pax_global_header00006660000000000000000000000064145616140540014520gustar00rootroot0000000000000052 comment=4322ad0c3a95e67c02f1204671565cd7b1a6e74b so-vits-svc-npu/000077500000000000000000000000001456161405400140615ustar00rootroot00000000000000so-vits-svc-npu/.gitattributes000066400000000000000000000000231456161405400167470ustar00rootroot00000000000000* text=auto eol=lf so-vits-svc-npu/.github/000077500000000000000000000000001456161405400154215ustar00rootroot00000000000000so-vits-svc-npu/.github/ISSUE_TEMPLATE/000077500000000000000000000000001456161405400176045ustar00rootroot00000000000000so-vits-svc-npu/.github/ISSUE_TEMPLATE/ask_for_help.yaml000066400000000000000000000074011456161405400231260ustar00rootroot00000000000000name: 请求帮助 description: 遇到了无法自行解决的错误 title: '[Help]: ' labels: [ "help wanted" ] body: - type: markdown attributes: value: | #### 提问前请先自己去尝试解决，比如查看[本仓库wiki](https://github.com/svc-develop-team/so-vits-svc/wiki)，也可以借助chatgpt或一些搜索引擎（谷歌/必应/New Bing/StackOverflow等等）。如果实在无法自己解决再发issue，在提issue之前，请先了解《[提问的智慧](https://github.com/ryanhanwu/How-To-Ask-Questions-The-Smart-Way/blob/main/README-zh_CN.md)》。 --- ### 什么样的issue会被直接close 1. 伸手党 2. 一键包/环境包相关 3. 提供的信息不全 4. 低级的如缺少依赖而导致无法运行的问题 4. 所用的数据集是无授权数据集(游戏角色/二次元人物暂不归为此类，但是训练时候也要小心谨慎。如果能联系到官方，必须先和官方联系并核实清楚) --- - type: checkboxes id: Clause attributes: label: 请勾选下方的确认框。 options: - label: "我已仔细阅读[README.md](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/README_zh_CN.md)和[wiki中的Quick solution](https://github.com/svc-develop-team/so-vits-svc/wiki/Quick-solution)。" required: true - label: "我已通过各种搜索引擎排查问题，我要提出的问题并不常见。" required: true - label: "我未在使用由第三方用户提供的一键包/环境包。" required: true - type: markdown attributes: value: | # 请根据实际使用环境填写以下信息 - type: input id: System attributes: label: 系统平台版本号 description: Windows执行`winver` | Linux执行`uname -a` validations: required: true - type: input id: GPU attributes: label: GPU 型号 description: 执行`nvidia-smi` validations: required: true - type: input id: PythonVersion attributes: label: Python版本 description: 执行`python -V` validations: required: true - type: input id: PyTorchVersion attributes: label: PyTorch版本 description: 执行`pip show torch` validations: required: true - type: dropdown id: Branch attributes: label: sovits分支 options: - 4.0(默认) - 4.0-v2 - 3.0-32k - 3.0-48k validations: required: true - type: input id: DatasetSource attributes: label: 数据集来源（用于判断数据集质量） description: 如：UVR处理过的vtb直播音频、录音棚录制 validations: required: true - type: input id: WhereOccurs attributes: label: 出现问题的环节或执行的命令 description: 如：预处理、训练、`python preprocess_hubert_f0.py` validations: required: true - type: textarea id: Description attributes: label: 问题描述 description: 在这里描述自己的问题，越详细越好 validations: required: true - type: textarea id: Log attributes: label: 日志 description: 将从执行命令到执行完毕输出的所有信息（包括你所执行的命令）粘贴到[pastebin.com](https://pastebin.com/)并把剪贴板链接贴到这里，日志量少的话也可以直接贴在下面 render: python validations: required: true - type: textarea id: ValidOneClick attributes: label: 截图`so-vits-svc`、`logs/44k`文件夹并粘贴到此处 validations: required: true - type: textarea id: Supplementary attributes: label: 补充说明 so-vits-svc-npu/.github/ISSUE_TEMPLATE/ask_for_help_en_US.yaml000066400000000000000000000077461456161405400242330ustar00rootroot00000000000000name: Ask for help description: Encountered an error cannot be resolved by self title: '[Help]: ' labels: [ "help wanted" ] body: - type: markdown attributes: value: | #### Please try to solve the problem yourself before asking for help. At first you can read *[repo wiki](https://github.com/svc-develop-team/so-vits-svc/wiki)*. Then you can use chatgpt or some search engines like google, bing, new bing and StackOverflow until you really find that you can't solve it by yourself. And before you raise an issue, please understand *[How To Ask Questions The Smart Way](http://www.catb.org/~esr/faqs/smart-questions.html)* in advance. --- ### What kind of issue will be closed immediately 1. Beggars or Free Riders 2. One click package / Environment package (Not using `pip install -r requirement.txt`) 3. Incomplete information 4. Stupid issues such as miss a dependency package 4. Using unlicenced dataset (Game characters / anime characters are not included in this category temporarily but you still need to pay attention. If you can contact the official, you must contact the official and verify it at first.) --- - type: checkboxes id: Clause attributes: label: Please check the checkboxes below. options: - label: "I have read *[README.md](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/README.md)* and *[Quick solution in wiki](https://github.com/svc-develop-team/so-vits-svc/wiki/Quick-solution)* carefully." required: true - label: "I have been troubleshooting issues through various search engines. The questions I want to ask are not common." required: true - label: "I am NOT using one click package / environment package." required: true - type: markdown attributes: value: | # Please fill in the following information according to your actual environment - type: input id: System attributes: label: OS version description: Windows run `winver` | Linux run `uname -a` validations: required: true - type: input id: GPU attributes: label: GPU description: Run `nvidia-smi` validations: required: true - type: input id: PythonVersion attributes: label: Python version description: Run `python -V` validations: required: true - type: input id: PyTorchVersion attributes: label: PyTorch version description: Run `pip show torch` validations: required: true - type: dropdown id: Branch attributes: label: Branch of sovits options: - 4.0(Default) - 4.0-v2 - 3.0-32k - 3.0-48k validations: required: true - type: input id: DatasetSource attributes: label: Dataset source (Used to judge the dataset quality) description: Such as UVR-processed streaming audio / Recorded in recording studio validations: required: true - type: input id: WhereOccurs attributes: label: Where thr problem occurs or what command you executed description: Such as Preprocessing / Training / `python preprocess_hubert_f0.py` validations: required: true - type: textarea id: Description attributes: label: Problem description description: Describe your problem here, the more detailed the better. validations: required: true - type: textarea id: Log attributes: label: Log description: All information output from the command you executed to the end of execution (include the command). It can also be directly posted below if there is only few text. render: python validations: required: true - type: textarea id: ValidOneClick attributes: label: Screenshot `so-vits-svc` and `logs/44k` folders and paste here validations: required: true - type: textarea id: Supplementary attributes: label: Supplementary description so-vits-svc-npu/.github/ISSUE_TEMPLATE/bug_report.yaml000066400000000000000000000041341456161405400226420ustar00rootroot00000000000000name: 问题回报 description: 遇到了BUG？！ title: '[Bug]: ' labels: [ "bug?" ] body: - type: markdown attributes: value: | # 请根据实际使用环境填写以下信息 - type: input id: System attributes: label: 系统平台版本号 description: Windows执行`winver` | Linux执行`uname -a` validations: required: true - type: input id: GPU attributes: label: GPU 型号 description: 执行`nvidia-smi` validations: required: true - type: input id: PythonVersion attributes: label: Python版本 description: 执行`python -V` validations: required: true - type: input id: PyTorchVersion attributes: label: PyTorch版本 description: 执行`pip show torch` validations: required: true - type: dropdown id: Branch attributes: label: sovits分支 options: - 4.0(默认) - 4.0-v2 - 3.0-32k - 3.0-48k validations: required: true - type: input id: DatasetSource attributes: label: 数据集来源（用于判断数据集质量） description: 如：UVR处理过的vtb直播音频、录音棚录制 validations: required: true - type: input id: WhereOccurs attributes: label: 出现问题的环节或执行的命令 description: 如：预处理、训练、`python preprocess_hubert_f0.py` validations: required: true - type: textarea id: Description attributes: label: 情况描述 description: 在这里描述遇到的情况，越详细越好 validations: required: true - type: textarea id: Log attributes: label: 日志 description: 将从执行命令到执行完毕输出的所有信息（包括你所执行的命令）粘贴到[pastebin.com](https://pastebin.com/)并把剪贴板链接贴到这里，日志量少的话也可以直接贴在下面 render: python validations: required: true - type: textarea id: Supplementary attributes: label: 补充说明 so-vits-svc-npu/.github/ISSUE_TEMPLATE/bug_report_en_US.yaml000066400000000000000000000043301456161405400237310ustar00rootroot00000000000000name: Bug report description: Encountered an bug?! title: '[Bug]: ' labels: [ "bug?" ] body: - type: markdown attributes: value: | # Please fill in the following information according to your actual environment - type: input id: System attributes: label: OS version description: Windows run `winver` | Linux run `uname -a` validations: required: true - type: input id: GPU attributes: label: GPU description: Run `nvidia-smi` validations: required: true - type: input id: PythonVersion attributes: label: Python version description: Run `python -V` validations: required: true - type: input id: PyTorchVersion attributes: label: PyTorch version description: Run `pip show torch` validations: required: true - type: dropdown id: Branch attributes: label: Branch of sovits options: - 4.0(Default) - 4.0-v2 - 3.0-32k - 3.0-48k validations: required: true - type: input id: DatasetSource attributes: label: Dataset source (Used to judge the dataset quality) description: Such as UVR-processed streaming audio / Recorded in recording studio validations: required: true - type: input id: WhereOccurs attributes: label: Where thr problem occurs or what command you executed description: Such as Preprocessing / Training / `python preprocess_hubert_f0.py` validations: required: true - type: textarea id: Description attributes: label: Situation description description: Describe your situation here, the more detailed the better. validations: required: true - type: textarea id: Log attributes: label: Log description: All information output from the command you executed to the end of execution (include the command). You can paste them to [pastebin.com](https://pastebin.com/) then paste the short link here. It can also be directly posted below if there is only few text. render: python validations: required: true - type: textarea id: Supplementary attributes: label: Supplementary description so-vits-svc-npu/.github/ISSUE_TEMPLATE/config.yml000066400000000000000000000005321456161405400215740ustar00rootroot00000000000000blank_issues_enabled: false contact_links: - name: 讨论区 / Discussions url: https://github.com/svc-develop-team/so-vits-svc/discussions about: 简单的询问/讨论请转至讨论区或发起一个低优先级的Default issue / For simple inquiries / discussions, please go to the discussions or raise a low priority Default issue so-vits-svc-npu/.github/ISSUE_TEMPLATE/default.md000066400000000000000000000005341456161405400215540ustar00rootroot00000000000000--- name: Default issue about: 如果模板中没有你想发起的issue类型，可以选择此项，但这个issue也许会获得一个较低的处理优先级 / If there is no issue type you want to raise, you can start with this one. But this issue maybe will get a lower priority to deal with. title: '' labels: 'not urgent' assignees: '' --- so-vits-svc-npu/.github/workflows/000077500000000000000000000000001456161405400174565ustar00rootroot00000000000000so-vits-svc-npu/.github/workflows/reviewdog.yml000066400000000000000000000005651456161405400222020ustar00rootroot00000000000000name: Ruff Autofix on: [pull_request] jobs: ruff: permissions: checks: write contents: read pull-requests: write runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: chartboost/ruff-action@v1 with: args: --fix -e - uses: reviewdog/action-suggester@v1 with: tool_name: ruff so-vits-svc-npu/.github/workflows/ruff.yml000066400000000000000000000002421456161405400211410ustar00rootroot00000000000000name: Ruff on: [push, pull_request] jobs: ruff: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: chartboost/ruff-action@v1 so-vits-svc-npu/.gitignore000066400000000000000000000050171456161405400160540ustar00rootroot00000000000000 # Created by https://www.toptal.com/developers/gitignore/api/python # Edit at https://www.toptal.com/developers/gitignore?templates=python ### Python ### # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so checkpoints/ # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ pytestdebug.log # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ doc/_build/ # PyBuilder target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # End of https://www.toptal.com/developers/gitignore/api/python /shelf/ /workspace.xml dataset dataset_raw raw results inference/chunks_temp.json logs hubert/checkpoint_best_legacy_500.pt configs/config.json filelists/test.txt filelists/train.txt filelists/val.txt .idea/ .vscode/ .idea/modules.xml .idea/so-vits-svc.iml .idea/vcs.xml .idea/inspectionProfiles/profiles_settings.xml .idea/inspectionProfiles/Project_Default.xml pretrain/ .vscode/launch.json trained/**/ configs/ filelists/* filelists/val.txt test_pipe.py test_queue.py so-vits-svc-npu/.ruff.toml000066400000000000000000000001451456161405400157760ustar00rootroot00000000000000select = ["E", "F", "I"] # Never enforce `E501` (line length violations). ignore = ["E501", "E741"] so-vits-svc-npu/0_preprocess_spec.py000066400000000000000000000036661456161405400200640ustar00rootroot00000000000000import argparse import os from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import librosa import mindtorch.torch as torch import mindtorch.torch.multiprocessing as mp from tqdm import tqdm import utils from modules.mel_processing import spectrogram_torch hps = utils.get_hparams_from_file("configs/config.json") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length def process_batch(file_chunk): for filename in tqdm(file_chunk): wav, sr = librosa.load(filename, sr=sampling_rate) audio_norm = torch.FloatTensor(wav) audio_norm = audio_norm.unsqueeze(0) spec_path = filename.replace(".wav", ".spec.pt") if not os.path.exists(spec_path): spec = spectrogram_torch(audio_norm, hps.data.filter_length, hps.data.sampling_rate, hps.data.hop_length, hps.data.win_length, center=False) spec = torch.squeeze(spec, 0) torch.save(spec, spec_path) def parallel_process(filenames, num_processes): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk)) for task in tasks: task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=20) args = parser.parse_args() filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) mp.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes) so-vits-svc-npu/1_preprocess_soft.py000066400000000000000000000035011456161405400200720ustar00rootroot00000000000000import argparse import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import librosa import mindtorch.torch as torch from tqdm import tqdm import utils hps = utils.get_hparams_from_file("configs/config.json") sampling_rate = hps.data.sampling_rate speech_encoder = hps["model"]["speech_encoder"] def process_batch(file_chunk): device = "npu:0" hmodel = utils.get_speech_encoder(speech_encoder, device=device) for filename in tqdm(file_chunk): wav, sr = librosa.load(filename, sr=sampling_rate) soft_path = filename + ".soft.pt" wav16k = librosa.resample(wav, orig_sr=sr, target_sr=16000) wav16k = torch.from_numpy(wav16k).to(device) c = hmodel.encoder(wav16k) torch.save(c.cpu(), soft_path) def parallel_process(filenames, num_processes): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk)) for task in tasks: task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=1) args = parser.parse_args() filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) multiprocessing.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes)so-vits-svc-npu/2_preprocess_vol.py000066400000000000000000000035401456161405400177230ustar00rootroot00000000000000import argparse import logging import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import librosa import numpy as np import mindtorch.torch as torch from tqdm import tqdm import utils hps = utils.get_hparams_from_file("configs/config.json") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length def process_batch(file_chunk): for filename in tqdm(file_chunk): wav, sr = librosa.load(filename, sr=sampling_rate) audio_norm = torch.FloatTensor(wav) audio_norm = audio_norm.unsqueeze(0) volume_path = filename + ".vol.npy" volume_extractor = utils.Volume_Extractor(hop_length) if not os.path.exists(volume_path): volume = volume_extractor.extract(audio_norm) np.save(volume_path, volume.to('cpu').numpy()) def parallel_process(filenames, num_processes): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk)) for task in tqdm(tasks): task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=1) args = parser.parse_args() filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) multiprocessing.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes)so-vits-svc-npu/3_preprocess_mel.py000066400000000000000000000042451456161405400177040ustar00rootroot00000000000000import argparse import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import librosa import mindtorch.torch as torch from tqdm import tqdm import diffusion.logger.utils as du import utils from diffusion.vocoder import Vocoder hps = utils.get_hparams_from_file("configs/config.json") dconfig = du.load_config("configs/diffusion.yaml") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length def process_batch(file_chunk, mel_extractor=None): for filename in tqdm(file_chunk): wav, sr = librosa.load(filename, sr=sampling_rate) audio_norm = torch.FloatTensor(wav) audio_norm = audio_norm.unsqueeze(0) device = torch.device("npu:0") mel_path = filename + ".mel.pt" if not os.path.exists(mel_path) and mel_extractor is not None: mel_t = mel_extractor.extract(audio_norm.to(device), sampling_rate) mel = mel_t.squeeze().to('cpu') torch.save(mel, mel_path) def parallel_process(filenames, num_processes, mel_extractor): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk, mel_extractor)) for task in tqdm(tasks): task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=1) args = parser.parse_args() device = torch.device("npu:0") mel_extractor = Vocoder(dconfig.vocoder.type, dconfig.vocoder.ckpt, device=device) filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) multiprocessing.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes, mel_extractor)so-vits-svc-npu/4_preprocess_diff.py000066400000000000000000000054201456161405400200340ustar00rootroot00000000000000import argparse import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import random import librosa import numpy as np import mindtorch.torch as torch from tqdm import tqdm import diffusion.logger.utils as du import utils from diffusion.vocoder import Vocoder hps = utils.get_hparams_from_file("configs/config.json") dconfig = du.load_config("configs/diffusion.yaml") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length def process_batch(file_chunk, mel_extractor=None): for filename in tqdm(file_chunk): wav, sr = librosa.load(filename, sr=sampling_rate) audio_norm = torch.FloatTensor(wav) audio_norm = audio_norm.unsqueeze(0) volume_extractor = utils.Volume_Extractor(hop_length) aug_mel_path = filename + ".aug_mel.npy" aug_vol_path = filename + ".aug_vol.npy" max_amp = float(torch.max(torch.abs(audio_norm))) + 1e-5 max_shift = min(1, np.log10(1/max_amp)) log10_vol_shift = random.uniform(-1, max_shift) keyshift = random.uniform(-5, 5) if mel_extractor is not None: aug_mel_t = mel_extractor.extract(audio_norm * (10 ** log10_vol_shift), sampling_rate, keyshift = keyshift) aug_mel = aug_mel_t.squeeze().to('cpu').numpy() aug_vol = volume_extractor.extract(audio_norm * (10 ** log10_vol_shift)) if not os.path.exists(aug_mel_path): np.save(aug_mel_path,np.asanyarray((aug_mel,keyshift),dtype=object)) if not os.path.exists(aug_vol_path): np.save(aug_vol_path,aug_vol.to('cpu').numpy()) def parallel_process(filenames, num_processes, mel_extractor): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk, mel_extractor)) for task in tqdm(tasks): task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=1) args = parser.parse_args() device = torch.device("npu:0") mel_extractor = Vocoder(dconfig.vocoder.type, dconfig.vocoder.ckpt, device = device) filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) multiprocessing.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes, mel_extractor)so-vits-svc-npu/5_preprocess_f0.py000066400000000000000000000033731456161405400174370ustar00rootroot00000000000000import argparse import os from random import shuffle import librosa import numpy as np from tqdm import tqdm import utils from concurrent.futures import ProcessPoolExecutor from glob import glob hps = utils.get_hparams_from_file("configs/config.json") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length def process_batch(filenames, f0p): for filename in tqdm(filenames): wav, sr = librosa.load(filename, sr=sampling_rate) f0_path = filename + ".f0.npy" f0_predictor = utils.get_f0_predictor(f0p,sampling_rate=sampling_rate, hop_length=hop_length,device=None,threshold=0.05) f0,uv = f0_predictor.compute_f0_uv(wav) np.save(f0_path, np.asanyarray((f0,uv),dtype=object)) def parallel_process(filenames, f0p, num_processes): with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] tasks.append(executor.submit(process_batch, file_chunk, f0p)) for task in tasks: task.result() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--in_dir", type=str, default="dataset/44k") parser.add_argument('--num_processes', type=int, default=12) parser.add_argument('--f0_predictor', type=str, default="rmvpe") args = parser.parse_args() f0p = args.f0_predictor filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) shuffle(filenames) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, f0p, num_processes)so-vits-svc-npu/LICENSE000066400000000000000000001033331456161405400150710ustar00rootroot00000000000000 GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU Affero General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Remote Network Interaction; Use with the GNU General Public License. Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements. You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see . so-vits-svc-npu/README.md000066400000000000000000000026361456161405400153470ustar00rootroot00000000000000# sovits-npu-mindtorch ## 原始readme [**English**](./README_sovits.md) | [**中文简体**](./README_zh_CN.md) ### 开机 选择`mindtorch0.2_mindspore2.2.1_torchnpu2.1.0_cann7.0rc1`镜像 数据集勾选上`so-vits-svc-npu.zip`和你的数据集 ### 解压程序、数据集&安装依赖&解决乱七八糟的问题 ```shell su pip uninstall mindtorch -y git clone https://git.openi.org.cn/OpenI/MSAdapter.git cd MSAdapter python setup.py install cd .. conda install -c conda-forge libsndfile -y conda install ffmpeg -y export LD_PRELOAD=/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/faiss_cpu.libs/libgomp-d22c30c5.so.1.0.0:/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/torch.libs/libgomp-6e1a1d1b.so.1.0.0 ``` ps.运行以下代码前，修改`dataset.zip`为文件实际名称 ```shell cp /dataset/so-vits-svc-npu.zip ./ unzip so-vits-svc-npu.zip rm -f so-vits-svc-npu.zip cp /dataset/dataset.zip ./so-vits-svc-npu/dataset_raw cd ./so-vits-svc-npu/dataset_raw unzip dataset.zip rm -f dataset.zip cd .. pip install -r requirements.txt ``` ### 重采样至44.1k ```shell bash resample.sh ``` ### 创建filelists ```shell python3 preprocess_flist_config.py --speech_encoder vec768l12 --vol_aug ``` ### 生成hubert和f0 ```shell python3 preprocess_hubert_f0.py --f0_predictor rmvpe --num_processes 2 --use_diff ``` ### 训练 ```shell python3 train.py -c configs/config.json -m 44k ```so-vits-svc-npu/README_sovits.md000066400000000000000000001116211456161405400167510ustar00rootroot00000000000000
LOGO # SoftVC VITS Singing Voice Conversion [**English**](./README_sovits.md) | [**中文简体**](./README_zh_CN.md) [![Open In Colab](https://img.shields.io/badge/Colab-F9AB00?style=for-the-badge&logo=googlecolab&color=525252)](https://colab.research.google.com/github/svc-develop-team/so-vits-svc/blob/4.1-Stable/sovits4_for_colab.ipynb) [![Licence](https://img.shields.io/badge/LICENSE-AGPL3.0-green.svg?style=for-the-badge)](https://github.com/svc-develop-team/so-vits-svc/blob/4.1-Stable/LICENSE) This round of limited time update is coming to an end, the warehouse will enter the Archieve state, please know
> ✨ A studio that contains visible f0 editor, speaker mix timeline editor and other features (Where the Onnx models are used) : [MoeVoiceStudio](https://github.com/NaruseMioShirakana/MoeVoiceStudio) > ✨ A fork with a greatly improved user interface: [34j/so-vits-svc-fork](https://github.com/34j/so-vits-svc-fork) > ✨ A client supports real-time conversion: [w-okada/voice-changer](https://github.com/w-okada/voice-changer) **This project differs fundamentally from VITS, as it focuses on Singing Voice Conversion (SVC) rather than Text-to-Speech (TTS). In this project, TTS functionality is not supported, and VITS is incapable of performing SVC tasks. It's important to note that the models used in these two projects are not interchangeable or universally applicable.** ## Announcement The purpose of this project was to enable developers to have their beloved anime characters perform singing tasks. The developers' intention was to focus solely on fictional characters and avoid any involvement of real individuals, anything related to real individuals deviates from the developer's original intention. ## Disclaimer This project is an open-source, offline endeavor, and all members of SvcDevelopTeam, as well as other developers and maintainers involved (hereinafter referred to as contributors), have no control over the project. The contributors have never provided any form of assistance to any organization or individual, including but not limited to dataset extraction, dataset processing, computing support, training support, inference, and so on. The contributors do not and cannot be aware of the purposes for which users utilize the project. Therefore, any AI models and synthesized audio produced through the training of this project are unrelated to the contributors. Any issues or consequences arising from their use are the sole responsibility of the user. This project is run completely offline and does not collect any user information or gather user input data. Therefore, contributors to this project are not aware of all user input and models and therefore are not responsible for any user input. This project serves as a framework only and does not possess speech synthesis functionality by itself. All functionalities require users to train the models independently. Furthermore, this project does not come bundled with any models, and any secondary distributed projects are independent of the contributors of this project. ## 📏 Terms of Use # Warning: Please ensure that you address any authorization issues related to the dataset on your own. You bear full responsibility for any problems arising from the usage of non-authorized datasets for training, as well as any resulting consequences. The repository and its maintainer, svc develop team, disclaim any association with or liability for the consequences. 1. This project is exclusively established for academic purposes, aiming to facilitate communication and learning. It is not intended for deployment in production environments. 2. Any sovits-based video posted to a video platform must clearly specify in the introduction the input source vocals and audio used for the voice changer conversion, e.g., if you use someone else's video/audio and convert it by separating the vocals as the input source, you must give a clear link to the original video or music; if you use your own vocals or a voice synthesized by another voice synthesis engine as the input source, you must also state this in your introduction. 3. You are solely responsible for any infringement issues caused by the input source and all consequences. When using other commercial vocal synthesis software as an input source, please ensure that you comply with the regulations of that software, noting that the regulations of many vocal synthesis engines explicitly state that they cannot be used to convert input sources! 4. Engaging in illegal activities, as well as religious and political activities, is strictly prohibited when using this project. The project developers vehemently oppose the aforementioned activities. If you disagree with this provision, the usage of the project is prohibited. 5. If you continue to use the program, you will be deemed to have agreed to the terms and conditions set forth in README and README has discouraged you and is not responsible for any subsequent problems. 6. If you intend to employ this project for any other purposes, kindly contact and inform the maintainers of this repository in advance. ## 📝 Model Introduction The singing voice conversion model uses SoftVC content encoder to extract speech features from the source audio. These feature vectors are directly fed into VITS without the need for conversion to a text-based intermediate representation. As a result, the pitch and intonations of the original audio are preserved. Meanwhile, the vocoder was replaced with [NSF HiFiGAN](https://github.com/openvpi/DiffSinger/tree/refactor/modules/nsf_hifigan) to solve the problem of sound interruption. ### 🆕 4.1-Stable Version Update Content - Feature input is changed to the 12th Layer of [Content Vec](https://github.com/auspicious3000/contentvec) Transformer output, And compatible with 4.0 branches. - Update the shallow diffusion, you can use the shallow diffusion model to improve the sound quality. - Added Whisper-PPG encoder support - Added static/dynamic sound fusion - Added loudness embedding - Added Functionality of feature retrieval from [RVC](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) ### 🆕 Questions about compatibility with the 4.0 model - To support the 4.0 model and incorporate the speech encoder, you can make modifications to the `config.json` file. Add the `speech_encoder` field to the "model" section as shown below: ``` "model": { ......... "ssl_dim": 256, "n_speakers": 200, "speech_encoder":"vec256l9" } ``` ### 🆕 Shallow diffusion ![Diagram](shadowdiffusion.png) ## 💬 Python Version Based on our testing, we have determined that the project runs stable on `Python 3.8.9`. ## 📥 Pre-trained Model Files #### **Required** **You need to select one encoder from the list below** ##### **1. If using contentvec as speech encoder(recommended)** `vec768l12` and `vec256l9` require the encoder - ContentVec: [checkpoint_best_legacy_500.pt](https://ibm.box.com/s/z1wgl1stco8ffooyatzdwsqn2psd9lrr) - Place it under the `pretrain` directory Or download the following ContentVec, which is only 199MB in size but has the same effect: - ContentVec: [hubert_base.pt](https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt) - Change the file name to `checkpoint_best_legacy_500.pt` and place it in the `pretrain` directory ```shell # contentvec wget -P pretrain/ https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt -O checkpoint_best_legacy_500.pt # Alternatively, you can manually download and place it in the hubert directory ``` ##### **2. If hubertsoft is used as the speech encoder** - soft vc hubert: [hubert-soft-0d54a1f4.pt](https://github.com/bshall/hubert/releases/download/v0.1/hubert-soft-0d54a1f4.pt) - Place it under the `pretrain` directory ##### **3. If whisper-ppg as the encoder** - download model at [medium.pt](https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt), the model fits `whisper-ppg` - or download model at [large-v2.pt](https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt), the model fits `whisper-ppg-large` - Place it under the `pretrain` directory ##### **4. If cnhubertlarge as the encoder** - download model at [chinese-hubert-large-fairseq-ckpt.pt](https://huggingface.co/TencentGameMate/chinese-hubert-large/resolve/main/chinese-hubert-large-fairseq-ckpt.pt) - Place it under the `pretrain` directory ##### **5. If dphubert as the encoder** - download model at [DPHuBERT-sp0.75.pth](https://huggingface.co/pyf98/DPHuBERT/resolve/main/DPHuBERT-sp0.75.pth) - Place it under the `pretrain` directory ##### **6. If WavLM is used as the encoder** - download model at [WavLM-Base+.pt](https://valle.blob.core.windows.net/share/wavlm/WavLM-Base+.pt?sv=2020-08-04&st=2023-03-01T07%3A51%3A05Z&se=2033-03-02T07%3A51%3A00Z&sr=c&sp=rl&sig=QJXmSJG9DbMKf48UDIU1MfzIro8HQOf3sqlNXiflY1I%3D), the model fits `wavlmbase+` - Place it under the `pretrain` directory ##### **7. If OnnxHubert/ContentVec as the encoder** - download model at [MoeSS-SUBModel](https://huggingface.co/NaruseMioShirakana/MoeSS-SUBModel/tree/main) - Place it under the `pretrain` directory #### **List of Encoders** - "vec768l12" - "vec256l9" - "vec256l9-onnx" - "vec256l12-onnx" - "vec768l9-onnx" - "vec768l12-onnx" - "hubertsoft-onnx" - "hubertsoft" - "whisper-ppg" - "cnhubertlarge" - "dphubert" - "whisper-ppg-large" - "wavlmbase+" #### **Optional(Strongly recommend)** - Pre-trained model files: `G_0.pth` `D_0.pth` - Place them under the `logs/44k` directory - Diffusion model pretraining base model file: `model_0.pt` - Put it in the `logs/44k/diffusion` directory Get Sovits Pre-trained model from svc-develop-team(TBD) or anywhere else. Diffusion model references [Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) diffusion model. The pre-trained diffusion model is universal with the DDSP-SVC's. You can go to [Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC)'s repo to get the pre-trained diffusion model. While the pretrained model typically does not pose copyright concerns, it is essential to remain vigilant. It is advisable to consult with the author beforehand or carefully review the description to ascertain the permissible usage of the model. This helps ensure compliance with any specified guidelines or restrictions regarding its utilization. #### **Optional(Select as Required)** ##### NSF-HIFIGAN If you are using the `NSF-HIFIGAN enhancer` or `shallow diffusion`, you will need to download the pre-trained NSF-HIFIGAN model. - Pre-trained NSF-HIFIGAN Vocoder: [nsf_hifigan_20221211.zip](https://github.com/openvpi/vocoders/releases/download/nsf-hifigan-v1/nsf_hifigan_20221211.zip) - Unzip and place the four files under the `pretrain/nsf_hifigan` directory ```shell # nsf_hifigan wget -P pretrain/ https://github.com/openvpi/vocoders/releases/download/nsf-hifigan-v1/nsf_hifigan_20221211.zip unzip -od pretrain/nsf_hifigan pretrain/nsf_hifigan_20221211.zip # Alternatively, you can manually download and place it in the pretrain/nsf_hifigan directory # URL: https://github.com/openvpi/vocoders/releases/tag/nsf-hifigan-v1 ``` ##### RMVPE If you are using the `rmvpe` F0 Predictor, you will need to download the pre-trained RMVPE model. + download model at [rmvpe.zip](https://github.com/yxlllc/RMVPE/releases/download/230917/rmvpe.zip), this weight is recommended. + unzip `rmvpe.zip`，and rename the `model.pt` file to `rmvpe.pt` and place it under the `pretrain` directory. - ~~download model at [rmvpe.pt](https://huggingface.co/datasets/ylzz1997/rmvpe_pretrain_model/resolve/main/rmvpe.pt)~~ - ~~Place it under the `pretrain` directory~~ ##### FCPE(Preview version) [FCPE(Fast Context-base Pitch Estimator)](https://github.com/CNChTu/MelPE) is a dedicated F0 predictor designed for real-time voice conversion and will become the preferred F0 predictor for sovits real-time voice conversion in the future.(The paper is being written) If you are using the `fcpe` F0 Predictor, you will need to download the pre-trained FCPE model. - download model at [fcpe.pt](https://huggingface.co/datasets/ylzz1997/rmvpe_pretrain_model/resolve/main/fcpe.pt) - Place it under the `pretrain` directory ## 📊 Dataset Preparation Simply place the dataset in the `dataset_raw` directory with the following file structure: ``` dataset_raw ├───speaker0 │ ├───xxx1-xxx1.wav │ ├───... │ └───Lxx-0xx8.wav └───speaker1 ├───xx2-0xxx2.wav ├───... └───xxx7-xxx007.wav ``` There are no specific restrictions on the format of the name for each audio file (naming conventions such as `000001.wav` to `999999.wav` are also valid), but the file type must be `WAV``. You can customize the speaker's name as showed below: ``` dataset_raw └───suijiSUI ├───1.wav ├───... └───25788785-20221210-200143-856_01_(Vocals)_0_0.wav ``` ## 🛠️ Preprocessing ### 0. Slice audio To avoid video memory overflow during training or pre-processing, it is recommended to limit the length of audio clips. Cutting the audio to a length of "5s - 15s" is more recommended. Slightly longer times are acceptable, however, excessively long clips may cause problems such as `torch.cuda.OutOfMemoryError`. To facilitate the slicing process, you can use [audio-slicer-GUI](https://github.com/flutydeer/audio-slicer) or [audio-slicer-CLI](https://github.com/openvpi/audio-slicer) In general, only the `Minimum Interval` needs to be adjusted. For spoken audio, the default value usually suffices, while for singing audio, it can be adjusted to around `100` or even `50`, depending on the specific requirements. After slicing, it is recommended to remove any audio clips that are excessively long or too short. **If you are using whisper-ppg encoder for training, the audio clips must shorter than 30s.** ### 1. Resample to 44100Hz and mono ```shell python resample.py ``` #### Cautions Although this project has resample.py scripts for resampling, mono and loudness matching, the default loudness matching is to match to 0db. This can cause damage to the sound quality. While python's loudness matching package pyloudnorm does not limit the level, this can lead to sonic boom. Therefore, it is recommended to consider using professional sound processing software, such as `adobe audition` for loudness matching. If you are already using other software for loudness matching, add the parameter `-skip_loudnorm` to the run command: ```shell python resample.py --skip_loudnorm ``` ### 2. Automatically split the dataset into training and validation sets, and generate configuration files. ```shell python preprocess_flist_config.py --speech_encoder vec768l12 ``` speech_encoder has the following options ``` vec768l12 vec256l9 hubertsoft whisper-ppg cnhubertlarge dphubert whisper-ppg-large wavlmbase+ ``` If the speech_encoder argument is omitted, the default value is `vec768l12` **Use loudness embedding** Add `--vol_aug` if you want to enable loudness embedding: ```shell python preprocess_flist_config.py --speech_encoder vec768l12 --vol_aug ``` After enabling loudness embedding, the trained model will match the loudness of the input source; otherwise, it will match the loudness of the training set. #### You can modify some parameters in the generated config.json and diffusion.yaml * `keep_ckpts`: Keep the the the number of previous models during training. Set to `0` to keep them all. Default is `3`. * `all_in_mem`: Load all dataset to RAM. It can be enabled when the disk IO of some platforms is too low and the system memory is **much larger** than your dataset. * `batch_size`: The amount of data loaded to the GPU for a single training session can be adjusted to a size lower than the GPU memory capacity. * `vocoder_name`: Select a vocoder. The default is `nsf-hifigan`. ##### diffusion.yaml * `cache_all_data`: Load all dataset to RAM. It can be enabled when the disk IO of some platforms is too low and the system memory is **much larger** than your dataset. * `duration`: The duration of the audio slicing during training, can be adjusted according to the size of the video memory, **Note: this value must be less than the minimum time of the audio in the training set!** * `batch_size`: The amount of data loaded to the GPU for a single training session can be adjusted to a size lower than the video memory capacity. * `timesteps`: The total number of steps in the diffusion model, which defaults to 1000. * `k_step_max`: Training can only train `k_step_max` step diffusion to save training time, note that the value must be less than `timesteps`, 0 is to train the entire diffusion model, **Note: if you do not train the entire diffusion model will not be able to use only_diffusion!** ##### **List of Vocoders** ``` nsf-hifigan nsf-snake-hifigan ``` ### 3. Generate hubert and f0 ```shell python preprocess_hubert_f0.py --f0_predictor dio ``` f0_predictor has the following options ``` crepe dio pm harvest rmvpe fcpe ``` If the training set is too noisy,it is recommended to use `crepe` to handle f0 If the f0_predictor parameter is omitted, the default value is `rmvpe` If you want shallow diffusion (optional), you need to add the `--use_diff` parameter, for example: ```shell python preprocess_hubert_f0.py --f0_predictor dio --use_diff ``` **Speed Up preprocess** If your dataset is pretty large,you can increase the param `--num_processes` like that: ```shell python preprocess_hubert_f0.py --f0_predictor dio --num_processes 8 ``` All the worker will be assigned to different GPU if you have more than one GPUs. After completing the above steps, the dataset directory will contain the preprocessed data, and the dataset_raw folder can be deleted. ## 🏋️‍ Training ### Sovits Model ```shell python train.py -c configs/config.json -m 44k ``` Here is a method that allows you to pause training and save the model at any time. Simply create a `stop.txt` file in the running directory, and the training will stop and save the model after the next step. ### Diffusion Model (optional) If the shallow diffusion function is needed, the diffusion model needs to be trained. The diffusion model training method is as follows: ```shell python train_diff.py -c configs/diffusion.yaml ``` During training, the model files will be saved to `logs/44k`, and the diffusion model will be saved to `logs/44k/diffusion` ## 🤖 Inference Use [inference_main.py](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/inference_main.py) ```shell # Example python inference_main.py -m "logs/44k/G_30400.pth" -c "configs/config.json" -n "君の知らない物語-src.wav" -t 0 -s "nen" ``` Required parameters: - `-m` | `--model_path`: path to the model. - `-c` | `--config_path`: path to the configuration file. - `-n` | `--clean_names`: a list of wav file names located in the `raw` folder. - `-t` | `--trans`: pitch shift, supports positive and negative (semitone) values. - `-s` | `--spk_list`: Select the speaker ID to use for conversion. - `-cl` | `--clip`: Forced audio clipping, set to 0 to disable(default), setting it to a non-zero value (duration in seconds) to enable. Optional parameters: see the next section - `-lg` | `--linear_gradient`: The cross fade length of two audio slices in seconds. If there is a discontinuous voice after forced slicing, you can adjust this value. Otherwise, it is recommended to use the default value of 0. - `-f0p` | `--f0_predictor`: Select a F0 predictor, options are `crepe`, `pm`, `dio`, `harvest`, `rmvpe`,`fcpe`, default value is `pm`(note: f0 mean pooling will be enable when using `crepe`) - `-a` | `--auto_predict_f0`: automatic pitch prediction, do not enable this when converting singing voices as it can cause serious pitch issues. - `-cm` | `--cluster_model_path`: Cluster model or feature retrieval index path, if left blank, it will be automatically set as the default path of these models. If there is no training cluster or feature retrieval, fill in at will. - `-cr` | `--cluster_infer_ratio`: The proportion of clustering scheme or feature retrieval ranges from 0 to 1. If there is no training clustering model or feature retrieval, the default is 0. - `-eh` | `--enhance`: Whether to use NSF_HIFIGAN enhancer, this option has certain effect on sound quality enhancement for some models with few training sets, but has negative effect on well-trained models, so it is disabled by default. - `-shd` | `--shallow_diffusion`: Whether to use shallow diffusion, which can solve some electrical sound problems after use. This option is disabled by default. When this option is enabled, NSF_HIFIGAN enhancer will be disabled - `-usm` | `--use_spk_mix`: whether to use dynamic voice fusion - `-lea` | `--loudness_envelope_adjustment`：The adjustment of the input source's loudness envelope in relation to the fusion ratio of the output loudness envelope. The closer to 1, the more the output loudness envelope is used - `-fr` | `--feature_retrieval`：Whether to use feature retrieval If clustering model is used, it will be disabled, and `cm` and `cr` parameters will become the index path and mixing ratio of feature retrieval Shallow diffusion settings: - `-dm` | `--diffusion_model_path`: Diffusion model path - `-dc` | `--diffusion_config_path`: Diffusion config file path - `-ks` | `--k_step`: The larger the number of k_steps, the closer it is to the result of the diffusion model. The default is 100 - `-od` | `--only_diffusion`: Whether to use Only diffusion mode, which does not load the sovits model to only use diffusion model inference - `-se` | `--second_encoding`：which involves applying an additional encoding to the original audio before shallow diffusion. This option can yield varying results - sometimes positive and sometimes negative. ### Cautions If inferencing using `whisper-ppg` speech encoder, you need to set `--clip` to 25 and `-lg` to 1. Otherwise it will fail to infer properly. ## 🤔 Optional Settings If you are satisfied with the previous results, or if you do not feel you understand what follows, you can skip it and it will have no effect on the use of the model. The impact of these optional settings mentioned is relatively small, and while they may have some impact on specific datasets, in most cases the difference may not be significant. ### Automatic f0 prediction During the training of the 4.0 model, an f0 predictor is also trained, which enables automatic pitch prediction during voice conversion. However, if the results are not satisfactory, manual pitch prediction can be used instead. Please note that when converting singing voices, it is advised not to enable this feature as it may cause significant pitch shifting. - Set `auto_predict_f0` to `true` in `inference_main.py`. ### Cluster-based timbre leakage control Introduction: The clustering scheme implemented in this model aims to reduce timbre leakage and enhance the similarity of the trained model to the target's timbre, although the effect may not be very pronounced. However, relying solely on clustering can reduce the model's clarity and make it sound less distinct. Therefore, a fusion method is adopted in this model to control the balance between the clustering and non-clustering approaches. This allows manual adjustment of the trade-off between "sounding like the target's timbre" and "have clear enunciation" to find an optimal balance. No changes are required in the existing steps. Simply train an additional clustering model, which incurs relatively low training costs. - Training process: - Train on a machine with good CPU performance. According to extant experience, it takes about 4 minutes to train each speaker on a Tencent Cloud machine with 6-core CPU. - Execute `python cluster/train_cluster.py`. The output model will be saved in `logs/44k/kmeans_10000.pt`. - The clustering model can currently be trained using the gpu by executing `python cluster/train_cluster.py --gpu` - Inference process: - Specify `cluster_model_path` in `inference_main.py`. If not specified, the default is `logs/44k/kmeans_10000.pt`. - Specify `cluster_infer_ratio` in `inference_main.py`, where `0` means not using clustering at all, `1` means only using clustering, and usually `0.5` is sufficient. ### Feature retrieval Introduction: As with the clustering scheme, the timbre leakage can be reduced, the enunciation is slightly better than clustering, but it will reduce the inference speed. By employing the fusion method, it becomes possible to linearly control the balance between feature retrieval and non-feature retrieval, allowing for fine-tuning of the desired proportion. - Training process: First, it needs to be executed after generating hubert and f0: ```shell python train_index.py -c configs/config.json ``` The output of the model will be in `logs/44k/feature_and_index.pkl` - Inference process: - The `--feature_retrieval` needs to be formulated first, and the clustering mode automatically switches to the feature retrieval mode. - Specify `cluster_model_path` in `inference_main.py`. If not specified, the default is `logs/44k/feature_and_index.pkl`. - Specify `cluster_infer_ratio` in `inference_main.py`, where `0` means not using feature retrieval at all, `1` means only using feature retrieval, and usually `0.5` is sufficient. ## 🗜️ Model compression The generated model contains data that is needed for further training. If you confirm that the model is final and not be used in further training, it is safe to remove these data to get smaller file size (about 1/3). ```shell # Example python compress_model.py -c="configs/config.json" -i="logs/44k/G_30400.pth" -o="logs/44k/release.pth" ``` ## 👨‍🔧 Timbre mixing ### Static Tone Mixing **Refer to `webUI.py` file for stable Timbre mixing of the gadget/lab feature.** Introduction: This function can combine multiple models into one model (convex combination or linear combination of multiple model parameters) to create mixed voice that do not exist in reality **Note:** 1. This feature is only supported for single-speaker models 2. If you force a multi-speaker model, it is critical to make sure there are the same number of speakers in each model. This will ensure that sounds with the same SpeakerID can be mixed correctly. 3. Ensure that the `model` fields in config.json of all models to be mixed are the same 4. The mixed model can use any config.json file from the models being synthesized. However, the clustering model will not be functional after mixed. 5. When batch uploading models, it is best to put the models into a folder and upload them together after selecting them 6. It is suggested to adjust the mixing ratio between 0 and 100, or to other numbers, but unknown effects will occur in the linear combination mode 7. After mixing, the file named output.pth will be saved in the root directory of the project 8. Convex combination mode will perform Softmax to add the mix ratio to 1, while linear combination mode will not ### Dynamic timbre mixing **Refer to the `spkmix.py` file for an introduction to dynamic timbre mixing** Character mix track writing rules: Role ID: \[\[Start time 1, end time 1, start value 1, start value 1], [Start time 2, end time 2, start value 2]] The start time must be the same as the end time of the previous one. The first start time must be 0, and the last end time must be 1 (time ranges from 0 to 1). All roles must be filled in. For unused roles, fill \[\[0., 1., 0., 0.]] The fusion value can be filled in arbitrarily, and the linear change from the start value to the end value within the specified period of time. The internal linear combination will be automatically guaranteed to be 1 (convex combination condition), so it can be used safely Use the `--use_spk_mix` parameter when reasoning to enable dynamic timbre mixing ## 📤 Exporting to Onnx Use [onnx_export.py](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/onnx_export.py) - Create a folder named `checkpoints` and open it - Create a folder in the `checkpoints` folder as your project folder, naming it after your project, for example `aziplayer` - Rename your model as `model.pth`, the configuration file as `config.json`, and place them in the `aziplayer` folder you just created - Modify `"NyaruTaffy"` in `path = "NyaruTaffy"` in [onnx_export.py](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/onnx_export.py) to your project name, `path = "aziplayer"`（onnx_export_speaker_mix makes you can mix speaker's voice） - Run [onnx_export.py](https://github.com/svc-develop-team/so-vits-svc/blob/4.0/onnx_export.py) - Wait for it to finish running. A `model.onnx` will be generated in your project folder, which is the exported model. Note: For Hubert Onnx models, please use the models provided by MoeSS. Currently, they cannot be exported on their own (Hubert in fairseq has many unsupported operators and things involving constants that can cause errors or result in problems with the input/output shape and results when exported.) ## 📎 Reference | URL | Designation | Title | Implementation Source | | --- | ----------- | ----- | --------------------- | |[2106.06103](https://arxiv.org/abs/2106.06103) | VITS (Synthesizer)| Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech | [jaywalnut310/vits](https://github.com/jaywalnut310/vits) | |[2111.02392](https://arxiv.org/abs/2111.02392) | SoftVC (Speech Encoder)| A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion | [bshall/hubert](https://github.com/bshall/hubert) | |[2204.09224](https://arxiv.org/abs/2204.09224) | ContentVec (Speech Encoder)| ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers | [auspicious3000/contentvec](https://github.com/auspicious3000/contentvec) | |[2212.04356](https://arxiv.org/abs/2212.04356) | Whisper (Speech Encoder) | Robust Speech Recognition via Large-Scale Weak Supervision | [openai/whisper](https://github.com/openai/whisper) | |[2110.13900](https://arxiv.org/abs/2110.13900) | WavLM (Speech Encoder) | WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing | [microsoft/unilm/wavlm](https://github.com/microsoft/unilm/tree/master/wavlm) | |[2305.17651](https://arxiv.org/abs/2305.17651) | DPHubert (Speech Encoder) | DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models | [pyf98/DPHuBERT](https://github.com/pyf98/DPHuBERT) | |[DOI:10.21437/Interspeech.2017-68](http://dx.doi.org/10.21437/Interspeech.2017-68) | Harvest (F0 Predictor) | Harvest: A high-performance fundamental frequency estimator from speech signals | [mmorise/World/harvest](https://github.com/mmorise/World/blob/master/src/harvest.cpp) | |[aes35-000039](https://www.aes.org/e-lib/online/browse.cfm?elib=15165) | Dio (F0 Predictor) | Fast and reliable F0 estimation method based on the period extraction of vocal fold vibration of singing voice and speech | [mmorise/World/dio](https://github.com/mmorise/World/blob/master/src/dio.cpp) | |[8461329](https://ieeexplore.ieee.org/document/8461329) | Crepe (F0 Predictor) | Crepe: A Convolutional Representation for Pitch Estimation | [maxrmorrison/torchcrepe](https://github.com/maxrmorrison/torchcrepe) | |[DOI:10.1016/j.wocn.2018.07.001](https://doi.org/10.1016/j.wocn.2018.07.001) | Parselmouth (F0 Predictor) | Introducing Parselmouth: A Python interface to Praat | [YannickJadoul/Parselmouth](https://github.com/YannickJadoul/Parselmouth) | |[2306.15412v2](https://arxiv.org/abs/2306.15412v2) | RMVPE (F0 Predictor) | RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music | [Dream-High/RMVPE](https://github.com/Dream-High/RMVPE) | |[2010.05646](https://arxiv.org/abs/2010.05646) | HIFIGAN (Vocoder) | HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis | [jik876/hifi-gan](https://github.com/jik876/hifi-gan) | |[1810.11946](https://arxiv.org/abs/1810.11946.pdf) | NSF (Vocoder) | Neural source-filter-based waveform model for statistical parametric speech synthesis | [openvpi/DiffSinger/modules/nsf_hifigan](https://github.com/openvpi/DiffSinger/tree/refactor/modules/nsf_hifigan) |[2006.08195](https://arxiv.org/abs/2006.08195) | Snake (Vocoder) | Neural Networks Fail to Learn Periodic Functions and How to Fix It | [EdwardDixon/snake](https://github.com/EdwardDixon/snake) |[2105.02446v3](https://arxiv.org/abs/2105.02446v3) | Shallow Diffusion (PostProcessing)| DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism | [CNChTu/Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) | |[K-means](https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=01D65490BADCC216F350D06F84D721AD?doi=10.1.1.308.8619&rep=rep1&type=pdf) | Feature K-means Clustering (PreProcessing)| Some methods for classification and analysis of multivariate observations | This repo | | | Feature TopK Retrieval (PreProcessing)| Retrieval based Voice Conversion | [RVC-Project/Retrieval-based-Voice-Conversion-WebUI](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) | | | whisper ppg| whisper ppg | [PlayVoice/whisper_ppg](https://github.com/PlayVoice/whisper_ppg) | | | bigvgan| bigvgan | [PlayVoice/so-vits-svc-5.0](https://github.com/PlayVoice/so-vits-svc-5.0/tree/bigvgan-mix-v2/vits_decoder/alias) | ## ☀️ Previous contributors For some reason the author deleted the original repository. Because of the negligence of the organization members, the contributor list was cleared because all files were directly reuploaded to this repository at the beginning of the reconstruction of this repository. Now add a previous contributor list to README.md. *Some members have not listed according to their personal wishes.*

MistEO
XiaoMiku01
しぐれ
TomoGaSukunai
Plachtaa
zd小达
凍聲響世
## 📚 Some legal provisions for reference #### Any country, region, organization, or individual using this project must comply with the following laws. #### 《民法典》 ##### 第一千零一十九条 任何组织或者个人不得以丑化、污损，或者利用信息技术手段伪造等方式侵害他人的肖像权。未经肖像权人同意，不得制作、使用、公开肖像权人的肖像，但是法律另有规定的除外。未经肖像权人同意，肖像作品权利人不得以发表、复制、发行、出租、展览等方式使用或者公开肖像权人的肖像。对自然人声音的保护，参照适用肖像权保护的有关规定。 ##### 第一千零二十四条 【名誉权】民事主体享有名誉权。任何组织或者个人不得以侮辱、诽谤等方式侵害他人的名誉权。 ##### 第一千零二十七条 【作品侵害名誉权】行为人发表的文学、艺术作品以真人真事或者特定人为描述对象，含有侮辱、诽谤内容，侵害他人名誉权的，受害人有权依法请求该行为人承担民事责任。行为人发表的文学、艺术作品不以特定人为描述对象，仅其中的情节与该特定人的情况相似的，不承担民事责任。 #### 《[中华人民共和国宪法](http://www.gov.cn/guoqing/2018-03/22/content_5276318.htm)》 #### 《[中华人民共和国刑法](http://gongbao.court.gov.cn/Details/f8e30d0689b23f57bfc782d21035c3.html?sw=%E4%B8%AD%E5%8D%8E%E4%BA%BA%E6%B0%91%E5%85%B1%E5%92%8C%E5%9B%BD%E5%88%91%E6%B3%95)》 #### 《[中华人民共和国民法典](http://gongbao.court.gov.cn/Details/51eb6750b8361f79be8f90d09bc202.html)》 #### 《[中华人民共和国合同法](http://www.npc.gov.cn/zgrdw/npc/lfzt/rlyw/2016-07/01/content_1992739.htm)》 ## 💪 Thanks to all contributors for their efforts so-vits-svc-npu/README_zh_CN.md000066400000000000000000001005231456161405400164220ustar00rootroot00000000000000
LOGO # SoftVC VITS Singing Voice Conversion(HuanLin Ver) [**English**](./README.md) | [**中文简体**](./README_zh_CN.md) [![在Google Cloab中打开](https://img.shields.io/badge/Colab-F9AB00?style=for-the-badge&logo=googlecolab&color=525252)](https://colab.research.google.com/github/svc-develop-team/so-vits-svc/blob/4.1-Stable/sovits4_for_colab.ipynb) [![LICENSE](https://img.shields.io/badge/LICENSE-AGPL3.0-green.svg?style=for-the-badge)](https://github.com/svc-develop-team/so-vits-svc/blob/4.1-Stable/LICENSE) 本轮限时更新即将结束，仓库将进入Archieve状态，望周知
#### ✨ 带有 F0 曲线编辑器，角色混合时间轴编辑器的推理端 (Onnx 模型的用途）: [MoeVoiceStudio](https://github.com/NaruseMioShirakana/MoeVoiceStudio) #### ✨ 改善了交互的一个分支推荐: [34j/so-vits-svc-fork](https://github.com/34j/so-vits-svc-fork) #### ✨ 支持实时转换的一个客户端: [w-okada/voice-changer](https://github.com/w-okada/voice-changer) **本项目与 Vits 有着根本上的不同。Vits 是 TTS，本项目是 SVC。本项目无法实现 TTS，Vits 也无法实现 SVC，这两个项目的模型是完全不通用的。** ## 重要通知 这个项目是为了让开发者最喜欢的动画角色唱歌而开发的，任何涉及真人的东西都与开发者的意图背道而驰。 ## 声明 本项目为开源、离线的项目，SvcDevelopTeam 的所有成员与本项目的所有开发者以及维护者（以下简称贡献者）对本项目没有控制力。本项目的贡献者从未向任何组织或个人提供包括但不限于数据集提取、数据集加工、算力支持、训练支持、推理等一切形式的帮助；本项目的贡献者不知晓也无法知晓使用者使用该项目的用途。故一切基于本项目训练的 AI 模型和合成的音频都与本项目贡献者无关。一切由此造成的问题由使用者自行承担。 此项目完全离线运行，不能收集任何用户信息或获取用户输入数据。因此，这个项目的贡献者不知道所有的用户输入和模型，因此不负责任何用户输入。 本项目只是一个框架项目，本身并没有语音合成的功能，所有的功能都需要用户自己训练模型。同时，这个项目没有任何模型，任何二次分发的项目都与这个项目的贡献者无关。 ## 📏 使用规约 # Warning：请自行解决数据集授权问题，禁止使用非授权数据集进行训练！任何由于使用非授权数据集进行训练造成的问题，需自行承担全部责任和后果！与仓库、仓库维护者、svc develop team 无关！ 1. 本项目是基于学术交流目的建立，仅供交流与学习使用，并非为生产环境准备。 2. 任何发布到视频平台的基于 sovits 制作的视频，都必须要在简介明确指明用于变声器转换的输入源歌声、音频，例如：使用他人发布的视频 / 音频，通过分离的人声作为输入源进行转换的，必须要给出明确的原视频、音乐链接；若使用是自己的人声，或是使用其他歌声合成引擎合成的声音作为输入源进行转换的，也必须在简介加以说明。 3. 由输入源造成的侵权问题需自行承担全部责任和一切后果。使用其他商用歌声合成软件作为输入源时，请确保遵守该软件的使用条例，注意，许多歌声合成引擎使用条例中明确指明不可用于输入源进行转换！ 4. 禁止使用该项目从事违法行为与宗教、政治等活动，该项目维护者坚决抵制上述行为，不同意此条则禁止使用该项目。 5. 继续使用视为已同意本仓库 README 所述相关条例，本仓库 README 已进行劝导义务，不对后续可能存在问题负责。 6. 如果将此项目用于任何其他企划，请提前联系并告知本仓库作者，十分感谢。 ## 📝 模型简介 歌声音色转换模型，通过 SoftVC 内容编码器提取源音频语音特征，与 F0 同时输入 VITS 替换原本的文本输入达到歌声转换的效果。同时，更换声码器为 [NSF HiFiGAN](https://github.com/openvpi/DiffSinger/tree/refactor/modules/nsf_hifigan) 解决断音问题。 ### 🆕 4.1-Stable 版本更新内容 + 特征输入更换为 [Content Vec](https://github.com/auspicious3000/contentvec) 的第 12 层 Transformer 输出，并兼容 4.0 分支 + 更新浅层扩散，可以使用浅层扩散模型提升音质 + 增加 whisper 语音编码器的支持 + 增加静态/动态声线融合 + 增加响度嵌入 + 增加特征检索，来自于 [RVC](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) ### 🆕 关于兼容 4.0 模型的问题 + 可通过修改 4.0 模型的 config.json 对 4.0 的模型进行支持，需要在 config.json 的 model 字段中添加 speech_encoder 字段，具体见下 ``` "model": { ......... "ssl_dim": 256, "n_speakers": 200, "speech_encoder":"vec256l9" } ``` ### 🆕 关于浅扩散 ![Diagram](shadowdiffusion.png) ## 💬 关于 Python 版本问题 在进行测试后，我们认为`Python 3.8.9`能够稳定地运行该项目 ## 📥 预先下载的模型文件 #### **必须项** **以下编码器需要选择一个使用** ##### **1. 若使用 contentvec 作为声音编码器（推荐）** `vec768l12`与`vec256l9` 需要该编码器 + contentvec ：[checkpoint_best_legacy_500.pt](https://ibm.box.com/s/z1wgl1stco8ffooyatzdwsqn2psd9lrr) + 放在`pretrain`目录下 或者下载下面的 ContentVec，大小只有 199MB，但效果相同： + contentvec ：[hubert_base.pt](https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt) + 将文件名改为`checkpoint_best_legacy_500.pt`后，放在`pretrain`目录下 ```shell # contentvec wget -P pretrain/ https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt -O checkpoint_best_legacy_500.pt # 也可手动下载放在 pretrain 目录 ``` ##### **2. 若使用 hubertsoft 作为声音编码器** + soft vc hubert：[hubert-soft-0d54a1f4.pt](https://github.com/bshall/hubert/releases/download/v0.1/hubert-soft-0d54a1f4.pt) + 放在`pretrain`目录下 ##### **3. 若使用 Whisper-ppg 作为声音编码器** + 下载模型 [medium.pt](https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt), 该模型适配`whisper-ppg` + 下载模型 [large-v2.pt](https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt), 该模型适配`whisper-ppg-large` + 放在`pretrain`目录下 ##### **4. 若使用 cnhubertlarge 作为声音编码器** + 下载模型 [chinese-hubert-large-fairseq-ckpt.pt](https://huggingface.co/TencentGameMate/chinese-hubert-large/resolve/main/chinese-hubert-large-fairseq-ckpt.pt) + 放在`pretrain`目录下 ##### **5. 若使用 dphubert 作为声音编码器** + 下载模型 [DPHuBERT-sp0.75.pth](https://huggingface.co/pyf98/DPHuBERT/resolve/main/DPHuBERT-sp0.75.pth) + 放在`pretrain`目录下 ##### **6. 若使用 WavLM 作为声音编码器** + 下载模型 [WavLM-Base+.pt](https://valle.blob.core.windows.net/share/wavlm/WavLM-Base+.pt?sv=2020-08-04&st=2023-03-01T07%3A51%3A05Z&se=2033-03-02T07%3A51%3A00Z&sr=c&sp=rl&sig=QJXmSJG9DbMKf48UDIU1MfzIro8HQOf3sqlNXiflY1I%3D), 该模型适配`wavlmbase+` + 放在`pretrain`目录下 ##### **7. 若使用 OnnxHubert/ContentVec 作为声音编码器** + 下载模型 [MoeSS-SUBModel](https://huggingface.co/NaruseMioShirakana/MoeSS-SUBModel/tree/main) + 放在`pretrain`目录下 #### **编码器列表** - "vec768l12" - "vec256l9" - "vec256l9-onnx" - "vec256l12-onnx" - "vec768l9-onnx" - "vec768l12-onnx" - "hubertsoft-onnx" - "hubertsoft" - "whisper-ppg" - "cnhubertlarge" - "dphubert" - "whisper-ppg-large" - "wavlmbase+" #### **可选项（强烈建议使用）** + 预训练底模文件： `G_0.pth` `D_0.pth` + 放在`logs/44k`目录下 + 扩散模型预训练底模文件： `model_0.pt` + 放在`logs/44k/diffusion`目录下 从 svc-develop-team（待定）或任何其他地方获取 Sovits 底模 扩散模型引用了 [Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) 的 Diffusion Model，底模与 [Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) 的扩散模型底模通用，可以去 [Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) 获取扩散模型的底模 虽然底模一般不会引起什么版权问题，但还是请注意一下，比如事先询问作者，又或者作者在模型描述中明确写明了可行的用途 #### **可选项（根据情况选择）** ##### NSF-HIFIGAN 如果使用`NSF-HIFIGAN 增强器`或`浅层扩散`的话，需要下载预训练的 NSF-HIFIGAN 模型，如果不需要可以不下载 + 预训练的 NSF-HIFIGAN 声码器 ：[nsf_hifigan_20221211.zip](https://github.com/openvpi/vocoders/releases/download/nsf-hifigan-v1/nsf_hifigan_20221211.zip) + 解压后，将四个文件放在`pretrain/nsf_hifigan`目录下 ```shell # nsf_hifigan wget -P pretrain/ https://github.com/openvpi/vocoders/releases/download/nsf-hifigan-v1/nsf_hifigan_20221211.zip unzip -od pretrain/nsf_hifigan pretrain/nsf_hifigan_20221211.zip # 也可手动下载放在 pretrain/nsf_hifigan 目录 # 地址：https://github.com/openvpi/vocoders/releases/tag/nsf-hifigan-v1 ``` ##### RMVPE 如果使用`rmvpe`F0预测器的话，需要下载预训练的 RMVPE 模型 + 下载模型[rmvpe.zip](https://github.com/yxlllc/RMVPE/releases/download/230917/rmvpe.zip)，目前首推该权重。 + 解压缩`rmvpe.zip`，并将其中的`model.pt`文件改名为`rmvpe.pt`并放在`pretrain`目录下 + ~~下载模型 [rmvpe.pt](https://huggingface.co/datasets/ylzz1997/rmvpe_pretrain_model/resolve/main/rmvpe.pt)~~ + ~~放在`pretrain`目录下~~ ##### FCPE(预览版) > 你说的对,但是[FCPE](https://github.com/CNChTu/MelPE)是由svc-develop-team自主研发的一款全新的F0预测器，后面忘了 [FCPE(Fast Context-base Pitch Estimator)](https://github.com/CNChTu/MelPE)是一个为实时语音转换所设计的专用F0预测器，他将在未来成为Sovits实时语音转换的首选F0预测器.（论文未来会有的） 如果使用 `fcpe` F0预测器的话，需要下载预训练的 FCPE 模型 + 下载模型 [fcpe.pt](https://huggingface.co/datasets/ylzz1997/rmvpe_pretrain_model/resolve/main/fcpe.pt) + 放在`pretrain`目录下 ## 📊 数据集准备 仅需要以以下文件结构将数据集放入 dataset_raw 目录即可。 ``` dataset_raw ├───speaker0 │ ├───xxx1-xxx1.wav │ ├───... │ └───Lxx-0xx8.wav └───speaker1 ├───xx2-0xxx2.wav ├───... └───xxx7-xxx007.wav ``` 对于每一个音频文件的名称并没有格式的限制(`000001.wav`~`999999.wav`之类的命名方式也是合法的)，不过文件类型必须是`wav`。 可以自定义说话人名称 ``` dataset_raw └───suijiSUI ├───1.wav ├───... └───25788785-20221210-200143-856_01_(Vocals)_0_0.wav ``` ## 🛠️ 数据预处理 ### 0. 音频切片 将音频切片至`5s - 15s`, 稍微长点也无伤大雅，实在太长可能会导致训练中途甚至预处理就爆显存 可以使用 [audio-slicer-GUI](https://github.com/flutydeer/audio-slicer)、[audio-slicer-CLI](https://github.com/openvpi/audio-slicer) 一般情况下只需调整其中的`Minimum Interval`，普通陈述素材通常保持默认即可，歌唱素材可以调整至`100`甚至`50` 切完之后手动删除过长过短的音频 **如果你使用 Whisper-ppg 声音编码器进行训练，所有的切片长度必须小于 30s** ### 1. 重采样至 44100Hz 单声道 ```shell python resample.py ``` #### 注意 虽然本项目拥有重采样、转换单声道与响度匹配的脚本 resample.py，但是默认的响度匹配是匹配到 0db。这可能会造成音质的受损。而 python 的响度匹配包 pyloudnorm 无法对电平进行压限，这会导致爆音。所以建议可以考虑使用专业声音处理软件如`adobe audition`等软件做响度匹配处理。若已经使用其他软件做响度匹配，可以在运行上述命令时添加`--skip_loudnorm`跳过响度匹配步骤。如： ```shell python resample.py --skip_loudnorm ``` ### 2. 自动划分训练集、验证集，以及自动生成配置文件 ```shell python preprocess_flist_config.py --speech_encoder vec768l12 ``` speech_encoder 拥有以下选择 ``` vec768l12 vec256l9 hubertsoft whisper-ppg whisper-ppg-large cnhubertlarge dphubert wavlmbase+ ``` 如果省略 speech_encoder 参数，默认值为 vec768l12 **使用响度嵌入** 若使用响度嵌入，需要增加`--vol_aug`参数，比如： ```shell python preprocess_flist_config.py --speech_encoder vec768l12 --vol_aug ``` 使用后训练出的模型将匹配到输入源响度，否则为训练集响度。 #### 此时可以在生成的 config.json 与 diffusion.yaml 修改部分参数 ##### config.json * `keep_ckpts`：训练时保留最后几个模型，`0`为保留所有，默认只保留最后`3`个 * `all_in_mem`：加载所有数据集到内存中，某些平台的硬盘 IO 过于低下、同时内存容量 **远大于** 数据集体积时可以启用 * `batch_size`：单次训练加载到 GPU 的数据量，调整到低于显存容量的大小即可 * `vocoder_name` : 选择一种声码器，默认为`nsf-hifigan`. ##### diffusion.yaml * `cache_all_data`：加载所有数据集到内存中，某些平台的硬盘 IO 过于低下、同时内存容量 **远大于** 数据集体积时可以启用 * `duration`：训练时音频切片时长，可根据显存大小调整，**注意，该值必须小于训练集内音频的最短时间！** * `batch_size`：单次训练加载到 GPU 的数据量，调整到低于显存容量的大小即可 * `timesteps` : 扩散模型总步数，默认为 1000. * `k_step_max` : 训练时可仅训练`k_step_max`步扩散以节约训练时间，注意，该值必须小于`timesteps`，0 为训练整个扩散模型，**注意，如果不训练整个扩散模型将无法使用仅扩散模型推理！** ##### **声码器列表** ``` nsf-hifigan nsf-snake-hifigan ``` ### 3. 生成 hubert 与 f0 ```shell python preprocess_hubert_f0.py --f0_predictor dio ``` f0_predictor 拥有以下选择 ``` crepe dio pm harvest rmvpe fcpe ``` 如果训练集过于嘈杂，请使用 crepe 处理 f0 如果省略 f0_predictor 参数，默认值为 rmvpe 尚若需要浅扩散功能（可选），需要增加--use_diff 参数，比如 ```shell python preprocess_hubert_f0.py --f0_predictor dio --use_diff ``` **加速预处理** 如若您的数据集比较大，可以尝试添加`--num_processes`参数： ```shell python preprocess_hubert_f0.py --f0_predictor dio --use_diff --num_processes 8 ``` 所有的Workers会被自动分配到多个线程上 执行完以上步骤后 dataset 目录便是预处理完成的数据，可以删除 dataset_raw 文件夹了 ## 🏋️‍ 训练 ### 主模型训练 ```shell python train.py -c configs/config.json -m 44k ``` 这里提供了一个可以随时暂停训练并保存模型的方法，在运行目录下面新建一个 `stop.txt` 即可，训练会在下一个 step 结束后停止并保存模型 ### 扩散模型（可选） 尚若需要浅扩散功能，需要训练扩散模型，扩散模型训练方法为： ```shell python train_diff.py -c configs/diffusion.yaml ``` 模型训练结束后，模型文件保存在`logs/44k`目录下，扩散模型在`logs/44k/diffusion`下 ## 🤖 推理 使用 [inference_main.py](inference_main.py) ```shell # 例 python inference_main.py -m "logs/44k/G_30400.pth" -c "configs/config.json" -n "君の知らない物語-src.wav" -t 0 -s "nen" ``` 必填项部分： + `-m` | `--model_path`：模型路径 (接受参数 `{lastest}`，示例: `"logs/44k/{lastest}"`，将获取最新保存点) + `-c` | `--config_path`：配置文件路径，缺省时使用 `logs/44k/config.json` + `-t` | `--trans`：音高调整，支持正负（半音） + `-s` | `--spk_list`：合成目标说话人名称，缺省时我也不知道怎么干 + `-n` | `--clean_names`：wav 文件名列表，放在 raw 文件夹下，缺省时 fallback 到 raw 文件夹中所有 wav 文件 可选项部分：部分具体见下一节 + `-cl` | `--clip`：音频强制切片，默认 0 为自动切片，单位为秒/s + `-lg` | `--linear_gradient`：两段音频切片的交叉淡入长度，如果强制切片后出现人声不连贯可调整该数值，如果连贯建议采用默认值 0，单位为秒 + `-f0p` | `--f0_predictor`：选择 F0 预测器，可选择 crepe,pm,dio,harvest,rmvpe,fcpe, 默认为 pm（注意：crepe 为原 F0 使用均值滤波器） + `-a` | `--auto_predict_f0`：语音转换自动预测音高，转换歌声时不要打开这个会严重跑调 + `-cm` | `--cluster_model_path`：聚类模型或特征检索索引路径，留空则自动设为各方案模型的默认路径，如果没有训练聚类或特征检索则随便填 + `-cr` | `--cluster_infer_ratio`：聚类方案或特征检索占比，范围 0-1，若没有训练聚类模型或特征检索则默认 0 即可 + `-eh` | `--enhance`：是否使用 NSF_HIFIGAN 增强器，该选项对部分训练集少的模型有一定的音质增强效果，但是对训练好的模型有反面效果，默认关闭 + `-shd` | `--shallow_diffusion`：是否使用浅层扩散，使用后可解决一部分电音问题，默认关闭，该选项打开时，NSF_HIFIGAN 增强器将会被禁止 + `-usm` | `--use_spk_mix`：是否使用角色融合/动态声线融合 + `-lea` | `--loudness_envelope_adjustment`：输入源响度包络替换输出响度包络融合比例，越靠近 1 越使用输出响度包络 + `-fr` | `--feature_retrieval`：是否使用特征检索，如果使用聚类模型将被禁用，且 cm 与 cr 参数将会变成特征检索的索引路径与混合比例 浅扩散设置： + `-dm` | `--diffusion_model_path`：扩散模型路径 + `-dc` | `--diffusion_config_path`：扩散模型配置文件路径 + `-ks` | `--k_step`：扩散步数，越大越接近扩散模型的结果，默认 100 + `-od` | `--only_diffusion`：纯扩散模式，该模式不会加载 sovits 模型，以扩散模型推理 + `-se` | `--second_encoding`：二次编码，浅扩散前会对原始音频进行二次编码，玄学选项，有时候效果好，有时候效果差 ### 注意！ 如果使用`whisper-ppg` 声音编码器进行推理，需要将`--clip`设置为 25，`-lg`设置为 1。否则将无法正常推理。 ## 🤔 可选项 如果前面的效果已经满意，或者没看明白下面在讲啥，那后面的内容都可以忽略，不影响模型使用（这些可选项影响比较小，可能在某些特定数据上有点效果，但大部分情况似乎都感知不太明显） ### 自动 f0 预测 4.0 模型训练过程会训练一个 f0 预测器，对于语音转换可以开启自动音高预测，如果效果不好也可以使用手动的，但转换歌声时请不要启用此功能！！！会严重跑调！！ + 在 inference_main 中设置 auto_predict_f0 为 true 即可 ### 聚类音色泄漏控制 介绍：聚类方案可以减小音色泄漏，使得模型训练出来更像目标的音色（但其实不是特别明显），但是单纯的聚类方案会降低模型的咬字（会口齿不清）（这个很明显），本模型采用了融合的方式，可以线性控制聚类方案与非聚类方案的占比，也就是可以手动在"像目标音色" 和 "咬字清晰" 之间调整比例，找到合适的折中点 使用聚类前面的已有步骤不用进行任何的变动，只需要额外训练一个聚类模型，虽然效果比较有限，但训练成本也比较低 + 训练过程： + 使用 cpu 性能较好的机器训练，据我的经验在腾讯云 6 核 cpu 训练每个 speaker 需要约 4 分钟即可完成训练 + 执行`python cluster/train_cluster.py`，模型的输出会在`logs/44k/kmeans_10000.pt` + 聚类模型目前可以使用 gpu 进行训练，执行`python cluster/train_cluster.py --gpu` + 推理过程： + `inference_main.py`中指定`cluster_model_path` 为模型输出文件，留空则默认为`logs/44k/kmeans_10000.pt` + `inference_main.py`中指定`cluster_infer_ratio`，`0`为完全不使用聚类，`1`为只使用聚类，通常设置`0.5`即可 ### 特征检索 介绍：跟聚类方案一样可以减小音色泄漏，咬字比聚类稍好，但会降低推理速度，采用了融合的方式，可以线性控制特征检索与非特征检索的占比， + 训练过程： 首先需要在生成 hubert 与 f0 后执行： ```shell python train_index.py -c configs/config.json ``` 模型的输出会在`logs/44k/feature_and_index.pkl` + 推理过程： + 需要首先指定`--feature_retrieval`，此时聚类方案会自动切换到特征检索方案 + `inference_main.py`中指定`cluster_model_path` 为模型输出文件，留空则默认为`logs/44k/feature_and_index.pkl` + `inference_main.py`中指定`cluster_infer_ratio`，`0`为完全不使用特征检索，`1`为只使用特征检索，通常设置`0.5`即可 ## 🗜️ 模型压缩 生成的模型含有继续训练所需的信息。如果确认不再训练，可以移除模型中此部分信息，得到约 1/3 大小的最终模型。 使用 [compress_model.py](compress_model.py) ```shell # 例 python compress_model.py -c="configs/config.json" -i="logs/44k/G_30400.pth" -o="logs/44k/release.pth" ``` ## 👨‍🔧 声线混合 ### 静态声线混合 **参考`webUI.py`文件中，小工具/实验室特性的静态声线融合。** 介绍：该功能可以将多个声音模型合成为一个声音模型（多个模型参数的凸组合或线性组合），从而制造出现实中不存在的声线 **注意：** 1. 该功能仅支持单说话人的模型 2. 如果强行使用多说话人模型，需要保证多个模型的说话人数量相同，这样可以混合同一个 SpaekerID 下的声音 3. 保证所有待混合模型的 config.json 中的 model 字段是相同的 4. 输出的混合模型可以使用待合成模型的任意一个 config.json，但聚类模型将不能使用 5. 批量上传模型的时候最好把模型放到一个文件夹选中后一起上传 6. 混合比例调整建议大小在 0-100 之间，也可以调为其他数字，但在线性组合模式下会出现未知的效果 7. 混合完毕后，文件将会保存在项目根目录中，文件名为 output.pth 8. 凸组合模式会将混合比例执行 Softmax 使混合比例相加为 1，而线性组合模式不会 ### 动态声线混合 **参考`spkmix.py`文件中关于动态声线混合的介绍** 角色混合轨道 编写规则： 角色 ID : \[\[起始时间 1, 终止时间 1, 起始数值 1, 起始数值 1], [起始时间 2, 终止时间 2, 起始数值 2, 起始数值 2]] 起始时间和前一个的终止时间必须相同，第一个起始时间必须为 0，最后一个终止时间必须为 1 （时间的范围为 0-1） 全部角色必须填写，不使用的角色填、[\[0., 1., 0., 0.]] 即可 融合数值可以随便填，在指定的时间段内从起始数值线性变化为终止数值，内部会自动确保线性组合为 1（凸组合条件），可以放心使用 推理的时候使用`--use_spk_mix`参数即可启用动态声线混合 ## 📤 Onnx 导出 使用 [onnx_export.py](onnx_export.py) + 新建文件夹：`checkpoints` 并打开 + 在`checkpoints`文件夹中新建一个文件夹作为项目文件夹，文件夹名为你的项目名称，比如`aziplayer` + 将你的模型更名为`model.pth`，配置文件更名为`config.json`，并放置到刚才创建的`aziplayer`文件夹下 + 将 [onnx_export.py](onnx_export.py) 中`path = "NyaruTaffy"` 的 `"NyaruTaffy"` 修改为你的项目名称，`path = "aziplayer" (onnx_export_speaker_mix，为支持角色混合的 onnx 导出）` + 运行 [onnx_export.py](onnx_export.py) + 等待执行完毕，在你的项目文件夹下会生成一个`model.onnx`，即为导出的模型 注意：Hubert Onnx 模型请使用 MoeSS 提供的模型，目前无法自行导出（fairseq 中 Hubert 有不少 onnx 不支持的算子和涉及到常量的东西，在导出时会报错或者导出的模型输入输出 shape 和结果都有问题） ## 📎 引用及论文 | URL | 名称 | 标题 | 源码 | | --- | ----------- | ----- | --------------------- | |[2106.06103](https://arxiv.org/abs/2106.06103) | VITS (Synthesizer)| Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech | [jaywalnut310/vits](https://github.com/jaywalnut310/vits) | |[2111.02392](https://arxiv.org/abs/2111.02392) | SoftVC (Speech Encoder)| A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion | [bshall/hubert](https://github.com/bshall/hubert) | |[2204.09224](https://arxiv.org/abs/2204.09224) | ContentVec (Speech Encoder)| ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers | [auspicious3000/contentvec](https://github.com/auspicious3000/contentvec) | |[2212.04356](https://arxiv.org/abs/2212.04356) | Whisper (Speech Encoder) | Robust Speech Recognition via Large-Scale Weak Supervision | [openai/whisper](https://github.com/openai/whisper) | |[2110.13900](https://arxiv.org/abs/2110.13900) | WavLM (Speech Encoder) | WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing | [microsoft/unilm/wavlm](https://github.com/microsoft/unilm/tree/master/wavlm) | |[2305.17651](https://arxiv.org/abs/2305.17651) | DPHubert (Speech Encoder) | DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models | [pyf98/DPHuBERT](https://github.com/pyf98/DPHuBERT) | |[DOI:10.21437/Interspeech.2017-68](http://dx.doi.org/10.21437/Interspeech.2017-68) | Harvest (F0 Predictor) | Harvest: A high-performance fundamental frequency estimator from speech signals | [mmorise/World/harvest](https://github.com/mmorise/World/blob/master/src/harvest.cpp) | |[aes35-000039](https://www.aes.org/e-lib/online/browse.cfm?elib=15165) | Dio (F0 Predictor) | Fast and reliable F0 estimation method based on the period extraction of vocal fold vibration of singing voice and speech | [mmorise/World/dio](https://github.com/mmorise/World/blob/master/src/dio.cpp) | |[8461329](https://ieeexplore.ieee.org/document/8461329) | Crepe (F0 Predictor) | Crepe: A Convolutional Representation for Pitch Estimation | [maxrmorrison/torchcrepe](https://github.com/maxrmorrison/torchcrepe) | |[DOI:10.1016/j.wocn.2018.07.001](https://doi.org/10.1016/j.wocn.2018.07.001) | Parselmouth (F0 Predictor) | Introducing Parselmouth: A Python interface to Praat | [YannickJadoul/Parselmouth](https://github.com/YannickJadoul/Parselmouth) | |[2306.15412v2](https://arxiv.org/abs/2306.15412v2) | RMVPE (F0 Predictor) | RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music | [Dream-High/RMVPE](https://github.com/Dream-High/RMVPE) | |[2010.05646](https://arxiv.org/abs/2010.05646) | HIFIGAN (Vocoder) | HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis | [jik876/hifi-gan](https://github.com/jik876/hifi-gan) | |[1810.11946](https://arxiv.org/abs/1810.11946.pdf) | NSF (Vocoder) | Neural source-filter-based waveform model for statistical parametric speech synthesis | [openvpi/DiffSinger/modules/nsf_hifigan](https://github.com/openvpi/DiffSinger/tree/refactor/modules/nsf_hifigan) |[2006.08195](https://arxiv.org/abs/2006.08195) | Snake (Vocoder) | Neural Networks Fail to Learn Periodic Functions and How to Fix It | [EdwardDixon/snake](https://github.com/EdwardDixon/snake) |[2105.02446v3](https://arxiv.org/abs/2105.02446v3) | Shallow Diffusion (PostProcessing)| DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism | [CNChTu/Diffusion-SVC](https://github.com/CNChTu/Diffusion-SVC) | |[K-means](https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=01D65490BADCC216F350D06F84D721AD?doi=10.1.1.308.8619&rep=rep1&type=pdf) | Feature K-means Clustering (PreProcessing)| Some methods for classification and analysis of multivariate observations | 本代码库 | | | Feature TopK Retrieval (PreProcessing)| Retrieval based Voice Conversion | [RVC-Project/Retrieval-based-Voice-Conversion-WebUI](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) | ## ☀️ 旧贡献者 因为某些原因原作者进行了删库处理，本仓库重建之初由于组织成员疏忽直接重新上传了所有文件导致以前的 contributors 全部木大，现在在 README 里重新添加一个旧贡献者列表 *某些成员已根据其个人意愿不将其列出*

MistEO
XiaoMiku01
しぐれ
TomoGaSukunai
Plachtaa
zd 小达
凍聲響世
## 📚 一些法律条例参考 #### 任何国家，地区，组织和个人使用此项目必须遵守以下法律 #### 《民法典》 ##### 第一千零一十九条 任何组织或者个人不得以丑化、污损，或者利用信息技术手段伪造等方式侵害他人的肖像权。未经肖像权人同意，不得制作、使用、公开肖像权人的肖像，但是法律另有规定的除外。未经肖像权人同意，肖像作品权利人不得以发表、复制、发行、出租、展览等方式使用或者公开肖像权人的肖像。对自然人声音的保护，参照适用肖像权保护的有关规定。 ##### 第一千零二十四条 【名誉权】民事主体享有名誉权。任何组织或者个人不得以侮辱、诽谤等方式侵害他人的名誉权。 ##### 第一千零二十七条 【作品侵害名誉权】行为人发表的文学、艺术作品以真人真事或者特定人为描述对象，含有侮辱、诽谤内容，侵害他人名誉权的，受害人有权依法请求该行为人承担民事责任。行为人发表的文学、艺术作品不以特定人为描述对象，仅其中的情节与该特定人的情况相似的，不承担民事责任。 #### 《[中华人民共和国宪法](http://www.gov.cn/guoqing/2018-03/22/content_5276318.htm)》 #### 《[中华人民共和国刑法](http://gongbao.court.gov.cn/Details/f8e30d0689b23f57bfc782d21035c3.html?sw=中华人民共和国刑法)》 #### 《[中华人民共和国民法典](http://gongbao.court.gov.cn/Details/51eb6750b8361f79be8f90d09bc202.html)》 #### 《[中华人民共和国合同法](http://www.npc.gov.cn/zgrdw/npc/lfzt/rlyw/2016-07/01/content_1992739.htm)》 ## 💪 感谢所有的贡献者 so-vits-svc-npu/cluster/000077500000000000000000000000001456161405400155425ustar00rootroot00000000000000so-vits-svc-npu/cluster/__init__.py000066400000000000000000000015631456161405400176600ustar00rootroot00000000000000import torch from sklearn.cluster import KMeans def get_cluster_model(ckpt_path): checkpoint = torch.load(ckpt_path) kmeans_dict = {} for spk, ckpt in checkpoint.items(): km = KMeans(ckpt["n_features_in_"]) km.__dict__["n_features_in_"] = ckpt["n_features_in_"] km.__dict__["_n_threads"] = ckpt["_n_threads"] km.__dict__["cluster_centers_"] = ckpt["cluster_centers_"] kmeans_dict[spk] = km return kmeans_dict def get_cluster_result(model, x, speaker): """ x: np.array [t, 256] return cluster class result """ return model[speaker].predict(x) def get_cluster_center_result(model, x,speaker): """x: np.array [t, 256]""" predict = model[speaker].predict(x) return model[speaker].cluster_centers_[predict] def get_center(model, x,speaker): return model[speaker].cluster_centers_[x] so-vits-svc-npu/cluster/kmeans.py000066400000000000000000000167631456161405400174070ustar00rootroot00000000000000from time import time import numpy as np import pynvml import mindtorch.torch as torch from mindtorch.torch.nn.functional import normalize # device=torch.device("cuda:0") def _kpp(data: torch.Tensor, k: int, sample_size: int = -1): """ Picks k points in the data based on the kmeans++ method. Parameters ---------- data : torch.Tensor Expect a rank 1 or 2 array. Rank 1 is assumed to describe 1-D data, rank 2 multidimensional data, in which case one row is one observation. k : int Number of samples to generate. sample_size : int sample data to avoid memory overflow during calculation Returns ------- init : ndarray A 'k' by 'N' containing the initial centroids. References ---------- .. [1] D. Arthur and S. Vassilvitskii, "k-means++: the advantages of careful seeding", Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 2007. .. [2] scipy/cluster/vq.py: _kpp """ batch_size=data.shape[0] if batch_size>sample_size: data = data[torch.randint(0, batch_size,[sample_size], device=data.device)] dims = data.shape[1] if len(data.shape) > 1 else 1 init = torch.zeros((k, dims)).to(data.device) r = torch.distributions.uniform.Uniform(0, 1) for i in range(k): if i == 0: init[i, :] = data[torch.randint(data.shape[0], [1])] else: D2 = torch.cdist(init[:i, :][None, :], data[None, :], p=2)[0].amin(dim=0) probs = D2 / torch.sum(D2) cumprobs = torch.cumsum(probs, dim=0) init[i, :] = data[torch.searchsorted(cumprobs, r.sample([1]).to(data.device))] return init class KMeansGPU: ''' Kmeans clustering algorithm implemented with PyTorch Parameters: n_clusters: int, Number of clusters max_iter: int, default: 100 Maximum number of iterations tol: float, default: 0.0001 Tolerance verbose: int, default: 0 Verbosity mode: {'euclidean', 'cosine'}, default: 'euclidean' Type of distance measure init_method: {'random', 'point', '++'} Type of initialization minibatch: {None, int}, default: None Batch size of MinibatchKmeans algorithm if None perform full KMeans algorithm Attributes: centroids: torch.Tensor, shape: [n_clusters, n_features] cluster centroids ''' def __init__(self, n_clusters, max_iter=200, tol=1e-4, verbose=0, mode="euclidean",device=torch.device("cuda:0")): self.n_clusters = n_clusters self.max_iter = max_iter self.tol = tol self.verbose = verbose self.mode = mode self.device=device pynvml.nvmlInit() gpu_handle = pynvml.nvmlDeviceGetHandleByIndex(device.index) info = pynvml.nvmlDeviceGetMemoryInfo(gpu_handle) self.minibatch=int(33e6/self.n_clusters*info.free/ 1024 / 1024 / 1024) print("free_mem/GB:",info.free/ 1024 / 1024 / 1024,"minibatch:",self.minibatch) @staticmethod def cos_sim(a, b): """ Compute cosine similarity of 2 sets of vectors Parameters: a: torch.Tensor, shape: [m, n_features] b: torch.Tensor, shape: [n, n_features] """ return normalize(a, dim=-1) @ normalize(b, dim=-1).transpose(-2, -1) @staticmethod def euc_sim(a, b): """ Compute euclidean similarity of 2 sets of vectors Parameters: a: torch.Tensor, shape: [m, n_features] b: torch.Tensor, shape: [n, n_features] """ return 2 * a @ b.transpose(-2, -1) -(a**2).sum(dim=1)[..., :, None] - (b**2).sum(dim=1)[..., None, :] def max_sim(self, a, b): """ Compute maximum similarity (or minimum distance) of each vector in a with all of the vectors in b Parameters: a: torch.Tensor, shape: [m, n_features] b: torch.Tensor, shape: [n, n_features] """ if self.mode == 'cosine': sim_func = self.cos_sim elif self.mode == 'euclidean': sim_func = self.euc_sim sim = sim_func(a, b) max_sim_v, max_sim_i = sim.max(dim=-1) return max_sim_v, max_sim_i def fit_predict(self, X): """ Combination of fit() and predict() methods. This is faster than calling fit() and predict() seperately. Parameters: X: torch.Tensor, shape: [n_samples, n_features] centroids: {torch.Tensor, None}, default: None if given, centroids will be initialized with given tensor if None, centroids will be randomly chosen from X Return: labels: torch.Tensor, shape: [n_samples] mini_=33kk/k*remain mini=min(mini_,fea_shape) offset=log2(k/1000)*1.5 kpp_all=min(mini_*10/offset,fea_shape) kpp_sample=min(mini_/12/offset,fea_shape) """ assert isinstance(X, torch.Tensor), "input must be torch.Tensor" assert X.dtype in [torch.half, torch.float, torch.double], "input must be floating point" assert X.ndim == 2, "input must be a 2d tensor with shape: [n_samples, n_features] " # print("verbose:%s"%self.verbose) offset = np.power(1.5,np.log(self.n_clusters / 1000))/np.log(2) with torch.no_grad(): batch_size= X.shape[0] # print(self.minibatch, int(self.minibatch * 10 / offset), batch_size) start_time = time() if (self.minibatch*10//offset< batch_size): x = X[torch.randint(0, batch_size,[int(self.minibatch*10/offset)])].to(self.device) else: x = X.to(self.device) # print(x.device) self.centroids = _kpp(x, self.n_clusters, min(int(self.minibatch/12/offset),batch_size)) del x torch.cuda.empty_cache() # self.centroids = self.centroids.to(self.device) num_points_in_clusters = torch.ones(self.n_clusters, device=self.device, dtype=X.dtype)#全1 closest = None#[3098036]#int64 if(self.minibatch>=batch_size//2 and self.minibatch=batch_size): X=X.to(self.device) for i in range(self.max_iter): iter_time = time() if self.minibatch= 2: print('iter:', i, 'error:', error.item(), 'time spent:', round(time()-iter_time, 4)) if error <= self.tol: break if self.verbose >= 1: print(f'used {i+1} iterations ({round(time()-start_time, 4)}s) to cluster {batch_size} items into {self.n_clusters} clusters') return closest so-vits-svc-npu/cluster/train_cluster.py000066400000000000000000000057611456161405400210030ustar00rootroot00000000000000import argparse import logging import os import time from pathlib import Path import numpy as np import torch import tqdm from kmeans import KMeansGPU from sklearn.cluster import KMeans, MiniBatchKMeans logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) def train_cluster(in_dir, n_clusters, use_minibatch=True, verbose=False,use_gpu=False):#gpu_minibatch真拉，虽然库支持但是也不考虑 if str(in_dir).endswith(".ipynb_checkpoints"): logger.info(f"Ignore {in_dir}") logger.info(f"Loading features from {in_dir}") features = [] nums = 0 for path in tqdm.tqdm(in_dir.glob("*.soft.pt")): # for name in os.listdir(in_dir): # path="%s/%s"%(in_dir,name) features.append(torch.load(path,map_location="cpu").squeeze(0).numpy().T) # print(features[-1].shape) features = np.concatenate(features, axis=0) print(nums, features.nbytes/ 1024**2, "MB , shape:",features.shape, features.dtype) features = features.astype(np.float32) logger.info(f"Clustering features of shape: {features.shape}") t = time.time() if(use_gpu is False): if use_minibatch: kmeans = MiniBatchKMeans(n_clusters=n_clusters,verbose=verbose, batch_size=4096, max_iter=80).fit(features) else: kmeans = KMeans(n_clusters=n_clusters,verbose=verbose).fit(features) else: kmeans = KMeansGPU(n_clusters=n_clusters, mode='euclidean', verbose=2 if verbose else 0,max_iter=500,tol=1e-2)# features=torch.from_numpy(features)#.to(device) kmeans.fit_predict(features)# print(time.time()-t, "s") x = { "n_features_in_": kmeans.n_features_in_ if use_gpu is False else features.shape[1], "_n_threads": kmeans._n_threads if use_gpu is False else 4, "cluster_centers_": kmeans.cluster_centers_ if use_gpu is False else kmeans.centroids.cpu().numpy(), } print("end") return x if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=Path, default="./dataset/44k", help='path of training data directory') parser.add_argument('--output', type=Path, default="logs/44k", help='path of model output directory') parser.add_argument('--gpu',action='store_true', default=False , help='to use GPU') args = parser.parse_args() checkpoint_dir = args.output dataset = args.dataset use_gpu = args.gpu n_clusters = 10000 ckpt = {} for spk in os.listdir(dataset): if os.path.isdir(dataset/spk): print(f"train kmeans for {spk}...") in_dir = dataset/spk x = train_cluster(in_dir, n_clusters,use_minibatch=False,verbose=False,use_gpu=use_gpu) ckpt[spk] = x checkpoint_path = checkpoint_dir / f"kmeans_{n_clusters}.pt" checkpoint_path.parent.mkdir(exist_ok=True, parents=True) torch.save( ckpt, checkpoint_path, ) so-vits-svc-npu/compress_model.py000066400000000000000000000044031456161405400174470ustar00rootroot00000000000000from collections import OrderedDict import mindtorch.torch as torch import utils from models import SynthesizerTrn def copyStateDict(state_dict): if list(state_dict.keys())[0].startswith('module'): start_idx = 1 else: start_idx = 0 new_state_dict = OrderedDict() for k, v in state_dict.items(): name = ','.join(k.split('.')[start_idx:]) new_state_dict[name] = v return new_state_dict def removeOptimizer(config: str, input_model: str, ishalf: bool, output_model: str): hps = utils.get_hparams_from_file(config) net_g = SynthesizerTrn(hps.data.filter_length // 2 + 1, hps.train.segment_size // hps.data.hop_length, **hps.model) optim_g = torch.optim.AdamW(net_g.parameters(), hps.train.learning_rate, betas=hps.train.betas, eps=hps.train.eps) state_dict_g = torch.load(input_model, map_location="cpu") new_dict_g = copyStateDict(state_dict_g) keys = [] for k, v in new_dict_g['model'].items(): if "enc_q" in k: continue # noqa: E701 keys.append(k) new_dict_g = {k: new_dict_g['model'][k].half() for k in keys} if ishalf else {k: new_dict_g['model'][k] for k in keys} torch.save( { 'model': new_dict_g, 'iteration': 0, 'optimizer': optim_g.state_dict(), 'learning_rate': 0.0001 }, output_model) if __name__ == "__main__": import argparse parser = argparse.ArgumentParser() parser.add_argument("-c", "--config", type=str, default='configs/config.json') parser.add_argument("-i", "--input", type=str) parser.add_argument("-o", "--output", type=str, default=None) parser.add_argument('-hf', '--half', action='store_true', default=False, help='Save as FP16') args = parser.parse_args() output = args.output if output is None: import os.path filename, ext = os.path.splitext(args.input) half = "_half" if args.half else "" output = filename + "_release" + half + ext removeOptimizer(args.config, args.input, args.half, output)so-vits-svc-npu/configs/000077500000000000000000000000001456161405400155115ustar00rootroot00000000000000so-vits-svc-npu/configs/config.json000066400000000000000000000037311456161405400176550ustar00rootroot00000000000000{ "train": { "log_interval": 200, "eval_interval": 800, "seed": 1234, "epochs": 10000, "learning_rate": 0.0001, "betas": [ 0.8, 0.99 ], "eps": 1e-09, "batch_size": 6, "fp16_run": false, "half_type": "fp16", "lr_decay": 0.999875, "segment_size": 10240, "init_lr_ratio": 1, "warmup_epochs": 0, "c_mel": 45, "c_kl": 1.0, "use_sr": true, "max_speclen": 512, "port": "8001", "keep_ckpts": 3, "all_in_mem": false, "vol_aug": false }, "data": { "training_files": "filelists/train.txt", "validation_files": "filelists/val.txt", "max_wav_value": 32768.0, "sampling_rate": 44100, "filter_length": 2048, "hop_length": 512, "win_length": 2048, "n_mel_channels": 80, "mel_fmin": 0.0, "mel_fmax": 22050, "unit_interpolate_mode": "nearest" }, "model": { "inter_channels": 192, "hidden_channels": 192, "filter_channels": 768, "n_heads": 2, "n_layers": 6, "kernel_size": 3, "p_dropout": 0.1, "resblock": "1", "resblock_kernel_sizes": [ 3, 7, 11 ], "resblock_dilation_sizes": [ [ 1, 3, 5 ], [ 1, 3, 5 ], [ 1, 3, 5 ] ], "upsample_rates": [ 8, 8, 2, 2, 2 ], "upsample_initial_channel": 512, "upsample_kernel_sizes": [ 16, 16, 4, 4, 4 ], "n_layers_q": 3, "n_layers_trans_flow": 3, "n_flow_layer": 4, "use_spectral_norm": false, "gin_channels": 768, "ssl_dim": 768, "n_speakers": 1, "vocoder_name": "nsf-hifigan", "speech_encoder": "vec768l12", "speaker_embedding": false, "vol_embedding": false, "use_depthwise_conv": false, "flow_share_parameter": false, "use_automatic_f0_prediction": true, "use_transformer_flow": false }, "spk": { "TEST": 0 } }so-vits-svc-npu/configs/diffusion.yaml000066400000000000000000000016371456161405400203720ustar00rootroot00000000000000data: block_size: 512 cnhubertsoft_gate: 10 duration: 2 encoder: vec768l12 encoder_hop_size: 320 encoder_out_channels: 768 encoder_sample_rate: 16000 extensions: - wav sampling_rate: 44100 training_files: filelists/train.txt unit_interpolate_mode: nearest validation_files: filelists/val.txt device: cuda env: expdir: logs/44k/diffusion gpu_id: 0 infer: method: dpm-solver++ speedup: 10 model: k_step_max: 0 n_chans: 512 n_hidden: 256 n_layers: 20 n_spk: 1 timesteps: 1000 type: Diffusion use_pitch_aug: true spk: TEST: 0 train: amp_dtype: fp32 batch_size: 48 cache_all_data: true cache_device: cpu cache_fp16: true decay_step: 100000 epochs: 100000 gamma: 0.5 interval_force_save: 5000 interval_log: 10 interval_val: 2000 lr: 0.0001 num_workers: 4 save_opt: false weight_decay: 0 vocoder: ckpt: pretrain/nsf_hifigan/model type: nsf-hifigan so-vits-svc-npu/configs_template/000077500000000000000000000000001456161405400174045ustar00rootroot00000000000000so-vits-svc-npu/configs_template/config_template.json000066400000000000000000000034651456161405400234470ustar00rootroot00000000000000{ "train": { "log_interval": 200, "eval_interval": 800, "seed": 1234, "epochs": 10000, "learning_rate": 0.0001, "betas": [ 0.8, 0.99 ], "eps": 1e-09, "batch_size": 6, "fp16_run": false, "half_type": "fp16", "lr_decay": 0.999875, "segment_size": 10240, "init_lr_ratio": 1, "warmup_epochs": 0, "c_mel": 45, "c_kl": 1.0, "use_sr": true, "max_speclen": 512, "port": "8001", "keep_ckpts": 3, "all_in_mem": false, "vol_aug":false }, "data": { "training_files": "filelists/train.txt", "validation_files": "filelists/val.txt", "max_wav_value": 32768.0, "sampling_rate": 44100, "filter_length": 2048, "hop_length": 512, "win_length": 2048, "n_mel_channels": 80, "mel_fmin": 0.0, "mel_fmax": 22050, "unit_interpolate_mode":"nearest" }, "model": { "inter_channels": 192, "hidden_channels": 192, "filter_channels": 768, "n_heads": 2, "n_layers": 6, "kernel_size": 3, "p_dropout": 0.1, "resblock": "1", "resblock_kernel_sizes": [3,7,11], "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]], "upsample_rates": [ 8, 8, 2, 2, 2], "upsample_initial_channel": 512, "upsample_kernel_sizes": [16,16, 4, 4, 4], "n_layers_q": 3, "n_layers_trans_flow": 3, "n_flow_layer": 4, "use_spectral_norm": false, "gin_channels": 768, "ssl_dim": 768, "n_speakers": 200, "vocoder_name":"nsf-hifigan", "speech_encoder":"vec768l12", "speaker_embedding":false, "vol_embedding":false, "use_depthwise_conv":false, "flow_share_parameter": false, "use_automatic_f0_prediction": true, "use_transformer_flow": false }, "spk": { "nyaru": 0, "huiyu": 1, "nen": 2, "paimon": 3, "yunhao": 4 } }so-vits-svc-npu/configs_template/config_tiny_template.json000066400000000000000000000034631456161405400245100ustar00rootroot00000000000000{ "train": { "log_interval": 200, "eval_interval": 800, "seed": 1234, "epochs": 10000, "learning_rate": 0.0001, "betas": [ 0.8, 0.99 ], "eps": 1e-09, "batch_size": 6, "fp16_run": false, "half_type": "fp16", "lr_decay": 0.999875, "segment_size": 10240, "init_lr_ratio": 1, "warmup_epochs": 0, "c_mel": 45, "c_kl": 1.0, "use_sr": true, "max_speclen": 512, "port": "8001", "keep_ckpts": 3, "all_in_mem": false, "vol_aug":false }, "data": { "training_files": "filelists/train.txt", "validation_files": "filelists/val.txt", "max_wav_value": 32768.0, "sampling_rate": 44100, "filter_length": 2048, "hop_length": 512, "win_length": 2048, "n_mel_channels": 80, "mel_fmin": 0.0, "mel_fmax": 22050, "unit_interpolate_mode":"nearest" }, "model": { "inter_channels": 192, "hidden_channels": 192, "filter_channels": 512, "n_heads": 2, "n_layers": 6, "kernel_size": 3, "p_dropout": 0.1, "resblock": "1", "resblock_kernel_sizes": [3,7,11], "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]], "upsample_rates": [ 8, 8, 2, 2, 2], "upsample_initial_channel": 400, "upsample_kernel_sizes": [16,16, 4, 4, 4], "n_layers_q": 3, "n_layers_trans_flow": 3, "n_flow_layer": 4, "use_spectral_norm": false, "gin_channels": 768, "ssl_dim": 768, "n_speakers": 200, "vocoder_name":"nsf-hifigan", "speech_encoder":"vec768l12", "speaker_embedding":false, "vol_embedding":false, "use_depthwise_conv":true, "flow_share_parameter": true, "use_automatic_f0_prediction": true, "use_transformer_flow": false }, "spk": { "nyaru": 0, "huiyu": 1, "nen": 2, "paimon": 3, "yunhao": 4 } }so-vits-svc-npu/configs_template/diffusion_template.yaml000066400000000000000000000031461456161405400241550ustar00rootroot00000000000000data: sampling_rate: 44100 block_size: 512 # Equal to hop_length duration: 2 # Audio duration during training, must be less than the duration of the shortest audio clip encoder: 'vec768l12' # 'hubertsoft', 'vec256l9', 'vec768l12' cnhubertsoft_gate: 10 encoder_sample_rate: 16000 encoder_hop_size: 320 encoder_out_channels: 768 # 256 if using 'hubertsoft' training_files: "filelists/train.txt" validation_files: "filelists/val.txt" extensions: # List of extension included in the data collection - wav unit_interpolate_mode: "nearest" model: type: 'Diffusion' n_layers: 20 n_chans: 512 n_hidden: 256 use_pitch_aug: true timesteps : 1000 k_step_max: 0 # must <= timesteps, If it is 0, train all n_spk: 1 # max number of different speakers device: cuda vocoder: type: 'nsf-hifigan' ckpt: 'pretrain/nsf_hifigan/model' infer: speedup: 10 method: 'dpm-solver++' # 'pndm' or 'dpm-solver' or 'ddim' or 'unipc' or 'dpm-solver++' env: expdir: logs/44k/diffusion gpu_id: 0 train: num_workers: 4 # If your cpu and gpu are both very strong, set to 0 may be faster! amp_dtype: fp32 # fp32, fp16 or bf16 (fp16 or bf16 may be faster if it is supported by your gpu) batch_size: 48 cache_all_data: true # Save Internal-Memory or Graphics-Memory if it is false, but may be slow cache_device: 'cpu' # Set to 'cuda' to cache the data into the Graphics-Memory, fastest speed for strong gpu cache_fp16: true epochs: 100000 interval_log: 10 interval_val: 2000 interval_force_save: 5000 lr: 0.0001 decay_step: 100000 gamma: 0.5 weight_decay: 0 save_opt: false spk: 'nyaru': 0so-vits-svc-npu/data_utils.py000066400000000000000000000156341456161405400165750ustar00rootroot00000000000000import os import random import numpy as np import mindtorch.torch as torch import mindtorch.torch.utils.data import utils from modules.mel_processing import spectrogram_torch from utils import load_filepaths_and_text, load_wav_to_torch # import h5py """Multi speaker version""" class TextAudioSpeakerLoader(torch.utils.data.Dataset): """ 1) loads audio, speaker_id, text pairs 2) normalizes text and converts them to sequences of integers 3) computes spectrograms from audio files. """ def __init__(self, audiopaths, hparams, all_in_mem: bool = False, vol_aug: bool = True): self.audiopaths = load_filepaths_and_text(audiopaths) self.hparams = hparams self.max_wav_value = hparams.data.max_wav_value self.sampling_rate = hparams.data.sampling_rate self.filter_length = hparams.data.filter_length self.hop_length = hparams.data.hop_length self.win_length = hparams.data.win_length self.unit_interpolate_mode = hparams.data.unit_interpolate_mode self.sampling_rate = hparams.data.sampling_rate self.use_sr = hparams.train.use_sr self.spec_len = hparams.train.max_speclen self.spk_map = hparams.spk self.vol_emb = hparams.model.vol_embedding self.vol_aug = hparams.train.vol_aug and vol_aug random.seed(1234) random.shuffle(self.audiopaths) self.all_in_mem = all_in_mem if self.all_in_mem: self.cache = [self.get_audio(p[0]) for p in self.audiopaths] def get_audio(self, filename): filename = filename.replace("\\", "/") audio, sampling_rate = load_wav_to_torch(filename) if sampling_rate != self.sampling_rate: raise ValueError( "Sample Rate not match. Expect {} but got {} from {}".format( self.sampling_rate, sampling_rate, filename)) audio_norm = audio / self.max_wav_value audio_norm = audio_norm.unsqueeze(0) spec_filename = filename.replace(".wav", ".spec.pt") # Ideally, all data generated after Mar 25 should have .spec.pt if os.path.exists(spec_filename): spec = torch.load(spec_filename) else: spec = spectrogram_torch(audio_norm, self.filter_length, self.sampling_rate, self.hop_length, self.win_length, center=False) spec = torch.squeeze(spec, 0) torch.save(spec, spec_filename) spk = filename.split("/")[-2] spk = torch.LongTensor([self.spk_map[spk]]) f0, uv = np.load(filename + ".f0.npy",allow_pickle=True) f0 = torch.FloatTensor(np.array(f0,dtype=float)) uv = torch.FloatTensor(np.array(uv,dtype=float)) c = torch.load(filename+ ".soft.pt") c = utils.repeat_expand_2d(c.squeeze(0), f0.shape[0], mode=self.unit_interpolate_mode) if self.vol_emb: volume_path = filename + ".vol.npy" volume = np.load(volume_path) volume = torch.from_numpy(volume).float() else: volume = None lmin = min(c.size(-1), spec.size(-1)) assert abs(c.size(-1) - spec.size(-1)) < 3, (c.size(-1), spec.size(-1), f0.shape, filename) assert abs(audio_norm.shape[1]-lmin * self.hop_length) < 3 * self.hop_length spec, c, f0, uv = spec[:, :lmin], c[:, :lmin], f0[:lmin], uv[:lmin] audio_norm = audio_norm[:, :lmin * self.hop_length] if volume is not None: volume = volume[:lmin] return c, f0, spec, audio_norm, spk, uv, volume def random_slice(self, c, f0, spec, audio_norm, spk, uv, volume): # if spec.shape[1] < 30: # print("skip too short audio:", filename) # return None if random.choice([True, False]) and self.vol_aug and volume is not None: max_amp = float(torch.max(torch.abs(audio_norm))) + 1e-5 max_shift = min(1, np.log10(1/max_amp)) log10_vol_shift = random.uniform(-1, max_shift) audio_norm = audio_norm * (10 ** log10_vol_shift) volume = volume * (10 ** log10_vol_shift) spec = spectrogram_torch(audio_norm, self.hparams.data.filter_length, self.hparams.data.sampling_rate, self.hparams.data.hop_length, self.hparams.data.win_length, center=False)[0] if spec.shape[1] > 800: start = random.randint(0, spec.shape[1]-800) end = start + 790 spec, c, f0, uv = spec[:, start:end], c[:, start:end], f0[start:end], uv[start:end] audio_norm = audio_norm[:, start * self.hop_length : end * self.hop_length] if volume is not None: volume = volume[start:end] return c, f0, spec, audio_norm, spk, uv,volume def __getitem__(self, index): if self.all_in_mem: return self.random_slice(*self.cache[index]) else: return self.random_slice(*self.get_audio(self.audiopaths[index][0])) def __len__(self): return len(self.audiopaths) class TextAudioCollate: def __call__(self, batch): batch = [b for b in batch if b is not None] input_lengths, ids_sorted_decreasing = torch.sort( torch.LongTensor([x[0].shape[1] for x in batch]), dim=0, descending=True) max_c_len = max([x[0].size(1) for x in batch]) max_wav_len = max([x[3].size(1) for x in batch]) lengths = torch.LongTensor(len(batch)) c_padded = torch.FloatTensor(len(batch), batch[0][0].shape[0], max_c_len) f0_padded = torch.FloatTensor(len(batch), max_c_len) spec_padded = torch.FloatTensor(len(batch), batch[0][2].shape[0], max_c_len) wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len) spkids = torch.LongTensor(len(batch), 1) uv_padded = torch.FloatTensor(len(batch), max_c_len) volume_padded = torch.FloatTensor(len(batch), max_c_len) c_padded.zero_() spec_padded.zero_() f0_padded.zero_() wav_padded.zero_() uv_padded.zero_() volume_padded.zero_() for i in range(len(ids_sorted_decreasing)): row = batch[ids_sorted_decreasing[i]] c = row[0] c_padded[i, :, :c.size(1)] = c lengths[i] = c.size(1) f0 = row[1] f0_padded[i, :f0.size(0)] = f0 spec = row[2] spec_padded[i, :, :spec.size(1)] = spec wav = row[3] wav_padded[i, :, :wav.size(1)] = wav spkids[i, 0] = row[4] uv = row[5] uv_padded[i, :uv.size(0)] = uv volume = row[6] if volume is not None: volume_padded[i, :volume.size(0)] = volume else : volume_padded = None return c_padded, f0_padded, spec_padded, wav_padded, spkids, lengths, uv_padded, volume_padded so-vits-svc-npu/dataset_raw/000077500000000000000000000000001456161405400163575ustar00rootroot00000000000000so-vits-svc-npu/dataset_raw/wav_structure.txt000066400000000000000000000005131456161405400220340ustar00rootroot00000000000000数据集准备 raw ├───speaker0 │ ├───xxx1-xxx1.wav │ ├───... │ └───Lxx-0xx8.wav └───speaker1 ├───xx2-0xxx2.wav ├───... └───xxx7-xxx007.wav 此外还需要编辑config.json "n_speakers": 10 "spk":{ "speaker0": 0, "speaker1": 1, } so-vits-svc-npu/diffusion/000077500000000000000000000000001456161405400160475ustar00rootroot00000000000000so-vits-svc-npu/diffusion/__init__.py000066400000000000000000000000001456161405400201460ustar00rootroot00000000000000so-vits-svc-npu/diffusion/data_loaders.py000066400000000000000000000253011456161405400210440ustar00rootroot00000000000000import os import random import librosa import numpy as np import mindtorch.torch as torch from mindtorch.torch.utils.data import Dataset from tqdm import tqdm from utils import repeat_expand_2d def traverse_dir( root_dir, extensions, amount=None, str_include=None, str_exclude=None, is_pure=False, is_sort=False, is_ext=True): file_list = [] cnt = 0 for root, _, files in os.walk(root_dir): for file in files: if any([file.endswith(f".{ext}") for ext in extensions]): # path mix_path = os.path.join(root, file) pure_path = mix_path[len(root_dir)+1:] if is_pure else mix_path # amount if (amount is not None) and (cnt == amount): if is_sort: file_list.sort() return file_list # check string if (str_include is not None) and (str_include not in pure_path): continue if (str_exclude is not None) and (str_exclude in pure_path): continue if not is_ext: ext = pure_path.split('.')[-1] pure_path = pure_path[:-(len(ext)+1)] file_list.append(pure_path) cnt += 1 if is_sort: file_list.sort() return file_list def get_data_loaders(args, whole_audio=False): data_train = AudioDataset( filelists = args.data.training_files, waveform_sec=args.data.duration, hop_size=args.data.block_size, sample_rate=args.data.sampling_rate, load_all_data=args.train.cache_all_data, whole_audio=whole_audio, extensions=args.data.extensions, n_spk=args.model.n_spk, spk=args.spk, device=args.train.cache_device, fp16=args.train.cache_fp16, unit_interpolate_mode = args.data.unit_interpolate_mode, use_aug=True) loader_train = torch.utils.data.DataLoader( data_train , batch_size=args.train.batch_size if not whole_audio else 1, shuffle=True, num_workers=args.train.num_workers if args.train.cache_device=='cpu' else 0, persistent_workers=(args.train.num_workers > 0) if args.train.cache_device=='cpu' else False, pin_memory=True if args.train.cache_device=='cpu' else False ) data_valid = AudioDataset( filelists = args.data.validation_files, waveform_sec=args.data.duration, hop_size=args.data.block_size, sample_rate=args.data.sampling_rate, load_all_data=args.train.cache_all_data, whole_audio=True, spk=args.spk, extensions=args.data.extensions, unit_interpolate_mode = args.data.unit_interpolate_mode, n_spk=args.model.n_spk) loader_valid = torch.utils.data.DataLoader( data_valid, batch_size=1, shuffle=False, num_workers=0, pin_memory=True ) return loader_train, loader_valid class AudioDataset(Dataset): def __init__( self, filelists, waveform_sec, hop_size, sample_rate, spk, load_all_data=True, whole_audio=False, extensions=['wav'], n_spk=1, device='cpu', fp16=False, use_aug=False, unit_interpolate_mode = 'left' ): super().__init__() self.waveform_sec = waveform_sec self.sample_rate = sample_rate self.hop_size = hop_size self.filelists = filelists self.whole_audio = whole_audio self.use_aug = use_aug self.data_buffer={} self.pitch_aug_dict = {} self.unit_interpolate_mode = unit_interpolate_mode # np.load(os.path.join(self.path_root, 'pitch_aug_dict.npy'), allow_pickle=True).item() if load_all_data: print('Load all the data filelists:', filelists) else: print('Load the f0, volume data filelists:', filelists) with open(filelists,"r") as f: self.paths = f.read().splitlines() for name_ext in tqdm(self.paths, total=len(self.paths)): path_audio = name_ext duration = librosa.get_duration(filename = path_audio, sr = self.sample_rate) path_f0 = name_ext + ".f0.npy" f0,_ = np.load(path_f0,allow_pickle=True) f0 = torch.from_numpy(np.array(f0,dtype=float)).float().unsqueeze(-1).to(device) path_volume = name_ext + ".vol.npy" volume = np.load(path_volume) volume = torch.from_numpy(volume).float().unsqueeze(-1).to(device) path_augvol = name_ext + ".aug_vol.npy" aug_vol = np.load(path_augvol) aug_vol = torch.from_numpy(aug_vol).float().unsqueeze(-1).to(device) if n_spk is not None and n_spk > 1: spk_name = name_ext.split("/")[-2] spk_id = spk[spk_name] if spk_name in spk else 0 if spk_id < 0 or spk_id >= n_spk: raise ValueError(' [x] Muiti-speaker traing error : spk_id must be a positive integer from 0 to n_spk-1 ') else: spk_id = 0 spk_id = torch.LongTensor(np.array([spk_id])).to(device) if load_all_data: ''' audio, sr = librosa.load(path_audio, sr=self.sample_rate) if len(audio.shape) > 1: audio = librosa.to_mono(audio) audio = torch.from_numpy(audio).to(device) ''' path_mel = name_ext + ".mel.npy" mel = np.load(path_mel) mel = torch.from_numpy(mel).to(device) path_augmel = name_ext + ".aug_mel.npy" aug_mel,keyshift = np.load(path_augmel, allow_pickle=True) aug_mel = np.array(aug_mel,dtype=float) aug_mel = torch.from_numpy(aug_mel).to(device) self.pitch_aug_dict[name_ext] = keyshift path_units = name_ext + ".soft.pt" units = torch.load(path_units).to(device) units = units[0] units = repeat_expand_2d(units,f0.size(0),unit_interpolate_mode).transpose(0,1) if fp16: mel = mel.half() aug_mel = aug_mel.half() units = units.half() self.data_buffer[name_ext] = { 'duration': duration, 'mel': mel, 'aug_mel': aug_mel, 'units': units, 'f0': f0, 'volume': volume, 'aug_vol': aug_vol, 'spk_id': spk_id } else: path_augmel = name_ext + ".aug_mel.npy" aug_mel,keyshift = np.load(path_augmel, allow_pickle=True) self.pitch_aug_dict[name_ext] = keyshift self.data_buffer[name_ext] = { 'duration': duration, 'f0': f0, 'volume': volume, 'aug_vol': aug_vol, 'spk_id': spk_id } def __getitem__(self, file_idx): name_ext = self.paths[file_idx] data_buffer = self.data_buffer[name_ext] # check duration. if too short, then skip if data_buffer['duration'] < (self.waveform_sec + 0.1): return self.__getitem__( (file_idx + 1) % len(self.paths)) # get item return self.get_data(name_ext, data_buffer) def get_data(self, name_ext, data_buffer): name = os.path.splitext(name_ext)[0] frame_resolution = self.hop_size / self.sample_rate duration = data_buffer['duration'] waveform_sec = duration if self.whole_audio else self.waveform_sec # load audio idx_from = 0 if self.whole_audio else random.uniform(0, duration - waveform_sec - 0.1) start_frame = int(idx_from / frame_resolution) units_frame_len = int(waveform_sec / frame_resolution) aug_flag = random.choice([True, False]) and self.use_aug ''' audio = data_buffer.get('audio') if audio is None: path_audio = os.path.join(self.path_root, 'audio', name) + '.wav' audio, sr = librosa.load( path_audio, sr = self.sample_rate, offset = start_frame * frame_resolution, duration = waveform_sec) if len(audio.shape) > 1: audio = librosa.to_mono(audio) # clip audio into N seconds audio = audio[ : audio.shape[-1] // self.hop_size * self.hop_size] audio = torch.from_numpy(audio).float() else: audio = audio[start_frame * self.hop_size : (start_frame + units_frame_len) * self.hop_size] ''' # load mel mel_key = 'aug_mel' if aug_flag else 'mel' mel = data_buffer.get(mel_key) if mel is None: mel = name_ext + ".mel.npy" mel = np.load(mel) mel = mel[start_frame : start_frame + units_frame_len] mel = torch.from_numpy(mel).float() else: mel = mel[start_frame : start_frame + units_frame_len] # load f0 f0 = data_buffer.get('f0') aug_shift = 0 if aug_flag: aug_shift = self.pitch_aug_dict[name_ext] f0_frames = 2 ** (aug_shift / 12) * f0[start_frame : start_frame + units_frame_len] # load units units = data_buffer.get('units') if units is None: path_units = name_ext + ".soft.pt" units = torch.load(path_units) units = units[0] units = repeat_expand_2d(units,f0.size(0),self.unit_interpolate_mode).transpose(0,1) units = units[start_frame : start_frame + units_frame_len] # load volume vol_key = 'aug_vol' if aug_flag else 'volume' volume = data_buffer.get(vol_key) volume_frames = volume[start_frame : start_frame + units_frame_len] # load spk_id spk_id = data_buffer.get('spk_id') # load shift aug_shift = torch.from_numpy(np.array([[aug_shift]])).float() return dict(mel=mel, f0=f0_frames, volume=volume_frames, units=units, spk_id=spk_id, aug_shift=aug_shift, name=name, name_ext=name_ext) def __len__(self): return len(self.paths)so-vits-svc-npu/diffusion/diffusion.py000066400000000000000000000412351456161405400204140ustar00rootroot00000000000000from collections import deque from functools import partial from inspect import isfunction import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import nn from tqdm import tqdm def exists(x): return x is not None def default(val, d): if exists(val): return val return d() if isfunction(d) else d def extract(a, t, x_shape): b, *_ = t.shape out = a.gather(-1, t) return out.reshape(b, *((1,) * (len(x_shape) - 1))) def noise_like(shape, device, repeat=False): def repeat_noise(): return torch.randn((1, *shape[1:]), device=device).repeat(shape[0], *((1,) * (len(shape) - 1))) def noise(): return torch.randn(shape, device=device) return repeat_noise() if repeat else noise() def linear_beta_schedule(timesteps, max_beta=0.02): """ linear schedule """ betas = np.linspace(1e-4, max_beta, timesteps) return betas def cosine_beta_schedule(timesteps, s=0.008): """ cosine schedule as proposed in https://openreview.net/forum?id=-NEXDKk8gZ """ steps = timesteps + 1 x = np.linspace(0, steps, steps) alphas_cumprod = np.cos(((x / steps) + s) / (1 + s) * np.pi * 0.5) ** 2 alphas_cumprod = alphas_cumprod / alphas_cumprod[0] betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1]) return np.clip(betas, a_min=0, a_max=0.999) beta_schedule = { "cosine": cosine_beta_schedule, "linear": linear_beta_schedule, } class GaussianDiffusion(nn.Module): def __init__(self, denoise_fn, out_dims=128, timesteps=1000, k_step=1000, max_beta=0.02, spec_min=-12, spec_max=2): super().__init__() self.denoise_fn = denoise_fn self.out_dims = out_dims betas = beta_schedule['linear'](timesteps, max_beta=max_beta) alphas = 1. - betas alphas_cumprod = np.cumprod(alphas, axis=0) alphas_cumprod_prev = np.append(1., alphas_cumprod[:-1]) timesteps, = betas.shape self.num_timesteps = int(timesteps) self.k_step = k_step if k_step>0 and k_step 1: if method == 'dpm-solver' or method == 'dpm-solver++': from .dpm_solver_pytorch import ( DPM_Solver, NoiseScheduleVP, model_wrapper, ) # 1. Define the noise schedule. noise_schedule = NoiseScheduleVP(schedule='discrete', betas=self.betas[:t]) # 2. Convert your discrete-time `model` to the continuous-time # noise prediction model. Here is an example for a diffusion model # `model` with the noise prediction type ("noise") . def my_wrapper(fn): def wrapped(x, t, **kwargs): ret = fn(x, t, **kwargs) if use_tqdm: self.bar.update(1) return ret return wrapped model_fn = model_wrapper( my_wrapper(self.denoise_fn), noise_schedule, model_type="noise", # or "x_start" or "v" or "score" model_kwargs={"cond": cond} ) # 3. Define dpm-solver and sample by singlestep DPM-Solver. # (We recommend singlestep DPM-Solver for unconditional sampling) # You can adjust the `steps` to balance the computation # costs and the sample quality. if method == 'dpm-solver': dpm_solver = DPM_Solver(model_fn, noise_schedule, algorithm_type="dpmsolver") elif method == 'dpm-solver++': dpm_solver = DPM_Solver(model_fn, noise_schedule, algorithm_type="dpmsolver++") steps = t // infer_speedup if use_tqdm: self.bar = tqdm(desc="sample time step", total=steps) x = dpm_solver.sample( x, steps=steps, order=2, skip_type="time_uniform", method="multistep", ) if use_tqdm: self.bar.close() elif method == 'pndm': self.noise_list = deque(maxlen=4) if use_tqdm: for i in tqdm( reversed(range(0, t, infer_speedup)), desc='sample time step', total=t // infer_speedup, ): x = self.p_sample_plms( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) else: for i in reversed(range(0, t, infer_speedup)): x = self.p_sample_plms( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) elif method == 'ddim': if use_tqdm: for i in tqdm( reversed(range(0, t, infer_speedup)), desc='sample time step', total=t // infer_speedup, ): x = self.p_sample_ddim( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) else: for i in reversed(range(0, t, infer_speedup)): x = self.p_sample_ddim( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) elif method == 'unipc': from .uni_pc import NoiseScheduleVP, UniPC, model_wrapper # 1. Define the noise schedule. noise_schedule = NoiseScheduleVP(schedule='discrete', betas=self.betas[:t]) # 2. Convert your discrete-time `model` to the continuous-time # noise prediction model. Here is an example for a diffusion model # `model` with the noise prediction type ("noise") . def my_wrapper(fn): def wrapped(x, t, **kwargs): ret = fn(x, t, **kwargs) if use_tqdm: self.bar.update(1) return ret return wrapped model_fn = model_wrapper( my_wrapper(self.denoise_fn), noise_schedule, model_type="noise", # or "x_start" or "v" or "score" model_kwargs={"cond": cond} ) # 3. Define uni_pc and sample by multistep UniPC. # You can adjust the `steps` to balance the computation # costs and the sample quality. uni_pc = UniPC(model_fn, noise_schedule, variant='bh2') steps = t // infer_speedup if use_tqdm: self.bar = tqdm(desc="sample time step", total=steps) x = uni_pc.sample( x, steps=steps, order=2, skip_type="time_uniform", method="multistep", ) if use_tqdm: self.bar.close() else: raise NotImplementedError(method) else: if use_tqdm: for i in tqdm(reversed(range(0, t)), desc='sample time step', total=t): x = self.p_sample(x, torch.full((b,), i, device=device, dtype=torch.long), cond) else: for i in reversed(range(0, t)): x = self.p_sample(x, torch.full((b,), i, device=device, dtype=torch.long), cond) x = x.squeeze(1).transpose(1, 2) # [B, T, M] return self.denorm_spec(x) def norm_spec(self, x): return (x - self.spec_min) / (self.spec_max - self.spec_min) * 2 - 1 def denorm_spec(self, x): return (x + 1) / 2 * (self.spec_max - self.spec_min) + self.spec_min so-vits-svc-npu/diffusion/diffusion_onnx.py000066400000000000000000000601351456161405400214560ustar00rootroot00000000000000import math from collections import deque from functools import partial from inspect import isfunction import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import nn from mindtorch.torch.nn import Conv1d, Mish from tqdm import tqdm def exists(x): return x is not None def default(val, d): if exists(val): return val return d() if isfunction(d) else d def extract(a, t): return a[t].reshape((1, 1, 1, 1)) def noise_like(shape, device, repeat=False): def repeat_noise(): return torch.randn((1, *shape[1:]), device=device).repeat(shape[0], *((1,) * (len(shape) - 1))) def noise(): return torch.randn(shape, device=device) return repeat_noise() if repeat else noise() def linear_beta_schedule(timesteps, max_beta=0.02): """ linear schedule """ betas = np.linspace(1e-4, max_beta, timesteps) return betas def cosine_beta_schedule(timesteps, s=0.008): """ cosine schedule as proposed in https://openreview.net/forum?id=-NEXDKk8gZ """ steps = timesteps + 1 x = np.linspace(0, steps, steps) alphas_cumprod = np.cos(((x / steps) + s) / (1 + s) * np.pi * 0.5) ** 2 alphas_cumprod = alphas_cumprod / alphas_cumprod[0] betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1]) return np.clip(betas, a_min=0, a_max=0.999) beta_schedule = { "cosine": cosine_beta_schedule, "linear": linear_beta_schedule, } def extract_1(a, t): return a[t].reshape((1, 1, 1, 1)) def predict_stage0(noise_pred, noise_pred_prev): return (noise_pred + noise_pred_prev) / 2 def predict_stage1(noise_pred, noise_list): return (noise_pred * 3 - noise_list[-1]) / 2 def predict_stage2(noise_pred, noise_list): return (noise_pred * 23 - noise_list[-1] * 16 + noise_list[-2] * 5) / 12 def predict_stage3(noise_pred, noise_list): return (noise_pred * 55 - noise_list[-1] * 59 + noise_list[-2] * 37 - noise_list[-3] * 9) / 24 class SinusoidalPosEmb(nn.Module): def __init__(self, dim): super().__init__() self.dim = dim self.half_dim = dim // 2 self.emb = 9.21034037 / (self.half_dim - 1) self.emb = torch.exp(torch.arange(self.half_dim) * torch.tensor(-self.emb)).unsqueeze(0) self.emb = self.emb.cpu() def forward(self, x): emb = self.emb * x emb = torch.cat((emb.sin(), emb.cos()), dim=-1) return emb class ResidualBlock(nn.Module): def __init__(self, encoder_hidden, residual_channels, dilation): super().__init__() self.residual_channels = residual_channels self.dilated_conv = Conv1d(residual_channels, 2 * residual_channels, 3, padding=dilation, dilation=dilation) self.diffusion_projection = nn.Linear(residual_channels, residual_channels) self.conditioner_projection = Conv1d(encoder_hidden, 2 * residual_channels, 1) self.output_projection = Conv1d(residual_channels, 2 * residual_channels, 1) def forward(self, x, conditioner, diffusion_step): diffusion_step = self.diffusion_projection(diffusion_step).unsqueeze(-1) conditioner = self.conditioner_projection(conditioner) y = x + diffusion_step y = self.dilated_conv(y) + conditioner gate, filter_1 = torch.split(y, [self.residual_channels, self.residual_channels], dim=1) y = torch.sigmoid(gate) * torch.tanh(filter_1) y = self.output_projection(y) residual, skip = torch.split(y, [self.residual_channels, self.residual_channels], dim=1) return (x + residual) / 1.41421356, skip class DiffNet(nn.Module): def __init__(self, in_dims, n_layers, n_chans, n_hidden): super().__init__() self.encoder_hidden = n_hidden self.residual_layers = n_layers self.residual_channels = n_chans self.input_projection = Conv1d(in_dims, self.residual_channels, 1) self.diffusion_embedding = SinusoidalPosEmb(self.residual_channels) dim = self.residual_channels self.mlp = nn.Sequential( nn.Linear(dim, dim * 4), Mish(), nn.Linear(dim * 4, dim) ) self.residual_layers = nn.ModuleList([ ResidualBlock(self.encoder_hidden, self.residual_channels, 1) for i in range(self.residual_layers) ]) self.skip_projection = Conv1d(self.residual_channels, self.residual_channels, 1) self.output_projection = Conv1d(self.residual_channels, in_dims, 1) nn.init.zeros_(self.output_projection.weight) def forward(self, spec, diffusion_step, cond): x = spec.squeeze(0) x = self.input_projection(x) # x [B, residual_channel, T] x = F.relu(x) # skip = torch.randn_like(x) diffusion_step = diffusion_step.float() diffusion_step = self.diffusion_embedding(diffusion_step) diffusion_step = self.mlp(diffusion_step) x, skip = self.residual_layers[0](x, cond, diffusion_step) # noinspection PyTypeChecker for layer in self.residual_layers[1:]: x, skip_connection = layer.forward(x, cond, diffusion_step) skip = skip + skip_connection x = skip / math.sqrt(len(self.residual_layers)) x = self.skip_projection(x) x = F.relu(x) x = self.output_projection(x) # [B, 80, T] return x.unsqueeze(1) class AfterDiffusion(nn.Module): def __init__(self, spec_max, spec_min, v_type='a'): super().__init__() self.spec_max = spec_max self.spec_min = spec_min self.type = v_type def forward(self, x): x = x.squeeze(1).permute(0, 2, 1) mel_out = (x + 1) / 2 * (self.spec_max - self.spec_min) + self.spec_min if self.type == 'nsf-hifigan-log10': mel_out = mel_out * 0.434294 return mel_out.transpose(2, 1) class Pred(nn.Module): def __init__(self, alphas_cumprod): super().__init__() self.alphas_cumprod = alphas_cumprod def forward(self, x_1, noise_t, t_1, t_prev): a_t = extract(self.alphas_cumprod, t_1).cpu() a_prev = extract(self.alphas_cumprod, t_prev).cpu() a_t_sq, a_prev_sq = a_t.sqrt().cpu(), a_prev.sqrt().cpu() x_delta = (a_prev - a_t) * ((1 / (a_t_sq * (a_t_sq + a_prev_sq))) * x_1 - 1 / ( a_t_sq * (((1 - a_prev) * a_t).sqrt() + ((1 - a_t) * a_prev).sqrt())) * noise_t) x_pred = x_1 + x_delta.cpu() return x_pred class GaussianDiffusion(nn.Module): def __init__(self, out_dims=128, n_layers=20, n_chans=384, n_hidden=256, timesteps=1000, k_step=1000, max_beta=0.02, spec_min=-12, spec_max=2): super().__init__() self.denoise_fn = DiffNet(out_dims, n_layers, n_chans, n_hidden) self.out_dims = out_dims self.mel_bins = out_dims self.n_hidden = n_hidden betas = beta_schedule['linear'](timesteps, max_beta=max_beta) alphas = 1. - betas alphas_cumprod = np.cumprod(alphas, axis=0) alphas_cumprod_prev = np.append(1., alphas_cumprod[:-1]) timesteps, = betas.shape self.num_timesteps = int(timesteps) self.k_step = k_step self.noise_list = deque(maxlen=4) to_torch = partial(torch.tensor, dtype=torch.float32) self.register_buffer('betas', to_torch(betas)) self.register_buffer('alphas_cumprod', to_torch(alphas_cumprod)) self.register_buffer('alphas_cumprod_prev', to_torch(alphas_cumprod_prev)) # calculations for diffusion q(x_t | x_{t-1}) and others self.register_buffer('sqrt_alphas_cumprod', to_torch(np.sqrt(alphas_cumprod))) self.register_buffer('sqrt_one_minus_alphas_cumprod', to_torch(np.sqrt(1. - alphas_cumprod))) self.register_buffer('log_one_minus_alphas_cumprod', to_torch(np.log(1. - alphas_cumprod))) self.register_buffer('sqrt_recip_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod))) self.register_buffer('sqrt_recipm1_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod - 1))) # calculations for posterior q(x_{t-1} | x_t, x_0) posterior_variance = betas * (1. - alphas_cumprod_prev) / (1. - alphas_cumprod) # above: equal to 1. / (1. / (1. - alpha_cumprod_tm1) + alpha_t / beta_t) self.register_buffer('posterior_variance', to_torch(posterior_variance)) # below: log calculation clipped because the posterior variance is 0 at the beginning of the diffusion chain self.register_buffer('posterior_log_variance_clipped', to_torch(np.log(np.maximum(posterior_variance, 1e-20)))) self.register_buffer('posterior_mean_coef1', to_torch( betas * np.sqrt(alphas_cumprod_prev) / (1. - alphas_cumprod))) self.register_buffer('posterior_mean_coef2', to_torch( (1. - alphas_cumprod_prev) * np.sqrt(alphas) / (1. - alphas_cumprod))) self.register_buffer('spec_min', torch.FloatTensor([spec_min])[None, None, :out_dims]) self.register_buffer('spec_max', torch.FloatTensor([spec_max])[None, None, :out_dims]) self.ad = AfterDiffusion(self.spec_max, self.spec_min) self.xp = Pred(self.alphas_cumprod) def q_mean_variance(self, x_start, t): mean = extract(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start variance = extract(1. - self.alphas_cumprod, t, x_start.shape) log_variance = extract(self.log_one_minus_alphas_cumprod, t, x_start.shape) return mean, variance, log_variance def predict_start_from_noise(self, x_t, t, noise): return ( extract(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t - extract(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape) * noise ) def q_posterior(self, x_start, x_t, t): posterior_mean = ( extract(self.posterior_mean_coef1, t, x_t.shape) * x_start + extract(self.posterior_mean_coef2, t, x_t.shape) * x_t ) posterior_variance = extract(self.posterior_variance, t, x_t.shape) posterior_log_variance_clipped = extract(self.posterior_log_variance_clipped, t, x_t.shape) return posterior_mean, posterior_variance, posterior_log_variance_clipped def p_mean_variance(self, x, t, cond): noise_pred = self.denoise_fn(x, t, cond=cond) x_recon = self.predict_start_from_noise(x, t=t, noise=noise_pred) x_recon.clamp_(-1., 1.) model_mean, posterior_variance, posterior_log_variance = self.q_posterior(x_start=x_recon, x_t=x, t=t) return model_mean, posterior_variance, posterior_log_variance @torch.no_grad() def p_sample(self, x, t, cond, clip_denoised=True, repeat_noise=False): b, *_, device = *x.shape, x.device model_mean, _, model_log_variance = self.p_mean_variance(x=x, t=t, cond=cond) noise = noise_like(x.shape, device, repeat_noise) # no noise when t == 0 nonzero_mask = (1 - (t == 0).float()).reshape(b, *((1,) * (len(x.shape) - 1))) return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise @torch.no_grad() def p_sample_plms(self, x, t, interval, cond, clip_denoised=True, repeat_noise=False): """ Use the PLMS method from [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://arxiv.org/abs/2202.09778). """ def get_x_pred(x, noise_t, t): a_t = extract(self.alphas_cumprod, t) a_prev = extract(self.alphas_cumprod, torch.max(t - interval, torch.zeros_like(t))) a_t_sq, a_prev_sq = a_t.sqrt(), a_prev.sqrt() x_delta = (a_prev - a_t) * ((1 / (a_t_sq * (a_t_sq + a_prev_sq))) * x - 1 / ( a_t_sq * (((1 - a_prev) * a_t).sqrt() + ((1 - a_t) * a_prev).sqrt())) * noise_t) x_pred = x + x_delta return x_pred noise_list = self.noise_list noise_pred = self.denoise_fn(x, t, cond=cond) if len(noise_list) == 0: x_pred = get_x_pred(x, noise_pred, t) noise_pred_prev = self.denoise_fn(x_pred, max(t - interval, 0), cond=cond) noise_pred_prime = (noise_pred + noise_pred_prev) / 2 elif len(noise_list) == 1: noise_pred_prime = (3 * noise_pred - noise_list[-1]) / 2 elif len(noise_list) == 2: noise_pred_prime = (23 * noise_pred - 16 * noise_list[-1] + 5 * noise_list[-2]) / 12 else: noise_pred_prime = (55 * noise_pred - 59 * noise_list[-1] + 37 * noise_list[-2] - 9 * noise_list[-3]) / 24 x_prev = get_x_pred(x, noise_pred_prime, t) noise_list.append(noise_pred) return x_prev def q_sample(self, x_start, t, noise=None): noise = default(noise, lambda: torch.randn_like(x_start)) return ( extract(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start + extract(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise ) def p_losses(self, x_start, t, cond, noise=None, loss_type='l2'): noise = default(noise, lambda: torch.randn_like(x_start)) x_noisy = self.q_sample(x_start=x_start, t=t, noise=noise) x_recon = self.denoise_fn(x_noisy, t, cond) if loss_type == 'l1': loss = (noise - x_recon).abs().mean() elif loss_type == 'l2': loss = F.mse_loss(noise, x_recon) else: raise NotImplementedError() return loss def org_forward(self, condition, init_noise=None, gt_spec=None, infer=True, infer_speedup=100, method='pndm', k_step=1000, use_tqdm=True): """ conditioning diffusion, use fastspeech2 encoder output as the condition """ cond = condition b, device = condition.shape[0], condition.device if not infer: spec = self.norm_spec(gt_spec) t = torch.randint(0, self.k_step, (b,), device=device).long() norm_spec = spec.transpose(1, 2)[:, None, :, :] # [B, 1, M, T] return self.p_losses(norm_spec, t, cond=cond) else: shape = (cond.shape[0], 1, self.out_dims, cond.shape[2]) if gt_spec is None: t = self.k_step if init_noise is None: x = torch.randn(shape, device=device) else: x = init_noise else: t = k_step norm_spec = self.norm_spec(gt_spec) norm_spec = norm_spec.transpose(1, 2)[:, None, :, :] x = self.q_sample(x_start=norm_spec, t=torch.tensor([t - 1], device=device).long()) if method is not None and infer_speedup > 1: if method == 'dpm-solver': from .dpm_solver_pytorch import ( DPM_Solver, NoiseScheduleVP, model_wrapper, ) # 1. Define the noise schedule. noise_schedule = NoiseScheduleVP(schedule='discrete', betas=self.betas[:t]) # 2. Convert your discrete-time `model` to the continuous-time # noise prediction model. Here is an example for a diffusion model # `model` with the noise prediction type ("noise") . def my_wrapper(fn): def wrapped(x, t, **kwargs): ret = fn(x, t, **kwargs) if use_tqdm: self.bar.update(1) return ret return wrapped model_fn = model_wrapper( my_wrapper(self.denoise_fn), noise_schedule, model_type="noise", # or "x_start" or "v" or "score" model_kwargs={"cond": cond} ) # 3. Define dpm-solver and sample by singlestep DPM-Solver. # (We recommend singlestep DPM-Solver for unconditional sampling) # You can adjust the `steps` to balance the computation # costs and the sample quality. dpm_solver = DPM_Solver(model_fn, noise_schedule) steps = t // infer_speedup if use_tqdm: self.bar = tqdm(desc="sample time step", total=steps) x = dpm_solver.sample( x, steps=steps, order=3, skip_type="time_uniform", method="singlestep", ) if use_tqdm: self.bar.close() elif method == 'pndm': self.noise_list = deque(maxlen=4) if use_tqdm: for i in tqdm( reversed(range(0, t, infer_speedup)), desc='sample time step', total=t // infer_speedup, ): x = self.p_sample_plms( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) else: for i in reversed(range(0, t, infer_speedup)): x = self.p_sample_plms( x, torch.full((b,), i, device=device, dtype=torch.long), infer_speedup, cond=cond ) else: raise NotImplementedError(method) else: if use_tqdm: for i in tqdm(reversed(range(0, t)), desc='sample time step', total=t): x = self.p_sample(x, torch.full((b,), i, device=device, dtype=torch.long), cond) else: for i in reversed(range(0, t)): x = self.p_sample(x, torch.full((b,), i, device=device, dtype=torch.long), cond) x = x.squeeze(1).transpose(1, 2) # [B, T, M] return self.denorm_spec(x).transpose(2, 1) def norm_spec(self, x): return (x - self.spec_min) / (self.spec_max - self.spec_min) * 2 - 1 def denorm_spec(self, x): return (x + 1) / 2 * (self.spec_max - self.spec_min) + self.spec_min def get_x_pred(self, x_1, noise_t, t_1, t_prev): a_t = extract(self.alphas_cumprod, t_1) a_prev = extract(self.alphas_cumprod, t_prev) a_t_sq, a_prev_sq = a_t.sqrt(), a_prev.sqrt() x_delta = (a_prev - a_t) * ((1 / (a_t_sq * (a_t_sq + a_prev_sq))) * x_1 - 1 / ( a_t_sq * (((1 - a_prev) * a_t).sqrt() + ((1 - a_t) * a_prev).sqrt())) * noise_t) x_pred = x_1 + x_delta return x_pred def OnnxExport(self, project_name=None, init_noise=None, hidden_channels=256, export_denoise=True, export_pred=True, export_after=True): cond = torch.randn([1, self.n_hidden, 10]).cpu() if init_noise is None: x = torch.randn((1, 1, self.mel_bins, cond.shape[2]), dtype=torch.float32).cpu() else: x = init_noise pndms = 100 org_y_x = self.org_forward(cond, init_noise=x) device = cond.device n_frames = cond.shape[2] step_range = torch.arange(0, self.k_step, pndms, dtype=torch.long, device=device).flip(0) plms_noise_stage = torch.tensor(0, dtype=torch.long, device=device) noise_list = torch.zeros((0, 1, 1, self.mel_bins, n_frames), device=device) ot = step_range[0] ot_1 = torch.full((1,), ot, device=device, dtype=torch.long) if export_denoise: torch.onnx.export( self.denoise_fn, (x.cpu(), ot_1.cpu(), cond.cpu()), f"{project_name}_denoise.onnx", input_names=["noise", "time", "condition"], output_names=["noise_pred"], dynamic_axes={ "noise": [3], "condition": [2] }, opset_version=16 ) for t in step_range: t_1 = torch.full((1,), t, device=device, dtype=torch.long) noise_pred = self.denoise_fn(x, t_1, cond) t_prev = t_1 - pndms t_prev = t_prev * (t_prev > 0) if plms_noise_stage == 0: if export_pred: torch.onnx.export( self.xp, (x.cpu(), noise_pred.cpu(), t_1.cpu(), t_prev.cpu()), f"{project_name}_pred.onnx", input_names=["noise", "noise_pred", "time", "time_prev"], output_names=["noise_pred_o"], dynamic_axes={ "noise": [3], "noise_pred": [3] }, opset_version=16 ) x_pred = self.get_x_pred(x, noise_pred, t_1, t_prev) noise_pred_prev = self.denoise_fn(x_pred, t_prev, cond=cond) noise_pred_prime = predict_stage0(noise_pred, noise_pred_prev) elif plms_noise_stage == 1: noise_pred_prime = predict_stage1(noise_pred, noise_list) elif plms_noise_stage == 2: noise_pred_prime = predict_stage2(noise_pred, noise_list) else: noise_pred_prime = predict_stage3(noise_pred, noise_list) noise_pred = noise_pred.unsqueeze(0) if plms_noise_stage < 3: noise_list = torch.cat((noise_list, noise_pred), dim=0) plms_noise_stage = plms_noise_stage + 1 else: noise_list = torch.cat((noise_list[-2:], noise_pred), dim=0) x = self.get_x_pred(x, noise_pred_prime, t_1, t_prev) if export_after: torch.onnx.export( self.ad, x.cpu(), f"{project_name}_after.onnx", input_names=["x"], output_names=["mel_out"], dynamic_axes={ "x": [3] }, opset_version=16 ) x = self.ad(x) print((x == org_y_x).all()) return x def forward(self, condition=None, init_noise=None, pndms=None, k_step=None): cond = condition x = init_noise device = cond.device n_frames = cond.shape[2] step_range = torch.arange(0, k_step.item(), pndms.item(), dtype=torch.long, device=device).flip(0) plms_noise_stage = torch.tensor(0, dtype=torch.long, device=device) noise_list = torch.zeros((0, 1, 1, self.mel_bins, n_frames), device=device) for t in step_range: t_1 = torch.full((1,), t, device=device, dtype=torch.long) noise_pred = self.denoise_fn(x, t_1, cond) t_prev = t_1 - pndms t_prev = t_prev * (t_prev > 0) if plms_noise_stage == 0: x_pred = self.get_x_pred(x, noise_pred, t_1, t_prev) noise_pred_prev = self.denoise_fn(x_pred, t_prev, cond=cond) noise_pred_prime = predict_stage0(noise_pred, noise_pred_prev) elif plms_noise_stage == 1: noise_pred_prime = predict_stage1(noise_pred, noise_list) elif plms_noise_stage == 2: noise_pred_prime = predict_stage2(noise_pred, noise_list) else: noise_pred_prime = predict_stage3(noise_pred, noise_list) noise_pred = noise_pred.unsqueeze(0) if plms_noise_stage < 3: noise_list = torch.cat((noise_list, noise_pred), dim=0) plms_noise_stage = plms_noise_stage + 1 else: noise_list = torch.cat((noise_list[-2:], noise_pred), dim=0) x = self.get_x_pred(x, noise_pred_prime, t_1, t_prev) x = self.ad(x) return x so-vits-svc-npu/diffusion/dpm_solver_pytorch.py000066400000000000000000002052521456161405400223510ustar00rootroot00000000000000import mindtorch.torch as torch class NoiseScheduleVP: def __init__( self, schedule='discrete', betas=None, alphas_cumprod=None, continuous_beta_0=0.1, continuous_beta_1=20., dtype=torch.float32, ): """Create a wrapper class for the forward SDE (VP type). *** Update: We support discrete-time diffusion models by implementing a picewise linear interpolation for log_alpha_t. We recommend to use schedule='discrete' for the discrete-time diffusion models, especially for high-resolution images. *** The forward SDE ensures that the condition distribution q_{t|0}(x_t | x_0) = N ( alpha_t * x_0, sigma_t^2 * I ). We further define lambda_t = log(alpha_t) - log(sigma_t), which is the half-logSNR (described in the DPM-Solver paper). Therefore, we implement the functions for computing alpha_t, sigma_t and lambda_t. For t in [0, T], we have: log_alpha_t = self.marginal_log_mean_coeff(t) sigma_t = self.marginal_std(t) lambda_t = self.marginal_lambda(t) Moreover, as lambda(t) is an invertible function, we also support its inverse function: t = self.inverse_lambda(lambda_t) =============================================================== We support both discrete-time DPMs (trained on n = 0, 1, ..., N-1) and continuous-time DPMs (trained on t in [t_0, T]). 1. For discrete-time DPMs: For discrete-time DPMs trained on n = 0, 1, ..., N-1, we convert the discrete steps to continuous time steps by: t_i = (i + 1) / N e.g. for N = 1000, we have t_0 = 1e-3 and T = t_{N-1} = 1. We solve the corresponding diffusion ODE from time T = 1 to time t_0 = 1e-3. Args: betas: A `torch.Tensor`. The beta array for the discrete-time DPM. (See the original DDPM paper for details) alphas_cumprod: A `torch.Tensor`. The cumprod alphas for the discrete-time DPM. (See the original DDPM paper for details) Note that we always have alphas_cumprod = cumprod(1 - betas). Therefore, we only need to set one of `betas` and `alphas_cumprod`. **Important**: Please pay special attention for the args for `alphas_cumprod`: The `alphas_cumprod` is the \hat{alpha_n} arrays in the notations of DDPM. Specifically, DDPMs assume that q_{t_n | 0}(x_{t_n} | x_0) = N ( \sqrt{\hat{alpha_n}} * x_0, (1 - \hat{alpha_n}) * I ). Therefore, the notation \hat{alpha_n} is different from the notation alpha_t in DPM-Solver. In fact, we have alpha_{t_n} = \sqrt{\hat{alpha_n}}, and log(alpha_{t_n}) = 0.5 * log(\hat{alpha_n}). 2. For continuous-time DPMs: We support the linear VPSDE for the continuous time setting. The hyperparameters for the noise schedule are the default settings in Yang Song's ScoreSDE: Args: beta_min: A `float` number. The smallest beta for the linear schedule. beta_max: A `float` number. The largest beta for the linear schedule. T: A `float` number. The ending time of the forward process. =============================================================== Args: schedule: A `str`. The noise schedule of the forward SDE. 'discrete' for discrete-time DPMs, 'linear' for continuous-time DPMs. Returns: A wrapper object of the forward SDE (VP type). =============================================================== Example: # For discrete-time DPMs, given betas (the beta array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', betas=betas) # For discrete-time DPMs, given alphas_cumprod (the \hat{alpha_n} array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', alphas_cumprod=alphas_cumprod) # For continuous-time DPMs (VPSDE), linear schedule: >>> ns = NoiseScheduleVP('linear', continuous_beta_0=0.1, continuous_beta_1=20.) """ if schedule not in ['discrete', 'linear']: raise ValueError("Unsupported noise schedule {}. The schedule needs to be 'discrete' or 'linear'".format(schedule)) self.schedule = schedule if schedule == 'discrete': if betas is not None: log_alphas = 0.5 * torch.log(1 - betas).cumsum(dim=0) else: assert alphas_cumprod is not None log_alphas = 0.5 * torch.log(alphas_cumprod) self.T = 1. self.log_alpha_array = self.numerical_clip_alpha(log_alphas).reshape((1, -1,)).to(dtype=dtype) self.total_N = self.log_alpha_array.shape[1] self.t_array = torch.linspace(0., 1., self.total_N + 1)[1:].reshape((1, -1)).to(dtype=dtype) else: self.T = 1. self.total_N = 1000 self.beta_0 = continuous_beta_0 self.beta_1 = continuous_beta_1 def numerical_clip_alpha(self, log_alphas, clipped_lambda=-5.1): """ For some beta schedules such as cosine schedule, the log-SNR has numerical isssues. We clip the log-SNR near t=T within -5.1 to ensure the stability. Such a trick is very useful for diffusion models with the cosine schedule, such as i-DDPM, guided-diffusion and GLIDE. """ log_sigmas = 0.5 * torch.log(1. - torch.exp(2. * log_alphas)) lambs = log_alphas - log_sigmas idx = torch.searchsorted(torch.flip(lambs, [0]), clipped_lambda) if idx > 0: log_alphas = log_alphas[:-idx] return log_alphas def marginal_log_mean_coeff(self, t): """ Compute log(alpha_t) of a given continuous-time label t in [0, T]. """ if self.schedule == 'discrete': return interpolate_fn(t.reshape((-1, 1)), self.t_array.to(t.device), self.log_alpha_array.to(t.device)).reshape((-1)) elif self.schedule == 'linear': return -0.25 * t ** 2 * (self.beta_1 - self.beta_0) - 0.5 * t * self.beta_0 def marginal_alpha(self, t): """ Compute alpha_t of a given continuous-time label t in [0, T]. """ return torch.exp(self.marginal_log_mean_coeff(t)) def marginal_std(self, t): """ Compute sigma_t of a given continuous-time label t in [0, T]. """ return torch.sqrt(1. - torch.exp(2. * self.marginal_log_mean_coeff(t))) def marginal_lambda(self, t): """ Compute lambda_t = log(alpha_t) - log(sigma_t) of a given continuous-time label t in [0, T]. """ log_mean_coeff = self.marginal_log_mean_coeff(t) log_std = 0.5 * torch.log(1. - torch.exp(2. * log_mean_coeff)) return log_mean_coeff - log_std def inverse_lambda(self, lamb): """ Compute the continuous-time label t in [0, T] of a given half-logSNR lambda_t. """ if self.schedule == 'linear': tmp = 2. * (self.beta_1 - self.beta_0) * torch.logaddexp(-2. * lamb, torch.zeros((1,)).to(lamb)) Delta = self.beta_0**2 + tmp return tmp / (torch.sqrt(Delta) + self.beta_0) / (self.beta_1 - self.beta_0) elif self.schedule == 'discrete': log_alpha = -0.5 * torch.logaddexp(torch.zeros((1,)).to(lamb.device), -2. * lamb) t = interpolate_fn(log_alpha.reshape((-1, 1)), torch.flip(self.log_alpha_array.to(lamb.device), [1]), torch.flip(self.t_array.to(lamb.device), [1])) return t.reshape((-1,)) def model_wrapper( model, noise_schedule, model_type="noise", model_kwargs={}, guidance_type="uncond", condition=None, unconditional_condition=None, guidance_scale=1., classifier_fn=None, classifier_kwargs={}, ): """Create a wrapper function for the noise prediction model. DPM-Solver needs to solve the continuous-time diffusion ODEs. For DPMs trained on discrete-time labels, we need to firstly wrap the model function to a noise prediction model that accepts the continuous time as the input. We support four types of the diffusion model by setting `model_type`: 1. "noise": noise prediction model. (Trained by predicting noise). 2. "x_start": data prediction model. (Trained by predicting the data x_0 at time 0). 3. "v": velocity prediction model. (Trained by predicting the velocity). The "v" prediction is derivation detailed in Appendix D of [1], and is used in Imagen-Video [2]. [1] Salimans, Tim, and Jonathan Ho. "Progressive distillation for fast sampling of diffusion models." arXiv preprint arXiv:2202.00512 (2022). [2] Ho, Jonathan, et al. "Imagen Video: High Definition Video Generation with Diffusion Models." arXiv preprint arXiv:2210.02303 (2022). 4. "score": marginal score function. (Trained by denoising score matching). Note that the score function and the noise prediction model follows a simple relationship: ``` noise(x_t, t) = -sigma_t * score(x_t, t) ``` We support three types of guided sampling by DPMs by setting `guidance_type`: 1. "uncond": unconditional sampling by DPMs. The input `model` has the following format: `` model(x, t_input, **model_kwargs) -> noise | x_start | v | score `` 2. "classifier": classifier guidance sampling [3] by DPMs and another classifier. The input `model` has the following format: `` model(x, t_input, **model_kwargs) -> noise | x_start | v | score `` The input `classifier_fn` has the following format: `` classifier_fn(x, t_input, cond, **classifier_kwargs) -> logits(x, t_input, cond) `` [3] P. Dhariwal and A. Q. Nichol, "Diffusion models beat GANs on image synthesis," in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 8780-8794. 3. "classifier-free": classifier-free guidance sampling by conditional DPMs. The input `model` has the following format: `` model(x, t_input, cond, **model_kwargs) -> noise | x_start | v | score `` And if cond == `unconditional_condition`, the model output is the unconditional DPM output. [4] Ho, Jonathan, and Tim Salimans. "Classifier-free diffusion guidance." arXiv preprint arXiv:2207.12598 (2022). The `t_input` is the time label of the model, which may be discrete-time labels (i.e. 0 to 999) or continuous-time labels (i.e. epsilon to T). We wrap the model function to accept only `x` and `t_continuous` as inputs, and outputs the predicted noise: `` def model_fn(x, t_continuous) -> noise: t_input = get_model_input_time(t_continuous) return noise_pred(model, x, t_input, **model_kwargs) `` where `t_continuous` is the continuous time labels (i.e. epsilon to T). And we use `model_fn` for DPM-Solver. =============================================================== Args: model: A diffusion model with the corresponding format described above. noise_schedule: A noise schedule object, such as NoiseScheduleVP. model_type: A `str`. The parameterization type of the diffusion model. "noise" or "x_start" or "v" or "score". model_kwargs: A `dict`. A dict for the other inputs of the model function. guidance_type: A `str`. The type of the guidance for sampling. "uncond" or "classifier" or "classifier-free". condition: A pytorch tensor. The condition for the guided sampling. Only used for "classifier" or "classifier-free" guidance type. unconditional_condition: A pytorch tensor. The condition for the unconditional sampling. Only used for "classifier-free" guidance type. guidance_scale: A `float`. The scale for the guided sampling. classifier_fn: A classifier function. Only used for the classifier guidance. classifier_kwargs: A `dict`. A dict for the other inputs of the classifier function. Returns: A noise prediction model that accepts the noised data and the continuous time as the inputs. """ def get_model_input_time(t_continuous): """ Convert the continuous-time `t_continuous` (in [epsilon, T]) to the model input time. For discrete-time DPMs, we convert `t_continuous` in [1 / N, 1] to `t_input` in [0, 1000 * (N - 1) / N]. For continuous-time DPMs, we just use `t_continuous`. """ if noise_schedule.schedule == 'discrete': return (t_continuous - 1. / noise_schedule.total_N) * noise_schedule.total_N else: return t_continuous def noise_pred_fn(x, t_continuous, cond=None): t_input = get_model_input_time(t_continuous) if cond is None: output = model(x, t_input, **model_kwargs) else: output = model(x, t_input, cond, **model_kwargs) if model_type == "noise": return output elif model_type == "x_start": alpha_t, sigma_t = noise_schedule.marginal_alpha(t_continuous), noise_schedule.marginal_std(t_continuous) return (x - expand_dims(alpha_t, x.dim()) * output) / expand_dims(sigma_t, x.dim()) elif model_type == "v": alpha_t, sigma_t = noise_schedule.marginal_alpha(t_continuous), noise_schedule.marginal_std(t_continuous) return expand_dims(alpha_t, x.dim()) * output + expand_dims(sigma_t, x.dim()) * x elif model_type == "score": sigma_t = noise_schedule.marginal_std(t_continuous) return -expand_dims(sigma_t, x.dim()) * output def cond_grad_fn(x, t_input): """ Compute the gradient of the classifier, i.e. nabla_{x} log p_t(cond | x_t). """ with torch.enable_grad(): x_in = x.detach().requires_grad_(True) log_prob = classifier_fn(x_in, t_input, condition, **classifier_kwargs) return torch.autograd.grad(log_prob.sum(), x_in)[0] def model_fn(x, t_continuous): """ The noise predicition model function that is used for DPM-Solver. """ if guidance_type == "uncond": return noise_pred_fn(x, t_continuous) elif guidance_type == "classifier": assert classifier_fn is not None t_input = get_model_input_time(t_continuous) cond_grad = cond_grad_fn(x, t_input) sigma_t = noise_schedule.marginal_std(t_continuous) noise = noise_pred_fn(x, t_continuous) return noise - guidance_scale * expand_dims(sigma_t, x.dim()) * cond_grad elif guidance_type == "classifier-free": if guidance_scale == 1. or unconditional_condition is None: return noise_pred_fn(x, t_continuous, cond=condition) else: x_in = torch.cat([x] * 2) t_in = torch.cat([t_continuous] * 2) c_in = torch.cat([unconditional_condition, condition]) noise_uncond, noise = noise_pred_fn(x_in, t_in, cond=c_in).chunk(2) return noise_uncond + guidance_scale * (noise - noise_uncond) assert model_type in ["noise", "x_start", "v", "score"] assert guidance_type in ["uncond", "classifier", "classifier-free"] return model_fn class DPM_Solver: def __init__( self, model_fn, noise_schedule, algorithm_type="dpmsolver++", correcting_x0_fn=None, correcting_xt_fn=None, thresholding_max_val=1., dynamic_thresholding_ratio=0.995, ): """Construct a DPM-Solver. We support both DPM-Solver (`algorithm_type="dpmsolver"`) and DPM-Solver++ (`algorithm_type="dpmsolver++"`). We also support the "dynamic thresholding" method in Imagen[1]. For pixel-space diffusion models, you can set both `algorithm_type="dpmsolver++"` and `correcting_x0_fn="dynamic_thresholding"` to use the dynamic thresholding. The "dynamic thresholding" can greatly improve the sample quality for pixel-space DPMs with large guidance scales. Note that the thresholding method is **unsuitable** for latent-space DPMs (such as stable-diffusion). To support advanced algorithms in image-to-image applications, we also support corrector functions for both x0 and xt. Args: model_fn: A noise prediction model function which accepts the continuous-time input (t in [epsilon, T]): `` def model_fn(x, t_continuous): return noise `` The shape of `x` is `(batch_size, **shape)`, and the shape of `t_continuous` is `(batch_size,)`. noise_schedule: A noise schedule object, such as NoiseScheduleVP. algorithm_type: A `str`. Either "dpmsolver" or "dpmsolver++". correcting_x0_fn: A `str` or a function with the following format: ``` def correcting_x0_fn(x0, t): x0_new = ... return x0_new ``` This function is to correct the outputs of the data prediction model at each sampling step. e.g., ``` x0_pred = data_pred_model(xt, t) if correcting_x0_fn is not None: x0_pred = correcting_x0_fn(x0_pred, t) xt_1 = update(x0_pred, xt, t) ``` If `correcting_x0_fn="dynamic_thresholding"`, we use the dynamic thresholding proposed in Imagen[1]. correcting_xt_fn: A function with the following format: ``` def correcting_xt_fn(xt, t, step): x_new = ... return x_new ``` This function is to correct the intermediate samples xt at each sampling step. e.g., ``` xt = ... xt = correcting_xt_fn(xt, t, step) ``` thresholding_max_val: A `float`. The max value for thresholding. Valid only when use `dpmsolver++` and `correcting_x0_fn="dynamic_thresholding"`. dynamic_thresholding_ratio: A `float`. The ratio for dynamic thresholding (see Imagen[1] for details). Valid only when use `dpmsolver++` and `correcting_x0_fn="dynamic_thresholding"`. [1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022b. """ self.model = lambda x, t: model_fn(x, t.expand((x.shape[0]))) self.noise_schedule = noise_schedule assert algorithm_type in ["dpmsolver", "dpmsolver++"] self.algorithm_type = algorithm_type if correcting_x0_fn == "dynamic_thresholding": self.correcting_x0_fn = self.dynamic_thresholding_fn else: self.correcting_x0_fn = correcting_x0_fn self.correcting_xt_fn = correcting_xt_fn self.dynamic_thresholding_ratio = dynamic_thresholding_ratio self.thresholding_max_val = thresholding_max_val def dynamic_thresholding_fn(self, x0, t): """ The dynamic thresholding method. """ dims = x0.dim() p = self.dynamic_thresholding_ratio s = torch.quantile(torch.abs(x0).reshape((x0.shape[0], -1)), p, dim=1) s = expand_dims(torch.maximum(s, self.thresholding_max_val * torch.ones_like(s).to(s.device)), dims) x0 = torch.clamp(x0, -s, s) / s return x0 def noise_prediction_fn(self, x, t): """ Return the noise prediction model. """ return self.model(x, t) def data_prediction_fn(self, x, t): """ Return the data prediction model (with corrector). """ noise = self.noise_prediction_fn(x, t) alpha_t, sigma_t = self.noise_schedule.marginal_alpha(t), self.noise_schedule.marginal_std(t) x0 = (x - sigma_t * noise) / alpha_t if self.correcting_x0_fn is not None: x0 = self.correcting_x0_fn(x0, t) return x0 def model_fn(self, x, t): """ Convert the model to the noise prediction model or the data prediction model. """ if self.algorithm_type == "dpmsolver++": return self.data_prediction_fn(x, t) else: return self.noise_prediction_fn(x, t) def get_time_steps(self, skip_type, t_T, t_0, N, device): """Compute the intermediate time steps for sampling. Args: skip_type: A `str`. The type for the spacing of the time steps. We support three types: - 'logSNR': uniform logSNR for the time steps. - 'time_uniform': uniform time for the time steps. (**Recommended for high-resolutional data**.) - 'time_quadratic': quadratic time for the time steps. (Used in DDIM for low-resolutional data.) t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). N: A `int`. The total number of the spacing of the time steps. device: A torch device. Returns: A pytorch tensor of the time steps, with the shape (N + 1,). """ if skip_type == 'logSNR': lambda_T = self.noise_schedule.marginal_lambda(torch.tensor(t_T).to(device)) lambda_0 = self.noise_schedule.marginal_lambda(torch.tensor(t_0).to(device)) logSNR_steps = torch.linspace(lambda_T.cpu().item(), lambda_0.cpu().item(), N + 1).to(device) return self.noise_schedule.inverse_lambda(logSNR_steps) elif skip_type == 'time_uniform': return torch.linspace(t_T, t_0, N + 1).to(device) elif skip_type == 'time_quadratic': t_order = 2 t = torch.linspace(t_T**(1. / t_order), t_0**(1. / t_order), N + 1).pow(t_order).to(device) return t else: raise ValueError("Unsupported skip_type {}, need to be 'logSNR' or 'time_uniform' or 'time_quadratic'".format(skip_type)) def get_orders_and_timesteps_for_singlestep_solver(self, steps, order, skip_type, t_T, t_0, device): """ Get the order of each step for sampling by the singlestep DPM-Solver. We combine both DPM-Solver-1,2,3 to use all the function evaluations, which is named as "DPM-Solver-fast". Given a fixed number of function evaluations by `steps`, the sampling procedure by DPM-Solver-fast is: - If order == 1: We take `steps` of DPM-Solver-1 (i.e. DDIM). - If order == 2: - Denote K = (steps // 2). We take K or (K + 1) intermediate time steps for sampling. - If steps % 2 == 0, we use K steps of DPM-Solver-2. - If steps % 2 == 1, we use K steps of DPM-Solver-2 and 1 step of DPM-Solver-1. - If order == 3: - Denote K = (steps // 3 + 1). We take K intermediate time steps for sampling. - If steps % 3 == 0, we use (K - 2) steps of DPM-Solver-3, and 1 step of DPM-Solver-2 and 1 step of DPM-Solver-1. - If steps % 3 == 1, we use (K - 1) steps of DPM-Solver-3 and 1 step of DPM-Solver-1. - If steps % 3 == 2, we use (K - 1) steps of DPM-Solver-3 and 1 step of DPM-Solver-2. ============================================ Args: order: A `int`. The max order for the solver (2 or 3). steps: A `int`. The total number of function evaluations (NFE). skip_type: A `str`. The type for the spacing of the time steps. We support three types: - 'logSNR': uniform logSNR for the time steps. - 'time_uniform': uniform time for the time steps. (**Recommended for high-resolutional data**.) - 'time_quadratic': quadratic time for the time steps. (Used in DDIM for low-resolutional data.) t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). device: A torch device. Returns: orders: A list of the solver order of each step. """ if order == 3: K = steps // 3 + 1 if steps % 3 == 0: orders = [3,] * (K - 2) + [2, 1] elif steps % 3 == 1: orders = [3,] * (K - 1) + [1] else: orders = [3,] * (K - 1) + [2] elif order == 2: if steps % 2 == 0: K = steps // 2 orders = [2,] * K else: K = steps // 2 + 1 orders = [2,] * (K - 1) + [1] elif order == 1: K = 1 orders = [1,] * steps else: raise ValueError("'order' must be '1' or '2' or '3'.") if skip_type == 'logSNR': # To reproduce the results in DPM-Solver paper timesteps_outer = self.get_time_steps(skip_type, t_T, t_0, K, device) else: timesteps_outer = self.get_time_steps(skip_type, t_T, t_0, steps, device)[torch.cumsum(torch.tensor([0,] + orders), 0).to(device)] return timesteps_outer, orders def denoise_to_zero_fn(self, x, s): """ Denoise at the final step, which is equivalent to solve the ODE from lambda_s to infty by first-order discretization. """ return self.data_prediction_fn(x, s) def dpm_solver_first_update(self, x, s, t, model_s=None, return_intermediate=False): """ DPM-Solver-1 (equivalent to DDIM) from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s`. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ ns = self.noise_schedule lambda_s, lambda_t = ns.marginal_lambda(s), ns.marginal_lambda(t) h = lambda_t - lambda_s log_alpha_s, log_alpha_t = ns.marginal_log_mean_coeff(s), ns.marginal_log_mean_coeff(t) sigma_s, sigma_t = ns.marginal_std(s), ns.marginal_std(t) alpha_t = torch.exp(log_alpha_t) if self.algorithm_type == "dpmsolver++": phi_1 = torch.expm1(-h) if model_s is None: model_s = self.model_fn(x, s) x_t = ( sigma_t / sigma_s * x - alpha_t * phi_1 * model_s ) if return_intermediate: return x_t, {'model_s': model_s} else: return x_t else: phi_1 = torch.expm1(h) if model_s is None: model_s = self.model_fn(x, s) x_t = ( torch.exp(log_alpha_t - log_alpha_s) * x - (sigma_t * phi_1) * model_s ) if return_intermediate: return x_t, {'model_s': model_s} else: return x_t def singlestep_dpm_solver_second_update(self, x, s, t, r1=0.5, model_s=None, return_intermediate=False, solver_type='dpmsolver'): """ Singlestep solver DPM-Solver-2 from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). r1: A `float`. The hyperparameter of the second-order solver. model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s` and `s1` (the intermediate time). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ if solver_type not in ['dpmsolver', 'taylor']: raise ValueError("'solver_type' must be either 'dpmsolver' or 'taylor', got {}".format(solver_type)) if r1 is None: r1 = 0.5 ns = self.noise_schedule lambda_s, lambda_t = ns.marginal_lambda(s), ns.marginal_lambda(t) h = lambda_t - lambda_s lambda_s1 = lambda_s + r1 * h s1 = ns.inverse_lambda(lambda_s1) log_alpha_s, log_alpha_s1, log_alpha_t = ns.marginal_log_mean_coeff(s), ns.marginal_log_mean_coeff(s1), ns.marginal_log_mean_coeff(t) sigma_s, sigma_s1, sigma_t = ns.marginal_std(s), ns.marginal_std(s1), ns.marginal_std(t) alpha_s1, alpha_t = torch.exp(log_alpha_s1), torch.exp(log_alpha_t) if self.algorithm_type == "dpmsolver++": phi_11 = torch.expm1(-r1 * h) phi_1 = torch.expm1(-h) if model_s is None: model_s = self.model_fn(x, s) x_s1 = ( (sigma_s1 / sigma_s) * x - (alpha_s1 * phi_11) * model_s ) model_s1 = self.model_fn(x_s1, s1) if solver_type == 'dpmsolver': x_t = ( (sigma_t / sigma_s) * x - (alpha_t * phi_1) * model_s - (0.5 / r1) * (alpha_t * phi_1) * (model_s1 - model_s) ) elif solver_type == 'taylor': x_t = ( (sigma_t / sigma_s) * x - (alpha_t * phi_1) * model_s + (1. / r1) * (alpha_t * (phi_1 / h + 1.)) * (model_s1 - model_s) ) else: phi_11 = torch.expm1(r1 * h) phi_1 = torch.expm1(h) if model_s is None: model_s = self.model_fn(x, s) x_s1 = ( torch.exp(log_alpha_s1 - log_alpha_s) * x - (sigma_s1 * phi_11) * model_s ) model_s1 = self.model_fn(x_s1, s1) if solver_type == 'dpmsolver': x_t = ( torch.exp(log_alpha_t - log_alpha_s) * x - (sigma_t * phi_1) * model_s - (0.5 / r1) * (sigma_t * phi_1) * (model_s1 - model_s) ) elif solver_type == 'taylor': x_t = ( torch.exp(log_alpha_t - log_alpha_s) * x - (sigma_t * phi_1) * model_s - (1. / r1) * (sigma_t * (phi_1 / h - 1.)) * (model_s1 - model_s) ) if return_intermediate: return x_t, {'model_s': model_s, 'model_s1': model_s1} else: return x_t def singlestep_dpm_solver_third_update(self, x, s, t, r1=1./3., r2=2./3., model_s=None, model_s1=None, return_intermediate=False, solver_type='dpmsolver'): """ Singlestep solver DPM-Solver-3 from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). r1: A `float`. The hyperparameter of the third-order solver. r2: A `float`. The hyperparameter of the third-order solver. model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. model_s1: A pytorch tensor. The model function evaluated at time `s1` (the intermediate time given by `r1`). If `model_s1` is None, we evaluate the model at `s1`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s`, `s1` and `s2` (the intermediate times). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ if solver_type not in ['dpmsolver', 'taylor']: raise ValueError("'solver_type' must be either 'dpmsolver' or 'taylor', got {}".format(solver_type)) if r1 is None: r1 = 1. / 3. if r2 is None: r2 = 2. / 3. ns = self.noise_schedule lambda_s, lambda_t = ns.marginal_lambda(s), ns.marginal_lambda(t) h = lambda_t - lambda_s lambda_s1 = lambda_s + r1 * h lambda_s2 = lambda_s + r2 * h s1 = ns.inverse_lambda(lambda_s1) s2 = ns.inverse_lambda(lambda_s2) log_alpha_s, log_alpha_s1, log_alpha_s2, log_alpha_t = ns.marginal_log_mean_coeff(s), ns.marginal_log_mean_coeff(s1), ns.marginal_log_mean_coeff(s2), ns.marginal_log_mean_coeff(t) sigma_s, sigma_s1, sigma_s2, sigma_t = ns.marginal_std(s), ns.marginal_std(s1), ns.marginal_std(s2), ns.marginal_std(t) alpha_s1, alpha_s2, alpha_t = torch.exp(log_alpha_s1), torch.exp(log_alpha_s2), torch.exp(log_alpha_t) if self.algorithm_type == "dpmsolver++": phi_11 = torch.expm1(-r1 * h) phi_12 = torch.expm1(-r2 * h) phi_1 = torch.expm1(-h) phi_22 = torch.expm1(-r2 * h) / (r2 * h) + 1. phi_2 = phi_1 / h + 1. phi_3 = phi_2 / h - 0.5 if model_s is None: model_s = self.model_fn(x, s) if model_s1 is None: x_s1 = ( (sigma_s1 / sigma_s) * x - (alpha_s1 * phi_11) * model_s ) model_s1 = self.model_fn(x_s1, s1) x_s2 = ( (sigma_s2 / sigma_s) * x - (alpha_s2 * phi_12) * model_s + r2 / r1 * (alpha_s2 * phi_22) * (model_s1 - model_s) ) model_s2 = self.model_fn(x_s2, s2) if solver_type == 'dpmsolver': x_t = ( (sigma_t / sigma_s) * x - (alpha_t * phi_1) * model_s + (1. / r2) * (alpha_t * phi_2) * (model_s2 - model_s) ) elif solver_type == 'taylor': D1_0 = (1. / r1) * (model_s1 - model_s) D1_1 = (1. / r2) * (model_s2 - model_s) D1 = (r2 * D1_0 - r1 * D1_1) / (r2 - r1) D2 = 2. * (D1_1 - D1_0) / (r2 - r1) x_t = ( (sigma_t / sigma_s) * x - (alpha_t * phi_1) * model_s + (alpha_t * phi_2) * D1 - (alpha_t * phi_3) * D2 ) else: phi_11 = torch.expm1(r1 * h) phi_12 = torch.expm1(r2 * h) phi_1 = torch.expm1(h) phi_22 = torch.expm1(r2 * h) / (r2 * h) - 1. phi_2 = phi_1 / h - 1. phi_3 = phi_2 / h - 0.5 if model_s is None: model_s = self.model_fn(x, s) if model_s1 is None: x_s1 = ( (torch.exp(log_alpha_s1 - log_alpha_s)) * x - (sigma_s1 * phi_11) * model_s ) model_s1 = self.model_fn(x_s1, s1) x_s2 = ( (torch.exp(log_alpha_s2 - log_alpha_s)) * x - (sigma_s2 * phi_12) * model_s - r2 / r1 * (sigma_s2 * phi_22) * (model_s1 - model_s) ) model_s2 = self.model_fn(x_s2, s2) if solver_type == 'dpmsolver': x_t = ( (torch.exp(log_alpha_t - log_alpha_s)) * x - (sigma_t * phi_1) * model_s - (1. / r2) * (sigma_t * phi_2) * (model_s2 - model_s) ) elif solver_type == 'taylor': D1_0 = (1. / r1) * (model_s1 - model_s) D1_1 = (1. / r2) * (model_s2 - model_s) D1 = (r2 * D1_0 - r1 * D1_1) / (r2 - r1) D2 = 2. * (D1_1 - D1_0) / (r2 - r1) x_t = ( (torch.exp(log_alpha_t - log_alpha_s)) * x - (sigma_t * phi_1) * model_s - (sigma_t * phi_2) * D1 - (sigma_t * phi_3) * D2 ) if return_intermediate: return x_t, {'model_s': model_s, 'model_s1': model_s1, 'model_s2': model_s2} else: return x_t def multistep_dpm_solver_second_update(self, x, model_prev_list, t_prev_list, t, solver_type="dpmsolver"): """ Multistep solver DPM-Solver-2 from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ if solver_type not in ['dpmsolver', 'taylor']: raise ValueError("'solver_type' must be either 'dpmsolver' or 'taylor', got {}".format(solver_type)) ns = self.noise_schedule model_prev_1, model_prev_0 = model_prev_list[-2], model_prev_list[-1] t_prev_1, t_prev_0 = t_prev_list[-2], t_prev_list[-1] lambda_prev_1, lambda_prev_0, lambda_t = ns.marginal_lambda(t_prev_1), ns.marginal_lambda(t_prev_0), ns.marginal_lambda(t) log_alpha_prev_0, log_alpha_t = ns.marginal_log_mean_coeff(t_prev_0), ns.marginal_log_mean_coeff(t) sigma_prev_0, sigma_t = ns.marginal_std(t_prev_0), ns.marginal_std(t) alpha_t = torch.exp(log_alpha_t) h_0 = lambda_prev_0 - lambda_prev_1 h = lambda_t - lambda_prev_0 r0 = h_0 / h D1_0 = (1. / r0) * (model_prev_0 - model_prev_1) if self.algorithm_type == "dpmsolver++": phi_1 = torch.expm1(-h) if solver_type == 'dpmsolver': x_t = ( (sigma_t / sigma_prev_0) * x - (alpha_t * phi_1) * model_prev_0 - 0.5 * (alpha_t * phi_1) * D1_0 ) elif solver_type == 'taylor': x_t = ( (sigma_t / sigma_prev_0) * x - (alpha_t * phi_1) * model_prev_0 + (alpha_t * (phi_1 / h + 1.)) * D1_0 ) else: phi_1 = torch.expm1(h) if solver_type == 'dpmsolver': x_t = ( (torch.exp(log_alpha_t - log_alpha_prev_0)) * x - (sigma_t * phi_1) * model_prev_0 - 0.5 * (sigma_t * phi_1) * D1_0 ) elif solver_type == 'taylor': x_t = ( (torch.exp(log_alpha_t - log_alpha_prev_0)) * x - (sigma_t * phi_1) * model_prev_0 - (sigma_t * (phi_1 / h - 1.)) * D1_0 ) return x_t def multistep_dpm_solver_third_update(self, x, model_prev_list, t_prev_list, t, solver_type='dpmsolver'): """ Multistep solver DPM-Solver-3 from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ ns = self.noise_schedule model_prev_2, model_prev_1, model_prev_0 = model_prev_list t_prev_2, t_prev_1, t_prev_0 = t_prev_list lambda_prev_2, lambda_prev_1, lambda_prev_0, lambda_t = ns.marginal_lambda(t_prev_2), ns.marginal_lambda(t_prev_1), ns.marginal_lambda(t_prev_0), ns.marginal_lambda(t) log_alpha_prev_0, log_alpha_t = ns.marginal_log_mean_coeff(t_prev_0), ns.marginal_log_mean_coeff(t) sigma_prev_0, sigma_t = ns.marginal_std(t_prev_0), ns.marginal_std(t) alpha_t = torch.exp(log_alpha_t) h_1 = lambda_prev_1 - lambda_prev_2 h_0 = lambda_prev_0 - lambda_prev_1 h = lambda_t - lambda_prev_0 r0, r1 = h_0 / h, h_1 / h D1_0 = (1. / r0) * (model_prev_0 - model_prev_1) D1_1 = (1. / r1) * (model_prev_1 - model_prev_2) D1 = D1_0 + (r0 / (r0 + r1)) * (D1_0 - D1_1) D2 = (1. / (r0 + r1)) * (D1_0 - D1_1) if self.algorithm_type == "dpmsolver++": phi_1 = torch.expm1(-h) phi_2 = phi_1 / h + 1. phi_3 = phi_2 / h - 0.5 x_t = ( (sigma_t / sigma_prev_0) * x - (alpha_t * phi_1) * model_prev_0 + (alpha_t * phi_2) * D1 - (alpha_t * phi_3) * D2 ) else: phi_1 = torch.expm1(h) phi_2 = phi_1 / h - 1. phi_3 = phi_2 / h - 0.5 x_t = ( (torch.exp(log_alpha_t - log_alpha_prev_0)) * x - (sigma_t * phi_1) * model_prev_0 - (sigma_t * phi_2) * D1 - (sigma_t * phi_3) * D2 ) return x_t def singlestep_dpm_solver_update(self, x, s, t, order, return_intermediate=False, solver_type='dpmsolver', r1=None, r2=None): """ Singlestep DPM-Solver with the order `order` from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). order: A `int`. The order of DPM-Solver. We only support order == 1 or 2 or 3. return_intermediate: A `bool`. If true, also return the model value at time `s`, `s1` and `s2` (the intermediate times). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. r1: A `float`. The hyperparameter of the second-order or third-order solver. r2: A `float`. The hyperparameter of the third-order solver. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ if order == 1: return self.dpm_solver_first_update(x, s, t, return_intermediate=return_intermediate) elif order == 2: return self.singlestep_dpm_solver_second_update(x, s, t, return_intermediate=return_intermediate, solver_type=solver_type, r1=r1) elif order == 3: return self.singlestep_dpm_solver_third_update(x, s, t, return_intermediate=return_intermediate, solver_type=solver_type, r1=r1, r2=r2) else: raise ValueError("Solver order must be 1 or 2 or 3, got {}".format(order)) def multistep_dpm_solver_update(self, x, model_prev_list, t_prev_list, t, order, solver_type='dpmsolver'): """ Multistep DPM-Solver with the order `order` from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). order: A `int`. The order of DPM-Solver. We only support order == 1 or 2 or 3. solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. """ if order == 1: return self.dpm_solver_first_update(x, t_prev_list[-1], t, model_s=model_prev_list[-1]) elif order == 2: return self.multistep_dpm_solver_second_update(x, model_prev_list, t_prev_list, t, solver_type=solver_type) elif order == 3: return self.multistep_dpm_solver_third_update(x, model_prev_list, t_prev_list, t, solver_type=solver_type) else: raise ValueError("Solver order must be 1 or 2 or 3, got {}".format(order)) def dpm_solver_adaptive(self, x, order, t_T, t_0, h_init=0.05, atol=0.0078, rtol=0.05, theta=0.9, t_err=1e-5, solver_type='dpmsolver'): """ The adaptive step size solver based on singlestep DPM-Solver. Args: x: A pytorch tensor. The initial value at time `t_T`. order: A `int`. The (higher) order of the solver. We only support order == 2 or 3. t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). h_init: A `float`. The initial step size (for logSNR). atol: A `float`. The absolute tolerance of the solver. For image data, the default setting is 0.0078, followed [1]. rtol: A `float`. The relative tolerance of the solver. The default setting is 0.05. theta: A `float`. The safety hyperparameter for adapting the step size. The default setting is 0.9, followed [1]. t_err: A `float`. The tolerance for the time. We solve the diffusion ODE until the absolute error between the current time and `t_0` is less than `t_err`. The default setting is 1e-5. solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_0: A pytorch tensor. The approximated solution at time `t_0`. [1] A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas, "Gotta go fast when generating data with score-based models," arXiv preprint arXiv:2105.14080, 2021. """ ns = self.noise_schedule s = t_T * torch.ones((1,)).to(x) lambda_s = ns.marginal_lambda(s) lambda_0 = ns.marginal_lambda(t_0 * torch.ones_like(s).to(x)) h = h_init * torch.ones_like(s).to(x) x_prev = x nfe = 0 if order == 2: r1 = 0.5 def lower_update(x, s, t): return self.dpm_solver_first_update(x, s, t, return_intermediate=True) def higher_update(x, s, t, **kwargs): return self.singlestep_dpm_solver_second_update(x, s, t, r1=r1, solver_type=solver_type, **kwargs) elif order == 3: r1, r2 = 1. / 3., 2. / 3. def lower_update(x, s, t): return self.singlestep_dpm_solver_second_update(x, s, t, r1=r1, return_intermediate=True, solver_type=solver_type) def higher_update(x, s, t, **kwargs): return self.singlestep_dpm_solver_third_update(x, s, t, r1=r1, r2=r2, solver_type=solver_type, **kwargs) else: raise ValueError("For adaptive step size solver, order must be 2 or 3, got {}".format(order)) while torch.abs((s - t_0)).mean() > t_err: t = ns.inverse_lambda(lambda_s + h) x_lower, lower_noise_kwargs = lower_update(x, s, t) x_higher = higher_update(x, s, t, **lower_noise_kwargs) delta = torch.max(torch.ones_like(x).to(x) * atol, rtol * torch.max(torch.abs(x_lower), torch.abs(x_prev))) def norm_fn(v): return torch.sqrt(torch.square(v.reshape((v.shape[0], -1))).mean(dim=-1, keepdim=True)) E = norm_fn((x_higher - x_lower) / delta).max() if torch.all(E <= 1.): x = x_higher s = t x_prev = x_lower lambda_s = ns.marginal_lambda(s) h = torch.min(theta * h * torch.float_power(E, -1. / order).float(), lambda_0 - lambda_s) nfe += order print('adaptive solver nfe', nfe) return x def add_noise(self, x, t, noise=None): """ Compute the noised input xt = alpha_t * x + sigma_t * noise. Args: x: A `torch.Tensor` with shape `(batch_size, *shape)`. t: A `torch.Tensor` with shape `(t_size,)`. Returns: xt with shape `(t_size, batch_size, *shape)`. """ alpha_t, sigma_t = self.noise_schedule.marginal_alpha(t), self.noise_schedule.marginal_std(t) if noise is None: noise = torch.randn((t.shape[0], *x.shape), device=x.device) x = x.reshape((-1, *x.shape)) xt = expand_dims(alpha_t, x.dim()) * x + expand_dims(sigma_t, x.dim()) * noise if t.shape[0] == 1: return xt.squeeze(0) else: return xt def inverse(self, x, steps=20, t_start=None, t_end=None, order=2, skip_type='time_uniform', method='multistep', lower_order_final=True, denoise_to_zero=False, solver_type='dpmsolver', atol=0.0078, rtol=0.05, return_intermediate=False, ): """ Inverse the sample `x` from time `t_start` to `t_end` by DPM-Solver. For discrete-time DPMs, we use `t_start=1/N`, where `N` is the total time steps during training. """ t_0 = 1. / self.noise_schedule.total_N if t_start is None else t_start t_T = self.noise_schedule.T if t_end is None else t_end assert t_0 > 0 and t_T > 0, "Time range needs to be greater than 0. For discrete-time DPMs, it needs to be in [1 / N, 1], where N is the length of betas array" return self.sample(x, steps=steps, t_start=t_0, t_end=t_T, order=order, skip_type=skip_type, method=method, lower_order_final=lower_order_final, denoise_to_zero=denoise_to_zero, solver_type=solver_type, atol=atol, rtol=rtol, return_intermediate=return_intermediate) def sample(self, x, steps=20, t_start=None, t_end=None, order=2, skip_type='time_uniform', method='multistep', lower_order_final=True, denoise_to_zero=False, solver_type='dpmsolver', atol=0.0078, rtol=0.05, return_intermediate=False, ): """ Compute the sample at time `t_end` by DPM-Solver, given the initial `x` at time `t_start`. ===================================================== We support the following algorithms for both noise prediction model and data prediction model: - 'singlestep': Singlestep DPM-Solver (i.e. "DPM-Solver-fast" in the paper), which combines different orders of singlestep DPM-Solver. We combine all the singlestep solvers with order <= `order` to use up all the function evaluations (steps). The total number of function evaluations (NFE) == `steps`. Given a fixed NFE == `steps`, the sampling procedure is: - If `order` == 1: - Denote K = steps. We use K steps of DPM-Solver-1 (i.e. DDIM). - If `order` == 2: - Denote K = (steps // 2) + (steps % 2). We take K intermediate time steps for sampling. - If steps % 2 == 0, we use K steps of singlestep DPM-Solver-2. - If steps % 2 == 1, we use (K - 1) steps of singlestep DPM-Solver-2 and 1 step of DPM-Solver-1. - If `order` == 3: - Denote K = (steps // 3 + 1). We take K intermediate time steps for sampling. - If steps % 3 == 0, we use (K - 2) steps of singlestep DPM-Solver-3, and 1 step of singlestep DPM-Solver-2 and 1 step of DPM-Solver-1. - If steps % 3 == 1, we use (K - 1) steps of singlestep DPM-Solver-3 and 1 step of DPM-Solver-1. - If steps % 3 == 2, we use (K - 1) steps of singlestep DPM-Solver-3 and 1 step of singlestep DPM-Solver-2. - 'multistep': Multistep DPM-Solver with the order of `order`. The total number of function evaluations (NFE) == `steps`. We initialize the first `order` values by lower order multistep solvers. Given a fixed NFE == `steps`, the sampling procedure is: Denote K = steps. - If `order` == 1: - We use K steps of DPM-Solver-1 (i.e. DDIM). - If `order` == 2: - We firstly use 1 step of DPM-Solver-1, then use (K - 1) step of multistep DPM-Solver-2. - If `order` == 3: - We firstly use 1 step of DPM-Solver-1, then 1 step of multistep DPM-Solver-2, then (K - 2) step of multistep DPM-Solver-3. - 'singlestep_fixed': Fixed order singlestep DPM-Solver (i.e. DPM-Solver-1 or singlestep DPM-Solver-2 or singlestep DPM-Solver-3). We use singlestep DPM-Solver-`order` for `order`=1 or 2 or 3, with total [`steps` // `order`] * `order` NFE. - 'adaptive': Adaptive step size DPM-Solver (i.e. "DPM-Solver-12" and "DPM-Solver-23" in the paper). We ignore `steps` and use adaptive step size DPM-Solver with a higher order of `order`. You can adjust the absolute tolerance `atol` and the relative tolerance `rtol` to balance the computatation costs (NFE) and the sample quality. - If `order` == 2, we use DPM-Solver-12 which combines DPM-Solver-1 and singlestep DPM-Solver-2. - If `order` == 3, we use DPM-Solver-23 which combines singlestep DPM-Solver-2 and singlestep DPM-Solver-3. ===================================================== Some advices for choosing the algorithm: - For **unconditional sampling** or **guided sampling with small guidance scale** by DPMs: Use singlestep DPM-Solver or DPM-Solver++ ("DPM-Solver-fast" in the paper) with `order = 3`. e.g., DPM-Solver: >>> dpm_solver = DPM_Solver(model_fn, noise_schedule, algorithm_type="dpmsolver") >>> x_sample = dpm_solver.sample(x, steps=steps, t_start=t_start, t_end=t_end, order=3, skip_type='time_uniform', method='singlestep') e.g., DPM-Solver++: >>> dpm_solver = DPM_Solver(model_fn, noise_schedule, algorithm_type="dpmsolver++") >>> x_sample = dpm_solver.sample(x, steps=steps, t_start=t_start, t_end=t_end, order=3, skip_type='time_uniform', method='singlestep') - For **guided sampling with large guidance scale** by DPMs: Use multistep DPM-Solver with `algorithm_type="dpmsolver++"` and `order = 2`. e.g. >>> dpm_solver = DPM_Solver(model_fn, noise_schedule, algorithm_type="dpmsolver++") >>> x_sample = dpm_solver.sample(x, steps=steps, t_start=t_start, t_end=t_end, order=2, skip_type='time_uniform', method='multistep') We support three types of `skip_type`: - 'logSNR': uniform logSNR for the time steps. **Recommended for low-resolutional images** - 'time_uniform': uniform time for the time steps. **Recommended for high-resolutional images**. - 'time_quadratic': quadratic time for the time steps. ===================================================== Args: x: A pytorch tensor. The initial value at time `t_start` e.g. if `t_start` == T, then `x` is a sample from the standard normal distribution. steps: A `int`. The total number of function evaluations (NFE). t_start: A `float`. The starting time of the sampling. If `T` is None, we use self.noise_schedule.T (default is 1.0). t_end: A `float`. The ending time of the sampling. If `t_end` is None, we use 1. / self.noise_schedule.total_N. e.g. if total_N == 1000, we have `t_end` == 1e-3. For discrete-time DPMs: - We recommend `t_end` == 1. / self.noise_schedule.total_N. For continuous-time DPMs: - We recommend `t_end` == 1e-3 when `steps` <= 15; and `t_end` == 1e-4 when `steps` > 15. order: A `int`. The order of DPM-Solver. skip_type: A `str`. The type for the spacing of the time steps. 'time_uniform' or 'logSNR' or 'time_quadratic'. method: A `str`. The method for sampling. 'singlestep' or 'multistep' or 'singlestep_fixed' or 'adaptive'. denoise_to_zero: A `bool`. Whether to denoise to time 0 at the final step. Default is `False`. If `denoise_to_zero` is `True`, the total NFE is (`steps` + 1). This trick is firstly proposed by DDPM (https://arxiv.org/abs/2006.11239) and score_sde (https://arxiv.org/abs/2011.13456). Such trick can improve the FID for diffusion models sampling by diffusion SDEs for low-resolutional images (such as CIFAR-10). However, we observed that such trick does not matter for high-resolutional images. As it needs an additional NFE, we do not recommend it for high-resolutional images. lower_order_final: A `bool`. Whether to use lower order solvers at the final steps. Only valid for `method=multistep` and `steps < 15`. We empirically find that this trick is a key to stabilizing the sampling by DPM-Solver with very few steps (especially for steps <= 10). So we recommend to set it to be `True`. solver_type: A `str`. The taylor expansion type for the solver. `dpmsolver` or `taylor`. We recommend `dpmsolver`. atol: A `float`. The absolute tolerance of the adaptive step size solver. Valid when `method` == 'adaptive'. rtol: A `float`. The relative tolerance of the adaptive step size solver. Valid when `method` == 'adaptive'. return_intermediate: A `bool`. Whether to save the xt at each step. When set to `True`, method returns a tuple (x0, intermediates); when set to False, method returns only x0. Returns: x_end: A pytorch tensor. The approximated solution at time `t_end`. """ t_0 = 1. / self.noise_schedule.total_N if t_end is None else t_end t_T = self.noise_schedule.T if t_start is None else t_start assert t_0 > 0 and t_T > 0, "Time range needs to be greater than 0. For discrete-time DPMs, it needs to be in [1 / N, 1], where N is the length of betas array" if return_intermediate: assert method in ['multistep', 'singlestep', 'singlestep_fixed'], "Cannot use adaptive solver when saving intermediate values" if self.correcting_xt_fn is not None: assert method in ['multistep', 'singlestep', 'singlestep_fixed'], "Cannot use adaptive solver when correcting_xt_fn is not None" device = x.device intermediates = [] with torch.no_grad(): if method == 'adaptive': x = self.dpm_solver_adaptive(x, order=order, t_T=t_T, t_0=t_0, atol=atol, rtol=rtol, solver_type=solver_type) elif method == 'multistep': assert steps >= order timesteps = self.get_time_steps(skip_type=skip_type, t_T=t_T, t_0=t_0, N=steps, device=device) assert timesteps.shape[0] - 1 == steps # Init the initial values. step = 0 t = timesteps[step] t_prev_list = [t] model_prev_list = [self.model_fn(x, t)] if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) # Init the first `order` values by lower order multistep DPM-Solver. for step in range(1, order): t = timesteps[step] x = self.multistep_dpm_solver_update(x, model_prev_list, t_prev_list, t, step, solver_type=solver_type) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) t_prev_list.append(t) model_prev_list.append(self.model_fn(x, t)) # Compute the remaining values by `order`-th order multistep DPM-Solver. for step in range(order, steps + 1): t = timesteps[step] # We only use lower order for steps < 10 if lower_order_final and steps < 10: step_order = min(order, steps + 1 - step) else: step_order = order x = self.multistep_dpm_solver_update(x, model_prev_list, t_prev_list, t, step_order, solver_type=solver_type) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) for i in range(order - 1): t_prev_list[i] = t_prev_list[i + 1] model_prev_list[i] = model_prev_list[i + 1] t_prev_list[-1] = t # We do not need to evaluate the final model value. if step < steps: model_prev_list[-1] = self.model_fn(x, t) elif method in ['singlestep', 'singlestep_fixed']: if method == 'singlestep': timesteps_outer, orders = self.get_orders_and_timesteps_for_singlestep_solver(steps=steps, order=order, skip_type=skip_type, t_T=t_T, t_0=t_0, device=device) elif method == 'singlestep_fixed': K = steps // order orders = [order,] * K timesteps_outer = self.get_time_steps(skip_type=skip_type, t_T=t_T, t_0=t_0, N=K, device=device) for step, order in enumerate(orders): s, t = timesteps_outer[step], timesteps_outer[step + 1] timesteps_inner = self.get_time_steps(skip_type=skip_type, t_T=s.item(), t_0=t.item(), N=order, device=device) lambda_inner = self.noise_schedule.marginal_lambda(timesteps_inner) h = lambda_inner[-1] - lambda_inner[0] r1 = None if order <= 1 else (lambda_inner[1] - lambda_inner[0]) / h r2 = None if order <= 2 else (lambda_inner[2] - lambda_inner[0]) / h x = self.singlestep_dpm_solver_update(x, s, t, order, solver_type=solver_type, r1=r1, r2=r2) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) else: raise ValueError("Got wrong method {}".format(method)) if denoise_to_zero: t = torch.ones((1,)).to(device) * t_0 x = self.denoise_to_zero_fn(x, t) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step + 1) if return_intermediate: intermediates.append(x) if return_intermediate: return x, intermediates else: return x ############################################################# # other utility functions ############################################################# def interpolate_fn(x, xp, yp): """ A piecewise linear function y = f(x), using xp and yp as keypoints. We implement f(x) in a differentiable way (i.e. applicable for autograd). The function f(x) is well-defined for all x-axis. (For x beyond the bounds of xp, we use the outmost points of xp to define the linear function.) Args: x: PyTorch tensor with shape [N, C], where N is the batch size, C is the number of channels (we use C = 1 for DPM-Solver). xp: PyTorch tensor with shape [C, K], where K is the number of keypoints. yp: PyTorch tensor with shape [C, K]. Returns: The function values f(x), with shape [N, C]. """ N, K = x.shape[0], xp.shape[1] all_x = torch.cat([x.unsqueeze(2), xp.unsqueeze(0).repeat((N, 1, 1))], dim=2) sorted_all_x, x_indices = torch.sort(all_x, dim=2) x_idx = torch.argmin(x_indices, dim=2) cand_start_idx = x_idx - 1 start_idx = torch.where( torch.eq(x_idx, 0), torch.tensor(1, device=x.device), torch.where( torch.eq(x_idx, K), torch.tensor(K - 2, device=x.device), cand_start_idx, ), ) end_idx = torch.where(torch.eq(start_idx, cand_start_idx), start_idx + 2, start_idx + 1) start_x = torch.gather(sorted_all_x, dim=2, index=start_idx.unsqueeze(2)).squeeze(2) end_x = torch.gather(sorted_all_x, dim=2, index=end_idx.unsqueeze(2)).squeeze(2) start_idx2 = torch.where( torch.eq(x_idx, 0), torch.tensor(0, device=x.device), torch.where( torch.eq(x_idx, K), torch.tensor(K - 2, device=x.device), cand_start_idx, ), ) y_positions_expanded = yp.unsqueeze(0).expand(N, -1, -1) start_y = torch.gather(y_positions_expanded, dim=2, index=start_idx2.unsqueeze(2)).squeeze(2) end_y = torch.gather(y_positions_expanded, dim=2, index=(start_idx2 + 1).unsqueeze(2)).squeeze(2) cand = start_y + (x - start_x) * (end_y - start_y) / (end_x - start_x) return cand def expand_dims(v, dims): """ Expand the tensor `v` to the dim `dims`. Args: `v`: a PyTorch tensor with shape [N]. `dim`: a `int`. Returns: a PyTorch tensor with shape [N, 1, 1, ..., 1] and the total dimension is `dims`. """ return v[(...,) + (None,)*(dims - 1)]so-vits-svc-npu/diffusion/how to export onnx.md000066400000000000000000000003111456161405400220310ustar00rootroot00000000000000- Open [onnx_export](onnx_export.py) - project_name = "dddsp" change "project_name" to your project name - model_path = f'{project_name}/model_500000.pt' change "model_path" to your model path - Runso-vits-svc-npu/diffusion/infer_gt_mel.py000066400000000000000000000060231456161405400210540ustar00rootroot00000000000000import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from diffusion.unit2mel import load_model_vocoder class DiffGtMel: def __init__(self, project_path=None, device=None): self.project_path = project_path if device is not None: self.device = device else: self.device = 'cuda' if torch.cuda.is_available() else 'cpu' self.model = None self.vocoder = None self.args = None def flush_model(self, project_path, ddsp_config=None): if (self.model is None) or (project_path != self.project_path): model, vocoder, args = load_model_vocoder(project_path, device=self.device) if self.check_args(ddsp_config, args): self.model = model self.vocoder = vocoder self.args = args def check_args(self, args1, args2): if args1.data.block_size != args2.data.block_size: raise ValueError("DDSP与DIFF模型的block_size不一致") if args1.data.sampling_rate != args2.data.sampling_rate: raise ValueError("DDSP与DIFF模型的sampling_rate不一致") if args1.data.encoder != args2.data.encoder: raise ValueError("DDSP与DIFF模型的encoder不一致") return True def __call__(self, audio, f0, hubert, volume, acc=1, spk_id=1, k_step=0, method='pndm', spk_mix_dict=None, start_frame=0): input_mel = self.vocoder.extract(audio, self.args.data.sampling_rate) out_mel = self.model( hubert, f0, volume, spk_id=spk_id, spk_mix_dict=spk_mix_dict, gt_spec=input_mel, infer=True, infer_speedup=acc, method=method, k_step=k_step, use_tqdm=False) if start_frame > 0: out_mel = out_mel[:, start_frame:, :] f0 = f0[:, start_frame:, :] output = self.vocoder.infer(out_mel, f0) if start_frame > 0: output = F.pad(output, (start_frame * self.vocoder.vocoder_hop_size, 0)) return output def infer(self, audio, f0, hubert, volume, acc=1, spk_id=1, k_step=0, method='pndm', silence_front=0, use_silence=False, spk_mix_dict=None): start_frame = int(silence_front * self.vocoder.vocoder_sample_rate / self.vocoder.vocoder_hop_size) if use_silence: audio = audio[:, start_frame * self.vocoder.vocoder_hop_size:] f0 = f0[:, start_frame:, :] hubert = hubert[:, start_frame:, :] volume = volume[:, start_frame:, :] _start_frame = 0 else: _start_frame = start_frame audio = self.__call__(audio, f0, hubert, volume, acc=acc, spk_id=spk_id, k_step=k_step, method=method, spk_mix_dict=spk_mix_dict, start_frame=_start_frame) if use_silence: if start_frame > 0: audio = F.pad(audio, (start_frame * self.vocoder.vocoder_hop_size, 0)) return audio so-vits-svc-npu/diffusion/logger/000077500000000000000000000000001456161405400173265ustar00rootroot00000000000000so-vits-svc-npu/diffusion/logger/__init__.py000066400000000000000000000000001456161405400214250ustar00rootroot00000000000000so-vits-svc-npu/diffusion/logger/saver.py000066400000000000000000000100301456161405400210120ustar00rootroot00000000000000''' author: wayn391@mastertones ''' import datetime import os import time import matplotlib.pyplot as plt import mindtorch.torch as torch import yaml from mindtorch.torch.utils.tensorboard import SummaryWriter class Saver(object): def __init__( self, args, initial_global_step=-1): self.expdir = args.env.expdir self.sample_rate = args.data.sampling_rate # cold start self.global_step = initial_global_step self.init_time = time.time() self.last_time = time.time() # makedirs os.makedirs(self.expdir, exist_ok=True) # path self.path_log_info = os.path.join(self.expdir, 'log_info.txt') # ckpt os.makedirs(self.expdir, exist_ok=True) # writer self.writer = SummaryWriter(os.path.join(self.expdir, 'logs')) # save config path_config = os.path.join(self.expdir, 'config.yaml') with open(path_config, "w") as out_config: yaml.dump(dict(args), out_config) def log_info(self, msg): '''log method''' if isinstance(msg, dict): msg_list = [] for k, v in msg.items(): tmp_str = '' if isinstance(v, int): tmp_str = '{}: {:,}'.format(k, v) else: tmp_str = '{}: {}'.format(k, v) msg_list.append(tmp_str) msg_str = '\n'.join(msg_list) else: msg_str = msg # dsplay print(msg_str) # save with open(self.path_log_info, 'a') as fp: fp.write(msg_str+'\n') def log_value(self, dict): for k, v in dict.items(): self.writer.add_scalar(k, v, self.global_step) def log_spec(self, name, spec, spec_out, vmin=-14, vmax=3.5): spec_cat = torch.cat([(spec_out - spec).abs() + vmin, spec, spec_out], -1) spec = spec_cat[0] if isinstance(spec, torch.Tensor): spec = spec.cpu().numpy() fig = plt.figure(figsize=(12, 9)) plt.pcolor(spec.T, vmin=vmin, vmax=vmax) plt.tight_layout() self.writer.add_figure(name, fig, self.global_step) def log_audio(self, dict): for k, v in dict.items(): self.writer.add_audio(k, v, global_step=self.global_step, sample_rate=self.sample_rate) def get_interval_time(self, update=True): cur_time = time.time() time_interval = cur_time - self.last_time if update: self.last_time = cur_time return time_interval def get_total_time(self, to_str=True): total_time = time.time() - self.init_time if to_str: total_time = str(datetime.timedelta( seconds=total_time))[:-5] return total_time def save_model( self, model, optimizer, name='model', postfix='', to_json=False): # path if postfix: postfix = '_' + postfix path_pt = os.path.join( self.expdir , name+postfix+'.pt') # check print(' [*] model checkpoint saved: {}'.format(path_pt)) # save if optimizer is not None: torch.save({ 'global_step': self.global_step, 'model': model.state_dict(), 'optimizer': optimizer.state_dict()}, path_pt) else: torch.save({ 'global_step': self.global_step, 'model': model.state_dict()}, path_pt) def delete_model(self, name='model', postfix=''): # path if postfix: postfix = '_' + postfix path_pt = os.path.join( self.expdir , name+postfix+'.pt') # delete if os.path.exists(path_pt): os.remove(path_pt) print(' [*] model checkpoint deleted: {}'.format(path_pt)) def global_step_increment(self): self.global_step += 1 so-vits-svc-npu/diffusion/logger/utils.py000066400000000000000000000072551456161405400210510ustar00rootroot00000000000000import json import os import mindtorch.torch as torch import yaml def traverse_dir( root_dir, extensions, amount=None, str_include=None, str_exclude=None, is_pure=False, is_sort=False, is_ext=True): file_list = [] cnt = 0 for root, _, files in os.walk(root_dir): for file in files: if any([file.endswith(f".{ext}") for ext in extensions]): # path mix_path = os.path.join(root, file) pure_path = mix_path[len(root_dir)+1:] if is_pure else mix_path # amount if (amount is not None) and (cnt == amount): if is_sort: file_list.sort() return file_list # check string if (str_include is not None) and (str_include not in pure_path): continue if (str_exclude is not None) and (str_exclude in pure_path): continue if not is_ext: ext = pure_path.split('.')[-1] pure_path = pure_path[:-(len(ext)+1)] file_list.append(pure_path) cnt += 1 if is_sort: file_list.sort() return file_list class DotDict(dict): def __getattr__(*args): val = dict.get(*args) return DotDict(val) if type(val) is dict else val __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ def get_network_paras_amount(model_dict): info = dict() for model_name, model in model_dict.items(): # all_params = sum(p.numel() for p in model.parameters()) trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad) info[model_name] = trainable_params return info def load_config(path_config): with open(path_config, "r") as config: args = yaml.safe_load(config) args = DotDict(args) # print(args) return args def save_config(path_config,config): config = dict(config) with open(path_config, "w") as f: yaml.dump(config, f) def to_json(path_params, path_json): params = torch.load(path_params, map_location=torch.device('cpu')) raw_state_dict = {} for k, v in params.items(): val = v.flatten().numpy().tolist() raw_state_dict[k] = val with open(path_json, 'w') as outfile: json.dump(raw_state_dict, outfile,indent= "\t") def convert_tensor_to_numpy(tensor, is_squeeze=True): if is_squeeze: tensor = tensor.squeeze() if tensor.requires_grad: tensor = tensor.detach() if tensor.is_cuda: tensor = tensor.cpu() return tensor.numpy() def load_model( expdir, model, optimizer, name='model', postfix='', device='cpu'): if postfix == '': postfix = '_' + postfix path = os.path.join(expdir, name+postfix) path_pt = traverse_dir(expdir, ['pt'], is_ext=False) global_step = 0 if len(path_pt) > 0: steps = [s[len(path):] for s in path_pt] maxstep = max([int(s) if s.isdigit() else 0 for s in steps]) if maxstep >= 0: path_pt = path+str(maxstep)+'.pt' else: path_pt = path+'best.pt' print(' [*] restoring model from', path_pt) ckpt = torch.load(path_pt, map_location=torch.device(device)) global_step = ckpt['global_step'] model.load_state_dict(ckpt['model'], strict=False) if ckpt.get("optimizer") is not None: optimizer.load_state_dict(ckpt['optimizer']) return global_step, model, optimizer so-vits-svc-npu/diffusion/onnx_export.py000066400000000000000000000212771456161405400210150ustar00rootroot00000000000000import os import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F import yaml from diffusion_onnx import GaussianDiffusion class DotDict(dict): def __getattr__(*args): val = dict.get(*args) return DotDict(val) if type(val) is dict else val __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ def load_model_vocoder( model_path, device='cpu'): config_file = os.path.join(os.path.split(model_path)[0], 'config.yaml') with open(config_file, "r") as config: args = yaml.safe_load(config) args = DotDict(args) # load model model = Unit2Mel( args.data.encoder_out_channels, args.model.n_spk, args.model.use_pitch_aug, 128, args.model.n_layers, args.model.n_chans, args.model.n_hidden, args.model.timesteps, args.model.k_step_max) print(' [Loading] ' + model_path) ckpt = torch.load(model_path, map_location=torch.device(device)) model.to(device) model.load_state_dict(ckpt['model']) model.eval() return model, args class Unit2Mel(nn.Module): def __init__( self, input_channel, n_spk, use_pitch_aug=False, out_dims=128, n_layers=20, n_chans=384, n_hidden=256, timesteps=1000, k_step_max=1000): super().__init__() self.unit_embed = nn.Linear(input_channel, n_hidden) self.f0_embed = nn.Linear(1, n_hidden) self.volume_embed = nn.Linear(1, n_hidden) if use_pitch_aug: self.aug_shift_embed = nn.Linear(1, n_hidden, bias=False) else: self.aug_shift_embed = None self.n_spk = n_spk if n_spk is not None and n_spk > 1: self.spk_embed = nn.Embedding(n_spk, n_hidden) self.timesteps = timesteps if timesteps is not None else 1000 self.k_step_max = k_step_max if k_step_max is not None and k_step_max>0 and k_step_max 1: # [N, S] * [S, B, 1, H] g = g.reshape((g.shape[0], g.shape[1], 1, 1, 1)) # [N, S, B, 1, 1] g = g * self.speaker_map # [N, S, B, 1, H] g = torch.sum(g, dim=1) # [N, 1, B, 1, H] g = g.transpose(0, -1).transpose(0, -2).squeeze(0) # [B, H, N] x = x.transpose(1, 2) + g return x else: return x.transpose(1, 2) def init_spkembed(self, units, f0, volume, spk_id = None, spk_mix_dict = None, aug_shift = None, gt_spec=None, infer=True, infer_speedup=10, method='dpm-solver', k_step=300, use_tqdm=True): ''' input: B x n_frames x n_unit return: dict of B x n_frames x feat ''' x = self.unit_embed(units) + self.f0_embed((1+ f0 / 700).log()) + self.volume_embed(volume) if self.n_spk is not None and self.n_spk > 1: if spk_mix_dict is not None: spk_embed_mix = torch.zeros((1,1,self.hidden_size)) for k, v in spk_mix_dict.items(): spk_id_torch = torch.LongTensor(np.array([[k]])).to(units.device) spk_embeddd = self.spk_embed(spk_id_torch) self.speaker_map[k] = spk_embeddd spk_embed_mix = spk_embed_mix + v * spk_embeddd x = x + spk_embed_mix else: x = x + self.spk_embed(spk_id - 1) self.speaker_map = self.speaker_map.unsqueeze(0) self.speaker_map = self.speaker_map.detach() return x.transpose(1, 2) def OnnxExport(self, project_name=None, init_noise=None, export_encoder=True, export_denoise=True, export_pred=True, export_after=True): hubert_hidden_size = 768 n_frames = 100 hubert = torch.randn((1, n_frames, hubert_hidden_size)) mel2ph = torch.arange(end=n_frames).unsqueeze(0).long() f0 = torch.randn((1, n_frames)) volume = torch.randn((1, n_frames)) spk_mix = [] spks = {} if self.n_spk is not None and self.n_spk > 1: for i in range(self.n_spk): spk_mix.append(1.0/float(self.n_spk)) spks.update({i:1.0/float(self.n_spk)}) spk_mix = torch.tensor(spk_mix) spk_mix = spk_mix.repeat(n_frames, 1) self.init_spkembed(hubert, f0.unsqueeze(-1), volume.unsqueeze(-1), spk_mix_dict=spks) self.forward(hubert, mel2ph, f0, volume, spk_mix) if export_encoder: torch.onnx.export( self, (hubert, mel2ph, f0, volume, spk_mix), f"{project_name}_encoder.onnx", input_names=["hubert", "mel2ph", "f0", "volume", "spk_mix"], output_names=["mel_pred"], dynamic_axes={ "hubert": [1], "f0": [1], "volume": [1], "mel2ph": [1], "spk_mix": [0], }, opset_version=16 ) self.decoder.OnnxExport(project_name, init_noise=init_noise, export_denoise=export_denoise, export_pred=export_pred, export_after=export_after) def ExportOnnx(self, project_name=None): hubert_hidden_size = 768 n_frames = 100 hubert = torch.randn((1, n_frames, hubert_hidden_size)) mel2ph = torch.arange(end=n_frames).unsqueeze(0).long() f0 = torch.randn((1, n_frames)) volume = torch.randn((1, n_frames)) spk_mix = [] spks = {} if self.n_spk is not None and self.n_spk > 1: for i in range(self.n_spk): spk_mix.append(1.0/float(self.n_spk)) spks.update({i:1.0/float(self.n_spk)}) spk_mix = torch.tensor(spk_mix) self.orgforward(hubert, f0.unsqueeze(-1), volume.unsqueeze(-1), spk_mix_dict=spks) self.forward(hubert, mel2ph, f0, volume, spk_mix) torch.onnx.export( self, (hubert, mel2ph, f0, volume, spk_mix), f"{project_name}_encoder.onnx", input_names=["hubert", "mel2ph", "f0", "volume", "spk_mix"], output_names=["mel_pred"], dynamic_axes={ "hubert": [1], "f0": [1], "volume": [1], "mel2ph": [1] }, opset_version=16 ) condition = torch.randn(1,self.decoder.n_hidden,n_frames) noise = torch.randn((1, 1, self.decoder.mel_bins, condition.shape[2]), dtype=torch.float32) pndm_speedup = torch.LongTensor([100]) K_steps = torch.LongTensor([1000]) self.decoder = torch.jit.script(self.decoder) self.decoder(condition, noise, pndm_speedup, K_steps) torch.onnx.export( self.decoder, (condition, noise, pndm_speedup, K_steps), f"{project_name}_diffusion.onnx", input_names=["condition", "noise", "pndm_speedup", "K_steps"], output_names=["mel"], dynamic_axes={ "condition": [2], "noise": [3], }, opset_version=16 ) if __name__ == "__main__": project_name = "dddsp" model_path = f'{project_name}/model_500000.pt' model, _ = load_model_vocoder(model_path) # 分开Diffusion导出（需要使用MoeSS/MoeVoiceStudio或者自己编写Pndm/Dpm采样） model.OnnxExport(project_name, export_encoder=True, export_denoise=True, export_pred=True, export_after=True) # 合并Diffusion导出（Encoder和Diffusion分开，直接将Encoder的结果和初始噪声输入Diffusion即可） # model.ExportOnnx(project_name) so-vits-svc-npu/diffusion/solver.py000066400000000000000000000162151456161405400177400ustar00rootroot00000000000000import time import librosa import numpy as np import mindtorch.torch as torch from mindtorch.torch import autocast from mindtorch.torch.cuda.amp import GradScaler from diffusion.logger import utils from diffusion.logger.saver import Saver def test(args, model, vocoder, loader_test, saver): print(' [*] testing...') model.eval() # losses test_loss = 0. # intialization num_batches = len(loader_test) rtf_all = [] # run with torch.no_grad(): for bidx, data in enumerate(loader_test): fn = data['name'][0].split("/")[-1] speaker = data['name'][0].split("/")[-2] print('--------') print('{}/{} - {}'.format(bidx, num_batches, fn)) # unpack data for k in data.keys(): if not k.startswith('name'): data[k] = data[k].to(args.device) print('>>', data['name'][0]) # forward st_time = time.time() mel = model( data['units'], data['f0'], data['volume'], data['spk_id'], gt_spec=None if model.k_step_max == model.timesteps else data['mel'], infer=True, infer_speedup=args.infer.speedup, method=args.infer.method, k_step=model.k_step_max ) signal = vocoder.infer(mel, data['f0']) ed_time = time.time() # RTF run_time = ed_time - st_time song_time = signal.shape[-1] / args.data.sampling_rate rtf = run_time / song_time print('RTF: {} | {} / {}'.format(rtf, run_time, song_time)) rtf_all.append(rtf) # loss for i in range(args.train.batch_size): loss = model( data['units'], data['f0'], data['volume'], data['spk_id'], gt_spec=data['mel'], infer=False, k_step=model.k_step_max) test_loss += loss.item() # log mel saver.log_spec(f"{speaker}_{fn}.wav", data['mel'], mel) # log audi path_audio = data['name_ext'][0] audio, sr = librosa.load(path_audio, sr=args.data.sampling_rate) if len(audio.shape) > 1: audio = librosa.to_mono(audio) audio = torch.from_numpy(audio).unsqueeze(0).to(signal) saver.log_audio({f"{speaker}_{fn}_gt.wav": audio,f"{speaker}_{fn}_pred.wav": signal}) # report test_loss /= args.train.batch_size test_loss /= num_batches # check print(' [test_loss] test_loss:', test_loss) print(' Real Time Factor', np.mean(rtf_all)) return test_loss def train(args, initial_global_step, model, optimizer, scheduler, vocoder, loader_train, loader_test): # saver saver = Saver(args, initial_global_step=initial_global_step) # model size params_count = utils.get_network_paras_amount({'model': model}) saver.log_info('--- model size ---') saver.log_info(params_count) # run num_batches = len(loader_train) model.train() saver.log_info('======= start training =======') scaler = GradScaler() if args.train.amp_dtype == 'fp32': dtype = torch.float32 elif args.train.amp_dtype == 'fp16': dtype = torch.float16 elif args.train.amp_dtype == 'bf16': dtype = torch.bfloat16 else: raise ValueError(' [x] Unknown amp_dtype: ' + args.train.amp_dtype) saver.log_info("epoch|batch_idx/num_batches|output_dir|batch/s|lr|time|step") for epoch in range(args.train.epochs): for batch_idx, data in enumerate(loader_train): saver.global_step_increment() optimizer.zero_grad() # unpack data for k in data.keys(): if not k.startswith('name'): data[k] = data[k].to(args.device) # forward if dtype == torch.float32: loss = model(data['units'].float(), data['f0'], data['volume'], data['spk_id'], aug_shift = data['aug_shift'], gt_spec=data['mel'].float(), infer=False, k_step=model.k_step_max) else: with autocast(device_type=args.device, dtype=dtype): loss = model(data['units'], data['f0'], data['volume'], data['spk_id'], aug_shift = data['aug_shift'], gt_spec=data['mel'], infer=False, k_step=model.k_step_max) # handle nan loss if torch.isnan(loss): raise ValueError(' [x] nan loss ') else: # backpropagate if dtype == torch.float32: loss.backward() optimizer.step() else: scaler.scale(loss).backward() scaler.step(optimizer) scaler.update() scheduler.step() # log loss if saver.global_step % args.train.interval_log == 0: current_lr = optimizer.param_groups[0]['lr'] saver.log_info( 'epoch: {} | {:3d}/{:3d} | {} | batch/s: {:.2f} | lr: {:.6} | loss: {:.3f} | time: {} | step: {}'.format( epoch, batch_idx, num_batches, args.env.expdir, args.train.interval_log/saver.get_interval_time(), current_lr, loss.item(), saver.get_total_time(), saver.global_step ) ) saver.log_value({ 'train/loss': loss.item() }) saver.log_value({ 'train/lr': current_lr }) # validation if saver.global_step % args.train.interval_val == 0: optimizer_save = optimizer if args.train.save_opt else None # save latest saver.save_model(model, optimizer_save, postfix=f'{saver.global_step}') last_val_step = saver.global_step - args.train.interval_val if last_val_step % args.train.interval_force_save != 0: saver.delete_model(postfix=f'{last_val_step}') # run testing set test_loss = test(args, model, vocoder, loader_test, saver) # log loss saver.log_info( ' --- --- \nloss: {:.3f}. '.format( test_loss, ) ) saver.log_value({ 'validation/loss': test_loss }) model.train() so-vits-svc-npu/diffusion/uni_pc.py000066400000000000000000000766111456161405400177110ustar00rootroot00000000000000import math import mindtorch.torch as torch class NoiseScheduleVP: def __init__( self, schedule='discrete', betas=None, alphas_cumprod=None, continuous_beta_0=0.1, continuous_beta_1=20., dtype=torch.float32, ): """Create a wrapper class for the forward SDE (VP type). *** Update: We support discrete-time diffusion models by implementing a picewise linear interpolation for log_alpha_t. We recommend to use schedule='discrete' for the discrete-time diffusion models, especially for high-resolution images. *** The forward SDE ensures that the condition distribution q_{t|0}(x_t | x_0) = N ( alpha_t * x_0, sigma_t^2 * I ). We further define lambda_t = log(alpha_t) - log(sigma_t), which is the half-logSNR (described in the DPM-Solver paper). Therefore, we implement the functions for computing alpha_t, sigma_t and lambda_t. For t in [0, T], we have: log_alpha_t = self.marginal_log_mean_coeff(t) sigma_t = self.marginal_std(t) lambda_t = self.marginal_lambda(t) Moreover, as lambda(t) is an invertible function, we also support its inverse function: t = self.inverse_lambda(lambda_t) =============================================================== We support both discrete-time DPMs (trained on n = 0, 1, ..., N-1) and continuous-time DPMs (trained on t in [t_0, T]). 1. For discrete-time DPMs: For discrete-time DPMs trained on n = 0, 1, ..., N-1, we convert the discrete steps to continuous time steps by: t_i = (i + 1) / N e.g. for N = 1000, we have t_0 = 1e-3 and T = t_{N-1} = 1. We solve the corresponding diffusion ODE from time T = 1 to time t_0 = 1e-3. Args: betas: A `torch.Tensor`. The beta array for the discrete-time DPM. (See the original DDPM paper for details) alphas_cumprod: A `torch.Tensor`. The cumprod alphas for the discrete-time DPM. (See the original DDPM paper for details) Note that we always have alphas_cumprod = cumprod(1 - betas). Therefore, we only need to set one of `betas` and `alphas_cumprod`. **Important**: Please pay special attention for the args for `alphas_cumprod`: The `alphas_cumprod` is the \hat{alpha_n} arrays in the notations of DDPM. Specifically, DDPMs assume that q_{t_n | 0}(x_{t_n} | x_0) = N ( \sqrt{\hat{alpha_n}} * x_0, (1 - \hat{alpha_n}) * I ). Therefore, the notation \hat{alpha_n} is different from the notation alpha_t in DPM-Solver. In fact, we have alpha_{t_n} = \sqrt{\hat{alpha_n}}, and log(alpha_{t_n}) = 0.5 * log(\hat{alpha_n}). 2. For continuous-time DPMs: We support two types of VPSDEs: linear (DDPM) and cosine (improved-DDPM). The hyperparameters for the noise schedule are the default settings in DDPM and improved-DDPM: Args: beta_min: A `float` number. The smallest beta for the linear schedule. beta_max: A `float` number. The largest beta for the linear schedule. cosine_s: A `float` number. The hyperparameter in the cosine schedule. cosine_beta_max: A `float` number. The hyperparameter in the cosine schedule. T: A `float` number. The ending time of the forward process. =============================================================== Args: schedule: A `str`. The noise schedule of the forward SDE. 'discrete' for discrete-time DPMs, 'linear' or 'cosine' for continuous-time DPMs. Returns: A wrapper object of the forward SDE (VP type). =============================================================== Example: # For discrete-time DPMs, given betas (the beta array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', betas=betas) # For discrete-time DPMs, given alphas_cumprod (the \hat{alpha_n} array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', alphas_cumprod=alphas_cumprod) # For continuous-time DPMs (VPSDE), linear schedule: >>> ns = NoiseScheduleVP('linear', continuous_beta_0=0.1, continuous_beta_1=20.) """ if schedule not in ['discrete', 'linear', 'cosine']: raise ValueError("Unsupported noise schedule {}. The schedule needs to be 'discrete' or 'linear' or 'cosine'".format(schedule)) self.schedule = schedule if schedule == 'discrete': if betas is not None: log_alphas = 0.5 * torch.log(1 - betas).cumsum(dim=0) else: assert alphas_cumprod is not None log_alphas = 0.5 * torch.log(alphas_cumprod) self.total_N = len(log_alphas) self.T = 1. self.t_array = torch.linspace(0., 1., self.total_N + 1)[1:].reshape((1, -1)).to(dtype=dtype) self.log_alpha_array = log_alphas.reshape((1, -1,)).to(dtype=dtype) else: self.total_N = 1000 self.beta_0 = continuous_beta_0 self.beta_1 = continuous_beta_1 self.cosine_s = 0.008 self.cosine_beta_max = 999. self.cosine_t_max = math.atan(self.cosine_beta_max * (1. + self.cosine_s) / math.pi) * 2. * (1. + self.cosine_s) / math.pi - self.cosine_s self.cosine_log_alpha_0 = math.log(math.cos(self.cosine_s / (1. + self.cosine_s) * math.pi / 2.)) self.schedule = schedule if schedule == 'cosine': # For the cosine schedule, T = 1 will have numerical issues. So we manually set the ending time T. # Note that T = 0.9946 may be not the optimal setting. However, we find it works well. self.T = 0.9946 else: self.T = 1. def marginal_log_mean_coeff(self, t): """ Compute log(alpha_t) of a given continuous-time label t in [0, T]. """ if self.schedule == 'discrete': return interpolate_fn(t.reshape((-1, 1)), self.t_array.to(t.device), self.log_alpha_array.to(t.device)).reshape((-1)) elif self.schedule == 'linear': return -0.25 * t ** 2 * (self.beta_1 - self.beta_0) - 0.5 * t * self.beta_0 elif self.schedule == 'cosine': def log_alpha_fn(s): return torch.log(torch.cos((s + self.cosine_s) / (1.0 + self.cosine_s) * math.pi / 2.0)) log_alpha_t = log_alpha_fn(t) - self.cosine_log_alpha_0 return log_alpha_t def marginal_alpha(self, t): """ Compute alpha_t of a given continuous-time label t in [0, T]. """ return torch.exp(self.marginal_log_mean_coeff(t)) def marginal_std(self, t): """ Compute sigma_t of a given continuous-time label t in [0, T]. """ return torch.sqrt(1. - torch.exp(2. * self.marginal_log_mean_coeff(t))) def marginal_lambda(self, t): """ Compute lambda_t = log(alpha_t) - log(sigma_t) of a given continuous-time label t in [0, T]. """ log_mean_coeff = self.marginal_log_mean_coeff(t) log_std = 0.5 * torch.log(1. - torch.exp(2. * log_mean_coeff)) return log_mean_coeff - log_std def inverse_lambda(self, lamb): """ Compute the continuous-time label t in [0, T] of a given half-logSNR lambda_t. """ if self.schedule == 'linear': tmp = 2. * (self.beta_1 - self.beta_0) * torch.logaddexp(-2. * lamb, torch.zeros((1,)).to(lamb)) Delta = self.beta_0**2 + tmp return tmp / (torch.sqrt(Delta) + self.beta_0) / (self.beta_1 - self.beta_0) elif self.schedule == 'discrete': log_alpha = -0.5 * torch.logaddexp(torch.zeros((1,)).to(lamb.device), -2. * lamb) t = interpolate_fn(log_alpha.reshape((-1, 1)), torch.flip(self.log_alpha_array.to(lamb.device), [1]), torch.flip(self.t_array.to(lamb.device), [1])) return t.reshape((-1,)) else: log_alpha = -0.5 * torch.logaddexp(-2. * lamb, torch.zeros((1,)).to(lamb)) def t_fn(log_alpha_t): return torch.arccos(torch.exp(log_alpha_t + self.cosine_log_alpha_0)) * 2.0 * (1.0 + self.cosine_s) / math.pi - self.cosine_s t = t_fn(log_alpha) return t def model_wrapper( model, noise_schedule, model_type="noise", model_kwargs={}, guidance_type="uncond", condition=None, unconditional_condition=None, guidance_scale=1., classifier_fn=None, classifier_kwargs={}, ): """Create a wrapper function for the noise prediction model. """ def get_model_input_time(t_continuous): """ Convert the continuous-time `t_continuous` (in [epsilon, T]) to the model input time. For discrete-time DPMs, we convert `t_continuous` in [1 / N, 1] to `t_input` in [0, 1000 * (N - 1) / N]. For continuous-time DPMs, we just use `t_continuous`. """ if noise_schedule.schedule == 'discrete': return (t_continuous - 1. / noise_schedule.total_N) * noise_schedule.total_N else: return t_continuous def noise_pred_fn(x, t_continuous, cond=None): t_input = get_model_input_time(t_continuous) if cond is None: output = model(x, t_input, **model_kwargs) else: output = model(x, t_input, cond, **model_kwargs) if model_type == "noise": return output elif model_type == "x_start": alpha_t, sigma_t = noise_schedule.marginal_alpha(t_continuous), noise_schedule.marginal_std(t_continuous) return (x - alpha_t * output) / sigma_t elif model_type == "v": alpha_t, sigma_t = noise_schedule.marginal_alpha(t_continuous), noise_schedule.marginal_std(t_continuous) return alpha_t * output + sigma_t * x elif model_type == "score": sigma_t = noise_schedule.marginal_std(t_continuous) return -sigma_t * output def cond_grad_fn(x, t_input): """ Compute the gradient of the classifier, i.e. nabla_{x} log p_t(cond | x_t). """ with torch.enable_grad(): x_in = x.detach().requires_grad_(True) log_prob = classifier_fn(x_in, t_input, condition, **classifier_kwargs) return torch.autograd.grad(log_prob.sum(), x_in)[0] def model_fn(x, t_continuous): """ The noise predicition model function that is used for DPM-Solver. """ if guidance_type == "uncond": return noise_pred_fn(x, t_continuous) elif guidance_type == "classifier": assert classifier_fn is not None t_input = get_model_input_time(t_continuous) cond_grad = cond_grad_fn(x, t_input) sigma_t = noise_schedule.marginal_std(t_continuous) noise = noise_pred_fn(x, t_continuous) return noise - guidance_scale * sigma_t * cond_grad elif guidance_type == "classifier-free": if guidance_scale == 1. or unconditional_condition is None: return noise_pred_fn(x, t_continuous, cond=condition) else: x_in = torch.cat([x] * 2) t_in = torch.cat([t_continuous] * 2) c_in = torch.cat([unconditional_condition, condition]) noise_uncond, noise = noise_pred_fn(x_in, t_in, cond=c_in).chunk(2) return noise_uncond + guidance_scale * (noise - noise_uncond) assert model_type in ["noise", "x_start", "v"] assert guidance_type in ["uncond", "classifier", "classifier-free"] return model_fn class UniPC: def __init__( self, model_fn, noise_schedule, algorithm_type="data_prediction", correcting_x0_fn=None, correcting_xt_fn=None, thresholding_max_val=1., dynamic_thresholding_ratio=0.995, variant='bh1' ): """Construct a UniPC. We support both data_prediction and noise_prediction. """ self.model = lambda x, t: model_fn(x, t.expand((x.shape[0]))) self.noise_schedule = noise_schedule assert algorithm_type in ["data_prediction", "noise_prediction"] if correcting_x0_fn == "dynamic_thresholding": self.correcting_x0_fn = self.dynamic_thresholding_fn else: self.correcting_x0_fn = correcting_x0_fn self.correcting_xt_fn = correcting_xt_fn self.dynamic_thresholding_ratio = dynamic_thresholding_ratio self.thresholding_max_val = thresholding_max_val self.variant = variant self.predict_x0 = algorithm_type == "data_prediction" def dynamic_thresholding_fn(self, x0, t=None): """ The dynamic thresholding method. """ dims = x0.dim() p = self.dynamic_thresholding_ratio s = torch.quantile(torch.abs(x0).reshape((x0.shape[0], -1)), p, dim=1) s = expand_dims(torch.maximum(s, self.thresholding_max_val * torch.ones_like(s).to(s.device)), dims) x0 = torch.clamp(x0, -s, s) / s return x0 def noise_prediction_fn(self, x, t): """ Return the noise prediction model. """ return self.model(x, t) def data_prediction_fn(self, x, t): """ Return the data prediction model (with corrector). """ noise = self.noise_prediction_fn(x, t) alpha_t, sigma_t = self.noise_schedule.marginal_alpha(t), self.noise_schedule.marginal_std(t) x0 = (x - sigma_t * noise) / alpha_t if self.correcting_x0_fn is not None: x0 = self.correcting_x0_fn(x0) return x0 def model_fn(self, x, t): """ Convert the model to the noise prediction model or the data prediction model. """ if self.predict_x0: return self.data_prediction_fn(x, t) else: return self.noise_prediction_fn(x, t) def get_time_steps(self, skip_type, t_T, t_0, N, device): """Compute the intermediate time steps for sampling. """ if skip_type == 'logSNR': lambda_T = self.noise_schedule.marginal_lambda(torch.tensor(t_T).to(device)) lambda_0 = self.noise_schedule.marginal_lambda(torch.tensor(t_0).to(device)) logSNR_steps = torch.linspace(lambda_T.cpu().item(), lambda_0.cpu().item(), N + 1).to(device) return self.noise_schedule.inverse_lambda(logSNR_steps) elif skip_type == 'time_uniform': return torch.linspace(t_T, t_0, N + 1).to(device) elif skip_type == 'time_quadratic': t_order = 2 t = torch.linspace(t_T**(1. / t_order), t_0**(1. / t_order), N + 1).pow(t_order).to(device) return t else: raise ValueError("Unsupported skip_type {}, need to be 'logSNR' or 'time_uniform' or 'time_quadratic'".format(skip_type)) def get_orders_and_timesteps_for_singlestep_solver(self, steps, order, skip_type, t_T, t_0, device): """ Get the order of each step for sampling by the singlestep DPM-Solver. """ if order == 3: K = steps // 3 + 1 if steps % 3 == 0: orders = [3,] * (K - 2) + [2, 1] elif steps % 3 == 1: orders = [3,] * (K - 1) + [1] else: orders = [3,] * (K - 1) + [2] elif order == 2: if steps % 2 == 0: K = steps // 2 orders = [2,] * K else: K = steps // 2 + 1 orders = [2,] * (K - 1) + [1] elif order == 1: K = steps orders = [1,] * steps else: raise ValueError("'order' must be '1' or '2' or '3'.") if skip_type == 'logSNR': # To reproduce the results in DPM-Solver paper timesteps_outer = self.get_time_steps(skip_type, t_T, t_0, K, device) else: timesteps_outer = self.get_time_steps(skip_type, t_T, t_0, steps, device)[torch.cumsum(torch.tensor([0,] + orders), 0).to(device)] return timesteps_outer, orders def denoise_to_zero_fn(self, x, s): """ Denoise at the final step, which is equivalent to solve the ODE from lambda_s to infty by first-order discretization. """ return self.data_prediction_fn(x, s) def multistep_uni_pc_update(self, x, model_prev_list, t_prev_list, t, order, **kwargs): if len(t.shape) == 0: t = t.view(-1) if 'bh' in self.variant: return self.multistep_uni_pc_bh_update(x, model_prev_list, t_prev_list, t, order, **kwargs) else: assert self.variant == 'vary_coeff' return self.multistep_uni_pc_vary_update(x, model_prev_list, t_prev_list, t, order, **kwargs) def multistep_uni_pc_vary_update(self, x, model_prev_list, t_prev_list, t, order, use_corrector=True): #print(f'using unified predictor-corrector with order {order} (solver type: vary coeff)') ns = self.noise_schedule assert order <= len(model_prev_list) # first compute rks t_prev_0 = t_prev_list[-1] lambda_prev_0 = ns.marginal_lambda(t_prev_0) lambda_t = ns.marginal_lambda(t) model_prev_0 = model_prev_list[-1] sigma_prev_0, sigma_t = ns.marginal_std(t_prev_0), ns.marginal_std(t) log_alpha_t = ns.marginal_log_mean_coeff(t) alpha_t = torch.exp(log_alpha_t) h = lambda_t - lambda_prev_0 rks = [] D1s = [] for i in range(1, order): t_prev_i = t_prev_list[-(i + 1)] model_prev_i = model_prev_list[-(i + 1)] lambda_prev_i = ns.marginal_lambda(t_prev_i) rk = (lambda_prev_i - lambda_prev_0) / h rks.append(rk) D1s.append((model_prev_i - model_prev_0) / rk) rks.append(1.) rks = torch.tensor(rks, device=x.device) K = len(rks) # build C matrix C = [] col = torch.ones_like(rks) for k in range(1, K + 1): C.append(col) col = col * rks / (k + 1) C = torch.stack(C, dim=1) if len(D1s) > 0: D1s = torch.stack(D1s, dim=1) # (B, K) C_inv_p = torch.linalg.inv(C[:-1, :-1]) A_p = C_inv_p if use_corrector: #print('using corrector') C_inv = torch.linalg.inv(C) A_c = C_inv hh = -h if self.predict_x0 else h h_phi_1 = torch.expm1(hh) h_phi_ks = [] factorial_k = 1 h_phi_k = h_phi_1 for k in range(1, K + 2): h_phi_ks.append(h_phi_k) h_phi_k = h_phi_k / hh - 1 / factorial_k factorial_k *= (k + 1) model_t = None if self.predict_x0: x_t_ = ( sigma_t / sigma_prev_0 * x - alpha_t * h_phi_1 * model_prev_0 ) # now predictor x_t = x_t_ if len(D1s) > 0: # compute the residuals for predictor for k in range(K - 1): x_t = x_t - alpha_t * h_phi_ks[k + 1] * torch.einsum('bkchw,k->bchw', D1s, A_p[k]) # now corrector if use_corrector: model_t = self.model_fn(x_t, t) D1_t = (model_t - model_prev_0) x_t = x_t_ k = 0 for k in range(K - 1): x_t = x_t - alpha_t * h_phi_ks[k + 1] * torch.einsum('bkchw,k->bchw', D1s, A_c[k][:-1]) x_t = x_t - alpha_t * h_phi_ks[K] * (D1_t * A_c[k][-1]) else: log_alpha_prev_0, log_alpha_t = ns.marginal_log_mean_coeff(t_prev_0), ns.marginal_log_mean_coeff(t) x_t_ = ( (torch.exp(log_alpha_t - log_alpha_prev_0)) * x - (sigma_t * h_phi_1) * model_prev_0 ) # now predictor x_t = x_t_ if len(D1s) > 0: # compute the residuals for predictor for k in range(K - 1): x_t = x_t - sigma_t * h_phi_ks[k + 1] * torch.einsum('bkchw,k->bchw', D1s, A_p[k]) # now corrector if use_corrector: model_t = self.model_fn(x_t, t) D1_t = (model_t - model_prev_0) x_t = x_t_ k = 0 for k in range(K - 1): x_t = x_t - sigma_t * h_phi_ks[k + 1] * torch.einsum('bkchw,k->bchw', D1s, A_c[k][:-1]) x_t = x_t - sigma_t * h_phi_ks[K] * (D1_t * A_c[k][-1]) return x_t, model_t def multistep_uni_pc_bh_update(self, x, model_prev_list, t_prev_list, t, order, x_t=None, use_corrector=True): #print(f'using unified predictor-corrector with order {order} (solver type: B(h))') ns = self.noise_schedule assert order <= len(model_prev_list) # first compute rks t_prev_0 = t_prev_list[-1] lambda_prev_0 = ns.marginal_lambda(t_prev_0) lambda_t = ns.marginal_lambda(t) model_prev_0 = model_prev_list[-1] sigma_prev_0, sigma_t = ns.marginal_std(t_prev_0), ns.marginal_std(t) log_alpha_prev_0, log_alpha_t = ns.marginal_log_mean_coeff(t_prev_0), ns.marginal_log_mean_coeff(t) alpha_t = torch.exp(log_alpha_t) h = lambda_t - lambda_prev_0 rks = [] D1s = [] for i in range(1, order): t_prev_i = t_prev_list[-(i + 1)] model_prev_i = model_prev_list[-(i + 1)] lambda_prev_i = ns.marginal_lambda(t_prev_i) rk = (lambda_prev_i - lambda_prev_0) / h rks.append(rk) D1s.append((model_prev_i - model_prev_0) / rk) rks.append(1.) rks = torch.tensor(rks, device=x.device) R = [] b = [] hh = -h if self.predict_x0 else h h_phi_1 = torch.expm1(hh) # h\phi_1(h) = e^h - 1 h_phi_k = h_phi_1 / hh - 1 factorial_i = 1 if self.variant == 'bh1': B_h = hh elif self.variant == 'bh2': B_h = torch.expm1(hh) else: raise NotImplementedError() for i in range(1, order + 1): R.append(torch.pow(rks, i - 1)) b.append(h_phi_k * factorial_i / B_h) factorial_i *= (i + 1) h_phi_k = h_phi_k / hh - 1 / factorial_i R = torch.stack(R) b = torch.cat(b) # now predictor use_predictor = len(D1s) > 0 and x_t is None if len(D1s) > 0: D1s = torch.stack(D1s, dim=1) # (B, K) if x_t is None: # for order 2, we use a simplified version if order == 2: rhos_p = torch.tensor([0.5], device=b.device) else: rhos_p = torch.linalg.solve(R[:-1, :-1], b[:-1]) else: D1s = None if use_corrector: #print('using corrector') # for order 1, we use a simplified version if order == 1: rhos_c = torch.tensor([0.5], device=b.device) else: rhos_c = torch.linalg.solve(R, b) model_t = None if self.predict_x0: x_t_ = ( sigma_t / sigma_prev_0 * x - alpha_t * h_phi_1 * model_prev_0 ) if x_t is None: if use_predictor: pred_res = torch.einsum('k,bkchw->bchw', rhos_p, D1s) else: pred_res = 0 x_t = x_t_ - alpha_t * B_h * pred_res if use_corrector: model_t = self.model_fn(x_t, t) if D1s is not None: corr_res = torch.einsum('k,bkchw->bchw', rhos_c[:-1], D1s) else: corr_res = 0 D1_t = (model_t - model_prev_0) x_t = x_t_ - alpha_t * B_h * (corr_res + rhos_c[-1] * D1_t) else: x_t_ = ( torch.exp(log_alpha_t - log_alpha_prev_0) * x - sigma_t * h_phi_1 * model_prev_0 ) if x_t is None: if use_predictor: pred_res = torch.einsum('k,bkchw->bchw', rhos_p, D1s) else: pred_res = 0 x_t = x_t_ - sigma_t * B_h * pred_res if use_corrector: model_t = self.model_fn(x_t, t) if D1s is not None: corr_res = torch.einsum('k,bkchw->bchw', rhos_c[:-1], D1s) else: corr_res = 0 D1_t = (model_t - model_prev_0) x_t = x_t_ - sigma_t * B_h * (corr_res + rhos_c[-1] * D1_t) return x_t, model_t def sample(self, x, steps=20, t_start=None, t_end=None, order=2, skip_type='time_uniform', method='multistep', lower_order_final=True, denoise_to_zero=False, atol=0.0078, rtol=0.05, return_intermediate=False, ): """ Compute the sample at time `t_end` by UniPC, given the initial `x` at time `t_start`. """ t_0 = 1. / self.noise_schedule.total_N if t_end is None else t_end t_T = self.noise_schedule.T if t_start is None else t_start assert t_0 > 0 and t_T > 0, "Time range needs to be greater than 0. For discrete-time DPMs, it needs to be in [1 / N, 1], where N is the length of betas array" if return_intermediate: assert method in ['multistep', 'singlestep', 'singlestep_fixed'], "Cannot use adaptive solver when saving intermediate values" if self.correcting_xt_fn is not None: assert method in ['multistep', 'singlestep', 'singlestep_fixed'], "Cannot use adaptive solver when correcting_xt_fn is not None" device = x.device intermediates = [] with torch.no_grad(): if method == 'multistep': assert steps >= order timesteps = self.get_time_steps(skip_type=skip_type, t_T=t_T, t_0=t_0, N=steps, device=device) assert timesteps.shape[0] - 1 == steps # Init the initial values. step = 0 t = timesteps[step] t_prev_list = [t] model_prev_list = [self.model_fn(x, t)] if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) # Init the first `order` values by lower order multistep UniPC. for step in range(1, order): t = timesteps[step] x, model_x = self.multistep_uni_pc_update(x, model_prev_list, t_prev_list, t, step, use_corrector=True) if model_x is None: model_x = self.model_fn(x, t) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) t_prev_list.append(t) model_prev_list.append(model_x) # Compute the remaining values by `order`-th order multistep DPM-Solver. for step in range(order, steps + 1): t = timesteps[step] if lower_order_final: step_order = min(order, steps + 1 - step) else: step_order = order if step == steps: #print('do not run corrector at the last step') use_corrector = False else: use_corrector = True x, model_x = self.multistep_uni_pc_update(x, model_prev_list, t_prev_list, t, step_order, use_corrector=use_corrector) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step) if return_intermediate: intermediates.append(x) for i in range(order - 1): t_prev_list[i] = t_prev_list[i + 1] model_prev_list[i] = model_prev_list[i + 1] t_prev_list[-1] = t # We do not need to evaluate the final model value. if step < steps: if model_x is None: model_x = self.model_fn(x, t) model_prev_list[-1] = model_x else: raise ValueError("Got wrong method {}".format(method)) if denoise_to_zero: t = torch.ones((1,)).to(device) * t_0 x = self.denoise_to_zero_fn(x, t) if self.correcting_xt_fn is not None: x = self.correcting_xt_fn(x, t, step + 1) if return_intermediate: intermediates.append(x) if return_intermediate: return x, intermediates else: return x ############################################################# # other utility functions ############################################################# def interpolate_fn(x, xp, yp): """ A piecewise linear function y = f(x), using xp and yp as keypoints. We implement f(x) in a differentiable way (i.e. applicable for autograd). The function f(x) is well-defined for all x-axis. (For x beyond the bounds of xp, we use the outmost points of xp to define the linear function.) Args: x: PyTorch tensor with shape [N, C], where N is the batch size, C is the number of channels (we use C = 1 for DPM-Solver). xp: PyTorch tensor with shape [C, K], where K is the number of keypoints. yp: PyTorch tensor with shape [C, K]. Returns: The function values f(x), with shape [N, C]. """ N, K = x.shape[0], xp.shape[1] all_x = torch.cat([x.unsqueeze(2), xp.unsqueeze(0).repeat((N, 1, 1))], dim=2) sorted_all_x, x_indices = torch.sort(all_x, dim=2) x_idx = torch.argmin(x_indices, dim=2) cand_start_idx = x_idx - 1 start_idx = torch.where( torch.eq(x_idx, 0), torch.tensor(1, device=x.device), torch.where( torch.eq(x_idx, K), torch.tensor(K - 2, device=x.device), cand_start_idx, ), ) end_idx = torch.where(torch.eq(start_idx, cand_start_idx), start_idx + 2, start_idx + 1) start_x = torch.gather(sorted_all_x, dim=2, index=start_idx.unsqueeze(2)).squeeze(2) end_x = torch.gather(sorted_all_x, dim=2, index=end_idx.unsqueeze(2)).squeeze(2) start_idx2 = torch.where( torch.eq(x_idx, 0), torch.tensor(0, device=x.device), torch.where( torch.eq(x_idx, K), torch.tensor(K - 2, device=x.device), cand_start_idx, ), ) y_positions_expanded = yp.unsqueeze(0).expand(N, -1, -1) start_y = torch.gather(y_positions_expanded, dim=2, index=start_idx2.unsqueeze(2)).squeeze(2) end_y = torch.gather(y_positions_expanded, dim=2, index=(start_idx2 + 1).unsqueeze(2)).squeeze(2) cand = start_y + (x - start_x) * (end_y - start_y) / (end_x - start_x) return cand def expand_dims(v, dims): """ Expand the tensor `v` to the dim `dims`. Args: `v`: a PyTorch tensor with shape [N]. `dim`: a `int`. Returns: a PyTorch tensor with shape [N, 1, 1, ..., 1] and the total dimension is `dims`. """ return v[(...,) + (None,)*(dims - 1)]so-vits-svc-npu/diffusion/unit2mel.py000066400000000000000000000145211456161405400201630ustar00rootroot00000000000000import os import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import yaml from .diffusion import GaussianDiffusion from .vocoder import Vocoder from .wavenet import WaveNet class DotDict(dict): def __getattr__(*args): val = dict.get(*args) return DotDict(val) if type(val) is dict else val __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ def load_model_vocoder( model_path, device='cpu', config_path = None ): if config_path is None: config_file = os.path.join(os.path.split(model_path)[0], 'config.yaml') else: config_file = config_path with open(config_file, "r") as config: args = yaml.safe_load(config) args = DotDict(args) # load vocoder vocoder = Vocoder(args.vocoder.type, args.vocoder.ckpt, device=device) # load model model = Unit2Mel( args.data.encoder_out_channels, args.model.n_spk, args.model.use_pitch_aug, vocoder.dimension, args.model.n_layers, args.model.n_chans, args.model.n_hidden, args.model.timesteps, args.model.k_step_max ) print(' [Loading] ' + model_path) ckpt = torch.load(model_path, map_location=torch.device(device)) model.to(device) model.load_state_dict(ckpt['model']) model.eval() print(f'Loaded diffusion model, sampler is {args.infer.method}, speedup: {args.infer.speedup} ') return model, vocoder, args class Unit2Mel(nn.Module): def __init__( self, input_channel, n_spk, use_pitch_aug=False, out_dims=128, n_layers=20, n_chans=384, n_hidden=256, timesteps=1000, k_step_max=1000 ): super().__init__() self.unit_embed = nn.Linear(input_channel, n_hidden) self.f0_embed = nn.Linear(1, n_hidden) self.volume_embed = nn.Linear(1, n_hidden) if use_pitch_aug: self.aug_shift_embed = nn.Linear(1, n_hidden, bias=False) else: self.aug_shift_embed = None self.n_spk = n_spk if n_spk is not None and n_spk > 1: self.spk_embed = nn.Embedding(n_spk, n_hidden) self.timesteps = timesteps if timesteps is not None else 1000 self.k_step_max = k_step_max if k_step_max is not None and k_step_max>0 and k_step_max 1: if spk_mix_dict is not None: spk_embed_mix = torch.zeros((1,1,self.hidden_size)) for k, v in spk_mix_dict.items(): spk_id_torch = torch.LongTensor(np.array([[k]])).to(units.device) spk_embeddd = self.spk_embed(spk_id_torch) self.speaker_map[k] = spk_embeddd spk_embed_mix = spk_embed_mix + v * spk_embeddd x = x + spk_embed_mix else: x = x + self.spk_embed(spk_id - 1) self.speaker_map = self.speaker_map.unsqueeze(0) self.speaker_map = self.speaker_map.detach() return x.transpose(1, 2) def init_spkmix(self, n_spk): self.speaker_map = torch.zeros((n_spk,1,1,self.n_hidden)) hubert_hidden_size = self.input_channel n_frames = 10 hubert = torch.randn((1, n_frames, hubert_hidden_size)) f0 = torch.randn((1, n_frames)) volume = torch.randn((1, n_frames)) spks = {} for i in range(n_spk): spks.update({i:1.0/float(self.n_spk)}) self.init_spkembed(hubert, f0.unsqueeze(-1), volume.unsqueeze(-1), spk_mix_dict=spks) def forward(self, units, f0, volume, spk_id = None, spk_mix_dict = None, aug_shift = None, gt_spec=None, infer=True, infer_speedup=10, method='dpm-solver', k_step=300, use_tqdm=True): ''' input: B x n_frames x n_unit return: dict of B x n_frames x feat ''' if not self.training and gt_spec is not None and k_step>self.k_step_max: raise Exception("The shallow diffusion k_step is greater than the maximum diffusion k_step(k_step_max)!") if not self.training and gt_spec is None and self.k_step_max!=self.timesteps: raise Exception("This model can only be used for shallow diffusion and can not infer alone!") x = self.unit_embed(units) + self.f0_embed((1+ f0 / 700).log()) + self.volume_embed(volume) if self.n_spk is not None and self.n_spk > 1: if spk_mix_dict is not None: for k, v in spk_mix_dict.items(): spk_id_torch = torch.LongTensor(np.array([[k]])).to(units.device) x = x + v * self.spk_embed(spk_id_torch) else: if spk_id.shape[1] > 1: g = spk_id.reshape((spk_id.shape[0], spk_id.shape[1], 1, 1, 1)) # [N, S, B, 1, 1] g = g * self.speaker_map # [N, S, B, 1, H] g = torch.sum(g, dim=1) # [N, 1, B, 1, H] g = g.transpose(0, -1).transpose(0, -2).squeeze(0) # [B, H, N] x = x + g else: x = x + self.spk_embed(spk_id) if self.aug_shift_embed is not None and aug_shift is not None: x = x + self.aug_shift_embed(aug_shift / 5) x = self.decoder(x, gt_spec=gt_spec, infer=infer, infer_speedup=infer_speedup, method=method, k_step=k_step, use_tqdm=use_tqdm) return x so-vits-svc-npu/diffusion/vocoder.py000066400000000000000000000065071456161405400200720ustar00rootroot00000000000000import mindtorch.torch as torch from mindtorch.torchaudio.transforms import Resample from vdecoder.nsf_hifigan.models import load_config, load_model from vdecoder.nsf_hifigan.nvSTFT import STFT class Vocoder: def __init__(self, vocoder_type, vocoder_ckpt, device = None): if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device if vocoder_type == 'nsf-hifigan': self.vocoder = NsfHifiGAN(vocoder_ckpt, device = device) elif vocoder_type == 'nsf-hifigan-log10': self.vocoder = NsfHifiGANLog10(vocoder_ckpt, device = device) else: raise ValueError(f" [x] Unknown vocoder: {vocoder_type}") self.resample_kernel = {} self.vocoder_sample_rate = self.vocoder.sample_rate() self.vocoder_hop_size = self.vocoder.hop_size() self.dimension = self.vocoder.dimension() def extract(self, audio, sample_rate, keyshift=0): # resample if sample_rate == self.vocoder_sample_rate: audio_res = audio else: key_str = str(sample_rate) if key_str not in self.resample_kernel: self.resample_kernel[key_str] = Resample(sample_rate, self.vocoder_sample_rate, lowpass_filter_width = 128).to(self.device) audio_res = self.resample_kernel[key_str](audio) # extract mel = self.vocoder.extract(audio_res, keyshift=keyshift) # B, n_frames, bins return mel def infer(self, mel, f0): f0 = f0[:,:mel.size(1),0] # B, n_frames audio = self.vocoder(mel, f0) return audio class NsfHifiGAN(torch.nn.Module): def __init__(self, model_path, device=None): super().__init__() if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device self.model_path = model_path self.model = None self.h = load_config(model_path) self.stft = STFT( self.h.sampling_rate, self.h.num_mels, self.h.n_fft, self.h.win_size, self.h.hop_size, self.h.fmin, self.h.fmax) def sample_rate(self): return self.h.sampling_rate def hop_size(self): return self.h.hop_size def dimension(self): return self.h.num_mels def extract(self, audio, keyshift=0): mel = self.stft.get_mel(audio, keyshift=keyshift).transpose(1, 2) # B, n_frames, bins return mel def forward(self, mel, f0): if self.model is None: print('| Load HifiGAN: ', self.model_path) self.model, self.h = load_model(self.model_path, device=self.device) with torch.no_grad(): c = mel.transpose(1, 2) audio = self.model(c, f0) return audio class NsfHifiGANLog10(NsfHifiGAN): def forward(self, mel, f0): if self.model is None: print('| Load HifiGAN: ', self.model_path) self.model, self.h = load_model(self.model_path, device=self.device) with torch.no_grad(): c = 0.434294 * mel.transpose(1, 2) audio = self.model(c, f0) return audioso-vits-svc-npu/diffusion/wavenet.py000066400000000000000000000074031456161405400200760ustar00rootroot00000000000000import math from math import sqrt import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn import Mish class Conv1d(torch.nn.Conv1d): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) nn.init.kaiming_normal_(self.weight) class SinusoidalPosEmb(nn.Module): def __init__(self, dim): super().__init__() self.dim = dim def forward(self, x): device = x.device half_dim = self.dim // 2 emb = math.log(10000) / (half_dim - 1) emb = torch.exp(torch.arange(half_dim, device=device) * -emb) emb = x[:, None] * emb[None, :] emb = torch.cat((emb.sin(), emb.cos()), dim=-1) return emb class ResidualBlock(nn.Module): def __init__(self, encoder_hidden, residual_channels, dilation): super().__init__() self.residual_channels = residual_channels self.dilated_conv = nn.Conv1d( residual_channels, 2 * residual_channels, kernel_size=3, padding=dilation, dilation=dilation ) self.diffusion_projection = nn.Linear(residual_channels, residual_channels) self.conditioner_projection = nn.Conv1d(encoder_hidden, 2 * residual_channels, 1) self.output_projection = nn.Conv1d(residual_channels, 2 * residual_channels, 1) def forward(self, x, conditioner, diffusion_step): diffusion_step = self.diffusion_projection(diffusion_step).unsqueeze(-1) conditioner = self.conditioner_projection(conditioner) y = x + diffusion_step y = self.dilated_conv(y) + conditioner # Using torch.split instead of torch.chunk to avoid using onnx::Slice gate, filter = torch.split(y, [self.residual_channels, self.residual_channels], dim=1) y = torch.sigmoid(gate) * torch.tanh(filter) y = self.output_projection(y) # Using torch.split instead of torch.chunk to avoid using onnx::Slice residual, skip = torch.split(y, [self.residual_channels, self.residual_channels], dim=1) return (x + residual) / math.sqrt(2.0), skip class WaveNet(nn.Module): def __init__(self, in_dims=128, n_layers=20, n_chans=384, n_hidden=256): super().__init__() self.input_projection = Conv1d(in_dims, n_chans, 1) self.diffusion_embedding = SinusoidalPosEmb(n_chans) self.mlp = nn.Sequential( nn.Linear(n_chans, n_chans * 4), Mish(), nn.Linear(n_chans * 4, n_chans) ) self.residual_layers = nn.ModuleList([ ResidualBlock( encoder_hidden=n_hidden, residual_channels=n_chans, dilation=1 ) for i in range(n_layers) ]) self.skip_projection = Conv1d(n_chans, n_chans, 1) self.output_projection = Conv1d(n_chans, in_dims, 1) nn.init.zeros_(self.output_projection.weight) def forward(self, spec, diffusion_step, cond): """ :param spec: [B, 1, M, T] :param diffusion_step: [B, 1] :param cond: [B, M, T] :return: """ x = spec.squeeze(1) x = self.input_projection(x) # [B, residual_channel, T] x = F.relu(x) diffusion_step = self.diffusion_embedding(diffusion_step) diffusion_step = self.mlp(diffusion_step) skip = [] for layer in self.residual_layers: x, skip_connection = layer(x, cond, diffusion_step) skip.append(skip_connection) x = torch.sum(torch.stack(skip), dim=0) / sqrt(len(self.residual_layers)) x = self.skip_projection(x) x = F.relu(x) x = self.output_projection(x) # [B, mel_bins, T] return x[:, None, :, :] so-vits-svc-npu/edgetts/000077500000000000000000000000001456161405400155205ustar00rootroot00000000000000so-vits-svc-npu/edgetts/tts.py000066400000000000000000000027641456161405400167150ustar00rootroot00000000000000import asyncio import random import sys import edge_tts from edge_tts import VoicesManager from langdetect import DetectorFactory, detect DetectorFactory.seed = 0 TEXT = sys.argv[1] LANG = detect(TEXT) if sys.argv[2] == "Auto" else sys.argv[2] RATE = sys.argv[3] VOLUME = sys.argv[4] GENDER = sys.argv[5] if len(sys.argv) == 6 else None OUTPUT_FILE = "tts.wav" print("Running TTS...") print(f"Text: {TEXT}, Language: {LANG}, Gender: {GENDER}, Rate: {RATE}, Volume: {VOLUME}") async def _main() -> None: voices = await VoicesManager.create() if GENDER is not None: # From "zh-cn" to "zh-CN" etc. if LANG == "zh-cn" or LANG == "zh-tw": LOCALE = LANG[:-2] + LANG[-2:].upper() voice = voices.find(Gender=GENDER, Locale=LOCALE) else: voice = voices.find(Gender=GENDER, Language=LANG) VOICE = random.choice(voice)["Name"] print(f"Using random {LANG} voice: {VOICE}") else: VOICE = LANG communicate = edge_tts.Communicate(text = TEXT, voice = VOICE, rate = RATE, volume = VOLUME) await communicate.save(OUTPUT_FILE) if __name__ == "__main__": if sys.platform.startswith("win"): asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) asyncio.run(_main()) else: loop = asyncio.get_event_loop_policy().get_event_loop() try: loop.run_until_complete(_main()) finally: loop.close() so-vits-svc-npu/edgetts/tts_voices.py000066400000000000000000000250251456161405400202600ustar00rootroot00000000000000#List of Supported Voices for edge_TTS SUPPORTED_VOICES = { 'zh-CN-XiaoxiaoNeural': 'zh-CN', 'zh-CN-XiaoyiNeural': 'zh-CN', 'zh-CN-YunjianNeural': 'zh-CN', 'zh-CN-YunxiNeural': 'zh-CN', 'zh-CN-YunxiaNeural': 'zh-CN', 'zh-CN-YunyangNeural': 'zh-CN', 'zh-HK-HiuGaaiNeural': 'zh-HK', 'zh-HK-HiuMaanNeural': 'zh-HK', 'zh-HK-WanLungNeural': 'zh-HK', 'zh-TW-HsiaoChenNeural': 'zh-TW', 'zh-TW-YunJheNeural': 'zh-TW', 'zh-TW-HsiaoYuNeural': 'zh-TW', 'af-ZA-AdriNeural': 'af-ZA', 'af-ZA-WillemNeural': 'af-ZA', 'am-ET-AmehaNeural': 'am-ET', 'am-ET-MekdesNeural': 'am-ET', 'ar-AE-FatimaNeural': 'ar-AE', 'ar-AE-HamdanNeural': 'ar-AE', 'ar-BH-AliNeural': 'ar-BH', 'ar-BH-LailaNeural': 'ar-BH', 'ar-DZ-AminaNeural': 'ar-DZ', 'ar-DZ-IsmaelNeural': 'ar-DZ', 'ar-EG-SalmaNeural': 'ar-EG', 'ar-EG-ShakirNeural': 'ar-EG', 'ar-IQ-BasselNeural': 'ar-IQ', 'ar-IQ-RanaNeural': 'ar-IQ', 'ar-JO-SanaNeural': 'ar-JO', 'ar-JO-TaimNeural': 'ar-JO', 'ar-KW-FahedNeural': 'ar-KW', 'ar-KW-NouraNeural': 'ar-KW', 'ar-LB-LaylaNeural': 'ar-LB', 'ar-LB-RamiNeural': 'ar-LB', 'ar-LY-ImanNeural': 'ar-LY', 'ar-LY-OmarNeural': 'ar-LY', 'ar-MA-JamalNeural': 'ar-MA', 'ar-MA-MounaNeural': 'ar-MA', 'ar-OM-AbdullahNeural': 'ar-OM', 'ar-OM-AyshaNeural': 'ar-OM', 'ar-QA-AmalNeural': 'ar-QA', 'ar-QA-MoazNeural': 'ar-QA', 'ar-SA-HamedNeural': 'ar-SA', 'ar-SA-ZariyahNeural': 'ar-SA', 'ar-SY-AmanyNeural': 'ar-SY', 'ar-SY-LaithNeural': 'ar-SY', 'ar-TN-HediNeural': 'ar-TN', 'ar-TN-ReemNeural': 'ar-TN', 'ar-YE-MaryamNeural': 'ar-YE', 'ar-YE-SalehNeural': 'ar-YE', 'az-AZ-BabekNeural': 'az-AZ', 'az-AZ-BanuNeural': 'az-AZ', 'bg-BG-BorislavNeural': 'bg-BG', 'bg-BG-KalinaNeural': 'bg-BG', 'bn-BD-NabanitaNeural': 'bn-BD', 'bn-BD-PradeepNeural': 'bn-BD', 'bn-IN-BashkarNeural': 'bn-IN', 'bn-IN-TanishaaNeural': 'bn-IN', 'bs-BA-GoranNeural': 'bs-BA', 'bs-BA-VesnaNeural': 'bs-BA', 'ca-ES-EnricNeural': 'ca-ES', 'ca-ES-JoanaNeural': 'ca-ES', 'cs-CZ-AntoninNeural': 'cs-CZ', 'cs-CZ-VlastaNeural': 'cs-CZ', 'cy-GB-AledNeural': 'cy-GB', 'cy-GB-NiaNeural': 'cy-GB', 'da-DK-ChristelNeural': 'da-DK', 'da-DK-JeppeNeural': 'da-DK', 'de-AT-IngridNeural': 'de-AT', 'de-AT-JonasNeural': 'de-AT', 'de-CH-JanNeural': 'de-CH', 'de-CH-LeniNeural': 'de-CH', 'de-DE-AmalaNeural': 'de-DE', 'de-DE-ConradNeural': 'de-DE', 'de-DE-KatjaNeural': 'de-DE', 'de-DE-KillianNeural': 'de-DE', 'el-GR-AthinaNeural': 'el-GR', 'el-GR-NestorasNeural': 'el-GR', 'en-AU-NatashaNeural': 'en-AU', 'en-AU-WilliamNeural': 'en-AU', 'en-CA-ClaraNeural': 'en-CA', 'en-CA-LiamNeural': 'en-CA', 'en-GB-LibbyNeural': 'en-GB', 'en-GB-MaisieNeural': 'en-GB', 'en-GB-RyanNeural': 'en-GB', 'en-GB-SoniaNeural': 'en-GB', 'en-GB-ThomasNeural': 'en-GB', 'en-HK-SamNeural': 'en-HK', 'en-HK-YanNeural': 'en-HK', 'en-IE-ConnorNeural': 'en-IE', 'en-IE-EmilyNeural': 'en-IE', 'en-IN-NeerjaNeural': 'en-IN', 'en-IN-PrabhatNeural': 'en-IN', 'en-KE-AsiliaNeural': 'en-KE', 'en-KE-ChilembaNeural': 'en-KE', 'en-NG-AbeoNeural': 'en-NG', 'en-NG-EzinneNeural': 'en-NG', 'en-NZ-MitchellNeural': 'en-NZ', 'en-NZ-MollyNeural': 'en-NZ', 'en-PH-JamesNeural': 'en-PH', 'en-PH-RosaNeural': 'en-PH', 'en-SG-LunaNeural': 'en-SG', 'en-SG-WayneNeural': 'en-SG', 'en-TZ-ElimuNeural': 'en-TZ', 'en-TZ-ImaniNeural': 'en-TZ', 'en-US-AnaNeural': 'en-US', 'en-US-AriaNeural': 'en-US', 'en-US-ChristopherNeural': 'en-US', 'en-US-EricNeural': 'en-US', 'en-US-GuyNeural': 'en-US', 'en-US-JennyNeural': 'en-US', 'en-US-MichelleNeural': 'en-US', 'en-ZA-LeahNeural': 'en-ZA', 'en-ZA-LukeNeural': 'en-ZA', 'es-AR-ElenaNeural': 'es-AR', 'es-AR-TomasNeural': 'es-AR', 'es-BO-MarceloNeural': 'es-BO', 'es-BO-SofiaNeural': 'es-BO', 'es-CL-CatalinaNeural': 'es-CL', 'es-CL-LorenzoNeural': 'es-CL', 'es-CO-GonzaloNeural': 'es-CO', 'es-CO-SalomeNeural': 'es-CO', 'es-CR-JuanNeural': 'es-CR', 'es-CR-MariaNeural': 'es-CR', 'es-CU-BelkysNeural': 'es-CU', 'es-CU-ManuelNeural': 'es-CU', 'es-DO-EmilioNeural': 'es-DO', 'es-DO-RamonaNeural': 'es-DO', 'es-EC-AndreaNeural': 'es-EC', 'es-EC-LuisNeural': 'es-EC', 'es-ES-AlvaroNeural': 'es-ES', 'es-ES-ElviraNeural': 'es-ES', 'es-ES-ManuelEsCUNeural': 'es-ES', 'es-GQ-JavierNeural': 'es-GQ', 'es-GQ-TeresaNeural': 'es-GQ', 'es-GT-AndresNeural': 'es-GT', 'es-GT-MartaNeural': 'es-GT', 'es-HN-CarlosNeural': 'es-HN', 'es-HN-KarlaNeural': 'es-HN', 'es-MX-DaliaNeural': 'es-MX', 'es-MX-JorgeNeural': 'es-MX', 'es-MX-LorenzoEsCLNeural': 'es-MX', 'es-NI-FedericoNeural': 'es-NI', 'es-NI-YolandaNeural': 'es-NI', 'es-PA-MargaritaNeural': 'es-PA', 'es-PA-RobertoNeural': 'es-PA', 'es-PE-AlexNeural': 'es-PE', 'es-PE-CamilaNeural': 'es-PE', 'es-PR-KarinaNeural': 'es-PR', 'es-PR-VictorNeural': 'es-PR', 'es-PY-MarioNeural': 'es-PY', 'es-PY-TaniaNeural': 'es-PY', 'es-SV-LorenaNeural': 'es-SV', 'es-SV-RodrigoNeural': 'es-SV', 'es-US-AlonsoNeural': 'es-US', 'es-US-PalomaNeural': 'es-US', 'es-UY-MateoNeural': 'es-UY', 'es-UY-ValentinaNeural': 'es-UY', 'es-VE-PaolaNeural': 'es-VE', 'es-VE-SebastianNeural': 'es-VE', 'et-EE-AnuNeural': 'et-EE', 'et-EE-KertNeural': 'et-EE', 'fa-IR-DilaraNeural': 'fa-IR', 'fa-IR-FaridNeural': 'fa-IR', 'fi-FI-HarriNeural': 'fi-FI', 'fi-FI-NooraNeural': 'fi-FI', 'fil-PH-AngeloNeural': 'fil-PH', 'fil-PH-BlessicaNeural': 'fil-PH', 'fr-BE-CharlineNeural': 'fr-BE', 'fr-BE-GerardNeural': 'fr-BE', 'fr-CA-AntoineNeural': 'fr-CA', 'fr-CA-JeanNeural': 'fr-CA', 'fr-CA-SylvieNeural': 'fr-CA', 'fr-CH-ArianeNeural': 'fr-CH', 'fr-CH-FabriceNeural': 'fr-CH', 'fr-FR-DeniseNeural': 'fr-FR', 'fr-FR-EloiseNeural': 'fr-FR', 'fr-FR-HenriNeural': 'fr-FR', 'ga-IE-ColmNeural': 'ga-IE', 'ga-IE-OrlaNeural': 'ga-IE', 'gl-ES-RoiNeural': 'gl-ES', 'gl-ES-SabelaNeural': 'gl-ES', 'gu-IN-DhwaniNeural': 'gu-IN', 'gu-IN-NiranjanNeural': 'gu-IN', 'he-IL-AvriNeural': 'he-IL', 'he-IL-HilaNeural': 'he-IL', 'hi-IN-MadhurNeural': 'hi-IN', 'hi-IN-SwaraNeural': 'hi-IN', 'hr-HR-GabrijelaNeural': 'hr-HR', 'hr-HR-SreckoNeural': 'hr-HR', 'hu-HU-NoemiNeural': 'hu-HU', 'hu-HU-TamasNeural': 'hu-HU', 'id-ID-ArdiNeural': 'id-ID', 'id-ID-GadisNeural': 'id-ID', 'is-IS-GudrunNeural': 'is-IS', 'is-IS-GunnarNeural': 'is-IS', 'it-IT-DiegoNeural': 'it-IT', 'it-IT-ElsaNeural': 'it-IT', 'it-IT-IsabellaNeural': 'it-IT', 'ja-JP-KeitaNeural': 'ja-JP', 'ja-JP-NanamiNeural': 'ja-JP', 'jv-ID-DimasNeural': 'jv-ID', 'jv-ID-SitiNeural': 'jv-ID', 'ka-GE-EkaNeural': 'ka-GE', 'ka-GE-GiorgiNeural': 'ka-GE', 'kk-KZ-AigulNeural': 'kk-KZ', 'kk-KZ-DauletNeural': 'kk-KZ', 'km-KH-PisethNeural': 'km-KH', 'km-KH-SreymomNeural': 'km-KH', 'kn-IN-GaganNeural': 'kn-IN', 'kn-IN-SapnaNeural': 'kn-IN', 'ko-KR-InJoonNeural': 'ko-KR', 'ko-KR-SunHiNeural': 'ko-KR', 'lo-LA-ChanthavongNeural': 'lo-LA', 'lo-LA-KeomanyNeural': 'lo-LA', 'lt-LT-LeonasNeural': 'lt-LT', 'lt-LT-OnaNeural': 'lt-LT', 'lv-LV-EveritaNeural': 'lv-LV', 'lv-LV-NilsNeural': 'lv-LV', 'mk-MK-AleksandarNeural': 'mk-MK', 'mk-MK-MarijaNeural': 'mk-MK', 'ml-IN-MidhunNeural': 'ml-IN', 'ml-IN-SobhanaNeural': 'ml-IN', 'mn-MN-BataaNeural': 'mn-MN', 'mn-MN-YesuiNeural': 'mn-MN', 'mr-IN-AarohiNeural': 'mr-IN', 'mr-IN-ManoharNeural': 'mr-IN', 'ms-MY-OsmanNeural': 'ms-MY', 'ms-MY-YasminNeural': 'ms-MY', 'mt-MT-GraceNeural': 'mt-MT', 'mt-MT-JosephNeural': 'mt-MT', 'my-MM-NilarNeural': 'my-MM', 'my-MM-ThihaNeural': 'my-MM', 'nb-NO-FinnNeural': 'nb-NO', 'nb-NO-PernilleNeural': 'nb-NO', 'ne-NP-HemkalaNeural': 'ne-NP', 'ne-NP-SagarNeural': 'ne-NP', 'nl-BE-ArnaudNeural': 'nl-BE', 'nl-BE-DenaNeural': 'nl-BE', 'nl-NL-ColetteNeural': 'nl-NL', 'nl-NL-FennaNeural': 'nl-NL', 'nl-NL-MaartenNeural': 'nl-NL', 'pl-PL-MarekNeural': 'pl-PL', 'pl-PL-ZofiaNeural': 'pl-PL', 'ps-AF-GulNawazNeural': 'ps-AF', 'ps-AF-LatifaNeural': 'ps-AF', 'pt-BR-AntonioNeural': 'pt-BR', 'pt-BR-FranciscaNeural': 'pt-BR', 'pt-PT-DuarteNeural': 'pt-PT', 'pt-PT-RaquelNeural': 'pt-PT', 'ro-RO-AlinaNeural': 'ro-RO', 'ro-RO-EmilNeural': 'ro-RO', 'ru-RU-DmitryNeural': 'ru-RU', 'ru-RU-SvetlanaNeural': 'ru-RU', 'si-LK-SameeraNeural': 'si-LK', 'si-LK-ThiliniNeural': 'si-LK', 'sk-SK-LukasNeural': 'sk-SK', 'sk-SK-ViktoriaNeural': 'sk-SK', 'sl-SI-PetraNeural': 'sl-SI', 'sl-SI-RokNeural': 'sl-SI', 'so-SO-MuuseNeural': 'so-SO', 'so-SO-UbaxNeural': 'so-SO', 'sq-AL-AnilaNeural': 'sq-AL', 'sq-AL-IlirNeural': 'sq-AL', 'sr-RS-NicholasNeural': 'sr-RS', 'sr-RS-SophieNeural': 'sr-RS', 'su-ID-JajangNeural': 'su-ID', 'su-ID-TutiNeural': 'su-ID', 'sv-SE-MattiasNeural': 'sv-SE', 'sv-SE-SofieNeural': 'sv-SE', 'sw-KE-RafikiNeural': 'sw-KE', 'sw-KE-ZuriNeural': 'sw-KE', 'sw-TZ-DaudiNeural': 'sw-TZ', 'sw-TZ-RehemaNeural': 'sw-TZ', 'ta-IN-PallaviNeural': 'ta-IN', 'ta-IN-ValluvarNeural': 'ta-IN', 'ta-LK-KumarNeural': 'ta-LK', 'ta-LK-SaranyaNeural': 'ta-LK', 'ta-MY-KaniNeural': 'ta-MY', 'ta-MY-SuryaNeural': 'ta-MY', 'ta-SG-AnbuNeural': 'ta-SG', 'ta-SG-VenbaNeural': 'ta-SG', 'te-IN-MohanNeural': 'te-IN', 'te-IN-ShrutiNeural': 'te-IN', 'th-TH-NiwatNeural': 'th-TH', 'th-TH-PremwadeeNeural': 'th-TH', 'tr-TR-AhmetNeural': 'tr-TR', 'tr-TR-EmelNeural': 'tr-TR', 'uk-UA-OstapNeural': 'uk-UA', 'uk-UA-PolinaNeural': 'uk-UA', 'ur-IN-GulNeural': 'ur-IN', 'ur-IN-SalmanNeural': 'ur-IN', 'ur-PK-AsadNeural': 'ur-PK', 'ur-PK-UzmaNeural': 'ur-PK', 'uz-UZ-MadinaNeural': 'uz-UZ', 'uz-UZ-SardorNeural': 'uz-UZ', 'vi-VN-HoaiMyNeural': 'vi-VN', 'vi-VN-NamMinhNeural': 'vi-VN', 'zu-ZA-ThandoNeural': 'zu-ZA', 'zu-ZA-ThembaNeural': 'zu-ZA', } SUPPORTED_LANGUAGES = [ "Auto", *SUPPORTED_VOICES.keys() ]so-vits-svc-npu/export_index_for_onnx.py000066400000000000000000000006331456161405400210550ustar00rootroot00000000000000import os import pickle import faiss path = "crs" indexs_file_path = f"checkpoints/{path}/feature_and_index.pkl" indexs_out_dir = f"checkpoints/{path}/" with open("feature_and_index.pkl",mode="rb") as f: indexs = pickle.load(f) for k in indexs: print(f"Save {k} index") faiss.write_index( indexs[k], os.path.join(indexs_out_dir,f"Index-{k}.index") ) print("Saved all index")so-vits-svc-npu/filelists/000077500000000000000000000000001456161405400160575ustar00rootroot00000000000000so-vits-svc-npu/filelists/test.txt000066400000000000000000000001741456161405400176010ustar00rootroot00000000000000./dataset/44k/taffy/000562.wav ./dataset/44k/nyaru/000011.wav ./dataset/44k/nyaru/000008.wav ./dataset/44k/taffy/000563.wav so-vits-svc-npu/filelists/train.txt000066400000000000000000000012201456161405400177300ustar00rootroot00000000000000./dataset/44k/TEST/354440190 (9).wav ./dataset/44k/TEST/354440190 (6).wav ./dataset/44k/TEST/354440190 (1).wav ./dataset/44k/TEST/354440190 (11).wav ./dataset/44k/TEST/354440190 (10).wav ./dataset/44k/TEST/354440190 (15).wav ./dataset/44k/TEST/354440190 (12).wav ./dataset/44k/TEST/354440190 (2).wav ./dataset/44k/TEST/354440190 (7).wav ./dataset/44k/TEST/vo_tips_TCG001_4_littlePrince_01.wav ./dataset/44k/TEST/vo_tips_TCG001_37_littlePrince_01.wav ./dataset/44k/TEST/354440190 (5).wav ./dataset/44k/TEST/vo_tips_TCG001_4_littlePrince_02.wav ./dataset/44k/TEST/354440190 (4).wav ./dataset/44k/TEST/354440190 (14).wav ./dataset/44k/TEST/354440190 (13).wav so-vits-svc-npu/filelists/val.txt000066400000000000000000000001121456161405400173740ustar00rootroot00000000000000./dataset/44k/TEST/354440190 (3).wav ./dataset/44k/TEST/354440190 (8).wav so-vits-svc-npu/flask_api.py000066400000000000000000000050521456161405400163660ustar00rootroot00000000000000import io import logging import soundfile import mindtorch.torch as torch import mindtorch.torchaudio as torch from flask import Flask, request, send_file from flask_cors import CORS from inference.infer_tool import RealTimeVC, Svc app = Flask(__name__) CORS(app) logging.getLogger('numba').setLevel(logging.WARNING) @app.route("/voiceChangeModel", methods=["POST"]) def voice_change_model(): request_form = request.form wave_file = request.files.get("sample", None) # 变调信息 f_pitch_change = float(request_form.get("fPitchChange", 0)) # DAW所需的采样率 daw_sample = int(float(request_form.get("sampleRate", 0))) speaker_id = int(float(request_form.get("sSpeakId", 0))) # http获得wav文件并转换 input_wav_path = io.BytesIO(wave_file.read()) # 模型推理 if raw_infer: # out_audio, out_sr = svc_model.infer(speaker_id, f_pitch_change, input_wav_path) out_audio, out_sr = svc_model.infer(speaker_id, f_pitch_change, input_wav_path, cluster_infer_ratio=0, auto_predict_f0=False, noice_scale=0.4, f0_filter=False) tar_audio = torchaudio.functional.resample(out_audio, svc_model.target_sample, daw_sample) else: out_audio = svc.process(svc_model, speaker_id, f_pitch_change, input_wav_path, cluster_infer_ratio=0, auto_predict_f0=False, noice_scale=0.4, f0_filter=False) tar_audio = torchaudio.functional.resample(torch.from_numpy(out_audio), svc_model.target_sample, daw_sample) # 返回音频 out_wav_path = io.BytesIO() soundfile.write(out_wav_path, tar_audio.cpu().numpy(), daw_sample, format="wav") out_wav_path.seek(0) return send_file(out_wav_path, download_name="temp.wav", as_attachment=True) if __name__ == '__main__': # 启用则为直接切片合成，False为交叉淡化方式 # vst插件调整0.3-0.5s切片时间可以降低延迟，直接切片方法会有连接处爆音、交叉淡化会有轻微重叠声音 # 自行选择能接受的方法，或将vst最大切片时间调整为1s，此处设为Ture，延迟大音质稳定一些 raw_infer = True # 每个模型和config是唯一对应的 model_name = "logs/32k/G_174000-Copy1.pth" config_name = "configs/config.json" cluster_model_path = "logs/44k/kmeans_10000.pt" svc_model = Svc(model_name, config_name, cluster_model_path=cluster_model_path) svc = RealTimeVC() # 此处与vst插件对应，不建议更改 app.run(port=6842, host="0.0.0.0", debug=False, threaded=False) so-vits-svc-npu/flask_api_full_song.py000066400000000000000000000041241456161405400204350ustar00rootroot00000000000000import io import numpy as np import soundfile from flask import Flask, request, send_file from inference import infer_tool, slicer app = Flask(__name__) @app.route("/wav2wav", methods=["POST"]) def wav2wav(): request_form = request.form audio_path = request_form.get("audio_path", None) # wav文件地址 tran = int(float(request_form.get("tran", 0))) # 音调 spk = request_form.get("spk", 0) # 说话人(id或者name都可以,具体看你的config) wav_format = request_form.get("wav_format", 'wav') # 范围文件格式 infer_tool.format_wav(audio_path) chunks = slicer.cut(audio_path, db_thresh=-40) audio_data, audio_sr = slicer.chunks2audio(audio_path, chunks) audio = [] for (slice_tag, data) in audio_data: print(f'#=====segment start, {round(len(data) / audio_sr, 3)}s======') length = int(np.ceil(len(data) / audio_sr * svc_model.target_sample)) if slice_tag: print('jump empty segment') _audio = np.zeros(length) else: # padd pad_len = int(audio_sr * 0.5) data = np.concatenate([np.zeros([pad_len]), data, np.zeros([pad_len])]) raw_path = io.BytesIO() soundfile.write(raw_path, data, audio_sr, format="wav") raw_path.seek(0) out_audio, out_sr = svc_model.infer(spk, tran, raw_path) svc_model.clear_empty() _audio = out_audio.cpu().numpy() pad_len = int(svc_model.target_sample * 0.5) _audio = _audio[pad_len:-pad_len] audio.extend(list(infer_tool.pad_array(_audio, length))) out_wav_path = io.BytesIO() soundfile.write(out_wav_path, audio, svc_model.target_sample, format=wav_format) out_wav_path.seek(0) return send_file(out_wav_path, download_name=f"temp.{wav_format}", as_attachment=True) if __name__ == '__main__': model_name = "logs/44k/G_60000.pth" # 模型地址 config_name = "configs/config.json" # config地址 svc_model = infer_tool.Svc(model_name, config_name) app.run(port=1145, host="0.0.0.0", debug=False, threaded=False) so-vits-svc-npu/inference/000077500000000000000000000000001456161405400160175ustar00rootroot00000000000000so-vits-svc-npu/inference/__init__.py000066400000000000000000000000001456161405400201160ustar00rootroot00000000000000so-vits-svc-npu/inference/infer_tool.py000066400000000000000000000616311456161405400205400ustar00rootroot00000000000000import gc import hashlib import io import json import logging import os import pickle import time from pathlib import Path import librosa import numpy as np # import onnxruntime import soundfile import mindtorch.torch as torch import mindtorch.torchaudio as torchaudio import cluster import utils from diffusion.unit2mel import load_model_vocoder from inference import slicer from models import SynthesizerTrn import logger logging.getLogger('matplotlib').setLevel(logging.WARNING) def read_temp(file_name): if not os.path.exists(file_name): with open(file_name, "w") as f: f.write(json.dumps({"info": "temp_dict"})) return {} else: try: with open(file_name, "r") as f: data = f.read() data_dict = json.loads(data) if os.path.getsize(file_name) > 50 * 1024 * 1024: f_name = file_name.replace("\\", "/").split("/")[-1] print(f"clean {f_name}") for wav_hash in list(data_dict.keys()): if int(time.time()) - int(data_dict[wav_hash]["time"]) > 14 * 24 * 3600: del data_dict[wav_hash] except Exception as e: print(e) print(f"{file_name} error,auto rebuild file") data_dict = {"info": "temp_dict"} return data_dict def write_temp(file_name, data): with open(file_name, "w") as f: f.write(json.dumps(data)) def timeit(func): def run(*args, **kwargs): t = time.time() res = func(*args, **kwargs) print('executing \'%s\' costed %.3fs' % (func.__name__, time.time() - t)) return res return run def format_wav(audio_path): if Path(audio_path).suffix == '.wav': return raw_audio, raw_sample_rate = librosa.load(audio_path, mono=True, sr=None) soundfile.write(Path(audio_path).with_suffix(".wav"), raw_audio, raw_sample_rate) def get_end_file(dir_path, end): file_lists = [] for root, dirs, files in os.walk(dir_path): files = [f for f in files if f[0] != '.'] dirs[:] = [d for d in dirs if d[0] != '.'] for f_file in files: if f_file.endswith(end): file_lists.append(os.path.join(root, f_file).replace("\\", "/")) return file_lists def get_md5(content): return hashlib.new("md5", content).hexdigest() def fill_a_to_b(a, b): if len(a) < len(b): for _ in range(0, len(b) - len(a)): a.append(a[0]) def mkdir(paths: list): for path in paths: if not os.path.exists(path): os.mkdir(path) def pad_array(arr, target_length): current_length = arr.shape[0] if current_length >= target_length: return arr else: pad_width = target_length - current_length pad_left = pad_width // 2 pad_right = pad_width - pad_left padded_arr = np.pad(arr, (pad_left, pad_right), 'constant', constant_values=(0, 0)) return padded_arr def split_list_by_n(list_collection, n, pre=0): for i in range(0, len(list_collection), n): yield list_collection[i-pre if i-pre>=0 else i: i + n] class F0FilterException(Exception): pass class Svc(object): def __init__(self, net_g_path, config_path, device=None, cluster_model_path="logs/44k/kmeans_10000.pt", nsf_hifigan_enhance = False, diffusion_model_path="logs/44k/diffusion/model_0.pt", diffusion_config_path="configs/diffusion.yaml", shallow_diffusion = False, only_diffusion = False, spk_mix_enable = False, feature_retrieval = False ): self.net_g_path = net_g_path self.only_diffusion = only_diffusion self.shallow_diffusion = shallow_diffusion self.feature_retrieval = feature_retrieval if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.net_g_ms = None if not self.only_diffusion: self.hps_ms = utils.get_hparams_from_file(config_path,True) self.target_sample = self.hps_ms.data.sampling_rate self.hop_size = self.hps_ms.data.hop_length self.spk2id = self.hps_ms.spk self.unit_interpolate_mode = self.hps_ms.data.unit_interpolate_mode if self.hps_ms.data.unit_interpolate_mode is not None else 'left' self.vol_embedding = self.hps_ms.model.vol_embedding if self.hps_ms.model.vol_embedding is not None else False self.speech_encoder = self.hps_ms.model.speech_encoder if self.hps_ms.model.speech_encoder is not None else 'vec768l12' self.nsf_hifigan_enhance = nsf_hifigan_enhance if self.shallow_diffusion or self.only_diffusion: if os.path.exists(diffusion_model_path) and os.path.exists(diffusion_model_path): self.diffusion_model,self.vocoder,self.diffusion_args = load_model_vocoder(diffusion_model_path,self.dev,config_path=diffusion_config_path) if self.only_diffusion: self.target_sample = self.diffusion_args.data.sampling_rate self.hop_size = self.diffusion_args.data.block_size self.spk2id = self.diffusion_args.spk self.dtype = torch.float32 self.speech_encoder = self.diffusion_args.data.encoder self.unit_interpolate_mode = self.diffusion_args.data.unit_interpolate_mode if self.diffusion_args.data.unit_interpolate_mode is not None else 'left' if spk_mix_enable: self.diffusion_model.init_spkmix(len(self.spk2id)) else: print("No diffusion model or config found. Shallow diffusion mode will False") self.shallow_diffusion = self.only_diffusion = False # load hubert and model if not self.only_diffusion: self.load_model(spk_mix_enable) self.hubert_model = utils.get_speech_encoder(self.speech_encoder,device=self.dev) self.volume_extractor = utils.Volume_Extractor(self.hop_size) else: self.hubert_model = utils.get_speech_encoder(self.diffusion_args.data.encoder,device=self.dev) self.volume_extractor = utils.Volume_Extractor(self.diffusion_args.data.block_size) if os.path.exists(cluster_model_path): if self.feature_retrieval: with open(cluster_model_path,"rb") as f: self.cluster_model = pickle.load(f) self.big_npy = None self.now_spk_id = -1 else: self.cluster_model = cluster.get_cluster_model(cluster_model_path) else: self.feature_retrieval=False if self.shallow_diffusion : self.nsf_hifigan_enhance = False if self.nsf_hifigan_enhance: from modules.enhancer import Enhancer self.enhancer = Enhancer('nsf-hifigan', 'pretrain/nsf_hifigan/model',device=self.dev) def load_model(self, spk_mix_enable=False): # get model configuration self.net_g_ms = SynthesizerTrn( self.hps_ms.data.filter_length // 2 + 1, self.hps_ms.train.segment_size // self.hps_ms.data.hop_length, **self.hps_ms.model) _ = utils.load_checkpoint(self.net_g_path, self.net_g_ms, None) self.dtype = list(self.net_g_ms.parameters())[0].dtype if "half" in self.net_g_path and torch.cuda.is_available(): _ = self.net_g_ms.half().eval().to(self.dev) else: _ = self.net_g_ms.eval().to(self.dev) if spk_mix_enable: self.net_g_ms.EnableCharacterMix(len(self.spk2id), self.dev) def get_unit_f0(self, wav, tran, cluster_infer_ratio, speaker, f0_filter ,f0_predictor,cr_threshold=0.05): if not hasattr(self,"f0_predictor_object") or self.f0_predictor_object is None or f0_predictor != self.f0_predictor_object.name: self.f0_predictor_object = utils.get_f0_predictor(f0_predictor,hop_length=self.hop_size,sampling_rate=self.target_sample,device=self.dev,threshold=cr_threshold) f0, uv = self.f0_predictor_object.compute_f0_uv(wav) if f0_filter and sum(f0) == 0: raise F0FilterException("No voice detected") f0 = torch.FloatTensor(f0).to(self.dev) uv = torch.FloatTensor(uv).to(self.dev) f0 = f0 * 2 ** (tran / 12) f0 = f0.unsqueeze(0) uv = uv.unsqueeze(0) wav = torch.from_numpy(wav).to(self.dev) if not hasattr(self,"audio16k_resample_transform"): self.audio16k_resample_transform = torchaudio.transforms.Resample(self.target_sample, 16000).to(self.dev) wav16k = self.audio16k_resample_transform(wav[None,:])[0] c = self.hubert_model.encoder(wav16k) c = utils.repeat_expand_2d(c.squeeze(0), f0.shape[1],self.unit_interpolate_mode) if cluster_infer_ratio !=0: if self.feature_retrieval: speaker_id = self.spk2id.get(speaker) if not speaker_id and type(speaker) is int: if len(self.spk2id.__dict__) >= speaker: speaker_id = speaker if speaker_id is None: raise RuntimeError("The name you entered is not in the speaker list!") feature_index = self.cluster_model[speaker_id] feat_np = np.ascontiguousarray(c.transpose(0,1).cpu().numpy()) if self.big_npy is None or self.now_spk_id != speaker_id: self.big_npy = feature_index.reconstruct_n(0, feature_index.ntotal) self.now_spk_id = speaker_id print("starting feature retrieval...") score, ix = feature_index.search(feat_np, k=8) weight = np.square(1 / score) weight /= weight.sum(axis=1, keepdims=True) npy = np.sum(self.big_npy[ix] * np.expand_dims(weight, axis=2), axis=1) c = cluster_infer_ratio * npy + (1 - cluster_infer_ratio) * feat_np c = torch.FloatTensor(c).to(self.dev).transpose(0,1) print("end feature retrieval...") else: cluster_c = cluster.get_cluster_center_result(self.cluster_model, c.cpu().numpy().T, speaker).T cluster_c = torch.FloatTensor(cluster_c).to(self.dev) c = cluster_infer_ratio * cluster_c + (1 - cluster_infer_ratio) * c c = c.unsqueeze(0) return c, f0, uv def infer(self, speaker, tran, raw_path, cluster_infer_ratio=0, auto_predict_f0=False, noice_scale=0.4, f0_filter=False, f0_predictor='pm', enhancer_adaptive_key = 0, cr_threshold = 0.05, k_step = 100, frame = 0, spk_mix = False, second_encoding = False, loudness_envelope_adjustment = 1 ): torchaudio.set_audio_backend("soundfile") wav, sr = torchaudio.load(raw_path) if not hasattr(self,"audio_resample_transform") or self.audio16k_resample_transform.orig_freq != sr: self.audio_resample_transform = torchaudio.transforms.Resample(sr,self.target_sample) wav = self.audio_resample_transform(wav).numpy()[0] if spk_mix: c, f0, uv = self.get_unit_f0(wav, tran, 0, None, f0_filter,f0_predictor,cr_threshold=cr_threshold) n_frames = f0.size(1) sid = speaker[:, frame:frame+n_frames].transpose(0,1) else: speaker_id = self.spk2id.get(speaker) if not speaker_id and type(speaker) is int: if len(self.spk2id.__dict__) >= speaker: speaker_id = speaker if speaker_id is None: raise RuntimeError("The name you entered is not in the speaker list!") sid = torch.LongTensor([int(speaker_id)]).to(self.dev).unsqueeze(0) c, f0, uv = self.get_unit_f0(wav, tran, cluster_infer_ratio, speaker, f0_filter,f0_predictor,cr_threshold=cr_threshold) n_frames = f0.size(1) c = c.to(self.dtype) f0 = f0.to(self.dtype) uv = uv.to(self.dtype) with torch.no_grad(): start = time.time() vol = None if not self.only_diffusion: vol = self.volume_extractor.extract(torch.FloatTensor(wav).to(self.dev)[None,:])[None,:].to(self.dev) if self.vol_embedding else None audio,f0 = self.net_g_ms.infer(c, f0=f0, g=sid, uv=uv, predict_f0=auto_predict_f0, noice_scale=noice_scale,vol=vol) audio = audio[0,0].data.float() audio_mel = self.vocoder.extract(audio[None,:],self.target_sample) if self.shallow_diffusion else None else: audio = torch.FloatTensor(wav).to(self.dev) audio_mel = None if self.dtype != torch.float32: c = c.to(torch.float32) f0 = f0.to(torch.float32) uv = uv.to(torch.float32) if self.only_diffusion or self.shallow_diffusion: vol = self.volume_extractor.extract(audio[None,:])[None,:,None].to(self.dev) if vol is None else vol[:,:,None] if self.shallow_diffusion and second_encoding: if not hasattr(self,"audio16k_resample_transform"): self.audio16k_resample_transform = torchaudio.transforms.Resample(self.target_sample, 16000).to(self.dev) audio16k = self.audio16k_resample_transform(audio[None,:])[0] c = self.hubert_model.encoder(audio16k) c = utils.repeat_expand_2d(c.squeeze(0), f0.shape[1],self.unit_interpolate_mode) f0 = f0[:,:,None] c = c.transpose(-1,-2) audio_mel = self.diffusion_model( c, f0, vol, spk_id = sid, spk_mix_dict = None, gt_spec=audio_mel, infer=True, infer_speedup=self.diffusion_args.infer.speedup, method=self.diffusion_args.infer.method, k_step=k_step) audio = self.vocoder.infer(audio_mel, f0).squeeze() if self.nsf_hifigan_enhance: audio, _ = self.enhancer.enhance( audio[None,:], self.target_sample, f0[:,:,None], self.hps_ms.data.hop_length, adaptive_key = enhancer_adaptive_key) if loudness_envelope_adjustment != 1: audio = utils.change_rms(wav,self.target_sample,audio,self.target_sample,loudness_envelope_adjustment) use_time = time.time() - start print("vits use time:{}".format(use_time)) return audio, audio.shape[-1], n_frames def clear_empty(self): # clean up vram torch.cuda.empty_cache() def unload_model(self): # unload model self.net_g_ms = self.net_g_ms.to("cpu") del self.net_g_ms if hasattr(self,"enhancer"): self.enhancer.enhancer = self.enhancer.enhancer.to("cpu") del self.enhancer.enhancer del self.enhancer gc.collect() def slice_inference(self, raw_audio_path, spk, tran, slice_db, cluster_infer_ratio, auto_predict_f0, noice_scale, pad_seconds=0.5, clip_seconds=0, lg_num=0, lgr_num =0.75, f0_predictor='pm', enhancer_adaptive_key = 0, cr_threshold = 0.05, k_step = 100, use_spk_mix = False, second_encoding = False, loudness_envelope_adjustment = 1 ): if use_spk_mix: if len(self.spk2id) == 1: spk = self.spk2id.keys()[0] use_spk_mix = False wav_path = Path(raw_audio_path).with_suffix('.wav') chunks = slicer.cut(wav_path, db_thresh=slice_db) audio_data, audio_sr = slicer.chunks2audio(wav_path, chunks) per_size = int(clip_seconds*audio_sr) lg_size = int(lg_num*audio_sr) lg_size_r = int(lg_size*lgr_num) lg_size_c_l = (lg_size-lg_size_r)//2 lg_size_c_r = lg_size-lg_size_r-lg_size_c_l lg = np.linspace(0,1,lg_size_r) if lg_size!=0 else 0 if use_spk_mix: assert len(self.spk2id) == len(spk) audio_length = 0 for (slice_tag, data) in audio_data: aud_length = int(np.ceil(len(data) / audio_sr * self.target_sample)) if slice_tag: audio_length += aud_length // self.hop_size continue if per_size != 0: datas = split_list_by_n(data, per_size,lg_size) else: datas = [data] for k,dat in enumerate(datas): pad_len = int(audio_sr * pad_seconds) per_length = int(np.ceil(len(dat) / audio_sr * self.target_sample)) a_length = per_length + 2 * pad_len audio_length += a_length // self.hop_size audio_length += len(audio_data) spk_mix_tensor = torch.zeros(size=(len(spk), audio_length)).to(self.dev) for i in range(len(spk)): last_end = None for mix in spk[i]: if mix[3]<0. or mix[2]<0.: raise RuntimeError("mix value must higer Than zero!") begin = int(audio_length * mix[0]) end = int(audio_length * mix[1]) length = end - begin if length<=0: raise RuntimeError("begin Must lower Than end!") step = (mix[3] - mix[2])/length if last_end is not None: if last_end != begin: raise RuntimeError("[i]EndTime Must Equal [i+1]BeginTime!") last_end = end if step == 0.: spk_mix_data = torch.zeros(length).to(self.dev) + mix[2] else: spk_mix_data = torch.arange(mix[2],mix[3],step).to(self.dev) if(len(spk_mix_data)0 or p_len - len(f0) - pad_size>0): f0 = np.pad(f0,[[pad_size,p_len - len(f0) - pad_size]], mode='constant') f0 *= pow(2, f0_up_key / 12) f0_mel = 1127 * np.log(1 + f0 / 700) f0_mel[f0_mel > 0] = (f0_mel[f0_mel > 0] - f0_mel_min) * 254 / (f0_mel_max - f0_mel_min) + 1 f0_mel[f0_mel <= 1] = 1 f0_mel[f0_mel > 255] = 255 f0_coarse = np.rint(f0_mel).astype(np.int) return f0_coarse, f0 def clean_pitch(input_pitch): num_nan = np.sum(input_pitch == 1) if num_nan / len(input_pitch) > 0.9: input_pitch[input_pitch != 1] = 1 return input_pitch def plt_pitch(input_pitch): input_pitch = input_pitch.astype(float) input_pitch[input_pitch == 1] = np.nan return input_pitch def f0_to_pitch(ff): f0_pitch = 69 + 12 * np.log2(ff / 440) return f0_pitch def fill_a_to_b(a, b): if len(a) < len(b): for _ in range(0, len(b) - len(a)): a.append(a[0]) def mkdir(paths: list): for path in paths: if not os.path.exists(path): os.mkdir(path) class VitsSvc(object): def __init__(self): self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") self.SVCVITS = None self.hps = None self.speakers = None self.hubert_soft = utils.get_hubert_model() def set_device(self, device): self.device = torch.device(device) self.hubert_soft.to(self.device) if self.SVCVITS is not None: self.SVCVITS.to(self.device) def loadCheckpoint(self, path): self.hps = utils.get_hparams_from_file(f"checkpoints/{path}/config.json") self.SVCVITS = SynthesizerTrn( self.hps.data.filter_length // 2 + 1, self.hps.train.segment_size // self.hps.data.hop_length, **self.hps.model) _ = utils.load_checkpoint(f"checkpoints/{path}/model.pth", self.SVCVITS, None) _ = self.SVCVITS.eval().to(self.device) self.speakers = self.hps.spk def get_units(self, source, sr): source = source.unsqueeze(0).to(self.device) with torch.inference_mode(): units = self.hubert_soft.units(source) return units def get_unit_pitch(self, in_path, tran): source, sr = torchaudio.load(in_path) source = torchaudio.functional.resample(source, sr, 16000) if len(source.shape) == 2 and source.shape[1] >= 2: source = torch.mean(source, dim=0).unsqueeze(0) soft = self.get_units(source, sr).squeeze(0).cpu().numpy() f0_coarse, f0 = get_f0(source.cpu().numpy()[0], soft.shape[0]*2, tran) return soft, f0 def infer(self, speaker_id, tran, raw_path): speaker_id = self.speakers[speaker_id] sid = torch.LongTensor([int(speaker_id)]).to(self.device).unsqueeze(0) soft, pitch = self.get_unit_pitch(raw_path, tran) f0 = torch.FloatTensor(clean_pitch(pitch)).unsqueeze(0).to(self.device) stn_tst = torch.FloatTensor(soft) with torch.no_grad(): x_tst = stn_tst.unsqueeze(0).to(self.device) x_tst = torch.repeat_interleave(x_tst, repeats=2, dim=1).transpose(1, 2) audio,_ = self.SVCVITS.infer(x_tst, f0=f0, g=sid)[0,0].data.float() return audio, audio.shape[-1] def inference(self,srcaudio,chara,tran,slice_db): sampling_rate, audio = srcaudio audio = (audio / np.iinfo(audio.dtype).max).astype(np.float32) if len(audio.shape) > 1: audio = librosa.to_mono(audio.transpose(1, 0)) if sampling_rate != 16000: audio = librosa.resample(audio, orig_sr=sampling_rate, target_sr=16000) soundfile.write("tmpwav.wav", audio, 16000, format="wav") chunks = slicer.cut("tmpwav.wav", db_thresh=slice_db) audio_data, audio_sr = slicer.chunks2audio("tmpwav.wav", chunks) audio = [] for (slice_tag, data) in audio_data: length = int(np.ceil(len(data) / audio_sr * self.hps.data.sampling_rate)) raw_path = io.BytesIO() soundfile.write(raw_path, data, audio_sr, format="wav") raw_path.seek(0) if slice_tag: _audio = np.zeros(length) else: out_audio, out_sr = self.infer(chara, tran, raw_path) _audio = out_audio.cpu().numpy() audio.extend(list(_audio)) audio = (np.array(audio) * 32768.0).astype('int16') return (self.hps.data.sampling_rate,audio) so-vits-svc-npu/inference/slicer.py000066400000000000000000000151331456161405400176550ustar00rootroot00000000000000import librosa import mindtorch.torch as torch import mindtorch.torchaudio as torchaudio class Slicer: def __init__(self, sr: int, threshold: float = -40., min_length: int = 5000, min_interval: int = 300, hop_size: int = 20, max_sil_kept: int = 5000): if not min_length >= min_interval >= hop_size: raise ValueError('The following condition must be satisfied: min_length >= min_interval >= hop_size') if not max_sil_kept >= hop_size: raise ValueError('The following condition must be satisfied: max_sil_kept >= hop_size') min_interval = sr * min_interval / 1000 self.threshold = 10 ** (threshold / 20.) self.hop_size = round(sr * hop_size / 1000) self.win_size = min(round(min_interval), 4 * self.hop_size) self.min_length = round(sr * min_length / 1000 / self.hop_size) self.min_interval = round(min_interval / self.hop_size) self.max_sil_kept = round(sr * max_sil_kept / 1000 / self.hop_size) def _apply_slice(self, waveform, begin, end): if len(waveform.shape) > 1: return waveform[:, begin * self.hop_size: min(waveform.shape[1], end * self.hop_size)] else: return waveform[begin * self.hop_size: min(waveform.shape[0], end * self.hop_size)] # @timeit def slice(self, waveform): if len(waveform.shape) > 1: samples = librosa.to_mono(waveform) else: samples = waveform if samples.shape[0] <= self.min_length: return {"0": {"slice": False, "split_time": f"0,{len(waveform)}"}} rms_list = librosa.feature.rms(y=samples, frame_length=self.win_size, hop_length=self.hop_size).squeeze(0) sil_tags = [] silence_start = None clip_start = 0 for i, rms in enumerate(rms_list): # Keep looping while frame is silent. if rms < self.threshold: # Record start of silent frames. if silence_start is None: silence_start = i continue # Keep looping while frame is not silent and silence start has not been recorded. if silence_start is None: continue # Clear recorded silence start if interval is not enough or clip is too short is_leading_silence = silence_start == 0 and i > self.max_sil_kept need_slice_middle = i - silence_start >= self.min_interval and i - clip_start >= self.min_length if not is_leading_silence and not need_slice_middle: silence_start = None continue # Need slicing. Record the range of silent frames to be removed. if i - silence_start <= self.max_sil_kept: pos = rms_list[silence_start: i + 1].argmin() + silence_start if silence_start == 0: sil_tags.append((0, pos)) else: sil_tags.append((pos, pos)) clip_start = pos elif i - silence_start <= self.max_sil_kept * 2: pos = rms_list[i - self.max_sil_kept: silence_start + self.max_sil_kept + 1].argmin() pos += i - self.max_sil_kept pos_l = rms_list[silence_start: silence_start + self.max_sil_kept + 1].argmin() + silence_start pos_r = rms_list[i - self.max_sil_kept: i + 1].argmin() + i - self.max_sil_kept if silence_start == 0: sil_tags.append((0, pos_r)) clip_start = pos_r else: sil_tags.append((min(pos_l, pos), max(pos_r, pos))) clip_start = max(pos_r, pos) else: pos_l = rms_list[silence_start: silence_start + self.max_sil_kept + 1].argmin() + silence_start pos_r = rms_list[i - self.max_sil_kept: i + 1].argmin() + i - self.max_sil_kept if silence_start == 0: sil_tags.append((0, pos_r)) else: sil_tags.append((pos_l, pos_r)) clip_start = pos_r silence_start = None # Deal with trailing silence. total_frames = rms_list.shape[0] if silence_start is not None and total_frames - silence_start >= self.min_interval: silence_end = min(total_frames, silence_start + self.max_sil_kept) pos = rms_list[silence_start: silence_end + 1].argmin() + silence_start sil_tags.append((pos, total_frames + 1)) # Apply and return slices. if len(sil_tags) == 0: return {"0": {"slice": False, "split_time": f"0,{len(waveform)}"}} else: chunks = [] # 第一段静音并非从头开始，补上有声片段 if sil_tags[0][0]: chunks.append( {"slice": False, "split_time": f"0,{min(waveform.shape[0], sil_tags[0][0] * self.hop_size)}"}) for i in range(0, len(sil_tags)): # 标识有声片段（跳过第一段） if i: chunks.append({"slice": False, "split_time": f"{sil_tags[i - 1][1] * self.hop_size},{min(waveform.shape[0], sil_tags[i][0] * self.hop_size)}"}) # 标识所有静音片段 chunks.append({"slice": True, "split_time": f"{sil_tags[i][0] * self.hop_size},{min(waveform.shape[0], sil_tags[i][1] * self.hop_size)}"}) # 最后一段静音并非结尾，补上结尾片段 if sil_tags[-1][1] * self.hop_size < len(waveform): chunks.append({"slice": False, "split_time": f"{sil_tags[-1][1] * self.hop_size},{len(waveform)}"}) chunk_dict = {} for i in range(len(chunks)): chunk_dict[str(i)] = chunks[i] return chunk_dict def cut(audio_path, db_thresh=-30, min_len=5000): audio, sr = librosa.load(audio_path, sr=None) slicer = Slicer( sr=sr, threshold=db_thresh, min_length=min_len ) chunks = slicer.slice(audio) return chunks def chunks2audio(audio_path, chunks): chunks = dict(chunks) audio, sr = torchaudio.load(audio_path) if len(audio.shape) == 2 and audio.shape[1] >= 2: audio = torch.mean(audio, dim=0).unsqueeze(0) audio = audio.cpu().numpy()[0] result = [] for k, v in chunks.items(): tag = v["split_time"].split(",") if tag[0] != tag[1]: result.append((v["slice"], audio[int(tag[0]):int(tag[1])])) return result, sr so-vits-svc-npu/inference_main.py000066400000000000000000000236561456161405400174110ustar00rootroot00000000000000import logging # from loguru import logger import logger import os import soundfile from inference import infer_tool from inference.infer_tool import Svc from spkmix import spk_mix_map logging.getLogger('numba').setLevel(logging.WARNING) chunks_dict = infer_tool.read_temp("inference/chunks_temp.json") def isWavFile(file_name): return file_name.endswith('.wav') def getLastestCheckpoint(path): files = [ f for f in os.listdir(path) if f.endswith('.pth') and f.startswith('G_')] if len(files) == 0: logger.error(f"no checkpoint in {path}") return None latest_file = max(files) return os.path.join(path, latest_file) def main(): import argparse parser = argparse.ArgumentParser(description='sovits4 inference') # 一定要设置的部分 parser.add_argument('-m', '--model_path', type=str, default="logs/44k/{lastest}", help='模型路径') parser.add_argument('-c', '--config_path', type=str, default="logs/44k/config.json", help='配置文件路径') parser.add_argument('-t', '--trans', type=int, nargs='+', default=[0], help='音高调整，支持正负（半音）') parser.add_argument('-s', '--spk_list', type=str, nargs='+', default=['buyizi'], help='合成目标说话人名称') # 可选项部分 parser.add_argument('-n', '--clean_names', type=str, nargs='+', default=[f for f in os.listdir("raw") if isWavFile(f)], help='wav文件名列表，放在raw文件夹下且后缀为.wav') parser.add_argument('-cl', '--clip', type=float, default=0, help='音频强制切片，默认0为自动切片，单位为秒/s') parser.add_argument('-a', '--auto_predict_f0', action='store_true', default=False, help='语音转换自动预测音高，转换歌声时不要打开这个会严重跑调') parser.add_argument('-cm', '--cluster_model_path', type=str, default="", help='聚类模型或特征检索索引路径，留空则自动设为各方案模型的默认路径，如果没有训练聚类或特征检索则随便填') parser.add_argument('-cr', '--cluster_infer_ratio', type=float, default=0, help='聚类方案或特征检索占比，范围0-1，若没有训练聚类模型或特征检索则默认0即可') parser.add_argument('-lg', '--linear_gradient', type=float, default=0, help='两段音频切片的交叉淡入长度，如果强制切片后出现人声不连贯可调整该数值，如果连贯建议采用默认值0，单位为秒') parser.add_argument('-f0p', '--f0_predictor', type=str, default="pm", help='选择F0预测器,可选择crepe,pm,dio,harvest,rmvpe,fcpe默认为pm(注意：crepe为原F0使用均值滤波器)') parser.add_argument('-eh', '--enhance', action='store_true', default=False, help='是否使用NSF_HIFIGAN增强器,该选项对部分训练集少的模型有一定的音质增强效果，但是对训练好的模型有反面效果，默认关闭') parser.add_argument('-shd', '--shallow_diffusion', action='store_true', default=False, help='是否使用浅层扩散，使用后可解决一部分电音问题，默认关闭，该选项打开时，NSF_HIFIGAN增强器将会被禁止') parser.add_argument('-usm', '--use_spk_mix', action='store_true', default=False, help='是否使用角色融合') parser.add_argument('-lea', '--loudness_envelope_adjustment', type=float, default=1, help='输入源响度包络替换输出响度包络融合比例，越靠近1越使用输出响度包络') parser.add_argument('-fr', '--feature_retrieval', action='store_true', default=False, help='是否使用特征检索，如果使用聚类模型将被禁用，且cm与cr参数将会变成特征检索的索引路径与混合比例') # 浅扩散设置 parser.add_argument('-dm', '--diffusion_model_path', type=str, default="logs/44k/diffusion/model_0.pt", help='扩散模型路径') parser.add_argument('-dc', '--diffusion_config_path', type=str, default="logs/44k/diffusion/config.yaml", help='扩散模型配置文件路径') parser.add_argument('-ks', '--k_step', type=int, default=100, help='扩散步数，越大越接近扩散模型的结果，默认100') parser.add_argument('-se', '--second_encoding', action='store_true', default=False, help='二次编码，浅扩散前会对原始音频进行二次编码，玄学选项，有时候效果好，有时候效果差') parser.add_argument('-od', '--only_diffusion', action='store_true', default=False, help='纯扩散模式，该模式不会加载sovits模型，以扩散模型推理') # 不用动的部分 parser.add_argument('-sd', '--slice_db', type=int, default=-40, help='默认-40，嘈杂的音频可以-30，干声保留呼吸可以-50') parser.add_argument('-d', '--device', type=str, default=None, help='推理设备，None则为自动选择cpu和gpu') parser.add_argument('-ns', '--noice_scale', type=float, default=0.4, help='噪音级别，会影响咬字和音质，较为玄学') parser.add_argument('-p', '--pad_seconds', type=float, default=0.5, help='推理音频pad秒数，由于未知原因开头结尾会有异响，pad一小段静音段后就不会出现') parser.add_argument('-wf', '--wav_format', type=str, default='flac', help='音频输出格式') parser.add_argument('-lgr', '--linear_gradient_retain', type=float, default=0.75, help='自动音频切片后，需要舍弃每段切片的头尾。该参数设置交叉长度保留的比例，范围0-1,左开右闭') parser.add_argument('-eak', '--enhancer_adaptive_key', type=int, default=0, help='使增强器适应更高的音域(单位为半音数)|默认为0') parser.add_argument('-ft', '--f0_filter_threshold', type=float, default=0.05,help='F0过滤阈值，只有使用crepe时有效. 数值范围从0-1. 降低该值可减少跑调概率，但会增加哑音') args = parser.parse_args() clean_names = args.clean_names trans = args.trans spk_list = args.spk_list slice_db = args.slice_db wav_format = args.wav_format auto_predict_f0 = args.auto_predict_f0 cluster_infer_ratio = args.cluster_infer_ratio noice_scale = args.noice_scale pad_seconds = args.pad_seconds clip = args.clip lg = args.linear_gradient lgr = args.linear_gradient_retain f0p = args.f0_predictor enhance = args.enhance enhancer_adaptive_key = args.enhancer_adaptive_key cr_threshold = args.f0_filter_threshold diffusion_model_path = args.diffusion_model_path diffusion_config_path = args.diffusion_config_path k_step = args.k_step only_diffusion = args.only_diffusion shallow_diffusion = args.shallow_diffusion use_spk_mix = args.use_spk_mix second_encoding = args.second_encoding loudness_envelope_adjustment = args.loudness_envelope_adjustment model_path = args.model_path if "{lastest}" in model_path: model_path = getLastestCheckpoint(model_path[:model_path.find("{lastest}")]) logger.info("Auto choose {}", model_path) if cluster_infer_ratio != 0: if args.cluster_model_path == "": if args.feature_retrieval: # 若指定了占比但没有指定模型路径，则按是否使用特征检索分配默认的模型路径 args.cluster_model_path = "logs/44k/feature_and_index.pkl" else: args.cluster_model_path = "logs/44k/kmeans_10000.pt" else: # 若未指定占比，则无论是否指定模型路径，都将其置空以避免之后的模型加载 args.cluster_model_path = "" svc_model = Svc(model_path, args.config_path, args.device, args.cluster_model_path, enhance, diffusion_model_path, diffusion_config_path, shallow_diffusion, only_diffusion, use_spk_mix, args.feature_retrieval) infer_tool.mkdir(["raw", "results"]) if len(spk_mix_map)<=1: use_spk_mix = False if use_spk_mix: spk_list = [spk_mix_map] infer_tool.fill_a_to_b(trans, clean_names) for clean_name, tran in zip(clean_names, trans): raw_audio_path = f"raw/{clean_name}" if "." not in raw_audio_path: raw_audio_path += ".wav" infer_tool.format_wav(raw_audio_path) for spk in spk_list: kwarg = { "raw_audio_path" : raw_audio_path, "spk" : spk, "tran" : tran, "slice_db" : slice_db, "cluster_infer_ratio" : cluster_infer_ratio, "auto_predict_f0" : auto_predict_f0, "noice_scale" : noice_scale, "pad_seconds" : pad_seconds, "clip_seconds" : clip, "lg_num": lg, "lgr_num" : lgr, "f0_predictor" : f0p, "enhancer_adaptive_key" : enhancer_adaptive_key, "cr_threshold" : cr_threshold, "k_step":k_step, "use_spk_mix":use_spk_mix, "second_encoding":second_encoding, "loudness_envelope_adjustment":loudness_envelope_adjustment } try: audio = svc_model.slice_inference(**kwarg) except Exception as e: logger.error(f"Error in {clean_name} {spk} {tran}") logger.error(e) return key = "auto" if auto_predict_f0 else f"{tran}key" cluster_name = "" if cluster_infer_ratio == 0 else f"_{cluster_infer_ratio}" isdiffusion = "sovits" if shallow_diffusion : isdiffusion = "sovdiff" if only_diffusion : isdiffusion = "diff" if use_spk_mix: spk = "spk_mix" res_path = f'results/{clean_name}_{key}_{spk}{cluster_name}_{isdiffusion}_{f0p}.{wav_format}' soundfile.write(res_path, audio, svc_model.target_sample, format=wav_format) svc_model.clear_empty() if __name__ == '__main__': main() so-vits-svc-npu/logger/000077500000000000000000000000001456161405400153405ustar00rootroot00000000000000so-vits-svc-npu/logger/__init__.py000066400000000000000000000036241456161405400174560ustar00rootroot00000000000000from loguru import logger from rich.console import Console from rich.logging import RichHandler from rich.progress import Progress as _Progress from rich.progress import ( Progress, TextColumn, BarColumn, TimeElapsedColumn, TimeRemainingColumn, ) import time import os import datetime console = Console(stderr=None) logger.remove() def format_level(str, length): if len(str) < length: str = str + " " * (length - len(str)) else: str = str # 给 str 上对应 level 的颜色 if str == "INFO ": str = f"[bold green]{str}[/bold green]" elif str == "WARNING": str = f"[bold yellow]{str}[/bold yellow]" elif str == "ERROR ": str = f"[bold red]{str}[/bold red]" elif str == "DEBUG ": str = f"[bold cyan]{str}[/bold cyan]" return str def default_format(record): return f"[green]{record['time'].strftime('%Y-%m-%d %H:%M:%S')}[/green] | [level]{format_level(record['level'].name,7)}[/level] | [cyan]{record['file'].path.replace(os.getcwd()+os.sep,'')}:{record['line']}[/cyan] - [level]{record['message']}[/level]\n" logger.add(lambda m: console.print(m, end=""), format=default_format, colorize=True) def addLogger(path): logger.add( path, format="{time:YYYY-MM-DD HH:mm:ss.SSS} | " "{level: <8} | " "{name}:{function}:{line} - {message}", colorize=True, ) info = logger.info error = logger.error warning = logger.warning warn = logger.warning debug = logger.debug def hps(hps): console.print(hps) def Progress(): return _Progress( TextColumn("[progress.description]{task.description}"), # TextColumn("[progress.description]W"), BarColumn(), TextColumn("[progress.percentage]{task.percentage:>3.0f}%"), TimeRemainingColumn(), TextColumn("[red]*Elapsed[/red]"), TimeElapsedColumn(), console=console, )so-vits-svc-npu/logs/000077500000000000000000000000001456161405400150255ustar00rootroot00000000000000so-vits-svc-npu/logs/44k/000077500000000000000000000000001456161405400154275ustar00rootroot00000000000000so-vits-svc-npu/logs/44k/diffusion/000077500000000000000000000000001456161405400174155ustar00rootroot00000000000000so-vits-svc-npu/logs/44k/diffusion/put_diffusion_pretrained_model_here000066400000000000000000000000001456161405400266040ustar00rootroot00000000000000so-vits-svc-npu/logs/44k/put_pretrained_model_here000066400000000000000000000000001456161405400225500ustar00rootroot00000000000000so-vits-svc-npu/models.py000066400000000000000000000500551456161405400157230ustar00rootroot00000000000000import mindtorch.torch as torch from mindtorch.torch import nn from mindtorch.torch.nn import Conv1d, Conv2d from mindtorch.torch.nn import functional as F from mindtorch.torch.nn.utils import spectral_norm, weight_norm import modules.attentions as attentions import modules.commons as commons import modules.modules as modules import utils from modules.commons import get_padding from utils import f0_to_coarse class ResidualCouplingBlock(nn.Module): def __init__(self, channels, hidden_channels, kernel_size, dilation_rate, n_layers, n_flows=4, gin_channels=0, share_parameter=False ): super().__init__() self.channels = channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.dilation_rate = dilation_rate self.n_layers = n_layers self.n_flows = n_flows self.gin_channels = gin_channels self.flows = nn.ModuleList() self.wn = modules.WN(hidden_channels, kernel_size, dilation_rate, n_layers, p_dropout=0, gin_channels=gin_channels) if share_parameter else None for i in range(n_flows): self.flows.append( modules.ResidualCouplingLayer(channels, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=gin_channels, mean_only=True, wn_sharing_parameter=self.wn)) self.flows.append(modules.Flip()) def forward(self, x, x_mask, g=None, reverse=False): if not reverse: for flow in self.flows: x, _ = flow(x, x_mask, g=g, reverse=reverse) else: for flow in reversed(self.flows): x = flow(x, x_mask, g=g, reverse=reverse) return x class TransformerCouplingBlock(nn.Module): def __init__(self, channels, hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, n_flows=4, gin_channels=0, share_parameter=False ): super().__init__() self.channels = channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.n_layers = n_layers self.n_flows = n_flows self.gin_channels = gin_channels self.flows = nn.ModuleList() self.wn = attentions.FFT(hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, isflow = True, gin_channels = self.gin_channels) if share_parameter else None for i in range(n_flows): self.flows.append( modules.TransformerCouplingLayer(channels, hidden_channels, kernel_size, n_layers, n_heads, p_dropout, filter_channels, mean_only=True, wn_sharing_parameter=self.wn, gin_channels = self.gin_channels)) self.flows.append(modules.Flip()) def forward(self, x, x_mask, g=None, reverse=False): if not reverse: for flow in self.flows: x, _ = flow(x, x_mask, g=g, reverse=reverse) else: for flow in reversed(self.flows): x = flow(x, x_mask, g=g, reverse=reverse) return x class Encoder(nn.Module): def __init__(self, in_channels, out_channels, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=0): super().__init__() self.in_channels = in_channels self.out_channels = out_channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.dilation_rate = dilation_rate self.n_layers = n_layers self.gin_channels = gin_channels self.pre = nn.Conv1d(in_channels, hidden_channels, 1) self.enc = modules.WN(hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=gin_channels) self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1) def forward(self, x, x_lengths, g=None): # print(x.shape,x_lengths.shape) x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype) x = self.pre(x) * x_mask x = self.enc(x, x_mask, g=g) stats = self.proj(x) * x_mask m, logs = torch.split(stats, self.out_channels, dim=1) z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask return z, m, logs, x_mask class TextEncoder(nn.Module): def __init__(self, out_channels, hidden_channels, kernel_size, n_layers, gin_channels=0, filter_channels=None, n_heads=None, p_dropout=None): super().__init__() self.out_channels = out_channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.n_layers = n_layers self.gin_channels = gin_channels self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1) self.f0_emb = nn.Embedding(256, hidden_channels) self.enc_ = attentions.Encoder( hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout) def forward(self, x, x_mask, f0=None, noice_scale=1): x = x + self.f0_emb(f0).transpose(1, 2) x = self.enc_(x * x_mask, x_mask) stats = self.proj(x) * x_mask m, logs = torch.split(stats, self.out_channels, dim=1) z = (m + torch.randn_like(m) * torch.exp(logs) * noice_scale) * x_mask return z, m, logs, x_mask class DiscriminatorP(torch.nn.Module): def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False): super(DiscriminatorP, self).__init__() self.period = period self.use_spectral_norm = use_spectral_norm norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))), norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))), norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))), norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))), norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(get_padding(kernel_size, 1), 0))), ]) self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0))) def forward(self, x): fmap = [] # 1d to 2d b, c, t = x.shape if t % self.period != 0: # pad first n_pad = self.period - (t % self.period) x = F.pad(x, (0, n_pad), "reflect") t = t + n_pad x = x.view(b, c, t // self.period, self.period) for l in self.convs: x = l(x) x = F.leaky_relu(x, modules.LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class DiscriminatorS(torch.nn.Module): def __init__(self, use_spectral_norm=False): super(DiscriminatorS, self).__init__() norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv1d(1, 16, 15, 1, padding=7)), norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)), norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)), norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)), norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)), norm_f(Conv1d(1024, 1024, 5, 1, padding=2)), ]) self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1)) def forward(self, x): fmap = [] for l in self.convs: x = l(x) x = F.leaky_relu(x, modules.LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiPeriodDiscriminator(torch.nn.Module): def __init__(self, use_spectral_norm=False): super(MultiPeriodDiscriminator, self).__init__() periods = [2, 3, 5, 7, 11] discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)] discs = discs + [DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods] self.discriminators = nn.ModuleList(discs) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) y_d_gs.append(y_d_g) fmap_rs.append(fmap_r) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs class SpeakerEncoder(torch.nn.Module): def __init__(self, mel_n_channels=80, model_num_layers=3, model_hidden_size=256, model_embedding_size=256): super(SpeakerEncoder, self).__init__() self.lstm = nn.LSTM(mel_n_channels, model_hidden_size, model_num_layers, batch_first=True) self.linear = nn.Linear(model_hidden_size, model_embedding_size) self.relu = nn.ReLU() def forward(self, mels): self.lstm.flatten_parameters() _, (hidden, _) = self.lstm(mels) embeds_raw = self.relu(self.linear(hidden[-1])) return embeds_raw / torch.norm(embeds_raw, dim=1, keepdim=True) def compute_partial_slices(self, total_frames, partial_frames, partial_hop): mel_slices = [] for i in range(0, total_frames - partial_frames, partial_hop): mel_range = torch.arange(i, i + partial_frames) mel_slices.append(mel_range) return mel_slices def embed_utterance(self, mel, partial_frames=128, partial_hop=64): mel_len = mel.size(1) last_mel = mel[:, -partial_frames:] if mel_len > partial_frames: mel_slices = self.compute_partial_slices(mel_len, partial_frames, partial_hop) mels = list(mel[:, s] for s in mel_slices) mels.append(last_mel) mels = torch.stack(tuple(mels), 0).squeeze(1) with torch.no_grad(): partial_embeds = self(mels) embed = torch.mean(partial_embeds, axis=0).unsqueeze(0) # embed = embed / torch.linalg.norm(embed, 2) else: with torch.no_grad(): embed = self(last_mel) return embed class F0Decoder(nn.Module): def __init__(self, out_channels, hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, spk_channels=0): super().__init__() self.out_channels = out_channels self.hidden_channels = hidden_channels self.filter_channels = filter_channels self.n_heads = n_heads self.n_layers = n_layers self.kernel_size = kernel_size self.p_dropout = p_dropout self.spk_channels = spk_channels self.prenet = nn.Conv1d(hidden_channels, hidden_channels, 3, padding=1) self.decoder = attentions.FFT( hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout) self.proj = nn.Conv1d(hidden_channels, out_channels, 1) self.f0_prenet = nn.Conv1d(1, hidden_channels, 3, padding=1) self.cond = nn.Conv1d(spk_channels, hidden_channels, 1) def forward(self, x, norm_f0, x_mask, spk_emb=None): x = torch.detach(x) if (spk_emb is not None): x = x + self.cond(spk_emb) x += self.f0_prenet(norm_f0) x = self.prenet(x) * x_mask x = self.decoder(x * x_mask, x_mask) x = self.proj(x) * x_mask return x class SynthesizerTrn(nn.Module): """ Synthesizer for Training """ def __init__(self, spec_channels, segment_size, inter_channels, hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, resblock, resblock_kernel_sizes, resblock_dilation_sizes, upsample_rates, upsample_initial_channel, upsample_kernel_sizes, gin_channels, ssl_dim, n_speakers, sampling_rate=44100, vol_embedding=False, vocoder_name = "nsf-hifigan", use_depthwise_conv = False, use_automatic_f0_prediction = True, flow_share_parameter = False, n_flow_layer = 4, n_layers_trans_flow = 3, use_transformer_flow = False, **kwargs): super().__init__() self.spec_channels = spec_channels self.inter_channels = inter_channels self.hidden_channels = hidden_channels self.filter_channels = filter_channels self.n_heads = n_heads self.n_layers = n_layers self.kernel_size = kernel_size self.p_dropout = p_dropout self.resblock = resblock self.resblock_kernel_sizes = resblock_kernel_sizes self.resblock_dilation_sizes = resblock_dilation_sizes self.upsample_rates = upsample_rates self.upsample_initial_channel = upsample_initial_channel self.upsample_kernel_sizes = upsample_kernel_sizes self.segment_size = segment_size self.gin_channels = gin_channels self.ssl_dim = ssl_dim self.vol_embedding = vol_embedding self.emb_g = nn.Embedding(n_speakers, gin_channels) self.use_depthwise_conv = use_depthwise_conv self.use_automatic_f0_prediction = use_automatic_f0_prediction self.n_layers_trans_flow = n_layers_trans_flow if vol_embedding: self.emb_vol = nn.Linear(1, hidden_channels) self.pre = nn.Conv1d(ssl_dim, hidden_channels, kernel_size=5, padding=2) self.enc_p = TextEncoder( inter_channels, hidden_channels, filter_channels=filter_channels, n_heads=n_heads, n_layers=n_layers, kernel_size=kernel_size, p_dropout=p_dropout ) hps = { "sampling_rate": sampling_rate, "inter_channels": inter_channels, "resblock": resblock, "resblock_kernel_sizes": resblock_kernel_sizes, "resblock_dilation_sizes": resblock_dilation_sizes, "upsample_rates": upsample_rates, "upsample_initial_channel": upsample_initial_channel, "upsample_kernel_sizes": upsample_kernel_sizes, "gin_channels": gin_channels, "use_depthwise_conv":use_depthwise_conv } modules.set_Conv1dModel(self.use_depthwise_conv) if vocoder_name == "nsf-hifigan": from vdecoder.hifigan.models import Generator self.dec = Generator(h=hps) elif vocoder_name == "nsf-snake-hifigan": from vdecoder.hifiganwithsnake.models import Generator self.dec = Generator(h=hps) else: print("[?] Unkown vocoder: use default(nsf-hifigan)") from vdecoder.hifigan.models import Generator self.dec = Generator(h=hps) self.enc_q = Encoder(spec_channels, inter_channels, hidden_channels, 5, 1, 16, gin_channels=gin_channels) if use_transformer_flow: self.flow = TransformerCouplingBlock(inter_channels, hidden_channels, filter_channels, n_heads, n_layers_trans_flow, 5, p_dropout, n_flow_layer, gin_channels=gin_channels, share_parameter= flow_share_parameter) else: self.flow = ResidualCouplingBlock(inter_channels, hidden_channels, 5, 1, n_flow_layer, gin_channels=gin_channels, share_parameter= flow_share_parameter) if self.use_automatic_f0_prediction: self.f0_decoder = F0Decoder( 1, hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, spk_channels=gin_channels ) self.emb_uv = nn.Embedding(2, hidden_channels) self.character_mix = False def EnableCharacterMix(self, n_speakers_map, device): self.speaker_map = torch.zeros((n_speakers_map, 1, 1, self.gin_channels)).to(device) for i in range(n_speakers_map): self.speaker_map[i] = self.emb_g(torch.LongTensor([[i]]).to(device)) self.speaker_map = self.speaker_map.unsqueeze(0).to(device) self.character_mix = True def forward(self, c, f0, uv, spec, g=None, c_lengths=None, spec_lengths=None, vol = None): g = self.emb_g(g).transpose(1,2) # vol proj vol = self.emb_vol(vol[:,:,None]).transpose(1,2) if vol is not None and self.vol_embedding else 0 # ssl prenet x_mask = torch.unsqueeze(commons.sequence_mask(c_lengths, c.size(2)), 1).to(c.dtype) x = self.pre(c) * x_mask + self.emb_uv(uv.long()).transpose(1,2) + vol # f0 predict if self.use_automatic_f0_prediction: lf0 = 2595. * torch.log10(1. + f0.unsqueeze(1) / 700.) / 500 norm_lf0 = utils.normalize_f0(lf0, x_mask, uv) pred_lf0 = self.f0_decoder(x, norm_lf0, x_mask, spk_emb=g) else: lf0 = 0 norm_lf0 = 0 pred_lf0 = 0 # encoder z_ptemp, m_p, logs_p, _ = self.enc_p(x, x_mask, f0=f0_to_coarse(f0)) z, m_q, logs_q, spec_mask = self.enc_q(spec, spec_lengths, g=g) # flow z_p = self.flow(z, spec_mask, g=g) z_slice, pitch_slice, ids_slice = commons.rand_slice_segments_with_pitch(z, f0, spec_lengths, self.segment_size) # nsf decoder o = self.dec(z_slice, g=g, f0=pitch_slice) return o, ids_slice, spec_mask, (z, z_p, m_p, logs_p, m_q, logs_q), pred_lf0, norm_lf0, lf0 @torch.no_grad() def infer(self, c, f0, uv, g=None, noice_scale=0.35, seed=52468, predict_f0=False, vol = None): if c.device == torch.device("cuda"): torch.cuda.manual_seed_all(seed) else: torch.manual_seed(seed) c_lengths = (torch.ones(c.size(0)) * c.size(-1)).to(c.device) if self.character_mix and len(g) > 1: # [N, S] * [S, B, 1, H] g = g.reshape((g.shape[0], g.shape[1], 1, 1, 1)) # [N, S, B, 1, 1] g = g * self.speaker_map # [N, S, B, 1, H] g = torch.sum(g, dim=1) # [N, 1, B, 1, H] g = g.transpose(0, -1).transpose(0, -2).squeeze(0) # [B, H, N] else: if g.dim() == 1: g = g.unsqueeze(0) g = self.emb_g(g).transpose(1, 2) x_mask = torch.unsqueeze(commons.sequence_mask(c_lengths, c.size(2)), 1).to(c.dtype) # vol proj vol = self.emb_vol(vol[:,:,None]).transpose(1,2) if vol is not None and self.vol_embedding else 0 x = self.pre(c) * x_mask + self.emb_uv(uv.long()).transpose(1, 2) + vol if self.use_automatic_f0_prediction and predict_f0: lf0 = 2595. * torch.log10(1. + f0.unsqueeze(1) / 700.) / 500 norm_lf0 = utils.normalize_f0(lf0, x_mask, uv, random_scale=False) pred_lf0 = self.f0_decoder(x, norm_lf0, x_mask, spk_emb=g) f0 = (700 * (torch.pow(10, pred_lf0 * 500 / 2595) - 1)).squeeze(1) z_p, m_p, logs_p, c_mask = self.enc_p(x, x_mask, f0=f0_to_coarse(f0), noice_scale=noice_scale) z = self.flow(z_p, c_mask, g=g, reverse=True) o = self.dec(z * c_mask, g=g, f0=f0) return o,f0 so-vits-svc-npu/modules/000077500000000000000000000000001456161405400155315ustar00rootroot00000000000000so-vits-svc-npu/modules/DSConv.py000066400000000000000000000057261456161405400172510ustar00rootroot00000000000000import mindtorch.torch.nn as nn from mindtorch.torch.nn.utils import remove_weight_norm, weight_norm class Depthwise_Separable_Conv1D(nn.Module): def __init__( self, in_channels, out_channels, kernel_size, stride = 1, padding = 0, dilation = 1, bias = True, padding_mode = 'zeros', # TODO: refine this type device=None, dtype=None ): super().__init__() self.depth_conv = nn.Conv1d(in_channels=in_channels, out_channels=in_channels, kernel_size=kernel_size, groups=in_channels,stride = stride,padding=padding,dilation=dilation,bias=bias,padding_mode=padding_mode,device=device,dtype=dtype) self.point_conv = nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=1, bias=bias, device=device,dtype=dtype) def forward(self, input): return self.point_conv(self.depth_conv(input)) def weight_norm(self): self.depth_conv = weight_norm(self.depth_conv, name = 'weight') self.point_conv = weight_norm(self.point_conv, name = 'weight') def remove_weight_norm(self): self.depth_conv = remove_weight_norm(self.depth_conv, name = 'weight') self.point_conv = remove_weight_norm(self.point_conv, name = 'weight') class Depthwise_Separable_TransposeConv1D(nn.Module): def __init__( self, in_channels, out_channels, kernel_size, stride = 1, padding = 0, output_padding = 0, bias = True, dilation = 1, padding_mode = 'zeros', # TODO: refine this type device=None, dtype=None ): super().__init__() self.depth_conv = nn.ConvTranspose1d(in_channels=in_channels, out_channels=in_channels, kernel_size=kernel_size, groups=in_channels,stride = stride,output_padding=output_padding,padding=padding,dilation=dilation,bias=bias,padding_mode=padding_mode,device=device,dtype=dtype) self.point_conv = nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=1, bias=bias, device=device,dtype=dtype) def forward(self, input): return self.point_conv(self.depth_conv(input)) def weight_norm(self): self.depth_conv = weight_norm(self.depth_conv, name = 'weight') self.point_conv = weight_norm(self.point_conv, name = 'weight') def remove_weight_norm(self): remove_weight_norm(self.depth_conv, name = 'weight') remove_weight_norm(self.point_conv, name = 'weight') def weight_norm_modules(module, name = 'weight', dim = 0): if isinstance(module,Depthwise_Separable_Conv1D) or isinstance(module,Depthwise_Separable_TransposeConv1D): module.weight_norm() return module else: return weight_norm(module,name,dim) def remove_weight_norm_modules(module, name = 'weight'): if isinstance(module,Depthwise_Separable_Conv1D) or isinstance(module,Depthwise_Separable_TransposeConv1D): module.remove_weight_norm() else: remove_weight_norm(module,name)so-vits-svc-npu/modules/F0Predictor/000077500000000000000000000000001456161405400176525ustar00rootroot00000000000000so-vits-svc-npu/modules/F0Predictor/CrepeF0Predictor.py000066400000000000000000000026031456161405400233250ustar00rootroot00000000000000import mindtorch.torch as torch from modules.F0Predictor.crepe import CrepePitchExtractor from modules.F0Predictor.F0Predictor import F0Predictor class CrepeF0Predictor(F0Predictor): def __init__(self,hop_length=512,f0_min=50,f0_max=1100,device=None,sampling_rate=44100,threshold=0.05,model="full"): self.F0Creper = CrepePitchExtractor(hop_length=hop_length,f0_min=f0_min,f0_max=f0_max,device=device,threshold=threshold,model=model) self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max self.device = device self.threshold = threshold self.sampling_rate = sampling_rate self.name = "crepe" def compute_f0(self,wav,p_len=None): x = torch.FloatTensor(wav).to(self.device) if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" f0,uv = self.F0Creper(x[None,:].float(),self.sampling_rate,pad_to=p_len) return f0 def compute_f0_uv(self,wav,p_len=None): x = torch.FloatTensor(wav).to(self.device) if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" f0,uv = self.F0Creper(x[None,:].float(),self.sampling_rate,pad_to=p_len) return f0,uvso-vits-svc-npu/modules/F0Predictor/DioF0Predictor.py000066400000000000000000000051511456161405400230030ustar00rootroot00000000000000import numpy as np import pyworld from modules.F0Predictor.F0Predictor import F0Predictor class DioF0Predictor(F0Predictor): def __init__(self,hop_length=512,f0_min=50,f0_max=1100,sampling_rate=44100): self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max self.sampling_rate = sampling_rate self.name = "dio" def interpolate_f0(self,f0): ''' 对F0进行插值处理 ''' vuv_vector = np.zeros_like(f0, dtype=np.float32) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 nzindex = np.nonzero(f0)[0] data = f0[nzindex] nzindex = nzindex.astype(np.float32) time_org = self.hop_length / self.sampling_rate * nzindex time_frame = np.arange(f0.shape[0]) * self.hop_length / self.sampling_rate if data.shape[0] <= 0: return np.zeros(f0.shape[0], dtype=np.float32),vuv_vector if data.shape[0] == 1: return np.ones(f0.shape[0], dtype=np.float32) * f0[0],vuv_vector f0 = np.interp(time_frame, time_org, data, left=data[0], right=data[-1]) return f0,vuv_vector def resize_f0(self,x, target_len): source = np.array(x) source[source<0.001] = np.nan target = np.interp(np.arange(0, len(source)*target_len, len(source))/ target_len, np.arange(0, len(source)), source) res = np.nan_to_num(target) return res def compute_f0(self,wav,p_len=None): if p_len is None: p_len = wav.shape[0]//self.hop_length f0, t = pyworld.dio( wav.astype(np.double), fs=self.sampling_rate, f0_floor=self.f0_min, f0_ceil=self.f0_max, frame_period=1000 * self.hop_length / self.sampling_rate, ) f0 = pyworld.stonemask(wav.astype(np.double), f0, t, self.sampling_rate) for index, pitch in enumerate(f0): f0[index] = round(pitch, 1) return self.interpolate_f0(self.resize_f0(f0, p_len))[0] def compute_f0_uv(self,wav,p_len=None): if p_len is None: p_len = wav.shape[0]//self.hop_length f0, t = pyworld.dio( wav.astype(np.double), fs=self.sampling_rate, f0_floor=self.f0_min, f0_ceil=self.f0_max, frame_period=1000 * self.hop_length / self.sampling_rate, ) f0 = pyworld.stonemask(wav.astype(np.double), f0, t, self.sampling_rate) for index, pitch in enumerate(f0): f0[index] = round(pitch, 1) return self.interpolate_f0(self.resize_f0(f0, p_len)) so-vits-svc-npu/modules/F0Predictor/F0Predictor.py000066400000000000000000000006451456161405400223520ustar00rootroot00000000000000class F0Predictor(object): def compute_f0(self,wav,p_len): ''' input: wav:[signal_length] p_len:int output: f0:[signal_length//hop_length] ''' pass def compute_f0_uv(self,wav,p_len): ''' input: wav:[signal_length] p_len:int output: f0:[signal_length//hop_length],uv:[signal_length//hop_length] ''' passso-vits-svc-npu/modules/F0Predictor/FCPEF0Predictor.py000066400000000000000000000077021456161405400230110ustar00rootroot00000000000000from typing import Union import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from modules.F0Predictor.F0Predictor import F0Predictor from .fcpe.model import FCPEInfer class FCPEF0Predictor(F0Predictor): def __init__(self, hop_length=512, f0_min=50, f0_max=1100, dtype=torch.float32, device=None, sampling_rate=44100, threshold=0.05): self.fcpe = FCPEInfer(model_path="pretrain/fcpe.pt", device=device, dtype=dtype) self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max if device is None: self.device = 'cuda' if torch.cuda.is_available() else 'cpu' else: self.device = device self.threshold = threshold self.sampling_rate = sampling_rate self.dtype = dtype self.name = "fcpe" def repeat_expand( self, content: Union[torch.Tensor, np.ndarray], target_len: int, mode: str = "nearest" ): ndim = content.ndim if content.ndim == 1: content = content[None, None] elif content.ndim == 2: content = content[None] assert content.ndim == 3 is_np = isinstance(content, np.ndarray) if is_np: content = torch.from_numpy(content) results = torch.nn.functional.interpolate(content, size=target_len, mode=mode) if is_np: results = results.numpy() if ndim == 1: return results[0, 0] elif ndim == 2: return results[0] def post_process(self, x, sampling_rate, f0, pad_to): if isinstance(f0, np.ndarray): f0 = torch.from_numpy(f0).float().to(x.device) if pad_to is None: return f0 f0 = self.repeat_expand(f0, pad_to) vuv_vector = torch.zeros_like(f0) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 # 去掉0频率, 并线性插值 nzindex = torch.nonzero(f0).squeeze() f0 = torch.index_select(f0, dim=0, index=nzindex).cpu().numpy() time_org = self.hop_length / sampling_rate * nzindex.cpu().numpy() time_frame = np.arange(pad_to) * self.hop_length / sampling_rate vuv_vector = F.interpolate(vuv_vector[None, None, :], size=pad_to)[0][0] if f0.shape[0] <= 0: return torch.zeros(pad_to, dtype=torch.float, device=x.device).cpu().numpy(), vuv_vector.cpu().numpy() if f0.shape[0] == 1: return (torch.ones(pad_to, dtype=torch.float, device=x.device) * f0[ 0]).cpu().numpy(), vuv_vector.cpu().numpy() # 大概可以用 torch 重写? f0 = np.interp(time_frame, time_org, f0, left=f0[0], right=f0[-1]) # vuv_vector = np.ceil(scipy.ndimage.zoom(vuv_vector,pad_to/len(vuv_vector),order = 0)) return f0, vuv_vector.cpu().numpy() def compute_f0(self, wav, p_len=None): x = torch.FloatTensor(wav).to(self.dtype).to(self.device) if p_len is None: p_len = x.shape[0] // self.hop_length else: assert abs(p_len - x.shape[0] // self.hop_length) < 4, "pad length error" f0 = self.fcpe(x, sr=self.sampling_rate, threshold=self.threshold)[0,:,0] if torch.all(f0 == 0): rtn = f0.cpu().numpy() if p_len is None else np.zeros(p_len) return rtn, rtn return self.post_process(x, self.sampling_rate, f0, p_len)[0] def compute_f0_uv(self, wav, p_len=None): x = torch.FloatTensor(wav).to(self.dtype).to(self.device) if p_len is None: p_len = x.shape[0] // self.hop_length else: assert abs(p_len - x.shape[0] // self.hop_length) < 4, "pad length error" f0 = self.fcpe(x, sr=self.sampling_rate, threshold=self.threshold)[0,:,0] if torch.all(f0 == 0): rtn = f0.cpu().numpy() if p_len is None else np.zeros(p_len) return rtn, rtn return self.post_process(x, self.sampling_rate, f0, p_len)so-vits-svc-npu/modules/F0Predictor/HarvestF0Predictor.py000066400000000000000000000047341456161405400237120ustar00rootroot00000000000000import numpy as np import pyworld from modules.F0Predictor.F0Predictor import F0Predictor class HarvestF0Predictor(F0Predictor): def __init__(self,hop_length=512,f0_min=50,f0_max=1100,sampling_rate=44100): self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max self.sampling_rate = sampling_rate self.name = "harvest" def interpolate_f0(self,f0): ''' 对F0进行插值处理 ''' vuv_vector = np.zeros_like(f0, dtype=np.float32) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 nzindex = np.nonzero(f0)[0] data = f0[nzindex] nzindex = nzindex.astype(np.float32) time_org = self.hop_length / self.sampling_rate * nzindex time_frame = np.arange(f0.shape[0]) * self.hop_length / self.sampling_rate if data.shape[0] <= 0: return np.zeros(f0.shape[0], dtype=np.float32),vuv_vector if data.shape[0] == 1: return np.ones(f0.shape[0], dtype=np.float32) * f0[0],vuv_vector f0 = np.interp(time_frame, time_org, data, left=data[0], right=data[-1]) return f0,vuv_vector def resize_f0(self,x, target_len): source = np.array(x) source[source<0.001] = np.nan target = np.interp(np.arange(0, len(source)*target_len, len(source))/ target_len, np.arange(0, len(source)), source) res = np.nan_to_num(target) return res def compute_f0(self,wav,p_len=None): if p_len is None: p_len = wav.shape[0]//self.hop_length f0, t = pyworld.harvest( wav.astype(np.double), fs=self.hop_length, f0_ceil=self.f0_max, f0_floor=self.f0_min, frame_period=1000 * self.hop_length / self.sampling_rate, ) f0 = pyworld.stonemask(wav.astype(np.double), f0, t, self.fs) return self.interpolate_f0(self.resize_f0(f0, p_len))[0] def compute_f0_uv(self,wav,p_len=None): if p_len is None: p_len = wav.shape[0]//self.hop_length f0, t = pyworld.harvest( wav.astype(np.double), fs=self.sampling_rate, f0_floor=self.f0_min, f0_ceil=self.f0_max, frame_period=1000 * self.hop_length / self.sampling_rate, ) f0 = pyworld.stonemask(wav.astype(np.double), f0, t, self.sampling_rate) return self.interpolate_f0(self.resize_f0(f0, p_len)) so-vits-svc-npu/modules/F0Predictor/PMF0Predictor.py000066400000000000000000000052351456161405400226070ustar00rootroot00000000000000import numpy as np import parselmouth from modules.F0Predictor.F0Predictor import F0Predictor class PMF0Predictor(F0Predictor): def __init__(self,hop_length=512,f0_min=50,f0_max=1100,sampling_rate=44100): self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max self.sampling_rate = sampling_rate self.name = "pm" def interpolate_f0(self,f0): ''' 对F0进行插值处理 ''' vuv_vector = np.zeros_like(f0, dtype=np.float32) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 nzindex = np.nonzero(f0)[0] data = f0[nzindex] nzindex = nzindex.astype(np.float32) time_org = self.hop_length / self.sampling_rate * nzindex time_frame = np.arange(f0.shape[0]) * self.hop_length / self.sampling_rate if data.shape[0] <= 0: return np.zeros(f0.shape[0], dtype=np.float32),vuv_vector if data.shape[0] == 1: return np.ones(f0.shape[0], dtype=np.float32) * f0[0],vuv_vector f0 = np.interp(time_frame, time_org, data, left=data[0], right=data[-1]) return f0,vuv_vector def compute_f0(self,wav,p_len=None): x = wav if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" time_step = self.hop_length / self.sampling_rate * 1000 f0 = parselmouth.Sound(x, self.sampling_rate).to_pitch_ac( time_step=time_step / 1000, voicing_threshold=0.6, pitch_floor=self.f0_min, pitch_ceiling=self.f0_max).selected_array['frequency'] pad_size=(p_len - len(f0) + 1) // 2 if(pad_size>0 or p_len - len(f0) - pad_size>0): f0 = np.pad(f0,[[pad_size,p_len - len(f0) - pad_size]], mode='constant') f0,uv = self.interpolate_f0(f0) return f0 def compute_f0_uv(self,wav,p_len=None): x = wav if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" time_step = self.hop_length / self.sampling_rate * 1000 f0 = parselmouth.Sound(x, self.sampling_rate).to_pitch_ac( time_step=time_step / 1000, voicing_threshold=0.6, pitch_floor=self.f0_min, pitch_ceiling=self.f0_max).selected_array['frequency'] pad_size=(p_len - len(f0) + 1) // 2 if(pad_size>0 or p_len - len(f0) - pad_size>0): f0 = np.pad(f0,[[pad_size,p_len - len(f0) - pad_size]], mode='constant') f0,uv = self.interpolate_f0(f0) return f0,uv so-vits-svc-npu/modules/F0Predictor/RMVPEF0Predictor.py000066400000000000000000000076161456161405400231710ustar00rootroot00000000000000from typing import Union import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from modules.F0Predictor.F0Predictor import F0Predictor from .rmvpe import RMVPE class RMVPEF0Predictor(F0Predictor): def __init__(self,hop_length=512,f0_min=50,f0_max=1100, dtype=torch.float32, device=None,sampling_rate=44100,threshold=0.05): self.rmvpe = RMVPE(model_path="pretrain/rmvpe.pt",dtype=dtype,device=device) self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max if device is None: self.device = 'cuda' if torch.cuda.is_available() else 'cpu' else: self.device = device self.threshold = threshold self.sampling_rate = sampling_rate self.dtype = dtype self.name = "rmvpe" def repeat_expand( self, content: Union[torch.Tensor, np.ndarray], target_len: int, mode: str = "nearest" ): ndim = content.ndim if content.ndim == 1: content = content[None, None] elif content.ndim == 2: content = content[None] assert content.ndim == 3 is_np = isinstance(content, np.ndarray) if is_np: content = torch.from_numpy(content) results = torch.nn.functional.interpolate(content, size=target_len, mode=mode) if is_np: results = results.numpy() if ndim == 1: return results[0, 0] elif ndim == 2: return results[0] def post_process(self, x, sampling_rate, f0, pad_to): if isinstance(f0, np.ndarray): f0 = torch.from_numpy(f0).float().to(x.device) if pad_to is None: return f0 f0 = self.repeat_expand(f0, pad_to) vuv_vector = torch.zeros_like(f0) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 # 去掉0频率, 并线性插值 nzindex = torch.nonzero(f0).squeeze() f0 = torch.index_select(f0, dim=0, index=nzindex).cpu().numpy() time_org = self.hop_length / sampling_rate * nzindex.cpu().numpy() time_frame = np.arange(pad_to) * self.hop_length / sampling_rate vuv_vector = F.interpolate(vuv_vector[None,None,:],size=pad_to)[0][0] if f0.shape[0] <= 0: return torch.zeros(pad_to, dtype=torch.float, device=x.device).cpu().numpy(),vuv_vector.cpu().numpy() if f0.shape[0] == 1: return (torch.ones(pad_to, dtype=torch.float, device=x.device) * f0[0]).cpu().numpy() ,vuv_vector.cpu().numpy() # 大概可以用 torch 重写? f0 = np.interp(time_frame, time_org, f0, left=f0[0], right=f0[-1]) #vuv_vector = np.ceil(scipy.ndimage.zoom(vuv_vector,pad_to/len(vuv_vector),order = 0)) return f0,vuv_vector.cpu().numpy() def compute_f0(self,wav,p_len=None): x = torch.FloatTensor(wav).to(self.dtype).to(self.device) if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" f0 = self.rmvpe.infer_from_audio(x,self.sampling_rate,self.threshold) if torch.all(f0 == 0): rtn = f0.cpu().numpy() if p_len is None else np.zeros(p_len) return rtn,rtn return self.post_process(x,self.sampling_rate,f0,p_len)[0] def compute_f0_uv(self,wav,p_len=None): x = torch.FloatTensor(wav).to(self.dtype).to(self.device) if p_len is None: p_len = x.shape[0]//self.hop_length else: assert abs(p_len-x.shape[0]//self.hop_length) < 4, "pad length error" f0 = self.rmvpe.infer_from_audio(x,self.sampling_rate,self.threshold) if torch.all(f0 == 0): rtn = f0.cpu().numpy() if p_len is None else np.zeros(p_len) return rtn,rtn return self.post_process(x,self.sampling_rate,f0,p_len)so-vits-svc-npu/modules/F0Predictor/__init__.py000066400000000000000000000000001456161405400217510ustar00rootroot00000000000000so-vits-svc-npu/modules/F0Predictor/crepe.py000066400000000000000000000262301456161405400213250ustar00rootroot00000000000000from typing import Optional, Union try: from typing import Literal except Exception: from typing_extensions import Literal import numpy as np import mindtorch.torch as torch import torchcrepe from mindtorch.torch import nn from mindtorch.torch.nn import functional as F #from:https://github.com/fishaudio/fish-diffusion def repeat_expand( content: Union[torch.Tensor, np.ndarray], target_len: int, mode: str = "nearest" ): """Repeat content to target length. This is a wrapper of torch.nn.functional.interpolate. Args: content (torch.Tensor): tensor target_len (int): target length mode (str, optional): interpolation mode. Defaults to "nearest". Returns: torch.Tensor: tensor """ ndim = content.ndim if content.ndim == 1: content = content[None, None] elif content.ndim == 2: content = content[None] assert content.ndim == 3 is_np = isinstance(content, np.ndarray) if is_np: content = torch.from_numpy(content) results = torch.nn.functional.interpolate(content, size=target_len, mode=mode) if is_np: results = results.numpy() if ndim == 1: return results[0, 0] elif ndim == 2: return results[0] class BasePitchExtractor: def __init__( self, hop_length: int = 512, f0_min: float = 50.0, f0_max: float = 1100.0, keep_zeros: bool = True, ): """Base pitch extractor. Args: hop_length (int, optional): Hop length. Defaults to 512. f0_min (float, optional): Minimum f0. Defaults to 50.0. f0_max (float, optional): Maximum f0. Defaults to 1100.0. keep_zeros (bool, optional): Whether keep zeros in pitch. Defaults to True. """ self.hop_length = hop_length self.f0_min = f0_min self.f0_max = f0_max self.keep_zeros = keep_zeros def __call__(self, x, sampling_rate=44100, pad_to=None): raise NotImplementedError("BasePitchExtractor is not callable.") def post_process(self, x, sampling_rate, f0, pad_to): if isinstance(f0, np.ndarray): f0 = torch.from_numpy(f0).float().to(x.device) if pad_to is None: return f0 f0 = repeat_expand(f0, pad_to) if self.keep_zeros: return f0 vuv_vector = torch.zeros_like(f0) vuv_vector[f0 > 0.0] = 1.0 vuv_vector[f0 <= 0.0] = 0.0 # 去掉0频率, 并线性插值 nzindex = torch.nonzero(f0).squeeze() f0 = torch.index_select(f0, dim=0, index=nzindex).cpu().numpy() time_org = self.hop_length / sampling_rate * nzindex.cpu().numpy() time_frame = np.arange(pad_to) * self.hop_length / sampling_rate vuv_vector = F.interpolate(vuv_vector[None,None,:],size=pad_to)[0][0] if f0.shape[0] <= 0: return torch.zeros(pad_to, dtype=torch.float, device=x.device),vuv_vector.cpu().numpy() if f0.shape[0] == 1: return torch.ones(pad_to, dtype=torch.float, device=x.device) * f0[0],vuv_vector.cpu().numpy() # 大概可以用 torch 重写? f0 = np.interp(time_frame, time_org, f0, left=f0[0], right=f0[-1]) #vuv_vector = np.ceil(scipy.ndimage.zoom(vuv_vector,pad_to/len(vuv_vector),order = 0)) return f0,vuv_vector.cpu().numpy() class MaskedAvgPool1d(nn.Module): def __init__( self, kernel_size: int, stride: Optional[int] = None, padding: Optional[int] = 0 ): """An implementation of mean pooling that supports masked values. Args: kernel_size (int): The size of the median pooling window. stride (int, optional): The stride of the median pooling window. Defaults to None. padding (int, optional): The padding of the median pooling window. Defaults to 0. """ super(MaskedAvgPool1d, self).__init__() self.kernel_size = kernel_size self.stride = stride or kernel_size self.padding = padding def forward(self, x, mask=None): ndim = x.dim() if ndim == 2: x = x.unsqueeze(1) assert ( x.dim() == 3 ), "Input tensor must have 2 or 3 dimensions (batch_size, channels, width)" # Apply the mask by setting masked elements to zero, or make NaNs zero if mask is None: mask = ~torch.isnan(x) # Ensure mask has the same shape as the input tensor assert x.shape == mask.shape, "Input tensor and mask must have the same shape" masked_x = torch.where(mask, x, torch.zeros_like(x)) # Create a ones kernel with the same number of channels as the input tensor ones_kernel = torch.ones(x.size(1), 1, self.kernel_size, device=x.device) # Perform sum pooling sum_pooled = nn.functional.conv1d( masked_x, ones_kernel, stride=self.stride, padding=self.padding, groups=x.size(1), ) # Count the non-masked (valid) elements in each pooling window valid_count = nn.functional.conv1d( mask.float(), ones_kernel, stride=self.stride, padding=self.padding, groups=x.size(1), ) valid_count = valid_count.clamp(min=1) # Avoid division by zero # Perform masked average pooling avg_pooled = sum_pooled / valid_count # Fill zero values with NaNs avg_pooled[avg_pooled == 0] = float("nan") if ndim == 2: return avg_pooled.squeeze(1) return avg_pooled class MaskedMedianPool1d(nn.Module): def __init__( self, kernel_size: int, stride: Optional[int] = None, padding: Optional[int] = 0 ): """An implementation of median pooling that supports masked values. This implementation is inspired by the median pooling implementation in https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598 Args: kernel_size (int): The size of the median pooling window. stride (int, optional): The stride of the median pooling window. Defaults to None. padding (int, optional): The padding of the median pooling window. Defaults to 0. """ super(MaskedMedianPool1d, self).__init__() self.kernel_size = kernel_size self.stride = stride or kernel_size self.padding = padding def forward(self, x, mask=None): ndim = x.dim() if ndim == 2: x = x.unsqueeze(1) assert ( x.dim() == 3 ), "Input tensor must have 2 or 3 dimensions (batch_size, channels, width)" if mask is None: mask = ~torch.isnan(x) assert x.shape == mask.shape, "Input tensor and mask must have the same shape" masked_x = torch.where(mask, x, torch.zeros_like(x)) x = F.pad(masked_x, (self.padding, self.padding), mode="reflect") mask = F.pad( mask.float(), (self.padding, self.padding), mode="constant", value=0 ) x = x.unfold(2, self.kernel_size, self.stride) mask = mask.unfold(2, self.kernel_size, self.stride) x = x.contiguous().view(x.size()[:3] + (-1,)) mask = mask.contiguous().view(mask.size()[:3] + (-1,)).to(x.device) # Combine the mask with the input tensor #x_masked = torch.where(mask.bool(), x, torch.fill_(torch.zeros_like(x),float("inf"))) x_masked = torch.where(mask.bool(), x, torch.FloatTensor([float("inf")]).to(x.device)) # Sort the masked tensor along the last dimension x_sorted, _ = torch.sort(x_masked, dim=-1) # Compute the count of non-masked (valid) values valid_count = mask.sum(dim=-1) # Calculate the index of the median value for each pooling window median_idx = (torch.div((valid_count - 1), 2, rounding_mode='trunc')).clamp(min=0) # Gather the median values using the calculated indices median_pooled = x_sorted.gather(-1, median_idx.unsqueeze(-1).long()).squeeze(-1) # Fill infinite values with NaNs median_pooled[torch.isinf(median_pooled)] = float("nan") if ndim == 2: return median_pooled.squeeze(1) return median_pooled class CrepePitchExtractor(BasePitchExtractor): def __init__( self, hop_length: int = 512, f0_min: float = 50.0, f0_max: float = 1100.0, threshold: float = 0.05, keep_zeros: bool = False, device = None, model: Literal["full", "tiny"] = "full", use_fast_filters: bool = True, decoder="viterbi" ): super().__init__(hop_length, f0_min, f0_max, keep_zeros) if decoder == "viterbi": self.decoder = torchcrepe.decode.viterbi elif decoder == "argmax": self.decoder = torchcrepe.decode.argmax elif decoder == "weighted_argmax": self.decoder = torchcrepe.decode.weighted_argmax else: raise "Unknown decoder" self.threshold = threshold self.model = model self.use_fast_filters = use_fast_filters self.hop_length = hop_length if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) if self.use_fast_filters: self.median_filter = MaskedMedianPool1d(3, 1, 1).to(device) self.mean_filter = MaskedAvgPool1d(3, 1, 1).to(device) def __call__(self, x, sampling_rate=44100, pad_to=None): """Extract pitch using crepe. Args: x (torch.Tensor): Audio signal, shape (1, T). sampling_rate (int, optional): Sampling rate. Defaults to 44100. pad_to (int, optional): Pad to length. Defaults to None. Returns: torch.Tensor: Pitch, shape (T // hop_length,). """ assert x.ndim == 2, f"Expected 2D tensor, got {x.ndim}D tensor." assert x.shape[0] == 1, f"Expected 1 channel, got {x.shape[0]} channels." x = x.to(self.dev) f0, pd = torchcrepe.predict( x, sampling_rate, self.hop_length, self.f0_min, self.f0_max, pad=True, model=self.model, batch_size=1024, device=x.device, return_periodicity=True, decoder=self.decoder ) # Filter, remove silence, set uv threshold, refer to the original warehouse readme if self.use_fast_filters: pd = self.median_filter(pd) else: pd = torchcrepe.filter.median(pd, 3) pd = torchcrepe.threshold.Silence(-60.0)(pd, x, sampling_rate, self.hop_length) f0 = torchcrepe.threshold.At(self.threshold)(f0, pd) if self.use_fast_filters: f0 = self.mean_filter(f0) else: f0 = torchcrepe.filter.mean(f0, 3) f0 = torch.where(torch.isnan(f0), torch.full_like(f0, 0), f0)[0] if torch.all(f0 == 0): rtn = f0.cpu().numpy() if pad_to is None else np.zeros(pad_to) return rtn,rtn return self.post_process(x, sampling_rate, f0, pad_to) so-vits-svc-npu/modules/F0Predictor/fcpe/000077500000000000000000000000001456161405400205675ustar00rootroot00000000000000so-vits-svc-npu/modules/F0Predictor/fcpe/__init__.py000066400000000000000000000001741456161405400227020ustar00rootroot00000000000000from .model import FCPEInfer # noqa: F401 from .nvSTFT import STFT # noqa: F401 from .pcmer import PCmer # noqa: F401 so-vits-svc-npu/modules/F0Predictor/fcpe/model.py000066400000000000000000000241211456161405400222410ustar00rootroot00000000000000import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn.utils import weight_norm from mindtorch.torchaudio.transforms import Resample from .nvSTFT import STFT from .pcmer import PCmer def l2_regularization(model, l2_alpha): l2_loss = [] for module in model.modules(): if type(module) is nn.Conv2d: l2_loss.append((module.weight ** 2).sum() / 2.0) return l2_alpha * sum(l2_loss) class FCPE(nn.Module): def __init__( self, input_channel=128, out_dims=360, n_layers=12, n_chans=512, use_siren=False, use_full=False, loss_mse_scale=10, loss_l2_regularization=False, loss_l2_regularization_scale=1, loss_grad1_mse=False, loss_grad1_mse_scale=1, f0_max=1975.5, f0_min=32.70, confidence=False, threshold=0.05, use_input_conv=True ): super().__init__() if use_siren is True: raise ValueError("Siren is not supported yet.") if use_full is True: raise ValueError("Full model is not supported yet.") self.loss_mse_scale = loss_mse_scale if (loss_mse_scale is not None) else 10 self.loss_l2_regularization = loss_l2_regularization if (loss_l2_regularization is not None) else False self.loss_l2_regularization_scale = loss_l2_regularization_scale if (loss_l2_regularization_scale is not None) else 1 self.loss_grad1_mse = loss_grad1_mse if (loss_grad1_mse is not None) else False self.loss_grad1_mse_scale = loss_grad1_mse_scale if (loss_grad1_mse_scale is not None) else 1 self.f0_max = f0_max if (f0_max is not None) else 1975.5 self.f0_min = f0_min if (f0_min is not None) else 32.70 self.confidence = confidence if (confidence is not None) else False self.threshold = threshold if (threshold is not None) else 0.05 self.use_input_conv = use_input_conv if (use_input_conv is not None) else True self.cent_table_b = torch.Tensor( np.linspace(self.f0_to_cent(torch.Tensor([f0_min]))[0], self.f0_to_cent(torch.Tensor([f0_max]))[0], out_dims)) self.register_buffer("cent_table", self.cent_table_b) # conv in stack _leaky = nn.LeakyReLU() self.stack = nn.Sequential( nn.Conv1d(input_channel, n_chans, 3, 1, 1), nn.GroupNorm(4, n_chans), _leaky, nn.Conv1d(n_chans, n_chans, 3, 1, 1)) # transformer self.decoder = PCmer( num_layers=n_layers, num_heads=8, dim_model=n_chans, dim_keys=n_chans, dim_values=n_chans, residual_dropout=0.1, attention_dropout=0.1) self.norm = nn.LayerNorm(n_chans) # out self.n_out = out_dims self.dense_out = weight_norm( nn.Linear(n_chans, self.n_out)) def forward(self, mel, infer=True, gt_f0=None, return_hz_f0=False, cdecoder = "local_argmax"): """ input: B x n_frames x n_unit return: dict of B x n_frames x feat """ if cdecoder == "argmax": self.cdecoder = self.cents_decoder elif cdecoder == "local_argmax": self.cdecoder = self.cents_local_decoder if self.use_input_conv: x = self.stack(mel.transpose(1, 2)).transpose(1, 2) else: x = mel x = self.decoder(x) x = self.norm(x) x = self.dense_out(x) # [B,N,D] x = torch.sigmoid(x) if not infer: gt_cent_f0 = self.f0_to_cent(gt_f0) # mel f0 #[B,N,1] gt_cent_f0 = self.gaussian_blurred_cent(gt_cent_f0) # #[B,N,out_dim] loss_all = self.loss_mse_scale * F.binary_cross_entropy(x, gt_cent_f0) # bce loss # l2 regularization if self.loss_l2_regularization: loss_all = loss_all + l2_regularization(model=self, l2_alpha=self.loss_l2_regularization_scale) x = loss_all if infer: x = self.cdecoder(x) x = self.cent_to_f0(x) if not return_hz_f0: x = (1 + x / 700).log() return x def cents_decoder(self, y, mask=True): B, N, _ = y.size() ci = self.cent_table[None, None, :].expand(B, N, -1) rtn = torch.sum(ci * y, dim=-1, keepdim=True) / torch.sum(y, dim=-1, keepdim=True) # cents: [B,N,1] if mask: confident = torch.max(y, dim=-1, keepdim=True)[0] confident_mask = torch.ones_like(confident) confident_mask[confident <= self.threshold] = float("-INF") rtn = rtn * confident_mask if self.confidence: return rtn, confident else: return rtn def cents_local_decoder(self, y, mask=True): B, N, _ = y.size() ci = self.cent_table[None, None, :].expand(B, N, -1) confident, max_index = torch.max(y, dim=-1, keepdim=True) local_argmax_index = torch.arange(0,9).to(max_index.device) + (max_index - 4) local_argmax_index[local_argmax_index<0] = 0 local_argmax_index[local_argmax_index>=self.n_out] = self.n_out - 1 ci_l = torch.gather(ci,-1,local_argmax_index) y_l = torch.gather(y,-1,local_argmax_index) rtn = torch.sum(ci_l * y_l, dim=-1, keepdim=True) / torch.sum(y_l, dim=-1, keepdim=True) # cents: [B,N,1] if mask: confident_mask = torch.ones_like(confident) confident_mask[confident <= self.threshold] = float("-INF") rtn = rtn * confident_mask if self.confidence: return rtn, confident else: return rtn def cent_to_f0(self, cent): return 10. * 2 ** (cent / 1200.) def f0_to_cent(self, f0): return 1200. * torch.log2(f0 / 10.) def gaussian_blurred_cent(self, cents): # cents: [B,N,1] mask = (cents > 0.1) & (cents < (1200. * np.log2(self.f0_max / 10.))) B, N, _ = cents.size() ci = self.cent_table[None, None, :].expand(B, N, -1) return torch.exp(-torch.square(ci - cents) / 1250) * mask.float() class FCPEInfer: def __init__(self, model_path, device=None, dtype=torch.float32): if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device ckpt = torch.load(model_path, map_location=torch.device(self.device)) self.args = DotDict(ckpt["config"]) self.dtype = dtype model = FCPE( input_channel=self.args.model.input_channel, out_dims=self.args.model.out_dims, n_layers=self.args.model.n_layers, n_chans=self.args.model.n_chans, use_siren=self.args.model.use_siren, use_full=self.args.model.use_full, loss_mse_scale=self.args.loss.loss_mse_scale, loss_l2_regularization=self.args.loss.loss_l2_regularization, loss_l2_regularization_scale=self.args.loss.loss_l2_regularization_scale, loss_grad1_mse=self.args.loss.loss_grad1_mse, loss_grad1_mse_scale=self.args.loss.loss_grad1_mse_scale, f0_max=self.args.model.f0_max, f0_min=self.args.model.f0_min, confidence=self.args.model.confidence, ) model.to(self.device).to(self.dtype) model.load_state_dict(ckpt['model']) model.eval() self.model = model self.wav2mel = Wav2Mel(self.args, dtype=self.dtype, device=self.device) @torch.no_grad() def __call__(self, audio, sr, threshold=0.05): self.model.threshold = threshold audio = audio[None,:] mel = self.wav2mel(audio=audio, sample_rate=sr).to(self.dtype) f0 = self.model(mel=mel, infer=True, return_hz_f0=True) return f0 class Wav2Mel: def __init__(self, args, device=None, dtype=torch.float32): # self.args = args self.sampling_rate = args.mel.sampling_rate self.hop_size = args.mel.hop_size if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device self.dtype = dtype self.stft = STFT( args.mel.sampling_rate, args.mel.num_mels, args.mel.n_fft, args.mel.win_size, args.mel.hop_size, args.mel.fmin, args.mel.fmax ) self.resample_kernel = {} def extract_nvstft(self, audio, keyshift=0, train=False): mel = self.stft.get_mel(audio, keyshift=keyshift, train=train).transpose(1, 2) # B, n_frames, bins return mel def extract_mel(self, audio, sample_rate, keyshift=0, train=False): audio = audio.to(self.dtype).to(self.device) # resample if sample_rate == self.sampling_rate: audio_res = audio else: key_str = str(sample_rate) if key_str not in self.resample_kernel: self.resample_kernel[key_str] = Resample(sample_rate, self.sampling_rate, lowpass_filter_width=128) self.resample_kernel[key_str] = self.resample_kernel[key_str].to(self.dtype).to(self.device) audio_res = self.resample_kernel[key_str](audio) # extract mel = self.extract_nvstft(audio_res, keyshift=keyshift, train=train) # B, n_frames, bins n_frames = int(audio.shape[1] // self.hop_size) + 1 if n_frames > int(mel.shape[1]): mel = torch.cat((mel, mel[:, -1:, :]), 1) if n_frames < int(mel.shape[1]): mel = mel[:, :n_frames, :] return mel def __call__(self, audio, sample_rate, keyshift=0, train=False): return self.extract_mel(audio, sample_rate, keyshift=keyshift, train=train) class DotDict(dict): def __getattr__(*args): val = dict.get(*args) return DotDict(val) if type(val) is dict else val __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ so-vits-svc-npu/modules/F0Predictor/fcpe/nvSTFT.py000066400000000000000000000125521456161405400222720ustar00rootroot00000000000000import os import librosa import numpy as np import soundfile as sf import mindtorch.torch as torch import mindtorch.torch.nn.functional as F import mindtorch.torch.utils.data from librosa.filters import mel as librosa_mel_fn os.environ["LRU_CACHE_CAPACITY"] = "3" def load_wav_to_torch(full_path, target_sr=None, return_empty_on_exception=False): sampling_rate = None try: data, sampling_rate = sf.read(full_path, always_2d=True)# than soundfile. except Exception as ex: print(f"'{full_path}' failed to load.\nException:") print(ex) if return_empty_on_exception: return [], sampling_rate or target_sr or 48000 else: raise Exception(ex) if len(data.shape) > 1: data = data[:, 0] assert len(data) > 2# check duration of audio file is > 2 samples (because otherwise the slice operation was on the wrong dimension) if np.issubdtype(data.dtype, np.integer): # if audio data is type int max_mag = -np.iinfo(data.dtype).min # maximum magnitude = min possible value of intXX else: # if audio data is type fp32 max_mag = max(np.amax(data), -np.amin(data)) max_mag = (2**31)+1 if max_mag > (2**15) else ((2**15)+1 if max_mag > 1.01 else 1.0) # data should be either 16-bit INT, 32-bit INT or [-1 to 1] float32 data = torch.FloatTensor(data.astype(np.float32))/max_mag if (torch.isinf(data) | torch.isnan(data)).any() and return_empty_on_exception:# resample will crash with inf/NaN inputs. return_empty_on_exception will return empty arr instead of except return [], sampling_rate or target_sr or 48000 if target_sr is not None and sampling_rate != target_sr: data = torch.from_numpy(librosa.core.resample(data.numpy(), orig_sr=sampling_rate, target_sr=target_sr)) sampling_rate = target_sr return data, sampling_rate def dynamic_range_compression(x, C=1, clip_val=1e-5): return np.log(np.clip(x, a_min=clip_val, a_max=None) * C) def dynamic_range_decompression(x, C=1): return np.exp(x) / C def dynamic_range_compression_torch(x, C=1, clip_val=1e-5): return torch.log(torch.clamp(x, min=clip_val) * C) def dynamic_range_decompression_torch(x, C=1): return torch.exp(x) / C class STFT(): def __init__(self, sr=22050, n_mels=80, n_fft=1024, win_size=1024, hop_length=256, fmin=20, fmax=11025, clip_val=1e-5): self.target_sr = sr self.n_mels = n_mels self.n_fft = n_fft self.win_size = win_size self.hop_length = hop_length self.fmin = fmin self.fmax = fmax self.clip_val = clip_val self.mel_basis = {} self.hann_window = {} def get_mel(self, y, keyshift=0, speed=1, center=False, train=False): sampling_rate = self.target_sr n_mels = self.n_mels n_fft = self.n_fft win_size = self.win_size hop_length = self.hop_length fmin = self.fmin fmax = self.fmax clip_val = self.clip_val factor = 2 ** (keyshift / 12) n_fft_new = int(np.round(n_fft * factor)) win_size_new = int(np.round(win_size * factor)) hop_length_new = int(np.round(hop_length * speed)) if not train: mel_basis = self.mel_basis hann_window = self.hann_window else: mel_basis = {} hann_window = {} if torch.min(y) < -1.: print('min value is ', torch.min(y)) if torch.max(y) > 1.: print('max value is ', torch.max(y)) mel_basis_key = str(fmax)+'_'+str(y.device) if mel_basis_key not in mel_basis: mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=n_mels, fmin=fmin, fmax=fmax) mel_basis[mel_basis_key] = torch.from_numpy(mel).float().to(y.device) keyshift_key = str(keyshift)+'_'+str(y.device) if keyshift_key not in hann_window: hann_window[keyshift_key] = torch.hann_window(win_size_new).to(y.device) pad_left = (win_size_new - hop_length_new) //2 pad_right = max((win_size_new- hop_length_new + 1) //2, win_size_new - y.size(-1) - pad_left) if pad_right < y.size(-1): mode = 'reflect' else: mode = 'constant' y = torch.nn.functional.pad(y.unsqueeze(1), (pad_left, pad_right), mode = mode) y = y.squeeze(1) spec = torch.stft(y, n_fft_new, hop_length=hop_length_new, win_length=win_size_new, window=hann_window[keyshift_key], center=center, pad_mode='reflect', normalized=False, onesided=True, return_complex=True) spec = torch.sqrt(spec.real.pow(2) + spec.imag.pow(2) + (1e-9)) if keyshift != 0: size = n_fft // 2 + 1 resize = spec.size(1) if resize < size: spec = F.pad(spec, (0, 0, 0, size-resize)) spec = spec[:, :size, :] * win_size / win_size_new spec = torch.matmul(mel_basis[mel_basis_key], spec) spec = dynamic_range_compression_torch(spec, clip_val=clip_val) return spec def __call__(self, audiopath): audio, sr = load_wav_to_torch(audiopath, target_sr=self.target_sr) spect = self.get_mel(audio.unsqueeze(0)).squeeze(0) return spect stft = STFT() so-vits-svc-npu/modules/F0Predictor/fcpe/pcmer.py000066400000000000000000000331151456161405400222520ustar00rootroot00000000000000import math from functools import partial import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from einops import rearrange, repeat from local_attention import LocalAttention from mindtorch.torch import nn #import fast_transformers.causal_product.causal_product_cuda def softmax_kernel(data, *, projection_matrix, is_query, normalize_data=True, eps=1e-4, device = None): b, h, *_ = data.shape # (batch size, head, length, model_dim) # normalize model dim data_normalizer = (data.shape[-1] ** -0.25) if normalize_data else 1. # what is ration?, projection_matrix.shape[0] --> 266 ratio = (projection_matrix.shape[0] ** -0.5) projection = repeat(projection_matrix, 'j d -> b h j d', b = b, h = h) projection = projection.type_as(data) #data_dash = w^T x data_dash = torch.einsum('...id,...jd->...ij', (data_normalizer * data), projection) # diag_data = D**2 diag_data = data ** 2 diag_data = torch.sum(diag_data, dim=-1) diag_data = (diag_data / 2.0) * (data_normalizer ** 2) diag_data = diag_data.unsqueeze(dim=-1) #print () if is_query: data_dash = ratio * ( torch.exp(data_dash - diag_data - torch.max(data_dash, dim=-1, keepdim=True).values) + eps) else: data_dash = ratio * ( torch.exp(data_dash - diag_data + eps))#- torch.max(data_dash)) + eps) return data_dash.type_as(data) def orthogonal_matrix_chunk(cols, qr_uniform_q = False, device = None): unstructured_block = torch.randn((cols, cols), device = device) q, r = torch.linalg.qr(unstructured_block.cpu(), mode='reduced') q, r = map(lambda t: t.to(device), (q, r)) # proposed by @Parskatt # to make sure Q is uniform https://arxiv.org/pdf/math-ph/0609050.pdf if qr_uniform_q: d = torch.diag(r, 0) q *= d.sign() return q.t() def exists(val): return val is not None def empty(tensor): return tensor.numel() == 0 def default(val, d): return val if exists(val) else d def cast_tuple(val): return (val,) if not isinstance(val, tuple) else val class PCmer(nn.Module): """The encoder that is used in the Transformer model.""" def __init__(self, num_layers, num_heads, dim_model, dim_keys, dim_values, residual_dropout, attention_dropout): super().__init__() self.num_layers = num_layers self.num_heads = num_heads self.dim_model = dim_model self.dim_values = dim_values self.dim_keys = dim_keys self.residual_dropout = residual_dropout self.attention_dropout = attention_dropout self._layers = nn.ModuleList([_EncoderLayer(self) for _ in range(num_layers)]) # METHODS ######################################################################################################## def forward(self, phone, mask=None): # apply all layers to the input for (i, layer) in enumerate(self._layers): phone = layer(phone, mask) # provide the final sequence return phone # ==================================================================================================================== # # CLASS _ E N C O D E R L A Y E R # # ==================================================================================================================== # class _EncoderLayer(nn.Module): """One layer of the encoder. Attributes: attn: (:class:`mha.MultiHeadAttention`): The attention mechanism that is used to read the input sequence. feed_forward (:class:`ffl.FeedForwardLayer`): The feed-forward layer on top of the attention mechanism. """ def __init__(self, parent: PCmer): """Creates a new instance of ``_EncoderLayer``. Args: parent (Encoder): The encoder that the layers is created for. """ super().__init__() self.conformer = ConformerConvModule(parent.dim_model) self.norm = nn.LayerNorm(parent.dim_model) self.dropout = nn.Dropout(parent.residual_dropout) # selfatt -> fastatt: performer! self.attn = SelfAttention(dim = parent.dim_model, heads = parent.num_heads, causal = False) # METHODS ######################################################################################################## def forward(self, phone, mask=None): # compute attention sub-layer phone = phone + (self.attn(self.norm(phone), mask=mask)) phone = phone + (self.conformer(phone)) return phone def calc_same_padding(kernel_size): pad = kernel_size // 2 return (pad, pad - (kernel_size + 1) % 2) # helper classes class Swish(nn.Module): def forward(self, x): return x * x.sigmoid() class Transpose(nn.Module): def __init__(self, dims): super().__init__() assert len(dims) == 2, 'dims must be a tuple of two dimensions' self.dims = dims def forward(self, x): return x.transpose(*self.dims) class GLU(nn.Module): def __init__(self, dim): super().__init__() self.dim = dim def forward(self, x): out, gate = x.chunk(2, dim=self.dim) return out * gate.sigmoid() class DepthWiseConv1d(nn.Module): def __init__(self, chan_in, chan_out, kernel_size, padding): super().__init__() self.padding = padding self.conv = nn.Conv1d(chan_in, chan_out, kernel_size, groups = chan_in) def forward(self, x): x = F.pad(x, self.padding) return self.conv(x) class ConformerConvModule(nn.Module): def __init__( self, dim, causal = False, expansion_factor = 2, kernel_size = 31, dropout = 0.): super().__init__() inner_dim = dim * expansion_factor padding = calc_same_padding(kernel_size) if not causal else (kernel_size - 1, 0) self.net = nn.Sequential( nn.LayerNorm(dim), Transpose((1, 2)), nn.Conv1d(dim, inner_dim * 2, 1), GLU(dim=1), DepthWiseConv1d(inner_dim, inner_dim, kernel_size = kernel_size, padding = padding), #nn.BatchNorm1d(inner_dim) if not causal else nn.Identity(), Swish(), nn.Conv1d(inner_dim, dim, 1), Transpose((1, 2)), nn.Dropout(dropout) ) def forward(self, x): return self.net(x) def linear_attention(q, k, v): if v is None: #print (k.size(), q.size()) out = torch.einsum('...ed,...nd->...ne', k, q) return out else: k_cumsum = k.sum(dim = -2) #k_cumsum = k.sum(dim = -2) D_inv = 1. / (torch.einsum('...nd,...d->...n', q, k_cumsum.type_as(q)) + 1e-8) context = torch.einsum('...nd,...ne->...de', k, v) #print ("TRUEEE: ", context.size(), q.size(), D_inv.size()) out = torch.einsum('...de,...nd,...n->...ne', context, q, D_inv) return out def gaussian_orthogonal_random_matrix(nb_rows, nb_columns, scaling = 0, qr_uniform_q = False, device = None): nb_full_blocks = int(nb_rows / nb_columns) #print (nb_full_blocks) block_list = [] for _ in range(nb_full_blocks): q = orthogonal_matrix_chunk(nb_columns, qr_uniform_q = qr_uniform_q, device = device) block_list.append(q) # block_list[n] is a orthogonal matrix ... (model_dim * model_dim) #print (block_list[0].size(), torch.einsum('...nd,...nd->...n', block_list[0], torch.roll(block_list[0],1,1))) #print (nb_rows, nb_full_blocks, nb_columns) remaining_rows = nb_rows - nb_full_blocks * nb_columns #print (remaining_rows) if remaining_rows > 0: q = orthogonal_matrix_chunk(nb_columns, qr_uniform_q = qr_uniform_q, device = device) #print (q[:remaining_rows].size()) block_list.append(q[:remaining_rows]) final_matrix = torch.cat(block_list) if scaling == 0: multiplier = torch.randn((nb_rows, nb_columns), device = device).norm(dim = 1) elif scaling == 1: multiplier = math.sqrt((float(nb_columns))) * torch.ones((nb_rows,), device = device) else: raise ValueError(f'Invalid scaling {scaling}') return torch.diag(multiplier) @ final_matrix class FastAttention(nn.Module): def __init__(self, dim_heads, nb_features = None, ortho_scaling = 0, causal = False, generalized_attention = False, kernel_fn = nn.ReLU(), qr_uniform_q = False, no_projection = False): super().__init__() nb_features = default(nb_features, int(dim_heads * math.log(dim_heads))) self.dim_heads = dim_heads self.nb_features = nb_features self.ortho_scaling = ortho_scaling self.create_projection = partial(gaussian_orthogonal_random_matrix, nb_rows = self.nb_features, nb_columns = dim_heads, scaling = ortho_scaling, qr_uniform_q = qr_uniform_q) projection_matrix = self.create_projection() self.register_buffer('projection_matrix', projection_matrix) self.generalized_attention = generalized_attention self.kernel_fn = kernel_fn # if this is turned on, no projection will be used # queries and keys will be softmax-ed as in the original efficient attention paper self.no_projection = no_projection self.causal = causal @torch.no_grad() def redraw_projection_matrix(self): projections = self.create_projection() self.projection_matrix.copy_(projections) del projections def forward(self, q, k, v): device = q.device if self.no_projection: q = q.softmax(dim = -1) k = torch.exp(k) if self.causal else k.softmax(dim = -2) else: create_kernel = partial(softmax_kernel, projection_matrix = self.projection_matrix, device = device) q = create_kernel(q, is_query = True) k = create_kernel(k, is_query = False) attn_fn = linear_attention if not self.causal else self.causal_linear_fn if v is None: out = attn_fn(q, k, None) return out else: out = attn_fn(q, k, v) return out class SelfAttention(nn.Module): def __init__(self, dim, causal = False, heads = 8, dim_head = 64, local_heads = 0, local_window_size = 256, nb_features = None, feature_redraw_interval = 1000, generalized_attention = False, kernel_fn = nn.ReLU(), qr_uniform_q = False, dropout = 0., no_projection = False): super().__init__() assert dim % heads == 0, 'dimension must be divisible by number of heads' dim_head = default(dim_head, dim // heads) inner_dim = dim_head * heads self.fast_attention = FastAttention(dim_head, nb_features, causal = causal, generalized_attention = generalized_attention, kernel_fn = kernel_fn, qr_uniform_q = qr_uniform_q, no_projection = no_projection) self.heads = heads self.global_heads = heads - local_heads self.local_attn = LocalAttention(window_size = local_window_size, causal = causal, autopad = True, dropout = dropout, look_forward = int(not causal), rel_pos_emb_config = (dim_head, local_heads)) if local_heads > 0 else None #print (heads, nb_features, dim_head) #name_embedding = torch.zeros(110, heads, dim_head, dim_head) #self.name_embedding = nn.Parameter(name_embedding, requires_grad=True) self.to_q = nn.Linear(dim, inner_dim) self.to_k = nn.Linear(dim, inner_dim) self.to_v = nn.Linear(dim, inner_dim) self.to_out = nn.Linear(inner_dim, dim) self.dropout = nn.Dropout(dropout) @torch.no_grad() def redraw_projection_matrix(self): self.fast_attention.redraw_projection_matrix() #torch.nn.init.zeros_(self.name_embedding) #print (torch.sum(self.name_embedding)) def forward(self, x, context = None, mask = None, context_mask = None, name=None, inference=False, **kwargs): _, _, _, h, gh = *x.shape, self.heads, self.global_heads cross_attend = exists(context) context = default(context, x) context_mask = default(context_mask, mask) if not cross_attend else context_mask #print (torch.sum(self.name_embedding)) q, k, v = self.to_q(x), self.to_k(context), self.to_v(context) q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = h), (q, k, v)) (q, lq), (k, lk), (v, lv) = map(lambda t: (t[:, :gh], t[:, gh:]), (q, k, v)) attn_outs = [] #print (name) #print (self.name_embedding[name].size()) if not empty(q): if exists(context_mask): global_mask = context_mask[:, None, :, None] v.masked_fill_(~global_mask, 0.) if cross_attend: pass #print (torch.sum(self.name_embedding)) #out = self.fast_attention(q,self.name_embedding[name],None) #print (torch.sum(self.name_embedding[...,-1:])) else: out = self.fast_attention(q, k, v) attn_outs.append(out) if not empty(lq): assert not cross_attend, 'local attention is not compatible with cross attention' out = self.local_attn(lq, lk, lv, input_mask = mask) attn_outs.append(out) out = torch.cat(attn_outs, dim = 1) out = rearrange(out, 'b h n d -> b n (h d)') out = self.to_out(out) return self.dropout(out)so-vits-svc-npu/modules/F0Predictor/rmvpe/000077500000000000000000000000001456161405400210035ustar00rootroot00000000000000so-vits-svc-npu/modules/F0Predictor/rmvpe/__init__.py000066400000000000000000000004331456161405400231140ustar00rootroot00000000000000from .constants import * # noqa: F403 from .inference import RMVPE # noqa: F401 from .model import E2E, E2E0 # noqa: F401 from .spec import MelSpectrogram # noqa: F401 from .utils import ( # noqa: F401 cycle, summary, to_local_average_cents, to_viterbi_cents, ) so-vits-svc-npu/modules/F0Predictor/rmvpe/constants.py000066400000000000000000000002131456161405400233650ustar00rootroot00000000000000SAMPLE_RATE = 16000 N_CLASS = 360 N_MELS = 128 MEL_FMIN = 30 MEL_FMAX = SAMPLE_RATE // 2 WINDOW_LENGTH = 1024 CONST = 1997.3794084376191 so-vits-svc-npu/modules/F0Predictor/rmvpe/deepunet.py000066400000000000000000000164421456161405400231750ustar00rootroot00000000000000import mindtorch.torch as torch import mindtorch.torch.nn as nn from .constants import N_MELS class ConvBlockRes(nn.Module): def __init__(self, in_channels, out_channels, momentum=0.01): super(ConvBlockRes, self).__init__() self.conv = nn.Sequential( nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False), nn.BatchNorm2d(out_channels, momentum=momentum), nn.ReLU(), nn.Conv2d(in_channels=out_channels, out_channels=out_channels, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False), nn.BatchNorm2d(out_channels, momentum=momentum), nn.ReLU(), ) if in_channels != out_channels: self.shortcut = nn.Conv2d(in_channels, out_channels, (1, 1)) self.is_shortcut = True else: self.is_shortcut = False def forward(self, x): if self.is_shortcut: return self.conv(x) + self.shortcut(x) else: return self.conv(x) + x class ResEncoderBlock(nn.Module): def __init__(self, in_channels, out_channels, kernel_size, n_blocks=1, momentum=0.01): super(ResEncoderBlock, self).__init__() self.n_blocks = n_blocks self.conv = nn.ModuleList() self.conv.append(ConvBlockRes(in_channels, out_channels, momentum)) for i in range(n_blocks - 1): self.conv.append(ConvBlockRes(out_channels, out_channels, momentum)) self.kernel_size = kernel_size if self.kernel_size is not None: self.pool = nn.AvgPool2d(kernel_size=kernel_size) def forward(self, x): for i in range(self.n_blocks): x = self.conv[i](x) if self.kernel_size is not None: return x, self.pool(x) else: return x class ResDecoderBlock(nn.Module): def __init__(self, in_channels, out_channels, stride, n_blocks=1, momentum=0.01): super(ResDecoderBlock, self).__init__() out_padding = (0, 1) if stride == (1, 2) else (1, 1) self.n_blocks = n_blocks self.conv1 = nn.Sequential( nn.ConvTranspose2d(in_channels=in_channels, out_channels=out_channels, kernel_size=(3, 3), stride=stride, padding=(1, 1), output_padding=out_padding, bias=False), nn.BatchNorm2d(out_channels, momentum=momentum), nn.ReLU(), ) self.conv2 = nn.ModuleList() self.conv2.append(ConvBlockRes(out_channels * 2, out_channels, momentum)) for i in range(n_blocks-1): self.conv2.append(ConvBlockRes(out_channels, out_channels, momentum)) def forward(self, x, concat_tensor): x = self.conv1(x) x = torch.cat((x, concat_tensor), dim=1) for i in range(self.n_blocks): x = self.conv2[i](x) return x class Encoder(nn.Module): def __init__(self, in_channels, in_size, n_encoders, kernel_size, n_blocks, out_channels=16, momentum=0.01): super(Encoder, self).__init__() self.n_encoders = n_encoders self.bn = nn.BatchNorm2d(in_channels, momentum=momentum) self.layers = nn.ModuleList() self.latent_channels = [] for i in range(self.n_encoders): self.layers.append(ResEncoderBlock(in_channels, out_channels, kernel_size, n_blocks, momentum=momentum)) self.latent_channels.append([out_channels, in_size]) in_channels = out_channels out_channels *= 2 in_size //= 2 self.out_size = in_size self.out_channel = out_channels def forward(self, x): concat_tensors = [] x = self.bn(x) for i in range(self.n_encoders): _, x = self.layers[i](x) concat_tensors.append(_) return x, concat_tensors class Intermediate(nn.Module): def __init__(self, in_channels, out_channels, n_inters, n_blocks, momentum=0.01): super(Intermediate, self).__init__() self.n_inters = n_inters self.layers = nn.ModuleList() self.layers.append(ResEncoderBlock(in_channels, out_channels, None, n_blocks, momentum)) for i in range(self.n_inters-1): self.layers.append(ResEncoderBlock(out_channels, out_channels, None, n_blocks, momentum)) def forward(self, x): for i in range(self.n_inters): x = self.layers[i](x) return x class Decoder(nn.Module): def __init__(self, in_channels, n_decoders, stride, n_blocks, momentum=0.01): super(Decoder, self).__init__() self.layers = nn.ModuleList() self.n_decoders = n_decoders for i in range(self.n_decoders): out_channels = in_channels // 2 self.layers.append(ResDecoderBlock(in_channels, out_channels, stride, n_blocks, momentum)) in_channels = out_channels def forward(self, x, concat_tensors): for i in range(self.n_decoders): x = self.layers[i](x, concat_tensors[-1-i]) return x class TimbreFilter(nn.Module): def __init__(self, latent_rep_channels): super(TimbreFilter, self).__init__() self.layers = nn.ModuleList() for latent_rep in latent_rep_channels: self.layers.append(ConvBlockRes(latent_rep[0], latent_rep[0])) def forward(self, x_tensors): out_tensors = [] for i, layer in enumerate(self.layers): out_tensors.append(layer(x_tensors[i])) return out_tensors class DeepUnet(nn.Module): def __init__(self, kernel_size, n_blocks, en_de_layers=5, inter_layers=4, in_channels=1, en_out_channels=16): super(DeepUnet, self).__init__() self.encoder = Encoder(in_channels, N_MELS, en_de_layers, kernel_size, n_blocks, en_out_channels) self.intermediate = Intermediate(self.encoder.out_channel // 2, self.encoder.out_channel, inter_layers, n_blocks) self.tf = TimbreFilter(self.encoder.latent_channels) self.decoder = Decoder(self.encoder.out_channel, en_de_layers, kernel_size, n_blocks) def forward(self, x): x, concat_tensors = self.encoder(x) x = self.intermediate(x) concat_tensors = self.tf(concat_tensors) x = self.decoder(x, concat_tensors) return x class DeepUnet0(nn.Module): def __init__(self, kernel_size, n_blocks, en_de_layers=5, inter_layers=4, in_channels=1, en_out_channels=16): super(DeepUnet0, self).__init__() self.encoder = Encoder(in_channels, N_MELS, en_de_layers, kernel_size, n_blocks, en_out_channels) self.intermediate = Intermediate(self.encoder.out_channel // 2, self.encoder.out_channel, inter_layers, n_blocks) self.tf = TimbreFilter(self.encoder.latent_channels) self.decoder = Decoder(self.encoder.out_channel, en_de_layers, kernel_size, n_blocks) def forward(self, x): x, concat_tensors = self.encoder(x) x = self.intermediate(x) x = self.decoder(x, concat_tensors) return x so-vits-svc-npu/modules/F0Predictor/rmvpe/inference.py000066400000000000000000000047251456161405400233230ustar00rootroot00000000000000import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torchaudio.transforms import Resample from .constants import * # noqa: F403 from .model import E2E0 from .spec import MelSpectrogram from .utils import to_local_average_cents, to_viterbi_cents class RMVPE: def __init__(self, model_path, device=None, dtype = torch.float32, hop_length=160): self.resample_kernel = {} if device is None: self.device = 'cuda' if torch.cuda.is_available() else 'cpu' else: self.device = device model = E2E0(4, 1, (2, 2)) ckpt = torch.load(model_path, map_location=torch.device(self.device)) model.load_state_dict(ckpt['model']) model = model.to(dtype).to(self.device) model.eval() self.model = model self.dtype = dtype self.mel_extractor = MelSpectrogram(N_MELS, SAMPLE_RATE, WINDOW_LENGTH, hop_length, None, MEL_FMIN, MEL_FMAX) # noqa: F405 self.resample_kernel = {} def mel2hidden(self, mel): with torch.no_grad(): n_frames = mel.shape[-1] mel = F.pad(mel, (0, 32 * ((n_frames - 1) // 32 + 1) - n_frames), mode='constant') hidden = self.model(mel) return hidden[:, :n_frames] def decode(self, hidden, thred=0.03, use_viterbi=False): if use_viterbi: cents_pred = to_viterbi_cents(hidden, thred=thred) else: cents_pred = to_local_average_cents(hidden, thred=thred) f0 = torch.Tensor([10 * (2 ** (cent_pred / 1200)) if cent_pred else 0 for cent_pred in cents_pred]).to(self.device) return f0 def infer_from_audio(self, audio, sample_rate=16000, thred=0.05, use_viterbi=False): audio = audio.unsqueeze(0).to(self.dtype).to(self.device) if sample_rate == 16000: audio_res = audio else: key_str = str(sample_rate) if key_str not in self.resample_kernel: self.resample_kernel[key_str] = Resample(sample_rate, 16000, lowpass_filter_width=128) self.resample_kernel[key_str] = self.resample_kernel[key_str].to(self.dtype).to(self.device) audio_res = self.resample_kernel[key_str](audio) mel_extractor = self.mel_extractor.to(self.device) mel = mel_extractor(audio_res, center=True).to(self.dtype) hidden = self.mel2hidden(mel) f0 = self.decode(hidden.squeeze(0), thred=thred, use_viterbi=use_viterbi) return f0 so-vits-svc-npu/modules/F0Predictor/rmvpe/model.py000066400000000000000000000047641456161405400224700ustar00rootroot00000000000000from mindtorch.torch import nn from .constants import * # noqa: F403 from .deepunet import DeepUnet, DeepUnet0 from .seq import BiGRU from .spec import MelSpectrogram class E2E(nn.Module): def __init__(self, hop_length, n_blocks, n_gru, kernel_size, en_de_layers=5, inter_layers=4, in_channels=1, en_out_channels=16): super(E2E, self).__init__() self.mel = MelSpectrogram(N_MELS, SAMPLE_RATE, WINDOW_LENGTH, hop_length, None, MEL_FMIN, MEL_FMAX) # noqa: F405 self.unet = DeepUnet(kernel_size, n_blocks, en_de_layers, inter_layers, in_channels, en_out_channels) self.cnn = nn.Conv2d(en_out_channels, 3, (3, 3), padding=(1, 1)) if n_gru: self.fc = nn.Sequential( BiGRU(3 * N_MELS, 256, n_gru), # noqa: F405 nn.Linear(512, N_CLASS), # noqa: F405 nn.Dropout(0.25), nn.Sigmoid() ) else: self.fc = nn.Sequential( nn.Linear(3 * N_MELS, N_CLASS), # noqa: F405 nn.Dropout(0.25), nn.Sigmoid() ) def forward(self, x): mel = self.mel(x.reshape(-1, x.shape[-1])).transpose(-1, -2).unsqueeze(1) x = self.cnn(self.unet(mel)).transpose(1, 2).flatten(-2) # x = self.fc(x) hidden_vec = 0 if len(self.fc) == 4: for i in range(len(self.fc)): x = self.fc[i](x) if i == 0: hidden_vec = x return hidden_vec, x class E2E0(nn.Module): def __init__(self, n_blocks, n_gru, kernel_size, en_de_layers=5, inter_layers=4, in_channels=1, en_out_channels=16): super(E2E0, self).__init__() self.unet = DeepUnet0(kernel_size, n_blocks, en_de_layers, inter_layers, in_channels, en_out_channels) self.cnn = nn.Conv2d(en_out_channels, 3, (3, 3), padding=(1, 1)) if n_gru: self.fc = nn.Sequential( BiGRU(3 * N_MELS, 256, n_gru), # noqa: F405 nn.Linear(512, N_CLASS), # noqa: F405 nn.Dropout(0.25), nn.Sigmoid() ) else: self.fc = nn.Sequential( nn.Linear(3 * N_MELS, N_CLASS), # noqa: F405 nn.Dropout(0.25), nn.Sigmoid() ) def forward(self, mel): mel = mel.transpose(-1, -2).unsqueeze(1) x = self.cnn(self.unet(mel)).transpose(1, 2).flatten(-2) x = self.fc(x) return x so-vits-svc-npu/modules/F0Predictor/rmvpe/seq.py000066400000000000000000000012221456161405400221420ustar00rootroot00000000000000import mindtorch.torch.nn as nn class BiGRU(nn.Module): def __init__(self, input_features, hidden_features, num_layers): super(BiGRU, self).__init__() self.gru = nn.GRU(input_features, hidden_features, num_layers=num_layers, batch_first=True, bidirectional=True) def forward(self, x): return self.gru(x)[0] class BiLSTM(nn.Module): def __init__(self, input_features, hidden_features, num_layers): super(BiLSTM, self).__init__() self.lstm = nn.LSTM(input_features, hidden_features, num_layers=num_layers, batch_first=True, bidirectional=True) def forward(self, x): return self.lstm(x)[0] so-vits-svc-npu/modules/F0Predictor/rmvpe/spec.py000066400000000000000000000044661456161405400223210ustar00rootroot00000000000000import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from librosa.filters import mel class MelSpectrogram(torch.nn.Module): def __init__( self, n_mel_channels, sampling_rate, win_length, hop_length, n_fft=None, mel_fmin=0, mel_fmax=None, clamp = 1e-5 ): super().__init__() n_fft = win_length if n_fft is None else n_fft self.hann_window = {} mel_basis = mel( sr=sampling_rate, n_fft=n_fft, n_mels=n_mel_channels, fmin=mel_fmin, fmax=mel_fmax, htk=True) mel_basis = torch.from_numpy(mel_basis).float() self.register_buffer("mel_basis", mel_basis) self.n_fft = win_length if n_fft is None else n_fft self.hop_length = hop_length self.win_length = win_length self.sampling_rate = sampling_rate self.n_mel_channels = n_mel_channels self.clamp = clamp def forward(self, audio, keyshift=0, speed=1, center=True): factor = 2 ** (keyshift / 12) n_fft_new = int(np.round(self.n_fft * factor)) win_length_new = int(np.round(self.win_length * factor)) hop_length_new = int(np.round(self.hop_length * speed)) keyshift_key = str(keyshift)+'_'+str(audio.device) if keyshift_key not in self.hann_window: self.hann_window[keyshift_key] = torch.hann_window(win_length_new).to(audio.device) fft = torch.stft( audio, n_fft=n_fft_new, hop_length=hop_length_new, win_length=win_length_new, window=self.hann_window[keyshift_key], center=center, return_complex=True) magnitude = torch.sqrt(fft.real.pow(2) + fft.imag.pow(2)) if keyshift != 0: size = self.n_fft // 2 + 1 resize = magnitude.size(1) if resize < size: magnitude = F.pad(magnitude, (0, 0, 0, size-resize)) magnitude = magnitude[:, :size, :] * self.win_length / win_length_new mel_output = torch.matmul(self.mel_basis, magnitude) log_mel_spec = torch.log(torch.clamp(mel_output, min=self.clamp)) return log_mel_specso-vits-svc-npu/modules/F0Predictor/rmvpe/utils.py000066400000000000000000000072411456161405400225210ustar00rootroot00000000000000import sys from functools import reduce import librosa import numpy as np import mindtorch.torch as torch from mindtorch.torch.nn.modules.module import _addindent from .constants import * # noqa: F403 def cycle(iterable): while True: for item in iterable: yield item def summary(model, file=sys.stdout): def repr(model): # We treat the extra repr like the sub-module, one item per line extra_lines = [] extra_repr = model.extra_repr() # empty string will be split into list [''] if extra_repr: extra_lines = extra_repr.split('\n') child_lines = [] total_params = 0 for key, module in model._modules.items(): mod_str, num_params = repr(module) mod_str = _addindent(mod_str, 2) child_lines.append('(' + key + '): ' + mod_str) total_params += num_params lines = extra_lines + child_lines for name, p in model._parameters.items(): if hasattr(p, 'shape'): total_params += reduce(lambda x, y: x * y, p.shape) main_str = model._get_name() + '(' if lines: # simple one-liner info, which most builtin Modules will use if len(extra_lines) == 1 and not child_lines: main_str += extra_lines[0] else: main_str += '\n ' + '\n '.join(lines) + '\n' main_str += ')' if file is sys.stdout: main_str += ', \033[92m{:,}\033[0m params'.format(total_params) else: main_str += ', {:,} params'.format(total_params) return main_str, total_params string, count = repr(model) if file is not None: if isinstance(file, str): file = open(file, 'w') print(string, file=file) file.flush() return count def to_local_average_cents(salience, center=None, thred=0.05): """ find the weighted average cents near the argmax bin """ if not hasattr(to_local_average_cents, 'cents_mapping'): # the bin number-to-cents mapping to_local_average_cents.cents_mapping = ( 20 * torch.arange(N_CLASS) + CONST).to(salience.device) # noqa: F405 if salience.ndim == 1: if center is None: center = int(torch.argmax(salience)) start = max(0, center - 4) end = min(len(salience), center + 5) salience = salience[start:end] product_sum = torch.sum( salience * to_local_average_cents.cents_mapping[start:end]) weight_sum = torch.sum(salience) return product_sum / weight_sum if torch.max(salience) > thred else 0 if salience.ndim == 2: return torch.Tensor([to_local_average_cents(salience[i, :], None, thred) for i in range(salience.shape[0])]).to(salience.device) raise Exception("label should be either 1d or 2d ndarray") def to_viterbi_cents(salience, thred=0.05): # Create viterbi transition matrix if not hasattr(to_viterbi_cents, 'transition'): xx, yy = torch.meshgrid(range(N_CLASS), range(N_CLASS)) # noqa: F405 transition = torch.maximum(30 - abs(xx - yy), 0) transition = transition / transition.sum(axis=1, keepdims=True) to_viterbi_cents.transition = transition # Convert to probability prob = salience.T prob = prob / prob.sum(axis=0) # Perform viterbi decoding path = librosa.sequence.viterbi(prob.detach().cpu().numpy(), to_viterbi_cents.transition).astype(np.int64) return torch.Tensor([to_local_average_cents(salience[i, :], path[i], thred) for i in range(len(path))]).to(salience.device) so-vits-svc-npu/modules/__init__.py000066400000000000000000000000001456161405400176300ustar00rootroot00000000000000so-vits-svc-npu/modules/attentions.py000066400000000000000000000336101456161405400202760ustar00rootroot00000000000000import math import mindtorch.torch as torch from mindtorch.torch import nn from mindtorch.torch.nn import functional as F import modules.commons as commons from modules.DSConv import weight_norm_modules from modules.modules import LayerNorm class FFT(nn.Module): def __init__(self, hidden_channels, filter_channels, n_heads, n_layers=1, kernel_size=1, p_dropout=0., proximal_bias=False, proximal_init=True, isflow = False, **kwargs): super().__init__() self.hidden_channels = hidden_channels self.filter_channels = filter_channels self.n_heads = n_heads self.n_layers = n_layers self.kernel_size = kernel_size self.p_dropout = p_dropout self.proximal_bias = proximal_bias self.proximal_init = proximal_init if isflow: cond_layer = torch.nn.Conv1d(kwargs["gin_channels"], 2*hidden_channels*n_layers, 1) self.cond_pre = torch.nn.Conv1d(hidden_channels, 2*hidden_channels, 1) self.cond_layer = weight_norm_modules(cond_layer, name='weight') self.gin_channels = kwargs["gin_channels"] self.drop = nn.Dropout(p_dropout) self.self_attn_layers = nn.ModuleList() self.norm_layers_0 = nn.ModuleList() self.ffn_layers = nn.ModuleList() self.norm_layers_1 = nn.ModuleList() for i in range(self.n_layers): self.self_attn_layers.append( MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, proximal_bias=proximal_bias, proximal_init=proximal_init)) self.norm_layers_0.append(LayerNorm(hidden_channels)) self.ffn_layers.append( FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout, causal=True)) self.norm_layers_1.append(LayerNorm(hidden_channels)) def forward(self, x, x_mask, g = None): """ x: decoder input h: encoder output """ if g is not None: g = self.cond_layer(g) self_attn_mask = commons.subsequent_mask(x_mask.size(2)).to(device=x.device, dtype=x.dtype) x = x * x_mask for i in range(self.n_layers): if g is not None: x = self.cond_pre(x) cond_offset = i * 2 * self.hidden_channels g_l = g[:,cond_offset:cond_offset+2*self.hidden_channels,:] x = commons.fused_add_tanh_sigmoid_multiply( x, g_l, torch.IntTensor([self.hidden_channels])) y = self.self_attn_layers[i](x, x, self_attn_mask) y = self.drop(y) x = self.norm_layers_0[i](x + y) y = self.ffn_layers[i](x, x_mask) y = self.drop(y) x = self.norm_layers_1[i](x + y) x = x * x_mask return x class Encoder(nn.Module): def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., window_size=4, **kwargs): super().__init__() self.hidden_channels = hidden_channels self.filter_channels = filter_channels self.n_heads = n_heads self.n_layers = n_layers self.kernel_size = kernel_size self.p_dropout = p_dropout self.window_size = window_size self.drop = nn.Dropout(p_dropout) self.attn_layers = nn.ModuleList() self.norm_layers_1 = nn.ModuleList() self.ffn_layers = nn.ModuleList() self.norm_layers_2 = nn.ModuleList() for i in range(self.n_layers): self.attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, window_size=window_size)) self.norm_layers_1.append(LayerNorm(hidden_channels)) self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout)) self.norm_layers_2.append(LayerNorm(hidden_channels)) def forward(self, x, x_mask): attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1) x = x * x_mask for i in range(self.n_layers): y = self.attn_layers[i](x, x, attn_mask) y = self.drop(y) x = self.norm_layers_1[i](x + y) y = self.ffn_layers[i](x, x_mask) y = self.drop(y) x = self.norm_layers_2[i](x + y) x = x * x_mask return x class Decoder(nn.Module): def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., proximal_bias=False, proximal_init=True, **kwargs): super().__init__() self.hidden_channels = hidden_channels self.filter_channels = filter_channels self.n_heads = n_heads self.n_layers = n_layers self.kernel_size = kernel_size self.p_dropout = p_dropout self.proximal_bias = proximal_bias self.proximal_init = proximal_init self.drop = nn.Dropout(p_dropout) self.self_attn_layers = nn.ModuleList() self.norm_layers_0 = nn.ModuleList() self.encdec_attn_layers = nn.ModuleList() self.norm_layers_1 = nn.ModuleList() self.ffn_layers = nn.ModuleList() self.norm_layers_2 = nn.ModuleList() for i in range(self.n_layers): self.self_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, proximal_bias=proximal_bias, proximal_init=proximal_init)) self.norm_layers_0.append(LayerNorm(hidden_channels)) self.encdec_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout)) self.norm_layers_1.append(LayerNorm(hidden_channels)) self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout, causal=True)) self.norm_layers_2.append(LayerNorm(hidden_channels)) def forward(self, x, x_mask, h, h_mask): """ x: decoder input h: encoder output """ self_attn_mask = commons.subsequent_mask(x_mask.size(2)).to(device=x.device, dtype=x.dtype) encdec_attn_mask = h_mask.unsqueeze(2) * x_mask.unsqueeze(-1) x = x * x_mask for i in range(self.n_layers): y = self.self_attn_layers[i](x, x, self_attn_mask) y = self.drop(y) x = self.norm_layers_0[i](x + y) y = self.encdec_attn_layers[i](x, h, encdec_attn_mask) y = self.drop(y) x = self.norm_layers_1[i](x + y) y = self.ffn_layers[i](x, x_mask) y = self.drop(y) x = self.norm_layers_2[i](x + y) x = x * x_mask return x class MultiHeadAttention(nn.Module): def __init__(self, channels, out_channels, n_heads, p_dropout=0., window_size=None, heads_share=True, block_length=None, proximal_bias=False, proximal_init=False): super().__init__() assert channels % n_heads == 0 self.channels = channels self.out_channels = out_channels self.n_heads = n_heads self.p_dropout = p_dropout self.window_size = window_size self.heads_share = heads_share self.block_length = block_length self.proximal_bias = proximal_bias self.proximal_init = proximal_init self.attn = None self.k_channels = channels // n_heads self.conv_q = nn.Conv1d(channels, channels, 1) self.conv_k = nn.Conv1d(channels, channels, 1) self.conv_v = nn.Conv1d(channels, channels, 1) self.conv_o = nn.Conv1d(channels, out_channels, 1) self.drop = nn.Dropout(p_dropout) if window_size is not None: n_heads_rel = 1 if heads_share else n_heads rel_stddev = self.k_channels**-0.5 self.emb_rel_k = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev) self.emb_rel_v = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev) nn.init.xavier_uniform_(self.conv_q.weight) nn.init.xavier_uniform_(self.conv_k.weight) nn.init.xavier_uniform_(self.conv_v.weight) if proximal_init: with torch.no_grad(): self.conv_k.weight.copy_(self.conv_q.weight) self.conv_k.bias.copy_(self.conv_q.bias) def forward(self, x, c, attn_mask=None): q = self.conv_q(x) k = self.conv_k(c) v = self.conv_v(c) x, self.attn = self.attention(q, k, v, mask=attn_mask) x = self.conv_o(x) return x def attention(self, query, key, value, mask=None): # reshape [b, d, t] -> [b, n_h, t, d_k] b, d, t_s, t_t = (*key.size(), query.size(2)) query = query.view(b, self.n_heads, self.k_channels, t_t).transpose(2, 3) key = key.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3) value = value.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3) scores = torch.matmul(query / math.sqrt(self.k_channels), key.transpose(-2, -1)) if self.window_size is not None: assert t_s == t_t, "Relative attention is only available for self-attention." key_relative_embeddings = self._get_relative_embeddings(self.emb_rel_k, t_s) rel_logits = self._matmul_with_relative_keys(query /math.sqrt(self.k_channels), key_relative_embeddings) scores_local = self._relative_position_to_absolute_position(rel_logits) scores = scores + scores_local if self.proximal_bias: assert t_s == t_t, "Proximal bias is only available for self-attention." scores = scores + self._attention_bias_proximal(t_s).to(device=scores.device, dtype=scores.dtype) if mask is not None: scores = scores.masked_fill(mask == 0, -1e4) if self.block_length is not None: assert t_s == t_t, "Local attention is only available for self-attention." block_mask = torch.ones_like(scores).triu(-self.block_length).tril(self.block_length) scores = scores.masked_fill(block_mask == 0, -1e4) p_attn = F.softmax(scores, dim=-1) # [b, n_h, t_t, t_s] p_attn = self.drop(p_attn) output = torch.matmul(p_attn, value) if self.window_size is not None: relative_weights = self._absolute_position_to_relative_position(p_attn) value_relative_embeddings = self._get_relative_embeddings(self.emb_rel_v, t_s) output = output + self._matmul_with_relative_values(relative_weights, value_relative_embeddings) output = output.transpose(2, 3).contiguous().view(b, d, t_t) # [b, n_h, t_t, d_k] -> [b, d, t_t] return output, p_attn def _matmul_with_relative_values(self, x, y): """ x: [b, h, l, m] y: [h or 1, m, d] ret: [b, h, l, d] """ ret = torch.matmul(x, y.unsqueeze(0)) return ret def _matmul_with_relative_keys(self, x, y): """ x: [b, h, l, d] y: [h or 1, m, d] ret: [b, h, l, m] """ ret = torch.matmul(x, y.unsqueeze(0).transpose(-2, -1)) return ret def _get_relative_embeddings(self, relative_embeddings, length): 2 * self.window_size + 1 # Pad first before slice to avoid using cond ops. pad_length = max(length - (self.window_size + 1), 0) slice_start_position = max((self.window_size + 1) - length, 0) slice_end_position = slice_start_position + 2 * length - 1 if pad_length > 0: padded_relative_embeddings = F.pad( relative_embeddings, commons.convert_pad_shape([[0, 0], [pad_length, pad_length], [0, 0]])) else: padded_relative_embeddings = relative_embeddings used_relative_embeddings = padded_relative_embeddings[:,slice_start_position:slice_end_position] return used_relative_embeddings def _relative_position_to_absolute_position(self, x): """ x: [b, h, l, 2*l-1] ret: [b, h, l, l] """ batch, heads, length, _ = x.size() # Concat columns of pad to shift from relative to absolute indexing. x = F.pad(x, commons.convert_pad_shape([[0,0],[0,0],[0,0],[0,1]])) # Concat extra elements so to add up to shape (len+1, 2*len-1). x_flat = x.view([batch, heads, length * 2 * length]) x_flat = F.pad(x_flat, commons.convert_pad_shape([[0,0],[0,0],[0,length-1]])) # Reshape and slice out the padded elements. x_final = x_flat.view([batch, heads, length+1, 2*length-1])[:, :, :length, length-1:] return x_final def _absolute_position_to_relative_position(self, x): """ x: [b, h, l, l] ret: [b, h, l, 2*l-1] """ batch, heads, length, _ = x.size() # padd along column x = F.pad(x, commons.convert_pad_shape([[0, 0], [0, 0], [0, 0], [0, length-1]])) x_flat = x.view([batch, heads, length**2 + length*(length -1)]) # add 0's in the beginning that will skew the elements after reshape x_flat = F.pad(x_flat, commons.convert_pad_shape([[0, 0], [0, 0], [length, 0]])) x_final = x_flat.view([batch, heads, length, 2*length])[:,:,:,1:] return x_final def _attention_bias_proximal(self, length): """Bias for self-attention to encourage attention to close positions. Args: length: an integer scalar. Returns: a Tensor with shape [1, 1, length, length] """ r = torch.arange(length, dtype=torch.float32) diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1) return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0) class FFN(nn.Module): def __init__(self, in_channels, out_channels, filter_channels, kernel_size, p_dropout=0., activation=None, causal=False): super().__init__() self.in_channels = in_channels self.out_channels = out_channels self.filter_channels = filter_channels self.kernel_size = kernel_size self.p_dropout = p_dropout self.activation = activation self.causal = causal if causal: self.padding = self._causal_padding else: self.padding = self._same_padding self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size) self.conv_2 = nn.Conv1d(filter_channels, out_channels, kernel_size) self.drop = nn.Dropout(p_dropout) def forward(self, x, x_mask): x = self.conv_1(self.padding(x * x_mask)) if self.activation == "gelu": x = x * torch.sigmoid(1.702 * x) else: x = torch.relu(x) x = self.drop(x) x = self.conv_2(self.padding(x * x_mask)) return x * x_mask def _causal_padding(self, x): if self.kernel_size == 1: return x pad_l = self.kernel_size - 1 pad_r = 0 padding = [[0, 0], [0, 0], [pad_l, pad_r]] x = F.pad(x, commons.convert_pad_shape(padding)) return x def _same_padding(self, x): if self.kernel_size == 1: return x pad_l = (self.kernel_size - 1) // 2 pad_r = self.kernel_size // 2 padding = [[0, 0], [0, 0], [pad_l, pad_r]] x = F.pad(x, commons.convert_pad_shape(padding)) return x so-vits-svc-npu/modules/commons.py000066400000000000000000000131351456161405400175610ustar00rootroot00000000000000import math import mindtorch.torch as torch from mindtorch.torch.nn import functional as F def slice_pitch_segments(x, ids_str, segment_size=4): ret = torch.zeros_like(x[:, :segment_size]) for i in range(x.size(0)): idx_str = ids_str[i] idx_end = idx_str + segment_size ret[i] = x[i, idx_str:idx_end] return ret def rand_slice_segments_with_pitch(x, pitch, x_lengths=None, segment_size=4): b, d, t = x.size() if x_lengths is None: x_lengths = t ids_str_max = x_lengths - segment_size + 1 ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long) ret = slice_segments(x, ids_str, segment_size) ret_pitch = slice_pitch_segments(pitch, ids_str, segment_size) return ret, ret_pitch, ids_str def init_weights(m, mean=0.0, std=0.01): classname = m.__class__.__name__ if "Depthwise_Separable" in classname: m.depth_conv.weight.data.normal_(mean, std) m.point_conv.weight.data.normal_(mean, std) elif classname.find("Conv") != -1: m.weight.data.normal_(mean, std) def get_padding(kernel_size, dilation=1): return int((kernel_size*dilation - dilation)/2) def convert_pad_shape(pad_shape): l = pad_shape[::-1] pad_shape = [item for sublist in l for item in sublist] return pad_shape def intersperse(lst, item): result = [item] * (len(lst) * 2 + 1) result[1::2] = lst return result def kl_divergence(m_p, logs_p, m_q, logs_q): """KL(P||Q)""" kl = (logs_q - logs_p) - 0.5 kl += 0.5 * (torch.exp(2. * logs_p) + ((m_p - m_q)**2)) * torch.exp(-2. * logs_q) return kl def rand_gumbel(shape): """Sample from the Gumbel distribution, protect from overflows.""" uniform_samples = torch.rand(shape) * 0.99998 + 0.00001 return -torch.log(-torch.log(uniform_samples)) def rand_gumbel_like(x): g = rand_gumbel(x.size()).to(dtype=x.dtype, device=x.device) return g def slice_segments(x, ids_str, segment_size=4): ret = torch.zeros_like(x[:, :, :segment_size]) for i in range(x.size(0)): idx_str = ids_str[i] idx_end = idx_str + segment_size ret[i] = x[i, :, idx_str:idx_end] return ret def rand_slice_segments(x, x_lengths=None, segment_size=4): b, d, t = x.size() if x_lengths is None: x_lengths = t ids_str_max = x_lengths - segment_size + 1 ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long) ret = slice_segments(x, ids_str, segment_size) return ret, ids_str def rand_spec_segments(x, x_lengths=None, segment_size=4): b, d, t = x.size() if x_lengths is None: x_lengths = t ids_str_max = x_lengths - segment_size ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long) ret = slice_segments(x, ids_str, segment_size) return ret, ids_str def get_timing_signal_1d( length, channels, min_timescale=1.0, max_timescale=1.0e4): position = torch.arange(length, dtype=torch.float) num_timescales = channels // 2 log_timescale_increment = ( math.log(float(max_timescale) / float(min_timescale)) / (num_timescales - 1)) inv_timescales = min_timescale * torch.exp( torch.arange(num_timescales, dtype=torch.float) * -log_timescale_increment) scaled_time = position.unsqueeze(0) * inv_timescales.unsqueeze(1) signal = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], 0) signal = F.pad(signal, [0, 0, 0, channels % 2]) signal = signal.view(1, channels, length) return signal def add_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4): b, channels, length = x.size() signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale) return x + signal.to(dtype=x.dtype, device=x.device) def cat_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4, axis=1): b, channels, length = x.size() signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale) return torch.cat([x, signal.to(dtype=x.dtype, device=x.device)], axis) def subsequent_mask(length): mask = torch.tril(torch.ones(length, length)).unsqueeze(0).unsqueeze(0) return mask @torch.jit.script def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels): n_channels_int = n_channels[0] in_act = input_a + input_b t_act = torch.tanh(in_act[:, :n_channels_int, :]) s_act = torch.sigmoid(in_act[:, n_channels_int:, :]) acts = t_act * s_act return acts def shift_1d(x): x = F.pad(x, convert_pad_shape([[0, 0], [0, 0], [1, 0]]))[:, :, :-1] return x def sequence_mask(length, max_length=None): if max_length is None: max_length = length.max() x = torch.arange(max_length, dtype=length.dtype, device=length.device) return x.unsqueeze(0) < length.unsqueeze(1) def generate_path(duration, mask): """ duration: [b, 1, t_x] mask: [b, 1, t_y, t_x] """ b, _, t_y, t_x = mask.shape cum_duration = torch.cumsum(duration, -1) cum_duration_flat = cum_duration.view(b * t_x) path = sequence_mask(cum_duration_flat, t_y).to(mask.dtype) path = path.view(b, t_x, t_y) path = path - F.pad(path, convert_pad_shape([[0, 0], [1, 0], [0, 0]]))[:, :-1] path = path.unsqueeze(1).transpose(2,3) * mask return path def clip_grad_value_(parameters, clip_value, norm_type=2): if isinstance(parameters, torch.Tensor): parameters = [parameters] parameters = list(filter(lambda p: p.grad is not None, parameters)) norm_type = float(norm_type) if clip_value is not None: clip_value = float(clip_value) total_norm = 0 for p in parameters: param_norm = p.grad.data.norm(norm_type) total_norm += param_norm.item() ** norm_type if clip_value is not None: p.grad.data.clamp_(min=-clip_value, max=clip_value) total_norm = total_norm ** (1. / norm_type) return total_norm so-vits-svc-npu/modules/enhancer.py000066400000000000000000000104531456161405400176710ustar00rootroot00000000000000import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torchaudio.transforms import Resample from vdecoder.nsf_hifigan.models import load_model from vdecoder.nsf_hifigan.nvSTFT import STFT class Enhancer: def __init__(self, enhancer_type, enhancer_ckpt, device=None): if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device if enhancer_type == 'nsf-hifigan': self.enhancer = NsfHifiGAN(enhancer_ckpt, device=self.device) else: raise ValueError(f" [x] Unknown enhancer: {enhancer_type}") self.resample_kernel = {} self.enhancer_sample_rate = self.enhancer.sample_rate() self.enhancer_hop_size = self.enhancer.hop_size() def enhance(self, audio, # 1, T sample_rate, f0, # 1, n_frames, 1 hop_size, adaptive_key = 0, silence_front = 0 ): # enhancer start time start_frame = int(silence_front * sample_rate / hop_size) real_silence_front = start_frame * hop_size / sample_rate audio = audio[:, int(np.round(real_silence_front * sample_rate)) : ] f0 = f0[: , start_frame :, :] # adaptive parameters adaptive_factor = 2 ** ( -adaptive_key / 12) adaptive_sample_rate = 100 * int(np.round(self.enhancer_sample_rate / adaptive_factor / 100)) real_factor = self.enhancer_sample_rate / adaptive_sample_rate # resample the ddsp output if sample_rate == adaptive_sample_rate: audio_res = audio else: key_str = str(sample_rate) + str(adaptive_sample_rate) if key_str not in self.resample_kernel: self.resample_kernel[key_str] = Resample(sample_rate, adaptive_sample_rate, lowpass_filter_width = 128).to(self.device) audio_res = self.resample_kernel[key_str](audio) n_frames = int(audio_res.size(-1) // self.enhancer_hop_size + 1) # resample f0 f0_np = f0.squeeze(0).squeeze(-1).cpu().numpy() f0_np *= real_factor time_org = (hop_size / sample_rate) * np.arange(len(f0_np)) / real_factor time_frame = (self.enhancer_hop_size / self.enhancer_sample_rate) * np.arange(n_frames) f0_res = np.interp(time_frame, time_org, f0_np, left=f0_np[0], right=f0_np[-1]) f0_res = torch.from_numpy(f0_res).unsqueeze(0).float().to(self.device) # 1, n_frames # enhance enhanced_audio, enhancer_sample_rate = self.enhancer(audio_res, f0_res) # resample the enhanced output if adaptive_factor != 0: key_str = str(adaptive_sample_rate) + str(enhancer_sample_rate) if key_str not in self.resample_kernel: self.resample_kernel[key_str] = Resample(adaptive_sample_rate, enhancer_sample_rate, lowpass_filter_width = 128).to(self.device) enhanced_audio = self.resample_kernel[key_str](enhanced_audio) # pad the silence frames if start_frame > 0: enhanced_audio = F.pad(enhanced_audio, (int(np.round(enhancer_sample_rate * real_silence_front)), 0)) return enhanced_audio, enhancer_sample_rate class NsfHifiGAN(torch.nn.Module): def __init__(self, model_path, device=None): super().__init__() if device is None: device = 'cuda' if torch.cuda.is_available() else 'cpu' self.device = device print('| Load HifiGAN: ', model_path) self.model, self.h = load_model(model_path, device=self.device) def sample_rate(self): return self.h.sampling_rate def hop_size(self): return self.h.hop_size def forward(self, audio, f0): stft = STFT( self.h.sampling_rate, self.h.num_mels, self.h.n_fft, self.h.win_size, self.h.hop_size, self.h.fmin, self.h.fmax) with torch.no_grad(): mel = stft.get_mel(audio) enhanced_audio = self.model(mel, f0[:,:mel.size(-1)]).view(-1) return enhanced_audio, self.h.sampling_rateso-vits-svc-npu/modules/losses.py000066400000000000000000000024171456161405400174170ustar00rootroot00000000000000import mindtorch.torch as torch def feature_loss(fmap_r, fmap_g): loss = 0 for dr, dg in zip(fmap_r, fmap_g): for rl, gl in zip(dr, dg): rl = rl.float().detach() gl = gl.float() loss += torch.mean(torch.abs(rl - gl)) return loss * 2 def discriminator_loss(disc_real_outputs, disc_generated_outputs): loss = 0 r_losses = [] g_losses = [] for dr, dg in zip(disc_real_outputs, disc_generated_outputs): dr = dr.float() dg = dg.float() r_loss = torch.mean((1-dr)**2) g_loss = torch.mean(dg**2) loss += (r_loss + g_loss) r_losses.append(r_loss.item()) g_losses.append(g_loss.item()) return loss, r_losses, g_losses def generator_loss(disc_outputs): loss = 0 gen_losses = [] for dg in disc_outputs: dg = dg.float() l = torch.mean((1-dg)**2) gen_losses.append(l) loss += l return loss, gen_losses def kl_loss(z_p, logs_q, m_p, logs_p, z_mask): """ z_p, logs_q: [b, h, t_t] m_p, logs_p: [b, h, t_t] """ z_p = z_p.float() logs_q = logs_q.float() m_p = m_p.float() logs_p = logs_p.float() z_mask = z_mask.float() #print(logs_p) kl = logs_p - logs_q - 0.5 kl += 0.5 * ((z_p - m_p)**2) * torch.exp(-2. * logs_p) kl = torch.sum(kl * z_mask) l = kl / torch.sum(z_mask) return l so-vits-svc-npu/modules/mel_processing.py000066400000000000000000000052321456161405400211160ustar00rootroot00000000000000import mindtorch.torch as torch import mindtorch.torch.utils.data from librosa.filters import mel as librosa_mel_fn MAX_WAV_VALUE = 32768.0 def dynamic_range_compression_torch(x, C=1, clip_val=1e-5): """ PARAMS ------ C: compression factor """ return torch.log(torch.clamp(x, min=clip_val) * C) def dynamic_range_decompression_torch(x, C=1): """ PARAMS ------ C: compression factor used to compress """ return torch.exp(x) / C def spectral_normalize_torch(magnitudes): output = dynamic_range_compression_torch(magnitudes) return output def spectral_de_normalize_torch(magnitudes): output = dynamic_range_decompression_torch(magnitudes) return output mel_basis = {} hann_window = {} def spectrogram_torch(y, n_fft, sampling_rate, hop_size, win_size, center=False): if torch.min(y) < -1.: print('min value is ', torch.min(y)) if torch.max(y) > 1.: print('max value is ', torch.max(y)) global hann_window dtype_device = str(y.dtype) + '_' + str(y.device) wnsize_dtype_device = str(win_size) + '_' + dtype_device if wnsize_dtype_device not in hann_window: hann_window[wnsize_dtype_device] = torch.hann_window(win_size).to(dtype=y.dtype, device=y.device) y = torch.nn.functional.pad(y.unsqueeze(1), (int((n_fft-hop_size)/2), int((n_fft-hop_size)/2)), mode='reflect') y = y.squeeze(1) y_dtype = y.dtype if y.dtype == torch.bfloat16: y = y.to(torch.float32) spec = torch.stft(y, n_fft, hop_length=hop_size, win_length=win_size, window=hann_window[wnsize_dtype_device], center=center, pad_mode='reflect', normalized=False, onesided=True, return_complex=True) spec = torch.view_as_real(spec).to(y_dtype) spec = torch.sqrt(spec.pow(2).sum(-1) + 1e-6) return spec def spec_to_mel_torch(spec, n_fft, num_mels, sampling_rate, fmin, fmax): global mel_basis dtype_device = str(spec.dtype) + '_' + str(spec.device) fmax_dtype_device = str(fmax) + '_' + dtype_device if fmax_dtype_device not in mel_basis: mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax) mel_basis[fmax_dtype_device] = torch.from_numpy(mel).to(dtype=spec.dtype, device=spec.device) spec = torch.matmul(mel_basis[fmax_dtype_device], spec) spec = spectral_normalize_torch(spec) return spec def mel_spectrogram_torch(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False): spec = spectrogram_torch(y, n_fft, sampling_rate, hop_size, win_size, center) spec = spec_to_mel_torch(spec, n_fft, num_mels, sampling_rate, fmin, fmax) return spec so-vits-svc-npu/modules/modules.py000066400000000000000000000277141456161405400175660ustar00rootroot00000000000000import mindtorch.torch as torch from mindtorch.torch import nn from mindtorch.torch.nn import functional as F import modules.attentions as attentions import modules.commons as commons from modules.commons import get_padding, init_weights from modules.DSConv import ( Depthwise_Separable_Conv1D, remove_weight_norm_modules, weight_norm_modules, ) LRELU_SLOPE = 0.1 Conv1dModel = nn.Conv1d def set_Conv1dModel(use_depthwise_conv): global Conv1dModel Conv1dModel = Depthwise_Separable_Conv1D if use_depthwise_conv else nn.Conv1d class LayerNorm(nn.Module): def __init__(self, channels, eps=1e-5): super().__init__() self.channels = channels self.eps = eps self.gamma = nn.Parameter(torch.ones(channels)) self.beta = nn.Parameter(torch.zeros(channels)) def forward(self, x): x = x.transpose(1, -1) x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps) return x.transpose(1, -1) class ConvReluNorm(nn.Module): def __init__(self, in_channels, hidden_channels, out_channels, kernel_size, n_layers, p_dropout): super().__init__() self.in_channels = in_channels self.hidden_channels = hidden_channels self.out_channels = out_channels self.kernel_size = kernel_size self.n_layers = n_layers self.p_dropout = p_dropout assert n_layers > 1, "Number of layers should be larger than 0." self.conv_layers = nn.ModuleList() self.norm_layers = nn.ModuleList() self.conv_layers.append(Conv1dModel(in_channels, hidden_channels, kernel_size, padding=kernel_size//2)) self.norm_layers.append(LayerNorm(hidden_channels)) self.relu_drop = nn.Sequential( nn.ReLU(), nn.Dropout(p_dropout)) for _ in range(n_layers-1): self.conv_layers.append(Conv1dModel(hidden_channels, hidden_channels, kernel_size, padding=kernel_size//2)) self.norm_layers.append(LayerNorm(hidden_channels)) self.proj = nn.Conv1d(hidden_channels, out_channels, 1) self.proj.weight.data.zero_() self.proj.bias.data.zero_() def forward(self, x, x_mask): x_org = x for i in range(self.n_layers): x = self.conv_layers[i](x * x_mask) x = self.norm_layers[i](x) x = self.relu_drop(x) x = x_org + self.proj(x) return x * x_mask class WN(torch.nn.Module): def __init__(self, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=0, p_dropout=0): super(WN, self).__init__() assert(kernel_size % 2 == 1) self.hidden_channels =hidden_channels self.kernel_size = kernel_size, self.dilation_rate = dilation_rate self.n_layers = n_layers self.gin_channels = gin_channels self.p_dropout = p_dropout self.in_layers = torch.nn.ModuleList() self.res_skip_layers = torch.nn.ModuleList() self.drop = nn.Dropout(p_dropout) if gin_channels != 0: cond_layer = torch.nn.Conv1d(gin_channels, 2*hidden_channels*n_layers, 1) self.cond_layer = weight_norm_modules(cond_layer, name='weight') for i in range(n_layers): dilation = dilation_rate ** i padding = int((kernel_size * dilation - dilation) / 2) in_layer = Conv1dModel(hidden_channels, 2*hidden_channels, kernel_size, dilation=dilation, padding=padding) in_layer = weight_norm_modules(in_layer, name='weight') self.in_layers.append(in_layer) # last one is not necessary if i < n_layers - 1: res_skip_channels = 2 * hidden_channels else: res_skip_channels = hidden_channels res_skip_layer = torch.nn.Conv1d(hidden_channels, res_skip_channels, 1) res_skip_layer = weight_norm_modules(res_skip_layer, name='weight') self.res_skip_layers.append(res_skip_layer) def forward(self, x, x_mask, g=None, **kwargs): output = torch.zeros_like(x) n_channels_tensor = torch.IntTensor([self.hidden_channels]) if g is not None: g = self.cond_layer(g) for i in range(self.n_layers): x_in = self.in_layers[i](x) if g is not None: cond_offset = i * 2 * self.hidden_channels g_l = g[:,cond_offset:cond_offset+2*self.hidden_channels,:] else: g_l = torch.zeros_like(x_in) acts = commons.fused_add_tanh_sigmoid_multiply( x_in, g_l, n_channels_tensor) acts = self.drop(acts) res_skip_acts = self.res_skip_layers[i](acts) if i < self.n_layers - 1: res_acts = res_skip_acts[:,:self.hidden_channels,:] x = (x + res_acts) * x_mask output = output + res_skip_acts[:,self.hidden_channels:,:] else: output = output + res_skip_acts return output * x_mask def remove_weight_norm(self): if self.gin_channels != 0: remove_weight_norm_modules(self.cond_layer) for l in self.in_layers: remove_weight_norm_modules(l) for l in self.res_skip_layers: remove_weight_norm_modules(l) class ResBlock1(torch.nn.Module): def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5)): super(ResBlock1, self).__init__() self.convs1 = nn.ModuleList([ weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))), weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=dilation[2], padding=get_padding(kernel_size, dilation[2]))) ]) self.convs1.apply(init_weights) self.convs2 = nn.ModuleList([ weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))) ]) self.convs2.apply(init_weights) def forward(self, x, x_mask=None): for c1, c2 in zip(self.convs1, self.convs2): xt = F.leaky_relu(x, LRELU_SLOPE) if x_mask is not None: xt = xt * x_mask xt = c1(xt) xt = F.leaky_relu(xt, LRELU_SLOPE) if x_mask is not None: xt = xt * x_mask xt = c2(xt) x = xt + x if x_mask is not None: x = x * x_mask return x def remove_weight_norm(self): for l in self.convs1: remove_weight_norm_modules(l) for l in self.convs2: remove_weight_norm_modules(l) class ResBlock2(torch.nn.Module): def __init__(self, channels, kernel_size=3, dilation=(1, 3)): super(ResBlock2, self).__init__() self.convs = nn.ModuleList([ weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm_modules(Conv1dModel(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))) ]) self.convs.apply(init_weights) def forward(self, x, x_mask=None): for c in self.convs: xt = F.leaky_relu(x, LRELU_SLOPE) if x_mask is not None: xt = xt * x_mask xt = c(xt) x = xt + x if x_mask is not None: x = x * x_mask return x def remove_weight_norm(self): for l in self.convs: remove_weight_norm_modules(l) class Log(nn.Module): def forward(self, x, x_mask, reverse=False, **kwargs): if not reverse: y = torch.log(torch.clamp_min(x, 1e-5)) * x_mask logdet = torch.sum(-y, [1, 2]) return y, logdet else: x = torch.exp(x) * x_mask return x class Flip(nn.Module): def forward(self, x, *args, reverse=False, **kwargs): x = torch.flip(x, [1]) if not reverse: logdet = torch.zeros(x.size(0)).to(dtype=x.dtype, device=x.device) return x, logdet else: return x class ElementwiseAffine(nn.Module): def __init__(self, channels): super().__init__() self.channels = channels self.m = nn.Parameter(torch.zeros(channels,1)) self.logs = nn.Parameter(torch.zeros(channels,1)) def forward(self, x, x_mask, reverse=False, **kwargs): if not reverse: y = self.m + torch.exp(self.logs) * x y = y * x_mask logdet = torch.sum(self.logs * x_mask, [1,2]) return y, logdet else: x = (x - self.m) * torch.exp(-self.logs) * x_mask return x class ResidualCouplingLayer(nn.Module): def __init__(self, channels, hidden_channels, kernel_size, dilation_rate, n_layers, p_dropout=0, gin_channels=0, mean_only=False, wn_sharing_parameter=None ): assert channels % 2 == 0, "channels should be divisible by 2" super().__init__() self.channels = channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.dilation_rate = dilation_rate self.n_layers = n_layers self.half_channels = channels // 2 self.mean_only = mean_only self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1) self.enc = WN(hidden_channels, kernel_size, dilation_rate, n_layers, p_dropout=p_dropout, gin_channels=gin_channels) if wn_sharing_parameter is None else wn_sharing_parameter self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1) self.post.weight.data.zero_() self.post.bias.data.zero_() def forward(self, x, x_mask, g=None, reverse=False): x0, x1 = torch.split(x, [self.half_channels]*2, 1) h = self.pre(x0) * x_mask h = self.enc(h, x_mask, g=g) stats = self.post(h) * x_mask if not self.mean_only: m, logs = torch.split(stats, [self.half_channels]*2, 1) else: m = stats logs = torch.zeros_like(m) if not reverse: x1 = m + x1 * torch.exp(logs) * x_mask x = torch.cat([x0, x1], 1) logdet = torch.sum(logs, [1,2]) return x, logdet else: x1 = (x1 - m) * torch.exp(-logs) * x_mask x = torch.cat([x0, x1], 1) return x class TransformerCouplingLayer(nn.Module): def __init__(self, channels, hidden_channels, kernel_size, n_layers, n_heads, p_dropout=0, filter_channels=0, mean_only=False, wn_sharing_parameter=None, gin_channels = 0 ): assert channels % 2 == 0, "channels should be divisible by 2" super().__init__() self.channels = channels self.hidden_channels = hidden_channels self.kernel_size = kernel_size self.n_layers = n_layers self.half_channels = channels // 2 self.mean_only = mean_only self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1) self.enc = attentions.FFT(hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, isflow = True, gin_channels = gin_channels) if wn_sharing_parameter is None else wn_sharing_parameter self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1) self.post.weight.data.zero_() self.post.bias.data.zero_() def forward(self, x, x_mask, g=None, reverse=False): x0, x1 = torch.split(x, [self.half_channels]*2, 1) h = self.pre(x0) * x_mask h = self.enc(h, x_mask, g=g) stats = self.post(h) * x_mask if not self.mean_only: m, logs = torch.split(stats, [self.half_channels]*2, 1) else: m = stats logs = torch.zeros_like(m) if not reverse: x1 = m + x1 * torch.exp(logs) * x_mask x = torch.cat([x0, x1], 1) logdet = torch.sum(logs, [1,2]) return x, logdet else: x1 = (x1 - m) * torch.exp(-logs) * x_mask x = torch.cat([x0, x1], 1) return x so-vits-svc-npu/preprocess_flist_config.py000066400000000000000000000123121456161405400213450ustar00rootroot00000000000000import argparse import json import os import re import wave from random import shuffle import logger from tqdm import tqdm import diffusion.logger.utils as du import time pattern = re.compile(r'^[\.a-zA-Z0-9_\/]+$') def get_wav_duration(file_path): try: with wave.open(file_path, 'rb') as wav_file: # 获取音频帧数 n_frames = wav_file.getnframes() # 获取采样率 framerate = wav_file.getframerate() # 计算时长（秒） return n_frames / float(framerate) except Exception as e: logger.error(f"Reading {file_path}") raise e if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--train_list", type=str, default="./filelists/train.txt", help="path to train list") parser.add_argument("--val_list", type=str, default="./filelists/val.txt", help="path to val list") parser.add_argument("--source_dir", type=str, default="./dataset/44k", help="path to source dir") parser.add_argument("--speech_encoder", type=str, default="vec768l12", help="choice a speech encoder|'vec768l12','vec256l9','hubertsoft','whisper-ppg','cnhubertlarge','dphubert','whisper-ppg-large','wavlmbase+'") parser.add_argument("--vol_aug", action="store_true", help="Whether to use volume embedding and volume augmentation") parser.add_argument("--tiny", action="store_true", help="Whether to train sovits tiny") args = parser.parse_args() config_template = json.load(open("configs_template/config_tiny_template.json")) if args.tiny else json.load(open("configs_template/config_template.json")) train = [] val = [] idx = 0 spk_dict = {} spk_id = 0 with logger.Progress() as progress: for speaker in progress.track(os.listdir(args.source_dir), description="Processing Speakers"): spk_dict[speaker] = spk_id spk_id += 1 wavs = [] for file_name in os.listdir(os.path.join(args.source_dir, speaker)): if not file_name.endswith("wav"): continue if file_name.startswith("."): continue file_path = "/".join([args.source_dir, speaker, file_name]) if not pattern.match(file_name): logger.warning("Detected non-ASCII file name: " + file_path) if get_wav_duration(file_path) < 0.3: logger.info("Skip too short audio: " + file_path) continue wavs.append(file_path) shuffle(wavs) train += wavs[2:] val += wavs[:2] shuffle(train) shuffle(val) logger.info("Writing " + args.train_list) with open(args.train_list, "w") as f: for fname in progress.track(train, description="Writing train list"): wavpath = fname f.write(wavpath + "\n") logger.info("Writing " + args.val_list) with open(args.val_list, "w") as f: for fname in progress.track(val, description="Writing val list"): wavpath = fname f.write(wavpath + "\n") d_config_template = du.load_config("configs_template/diffusion_template.yaml") d_config_template["model"]["n_spk"] = spk_id d_config_template["data"]["encoder"] = args.speech_encoder d_config_template["spk"] = spk_dict config_template["spk"] = spk_dict config_template["model"]["n_speakers"] = spk_id config_template["model"]["speech_encoder"] = args.speech_encoder if args.speech_encoder == "vec768l12" or args.speech_encoder == "dphubert" or args.speech_encoder == "wavlmbase+": config_template["model"]["ssl_dim"] = config_template["model"]["filter_channels"] = config_template["model"]["gin_channels"] = 768 d_config_template["data"]["encoder_out_channels"] = 768 elif args.speech_encoder == "vec256l9" or args.speech_encoder == 'hubertsoft': config_template["model"]["ssl_dim"] = config_template["model"]["gin_channels"] = 256 d_config_template["data"]["encoder_out_channels"] = 256 elif args.speech_encoder == "whisper-ppg" or args.speech_encoder == 'cnhubertlarge': config_template["model"]["ssl_dim"] = config_template["model"]["filter_channels"] = config_template["model"]["gin_channels"] = 1024 d_config_template["data"]["encoder_out_channels"] = 1024 elif args.speech_encoder == "whisper-ppg-large": config_template["model"]["ssl_dim"] = config_template["model"]["filter_channels"] = config_template["model"]["gin_channels"] = 1280 d_config_template["data"]["encoder_out_channels"] = 1280 if args.vol_aug: config_template["train"]["vol_aug"] = config_template["model"]["vol_embedding"] = True if args.tiny: config_template["model"]["filter_channels"] = 512 logger.info("Writing to configs/config.json") with open("configs/config.json", "w") as f: json.dump(config_template, f, indent=2) logger.info("Writing to configs/diffusion.yaml") du.save_config("configs/diffusion.yaml",d_config_template) so-vits-svc-npu/preprocess_hubert_f0.py000066400000000000000000000170201456161405400205560ustar00rootroot00000000000000import argparse import logging from multiprocessing import Pipe import os import random from concurrent.futures import ProcessPoolExecutor from glob import glob from random import shuffle import librosa import numpy as np import mindtorch.torch as torch import mindtorch.torch.multiprocessing as mp import logger from tqdm import tqdm import diffusion.logger.utils as du import utils from diffusion.vocoder import Vocoder from modules.mel_processing import spectrogram_torch logging.getLogger("numba").setLevel(logging.WARNING) logging.getLogger("matplotlib").setLevel(logging.WARNING) hps = utils.get_hparams_from_file("configs/config.json") dconfig = du.load_config("configs/diffusion.yaml") sampling_rate = hps.data.sampling_rate hop_length = hps.data.hop_length speech_encoder = hps["model"]["speech_encoder"] def process_one(filename, hmodel, f0p, device, diff=False, mel_extractor=None): wav, sr = librosa.load(filename, sr=sampling_rate) audio_norm = torch.FloatTensor(wav) audio_norm = audio_norm.unsqueeze(0) soft_path = filename + ".soft.pt" if not os.path.exists(soft_path): wav16k = librosa.resample(wav, orig_sr=sampling_rate, target_sr=16000) wav16k = torch.from_numpy(wav16k).to(device) c = hmodel.encoder(wav16k) torch.save(c.cpu(), soft_path) f0_path = filename + ".f0.npy" if not os.path.exists(f0_path): f0_predictor = utils.get_f0_predictor(f0p,sampling_rate=sampling_rate, hop_length=hop_length,device=None,threshold=0.05) f0,uv = f0_predictor.compute_f0_uv( wav ) np.save(f0_path, np.asanyarray((f0,uv),dtype=object)) spec_path = filename.replace(".wav", ".spec.pt") if not os.path.exists(spec_path): # Process spectrogram # The following code can't be replaced by torch.FloatTensor(wav) # because load_wav_to_torch return a tensor that need to be normalized if sr != hps.data.sampling_rate: raise ValueError( "{} SR doesn't match target {} SR".format( sr, hps.data.sampling_rate ) ) #audio_norm = audio / hps.data.max_wav_value spec = spectrogram_torch( audio_norm, hps.data.filter_length, hps.data.sampling_rate, hps.data.hop_length, hps.data.win_length, center=False, ) spec = torch.squeeze(spec, 0) torch.save(spec, spec_path) if diff or hps.model.vol_embedding: volume_path = filename + ".vol.npy" volume_extractor = utils.Volume_Extractor(hop_length) if not os.path.exists(volume_path): volume = volume_extractor.extract(audio_norm) np.save(volume_path, volume.to('cpu').numpy()) if diff: mel_path = filename + ".mel.npy" if not os.path.exists(mel_path) and mel_extractor is not None: mel_t = mel_extractor.extract(audio_norm.to(device), sampling_rate) mel = mel_t.squeeze().to('cpu').numpy() np.save(mel_path, mel) aug_mel_path = filename + ".aug_mel.npy" aug_vol_path = filename + ".aug_vol.npy" max_amp = float(torch.max(torch.abs(audio_norm))) + 1e-5 max_shift = min(1, np.log10(1/max_amp)) log10_vol_shift = random.uniform(-1, max_shift) keyshift = random.uniform(-5, 5) if mel_extractor is not None: aug_mel_t = mel_extractor.extract(audio_norm * (10 ** log10_vol_shift), sampling_rate, keyshift = keyshift) aug_mel = aug_mel_t.squeeze().to('cpu').numpy() aug_vol = volume_extractor.extract(audio_norm * (10 ** log10_vol_shift)) if not os.path.exists(aug_mel_path): np.save(aug_mel_path,np.asanyarray((aug_mel,keyshift),dtype=object)) if not os.path.exists(aug_vol_path): np.save(aug_vol_path,aug_vol.to('cpu').numpy()) def process_batch(pipe, file_chunk, f0p, diff=False, mel_extractor=None, device="cpu"): # logger.info("Loading speech encoder for content...") rank = mp.current_process()._identity rank = rank[0] if len(rank) > 0 else 0 if torch.cuda.is_available(): device = torch.device("npu:0") # logger.info(f"Rank {rank} uses device {device}") hmodel = utils.get_speech_encoder(speech_encoder, device=device, log=False) # logger.info(f"Loaded speech encoder for rank {rank}") # for filename in tqdm(file_chunk, position = rank): # process_one(filename, hmodel, f0p, device, diff, mel_extractor) for filename in file_chunk: process_one(filename, hmodel, f0p, device, diff, mel_extractor) pipe.send(1) # logger.info(f"Rank {rank} finished {filename}") def parallel_process(filenames, num_processes, f0p, diff, mel_extractor, device): with logger.Progress() as progress: task = progress.add_task("Preprocessing", total=len(filenames)) with ProcessPoolExecutor(max_workers=num_processes) as executor: tasks = [] pipes = [] for i in range(num_processes): start = int(i * len(filenames) / num_processes) end = int((i + 1) * len(filenames) / num_processes) file_chunk = filenames[start:end] parent_conn, child_conn = Pipe() pipes.append((parent_conn, child_conn)) tasks.append(executor.submit(process_batch, child_conn, file_chunk, f0p, diff, mel_extractor, device=device)) while True: # if all([task.done() for task in tasks]): # break for parent_conn, child_conn in pipes: # if not parent_conn.empty(): parent_conn.recv() # logger.debug("Received message") progress.advance(task) # progress.refresh() if progress.finished: break if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('-d', '--device', type=str, default=None) parser.add_argument( "--in_dir", type=str, default="dataset/44k", help="path to input dir" ) parser.add_argument( '--use_diff',action='store_true', help='Whether to use the diffusion model' ) parser.add_argument( '--f0_predictor', type=str, default="rmvpe", help='Select F0 predictor, can select crepe,pm,dio,harvest,rmvpe,fcpe|default: pm(note: crepe is original F0 using mean filter)' ) parser.add_argument( '--num_processes', type=int, default=1, help='You are advised to set the number of processes to the same as the number of CPU cores' ) args = parser.parse_args() f0p = args.f0_predictor device = args.device if device is None: device = torch.device("npu:0") print(speech_encoder) logger.info("Using device: " + str(device)) logger.info("Using SpeechEncoder: " + speech_encoder) logger.info("Using extractor: " + f0p) logger.info("Using diff Mode: " + str(args.use_diff)) if args.use_diff: print("use_diff") print("Loading Mel Extractor...") mel_extractor = Vocoder(dconfig.vocoder.type, dconfig.vocoder.ckpt, device=device) print("Loaded Mel Extractor.") else: mel_extractor = None filenames = glob(f"{args.in_dir}/*/*.wav", recursive=True) # [:10] shuffle(filenames) mp.set_start_method("spawn", force=True) num_processes = args.num_processes if num_processes == 0: num_processes = os.cpu_count() parallel_process(filenames, num_processes, f0p, args.use_diff, mel_extractor, device) so-vits-svc-npu/pretrain/000077500000000000000000000000001456161405400157055ustar00rootroot00000000000000so-vits-svc-npu/pretrain/__init__.py000066400000000000000000000000001456161405400200040ustar00rootroot00000000000000so-vits-svc-npu/pretrain/meta.py000066400000000000000000000032071456161405400172070ustar00rootroot00000000000000def download_dict(): return { "vec768l12": { "url": "https://ibm.ent.box.com/shared/static/z1wgl1stco8ffooyatzdwsqn2psd9lrr", "output": "./pretrain/checkpoint_best_legacy_500.pt" }, "vec256l9": { "url": "https://ibm.ent.box.com/shared/static/z1wgl1stco8ffooyatzdwsqn2psd9lrr", "output": "./pretrain/checkpoint_best_legacy_500.pt" }, "hubertsoft": { "url": "https://github.com/bshall/hubert/releases/download/v0.1/hubert-soft-0d54a1f4.pt", "output": "./pretrain/hubert-soft-0d54a1f4.pt" }, "whisper-ppg-small": { "url": "https://openaipublic.azureedge.net/main/whisper/models/9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/small.pt", "output": "./pretrain/small.pt" }, "whisper-ppg": { "url": "https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt", "output": "./pretrain/medium.pt" }, "whisper-ppg-large": { "url": "https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt", "output": "./pretrain/large-v2.pt" } } def get_speech_encoder(config_path="configs/config.json"): import json with open(config_path, "r") as f: data = f.read() config = json.loads(data) speech_encoder = config["model"]["speech_encoder"] dict = download_dict() return dict[speech_encoder]["url"], dict[speech_encoder]["output"] so-vits-svc-npu/pretrain/nsf_hifigan/000077500000000000000000000000001456161405400201605ustar00rootroot00000000000000so-vits-svc-npu/pretrain/nsf_hifigan/put_nsf_hifigan_ckpt_here000066400000000000000000000000001456161405400252600ustar00rootroot00000000000000so-vits-svc-npu/pretrain/put_hubert_ckpt_here000066400000000000000000000000001456161405400220230ustar00rootroot00000000000000so-vits-svc-npu/raw/000077500000000000000000000000001456161405400146525ustar00rootroot00000000000000so-vits-svc-npu/raw/put_raw_wav_here000066400000000000000000000000001456161405400201240ustar00rootroot00000000000000so-vits-svc-npu/requirements.txt000066400000000000000000000004611456161405400173460ustar00rootroot00000000000000ffmpeg-python Flask Flask_Cors gradio>=3.7.0 numpy==1.23.5 pyworld scipy==1.10.0 SoundFile==0.12.1 torchcrepe tqdm rich loguru scikit-maad fairseq==0.12.2 librosa==0.9.1 torch==2.1.0 torchaudio==2.1.0 tensorboard tensorboardX transformers edge_tts langdetect pyyaml pynvml faiss-cpu einops local_attentionso-vits-svc-npu/resample.sh000066400000000000000000000003121456161405400162210ustar00rootroot00000000000000for dir in dataset_raw/*/; do target_dir="dataset/44k/${dir#dataset_raw/}" mkdir -p "$target_dir" for f in "${dir}"*.wav; do ffmpeg -i "$f" -ar 44100 "$target_dir/$(basename "$f")" done doneso-vits-svc-npu/shadowdiffusion.png000066400000000000000000017650421456161405400200020ustar00rootroot00000000000000PNG  IHDRA pHYs  iTXtXML:com.adobe.xmp }a %={2eJNNN{NNNR }lS[[&  ь 𗤬>|/l   "AڬO:UO$  SD4# k¢lK#u֭[lݺ 2e_uejj3}||,--ǧT;+׹nݺ{]5B~pT^n_W_˸c)OEy>>>ƍ{/[HAA AAv*?%WSNI]=JLYm׮]2%%{q//ݏW^yzzN mV~ߝU8={|dرR˸LPP_z3pXX[WNNϜ+AADAq_cccaKN:-^O3<<ĉ?(3>|||zi[na9nܸ*8qo)9&&{{>|xvvvQXxa~\  '"Atuu.'&&&_L رcgYpĉ^^^:uj*W.!aaaիrSN}qڵ?*&N:k;SRRy|||rhwT9qDSNEpppdjڵ_ߣ7S -)2¾^f'N#AA#]mA? hڵk^/4}&fTC+\vMj m Ř9%%(kC*gR'((w~pʩ:z+%%0T"dC*  E^3 ߁n¿0͛]~kSNyxxH-/=jӦMVjkk/Y?3A_[9nܸ_zyyU>''T H-{YJ|||*L#Gu]r !AA߯k/ ={d[{RB ~]NJJZnߋ|]-uVww?6ljjQ5jׯ_gYj%?؅FقH}SQ?|uW^}_'.8 7&A ::벁/l}#w׭[/ݥiq~&ShjjKSQ~?|uo  By"A,,,?E[[AAA+:uΏn-甏fTTT~aKAA[ь 𗗚uֲ#G՟e˖錮Z>;;벶O/O9㘟䟶$((g% _fAE}]2UUկ;YTPǴ{k׮r?mp*11$éS|J[>>_̬]{$O+߱!O2k:sĉdرc?.A*SNyoݺ5%_  @D3 _O^ʖܾ>IMM]xnݺ_&Э[?ʕ+_?O9maekkܳu  hFAKJMMXXX޽2SOԩS+͝8qj^Y`aaskkG^z]~]jpرԟr*w2dH׿ Uǎ.Sj%e|||sAA+ь WիWMMMm۾`ffV:u:qℛ6_xO֭:ujٗ?-HYYy_lӦw֭4+ ooO9^^^e111e(->>~ݺumڴqKS{h}||,-- 7#  EDرcW\Tvڹ}Ys_ H{r77|{]5/XxSe[g33gٲeaÆ}%C}ّ~PRA^3 _gPPЖ-[rǏ^OOϯ=&v;vǏ鉇GXX|_??eb5v={%[lԩS={qAA6DA/y<||||}}O8Qrر;v999W\ٱc׮K,óX[[]~=11|濜Thkk?~|AAAٳvNNN?ْCv100{ ˫~"A36    J hAAAAeD4#   ˈhFAAAь    /#AAAA_FD3    fAAAA~    2"AAAAeD4#   ˈhFAAAь    /#AAAA_FD3    fAAAA~    2"AAAAeD4#   ~7 AAAAVU%z    2"AAAAe;?    _bkFAAAь    /#AAAA_FD3    fAAAA~    2"AAAAeD4#   ˈhFAAAь    /#AAAA_FD3    fAAAA~    2"AAAAed#VUB[`&v-./ʚ%=<0}lQKHL=[SnJ3'm{;4rW2vחxS8xWAWA}C3FƚJg. ѨHTŮn/pǒb-fWflV*s@tڣ /q޴8)^%&Ep>F1Dՙ 8 _@`zLM L銵Z̡*Yq $9x Ob'0q$S6> GZy\S#>t Pΰ }n]Qs %ޤN"H5 hH Cw!IG|D hZH4|1g12Kt8؝NkD`2!+Ж'6Yժ`͏#fwDFCb5XmwbpuԇUmga̠f9||gCib0x/0Lc`\ݖ Oz讅3m=ԒElXoFq8Nʗ9JSugu>c0~@\LnMV7fj?o| fWO>4ŇLD4q wMqt؟":ȡ{i.C]aݏ$AA"۬wrYRPq )ۊJ^+V%䛕 @B\~%ݒTc2NrPMSK%`يʣT*Vl?2T~MeT*J@߮ar+;VX,)۰TRTio*sܖRTJ*8u%mMoDJ*)\v *]LV쑥Ҋ+єoOc+J%/N ĕ>VA?'hp~Uޥ|@! H+ >͞ && hwϠӆSgO ?DZ m0mh<VOrxdޏn]#%qoX U:to]gl[mؾLLj(KSh ]p}H:\w2" - N۸XJdP`3al>\\(xoMv»0r7 #l  OsͻE -6^PvgQPt#\dbZSAq=0پ!Xpv)RezBЄ&<6}VD7A~>U'rXΉ\W1b x2B&S,;ڒq0%~^8i"l ndį bJxHLOy}sHb< -nE P4}F G AKZ!k -50Tᱞ}N$r'*Y{Xyײ|)20 NFU'Tmcr=eˑ.*cn=oRޅ{Džud_R2?ǹ40Zt(Ts]'_E;~YD;gѿv7QcuG/m8_6t&,iWӷur}y#Q>z%I ?<*>5A+vMrMzkotW)l|!box>Y {7+ƫ[tɌS6DSsfnJN uί]ZSC( Oan.=o-Gl]qlНf" 4Pa%)xϥ57|Z)lxSJjSIzt0nNvOENYOouܡ-M>caxڽ2ZGd*{*&FV w:oW7gmJd~T3Q)r'׶ܮ繴XT흪G9xhgj߫ޱ.*e_^jX ~Ku͖_h)ysjnxDM~FDOg%pn P}GmO9%ՌsQQCYK%O3vNt:zUv hW(SV\Nm*wFoݸKU쌙ۈ *}ү<겆L^zn~jqϻaٞ+Ts#?^iT׎TvJ2ޱokxլg ܨ#NS,Tz^{Q}2GiE]1cjٯ0phpEC CvzC FJ>nB{Z4XէJE_^4huvɊU*(묙{&2ԴRqR*fWW'owrl_!%r@NiFnX K$*(3JJJd*lX*++`Ybo! h2K@n+-hH %J%Y@5oKPTMTI%H*=nQRZ1vƤ(Q+RIALW,Vr&ݬ͔Jw7VINm'7pY+d/2RIGCi;F0,1! D1[qX ijxp;\:1&0 `0_26|M}/0g.j>Wm {e+jDpg <Ǜh_o_4 t'|d"-<vC/¼l9G/9E:0ۤ0y" et'(M7e}N Cvμ&1gl0 F-=i+qI w6 @h|kƃ7:?Fwe$Vcď#7ff݆f́e [z4bپ}Qòst! uP3p!@H*drT $̉D,޳C x@@VaqY}il^T\ʧ3L~sAJu/2/uO$.`Eq3ݙԱ5-bHqxD8@1BDYL5;' 4i԰ծ&;ع4t>#:ZƼi&~=Rq`r/s3{2}VmkSa[6mC6Դ5.SOrAWӄE<ӭԛ;_&8ˆ^3:Twcd:m]g?/l~J S-Y7Jʺ-,Gi=C܊TX2 *KOnp{c +˛U[#f[↵R w O;s=鋕MzI rj7}2[bsxﷄI|ʵJćU=ڰv䋖~Uotylj|G}.bRU`]G7ӣn[{=i씭W'R֗[ t!]wjq /U]tE]K-6:)h]/iWXrAٵJ'C3{NwA˫N|piԗ0W;lJ:XSE_A<85ԨfLBpNjjdwHH;_wS%DΙV”44 oܠvԉU}v.:_n[VZ+ҏ(׋jv'YR>6jÿM5fF_.|l'ajNr_yQV^w{jY:wn~R:U==Ⱦo?M~Fh/nCi [7:ZIVAIWQzMuotIt}9+uf=q]khow^ҲӪ$mq1Һ1D5F1:61=`G&o8,ߵzF)lj4kXHM|Q@&I7F Y RD6o5-<`Yεn " Aq|-$O Vwdz3hh.f?6<( 9#G =/G HlC h33bϱ`9>Ld]?",^ Nt6`D4viNpy7b%l<+!`B ":=Ře[y1Gg 2JIf@yptwp^s|GYfG3C)6?;?-\yI3MNn7ҵ{z*QMyn-Qu4y-T5|T֤[wF6j:O|W%Ά~捻m3ΐO緎!  S+.^mw mgɫ'1tN{o?6X=RDYCZvfzc5R3 P\bt{+/[7+!曽5jھuDN1'6O( kU7wީڦyȍp[Ĥ-Z/m\½_8nGi PMSܻfQ;E+-crMKZ7 .N(UQ _ gx||t/11|hF'vϵ왡}S5"/`AUZidzd]VUB7s]l vCd&[%;2,-+٬!el.TJ5P3/K{)|/]-dFhϕy7sKժu)aFօHqr֡f[3C5oZ,tA,Z`KԊy*~>Jۻa~LYZTU6ȡԹ 姭kznY 4ƮwmY]τj۟;|"Ov5fE-l0\eϴT,'EbuΧ&>ܣ@W2ũ5c iZ"Q=(4Sd^eыǬz6fjFG],`.U4#o[>w`ܙ):Sۻ_1yt6zC¤!]<`kxK[p/0U9h I,``~ |[1`AG.[[arvg,G(ǍY?-|{gRnXFf>\9T)@Q W..-?%TyH* a2h:Fj4h $-ʃVJQXqeqT:o0n4#%RRĕɿ|HfK˗a&b(#qZ42I SX"e(bLE\,%Sci KJf.T`oV);fJ+G3GU:R) 7ޝcӥf <ӳԖ0re؝40haH_p}[ &ۂR-6?m5͝G[R.MUA~,hMLx9v雦ЅG1? V<5'\D &%]3.<@Xyg{­k2kƸWTy@<Ͱ͔z8ѦhN@A;G#GG(B6fLm'&ʼ!A-hls5+] HCH 2 ěOW8 dq>ItЩj:) F.ǝD-FbDz&P̐;n]ΰ3XlD^Ό 8ܺI:zY],0> Ľ'&!3#ZP_Y* L 2 I^[1g ]0l( #WgHj1b! =jo/Rh҂4z5v af333]_#`8Gܩl%hYL+͡>AC94ГT3ӌ2@mݨS's>X{Ft>7!.&ִ&>%>zp; \^  ctp5NVo=avLh-(.=򙙒S{/zi `(*:/ԴSrEʻ8L'B H0$w3^宕[W(Nnc{UלHo&vG`4/ҥQ$.|ybº=ȬxqѨg=wFÇv,U2P_3B2*%u+fmM;?KnyzmZGN8h}e9OR+r4]&Ec^UR~ֿ0GO]jun^;t^xqihdZzBv=CB,C|^ꮿ͹tHAꡆ nSyVE=XE[I)l|] _CZ>{T iOI:"? sqO“YVߴE*|x' HTyǹ0Uˮav >Uhb#Y)6=ϦZ^i^=`?.cؐ'(x׼ۺS}oV[jiq3u߈c}&q>`Ըȣn/f/6߶2!e8(êGZfǗ 5X"1mx.!1SF\aEuTwXyw$Im-BzW/JN%SBhZzUc9y%#Gbܚav{%>Y XTWӛaso%}ðuB=vm9ɼZO\S5;;]Y-C1C[K_]̰As]# }^{ԤMщ mR }nPziR_d7ȥ_R7C8~^ZG"Q-׸#> PQ}HhVYkFi0\c`ɱ<,~~WJ -?L,U\Ri7R*dzݣ+g .R IѥJ)WJt6ң@mXQRE\3 -˨TBң(+OSL)TBRzvH #F3vwFʣT^Jm皡),T] ,e~:R;4iJKJ~.T\'۟R Xڌ[ϒok=Z7Ufhf]GrVv zDS,_sOnc͜!leCDe!j ]쭂E9වыU;c2h|Wnw2w T'ÆqB#& [;&MqQ*/v#]i0G]7ӱ=Xws|54 g׿>#vW<5jS=\|ݲoZ#ݰjb-Ioܮ&g x=x*Qx}Y)NV̬*',IӔ"?94.VjJJ#S,S1]9B;+[{Ҫ GW7\9 K G%UKדGa?╂4l5&Ҩ A5$ot~Wv-P`lP5Lc 񡃆Vp.1<f~F<' F?kؼ%X(or;`ŒunjY\??CL^>;~A\/z%f X-ݜ 0aFz!nc6٫8&A;s{'#O6jNŵ2pY1[rv[kfLuUr[!+񺲃?4SM47%ԏ}NW0'SUҳ̉[s[rVJ5jxv!Ѧo*ʟrhyR6ĥ;߰h[Mt3tN\1\qfcwz9fWzWku2&<:ڹG^-2ne hTZiƍ3Y2Ծ]ڵr 5}7m҄sj-r8jT9Moܪ_Ӥ-l]=-O˓|9~l*E=ZZ}9ܒi˚i%LmUg;,n~N쀺c{&dytvb.C58%>qiX֞]/uJh~ӯ>8w{ŗ6n_UCFNe9wVm ?$ȫyjDͱx_No4i[mr7wjߌqz2U~Rv⣏} VyjuѮCO)s5 }Q  RP*٧Hf &DR4PT*%=K9p|۱)QTR".Sʷ`ͨ4 mlʲ9,*|snhByc*e*w[*Cr+۸R+$JlXRJUV2 YR$-ܴRTf*Ϙ+5|E1K }vתIJTR"9fJr2LrRJ+dX"%#eC4ct1^QFx9% Pbݯf>};ֵ{b|71?#0/Dm<O!nSC6E;ħ~Ojffb% 'Wml"s`f'>^G7 ^Zo#^ D{sYc0u)5 m5A 7kpX& ^`Rf{73T_ sWȢm|bF6#A wQ/ܽFU@~Jn@d~|?vg-;uع*ق9PӅ$\ @93!G\̄7 c!|ڌu8 2l.28z}dpȕS'3ެ̠6@;N@'ڱGy0eaB'"s)9b|@#;Imom,܉)Uۂ7fiG8 |&T%6 oڍ $`yU-;W̿8HK@PR28p=mǶ> 0٭6_ @]iƌfC4Tfz?X.Oy<-M 0f2 @XȐhflug3|u6[{X>&ט(B⦌{(%TmxK}N[nbYO -Nsh5Žj^c lOb`=dz%Vf2PyP|xglLkazSLk;-ԍ/7a*Z^>mT5{ O?Rl9$i̛.R5KFX7Tj+#Y[Fe--_=wᔰ8ZsLT=7sΩ5Fd=rWӠIC'.?T, ;&|1U+W%{.  .E D=ު$竭:ɯ/o5TK r1cdxzچܯj3r۹_ӼS#>]fA uIYƍΦn8xh7rfB|b5(wVG< Fا};CYW=V dv}jAi/93xFmgU}Dۙ6JYoӚŶ-+=2}u1 XqttĀ79s&D^w6Og>pӶjE 6-[d߽ﳕf:{~\$:gdtWdžX?a\^^Љ:[ͽ&[ޯ ̌][ݸ&㏦+ke|#*^dnZg>OO*Ҹ1,Þ;\xCcەi[o`P,W,=⮶L苌7^W[=aۺZ]ц:/͎Vߨj՞a0,k>P7pkw3bv5cQUhrWOS%+kHZ5uҗu"%dzrW7bܶ5uϵBO)qZo{ʀ*)Ri{VHfP>a܊R *|G **G架HJ?"eՊܸRY)K Ro*&rYeI7#>کT*v)WUaKDWJ}*&Ǖ|)**^U 䑤B4S)bRʱ'dl6dXXQPq"%ꥥU+Qb-ryDDFqLJFDVJiS ŗ_h7G@ ާG4@wa,G\OAOm#i(a:PDnn`F.A x-^z9>&ZקUpr'q !&_M@*d>ɟzw"*qgX@-;E{wUpO FuFSwޘ9'2c8Bvr_P;п3@]|Ag(Ro$@ zq't_gӁ4tO^wT>g}?F݆q3~sLCߦtD@x8QqTa_6!Zj+o/GG@"` xJb; %Lg"lXgb/b 峌1!tl荭l`cd0[e"ϮdL=.twz92"aDx{:( bÁEd]9PI*qǣv̑j`:Y("s.T"HAM`Zo)o@860PL9v=XBnjy;=Kl绣8Ga]9hn/j}5+cE|ON`E|2] P…a;ZSGgԨe4xna+g9h&onsBC[y[%CXF-9δ>x.T<ixK(IT{ b>Sx f[a[q9}+٩Szޝ}VOW(IX.t>T@.m[ մ)I< 13,R_& y|Hf݉.ZGԢɫcM,bao5Tz=VO?rlg SeC eF>=`uه<+058{RA+/vU5,d[vyM7ܩLblŭm[f54nE/ ESS*ޠD>qޞZSe$_pY}n2շ 7VHiQHs]h)|2UKK62磻*#:{^SQol!t~drW1♟b"NzjR@5%߾ٳL}]ug7e dOtovwjU4=s OLTq]χcOJ ]7hj9JN}-d6Tx}ӚtuB7[m/8;*ϰ_`\' tJq+?fh4wH ^O!A[ )pO(F((HQ*Ϩtg^Dj^#ATL>Yڀ7 foiib)]!KR*#`l9.6ܷ3 }n)O_m~.d *J9RJѨlRgW)7Wj'tGkX*\\)\.V.AL'bTHK$%L!d*%DL(?ZPZ)*Q+edJQ׌7YL\4H")T[VR"W.꾺cLD.4EE# di ԜЩ[џ ]:=LHGQAd!Y&Ox햫<+C&(6tk!M`]pĻB`4Ѥ|ۘn9UQ)@q`8ծf#@OA&=s䈙!(fshI. hwp[Zl6?!<tD</~4( Na5 bFIgy$դ/M4ї[A̹evA@\N)Bė@l" 2S03܂P6P6غ\.&SGmG޹c: Fm͇шn; iQ5}6DMJ4وX}-}тf⬤ѾЇ>kX&sBOg  #* 늾gmoazKӔ)dj˗fhݱgܬ"yhĸO#~^`XXy@&,j-ѕY4&( 1yУXUe %]L'1hC+6 ,d|TM^`)W׷G4R%|Qd3?ͬ[ZRkejOojP#&#T~ ob!Y+ho[Uj\\%]CFP#:w79PG$!VȗW^8JIIE݃8VQY]a8e*ͣ\4KN t a6&]{H]+#_"$R^;QoT pJI쳝*Kbxr`]-4_ӂu 5<]BJxkyy^À5,6kxC^8⇾u4x]G+!eo5p ]f_ArUvb/0l :g<;ƴQҘ/*)YW͓#SZvwmah\ͧ-140`lHsYMv>}Ѱ!RbjĶHD#Q?uln޲zYoO5ϥQp]rB4L;NVtuo1U훗h3s/J6Lt=v1޵ͣ ~Z;t|~w`Qm?[FxߺhuHzѝE0Y[uV=Eя>_~FN_N^U<Uh :-mvaG4-w0S٩_bya{viې?w68ug63Rĥn!e'(0EuZ>Ko݇7IBVryMR3S![D^բ-\blzb@7(4#h gyșͲ#߰LT].ȠG5pWMؾ@SG'JaFEU HF2VNѭC r{uC[-5u4:+ޫꡟY>&{}$g(͇'(=mwFL> -kAE>s;ܒxzۢzS|{#dsgwϰ߷XZZ߼FEC;ZT^ggO"S[=s1Ntv lX T1`P7U(ST> ,K0J.a-2~aзSpY~Z{I*i(J W$*|'P )m+Iuv\!z i}ooɕB^Z"eD7o U:ߋU^U!h8*-!|v2#C!ɯT۷CJetbV2/-ʔWfd$%fuȕ;벾RCSi s1kta " fdLكVWHe#8GIt"3!>#xLQCp=Wm[h;8R4afGُHՙɃ0nnrj6iäD̿t#UA<Bg7>5%nX6K^L8 rFvm 2zDt LcH[;ڎL1W1h1ԏp@Nig:!ftoMmn`E5/D_Tsύ)P]ޣ/#{݅-Ϧ5]Ěpf/hBt M11 Exїp{8`##:: ^zYh [Dif4GpAc^"`bFiWNÓ8ς%2ӉX –8@67{cXL&0CU>wITy( +35Msuq g% U/0ww"9_,<5RvVpRQU'cqu^(WTp?JbHΛ)!-eUoSScy-w1g[EWOq!%]3jIbG^w6=vCVݹUO:z2qI񵕷ߺ#Xp~y׿J ^Zg]B?׊}/o}қAI~~;;|e/y,vhaۣoUOzqU3K݌7o;wM&Y7fNW;ilDKrOzmTdϞs/;Xrڧ,$j/$U]6#l@rd`lՒ5}&V]RO=ތ(4Z8;n'O:* \;?-ȞTWu>ܱ2kz"s] zdI_ޫ]Sa.92et(ݒb=PuG_M>}Nw]94Tj5/mP]CN :lI뤑Gߧ{o]t^uQ'SWCf첣ZIُuڷGI^;tKm ^ Y.fNmmj[A}fY'{{QaY' WH5]>7SͮU빝-8"XKͭs7ԥNYU[ؘg$6 w>N oTO{<нE~1DJ*k}m~}ӌ*8tOu*,6;)y^ݢu|Z{!肞 G^et4ɒ}j93] Pw%0< <0;=wkgth0ՙ>@+Z'V{ I)r?@S K5 [`Th WV?Rn2ҽ~T 0?]5ƞȐ/0/3A{ 6RVwfH1psE崇l:Fl R"nvO%]r#35ΡNy|0v5WS'mt(7:vt"[*>1}>\E7\`5& TLS&3wlP|gNܐ`1hpgֵJVDI{h>Z7|mA6Qjզ8{Ÿ.K-̚S&X;ZiXM㺪GM\vw+q>l]i]N/F,'o'B^Pc~:hn~;T_. k$hudFDIڲb;><:A6XX{\Wbgk Ck5%\lmm4\b|dltQϫe7wxeGt1oy_=o 4`aoq}Wy~vM~ϋ,6x i5#+mZ7OK\޻҉睲J9:ewmg-f-8Y0s/c.Ñu]8Nq5,t{r6ָei*svKn鮫Тʬymf7H2{fq/kcc&}r>gwhc+t]UV&GL`~^o,"H悶JW_tkt6ǽ95x7+{: xaqzW5"LowRZzKbYlTq5JeJƥh>IڛȴM≆!X^U[VؓQgC۩ڎl1Xl+$-ix?.WzNJ|{/0%aoiM5Q)_&tk ѷٗSg&5 PȺtcɆJ.S;}zh,I#6 ]Jt’r";fZ] {EC:ۚV3g>-|ҥpjU{}ٽmA*uH}P3ͮUq/Is%P>.i՟P&i/'/}Q |@ % i/|"ie~E߶@+*4Gy+J+~L)QϽ˽Z+MUڬt*]- $2\%“ROmt$TzvV^R:6Ơ}Wc9^~L݊,ɼ7P{Whp*v&3.8Nč0F/޴cm ?paz,Lci41IF J_f.2C« Hå 7Ѣ?%, l|] x&u!#p/:9&q&LYoE#&@ٽ2X F}n?hESzFԏ(2`(Ǽ C.;Rf^#Q [J ZPOL[/N 8ݣ.%}&ǠK4"1A|ф0I\Ӂqp[0KiJ!N^mܪh'@gi"f)fm0@FPֈSYߚLyK"RN\]ϋvP5Y@2ikdr](n4|Qr;Aeϖ{U J ޼3cƀ7ѲY_Vd mWxL ̡ܷwIL FGT%D@3`iC]137|@Rgye:7|I߰^q!cw!i#j&ͣ>e/L4V-"A69F=ћq`ԱTHXL_D~ $#眪CyD=(ョQuwUT7TkءHȦPA%>vus^=ӵza[?=Ʈ J|Q-.͝9rm'xռqti+G[ xeBW{!i:'*Nl{74ъ*WTb*>Y5Lul5-ǨX{vh;ݬڛʵhg1o&|S(Phhs٭جap}{VaMy؀A*QO,u|mu&[9yPuE\^ tނk\cM1.YÞ#\ թs?s_#v~Zph-}aVLlڪq?~;6M)%2|]O'jEBzZi|_V7Y|8.ʿURRjf޵S2,)U,x?CJFr[jRufۙ7nw(RxB6\+]zx=LO/\^ݼԌE;OϏ(E.kSZu}w aKk1Q|bI}[W umQdF]3\-^6• MS-|^?"w1ڻ.V}u"U[~Nv[䮝\,_L6*n./IU}$sSk}v V6QP[֐&[e,hv~:W.~HaQ\GӺJ%־Yˌ445 j{_=3ymjUMOVyl? j8N;|1}4Ly{l, d9]4𬑼ؙy[du3Q.miZ%쀽~G  !pnfufj &6+Ʉw4fhXk78!aSg|eCu9 퇭6#p m.2:d$*{&' f&1oAϬ۰afl?x/~~7<`Jj89 GY79CYveV# QX?hLt|ۋ\QsS}@ď`N/Ml=1M"f&^іBLК127޻'(%2̹.PgM\g%< kyTĀz !ح-vg , ލ! @=% jV6}РOY\ hc& DSzbp|_. Ey攅?37ć qPF%+Q3} "dNnLtr)h.GD3剮}s<%v)5vo*Q֧`hi8i58! g&ZǘL:ƹpofxvPMkGeQtpg310|:>ef!El # PYi0؜M#Fqzsa=uP ȣKy<_VR foʼnG<1 *m&^X&Lƫ <7`CNd 6SvȱK7AJO؛=u=#\3'>Œ ZNjUmI^} jԋx_#1X6 S5L MVyפc¥%c>ui[ۦ{nc=n_^x}Xwڗ.=2O ,d5'K|_Kzן$ܵ2.BJTWէݹ6{ xuzf$RGI~uA3l|Avwy[ZJ4㗟utqZ[]_ߺP{i j]5}J<81IOg԰Fj+UO(jU_l9RUG˺i͊ŸV/zI O_yzMgt+fNߩ,G՛$> bl5.ɸ4Qu[z~zũ`ޯت7>'S x5]e= RÁRRda-VM3Tﵱl XCY x{{|f8C &d?&A8gW4BMH]j>]  VÖH;횙 qGCߓe=0l0 O((% !Lv>io Croy:Ijb?1\p4n(t۳^m} x1wPĊsM)Bۘ5?y HȬ[ۍd & 8W |GʵVqB !#ZV8ӊkƌJsK-7K>6=sƒLy4_}g{M%ԏ b0#+ ϩ6%`/@WPZlƤ욬wW,ɞ| s^35R"@-mǓl_ͳXۮ669W_|ɖϪ~'IA }9sOX,BÁw3܌4ŨizBƧ?|ddؐ]a'ZtnacQ=ξYhbf__◜A 黺t7W=XrEY圍~?cؚ]7_}~o.ٔgw7ӗ[5hSU/O'm2L;wI胉fs;̱,=]q܌@3 ݶ6VWLydpmK$:8aC0֖ﵱ'^1QBL(n 2Bhde_^tLC4|ܲٙ sk1V~r nnW0폺Ҕ:.ŃPߕKvnhdw4 ZOφ:^/ .O=mU'-TY4Z])څ5Q78c}QILeV+BVոódau9p[I,>}k09C»b}תqG؎:tFݽ/<1 3r_/k}te\ٶ=7 A AB@!D!$@; {Nػw}=~CQVժUU+QsmLY{=^Lܓ#˺fތöZӛLg?#]5Kd3YzbʆZh[׹kr:}쳽F|.73{uSMQ.OX"UaaM2/.[1;eXcVxoשI6/.ի_n^C;xC?TۄXSq{Lέrˠʫ]L^̬]T1tgaQe <)7U*b}xln9%7iڧVyK:=xD萪%S 8\NBWydt%C߼==Z0 uǭB xWCv?a+/|^a{8hNC)5s>Ds-aFJY޻d()#,P[8;}Ӥ8+D0@̂q3r#fV.%m(e8o=HMvQD 07tP F)4fJ;{A@il3Fp"#XMS=Z产W+ 6&d3 (7uY 6)+]g=-ؚ@lH!)s䂱 f=*)3as?4C᳔w{q_F}Kvsjaqɕr^]6OO?ڡLξZ$$觌*{Xk<b]:g*}QTtBNkk4t_\Kj|UDu֮6k'\7J /1sZ(n7}^aMk}_mmUn71~Tvoۥj7VȖЕu /Z*XuvR|mqnAg7Pqodҡ*UKuZOoQJͮܦtZj@fzA}wpEcoU-Z잫gL-c;z/Q3z{+0)mP©v{ǵ9=⁆n{ΉK]Z_nbc\EjSH2pr6R-DY:VX~/oN߯F7=;pE;vMvY.<-:w k;9;9|JrV{?頺~Pss|bY% * r)m@e];M-ף~g)uW[KR*CߵLZ6N˾:/?mu>KsbՁOL6`y(6'jg |u^՘RO^Xm0>],+/z8D-M߼!9IqgLKO?|F%!+Rk)Q}Q攽Zu-MKy,7݉= 4)*HL&KIQ3 b Ec_H$+U!P;~qXT)& A?+Ak5tG`4lZ2P+|ܙĤ!QvyOrK?c@@gdfv8FpKmB sm8}-aH?xsZ &nP kW`A.CIoKlD]pX!a:0+t=>cv,O?ݽ4873]ЮpYͧY< l!N\ |9W* UsmK!jeU+Wtc7po?,h!͢Cr&62 ^CZg]_x<狯V녾$P2)'N;hpuP1Ϊk*ǙIѷ6VX3θFp`-v-}dfM;v1nڃަVcErxo}הc+煝̛W׺'(A3]k@^koGp7Fe 7K7tNn3nI9`;(/_~|O_/xMr/vZc˫GYxdG,Y[ƹ;?~FM窒9uc\NmUƫ_D]ըToC!wNAa!ev]w pDgm1~1Rk.N߶,Ktsv~yJCiG|!5ԧuKd_zqn2UK[ՔK)ԘCů|µ>,7wP?*:,Jӽy_ng"KPɪŏ'v3 6rKk;Toe+ oO~1zgVj2VtÁ8GjSӇcKYKi'ʒ|ԛV ,uPK4sזO.5Tŭoj[)Ct ~ekȶ@һ8lNIZRAL /u ˣn`{G>M׺hoJ24̻WӮٛaܢ [Z)IdբgWً+R*4ģo:]C> \9{pa,K#k;~Iw$6{#q2F*Rh?SkBL+X}Rf/0[pXnXqT`-O1+:Rf>>u)p+UhΎ`XVK%!6S"7kS y`@W=tl(0Q^ bVvP"7#`3¶= ·*-}] Lkhc;;@|h{7ěVJ) Ҧ.w~{eSb3J֛#Fl&ԝζS;Nr k@Uv-)?'- ~O_d|9NܢΖtw4LD^_Y˩OT=ԯY mշyZ7&% D>洶FsT&_V,|}'1+؞aW??MGyHS_bo4r~ܕQj;X&sQA9i6щ1CTy_p[hĿtW?ng[4LIGlǒ\w?视UV䘗(s.aBڏxo9!ym;ٴ44jq[vzĻRyK_ZQ/=Lž^ϞB2F|%W{U&FG>׼}2P:O!HSbt(*, pNd|}NT+]vP5J;ˉKz YxNi5qjQc*7Klת v00^xcfx~,U||>WRߩ80{8hVaY=2O魨 Ϙ wIDخ[X\ZGl 'ѹ^0ךCkR Mh;$P|Fҧz< 5l"CEWC''F[]Pz8֝:;!i#NW^biSdQei7Uuw{yq0oU=Siy0DlqKkQt{tV} z{Ccziβ+SF)W_3 TΏ5n׭ᅶEg/)jfܩr8#cX]'^~vmc.Үh!|qsoI`~_HM~3(HZنk)є 5%˂e~4X)!)d|$BQX HR,nDQBO(eD榔Jޤ@uE$VanHNu%Q!߆$j4! gI2'I $= H$ †AD V$m_|u/7҅`ljkTy<>G5*ܾXy9ZF 4 d!4#b3_~u Wɶ+<@(6a2“3-NZzež,gXNmp%U_aWՄ8Fe}G̞ʤ\6Mb)t6N|'3% |jԗ;D> c$YczcL×@+O17PځU*Cp9.( e'&jO7憽Չ*)X:ԋv}/^GEn=ܡ3)PJTQ;wQ# 25KéYˡ席LA؄Wc9x7b N(SU|AL72\:=y /C#3}H4E7ɝ^؆gSp#]S:3-QiQ]$u̍L#ݧ7c1/E c hB1)h7тo]_|ȸHy7y!%vJX"%N2@D\ܽ3^Nƀ50o0Q)ǪPnCADʓJS&<(3uzti?G񉚾_x*@LLu >h?}e}|"xMev_>Ç;d*_tw[[Aty!%m6(7ZݥN3n`悏6PJ0ؔ(5|sO :KEEŮ]lgʗ/_49uԒ%Kg̚O5 EY~={vʔ)s=v67nQ:v(#GǼs|`G|i(#ʮP7ED)cc suN=wOQ뾖^}4V]w򆸥]G`衲K2GV]RT2ᩌf\R?Ȝo-;#V|iT%#Բv%vk d7A,**7L<]4(ǣo;(7&8 mse+ДU<Sζ2/7vyM\ZXYtoGk.yj~lϔͤ2,Iv*8ty˄;Uޭ8X\Nt(}4(Mk zEdheD*t%׶9UR"Έ+3ַs^-XnWVDAY=n01{;cP'~%^<,;):߇Zy{DN㩴djqU~q-U8JEe+2I83ETqvc-v4 \c=hgL79[A# -r1~V=26܊ U+nنVzU9xzզUTT%mhhqh@vehOVRu= jde0F+醧*wKnM>N7-s[. x` ZNA_)NŰHLfS&WÕ$ba&wƉ_, 1 %$MJYa Q3"IY6xd D&)Y5@h5Ēn5R6'%Q QOQ3R̲fnr0]0$eŷI5P&fpQ3$a)bWUŢfddj D?_7)aU$S$4jF`{w5CKi%PoZ3rq" 31'^t~o1 $ Ęt`<h,ˁӘnMG7 >DȒ"AfpCRN#N(ao*@ VnUL >N|N9#ޔ.ė'x0?w3fOao5[B~#f^g,q8A׿/d|<SspSܭHP|͋P/A?ռs)%8r9&Cjo {O35du)w5spxmeteW^⹯qP3cJǾdۯ1ĹXrz%21+}Mؽ9~w-ټ"0z>eV?GLИ~UK𙺦fn ]3GFQ'zz j }?O G:ŗxgk zS@[ 5:RKF `d'u|w+ p(`X;mns&O3i'0ĉ 'E\Mi`Xb1&>iTTT2HLLիW~~?fvܩS6.͛7 ,]3]͛7;wg3]-[N;֐&o7{yUͷw_d-xY~byN.Y7!05mĘ3#g>Lꥣlfh_WSVFw2웞TFb5ўW3eܚŷXInvCFG꿽);@;&.F  fE`|DŽIܯu;|e?E+O>o2 lŴJr-no_zէ%β-RkZY*hmH1Òk|U<3\UOhўIsZ==sPn7o1pso:%N>&>kq佥׋ܬw6(x0l Z.0 {gB{]_yH|i]`ygCvX 䌋}]һsZ?{Z>wq݋/OkWrzh(ayx:%UDC뾊Iu+}X61pFZڸVlOIFgfia {K}aN20+ƒ-}GfZm *U|̨]X'泗>w,2m^!x\4ms]?Zvz_2dӂ^ ű!m0vM~,)}~,xlӢ ɨ5zR%ފ':J12  G{JJǧ|-\7GP.lNga6I*Z y7>'\:{g{p ++iO)k\zS_̷_Ot{`H"13,<8ܑ6i&(.+zlQ^ SM̷^O^OR|Za/Q/MV`C?nY8lu'kPoQowjBy1,,!П~5fȉR^MM)ǐ K5C⻝x$ƿ68 n&%w6١WFH 4PSjX(6~obOHBʈP %"aSQR8$!bP;0SSJl>;f\>b\DaM ~ZжՠtL $pɥ=p(Q7V38Թëǐ1p[e's >MX;0/<( 0I;N "@{SOJS"`1Z〯.a x< <>z3јgȝf | GHVy%ϡY=8n=ov!"].Ϫ ?+ȃOf_C <`L@d>| C2}@e " ҪW|cW"ی J G" 4 ", " f<`EW~X[5~2!5[u:p{o`p=qwxb0P%FKeMJ/;=ڝU8|;co<5lMQ)2upATj `@y`:Z_ ̝6=nDN&kemJO?.rAx}ӱJy"KhLcB^f:mA4-q.A&'̲942jĩbh[p𢋟C2&&8k!{4ИPFN Ƅ8t>.G MvLOo3P*/6Bێ`-8 ?A08G.9G9y| O7o]\Xn]JJ ֭˗/7˜9sMLLzꟵ{%K|k42gϞ?w[j7oWnHkTŖ>;xۢe&[k[yS M.Mja6S.zs7%#Fz5޷ﷂ>]znP9ةUÕ sW4hk> 5K] A@ VlV1U{ V;rҖpP2}K+H 頕!NF?NgSۇf}9Uʴ[ jZ'\7X%5rNs'%֘7˧ܦ:=r~ـ}Fy3~6?W|U,VF'@ac^&fe4d8%P4xļG/d~y+?)<>L7ibau l; i_plj5w\q"Z'nDyLa7IұoTce|ֹ6ԺdgVwN4uF _ݣ)gsi.wxQAIohξ7Ah]#) '~.;̱yJko%ntDsb>/+w{tf[Ѧ 0q;+ЫAv[蹰"u\cQQ3"JsnWZ5=.:PJP/I M1[%4F0M܃ B*T'ni#* gc\o!=3yB_``: NTg6vJVL89-fi N<8ODKdXW _PZ%x@2b ͟HÃ'G|v }'&򀞊޾h`#Hx voG "jS|ގ-d*ݔv5mT|j֠c<@z8p9}݈5`N)悍\vBC ?ȘЭK 8~sc58A2**JU.:o 3ыW{偠kT1yCZ f27:5ÃdRϸE2GWg'͠X;ec8 ZAL2Ѩǘ@9]ۉq_`@i!pKaqeww T=|)00T\LoI_惮GN^ 5m{yRfDfh8<=?dkyny7t\/ZDւ`VyD8>cmdd}യU.Q軪8p˽Q41iw{UztS7WXFh =~ogt-7֭_ +t,ʶ`" =U\93Mvw/ n4&MxNXʘ^ KQQlz +33[i*119s=OOOzxx4= Q?S?|Ixϛu։ +M .K'm󄦟Qb,*//iM֛$6H{u|5KDgoյݻ*3T술5}E^iY?BVRRڭ,v3ŪNᮧwUU_c7g.;5}0NJI_kN(=,=wst0`ء7>FerU 3Qss ~U埽/L|1MJ&JPYv-O}TfJx=ƺU|B[F;i T|VeR_R07&O=G83tm5 eNpZlϲk.Zd޸t3 fTl٭cOIP{~k\{yCi=?n*\%6W $]0`]jVwI5*?L9OIZ?=;Bgm~N$/Sxߏ4za˩WPS_mc5T[zڤQrc>nv }Wc<:ufQ͜;_ݱb_duC:tbΚsCO44[D;l2q me_gsϽ|k(.(foxV%|vGo<\VJSdyBܶm.Gc=;f`ln$zeU)7Tnҭ -Sf0GAOugIR^Cn{5׺$EUč TfqҭWntY'& Mj7Wg gWԺun}cH t{F\_dm+JtU¶*mNcllݙ/'/}*?A٠IGf]9[US[OYc׸[l{쀲^m=֢ߐtsn͘tݡ0}}5].tE8Co3u?kiL (jaue[_k^mWV9ڑm?,+3G, jUjL_s#<ǚ?zvjޯuit+eͭE}st{-wQ琿7߻hUG`F=IPxR!݋j,$&;4A۽,!9)zOJ(`[k$۠&Mr NjI,/!&û+?-C$r=[.HB`_ 7B 0iDD}Ƣ_f$&n Tj]AnuIt"VndE"OdR 2beeTd :4ۡxפ'z/@vڹ+<'33L !?V f90a{ yy tD- <Jo4S=h\4b3  @iGI,8'E#~:?:|愶w<͙CsF ,,̃oZ=!+eFۚ7hXiJPO<4E(W8u X!K`FԖMƢ$Dy:Zp;٘3ıO6/J/‡gyyy?Wr'q5@~~JT^5͛Fyƍ&|4Ajj蟌oc0Ο GGF믌^vkY~m`BE v:2EFwM᪻G6wn\\tbCSP=}^gփ?]Ճt;#VQcK>xwMuj,87DDer;?kpM]߷vbn秅8V2;L3G^z$gV[Տo`ZO..̡*X_mfIÀtz_ȺԶ,wk^<E\W׷aе:^?9[]Q3 5#tox>m繱^U*!olnQ9;\|$afOdT-ilXw!AƵsv`uٳY1yyUv (-U6곫*̨Kkt=ˢş-<}ow36w=Pq=ɴv%vuIJjnaV`<{n|{G'EyyjW%%=P>Z8 d[|znmЋe"m촜X=d5߮v!ZX::.Ufl苜qvYٲ9GuXgQvOW&i 9#vڗ  s{FViA?ZM?uK+fA1=$k͈̳˾ 9 ګӊWmXԺGo}FۄH1\BpR̺Bݲeڢ87'e{ \wR5%}}ۋs#|ꦮvfާOQ*m]K2m\R{gɽyPdUcLGϩ9=OwH*aYEmU~|wܭMyj.jU_ߝ"=f4'd7`o *QvVn'Mwv+CwZ;8T1 I 0@@a"pt0,?r\/ۅ P= f֑ESQhô2 o{~V |`Z3yLż:b"h81:ǏHta 9,b~ pت (Ѝy L58Dj2l,c&E3 )E19P_ &`qu@g= 8·UF W4p6yto*8S-D VyL@Bmsx'$Dxs)U"pg3.W>~KK'>)G%m{1"< Ga$5&QxƒhGX5{ t|VۑLg"-}N*7qp(\ZἚNF`qt@ܐMI%ƠsV83|i>+nI2/ D8.1b#r#w feN\u7(>Ѹ&s`0…GYHX<@N;.G%`3 iNetmY pstuIY`&R98ItHy=w~_m%*:::#Gω'Jl޽{ƅF`iiޚ̣[n4A```œ9s$EE#F4.GEE}w6:Ӷ#)>g-t'VGhǻ6dO\ɀ7ԞX[tl{@ ,UѬ{N%}kbQ5&'^PʹWTȮzeV5y(ҼXawIȊ' /K#0ȭ:u_/Gw?pUӠ={?J34X#я֡cu?)gt`.ԭ =^&s[e*_j4pRA5Ujͮ9?h/e',lF >uXV_S'bw|_2S%/*:q2eׄڝ>ի6롶/n 巺 .V$m7Yܸ,NZk!5uFȮWgv?"<՞{G9jRk|r8\{};wTLUt۟UZ/nnlq5oz䜾ݞlj9XɖZۮ%\ve}K~(c5NB[:֕ϞJ漈},Ĵ"kO&JMtgQj\+;ܪNhZ ՜Mn^y0N^}ț[_kY#8ݒ\/sZY$ZSRDH~U5 %=nd}uVYl|Nj'1n5_SU *7:nPi>VәΧ$wT-i`! Nd䖚&ʕ娮}fx=U[/Wly8hz,5l?G&*7n-M[u7Vfv œE W m[:s[7S゛>Q~v_ߑ_W8tNjZtl; Aފ7}< >BY7crn'o=+9oi质E-ڝ1*,IO]K=4nlkrl~z: 'j?S`ymgk:#Ȧ\:OPo۠:j|=ԃ cR#oJ4WWi"'~I+w4?U~fAnb$'b D(bȈ +F_7ɹ]3%U?nB0 (`),@l? HT'Kl*HARޙ=FSƨS$ {H)C,Q/6R$U+OK3/;3`:G>w@| ̴`;Fq. J8Pi=.'L0y!nXKX10q HʛM3'EX⅏Gca\I@ؘLL%s~6opW擘h6r0׃t1;=`vIB%j,`N3bLz`]://"gZ pYfX|g#08$h]pvBMjTHb b-#'SdZBX:BcpMLQc[e]"DQH3W2i/3,"qiE]M =yk 4jvi>_})o^\stw|P 0 /w1>zT%(J(v J=1h_ :ϖ2  bI 1no|*̊g]7e(WoX-v^ Q/Ǘa_+¹۳;KrnMal Dk]q.c!:@A&Q0r؀)( muJe}&aa8m'#"xXQ7BMof[r5 L.WD$H!T5p7@ bx.xzԟa"c0y12?`X;[ha:]:n/,!->>o߾[~jѢEゥxP#TTTJF3z1uߛKIBRRҿgJEu3uN]oZחP*4x/C)6[aZNt6?Zo?uc>q!=T2 /?qtOÃBgz˥ o+dᦾLuȒW!ltHa֢5U RΞ,f,ۢp^O6 IGöM.Gpor_Kؗ[۱XYF(z ~ٵ٘s*:^~uM! ћKo0X}CŏynywSz}[KujJ+*ߡ-2ѵPX>^ySxa/{wF*it}V1ehMs#͒7.1asYKW͘czߓ~ |_Y|cwUFeD\MR+{^Z :Tv\U+N$9[דR=5X6(Z8!E$7Q{1CtafÍ}{{2Hʰz; )aczU\*I'ao^hҍ[wm1@O4/e4IJW:,T>.,me.{^A+oQpA:č"08]= 5ȞJFã_TZ򩭠NgCGnfR^gzJ >Q;n/tdo^[%yy穽?}`m|(¯w/eJ_ʼcG8LتW|̷n)7/?h?I }vP+U]k;G< %g7Kh|bAnb ?(M^&B)%O X,"A7%d$Ib}7HbHR;IFimWq X %H 3IN&<.Ir_DKl dY S $Q$piZFB܄,BC ^q{a N=k-q=D&d犿"ՋMXS/UQ'ۄR5 ߽[?2%(AeKJaBY-Dޢ~ ;w @@ \^[K+8I؂)dڡc):B47]?ilZ nG /Z>fKҎf򫷶o;0m眊:aTpe x.ɚ2Ok}|2`rd1ϙtQ&9tȀx%X2Okgۿ(my^ D ;ӐO'4p=u B;ύ==A[-ʸ<B-hP|2a/Mg> 2x d>SLX`>!ۖm `3'@ѧHo-c:Jc qg}kӋgiL/ -X a}:f(/ n 33I{.OD@&w->]7p9K2@A.'٧Lބ[cB- m&^(76Mg,kќGA:P*.]Fz߽'gLWxpzp^H;_ בG@[bĪDXL xoU4fn|Aw0pk1K2Ǝ{ԩƼcǎ;v}̙%)???335IDf'55GG?/J3|CCߗ%h7A~~~p=o_6ۤ D^[هzadUUuRjA  }ƈz>"QvuZ^aֳud?j>/2J~Î/o}UP+ޏpjzV35qvfi}'z6di*SGyVڴ]?PzȱWiiFGIU)u`ykɫEaw_vkjvNn:d#45-sp 4iOa箼;sVl=xӤet3eE%3yLkw2a] pfTt7=pgagt?1[y$rPiNvf>/ݮYjd{Ѷ*~]__V4HncZKau}1ziY2ڣU(%<(:7[Vt^Oi[zkɭcUVzM]"C( &Umzª1 %Wj=[,X+K}'osD[L^Ķ;gnN%R/ޯv )ʖo*X GWB];/ ӯqVW8j3ڪn ~km{ 7=K>SCah[P Mi9vr_ϾèבznTI zΛeW(UYq4Tÿ/QԪm ew\9Qu+=Ȥʩyҧ"7ϧ&]ڶ1~\>W#pg+]1yfns.Yih =HH؋%GR2p IJ>HRF잰a]w"R !Ks7{Η~Bʇ w |ؓޛޝdE{>Ns{ө{m0P @;['v?s~!eW@_^[vHHf YK:4xT-leSt<Sh" ^؅#2`:Vٴ/gmpW+-!8?r&a=gУJ i`$Ǻ$v H~?BĄG ؇L3Ղ>wgt: po26%@`<Ńq$6Zvbߔ;H`* n/!5'0c?d N@@Y?o޼>|8߿4s>e"455;~_.//+M,E뿈g|pdvYjT&tANY;1^ezk2U)Wv%Dp8>69!?2yknYC`dvNt|Vp’P_g='}y:oژi.;+j|:O&wyeRݴgzPTGx}[ha` sj~wRW|D-d.NֶF&Űx.aѱZJu晶=mɆ Fuq#FJ}Sor-[Wa'fnil컶+T>^yOQVU[ Y[ϺUﰺt]`gL54^' MWh3"*Tgʬg#dxAZվCw'*8poruK—C>v󽫃yd}2^rR5Is uj|I8o|U h.%O*-m55u`'vOzo94&It/S{iq|]͜YTѻF]%Z[uk W2Ϝ_*A5uJ=kST?dኣ;$sJcFhw#ّLn4Zz3gRzQUu;Ķ];:>ko6TO|qA#[w}hGj+UbRg;BGCI=pY/3=UErwoyb&hfxB\95 a纼i ,u-%HrzqjC[nZN&۳6/)5deĸOGFWzݴ>Wicc_>d܇6 )9sUČt]jƣSj;lr1nnǨfy !usVzIh벪)LSGty׵R.0WP<^1\lDDr!S+ O(B7\ދx'OUi`J@dȤÜdaWQv&ΔDl(@~YLPwBlj(*C .]aB{aIbr8ӃPެ"E 4$^Js}71?{ѮStUOd@$}fN|wY+X 'YPndiV8tB gvvd9{FƑg;`md"$koK};<!y3""Bo V:*lJ{;̹/L@`suT|`p%IJ㙩(@>A"_ bjw)nB/wxfL8D{N #yn9 Da?ތۈ1#3`:k8O ]- vx/+ "ONpDŽC%(,lǶ}X> 4a3b.șԱD6 Q XOH ).dXwa`n) BW9o-Z\6%ЫYULW2`\AhT]8hJIup>pƀe'A<1 %rOZD}O'56n}cXDaz̞Eڹ8cCsuFIefIg &fDL0K7tttnܸ{F,YDWWWܝ:??ԨQ֍3g?P^^_3YYϣbiD.7>z/Ȕ6q^.Ӹ6LO1 ݼ.n6Qv1]aҮVx%N׊΋ Sb)/ac#g;)*3ܻ}p܍4展}#6{oά~cG懝`,y_A n>69o~Bj敫vZzfȘd'GC6-6X񛊏/1@{.Y$w+(ay׹~Ozt>Ehy>! EVOiLq>O4PIwپԾ{>JY0͸VM"W׈bBU|ڭضIVwH+ѕS)l2 1H4ڥK-C 8ܛu*r]Ri7?ghEռ|s+mźzGU8XE@s?[SO u64WG.=X_yrˤvW?~<֫=w 'zhvB64{Akg^ߪYa|ucePD9 ?5"ա͡@儁oN})S^UFGmVZWsy~mia! 8D[N$S};(ֶ"g}&̲k׶/B-~,̱fZ񃮕GĹTi=Jz /u;}4Ŭi2q˝74/>aWp ޟ,.}Eִ즥)l08p㝷#ЌMl/&,!KDZǬ!^]8ˉl {0Qty^@YZп >OɼYy-`WYE@]c.k,tSfNmHaCI B`E<M0xv=f\*[|+0xJv|Cc2t| @Ʉ;3}#P27@fF"^H‡c4 av4[20jv1;8<\!}T dSD t1(Ľ B$*XO| 1  ci1i=b/!/g@`aD"p_}`Π͚ؾ 4# #A 6˩ZǸs-B Nz?S$l"i_VmUZ7-Rp(I8,M~w=(S"x>c DTЛ>,͜0oU 9~G;O'S ۝ txeP^+4> rK~O 9>H PQ$`@ 8^'_H|z70S?WCGy}ǎ37os|at_ߗ4r`` ~_v^yA ̇(lT]NCCdrߚ+랋nSQr]d1W%}K(GK͚^Z=H_^|/m;S<M4+UN.fdoywX,esѯ5NIMZ1E c.x-6u}թ|tʅYx<ϴHv.rNZQjٙ^MݿK)xtmW9F5ZI-MFCUFY tL'WW|SʇgTR,]S^AmnD7L9o}m[^zj~XN;nbr$-Lz|}aMW츄d;M94KҝFhMC/&o G7voٖXuƈ,xڦjʖ~'j\YX;l} 3bV[OrJbxփuU#^eX¸Є17_V 'US7ȈS]~leeJH7;rdf kl #jͣ50\yJ s~8cvSH5dbT]v6}ldoPܼ YV0Ɋv1#zJNIbQsC'=agOf7ߎ` OOtP}V-U_~1Ц si[#JkFZn U)Ը֭WzV10oeuHF4eyyVV&$1ɣq|jm/ktna٪Di;%;=dΌ{[]>|^IO.b*? zouA0S'}p \JW⏆[;^>#-"Oc~Kj8&$aX@6g+4H;Db"۶:䜳 I@A@D@PTs9a96sN`*d$tW}_k1b]vf5PSz3Wqn?l QSD?OGqjI ѯ&$b f`4 IH4R!ﾾxOZ73~CTp+`~@aMD$ @(? qd}=&wEci&jpF "qgL96mpkn.GAX4Q 7b }>1|Crz<(;D;5kNIoGm@F& 2\^ss,)7V5>!`!y`Xs㰖h& \@oȆvuG[۠˸),6B{z;|7AH_Cvig>=ϝcQ{II^; Õt[}}}"؊˕#Mb¸ooF6XzyAuYh.:qDFV. iLYEu@ ͊Bi120ɫt;|" bax:M&\Kݧdlyq=m:)726]]a s$cn.HGNl,W5m/eqv(Oǘ:1 r|$ 8o\ލ׸ˉ1 %b0{cbM/a@)"h@l}O&^^|Nȉ/k^dԵ?6iߋsL}? ;$]I?짜+t"03.nxv`RJU!~-2 ;JE蒶=3reܱnά<6`tqoGoC*iiSWٿi TXfRa#}gRR'd?ywU/bgtݞG UgYI~bfSwQEفuSNt/Lo}5wLנ-dӤO.<#j3OSVNezB_[/ =ώ ]"=t`ÐU,w[E{:7 c=^ӶtMzh.r6G]륟 .XU[DVTvoh8ЃC_ L.HQ wbV~V3F C!\wWLT~&R;c0nG}yV#W&Í,[u2nc:뷞WPgXPl5G+mҮ=Њ=ik?ӡ߮hJ}-im-2_vX:|{~s6L侢ݏK.99P_)'qtڡmr$9תg.2zS^6vESfqo`;fg/ژ3Pٗ?p;E3BS:MzbD.~=.z>xn֜gJj4*rmYub8OnO4(/om%͌GiҒvgd:uQMXBkJF6E yBR1sw~x_~A5E'ڸEW}cSν25jtk[K(Μmo.ELXgYm}Ͷ)^Ihi)_zvlGWwxnAn*? R5v.i#-j+T Uk%qK~ȠHr + ,#Ω?6y_4y'Є:mEBACDS C" ]OL &NR6OmepCQs4A3X B#r9~(B1rr07 qQ ֌07 QSukM͜&@cQSVGaI4^b3DۑF8cN>않%@:rZ!DAYWq{l{Rɝ4|)03׵gZΝxUk? 77Q%F 蕌k vA.LL疁.-8? ؁|fux/XhS6 a,x"=YQn;k6@sS~0#q-AૄonfF>^cEY^pHGA }߂!R= !jG`g&߀(C 's&Y^w(wBčor ;8C3=g`If̀b 0OѶN:'n2Y"E0 /gS(CΛ,$=>xc 0v7Ixɳ/,&/ܸq/_~H/3?ZYYҏB4~Xgs֨;6S4-JN75R[:W ʗ%tayBt7J{oTí5]j5;X-$aQ V"Ifio30QnVļ6.'y[5dɂ.볶Mz>&jfK=k;Ȗ~,:HnE:*ZiPZVjoښoT6#Q yӊ7>bkI}nh/5eGcZl\sB8bZ7R{Ƽ0RUz@>@|ܦq]KuZjKBjv$k 6Z zfڷdrN9o]Yk^G0y;[=;Iۮ˗n-F`)wwA$?U^ߒ\{=a_7-:䆚57c9j^keu4f@UdRyV#}>z[a\VM=R l ouORsY PFElh6cD_jc=yy@~yku/)\[kڎYYzGF𕯶jXW{)kB"/WMsk @ Mk*IRͥ )Oh|,Cb[ W )aSɯ3L6NI ęb4(xS9ϸ16;gb܅Mɸ?V Kz'( D"PDM,njf>TB'1/1=Lّ:YuWr !Bv|UUDydo{ ZR[S/VwPVUt+&Z> mp@+ ,;haf>S9D듁3XP1g-4־HLƒ p1eى{E!%MFsx烀涘9t;ub%u^, rd)0c߉x@]>VұGh}qSCa"wbn=8v⯎5mA(%茢楛03#Qp]vQg(;E*D.jVocL9O'Z. }rwE\E\BA@rb"o'aBXn_cP)IKWYВ rVrWA,BݓN"ezE[RV4VcDAM_n]P /E>Ttfh)UԺ&|:u{;ak@Cܐ\9d.+Xn/i{A+GnV;/VHa\kcvۆhH]p=?͐BˍrEtxF 4eRj39t0Ww`Y{֤jH?UdmKC~ˌ1Sq"KqOڤW&gN+Nn|W:lN=[JW9aEiONpmcuw_~Nd'?4풣i2V=(⸔Dxς Wh&۠nd)ѐh`ӧR=]fg;lVXkMinth%Q֝bJhN(|YkhZxQhM[jJu ?Al19$[TZ;IjldE'VTSp+X,SnT1*C/tT\[ᆼZ0$}lzH;V</}-92erA~]TK U* kok约+g x珌Zol-D0j-XZ-ܬ.ܶ_[޲v7j]8a.1*]1SRoNڤ$^|o}AQyQ')ϼe.>MbMK/FzPհi}´U}7vTw窚j#yY0FuwS #LyW%hBdbf1ÂL.+eʳű3:\*TorBe.gq_Jڥ} r*U.&yzY+鳭=].W.lQBi~H-Woo˺}jAbVeoG{a0%2Y &~/*RhU+P໳h>"39z U{Z=_ş  +$=ds?8ǿ&Ŋ/Lu&@uMQQ+!zɦz]Ũ;?Ȗj8mG?l7@BLnՀh̪VK%P/@Blm?1TV'?0\x$D≪&l ]{M˙ҿy(q3yY?iN`%S/YΖhQ@`@ׁ2s |5GGY#77CW+ԣ7qKrwmf8qQtF)b%ewF C+HbBWaJ}Abl7S@ x p'lMpjqJwjB-9 @-A2Bi9!L2ǀ,) XX~ >%|yqt 0#`rǁEmw v$&;r\f[<7|__s iA6h+u ~X iE/K04[ 3aZ _NP 023f=úN#BAyxo6sW13qah$ ATԘU9l=sap\@Os9LpNf#5f3DtF(G#"dFZW[1WC^d8Vm 7` 'xK~( (,&ZnP fe \55q&3GwENě h"fcݢ c Cgκ&l^u8Uc8.RёʸPޯ|pJQ_{/ ǜcWE׮we&a n/pRR5EV',`v#ȔL04?uɃ=Ϝ2OC> ;(00T5|(EFGhX>FhiC6`=;n)A_j3"2I =gz!KFS6_$?죤Wj߾}?h7ɓSSSoQClfE|>|klB } g}g>)>sy뵧iH,ٵ~hCyNK>s'GlKkL>EsZKVӋ;,;b}-+ yo&>Kt(1@AI:j(u۪ƣj}SA{^ۈ2af'+LELW#,̥2V[[s2ttXW7*GW_!fU*|.>y{1amtעP@eD設V5@U6# OsV*w>,V__QYwPV7iG'dDŹsjt-IFOREViխwC5:Ie"^NfI^UhUfٴKY+{زROmuCv#tUCόqQk-5.*?}h6*.ذ*{GJ* ^\0HT^1;jo왚?&}zf'v}AfsjyiMyHд[Cd%#GT[q7j T R쮊)55êO:Ah_&o*qϛ[(;ЗʃwT6L2|}ヺBj cPgAswpThNxrY#wϓjZ5~3?UxSի>,i)8*]m?߮ZoUY+ ?~)ДKpߢL3Oh9u*ecB{Vi)Μr& ,S͋5~yYuJ|2_`hwj=eD}W<UYV򚗭0!-',ۛq F!yܫ6[y#W`iR ?aq, M,L<)FJ{Or1͟ ?_7@ DuMєOtl)"&g+#. @3fzMyf.?bJY $m A J2@1q̞XQL%qlje*,jNT5"!c9K";1~;A(eSa<7L\/[.(us3`U]>18xyKs2$;2c_AtYPX#Q<`'> [=h G [KE#@O3NG)W! 4l>nr+`0b`mZ4l X_aHO6PL,Kb#BcԦ'獵8`)+'~@Q"K s^a~&^~iy(Fh@w3@$] ؜pD)Zd[sk:-! D[0veft ,cW'Z-sox0&,qqE'caAήˀ)yidipK`vv:GܛK) 8 ɘZ˽sd ZX0V'2eL "~-!(LizҰÓft*;b"=\~x8~T^[0 Wt.qf6g졏t@ztnD T"gMg_镴9!,6E=h Gf0| " \@ZDFa{*+;İtl@m\= .nMrS%J3p:{\t1n|[A<Ĵy|!_I5FO#a;7Rc.{1Mxk^ggt?A{D#Oֈ{yfEEE__ߟQ{ƌ?dgwԩS ;w%K~.??6331}| 7o>sLϞ=3Akiiqαn^쨲_:]b8ゆ)tQkݭ":O ;_8j(]_'W}aR8N:VY`3NB~ck }A"mvIK >OAC2&]lrUe5m1BֺcwY^*yv[oˢW|]8ljP4O/ ojh,}xc}#ڮO.ar<4a{F['t[&8lGO}f R5ӓ<#8LZ;ñ=VXT3v|ÖjE['W"j ' JJC #TV_pxuZBEڊ>*Z*+V׋Hu~imݟKyyz 9WƲk>tXvȭO2>W^Tӗj_ڦW<-Yղξ\8\eg>ؖn`ow{_^QNFYk]r*NKWej[lPVO /V}7T3媕ߗ=JJbZ9;;Rkyc5kK_[(9̚)6t?4nT(z$msD"r,]V9gXnE;,Gсn'CJ\96txj .{7ߪyV|,YةxCSMU%!i7V,$.7wVai?LxQxϔKfMCˢLk>n(ˎ}eU.?"޼g-2uKx|yCYѽM=ˢtj'Y5[0O6YJ)Q]+b{f_tyaTYn_69%iqsǕe=i7n3+DX\=VL΍SWnyQr5+c!e,jFL8Gs K~ 2 B? {ZE㬐A,?A\'?iB$| 6 $$Dy1ԒN hϨ%LbUQnG7~G?f~i煉c^~"qg4hEϙf5݃q:aXBB " 7D?X;C߼δ5C24.LZ|udBM">ݏKGL@_~N܉wuMх doʲ@@*-? x1Yy G+ߛL8  I!!# P1xOWrqDlj^fR!&0•L>L{+ǸFKk MQq صWJVξ2"1ܑӍ%=,9@FmʹsA uT{yhzC i6(<Pˋ&{)ݡSմ&SOB0,w~}u^t>^',N0cx {D1- -BЊ.Rf ;Itp9L3.3񄛳߮ЫGMQ~m IFI"ͫk ; P. bk-]Ĵ+A$L B23Ⱦc1q@69qQ! 0+R08:+kL"#2?{ zuW㶡0ahl>=1&T#Ͼīܵ_/,cL ՘E{ A&OeskL~vQH3 5FĞLP7LOhQ2'6sU,@B ZL'¬G(2,g﫨B?8AL:^ < TNT6"b1nܸ_Μ93uƯK,hu7n }lٲƍ}||\]]՞߽{ 暽ۏMmۓ'.-ڰK9go1 3Z=bZnj+m{2[9j矅J3}h t}n0좖SzVxܖ6QT`RTzyFI]ysJ;"g[l\agE7D瑥Kqb '4/1_uÙcJqӁ쳃_&u|TQٶD}s#g>11^^]>ZOՇ#M(~1DفOvKO15{y5[EL,nɃƪ*;'SYNm35oڠ.ӵ q l ˳<~v-u/oTT.XD{}VF[R^#qCQYFb6.SۙzvHjzw EIb+]W{ y4qyH:AZ &J(% ou>5XWq4/ضM/)G|L9iUyj¤ǫj(/ic|QhR>sfF("{&$nbNg[R00tʹG2"{t?ckCȪovDco1Uj[$긜o1emrl`qF{o ;O wA1>'J Ew{9*+lYw?krul7bώ :|Wjeϖ):e>+7/Q7ƶs Illa/Yī\LNzNyG2JRQ#uZE૛qiNOKiXղIXaR;,/o b@(wfL>Aһ{|@G 81A՛:&2(b{9..Cl/OIc]5 msa=YgX4ͺdf;ݭNC8[:6ѱB،cLq~t4E -ZáX_n`vp5됵=p'`ޏ@rct,[ۀU qFkndy<'`"<+FwE=(gbeۉn-9Lq!tM!{34nK\-Pe/[~̊9;1m>NaՌqg.eܠJ#8}-I߄/Av`}n#qZsn$ +uPG8(9o#D$qmg#ljjƧtA  {aPRmw!97.;ċMU@w a]$Uq`D7_,>QI͆!kOp߸buMso6ཻRݛÄsxz 8g)$ՙXXL{"ehebꍍRLBΧKeW΁.E-T c/\b8O_O Q48?-lH? 0_@HT{fc)J(WX'!`#տ}AdZL6g/;v%h~S wRγ@P6DuaX|x?XR̙s&׭['ӧdO9sfΜ9אÇz>rϟ!\jZB3G[%١$:+QOt~ 75W,;7)jfjH֝/_5иIC[pb'>\ھ06DHN-(IXk="aۮZ&m#Q}) Rx[XB_=NzC%zvoOjH1a_CyÆOyNwcdPWI7ߴNoSn.hDuadp^Yƞy;q|,IjRrO~TEJ +(as4R@Sj!7S%: {z؛em*ijaѧ7jo{pFUFo紝*O<<2` ݲ>DtCy(!gm<_|YqP l)Wn6zv_pVe+[i1B@GUú_,cEPKT-uz4C3G#~J![U+s!qjfeҴ5Vj nMM"CQ3pXS7I4XX?'؈D@ +Ĥ4&>KH$$$5-4w$s64R7!jT-RMnjf:ӻ a4!átB=w^ |x@Wh1_Xz^cV0  #%%wJ5aN@!5f ӻx4wHbC'hZ> D4s~N(c!@ ?i71]Qg.{MgK`f f6Rc]8fXN  A9]`F~^If͊T&!Y{v5-/vN\1qǠsvv:ى1Q6r I[]?qGAvL1$dD wSh'F x0'pzbN `e} σ}`t%˄;=>`H~8vo"jhٳd-,,|||>|p%7m3A SS"_ GyͨQ|||***z;//n:f즜<~9Þ*zy|β0lUf8Wj>W9҆?V%}.j9\k, BUFlwqk6kZ6mo/تJBWLE)^Go'Un^s7>毇:w5/O`egfܮanҵɷ uJ6g}RU)~_:>[[P[8=QXT$,jYUcFӪsA^uwa^5Ig?iһ[/.ϫ ޭSY>zT0!v·>ʓneT3~@"EC*oGOwrPxgM5%[e6}sF3] V~SToߥDC |Te^WE{hZQZ (tm7Kڣjح ش`IwP·J;^ ^Zg<&eB'lvm4+:6N6г#k]_ɿvjumꄪfߟ-ڭH3 1*yIꢞ.fO8ݲ:TY=:2u{ {ѐ_>jNpDrӭ @A˕Mn(-U8Gx] aE׽_8seN,-s-s4Vם^9l[~۬#=*[xSjTYhƙS %9f WgV!FPS@Րl A9$ʚB@f(D? *$5ͳl $7Nusyeg~("\#B@$3$~tP=Jl=_;_2bs)Z hQ@XU0+>-I4/>-jg_5_GFPCT#CT:Zj馨jZZjOWҴ h8ms|辿{ &y .`\ OI,gIUiIϜУs KӘKDnQ1(hO෌0^{{7D/Bg|-#t_7ǠA~qJd dJB\")~u1PnK-+.6D'^tS$VVw~N;UfVf\ZXhs;meJNnkj' Y=S*dGW=&`rɝNU=%z .^{j7л^ȇ]$^ _s{~ºw;T?9^ص˒[>D}!/V:VHu=. RE dgVJ\ 8=fC]I^{=N6Qh]_S'Ƌvbt`~a3ͰmWxw]F:BN~]dϬ^uN(69>Vܯ]|N]L񟫕dTշ3#vZ uZ.f~(amFLE̗&}*mmP1cͽ%f W=CYuAy9hdCESbccm:uv^ ,(9`ۚC>f`յewg%޹w/,1^ Fi(c-.5vHɔkC{f Sdt\ I[ҭz8dYw>~a߁!',^ {a\4WߙK0+wbLy*B-Jw.>yjχ33HvZvATHV[s_кkUdVS\N¾rF[6pq:e\u*7ԃd89S_Đљ[QV˂dQx1K)`Whe? .o~I@g9Pïm"YBb_ى _^|Xi#CIܤ1K;&Q"&!i,f{5RA4πaH ވ~wHlFSD*f%XP\LÀWIcl)nPBqUQ6?od ۫ӟv HmdHi4 1V  ~PB!PH"QӫuT.A9UdM b 4D25(~Y05;p9F/g#boͺ^A;[-ၷ,S`xSv\>DQסݓfy=;lA|=Ր4vF?'- unk6ˮ*_bjZGE'wN|Rܐ;D( ' P)d ]$!y$#lx9% YL.j*M ư |5}<(f:vwW/M nxmkoZ ݎ cK0T Й'IŸ@f}@K1yda":znW@r5u) fSyoio0Ujי48{x$xϾHƑ5X_Jә(e?W0Ğ7ifRJc #<*dAVEl°+ytiP +1zaV MX A2".(0KZ2*0:`dph}.ŝzF90iݧߵT?X:eDZ^wC e;Rw;>Y[)2LR@ї/+[ W>qw&1:X5sN"#U=/oF65I]RP|oti+ C3ZSxd#7Ey*ZژF3d&v9}sB L6|b5U4pP:}M_rVX vV8XmvysC)|*XsdTM'z,Ebn휋ǦZ!eX1*GZSgQ=j2i^iʢ^/oEflM>봵FAҖ rvqH* TJ#3k[#Ye;3ډA3[JdMՋꤢpIZZ9k$cG ֚UoEJx][)eLIt˅:j4u+ƜHhb.y&,\9Hŋ-5l\\峦g7#eN_[]{+A,zzTS7FyO&K34 d0t Ň3\ܳrXJ;ƺj~ W\@AщW]J-}K[a=9śCfaj׫Su·Qz'9?z9⮌>uS4Qa&ifODvئigdb'VXi+"CZ);rPzNUR{RƦ@std/'0I&9OSimm3ե]Oug4=5TdZM(yojT_UF4ܭ 3lF SxsnlluOſ%~VL4~6BP}3ErfP l1+VҴ-/7MC TA0/m$VJ,Ɖq @ES^2Њ <n9$҇A9Ї!bZA[KJIf26Z@!7M%fޣd78)~[#3J6Pn A+7Fض3:#O1n3B v+g4 v^-6>:0oLn`v5qS*ÿZrc|JwLhx9 x%2"-t 0Q>C0q"{r ӼM/A/)#z9 vTxgjnbbU&!: ,)8O ixhL:UbSQ5oni{=FH%X} b\}g/5m/ҳ^S~P{jY {ydH2f{kq2y=WK}vevǶEzeW盜c~^θAf~%^VNE.z-o~j4EVn{-슖F{[1NealEm3NBM}jc^]Ué$eAzWPxwSE麯{nU~Vq "蝼tufǡEz|ꊀWuʃw)1%{m1:kޠ*x*#M=-Y.O: |VլYk>n~9rWŧnjU!–68e*6|j9ok}̳?_žnfHY`gBҡ4?4޽BbV ?EUO:jxqłlG#Gy{d%67XF /q4qEc phar_b=_uʤ Ka7|TN0N5J˚4rk{]Ec 'Z)4|hQz/<'5o~N_'4'{|J`1{nMg>16WBvXv OE~CkͲ?UZؖ2qqj* OΎl_vj/%ft̹JT}J/bf+҉TtRU5Ѷ7W c6[pE;4_ljJo%Ԫ Mxs&H+їh"/ұR |e-rUc⍊X.q2fg=a=y_(sXgWq+Ls#OL_a8*b#=dMՇY{b+Ozgds~F!))1EtAߜ=j3{|ZA6"duy(ȱ.PÔ6&>qOy6RНʱ]DBJТӝϬrرN)XɦVp~q|1__Audh@2e$͵؊Z2IO/Y>wU$N5#5rFG%)Q͈ 5u fJ"cCP/ qQ?dqcc^I49Ebqr!#3)fZj~h#!j>UHBDMGXz:7a~3\5(Y ɲfhJA\5# \3՘r͆"\Hc@R$hvŸEf SL851@e㑰H:Xkwa &F\%yJ͍^xȃ΄2QhC @bfr{3l`Cs@lu nA[q 9?.qzX5D>&g1ħv`|G{Ő%.{QjUVap`f@SyG8̶-`y3YLXV (zB莻P[3 MeΈT&3ƛ zѸҼE$# dK)L䳌oNcy'e20 `H QKD@V5 $ )^]@2ܺ's}6A>@WS!CDRkͪ c6ұPwjʛT0fqʆ2#=?ܭ /`J@/ >VLc7N5vRR !>TE,8Ëڃ50A=f\o 10ulƩۅGS"0̉ہ,]$PwvO1]syJi29 m-'ɠ:V l ޓ?Oa'Yj̐ؐẇ[`bY 32pV740lO'0S!@9 #4yDĻAL| ߈cf&;`-W*]$LMvDrzL :8h?'$NHKkC`T,A>j ^h|x$<(M+B>d{Ϙ\aW]e^{> -a)QcaչFv{u}SeݖKcFTZZ1k{ m{ʬ{W+]nΔi“1Z߽dΈӡBU?uAj5_^! 4먺jg;J^w>Cyz}7kvp+DsH9ǯRFf 2R[V&^AuA{jrlwQ <ʼU O\:OjG.enߣygv_ R}m&]G90Xdg}~_fjZ{U/fMkqRbZoy{}|SyGI%s1j'לwkoSɝRy=f[b\:0n'Gd}y -NFZ]%Do̲10UUXI<}Ycϫ+6t1lg;%z#2vʘr($=pNue%Z)ᖅag .Llp޾ׯvzW]Rk1DAX橙jb$ۤk*&\, Y Y)0劒-O$$TjOYhv-ĶxqaЁ/sXM>sV~.-2:$yqdxOi%"SW7QoZyXNΫi3#}b zLS7Ҳ\q¾v̔g R}8٦7yGnN;!JMuB >h3/"_񟎿bU,)m\e!R!ABy"y$o>)?^ȶ_eeyD6M Aޖ8p=/_~6ۀ8>_:?e3{Sd V&ۍTns|)5O|"]aNК81hh fsژB bp11,m $k~%RsxyIʑj1 G1vkkY|F4Vxdr/g{LG0cRb0&.ޑWaElyU*y~@a#FA=OSx Ɏ8@5f'X l"7c^{<x j;f ĎLgoK˒iwtd0)˚/_9J&bb~+`2uf^^֐QI a.C*L86T7G!4W0"bp~{6t=ɯ5OoF=ZK+!D $,`p1DSra(Fc0|ix4Hrf'uePV "e{1:ňُ@Teb qt=؞RriAK]xPȩ`>hP/T8K[빻P;==m \$tƉqi_WS״ļøڱw-5*5f]UjEb@MCbґlWqdj(J?+v'l25D{ԑ+]Be̜*mi7D;o()]rgӡ9CRk[q}s/z|fj݊aW:bkwv"]J-D'bvʉv_.gIњnz1fۢj2{]y_l(ڟuQ[P_Aruϫ+S$ 5&A+-8P{Q)nSc.:nvEc^LlRPӡu%; L|!Q1Pbj@fۯ טEWK?[ı+8~ֺgV!=љfV{7KwC'bTSXu~E΄-Sy1X!Ӧɽ;'N.gRWf.2gx4ԇ!0TA^3c ĵ(o{Qcy+ǂ[~P^{Ʌc8:59$$zN&+C&SzݓZA:gf;U|m~2$%n׶QnNݎee/nVo~^7=Juo|«)UQkߖYI_N+tmmو|B$Vr_VJ9W%USު͓YQIZ= ڗgµ0Xe Wz+)K$rSGIņLkpҔZu^2qE2:: 7ĔmSҜ%[&˪ch5Ng e1@m56ÐȱW֡:<&7Z2 :m̈}Ji1x &ٕM}eKiLNq͕c>~Sg쪪_kO:-SggHMu.S5hjQ-Zi[8Y댤ަr,Տ)murXV:ZI 3/puqZ"X3%j-֫]^yͭ6dJK-]225J?JG5}C#ӬzC(O[%ͽfHA<Ջ!K6#i~ \QFW)B),d;⟥$57I6D ! Sb47iTlˠaa0ht𗑉IB9YuI5sWi|t^%!~ÙJ5S fb q9ffÐlSdQ͐)*uEl]"ŀ"2|-c%>y!ON +cR[IL(|[_O$A * KPֹxSGm&+%k&;_D[FGV@#MLF \uX#TALǁIO^mHulhGWٮ7>eQ1_#f'AP љoSO3)&Z.Isv0Ç;@ b6v3mA*n'UP}/~3_ʹ24e`<2R&g 4|i2}&Ixq?chZMvڄ%{zd}SV2%q࿈ <jEE!ʘCwYA~bl5yw;wFx+x/PP"tY>P~Xuh0|(G x6p҇DmP"CFm2e[GO=/ZT3 q 1!zJ~b//:Jjycc%N ^磝6޵#o*.1p'cG j*lR,@P\ +淒s5c3Mz\_PcAлPwc1MvTJc<1sq-F'azrڼ 6m\s½gKnK~b+H(s(YTel_c% an|O s'F,C1@G~TaoU @Lq™Fxe W8["5ejple;(YGNj 7H->Oݵ)>NF}M0uV4*J}Vۭ#LBtʓy!CZڛ{}ӫw2HTQqe|,Qcie $J)Wi47aLvyaj 64r1E}3GN\~>hi.Ao,7oP꼃}G V%:e򉆅NĎӟL~HbS䪒>]ج76|W:ߑ㯷j APtT͠9قy~./o&Z8ub/鈂SaD.sv͊sKڮPx$C]Cߡj\_luh)C[ňƘDspeU2GWt4ם;rEWਕql 8u U4SY;9MqRͺxo޽{ƕ{Ы`3'|Mx+].3_\Louf}bȐe!h1E^jž2+H{i-USՊ4E {] t7z_K,vPьi=ҽ`NaۏvV_yT=lnXۇƦ;f Jvnfm^ly%߿.S]oJZ;IVw:7|f9M~įu =\ž|ig*>^fpQK'yhsʈV}*Ky&;Y{`txQ_L:荇 ߖ+y WQ$qFsfN^喺U9vwyg7H|ԶqDo/*wQ[[:ݟMI,vM*L޲ӱ[JJ1rZJW)OKz9dfQ{=hj~5r~sU_ϗ?r4jZL*TD;&+q u;ܡeABMIޝnrAC+5?J}KP04 jTB9d~qm%bӋf9yOH6م?㡊Kje#O4KA{Hj$&DX/F,>&4a.1?S3BM" 1b6R=Ĕsj),d4gX]MF3jDj𯅻9cQMR!FD?/|w.,l2"iFD͏S\ӯQ3J#Ђ5ROaMƣ9S_c 컢wl^ @4Ds8 InU/{GFtՓ,<4;z6jn M2J Ĕo[ 0q jCHՅX&@g SA7GMl͝`lWiVkoœVĠ5GxfĔq!Zp>hd\8(^N|<53QMM YgCGOb< IϊsKsGa3KQF)p=Y5*1$r8ȑpKPnR0pZ,I/g|0:(3XܚFt9ܮ@Iko`lLv4 qN*3XEcץi˃40V>\7}f铗1s's JCҘZ)LV4ٟm @,1wqAcY*Jzzໍ(hiaE/}jj *-roVʭ@OChAie'a?0A]u~{ն!*X`݉(* ((JJKHJH>?Xg/w{s}׺.s1 |Ul-U c@ xgMnn^M5M@0p<tuc^w\n?Q ott&FItYy{*Qi{s':µZZ~*8c&r^k/P+NS15IV7hVYʫ˨TDaT 5uᄾa%ʖ]%&~|HU-[p7~3jWi=S J)ZxTœ+~]v\j nfrؼAٙY)b¸k{ڮ꼼-&9 /|V]sҫQ:m-r4گ5-Kw>Q鍚yRTfF7Բ;i xNyDqIҗM8co VYq^Fj9;|"3THc:7~h,Ъ!j){MeRIaQtRePKm!Y&cGuD+.L <Ҫ\Q][|wi̳XDwܥl"˪=kL`cܣ bmkT뷰!}v{jG=S-A6.[t˰f`e)'NS\tuj ?~x7R3s̑s-@Gڬ\oPAEqJh2$pUqNs?WVPh<7ڴcZbyUv ^'5וr|rUw괯IB-PN_Jp/ZT_{E͈$RvI%f4Ӫ@Xsi5-ZbZl;[غ4Q AiAmcLZfq{_fjPuTC  g8Z IRZК4K0Kf$1i宜[B5Bg >oG ka҄o +ſ5Aœ6dV&PLϖӥwڇL<.쏫o s>앻 3aM ^=v̞}ۦ9MF `^ S|`B >Y1`E,9zUڪ~Nm>Q'?ܛ q[8%{^GWVR:}w>+Xc̈́&6CA瞟 ^ q憠IdC204#do0AjR:25oVah` > %h?g5<3V~}?x87-#8i/ъs!8b1fL9`% U ZUT=TMx³^e?:Grޘ!NϑL&\D1nD@? ":u|$2KFq4 +@QKqq:=X`<~ YPLn]L1[ B1C P %R10k?}qO]?:?/мlbݘ~ Iy;,Fh7Txt܃v+@4%؟n;'/x8[/uhZF HV!x-(-N q_ K>8!0/b~?M0 8gǴ„p& CfmI.T"1 ]=iȂy&7ׄ\~tƦ\/),,3]DHy-Z >|Vl7lc÷sUnEi*cg Jfz cqaPYF7G ?+%~s,:^z8Z6XcoRZSeC̣Ω9 Q!Ч.ne}ONuؐfgq$T$;~2Sr]RnlGHpN9n\nΣJAҴƮ/?J,-r^|Q1cs͞jlx}~ownzQKLF0I?HoEkz5-,NPmsU3O^ U+?$%kn+4>BB6sO79:mϯ|W6|`?tVP߮A: /+32Z]YH4<<|^NwF%\3ք]3O;V1 0E;gQ1J1må޵s ~\aqͶ5 MW{czKܶ=ݹ9eKxh(L*+>TEw8ypiY'ύ/.Xhk٪v:&N_iY*'3{=uVu=9v֟qJw<:e\=DJ O|&?ui7="ѝވi^8+=oڕi6vZ[vX2jSo,ԩlyQ{}H:fhVհmn=#UzY9խ& GbB\rΉo)zl|󔥵DdOxzSMn"rnC`f5=#w=Cv?_4 T^*߸nQx,Y?I}ݾ_mgRFj\uk]Cgguv3K\2 5okEWdW9+fK-=:]1p;oV 7m=俜b[=Z%gl55߉*Z^@kI[+xDD45h;Uӻpk^9B"9 TׂQrI>uRJ$9bao[ӋgC>]pm;͋vv(GqE@C #wLY oG%R0hy‡%ז`4PCtp*+z5T1J?y MvzL`bZ. >ΖpGF}ܑ=QA' Segc4߽ܲ7,*ky( j>01i$`t!ۀW޶W0~&Q$ F%Q͝2G#0kҜlHquN P2ó7X ld`m^CXK3[  !WJ&>|9s 塥}}\ 9  F4hZEHg1~AvD@ϲCIQ1 jLcs MC9{32)`f tUBġd[O hg2&c9q<|Nb*SZ ZOAtHUhDœ, / @8'{\`$s>2Ax6wմK)>"}7HB(t]o3F&0]΃]Mf _FxH МǼ;c_@np`oh`*/K,]y=,вFM<*uݴAC*=+},;66b (\_8M}O?n_xyסF%M҂?sʏv^U46LDIR緲)~ߢ 6?4]k=nAָӭbՉD0Y7֞歍hѳ\.K C.I֩h.%:<.Pj^ɩe#%_,UO'3g\.ѭDQ;URM3Rs+? YiWyyPVRqX'I[w묔hK-ku`|ej.&q Zqop;Ojk~+ncEe zZw]ϤUW"4MNė-4<߼}QO_ qwafCקrI^ӊERul {~Xjsy*X]Oҋ^sU\~j>5x紐yc١혝r5+ ZFe+N̿sOv5]Zim]^$v'g^X*G{ȀϏZ:Of?Z6Xz'/sz(_WW2r/rI‡׻,+cdGa=ށ{:s|'| rr߂&=]%{S77?hڟB+)}64V.yBBoxl⭞8Էh"+@P쇱GaëGʍa3FZ "V p$6fsi.@=^c@=.CXC nJp<\!dl#"$OAj ,I"10p͵9^6iIe;2U#[ji ~#3X@/ H=Pl سʚ(jDFR&Y\Ff10vr!)uѬDp!Dxo2T#{1PV"̠PYHI&ʅCp؊{ Cӌ E> 83bȎ1?ymE t=ف\rLL1D:ѡdD-w>y"ƥF0|I=9Ȅܷ>$+~r  /p" З,vy8gx*3I/Ax163h=Hg p1w)r@p!?5QKq&#%nG.$=#0X0=pIȡ'H%nXLaFG qJ#0ޛl[N_r2Im8hGTⶈh#DmzY!hHs nR@tౠ 0Q-$C0lc2E(@* ?K# zѬ <&p3>|qHlP֞`t~GܿpwT[&/d3H/S+Y8XtHo].ӭхuK w=o,*ͤ_$VoJ}FPi;qwI"/J. C/]mJfPǜ3(.f^k^m {"$?6y62 l}ck+1IQҫŪKHz=mWO9A|Ǫ8BG._?%V~V{ǝMRV{ .B13w ȔO3x W#Q+Ix\'Klώ$%nR5;!~^XߠQfS_w5S>ythdZ1c¤}Vg3?}. m#e3z}hS?նO-Oȷt_{o&n9X7CONϺ{ArB]"6S^LiU"|Zoeձm78,)#롼;-V d`4ea9/tKP]bSƾ{LH S7NsEșdC<3^b VvQNSoM?s򷯻Yq͇ZHtz׿{2UďK:2 ͂*76g?j#ԣR\ G\TZ&-ԹJŧ*2T$'Nb&I<{4T:-^ ^ /Sko"=UrՄnO=9p۫#VЕ,VTuߋUQO2aMvVb ;J5KK;FA~q}|sY{cGltkOβ, ̬MWW=MΚRI~ ܒ, Pt UKp 19>\LEkY޶ju gL1Y|:iN~'ϛ^=Bstܐ'c*PymF .?>;]5D>AmS9s;Ⱥv;}O294c#2A$# ZV" _@hl}y!5Xk-,D TϨ1Tc3 IRؒ $ D IT+uQ@'@-f7Q$/Fls ,kLeWԂߝBH6cz["nAH$ EBeTsDBK'AukTڬB-*d_'6K2*`K7Q3{x[6=>ດH%߱# fwl&:$t*"lƣ7Cf "(32->I΃@wWhѩ*ZAؼ )3p~9 <<xA4b@Ti+F1M~@=@iWKs` JDؐ4nX7é|Vi/70fhD`A@7 $@x>x ?G1fWQwڈs`p&oŀ8Lq݈a338uB{4h13+5z0`~^7p\$&DS@)G9a'h5iXIwQl=}%oZ ( FF.lr#D,QCnT=ʟ#Xb7]hUa6q`8x?|p&h.TplcR4e`һX#iMY5٬ɎYAH~O6K&P.N5T'r{>&8L̷z ^wf i ޵7<@ 0.!1fĘL A憂n44=|O8mҍ#"L`c6G zsy lSX;r*M 'QSPfƄC{9[$J^x-S ADL96Vjކ(-9,}eUZڬu:ݝzAG3#۵p8?Bxt;뚟v. ܩ9kqc)Kީx~#ՓgY=eKQ>1~7vc++Qܩ϶"O8t+K)R2B@˿g5:cW O?>>iKlLlϮ7U&ˈ./<9|RXߗK>۩g}7u֚.>s dшF5vI" E5>[?27;$'.< +4Y$q{g e2_m3qa[?d7`{gB>]{`&Թ_W$'z#MJKY|:mx. zcEӖmĸG4r -ܮ/ՠW=a@$2X }+?'@`>Jt3E_c`BA&nlj,vdŸ μ0xk`r"Ldx]1 +AWB8  4u1;ibܷYQx= D  y0x'!^h*Oxє0$`Ec?wf ^!߼<;LJNj< &@,p.O[^5$9s@ )\;,Ĭ&W_";" hGnoŞ'X $wk5 P&8YHEchݽ±TqeI2%:F)P[ix=81W̠icwzE|.bm?2~kTΣ.UKCf$mŚ(~ > Hρۑ2p#ʄTB6ۆz ZZ&s'~bDc"`4TI`4"Н#t`Oygi{q/ۇw6?ͧq511M˧g1!FO~ཝ+W_o;=P+uۉ～}Pfލk.{ƆK6DKg=Ն*Dqs4+c5P֍Y:}I/-ЯnN&noZjE(KUX~sۺcD26qIߞ(8J:w0K|hkOW])1S }I}W.׾Tgq5Hnn}5quɮw-z/pM߳;º+Z*nh'z9DQ߶_(+#5kցk=dO^csby}eQ~lYk?d \}QJtt_eOUEuU6ae&e_'p^vIwPrm׈B3܇WK?g|ܻgo~tc^opéRwTsnD kgq㛿כz};im4M qcʆ]$N$L3ė7~3'ÂRQx]ݒi4Y~mw߹%g|h̆O):}R ]%0ɯ j9BsI QyA3ymhOWZA3/럑(c1~oZ3ǞR.$d*<ß\ާ^29F}Y='?g.f@]׬^1&=|V|T*t).(Py>'t'0JYYjm x+'\][njczDEtUM[~uOH mUnt[YvZXMÇ Q#%]|s:Jk1̶/P]Р9q :ƏJLfX:[TVn0;~i +U,9\?\̋?{mK6򖡝'۫iN ι}1WaAO0[gg='$nU_B< .f7LWz {L|I̪CiT8 ePc#>Ta;D(ln!D̗Up)}PXv+,nj$' ivq3vq(y7p[GV%*̐ m/X-Ptd {bONJ`RI(nC)&De~zXh!A/#q]% |OLxyd{;2[< cTAF>zM9iI_E}A"Qz̊ķ#<+_G}Dg;.1L`|hF{16tP7q ~s]-IuԷonikwVNr.:7 ݺ*RZ֊B Գa_)BAMQQO })jZxa^NSٹBSdM]2(-_ jK(\X?fЄ"tO2$:Ke8o{'MRWVF[F&9Q9 ϺNw8~odžyI9;oT,U+78)L_U {&_8V :RX:fӥâ ,$ |I=up jr% ^w~Nt7$,[usMSiU𙿵Ή:k\W$vG^}7zWVIYpXًjd4&W z~$<^*.MU?:hQyz#K;v77!;Al]%>)XܽZOw^Kvu~-c;Ҭn?[zsR/l\j܎U=¬F߲Wpy!v;cItlsC5iStr5Ω7ӕk ۿjׯP5MǹbCd &ۼՌ mo+0$eoXG} {~5~͏R/Y}y0V?Ji*Hˤ~")0d"!tuu 79[lWU=tؓƢOK޼kXsr=)ł`:e; {hς_OEy+{`Ƙ6)}۸ؽ3w:9zͲr}ь,ѬAϝN۪zin^ -zXJ@$ՂU)l A-(h18ak6[=ˆEe [D4gmeZi&w[ `q$ZkN$DҿPU}'[&I3KOJrA8#tn!Q+5$-B1_(&%lF-))j"In>pxKj5K I j/XdG<Շw{(O &tJx@ݢqW@W0.ѡчzyI("oÝ^A1@Kz`1q&3 ;?&Cžy76npP 3O^8 8s +Ic<_2X3ۣ[vbŒ]m5ϡsA`k"c OBG}1.ׄ[0[#"&>i<!DIp̙ #0q%nkqL$ ]33c1"(7Pf"63 HXȈ"`wCtü BK\vmx u^t?v#q#"O|2",@#:S0p2 `kT}`&zǙuξYCu~ԏ A@pB# t @ل(H8FtGGb^rvwhiXe"zct@,̕bs^v89-H 5hC; OhGF*A:0bƖ0hp0­:TaáIXI3A}ՙAӅy ș[hntk]r/D1J56i߉o-|b0˶=b)Pjּ2H~isQ\V6ݻ F$x]3r/Wñ"a"n*ɲ#?J+W_~ek4Iq;NNϖ~[xhjlJ\`ט 7ɱ֑*짾^?*U0!fM ~(pgo:96}gFI `V z xR#CxG978H7TэP9sS'ZQ05amIdUWy>,:׀r)WӲ0U_~(iƏ".xe KWFelbr{7%%weBϐT=7-U*ܡu"khgKd#͝( t>8:Iu'RK2K:GũN}3/WeM1^W{=K\}滍MhUe+G*\ONrҶ%w7h5G̍ c  g\JZPC?M O><'jɽ`wo%z\]0דwBe}_'[+e)UH/>ṋuDwdzK_Q/pw]ζy}SՉm=\WL=.;C=uJƨWͻ1oY]Cvq+AS^w{gulzu/!fA.F51y&A=D{Qb݋uMs :wL} #2:Ád7,+Z#"Tmϡrʶk7Jyݫs,ver~av|FY/=Qt_uwvR e/؉ N0yl+GZB2G"geȻFHfy56D,[Jhh$}5ӫ2qsU^H&(;(HOv7˽f#neq6oUl+iVn*#y٥Sbaj J2*w%r)Bc{QzͶ(cYg'˔Z+x0C{҆r䮖󖴗S> }鴁6ZHSqLf{=dNu^^~t?kNv˪KVJ-"kuD[Vu8V[ݔjoIZTt@h~/"y?Duk ړ6C#وa'7A-u~VZjxv3H!|f-Wj!ʹ # Z\M &`n> bLVfP%FuDTը_HܢܸH[A1~y(P\0 z!0p}MPKҌ3JAIJPʕ5+$$Aya6#dݸSu=69| x>`E7Ō&CJH [aam?WAh "w[ҏ^l ^ݟ!`=kuyL^bce]f*DžkZm1 f|&NGgq`r]{ ?d5.@8-AN"~?ġsWܱ1`6?5{P 7E&z۫wp_BepӖ`Dcy_|od,T(~X>{A<:͛`ng5 `_l~muQ|Mө`ʲ9a W0t| n"xD? ";lt5sda  iUxV*eND`0LƄ^hߋFaB l CFP)⼩ ľz@yd '7fqe8RIe-B|}m<#/ U8Σk{">5 Tл- cא%,0@%0(;(Cm6,ozp4`h.d17f .\w;OutF>DDucKg;7Y.PBs>^#e. $qѻOEQG6@֪ӣLJ,K;nyRxT)'.XZ]XvBW%GtaN^UyîY*vde%5}Df(Q+7ȏ(mK<ļ suuxUq>]ܭ%o=.<(O\XiMx&zoB6>jqu.xJqq#J2r tܝvtڔ=bǹIwv~lw ۠aE&Qb\ްmLBc]I؎X1d."^O\{\}=_ }T m멯f}wwn/BƌCps9U&; p5h/~7s7NFBKIC iU2yt䶢Ԅ%T.}[?"*61ŠW⋹+2m,vG(ZU\M=sd‡u]O.8tKzʥ}/gw|7xcKtt`RՉD~RVm6ËUn~4PSe;YK=°ĸ~ѻ h9ک#5(]#;T󇟣{ pӴhJznK".`޽_A-D^~j$J O$\zwܝct:W2,wJr޹'=b>65 (Ycawsa]V}eoíngEM Q%y_Ŀz y^Hi@Eai-!E$ GKv" nE@+[kRH]Z"N+$ѪzlkM%EH -&jѲ2/ ,=لCX'@} ABj$Ǥ5k)jMnE h`#h$DH3?՘27jy)[2P-kh,ƠJyj!V..tJ39ͤZk qY%_ aUW&`$Hf#ZN X|" $_J@{0.sitn,[ ^nf4T ›0yhM#tʡ1|~4 ~-I?ZORwO!`4gFN . %Ls-\E3>ު}LB:GhO9ЧS_LCDL G| r 1u`7 nr:<{a.0Ԍ9m??' L0 t ?o0#3_{30GXՕRpYLPc; Lxع$T3O`f S{>"Q9 m!a?"WiR B}$<%}K@5 QK_+6pv0ScjQ@pħw{{ w$ߡ;O0Y SM@V<}!f'%`餎M2_fb&\X#2j &b#"Е0{}#5DF^D2@d~9qd*oqt&uC,`6CKx)9IIs%"_ PUUTQQH+Xrr(s_曌kw+^m)0MbfS/p K+گIpo]ѯf[G1dᙤj+#+ua½0~Jy6ҦJ.~zʬgG'"s٨}s徆RKRnE-T t~hn2gUdKMJtީ<)ٹWl=됅JS>=7tӕO5N$JHԱMLg|0zf|A7-4*J|F f◖);Nm/P'97/[K`c=h~yE컲Ti M/K~VgυkQIi B?H5:iseMvFcp/wKzt-̍ .47>Ӎ}O7Z[WcMbV3v!~Y?eԇ(u6e᫊r~*u7Y]Gׂmvo$m9L{+xvTD+3};Oi﹮__(WW^ޥ)*Z2)/ /x]&q||t~3TbOKj=_Po_C(j[nb_0zPAjSgpPDPqgEDKPO+zfb۔AR<n-Y^ !hXCs$;)H-Ge/1K6'dzI9oAZI Y6u(%"l>[R ,ۥ"q?)& ,C3=_ĸ9[mM@oi\RD{οQ,՜ͣbY[E`o ehQQR )_+UqQD36VH6KJfՋqQ'fWoibB[AЌlۀP lk:Z6矏4@;pOGL4&08U XTۀq1h$Ȧ/;͸C'.SBͦ23q>$wyɷKEȁW.vp =#Y;#`2 0!.@wr #2|9n3}iP'?C3#dEuW{W^0:.ƕ\jCc]>ܹ)~O:}02#j^j peG{[12W>èP.(mB$+oGlBbæܦ)1sܲ9%WC*9|Qu;I=>G7S` 2JU<Ըp q3a–5~hLBؗyh^x9Hj# 9(Ԥ"$>p *49W[(WzLSAlDZT3# ,&+kܤ`Gl`ua(S%o;z&}=Hh8Żq23 / +\R)d`Ao|<`)V\:s9 B<4`Zyg s5yX%*o:s8eD[,NE'23e8IG &)c 6 2:(ә4,v(N [y ] < e-eӷwѵaU6(IgCpƁBA/ K;Z#gm:>-a)^C 3LO\!''W\\ޱcǥK4)jbTìfx+&cЯO*rmV`8Goщ7;~p~[Oi> m(q>rD*-W/q:WRqfʋwpG)/4ԙ{*t1V{#lZy&{&WuUǰcA陟Ċ:޳|rZG . WBtt{Zɪu]4Gi_x-?+ "djOM.F.vr%w7,Ȫ>&]: L\ow9Eʌ~yiѷޯ;1Olpl̥u[xIWY u)1o'ܾ ike}OKI? oAv;V&Ӽ%. mM}=wN :NC,{2wWzdhkJR*G2zmZ=nʃi9>"eNO1t1P2Izk :՞Y_=0¬x3 j|m8 Ke_r6|ߑXV*tZmvu16?C]z -jAX %5vh{|?;Cg+@YܨY7E/qHggc [SajK]ZjR1c.P4|g~AA2h"Y|u2ƣ \wJ=˙;TT\c:xvī*tYWl|3 /#9 m~/?3yfCy |av%?ߙg=O,;hdzaAf.՘.=w.afLn؄U1X`U!Lzgn? v^>pPiS*S$ m_!K^OPJ&rC$+jh zlU?jj^|R}tYR)j'U հxAtVfn+*L >I{LQGV$]iߋ.^у͛_ w̃)tl׈GV<bKx銃@ 0 Ydn?& 0;Kx]4|CΰǪ* WIԑ#AOwygJ4nnj9_aM08SG4) y\,m:#KR w(m 2fe!&zn NEyW}[ F"R |]$O7ƃSK 1L_3eƪMEMOr;~D q>jEf+z- T|J#lXJvVۦp4/H}N}oR~-rN^Ӕ?Qᓭq:q{ܳosd'x,b={ BA5,߉A` 7U,sOD% .f|{M3Q7pK6VZ9# :jLƑ1N(Vu[ *N)o&Sfs :e2O3#tc󘉱8k FIÛ ' |oܡsO8T}@&$϶*ۑ5;`pت yrHwznt^1| ŗGeٔeu! F@x }V'T`"<:# @/^Mbv" xy6;#|d4sNq[q1xΈ%@7T3K\9sΣ|*tA=s''W83qձc Bbbb ^my'緆{QgZZK{U`v䕚*+5({]jV .gYbN*~֩vhЧ\WfNGӐ8+fmUD2Reh-"u@wsz|Sy9\!S.{Df\]bD\Hg1'sh,쬤(yG-MItM}uiNr^BaOQ?71SoUU&AF^.vъ#v3vehSǥWT;ȴɆQ[<'i!G_Wj:ֻēC{s몢]FR*Q+_4Zqkk wL?_XQ'ZrgR?vRXN]-c@ij6d,5Tj<6#{id%mhZ `_kr;X4I.!ȫg19@՟mu\.Uh絉M{Z4 ՍU?M/ ʮžǽ7L3Z~jλ]~>,2qc׈IkDۨBƉ}D[,;i~8NsN^SZ_ׯ[nҷ`4;}mb +UĔS3_kX-QJ4sދ>C<c^taKMG*7Bc=+T]B^ԟ&&m&D7Z.6NWPUJhF'En>Ӡڕ¸~Qt_]i0 gKÝ9BŲ%( S1`Q}MW}>F jQ'yPeIxͥMPj] =coF8tuv˂^_.<@UzlQshm*w9"u$-O><g~Vʛks> 1K9V9}eiG/_pƍC?e$4؞$~o=BhŞEHʼnR #L#h!Z>5-K$h?nyVfH\(2c~3kDBjYhosZaZ$45Dj4B2Hl6~6ř[mfKFBgsĒ"H2=X칹yj -,{@- tߞPYC_2f5})Ccڝ1#uƌ ˼ufV!jCq` "|{$2L>pوЇA4*Oq̼5n1,!ʕ7h6`M >8ǀ&Don2akߤ1?$X˭}Uug>bO.KE7o:=;ϣ''=U}ól:Sݣ̉ˣ=x`|B IdFAYbo>o-C 1>ΙPơZ4Z*>w\I_xaf1"A4dL1֐P4Z B`}%Kq;uO=D'-Q/`Iq9d[G~9;>d95>vƃx\! 48~p$2M"> gaMFȌ@o&,R|ȪŰ@g'AQNE%@˵7=2񨫸O dAw%1+?挾0FpRz_6inb)=Fc[oRfOCk $| #;`~rT$< Hۇ08~qIqز|0z*OR\u?(hF~4pt j¶; ÞO3e2p2݁+T Zf_uɝzP w RVod)}Ųg R3kx_yꝐ:y>N~Uxu&ꖝ׻ufCU.B޼o*|1Uawkٍ$nt:W=xs |z^Kv)Cɸntzbb+Ȩ<ώbv`* o-mP]{lFZO,VT5^wT òdu{?&DltٲϷν7?S@+[}΋y1 5`#ۃWs&e2FVVּ܋_=sRuVѮiBY&6Zus#\eGkhh8>--fORܺvsEWVw;j>5OWFͭQ< 3f~cB<ʂj=^{8\l58`5c]-K=* 9'/TMSJĿ*Lsf[Y4j6ҕ 7w4j#m&ҨSʫ?u~X.nCLW)卥czQ(8~Ω˅\z.cutvჷBۚFgHo4Zsm2_ߤVf$ZfDR-"ZuEsS?տZ{= Z6s+66݀,jxv+^<1%n {54?QPIDb9G_R,mFYՀފWV[fXdukEÖbB4КTCR-Zt [-HAIa+vN4C$jŹQCWh߅  @ Fr,; >laN4?@0 =]Oxmh58'! yy $ѧ#< p) a*$N}]g;hjZ-3^Ab?"7ayIaM0+YZ3Ӂ *׾wdZς+0jN}&L8⏳?@Q|ATFj `TL0}pj)vN ,oDS>OafdvނmW]3oCr>BWu8}dcUǗF o0prt|v/vh9}`/NlӶ<0 5/>cq=œtZ 2`lq|{k`U_<>5CMtW0iC>4戬(< ==]/Ӏ1 DPXvaێ!sn71K Npx9PyCH~z-a3i9DMЧ߱GA-d8nS) k̐|OP6 7 Fts $q%.m qILg'Xk3@TIpؕ#qwٌHw^ (Ff#ٚ( Jʬpa!F+ix aèƸ%|~$@SPw[@yu_^ R#ȳBuDv#ewP'=*^ C6g.1K|j@ Z8Y4`^H|Vՙ/Ӵ}:_%J]d𾛥ֶ-GHl<jچQs-D i[+oSV ||aaz;us >xIΟww1>zoőIWwz7 fyׅ&nԋSO0|C=*|cF/9()w{@Dq169 _mxB^Խ (J, hI.68`ڂn~ O(띾1 $Ľfvcj\vkW5~#&:*0;nӗ=e&?&ɮLS;l6-(B2@~TwsKˁŪ8IZ8󆬦]G ?m2seY1l6Ot"w4+\5z3Y0*Q⨛Ìท; UP05PB{IjΌ3SQ"UQ)J 7;7J0mr%Ag%S߫ՏQA:͈v׽x Jd%Cd7u;b-Įĥ_1# [ 6ZpgtfEE_ɉa_zXs՚^}Nc ?VH|-4??h*_-\_ݗ_tX$ʹd`෍H-,Hf$[\ߪoZPS Kk喼eL7lUi޴j[ ߥ{V|[ K3 ԲB7 Aރj!h! Y*@$ZD͈~fDhEbnѪNQ3ܺ4ӢM-o2y(llMUkD%425oi,m27o"4HuY`Z+.hEa$Ӑ5-:߅ 6,$`졚hKN<+B 890f~5,B 3~amXps[/8Թb%{;]e0^S)(tAYp&c[AKELUG7yp @ S~rpI0߇cܖi@&AQ@BpP=ӞxZe%1|=ŔY&[ T&: pqp#cM.cV 1@O 2 7,>O#r,{PQz/#]KG4w(12_!ei7@|i}5. \>̽(l4a2{ȕ#ʑCŽ}JE7#p?t Q`?*32!:!gym006;whhkw/w n(-{ހ6QOZhdC"^rgPnINy֝% ƒƗA9U|$YЃh?l` a,|(4ơ r<087ͲLD~PĆ =WeDfp9e#4n aUe3I(EݙQ7_m) P=/&•M 0^MR+TܺAy̖IȍeP2S /Vf v۷?fBڸd'=c\%IvS8cjO4O}hǒ.= OXT0]m4 L}l*}S8p~S2"/4oHt4mCHQ- 6<6!B{野O`]W_OѺZ9=F%DFwI|_}"[5⦉j?;O?m]:uM>S RE1e*MwQ؁W`e;ڿG*끭]Iy{M/ҩ}<.*1wdgwfTlX͝r_k:}핽}L s^{؞íz3__i/%gڽVK;ܥZΰYwg5u35rʂ9br;| _mTL7}-$wVb3j;ƳyU_GWݎ$=aҶk(k~@x JT9=8A>B~'U#]vhXUm+_awÚ%5NjKËW抔l=~82'5bѶU^ݸbS0:%M+mow6eIߖ [`=FkP+_X'h~X|V(Lfn+"R&1HĹt3 DRyIk~H9hPiA%4*Iҿr[Nַ*D:)O~nT`p Pq[;pM%삃sˑ=)gӲu]a}C܌xRb@UZzeܺ 6ծ wr|P&~!kyB7[*̮2rz}SsUR}vS#l. 'fՏũN*>==Tb,6\l*5:/ ir9*SѸͧ|u_T<瓁UÖrc54rvsVזfFzy-#8u;SۡeeEtB'mô:μCS;$?.MkDMwS1HpBpMhqX]!= u & LfW۽?tb`i<, M2Oy/;[lrTki_t8Ql~L:ܨkS .ل{$NƋV^(Xk^ZXig5;uvM+[Xe4M%ڏJ{_nWc/>š䖚ZK{.yU*Zպ{:)^yf5¹Uvs1pT`5k8#$5N2f԰lZiҡE-ڏ^}o BPLO;߫~ޘQE.Y8«p&Ff԰1c:Gۄքxw? lĖ8oz 9`i\2`~ 6R0הƥC'U^p c 1Ÿ`6a8i68XZہ5~uhvLŹs_0Q`hն#oWQe0VAi[9E:D`n7=,5,r}Cb$"|$L?Л3SE ye_'R9S;țnU'Rs!/b؈A\v (S,8Մ:900N@c"pRԗ>L ўBL&>t8-p¾d6A| 1%(2Z4} ךּhχ/7y0I[hcLfmAhѶ>#l{NTGe1/{EYcM|aClT!7guQ6'3*ɝHxRh.Oթ ƚ_tUt*z[R/aÿct/%2,bv['ϩrCp༆MP)p7he宱]OWck'B-+ dWp(܊ZDEvՠ]qa:cͼ{zEe_Wc@=ڵʍ޳:l~V3x,qZR+?hZ.tS6i)qNZz)ﻯ^Fī{"B櫶\<5-Nr/껶}wqÌPZήT~@ #~ÍO CZ+KZ9Wۀz^CcR=>9.ݷ&9ottRj. |0mz_E sGOw?bNxWu;q]ijrwQ~(R >X4A~vO>\o%)['h6dO"cHVU֩tK7r-n^QգdJ9NMO`E'rlw=-,ОP3c/~]zҗQqwv)pCWHnߗ74ogᨿA>"/).VOt[LGsbVRgVDWŪ!% _h.py͙qBYy>!Lb㠗eV}Ydr*Y@V_ Vl=>I?Stڗo4*}4fY}uoz>?1jeM_o6eE*X_4NL 9pį\[y 誊 r4p{ʾ>XzЧO)kPlJ.of}kO% sY:M|˧mCZ}ZEQnMY%Nt6 M h{yeqiFO [gܶϲ mxemwb[a}̇r甊*z(^cz~~O2%X@\46HIkzU甑~\ ~5*!n$$hjcY0eϹ?,"hjX.9V)d5;cӏ&5Rf(HGځR3X4.qLGTHXA]HNN4S$a)?|mLiOc&n1ik.h8.ee㳻b~{-}4XɄq!N+ƶ|y@2&XbA<=V"^awV}@' x"T'3o@k0,F 6V6 |\p~E_y%m7b3)[J`~`$\F]BG18z{RoAcyƄ5@yr0LfO"0҇7SyG&d `e|B@{N?M`Ow#5I"&BRr۵@Hv^iG " No+kLu^;6~~$L' D ΃]:cCX>NӿsݺNPA@4m/Ćpô O|{? 4gxa< !TN@"do>MuāC8@ r{?qԬaH+!AhЄ8YT^p}]0Z >(WwR2k72g|SOʳ(L\x XLĿ |LX]:&4 G7OA D#r*a`: P9;1 i/#vSBL TY';1+>Obx8Yq.;SQ $ !.7_4o` >7"sh)OWwR&tuV|A0eiʧ_hL4ܸ<$IOs)hQi"/߼$]sYҘџlECv88N܎lj9Fժq1'@CBvkqr+r^)s|`FxU٥/-3wnQ=<\w/C%Kzwܧ8Ǽsgg&fED<.i{czޟ߿,(HNd'J//MЫiJ}]nh}u̗c$ޝvQa-J߼2P9_dzȈ;$(bU\` [3ﮍ~8Dcfݣk-IJ}ا<Ӽq:5V%W,? xummߌkǼ67TUR9T=5`Té!kB#7t ?V́:]Hx m{gٷuϮ)ǿ: 4r- nN|C}ek)sEg;X`okx ҃pvmh-F^o҈kg 6K=aA1.&#nRmǰżE6g''u+_,LphoG=f:,X>ѯ2P"-j孭' c+ ;&]yLWEΙg %-:gUPi4ei/FERiI);>z4],;Hx|}>oߊ}/*ܿ:2;I4pJ$  H*D _Ьv@?@oT "Hh4O$⦐F!d'4E jT3jT'@4X% @* /I%UIW E&'ʋ~BS++5Zz_VJuMʋ@^o񿭩J:PdX a/65Ch+{. TYspAi 4^⎈끮}7#3P3jSX Fer{ US> LLIki0.x)=^}3Ym#,>ef ut?"@"HTnA| cb0c¨%R ;3?<@ zGO, 21ہZ$~:lS"h6MN*cF p ؃-yLQ ρq]V £LWT0`90QRfc^M`\h{OGg dCcʪ 뇺Z]H`LFGN>^|;sH{a28M/_@ 0#zxLjk a'Ngq{Wo'>Tb ـ[P$/&YQ"}ʍncz`*δ;XiLo~iY8"~xUU,|KwS'<|ht*N"}X8nOX[t7s[WSuO ,pqaݾUvEg$2yad y2LRbSoa/یun$mQcqG˟fÇ{t,xL*0U8u`a6nžmQwH*$ڷƮ$[0`I'BM !Ӟ6V\禵څ?ڍٓ\w2ÓHIbAx/yIҤhestg:^)\a~>m*G~\9R,NsI[mYiAZOW6!]꓾n/nqR6h׻,gRcq"y] /$2ҍR=Ql}g<|S*(MZZL>hאgLDWa-f(\n~JE JZ)r۫ۮN"-uc+zL&ʳCL%Oe=>_nW\&O6ceW]U5xSz/-g(NQK%X3Bd֮&x־SPfpȡECC^fr]х Zgg5,7Vl]p@80X٪/CiO[6._>kˁ: fǺB&A|=^<w7TwIP XP ?0p1W,M!xp3?| nn P4#&A34WaAҸ3"5ua!1'@K#lX?Ql `yF3l3_{1@'ϐ@ ;4H ACSH$hI ԇ;U|HF)o -^3~ 3sg.&h Za]VQ9BrR¤l\:FTōv4ҹ- bA*Akē9K[Srn+?wa!;-&~` J3{cu& bOx](wh]@[u8˱O&Ҝ-0β 096Zs;fƥXp[H`lJzNGhͭq 2i 4&@q/7d5c؋2#0[^^vPUU֭[_3 G9/_ 7Oj._Vl笛苮*ޏY#au۵Snߜ4fO2X6Z,7Oqsʝ'C\dd뿦J=]͝_U0fT~yN_̒,u*)W4@rX[M'{C diXAfB[ۨ wbKW?;H#]/ZǥGm_KdBg(KK0:vNz)> j2&3l.kG(?9myDbmC]DW>73DW!nlvcvϻdoC b$uyw"ɬ'+t}{r_BְpIex9B*_r-Z墪6Oy;vֲʫzGLE2+llj^gթ9fJZGOvN+5cEgW+[qTlIRN_4Z'ynWXoI;#$6->>?^u\]]HtZOM,L ?XVJdI=!V6 ɔʋ<7?O3D!EI$&ϑHQFJ&d4ӉHB"Yg֤44-5OG iַܤ!ɉVHiYHEZ ?)*#|bfH~R5*nb#R]"bV$i. c6/2rK B^)/E"b?[њҞO/JU͚ųawA*XQW? C!p oF!Z2h=FN{ĺsۖfL3 y 6y@A D&& (u]YЦoׄ{O/DE پ81Cbƚkۦ p&Z`1vjbn;xfޞmBLhrhW $i`u B'l35A: {1W6v1@':̩#bD{mZ ~+s+@ DN峖w=XJP_mwc3 yI[)!{z^n13@ )a]C,ʳuqk9it G1?T c@.c`1؏ UN T<%fClQ9xp9b@B/v?۱W3@=$ ` m^|Ǧ-@ Іx{8v1'bD_8sND#K45F@@@~2jjjnnn4:Ǡy L=B'9 Ү}Է6R8*O yn#cy:@bmSWl \6i҇ul7d*NHuisϹWA;'}⧹3-qp;A#b-ꬸ57]}]7^KVYO7+aͥOښܑI\5[ъWzk<77O5C4l$?'9O.nZհW8Y{JZ<9sy`ɡ)Y+t U^{h< XVyyܰq=|Ix/X([I}0}])+,9ݮTRS޳6RCˀDñ/J&XȩW}w߹VVh_*;; ;ZNSκ9`z&kk;o-͹2lIsVNռx.TZae_SME}[ݹӸ˸޵FOo6z;v>T}!rS\&_`2Z&(%-ȣ8-[,1Gɖڶ-f1zi;VϞH*Ax<HθHc}6D|cH_#WYi6vq` :ȅU$XD}}^?͙^ c b"Jpf!` `ܺh+D,F ϢΙAx4%@~e:H`kBeh;x], [I{} 7:ѱ0aev~ xoیn3B? |@&qY Y *jR;-/k`^xk\O ӥ^&R.|E]n Ν'`K7[ws8\P@Ԃ}b8e /Q$zB (M WoȝBfQ=0}tzm7NLV BwFf&aֆ}f:uՋK`0>DG˩X`;*tWn 1GANCot$&h"LiwBD@88OtiK7c3h ufL@A6]Nz; )p]:0|oXq_R\6*?雌ſwzA[Ok ;_͐86բu3rHL7efZ O~ӭ}Fo^՘qnsv6װiS?.Y/+Θcf$9U~a|eNM/ 5,|&[+[.y'\4[lz^&DNf$kQ7O׌7xG9M[Nw ^cs&8kf^ ~mcZ8{ٳ U6;ӣ-ìfۨOk]=IBxF!VCeE\j:,CPf 7 m朜%{C71CndzLj- >\Xx~m#)δAnK3C_i;d9hGm|N^h0"̞6yOL7DP `05ӓvV9?DcUsp{3.VZW!v-V?#ڊ+^/XkI]n玭 G<֪+@md|ᗤ5$1~ Hw873W8X/TCWu\ֱN/ZΙ5I)XY zqibMԥ^ip7:{}h{W=zCv +EZU4BN9~uexDŽ[Vᡇv*t0cv?U ls8y:<Ŵ-#^`N?*s&q_9sLim-@wou ^*o`42;_ȺpjݸƋ7Y3<}ZY."MNvZ$3& /g?דU Mx[ ^={)}QZߖKZG|^ۢ}Z_6 XV'dRhКKI%G/}3Ϩ Ȋ՞ϑ]]㍶<ܦu = ),ouABFJBxjS %[ff@i첊=;.We?ƙw{vl.1j/ 30i-$$gNnW唤>-/>yr? p|y-UwT_/?׻Pw52\ʽW;跭5uT7iC=L^\rgVl?}袟8l^h`uV$c_iLCCڗjpGvRo7Kr%_ɔSTw;Nu6KssqedvseKT bO:$ފYC]Ue­e 3Ν;YnYƍyu%''uֺuloo_=<<7۷GDZyy[nUVV61p?F^fݺuu-KK}~y濧+B" X 4 X@$?mlADMwk\2 gpO^g#FI@\ qLHML:H K3 vF ͭj~ i.̐S'@̏xRy٦#;??M}|c#hZ22y&PTSVm%:E5PV$@ #Y7q4f4o\ ~ i=)1@N,P 5?.Y3.aL= nOX3 `^/Y+?FJlX<[ZЅpaPF,<,6AWAf>B LZ9;Ǒ =^<Њ -m4̇@k;V:Jpm UTUL9_Ą)plyaG´{PL2B,1;0* q̛[_J(u3z@3<!̕jZCٻ<SVGaqY$h i| [dά3 WU@^ `sP:eÃy]e/(7lKX`Bx$!,QQ\'\PvM}dXeٹTCa xAYGHe]Olk>lMs+Խd/`2/1[:C":~U%whg7=^  y@캢7'py ncg]QYYyU''av_ Gs6c)Kv/RGH41.*F+76 cӝjKA4Q&E oZVE9rW=˧'k]CA UL7m"^ګBvSJGgQ5 kag}a+ibb&/2V^BguvV.+r?muՏܧܬ6s F" n"=NW [ ++h?8ӧIջɹЛjD; &젼/}L`^̒Ϭ%zeޭD!I:e+:8?z.ۡZV{u]5׆oJY5c_'=BZ_ /Y^4G=M߳N)x_(mvxO~Pлc{Q)+UF}r׋ag'1!xƧϱ:aw#{s&1I+1woxҮGkZy-ckMj[_s\e]n喵pO*0Bb蟯f,l>V#(^/5m^W~ҟ{un< ֙z1b{7@N1Fm>:?F]qHN!k]'O̘8!^B˩_?H_?ϭP3YGsC+% um @Yhs']]n,L٢oJJz}W/_=KYH͚<גMKaee2G?\ޟ|ٳgMd_ti&Y3!ŦI9q~$nVYej@PvItD'WgI J ]EYJrHuNi4O) fDJE=X,GY?67-T.HhH~:32ۄ楠% ,/4= pBHɬ"irUk  2ʩ52?>bX(@)9P"@|c P.n2U$J43c1%Y2Rj7 T Ye{;p_F8g@eM53^裳B9w7R&#W-wӁ!Wakto?BhB$0v=;b z`m(ao5p%ΉEӑwv_:¨\e>eL?|7aƆ~[(x &S3/ L+P}IsytvD;r:P K D_FL 狺.ѷ^̝h1nuW_{Z$32;!&B]v!ǐl -ʅtO#8QA{s9. 'Z@X LLtF'M>dN:% Gs]0`,#ff3{- &>ƥ L@.HCŕMA%FmbA^hv|(q@zTyI@ j9x^ɴh uWbA)w N\ZKQ#/C\N@`f|#tdx:"@nzՊd>wĺ>D"֧P|:oB`0e! P>`6].YxӽڂRT"[9b{ 1 }+fLCAY Lك14S(byH%tb*\J mzy{ -HF..kcuK&| `ꀁ?2{mv_6mĬy~=w> ӭ{첶Aڵ a^ flWJ2[#n \y}eLS_ߘWdVcWE%#J-zn\gI͝+^s:pE`MWyfm'#;4E8qoABuҙ!ϯ\Q?;j+}ڶ6c`ECO3c- fݫo6%ٲoQBx%=]¯(*ܮ=)B R{9cc8+^}4վ(XcйzzF1.Vjuc%?Qa{<4ZNЎm#߬z&yN]qP}hū=XiSO=;rD-3ݫk8VaJ|3F67ޯuǦޒA93kH%wܐnvӃ[95kӇv3aCEVQ򗞧Ffsw`9$ V PU4X,lDV'[j1&~$m 2q{ƐZs9D?T ' ܣD +H bA 5DUeA^V'Ue@E$Pl  I444RBn^YFV,l@ ~S. keVZkꦨ @ 2&ߥiR] $AYlYP(5@$%ljQb D|J_Un]GIi 9>0AN߇vSFg׭ Ar o|}ث :@S_Li|foKO,r%n* 6!,@q8SAy@s \%? 6@@~\C(Gqqr쇗ΫNqg`0=k ero V !q{)vP O;&{Xq0Qs(+3@oF<ؾwj 2y2aq.g۾hleǁ#T`d?ܞ*T9M P8TɕQ5 D1 y0=]SHd`j4cεA醑}m:.Ԉ;L1Hd̏$:࿡KMX^'`♠یHqT, 0_a"T=)HA,هk8ĆQpjx1VLuԅ Ԁl bODJ֮ ; F>}a4J8J9_=B$b!c)O؞@&;Xao+VЋyqI-]z{a'G^"a̧GjtE6 "C :w1v oMm8vΕx\'WpvX ] yrc>ECPt l;,ٖV>[I(:;QCqN`Z;gj >wFhbἰX_ܐwo/.E O'$0Pj*M}~TI>8u}FN8>nx&#}o`/O-EFjLϸ~Ȓ Gؐ Ybsu<W߷匚n?RiwoOuF4SN6Zя/F U}h ~J)i'hקCڧ:-k}V_U\U4нSTz6˲4Ze!8>fl+l۾ߜ{S-W5_0#]ayk}{׳M\Ph_ Lѿ^iɪNOB'IJT]fm@=Nu"'m/Oeyz;7/'}je^Թ{?&S~Zfզ j,:||+8U+T>jneM2exKnʉK+B$}'LqE+EGԠ 4aZ93v _G&Rou3[ޱʗ$ώViTOSc]f%o^::dQk~m3.U0mg[7!xԸH/WU7UYR`$8jېN1c3~_*UWw5ɢ\/pɫSPbڜl/[.&+/~՜[Sŀ_nݭ$ߡ]יMQ(ŢTC^aCV~+Sé rV[|;VחܶSW3{t%+$͝8QGs l=.6xMg`bmyF]ode\۝4}ޥ;f&FNEeyQ63)oN8zڳ߳.7<\rofU8Q_85T[BhkDVm>w-&ڽyDR4ht4YQT}ơ5NII;q!ֹΌܻwoh0ɢhn|ׂ<Sd좑 h1D 7[jP*PN^"%a;3ŏG{vRW B)d_1Q\{<`bQV$b>>bܴ6'J3El`ݔv9ơ; 3'2q9ZOh2/;Wn]h:3 6q8QZ(Q^h0pК:VKeB2 sj*.fD9ς9޲UvŌ6}XHHOkLF%aŠѤcgΰO_xQ蚞~YKz5Q=s:xLyߩkYՆ46H4(-7&t购__j8wGTh+᝺tȎI+ͪb:J 047w]S=|nEʖmCʭ^+U\iy\f`SnP{'Wg9KkV(vS3_5;XL]4FJExC熆:Uβ2bw˩W(gj9ã<{Ebܺeq/1x摝bwUn{ݭǽ53c{-N͞[AĹ>v*<~8s|:!!LWx鼏J :X68I[VՔ "HԱ} I?knz-uE9+=WrCtIVR#ةhK+ѵ4w]g&I"|mE-Fh--vv˂v7 {D+u#GY9[q:-!l=v5iemԺx~|R.LB"ӄmȩwvN!#.ToXMERtP}Ɓ FLW=y|AOx(?*g!F;ۈK褭ayu]l߂]ί?y e]x3šZQ/'4aP2z~pDݹk {O+dڴA㿞=BvX[[1'[l\GGυ+jͱf"@!F炡o͒M)* ٿʭCv 3,i;цYK~xflԖVc,RtQ.GZh*6ڔ|F˒j!z3Qaý&Kt$q-vbQ˴G/Riq4)6M,#rYo3g6fXZZ^~w;{FIэ7g͚}gI<u֍>Gsرc~? X,hV2, kzZ<$BwsTڅ64E M|Q?&#?XԬ}4KeozP,lG&ƍI e@ Rh.Zi-KE2fbAd@ HN,y%Vٮ"OF{#HM],l B H psP 4n{uVt_YPSӫ QL}? 9Mou⟝5舥bXJCVs⛈B1"Lπ)0FsqkQw Ե(ϗ; '0 B#PNtO*Ә+ rP]+'qyuX~, @P=~wX!輎&a>to @'{S#~HkDГs3mn4:0hfWzލj= \Zut|/ L B$e/`{\9ףM/'8ё`,0Q{0hbHu 2WN$;CBAT$ػ#S(xN@&{1v`FWXs"#U_O.Y̡L՘|V`z#X=&`۞u `wDj( ĘD˱pb{a>;H* '`6voΏD9ȏN8DEqiFá`\de`jK\Ur|(^:EZsM[յJۅܟTyͮ]df w0*l#J-Z4jU~VY;/[b^4j*?'VV8s+q}vsu6=ۈ'O>jʄ* ;间1NnnwuyIp}~cN_ޛ4Z ;:mL_\ιJ }%x. ]O׵;*hTI*f-+}5IlNlWn1F׮;# {Z`UAjԌŏ2Z?q' jǚFjP)yH?ZNymnV_.ՒQhSp'n[Y^Ţ%i^1_߷wO;oXW%螏3;Uɓ㕟>hԶ^1W{Vt+2bW뵗u?fe6IAf_X7:]^5IVjgd K8{yS^֨w_4;Nw/w(zٵwͱ䧉+zn[G=Woս1yo& m#CK|s{l9H}WnT{ ,34뀹m7|>>ަGիC-{z??cuBmc^Ots<jƨ^{*//7n_S!-Zi˟S} &|O9v옃矛-]*5jԨ?3f|?䯵fgg344lз.LI)?K@⦀ PSPg~ i"?H@H h+&A,I@,i)deEM1M!/+kDDLb*ivIB;B()mqT/(kljYQLE2Rtaӕ3'D2u ¦mzٚz ,hFGBM$`AXb@tl]MfIIe5#1Rvt-fY(r8^P+?]R4k#^a* 겉͗J mqS&`8 r2\ .Ǯ<<խPfb7q;S\3C@k\8Cy7\}oa͎h$[_b,&6a,En{Zi+2b CS]6 +SVe vpߔ0OzvSŽƃjlKAxfoe:-&檩-nXD21dVX x &8&o[}i(a+VP śPx]Aqåf`iGhj'$[9 72'gp#@ [vP`.yZ3o0h6 ' SΟkIϭ^ع=x1hSq2c=@h*pL LJ 1>#L\Lȝ]O3xO}`$Au'Fhvߓ@eԴ b{^w x ֽ'yPZ N4kV!qh>+r>>FڃfOcP9p l\͚#ݖ.Hdl ^sB+`@[Va:<R/Vѳ*eMҺWLrFeEXgJmp]F &^m[;K^ica :3ŋȪVvHڇkFWՎ75C* *qDS\4XK#ިԠa^Y3Z]K=\3 #{v1~GQԂ2] +L*U;֢y*Dqltt5/ʦyWmadR &SM6)"am]4Zd7(9:i73[k>߾j~՜d Y1%4}^qv/s&`ӥ TsG5fN+cۮL 5% > YKOag/o%]c-Bt4Ejٚ߾ϊ[ZxamUm'v4v]fז]n5rO7͌_'2+ִN#5;qrPK'f=t**[tJzi߲^=̹cͪ`އk=2bФrm3I;EJfuXߘ/b [ ]Υkl)U&N<:l6xƘ+*zxC6IAoVoŢq GՓvM^nh]S.:V\K0o>bR.;p3՝2gJi_C.FtL|^0-gN˵_>c~V:&>Q:Ś \t܏1oպ LL{mP#EB7_ᱶST*Z1 |-p˭$1fM:<([6b˺!DI!ݝsqww&A D1b(ݻ!wgCS]uSΪ׾D<c?)lA"([0:4<+g|ms#ACI p:C` 4()¹eX>`lާ2~@K?BG(lLlJmcq0'+pOi7yN|.Z0x7l%?a>[Xܭ!3c}oA[؎@@|wG"_@ o *0c .]|)c`8< Q⏥g|ȴ;<5(+;+A if }>2c  fx|QӶCdTf~F*x 0r60s$gC2@ӻe1_Ϟ[Β9T߁MiZEp4O391~q ]r{6,z'{\؂;ca.4eC);3>}L)b5D2K qqI;H^~ѼEP&euǪi>O70 j?Ěw;CŎ1 nTPy$[T\0}uĂ7ٱZ?ѱWz-F^P{'Pm„sʊ;y(aEj^TK{>eG#˴?|JXGWbeLV~̻NoxאU)^S寸ֿk{+&׹~lp{qcZ+ QuN&6s9yx̳_f7(׎=o`M8ut[ 1x]-O(ݥ*Ǎk!I!6PyJ[w)p*2ѳ1I 5'I'[@3վ@]Y.;v*W~Bmh'nۜtt%X$X.yo B?c``Op7OWrԟpˑDU2ī^Y_Z'm`xLf{§qgjo;uiYeYzs>[(V,xeJ[Vi͢y#NiߦWym5^9迫ZVt+@>\lΥ~wݺ, -nkGbo&~ViĂ Go-[Xsfzk;FMmm`Bcչ gYCoo;#O}""EC/:b[rkgkKF;_T~/NoR:v]:A_y .TTT׶sڵ- tR]]wįt̯4Yq)Ճjj$ɻ}WF5T jjUB.o qCT4i^`^X]#_2 &cK-oE1ɋH*M,_y I: %E(;ʹ/ ZOTʔ7AqR^jsJMPY)\%zF?~¯WWo@?vIHk<ǩ ē4ͣ1 nRY|,^"-@^\F=c@:Sٯ{",В ;0  ghAIga`>jCwh]`Pb<<:5KĶAJ̤ՀK-7e 9Of!0| g68wLf7 &/y @!Oj0'\& v٪Bu,8.~>ܠe>Xْk+5%>8 )Dbkjfʠ t RpH!8?{|>ߴ!p!,v! n+^!6}c;N/-rHqn6|V Ř_˸w~8`,ik_WerY |6.ǕJuGh%6^ډ-YcڭE}Me\Ţam @sOn1ǹMS?^EiɇdžHfCEhcr(;!{Qk5[1X~hZн=Y;dB]lro}n~7|p5q_i5T0?ΓUSx0xD&hT/xYZ[LPex˧BS#}1~uuyKa:_\3k׿?QQRa38澍01*[[-:P|lh#C]k;dمi#pYlt_㨓ui{%ur|7HZŶq)_:y~<=Gq fe8"uF(٭)'x%k} v|PF-R>Y=R?9-N~_@j9biJo Wohx?;XsĀfPc2yH3Qf"0 m`\;DAZ[ {¾mK:|ɋ1}Ux2®-&"3pʈ- 4P4A hN~ |q=cA\y7:1&Rԏ'$@VѮN`mZC8~R_>D0N xZt  O~wo*3C|YwC pXt*?X~H@DF/~qZ)^ $^ &P oǺL,ܹn?R|\=Lm`>Jژ 7G/=G[Aɭy޳RFWʀԏ_=2os6$T ^f+ި}0qZ'XY8ls۩+V<oa]|ޤp[|ӸCyƉ+Ld`~ck{G"6v#m`Y;R4L{-Cq *z̝0UK8#ҿ_tBBLKrhL0ى4kkο __6hznUi4" NŖ/8swXܲ<`A])؟حMT) ٷMfh:9[þ~Q3(x\YeCG+⨑Ijm?3ZR=]m7'+1S_^Sp剙gepf8@wꚛ+:߱GV9]2b y{>_%l݇Ln^ixuw]/_hNlwaCUZf*]+rŲ on4 ;"]d²K*HI ~s "(,s-(CߎM=l:ZX4PuؖV]ɋai?rܪ!rtWkmL˨D'hSrHHP;2-KV0~8bՑ|aVqSoI(iuA Vg+̯im+Y`Coz&},1`%cJ :hW=g@/CmyuՃC ,oZzvFXF}3Tn)f$|=_`9ʒضSmؐa6<0n߾jL~Ґ'q}N%7cۈ;J# :8{V,~]ɑнǻvhhC &~Ri{B;;ʽVbXaiUz :y?GHyLt{kb3^8Z)} szUݷ$ڬ\M;-}o?ZÇ,}y7HGGg̼wޟ>mW_y.JKK?;YYY_+ i ޽̓$> _5KQ꿃!q|+G^$CBV]nP,%+jLSPBY XbHphFYUP\VF\&-.W KH:#r_$+*7H AdE K¦5\|ͯPTQVTU eJ%&PPSi*%ʟPVTm%j&ʊu  ,?b@$6E}Z|;y`jñ/,$pCg|n⋸ m[L4);1NE'"OZA-A|% r]A b0%@Gњ#x3iPVE u; G!vfY ("0$qʞQӴxwH04w  K~| 9~1`= 3;";Wk=W`THFwyJ6ɣ(<\y1]: 9|9'+ MJ_䖌!أ{cӇz/d}˜uwffo۟![@"ݲSv(׃[@ h9EW<5( ̿1vϝ#`yGfEBwݥ}ݱ ht= %o@Nv`Ñ=,bZJEAX*˂Y C^~'ErN_r KۛG:_IsT"I9޼R O~| rK{'.XVz^W<׌20GǁyiuW:!{fECVj⋾#{&|:YቷtwLv׾>:r^S%Wf&{S>(;d8hS%-Jh`C^lgrPyp˪K%8xCl*FS{[^(aD| NVT13bը`Ѯ2.3[]aAa^zs?QyJQt_E =6֭sumn.8.T'F!b9W-5v vmCo,5B톆BŇz7,.T;s;oVs\.Z-ɽ={1lvJ쳥ii2y҅ϻZ\>4:c7'G` j4fv򽺱z $[Pqcw˪YF:p^f;rдxHn"݆~,Jv|+"ygݐw~RvS歮]#~<0qFMBa+DmktJNu]xc|z%EpeߞΝZh7Pun-Ybh97kV& VZE ̀{$׋?tw9cNKy+=dyLJ+/v6ݤYΕ˴JiiVM뇽[mX}y]+V˄-|j(k\Qͳҕ.c[it赴߭CG,NnݺGݷo,Y򓝹~4JQQgV``3/^VTTu$RB,W@!L7v3L\UbG"5#  2K)U*nHhDHkI\@B 6H,7kIFl$NC$A>$/CM "qShT'\ d)iQ4l'Rh.ejV h0h"~i6mH&$8%730b,Eg̿8<_.SR"^~GzNv⹇^앂W0t 6> ZK}e&ghi.t84g3n3 #u5D+Jg?*{9bޜ]`? ,)q#zCu=`B/ߪDG`0}c\=ВKʳfDŽ}/S8lx`P|} bsU&Du9xʬQXwp7'<14*.0@_.Q~a{´:T{e2%GO76qbaW-+GV6+ b(P I}dPcضfYL+Üt+O5&HTbp=h3:!W/c"13VeP"| ˧. b.5sߊ0`ϼL nx{gAgs-* @Xn!~V F2 ](CR۵dK cYGݔf\^_w]IY{axxjqF; +0P{GUVDu<pz 5^6t&ha?Vl+2&ox s0wË纝i:g"e%0F@okqjҦij88`2]ObZJ %J H7+"=Vӥd\~/@̘ \m=aN?r6r-w\C(-nA`;mKl:5uKzeW֦AcW~,{Xsl㧆Vgqyә;֎1N)u -M2sbb:O$GvmngDl=֞_S`'tjβ\/~Q|AN?1_ޚA.zS?ExLzClD*˥5kr[IvC"ќ:7Meo14u͔M*G֖ʱMU{F7dX$9_YjϮ?b\D':uRcW4z<2ݦ᝻˨V zP.&!XQ8-C_TKTVT;L)T^K\n;+e>ilYzE=PeYJ'ε)uZkV}4>kg?R;bV4ḳ][wxTsR9.LjuG|]ELj{UGfnhBL ?_x.Vj:}-~uuyqzyX\P7t#[V38U\`XYXtCA=F~n>[6/ xckVOi'QisCG7a=_Η+=+m8w+˝y?xjw5)ƻF]cϑ8KxyVV>[}Qhch֔ e"jtySoE׮]2ݼys G̙3SN뀿^>gݻdɒ懟={gէOK٠:H֭[#Fh׳g[q~?5]f$n\,,1n$w R7" I%̿ԙ&rݒ;dM$iFME׉}nCT#_ٵfFJ0J̌pSqBXhYj5.RhlipIV44mMZX/ݬwmc[/"Qq-#7ER\M5,m63$\"EmA IU=x%]N-rfw[?M͠`%4w&u݋51PKܾ_Y#ec8,VwIc~v R(}iٹ82E$LNys'7o۬g\=^IS"m~~ee] %[@ك"H XLcXbh]- 9 и\x=ČWP A,Bpof[Ft_,E<^{1} A|}f'TQaj17/7 ǗWYXC{8ژB6<[8?o h #3IʝѢ|A Cv`0'` 7RtO7hQ;`{KV#۞ǀ32-=pxO!Dwn]~k9~!P3p^| [g B&}t;diIIPW!,'6fbZP 8h63ЎH04a-H}ա.{@s lK1 C/":TpծD/πR|M1/1nU CHv* DYaK R&pG?+<c%4n{ˉpZCbS~N%#ЀyC0rA: ܉W'QctU.+.s ;g3t8Եh!`~gn 03qځ‘8[Ş&Bk}_s]m:ZE?"xVY d Zn Iy5C!~؍!63;tbR$TaM҂̮Y|Lkiaֹo/lu~9yʳhD*K#uUo]VEȻ vyjגW{*Ś0ŪWtݐ4{w&(ϟW%PZgkf*?qy픙V [ap25"Q!*J.T~X6\o\#բev >E ?5يccLJ3e p0:gWMT~l~׍B?8}RT؍Kh/,YfH懝zT~n)ؽ籴ܖ[f"qn/}mjpc7u֏մ*zN- ?*T}Ofv%wUUNqmMUec D_|IQWMzj;+ :%/`hw<˕-; ߰Q_եթ ( ҂1{{ \Ӫ1,KZ أ.'gЖ㢞jamϵi EKde{/2Y1ɡ&RJTu@&$"S~/_[!=zt&_s~+W3۷tRVVV_|iOK.mbrssuuu>7R _WBttC?ɿ6V`PHZ5;^rA s l'Il=Nl`ш C>ѴB rq%3515cH(Ԍ@Y̍1Hؔ`iԯd E17GmT}}# =MߐA@0 10c?&!L8+Bz)Ŧ Ăf ^ n̉4'.]녿͆Z,V&'5fʅeհ5Cu_/17ڰMfvU(B]bj4oYŒ~moA͌Xtlyu\ľzL,OG/O@~Oށ8|3C hM0D/"1ojn[r nL:͵Mxʠ !W 1CELh"/(l'!fsp4hNg>d3PđZtӵ1 (0|@kX К>6#p֠`}20zųgFOen<7b@"6Rkl=x/H ~Mg%< \NW?:]S; @`i$ܨ5 lkd'kp4}GdwjbOeà\P֞3AB ǽ DɌTĬNğB 4ޞ4a@s|cPD@ H O~Gu 5,j~LBp!:0Q ;XX ևڌɆ yD8={!)'ο>KJJ$?vܖҺ <-OXlI\,2)Vc놻u[6FUB+z q'16{={7J@sO\M|eŻW 8JHE~c>q%S u?aAŹNf:JR^/KavQҵtݠNl3sB:,e&+_4߷͚{Z֪o_Θne݄>:Uۏ`~rXjo{MCplE{kUEڛ(tjhVW/tr60/1{,ն׊ۗdԍ(ޥ<$Ǜ9wifTu/_J/06fWe{뮋luFUA{w hHeV oԸZ"_sɨ^ʏp+t@E)'wSJGl\6{Z,ujAj?x~o^6 YޫHXܪʻ%eYrǎ1u Suz]){ΣQOߙ@36:v[QzƸvaèA:uSK`yjhA 婩m}HZPR::9 Z_ߥU_n뒗r6(@K-7:D$Dm-v8QKՐ12nuGNWRHqs$5v>)L/ !,8NeLNvb[W{D` gMfO e؏'RQvcw``d%J`'|g.=k&~<'0|y0YL^_WRNT*P' 8Ƀ@sY\gF-&qw5-WECέKq5p<ͪ,n'~qW|Ƃbˏo<2)LEM&S’A_vբS+y^L=<}=%zJ/饮W,*kpAR֭e^~]kxCX*xe>>S{DzlBM)k{jVir1Cӳg_WrT(>~OXO{qV<9xT76W'vvFr]ճ55^=i["Oܫ}Bn4yghfԲ*T:"}]u{d[=XknչMP\@+.eyv!⨌jI7[ʗy.i"OvP*{36)& ǺzO;i"] *=n}7am3s[5b͍^G*#j/>_[g KE$$fo]tvZso&}My=zvoS37o޼{L ϩF-T\5#)B$In4!)HЧ7rai0.;I@@"5Ch$M2 !%U/V(7hHYLB)/IĐ$la4b%o nxXL҄QH>bQ4@\לE ~ev$R3 rJ5M6 XܜuA ,b$FsQdM3q3VNԄ!Bul]/JCׄXFAVB\/HHD5qmY;q7RH/‹90lщ/çSiǠAI+|sp) ka co U6}d3@Co=p#$P}'@{ɢyy`Y^$7)9JM%|-J}:_F](K{mPҵX9򱬆齫.%z3lrmg>U=KVхI*ycU+m]ڝV*|Ķf 8ڲj|ҝ cyk]u k}HmtzBOn|iYl9øe|[_~>a˵I}`j֮J\p ~O4ڱmS>/8.,}c{ǡfO.S0sՖ7o>pJlUy ^*/w<ڝ3jvILY}oVAѲn*z)v^M@\NL2$?YKŷu=b+M;U qޱ׍ {ۮ} e2e³ɵU;U вQcǩxX=NR[Vt"cA`Gb6E鰍~t%묕S&fдeK'sJ3n^=4,EOtZ;௛5-/uk*Y|;tEχzxxDGGsMN;wD^?F8raUΜ93w?6(('/%mZs"-7_Y?܂7Iѕ`cnfbAAZ,r"|OdEi J@д3 M!%%n4I$+M &&H5ET]t#TIWI q1piIB /"Yq@JܬzSbi ej@YZE1*@MR] 4˵~rrih)+V4bBmeMQI5$ ,-JtfjRgׂM'̿:m]d7!lfn x"pIܚI7uvJN@jغRR.}JL}b"G 32 A9poM43Nq1@0e|#+yf9 1ҋ`CVu?яcBҦ\*E m; x\=/ )K7\FImuzrH+4}nL[M\~@y3̫@t-\._bp *T^O1Iڵʘ: (wwa iN`\"Nۇ Zd [[u9K|PӲLj˷3PFUП 7`~{[s;iDF p)bN2\ ѣ<N3 8&s'62V&^!4F/^yЭ˫ afs"`=$ mAמ0`7gHL+INaۭWy[&/dIIlc&k+3j@ʄO*g-А}2Vc/!ٿz<82nfc1`fmu`?xMXґFܡ'〹 i:u2V&dq)"F2'u`"`?@>X(e!~"p%M& !FL0ѣ4t4XTEE'y1!C 5Y6(Cwg!mx['䭄LO `:T3;,)泽zxJ1JepBys|<<O3:( zzw_:i]*=Z?hqDFp'=Uc$yd.-oE~R<ۖ7N38T|3N~wG mkfoZ՜ |m>wh5+U衮쳞X=uD/hAR>șhZ/ ;<"ҿrz5BY>.BFjql*O76Usk3LBV$<實~sY+YmzgpA/w3.IS"&.=Qj*RjJ _V@i3uܓErK_"p>pWǕnQb^ѷOn'//oa7C>"#IqMeqЛIlϾ=d=GSfѧi 'S&M}gǾ!ߖ| ^+n%ܡ8tgݶV? lLާֽC'i5)gۤ%v: lt[%tj:pCV;˜f ҳ:ȯ`C-֕wRZ ULl1J׽^m }e껗j(z']zgDwkbsO{vAL V=.Tuwqy~Ƿ>P6fR)3TzJWy/U:*a_lF9.펟 vm1r&)1 5'( W=.=GKlgG;*|JR$\e_Z)gPr%~sk~q \7<(2>@GQ* F+?Lqu|R̒o|E~tUgsʇΝt^Py+%ػӑ(ve\?h ߊi_onݺOXx'~ZXYYmٲ?  .4Y_O`ݺuvvvRU5ӰG~pF5dȐ)N;a>@1_7Hz͒Z/$x͈haB<;xHVP1I15QH䌚 zb/,H\@(`PBIsIu3wB{JLAHRkuMfbQ#}VlH$a:ۚP,nÉf|D}@[Jȵ@^JjF^FQi~^sH$Ռ&tF{ f"jT.+m FqRU_7 Moo5#jOX^3b4(7,h'1j,rĚHh""dLj99` 2}.ay4^g$Nܘ021&K5-9[9xy֠aM ۵GQՉ EgTf `.өҩ~2QM< -qDSߙz1e#ЃN񗙰!~O1ӀXKk{bg6A1}#`֕ 7q1縷j`!:FCȟ|t1֑ V҃ݗ񄬀čUqTLyCI#~~[HņK֞# h@(g҅]busCJ",967{P҆(V_=ܣ[`AוE`(LGK u`P̺r9'_5oaf_ 3[AkFHx4n&&.3US4;GG-P*Q žY/a LT܋W5W4`-ȠYV (`KE)ACTS. |6y .{fZ `Ͽ?Hљa m^Z|玟88GVP16B|q&P%y;r.Ʃ=cmp-/FZ$=Ƶ< F\m! [ 7lh~laܕ BJ&$бC[ A @hBK%213VԌj͎36Bjm|NvvooTxg%񛋗mqTo|۶Woo\6^Z:1؉l!ޔFM%f̲˹Q#}ĮhS9o]}0tgS7"вWM=y"MEJOٱx1Qjc-FK=go}ͬuRdcw]ce~ۧ *> (O*W5i"hV*#+plvЪ=lݙQcas\^˖Kqe>[;;SQ9_qlgkYxR^reΗOn\W%xbcW%>>jnӛ2:h[|gRo˃҉dz>myB[S2;8 _h ?N?(x%bvOswjS~EQN6A.]Pqs̹\㏄ZkCMR\i;|NXC_uMBbس,˭F+^N̿V|j66;훲d[DO5G}y+KOڢ|WZX1iץ Ax=4{c0例\o"wi|VtMSкMC:^|.8}`Cy~tJq5'wx{\n\=fV[KսbmayFBw-Q0y.1SN]'{Ū"]A;ER+}||T)OsLAFev/o')f:[qۿ;Z*뽬^r[6*Xt#lCu#UcP Lg&OP.Q̘7mN<騙RNvg5QSWڬR/V4f/]07Y(Jv% .܄(J`a?Kj4]$tCҁR$H5wG#wwM,Ч[3T``ݜN ۽0y(|h9Ǿe \ ^ :v#Z?ҰLSi[yc3L"( #RWXy1PPu&"#;0 r? $XWO{pٓ`3n.dR@D={^;WklQ"s [^wRi/{&RoEper{mX;Hda/[/fN ž5=_@^YI]u=o=8 i_S䴕zfC]YnU/h/UNէl^OY@#3>=SMVDb{VTN4wst]77uz9k?)b6d}zvzy l[YR.OE۬^ў;}Zq*Qzzu锠3!3}L+; JJ,Yt=;s/YN1*~+f6S3jialIfԌ%Fbi$tfC}/5- @ ,TӪ\!#7-E,A#Q<"IiBH EͩIci Lu#jͫOꋋQ33$KKʊYio% 5*4MR")͆za3W X,AjF5ll"cTUubjQi5#lL0Kik57 GĿ5 ՝~slX&ε|Eư#2“VZt(`!%'g%݉x22TuOD#enџIW 70X7d-KԱFS@5c3h= `2c8L8b~`rB0a.EG1˸p"8u+]ӋӉ^0*. j$*m@8 +lBJ=P`c*:'f1-`>L0oO a6ڟu#hPGpWdN"GIK<À&=˘1ւ4 2F(9v&$LB'fhhZ`ɥm0٤O:>OEzW^^^ Ư'244lp37_o n9s挫)oANk׭[);ꌻξ5Ub!p]=>F0qsT+s zGmbz/D=Or4J%˳*U1\BR+ʩrsG=Tͽ '4Ypy僳7磻j3O[.kom&^ tWe-F۵p`rJUQ4eE+-.ߘs鞗frʹBLyX۳K1yONyi9miݽ?6k-GNfFǫw yۈc%?8osXܖu$u738w_ꕛ*eěU_z_ϝu\!zix3sڒԜclYܕ96C~(6omݖHȸ,f@m ÌTxjlbUk i=YvZsQ?.rbT͞:<Vv9-1bh[m$Ge/ebN7o%\XLQѥ5T9R̨7ss1̬2|a+qjsssHf̽G]l 'i|Od|Ki w}r/_609f_=7eV=Ń&*L]dWZZN8VɻCzņz;}7R7g_T^=uYO驚6e#3J}re ARJAIqH'tIxu0nSrhPAڀζ?>ykŵ uOڳݸG;l^WJmOS| }j8^ެ~ΑoW2:UCX(4p H>Aq˹IuTD՞=nn:z ' |+Ɗ:+$zȏ.*rmehWV>c4wtI}]^2Dž'=>k*%-M/L(]A7/Q-v+2nI&9x}Y!ouT}Ru dgehj֨_V2"06κ'filԡQDjF'Lْ B'LcD_Ukңngf WcVx<*;Ás[P'0%[؆{$:0OLȬd8;Y뽘'7 0fxEf9HH,0c6]pƎfG| ~!g%V*y[SԶ޹2qKBcf'5R(2,ˀ 7"bF>)js`ĮC)| ɀ%%(B[ $xuW+&R #lxOٮb@&z~,9pJv31Gxqqb'\r?P3M+Z ֳ;k{EnS`T{^]"g+N)\E+@`a6H}&S >8okADlJsm仦sZv4L/\mQy h\nӶ尫+kڏþX|( o" 2@ $rbLTUn`91>{貢;"ީ,B+hMF6c734;+|JFWҢjw Qejuzaw!S:MMӻR!qF]Gt]Jr@Vj'l?Vڴ|TYOId {50y/^;{±jlնg:9+>ghY%wZ_MH?#nW&|Ӆi2N5^wz.{"{kߝۗy-it DWz/&, p_4~5qeRuҞmDqҥֽtj@RֵۤљMAk} xxŶ%U7\.(SKkː75ը/7.vt/8Cc/Pv_o5Ȓ|7In8xYʊƮ.W z9 &C(A\Q",CyJ*ڶy윞^ C}|nۊ:et{ja>4ͨ^'h/?BH]GB Ԍ$!A#ENIrE\D"j")X(hUӤ54"jōz" \ߌ!⦙l8nd2 vjMRuoM"|l#i 1{ HdLX6 5#gi Q35~DR\׍TNr ųQ3J,IJM74 rv6k,#5=v M|" '͙q[gd0v+@…B0tat&zfѰYTDϹ'`{/ bJʅ-z`rl,ODM`TCnIv{r>> `:!i,_f\+et.HpGg;<ըƜz1 0-e M|x]PX@W#.0XlZ~\p`@LeMׄw߷WLd&b ЙZU}:mxz8~`bx`|;o#XA:8뻭w5]W/m#/.%%/QQQѧOOFCB26l0qĿ=H``e\r=//ZjMmt.VrZb*K[HVR7&ɹۃ6j g_Pni$Zk德޲CG|gǨt]?=s~Wܐ,."o?K ^Vx{mb7SzEQJNѫP)+z^+KIN4zݢ[J_;T'8uTmU7{z=}) t՗]׷zS7_W`-R0.yiÞx|j3=FYlOmjly;U:Z *rx9K/r|5dF灁[Gt0~D 7{2:~~=?@=w淸R9:ea7y/"/kL߮6_Rצ.6U6dpư̜t#zFy Z"@+/Lzi#CH\PLCh+UX_Fk:ʶA{^=k>ck+{Rn⛆LA`eK܊n)JH{Np!O Һ+ t4uFNbt۾uJ~]̒qN;Hjg{}v}s>X^ge&lecq5vw;^Hn1y}kNJW/mVn9Kj3qz>Q[UR4O$\(9Pȥ7LCH<~GUd ʶkQ7+mZzE=l.,? M@JHX$/KsM2"Ȉ~T)5:1{FjT P-uF?`5v!b#HD8ډvhHD? bJVIQW)YaT'I.%T(@PjHHV:q$Uj(֑\h,hLIJILBROԕU+PCVVNPEZIIaT;98EP=PPd J IH~Lt˫f (DbX u+ohSzut`<蘂W&y"Pe<@Bpc)]0ƀZ4 F)dY_D`9cҾk͔C@]h1EtN>%rރ(':fA&Ԑ3Pt- `^+0`Wέ4Ov$ jL7|sSp`- @x4nz~nW?:c|@!2ʟ\1GlmEx1cwfj8]>/Q@!y v4cx7@F@;eͅ˱s3s:5@?RD*PJLg0&N2,/S-3} w{ω1)JBqJDƎHaVLkı}AyDiBV0-F7d7CV"S +A}sA6l …a4Go[NII3Ynn ~Q?6CCi9Do@d DU zP,>Ru7<)|;8rxPF&"0X4q٠AcCϑD LT@l+ &%' 5ۙGFDOh2.]y:+DgF!Hp< `dC홢iw(uzSm6rN02Aƌd7`ApjYwZjfƌ ׮]i-}UV_?s:uȪUƎkjj𴡁Ǐ+fRRR~>?~F_;k׮ 4Lי\~3f0#;xp{|ͬ4Y2-5*X.,^uTU1Vf8pMi[s׏5`\齝vp&Oxf>j3c#(Sϋ9%;MmeS?}cޅ;<6dsQ۲#sV:o->?+eƞL4q[=P {7F ٙ۝ e!638B*vmcZfJZ\WoKJ9YaÛci9-4.|60anONu^n|xb`w[?OTdS4E;.UQiAxU*_f\j^Y.hn=sT麗?J/|*RfY:虑̈́y4gi y;[53d'|{w~~jME6l1xdfDAֲfOYs鶹tSi'%Mnr#LU[yxXLaPzu|twwn9&G:_Fye[mpU,F LA3X<{ iZf~ۑZ~e%p^k0 AL@{P$ɍ5_rVl yItҀ*LL>^=ӽ=\Pb}9hSFX7@ AӺQF]H4m) \XPx*(Zſi5gץIrq V2;ED`S ״ܕt9 ~|K|"8,Ah9b7tt28 7Wa!gq{ /H L<'9Kj Wi1y251J#*->[dţ !ϫ/v@b|_+63.>ؐ^ 0]\@qaC֯Q2d+^}kߢi i]J&0`dTxq OӉJx#6 fr6/#>ڋ#7a%4@'y"rВ~É.2]Bbo~x]Ҟ[+3!S|i4)A)V = ػeo7$"0rd]Vs0h`ii3Wyyy?SCP++?[oݻ.hFČlj(\ËXl00R5ǣ_Vti/cɳ-R./%z'sx[W,lv~aJZuTv,a~I\~KMZH\*gܒK7f?yG.Qdy| v߭k3bF O)j('jqTmmrNTh儥ϟ|T|edKOpˠkF OQwWNhXߥzynxxtYG;kozNvNOb=KÝb˘+; OUK ʅYvFtrvԸ_D]<4wi~WF+h|jcaKru5E>օGhvz2g;QZ%Uԯ Z9tG桌 s-%t++cټsG<\0vMۆvw&SՎe/y8*3cX%N ^6rOfe.o2&l~xo?l_rayZq*tX޻Yم_]3<騄֙-<ЬR6o{泶})o7|+&hT'*ie8X};pkz$h>áuB]8謮WR J!uڗ~-dqg'T#-$%)E^(,/<3iݎ+oiž=NPWI#f.G0JYgI.F[b{PPBXV4k]^.=sZXsݓݿ^.vKkY+XGδ{>$IEө4~HX[0!%R%暁U1jJw\z:}u@urM}V]йu2=n.iH|RuI+P^4{GSUN:v^?˭G^ݾ5cB* >Lt|(Z1.JnxyF5 풣#`UCV}r}:hPu0͗S6^K_+}u@)2R/F.g*eH8 4-8 `4ûO &,PGM(BIJa}6:+/445tϙs@L"$wGs RQ05.+˗s]y[0@` OkL {gC!>$dЁB;~NK =Ê/x 苍r(yәPC gQpc1XR@xbH`=3xדDc|Pibia:n#opb\TVCo+Xo-L3LϮ6yLSNVM4;B4r^w e'vec f(2#  XjrޮaKiVpx H@7mFDgk΀.9->^K{'bɗ@4myԉ c2I]_%#;V,E<@h%` `0vXm -,ޅ*La\G`}SPMS ~3-F'N؊QDWW!"t(:/$Q:@ؓvX,1@070 pJ@L <2,8%+~[|Kh4iƷY 0)@<26McK`pW c2r"!GلMÌ9ټy>hk Agaxea4S}?ƅ tx}wVi΅dZ`i-a(!?ֳ;FwDnGxH_؊g&Ҷ mm7|GLt?3N 4~z7`ʔ) =zgC=5cȪNk>5/¬>匤uEIWK|l#˯ :՜)*"ʄV9/"}s)'F-_smw5?x{нr(]*͹7 +k鹗&' kh_׮$V^c RLN _ ⡏*{,hWug+ogŨ {?3^4O)M?*KcQrY[+ wS$p,Sˤ5_!7UC6e\G͞=xI_{/1Of% ^%&'oꁐfA7.;3MɌNﯦyt.N{>yW S޾!kxگLXAK`TkZ<sv=;umPۮꂥgS-g'L[mWD}GYeaԨNZv =3zZVR2FoJ3F,j#ŇG2{yr]z QUOLxI,Mz6Zw,䨤oG|Lҏa9nB|Նvq`g΋=+8$fZ+UZQdN꥔KwDmUQף5[^QqLWF -[+KȎ Y6No{[ ǮJk:R_;tI@F4q'AţJ}CDD_TD$"QC!1DuaF$011izm54n k<HDT $G$JlD5F  !vBHi[$j竦Z'2%rbTW}XL*kũC(iA?j7R 14%f!NDBO©?*o0fa> L$P:y5V'G3@ ïp@֤H!NGiҘCN."ئ0E|bRKvvMfw.$Ǔpt8x#nJ^Rqo1_8#'ܪghj8rԻKĠVX0*R8g3baW(M{Xq FR6h+ ML!T/N_fh( 3jNAW]7]GAc*Y $-)TF`Gդ#1ЂV2<e:3QAX}EE-I"fvLrA3Bsb)#`~L5X4(c/4 ^"%5K\߱?]ߘݠ˷C8e?ٛ^NΌ\5Sn[-)p[3m-y{>~?D4!n+XKlf81;(RhZNmDsHS{(b qwmA9<w30KfI*85x4O#F 3wO!v!0؄mvz[61O<R#0E#cD2; <ԧ[{~KPc$ea`W6Lh/LЍNc@UsZ~΋W`Fʪ@"9)VsaTv8R@bwk֪Fxqly!O|MyӺs'EEϼڴiqff_ǧܺu?3NxmÃ3fY1!Cf8t?{SS3f4ZZ=4#W[Uyхn-%+"W,hce5iMOݪnu_~7T%ASl߼FJS,^Tk y.VO ۗUE l_ (Ni?L>]ۻ+lcgaq׺/ӑس$7:˕ ׷e։b4a4ծ~Πd 47٨7,٦0Z!d'g̍Ze.n|Y`%ձO"ky-D6enoyԕi|v-feZ{P])KƪU%nkHl9iS[Y:d P>7۵,y XiAu*VE6T]-<0e5bH@k"m lf. !5+D3Tvm+hIa^/c\gy::o#$Qg|`x!:]O0<82 R灚M@C\`tMBϧ|zGAa|:tc@~(aLd̜8B @n3=*c,x׋>9vH /C+N bB||ȘL?שU"[~!e\AD=ahVE Ώ=`- 1mH59st)m^q'+ Bk:0tm o6 4fS;1.Bmj6<,ʃ:QcNN36,f\}#`M*}guo@S2qy(n)('`ܑWq"72a)'  #ÚbPdB[ 6+'d5lod_ xC(gO@iO `_"F;0= d>GCumA6k*{%9h;)az=.A=l\{=4k~܇U ;z1G~`"ۙ?m%gth\.7?cQ<>YLt <J 5Z(V XC٣ ɸHuXz2A݊?7!IȠG53@ˁ@9 ߍ5@i#a8 93y}{wO]OzcWqm U()|OQ80#:!O+l &vxLL0Cӧ<,Kyt?{1?Fn:77䅘r OOϩS57FhϞ=W˗/lުeF,&Zu[3]RI?˜4?yc&"->D('k7ޢ' !)2K|Egk>t9!zрJ}ZZ^3?t@}-鉮Gu._wO`UcWEO/Ff67nfR}2W]lQ+yhڊ| -T+M{oʳ(OG(Xgk<6|^oWKB*FT|Z'_6y@Q~ǥ> bk<4eEY㋤YM Y 0鶗nkSs5*d: ѣ_Vs@+'WUK\ԹV_qR`SvqoE,3ɘbn~mY}7;`,%Aq.;wK_faoC6IUrR+;qe\9y iH>CϚfԴ*h+uxj0IݼӃ5|(^U{ mg@#Kk.'sAF<4prx|Mɤ|}7YX3Sot:g }oYYx]'6}7d߲Gn >fs|%MpLem{ HȩL)[Skbb65ejd7*9Kx$(KjqªKWuHJV~:ŨcO;?V]ϦoRĵ su| [i뇽4sGw_9LVvë@]uGMf*;Ht> 4!ѧRDZhz$SHo'Q\.噖N&=EY-kyYmOrBݡ]Kʿ~6J0oQC$ H1vM6%۔H UW`!73ʢKr2ƍo75I\ȣPh<|B `)TQD"4bTJU7k@`ܰd7%AN,h@DLKgX: aWI~њ!)G.'W#'W+QQZAJ,Rѯ2QcYCFP'/d4vhIlEo#3*ejу)H,lC($+l#qTXS:3zi e^/ĸAgpQckkc8 ܢԌf GiƘL 0893O\#݈/`MHf9\ ss%μN{xqe;-"ii z*؅f Zٓ]^p hzim9lrF@z BB(`ZЉ=9 L: r$S[ ǀ ; x8̻+*p3Ag &Pk;8d@Zsy$U b@7v&D3%t FCY O 8+x?h55CN# 5=#ÜtF>'T4+5+h-c-숕)X3)vE]DmIKɥۏZ뗨m'uV.0*SiĢ^t~ԬDշP6p鐱# s=,l2Z[ZPֽUǸVѩSѮ,i$ԲSsˇ hҸ.fnL fv;EˬG϶=0PieuU͵׶1RIU3@eb XϖNZ P4i*>eaWeӎڟ´?v!&?BuίNi#0dqVҝ,Ct̹fb|~^""8KsIė^)}R!jqDJ5[;{12K4:%zwOj?D8Gk?//\;ǿWf_`0m$. ͧ~A5>yxwlhKYg^bb<YeN +>D~};lxi#2oeq9UPZ *-y8Qn ʚ֭ ÁnqقA 6e23A-Հby2t9luW0g LQY;dUyn^ $bZ/"S}omi ?c~Mq"ѡV3W>u L7NYw- S2l)sȍgZEE{3((3okKw}|MK$  ~&BK *x|R<ú7ߏ;:.3ӡIK?mn]٤j}­J$/t&LyBAm-~l'DkMz|gk9ucB½I3T{oƒ[ZC^UG,B[ċ?)O[oT^, O+]O :nUK(Ds[m2:h8oߋöqMU-u:trk^i髆I9/*}o0ƍ{M3_kϻ;^-m(fs郮ޞGFhZ?iy،TF)zZo /j}H`Wi^ÛOGmZ '_lqt^eO A*\bz :Ɔd昩HخF֧ͪEE]T%fAǶmԓ By70*s|ȮzV&AO OSeub SE3E/vQQ]74SYsmn흓(8ZՌΞ.g'葹xue7.cE|dj3L+Jt q7Yn֐h=OXEu1}\S(r^fe}7hUA ݚy8EȚ$}=]R>-q?"***_Oh,*(i4>4~Mlƈ EZDljo5nj1D"EM_&K:SC\-R BBR*&sE GDL̨ ĨV*jbQ"qDZ8"6B$$'BnDo,TԱ8h߇VH1֚K+r`I'Sd~Htz0NAk2GD̪ixP)BĽqs fe8q?ٟ醫C\ ޸JCCsmi_wK!ܙ|Gv3Gxg>9D {n4S")ZQQ9ޣCHh01Ve"2?Ѓ0}Xg@whO~gP)DdtkȔMWW@x8$h)FD}#S 1?hWi2ˇ; N<blB[џ[>LT \ν\{b$`Cg@pgP%h;gux3vg/O>;X+Ix̩ CsD~`\fLbR?ySu 4*Iꠅf. MNN^-)L+kGK!ܢy_?f3,ߒ1&A#@̮7+A$xϰͅu}۷_zuPPnС۷o۷oOv_8_Bj&kdv/Ge'?R.IW2u/:lCk,R.U?~`ڸQrz2&LBqVlN:$Ncv[g=)O&[N~^8Kbzk͹VVsT,k_y|ܴVV{;3Ț.ښ;D~=fJ. 5")hUр"-hڊ̴;{-Sjz3V]>pА;jL`?d +?>xYp1&a՚m۩1ernGwJ6r4Nu0  ,XrI3h<@u͘Կxj)~{dO9V(:GSLΝ}2i;'^yU2qcom!ao+%lNdaH},Tii#{xɶ#e蓞x1ޝl`[$o] KޣvO.zOjY8Uۅu[s[I=!q6&op~S>l6(Saʴ &+ݏnZZ V7ǡO_J_^纵J};d^)D*LȜ&~|UmE\@ʢY^rqe%ɭjceG钻>eԟ6RVToi>UN/u=.lSV.NeW$E7%,^eu'}奯]Ț5Yw{ִ42JTO9鰶C]~aR_\=qHQfVTlV ]$=;1Dn *zE)i x7xk"o7|Sbk>=Kf5&?ޣ9[[e؃Aw`k  ۽? }"Km Y 3L-[E7oF@|Zqa$;Vx|{@e&0aȥ@;ў O BIЂD݁lp̈́sCN3i^0zW,c< P8Fމ b`/ghKF0 ĥ3R!\DXGLd!pۨˑ1h;0&|-^Д``(5 S(sy(&o]0].]M+ƯjNED_f(=Fu> b\;^ 2[M gdlc[:;@ [[ @a1S~X|>WAK Աfzz?. ̻v8 '=JKNCOk'KKV@trnJ {;V+IiU߹Vo{U+~Nfw{G9 _U~3l}[N&5ɛ2'sähL5Nʹȋ%u-[ txh!rU=`^ݣ9Yoe̓un i 9=>tIKxK-m6Y`ub#Jˠ^>k;^f"/P²=;[/VT]SG׾JV?bbEX/; |,#]^T/XmsMʊ; Ѳ*{ ^螬{ѷGd{8^g*`j}=;Z-7BÌkf ڪ):k_ݮ*~ҎT*" ]~Ɨv-1,> *WV6Tx c#[LLׯʶ󩍂 J;w+*=3jYA>mQKVWl䷉qyu/Mt:RcIt))[ =sOXz{l׈0k6n))pmŐ o+?`s&Ɇ_4/rQ%ZVRzUeRX\RѪB-| "ijrI]FT-\x\gY}wLs=N}^%r𡌷!fz}sgߢd|T`7#cV|KJӷӳM)"6=e^t}=F|Ѽn iygDQTyfw}*3Pi`EQ̌^Ts#wzvJ|:/$MO£<[#{9@M|m;U+Lk3u1\Ul7Fp;oo3M|HT5ӡRmXٛ؈L+zrSxֽ61iE>iX.5[`!O^̮\'uW}C4?UJH;Bi5#P?cgΜڱcGϞ=\\\/^ѣ'4?JEX$+D2`_j\L8M֡@R#- ,#b_CV܄M. @j&mm/87e-$Q#@ Qc0&oCx]S5BPAe (#AqHBAW6Zp&UmoIzɚ?()#D9PU?.T)V\ZFZI1Du,EԸA 6 Gy偨H_sN؁0%ZĘv|DC֡?q^L|"ǃQ~A8E^~_}ǐngf?& -6u$Lb$S=ۣ~y B e Yw1'\%w8vԪᡦ&K~y[tP> ؇9eB>ن&Uq>WљUT'zf+%jMt5O|ޗw˚i1>譶^L^3VGj| MpZ^35~主\Ujwٽ~FgO/4XRߡ,^NzH~}ͲLhL~MNSq]~Xy|[ y>Յ(ٝA-ҕM.iW/{|;ё{zHϼlp {]_ .0)A4Џͦ'uȋw8̀lSm^oR3)jBΝG*U?'f:>=t|}}:$JM .Jni}.=,xe^-{wR*n0al5Υ\V"NA/c{3Can|/KGrvu :&Ԗ a I.y{e*7^`nsV;eZح,Uo"6}&ܸ/׮" lM}ЁV[٠*v@zg#m\Y3^ل9O4ϋH;?-9u/G^Q1ꁁ֨G[vh~dƴǷ_0I/[l\DQGk3{Z>E6qRKc#M|+9h/o[X"u\o cmvj=X]߫®K^?jnțR353Lt-d4ԁǿiORty-oZJp˞2L_=0)Et.q>՘0]w4sWѻ^?ZU~>V/"ޑx\&9yU;c۪y3cEWk[_baTަ;H-1U}n{C KZO8Dr'#R/rw/^KKgmGB׶Zru96(K9ll]PypZLe[ݞ*sy)굋SgvYW.[ 1|KHH ޽iPH6`i-5T"1=z%8BL@!55#a)f qߍ'&Ӎ M13i š̹aOD&1C ǯP+J q$Y$* jb|M6 HI dOjk%K+^]*NjNOrKK(NJX'! H5_zqHq#,)xj~V/C%Nd׬ b,!H_'܏ 0JO[Щ}dpiuu^T*C0NPRFdGtbH5!OO&Wd,8pBhK/!Nl^[xR/>N{th?axqÉj!h!pt!i9nskIxd[XLJ*?jx0ױwIƂϥ >kuN2*~..{}}?z⢄Sp52+IXW#f:-1<LnPCXK~W2qC w쀾ϼ{Lq`qx6=!A%Xf&=%SÁyF ZWox [I  f !n 3!{e&4j6A98 Q9Ä+0SΓhXVb50n> 8`'WPL$iqNX$(/kgzМ4𰾗1!a|8@&a@SKr FZn6}k2Q0ph<Ќ9  Ql`v x Lvb48Kufz ya0<ę )B-9c H{0݊ x[ȁ HLc/&C?&:]&N8qb֓WsʿuWݻw^. 'VJ8^hQ{zb¼W4{JYE]Nz qnۥi+{ .MKyDJҹ)^BJR͗X]MZ =u̧Zy:L[;k:Q2<{!euN,})en\,jEXGaMO:X⩯톉;'`u42ʝl=[hN=i]/nwZGY!-Y!YF%J{O0V'yY+1^ܴ#'5ґRr/$}a?Y!oǴmԴچȼk%Zs F|*&=څ$\,`mZ^fӭ:Hpwrx;fcԻ5.F^*kXEN(},`Θˊ>ʒ5/5ک)}x3 '$y=ӍUgS#WcFgLPFO<p^ Q\3В֡V޼ut"(Okl;]|T9%%.n@mR#t@3EsMLpW--`2QF⩘=]8 ;lY썛ß͕eVKW5i;O!Xn w ԰@Yt8 |jCܬҽ zb@ _KqoKόp:a~QeI^KB͢8:y9rNF} |Q^b'Q{ *ؕ/UyAjR&8&]-BKNE#zŇj^/Jӑ%XcakٕEY]7^1YNBa 9yv_\wqҥѣG M'MPP"r:݊&YDu[ RFh.ߍ&YD[JSčbB>"!Q_@Bhn,%)fN(o&T_#!oH|E{ #xc&@f/}U" nᲔdW&S|O啫ժQV(B)1?BY%ĥY;Hgo_TU#U{3e+X*DpTz7!MC( .ce#K+6)FR94EX$[CS"x.>&o3-&""%N}8e٦qy\7zCl@@[F|Ǩ@^ `5rj"}oӈw݁9b`C: @U Zp`/A@g2̍u.U`ؿ8я&8 !P D+ƌZ? w}J%7(X-R C@z֪[y f;B^2h@\(yc[ʄxpC\ς+su0"[$0ug?8;Y9@ 8scBH`R'01k@ߎr1m hǭaNgЧJP 1 9^P&@$ .5`HNµ0A<=5?faw4;1zOI^#lL4hvG9j(bmk{q3([.>a)k0F*߮Ehǻx*7yBz"Z|ٗZS 3 >8n@sf^@uFr0m^)+*qͬǸ@-hR=Q;W< 2nu;(t}C{`A-Q"<0nVNEF`#vi,4LDg<t5_۽bGjf h i}ȣiC A 0R>NbPhDB]7r\ UՃwץhsPI~q$ۓ;+DZȹx^_W؆~>}៍ܻw g{^~cƍk𘘘yOOϿ:"lNY2'H<'Ņ9{)7L2uحSɟzIVHLѻҷN)N2v«dwvO),8 '^*Wg$?rUZf lZvx'xt序Q n $ۻݍO,d7orACen>]V*JlZ&t(֦=7QzjmBʂ qO=^f7wX%t?aE=^v1 .GM|Zqg OmhlziŸj庵_甡"@#yh ?г.1?h`سn/>t|=cc Z>Wy?]PV*:?DՃj֖ɯ%_̤]վf˒٧'i+[}t1oK ZJe,{Y2xPꖯJb9s N}NpA^>ggiJqSM|o_:ۉ2fg.Vޅzc^$ʗڴ߫bݰd{\{9=쳋Y$h؍c|Ӯx>8֢ -g> NΕ/!'RAQƢ͎%gzPadTH5wkoc) N'pۍO&0ΌjZUy#f[b4wؙ/Ln v廿{q:h\ξS9p4ݿwo_kgǿABPCA-u4!SXVDr4P'@m#跦/mI$CJ dH#^#"b@$oHH @Z()S/PRJiz?JH4BJDҍH [-K1Ө%~kˈ#PDJ,R"Yd! )L `) R*2Ubh)*aTl\b=1 T1T-jZC"}b1*PRCJ׳`IMۿuA,Fe2@}_Cp^p/q^7 P< Lx! YUR :YsK e&A>`Ԗ_^Ã1Yr*bbE|-IÄ{ҞmbGe2xĸSC}e4{HD$"ކ0lz8Mc߾<1f`!,alsS*r6A\A *U `>M;.`wкҌxVD?r0,v%c A}?{* G7ǚ%ꇷΝj1ЎoY4G3&G`jޛƝpaw\& `eeAttZdˠ=𐖃3a:'.fOz7 `J$X1`okuuD;p[rGa3b9/bX:d Axv<˘hj3^6].`4nRz3_vvܠV%d@S A V t> .B=-s}D'Ljecq|ĄՌg4=x<2LVXp'|7 a2JcU.CʼnN&! ?Z![g@ܹs ZY3 ۷o[ZZؒw%K4|lN .++}7gϞ011iv⯬ ĆNׯ&55!̆,66qfwC󭜞h2?ko ٮCI̳7RJ8(|<\)iv|a#EZ?4:S8:7ULF\X=Rú&hL'mꢆ)ymO0v ) DC[گy!c+2KRyMSΝꙇMsr:.uZ>~6$}̌ s;Io.$ OWh{m1;8@Qƅ!GXyuڒF7C ~%dujᐘOte}ҋ>Y'2?Nʽֶ9_#-vf>Hr %}"2;kt1>T/23\UYb?$9{_-|W:exQmKOu3˶öJRw^۴L^{˿V;y ͽ t-waCQ3+gJ764s}-{|pOyWb`e/9H.FCgc-˞$e?^)xtJ-]ʩJ쬄Z'/F k#2FvzH´Uݭ例KSc"3YS B nPSJ<E8_YwM92 "߶PK6[cPFX %7:n$!6e/bA3A-mq3~ K ~;oMD]3L] u/? A`zP'n+-\S@(jzŢ o MY322M7I 5ѽTM/ k)^X-nE)5(f)VmSjښy7{hN~A3$~gȚi&-o +Z _ 6x `;/G#¯=f 8N|) ? sB'n3yn}"Gvr9/z7A=1@O ^H\2;]yx~I=eg)Rzx ,&:2^Q|Mdp0Af ?a 9CGـJ}_T$aJ=c`|W {2x*N~MTUFTϦpLg a:e;b]$d AH+ /m`,p1 78@;~!F$X' ~Eo+Zl=ca% =*~rahf1%M8 "M~5OENsG.Ahv-s4YM xSZf,Ǒ4A%&+08Kק{äȁ6Ҩx2]cEilKZ*4;BՌ0!ްX2=Ex9 0~ikn-!ؤ5s@gQ% 7_0ڇ#zb9]N/GTaFL 'Qq0!ïQsXX rJņƃL~iExN=ϓ:aLw"N;&n1;7deѽ{7n4aDEE4߹hܹǎO@3o߾w͛7K߳թ;x+5YG-_t!]_^Pބ-Z6Rxl,73 Ζ^6UF.oRAJp}~ /n WlS*wte 鈧ڿRj⯑kBcuQrLxiE/Wh_3!Eqmj:_x~GFUtX?;Bݳ5Gz%LNҐIA2QuewLHts%w㯿vӛn֣}eD/?y3C1G!FK4ӀOedq`7=u&H<;+D;\~J=\x*@nޙg<=i#%#+9Q1=|jSXݮu+Q}ZϜp4{d#raU:Iғ-7=ȇ(ܻ[N~+wIncV]dgf!]VSt_&]|^CT\=FU%J`mHUi[v-*&W9ռSޓ{>[+5j,:k0g:L8l{Ϫq{]ֽ#2Ŷ1eh;# ;)TtQ4j7Jޝ_heA~tܪKˈ A3■Fd Yy&c **ybpHeߢJ$Y jԵ[ I4SłLa,KCSq227bH/A3j-9h;95]ڄa,BðnGu Ն0 G C@w6(I9j !s3 [&vAZ%tbNiUat}фu]*4ռצ\ X>3߮r}^˧E?k蕚HS).b/7X;O٥Gwڡ~pVU72ꢟILJ}4/ !$;K:&~*Bf[soSQQ~×U+%HP xvɈ7"~J39<>o=u^WM&GyZҦ=. i12gEVQ1߶l[YqDljvagyĽV! N:mҔ=6O>ʎR|45=*k 6d<&3Z^{TG)z-【4۴Z6<tM4 .SO^zn۽{wpp_p?3Ѽ?7FWB;8Sfw~iC a1X3 W,wHl<§2(U\m#R7ޝQOںqoDu֏C+䝀h f+XQon;{y30!sL|jRe>0=lԩX!}yA`-d`2gx0Yk-e[*JdyD^`Hsмv(m´C8&pepDv<d?Wxt] OI-  w拏,$]sOO5 788_l F]5ݷ@4 ]w(.N @SsӶƵ !3̃+Q PC6>h3K"}`!XUf&gۇ@DLHYaA$c>x6r˷m(KFz|0}Tt۹9'u[`~~L+!>WKp9 <=K) /pz,BEo@@pk{\s`̈/"@\sA= (H F>Y4zL` e\$\|a(H& ,!F_3{7ᄤW6^w5vX$abZq 1X(~CN6q0%w-=`T/;#as~FmvuL>}OCZzc^Eb~ڹ%WӔՎ>b{CWJz7_ I;=h}}`H!㋜;t V$W?;pډ=i'8ź:J>6IvA_Mږ m߮kvnX^gU_- zwNNmMytb(M|dgqr(zXV 똛'W>KuaӋjZ*a='?ZqdY6hܓ~F WGK_]_ EM;mWa@mOodՙ\q:&M!Z!rȲ> J_C} Y1V}ʃz%\ӮW[CT{+LϚz])xپN6[RrSڈ &B">r~]C%6JV]߯4Va_)havK ĒNvZ{sr7\_zҥey1fC>ewi܊яZ|*oզG+ ʾ+HHHZ5"J!·z}:rz図"Z*++|HbLoZZ}#~ha yv:bZbBu] 6-{ >@4ܼroY[7\UoXkY{+uѸׇ> ñ_~bWơjc=a\+N(o})[LA9s6K]DžC >}r_`4;<6Yʰe8"#he$v5Oɛ̄=ڼUNҔ&P&3&0O awɁW@uE8s'?C\;}5ָV#1e$=ˆ; "4xL%b/?FƈZ;fj7'/x])#4?#.J|!fD>!ZjGNՠMw?exF m`-{0.4|=k’0fORAt{}0-%e6ƱD 0p7P{5]Q4Vܺȝb\VC^6l:Q $Q 8$i`y̽FMV()A>}ryo#p $ fxAn| M8h:ʮ : \ LY;r_gB̪XpӴPB>65 Ce\>z@̀#3ob~|1x9)"T'(ʠ'V):3 UW"^[{ `eDfnzO]َyy5`\>{,8],{-dx&92=^]8|#gNGpaaHxx?3`Rݡ S`<?Qj?ѯ3">Z\K#˜MWà01/OWwA?UC E]KZՌͬ/- =,΀[RYy<*tI /E0Fs\tBWRS&Z X1)e"%$%RZ%6O\eʀz2~mb&Vb5 /=^j +Kg7aguW{}q] mama{`v/V,Q۝Tn6<+0tʁW%nuq5}3vbQiT[)ŝ 2]l1+iej:^*.s6@aʙǖY^{}lEݯ:W,'o0r# ,̓M3QOvNih߻0eƓػuA98JjW$Mú~7Z\wYݩjX3XPyqw|Qwt/3~|Q&hDdn`$=)XR1/)fzf>x>ȌQwpkD!?%"MٕkaRS_|o:ϛ{p3  "k\/Ffka%ZX($X7-@i-pH6/ k[zhB "P vq%9XL[BMcЂm'r_`4@t@@RcǏ_Fm̈́4b<#9A`Q[%ӈ{S1̤Ø<\Dzmtdr!1'8Sj8-gF 2_B02T?I Q ҇8i# V޸`(g@{Jمt3`O| rm¡%\]r|s[=c6A{hxfR+q$_-&Jja(?9Aˠ}ofIj )t7|2{3R"7שiW%09NPu 5nQ[U]6c.u5mW5šIFe}Knf3[uAd=!&_9(%}YyYDP:Z_gTqI8t[a= V_Y'Qv|mVœOkL ie'Se|ۣOR86~|Oj¶ jkӑg6;bƀwLnCaRVdD#i_B]AJwԻO;U+d]K*MM [YRzυwI`%ӯ3\L=s [6hFym9UnCsOY~0Jc7Jv L*%>b.a;TZۥݢ[Kq!QycBmGvON3{qEȧx V3MAcjlϤQuz'sxCkoњy{[Kp QZJY+8sc݅OJWڱqEOKۻx[OZxPحw[iHJ<4I3e}ߜZ1.MMYBJS0XfeMɺ}N,j.q|PpQ8eأÚ"x|8Zn*a5/Rd1=P9)M.{2Symvkݖ M=BnĂ ݓ *L}淪OihXzR]鍊l{ݓ.ɉooI?햰K{ʏBUkE8N%~.Nz Uv۷|ڢY;_j64٪iQ5Nnu~=.i +DYcF\Z֣Q nF{MS+v(rwޝ۱0G:_=̪ۈQ۩BiFS@qW6?ƿ7>|Я_渌R׮]paj-LIY,A3-) Z2Gi>wS`Hь 7>IH-A3ws `X毝m^BS3tIT'7f s5=%7f1kEhdzj50BZI(tL}shBocT\)UDpvܜ`Bbj6~q QD#N@PѼlu a&pm||Z| D-MFxvh_5ӚK< V`a?@d`~fe^q0xFa#9пnBV, x|Է V߃}uV*'qn8Ba6bHW8C]8pF7ޣKA.%2}K =kJ) s>OC) r%\ie3{2Cxq2`"CH_ϫ7-I݁QC 69%Ib!  G`>@ x= bxj3`ÔL_Q7C.dzix*L!#1 "Rn9jH1 q %&cq:O,ɰV.`7k8q2B{a둁@#`ꢯ%׮n ]x8!ip t%>*s%f~E nY.c$8clF,{~ЛFl6R' 1Qo ,*ևPpG =g 8M(g7.*/ ftQtְh 1B|'K`|f@3K~]j5p9|e | Ř(4}`lg+:=cTF>S13cPޥ(\"33qv Hk֚{ws @Mw!}H:GkA>oZP5'X;r@n_CCg/Y1  *vH?'<^/r AݗcUL-O O!SFXo+uf_Ku^X}fazzsI2:`yj^}I%9|oN ڟUU4ڳ6\X7]K\{zZZ~X3@j]tpHjbu 2|>;|gל,WQy%?Sd13RRnU 5Wh_y2+Z˭A=\>kBՆ; ՍŬWn@ZBɣm-6a&Qʁ2F_*FZ%{H55?ZV?MY[/3:(Hɾm}sdnGd_[iTklF#,'l~/(ttsOnv$3s-hk-Bta`vIIJogv42G*ʱ/DK.?zwӤtU;B0y쎴:#;MݠXszФ½,=Ҭ(ېqqU%s/kk}8#nѯjroՙ][wO43?fMq% cR/:+Ч:B;ջ6n/.}wK;Gͦ7~Qxmk?&mhxl=Au1?b.\5$? 6C>#>ى 9I+ 9(rwD;{ oZuhgr cǦ4)k%zB`F pG5r2ՙDk;>\  Gqx,•\>N n"~aG3m66^a;t2Gwu)0q}Ӱ8FӀp>dOU+￵帠bcdS&qlxЎ+0 f]Ě_U,w:`p/Ne+wYĄʙ/clq~~`ѩEG/Pj{T[Q>3Xl*aJdgkS7’ ժ2] xܷBn:!6H/cGk^HhFtv50lNӠRI^+ ˘gT۴Z:HO0 =].Mw^w urNjqI<6)oyX?<=Q=hd;rBi]1nynׇS#~*<&,_oZmrjz'ۦܲ.[~t|An`glJmҾ ;m6LOɷYwRy~MR֝2Fu}~DCNfk5x#37]OmQcW~X2c0Ͼt*U: _gfUrH z~ke29r咯ҜVyabYwʚm~'1.qxleqhiNf*:{ VimR.54i3Ͻ.`dgَN:ze>af1E[k$NQ =XCi#-n;qVׂ:|?޿lan;;MSQ]QB=+ݰf}l^;GջVYG\v!̰uOBG.;tƶFKu3;JҿPwL^7yܱY̫Ec۷Y2F/- 8͢dv?3H)?4ш8>ULӽ2%D9WAјGNuՕY][xf~Gvc g:O!o3 c%-.ؕ.yU]^Zs)GeWN}OXDio5sS;%vF?+i64Kq&y%F.Ldu˧YZ *ۣT9K6x{~_PA}^Uށ^7Reڪb6$o6+۶K\YM8.Ǚm=tSǧN*[XU %ġUfr}? WWW77;EdD oEx.-fp@R`fZݜӌB7Z0xq+'4LqsL>XSIB !F 2uuj*ykLĿsDTW+5kM,jL75B q}3FHke3Nh<ռ\7q#%jfbL*UM6nj ŒڿU4zXrHJ^KR^Z= w DF`Fmre͘EBĚyXRL"55nUKMSCwG`쐋Y5W)Xʱxp ;Ƈy'gE1`?ƐD[F ?9qx? Rf0WM3 Lil3)q:E0%g.P8œ tl#c=wq듾bZ~<.=`)E !eRv  ~҉Gn1FK`@Y 0 U^duږ|Hu "> ` :MXI`@," kơd\DZ)ۯEǟ1נr0;Ѽ8zt (W:0$}5@.H\ BV G*V sp"/dM7h|zLB1|^!%T7p-ڋxB/`w&E+v6}x4nt2ƚrk2hvӐHymwD",gWgF~CO+Ź^aCh'^,8W_ql-V^væuϢNJ jy5\RtwSJ(x_uW9]^Ӻj5`6:6գ,u~4` ɺuLim.␌k0[S= z,*zZØmTA]ehi:xOk|sy J3/br#;*4z덝^wuTkY_l-)ԌxwM`NՖ;c)]kױ/~o_oa/U/6+-B~vw"ousPIm&y{T"? T%tki"ޛ7,ro8江ΠԎqFt,|kOH}frLӟ(Gijat%X/2$ZK/PHQYh5RU w{jNkY!I^'$o7]7޷ݭf@ֹOeۯ[zuʆDyykp4|f-ʈh$jOK KPVtzefxHIҚ-x5Z4r 4ؒmH j&AjIP u5C\Cozfj j\4f 4Š.>-iZq] eU4A322M!EwU#WQ7s+sӛ")m%6N,5k^V!4ǵꔷ5VͶ(hlܒIMg䏍[xBL&34e?0x[t%zq86 W4 eڠ×|Qq@p r!a 8J)xv Omxp=4Q><̠y),Zu>8tO bztKx_-G ~ѡ[a 4!8 Fs&.2/V@ i} `1xѣj x%D\`ɘԈw?RR9Gzm(;`" x ΟbuX=U57iN-$Zk aɖdNB$y("d3;N}+ֽ`3tNXdȼf{+/Lu'լ-ѻ IKg#8 ˘cavSp_>Osb'\Ӏ!d|86>JP- %9nx6g._!13!{ 4-QvxܽʂB2r+6dՈpQC<4 4#ݚd̊@\g©f֖j;tܭiCz3!J_W^{S\#cܡ㊣s\&@F75GEv+UG.J;rS$V䲢- (]WǡkԺ~ؑ>!_B]} :JX?4r_.pӬ Ȫ_Bߩ{մ&ߺ&WB440{Afy~A^]\ ' ]J.3 is&x;tUpCBr m+BtBSLƶ8034!7>^|®ok mIǂWCH59-r4Sԟ? 3fwqilc@ܨV l F6aH l@(M݈%HnF'bH5P4+fau: h[EO(HFDߠ&h%hZ$i2-&fj4pcf_lImTUѬ 6旨w03$H,l,c@QZVo%B%+5za)"h1=M-I mҁ 4GM8"kÿ5QJI gtF.(Si4%Rק]?o@U nSBocGXh5(T68ݶ(WsLQt|.Ϧ]+7)` Fr'|P Nd"(u(/A?kK.r=q( 󻴊vp20I6Oin6wgeNJOpk;0y򠊾r1Za ;jo# "4TKB89m >a?neo0?pZ "\O/6;o}dd8j<]LpGl0޼ا6DW嘩m @ͳ}].sWU %hIWk,~TbEހU0玁j[*& 3Iz`[';i`9Ń- u$F\2':axX Bߎa!rȼ \fccnsAe)X|JlL`6RN).TŶL # l=ޏx\z0iX!8;\#n?敧JwCiɒ'`GDXƴ;&cZM6LA`|<ƀ9cz/Fe,@ظc 0?YA Ч_Tu¹X[~l8gOX98wph//J{=2GُEވ#;E;߬$'0^:s¼C&eRt/t,NyK^/Bl Ug\9(}oFEx^E7lg%A4r>v>[z !A=)^?]SVRq2-v?tBO;ޒjr +=n`e*NDiHm>Nrqh?&j΄H2}$g!x/#rՠY.Dd qmRn|E@jJWcoК]ܥýW) ϑ+Nvh˯A]-2<듫QW`: B7Y,Jl{rR#);)kľLW'[Kf~1#Ӿ&tfAuy$*T5},W=ɮ ۨXק]rRb%Qt%co^v1_~f:gC[IG8zspVjokIhIq|[kSE\lV_4y#)|ٰ}US1_=ڣi5\dft^ĢP1U,D~cbyeIj?R8 埢/5A+ # MX#A1a 0O`NOt58fSbf"a1A_'w0= ߙ>8< 8x 53#l/:.%=?_h{G#R9Jji7o ǻQ`p! +;3q@;nUG)Mr8փ1y0PG0:zu@~p`ߠ10\!dt ?dE\a, `sky3q0f,d!♭DG@"x J" ( /LilH- nE*AV ]E).  sǑ0p?$So?9zuB :a'[+-2ýAù\SxpgYpODħ!=kpu+kUDhEcیu[txOt t xrXH#&~>(5gҝSFńb4l9x`JB( c\&nDC'p뾐H{cgbCcu &zw"pa]6a~Q;MyŞθO[eB @[tsj[sUD[ϟƿ !ҰWRgV9apAW_W#}N6/ęez2a衽懏Vy\g]-yri5[ %oU:}^3ֻ3w_'s=>(9Ҥ KeVI!t薗Ql_PJP{IhtT6Ց_3tp4?K1):o^mȺV7fHZJǫnLWבwHCs>̭ ںՇW *ӝ3ל(m[d#M־VԢ+ vIVxc>poK6j6\ں`H[Dvb>Q2OWEtEOVD?h3W™ >ݷ?bpG6QjG[(SC;\|]bS\k%,7){)_nZ_'ɳ~3_=T 3uY]tL(d=LWVGfUҾ^v8bT#'\WLcrXjKMg=Ub%QW&9JrWs*++5,[[2rhOqN NΞu7 'pTفab$P+)V:K}IN="?6dCN*v~i˟R,=ŢǷƭ(m̈kuerbz{mh#ˬYv~ӗbzGʯEZ.ց_#}9Y 5MɈrr"k\` +<NhoFe>VKnGZ!Ȭݬ_4ˊ:ݼշڅ= z:GSDGC9VMIM>|Qχs=޹>PnW^}gg b?k-B3- jYI\N[~xK/5,4Cb-x4 5DR-U_oLhpk@ԼxPR$![Ţ:6f*~# * TpK D"B(,'i6-V [Epbrm]duҒM+j )X܈X$-Y''5DĿ}sqK5@$l2BTR*_;4S!h^(JP Aohn=󇮰pl&n"V B*hT;44ݛMVp\ RPm;y 0^`OϪGʨ'/I |~ÈĆ ;sln@5Er&< 8)YpSHu)X ]V08 g؄ q#ڋy|L骰١ c7c, 0{ I^k,yoinӬ1- 0iIu=aV r wR4|gŐ@[Qj~<7󪞗R%wO/)bЎ?KF;sq>|(O+;߄#\h W;ޒe>)]y hz32VKVĒ2ޏ*6zc^IĥN&ގ{o^Šy^,ׇ=N$W@' SRɮ@5EF F![%ԡ؏ ޢ. y%Z ~Hmqĺτg`ߕn^`:Ԕ6yh64FJufK2=9@*2;;ieuҁ}0-Fh(@\/ /2 0*D40ȀFx.pl+A癋{!ZAҠ%O:[o|e36UCQìgYv,e*| qd)o0 5b3=l+̴.0Qja>+yj1. J@_;#x)F:]z_Y[Jܯ'3 ƙNǍW e+%4J?>jRia/ b'_*'9P1@Cbi{+TM"GN8Kw_s[_m0PoEn[Rاz.pʼ[qYeۚfRICU~Wz wD OX9jw]c/ ^O73gdx&r5MqBJWCUY_RS7'v̳}.'CKGg'ȟSpN9g .*>gj1'2pv(^v2w@/L?(-*>u5͌{UM;bn:!lTF(lz H܈"7 nhshfĂFۄf5v̚ijз暵.Mn+:f|" t=AD[.,@oO? DXA<*"烺#$b@r)3B: 4QMQ>13?ǡ(eю_.C#Z*`g\ES/A>U:H֕,lƠٗ_雼ePxGxɊ6(pK7 ՞؉'A=!ylY4"$ݒ?{»r3`Nc3۱ ћ^ 4 0/s;3G`b`|>Lh}KciqkdR`ōs؏!Lxn<#0gkn n7q E3q?ݘ\Ƅ$Y Ocv:eK8 kt2=akTlrVU {TyPP֌V> 0ʀ&`|8 ~1 , xf.fAɎ>\-z;. /""$kl%&ygiDFV*/1""@ @|`qV{p2m ,oem'nU l +"[ KV]OEe? o˞z~~&2E3}\@,$Y1s,G"}ԏnMCxܻ WB!cmy̬!'%Ԉ10db;6F_JP97CN0SV.x>Ap6J_W:O mOm$3nH~~ӄ9oZt4O dlz,~Mȗ=:;v/MfYi׊b;rpNAvkj'ml7g+)D(X.VRSXEO'^v{8urC;\5RsM'm"Tdo^yX(uUKDou4zAwHJ+>;@cv*+BP*_+; v.)>Oݣ== =^l`WfHl GWk}JCKBnSӷbH |=*_3݅wOOWާfj,x܎l^>Kx6$ Xi\&HSޤs.,x)+SoTjbpᖓ̲zu{o.=(RPqwuesK$uuscn{/Y Lu̒S;4Pa5nulɟ{3&aV3kOML.jK;P7.sFŤcFf<-!jfG B3ynU-q/{yR S6NQJ\eRKDE=ay盩ƭ>t_U)w('/g-Y ,&%՘ا&} E)vҧOxƫ0l70^G0sTLG 8>/c@ld6 X gz쨭WnܻbQ  Q'~h\puB(|ky`2n`>fDenN ߓN\x2Ti7+[/ &AřxX+ܥގZ;K`];vY2]%=)63s?VtRVc)|p0Nb 8ĂCCRgLWgkzL[m1:?"U,L;ҸTD7f-./ &SqQIF'Ǔ+6 'F_b9gr' ?p&T0kT;==z G1 l@NTFIuHT! ԕ19~UR0ךzNsx`1މLx_C\*y!"5kr~hٲK"x [Qt/vH7_"gL_WwP1Ү0ez?^O٩_9&h1Z!]9d,*O*ϴ :h6e Wt7d{_` 1/.֙pUW7z3O]8an2j?M{'E?ӢF~ 6b%n Jj?÷fֶdfcW ,V%U|q=ccukminCTTo3f\ݕ}50*035gכ=mJ#[{08Y`A[+˷ XsDه_zsڌwRݸj i}n)y\Xx}ڹ%q>D<-k];vշQu#凬tEھiLSՕe9?ݣY@S]>ښ;>Y!ZfOg|kQ3`WJoKZخ1Ϣ,'d| T|545N|$;e]3SQU C=P~Ӗ^\vЅ듂uw7ۣwtԯ"WǡPB)~ $8~$|0_c%y=4h L&li(Aj?NςL ՃF\S'R\.G H5ʓE#KZ? @$·oQ)DET O(*EJq=H+5JP[) ǪA!,j, V7#?/Nz`@RT'?<kx]'UN]r1&c8]xdtUv{3nƈ'=f!f4 WC8;hN^m6=@?%euO`3-@G\Cxn]}װ!}u{xin r}Oq/D:C";ڲR{E~=Q't) ]CLB_>riq2оkvT* wj쾚^hx; TZ^bxh#Ā(3 (I8@~ժ~jO7@@cώ2 ,a6`sxS3{ݯBQ]/ )?1Q$ cyT|iut$ y.Đn}  Lyh1 GNxѳ}YSRf/7 Mie ĺYB_0:^3n9 z=ib&Bqr>2D\ m1^F׺oPLZ~ORdq= Z^٣q .l?Ho {$hߞp$[6PL ">EOOƴOlmp@L/23dYԹ ZyOw-v-q ̊A-cH}j9m<ON} cp-Λc7ާ֌v]@??믜M%3y0'6J@jA5z?2ݪb.{b  (K/SW vy`sXK]M7AFgX3}MD uy+Ehިe휫> LskXÅ9+bw0GZCYOtI|Uxww,R!fl V}%8{ ]/Z9 Vrg^ukDW}YwmiiEJ˓ 7Zďk-䓾$m"Xn:-3c(YOŰVK7^ U>B En<ħ4 OO`z<\'IrfW(9gѳ. Lm"Fv3TxVQTk-7R:SSjԒ%Xw}xVnz)xHŝ,Z_2snZ^<뼉C}ۙkGE9d:q8{"sqR6Z®[GD ~Bӏ){4C0iYny AUm'T'?Az)I~:=᢬TH#C"9X~DTQ5DN6qyt_\Vx:,՚V߯H?(3ʚ!aR5 ]٣:(rȶbPSɣ,^:&hL~eRe>MZ) ЫݠUz0u><Ȍ\ؐ e nj=Gqz#Gva'8}BxNsN0g%љ;Massn}0';pOI U=@ n} =DmzA>\IS6mFng^>q&~ecLoY#杧hkJ}@0g2?0¤˿RBgz 0 @dH< N,CfSEwq]dBk&V. uV=N8D4i?trKKahXL«Y{P1":=~b1-p: U x"a"˱O?[\mKҠjjVɨb01aȡA+KW+bLm, eX֡<$p57C:UkX.KGJ^a1žG@Pħ. :.0B2H $@`e௒T^ŧ )X]{|Κ#^>7;WQMXeZ[1Y): [=IhnY.I'oFNJrv}N>}c_ſjc_lZ:lgo]>)OCwG=%=dz:*z0}u˚:énUsʽb&Cviy/4:\dPnTR[yEfG Lַs>ZberPFU+G7V+R%tyW=ꬷed1"U~{w]ٺosF,x.Sj]kq;WBXvG4 Enjg?SUoR;}j݉ު*":e( =ʧϓ>Zml~evgcl;{ m뉋6?T:[]V+mL\IVU]lwLXT:[7O _o󚲞j_t<߯W uTfj:]ڥM'4kiM#`;C]kr, 6u~pU>I;N#\L<+.ߕMSQ%T'Tz|f.. R큜agot)53\E^vƖW^J3qN Jo-h =nҷء@cs\=m?J\&kgL㹿#M*.a1REL=G3z+w2ц8af1%|4`uQ7$BaR=9 D?~č;$UQ_h(B Err$"\:\HRA PӏeD,*XAj-Uu@YT40(Ʌ rRƔ_X@φǪ1Uz!d4HH!hp5'fIRj19E#YFnIMfA\MX@#I/`Li jTjO9}IXS' nuD8Ϊp"v;V}d6<<%P)}Zm Z&? WZ`xO\~BĦqjDvs|J&{7Gʃ5tя)s/}uoE %0<O%dugYl!lHBjX..u"Ư9Vi΢itqΎ4_w1<,yI\#0 ٍ :8>p, › s]E+-wæ8wx@[C.s)h/gnE1G>?mO<0_զ62#,^A+j6F>#rmfJXz0#;|rekL8}O6߲h:BH8oEh^W 2?@hԑA 0y {Ί^r>,zn}֋}\]ZݖS2^w{"<GKH^q"@@ >kBG+`7Mݿev`1z#aeXĝwP OkQLVgy1/y 񽡈u DR u@^l^#qY}Ae>,6;9hdiM 'LԄ|j*f\oEkFƊwƱLob*x89~xDaoE-e70P7$ D@D vcMMG淄5+W]'Tuwr"wy+:COxs##/R?jg]qMV8}rmTpGY[=᪒jG]3շI+zcq/Ě˅t$V;V|і= /IR4뇈f9^s}sv]hlJkѨy5ovNN7IVں3s:8+`O]]$ F$:ז޿P}Hnugn/@[mkA5;ڍ>5r('V;z;ozc)/l<±:7jM9.*ۙ|˒>|ya}'746 *ۘM(unqF3/V?)-u.Jt"]'I]'4s~(>?ﲜu $=`\jJ3yvkn޵Vbreefk2+H=U O2W-&rYq`//Lre1ɰٖü=ZvVECYlZBpIՂ{NQs:q.|7hL8 Obژ~iaaҀ kxΦE97'ʍ_d p˹e-i|taWhuSR ii+Wꑠ(4Jȿz+y.eXgnxjM4S@Cg`j$hp$ 05Ө@!.Z0qH&jQWɔ&DRU.BR2jZzOŸh>BBA@ޠS$I GCVb.X"Xw_ĤTY 2OH.lDWѐU?҅XFhX[({g~@,(D򆏁@ EFD$15V%TT6p=jdX;4)~:X/?6n?2"OYug@SfL?wdbZL0E%φB441ݏ nx[<|2'Hf: t/GwBRi%#F~RTnabb3:g<|t(?[ܬ[j NOxC iݩLspq*SL~[%1Ji\Dfo})H PxrʲG߅e1hW%i$`p2堳t6i ?a aڋW&lu:@Є9g>>O'kjIX7c2QJT lD7#qe)] ߆C5Svgzhnjv& e{_:d囧z//ߨaܢ7~ΪqUY0sz+Kg.T{RhsZlmz4UJ=&ŕv Sv=8/b_Qso.Kk: նF {o҆ruسf=VaxTZVQb(Ht_U9qsVu&bX{ŦImf&\x`i~\mQ Q1D=~`ԼjȖY*&!}ӌí8|ϻν]uOgׂ˷TS`ѿz+Wu4z6'F޺vƘ^w QBX H3`]?Gr$QDI,SV_eT@*jk$2AR #FDY9HzVF$Yy n{3? D0 #KE jDꪘ54hB("B$rB7VFBŏsB`6EBAR! z\^\5ap5lvzAK(br4f*E{фhKc(hVאx? .| JFO9o|jZlDgڜ 0JΠ}R8z_:>\ +eOهE =hv:#J89ģJIN I})?UAhtL2q-; \eA P'? /@`<">[ž.({F$h7"WG&a~XU X2Þr"Z:><<< 0xB0QXyG'rN$¤aA&yR ^ 7upN@kޖ6 ג4 r3y3{C Zg 1ڂ޻A "sU8ʆ||G ?:m&<׆A!8 0%twnV~ t⯘'iσuu b]=I64!$nRVu^mCz'@PN7 'frE.<AU4}]>%1ljQ Btae-140ɎL-0 G(oe٬7q>)G8>u e&.hQÊ,%c#І~Q9t-f!HV/%|s7Q;}r_ }3߅c1~UJPo \UyQ؃go:wF)<=1~^NArWy`pj%vRe#-aJKUy_}<~fGkLvB4nbXp-ћ}4H{jH'eI1sL2SǙk9 ,Bl3+mbi^Qฮ6J U^EVMQEl݃`}OWYfj*OL ^\|h2wlmxuYrϭj9[fO{5)7)W-_@W*N ?.c1}zdKuƾ!R.r}Y|Q6η, [qD~~Ƶi$ :ٹVͯNݹIfMkGf۴k-{ԍޫ۷_RVj8C_.]&'JR[+=m֧&,wrl[0w@Wu%.fEWjeo>h|{xK9;GhQO;o!FG_3F{C-YK1ejIE5WIܮil<ըg{nٴ۸:!{YIc|7)bqg "r)_WsTZ]wgĹ7֜hLn6Ћ}봬ОM^&+ӷ$2WoWՏYKf[֗l*wjNzC&-6Cۯs:KD0ɍi7a7O7ƵZ179lڟ7=|4!㱬jY~3K_"(|$laI.Qxƚh'Sr\ؠvFYI.ψh(Y$ȔI y!4,d, PЀ<҈O24 YA`F,C1.X,Un: C$ژ5"r\Q)1%~Jop#&A* fk ЀYV'ؙ>~A"KPb(23ӷC$E%<`!ݴǙ&x V{SHh 92Mh $zwB^-&#$Bw Ja#WY:p_3uZǼJUn(y JCL8;׊X,orYVdY>ӏAr{m OouNXNx`d>.:"D.] S 1x -Ȅ|zw=/0[2on坿 rC<}!yhpX bJV'JW:^'{@wjp{:@>ISn-o8ªN`,?:#/{-B 6ҋ|z>Nw8_ob[12zg 3lvأ"ײz9Doj,}|RiӀE˲҃k"iqҫe&q$-6Z|a~~ iWoww2su+__fN[Dk)nuQ[=}3ɡRpzւӹA+k[->2*b4KIẏtW 7zf!kQgqAS ^l2dƵϲ|[e1ainr>nPqL$!]U+ϿyzDg7в; te`ף.rw_F MʢK翍9bw:(>>u$_n"E|=xQ_-#Gbok[OcT\1J hs%6^Zpy6Fg_ 30pewj*2rD]ԩN~Qh'j-}lK<2VL*HZ&Pɰv҇jfgUFJ;ugWVe·w C޽ں"@̍O;}ΩV%8~EFқ.Z454dn/폼qI cQq*?-d厌Ǻp2y[K)BK3?+~Q>u$iWR {ah!ӓ5dvj;|f?hwu*\2yۤ]_IAfb# w[?4)rXE*ܭι6z'"\UKٮìa}T6J[Bv?m PzvK9gի_.kvP(TWԏ?* S3?s\_'`OBn 1wXj&ȏ"5(P Tِ@\,))'ȨD*? 7jFQ?;STVRVjR ↩ZiPV?g72J?H!PDUɅ \Jǫ (+KTU* ~3 QsA' !aE*֣21FY"s"EEÂ&TФ5y5ZFS3? Ѡ ~M-϶bx`;Y|G&nr<|"3ps'T7<g=99rMh9xN$9<*e 0&<ǘm`,Q҅S_`LP$mk?Ġ3awpG8"Cbl㞼>ZCZ.>׼= H{>6c#`RݘH76#;i4rhv`%=iV۫.]Q&c/M4As;70:3=)$#Ե/\$Vpa++ 0q:UʌkoOhLMg:m'biӒdL W`d.! ՠE>q0+AVjx ]4U`hqAg#d$fNےƎ5#_Oƫ_OYqG^I(dGyO1A#$\FÑ[7a?O;T"s%kB^LR&Vd`Ұdq)1 =t9M8ALB†g|T9H`_na?8 C[8FC{O7(͈S he/y šfzO6@}0"xy*ĻjFɔى{i̮ꖅW?ۦI%ϻ~w;G' lYA|˳V=8t֡{OO5V1kZz_+'lYv~KrEY~˽M_$|$GÜ׋`⻃^_lrZ4ip$ܓ:R)-_3ƙg;\գ䞒cJfrFG-ֳK5(^VEߵ06jSA %19]F" gƕ&ߛzOv=;\9>=UaAjR>Վo5tmm&vvԱrP[' MIK^L,gLp/܏ՕH3ZZ[Z.X| MjY'y@S.jj;bw'-ߠㇱwzs'dN9wzhhݮeXξJ\Wڪ}۾a-SQkR|->};h夜 &W_py6O4{-ti~/]~T*|y>NKD.Lzʝ6q | yoԧcls+b:ft!,\/y{y|mmsF/y9o<٣MpFśZt%N>m5;)Qߤk;O ~m[ovᥑsznq{j~=MJ֖*凬(|+;恳$'JC~/(^gSۡ B75$y .M3dɣCou|3(㇗M:t"4_­,]͋wkѢ}^kk]NߛvXB8IJhk'1-/9#usLo?gr}]6ta)Mi_D+Ksmho PbCKJ !'Ն"2CJC*@bH!lOPԕ$$ߪLwתhCDRTrX,5%Jߡ\.ԃDEYZB0TD?B@U P)JXX(r4bwݨ1GheɅ kJ?ZT%QRAJ $rq>$FΔ?YzY}$ҿꡑh +?@إd?xY3 HwA̛P 24j xܢkG=u9g|1ӐDvPr] 6|Y80EyЉ!&$͝(^pہhqܚ]3?R~J\yECn7 3SL _-vdMyD]jiBW20uܖщ5ܙ)DfDS1g3uI7iP/ RDla7Mc'6OɓY=k) s-"B1Nkv;;9d]pQ(I\m%qٌqbl 1j2UqSy <W[|_׿YJ3#<aK6jUTr]aP? stv z2?;3v1: |ft{E$b؜9Б$c <06ay<g={+9*0GЂ`m0 oI8a)L.k\f5}(]o^]]}eX:i]f`ݾd؝Zi^/Ou{owN0ڃ1c.7;cF1W2PA@ 5- nk?Wa}AamaEZ&x=*⚤n S7xW'˛^q_}T8VhNAxeϕ}'\ݭL>Z%ݏ&sBUlS\5-^+"gsftqAR}vÅW[iߒQEҐϿy)X5&٫fmt>714 *r׈+3p`83A>=mzlyMcgAo%i**αE˫f>mΏUl奣w>.ĩ.6k>٫WHU`U7lf M _4X,\?{uT\Q;iDTN;D_?9LOSZ5ۑ¢"g.Q"Ηn&ə٭fJ^y߲t_FדѭJ7j(zԻ}2$V2.wUba871OZ\M{૦z_gʎZ~vܬ;ҥ/f&uo?j[1 怈.0MGdk=Pjy(NWʴﰺ.+Zo B5/f~U+mir.ɫj ZRMk< 3nO:>-Q ZO+t41~I~ˉ:uI}w=j#^}Q9i*{{ #V_$Jx"]ϽV6ܙ񮭑)7c+Eu,y{P}:o+崘hsK eClb,YW+%L _#@rE3G1ZaeG -}=rNfVJU}u?ЃڟػV\vlm/i]Lߥ8;4jÆ=Ef*{=ge䗸ܞΥ:/ZoݮCի_m7BCPs4|ʍf*~ʸBsMӆfMbPAAM3BOu1ʍ!E?1!(1$QFU*?XZ]ALH5?fY!M 1V(Pbja_$>$lI$R((TKя;\Ԁ'ը֌B3=*-PѰJ.z8̐֊?Xbq*olpMW)TRQGƓ_?χy'n.!>J6mo9SiOK%ħ4l.<hr nFc@ݶm…;ˆŬk8~g4`tC@؊n^' U0xƔ5 T@@4~:aWp# ϙAZfèij7B^M4&)3*f4i9Ρ1N>Cܱ=&;C_2aI?L'\ $lT`:ҥ t/o0fcl-$"1 i @ٌy/q~A \NbV` berR06xVG،96`XVl1J]xҢcVj;˰=>Mdr=}v!Cj: ;Ҽ;d4鏾8 "EJ7lO4,T$cV k5h%s7XaW_>^O~H Y{/"'v*~<)AVV10\zLS5{h"xC䡰eh;{UM[2v&3,9e@`XI[w -+p!~kwh)VQ5sKHknӢxX gFv\VT跮T >-*Vtty6}JlLYuSSͷc Gs^act_޾ocG FZg9$?1EKIYIocQ&)͚5vU}c[\e_Ȭ-=GC⮧xz^LU¢K8N~XH8*siʬWì5G ӱ"Ρ_־ĽdrBbZpsG\ҵ.o׶-R14Sni; ?6ީliov4nHŞ-̝[ 6wOǿ/0{\g6銯]XvϾUupW;[5W6RiFZ阔d) k}4^1=s XetRʭOʢ*(Ϟ&m,Ui~i{ C ϚշEBC?ty|OMw~(|=0rI &[nakV>Ɗ0R*;؝C"ݴ}ægV<>{gdNRaww׳@UJOr6N/ev{7_&x/1(|RoJ;Qݗ ??w Z|JejFx@eLsIs~f'LbsĻmrxv;k͉/4vje ig^x9/~vfkfqۗ'me~J$߇ո?U<{S7GjW}|/remޭ):yvG#Tlb鍸otӹz<4mb(fnd41E=vfԛp|iKk9y ^|NTY:ʧ6H&9tެHG6W/Q`X,l {L"FИ;o  :  dP$VD\$` Jb!(dLDB!* Sd a= @ zh..P3+1.'BO@7`I$R=ȤTTz˄{. L j5U?J\[\[wTCEZEq}TVrJzVA!K(GZD`P GG4ʉaD?/l9M̀5ђ471^REjV᷺RYDLDhge(&k9Z;( vE0Nq3RBLh/Шk83#?c&`E{ @OQkLe7  3/@Tʗ@Sx==[5A !jAĽ îq 4bF <:M4оW27.h LRA |$d`, ~G?/]4T I}XiΠF1u)Фl2B- #6yQ$Ki |`%!Kxx@%;:mc ~a/_m`>~muw ~Fy{K@2P& %~7O-{ޥ8NZK*shOݳ~.(lC=؅û.E_,osIj^hL! CH;V{tx/뻞b$ ^cC[@.ɨ:jۤ +2؍WA@(@Ley F[r f=a}i,,M([Z [>gOI`Fi`;K;ڄgS+?塡:uRWW'7?+Y?~, cƘ+G:im\3->Nr#JjQ[td&̴ҽrc)C{#^^\0'zDܫ AaSQHfl=|?K=NZ} }P[TfJk#^w^Z{ruMLA`3p,7g@;9+cQ=aӘz7CozlTFѢ2RDc/;S'Hl`~kC^.VNoI֜ԥu6Tm3La+/ɲ\ũ4tF}psmR|j ^>m|UB0yQjՒ62(n[Vٳv]^nyaJw_6lK4EX]h{B,שЬuM }o]P鞥&~2u Ѿ[j']{>p-/V1P5PΛG*z%IlqO8PKԮPw*roeFVԥ 6e+=n}yi$ '?jX^Oo:~͂N\,vZ[]鴖ͥ+^^ZUXq5"1PI~g&vo*,}ɨ_ޚQ$>}lIԔDǔOvsyrR }>>\KաfkU 0P[dj6g-c}z Tϐov+n'98r?+dskJ8^2/26@;AUcv5tI[Ǻu2M{:_ߦIL {n6ҪuanWeK5 <:髬nū\JeTW}ƾ#kMUϯ\^ɾlM5?¬]}LͲUMseMmnķY8: ֭B`{VKGώ̑Hi8v8Ⱦ \MZ#MP}AqgO6s3m $έ)~m*W]J:_7d UGDRԹk`oܘ!IȂ Ԁ4,QLA/ IKQ#;,(D?[#><2LQ Q^RV V =*IP{ꏳ~$Ip_"(]_VF$5f9q"?_Rc덐_N5N a O T)q~ʌGcr ~j581Qc:2(6?w̿fK$[3r> ;L'1lNyaὠ0y^U0a&W0uo쑊kD#@8mW]O`Jΰ1~0Dg4%{-ϤWp<:iJ"&w<ob,"厢L8L%i #.˾a2zPƒmጝ v3xyyM7 pjjuι|8ӧC$2*fs{XES6)(A Fw/LL: a^]F{"ov' A6vBGMog%ŭp2/x 5: 1WNÀbPBУUzM:H ׊@ɌAݞssGʣ쇹[%8 b; DDDol~Ʊd{̇Ȍ0`/x,u>nÛLgp੢>{r;qw7CZAAK]V=@ۭ @ݙV⃈N{60U9F#= #~_ g;_||5\#M gn;`k!Bj?y}DX fVD"P׾5,4~dg$8z&մAt^WwRX*c !88`u|aQt~`uZ ,2صLx>-!dnAK0 ,> t{D'NObY!^rA}'7 2N3G@znKɼuK,qww744,((+/[ihcN-Visv,:MZ8tS'?\ca$kUkwhSBUƼ 9'Nd;Oɸm*)ZqS(zN`zEZvzxoF;םYQ'.FcK7A1wwyuY7]nJ~*09Xc mp|_3M7J&/-A염,*!vRN+W,m/Xm/XSI'_iܐn#-̙iNm13]{c}G5*>UYjSѕ.s2$UNv&Rk % ,,MIY4>߰EħizKe!_ФU_]Jmf|yb[U5l:3 { 5:Vx#,>f_[iXUT޲hTaKIv/JM5*[1wT>eƶ&.:ѽū( }o]*&sk7wgo?9\2t[msV1@Ljm2ڗL/n^m^c&1opy x5 _oGU㗟 [/OF:ʇ[DύK6͋ŻU'{ m).na=<~HX)A||4^xdLtgh:+Έ f|tNWZE,Q`nS똜Ls,zw+&ؙp|(Q ĽWm\AnɌ6ī*~/4%ƀܝ:{sU=GkgI!З% ϶FlCʸ*Y괮ǀֆ"8 xHZ+ K5ŤLs 1}9;N&2@YV/^8\8>6@}%c[ iz>x՝+GTP笥$T;X9sE涇ՒӘV޵y.='Z%{;hF;)HE*K>+ow ,M{0u=ĉK~8 sV~d85\vmjwt馫:ۿV?qb1" MEQ#>Y}aBIl~E4q¯%nۯN S&JG,,l/| 5cU{zx(tA+bnAs͵׶mAcKםW6Is=z_sҫvyqw/Վvtе0oj^l':i5 35^|h77JW)j<m.rJGqeH5CtJA3<c\ͫŸ:i'v~QgrUtLwAyUF^vQ˗1(k 5W/g_sh᧮Owv_^`_2Jt44R)zK-~|;!e)-߫DMdϵ;t=i~zןx$HIõ%4Gr"fB:1GZgcNZ;mjU5&xha?.;F)U (y:^@ T'너2Sd2B^%>Q}5@C@ ǚÀ䇽\*""T@P4Ʉ:՘ݲP!jnCP#?oNEI\ߓH$mqjEm}֌R,!@ހQETۀYQ/@-㧱f9@E*RohHEL"i!B(߄\Fh5SՐҀ #:,3LhD2I,4(mH$k0uҿ c#ÂFnV녲G~Pe +X3;qLovz&سL98@E{WgP>A_w Zi7_ ypL9s9aMm j9@S! z<sLδ}=t.ĭ!rAWtqf #q=B x3Ɓ1(e"Nbr"w"~<>Nxe@->>Wy> >Qhʺ8tS؊}ND AOR0m3vN`@7Y;K1q{u>W@x]r/$x: 6`}F-f3h?T`4Ȧ|B?,.`q< 0izȸTa0/WN1!#okb*a AsD{R~@{$H{#0=l E 42-^@X,a,?QCNtT+sm}Re̡V?>I;+xQ*l;T(W.0TfvV8dJ[܉3(?EΥdOHPؗiPL2'U}EƧYe Ͽ2UM4E7|ȨTEݭq>׆KUVԭIRz&c¹BK3U?sk}ϗ?hd]Y;2.).)x1^>.^b1epzt ΌnUnؿD؂^3R{tz.1XK#Zcz-]&ڤM#~O*hk&zssǕv3S>vfvrV,yy[>wH! FdB'Y\PrAI$SQԃTJ X̬ J |2>4Д.A,C(hȬ!7|6_?^&CH'"f ?ߜ,jjANoLkFS$bXw,#͚GD %chO89q`n,&FĬZ*f"~( O`8~je.DJ<'ޣ S,&qWx40|0o hǡDY 'Yjd}Ѭ8PSff κt`+N-D a!J#`qkF59)8 ,n_dxX3K}FAS|A{ZDR"5{6nޏB\"#!B$FoشI/\~t0ls&!݃(Yk9ɻ DP >;z*] Q_=p~EЃ-⧕Vp6߇>A=1s۠( !0'GhNIAINIDGMa[-fxici/d~Vz䇸( *\=nKW?m6OIw /o5gṐ0$Y__S;uG-1W^q)ľGG$mZ); BzwisjR~Y3Xvu:pxk𮵊0\8e)UmmJrf }H[ o$O9݅\22jFn+M좬XgT+?]QX,7Ļf(&ˋԒ(dN̔ ME: CwsX_J5YcgW\~uS"#Ȁ6)?6شa!+;꫔K:ʘYBSq_Jc?5(+BMeYCFChH' f7zR 6Bgf(ŏBë 6)$oТX,5dmSF@,lsHRϦV WSWRAeLZ*_Z$*QP<(qUu8~k-;0 nn}P$T@`w;s?c1cu_G(:(P*(5V  Ie$TW(Hv<]u!HkH&"/b eb}FX{r"a7F9e9טH ΐ)0zX$WIAER}#n-߁OC3 Z}&lZ]y1`^GM 30#ߑ+桏1* {ga~3هH:jv%nCA0`/ӴqL~a"8˰sykx"PE}&#*b{x4 {t nh9(4gwVmٮ͉w-4ΚC(Q x~o Bg9e[-K*GdyW;bAG hm]D?V:KXmǿimuܙ@ n;), b *aEC5bH")X9amZrMup2% iݟ+,'  ]/Q1 zA s t6Oa%|SdE?xjŜӝ}6aS6Bqƌ70^CD`|%u>f~a c '[2ghhLMOl{gR?Aҏܕ\G'{p7m J V B9b⧿v{qE"J 3⑚G: `WGQ{sdB? =eimٝkTc˕5*X<خ[JgCROu;5)OfUOFw$ekܮ95Ǚ y(Y31 ݥ5uOi̔fS\ ͊PQol3IS}f+*r^lx6s{U5[70)(_tjk}Tsk⬶n5p%ޠUv)f6YbCbڙnu)U{pDpڽ}(l,h&t|cӥ7,<)<\~̾4D /co7MqII&w$IƭuBgVP4-> ٜX1Ƕnb*KY\#F#-%Q:'.%pM7 ]=&|0M Sϼv)y$=Gxo Nq̹}#i\+_.uk{"Ys_y/ ^wH;5}nz$ml17G*mA5Yd6S8o’ofxU㜠J7u4zr5M@EbeX[/>i/vp]| )nґg^aQ*{כ;ҥjrss}Z(K2:T}_+s]jy:m-n5[!6xe_i>* {q#^v-cʼSЌCUT<_4`Iy5?iVNt̶)z*|iԗ%_Jgn$IĊRPU\צ!2ܘT"5↻aDX!bfFP?_ujXG(ɧp;UJE %s,rW*V%̵-^NCBGQ֨5! `F+ڨI$ шjQ TP|ZA*TlfSa@&oI.LbDڰ*NB Y^j1j03J5ңQ F#ymPr{L ׍FYHV7YB ('UymX,d{U$@科3Oſ4C@+<`r1a$e@D ́ =#YmMuylq9q 80r&[hfUp ^x Šr"`71ؗ^0R5W8s&"Lb&`1ɦqlΣ"6?]M~hvdnL(M|ih;, 孈ccIr[ |~~hDZG{xu-&ZZwa aoLFP@?]Ζ~T!waEbt~>?f^Nh#x=Wvr#&Ǽ^;[W6!l=uiI―桠"Xf t ]B_9{H5< B6`^GZc{'hÔO^`Lq!T] ʠ>#]yOuډq|im/L;)t>/t@#<@}߂YJEHd<0u.FOv11uCII쟯4RSS ̷ʶoex9C/wV}LT2hC۾fe7oط_e%gnmUҟa 缲_/W-;9OQ Otrv+edtk%-3NV~qٔ_y˓*nKjYFw^602o|[ۢmϲavJM;=oiFuvXDa@сȰw'ғׇoػTtS N>ATlǂ:.VXuvݼ;*WԔ7.[{YKh|mcmtPXnUa{ yb8>4kwgn6@Y^ӰPx͖ .%S} h45؅_J93etzHOFfcmbT,QUSvm/T%B- Ԛ68Q(' OH1TRD a*iCy 0#"h(W4fTEMz ,{P"H倶JB!eׁf%4C rXӪZ,*ŪqPV}(HEJ [ DdR@~Կp,kx"P 6ikW(H*#5jMlL Pݾ6Ry2S3zny`8#@Hgha60kO:ӻIsDUv,}L;xCZ@3xẎ9c\*u;n۾v+68[}Xg0Wvǰ_':jV :?N/30\Ȥ$I/>1$%~G 牆mЃ4ʹL3c=ssLBb0nΡ;fp }Z=ZbUҳܜ~p5worIElD̬}T(U =Q4l @zCĘ |?)fÙހ6|\m hru)jZ^Vq`0ޓ#r!cv7d>=jFB{<ͽڷW쁭{ꆖ3!9|ʻ昻^*3nt vOF]s_d>W/U5]Ʈ`j:b!#'<==GPE7Uj}g} 0 Zٷ@SMfu T}_3XG:&+U[_2Ay԰#b7AyJ^˞er4bm&r75Kh{\xȈfYU-{GWXtiŸ3]mEm}_Maa7M*{;e~_qQJ}>>Tqwږi֢O^O;5vU>g ^yF%V.$IH\7HƊG}\3^2ƿ{oOH6zYɷn?,Fu3U+/kIڸ=T)1(l97(\!{{&GS.W7u ] 愸A#z}EF_7<"ZyZ߃ :!Y;pPz_^|1J7ٿBEX%"f>[B,n{)]"{CLu5C1m!BD"DEERf޼ySZZ_L?Zsa=2.U߷vϤϿ|V'ж*fퟭ¬I&4AWU班E?K?<+`^ T)kZ\mմɨ:o4Kb C=IE;J[Ht>w^J.A~s>F&]5*g/\}F1WHXoʚ*Uxό)tHB/qJq5 c,5϶3Vm 3k~L͠K=z tN玺;9cqKb8W-n7zh$c~|2::5O/;sH3+ēsm{V?.UxUqt M8+6&Lؑ?B[XX֥]Ŭrt1{ZH/u5:}!#Wԩ4=ls=eskx}ҥHU*K=/rhQY9*ӪZn"yf~$j{QO]G˧m|8] ƣeh]ڟm;c)~=1햎[Twil9 =wTyQuk`>Ӏ!NM[+pl.NJ?.K:lG3w\Vl"i֙Cͮ |~ bulE^EshdQ{pY΅C_7w}tbO3Zt&yB$\[3c.J) ._8ΗĖ-^'8vD5M(:iu^jݥo,R43I?ߦbZv3}uN]z) ;}Tycjr3) 'w%`x]>o5dэކ識UHzõY/~+# h4 ,oԧ46,W"RairYayJ`?ȵ9$HEfuH&QJa h"fpCWriJc 7亪(+T  (Ք(VJUѧTD5*A3u젉d"ƈRUjj V!RVVUnPGMZM!]H,k,\ f#L&'2,Nsë5FTu*4QRcr%71hENcZ3LKДrD{ AZDGoM"W6xt1 m` >8 P)TzطRbb9 +`HuNCo1'd2Ձh9x sdݼN+$nW0{~G_ؽ)]i?Pn`_{Z1z9|#YEoVacIïlg?M`yGP*؆Ovz *.>݁)ܴin3*0x6;03S]e?L᙮-13݂cҰG8WST+ރovtAMjq/1MZlZAkfFRDۖM{4}09'jgdS}$6gzIH4eB[SGEHmA*e|ȥIF>9X&<1y}>Mh25Z_Z˴X=ۍσ#i^`VH}Rk itO]3I륽$7VvyQ5GJv^oqjE-zft|5FzELqN/=](Y*(N4gvŴƶOV6˯&[]K~ޏk$q=f:M8av :,6vLY]i[jpjB;F}`0,Z&YmB,Dky[KRTCY$U]4S#]"`}ې=:9s7s]&>I/K+\u*na:tܢgJݿ4^;_9-:7,!(,(IxܘoĹx=)uM &M^}Y)V:nV|JNɳ)Ӫ}֝bT)♠ChW[s$=s[y&_l*)Q?|7MY`{ZTjL]~=)ܱ&-7˭n3L5>܍=vex9#,}_x4yxK =o;/-R3z-FK/6,NfH̄) $GN?j*G &*~۫F ӵ5/J-_|+onA5ě kt:ni;'O4G10lrơ=BmfUUU.G=f=x;dmut\ǎUT~+ )23.Ӧ~F^L OӡS]Qt3 Rrݭ+ƹ]`?֥Uu6m^ԽTzQv6M KuO%48Ӽ!]ST%.<H#!I7T]hXiTrHlFkbxG'/vTR3 vZ?&3cEB«p\;ka9S\_IwZv㔃*?lzM]i[ի_N'w`T׊XЈ()J*-A>Tklj 5*u)R*i`DTP-'k"h,JCJMn4HSۉ2܉%5J?Y'5 5 I(L$fUՈj*0Q=b<5{58j.KS T 56XJu\J %?DEL()7"u[\ZQ'PR0KMFM!4S&V4DZ/uK(#\BT$4$BII% $rv㍆T4#>"!j8A6opcJ2ad@-VAN 437]` r"ʀ?5da3V308 "} m:~Q^#طSvH|xb*n1FWLN1s|&6O48&)͵K2^1 _#Ի?`-}k~ٔv{ .zQ-.s;~20Olc A@Ouxw gu5ט ڈ6(v=mBE=1%O>]r3Iq?c<2a1۳t eYu,o MLzΩ ݥҼ93mQke"'e"F8m9 ˕0k,82Lpk`ڎ&777v94{ 1,reHb|ޕ(ip)$dخ$R?61u;ٟ4ӥKxu 2 nsR`Uk7y_2ĄJ:፫{c8Y 書xZ°:ZBP5Q,##+7-tnGL Xw2V Z^9a:V2grva0f5?ϑ;h!5)l4#~ vn[!` b\{Af‘տ?)e}ŧX(W,L$S"+fcv@uڿY= Ԋq-@^⛁p1Z я,oǘoC~4~"%u;| < 2a7聛"|+q#k?Bcɒ%>,**Î;zukJJJzݻDsφEO>V8y r:KOۺ؞uү>mf˼N=Ͳ&/GٱI BS kd+يb4]i6Ne}=Z8)Pn_fm/Y7LoE#rvT,/SO ;f5r}\ ^;͈:5oŴTi~؄&I uӔnj| ]_kayiGwVZX@32*ժWeּwE^XI\-okb͉'+ѲuOLQ]m0۝U˖OW<:Ϡ۩oNw%rs>#pt2.:o[l&ܾzUPek  }fJ-fΩ-R.{k uH\bsOA>|rlsz ?OgwUJR(iCc;,8UApyӺwqhg}]|v \70{\9&u^}vEشgw{';[x ]S̰ɌA6.i^t4oul{r[T*JSYXǤ_ԈuJ3m'[; $ 5!7ZihťmOfE.^`qUP~ ,;B^4TЦm8 :bW >v~ jӖz_ $&:H'dGtv=`G/"YjLN.Nc8:r6m{?js `f0e|0Limgi13ґ`4UMqޜA1֭w[ߏ4{RNk13|ip*%ܢ\!pr~Asm`0SY VbCp˙r ~   &jT PK`x^Exf[|lCFFcD]8 fPD0 't^0*h?UmR%]g_xL 4}tJ ̆+cGȗ>8҂[ՑǷ:>31c14ݛZ{-SQ`zmM D5 5 jyPig .xNR e<[*Mw+ EL,D [Oco#6Gd{*،+ꂑ~E3_&q sBy't)bahN; EMBq+F$n6f[LEl'/`f,Sn=`l] P5 }cMϘ)~{On(9h/aa/Q xp'[!лЂ^ k}FhlSE6`n.ZAa?\/Mw S `ЙhnkxuO)t{j/+OQhadw{2U|>]zWesa:O罿}~XQ`T{žQ#OvNϤBVwYB"Z=奐V? n6.ϩY oT{hxLwFɸu5v18O7mSyM\ң._L[LkFo#h?LqM?uNH6]Zs܇ >v?RÃB' r,M>[ƒ' ?;&}:uhӓ'OyUw78gUi3IkYᮊoCJf5SC߇:_^kp”Mԫ{f94DۼE>l89z\C*c_/T;[^uK'ah`q}C,NWuy~&/5r5% ZDYzW/Q {GnڟCo^Kud;(:>|uY?5hrE&ȟ۰GFk^W*r*<+ټX 11bxS11r1j;j7ľL/q1N̽rv贈_ tH x7ۤ?w` #p: q\I8 " m"&n~s]h431)0*a:i\ zr^6bpAƎ`7]Zܚ9D̕I+w"@p|Qrkj=V„h3M\h}x//p*k?vꍜ ŀ#@D|IL*8gĜ;n_@'Po' LgGpT'-7 VJ?u۶5r .*iyi3/N񸕭UͦѧLKKZf E~ӍS:* 5KII:b*1obz;[_o4N'|Wf*6#cH3!"^/UX3"Fղ.-m@.Ez_tS{ocD|uY([z8Y=Ϯ}Bs=6mkf<)zg[Qvw:5)w^w`fïjHk.o2shtC}; bۊͪbUl<8sע'6='RZ:vjd5-G)iSlè]f_%?X\=}ǜ_ov0Y퇝c)sɬ-Lz+kѤhZOKӀ_uߤbQO,*3hvS$NnSQFfZ]d+^y hi޹>NSO:%g{?#_8If9`4»,etUph'MZa &@s1 "cd%kw-@/uP @q«'ڟ)u!y*Lđ56`S>$+,tOڠGt0v1ƀXHq2{`S }so_r4p Y%OFtWU<:2g(`aI݁VcvߔmZ"[ylm =(21 "F*n(+dPhVYxmRq0jec0=7ѹOoٌb~p;_[|U{EYdDDd*nmg' Y2a_Y!kV=眾^o_q??y~z!lt)mgqs(1|}4|f~&-v~];o#iϾ^VC6o ;y'u]v@&bo\\x>-⌂.i]lJD݋s|nTa'^Ãˢ4[2`tӜg]oVYYZ衊"|sK\jӳXqb9|ox܏ ~x7\ݟqf&OY8DYlq nibkA3'MUq:'9cl{__=䍃MO9.u5L Rlr(kM+?<0˘MWNYt?j3>}kSaGdρJZ4OW}4tα~JmN-hPtxiqާ7mG{#W\ʲXb]Hnq+{mwӫ d*|.2*VkdӈOaC,-#Nf5;95n]^dWVJB]t]|p_Jtd'~!2ȣ0"@NFT d܈3qc{rbm3++uwhH匊7UmF]xͳu*42i 5evרuf@ꇼ:|5hbëU X,.u !Bfie BC@"'"r ,OԅDJ90)IP*ibTy 9hO70jLFe$o>-'` P]qPL%Q͟uHyUF  CnA֨H@3fc |͐dJ#x@@(F{1`&Dň0yYzK-yp4m:1h 9mPϝ'Č_5 > s҇mK3%8B!a3P*P eL?2v\{4PsIFp,}A77Y$:_@ 1Ws1#fxjɈ2Ž7G  PxfzIfmo.Ok7d{gbV˦2}{D*L?KxĴZ(q^ᠳOnp;H DdCo# y~b) `#s5pP,%->r s2u;%/}`c/@t=2;x6l:O=9šKpbo"*hP7v~F2S{ѤwbG0h/[H=aCeD.х<b[CXuA0>{O"_Ly0lv`rN>wf5 L%;xȷFࠊP9rDy`}'* rFXDGɃfbz.QRj Ja1=#4IܬiW8uK8a | 4~)[T>` 4c`־#2PgR C@^x:ԚFN$\Ť}83 }d@LZ&22r/ө L۲ Ά~*Vq3iQ;+bm4E?\|rR.Nm8RqGaK^ |69 '=^*\/2nJMSjai:PMذJHo9cUpCO'u":5EiɃV%Ϲ'KSK^<'F%'94GQ#|IU@혹Nǃ715ޞtb}sowFgsk}'B-nqr} nꇺ*L^5kyK=9?V={<$N/UN]|cp}ZhCaVuլtuxI~;71KU5Dyo#V}Ӵn;m^Ww'3mkBj&S<,oij? 4<.z0p{N'=<٪Id]c c "+JLo:by9_T=uJҭLKx(nѱ3ZfBz[Ff4-ĭlFZ]ȦgT&}zQuZQZgRqhh >ڪ'KSJZuq:9"}5=="Iy~Sm|x&~kz7&jq {lTэql:hFȗLx{>yc7Ҋo;t)ԛeTAlE`Ov"}7; ]Uhf=~ zpj5#}GYWw){L3Q{ɺJevUra3#eTE<4#T5FS8Q(1MpdF/@T_V"\DbE-j*Wqa4T)Ifb ,)&B7*,ףQB 2>o7@4dR~xr<`!˳ k^7HQ*jwnBF ! ,?D 2uC ߔܭ3CV% IGUI# !F}UmHE4~Thɸvyh[zZx z =3}GP7.OQBx'@ —%sL JWx$'b?E#F8rٹg֕@0(QO1xC|:@T  ǀ8ǝ| S1/%lg1+iFpLr5F9z`n@^c S73_yh]5(% l@Xxqy *֋TE +L`{㉚>ݰEO7p(g5ye m i%LĤMnbq:`=@F3 U.$^#^Cy|`&" `!b|ȚA4`\.{ 6GAb%gpE|EV>1 "fv"5R$v2 pF@ ?]f٘Kpm Z2(&B[Az4Q  6bfROfCF@pO*n-^qZ,b2`Mby y?a%l n LZv^ H!@LhNzG@<\@!H  Oԟ3m-l ^W¶}-tlO]0= w!3IdG[Q2;x  wȘIikͷltu$e=dQ5cѣjE.|rm1~/Zs4*.AmP-( oGJ 4Ow6wiLo7OU|j~ws,m3XFY,Q_Viv,,1noQҽW /_zjAٙGfkql_ʹU((Vqϖ5[f/vx都Uo`\R;PV eO&~u/XtjQD?Ŗf_?uUZUC[>f\;yط~z+=wu\XYƪrЌT _UPKhHR9LAnGWnKH%9cYP'ۆ*JreQ*)7W+VT7ɨQ>BL$oT$zO/bdQړE!,E# b"IC >{1؄7g?f&C"=#4Ȼ3P,BE!5a$gՐJ$r']!qM&7TJ r9[Ncm421Ld]HάM5jb-F,A Ӏŀ|leř IO{Ԛ2@U0RBTDYp5:*`l. XQN콳~*XX &µ?_d"`4j`G7ũcyC5Oh`0>qKYi1n7qف+T#y6{Ycm r `?m05!ɜv0͈Ӵ͸jʿoU_R o)*,U )AmP H 8up^g|:}?i>twb *~o=~Out$6i<bM,ӚD0ߟ?<΢>mIi#NJo "sYC2 #h;D̕  ADxpH| ^^{Q/sEtD/^.?n;;(UvG n:E GE&mgPEU j)j .KN-)24o z{.^sGMsɣֻV yO ϶ZTIm_M(DF/6DeErcNΙ Zz)lx=:Zv|/oW&mMސ}xFIW{N)6S&.oL~V^֞=?mIUEbTmH37F`?3^8KmZ[EQq-#sHCϿ&ǬG'(kqŔ)ڮzsX o-a]sߎxgƬu=rm`m)}B:=Jly7hz+ȭö1I' "5mԙ83U׊I&UJ =%ohA7ZSlr>vktǶi[;iǗ;'8R@SwD3ڽe*)]Zd?UJAQɳgd_FUƽ6KUj1Y6843&//Kpk%Q,\):.WM^զ\ֽ7KZ5X(Pez2S#K~6OMslqw,?tz~ֻFJDg{ j2x@Sm_fZasdOXzIgxOc.Z[.r]s*qσ;:fFܣ>i-#c]WX$}tKص Zkum Fm)\SG^4i'r.yQO;?t 5P}@ajooYŽ]nګkdkEysPϣ:*ݓ].2xDc=+u;>Y]mRu=)։-Z%܄ի_nK΀LTQ\BDpTF=ICfR 5ȳ6D \BMP"nh>7T( (${r(ZA\#'EjDOG UlhLC$ z 7,Sf4 zGL#Z4#lD䈄wd7@zFBҐ5YP6''Yu}hru1L)AQ C3"܄m. C*\씥rhQKcx%4q 4 kaLa;3OD`b9;/9,A"5ǔWsP^HSNn>!Cv@7%b8!!1Jxp&dzI,FZA%{ћv\G 0 N lce_E!"v'["IgWB};p`|!t'"_RP0#7/2D|JSҽ[JD20"Bϫ6!uӯzvK}vpߑWl,g0"f"gNZL.$ Lj"ģieJôhoaj N&XJ#8H((spARnN4g1/$b95:ܙ3C5p-4p /GU+L\㦎8>\ " #Z*tl{p^Np<ȑKA)Lm1"pb|Ys$٢ [2g֎mZ^9cD fxЯ4R3T?!ɗ褌ǂ.&`mk8.x_Ƥv_7~8ZFɡo]ֽLqMUJ*i( NSSpL)ܒ*Y([xXf I_a5m#SKC&©I顖 DygENKc?v#E1Uǯt~9aI_8y:n۬H}ar޾j+6:7y2ń7k~p\bBB5ZX+'6H%'eF Kpo;q3G˗#ϻT~`F/Mܔ+fsGdÊVYkU ;ydGۻ7|,z1%ܫxˆL:~-.'5YOK{F&ݚ`hV2Pq_Q+Y0(/μH݌˿dUƅG_epYn*hK[u{pG%Ӧ ٵҲbSq?qձ[R~vS ~CRm)~U`iUes3+pbfvۓgw/$7jf&_vW="}#oWi ]M骱cy :۾ nɯ4^q*KFlu{';lm9N^[Z5Kvq?QDhVx7L/kNx,Le򄩇 ja9%=6Γ֘~fSږG-ZTTӱLŷx1ю{Wmh6nZZ쐴WݺvͳJbbcָny%ؤTFD$]P\o*N±#LC}CD#-R1Óo$,>'/}ە,T㤛%G= ="BNuz,=rJKy.w9*{+w^1gFa~ ccu7hemd坦G.e ຩZ#޶(0\fRh%cޯ%̊ ^p_5 (ldJ* *5 Ja#,ok+P6B։Z$m1gyD@&wIV! $rPjD@$jL W,o u5 B\&TctTՈ4EuA`XBP?k j,#HB{bc\ FFf5I$)1A3ƀ*^BIA*/KNcH'4Q@P&5#溌Ʋi#TFVqzsm2y`bV,]Iy@))Ҋa;d?4W94< 4ZOgYG̦-n PYkS3DM|tLqxoThZW!cB60qR` >—qȊxr9(!6c#ӠC14 IW1 'i'| W_>[!i¾el7Gx smrq0~( |WFn?40<7lՔ}7 ch$:z܊=T B4mQ843a҂+ "`tƝA * ߂໏oGqHTD*1+R3@xt?ã'Rlf%X sPy䷏if$G3 AR#c7o1i$b\\K%PB  ŶTESڣW9W#D85?q/gLM "Fw?dn1H=w A.TuT~r0T4@BWs(aI>d6}Q)ƂϿF_] xU=ĜPO"z, a0,w#ɉSWf>Uh ?w@8]Ga7 6h[n^ԓa1LHKJqo<:#`'ݙ{zd>>3UZNgQRfg{"ᐩHWy.5r+ C8oR{{f#MZyGZvvʁ 6w2Ju . }lLO/N)ߵd=>}nRdC cӅEʃev1}^,^3jڵKLym4BvmzoN\5i#VXVwR4}pHmUL_6þX7߯}XnఴeznlЈ}yxYZ/Xx]CUډ:_jpd'dtYfα_}_g5i#-|u 3/βd5% L4.9'xOԑ6zDsݤ /I:F44K*߆!-d֙~\Ձ|f,ΜongѶCN_/˳+w(lSWG[o7FO:,R443^ ^5D-l){5kgʡSM~Ov)и+ꝵJgl"JZK#]^[=?m^6\F#nlU|ؠ@G4$G3+;~^J:aX}7_u LovW#,Oƹ!M-;gok/ns{EmOK~|padTHTͮP p1u 5DSp /i6$/'yo&g~!W%{}j|)HW/r02JJsZ#Rڒ+yp]ꄝ&:rmShe/HQQi.(\g9SM ב]~-c{|w`Mz祫mԼu27:-lZzl f#\Z jgl9iD88Uȓ\Od-am>2utJ5, ˕&T82֖u>p=Ms>/fy8o9f~m7?>Za`ԈW -jnUW΂Aʱ,=U*1n? 8|Z[vN뛴KSͫ(VcѬcnߧuPm)6_JB3Pgvb=5$!ФZP7C@@2y#8 sbKƄ{kfꆢXIY^@ZF.iH\-4lP VTnH a\gEb9OP^X%gTTQTS2 ɛ(q}?nEA=P P#)HrXdbA=1uAcLJwBЌ! E2 RT)JI,D$@ ghLCSc !׽ًQ0!*Xu_\w8D;^%b kY\BF61c'mCpoU؛W&$ZGMŕi& Vh~d[ Apkrw~D|; "(?g@mvI`R ,O1`  %b1,= An MJ2J͘  ?!}6gQk~]>`(ŠA{-p;^2U-$x |=qvwe|Pحj1;DPL%V~O1VcOF2/ @XiC82c^DF߅.:iY u5[#bod,oaN(ylϳ``GNӱDq?ϼq1`#a~]l'F(>Xy--GJ]]ZXxX hހ>YɋPΈő<5# -h.N:Cwuv v5ʥuȁF=~Bz>- gDd r;%еC0m hb!aџT ;s>ptM} ~~ >FR[N nO#8 XRU7QKB 9JmYfzt.-?їU.j,7೒]5uፗMQk&/]JP5P/;~/$gwlhP=^;:\^2nPKIo6Mvf}"$T`Ҩ6ivoYW%Z\c_$>:WwnGF}mmMv 3,[OUvvJeXͲyG/(zWrRN;c{W.z%dcV.ڡO((p"æ^[nm97< Y~r1ߋ&Vik3pNe Z= n?=kG^%7Z4:bne*{^9Ev>~_G9Ov.}t-钒^D KU~t(_c- \qͷ*%*J95 ުk$>Β|U}s=^oUUI~z791n| \夰iC>7)$wucLYLYSDʖ2_rD>w\|ԥO^Os˙c#MS^BF7Nr,l:iMSԏ[U:.uJCyת})/Sy'rőer43$r5^rA ,X}Ûb@tpݩLO?2ZsZ%cN.o38l>.w_%9y&[_/6;uQ'{e>P;2`sƔ\HnP,q'eO{ME>#4w0=[V2n\I 7=ʸRjܖNUM#̻>IWZth̭Foߗ} إ ^XaSlUDK&ۀn jE6c0|85]bƶDu]^jz)S_m: ֣VQвVFq~qNF73pxD=tbt~ÎW*Қ3_ϫF-H/o{"cl`E}MC+ :ʸlΠ3M }BGg#o~_TpnS)[9,|ձʙ#S/;{R{^?觤wt_ x[q:t'0*I,0}taEM&*;jҼw~M}V۟)o?Z$t 北TsC˱2VVWk߼ l%_qfC5EJ-=-{]kREmyA5 e':Fd y8yjRKiV_v{ +u}=jԫMل9zuM鲑<$_Qni!C_9Fp;u$_hbb${:?}·s tj)vگ0$TilY34V74F_vRFP3S갧]k߻_}{ۭ[J>,-,> Yjʮ_K&4i[V(eуwWz"x(`{|ܰ;7mgc~XwezL3en+m@o}D:/.oOʧw%U4Fߺ鿹&on:Njz z\5v`{y^{ǾqnS`y@)* (n(_QYǡa4I\{+t8TBNWT&Wb#4 pL94$x"O7Nk8e-i`X/eer N @Hp(`"|0.l1Rm1Ê baxɱh#rc%@w3DRA0cp(M!Q & >׿0fNWD~'1<h"3iw+W?@GNՠu˓F-(]Fa&3vNV+o"qNضt |`&G08tz>Ʀn`F3p:]bvXF^iu,`%EaV%T1!?K:4͂Xv6 3{5!(]q.Y3m£+AZ \e3B6[d `ܽ0n`, 1l%%jC&_/ft"&oMߺWA5oG(=LJCu+ӓCiU%}:& }ɏm0{u--ZOynɏ?w_Nٜ2!-[}iYjUc07OX:"AOw;}D=Vt\`yu{_o溱vt%ikp}+;ݴ]. }CSg7-9]fُt"8f;'׭f~.|:&R#^65)JwR SV9/ i[u +a 5s?lMdkCQɧ\p̮Xh>1U.otӆx[8F$S+ݝDͭҺs.iɽG\"ԹUiÚWzP&bCoJZe bG|'L:xmVn_өu :?ygM7iW =s{(rO}fKmQJTwlV4M'<7_gCl>F_ M5e|w-gglx}7ŷ$~K*ovls%P2pf3&mAEBrshYzx*-:X]l$0P4cL&%Q`H1ui Xe?͝,@qT1[2W6)|g1kX4[qߍ#&|㐩p[H8hܙ} ^9 ۷iM&yi D;f3!^ϟb bpg~& e&4j}2o [} 1\~Om2Os6bą F$  iz*GK@k?YMV*@S*M4np: ^}!`1 X8/u+LS\)-slX7'|fa3B?vY#/5(1K 41版d`<tF\9`@#Arߨ;wp/iao]qs6e1hj2>|0r@!LWylxgmO~vdq4Kb0p`hVzwm#+=[.r8j7V& sGcYI 9 5~ Q݁'; Ls8üY06a0L ٵƂ"!l1~jwA :1t&@{@hd1p8zg?s Acg(Zo: !"P^iP馠-UxgawaZhM+B? fe6-:f*ߓW7yg[(ѳZ/'r}@%-F>0|;_M2:'C.6R۬ ]y|-Lը5'|BXy'wi^¯}6ښ*M MU:#TFX$e_V6jUۏ) ѿ-KYGhMRK஭/}ÆӜo,A)f;6IziF'GfHt¹" K{=[Tk炁ofa75vPH;0C^Jv^6f^CMCQ}竾G [RvPOtڈ$w鈂32'۬1]vE'=[*<qZ{~cOתMnXf憇|%ĭ˷ooP흒]S}lq$oL~ڜAj/u{[&f([-1 jRR\Q+T3'?|=ףX ^}3o/Cd kңbS5)g>.ttq;'']p5..xmܹ #ͦ|FIYoeQc7E#~{ޱ7Zqq|"yئ4h~n^ ηϷ,d[1jwc .I|>Ѿ&֙=\.ٴUM-t޷w2e[R-6{fq#NԈկЮ^ O;e͛9dޔ̧c8tꓦV~YgQU5ּ̖ͣM hzM ثx؁e"и+o5=1Nۜo)]Ⱦ7ƽ|wT{Zt5Hj-N) `P+Ց_yw'^_=KUrz+3}~a/-uERl%vL KLDj"gr\AA(QHL5(P+JV\QP/$*)d`*Wh+UVCL)*ITl̯T/Rkp*5e2*B!C!H&Ma,~u.idٯq Uoc\X)- $ 4ZRQJ~fI?*Ԫ7FTUa4$*)aFL$!7.UȴbDId I`yS\^YEB.Rf$@3(JH05"~:j@1Q[Yz{씿l[L8j3\t}G4ɔxh<+ a1@. ?~es{3IԭղŸ>Lth0ﹾn߱XAbJb"wZwpxKhPC+]T~2~FU'vc`5Dv›IA`qѫ#%fI !&]މ 0Jyew{D^`,s跄O!rfж.Xn)VKHoGN!RS:e6( <<Ys gsi/ڜB"ƞN`bQt Gt&'`,/K0@2/H1nWǡ_D Pxǰ&)߈qpN`8|J0 a_oHzvze s6wN mqAc8Ml+j(ʺG%|@-{,47=̑|j&xNhP؃;Q|AFD'0){3'O4i]``˿rjF?A,vm`5vwQea@_ ^}C)ptu`+*Ix2NeXCRՐT#Ց.WoƮnORxo9ov4r݌Y# [jSmɺ޳\_^" g{T-pYJ̔E_[1QD^ĕW[n[eFua_d^|^jIT3Nyl|:-jg7pE4Lo ܓX>{؄.kk:PۯJENaC9:E}_U?&ύb=ym~Ь,t}7ɗ;}o}:ab&\\3ۿklV o=zeL55oy+%}EQT拺ivzaJ6_sRZSQ8,z6DQ0kTStrJ[3+˛撼Yێ=oLvdGn5 6Y,`Z%E4߳gFSYSYvOo-]L =[yzg0{5/7mVݿKa-oH74q΍5g;{X.tZ&~]-tqwݞWj}=,Y⻼k}jz͜VNNY=L}U߳6:xiγ^ }='mznhk/7Ƭݢ9NOkax5UR1&4%lwF}r-쒗?tD:Q5-^0ti[Uz> ^MZ?5 ،L[n9ނc4&{N:%D~ Ώ9fԀO-Ӗz:|ά>ۿI3S(3ϛdzvި7[.|b,WZK VF7BTBUuժ<7|7"ڒA}I^D*ިnLP1_U#Tf4UUlW.UL5aCEDHҀAVvnvVRWFI U5UUlQ̐@Z7y$ԏ\qkeZyÄRzwZ$䢆C԰ Q3bnWYa R"J³[儴x w8F~т E0gqDND8z~_tecMelx۝Oԙa89t܏i2xDC-h!0p=>pXu"pKzgӭe ?4C˧diͿ-?x[JqZMf⋐PZ7k$JL`O>_g˱Z1 X: ?pl񊩼",DDOrbB`(2@Sftg)B f1Í q b.H:Vj. gG<$0Ngk%$Zm ίa&8HfW?Q)&>xzEnx3s ͂#)Ӗk %t.rf >[)Z {1p7~$PK[G ( 9fCxԦ7-ޮq/Ύ #3{}hZ-h$M-GwtVπ˳KG}0cT{{V00Dxu NQ`lŴe؜a 0w# m{E-ξmڿ\Iy6ہM}|嘅C_/L=/Yݤoei'=XrJjG-UwՋX7:jl:N0b>3-[q״}gؘn&[%N4 [9Y"7m|2U#C2-qj֤/S,,xwԭD7>3*(56=Uۼj[~N_fѬ<\ͭ:EVڒ TBO9_RX~UJ98V:9q5.Cp|?7t{)tydUu|O\0)2*ٖ$f=5eQ$4.#Go&Oė{NeYb~iwCݑ)i}{e_@#u=Z_yGZ_q 'U;{sQNb/UJ.кUS_U.onI=ʏ;c]UwC Zl/Q$5T\_Iz=o~/M-b7jI17vcNOI^@wݳ^,(}?UY|Z ^`Hv˔2=z[(묲ڨ1.'*72luSCwZ~Om=N>ޤI,s]s: K{;]d8ʅK|YIg· CB*F;!-ڵ4y~ʋ4cSURVĩ6V?J[V^"SYU&_MPx1MYhzlY/(zWwT S Kag[)rAZ3E{!ϧ] "ճ__% < p} * @N Q-E me"B"5kTik/ma!\ %5R tTR32>"TgXUV^$X5#52j,'VU_(,( e H_9,'RG¿tBIHXH)MQjQ[UTUZZU QQnQDrjz f-BE T)&Ȅ *3Ōb)SA5SkxF/5z4GӐ QI^_53㯆f#D[}8ۯ:O\vdAx2\7SߖN#={ uz<YImp%ޓBEUL]+ooT) @! {6f{~C9 l uMDULjf@m7Vf4 M;!5ڀs{Q.q;JH'2y];Ksj225pCc ܧ0-zoH~޳s@:%q)sUW` _m sL/N~/&]gX;$#z'-Hn+8!M$SQq?HʞǑP_{XxZWf\s(c 1|f*?ks;^fS~;3N&vo62m7xN#UsSK2\qz7z@e ϕyM`=Kf|=G+$0=@g~KC/m~찍gv= $2" 1a>MČ.Fq ^q܂廡yxO:UmyAzަS~z-}jln+Haa1HM5h]#Wri?Q=Bt֎Ѩ`7Mn;ϏO3O_Y6Ir݉%޸j U*<>:S&Vm3ڪ\lӺ]alų  :&Oj43i7(xnV' HF1#.c%>NhgD]@hu~pbOMOacqͻ r}z*XW$G;N]'U;Fn;Ԫ 9E<*Q~3|W<Ȕ.+?6]N3s}^k5W3'(?Ґ7I_IXub϶uze^cU_WlsRfw:4Hw7hxy%qNK1[QJѓ^==RviGT*LͫȷmQ޿󲻷/pq年'8yQ wG̎ڐo-LM߫^<*1LM}tVtlJbMƘ/۔8r086w*[7hG9o)w*.QFOؒ;M&ͽyMxkZ.kTAtL#'yg;ukWyX;OEj;*vѧ372YOc=K.~%y@s/4;ӴWW*')E/[{ۜo4[ sVoVO6|tU6KJ/?7w䙛&.;'[t)>_gEWMN6 o NT2W5H+O|;wfJXcdmy|+PrwsEDžʾ֠U7/WTljY;`FC9j(\Y`ρ.^2Ԥv6bwb^C$tuEy7?Ag߿ZJUJ 4" eOQÿTbjU V"o4ȅ 2*_9#X^Ϛ=3,{ 0ra򍬦&v?_ҴvG&L,D'86&)<ĿyeVH'we`_/TH.x$~E0i.$~7e[RA`D&*4dw9 NxUw]h&c\ǎ g#!/Ahjƣe v\~A{\x<1RģAOdf|8c)|qaA< vrO`)3AQp1UBJd]/Zs7jAAazM.rT[ ȥlW6:`kQ,YϬr$`,0tW9 MOW_U~do7m]tMA.E1.|x@oWc̻W.{j#%=IJ,S](>%weMFV :Z[Sϫ1hdYxԾX/C,սzx^Kїf]z͍3QzzEstl)k_zdlwmݹ쵬:,ˏ*C34?7,բ;uota@2-sCOUQio"ŮCٻ--}}MEs5c ߶YX28McjyxGl>s ek3.4+5@-a_TH"ĄE=BUy/nmYKVy(l'vղ1ۯT#(0oj;y#*Wwg"|Uߡ&f:|׈MǚD4\}YtL/nX.7|iX jz71eL׻#Xx)fxJfl;^$mxb˸%#\>lNH|Z{$hutTm􅮿ݵ&?~FUڻܳ5ЌR-%fR^܎f\ e6^a{Z:J3)c*ޭr+,yӽ'N&$??sWi]랳OM SSWZ0(4p76xE&L3M0TQ0 Sճ__u .CR%#AEB5jUp""LIpF$2T!ZV";1jjÅPK'Nec5ףV]O#LqM.Ȕ8&52aB_OLX/T-[NDBXK=`(% rAG,4eF !J,_WL 5G3P0%{+.ik41L W0y@VST) (+5#b{WT[5xĒ<I~CNJXa=%]5R6\ Â_E4nlABӬT3-( 4w̒i1m vYӖ( $m^rs:q'y)MЖu$D5P|c[< ه Þv&}C&E+C?g4oK6Ą W>yD՘q'?2!A&I/06p(QFS,H$Ms=RhP;k[{[y "򛇥΁K&fqbB f)>0'&PBvIsQg:b<l<Opyk r0c.0 @qQN}qO T RW@QߥO Ǖ5P^` :r`#x%w0z4_G{;5EWo\Ǘgw 7 TYkE;PzQ]5uM#H-/9:$ /]}ooU>x}|2Vj㎦jID}uSJwn9e\5w÷}7OQ~HkznEls. Rj^Uc﷓8KN="qZ`=Waa:NVi`y=J;9껩^!^1EadǷӷS^t񫕧j.tҸ:eF^/_VtɸYZV,ɪ=f5QjE6&w{PJiŨ9K7>(Jj1˛T2=I*w62iCfN8Y2DCFA!nfl*]ص"ǛwbbM<*d?n'15maQw­=:XO|U YuϋY2#7~ yD"K-]V][ciKݺz.޳~,^|$P}}/Sj/NޙЩsE<\(I̱,:U筙裆Ҽ{M+nw)ǎkԆ_=KUqcLٯ6r)U@,*5\JL$2LRr9mm, j bɄzP-PaR]ELR6_^T-PA΍\*U&41C.G}V@x5"Ca  䤸&gR\6p }mb*R@p+IRKr4uRQRSaTShػ*n$VFڰH$w5L@T^•( L QIJUڟ҂.AD8(@Y F{!B5#lGL}쿨S) #*r$o% |H *@{\,c9 `}̥{{l{|* :#b~',=4tơ;@[k.Qt}& |-ʊ3h*p kpw~MYbL6-@GF a "7Wwjn@*Z("9<МRf>ZuXrKi @u,`ĥm}u8] xb[tm~'㭜G!Ax9?f*D7@"#ē@y۠[\dĺm1G+I6c_]5;fĖ/@F }d>nG9<*Lm~R2n|3nAxx݇pP8ڡ5 -Uf0E0`sTλa-޴0_{\wK޶0XW! 8%e̱s]) _v2in;[LxCʰAwPN800h0pmoށOMat7bp@U'{i"S لCurKL|1d0&{>]tgE|i3B&HNXjF?wYElׅU{&uM{°Uw!=q>ۼZ񫪸'(%>OwNT-:v+~~i7)ۥ;pCȡgjZmX]J>u7m~xgHltE&m.?{ta|n,>fC{<”{\. 13#ɾȸUKn: ӉsFtw߈8EصtRvwkݝg1^h~r]V7.ww)F˭vc{GRA'T }[}9aEmދ UJZ=-56HYZs󕪚_>*_r9!Ǩa{Y7£mզRuY' ]h`$~2D<Y_Rwt7}.lX]p<0ΠK^ׄK{fOknM;_^ZvY\ϧ'yrڞw<80cNn}. PsmΣ}K=,p(]rVU}7溭׷RߦIeSˬe~sT^"ꭑz}Ŭp'kה'oO漵"l@Hǥ>xaQG],ݜnZmZV|^?X-yM5;nb`{xڑ Q-m8EpXC铤ˤXf~)Ñnb2le =?(2mot9ǛF`nY: ~ȥ$WD2aCh$WP[+(M&(0~e+|BP3*k-&e%ESZ^4d"ID.TS),W/#IɈ5R5DSP,Er@R#I*a#r%oK,'Eb6r>V,B&U@').Y_Q nBGnW_?!u[BORc*3E͈%`pXVLH df Ƥ.7 hFݨ(PRTV7ڳ  6Î$;P뻽^hX8@~q\bV_z0tKck0>%R|ONGPQ;"Г.0 F}Eoj}\k\7`^&6z(9n]H@H|8/`О9 [/>a@?'{; Ԥsh)o\7eTM˦U6⥄m ،ltu!~PSdêf-VzWڱIK *' tglIBaE.C#Xg;|vp~%.w۩uh&v2䕚+NLlRZPC6AAx[-p3wqa  }ԕ+; *>WԹIӺ4mLG VUٍYtfڪ$̒5{iyC3ޡz0{ zvM2~ݽtBk?4f|]we+M wO TjAˢnMԇ?:.kvuMsVnj0SzN[(V+󊖇LAz둕}f=aJ# cՑwל{3F:|+zZ)v4KiL e;g+[aɢ>.:_fgǗ߽gד>O)cQѲs6J5W\䓽Zn^f\}۬E45W%}?.V}ZJRS3ިx_̓őo)rkvųأ̖<[$I-$OeVݚ9gWeN_tNiaNs%V{J_Zac"Ji]'YԯH;kEjdkboC׌'X|$09]n'podq샒/-U /W{X"7viTWU=> +u[}8|9~ .=?\kN)zLG7Ev;Oŧ^pcM۩6#=/AiwtY!iqEZ +=+NI5 IX$4HREKK{G 2WPN$Gڴg5s0(ɡШPrV4ç@Ȥ$r5#4b[^\]ݐaYCjMΊA,ude}D"u,)53jkEzM^"!"4 Wƶ GH9Ab<UF1zт{[L(R# 65zNln5mv}Њ}ObzDmw]xp{>%}GvF```)݇B,7}xB?$\ms͙9nG,ğahL`NR2 nlCEB_4U 1@**ȠO=EVllELы'y|^N zḟS(43lws'! p%&!ߋ`L"D<DuD,ɗl1)< MkDSk2Er%h q}LV3q!x ڑ~ǝJBlKs8ќׁIf#1ӄ/bWAXzsFs?@;7Sߐ]f'Lt ) 1hRЕh,'3 8GDD<K;qL%xJۍَ =p4?Mهx=XG /fKP 3Wc3>1bz Jbzpa1 4KNC2^0{q xd!3z4g4-KahCcG`>=10踵!b;vz{>L\ANh ]O'oap|/MBM LqcWsoL~s(,څv-˿rhoQSFkZ؊6'\kQ{s͓[ubM?ܶw^]&YA+?H8cm[f[\yEb :tN;*Z z=})*kv'd= xmT2yԦMroMr [|Mo<3;2Xߜͫ yub6ðm:쏞G>w5ښUVk$.{ "%ǃò:Ƽ'35 ]ƽ~]x"=ΘeL3lxAMt|W9  ϫx*ɈQo^u/Oܪ=rlJŏ䕵櫟7B:/[XbT YV@Yf`or6/B㢣DsPq,m+F[1x䍕o^fGɳ5{[.j8T~r !'uz>ĝj'zfW'??&x]]^Uތͼ?趯ةNaF,kS=:{j#a+l{?ｦ,e ~om/*-(ua⾊yN x։Zc ͳ{'{aŷ;5NLZ>qFrEgsUdc+WԒVK$8l`'q5 \>Wsy{=]E:)5)Ͽna} X}7xKx[w 1mOkSKI̿ \cFzTdjA3lBnץƍ7}fdb@6u0cS7:%ڻ@#YUg_ڲ"=}G.a1{ZL?ǥv']?ddWJߜ]o{ƨS%C/NW,2ڛ|{ })ud)z/K.ȥ3*ʔ ʅ\.7\.?E7LL.Ʌ j$HDuwW0qr%H 1RD P)Q*ViOQ 2aHZICHBrŨc" AB@95u_FXGԇ!X&8 dRae\ S@DTQbDZ! dbYR+\);P%B'/?~FfQCȕ!SUׇ\!DC0Hu1?\LF@"M^"i!\Fh b`HS1p t>z_=@cЅ^ ⦳iIb$ //٭ iIq*. dZ?`\HV٩gpl:Np 3M2!<01 '[R7DKسp:>an oyX}v R ]@֏-qe(+_{(p_!w0F%o݂*d1Lh xLܞ得2K u `O,n-P@^9y`#zjB?:< ѕf9 0GQv_x5Dc+-VN7tl8e ܐdEe8@Z191@:+  oMgN$Ĕ| V,e `?HeMBc,}!8M$0â 揣IF?6wu|W`i7]k\y&%XIXb! 14j73tR"h^ ^24mqĎ;n.>q/X;QVߟ(00nKZZZ`` իbbb}׮]?TuWȁ#?zhuɘVVN]g1_gyƎ*396ˬeY׹^V}zDgU[*ftqtP; =mH{nK?pcd߾6S6Ƙݶfմ=cŲjyi'6sR{PN kiK+S>%RE?sƄ>Ud*1ǵʹ[Ksn5'_ o SLY4Scn-NbTLno(i-OPs?|} EД=ӂtM۪k{khftl~ >Z]k56!צv|qqQO_^[(?S,SM~ndtΆyo!nY~}H}.omY]#ߦWuZsRtvݨtiXqOrTsT_U'=8'v\u4=Fҿc}|_egO(fĪ4[LU8Z뵚||D0p%r[;"#/R +?ib@鷱\ֵnzuɂ#tڗdt(Ntr+Ug-_]u?iʲ~?.C_;5_{渓tW+4^o>+*6y\(^:CsgBe. ]d{d~A̓|s(fućGݭF]xߜ0-+isx<ȡ{eM<'E6.&'n4 ,i-ؚyQށImsUcHj[wl]fBjj?ںhC5U~UW,uBanm`G^{ Ūg|d^o{-Z)7#QȲ˄83u D)gnG]d\,m-bi÷:>8orOv=wQ$y@V+s"$HY@&V+U5bDU ֊rdLN5T*S2w(z >T%) 䍬]ԿF\d Bi}HDچzhtJ@ rjA,/aՃm!'F&)1H HHIV#l'JF+ WXbnUf\E`h{ߦfVPkZ32J.MF`AU:-{SɋL 46|҅)hjtXD$ (@nx h U۷\@Nzmk?M&s`- xLtBnIc5\Ucir ͨ/^w羷-\h́cFUQZz [+_ZDo-Bcq2bx 4n= Go28f ӝV1~s.P8;>9DDh$ 'aEk_(+ ΟlB wc\tkfsuNhJ'a'.N3|_/y2CNB!P6dmG;  1'Go40S;X.c:efi ‚@ޑsq-fhC2GG6w!` k1p@ K*ov]:Swm~?؊4wW>HGj-m̘p(`"\W#hY^4m >QMb2So<Dꘌ69)쎪18>?pC0!5}A.8#}t`e.|4tq> 5@m,[Lm`4606b5Ӱ!i[G'"ok˞IbG0L:~Si> 7z@9NƜ shmmp@@HKK1bk@@@rrrp€{{Fi***V^]wU ]XLL=6lPI\]]W^]Q0R$0H9M/ץ}\й{m U+tg+0wVv;,ҥ_iQi(]i֩>6}@o:V'* ,r,(h*S.|D:X){ع`lȈԎ53L]x71tCc[f43kXQK|7|U`B,f3Fb=k(q n6 RPde257pO]q~dMfOt'oQ&׏J50̈(h hxD.޷)5'$3Ca=t7 usmO^x7qGb;?c[F%c.S@ WV*0ǀyx(I, 2a@;FoxۜIF@IY:mŸ^>@Ozs},%S3uu=ϡ҅AnD+w D,!b0Z\;Zl121/SQ<6_Eأ!r x۹TѸ#:ϙq>}U[?~Utw *`((vwwww׶b(JwNJq9g}1ךsktc\k(2JogWGNDDDf]6x?I*O>3G޴iƔ v2MйӥCIi`Ltqvfü~`v[wEF!+n>*duwӖgJKlhVVk%0zupӾR|.V5`hY{|3Ȇe)K^W U R˸}^c+{_;T-qלsKRz%rGihEg׮zqCIn]3-:tUEBO}3&8䙤U*T2kUEÀOÞ8~ }WF23w1K ^A2& J{W' g96Y͒wOvt|jw A@RHkz(t݈z{ [|uH|tMZ;#W[kPUg :(Sk;4˛/s Zl?W_޻dygбIK;Ta6-c^Ԁ[MoDk͡Qm~PgGШݕloa~¾Q\<%d>LN6zŚ}5"l *AkNLq^m$_WW ܾ2ƏmMؿqFfv<\=սťR,7eM+7zl%r߾/na]?Qӽ&η_D-uy$H]Lo5Ȯ0-a[Yўc4EY:7,(~q䏋VL*ϱB?զz'?\T͙ڰ=x:M7?洛vyz!!i̳Y:^WN6P柬;wλwETe3B Wfg5P|`Ue*.'ӼTްAZeg:fq9w(EӿZ.#7PU&VWNGE}kA EB*D*%XX@ NT+TnKap+7?sM#@! H!MYQ)*AXA5q}+Z=,%M ؽHLNA6 YKB- uu$rZmmR#*j wjuT0jZ* e^@"Dy\'/˛(VժJרj:])MKPɂZԺ |Z:Ox†3p؜ţg@%<+Q9/ِh1 }t>al[۝.$N%wS ȥvUEh\!YK{r.h`۝0L7Va61" :V})`\@+&`ȍ,gBn`nJ7Mﻹ=L80,FwIx%s\`A"heLzqg78c0PnL#wf6q.nvC 5!KxFńB&SVT% VO@?rm 45 o̗x@k>R7`x%0)pTA4їrBa9)6yĴHV rmLvv\M̭e|S7s1 1\At!ѣ`6ՓKIۛl&41{7 -zg4ΑK & (@Xӷ5O'b~WXLhމA@۸)">Ntp_qg/umn B=>ЖAV{xj | )p˒ uu&Ux!-[:~7 A "n AGU5OfKd@Zt7@YW `BLo iI?!LLpf~̌;qXqMt|.z"Fw \[NۊAד?]&}B>nAP\vh׷SN?s̙;&$$AV^͛ev!Crg\SYzu||#\vmrJ5?Qxx̙3_VZ7~s@q&D2 -݌'D#"«wY?9,-rmVl:iwvm׽"-٢&æLJf3˵{l޺aZnu]@iLYyF0mcf,p~'&_$={;6Jjt"J/Mޥ yIENWhjhYŲxO_K[-:"ͷOܟ|W@1.#_?2;w4WM+]r_s勴AJq[/]]yot&xQa֑O]q[%m.|1֦:XRz̝+[[ֿmǯTGNI8A/i{8%َƨ9 ÇzH۷ڭn.کCVJ͘.9{hݨe!/r\vкi wj-1,:lbS-flmE!Mt,}CV#-C&6o}wƽf|SOjeOG_o޲SR{o7{=1Nz_2![Lr;^V_:"tޅWޜ;I;6JWmmg޴Wz]y䤂IRo8垢ѹYToJM,R#KJws #/;H͋9R h}Fi!iz S˴\RO&7݆!!'f:;]t|ڳGÖc@_-o%jG|kN-H:q8./F=}6]+ͬamʣ2F[F뺻Yl~tĞ*ٳcG,;wDu?RJ%H )"'TqF(P2B@P$ D\SF" H`RBTɅ?)N#/*u(,W~u0 Ju% T(#_2/RԆH+2")QU due(Ps/pTN2jR9D kvFɯ:S@Yڧ?ݔPb"ԶaA//B9QleK4{͸CK&ٸuv}|]AN$㏿3Ӷ"h?nqk؜+kX6`O4 YM$i: .ŭvp<*n8O% >Vν vB07@8mHcp{O9ʴdTP맠4;3)x& !zŘ)fS+dw=~!"d~L\Ax9oS੏q3@eN v^zn?D蠀1 ٴ3aNK=O3ЗqI9p.olQeOsR1)bFTªL`$OC12q  >HFv_@_hdh'N6Q}q OE#2HZG >[~)iwWD\?+愓Ү v'M Wщ=L$బJ Fdڰ!pD ;1N Ot1gPX{- Ncz@̖h?c>ysha޸GwT&HkЎv̿MOВA@d/WrD3})h-(Dp tqL񁮀 QAc \1'#oф}v~7 'w"6zLqo7]6e Jsԋ8Mn4h^䵑޿({]ޝyW\ez!cj='|0iIyK'c}iz`QqoW⺰珛 LEkW*7e%y$YViǖ/ޟ,zE_u^Ly3mHÿqy_Iu56LnݱAE2BK*4${&WԸC $.gmDUO=˒˖li>ϋMhp'r2v{ʉK*snpcO[sdfaA3m)k* c( H;~S!Ole7y=^1v8SaVemv72/;ޘW&grLǾR0/W )iTN׶1WB7|IR~7E QVw-4NY"|;pmj 6حi}"]/u[;mMT[M_=Z̚8XE<5Ce>=I}ꭁZB'qx"* 3ä 0_3Lx!rb(v4`37Zva\SާtXEcCQz0} ɞxT|90ڎ q@e 'HL=, ϡhs^C2 %/fC+4LVB r&acY컓8iIŃw7XLǩ >z=Ǟ B2^-AZ7F@Kt"g-]^ba(R_"j@ֲ ~;d7`5aHlRh10vIwTMA̋*uD`e)rp>[P6+/%<c"}b=ׂq>o;Z|%Z^6~e{{m0%]Q'`R<ɶ} J 7i h2>Űfw|{1/ A졍 ^`卦z.@fO^1Fw!">oLSmDb$SbCb$2Vf ZoޜϘ34W ؚDRpS fkg W1f^QsY f} 8>o .ɐG?Ds wU.5&ZgR%og4gXG`8GUvgO1¶gJƛ6gV˵k|||.-ӰZzj=]W70 O9*gTw! m/j|; S-,_lZdd[#Iito}xK쉷I.y~vRDy};v-[oOLJ/40.̖Mm d^!/L릛^Jx>tvQy&?uKOBX|dĻ_-h]>g9NqXoΗf΅\&5lb{1 ~ )hڑf; h7,p9Eow4&etyx2rs~?ٽ$4Ղ`'Ysr&ùhe_nc8Mm~$ LmLB?>pQ(@s[d#b2Tr rppk h).+gbO9X`ӫLĞZ<ӅƯIxD6$9m [OYJHHm|\}i0m@2*Z{>uىۘ4N3'ѣ]G`Ώg' )Lh"ќ=``S>IcC[39f_ @'ċ<ƅTz =rC_SW)L7!{܃-ph.AB8+QJ>~!]@I{<ɤ7=ŞE8'^]e]FKmtD2:Eِ96ODߨBL@ gYy+&xфz_ Mڈ|)_yQ  өS$hvjjj͗޼ySaoo_ѣO5^2~r֭σϜ9vBMuhhh :z;22//ܮ];aaw+7) 1mdklcd_m*>uNPat/LbiN9?5vU_5Kon; [Z,%[}McKk`M;~kR@̋]z Wj-3iiSpKm$nO3J6$<|`iJhQfF]?|-u#byo5h~VFEo7z:7IsH@neѷ 9ؤ9O壵eӼu]~؛?]H޵( HZU1$*_Fļ?gYCG?8ԼŖw*=w0p\ۦD\*QNo-r2z*gU% .~Yz]+ݗO1hUґ2 r:u['6k Nrzw~hvX73IӚ?ؿBkQag;՛iw!9w=wymlLoʥ;q M,ցg nWf^HI28YlX_U?o5Ioٴ|-ٛ[?bȏFU wJ-ky-PGg[}ͧ惗nCz'UƖ;5-~_6\Mۼxl+ _;:}RҺ-v65-s3ʓe~m9E$/O_jS[-VJe36*VM.jpWSt'=4ٍ Aм݃QTnY\;vE6tvfчEwr_RI=Mlݲ9T}N츲vF,]󣢈i=kPϴT-bFL8q׵3dQ)rgǽ3cKYK{A#^&ٷoc^ƫ*;+z^?)YP ïj)ߤ/74`ZЏ~گ[ˁK~^,n^pD=$ٶۿwuZZZX]@^2\WB@BD(Rr%TIJ:\(`%Q|W*JDJ(uT/)#Z|:iH!;:B,+Q6叢2%K,SBUUQXU@$)HPWu| K"H _LTJ۩SL*"VfUO Ax:~uQMi@(Q4C&zG6J? R(;e_Du9 /(Ds?-ͼ}⦈CUw0߹YXs8>` ţ2c6IUsf\F?vR=9¶PW[SpV1`h4773G@8;Ǜ.ka9wf %n8a9RXj0066~6&1F3G_  c;Lض!{9g3_{0ۃka9-ݔ]$<"FBTܧ@OK0J{A̛ CL@(@O)"9=BPH􎻙qLaxxW1"1:L`2 ɡ  x]pv<8:mhDg .π>h= fvZ޺@-s-[Q.) [ӏT3N?eŝ)`z9dF_[}Վt1گ>ňHy&_ !ynOoFDD"7P` ,{SLfH5: L~8Dی813.Npg!i2=~?1l ld]> A`2Ng6`.MwYuFJ@` Q&gϞ:OիK,ǯ_ԩՂ\2k֬yMF\\__o[JM2ٕ5ΟtM~}0bO},Kǫe~Σ8W>~uJDX}}o9-H;|" m-]F{Ij#>](ݽGֶ?;~MVc8)M2%+"S&6=H䑣dS&OqܷI[-tI>y}T0yۦ( w\9Qgol+*W}4֑VWܓ[ll|ѥq_:$tu7[cN+On@j 43+.-v-Bvt33 /Ŧ "6L!ylR6;TX}Q>ť}v~>-~9b6N >yQal%g,.8]*|16(DqЂR>X0p7Ҽܰoڕ|omƕ)mc5ǒNq=aا]W&Fw>W)Xg$iT+2L~AS!7J͜o_xE*^Q7ebwGNi{SNe1ݿjкn;ﳄ'M$lu5S_SMVijZzg*VAӵs|tqqOCЌNQu{}e9-$2a5]#Df7J?oᇕf^z#ӏ o9CX=ᕠT˕Y+Ղ r[hpR֐GԞ=5ϛ=qzӇSQxp_Ͳ2`2}/66'.Ѣk̋0Psޟ>pieRJ_ZTJTCZG(jvʅUReTArP*R*VBJXY,QBV!FKGgUH"UB$b+CLLM':JhՅvXM*Rv~aUf-(XWiF¨XiD,+kE]BR&f/ˈjk.h7X'jlSi+ii]|7ú"WI ˉ L$3?Vh[U 6< ܶϔzQǁW!Aܦ!~`+:L?} ^5Zo'ɾˆ##@@FT<Ծ#mp$0s A@$0#h\#|X yZ^U^߶Af--7H6sHPhC:!0[mH]QKsqvvo!Ap{;$Y?\ܱ8 h2hi i7+Z>nGWG<F؀ 0ub,Ŷ!]wv=">䱋&ljDl7 d7w`kfvT`e!<͈驵fJZYkt;6l&+EVf=9} @Ħ'/@hߝtZY 0%ʼkH hf6͹:Q@D~>=ڇwu9XJ$eØ`G0"[ggLYI~K:h^m.@K0VCWh63c1`CKv jtAkmw9֦K2L\W񛃰$߬N3lÙG9qGꁀ3 z,d1Ƀry;۷oT3Kva2zoYWn11ZYYWfԌGM4PObԨQ?O}Сf͚ux֜Ґ!CǤI~'cKܝտN $Gw12_jb;א՝nVoUFdeݫbUމfM>go) _sc~" 9o7~k3t9Ai{ ~a}Ϗt5v5Nv#g&%Z7-gTegb>3i]b60~hSD|n2zcI* I|>xϤтO[n--8Hn%8ÙB r4hzm==Cؖ]V_on[ ?_ 1EWu ?JFJwo! ngosr{BOtU"ӝNo}KT[fbk&_KD.$Yc׼$wJOm/tEFQϛ]ԢG&R}ؿQ=im׿6Pi׷*)$\m{Ă>l*f9f̓&O>c쌟}LTAgwuϿwPiljgɲO| ft3-W흉S^+;ry~w\0xF[ŋ/ɿd ;6ߵ0jbʔD5ҽT\LAOJqIq<;\l/|˜Me[tJa vػo_ةXЯAka/7lǦӊˊMlY+tF7e)]53;iO?$둾Խ)9N>y@bL^8ةR#ۺ(y|wa]Ty~l mC& bZβY~ zw8m֯o?vڢ̵-veiL{H Bv=Uįm tr )QTqt>˻LǨUü|_oCWgΟ֬!.AFb[Vy\+F!eW c-(3IY?K̙9?:F{} _:WVHXP9F< mN# V6i;:2eLkberE 4U(H.WF*UVWUVZ0%B0j/F\!*k! k˪  LfX,*"jjbDMIj2 -I*U*"L" Y#Q/@]_$ BSW@$ vwpq;V(SWFO"H'uʐ@!ɕb%~_jZa]D VN'kT6L?f&" .k䎖]L`1#f=xҮd:An_>m*ϞPcѢY(c-87z`":L;M_`х"ܣwc.0ES̏ILѹ%X\;$Wܛ?dLҎ'q =X[4`g'#G*@~X-&zi oaM0&lǑØ~ 13 g\0ƧŸi | 9 mt(O.*Rg0;1"[UPHC iEj;E*|]ڰ,"xDK#O̠CBOo`Ej EAe?cV d9i2S:32}\0}E L4>!)lNTfe pXwM|v3If3nOǦ;z=!A3 Mʀ)9B0DCAX*&3aBP( c:xo!n8(߮3X>Kop~a~ҥݍoF aj>hy#܃z'bjIt"@<-= r2Ly 6elON6GJn;W0'a/YMfTw2&Np4]@{0a7VtBB\ QUhFfff ?.Ư_;w+WG?^m%k׮QF)OųoNnkۧ3|Ѿ@13h@7F{Ą?{|Q5{*uj y~4~7^CֲYGD:0ËOݿu:Gu3]7nk?Nh9vc"f/+s00}:_><.Fp^f___kg:A2Ul_XN%2횴u_{wh.䙿f'=֣mno5.wktaz 酱Iwt(ƽ|Pڇ/WM!jiw5ȿosyit ñSY5 wv0M |G WxaoaU5DZCc`Y~wmj{,פ 5O{tsuO Y-V);D$p>.6[Wubޥ}^JD !C_VVČDhxgKo]ϸ{}[?! p#ft`H)adc}͈_绞 4&qULKpK4#ԁOR\6pKߗhy%v\f7`$(M4"`ivA/nOEYj71z|BjKONvSj_|yٗ.]:t .\͛75B.}ƨ9dEatI`1ƥQ X;Z3C˜z:NR4kF)<)Sy[# `g[U/9N.qIʯNqs&f5z[o9aSߏl6^G{Ž]﷛ECtի@z>yF~̠ʓd=NyՏdY'Q ݗ9Kbc5[cok/oe)ˍZLXU_0?`~_-:ӆoqR>07ż^FY=mj-UϺnׂYWm_aJvuB3s(`՜ʆs۔ Nϐ \kodbS[Bҷ->(ܼlj\=ZZf6)/}mݭ,c8`Ͳz_Xt\w0'&R:[Qp LT9#QoWx&p͎x&vlQ+[(rN:f[hv^vC哓6q7/ /񝑹aV,NZmjˮMO7}+u%I{}`qnni3}ז=^ia@a$izo*foOν_uhz[u˩'cMp&,-4~IXhNgG`3&ޝ둾4_H l5*U>ԧފ$횻12J[a0xE^B^^_ayńhsbm/ lGvjfg2BaΊ@DtCT|uP[Ǒ{Df ~hӧeoz?xGqQ4g .8yd+ >wF,I.~DO Q AQ)( jI3$Pk(K q-i71re"B"RZV{YNJiV՞5&\KhZ"bLE"U~@.*c 9)W,E?ߩ" &'$ M/dR&9" eRE+Su/Қ;9)+("kbL,-P"B7&*B`ENJ}j5%n@$Vk]M+ݢH2Z&ԕR6<@-rS]c]ݱTZ$DmÏ`D'Lb>Xwx"HP4\1Srٴe(wɝSb09  ? K7AOi,6(e\7qh_s 놻/ʹÎA)qv=Ne|ta _v f:; {D( WYq]q~\lKׁNƝlsqg_,"p΄z2z`tm̜4bb^3WF%[&*c.#1XH_2ɋU z;"ȔSrVB66`{u nA0t!1oΔqCDN" @&'Wԍ? rpę0/$n\\>>gh6u21P'yr:g=<0 =7 3ΙlmB#O-^Wx>q>obWf%Z0SD`T:ޚgC&N_(+n$A(,ܘgUMzſ=@%Y(f}@[mp&cݩ9h@m6x~Dߋ`9b3}hcchz@{?8Gajjs;==y%efxzzzzzY& `Ȑ!?۴iSS9SMĖe_/6 `rNKG4K|hzƓ2d%?wjnhzh~w9 :y ^[6ѓz*cʾ޲$3zm]Mmps:vF}o4 5yv\LTK\h4 X vi|MS{̶Pl_6LK|6?-fkb6w6.黖iTԚ( #,4gL2S 粒Qw~j'-U^Zb=|iGˡ[y|uMeUG ?]|4=e5E-Ez]JG:ANnthZok/Wι!o$s&ahtCV﮽_h}pE{%MuIiQynz[fv˪z7եMrp7N?r78flOY̮7XKCN6A$.SWݗFz`Dg'trݷmۦ4^Q:twdv/}T{I3ʘAy]u`yg7vPo68A_n2Y|—`wwIoxX؋=@0L{`\1L t6*K`֫`LWSAo0h@lIy μ#=/ٌ@-҅v yíx}y]zҜy)4ނB2\1v[3w.~Jv5,9'{C:U{FhA@9xRo6M&0<;&r xw^o een^i d<_:#K^Dp1 }Vj3x6G$gdĶ~/1+` N̴WS1rWɽt /v+jWv :|}7,䭍1q ӷ+bSA1;>xs)1ŀv(!dB'jym(rx st}d;fab`>xsTy'baD QGf>lFuiial 3 Py 1>UC;@3Qy6@Z`Մ |6->$/GyԋY~"?/Ұa{g466Vj>Z˺Qiy-N(떬Go /&ɷ.^zom°[n}ǟWkgҵ*u:[ca۸˒%k|8EYrVcϵB ^i#W{e5ծ|Xz˳{l`h&?$vG jrۋUw̻Hޡ.N&KO3^vm33r[|֡Sy(.rTٽn>NhqFtҠM-;'G"/2S<|)(I:&TOݥӢ!)|/Yh*(U&zu"8 L{Ѥ~&旦p,Tzq 6*)=D'@zV>W~v%=ߴ:=N,L:4tii5Hȗ6P$8ni0em'ՙmV1jxYkF} *cOs3ΨZMܪ6-z )WYm{y o(NҨwv`)w":H=çzǿnR?lT3aҤc5t(H>G?d?ߵ(҈e7]^$اq9rf[z_Z*5 {ZnWEB"PSڎבSPE=2kSBPQ%9 LQ$-Q2MI|4jQ$jHRZ9*;bLU*PV+넙dreMR Uk$ jҀУ`(jM2@ì_nMaeY*B L!)9T)V*NUE|FLVAB\_@<##j& ՕSWGv)ף1C ּxDu|)O_n,\\kPX4xㅺg?p *Dݸ3x;F%t 9Q?  r38烱s@_D; sO%oV10Tmbf+v a Kq&&kx1}`iLP,:OTQqB wp̳A0Z`=l8"7Ђi Ї8%t)6F{>ڱ4$3( nv}I bOJ+[ؒj4sXATH K0iƶyOI\HGYݧ921qa1i?f=Z:>̥V9nB$ 4Ѐ(N49D*unL?Wਟ#T?!5B2WlWoҟ˜\YhLs-@X fXS1aVt_9LզV14dqi/ᅞ r`shX3nb:܁ ʸp;]i]]`M[I_cW0[%a2xuS+`RavX8:Κ{ligE lg w9`Ҟcq-8dk;onp=TJU֡R"iqz/ƫdcjup'}A7LYH!GhcYW~#;pZ_FpY<.e[mxiХڭm[ ]i8L'ܹsu˵kIÇWtٳgDՆ5'N8qbWEӨ5ՙ)͜9PhY cwyRzdɩt١YhyU6"_ PͣIc:-v()&>MŸPCR7[8{kو<+=޳oZ$P7k}1*A+#˫MoLt=Gy^{%hrd^}mz2 Ej̙UۥoɭRK~-uKXo kk/L=P*>x3{{%ʾʷ;e9/s'UuNvEGf2l NJ ƺ(tWe{ͭf|/`&jߜV>0q蹥_"/i!L`a|ٽE=\uMpCEK>t[ooM lni2;oq|q'N;Ǖ?y0 JmEO}5jcv=6 m<r6GJ˵˴3 l <~&nsEU]iqTԸwLt%پxbǠזXgYhml. ϴG87'fL3b~TSjݰu輇Fɢ7%,70je䞫$v:[]5J֟nSf&ruo|غDf\ϛ_ho&hVp3LUBZ~vcnK}٨' #8PeES`ZVBӥ }_;/7|HpMyU],JoW|>ϡfs߅vZMBhڹ5J0\LҪW-h`Qf_;ōΏ!zBGiM\ ~-esa6*߯7ixk,]fl|흡~IiUQ>j!!by4v{;K5\fml#ۥi~0`h/wZҤVۋ M-_[kj#gO?עHP TE9Ab#@X.F*+Qrb#A(DbBK!UP~UX4^)Q"תKWhJU2I @HTGLR*S.W*/u"Ƈ2R+6(RQU ]B2^ê$P(RPO5JjdT*(hRt AmRQGNi_^RWV^mIXy/B&%WԥZ9]uX*K~BB`UL#_d*IP^+-J,Zΐ]TJ)>n]/jCӠ'B{B<\D{vg@;f23ϽAtqN o-`pwF7 ޾}>Lo(si"jCo2&q3p 3K>/Od>Jo,gܚR >jSKDSI`f' }F|5JG&_ !d_5D4L{/y *i|?30!0ԘJ؃Щ`z wN?UhjbDAA;a<Ĝ#ڥx+@!1irO11`aW]7@84k/b'Й/twiAV=O [=0W 35yd9D"rӕƔ1ǁgJy'k`657bI=4wB3o^>&"9` :cZYUAm" $uF^Nsl.@vh Q7i/֞P B ,3(iz*\A[o"vS.&z Rt-Psޝ)'Z/ )'x8:7x50CabNBdhLHK1  lVk(g$ukK|ܼV!2&Q av^ A<",uÊqn7T?0["fӜf|Š7@ .¯ߣ˹L M=11!P"0]& ml8ÌQN=bF! c>3Å o^1X,Hu fDT3ms wx+vˁ@0j7GW 0GCX̩<M\!d8ن^ȐF2sw=k48KM;*)Au(cޙIaKA&v_$i=B <_`͞$Cf2)R-ڣGͭ W"""~̜9_5kԔ]jFTTT}M޽{=l6MQPU%;xx;*uCҲojvϳZrq)Xf b77Oq?${a -q\Fq;J\f,䀎1.[ilW\q>oLJavgyk1~N 7~3*zܥpJilrU熕ōUwWh}/3,V&z^n1}˦UBJ[4N䋞i&4|d9(s׾㗪O}%>xyGۆF*vW)mDĩ˗.;;~5#n_vҞSfM?,Z~Hwd Sy:6_qe//ty`o}hM9{aC|{wN,0Q$"&c7^>5Nwst҆K}+o2 .rbq >hit=^:yWU8G-ݵ?~?h׋8 E%Ï4??8Kw~0t i1Y"ٶ%g-mEK88^]xrvZg-w"RC׈iiѦM<_fDFl<芛:^ g 7Vn,u 5I;Uy.,ğ~ȚX1pׅ}/q۲/{auk_[VZVTAϵ OoG!@*`eTCaP(I3,)#$*ʨJ*PTj2\(`%_״X!˕Pҟh+u%Jh+4EJHM,PHJB&RBV+@OUyPϭJ*ͬ?ɬʬT&J3GDTHJ\DB$g ZJuj!PE"*j_\@")&i*iR 4Jhj_M az ԄҚ[5bRVa+J61 /n-Kaa(n(_Iſx?('1lnq8 Q}Ր `b9&5F*@bCOA`m<4тb x:1*yc,n _+tAP'b`"QLhe:J4yϥ}; Bo8OfD W{0>A3 b`}nق6fsy:(p*+@u^NŌV-\8 Hטzb9B/[^@98ʗ!.fXe\ tia\Gs5~ ]X4&4c2oow/Mj8J8Jt amtdnٮF<=38eơ jtA*\Pd@D1&/ 3l:Pn8.>94>`[ 7{븪+hQi,[n[lLTlTDR$Q)X1?C~{5ךsߗ1k^n+,o X9#zW |ePq M5<ħCGoϹ4*s@T>v#Em" 295[!^/L 6Wk}Kݓ潁Ų}OJt^67Y_&%Th]:;hp;Dy*k: _Oo[(|Z٤ف<`~ְIԗ_?`\Hώ:79LjmkJ ]w}Wݢ#g13MEt7ug%*x#G{|vGo;W=7Y=,wUuV%4R#M{*/ۛOSHvpKLI[e9Q[ {~׶qTՀRw[ܜsc鋁9nIݾ [Y.vSD_Op*Qf*c|L$y-;?ESJ]e S5ϻ,EA!#M|qвl'_C-F-[CleݜȻ̥ú/wO(~YSҾ&[_[7UMz;gՒw5w+Ko'v5(]ttO@{dGzo0cwºNbx3u9&_#iDU @]= X$UVWjʕKRMk@^B(%@]*֨W/m2uQeRHD$WrlP2QLXdvELTX\*~׫reڵGvzffEWjvJN^ڰ*4kUFffeʭV%U{d,WRH!W2jܚkeA\\X$A@PɢzBrq5dYe RzU!UhHT-5@HQ/?*Y7]64Yfv$=$OXM` TWTtAG3e^@kئ^(Lt{B_/@D(1'P;9規"7 w\I^AAv. ֌3t0bh!Z#{N]rzK/nhN ~kx>ùx F92 43^ͤX{w x@3WйP> +x{Ű{0h1-GgL"_ݻĻ(aE}tB 2Ps럣ۊӌqGYpM=n6GHV`r5X >GBLla L =`+'>L$[pdۋEzV@0(fD^礄݁a1R528@x ކ{ڲ5`hvDмc|% Td)H s@?nɑHJ 4g)Zqqm4}6gP3F %)O[ 9(v'7`8QQݎ fͼi8yQ^XAfd"slIumh Yă-M/ܤQ?}[iPs1krI7;^}ٌy׏-z_ ?lvP!j'@/-h=)ֻUٯ㸒6& TstJzV>H } /{{^hݫ<`cvSZ:=W5ç}ZFgip#o= Q4Ng^_3:ʹ;G UbR&ybcQ28vߙNn[x@QkE.GzK=7G\w:g%O3?cAe+ _+%jx:1FL_;ԁ/mֳK׍98y0=D);;KzR[q<h gdo[>zȱ |S&KeGT|ϴ-Ҕ tޫQ zUv$9naYi5v?T>Nѥc/R|+ݬsᲊnQxOXcߣl~eN׍O㲹Z6t/ƵiAM9n6!}4`~˯}Pmjv_׎6}cC(`Q(QT *֏X(7:6VKDҺQY ,m$FIRIuj?|LjɩZ\0IUJ\UKYB]!5-72A3>u$Ȅ iA\5B(Vm`ʥ*y} 2?.J]5Q(R50 PVk^LL22nkZ&ə0WP+.Vs]RCT,TA}c@`a#Y]\祱0&/k77΁gk{?Z=3c_\uI2]%n5Dh ux/}>xg2[vALhNB2O:` 7m7@ ZD^y'G_*+h2Qa/g)NTޙG#tF V5c xFsK.ﰋ;SKjܰ2 U=;YoKVJkjlZBC>=b%K$akΛۆ9H\ m`12Xg:Kty xb_\J#W>f?ܠs<@3^g !;3Nd]KgfY2џ1y<7qC/L'^?KWqs=%2ۋ%qϋf'f8gv끝%b2|n={ӣ1h3'r~80iA@c:B)@`){uL%@Hdt@h1:фM@5q_*,'~tkJ2Tp"]iXv`&oA3~tҚ4oڴef34>[i#nÁHoGei̴Ito65CTve2/eײjNmjKjn'ݣ-OWOVN<кXp>~ x.ֳ~W³hƷp{&o;~PXlz̋/4;׌n֭P?wV;.|/sv9;,OS9JPwюBWx,[,y\r&=dOʗc|UM-G~Tx^T$Ll?hmzG`+FnuQ=x:loI>YNROtf䲽Wz :WuygWzYͽ$JڣX* ¶Yq蕇յsN.m\J-UODQ^-;v{¹eW6VtgD:Q\G}Ω#nQS&jF?52oX}YE>c<Tn]+կ7ۥxҤݜt 7L0=jgaXPUіQ 7*~e;Kz=uIִvmJo: r{"0ɞM³373= VOŇHI7 Ww|(](\=y'2I{庖U> ?l \;}MeEQ]gqoVb_|nzԾߛ)V9~)P3.ÿ,cb lf^ ؽE+-ms]Iqk0q_'u _,v8h;A-伧DԯM)zw7LXup979 4#G^\~! c*R1W$Qjf.!Z^-VD"ɄvU mcR(SӨQW?6 P'aH BqUШhR"M(Q4S-Jf`KfH$ڗhf"iC7eT(chGJ ժH J\*WriXI\ІYM\&SW+KVUJDk5jiChQ%K~bYREq rA@P\+14CF2X+4pipkm[-B3! 0g* ™m gyAM1wwڙ"k{32P$m4 aGV8x5w\q+& ^hfis0Y3"h '3e4`0Ѷv Bq! 4it\fz '8OW3}v*9\V;Og$P&!>lȕgfnN{x遠nۙ=@EmiAa[syѫa6bm όk@ 0𸦿/$IzO*'(3_OccA> <:v'/y?ܐh3Ћ-@3M5@ . N `on& 9Ĵg2c1@ PQ1༏C`D퀡?Tx<&a*^C/o5q+oPaMJF4t{XLc̩2G0m,;_۽<x9DgVc.qwu|oʙfV2H z]{%XoUhљ7{o0u 'Jnh͸ 8'4\1U cP,6faHdAm5A`ƂGXL#@m9u{c uʹP@XYm!eQ +gx&@̠m'c) 3V $ |Fbj =:ln@.9V-jAw{14F%fq..Yyf啊]^46LaZ; +g ㏥J%j?.iqPmfBuʽ-<=4ln>0ynow]uGa;F1mB Nｿ߻Z\jKRei^Azl.y`DҒ𠡂o{Y"i#N&V=Ev}=[MTti?Wy!WD? 3G7jUS4(BA0]Sjśksk,C&珋>{2oC[FN /Ymbo7k"(m̰VB^BFS>*sf>ꆿ*~Wk0%xȟN]_(d7E; 6bEѾOvc'-5~wZt9ë~qk,zu!S].Fo|>g B+|yc\j$67D:8ξ>;C9?[Uzqc]k{%וcoz4wjG*o?mն|[*)z^/]mzovjoll)IDX*)V*_EHs! koWUJ!KQF`VVh*T^k(RlHa*$Q #} $HX0$v$UI"YA3hR\-i%.P%aP(!6@uT/U*u40YJLd_eU*I}4SJ@z \XT\58TbX>) 7Kƕ% Ufaa-4S A}${B횑+$jShFNuZi?~ܫ?^mZ6x&`_NTD <.3;d,;?]8ػ9؍*oNr#p,[hf ]ܧ``dm`a₴#` <L@go|ӫ2Esl 0 z-!ݤ xv@[:6(?N>܆|S0j.'cՓK 2+Q͛E"*8a4fi!LDv1P:0ϙf"b7(w$LQ(c`= Tqo8nfuDx6Ȧ+h:NUjX Zsn %&yxLĝ+zx nTяܺ=coq6e^ i%,n2"B PNxf9|&r Bΰ =m x7zOD`#BA`)_-)6D|vMs&C{IBЪOiYt;+^ |Ea@Kq(;: 1Z%0eIѯ{ =]YDΌ0@;H 5]|cyԢ_DB'x5 L6|#=9UgUGMOSG 3Cwĩc/ z5E =먛hY| hc~?Ͼ[dj'ƚ[Q}T%4ъ|ء혃ӗ\[ͦ4=d u Fqo]'V* Y>Z^B2$ϊUG R;;LR~q ]Mfyw!BcӼٙ'yqnFn˞]Ήf6N 8ino^*!][V=0to򦲰z3&a*>]nlي@[paDXY|2y]o4 bʽT?ߓ3vI b'V}AqQƳZ;.z/@wv39#)QK^Unj "VjCogQ:pn:~{o)KM}de9A#vuxw Q뙫-Dhs_˽"6bk5 rHjLLAwz' sN>3vyf'ݒ:[lfntv1ٖZ~n+{젒 Ϳ}/zr˨OFQ-4)V*cM((_b27zBI$L 53R])r!TT+.Z ru%&(W*V_ZKeRc9B2+w۫Z@MYb"faay#t\ZTW^MDZU*Vj4"$3m1SCUY%H?r\Fx\ ƀ@ARQXZofURVגlOjlZ5fn,'lf̿y@?)  (`41ZfFn6?E&{wis~TkA<t%pu2~k_YCn3m[]~Iggw.u|MI]Wi1{fd⋃qC;@'1PM@ilf0?]hV,As~ۇ-+pcĿlo1}>?~=S5c̵өf_|Q<Yv&wGRowKhL=kTQA`:sC> csc6L:~/5$~5&Z CcsJ̯feSgm{U?MmIw8bGσADuHh|O1i{s S>_iy4c &*gv }=4r^ Iجl1M!&8a@Ϝ6~zmnߘ%:jѱ^ !7OTQi͢v]X:Ε 1?ҴۛDĝ?Up2{Qr~ꘂ|[u駢n-q"KEߝҤnoT.}!SJ~yn[aNgVĆD}Z}>Ymx\}鉟F^7g4~s8AͫKp~QW,]VbY$AN6:t t=X&FtO#u?:\w6:-L4Û`0f6^&q/)͈i^4~y5}+57;VXEaכz://X>Տpy ؏+MNuVahΤ2ifkTMK[ ×=C56y8q*?0B+c} *xlT8 ]Ï6^d5|gQ= mld(7*JdDBb>JI} Z(mQ^פC&%2TV%~׉@Y&TD2~BUCl$$ym^ QΔJŊ]3\ا> BRPԖTN= U f$B@S5b ΪPM,jԿ*ʕ BBVN*4Dc$oJ$R\[M߷ȹƅ,AVGEFEfqRP*S |qTaY` S#1r1P.h̤zh̿} 4]A`N1e 5 Iu Q5Ohh6PRyokxK7<4g5nDS#&@O!4uDĄ77>4="B@lYo cn N@ !|FM^5]c'?vǺv|4atT~Z8SP@2"\v'$@{8l~)֕a4Mb֛G>q.px ~F.a OD Apv0@MQw?o&Bs $v\tY<"03h8.#;QУN㐓pxŭ:, & Z'_vM&k={Vk]EVLh<|zTyqj&;wAXj'/hEY'K*on4S o7993c˞^~eϏ.H;pgXHy|FK3%SZj1ٙ_p9@]~(z`ܛ\y0cBܚ %%"nV}qV\Mss: [G2t՝XĠQV֖[\*|-;?HcaZFхK[|TYdޱ/K|m+ٻo[)4~W[GD]Bdn_;njܦqk-M^]6WhSzZ&tVY,|;xC7,9y86oGت_.~-OIky03m}k+GO >56S+7똼}CzEk. >4:Y9s%-/FX"otU:r7-4kVΤQHZm93.{LJuw4az{pәAJyӍ hg֫-ĩ Q;(sQbĖiv#3 ݿ>Xf²iY\/l̷ k) A˨ꂒ&#$ (U :VID&)UBY˕$weBq-a#Aj%#,$2ACTY)߼#ֹDFRj}[](hlʬR\R(u14(#^3evsmAJ2j} \pvL$7kjυɘLA32y} \\H Q#ncˉ =U*庠Һ_UQU}gT[iʉJP0) rE3,XƍtZcf7-m*O@3(Mup܎p=:p'b\b  '21q2lC7vO-Tt'S>#h5V AˇFֆ@QuUc?D+qkq}QhoWhIGbhU@?g5ҧysW6fn D(=e#n&#?|Q| 49u@;17L|<A(w-Fg~(`|ێŮ^BA p|m!g%A@78/CROqQMD u0 x[gѮXw~`;K;4}Ԣ+ w`WV!)q:bi} -ۉgF~jH0>y|P~ XjfXvzv`#F^MgA*n/sinnj ~>kZq K>`G'}i7xuv@;'Zsi*A 1Mw*p17~`0KA|D@g¯*?l0߱Hli bltcصA8 p]gQ0瞉b̎Ԙ& Ӝ |îOÌF <}olA8}.dN@荓-\jKNyve=WL@07|"!iψ,FʿmӞx%p*~2)39F:f{]dդLG*1 dUWZ&0u#b~o{W-MO,g9M?ICb7U;$]2R]n6--zchhYtWޓ*'Yt/7Tȴ,E(}ꯢ{ueNyEs =4vZTe<H様d>: ocǤzr(yg{Q)CeQ>.5;1>y,rFfN7ΆryGHG573R11R+*kQ)e%iX˰*uۥ]`JsnC-DoUbf%ﻳc/&VV课4i =[xƘ!V).-11DO'jm3/_~+Ό $I($'qE2Y]듚ڝ5Ro( rj0l"T$CA3ֱɤJAPA*6J.'԰ył:LU HrL@\$u"( j!PQTo ` *X5G#k 4^5ǵ/\JܠS85uR\B1XPT)Tշ,Kj15.I#4ss O1Sa kLDR IH=R#X^->yRBYe{3ХfufОun_Yt3<x ՝N6@BҚWen2Cu.a0PUP*9a+^UD $gpKKAANɘQ̇ѼW|w%,8" 5k$v G6]niW3DOzMZUd@`>Mθ|I 9e\w/u| "x s`%_Dvi~O5Axh3Emgt'[vp^ s38|tJbB=]xE~D`V҂=Z0|uJ)m/1QXp?_6qs,k2dXp2+y 1!bvGD=L=7 qPOaO7 ?ͳDq;h}Gux׶0-W _|dѥVy޴K9iw`x3GzoeSæDžV]]6nQkP`;Y)/OK2O[3lQc5|.t8ݖEz[#{^E ]a{J2񛶡g!=tV-%N%wH.I\ظgyw2"_ԻnqO URUdCfB񰴻J>"ݦgD [U.K\ګƊ^/Zؔ7Oy;>Q~UL,ͱTD79pH'95r%1T׬޾Tdzї&YNt^3겻~׃QZF(١\EIϺN^T{O\uF7m FZEd7&ruyR$tڪH$ӟ狛-뜷8v׎=tŅelb4mU,,|͙qsV&PLҩѾgں$΁w:´9"=Kۿo6Z\I18hT6*-7Ϯ{݉aN]4UoaϤ7$:R3Iǽ%CixrlQ`L],:`;4 NVW%=3s;o*9h@VV[wLR+m/Q-.4}۬7U߹wsAW 2R 3rae_m1?SUO8.z'.`ryߍ.~JjH%JJbDZU%f)KA%\0Û? b3 Fc ;UO UHZ \H׆rr4$(՚s#kkYcE*eu|pqJnD"AB XZ>>L\PRRi`W%4TUToLN 6FLNX^3 (+ցP۵.T*nf**y|oI 4`RYsa\E3B3;- &pe |\|Q8/ O1KГ-\p;f>Leq'[0ևidk61S bAے)0x5Ci%0B2n o1"bOʄ f .0  BP"MSZtN=@33 @LЛ5s= 8@jh|h{s=/a`K$R5K4T$XPˆ.qY=wdZ08^(&n י }Ii/PjN.7`3F۟kk9r3YJހxtMڂL\j ФC 3vsI2ʀvq-+Afly^&rdKx|΀/j8w  m}? \cܦ)  o&;JXlbסdE(f[вG\L@QrQL7F,Nu2b>s0YO$miCZI4;BL/6xv'^z1 w6u,{L&8wm`P~N1fcۍbYTtӐÿ@1e/+A >#pd_*4݊hQ>Sl%z_"DcUʸL͘|4]7pQהm]PheӜV?nfv&MsKܻ<(32./\r}E<^qSmTUӴ_~|0δ?{to^! -DH:ݫOZ;WIZՒ1.N *^ܷT1$:W\ufcMpOk9nX$*Cwl+tSK!z ݦ Xs˱)=#M MWɢGh.4uǟϼE F -]3ۿEmEUFV~VSEz?=ݼ˥c.k3U¶JzfT5]|v] R= cNp 7sX{٘.WJnζh՝U $zV&lp#(^QSJxt_q_0Is@IVE$ݫ.>zQt6!7Z9`E?蛧gq׳s 8:ˎ-qugC{鬺QzyFyʑFe{9zxBU[_-}s)5hK5s!o$+"J*ZWW+s35aR!"TXNLFhF*kZ"Z@.4$LoX.RX NuqJ q꽨T>KtW@=lO&P]H 럸XX4zSUKs"b'J n5z@0`2ȁE7Փ#qN;n[Q#a(Z)0N2M3n'fsgt4 <Ȣ/?w_ء2z OcU-)l%|&\$~⋆rc~N"2[Wp?~.Jx) la |j1;bźx ؑn49SI̸C4k羧>:WPm' A`)9f(*V8gUT&,ۍդ)ı< ~dq|XZEG/`ߤŬ?`=da-[y |@*7?MAóL) xdހa^ξy#pSA <4:3Fr@8_&ԡ_z&9XH8t[dgW!R]vίF`\N^ɬ4LԖLQ?rcɉ)~k囩pUoJ~n( XSXε"C:QXD6xa)Qbysu>exҾ ]M ZJ|iIƴC8 v/ &׬yDN^]dF)[)*v4!Iaz=B7wpv/l|b]A9?]AzϨJ9(ܑɗ0 y*v*%a/s,;6@$##<'cI`{ hmZIxU(^V}.T/c2- '߽4m3c-쩯3 wصv;NX&PY)ӖI5e*_igc7ܴ$~&1^uY!i/y0l}~{43][?vNTlvd@%V?捽收M VVWbHfWnc5^S]7 :c'm΄]ۣeڱxp՟OU-C`q좱mRHz'^ :o^6\7.nT&&ZKoF|xǪuƐJW}˵ N5RAccV떛3\{×anZEU:J)g)4rӨYAQ-\hv^_a1F鸤>Hs~08oh9ɉ;?wʦ[fTƕ~nJZ=R݈ BGc -;E2ɱ~U(V@YHc$[d{=&^7On`eTu:Uh͝*%3t$ :w?|G .H&:WK0Ȥ$o#ojZP&BP}˛90+0 a$bP>Fd!gԶ5}Ay '2e܈oD\VϢuQN l}q,)D*)X,Qlf deBZI(4S%ILYbDLߢ,f U#%jʢj%q3Y:Ӳ*2IR**1Z@/\ Fv$/׵/4/}QYq8-@$͘c2DmWBJ>f?[!y͞S,H^Eyy#ݭVb˖yiT|4\~.v1~cEszH+M7n#[) 8{ ܁ɸq^BؑNYȆcc{_Otm4dR ;Q^D'Y8b8 G`0wZ@7sfw>f`roX}р.l.ǜ #mZD/3 V_?K3ݕ#$~[SpqvWm2gN婻.+|xz.deIէdy_}-g-L9~3*F O5SVW_:| Pn2ڹGj5W{CmQ3 =d>KcXgӍ8RpSr ZoÀkUŏ29Dau"`7IAux;ϘfS~Y~~ zz-0tU15Cڨ[kuA]su꫷274xZT˶E#ƞ [:Vru}\,͵.R+v֑Ԡ]l5szm{ULM{jmc7U,C?H+Lnz)9b8M|Ž* *v;G)mʮ>o*=i;:*eu̔C.Ot~k =P Z ܇IITٿ}e'<%M`7˫V5cf9nE}*>0]M tPnӾIYmf8:zk,|zts%*ydd>YZyJN&*On53<)VhexV~tX Q3z kW)SlYyf`>)Dh*DU ,c=T'H a[~4m2U⪆;Jd$b _!Lo((7i(RPIH)_WHE L~jI4#nfh0pc+L2a+&K3U-F3J߭14S)UҪ˫%.kA"{]$P^K j.Y3W40`,VԶ&AÜDUb}@ mD6T#VߍfV8t8-Xa@3E1ne◗0 <>Ś HlU1-c7$b5Cn+֠! q:qu6b T1L#=.E /i=U˃[5ݦ( 7 &\qbR9&kpeJz$&ƽceS0H43eyN3އb؜8IRyC]W4h'Dk6y~8y$Fc ^ ն-A-0F}ц$ ,!B>*-Y=lTA=$nkR,JU2M ǢƞA9+'-I@ID-)]$c뵭48e`\9L_=XocA5‘:3QHLZesUp}Jz_Lէ# 07 c,+n UX`\KDu Є3~V=ZfV4+/ h2\g,D2<>a|ůӣJ0/E򲜯.h 7t;"ۘDXcƻ!2&gU a6$P0*6Z:Ӿ 32q\fNy7 `N^拱vXHX >Gj.p{ ¤D. y}wwrZE#"GĠElSrX8[dD30ʼn!a9"w<@P=im'1_`~FxA?K-:)8幮q#M19N>v-/}ˍ̟^\*l8'~Ͽ/nU!km$0K3.Ui5aBUȼ~;W˴}-_i.MD]s+[N HVLIO8Z1C8Tn^a>k-&WͬX+=0syKϺjӲݲfӗk&uQ8SA+4vVv;7!v!EmW8շuR*yanb #M/Or;_*тy*H"zkui!3#6VMuǏI{o5&IWaqs@W Jzgg|]]U}i%6='=:Ӕꝧ{hm.|Itri^֓M9n>4/AKP٣ Ϟ;xۃ=J,t~Nv_͢ '?ѓ%jv)0s5Ήe^&ٴbmfu|݊{5IklYF Q㛬y/oy4[G?9~:W.MIr@Ζ{c Ӿɯ uNֹezGN'T84IzCf^Uܔ͝KTyU|BǕ/<֏zϳJѻĕnU}6j%-n3U߉BQqэ9i?n}[9tk&U7A~8,Anc5dnPy|طV/ztsl}jcԥ)N9!gS↹LՋ+O 2m>ȝ _bˡ3C|m?qv۴ɹՂ%_g8\ĵ3C+:(Ee׌F\}~Pҵ3lBˮ9^ÊÊl57zSL֧ <8ϻ7M@(lDAb"DG FHݽS !&j`q.d)hzot$BzLsGZ_@3Qh6*$[NO $+@)Hx;ևڣyʘHz.(!W7Ln v@mh&TL~K:M 'yz\\ ExN$|EP 캨) 0 : x(t]\.L˜Jm('LG-nYto(:m5ؔ\"(3F/bqﺀV=mEKƀG%p0UӚM6 S: ]aP-+pGnBQf-? |8$TQpqb i}:6[{}&uˢ}ŭ4 %2}u1}WO?¸&?rird^rnȬ_>uow6HulݾtOzqKC]۶b8ݶ~bVhgrnr~ο/4gZ綘]zp@m/i4|<*JЦ:_i1TSᗡ v0OGE``H[=AG.6"5kmf0*( зw?L7+˜m5eˋ# y|}-S^~%Y_4W;7a{ vWocZݪW< j*[f#M/jEQǴUҊm)}t'Xy JJx5mwx" fZǘĞݫ隩ѭv_XedЫ帣$|,Oq[ _$goN-A-S;\1Vo5`oׄ>3좻{(_(,e0i1=_|o8G_d2H.'K&LNR?J&#~ 7R\@L"ybIRKDK,(@VӹQ2W%JPTWMߋk+jpF K@r1IAΌzE$e JN H~n@H^ ߿M"y_prJVFK W)&3"׿5% z% r7z^x%gHQd?x*w+sWWz zmkiw5A2 ;@/C5pG7(rZċ1 (aj+ O`&# cMG t1u3pȍp G0. `9^Hn.X Tya2ǤDl-yb0@>v#G/.t0>1 x&hv1[7S&Kh ;NC;ⓧG `z|m r@@8 z^̣\'H݅Aaw1R+4|b&|XOW.<7"XM(A.*vzt{to 4.aQ_ `Cok"q';1]w b]I}4Dހ mx'`Вۢ0Z1񔽈Х?񃺀Go̾2x7CÛ:R ,mn0Q:!c_6:3E@ϠU`ӕ;0&%=/) @ ܦ;*W7l/zR kL d8<`UE$/EDȼQB?~1\./EqKLcИ B@pr9ݖOQPmƷ`h)0$" ʷا3b HO=H/Sx1`q 3 3c Xsivk1͘sU:QY>eOO\[ Onvg?WqNF-dDu;2qM?buS7A+O-=ge>Ss]<w64}fdm'87HnM+[Xwӟ>]ť:W1j]~Gu*K=ifCf"<5^+}R40Lfk١~ĢM-ZZ3G2xkTG+FRaX_4k{-T2}2n99BEmhUEs%(IFb9眗9/s˜`(* CQPwy߹w_9sjj3Y?h=So[G}Yi#1,g;obp v 5JNO*zv.^nS;B6y|;Ҩ4SO? rjiٮeTDZ><[]{1r^oSUo{@\+?$rlOSW0DM>~Ml9C7#m`}!Qj_-Ek: o+yi Uwt!XWTpwOpj]{5Z tnY{aTvyQ/GM0u\>tqfnHØg-خPh5VZ?u_YlQٸ=O=y+{'K~zY nc5n'[ggcQaVJ4@Vƽz^?a{9|m'҄>Yt{Eygt:ud8{MN!:vw6s(v3ؤwc|b:ym),o"ٛw@02k_!@Ș [lX^N7C3ȿo!6ju<Ĵf0mj?+NXdRv=k[W83~;{*MiaZ\j_t-NT MS{Hڡ瀼D:~{]v>Ͳ+9۽ M^}N9%?IL͒kn7WnnUX5ڏ{RYDw)nl҇_w{GZB*G:7M (VoЪO/hGoιo{8sjF6 7B.B\?eLL+hzIU RE]Q jUR\/UZغ) $Y]%+e"T$W XJ 24< jkH*5H!XyOB R( Ŋ&@/DdQCTW d RE$WH(  @]4B$E d,l T0 ACxC0!4 S]5_)T1 j F.)_5_>n?35{I e_ACx~4v9h Jn;숾WRb& BUo4%q[e |ߟj3 x.Q xlKHª[o0Zݠؖ~~`ΏCF0c ']9oNŠԯ~AV |? @\>PxdoPBrEY4ƙ/Qَ*M$iL~8̦gd!4}O@yl?i?Tc@ks'$M|O\@azr|*f,I8  rV73KX,Ͽ:[ $^8W8=g2\9E[&j|.]JD:6h쬕l/FG2^;_vB\nߺ~_a"˴4QG鄽]hͻiwD{Z>ZyYi?N va?sڿ}fhd]Sk5{py%hlѳBQ^lQz4zSs ّE_҅E|DZfQ*o\FOQ쩚p,#wMl>j ,- ]y8x@(ah;[@{,o@+a;fVIH- -);Q2v11Nr,YqC\\-'X;ߺ.z>>.]U2=tJUvzwZǟBvSy!oǪ?LWzyބ%t~BS ̩QUqM-vAvx٪=#_O{y*6njӍ#_ $v>pFN]P[d5 ߌ ^e6'MU5o%=F<ъ1ٷ6g[KwJgiEKc!~0[8*^#}[htv!7'η5#=Yw}c/;5M#WZkq'U6 )=7w|䦷Qܞ*&_Ǿ+(<,[x~xo:| k;d̕]5;@`ݖ6Ř!t'8OO`<k/XxQOfSϦ *@c>x>yLFx/ [0zfET߮5I[5IZ.o8ĘD9)qS.:.̗]тMZCP]:D` İ@Ymwc\1E#b3aE< 1Ь֓t`Z7t @ a  L8c qF*`@P 0*{p51N|8褅r?v~YZ5“4lRU.vf?فt_S3 6z[sĜqDz'o4/8X(@& XوS '0t{ =e.;uv iQ|P%炿k9g&P!pb)Ou BY0v")"oD5ʨU@/C &`-65/9|j $j_&`Ac\P>^_fTxK˾&d.b-{!(u'fO^`y}tФ$M ȁ=Jns9D#-J%:Tia^@-˙<81>- wmz~Z;K|w74A.L,?9οo$xi1J\U˱W HkKZ?3vhN$wzѦ0xnؠ=V^8VYfn}#K*F oe?e^*ey~ƅtU߳7**N}r/֮sxRtAȟQ +-D]vϾ szX4>aw)} Tۼ٣I/ۏ]v /2Ȣc꠲%~vn>)$ܮCǬۘŜ^;ԀRm)Rvv9& . AdךzgdЙk:(yv~&v[SfK6jԾև[̨YB*G5fHFLlif;㸪Ci357oe_Je-|K.4qi,ÍFu>KB/zXrǕ_~`/7ݯ!Fj vRk}x_1˨qa4ˤ_ޏ,,e-vܺ7f)'=TA. %'۠>e[L/:,Jg&˲R5zZGύW6mb E㌔#:\6)nFv|`ew:9pJ/KP Aٶӯ5.Utymsܰ_ioQe'!~M{V^ °jw.˴f[h9{ǥUmo9X^ L-M^Z t.q6V: : 9ofg+.>!/i\zk-]4lR9g7Bt^6 3Ua..\5Z=/~U9QV/}"6Rcr.,8p!?7B:W}ZRu-Wj';6B+&BNn?jQ4\S'sfѳ#5z74_Jq`PimI^#TJFERUiՑ)~Z- PIJjڨ̐ʄry=h쮉~! 6 oUXMJ8+\I FMF]CP(@ nI 7BR|3[6<ra,!BQP( k0aߘ!cQ]G_ns p.Bd%UJ`\$@ɏR*iQnUMDKil%ryT!ׁ($_'K$WB ԳĪ~]*5J蛀e2U/T+/ @V-EJ̨(5*tM%K J>$S}L'j2h7Hi\ OO{ 0S_ ''1q1e +澠͙ 8v 嫀=S*ڊ.*78c< +&n6DȄA#ls˚p w<_Ƕo;+ĺ>WyALAC:> < 9qdUNo"a@@|%6~~7o ;\QNk}A ~LU^HUpLt852`6闶2B/fz{p8 Տ ]6bBrT=D"!Uf4|_[sI%]ށn}nn5xwυ="7;倯GᡱL%^0{HlJ3Nԗ^}2ӫnH>"|%s}c, ͊l=aʦ #^P][u*$YNN.Ҋl]m!YmdNZѴ˦~Ύ7o~gōnm={x٦wƚzw@ӳ(ujC<9iC3'ӄLP}7M35}SL?9ũ۟j/>;^w8Q4h5Bg }`ДO9̂lA+h{O p*p~:z7c>nZAMUjPh'L6~G~uPtU7 ,vc~u]YVR&}0>g\1#.u`5^?vxiG?ٔbua3:ڷ^psg VS3>{Hi-N9[^<wz~ g.g/C:%ÿP;jrܸ qWqsȒ,ؐYу5o(<]<\X9Vw_HHWq& JUMƄ<+(2zSu=!M#_lmv,keѲSdWf{?HU}٩ưZ)zU7m3>X Mp}T;m$)^ߌq{a0I𞛽0]%Aco|~?ٍ|#B343͜aUMú.~1i\mh?E=×t=4oX[obj`-ڔ,< gm^<۩jwiEm[M7ſWRǿx쮅fJ5B_P P[ eujFe2)_j$"ÎBA\#`T DT!B(Rb=In5ŒmQ1jk $¿NTfOT y%E2m-iƢ SqCBLrR%Jp\SY$w/Jq!tʆYbfr_aQfU6)2ԇf`T!+k{ eTiT2˔lΙQZ^TRVR`!ʤu+JJSSvBVB5J=C33G`eA7֌g8U߾y _y5NJ%:G~Lz|ߌ sx™'<&YvaM31&Zƈ6ޕ#y~b)NJ*tsb(  AߖNǪn`YbY81#H?BKn8b#ڛǎSt"ֺ f_8\ \)jDՈ 6lpuk99Enˆ@&B>>(aJ }]_%A~qWEjlᚉT6-ꩨqp9'Ygb[װ,3h5{N |j㫱V shSsS&21s$nV8<+o&g",@-zKy[l*y/`7yypip 'A}9 :0BN]8 q 4DT7U*}ۃҠ ` 3^ȱY>Oql8MHFtA'U1tt˛N95dd 6&}{8<9ӿhJЏr]UA4Z:`5#3lȘ] 8<tT{1~\$# *t3b7P,;px fm τ*' ׀/p&LյL1ڪ>?xr;Av /\:_,t~xe`oX4E9Rᷘ}3\E04߸ # n]~}kGD9 ?)P!ˆŃSboztrgll?<'6~G>|YbSX#mȼlf˄7QG v? T4n7T#šoRq6-y($Bp1l̊yCʙ!t5qypR,01۲<0hfXg#=}#c`XQ .h%q,äoFx^:C}yWI] IZht.Q`%sAEG{Q{ƨ,٩7'hcsvy27ݓV间H\Iؤ.t6'.z%M]$iCwӦ5ab:l?bC[F>;U<㌭^}sٷ7bЉZediuO돯NC4uVNjd>kGU˯\f!س2YV%?$~s+%,!koÎ>"Q⟻vZKxWEtc{% :_mk)c}$$gy3}Ե;=þK {;hj -v\[7U#پr0'~byql]N;ma]ɶ?[Yhž*c^];ŷՉ\Eh+?4k~׬Eό=tùCRmuU"o5'2zVɧ RY.}.)鶾.s1 U&Ǻ\Gi̖ᰮni.I̾ߙ}W}2\x@ệyuF#w2~|Tvz߱_Jyp`,Ԕ vdb `:[sTTH( ?iP2kF&eEb҉:jU a]Hz** Q ʘdJRj!]gfT~+ug +T*KLB CC~AIڱ_A34rQ=hFI1Lʕh4qWKz毠aIRn^UΤQj!Z`N%RXDnԨWׁf$JMjih旮 ) CUQju:'_C3J 1P<74ߺ4yZwI'E>@g`$JH;zL/0xc,´*r:|#9۪jx]Qmf_y2nM#r ##o<ǹT7 n@Z[&06uWN7.:pMcw@ɰw!G2|8c2=tygR0\Z{Lx!E8 .-yt3I<*@AwH =pSc!-Ӟ[WcsT5:Nt*Iv[ v,ZB\Ja5+O>1 P X@`a/s#γP s)ONks$“j7Ƀ { ܟ1m@mCY(!cѴ;9t4oz :/6/Kg{bu/jOI5Ǯ8,4ךB;E;w_d}@Ku1[iF[ÚL|ʃ5*&ldZ45 S8j= y?13(}.-%g73[Et<;t-`cՑM_recƶnu>#{{wm++i=C_w/a h,ܫb9cM,Ykgl~/Qh㭞b)*j 3E?MMTzQ@#n7*ƶ MK`@w,/~Jb%FU-F/On&|}ӗj|- %kVF=5uztqqWʞ7e[:u6Gu`bTsbt^eQ[#ƭyYBQx!+Y-w%w٫|?;}W[Vȃi_.0l,q!>KMD/ѐJ{{@&2=&@Q[萲4mtLE酎F9DV71V=_u=j$;aj$Vn2|MGSVߐj<7;Uv,1+}ba+ogfYv鹻wyckt =ɵ^xIGeXɯe?3pn`}.#ixQ7^, s\qC>5 o-?qMV]49 \ y}BWxT~˝ hUfZ28:?y WB]~纯JR|;Z99?Vk1:6ehlAK6>Ki=WJﳗi¼Ȳ. ^VYnliƍ7Lzj[3֩s u.so\R9rBFV5"uݷ{ɞOF^mݡ{Բ=pX':xЭ^gIrPc~/~04@˱WRǿx] Ԋ~UMFSa jº1RP$k"WXI7W!(o;k/Ќ\*T!横J5Vt7`1KLmP=Dƿ()*A35W{]MD[TO- 9 Φ(ϑ+=PTHe֌2:!\K"冀\2&*.zb,.7W/ip@)PAChki' jP@hTE5\g0Hik* JkT4~%T%) e=_B3VjҜ4FQ߆f;4tR ݸ|~tQW}f.ǡRwy1݆=q մC(5&[[] F!vGлZT(% {‡CQ㐹[K@_ѧ!.>A4YG| \_ӛӍCLۍ,`) NX4#p@|v'~0$MR7vsA 1/)O ?o-]};QzAWZJP&y-u.0ukG ]C R0in{:lrX1`3  $eƕ7}?snOҺ|05vwS<@1 k.GAɀV׬BFU^`v}Ù|r1|y)^>2Tӷ<5ቈ;B nFk6PDsBW#@k/R dM%+X>oҖd g-D;p̵bsh"*j7]5LW󩄀oL~?8(0lyμA^ &u M2%nj>YG;ƋV JMڠۖ NҾ:?Z"P*|5~rUŧ|P-z2]$;["rUI^*uG\pSMuQtU4Zg]?[XOR_Mi=*HUEaQ9;6e}9x-xjlxueD;K[=WK\Kr;m%Rl Ӡ 5}>Zɪ';^~=gN8~zhU\+6j|ģ R^S: r/JNtmDRmYzQE#̴ NQ~E%o^[cC;_Qk%jVSҮFw?riSr'aM'_huuk'VlF\佾Ѵ`꺻6[佺 QO%ˬvly|r;+{EmܮB'Th[F =/SkMol19ϥ]6|դJ]>0T+sKcJ`q mjdז=5Ү~':ҶavNcC[ͳ 7$b6ر𠟑}>|X[zΩҟ &-Q+;ѭpRG=TMѤ{}PxRU˷cRdlj(ڞ53c6I![ 6_gཚϊ/=ܲfJ׌jf'˚9}'7lĕ m{viWE V_heaOqW'᫮5%oBs7e˄͡^_~3!.]a O3<~{|h\5K8k)q_sMӃS}l r\Ito% MAՊ C*KPcjSBT( n[Gn#@ )@ j *h"R(- (ٹ1 R )@_? O?_L篆B(`@Q/1?zi"U!//|E`/GQO@wp~2,X}j؅0 ijg]/lJz&>>=yS ,^+z'¶o@#u9K6t,F2XaCHG4-= ;L!5FWvsGڄR&F[ॠX!+Ӵ+VXQ[wwool'ZZYhx+pOmy`:AvMԔOjaHj1=Zp;bX<ц i_f`H0h|W㥠ڹZK_7?y2>0LQ=)|Up"?ұ;",~EݑWnCt84ɊӎCMgt𽑊{9;Ŋa Y)[ՆKf)]*Lhjƛ-zaD짾FZu11^>|fR簥?4{,xtvGOݹ6zazzO!zWqW墤XIHͷF}*>weΏ[N ZƫnvKB*xruMz>NO mP,3ߜ.k1ǀD7I,H3&oI.h S}?HyǨ =;K]?٫&y]S>r]Jȣ(_- ZԎz[:c-_zOkZؙ%JsЈ^kt-;ּrQб9 )_0PSBE{? ,`謴 涡E3e%t;6i))Q Լ:GS)fZOo8Gz5vFG+*JsІ>hehjQ)5$[&{iV&~-?<3V7i94U`OUH}fߴd-xӬUNZ%e{[$Ig,P׉sfFzIB_}ւ Bjޢp'?[q4;Yy:InK{V6,_,<n?oөk5uDPcW-n]փFjs |~Y]󸱇wcԬ!?Kzm(ɝ:zm&m^hvmH.jaeE$JJyF$x* JK BGCR篨ǚ{~QȄu$ o^fްJD\_5NR-ԫAKLBAcnfkFOu55ݐ1B$E)jY3u12kL BDX3_fd,*Uʬ\aA:4Tm+0^y!k(D\n]Ěa)Que-h+HlTIT$uY3RΚQrZ2)XA Xq}̢G&Ӑ =c̍Ű9@>s@ &bnN叾%43 Q! 8 $_&6o0 |~cb&0G:n8K׃8,>~R tnrS? qv9x`2NUR0TF-&mfh֎N"JYڑ8M ZΥhV1m" pFoHPR{ViO 2m]^LA(}77/ѱ@7gLᄨdrmRYa0ү3n2r&$N'zeiN@00-M` Z'Ncl)[&ksJ:>Sč;1e-jym~An4! L%H>˭aLa=/0FDmqe Հ.)A41 !&܄GDxl [g~a`F8 @9=p˹q۟[:11ln*9=BCzrp;o x`$v‹d߯Z4D>cTFSe6oO!z8Ğ ~\3,},"W6r gA}'ۦDʳd12FߙÁO>NEs @h`{%c6Y,b 4V i%-:0B Kf )7j|:Thr}^|SɹVKN]X:a;C˓+'vR9QU+yD7VQ9f\ {ZY~]U̪3W{)kcR_wo|Xcn7&\e'u|?Ip}w}{JFHTVw[t{`fiLR~`g_|tfx _D]bHsމ*JL{E-Kt"@.՗LrjWF8_ը]WUCly;*T|5fz7R['m$z9:}8RK\ ?"ԨzȽ_g=^;DDimKB`ap!7.Pl%f8yjּ%g>UG q5ANN#yC{rNf.i=ާ-6SIr}1Z_Yq }ݱmEGqz(V{]He'n85.cLK1j{|Ȍ9%{wIgIѥ>_-T()^x0,F"Lsx=I\fc]/yxxeTׄzM((sɩ9ک^%la/.Ze3K7YxwJn6h4 g:Xfo7;fz`ErWo N*N'?^QP6e䏝*|pZIMFE.WQJ|kvU^+҃ŅO;;?(WZH{~]q_֧iվ*m4Ru~ޙi<"8Frb:X|^=u{ܤOgOke묚p~+fT5}ECq<3%ܣ^}-ȏ?41wN*=hi[1Ah$4tH7_J1\r+A2R\XI %GjTPwͽB"PBTdbՆ}kCC U#"Tm ԕ$չbBX5rTQX jJڨT7t} B)H~Սj.45RN/2+2 QD:2̚bZWIGeMX US\ů0Bz^3Kd*zڐs}< @Ǣ#Vmx_X\LKU*<"/hFR#i-ʔ@dVITSw?oA(ԙ͏K+(FkO 197#jVJMv" ͣt3LxB5Ѝk|Rx0fz8F0-'p-lfzۅ6c@}#lG_J!=Pn=ɗpӪ`e :<}2݉g^v/z$Qf^<"ZQO/B˞:n K1m1Lq&fCԗHQinGoA-ExZm~mD]{Vb'{g(4[d4\g  LVQAg>3a/6~ZpSg=f.Og7Ut b'/[HUL`Ȣ̳tZ`nC1C2`c8 *"1&b> khN.ئFj^ '.L0 NHC2'L;k-SP "B'*!΃6&J.3v?e.byHZli֛K4߁H+4< Fkѹoh,* iq_$U QpduN0_쿧&d"_uf.^9ѢvxYM3 gVw5Z»pP :NхwDEvUU/OM!:66y.HO z Y҇@=&Ɩ8c'M悝d}NK"Lם -*mz{ݔ*a*{tw:߆:cv~k^`VHm@YCs|K?p#O3}>#aю$)->Ęvy*ia\dF5uAPj[ME]TZ /Nbtry[G[2Vp*pԯC]7?wΟ\F}}1-(;/=T n޵/jiTJwSsEyS&6Z}#iU߼95q0Lnr wᤗOfl#]U?F%E id!WB[z15GZu^SK ^L39;Zj;w;?}8jm~ZG-̡wgBFv~g.s9m[fp:_:fGV768ȵ 4ӗ^3mjˆHbht`(*qn6VFVnr0K (jͫ?PGHEDo2z}o#MK!^˜mO7MA<84+yKL>(#̘#6U䌫<!<Pd/Mc6?=2L1J:n[U#!V<1LGzӗt2B5+0 Q$@pZzr[ƳLǓ6i׶C:@; 4$z ygx 'KLUa r#[d:  n܄cܥ/0՜'9.0׿/? $J̸-w^!&s5ɢ2wjgz$]d+gB= աكCwoq23q-d4VMZwPØkS t_H՝ Ή)7'gI?'i?W7o= $)edhp]*:n5;Jh,}>N.|n.:߀Bz HS=wf<{ᨼDL/E߯﫫7(Ug֑ GormhQ-D$M WRǿߩ3B. (QtY/~J$YԆJXQIU&DqC򋞨. S{ZSPZYBRS,SըJB3R/dr3EW\eQQarߘQ"ST3P\$Qb͔IT ? //̐MLF]HJ<gJCЌjZFDQjHh)JL1RBl%@} )i7銪tD@@g^fDzS\U%l-(8T$ oJPM _UU*^]ߍS0? a! Xu!-FH 3Pؽɀ3W.!uqTX'SC 6r /-uzږ B ba ϸL"U"Ԕvm4<(:z|;4ZCkmQbpwb>Oө !s~5۠~W " ,(p'S6{,$hzXci7肴 \t jh#>54N/M+etyKMcǘGMA0loDw0-ꬅj؇FrގUݶ6(SF Eka;N>V|Ђ `t&gyw{uFԴ~NV ;PȲh(! n<3/ ꬎq ڗ 1QI 2x]iV!7}1 f'KKQN {R/C<(X}Ђ fŒIb%VK:õ1>^gߐX~ڦMϖ tgAu=G@3m 9wy@neMߣmkH9~,qWL1w8)e$Gl=oYۉj  dO0'2kwAO2"«hj_ȿ FZ|n@ҾGA$Z۲*5cr!]8W4)sI>M(zV,ig't05PT\3қg=*N5wwDQ>Et/L7yѓBzf[}]8뒢]w/=}l'L>xY'd{=<6= |utFf YU&"N<Ҵ2(l\`mwKj7LgrG7C* "sf7jv~eYݻ>m/j%"g\E_u~)&{V"Ϊ ]E7݂ =Ҿkъi5-J~H|{#xݕ;I4*|lHZUCfGNv^%RvHpR*j(JXCXHU_UZTV%NF %JZ!%2rECbBaFYhTVRp_ԯ$W+ T6lTvfFLNj俨$JqAyUe5udʍ ehC(GihT5GJ#J]bA=F$UJYX^&G,bFYz/A.(dJ,B@ v@ ~qnwuK/SO+(Tr /+jqU*UJ^5Z鍓ͤ[ ?; ֌X:/Z@K&C ҝD-K0sF!1ڃ Mp K Ya Z  >T?[E3%'ae3w{xvڌM &"b@p1 -ÀHc :z`aZq4"{ԔCE6AA6+hfNG= `ǂV{Ǡ݅bNԆ3U~5i k3m.:=g@[? ]s3 _#{xwzKVoq$~g@TlԊ￁'~◼"v ]`eL]EwC1zl2&bO"`\ :#挕?rAכ!S'-p}w[@ 2a,$`*pAq&8oȗL[oʘ c ?\D笁L0qR?^:-}j lnB;qTj]:LΣ%VbfHD,K>\ l ΘH847ƒL6A3[[lYH4%5`)aiDij HÁ-Ny%6`<`tyU귤`1AfpOȘ?MQ"fW HEtguPBdE^w;%*F/%*]d sh`{&fhL^/d6r@ᚣOH/.uϫoСO0Z2%{р7^!@F hVI%iP`4725"_u#nwo[37ʷjeW{sϔYɋ&f2u r\h4yy{?fIJxfQ<(3';@rX{t jJuF凮 B ׍-+qtvy<^!Ѩԟ97]1ت.F?Hm  ]שgAN45wit4 {xi8VF_&uo]PiW](nq6tuË^| ~{;ĶEhUm5/K:}E׌NYuF0C#'Qk3}pa}jFEEvMʤ4wnFKBC,vFD30:Ke6¼vQauOSmܟt|ZsqWsD[k 6"6zLn~7ԒP3YMF?%lg9vm[i}E}˨Í$RkiKAGw<6._>Sɽn鐹PܮlcNwffzAAkN9*|b[N+1SӋη.wq\=Ս|!D6ԵKKG#׈]>Zvlji'S:4؞H6f,8k=^ݕZ*OnE w#ŧCQbQH)wWU*ѦQR~ZsZ;;\]ύ% Zd4DnnsydsU CI;xS^JSmX< {MK#Kf{ڋ'ySꓹjt$rTUH\UQ;^kڥI/v$L{k2D'w?ngAO[qUgm&ds`c/%[}yZ{ʋ^[pTS%.gZ8&DHO['0_ؐP6%`hX;M~ ʸ~!ѹvƼ{ `Y\w2Q˕a pr sM5h.LodYFIxF&/i^sPG~B4L#ţCu[G3>u1fw gS?gc=Ǽ.47aR>y"x"g`tWt.o$XDyLRR;dTlitmylpMnOD IWaNoP;0&ԡڵV %f!{\ه`oBi.gpA\&(d xq+2Q{@ XN&`5l!6孺Mnq3/\L%8PxCxBO`KGCFQ<&4 ?M$^0.4>b&E46AA6,a+@Ck!./_/aQcS _>ү&WGYW=+~f oux- rYYT }neF%?C:gLhwQ(O~~g^>4ٓl7'ʏ ˽۶yb~y̗ݭn뮾ĚߡKE~A6>4-3)}s.mث>W-פ hlR:äha"K *gt~1Cg{«]"߇O?fk>gVihh.q~]3oFkqS'NvDǷOJDvtbǕڮŶ>z5_Od^³Xuq'.uwbu+$ja,}ۦ;GEVzum=r69nM})o֋'^)Ylq6`}ꇗ6{J#ͱdwݰUyOo,-K4x>V'l-Q +WYV=w}ژ:j&a~NN٠W>V?)r4$~4qBE*kB.i5"eeSk{=o2n>Cw8zR&\ux݆8 X̛Z ToDbŌ(z-v7 ߻cTzLWRǿL{7 FMT`v-Q/QLJbUQh)K dJ,(DmwP(,(2zZ-5 P|Q)WK~TU7! ~ɳzb-*B!"Rm[5#Qԓ)R%hZE"i7D(/_) J'( 3GbH፫h*9|I0*A!(ZLJII _㦐+C3Ĥ\`4 5qmJ-2*u.A˫zRFIVJ a&e\5\KhFˁ4ohDe U@B%? G&at^ངz3+3F fFCp)=P<@;sbO 3&̀h{ Ѥyb8c 4YY(<ܑ.G7{&PxN]$<Hg8v ?kf\F\5esx39#Ճ=ޯºÞq!o!qTZ<3)ҨjD9"ĕ$6!c61k DL 162q;U%8a)`̄?C1x aYcXrVc2;a7[fvE0zwE3DsNM*aWl j} ̯u1 " N1LYxS /l,U017!Z eh?]<ҩ{+BQdNbn@RQMV?[;buGT&Ӏ'+ V2n[_v׈0.!NgL83qEF`?+ q[ƒ ';ħ_fBsg__3MI~NLHd.^dHشA<6b;&H_AӾF=l"lܣ䩎߁={or}w]M+槨ONy]a.[nOSGv3e_3erƲ&z#ڤEnڤ〗WX];I; 1)ڦ{ąyvY]n.hެ/ HMEVR1Z wnE*XWŊq%0åx=Fߪ6w} {Zōc[.W\1g;F? r)m٫sQ;-zz)9A1FW~boȻ3awq%":~i`WM 5ULXnGN6?V{Y}H{IzDF[{ܼ,Q$Lf#.y_^g+[>]ץkQ^nZ_a㣨=[՝UD"(Pu@u\Pqe :npW0Qt\$. $G餁@zJ-;&urιSw]uOo~o׏⳵=|[{"m聾euuxi)+使5㡎~<Ek C\qKst}:"4g[5n 5{B9/3_>% j>_{gX-)oSZhM旭k۰ͺjOG=uƢ3KZ.c;͇~Y:ϟ˜R.,>ow=礇^Dɯ7lv`jR7Ӿ:)ɉlrĹ_kB.OJz|g5n'#Rw&o}l{ߎ@Ǐ_j438t̏\:u^M慇S|qк֦|R~پOVm3oƙ6/6/}{;6[idoWE[q{/2{_r?6Im{ٺh`~m/F u'GuN'?EVژʱ=Y3aaRxRdM0#,kYVO,ikiRqY04#6 CDDvP`19<)Z!$be@mݚm#ڱ2ev Fa\A4 [+*]:Na7{Mf+DݞI=>So+NnStD?i7<ӴۺԒp}&5$E)K#M>RYS9_LRwGݱ-MXOdSV*|V,Woߏ j|Z&V/xM-#ZMB>LӻO-<4,U\ ~r>Mt=STCE#u[\DOqOF!QZ6D)HLLHߚiy. R'1Nc!?.scܦTi3t_N4zr巪:ݡ]x(u(>nL幓\6S{"1:V'/ЪdzXp>BQNqbc3szc+-&ɤ5DmPtgpw?'Yݲ+"j0@WLOM3-U*Egl~ fڧ, Vnj DEObz&5G}ya/msعa}q? oP|%"4-TK%7넢thR'c]r~0+e.;qmOMs+'}!sߺhcƬs/.z59.*Nū^}-^ܬQ /Y5޶IC;]Xp{k-_?֤ёn𳒴^ǯ ˷=: _}9iڷkOOg&;}OߚcS?L+u}oZY+z?X-Z~8'vea׮4heѻzVnWW?kjW4|ԝ|rߎMn]lfOaɗ-j 露bi.ЈEnƒGNvZ?*_{u~;asd}ی)oy+]x忿Oj޼[ݣVq<㴺 -$G(^K>2nu.YeFFevn_0m:v}Ζ٤Jm4Gu|"7yaկ~˷?r^ֽ=ܜ(ޣܹFd~}]OkQJG6>6>g>x0=[i?bN{'3~y K\5}_qEjINYDW ~OD))I&\:1C3웄RFc9FKyKW*Ual+;t2'rp1 .ϡp*,ǜ;E޸UtGǶHeN^fh*|J*"G 'X}B R,~W'n&lPW8zӐ7Հit^E F<b&7HFŗ?v=Ҋ㞞__~tLJ*%sV~_9gE:|\1}B=?PmgBZ ^t,wɷ?EB̤fv)f<ˉ(:~J͈HRԖTVof"E]P>C(5H tZDي&>G DҺY0 K!CM5WRZ7AEbDBtXf|YtH_T5q>FGgD?U?xz-~B+V_C^%G*ZJH?ʽ<4TQ+v}{OϦh3FD.״GȥE"E3Ѓ{OSO#ŊúWV(M&5w"I.*NQ ܘiRķ^2EZ٘vţtdNUŁѹΦú+6=Is:i?*[~jx~N93eŪcton{9][t!GYں]8*ftigCڬpFJ[3L>oaZy'lP]k-iо黫:'s| ߳hnm#ԫ??vׯ\CC[e{u_ O4o&;;6J75u]_höuY܏ʏwz9I_nXwz-lT&]mַ7s~bͧ4U?>s~4;6_t˶µQi;X^|NmoWv%/7^ʞt#NuCIY2c3.f+na7;|_I]oyCO/>r]C΢Am⶯ooݎ.ifO;hC?:%k'}aԂu~sNnqeo}|{& g|{Zg9|~ҦϪzr1Oyjmm+ ,: 4G,ힻg!_6jw+'r5jk[fw57*Qtfii͌]pk?rչyqqֆf?2c?_pm훎a͏/A&K.Iۺh鹏Y}9_yך6&*=Y3]?ٟ7:/Te{n>nRnٖWyw=C|TaW'n}c xOe#v~~/}ۋd_1n}o8qoJϷؼ*uQ'|peΜu=37|%%Vy#U+zv4TA]SEgϥ=n#*)O{8 m^Kt5q+Eb:e 3H{~z{NIgPU#a щgw giRRkhXorM'2wWT/|ETcEWQe~Xzh׊>mNTqktH j1~:ұӣxGԟuD-nk?ҷyՅ?DD4:{)%ު:-g"ՎF]5/xpЫxdSTBTj: ;Mܬӌ ΤMD7#M/*bzKQy\MHC+ O3"Zc(ŊasxaԆHLtCO11^plDIѓ+8Rg*HnGi]x\G[+Ukt.)7Dsa^dRT)oL}/&I|'~1EL+k۩ǾqD:[˳R^S&Sq?JoPMSswop?u$k۪hh7I-[gv;oO'|ߦ8_6GzsI+-~R"Z~|C+/؆WS/E*'U$[PuFM~UT&oqږ]T&1if&O|4e4"Ŵh{#Lb"RD4_)i>e5_9uSk7@ʈ-9tFuXAzz6 u>5{֊)3>SA_OS Ky?uu?Z4΢)׹m77#eC/ ퟛc դ]Öli!u^K'|+'%i[EoyU+<.c}>]-6cރ?ja#_iy.{zWc~q_Pr}cJ>?mi0MӓGgvU_;k{n\n|j4&F{61#û22Zq:ҋFm\wt/Eݔ4$|tp[}f.+q"]5[g}}V?R/w'/Ѫxm&h5> _sW k j7_^g.[vOo5/?6D/ϜwmGN94iw|9i&?\0wέuk?^6-;}fgFz:;[)4&K/~K_ SX0$zgv4򯊼!]߳{.~q?I~ۻht_n^_-iݮtkzkm[u1 ^ N PLRryrrʆlvEC3-dD@"\RLW)-U>";ZQF 3i*ʩ:@IQ Ͱѵ2J`H ڹm`B3-*7B3eЌԬq;Uyt}A k3`A!)4ĐmЌTISy434CB8Q, PO0dTX%cILd4١RE=H#%Ȕd1f׈|QWECFm\{'ul_@4L-EMo 'pҕ Ozo;h']z TL|:慡̳bQw͝![?wRin3^hբ<_݈;]02y':d֊}Nq'3_>dpFU{UԘT!ʾߧrjt?6iox\u4k( #"E/ˍ~%Ri.5hPt6)IFu*Hoݡ9jɜ nrD:0tRò7ԉ.L*B4H÷^K5QόwT״RֹD4l}'"K4p"MdYJe%j%q"uh6>&ZVUё ږEJ}i|H)g1=f6i&u=s ω*V'D̼_hM|oɋ'vshxzc֟~{%1yWݕ|UgEŗ KlRWVN|?-n\DwUg3Ox6qmSԙhBuZ7O!89ʲT7法t$ه^]yYj+L4&&R1#z>hu\OoNĵ?I5']OEۜ[_zsy WKH^V~[׺ݟkE#5ur_ҩUOPzo{ƟX-昒jYUa3lʡ}׬4}S+mix1}}m w׿wU8MYzյ /됍Pxk6H>?Rʭۿf5G~5ҡ/6Ƥ7Ųoڍ趴d748nz>{l\lNy+FIRZ-j< ?밤Czv+(ٗ7q\nfMVt+>Ȍ-;f͡N %aSxEHl_2wx~iXɇt9${߆QѦ?mYxVU >kCOhK;Oܼɣ Nh:ڔwE'%p睞x.'>ش݉Kjg}_4Q!|=GSEtl)E|xZoyI_}q٭5g8XkBZɵ6踜3knQXﶼp俑->^eg~tz,:jӹvFZO^ؠU>'雳OE5fdPҵʠo{l(;eWM'/MZ\۵:GSӜVy_~Zpwd esE}Vs;~i4|eoKk:kK\\F^ּ[>7:'9v=Krꩫ9}`/ߺi֖,k<竒F?<Y;\!獊 4_c.̢fwmV)F ᕧ79#iC Yvmm/lV e߅?q'llr__>|eu\zo5h|XM-Z\:-3zeIO۸S}s?S"Ϟz1:-Kj5oCm+˖߈ZԞ9)]Њ5>r̀ʖ֩S޻I_)8uy_DJS#&x$1}9bhre0kuY4붐<ҖQyfGi,ב{1aF ^cYtCR:cb )D9 \jۑ0w`u9-aƬABh&0(ͰxT<K%f9.d(1z"f(d̷|9Dbdm'U&W+4c֞y q0f@6{f7Z?^%^?VX˗NOԏ^͉2`r=ƛ[dtkKvYT!3&-?zcK<;^nZޑђ[>n)K5okn-ݘ<ꖯs:. )>wCVf-=NS׎kkWpvO5d|G gI7i7V;{n.V4x|z-[E3\wtFcښ~$Kq^~X>Sdlm|YxUѓˇ8S{,o-ZW\㒋f=Щ׎[T S6zӿ {ڷۮo}ZE-'9ϗ,s\yBMFNkRl?nԚ1c~-4mh;pڍ3oYhUItPy/ZbW ;ɜV1k20W&%ڡ'oqi?V0zM彮L{1~s_^KQV֮<؁M|4ʓ>O?){ O4,4g閵?Zuiqo,s9/~}ͮN*ِvry~kT#pZH~_Yݗ9hSY~[ֶRLVygF]uUv>nW*>↯^MYv̊رpcۜro=n.>eLP;ͫsSg]nY'l＄Z5ۏ|K;Rkq/RVȼi_4k-=^JYӭ׶iOjk 롹3;;of(̮n Bh aHfAh aHfAh aHfAh aHfAh aHfAh aHfAh aHfAhDNj߾}aaa@ y,\K/%UV92 Bh ?^zM2222yDH1se˖~[~xY3rssAh a}aРAnsnnț~iƍ077}'M=x,s 977w…U9//o޽NZXXY(?QFVk 233cXTTtO>ҸqhKz!&L0T K/o]I 4ҥ˚5k/vҥꃯZw999#GH%%%r^VC0aիMG? 4>5o<7.S\\<{쬬6>cQQ7`ޡ`ĉѯ5$ӧG2k֬8''';;!y`رуW[iZ (33sٲeDT#- 4(Ș5kV /fƍ}]j@g2dyoKVV֜9sD մo=رc](hҤInxEPjoV,.A k(FZZwgAAyyyѸLNN!̜u~~~^^^C.!Xcƌp_Ϝ93# B3Oeee s1?K ,p_dddTDfhHtjEw1cFD>vX}SSS:,W\8L:5:^QQQ[T;nO2qܳ>{ȩsss ڍ]LkΝ;o V2'Nb(999ɔ)S=W/LFFF;vlAAAg)p)S=c^5k$ 6=zt6jbY#S/X / ++k͚5ˏuowh׶?d>u衇VUw7F ZjU)Gy=ɓtRC5j͝B%~%===ר:r޼ycǦyTXXضmӧW;`?}N9 &;;unnn޽c`Ͽ˶lRHtR|&LSAh#be͛7 Ջ3 6|_ݘˤĨx38}1}X5M7nWkOnSxcѽ\O=";;;9u߿#b"jݺbU 8s=zSw):!OgɅ{~"؟3}m4=+»F&,))3AjރxLkX-fM5jT >0:D)Ӧ{.^˗/w_dddxP}cȑ& ?d={/5MEEEF*ÇR# zl2ȩwUD 55gsСǏ-,,4iRtVΌ &ԩS'77כqI&7g􏥮)SWZնm۩S~Sm߾wQ8pKi;w";;یx#uԩS %zqիٳs„ ѹ*222f͚s3Q1;/K.YtɓFӰ`uM7'=qxETT`j9I1xSW1 k7rpC 0 RPPU6cǎ?8+gfϞ]1M&zhL,999Uϋ/LBJFFtI6mKIԩcMi H233̙3{@@$###''ߍ|!Cx>qK.ٰaC`_"afeȅ[ZF&ܥ5\ >B#2DgjH@~x丧bhbÆ UoODnȾ#QAAO=q,XJȚWt.[l„ 5'Nٳ9{w͞://oN2cǎyI=U7Λ7'dggwOq!42f^p BA@ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 40$ B3 @ 4`ChH嗫l…/> %zW^rs9Ypa{}8.]8 իWk׮\pAܙ  69};4] 8\K/`Çoٲo߾m'dx@hT "q.Bot&!~%jxP̜1(33sٲeD`e˖F9CDmתUիeO>9c9J*!k 7uTT =V^hqܸ E2 O.\x饗ѪUFԼh<h…_}͛oD%? W 0ࡇU4ffz2e edd<36 ݻ\'t]7`O0 %\+WիWP&F3p@@hlh-VR@vB3p` MEEE۷W .bBsI&O:5YQQQG.))#^\YXX]7nwTyyy5r{ˋnq]l7.߼%%%a?޼i0Sf@ɳwYII?aKF"nӧ{7vLh;p?<===֨fΜ}]ݻj=,\wށ=z>qyO?A8P kfהr)b m۶cQVݻw U$==}3fTycǎٸ]rv$\^^>}z߾}ȋl7j&8P 4kO(^f P\\إK.fgg/X WAAĉ_5jT>3Έ:VMƍ'O1bDW@VVrĉ1dȐݾj=Ctk׮:t;悂9996V뮻I92Mp_;6:rȑ#GvˢgϞxSNqX'Ow `RXX!,iB8 kf׬Z}g5///r+!333''}m隖6vXu1cƤ3%WkIbM2Esjj?_}}8(~E_ @5Ѣ;D-X c%Ç<+Z$4E; <8Q]j=IhSzzz4+A݌z`̮3fbW_}un-SSS:^>}D#y`蚬QA\'rYgU-ZjKLo ~fvySO5>'OZ;*'rܰwK\v͸LJW*%/:`bU/<(QLV$ar{@[xM`ґC*{""JHq1~󍋏dFHc|_~~)M,--mڴi ,vq7iӦ5;sT^Ěs/kpTB. x4-fh,^z=6l"ݻw4TѬYcnٲ%LvfV0gfCwK<\'18ȰXm%Db|' ()~c8:+QA l""f;mB*r!$<"|τ%UK Y;kUn4+m4L)6RhOլ(:2N'^I&'h Dd 6+*gܿI9{*2z͐ٯ;')q)')FJ#9Q{谰N&mDH |vwV,WU0C'*SR`0%,ZL3ÅaYϐT6[v FjߎF3p`Aht)čSqn"???77YfΜG6ѣAz]?bĈ]#^rj䞴m6?sv6n8a„xӧ_s5RݻK0@ *=""UN 6q8K+C\mL7-]yC3v1KύViyt؟Pber*HyVNJ"Y{>a&14Sj|[F"xbR [}ũEBX?FTEQqRH?O(p0$MPSDە ͐eR\ YBR9"BhdrjZ74.<Vxe;v8xVm'dǚ h8|b b o%322QF͞={Ȑ!澅Fr_;633S<{tbA8tvE.97O͕O!b3 1+lh8V`y"n:SYS@v3Llob7cN8Y! #4Pl*srrڷo lܸ_6^8qy…CF& ϯ"qWO܍Xт ̼}9`#*'o V9ڗ">~B0lYP`HbъþȂMD`HB3 gٳgWձcNJA!CdeeUQ>}2223fL^b̜2eJ%WkIjjibaU_7M2eر`Xs5`XUXbj2 (B$l!ƿ<G[-m#V! ;h.EKW(|is(!!IzwQ 2&!ƃ*US0b:D*)ZzB+6n{!;Y14 ƃ41Z)Gef]v@|`FWƤv V Ihl'byd\U.DRLMӬ;QVE-*(Tf>x,)4Raaf6'q Vf3FzfO3/L HC9-~Ŭ%d=%1;jS;J!6 P#W|&X%q1zx%L-^-eGBBwXvE)mq&-L%NfbeGi2wGk_OI9PZR }@֕~nCD/̶|'2j+65rC36"@+FgP# %Ic4kܝDl9,>ZMZJEqE[wq߾3-TfȘJX6-.*7t 9at2Kg-zЌ94-%V[؊aRa8GHIE/?&-sN()PE9(VA)ذV8#1IR7A|쏕Thҥ g%TT̈r^8+sl%ϱtǑ&/߽X8J}|,B-z$"6e `YNs4f .얊?78̚1ʍB,{SOl 5Gdef0fG|I0ׅ/&bk\.o 11/Jhd al)(SD*a!]Ewb%ƈ䲤&K ,befJIHQ*<)7%ΰ-c1f?BdO1Cڮ ąȬa Rސct&RU*6ܢrcG# Ĝd1C%J!6g"JҶ&?Qy(3kx)cЌg/hJ ȍN!xc\yM96A WLtl 4fhƲeF_#f0~MF& YqTc)bX h(۸ EwXDnGw wֹ0B?)]BJ%PW3eyD'Ǜbs10GO,![B#r*٢]Ќ@J`q2,3i . ,Ux~bo`_,HR3PдOҮFh)MgN tuav)LZȍL&_hl (NdZ5 !97l@`fI3UIA2+"-GB1@ Ahf7.+ }2~t0QxVM_ RcGv,FiI !RF)BH:b3QRB֌(m^0P5)g7t剖 ko#̳LE ~%eĨR S,ücDDJ?LMlٱ*^9q|165̕M +P sK؜}ҋMc kKCKDdkEJ3݆ɳ!]ܩ{B+͌zIHmq\\q\b@QZ۱M(UnƤBfjb4%VORlmkxT$TO|S-3᠕R6z͈όB]QR $͕b3`D(,\;N{G#03`p7l|3sK 2ⰔXTҎfuIʒ X w"-u 6+;kXbز&7Kþ-897! JI1鑞Eߎʒ1CfWi\IGҠX*aVr˯6r{|38o*c`Ƅ B3'l8Bt)~MY?L$dفU;.N}{c:bhF̬+f/2B3CfLJr6bmL-VvkX; Ӽ mtGxpցЌMnJGma#O6(#R RxM阏$釤!!|s(T\j3mos1(0vXڗ㍬6AeW2Pc ~ X/am;>>؞k-L>2EL\yBU9^3] qXxWX&=g̲B2FIJ, I@gKRO;vVR-t< ΏrpȖZ-lN="fhH]Y9bD|$-rfGqoMvv|0-{k,bHJb CK5Ÿ́l-MK2qbcR.Qf*GC$ )RH|0%)\F`?#[1I,`Yv VL Q´M:btSf#yn$fЌY&]x8GDu!vCK5IJ$GŶ1:a`)PHb0m RfI br*݀ j#IWO &[ Si/B%s`[Fl+^@JEIgK%$; Ne-䗰V:"'51bQw&+F6ɜ|Fќs0$$} *Kk.q\-_L$rR=;٤Mj6ƚUXiwW9 )ix8P(KSs0  ޑ8'lrc"#,TB)Jr2.Jd J6h>34gVf"ƻұP ͨ2ey? Ӷ2aKK1G"&;pbM%ssҭʖ=/['̪\H1)3(V'D4UĄ *%rJ& OxMH$>vdL K7Qb@C5# %mMb؂]l veNmd{RdE 'HגF+C ,bK[(RDQfYUH> LEH,NYgL!Q[$fSP$E@\ܧMe>ƙb̩$L?FZs%p) #*XO q#.NcŚ(Ɋ SVneg}&2&*7H;<*V*/݊HBJx[y)FpW1"ߜ\sK kR,fk G_la J}ioA%zͨ\!t1 aI_ŌAfK[ooek+dLc„U}73LQ,L`B)XFK=C;iI7#Ē>*R1P ̂;6sEm@fUVBژKU'̐::~))ޅ:94ϾI(@ PX,qmsGw_CLaT" W:с'dH\x7ҳ++vXYd'e1=#JQ15$LyE263%(lZq=-ֱQfDd11ѫH `bs)c_bw"k--ԅPƊ،0Zw}P wǼt~_+)vܩZ0@\֤ȚasZVLmIwU5`#hl S=W6{Vˑ@hFWD\n  lHhrcFs3zc*|A~FL!igJVlmZ3^LaOblqe<0Lڈ"aۨrU?3f >l?Ë)ma&R JI7BAhʜG 4PiUL +eΛÚ'|c2s1Ƕ|!<+R0Hy?*bT9ԟXپ$NB/S`-}2N/^4T e&e21q,~8!q3i*z1k&Ff;e Q&ӯ=o'p0 B3ԅv9rdbS==ƒ06d'I"%/LbDl@O'\4WeX)-qG|ˣ`NJXZ*ac8O%M7nKw)ϡv(%xF‡FhFH'۲ȲvWnHQD!w"|Q01|!҅B,!NY,&ی B`P`XWbYh6S4uhN02Xx63fQH(vu+Vr%9yQ**Ĭ` 1)l9ˎ8/`#?ENFQP}3Ҋq]-qfh)?FW^B[fH h2{S`Sm's2̠Dܶ5~q$VU&\+%zf`.rͤ^3ڟJm *-W|6P!Gʡ08jw$ ,Y p8p4!O ;9~A@P4&)P|-nԱCwEH^Wjd 5`*%QjBf0E@FLe^R"0tc,U,U%d{d ˌ(mBRÎ嘛h!ĥlm6A\UC gF$z,u  XpԩS'Mף&ջqji̍;?rׁ*]ڇ%p$+PAKK26i6U9QBq/ڹbs#fbm,v ӿ0.ֿ͘vwB,&&vEʄ BXJf,ۃ*%a),D4n :O^[ q1GZ"tuOcM⹉1vU"}>MOn? ͕iú`6Hnڠřw~Ώ\藎ϻJ6>D+}Gk`f(hO.7;iN)Tؖ'ǒ/#]:B[R1 ;ЦC j4J-N-=oK-5N%\"vSBՌKKݠ_;$m-D[,NpnN-lcܤc1vv *TStEfAS- Q,&e }䟵o_6=rn$ţ o}>,|{?w2Qߊ;uΦV4觯~ln} N=nщw&-y["[oP ʾ!J +1۠-r)4g:Ƙh)2gW埨ZIXBIkFF,ĤXY'dHb 4aR|TI%,Lm0QRF?̣~OT\)̸ClR2vTIJ;3Aqpmfa߁@'KhÖ4zYփ;ӥx%=j a"y7gw {Ŭ1 6? |Mٶt4cd8CS`%L&(EGt7c<0 Dbw^ue ߎfpH~gR1}ɊtI$T8pLjY 28j{3I#*8-.fB3UJ'kI<;DW&\fr%W}`,T? uTݴnn3nLco31/v'A/Lan.X) vu=wj>!=ԕcoњ5([3K'D䟡yY؊98擞 "V`6E"ҚD! ɳx eB#8w|y:b9ve?0K8Vɤ)=^P?K;H0f&XSPr'At0Lq[GYEhCcJ PlktXpX9QTHWgbavf8H]TUcQպGNjR-5uUVQ]ƕq=Vau;Ӊh˲{r/ȑ8BGr>{;{~ڰnfKF7LjT:uNJFn]hd}[uZ.rxPPGaa!??]9n8呗W۷n?iҤ=UP,\pҤIǏ/,, pG4hP:uy'''N-LJޑ2>${Ŭ*T .۔XJ\TڹD%FUxR^ʍ .ڱ]X[BZaFpTZcbo$ڷ밮\4ܹ膬P`q겮YR 䎳f8Fonp _c,Z"M\H=KH^(́EGF%}Y+_j808]p_id4RbR5o:Fv_έ3'{d.ԅP 5o>}qQFF3<ӫW=?'L]3a„ &L2K.q\}Փ'On5mڴ6),,a(wo955uϯb_cϖT{o.[yԸT`Ҙ;GK:V?I%TZ\B_] 9<Y;A4ŜmdnQ6+#݃@cΒV*Ėe$lǶvNy] qBq~Ъw(2u NWnܥQ5 7HmDSwuGGwud`.O^YvDnm飯=H%dBߡwݠ;#fonԅFϸk.4z9|ͼB7jۿX#m]~\XXإK={!Cvu/WNNȑ#wuTٜk׺*oeF=m4K\wq1N|eOK~tM2#-e_6 )|a*d3;qOJ[BGOl<+#\4a׌(1zDPfF؈whaH-{pJmMRэeJʕ^iH:Y㯽rY?4#l!4#w-GNJ.~UF"5EϷXq"Jm׼F$xyo]@q&z9zߵ/{.t5=1C]h1{xueE_ 5F=̉;ӧgeeђUV5ȑ#,Xлw8Ҫddd̚5+=={?xzz3jԨdI*JmI!"x6zK1zF,Nl٪ܣyKxɘĂa?XU BIJx%TDdX!' &8̚Z%3[bӳgſy*XlFR3 s*2bVB%G.9șϝ [fBMa3yjry8X֚meR`8p3é*fʍ`՛szT~6 m9Q0Yedo1̊@IoJ! 9*8_̎;X$(Nl<ϴb=QBT\-R|u$vt8lfKQЌPxVJ,\]xa*mnCc|dź`GҎqEMa"m?&8L\˸I7 8ƛ"Fk$ywVpjdƼ<:^Nm04B6j؞dhн]3U9uBE> fiלv.4֗zu2SIx]mCÇp ,ؽLVVVff>---v!.H}6ӧO^رcSQjjYgƀ-[v`fM>{unl $+"l6ImMɧRڊ/8NM MMmIȡPȴ{ɛgk֊U;vӰDDn(_Kʁ |6Jv^Q|7Y'2;GfC;ڶRBm?+'rD`g&b+@gҧV|[d?~bK꼣t1E3SSq|3'/ox^foAD-қjh< bZXgxDm/ڢDfl-CD{JhAD=ߺ#L"YD+/n馪5^y iWtItG/222Ĝ"W/f߿n}a*LDΊ NZ&-eP!U1Y 1nK`a0`MJv*-3c{&c6/26sef+Tp[ eAK{Ǧl! #b0nWeiۘ@VBG R6v4MٹhQ'~b NQ:޷uڷ,^8QJD]4ho`y(SpCh`Bwƍ,"jժUw}kWz衱ԠAEnbm+4Z7Vm$""6Fr 0:,fOq69׈T6/.Hڟ a!eGy+k#B8v[&'ٖcj`sJk_ kKK3wCw\2)!nj8`yWehG+m@;"{-URqnV!**4IwLL%QJi[RLJK7oU~x0=rn\Y$T 3O"g\+V$65%tETlaKGDR:[`)6a!]uXP|R/Kw( mAl%Ԁ9Te؟XWoe6VlNDlX_P"̈#Xl{ NX06i*W#c|~*تpt9"$s"D6!`H[Mb&3G }=yx/'^[d~?kai$PA]hX[-[=:Y ǽ'M83fä  C@&)Xf* #ۿ6 [Á '8EQRfh Jh&6:ƽR?!IINrΧp*ȱ(LiYȉ-W 0&!Lb.zDo71%3%Mv9S!)spĈI_\IRG+6*;F8piE[( 6b&jqEaЌ)0S㯹cˌi$>;V?"պu횷ٛVU":䄮Ávzf&z W 4sC]h.Z)_~K.KJJ 3:t .),,4iR.ﺴ)SWZնm۩SE7(**m߾}4t`a[lҡ [;*< q<ŭ .%/΅"Ңb.tJ"u1f9YB\9oaaфcъKly碈mK$#Ųeq`l]"ʭRfQibsqobRdRϭVnſ8r۱X6N`INU,$'W:!Θ]ب.l~߆r .2G̳Kc;N57_i'vMun R  .AA u?m]hܸqƍs_Ϟ=ۻY^fϞ=th&L0axYfWs3y]r%K.'s1C? ]'1Lʗ<8DJ*0Q~bB LL#v0y氊Gı ?|o!_sʕ6.G>pP:hZC&d$B7κ)S2dHAA_kcǎ?~˸>K9𪙈8#5SAeDxc<[Dqe˒<aY%LL6b8Vi)fNWΫRKYdҊ}Hdkǘ8(3J&eT)My~7 `/2;%W .زfŎVlsm-V0ݸʜ8ŎS]Z(]T,qLPVnB")Ph%Um҇?8DrEY?<f8@!4Bl.VZ=n܌[&9gΜuyC.۷ˈ.#F̛7'F;w| e*U6WlJ*f'6q(,P_oS)”$G| 6\Tر=g,mQHPJØSUD,2f#!oHi;6gj1Ԕ[Ngl=kb̓-~EFKqsQlUU&1LG0gcJj&sk3H2lo4Se];^÷|GDx R<&;ֿ 4sŃjspA+D|bNNzNV.d{/4odIפƬ8rЄu:L`3_!"]ix|{"4˭+i߲ bì%\o/ߴ5zDs"ŷZ5!~MB3!ԅP .N"{t+ "2{5rfyo#nj0i銥cQJgGi3lA~I 8d^-3¢y;h68 Bh`:P 5䣏>իWݺuYܿf!aRIûo"VBI|٠{"J+h|&j33H|5ZѴ8!&*S䙒I;@5%QPHB3!R!ʣ"Ќ䈥v6Zq0n9Rmpa&3lLDcLfSg6(ٗ&2Kt9 k+Io~[kبu*se0. Pq6mjҤI.] 0`}xFY%7= +!4O|\AdEqH=J4 T l$gMj?+m$X!erhmjg="ʌp"14Ȉarf#b:! fBm[OZFh&u2kIi[#|-v<nGF_r¡`ph=w\ZK3L1،.3yEl|pHZ 1KvJ$_2p@@hf7.tdP 5s! .`>mݺu0PsΗR\%f%N[aF,wRl! <+Fh@q%XckqQp۶|w9soA1 j@cIFB?6ɓyݱ[:GPZ1b؄V$h M h U H!?JaYURԯ{~5}9csw{ss5\sF5g`Av_hq7W EpkĒs Q2VbM8\so'4f}O;DdB}g߅&\k [n>x=&@GG }ܹP|##o~˿xR2/C(;Zˀ:3/5Tk:-Y,˛;R87p}D jeuMeI7P](qoncN}{^P gI9izgǟ"C*lݤSV&WD+3UY4o^䬒HNO|b|B~vL')0G> !AǃJK Cx:m0r:SLSfx\(xiՙZ=7~O> +f_~_ cl~5)7BEs%5Jՠ(5pvԩfwG(hg %֛8PZBNnvyi*Rk-䣂%nG6 MDO2QJdR6ٴsyɌKh<Ѝױ߄\[JX^@SB'x1 p~0 :s$bT'b {q~v6䋱 Bx(eH.8rlTn8Z@ FH(S J(UDfD- ^,Q8WXdI5OaC;ꪠM{jVKܜכ f$;IhVmnj䇯eZtb}ⰿE"6<$RlH5Id BO4)"|s/q&Z6+ȮHF%Re_c+|Wv1KE\M\iݥ%*Im>M58K͗e%ZJOʌmv[CR$k)fis0iE U$ͼW}Ӫ>_1^xY4+ǫe.v}}?3OcȎ5'_TG!#Y55ȚQadlFպbZx#AU3qd,Jms\kekaV:f_zԥFd'oEbZvl{Uˮujt]1bl"ŨJ6)osȆc"͸hYV`ݤ B| spi0Sv}[YLdCłvw_'~ŷǯbN>~{;>7}/ƢW|&fB_`.ԎD^iRe^uoԶ`(ܔSw想MF'ofєeԹf4_Gp|?i۪Lߎfo^~Gi>qo!og[Ə?w% H3gs ַ7ug}>Gix/U`. cZj +m;Sc(!%k:QtlltP SK2Ec<~Z X`Ma&?KQ/p"DTF͹̭#cK^F~5SN{@T)( wAdqVpi.ElrA07p"jf$}m|XUvixG.Kٷewo:҆d`k}'Eeoͥ#9,Y6=US O~ʯ_?}3/ï̒^V1 ͼv\ Bʝ;w~'~⻾뻞M,Og~bDX[U au> #b SNvʹ5>%\UvnQ!R*"8LS(FrL$zAv.I&dn詈TdLI LJcu*(]N= +Ə?KDOkyyy__By=i5 B0 ^ny[Wۿ~׾y[~⍿_"?os/B@y͂Ч̅ .a\ǁ EjCa mj, 3Jԕ@5Id.t[1s0ԏĮ&7ĉ]*Ri*MUV͹fn-m%veLLaOT[2"NR֎>ƸtXg˓ohcp7YӌROa*.WZ5oݻ;KR7b*(iHdAZV}Sf [{aG.Z=|7乗^9 ̀W̅*jH& ~EI5hfqL Cy^4Ȧk:(*Ք V5zvmHdQdzjX=RKTT\ T"YHV@rMֱ Kh m4C|VӠ^;l&Ar$1"ڌe(L:.Cⵗ8PXL,կ}tz'}P ~wYu)v0Gɯo!/?G}+χ7'=;Z+ P^eAϱJ?qji<3x3{1AqFNS$FD/i*,S}d wK&NL~q&6FfDZF'ͭm݈ =T44"gG~=)wIw QJO56vՓgĴLr*1KSQ?I\5KG1)fB/߃W}K|w|7Mn|=D{g{x둞?b>WvK3V_xiB`.W;|GHk&lx 1¿Q^7YDf39ck0񗂢%Is+͐wȔ"%滌Qh2vJ9C#h ZJ)#ZljQ]zT[;H0Dԩ?–]ytfV)isU>9"~eIS )lbF4s^3Ow] W5߻?oCC?7};~c{_-ǿe6^8'f@ B̅)Xix1q%Jz\s&}:K3dEڮO,W-fjMTyLf6siUԂl5qGCDdNx%72<4x}O~OXn_[w|շxiI3/^x*4s/+ 6vhN#A"afWd2_E QɜM8YҌ0$jH}X1_5MD4#X_+RdPeDKwr-1¹IlSIKQzו? 5(Ͼg6Kɥ!/G4݂Kem޹R d]MYBdC_9BSh+p4Ce#h?|_mGz·Ox?HO/Ow> >cs4/Df Bx1Xk/8atLT{DYyasgܨ{9Ҏ>EY3ZJ0ԍS2D1f\栐0 c2בvbmF5kƫoa6_J5Ho3--/6uc6UfɧAG%\"ꋺr}>"ڌv"FnĽ&5 OG_ O _g_ğxۿλTi<0 kcuCPf_oN |8lkM8R"+ʐkN8şR4GWtOhFZDžT>&X?Mïlr6%\)XgVf7-e̔\(r񴲿؅\iyp9ʒ"#7Q9Ҍ\{̨!fXGJ>&`U; OIc\Z㼝 ewǿ>/ӻ?{/-ؓkH3 0rPzCwDLm⺿N (VѦ4~iU)6:5s᱈ƣLF HYSvOv >8>92D/ 62QRٜn9aگ~ZIe%i22X@g\R$Lm0[JSĕW<[po HГx᧜C"x}""zV3K[c=#ѯ檈#_<5  }`.>|hI.)i5l{c9̸L,8VHIA5vMɭ5mIulMCDlrՔ}̚]QcCVQTs_c&2Gc27qIf$8)HԆGsU7Rxb$5FǼHlb ú ,to&4ѝ x3/|wg|\W?/D_o~m?O~|/S_]x ̅>2 Sѱ74%w9&f_;dT$ pɹ!kR#i&j9Z %""f̅b.lj DHkTJ&^@KL#lT;bˆAa"r.mw遈0b}[Ma50N]CSxruzKYSzbN)l-Se2ثd"ھd椽NBDLC k}V 5罿I*\m ReF˚KcDޠf@4-\Ɓ2 "?؏ď7}pȷ=~~E}_o[~Wg~ݗ}௼?moo$O7~oĖ^@y\0P^g_>zij9i&S4VlF9(vcljmU2Wє,$(2Oe@7l¥\|cZdnΘ#옯zX=~l8#8sؘ-fǎm Mu,ٍYi˾萷F0&s&)W'ȑfJ+@dT'XfTJn A.c@بEøc"xpg>؏_'|T? 祭1yq]cDcg>[w|?/pg^fcP^o\ \;a~m"E"Dubʸͫ\n/&iToMgLOyaJcIbfaM.b*֬ř_Q  >l)sWu(ti9IJ5K7E4e럸&AFԂ6#LMiSq8#ٔ$0Qf 7_{'T'p_gyw]Ñw|s[>?"%,4̅𺃯Mh"Uq۸=.:r/QrJg~#{/|7%}/}yKGm8v|r 4zs D{ic A CI3reWD:qTg(Y͠S9iw6au5n<OcҭݘכuX.ޙaoḊn>*ب7,c+I#L̶sAlȜȘq *C)x`4/76B(i$T."ͨV饙7C& mz7:O7?EWK;gxߟ7>#{O`qg;Wz7~O{-u ; %̅p"t4cbã`i2g1OلǨ款'egEIRvSqfڇ4fXhNݲhj˿݋z #N mLAI;׌mڝ3'bm[n&uJA zI+]ZfD[-o0ہkF8jesLY6XVW[M-EqcճEF#2<%XS${>_`r/s^̑O{?& iUB= = wmAiq򤱋'f8RS~#f+M=,AH'Gj\\.QTjUF9y塄qlbDBh*FLWw waS'kRLbřڃYivn У4k9XMkFBn(A 34\zw~w~훟Њ^e@@yPB8qdLAoBh3u[b`Mede//MyH1V,E3H<^MS_)(U-3.&raJvg"[-3WO@Y]4/&Sa2Vq:7ilѭ\LMme-eLS$׽N=OΚQ66o{xw%pf ͼ:\s߂e7|oz k- fjjpT{*WuRE&*$w"(mUF(u%I&lNSH`O[ڗSQM|CRK6xb,ئ^sM봔fJhrkmaYVFJ;#SMh~<,"ANUVƵr2r[=9sm6iwS%9^i1)h>/:h3fk;lŦK, K}Ow Lv=>/w?|ӓ_d <`.DD?Eʅoj)`f$#|7A0eWi֯yVx4 ͼ\(<;C >#GG.H1q7F[kMl;7%4U\3pDT&_L}.{aSdsIN< 6:5QU"mާXsH8*|y޻u4ƨtƣyASHL杭Yℂ[]4u%bfTsxF I&JxL_O} Of^`.cǼdAPZlcAV抎I&%J\p0RrH[z./P?7⤁4-2nWCx9x$ ̅͜^5ѵ\gn69j99nȖ{iӅD]Ŕ E1ں4j1xN8kBĭߣZ#LկBƃnr<;Kp E{rg7<yʥ~rK5%Xb'\F Vm|8; FrjW5< o2Z;E7QׄqL/zp\iFisW_OId Z"Qe5RW Ñilc5 6Mex:c1[MչN;/5a%n<κ,_֒.\R)~Xf\G8g"V֩4F=ZD|pZ'ew --*[:cS)r!;eZ]Ô^&[E%\3$mH){M4d7]oVb?,i}R;`6tyG_8H3O  8OS 6~[բEEF.ieDkuEEvt Hu>)o6ܫCSTdрǡ̗ۨl[*]%MD} 1fjׯ MdAKq[R'TTƬD+tuxyR0QcU )N-›&$R-ޯh0"NdZy>9 62N^LIn1CpY,JwD;h3<]/}_n xN LWLw܎e1m Qb|qU*yL!"3a66TI5i f6T-B,ejՂj|p5]Ros,Aqh>[ְIIF9=б|HOګoUc'TFY;b41b53ESȮ֜JtCs0~؅l"z଼Xo_~Y8H3O <ͤ(鞬~.D*}7mb~Og'iʹ@M>ii撊r˴"8 yiFMO5s&w_Xmxڰ`-x)GbݗmEdH{LD.7D:mpb6spN%>81ɑ1&E#Yn\]i.4ڌrgbd%Hƻ_ӗQm<f0 x9nSoEI3* 8Svɷf5/+RvZOa,"]DYDA*MM}.Zk4&tw{]3wQ$Kw Y?.tkSH ͈r/?)zu1wcҝ|%4r UIykdp^>p\e ^iP=K/?9xtyJ\(iCoc{sڬDVJ"ۀD_gc9}SI`髝'Jy|`bM+zl"O!RI,Ρ% <Ȟ25M}6[@r^Mm6&r>AWڬ&o>5Qx,2#5u|Uߨ-[FjlK6D͸":64? d@CN?ϽS'Q>UVjH d۳M7,]QsiȚq0C=[~lrsbK.f)ys7jNfLͣPrk>ܜ#oiιW:V;mv1=Pv.v#I/. \ hi"JQ)&9jZW# R-P'.6C|1*\,rs|hRsL/X.'4Ǧy'>OI3mE;q?5ʚkkltYLvmj\Qaev=7y#81%/ΥJ5V]11.٦Q{K1"E9k>/ 9|cEн<49٩`\dװFf\g.SJ1g٩0"H'evlj@6{-!-SjYa&Lqi"Jה$ދqoNo{Tv.8y'.^ʾ9ȤY*t&V͛~Nؼ #(]rEӒ6wk:D"5Ry]Y2~…&ҌNd` DTY1<f<":ݘ͝E+E0QVENitBj!j*:Ю%Q/줁7V(ѡ>X%PI{ɉDtY@(YkZ\RS.cJ4fFAJ?u璒4q`ʌ\eV!9u;[F{Rکێ}^=Ex1Ƹ6BebLs$h U2$$ls\o3%g9jpfP[خȚ fїᇒHZz..*b+ OUeq[&[ 3;Fmn%aɩD[ɇnS/2! &YX1˝u>9'q-_*&[jBe^JC-Xmֽ(4Y㠘,*N1T4o] )bB]Wüj>CKk |R䄪6~]ƪ4d9)er/HXTVn|)Z"RgOH3 B7j|3ۗ\51gfƛ IYZŵPS3OezXaFii U4dm6l|G|S.^jxbCʹkqZD7rh4"t&eN Hٙ*]%K %oh3`6qV*)53Reke6.\XsZ :n(2T;qئS@Ժ,$#cFAa@>g14$0IO> s \[ɲ=j J;+G](blmy[HMمpt^+F]uZ|:(28DȥBaE"N0vWшPիSdD& %Fof5`"ިUru܈N2fLҌfXe{sH]fdiLnRM,|"nUQj:oQ+G\ϙ1< 8 -BD,6FAoN{X-ʲ*18T2l5mY]L.8G?+Gu{|.%BqΠ0wMQ2.ULܦےpmͽG>x\-U86Tū0ֺSƬ"*L4XV&/5J{JNp窺fu32I( FtVBnp*iilDZQ0N]y0 yG:Oe Kx/PEI)qlf_y`Q JfcXj/(:t5)s-zN;x02O'!pM5*x,Jo۔mPr% L^/.K3\23ګWIo?D~ͦ=P+FM``<f2&u+k\b.7ha)Hk'O0nȌ7gp(k^cvgGFlUSTK~Hmr 1f'4`&6lو1=rLI񬚲 '&-TO1o:J3$Afg *yiH~aڂ4)<Ԇ[P>h1F >d_?b'q(꤁I7/ԚB`8rko8Ih<}"Yw1旺榴Ȭ#ZY@Zj ^7ؽW6%FrLxto~Ƨϯ%!5dޓCuX>$2XoևfHnʦk~.238<:fb%ȐkF6IVP]&&e|0I0^T(k#4"?DRX&֭y o):Y'$STq6#6M6:V,k.%2#ۤ-rjK[`cܷ5M|#F~Z'EҌhvf"KQ(͌G̈r_cUI.:Apm]_G{25K<#a/ƅY׽4H3N%a7,cGUmD^jkN)ƾiRBTkj%!-.nѰҠ@5{K]œ^G<-˰iۅ Eyqf1^}RonO !?8f8 X0[O،PBj Q5R84fdJ.c#?#.=级4< 8ȏ8aFlEƔrU~k=P)ZGciIf F))fD̒?H| p]w٩Z礤]H'ql~~L+t%5)Ef7CMCDD,kfԘ>Gs.Ɍ,vkbQ &'1dD:LiW-Ý)Q0S_'v6<:fDJDF3!r"L)N7u&&s5#ZJdsLGɘ!5|:apl:iFBD!)Y[raR}rkZƥ{Q-R|Ѱ~7st{tb@ڄHNa."~̶ۼR0YJX _v:^~_WLNb7MȩyUTP$6H3 xa'HFe1jLtp2v KZJusD{p\nQ^Š,:aojÞe\! MLf;j)ݼHC+SIjnL7T/l+%23;ӌu)澶-mml4OP)Ul|`-|fb^w ] %5i~%&7 Wlln23IM6Tz"PȼC6ܢ^Y>r]L $9BgH` Zc'qLC=Dx`(\HmmIus|iT(FIWTT4Go),l6ۙ\E3KۍiSޒ*ѕdrʬϤSیʍҰԴ¦RwjԩvwUJ^i"r*:Ln*NؼS+WS r"%74Y\;NM) [m|Y|k&!kx2@pw\fTT4;EF)I38$aaN=IF.. ،jIL0mgQa9>۴SnN̹f7.2msyh;}Y5:12"BkN~vi: N2Jzh,&**ե(w73i檊JiXTK\ꔤՍYmSm7+Rٹ&UC4J GU5 1G>(x3HJn|K`!2Umq| נiI\A, `3vqK؈l0S\&S7Hu'm'ivR*6?rNK9Pޝ w~+ʼq]5fĔ'\Q=Y)-tb4W\Y VThӹSn\q·g :G UkRU#h22|Ft{unpW; (q.2駥?]{ikzr60H3NB/L~>p DOjܵ0drtk;Ug}U)n!=!L:K)4g7^y&/Q/>yȡ{eӈH{fӴ8iC77:tSV~ln3\fՇ!$f"oZMvEes'5u&M͹xgгn}]IBݘ geiVD .k!"**[ЈDy. \3D]6uYZ,br1 46cn2X+j s]M.1?TjsSFni-((~?n;vS5J hи.w[Ԓ#:xa'Zk3$"'q; :R_u YL60*&u'&+u^b lJcQީELQ)CS?92T%jJ"RK&4IiPh 0JۃHI˸_e鶝,$n3S&2uDlN;$^K.:4In۹7ue:fό;7mKI3vEQdOvCm/u +1+wm<ﺙ`Ǖ|NN576`xݫG*.֜[cBC&t*@a]6R]  [᜵P%o.EXsTZw}u8$ɭ_Fِ(Nb᮶I(W$"Ȥ׽yk~ $1fdUe&iszH-e)ys뚊;S%ǀOJ39IADk$V-[es>lnUAƼXd]K FY?lfT5u*lQ(ظu -@Yk}UKcf$uW1tLvkc&G5IWA(BuQiXNT#4f-&'14$.+kPCFXJTېU--lyuے(D=n3[LԸ8Ħդ*^E:uJs mmF-<IlypSQa3[9dbQs5qMTPyB,Tbbkn1d0spo'66av:$a4-墨ɭiH X1!JcyIu Rx4 8{4JDTևdN\cE鎬[>l.'ǃh璆d $HaON")کum%P-#IDqߩ4p{IӤ&S7Y8OƖ]60y*_J7CF̏9&&WdY9DۗZW⧓РtW^<+ĺ.Z{b]4mMNb#o{ּ{W&JSMM\nyn %;Suoy2!5-] H|֌FO_M5#svU & ޘGc⥙EXfq,Bw{۲vI/sM'\7m?6PmI2lڢ%mJS٥ 8IɀNX6lHڭը+~%lhEiɽl!ٷjcv]J^qMjaj^7!`< MSY8XMsIXrT@0ҽqHH[;e$$p͘8(Io/Y|#xek~)(_7fo6ךnK3f_%i[ _X| GsL}}TD>hزI,8r?v8Z9tڵרyHec㚊 fʹE&f;}4 K▸d2:1J'D x 8x!b$YZDLPT4adiR2~)7cI:iTWTjZ3*KxȥХkhfՒ䓥ȧT#_4LdV [XDS#Dd i\kn0h$2rQ>ܫGG'qliI'"Il3ٺ"i6QŴL1_T CkdS5Q.1X6ڻ9mt5fem_E^oNxN5syq3OnsyL|C~PuּĺZÕ3R;[akBjҾ[c.oxptRry]ƽl{'$xZ7^lӪV+#ܷPѱ' 1pQ.Ź)cB:9('!;/G{<:fޙHsr *^s.3{}.š(3Jj7U\~*Pa3(9+Gmݹ-Yi:\ >ff熬 s{ߩ`bjS9seSFڃKMtk3bm}"Us!M3KݹC͋B2]i5f^qvɗgoTۥI3ru *uXQkid#v-fYz'S`_a2?/&m{Xj'J|xD 8aO&]K} ML^m5֚2nK/뒽4Ur½m97~Zߜq+0ʶ{,7҃z|p\zʇMn=s}ԇ^JP$v_`y Fq:f5bG׋Itn1g\>!_ǵQ1饙ŠM dsWl2Tjh>_N59&Vd#w%JJ00%jfIOH3NYyTLaɜ4rm_-{ҠQ]dS^c2)\ @ ^gHY'yscڏse;Q#io`[VIFV!lKwmcv/Qwvk$#C#iFLK}B5wҌzM(ӹ#3].7f 24L\jgU׿(щ,xqZ-yw_46ŦDA9r<^c?xt 8 RĎ8P2e7^nJ.֤ED f9zNn/?y?.2yC܌2u2W/1J~؋2irIt;S/<R^cʌ5C"{?Bd,5QL 7!zn>r 3zM!lf.4 j^I.&HUP`OJUJNh [ܓxi&%Vx 8 myu0Efҡ֕(ic]UN:Ma!qYYlrYjkWNNHh)qyX3]?-v)I{FҌN/T D <I,7@Rbe7]eAIk].8wH'x|;; 9Peb@p"F{ j֥_?)-58yCx~d^~rO7gj픍Ɖ4Zjt}R9^P ^TD|]wc#~!Lc.)k"iIO9"CԉMeiqsIeIS*el2"HS/R2i-y*nإ2a rs7%טM?Mx@nscϭ|xE<[iDv_β]ֱETiZ5}<؛_˳n|7:ѭ|h?;tures3ɵtpΚB F\ uO6fR^!OH3NcD fjך7s 2m1*wjF1m&S.醋\57^ff^ng n'RNOG9^7ܧޚ.?yĈtINm/ݛ2;onNݏK-AeƬrWlkIAp qj<ón_^\d2?Ia0ŚxҧnԳL?ym\ꃇ5yIA0w8G([za6hx 8TUTu^[i pLlt4v 皝4kU:ϣ4ۮqٴ\6tF׏_-ϢĖ\VNKJ|4BtgrYʜ{u|".2L+j`Pdi88=dDF;I3>Uw5| \ݛgu|u (ɜXmD3 Rj3< Ͷi9}~߻i67d7R'ܔ4.V|ftd؎KZ:N︹7n7:$$n39fƶi3W˦yiszZJM(-lۃװ:hmFIo2C;Kmj`*4yfȌJM u-encSau>CiY3"5.&> n[T]Nbو}V r{y fш\/w| w,%wNs٧u6;u}ITNژx,Q2R;u.fMdɽm ([&q&.aQ94թ.iݺfTK[c2y+ 7::gZq$FĽZMike0d &G&--'({Q6kY6?B3FD SLĽReil2ۖc1]Ǣ!iI|Y"}g&ћͲcP'.m:EZQRvnEHSq6Jջ6Ԍ]ղηQ/X4yŽ:2]FE> Z.mVuCRf"~椙aZʒjlDJ7bD~oX&ibjn;f^?*b#2iRkTm ffgXwǃt1$ھ˒/vXuiF؊}PbŜVD  x7ѭpsֵg0ۍwQ h=r9-NɚfoB螭tt{t{7avEWj5yt1̋#1265Hp>v91[RpYU^i%:F$Miۚҡ laqSBQ8c"ĸ_x|;4]Dl=Pe^[I+l8#fs+5xt 8߾ID>-:.f3Q.ʮLfj8giJuUs;l.?:uK,ujϼq%;XkЉcv=[R,4N+Ԙy' CFQ;%J'IєMf~m@ZONJ qja49}^ +1ȫHiskJ˸6;jnjroeXr+U"?$7.pq&7 w(*55n%㰐WCM~gSQj<R P&Y'_% ip.Uc "C/o&6"dδA7A وtd'e jsUp3sd9L,<)sQ^qa-ɫqKuXi¢(;9>LSeJ8sp&" <#PU:}XL[Q M>*N6iކ""SֽP5Ȯ:<ZWse9XZ[ZccVnWS7jCtffgm2g"kCgCKEŁ4cJ3+VVp9I̸:0z_pvF0_+FuT<&fıPvvlk{<=.s&yRhZּ`*l0XˬKÌTJE,_3nXY&"ٳ I"gM~ .r3=ޑaL3GKcJ %N-ao7TGp (7Snawb^?k\QGtEG ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p6 H3g ـ4p67 AX`IENDB`so-vits-svc-npu/spkmix.py000066400000000000000000000014051456161405400157460ustar00rootroot00000000000000# 角色混合轨道 编写规则： # 角色ID : [[起始时间1, 终止时间1, 起始数值1, 起始数值1], [起始时间2, 终止时间2, 起始数值2, 起始数值2]] # 起始时间和前一个的终止时间必须相同，第一个起始时间必须为0，最后一个终止时间必须为1 （时间的范围为0-1） # 全部角色必须填写，不使用的角色填[[0., 1., 0., 0.]]即可 # 融合数值可以随便填，在指定的时间段内从起始数值线性变化为终止数值，内部会自动确保线性组合为1，可以放心使用 spk_mix_map = { 0 : [[0., 0.5, 1, 0.5], [0.5, 1, 0.5, 1]], 1 : [[0., 0.35, 1, 0.5], [0.35, 0.75, 0.75, 1], [0.75, 1, 0.45, 1]], 2 : [[0., 0.35, 1, 0.5], [0.35, 0.75, 0.75, 1], [0.75, 1, 0.45, 1]] }so-vits-svc-npu/train.py000066400000000000000000000337061456161405400155610ustar00rootroot00000000000000import logging import multiprocessing import os import time from rich import progress import mindtorch.torch as torch import mindtorch.torch.distributed as dist import mindtorch.torch.multiprocessing as mp from mindtorch.torch.cuda.amp import GradScaler, autocast from mindtorch.torch.nn import functional as F from mindtorch.torch.utils.data import DataLoader from tensorboardX import SummaryWriter import modules.commons as commons import utils from data_utils import TextAudioCollate, TextAudioSpeakerLoader from models import ( MultiPeriodDiscriminator, SynthesizerTrn, ) from modules.losses import discriminator_loss, feature_loss, generator_loss, kl_loss from modules.mel_processing import mel_spectrogram_torch, spec_to_mel_torch logging.getLogger('matplotlib').setLevel(logging.WARNING) logging.getLogger('numba').setLevel(logging.WARNING) torch.backends.cudnn.benchmark = True global_step = 0 start_time = time.time() # os.environ['TORCH_DISTRIBUTED_DEBUG'] = 'INFO' def main(): hps = utils.get_hparams() n_gpus = 1 os.environ['MASTER_ADDR'] = 'localhost' os.environ['MASTER_PORT'] = hps.train.port run(0,1,hps) def run(rank, n_gpus, hps): global global_step if rank == 0: logger = utils.get_logger(hps.model_dir) logger.hps(hps) utils.check_git_hash(hps.model_dir) writer = SummaryWriter(log_dir=hps.model_dir) writer_eval = SummaryWriter(log_dir=os.path.join(hps.model_dir, "eval")) # for pytorch on win, backend use gloo torch.manual_seed(hps.train.seed) torch.cuda.set_device(rank) collate_fn = TextAudioCollate() all_in_mem = hps.train.all_in_mem # If you have enough memory, turn on this option to avoid disk IO and speed up training. train_dataset = TextAudioSpeakerLoader(hps.data.training_files, hps, all_in_mem=all_in_mem) num_workers = 8 if multiprocessing.cpu_count() > 4 else multiprocessing.cpu_count() if all_in_mem: num_workers = 0 train_loader = DataLoader(train_dataset, num_workers=num_workers, shuffle=False, pin_memory=True,persistent_workers=True, batch_size=hps.train.batch_size, collate_fn=collate_fn) if rank == 0: eval_dataset = TextAudioSpeakerLoader(hps.data.validation_files, hps, all_in_mem=all_in_mem,vol_aug = False) eval_loader = DataLoader(eval_dataset, num_workers=1, shuffle=False, batch_size=1, pin_memory=False, drop_last=False, collate_fn=collate_fn) net_g = SynthesizerTrn( hps.data.filter_length // 2 + 1, hps.train.segment_size // hps.data.hop_length, **hps.model).cuda(rank) net_d = MultiPeriodDiscriminator(hps.model.use_spectral_norm).cuda(rank) optim_g = torch.optim.AdamW( net_g.parameters(), hps.train.learning_rate, betas=hps.train.betas, eps=hps.train.eps) optim_d = torch.optim.AdamW( net_d.parameters(), hps.train.learning_rate, betas=hps.train.betas, eps=hps.train.eps) skip_optimizer = False try: _, _, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(hps.model_dir, "G_*.pth"), net_g, optim_g, skip_optimizer) _, _, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d, skip_optimizer) epoch_str = max(epoch_str, 1) name=utils.latest_checkpoint_path(hps.model_dir, "D_*.pth") global_step=int(name[name.rfind("_")+1:name.rfind(".")])+1 #global_step = (epoch_str - 1) * len(train_loader) except Exception: print("load old checkpoint failed...") epoch_str = 1 global_step = 0 if skip_optimizer: epoch_str = 1 global_step = 0 warmup_epoch = hps.train.warmup_epochs scheduler_g = torch.optim.lr_scheduler.ExponentialLR(optim_g, gamma=hps.train.lr_decay, last_epoch=epoch_str - 2) scheduler_d = torch.optim.lr_scheduler.ExponentialLR(optim_d, gamma=hps.train.lr_decay, last_epoch=epoch_str - 2) scaler = GradScaler(enabled=hps.train.fp16_run) with logger.Progress() as progress: for epoch in range(epoch_str, hps.train.epochs + 1): # set up warm-up learning rate if epoch <= warmup_epoch: for param_group in optim_g.param_groups: param_group['lr'] = hps.train.learning_rate / warmup_epoch * epoch for param_group in optim_d.param_groups: param_group['lr'] = hps.train.learning_rate / warmup_epoch * epoch # training if rank == 0: train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval], progress) else: train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, None], None, None, progress) # update learning rate scheduler_g.step() scheduler_d.step() def train_and_evaluate(rank, epoch, hps, nets, optims, schedulers, scaler, loaders, logger, writers, progress: progress.Progress): net_g, net_d = nets optim_g, optim_d = optims scheduler_g, scheduler_d = schedulers train_loader, eval_loader = loaders if writers is not None: writer, writer_eval = writers half_type = torch.bfloat16 if hps.train.half_type=="bf16" else torch.float16 global global_step net_g.train() net_d.train() enumerated_train_loader = enumerate(train_loader) task = progress.add_task(f"Epoch {epoch}", total=len(train_loader)) for batch_idx, items in enumerated_train_loader: c, f0, spec, y, spk, lengths, uv,volume = items g = spk.cuda(rank, non_blocking=True) spec, y = spec.cuda(rank, non_blocking=True), y.cuda(rank, non_blocking=True) c = c.cuda(rank, non_blocking=True) f0 = f0.cuda(rank, non_blocking=True) uv = uv.cuda(rank, non_blocking=True) lengths = lengths.cuda(rank, non_blocking=True) mel = spec_to_mel_torch( spec, hps.data.filter_length, hps.data.n_mel_channels, hps.data.sampling_rate, hps.data.mel_fmin, hps.data.mel_fmax) with autocast(enabled=hps.train.fp16_run, dtype=half_type): y_hat, ids_slice, z_mask, \ (z, z_p, m_p, logs_p, m_q, logs_q), pred_lf0, norm_lf0, lf0 = net_g(c, f0, uv, spec, g=g, c_lengths=lengths, spec_lengths=lengths,vol = volume) y_mel = commons.slice_segments(mel, ids_slice, hps.train.segment_size // hps.data.hop_length) y_hat_mel = mel_spectrogram_torch( y_hat.squeeze(1), hps.data.filter_length, hps.data.n_mel_channels, hps.data.sampling_rate, hps.data.hop_length, hps.data.win_length, hps.data.mel_fmin, hps.data.mel_fmax ) y = commons.slice_segments(y, ids_slice * hps.data.hop_length, hps.train.segment_size) # slice # Discriminator y_d_hat_r, y_d_hat_g, _, _ = net_d(y, y_hat.detach()) with autocast(enabled=False, dtype=half_type): loss_disc, losses_disc_r, losses_disc_g = discriminator_loss(y_d_hat_r, y_d_hat_g) loss_disc_all = loss_disc optim_d.zero_grad() scaler.scale(loss_disc_all).backward() scaler.unscale_(optim_d) grad_norm_d = commons.clip_grad_value_(net_d.parameters(), None) scaler.step(optim_d) with autocast(enabled=hps.train.fp16_run, dtype=half_type): # Generator y_d_hat_r, y_d_hat_g, fmap_r, fmap_g = net_d(y, y_hat) with autocast(enabled=False, dtype=half_type): loss_mel = F.l1_loss(y_mel, y_hat_mel) * hps.train.c_mel loss_kl = kl_loss(z_p, logs_q, m_p, logs_p, z_mask) * hps.train.c_kl loss_fm = feature_loss(fmap_r, fmap_g) loss_gen, losses_gen = generator_loss(y_d_hat_g) loss_lf0 = F.mse_loss(pred_lf0, lf0) if net_g.module.use_automatic_f0_prediction else 0 loss_gen_all = loss_gen + loss_fm + loss_mel + loss_kl + loss_lf0 optim_g.zero_grad() scaler.scale(loss_gen_all).backward() scaler.unscale_(optim_g) grad_norm_g = commons.clip_grad_value_(net_g.parameters(), None) scaler.step(optim_g) scaler.update() if rank == 0: if global_step % hps.train.log_interval == 0: lr = optim_g.param_groups[0]['lr'] losses = [loss_disc, loss_gen, loss_fm, loss_mel, loss_kl] reference_loss=0 for i in losses: reference_loss += i logger.info('Train Epoch: {} [{:.0f}%]'.format( epoch, 100. * batch_idx / len(train_loader))) logger.info(f"Losses: {[x.item() for x in losses]}, step: {global_step}, lr: {lr}, reference_loss: {reference_loss}") scalar_dict = {"loss/g/total": loss_gen_all, "loss/d/total": loss_disc_all, "learning_rate": lr, "grad_norm_d": grad_norm_d, "grad_norm_g": grad_norm_g} scalar_dict.update({"loss/g/fm": loss_fm, "loss/g/mel": loss_mel, "loss/g/kl": loss_kl, "loss/g/lf0": loss_lf0}) image_dict = { "slice/mel_org": utils.plot_spectrogram_to_numpy(y_mel[0].data.cpu().numpy()), "slice/mel_gen": utils.plot_spectrogram_to_numpy(y_hat_mel[0].data.cpu().numpy()), "all/mel": utils.plot_spectrogram_to_numpy(mel[0].data.cpu().numpy()) } if net_g.module.use_automatic_f0_prediction: image_dict.update({ "all/lf0": utils.plot_data_to_numpy(lf0[0, 0, :].cpu().numpy(), pred_lf0[0, 0, :].detach().cpu().numpy()), "all/norm_lf0": utils.plot_data_to_numpy(lf0[0, 0, :].cpu().numpy(), norm_lf0[0, 0, :].detach().cpu().numpy()) }) utils.summarize( writer=writer, global_step=global_step, images=image_dict, scalars=scalar_dict ) # 达到保存步数或者 stop 文件存在 if global_step % hps.train.eval_interval == 0 or os.path.exists("stop.txt"): if os.path.exists("stop.txt"): logger.info("stop.txt found, stop training") evaluate(hps, net_g, eval_loader, writer_eval) utils.save_checkpoint(net_g, optim_g, hps.train.learning_rate, epoch, os.path.join(hps.model_dir, "G_{}.pth".format(global_step))) utils.save_checkpoint(net_d, optim_d, hps.train.learning_rate, epoch, os.path.join(hps.model_dir, "D_{}.pth".format(global_step))) keep_ckpts = getattr(hps.train, 'keep_ckpts', 0) if keep_ckpts > 0: utils.clean_checkpoints(path_to_models=hps.model_dir, n_ckpts_to_keep=keep_ckpts, sort_by_time=True) if os.path.exists("stop.txt"): logger.info("good bye!") os.remove("stop.txt") os._exit(0) global_step += 1 progress.advance(task) progress.remove_task(task) if rank == 0: global start_time now = time.time() duration = format(now - start_time, '.2f') # 这里原本是 durtaion，让我看看是谁拼错了（ logger.info(f'Epoch: {epoch} finished, cost {duration} s, {round(float(duration)/len(train_loader),3)}s per batch') start_time = now def evaluate(hps, generator, eval_loader, writer_eval): generator.eval() image_dict = {} audio_dict = {} with torch.no_grad(): for batch_idx, items in enumerate(eval_loader): c, f0, spec, y, spk, _, uv,volume = items g = spk[:1].cuda(0) spec, y = spec[:1].cuda(0), y[:1].cuda(0) c = c[:1].cuda(0) f0 = f0[:1].cuda(0) uv= uv[:1].cuda(0) if volume is not None: volume = volume[:1].cuda(0) mel = spec_to_mel_torch( spec, hps.data.filter_length, hps.data.n_mel_channels, hps.data.sampling_rate, hps.data.mel_fmin, hps.data.mel_fmax) y_hat,_ = generator.module.infer(c, f0, uv, g=g,vol = volume) y_hat_mel = mel_spectrogram_torch( y_hat.squeeze(1).float(), hps.data.filter_length, hps.data.n_mel_channels, hps.data.sampling_rate, hps.data.hop_length, hps.data.win_length, hps.data.mel_fmin, hps.data.mel_fmax ) audio_dict.update({ f"gen/audio_{batch_idx}": y_hat[0], f"gt/audio_{batch_idx}": y[0] }) image_dict.update({ "gen/mel": utils.plot_spectrogram_to_numpy(y_hat_mel[0].cpu().numpy()), "gt/mel": utils.plot_spectrogram_to_numpy(mel[0].cpu().numpy()) }) utils.summarize( writer=writer_eval, global_step=global_step, images=image_dict, audios=audio_dict, audio_sampling_rate=hps.data.sampling_rate ) generator.train() if __name__ == "__main__": main()so-vits-svc-npu/train_diff.py000066400000000000000000000050631456161405400165440ustar00rootroot00000000000000import argparse import mindtorch.torch as torch from loguru import logger from torch.optim import lr_scheduler from diffusion.data_loaders import get_data_loaders from diffusion.logger import utils from diffusion.solver import train from diffusion.unit2mel import Unit2Mel from diffusion.vocoder import Vocoder def parse_args(args=None, namespace=None): """Parse command-line arguments.""" parser = argparse.ArgumentParser() parser.add_argument( "-c", "--config", type=str, required=True, help="path to the config file") return parser.parse_args(args=args, namespace=namespace) if __name__ == '__main__': # parse commands cmd = parse_args() # load config args = utils.load_config(cmd.config) logger.info(' > config:'+ cmd.config) logger.info(' > exp:'+ args.env.expdir) # load vocoder vocoder = Vocoder(args.vocoder.type, args.vocoder.ckpt, device=args.device) # load model model = Unit2Mel( args.data.encoder_out_channels, args.model.n_spk, args.model.use_pitch_aug, vocoder.dimension, args.model.n_layers, args.model.n_chans, args.model.n_hidden, args.model.timesteps, args.model.k_step_max ) logger.info(f' > Now model timesteps is {model.timesteps}, and k_step_max is {model.k_step_max}') # load parameters optimizer = torch.optim.AdamW(model.parameters()) initial_global_step, model, optimizer = utils.load_model(args.env.expdir, model, optimizer, device=args.device) for param_group in optimizer.param_groups: param_group['initial_lr'] = args.train.lr param_group['lr'] = args.train.lr * (args.train.gamma ** max(((initial_global_step-2)//args.train.decay_step),0) ) param_group['weight_decay'] = args.train.weight_decay scheduler = lr_scheduler.StepLR(optimizer, step_size=args.train.decay_step, gamma=args.train.gamma,last_epoch=initial_global_step-2) # device if args.device == 'cuda': torch.cuda.set_device(args.env.gpu_id) model.to(args.device) for state in optimizer.state.values(): for k, v in state.items(): if torch.is_tensor(v): state[k] = v.to(args.device) # datas loader_train, loader_valid = get_data_loaders(args, whole_audio=False) # run train(args, initial_global_step, model, optimizer, scheduler, vocoder, loader_train, loader_valid) so-vits-svc-npu/train_index.py000066400000000000000000000015511456161405400167410ustar00rootroot00000000000000import argparse import os import pickle import utils if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( "--root_dir", type=str, default="dataset/44k", help="path to root dir" ) parser.add_argument('-c', '--config', type=str, default="./configs/config.json", help='JSON file for configuration') parser.add_argument( "--output_dir", type=str, default="logs/44k", help="path to output dir" ) args = parser.parse_args() hps = utils.get_hparams_from_file(args.config) spk_dic = hps.spk result = {} for k,v in spk_dic.items(): print(f"now, index {k} feature...") index = utils.train_index(k,args.root_dir) result[v] = index with open(os.path.join(args.output_dir,"feature_and_index.pkl"),"wb") as f: pickle.dump(result,f)so-vits-svc-npu/trained/000077500000000000000000000000001456161405400155075ustar00rootroot00000000000000so-vits-svc-npu/trained/put_trained_checkpoints_here000066400000000000000000000000001456161405400233330ustar00rootroot00000000000000so-vits-svc-npu/utils.py000066400000000000000000000507111456161405400155770ustar00rootroot00000000000000import argparse import glob import json import logging import os import re import subprocess import sys import traceback from multiprocessing import cpu_count import faiss import librosa import numpy as np import mindtorch.torch as torch from scipy.io.wavfile import read from sklearn.cluster import MiniBatchKMeans from mindtorch.torch.nn import functional as F import logger MATPLOTLIB_FLAG = False logging.basicConfig(stream=sys.stdout, level=logging.WARN) # logger = logging f0_bin = 256 f0_max = 1100.0 f0_min = 50.0 f0_mel_min = 1127 * np.log(1 + f0_min / 700) f0_mel_max = 1127 * np.log(1 + f0_max / 700) def normalize_f0(f0, x_mask, uv, random_scale=True): # calculate means based on x_mask uv_sum = torch.sum(uv, dim=1, keepdim=True) uv_sum[uv_sum == 0] = 9999 means = torch.sum(f0[:, 0, :] * uv, dim=1, keepdim=True) / uv_sum if random_scale: factor = torch.Tensor(f0.shape[0], 1).uniform_(0.8, 1.2).to(f0.device) else: factor = torch.ones(f0.shape[0], 1).to(f0.device) # normalize f0 based on means and factor f0_norm = (f0 - means.unsqueeze(-1)) * factor.unsqueeze(-1) if torch.isnan(f0_norm).any(): exit(0) return f0_norm * x_mask def plot_data_to_numpy(x, y): global MATPLOTLIB_FLAG if not MATPLOTLIB_FLAG: import matplotlib matplotlib.use("Agg") MATPLOTLIB_FLAG = True mpl_logger = logging.getLogger('matplotlib') mpl_logger.setLevel(logging.WARNING) import matplotlib.pylab as plt import numpy as np fig, ax = plt.subplots(figsize=(10, 2)) plt.plot(x) plt.plot(y) plt.tight_layout() fig.canvas.draw() data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='') data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,)) plt.close() return data def f0_to_coarse(f0): f0_mel = 1127 * (1 + f0 / 700).log() a = (f0_bin - 2) / (f0_mel_max - f0_mel_min) b = f0_mel_min * a - 1. f0_mel = torch.where(f0_mel > 0, f0_mel * a - b, f0_mel) # torch.clip_(f0_mel, min=1., max=float(f0_bin - 1)) f0_coarse = torch.round(f0_mel).long() f0_coarse = f0_coarse * (f0_coarse > 0) f0_coarse = f0_coarse + ((f0_coarse < 1) * 1) f0_coarse = f0_coarse * (f0_coarse < f0_bin) f0_coarse = f0_coarse + ((f0_coarse >= f0_bin) * (f0_bin - 1)) return f0_coarse def get_content(cmodel, y): with torch.no_grad(): c = cmodel.extract_features(y.squeeze(1))[0] c = c.transpose(1, 2) return c def get_f0_predictor(f0_predictor,hop_length,sampling_rate,**kargs): if f0_predictor == "pm": from modules.F0Predictor.PMF0Predictor import PMF0Predictor f0_predictor_object = PMF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate) elif f0_predictor == "crepe": from modules.F0Predictor.CrepeF0Predictor import CrepeF0Predictor f0_predictor_object = CrepeF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate,device=kargs["device"],threshold=kargs["threshold"]) elif f0_predictor == "harvest": from modules.F0Predictor.HarvestF0Predictor import HarvestF0Predictor f0_predictor_object = HarvestF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate) elif f0_predictor == "dio": from modules.F0Predictor.DioF0Predictor import DioF0Predictor f0_predictor_object = DioF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate) elif f0_predictor == "rmvpe": from modules.F0Predictor.RMVPEF0Predictor import RMVPEF0Predictor f0_predictor_object = RMVPEF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate,dtype=torch.float32 ,device=kargs["device"],threshold=kargs["threshold"]) elif f0_predictor == "fcpe": from modules.F0Predictor.FCPEF0Predictor import FCPEF0Predictor f0_predictor_object = FCPEF0Predictor(hop_length=hop_length,sampling_rate=sampling_rate,dtype=torch.float32 ,device=kargs["device"],threshold=kargs["threshold"]) else: raise Exception("Unknown f0 predictor") return f0_predictor_object def get_speech_encoder(speech_encoder,device=None,log=True,**kargs): if speech_encoder == "vec768l12": from vencoder.ContentVec768L12 import ContentVec768L12 speech_encoder_object = ContentVec768L12(device = device,log=log) elif speech_encoder == "vec256l9": from vencoder.ContentVec256L9 import ContentVec256L9 speech_encoder_object = ContentVec256L9(device = device,log=log) elif speech_encoder == "vec256l9-onnx": from vencoder.ContentVec256L9_Onnx import ContentVec256L9_Onnx speech_encoder_object = ContentVec256L9_Onnx(device = device,log=log) elif speech_encoder == "vec256l12-onnx": from vencoder.ContentVec256L12_Onnx import ContentVec256L12_Onnx speech_encoder_object = ContentVec256L12_Onnx(device = device,log=log) elif speech_encoder == "vec768l9-onnx": from vencoder.ContentVec768L9_Onnx import ContentVec768L9_Onnx speech_encoder_object = ContentVec768L9_Onnx(device = device,log=log) elif speech_encoder == "vec768l12-onnx": from vencoder.ContentVec768L12_Onnx import ContentVec768L12_Onnx speech_encoder_object = ContentVec768L12_Onnx(device = device,log=log) elif speech_encoder == "hubertsoft-onnx": from vencoder.HubertSoft_Onnx import HubertSoft_Onnx speech_encoder_object = HubertSoft_Onnx(device = device,log=log) elif speech_encoder == "hubertsoft": from vencoder.HubertSoft import HubertSoft speech_encoder_object = HubertSoft(device = device,log=log) elif speech_encoder == "whisper-ppg": from vencoder.WhisperPPG import WhisperPPG speech_encoder_object = WhisperPPG(device = device,log=log) elif speech_encoder == "cnhubertlarge": from vencoder.CNHubertLarge import CNHubertLarge speech_encoder_object = CNHubertLarge(device = device,log=log) elif speech_encoder == "dphubert": from vencoder.DPHubert import DPHubert speech_encoder_object = DPHubert(device = device,log=log) elif speech_encoder == "whisper-ppg-large": from vencoder.WhisperPPGLarge import WhisperPPGLarge speech_encoder_object = WhisperPPGLarge(device = device,log=log) elif speech_encoder == "wavlmbase+": from vencoder.WavLMBasePlus import WavLMBasePlus speech_encoder_object = WavLMBasePlus(device = device,log=log) else: raise Exception("Unknown speech encoder") return speech_encoder_object def load_checkpoint(checkpoint_path, model, optimizer=None, skip_optimizer=False): assert os.path.isfile(checkpoint_path) checkpoint_dict = torch.load(checkpoint_path, map_location='cpu') iteration = checkpoint_dict['iteration'] learning_rate = checkpoint_dict['learning_rate'] if optimizer is not None and not skip_optimizer and checkpoint_dict['optimizer'] is not None: optimizer.load_state_dict(checkpoint_dict['optimizer']) saved_state_dict = checkpoint_dict['model'] model = model.to(list(saved_state_dict.values())[0].dtype) if hasattr(model, 'module'): state_dict = model.module.state_dict() else: state_dict = model.state_dict() new_state_dict = {} for k, v in state_dict.items(): try: # assert "dec" in k or "disc" in k # print("load", k) new_state_dict[k] = saved_state_dict[k] assert saved_state_dict[k].shape == v.shape, (saved_state_dict[k].shape, v.shape) except Exception: if "enc_q" not in k or "emb_g" not in k: logger.error("%s is not in the checkpoint,please check your checkpoint.If you're using pretrain model,just ignore this warning." % k) logger.info("%s is not in the checkpoint" % k) new_state_dict[k] = v if hasattr(model, 'module'): model.module.load_state_dict(new_state_dict) else: model.load_state_dict(new_state_dict) logger.info("Loaded checkpoint '{}' (iteration {})".format( checkpoint_path, iteration)) return model, optimizer, learning_rate, iteration def save_checkpoint(model, optimizer, learning_rate, iteration, checkpoint_path): logger.info("Saving model and optimizer state at iteration {} to {}".format( iteration, checkpoint_path)) if hasattr(model, 'module'): state_dict = model.module.state_dict() else: state_dict = model.state_dict() torch.save({'model': state_dict, 'iteration': iteration, 'optimizer': optimizer.state_dict(), 'learning_rate': learning_rate}, checkpoint_path) def clean_checkpoints(path_to_models='logs/44k/', n_ckpts_to_keep=2, sort_by_time=True): """Freeing up space by deleting saved ckpts Arguments: path_to_models -- Path to the model directory n_ckpts_to_keep -- Number of ckpts to keep, excluding G_0.pth and D_0.pth sort_by_time -- True -> chronologically delete ckpts False -> lexicographically delete ckpts """ ckpts_files = [f for f in os.listdir(path_to_models) if os.path.isfile(os.path.join(path_to_models, f))] def name_key(_f): return int(re.compile("._(\\d+)\\.pth").match(_f).group(1)) def time_key(_f): return os.path.getmtime(os.path.join(path_to_models, _f)) sort_key = time_key if sort_by_time else name_key def x_sorted(_x): return sorted([f for f in ckpts_files if f.startswith(_x) and not f.endswith("_0.pth")], key=sort_key) to_del = [os.path.join(path_to_models, fn) for fn in (x_sorted('G')[:-n_ckpts_to_keep] + x_sorted('D')[:-n_ckpts_to_keep])] def del_info(fn): return logger.info(f".. Free up space by deleting ckpt {fn}") def del_routine(x): return [os.remove(x), del_info(x)] [del_routine(fn) for fn in to_del] def summarize(writer, global_step, scalars={}, histograms={}, images={}, audios={}, audio_sampling_rate=22050): for k, v in scalars.items(): writer.add_scalar(k, v, global_step) for k, v in histograms.items(): writer.add_histogram(k, v, global_step) for k, v in images.items(): writer.add_image(k, v, global_step, dataformats='HWC') for k, v in audios.items(): writer.add_audio(k, v, global_step, audio_sampling_rate) def latest_checkpoint_path(dir_path, regex="G_*.pth"): f_list = glob.glob(os.path.join(dir_path, regex)) f_list.sort(key=lambda f: int("".join(filter(str.isdigit, f)))) x = f_list[-1] print(x) return x def plot_spectrogram_to_numpy(spectrogram): global MATPLOTLIB_FLAG if not MATPLOTLIB_FLAG: import matplotlib matplotlib.use("Agg") MATPLOTLIB_FLAG = True mpl_logger = logging.getLogger('matplotlib') mpl_logger.setLevel(logging.WARNING) import matplotlib.pylab as plt import numpy as np fig, ax = plt.subplots(figsize=(10,2)) im = ax.imshow(spectrogram, aspect="auto", origin="lower", interpolation='none') plt.colorbar(im, ax=ax) plt.xlabel("Frames") plt.ylabel("Channels") plt.tight_layout() fig.canvas.draw() data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='') data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,)) plt.close() return data def plot_alignment_to_numpy(alignment, info=None): global MATPLOTLIB_FLAG if not MATPLOTLIB_FLAG: import matplotlib matplotlib.use("Agg") MATPLOTLIB_FLAG = True mpl_logger = logging.getLogger('matplotlib') mpl_logger.setLevel(logging.WARNING) import matplotlib.pylab as plt import numpy as np fig, ax = plt.subplots(figsize=(6, 4)) im = ax.imshow(alignment.transpose(), aspect='auto', origin='lower', interpolation='none') fig.colorbar(im, ax=ax) xlabel = 'Decoder timestep' if info is not None: xlabel += '\n\n' + info plt.xlabel(xlabel) plt.ylabel('Encoder timestep') plt.tight_layout() fig.canvas.draw() data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='') data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,)) plt.close() return data def load_wav_to_torch(full_path): sampling_rate, data = read(full_path) return torch.FloatTensor(data.astype(np.float32)), sampling_rate def load_filepaths_and_text(filename, split="|"): with open(filename, encoding='utf-8') as f: filepaths_and_text = [line.strip().split(split) for line in f] return filepaths_and_text def get_hparams(init=True): parser = argparse.ArgumentParser() parser.add_argument('-c', '--config', type=str, default="./configs/config.json", help='JSON file for configuration') parser.add_argument('-m', '--model', type=str, required=True, help='Model name') args = parser.parse_args() model_dir = os.path.join("./logs", args.model) if not os.path.exists(model_dir): os.makedirs(model_dir) config_path = args.config config_save_path = os.path.join(model_dir, "config.json") if init: with open(config_path, "r") as f: data = f.read() with open(config_save_path, "w") as f: f.write(data) else: with open(config_save_path, "r") as f: data = f.read() config = json.loads(data) hparams = HParams(**config) hparams.model_dir = model_dir return hparams def get_hparams_from_dir(model_dir): config_save_path = os.path.join(model_dir, "config.json") with open(config_save_path, "r") as f: data = f.read() config = json.loads(data) hparams =HParams(**config) hparams.model_dir = model_dir return hparams def get_hparams_from_file(config_path, infer_mode = False): with open(config_path, "r") as f: data = f.read() config = json.loads(data) hparams =HParams(**config) if not infer_mode else InferHParams(**config) return hparams def check_git_hash(model_dir): source_dir = os.path.dirname(os.path.realpath(__file__)) if not os.path.exists(os.path.join(source_dir, ".git")): logger.warn("{} is not a git repository, therefore hash value comparison will be ignored.".format( source_dir )) return cur_hash = subprocess.getoutput("git rev-parse HEAD") path = os.path.join(model_dir, "githash") if os.path.exists(path): saved_hash = open(path).read() if saved_hash != cur_hash: logger.warn("git hash values are different. {}(saved) != {}(current)".format( saved_hash[:8], cur_hash[:8])) else: open(path, "w").write(cur_hash) def get_logger(model_dir, filename="train.log"): import logger logger.addLogger(os.path.join(model_dir, filename)) # logger = logging.getLogger(os.path.basename(model_dir)) # logger.setLevel(logging.DEBUG) # formatter = logging.Formatter("%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s") # if not os.path.exists(model_dir): # os.makedirs(model_dir) # h = logging.FileHandler(os.path.join(model_dir, filename)) # h.setLevel(logging.DEBUG) # h.setFormatter(formatter) # logger.addHandler(h) return logger def repeat_expand_2d(content, target_len, mode = 'left'): # content : [h, t] return repeat_expand_2d_left(content, target_len) if mode == 'left' else repeat_expand_2d_other(content, target_len, mode) def repeat_expand_2d_left(content, target_len): # content : [h, t] src_len = content.shape[-1] target = torch.zeros([content.shape[0], target_len], dtype=torch.float).to(content.device) temp = torch.arange(src_len+1) * target_len / src_len current_pos = 0 for i in range(target_len): if i < temp[current_pos+1]: target[:, i] = content[:, current_pos] else: current_pos += 1 target[:, i] = content[:, current_pos] return target # mode : 'nearest'| 'linear'| 'bilinear'| 'bicubic'| 'trilinear'| 'area' def repeat_expand_2d_other(content, target_len, mode = 'nearest'): # content : [h, t] content = content[None,:,:] target = F.interpolate(content,size=target_len,mode=mode)[0] return target def mix_model(model_paths,mix_rate,mode): mix_rate = torch.FloatTensor(mix_rate)/100 model_tem = torch.load(model_paths[0]) models = [torch.load(path)["model"] for path in model_paths] if mode == 0: mix_rate = F.softmax(mix_rate,dim=0) for k in model_tem["model"].keys(): model_tem["model"][k] = torch.zeros_like(model_tem["model"][k]) for i,model in enumerate(models): model_tem["model"][k] += model[k]*mix_rate[i] torch.save(model_tem,os.path.join(os.path.curdir,"output.pth")) return os.path.join(os.path.curdir,"output.pth") def change_rms(data1, sr1, data2, sr2, rate): # 1是输入音频，2是输出音频,rate是2的占比 from RVC # print(data1.max(),data2.max()) rms1 = librosa.feature.rms( y=data1, frame_length=sr1 // 2 * 2, hop_length=sr1 // 2 ) # 每半秒一个点 rms2 = librosa.feature.rms(y=data2.detach().cpu().numpy(), frame_length=sr2 // 2 * 2, hop_length=sr2 // 2) rms1 = torch.from_numpy(rms1).to(data2.device) rms1 = F.interpolate( rms1.unsqueeze(0), size=data2.shape[0], mode="linear" ).squeeze() rms2 = torch.from_numpy(rms2).to(data2.device) rms2 = F.interpolate( rms2.unsqueeze(0), size=data2.shape[0], mode="linear" ).squeeze() rms2 = torch.max(rms2, torch.zeros_like(rms2) + 1e-6) data2 *= ( torch.pow(rms1, torch.tensor(1 - rate)) * torch.pow(rms2, torch.tensor(rate - 1)) ) return data2 def train_index(spk_name,root_dir = "dataset/44k/"): #from: RVC https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI n_cpu = cpu_count() print("The feature index is constructing.") exp_dir = os.path.join(root_dir,spk_name) listdir_res = [] for file in os.listdir(exp_dir): if ".wav.soft.pt" in file: listdir_res.append(os.path.join(exp_dir,file)) if len(listdir_res) == 0: raise Exception("You need to run preprocess_hubert_f0.py!") npys = [] for name in sorted(listdir_res): phone = torch.load(name)[0].transpose(-1,-2).numpy() npys.append(phone) big_npy = np.concatenate(npys, 0) big_npy_idx = np.arange(big_npy.shape[0]) np.random.shuffle(big_npy_idx) big_npy = big_npy[big_npy_idx] if big_npy.shape[0] > 2e5: # if(1): info = "Trying doing kmeans %s shape to 10k centers." % big_npy.shape[0] print(info) try: big_npy = ( MiniBatchKMeans( n_clusters=10000, verbose=True, batch_size=256 * n_cpu, compute_labels=False, init="random", ) .fit(big_npy) .cluster_centers_ ) except Exception: info = traceback.format_exc() print(info) n_ivf = min(int(16 * np.sqrt(big_npy.shape[0])), big_npy.shape[0] // 39) index = faiss.index_factory(big_npy.shape[1] , "IVF%s,Flat" % n_ivf) index_ivf = faiss.extract_index_ivf(index) # index_ivf.nprobe = 1 index.train(big_npy) batch_size_add = 8192 for i in range(0, big_npy.shape[0], batch_size_add): index.add(big_npy[i : i + batch_size_add]) # faiss.write_index( # index, # f"added_{spk_name}.index" # ) print("Successfully build index") return index class HParams(): def __init__(self, **kwargs): for k, v in kwargs.items(): if type(v) == dict: v = HParams(**v) self[k] = v def keys(self): return self.__dict__.keys() def items(self): return self.__dict__.items() def values(self): return self.__dict__.values() def __len__(self): return len(self.__dict__) def __getitem__(self, key): return getattr(self, key) def __setitem__(self, key, value): return setattr(self, key, value) def __contains__(self, key): return key in self.__dict__ def __repr__(self): return self.__dict__.__repr__() def get(self,index): return self.__dict__.get(index) class InferHParams(HParams): def __init__(self, **kwargs): for k, v in kwargs.items(): if type(v) == dict: v = InferHParams(**v) self[k] = v def __getattr__(self,index): return self.get(index) class Volume_Extractor: def __init__(self, hop_size = 512): self.hop_size = hop_size def extract(self, audio): # audio: 2d tensor array if not isinstance(audio,torch.Tensor): audio = torch.Tensor(audio) n_frames = int(audio.size(-1) // self.hop_size) audio2 = audio ** 2 audio2 = torch.nn.functional.pad(audio2, (int(self.hop_size // 2), int((self.hop_size + 1) // 2)), mode = 'reflect') volume = torch.nn.functional.unfold(audio2[:,None,None,:],(1,self.hop_size),stride=self.hop_size)[:,:,:n_frames].mean(dim=1)[0] volume = torch.sqrt(volume) return volume so-vits-svc-npu/vdecoder/000077500000000000000000000000001456161405400156545ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/__init__.py000066400000000000000000000000001456161405400177530ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/hifigan/000077500000000000000000000000001456161405400172615ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/hifigan/env.py000066400000000000000000000006121456161405400204220ustar00rootroot00000000000000import os import shutil class AttrDict(dict): def __init__(self, *args, **kwargs): super(AttrDict, self).__init__(*args, **kwargs) self.__dict__ = self def build_env(config, config_name, path): t_path = os.path.join(path, config_name) if config != t_path: os.makedirs(path, exist_ok=True) shutil.copyfile(config, os.path.join(path, config_name)) so-vits-svc-npu/vdecoder/hifigan/models.py000066400000000000000000000525361456161405400211310ustar00rootroot00000000000000import json import os import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn import AvgPool1d, Conv1d, Conv2d, ConvTranspose1d from mindtorch.torch.nn.utils import remove_weight_norm, spectral_norm, weight_norm from .env import AttrDict from .utils import get_padding, init_weights LRELU_SLOPE = 0.1 def load_model(model_path, device='cuda'): config_file = os.path.join(os.path.split(model_path)[0], 'config.json') with open(config_file) as f: data = f.read() global h json_config = json.loads(data) h = AttrDict(json_config) generator = Generator(h).to(device) cp_dict = torch.load(model_path) generator.load_state_dict(cp_dict['generator']) generator.eval() generator.remove_weight_norm() del cp_dict return generator, h class ResBlock1(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3, 5)): super(ResBlock1, self).__init__() self.h = h self.convs1 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[2], padding=get_padding(kernel_size, dilation[2]))) ]) self.convs1.apply(init_weights) self.convs2 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))) ]) self.convs2.apply(init_weights) def forward(self, x): for c1, c2 in zip(self.convs1, self.convs2): xt = F.leaky_relu(x, LRELU_SLOPE) xt = c1(xt) xt = F.leaky_relu(xt, LRELU_SLOPE) xt = c2(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs1: remove_weight_norm(l) for l in self.convs2: remove_weight_norm(l) class ResBlock2(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3)): super(ResBlock2, self).__init__() self.h = h self.convs = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))) ]) self.convs.apply(init_weights) def forward(self, x): for c in self.convs: xt = F.leaky_relu(x, LRELU_SLOPE) xt = c(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs: remove_weight_norm(l) def padDiff(x): return F.pad(F.pad(x, (0,0,-1,1), 'constant', 0) - x, (0,0,0,-1), 'constant', 0) class SineGen(torch.nn.Module): """ Definition of sine generator SineGen(samp_rate, harmonic_num = 0, sine_amp = 0.1, noise_std = 0.003, voiced_threshold = 0, flag_for_pulse=False) samp_rate: sampling rate in Hz harmonic_num: number of harmonic overtones (default 0) sine_amp: amplitude of sine-wavefrom (default 0.1) noise_std: std of Gaussian noise (default 0.003) voiced_thoreshold: F0 threshold for U/V classification (default 0) flag_for_pulse: this SinGen is used inside PulseGen (default False) Note: when flag_for_pulse is True, the first time step of a voiced segment is always sin(np.pi) or cos(0) """ def __init__(self, samp_rate, harmonic_num=0, sine_amp=0.1, noise_std=0.003, voiced_threshold=0, flag_for_pulse=False): super(SineGen, self).__init__() self.sine_amp = sine_amp self.noise_std = noise_std self.harmonic_num = harmonic_num self.dim = self.harmonic_num + 1 self.sampling_rate = samp_rate self.voiced_threshold = voiced_threshold self.flag_for_pulse = flag_for_pulse self.onnx = False def _f02uv(self, f0): # generate uv signal uv = (f0 > self.voiced_threshold).type(torch.float32) return uv def _f02sine(self, f0_values): """ f0_values: (batchsize, length, dim) where dim indicates fundamental tone and overtones """ # convert to F0 in rad. The interger part n can be ignored # because 2 * np.pi * n doesn't affect phase rad_values = (f0_values / self.sampling_rate) % 1 # initial phase noise (no noise for fundamental component) rand_ini = torch.rand(f0_values.shape[0], f0_values.shape[2], \ device=f0_values.device) rand_ini[:, 0] = 0 rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini # instantanouse phase sine[t] = sin(2*pi \sum_i=1 ^{t} rad) if not self.flag_for_pulse: # for normal case # To prevent torch.cumsum numerical overflow, # it is necessary to add -1 whenever \sum_k=1^n rad_value_k > 1. # Buffer tmp_over_one_idx indicates the time step to add -1. # This will not change F0 of sine because (x-1) * 2*pi = x * 2*pi tmp_over_one = torch.cumsum(rad_values, 1) % 1 tmp_over_one_idx = (padDiff(tmp_over_one)) < 0 cumsum_shift = torch.zeros_like(rad_values) cumsum_shift[:, 1:, :] = tmp_over_one_idx * -1.0 sines = torch.sin(torch.cumsum(rad_values + cumsum_shift, dim=1) * 2 * np.pi) else: # If necessary, make sure that the first time step of every # voiced segments is sin(pi) or cos(0) # This is used for pulse-train generation # identify the last time step in unvoiced segments uv = self._f02uv(f0_values) uv_1 = torch.roll(uv, shifts=-1, dims=1) uv_1[:, -1, :] = 1 u_loc = (uv < 1) * (uv_1 > 0) # get the instantanouse phase tmp_cumsum = torch.cumsum(rad_values, dim=1) # different batch needs to be processed differently for idx in range(f0_values.shape[0]): temp_sum = tmp_cumsum[idx, u_loc[idx, :, 0], :] temp_sum[1:, :] = temp_sum[1:, :] - temp_sum[0:-1, :] # stores the accumulation of i.phase within # each voiced segments tmp_cumsum[idx, :, :] = 0 tmp_cumsum[idx, u_loc[idx, :, 0], :] = temp_sum # rad_values - tmp_cumsum: remove the accumulation of i.phase # within the previous voiced segment. i_phase = torch.cumsum(rad_values - tmp_cumsum, dim=1) # get the sines sines = torch.cos(i_phase * 2 * np.pi) return sines def forward(self, f0, upp=None): """ sine_tensor, uv = forward(f0) input F0: tensor(batchsize=1, length, dim=1) f0 for unvoiced steps should be 0 output sine_tensor: tensor(batchsize=1, length, dim) output uv: tensor(batchsize=1, length, 1) """ if self.onnx: with torch.no_grad(): f0 = f0[:, None].transpose(1, 2) f0_buf = torch.zeros(f0.shape[0], f0.shape[1], self.dim, device=f0.device) # fundamental component f0_buf[:, :, 0] = f0[:, :, 0] for idx in np.arange(self.harmonic_num): f0_buf[:, :, idx + 1] = f0_buf[:, :, 0] * ( idx + 2 ) # idx + 2: the (idx+1)-th overtone, (idx+2)-th harmonic rad_values = (f0_buf / self.sampling_rate) % 1 ###%1意味着n_har的乘积无法后处理优化 rand_ini = torch.rand( f0_buf.shape[0], f0_buf.shape[2], device=f0_buf.device ) rand_ini[:, 0] = 0 rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini tmp_over_one = torch.cumsum(rad_values, 1) # % 1 #####%1意味着后面的cumsum无法再优化 tmp_over_one *= upp tmp_over_one = F.interpolate( tmp_over_one.transpose(2, 1), scale_factor=upp, mode="linear", align_corners=True, ).transpose(2, 1) rad_values = F.interpolate( rad_values.transpose(2, 1), scale_factor=upp, mode="nearest" ).transpose( 2, 1 ) ####### tmp_over_one %= 1 tmp_over_one_idx = (tmp_over_one[:, 1:, :] - tmp_over_one[:, :-1, :]) < 0 cumsum_shift = torch.zeros_like(rad_values) cumsum_shift[:, 1:, :] = tmp_over_one_idx * -1.0 sine_waves = torch.sin( torch.cumsum(rad_values + cumsum_shift, dim=1) * 2 * np.pi ) sine_waves = sine_waves * self.sine_amp uv = self._f02uv(f0) uv = F.interpolate( uv.transpose(2, 1), scale_factor=upp, mode="nearest" ).transpose(2, 1) noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3 noise = noise_amp * torch.randn_like(sine_waves) sine_waves = sine_waves * uv + noise return sine_waves, uv, noise else: with torch.no_grad(): # fundamental component fn = torch.multiply(f0, torch.FloatTensor([[range(1, self.harmonic_num + 2)]]).to(f0.device)) # generate sine waveforms sine_waves = self._f02sine(fn) * self.sine_amp # generate uv signal # uv = torch.ones(f0.shape) # uv = uv * (f0 > self.voiced_threshold) uv = self._f02uv(f0) # noise: for unvoiced should be similar to sine_amp # std = self.sine_amp/3 -> max value ~ self.sine_amp # . for voiced regions is self.noise_std noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3 noise = noise_amp * torch.randn_like(sine_waves) # first: set the unvoiced part to 0 by uv # then: additive noise sine_waves = sine_waves * uv + noise return sine_waves, uv, noise class SourceModuleHnNSF(torch.nn.Module): """ SourceModule for hn-nsf SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0) sampling_rate: sampling_rate in Hz harmonic_num: number of harmonic above F0 (default: 0) sine_amp: amplitude of sine source signal (default: 0.1) add_noise_std: std of additive Gaussian noise (default: 0.003) note that amplitude of noise in unvoiced is decided by sine_amp voiced_threshold: threhold to set U/V given F0 (default: 0) Sine_source, noise_source = SourceModuleHnNSF(F0_sampled) F0_sampled (batchsize, length, 1) Sine_source (batchsize, length, 1) noise_source (batchsize, length 1) uv (batchsize, length, 1) """ def __init__(self, sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0): super(SourceModuleHnNSF, self).__init__() self.sine_amp = sine_amp self.noise_std = add_noise_std # to produce sine waveforms self.l_sin_gen = SineGen(sampling_rate, harmonic_num, sine_amp, add_noise_std, voiced_threshod) # to merge source harmonics into a single excitation self.l_linear = torch.nn.Linear(harmonic_num + 1, 1) self.l_tanh = torch.nn.Tanh() def forward(self, x, upp=None): """ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled) F0_sampled (batchsize, length, 1) Sine_source (batchsize, length, 1) noise_source (batchsize, length 1) """ # source for harmonic branch sine_wavs, uv, _ = self.l_sin_gen(x, upp) sine_merge = self.l_tanh(self.l_linear(sine_wavs.to(self.l_linear.weight.dtype))) # source for noise branch, in the same shape as uv noise = torch.randn_like(uv) * self.sine_amp / 3 return sine_merge, noise, uv class Generator(torch.nn.Module): def __init__(self, h): super(Generator, self).__init__() self.h = h self.num_kernels = len(h["resblock_kernel_sizes"]) self.num_upsamples = len(h["upsample_rates"]) self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(h["upsample_rates"])) self.m_source = SourceModuleHnNSF( sampling_rate=h["sampling_rate"], harmonic_num=8) self.noise_convs = nn.ModuleList() self.conv_pre = weight_norm(Conv1d(h["inter_channels"], h["upsample_initial_channel"], 7, 1, padding=3)) resblock = ResBlock1 if h["resblock"] == '1' else ResBlock2 self.ups = nn.ModuleList() for i, (u, k) in enumerate(zip(h["upsample_rates"], h["upsample_kernel_sizes"])): c_cur = h["upsample_initial_channel"] // (2 ** (i + 1)) self.ups.append(weight_norm( ConvTranspose1d(h["upsample_initial_channel"] // (2 ** i), h["upsample_initial_channel"] // (2 ** (i + 1)), k, u, padding=(k - u +1 ) // 2))) if i + 1 < len(h["upsample_rates"]): # stride_f0 = np.prod(h["upsample_rates"][i + 1:]) self.noise_convs.append(Conv1d( 1, c_cur, kernel_size=stride_f0 * 2, stride=stride_f0, padding=(stride_f0+1) // 2)) else: self.noise_convs.append(Conv1d(1, c_cur, kernel_size=1)) self.resblocks = nn.ModuleList() for i in range(len(self.ups)): ch = h["upsample_initial_channel"] // (2 ** (i + 1)) for j, (k, d) in enumerate(zip(h["resblock_kernel_sizes"], h["resblock_dilation_sizes"])): self.resblocks.append(resblock(h, ch, k, d)) self.conv_post = weight_norm(Conv1d(ch, 1, 7, 1, padding=3)) self.ups.apply(init_weights) self.conv_post.apply(init_weights) self.cond = nn.Conv1d(h['gin_channels'], h['upsample_initial_channel'], 1) self.upp = np.prod(h["upsample_rates"]) self.onnx = False def OnnxExport(self): self.onnx = True self.m_source.l_sin_gen.onnx = True def forward(self, x, f0, g=None): # print(1,x.shape,f0.shape,f0[:, None].shape) if not self.onnx: f0 = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t # print(2,f0.shape) har_source, noi_source, uv = self.m_source(f0, self.upp) har_source = har_source.transpose(1, 2) x = self.conv_pre(x) x = x + self.cond(g) # print(124,x.shape,har_source.shape) for i in range(self.num_upsamples): x = F.leaky_relu(x, LRELU_SLOPE) # print(3,x.shape) x = self.ups[i](x) x_source = self.noise_convs[i](har_source) # print(4,x_source.shape,har_source.shape,x.shape) x = x + x_source xs = None for j in range(self.num_kernels): if xs is None: xs = self.resblocks[i * self.num_kernels + j](x) else: xs += self.resblocks[i * self.num_kernels + j](x) x = xs / self.num_kernels x = F.leaky_relu(x) x = self.conv_post(x) x = torch.tanh(x) return x def remove_weight_norm(self): print('Removing weight norm...') for l in self.ups: remove_weight_norm(l) for l in self.resblocks: l.remove_weight_norm() remove_weight_norm(self.conv_pre) remove_weight_norm(self.conv_post) class DiscriminatorP(torch.nn.Module): def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False): super(DiscriminatorP, self).__init__() self.period = period norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(2, 0))), ]) self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0))) def forward(self, x): fmap = [] # 1d to 2d b, c, t = x.shape if t % self.period != 0: # pad first n_pad = self.period - (t % self.period) x = F.pad(x, (0, n_pad), "reflect") t = t + n_pad x = x.view(b, c, t // self.period, self.period) for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiPeriodDiscriminator(torch.nn.Module): def __init__(self, periods=None): super(MultiPeriodDiscriminator, self).__init__() self.periods = periods if periods is not None else [2, 3, 5, 7, 11] self.discriminators = nn.ModuleList() for period in self.periods: self.discriminators.append(DiscriminatorP(period)) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs class DiscriminatorS(torch.nn.Module): def __init__(self, use_spectral_norm=False): super(DiscriminatorS, self).__init__() norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv1d(1, 128, 15, 1, padding=7)), norm_f(Conv1d(128, 128, 41, 2, groups=4, padding=20)), norm_f(Conv1d(128, 256, 41, 2, groups=16, padding=20)), norm_f(Conv1d(256, 512, 41, 4, groups=16, padding=20)), norm_f(Conv1d(512, 1024, 41, 4, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 41, 1, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 5, 1, padding=2)), ]) self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1)) def forward(self, x): fmap = [] for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiScaleDiscriminator(torch.nn.Module): def __init__(self): super(MultiScaleDiscriminator, self).__init__() self.discriminators = nn.ModuleList([ DiscriminatorS(use_spectral_norm=True), DiscriminatorS(), DiscriminatorS(), ]) self.meanpools = nn.ModuleList([ AvgPool1d(4, 2, padding=2), AvgPool1d(4, 2, padding=2) ]) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): if i != 0: y = self.meanpools[i - 1](y) y_hat = self.meanpools[i - 1](y_hat) y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs def feature_loss(fmap_r, fmap_g): loss = 0 for dr, dg in zip(fmap_r, fmap_g): for rl, gl in zip(dr, dg): loss += torch.mean(torch.abs(rl - gl)) return loss * 2 def discriminator_loss(disc_real_outputs, disc_generated_outputs): loss = 0 r_losses = [] g_losses = [] for dr, dg in zip(disc_real_outputs, disc_generated_outputs): r_loss = torch.mean((1 - dr) ** 2) g_loss = torch.mean(dg ** 2) loss += (r_loss + g_loss) r_losses.append(r_loss.item()) g_losses.append(g_loss.item()) return loss, r_losses, g_losses def generator_loss(disc_outputs): loss = 0 gen_losses = [] for dg in disc_outputs: l = torch.mean((1 - dg) ** 2) gen_losses.append(l) loss += l return loss, gen_losses so-vits-svc-npu/vdecoder/hifigan/nvSTFT.py000066400000000000000000000105411456161405400207600ustar00rootroot00000000000000import os import librosa import numpy as np import soundfile as sf import mindtorch.torch as torch import mindtorch.torch.utils.data from librosa.filters import mel as librosa_mel_fn os.environ["LRU_CACHE_CAPACITY"] = "3" def load_wav_to_torch(full_path, target_sr=None, return_empty_on_exception=False): sampling_rate = None try: data, sampling_rate = sf.read(full_path, always_2d=True)# than soundfile. except Exception as ex: print(f"'{full_path}' failed to load.\nException:") print(ex) if return_empty_on_exception: return [], sampling_rate or target_sr or 32000 else: raise Exception(ex) if len(data.shape) > 1: data = data[:, 0] assert len(data) > 2# check duration of audio file is > 2 samples (because otherwise the slice operation was on the wrong dimension) if np.issubdtype(data.dtype, np.integer): # if audio data is type int max_mag = -np.iinfo(data.dtype).min # maximum magnitude = min possible value of intXX else: # if audio data is type fp32 max_mag = max(np.amax(data), -np.amin(data)) max_mag = (2**31)+1 if max_mag > (2**15) else ((2**15)+1 if max_mag > 1.01 else 1.0) # data should be either 16-bit INT, 32-bit INT or [-1 to 1] float32 data = torch.FloatTensor(data.astype(np.float32))/max_mag if (torch.isinf(data) | torch.isnan(data)).any() and return_empty_on_exception:# resample will crash with inf/NaN inputs. return_empty_on_exception will return empty arr instead of except return [], sampling_rate or target_sr or 32000 if target_sr is not None and sampling_rate != target_sr: data = torch.from_numpy(librosa.core.resample(data.numpy(), orig_sr=sampling_rate, target_sr=target_sr)) sampling_rate = target_sr return data, sampling_rate def dynamic_range_compression(x, C=1, clip_val=1e-5): return np.log(np.clip(x, a_min=clip_val, a_max=None) * C) def dynamic_range_decompression(x, C=1): return np.exp(x) / C def dynamic_range_compression_torch(x, C=1, clip_val=1e-5): return torch.log(torch.clamp(x, min=clip_val) * C) def dynamic_range_decompression_torch(x, C=1): return torch.exp(x) / C class STFT(): def __init__(self, sr=22050, n_mels=80, n_fft=1024, win_size=1024, hop_length=256, fmin=20, fmax=11025, clip_val=1e-5): self.target_sr = sr self.n_mels = n_mels self.n_fft = n_fft self.win_size = win_size self.hop_length = hop_length self.fmin = fmin self.fmax = fmax self.clip_val = clip_val self.mel_basis = {} self.hann_window = {} def get_mel(self, y, center=False): sampling_rate = self.target_sr n_mels = self.n_mels n_fft = self.n_fft win_size = self.win_size hop_length = self.hop_length fmin = self.fmin fmax = self.fmax clip_val = self.clip_val if torch.min(y) < -1.: print('min value is ', torch.min(y)) if torch.max(y) > 1.: print('max value is ', torch.max(y)) if fmax not in self.mel_basis: mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=n_mels, fmin=fmin, fmax=fmax) self.mel_basis[str(fmax)+'_'+str(y.device)] = torch.from_numpy(mel).float().to(y.device) self.hann_window[str(y.device)] = torch.hann_window(self.win_size).to(y.device) y = torch.nn.functional.pad(y.unsqueeze(1), (int((n_fft-hop_length)/2), int((n_fft-hop_length)/2)), mode='reflect') y = y.squeeze(1) spec = torch.stft(y, n_fft, hop_length=hop_length, win_length=win_size, window=self.hann_window[str(y.device)], center=center, pad_mode='reflect', normalized=False, onesided=True) # print(111,spec) spec = torch.sqrt(spec.pow(2).sum(-1)+(1e-9)) # print(222,spec) spec = torch.matmul(self.mel_basis[str(fmax)+'_'+str(y.device)], spec) # print(333,spec) spec = dynamic_range_compression_torch(spec, clip_val=clip_val) # print(444,spec) return spec def __call__(self, audiopath): audio, sr = load_wav_to_torch(audiopath, target_sr=self.target_sr) spect = self.get_mel(audio.unsqueeze(0)).squeeze(0) return spect stft = STFT() so-vits-svc-npu/vdecoder/hifigan/utils.py000066400000000000000000000035471456161405400210040ustar00rootroot00000000000000import glob import os # matplotlib.use("Agg") import matplotlib.pylab as plt import mindtorch.torch as torch from mindtorch.torch.nn.utils import weight_norm def plot_spectrogram(spectrogram): fig, ax = plt.subplots(figsize=(10, 2)) im = ax.imshow(spectrogram, aspect="auto", origin="lower", interpolation='none') plt.colorbar(im, ax=ax) fig.canvas.draw() plt.close() return fig def init_weights(m, mean=0.0, std=0.01): classname = m.__class__.__name__ if classname.find("Conv") != -1: m.weight.data.normal_(mean, std) def apply_weight_norm(m): classname = m.__class__.__name__ if classname.find("Conv") != -1: weight_norm(m) def get_padding(kernel_size, dilation=1): return int((kernel_size*dilation - dilation)/2) def load_checkpoint(filepath, device): assert os.path.isfile(filepath) print("Loading '{}'".format(filepath)) checkpoint_dict = torch.load(filepath, map_location=device) print("Complete.") return checkpoint_dict def save_checkpoint(filepath, obj): print("Saving checkpoint to {}".format(filepath)) torch.save(obj, filepath) print("Complete.") def del_old_checkpoints(cp_dir, prefix, n_models=2): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) # get checkpoint paths cp_list = sorted(cp_list)# sort by iter if len(cp_list) > n_models: # if more than n_models models are found for cp in cp_list[:-n_models]:# delete the oldest models other than lastest n_models open(cp, 'w').close()# empty file contents os.unlink(cp)# delete file (move to trash when using Colab) def scan_checkpoint(cp_dir, prefix): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) if len(cp_list) == 0: return None return sorted(cp_list)[-1] so-vits-svc-npu/vdecoder/hifiganwithsnake/000077500000000000000000000000001456161405400211775ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/hifiganwithsnake/alias/000077500000000000000000000000001456161405400222705ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/hifiganwithsnake/alias/__init__.py000066400000000000000000000003621456161405400244020ustar00rootroot00000000000000# Adapted from https://github.com/junjun3518/alias-free-torch under the Apache License 2.0 # LICENSE is in incl_licenses directory. from .act import * # noqa: F403 from .filter import * # noqa: F403 from .resample import * # noqa: F403 so-vits-svc-npu/vdecoder/hifiganwithsnake/alias/act.py000066400000000000000000000106601456161405400234140ustar00rootroot00000000000000# Adapted from https://github.com/junjun3518/alias-free-torch under the Apache License 2.0 # LICENSE is in incl_licenses directory. import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch import pow, sin from mindtorch.torch.nn import Parameter from .resample import DownSample1d, UpSample1d class Activation1d(nn.Module): def __init__(self, activation, up_ratio: int = 2, down_ratio: int = 2, up_kernel_size: int = 12, down_kernel_size: int = 12): super().__init__() self.up_ratio = up_ratio self.down_ratio = down_ratio self.act = activation self.upsample = UpSample1d(up_ratio, up_kernel_size) self.downsample = DownSample1d(down_ratio, down_kernel_size) # x: [B,C,T] def forward(self, x): x = self.upsample(x) x = self.act(x) x = self.downsample(x) return x class SnakeBeta(nn.Module): ''' A modified Snake function which uses separate parameters for the magnitude of the periodic components Shape: - Input: (B, C, T) - Output: (B, C, T), same shape as the input Parameters: - alpha - trainable parameter that controls frequency - beta - trainable parameter that controls magnitude References: - This activation function is a modified version based on this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda: https://arxiv.org/abs/2006.08195 Examples: >>> a1 = snakebeta(256) >>> x = torch.randn(256) >>> x = a1(x) ''' def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=False): ''' Initialization. INPUT: - in_features: shape of the input - alpha - trainable parameter that controls frequency - beta - trainable parameter that controls magnitude alpha is initialized to 1 by default, higher values = higher-frequency. beta is initialized to 1 by default, higher values = higher-magnitude. alpha will be trained along with the rest of your model. ''' super(SnakeBeta, self).__init__() self.in_features = in_features # initialize alpha self.alpha_logscale = alpha_logscale if self.alpha_logscale: # log scale alphas initialized to zeros self.alpha = Parameter(torch.zeros(in_features) * alpha) self.beta = Parameter(torch.zeros(in_features) * alpha) else: # linear scale alphas initialized to ones self.alpha = Parameter(torch.ones(in_features) * alpha) self.beta = Parameter(torch.ones(in_features) * alpha) self.alpha.requires_grad = alpha_trainable self.beta.requires_grad = alpha_trainable self.no_div_by_zero = 0.000000001 def forward(self, x): ''' Forward pass of the function. Applies the function to the input elementwise. SnakeBeta = x + 1/b * sin^2 (xa) ''' alpha = self.alpha.unsqueeze( 0).unsqueeze(-1) # line up with x to [B, C, T] beta = self.beta.unsqueeze(0).unsqueeze(-1) if self.alpha_logscale: alpha = torch.exp(alpha) beta = torch.exp(beta) x = x + (1.0 / (beta + self.no_div_by_zero)) * pow(sin(x * alpha), 2) return x class Mish(nn.Module): """ Mish activation function is proposed in "Mish: A Self Regularized Non-Monotonic Neural Activation Function" paper, https://arxiv.org/abs/1908.08681. """ def __init__(self): super().__init__() def forward(self, x): return x * torch.tanh(F.softplus(x)) class SnakeAlias(nn.Module): def __init__(self, channels, up_ratio: int = 2, down_ratio: int = 2, up_kernel_size: int = 12, down_kernel_size: int = 12, C = None): super().__init__() self.up_ratio = up_ratio self.down_ratio = down_ratio self.act = SnakeBeta(channels, alpha_logscale=True) self.upsample = UpSample1d(up_ratio, up_kernel_size, C) self.downsample = DownSample1d(down_ratio, down_kernel_size, C) # x: [B,C,T] def forward(self, x, C=None): x = self.upsample(x, C) x = self.act(x) x = self.downsample(x) return xso-vits-svc-npu/vdecoder/hifiganwithsnake/alias/filter.py000066400000000000000000000101701456161405400241260ustar00rootroot00000000000000# Adapted from https://github.com/junjun3518/alias-free-torch under the Apache License 2.0 # LICENSE is in incl_licenses directory. import math import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F if 'sinc' in dir(torch): sinc = torch.sinc else: # This code is adopted from adefossez's julius.core.sinc under the MIT License # https://adefossez.github.io/julius/julius/core.html # LICENSE is in incl_licenses directory. def sinc(x: torch.Tensor): """ Implementation of sinc, i.e. sin(pi * x) / (pi * x) __Warning__: Different to julius.sinc, the input is multiplied by `pi`! """ return torch.where(x == 0, torch.tensor(1., device=x.device, dtype=x.dtype), torch.sin(math.pi * x) / math.pi / x) # This code is adopted from adefossez's julius.lowpass.LowPassFilters under the MIT License # https://adefossez.github.io/julius/julius/lowpass.html # LICENSE is in incl_licenses directory. def kaiser_sinc_filter1d(cutoff, half_width, kernel_size): # return filter [1,1,kernel_size] even = (kernel_size % 2 == 0) half_size = kernel_size // 2 #For kaiser window delta_f = 4 * half_width A = 2.285 * (half_size - 1) * math.pi * delta_f + 7.95 if A > 50.: beta = 0.1102 * (A - 8.7) elif A >= 21.: beta = 0.5842 * (A - 21)**0.4 + 0.07886 * (A - 21.) else: beta = 0. window = torch.kaiser_window(kernel_size, beta=beta, periodic=False) # ratio = 0.5/cutoff -> 2 * cutoff = 1 / ratio if even: time = (torch.arange(-half_size, half_size) + 0.5) else: time = torch.arange(kernel_size) - half_size if cutoff == 0: filter_ = torch.zeros_like(time) else: filter_ = 2 * cutoff * window * sinc(2 * cutoff * time) # Normalize filter to have sum = 1, otherwise we will have a small leakage # of the constant component in the input signal. filter_ /= filter_.sum() filter = filter_.view(1, 1, kernel_size) return filter class LowPassFilter1d(nn.Module): def __init__(self, cutoff=0.5, half_width=0.6, stride: int = 1, padding: bool = True, padding_mode: str = 'replicate', kernel_size: int = 12, C=None): # kernel_size should be even number for stylegan3 setup, # in this implementation, odd number is also possible. super().__init__() if cutoff < -0.: raise ValueError("Minimum cutoff must be larger than zero.") if cutoff > 0.5: raise ValueError("A cutoff above 0.5 does not make sense.") self.kernel_size = kernel_size self.even = (kernel_size % 2 == 0) self.pad_left = kernel_size // 2 - int(self.even) self.pad_right = kernel_size // 2 self.stride = stride self.padding = padding self.padding_mode = padding_mode filter = kaiser_sinc_filter1d(cutoff, half_width, kernel_size) self.register_buffer("filter", filter) self.conv1d_block = None if C is not None: self.conv1d_block = [nn.Conv1d(C,C,kernel_size,stride=self.stride, groups=C, bias=False),] self.conv1d_block[0].weight = nn.Parameter(self.filter.expand(C, -1, -1)) self.conv1d_block[0].requires_grad_(False) #input [B, C, T] def forward(self, x): if self.conv1d_block[0].weight.device != x.device: self.conv1d_block[0] = self.conv1d_block[0].to(x.device) if self.conv1d_block is None: _, C, _ = x.shape if self.padding: x = F.pad(x, (self.pad_left, self.pad_right), mode=self.padding_mode) out = F.conv1d(x, self.filter.expand(C, -1, -1), stride=self.stride, groups=C) else: if self.padding: x = F.pad(x, (self.pad_left, self.pad_right), mode=self.padding_mode) out = self.conv1d_block[0](x) return outso-vits-svc-npu/vdecoder/hifiganwithsnake/alias/resample.py000066400000000000000000000062561456161405400244630ustar00rootroot00000000000000# Adapted from https://github.com/junjun3518/alias-free-torch under the Apache License 2.0 # LICENSE is in incl_licenses directory. import mindtorch.torch.nn as nn from mindtorch.torch.nn import functional as F from .filter import LowPassFilter1d, kaiser_sinc_filter1d class UpSample1d(nn.Module): def __init__(self, ratio=2, kernel_size=None, C=None): super().__init__() self.ratio = ratio self.kernel_size = int(6 * ratio // 2) * 2 if kernel_size is None else kernel_size self.stride = ratio self.pad = self.kernel_size // ratio - 1 self.pad_left = self.pad * self.stride + (self.kernel_size - self.stride) // 2 self.pad_right = self.pad * self.stride + (self.kernel_size - self.stride + 1) // 2 filter = kaiser_sinc_filter1d(cutoff=0.5 / ratio, half_width=0.6 / ratio, kernel_size=self.kernel_size) self.register_buffer("filter", filter) self.conv_transpose1d_block = None if C is not None: self.conv_transpose1d_block = [nn.ConvTranspose1d(C, C, kernel_size=self.kernel_size, stride=self.stride, groups=C, bias=False ),] self.conv_transpose1d_block[0].weight = nn.Parameter(self.filter.expand(C, -1, -1).clone()) self.conv_transpose1d_block[0].requires_grad_(False) # x: [B, C, T] def forward(self, x, C=None): if self.conv_transpose1d_block[0].weight.device != x.device: self.conv_transpose1d_block[0] = self.conv_transpose1d_block[0].to(x.device) if self.conv_transpose1d_block is None: if C is None: _, C, _ = x.shape # print("snake.conv_t.in:",x.shape) x = F.pad(x, (self.pad, self.pad), mode='replicate') x = self.ratio * F.conv_transpose1d( x, self.filter.expand(C, -1, -1), stride=self.stride, groups=C) # print("snake.conv_t.out:",x.shape) x = x[..., self.pad_left:-self.pad_right] else: x = F.pad(x, (self.pad, self.pad), mode='replicate') x = self.ratio * self.conv_transpose1d_block[0](x) x = x[..., self.pad_left:-self.pad_right] return x class DownSample1d(nn.Module): def __init__(self, ratio=2, kernel_size=None, C=None): super().__init__() self.ratio = ratio self.kernel_size = int(6 * ratio // 2) * 2 if kernel_size is None else kernel_size self.lowpass = LowPassFilter1d(cutoff=0.5 / ratio, half_width=0.6 / ratio, stride=ratio, kernel_size=self.kernel_size, C=C) def forward(self, x): xx = self.lowpass(x) return xxso-vits-svc-npu/vdecoder/hifiganwithsnake/env.py000066400000000000000000000006121456161405400223400ustar00rootroot00000000000000import os import shutil class AttrDict(dict): def __init__(self, *args, **kwargs): super(AttrDict, self).__init__(*args, **kwargs) self.__dict__ = self def build_env(config, config_name, path): t_path = os.path.join(path, config_name) if config != t_path: os.makedirs(path, exist_ok=True) shutil.copyfile(config, os.path.join(path, config_name)) so-vits-svc-npu/vdecoder/hifiganwithsnake/models.py000066400000000000000000000544531456161405400230470ustar00rootroot00000000000000import json import os import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn import AvgPool1d, Conv1d, Conv2d, ConvTranspose1d from mindtorch.torch.nn.utils import remove_weight_norm, spectral_norm, weight_norm from vdecoder.hifiganwithsnake.alias.act import SnakeAlias from .env import AttrDict from .utils import get_padding, init_weights LRELU_SLOPE = 0.1 def load_model(model_path, device='cuda'): config_file = os.path.join(os.path.split(model_path)[0], 'config.json') with open(config_file) as f: data = f.read() global h json_config = json.loads(data) h = AttrDict(json_config) generator = Generator(h).to(device) cp_dict = torch.load(model_path) generator.load_state_dict(cp_dict['generator']) generator.eval() generator.remove_weight_norm() del cp_dict return generator, h class ResBlock1(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3, 5), C=None): super(ResBlock1, self).__init__() self.h = h self.convs1 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[2], padding=get_padding(kernel_size, dilation[2]))) ]) self.convs1.apply(init_weights) self.convs2 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))) ]) self.convs2.apply(init_weights) self.num_layers = len(self.convs1) + len(self.convs2) self.activations = nn.ModuleList([ SnakeAlias(channels, C=C) for _ in range(self.num_layers) ]) def forward(self, x, DIM=None): acts1, acts2 = self.activations[::2], self.activations[1::2] for c1, c2, a1, a2 in zip(self.convs1, self.convs2, acts1, acts2): xt = a1(x, DIM) xt = c1(xt) xt = a2(xt, DIM) xt = c2(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs1: remove_weight_norm(l) for l in self.convs2: remove_weight_norm(l) class ResBlock2(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3), C=None): super(ResBlock2, self).__init__() self.h = h self.convs = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))) ]) self.convs.apply(init_weights) self.num_layers = len(self.convs) self.activations = nn.ModuleList([ SnakeAlias(channels, C=C) for _ in range(self.num_layers) ]) def forward(self, x, DIM=None): for c,a in zip(self.convs, self.activations): xt = a(x, DIM) xt = c(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs: remove_weight_norm(l) def padDiff(x): return F.pad(F.pad(x, (0,0,-1,1), 'constant', 0) - x, (0,0,0,-1), 'constant', 0) class SineGen(torch.nn.Module): """ Definition of sine generator SineGen(samp_rate, harmonic_num = 0, sine_amp = 0.1, noise_std = 0.003, voiced_threshold = 0, flag_for_pulse=False) samp_rate: sampling rate in Hz harmonic_num: number of harmonic overtones (default 0) sine_amp: amplitude of sine-wavefrom (default 0.1) noise_std: std of Gaussian noise (default 0.003) voiced_thoreshold: F0 threshold for U/V classification (default 0) flag_for_pulse: this SinGen is used inside PulseGen (default False) Note: when flag_for_pulse is True, the first time step of a voiced segment is always sin(np.pi) or cos(0) """ def __init__(self, samp_rate, harmonic_num=0, sine_amp=0.1, noise_std=0.003, voiced_threshold=0, flag_for_pulse=False): super(SineGen, self).__init__() self.sine_amp = sine_amp self.noise_std = noise_std self.harmonic_num = harmonic_num self.dim = self.harmonic_num + 1 self.sampling_rate = samp_rate self.voiced_threshold = voiced_threshold self.flag_for_pulse = flag_for_pulse self.onnx = False def _f02uv(self, f0): # generate uv signal uv = (f0 > self.voiced_threshold).type(torch.float32) return uv def _f02sine(self, f0_values): """ f0_values: (batchsize, length, dim) where dim indicates fundamental tone and overtones """ # convert to F0 in rad. The interger part n can be ignored # because 2 * np.pi * n doesn't affect phase rad_values = (f0_values / self.sampling_rate) % 1 # initial phase noise (no noise for fundamental component) rand_ini = torch.rand(f0_values.shape[0], f0_values.shape[2], \ device=f0_values.device) rand_ini[:, 0] = 0 rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini # instantanouse phase sine[t] = sin(2*pi \sum_i=1 ^{t} rad) if not self.flag_for_pulse: # for normal case # To prevent torch.cumsum numerical overflow, # it is necessary to add -1 whenever \sum_k=1^n rad_value_k > 1. # Buffer tmp_over_one_idx indicates the time step to add -1. # This will not change F0 of sine because (x-1) * 2*pi = x * 2*pi tmp_over_one = torch.cumsum(rad_values, 1) % 1 tmp_over_one_idx = (padDiff(tmp_over_one)) < 0 cumsum_shift = torch.zeros_like(rad_values) cumsum_shift[:, 1:, :] = tmp_over_one_idx * -1.0 sines = torch.sin(torch.cumsum(rad_values + cumsum_shift, dim=1) * 2 * np.pi) else: # If necessary, make sure that the first time step of every # voiced segments is sin(pi) or cos(0) # This is used for pulse-train generation # identify the last time step in unvoiced segments uv = self._f02uv(f0_values) uv_1 = torch.roll(uv, shifts=-1, dims=1) uv_1[:, -1, :] = 1 u_loc = (uv < 1) * (uv_1 > 0) # get the instantanouse phase tmp_cumsum = torch.cumsum(rad_values, dim=1) # different batch needs to be processed differently for idx in range(f0_values.shape[0]): temp_sum = tmp_cumsum[idx, u_loc[idx, :, 0], :] temp_sum[1:, :] = temp_sum[1:, :] - temp_sum[0:-1, :] # stores the accumulation of i.phase within # each voiced segments tmp_cumsum[idx, :, :] = 0 tmp_cumsum[idx, u_loc[idx, :, 0], :] = temp_sum # rad_values - tmp_cumsum: remove the accumulation of i.phase # within the previous voiced segment. i_phase = torch.cumsum(rad_values - tmp_cumsum, dim=1) # get the sines sines = torch.cos(i_phase * 2 * np.pi) return sines def forward(self, f0, upp=None): """ sine_tensor, uv = forward(f0) input F0: tensor(batchsize=1, length, dim=1) f0 for unvoiced steps should be 0 output sine_tensor: tensor(batchsize=1, length, dim) output uv: tensor(batchsize=1, length, 1) """ if self.onnx: with torch.no_grad(): f0 = f0[:, None].transpose(1, 2) f0_buf = torch.zeros(f0.shape[0], f0.shape[1], self.dim, device=f0.device) # fundamental component f0_buf[:, :, 0] = f0[:, :, 0] for idx in np.arange(self.harmonic_num): f0_buf[:, :, idx + 1] = f0_buf[:, :, 0] * ( idx + 2 ) # idx + 2: the (idx+1)-th overtone, (idx+2)-th harmonic rad_values = (f0_buf / self.sampling_rate) % 1 ###%1意味着n_har的乘积无法后处理优化 rand_ini = torch.rand( f0_buf.shape[0], f0_buf.shape[2], device=f0_buf.device ) rand_ini[:, 0] = 0 rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini tmp_over_one = torch.cumsum(rad_values, 1) # % 1 #####%1意味着后面的cumsum无法再优化 tmp_over_one *= upp tmp_over_one = F.interpolate( tmp_over_one.transpose(2, 1), scale_factor=upp, mode="linear", align_corners=True, ).transpose(2, 1) rad_values = F.interpolate( rad_values.transpose(2, 1), scale_factor=upp, mode="nearest" ).transpose( 2, 1 ) ####### tmp_over_one %= 1 tmp_over_one_idx = (tmp_over_one[:, 1:, :] - tmp_over_one[:, :-1, :]) < 0 cumsum_shift = torch.zeros_like(rad_values) cumsum_shift[:, 1:, :] = tmp_over_one_idx * -1.0 sine_waves = torch.sin( torch.cumsum(rad_values + cumsum_shift, dim=1) * 2 * np.pi ) sine_waves = sine_waves * self.sine_amp uv = self._f02uv(f0) uv = F.interpolate( uv.transpose(2, 1), scale_factor=upp, mode="nearest" ).transpose(2, 1) noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3 noise = noise_amp * torch.randn_like(sine_waves) sine_waves = sine_waves * uv + noise return sine_waves, uv, noise else: with torch.no_grad(): # fundamental component fn = torch.multiply(f0, torch.FloatTensor([[range(1, self.harmonic_num + 2)]]).to(f0.device)) # generate sine waveforms sine_waves = self._f02sine(fn) * self.sine_amp # generate uv signal # uv = torch.ones(f0.shape) # uv = uv * (f0 > self.voiced_threshold) uv = self._f02uv(f0) # noise: for unvoiced should be similar to sine_amp # std = self.sine_amp/3 -> max value ~ self.sine_amp # . for voiced regions is self.noise_std noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3 noise = noise_amp * torch.randn_like(sine_waves) # first: set the unvoiced part to 0 by uv # then: additive noise sine_waves = sine_waves * uv + noise return sine_waves, uv, noise class SourceModuleHnNSF(torch.nn.Module): """ SourceModule for hn-nsf SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0) sampling_rate: sampling_rate in Hz harmonic_num: number of harmonic above F0 (default: 0) sine_amp: amplitude of sine source signal (default: 0.1) add_noise_std: std of additive Gaussian noise (default: 0.003) note that amplitude of noise in unvoiced is decided by sine_amp voiced_threshold: threhold to set U/V given F0 (default: 0) Sine_source, noise_source = SourceModuleHnNSF(F0_sampled) F0_sampled (batchsize, length, 1) Sine_source (batchsize, length, 1) noise_source (batchsize, length 1) uv (batchsize, length, 1) """ def __init__(self, sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0): super(SourceModuleHnNSF, self).__init__() self.sine_amp = sine_amp self.noise_std = add_noise_std # to produce sine waveforms self.l_sin_gen = SineGen(sampling_rate, harmonic_num, sine_amp, add_noise_std, voiced_threshod) # to merge source harmonics into a single excitation self.l_linear = torch.nn.Linear(harmonic_num + 1, 1) self.l_tanh = torch.nn.Tanh() def forward(self, x, upp=None): """ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled) F0_sampled (batchsize, length, 1) Sine_source (batchsize, length, 1) noise_source (batchsize, length 1) """ # source for harmonic branch sine_wavs, uv, _ = self.l_sin_gen(x, upp) sine_merge = self.l_tanh(self.l_linear(sine_wavs.to(self.l_linear.weight.dtype))) # source for noise branch, in the same shape as uv noise = torch.randn_like(uv) * self.sine_amp / 3 return sine_merge, noise, uv class Generator(torch.nn.Module): def __init__(self, h): super(Generator, self).__init__() self.h = h self.num_kernels = len(h["resblock_kernel_sizes"]) self.num_upsamples = len(h["upsample_rates"]) self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(h["upsample_rates"])) self.m_source = SourceModuleHnNSF( sampling_rate=h["sampling_rate"], harmonic_num=8) self.noise_convs = nn.ModuleList() self.conv_pre = weight_norm(Conv1d(h["inter_channels"], h["upsample_initial_channel"], 7, 1, padding=3)) resblock = ResBlock1 if h["resblock"] == '1' else ResBlock2 self.ups = nn.ModuleList() for i, (u, k) in enumerate(zip(h["upsample_rates"], h["upsample_kernel_sizes"])): c_cur = h["upsample_initial_channel"] // (2 ** (i + 1)) self.ups.append(weight_norm( ConvTranspose1d(h["upsample_initial_channel"] // (2 ** i), h["upsample_initial_channel"] // (2 ** (i + 1)), k, u, padding=(k - u + 1) // 2))) if i + 1 < len(h["upsample_rates"]): # stride_f0 = np.prod(h["upsample_rates"][i + 1:]) self.noise_convs.append(Conv1d( 1, c_cur, kernel_size=stride_f0 * 2, stride=stride_f0, padding=(stride_f0+ 1) // 2)) else: self.noise_convs.append(Conv1d(1, c_cur, kernel_size=1)) self.resblocks = nn.ModuleList() self.snakes = nn.ModuleList() for i in range(len(self.ups)): ch = h["upsample_initial_channel"] // (2 ** (i + 1)) self.snakes.append(SnakeAlias(h["upsample_initial_channel"] // (2 ** (i)), C = h["upsample_initial_channel"] >> i)) for j, (k, d) in enumerate(zip(h["resblock_kernel_sizes"], h["resblock_dilation_sizes"])): self.resblocks.append(resblock(h, ch, k, d, C = h["upsample_initial_channel"] >> (i + 1))) self.conv_post = weight_norm(Conv1d(ch, 1, 7, 1, padding=3)) self.ups.apply(init_weights) self.conv_post.apply(init_weights) self.snake_post = SnakeAlias(ch, C = h["upsample_initial_channel"] >> len(self.ups)) self.cond = nn.Conv1d(h['gin_channels'], h['upsample_initial_channel'], 1) self.upp = np.prod(h["upsample_rates"]) self.onnx = False def OnnxExport(self): self.onnx = True self.m_source.l_sin_gen.onnx = True def forward(self, x, f0, g=None): # print(1,x.shape,f0.shape,f0[:, None].shape) if not self.onnx: f0 = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t # print(2,f0.shape) har_source, noi_source, uv = self.m_source(f0, self.upp) har_source = har_source.transpose(1, 2) x = self.conv_pre(x) x = x + self.cond(g) # print(124,x.shape,har_source.shape) for i in range(self.num_upsamples): # print(f"self.snakes.{i}.pre:", x.shape) x = self.snakes[i](x) # print(f"self.snakes.{i}.after:", x.shape) x = self.ups[i](x) x_source = self.noise_convs[i](har_source) # print(4,x_source.shape,har_source.shape,x.shape) x = x + x_source xs = None for j in range(self.num_kernels): if xs is None: xs = self.resblocks[i * self.num_kernels + j](x) else: xs += self.resblocks[i * self.num_kernels + j](x) # print(f"self.resblocks.{i}.after:", xs.shape) x = xs / self.num_kernels x = self.snake_post(x) x = self.conv_post(x) x = torch.tanh(x) return x def remove_weight_norm(self): print('Removing weight norm...') for l in self.ups: remove_weight_norm(l) for l in self.resblocks: l.remove_weight_norm() remove_weight_norm(self.conv_pre) remove_weight_norm(self.conv_post) class DiscriminatorP(torch.nn.Module): def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False): super(DiscriminatorP, self).__init__() self.period = period norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(2, 0))), ]) self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0))) def forward(self, x): fmap = [] # 1d to 2d b, c, t = x.shape if t % self.period != 0: # pad first n_pad = self.period - (t % self.period) x = F.pad(x, (0, n_pad), "reflect") t = t + n_pad x = x.view(b, c, t // self.period, self.period) for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiPeriodDiscriminator(torch.nn.Module): def __init__(self, periods=None): super(MultiPeriodDiscriminator, self).__init__() self.periods = periods if periods is not None else [2, 3, 5, 7, 11] self.discriminators = nn.ModuleList() for period in self.periods: self.discriminators.append(DiscriminatorP(period)) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs class DiscriminatorS(torch.nn.Module): def __init__(self, use_spectral_norm=False): super(DiscriminatorS, self).__init__() norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv1d(1, 128, 15, 1, padding=7)), norm_f(Conv1d(128, 128, 41, 2, groups=4, padding=20)), norm_f(Conv1d(128, 256, 41, 2, groups=16, padding=20)), norm_f(Conv1d(256, 512, 41, 4, groups=16, padding=20)), norm_f(Conv1d(512, 1024, 41, 4, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 41, 1, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 5, 1, padding=2)), ]) self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1)) def forward(self, x): fmap = [] for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiScaleDiscriminator(torch.nn.Module): def __init__(self): super(MultiScaleDiscriminator, self).__init__() self.discriminators = nn.ModuleList([ DiscriminatorS(use_spectral_norm=True), DiscriminatorS(), DiscriminatorS(), ]) self.meanpools = nn.ModuleList([ AvgPool1d(4, 2, padding=2), AvgPool1d(4, 2, padding=2) ]) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): if i != 0: y = self.meanpools[i - 1](y) y_hat = self.meanpools[i - 1](y_hat) y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs def feature_loss(fmap_r, fmap_g): loss = 0 for dr, dg in zip(fmap_r, fmap_g): for rl, gl in zip(dr, dg): loss += torch.mean(torch.abs(rl - gl)) return loss * 2 def discriminator_loss(disc_real_outputs, disc_generated_outputs): loss = 0 r_losses = [] g_losses = [] for dr, dg in zip(disc_real_outputs, disc_generated_outputs): r_loss = torch.mean((1 - dr) ** 2) g_loss = torch.mean(dg ** 2) loss += (r_loss + g_loss) r_losses.append(r_loss.item()) g_losses.append(g_loss.item()) return loss, r_losses, g_losses def generator_loss(disc_outputs): loss = 0 gen_losses = [] for dg in disc_outputs: l = torch.mean((1 - dg) ** 2) gen_losses.append(l) loss += l return loss, gen_losses so-vits-svc-npu/vdecoder/hifiganwithsnake/nvSTFT.py000066400000000000000000000105411456161405400226760ustar00rootroot00000000000000import os import librosa import numpy as np import soundfile as sf import mindtorch.torch as torch import mindtorch.torch.utils.data from librosa.filters import mel as librosa_mel_fn os.environ["LRU_CACHE_CAPACITY"] = "3" def load_wav_to_torch(full_path, target_sr=None, return_empty_on_exception=False): sampling_rate = None try: data, sampling_rate = sf.read(full_path, always_2d=True)# than soundfile. except Exception as ex: print(f"'{full_path}' failed to load.\nException:") print(ex) if return_empty_on_exception: return [], sampling_rate or target_sr or 32000 else: raise Exception(ex) if len(data.shape) > 1: data = data[:, 0] assert len(data) > 2# check duration of audio file is > 2 samples (because otherwise the slice operation was on the wrong dimension) if np.issubdtype(data.dtype, np.integer): # if audio data is type int max_mag = -np.iinfo(data.dtype).min # maximum magnitude = min possible value of intXX else: # if audio data is type fp32 max_mag = max(np.amax(data), -np.amin(data)) max_mag = (2**31)+1 if max_mag > (2**15) else ((2**15)+1 if max_mag > 1.01 else 1.0) # data should be either 16-bit INT, 32-bit INT or [-1 to 1] float32 data = torch.FloatTensor(data.astype(np.float32))/max_mag if (torch.isinf(data) | torch.isnan(data)).any() and return_empty_on_exception:# resample will crash with inf/NaN inputs. return_empty_on_exception will return empty arr instead of except return [], sampling_rate or target_sr or 32000 if target_sr is not None and sampling_rate != target_sr: data = torch.from_numpy(librosa.core.resample(data.numpy(), orig_sr=sampling_rate, target_sr=target_sr)) sampling_rate = target_sr return data, sampling_rate def dynamic_range_compression(x, C=1, clip_val=1e-5): return np.log(np.clip(x, a_min=clip_val, a_max=None) * C) def dynamic_range_decompression(x, C=1): return np.exp(x) / C def dynamic_range_compression_torch(x, C=1, clip_val=1e-5): return torch.log(torch.clamp(x, min=clip_val) * C) def dynamic_range_decompression_torch(x, C=1): return torch.exp(x) / C class STFT(): def __init__(self, sr=22050, n_mels=80, n_fft=1024, win_size=1024, hop_length=256, fmin=20, fmax=11025, clip_val=1e-5): self.target_sr = sr self.n_mels = n_mels self.n_fft = n_fft self.win_size = win_size self.hop_length = hop_length self.fmin = fmin self.fmax = fmax self.clip_val = clip_val self.mel_basis = {} self.hann_window = {} def get_mel(self, y, center=False): sampling_rate = self.target_sr n_mels = self.n_mels n_fft = self.n_fft win_size = self.win_size hop_length = self.hop_length fmin = self.fmin fmax = self.fmax clip_val = self.clip_val if torch.min(y) < -1.: print('min value is ', torch.min(y)) if torch.max(y) > 1.: print('max value is ', torch.max(y)) if fmax not in self.mel_basis: mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=n_mels, fmin=fmin, fmax=fmax) self.mel_basis[str(fmax)+'_'+str(y.device)] = torch.from_numpy(mel).float().to(y.device) self.hann_window[str(y.device)] = torch.hann_window(self.win_size).to(y.device) y = torch.nn.functional.pad(y.unsqueeze(1), (int((n_fft-hop_length)/2), int((n_fft-hop_length)/2)), mode='reflect') y = y.squeeze(1) spec = torch.stft(y, n_fft, hop_length=hop_length, win_length=win_size, window=self.hann_window[str(y.device)], center=center, pad_mode='reflect', normalized=False, onesided=True) # print(111,spec) spec = torch.sqrt(spec.pow(2).sum(-1)+(1e-9)) # print(222,spec) spec = torch.matmul(self.mel_basis[str(fmax)+'_'+str(y.device)], spec) # print(333,spec) spec = dynamic_range_compression_torch(spec, clip_val=clip_val) # print(444,spec) return spec def __call__(self, audiopath): audio, sr = load_wav_to_torch(audiopath, target_sr=self.target_sr) spect = self.get_mel(audio.unsqueeze(0)).squeeze(0) return spect stft = STFT() so-vits-svc-npu/vdecoder/hifiganwithsnake/utils.py000066400000000000000000000035471456161405400227220ustar00rootroot00000000000000import glob import os # matplotlib.use("Agg") import matplotlib.pylab as plt import mindtorch.torch as torch from mindtorch.torch.nn.utils import weight_norm def plot_spectrogram(spectrogram): fig, ax = plt.subplots(figsize=(10, 2)) im = ax.imshow(spectrogram, aspect="auto", origin="lower", interpolation='none') plt.colorbar(im, ax=ax) fig.canvas.draw() plt.close() return fig def init_weights(m, mean=0.0, std=0.01): classname = m.__class__.__name__ if classname.find("Conv") != -1: m.weight.data.normal_(mean, std) def apply_weight_norm(m): classname = m.__class__.__name__ if classname.find("Conv") != -1: weight_norm(m) def get_padding(kernel_size, dilation=1): return int((kernel_size*dilation - dilation)/2) def load_checkpoint(filepath, device): assert os.path.isfile(filepath) print("Loading '{}'".format(filepath)) checkpoint_dict = torch.load(filepath, map_location=device) print("Complete.") return checkpoint_dict def save_checkpoint(filepath, obj): print("Saving checkpoint to {}".format(filepath)) torch.save(obj, filepath) print("Complete.") def del_old_checkpoints(cp_dir, prefix, n_models=2): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) # get checkpoint paths cp_list = sorted(cp_list)# sort by iter if len(cp_list) > n_models: # if more than n_models models are found for cp in cp_list[:-n_models]:# delete the oldest models other than lastest n_models open(cp, 'w').close()# empty file contents os.unlink(cp)# delete file (move to trash when using Colab) def scan_checkpoint(cp_dir, prefix): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) if len(cp_list) == 0: return None return sorted(cp_list)[-1] so-vits-svc-npu/vdecoder/nsf_hifigan/000077500000000000000000000000001456161405400201275ustar00rootroot00000000000000so-vits-svc-npu/vdecoder/nsf_hifigan/env.py000066400000000000000000000006121456161405400212700ustar00rootroot00000000000000import os import shutil class AttrDict(dict): def __init__(self, *args, **kwargs): super(AttrDict, self).__init__(*args, **kwargs) self.__dict__ = self def build_env(config, config_name, path): t_path = os.path.join(path, config_name) if config != t_path: os.makedirs(path, exist_ok=True) shutil.copyfile(config, os.path.join(path, config_name)) so-vits-svc-npu/vdecoder/nsf_hifigan/models.py000066400000000000000000000401531456161405400217670ustar00rootroot00000000000000import json import os import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn import AvgPool1d, Conv1d, Conv2d, ConvTranspose1d from mindtorch.torch.nn.utils import remove_weight_norm, spectral_norm, weight_norm from .env import AttrDict from .utils import get_padding, init_weights LRELU_SLOPE = 0.1 def load_model(model_path, device='cuda'): h = load_config(model_path) generator = Generator(h).to(device) cp_dict = torch.load(model_path, map_location=device) generator.load_state_dict(cp_dict['generator']) generator.eval() generator.remove_weight_norm() del cp_dict return generator, h def load_config(model_path): config_file = os.path.join(os.path.split(model_path)[0], 'config.json') with open(config_file) as f: data = f.read() json_config = json.loads(data) h = AttrDict(json_config) return h class ResBlock1(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3, 5)): super(ResBlock1, self).__init__() self.h = h self.convs1 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[2], padding=get_padding(kernel_size, dilation[2]))) ]) self.convs1.apply(init_weights) self.convs2 = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1, padding=get_padding(kernel_size, 1))) ]) self.convs2.apply(init_weights) def forward(self, x): for c1, c2 in zip(self.convs1, self.convs2): xt = F.leaky_relu(x, LRELU_SLOPE) xt = c1(xt) xt = F.leaky_relu(xt, LRELU_SLOPE) xt = c2(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs1: remove_weight_norm(l) for l in self.convs2: remove_weight_norm(l) class ResBlock2(torch.nn.Module): def __init__(self, h, channels, kernel_size=3, dilation=(1, 3)): super(ResBlock2, self).__init__() self.h = h self.convs = nn.ModuleList([ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0], padding=get_padding(kernel_size, dilation[0]))), weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1], padding=get_padding(kernel_size, dilation[1]))) ]) self.convs.apply(init_weights) def forward(self, x): for c in self.convs: xt = F.leaky_relu(x, LRELU_SLOPE) xt = c(xt) x = xt + x return x def remove_weight_norm(self): for l in self.convs: remove_weight_norm(l) class SineGen(torch.nn.Module): """ Definition of sine generator SineGen(samp_rate, harmonic_num = 0, sine_amp = 0.1, noise_std = 0.003, voiced_threshold = 0, flag_for_pulse=False) samp_rate: sampling rate in Hz harmonic_num: number of harmonic overtones (default 0) sine_amp: amplitude of sine-wavefrom (default 0.1) noise_std: std of Gaussian noise (default 0.003) voiced_thoreshold: F0 threshold for U/V classification (default 0) flag_for_pulse: this SinGen is used inside PulseGen (default False) Note: when flag_for_pulse is True, the first time step of a voiced segment is always sin(np.pi) or cos(0) """ def __init__(self, samp_rate, harmonic_num=0, sine_amp=0.1, noise_std=0.003, voiced_threshold=0): super(SineGen, self).__init__() self.sine_amp = sine_amp self.noise_std = noise_std self.harmonic_num = harmonic_num self.dim = self.harmonic_num + 1 self.sampling_rate = samp_rate self.voiced_threshold = voiced_threshold def _f02uv(self, f0): # generate uv signal uv = torch.ones_like(f0) uv = uv * (f0 > self.voiced_threshold) return uv @torch.no_grad() def forward(self, f0, upp): """ sine_tensor, uv = forward(f0) input F0: tensor(batchsize=1, length, dim=1) f0 for unvoiced steps should be 0 output sine_tensor: tensor(batchsize=1, length, dim) output uv: tensor(batchsize=1, length, 1) """ f0 = f0.unsqueeze(-1) fn = torch.multiply(f0, torch.arange(1, self.dim + 1, device=f0.device).reshape((1, 1, -1))) rad_values = (fn / self.sampling_rate) % 1 ###%1意味着n_har的乘积无法后处理优化 rand_ini = torch.rand(fn.shape[0], fn.shape[2], device=fn.device) rand_ini[:, 0] = 0 rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini is_half = rad_values.dtype is not torch.float32 tmp_over_one = torch.cumsum(rad_values.double(), 1) # % 1 #####%1意味着后面的cumsum无法再优化 if is_half: tmp_over_one = tmp_over_one.half() else: tmp_over_one = tmp_over_one.float() tmp_over_one *= upp tmp_over_one = F.interpolate( tmp_over_one.transpose(2, 1), scale_factor=upp, mode='linear', align_corners=True ).transpose(2, 1) rad_values = F.interpolate(rad_values.transpose(2, 1), scale_factor=upp, mode='nearest').transpose(2, 1) tmp_over_one %= 1 tmp_over_one_idx = (tmp_over_one[:, 1:, :] - tmp_over_one[:, :-1, :]) < 0 cumsum_shift = torch.zeros_like(rad_values) cumsum_shift[:, 1:, :] = tmp_over_one_idx * -1.0 rad_values = rad_values.double() cumsum_shift = cumsum_shift.double() sine_waves = torch.sin(torch.cumsum(rad_values + cumsum_shift, dim=1) * 2 * np.pi) if is_half: sine_waves = sine_waves.half() else: sine_waves = sine_waves.float() sine_waves = sine_waves * self.sine_amp uv = self._f02uv(f0) uv = F.interpolate(uv.transpose(2, 1), scale_factor=upp, mode='nearest').transpose(2, 1) noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3 noise = noise_amp * torch.randn_like(sine_waves) sine_waves = sine_waves * uv + noise return sine_waves, uv, noise class SourceModuleHnNSF(torch.nn.Module): """ SourceModule for hn-nsf SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0) sampling_rate: sampling_rate in Hz harmonic_num: number of harmonic above F0 (default: 0) sine_amp: amplitude of sine source signal (default: 0.1) add_noise_std: std of additive Gaussian noise (default: 0.003) note that amplitude of noise in unvoiced is decided by sine_amp voiced_threshold: threhold to set U/V given F0 (default: 0) Sine_source, noise_source = SourceModuleHnNSF(F0_sampled) F0_sampled (batchsize, length, 1) Sine_source (batchsize, length, 1) noise_source (batchsize, length 1) uv (batchsize, length, 1) """ def __init__(self, sampling_rate, harmonic_num=0, sine_amp=0.1, add_noise_std=0.003, voiced_threshod=0): super(SourceModuleHnNSF, self).__init__() self.sine_amp = sine_amp self.noise_std = add_noise_std # to produce sine waveforms self.l_sin_gen = SineGen(sampling_rate, harmonic_num, sine_amp, add_noise_std, voiced_threshod) # to merge source harmonics into a single excitation self.l_linear = torch.nn.Linear(harmonic_num + 1, 1) self.l_tanh = torch.nn.Tanh() def forward(self, x, upp): sine_wavs, uv, _ = self.l_sin_gen(x, upp) sine_merge = self.l_tanh(self.l_linear(sine_wavs)) return sine_merge class Generator(torch.nn.Module): def __init__(self, h): super(Generator, self).__init__() self.h = h self.num_kernels = len(h.resblock_kernel_sizes) self.num_upsamples = len(h.upsample_rates) self.m_source = SourceModuleHnNSF( sampling_rate=h.sampling_rate, harmonic_num=8 ) self.noise_convs = nn.ModuleList() self.conv_pre = weight_norm(Conv1d(h.num_mels, h.upsample_initial_channel, 7, 1, padding=3)) resblock = ResBlock1 if h.resblock == '1' else ResBlock2 self.ups = nn.ModuleList() for i, (u, k) in enumerate(zip(h.upsample_rates, h.upsample_kernel_sizes)): c_cur = h.upsample_initial_channel // (2 ** (i + 1)) self.ups.append(weight_norm( ConvTranspose1d(h.upsample_initial_channel // (2 ** i), h.upsample_initial_channel // (2 ** (i + 1)), k, u, padding=(k - u) // 2))) if i + 1 < len(h.upsample_rates): # stride_f0 = int(np.prod(h.upsample_rates[i + 1:])) self.noise_convs.append(Conv1d( 1, c_cur, kernel_size=stride_f0 * 2, stride=stride_f0, padding=stride_f0 // 2)) else: self.noise_convs.append(Conv1d(1, c_cur, kernel_size=1)) self.resblocks = nn.ModuleList() ch = h.upsample_initial_channel for i in range(len(self.ups)): ch //= 2 for j, (k, d) in enumerate(zip(h.resblock_kernel_sizes, h.resblock_dilation_sizes)): self.resblocks.append(resblock(h, ch, k, d)) self.conv_post = weight_norm(Conv1d(ch, 1, 7, 1, padding=3)) self.ups.apply(init_weights) self.conv_post.apply(init_weights) self.upp = int(np.prod(h.upsample_rates)) def forward(self, x, f0): har_source = self.m_source(f0, self.upp).transpose(1, 2) x = self.conv_pre(x) for i in range(self.num_upsamples): x = F.leaky_relu(x, LRELU_SLOPE) x = self.ups[i](x) x_source = self.noise_convs[i](har_source) x = x + x_source xs = None for j in range(self.num_kernels): if xs is None: xs = self.resblocks[i * self.num_kernels + j](x) else: xs += self.resblocks[i * self.num_kernels + j](x) x = xs / self.num_kernels x = F.leaky_relu(x) x = self.conv_post(x) x = torch.tanh(x) return x def remove_weight_norm(self): print('Removing weight norm...') for l in self.ups: remove_weight_norm(l) for l in self.resblocks: l.remove_weight_norm() remove_weight_norm(self.conv_pre) remove_weight_norm(self.conv_post) class DiscriminatorP(torch.nn.Module): def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False): super(DiscriminatorP, self).__init__() self.period = period norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))), norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(2, 0))), ]) self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0))) def forward(self, x): fmap = [] # 1d to 2d b, c, t = x.shape if t % self.period != 0: # pad first n_pad = self.period - (t % self.period) x = F.pad(x, (0, n_pad), "reflect") t = t + n_pad x = x.view(b, c, t // self.period, self.period) for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiPeriodDiscriminator(torch.nn.Module): def __init__(self, periods=None): super(MultiPeriodDiscriminator, self).__init__() self.periods = periods if periods is not None else [2, 3, 5, 7, 11] self.discriminators = nn.ModuleList() for period in self.periods: self.discriminators.append(DiscriminatorP(period)) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs class DiscriminatorS(torch.nn.Module): def __init__(self, use_spectral_norm=False): super(DiscriminatorS, self).__init__() norm_f = weight_norm if use_spectral_norm is False else spectral_norm self.convs = nn.ModuleList([ norm_f(Conv1d(1, 128, 15, 1, padding=7)), norm_f(Conv1d(128, 128, 41, 2, groups=4, padding=20)), norm_f(Conv1d(128, 256, 41, 2, groups=16, padding=20)), norm_f(Conv1d(256, 512, 41, 4, groups=16, padding=20)), norm_f(Conv1d(512, 1024, 41, 4, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 41, 1, groups=16, padding=20)), norm_f(Conv1d(1024, 1024, 5, 1, padding=2)), ]) self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1)) def forward(self, x): fmap = [] for l in self.convs: x = l(x) x = F.leaky_relu(x, LRELU_SLOPE) fmap.append(x) x = self.conv_post(x) fmap.append(x) x = torch.flatten(x, 1, -1) return x, fmap class MultiScaleDiscriminator(torch.nn.Module): def __init__(self): super(MultiScaleDiscriminator, self).__init__() self.discriminators = nn.ModuleList([ DiscriminatorS(use_spectral_norm=True), DiscriminatorS(), DiscriminatorS(), ]) self.meanpools = nn.ModuleList([ AvgPool1d(4, 2, padding=2), AvgPool1d(4, 2, padding=2) ]) def forward(self, y, y_hat): y_d_rs = [] y_d_gs = [] fmap_rs = [] fmap_gs = [] for i, d in enumerate(self.discriminators): if i != 0: y = self.meanpools[i - 1](y) y_hat = self.meanpools[i - 1](y_hat) y_d_r, fmap_r = d(y) y_d_g, fmap_g = d(y_hat) y_d_rs.append(y_d_r) fmap_rs.append(fmap_r) y_d_gs.append(y_d_g) fmap_gs.append(fmap_g) return y_d_rs, y_d_gs, fmap_rs, fmap_gs def feature_loss(fmap_r, fmap_g): loss = 0 for dr, dg in zip(fmap_r, fmap_g): for rl, gl in zip(dr, dg): loss += torch.mean(torch.abs(rl - gl)) return loss * 2 def discriminator_loss(disc_real_outputs, disc_generated_outputs): loss = 0 r_losses = [] g_losses = [] for dr, dg in zip(disc_real_outputs, disc_generated_outputs): r_loss = torch.mean((1 - dr) ** 2) g_loss = torch.mean(dg ** 2) loss += (r_loss + g_loss) r_losses.append(r_loss.item()) g_losses.append(g_loss.item()) return loss, r_losses, g_losses def generator_loss(disc_outputs): loss = 0 gen_losses = [] for dg in disc_outputs: l = torch.mean((1 - dg) ** 2) gen_losses.append(l) loss += l return loss, gen_losses so-vits-svc-npu/vdecoder/nsf_hifigan/nvSTFT.py000066400000000000000000000124241456161405400216300ustar00rootroot00000000000000import os import librosa import numpy as np import soundfile as sf import mindtorch.torch as torch import mindtorch.torch.nn.functional as F import mindtorch.torch.utils.data from librosa.filters import mel as librosa_mel_fn os.environ["LRU_CACHE_CAPACITY"] = "3" def load_wav_to_torch(full_path, target_sr=None, return_empty_on_exception=False): sampling_rate = None try: data, sampling_rate = sf.read(full_path, always_2d=True)# than soundfile. except Exception as ex: print(f"'{full_path}' failed to load.\nException:") print(ex) if return_empty_on_exception: return [], sampling_rate or target_sr or 48000 else: raise Exception(ex) if len(data.shape) > 1: data = data[:, 0] assert len(data) > 2# check duration of audio file is > 2 samples (because otherwise the slice operation was on the wrong dimension) if np.issubdtype(data.dtype, np.integer): # if audio data is type int max_mag = -np.iinfo(data.dtype).min # maximum magnitude = min possible value of intXX else: # if audio data is type fp32 max_mag = max(np.amax(data), -np.amin(data)) max_mag = (2**31)+1 if max_mag > (2**15) else ((2**15)+1 if max_mag > 1.01 else 1.0) # data should be either 16-bit INT, 32-bit INT or [-1 to 1] float32 data = torch.FloatTensor(data.astype(np.float32))/max_mag if (torch.isinf(data) | torch.isnan(data)).any() and return_empty_on_exception:# resample will crash with inf/NaN inputs. return_empty_on_exception will return empty arr instead of except return [], sampling_rate or target_sr or 48000 if target_sr is not None and sampling_rate != target_sr: data = torch.from_numpy(librosa.core.resample(data.numpy(), orig_sr=sampling_rate, target_sr=target_sr)) sampling_rate = target_sr return data, sampling_rate def dynamic_range_compression(x, C=1, clip_val=1e-5): return np.log(np.clip(x, a_min=clip_val, a_max=None) * C) def dynamic_range_decompression(x, C=1): return np.exp(x) / C def dynamic_range_compression_torch(x, C=1, clip_val=1e-5): return torch.log(torch.clamp(x, min=clip_val) * C) def dynamic_range_decompression_torch(x, C=1): return torch.exp(x) / C class STFT(): def __init__(self, sr=22050, n_mels=80, n_fft=1024, win_size=1024, hop_length=256, fmin=20, fmax=11025, clip_val=1e-5): self.target_sr = sr self.n_mels = n_mels self.n_fft = n_fft self.win_size = win_size self.hop_length = hop_length self.fmin = fmin self.fmax = fmax self.clip_val = clip_val self.mel_basis = {} self.hann_window = {} def get_mel(self, y, keyshift=0, speed=1, center=False): sampling_rate = self.target_sr n_mels = self.n_mels n_fft = self.n_fft win_size = self.win_size hop_length = self.hop_length fmin = self.fmin fmax = self.fmax clip_val = self.clip_val factor = 2 ** (keyshift / 12) n_fft_new = int(np.round(n_fft * factor)) win_size_new = int(np.round(win_size * factor)) hop_length_new = int(np.round(hop_length * speed)) if torch.min(y) < -1.: print('min value is ', torch.min(y)) if torch.max(y) > 1.: print('max value is ', torch.max(y)) mel_basis_key = str(fmax)+'_'+str(y.device) if mel_basis_key not in self.mel_basis: mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=n_mels, fmin=fmin, fmax=fmax) self.mel_basis[mel_basis_key] = torch.from_numpy(mel).float().to(y.device) keyshift_key = str(keyshift)+'_'+str(y.device) if keyshift_key not in self.hann_window: self.hann_window[keyshift_key] = torch.hann_window(win_size_new).to(y.device) pad_left = (win_size_new - hop_length_new) //2 pad_right = max((win_size_new- hop_length_new + 1) //2, win_size_new - y.size(-1) - pad_left) if pad_right < y.size(-1): mode = 'reflect' else: mode = 'constant' y = torch.nn.functional.pad(y.unsqueeze(1), (pad_left, pad_right), mode = mode) y = y.squeeze(1) spec = torch.stft(y, n_fft_new, hop_length=hop_length_new, win_length=win_size_new, window=self.hann_window[keyshift_key], center=center, pad_mode='reflect', normalized=False, onesided=True, return_complex=False) # print(111,spec) spec = torch.sqrt(spec.pow(2).sum(-1)+(1e-9)) if keyshift != 0: size = n_fft // 2 + 1 resize = spec.size(1) if resize < size: spec = F.pad(spec, (0, 0, 0, size-resize)) spec = spec[:, :size, :] * win_size / win_size_new # print(222,spec) spec = torch.matmul(self.mel_basis[mel_basis_key], spec) # print(333,spec) spec = dynamic_range_compression_torch(spec, clip_val=clip_val) # print(444,spec) return spec def __call__(self, audiopath): audio, sr = load_wav_to_torch(audiopath, target_sr=self.target_sr) spect = self.get_mel(audio.unsqueeze(0)).squeeze(0) return spect stft = STFT() so-vits-svc-npu/vdecoder/nsf_hifigan/utils.py000066400000000000000000000035701456161405400216460ustar00rootroot00000000000000import glob import os import matplotlib import matplotlib.pylab as plt import mindtorch.torch as torch from mindtorch.torch.nn.utils import weight_norm matplotlib.use("Agg") def plot_spectrogram(spectrogram): fig, ax = plt.subplots(figsize=(10, 2)) im = ax.imshow(spectrogram, aspect="auto", origin="lower", interpolation='none') plt.colorbar(im, ax=ax) fig.canvas.draw() plt.close() return fig def init_weights(m, mean=0.0, std=0.01): classname = m.__class__.__name__ if classname.find("Conv") != -1: m.weight.data.normal_(mean, std) def apply_weight_norm(m): classname = m.__class__.__name__ if classname.find("Conv") != -1: weight_norm(m) def get_padding(kernel_size, dilation=1): return int((kernel_size*dilation - dilation)/2) def load_checkpoint(filepath, device): assert os.path.isfile(filepath) print("Loading '{}'".format(filepath)) checkpoint_dict = torch.load(filepath, map_location=device) print("Complete.") return checkpoint_dict def save_checkpoint(filepath, obj): print("Saving checkpoint to {}".format(filepath)) torch.save(obj, filepath) print("Complete.") def del_old_checkpoints(cp_dir, prefix, n_models=2): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) # get checkpoint paths cp_list = sorted(cp_list)# sort by iter if len(cp_list) > n_models: # if more than n_models models are found for cp in cp_list[:-n_models]:# delete the oldest models other than lastest n_models open(cp, 'w').close()# empty file contents os.unlink(cp)# delete file (move to trash when using Colab) def scan_checkpoint(cp_dir, prefix): pattern = os.path.join(cp_dir, prefix + '????????') cp_list = glob.glob(pattern) if len(cp_list) == 0: return None return sorted(cp_list)[-1] so-vits-svc-npu/vencoder/000077500000000000000000000000001456161405400156665ustar00rootroot00000000000000so-vits-svc-npu/vencoder/CNHubertLarge.py000066400000000000000000000024521456161405400206700ustar00rootroot00000000000000import mindtorch.torch as torch from fairseq import checkpoint_utils from vencoder.encoder import SpeechEncoder class CNHubertLarge(SpeechEncoder): def __init__(self, vec_path="pretrain/chinese-hubert-large-fairseq-ckpt.pt", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 1024 models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task( [vec_path], suffix="", ) if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.model = models[0].to(self.dev) self.model.eval() def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) padding_mask = torch.BoolTensor(feats.shape).fill_(False) inputs = { "source": feats.to(wav.device), "padding_mask": padding_mask.to(wav.device) } with torch.no_grad(): logits = self.model.extract_features(**inputs) return logits[0].transpose(1, 2)so-vits-svc-npu/vencoder/ContentVec256L12_Onnx.py000066400000000000000000000023441456161405400220310ustar00rootroot00000000000000import onnxruntime import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder class ContentVec256L12_Onnx(SpeechEncoder): def __init__(self, vec_path="pretrain/vec-256-layer-12.onnx", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 256 if device is None: self.dev = torch.device("cpu") else: self.dev = torch.device(device) if device == 'cuda' or device == torch.device("cuda"): providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] else: providers = ['CPUExecutionProvider'] self.model = onnxruntime.InferenceSession(vec_path, providers=providers) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) feats = feats.unsqueeze(0).cpu().detach().numpy() onnx_input = {self.model.get_inputs()[0].name: feats} logits = self.model.run(None, onnx_input) return torch.tensor(logits[0]).transpose(1, 2).to(self.dev) so-vits-svc-npu/vencoder/ContentVec256L9.py000066400000000000000000000025771456161405400207650ustar00rootroot00000000000000import mindtorch.torch as torch from fairseq import checkpoint_utils from vencoder.encoder import SpeechEncoder class ContentVec256L9(SpeechEncoder): def __init__(self, vec_path="pretrain/checkpoint_best_legacy_500.pt", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task( [vec_path], suffix="", ) self.hidden_dim = 256 if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.model = models[0].to(self.dev) self.model.eval() def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) padding_mask = torch.BoolTensor(feats.shape).fill_(False) inputs = { "source": feats.to(wav.device), "padding_mask": padding_mask.to(wav.device), "output_layer": 9, # layer 9 } with torch.no_grad(): logits = self.model.extract_features(**inputs) feats = self.model.final_proj(logits[0]) return feats.transpose(1, 2) so-vits-svc-npu/vencoder/ContentVec256L9_Onnx.py000066400000000000000000000024471456161405400217630ustar00rootroot00000000000000import onnxruntime import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder class ContentVec256L9_Onnx(SpeechEncoder): def __init__(self, vec_path="pretrain/vec-256-layer-9.onnx", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 256 if device is None: self.dev = torch.device("cpu") else: self.dev = torch.device(device) if device == 'cpu' or device == torch.device("cpu") or device is None: providers = ['CPUExecutionProvider'] elif device == 'cuda' or device == torch.device("cuda"): providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] self.model = onnxruntime.InferenceSession(vec_path, providers=providers) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) feats = feats.unsqueeze(0).cpu().detach().numpy() onnx_input = {self.model.get_inputs()[0].name: feats} logits = self.model.run(None, onnx_input) return torch.tensor(logits[0]).transpose(1, 2).to(self.dev) so-vits-svc-npu/vencoder/ContentVec768L12.py000066400000000000000000000025211456161405400210340ustar00rootroot00000000000000import mindtorch.torch as torch from fairseq import checkpoint_utils from vencoder.encoder import SpeechEncoder class ContentVec768L12(SpeechEncoder): def __init__(self, vec_path="pretrain/checkpoint_best_legacy_500.pt", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 768 models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task( [vec_path], suffix="", ) if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.model = models[0].to(self.dev) self.model.eval() def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) padding_mask = torch.BoolTensor(feats.shape).fill_(False) inputs = { "source": feats.to(wav.device), "padding_mask": padding_mask.to(wav.device), "output_layer": 12, # layer 12 } with torch.no_grad(): logits = self.model.extract_features(**inputs) return logits[0].transpose(1, 2) so-vits-svc-npu/vencoder/ContentVec768L12_Onnx.py000066400000000000000000000023601456161405400220370ustar00rootroot00000000000000import onnxruntime import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder class ContentVec768L12_Onnx(SpeechEncoder): def __init__(self, vec_path="pretrain/vec-768-layer-12.onnx", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 768 if device is None: self.dev = torch.device("cpu") else: self.dev = torch.device(device) if device == 'cuda' or device == torch.device("cuda"): providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] else: providers = ['CPUExecutionProvider'] self.model = onnxruntime.InferenceSession(vec_path, providers=providers) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) feats = feats.unsqueeze(0).cpu().detach().numpy() onnx_input = {self.model.get_inputs()[0].name: feats} logits = self.model.run(None, onnx_input) return torch.tensor(logits[0]).transpose(1, 2).to(self.dev) so-vits-svc-npu/vencoder/ContentVec768L9_Onnx.py000066400000000000000000000023201456161405400217610ustar00rootroot00000000000000import onnxruntime import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder class ContentVec768L9_Onnx(SpeechEncoder): def __init__(self,vec_path = "pretrain/vec-768-layer-9.onnx",device=None): super().__init__() import logger logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 768 if device is None: self.dev = torch.device("cpu") else: self.dev = torch.device(device) if device == 'cuda' or device == torch.device("cuda"): providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] else: providers = ['CPUExecutionProvider'] self.model = onnxruntime.InferenceSession(vec_path, providers=providers) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) feats = feats.unsqueeze(0).cpu().detach().numpy() onnx_input = {self.model.get_inputs()[0].name: feats} logits = self.model.run(None, onnx_input) return torch.tensor(logits[0]).transpose(1, 2).to(self.dev) so-vits-svc-npu/vencoder/DPHubert.py000066400000000000000000000021361456161405400177170ustar00rootroot00000000000000import mindtorch.torch as torch from vencoder.dphubert.model import wav2vec2_model from vencoder.encoder import SpeechEncoder class DPHubert(SpeechEncoder): def __init__(self, vec_path="pretrain/DPHuBERT-sp0.75.pth", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) ckpt = torch.load(vec_path) self.hidden_dim = 768 self.model = wav2vec2_model(**ckpt["config"]).to(self.dev) self.model.load_state_dict(ckpt["state_dict"], strict=False) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats[None, :] with torch.no_grad(): with torch.inference_mode(): units = self.model(feats)[0] return units.transpose(1,2) so-vits-svc-npu/vencoder/HubertSoft.py000066400000000000000000000020361456161405400203260ustar00rootroot00000000000000import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder from vencoder.hubert import hubert_model class HubertSoft(SpeechEncoder): def __init__(self, vec_path="pretrain/hubert-soft-0d54a1f4.pt", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) hubert_soft = hubert_model.hubert_soft(vec_path) if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.hidden_dim = 256 self.model = hubert_soft.to(self.dev) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats[None,None,:] with torch.no_grad(): with torch.inference_mode(): units = self.model.units(feats) return units.transpose(1,2) so-vits-svc-npu/vencoder/HubertSoft_Onnx.py000066400000000000000000000023451456161405400213330ustar00rootroot00000000000000import onnxruntime import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder class HubertSoft_Onnx(SpeechEncoder): def __init__(self, vec_path="pretrain/hubert-soft.onnx", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) self.hidden_dim = 256 if device is None: self.dev = torch.device("cpu") else: self.dev = torch.device(device) if device == 'cuda' or device == torch.device("cuda"): providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] else: providers = ['CPUExecutionProvider'] self.model = onnxruntime.InferenceSession(vec_path, providers=providers) def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() feats = feats.view(1, -1) feats = feats.unsqueeze(0).cpu().detach().numpy() onnx_input = {self.model.get_inputs()[0].name: feats} logits = self.model.run(None, onnx_input) return torch.tensor(logits[0]).transpose(1, 2).to(self.dev) so-vits-svc-npu/vencoder/WavLMBasePlus.py000066400000000000000000000024141456161405400206660ustar00rootroot00000000000000import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder from vencoder.wavlm.WavLM import WavLM, WavLMConfig class WavLMBasePlus(SpeechEncoder): def __init__(self, vec_path="pretrain/WavLM-Base+.pt", device=None, log=True): super().__init__() import logger if log: logger.info("load model(s) from {}".format(vec_path)) checkpoint = torch.load(vec_path) self.cfg = WavLMConfig(checkpoint['cfg']) if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) self.hidden_dim = self.cfg.encoder_embed_dim self.model = WavLM(self.cfg) self.model.load_state_dict(checkpoint['model']) self.model.to(self.dev).eval() def encoder(self, wav): feats = wav if feats.dim() == 2: # double channels feats = feats.mean(-1) assert feats.dim() == 1, feats.dim() if self.cfg.normalize: feats = torch.nn.functional.layer_norm(feats, feats.shape) with torch.no_grad(): with torch.inference_mode(): units = self.model.extract_features(feats[None, :])[0] return units.transpose(1, 2) so-vits-svc-npu/vencoder/WhisperPPG.py000066400000000000000000000023011456161405400202240ustar00rootroot00000000000000import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder from vencoder.whisper.audio import log_mel_spectrogram, pad_or_trim from vencoder.whisper.model import ModelDimensions, Whisper class WhisperPPG(SpeechEncoder): def __init__(self, vec_path="pretrain/medium.pt", device=None): super().__init__() if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) checkpoint = torch.load(vec_path, map_location=device) dims = ModelDimensions(**checkpoint["dims"]) model = Whisper(dims) model.load_state_dict(checkpoint["model_state_dict"]) self.hidden_dim = dims self.model = model.to(self.dev) def encoder(self, wav): audio = wav audln = audio.shape[0] ppgln = audln // 320 audio = pad_or_trim(audio) mel = log_mel_spectrogram(audio).to(self.dev) with torch.no_grad(): ppg = self.model.encoder(mel.unsqueeze(0)).squeeze().data.cpu().float().numpy() ppg = torch.FloatTensor(ppg[:ppgln, ]).to(self.dev) return ppg[None, :, :].transpose(1, 2) so-vits-svc-npu/vencoder/WhisperPPGLarge.py000066400000000000000000000023101456161405400211770ustar00rootroot00000000000000import mindtorch.torch as torch from vencoder.encoder import SpeechEncoder from vencoder.whisper.audio import log_mel_spectrogram, pad_or_trim from vencoder.whisper.model import ModelDimensions, Whisper class WhisperPPGLarge(SpeechEncoder): def __init__(self, vec_path="pretrain/large-v2.pt", device=None): super().__init__() if device is None: self.dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") else: self.dev = torch.device(device) checkpoint = torch.load(vec_path, map_location=device) dims = ModelDimensions(**checkpoint["dims"]) model = Whisper(dims) model.load_state_dict(checkpoint["model_state_dict"]) self.hidden_dim = dims self.model = model.to(self.dev) def encoder(self, wav): audio = wav audln = audio.shape[0] ppgln = audln // 320 audio = pad_or_trim(audio) mel = log_mel_spectrogram(audio).to(self.dev) with torch.no_grad(): ppg = self.model.encoder(mel.unsqueeze(0)).squeeze().data.cpu().float().numpy() ppg = torch.FloatTensor(ppg[:ppgln, ]).to(self.dev) return ppg[None, :, :].transpose(1, 2) so-vits-svc-npu/vencoder/__init__.py000066400000000000000000000000001456161405400177650ustar00rootroot00000000000000so-vits-svc-npu/vencoder/dphubert/000077500000000000000000000000001456161405400175035ustar00rootroot00000000000000so-vits-svc-npu/vencoder/dphubert/__init__.py000066400000000000000000000000001456161405400216020ustar00rootroot00000000000000so-vits-svc-npu/vencoder/dphubert/components.py000066400000000000000000001624701456161405400222540ustar00rootroot00000000000000"""Building blocks for speech SSL models supporting pruning. Originally from: https://github.com/pytorch/audio/blob/main/torchaudio/models/wav2vec2/components.py """ import math from collections import defaultdict from typing import List, Optional, Tuple import mindtorch.torch as torch from mindtorch.torch import Tensor, nn from mindtorch.torch.nn import Module from .hardconcrete import HardConcrete from .pruning_utils import ( prune_conv1d_layer, prune_layer_norm, prune_linear_layer, ) def _init_transformer_params(module): """ Initialize the weights of Transformer module in Wav2Vec2/HuBERT. If the module is ``nn.Linear``, normalize the weight with mean 0 and standard deviation 0.02. If ``bias`` is set to ``True`` in the module, set ``bias`` to 0. If the module is ``nn.Embedding``, normalize the weight with mean 0 and standard deviation 0.02. If ``padding_idx`` is not None, set the weight of padding to 0. Note: Ths method corresponds to `init_bert_params `__ in the original ``fairseq`` implementation. """ def normal_(data): data.copy_(data.cpu().normal_(mean=0.0, std=0.02).to(data.device)) if isinstance(module, nn.Linear): normal_(module.weight.data) if module.bias is not None: module.bias.data.zero_() if isinstance(module, nn.Embedding): normal_(module.weight.data) if module.padding_idx is not None: module.weight.data[module.padding_idx].zero_() class LayerNorm(nn.LayerNorm): """Layer norm with transpose""" def forward(self, input: Tensor) -> Tensor: x = input.transpose(-2, -1) x = nn.functional.layer_norm(x, self.normalized_shape, self.weight, self.bias, self.eps) x = x.transpose(-2, -1) return x class ConvLayerBlock(Module): """Convolution unit of FeatureExtractor""" def __init__( self, in_channels: int, out_channels: int, kernel_size: int, stride: int, bias: bool, layer_norm: Optional[Module], prune_conv_channels: bool = False, ): super().__init__() self.kernel_size = kernel_size self.stride = stride self.layer_norm = layer_norm self.conv = nn.Conv1d( in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride, bias=bias, ) if prune_conv_channels: self.hard_concrete = HardConcrete(n_in=out_channels, init_mean=0.01) else: self.hard_concrete = None def forward( self, x: Tensor, length: Optional[Tensor], ) -> Tuple[Tensor, Optional[Tensor]]: """ Args: x (Tensor): Shape: ``[batch, in_channels, in_frame]``. length (Tensor or None, optional): Shape ``[batch, ]``. Returns: Tensor: Shape ``[batch, out_channels, out_frames]``. Optional[Tensor]: Shape ``[batch, ]``. """ x = self.conv(x) if self.layer_norm is not None: x = self.layer_norm(x) x = nn.functional.gelu(x) if self.hard_concrete is not None: channel_mask = self.hard_concrete() # hard concrete mask, (out_channels,) x = x * channel_mask.unsqueeze(-1) if length is not None: length = torch.div(length - self.kernel_size, self.stride, rounding_mode="floor") + 1 # When input length is 0, the resulting length can be negative. So fix it here. length = torch.max(torch.zeros_like(length), length) return x, length def get_num_params_and_out_channels(self, in_channels): if self.hard_concrete is not None: out_channels = self.hard_concrete.l0_norm() else: out_channels = self.conv.out_channels num_params = in_channels * out_channels * self.kernel_size if self.conv.bias is not None: num_params += out_channels if self.layer_norm is not None: num_params += out_channels * 2 return num_params, out_channels class FeatureExtractor(Module): """Extract features from audio Args: conv_layers (nn.ModuleList): convolution layers """ def __init__( self, conv_layers: nn.ModuleList, ): super().__init__() self.conv_layers = conv_layers # NOTE: a dummy weight used to save the soft mask of the last conv layer self.dummy_weight = nn.Parameter( torch.ones(conv_layers[-1].conv.out_channels, dtype=torch.float32), requires_grad=False ) def forward( self, x: Tensor, length: Optional[Tensor], ) -> Tuple[Tensor, Optional[Tensor]]: """ Args: x (Tensor): Input Tensor representing a batch of audio, shape: ``[batch, time]``. length (Tensor or None, optional): Valid length of each input sample. shape: ``[batch, ]``. Returns: Tensor: The resulting feature, shape: ``[batch, frame, feature]`` Optional[Tensor]: Valid length of each output sample. shape: ``[batch, ]``. """ if x.ndim != 2: raise ValueError("Expected the input Tensor to be 2D (batch, time), " "but received {list(x.shape)}") x = x.unsqueeze(1) # (batch, channel==1, frame) for layer in self.conv_layers: x, length = layer(x, length) # (batch, feature, frame) x = x.transpose(1, 2) # (batch, frame, feature) x = x * self.dummy_weight return x, length def get_num_params_and_final_out_channels(self): in_channels = 1 num_params = 0 for layer in self.conv_layers: layer_params, in_channels = layer.get_num_params_and_out_channels(in_channels) num_params += layer_params num_params += in_channels # dummy weight return num_params, in_channels def prune(self): """"Prune conv layers and dummy weight based on hardconcrete parameters. This is an in-place operation. """ new_config = [] # [(output_channel, kernel_size, stride), ...] for idx, layer in enumerate(self.conv_layers): if layer.hard_concrete is not None: assert not layer.hard_concrete.training mask = layer.hard_concrete() # (out_features,) index = mask.nonzero().squeeze(-1) # 2D -> 1D assert len(index) > 0, f"Conv channels pruned to zero at index {idx}" new_config.append( (len(index), layer.kernel_size, layer.stride) ) # prune the current layer prune_conv1d_layer(layer.conv, index, "output") if layer.layer_norm is not None: prune_layer_norm(layer.layer_norm, index) # prune the next layer if idx == len(self.conv_layers) - 1: self.dummy_weight.data *= mask self.dummy_weight = nn.Parameter( self.dummy_weight.index_select(0, index).clone().detach(), requires_grad=False ) else: self.conv_layers[idx+1].conv.weight.data *= mask.unsqueeze(-1) prune_conv1d_layer(self.conv_layers[idx+1].conv, index, dim="input") layer.hard_concrete = None else: new_config.append( (layer.conv.out_channels, layer.kernel_size, layer.stride) ) index = torch.arange(layer.conv.out_channels, dtype=torch.long) return new_config, index class FeatureProjection(Module): """Layer that connects FeatureExtractor and Encoder Projects features to encoder dimension. Args: in_features (int): Input feature dim. out_features (int): Output feature dim. dropout (float): Dropout probability. """ def __init__( self, in_features: int, out_features: int, dropout: float, ): super().__init__() self.layer_norm = nn.LayerNorm(in_features) self.projection = nn.Linear( in_features, out_features, ) self.dropout = nn.Dropout(dropout) def forward(self, x): """ Args: x (Tensor): Feature Tensor. shape: ``[batch, frame, in_feature]`` Returns: Tensor: Projected features. ``[batch, frame, out_feature]``. """ x = self.layer_norm(x) x = self.projection(x) x = self.dropout(x) return x def get_num_params(self, in_features): return in_features * 2 + (in_features + 1) * self.projection.out_features class ConvolutionalPositionalEmbedding(Module): """Positional embedding which is placed at the beginning of Transformer. Args: embed_dim (int): Feature dimension of the input Tensor. kernel_size (int): The number of frames to be use. groups (int): The number of groups in feature dimensions. """ def __init__( self, embed_dim: int, kernel_size: int, groups: int, ): super().__init__() self.embed_dim = embed_dim self.kernel_size = kernel_size self.conv = nn.Conv1d( in_channels=embed_dim, out_channels=embed_dim, kernel_size=kernel_size, padding=kernel_size // 2, groups=groups, ) self.conv = nn.utils.weight_norm(self.conv, name="weight", dim=2) self.num_remove: int = 1 if kernel_size % 2 == 0 else 0 def __prepare_scriptable__(self): for hook in self.conv._forward_pre_hooks.values(): # The hook we want to remove is an instance of WeightNorm class, so # normally we would do `if isinstance(...)` but this class is not accessible # because of shadowing, so we check the module name directly. # https://github.com/pytorch/pytorch/blob/be0ca00c5ce260eb5bcec3237357f7a30cc08983/torch/nn/utils/__init__.py#L3 if hook.__module__ == "torch.nn.utils.weight_norm" and hook.__class__.__name__ == "WeightNorm": torch.nn.utils.remove_weight_norm(self.conv) return self def forward(self, x): """ Args: x (Tensor): shape ``[batch, frame, feature]``. Returns: Tensor: The resulting feature. Shape ``[batch, frame, feature]``. """ x = x.transpose(-2, -1) x = self.conv(x) if self.num_remove > 0: x = x[..., : -self.num_remove] x = torch.nn.functional.gelu(x) x = x.transpose(-2, -1) return x class SelfAttention(Module): """Multihead Self Attention module Args: embed_dim (int): Total dimension of the model. num_heads (int): The number of heads. dropout (float, optional): Dropout probability on attn_output_weights. Default: ``0.0`` """ def __init__( self, embed_dim: int, num_heads: int, head_dim: int, dropout: float = 0.0, prune_heads: bool = False, # whether to prune attention heads prune_layer: bool = False, # whether to prune entire attention layers ): super().__init__() self.embed_dim = embed_dim self.num_heads = num_heads self.head_dim = head_dim self.dropout = torch.nn.Dropout(dropout) self.scaling = self.head_dim**-0.5 self.k_proj = nn.Linear(embed_dim, num_heads * head_dim, bias=True) self.v_proj = nn.Linear(embed_dim, num_heads * head_dim, bias=True) self.q_proj = nn.Linear(embed_dim, num_heads * head_dim, bias=True) self.out_proj = nn.Linear(num_heads * head_dim, embed_dim, bias=True) if prune_heads: self.hard_concrete_for_heads = HardConcrete(n_in=num_heads, init_mean=0.01) else: self.hard_concrete_for_heads = None if prune_layer: self.hard_concrete_for_layer = HardConcrete(n_in=1, init_mean=0.01) else: self.hard_concrete_for_layer = None def forward( self, x: Tensor, attention_mask: Optional[Tensor] = None, position_bias: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, ) -> Tuple[Tensor, Optional[Tensor]]: """ Args: x (Tensor): shape: ``[batch_size, sequence_length, embed_dim]``. attention_mask (Tensor or ``None``, optional): shape: ``[batch_size, 1, sequence_length, sequence_length]`` position_bias: Not used. Only for the compatibility with :py:class:`WavLMSelfAttention`. key_padding_mask (Tensor or ``None``): Not used. Only for the compatibility with :py:class:`WavLMSelfAttention`. Returns: (Tensor, ``None``): The resulting attention output and ``None`` (necessary for compatibility with :py:class:`WavLMSelAttention`). Attention output shape: ``[batch, sequence_length, embed_dim]``. """ if x.ndim != 3 or x.shape[2] != self.embed_dim: raise ValueError( f"The expected input shape is (batch, sequence, embed_dim=={self.embed_dim}). " f"Found {x.shape}." ) batch_size, length, embed_dim = x.size() shape = (batch_size, length, self.num_heads, self.head_dim) q = self.q_proj(x).view(*shape).transpose(2, 1) # B, nH, L, Hd k = self.k_proj(x).view(*shape).permute(0, 2, 3, 1) # B, nH, Hd, L v = self.v_proj(x).view(*shape).transpose(2, 1) # B, nH, L, Hd # scale down q to avoid value overflow. weights = (self.scaling * q) @ k # B, nH, L, L if attention_mask is not None: weights += attention_mask # subtracting a constant value from the tensor won't change the output of softmax. # apply the subtraction to avoid value overflow in torch.nn.functional.softmax. # for more details, please see Equation 7 in https://arxiv.org/abs/2112.08778 weights = weights - weights.max(dim=-1, keepdim=True)[0] weights = torch.nn.functional.softmax(weights, dim=-1) weights = self.dropout(weights) output = weights @ v # B, nH, L, Hd if self.hard_concrete_for_heads is not None: head_mask = self.hard_concrete_for_heads() # (nH,) output = output * head_mask.unsqueeze(-1).unsqueeze(-1) output = output.transpose(2, 1).reshape(batch_size, length, self.num_heads * self.head_dim) output = self.out_proj(output) if self.hard_concrete_for_layer is not None: layer_mask = self.hard_concrete_for_layer() # (1,) output = output * layer_mask return output, None # Necessary for compatibility with WavLMSelAttention def get_num_params(self): if self.hard_concrete_for_heads is not None: num_heads = self.hard_concrete_for_heads.l0_norm() else: num_heads = self.num_heads num_params = (self.embed_dim + 1) * num_heads * self.head_dim * 3 \ + (num_heads * self.head_dim + 1) * self.embed_dim if self.hard_concrete_for_layer is not None: num_params *= self.hard_concrete_for_layer.l0_norm() return num_params def prune(self): new_config = { "use_attention": True, "num_heads": self.num_heads, } if self.hard_concrete_for_layer is not None: assert not self.hard_concrete_for_layer.training layer_mask = self.hard_concrete_for_layer() # (1,) self.out_proj.weight.data *= layer_mask self.out_proj.bias.data *= layer_mask if layer_mask == 0: new_config["use_attention"] = False self.hard_concrete_for_layer = None if self.hard_concrete_for_heads is not None: assert not self.hard_concrete_for_heads.training head_mask = self.hard_concrete_for_heads() # (num_heads,) new_config["num_heads"] = len(head_mask.nonzero()) if new_config["num_heads"] == 0: new_config["use_attention"] = False else: full_mask = head_mask.repeat_interleave(self.head_dim) full_index = full_mask.nonzero().squeeze(-1) # 1D prune_linear_layer(self.k_proj, full_index, "output") prune_linear_layer(self.v_proj, full_index, "output") prune_linear_layer(self.q_proj, full_index, "output") self.out_proj.weight.data *= full_mask prune_linear_layer(self.out_proj, full_index, "input") self.hard_concrete_for_heads = None return new_config class WavLMSelfAttention(SelfAttention): """Multi-headed self-attention for WavLM model :cite:`chen2022wavlm`. Args: embed_dim (int): Total dimension of the model. num_heads (int): The number of heads. dropout (float, optional): Dropout probability on attn_output_weights. (Default: to ``0.0``) bias (bool, optional): If ``True``, add bias to input / output projection layers. (Default: ``True``) has_relative_attention_bias (bool, optional): If ``True``, apply relative position embedding. Necessary in the first encoder layer, but not in the subsequent ones. (Default: ``False``) num_buckets (int, optional): Number of buckets for relative position embedding. (Default: ``32``) max_distance (int, optional): Naximum distance for relative position embedding. (Default: ``128``) gru_rel_pos (bool, optional): If ``True``, apply gated relative position embedding. (Default: ``False``) """ def __init__( self, embed_dim: int, total_num_heads: int, remaining_heads: Optional[List[int]] = None, dropout: float = 0.0, bias: bool = True, has_relative_attention_bias: bool = False, num_buckets: int = 32, max_distance: int = 128, gru_rel_pos: bool = True, prune_heads: bool = False, prune_layer: bool = False, ): self.total_num_heads = total_num_heads if remaining_heads is None: self.remaining_heads = list(range(total_num_heads)) else: self.remaining_heads = remaining_heads # list of indices self.head_dim = embed_dim // total_num_heads super().__init__(embed_dim, len(self.remaining_heads), self.head_dim, dropout, prune_heads, prune_layer) self.has_relative_attention_bias = has_relative_attention_bias self.num_buckets = num_buckets self.max_distance = max_distance if has_relative_attention_bias: self.rel_attn_embed = nn.Embedding(num_buckets, total_num_heads) else: self.rel_attn_embed = None # override linear layers to customize bias self.k_proj = nn.Linear(embed_dim, len(self.remaining_heads) * self.head_dim, bias=bias) self.v_proj = nn.Linear(embed_dim, len(self.remaining_heads) * self.head_dim, bias=bias) self.q_proj = nn.Linear(embed_dim, len(self.remaining_heads) * self.head_dim, bias=bias) self.out_proj = nn.Linear(len(self.remaining_heads) * self.head_dim, embed_dim, bias=bias) self.gru_rel_pos = gru_rel_pos if self.gru_rel_pos: self.gru_rel_pos_linear = nn.Linear(self.head_dim, 8) self.gru_rel_pos_const = nn.Parameter(torch.ones(1, total_num_heads, 1, 1)) self.has_position_bias = True def compute_bias(self, query_length: int, key_length: int) -> Tensor: """Compute relative position embeddings for WavLM model. Args: query_length (int): Query position can take values between 0 and ``query_length - 1``. key_length (int): Key position can take values between 0 and ``key_length - 1``. Returns: Tensor of shape `(num_heads, query_length, key_length)`, relative positions embeddings """ context_position = torch.arange(query_length, dtype=torch.long)[:, None] memory_position = torch.arange(key_length, dtype=torch.long)[None, :] relative_position = memory_position - context_position # Shape (query_length, key_length) relative_position_bucket = self._relative_positions_bucket(relative_position, bidirectional=True) relative_position_bucket = relative_position_bucket.to(self.rel_attn_embed.weight.device) values = self.rel_attn_embed(relative_position_bucket) # Shape (query_length, key_length, num_heads) values = values.permute([2, 0, 1]) return values def _relative_positions_bucket(self, relative_positions: Tensor, bidirectional: bool = True): """Compute relative position buckets for WavLM model. Computation similar to formula (5) in WavLM paper :cite:`chen2022wavlm`. Args: relative_positions (Tensor): Relative offsets between query and key positions, of shape ``(query_length, key_length)``. bidirectional (bool): If ``True``, values will be filled both above and below the diagonal in the resulting matrix. If ``False``, the elements above the diagonal (i.e. with negative relative offsets) will be set to zero. (Default ``True``) Returns: Tensor of shape ``(query_length, key_length)`` filled bucketed values of with relative positions. """ num_buckets = self.num_buckets max_distance = self.max_distance # Shape (query_length, key_length) relative_buckets = torch.zeros_like(relative_positions, dtype=torch.long) if bidirectional: num_buckets = num_buckets // 2 relative_buckets += (relative_positions > 0).to(torch.long) * num_buckets relative_positions = torch.abs(relative_positions) else: relative_positions = -torch.min(relative_positions, torch.zeros_like(relative_positions)) max_exact = num_buckets // 2 is_small = relative_positions < max_exact relative_postion_if_large = max_exact + ( torch.log(relative_positions.float() / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact) ).to(torch.long) relative_postion_if_large = torch.min( relative_postion_if_large, torch.full_like(relative_postion_if_large, num_buckets - 1) ) relative_buckets += torch.where(is_small, relative_positions, relative_postion_if_large) return relative_buckets def forward( self, query: Tensor, attention_mask: Optional[Tensor] = None, position_bias: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, ) -> Tuple[Tensor, Optional[Tensor]]: """ Args: query (Tensor): Input of shape ``(batch_size, src_len, embed_dim)``. key_padding_mask (Tensor or None, optional): Mask to exclude keys that are pads, of shape `(batch, src_len)`, where padding elements are indicated by 1s. (Default: ``None``) attn_mask: Needs to be ``None``. The argument exists for compatibility with ``EncoderLayer``. (Default: ``None``) position_bias (Tensor or None, optional): Position bias of shape ``(batch_size * num_heads, src_len, src_len)``. When used inside WavLM model encoder, will be generated in the first layer and then passed from each encoder layer to the next one. (Default: ``None``) Returns: attn_output (Tensor): Attention output of shape ``(batch_size, src_len, embed_dim)``. position_bias (Tensor or None): Position bias of shape ``(batch_size * num_heads, src_len, src_len)``. """ bsz, seq_len, embed_dim = query.size() assert embed_dim == self.embed_dim assert key_padding_mask is None # only for the first layer if self.rel_attn_embed is not None and position_bias is None: position_bias = self.compute_bias(seq_len, seq_len) position_bias = position_bias.unsqueeze(0).repeat(bsz, 1, 1, 1).view(bsz * self.total_num_heads, seq_len, seq_len) attn_mask_rel_pos: Optional[Tensor] = None if position_bias is not None: attn_mask_rel_pos = position_bias if self.gru_rel_pos: # Apply gating on relative position bias query_layer = query.view(bsz, seq_len, self.total_num_heads, -1) query_layer = query_layer.permute(0, 2, 1, 3) gate_a, gate_b = torch.sigmoid( self.gru_rel_pos_linear(query_layer).view(bsz, self.total_num_heads, seq_len, 2, 4).sum(-1, keepdim=False) ).chunk(2, dim=-1) gate_a_1 = gate_a * (gate_b * self.gru_rel_pos_const - 1.0) + 2.0 attn_mask_rel_pos = gate_a_1.view(bsz * self.total_num_heads, -1, 1) * position_bias attn_mask_rel_pos = attn_mask_rel_pos.view((-1, seq_len, seq_len)) attn_mask_rel_pos = attn_mask_rel_pos.reshape(bsz, self.total_num_heads, seq_len, seq_len)[:, self.remaining_heads, :, :] attn_mask = attn_mask_rel_pos if attention_mask is not None: attn_mask = attn_mask + attention_mask if key_padding_mask is not None: attn_mask = attn_mask.masked_fill( key_padding_mask.reshape(bsz, 1, 1, seq_len), float("-inf") ) attn_output, _ = super().forward(query, attention_mask=attn_mask) return attn_output, position_bias def prune(self): new_config = { "use_attention": True, "remaining_heads": self.remaining_heads, } if self.hard_concrete_for_layer is not None: assert not self.hard_concrete_for_layer.training layer_mask = self.hard_concrete_for_layer() # (1,) self.out_proj.weight.data *= layer_mask self.out_proj.bias.data *= layer_mask if layer_mask == 0: new_config["use_attention"] = False self.hard_concrete_for_layer = None if self.hard_concrete_for_heads is not None: assert not self.hard_concrete_for_heads.training head_mask = self.hard_concrete_for_heads() # (num_heads,) new_config["remaining_heads"] = head_mask.nonzero().squeeze(-1).tolist() if len(new_config["remaining_heads"]) == 0: new_config["use_attention"] = False else: full_mask = head_mask.repeat_interleave(self.head_dim) full_index = full_mask.nonzero().squeeze(-1) # 1D prune_linear_layer(self.k_proj, full_index, "output") prune_linear_layer(self.v_proj, full_index, "output") prune_linear_layer(self.q_proj, full_index, "output") self.out_proj.weight.data *= full_mask prune_linear_layer(self.out_proj, full_index, "input") self.hard_concrete_for_heads = None return new_config class FeedForward(Module): """Layer that follows attention layer in encoder layer.""" def __init__( self, io_features: int, intermediate_features: int, intermediate_dropout: float, output_dropout: float, prune_intermediate: bool = False, prune_layer: bool = False, ): super().__init__() self.intermediate_dense = nn.Linear(io_features, intermediate_features) self.intermediate_dropout = nn.Dropout(intermediate_dropout) self.output_dense = nn.Linear(intermediate_features, io_features) self.output_dropout = nn.Dropout(output_dropout) if prune_intermediate: self.hard_concrete_for_intermediate = HardConcrete( n_in=intermediate_features, init_mean=0.5 ) else: self.hard_concrete_for_intermediate = None if prune_layer: self.hard_concrete_for_layer = HardConcrete(n_in=1, init_mean=0.01) else: self.hard_concrete_for_layer = None def forward(self, x): """ Args: x (Tensor): shape: `(batch, sequence_length, io_features)` Returns: x (Tensor): shape: `(batch, sequence_length, io_features)` """ x = self.intermediate_dense(x) x = torch.nn.functional.gelu(x) x = self.intermediate_dropout(x) if self.hard_concrete_for_intermediate is not None: intermediate_mask = self.hard_concrete_for_intermediate() # (intermediate_features,) x = x * intermediate_mask x = self.output_dense(x) x = self.output_dropout(x) if self.hard_concrete_for_layer is not None: layer_mask = self.hard_concrete_for_layer() # (1,) x = x * layer_mask return x def get_num_params(self): io_features = self.intermediate_dense.in_features if self.hard_concrete_for_intermediate is not None: intermediate_features = self.hard_concrete_for_intermediate.l0_norm() else: intermediate_features = self.intermediate_dense.out_features num_params = (io_features + 1) * intermediate_features + (intermediate_features + 1) * io_features if self.hard_concrete_for_layer is not None: num_params *= self.hard_concrete_for_layer.l0_norm() return num_params def prune(self): new_config = { "use_feed_forward": True, "ff_interm_features": self.intermediate_dense.out_features } if self.hard_concrete_for_layer is not None: assert not self.hard_concrete_for_layer.training layer_mask = self.hard_concrete_for_layer() self.output_dense.weight.data *= layer_mask self.output_dense.bias.data *= layer_mask if layer_mask == 0: new_config["use_feed_forward"] = False self.hard_concrete_for_layer = None if self.hard_concrete_for_intermediate is not None: assert not self.hard_concrete_for_intermediate.training interm_mask = self.hard_concrete_for_intermediate() interm_index = interm_mask.nonzero().squeeze(-1) # NOTE: must specify dim=-1 new_config["ff_interm_features"] = len(interm_index) if new_config["ff_interm_features"] == 0: new_config["use_feed_forward"] = False else: prune_linear_layer(self.intermediate_dense, interm_index, "output") self.output_dense.weight.data *= interm_mask prune_linear_layer(self.output_dense, interm_index, "input") self.hard_concrete_for_intermediate = None return new_config class EncoderLayer(Module): """A layer unit in encoder. Combines multihead self attention and feed forward.""" def __init__( self, attention: Optional[Module], # can be None if the entire layer is pruned dropout: float, layer_norm_first: bool, feed_forward: Optional[Module], # can be None if the entire layer is pruned embed_dim: int, ): super().__init__() self.attention = attention self.dropout = nn.Dropout(dropout) self.layer_norm = nn.LayerNorm(embed_dim) self.layer_norm_first = layer_norm_first self.feed_forward = feed_forward self.final_layer_norm = nn.LayerNorm(embed_dim) self.embed_dim = embed_dim def forward( self, x: Tensor, attention_mask: Optional[Tensor] = None, position_bias: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, ) -> Tuple[Tensor, Optional[Tensor]]: """ Args: x (Tensor): Input of shape ``(batch, sequence_length, embed_dim)``. attention_mask (Tensor or ``None``, optional): attention mask of shape ``(batch, 1, sequence_length, sequence_length)``. (Default: ``None``) position_bias (Tensor or ``None``, optional): position bias of shape ``(batch_size * num_heads, src_len, src_len)``. Only necessary for WavLM model, ``None`` otherwise. (Default: ``None``) key_padding_mask (Tensor or ``None``, optional): key padding mask of shape ``(batch_size, src_len)``. Only used for WavLM model, ignored otherwise. (Default: ``None``) Returns: (x, position_bias): Shapes are the same as in the input. Position bias is only relevant for WaLM model, ``None`` otherwise. """ if self.attention is not None: residual = x if self.layer_norm_first: x = self.layer_norm(x) x, position_bias = self.attention( x, attention_mask=attention_mask, position_bias=position_bias, key_padding_mask=key_padding_mask ) x = self.dropout(x) x = residual + x if self.layer_norm_first: if self.feed_forward is not None: x = x + self.feed_forward(self.final_layer_norm(x)) else: # NOTE: for post norm, the layer norms should always be applied even if the layers are pruned. x = self.layer_norm(x) if self.feed_forward is not None: x = x + self.feed_forward(x) x = self.final_layer_norm(x) return x, position_bias def get_num_params(self): num_params = self.embed_dim * 2 * 2 # two layer norms if self.attention is not None: num_params += self.attention.get_num_params() if self.feed_forward is not None: num_params += self.feed_forward.get_num_params() return num_params class Transformer(Module): def __init__( self, pos_conv_embed: Module, dropout: float, layers: Module, layer_norm_first: bool, layer_drop: float, ): super().__init__() self.pos_conv_embed = pos_conv_embed self.layer_norm = nn.LayerNorm(pos_conv_embed.embed_dim) self.layer_norm_first = layer_norm_first self.layer_drop = layer_drop self.dropout = nn.Dropout(dropout) self.layers = layers def _preprocess(self, x: Tensor): x = x + self.pos_conv_embed(x) if self.layer_norm_first: x = self.layer_norm(x) x = self.dropout(x) return x def forward( self, x: Tensor, attention_mask: Optional[Tensor] = None, position_bias: Optional[Tensor] = None, ) -> Tensor: x = self._preprocess(x) for layer in self.layers: if not (self.training and torch.rand(1).item() <= self.layer_drop): x, position_bias = layer(x, attention_mask, position_bias=position_bias) if not self.layer_norm_first: x = self.layer_norm(x) return x def get_intermediate_outputs( self, x: Tensor, attention_mask: Optional[Tensor] = None, num_layers: Optional[int] = None, position_bias: Optional[Tensor] = None, ) -> List[Tensor]: if num_layers is not None: if not 0 < num_layers <= len(self.layers): raise ValueError(f"`num_layers` must be between [1, {len(self.layers)}]") ret: List[Tensor] = [] x = self._preprocess(x) for layer in self.layers: x, position_bias = layer(x, attention_mask, position_bias=position_bias) ret.append(x) if num_layers is not None and len(ret) >= num_layers: return ret return ret def get_num_params(self): # pos_conv_embed and layer_norm num_params = sum(p.numel() for p in self.pos_conv_embed.parameters()) + self.pos_conv_embed.embed_dim * 2 for layer in self.layers: num_params += layer.get_num_params() return num_params def prune(self): new_config = defaultdict(list) for layer in self.layers: attention_config = layer.attention.prune() new_config["use_attention"].append(attention_config["use_attention"]) if "remaining_heads" in attention_config: new_config["remaining_heads"].append(attention_config["remaining_heads"]) else: new_config["num_heads"].append(attention_config["num_heads"]) if not attention_config["use_attention"]: layer.attention = None ff_config = layer.feed_forward.prune() new_config["use_feed_forward"].append(ff_config["use_feed_forward"]) new_config["ff_interm_features"].append(ff_config["ff_interm_features"]) if not ff_config["use_feed_forward"]: layer.feed_forward = None return new_config class Encoder(Module): def __init__( self, feature_projection: Module, transformer: Module, ): super().__init__() self.feature_projection = feature_projection self.transformer = transformer def _preprocess( self, features: Tensor, lengths: Optional[Tensor] = None, ) -> Tuple[Tensor, Optional[Tensor]]: x = self.feature_projection(features) mask: Optional[Tensor] = None if lengths is not None: batch_size, max_len, _ = x.shape # create mask for padded elements and zero-out them mask = torch.arange(max_len, device=lengths.device).expand(batch_size, max_len) >= lengths[:, None] x[mask] = 0.0 # extend the mask to attention shape and set weight mask = -10000.0 * mask[:, None, None, :].to(dtype=features.dtype) mask = mask.expand(batch_size, 1, max_len, max_len) return x, mask def forward( self, features: Tensor, lengths: Optional[Tensor] = None, ) -> Tensor: x, mask = self._preprocess(features, lengths) x = self.transformer(x, attention_mask=mask) return x def extract_features( self, features: Tensor, lengths: Optional[Tensor] = None, num_layers: Optional[int] = None, ) -> List[Tensor]: x, masks = self._preprocess(features, lengths) interm = self.transformer.get_intermediate_outputs(x, attention_mask=masks, num_layers=num_layers) return [x] + interm def get_num_params(self, in_features): """Calculate the current model size.""" feature_projection_size = self.feature_projection.get_num_params(in_features) transformer_size = self.transformer.get_num_params() return feature_projection_size + transformer_size def prune(self, conv_out_index): """In-place pruning of submodules.""" prune_layer_norm(self.feature_projection.layer_norm, conv_out_index) prune_linear_layer(self.feature_projection.projection, conv_out_index, "input") transformer_config = self.transformer.prune() return transformer_config ################################################################################ def _get_feature_extractor( norm_mode: str, shapes: List[Tuple[int, int, int]], bias: bool, prune_conv_channels: bool = False, ) -> FeatureExtractor: """ Args: norm_mode (str): Either "group_norm" or "layer_norm". If "group_norm", then a single normalization is applied in the first convolution block. Otherwise, all the convolution blocks will have layer normalization. This option corresponds to "extractor_mode" from fairseq. Expected values are "group_norm" for Base arch, and "layer_norm" for Large arch. shapes (list of tuple of int): Configuration of convolution layers. List of convolution configuration, i.e. ``[(output_channel, kernel_size, stride), ...]`` This option corresponds to "conv_feature_layers" from fairseq. Expected values are ``[(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2`` for all the architectures. bias (bool): Whether to include bias term to each convolution operation. This option corresponds to "conv_bias" from fairseq. Expected values are False for Base arch, and True for Large arch. See Also: * Original implementation https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L666-L733 * "extractor_mode" - Def and base: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L38-L45 - Large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L52 * "conv_feature_layers" - Def, base and large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L94-L100 * "conv_bias" - Def and base: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L101-L103 - Large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L61 """ if norm_mode not in ["group_norm", "layer_norm"]: raise ValueError("Invalid norm mode") blocks = [] in_channels = 1 for i, (out_channels, kernel_size, stride) in enumerate(shapes): normalization = None if norm_mode == "group_norm" and i == 0: normalization = nn.GroupNorm( num_groups=out_channels, num_channels=out_channels, affine=True, ) elif norm_mode == "layer_norm": normalization = LayerNorm( normalized_shape=out_channels, elementwise_affine=True, ) blocks.append( ConvLayerBlock( in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride, bias=bias, layer_norm=normalization, prune_conv_channels=prune_conv_channels, ) ) in_channels = out_channels return FeatureExtractor(nn.ModuleList(blocks)) def _get_encoder( in_features: int, embed_dim: int, dropout_input: float, pos_conv_kernel: int, pos_conv_groups: int, num_layers: int, use_attention: List[bool], use_feed_forward: List[bool], num_heads: List[int], head_dim: int, attention_dropout: float, ff_interm_features: List[int], ff_interm_dropout: float, dropout: float, layer_norm_first: bool, layer_drop: float, prune_attention_heads: bool = False, prune_attention_layer: bool = False, prune_feed_forward_intermediate: bool = False, prune_feed_forward_layer: bool = False, ) -> Encoder: """ Args: in_features (int): The number of input features. embed_dim (int): The dimension of embedding. This option corresponds to "encoder_embed_dim" from fairseq. Expected values are 768 for Base arch, and 1024 for Large arch. dropout_input (float): The dropout probability applied after the input feature is projected to ``embed_dim``. This option corresponds to "dropout_input" from fairseq. Expected values are 0.1 for both Base and Large arch. pos_conv_kernel (int): The kernel size of convolutional positional embeddings. This option corresponds to "conv_pos" from fairseq. Expected values are 128 for both Base and Large arch. pos_conv_groups (int): The number of groups of convolutional positional embeddings. This option corresponds to "conv_pos_groups" from fairseq. Expected values are 16 for both Base and Large arch. num_layers (int): The number of self attention layers in transformer block. This option corresponds to "encoder_layers" from fairseq. Expected values are 12 for Base and 24 for Large arch. num_heads (int): The number of heads in self attention layers. This option corresponds to "encoder_attention_heads" from fairseq. Expected values are 12 for Base and 16 for Large arch. attention_dropout (float): The dropout probability applied after softmax in self-attention layer. This option corresponds to "attention_dropout" from fairseq. Expected values are 0.1 for Base and 0.0 for Large arch. ff_interm_features (int): The dimension of hidden features in feed forward layer. This option corresponds to "encoder_ffn_embed_dim" from fairseq. Expected values are 3072 for Base and 4096 for Large arch. ff_interm_dropout (float): The dropout probability applied in feedforward layer. This option correspinds to "activation_dropout" from fairseq. Expected values are 0.1 for both Base and Large arch. dropout (float): The dropout probability applied at the end of feed forward layer. This option corresponds to "dropout" from fairseq. Expected values are 0.1 for Base and 0.0 for Large arch. layer_norm_first (bool): Control the order of layer norm in transformer layer and each encoder layer. If True, in transformer layer, layer norm is applied before features are fed to encoder layers. In encoder layer, two layer norms are applied before and after self attention. If False, in transformer layer, layer norm is applied after features are fed to encoder layers. In encoder layer, two layer norms are applied after self attention, before and after feed forward. This option corresponds to "layer_norm_first" from fairseq. Expected values are False for Base and True for Large arch. layer_drop (float): Probability to drop each encoder layer during training. This option corresponds to "layerdrop" from fairseq. Expected values are 0.1 for both Base and Large arch. See Also: * "encoder_embed_dim" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L49-L51 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L64 * "dropout_input" - Def, base and large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L75-L78 * "conv_pos" - Def, base and large NOTE: The description is wrong. https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L204-L207 - Usage https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L756 * "conv_pos_groups" - Def, base and large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L208-L211 * "encoder_layers" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L46-L48 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L63 * "encoder_attention_heads" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L55-L57 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L66 * "attention_dropout" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L66-L68 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L60 * "encoder_ffn_embed_dim" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L52-L54 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L65 * "activation_dropout" - Def https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L69-L71 - Base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/base_960h.yaml#L55 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/vox_960h.yaml#L55 * "dropout" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L63-L65 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L59 * "layer_norm_first" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L91-L93 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L53 * "layerdrop" - Def https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L72-L74 - Base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/base_960h.yaml#L54 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/vox_960h.yaml#L54 """ feature_projection = FeatureProjection(in_features, embed_dim, dropout_input) pos_conv = ConvolutionalPositionalEmbedding(embed_dim, pos_conv_kernel, pos_conv_groups) # Original impl # https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L768-L782 encoder_layers = nn.ModuleList() for idx in range(num_layers): if use_attention[idx]: attention = SelfAttention( embed_dim=embed_dim, num_heads=num_heads[idx], head_dim=head_dim, dropout=attention_dropout, prune_heads=prune_attention_heads, prune_layer=prune_attention_layer, ) else: attention = None if use_feed_forward[idx]: feed_forward = FeedForward( io_features=embed_dim, intermediate_features=ff_interm_features[idx], intermediate_dropout=ff_interm_dropout, output_dropout=dropout, prune_intermediate=prune_feed_forward_intermediate, prune_layer=prune_feed_forward_layer, ) else: feed_forward = None encoder_layers.append( EncoderLayer( attention=attention, dropout=dropout, layer_norm_first=layer_norm_first, feed_forward=feed_forward, embed_dim=embed_dim, ) ) transformer = Transformer( pos_conv_embed=pos_conv, dropout=dropout, layers=encoder_layers, layer_norm_first=not layer_norm_first, layer_drop=layer_drop, ) return Encoder(feature_projection, transformer) def _get_wavlm_encoder( in_features: int, embed_dim: int, dropout_input: float, pos_conv_kernel: int, pos_conv_groups: int, num_layers: int, use_attention: List[bool], use_feed_forward: List[bool], total_num_heads: List[int], remaining_heads: List[List[int]], num_buckets: int, max_distance: int, attention_dropout: float, ff_interm_features: List[int], ff_interm_dropout: float, dropout: float, layer_norm_first: bool, layer_drop: float, prune_attention_heads: bool = False, prune_attention_layer: bool = False, prune_feed_forward_intermediate: bool = False, prune_feed_forward_layer: bool = False, ) -> Encoder: """ Construct encoder for WavLM model :cite:`chen2022wavlm`. The structure of the encoder and most of the argments are the same as in :py:func:`_get_encoder` so refer there for documentation. The only difference from Wav2Vec2 encoder is usage of `WavLMSelfAttention` instead of `SelfAttention` and two additional parameters: `num_buckets` and `max_distance`. Args: in_features (int): See :py:func:`_get_encoder`. embed_dim (int): See :py:func:`_get_encoder`. dropout_input (float): See :py:func:`_get_encoder`. pos_conv_kernel (int): See :py:func:`_get_encoder`. pos_conv_groups (int): See :py:func:`_get_encoder`. num_layers (int): See :py:func:`_get_encoder`. num_heads (int): See :py:func:`_get_encoder`. num_buckets (int): Number of buckets for relative position embedding. max_distance (int): Maximum distance for relative position embedding. attention_dropout (float): See :py:func:`_get_encoder`. ff_interm_features (int): See :py:func:`_get_encoder`. ff_interm_dropout (float): See :py:func:`_get_encoder`. dropout (float): See :py:func:`_get_encoder`. layer_norm_first (bool): See :py:func:`_get_encoder`. layer_drop (float): See :py:func:`_get_encoder`. """ feature_projection = FeatureProjection(in_features, embed_dim, dropout_input) pos_conv = ConvolutionalPositionalEmbedding(embed_dim, pos_conv_kernel, pos_conv_groups) # Original impl # https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L768-L782 encoder_layers = nn.ModuleList() for i in range(num_layers): if use_attention[i]: attention = WavLMSelfAttention( embed_dim=embed_dim, total_num_heads=total_num_heads[i], remaining_heads=remaining_heads[i], dropout=attention_dropout, has_relative_attention_bias=(i == 0), # Position embedding is only necessary in the first layer. num_buckets=num_buckets, max_distance=max_distance, prune_heads=prune_attention_heads, prune_layer=prune_attention_layer, ) else: attention = None if use_feed_forward[i]: feed_forward = FeedForward( io_features=embed_dim, intermediate_features=ff_interm_features[i], intermediate_dropout=ff_interm_dropout, output_dropout=dropout, prune_intermediate=prune_feed_forward_intermediate, prune_layer=prune_feed_forward_layer, ) else: feed_forward = None encoder_layers.append( EncoderLayer( attention=attention, dropout=dropout, layer_norm_first=layer_norm_first, feed_forward=feed_forward, embed_dim=embed_dim, ) ) transformer = Transformer( pos_conv_embed=pos_conv, dropout=dropout, layers=encoder_layers, layer_norm_first=not layer_norm_first, layer_drop=layer_drop, ) return Encoder(feature_projection, transformer) def _get_padding_mask(input: Tensor, lengths: Tensor) -> Tensor: """Generate the padding mask given the padded input and the lengths Tensors. Args: input (Tensor): The padded Tensor of dimension `[batch, max_len, frequency]`. lengths (Tensor): The lengths Tensor of dimension `[batch,]`. Returns: (Tensor): The padding mask. """ batch_size, max_len, _ = input.shape mask = torch.arange(max_len, device=lengths.device).expand(batch_size, max_len) >= lengths[:, None] return mask class GradMultiply(torch.autograd.Function): @staticmethod def forward(ctx, x, scale): ctx.scale = scale res = x.new(x) return res @staticmethod def backward(ctx, grad): return grad * ctx.scale, None so-vits-svc-npu/vencoder/dphubert/hardconcrete.py000066400000000000000000000077711456161405400225320ustar00rootroot00000000000000"""Implementation of the hard Concrete distribution. Originally from: https://github.com/asappresearch/flop/blob/master/flop/hardconcrete.py """ import math import mindtorch.torch as torch import mindtorch.torch.nn as nn class HardConcrete(nn.Module): """A HarcConcrete module. Use this module to create a mask of size N, which you can then use to perform L0 regularization. To obtain a mask, simply run a forward pass through the module with no input data. The mask is sampled in training mode, and fixed during evaluation mode, e.g.: >>> module = HardConcrete(n_in=100) >>> mask = module() >>> norm = module.l0_norm() """ def __init__( self, n_in: int, init_mean: float = 0.5, init_std: float = 0.01, temperature: float = 2/3, # from CoFi stretch: float = 0.1, eps: float = 1e-6 ) -> None: """Initialize the HardConcrete module. Parameters ---------- n_in : int The number of hard concrete variables in this mask. init_mean : float, optional Initial drop rate for hard concrete parameter, by default 0.5., init_std: float, optional Used to initialize the hard concrete parameters, by default 0.01. temperature : float, optional Temperature used to control the sharpness of the distribution, by default 1.0 stretch : float, optional Stretch the sampled value from [0, 1] to the interval [-stretch, 1 + stretch], by default 0.1. """ super().__init__() self.n_in = n_in self.limit_l = -stretch self.limit_r = 1.0 + stretch self.log_alpha = nn.Parameter(torch.zeros(n_in)) self.beta = temperature self.init_mean = init_mean self.init_std = init_std self.bias = -self.beta * math.log(-self.limit_l / self.limit_r) self.eps = eps self.compiled_mask = None self.reset_parameters() def reset_parameters(self): """Reset the parameters of this module.""" self.compiled_mask = None mean = math.log(1 - self.init_mean) - math.log(self.init_mean) self.log_alpha.data.normal_(mean, self.init_std) def l0_norm(self) -> torch.Tensor: """Compute the expected L0 norm of this mask. Returns ------- torch.Tensor The expected L0 norm. """ return (self.log_alpha + self.bias).sigmoid().sum() def forward(self) -> torch.Tensor: """Sample a hard concrete mask. Returns ------- torch.Tensor The sampled binary mask """ if self.training: # Reset the compiled mask self.compiled_mask = None # Sample mask dynamically u = self.log_alpha.new(self.n_in).uniform_(self.eps, 1 - self.eps) s = torch.sigmoid((torch.log(u / (1 - u)) + self.log_alpha) / self.beta) s = s * (self.limit_r - self.limit_l) + self.limit_l mask = s.clamp(min=0., max=1.) else: # Compile new mask if not cached if self.compiled_mask is None: # Get expected sparsity expected_num_zeros = self.n_in - self.l0_norm().item() num_zeros = round(expected_num_zeros) # Approximate expected value of each mask variable z; # We use an empirically validated magic number 0.8 soft_mask = torch.sigmoid(self.log_alpha / self.beta * 0.8) # Prune small values to set to 0 _, indices = torch.topk(soft_mask, k=num_zeros, largest=False) soft_mask[indices] = 0. self.compiled_mask = soft_mask mask = self.compiled_mask return mask def extra_repr(self) -> str: return str(self.n_in) def __repr__(self) -> str: return "{}({})".format(self.__class__.__name__, self.extra_repr()) so-vits-svc-npu/vencoder/dphubert/model.py000066400000000000000000001136121456161405400211610ustar00rootroot00000000000000"""Speech SSL models supporting pruning. Originally from: https://github.com/pytorch/audio/blob/main/torchaudio/models/wav2vec2/model.py """ import math from typing import List, Optional, Tuple import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import Tensor from mindtorch.torch.nn import Module from . import components class Wav2Vec2Model(Module): """Acoustic model used in *wav2vec 2.0* :cite:`baevski2020wav2vec`. Note: To build the model, please use one of the factory functions. :py:func:`wav2vec2_model`, :py:func:`wav2vec2_base`, :py:func:`wav2vec2_large`, :py:func:`wav2vec2_large_lv60k`, :py:func:`hubert_base`, :py:func:`hubert_large`, and :py:func:`hubert_xlarge`. See Also: * :class:`torchaudio.pipelines.Wav2Vec2Bundle`: Pretrained models (without fine-tuning) * :class:`torchaudio.pipelines.Wav2Vec2ASRBundle`: ASR pipelines with pretrained models. Args: feature_extractor (torch.nn.Module): Feature extractor that extracts feature vectors from raw audio Tensor. encoder (torch.nn.Module): Encoder that converts the audio features into the sequence of probability distribution (in negative log-likelihood) over labels. aux (torch.nn.Module or None, optional): Auxiliary module. If provided, the output from encoder is passed to this module. """ # noqa: E501 def __init__( self, normalize_waveform: bool, feature_extractor: Module, encoder: Module, aux: Optional[Module] = None, ): super().__init__() self.normalize_waveform = normalize_waveform self.feature_extractor = feature_extractor self.encoder = encoder self.aux = aux @torch.jit.export def extract_features( self, waveforms: Tensor, lengths: Optional[Tensor] = None, num_layers: Optional[int] = None, ) -> Tuple[List[Tensor], Optional[Tensor]]: """Extract feature vectors from raw waveforms This returns the list of outputs from the intermediate layers of transformer block in encoder. Args: waveforms (Tensor): Audio tensor of shape `(batch, frames)`. lengths (Tensor or None, optional): Indicates the valid length of each audio in the batch. Shape: `(batch, )`. When the ``waveforms`` contains audios with different durations, by providing ``lengths`` argument, the model will compute the corresponding valid output lengths and apply proper mask in transformer attention layer. If ``None``, it is assumed that the entire audio waveform length is valid. num_layers (int or None, optional): If given, limit the number of intermediate layers to go through. Providing `1` will stop the computation after going through one intermediate layers. If not given, the outputs from all the intermediate layers are returned. Returns: (List[Tensor], Optional[Tensor]): List of Tensors Features from requested layers. Each Tensor is of shape: `(batch, time frame, feature dimension)` Tensor or None If ``lengths`` argument was provided, a Tensor of shape `(batch, )` is returned. It indicates the valid length in time axis of each feature Tensor. """ if self.normalize_waveform: if lengths is not None: waveforms = [ F.layer_norm(wave[:length], (length,)) for wave, length in zip(waveforms, lengths) ] waveforms = torch.nn.utils.rnn.pad_sequence(waveforms, batch_first=True) else: waveforms = F.layer_norm(waveforms, waveforms.shape[-1:]) x, lengths = self.feature_extractor(waveforms, lengths) x = self.encoder.extract_features(x, lengths, num_layers) # (num_layers+1,), including the input return x, lengths def get_num_params(self): """Calculate the current size.""" feature_extractor_size, encoder_in_features = self.feature_extractor.get_num_params_and_final_out_channels() encoder_size = self.encoder.get_num_params(encoder_in_features) return feature_extractor_size + encoder_size def prune(self): self.eval() # must be in eval mode conv_config, conv_out_index = self.feature_extractor.prune() # [(output_channel, kernel_size, stride), ...] transformer_config = self.encoder.prune(conv_out_index) # NOTE: this is a defaultdict(list) use_attention = transformer_config["use_attention"] use_feed_forward = transformer_config["use_feed_forward"] num_heads = transformer_config["num_heads"] # can be [] remaining_heads = transformer_config["remaining_heads"] # can be [] ff_interm_features = transformer_config["ff_interm_features"] return conv_config, use_attention, use_feed_forward, num_heads, remaining_heads, ff_interm_features def forward( self, waveforms: Tensor, lengths: Optional[Tensor] = None, ) -> Tuple[Tensor, Optional[Tensor]]: """Compute the sequence of probability distribution over labels. Args: waveforms (Tensor): Audio tensor of shape `(batch, frames)`. lengths (Tensor or None, optional): Indicates the valid length of each audio in the batch. Shape: `(batch, )`. When the ``waveforms`` contains audios with different durations, by providing ``lengths`` argument, the model will compute the corresponding valid output lengths and apply proper mask in transformer attention layer. If ``None``, it is assumed that all the audio in ``waveforms`` have valid length. Default: ``None``. Returns: (Tensor, Optional[Tensor]): Tensor The sequences of probability distribution (in logit) over labels. Shape: `(batch, frames, num labels)`. Tensor or None If ``lengths`` argument was provided, a Tensor of shape `(batch, )` is returned. It indicates the valid length in time axis of the output Tensor. """ if self.normalize_waveform: if lengths is not None: waveforms = [ F.layer_norm(wave[:length], (length,)) for wave, length in zip(waveforms, lengths) ] waveforms = torch.nn.utils.rnn.pad_sequence(waveforms, batch_first=True) else: waveforms = F.layer_norm(waveforms, waveforms.shape[-1:]) x, lengths = self.feature_extractor(waveforms, lengths) x = self.encoder(x, lengths) if self.aux is not None: x = self.aux(x) return x, lengths def wav2vec2_model(**configs) -> Wav2Vec2Model: """Wraps the original wav2vec2_model and wavlm_model.""" if "encoder_remaining_heads" in configs: return wavlm_model(**configs) return wav2vec2_model_original(**configs) def wav2vec2_model_original( extractor_mode: str, extractor_conv_layer_config: Optional[List[Tuple[int, int, int]]], extractor_conv_bias: bool, encoder_embed_dim: int, encoder_projection_dropout: float, encoder_pos_conv_kernel: int, encoder_pos_conv_groups: int, encoder_num_layers: int, encoder_use_attention: List[bool], encoder_use_feed_forward: List[bool], encoder_num_heads: List[int], encoder_head_dim: int, encoder_attention_dropout: float, encoder_ff_interm_features: List[int], encoder_ff_interm_dropout: float, encoder_dropout: float, encoder_layer_norm_first: bool, encoder_layer_drop: float, aux_num_out: Optional[int], normalize_waveform: bool, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds custom :class:`~torchaudio.models.Wav2Vec2Model`. Note: The "feature extractor" below corresponds to `ConvFeatureExtractionModel `__ in the original ``fairseq`` implementation. This is referred as "(convolutional) feature encoder" in the *wav2vec 2.0* :cite:`baevski2020wav2vec` paper. The "encoder" below corresponds to `TransformerEncoder `__, and this is referred as "Transformer" in the paper. Args: extractor_mode (str): Operation mode of feature extractor. Valid values are ``"group_norm"`` or ``"layer_norm"``. If ``"group_norm"``, then a single normalization is applied in the first convolution block. Otherwise, all the convolution blocks will have layer normalization. This option corresponds to ``extractor_mode`` from ``fairseq``. extractor_conv_layer_config (list of integer tuples or None): Configuration of convolution layers in feature extractor. List of convolution configuration, i.e. ``[(output_channel, kernel_size, stride), ...]`` If ``None`` is provided, then the following default value is used. .. code-block:: python [ (512, 10, 5), (512, 3, 2), (512, 3, 2), (512, 3, 2), (512, 3, 2), (512, 2, 2), (512, 2, 2), ] This option corresponds to ``conv_feature_layers`` from ``fairseq``. extractor_conv_bias (bool): Whether to include bias term to each convolution operation. This option corresponds to ``conv_bias`` from ``fairseq``. encoder_embed_dim (int): The dimension of embedding in encoder. This option corresponds to ``encoder_embed_dim`` from ``fairseq``. encoder_projection_dropout (float): The dropout probability applied after the input feature is projected to ``encoder_embed_dim``. This option corresponds to ``dropout_input`` from ``fairseq``. encoder_pos_conv_kernel (int): The kernel size of convolutional positional embeddings. This option corresponds to ``conv_pos`` from ``fairseq``. encoder_pos_conv_groups (int): The number of groups of convolutional positional embeddings. This option corresponds to ``conv_pos_groups`` from ``fairseq``. encoder_num_layers (int): The number of self attention layers in transformer block. This option corresponds to ``encoder_layers`` from ``fairseq``. encoder_num_heads (int): The number of heads in self attention layers. This option corresponds to ``encoder_attention_heads`` from ``fairseq``. encoder_attention_dropout (float): The dropout probability applied after softmax in self-attention layer. This option corresponds to ``attention_dropout`` from ``fairseq``. encoder_ff_interm_features (int): The dimension of hidden features in feed forward layer. This option corresponds to ``encoder_ffn_embed_dim`` from ``fairseq``. encoder_ff_interm_dropout (float): The dropout probability applied in feedforward layer. This option correspinds to ``activation_dropout`` from ``fairseq``. encoder_dropout (float): The dropout probability applied at the end of feed forward layer. This option corresponds to ``dropout`` from ``fairseq``. encoder_layer_norm_first (bool): Control the order of layer norm in transformer layer and each encoder layer. If True, in transformer layer, layer norm is applied before features are fed to encoder layers. In encoder layer, two layer norms are applied before and after self attention. If False, in transformer layer, layer norm is applied after features are fed to encoder layers. In encoder layer, two layer norms are applied after self attention, before and after feed forward. This option corresponds to ``layer_norm_first`` from ``fairseq``. encoder_layer_drop (float): Probability to drop each encoder layer during training. This option corresponds to ``layerdrop`` from ``fairseq``. aux_num_out (int or None): When provided, attach an extra linear layer on top of encoder, which can be used for fine-tuning. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 if extractor_conv_layer_config is None: extractor_conv_layer_config = [(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2 feature_extractor = components._get_feature_extractor( extractor_mode, extractor_conv_layer_config, extractor_conv_bias, prune_conv_channels=extractor_prune_conv_channels, ) encoder = components._get_encoder( in_features=extractor_conv_layer_config[-1][0], embed_dim=encoder_embed_dim, dropout_input=encoder_projection_dropout, pos_conv_kernel=encoder_pos_conv_kernel, pos_conv_groups=encoder_pos_conv_groups, num_layers=encoder_num_layers, use_attention=encoder_use_attention, use_feed_forward=encoder_use_feed_forward, num_heads=encoder_num_heads, head_dim=encoder_head_dim, attention_dropout=encoder_attention_dropout, ff_interm_features=encoder_ff_interm_features, ff_interm_dropout=encoder_ff_interm_dropout, dropout=encoder_dropout, layer_norm_first=encoder_layer_norm_first, layer_drop=encoder_layer_drop, prune_attention_heads=encoder_prune_attention_heads, prune_attention_layer=encoder_prune_attention_layer, prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) aux = None if aux_num_out is not None: aux = torch.nn.Linear(in_features=encoder_embed_dim, out_features=aux_num_out) return Wav2Vec2Model(normalize_waveform, feature_extractor, encoder, aux) def wav2vec2_base( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.1, encoder_ff_interm_dropout: float = 0.1, encoder_dropout: float = 0.1, encoder_layer_drop: float = 0.1, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "base" :class:`~torchaudio.models.Wav2Vec2Model` from *wav2vec 2.0* :cite:`baevski2020wav2vec` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="group_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=768, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=12, encoder_num_heads=12, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=3072, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=False, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def wav2vec2_large( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.1, encoder_ff_interm_dropout: float = 0.1, encoder_dropout: float = 0.1, encoder_layer_drop: float = 0.1, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "large" :class:`~torchaudio.models.Wav2Vec2Model` from *wav2vec 2.0* :cite:`baevski2020wav2vec` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="group_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=1024, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=24, encoder_num_heads=16, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=4096, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=False, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def wav2vec2_large_lv60k( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.0, encoder_ff_interm_dropout: float = 0.1, encoder_dropout: float = 0.0, encoder_layer_drop: float = 0.1, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "large lv-60k" :class:`~torchaudio.models.Wav2Vec2Model` from *wav2vec 2.0* :cite:`baevski2020wav2vec` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="layer_norm", extractor_conv_layer_config=None, extractor_conv_bias=True, encoder_embed_dim=1024, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=24, encoder_num_heads=16, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=4096, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=True, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def hubert_base( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.1, encoder_ff_interm_dropout: float = 0.0, encoder_dropout: float = 0.1, encoder_layer_drop: float = 0.05, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "base" :class:`HuBERT ` from *HuBERT* :cite:`hsu2021hubert` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="group_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=768, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=12, encoder_use_attention=[True] * 12, encoder_use_feed_forward=[True] * 12, encoder_num_heads=[12] * 12, encoder_head_dim=64, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=[3072] * 12, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=False, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def hubert_large( encoder_projection_dropout: float = 0.0, encoder_attention_dropout: float = 0.0, encoder_ff_interm_dropout: float = 0.0, encoder_dropout: float = 0.0, encoder_layer_drop: float = 0.0, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "large" :class:`HuBERT ` from *HuBERT* :cite:`hsu2021hubert` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="layer_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=1024, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=24, encoder_num_heads=16, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=4096, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=True, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def hubert_xlarge( encoder_projection_dropout: float = 0.0, encoder_attention_dropout: float = 0.0, encoder_ff_interm_dropout: float = 0.0, encoder_dropout: float = 0.0, encoder_layer_drop: float = 0.0, aux_num_out: Optional[int] = None, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds "extra large" :class:`HuBERT ` from *HuBERT* :cite:`hsu2021hubert` Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ # noqa: E501 return wav2vec2_model( extractor_mode="layer_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=1280, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=48, encoder_num_heads=16, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=5120, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=True, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, extractor_prune_conv_channels=extractor_prune_conv_channels, encoder_prune_attention_heads=encoder_prune_attention_heads, encoder_prune_attention_layer=encoder_prune_attention_layer, encoder_prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, encoder_prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) def _init_hubert_pretrain_model(module): if isinstance(module, components.LayerNorm): torch.nn.init.kaiming_normal_(module.conv.weight) elif isinstance(module, components.ConvolutionalPositionalEmbedding): # normalize the weight to normal distribution. std = math.sqrt(4.0 / (module.embed_dim * module.kernel_size)) torch.nn.init.normal_(module.conv.weight, mean=0.0, std=std) torch.nn.init.constant_(module.conv.bias, 0.0) elif isinstance(module, components.SelfAttention): # normalize the query, key, value, and out_proj parameters in self attention module. torch.nn.init.xavier_uniform_(module.k_proj.weight, gain=1 / math.sqrt(2)) torch.nn.init.xavier_uniform_(module.v_proj.weight, gain=1 / math.sqrt(2)) torch.nn.init.xavier_uniform_(module.q_proj.weight, gain=1 / math.sqrt(2)) torch.nn.init.xavier_uniform_(module.out_proj.weight) torch.nn.init.constant_(module.out_proj.bias, 0.0) elif isinstance(module, components.Transformer): module.apply(components._init_transformer_params) else: pass def wavlm_model( extractor_mode: str, extractor_conv_layer_config: Optional[List[Tuple[int, int, int]]], extractor_conv_bias: bool, encoder_embed_dim: int, encoder_projection_dropout: float, encoder_pos_conv_kernel: int, encoder_pos_conv_groups: int, encoder_num_layers: int, encoder_use_attention: List[bool], encoder_use_feed_forward: List[bool], encoder_total_num_heads: List[int], encoder_remaining_heads: List[List[int]], encoder_num_buckets: int, encoder_max_distance: int, encoder_attention_dropout: float, encoder_ff_interm_features: List[int], encoder_ff_interm_dropout: float, encoder_dropout: float, encoder_layer_norm_first: bool, encoder_layer_drop: float, aux_num_out: Optional[int], normalize_waveform: bool, extractor_prune_conv_channels: bool = False, encoder_prune_attention_heads: bool = False, encoder_prune_attention_layer: bool = False, encoder_prune_feed_forward_intermediate: bool = False, encoder_prune_feed_forward_layer: bool = False, ) -> Wav2Vec2Model: """Builds custom WaveLM model :cite:`chen2022wavlm`. The architecture is compatible with Wav2Vec2 model :cite:`baevski2020wav2vec`, and so the output object is :class:`~torchaudio.models.Wav2Vec2Model`. Most of the arguments have the same meaning as in :py:func:`wav2vec2_model` so please refer there for documentation. Args: extractor_mode (str): Operation mode of feature extractor. See :py:func:`wav2vec2_model`. extractor_conv_layer_config (list of integer tuples or None): See :py:func:`wav2vec2_model`. extractor_conv_bias (bool): See :py:func:`wav2vec2_model`. encoder_embed_dim (int): See :py:func:`wav2vec2_model`. encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_pos_conv_kernel (int): See :py:func:`wav2vec2_model`. encoder_pos_conv_groups (int): See :py:func:`wav2vec2_model`. encoder_num_layers (int): See :py:func:`wav2vec2_model`. encoder_num_heads (int): See :py:func:`wav2vec2_model`. encoder_num_buckets (int): Number of buckets for relative position embedding. encoder_max_distance (int): Maximum distance for relative position embedding. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_features (int): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_norm_first (bool): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int or None): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ if extractor_conv_layer_config is None: extractor_conv_layer_config = [(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2 feature_extractor = components._get_feature_extractor( extractor_mode, extractor_conv_layer_config, extractor_conv_bias, prune_conv_channels=extractor_prune_conv_channels, ) encoder = components._get_wavlm_encoder( in_features=extractor_conv_layer_config[-1][0], embed_dim=encoder_embed_dim, dropout_input=encoder_projection_dropout, pos_conv_kernel=encoder_pos_conv_kernel, pos_conv_groups=encoder_pos_conv_groups, num_layers=encoder_num_layers, use_attention=encoder_use_attention, use_feed_forward=encoder_use_feed_forward, total_num_heads=encoder_total_num_heads, remaining_heads=encoder_remaining_heads, num_buckets=encoder_num_buckets, max_distance=encoder_max_distance, attention_dropout=encoder_attention_dropout, ff_interm_features=encoder_ff_interm_features, ff_interm_dropout=encoder_ff_interm_dropout, dropout=encoder_dropout, layer_norm_first=encoder_layer_norm_first, layer_drop=encoder_layer_drop, prune_attention_heads=encoder_prune_attention_heads, prune_attention_layer=encoder_prune_attention_layer, prune_feed_forward_intermediate=encoder_prune_feed_forward_intermediate, prune_feed_forward_layer=encoder_prune_feed_forward_layer, ) aux = None if aux_num_out is not None: aux = torch.nn.Linear(in_features=encoder_embed_dim, out_features=aux_num_out) return Wav2Vec2Model(normalize_waveform, feature_extractor, encoder, aux) def wavlm_base( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.1, encoder_ff_interm_dropout: float = 0.1, encoder_dropout: float = 0.1, encoder_layer_drop: float = 0.1, aux_num_out: Optional[int] = None, ) -> Wav2Vec2Model: """Builds "base" WaveLM model :cite:`chen2022wavlm`. The architecture is compatible with Wav2Vec2 model :cite:`baevski2020wav2vec`, and so the output class is :class:`~torchaudio.models.Wav2Vec2Model`. Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ return wavlm_model( extractor_mode="group_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=768, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=12, encoder_num_heads=12, encoder_num_buckets=320, encoder_max_distance=800, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=3072, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=False, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, ) def wavlm_large( encoder_projection_dropout: float = 0.1, encoder_attention_dropout: float = 0.1, encoder_ff_interm_dropout: float = 0.0, encoder_dropout: float = 0.1, encoder_layer_drop: float = 0.1, aux_num_out: Optional[int] = None, ) -> Wav2Vec2Model: """Builds "large" WaveLM model :cite:`chen2022wavlm`. The architecture is compatible with Wav2Vec2 model :cite:`baevski2020wav2vec`, and so the output class is :class:`~torchaudio.models.Wav2Vec2Model`. Args: encoder_projection_dropout (float): See :py:func:`wav2vec2_model`. encoder_attention_dropout (float): See :py:func:`wav2vec2_model`. encoder_ff_interm_dropout (float): See :py:func:`wav2vec2_model`. encoder_dropout (float): See :py:func:`wav2vec2_model`. encoder_layer_drop (float): See :py:func:`wav2vec2_model`. aux_num_out (int, optional): See :py:func:`wav2vec2_model`. Returns: Wav2Vec2Model: The resulting model. """ return wavlm_model( extractor_mode="layer_norm", extractor_conv_layer_config=None, extractor_conv_bias=False, encoder_embed_dim=1024, encoder_projection_dropout=encoder_projection_dropout, encoder_pos_conv_kernel=128, encoder_pos_conv_groups=16, encoder_num_layers=24, encoder_num_heads=16, encoder_num_buckets=320, encoder_max_distance=800, encoder_attention_dropout=encoder_attention_dropout, encoder_ff_interm_features=4096, encoder_ff_interm_dropout=encoder_ff_interm_dropout, encoder_dropout=encoder_dropout, encoder_layer_norm_first=True, encoder_layer_drop=encoder_layer_drop, aux_num_out=aux_num_out, ) so-vits-svc-npu/vencoder/dphubert/pruning_utils.py000066400000000000000000000035521456161405400227640ustar00rootroot00000000000000"""Utility functions for pruning.""" from typing import Union import mindtorch.torch as torch import mindtorch.torch.nn as nn def prune_linear_layer(layer: nn.Linear, index: torch.LongTensor, dim: str): "Prune linear layer in place." # NOTE: weight: (out_features, in_features), bias: (out_features,) if dim == "input": dim = 1 layer.in_features = len(index) elif dim == "output": dim = 0 layer.out_features = len(index) else: raise ValueError layer.weight = nn.Parameter(layer.weight.index_select(dim, index).clone().detach()) if layer.bias is not None and dim == 0: layer.bias = nn.Parameter(layer.bias.index_select(0, index).clone().detach()) def prune_conv1d_layer(layer: nn.Conv1d, index: torch.LongTensor, dim: str): """Prune conv1d in place.""" # NOTE: weight: (out_channels, in_channels, kernel_size), bias: (out_channels,) if dim == "input": dim = 1 layer.in_channels = len(index) elif dim == "output": dim = 0 layer.out_channels = len(index) else: raise ValueError layer.weight = nn.Parameter(layer.weight.index_select(dim, index).clone().detach()) if layer.bias is not None and dim == 0: layer.bias = nn.Parameter(layer.bias.index_select(0, index).clone().detach()) def prune_layer_norm(layernorm: Union[nn.LayerNorm, nn.GroupNorm], index: torch.LongTensor): """Prune layer norm or group norm in place.""" layernorm.weight = nn.Parameter(layernorm.weight.index_select(0, index).clone().detach()) layernorm.bias = nn.Parameter(layernorm.bias.index_select(0, index).clone().detach()) if isinstance(layernorm, nn.LayerNorm): layernorm.normalized_shape = (len(index),) elif isinstance(layernorm, nn.GroupNorm): layernorm.num_groups = len(index) layernorm.num_channels = len(index) so-vits-svc-npu/vencoder/dphubert/utils/000077500000000000000000000000001456161405400206435ustar00rootroot00000000000000so-vits-svc-npu/vencoder/dphubert/utils/__init__.py000066400000000000000000000000001456161405400227420ustar00rootroot00000000000000so-vits-svc-npu/vencoder/dphubert/utils/import_huggingface_wavlm.py000066400000000000000000000127061456161405400262720ustar00rootroot00000000000000"""Import Hugging Face transformers's wav2vec2.0 pretrained weights to torchaudios's format. Originally from: https://github.com/pytorch/audio/blob/main/torchaudio/models/wav2vec2/utils/import_huggingface.py """ import logging from typing import Any, Dict from mindtorch.torch.nn import Module from ..model import Wav2Vec2Model, wav2vec2_model, wavlm_model _LG = logging.getLogger(__name__) def _get_config(cfg): config = { "extractor_mode": f"{cfg.feat_extract_norm}_norm", "extractor_conv_layer_config": list(zip(cfg.conv_dim, cfg.conv_kernel, cfg.conv_stride)), "extractor_conv_bias": cfg.conv_bias, "encoder_embed_dim": cfg.hidden_size, "encoder_projection_dropout": cfg.feat_proj_dropout, "encoder_pos_conv_kernel": cfg.num_conv_pos_embeddings, "encoder_pos_conv_groups": cfg.num_conv_pos_embedding_groups, "encoder_num_layers": cfg.num_hidden_layers, "encoder_num_heads": cfg.num_attention_heads, "encoder_attention_dropout": cfg.attention_dropout, "encoder_ff_interm_features": cfg.intermediate_size, "encoder_ff_interm_dropout": cfg.activation_dropout, "encoder_dropout": cfg.hidden_dropout, "encoder_layer_norm_first": cfg.do_stable_layer_norm, "encoder_layer_drop": cfg.layerdrop, } return config def _get_config_wavlm(cfg): config = { "extractor_mode": f"{cfg.feat_extract_norm}_norm", "extractor_conv_layer_config": list(zip(cfg.conv_dim, cfg.conv_kernel, cfg.conv_stride)), "extractor_conv_bias": cfg.conv_bias, "encoder_embed_dim": cfg.hidden_size, "encoder_projection_dropout": cfg.feat_proj_dropout, "encoder_pos_conv_kernel": cfg.num_conv_pos_embeddings, "encoder_pos_conv_groups": cfg.num_conv_pos_embedding_groups, "encoder_num_layers": cfg.num_hidden_layers, "encoder_use_attention": [True] * cfg.num_hidden_layers, "encoder_use_feed_forward": [True] * cfg.num_hidden_layers, "encoder_total_num_heads": [cfg.num_attention_heads for _ in range(cfg.num_hidden_layers)], "encoder_remaining_heads": [list(range(cfg.num_attention_heads)) for _ in range(cfg.num_hidden_layers)], "encoder_num_buckets": cfg.num_buckets, "encoder_max_distance": cfg.max_bucket_distance, "encoder_attention_dropout": cfg.attention_dropout, "encoder_ff_interm_features": [cfg.intermediate_size for _ in range(cfg.num_hidden_layers)], "encoder_ff_interm_dropout": cfg.activation_dropout, "encoder_dropout": cfg.hidden_dropout, "encoder_layer_norm_first": cfg.do_stable_layer_norm, "encoder_layer_drop": cfg.layerdrop, "normalize_waveform": cfg.feat_extract_norm == "layer", } return config def _build(config, original): is_for_ctc = original.__class__.__name__ in ["Wav2Vec2ForCTC", "WavLMForCTC"] if is_for_ctc: aux_num_out = original.config.vocab_size wav2vec2 = original.wav2vec2 else: _LG.warning( "The model is not an instance of Wav2Vec2ForCTC or WavLMForCTC. " '"lm_head" module is not imported.' ) aux_num_out = None wav2vec2 = original is_wavlm = original.__class__.__name__ in ["WavLMModel", "WavLMForCTC"] if is_wavlm: imported = wavlm_model(**config, aux_num_out=aux_num_out) else: imported = wav2vec2_model(**config, aux_num_out=aux_num_out) print(imported.feature_extractor.load_state_dict(wav2vec2.feature_extractor.state_dict(), strict=False)) print(imported.encoder.feature_projection.load_state_dict(wav2vec2.feature_projection.state_dict(), strict=False)) encoder_state_dict = wav2vec2.encoder.state_dict() if is_wavlm: # Rename paramaters of linear transformations for compatibility with the HF model transform_wavlm_encoder_state(encoder_state_dict, config["encoder_num_layers"]) print(imported.encoder.transformer.load_state_dict(encoder_state_dict, strict=False)) if is_for_ctc: imported.aux.load_state_dict(original.lm_head.state_dict()) return imported def transform_wavlm_encoder_state(state: Dict[str, Any], encoder_num_layers: int): """Converts WavLM encoder state from HuggingFace format. In particular, concatenates linear projection weights and biases to align with the structure of ``torch.nn.MultiheadAttention``. """ pass def import_huggingface_model(original: Module) -> Wav2Vec2Model: """Builds :class:`Wav2Vec2Model` from the corresponding model object of `Transformers `_. Args: original (torch.nn.Module): An instance of ``Wav2Vec2ForCTC`` from ``transformers``. Returns: Wav2Vec2Model: Imported model. Example >>> from torchaudio.models.wav2vec2.utils import import_huggingface_model >>> >>> original = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h") >>> model = import_huggingface_model(original) >>> >>> waveforms, _ = torchaudio.load("audio.wav") >>> logits, _ = model(waveforms) """ _LG.info("Importing model.") _LG.info("Loading model configuration.") is_wavlm = original.__class__.__name__ in ["WavLMModel", "WavLMForCTC"] if is_wavlm: config = _get_config_wavlm(original.config) else: config = _get_config(original.config) _LG.debug(" - config: %s", config) _LG.info("Building model.") imported = _build(config, original) return imported so-vits-svc-npu/vencoder/encoder.py000066400000000000000000000005541456161405400176630ustar00rootroot00000000000000class SpeechEncoder(object): def __init__(self, vec_path="pretrain/checkpoint_best_legacy_500.pt", device=None): self.model = None # This is Model self.hidden_dim = 768 pass def encoder(self, wav): """ input: wav:[signal_length] output: embedding:[batchsize,hidden_dim,wav_frame] """ pass so-vits-svc-npu/vencoder/hubert/000077500000000000000000000000001456161405400171575ustar00rootroot00000000000000so-vits-svc-npu/vencoder/hubert/__init__.py000066400000000000000000000000001456161405400212560ustar00rootroot00000000000000so-vits-svc-npu/vencoder/hubert/hubert_model.py000066400000000000000000000163221456161405400222060ustar00rootroot00000000000000import copy import random from typing import Optional, Tuple import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as t_func from mindtorch.torch.nn.modules.utils import consume_prefix_in_state_dict_if_present class Hubert(nn.Module): def __init__(self, num_label_embeddings: int = 100, mask: bool = True): super().__init__() self._mask = mask self.feature_extractor = FeatureExtractor() self.feature_projection = FeatureProjection() self.positional_embedding = PositionalConvEmbedding() self.norm = nn.LayerNorm(768) self.dropout = nn.Dropout(0.1) self.encoder = TransformerEncoder( nn.TransformerEncoderLayer( 768, 12, 3072, activation="gelu", batch_first=True ), 12, ) self.proj = nn.Linear(768, 256) self.masked_spec_embed = nn.Parameter(torch.FloatTensor(768).uniform_()) self.label_embedding = nn.Embedding(num_label_embeddings, 256) def mask(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: mask = None if self.training and self._mask: mask = _compute_mask((x.size(0), x.size(1)), 0.8, 10, x.device, 2) x[mask] = self.masked_spec_embed.to(x.dtype) return x, mask def encode( self, x: torch.Tensor, layer: Optional[int] = None ) -> Tuple[torch.Tensor, torch.Tensor]: x = self.feature_extractor(x) x = self.feature_projection(x.transpose(1, 2)) x, mask = self.mask(x) x = x + self.positional_embedding(x) x = self.dropout(self.norm(x)) x = self.encoder(x, output_layer=layer) return x, mask def logits(self, x: torch.Tensor) -> torch.Tensor: logits = torch.cosine_similarity( x.unsqueeze(2), self.label_embedding.weight.unsqueeze(0).unsqueeze(0), dim=-1, ) return logits / 0.1 def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: x, mask = self.encode(x) x = self.proj(x) logits = self.logits(x) return logits, mask class HubertSoft(Hubert): def __init__(self): super().__init__() @torch.inference_mode() def units(self, wav: torch.Tensor) -> torch.Tensor: wav = t_func.pad(wav, ((400 - 320) // 2, (400 - 320) // 2)) x, _ = self.encode(wav) return self.proj(x) class FeatureExtractor(nn.Module): def __init__(self): super().__init__() self.conv0 = nn.Conv1d(1, 512, 10, 5, bias=False) self.norm0 = nn.GroupNorm(512, 512) self.conv1 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv2 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv3 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv4 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv5 = nn.Conv1d(512, 512, 2, 2, bias=False) self.conv6 = nn.Conv1d(512, 512, 2, 2, bias=False) def forward(self, x: torch.Tensor) -> torch.Tensor: x = t_func.gelu(self.norm0(self.conv0(x))) x = t_func.gelu(self.conv1(x)) x = t_func.gelu(self.conv2(x)) x = t_func.gelu(self.conv3(x)) x = t_func.gelu(self.conv4(x)) x = t_func.gelu(self.conv5(x)) x = t_func.gelu(self.conv6(x)) return x class FeatureProjection(nn.Module): def __init__(self): super().__init__() self.norm = nn.LayerNorm(512) self.projection = nn.Linear(512, 768) self.dropout = nn.Dropout(0.1) def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.norm(x) x = self.projection(x) x = self.dropout(x) return x class PositionalConvEmbedding(nn.Module): def __init__(self): super().__init__() self.conv = nn.Conv1d( 768, 768, kernel_size=128, padding=128 // 2, groups=16, ) self.conv = nn.utils.weight_norm(self.conv, name="weight", dim=2) def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.conv(x.transpose(1, 2)) x = t_func.gelu(x[:, :, :-1]) return x.transpose(1, 2) class TransformerEncoder(nn.Module): def __init__( self, encoder_layer: nn.TransformerEncoderLayer, num_layers: int ) -> None: super(TransformerEncoder, self).__init__() self.layers = nn.ModuleList( [copy.deepcopy(encoder_layer) for _ in range(num_layers)] ) self.num_layers = num_layers def forward( self, src: torch.Tensor, mask: torch.Tensor = None, src_key_padding_mask: torch.Tensor = None, output_layer: Optional[int] = None, ) -> torch.Tensor: output = src for layer in self.layers[:output_layer]: output = layer( output, src_mask=mask, src_key_padding_mask=src_key_padding_mask ) return output def _compute_mask( shape: Tuple[int, int], mask_prob: float, mask_length: int, device: torch.device, min_masks: int = 0, ) -> torch.Tensor: batch_size, sequence_length = shape if mask_length < 1: raise ValueError("`mask_length` has to be bigger than 0.") if mask_length > sequence_length: raise ValueError( f"`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: {mask_length} and `sequence_length`: {sequence_length}`" ) # compute number of masked spans in batch num_masked_spans = int(mask_prob * sequence_length / mask_length + random.random()) num_masked_spans = max(num_masked_spans, min_masks) # make sure num masked indices <= sequence_length if num_masked_spans * mask_length > sequence_length: num_masked_spans = sequence_length // mask_length # SpecAugment mask to fill mask = torch.zeros((batch_size, sequence_length), device=device, dtype=torch.bool) # uniform distribution to sample from, make sure that offset samples are < sequence_length uniform_dist = torch.ones( (batch_size, sequence_length - (mask_length - 1)), device=device ) # get random indices to mask mask_indices = torch.multinomial(uniform_dist, num_masked_spans) # expand masked indices to masked spans mask_indices = ( mask_indices.unsqueeze(dim=-1) .expand((batch_size, num_masked_spans, mask_length)) .reshape(batch_size, num_masked_spans * mask_length) ) offsets = ( torch.arange(mask_length, device=device)[None, None, :] .expand((batch_size, num_masked_spans, mask_length)) .reshape(batch_size, num_masked_spans * mask_length) ) mask_idxs = mask_indices + offsets # scatter indices to mask mask = mask.scatter(1, mask_idxs, True) return mask def hubert_soft( path: str, ) -> HubertSoft: r"""HuBERT-Soft from `"A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion"`. Args: path (str): path of a pretrained model """ hubert = HubertSoft() checkpoint = torch.load(path) consume_prefix_in_state_dict_if_present(checkpoint, "module.") hubert.load_state_dict(checkpoint) hubert.eval() return hubert so-vits-svc-npu/vencoder/hubert/hubert_model_onnx.py000066400000000000000000000160511456161405400232470ustar00rootroot00000000000000import copy import random from typing import Optional, Tuple import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as t_func from mindtorch.torch.nn.modules.utils import consume_prefix_in_state_dict_if_present class Hubert(nn.Module): def __init__(self, num_label_embeddings: int = 100, mask: bool = True): super().__init__() self._mask = mask self.feature_extractor = FeatureExtractor() self.feature_projection = FeatureProjection() self.positional_embedding = PositionalConvEmbedding() self.norm = nn.LayerNorm(768) self.dropout = nn.Dropout(0.1) self.encoder = TransformerEncoder( nn.TransformerEncoderLayer( 768, 12, 3072, activation="gelu", batch_first=True ), 12, ) self.proj = nn.Linear(768, 256) self.masked_spec_embed = nn.Parameter(torch.FloatTensor(768).uniform_()) self.label_embedding = nn.Embedding(num_label_embeddings, 256) def mask(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: mask = None if self.training and self._mask: mask = _compute_mask((x.size(0), x.size(1)), 0.8, 10, x.device, 2) x[mask] = self.masked_spec_embed.to(x.dtype) return x, mask def encode( self, x: torch.Tensor, layer: Optional[int] = None ) -> Tuple[torch.Tensor, torch.Tensor]: x = self.feature_extractor(x) x = self.feature_projection(x.transpose(1, 2)) x, mask = self.mask(x) x = x + self.positional_embedding(x) x = self.dropout(self.norm(x)) x = self.encoder(x, output_layer=layer) return x, mask def logits(self, x: torch.Tensor) -> torch.Tensor: logits = torch.cosine_similarity( x.unsqueeze(2), self.label_embedding.weight.unsqueeze(0).unsqueeze(0), dim=-1, ) return logits / 0.1 class HubertSoft(Hubert): def __init__(self): super().__init__() def units(self, wav: torch.Tensor) -> torch.Tensor: wav = t_func.pad(wav, ((400 - 320) // 2, (400 - 320) // 2)) x, _ = self.encode(wav) return self.proj(x) def forward(self, x): return self.units(x) class FeatureExtractor(nn.Module): def __init__(self): super().__init__() self.conv0 = nn.Conv1d(1, 512, 10, 5, bias=False) self.norm0 = nn.GroupNorm(512, 512) self.conv1 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv2 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv3 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv4 = nn.Conv1d(512, 512, 3, 2, bias=False) self.conv5 = nn.Conv1d(512, 512, 2, 2, bias=False) self.conv6 = nn.Conv1d(512, 512, 2, 2, bias=False) def forward(self, x: torch.Tensor) -> torch.Tensor: x = t_func.gelu(self.norm0(self.conv0(x))) x = t_func.gelu(self.conv1(x)) x = t_func.gelu(self.conv2(x)) x = t_func.gelu(self.conv3(x)) x = t_func.gelu(self.conv4(x)) x = t_func.gelu(self.conv5(x)) x = t_func.gelu(self.conv6(x)) return x class FeatureProjection(nn.Module): def __init__(self): super().__init__() self.norm = nn.LayerNorm(512) self.projection = nn.Linear(512, 768) self.dropout = nn.Dropout(0.1) def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.norm(x) x = self.projection(x) x = self.dropout(x) return x class PositionalConvEmbedding(nn.Module): def __init__(self): super().__init__() self.conv = nn.Conv1d( 768, 768, kernel_size=128, padding=128 // 2, groups=16, ) self.conv = nn.utils.weight_norm(self.conv, name="weight", dim=2) def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.conv(x.transpose(1, 2)) x = t_func.gelu(x[:, :, :-1]) return x.transpose(1, 2) class TransformerEncoder(nn.Module): def __init__( self, encoder_layer: nn.TransformerEncoderLayer, num_layers: int ) -> None: super(TransformerEncoder, self).__init__() self.layers = nn.ModuleList( [copy.deepcopy(encoder_layer) for _ in range(num_layers)] ) self.num_layers = num_layers def forward( self, src: torch.Tensor, mask: torch.Tensor = None, src_key_padding_mask: torch.Tensor = None, output_layer: Optional[int] = None, ) -> torch.Tensor: output = src for layer in self.layers[:output_layer]: output = layer( output, src_mask=mask, src_key_padding_mask=src_key_padding_mask ) return output def _compute_mask( shape: Tuple[int, int], mask_prob: float, mask_length: int, device: torch.device, min_masks: int = 0, ) -> torch.Tensor: batch_size, sequence_length = shape if mask_length < 1: raise ValueError("`mask_length` has to be bigger than 0.") if mask_length > sequence_length: raise ValueError( f"`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: {mask_length} and `sequence_length`: {sequence_length}`" ) # compute number of masked spans in batch num_masked_spans = int(mask_prob * sequence_length / mask_length + random.random()) num_masked_spans = max(num_masked_spans, min_masks) # make sure num masked indices <= sequence_length if num_masked_spans * mask_length > sequence_length: num_masked_spans = sequence_length // mask_length # SpecAugment mask to fill mask = torch.zeros((batch_size, sequence_length), device=device, dtype=torch.bool) # uniform distribution to sample from, make sure that offset samples are < sequence_length uniform_dist = torch.ones( (batch_size, sequence_length - (mask_length - 1)), device=device ) # get random indices to mask mask_indices = torch.multinomial(uniform_dist, num_masked_spans) # expand masked indices to masked spans mask_indices = ( mask_indices.unsqueeze(dim=-1) .expand((batch_size, num_masked_spans, mask_length)) .reshape(batch_size, num_masked_spans * mask_length) ) offsets = ( torch.arange(mask_length, device=device)[None, None, :] .expand((batch_size, num_masked_spans, mask_length)) .reshape(batch_size, num_masked_spans * mask_length) ) mask_idxs = mask_indices + offsets # scatter indices to mask mask = mask.scatter(1, mask_idxs, True) return mask def hubert_soft( path: str, ) -> HubertSoft: r"""HuBERT-Soft from `"A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion"`. Args: path (str): path of a pretrained model """ hubert = HubertSoft() checkpoint = torch.load(path) consume_prefix_in_state_dict_if_present(checkpoint, "module.") hubert.load_state_dict(checkpoint) hubert.eval() return hubert so-vits-svc-npu/vencoder/wavlm/000077500000000000000000000000001456161405400170145ustar00rootroot00000000000000so-vits-svc-npu/vencoder/wavlm/WavLM.py000066400000000000000000000657131456161405400203700ustar00rootroot00000000000000# -------------------------------------------------------- # WavLM: Large-Scale Self-Supervised Pre-training for Full Stack Speech Processing (https://arxiv.org/abs/2110.13900.pdf) # Github source: https://github.com/microsoft/unilm/tree/master/wavlm # Copyright (c) 2021 Microsoft # Licensed under The MIT License [see LICENSE for details] # Based on fairseq code bases # https://github.com/pytorch/fairseq # -------------------------------------------------------- import logging import math from typing import List, Optional, Tuple import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn as nn import mindtorch.torch.nn.functional as F from mindtorch.torch.nn import LayerNorm from vencoder.wavlm.modules import ( Fp32GroupNorm, Fp32LayerNorm, GLU_Linear, GradMultiply, MultiheadAttention, SamePad, TransposeLast, get_activation_fn, init_bert_params, ) logger = logging.getLogger(__name__) def compute_mask_indices( shape: Tuple[int, int], padding_mask: Optional[torch.Tensor], mask_prob: float, mask_length: int, mask_type: str = "static", mask_other: float = 0.0, min_masks: int = 0, no_overlap: bool = False, min_space: int = 0, ) -> np.ndarray: """ Computes random mask spans for a given shape Args: shape: the the shape for which to compute masks. should be of size 2 where first element is batch size and 2nd is timesteps padding_mask: optional padding mask of the same size as shape, which will prevent masking padded elements mask_prob: probability for each token to be chosen as start of the span to be masked. this will be multiplied by number of timesteps divided by length of mask span to mask approximately this percentage of all elements. however due to overlaps, the actual number will be smaller (unless no_overlap is True) mask_type: how to compute mask lengths static = fixed size uniform = sample from uniform distribution [mask_other, mask_length*2] normal = sample from normal distribution with mean mask_length and stdev mask_other. mask is min 1 element poisson = sample from possion distribution with lambda = mask length min_masks: minimum number of masked spans no_overlap: if false, will switch to an alternative recursive algorithm that prevents spans from overlapping min_space: only used if no_overlap is True, this is how many elements to keep unmasked between spans """ bsz, all_sz = shape mask = np.full((bsz, all_sz), False) all_num_mask = int( # add a random number for probabilistic rounding mask_prob * all_sz / float(mask_length) + np.random.rand() ) all_num_mask = max(min_masks, all_num_mask) mask_idcs = [] for i in range(bsz): if padding_mask is not None: sz = all_sz - padding_mask[i].long().sum().item() num_mask = int( # add a random number for probabilistic rounding mask_prob * sz / float(mask_length) + np.random.rand() ) num_mask = max(min_masks, num_mask) else: sz = all_sz num_mask = all_num_mask if mask_type == "static": lengths = np.full(num_mask, mask_length) elif mask_type == "uniform": lengths = np.random.randint(mask_other, mask_length * 2 + 1, size=num_mask) elif mask_type == "normal": lengths = np.random.normal(mask_length, mask_other, size=num_mask) lengths = [max(1, int(round(x))) for x in lengths] elif mask_type == "poisson": lengths = np.random.poisson(mask_length, size=num_mask) lengths = [int(round(x)) for x in lengths] else: raise Exception("unknown mask selection " + mask_type) if sum(lengths) == 0: lengths[0] = min(mask_length, sz - 1) if no_overlap: mask_idc = [] def arrange(s, e, length, keep_length): span_start = np.random.randint(s, e - length) mask_idc.extend(span_start + i for i in range(length)) new_parts = [] if span_start - s - min_space >= keep_length: new_parts.append((s, span_start - min_space + 1)) if e - span_start - keep_length - min_space > keep_length: new_parts.append((span_start + length + min_space, e)) return new_parts parts = [(0, sz)] min_length = min(lengths) for length in sorted(lengths, reverse=True): lens = np.fromiter( (e - s if e - s >= length + min_space else 0 for s, e in parts), np.int, ) l_sum = np.sum(lens) if l_sum == 0: break probs = lens / np.sum(lens) c = np.random.choice(len(parts), p=probs) s, e = parts.pop(c) parts.extend(arrange(s, e, length, min_length)) mask_idc = np.asarray(mask_idc) else: min_len = min(lengths) if sz - min_len <= num_mask: min_len = sz - num_mask - 1 mask_idc = np.random.choice(sz - min_len, num_mask, replace=False) mask_idc = np.asarray( [ mask_idc[j] + offset for j in range(len(mask_idc)) for offset in range(lengths[j]) ] ) mask_idcs.append(np.unique(mask_idc[mask_idc < sz])) min_len = min([len(m) for m in mask_idcs]) for i, mask_idc in enumerate(mask_idcs): if len(mask_idc) > min_len: mask_idc = np.random.choice(mask_idc, min_len, replace=False) mask[i, mask_idc] = True return mask class WavLMConfig: def __init__(self, cfg=None): self.extractor_mode: str = "default" # mode for feature extractor. default has a single group norm with d groups in the first conv block, whereas layer_norm has layer norms in every block (meant to use with normalize=True) self.encoder_layers: int = 12 # num encoder layers in the transformer self.encoder_embed_dim: int = 768 # encoder embedding dimension self.encoder_ffn_embed_dim: int = 3072 # encoder embedding dimension for FFN self.encoder_attention_heads: int = 12 # num encoder attention heads self.activation_fn: str = "gelu" # activation function to use self.layer_norm_first: bool = False # apply layernorm first in the transformer self.conv_feature_layers: str = "[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2" # string describing convolutional feature extraction layers in form of a python list that contains [(dim, kernel_size, stride), ...] self.conv_bias: bool = False # include bias in conv encoder self.feature_grad_mult: float = 1.0 # multiply feature extractor var grads by this self.normalize: bool = False # normalize input to have 0 mean and unit variance during training # dropouts self.dropout: float = 0.1 # dropout probability for the transformer self.attention_dropout: float = 0.1 # dropout probability for attention weights self.activation_dropout: float = 0.0 # dropout probability after activation in FFN self.encoder_layerdrop: float = 0.0 # probability of dropping a tarnsformer layer self.dropout_input: float = 0.0 # dropout to apply to the input (after feat extr) self.dropout_features: float = 0.0 # dropout to apply to the features (after feat extr) # masking self.mask_length: int = 10 # mask length self.mask_prob: float = 0.65 # probability of replacing a token with mask self.mask_selection: str = "static" # how to choose mask length self.mask_other: float = 0 # secondary mask argument (used for more complex distributions), see help in compute_mask_indicesh self.no_mask_overlap: bool = False # whether to allow masks to overlap self.mask_min_space: int = 1 # min space between spans (if no overlap is enabled) # channel masking self.mask_channel_length: int = 10 # length of the mask for features (channels) self.mask_channel_prob: float = 0.0 # probability of replacing a feature with 0 self.mask_channel_selection: str = "static" # how to choose mask length for channel masking self.mask_channel_other: float = 0 # secondary mask argument (used for more complex distributions), see help in compute_mask_indices self.no_mask_channel_overlap: bool = False # whether to allow channel masks to overlap self.mask_channel_min_space: int = 1 # min space between spans (if no overlap is enabled) # positional embeddings self.conv_pos: int = 128 # number of filters for convolutional positional embeddings self.conv_pos_groups: int = 16 # number of groups for convolutional positional embedding # relative position embedding self.relative_position_embedding: bool = False # apply relative position embedding self.num_buckets: int = 320 # number of buckets for relative position embedding self.max_distance: int = 1280 # maximum distance for relative position embedding self.gru_rel_pos: bool = False # apply gated relative position embedding if cfg is not None: self.update(cfg) def update(self, cfg: dict): self.__dict__.update(cfg) class WavLM(nn.Module): def __init__( self, cfg: WavLMConfig, ) -> None: super().__init__() logger.info(f"WavLM Config: {cfg.__dict__}") self.cfg = cfg feature_enc_layers = eval(cfg.conv_feature_layers) self.embed = feature_enc_layers[-1][0] self.feature_extractor = ConvFeatureExtractionModel( conv_layers=feature_enc_layers, dropout=0.0, mode=cfg.extractor_mode, conv_bias=cfg.conv_bias, ) self.post_extract_proj = ( nn.Linear(self.embed, cfg.encoder_embed_dim) if self.embed != cfg.encoder_embed_dim else None ) self.mask_prob = cfg.mask_prob self.mask_selection = cfg.mask_selection self.mask_other = cfg.mask_other self.mask_length = cfg.mask_length self.no_mask_overlap = cfg.no_mask_overlap self.mask_min_space = cfg.mask_min_space self.mask_channel_prob = cfg.mask_channel_prob self.mask_channel_selection = cfg.mask_channel_selection self.mask_channel_other = cfg.mask_channel_other self.mask_channel_length = cfg.mask_channel_length self.no_mask_channel_overlap = cfg.no_mask_channel_overlap self.mask_channel_min_space = cfg.mask_channel_min_space self.dropout_input = nn.Dropout(cfg.dropout_input) self.dropout_features = nn.Dropout(cfg.dropout_features) self.feature_grad_mult = cfg.feature_grad_mult self.mask_emb = nn.Parameter( torch.FloatTensor(cfg.encoder_embed_dim).uniform_() ) self.encoder = TransformerEncoder(cfg) self.layer_norm = LayerNorm(self.embed) def apply_mask(self, x, padding_mask): B, T, C = x.shape if self.mask_prob > 0: mask_indices = compute_mask_indices( (B, T), padding_mask, self.mask_prob, self.mask_length, self.mask_selection, self.mask_other, min_masks=2, no_overlap=self.no_mask_overlap, min_space=self.mask_min_space, ) mask_indices = torch.from_numpy(mask_indices).to(x.device) x[mask_indices] = self.mask_emb else: mask_indices = None if self.mask_channel_prob > 0: mask_channel_indices = compute_mask_indices( (B, C), None, self.mask_channel_prob, self.mask_channel_length, self.mask_channel_selection, self.mask_channel_other, no_overlap=self.no_mask_channel_overlap, min_space=self.mask_channel_min_space, ) mask_channel_indices = ( torch.from_numpy(mask_channel_indices) .to(x.device) .unsqueeze(1) .expand(-1, T, -1) ) x[mask_channel_indices] = 0 return x, mask_indices def forward_padding_mask( self, features: torch.Tensor, padding_mask: torch.Tensor, ) -> torch.Tensor: extra = padding_mask.size(1) % features.size(1) if extra > 0: padding_mask = padding_mask[:, :-extra] padding_mask = padding_mask.view( padding_mask.size(0), features.size(1), -1 ) padding_mask = padding_mask.all(-1) return padding_mask def extract_features( self, source: torch.Tensor, padding_mask: Optional[torch.Tensor] = None, mask: bool = False, ret_conv: bool = False, output_layer: Optional[int] = None, ret_layer_results: bool = False, ): if self.feature_grad_mult > 0: features = self.feature_extractor(source) if self.feature_grad_mult != 1.0: features = GradMultiply.apply(features, self.feature_grad_mult) else: with torch.no_grad(): features = self.feature_extractor(source) features = features.transpose(1, 2) features = self.layer_norm(features) if padding_mask is not None: padding_mask = self.forward_padding_mask(features, padding_mask) if self.post_extract_proj is not None: features = self.post_extract_proj(features) features = self.dropout_input(features) if mask: x, mask_indices = self.apply_mask( features, padding_mask ) else: x = features # feature: (B, T, D), float # target: (B, T), long # x: (B, T, D), float # padding_mask: (B, T), bool # mask_indices: (B, T), bool x, layer_results = self.encoder( x, padding_mask=padding_mask, layer=None if output_layer is None else output_layer - 1 ) res = {"x": x, "padding_mask": padding_mask, "features": features, "layer_results": layer_results} feature = res["features"] if ret_conv else res["x"] if ret_layer_results: feature = (feature, res["layer_results"]) return feature, res["padding_mask"] class ConvFeatureExtractionModel(nn.Module): def __init__( self, conv_layers: List[Tuple[int, int, int]], dropout: float = 0.0, mode: str = "default", conv_bias: bool = False, conv_type: str = "default" ): super().__init__() assert mode in {"default", "layer_norm"} def block( n_in, n_out, k, stride, is_layer_norm=False, is_group_norm=False, conv_bias=False, ): def make_conv(): conv = nn.Conv1d(n_in, n_out, k, stride=stride, bias=conv_bias) nn.init.kaiming_normal_(conv.weight) return conv assert (is_layer_norm and is_group_norm) is False, "layer norm and group norm are exclusive" if is_layer_norm: return nn.Sequential( make_conv(), nn.Dropout(p=dropout), nn.Sequential( TransposeLast(), Fp32LayerNorm(dim, elementwise_affine=True), TransposeLast(), ), nn.GELU(), ) elif is_group_norm: return nn.Sequential( make_conv(), nn.Dropout(p=dropout), Fp32GroupNorm(dim, dim, affine=True), nn.GELU(), ) else: return nn.Sequential(make_conv(), nn.Dropout(p=dropout), nn.GELU()) self.conv_type = conv_type if self.conv_type == "default": in_d = 1 self.conv_layers = nn.ModuleList() for i, cl in enumerate(conv_layers): assert len(cl) == 3, "invalid conv definition: " + str(cl) (dim, k, stride) = cl self.conv_layers.append( block( in_d, dim, k, stride, is_layer_norm=mode == "layer_norm", is_group_norm=mode == "default" and i == 0, conv_bias=conv_bias, ) ) in_d = dim elif self.conv_type == "conv2d": in_d = 1 self.conv_layers = nn.ModuleList() for i, cl in enumerate(conv_layers): assert len(cl) == 3 (dim, k, stride) = cl self.conv_layers.append( torch.nn.Conv2d(in_d, dim, k, stride) ) self.conv_layers.append(torch.nn.ReLU()) in_d = dim elif self.conv_type == "custom": in_d = 1 idim = 80 self.conv_layers = nn.ModuleList() for i, cl in enumerate(conv_layers): assert len(cl) == 3 (dim, k, stride) = cl self.conv_layers.append( torch.nn.Conv2d(in_d, dim, k, stride, padding=1) ) self.conv_layers.append( torch.nn.LayerNorm([dim, idim]) ) self.conv_layers.append(torch.nn.ReLU()) in_d = dim if (i + 1) % 2 == 0: self.conv_layers.append( torch.nn.MaxPool2d(2, stride=2, ceil_mode=True) ) idim = int(math.ceil(idim / 2)) else: pass def forward(self, x, mask=None): # BxT -> BxCxT x = x.unsqueeze(1) if self.conv_type == "custom": for conv in self.conv_layers: if isinstance(conv, nn.LayerNorm): x = x.transpose(1, 2) x = conv(x).transpose(1, 2) else: x = conv(x) x = x.transpose(2, 3).contiguous() x = x.view(x.size(0), -1, x.size(-1)) else: for conv in self.conv_layers: x = conv(x) if self.conv_type == "conv2d": b, c, t, f = x.size() x = x.transpose(2, 3).contiguous().view(b, c * f, t) return x class TransformerEncoder(nn.Module): def __init__(self, args): super().__init__() self.dropout = args.dropout self.embedding_dim = args.encoder_embed_dim self.pos_conv = nn.Conv1d( self.embedding_dim, self.embedding_dim, kernel_size=args.conv_pos, padding=args.conv_pos // 2, groups=args.conv_pos_groups, ) dropout = 0 std = math.sqrt((4 * (1.0 - dropout)) / (args.conv_pos * self.embedding_dim)) nn.init.normal_(self.pos_conv.weight, mean=0, std=std) nn.init.constant_(self.pos_conv.bias, 0) self.pos_conv = nn.utils.weight_norm(self.pos_conv, name="weight", dim=2) self.pos_conv = nn.Sequential(self.pos_conv, SamePad(args.conv_pos), nn.GELU()) if hasattr(args, "relative_position_embedding"): self.relative_position_embedding = args.relative_position_embedding self.num_buckets = args.num_buckets self.max_distance = args.max_distance else: self.relative_position_embedding = False self.num_buckets = 0 self.max_distance = 0 self.layers = nn.ModuleList( [ TransformerSentenceEncoderLayer( embedding_dim=self.embedding_dim, ffn_embedding_dim=args.encoder_ffn_embed_dim, num_attention_heads=args.encoder_attention_heads, dropout=self.dropout, attention_dropout=args.attention_dropout, activation_dropout=args.activation_dropout, activation_fn=args.activation_fn, layer_norm_first=args.layer_norm_first, has_relative_attention_bias=(self.relative_position_embedding and i == 0), num_buckets=self.num_buckets, max_distance=self.max_distance, gru_rel_pos=args.gru_rel_pos, ) for i in range(args.encoder_layers) ] ) self.layer_norm_first = args.layer_norm_first self.layer_norm = LayerNorm(self.embedding_dim) self.layerdrop = args.encoder_layerdrop self.apply(init_bert_params) def forward(self, x, padding_mask=None, streaming_mask=None, layer=None): x, layer_results = self.extract_features(x, padding_mask, streaming_mask, layer) if self.layer_norm_first and layer is None: x = self.layer_norm(x) return x, layer_results def extract_features(self, x, padding_mask=None, streaming_mask=None, tgt_layer=None): if padding_mask is not None: x[padding_mask] = 0 x_conv = self.pos_conv(x.transpose(1, 2)) x_conv = x_conv.transpose(1, 2) x = x + x_conv if not self.layer_norm_first: x = self.layer_norm(x) x = F.dropout(x, p=self.dropout, training=self.training) # B x T x C -> T x B x C x = x.transpose(0, 1) layer_results = [] z = None if tgt_layer is not None: layer_results.append((x, z)) r = None pos_bias = None for i, layer in enumerate(self.layers): dropout_probability = np.random.random() if not self.training or (dropout_probability > self.layerdrop): x, z, pos_bias = layer(x, self_attn_padding_mask=padding_mask, need_weights=False, self_attn_mask=streaming_mask, pos_bias=pos_bias) if tgt_layer is not None: layer_results.append((x, z)) if i == tgt_layer: r = x break if r is not None: x = r # T x B x C -> B x T x C x = x.transpose(0, 1) return x, layer_results class TransformerSentenceEncoderLayer(nn.Module): """ Implements a Transformer Encoder Layer used in BERT/XLM style pre-trained models. """ def __init__( self, embedding_dim: float = 768, ffn_embedding_dim: float = 3072, num_attention_heads: float = 8, dropout: float = 0.1, attention_dropout: float = 0.1, activation_dropout: float = 0.1, activation_fn: str = "relu", layer_norm_first: bool = False, has_relative_attention_bias: bool = False, num_buckets: int = 0, max_distance: int = 0, rescale_init: bool = False, gru_rel_pos: bool = False, ) -> None: super().__init__() # Initialize parameters self.embedding_dim = embedding_dim self.dropout = dropout self.activation_dropout = activation_dropout # Initialize blocks self.activation_name = activation_fn self.activation_fn = get_activation_fn(activation_fn) self.self_attn = MultiheadAttention( self.embedding_dim, num_attention_heads, dropout=attention_dropout, self_attention=True, has_relative_attention_bias=has_relative_attention_bias, num_buckets=num_buckets, max_distance=max_distance, rescale_init=rescale_init, gru_rel_pos=gru_rel_pos, ) self.dropout1 = nn.Dropout(dropout) self.dropout2 = nn.Dropout(self.activation_dropout) self.dropout3 = nn.Dropout(dropout) self.layer_norm_first = layer_norm_first # layer norm associated with the self attention layer self.self_attn_layer_norm = LayerNorm(self.embedding_dim) if self.activation_name == "glu": self.fc1 = GLU_Linear(self.embedding_dim, ffn_embedding_dim, "swish") else: self.fc1 = nn.Linear(self.embedding_dim, ffn_embedding_dim) self.fc2 = nn.Linear(ffn_embedding_dim, self.embedding_dim) # layer norm associated with the position wise feed-forward NN self.final_layer_norm = LayerNorm(self.embedding_dim) def forward( self, x: torch.Tensor, self_attn_mask: torch.Tensor = None, self_attn_padding_mask: torch.Tensor = None, need_weights: bool = False, pos_bias=None ): """ LayerNorm is applied either before or after the self-attention/ffn modules similar to the original Transformer imlementation. """ residual = x if self.layer_norm_first: x = self.self_attn_layer_norm(x) x, attn, pos_bias = self.self_attn( query=x, key=x, value=x, key_padding_mask=self_attn_padding_mask, need_weights=False, attn_mask=self_attn_mask, position_bias=pos_bias ) x = self.dropout1(x) x = residual + x residual = x x = self.final_layer_norm(x) if self.activation_name == "glu": x = self.fc1(x) else: x = self.activation_fn(self.fc1(x)) x = self.dropout2(x) x = self.fc2(x) x = self.dropout3(x) x = residual + x else: x, attn, pos_bias = self.self_attn( query=x, key=x, value=x, key_padding_mask=self_attn_padding_mask, need_weights=need_weights, attn_mask=self_attn_mask, position_bias=pos_bias ) x = self.dropout1(x) x = residual + x x = self.self_attn_layer_norm(x) residual = x if self.activation_name == "glu": x = self.fc1(x) else: x = self.activation_fn(self.fc1(x)) x = self.dropout2(x) x = self.fc2(x) x = self.dropout3(x) x = residual + x x = self.final_layer_norm(x) return x, attn, pos_bias so-vits-svc-npu/vencoder/wavlm/modules.py000066400000000000000000000763741456161405400210570ustar00rootroot00000000000000# -------------------------------------------------------- # WavLM: Large-Scale Self-Supervised Pre-training for Full Stack Speech Processing (https://arxiv.org/abs/2110.13900.pdf) # Github source: https://github.com/microsoft/unilm/tree/master/wavlm # Copyright (c) 2021 Microsoft # Licensed under The MIT License [see LICENSE for details] # Based on fairseq code bases # https://github.com/pytorch/fairseq # -------------------------------------------------------- import math import warnings from typing import Dict, Optional, Tuple import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import Tensor, nn from mindtorch.torch.nn import Parameter class TransposeLast(nn.Module): def __init__(self, deconstruct_idx=None): super().__init__() self.deconstruct_idx = deconstruct_idx def forward(self, x): if self.deconstruct_idx is not None: x = x[self.deconstruct_idx] return x.transpose(-2, -1) class Fp32LayerNorm(nn.LayerNorm): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) def forward(self, input): output = F.layer_norm( input.float(), self.normalized_shape, self.weight.float() if self.weight is not None else None, self.bias.float() if self.bias is not None else None, self.eps, ) return output.type_as(input) class Fp32GroupNorm(nn.GroupNorm): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) def forward(self, input): output = F.group_norm( input.float(), self.num_groups, self.weight.float() if self.weight is not None else None, self.bias.float() if self.bias is not None else None, self.eps, ) return output.type_as(input) class GradMultiply(torch.autograd.Function): @staticmethod def forward(ctx, x, scale): ctx.scale = scale res = x.new(x) return res @staticmethod def backward(ctx, grad): return grad * ctx.scale, None class SamePad(nn.Module): def __init__(self, kernel_size, causal=False): super().__init__() if causal: self.remove = kernel_size - 1 else: self.remove = 1 if kernel_size % 2 == 0 else 0 def forward(self, x): if self.remove > 0: x = x[:, :, : -self.remove] return x class Swish(nn.Module): """Swish function """ def __init__(self): """Construct an MultiHeadedAttention object.""" super(Swish, self).__init__() self.act = torch.nn.Sigmoid() def forward(self, x): return x * self.act(x) class GLU_Linear(nn.Module): def __init__(self, input_dim, output_dim, glu_type="sigmoid", bias_in_glu=True): super(GLU_Linear, self).__init__() self.glu_type = glu_type self.output_dim = output_dim if glu_type == "sigmoid": self.glu_act = torch.nn.Sigmoid() elif glu_type == "swish": self.glu_act = Swish() elif glu_type == "relu": self.glu_act = torch.nn.ReLU() elif glu_type == "gelu": self.glu_act = torch.nn.GELU() if bias_in_glu: self.linear = nn.Linear(input_dim, output_dim * 2, True) else: self.linear = nn.Linear(input_dim, output_dim * 2, False) def forward(self, x): # to be consistent with GLU_Linear, we assume the input always has the #channel (#dim) in the last dimension of the tensor, so need to switch the dimension first for 1D-Conv case x = self.linear(x) if self.glu_type == "bilinear": x = (x[:, :, 0:self.output_dim] * x[:, :, self.output_dim:self.output_dim * 2]) else: x = (x[:, :, 0:self.output_dim] * self.glu_act(x[:, :, self.output_dim:self.output_dim * 2])) return x def gelu_accurate(x): if not hasattr(gelu_accurate, "_a"): gelu_accurate._a = math.sqrt(2 / math.pi) return ( 0.5 * x * (1 + torch.tanh(gelu_accurate._a * (x + 0.044715 * torch.pow(x, 3)))) ) def gelu(x: torch.Tensor) -> torch.Tensor: return torch.nn.functional.gelu(x.float()).type_as(x) def get_activation_fn(activation: str): """Returns the activation function corresponding to `activation`""" if activation == "relu": return F.relu elif activation == "gelu": return gelu elif activation == "gelu_fast": warnings.warn( "--activation-fn=gelu_fast has been renamed to gelu_accurate" ) return gelu_accurate elif activation == "gelu_accurate": return gelu_accurate elif activation == "tanh": return torch.tanh elif activation == "linear": return lambda x: x elif activation == "glu": return lambda x: x else: raise RuntimeError("--activation-fn {} not supported".format(activation)) def init_bert_params(module): """ Initialize the weights specific to the BERT Model. This overrides the default initializations depending on the specified arguments. 1. If normal_init_linear_weights is set then weights of linear layer will be initialized using the normal distribution and bais will be set to the specified value. 2. If normal_init_embed_weights is set then weights of embedding layer will be initialized using the normal distribution. 3. If normal_init_proj_weights is set then weights of in_project_weight for MultiHeadAttention initialized using the normal distribution (to be validated). """ def normal_(data): # with FSDP, module params will be on CUDA, so we cast them back to CPU # so that the RNG is consistent with and without FSDP data.copy_( data.cpu().normal_(mean=0.0, std=0.02).to(data.device) ) if isinstance(module, nn.Linear): normal_(module.weight.data) if module.bias is not None: module.bias.data.zero_() if isinstance(module, nn.Embedding): normal_(module.weight.data) if module.padding_idx is not None: module.weight.data[module.padding_idx].zero_() if isinstance(module, MultiheadAttention): normal_(module.q_proj.weight.data) normal_(module.k_proj.weight.data) normal_(module.v_proj.weight.data) def quant_noise(module, p, block_size): """ Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described in "Training with Quantization Noise for Extreme Model Compression" Args: - module: nn.Module - p: amount of Quantization Noise - block_size: size of the blocks for subsequent quantization with iPQ Remarks: - Module weights must have the right sizes wrt the block size - Only Linear, Embedding and Conv2d modules are supported for the moment - For more detail on how to quantize by blocks with convolutional weights, see "And the Bit Goes Down: Revisiting the Quantization of Neural Networks" - We implement the simplest form of noise here as stated in the paper which consists in randomly dropping blocks """ # if no quantization noise, don't register hook if p <= 0: return module # supported modules assert isinstance(module, (nn.Linear, nn.Embedding, nn.Conv2d)) # test whether module.weight has the right sizes wrt block_size is_conv = module.weight.ndim == 4 # 2D matrix if not is_conv: assert ( module.weight.size(1) % block_size == 0 ), "Input features must be a multiple of block sizes" # 4D matrix else: # 1x1 convolutions if module.kernel_size == (1, 1): assert ( module.in_channels % block_size == 0 ), "Input channels must be a multiple of block sizes" # regular convolutions else: k = module.kernel_size[0] * module.kernel_size[1] assert k % block_size == 0, "Kernel size must be a multiple of block size" def _forward_pre_hook(mod, input): # no noise for evaluation if mod.training: if not is_conv: # gather weight and sizes weight = mod.weight in_features = weight.size(1) out_features = weight.size(0) # split weight matrix into blocks and randomly drop selected blocks mask = torch.zeros( in_features // block_size * out_features, device=weight.device ) mask.bernoulli_(p) mask = mask.repeat_interleave(block_size, -1).view(-1, in_features) else: # gather weight and sizes weight = mod.weight in_channels = mod.in_channels out_channels = mod.out_channels # split weight matrix into blocks and randomly drop selected blocks if mod.kernel_size == (1, 1): mask = torch.zeros( int(in_channels // block_size * out_channels), device=weight.device, ) mask.bernoulli_(p) mask = mask.repeat_interleave(block_size, -1).view(-1, in_channels) else: mask = torch.zeros( weight.size(0), weight.size(1), device=weight.device ) mask.bernoulli_(p) mask = ( mask.unsqueeze(2) .unsqueeze(3) .repeat(1, 1, mod.kernel_size[0], mod.kernel_size[1]) ) # scale weights and apply mask mask = mask.to( torch.bool ) # x.bool() is not currently supported in TorchScript s = 1 / (1 - p) mod.weight.data = s * weight.masked_fill(mask, 0) module.register_forward_pre_hook(_forward_pre_hook) return module class MultiheadAttention(nn.Module): """Multi-headed attention. See "Attention Is All You Need" for more details. """ def __init__( self, embed_dim, num_heads, kdim=None, vdim=None, dropout=0.0, bias=True, add_bias_kv=False, add_zero_attn=False, self_attention=False, encoder_decoder_attention=False, q_noise=0.0, qn_block_size=8, has_relative_attention_bias=False, num_buckets=32, max_distance=128, gru_rel_pos=False, rescale_init=False, ): super().__init__() self.embed_dim = embed_dim self.kdim = kdim if kdim is not None else embed_dim self.vdim = vdim if vdim is not None else embed_dim self.qkv_same_dim = self.kdim == embed_dim and self.vdim == embed_dim self.num_heads = num_heads self.dropout_module = nn.Dropout(dropout) self.has_relative_attention_bias = has_relative_attention_bias self.num_buckets = num_buckets self.max_distance = max_distance if self.has_relative_attention_bias: self.relative_attention_bias = nn.Embedding(num_buckets, num_heads) self.head_dim = embed_dim // num_heads self.q_head_dim = self.head_dim self.k_head_dim = self.head_dim assert ( self.head_dim * num_heads == self.embed_dim ), "embed_dim must be divisible by num_heads" self.scaling = self.head_dim ** -0.5 self.self_attention = self_attention self.encoder_decoder_attention = encoder_decoder_attention assert not self.self_attention or self.qkv_same_dim, ( "Self-attention requires query, key and " "value to be of the same size" ) k_bias = True if rescale_init: k_bias = False k_embed_dim = embed_dim q_embed_dim = embed_dim self.k_proj = quant_noise( nn.Linear(self.kdim, k_embed_dim, bias=k_bias), q_noise, qn_block_size ) self.v_proj = quant_noise( nn.Linear(self.vdim, embed_dim, bias=bias), q_noise, qn_block_size ) self.q_proj = quant_noise( nn.Linear(embed_dim, q_embed_dim, bias=bias), q_noise, qn_block_size ) self.out_proj = quant_noise( nn.Linear(embed_dim, embed_dim, bias=bias), q_noise, qn_block_size ) if add_bias_kv: self.bias_k = Parameter(torch.Tensor(1, 1, embed_dim)) self.bias_v = Parameter(torch.Tensor(1, 1, embed_dim)) else: self.bias_k = self.bias_v = None self.add_zero_attn = add_zero_attn self.gru_rel_pos = gru_rel_pos if self.gru_rel_pos: self.grep_linear = nn.Linear(self.q_head_dim, 8) self.grep_a = nn.Parameter(torch.ones(1, num_heads, 1, 1)) self.reset_parameters() def reset_parameters(self): if self.qkv_same_dim: # Empirically observed the convergence to be much better with # the scaled initialization nn.init.xavier_uniform_(self.k_proj.weight, gain=1 / math.sqrt(2)) nn.init.xavier_uniform_(self.v_proj.weight, gain=1 / math.sqrt(2)) nn.init.xavier_uniform_(self.q_proj.weight, gain=1 / math.sqrt(2)) else: nn.init.xavier_uniform_(self.k_proj.weight) nn.init.xavier_uniform_(self.v_proj.weight) nn.init.xavier_uniform_(self.q_proj.weight) nn.init.xavier_uniform_(self.out_proj.weight) if self.out_proj.bias is not None: nn.init.constant_(self.out_proj.bias, 0.0) if self.bias_k is not None: nn.init.xavier_normal_(self.bias_k) if self.bias_v is not None: nn.init.xavier_normal_(self.bias_v) if self.has_relative_attention_bias: nn.init.xavier_normal_(self.relative_attention_bias.weight) def _relative_positions_bucket(self, relative_positions, bidirectional=True): num_buckets = self.num_buckets max_distance = self.max_distance relative_buckets = 0 if bidirectional: num_buckets = num_buckets // 2 relative_buckets += (relative_positions > 0).to(torch.long) * num_buckets relative_positions = torch.abs(relative_positions) else: relative_positions = -torch.min(relative_positions, torch.zeros_like(relative_positions)) max_exact = num_buckets // 2 is_small = relative_positions < max_exact relative_postion_if_large = max_exact + ( torch.log(relative_positions.float() / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact) ).to(torch.long) relative_postion_if_large = torch.min( relative_postion_if_large, torch.full_like(relative_postion_if_large, num_buckets - 1) ) relative_buckets += torch.where(is_small, relative_positions, relative_postion_if_large) return relative_buckets def compute_bias(self, query_length, key_length): context_position = torch.arange(query_length, dtype=torch.long)[:, None] memory_position = torch.arange(key_length, dtype=torch.long)[None, :] relative_position = memory_position - context_position relative_position_bucket = self._relative_positions_bucket( relative_position, bidirectional=True ) relative_position_bucket = relative_position_bucket.to(self.relative_attention_bias.weight.device) values = self.relative_attention_bias(relative_position_bucket) values = values.permute([2, 0, 1]) return values def forward( self, query, key: Optional[Tensor], value: Optional[Tensor], key_padding_mask: Optional[Tensor] = None, incremental_state: Optional[Dict[str, Dict[str, Optional[Tensor]]]] = None, need_weights: bool = True, static_kv: bool = False, attn_mask: Optional[Tensor] = None, before_softmax: bool = False, need_head_weights: bool = False, position_bias: Optional[Tensor] = None ) -> Tuple[Tensor, Optional[Tensor], Optional[Tensor]]: """Input shape: Time x Batch x Channel Args: key_padding_mask (ByteTensor, optional): mask to exclude keys that are pads, of shape `(batch, src_len)`, where padding elements are indicated by 1s. need_weights (bool, optional): return the attention weights, averaged over heads (default: False). attn_mask (ByteTensor, optional): typically used to implement causal attention, where the mask prevents the attention from looking forward in time (default: None). before_softmax (bool, optional): return the raw attention weights and values before the attention softmax. need_head_weights (bool, optional): return the attention weights for each head. Implies *need_weights*. Default: return the average attention weights over all heads. """ if need_head_weights: need_weights = True is_tpu = query.device.type == "xla" tgt_len, bsz, embed_dim = query.size() src_len = tgt_len assert embed_dim == self.embed_dim assert list(query.size()) == [tgt_len, bsz, embed_dim] if key is not None: src_len, key_bsz, _ = key.size() if not torch.jit.is_scripting(): assert key_bsz == bsz assert value is not None assert src_len, bsz == value.shape[:2] if self.has_relative_attention_bias and position_bias is None: position_bias = self.compute_bias(tgt_len, src_len) position_bias = position_bias.unsqueeze(0).repeat(bsz, 1, 1, 1).view(bsz * self.num_heads, tgt_len, src_len) if ( not is_tpu # don't use PyTorch version on TPUs and incremental_state is None and not static_kv # A workaround for quantization to work. Otherwise JIT compilation # treats bias in linear module as method. and not torch.jit.is_scripting() and self.q_head_dim == self.head_dim ): assert key is not None and value is not None assert attn_mask is None attn_mask_rel_pos = None if position_bias is not None: attn_mask_rel_pos = position_bias if self.gru_rel_pos: query_layer = query.transpose(0, 1) new_x_shape = query_layer.size()[:-1] + (self.num_heads, -1) query_layer = query_layer.view(*new_x_shape) query_layer = query_layer.permute(0, 2, 1, 3) _B, _H, _L, __ = query_layer.size() gate_a, gate_b = torch.sigmoid(self.grep_linear(query_layer).view( _B, _H, _L, 2, 4).sum(-1, keepdim=False)).chunk(2, dim=-1) gate_a_1 = gate_a * (gate_b * self.grep_a - 1.0) + 2.0 attn_mask_rel_pos = gate_a_1.view(bsz * self.num_heads, -1, 1) * position_bias attn_mask_rel_pos = attn_mask_rel_pos.view((-1, tgt_len, tgt_len)) k_proj_bias = self.k_proj.bias if k_proj_bias is None: k_proj_bias = torch.zeros_like(self.q_proj.bias) x, attn = F.multi_head_attention_forward( query, key, value, self.embed_dim, self.num_heads, torch.empty([0]), torch.cat((self.q_proj.bias, self.k_proj.bias, self.v_proj.bias)), self.bias_k, self.bias_v, self.add_zero_attn, self.dropout_module.p, self.out_proj.weight, self.out_proj.bias, self.training, # self.training or self.dropout_module.apply_during_inference, key_padding_mask, need_weights, attn_mask_rel_pos, use_separate_proj_weight=True, q_proj_weight=self.q_proj.weight, k_proj_weight=self.k_proj.weight, v_proj_weight=self.v_proj.weight, ) return x, attn, position_bias if incremental_state is not None: saved_state = self._get_input_buffer(incremental_state) if saved_state is not None and "prev_key" in saved_state: # previous time steps are cached - no need to recompute # key and value if they are static if static_kv: assert self.encoder_decoder_attention and not self.self_attention key = value = None else: saved_state = None if self.self_attention: q = self.q_proj(query) k = self.k_proj(query) v = self.v_proj(query) elif self.encoder_decoder_attention: # encoder-decoder attention q = self.q_proj(query) if key is None: assert value is None k = v = None else: k = self.k_proj(key) v = self.v_proj(key) else: assert key is not None and value is not None q = self.q_proj(query) k = self.k_proj(key) v = self.v_proj(value) q *= self.scaling if self.bias_k is not None: assert self.bias_v is not None k = torch.cat([k, self.bias_k.repeat(1, bsz, 1)]) v = torch.cat([v, self.bias_v.repeat(1, bsz, 1)]) if attn_mask is not None: attn_mask = torch.cat( [attn_mask, attn_mask.new_zeros(attn_mask.size(0), 1)], dim=1 ) if key_padding_mask is not None: key_padding_mask = torch.cat( [ key_padding_mask, key_padding_mask.new_zeros(key_padding_mask.size(0), 1), ], dim=1, ) q = ( q.contiguous() .view(tgt_len, bsz * self.num_heads, self.q_head_dim) .transpose(0, 1) ) if k is not None: k = ( k.contiguous() .view(-1, bsz * self.num_heads, self.k_head_dim) .transpose(0, 1) ) if v is not None: v = ( v.contiguous() .view(-1, bsz * self.num_heads, self.head_dim) .transpose(0, 1) ) if saved_state is not None: # saved states are stored with shape (bsz, num_heads, seq_len, head_dim) if "prev_key" in saved_state: _prev_key = saved_state["prev_key"] assert _prev_key is not None prev_key = _prev_key.view(bsz * self.num_heads, -1, self.head_dim) if static_kv: k = prev_key else: assert k is not None k = torch.cat([prev_key, k], dim=1) src_len = k.size(1) if "prev_value" in saved_state: _prev_value = saved_state["prev_value"] assert _prev_value is not None prev_value = _prev_value.view(bsz * self.num_heads, -1, self.head_dim) if static_kv: v = prev_value else: assert v is not None v = torch.cat([prev_value, v], dim=1) prev_key_padding_mask: Optional[Tensor] = None if "prev_key_padding_mask" in saved_state: prev_key_padding_mask = saved_state["prev_key_padding_mask"] assert k is not None and v is not None key_padding_mask = MultiheadAttention._append_prev_key_padding_mask( key_padding_mask=key_padding_mask, prev_key_padding_mask=prev_key_padding_mask, batch_size=bsz, src_len=k.size(1), static_kv=static_kv, ) saved_state["prev_key"] = k.view(bsz, self.num_heads, -1, self.head_dim) saved_state["prev_value"] = v.view(bsz, self.num_heads, -1, self.head_dim) saved_state["prev_key_padding_mask"] = key_padding_mask # In this branch incremental_state is never None assert incremental_state is not None incremental_state = self._set_input_buffer(incremental_state, saved_state) assert k is not None assert k.size(1) == src_len # This is part of a workaround to get around fork/join parallelism # not supporting Optional types. if key_padding_mask is not None and key_padding_mask.dim() == 0: key_padding_mask = None if key_padding_mask is not None: assert key_padding_mask.size(0) == bsz assert key_padding_mask.size(1) == src_len if self.add_zero_attn: assert v is not None src_len += 1 k = torch.cat([k, k.new_zeros((k.size(0), 1) + k.size()[2:])], dim=1) v = torch.cat([v, v.new_zeros((v.size(0), 1) + v.size()[2:])], dim=1) if attn_mask is not None: attn_mask = torch.cat( [attn_mask, attn_mask.new_zeros(attn_mask.size(0), 1)], dim=1 ) if key_padding_mask is not None: key_padding_mask = torch.cat( [ key_padding_mask, torch.zeros(key_padding_mask.size(0), 1).type_as( key_padding_mask ), ], dim=1, ) attn_weights = torch.bmm(q, k.transpose(1, 2)) attn_weights = self.apply_sparse_mask(attn_weights, tgt_len, src_len, bsz) assert list(attn_weights.size()) == [bsz * self.num_heads, tgt_len, src_len] if attn_mask is not None: attn_mask = attn_mask.unsqueeze(0) attn_weights += attn_mask if key_padding_mask is not None: # don't attend to padding symbols attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) if not is_tpu: attn_weights = attn_weights.masked_fill( key_padding_mask.unsqueeze(1).unsqueeze(2).to(torch.bool), float("-inf"), ) else: attn_weights = attn_weights.transpose(0, 2) attn_weights = attn_weights.masked_fill(key_padding_mask, float("-inf")) attn_weights = attn_weights.transpose(0, 2) attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len) if before_softmax: return attn_weights, v, position_bias if position_bias is not None: if self.gru_rel_pos == 1: query_layer = q.view(bsz, self.num_heads, tgt_len, self.q_head_dim) _B, _H, _L, __ = query_layer.size() gate_a, gate_b = torch.sigmoid(self.grep_linear(query_layer).view( _B, _H, _L, 2, 4).sum(-1, keepdim=False)).chunk(2, dim=-1) gate_a_1 = gate_a * (gate_b * self.grep_a - 1.0) + 2.0 position_bias = gate_a_1.view(bsz * self.num_heads, -1, 1) * position_bias position_bias = position_bias.view(attn_weights.size()) attn_weights = attn_weights + position_bias attn_weights_float = F.softmax( attn_weights, dim=-1 ) attn_weights = attn_weights_float.type_as(attn_weights) attn_probs = self.dropout_module(attn_weights) assert v is not None attn = torch.bmm(attn_probs, v) assert list(attn.size()) == [bsz * self.num_heads, tgt_len, self.head_dim] attn = attn.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim) attn = self.out_proj(attn) attn_weights: Optional[Tensor] = None if need_weights: attn_weights = attn_weights_float.view( bsz, self.num_heads, tgt_len, src_len ).transpose(1, 0) if not need_head_weights: # average attention weights over heads attn_weights = attn_weights.mean(dim=0) return attn, attn_weights, position_bias @staticmethod def _append_prev_key_padding_mask( key_padding_mask: Optional[Tensor], prev_key_padding_mask: Optional[Tensor], batch_size: int, src_len: int, static_kv: bool, ) -> Optional[Tensor]: # saved key padding masks have shape (bsz, seq_len) if prev_key_padding_mask is not None and static_kv: new_key_padding_mask = prev_key_padding_mask elif prev_key_padding_mask is not None and key_padding_mask is not None: new_key_padding_mask = torch.cat( [prev_key_padding_mask.float(), key_padding_mask.float()], dim=1 ) # During incremental decoding, as the padding token enters and # leaves the frame, there will be a time when prev or current # is None elif prev_key_padding_mask is not None: if src_len > prev_key_padding_mask.size(1): filler = torch.zeros( (batch_size, src_len - prev_key_padding_mask.size(1)), device=prev_key_padding_mask.device, ) new_key_padding_mask = torch.cat( [prev_key_padding_mask.float(), filler.float()], dim=1 ) else: new_key_padding_mask = prev_key_padding_mask.float() elif key_padding_mask is not None: if src_len > key_padding_mask.size(1): filler = torch.zeros( (batch_size, src_len - key_padding_mask.size(1)), device=key_padding_mask.device, ) new_key_padding_mask = torch.cat( [filler.float(), key_padding_mask.float()], dim=1 ) else: new_key_padding_mask = key_padding_mask.float() else: new_key_padding_mask = prev_key_padding_mask return new_key_padding_mask def _get_input_buffer( self, incremental_state: Optional[Dict[str, Dict[str, Optional[Tensor]]]] ) -> Dict[str, Optional[Tensor]]: result = self.get_incremental_state(incremental_state, "attn_state") if result is not None: return result else: empty_result: Dict[str, Optional[Tensor]] = {} return empty_result def _set_input_buffer( self, incremental_state: Dict[str, Dict[str, Optional[Tensor]]], buffer: Dict[str, Optional[Tensor]], ): return self.set_incremental_state(incremental_state, "attn_state", buffer) def apply_sparse_mask(self, attn_weights, tgt_len: int, src_len: int, bsz: int): return attn_weights so-vits-svc-npu/vencoder/whisper/000077500000000000000000000000001456161405400173475ustar00rootroot00000000000000so-vits-svc-npu/vencoder/whisper/__init__.py000066400000000000000000000000001456161405400214460ustar00rootroot00000000000000so-vits-svc-npu/vencoder/whisper/audio.py000066400000000000000000000077731456161405400210400ustar00rootroot00000000000000from functools import lru_cache from typing import Union import ffmpeg import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from librosa.filters import mel as librosa_mel_fn from .utils import exact_div # hard-coded audio hyperparameters SAMPLE_RATE = 16000 N_FFT = 400 N_MELS = 80 HOP_LENGTH = 160 CHUNK_LENGTH = 30 N_SAMPLES = CHUNK_LENGTH * SAMPLE_RATE # 480000: number of samples in a chunk N_FRAMES = exact_div(N_SAMPLES, HOP_LENGTH) # 3000: number of frames in a mel spectrogram input def load_audio(file: str, sr: int = SAMPLE_RATE): """ Open an audio file and read as mono waveform, resampling as necessary Parameters ---------- file: str The audio file to open sr: int The sample rate to resample the audio if necessary Returns ------- A NumPy array containing the audio waveform, in float32 dtype. """ try: # This launches a subprocess to decode audio while down-mixing and resampling as necessary. # Requires the ffmpeg CLI and `ffmpeg-python` package to be installed. out, _ = ( ffmpeg.input(file, threads=0) .output("-", format="s16le", acodec="pcm_s16le", ac=1, ar=sr) .run(cmd=["ffmpeg", "-nostdin"], capture_stdout=True, capture_stderr=True) ) except ffmpeg.Error as e: raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0 def pad_or_trim(array, length: int = N_SAMPLES, *, axis: int = -1): """ Pad or trim the audio array to N_SAMPLES, as expected by the encoder. """ if torch.is_tensor(array): if array.shape[axis] > length: array = array.index_select(dim=axis, index=torch.arange(length, device=array.device)) if array.shape[axis] < length: pad_widths = [(0, 0)] * array.ndim pad_widths[axis] = (0, length - array.shape[axis]) array = F.pad(array, [pad for sizes in pad_widths[::-1] for pad in sizes]) else: if array.shape[axis] > length: array = array.take(indices=range(length), axis=axis) if array.shape[axis] < length: pad_widths = [(0, 0)] * array.ndim pad_widths[axis] = (0, length - array.shape[axis]) array = np.pad(array, pad_widths) return array @lru_cache(maxsize=None) def mel_filters(device, n_mels: int = N_MELS) -> torch.Tensor: """ load the mel filterbank matrix for projecting STFT into a Mel spectrogram. Allows decoupling librosa dependency; saved using: np.savez_compressed( "mel_filters.npz", mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80), ) """ assert n_mels == 80, f"Unsupported n_mels: {n_mels}" return torch.from_numpy(librosa_mel_fn(sr=SAMPLE_RATE,n_fft=N_FFT,n_mels=n_mels)).to(device) def log_mel_spectrogram(audio: Union[str, np.ndarray, torch.Tensor], n_mels: int = N_MELS): """ Compute the log-Mel spectrogram of Parameters ---------- audio: Union[str, np.ndarray, torch.Tensor], shape = (*) The path to audio or either a NumPy array or Tensor containing the audio waveform in 16 kHz n_mels: int The number of Mel-frequency filters, only 80 is supported Returns ------- torch.Tensor, shape = (80, n_frames) A Tensor that contains the Mel spectrogram """ if not torch.is_tensor(audio): if isinstance(audio, str): audio = load_audio(audio) audio = torch.from_numpy(audio) window = torch.hann_window(N_FFT).to(audio.device) stft = torch.stft(audio, N_FFT, HOP_LENGTH, window=window, return_complex=True) magnitudes = stft[..., :-1].abs() ** 2 filters = mel_filters(audio.device, n_mels) mel_spec = filters @ magnitudes log_spec = torch.clamp(mel_spec, min=1e-10).log10() log_spec = torch.maximum(log_spec, log_spec.max() - 8.0) log_spec = (log_spec + 4.0) / 4.0 return log_spec so-vits-svc-npu/vencoder/whisper/decoding.py000066400000000000000000000727751456161405400215170ustar00rootroot00000000000000from dataclasses import dataclass, field from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Sequence, Tuple, Union import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import Tensor from mindtorch.torch.distributions import Categorical from .audio import CHUNK_LENGTH from .tokenizer import Tokenizer, get_tokenizer from .utils import compression_ratio if TYPE_CHECKING: from .model import Whisper @torch.no_grad() def detect_language(model: "Whisper", mel: Tensor, tokenizer: Tokenizer = None) -> Tuple[Tensor, List[dict]]: """ Detect the spoken language in the audio, and return them as list of strings, along with the ids of the most probable language tokens and the probability distribution over all language tokens. This is performed outside the main decode loop in order to not interfere with kv-caching. Returns ------- language_tokens : Tensor, shape = (n_audio,) ids of the most probable language tokens, which appears after the startoftranscript token. language_probs : List[Dict[str, float]], length = n_audio list of dictionaries containing the probability distribution over all languages. """ if tokenizer is None: tokenizer = get_tokenizer(model.is_multilingual) if tokenizer.language is None or tokenizer.language_token not in tokenizer.sot_sequence: raise ValueError("This model doesn't have language tokens so it can't perform lang id") single = mel.ndim == 2 if single: mel = mel.unsqueeze(0) # skip encoder forward pass if already-encoded audio features were given if mel.shape[-2:] != (model.dims.n_audio_ctx, model.dims.n_audio_state): mel = model.encoder(mel) # forward pass using a single token, startoftranscript n_audio = mel.shape[0] x = torch.tensor([[tokenizer.sot]] * n_audio).to(mel.device) # [n_audio, 1] logits = model.logits(x, mel)[:, 0] # collect detected languages; suppress all non-language tokens mask = torch.ones(logits.shape[-1], dtype=torch.bool) mask[list(tokenizer.all_language_tokens)] = False logits[:, mask] = -np.inf language_tokens = logits.argmax(dim=-1) language_token_probs = logits.softmax(dim=-1).cpu() language_probs = [ { c: language_token_probs[i, j].item() for j, c in zip(tokenizer.all_language_tokens, tokenizer.all_language_codes) } for i in range(n_audio) ] if single: language_tokens = language_tokens[0] language_probs = language_probs[0] return language_tokens, language_probs @dataclass(frozen=True) class DecodingOptions: task: str = "transcribe" # whether to perform X->X "transcribe" or X->English "translate" language: Optional[str] = None # language that the audio is in; uses detected language if None # sampling-related options temperature: float = 0.0 sample_len: Optional[int] = None # maximum number of tokens to sample best_of: Optional[int] = None # number of independent samples to collect, when t > 0 beam_size: Optional[int] = None # number of beams in beam search, when t == 0 patience: Optional[float] = None # patience in beam search (https://arxiv.org/abs/2204.05424) # options for ranking generations (either beams or best-of-N samples) length_penalty: Optional[float] = None # "alpha" in Google NMT, None defaults to length norm # prompt, prefix, and token suppression prompt: Optional[Union[str, List[int]]] = None # text or tokens for the previous context prefix: Optional[Union[str, List[int]]] = None # text or tokens to prefix the current context suppress_blank: bool = True # this will suppress blank outputs # list of tokens ids (or comma-separated token ids) to suppress # "-1" will suppress a set of symbols as defined in `tokenizer.non_speech_tokens()` suppress_tokens: Optional[Union[str, Iterable[int]]] = "-1" # timestamp sampling options without_timestamps: bool = False # use <|notimestamps|> to sample text tokens only max_initial_timestamp: Optional[float] = 1.0 # the initial timestamp cannot be later than this # implementation details fp16: bool = True # use fp16 for most of the calculation @dataclass(frozen=True) class DecodingResult: audio_features: Tensor language: str language_probs: Optional[Dict[str, float]] = None tokens: List[int] = field(default_factory=list) text: str = "" avg_logprob: float = np.nan no_speech_prob: float = np.nan temperature: float = np.nan compression_ratio: float = np.nan class Inference: def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: """Perform a forward pass on the decoder and return per-token logits""" raise NotImplementedError def rearrange_kv_cache(self, source_indices) -> None: """Update the key-value cache according to the updated beams""" raise NotImplementedError def cleanup_caching(self) -> None: """Clean up any resources or hooks after decoding is finished""" pass class PyTorchInference(Inference): def __init__(self, model: "Whisper", initial_token_length: int): self.model: "Whisper" = model self.initial_token_length = initial_token_length self.kv_cache = {} self.hooks = [] def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: if not self.kv_cache: self.kv_cache, self.hooks = self.model.install_kv_cache_hooks() if tokens.shape[-1] > self.initial_token_length: # only need to use the last token except in the first forward pass tokens = tokens[:, -1:] return self.model.decoder(tokens, audio_features, kv_cache=self.kv_cache) def cleanup_caching(self): for hook in self.hooks: hook.remove() self.kv_cache = {} self.hooks = [] def rearrange_kv_cache(self, source_indices): for module, tensor in self.kv_cache.items(): # update the key/value cache to contain the selected sequences self.kv_cache[module] = tensor[source_indices].detach() class SequenceRanker: def rank(self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]]) -> List[int]: """ Given a list of groups of samples and their cumulative log probabilities, return the indices of the samples in each group to select as the final result """ raise NotImplementedError class MaximumLikelihoodRanker(SequenceRanker): """ Select the sample with the highest log probabilities, penalized using either a simple length normalization or Google NMT paper's length penalty """ def __init__(self, length_penalty: Optional[float]): self.length_penalty = length_penalty def rank(self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]]): def scores(logprobs, lengths): result = [] for logprob, length in zip(logprobs, lengths): if self.length_penalty is None: penalty = length else: # from the Google NMT paper penalty = ((5 + length) / 6) ** self.length_penalty result.append(logprob / penalty) return result # get the sequence with the highest score lengths = [[len(t) for t in s] for s in tokens] return [np.argmax(scores(p, l)) for p, l in zip(sum_logprobs, lengths)] class TokenDecoder: def reset(self): """Initialize any stateful variables for decoding a new sequence""" def update(self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor) -> Tuple[Tensor, bool]: """Specify how to select the next token, based on the current trace and logits Parameters ---------- tokens : Tensor, shape = (n_batch, current_sequence_length) all tokens in the context so far, including the prefix and sot_sequence tokens logits : Tensor, shape = (n_batch, vocab_size) per-token logits of the probability distribution at the current step sum_logprobs : Tensor, shape = (n_batch) cumulative log probabilities for each sequence Returns ------- tokens : Tensor, shape = (n_batch, current_sequence_length + 1) the tokens, appended with the selected next token completed : bool True if all sequences has reached the end of text """ raise NotImplementedError def finalize( self, tokens: Tensor, sum_logprobs: Tensor ) -> Tuple[Sequence[Sequence[Tensor]], List[List[float]]]: """Finalize search and return the final candidate sequences Parameters ---------- tokens : Tensor, shape = (n_audio, n_group, current_sequence_length) all tokens in the context so far, including the prefix and sot_sequence sum_logprobs : Tensor, shape = (n_audio, n_group) cumulative log probabilities for each sequence Returns ------- tokens : Sequence[Sequence[Tensor]], length = n_audio sequence of Tensors containing candidate token sequences, for each audio input sum_logprobs : List[List[float]], length = n_audio sequence of cumulative log probabilities corresponding to the above """ raise NotImplementedError class GreedyDecoder(TokenDecoder): def __init__(self, temperature: float, eot: int): self.temperature = temperature self.eot = eot def update(self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor) -> Tuple[Tensor, bool]: temperature = self.temperature if temperature == 0: next_tokens = logits.argmax(dim=-1) else: next_tokens = Categorical(logits=logits / temperature).sample() logprobs = F.log_softmax(logits.float(), dim=-1) current_logprobs = logprobs[torch.arange(logprobs.shape[0]), next_tokens] sum_logprobs += current_logprobs * (tokens[:, -1] != self.eot) next_tokens[tokens[:, -1] == self.eot] = self.eot tokens = torch.cat([tokens, next_tokens[:, None]], dim=-1) completed = (tokens[:, -1] == self.eot).all() return tokens, completed def finalize(self, tokens: Tensor, sum_logprobs: Tensor): # make sure each sequence has at least one EOT token at the end tokens = F.pad(tokens, (0, 1), value=self.eot) return tokens, sum_logprobs.tolist() class BeamSearchDecoder(TokenDecoder): def __init__(self, beam_size: int, eot: int, inference: Inference, patience: Optional[float] = None): self.beam_size = beam_size self.eot = eot self.inference = inference self.patience = patience or 1.0 self.max_candidates: int = round(beam_size * self.patience) self.finished_sequences = None assert self.max_candidates > 0, f"Invalid beam size ({beam_size}) or patience ({patience})" def reset(self): self.finished_sequences = None def update(self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor) -> Tuple[Tensor, bool]: if tokens.shape[0] % self.beam_size != 0: raise ValueError(f"{tokens.shape}[0] % {self.beam_size} != 0") n_audio = tokens.shape[0] // self.beam_size if self.finished_sequences is None: # for the first update self.finished_sequences = [{} for _ in range(n_audio)] logprobs = F.log_softmax(logits.float(), dim=-1) next_tokens, source_indices, finished_sequences = [], [], [] for i in range(n_audio): scores, sources, finished = {}, {}, {} # STEP 1: calculate the cumulative log probabilities for possible candidates for j in range(self.beam_size): idx = i * self.beam_size + j prefix = tokens[idx].tolist() for logprob, token in zip(*logprobs[idx].topk(self.beam_size + 1)): new_logprob = (sum_logprobs[idx] + logprob).item() sequence = tuple(prefix + [token.item()]) scores[sequence] = new_logprob sources[sequence] = idx # STEP 2: rank the candidates and keep the top beam_size sequences for each audio saved = 0 for sequence in sorted(scores, key=scores.get, reverse=True): if sequence[-1] == self.eot: finished[sequence] = scores[sequence] else: sum_logprobs[len(next_tokens)] = scores[sequence] next_tokens.append(sequence) source_indices.append(sources[sequence]) saved += 1 if saved == self.beam_size: break finished_sequences.append(finished) tokens = torch.tensor(next_tokens, device=tokens.device) self.inference.rearrange_kv_cache(source_indices) # add newly finished sequences to self.finished_sequences assert len(self.finished_sequences) == len(finished_sequences) for previously_finished, newly_finished in zip(self.finished_sequences, finished_sequences): for seq in sorted(newly_finished, key=newly_finished.get, reverse=True): if len(previously_finished) >= self.max_candidates: break # the candidate list is full previously_finished[seq] = newly_finished[seq] # mark as completed if all audio has enough number of samples completed = all( len(sequences) >= self.max_candidates for sequences in self.finished_sequences ) return tokens, completed def finalize(self, preceding_tokens: Tensor, sum_logprobs: Tensor): # collect all finished sequences, including patience, and add unfinished ones if not enough sum_logprobs = sum_logprobs.cpu() for i, sequences in enumerate(self.finished_sequences): if len(sequences) < self.beam_size: # when not enough sequences are finished for j in list(np.argsort(sum_logprobs[i]))[::-1]: sequence = preceding_tokens[i, j].tolist() + [self.eot] sequences[tuple(sequence)] = sum_logprobs[i][j].item() if len(sequences) >= self.beam_size: break tokens: List[List[Tensor]] = [ [torch.tensor(seq) for seq in sequences.keys()] for sequences in self.finished_sequences ] sum_logprobs: List[List[float]] = [ list(sequences.values()) for sequences in self.finished_sequences ] return tokens, sum_logprobs class LogitFilter: def apply(self, logits: Tensor, tokens: Tensor) -> None: """Apply any filtering or masking to logits in-place Parameters ---------- logits : Tensor, shape = (n_batch, vocab_size) per-token logits of the probability distribution at the current step tokens : Tensor, shape = (n_batch, current_sequence_length) all tokens in the context so far, including the prefix and sot_sequence tokens """ raise NotImplementedError class SuppressBlank(LogitFilter): def __init__(self, tokenizer: Tokenizer, sample_begin: int): self.tokenizer = tokenizer self.sample_begin = sample_begin def apply(self, logits: Tensor, tokens: Tensor): if tokens.shape[1] == self.sample_begin: logits[:, self.tokenizer.encode(" ") + [self.tokenizer.eot]] = -np.inf class SuppressTokens(LogitFilter): def __init__(self, suppress_tokens: Sequence[int]): self.suppress_tokens = list(suppress_tokens) def apply(self, logits: Tensor, tokens: Tensor): logits[:, self.suppress_tokens] = -np.inf class ApplyTimestampRules(LogitFilter): def __init__( self, tokenizer: Tokenizer, sample_begin: int, max_initial_timestamp_index: Optional[int] ): self.tokenizer = tokenizer self.sample_begin = sample_begin self.max_initial_timestamp_index = max_initial_timestamp_index def apply(self, logits: Tensor, tokens: Tensor): # suppress <|notimestamps|> which is handled by without_timestamps if self.tokenizer.no_timestamps is not None: logits[:, self.tokenizer.no_timestamps] = -np.inf # timestamps have to appear in pairs, except directly before EOT; mask logits accordingly for k in range(tokens.shape[0]): seq = [t for t in tokens[k, self.sample_begin :].tolist()] last_was_timestamp = len(seq) >= 1 and seq[-1] >= self.tokenizer.timestamp_begin penultimate_was_timestamp = len(seq) < 2 or seq[-2] >= self.tokenizer.timestamp_begin if last_was_timestamp: if penultimate_was_timestamp: # has to be non-timestamp logits[k, self.tokenizer.timestamp_begin :] = -np.inf else: # cannot be normal text tokens logits[k, : self.tokenizer.eot] = -np.inf if tokens.shape[1] == self.sample_begin: # suppress generating non-timestamp tokens at the beginning logits[:, : self.tokenizer.timestamp_begin] = -np.inf # apply the `max_initial_timestamp` option if self.max_initial_timestamp_index is not None: last_allowed = self.tokenizer.timestamp_begin + self.max_initial_timestamp_index logits[:, last_allowed + 1 :] = -np.inf # if sum of probability over timestamps is above any other token, sample timestamp logprobs = F.log_softmax(logits.float(), dim=-1) for k in range(tokens.shape[0]): timestamp_logprob = logprobs[k, self.tokenizer.timestamp_begin :].logsumexp(dim=-1) max_text_token_logprob = logprobs[k, : self.tokenizer.timestamp_begin].max() if timestamp_logprob > max_text_token_logprob: logits[k, : self.tokenizer.timestamp_begin] = -np.inf class DecodingTask: inference: Inference sequence_ranker: SequenceRanker decoder: TokenDecoder logit_filters: List[LogitFilter] def __init__(self, model: "Whisper", options: DecodingOptions): self.model = model language = options.language or "en" tokenizer = get_tokenizer(model.is_multilingual, language=language, task=options.task) self.tokenizer: Tokenizer = tokenizer self.options: DecodingOptions = self._verify_options(options) self.n_group: int = options.beam_size or options.best_of or 1 self.n_ctx: int = model.dims.n_text_ctx self.sample_len: int = options.sample_len or model.dims.n_text_ctx // 2 self.sot_sequence: Tuple[int] = tokenizer.sot_sequence if self.options.without_timestamps: self.sot_sequence = tokenizer.sot_sequence_including_notimestamps self.initial_tokens: Tuple[int] = self._get_initial_tokens() self.sample_begin: int = len(self.initial_tokens) self.sot_index: int = self.initial_tokens.index(tokenizer.sot) # inference: implements the forward pass through the decoder, including kv caching self.inference = PyTorchInference(model, len(self.initial_tokens)) # sequence ranker: implements how to rank a group of sampled sequences self.sequence_ranker = MaximumLikelihoodRanker(options.length_penalty) # decoder: implements how to select the next tokens, given the autoregressive distribution if options.beam_size is not None: self.decoder = BeamSearchDecoder( options.beam_size, tokenizer.eot, self.inference, options.patience ) else: self.decoder = GreedyDecoder(options.temperature, tokenizer.eot) # logit filters: applies various rules to suppress or penalize certain tokens self.logit_filters = [] if self.options.suppress_blank: self.logit_filters.append(SuppressBlank(self.tokenizer, self.sample_begin)) if self.options.suppress_tokens: self.logit_filters.append(SuppressTokens(self._get_suppress_tokens())) if not options.without_timestamps: precision = CHUNK_LENGTH / model.dims.n_audio_ctx # usually 0.02 seconds max_initial_timestamp_index = None if options.max_initial_timestamp: max_initial_timestamp_index = round(self.options.max_initial_timestamp / precision) self.logit_filters.append( ApplyTimestampRules(tokenizer, self.sample_begin, max_initial_timestamp_index) ) def _verify_options(self, options: DecodingOptions) -> DecodingOptions: if options.beam_size is not None and options.best_of is not None: raise ValueError("beam_size and best_of can't be given together") if options.temperature == 0: if options.best_of is not None: raise ValueError("best_of with greedy sampling (T=0) is not compatible") if options.patience is not None and options.beam_size is None: raise ValueError("patience requires beam_size to be given") if options.length_penalty is not None and not (0 <= options.length_penalty <= 1): raise ValueError("length_penalty (alpha) should be a value between 0 and 1") return options def _get_initial_tokens(self) -> Tuple[int]: tokens = list(self.sot_sequence) prefix = self.options.prefix prompt = self.options.prompt if prefix: prefix_tokens = ( self.tokenizer.encode(" " + prefix.strip()) if isinstance(prefix, str) else prefix ) if self.sample_len is not None: max_prefix_len = self.n_ctx // 2 - self.sample_len prefix_tokens = prefix_tokens[-max_prefix_len:] tokens = tokens + prefix_tokens if prompt: prompt_tokens = ( self.tokenizer.encode(" " + prompt.strip()) if isinstance(prompt, str) else prompt ) tokens = [self.tokenizer.sot_prev] + prompt_tokens[-(self.n_ctx // 2 - 1) :] + tokens return tuple(tokens) def _get_suppress_tokens(self) -> Tuple[int]: suppress_tokens = self.options.suppress_tokens if isinstance(suppress_tokens, str): suppress_tokens = [int(t) for t in suppress_tokens.split(",")] if -1 in suppress_tokens: suppress_tokens = [t for t in suppress_tokens if t >= 0] suppress_tokens.extend(self.tokenizer.non_speech_tokens) elif suppress_tokens is None or len(suppress_tokens) == 0: suppress_tokens = [] # interpret empty string as an empty list else: assert isinstance(suppress_tokens, list), "suppress_tokens must be a list" suppress_tokens.extend( [self.tokenizer.sot, self.tokenizer.sot_prev, self.tokenizer.sot_lm] ) if self.tokenizer.no_speech is not None: # no-speech probability is collected separately suppress_tokens.append(self.tokenizer.no_speech) return tuple(sorted(set(suppress_tokens))) def _get_audio_features(self, mel: Tensor): if self.options.fp16: mel = mel.half() if mel.shape[-2:] == (self.model.dims.n_audio_ctx, self.model.dims.n_audio_state): # encoded audio features are given; skip audio encoding print("encoded audio features are given; skip audio encoding") audio_features = mel else: print(mel.shape) print("===============================") audio_features = self.model.encoder(mel) if audio_features.dtype != (torch.float16 if self.options.fp16 else torch.float32): return TypeError(f"audio_features has an incorrect dtype: {audio_features.dtype}") return audio_features def _detect_language(self, audio_features: Tensor, tokens: Tensor): languages = [self.options.language] * audio_features.shape[0] lang_probs = None if self.options.language is None or self.options.task == "lang_id": lang_tokens, lang_probs = self.model.detect_language(audio_features, self.tokenizer) languages = [max(probs, key=probs.get) for probs in lang_probs] if self.options.language is None: tokens[:, self.sot_index + 1] = lang_tokens # write language tokens return languages, lang_probs def _main_loop(self, audio_features: Tensor, tokens: Tensor): assert audio_features.shape[0] == tokens.shape[0] n_batch = tokens.shape[0] sum_logprobs: Tensor = torch.zeros(n_batch, device=audio_features.device) no_speech_probs = [np.nan] * n_batch try: for i in range(self.sample_len): logits = self.inference.logits(tokens, audio_features) if i == 0 and self.tokenizer.no_speech is not None: # save no_speech_probs probs_at_sot = logits[:, self.sot_index].float().softmax(dim=-1) no_speech_probs = probs_at_sot[:, self.tokenizer.no_speech].tolist() # now we need to consider the logits at the last token only logits = logits[:, -1] # apply the logit filters, e.g. for suppressing or applying penalty to for logit_filter in self.logit_filters: logit_filter.apply(logits, tokens) # expand the tokens tensor with the selected next tokens tokens, completed = self.decoder.update(tokens, logits, sum_logprobs) if completed or tokens.shape[-1] > self.n_ctx: break finally: self.inference.cleanup_caching() return tokens, sum_logprobs, no_speech_probs @torch.no_grad() def run(self, mel: Tensor) -> List[DecodingResult]: self.decoder.reset() tokenizer: Tokenizer = self.tokenizer n_audio: int = mel.shape[0] audio_features: Tensor = self._get_audio_features(mel) # encoder forward pass tokens: Tensor = torch.tensor([self.initial_tokens]).repeat(n_audio, 1) # detect language if requested, overwriting the language token languages, language_probs = self._detect_language(audio_features, tokens) if self.options.task == "lang_id": return [ DecodingResult(audio_features=features, language=language, language_probs=probs) for features, language, probs in zip(audio_features, languages, language_probs) ] # repeat the audio & text tensors by the group size, for beam search or best-of-n sampling audio_features = audio_features.repeat_interleave(self.n_group, dim=0) tokens = tokens.repeat_interleave(self.n_group, dim=0).to(audio_features.device) # call the main sampling loop tokens, sum_logprobs, no_speech_probs = self._main_loop(audio_features, tokens) # reshape the tensors to have (n_audio, n_group) as the first two dimensions audio_features = audio_features[:: self.n_group] no_speech_probs = no_speech_probs[:: self.n_group] assert audio_features.shape[0] == len(no_speech_probs) == n_audio tokens = tokens.reshape(n_audio, self.n_group, -1) sum_logprobs = sum_logprobs.reshape(n_audio, self.n_group) # get the final candidates for each group, and slice between the first sampled token and EOT tokens, sum_logprobs = self.decoder.finalize(tokens, sum_logprobs) tokens: List[List[Tensor]] = [ [t[self.sample_begin : (t == tokenizer.eot).nonzero()[0, 0]] for t in s] for s in tokens ] # select the top-ranked sample in each group selected = self.sequence_ranker.rank(tokens, sum_logprobs) tokens: List[List[int]] = [t[i].tolist() for i, t in zip(selected, tokens)] texts: List[str] = [tokenizer.decode(t).strip() for t in tokens] sum_logprobs: List[float] = [lp[i] for i, lp in zip(selected, sum_logprobs)] avg_logprobs: List[float] = [lp / (len(t) + 1) for t, lp in zip(tokens, sum_logprobs)] fields = (texts, languages, tokens, audio_features, avg_logprobs, no_speech_probs) if len(set(map(len, fields))) != 1: raise RuntimeError(f"inconsistent result lengths: {list(map(len, fields))}") return [ DecodingResult( audio_features=features, language=language, tokens=tokens, text=text, avg_logprob=avg_logprob, no_speech_prob=no_speech_prob, temperature=self.options.temperature, compression_ratio=compression_ratio(text), ) for text, language, tokens, features, avg_logprob, no_speech_prob in zip(*fields) ] @torch.no_grad() def decode(model: "Whisper", mel: Tensor, options: DecodingOptions = DecodingOptions()) -> Union[DecodingResult, List[DecodingResult]]: """ Performs decoding of 30-second audio segment(s), provided as Mel spectrogram(s). Parameters ---------- model: Whisper the Whisper model instance mel: torch.Tensor, shape = (80, 3000) or (*, 80, 3000) A tensor containing the Mel spectrogram(s) options: DecodingOptions A dataclass that contains all necessary options for decoding 30-second segments Returns ------- result: Union[DecodingResult, List[DecodingResult]] The result(s) of decoding contained in `DecodingResult` dataclass instance(s) """ single = mel.ndim == 2 if single: mel = mel.unsqueeze(0) result = DecodingTask(model, options).run(mel) if single: result = result[0] return result so-vits-svc-npu/vencoder/whisper/model.py000066400000000000000000000227441456161405400210320ustar00rootroot00000000000000from dataclasses import dataclass from typing import Dict, Iterable, Optional import numpy as np import mindtorch.torch as torch import mindtorch.torch.nn.functional as F from mindtorch.torch import Tensor, nn from .decoding import decode as decode_function from .decoding import detect_language as detect_language_function @dataclass class ModelDimensions: n_mels: int n_audio_ctx: int n_audio_state: int n_audio_head: int n_audio_layer: int n_vocab: int n_text_ctx: int n_text_state: int n_text_head: int n_text_layer: int class LayerNorm(nn.LayerNorm): def forward(self, x: Tensor) -> Tensor: return super().forward(x.float()).type(x.dtype) class Linear(nn.Linear): def forward(self, x: Tensor) -> Tensor: return F.linear( x, self.weight.to(x.dtype), None if self.bias is None else self.bias.to(x.dtype) ) class Conv1d(nn.Conv1d): def _conv_forward(self, x: Tensor, weight: Tensor, bias: Optional[Tensor]) -> Tensor: return super()._conv_forward( x, weight.to(x.dtype), None if bias is None else bias.to(x.dtype) ) def sinusoids(length, channels, max_timescale=10000): """Returns sinusoids for positional embedding""" assert channels % 2 == 0 log_timescale_increment = np.log(max_timescale) / (channels // 2 - 1) inv_timescales = torch.exp(-log_timescale_increment * torch.arange(channels // 2)) scaled_time = torch.arange(length)[:, np.newaxis] * inv_timescales[np.newaxis, :] return torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1) class MultiHeadAttention(nn.Module): def __init__(self, n_state: int, n_head: int): super().__init__() self.n_head = n_head self.query = Linear(n_state, n_state) self.key = Linear(n_state, n_state, bias=False) self.value = Linear(n_state, n_state) self.out = Linear(n_state, n_state) def forward( self, x: Tensor, xa: Optional[Tensor] = None, mask: Optional[Tensor] = None, kv_cache: Optional[dict] = None, ): q = self.query(x) if kv_cache is None or xa is None or self.key not in kv_cache: # hooks, if installed (i.e. kv_cache is not None), will prepend the cached kv tensors; # otherwise, perform key/value projections for self- or cross-attention as usual. k = self.key(x if xa is None else xa) v = self.value(x if xa is None else xa) else: # for cross-attention, calculate keys and values once and reuse in subsequent calls. k = kv_cache[self.key] v = kv_cache[self.value] wv, qk = self.qkv_attention(q, k, v, mask) return self.out(wv), qk def qkv_attention(self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None): n_batch, n_ctx, n_state = q.shape scale = (n_state // self.n_head) ** -0.25 q = q.view(*q.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) * scale k = k.view(*k.shape[:2], self.n_head, -1).permute(0, 2, 3, 1) * scale v = v.view(*v.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) qk = q @ k if mask is not None: qk = qk + mask[:n_ctx, :n_ctx] qk = qk.float() w = F.softmax(qk, dim=-1).to(q.dtype) return (w @ v).permute(0, 2, 1, 3).flatten(start_dim=2), qk.detach() class ResidualAttentionBlock(nn.Module): def __init__(self, n_state: int, n_head: int, cross_attention: bool = False): super().__init__() self.attn = MultiHeadAttention(n_state, n_head) self.attn_ln = LayerNorm(n_state) self.cross_attn = MultiHeadAttention(n_state, n_head) if cross_attention else None self.cross_attn_ln = LayerNorm(n_state) if cross_attention else None n_mlp = n_state * 4 self.mlp = nn.Sequential(Linear(n_state, n_mlp), nn.GELU(), Linear(n_mlp, n_state)) self.mlp_ln = LayerNorm(n_state) def forward( self, x: Tensor, xa: Optional[Tensor] = None, mask: Optional[Tensor] = None, kv_cache: Optional[dict] = None, ): x = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)[0] if self.cross_attn: x = x + self.cross_attn(self.cross_attn_ln(x), xa, kv_cache=kv_cache)[0] x = x + self.mlp(self.mlp_ln(x)) return x class AudioEncoder(nn.Module): def __init__(self, n_mels: int, n_ctx: int, n_state: int, n_head: int, n_layer: int): super().__init__() self.conv1 = Conv1d(n_mels, n_state, kernel_size=3, padding=1) self.conv2 = Conv1d(n_state, n_state, kernel_size=3, stride=2, padding=1) self.register_buffer("positional_embedding", sinusoids(n_ctx, n_state)) self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( [ResidualAttentionBlock(n_state, n_head) for _ in range(n_layer)] ) self.ln_post = LayerNorm(n_state) def forward(self, x: Tensor): """ x : torch.Tensor, shape = (batch_size, n_mels, n_ctx) the mel spectrogram of the audio """ x = F.gelu(self.conv1(x)) x = F.gelu(self.conv2(x)) x = x.permute(0, 2, 1) len_x = x.shape[1] len_e = self.positional_embedding.shape[0] assert len_x <= len_e, "incorrect audio shape" pos_e = self.positional_embedding[:len_x, :] x = (x + pos_e).to(x.dtype) for block in self.blocks: x = block(x) x = self.ln_post(x) return x class TextDecoder(nn.Module): def __init__(self, n_vocab: int, n_ctx: int, n_state: int, n_head: int, n_layer: int): super().__init__() self.token_embedding = nn.Embedding(n_vocab, n_state) self.positional_embedding = nn.Parameter(torch.empty(n_ctx, n_state)) self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( [ResidualAttentionBlock(n_state, n_head, cross_attention=True) for _ in range(n_layer)] ) self.ln = LayerNorm(n_state) mask = torch.empty(n_ctx, n_ctx).fill_(-np.inf).triu_(1) self.register_buffer("mask", mask, persistent=False) def forward(self, x: Tensor, xa: Tensor, kv_cache: Optional[dict] = None): """ x : torch.LongTensor, shape = (batch_size, <= n_ctx) the text tokens xa : torch.Tensor, shape = (batch_size, n_mels, n_audio_ctx) the encoded audio features to be attended on """ offset = next(iter(kv_cache.values())).shape[1] if kv_cache else 0 x = self.token_embedding(x) + self.positional_embedding[offset : offset + x.shape[-1]] x = x.to(xa.dtype) for block in self.blocks: x = block(x, xa, mask=self.mask, kv_cache=kv_cache) x = self.ln(x) logits = (x @ torch.transpose(self.token_embedding.weight.to(x.dtype), 0, 1)).float() return logits class Whisper(nn.Module): def __init__(self, dims: ModelDimensions): super().__init__() self.dims = dims self.encoder = AudioEncoder( self.dims.n_mels, self.dims.n_audio_ctx, self.dims.n_audio_state, self.dims.n_audio_head, self.dims.n_audio_layer, ) self.decoder = TextDecoder( self.dims.n_vocab, self.dims.n_text_ctx, self.dims.n_text_state, self.dims.n_text_head, self.dims.n_text_layer, ) def embed_audio(self, mel: torch.Tensor): return self.encoder(mel) def logits(self, tokens: torch.Tensor, audio_features: torch.Tensor): return self.decoder(tokens, audio_features) def forward(self, mel: torch.Tensor, tokens: torch.Tensor) -> Dict[str, torch.Tensor]: return self.decoder(tokens, self.encoder(mel)) @property def device(self): return next(self.parameters()).device @property def is_multilingual(self): return self.dims.n_vocab == 51865 def install_kv_cache_hooks(self, cache: Optional[dict] = None): """ The `MultiHeadAttention` module optionally accepts `kv_cache` which stores the key and value tensors calculated for the previous positions. This method returns a dictionary that stores all caches, and the necessary hooks for the key and value projection modules that save the intermediate tensors to be reused during later calculations. Returns ------- cache : Dict[nn.Module, torch.Tensor] A dictionary object mapping the key/value projection modules to its cache hooks : List[RemovableHandle] List of PyTorch RemovableHandle objects to stop the hooks to be called """ cache = {**cache} if cache is not None else {} hooks = [] def save_to_cache(module, _, output): if module not in cache or output.shape[1] > self.decoder.positional_embedding.shape[0]: cache[module] = output # save as-is, for the first token or cross attention else: cache[module] = torch.cat([cache[module], output], dim=1).detach() return cache[module] def install_hooks(layer: nn.Module): if isinstance(layer, MultiHeadAttention): hooks.append(layer.key.register_forward_hook(save_to_cache)) hooks.append(layer.value.register_forward_hook(save_to_cache)) self.decoder.apply(install_hooks) return cache, hooks detect_language = detect_language_function decode = decode_function so-vits-svc-npu/vencoder/whisper/tokenizer.py000066400000000000000000000231771456161405400217450ustar00rootroot00000000000000import os from dataclasses import dataclass from functools import lru_cache from typing import List, Optional, Tuple, Union import numpy as np import mindtorch.torch as torch from transformers import GPT2TokenizerFast LANGUAGES = { "en": "english", "zh": "chinese", "de": "german", "es": "spanish", "ru": "russian", "ko": "korean", "fr": "french", "ja": "japanese", "pt": "portuguese", "tr": "turkish", "pl": "polish", "ca": "catalan", "nl": "dutch", "ar": "arabic", "sv": "swedish", "it": "italian", "id": "indonesian", "hi": "hindi", "fi": "finnish", "vi": "vietnamese", "he": "hebrew", "uk": "ukrainian", "el": "greek", "ms": "malay", "cs": "czech", "ro": "romanian", "da": "danish", "hu": "hungarian", "ta": "tamil", "no": "norwegian", "th": "thai", "ur": "urdu", "hr": "croatian", "bg": "bulgarian", "lt": "lithuanian", "la": "latin", "mi": "maori", "ml": "malayalam", "cy": "welsh", "sk": "slovak", "te": "telugu", "fa": "persian", "lv": "latvian", "bn": "bengali", "sr": "serbian", "az": "azerbaijani", "sl": "slovenian", "kn": "kannada", "et": "estonian", "mk": "macedonian", "br": "breton", "eu": "basque", "is": "icelandic", "hy": "armenian", "ne": "nepali", "mn": "mongolian", "bs": "bosnian", "kk": "kazakh", "sq": "albanian", "sw": "swahili", "gl": "galician", "mr": "marathi", "pa": "punjabi", "si": "sinhala", "km": "khmer", "sn": "shona", "yo": "yoruba", "so": "somali", "af": "afrikaans", "oc": "occitan", "ka": "georgian", "be": "belarusian", "tg": "tajik", "sd": "sindhi", "gu": "gujarati", "am": "amharic", "yi": "yiddish", "lo": "lao", "uz": "uzbek", "fo": "faroese", "ht": "haitian creole", "ps": "pashto", "tk": "turkmen", "nn": "nynorsk", "mt": "maltese", "sa": "sanskrit", "lb": "luxembourgish", "my": "myanmar", "bo": "tibetan", "tl": "tagalog", "mg": "malagasy", "as": "assamese", "tt": "tatar", "haw": "hawaiian", "ln": "lingala", "ha": "hausa", "ba": "bashkir", "jw": "javanese", "su": "sundanese", } # language code lookup by name, with a few language aliases TO_LANGUAGE_CODE = { **{language: code for code, language in LANGUAGES.items()}, "burmese": "my", "valencian": "ca", "flemish": "nl", "haitian": "ht", "letzeburgesch": "lb", "pushto": "ps", "panjabi": "pa", "moldavian": "ro", "moldovan": "ro", "sinhalese": "si", "castilian": "es", } @dataclass(frozen=True) class Tokenizer: """A thin wrapper around `GPT2TokenizerFast` providing quick access to special tokens""" tokenizer: "GPT2TokenizerFast" language: Optional[str] sot_sequence: Tuple[int] def encode(self, text, **kwargs): return self.tokenizer.encode(text, **kwargs) def decode(self, token_ids: Union[int, List[int], np.ndarray, torch.Tensor], **kwargs): return self.tokenizer.decode(token_ids, **kwargs) def decode_with_timestamps(self, tokens) -> str: """ Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". """ outputs = [[]] for token in tokens: if token >= self.timestamp_begin: timestamp = f"<|{(token - self.timestamp_begin) * 0.02:.2f}|>" outputs.append(timestamp) outputs.append([]) else: outputs[-1].append(token) outputs = [s if isinstance(s, str) else self.tokenizer.decode(s) for s in outputs] return "".join(outputs) @property @lru_cache() def eot(self) -> int: return self.tokenizer.eos_token_id @property @lru_cache() def sot(self) -> int: return self._get_single_token_id("<|startoftranscript|>") @property @lru_cache() def sot_lm(self) -> int: return self._get_single_token_id("<|startoflm|>") @property @lru_cache() def sot_prev(self) -> int: return self._get_single_token_id("<|startofprev|>") @property @lru_cache() def no_speech(self) -> int: return self._get_single_token_id("<|nospeech|>") @property @lru_cache() def no_timestamps(self) -> int: return self._get_single_token_id("<|notimestamps|>") @property @lru_cache() def timestamp_begin(self) -> int: return self.tokenizer.all_special_ids[-1] + 1 @property @lru_cache() def language_token(self) -> int: """Returns the token id corresponding to the value of the `language` field""" if self.language is None: raise ValueError("This tokenizer does not have language token configured") additional_tokens = dict( zip( self.tokenizer.additional_special_tokens, self.tokenizer.additional_special_tokens_ids, ) ) candidate = f"<|{self.language}|>" if candidate in additional_tokens: return additional_tokens[candidate] raise KeyError(f"Language {self.language} not found in tokenizer.") @property @lru_cache() def all_language_tokens(self) -> Tuple[int]: result = [] for token, token_id in zip( self.tokenizer.additional_special_tokens, self.tokenizer.additional_special_tokens_ids, ): if token.strip("<|>") in LANGUAGES: result.append(token_id) return tuple(result) @property @lru_cache() def all_language_codes(self) -> Tuple[str]: return tuple(self.decode([l]).strip("<|>") for l in self.all_language_tokens) @property @lru_cache() def sot_sequence_including_notimestamps(self) -> Tuple[int]: return tuple(list(self.sot_sequence) + [self.no_timestamps]) @property @lru_cache() def non_speech_tokens(self) -> Tuple[int]: """ Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech annotations, to prevent sampling texts that are not actually spoken in the audio, e.g. - ♪♪♪ - ( SPEAKING FOREIGN LANGUAGE ) - [DAVID] Hey there, keeping basic punctuations like commas, periods, question marks, exclamation points, etc. """ symbols = list("\"#()*+/:;<=>@[\\]^_`{|}~「」『』") symbols += "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split() # symbols that may be a single token or multiple tokens depending on the tokenizer. # In case they're multiple tokens, suppress the first token, which is safe because: # These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress # in generations, and in the 3-byte UTF-8 representation they share the first two bytes. miscellaneous = set("♩♪♫♬♭♮♯") assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous) # allow hyphens "-" and single quotes "'" between words, but not at the beginning of a word result = {self.tokenizer.encode(" -")[0], self.tokenizer.encode(" '")[0]} for symbol in symbols + list(miscellaneous): for tokens in [self.tokenizer.encode(symbol), self.tokenizer.encode(" " + symbol)]: if len(tokens) == 1 or symbol in miscellaneous: result.add(tokens[0]) return tuple(sorted(result)) def _get_single_token_id(self, text) -> int: tokens = self.tokenizer.encode(text) assert len(tokens) == 1, f"{text} is not encoded as a single token" return tokens[0] @lru_cache(maxsize=None) def build_tokenizer(name: str = "gpt2"): os.environ["TOKENIZERS_PARALLELISM"] = "false" path = os.path.join(os.path.dirname(__file__), "assets", name) tokenizer = GPT2TokenizerFast.from_pretrained(path) specials = [ "<|startoftranscript|>", *[f"<|{lang}|>" for lang in LANGUAGES.keys()], "<|translate|>", "<|transcribe|>", "<|startoflm|>", "<|startofprev|>", "<|nospeech|>", "<|notimestamps|>", ] tokenizer.add_special_tokens(dict(additional_special_tokens=specials)) return tokenizer @lru_cache(maxsize=None) def get_tokenizer( multilingual: bool, *, task: Optional[str] = None, # Literal["transcribe", "translate", None] language: Optional[str] = None, ) -> Tokenizer: if language is not None: language = language.lower() if language not in LANGUAGES: if language in TO_LANGUAGE_CODE: language = TO_LANGUAGE_CODE[language] else: raise ValueError(f"Unsupported language: {language}") if multilingual: tokenizer_name = "multilingual" task = task or "transcribe" language = language or "en" else: tokenizer_name = "gpt2" task = None language = None tokenizer = build_tokenizer(name=tokenizer_name) all_special_ids: List[int] = tokenizer.all_special_ids sot: int = all_special_ids[1] translate: int = all_special_ids[-6] transcribe: int = all_special_ids[-5] langs = tuple(LANGUAGES.keys()) sot_sequence = [sot] if language is not None: sot_sequence.append(sot + 1 + langs.index(language)) if task is not None: sot_sequence.append(transcribe if task == "transcribe" else translate) return Tokenizer(tokenizer=tokenizer, language=language, sot_sequence=tuple(sot_sequence)) so-vits-svc-npu/vencoder/whisper/utils.py000066400000000000000000000121521456161405400210620ustar00rootroot00000000000000import json import os import sys import zlib from typing import Callable, TextIO system_encoding = sys.getdefaultencoding() if system_encoding != "utf-8": def make_safe(string): # replaces any character not representable using the system default encoding with an '?', # avoiding UnicodeEncodeError (https://github.com/openai/whisper/discussions/729). return string.encode(system_encoding, errors="replace").decode(system_encoding) else: def make_safe(string): # utf-8 can encode any Unicode code point, so no need to do the round-trip encoding return string def exact_div(x, y): assert x % y == 0 return x // y def str2bool(string): str2val = {"True": True, "False": False} if string in str2val: return str2val[string] else: raise ValueError(f"Expected one of {set(str2val.keys())}, got {string}") def optional_int(string): return None if string == "None" else int(string) def optional_float(string): return None if string == "None" else float(string) def compression_ratio(text) -> float: text_bytes = text.encode("utf-8") return len(text_bytes) / len(zlib.compress(text_bytes)) def format_timestamp(seconds: float, always_include_hours: bool = False, decimal_marker: str = '.'): assert seconds >= 0, "non-negative timestamp expected" milliseconds = round(seconds * 1000.0) hours = milliseconds // 3_600_000 milliseconds -= hours * 3_600_000 minutes = milliseconds // 60_000 milliseconds -= minutes * 60_000 seconds = milliseconds // 1_000 milliseconds -= seconds * 1_000 hours_marker = f"{hours:02d}:" if always_include_hours or hours > 0 else "" return f"{hours_marker}{minutes:02d}:{seconds:02d}{decimal_marker}{milliseconds:03d}" class ResultWriter: extension: str def __init__(self, output_dir: str): self.output_dir = output_dir def __call__(self, result: dict, audio_path: str): audio_basename = os.path.basename(audio_path) output_path = os.path.join(self.output_dir, audio_basename + "." + self.extension) with open(output_path, "w", encoding="utf-8") as f: self.write_result(result, file=f) def write_result(self, result: dict, file: TextIO): raise NotImplementedError class WriteTXT(ResultWriter): extension: str = "txt" def write_result(self, result: dict, file: TextIO): for segment in result["segments"]: print(segment['text'].strip(), file=file, flush=True) class WriteVTT(ResultWriter): extension: str = "vtt" def write_result(self, result: dict, file: TextIO): print("WEBVTT\n", file=file) for segment in result["segments"]: print( f"{format_timestamp(segment['start'])} --> {format_timestamp(segment['end'])}\n" f"{segment['text'].strip().replace('-->', '->')}\n", file=file, flush=True, ) class WriteSRT(ResultWriter): extension: str = "srt" def write_result(self, result: dict, file: TextIO): for i, segment in enumerate(result["segments"], start=1): # write srt lines print( f"{i}\n" f"{format_timestamp(segment['start'], always_include_hours=True, decimal_marker=',')} --> " f"{format_timestamp(segment['end'], always_include_hours=True, decimal_marker=',')}\n" f"{segment['text'].strip().replace('-->', '->')}\n", file=file, flush=True, ) class WriteTSV(ResultWriter): """ Write a transcript to a file in TSV (tab-separated values) format containing lines like: \t\t Using integer milliseconds as start and end times means there's no chance of interference from an environment setting a language encoding that causes the decimal in a floating point number to appear as a comma; also is faster and more efficient to parse & store, e.g., in C++. """ extension: str = "tsv" def write_result(self, result: dict, file: TextIO): print("start", "end", "text", sep="\t", file=file) for segment in result["segments"]: print(round(1000 * segment['start']), file=file, end="\t") print(round(1000 * segment['end']), file=file, end="\t") print(segment['text'].strip().replace("\t", " "), file=file, flush=True) class WriteJSON(ResultWriter): extension: str = "json" def write_result(self, result: dict, file: TextIO): json.dump(result, file) def get_writer(output_format: str, output_dir: str) -> Callable[[dict, TextIO], None]: writers = { "txt": WriteTXT, "vtt": WriteVTT, "srt": WriteSRT, "tsv": WriteTSV, "json": WriteJSON, } if output_format == "all": all_writers = [writer(output_dir) for writer in writers.values()] def write_all(result: dict, file: TextIO): for writer in all_writers: writer(result, file) return write_all return writers[output_format](output_dir) so-vits-svc-npu/wav_upload.py000066400000000000000000000016261456161405400166010ustar00rootroot00000000000000import argparse import os import shutil from google.colab import files if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--type", type=str, required=True, help="type of file to upload") args = parser.parse_args() file_type = args.type basepath = os.getcwd() uploaded = files.upload() # 上传文件 assert(file_type in ['zip', 'audio']) if file_type == "zip": upload_path = "./upload/" for filename in uploaded.keys(): #将上传的文件移动到指定的位置上 shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, "userzip.zip")) elif file_type == "audio": upload_path = "./raw/" for filename in uploaded.keys(): #将上传的文件移动到指定的位置上 shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, filename))so-vits-svc-npu/webUI.py000066400000000000000000000574611456161405400154630ustar00rootroot00000000000000import glob import json import logging import os import re import subprocess import sys import time import traceback from itertools import chain from pathlib import Path import gradio as gr import librosa import numpy as np import soundfile import mindtorch.torch as torch from compress_model import removeOptimizer from edgetts.tts_voices import SUPPORTED_LANGUAGES from inference.infer_tool import Svc from utils import mix_model logging.getLogger('numba').setLevel(logging.WARNING) logging.getLogger('markdown_it').setLevel(logging.WARNING) logging.getLogger('urllib3').setLevel(logging.WARNING) logging.getLogger('matplotlib').setLevel(logging.WARNING) logging.getLogger('multipart').setLevel(logging.WARNING) model = None spk = None debug = False local_model_root = './trained' cuda = {} if torch.cuda.is_available(): for i in range(torch.cuda.device_count()): device_name = torch.cuda.get_device_properties(i).name cuda[f"CUDA:{i} {device_name}"] = f"cuda:{i}" def upload_mix_append_file(files,sfiles): try: if(sfiles is None): file_paths = [file.name for file in files] else: file_paths = [file.name for file in chain(files,sfiles)] p = {file:100 for file in file_paths} return file_paths,mix_model_output1.update(value=json.dumps(p,indent=2)) except Exception as e: if debug: traceback.print_exc() raise gr.Error(e) def mix_submit_click(js,mode): try: assert js.lstrip()!="" modes = {"凸组合":0, "线性组合":1} mode = modes[mode] data = json.loads(js) data = list(data.items()) model_path,mix_rate = zip(*data) path = mix_model(model_path,mix_rate,mode) return f"成功，文件被保存在了{path}" except Exception as e: if debug: traceback.print_exc() raise gr.Error(e) def updata_mix_info(files): try: if files is None : return mix_model_output1.update(value="") p = {file.name:100 for file in files} return mix_model_output1.update(value=json.dumps(p,indent=2)) except Exception as e: if debug: traceback.print_exc() raise gr.Error(e) def modelAnalysis(model_path,config_path,cluster_model_path,device,enhance,diff_model_path,diff_config_path,only_diffusion,use_spk_mix,local_model_enabled,local_model_selection): global model try: device = cuda[device] if "CUDA" in device else device cluster_filepath = os.path.split(cluster_model_path.name) if cluster_model_path is not None else "no_cluster" # get model and config path if (local_model_enabled): # local path model_path = glob.glob(os.path.join(local_model_selection, '*.pth'))[0] config_path = glob.glob(os.path.join(local_model_selection, '*.json'))[0] else: # upload from webpage model_path = model_path.name config_path = config_path.name fr = ".pkl" in cluster_filepath[1] model = Svc(model_path, config_path, device=device if device != "Auto" else None, cluster_model_path = cluster_model_path.name if cluster_model_path is not None else "", nsf_hifigan_enhance=enhance, diffusion_model_path = diff_model_path.name if diff_model_path is not None else "", diffusion_config_path = diff_config_path.name if diff_config_path is not None else "", shallow_diffusion = True if diff_model_path is not None else False, only_diffusion = only_diffusion, spk_mix_enable = use_spk_mix, feature_retrieval = fr ) spks = list(model.spk2id.keys()) device_name = torch.cuda.get_device_properties(model.dev).name if "cuda" in str(model.dev) else str(model.dev) msg = f"成功加载模型到设备{device_name}上\n" if cluster_model_path is None: msg += "未加载聚类模型或特征检索模型\n" elif fr: msg += f"特征检索模型{cluster_filepath[1]}加载成功\n" else: msg += f"聚类模型{cluster_filepath[1]}加载成功\n" if diff_model_path is None: msg += "未加载扩散模型\n" else: msg += f"扩散模型{diff_model_path.name}加载成功\n" msg += "当前模型的可用音色：\n" for i in spks: msg += i + " " return sid.update(choices = spks,value=spks[0]), msg except Exception as e: if debug: traceback.print_exc() raise gr.Error(e) def modelUnload(): global model if model is None: return sid.update(choices = [],value=""),"没有模型需要卸载!" else: model.unload_model() model = None torch.cuda.empty_cache() return sid.update(choices = [],value=""),"模型卸载完毕!" def vc_infer(output_format, sid, audio_path, truncated_basename, vc_transform, auto_f0, cluster_ratio, slice_db, noise_scale, pad_seconds, cl_num, lg_num, lgr_num, f0_predictor, enhancer_adaptive_key, cr_threshold, k_step, use_spk_mix, second_encoding, loudness_envelope_adjustment): global model _audio = model.slice_inference( audio_path, sid, vc_transform, slice_db, cluster_ratio, auto_f0, noise_scale, pad_seconds, cl_num, lg_num, lgr_num, f0_predictor, enhancer_adaptive_key, cr_threshold, k_step, use_spk_mix, second_encoding, loudness_envelope_adjustment ) model.clear_empty() #构建保存文件的路径，并保存到results文件夹内 str(int(time.time())) if not os.path.exists("results"): os.makedirs("results") key = "auto" if auto_f0 else f"{int(vc_transform)}key" cluster = "_" if cluster_ratio == 0 else f"_{cluster_ratio}_" isdiffusion = "sovits" if model.shallow_diffusion: isdiffusion = "sovdiff" if model.only_diffusion: isdiffusion = "diff" output_file_name = 'result_'+truncated_basename+f'_{sid}_{key}{cluster}{isdiffusion}.{output_format}' output_file = os.path.join("results", output_file_name) soundfile.write(output_file, _audio, model.target_sample, format=output_format) return output_file def vc_fn(sid, input_audio, output_format, vc_transform, auto_f0,cluster_ratio, slice_db, noise_scale,pad_seconds,cl_num,lg_num,lgr_num,f0_predictor,enhancer_adaptive_key,cr_threshold,k_step,use_spk_mix,second_encoding,loudness_envelope_adjustment): global model try: if input_audio is None: return "You need to upload an audio", None if model is None: return "You need to upload an model", None if getattr(model, 'cluster_model', None) is None and model.feature_retrieval is False: if cluster_ratio != 0: return "You need to upload an cluster model or feature retrieval model before assigning cluster ratio!", None #print(input_audio) audio, sampling_rate = soundfile.read(input_audio) #print(audio.shape,sampling_rate) if np.issubdtype(audio.dtype, np.integer): audio = (audio / np.iinfo(audio.dtype).max).astype(np.float32) #print(audio.dtype) if len(audio.shape) > 1: audio = librosa.to_mono(audio.transpose(1, 0)) # 未知原因Gradio上传的filepath会有一个奇怪的固定后缀，这里去掉 truncated_basename = Path(input_audio).stem[:-6] processed_audio = os.path.join("raw", f"{truncated_basename}.wav") soundfile.write(processed_audio, audio, sampling_rate, format="wav") output_file = vc_infer(output_format, sid, processed_audio, truncated_basename, vc_transform, auto_f0, cluster_ratio, slice_db, noise_scale, pad_seconds, cl_num, lg_num, lgr_num, f0_predictor, enhancer_adaptive_key, cr_threshold, k_step, use_spk_mix, second_encoding, loudness_envelope_adjustment) return "Success", output_file except Exception as e: if debug: traceback.print_exc() raise gr.Error(e) def text_clear(text): return re.sub(r"[\n\,\(\) ]", "", text) def vc_fn2(_text, _lang, _gender, _rate, _volume, sid, output_format, vc_transform, auto_f0,cluster_ratio, slice_db, noise_scale,pad_seconds,cl_num,lg_num,lgr_num,f0_predictor,enhancer_adaptive_key,cr_threshold, k_step,use_spk_mix,second_encoding,loudness_envelope_adjustment): global model try: if model is None: return "You need to upload an model", None if getattr(model, 'cluster_model', None) is None and model.feature_retrieval is False: if cluster_ratio != 0: return "You need to upload an cluster model or feature retrieval model before assigning cluster ratio!", None _rate = f"+{int(_rate*100)}%" if _rate >= 0 else f"{int(_rate*100)}%" _volume = f"+{int(_volume*100)}%" if _volume >= 0 else f"{int(_volume*100)}%" if _lang == "Auto": _gender = "Male" if _gender == "男" else "Female" subprocess.run([sys.executable, "edgetts/tts.py", _text, _lang, _rate, _volume, _gender]) else: subprocess.run([sys.executable, "edgetts/tts.py", _text, _lang, _rate, _volume]) target_sr = 44100 y, sr = librosa.load("tts.wav") resampled_y = librosa.resample(y, orig_sr=sr, target_sr=target_sr) soundfile.write("tts.wav", resampled_y, target_sr, subtype = "PCM_16") input_audio = "tts.wav" #audio, _ = soundfile.read(input_audio) output_file_path = vc_infer(output_format, sid, input_audio, "tts", vc_transform, auto_f0, cluster_ratio, slice_db, noise_scale, pad_seconds, cl_num, lg_num, lgr_num, f0_predictor, enhancer_adaptive_key, cr_threshold, k_step, use_spk_mix, second_encoding, loudness_envelope_adjustment) os.remove("tts.wav") return "Success", output_file_path except Exception as e: if debug: traceback.print_exc() # noqa: E701 raise gr.Error(e) def model_compression(_model): if _model == "": return "请先选择要压缩的模型" else: model_path = os.path.split(_model.name) filename, extension = os.path.splitext(model_path[1]) output_model_name = f"{filename}_compressed{extension}" output_path = os.path.join(os.getcwd(), output_model_name) removeOptimizer(_model.name, output_path) return f"模型已成功被保存在了{output_path}" def scan_local_models(): res = [] candidates = glob.glob(os.path.join(local_model_root, '**', '*.json'), recursive=True) candidates = set([os.path.dirname(c) for c in candidates]) for candidate in candidates: jsons = glob.glob(os.path.join(candidate, '*.json')) pths = glob.glob(os.path.join(candidate, '*.pth')) if (len(jsons) == 1 and len(pths) == 1): # must contain exactly one json and one pth file res.append(candidate) return res def local_model_refresh_fn(): choices = scan_local_models() return gr.Dropdown.update(choices=choices) def debug_change(): global debug debug = debug_button.value with gr.Blocks( theme=gr.themes.Base( primary_hue = gr.themes.colors.green, font=["Source Sans Pro", "Arial", "sans-serif"], font_mono=['JetBrains mono', "Consolas", 'Courier New'] ), ) as app: with gr.Tabs(): with gr.TabItem("推理"): gr.Markdown(value=""" So-vits-svc 4.0 推理 webui """) with gr.Row(variant="panel"): with gr.Column(): gr.Markdown(value=""" 模型设置 """) with gr.Tabs(): # invisible checkbox that tracks tab status local_model_enabled = gr.Checkbox(value=False, visible=False) with gr.TabItem('上传') as local_model_tab_upload: with gr.Row(): model_path = gr.File(label="选择模型文件") config_path = gr.File(label="选择配置文件") with gr.TabItem('本地') as local_model_tab_local: gr.Markdown(f'模型应当放置于{local_model_root}文件夹下') local_model_refresh_btn = gr.Button('刷新本地模型列表') local_model_selection = gr.Dropdown(label='选择模型文件夹', choices=[], interactive=True) with gr.Row(): diff_model_path = gr.File(label="选择扩散模型文件") diff_config_path = gr.File(label="选择扩散模型配置文件") cluster_model_path = gr.File(label="选择聚类模型或特征检索文件（没有可以不选）") device = gr.Dropdown(label="推理设备，默认为自动选择CPU和GPU", choices=["Auto",*cuda.keys(),"cpu"], value="Auto") enhance = gr.Checkbox(label="是否使用NSF_HIFIGAN增强,该选项对部分训练集少的模型有一定的音质增强效果，但是对训练好的模型有反面效果，默认关闭", value=False) only_diffusion = gr.Checkbox(label="是否使用全扩散推理，开启后将不使用So-VITS模型，仅使用扩散模型进行完整扩散推理，默认关闭", value=False) with gr.Column(): gr.Markdown(value=""" 左侧文件全部选择完毕后(全部文件模块显示download)，点击“加载模型”进行解析： """) model_load_button = gr.Button(value="加载模型", variant="primary") model_unload_button = gr.Button(value="卸载模型", variant="primary") sid = gr.Dropdown(label="音色（说话人）") sid_output = gr.Textbox(label="Output Message") with gr.Row(variant="panel"): with gr.Column(): gr.Markdown(value=""" 推理设置 """) auto_f0 = gr.Checkbox(label="自动f0预测，配合聚类模型f0预测效果更好,会导致变调功能失效（仅限转换语音，歌声勾选此项会究极跑调）", value=False) f0_predictor = gr.Dropdown(label="选择F0预测器,可选择crepe,pm,dio,harvest,rmvpe,默认为pm(注意：crepe为原F0使用均值滤波器)", choices=["pm","dio","harvest","crepe","rmvpe"], value="pm") vc_transform = gr.Number(label="变调（整数，可以正负，半音数量，升高八度就是12）", value=0) cluster_ratio = gr.Number(label="聚类模型/特征检索混合比例，0-1之间，0即不启用聚类/特征检索。使用聚类/特征检索能提升音色相似度，但会导致咬字下降（如果使用建议0.5左右）", value=0) slice_db = gr.Number(label="切片阈值", value=-40) output_format = gr.Radio(label="音频输出格式", choices=["wav", "flac", "mp3"], value = "wav") noise_scale = gr.Number(label="noise_scale 建议不要动，会影响音质，玄学参数", value=0.4) k_step = gr.Slider(label="浅扩散步数，只有使用了扩散模型才有效，步数越大越接近扩散模型的结果", value=100, minimum = 1, maximum = 1000) with gr.Column(): pad_seconds = gr.Number(label="推理音频pad秒数，由于未知原因开头结尾会有异响，pad一小段静音段后就不会出现", value=0.5) cl_num = gr.Number(label="音频自动切片，0为不切片，单位为秒(s)", value=0) lg_num = gr.Number(label="两端音频切片的交叉淡入长度，如果自动切片后出现人声不连贯可调整该数值，如果连贯建议采用默认值0，注意，该设置会影响推理速度，单位为秒/s", value=0) lgr_num = gr.Number(label="自动音频切片后，需要舍弃每段切片的头尾。该参数设置交叉长度保留的比例，范围0-1,左开右闭", value=0.75) enhancer_adaptive_key = gr.Number(label="使增强器适应更高的音域(单位为半音数)|默认为0", value=0) cr_threshold = gr.Number(label="F0过滤阈值，只有启动crepe时有效. 数值范围从0-1. 降低该值可减少跑调概率，但会增加哑音", value=0.05) loudness_envelope_adjustment = gr.Number(label="输入源响度包络替换输出响度包络融合比例，越靠近1越使用输出响度包络", value = 0) second_encoding = gr.Checkbox(label = "二次编码，浅扩散前会对原始音频进行二次编码，玄学选项，效果时好时差，默认关闭", value=False) use_spk_mix = gr.Checkbox(label = "动态声线融合", value = False, interactive = False) with gr.Tabs(): with gr.TabItem("音频转音频"): vc_input3 = gr.Audio(label="选择音频", type="filepath") vc_submit = gr.Button("音频转换", variant="primary") with gr.TabItem("文字转音频"): text2tts=gr.Textbox(label="在此输入要转译的文字。注意，使用该功能建议打开F0预测，不然会很怪") with gr.Row(): tts_gender = gr.Radio(label = "说话人性别", choices = ["男","女"], value = "男") tts_lang = gr.Dropdown(label = "选择语言，Auto为根据输入文字自动识别", choices=SUPPORTED_LANGUAGES, value = "Auto") tts_rate = gr.Slider(label = "TTS语音变速（倍速相对值）", minimum = -1, maximum = 3, value = 0, step = 0.1) tts_volume = gr.Slider(label = "TTS语音音量（相对值）", minimum = -1, maximum = 1.5, value = 0, step = 0.1) vc_submit2 = gr.Button("文字转换", variant="primary") with gr.Row(): with gr.Column(): vc_output1 = gr.Textbox(label="Output Message") with gr.Column(): vc_output2 = gr.Audio(label="Output Audio", interactive=False) with gr.TabItem("小工具/实验室特性"): gr.Markdown(value=""" So-vits-svc 4.0 小工具/实验室特性 """) with gr.Tabs(): with gr.TabItem("静态声线融合"): gr.Markdown(value=""" 介绍:该功能可以将多个声音模型合成为一个声音模型(多个模型参数的凸组合或线性组合)，从而制造出现实中不存在的声线 注意： 1.该功能仅支持单说话人的模型 2.如果强行使用多说话人模型，需要保证多个模型的说话人数量相同，这样可以混合同一个SpaekerID下的声音 3.保证所有待混合模型的config.json中的model字段是相同的 4.输出的混合模型可以使用待合成模型的任意一个config.json，但聚类模型将不能使用 5.批量上传模型的时候最好把模型放到一个文件夹选中后一起上传 6.混合比例调整建议大小在0-100之间，也可以调为其他数字，但在线性组合模式下会出现未知的效果 7.混合完毕后，文件将会保存在项目根目录中，文件名为output.pth 8.凸组合模式会将混合比例执行Softmax使混合比例相加为1，而线性组合模式不会 """) mix_model_path = gr.Files(label="选择需要混合模型文件") mix_model_upload_button = gr.UploadButton("选择/追加需要混合模型文件", file_count="multiple") mix_model_output1 = gr.Textbox( label="混合比例调整，单位/%", interactive = True ) mix_mode = gr.Radio(choices=["凸组合", "线性组合"], label="融合模式",value="凸组合",interactive = True) mix_submit = gr.Button("声线融合启动", variant="primary") mix_model_output2 = gr.Textbox( label="Output Message" ) mix_model_path.change(updata_mix_info,[mix_model_path],[mix_model_output1]) mix_model_upload_button.upload(upload_mix_append_file, [mix_model_upload_button,mix_model_path], [mix_model_path,mix_model_output1]) mix_submit.click(mix_submit_click, [mix_model_output1,mix_mode], [mix_model_output2]) with gr.TabItem("模型压缩工具"): gr.Markdown(value=""" 该工具可以实现对模型的体积压缩，在**不影响模型推理功能**的情况下，将原本约600M的So-VITS模型压缩至约200M, 大大减少了硬盘的压力。 **注意：压缩后的模型将无法继续训练，请在确认封炉后再压缩。** """) model_to_compress = gr.File(label="模型上传") compress_model_btn = gr.Button("压缩模型", variant="primary") compress_model_output = gr.Textbox(label="输出信息", value="") compress_model_btn.click(model_compression, [model_to_compress], [compress_model_output]) with gr.Tabs(): with gr.Row(variant="panel"): with gr.Column(): gr.Markdown(value=""" WebUI设置 """) debug_button = gr.Checkbox(label="Debug模式，如果向社区反馈BUG需要打开，打开后控制台可以显示具体错误提示", value=debug) # refresh local model list local_model_refresh_btn.click(local_model_refresh_fn, outputs=local_model_selection) # set local enabled/disabled on tab switch local_model_tab_upload.select(lambda: False, outputs=local_model_enabled) local_model_tab_local.select(lambda: True, outputs=local_model_enabled) vc_submit.click(vc_fn, [sid, vc_input3, output_format, vc_transform,auto_f0,cluster_ratio, slice_db, noise_scale,pad_seconds,cl_num,lg_num,lgr_num,f0_predictor,enhancer_adaptive_key,cr_threshold,k_step,use_spk_mix,second_encoding,loudness_envelope_adjustment], [vc_output1, vc_output2]) vc_submit2.click(vc_fn2, [text2tts, tts_lang, tts_gender, tts_rate, tts_volume, sid, output_format, vc_transform,auto_f0,cluster_ratio, slice_db, noise_scale,pad_seconds,cl_num,lg_num,lgr_num,f0_predictor,enhancer_adaptive_key,cr_threshold,k_step,use_spk_mix,second_encoding,loudness_envelope_adjustment], [vc_output1, vc_output2]) debug_button.change(debug_change,[],[]) model_load_button.click(modelAnalysis,[model_path,config_path,cluster_model_path,device,enhance,diff_model_path,diff_config_path,only_diffusion,use_spk_mix,local_model_enabled,local_model_selection],[sid,sid_output]) model_unload_button.click(modelUnload,[],[sid,sid_output]) os.system("start http://127.0.0.1:7860") app.launch()