pax_global_header00006660000000000000000000000064144426230430014514gustar00rootroot0000000000000052 comment=5c72f3357354374177d1268abab3da42a03cbb64 pytorch/000077500000000000000000000000001444262304300125505ustar00rootroot00000000000000pytorch/.github/000077500000000000000000000000001444262304300141105ustar00rootroot00000000000000pytorch/.github/CODEOWNERS000066400000000000000000000013671444262304300155120ustar00rootroot00000000000000# About code owners # https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners # # Code owners recieve review requests when parts of code they "own" change. *.py @rusty1s @wsad1 /torch_geometric/nn/ @EdisonLeeeee /torch_geometric/explain/ @RexYing @RBendias @dufourc1 /torch_geometric/data/ @mananshah99 /torch_geometric/loader/ @mananshah99 @pyg-team/intel-team /torch_geometric/sampler/ @mananshah99 @pyg-team/intel-team /torch_geometric/transforms/ @wsad1 /torch_geometric/utils/ @wsad1 /torch_geometric/datasets/ @wsad1 /torch_geometric/graphgym/ @JiaxuanYou /test/ @wsad1 /docs/ @rusty1s /graphgym/ @JiaxuanYou /examples/ @wsad1 /benchmark/ @pyg-team/intel-team pytorch/.github/CONTRIBUTING.md000066400000000000000000000123131444262304300163410ustar00rootroot00000000000000# Contributing to PyG If you are interested in contributing to PyG, your contributions will likely fall into one of the following two categories: 1. You want to implement a new feature: - In general, we accept any features as long as they fit the scope of this package. If you are unsure about this or need help on the design/implementation of your feature, post about it in an issue. 2. You want to fix a bug: - Feel free to send a Pull Request (PR) any time you encounter a bug. Please provide a clear and concise description of what the bug was. If you are unsure about if this is a bug at all or how to fix, post about it in an issue. Once you finish implementing a feature or bug-fix, please send a PR to https://github.com/pyg-team/pytorch_geometric. Your PR will be merged after one or more rounds of reviews by the [pyg-team](https://github.com/pyg-team). If your PR isn't merged anytime soon (*e.g.,* due to its large size, complexity or unavailability of reviewers), try moving your contribution to the [`torch_geometric.contrib`](https://pytorch-geometric.readthedocs.io/en/latest/modules/contrib.html) package. [`torch_geometric.contrib`](https://pytorch-geometric.readthedocs.io/en/latest/modules/contrib.html) has less rigourous review requirements and might lead to your PR getting merged faster. ## Developing PyG To develop PyG on your machine, here are some tips: 1. Ensure that you are running on one of the two latest PyTorch releases (*e.g.*, `1.12.0`): ```python import torch print(torch.__version__) ``` 2. Follow the [installation instructions](https://github.com/pyg-team/pytorch_geometric#installation) to install `pyg-lib`, `torch-scatter`, `torch-sparse`, `torch-cluster` and `torch-spline-conv` (if you haven't already): ```bash pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv -f https://data.pyg.org/whl/torch-${TORCH}+${CUDA}.html ``` where `${TORCH}` should be replaced by your PyTorch version (*e.g.*, `1.12.0`), and `${CUDA}` should be replaced by your CUDA version (*e.g.*, `cpu` or `cu116`). 3. Uninstall all existing PyG installations: ```bash pip uninstall torch-geometric pip uninstall torch-geometric # run this command twice ``` 4. Clone a copy of PyG from source: ```bash git clone https://github.com/pyg-team/pytorch_geometric cd pytorch_geometric ``` 5. If you already cloned PyG from source, update it: ```bash git pull ``` 6. Install PyG in editable mode: ```bash pip install -e ".[dev,full]" ``` This mode will symlink the Python files from the current local source tree into the Python install. Hence, if you modify a Python file, you do not need to reinstall PyG again and again. 7. Ensure that you have a working PyG installation by running the entire test suite with ```bash pytest ``` In case an error occurs, please first check if all sub-packages ([`pyg-lib`](https://github.com/pyg-team/pyg-lib), [`torch-scatter`](https://github.com/rusty1s/pytorch_scatter), [`torch-sparse`](https://github.com/rusty1s/pytorch_sparse), [`torch-cluster`](https://github.com/rusty1s/pytorch_cluster) and [`torch-spline-conv`](https://github.com/rusty1s/pytorch_spline_conv)) are on its latest reported version. 8. Install pre-commit hooks: ```bash pre-commit install ``` ## Unit Testing The PyG testing suite is located under `test/`. Run the entire test suite with ```bash pytest ``` or test individual files via, _e.g._, `pytest test/utils/test_convert.py`. ## Continuous Integration PyG uses [GitHub Actions](https://github.com/pyg-team/pytorch_geometric/actions) in combination with [CodeCov](https://codecov.io/github/pyg-team/pytorch_geometric?branch=master) for continuous integration. Everytime you send a Pull Request, your commit will be built and checked against the PyG guidelines: 1. Ensure that your code is formatted correctly by testing against the styleguide of [`flake8`](https://github.com/PyCQA/flake8). We use the [`Flake8-pyproject`](https://pypi.org/project/Flake8-pyproject/) plugin for configuration: ```bash flake8 . ``` If you do not want to format your code manually, we recommend to use [`yapf`](https://github.com/google/yapf). 2. Ensure that the entire test suite passes and that code coverage roughly stays the same. Please feel encouraged to provide a test with your submitted code. To test, either run ```bash pytest --cov ``` or ```bash FULL_TEST=1 pytest --cov ``` (which runs a set of additional but time-consuming tests) dependening on your needs. 3. Add your feature/bugfix to the [`CHANGELOG.md`](https://github.com/pyg-team/pytorch_geometric/blob/master/CHANGELOG.md?plain=1). If multiple PRs move towards integrating a single feature, it is advised to group them together into one bullet point. ## Building Documentation To build the documentation: 1. [Build and install](#developing-pyg) PyG from source. 2. Install [Sphinx](https://www.sphinx-doc.org/en/master/) theme via ```bash pip install git+https://github.com/pyg-team/pyg_sphinx_theme.git ``` 3. Generate the documentation via: ```bash cd docs make html ``` The documentation is now available to view by opening `docs/build/html/index.html`. pytorch/.github/ISSUE_TEMPLATE/000077500000000000000000000000001444262304300162735ustar00rootroot00000000000000pytorch/.github/ISSUE_TEMPLATE/bug-report.yml000066400000000000000000000050621444262304300211070ustar00rootroot00000000000000name: "πŸ› Bug Report" description: "Submit a report to help us reproduce and fix the bug" labels: bug body: - type: markdown attributes: value: > #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/pyg-team/pytorch_geometric/issues). # - type: textarea attributes: label: πŸ› Describe the bug description: | Please provide a clear and concise description of the bug. If relevant, add a minimal example so that we can reproduce the error by running the code. It is very important for the snippet to be as minimal as possible, so please take time to trim down any irrelevant code to help us debug efficiently. We are going to copy-paste your code and we expect to get the same result as you did: avoid any external data, and include the relevant imports: ```python # All necessary imports at the beginning import torch from torch_geometric.utils import to_undirected # A minimal reproducing example trimmed down to the essential parts: edge_index = torch.tensor([[0, 1, 2], [1, 2, 3]]) edge_index = to_undirected(edge_index, num_nodes=1) assert edge_index.size(1) == 6 # We expect that the number of edges is doubled. # NOTE: the bug is that num_nodes < edge_index.max() + 1 ``` Please also paste or describe the results you observe instead of the expected results. If you observe an error, please paste the error message including the **full** traceback of the exception. It may be relevant to wrap error messages in ```` ```triple quotes blocks``` ````. placeholder: | A clear and concise description of the bug. ```python # Sample code to reproduce the problem ``` ``` The error message you got, with the full traceback. ``` validations: required: true - type: textarea attributes: label: Environment description: | Please provide as much information as possible about your environment, such as your PyG (`print(torch_geometric.__version__)`) and PyTorch version (`print(torch.__version__)`), your OS (*e.g.*, Linux), and your Python version (*e.g.*, `3.9`): value: | * PyG version: * PyTorch version: * OS: * Python version: * CUDA/cuDNN version: * How you installed PyTorch and PyG (`conda`, `pip`, source): * Any other relevant information (*e.g.*, version of `torch-scatter`): pytorch/.github/ISSUE_TEMPLATE/config.yml000066400000000000000000000005341444262304300202650ustar00rootroot00000000000000blank_issues_enabled: false contact_links: - name: πŸ™ Ask a Question url: https://github.com/pyg-team/pytorch_geometric/discussions/new about: Ask and answer PyG related questions - name: πŸ’¬ Slack url: https://join.slack.com/t/torchgeometricco/shared_invite/zt-p6br3yuo-BxRoe36OHHLF6jYU8xHtBA about: Chat with our community pytorch/.github/ISSUE_TEMPLATE/documentation.yml000066400000000000000000000007671444262304300217010ustar00rootroot00000000000000name: "πŸ“š Typos and Doc Fixes" description: "Tell us about how we can improve our documentation" labels: documentation body: - type: textarea attributes: label: πŸ“š Describe the documentation issue description: | A clear and concise description of the issue. validations: required: true - type: textarea attributes: label: Suggest a potential alternative/fix description: | Tell us how we could improve the documentation in this regard. pytorch/.github/ISSUE_TEMPLATE/feature-request.yml000066400000000000000000000015411444262304300221400ustar00rootroot00000000000000name: "πŸš€ Feature Request" description: "Propose a new PyG feature" labels: feature body: - type: textarea attributes: label: πŸš€ The feature, motivation and pitch description: > A clear and concise description of the feature proposal. Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., *"I'm working on X and would like Y to be possible"*. If this is related to another GitHub issue, please link here too. validations: required: true - type: textarea attributes: label: Alternatives description: > A description of any alternative solutions or features you've considered, if any. - type: textarea attributes: label: Additional context description: > Add any other context or screenshots about the feature request. pytorch/.github/ISSUE_TEMPLATE/installation.yml000066400000000000000000000026731444262304300215270ustar00rootroot00000000000000name: "😡 Installation" description: "Report an installation problem" labels: installation body: - type: markdown attributes: value: > #### Before submitting an installation problem, please make sure the issue hasn't been already reported by searching through [the existing and past issues](https://github.com/pyg-team/pytorch_geometric/issues). # - type: textarea attributes: label: 😡 Describe the installation problem description: | Please provide a clear and concise description of the installation problem. If you have installation log files, please prvoide them here as well. It may be relevant to wrap the log files in ```` ```triple quotes blocks``` ````. placeholder: | A clear and concise description of the installation problem. validations: required: true - type: textarea attributes: label: Environment description: | Please provide as much information as possible about your environment, such as your PyG (`print(torch_geometric.__version__)`) and PyTorch version (`print(torch.__version__)`), your OS (*e.g.*, Linux), and your Python version (*e.g.*, `3.9`): value: | * PyG version: * PyTorch version: * OS: * Python version: * CUDA/cuDNN version: * How you installed PyTorch and PyG (`conda`, `pip`, source): * Any other relevant information (*e.g.*, version of `torch-scatter`): pytorch/.github/ISSUE_TEMPLATE/refactor.yml000066400000000000000000000011471444262304300206260ustar00rootroot00000000000000name: "πŸ›  Refactor" description: "Suggest a code refactor or deprecation" labels: refactor body: - type: textarea attributes: label: πŸ›  Proposed Refactor description: | A clear and concise description of the refactor proposal. Please outline the motivation for the proposal. If this is related to another GitHub issue, please link here too. validations: required: true - type: textarea attributes: label: Suggest a potential alternative/fix description: | Tell us how we could improve the code in this regard. validations: required: true pytorch/.github/actions/000077500000000000000000000000001444262304300155505ustar00rootroot00000000000000pytorch/.github/actions/setup/000077500000000000000000000000001444262304300167105ustar00rootroot00000000000000pytorch/.github/actions/setup/action.yml000066400000000000000000000043221444262304300207110ustar00rootroot00000000000000name: Setup inputs: python-version: required: false default: '3.8' torch-version: required: false default: '2.0.0' cuda-version: required: false default: cpu torchvision-version: required: false default: '0.15.0' full_install: required: false default: true runs: using: composite steps: - name: Set up Python ${{ inputs.python-version }} uses: actions/setup-python@v4.3.0 with: python-version: ${{ inputs.python-version }} check-latest: true cache: pip cache-dependency-path: | pyproject.toml - name: Install PyTorch ${{ inputs.torch-version }}+${{ inputs.cuda-version }} if: ${{ inputs.torch-version != 'nightly' }} run: | pip install torch==${{ inputs.torch-version }} --extra-index-url https://download.pytorch.org/whl/${{ inputs.cuda-version }} python -c "import torch; print('PyTorch:', torch.__version__)" python -c "import torch; print('CUDA:', torch.version.cuda)" shell: bash - name: Install PyTorch ${{ inputs.torch-version }}+${{ inputs.cuda-version }} if: ${{ inputs.torch-version == 'nightly' }} run: | pip install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/${{ inputs.cuda-version }} python -c "import torch; print('PyTorch:', torch.__version__)" python -c "import torch; print('CUDA:', torch.version.cuda)" shell: bash - name: Install pyg-lib # pyg-lib is currently only available on Linux. if: ${{ inputs.torch-version != 'nightly' && runner.os == 'Linux' }} run: | pip install pyg-lib -f https://data.pyg.org/whl/nightly/torch-${{ inputs.torch-version }}+${{ inputs.cuda-version }}.html shell: bash - name: Install extension packages if: ${{ inputs.full_install == 'true' && inputs.torch-version != 'nightly' }} run: | pip install torchvision==${{ inputs.torchvision-version }} --extra-index-url https://download.pytorch.org/whl/${{ inputs.cuda-version }} pip install torch-scatter torch-sparse torch-cluster torch-spline-conv -f https://data.pyg.org/whl/torch-${{ inputs.torch-version }}+${{ inputs.cuda-version }}.html shell: bash pytorch/.github/labeler.yml000066400000000000000000000011071444262304300162400ustar00rootroot00000000000000documentation: - docs/**/* example: - examples/**/* data: - torch_geometric/data/**/* dataset: - torch_geometric/io/**/* - torch_geometric/datasets/**/* sampler: - torch_geometric/sampler/**/* loader: - torch_geometric/loader/**/* nn: - torch_geometric/nn/**/* explain: - torch_geometric/explain/**/* transform: - torch_geometric/transforms/**/* utils: - torch_geometric/utils/**/* contrib: - torch_geometric/contrib/**/* graphgym: - graphgym/**/* - torch_geometric/graphgym/**/* benchmark: - benchmark/**/* - torch_geometric/profile/**/* pytorch/.github/workflows/000077500000000000000000000000001444262304300161455ustar00rootroot00000000000000pytorch/.github/workflows/building_pyg_conda.yml000066400000000000000000000050551444262304300225150ustar00rootroot00000000000000name: Building PyG Conda on: [workflow_dispatch] # yamllint disable-line rule:truthy jobs: conda-build: runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ubuntu-18.04, macos-10.15, windows-2019] python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] torch-version: [1.12.0, 1.13.0, 2.0.0] cuda-version: ['cpu', 'cu102', 'cu113', 'cu116', 'cu117', 'cu118'] exclude: - torch-version: 1.12.0 python-version: '3.11' - torch-version: 1.12.0 cuda-version: 'cu117' - torch-version: 1.12.0 cuda-version: 'cu118' - torch-version: 1.13.0 python-version: '3.11' - torch-version: 1.13.0 cuda-version: 'cu102' - torch-version: 1.13.0 cuda-version: 'cu113' - torch-version: 1.13.0 cuda-version: 'cu118' - torch-version: 2.0.0 python-version: '3.7' - torch-version: 2.0.0 cuda-version: 'cu102' - torch-version: 2.0.0 cuda-version: 'cu113' - torch-version: 2.0.0 cuda-version: 'cu115' - torch-version: 2.0.0 cuda-version: 'cu116' - os: macos-10.15 cuda-version: 'cu102' - os: macos-10.15 cuda-version: 'cu113' - os: macos-10.15 cuda-version: 'cu116' - os: macos-10.15 cuda-version: 'cu117' - os: macos-10.15 cuda-version: 'cu118' steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Conda for Python ${{ matrix.python-version }} uses: conda-incubator/setup-miniconda@v2 with: python-version: ${{ matrix.python-version }} - name: Free up disk space if: ${{ runner.os == 'Linux' }} run: | sudo rm -rf /usr/share/dotnet - name: Install Conda packages run: | conda install conda-build conda-verify --yes shell: bash -l {0} - name: Build Conda package run: | ./conda/pyg/build_conda.sh ${{ matrix.python-version }} ${{ matrix.torch-version }} ${{ matrix.cuda-version }} shell: bash -l {0} - name: Publish Conda package run: | conda install anaconda-client --yes anaconda upload --force --label main $HOME/conda-bld/*/*.tar.bz2 env: ANACONDA_API_TOKEN: ${{ secrets.PYG_ANACONDA_TOKEN }} shell: bash -l {0} pytorch/.github/workflows/building_rusty1s_conda.yml000066400000000000000000000051031444262304300233420ustar00rootroot00000000000000name: Building rusty1s Conda on: [workflow_dispatch] # yamllint disable-line rule:truthy jobs: conda-build: runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ubuntu-18.04, macos-10.15, windows-2019] python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] torch-version: [1.12.0, 1.13.0, 2.0.0] cuda-version: ['cpu', 'cu102', 'cu113', 'cu116', 'cu117', 'cu118'] exclude: - torch-version: 1.12.0 python-version: '3.11' - torch-version: 1.12.0 cuda-version: 'cu117' - torch-version: 1.12.0 cuda-version: 'cu118' - torch-version: 1.13.0 python-version: '3.11' - torch-version: 1.13.0 cuda-version: 'cu102' - torch-version: 1.13.0 cuda-version: 'cu113' - torch-version: 1.13.0 cuda-version: 'cu118' - torch-version: 2.0.0 python-version: '3.7' - torch-version: 2.0.0 cuda-version: 'cu102' - torch-version: 2.0.0 cuda-version: 'cu113' - torch-version: 2.0.0 cuda-version: 'cu115' - torch-version: 2.0.0 cuda-version: 'cu116' - os: macos-10.15 cuda-version: 'cu102' - os: macos-10.15 cuda-version: 'cu113' - os: macos-10.15 cuda-version: 'cu116' - os: macos-10.15 cuda-version: 'cu117' - os: macos-10.15 cuda-version: 'cu118' steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Conda for Python ${{ matrix.python-version }} uses: conda-incubator/setup-miniconda@v2 with: python-version: ${{ matrix.python-version }} - name: Free up disk space if: ${{ runner.os == 'Linux' }} run: | sudo rm -rf /usr/share/dotnet - name: Install Conda packages run: | conda install conda-build conda-verify --yes shell: bash -l {0} - name: Build Conda package run: | ./conda/pytorch-geometric/build_conda.sh ${{ matrix.python-version }} ${{ matrix.torch-version }} ${{ matrix.cuda-version }} shell: bash -l {0} - name: Publish Conda package run: | conda install anaconda-client --yes anaconda upload --force --label main $HOME/conda-bld/*/*.tar.bz2 env: ANACONDA_API_TOKEN: ${{ secrets.RUSTY1S_ANACONDA_TOKEN }} shell: bash -l {0} pytorch/.github/workflows/changelog.yml000066400000000000000000000005511444262304300206200ustar00rootroot00000000000000name: Changelog Enforcer on: # yamllint disable-line rule:truthy pull_request: types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled] jobs: changelog: runs-on: ubuntu-latest steps: - name: Enforce changelog entry uses: dangoslen/changelog-enforcer@v3 with: skipLabels: 'skip-changelog' pytorch/.github/workflows/documentation.yml000066400000000000000000000023611444262304300215430ustar00rootroot00000000000000name: Documentation on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: make_html: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 40 # Skip workflow if only certain files have been changed. - name: Get changed files id: changed-files-specific uses: tj-actions/changed-files@v34 with: files: | benchmark/** conda/** docker/** examples/** graphgym/** README.md CHANGELOG.md - name: Setup packages if: steps.changed-files-specific.outputs.only_changed != 'true' uses: ./.github/actions/setup with: full_install: false - name: Install main package if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pip install git+https://github.com/pyg-team/pyg_sphinx_theme.git pip install -e . - name: Build documentation if: steps.changed-files-specific.outputs.only_changed != 'true' run: | cd docs && make clean && make html SPHINXOPTS="-W" # Fail on warning. pytorch/.github/workflows/examples.yml000066400000000000000000000020401444262304300205020ustar00rootroot00000000000000name: Examples on: # yamllint disable-line rule:truthy workflow_dispatch: schedule: - cron: "0 7 * * *" # Everyday at 7:00am UTC/11:00pm PST jobs: pytest: if: github.repository == 'pyg-team/pytorch_geometric' runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Setup packages uses: ./.github/actions/setup - name: Install main package run: | pip install .[benchmark] - name: Run GCN on Cora run: | python examples/gcn.py --wandb env: WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }} - name: Run GAT on Cora run: | python examples/gat.py --wandb env: WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }} - name: Run GIN on MUTAG run: | python examples/mutag_gin.py --wandb env: WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }} - name: Run GNNExplainer run: | python examples/explain/gnn_explainer.py pytorch/.github/workflows/full_testing.yml000066400000000000000000000026741444262304300214000ustar00rootroot00000000000000name: Full Testing on: # yamllint disable-line rule:truthy workflow_dispatch: schedule: - cron: "0 6 * * *" # Everyday at 6:00am UTC/10:00pm PST jobs: pytest: if: github.repository == 'pyg-team/pytorch_geometric' runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ubuntu-latest, windows-latest] python-version: ['3.8', '3.10'] torch-version: [1.13.0, 2.0.0, nightly] include: - torch-version: 1.13.0 torchvision-version: 0.14.0 - torch-version: 2.0.0 torchvision-version: 0.15.0 - torch-version: nightly torchvision-version: nightly steps: - name: Checkout repository uses: actions/checkout@v3 - name: Setup packages uses: ./.github/actions/setup with: python-version: ${{ matrix.python-version }} torch-version: ${{ matrix.torch-version }} torchvision-version: ${{ matrix.torchvision-version }} - name: Install graphviz if: ${{ runner.os == 'Linux' }} run: | sudo apt-get install graphviz - name: Install main package run: | pip install -e .[full,test] - name: Run tests run: | FULL_TEST=1 pytest --cov --cov-report=xml shell: bash - name: Upload coverage uses: codecov/codecov-action@v2 with: fail_ci_if_error: false pytorch/.github/workflows/labeler.yml000066400000000000000000000012331444262304300202750ustar00rootroot00000000000000name: PR Labeler on: # yamllint disable-line rule:truthy pull_request: jobs: triage: if: github.repository == 'pyg-team/pytorch_geometric' runs-on: ubuntu-latest permissions: contents: read pull-requests: write steps: - name: Add PR labels uses: actions/labeler@v4 continue-on-error: true with: repo-token: "${{ secrets.GITHUB_TOKEN }}" sync-labels: true - name: Add PR author uses: samspills/assign-pr-to-author@v1.0 if: github.event_name == 'pull_request' continue-on-error: true with: repo-token: "${{ secrets.GITHUB_TOKEN }}" pytorch/.github/workflows/latest_testing.yml000066400000000000000000000022141444262304300217200ustar00rootroot00000000000000name: Testing PyTorch nightly on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: latest_pytest: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 40 # Skip workflow if only certain files have been changed. - name: Get changed files id: changed-files-specific uses: tj-actions/changed-files@v34 with: files: | benchmark/** conda/** docker/** docs/** examples/** graphgym/** README.md CHANGELOG.md - name: Setup packages if: steps.changed-files-specific.outputs.only_changed != 'true' uses: ./.github/actions/setup with: torch-version: nightly - name: Install main package if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pip install -e .[full,test] - name: Run tests if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pytest pytorch/.github/workflows/linting.yml000066400000000000000000000025341444262304300203400ustar00rootroot00000000000000name: Linting on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: pyright: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Setup packages uses: ./.github/actions/setup with: full_install: false - name: Install pyright run: | pip install pyright - name: Run pyright continue-on-error: true run: | pyright pylint: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4.3.0 with: python-version: 3.8 - name: Install dependencies run: pip install pylint - name: Run linting continue-on-error: true run: pylint **/*.py mypy: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4.3.0 with: python-version: 3.8 - name: Install dependencies run: pip install mypy - name: Check type hints continue-on-error: true run: | mypy --install-types --non-interactive --ignore-missing-imports -m torch_geometric pytorch/.github/workflows/minimal_testing.yml000066400000000000000000000022111444262304300220470ustar00rootroot00000000000000name: Testing minimal PyTorch 2.0 on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: minimal_pytest: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 40 # Skip workflow if only certain files have been changed. - name: Get changed files id: changed-files-specific uses: tj-actions/changed-files@v34 with: files: | benchmark/** conda/** docker/** docs/** examples/** graphgym/** README.md CHANGELOG.md - name: Setup packages if: steps.changed-files-specific.outputs.only_changed != 'true' uses: ./.github/actions/setup with: full_install: false - name: Install main package if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pip install -e .[test] - name: Run tests if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pytest pytorch/.github/workflows/nightly.yml000066400000000000000000000024221444262304300203460ustar00rootroot00000000000000name: Nightly Build on: # yamllint disable-line rule:truthy workflow_dispatch: schedule: - cron: "0 6 * * *" # Everyday at 6:00am UTC/10:00pm PST jobs: build: if: github.repository == 'pyg-team/pytorch_geometric' runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4.3.0 with: python-version: 3.8 - name: Set version run: echo "VERSION=$(sed -n "s/^__version__ = '\(.*\)'/\1/p" torch_geometric/__init__.py)" >> ${GITHUB_ENV} - name: Set time run: echo "TODAY=$(date +'%Y%m%d')" >> ${GITHUB_ENV} - name: Customize build version run: | sed -i "s/$VERSION/$VERSION.dev$TODAY/" torch_geometric/__init__.py sed -i '0,/name="torch_geometric"/s//name="pyg-nightly"/' pyproject.toml # Only change first occurence sed -i "s/version=\"$VERSION\"/version=\"$VERSION.dev$TODAY\"/" pyproject.toml - name: Build package run: | pip install --upgrade build python -m build - name: Publish package uses: pypa/gh-action-pypi-publish@release/v1 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} pytorch/.github/workflows/prev_testing.yml000066400000000000000000000022541444262304300214040ustar00rootroot00000000000000name: Testing PyTorch 1.13 on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: prev_pytest: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 40 # Skip workflow if only certain files have been changed. - name: Get changed files id: changed-files-specific uses: tj-actions/changed-files@v34 with: files: | benchmark/** conda/** docker/** docs/** examples/** graphgym/** README.md CHANGELOG.md - name: Setup packages if: steps.changed-files-specific.outputs.only_changed != 'true' uses: ./.github/actions/setup with: torch-version: 1.13.0 torchvision-version: 0.14.0 - name: Install main package if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pip install -e .[full,test] - name: Run tests if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pytest pytorch/.github/workflows/testing.yml000066400000000000000000000024501444262304300203460ustar00rootroot00000000000000name: Testing PyTorch 2.0 on: # yamllint disable-line rule:truthy push: branches: - master pull_request: jobs: pytest: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 40 # Skip workflow if only certain files have been changed. - name: Get changed files id: changed-files-specific uses: tj-actions/changed-files@v34 with: files: | benchmark/** conda/** docker/** docs/** examples/** graphgym/** README.md CHANGELOG.md - name: Setup packages if: steps.changed-files-specific.outputs.only_changed != 'true' uses: ./.github/actions/setup - name: Install main package if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pip install -e .[full,test] - name: Run tests if: steps.changed-files-specific.outputs.only_changed != 'true' run: | pytest --cov --cov-report=xml - name: Upload coverage if: steps.changed-files-specific.outputs.only_changed != 'true' uses: codecov/codecov-action@v2 with: fail_ci_if_error: false pytorch/.gitignore000066400000000000000000000005351444262304300145430ustar00rootroot00000000000000__pycache__/ .pytest_cache/ .DS_Store data/ build/ dist/ alpha/ runs/ wandb/ .cache/ .eggs/ lightning_logs/ outputs/ graphgym/datasets/ graphgym/results/ *.egg-info/ .ipynb_checkpoints .coverage .coverage.* coverage.xml .vscode .idea .venv *.out *.pt *.onnx examples/**/*.png examples/**/*.pdf benchmark/results/ !torch_geometric/data/ !test/data/ pytorch/.pre-commit-config.yaml000066400000000000000000000026541444262304300170400ustar00rootroot00000000000000ci: autofix_prs: true autoupdate_commit_msg: '[pre-commit.ci] pre-commit suggestions' autoupdate_schedule: quarterly repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: end-of-file-fixer name: End-of-file fixer - id: trailing-whitespace name: Remove trailing whitespaces - id: check-toml name: Check toml - id: check-yaml name: Check yaml exclude: | (?x)^( conda/pytorch-geometric/meta.yaml| conda/pyg/meta.yaml )$ - repo: https://github.com/adrienverge/yamllint.git rev: v1.30.0 hooks: - id: yamllint name: Lint yaml args: [-d, '{extends: default, rules: {line-length: disable, document-start: disable, truthy: {level: error}, braces: {max-spaces-inside: 1}}}'] # - repo: https://github.com/regebro/pyroma # rev: "4.2" # hooks: # - id: pyroma # name: Check packaging # args: [--min=10, .] - repo: https://github.com/google/yapf rev: v0.32.0 hooks: - id: yapf name: Format code additional_dependencies: [toml] - repo: https://github.com/pycqa/isort rev: 5.12.0 hooks: - id: isort name: Sort imports - repo: https://github.com/PyCQA/flake8 rev: 6.0.0 hooks: - id: flake8 name: Check PEP8 additional_dependencies: [Flake8-pyproject] pytorch/CHANGELOG.md000066400000000000000000002515451444262304300143750ustar00rootroot00000000000000# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). ## [2.4.0] - 2023-MM-DD ### Added - Added the `DirGNN` model for learning on directed graphs ([#7458](https://github.com/pyg-team/pytorch_geometric/pull/7458)) - Allow GPU tensors as input to `NodeLoader` and `LinkLoader` ([#7572](https://github.com/pyg-team/pytorch_geometric/pull/7572)) - Added an `embedding_device` option to allow for GPU inference in `BasicGNN` ([#7548](https://github.com/pyg-team/pytorch_geometric/pull/7548)) - Added `Performer` to `GPSConv` and remove `attn_dropout` argument from `GPSConv` ([#7465](https://github.com/pyg-team/pytorch_geometric/pull/7465)) - Enabled `LinkNeighborLoader` to return number of sampled nodes and edges per hop ([#7516](https://github.com/pyg-team/pytorch_geometric/pull/7516)) - Added the `HM` personalized fashion recommendation dataset ([#7515](https://github.com/pyg-team/pytorch_geometric/pull/7515)) - Added the `GraphMixer` model ([#7501](https://github.com/pyg-team/pytorch_geometric/pull/7501)) - Added the `disable_dynamic_shape` experimental flag ([#7246](https://github.com/pyg-team/pytorch_geometric/pull/7246), [#7534](https://github.com/pyg-team/pytorch_geometric/pull/7534)) - Added the option to override `use_segmm` selection in `HeteroLinear` ([#7474](https://github.com/pyg-team/pytorch_geometric/pull/7474)) - Added the `MovieLens-1M` heterogeneous dataset ([#7479](https://github.com/pyg-team/pytorch_geometric/pull/7479)) - Added a CPU-based and GPU-based `map_index` implementation ([#7493](https://github.com/pyg-team/pytorch_geometric/pull/7493)) - Added the `AmazonBook` heterogeneous dataset ([#7483](https://github.com/pyg-team/pytorch_geometric/pull/7483)) - Added hierarichial heterogeneous GraphSAGE example on OGB-MAG ([#7425](https://github.com/pyg-team/pytorch_geometric/pull/7425)) - Added the `torch_geometric.distributed` package ([#7451](https://github.com/pyg-team/pytorch_geometric/pull/7451), [#7452](https://github.com/pyg-team/pytorch_geometric/pull/7452)), [#7482](https://github.com/pyg-team/pytorch_geometric/pull/7482)) - Added the `GDELTLite` dataset ([#7442](https://github.com/pyg-team/pytorch_geometric/pull/7442)) - Added the `approx_knn` function for approximated nearest neighbor search ([#7421](https://github.com/pyg-team/pytorch_geometric/pull/7421)) - Added the `IGMCDataset` ([#7441](https://github.com/pyg-team/pytorch_geometric/pull/7441)) - Added a sparse `cross_entropy` implementation ([#7447](https://github.com/pyg-team/pytorch_geometric/pull/7447), [#7466](https://github.com/pyg-team/pytorch_geometric/pull/7466)) - Added the `MovieLens-100K` heterogeneous dataset ([#7398](https://github.com/pyg-team/pytorch_geometric/pull/7398)) - Added the `PMLP` model and an example ([#7370](https://github.com/pyg-team/pytorch_geometric/pull/7370), [#7543](https://github.com/pyg-team/pytorch_geometric/pull/7543)) - Added padding capabilities to `HeteroData.to_homogeneous()` in case feature dimensionalities do not match ([#7374](https://github.com/pyg-team/pytorch_geometric/pull/7374)) - Added an optional `batch_size` argument to `fps`, `knn`, `knn_graph`, `radius` and `radius_graph` ([#7368](https://github.com/pyg-team/pytorch_geometric/pull/7368)) - Added `PrefetchLoader` capabilities ([#7376](https://github.com/pyg-team/pytorch_geometric/pull/7376), [#7378](https://github.com/pyg-team/pytorch_geometric/pull/7378), [#7383](https://github.com/pyg-team/pytorch_geometric/pull/7383)) - Added an example for hierarichial sampling ([#7244](https://github.com/pyg-team/pytorch_geometric/pull/7244)) - Added KΓΉzu remote backend examples ([#7298](https://github.com/pyg-team/pytorch_geometric/pull/7298)) - Fixed tracing of `add_self_loops` for a dynamic number of nodes ([#7330](https://github.com/pyg-team/pytorch_geometric/pull/7330)) - Added an optional `add_pad_mask` argument to the `Pad` transform ([#7339](https://github.com/pyg-team/pytorch_geometric/pull/7339)) - Added `keep_inter_cluster_edges` option to `ClusterData` to support inter-subgraph edge connections when doing graph partitioning ([#7326](https://github.com/pyg-team/pytorch_geometric/pull/7326)) - Unify graph pooling framework ([#7308](https://github.com/pyg-team/pytorch_geometric/pull/7308)) - Added support for tuples as keys in `ModuleDict`/`ParameterDict` ([#7294](https://github.com/pyg-team/pytorch_geometric/pull/7294)) - Added `NodePropertySplit` transform for creating node-level splits using structural node properties ([#6894](https://github.com/pyg-team/pytorch_geometric/pull/6894)) - Added an option to preserve directed graphs in `CitationFull` datasets ([#7275](https://github.com/pyg-team/pytorch_geometric/pull/7275)) - Added support for `torch.sparse.Tensor` in `DataLoader` ([#7252](https://github.com/pyg-team/pytorch_geometric/pull/7252)) - Added `save` and `load` methods to `InMemoryDataset` ([#7250](https://github.com/pyg-team/pytorch_geometric/pull/7250), [#7413](https://github.com/pyg-team/pytorch_geometric/pull/7413)) - Added an example for heterogeneous GNN explanation via `CaptumExplainer` ([#7096](https://github.com/pyg-team/pytorch_geometric/pull/7096)) - Added `visualize_feature_importance` functionality to `HeteroExplanation` ([#7096](https://github.com/pyg-team/pytorch_geometric/pull/7096)) - Added a `AddRemainingSelfLoops` transform ([#7192](https://github.com/pyg-team/pytorch_geometric/pull/7192)) - Added `optimizer_resolver` ([#7209](https://github.com/pyg-team/pytorch_geometric/pull/7209)) - Added `type_ptr` argument to `HeteroLayerNorm` ([#7208](https://github.com/pyg-team/pytorch_geometric/pull/7208)) - Added an option to benchmark scripts to write PyTorch profiler results to CSV ([#7114](https://github.com/pyg-team/pytorch_geometric/pull/7114)) - Added subgraph type sampling option with bidirectional edge support ([#7199](https://github.com/pyg-team/pytorch_geometric/pull/7199), [#7200](https://github.com/pyg-team/pytorch_geometric/pull/7200)) - Added support for `"any"`-reductions in `scatter` ([#7198](https://github.com/pyg-team/pytorch_geometric/pull/7198)) - Added manual sampling interface to `NodeLoader` and `LinkLoader` ([#7197](https://github.com/pyg-team/pytorch_geometric/pull/7197)) - Extending `torch.sparse` support ([#7155](https://github.com/pyg-team/pytorch_geometric/pull/7155)) - Added edge weight support to `LightGCN` ([#7157](https://github.com/pyg-team/pytorch_geometric/pull/7157)) - Added `SparseTensor` support to`trim_to_layer` function ([#7089](https://github.com/pyg-team/pytorch_geometric/pull/7089)) - Added instructions for ROCm build wheels ([#7143](https://github.com/pyg-team/pytorch_geometric/pull/7143)) - Added a `ComposeFilters` class to compose `pre_filter` functions in `Dataset` ([#7097](https://github.com/pyg-team/pytorch_geometric/pull/7097)) - Added a time-step aware variant of the `EllipticBitcoinDataset` called `EllipticBitcoinTemporalDataset` ([#7011](https://github.com/pyg-team/pytorch_geometric/pull/7011)) - Added `to_dgl` and `from_dgl` conversion functions ([#7053](https://github.com/pyg-team/pytorch_geometric/pull/7053)) - Added support for `torch.jit.script` within `MessagePassing` layers without `torch_sparse` being installed ([#7061](https://github.com/pyg-team/pytorch_geometric/pull/7061), [#7062](https://github.com/pyg-team/pytorch_geometric/pull/7062)) - Added unbatching logic for `torch.sparse` tensors ([#7037](https://github.com/pyg-team/pytorch_geometric/pull/7037)) - Added the `RotatE` KGE model ([#7026](https://github.com/pyg-team/pytorch_geometric/pull/7026)) ### Changed - Added a `num_edges` parameter to the forward method of `HypergraphConv` ([#7560](https://github.com/pyg-team/pytorch_geometric/pull/7560)) - Fixed `get_mesh_laplacian` for `normalization="sym"` ([#7544](https://github.com/pyg-team/pytorch_geometric/pull/7544)) - Use `dim_size` to initialize output size of the `EquilibriumAggregation` layer ([#7530](https://github.com/pyg-team/pytorch_geometric/pull/7530)) - Added a `max_num_elements` parameter to the forward method of `GraphMultisetTransformer`, `GRUAggregation`, `LSTMAggregation` and `SetTransformerAggregation` ([#7529](https://github.com/pyg-team/pytorch_geometric/pull/7529)) - Fixed empty edge indices handling in `SparseTensor` ([#7519](https://github.com/pyg-team/pytorch_geometric/pull/7519)) - Move the `scaler` tensor in `GeneralConv` to the correct device ([#7484](https://github.com/pyg-team/pytorch_geometric/pull/7484)) - Fixed `HeteroLinear` bug when used via mixed precision ([#7473](https://github.com/pyg-team/pytorch_geometric/pull/7473)) - All transforms are now immutable, i.e., they perform a shallow-copy of the data and therefore do not longer modify data in-place ([#7429](https://github.com/pyg-team/pytorch_geometric/pull/7429)) - Set `output_size` in the `repeat_interleave` operation in `QuantileAggregation` ([#7426](https://github.com/pyg-team/pytorch_geometric/pull/7426)) - Fixed gradient computation of edge weights in `utils.spmm` ([#7428](https://github.com/pyg-team/pytorch_geometric/pull/7428)) - Re-factored `ClusterLoader` to integrate `pyg-lib` METIS routine ([#7416](https://github.com/pyg-team/pytorch_geometric/pull/7416)) - Fixed an index-out-of-range bug in `QuantileAggregation` when `dim_size` is passed ([#7407](https://github.com/pyg-team/pytorch_geometric/pull/7407)) - The `filter_per_worker` option will not get automatically inferred by default based on the device of the underlying data ([#7399](https://github.com/pyg-team/pytorch_geometric/pull/7399)) - Fixed a bug in `LightGCN.recommendation_loss()` to only use the embeddings of the nodes involved in the current mini-batch ([#7384](https://github.com/pyg-team/pytorch_geometric/pull/7384)) - Added an optional `max_num_elements` argument to `SortAggregation` ([#7367](https://github.com/pyg-team/pytorch_geometric/pull/7367)) - Added the option to pass `fill_value` as a `torch.tensor` to `utils.to_dense_batch` ([#7367](https://github.com/pyg-team/pytorch_geometric/pull/7367)) - Fixed a bug in which inputs where modified in-place in `to_hetero_with_bases` ([#7363](https://github.com/pyg-team/pytorch_geometric/pull/7363)) - Do not load `node_default` and `edge_default` attributes in `from_networkx` ([#7348](https://github.com/pyg-team/pytorch_geometric/pull/7348)) - Updated examples to use `NeighborLoader` instead of `NeighborSampler` ([#7152](https://github.com/pyg-team/pytorch_geometric/pull/7152)) - Fixed `HGTConv` utility function `_construct_src_node_feat` ([#7194](https://github.com/pyg-team/pytorch_geometric/pull/7194)) - Extend dataset summary to create stats for each node/edge type ([#7203](https://github.com/pyg-team/pytorch_geometric/pull/7203)) - Added an optional `batch_size` argument to `avg_pool_x` and `max_pool_x` ([#7216](https://github.com/pyg-team/pytorch_geometric/pull/7216)) - Fixed `subgraph` on unordered inputs ([#7187](https://github.com/pyg-team/pytorch_geometric/pull/7187)) - Allow missing node types in `HeteroDictLinear` ([#7185](https://github.com/pyg-team/pytorch_geometric/pull/7185)) - Optimized `from_networkx` memory footprint by reducing unnecessary copies ([#7119](https://github.com/pyg-team/pytorch_geometric/pull/7119)) - Added an optional `batch_size` argument to `LayerNorm`, `GraphNorm`, `InstanceNorm`, `GraphSizeNorm` and `PairNorm` ([#7135](https://github.com/pyg-team/pytorch_geometric/pull/7135)) - Improved code coverage ([#7093](https://github.com/pyg-team/pytorch_geometric/pull/7093), [#7195](https://github.com/pyg-team/pytorch_geometric/pull/7195)) - Fix `numpy` incompatiblity when reading files for `Planetoid` datasets ([#7141](https://github.com/pyg-team/pytorch_geometric/pull/7141)) - Added support for `Data.num_edges` for native `torch.sparse.Tensor` adjacency matrices ([#7104](https://github.com/pyg-team/pytorch_geometric/pull/7104)) - Fixed crash of heterogeneous data loaders if node or edge types are missing ([#7060](https://github.com/pyg-team/pytorch_geometric/pull/7060), [#7087](https://github.com/pyg-team/pytorch_geometric/pull/7087)) - Accelerated attention-based `MultiAggregation` ([#7077](https://github.com/pyg-team/pytorch_geometric/pull/7077)) - Edges in `HeterophilousGraphDataset` are now undirected by default ([#7065](https://github.com/pyg-team/pytorch_geometric/pull/7065)) - Fixed a bug in `FastHGTConv` that computed values via parameters used to compute the keys ([#7050](https://github.com/pyg-team/pytorch_geometric/pull/7050)) - Accelerated sparse tensor conversion routines ([#7042](https://github.com/pyg-team/pytorch_geometric/pull/7042), [#7043](https://github.com/pyg-team/pytorch_geometric/pull/7043)) - Change `torch_sparse.SparseTensor` logic to utilize `torch.sparse_csr` instead ([#7041](https://github.com/pyg-team/pytorch_geometric/pull/7041)) - Added an optional `batch_size` and `max_num_nodes` arguments to `MemPooling` layer ([#7239](https://github.com/pyg-team/pytorch_geometric/pull/7239)) - Fixed training issues of the GraphGPS example ([#7377](https://github.com/pyg-team/pytorch_geometric/pull/7377)) - Allowed `CaptumExplainer` to be called multiple times in a row ([#7391](https://github.com/pyg-team/pytorch_geometric/pull/7391)) ### Removed - Removed `layer_type` argument in `contrib.explain.GraphMaskExplainer` ([#7445](https://github.com/pyg-team/pytorch_geometric/pull/7445)) - Replaced `FastHGTConv` with `HGTConv` ([#7117](https://github.com/pyg-team/pytorch_geometric/pull/7117)) ## [2.3.0] - 2023-03-23 ### Added - Added a memory-efficient `utils.one_hot` implementation ([#7005](https://github.com/pyg-team/pytorch_geometric/pull/7005)) - Added `HeteroDictLinear` and an optimized `FastHGTConv` module ([#6178](https://github.com/pyg-team/pytorch_geometric/pull/6178), [#6998](https://github.com/pyg-team/pytorch_geometric/pull/6998)) - Added the `DenseGATConv` module ([#6928](https://github.com/pyg-team/pytorch_geometric/pull/6928)) - Added `trim_to_layer` utility function for more efficient `NeighborLoader` use-cases ([#6661](https://github.com/pyg-team/pytorch_geometric/pull/6661)) - Added the `DistMult` KGE model ([#6958](https://github.com/pyg-team/pytorch_geometric/pull/6958)) - Added `HeteroData.set_value_dict` functionality ([#6961](https://github.com/pyg-team/pytorch_geometric/pull/6961), [#6974](https://github.com/pyg-team/pytorch_geometric/pull/6974)) - Added PyTorch >= 2.0 support ([#6934](https://github.com/pyg-team/pytorch_geometric/pull/6934), [#7000](https://github.com/pyg-team/pytorch_geometric/pull/7000)) - Added PyTorch Lightning >= 2.0 support ([#6929](https://github.com/pyg-team/pytorch_geometric/pull/6929)) - Added the `ComplEx` KGE model ([#6898](https://github.com/pyg-team/pytorch_geometric/pull/6898)) - Added option to write benchmark results to csv ([#6888](https://github.com/pyg-team/pytorch_geometric/pull/6888)) - Added `HeteroLayerNorm` and `HeteroBatchNorm` layers ([#6838](https://github.com/pyg-team/pytorch_geometric/pull/6838)) - Added the `HeterophilousGraphDataset` suite ([#6846](https://github.com/pyg-team/pytorch_geometric/pull/6846)) - Added support for sparse tensor in full batch mode inference benchmark ([#6843](https://github.com/pyg-team/pytorch_geometric/pull/6843)) - Enabled `NeighborLoader` to return number of sampled nodes and edges per hop ([#6834](https://github.com/pyg-team/pytorch_geometric/pull/6834)) - Added `ZipLoader` to execute multiple `NodeLoader` or `LinkLoader` instances ([#6829](https://github.com/pyg-team/pytorch_geometric/issues/6829)) - Added common `utils.select` and `utils.narrow` functionality to support filtering of both tensors and lists ([#6162](https://github.com/pyg-team/pytorch_geometric/issues/6162)) - Support `normalization` customization in `get_mesh_laplacian` ([#6790](https://github.com/pyg-team/pytorch_geometric/issues/6790)) - Added the `TemporalEncoding` module ([#6785](https://github.com/pyg-team/pytorch_geometric/pull/6785)) - Added CPU-optimized `spmm_reduce` functionality via CSR format ([#6699](https://github.com/pyg-team/pytorch_geometric/pull/6699), [#6759](https://github.com/pyg-team/pytorch_geometric/pull/6759)) - Added support for the revised version of the `MD17` dataset ([#6734](https://github.com/pyg-team/pytorch_geometric/pull/6734)) - Added TorchScript support to the `RECT_L` model ([#6727](https://github.com/pyg-team/pytorch_geometric/pull/6727)) - Added TorchScript support to the `Node2Vec` model ([#6726](https://github.com/pyg-team/pytorch_geometric/pull/6726)) - Added `utils.to_edge_index` to convert sparse tensors to edge indices and edge attributes ([#6728](https://github.com/pyg-team/pytorch_geometric/issues/6728)) - Fixed expected data format in `PolBlogs` dataset ([#6714](https://github.com/pyg-team/pytorch_geometric/issues/6714)) - Added `SimpleConv` to perform non-trainable propagation ([#6718](https://github.com/pyg-team/pytorch_geometric/pull/6718)) - Added a `RemoveDuplicatedEdges` transform ([#6709](https://github.com/pyg-team/pytorch_geometric/pull/6709)) - Added TorchScript support to the `LINKX` model ([#6712](https://github.com/pyg-team/pytorch_geometric/pull/6712)) - Added `torch.jit` examples for `example/film.py` and `example/gcn.py`([#6602](https://github.com/pyg-team/pytorch_geometric/pull/6692)) - Added `Pad` transform ([#5940](https://github.com/pyg-team/pytorch_geometric/pull/5940), [#6697](https://github.com/pyg-team/pytorch_geometric/pull/6697), [#6731](https://github.com/pyg-team/pytorch_geometric/pull/6731), [#6758](https://github.com/pyg-team/pytorch_geometric/pull/6758)) - Added full batch mode to the inference benchmark ([#6631](https://github.com/pyg-team/pytorch_geometric/pull/6631)) - Added `cat` aggregation type to the `HeteroConv` class so that features can be concatenated during grouping ([#6634](https://github.com/pyg-team/pytorch_geometric/pull/6634)) - Added `torch.compile` support and benchmark study ([#6610](https://github.com/pyg-team/pytorch_geometric/pull/6610), [#6952](https://github.com/pyg-team/pytorch_geometric/pull/6952), [#6953](https://github.com/pyg-team/pytorch_geometric/pull/6953), [#6980](https://github.com/pyg-team/pytorch_geometric/pull/6980), [#6983](https://github.com/pyg-team/pytorch_geometric/pull/6983), [#6984](https://github.com/pyg-team/pytorch_geometric/pull/6984), [#6985](https://github.com/pyg-team/pytorch_geometric/pull/6985), [#6986](https://github.com/pyg-team/pytorch_geometric/pull/6986), [#6989](https://github.com/pyg-team/pytorch_geometric/pull/6989), [#7002](https://github.com/pyg-team/pytorch_geometric/pull/7002)) - Added the `AntiSymmetricConv` layer ([#6577](https://github.com/pyg-team/pytorch_geometric/pull/6577)) - Added a mixin for Huggingface model hub integration ([#5930](https://github.com/pyg-team/pytorch_geometric/pull/5930), [#6591](https://github.com/pyg-team/pytorch_geometric/pull/6591)) - Added support for accelerated GNN layers in `nn.conv.cugraph` via `cugraph-ops` ([#6278](https://github.com/pyg-team/pytorch_geometric/pull/6278), [#6388](https://github.com/pyg-team/pytorch_geometric/pull/6388), [#6412](https://github.com/pyg-team/pytorch_geometric/pull/6412)) - Added accelerated `index_sort` function from `pyg-lib` for faster sorting ([#6554](https://github.com/pyg-team/pytorch_geometric/pull/6554)) - Fix incorrect device in `EquilibriumAggregration` ([#6560](https://github.com/pyg-team/pytorch_geometric/pull/6560)) - Added bipartite graph support in `dense_to_sparse()` ([#6546](https://github.com/pyg-team/pytorch_geometric/pull/6546)) - Add CPU affinity support for more data loaders ([#6534](https://github.com/pyg-team/pytorch_geometric/pull/6534), [#6922](https://github.com/pyg-team/pytorch_geometric/pull/6922)) - Added the `BAMultiShapesDataset` ([#6541](https://github.com/pyg-team/pytorch_geometric/pull/6541)) - Added the interfaces of a graph pooling framework ([#6540](https://github.com/pyg-team/pytorch_geometric/pull/6540)) - Added automatic `n_id` and `e_id` attributes to mini-batches produced by `NodeLoader` and `LinkLoader` ([#6524](https://github.com/pyg-team/pytorch_geometric/pull/6524)) - Added `PGMExplainer` to `torch_geometric.contrib` ([#6149](https://github.com/pyg-team/pytorch_geometric/pull/6149), [#6588](https://github.com/pyg-team/pytorch_geometric/pull/6588), [#6589](https://github.com/pyg-team/pytorch_geometric/pull/6589)) - Added a `NumNeighbors` helper class for specifying the number of neighbors when sampling ([#6501](https://github.com/pyg-team/pytorch_geometric/pull/6501), [#6505](https://github.com/pyg-team/pytorch_geometric/pull/6505), [#6690](https://github.com/pyg-team/pytorch_geometric/pull/6690)) - Added caching to `is_node_attr()` and `is_edge_attr()` calls ([#6492](https://github.com/pyg-team/pytorch_geometric/pull/6492)) - Added `ToHeteroLinear` and `ToHeteroMessagePassing` modules to accelerate `to_hetero` functionality ([#5992](https://github.com/pyg-team/pytorch_geometric/pull/5992), [#6456](https://github.com/pyg-team/pytorch_geometric/pull/6456)) - Added `GraphMaskExplainer` ([#6284](https://github.com/pyg-team/pytorch_geometric/pull/6284)) - Added the `GRBCD` and `PRBCD` adversarial attack models ([#5972](https://github.com/pyg-team/pytorch_geometric/pull/5972)) - Added `dropout` option to `SetTransformer` and `GraphMultisetTransformer` ([#6484](https://github.com/pyg-team/pytorch_geometric/pull/6484)) - Added option to customize loader arguments for evaluation in `LightningNodeData` and `LightningLinkData` ([#6450](https://github.com/pyg-team/pytorch_geometric/pull/6450), [#6456](https://github.com/pyg-team/pytorch_geometric/pull/6456)) - Added option to customize `num_neighbors` in `NeighborSampler` after instantiation ([#6446](https://github.com/pyg-team/pytorch_geometric/pull/6446)) - Added the `Taobao` dataset and a corresponding example for it ([#6144](https://github.com/pyg-team/pytorch_geometric/pull/6144)) - Added `pyproject.toml` ([#6431](https://github.com/pyg-team/pytorch_geometric/pull/6431)) - Added the `torch_geometric.contrib` sub-package ([#6422](https://github.com/pyg-team/pytorch_geometric/pull/6422)) - Warn on using latest documentation ([#6418](https://github.com/pyg-team/pytorch_geometric/pull/6418)) - Added basic `pyright` type checker support ([#6415](https://github.com/pyg-team/pytorch_geometric/pull/6415)) - Added a new external resource for link prediction ([#6396](https://github.com/pyg-team/pytorch_geometric/pull/6396)) - Added `CaptumExplainer` ([#6383](https://github.com/pyg-team/pytorch_geometric/pull/6383), [#6387](https://github.com/pyg-team/pytorch_geometric/pull/6387), [#6433](https://github.com/pyg-team/pytorch_geometric/pull/6433), [#6487](https://github.com/pyg-team/pytorch_geometric/pull/6487), [#6966](https://github.com/pyg-team/pytorch_geometric/pull/6966)) - Added support for custom `HeteroData` mini-batch class in remote backends ([#6377](https://github.com/pyg-team/pytorch_geometric/pull/6377)) - Added the `GNNFF` model ([#5866](https://github.com/pyg-team/pytorch_geometric/pull/5866)) - Added `MLPAggregation`, `SetTransformerAggregation`, `GRUAggregation`, and `DeepSetsAggregation` as adaptive readout functions ([#6301](https://github.com/pyg-team/pytorch_geometric/pull/6301), [#6336](https://github.com/pyg-team/pytorch_geometric/pull/6336), [#6338](https://github.com/pyg-team/pytorch_geometric/pull/6338)) - Added `Dataset.to_datapipe` for converting PyG datasets into a torchdata `DataPipe`([#6141](https://github.com/pyg-team/pytorch_geometric/pull/6141)) - Added `to_nested_tensor` and `from_nested_tensor` functionality ([#6329](https://github.com/pyg-team/pytorch_geometric/pull/6329), [#6330](https://github.com/pyg-team/pytorch_geometric/pull/6330), [#6331](https://github.com/pyg-team/pytorch_geometric/pull/6331), [#6332](https://github.com/pyg-team/pytorch_geometric/pull/6332)) - Added the `GPSConv` Graph Transformer layer and example ([#6326](https://github.com/pyg-team/pytorch_geometric/pull/6326), [#6327](https://github.com/pyg-team/pytorch_geometric/pull/6327)) - Added `networkit` conversion utilities ([#6321](https://github.com/pyg-team/pytorch_geometric/pull/6321)) - Added global dataset attribute access via `dataset.{attr_name}` ([#6319](https://github.com/pyg-team/pytorch_geometric/pull/6319)) - Added the `TransE` KGE model and example ([#6314](https://github.com/pyg-team/pytorch_geometric/pull/6314)) - Added the Freebase `FB15k_237` dataset ([#3204](https://github.com/pyg-team/pytorch_geometric/pull/3204)) - Added `Data.update()` and `HeteroData.update()` functionality ([#6313](https://github.com/pyg-team/pytorch_geometric/pull/6313)) - Added `PGExplainer` ([#6204](https://github.com/pyg-team/pytorch_geometric/pull/6204)) - Added the `AirfRANS` dataset ([#6287](https://github.com/pyg-team/pytorch_geometric/pull/6287)) - Added `AttentionExplainer` ([#6279](https://github.com/pyg-team/pytorch_geometric/pull/6279)) - Added (un)faithfulness explainability metric ([#6090](https://github.com/pyg-team/pytorch_geometric/pull/6090)) - Added fidelity explainability metric ([#6116](https://github.com/pyg-team/pytorch_geometric/pull/6116), [#6510](https://github.com/pyg-team/pytorch_geometric/pull/6510)) - Added subgraph visualization of GNN explanations ([#6235](https://github.com/pyg-team/pytorch_geometric/pull/6235), [#6271](https://github.com/pyg-team/pytorch_geometric/pull/6271)) - Added weighted negative sampling option in `LinkNeighborLoader` ([#6264](https://github.com/pyg-team/pytorch_geometric/pull/6264)) - Added the `BA2MotifDataset` explainer dataset ([#6257](https://github.com/pyg-team/pytorch_geometric/pull/6257)) - Added `CycleMotif` motif generator to generate `n`-node cycle shaped motifs ([#6256](https://github.com/pyg-team/pytorch_geometric/pull/6256)) - Added the `InfectionDataset` to evaluate explanations ([#6222](https://github.com/pyg-team/pytorch_geometric/pull/6222)) - Added `characterization_score` and `fidelity_curve_auc` explainer metrics ([#6188](https://github.com/pyg-team/pytorch_geometric/pull/6188)) - Added `get_message_passing_embeddings` ([#6201](https://github.com/pyg-team/pytorch_geometric/pull/6201)) - Added the `PointGNNConv` layer ([#6194](https://github.com/pyg-team/pytorch_geometric/pull/6194)) - Added `GridGraph` graph generator to generate grid graphs ([#6220](https://github.com/pyg-team/pytorch_geometric/pull/6220) - Added explainability metrics for when ground truth is available ([#6137](https://github.com/pyg-team/pytorch_geometric/pull/6137)) - Added `visualize_feature_importance` to support node feature visualizations ([#6094](https://github.com/pyg-team/pytorch_geometric/pull/6094)) - Added heterogeneous graph support to `Explanation` framework ([#6091](https://github.com/pyg-team/pytorch_geometric/pull/6091), [#6218](https://github.com/pyg-team/pytorch_geometric/pull/6218)) - Added a `CustomMotif` motif generator ([#6179](https://github.com/pyg-team/pytorch_geometric/pull/6179)) - Added `ERGraph` graph generator to generate Ergos-Renyi (ER) graphs ([#6073](https://github.com/pyg-team/pytorch_geometric/pull/6073)) - Added `BAGraph` graph generator to generate Barabasi-Albert graphs - the usage of `datasets.BAShapes` is now deprecated ([#6072](https://github.com/pyg-team/pytorch_geometric/pull/6072) - Added explainability benchmark dataset framework ([#6104](https://github.com/pyg-team/pytorch_geometric/pull/6104)) - Added `seed_time` attribute to temporal `NodeLoader` outputs in case `input_time` is given ([#6196](https://github.com/pyg-team/pytorch_geometric/pull/6196)) - Added `Data.edge_subgraph` and `HeteroData.edge_subgraph` functionalities ([#6193](https://github.com/pyg-team/pytorch_geometric/pull/6193)) - Added `input_time` option to `LightningNodeData` and `transform_sampler_output` to `NodeLoader` and `LinkLoader` ([#6187](https://github.com/pyg-team/pytorch_geometric/pull/6187)) - Added `summary` for PyG/PyTorch models ([#5859](https://github.com/pyg-team/pytorch_geometric/pull/5859), [#6161](https://github.com/pyg-team/pytorch_geometric/pull/6161)) - Started adding `torch.sparse` support to PyG ([#5906](https://github.com/pyg-team/pytorch_geometric/pull/5906), [#5944](https://github.com/pyg-team/pytorch_geometric/pull/5944), [#6003](https://github.com/pyg-team/pytorch_geometric/pull/6003), [#6033](https://github.com/pyg-team/pytorch_geometric/pull/6033), [#6514](https://github.com/pyg-team/pytorch_geometric/pull/6514), [#6532](https://github.com/pyg-team/pytorch_geometric/pull/6532), [#6748](https://github.com/pyg-team/pytorch_geometric/pull/6748), [#6847](https://github.com/pyg-team/pytorch_geometric/pull/6847), [#6868](https://github.com/pyg-team/pytorch_geometric/pull/6868), [#6874](https://github.com/pyg-team/pytorch_geometric/pull/6874), [#6897](https://github.com/pyg-team/pytorch_geometric/pull/6897), [#6930](https://github.com/pyg-team/pytorch_geometric/pull/6930), [#6932](https://github.com/pyg-team/pytorch_geometric/pull/6932), [#6936](https://github.com/pyg-team/pytorch_geometric/pull/6936), [#6937](https://github.com/pyg-team/pytorch_geometric/pull/6937), [#6939](https://github.com/pyg-team/pytorch_geometric/pull/6939), [#6947](https://github.com/pyg-team/pytorch_geometric/pull/6947), [#6950](https://github.com/pyg-team/pytorch_geometric/pull/6950), [#6951](https://github.com/pyg-team/pytorch_geometric/pull/6951), [#6957](https://github.com/pyg-team/pytorch_geometric/pull/6957)) - Add `inputs_channels` back in training benchmark ([#6154](https://github.com/pyg-team/pytorch_geometric/pull/6154)) - Added support for dropping nodes in `utils.to_dense_batch` in case `max_num_nodes` is smaller than the number of nodes ([#6124](https://github.com/pyg-team/pytorch_geometric/pull/6124)) - Added the RandLA-Net architecture as an example ([#5117](https://github.com/pyg-team/pytorch_geometric/pull/5117)) ### Changed - Migrate to `pyproject.toml` for packaging ([#6880](https://github.com/pyg-team/pytorch_geometric/pull/6880)) - Drop internal usage of `__dunder__` names ([#6999](https://github.com/pyg-team/pytorch_geometric/issues/6999)) - Changed the interface of `sort_edge_index`, `coalesce` and `to_undirected` to only return single `edge_index` information in case the `edge_attr` argument is not specified ([#6875](https://github.com/pyg-team/pytorch_geometric/issues/6875), [#6879](https://github.com/pyg-team/pytorch_geometric/issues/6879), [#6893](https://github.com/pyg-team/pytorch_geometric/issues/6893)) - Fixed a bug in `to_hetero` when using an uninitialized submodule without implementing `reset_parameters` ([#6863](https://github.com/pyg-team/pytorch_geometric/issues/6790)) - Fixed a bug in `get_mesh_laplacian` ([#6790](https://github.com/pyg-team/pytorch_geometric/issues/6790)) - Fixed a bug in which masks were not properly masked in `GNNExplainer` on link prediction tasks ([#6787](https://github.com/pyg-team/pytorch_geometric/pull/6787)) - Allow the usage of `ChebConv` within `GNNExplainer` ([#6778](https://github.com/pyg-team/pytorch_geometric/pull/6778)) - Allow setting the `EdgeStorage.num_edges` property ([#6710](https://github.com/pyg-team/pytorch_geometric/pull/6710)) - Fixed a bug in `utils.bipartite_subgraph()` and updated docs of `HeteroData.subgraph()` ([#6654](https://github.com/pyg-team/pytorch_geometric/pull/6654)) - Properly reset the `data_list` cache of an `InMemoryDataset` when accessing `dataset.data` ([#6685](https://github.com/pyg-team/pytorch_geometric/pull/6685)) - Fixed a bug in `Data.subgraph()` and `HeteroData.subgraph()` ([#6613](https://github.com/pyg-team/pytorch_geometric/pull/6613)) - Fixed a bug in `PNAConv` and `DegreeScalerAggregation` to correctly incorporate degree statistics of isolated nodes ([#6609](https://github.com/pyg-team/pytorch_geometric/pull/6609)) - Improved code coverage ([#6523](https://github.com/pyg-team/pytorch_geometric/pull/6523), [#6538](https://github.com/pyg-team/pytorch_geometric/pull/6538), [#6555](https://github.com/pyg-team/pytorch_geometric/pull/6555), [#6558](https://github.com/pyg-team/pytorch_geometric/pull/6558), [#6568](https://github.com/pyg-team/pytorch_geometric/pull/6568), [#6573](https://github.com/pyg-team/pytorch_geometric/pull/6573), [#6578](https://github.com/pyg-team/pytorch_geometric/pull/6578), [#6597](https://github.com/pyg-team/pytorch_geometric/pull/6597), [#6600](https://github.com/pyg-team/pytorch_geometric/pull/6600), [#6618](https://github.com/pyg-team/pytorch_geometric/pull/6618), [#6619](https://github.com/pyg-team/pytorch_geometric/pull/6619), [#6621](https://github.com/pyg-team/pytorch_geometric/pull/6621), [#6623](https://github.com/pyg-team/pytorch_geometric/pull/6623), [#6637](https://github.com/pyg-team/pytorch_geometric/pull/6637), [#6638](https://github.com/pyg-team/pytorch_geometric/pull/6638), [#6640](https://github.com/pyg-team/pytorch_geometric/pull/6640), [#6645](https://github.com/pyg-team/pytorch_geometric/pull/6645), [#6648](https://github.com/pyg-team/pytorch_geometric/pull/6648), [#6647](https://github.com/pyg-team/pytorch_geometric/pull/6647), [#6653](https://github.com/pyg-team/pytorch_geometric/pull/6653), [#6657](https://github.com/pyg-team/pytorch_geometric/pull/6657), [#6662](https://github.com/pyg-team/pytorch_geometric/pull/6662), [#6664](https://github.com/pyg-team/pytorch_geometric/pull/6664), [#6667](https://github.com/pyg-team/pytorch_geometric/pull/6667), [#6668](https://github.com/pyg-team/pytorch_geometric/pull/6668), [#6669](https://github.com/pyg-team/pytorch_geometric/pull/6669), [#6670](https://github.com/pyg-team/pytorch_geometric/pull/6670), [#6671](https://github.com/pyg-team/pytorch_geometric/pull/6671), [#6673](https://github.com/pyg-team/pytorch_geometric/pull/6673), [#6675](https://github.com/pyg-team/pytorch_geometric/pull/6675), [#6676](https://github.com/pyg-team/pytorch_geometric/pull/6676), [#6677](https://github.com/pyg-team/pytorch_geometric/pull/6677), [#6678](https://github.com/pyg-team/pytorch_geometric/pull/6678), [#6681](https://github.com/pyg-team/pytorch_geometric/pull/6681), [#6683](https://github.com/pyg-team/pytorch_geometric/pull/6683), [#6703](https://github.com/pyg-team/pytorch_geometric/pull/6703), [#6720](https://github.com/pyg-team/pytorch_geometric/pull/6720), [#6735](https://github.com/pyg-team/pytorch_geometric/pull/6735), [#6736](https://github.com/pyg-team/pytorch_geometric/pull/6736), [#6763](https://github.com/pyg-team/pytorch_geometric/pull/6763), [#6781](https://github.com/pyg-team/pytorch_geometric/pull/6781), [#6797](https://github.com/pyg-team/pytorch_geometric/pull/6797), [#6799](https://github.com/pyg-team/pytorch_geometric/pull/6799), [#6824](https://github.com/pyg-team/pytorch_geometric/pull/6824), [#6858](https://github.com/pyg-team/pytorch_geometric/pull/6858)) - Fixed a bug in which `data.to_heterogeneous()` filtered attributs in the wrong dimension ([#6522](https://github.com/pyg-team/pytorch_geometric/pull/6522)) - Breaking Change: Temporal sampling will now also sample nodes with an equal timestamp to the seed time (requires `pyg-lib>0.1.0`) ([#6517](https://github.com/pyg-team/pytorch_geometric/pull/6517)) - Changed `DataLoader` workers with affinity to start at `cpu0` ([#6512](https://github.com/pyg-team/pytorch_geometric/pull/6512)) - Allow 1D input to `global_*_pool` functions ([#6504](https://github.com/pyg-team/pytorch_geometric/pull/6504)) - Add information about dynamic shapes in `RGCNConv` ([#6482](https://github.com/pyg-team/pytorch_geometric/pull/6482)) - Fixed the use of types removed in `numpy 1.24.0` ([#6495](https://github.com/pyg-team/pytorch_geometric/pull/6495)) - Fixed keyword parameters in `examples/mnist_voxel_grid.py` ([#6478](https://github.com/pyg-team/pytorch_geometric/pull/6478)) - Unified `LightningNodeData` and `LightningLinkData` code paths ([#6473](https://github.com/pyg-team/pytorch_geometric/pull/6473)) - Allow indices with any integer type in `RGCNConv` ([#6463](https://github.com/pyg-team/pytorch_geometric/pull/6463)) - Re-structured the documentation ([#6420](https://github.com/pyg-team/pytorch_geometric/pull/6420), [#6423](https://github.com/pyg-team/pytorch_geometric/pull/6423), [#6429](https://github.com/pyg-team/pytorch_geometric/pull/6429), [#6440](https://github.com/pyg-team/pytorch_geometric/pull/6440), [#6443](https://github.com/pyg-team/pytorch_geometric/pull/6443), [#6445](https://github.com/pyg-team/pytorch_geometric/pull/6445), [#6452](https://github.com/pyg-team/pytorch_geometric/pull/6452), [#6453](https://github.com/pyg-team/pytorch_geometric/pull/6453), [#6458](https://github.com/pyg-team/pytorch_geometric/pull/6458), [#6459](https://github.com/pyg-team/pytorch_geometric/pull/6459), [#6460](https://github.com/pyg-team/pytorch_geometric/pull/6460), [#6490](https://github.com/pyg-team/pytorch_geometric/pull/6490), [#6491](https://github.com/pyg-team/pytorch_geometric/pull/6491), [#6693](https://github.com/pyg-team/pytorch_geometric/pull/6693), [#6744](https://github.com/pyg-team/pytorch_geometric/pull/6744)) - Fix the default arguments of `DataParallel` class ([#6376](https://github.com/pyg-team/pytorch_geometric/pull/6376)) - Fix `ImbalancedSampler` on sliced `InMemoryDataset` ([#6374](https://github.com/pyg-team/pytorch_geometric/pull/6374)) - Breaking Change: Changed the interface and implementation of `GraphMultisetTransformer` ([#6343](https://github.com/pyg-team/pytorch_geometric/pull/6343)) - Fixed the approximate PPR variant in `transforms.GDC` to not crash on graphs with isolated nodes ([#6242](https://github.com/pyg-team/pytorch_geometric/pull/6242)) - Added a warning when accesing `InMemoryDataset.data` ([#6318](https://github.com/pyg-team/pytorch_geometric/pull/6318)) - Drop `SparseTensor` dependency in `GraphStore` ([#5517](https://github.com/pyg-team/pytorch_geometric/pull/5517)) - Replace `NeighborSampler` with `NeighborLoader` in the distributed sampling example ([#6204](https://github.com/pyg-team/pytorch_geometric/pull/6307)) - Fixed the filtering of node features in `transforms.RemoveIsolatedNodes` ([#6308](https://github.com/pyg-team/pytorch_geometric/pull/6308)) - Fixed a bug in `DimeNet` that causes a output dimension mismatch ([#6305](https://github.com/pyg-team/pytorch_geometric/pull/6305)) - Fixed `Data.to_heterogeneous()` with empty `edge_index` ([#6304](https://github.com/pyg-team/pytorch_geometric/pull/6304)) - Unify `Explanation.node_mask` and `Explanation.node_feat_mask` ([#6267](https://github.com/pyg-team/pytorch_geometric/pull/6267)) - Moved thresholding config of the `Explainer` to `Explanation` ([#6215](https://github.com/pyg-team/pytorch_geometric/pull/6215)) - Fixed a bug in the output order in `HeteroLinear` for un-sorted type vectors ([#6198](https://github.com/pyg-team/pytorch_geometric/pull/6198)) - Breaking Change: Move `ExplainerConfig` arguments to the `Explainer` class ([#6176](https://github.com/pyg-team/pytorch_geometric/pull/6176)) - Refactored `NeighborSampler` to be input-type agnostic ([#6173](https://github.com/pyg-team/pytorch_geometric/pull/6173)) - Infer correct CUDA device ID in `profileit` decorator ([#6164](https://github.com/pyg-team/pytorch_geometric/pull/6164)) - Correctly use edge weights in `GDC` example ([#6159](https://github.com/pyg-team/pytorch_geometric/pull/6159)) - Breaking Change: Moved PyTorch Lightning data modules to `torch_geometric.data.lightning` ([#6140](https://github.com/pyg-team/pytorch_geometric/pull/6140)) - Make `torch_sparse` an optional dependency ([#6132](https://github.com/pyg-team/pytorch_geometric/pull/6132), [#6134](https://github.com/pyg-team/pytorch_geometric/pull/6134), [#6138](https://github.com/pyg-team/pytorch_geometric/pull/6138), [#6139](https://github.com/pyg-team/pytorch_geometric/pull/6139), [#7387](https://github.com/pyg-team/pytorch_geometric/pull/7387)) - Optimized `utils.softmax` implementation ([#6113](https://github.com/pyg-team/pytorch_geometric/pull/6113), [#6155](https://github.com/pyg-team/pytorch_geometric/pull/6155), [#6805](https://github.com/pyg-team/pytorch_geometric/pull/6805)) - Optimized `topk` implementation for large enough graphs ([#6123](https://github.com/pyg-team/pytorch_geometric/pull/6123)) ### Removed - `torch-sparse` is now an optional dependency ([#6625](https://github.com/pyg-team/pytorch_geometric/pull/6625), [#6626](https://github.com/pyg-team/pytorch_geometric/pull/6626), [#6627](https://github.com/pyg-team/pytorch_geometric/pull/6627), [#6628](https://github.com/pyg-team/pytorch_geometric/pull/6628), [#6629](https://github.com/pyg-team/pytorch_geometric/pull/6629), [#6630](https://github.com/pyg-team/pytorch_geometric/pull/6630)) - Removed most of the `torch-scatter` dependencies ([#6394](https://github.com/pyg-team/pytorch_geometric/pull/6394), [#6395](https://github.com/pyg-team/pytorch_geometric/pull/6395), [#6399](https://github.com/pyg-team/pytorch_geometric/pull/6399), [#6400](https://github.com/pyg-team/pytorch_geometric/pull/6400), [#6615](https://github.com/pyg-team/pytorch_geometric/pull/6615), [#6617](https://github.com/pyg-team/pytorch_geometric/pull/6617)) - Removed the deprecated classes `GNNExplainer` and `Explainer` from `nn.models` ([#6382](https://github.com/pyg-team/pytorch_geometric/pull/6382)) - Removed `target_index` argument in the `Explainer` interface ([#6270](https://github.com/pyg-team/pytorch_geometric/pull/6270)) - Removed `Aggregation.set_validate_args` option ([#6175](https://github.com/pyg-team/pytorch_geometric/pull/6175)) ## [2.2.0] - 2022-12-01 ### Added - Extended `GNNExplainer` to support edge level explanations ([#6056](https://github.com/pyg-team/pytorch_geometric/pull/6056), [#6083](https://github.com/pyg-team/pytorch_geometric/pull/6083)) - Added CPU affinitization for `NodeLoader` ([#6005](https://github.com/pyg-team/pytorch_geometric/pull/6005)) - Added triplet sampling in `LinkNeighborLoader` ([#6004](https://github.com/pyg-team/pytorch_geometric/pull/6004)) - Added `FusedAggregation` of simple scatter reductions ([#6036](https://github.com/pyg-team/pytorch_geometric/pull/6036)) - Added a `to_smiles` function ([#6038](https://github.com/pyg-team/pytorch_geometric/pull/6038)) - Added option to make normalization coefficients trainable in `PNAConv` ([#6039](https://github.com/pyg-team/pytorch_geometric/pull/6039)) - Added `semi_grad` option in `VarAggregation` and `StdAggregation` ([#6042](https://github.com/pyg-team/pytorch_geometric/pull/6042)) - Allow for fused aggregations in `MultiAggregation` ([#6036](https://github.com/pyg-team/pytorch_geometric/pull/6036), [#6040](https://github.com/pyg-team/pytorch_geometric/pull/6040)) - Added `HeteroData` support for `to_captum_model` and added `to_captum_input` ([#5934](https://github.com/pyg-team/pytorch_geometric/pull/5934)) - Added `HeteroData` support in `RandomNodeLoader` ([#6007](https://github.com/pyg-team/pytorch_geometric/pull/6007)) - Added bipartite `GraphSAGE` example ([#5834](https://github.com/pyg-team/pytorch_geometric/pull/5834)) - Added `LRGBDataset` to include 5 datasets from the [Long Range Graph Benchmark](https://openreview.net/pdf?id=in7XC5RcjEn) ([#5935](https://github.com/pyg-team/pytorch_geometric/pull/5935)) - Added a warning for invalid node and edge type names in `HeteroData` ([#5990](https://github.com/pyg-team/pytorch_geometric/pull/5990)) - Added PyTorch 1.13 support ([#5975](https://github.com/pyg-team/pytorch_geometric/pull/5975)) - Added `int32` support in `NeighborLoader` ([#5948](https://github.com/pyg-team/pytorch_geometric/pull/5948)) - Add `dgNN` support and `FusedGATConv` implementation ([#5140](https://github.com/pyg-team/pytorch_geometric/pull/5140)) - Added `lr_scheduler_solver` and customized `lr_scheduler` classes ([#5942](https://github.com/pyg-team/pytorch_geometric/pull/5942)) - Add `to_fixed_size` graph transformer ([#5939](https://github.com/pyg-team/pytorch_geometric/pull/5939)) - Add support for symbolic tracing of `SchNet` model ([#5938](https://github.com/pyg-team/pytorch_geometric/pull/5938)) - Add support for customizable interaction graph in `SchNet` model ([#5919](https://github.com/pyg-team/pytorch_geometric/pull/5919)) - Started adding `torch.sparse` support to PyG ([#5906](https://github.com/pyg-team/pytorch_geometric/pull/5906), [#5944](https://github.com/pyg-team/pytorch_geometric/pull/5944), [#6003](https://github.com/pyg-team/pytorch_geometric/pull/6003), [#6633](https://github.com/pyg-team/pytorch_geometric/pull/6633)) - Added `HydroNet` water cluster dataset ([#5537](https://github.com/pyg-team/pytorch_geometric/pull/5537), [#5902](https://github.com/pyg-team/pytorch_geometric/pull/5902), [#5903](https://github.com/pyg-team/pytorch_geometric/pull/5903)) - Added explainability support for heterogeneous GNNs ([#5886](https://github.com/pyg-team/pytorch_geometric/pull/5886)) - Added `SparseTensor` support to `SuperGATConv` ([#5888](https://github.com/pyg-team/pytorch_geometric/pull/5888)) - Added TorchScript support for `AttentiveFP `([#5868](https://github.com/pyg-team/pytorch_geometric/pull/5868)) - Added `num_steps` argument to training and inference benchmarks ([#5898](https://github.com/pyg-team/pytorch_geometric/pull/5898)) - Added `torch.onnx.export` support ([#5877](https://github.com/pyg-team/pytorch_geometric/pull/5877), [#5997](https://github.com/pyg-team/pytorch_geometric/pull/5997)) - Enable VTune ITT in inference and training benchmarks ([#5830](https://github.com/pyg-team/pytorch_geometric/pull/5830), [#5878](https://github.com/pyg-team/pytorch_geometric/pull/5878)) - Add training benchmark ([#5774](https://github.com/pyg-team/pytorch_geometric/pull/5774)) - Added a "Link Prediction on MovieLens" Colab notebook ([#5823](https://github.com/pyg-team/pytorch_geometric/pull/5823)) - Added custom `sampler` support in `LightningDataModule` ([#5820](https://github.com/pyg-team/pytorch_geometric/pull/5820)) - Added a `return_semantic_attention_weights` argument `HANConv` ([#5787](https://github.com/pyg-team/pytorch_geometric/pull/5787)) - Added `disjoint` argument to `NeighborLoader` and `LinkNeighborLoader` ([#5775](https://github.com/pyg-team/pytorch_geometric/pull/5775)) - Added support for `input_time` in `NeighborLoader` ([#5763](https://github.com/pyg-team/pytorch_geometric/pull/5763)) - Added `disjoint` mode for temporal `LinkNeighborLoader` ([#5717](https://github.com/pyg-team/pytorch_geometric/pull/5717)) - Added `HeteroData` support for `transforms.Constant` ([#5700](https://github.com/pyg-team/pytorch_geometric/pull/5700)) - Added `np.memmap` support in `NeighborLoader` ([#5696](https://github.com/pyg-team/pytorch_geometric/pull/5696)) - Added `assortativity` that computes degree assortativity coefficient ([#5587](https://github.com/pyg-team/pytorch_geometric/pull/5587)) - Added `SSGConv` layer ([#5599](https://github.com/pyg-team/pytorch_geometric/pull/5599)) - Added `shuffle_node`, `mask_feature` and `add_random_edge` augmentation methdos ([#5548](https://github.com/pyg-team/pytorch_geometric/pull/5548)) - Added `dropout_path` augmentation that drops edges from a graph based on random walks ([#5531](https://github.com/pyg-team/pytorch_geometric/pull/5531)) - Add support for filling labels with dummy values in `HeteroData.to_homogeneous()` ([#5540](https://github.com/pyg-team/pytorch_geometric/pull/5540)) - Added `temporal_strategy` option to `neighbor_sample` ([#5576](https://github.com/pyg-team/pyg-lib/pull/5576)) - Added `torch_geometric.sampler` package to docs ([#5563](https://github.com/pyg-team/pytorch_geometric/pull/5563)) - Added the `DGraphFin` dynamic graph dataset ([#5504](https://github.com/pyg-team/pytorch_geometric/pull/5504)) - Added `dropout_edge` augmentation that randomly drops edges from a graph - the usage of `dropout_adj` is now deprecated ([#5495](https://github.com/pyg-team/pytorch_geometric/pull/5495)) - Added `dropout_node` augmentation that randomly drops nodes from a graph ([#5481](https://github.com/pyg-team/pytorch_geometric/pull/5481)) - Added `AddRandomMetaPaths` that adds edges based on random walks along a metapath ([#5397](https://github.com/pyg-team/pytorch_geometric/pull/5397)) - Added `WLConvContinuous` for performing WL refinement with continuous attributes ([#5316](https://github.com/pyg-team/pytorch_geometric/pull/5316)) - Added `print_summary` method for the `torch_geometric.data.Dataset` interface ([#5438](https://github.com/pyg-team/pytorch_geometric/pull/5438)) - Added `sampler` support to `LightningDataModule` ([#5456](https://github.com/pyg-team/pytorch_geometric/pull/5456), [#5457](https://github.com/pyg-team/pytorch_geometric/pull/5457)) - Added official splits to `MalNetTiny` dataset ([#5078](https://github.com/pyg-team/pytorch_geometric/pull/5078)) - Added `IndexToMask` and `MaskToIndex` transforms ([#5375](https://github.com/pyg-team/pytorch_geometric/pull/5375), [#5455](https://github.com/pyg-team/pytorch_geometric/pull/5455)) - Added `FeaturePropagation` transform ([#5387](https://github.com/pyg-team/pytorch_geometric/pull/5387)) - Added `PositionalEncoding` ([#5381](https://github.com/pyg-team/pytorch_geometric/pull/5381)) - Consolidated sampler routines behind `torch_geometric.sampler`, enabling ease of extensibility in the future ([#5312](https://github.com/pyg-team/pytorch_geometric/pull/5312), [#5365](https://github.com/pyg-team/pytorch_geometric/pull/5365), [#5402](https://github.com/pyg-team/pytorch_geometric/pull/5402), [#5404](https://github.com/pyg-team/pytorch_geometric/pull/5404)), [#5418](https://github.com/pyg-team/pytorch_geometric/pull/5418)) - Added `pyg-lib` neighbor sampling ([#5384](https://github.com/pyg-team/pytorch_geometric/pull/5384), [#5388](https://github.com/pyg-team/pytorch_geometric/pull/5388)) - Added `pyg_lib.segment_matmul` integration within `HeteroLinear` ([#5330](https://github.com/pyg-team/pytorch_geometric/pull/5330), [#5347](https://github.com/pyg-team/pytorch_geometric/pull/5347))) - Enabled `bf16` support in benchmark scripts ([#5293](https://github.com/pyg-team/pytorch_geometric/pull/5293), [#5341](https://github.com/pyg-team/pytorch_geometric/pull/5341)) - Added `Aggregation.set_validate_args` option to skip validation of `dim_size` ([#5290](https://github.com/pyg-team/pytorch_geometric/pull/5290)) - Added `SparseTensor` support to inference and training benchmark suite ([#5242](https://github.com/pyg-team/pytorch_geometric/pull/5242), [#5258](https://github.com/pyg-team/pytorch_geometric/pull/5258), [#5881](https://github.com/pyg-team/pytorch_geometric/pull/5881)) - Added experimental mode in inference benchmarks ([#5254](https://github.com/pyg-team/pytorch_geometric/pull/5254)) - Added node classification example instrumented with [Weights and Biases (W&B) logging](https://wandb.com) and [W&B Sweeps](https://wandb.com/sweeps) ([#5192](https://github.com/pyg-team/pytorch_geometric/pull/5192)) - Added experimental mode for `utils.scatter` ([#5232](https://github.com/pyg-team/pytorch_geometric/pull/5232), [#5241](https://github.com/pyg-team/pytorch_geometric/pull/5241), [#5386](https://github.com/pyg-team/pytorch_geometric/pull/5386)) - Added missing test labels in `HGBDataset` ([#5233](https://github.com/pyg-team/pytorch_geometric/pull/5233)) - Added `BaseStorage.get()` functionality ([#5240](https://github.com/pyg-team/pytorch_geometric/pull/5240)) - Added a test to confirm that `to_hetero` works with `SparseTensor` ([#5222](https://github.com/pyg-team/pytorch_geometric/pull/5222)) - Added `torch_geometric.explain` module with base functionality for explainability methods ([#5804](https://github.com/pyg-team/pytorch_geometric/pull/5804), [#6054](https://github.com/pyg-team/pytorch_geometric/pull/6054), [#6089](https://github.com/pyg-team/pytorch_geometric/pull/6089)) ### Changed - Moved and adapted `GNNExplainer` from `torch_geometric.nn` to `torch_geometric.explain.algorithm` ([#5967](https://github.com/pyg-team/pytorch_geometric/pull/5967), [#6065](https://github.com/pyg-team/pytorch_geometric/pull/6065)) - Optimized scatter implementations for CPU/GPU, both with and without backward computation ([#6051](https://github.com/pyg-team/pytorch_geometric/pull/6051), [#6052](https://github.com/pyg-team/pytorch_geometric/pull/6052)) - Support temperature value in `dense_mincut_pool` ([#5908](https://github.com/pyg-team/pytorch_geometric/pull/5908)) - Fixed a bug in which `VirtualNode` mistakenly treated node features as edge features ([#5819](https://github.com/pyg-team/pytorch_geometric/pull/5819)) - Fixed `setter` and `getter` handling in `BaseStorage` ([#5815](https://github.com/pyg-team/pytorch_geometric/pull/5815)) - Fixed `path` in `hetero_conv_dblp.py` example ([#5686](https://github.com/pyg-team/pytorch_geometric/pull/5686)) - Fix `auto_select_device` routine in GraphGym for PyTorch Lightning>=1.7 ([#5677](https://github.com/pyg-team/pytorch_geometric/pull/5677)) - Support `in_channels` with `tuple` in `GENConv` for bipartite message passing ([#5627](https://github.com/pyg-team/pytorch_geometric/pull/5627), [#5641](https://github.com/pyg-team/pytorch_geometric/pull/5641)) - Handle cases of not having enough possible negative edges in `RandomLinkSplit` ([#5642](https://github.com/pyg-team/pytorch_geometric/pull/5642)) - Fix `RGCN+pyg-lib` for `LongTensor` input ([#5610](https://github.com/pyg-team/pytorch_geometric/pull/5610)) - Improved type hint support ([#5842](https://github.com/pyg-team/pytorch_geometric/pull/5842), [#5603](https://github.com/pyg-team/pytorch_geometric/pull/5603), [#5659](https://github.com/pyg-team/pytorch_geometric/pull/5659), [#5664](https://github.com/pyg-team/pytorch_geometric/pull/5664), [#5665](https://github.com/pyg-team/pytorch_geometric/pull/5665), [#5666](https://github.com/pyg-team/pytorch_geometric/pull/5666), [#5667](https://github.com/pyg-team/pytorch_geometric/pull/5667), [#5668](https://github.com/pyg-team/pytorch_geometric/pull/5668), [#5669](https://github.com/pyg-team/pytorch_geometric/pull/5669), [#5673](https://github.com/pyg-team/pytorch_geometric/pull/5673), [#5675](https://github.com/pyg-team/pytorch_geometric/pull/5675), [#5673](https://github.com/pyg-team/pytorch_geometric/pull/5676), [#5678](https://github.com/pyg-team/pytorch_geometric/pull/5678), [#5682](https://github.com/pyg-team/pytorch_geometric/pull/5682), [#5683](https://github.com/pyg-team/pytorch_geometric/pull/5683), [#5684](https://github.com/pyg-team/pytorch_geometric/pull/5684), [#5685](https://github.com/pyg-team/pytorch_geometric/pull/5685), [#5687](https://github.com/pyg-team/pytorch_geometric/pull/5687), [#5688](https://github.com/pyg-team/pytorch_geometric/pull/5688), [#5695](https://github.com/pyg-team/pytorch_geometric/pull/5695), [#5699](https://github.com/pyg-team/pytorch_geometric/pull/5699), [#5701](https://github.com/pyg-team/pytorch_geometric/pull/5701), [#5702](https://github.com/pyg-team/pytorch_geometric/pull/5702), [#5703](https://github.com/pyg-team/pytorch_geometric/pull/5703), [#5706](https://github.com/pyg-team/pytorch_geometric/pull/5706), [#5707](https://github.com/pyg-team/pytorch_geometric/pull/5707), [#5710](https://github.com/pyg-team/pytorch_geometric/pull/5710), [#5714](https://github.com/pyg-team/pytorch_geometric/pull/5714), [#5715](https://github.com/pyg-team/pytorch_geometric/pull/5715), [#5716](https://github.com/pyg-team/pytorch_geometric/pull/5716), [#5722](https://github.com/pyg-team/pytorch_geometric/pull/5722), [#5724](https://github.com/pyg-team/pytorch_geometric/pull/5724), [#5725](https://github.com/pyg-team/pytorch_geometric/pull/5725), [#5726](https://github.com/pyg-team/pytorch_geometric/pull/5726), [#5729](https://github.com/pyg-team/pytorch_geometric/pull/5729), [#5730](https://github.com/pyg-team/pytorch_geometric/pull/5730), [#5731](https://github.com/pyg-team/pytorch_geometric/pull/5731), [#5732](https://github.com/pyg-team/pytorch_geometric/pull/5732), [#5733](https://github.com/pyg-team/pytorch_geometric/pull/5733), [#5743](https://github.com/pyg-team/pytorch_geometric/pull/5743), [#5734](https://github.com/pyg-team/pytorch_geometric/pull/5734), [#5735](https://github.com/pyg-team/pytorch_geometric/pull/5735), [#5736](https://github.com/pyg-team/pytorch_geometric/pull/5736), [#5737](https://github.com/pyg-team/pytorch_geometric/pull/5737), [#5738](https://github.com/pyg-team/pytorch_geometric/pull/5738), [#5747](https://github.com/pyg-team/pytorch_geometric/pull/5747), [#5752](https://github.com/pyg-team/pytorch_geometric/pull/5752), [#5753](https://github.com/pyg-team/pytorch_geometric/pull/5753), [#5754](https://github.com/pyg-team/pytorch_geometric/pull/5754), [#5756](https://github.com/pyg-team/pytorch_geometric/pull/5756), [#5757](https://github.com/pyg-team/pytorch_geometric/pull/5757), [#5758](https://github.com/pyg-team/pytorch_geometric/pull/5758), [#5760](https://github.com/pyg-team/pytorch_geometric/pull/5760), [#5766](https://github.com/pyg-team/pytorch_geometric/pull/5766), [#5767](https://github.com/pyg-team/pytorch_geometric/pull/5767), [#5768](https://github.com/pyg-team/pytorch_geometric/pull/5768)), [#5781](https://github.com/pyg-team/pytorch_geometric/pull/5781), [#5778](https://github.com/pyg-team/pytorch_geometric/pull/5778), [#5797](https://github.com/pyg-team/pytorch_geometric/pull/5797), [#5798](https://github.com/pyg-team/pytorch_geometric/pull/5798), [#5799](https://github.com/pyg-team/pytorch_geometric/pull/5799), [#5800](https://github.com/pyg-team/pytorch_geometric/pull/5800), [#5806](https://github.com/pyg-team/pytorch_geometric/pull/5806), [#5810](https://github.com/pyg-team/pytorch_geometric/pull/5810), [#5811](https://github.com/pyg-team/pytorch_geometric/pull/5811), [#5828](https://github.com/pyg-team/pytorch_geometric/pull/5828), [#5847](https://github.com/pyg-team/pytorch_geometric/pull/5847), [#5851](https://github.com/pyg-team/pytorch_geometric/pull/5851), [#5852](https://github.com/pyg-team/pytorch_geometric/pull/5852)) - Avoid modifying `mode_kwargs` in `MultiAggregation` ([#5601](https://github.com/pyg-team/pytorch_geometric/pull/5601)) - Changed `BatchNorm` to allow for batches of size one during training ([#5530](https://github.com/pyg-team/pytorch_geometric/pull/5530), [#5614](https://github.com/pyg-team/pytorch_geometric/pull/5614)) - Integrated better temporal sampling support by requiring that local neighborhoods are sorted according to time ([#5516](https://github.com/pyg-team/pytorch_geometric/issues/5516), [#5602](https://github.com/pyg-team/pytorch_geometric/issues/5602)) - Fixed a bug when applying several scalers with `PNAConv` ([#5514](https://github.com/pyg-team/pytorch_geometric/issues/5514)) - Allow `.` in `ParameterDict` key names ([#5494](https://github.com/pyg-team/pytorch_geometric/pull/5494)) - Renamed `drop_unconnected_nodes` to `drop_unconnected_node_types` and `drop_orig_edges` to `drop_orig_edge_types` in `AddMetapaths` ([#5490](https://github.com/pyg-team/pytorch_geometric/pull/5490)) - Improved `utils.scatter` performance by explicitly choosing better implementation for `add` and `mean` reduction ([#5399](https://github.com/pyg-team/pytorch_geometric/pull/5399)) - Fix `to_dense_adj` with empty `edge_index` ([#5476](https://github.com/pyg-team/pytorch_geometric/pull/5476)) - The `AttentionalAggregation` module can now be applied to compute attentin on a per-feature level ([#5449](https://github.com/pyg-team/pytorch_geometric/pull/5449)) - Ensure equal lenghts of `num_neighbors` across edge types in `NeighborLoader` ([#5444](https://github.com/pyg-team/pytorch_geometric/pull/5444)) - Fixed a bug in `TUDataset` in which node features were wrongly constructed whenever `node_attributes` only hold a single feature (_e.g._, in `PROTEINS`) ([#5441](https://github.com/pyg-team/pytorch_geometric/pull/5441)) - Breaking change: removed `num_neighbors` as an attribute of loader ([#5404](https://github.com/pyg-team/pytorch_geometric/pull/5404)) - `ASAPooling` is now jittable ([#5395](https://github.com/pyg-team/pytorch_geometric/pull/5395)) - Updated unsupervised `GraphSAGE` example to leverage `LinkNeighborLoader` ([#5317](https://github.com/pyg-team/pytorch_geometric/pull/5317)) - Replace in-place operations with out-of-place ones to align with `torch.scatter_reduce` API ([#5353](https://github.com/pyg-team/pytorch_geometric/pull/5353)) - Breaking bugfix: `PointTransformerConv` now correctly uses `sum` aggregation ([#5332](https://github.com/pyg-team/pytorch_geometric/pull/5332)) - Improve out-of-bounds error message in `MessagePassing` ([#5339](https://github.com/pyg-team/pytorch_geometric/pull/5339)) - Allow file names of a `Dataset` to be specified as either property and method ([#5338](https://github.com/pyg-team/pytorch_geometric/pull/5338)) - Fixed separating a list of `SparseTensor` within `InMemoryDataset` ([#5299](https://github.com/pyg-team/pytorch_geometric/pull/5299)) - Improved name resolving of normalization layers ([#5277](https://github.com/pyg-team/pytorch_geometric/pull/5277)) - Fail gracefully on `GLIBC` errors within `torch-spline-conv` ([#5276](https://github.com/pyg-team/pytorch_geometric/pull/5276)) - Fixed `Dataset.num_classes` in case a `transform` modifies `data.y` ([#5274](https://github.com/pyg-team/pytorch_geometric/pull/5274)) - Allow customization of the activation function within `PNAConv` ([#5262](https://github.com/pyg-team/pytorch_geometric/pull/5262)) - Do not fill `InMemoryDataset` cache on `dataset.num_features` ([#5264](https://github.com/pyg-team/pytorch_geometric/pull/5264)) - Changed tests relying on `dblp` datasets to instead use synthetic data ([#5250](https://github.com/pyg-team/pytorch_geometric/pull/5250)) - Fixed a bug for the initialization of activation function examples in `custom_graphgym` ([#5243](https://github.com/pyg-team/pytorch_geometric/pull/5243)) - Allow any integer tensors when checking edge_index input to message passing ([5281](https://github.com/pyg-team/pytorch_geometric/pull/5281)) ### Removed - Removed `scatter_reduce` option from experimental mode ([#5399](https://github.com/pyg-team/pytorch_geometric/pull/5399)) ## [2.1.0] - 2022-08-17 ### Added - Added the test for `DeepGCNLayer` ([#5704](https://github.com/pyg-team/pytorch_geometric/pull/5704)) - Allow `.` in `ModuleDict` key names ([#5227](https://github.com/pyg-team/pytorch_geometric/pull/5227)) - Added `edge_label_time` argument to `LinkNeighborLoader` ([#5137](https://github.com/pyg-team/pytorch_geometric/pull/5137), [#5173](https://github.com/pyg-team/pytorch_geometric/pull/5173)) - Let `ImbalancedSampler` accept `torch.Tensor` as input ([#5138](https://github.com/pyg-team/pytorch_geometric/pull/5138)) - Added `flow` argument to `gcn_norm` to correctly normalize the adjacency matrix in `GCNConv` ([#5149](https://github.com/pyg-team/pytorch_geometric/pull/5149)) - `NeighborSampler` supports graphs without edges ([#5072](https://github.com/pyg-team/pytorch_geometric/pull/5072)) - Added the `MeanSubtractionNorm` layer ([#5068](https://github.com/pyg-team/pytorch_geometric/pull/5068)) - Added `pyg_lib.segment_matmul` integration within `RGCNConv` ([#5052](https://github.com/pyg-team/pytorch_geometric/pull/5052), [#5096](https://github.com/pyg-team/pytorch_geometric/pull/5096)) - Support `SparseTensor` as edge label in `LightGCN` (#[5046](https://github.com/pyg-team/pytorch_geometric/issues/5046)) - Added support for `BasicGNN` models within `to_hetero` ([#5091](https://github.com/pyg-team/pytorch_geometric/pull/5091)) - Added support for computing weighted metapaths in `AddMetapaths` ([#5049](https://github.com/pyg-team/pytorch_geometric/pull/5049)) - Added inference benchmark suite ([#4915](https://github.com/pyg-team/pytorch_geometric/pull/4915)) - Added a dynamically sized batch sampler for filling a mini-batch with a variable number of samples up to a maximum size ([#4972](https://github.com/pyg-team/pytorch_geometric/pull/4972)) - Added fine grained options for setting `bias` and `dropout` per layer in the `MLP` model ([#4981](https://github.com/pyg-team/pytorch_geometric/pull/4981)) - Added `EdgeCNN` model ([#4991](https://github.com/pyg-team/pytorch_geometric/pull/4991)) - Added scalable `inference` mode in `BasicGNN` with layer-wise neighbor loading ([#4977](https://github.com/pyg-team/pytorch_geometric/pull/4977)) - Added inference benchmarks ([#4892](https://github.com/pyg-team/pytorch_geometric/pull/4892), [#5107](https://github.com/pyg-team/pytorch_geometric/pull/5107)) - Added PyTorch 1.12 support ([#4975](https://github.com/pyg-team/pytorch_geometric/pull/4975)) - Added `unbatch_edge_index` functionality for splitting an `edge_index` tensor according to a `batch` vector ([#4903](https://github.com/pyg-team/pytorch_geometric/pull/4903)) - Added node-wise normalization mode in `LayerNorm` ([#4944](https://github.com/pyg-team/pytorch_geometric/pull/4944)) - Added support for `normalization_resolver` ([#4926](https://github.com/pyg-team/pytorch_geometric/pull/4926), [#4951](https://github.com/pyg-team/pytorch_geometric/pull/4951), [#4958](https://github.com/pyg-team/pytorch_geometric/pull/4958), [#4959](https://github.com/pyg-team/pytorch_geometric/pull/4959)) - Added notebook tutorial for `torch_geometric.nn.aggr` package to documentation ([#4927](https://github.com/pyg-team/pytorch_geometric/pull/4927)) - Added support for `follow_batch` for lists or dictionaries of tensors ([#4837](https://github.com/pyg-team/pytorch_geometric/pull/4837)) - Added `Data.validate()` and `HeteroData.validate()` functionality ([#4885](https://github.com/pyg-team/pytorch_geometric/pull/4885)) - Added `LinkNeighborLoader` support to `LightningDataModule` ([#4868](https://github.com/pyg-team/pytorch_geometric/pull/4868)) - Added `predict()` support to the `LightningNodeData` module ([#4884](https://github.com/pyg-team/pytorch_geometric/pull/4884)) - Added `time_attr` argument to `LinkNeighborLoader` ([#4877](https://github.com/pyg-team/pytorch_geometric/pull/4877), [#4908](https://github.com/pyg-team/pytorch_geometric/pull/4908)) - Added a `filter_per_worker` argument to data loaders to allow filtering of data within sub-processes ([#4873](https://github.com/pyg-team/pytorch_geometric/pull/4873)) - Added a `NeighborLoader` benchmark script ([#4815](https://github.com/pyg-team/pytorch_geometric/pull/4815), [#4862](https://github.com/pyg-team/pytorch_geometric/pull/4862/files)) - Added support for `FeatureStore` and `GraphStore` in `NeighborLoader` ([#4817](https://github.com/pyg-team/pytorch_geometric/pull/4817), [#4851](https://github.com/pyg-team/pytorch_geometric/pull/4851), [#4854](https://github.com/pyg-team/pytorch_geometric/pull/4854), [#4856](https://github.com/pyg-team/pytorch_geometric/pull/4856), [#4857](https://github.com/pyg-team/pytorch_geometric/pull/4857), [#4882](https://github.com/pyg-team/pytorch_geometric/pull/4882), [#4883](https://github.com/pyg-team/pytorch_geometric/pull/4883), [#4929](https://github.com/pyg-team/pytorch_geometric/pull/4929), [#4992](https://github.com/pyg-team/pytorch_geometric/pull/4922), [#4962](https://github.com/pyg-team/pytorch_geometric/pull/4962), [#4968](https://github.com/pyg-team/pytorch_geometric/pull/4968), [#5037](https://github.com/pyg-team/pytorch_geometric/pull/5037), [#5088](https://github.com/pyg-team/pytorch_geometric/pull/5088), [#5270](https://github.com/pyg-team/pytorch_geometric/pull/5270), [#5307](https://github.com/pyg-team/pytorch_geometric/pull/5307), [#5318](https://github.com/pyg-team/pytorch_geometric/pull/5318)) - Added a `normalize` parameter to `dense_diff_pool` ([#4847](https://github.com/pyg-team/pytorch_geometric/pull/4847)) - Added `size=None` explanation to jittable `MessagePassing` modules in the documentation ([#4850](https://github.com/pyg-team/pytorch_geometric/pull/4850)) - Added documentation to the `DataLoaderIterator` class ([#4838](https://github.com/pyg-team/pytorch_geometric/pull/4838)) - Added `GraphStore` support to `Data` and `HeteroData` ([#4816](https://github.com/pyg-team/pytorch_geometric/pull/4816)) - Added `FeatureStore` support to `Data` and `HeteroData` ([#4807](https://github.com/pyg-team/pytorch_geometric/pull/4807), [#4853](https://github.com/pyg-team/pytorch_geometric/pull/4853)) - Added `FeatureStore` and `GraphStore` abstractions ([#4534](https://github.com/pyg-team/pytorch_geometric/pull/4534), [#4568](https://github.com/pyg-team/pytorch_geometric/pull/4568), [#5120](https://github.com/pyg-team/pytorch_geometric/pull/5120)) - Added support for dense aggregations in `global_*_pool` ([#4827](https://github.com/pyg-team/pytorch_geometric/pull/4827)) - Added Python version requirement ([#4825](https://github.com/pyg-team/pytorch_geometric/pull/4825)) - Added TorchScript support to `JumpingKnowledge` module ([#4805](https://github.com/pyg-team/pytorch_geometric/pull/4805)) - Added a `max_sample` argument to `AddMetaPaths` in order to tackle very dense metapath edges ([#4750](https://github.com/pyg-team/pytorch_geometric/pull/4750)) - Test `HANConv` with empty tensors ([#4756](https://github.com/pyg-team/pytorch_geometric/pull/4756), [#4841](https://github.com/pyg-team/pytorch_geometric/pull/4841)) - Added the `bias` vector to the `GCN` model definition in the "Create Message Passing Networks" tutorial ([#4755](https://github.com/pyg-team/pytorch_geometric/pull/4755)) - Added `transforms.RootedSubgraph` interface with two implementations: `RootedEgoNets` and `RootedRWSubgraph` ([#3926](https://github.com/pyg-team/pytorch_geometric/pull/3926)) - Added `ptr` vectors for `follow_batch` attributes within `Batch.from_data_list` ([#4723](https://github.com/pyg-team/pytorch_geometric/pull/4723)) - Added `torch_geometric.nn.aggr` package ([#4687](https://github.com/pyg-team/pytorch_geometric/pull/4687), [#4721](https://github.com/pyg-team/pytorch_geometric/pull/4721), [#4731](https://github.com/pyg-team/pytorch_geometric/pull/4731), [#4762](https://github.com/pyg-team/pytorch_geometric/pull/4762), [#4749](https://github.com/pyg-team/pytorch_geometric/pull/4749), [#4779](https://github.com/pyg-team/pytorch_geometric/pull/4779), [#4863](https://github.com/pyg-team/pytorch_geometric/pull/4863), [#4864](https://github.com/pyg-team/pytorch_geometric/pull/4864), [#4865](https://github.com/pyg-team/pytorch_geometric/pull/4865), [#4866](https://github.com/pyg-team/pytorch_geometric/pull/4866), [#4872](https://github.com/pyg-team/pytorch_geometric/pull/4872), [#4934](https://github.com/pyg-team/pytorch_geometric/pull/4934), [#4935](https://github.com/pyg-team/pytorch_geometric/pull/4935), [#4957](https://github.com/pyg-team/pytorch_geometric/pull/4957), [#4973](https://github.com/pyg-team/pytorch_geometric/pull/4973), [#4973](https://github.com/pyg-team/pytorch_geometric/pull/4973), [#4986](https://github.com/pyg-team/pytorch_geometric/pull/4986), [#4995](https://github.com/pyg-team/pytorch_geometric/pull/4995), [#5000](https://github.com/pyg-team/pytorch_geometric/pull/5000), [#5034](https://github.com/pyg-team/pytorch_geometric/pull/5034), [#5036](https://github.com/pyg-team/pytorch_geometric/pull/5036), [#5039](https://github.com/pyg-team/pytorch_geometric/issues/5039), [#4522](https://github.com/pyg-team/pytorch_geometric/pull/4522), [#5033](https://github.com/pyg-team/pytorch_geometric/pull/5033]), [#5085](https://github.com/pyg-team/pytorch_geometric/pull/5085), [#5097](https://github.com/pyg-team/pytorch_geometric/pull/5097), [#5099](https://github.com/pyg-team/pytorch_geometric/pull/5099), [#5104](https://github.com/pyg-team/pytorch_geometric/pull/5104), [#5113](https://github.com/pyg-team/pytorch_geometric/pull/5113), [#5130](https://github.com/pyg-team/pytorch_geometric/pull/5130), [#5098](https://github.com/pyg-team/pytorch_geometric/pull/5098), [#5191](https://github.com/pyg-team/pytorch_geometric/pull/5191)) - Added the `DimeNet++` model ([#4432](https://github.com/pyg-team/pytorch_geometric/pull/4432), [#4699](https://github.com/pyg-team/pytorch_geometric/pull/4699), [#4700](https://github.com/pyg-team/pytorch_geometric/pull/4700), [#4800](https://github.com/pyg-team/pytorch_geometric/pull/4800)) - Added an example of using PyG with PyTorch Ignite ([#4487](https://github.com/pyg-team/pytorch_geometric/pull/4487)) - Added `GroupAddRev` module with support for reducing training GPU memory ([#4671](https://github.com/pyg-team/pytorch_geometric/pull/4671), [#4701](https://github.com/pyg-team/pytorch_geometric/pull/4701), [#4715](https://github.com/pyg-team/pytorch_geometric/pull/4715), [#4730](https://github.com/pyg-team/pytorch_geometric/pull/4730)) - Added benchmarks via [`wandb`](https://wandb.ai/site) ([#4656](https://github.com/pyg-team/pytorch_geometric/pull/4656), [#4672](https://github.com/pyg-team/pytorch_geometric/pull/4672), [#4676](https://github.com/pyg-team/pytorch_geometric/pull/4676)) - Added `unbatch` functionality ([#4628](https://github.com/pyg-team/pytorch_geometric/pull/4628)) - Confirm that `to_hetero()` works with custom functions, _e.g._, `dropout_adj` ([4653](https://github.com/pyg-team/pytorch_geometric/pull/4653)) - Added the `MLP.plain_last=False` option ([4652](https://github.com/pyg-team/pytorch_geometric/pull/4652)) - Added a check in `HeteroConv` and `to_hetero()` to ensure that `MessagePassing.add_self_loops` is disabled ([4647](https://github.com/pyg-team/pytorch_geometric/pull/4647)) - Added `HeteroData.subgraph()`, `HeteroData.node_type_subgraph()` and `HeteroData.edge_type_subgraph()` support ([#4635](https://github.com/pyg-team/pytorch_geometric/pull/4635)) - Added the `AQSOL` dataset ([#4626](https://github.com/pyg-team/pytorch_geometric/pull/4626)) - Added `HeteroData.node_items()` and `HeteroData.edge_items()` functionality ([#4644](https://github.com/pyg-team/pytorch_geometric/pull/4644)) - Added PyTorch Lightning support in GraphGym ([#4511](https://github.com/pyg-team/pytorch_geometric/pull/4511), [#4516](https://github.com/pyg-team/pytorch_geometric/pull/4516) [#4531](https://github.com/pyg-team/pytorch_geometric/pull/4531), [#4689](https://github.com/pyg-team/pytorch_geometric/pull/4689), [#4843](https://github.com/pyg-team/pytorch_geometric/pull/4843)) - Added support for returning embeddings in `MLP` models ([#4625](https://github.com/pyg-team/pytorch_geometric/pull/4625)) - Added faster initialization of `NeighborLoader` in case edge indices are already sorted (via `is_sorted=True`) ([#4620](https://github.com/pyg-team/pytorch_geometric/pull/4620), [#4702](https://github.com/pyg-team/pytorch_geometric/pull/4702)) - Added `AddPositionalEncoding` transform ([#4521](https://github.com/pyg-team/pytorch_geometric/pull/4521)) - Added `HeteroData.is_undirected()` support ([#4604](https://github.com/pyg-team/pytorch_geometric/pull/4604)) - Added the `Genius` and `Wiki` datasets to `nn.datasets.LINKXDataset` ([#4570](https://github.com/pyg-team/pytorch_geometric/pull/4570), [#4600](https://github.com/pyg-team/pytorch_geometric/pull/4600)) - Added `nn.aggr.EquilibrumAggregation` implicit global layer ([#4522](https://github.com/pyg-team/pytorch_geometric/pull/4522)) - Added support for graph-level outputs in `to_hetero` ([#4582](https://github.com/pyg-team/pytorch_geometric/pull/4582)) - Added `CHANGELOG.md` ([#4581](https://github.com/pyg-team/pytorch_geometric/pull/4581)) - Added `HeteroData` support to the `RemoveIsolatedNodes` transform ([#4479](https://github.com/pyg-team/pytorch_geometric/pull/4479)) - Added `HeteroData.num_features` functionality ([#4504](https://github.com/pyg-team/pytorch_geometric/pull/4504)) - Added support for projecting features before propagation in `SAGEConv` ([#4437](https://github.com/pyg-team/pytorch_geometric/pull/4437)) - Added `Geom-GCN` splits to the `Planetoid` datasets ([#4442](https://github.com/pyg-team/pytorch_geometric/pull/4442)) - Added a `LinkNeighborLoader` for training scalable link predictions models [#4396](https://github.com/pyg-team/pytorch_geometric/pull/4396), [#4439](https://github.com/pyg-team/pytorch_geometric/pull/4439), [#4441](https://github.com/pyg-team/pytorch_geometric/pull/4441), [#4446](https://github.com/pyg-team/pytorch_geometric/pull/4446), [#4508](https://github.com/pyg-team/pytorch_geometric/pull/4508), [#4509](https://github.com/pyg-team/pytorch_geometric/pull/4509)) - Added an unsupervised `GraphSAGE` example on `PPI` ([#4416](https://github.com/pyg-team/pytorch_geometric/pull/4416)) - Added support for `LSTM` aggregation in `SAGEConv` ([#4379](https://github.com/pyg-team/pytorch_geometric/pull/4379)) - Added support for floating-point labels in `RandomLinkSplit` ([#4311](https://github.com/pyg-team/pytorch_geometric/pull/4311), [#4383](https://github.com/pyg-team/pytorch_geometric/pull/4383)) - Added support for `torch.data` `DataPipes` ([#4302](https://github.com/pyg-team/pytorch_geometric/pull/4302), [#4345](https://github.com/pyg-team/pytorch_geometric/pull/4345), [#4349](https://github.com/pyg-team/pytorch_geometric/pull/4349)) - Added support for the `cosine` argument in the `KNNGraph`/`RadiusGraph` transforms ([#4344](https://github.com/pyg-team/pytorch_geometric/pull/4344)) - Added support graph-level attributes in `networkx` conversion ([#4343](https://github.com/pyg-team/pytorch_geometric/pull/4343)) - Added support for renaming node types via `HeteroData.rename` ([#4329](https://github.com/pyg-team/pytorch_geometric/pull/4329)) - Added an example to load a trained PyG model in C++ ([#4307](https://github.com/pyg-team/pytorch_geometric/pull/4307)) - Added a `MessagePassing.explain_message` method to customize making explanations on messages ([#4278](https://github.com/pyg-team/pytorch_geometric/pull/4278), [#4448](https://github.com/pyg-team/pytorch_geometric/pull/4448))) - Added support for `GATv2Conv` in the `nn.models.GAT` model ([#4357](https://github.com/pyg-team/pytorch_geometric/pull/4357)) - Added `HeteroData.subgraph` functionality ([#4243](https://github.com/pyg-team/pytorch_geometric/pull/4243)) - Added the `MaskLabel` module and a corresponding masked label propagation example ([#4197](https://github.com/pyg-team/pytorch_geometric/pull/4197)) - Added temporal sampling support to `NeighborLoader` ([#4025](https://github.com/pyg-team/pytorch_geometric/pull/4025)) - Added an example for unsupervised heterogeneous graph learning based on "Deep Multiplex Graph Infomax" ([#3189](https://github.com/pyg-team/pytorch_geometric/pull/3189)) ### Changed - Changed docstring for `RandomLinkSplit` ([#5190](https://github.com/pyg-team/pytorch_geometric/issues/5190)) - Switched to PyTorch `scatter_reduce` implementation - experimental feature ([#5120](https://github.com/pyg-team/pytorch_geometric/pull/5120)) - Fixed `RGATConv` device mismatches for `f-scaled` mode ([#5187](https://github.com/pyg-team/pytorch_geometric/pull/5187)] - Allow for multi-dimensional `edge_labels` in `LinkNeighborLoader` ([#5186](https://github.com/pyg-team/pytorch_geometric/pull/5186)] - Fixed `GINEConv` bug with non-sequential input ([#5154](https://github.com/pyg-team/pytorch_geometric/pull/5154)] - Improved error message ([#5095](https://github.com/pyg-team/pytorch_geometric/pull/5095)) - Fixed `HGTLoader` bug which produced outputs with missing edge types ([#5067](https://github.com/pyg-team/pytorch_geometric/pull/5067)) - Fixed dynamic inheritance issue in data batching ([#5051](https://github.com/pyg-team/pytorch_geometric/pull/5051)) - Fixed `load_state_dict` in `Linear` with `strict=False` mode ([5094](https://github.com/pyg-team/pytorch_geometric/pull/5094)) - Fixed typo in `MaskLabel.ratio_mask` ([5093](https://github.com/pyg-team/pytorch_geometric/pull/5093)) - Fixed `data.num_node_features` computation for sparse matrices ([5089](https://github.com/pyg-team/pytorch_geometric/pull/5089)) - Fixed `torch.fx` bug with `torch.nn.aggr` package ([#5021](https://github.com/pyg-team/pytorch_geometric/pull/5021))) - Fixed `GenConv` test ([4993](https://github.com/pyg-team/pytorch_geometric/pull/4993)) - Fixed packaging tests for Python 3.10 ([4982](https://github.com/pyg-team/pytorch_geometric/pull/4982)) - Changed `act_dict` (part of `graphgym`) to create individual instances instead of reusing the same ones everywhere ([4978](https://github.com/pyg-team/pytorch_geometric/pull/4978)) - Fixed issue where one-hot tensors were passed to `F.one_hot` ([4970](https://github.com/pyg-team/pytorch_geometric/pull/4970)) - Fixed `bool` arugments in `argparse` in `benchmark/` ([#4967](https://github.com/pyg-team/pytorch_geometric/pull/4967)) - Fixed `BasicGNN` for `num_layers=1`, which now respects a desired number of `out_channels` ([#4943](https://github.com/pyg-team/pytorch_geometric/pull/4943)) - `len(batch)` will now return the number of graphs inside the batch, not the number of attributes ([#4931](https://github.com/pyg-team/pytorch_geometric/pull/4931)) - Fixed `data.subgraph` generation for 0-dim tensors ([#4932](https://github.com/pyg-team/pytorch_geometric/pull/4932)) - Removed unnecssary inclusion of self-loops when sampling negative edges ([#4880](https://github.com/pyg-team/pytorch_geometric/pull/4880)) - Fixed `InMemoryDataset` inferring wrong `len` for lists of tensors ([#4837](https://github.com/pyg-team/pytorch_geometric/pull/4837)) - Fixed `Batch.separate` when using it for lists of tensors ([#4837](https://github.com/pyg-team/pytorch_geometric/pull/4837)) - Correct docstring for SAGEConv ([#4852](https://github.com/pyg-team/pytorch_geometric/pull/4852)) - Fixed a bug in `TUDataset` where `pre_filter` was not applied whenever `pre_transform` was present - Renamed `RandomTranslate` to `RandomJitter` - the usage of `RandomTranslate` is now deprecated ([#4828](https://github.com/pyg-team/pytorch_geometric/pull/4828)) - Do not allow accessing edge types in `HeteroData` with two node types when there exists multiple relations between these types ([#4782](https://github.com/pyg-team/pytorch_geometric/pull/4782)) - Allow `edge_type == rev_edge_type` argument in `RandomLinkSplit` ([#4757](https://github.com/pyg-team/pytorch_geometric/pull/4757), [#5221](https://github.com/pyg-team/pytorch_geometric/pull/5221)) - Fixed a numerical instability in the `GeneralConv` and `neighbor_sample` tests ([#4754](https://github.com/pyg-team/pytorch_geometric/pull/4754)) - Fixed a bug in `HANConv` in which destination node features rather than source node features were propagated ([#4753](https://github.com/pyg-team/pytorch_geometric/pull/4753)) - Fixed versions of `checkout` and `setup-python` in CI ([#4751](https://github.com/pyg-team/pytorch_geometric/pull/4751)) - Fixed `protobuf` version ([#4719](https://github.com/pyg-team/pytorch_geometric/pull/4719)) - Fixed the ranking protocol bug in the RGCN link prediction example ([#4688](https://github.com/pyg-team/pytorch_geometric/pull/4688)) - Math support in Markdown ([#4683](https://github.com/pyg-team/pytorch_geometric/pull/4683)) - Allow for `setter` properties in `Data` ([#4682](https://github.com/pyg-team/pytorch_geometric/pull/4682), [#4686](https://github.com/pyg-team/pytorch_geometric/pull/4686)) - Allow for optional `edge_weight` in `GCN2Conv` ([#4670](https://github.com/pyg-team/pytorch_geometric/pull/4670)) - Fixed the interplay between `TUDataset` and `pre_transform` that modify node features ([#4669](https://github.com/pyg-team/pytorch_geometric/pull/4669)) - Make use of the `pyg_sphinx_theme` documentation template ([#4664](https://github.com/pyg-team/pyg-lib/pull/4664), [#4667](https://github.com/pyg-team/pyg-lib/pull/4667)) - Refactored reading molecular positions from sdf file for qm9 datasets ([4654](https://github.com/pyg-team/pytorch_geometric/pull/4654)) - Fixed `MLP.jittable()` bug in case `return_emb=True` ([#4645](https://github.com/pyg-team/pytorch_geometric/pull/4645), [#4648](https://github.com/pyg-team/pytorch_geometric/pull/4648)) - The generated node features of `StochasticBlockModelDataset` are now ordered with respect to their labels ([#4617](https://github.com/pyg-team/pytorch_geometric/pull/4617)) - Fixed typos in the documentation ([#4616](https://github.com/pyg-team/pytorch_geometric/pull/4616), [#4824](https://github.com/pyg-team/pytorch_geometric/pull/4824), [#4895](https://github.com/pyg-team/pytorch_geometric/pull/4895), [#5161](https://github.com/pyg-team/pytorch_geometric/pull/5161)) - The `bias` argument in `TAGConv` is now actually applied ([#4597](https://github.com/pyg-team/pytorch_geometric/pull/4597)) - Fixed subclass behavior of `process` and `download` in `Datsaet` ([#4586](https://github.com/pyg-team/pytorch_geometric/pull/4586)) - Fixed filtering of attributes for loaders in case `__cat_dim__ != 0` ([#4629](https://github.com/pyg-team/pytorch_geometric/pull/4629)) - Fixed `SparseTensor` support in `NeighborLoader` ([#4320](https://github.com/pyg-team/pytorch_geometric/pull/4320)) - Fixed average degree handling in `PNAConv` ([#4312](https://github.com/pyg-team/pytorch_geometric/pull/4312)) - Fixed a bug in `from_networkx` in case some attributes are PyTorch tensors ([#4486](https://github.com/pyg-team/pytorch_geometric/pull/4486)) - Added a missing clamp in `DimeNet` ([#4506](https://github.com/pyg-team/pytorch_geometric/pull/4506), [#4562](https://github.com/pyg-team/pytorch_geometric/pull/4562)) - Fixed the download link in `DBP15K` ([#4428](https://github.com/pyg-team/pytorch_geometric/pull/4428)) - Fixed an autograd bug in `DimeNet` when resetting parameters ([#4424](https://github.com/pyg-team/pytorch_geometric/pull/4424)) - Fixed bipartite message passing in case `flow="target_to_source"` ([#4418](https://github.com/pyg-team/pytorch_geometric/pull/4418)) - Fixed a bug in which `num_nodes` was not properly updated in the `FixedPoints` transform ([#4394](https://github.com/pyg-team/pytorch_geometric/pull/4394)) - PyTorch Lightning >= 1.6 support ([#4377](https://github.com/pyg-team/pytorch_geometric/pull/4377)) - Fixed a bug in which `GATConv` was not jittable ([#4347](https://github.com/pyg-team/pytorch_geometric/pull/4347)) - Fixed a bug in which the GraphGym config was not stored in each specific experiment directory ([#4338](https://github.com/pyg-team/pytorch_geometric/pull/4338)) - Fixed a bug in which `nn.models.GAT` did not produce `out_channels`-many output channels ([#4299](https://github.com/pyg-team/pytorch_geometric/pull/4299)) - Fixed mini-batching with empty lists as attributes ([#4293](https://github.com/pyg-team/pytorch_geometric/pull/4293)) - Fixed a bug in which `GCNConv` could not be combined with `to_hetero` on heterogeneous graphs with one node type ([#4279](https://github.com/pyg-team/pytorch_geometric/pull/4279)) ### Removed - Remove internal metrics in favor of `torchmetrics` ([#4287](https://github.com/pyg-team/pytorch_geometric/pull/4287)) pytorch/CITATION.cff000066400000000000000000000005531444262304300144450ustar00rootroot00000000000000--- cff-version: 1.2.0 message: "Please cite our paper if you use this code in your own work." title: "Fast Graph Representation Learning with PyTorch Geometric" authors: - family-names: "Fey" given-names: "Matthias" - family-names: "Lenssen" given-names: "Jan Eric" date-released: 2019-05-06 license: MIT url: "https://github.com/pyg-team/pytorch_geometric" pytorch/LICENSE000066400000000000000000000020531444262304300135550ustar00rootroot00000000000000Copyright (c) 2023 PyG Team Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. pytorch/README.md000066400000000000000000001667311444262304300140450ustar00rootroot00000000000000[pypi-image]: https://badge.fury.io/py/torch-geometric.svg [pypi-url]: https://pypi.python.org/pypi/torch-geometric [testing-image]: https://github.com/pyg-team/pytorch_geometric/actions/workflows/testing.yml/badge.svg [testing-url]: https://github.com/pyg-team/pytorch_geometric/actions/workflows/testing.yml [linting-image]: https://github.com/pyg-team/pytorch_geometric/actions/workflows/linting.yml/badge.svg [linting-url]: https://github.com/pyg-team/pytorch_geometric/actions/workflows/linting.yml [docs-image]: https://readthedocs.org/projects/pytorch-geometric/badge/?version=latest [docs-url]: https://pytorch-geometric.readthedocs.io/en/latest [coverage-image]: https://codecov.io/gh/pyg-team/pytorch_geometric/branch/master/graph/badge.svg [coverage-url]: https://codecov.io/github/pyg-team/pytorch_geometric?branch=master [contributing-image]: https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat [contributing-url]: https://github.com/pyg-team/pytorch_geometric/blob/master/.github/CONTRIBUTING.md [slack-image]: https://img.shields.io/badge/slack-pyg-brightgreen [slack-url]: https://data.pyg.org/slack.html

-------------------------------------------------------------------------------- [![PyPI Version][pypi-image]][pypi-url] [![Testing Status][testing-image]][testing-url] [![Linting Status][linting-image]][linting-url] [![Docs Status][docs-image]][docs-url] [![Contributing][contributing-image]][contributing-url] [![Slack][slack-image]][slack-url] **[Documentation](https://pytorch-geometric.readthedocs.io)** | **[Paper](https://arxiv.org/abs/1903.02428)** | **[Colab Notebooks and Video Tutorials](https://pytorch-geometric.readthedocs.io/en/latest/get_started/colabs.html)** | **[External Resources](https://pytorch-geometric.readthedocs.io/en/latest/external/resources.html)** | **[OGB Examples](https://github.com/snap-stanford/ogb/tree/master/examples)** **PyG** *(PyTorch Geometric)* is a library built upon [PyTorch](https://pytorch.org/) to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data. It consists of various methods for deep learning on graphs and other irregular structures, also known as *[geometric deep learning](http://geometricdeeplearning.com/)*, from a variety of published papers. In addition, it consists of easy-to-use mini-batch loaders for operating on many small and single giant graphs, [multi GPU-support](https://github.com/pyg-team/pytorch_geometric/tree/master/examples/multi_gpu), [`torch.compile`](https://pytorch-geometric.readthedocs.io/en/latest/tutorial/compile.html) support, [`DataPipe`](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/datapipe.py) support, a large number of common benchmark datasets (based on simple interfaces to create your own), the [GraphGym](https://pytorch-geometric.readthedocs.io/en/latest/advanced/graphgym.html) experiment manager, and helpful transforms, both for learning on arbitrary graphs as well as on 3D meshes or point clouds. **[Click here to join our Slack community!][slack-url]**

-------------------------------------------------------------------------------- * [Library Highlights](#library-highlights) * [Quick Tour for New Users](#quick-tour-for-new-users) * [Architecture Overview](#architecture-overview) * [Implemented GNN Models](#implemented-gnn-models) * [Installation](#installation) ## Library Highlights Whether you are a machine learning researcher or first-time user of machine learning toolkits, here are some reasons to try out PyG for machine learning on graph-structured data. * **Easy-to-use and unified API**: All it takes is 10-20 lines of code to get started with training a GNN model (see the next section for a [quick tour](#quick-tour-for-new-users)). PyG is *PyTorch-on-the-rocks*: It utilizes a tensor-centric API and keeps design principles close to vanilla PyTorch. If you are already familiar with PyTorch, utilizing PyG is straightforward. * **Comprehensive and well-maintained GNN models**: Most of the state-of-the-art Graph Neural Network architectures have been implemented by library developers or authors of research papers and are ready to be applied. * **Great flexibility**: Existing PyG models can easily be extended for conducting your own research with GNNs. Making modifications to existing models or creating new architectures is simple, thanks to its easy-to-use message passing API, and a variety of operators and utility functions. * **Large-scale real-world GNN models**: We focus on the need of GNN applications in challenging real-world scenarios, and support learning on diverse types of graphs, including but not limited to: scalable GNNs for graphs with millions of nodes; dynamic GNNs for node predictions over time; heterogeneous GNNs with multiple node types and edge types. * **GraphGym integration**: GraphGym lets users easily reproduce GNN experiments, is able to launch and analyze thousands of different GNN configurations, and is customizable by registering new modules to a GNN learning pipeline. ## Quick Tour for New Users In this quick tour, we highlight the ease of creating and training a GNN model with only a few lines of code. ### Train your own GNN model In the first glimpse of PyG, we implement the training of a GNN for classifying papers in a citation graph. For this, we load the [Cora](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Planetoid.html) dataset, and create a simple 2-layer GCN model using the pre-defined [`GCNConv`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCNConv.html): ```python import torch from torch import Tensor from torch_geometric.nn import GCNConv from torch_geometric.datasets import Planetoid dataset = Planetoid(root='.', name='Cora') class GCN(torch.nn.Module): def __init__(self, in_channels, hidden_channels, out_channels): super().__init__() self.conv1 = GCNConv(in_channels, hidden_channels) self.conv2 = GCNConv(hidden_channels, out_channels) def forward(self, x: Tensor, edge_index: Tensor) -> Tensor: # x: Node feature matrix of shape [num_nodes, in_channels] # edge_index: Graph connectivity matrix of shape [2, num_edges] x = self.conv1(x, edge_index).relu() x = self.conv2(x, edge_index) return x model = GCN(dataset.num_features, 16, dataset.num_classes) ```
We can now optimize the model in a training loop, similar to the standard PyTorch training procedure. ```python import torch.nn.functional as F data = dataset[0] optimizer = torch.optim.Adam(model.parameters(), lr=0.01) for epoch in range(200): pred = model(data.x, data.edge_index) loss = F.cross_entropy(pred[data.train_mask], data.y[data.train_mask]) # Backpropagation optimizer.zero_grad() loss.backward() optimizer.step() ```
More information about evaluating final model performance can be found in the corresponding [example](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn.py). ### Create your own GNN layer In addition to the easy application of existing GNNs, PyG makes it simple to implement custom Graph Neural Networks (see [here](https://pytorch-geometric.readthedocs.io/en/latest/tutorial/create_gnn.html) for the accompanying tutorial). For example, this is all it takes to implement the [edge convolutional layer](https://arxiv.org/abs/1801.07829) from Wang *et al.*: $$x_i^{\prime} ~ = ~ \max_{j \in \mathcal{N}(i)} ~ \textrm{MLP}_{\theta} \left( [ ~ x_i, ~ x_j - x_i ~ ] \right)$$ ```python import torch from torch import Tensor from torch.nn import Sequential, Linear, ReLU from torch_geometric.nn import MessagePassing class EdgeConv(MessagePassing): def __init__(self, in_channels, out_channels): super().__init__(aggr="max") # "Max" aggregation. self.mlp = Sequential( Linear(2 * in_channels, out_channels), ReLU(), Linear(out_channels, out_channels), ) def forward(self, x: Tensor, edge_index: Tensor) -> Tensor: # x: Node feature matrix of shape [num_nodes, in_channels] # edge_index: Graph connectivity matrix of shape [2, num_edges] return self.propagate(edge_index, x=x) # shape [num_nodes, out_channels] def message(self, x_j: Tensor, x_i: Tensor) -> Tensor: # x_j: Source node features of shape [num_edges, in_channels] # x_i: Target node features of shape [num_edges, in_channels] edge_features = torch.cat([x_i, x_j - x_i], dim=-1) return self.mlp(edge_features) # shape [num_edges, out_channels] ``` ### Manage experiments with GraphGym GraphGym allows you to manage and launch GNN experiments, using a highly modularized pipeline (see [here](https://pytorch-geometric.readthedocs.io/en/latest/advanced/graphgym.html) for the accompanying tutorial). ``` git clone https://github.com/pyg-team/pytorch_geometric.git cd pytorch_geometric/graphgym bash run_single.sh # run a single GNN experiment (node/edge/graph-level) bash run_batch.sh # run a batch of GNN experiments, using differnt GNN designs/datasets/tasks ``` Users are highly encouraged to check out the [documentation](https://pytorch-geometric.readthedocs.io/en/latest), which contains additional tutorials on the essential functionalities of PyG, including data handling, creation of datasets and a full list of implemented methods, transforms, and datasets. For a quick start, check out our [examples](https://github.com/pyg-team/pytorch_geometric/tree/master/examples) in `examples/`. ## Architecture Overview PyG provides a multi-layer framework that enables users to build Graph Neural Network solutions on both low and high levels. It comprises of the following components: * The PyG **engine** utilizes the powerful PyTorch deep learning framework with full [`torch.compile`](https://pytorch-geometric.readthedocs.io/en/latest/tutorial/compile.html) and [TorchScript](https://pytorch-geometric.readthedocs.io/en/latest/advanced/jit.html) support, as well as additions of efficient CPU/CUDA libraries for operating on sparse data, *e.g.*, [`pyg-lib`](https://github.com/pyg-team/pyg-lib). * The PyG **storage** handles data processing, transformation and loading pipelines. It is capable of handling and processing large-scale graph datasets, and provides effective solutions for heterogeneous graphs. It further provides a variety of sampling solutions, which enable training of GNNs on large-scale graphs. * The PyG **operators** bundle essential functionalities for implementing Graph Neural Networks. PyG supports important GNN building blocks that can be combined and applied to various parts of a GNN model, ensuring rich flexibility of GNN design. * Finally, PyG provides an abundant set of GNN **models**, and examples that showcase GNN models on standard graph benchmarks. Thanks to its flexibility, users can easily build and modify custom GNN models to fit their specific needs.

## Implemented GNN Models We list currently supported PyG models, layers and operators according to category: **GNN layers:** All Graph Neural Network layers are implemented via the **[`nn.MessagePassing`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.MessagePassing.html)** interface. A GNN layer specifies how to perform message passing, *i.e.* by designing different message, aggregation and update functions as defined [here](https://pytorch-geometric.readthedocs.io/en/latest/tutorial/create_gnn.html). These GNN layers can be stacked together to create Graph Neural Network models. * **[GCNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCNConv.html)** from Kipf and Welling: [Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907) (ICLR 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn.py)] * **[ChebConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.ChebConv.html)** from Defferrard *et al.*: [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://arxiv.org/abs/1606.09375) (NIPS 2016) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn.py#L36-L37)] * **[GATConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GATConv.html)** from VeličkoviΔ‡ *et al.*: [Graph Attention Networks](https://arxiv.org/abs/1710.10903) (ICLR 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gat.py)]
Expand to see all implemented GNN layers... * **[GCN2Conv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCN2Conv.html)** from Chen *et al.*: [Simple and Deep Graph Convolutional Networks](https://arxiv.org/abs/2007.02133) (ICML 2020) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn2_cora.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn2_ppi.py)] * **[SplineConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SplineConv.html)** from Fey *et al.*: [SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels](https://arxiv.org/abs/1711.08920) (CVPR 2018) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/cora.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/faust.py)] * **[NNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.NNConv.html)** from Gilmer *et al.*: [Neural Message Passing for Quantum Chemistry](https://arxiv.org/abs/1704.01212) (ICML 2017) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/qm9_nn_conv.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/mnist_nn_conv.py)] * **[CGConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.CGConv.html)** from Xie and Grossman: [Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.145301) (Physical Review Letters 120, 2018) * **[ECConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.ECConv.html)** from Simonovsky and Komodakis: [Edge-Conditioned Convolution on Graphs](https://arxiv.org/abs/1704.02901) (CVPR 2017) * **[EGConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.EGConv.html)** from Tailor *et al.*: [Adaptive Filters and Aggregator Fusion for Efficient Graph Convolutions](https://arxiv.org/abs/2104.01481) (GNNSys 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/egc.py)] * **[GATv2Conv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GATv2Conv.html)** from Brody *et al.*: [How Attentive are Graph Attention Networks?](https://arxiv.org/abs/2105.14491) (ICLR 2022) * **[TransformerConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.TransformerConv.html)** from Shi *et al.*: [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://arxiv.org/abs/2009.03509) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/unimp_arxiv.py)] * **[SAGEConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SAGEConv.html)** from Hamilton *et al.*: [Inductive Representation Learning on Large Graphs](https://arxiv.org/abs/1706.02216) (NIPS 2017) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/reddit.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/ogbn_products_sage.py), [**Example3**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_sage_unsup.py), [**Example4**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_sage_unsup_ppi.py)] * **[GraphConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GraphConv.html)** from, *e.g.*, Morris *et al.*: [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://arxiv.org/abs/1810.02244) (AAAI 2019) * **[GatedGraphConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GatedGraphConv.html)** from Li *et al.*: [Gated Graph Sequence Neural Networks](https://arxiv.org/abs/1511.05493) (ICLR 2016) * **[ResGatedGraphConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.ResGatedGraphConv.html)** from Bresson and Laurent: [Residual Gated Graph ConvNets](https://arxiv.org/abs/1711.07553) (CoRR 2017) * **[GINConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GINConv.html)** from Xu *et al.*: [How Powerful are Graph Neural Networks?](https://arxiv.org/abs/1810.00826) (ICLR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/mutag_gin.py)] * **[GINEConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GINEConv.html)** from Hu *et al.*: [Strategies for Pre-training Graph Neural Networks](https://arxiv.org/abs/1905.12265) (ICLR 2020) * **[ARMAConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.ARMAConv.html)** from Bianchi *et al.*: [Graph Neural Networks with Convolutional ARMA Filters](https://arxiv.org/abs/1901.01343) (CoRR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/arma.py)] * **[SGConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SGConv.html)** from Wu *et al.*: [Simplifying Graph Convolutional Networks](https://arxiv.org/abs/1902.07153) (CoRR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/sgc.py)] * **[APPNP](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.APPNP.html)** from Klicpera *et al.*: [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://arxiv.org/abs/1810.05997) (ICLR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/appnp.py)] * **[MFConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.MFConv.html)** from Duvenaud *et al.*: [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://arxiv.org/abs/1509.09292) (NIPS 2015) * **[AGNNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.AGNNConv.html)** from Thekumparampil *et al.*: [Attention-based Graph Neural Network for Semi-Supervised Learning](https://arxiv.org/abs/1803.03735) (CoRR 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/agnn.py)] * **[TAGConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.TAGConv.html)** from Du *et al.*: [Topology Adaptive Graph Convolutional Networks](https://arxiv.org/abs/1710.10370) (CoRR 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/tagcn.py)] * **[PNAConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.PNAConv.html)** from Corso *et al.*: [Principal Neighbourhood Aggregation for Graph Nets](https://arxiv.org/abs/2004.05718) (CoRR 2020) [**[Example](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pna.py)**] * **[FAConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.FAConv.html)** from Bo *et al.*: [Beyond Low-Frequency Information in Graph Convolutional Networks](https://arxiv.org/abs/2101.00797) (AAAI 2021) * **[PDNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.nn.conv.PDNConv.html)** from Rozemberczki *et al.*: [Pathfinder Discovery Networks for Neural Message Passing](https://arxiv.org/abs/2010.12878) (WWW 2021) * **[RGCNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.RGCNConv.html)** from Schlichtkrull *et al.*: [Modeling Relational Data with Graph Convolutional Networks](https://arxiv.org/abs/1703.06103) (ESWC 2018) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/rgcn.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/rgcn_link_pred.py)] * **[RGATConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.RGATConv.html)** from Busbridge *et al.*: [Relational Graph Attention Networks](https://arxiv.org/abs/1904.05811) (CoRR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/rgat.py)] * **[FiLMConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.FiLMConv.html)** from Brockschmidt: [GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation](https://arxiv.org/abs/1906.12192) (ICML 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/film.py)] * **[SignedConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SignedConv.html)** from Derr *et al.*: [Signed Graph Convolutional Network](https://arxiv.org/abs/1808.06354) (ICDM 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/signed_gcn.py)] * **[DNAConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.DNAConv.html)** from Fey: [Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks](https://arxiv.org/abs/1904.04849) (ICLR-W 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/dna.py)] * **[PANConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.PANConv.html)** from Ma *et al.*: [Path Integral Based Convolution and Pooling for Graph Neural Networks](https://arxiv.org/abs/2006.16811) (NeurIPS 2020) * **[PointNetConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.PointNetConv.html)** (including **[Iterative Farthest Point Sampling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.fps.html)**, dynamic graph generation based on **[nearest neighbor](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.knn_graph.html)** or **[maximum distance](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.radius_graph.html)**, and **[k-NN interpolation](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.unpool.knn_interpolate.html)** for upsampling) from Qi *et al.*: [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://arxiv.org/abs/1612.00593) (CVPR 2017) and [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://arxiv.org/abs/1706.02413) (NIPS 2017) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pointnet2_classification.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pointnet2_segmentation.py)] * **[EdgeConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.EdgeConv.html)** from Wang *et al.*: [Dynamic Graph CNN for Learning on Point Clouds](https://arxiv.org/abs/1801.07829) (CoRR, 2018) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/dgcnn_classification.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/dgcnn_segmentation.py)] * **[XConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.XConv.html)** from Li *et al.*: [PointCNN: Convolution On X-Transformed Points](https://arxiv.org/abs/1801.07791) (NeurIPS 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/point_cnn.py)] * **[PPFConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.PPFConv.html)** from Deng *et al.*: [PPFNet: Global Context Aware Local Features for Robust 3D Point Matching](https://arxiv.org/abs/1802.02669) (CVPR 2018) * **[GMMConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GMMConv.html)** from Monti *et al.*: [Geometric Deep Learning on Graphs and Manifolds using Mixture Model CNNs](https://arxiv.org/abs/1611.08402) (CVPR 2017) * **[FeaStConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.FeaStConv.html)** from Verma *et al.*: [FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis](https://arxiv.org/abs/1706.05206) (CVPR 2018) * **[PointTransformerConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.PointTransformerConv.html)** from Zhao *et al.*: [Point Transformer](https://arxiv.org/abs/2012.09164) (2020) * **[HypergraphConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.HypergraphConv.html)** from Bai *et al.*: [Hypergraph Convolution and Hypergraph Attention](https://arxiv.org/abs/1901.08150) (CoRR 2019) * **[GravNetConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GravNetConv.html)** from Qasim *et al.*: [Learning Representations of Irregular Particle-detector Geometry with Distance-weighted Graph Networks](https://arxiv.org/abs/1902.07987) (European Physics Journal C, 2019) * **[SuperGAT](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SuperGATConv.html)** from Kim and Oh: [How To Find Your Friendly Neighborhood: Graph Attention Design With Self-Supervision](https://openreview.net/forum?id=Wi5KUNlqWty) (ICLR 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/super_gat.py)] * **[HGTConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.HGTConv.html)** from Hu *et al.*: [Heterogeneous Graph Transformer](https://arxiv.org/abs/2003.01332) (WWW 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/hgt_dblp.py)] * **[HEATConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.HEATonv.html)** from Mo *et al.*: [Heterogeneous Edge-Enhanced Graph Attention Network For Multi-Agent Trajectory Prediction](https://arxiv.org/abs/2106.07161) (CoRR 2021) * **[SSGConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SSGConv.html)** from Zhu *et al.*: [Simple Spectral Graph Convolution](https://openreview.net/forum?id=CYO5T-YjWZV) (ICLR 2021) * **[FusedGATConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.FusedGATConv.html)** from Zhang *et al.*: [Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective](https://proceedings.mlsys.org/paper/2022/file/9a1158154dfa42caddbd0694a4e9bdc8-Paper.pdf) (MLSys 2022) * **[GPSConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GPSConv.html)** from RampΓ‘Ε‘ek *et al.*: [Recipe for a General, Powerful, Scalable Graph Transformer](https://arxiv.org/abs/2205.12454) (NeurIPS 2022) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_gps.py)]
**Pooling layers:** Graph pooling layers combine the vectorial representations of a set of nodes in a graph (or a subgraph) into a single vector representation that summarizes its properties of nodes. It is commonly applied to graph-level tasks, which require combining node features into a single graph representation. * **[Top-K Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.TopKPooling.html)** from Gao and Ji: [Graph U-Nets](https://arxiv.org/abs/1905.05178) (ICML 2019), Cangea *et al.*: [Towards Sparse Hierarchical Graph Classifiers](https://arxiv.org/abs/1811.01287) (NeurIPS-W 2018) and Knyazev *et al.*: [Understanding Attention and Generalization in Graph Neural Networks](https://arxiv.org/abs/1905.02850) (ICLR-W 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/proteins_topk_pool.py)] * **[DiffPool](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.dense.dense_diff_pool.html)** from Ying *et al.*: [Hierarchical Graph Representation Learning with Differentiable Pooling](https://arxiv.org/abs/1806.08804) (NeurIPS 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/proteins_diff_pool.py)]
Expand to see all implemented pooling layers... * **[Attentional Aggregation](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.aggr.AttentionalAggregation.html)** from Li *et al.*: [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://arxiv.org/abs/1904.12787) (ICML 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/global_attention.py)] * **[Set2Set](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.aggr.Set2Set.html)** from Vinyals *et al.*: [Order Matters: Sequence to Sequence for Sets](https://arxiv.org/abs/1511.06391) (ICLR 2016) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/set2set.py)] * **[Sort Aggregation](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.aggr.SortAggregation.html)** from Zhang *et al.*: [An End-to-End Deep Learning Architecture for Graph Classification](https://www.cse.wustl.edu/~muhan/papers/AAAI_2018_DGCNN.pdf) (AAAI 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/sort_pool.py)] * **[MinCut Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.dense.dense_mincut_pool.html)** from Bianchi *et al.*: [Spectral Clustering with Graph Neural Networks for Graph Pooling](https://arxiv.org/abs/1907.00481) (ICML 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/proteins_mincut_pool.py)] * **[DMoN Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.dense.DMoNPooling.html)** from Tsitsulin *et al.*: [Graph Clustering with Graph Neural Networks](https://arxiv.org/abs/2006.16904) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/proteins_dmon_pool.py)] * **[Graclus Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.graclus.html)** from Dhillon *et al.*: [Weighted Graph Cuts without Eigenvectors: A Multilevel Approach](http://www.cs.utexas.edu/users/inderjit/public_papers/multilevel_pami.pdf) (PAMI 2007) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/mnist_graclus.py)] * **[Voxel Grid Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.voxel_grid.html)** from, *e.g.*, Simonovsky and Komodakis: [Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs](https://arxiv.org/abs/1704.02901) (CVPR 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/mnist_voxel_grid.py)] * **[SAG Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.SAGPooling.html)** from Lee *et al.*: [Self-Attention Graph Pooling](https://arxiv.org/abs/1904.08082) (ICML 2019) and Knyazev *et al.*: [Understanding Attention and Generalization in Graph Neural Networks](https://arxiv.org/abs/1905.02850) (ICLR-W 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/sag_pool.py)] * **[Edge Pooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.EdgePooling.html)** from Diehl *et al.*: [Towards Graph Pooling by Edge Contraction](https://graphreason.github.io/papers/17.pdf) (ICML-W 2019) and Diehl: [Edge Contraction Pooling for Graph Neural Networks](https://arxiv.org/abs/1905.10990) (CoRR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/edge_pool.py)] * **[ASAPooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.ASAPooling.html)** from Ranjan *et al.*: [ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations](https://arxiv.org/abs/1911.07979) (AAAI 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/asap.py)] * **[PANPooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.PANPooling.html)** from Ma *et al.*: [Path Integral Based Convolution and Pooling for Graph Neural Networks](https://arxiv.org/abs/2006.16811) (NeurIPS 2020) * **[MemPooling](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.MemPooling.html)** from Khasahmadi *et al.*: [Memory-Based Graph Networks](https://arxiv.org/abs/2002.09518) (ICLR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/mem_pool.py)] * **[Graph Multiset Transformer](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.aggr.GraphMultisetTransformer.html)** from Baek *et al.*: [Accurate Learning of Graph Representations with Graph Multiset Pooling](https://arxiv.org/abs/2102.11533) (ICLR 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/proteins_gmt.py)] * **[Equilibrium Aggregation](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.aggr.EquilibriumAggregation.html)** from Bartunov *et al.*: [](https://arxiv.org/abs/2202.12795) (UAI 2022) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/equilibrium_median.py)]
**GNN models:** Our supported GNN models incorporate multiple message passing layers, and users can directly use these pre-defined models to make predictions on graphs. Unlike simple stacking of GNN layers, these models could involve pre-processing, additional learnable parameters, skip connections, graph coarsening, etc. * **[SchNet](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.SchNet.html)** from SchΓΌtt *et al.*: [SchNet: A Continuous-filter Convolutional Neural Network for Modeling Quantum Interactions](https://arxiv.org/abs/1706.08566) (NIPS 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/qm9_pretrained_schnet.py)] * **[DimeNet](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.DimeNet.html)** and **[DimeNetPlusPlus](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.DimeNetPlusPlus.html)** from Klicpera *et al.*: [Directional Message Passing for Molecular Graphs](https://arxiv.org/abs/2003.03123) (ICLR 2020) and [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://arxiv.org/abs/2011.14115) (NeurIPS-W 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/qm9_pretrained_dimenet.py)] * **[Node2Vec](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.Node2Vec.html)** from Grover and Leskovec: [node2vec: Scalable Feature Learning for Networks](https://arxiv.org/abs/1607.00653) (KDD 2016) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/node2vec.py)] * **[Deep Graph Infomax](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.DeepGraphInfomax.html)** from VeličkoviΔ‡ *et al.*: [Deep Graph Infomax](https://arxiv.org/abs/1809.10341) (ICLR 2019) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/infomax_transductive.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/infomax_inductive.py)] * **Deep Multiplex Graph Infomax** from Park *et al.*: [Unsupervised Attributed Multiplex Network Embedding](https://arxiv.org/abs/1911.06750) (AAAI 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/dmgi_unsup.py)] * **[Masked Label Prediction](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.MaskLabel.html)** from Shi *et al.*: [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://arxiv.org/abs/2009.03509) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/unimp_arxiv.py)] * **[PMLP](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.PMLP.html)** from Yang *et al.*: [Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs](https://arxiv.org/abs/2212.09034) (ICLR 2023)
Expand to see all implemented GNN models... * **[Jumping Knowledge](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.JumpingKnowledge.html)** from Xu *et al.*: [Representation Learning on Graphs with Jumping Knowledge Networks](https://arxiv.org/abs/1806.03536) (ICML 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/gin.py#L54-L106)] * A **[MetaLayer](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.MetaLayer.html)** for building any kind of graph network similar to the [TensorFlow Graph Nets library](https://github.com/deepmind/graph_nets) from Battaglia *et al.*: [Relational Inductive Biases, Deep Learning, and Graph Networks](https://arxiv.org/abs/1806.01261) (CoRR 2018) * **[MetaPath2Vec](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.MetaPath2Vec.html)** from Dong *et al.*: [metapath2vec: Scalable Representation Learning for Heterogeneous Networks](https://ericdongyx.github.io/papers/KDD17-dong-chawla-swami-metapath2vec.pdf) (KDD 2017) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/metapath2vec.py)] * All variants of **[Graph Autoencoders](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GAE.html)** and **[Variational Autoencoders](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.VGAE.html)** from: * [Variational Graph Auto-Encoders](https://arxiv.org/abs/1611.07308) from Kipf and Welling (NIPS-W 2016) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/autoencoder.py)] * [Adversarially Regularized Graph Autoencoder for Graph Embedding](https://arxiv.org/abs/1802.04407) from Pan *et al.* (IJCAI 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/argva_node_clustering.py)] * [Simple and Effective Graph Autoencoders with One-Hop Linear Models](https://arxiv.org/abs/2001.07614) from Salha *et al.* (ECML 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/autoencoder.py)] * **[SEAL](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/seal_link_pred.py)** from Zhang and Chen: [Link Prediction Based on Graph Neural Networks](https://arxiv.org/pdf/1802.09691.pdf) (NeurIPS 2018) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/seal_link_pred.py)] * **[RENet](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.RENet.html)** from Jin *et al.*: [Recurrent Event Network for Reasoning over Temporal Knowledge Graphs](https://arxiv.org/abs/1904.05530) (ICLR-W 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/renet.py)] * **[GraphUNet](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GraphUNet.html)** from Gao and Ji: [Graph U-Nets](https://arxiv.org/abs/1905.05178) (ICML 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_unet.py)] * **[AttentiveFP](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.AttentiveFP.html)** from Xiong *et al.*: [Pushing the Boundaries of Molecular Representation for Drug Discovery with the Graph Attention Mechanism](https://pubs.acs.org/doi/10.1021/acs.jmedchem.9b00959) (J. Med. Chem. 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/attentive_fp.py)] * **[DeepGCN](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.DeepGCNLayer.html)** and the **[GENConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GENConv.html)** from Li *et al.*: [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://arxiv.org/abs/1904.03751) (ICCV 2019) and [DeeperGCN: All You Need to Train Deeper GCNs](https://arxiv.org/abs/2006.07739) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/ogbn_proteins_deepgcn.py)] * **[RECT](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.RECT_L.html)** from Wang *et al.*: [Network Embedding with Completely-imbalanced Labels](https://ieeexplore.ieee.org/document/8979355) (TKDE 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/rect.py)] * **[GNNExplainer](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.explain.algorithm.GNNExplainer.html)** from Ying *et al.*: [GNNExplainer: Generating Explanations for Graph Neural Networks](https://arxiv.org/abs/1903.03894) (NeurIPS 2019) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/explain/gnn_explainer.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/explain/gnn_explainer_ba_shapes.py), [**Example3**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/explain/gnn_explainer_link_pred.py)] * **Graph-less Neural Networks** from Zhang *et al.*: [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://arxiv.org/abs/2110.08727) (CoRR 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/glnn.py)] * **[LINKX](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.LINKX.html)** from Lim *et al.*: [Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods](https://arxiv.org/abs/2110.14446) (NeurIPS 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/linkx.py)] * **[RevGNN](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GroupAddRev.html)** from Li *et al.*: [Training Graph Neural with 1000 Layers](https://arxiv.org/abs/2106.07476) (ICML 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/rev_gnn.py)] * **[TransE](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.kge.TransE.html)** from Bordes *et al.*: [Translating Embeddings for Modeling Multi-Relational Data](https://proceedings.neurips.cc/paper/2013/file/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf) (NIPS 2013) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/kge_fb15k_237.py)] * **[ComplEx](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.kge.ComplEx.html)** from Trouillon *et al.*: [Complex Embeddings for Simple Link Prediction](https://arxiv.org/abs/1606.06357) (ICML 2016) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/kge_fb15k_237.py)] * **[DistMult](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.kge.DistMult.html)** from Yang *et al.*: [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://arxiv.org/abs/1412.6575) (ICLR 2015) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/kge_fb15k_237.py)] * **[RotatE](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.kge.RotatE.html)** from Sun *et al.*: [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://arxiv.org/abs/1902.10197) (ICLR 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/kge_fb15k_237.py)]
**GNN operators and utilities:** PyG comes with a rich set of neural network operators that are commonly used in many GNN models. They follow an extensible design: It is easy to apply these operators and graph utilities to existing GNN layers and models to further enhance model performance. * **[DropEdge](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.dropout_edge)** from Rong *et al.*: [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://openreview.net/forum?id=Hkx1qkrKPr) (ICLR 2020) * **[DropNode](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.dropout_node)**, **[MaskFeature](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.mask_feature)** and **[AddRandomEdge](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.add_random_edge)** from You *et al.*: [Graph Contrastive Learning with Augmentations](https://arxiv.org/abs/2010.13902) (NeurIPS 2020) * **[DropPath](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.dropout_path)** from Li *et al.*: [MaskGAE: Masked Graph Modeling Meets Graph Autoencoders](https://arxiv.org/abs/2205.10053) (arXiv 2022) * **[ShuffleNode](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.shuffle_node)** from VeličkoviΔ‡ *et al.*: [Deep Graph Infomax](https://arxiv.org/abs/1809.10341) (ICLR 2019) * **[GraphNorm](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.norm.GraphNorm.html)** from Cai *et al.*: [GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training](https://proceedings.mlr.press/v139/cai21e.html) (ICML 2021) * **[GDC](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.transforms.GDC.html)** from Klicpera *et al.*: [Diffusion Improves Graph Learning](https://arxiv.org/abs/1911.05485) (NeurIPS 2019) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/gcn.py)]
Expand to see all implemented GNN operators and utilities... * **[GraphSizeNorm](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.norm.GraphSizeNorm.html)** from Dwivedi *et al.*: [Benchmarking Graph Neural Networks](https://arxiv.org/abs/2003.00982) (CoRR 2020) * **[PairNorm](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.norm.PairNorm.html)** from Zhao and Akoglu: [PairNorm: Tackling Oversmoothing in GNNs](https://arxiv.org/abs/1909.12223) (ICLR 2020) * **[MeanSubtractionNorm](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.norm.MeanSubtractionNorm.html)** from Yang *et al.*: [Revisiting "Over-smoothing" in Deep GCNs](https://arxiv.org/abs/2003.13663) (CoRR 2020) * **[DiffGroupNorm](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.norm.DiffGroupNorm.html)** from Zhou *et al.*: [Towards Deeper Graph Neural Networks with Differentiable Group Normalization](https://arxiv.org/abs/2006.06972) (NeurIPS 2020) * **[Tree Decomposition](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.tree_decomposition)** from Jin *et al.*: [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://arxiv.org/abs/1802.04364) (ICML 2018) * **[TGN](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.TGNMemory.html)** from Rossi *et al.*: [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://arxiv.org/abs/2006.10637) (GRL+ 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/tgn.py)] * **[Weisfeiler Lehman Operator](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.WLConv.html)** from Weisfeiler and Lehman: [A Reduction of a Graph to a Canonical Form and an Algebra Arising During this Reduction](https://www.iti.zcu.cz/wl2018/pdf/wl_paper_translation.pdf) (Nauchno-Technicheskaya Informatsia 1968) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/wl_kernel.py)] * **[Continuous Weisfeiler Lehman Operator](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.WLConvContinuous.html)** from Togninalli *et al.*: [Wasserstein Weisfeiler-Lehman Graph Kernels](https://arxiv.org/abs/1906.01277) (NeurIPS 2019) * **[Label Propagation](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.LabelPropagation.html)** from Zhu and Ghahramani: [Learning from Labeled and Unlabeled Data with Label Propagation](http://mlg.eng.cam.ac.uk/zoubin/papers/CMU-CALD-02-107.pdf) (CMU-CALD 2002) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/label_prop.py)] * **[Local Degree Profile](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.transforms.LocalDegreeProfile)** from Cai and Wang: [A Simple yet Effective Baseline for Non-attribute Graph Classification](https://arxiv.org/abs/1811.03508) (CoRR 2018) * **[CorrectAndSmooth](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.CorrectAndSmooth.html)** from Huang *et al.*: [Combining Label Propagation And Simple Models Out-performs Graph Neural Networks](https://arxiv.org/abs/2010.13993) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/correct_and_smooth.py)] * **[Gini](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.functional.gini.html)** and **[BRO](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.functional.bro.html)** regularization from Henderson *et al.*: [Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity](https://arxiv.org/abs/2105.04854) (ICML 2021) * **[RootedEgoNets](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.transforms.RootedEgoNets)** and **[RootedRWSubgraph](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.transforms.RootedRWSubgraph)** from Zhao *et al.*: [From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness](https://arxiv.org/abs/2110.03753) (ICLR 2022) * **[FeaturePropagation](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.transforms.FeaturePropagation)** from Rossi *et al.*: [On the Unreasonable Effectiveness of Feature Propagation in Learning on Graphs with Missing Node Features](https://arxiv.org/abs/2111.12128) (CoRR 2021)
**Scalable GNNs:** PyG supports the implementation of Graph Neural Networks that can scale to large-scale graphs. Such application is challenging since the entire graph, its associated features and the GNN parameters cannot fit into GPU memory. Many state-of-the-art scalability approaches tackle this challenge by sampling neighborhoods for mini-batch training, graph clustering and partitioning, or by using simplified GNN models. These approaches have been implemented in PyG, and can benefit from the above GNN layers, operators and models. * **[NeighborLoader](https://pytorch-geometric.readthedocs.io/en/latest/modules/loader.html#torch_geometric.loader.NeighborLoader)** from Hamilton *et al.*: [Inductive Representation Learning on Large Graphs](https://arxiv.org/abs/1706.02216) (NIPS 2017) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/reddit.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/ogbn_products_sage.py), [**Example3**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/ogbn_products_gat.py), [**Example4**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/to_hetero_mag.py)] * **[ClusterGCN](https://pytorch-geometric.readthedocs.io/en/latest/modules/loader.html#torch_geometric.loader.ClusterLoader)** from Chiang *et al.*: [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://arxiv.org/abs/1905.07953) (KDD 2019) [[**Example1**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/cluster_gcn_reddit.py), [**Example2**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/cluster_gcn_ppi.py)] * **[GraphSAINT](https://pytorch-geometric.readthedocs.io/en/latest/modules/loader.html#torch_geometric.loader.GraphSAINTSampler)** from Zeng *et al.*: [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://arxiv.org/abs/1907.04931) (ICLR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_saint.py)]
Expand to see all implemented scalable GNNs... * **[ShaDow](https://pytorch-geometric.readthedocs.io/en/latest/modules/loader.html#torch_geometric.loader.ShaDowKHopSampler)** from Zeng *et al.*: [Decoupling the Depth and Scope of Graph Neural Networks](https://arxiv.org/abs/2201.07858) (NeurIPS 2021) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/shadow.py)] * **[SIGN](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.transforms.SIGN.html)** from Rossi *et al.*: [SIGN: Scalable Inception Graph Neural Networks](https://arxiv.org/abs/2004.11198) (CoRR 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/sign.py)] * **[HGTLoader](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.loader.HGTLoader.html)** from Hu *et al.*: [Heterogeneous Graph Transformer](https://arxiv.org/abs/2003.01332) (WWW 2020) [[**Example**](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/to_hetero_mag.py)]
## Installation PyG is available for Python 3.7 to Python 3.11. ### Anaconda You can now install PyG via [Anaconda](https://anaconda.org/pyg/pyg) for all major OS/PyTorch/CUDA combinations πŸ€— If you have not yet installed PyTorch, install it via `conda` as described in the [official PyTorch documentation](https://pytorch.org/get-started/locally/). Given that you have PyTorch installed (`>=1.8.0`), simply run ``` conda install pyg -c pyg ``` ### PyPi From **PyG 2.3** onwards, you can install and use PyG **without any external library** required except for PyTorch. For this, simply run ``` pip install torch_geometric ``` PyG 2.3 requires that at least PyTorch 1.12 is installed. ### Additional Libraries If you want to utilize the full set of features from PyG, there exists several additional libraries you may want to install: * **[`pyg-lib`](https://github.com/pyg-team/pyg-lib)**: Heterogeneous GNN operators and graph sampling routines * **[`torch-scatter`](https://github.com/rusty1s/pytorch_scatter)**: Accelerated and efficient sparse reductions * **[`torch-sparse`](https://github.com/rusty1s/pytorch_sparse)**: [`SparseTensor`](https://pytorch-geometric.readthedocs.io/en/latest/advanced/sparse_tensor.html) support * **[`torch-cluster`](https://github.com/rusty1s/pytorch_cluster)**: Graph clustering routines * **[`torch-spline-conv`](https://github.com/rusty1s/pytorch_spline_conv)**: [`SplineConv`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.SplineConv.html) support These packages come with their own CPU and GPU kernel implementations based on the [PyTorch C++/CUDA/hip(ROCm) extension interface](https://github.com/pytorch/extension-cpp). For a basic usage of PyG, these dependencies are **fully optional**. We recommend to start with a minimal installation, and install additional dependencies once you start to actually need them. For ease of installation of these extensions, we provide `pip` wheels for all major OS/PyTorch/CUDA combinations, see [here](https://data.pyg.org/whl). #### PyTorch 2.0 To install the binaries for PyTorch 2.0.0, simply run ``` pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.0.0+${CUDA}.html ``` where `${CUDA}` should be replaced by either `cpu`, `cu117`, or `cu118` depending on your PyTorch installation. | | `cpu` | `cu117` | `cu118` | |-------------|-------|---------|---------| | **Linux** | βœ… | βœ… | βœ… | | **Windows** | βœ… | βœ… | βœ… | | **macOS** | βœ… | | | #### PyTorch 1.13 To install the binaries for PyTorch 1.13.0, simply run ``` pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-1.13.0+${CUDA}.html ``` where `${CUDA}` should be replaced by either `cpu`, `cu116`, or `cu117` depending on your PyTorch installation. | | `cpu` | `cu116` | `cu117` | |-------------|-------|---------|---------| | **Linux** | βœ… | βœ… | βœ… | | **Windows** | βœ… | βœ… | βœ… | | **macOS** | βœ… | | | **Note:** Binaries of older versions are also provided for PyTorch 1.4.0, PyTorch 1.5.0, PyTorch 1.6.0, PyTorch 1.7.0/1.7.1, PyTorch 1.8.0/1.8.1, PyTorch 1.9.0, PyTorch 1.10.0/1.10.1/1.10.2, PyTorch 1.11.0 and PyTorch 1.12.0/1.12.1 (following the same procedure). **For older versions, you might need to explicitly specify the latest supported version number** or install via `pip install --no-index` in order to prevent a manual installation from source. You can look up the latest supported version number [here](https://data.pyg.org/whl). ### Nightly and Master In case you want to experiment with the latest PyG features which are not fully released yet, either install the **nightly version** of PyG via ``` pip install pyg-nightly ``` or install PyG **from master** via ``` pip install git+https://github.com/pyg-team/pytorch_geometric.git ``` ### ROCm Wheels The external [`pyg-rocm-build` repository](https://github.com/Looong01/pyg-rocm-build) provides wheels and detailed instructions on how to install PyG for ROCm. If you have any questions about it, please open an issue [here](https://github.com/Looong01/pyg-rocm-build/issues). ## Cite Please cite [our paper](https://arxiv.org/abs/1903.02428) (and the respective papers of the methods used) if you use this code in your own work: ``` @inproceedings{Fey/Lenssen/2019, title={Fast Graph Representation Learning with {PyTorch Geometric}}, author={Fey, Matthias and Lenssen, Jan E.}, booktitle={ICLR Workshop on Representation Learning on Graphs and Manifolds}, year={2019}, } ``` Feel free to [email us](mailto:matthias.fey@tu-dortmund.de) if you wish your work to be listed in the [external resources](https://pytorch-geometric.readthedocs.io/en/latest/external/resources.html). If you notice anything unexpected, please open an [issue](https://github.com/pyg-team/pytorch_geometric/issues) and let us know. If you have any questions or are missing a specific feature, feel free [to discuss them with us](https://github.com/pyg-team/pytorch_geometric/discussions). We are motivated to constantly make PyG even better. pytorch/benchmark/000077500000000000000000000000001444262304300145025ustar00rootroot00000000000000pytorch/benchmark/README.md000066400000000000000000000014051444262304300157610ustar00rootroot00000000000000# PyG Benchmark Suite This benchmark suite provides evaluation scripts for **[semi-supervised node classification](https://github.com/pyg-team/pytorch_geometric/tree/master/benchmark/citation)**, **[graph classification](https://github.com/pyg-team/pytorch_geometric/tree/master/benchmark/kernel)**, and **[point cloud classification](https://github.com/pyg-team/pytorch_geometric/tree/master/benchmark/points)** and **[runtimes](https://github.com/pyg-team/pytorch_geometric/tree/master/benchmark/runtime)** in order to compare various methods in homogeneous evaluation scenarios. In particular, we take care to avoid to perform hyperparameter and model selection on the test set and instead use an additional validation set. ## Installation ``` $ pip install -e . ``` pytorch/benchmark/citation/000077500000000000000000000000001444262304300163145ustar00rootroot00000000000000pytorch/benchmark/citation/README.md000066400000000000000000000017311444262304300175750ustar00rootroot00000000000000# Semi-supervised Node Classification Evaluation scripts for various methods on the Cora, CiteSeer and PubMed citation networks. Each experiment is repeated 100 times on either a fixed train/val/test split or on multiple random splits: * **[GCN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/gcn.py)**: `python gcn.py` * **[GAT](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/gat.py)**: `python gat.py` * **[Cheby](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/cheb.py)**: `python cheb.py` * **[SGC](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/sgc.py)**: `python sgc.py` * **[ARMA](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/arma.py)**: `python arma.py` * **[APPNP](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/citation/appnp.py)**: `python appnp.py` Run the whole test suite via ``` $ ./run.sh ``` pytorch/benchmark/citation/__init__.py000066400000000000000000000002671444262304300204320ustar00rootroot00000000000000from .datasets import get_planetoid_dataset from .train_eval import random_planetoid_splits, run __all__ = [ 'get_planetoid_dataset', 'random_planetoid_splits', 'run', ] pytorch/benchmark/citation/appnp.py000066400000000000000000000046451444262304300200150ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch.nn import Linear from torch_geometric.nn import APPNP from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=10) parser.add_argument('--hidden', type=int, default=64) parser.add_argument('--dropout', type=float, default=0.5) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--K', type=int, default=10) parser.add_argument('--alpha', type=float, default=0.1) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.lin1 = Linear(dataset.num_features, args.hidden) self.lin2 = Linear(args.hidden, dataset.num_classes) self.prop1 = APPNP(args.K, args.alpha) def reset_parameters(self): self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = F.dropout(x, p=args.dropout, training=self.training) x = F.relu(self.lin1(x)) x = F.dropout(x, p=args.dropout, training=self.training) x = self.lin2(x) x = self.prop1(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', APPNP.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/arma.py000066400000000000000000000052771444262304300176210ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch_geometric.nn import ARMAConv from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=1000) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=100) parser.add_argument('--hidden', type=int, default=16) parser.add_argument('--dropout', type=float, default=0.5) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--num_stacks', type=int, default=1) parser.add_argument('--num_layers', type=int, default=1) parser.add_argument('--shared_weights', action='store_true') parser.add_argument('--skip_dropout', type=float, default=0.75) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.conv1 = ARMAConv(dataset.num_features, args.hidden, args.num_stacks, args.num_layers, args.shared_weights, dropout=args.skip_dropout) self.conv2 = ARMAConv(args.hidden, dataset.num_classes, args.num_stacks, args.num_layers, args.shared_weights, dropout=args.skip_dropout) def reset_parameters(self): self.conv1.reset_parameters() self.conv2.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = F.relu(self.conv1(x, edge_index)) x = F.dropout(x, p=args.dropout, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', ARMAConv.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/cheb.py000066400000000000000000000044071444262304300175740ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch_geometric.nn import ChebConv from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=10) parser.add_argument('--hidden', type=int, default=16) parser.add_argument('--dropout', type=float, default=0.5) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--num_hops', type=int, default=3) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.conv1 = ChebConv(dataset.num_features, args.hidden, args.num_hops) self.conv2 = ChebConv(args.hidden, dataset.num_classes, args.num_hops) def reset_parameters(self): self.conv1.reset_parameters() self.conv2.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = F.relu(self.conv1(x, edge_index)) x = F.dropout(x, p=args.dropout, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', ChebConv.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/datasets.py000066400000000000000000000011241444262304300204740ustar00rootroot00000000000000import os.path as osp import torch_geometric.transforms as T from torch_geometric.datasets import Planetoid def get_planetoid_dataset(name, normalize_features=False, transform=None): path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', name) dataset = Planetoid(path, name) if transform is not None and normalize_features: dataset.transform = T.Compose([T.NormalizeFeatures(), transform]) elif normalize_features: dataset.transform = T.NormalizeFeatures() elif transform is not None: dataset.transform = transform return dataset pytorch/benchmark/citation/gat.py000066400000000000000000000050501444262304300174410ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch_geometric.nn import GATConv from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=1000) parser.add_argument('--lr', type=float, default=0.005) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=100) parser.add_argument('--hidden', type=int, default=8) parser.add_argument('--dropout', type=float, default=0.6) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--heads', type=int, default=8) parser.add_argument('--output_heads', type=int, default=1) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.conv1 = GATConv(dataset.num_features, args.hidden, heads=args.heads, dropout=args.dropout) self.conv2 = GATConv(args.hidden * args.heads, dataset.num_classes, heads=args.output_heads, concat=False, dropout=args.dropout) def reset_parameters(self): self.conv1.reset_parameters() self.conv2.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = F.dropout(x, p=args.dropout, training=self.training) x = F.elu(self.conv1(x, edge_index)) x = F.dropout(x, p=args.dropout, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', GATConv.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/gcn.py000066400000000000000000000042561444262304300174440ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch_geometric.nn import GCNConv from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=10) parser.add_argument('--hidden', type=int, default=16) parser.add_argument('--dropout', type=float, default=0.5) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.conv1 = GCNConv(dataset.num_features, args.hidden) self.conv2 = GCNConv(args.hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() self.conv2.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = F.relu(self.conv1(x, edge_index)) x = F.dropout(x, p=args.dropout, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', GCNConv.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/inference.sh000077500000000000000000000133201444262304300206100ustar00rootroot00000000000000#!/bin/sh echo "Cora" echo "====" echo "GCN" python gcn.py --dataset=Cora --inference python gcn.py --dataset=Cora --random_splits --inference python gcn.py --dataset=Cora --inference --profile python gcn.py --dataset=Cora --random_splits --inference --profile echo "GAT" python gat.py --dataset=Cora --inference python gat.py --dataset=Cora --random_splits --inference python gat.py --dataset=Cora --inference --profile python gat.py --dataset=Cora --random_splits --inference --profile echo "Cheby" python cheb.py --dataset=Cora --num_hops=3 --inference python cheb.py --dataset=Cora --num_hops=3 --random_splits --inference python cheb.py --dataset=Cora --num_hops=3 --inference --profile python cheb.py --dataset=Cora --num_hops=3 --random_splits --inference --profile echo "SGC" python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 --inference python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 --random_splits --inference python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 --inference --profile python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 --random_splits --inference --profile echo "ARMA" python arma.py --dataset=Cora --num_stacks=2 --num_layers=1 --shared_weights=True --inference python arma.py --dataset=Cora --num_stacks=3 --num_layers=1 --shared_weights=True --random_splits --inference python arma.py --dataset=Cora --num_stacks=2 --num_layers=1 --shared_weights=True --inference --profile python arma.py --dataset=Cora --num_stacks=3 --num_layers=1 --shared_weights=True --random_splits --inference --profile echo "APPNP" python appnp.py --dataset=Cora --alpha=0.1 --inference python appnp.py --dataset=Cora --alpha=0.1 --random_splits --inference python appnp.py --dataset=Cora --alpha=0.1 --inference --profile python appnp.py --dataset=Cora --alpha=0.1 --random_splits --inference --profile echo "CiteSeer" echo "========" echo "GCN" python gcn.py --dataset=CiteSeer --inference python gcn.py --dataset=CiteSeer --random_splits --inference python gcn.py --dataset=CiteSeer --inference --profile python gcn.py --dataset=CiteSeer --random_splits --inference --profile echo "GAT" python gat.py --dataset=CiteSeer --inference python gat.py --dataset=CiteSeer --random_splits --inference python gat.py --dataset=CiteSeer --inference --profile python gat.py --dataset=CiteSeer --random_splits --inference --profile echo "Cheby" python cheb.py --dataset=CiteSeer --num_hops=2 --inference python cheb.py --dataset=CiteSeer --num_hops=3 --random_splits --inference python cheb.py --dataset=CiteSeer --num_hops=2 --inference --profile python cheb.py --dataset=CiteSeer --num_hops=3 --random_splits --inference --profile echo "SGC" python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 --inference python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 --random_splits --inference python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 --inference --profile python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 --random_splits --inference --profile echo "ARMA" python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights=True --inference python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights=True --random_splits --inference python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights=True --inference --profile python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights=True --random_splits --inference --profile echo "APPNP" python appnp.py --dataset=CiteSeer --alpha=0.1 --inference python appnp.py --dataset=CiteSeer --alpha=0.1 --random_splits --inference python appnp.py --dataset=CiteSeer --alpha=0.1 --inference --profile python appnp.py --dataset=CiteSeer --alpha=0.1 --random_splits --inference --profile echo "PubMed" echo "======" echo "GCN" python gcn.py --dataset=PubMed --inference python gcn.py --dataset=PubMed --random_splits --inference python gcn.py --dataset=PubMed --inference --profile python gcn.py --dataset=PubMed --random_splits --inference --profile echo "GAT" python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 --inference python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 --random_splits --inference python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 --inference --profile python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 --random_splits --inference --profile echo "Cheby" python cheb.py --dataset=PubMed --num_hops=2 --inference python cheb.py --dataset=PubMed --num_hops=2 --random_splits --inference python cheb.py --dataset=PubMed --num_hops=2 --inference --profile python cheb.py --dataset=PubMed --num_hops=2 --random_splits --inference --profile echo "SGC" python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 --inference python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 --random_splits --inference python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 --inference --profile python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 --random_splits --inference --profile echo "ARMA" python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0 --inference python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0.5 --random_splits --inference python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0 --inference --profile python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0.5 --random_splits --inference --profile echo "APPNP" python appnp.py --dataset=PubMed --alpha=0.1 --inference python appnp.py --dataset=PubMed --alpha=0.1 --random_splits --inference python appnp.py --dataset=PubMed --alpha=0.1 --inference --profile python appnp.py --dataset=PubMed --alpha=0.1 --random_splits --inference --profile pytorch/benchmark/citation/run.sh000077500000000000000000000046241444262304300174650ustar00rootroot00000000000000#!/bin/sh echo "Cora" echo "====" echo "GCN" python gcn.py --dataset=Cora python gcn.py --dataset=Cora --random_splits echo "GAT" python gat.py --dataset=Cora python gat.py --dataset=Cora --random_splits echo "Cheby" python cheb.py --dataset=Cora --num_hops=3 python cheb.py --dataset=Cora --num_hops=3 --random_splits echo "SGC" python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 python sgc.py --dataset=Cora --K=3 --weight_decay=0.0005 --random_splits echo "ARMA" python arma.py --dataset=Cora --num_stacks=2 --num_layers=1 --shared_weights python arma.py --dataset=Cora --num_stacks=3 --num_layers=1 --shared_weights --random_splits echo "APPNP" python appnp.py --dataset=Cora --alpha=0.1 python appnp.py --dataset=Cora --alpha=0.1 --random_splits echo "CiteSeer" echo "========" echo "GCN" python gcn.py --dataset=CiteSeer python gcn.py --dataset=CiteSeer --random_splits echo "GAT" python gat.py --dataset=CiteSeer python gat.py --dataset=CiteSeer --random_splits echo "Cheby" python cheb.py --dataset=CiteSeer --num_hops=2 python cheb.py --dataset=CiteSeer --num_hops=3 --random_splits echo "SGC" python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 python sgc.py --dataset=CiteSeer --K=2 --weight_decay=0.005 --random_splits echo "ARMA" python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights python arma.py --dataset=CiteSeer --num_stacks=3 --num_layers=1 --shared_weights --random_splits echo "APPNP" python appnp.py --dataset=CiteSeer --alpha=0.1 python appnp.py --dataset=CiteSeer --alpha=0.1 --random_splits echo "PubMed" echo "======" echo "GCN" python gcn.py --dataset=PubMed python gcn.py --dataset=PubMed --random_splits echo "GAT" python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 python gat.py --dataset=PubMed --lr=0.01 --weight_decay=0.001 --output_heads=8 --random_splits echo "Cheby" python cheb.py --dataset=PubMed --num_hops=2 python cheb.py --dataset=PubMed --num_hops=2 --random_splits echo "SGC" python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 python sgc.py --dataset=PubMed --K=2 --weight_decay=0.0005 --random_splits echo "ARMA" python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0 python arma.py --dataset=PubMed --num_stacks=2 --num_layers=1 --skip_dropout=0.5 --random_splits echo "APPNP" python appnp.py --dataset=PubMed --alpha=0.1 python appnp.py --dataset=PubMed --alpha=0.1 --random_splits pytorch/benchmark/citation/sgc.py000066400000000000000000000037211444262304300174450ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from citation import get_planetoid_dataset, random_planetoid_splits, run from torch_geometric.nn import SGConv from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--dataset', type=str, required=True) parser.add_argument('--random_splits', action='store_true') parser.add_argument('--runs', type=int, default=100) parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--lr', type=float, default=0.1) parser.add_argument('--weight_decay', type=float, default=0.0005) parser.add_argument('--early_stopping', type=int, default=10) parser.add_argument('--no_normalize_features', action='store_true') parser.add_argument('--K', type=int, default=2) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, dataset): super().__init__() self.conv1 = SGConv(dataset.num_features, dataset.num_classes, K=args.K, cached=True) def reset_parameters(self): self.conv1.reset_parameters() def forward(self, data): x, edge_index = data.x, data.edge_index x = self.conv1(x, edge_index) return F.log_softmax(x, dim=1) dataset = get_planetoid_dataset(args.dataset, not args.no_normalize_features) permute_masks = random_planetoid_splits if args.random_splits else None run(dataset, Net(dataset), args.runs, args.epochs, args.lr, args.weight_decay, args.early_stopping, args.inference, args.profile, args.bf16, args.compile, permute_masks) if args.profile: rename_profile_file('citation', SGConv.__name__, args.dataset, str(args.random_splits), 'inference' if args.inference else 'train') pytorch/benchmark/citation/statistics.py000066400000000000000000000007131444262304300210610ustar00rootroot00000000000000from citation import get_planetoid_dataset def print_dataset(dataset): data = dataset[0] print('Name', dataset) print('Nodes', data.num_nodes) print('Edges', data.num_edges // 2) print('Features', dataset.num_features) print('Classes', dataset.num_classes) print('Label rate', data.train_mask.sum().item() / data.num_nodes) print() for name in ['Cora', 'CiteSeer', 'PubMed']: print_dataset(get_planetoid_dataset(name)) pytorch/benchmark/citation/train_eval.py000066400000000000000000000124641444262304300210210ustar00rootroot00000000000000import time import torch import torch.nn.functional as F from torch import tensor from torch.optim import Adam import torch_geometric from torch_geometric.profile import timeit, torch_profile from torch_geometric.utils import index_to_mask device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') def random_planetoid_splits(data, num_classes): # Set new random planetoid splits: # * 20 * num_classes labels for training # * 500 labels for validation # * 1000 labels for testing indices = [] for i in range(num_classes): index = (data.y == i).nonzero().view(-1) index = index[torch.randperm(index.size(0))] indices.append(index) train_index = torch.cat([i[:20] for i in indices], dim=0) rest_index = torch.cat([i[20:] for i in indices], dim=0) rest_index = rest_index[torch.randperm(rest_index.size(0))] data.train_mask = index_to_mask(train_index, size=data.num_nodes) data.val_mask = index_to_mask(rest_index[:500], size=data.num_nodes) data.test_mask = index_to_mask(rest_index[500:1500], size=data.num_nodes) return data def run_train(dataset, model, runs, epochs, lr, weight_decay, early_stopping, profiling, use_compile, permute_masks=None, logger=None): val_losses, accs, durations = [], [], [] if use_compile: model = torch_geometric.compile(model) for run in range(runs): data = dataset[0] if permute_masks is not None: data = permute_masks(data, dataset.num_classes) data = data.to(device) model.to(device).reset_parameters() optimizer = Adam(model.parameters(), lr=lr, weight_decay=weight_decay) if torch.cuda.is_available(): torch.cuda.synchronize() t_start = time.perf_counter() best_val_loss = float('inf') test_acc = 0 val_loss_history = [] for epoch in range(1, epochs + 1): if run == runs - 1 and epoch == epochs: with timeit(): train(model, optimizer, data) else: train(model, optimizer, data) eval_info = evaluate(model, data) eval_info['epoch'] = epoch if logger is not None: logger(eval_info) if eval_info['val_loss'] < best_val_loss: best_val_loss = eval_info['val_loss'] test_acc = eval_info['test_acc'] val_loss_history.append(eval_info['val_loss']) if early_stopping > 0 and epoch > epochs // 2: tmp = tensor(val_loss_history[-(early_stopping + 1):-1]) if eval_info['val_loss'] > tmp.mean().item(): break if torch.cuda.is_available(): torch.cuda.synchronize() t_end = time.perf_counter() val_losses.append(best_val_loss) accs.append(test_acc) durations.append(t_end - t_start) loss, acc, duration = tensor(val_losses), tensor(accs), tensor(durations) print(f'Val Loss: {float(loss.mean()):.4f}, ' f'Test Accuracy: {float(acc.mean()):.3f} Β± {float(acc.std()):.3f}, ' f'Duration: {float(duration.mean()):.3f}s') if profiling: with torch_profile(): train(model, optimizer, data) @torch.no_grad() def run_inference(dataset, model, epochs, profiling, bf16, use_compile, permute_masks=None, logger=None): data = dataset[0] if permute_masks is not None: data = permute_masks(data, dataset.num_classes) data = data.to(device) model.to(device).reset_parameters() if use_compile: model = torch_geometric.compile(model) if torch.cuda.is_available(): amp = torch.cuda.amp.autocast(enabled=False) else: amp = torch.cpu.amp.autocast(enabled=bf16) if bf16: data.x = data.x.to(torch.bfloat16) with amp: for epoch in range(1, epochs + 1): if epoch == epochs: with timeit(): inference(model, data) else: inference(model, data) if profiling: with torch_profile(): inference(model, data) def run(dataset, model, runs, epochs, lr, weight_decay, early_stopping, inference, profiling, bf16, use_compile, permute_masks=None, logger=None): if not inference: run_train(dataset, model, runs, epochs, lr, weight_decay, early_stopping, profiling, use_compile, permute_masks, logger) else: run_inference(dataset, model, epochs, profiling, bf16, use_compile, permute_masks, logger) def train(model, optimizer, data): model.train() optimizer.zero_grad() out = model(data) loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask]) loss.backward() optimizer.step() @torch.no_grad() def evaluate(model, data): model.eval() out = model(data) outs = {} for key in ['train', 'val', 'test']: mask = data[f'{key}_mask'] loss = float(F.nll_loss(out[mask], data.y[mask])) pred = out[mask].argmax(1) acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item() outs[f'{key}_loss'] = loss outs[f'{key}_acc'] = acc return outs @torch.no_grad() def inference(model, data): model.eval() model(data) pytorch/benchmark/inference/000077500000000000000000000000001444262304300164405ustar00rootroot00000000000000pytorch/benchmark/inference/README.md000066400000000000000000000032041444262304300177160ustar00rootroot00000000000000# Inference Benchmark ## Environment setup 1. Confirm that PyG is properly installed. 2. Install dataset package: ```bash pip install ogb ``` 3. Install `jemalloc` for performance benchmark: ```bash cd ${workspace} git clone https://github.com/jemalloc/jemalloc.git cd jemalloc git checkout 5.2.1 ./autogen.sh ./configure --prefix=${workspace}/jemalloc-bin make make install ``` ## Running benchmark 1. Set environment variables: ```bash source activate env_name export DNNL_PRIMITIVE_CACHE_CAPACITY=1024 export KMP_BLOCKTIME=1 export KMP_AFFINITY=granularity=fine,compact,1,0 jemalloc_lib=${workspace}/jemalloc-bin/lib/libjemalloc.so export LD_PRELOAD="$jemalloc_lib" export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000" ``` 2. Core binding, *e.g.*, single socket / single core / 4 cores per instance: ```bash OMP_NUM_THREADS=${CORES} numactl -C 0-${LAST_CORE} -m 0 CMD...... ``` 3. Execute benchmarks, *e.g.*: ```bash python -u inference_benchmark.py --datasets=Reddit --models=gcn --eval-batch-sizes=512 --num-layers=2 --num-hidden-channels=64 python -u inference_benchmark.py --datasets=Reddit --models=gcn --eval-batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --use-sparse-tensor python -u inference_benchmark.py --datasets=ogbn-products --models=sage --eval-batch-sizes=512 --num-layers=2 --num-hidden-channels=64 python -u inference_benchmark.py --datasets=ogbn-products --models=sage --eval-batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --use-sparse-tensor ``` pytorch/benchmark/inference/inference_benchmark.py000066400000000000000000000326111444262304300227650ustar00rootroot00000000000000import argparse import warnings from collections import defaultdict from contextlib import nullcontext import torch from benchmark.utils import ( emit_itt, get_dataset_with_transformation, get_model, get_split_masks, save_benchmark_data, test, write_to_csv, ) from torch_geometric.loader import NeighborLoader from torch_geometric.nn import PNAConv from torch_geometric.profile import rename_profile_file, timeit, torch_profile supported_sets = { 'ogbn-mag': ['rgat', 'rgcn'], 'ogbn-products': ['edge_cnn', 'gat', 'gcn', 'pna', 'sage'], 'Reddit': ['edge_cnn', 'gat', 'gcn', 'pna', 'sage'], } @torch.no_grad() def full_batch_inference(model, data): model.eval() if hasattr(data, 'adj_t'): edge_index = data.adj_t else: edge_index = data.edge_index return model(data.x, edge_index) def run(args: argparse.ArgumentParser): csv_data = defaultdict(list) if args.write_csv == 'prof' and not args.profile: warnings.warn("Cannot write profile data to CSV because profiling is " "disabled") # cuda device is not suitable for full batch mode device = torch.device( 'cuda' if not args.full_batch and torch.cuda.is_available() else 'cpu') print('BENCHMARK STARTS') for dataset_name in args.datasets: assert dataset_name in supported_sets.keys( ), f"Dataset {dataset_name} isn't supported." print(f'Dataset: {dataset_name}') load_time = timeit() if args.measure_load_time else nullcontext() with load_time: result = get_dataset_with_transformation(dataset_name, args.root, args.use_sparse_tensor, args.bf16) dataset, num_classes, transformation = result data = dataset.to(device) hetero = True if dataset_name == 'ogbn-mag' else False mask = ('paper', None) if dataset_name == 'ogbn-mag' else None _, _, test_mask = get_split_masks(data, dataset_name) degree = None if args.num_layers != [1] and not hetero and args.num_steps != -1: raise ValueError("Layer-wise inference requires `steps=-1`") if torch.cuda.is_available(): amp = torch.cuda.amp.autocast(enabled=False) else: amp = torch.cpu.amp.autocast(enabled=args.bf16) inputs_channels = data[ 'paper'].num_features if dataset_name == 'ogbn-mag' \ else dataset.num_features for model_name in args.models: if model_name not in supported_sets[dataset_name]: print(f'Configuration of {dataset_name} + {model_name} ' f'not supported. Skipping.') continue with_loader = not args.full_batch or (model_name == 'pna' and degree is None) print(f'Evaluation bench for {model_name}:') for batch_size in args.eval_batch_sizes: num_nodes = data[ 'paper'].num_nodes if hetero else data.num_nodes sampler = torch.utils.data.RandomSampler( range(num_nodes), num_samples=args.num_steps * batch_size ) if args.num_steps != -1 and with_loader else None kwargs = { 'batch_size': batch_size, 'shuffle': False, 'num_workers': args.num_workers, } if not hetero: subgraph_loader = NeighborLoader( data, num_neighbors=[-1], # layer-wise inference input_nodes=mask, sampler=sampler, **kwargs, ) if with_loader else None if args.evaluate and not args.full_batch: test_loader = NeighborLoader( data, num_neighbors=[-1], # layer-wise inference input_nodes=test_mask, sampler=None, **kwargs, ) for layers in args.num_layers: num_neighbors = [args.hetero_num_neighbors] * layers if hetero: # batch-wise inference subgraph_loader = NeighborLoader( data, num_neighbors=num_neighbors, input_nodes=mask, sampler=sampler, **kwargs, ) if with_loader else None if args.evaluate and not args.full_batch: test_loader = NeighborLoader( data, num_neighbors=num_neighbors, input_nodes=test_mask, sampler=None, **kwargs, ) for hidden_channels in args.num_hidden_channels: print('----------------------------------------------') print(f'Batch size={batch_size}, ' f'Layers amount={layers}, ' f'Num_neighbors={num_neighbors}, ' f'Hidden features size={hidden_channels}, ' f'Sparse tensor={args.use_sparse_tensor}') params = { 'inputs_channels': inputs_channels, 'hidden_channels': hidden_channels, 'output_channels': num_classes, 'num_heads': args.num_heads, 'num_layers': layers, } if model_name == 'pna': if degree is None: degree = PNAConv.get_degree_histogram( subgraph_loader) print(f'Calculated degree for {dataset_name}.') params['degree'] = degree model = get_model( model_name, params, metadata=data.metadata() if hetero else None) model = model.to(device) # TODO: Migrate to ModelHubMixin. if args.ckpt_path: state_dict = torch.load(args.ckpt_path) model.load_state_dict(state_dict) model.eval() # Define context manager parameters: if args.cpu_affinity and with_loader: cpu_affinity = subgraph_loader.enable_cpu_affinity( args.loader_cores) else: cpu_affinity = nullcontext() profile = torch_profile( args.export_chrome_trace, csv_data, args.write_csv) if args.profile else nullcontext() itt = emit_itt( ) if args.vtune_profile else nullcontext() if args.full_batch and args.use_sparse_tensor: data = transformation(data) with cpu_affinity, amp, timeit() as time: for _ in range(args.warmup): if args.full_batch: full_batch_inference(model, data) else: model.inference(subgraph_loader, device, progress_bar=True) if args.warmup > 0: time.reset() with itt, profile: if args.full_batch: y = full_batch_inference(model, data) if args.evaluate: mask = data.test_mask pred = y[mask].argmax(1) test_acc = pred.eq(data.y[mask]).sum( ).item() / mask.sum().item() print(f'Full Batch Test Accuracy: \ {test_acc:.4f}') else: y = model.inference( subgraph_loader, device, progress_bar=True, ) if args.evaluate: test_acc = test( model, test_loader, device, hetero, progress_bar=True, ) print(f'Mini Batch Test Accuracy: \ {test_acc:.4f}') if args.profile and args.export_chrome_trace: rename_profile_file(model_name, dataset_name, str(batch_size), str(layers), str(hidden_channels), str(num_neighbors)) total_time = time.duration if args.num_steps != -1: total_num_samples = args.num_steps * batch_size else: total_num_samples = num_nodes throughput = total_num_samples / total_time latency = total_time / total_num_samples * 1000 print(f'Throughput: {throughput:.3f} samples/s') print(f'Latency: {latency:.3f} ms') num_records = 1 if args.write_csv == 'prof': # For profiling with PyTorch, we save the top-5 # most time consuming operations. Therefore, the # same data should be entered for each of them. num_records = 5 for _ in range(num_records): save_benchmark_data( csv_data, batch_size, layers, num_neighbors, hidden_channels, total_time, model_name, dataset_name, args.use_sparse_tensor, ) if args.write_csv: write_to_csv(csv_data, args.write_csv) if __name__ == '__main__': argparser = argparse.ArgumentParser('GNN inference benchmark') add = argparser.add_argument add('--datasets', nargs='+', default=['ogbn-mag', 'ogbn-products', 'Reddit'], type=str) add('--use-sparse-tensor', action='store_true', help='use torch_sparse.SparseTensor as graph storage format') add('--models', nargs='+', default=['edge_cnn', 'gat', 'gcn', 'pna', 'rgat', 'rgcn'], type=str) add('--root', default='../../data', type=str, help='relative path to look for the datasets') add('--eval-batch-sizes', nargs='+', default=[512, 1024, 2048, 4096, 8192], type=int) add('--num-layers', nargs='+', default=[2, 3], type=int) add('--num-hidden-channels', nargs='+', default=[64, 128, 256], type=int) add('--num-heads', default=2, type=int, help='number of hidden attention heads, applies only for gat and rgat') add('--hetero-num-neighbors', default=10, type=int, help='number of neighbors to sample per layer for hetero workloads') add('--num-workers', default=0, type=int) add('--num-steps', default=-1, type=int, help='number of steps, -1 means iterating through all the data') add('--warmup', default=1, type=int) add('--profile', action='store_true') add('--vtune-profile', action='store_true') add('--bf16', action='store_true') add('--cpu-affinity', action='store_true', help='Use DataLoader affinitzation.') add('--loader-cores', nargs='+', default=[], type=int, help="List of CPU core IDs to use for DataLoader workers") add('--measure-load-time', action='store_true') add('--full-batch', action='store_true', help='Use full batch mode') add('--evaluate', action='store_true') add('--ckpt_path', type=str, help='Checkpoint path for loading a model') add('--write-csv', choices=[None, 'bench', 'prof'], default=None, help='Write benchmark or PyTorch profile data to CSV') add('--export-chrome-trace', default=True, type=bool, help='Export chrome trace file. Works only with PyTorch profiler') run(argparser.parse_args()) pytorch/benchmark/kernel/000077500000000000000000000000001444262304300157625ustar00rootroot00000000000000pytorch/benchmark/kernel/README.md000066400000000000000000000034231444262304300172430ustar00rootroot00000000000000# Graph Classification Evaluation script for various methods on [common benchmark datasets](http://graphkernels.cs.tu-dortmund.de) via 10-fold cross validation, where a training fold is randomly sampled to serve as a validation set. Hyperparameter selection is performed for the number of hidden units and the number of layers with respect to the validation set: * **[GCN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/gcn.py)** * **[GraphSAGE](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/graph_sage.py)** * **[GIN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/gin.py)** * **[Graclus](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/graclus.py)** * **[Top-K Pooling](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/top_k.py)** * **[SAG Pooling](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/sag_pool.py)** * **[DiffPool](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/diff_pool.py)** * **[EdgePool](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/edge_pool.py)** * **[GlobalAttention](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/global_attention.py)** * **[Set2Set](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/set2set.py)** * **[SortPool](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/sort_pool.py)** * **[ASAPool](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/kernel/asap.py)** Run (or modify) the whole test suite via ``` $ python main.py ``` For more comprehensive time-measurement and memory usage information, you may use ``` $ python main_performance.py ``` pytorch/benchmark/kernel/__init__.py000066400000000000000000000002371444262304300200750ustar00rootroot00000000000000from .datasets import get_dataset from .train_eval import cross_validation_with_val_set __all__ = [ 'get_dataset', 'cross_validation_with_val_set', ] pytorch/benchmark/kernel/asap.py000066400000000000000000000040711444262304300172620ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import ( ASAPooling, GraphConv, JumpingKnowledge, global_mean_pool, ) class ASAP(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, ratio=0.8, dropout=0): super().__init__() self.conv1 = GraphConv(dataset.num_features, hidden, aggr='mean') self.convs = torch.nn.ModuleList() self.pools = torch.nn.ModuleList() self.convs.extend([ GraphConv(hidden, hidden, aggr='mean') for i in range(num_layers - 1) ]) self.pools.extend([ ASAPooling(hidden, ratio, dropout=dropout) for i in range((num_layers) // 2) ]) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear(num_layers * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() for pool in self.pools: pool.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch edge_weight = None x = F.relu(self.conv1(x, edge_index)) xs = [global_mean_pool(x, batch)] for i, conv in enumerate(self.convs): x = conv(x=x, edge_index=edge_index, edge_weight=edge_weight) x = F.relu(x) xs += [global_mean_pool(x, batch)] if i % 2 == 0 and i < len(self.convs) - 1: pool = self.pools[i // 2] x, edge_index, edge_weight, batch, _ = pool( x=x, edge_index=edge_index, edge_weight=edge_weight, batch=batch) x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/datasets.py000066400000000000000000000040521444262304300201450ustar00rootroot00000000000000import os.path as osp import torch import torch_geometric.transforms as T from torch_geometric.datasets import TUDataset from torch_geometric.utils import degree class NormalizedDegree: def __init__(self, mean, std): self.mean = mean self.std = std def __call__(self, data): deg = degree(data.edge_index[0], dtype=torch.float) deg = (deg - self.mean) / self.std data.x = deg.view(-1, 1) return data def get_dataset(name, sparse=True, cleaned=False): path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', name) dataset = TUDataset(path, name, cleaned=cleaned) dataset.data.edge_attr = None if dataset.data.x is None: max_degree = 0 degs = [] for data in dataset: degs += [degree(data.edge_index[0], dtype=torch.long)] max_degree = max(max_degree, degs[-1].max().item()) if max_degree < 1000: dataset.transform = T.OneHotDegree(max_degree) else: deg = torch.cat(degs, dim=0).to(torch.float) mean, std = deg.mean().item(), deg.std().item() dataset.transform = NormalizedDegree(mean, std) if not sparse: num_nodes = max_num_nodes = 0 for data in dataset: num_nodes += data.num_nodes max_num_nodes = max(data.num_nodes, max_num_nodes) # Filter out a few really large graphs in order to apply DiffPool. if name == 'REDDIT-BINARY': num_nodes = min(int(num_nodes / len(dataset) * 1.5), max_num_nodes) else: num_nodes = min(int(num_nodes / len(dataset) * 5), max_num_nodes) indices = [] for i, data in enumerate(dataset): if data.num_nodes <= num_nodes: indices.append(i) dataset = dataset.copy(torch.tensor(indices)) if dataset.transform is None: dataset.transform = T.ToDense(num_nodes) else: dataset.transform = T.Compose( [dataset.transform, T.ToDense(num_nodes)]) return dataset pytorch/benchmark/kernel/diff_pool.py000066400000000000000000000060561444262304300203040ustar00rootroot00000000000000from math import ceil import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import DenseSAGEConv, JumpingKnowledge, dense_diff_pool class Block(torch.nn.Module): def __init__(self, in_channels, hidden_channels, out_channels, mode='cat'): super().__init__() self.conv1 = DenseSAGEConv(in_channels, hidden_channels) self.conv2 = DenseSAGEConv(hidden_channels, out_channels) self.jump = JumpingKnowledge(mode) if mode == 'cat': self.lin = Linear(hidden_channels + out_channels, out_channels) else: self.lin = Linear(out_channels, out_channels) def reset_parameters(self): self.conv1.reset_parameters() self.conv2.reset_parameters() self.lin.reset_parameters() def forward(self, x, adj, mask=None): x1 = F.relu(self.conv1(x, adj, mask)) x2 = F.relu(self.conv2(x1, adj, mask)) return self.lin(self.jump([x1, x2])) class DiffPool(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, ratio=0.25): super().__init__() num_nodes = ceil(ratio * dataset[0].num_nodes) self.embed_block1 = Block(dataset.num_features, hidden, hidden) self.pool_block1 = Block(dataset.num_features, hidden, num_nodes) self.embed_blocks = torch.nn.ModuleList() self.pool_blocks = torch.nn.ModuleList() for i in range((num_layers // 2) - 1): num_nodes = ceil(ratio * num_nodes) self.embed_blocks.append(Block(hidden, hidden, hidden)) self.pool_blocks.append(Block(hidden, hidden, num_nodes)) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear((len(self.embed_blocks) + 1) * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.embed_block1.reset_parameters() self.pool_block1.reset_parameters() for embed_block, pool_block in zip(self.embed_blocks, self.pool_blocks): embed_block.reset_parameters() pool_block.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, adj, mask = data.x, data.adj, data.mask s = self.pool_block1(x, adj, mask) x = F.relu(self.embed_block1(x, adj, mask)) xs = [x.mean(dim=1)] x, adj, _, _ = dense_diff_pool(x, adj, s, mask) for i, (embed_block, pool_block) in enumerate( zip(self.embed_blocks, self.pool_blocks)): s = pool_block(x, adj) x = F.relu(embed_block(x, adj)) xs.append(x.mean(dim=1)) if i < len(self.embed_blocks) - 1: x, adj, _, _ = dense_diff_pool(x, adj, s) x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/edge_pool.py000066400000000000000000000035151444262304300202750ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import ( EdgePooling, GraphConv, JumpingKnowledge, global_mean_pool, ) class EdgePool(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = GraphConv(dataset.num_features, hidden, aggr='mean') self.convs = torch.nn.ModuleList() self.pools = torch.nn.ModuleList() self.convs.extend([ GraphConv(hidden, hidden, aggr='mean') for i in range(num_layers - 1) ]) self.pools.extend( [EdgePooling(hidden) for i in range((num_layers) // 2)]) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear(num_layers * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() for pool in self.pools: pool.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [global_mean_pool(x, batch)] for i, conv in enumerate(self.convs): x = F.relu(conv(x, edge_index)) xs += [global_mean_pool(x, batch)] if i % 2 == 0 and i < len(self.convs) - 1: pool = self.pools[i // 2] x, edge_index, batch, _ = pool(x, edge_index, batch=batch) x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/gcn.py000066400000000000000000000050561444262304300171110ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import GCNConv, JumpingKnowledge, global_mean_pool class GCN(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = GCNConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(GCNConv(hidden, hidden)) self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) for conv in self.convs: x = F.relu(conv(x, edge_index)) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ class GCNWithJK(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, mode='cat'): super().__init__() self.conv1 = GCNConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(GCNConv(hidden, hidden)) self.jump = JumpingKnowledge(mode) if mode == 'cat': self.lin1 = Linear(num_layers * hidden, hidden) else: self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [x] for conv in self.convs: x = F.relu(conv(x, edge_index)) xs += [x] x = self.jump(xs) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/gin.py000066400000000000000000000154121444262304300171140ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import BatchNorm1d as BN from torch.nn import Linear, ReLU, Sequential from torch_geometric.nn import GINConv, JumpingKnowledge, global_mean_pool class GIN0(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = GINConv( Sequential( Linear(dataset.num_features, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=False) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append( GINConv( Sequential( Linear(hidden, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=False)) self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = self.conv1(x, edge_index) for conv in self.convs: x = conv(x, edge_index) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ class GIN0WithJK(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, mode='cat'): super().__init__() self.conv1 = GINConv( Sequential( Linear(dataset.num_features, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=False) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append( GINConv( Sequential( Linear(hidden, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=False)) self.jump = JumpingKnowledge(mode) if mode == 'cat': self.lin1 = Linear(num_layers * hidden, hidden) else: self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = self.conv1(x, edge_index) xs = [x] for conv in self.convs: x = conv(x, edge_index) xs += [x] x = self.jump(xs) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ class GIN(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = GINConv( Sequential( Linear(dataset.num_features, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=True) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append( GINConv( Sequential( Linear(hidden, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=True)) self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = self.conv1(x, edge_index) for conv in self.convs: x = conv(x, edge_index) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ class GINWithJK(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, mode='cat'): super().__init__() self.conv1 = GINConv( Sequential( Linear(dataset.num_features, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=True) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append( GINConv( Sequential( Linear(hidden, hidden), ReLU(), Linear(hidden, hidden), ReLU(), BN(hidden), ), train_eps=True)) self.jump = JumpingKnowledge(mode) if mode == 'cat': self.lin1 = Linear(num_layers * hidden, hidden) else: self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = self.conv1(x, edge_index) xs = [x] for conv in self.convs: x = conv(x, edge_index) xs += [x] x = self.jump(xs) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/global_attention.py000066400000000000000000000024731444262304300216670ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import AttentionalAggregation, SAGEConv class GlobalAttentionNet(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = SAGEConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(SAGEConv(hidden, hidden)) self.att = AttentionalAggregation(Linear(hidden, 1)) self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.att.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) for conv in self.convs: x = F.relu(conv(x, edge_index)) x = self.att(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/graclus.py000066400000000000000000000035121444262304300177750ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.data import Batch from torch_geometric.nn import ( GraphConv, JumpingKnowledge, global_mean_pool, graclus, max_pool, ) class Graclus(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = GraphConv(dataset.num_features, hidden, aggr='mean') self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(GraphConv(hidden, hidden, aggr='mean')) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear(num_layers * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [global_mean_pool(x, batch)] for i, conv in enumerate(self.convs): x = F.relu(conv(x, edge_index)) xs += [global_mean_pool(x, batch)] if i % 2 == 0 and i < len(self.convs) - 1: cluster = graclus(edge_index, num_nodes=x.size(0)) data = Batch(x=x, edge_index=edge_index, batch=batch) data = max_pool(cluster, data) x, edge_index, batch = data.x, data.edge_index, data.batch x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/graph_sage.py000066400000000000000000000050771444262304300204450ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import JumpingKnowledge, SAGEConv, global_mean_pool class GraphSAGE(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = SAGEConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(SAGEConv(hidden, hidden)) self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) for conv in self.convs: x = F.relu(conv(x, edge_index)) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ class GraphSAGEWithJK(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, mode='cat'): super().__init__() self.conv1 = SAGEConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(SAGEConv(hidden, hidden)) self.jump = JumpingKnowledge(mode) if mode == 'cat': self.lin1 = Linear(num_layers * hidden, hidden) else: self.lin1 = Linear(hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.jump.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [x] for conv in self.convs: x = F.relu(conv(x, edge_index)) xs += [x] x = self.jump(xs) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/main.py000066400000000000000000000047411444262304300172660ustar00rootroot00000000000000import argparse from itertools import product from asap import ASAP from datasets import get_dataset from diff_pool import DiffPool from edge_pool import EdgePool from gcn import GCN, GCNWithJK from gin import GIN, GIN0, GIN0WithJK, GINWithJK from global_attention import GlobalAttentionNet from graclus import Graclus from graph_sage import GraphSAGE, GraphSAGEWithJK from sag_pool import SAGPool from set2set import Set2SetNet from sort_pool import SortPool from top_k import TopK from train_eval import cross_validation_with_val_set parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=100) parser.add_argument('--batch_size', type=int, default=128) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) args = parser.parse_args() layers = [1, 2, 3, 4, 5] hiddens = [16, 32, 64, 128] datasets = ['MUTAG', 'PROTEINS', 'IMDB-BINARY', 'REDDIT-BINARY'] # , 'COLLAB'] nets = [ GCNWithJK, GraphSAGEWithJK, GIN0WithJK, GINWithJK, Graclus, TopK, SAGPool, DiffPool, EdgePool, GCN, GraphSAGE, GIN0, GIN, GlobalAttentionNet, Set2SetNet, SortPool, ASAP, ] def logger(info): fold, epoch = info['fold'] + 1, info['epoch'] val_loss, test_acc = info['val_loss'], info['test_acc'] print(f'{fold:02d}/{epoch:03d}: Val Loss: {val_loss:.4f}, ' f'Test Accuracy: {test_acc:.3f}') results = [] for dataset_name, Net in product(datasets, nets): best_result = (float('inf'), 0, 0) # (loss, acc, std) print(f'--\n{dataset_name} - {Net.__name__}') for num_layers, hidden in product(layers, hiddens): dataset = get_dataset(dataset_name, sparse=Net != DiffPool) model = Net(dataset, num_layers, hidden) loss, acc, std = cross_validation_with_val_set( dataset, model, folds=10, epochs=args.epochs, batch_size=args.batch_size, lr=args.lr, lr_decay_factor=args.lr_decay_factor, lr_decay_step_size=args.lr_decay_step_size, weight_decay=0, logger=None, ) if loss < best_result[0]: best_result = (loss, acc, std) desc = f'{best_result[1]:.3f} Β± {best_result[2]:.3f}' print(f'Best result - {desc}') results += [f'{dataset_name} - {model}: {desc}'] results = '\n'.join(results) print(f'--\n{results}') pytorch/benchmark/kernel/main_performance.py000066400000000000000000000125021444262304300216410ustar00rootroot00000000000000import argparse from itertools import product import torch from datasets import get_dataset from gcn import GCN from gin import GIN from graph_sage import GraphSAGE from train_eval import eval_acc, inference_run, train import torch_geometric from torch_geometric import seed_everything from torch_geometric.loader import DataLoader from torch_geometric.profile import rename_profile_file, timeit, torch_profile seed_everything(0) parser = argparse.ArgumentParser() parser.add_argument( '--datasets', type=str, nargs='+', default=['MUTAG', 'PROTEINS', 'IMDB-BINARY', 'REDDIT-BINARY']) parser.add_argument('--models', type=str, nargs='+', default=['GCN', 'GraphSAGE', 'GIN']) parser.add_argument('--layers', type=int, nargs='+', default=[1, 2, 3]) parser.add_argument('--hiddens', type=int, nargs='+', default=[16, 32]) parser.add_argument('--epochs', type=int, default=100) parser.add_argument('--batch_size', type=int, default=128) parser.add_argument('--lr', type=float, default=0.01) parser.add_argument('--warmup_profile', type=int, default=1, help='Skip the first few runs') parser.add_argument('--goal_accuracy', type=int, default=1, help='The goal test accuracy') parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') if torch.cuda.is_available(): amp = torch.cuda.amp.autocast(enabled=False) else: amp = torch.cpu.amp.autocast(enabled=args.bf16) MODELS = { 'GCN': GCN, 'GraphSAGE': GraphSAGE, 'GIN': GIN, } def prepare_dataloader(dataset_name): dataset = get_dataset(dataset_name, sparse=True) num_train = int(len(dataset) * 0.8) num_val = int(len(dataset) * 0.1) train_dataset = dataset[:num_train] val_dataset = dataset[num_train:num_train + num_val] test_dataset = dataset[num_train + num_val:] train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True) val_loader = DataLoader(val_dataset, batch_size=args.batch_size, shuffle=False) test_loader = DataLoader(test_dataset, batch_size=args.batch_size, shuffle=False) return dataset, train_loader, val_loader, test_loader def run_train(): for dataset_name, model_name in product(args.datasets, args.models): dataset, train_loader, val_loader, test_loader = prepare_dataloader( dataset_name) Model = MODELS[model_name] for num_layers, hidden in product(args.layers, args.hiddens): print('--') print(f'{dataset_name} - {model_name}- {num_layers} - {hidden}') model = Model(dataset, num_layers, hidden).to(device) optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) if args.compile: model = torch_geometric.compile(model) loss_list = [] acc_list = [] for epoch in range(1, args.epochs + 1): if epoch == args.epochs: with timeit(): loss = train(model, optimizer, train_loader) else: loss = train(model, optimizer, train_loader) with timeit(log=False) as t: val_acc = eval_acc(model, val_loader) val_time = t.duration with timeit(log=False) as t: test_acc = eval_acc(model, test_loader) test_time = t.duration if epoch >= args.warmup_profile: loss_list.append(loss) acc_list.append([val_acc, val_time, test_acc, test_time]) if args.profile: with torch_profile(): train(model, optimizer, train_loader) rename_profile_file(model_name, dataset_name, str(num_layers), str(hidden), 'train') @torch.no_grad() def run_inference(): for dataset_name, model_name in product(args.datasets, args.models): dataset, _, _, test_loader = prepare_dataloader(dataset_name) Model = MODELS[model_name] for num_layers, hidden in product(args.layers, args.hiddens): print('--') print(f'{dataset_name} - {model_name}- {num_layers} - {hidden}') model = Model(dataset, num_layers, hidden).to(device) if args.compile: model = torch_geometric.compile(model) with amp: for epoch in range(1, args.epochs + 1): if epoch == args.epochs: with timeit(): inference_run(model, test_loader, args.bf16) else: inference_run(model, test_loader, args.bf16) if args.profile: with torch_profile(): inference_run(model, test_loader, args.bf16) rename_profile_file(model_name, dataset_name, str(num_layers), str(hidden), 'inference') if not args.inference: run_train() else: run_inference() pytorch/benchmark/kernel/sag_pool.py000066400000000000000000000036271444262304300201470ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import ( GraphConv, JumpingKnowledge, SAGPooling, global_mean_pool, ) class SAGPool(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, ratio=0.8): super().__init__() self.conv1 = GraphConv(dataset.num_features, hidden, aggr='mean') self.convs = torch.nn.ModuleList() self.pools = torch.nn.ModuleList() self.convs.extend([ GraphConv(hidden, hidden, aggr='mean') for i in range(num_layers - 1) ]) self.pools.extend( [SAGPooling(hidden, ratio) for i in range((num_layers) // 2)]) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear(num_layers * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() for pool in self.pools: pool.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [global_mean_pool(x, batch)] for i, conv in enumerate(self.convs): x = F.relu(conv(x, edge_index)) xs += [global_mean_pool(x, batch)] if i % 2 == 0 and i < len(self.convs) - 1: pool = self.pools[i // 2] x, edge_index, _, batch, _, _ = pool(x, edge_index, batch=batch) x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/set2set.py000066400000000000000000000024561444262304300177340ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import SAGEConv, Set2Set class Set2SetNet(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = SAGEConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(SAGEConv(hidden, hidden)) self.set2set = Set2Set(hidden, processing_steps=4) self.lin1 = Linear(2 * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.set2set.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) for conv in self.convs: x = F.relu(conv(x, edge_index)) x = self.set2set(x, batch) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/sort_pool.py000066400000000000000000000027241444262304300203610ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Conv1d, Linear from torch_geometric.nn import SAGEConv, SortAggregation class SortPool(torch.nn.Module): def __init__(self, dataset, num_layers, hidden): super().__init__() self.conv1 = SAGEConv(dataset.num_features, hidden) self.convs = torch.nn.ModuleList() for i in range(num_layers - 1): self.convs.append(SAGEConv(hidden, hidden)) self.pool = SortAggregation(k=30) self.conv1d = Conv1d(hidden, 32, 5) self.lin1 = Linear(32 * (30 - 5 + 1), hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() self.conv1d.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) for conv in self.convs: x = F.relu(conv(x, edge_index)) x = self.pool(x, batch) x = x.view(len(x), self.k, -1).permute(0, 2, 1) x = F.relu(self.conv1d(x)) x = x.view(len(x), -1) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/statistics.py000066400000000000000000000010721444262304300205260ustar00rootroot00000000000000from kernel.datasets import get_dataset def print_dataset(dataset): num_nodes = num_edges = 0 for data in dataset: num_nodes += data.num_nodes num_edges += data.num_edges print('Name', dataset) print('Graphs', len(dataset)) print('Nodes', num_nodes / len(dataset)) print('Edges', (num_edges // 2) / len(dataset)) print('Features', dataset.num_features) print('Classes', dataset.num_classes) print() for name in ['MUTAG', 'PROTEINS', 'COLLAB', 'IMDB-BINARY', 'REDDIT-BINARY']: print_dataset(get_dataset(name)) pytorch/benchmark/kernel/top_k.py000066400000000000000000000036261444262304300174570ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch.nn import Linear from torch_geometric.nn import ( GraphConv, JumpingKnowledge, TopKPooling, global_mean_pool, ) class TopK(torch.nn.Module): def __init__(self, dataset, num_layers, hidden, ratio=0.8): super().__init__() self.conv1 = GraphConv(dataset.num_features, hidden, aggr='mean') self.convs = torch.nn.ModuleList() self.pools = torch.nn.ModuleList() self.convs.extend([ GraphConv(hidden, hidden, aggr='mean') for i in range(num_layers - 1) ]) self.pools.extend( [TopKPooling(hidden, ratio) for i in range((num_layers) // 2)]) self.jump = JumpingKnowledge(mode='cat') self.lin1 = Linear(num_layers * hidden, hidden) self.lin2 = Linear(hidden, dataset.num_classes) def reset_parameters(self): self.conv1.reset_parameters() for conv in self.convs: conv.reset_parameters() for pool in self.pools: pool.reset_parameters() self.lin1.reset_parameters() self.lin2.reset_parameters() def forward(self, data): x, edge_index, batch = data.x, data.edge_index, data.batch x = F.relu(self.conv1(x, edge_index)) xs = [global_mean_pool(x, batch)] for i, conv in enumerate(self.convs): x = F.relu(conv(x, edge_index)) xs += [global_mean_pool(x, batch)] if i % 2 == 0 and i < len(self.convs) - 1: pool = self.pools[i // 2] x, edge_index, _, batch, _, _ = pool(x, edge_index, batch=batch) x = self.jump(xs) x = F.relu(self.lin1(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin2(x) return F.log_softmax(x, dim=-1) def __repr__(self): return self.__class__.__name__ pytorch/benchmark/kernel/train_eval.py000066400000000000000000000115351444262304300204650ustar00rootroot00000000000000import time import torch import torch.nn.functional as F from sklearn.model_selection import StratifiedKFold from torch import tensor from torch.optim import Adam from torch_geometric.loader import DataLoader from torch_geometric.loader import DenseDataLoader as DenseLoader device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') def cross_validation_with_val_set(dataset, model, folds, epochs, batch_size, lr, lr_decay_factor, lr_decay_step_size, weight_decay, logger=None): val_losses, accs, durations = [], [], [] for fold, (train_idx, test_idx, val_idx) in enumerate(zip(*k_fold(dataset, folds))): train_dataset = dataset[train_idx] test_dataset = dataset[test_idx] val_dataset = dataset[val_idx] if 'adj' in train_dataset[0]: train_loader = DenseLoader(train_dataset, batch_size, shuffle=True) val_loader = DenseLoader(val_dataset, batch_size, shuffle=False) test_loader = DenseLoader(test_dataset, batch_size, shuffle=False) else: train_loader = DataLoader(train_dataset, batch_size, shuffle=True) val_loader = DataLoader(val_dataset, batch_size, shuffle=False) test_loader = DataLoader(test_dataset, batch_size, shuffle=False) model.to(device).reset_parameters() optimizer = Adam(model.parameters(), lr=lr, weight_decay=weight_decay) if torch.cuda.is_available(): torch.cuda.synchronize() t_start = time.perf_counter() for epoch in range(1, epochs + 1): train_loss = train(model, optimizer, train_loader) val_losses.append(eval_loss(model, val_loader)) accs.append(eval_acc(model, test_loader)) eval_info = { 'fold': fold, 'epoch': epoch, 'train_loss': train_loss, 'val_loss': val_losses[-1], 'test_acc': accs[-1], } if logger is not None: logger(eval_info) if epoch % lr_decay_step_size == 0: for param_group in optimizer.param_groups: param_group['lr'] = lr_decay_factor * param_group['lr'] if torch.cuda.is_available(): torch.cuda.synchronize() t_end = time.perf_counter() durations.append(t_end - t_start) loss, acc, duration = tensor(val_losses), tensor(accs), tensor(durations) loss, acc = loss.view(folds, epochs), acc.view(folds, epochs) loss, argmin = loss.min(dim=1) acc = acc[torch.arange(folds, dtype=torch.long), argmin] loss_mean = loss.mean().item() acc_mean = acc.mean().item() acc_std = acc.std().item() duration_mean = duration.mean().item() print(f'Val Loss: {loss_mean:.4f}, Test Accuracy: {acc_mean:.3f} ' f'Β± {acc_std:.3f}, Duration: {duration_mean:.3f}') return loss_mean, acc_mean, acc_std def k_fold(dataset, folds): skf = StratifiedKFold(folds, shuffle=True, random_state=12345) test_indices, train_indices = [], [] for _, idx in skf.split(torch.zeros(len(dataset)), dataset.data.y): test_indices.append(torch.from_numpy(idx).to(torch.long)) val_indices = [test_indices[i - 1] for i in range(folds)] for i in range(folds): train_mask = torch.ones(len(dataset), dtype=torch.bool) train_mask[test_indices[i]] = 0 train_mask[val_indices[i]] = 0 train_indices.append(train_mask.nonzero(as_tuple=False).view(-1)) return train_indices, test_indices, val_indices def num_graphs(data): if hasattr(data, 'num_graphs'): return data.num_graphs else: return data.x.size(0) def train(model, optimizer, loader): model.train() total_loss = 0 for data in loader: optimizer.zero_grad() data = data.to(device) out = model(data) loss = F.nll_loss(out, data.y.view(-1)) loss.backward() total_loss += loss.item() * num_graphs(data) optimizer.step() return total_loss / len(loader.dataset) def eval_acc(model, loader): model.eval() correct = 0 for data in loader: data = data.to(device) with torch.no_grad(): pred = model(data).max(1)[1] correct += pred.eq(data.y.view(-1)).sum().item() return correct / len(loader.dataset) def eval_loss(model, loader): model.eval() loss = 0 for data in loader: data = data.to(device) with torch.no_grad(): out = model(data) loss += F.nll_loss(out, data.y.view(-1), reduction='sum').item() return loss / len(loader.dataset) @torch.no_grad() def inference_run(model, loader, bf16): model.eval() for data in loader: data = data.to(device) if bf16: data.x = data.x.to(torch.bfloat16) model(data) pytorch/benchmark/loader/000077500000000000000000000000001444262304300157505ustar00rootroot00000000000000pytorch/benchmark/loader/neighbor_loader.py000066400000000000000000000142711444262304300214520ustar00rootroot00000000000000import argparse import ast import os.path as osp from contextlib import nullcontext from timeit import default_timer import tqdm from ogb.nodeproppred import PygNodePropPredDataset import torch_geometric.transforms as T from torch_geometric.datasets import OGB_MAG from torch_geometric.loader import NeighborLoader from torch_geometric.profile import torch_profile def run(args: argparse.ArgumentParser): for dataset_name in args.datasets: print(f"Dataset: {dataset_name}") root = osp.join(args.root, dataset_name) transform = T.ToSparseTensor( remove_edge_index=False) if args.use_sparse_tensor else None if dataset_name == 'mag': transform = (T.ToUndirected(merge=True) if transform is None else T.Compose([T.ToUndirected(merge=True), transform])) dataset = OGB_MAG(root=root, transform=transform) train_idx = ('paper', dataset[0]['paper'].train_mask) eval_idx = ('paper', None) neighbor_sizes = (args.hetero_neighbor_sizes if args.hetero_neighbor_sizes else None) else: dataset = PygNodePropPredDataset(f'ogbn-{dataset_name}', root) split_idx = dataset.get_idx_split() train_idx = split_idx['train'] eval_idx = None neighbor_sizes = (args.homo_neighbor_sizes if args.homo_neighbor_sizes else None) data = dataset[0].to(args.device) average_times = [] profile = torch_profile() if args.profile else nullcontext() # run dataloader iteration if neighbor_sizes is not None: for num_neighbors in neighbor_sizes: print(f'Training sampling with {num_neighbors} neighbors') for batch_size in args.batch_sizes: train_loader = NeighborLoader( data, num_neighbors=num_neighbors, input_nodes=train_idx, batch_size=batch_size, shuffle=True, num_workers=args.num_workers, subgraph_type=args.subgraph_type, ) cpu_affinity = train_loader.enable_cpu_affinity( args.loader_cores ) if args.cpu_affinity else nullcontext() runtimes = [] num_iterations = 0 with profile, cpu_affinity: for runit in range(args.runs): start = default_timer() for batch in tqdm.tqdm(train_loader): num_iterations += 1 stop = default_timer() runtimes.append(round(stop - start, 3)) average_time = round(sum(runtimes) / args.runs, 3) print(f'batch size={batch_size}, ' f'iterations={num_iterations}, ' f'runtimes={runtimes}, ' f'average runtime={average_time}') average_times.append(average_time) eval_batch_sizes = (args.eval_batch_sizes if args.eval_batch_sizes else None) if eval_batch_sizes is not None: print('Evaluation sampling with all neighbors') for batch_size in eval_batch_sizes: subgraph_loader = NeighborLoader( data, num_neighbors=[-1], input_nodes=eval_idx, batch_size=batch_size, shuffle=False, num_workers=args.num_workers, ) cpu_affinity = subgraph_loader.enable_cpu_affinity( args.loader_cores) if args.cpu_affinity else nullcontext() runtimes = [] num_iterations = 0 with profile, cpu_affinity: for runit in range(args.runs): start = default_timer() for batch in tqdm.tqdm(subgraph_loader): num_iterations += 1 stop = default_timer() runtimes.append(round(stop - start, 3)) average_time = round(sum(runtimes) / args.runs, 3) print(f'batch size={batch_size}, ' f'iterations={num_iterations}, ' f'runtimes={runtimes}, ' f'average runtime={average_time}') average_times.append(average_time) print(f"Total time averages: {average_times}") if __name__ == '__main__': parser = argparse.ArgumentParser('NeighborLoader Sampling Benchmarking') add = parser.add_argument add('--device', default='cpu') add('--datasets', nargs="+", default=['arxiv', 'products', 'mag']) add('--root', default='../../data') add('--batch-sizes', default=[8192, 4096, 2048, 1024, 512], type=ast.literal_eval) add('--eval-batch-sizes', default=[16384, 8192, 4096, 2048, 1024, 512], type=ast.literal_eval) add('--homo-neighbor_sizes', default=[[10, 5], [15, 10, 5], [20, 15, 10]], type=ast.literal_eval) add('--hetero-neighbor_sizes', default=[[5], [10], [10, 5]], type=ast.literal_eval) add('--use-sparse-tensor', action='store_true', help='use torch_sparse.SparseTensor as graph storage format') add('--num-workers', type=int, default=0, help="Number of DataLoader workers to use.") add('--runs', type=int, default=3, help="Number of iterations for each test setting.") add('--profile', default=False, action='store_true', help="Run torch.profiler.") add('--cpu-affinity', default=False, action='store_true', help="Use DataLoader affinitzation.") add('--loader-cores', nargs='+', default=[], type=int, help="List of CPU core IDs to use for DataLoader workers.") add('--subgraph-type', type=str, default='directional', help="The type of the returned subgraph (directional, bidirectional)") run(parser.parse_args()) pytorch/benchmark/points/000077500000000000000000000000001444262304300160165ustar00rootroot00000000000000pytorch/benchmark/points/README.md000066400000000000000000000013321444262304300172740ustar00rootroot00000000000000# Point Cloud classification Evaluation scripts for various methods on the ModelNet10 dataset: * **[MPNN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/mpnn.py)**: `python mpnn.py` * **[PointNet++](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/point_net.py)**: `python point_net.py` * **[EdgeCNN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/edge_cnn.py)**: `python edge_cnn.py` * **[SplineCNN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/spline_cnn.py)**: `python spline_cnn.py` * **[PointCNN](https://github.com/pyg-team/pytorch_geometric/blob/master/benchmark/points/point_cnn.py)**: `python point_cnn.py` pytorch/benchmark/points/__init__.py000066400000000000000000000001531444262304300201260ustar00rootroot00000000000000from .datasets import get_dataset from .train_eval import run __all__ = [ 'get_dataset', 'run', ] pytorch/benchmark/points/datasets.py000066400000000000000000000011741444262304300202030ustar00rootroot00000000000000import os.path as osp import torch_geometric.transforms as T from torch_geometric.datasets import ModelNet def get_dataset(num_points): name = 'ModelNet10' path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', name) pre_transform = T.NormalizeScale() transform = T.SamplePoints(num_points) train_dataset = ModelNet(path, name='10', train=True, transform=transform, pre_transform=pre_transform) test_dataset = ModelNet(path, name='10', train=False, transform=transform, pre_transform=pre_transform) return train_dataset, test_dataset pytorch/benchmark/points/edge_cnn.py000066400000000000000000000043601444262304300201350ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from points.datasets import get_dataset from points.train_eval import run from torch.nn import Linear as Lin from torch.nn import ReLU from torch.nn import Sequential as Seq from torch_geometric.nn import DynamicEdgeConv, global_max_pool from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--batch_size', type=int, default=8) parser.add_argument('--lr', type=float, default=0.001) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) parser.add_argument('--weight_decay', type=float, default=0) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, num_classes): super().__init__() nn = Seq(Lin(6, 64), ReLU(), Lin(64, 64), ReLU(), Lin(64, 64), ReLU()) self.conv1 = DynamicEdgeConv(nn, k=20, aggr='max') nn = Seq(Lin(128, 128), ReLU(), Lin(128, 128), ReLU(), Lin(128, 256), ReLU()) self.conv2 = DynamicEdgeConv(nn, k=20, aggr='max') self.lin0 = Lin(256, 512) self.lin1 = Lin(512, 256) self.lin2 = Lin(256, 256) self.lin3 = Lin(256, num_classes) def forward(self, pos, batch): x = self.conv1(pos, batch) x = self.conv2(x, batch) x = F.relu(self.lin0(x)) x = global_max_pool(x, batch) x = F.relu(self.lin1(x)) x = F.relu(self.lin2(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin3(x) return F.log_softmax(x, dim=-1) train_dataset, test_dataset = get_dataset(num_points=1024) model = Net(train_dataset.num_classes) run(train_dataset, test_dataset, model, args.epochs, args.batch_size, args.lr, args.lr_decay_factor, args.lr_decay_step_size, args.weight_decay, args.inference, args.profile, args.bf16, args.compile) if args.profile: rename_profile_file('points', DynamicEdgeConv.__name__) pytorch/benchmark/points/mpnn.py000066400000000000000000000057271444262304300173530ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from points.datasets import get_dataset from points.train_eval import run from torch.nn import Linear as Lin from torch.nn import ReLU from torch.nn import Sequential as Seq from torch_geometric.nn import NNConv, fps, global_mean_pool, radius_graph from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--batch_size', type=int, default=8) parser.add_argument('--lr', type=float, default=0.001) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) parser.add_argument('--weight_decay', type=float, default=0) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, num_classes): super().__init__() nn = Seq(Lin(3, 25), ReLU(), Lin(25, 1 * 64)) self.conv1 = NNConv(1, 64, nn, aggr='mean') nn = Seq(Lin(3, 25), ReLU(), Lin(25, 64 * 64)) self.conv2 = NNConv(64, 64, nn, aggr='mean') nn = Seq(Lin(3, 25), ReLU(), Lin(25, 64 * 128)) self.conv3 = NNConv(64, 128, nn, aggr='mean') self.lin1 = torch.nn.Linear(128, 256) self.lin2 = torch.nn.Linear(256, 256) self.lin3 = torch.nn.Linear(256, num_classes) def forward(self, pos, batch): x = pos.new_ones((pos.size(0), 1)) radius = 0.2 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = pos[edge_index[1]] - pos[edge_index[0]] x = F.relu(self.conv1(x, edge_index, pseudo)) idx = fps(pos, batch, ratio=0.5) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 0.4 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = pos[edge_index[1]] - pos[edge_index[0]] x = F.relu(self.conv2(x, edge_index, pseudo)) idx = fps(pos, batch, ratio=0.25) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 1 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = pos[edge_index[1]] - pos[edge_index[0]] x = F.relu(self.conv3(x, edge_index, pseudo)) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.relu(self.lin2(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin3(x) return F.log_softmax(x, dim=-1) train_dataset, test_dataset = get_dataset(num_points=1024) model = Net(train_dataset.num_classes) run(train_dataset, test_dataset, model, args.epochs, args.batch_size, args.lr, args.lr_decay_factor, args.lr_decay_step_size, args.weight_decay, args.inference, args.profile, args.bf16, args.compile) if args.profile: rename_profile_file('points', NNConv.__name__) pytorch/benchmark/points/point_cnn.py000066400000000000000000000050341444262304300203610ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from points.datasets import get_dataset from points.train_eval import run from torch.nn import Linear as Lin from torch_geometric.nn import XConv, fps, global_mean_pool from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--batch_size', type=int, default=32) parser.add_argument('--lr', type=float, default=0.001) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) parser.add_argument('--weight_decay', type=float, default=0) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, num_classes): super().__init__() self.conv1 = XConv(0, 48, dim=3, kernel_size=8, hidden_channels=32) self.conv2 = XConv(48, 96, dim=3, kernel_size=12, hidden_channels=64, dilation=2) self.conv3 = XConv(96, 192, dim=3, kernel_size=16, hidden_channels=128, dilation=2) self.conv4 = XConv(192, 384, dim=3, kernel_size=16, hidden_channels=256, dilation=2) self.lin1 = Lin(384, 256) self.lin2 = Lin(256, 128) self.lin3 = Lin(128, num_classes) def forward(self, pos, batch): x = F.relu(self.conv1(None, pos, batch)) idx = fps(pos, batch, ratio=0.375) x, pos, batch = x[idx], pos[idx], batch[idx] x = F.relu(self.conv2(x, pos, batch)) idx = fps(pos, batch, ratio=0.334) x, pos, batch = x[idx], pos[idx], batch[idx] x = F.relu(self.conv3(x, pos, batch)) x = F.relu(self.conv4(x, pos, batch)) x = global_mean_pool(x, batch) x = F.relu(self.lin1(x)) x = F.relu(self.lin2(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin3(x) return F.log_softmax(x, dim=-1) train_dataset, test_dataset = get_dataset(num_points=1024) model = Net(train_dataset.num_classes) run(train_dataset, test_dataset, model, args.epochs, args.batch_size, args.lr, args.lr_decay_factor, args.lr_decay_step_size, args.weight_decay, args.inference, args.profile, args.bf16, args.compile) if args.profile: rename_profile_file('points', XConv.__name__) pytorch/benchmark/points/point_net.py000066400000000000000000000053111444262304300203670ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from points.datasets import get_dataset from points.train_eval import run from torch.nn import Linear as Lin from torch.nn import ReLU from torch.nn import Sequential as Seq from torch_geometric.nn import PointNetConv, fps, global_max_pool, radius_graph from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--batch_size', type=int, default=8) parser.add_argument('--lr', type=float, default=0.001) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) parser.add_argument('--weight_decay', type=float, default=0) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, num_classes): super().__init__() nn = Seq(Lin(3, 64), ReLU(), Lin(64, 64)) self.conv1 = PointNetConv(local_nn=nn) nn = Seq(Lin(67, 128), ReLU(), Lin(128, 128)) self.conv2 = PointNetConv(local_nn=nn) nn = Seq(Lin(131, 256), ReLU(), Lin(256, 256)) self.conv3 = PointNetConv(local_nn=nn) self.lin1 = Lin(256, 256) self.lin2 = Lin(256, 256) self.lin3 = Lin(256, num_classes) def forward(self, pos, batch): radius = 0.2 edge_index = radius_graph(pos, r=radius, batch=batch) x = F.relu(self.conv1(None, pos, edge_index)) idx = fps(pos, batch, ratio=0.5) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 0.4 edge_index = radius_graph(pos, r=radius, batch=batch) x = F.relu(self.conv2(x, pos, edge_index)) idx = fps(pos, batch, ratio=0.25) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 1 edge_index = radius_graph(pos, r=radius, batch=batch) x = F.relu(self.conv3(x, pos, edge_index)) x = global_max_pool(x, batch) x = F.relu(self.lin1(x)) x = F.relu(self.lin2(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin3(x) return F.log_softmax(x, dim=-1) train_dataset, test_dataset = get_dataset(num_points=1024) model = Net(train_dataset.num_classes) run(train_dataset, test_dataset, model, args.epochs, args.batch_size, args.lr, args.lr_decay_factor, args.lr_decay_step_size, args.weight_decay, args.inference, args.profile, args.bf16, args.compile) if args.profile: rename_profile_file('points', PointNetConv.__name__) pytorch/benchmark/points/spline_cnn.py000066400000000000000000000056631444262304300205320ustar00rootroot00000000000000import argparse import torch import torch.nn.functional as F from points.datasets import get_dataset from points.train_eval import run from torch.nn import Linear as Lin from torch_geometric.nn import SplineConv, fps, global_mean_pool, radius_graph from torch_geometric.profile import rename_profile_file parser = argparse.ArgumentParser() parser.add_argument('--epochs', type=int, default=200) parser.add_argument('--batch_size', type=int, default=8) parser.add_argument('--lr', type=float, default=0.001) parser.add_argument('--lr_decay_factor', type=float, default=0.5) parser.add_argument('--lr_decay_step_size', type=int, default=50) parser.add_argument('--weight_decay', type=float, default=0) parser.add_argument('--inference', action='store_true') parser.add_argument('--profile', action='store_true') parser.add_argument('--bf16', action='store_true') parser.add_argument('--compile', action='store_true') args = parser.parse_args() class Net(torch.nn.Module): def __init__(self, num_classes): super().__init__() self.conv1 = SplineConv(1, 64, dim=3, kernel_size=5) self.conv2 = SplineConv(64, 64, dim=3, kernel_size=5) self.conv3 = SplineConv(64, 128, dim=3, kernel_size=5) self.lin1 = Lin(128, 256) self.lin2 = Lin(256, 256) self.lin3 = Lin(256, num_classes) def forward(self, pos, batch): x = pos.new_ones((pos.size(0), 1)) radius = 0.2 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = (pos[edge_index[1]] - pos[edge_index[0]]) / (2 * radius) + 0.5 pseudo = pseudo.clamp(min=0, max=1) x = F.elu(self.conv1(x, edge_index, pseudo)) idx = fps(pos, batch, ratio=0.5) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 0.4 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = (pos[edge_index[1]] - pos[edge_index[0]]) / (2 * radius) + 0.5 pseudo = pseudo.clamp(min=0, max=1) x = F.elu(self.conv2(x, edge_index, pseudo)) idx = fps(pos, batch, ratio=0.25) x, pos, batch = x[idx], pos[idx], batch[idx] radius = 1 edge_index = radius_graph(pos, r=radius, batch=batch) pseudo = (pos[edge_index[1]] - pos[edge_index[0]]) / (2 * radius) + 0.5 pseudo = pseudo.clamp(min=0, max=1) x = F.elu(self.conv3(x, edge_index, pseudo)) x = global_mean_pool(x, batch) x = F.elu(self.lin1(x)) x = F.elu(self.lin2(x)) x = F.dropout(x, p=0.5, training=self.training) x = self.lin3(x) return F.log_softmax(x, dim=-1) train_dataset, test_dataset = get_dataset(num_points=1024) model = Net(train_dataset.num_classes) run(train_dataset, test_dataset, model, args.epochs, args.batch_size, args.lr, args.lr_decay_factor, args.lr_decay_step_size, args.weight_decay, args.inference, args.profile, args.bf16, args.compile) if args.profile: rename_profile_file('points', SplineConv.__name__) pytorch/benchmark/points/statistics.py000066400000000000000000000013571444262304300205700ustar00rootroot00000000000000from points.datasets import get_dataset from torch_geometric.transforms import RadiusGraph def print_dataset(train_dataset, test_dataset): num_nodes = num_edges = 0 for data in train_dataset: data = RadiusGraph(0.2)(data) num_nodes += data.num_nodes num_edges += data.num_edges for data in test_dataset: data = RadiusGraph(0.2)(data) num_nodes += data.num_nodes num_edges += data.num_edges num_graphs = len(train_dataset) + len(test_dataset) print('Graphs', num_graphs) print('Nodes', num_nodes / num_graphs) print('Edges', (num_edges // 2) / num_graphs) print('Label rate', len(train_dataset) / num_graphs) print() print_dataset(*get_dataset(num_points=1024)) pytorch/benchmark/points/train_eval.py000066400000000000000000000071611444262304300205210ustar00rootroot00000000000000import time import torch import torch.nn.functional as F from torch.optim import Adam import torch_geometric from torch_geometric.loader import DataLoader from torch_geometric.profile import timeit, torch_profile device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') def run_train(train_dataset, test_dataset, model, epochs, batch_size, use_compile, lr, lr_decay_factor, lr_decay_step_size, weight_decay): model = model.to(device) if use_compile: model = torch_geometric.compile(model) optimizer = Adam(model.parameters(), lr=lr, weight_decay=weight_decay) train_loader = DataLoader(train_dataset, batch_size, shuffle=True) test_loader = DataLoader(test_dataset, batch_size, shuffle=False) for epoch in range(1, epochs + 1): if torch.cuda.is_available(): torch.cuda.synchronize() t_start = time.perf_counter() train(model, optimizer, train_loader, device) test_acc = test(model, test_loader, device) if torch.cuda.is_available(): torch.cuda.synchronize() t_end = time.perf_counter() print(f'Epoch: {epoch:03d}, Test: {test_acc:.4f}, ' f'Duration: {t_end - t_start:.2f}') if epoch % lr_decay_step_size == 0: for param_group in optimizer.param_groups: param_group['lr'] = lr_decay_factor * param_group['lr'] @torch.no_grad() def run_inference(test_dataset, model, epochs, batch_size, profiling, bf16, use_compile): model = model.to(device) if use_compile: model = torch_geometric.compile(model) test_loader = DataLoader(test_dataset, batch_size, shuffle=False) if torch.cuda.is_available(): amp = torch.cuda.amp.autocast(enabled=False) else: amp = torch.cpu.amp.autocast(enabled=bf16) with amp: for epoch in range(1, epochs + 1): print("Epoch: ", epoch) if epoch == epochs: with timeit(): inference(model, test_loader, device, bf16) else: inference(model, test_loader, device, bf16) if profiling: with torch_profile(): inference(model, test_loader, device, bf16) def run(train_dataset, test_dataset, model, epochs, batch_size, lr, lr_decay_factor, lr_decay_step_size, weight_decay, inference, profiling, bf16, use_compile): if not inference: run_train(train_dataset, test_dataset, model, epochs, batch_size, use_compile, lr, lr_decay_factor, lr_decay_step_size, weight_decay) else: run_inference(test_dataset, model, epochs, batch_size, profiling, bf16, use_compile) def train(model, optimizer, train_loader, device): model.train() for data in train_loader: optimizer.zero_grad() data = data.to(device) out = model(data.pos, data.batch) loss = F.nll_loss(out, data.y) loss.backward() optimizer.step() @torch.no_grad() def test(model, test_loader, device): model.eval() correct = 0 for data in test_loader: data = data.to(device) pred = model(data.pos, data.batch).max(1)[1] correct += pred.eq(data.y).sum().item() test_acc = correct / len(test_loader.dataset) return test_acc @torch.no_grad() def inference(model, test_loader, device, bf16): model.eval() for data in test_loader: data = data.to(device) if bf16: data.pos = data.pos.to(torch.bfloat16) model = model.to(torch.bfloat16) model(data.pos, data.batch) pytorch/benchmark/runtime/000077500000000000000000000000001444262304300161655ustar00rootroot00000000000000pytorch/benchmark/runtime/README.md000066400000000000000000000002241444262304300174420ustar00rootroot00000000000000# Runtimes Run the test suite for PyG via ``` python main.py ``` Install `dgl` and run the test suite for DGL via ``` cd dgl python main.py ``` pytorch/benchmark/runtime/__init__.py000066400000000000000000000001051444262304300202720ustar00rootroot00000000000000from .train import train_runtime __all__ = [ 'train_runtime', ] pytorch/benchmark/runtime/dgl/000077500000000000000000000000001444262304300167335ustar00rootroot00000000000000pytorch/benchmark/runtime/dgl/gat.py000066400000000000000000000111741444262304300200640ustar00rootroot00000000000000import dgl.function as fn import torch import torch.nn.functional as F from dgl.nn.pytorch import EdgeSoftmax from torch.nn import Parameter from torch_geometric.nn.inits import glorot, zeros class GATConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels, heads=1, negative_slope=0.2, dropout=0): super().__init__() self.g = g self.in_channels = in_channels self.out_channels = out_channels self.heads = heads self.negative_slope = negative_slope self.dropout = dropout self.weight = Parameter(torch.Tensor(in_channels, heads * out_channels)) self.att = Parameter(torch.Tensor(1, heads, 2 * out_channels)) self.bias = Parameter(torch.Tensor(heads * out_channels)) self.reset_parameters() def reset_parameters(self): glorot(self.weight) glorot(self.att) zeros(self.bias) def gat_msg(self, edge): alpha = torch.cat([edge.src['x'], edge.dst['x']], dim=-1) alpha = (alpha * self.att).sum(dim=-1) alpha = F.leaky_relu(alpha, self.negative_slope) return {'m': edge.src['x'], 'a': alpha} def gat_reduce(self, node): alpha = torch.softmax(node.mailbox['a'], dim=1) alpha = F.dropout(alpha, p=self.dropout, training=self.training) x = (node.mailbox['m'] * alpha.unsqueeze(-1)).sum(dim=1) return {'x': x} def forward(self, x): x = torch.mm(x, self.weight).view(-1, self.heads, self.out_channels) self.g.ndata['x'] = x self.g.update_all(self.gat_msg, self.gat_reduce) x = self.g.ndata.pop('x') x = x.view(-1, self.heads * self.out_channels) x = x + self.bias return x class GAT(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.g = g self.conv1 = GATConv(g, in_channels, 8, 8, 0.6, 0.2) self.conv2 = GATConv(g, 64, out_channels, 1, 0.6, 0.2) def forward(self, x): x = F.dropout(x, p=0.6, training=self.training) x = F.elu(self.conv1(x)) x = F.dropout(x, p=0.6, training=self.training) x = self.conv2(x) return F.log_softmax(x, dim=1) class GATSPMVConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels, heads=1, negative_slope=0.2, dropout=0): super().__init__() self.g = g self.out_channels = out_channels self.heads = heads self.negative_slope = negative_slope self.dropout = dropout self.weight = Parameter(torch.Tensor(in_channels, heads * out_channels)) self.att_l = Parameter(torch.Tensor(heads, out_channels, 1)) self.att_r = Parameter(torch.Tensor(heads, out_channels, 1)) self.bias = Parameter(torch.Tensor(heads * out_channels)) self.softmax = EdgeSoftmax() self.reset_parameters() def reset_parameters(self): glorot(self.weight) glorot(self.att_l) glorot(self.att_r) zeros(self.bias) def forward(self, x): x = torch.matmul(x, self.weight) x = x.reshape((x.size(0), self.heads, -1)) # NxHxD' head_x = x.transpose(0, 1) # HxNxD' a1 = torch.bmm(head_x, self.att_l).transpose(0, 1) # NxHx1 a2 = torch.bmm(head_x, self.att_r).transpose(0, 1) # NxHx1 self.g.ndata.update({'x': x, 'a1': a1, 'a2': a2}) self.g.apply_edges(self.edge_attention) self.edge_softmax() self.g.update_all(fn.src_mul_edge('x', 'a', 'x'), fn.sum('x', 'x')) x = self.g.ndata['x'] / self.g.ndata['z'] # NxHxD' return x.view(-1, self.heads * self.out_channels) def edge_attention(self, edge): a = F.leaky_relu(edge.src['a1'] + edge.dst['a2'], self.negative_slope) return {'a': a} def edge_softmax(self): alpha, normalizer = self.softmax(self.g.edata['a'], self.g) self.g.ndata['z'] = normalizer if self.training and self.dropout > 0: alpha = F.dropout(alpha, p=self.dropout, training=True) self.g.edata['a'] = alpha class GATSPMV(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.g = g self.conv1 = GATSPMVConv(g, in_channels, 8, 8, 0.6, 0.2) self.conv2 = GATSPMVConv(g, 64, out_channels, 1, 0.6, 0.2) def forward(self, x): x = F.dropout(x, p=0.6, training=self.training) x = F.elu(self.conv1(x)) x = F.dropout(x, p=0.6, training=self.training) x = self.conv2(x) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/dgl/gcn.py000066400000000000000000000046131444262304300200600ustar00rootroot00000000000000import dgl.function as fn import torch import torch.nn.functional as F from torch.nn import Parameter from torch_geometric.nn.inits import glorot, zeros class GCNConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.g = g self.weight = Parameter(torch.Tensor(in_channels, out_channels)) self.bias = Parameter(torch.Tensor(out_channels)) self.reset_parameters() def reset_parameters(self): glorot(self.weight) zeros(self.bias) def gcn_msg(self, edge): return {'m': edge.src['x'] * edge.src['norm']} def gcn_reduce(self, node): return {'x': node.mailbox['m'].sum(dim=1) * node.data['norm']} def forward(self, x): self.g.ndata['x'] = torch.matmul(x, self.weight) self.g.update_all(self.gcn_msg, self.gcn_reduce) x = self.g.ndata.pop('x') x = x + self.bias return x class GCN(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.conv1 = GCNConv(g, in_channels, 16) self.conv2 = GCNConv(g, 16, out_channels) def forward(self, x): x = F.relu(self.conv1(x)) x = F.dropout(x, training=self.training) x = self.conv2(x) return F.log_softmax(x, dim=1) class GCNSPMVConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.g = g self.weight = Parameter(torch.Tensor(in_channels, out_channels)) self.bias = Parameter(torch.Tensor(out_channels)) self.reset_parameters() def reset_parameters(self): glorot(self.weight) zeros(self.bias) def forward(self, x): x = torch.matmul(x, self.weight) self.g.ndata['x'] = x * self.g.ndata['norm'] self.g.update_all(fn.copy_src(src='x', out='m'), fn.sum(msg='m', out='x')) x = self.g.ndata.pop('x') * self.g.ndata['norm'] x = x + self.bias return x class GCNSPMV(torch.nn.Module): def __init__(self, g, in_channels, out_channels): super().__init__() self.conv1 = GCNSPMVConv(g, in_channels, 16) self.conv2 = GCNSPMVConv(g, 16, out_channels) def forward(self, x): x = F.relu(self.conv1(x)) x = F.dropout(x, training=self.training) x = self.conv2(x) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/dgl/hidden.py000066400000000000000000000005061444262304300205410ustar00rootroot00000000000000import os import sys import warnings warnings.filterwarnings('ignore') class HiddenPrint: def __enter__(self): self._original_stdout = sys.stdout sys.stdout = open(os.devnull, 'w') def __exit__(self, exc_type, exc_val, exc_tb): sys.stdout.close() sys.stdout = self._original_stdout pytorch/benchmark/runtime/dgl/main.py000066400000000000000000000041101444262304300202250ustar00rootroot00000000000000from itertools import product import dgl import torch from dgl import DGLGraph from dgl.contrib.data import load_data from dgl.data import citation_graph from runtime.dgl.gat import GAT, GATSPMV from runtime.dgl.gcn import GCN, GCNSPMV from runtime.dgl.hidden import HiddenPrint from runtime.dgl.rgcn import RGCN, RGCNSPMV from runtime.dgl.train import train_runtime device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') with HiddenPrint(): Cora = citation_graph.load_cora() CiteSeer = citation_graph.load_citeseer() PubMed = citation_graph.load_pubmed() MUTAG = load_data('mutag') # fair comparison # One training run before we start tracking duration to warm up GPU. g = DGLGraph(Cora.graph) g.set_n_initializer(dgl.init.zero_initializer) g.add_edges(g.nodes(), g.nodes()) norm = torch.pow(g.in_degrees().float(), -0.5) norm[torch.isinf(norm)] = 0 g.ndata['norm'] = norm.unsqueeze(1).to(device) model = GCNSPMV(g, Cora.features.shape[1], Cora.num_labels).to(device) train_runtime(model, Cora, epochs=200, device=device) for d, Net in product([Cora, CiteSeer, PubMed], [GCN, GCNSPMV, GAT, GATSPMV]): g = DGLGraph(d.graph) g.set_n_initializer(dgl.init.zero_initializer) g.add_edges(g.nodes(), g.nodes()) norm = torch.pow(g.in_degrees().float(), -0.5) norm[torch.isinf(norm)] = 0 g.ndata['norm'] = norm.unsqueeze(1).to(device) model = Net(g, d.features.shape[1], d.num_labels).to(device) t = train_runtime(model, d, epochs=200, device=device) print(f'{d.name} - {Net.__name__}: {t:.2f}s') for d, Net in product([MUTAG], [RGCN, RGCNSPMV]): g = DGLGraph() g.add_nodes(d.num_nodes) g.add_edges(d.edge_src, d.edge_dst) edge_type = torch.from_numpy(d.edge_type).to(device) edge_norm = torch.from_numpy(d.edge_norm).to(device) g.edata.update({'type': edge_type, 'norm': edge_norm}) g.ndata['id'] = torch.arange(d.num_nodes, dtype=torch.long, device=device) model = Net(g, d.num_nodes, d.num_classes, d.num_rels) t = train_runtime(model, d, epochs=200, device=device) print(f'{d.name} - {Net.__name__}: {t:.2f}s') pytorch/benchmark/runtime/dgl/rgcn.py000066400000000000000000000116021444262304300202360ustar00rootroot00000000000000import dgl.function as fn import torch import torch.nn.functional as F from torch.nn import Parameter as Param from torch_geometric.nn.inits import uniform class RGCNConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels, num_relations, num_bases): super().__init__() self.g = g self.in_channels = in_channels self.out_channels = out_channels self.num_relations = num_relations self.num_bases = num_bases self.basis = Param(torch.Tensor(num_bases, in_channels, out_channels)) self.att = Param(torch.Tensor(num_relations, num_bases)) self.root = Param(torch.Tensor(in_channels, out_channels)) self.bias = Param(torch.Tensor(out_channels)) self.reset_parameters() def reset_parameters(self): size = self.num_bases * self.in_channels uniform(size, self.basis) uniform(size, self.att) uniform(size, self.root) uniform(size, self.bias) def rgcn_reduce(self, node): return {'x': node.mailbox['m'].sum(dim=1)} def forward(self, x): self.w = torch.matmul(self.att, self.basis.view(self.num_bases, -1)) self.w = self.w.view(self.num_relations, self.in_channels, self.out_channels) if x is None: def msg_func(edge): w = self.w.view(-1, self.out_channels) index = edge.data['type'] * self.in_channels + edge.src['id'] m = w.index_select(0, index) * edge.data['norm'].unsqueeze(1) return {'m': m} else: self.g.ndata['x'] = x def msg_func(edge): w = self.w.index_select(0, edge.data['type']) m = torch.bmm(edge.src['x'].unsqueeze(1), w).squeeze() m = m * edge.data['norm'].unsqueeze(1) return {'m': m} self.g.update_all(msg_func, self.rgcn_reduce) out = self.g.ndata.pop('x') if x is None: out = out + self.root else: out = out + torch.matmul(x, self.root) out = out + self.bias return out class RGCN(torch.nn.Module): def __init__(self, g, in_channels, out_channels, num_relations): super().__init__() self.conv1 = RGCNConv(g, in_channels, 16, num_relations, num_bases=30) self.conv2 = RGCNConv(g, 16, out_channels, num_relations, num_bases=30) def forward(self, x): x = F.relu(self.conv1(None)) x = self.conv2(x) return F.log_softmax(x, dim=1) class RGCNSPMVConv(torch.nn.Module): def __init__(self, g, in_channels, out_channels, num_relations, num_bases): super().__init__() self.g = g self.in_channels = in_channels self.out_channels = out_channels self.num_relations = num_relations self.num_bases = num_bases self.basis = Param(torch.Tensor(num_bases, in_channels, out_channels)) self.att = Param(torch.Tensor(num_relations, num_bases)) self.root = Param(torch.Tensor(in_channels, out_channels)) self.bias = Param(torch.Tensor(out_channels)) self.reset_parameters() def reset_parameters(self): size = self.num_bases * self.in_channels uniform(size, self.basis) uniform(size, self.att) uniform(size, self.root) uniform(size, self.bias) def forward(self, x): self.w = torch.matmul(self.att, self.basis.view(self.num_bases, -1)) self.w = self.w.view(self.num_relations, self.in_channels, self.out_channels) if x is None: def msg_func(edge): w = self.w.view(-1, self.out_channels) index = edge.data['type'] * self.in_channels + edge.src['id'] m = w.index_select(0, index) * edge.data['norm'].unsqueeze(1) return {'m': m} else: self.g.ndata['x'] = x def msg_func(edge): w = self.w.index_select(0, edge.data['type']) m = torch.bmm(edge.src['x'].unsqueeze(1), w).squeeze() m = m * edge.data['norm'].unsqueeze(1) return {'m': m} self.g.update_all(msg_func, fn.sum(msg='m', out='x')) out = self.g.ndata.pop('x') if x is None: out = out + self.root else: out = out + torch.matmul(x, self.root) out = out + self.bias return out class RGCNSPMV(torch.nn.Module): def __init__(self, g, in_channels, out_channels, num_relations): super().__init__() self.conv1 = RGCNSPMVConv(g, in_channels, 16, num_relations, num_bases=30) self.conv2 = RGCNSPMVConv(g, 16, out_channels, num_relations, num_bases=30) def forward(self, x): x = F.relu(self.conv1(None)) x = self.conv2(x) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/dgl/train.py000066400000000000000000000016261444262304300204270ustar00rootroot00000000000000import time import torch import torch.nn.functional as F def train_runtime(model, data, epochs, device): if hasattr(data, 'features'): x = torch.tensor(data.features, dtype=torch.float, device=device) else: x = None mask = data.train_mask if hasattr(data, 'train_mask') else data.train_idx y = torch.tensor(data.labels, dtype=torch.long, device=device)[mask] model = model.to(device) model.train() optimizer = torch.optim.Adam(model.parameters(), lr=0.01) if torch.cuda.is_available(): torch.cuda.synchronize() t_start = time.perf_counter() for epoch in range(epochs): optimizer.zero_grad() out = model(x) loss = F.nll_loss(out[mask], y.view(-1)) loss.backward() optimizer.step() if torch.cuda.is_available(): torch.cuda.synchronize() t_end = time.perf_counter() return t_end - t_start pytorch/benchmark/runtime/gat.py000066400000000000000000000011731444262304300173140ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch_geometric.nn import GATConv class GAT(torch.nn.Module): def __init__(self, in_channels, out_channels): super().__init__() self.conv1 = GATConv(in_channels, 8, heads=8, dropout=0.6) self.conv2 = GATConv(8 * 8, out_channels, dropout=0.6) def forward(self, data): x, edge_index = data.x, data.edge_index x = F.dropout(x, p=0.6, training=self.training) x = F.elu(self.conv1(x, edge_index)) x = F.dropout(x, p=0.6, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/gcn.py000066400000000000000000000010621444262304300173050ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch_geometric.nn import GCNConv class GCN(torch.nn.Module): def __init__(self, in_channels, out_channels): super().__init__() self.conv1 = GCNConv(in_channels, 16, cached=True) self.conv2 = GCNConv(16, out_channels, cached=True) def forward(self, data): x, edge_index = data.x, data.edge_index x = F.relu(self.conv1(x, edge_index)) x = F.dropout(x, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/main.py000066400000000000000000000023241444262304300174640ustar00rootroot00000000000000import os.path as osp from itertools import product import torch from runtime.gat import GAT from runtime.gcn import GCN from runtime.rgcn import RGCN from runtime.train import train_runtime from torch_geometric.datasets import Entities, Planetoid device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data') Cora = Planetoid(osp.join(root, 'Cora'), 'Cora') CiteSeer = Planetoid(osp.join(root, 'CiteSeer'), 'CiteSeer') PubMed = Planetoid(osp.join(root, 'PubMed'), 'PubMed') MUTAG = Entities(osp.join(root, 'EntitiesMUTAG'), 'MUTAG') # One training run before we start tracking duration to warm up GPU. model = GCN(Cora.num_features, Cora.num_classes) train_runtime(model, Cora[0], epochs=200, device=device) for d, Net in product([Cora, CiteSeer, PubMed], [GCN, GAT]): model = Net(d.num_features, d.num_classes) t = train_runtime(model, d[0], epochs=200, device=device) print(f'{str(d)[:-2]} - {Net.__name__}: {t:.2f}s') for d, Net in product([MUTAG], [RGCN]): model = Net(d[0].num_nodes, d.num_classes, d.num_relations) t = train_runtime(model, d[0], epochs=200, device=device) print(f'{str(d)[:-2]} - {Net.__name__}: {t:.2f}s') pytorch/benchmark/runtime/rgcn.py000066400000000000000000000012131444262304300174650ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch_geometric.nn import FastRGCNConv class RGCN(torch.nn.Module): def __init__(self, in_channels, out_channels, num_relations): super().__init__() self.conv1 = FastRGCNConv(in_channels, 16, num_relations, num_bases=30) self.conv2 = FastRGCNConv(16, out_channels, num_relations, num_bases=30) def forward(self, data): edge_index, edge_type = data.edge_index, data.edge_type x = F.relu(self.conv1(None, edge_index, edge_type)) x = self.conv2(x, edge_index, edge_type) return F.log_softmax(x, dim=1) pytorch/benchmark/runtime/train.py000066400000000000000000000014221444262304300176530ustar00rootroot00000000000000import time import torch import torch.nn.functional as F def train_runtime(model, data, epochs, device): optimizer = torch.optim.Adam(model.parameters(), lr=0.01) model = model.to(device) data = data.to(device) model.train() mask = data.train_mask if 'train_mask' in data else data.train_idx y = data.y[mask] if 'train_mask' in data else data.train_y if torch.cuda.is_available(): torch.cuda.synchronize() t_start = time.perf_counter() for epoch in range(epochs): optimizer.zero_grad() out = model(data) loss = F.nll_loss(out[mask], y) loss.backward() optimizer.step() if torch.cuda.is_available(): torch.cuda.synchronize() t_end = time.perf_counter() return t_end - t_start pytorch/benchmark/setup.py000066400000000000000000000005531444262304300162170ustar00rootroot00000000000000from setuptools import find_packages, setup setup( name='torch_geometric_benchmark', version='0.1.0', description='PyG Benchmark Suite', author='Matthias Fey', author_email='matthias.fey@tu-dortmund.de', url='https://github.com/pyg-team/pytorch_geometric_benchmark', install_requires=['scikit-learn'], packages=find_packages(), ) pytorch/benchmark/training/000077500000000000000000000000001444262304300163155ustar00rootroot00000000000000pytorch/benchmark/training/README.md000066400000000000000000000033331444262304300175760ustar00rootroot00000000000000# Training Benchmark ## Environment setup 1. Confirm that PyG is properly installed. 2. Install dataset package: ```bash pip install ogb ``` 3. Install `jemalloc` for performance benchmark: ```bash cd ${workspace} git clone https://github.com/jemalloc/jemalloc.git cd jemalloc git checkout 5.2.1 ./autogen.sh ./configure --prefix=${workspace}/jemalloc-bin make make install ``` ## Running benchmark 1. Set environment variables: ```bash source activate env_name export DNNL_PRIMITIVE_CACHE_CAPACITY=1024 export KMP_BLOCKTIME=1 export KMP_AFFINITY=granularity=fine,compact,1,0 jemalloc_lib=${workspace}/jemalloc-bin/lib/libjemalloc.so export LD_PRELOAD="$jemalloc_lib" export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000" ``` 2. Core binding, *e.g.*, single socket / single core / 4 cores per instance: ```bash OMP_NUM_THREADS=${CORES} numactl -C 0-${LAST_CORE} -m 0 CMD...... ``` 3. Execute benchmarks, *e.g.*: ```bash python training_benchmark.py --models=gcn --datasets=Reddit --num-workers=0 --batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --num-steps=50 python training_benchmark.py --models=gcn --datasets=Reddit --num-workers=0 --batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --num-steps=50 --use-sparse-tensor python training_benchmark.py --models=sage --datasets=ogbn-products --num-workers=0 --batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --num-steps=50 python training_benchmark.py --models=sage --datasets=ogbn-products --num-workers=0 --batch-sizes=512 --num-layers=2 --num-hidden-channels=64 --num-steps=50 --use-sparse-tensor ``` pytorch/benchmark/training/training_benchmark.py000066400000000000000000000350301444262304300225150ustar00rootroot00000000000000import argparse import ast import warnings from collections import defaultdict from contextlib import nullcontext import torch import torch.nn.functional as F from tqdm import tqdm from benchmark.utils import ( emit_itt, get_dataset, get_model, get_split_masks, save_benchmark_data, test, write_to_csv, ) from torch_geometric.loader import NeighborLoader from torch_geometric.nn import PNAConv from torch_geometric.profile import rename_profile_file, timeit, torch_profile supported_sets = { 'ogbn-mag': ['rgat', 'rgcn'], 'ogbn-products': ['edge_cnn', 'gat', 'gcn', 'pna', 'sage'], 'Reddit': ['edge_cnn', 'gat', 'gcn', 'pna', 'sage'], } def train_homo(model, loader, optimizer, device, progress_bar=True, desc="", trim=False): if progress_bar: loader = tqdm(loader, desc=desc) for batch in loader: optimizer.zero_grad() batch = batch.to(device) if hasattr(batch, 'adj_t'): edge_index = batch.adj_t else: edge_index = batch.edge_index if not trim: out = model(batch.x, edge_index) else: out = model( batch.x, edge_index, num_sampled_nodes_per_hop=batch.num_sampled_nodes, num_sampled_edges_per_hop=batch.num_sampled_edges, ) batch_size = batch.batch_size out = out[:batch_size] target = batch.y[:batch_size] loss = F.cross_entropy(out, target) loss.backward() optimizer.step() def train_hetero(model, loader, optimizer, device, progress_bar=True, desc="", trim=False): if trim: warnings.warn("Trimming not yet implemented for heterogeneous graphs") if progress_bar: loader = tqdm(loader, desc=desc) for batch in loader: optimizer.zero_grad() batch = batch.to(device) if len(batch.adj_t_dict) > 0: edge_index_dict = batch.adj_t_dict else: edge_index_dict = batch.edge_index_dict out = model(batch.x_dict, edge_index_dict) batch_size = batch['paper'].batch_size out = out['paper'][:batch_size] target = batch['paper'].y[:batch_size] loss = F.cross_entropy(out, target) loss.backward() optimizer.step() def run(args: argparse.ArgumentParser): csv_data = defaultdict(list) if args.write_csv == 'prof' and not args.profile: warnings.warn("Cannot write profile data to CSV because profiling is " "disabled") device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # If we use a custom number of steps, then we need to use RandomSampler, # which already does shuffle. shuffle = False if args.num_steps != -1 else True print('BENCHMARK STARTS') for dataset_name in args.datasets: assert dataset_name in supported_sets.keys( ), f"Dataset {dataset_name} isn't supported." print(f'Dataset: {dataset_name}') load_time = timeit() if args.measure_load_time else nullcontext() with load_time: data, num_classes = get_dataset(dataset_name, args.root, args.use_sparse_tensor, args.bf16) hetero = True if dataset_name == 'ogbn-mag' else False mask, val_mask, test_mask = get_split_masks(data, dataset_name) degree = None if torch.cuda.is_available(): amp = torch.cuda.amp.autocast(enabled=False) else: amp = torch.cpu.amp.autocast(enabled=args.bf16) inputs_channels = data[ 'paper'].num_features if dataset_name == 'ogbn-mag' \ else data.num_features for model_name in args.models: if model_name not in supported_sets[dataset_name]: print(f'Configuration of {dataset_name} + {model_name} ' f'not supported. Skipping.') continue print(f'Training bench for {model_name}:') for batch_size in args.batch_sizes: num_nodes = int(mask[-1].sum()) if hetero else int(mask.sum()) sampler = torch.utils.data.RandomSampler( range(num_nodes), num_samples=args.num_steps * batch_size) if args.num_steps != -1 else None for layers in args.num_layers: num_neighbors = args.num_neighbors if type(num_neighbors) is list: if len(num_neighbors) == 1: num_neighbors = num_neighbors * layers elif type(num_neighbors) is int: num_neighbors = [num_neighbors] * layers assert len( num_neighbors) == layers, \ f'''num_neighbors={num_neighbors} lenght != num of layers={layers}''' kwargs = { 'num_neighbors': num_neighbors, 'batch_size': batch_size, 'shuffle': shuffle, 'num_workers': args.num_workers, } subgraph_loader = NeighborLoader( data, input_nodes=mask, sampler=sampler, **kwargs, ) if args.evaluate: val_loader = NeighborLoader( data, input_nodes=val_mask, sampler=None, **kwargs, ) test_loader = NeighborLoader( data, input_nodes=test_mask, sampler=None, **kwargs, ) for hidden_channels in args.num_hidden_channels: print('----------------------------------------------') print(f'Batch size={batch_size}, ' f'Layers amount={layers}, ' f'Num_neighbors={num_neighbors}, ' f'Hidden features size={hidden_channels}, ' f'Sparse tensor={args.use_sparse_tensor}') params = { 'inputs_channels': inputs_channels, 'hidden_channels': hidden_channels, 'output_channels': num_classes, 'num_heads': args.num_heads, 'num_layers': layers, } if model_name == 'pna': if degree is None: degree = PNAConv.get_degree_histogram( subgraph_loader) print(f'Calculated degree for {dataset_name}.') params['degree'] = degree model = get_model( model_name, params, metadata=data.metadata() if hetero else None) model = model.to(device) model.train() optimizer = torch.optim.Adam(model.parameters(), lr=0.001) progress_bar = False if args.no_progress_bar else True train = train_hetero if hetero else train_homo # Define context manager parameters: cpu_affinity = subgraph_loader.enable_cpu_affinity( args.loader_cores ) if args.cpu_affinity else nullcontext() with amp, cpu_affinity: for _ in range(args.warmup): train( model, subgraph_loader, optimizer, device, progress_bar=progress_bar, desc="Warmup", trim=args.trim, ) with timeit(avg_time_divisor=args.num_epochs) as t: # becomes a no-op if vtune_profile == False with emit_itt(args.vtune_profile): for epoch in range(args.num_epochs): train( model, subgraph_loader, optimizer, device, progress_bar=progress_bar, desc=f"Epoch={epoch}", trim=args.trim, ) if args.evaluate: # In evaluate, throughput and # latency are not accurate. val_acc = test( model, val_loader, device, hetero, progress_bar=progress_bar) print( f'Val Accuracy: {val_acc:.4f}') if args.evaluate: test_acc = test(model, test_loader, device, hetero, progress_bar=progress_bar) print(f'Test Accuracy: {test_acc:.4f}') if args.profile: profile = torch_profile( args.export_chrome_trace, csv_data, args.write_csv) with profile: train(model, subgraph_loader, optimizer, device, progress_bar=progress_bar, desc="Profile training") if args.export_chrome_trace: rename_profile_file( model_name, dataset_name, str(batch_size), str(layers), str(hidden_channels), str(num_neighbors)) total_time = t.duration if args.num_steps != -1: total_num_samples = args.num_steps * batch_size else: total_num_samples = num_nodes throughput = total_num_samples / total_time latency = total_time / total_num_samples * 1000 print(f'Throughput: {throughput:.3f} samples/s') print(f'Latency: {latency:.3f} ms') num_records = 1 if args.write_csv == 'prof': # For profiling with PyTorch, we save the top-5 # most time consuming operations. Therefore, the # same data should be entered for each of them. num_records = 5 for _ in range(num_records): save_benchmark_data( csv_data, batch_size, layers, num_neighbors, hidden_channels, total_time, model_name, dataset_name, args.use_sparse_tensor, ) if args.write_csv: write_to_csv(csv_data, args.write_csv, training=True) if __name__ == '__main__': argparser = argparse.ArgumentParser('GNN training benchmark') add = argparser.add_argument add('--datasets', nargs='+', default=['ogbn-mag', 'ogbn-products', 'Reddit'], type=str) add('--use-sparse-tensor', action='store_true', help='use torch_sparse.SparseTensor as graph storage format') add('--models', nargs='+', default=['edge_cnn', 'gat', 'gcn', 'pna', 'rgat', 'rgcn'], type=str) add('--root', default='../../data', type=str, help='relative path to look for the datasets') add('--batch-sizes', nargs='+', default=[512, 1024, 2048, 4096, 8192], type=int) add('--num-layers', nargs='+', default=[2, 3], type=int) add('--num-hidden-channels', nargs='+', default=[64, 128, 256], type=int) add('--num-heads', default=2, type=int, help='number of hidden attention heads, applies only for gat and rgat') add('--num-neighbors', default=[10], type=ast.literal_eval, help='number of neighbors to sample per layer') add('--num-workers', default=2, type=int) add('--warmup', default=1, type=int) add('--profile', action='store_true') add('--vtune-profile', action='store_true') add('--bf16', action='store_true') add('--no-progress-bar', action='store_true', default=False, help='turn off using progress bar') add('--num-epochs', default=1, type=int) add('--num-steps', default=-1, type=int, help='number of steps, -1 means iterating through all the data') add('--cpu-affinity', action='store_true', help="Use DataLoader affinitzation.") add('--loader-cores', nargs='+', default=[], type=int, help="List of CPU core IDs to use for DataLoader workers.") add('--measure-load-time', action='store_true') add('--evaluate', action='store_true') add('--write-csv', choices=[None, 'bench', 'prof'], default=None, help='Write benchmark or PyTorch profile data to CSV') add('--export-chrome-trace', default=True, type=bool, help='Export chrome trace file. Works only with PyTorch profiler') add('--trim', action='store_true', help="Use `trim_to_layer` optimization") args = argparser.parse_args() run(args) pytorch/benchmark/utils/000077500000000000000000000000001444262304300156425ustar00rootroot00000000000000pytorch/benchmark/utils/__init__.py000066400000000000000000000006451444262304300177600ustar00rootroot00000000000000from .utils import emit_itt from .utils import get_dataset, get_dataset_with_transformation from .utils import get_model from .utils import get_split_masks from .utils import save_benchmark_data, write_to_csv from .utils import test __all__ = [ 'emit_itt', 'get_dataset', 'get_dataset_with_transformation', 'get_model', 'get_split_masks', 'save_benchmark_data', 'write_to_csv', 'test', ] pytorch/benchmark/utils/hetero_gat.py000066400000000000000000000033231444262304300203360ustar00rootroot00000000000000import torch from tqdm import tqdm from torch_geometric.nn import GAT, to_hetero class HeteroGAT(torch.nn.Module): def __init__(self, metadata, hidden_channels, num_layers, output_channels, num_heads): super().__init__() self.model = to_hetero( GAT((-1, -1), hidden_channels, num_layers, output_channels, add_self_loops=False, heads=num_heads), metadata) def forward(self, x_dict, edge_index_dict): return self.model(x_dict, edge_index_dict) @torch.no_grad() def inference(self, loader, device, progress_bar=False): self.model.eval() if progress_bar: loader = tqdm(loader, desc="Inference") for batch in loader: batch = batch.to(device) if len(batch.adj_t_dict) > 0: self.model(batch.x_dict, batch.adj_t_dict) else: self.model(batch.x_dict, batch.edge_index_dict) @torch.no_grad() def test(self, x, loader, device, progress_bar=False): self.model.eval() total_examples = total_correct = 0 if progress_bar: loader = tqdm(loader, desc="Evaluate") for batch in loader: batch = batch.to(device) if len(batch.adj_t_dict) > 0: out = self.model(batch.x_dict, batch.adj_t_dict) else: out = self.model(batch.x_dict, batch.edge_index_dict) batch_size = batch['paper'].batch_size out = out['paper'][:batch_size] pred = out.argmax(dim=-1) total_examples += batch_size total_correct += int((pred == batch['paper'].y[:batch_size]).sum()) return total_correct / total_examples pytorch/benchmark/utils/hetero_sage.py000066400000000000000000000032331444262304300205020ustar00rootroot00000000000000import torch from tqdm import tqdm from torch_geometric.nn import GraphSAGE, to_hetero class HeteroGraphSAGE(torch.nn.Module): def __init__(self, metadata, hidden_channels, num_layers, output_channels): super().__init__() self.model = to_hetero( GraphSAGE((-1, -1), hidden_channels, num_layers, output_channels), metadata) def forward(self, x_dict, edge_index_dict): return self.model(x_dict, edge_index_dict) @torch.no_grad() def inference(self, loader, device, progress_bar=False): self.model.eval() if progress_bar: loader = tqdm(loader, desc="Inference") for batch in loader: batch = batch.to(device) if len(batch.adj_t_dict) > 0: self.model(batch.x_dict, batch.adj_t_dict) else: self.model(batch.x_dict, batch.edge_index_dict) @torch.no_grad() def test(self, loader, device, progress_bar=False): self.model.eval() total_examples = total_correct = 0 if progress_bar: loader = tqdm(loader, desc="Evaluate") for batch in loader: batch = batch.to(device) if len(batch.adj_t_dict) > 0: out = self.model(batch.x_dict, batch.adj_t_dict) else: out = self.model(batch.x_dict, batch.edge_index_dict) batch_size = batch['paper'].batch_size out = out['paper'][:batch_size] pred = out.argmax(dim=-1) total_examples += batch_size total_correct += int((pred == batch['paper'].y[:batch_size]).sum()) return total_correct / total_examples pytorch/benchmark/utils/utils.py000066400000000000000000000155071444262304300173640ustar00rootroot00000000000000import os import os.path as osp from datetime import datetime import torch from ogb.nodeproppred import PygNodePropPredDataset from tqdm import tqdm import torch_geometric.transforms as T from torch_geometric.data import HeteroData from torch_geometric.datasets import OGB_MAG, Reddit from torch_geometric.nn import GAT, GCN, PNA, EdgeCNN, GraphSAGE from torch_geometric.utils import index_to_mask from .hetero_gat import HeteroGAT from .hetero_sage import HeteroGraphSAGE try: from torch.autograd.profiler import emit_itt except ImportError: from contextlib import contextmanager @contextmanager def emit_itt(*args, **kwargs): yield models_dict = { 'edge_cnn': EdgeCNN, 'gat': GAT, 'gcn': GCN, 'pna': PNA, 'sage': GraphSAGE, 'rgat': HeteroGAT, 'rgcn': HeteroGraphSAGE, } def get_dataset_with_transformation(name, root, use_sparse_tensor=False, bf16=False): path = osp.join(osp.dirname(osp.realpath(__file__)), root, name) transform = T.ToSparseTensor( remove_edge_index=False) if use_sparse_tensor else None if name == 'ogbn-mag': if transform is None: transform = T.ToUndirected(merge=True) else: transform = T.Compose([T.ToUndirected(merge=True), transform]) dataset = OGB_MAG(root=path, preprocess='metapath2vec', transform=transform) elif name == 'ogbn-products': if transform is None: transform = T.RemoveDuplicatedEdges() else: transform = T.Compose([T.RemoveDuplicatedEdges(), transform]) dataset = PygNodePropPredDataset('ogbn-products', root=path, transform=transform) elif name == 'Reddit': dataset = Reddit(root=path, transform=transform) data = dataset[0] if name == 'ogbn-products': split_idx = dataset.get_idx_split() data.train_mask = index_to_mask(split_idx['train'], size=data.num_nodes) data.val_mask = index_to_mask(split_idx['valid'], size=data.num_nodes) data.test_mask = index_to_mask(split_idx['test'], size=data.num_nodes) data.y = data.y.squeeze() if bf16: if isinstance(data, HeteroData): for node_type in data.node_types: data[node_type].x = data[node_type].x.to(torch.bfloat16) else: data.x = data.x.to(torch.bfloat16) return data, dataset.num_classes, transform def get_dataset(name, root, use_sparse_tensor=False, bf16=False): data, num_classes, _ = get_dataset_with_transformation( name, root, use_sparse_tensor, bf16) return data, num_classes def get_model(name, params, metadata=None): Model = models_dict.get(name, None) assert Model is not None, f'Model {name} not supported!' if name == 'rgat': return Model(metadata, params['hidden_channels'], params['num_layers'], params['output_channels'], params['num_heads']) if name == 'rgcn': return Model(metadata, params['hidden_channels'], params['num_layers'], params['output_channels']) if name == 'gat': return Model(params['inputs_channels'], params['hidden_channels'], params['num_layers'], params['output_channels'], heads=params['num_heads']) if name == 'pna': return Model(params['inputs_channels'], params['hidden_channels'], params['num_layers'], params['output_channels'], aggregators=['mean', 'min', 'max', 'std'], scalers=['identity', 'amplification', 'attenuation'], deg=params['degree']) return Model(params['inputs_channels'], params['hidden_channels'], params['num_layers'], params['output_channels']) def get_split_masks(data, dataset_name): if dataset_name == 'ogbn-mag': train_mask = ('paper', data['paper'].train_mask) test_mask = ('paper', data['paper'].test_mask) val_mask = ('paper', data['paper'].val_mask) else: train_mask = data.train_mask val_mask = data.val_mask test_mask = data.test_mask return train_mask, val_mask, test_mask def save_benchmark_data(csv_data, batch_size, layers, num_neighbors, hidden_channels, total_time, model_name, dataset_name, use_sparse_tensor): config = f'Batch size={batch_size}, ' \ f'#Layers={layers}, ' \ f'#Neighbors={num_neighbors}, ' \ f'#Hidden features={hidden_channels}' csv_data['DATE'].append(datetime.now().date()) csv_data['TIME (s)'].append(round(total_time, 2)) csv_data['MODEL'].append(model_name) csv_data['DATASET'].append(dataset_name) csv_data['CONFIG'].append(config) csv_data['SPARSE'].append(use_sparse_tensor) def write_to_csv(csv_data, write_csv='bench', training=False): import pandas as pd results_path = osp.join(osp.dirname(osp.realpath(__file__)), '../results/') os.makedirs(results_path, exist_ok=True) name = 'training' if training else 'inference' if write_csv == 'bench': csv_file_name = f'TOTAL_{name}_benchmark.csv' else: csv_file_name = f'TOTAL_prof_{name}_benchmark.csv' csv_path = osp.join(results_path, csv_file_name) index_label = 'TEST_ID' if write_csv == 'bench' else 'ID' with_header = not osp.exists(csv_path) df = pd.DataFrame(csv_data) df.to_csv(csv_path, mode='a', index_label=index_label, header=with_header) @torch.no_grad() def test(model, loader, device, hetero, progress_bar=True, desc="Evaluation") -> None: if progress_bar: loader = tqdm(loader, desc=desc) total_examples = total_correct = 0 if hetero: for batch in loader: batch = batch.to(device) if len(batch.adj_t_dict) > 0: edge_index_dict = batch.adj_t_dict else: edge_index_dict = batch.edge_index_dict out = model(batch.x_dict, edge_index_dict) batch_size = batch['paper'].batch_size out = out['paper'][:batch_size] pred = out.argmax(dim=-1) total_examples += batch_size total_correct += int((pred == batch['paper'].y[:batch_size]).sum()) else: for batch in loader: batch = batch.to(device) if hasattr(batch, 'adj_t'): edge_index = batch.adj_t else: edge_index = batch.edge_index out = model(batch.x, edge_index) batch_size = batch.batch_size out = out[:batch_size] pred = out.argmax(dim=-1) total_examples += batch_size total_correct += int((pred == batch.y[:batch_size]).sum()) return total_correct / total_examples pytorch/codecov.yml000066400000000000000000000003741444262304300147210ustar00rootroot00000000000000# see https://docs.codecov.io/docs/codecov-yaml coverage: range: 80..100 round: down precision: 2 status: project: default: target: 80% threshold: 1% patch: default: target: 80% threshold: 1% pytorch/conda/000077500000000000000000000000001444262304300136345ustar00rootroot00000000000000pytorch/conda/pyg/000077500000000000000000000000001444262304300144335ustar00rootroot00000000000000pytorch/conda/pyg/README.md000066400000000000000000000001151444262304300157070ustar00rootroot00000000000000``` ./build_conda.sh 3.9 2.0.0 cu117 # python, pytorch and cuda version ``` pytorch/conda/pyg/build_conda.sh000077500000000000000000000030601444262304300172340ustar00rootroot00000000000000#!/bin/bash export PYTHON_VERSION=$1 export TORCH_VERSION=$2 export CUDA_VERSION=$3 export CONDA_PYTORCH_CONSTRAINT="pytorch==${TORCH_VERSION%.*}.*" if [ "${CUDA_VERSION}" = "cpu" ]; then export CONDA_CUDATOOLKIT_CONSTRAINT="cpuonly # [not osx]" else case $CUDA_VERSION in cu118) export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.8.*" ;; cu117) export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.7.*" ;; cu116) if [ "${TORCH_VERSION}" = "1.12.0" ]; then export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.6.*" else export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.6.*" fi ;; cu115) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.5.*" ;; cu113) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.3.*" ;; cu111) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.1.*" ;; cu102) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==10.2.*" ;; cu101) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==10.1.*" ;; *) echo "Unrecognized CUDA_VERSION=$CUDA_VERSION" exit 1 ;; esac fi echo "PyTorch $TORCH_VERSION+$CUDA_VERSION" echo "- $CONDA_PYTORCH_CONSTRAINT" echo "- $CONDA_CUDATOOLKIT_CONSTRAINT" if [ "${TORCH_VERSION}" = "1.12.0" ] && [ "${CUDA_VERSION}" = "cu116" ]; then conda build . -c pytorch -c pyg -c default -c nvidia -c conda-forge --output-folder "$HOME/conda-bld" else conda build . -c pytorch -c pyg -c default -c nvidia --output-folder "$HOME/conda-bld" fi pytorch/conda/pyg/meta.yaml000066400000000000000000000020151444262304300162430ustar00rootroot00000000000000package: name: pyg version: 2.3.1 source: url: https://files.pythonhosted.org/packages/06/a5/9f5af849c4185da5ea55f70ef17e23f93355cd4e989d82cfc8ba2d8747af/torch_geometric-2.3.1.tar.gz requirements: host: - pip - python {{ environ.get('PYTHON_VERSION') }} run: - python {{ environ.get('PYTHON_VERSION') }} - {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }} - {{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT') }} - psutil - tqdm - jinja2 - pyparsing - numpy - scipy - requests - scikit-learn build: string: py{{ environ.get('PYTHON_VERSION').replace('.', '') }}_torch_{{ environ['TORCH_VERSION'] }}_{{ environ['CUDA_VERSION'] }} number: 1 script: pip install . test: imports: - torch_geometric - torch_geometric.nn - torch_geometric.data - torch_geometric.utils - torch_geometric.datasets - torch_geometric.transforms about: home: https://github.com/pyg-team/pytorch_geometric license: MIT summary: Graph Neural Network Library for PyTorch pytorch/conda/pytorch-geometric/000077500000000000000000000000001444262304300173005ustar00rootroot00000000000000pytorch/conda/pytorch-geometric/README.md000066400000000000000000000001151444262304300205540ustar00rootroot00000000000000``` ./build_conda.sh 3.9 2.0.0 cu117 # python, pytorch and cuda version ``` pytorch/conda/pytorch-geometric/build_conda.sh000077500000000000000000000030701444262304300221020ustar00rootroot00000000000000#!/bin/bash export PYTHON_VERSION=$1 export TORCH_VERSION=$2 export CUDA_VERSION=$3 export CONDA_PYTORCH_CONSTRAINT="pytorch==${TORCH_VERSION%.*}.*" if [ "${CUDA_VERSION}" = "cpu" ]; then export CONDA_CUDATOOLKIT_CONSTRAINT="cpuonly # [not osx]" else case $CUDA_VERSION in cu118) export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.8.*" ;; cu117) export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.7.*" ;; cu116) if [ "${TORCH_VERSION}" = "1.12.0" ]; then export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.6.*" else export CONDA_CUDATOOLKIT_CONSTRAINT="pytorch-cuda==11.6.*" fi ;; cu115) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.5.*" ;; cu113) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.3.*" ;; cu111) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==11.1.*" ;; cu102) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==10.2.*" ;; cu101) export CONDA_CUDATOOLKIT_CONSTRAINT="cudatoolkit==10.1.*" ;; *) echo "Unrecognized CUDA_VERSION=$CUDA_VERSION" exit 1 ;; esac fi echo "PyTorch $TORCH_VERSION+$CUDA_VERSION" echo "- $CONDA_PYTORCH_CONSTRAINT" echo "- $CONDA_CUDATOOLKIT_CONSTRAINT" if [ "${TORCH_VERSION}" = "1.12.0" ] && [ "${CUDA_VERSION}" = "cu116" ]; then conda build . -c pytorch -c rusty1s -c default -c nvidia -c conda-forge --output-folder "$HOME/conda-bld" else conda build . -c pytorch -c rusty1s -c default -c nvidia --output-folder "$HOME/conda-bld" fi pytorch/conda/pytorch-geometric/meta.yaml000066400000000000000000000020331444262304300211100ustar00rootroot00000000000000package: name: pytorch-geometric version: 2.3.1 source: url: https://files.pythonhosted.org/packages/06/a5/9f5af849c4185da5ea55f70ef17e23f93355cd4e989d82cfc8ba2d8747af/torch_geometric-2.3.1.tar.gz requirements: host: - pip - python {{ environ.get('PYTHON_VERSION') }} run: - python {{ environ.get('PYTHON_VERSION') }} - {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }} - {{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT') }} - psutil - tqdm - jinja2 - pyparsing - numpy - scipy - requests - scikit-learn build: string: py{{ environ.get('PYTHON_VERSION').replace('.', '') }}_torch_{{ environ['TORCH_VERSION'] }}_{{ environ['CUDA_VERSION'] }} number: 1 script: pip install . test: imports: - torch_geometric - torch_geometric.nn - torch_geometric.data - torch_geometric.utils - torch_geometric.datasets - torch_geometric.transforms about: home: https://github.com/pyg-team/pytorch_geometric license: MIT summary: Graph Neural Network Library for PyTorch pytorch/docker/000077500000000000000000000000001444262304300140175ustar00rootroot00000000000000pytorch/docker/Dockerfile000066400000000000000000000132771444262304300160230ustar00rootroot00000000000000FROM ubuntu:18.04 # metainformation LABEL org.opencontainers.image.version = "2.3.1" LABEL org.opencontainers.image.authors = "Matthias Fey" LABEL org.opencontainers.image.source = "https://github.com/pyg-team/pytorch_geometric" LABEL org.opencontainers.image.licenses = "MIT" LABEL org.opencontainers.image.base.name="docker.io/library/ubuntu:18.04" RUN apt-get update && apt-get install -y apt-transport-https ca-certificates && \ rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y --no-install-recommends apt-utils gnupg2 curl && \ curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \ echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \ echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list &&\ apt-get purge --autoremove -y curl && \ rm -rf /var/lib/apt/lists/* ENV CUDA_VERSION 10.1.243 ENV NCCL_VERSION 2.4.8 ENV CUDA_PKG_VERSION 10-1=$CUDA_VERSION-1 ENV CUDNN_VERSION 7.6.5.32 RUN apt-get update && apt-get install -y --no-install-recommends \ cuda-cudart-$CUDA_PKG_VERSION \ cuda-compat-10-1 && \ ln -s cuda-10.1 /usr/local/cuda && \ rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \ cuda-libraries-$CUDA_PKG_VERSION \ cuda-nvtx-$CUDA_PKG_VERSION \ libcublas10=10.2.1.243-1 \ libnccl2=$NCCL_VERSION-1+cuda10.1 && \ apt-mark hold libnccl2 && \ rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \ cuda-libraries-dev-$CUDA_PKG_VERSION \ cuda-nvml-dev-$CUDA_PKG_VERSION \ cuda-minimal-build-$CUDA_PKG_VERSION \ cuda-command-line-tools-$CUDA_PKG_VERSION \ libnccl-dev=$NCCL_VERSION-1+cuda10.1 \ libcublas-dev=10.2.1.243-1 \ && \ rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y --no-install-recommends \ libcudnn7=$CUDNN_VERSION-1+cuda10.1 \ libcudnn7-dev=$CUDNN_VERSION-1+cuda10.1 \ && \ apt-mark hold libcudnn7 && \ rm -rf /var/lib/apt/lists/* ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs # NVIDIA docker 1.0. LABEL com.nvidia.volumes.needed="nvidia_driver" LABEL com.nvidia.cuda.version="${CUDA_VERSION}" RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \ echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH} ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64 # NVIDIA container runtime. ENV NVIDIA_VISIBLE_DEVICES all ENV NVIDIA_DRIVER_CAPABILITIES compute,utility ENV NVIDIA_REQUIRE_CUDA "cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411" # PyTorch (Geometric) installation RUN rm /etc/apt/sources.list.d/cuda.list && \ rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-get update && apt-get install -y \ curl \ ca-certificates \ vim \ sudo \ git \ bzip2 \ libx11-6 \ && rm -rf /var/lib/apt/lists/* # Create a working directory. RUN mkdir /app WORKDIR /app # Create a non-root user and switch to it. RUN adduser --disabled-password --gecos '' --shell /bin/bash user \ && chown -R user:user /app RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user USER user # All users can use /home/user as their home directory. ENV HOME=/home/user RUN chmod 777 /home/user # Install Miniconda. RUN curl -so ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ && chmod +x ~/miniconda.sh \ && ~/miniconda.sh -b -p ~/miniconda \ && rm ~/miniconda.sh ENV PATH=/home/user/miniconda/bin:$PATH ENV CONDA_AUTO_UPDATE_CONDA=false # Create a Python 3.6 environment. RUN /home/user/miniconda/bin/conda install conda-build \ && /home/user/miniconda/bin/conda create -y --name py36 python=3.6.5 \ && /home/user/miniconda/bin/conda clean -ya ENV CONDA_DEFAULT_ENV=py36 ENV CONDA_PREFIX=/home/user/miniconda/envs/$CONDA_DEFAULT_ENV ENV PATH=$CONDA_PREFIX/bin:$PATH # CUDA 10.0-specific steps. RUN conda install -y -c pytorch \ cudatoolkit=10.1 \ "pytorch=1.4.0=py3.6_cuda10.1.243_cudnn7.6.3_0" \ torchvision=0.5.0=py36_cu101 \ && conda clean -ya # Install HDF5 Python bindings. RUN conda install -y h5py=2.8.0 \ && conda clean -ya RUN pip install h5py-cache==1.0 # Install TorchNet, a high-level framework for PyTorch. RUN pip install torchnet==0.0.4 # Install Requests, a Python library for making HTTP requests. RUN conda install -y requests=2.19.1 \ && conda clean -ya # Install Graphviz. RUN conda install -y graphviz=2.40.1 python-graphviz=0.8.4 \ && conda clean -ya # Install OpenCV3 Python bindings. RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \ libgtk2.0-0 \ libcanberra-gtk-module \ && sudo rm -rf /var/lib/apt/lists/* RUN conda install -y -c menpo opencv3=3.1.0 \ && conda clean -ya # Install PyG. RUN CPATH=/usr/local/cuda/include:$CPATH \ && LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \ && DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH RUN pip install scipy RUN pip install --no-index torch_scatter -f https://data.pyg.org/whl/torch-1.4.0+cu101.html \ && pip install --no-index torch_sparse -f https://data.pyg.org/whl/torch-1.4.0+cu101.html \ && pip install --no-index torch_cluster -f https://data.pyg.org/whl/torch-1.4.0+cu101.html \ && pip install --no-index torch_spline_conv -f https://data.pyg.org/whl/torch-1.4.0+cu101.html \ && pip install torch-geometric # Set the default command to python3. CMD ["python3"] pytorch/docker/README.md000066400000000000000000000051621444262304300153020ustar00rootroot00000000000000# Docker You can run PyG with CUDA 10.1 inside a docker image. The creation of [our dockerfile](https://github.com/pyg-team/pytorch_geometric/blob/master/docker/Dockerfile) refers to the dockerfiles provided by [NVIDIA](https://gitlab.com/nvidia/cuda/tree/ubuntu18.04) and [PyTorch](https://github.com/anibali/docker-pytorch). 1. Download the dockerfile to your host server. 2. `$ docker build -t "custom image name"` 3. `$ docker run --rm -it --init --runtime=nvidia --ipc=host --network=host --volume=$PWD:/app -e NVIDIA_VISIBLE_DEVICES=0 "custom image name" /bin/bash` If you encounter any problems, please feel free to contact . # Singularity You can run PyG inside a singularity image. An example singularity file can be found in this folder. You might have to modify the script; depending on your needs, modify the following: - **cuda version 10.1**: If you need another version, change `From: nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04` to the corresponding tag from . Same if you want to use anything but Ubuntu 16.04. Your host has to have at least this cuda version! - **python version 3.7.2**: If you need another version, change `pyenv install 3.7.2` and the following lines to the corresponding version. - **pytorch version 1.3.0**: If you need another version, change `pip install torch==1.3.0`. - **pytorch_geometric versions**: This uses specific versions for each of the `pytorch_geometric` requirements (scatter: 1.4.0, sparse: 0.4.3, cluster: 1.4.5, geometric 1.3.2). To change these, change the corresponding `git checkout` lines near the bottom. - **cuda compute capability 5.0 and 6.1**: If you run it on multiply systems (likely, with singularity), ensure that all compute capabilities are listed here. If you have the cuda samples installed, check it with (for example) `/usr/local/cuda-10.1/extras/demo_suite/deviceQuery | grep 'CUDA Capability'`; if not, check [here](https://en.wikipedia.org/wiki/CUDA#GPUs_supported). Note: If your harddisk runs full after multiple builds, this is known and apparently working as intended; delete the `/tmp/sbuild-XXXXXXXXX` files. ## Building and Using the Container To build the container, run `sudo singularity build geometric.sif singularity` then wait. Once finished, you can run the GAT example in the folder you built the image in by calling ``` wget https://raw.githubusercontent.com/pyg-team/pytorch_geometric/master/examples/gat.py ``` (to download the sample), then ``` singularity exec geometric.sif python3 gat.py ``` to run on the CPU, or ``` singularity exec --nv geometric.sif python3 gat.py ``` to run on the GPU. pytorch/docker/singularity000066400000000000000000000036221444262304300163170ustar00rootroot00000000000000Bootstrap: docker From: nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 %post CURDIR=$(pwd) # Set timezone to Etc/UTC for tzdata. See issue #4365 for more details. TZ=Etc/UTC && \ ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \ echo $TZ > /etc/timezone apt-get update -y apt-get install -y tmux nano git wget apt-get install -y --no-install-recommends \ build-essential \ gfortran \ libssl-dev \ zlib1g-dev \ libbz2-dev \ libreadline-dev \ libsqlite3-dev \ wget \ curl \ llvm \ libncurses5-dev \ xz-utils \ tk-dev \ libxml2-dev \ libxmlsec1-dev \ libffi-dev \ liblzma-dev \ liblapack-dev \ libopenblas-dev \ libhdf5-dev export PYENV_ROOT=/opt/pyenv export PATH="/opt/pyenv/bin:$PATH" curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash pyenv install 3.7.2 echo 'export PATH=/opt/pyenv/versions/3.7.2/bin/:$PATH' >> $SINGULARITY_ENVIRONMENT export PATH=/opt/pyenv/versions/3.7.2/bin/:$PATH pip install torch==1.3.0 mkdir -p $SINGULARITY_ROOTFS/tmp/sing_build_cuda cd $SINGULARITY_ROOTFS/tmp/sing_build_cuda export TORCH_CUDA_ARCH_LIST="5.0 6.1" git clone https://github.com/rusty1s/pytorch_scatter.git && \ cd ./pytorch_scatter && \ git checkout 1.4.0 && \ python3 -m pip install . && \ cd .. git clone https://github.com/rusty1s/pytorch_sparse.git && \ cd ./pytorch_sparse && \ git checkout 0.4.3 && \ python3 -m pip install . && \ cd .. git clone https://github.com/rusty1s/pytorch_cluster.git && \ cd ./pytorch_cluster && \ git checkout 1.4.5 && \ python3 -m pip install . && \ cd .. git clone https://github.com/pyg-team/pytorch_geometric.git && \ cd ./pytorch_geometric && \ git checkout 1.3.2 && \ python3 -m pip install . && \ cd .. cd $CURDIR rm -rf $SINGULARITY_ROOTFS/tmp/sing_build_cuda pytorch/docs/000077500000000000000000000000001444262304300135005ustar00rootroot00000000000000pytorch/docs/Makefile000066400000000000000000000003271444262304300151420ustar00rootroot00000000000000SPHINXBUILD = sphinx-build SPHINXPROJ = pytorch_geometric SOURCEDIR = source BUILDDIR = build .PHONY: help Makefile %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0) pytorch/docs/README.md000066400000000000000000000010111444262304300147500ustar00rootroot00000000000000# Building Documentation To build the documentation: 1. [Build and install](https://github.com/pyg-team/pytorch_geometric/blob/master/.github/CONTRIBUTING.md#developing-pytorch-geometric) PyG from source. 2. Install [Sphinx](https://www.sphinx-doc.org/en/master/) theme via ``` pip install git+https://github.com/pyg-team/pyg_sphinx_theme.git ``` 3. Generate the documentation file via: ``` cd docs make html ``` The documentation is now available to view by opening `docs/build/html/index.html`. pytorch/docs/requirements.txt000066400000000000000000000002261444262304300167640ustar00rootroot00000000000000https://download.pytorch.org/whl/cpu/torch-1.13.0%2Bcpu-cp38-cp38-linux_x86_64.whl numpy>=1.19.5 git+https://github.com/pyg-team/pyg_sphinx_theme.git pytorch/docs/source/000077500000000000000000000000001444262304300150005ustar00rootroot00000000000000pytorch/docs/source/.gitignore000066400000000000000000000000131444262304300167620ustar00rootroot00000000000000generated/ pytorch/docs/source/_figures/000077500000000000000000000000001444262304300166035ustar00rootroot00000000000000pytorch/docs/source/_figures/.gitignore000066400000000000000000000000221444262304300205650ustar00rootroot00000000000000*.aux *.log *.pdf pytorch/docs/source/_figures/architecture.pdf000066400000000000000000003652771444262304300220040ustar00rootroot00000000000000%PDF-1.5 %ΠΤΕΨ 8 0 obj << /Length 3739 /Filter /FlateDecode >> stream xΪΕ\Ks7ΎλWτQ:4ΓχγΆymjSϋˆΦήS6‡ΙX–]5’I©έΪ_Ώ›Αnjd;š€’Ψ£Oh?€ š9]OrϊξL>σ·‚?ε€&­£ΠήOR€δ§ύΝΩ/Θ|7eΑ3)LŠnϊΟ„π%ϋ₯„K!xό₯ή+aθTΐΑ‚Q!Lχן"©΅!Ί`'«„•Κ§ιf:›π!ΓNXγ­i0`^„`]˜¬η8f’°JG°2?N°Π48t ή©&6:Ε&¦P…φ\ω \?Όa£Hl£σ£dέΐ wΑˆο:λͺ6ΰή―κŠVBΘIγβ ”¨όΗ?οo¦ž&—όρ™žΏ¬cwˆFβΊ{%ς ι!MΖRΥ8=)Ζh§#αDΤΖz‚»¬PAjΥzˆ&εγ‚]φ~Lο‚[|_0›„”Μ«£ΫŒŽ€‹μΉΊ„2V0΄.k4ˆτ`ΜpLεC“m4 σΐΐ ;ΘΐͺˆJY%ΈK‘–tρ9™λ83hL‡Ώ6ΰ "aΘ•‘ΠΌ%Ο&Ί'Λ<ςΦt(YG4'l½HqP«Ϊ€{ίnS(L0 H‘ώj–UmuΏώrωα—3¨&Qcœ„}ΔΓΐYΤr™([ΐ/ώr£¦oξΞ.ρFΓ’Ϋά”{yˊ†ά ™ AΦCbβ$ Šg9Δ)H(g Δ:Qˆ4aτ#ϋ`(­ ]V’Ÿ‚’ΎŽ\dΌ«ga՚:,YΌυW!Ωr”oF Hυ| δΝ—a[γrrŽ}N·fΙ[KŠŒS‚©=¦Θ*TdrŠ|Ό»ίΏ«ςv’Υ°•dQ—,χγOrzΏϋ~ΚηX(QςZΚΐζ|ήNŠ{Έ±›τaΏ{|ΌΊ_'ζ•~}b^Ό77χ½|<`ΎƒbHχaΥ½Gb°x€ά#απν9"TyθΥψC Y,V [―t"X 4eY$`O„ ˜ CrςpX<4GC«Mm2|ΰ΅%,Θ¬cQ‘ΰicΜ‰ΒΒ@ν°J9Uθ#Β‚_˜^ , ξq*τaρaw°)WVκυQ1WƒΉ9%ϋ‘r6ωπye˜ς28t¦ 0ŠΧaUͺ eΛ^](‡>έ]γ‡­‰«gV6εg¦ύ ‚˜νŒ„L0pPp²O@$††α ‰(zΧ0¨Ίαη˜7š€ccΜ'Ϊ 9‡gƒd– D#-΄Œμ E¨»κτμΉ’„–XυFB”Š9φOίΖoHΥ"WΐͺνcΥ€6>™=πX όjWΥdψΙΓv'ΗG_<ξ5Ζ)"=d0oSΑ"Td2QΏΩ=ξΆ νdΧ]l™›1/ΟO%Aσ*€ˆΗ–%V~ΖθIX]dJΦ6ΐγ>yΞΟ¨„²Κ$ΞOΨ~€•I?aƒR6FCό$`Ο%†R –4¨εE(yl”Σψ ͺjμ—g”>th΅«MBΆo½ΦS”΄Qϋ?r›>Ό8E5l9qΗΓ?AΡ*Tdˆ’›#m/·’g±cn†œ&}&ΝΞO¨4R§Ž‘JΑ:EΕΦ (+‚α«IγiC9SΪΈT'iŒ#m8_›ζœ°ΉΦIκ†"”΄RmŽ†0ΊΘΩΪΐj^›€œ0παΒΦfήQΎ*λ k«Φlτ„γ„­Bœ°―/Ό;ΏΗwϋπΎ»Ώ€ƒνωΝ6Νφ―kbίΙ‰ΌμjΚ‹jy¦m K΅ ¬yEY<»*Μw„Dauα>9O·°‘ŠδP’ΖDΊΤς-]=@hWJ›z›€AU^4Zmc“Fώλs.)4>”Ύ6–„Ζ½‰ΗοKΪ'θ­€ΰ¨i ψ#(πζC+‚–˜ΌŒπ6`uN¨P;Τ ¨žTδP©εEiρδ—e³m(¬–Θ šCA9 Yt!dΟΥ&™ υrψž•PΨp€ΔR¨ΝΑ ͺ ŽHhΥϋЁվ6yaΰΓ’›UŸ—4Ύ‹1y'Πξε7p§0•/G0νΊ„™?ΐρwπ"U…rFόΫϋΫχσW»Ηύ»χ·Χ›½|υΔ* £ffΥ ŠMΘ΄ΖXzϊF‡…œ:ώFι?8ΕP€dpψώΦ—AΏ δόehεV›ƒVΌ)Γ Μ4η VΦ kœr]@Ρ€ƒζσpŒ‘˜ΑŒΕ₯0m΅hxs!2³τ‘ΡTNEd8q Μ3DR‘~‘α8ώϊέΟΈ₯k'Ο_νn.΄ [rBXbX—— $ΊΡψ•MΞk¦7η΅VR‘ <±kΥ“Ε·Ž4ƒ«ΚHΝΐj ›„ά0ςcIΞdε1J£εΙΔ“q"KC ς §‹TΗιWΏώ| εi-Y?Όϋb―ΖYΧ­ΕΦ!³ϋ0Ο>Φ3{tb)Ώ0Q ±ΖUz‘ͺBν\™MύλέξΝΥύΓ(|ωSΫπΝI€©?~l°Ι―~[+2αΛχL'²uΘ|ΈΊί=ζσ‡;ίXΈzbυfΈ(~Ϊ6δΌι8ή”wΐ«V[}‹»jΚΝΫΦέΌιπ-ο…Χ½ΐC‡Φžα<θ,ΞƒδžιΞ“σ¦©9oϊžσ¦;š_Ί¨Ž‘q›nμΦƒτZΈΪv¬/ 5ωfΓi:“Ιb!sΌ1Ydx_ςŸ˜{voξ~}ܐΆ“^sv±ε€ΙyΣ…μ)Λ[?ŒM·…‘ζƒX六ΌχΓΠΪ—αQ&<†Xχgή΄09gΫΣ›^θΌι˜rΞςήΟΌiΐΞ›νΦ‡kνόΜXv&|Sx’Φ˜h”=ήω)2Όρσχ #©Χ³;Όίξρύέ횿ύƒλΫ8‹Y'νψδΑη›ΣJC0Ι5+Ϊσ·’m‰‘ωHΘ ΨR$yΗίο33σ 3:=ϋ^}ΦΚΕ†ƒmL±w&iΝ§β λqœ¬8¬πj0›¬ΊeλΥ5₯6‚ςΕ2c—n‚‹/|tVΣl% _ ‘*sμΥ0œΪƒ\DCόM―†a[Μ­ 6ηώπλΓΰΒΔJ½>vͺfζΐO>β΅:ψΦΞ1u‘ƒ8P˜ΐ9’πμ`mG₯†ρΰax=gœ¦Ήω›V<|ΘkόζXn)#t•κύΓύΥdΟw·όφΏ΅o}΅M嫁W έbϋθμ+ƒε…Ί…²ΐβH.CΩ g’\@½*/Ξ›«G‹G56]ρΌ$πμ˜`ρ.ΎUX=Υ8΄_Ξν·o§Λiκ=Vϋ“΄ώύ•f~R’…ΔΧ½·•‡ΥΤXU”»Ϋ0Ώ›Ώ;Ν?Mρ?@oό6’‰x‹›˜b.Zη€η8oe+JZ―SΔ@φΔ:-ΰδΰz•α8ε£±ζ#4ފžXa‡WŠ"Ά9ρ‹ϋ2ov ƒΓ³ƒ3Θ^le/…•„zΨ㠈«œ’PI.{Υs*·Nƒ­ κ@νφΒs,j‘|SΔ#œT%#Β>`ƒL }’7Ω?ΐ¦ή$+,~ο₯7)ΐΠΡ†I3qV%ητCœ>Ιa ~1ηNQϊ–δ₯ΧωzVşΞ:#ιΧώ½Ν>χHω?{ZςΏ endstream endobj 4 0 obj << /Type /XObject /Subtype /Image /Width 1025 /Height 205 /BitsPerComponent 8 /ColorSpace /DeviceRGB /SMask 14 0 R /Length 8089 /Filter /FlateDecode >> stream xΪνέΝnγF†a_‡ ϊq[rSR&Πjv\؝d‘ϋɝΞb€Α€mJNϊ4΄Ω­8펻ͺX¬ͺsκ}Π’LΖ‹y>ΦίΕδ™N§———WWWοžΡ €2σωόeΝΪΘΠa»:ν ΪM.?{χ-΄›ΊZvΛΣΝ%M€&χΟP7υ?"€¬κ"DYΥDˆ²ͺ‰dU Θͺώ'@UύO€ «ϊŸ@Vυ? °ιt: ’έ Ύωu£Ρh<Σς‰(Šβ >p-DΞυ? 0“½ΪhB£ œ”4Κ hˆzͺ‰yF€Χ‰ ωkσΌhώ†sD DΊλ"ΰ«q ωλb±ΰ|ˆ€ΎϊŸ@x[σ)Š‚G Djκ"ΐ0ΜηsN€(¨‰D«1BL "ιυ?€@ "€¬κ" Kΰ„ˆ€ΈϊŸ@θ8G`½^sZ‰D€@PύO tΗς‘D"@a»κRͺε©,iF"ƒ‚ˆD€@„Žο›?·οiF")€@ˆ2©ΥŠfL6\uπΕΟiΧπ S˜•t*ϊΗ. ώwύΣiΛΧΓvύr4Ρ‘ZΥ»Υγ–Lα-ττΫ›Ϊ©ωk›zΚμ&όΆ"%Όuˆ΄ ¦ώίλ¦\wώνΩσ°{k“ ’_ύ<}τΠ  ΠV€ŒϊίΓψŸ₯σoΏΏ›MVΪl8S""€Ϋgγ™B@Iύτ ΡvΙΙΌgN7€τϊPY,@ΚΌΉ ΅X,HDN%@sύoκ²#€νf₯P""eυΏΛ±έo$¨ˆρxμΌ”eσδ€H¬7ΦΏχÞΕ!5DΫONG€‚ϊω“Ό·ύΥoΎη ͺ‰fpή‰ˆΔՍ?~YΨώφΣ―œCUΰβσ>t¨―[Αv#€|OžΝftδP?E›OUWLΦ¬ž,½]J5kϋ½VΆμνd2iΝΛO›Οηη»ίΥgηΫ‚ˆ›Cs~›£ψζυπϊ?Έόl³ΩδšζzωUj¦m‡φg³+E#OΗjΕϋ— υ6‹αL™G^7RS–eSŠ˜1MjΏζ§I¬mΌΟβoΛ<‡m›J;‘6qΫ έδ:ΉΎΎΦΎΘΞ ^{Ό]ςώ+ˊV+"€ΔŠΛͺ|Z,άˆ)\΄=Uw_Όϋ iΆ@RΪϊό›}}ΏxC€:ω«šΫTί—ΗΛΡ`<ϋm²;ώό#υίΪΓ­ςmΕTF€ΆsάΧϋ=«@AαMpψ¨Ώ¨jy­m{κm T_ύ)±jΒof’ώxŸiwθ΅5X>ϊό·ΪͺμΗΌy€Ώέi€2΄c>}υφΪ–Š―•OV Viχ²S@qh*XΉ5αf³9ΟJˆ.ύ¦‘šί’`vœΥέ¦(~ςΟi―vŠΚpα{F°ξΤˆθL"ΐyΖn  ~#@SŸhΊ>fX4ΕάlΨ_Zμ#„Jμ) [ή›©ή©ΑD“C³z"^d£#σ7₯a Eυ π˜Aγ<`Αή χqy€“ _oΗκΧoˆΫPL€h[Ζ[ογnMPLξδeYj}4€ΦΑ‘84Ω0Ωκ.…VυƒAΘFŽΖ£sξ τ&“I ν@wδΐΌ·=κίT8ˆ`xhVonU6TžŸΗ«)κq«šXέέλ.ΏKΑ„—fǐχέ#€ΥΫυ©psάxoϋ§ή2ν°}F7₯ˆˆVZ,D€ΔxQR@ΗPEΰN–(ο+rŽiv’ΡYξoΏs›βYόαxν|ψ•Ά 5˜—ω άόω%呐ΤZ@*#ΐp8|'Sψρl]Ύ\Q¦Πv?d)}CΎz@ΊD€”Ρ‘ H½sο8ύͺ’ͺχΞ HΩκ@τ˜šΎήαGS F£w’N]Κ-‰UŸΈΎ‘ξί>· "GSXB„£σ,ΰjyΊΡ3ώαΦ5θZˆ`^ηX½†J½P%D€³žf,^~6ŸΟ_ξ₯ΫΣ*£!S€σΥεύpΗ]ΕLώ=7Kν,€ΌDRά^3ΐΦi_Τοοο΄νˆqtέαJO―ΐͺΎΥτ^(Ν© :šΧcΡό(Ϋ‘ω~WΕ έ>OΗυ_6T°}ΐ½ΏoλδonRΠκΩ}ΗηrΧ6tΏŒ_ώΖvSμΩ³—{.ϋϊ²4©Ÿ")sήΰώφ;δΟ Ρ;=cTF«…A¬gVOΟ¦KΆ‰¬^B¦L’mΫλλk/E—m§^BɎspˆΞΥμΝ{ή–χe•θ1yίvϊtσάn€n{y8ΎUθR™Ϋ6Ns―fF΄wμσ\θ­6q]UMŸˆΚ`UΨΞkΣѐμsMzΫΧλ.υv[ΪyTeY¦Ÿ.ΘEΪb±x}“t>Gέ_87?a4υΤώ&ΓΙΊά¬VTˆ‹:ζwR~γ]Έ•»Ί›εq»:Tωv¨ŒV7mΫQ.:6Hφ€·m—J»οαΔ—Νι»Ό±ύVΗβwφn˜[ΣλCh’\˜W(o·m”oΗίοΑΩωsnpZθΓΎPί2Ž“~Ψ”aΆ?Όν:ύD0ί όηDΜ%μ²~~―£­I«ΩΦ?δ|>wώRtΉ<Β&w‰.ΧͺίΠύ}M’ΜϊP;uΤ»"‡Ζ9=ε£ΒΎqVD€4υ]’'Έœ~OŸίγώ§κ#€s]e#Ή£ŠΥwΦ°˜L§s»<Ϊ₯Ÿ"~μΧQ+ΨΛ™oΆL€/N²g+G—ϊ•Oϋ<:m—|"€Ηφ)Σχ―‘’δ‘ λΠ mχc Π`oef(_`:ŽF£Ιdrυ¬{Σόπφ/Ÿ΅Ύσ_ϋΦώꐍi>m!ΜΣAβ I5@M`xΘέΏk Šg"€€«•εζVD€Ώ8Y. $tƒ`«·š_άΐΓTςΒ܁өΜ·h’ ΐϋΗKdz΅ 3’U(ŠΧί#aΤ–γX˜ πš]†ΊΩ€} ΌOA¬mYΜ οΎsBχ}N9€ΦΛ“ΪQgΜWΞvίX"†υ‚6[6ψ Ϋ^€ΊR>@Šζ)χ ›ωGνυυ»Πj$εx«,q'₯γQK―Ÿ'“ s‰D€ΰρvΙ\€.lηΤΥ{"@τ‘?)”)μ $΄ € ΊμιΨυ&=°V €³ŒD½iή΅ˆD¨ΡT€‡ŠΰŠZOn˜L&:ͺ¦ΝfεWKOO)0I­³μ€dή `~/N§D"Π«γΈύνŽπ§ΓnEHsΞ―ˆ%΅c 1_•Tz»ρτWVΉ1€@Ό±–½‡ŠuAT[Ν&τ\φ‹_mΎšΊίη]°½\‰!{vDά%萡p€Ŏ[ΛέΑΆτΌh=«>±{+'Ϊ‡Eσρ䎩0?ΨλλλΐΏ4ΩaT:"€²›α|>§ΐγεQ–%€τκPΩnL/ΐ'Ά+‚ιπϊΒ½^+ΈŠf³Yΰ’Qz M†£ΛX4ΟN"" zΨXΘMm9†JθΎ`a"ΐωΆίόM»{ΧΛΤ7w2bΡ¨ΰΩ€#(ιΑ\ŠC"g²|Όωή6<²AΨ3ΫΉΐ'±»ζ@ΩRη©5oχη…ωvD€Ύ?›²TΛ\Ώυ­β—D"*e·Άo³ιpˆ‚ίψz5™L‚Υ΄:†©θˆ'wδSΉεΣ  lΫ"‰–²Ά‹1θiUFFHη‘1 ό’ΔL:φPφe‘ΐοε‘oΆ€€Υ;λj–έl»N*zΰα©Χε‘!}; Φ^RN Ӂ‰D"a»² λYη¦jyΪο‰ˆ[ۘŸG"O"@ά{)Ӂ‰@φ λ^€‡]‘Us’zπ9?5 ~ϊ«­¦²­ρ /`zΈ—ˆH,XΧ΄·οi.Σ*"ΎΑ|ƒ·FVΣp‘e.€²‘IΝgOφFdΎ½D Œϊ'ϋ±@Ήn|όωΆmu'{m"€‚ΚVΣ:$:z2Œέ ι7’π[ˆΐ79¬ τϋξ:Η†Ϊe·Ÿ2@zυhΎY©£αp˜αΦΑΚ6ι7’’(Μ/ζkK A"ΐŠίτGueίJβwR#„±Ωlzzό);}‰ΏGΝp]ΠΤ6ΆHΉa©‰D€ΦZΆλΜZΙΊ‰N› ޟ£ΡΘϋ΅ϊ™D€ξ5Œš€ΠΗε‘ηX " ₯Ύ=ζ4#ΰx›ιŒ "@0ζKχ›?; ¦ sG/€Π3}¦IV½D"0ΈŒ:ζ1θδ΄tκΓξ;"ϊx|˜?;ΒlΫJ8³šς©`:€ωρ;υ)߈¬¦όd8€ͺrΫβ?Θv;ΰηΑ”l FέΪϊή7&šͺ>« δ‘*ΈYΝΞ°#€D+tιψšϊ§₯C³¨Y1‰p8τΈΛ’Ύ.€τ#ΐEfƒ=Bn9‘γF”νΚ±D"RvΪοfθXχζoΫ€,]ΊΕ""@² ώΝ'ˆ¦νE«o0δP°yΉΖt܈¬bn7U"‘ΓNΑνŸΗ›₯YfΎV 0σ-Dί~n ξ1”~°ZυEtψ\θΈY]Ήu€ˆN{ΗΠόΡ7)ΐ­O€Φ52Š ΄²Zc€\V[<ˆρˆn—€@@šΆ+·PοŠ&Aθi‡Κ1 έ¨jίd"@ΚΟΑ7ΖWh-0DΤN9Όιυ8b-«pa9T,«B‘DwtνxΪ KE άί­][ Pv1Β³Z6δ«?Α|‹"@ά)·9XY`,€€d*χα@ fΒήίΝ[Η8•%Αž#χQΉ `υšWχŒ ‡–g/@V΅"HΑqΧ!TΛΣ―Rό±ZΧ^k\‰x³Οf3ηǐβpύθήYRyqϊέώLӍΘφς½r€YN›M½uο8ξ–.ΗβŽϊq»:RD=J^?G O™Πσ%₯¨ΆZΩIΦι? @_°νΘd8HDύS§α@MI\ίHΊΒ; jβRšˆ|k’΄|QΟ ―g‰?ͺ   ›νγ,(»Ω^_νο#€ž;t΄[†‰X!ΤnˆPΉ©Φmˆ™Oι}ΩςζšΠfl;?)qΧ;Uv#rθP_4€€t‹Ÿ₯|€oΎ?μ:υw4AIρ@Uξͺ?Y²Κiۏm­ς‰ "iϋ°/ΫΞ) ZώώcŠ«eΦΫ’γ‘5?Aχ@ˆΘό5ςyΫQ%uŒšήτF? "€ϊΊ‘€¦Λ"9_tœφϋDκaχ‡#’Όφ@_Εk8vHτ™χΎΤφΝyš΅νQτ1p]εθ"W0WΕΑG x^24ςkσΣΎ8ϊ9–BβͺGDΕΟDΓ“%bΪiΞ Ρ€ΉtΪ<‘I *oDŠ«G"¦Ϊg Xo#,‘y7―·K_G‘ζΠ&"€2γρΨΌύsxξH|Sκ–IjΆΎΏϋ€Φ‘[G@š_䎽?D YΗ­Ώπi Ν*Μ@šίξ–]όyύηγ/λLN:@Π) $[$Έ₯€θί)‡Ο<N‰VΚ²tξ HgΛ°«gΏwD e‡νo x^Q§xθgtΠύέόX}κwπυiλέ²ω±ΤŸD€”Λ0Ε£ˆ…ιl6ΩάbKτS/ρFδΦΤηΎΏΈΎ }/?—ή+"ΈΊ‡πrŒΠα‡{l΅›šw«>>ή‡Ϋ"«ΣMΠ”ŒΗγdO½¦«½ig‰—‡s}K€;ν‹Zf xΌαε?@aΉeΫ "€ΥAI©τ"~Ν³½Iμ*r$Ftψγ—Eπ½:όΩRόhWκΘΆ @kh‹ŸΔ+=Ώ+‘§ζ€§ P]ΥiWώ‡]qβ<DՊVcΛƒ²­τš«οΝ‰/ώˆΧI—>G" Οώ΅>²ΰ?>ΨN ΞΆ z€γρ8΅ 0™Lzέ7’”ŒuΫ@΄:m6‡]ό‘AΗjŘ"@&e€Ύ§L&Ω'‘Υ`’ZK–ΡYRK†z!FΤϋΟώŸυvΥ½r©ŸCΕR?D€ΌΚΰαpHk<ǚ)ά™BO}>7’&'zάΜ+δβŸD"πΈ]·E―³†Ÿ~ώέϋΣ―4Ά£ω|~e&Αš!σ ²Κp”ίm[ίIšί|εΣlΓQn•U°Λγ|‘4νι"Ρ /ΤdΏh@x‡6‡jYw‰OiβΣ ίΓ튲ZŸψ†OI1ΚL§ΣΆΔυ;Y ύ™\-΅³HΌ_!ϋ†f³Ω₯N7 ΫowM=Ώ·«γΣ.ΕαiμPρΌΎPσo–u΅|ϊ7ΥS'B]½?έpO@σD`Ό4™LΪw‰]^ψSJiu}}ݞί.½ν­&©ω P3ΠgEQ΄cφΪͺο<$ζεΏiŠυzM[εœΟ}ηΏωβRQΆ†0:伐‘l·2d΅)ΝHg^o6š ­,K‹f0d΅Φ7ΝHgΨ rG` 7“ΙΔ|§Wš Ξ|Πx<¦Ή€L"ہ ˜O&YEΪ n0°(σ.€ιtJsωDΪ Ξp;0Άt0Ÿ0i.@΄ρxΜZ @>ήΩ Ή€|"mHgΎΜλΑ`@s’ΡdΕ|ΐl6£ΉΡΜ·c"0 ω F °ΕbA‰ΛΛKσYΝL‹# ™°ZˆY@x‡βΨΦΣι”φBšΝfΎΖδΏ³Δ,` ΌΧ•ω|>·ύ!γρΨφύ£( Ϊμο¦ξ6%ύl6Ϋl6oί§Σ©Υδ_":ϊ戝ΆΞΏ|vυμ]74>žΫ όξF£„₯ώw˜k »(]Lb‰@³±ξΈΊΊ*Λ’f’0\ ˆχ€Τ §ρ?΄6έ`08oψΕϊ?@>Ϊέ~yωδζκ™—7›Ν†φ€˜N§m%oυΞΏEλΝησσ¬s©ίVώ‹Ε’Ι 4rφ+²L endstream endobj 14 0 obj << /Type /XObject /Subtype /Image /Width 1025 /Height 205 /BitsPerComponent 8 /ColorSpace /DeviceGray /Length 8313 /Filter /FlateDecode >> stream xΪνw`ΕφΗηή››$τN€ˆ:†* DMˆ€ΤΠ_x>5<@ΚCŠΰ RD@šJWTš€K3Ho††–vχGyθžΩ½wΫlΉ{ΟχΟ»{gfgχ|vvζΜ9„ P( …B‘P(υ*ΣvμκΨ (Tΐ©DμΨ΅—Ήϋ:‡}Α\ρΑΨ(λͺςϋΫΉΗB0WŸœμ”uΥƒγ ΐήώ9 Έφ@!Ψώ(@Ϋ?…`ϋG l €ν€B°ύ#P€ΆP&FRUΚ—Ιaώ «k  ασ‰0ΜώMΐLN¦ZΙΔ!M‹˜ΧaOsͺ> ΓμίπΆ$υ(…@ ΨΩΏ?ΰ‘ŽΟ‰+ˆ@ ΨΨΏίΰΎ²7υC P μίp_ι !(Νφο§Έ―m±₯Νώύw0€@i²Η}_€@i°—1)€Rm~Ž;V €@ΙPΌGδi»δηΰξτF Pκή·λΗΝt!(φŸέ–XΗ&‹hϊάΉKVm;~SΚd–Y™S”©6>¬cμŸ,vjΥ){/]ά5ΎΌ‘Xξλ̐κG|vά‡}₯svg˜2β7ξ>}³ΥOμϋŸ/rfyCEΖ” «ΰ‘ ·yχ€7¬ ²Tw@ ¬ώWδΜΌ?}|½ΫRx *#Oˆ`:€lδ+ώ³,B-VηX~κ €πOϋo NρΔXχUνk‘ΛΉW€@i³²ž΄ΐ’ δΩ_…t4€@i²Π{π¬Λ ,ΌͺqJ‹ύ“(ϊΌ0‹€ο€Œ*Jƒύ“*τ‰…- Rν,έΨMJƒύ“‚τKΥi]R§ιλͺ@ ΤΫ?!ΤδΪba'Πξ@Jƒύ“‘πΤ>–iœ›λ©‰@ ΤΫ? »Θ?υH΅@ΖRΧφ€Ro„4δ-ώY•XA?Γ‹K F Pκퟐ&©O=[‹X€΅Ι)€@i°B ']{θτv>b}Υπϊ#(-φ``]·]›ZNέ›Θ5α¦9”&ϋ7JL@¨Y€ͺΚμŸΐk„@ ύ‹Ζ™¬±Tw¬?† δχΓ'3@uΜΦγύοώΚ3Ψ‚ »aφp› ΈΖKΓηpςBΪΓ©WSv/Χ­“-[‘νΊy9’·ΑΛ―›οζaΔ ^hY› €ύŠ»Ύ‚g}ϋJnŠ˜Ύ³v<~`„ˆ~΄$L9\οL(.³ψ‚Cvy$;MγΠ€Π]-ϊΘΈ¨@΄²—υ³Ά‰D "Όη>2Ψ‘₯­N°χ8Εa4κΟK“N@Ύ°‘rΛ? €*ΛrΌΤ”½ξY‹φ«2δδM83΅‚zDOOυ^rΊ†gΉκ2ώ+Vθω’V@0όβα}₯”‚s!ΡΪΪ”4άΰ@γ΅r“l‹e€π€,_5ύτ‚Δ#ΣσΈμΤ)ž\jΰμψ“TΡ+Λ>Ρσύ@Ϋƒ-r \λ0ή‘oΐ‘w4΄u1m—0νφ(ΙCτss6xξŒTbf_3mξψSΚ'οξR G‡ƒrRGŽp’ύΤχύ@S­€½ΰbGςŽ΄‡Υ§9Ώ ή*FΞD­SšŠlmiνpŒΙ‘ͺΖW¬ˆ9Ε:ήΕ‘mφΛ,xCαΐ±θ;:Ώ|X5΅~Ν›Θ;tΡN‘ώ œΦΖΐxWΉ)έHpi@ξ•΅Όι—IΩͺ’(ώX]>"Ώ–_ξΉJbξdέί¦ X p ŸΔ?4 fT]Εpύ.ΓPύ˜Ί|€ίΡ€ˆŸdΤQέ[™M/ͺM£š5\&‚ώTRξ•š€·τ?ΠΗΦΐ˜+œ¨©Β­€YπsU0Π;θ=嗝ϊ€ι‚‡χN2ώVΰSαΧ{€"εwΦέΧ½.!UYPMT~wΈΌΔ2egoCT ­κλΗ7TGg*Iњw =K%ίσΆU­UΦjΊΡJ£Η9ϋC6ΞόC^_Π Ψo ψΛΊ†"΄_ϊLk`l•ΘwήPpΒg ΛN‡ˆώθO/’+ϋ– ’χnluΘo+ιb…/Λ>τΪί Œ;1|,74iΣΐω8@OxT`^ΕLz~Ιmΐν>ά)‘S"ΊξD(*: Σ…¨O_…=β¦ΗνεΰomTθλM7:³ Ρ ™ R?P»šχΨΦώ£ι^ZΖΎŽrτ.―–VΐNb ΰΎ>§Δ«gΦ34{]‡Rƒ’_]J°[©ΏΤ{΄ΊέpεG½m.\ΰΏ]_‡Jθ9•…@$επΡAμ€&œς•όΗN²³Ž¨@uA_¨θ”’ηa}ΰR)₯ο†{œ™Lp΄Œ’ΏSήν €£T7νΤ£’ϊτJŒΓ|, –Λƒet’ιQθΠL¬€tDΥ;«1t‡?ͺYŠCι~A-@Ήtΐ― =P‘c7Τ¦φ_Œήΐ1D—jhgγͺ¦ ©GVτƒDp’’ †ΐ ΰ”SwP~Ν*»˜Š Σ[F(­­Ό ©Eˆ^Ψ§tZqΈ:6ί¦θDOΥ₯š·©j™ €|tδΪZ^ΖΔΰ¨ 8(tEtΐB{ϊE έςχ)ΐVΕ!$ΘōV\Ο―ψp x£M0‹Ί‡λ™j\j2œΛ©ν•7°–?E œ²Kλ€"p³Λι`š@ΉΑAw¨]άV €ΑυΤτ1ύDŽœιK4ήFξb-ᨀ-0Ν°ψ‡TΨΑL@=aΔ>]DΎηΦ•U pπTΠ«,­7‚mΜ!²©<•¦γœlΏcScˆ>aKΠ‘XžΦ­&Κ·z³IˆxW$χΤ«>C&Ο‘U pψžθ€ηΰtΌΖΐ«π{­…,¨ :τΆM ±6ƒφUΆ%θ]Εt«i ”ΕΈγΕβJK| Ί¨88(tθ¦?fhvζ FDŸ!*Tφ΅ιΔjXaΈ(χqΧ:ϊR›+άΖ Ψ[ΔόΧOJ½/ί‘ @-P υρ„Υΰ0»)ΐvΛΐ΅μΥΧΩ^3Ϋα”-ό’_UM8ί8υ@φKΔσ`\•όΈ‹ς(ύxΞϋDw„ƒi» Ν9ځβB€YVy%ΉΑrγM·επ±ύP’2†΅ϊUυUU)#πdοeή’Βέ'ύχ­ ƒƒB'€Zϊ ΉζψΕPyξI;¨I›ςŒΞ»Π5`žύPΡ¨U@α:`EcTι₯‘+.{Λέm%£Ίn ήϊk%‘ £]…q"‡JΰYuΌέΟz˜kP;ΘΔƒ+²QUUŒΟ_΄|ωš1υ›΅Œλ3tΔ{_ώ”’ΑωTš¬ePΆς΄δ–ΰ0ΐ|Λ:~έιθ ΥΊz΄\yΦc=KΩΓtύͺrP±GŸe€+ΙΙ‡SR₯έζλ\yυΑIφ¦RυqΔΤLnfεΦ~·Zσ ά‘@>’{Ατeu¬‹²Ο0€zν’»I―ψΫηg'€‰ώ³”Y°]ΠA`£u2A˜ zj~šq#€ϊ¦ {²ό·ΞΟ²φΗ<thl*ΙΫΏ§Dό1ENˆˆδ΄pΐD΅Ν›¨b6N+yhϋqς?λΐ1‡€[–Η°Έ][%½`΅  ϋF#”ͺ’y«₯ΝΐνρŠBζ…ύΙσΟ>Ο]¬iΣ“ —Φ‹ΕνZ(ΠP+zɍ>ŠΠO&ϊD˜ Ο—eΦ8όΏΊ3@F#]›±Έ]oKϊ2kΐX9ϊ*\n`,ν’<.σ΅Lω$άΈς3ϋksoQ€ΨyXˆ^E=πΛ(±ureš΅ΰbn&Eͺ©r?Xyτα· œ^0Ÿ±ίΠΆ@ώKΐe¦ˆή XT·š¨ŒΛΫΝ@ΦΖ>*SΌόSζvΰ "©¬ΐœ~ωXά.0―8R,.Œ0E_sFε?‘O.0Y[ϊ«ί(q‡_Τj―ηMΤ&SΦƒΛd²«ζYΙγZD„!L‘q¨±Ζpƒ•œ§iήΎ±²Š{9 8δD€AӘά.°@όžΨξOΐ ’μd‘^Ρ«€νΐ΅f|&ζ+€AwήπrpXGŒΐnΰίΔδ~U“τΣ €Ψ[ŠEG>―WEXϊI νbJςΆuŸΌ=€m=VAŽ –ΰ1/g'€ŽŒ8&^υuω%ŽΧ`ŠyΚλTe²1,­†1}5\†+3pψ#Δ ¬gηaρXM$ΏΧ΄`•ή±θrDeμΡΓ#둨 DίL8Wτ°|>•8ΑZœΕΕΎ(@+`ΐ­rs4[A„| ’sΎα‡€vvStΣν͏Œv?€J,΅£δ[A+ oeM€9’3φd“ΈŽ?fίι.rpψ‰€Ήμ]‡ςKμ’`Œ…ΦsDΟΞsƒu©†JΖqΓεp‚”βί‰œœzθdΕ$ΗφIσΦ tHkΎx€jγXΊθyQ]ͺ’5Δ@Ζέ Β<ͺΐ ΰfΓ0ˆύΞΪ₯όKκΈpΐ$M’‡υt¨dUGw@iUhœΰx ­‘ς ‹xLbkπ'3ξ9υΐσLs™ Τͺ2 €₯μλ(›EmΖΟλŸ€λmBG0±]Ϋ8ΐ{ΈΕΧ?ώΑq’ΰώπc³tŽ”Uyο³φ74 }†5NˆqΘΌ92C΅_(ˆ³Aƒqžp€INζ°‘―χE€ϋ’―ψIΐ `ˆ —X.±* @2›i€Φ9tΑ ˆΖίϊ;“JF{ΫΤΐ ΎLΈ­šΠ™"ˆ ά7@£;f©™0¨Φf‘₯κ€ΣΕ¦χwΔs’c}Ύΐy—Ρ€©Ό4nθˆEΥ δ Ψθ6Σ΄\H€…ځ\ρA©oLψψz€ΐxb8:°‹ΈδΞα'‰~θ½ί0Ken 0Ek‘%~”Ή?Θο@>βΧ}ά!œλπD€ΛœŒ…²τ'™ˆBΒFwD˜¦Q"‘ug85Yξ„ ΔœΔΠTήπΡg3ί `1tΜ₯TΏMCΐ„Pz„Ž€žΥχ[n€Yrο!ΐr-.e1—…Ž'6LψΘe’…d-½P‚JσΨWmFƒbf=•ΓxΠ‰P­ΚwD°E}Έφηo ‹ΫγΆ†WδΓ€V|'€ΤP3@–QQΧTΊ"TΈ vιFλ ²‘r‹A˜¦‘b 6«\΅wΌ™),μ:+η“PΨHB9L'¦ ΥΫ«U}Έ·ƒBΎ"ϊ 9Υθ£ω¦i‘ξͺrΟ(ό­HQžΔ€‘Ύ'”@ s@ΦRύ=BMυ0YdV΄Ξ [hl9f)μˆh–­/ (.©Ε%±‚Ζ»„³σ<v“P…Ϊv­bOΠK™3¬PŸ~JΖ LSΉ+βyφ”ω΅”\(§mΰ8 g6@_³@₯ΪΈΙUKiεoA‡π"Ί€¬‘Η‰C¦©ΞmρT›ϋΛ/#8!]΄ŒC‰m@ώƒƒςσš€’·¨>O­¦¬ξΪ7δob€Κ4!LΣKΩβπ|ύŒΌςώϋΌx   ˜βly–αNj Cθ^ΏR]S©. τ@ρ$*.£*€‘^ΝαΌhg;ι!|Ρ Χ½όϋhb'•όϊO1rΑΣ Η:ΑΦ‹vς݁έςΈ‚„(_(s*Ήπ+ τx#w~šΟp³y»“ιν―'J{ ΅·+=DL)}Cΰ{9\ζj σ?4ϋC@jdχ’FΛ{ψΈΫβa ͺΤ;›σ“SZ‹‡n«<`Ωο;Θ:ϋ»ιpρr©ΓL±μΡ2₯,ο‹rtδFn•Γ 7…Ύ7J¦ΟXΎ·n„ξ„`¦Ξœ/eςώ ζεώώ:ΜϋT§‘Λ―ψόΛΔn€ίΚ(l.`<ο2L˜‚†fnχ…£ΰψ\l(ΥFΖΠ₯ŒήM#TκΉ4NRN&oέ΄σ`ΚuιsWζ&φ@iρb2œ«DZu&ήη&LGάqΑ_.–&†€δΪ-Φ—{ΫψžuΚίk‹ΗW€ZE§p υNbC ’ΫΜl°}’“0ΓΌMΒž[Ϊ)Ÿ!H‰ ’½yfΌΧyΘ²}ΎζTΣ]†Κ·’™ωίν­G-€8±«=ε4€Τ ρ1ΫWEVbς΅ύDΜχγ– ±‰ΎδεωΩ7-–žuΚU§ίόSτyU,ε“ΝΖώW#6@°ΨΌΗ(b>HρΓ^n†ηΐτ> „ΕΌ2q[–θ‰7δ8~1©xήλ3δ9½ivb|ΧΆΝγΊΗώpέQΡ6χD°Uύ“,μa±+ΘT‘ V)”μ란_<‘ΌηΘ=©²6ε²){JΣ“6 ΐXyηy΄šιΦz5Ξ ¨(쟡Δ α[΄άΆςuΖ ‘{΅4z7€ΉΦdώIaΔΞ ? .ΉE@\csTί·5ĐΠωZžΆϋΟάΧΤί‘5Q:ΆΜθA_ςe·U@Hk•·.{¬άyLζ dP†ϊη­*@œvKένΨίJΧvYΉh›D¬R^ΥgΐAωQ[u©΅[₯ωiŠŸϊ¨ΠΨTε·c[¬ƒΨd5[]ΑJ $φœϋv{¬‚l0z€Έ†ήTaώΧ_s€^Κ“pFΡέπ¬m¨{›¬€κ”Σ=±HΔΜLE7ny€’u!%)]ΎφVΈτ'@ƒά±KnΛ½GGGΠ"k€ΐεΆnV!‘IwδήΈœ΅ #ςκBž\ „[g†η“3GƒΠψ½·VΖ]9ŸΤؘζXΐΰz.뀐bSdΝήϊθI₯%λ€ϋάz_ζfφšV.y“΄ΝΚίfZ²ΡYκΚa΅œF΅₯Q<_MκΗo(jτΓ5žΞ0(0$v‘ΔLnΞΆxqΜF€ρΤ€qΧήo΄δ 3{[‚Ο0š0…i Š„··$™ž¦Ιω}γ¬Α՜ΧM@7ΤΆπmλριioftyιΐ–ltΔ+σŽϋ˜φŸσ²χ«&ς4—Š>Χkpβ€€wΖΌr›*!ΪŸ2 €³"{ΜΪp ŽΰΞ?o°΅ΆΕ»Ό»κΘ=j©bη‰­Š ’ΜV>0Hύ—?49ΈbΣ6]βί­}³šyό€Ÿ‘O·ˆλ5(ρΝψΎqΝ*‡γƒ‡²†βωφŸY;… $νδ`φ HͺΎL[c‡ P€©Μχ‘P(Qbυv Hj Ύ’±CP¨@Ώ½ϋ… $[#ϊb‡ P€7ψφ';… $αΰSμ*ΤL6ΑA‘Iσ@> v @Κ ΆDΏ…‚B’ϊ€xe°CP¨@>6` P€:` ° v Hϊ€Ν…‚B"@( ™Ψ!(T )|ΤΔA‘H“|ϋ߁‚B’Ίΐ«Ψ!(TΙω+ίώΣrc P€N`0;… 9@ςΎ¬Ψ#(T© ,ΑA‘Hω.ΤΑA‘HIΐώΧc‡ P€jYOc P£d`[°GP¨ϋΐώ= ±GP¨ΐQ_˜©~φ e;Υςv 3œΘ¬€}…BΩME²wŊFωόg6LΓΎB‘l§χmϋ@B ϊηbK‘ωsgΓ°―P(ΫiΛ£HŸί‰α₯ό.3αe\μ*Κ†_™ψ½ŸηO|£O\ί³ρΠζΟ-ΒB‘직œ,Ν]…BΩO›eΩf#μ)Κ~*œ% ±§P(ͺΏ,ϋOŽB‘μ¨-rμiv eCΙ–a‹]ΨQ(”5@Ξψί‰ύ„BΩRk%Ν?γ€£PvUξW·ωΆŸ’°“P(«ς΄³^Ν·Ξμ ΚήrΤ™tLΔϊ=[;Ή±sP¨@P…žξNηYΚψ’Ψ+(T JΤiΣ5>>Ύk«jαΨ( …B‘P( …Cών›Υ endstream endobj 19 0 obj << /Length1 1517 /Length2 8718 /Length3 0 /Length 9730 /Filter /FlateDecode >> stream xڍ΄T”ν6LIw#5 ]3tJƒt# 0Τΐ04Hˆ€ έ!%%JƒtHI—€„” ϊΎηœχœ_λϋΦ¬υΜsν}ν}ο½οk?Μ ZΊά260+ˆ"ΜΑΝΛΘ©λιρ‚ ?Δ‡ΝΜ¬E8AώeΗf~»Ca.b`ΘΑ!`ΔƒMŒx ͺΓ\*N^~――°ΰDE„ΑΕς`O¨ @ sΈc3ΛΑ\}ΰP;{ΔΓ9z°Y³xEE…Ήώ„dœ!p¨5Ψ FΨCœN΄;taΦPΒηΏR°IΨ#b@ ——Ψٝ·“dηxAφˆ;ξ ±όn v†όέ63@Οκώ—Cf‹πΓ!€ƒΤββώβαbNθ*«4]!.‘Υώ"pώ€—‡χίιώŽώκς'lm sv»ψ@]μΆP'@SQαΰ€]l~ΑNξ°‡x°'κΆz ό) P”Ρ€:ό»?wk8ΤαΞγuϊέ#πwš‡1+ΈΨΘΑœ!.wμίυΙCαλ‡ΉϋΎ\G˜—‹ίΏ-ΤΕΖφw6@}¨›DYώoΞƒ ϋ?6; Dψω7ΔΫΪψϋ=WΘ'οoσC~0W€νC¨-δαΫΟμ  ΰΏ:ώaσςl Φ€Δκ‚ύŸμfˆν_ψαώαPo€ θA~ΌΠοίΏίΜfsqςωύΟ•Τ5εe8nωίNYY˜7ΐ[ΐΝΗΟδ‰ ώ;‹ϊwˆTv±…D*φaJ*Ψσo°ύ½μ€Ξ₯{Π-ΐφ™›‚AΦήg± ωΣψο,W™oEŠNNόlώ?~°3ΤΙηoΖƒn=; {Ψ—₯@ώZ\uˆ ΤΓω½ΚπΓ.ΘΈΨ9ύ{PwE¨7ΔF Š°Ά£ΏΜϊΏχΜ κΡ‚ΉCYάΌ Πψ–ΛΪραλαώ Ι?.ΘΓξόχ‰ .Φ0›ίKΖ'(Γα`lΠƒ’ψ~ΌΫhρώ#bΗ†x β_ψΟΧρ†XcΟΝΐ¬ΕCjC?^TΛP{qoŽ>dή4Hcηφ›ƒςΈΒΗHf―Κ Y’Iθ"\άP`;“ž§ΏυΫk©Η¨έzνc―3±ΩŠύuœΌo¬`O¦—‹†[OzΛΦΝy°#j r» s›‡ΎV>Ι…WΟ3οΊή²…α°™Mν­*!Uœ›²/άΡϊoLƒ‹§˜σ¬ήNS2’#Έi19ˆΌ ¦Ξ~MηŒέΣ«ΔsbμGσϊ―ςΕ\Nϋ.UθρΉwP1QS’žO°ψΙ~OQ‘˜υ+)z³ςt6ΔΞƒ΄2άKο?ΤΞπCΜDM™˜δŸ}6ΜΠZ 6μΖΕΚΪΤ΅tε¨uέß₯δ΄υςR|–―Ω\ΤcΎΥŽuη΄QχŒΕΠπρΫZ›C;3Ϊ;SέϋQ;fβ;hb»Ž…½NΆΨ$»$ξ@ͺŽNsΊ%α_ΰYΚTγΧν0Άο:ι½ε­ϋ©―\αΐ£ 9€€ £Q˜y ώ¬†Dΐ«J΄Υ Λ΄“έd€j^QΗνxλNΰGή°qšˆκ΅―—MƒSX‘² q½ƒuόc² ότHDαœ‚™2Υ±‹'* 6ο₯'Zχ‘GΤΓ·Ο;-ΚςmˆžΨqΦ^ν=mϊ.Ή‰6eό6Ψά”j@©qF©$hœί0ν+ψ^ή›ˆLN­0Rh24Κ’­ŒΣ%π}ϊόή›oθ)”φŽΆM&•YΜδ²7ƒ'UŠ]Rv:ιͺ₯uf―Gβ$t'΅Bš;ξ b+ Ÿž•­› Σt'Ξ™U?y‘6l<ΩiόΥgΠ»ΏκMmxΝη`y“ψΙ€Lβ†ΝΣ⌒4­―΄.ΪβWΪ»'ΐg3<·£ΡΉσCJτ K.[Ρζ»}^±“κM“ήΞΝΣδ[FοSI‡%χρxΰ›$qΆ~ ω!ις휝·βf»ΐͺ6{±qž‹b΄εjΥ)Tη,§_ζ‹,‹AQKŸύθšςn{/–ΣΎ­Om§žŸNΰST}‘LV(,M',³¨~­©…Ι#³χFifX“-P›p‘MΊεΠkdx¦0λQNρΰηVΧ…8YεEky‘Ίς›\aWνOΡ kd₯τ©i#“}š ΖΏ ¦ΜΏ{SŸ%ešΩl•΅vΗ1G‹Β„mΣWhΙΠz…ϋ*σ΅X0!cHξq){L©ΏώΔa_ιή[ τΛrω†Δα”MjWqξӊΜ,Α‘Ÿ9© „;™χ χ©δλ$ύDμTΘCz@€q·ˆΗϊ‹γŠ;5ηq-ΌI…±/MKωxωAΌTέοjΝu€RΈί&Ώ=ι*r#Ε0ΟbΈ~ƒΒΑkηuΎΝΧ‘',ΚΝiζo—._7­φ]ϋξNπμ°ΏnβYM?ϊ₯ΰbΔϊ‘Ώμ|W0BΡ:π©‹Cίϊ§iτkοq^ΌΈΡ7Γίωƒ¬r”ΝΒ³";}škΕb†ΏαΩΞδ•.ξF%nξ]—ΨΪν3·»ζ]K@ι5ιΨXύJc§k‡π½Κοt·t!™¬"ςχΗCΤλu|c£[tηŠ•A²Σ%°Ιθρ?HΜΡ^v¦ε γμΝέ³&ΗU†α#kϊM(¦½ξ―eτ__Œ‰ .eh9ΛLυsζ>Ÿτ]|—iMNNiΙo‡£ε‰ΝςΈΊz£s·"ήμ\G'³LPΝΌJπVτɩ̏5N©Ο²{LΫ‹ŸΩw₯jŊ٠X·ΈPΓ‘ξd{Ÿ•αRqΓ†υςΒ’§Ι Ah‘²\ŽoQKΆYGΛ3=Ž[ηŒV―J-%‰&Μ-ικΩ’+'YѝӯhqTY TΧ'ρX%sˆ²V΅4ΈΔΤΏϋEκς?γΣΑ—2ΝY•β>š6'7Œf Zψ~¨HΞγfŸΦύθ܈:ηδΪ)\Vβ5<§ρ^ίu€Ψ²²ςt=C]΄ύ›©Šέ+/†υTάαξ=Αƒz_’*ε;P­BΔ άΜ‘}y^­‰–χί JΡ0ή°nvYΪ›£wακιΕ*—»(½†8}kΜ'ϊ4’psޝkIΌτkf#8-θΧ쀡„Ξ5bŸν)ZΉoΛΦ―ςΧΉϊν%ΐ}Ν«GzΨΜ–,ϋ4~‘l”ͺ½bN;B™Ζή-~φ.‚Ž„šλvτ‘&¨Jr;Tƒμ³>V/}μͺ0AO¨&’Ρ„ί?ϊHBύΩβΪχυξϊ«ΖJ²A›ΌZ紐1~‹?,kIϋmΌ’¨ύgCρž©άϋ›²Ύ+±Ά3…ΎžCJͺ>Z ]°ΫgL8ƒ\€)!ΌβW₯ζΊLΤ}ώœ™„xν»γΰ"ΛΠύt=XΑœK€€d“ρ=ΌΏΪ―NΡ©o|/ͺP–33υμύšGΌ€θΌδͺ%ŒΚ­mR°φbί{=%Vχ•–­(ύl…T‡α1GΏ¬·ν3+…/ρλWι0,ŠBY5GGΥ? Q[L;».d§r–ΤA_(ρΥϋΑ輦~½¦mχ|²Ψ(Ή/¬Ώo½8݈QŽΔΪ„Bξ¬A0ž>Κϋ&c»Θ`Ή―ό£dξώω"niš†:§θύ<€0Q”Ο3Ά‰»•Ό’`ΌŽHΖJίΑu|˜f—E’…΅ρ―lΑΦKι&γ ϊ₯ΆΛUiπ;ˆwΚ&΅•TάΆ$γ(Y2ςύ†QΗ«Γƒ¬ŠΧ;zEv²υNh$sΡ’ϊ·Š»*URŒΔ&f]8σ]Gνί|νΜνH(9ΣG ›NτΝΫ^nS “ͺ ΊBI€Έ£Sv•gpΟ€"Ž„Mκ½Q¨† dίOΌ+LBLdMΧΑ+Ό›Πua‘ν[¦6ru}`»Tm“Λs4OΟCϋ;¬ElιδιqmƒBύP Ϋ†ξΰΠˆ'±½*†w™Ηζš<κε“ŸƒΚθ#β­AZfğŽ[ΘuΫο’oΎ'PFεJbμ—Κ|›όkΓJ6gl΄fVΔqαΩXφς‘©ψ%\l—Ύ5tkΎˆ•*ozΓQt\^)±U²5’–”MBrΔΊξUr)ζ–‰σd»ΠHέvˆ‰oFdιU`pŞ=υJ~ՁϋκfhΟΎf¨η™―GAœ΄ΥK¬¨^,¨w½uί’%|˜ή±Mƒ‡―ΌΏ‹NK;ίb!I„ŒΗ«»«kσθ‰rӈιΰ•%{AWεΩDε¬Ψ£Hςƒφd"3œ b³qEεχzŽήG™vVΩJ‰6˜ά6πJo<[κΌOΓτWΔΒ*+3Η‘³“ηυΊoK7XsSϋϊQ΅μY5NZ;jYaΎθ FϊŠ Κ«ικΎ(ρ₯ήPž¬dΕΣ{Ω₯SΖ4ΙUHR²Μύ.‹T°ΏΜ’†.Y—ώωFΛK>^),L%5Ζ “₯¬Είΐ˜ν2tw\I k@z+|TΨF³ϋέΦ`¨›AT8ώ„ͺj±’"[̝‚Γ“ŽƒQ]b.ΝggvέATCΩτQZŸ`΄ύτ·ibδε75:«)²N*;MV²›ώ²Κχ ¦ΒXΕiο^ρ’Β7Β²Y©~‚ϋNΫ4%“Ώ„/σF­‚τΎH§0jS'yΉŽ:d’%NΫςžΧ‰³ž•Œ?ͺ7/r#c;όxΥ)ψ¬'"Bh5Η‘¦·ΐ²Iεt*Χcτ£ƒ U[xŸk.η‡a"ΗΩ’§lη„”ΩΏ\±¦{·ζUkLόθ›Τη“:±“; Υ.'}βŠ`€Œ‘jrΩΑΥ6y.“v¨£5L‘ε•87ΠCήL,ˆΓ[έ³Œ@ŒjF$hυe8 Ηώσγ0ιαŒΠ1χΛΞoζjδβΦJ=%₯ΊRέΈS;ͺO2¬―Α›_Ύ”Ok oJ‚ƒ‰αf§₯©.mΪΚR;“Ά=«IλPv?d BΘzδmu8›¬ŸjAQξέΛ γEhο2¦Ζ£‚2Δ*«Τv{Μ²”ό+u»ͺkΒ›OΞϊΫ1Ϊ‹.†hλ‹7·«;ZϋʜΆρΟj©/Σ7ύη%'‘ΊBψδ—‘[G‚ƒΞv~δ LY#=;Β44τyτšr½<=pΪhDHDNωŠ-Yζ”]NQλΤέ‘a‘'"‰7]fίo₯δβpΓϊεlun|y‰―‡φPiͺξ¬Ξ¦ ύΆ“ΖŠ―±V$α₯€(^Ÿ<,z‹Q)i@+”ό…νΞέOQίW΅Oλ+T Ύ“bΜ6ΨτŸ~―,t’dˆΒjjω]υςΤ=€ο •ŸΆ »Χc•]†αj ”ίγ1?Š^¦›`Ό•?έ‹½«‰Ύλa*+™€WΑ9ι Υ §4…Ά―„¬(–bτcSǟPκΫλγKθ> “‹Ψ½QFf:{Ζ•|?¬nιπl·(‹bPŠέ:aρΡςΟ”Œ{α5UΩΑO1NΘ<‹Έ(°3Ί1ς†iOlΥ¦P]©±DνΗuσl˜ΚΪΜaž„a„ ’ŠgώβσjS'~Ϋͺ„σ³Οh;¨dq=#οNΡΣJΘh…r#;Ρ€¬²^Σ›ι)Οβ9Θχ†ΘΝυθΆΆ7Ά‘t₯Q—ξι{ΌΩϋς1w,kYςyRjužφAͺό9kΨϋˆŒ·‹cm„λsrλ‹(υM9ΉΣόΧο“Ξ@_αΤΊrο·Pžv}ρΔ½6™Ίž2—`ΰ.βeͺTKZV'"Rζ–Ό#ŸNR>@β ·y’E¬"zΐ’‹ϊκI `¬Χ.f₯њ£U–™³€’lQΡΐΕ&L=`LΗΘ#έJ ΅)€3cΫZΚΒNέQΠΦ["ύFS’Sχς‹mnκ Š•ΟΊ$§nΎ6μDpεζV€mΨ9˜‘(ˆ4I‘¦ε|΅Gh-]Ρ‹ ΆGΧwψψ ¨%Wˆ!,6 Iτ†ΏΈ\Ρ ’.΄1mx†jΠΜ…ςQ.* I“όΧkνύΧF–%Ϋ–*²U«tφΕSnΗ;$"Έ β<'cŸ©Ϋ9zΪΥΤΟ+>ngZ΅'©«R₯‚ύ¬`ͺ™Κ[.ϊ‘°fj―i;Ϊώ„Ξιhx67Z)’£rΦ«R‚-dέ«˜ςΜsΣŒ3茞]T΄‰ΕŠqƒΕΎCΝϋwwI‚U%¬"^κΜœ bžo°#CQ›'ΛKΧΑEήδβcFΈ1–uΞΈη5£}θ“ΓΓRF+χU¨›$WΟcc—AΪξ’Lώ­7ρaΆΒ»Ύ―°λ"²€§Lϋ:—Mχ9 w‹WͺrXQΧUΔ¬Κψ‹Blω°5ΔψεδβΔ5 =%Ιϋλd€DŸΆO…ƒ&)ϋύH―Τ‡Z#±¨mΉxΌ- €,XΘyn2ΌΚΙ“oρΚj·„ ρ«<Žw—§1``ζσd§υΔM•ί·r ‰LΟύσΕζςaί™Σp‰+œ€γ”ώρwKH’ΓpτeΚ†`Ζ,«Dcϊ`“ΟΞxΩLήσ}€™A eΧΜ—)ΙwK Τl’³@z­Ω„&ZcΉΎΜHΝμ]έΰ ¬=PέNΠ48–βCΘθLΗτMέΠΑΡδsξαΧ!˜S}δsܝ™ΝΚrοcύxϋ[Βd"ΣVL^οo ζfbO ΧΗVhHρΤ¬v-֏G˜Ž{<³γzςDM­O9,όω‰ϋeͺχ<9ΐŸ}bpδlP±ƒ1φϊUΥ“πNΡ|4™g*ŸwI 7:+fηy¦PLWφuϊ&Ia Kn·ƒΚN₯jν:2DνU€ΫSϋΦ`ε°ώςlyՐ $ΌoqδAC―nΉi΅¬χT9Ή_ι±@nΏΕ6*t;Ό{kΚϋφΕΆΐk\»ΪB‡Βν1 ΊY%K¨³)<Fζ:hΖJpΔ‹qb+<Ϋ$.«25y(ouB\ΞΉ’[P»h6p?ί’/©ήvYUψ>-Γ0HΡβνyaδ{ζGοΌ²,εΣ©ŠrΔ΄w`JηS€9π‰vš’n0ΧVσQnΤ vλ•@γsζq«Η£z }{–5‚:βM…ˆΕδ’ΪZ[+ QfXά~έ΅ΟΟωŠT›x^)υι»―΅ΔJ˜TXΉΘmF>γ„ΩJ/rŽ/­4†φ" ?1³E›ΊΦ.ΠΫμΒ£W+£PAγeΰŠPE: \›\–#žG%Fαr’°[άQ±Ξ›ωi«uε IΒωOοV ½όοx«ση8œζχ\ ξΝ]αDάs9»-ό-€ Ϊ",c­G­EEθ=(οκ{r5½sE3&cM#ƒΥκΌϊ―δƒΓ{μ΄ νΣν΄9ϊΥdWjηzΰh‹` ‡ͺηv:`›7«4οCT1d,αfΈB Dκκh|γı勬˾˜™Kb €`³ΑRυϋάΝέ7Ρ*σBnqς/„-Ό}φΝ6“wPφB6†uA­\ΣtΤ¦Θ WnήS… Œ§kσ8ύRΉl‡‚YΓ™ύNͺž“½KgΚσΐΗΉ‘‹f»ΏD:j₯ΓG h[)ΤΦ'ΈG{\ιjxΝψB!Ze•?o܈;ΏτΓE—φZοΑF]h ‹8©ZΦC]D ]±iΈ†?ˆ2c›—“,š œΑυΥ§d Ω|­ΧŒM$ίZ… %d°ί°υb 0ƒiAορX–χΌΤ|ω…m~<‹ŒA ¬ασ¦SyIΒεγΤυK“Q†,ηΚωΣώ’‡t­z†ψκΘ׏»±f-ιηaυ.@αœz³5ίPω’pMΊ€zdΙƒΰ)‰$3L‘PU;Ν¨Δ κ) σkqֈo²RSŽϋT`ϋΈK&¦iήpd~wΝΑˆš.Yͺηϋ}”(€‘”,!•CžiΥM;φ§χΟ½o:‰djγq/J¬ =+ΏeDF§J¨‡„᳦ΙJœ#ώΎ#5Ο?Β³Λ‡λύFŒ:cPΦb71mˆ«εΙ¬¨p₯z3?‡Τσκ,χΧξm³Ύ†΄‰H=kaJŒ1wa»WgΉ Šƒ"1>l%rW²wPS_QΒό8ΔΦH˜ԍΪeUεƒΒπ\IΦίλο~Zn X–Ανh¦δb~·FwωαU‘΄Ή&q›κHξSVΈω漎¨Χ5©ιPRœ}υ“ΝΪAΩ―΄cgζφ²Ρ]½βυ§ΎΦΨ{ωi= ]y‹Ε鍋@‡S`TiO`ΠKΚΦ†SΕδ M @ΜΐBχD ½έ‡DτG y„a,d],–CϊΞΈ:Vg_ ΐ·°*§*)ˆ©uth#7:’Ιg­ξ¦ΑΧCχ‡xΒ΅§dž~‘φ¦§ΊtοclοX7{ΕV7Ψ¦n{»„…ŸΖ›rΔv«Š'v?κ`Ω”γ,CzΩIB1ZΉ³wξώ Μ2ΐMξτΒ86dκ_g^ωΉΚ—ž‚Qόǜsρ_EΟ}‘24pI χ(f²™• i*?‰TλΞ9Ύ`)Z4ŠΖj·ί>n/0ώόhT9hq†>­ΧX©]šΣ`›4¬œ<1 Ÿ‰₯|xΏύe™EŸ¦Αx‘Β1³³  NR;kΨτύ›Β£οιι${Š,ω‘^ο\ χ†9:mγ»δ…!΅{Χyβωι(ϋο¨―Un†R©$tUIζΒ’TΕŒ •xΚΩΑš¦Dθ‹ρ'6YΨ?Μδ2i :Τ+υ>ŸŒι57|ϊb΅($\43EJΙαYκŒϋ2€h‰3Λ(ΉCŠ@“Xέρύδ· §n&lRB?S{)RΦuL³χr9”9&0ή‚Άup tWD½p<Κ‘NˆNJ(rˆ$‘°°Ό–υbyqk‰ZAh~r«&鎹gηUδ°βΫ {‘smxhΪwfη&!λ―ϋ>ΰgMΨιϋ0kWΧΑ£:5€·ΘO–SΓON7![ΑT vΆ’§nY^e³jΣH^nαsΗh£x²˜ϊ˜κΏώ’Œ}”n΅[Υξκ0χ£ξύ-₯’©uiY- μ5»‘g/³%^¨Τ”§ςWJ©FHvfτœ:Mh€PςR9!χ½vI~ΏC°ˆ€Ο GŒ–Γt9%Εg•R* ͺ“τC_kiJj`qι»)όuw«~«―w-κΘ™]*ΜΑ$jξΰ|&mέ6-­φDςLΨPKg΅―ϊXΉ"Vb¨?‡ΥŒΒϊΏΞθζ1FΣΘ4Εq:^γ)ψΑWxVV lΗ‘@– Uζ’δ%iΡύΡ¬σqr#Τtη‹mŽφNλρK]VŒhΦƒ>†κ 9*QO'η9‚o5&ΥΥaίDίF8ηr±΅₯έΠ»$ Ÿν³δdξ±H Σ>jK—όBΖ­•">JpψQl’Η½ωΐ0υλ>{Ρ‰Η±¨Ώ“―λ› Ώ Χ*1‘Φ"*Jθg‘aZ‘‘© ύuή.;4'ό2ω`όέG~ΆΙ|‚γ•B!aώƒ‚‘ΝŸ}eI~’t—‘ήFc˜v—& %ΊςŸ‚ Ž‘3(jP…ήUhm“?ϋςΥdYsotΰˆΏNlhξhVΞQ―VτΕ¬;­R©ι\­ά]G„τωLd½‰¦%LΘ·7OMi³Ÿξ©SQ¬ œΞζμ.GdE‰hxΟΈ·Σ«ε4½Θ“±XΈ$Bυ3ζρ„·F`•‹δž’^Obž“ &χσžΟη_Σ(w§ŒkYM‚υ¬HL£Ϊψ–Βΐ5\Χ>Β_ΙC9_sͺΧ“‰ΐO•ζ> iΤΆψΞ αW5₯RMaΑ b甆S‘y[>{ ί ό%n[ΠΒ·H°žθΉpέ8E§”/΅”#1h5IRλ… ¦B :ZσΎξTbq†~{{΄-L(ͺΫY»αψ©CΠΔ%Ήg—ŒΓΟ`#e―AΎΈZΩK=Vu{s{‘Όz>B9I4ˆ<9\’'ͺτΉΥ»b97'ƒ0]ΗΡhR $ϊq9Z85€TΰŠ??š‡ƒ ±}0Ν« γUj¦—ηΡώV%TΈείεX vΟYι(άε(έ2δšΕiΐŸΈ—»Ιίρ‰-―ͺπ Ÿ ±)‘h:›Έ›Ω[B;"΄'z‘NϋυΧρ’c,ρ{ς} endstream endobj 21 0 obj << /Length1 1520 /Length2 45158 /Length3 0 /Length 45949 /Filter /FlateDecode >> stream xڜ·pe]·5ΫvNlΫΆmΫΆŽ­NΊcΫΆ­Žtl;?Ο½χΕύήͺΏΎ―NS{Μ5ζΨsΞ5V½ΙˆUθ„LŒΝΔμ]ι˜θΉͺfš^ΞfŠFfΆΆFtΒΆ¦fz6F&22U+W[³H!S7sv±r°ηώYΔΩΜΘυODΤΘυ―73€œ‘€‘ΐΔΔΝΜΖΝΔ `fdβό’ƒ37ΐΪΘΔΖ†LΔΑΡΛΩΚΒ•πΛ?\FvΊΏζΞUψ«€™§«™ύ_χu{„ι-[ϊ/=+Z€4½=@ήΑγOδ2sΩ(μΖf–FΆζσΏ5ΥTΔ”UώMCBYAMQ…Š neζjodgζb0±4r62qύΣ0ΐΓμOF¦¦f¦έ_Θ jiφηkdoω§–Ώ;€c£γόΣΙ_MώayXΉZώšΆΈƒ³…€ΥΥ‘›ΑόOΔό―ύŸͺ?™fφfΞgόΡύ' # ΣHt1§·7seψ“L&fo*β`ggfοκΓΔ0΅2qύΣ¨…•= Γ_Rͺ^Žf&€©™ωa9#Wg+O€#=##€ρ―Ο?τώΤmκ`oλυOΊόŸ)4€₯$Deiώ“]ώΑvπψΠq²2θXΈώμ<;;€‰™‰ΰχﲊFV]γ?³₯μΝώ˜εΏΚ7usόŸάΛiΚΏFψw)yW+“?ƒύΏΆΞ―]tΩ³cώΫ%L/Fωί-ˆ»ΩΪώ=dΚ0έƒldgeλυŸθ›©aφχ8(£ŠΚΏN_ΚΥΘΦΚDΘήΒφ!+q+O3SE+WK€Ή‘νŸξώŽ«Ω›š9ΫZΩ›):ΈXύuάtLŒŒkMΥΚΔΖήΜΕΐφίKF.&ά ΰ`f›Ω›ώ―‚ΔμML­μ-Μ,cδμlδΓψΗ³Μll&€ΥeΟ?;ψ§z{Χ?)G7WΏΏvζ/«°sΔώ ύ88 rDΦώ8Y *@\˜F@uΒ`ϊ/ ΐ`φ/πOͺΕΏ@6ƒΥΏ@Nƒν?αŸsΘ`/π”ΓΏ@fƒγΏ@VƒσΏΐ?Κ.Ω Γ›™‰›³σŸ‘ώ}\ώ τ°ΉΥŸ]43σ43Y]r0α ΅ν|Βυ ϋ=ΝΜ ?|u₯q †ν~€,ΒϋrΗR½"Τ»xύ²ƒrΖ\ϊΥb£‰Αh£Ών›͍±ΪΨ nnΌOλ;‹Βχzζ"λΖ|(ΣΣ£mΘbojqΥͺ<κq/Έ@…IλtσϋWφ‰/ykυS•ε$Eσ9=Θtl4$'' Hh—PvW΅pΠΒb·©Λ­ΰτE2ΎψΙKζΠ3ΫήπZ9-°Υn&ξι)Ό£Μ}φ)Ζq?;ŽŒα,`μ _ ―秨 '4sμ<),£Ξˆz§-ݎŒ‚D”EΊξύo§%τΎΠ+ήvœ9ίΙa „¨β#)\p]Ρ­Š“uen8s΅‰μΫ’1SyDoGin‘ŒΎaφL«_φlŒ¬)Χω:,™.Ÿ)Տμ―ς'‡Ϊ_cY:ŸώfοFόMψu1џ‘MμmΣϊP›γŽ•”Ÿ– ψπ1‹5*jΌœœ{ΰX;θ@”Ώk&xC7q.βε^ΛΖ~vΉ@§ΥK―‘e¬€w^ eνς―kΫlY€‡1ΕΝHŠ9›yUΊρo˜―uŸ)ϟΕοR€ώb9ͺπUρ.$J₯'•ty'`Hή{8RΫΕ<ƒέΞA_ό}’‰ƒ5Ζ.hζ>ΡO0)\RgζΧΘυL§]R)`³Η cNοu’AΊ`Ζo;Cc₯hKςΉvj«—*€ƒ² 5=ϋ”ΏΰΈΎδΞs΄kCηo¨όC'Mž„8³f1cg—Ρηα!I―Vφg°1cjΕͺ<„jš~*Θ2V hςΑφθHͺŽ |μR£ΗΫ—ΖD[B@ϊŸ·Οψx°τ¦yH΄ž‘₯QσŸ©ωBγ΅8[πΖh₯Γ‡76;툳JKε$΄6«#„&Ξίί N4N‰‘­SδGΚZ'm"§…₯CPΘΐΨDˆτξlZ‹¨Ε’‰τ4qš›Ο‘y:ˆqκ‚ΒK4ΘΧ‘ΡΉ¨a4–}Ψ„kΎΧ»Κ2κ$&[:ž5©>·°Υ¦ω(ΡχΨΔ“‹η)ώΞ(KΞ€Ω ΈΒDt²ήΗ\Τ‰ώ•4¦jΥ4pJηψHω ZΓλζΩήύ ΤJ’ynu¨ΉN²lπςQDŸ•›@=w΅ΤΉςΜΥBϊyΝή₯ξ͚V$œr€CΈ²ΔΤ.*–Έ¬z©hUaCš{₯ο\bPυΞ€•εΑgεxŸΌH:^#ΌΠξfΡ‰ΤΑP/7ΪΨqβoTπu€Ύ‡ΪέdΦoΟNfTιω[6e­“ώΒ"%ύXΣ:wΗ‡ͺΪ‡4³1ΒΛ&cρOK΅Q‰έΪ,άWήδυی€τfWo*(Ό0EvΚˆoۍ‘’z1 φ±)yα)oœj‡ΨΛ(Β6\eT΅ZUϋW‚φΧQx‚1°>₯V₯qboωπΫ{•˜ΛfeΜ}ήβŽξ7Γ—8οόl €c€R―]c"ωΝLdˆ’1xσχΟ b­ΡΏ•Ft1Χξ<%„b―πbΗha±”ˆ€ z’‡DoΎΨ{Q49ξΚΧρœš–βρQΛ»Enγ XU1†„1ψxž™,a­„γZHΝϋ1$ύ0›*Ζ샑L_WζŠOo3˜yœ \!€ΰHA`&ŠΜxYλFΎƒ?Ξ:‰ˆlUt=ωΈ˜!»Δk¨tJ_~RϋΪ‡ΨΟμΤ«C7³ϊtΘRPΊx­Ÿbε‹w0ϋŠLΚzΈLΎ„Φ ΄NcΏ²€τ%θΎωvΤh"η7GŸdĚӐξώ¨0PΤ€}ζw›0Ϊ£뎚pF…_°΄LΓΙT 䳘£'?δΐΌS+xΓužnΣ―RRΏξ§Nϋ|š§Ι›‰όΆgV2έϊ§—μτ—#ι΅―ˆhίh”ΏεWW sTtΑδΫJ»Ό1ϋ6wrηΧχD6ŒO&˜¬Q!λmφ³*] LΜapίΥYyσώ\ŒOz΄²_=‚΄©¬t±λœoN νΓSUcο@EΈ#‘œc™©avTs5 9±θzΊαΉ‡ώLΉ,ύŽΔlu[L!p[r£φ%lXvαI:€˜ά6 xϋΜEΔψήŸΪώ°έΙψA‘O³Θ2vιΕFμΔq›&•'™κŠΌ2Βj,ΉΦ{Έ(aΘWσ†% άφΥ ­0Žo0&X¦žUMHŠ»Β4­ϊD{,jzφœ/l‘Kn*G”…‹ηƒ[8`olΪ20΄œ§›Η³±O{ew6:mrΟη(ρκ‚ή]Ÿgq΅Ω©X»X{ Q~ΈΒ»’E²XέF€ƒΜ–ž%:TΉωΤΆ˜ύΐh‡Ί`–!YG_8ΏK±zn|#£2$ΥβΑ6Μ7yw9―Ώθ,‘<;uˆŠ”Τν…θ?‘BL…Έί{D•κr|―v9…'>iΖ8σΦWoθ/Χ,φnλ<`©ν'ΠlψE4P°ΧΞ—ήΦοJR£―ΐ8OΔW,ΌUXpsˆΛŽή5%«_D˜Θ,7ΰdUš9ΠΊͺ₯‘=:,,Ÿ@ Ζ³NΝΞίβ€±ΛTυα»ωj+CYˆŽδΕί1³ L~ a".΄$«Ύ'&r>΅8½9ŒŠ9˜> άΘGŠϊΘcyγxζƚ]ƒ‚o°΅ζތ›@“ ν"6Χ 5Ά‚&=%o΄·σ+Ψ•ΣΈc|Ύή:-=–„yrξΪ#[,0μ^―;Wϊ.猏ΛΰξxO^•’€fƒΑ%²U,¨φE‚H—$9cδΕ)kNg[ͺΆ`7aΖΝLWŠΤβΉμθ€šΑ5rΧΟ€='·Εη26—ΟτY‹Ÿsί*”hρό»VzH,"½€ΓϊvΗΩh³Qύ.€ =ϋΛκiЩŐ$ΕάzNlΠ ;b;Έά3,·ϊ9δOΪΒβθC1[ κΖUαΙ6τδΊ­Ύ1¬ άvβu£‘t`JQ·DΞh f’Γy žΜ±ͺa±οή ³k5l¬ΉΦŽiΖtn`ŽΒ΅Œ•«e)P}œΜ–Θ|Y飴nmuΌŒ–Λ'Ε‡†‹±]~.Ο[Sf+ΗŽF«6οθ€ϋ2:αgΙΉVq<:=•­“λ'fyωψ‚BΠxxΎ{/ γQρΩΖɚχz8na= “ΫŸyπX3Ώv=Ώ3zχ¬5ϋMΚ³Ίgz-@Z1άGX(n•ιX₯O«9«ΑΠ/ΤήJ£Ž{§ΨΠγΫΜ₯{>%p™λ’φr@ο8ΎŽΈ5"Ή"<.PB$΄Šωj›dŒσ—/ίv·,Ρ5B’|pΘΎe’L ΙΙόΝR ρΚf4Χ+x…ϊ˜OŠ*–y_/!zηNπͺ·BΏ³vΣΩΩeΔdξc‡$dΏe 8ς­yœ—7#ςp6Wͺ93,γo{ƒπn£ 4―φ*xύ΅ΕΈϊώΪV@ΠiFpΠ·_^Ρ3a©+N›Ζ΄C~Lr./d €ό£.G²γ¦Z w₯q ŒgMΒ₯j™ΑΔ\f^F$]Ίφ—a8ܘάΑG1P›Αϋ·§ϊr¦dΔ’Η&ώG[Ϊ9-°μ8Ω;‘~‘>ρρPΚή8Ρμ/€βgGη1Π†©ξ“HnίΒRΪ7Ό=<ώ’X[ύΓ,ˆ«A7yΌφΒΈtΘS­ΆύΓ^>0!SbވuΨΘΚ³ε&V’9΅½g©Œ2h>¨ΐΡΕͺ~#8kοER R‰²Η­H ~ΡhΨgŒ<ΰΡ)ρ3~ρpbώbΝGΩv»ΧηuΘΝ;₯₯ŒT[+gΦ™‹fθΚΎM€;§ ² n)†άΔή'=ͺf σ©*-–<EŒaqK:Δ|ΜκΙΉvšŸπLgGpΠ7:!ΌΑΊ'τsφ¨¨nΈoQή„³h ΟΨ³ŒNκΕίςτ™γJw7`υšb²˜ Αx2 ―·c$5―φ―~>”4ΰ',Pύζ<©qDv¨ΠΧ݌v°iΥΨ+άrΣΌά[€L=Ra"Ό9IG±ίZά'Μ)ΧΎτη²ιξρkΨ6Π(³ΞΆ_ίLžΨ2³ƒΞ/¦†&SRGθ΅ ¦ §D»Ž« Žƒ¦'›η4q†θΰIdjρn$βVY/Έ |μιΈΘŒ•ςΑYξλδG‡(kaΎ§¬b”Ι+υΣ²:lp΅$¬ζo–šΫ|‡ίΈcb―Υg5š^m*―?ζǞ44Yuu ω‘e+(”§}σ…²8+η€σ>!Ύϋ™΅ nΑ ;goωΝF΅~²Š>ύUαν™σ 8ίMˆ3φ°Œj͏ΎτŒŠύRV?IΙΰR±».1–9ϐ‘ˆύB”Χ3Κ[βA?$ w-«5Ξξ7/Cζ… X'‚«/…\ŽPcE#“HςŽχγbu3›φς#?N:W¨Ά~Δ κœΰβσπς#xg>.HgwR;5q†ή‰§) vΦ`>Vԟύ(rU29sεΝT+_‹ηlω0΅gΙΔ:π#8H ‘TΛ·γeŸŽμΪ¦ςp" fœsu”Βͺ–r}Ϊ¬‘/lΚ-§3<›:b΅ ’ιΒΏFζš l,$ώΨYV°’ψ¦‘œ5o2~±‘ΌQ6‰νxΤ&e±Š_;‡¦˜ΔZδ3Ωνcή4FHoΠ<1sˆb%ͺ‘4ψσȍΖjqμ K†—ρ2ξΛ ·85άΛLkyΚg»―ύΞE*‘eΝ™ΕEloΦ‚+e{Ή­¬sα'o•ΟD,hκ`یΦΫα!ΚΌωωϊŸfIMί Οw©F8ΗΥW8uΑχIΓΪskΦ&N ‰γeγwϋN ‰ˆ™eγ…!ΞNοˆ :Όγ=³¦ ‹ω•ξ'ξ*%Ώ §ŠίNρ…Ϋ””Η³γI―^ιΛuf \iͺεή ςGΑ¨ηy=7²ƒήεg^R" σcή ]bVc£_—D³$Φ—,ε‘Ψ`γΝΓ¬+FEΣ·I6LSy qbίΉ8«Ψΐš™» £Yτ v·23ΝαΆ”s:f!_β-,*InΒ/¨šqx³`vχεΑUπνο]ΐ™€ΠT―L‘Ε)κθχ~MX¦μps₯ΰυŠ?E½†Ξ?n%kfŽ8‘ν(~2¦ο‚ˆγD-ϊ6H%)Ϊ\(3Μ««Ex0€ΤGξϋ£uBJœt%έ=’Kϊ:$Sœ!ΩNλθ»™σOr*1Ι‰yΦΆ GΖ΄#ά•I"ήΨΰ_i΅qηίπnΛ-4_c9»’§m$φPz―ζy½ŽώΠβε.Ώ3g£"S΄η,Z ΄°ήΧXŽT{˜ζq’‘<τNΠΖ™’Οh£/Ÿή²―`‚€; ·=‹tƒ+¦μγK’¨Ο^ΉϊΪΆΘΟŸ)”)ΎΘB)Ea{?Ο2Μ?Š+ΞߚΗΪ“g€g‘ΚjπίsvυιyΝ½W0τa¦M1ŽΥd$ΘI„ΊΦ™QRέ{Šv –Ž{ΆxDκ!δ”ΈWγΚ’6ΟyͺJΖ'Φpά2rςΑ¬Cid<ωeΖv Λ—δ[ι'8Aή#pjfgθμυI°‘FSj2ϋ”ι-Χ'RCJ(F³ι­Ηνο»™GNpτΎΎ«@8ρχ¬ΘHs=λjξ”Μ+mδ~)VΊb[™_ΕΫγuήνNGέ“ μb*ΫέK}ΔX!τ[,ϋWXHύ4θΨ°ζJξ’©Ρ»Έ*{˜)…ΦΩ#+/HFƒ?ΡΕ•?΅χνkΙ$Zœ¦<Ω(le©BzΔ𳜠LN'*Ow―cΆώN}XυϋˆR~„P…ﱐάV!4ΠΚε!˜@ 睩ώIΠ2X‹‘ αpq8d„1φ3¬>7·7Ό‘ά]Έ8Ÿ†ήžZA΅ Σ%‡0·K=ΤGΰT|”|zaίOZjΡabξŽΙeΙβo6εU(ζ½-Ώ1*οKπh—žHš©8YͺπN?ϋχ9Š„ιYd+(0+Z–½="ωvͺ΄‘\Yvι₯²ΓƒξˆwN[yΫ~Α9™T’΅‘Β…Τ•$ΑυdlΧQ¨I/JLxP™JΙ\ ΪFmτQO€·'δλυ°±|1π‘%iΕ$œίΤΉœθt+Zώΰw¬ΕΔz‘–=άσΝ’zEνυ‰„?!ιΪ}."μ—όο\ά­‘¦Ϋ]bT§ΝMw½*Μυ3ͺ“«ΎEύη(σˆ^Μϊ±E¬‘Hs‹.ν8(ΠB Aπ4ƒ¨€„Η"ΖρZ|q횡Β{±ͺΝ3a Θ›A•>Ε†‚3ΊΕ δ…™!Ξξ­3χ±%IRœ$QB™ βδχGˆ°7όι HzΨ_ιΤ ’ϋœι~z#Ά?νeΦ†»0Šl͟‰,PbVŒ’ο}Ӝ«œCnΏ€NtBφh‡‚Cc\†fd0²ΰ‘ς#πVΌπ―L8p°™,±Ο²›w­¨ ­οΔζahZξρ|―?EšXy>KŽΠ"»-‰ή@DB 6Σ―™jΒ Αη‘ŠHΨA‘œ‚ϊνω͜zΫ{|w(YU5\/X’•J«jσX6{ξfάoL‘°M'Υΰ^ΐιΑΪ€%§¨ΪZ{oϊβ‹έBΐšνŸ·s}Ψw*gΣwΖΞLG};Χΰ„.εŠΦzdMHΎ€-™Σΰκ‘»³ΖS;~Š) Η…­7Š½ΏqQϊ” ώ%‹FͺεŠ₯N’ΕŠΨdp :fm>/EτΥ¨l}'κ6>ρΘ€…ͺΣu΅ˆ;Κ[ΓJ(‘# †Τ΄#ŒPΊω­λ4λnπh6Κ{~W[έ›>:ΏΈ„…LΪ¬όBŒΉωυE\Ν"ύ“ugν.―yJ΄\O5Ν Y‰Q5’‡=iU€46΄lμg:ΔΠ’φ• )@8«f€wiεύ‹ΜοhGƒ™ω¨d§―:’μ’ζ\g"τZŒμ›ΑˆΰŠ>.’H\;βσϊ8ΔΖ~GGΚ;uφS&ΦΚλh=T€b%ZŸHrιcЁΪ|Ο4" ΞΉ(Ν½cl:oˆžΩΰή,ν{vήχ¨Β‰h‚»·υύΰ‘ ‘*qρ=τ―lΖdF(‘Dn‹έChžΓF η‘ϊοΜΪΘΫh5ψ0‡°ύΙΩ QŒA•,-όΉύΤ[ω5Ώœ:›/εΚ2k)M pϙςΦΏƒπ‰χp†Ό4BgόΖ²„©’]tΙChη{ήY‡ϊΝd £―]¬-ωέRUzΚK₯|δcžΥŒoΰΡ₯μ#žΠŽ ‚IΊ5† W§¨’-JODλίG+8ί§ΡS(ͺΥk7―/&ζεD+ρCM›H—hƒ>ψ«ϊ(p2Gr”9„+σ({Rαμ΄|±koΝmuπ9‰·Ξ<„ *Π¦ZP5ΐ*Ύn„ΈY·‹ΒWﱎB…mώΪΤF sλ€qΗλής›±‚Ÿ£άτH(χ`ΪΘeΨΊ*κ³ΖHΘ2±E•₯CGΉΩR† X:+–Ž6°[Ό©liνw]ώt«3>‡vσ-ΰ6™υ…+%Ξ» gXŸ^»@œbΤ+θΞzΝΣ3uΐϋ 6?(!œ&–ί’οΦ υF”%79Ίύ‘yγ“ΨXAG€Mύ«S‘“Γ½ΰϊτŠΪL&Ώ»έΣ<σŒ”~Κޞ„―z9οχ€/δŽ(¨ΆΩΨ‘ιΧ³tφφžΚΚU…KSuΏΥ \2#β³±–,09 Χ;{ξγˆΝcpε ΪΩΐ㠎}ΒβR€ƒuυΌ―€Š‚Wρ»Αf{jρlΥ>ŒGv¦½ΓˆΈ…¨΅#Ώ³ŠΐΠσ,7Σς2ψ%~κRšNjΜl“ΚνςΊ¦ΆdoςεK ߜ‚§η yX9-4ρŽίTΝ|Ս>7“πς‚φ&ηΙ™)ρvΎ‘n«ΐcWgŸ@žΠό…~πEΔ{œΥOŒΕM—[ŠΰzΥ#uN\Α4PΑ§<©­^⋧}ίψd•eκͺK›c·‚w‹a‘ͺ3ΖO©XΡ)`Y‹eT |ο€ŽQ«5‘) ΑΊ—~΄I—ΤλΚΈrjΒΈΰ3ΐ I¨bZNQi+χ£§x‡Ϊ½ΚΞεcΉάš–Œ°&zυ9f±p†μΚ½ΌNxψζ ΠφΎF5 θΐLt2/„b;6ή#Þ}Hš[₯P»”+’ΡxΒΝ~U$ίΊ‘΅Y§ ­,m=Ά"φξd3¨ =iηΣ¨½5–£L Σ‰œ“χή;‰>n&ƒ=ƒ`΄‚€„šÏQk~ΒŠ±8Α‘4Μvσ|ΔEkάεaόΉ$ΪsXΈ― >€vždN1υ/ŒΝΧn+0ϋΪΪΡΑ ¨§έJ§΅²4κDv‰ΰ_––‘ η•νΉγρ0 9Žml+Π \ϋ\z†%‘"οΔ„N`ΏXgv¨²€ŸΏΣ lf0Ψ²τu_δ»…ΉΈ\Ιδt D‘SσρpρθYάώ$ΐιΆμ|_Α₯€ςhυDWΰrPόVμΆ½_Π3ώ6PΣR’΄Τ„τψ]¬`²­@”yD¬t"Άϊ Qς.Θ2—«BPb[+δπXχlžΙ}DΑͺkώ'(Ξ—;Κ¦aΓ2–ZJ6ƊZ΅„rθIP£τP[ί—΄2Lε‚ΛtCΑΛ ')wΞΕχ_Ώo£q YCo Ÿ»£ΰ[μ¨•ZΉΟΠUφ!»‘νκͺ6*SrλΦ ψ¦6 1ρΆδ5 ·„΅aŠœ₯Ρζ―,κ­’²yΑWgϋ4W~›ψf,oήώ©β•<;L%A>X‰έV&λœxΎž —Qžγ .YT±νέυcΫΫW“tφc―‚&φH…f.ϋc))¬ΗΩ žκu¨VφJ·„!λTl‘€oβm _Ό:œBϋΉ?ΦΒ‚’―Ί {«Špf‚Άͺy$cIΦ΄κ°H9΄Κ=σ”ΰ9bί(αχ•¨|p>CΆΊΡ87nϋ©Ο„ κ„tΰ‰ΜO·WΏ›ςΩ”„c14#‘ρ€ΚU›υw~ˆHΒεNn©°β9;ρ,ΐ4pIΩ¦ήWρE`OŸf<»„?ΕH`%ˆh2D†N’τ£qνώpͺ Iύ‡νθΛΜ΅ŽeδGQ–ΕΛ δj£εΣ€™'β@ΐRpZcE­Τω9α₯£°ΖΊ0Ύ°?εΈ‘Θ#OF¬Π\§S_U‘;P"ΫΧ@ #dπSƒ07&Ι"ΜΨ(½‰LΐΘ/\„–Z΅‡™  Š΅ΜΨΣ3“y }³ŸQ *=7†M—ΊΓλj3‹SŠΞ_^χ›ρ"QΡ`γΒΌυA-ζuK7 ΗΓψ+Œ“’†ž­Qφq`½O£ΎΥ¦ˆILΜq»hεiWμϋs†πΖD,Ι\R“OAτ0²υΛκo?€@ƒ$?[±ΉdL¬ΜTφλCΉ‡•DϋίόΖaSΎξ,Œϊh₯χWό.λ…H{τ]‹'?9+S…杴€wm3ϋΈz–φ6χk LiΩηξšΚeΨTkj­pΎ‡eAΔϋ<[fDwˆlά sw΄½Eζd4U” "ϊγυΫ]ίpΥ SՁͺ…24@k’ψκΐxε;]€Ε_œKw}χ²Αa'N'ίτ5’ˆύ’3²έ8=ΗY†ΟΎZΈψΩ ³•yˆΈΓ'ΤΕβŠξ*ΧY謊xŒzK[S‘οFέOTΆ£/ΟR`PΔΙsΧLtG+Υχj{1hΥA»u‚†KʊΟώ-PΐΞ{ΐξ­ξgβγ[ρτR”ΌϊeΛL™"Ιx€–2κωτΘΏU _C*Ήε±Οq―»tψ9f(ΟΐΦpυύjŒίΒγ'αQΑϊ‰\Š ϋFkΈ·—AO2λ$λvΎΎ„‡ΆΕpV ύVΉ™ Χ£•i<ˆυ;…~DΤSίψ₯ν€° zΒ•p2aύΕ άΆΌ±LΡͺΉXE* _5½±β#φšξ %<Χ·S ωιqΜadiή†™|Co«gΙ$c•¬#Ο^Λ‘βΡκˆχς~Ÿ}γε:TB 3H φ΅ŽnλJΎΟ ™jˆόθ’7κ€;MŽωΪCevZ~ΎΚ(£Πεζε˜6^§Ξ{ENHNΘ\HQΚ±A ΡsγH|…Β8ΘYΩ΅Yφg·²–lO—‚Gτ'όήΝ.Π§Ρ€ϋ›iL‚)W•’ƒ}Α3†£&ΑlΑB]Ψ/L_τ2Ψtoβ·ΧέΟ6‰!Έγn¬[)βZ³‘‘Ρ·ρ[ζwYuߏ€ΰKg»n‡aδALUφΗμ\Jb yΪs‘K$§P'ESOlS}4Λ|Š€’-Ι΅Μν4ΓSrΘYΦ;<’ZωΔ ΡΠ €Κρ¦UώΉώ½εκYfΓ8K•r#βw[ ·ΒOu½υEν‹’`g½h~ΖΘτMςΥνO`³9·œΰ"UΪwiΊxY2.Ibά οqšOrΙeT…zν|R›}gγZαΫQœ9Κr‡«•Χύ&>α"ΔϊW¬£mρoξd™Ξ%€„ύxʘg=πQ "yΨgy8›nΐωΥ{ΟΪΕd~ήK’Νΐώϋ$α“t"”όFg ytE΄>ΕT±œVn€‘ZwϋΗTrΫk)u"ΡgΞ›μSο@Q~½φπcu?±RSxmΛK%ΐŸ^h2z$ϊ’#ρσΣe'U_ΏSo©sβζ»}jE,˜AvBi*²™ΉEjρυνλ©μpΘ=ΨG³ήi³0ΥSl^Οξ፧1ΎAΌ₯šctnδY1F–UŽΨœξήπ:ξ,όμΏJzt΅ηŠD,ŽOρ€ύ5>0ι†ΗΞ’£@5 ·•ΖέΠ%»QŒby»/–9‰…V ΌSL,ŸΡΌ}\ίΖ!EήΠFΌ΄;ψa]Wr:ANΡήΣE2vεKšάΙ@]V8ω^ŒΙ>?Ζ^Ώ)οˆ~€ΚO†=ŎmŸ³S”˜˜Έ΅³ W›ΓzhĊpVŸ*ɍΆζS}Š—Wn,Σ$Cέ΅ύΝΚ΄Β½θ_žAŠ6zŠoχ ΥΠ›ΩPζ[˜Y½˜;ιFλ{aϊ;Λ~T΅ψ―MyΖβ/sσΙΗkj €ž‰c‡M&ΰφz†©at"Ώœ‘ΌYϋŽ14V !mιώ"βΔ*E–Ω 5ψό_?ή•λˆš ‡<Β(-Μyγ“ °ΙοJšςb,ΫωΤ’1Nδ–&hπ1˜Ωc‘Λ,βx¬2Χυqdλz¬,n½ά‡;SŜc{K5 όφ’Ο|‘°[σb…ΨΚz―;•”κί Σχκπθ!ΧοΠW ŠΚXi‘t ζG\?fδ3±hωύj“ˆC”Α>q –¦θ>›ƒF΄3τΒTVαKΙ%ΐ+&–¬n˜/!Φ@φ’σ“³n4¨+`°―}z’ΙN\νσׁν7ψV dŒ…0~―ΈξpζΩΉtykμS³ƒQ0G8bΔ)«±Sφb΄^Ž==νšυσVΖ»΅h(ΏΓu€₯ρ w«ΥuͺN’«./•œ^NΊΫθΨ(Ξ.šΐΉ#€ξ5z†£jεBΟΕοkγΦZ.ž)U‘M %aΈυtΡαhί΄0'(όςHόs§ξ4ΩHϋνc>ηtΊ—Eλ‘„β3Ÿ΄T}§…1+ΝZΞ0ΏύEΗYΪπΖχZύυνδwAΉ&&~•jPΧW?ό+ΘFn€FΣ †BqA²»7UρΥDO{ΏB:'Υ«1ΈΗF4Ε2SFΨυΞήήΨΫΤASI^ΒaXvήέΌΖyιονΕ@A41±λ¦‡τ(!ΗΜ±Η‹](ψm^λ3Ο%i©’ΊFξ(qfΆzχ}α[QŽΈ΅’ +v2V+i,1„ε”ώ·37F…,έΰπΚΤEƒ―ιd\CdMŒ );ߚΆ•^’p±~•#ΤέςΉΙ•IWOƒΑ_’ύ€ΟT ΞΏ4« 5Ž=ΎBΦ6M ©oB ΎP΄AΑ­Ÿ«T«~2;Ξvαv+Ζφ–iζ8F"™N^ΑŠΓΊΟλWύVΖφ¨ΏΤ«dAΎζ&Š—£ ~M §Δ.! χ‹‘˜OLN »ŒϊΖΫYBΖBQr6ΡΌΰΩ[Έη*iΔF’DψQ›r„—ώe`7"-z>l‘G‘θΕχƒ₯V†hμn)’ž?!'pǝVσ|pOdάQWΓ­•T"ΖHIέl›™εMœε³ ψμTU‘¦wΐΩem₯»ΣŠz4}I?P ‡bΙ’:D:-I=Θ8A£m΅ΉΡΩΌ1JfΧi–=-—όΥ[U^jήρ–!‚σSΉ„Œ–˜0ς&Τ8ωώΑ>m5ύ* B­Ώ£š¦ 4žΜϋœΨ|6>΅ s{a˜¬Ό1ήfe΄ΛdEbθγΟ_'{Σλεcž°δŽ2ͺ:ΧmτΰΕN¦)ΕEξψΞM6 ΎN’―b¬Q™ύžΜˆ½Q­\έv!ω‘r_ϊ[ Ο“Šή¬2C<˜–xΎώ‚θS››—JKȊ™YN–X/ιΪΑ §œ΅-¬<_sx»p끻ρƒ=˜ G§]xχlμ6Ο/Ν4}χωΘ‘ΉΕ5Ν‡ΛηD0=ς"ς +Δ€65dhΐIγϊ f!ύ²’Ε1ύ;ƒ}j€ε €#‰Q%‡Ί΅|ss’v0 ΪηE›!zίηϊ!Ρo‡K₯Ιl½(Υ'7>˜±$νƒnνŽSυj ΕB—rΊjHHα:¬,ˆK­€"»0M¨οΔΦϋ @Έ†° -δI~εvΙΏ˜Ϊ»#͎ZL΅šzύhΗmaλNrWΛQύ,΄·ΕΚυ4ΆηΌβθfΖ.Α!pφς‘ΰŸη]‹Zre4n%π§ρ›¬:ˆ&±^ εJϋ«ΑΞШϝ^ 1[@ο|Ρ9B"α^γ‘j ,Ζ*΅H²Ί:όoɟΗΞC’*²»0e‹φ't™ΚtέΚ9ςΡQuΣlq>Zέ ‹JNYΊp*―­Ή|τά(,@Ž0ڌυwmcΫbh© ―άΡ§Ρ ρΫXδ‰Z-ΨVx G1₯Υ&€‘SG¬­m¨ U2~΄Lžη0ΪΤΒA·)—RυΪKŒH’ΰΝ°!p1VU+K5όίΓρ0ƒ’mΤͺπ/Γ €aΦΩUΞ*Ο…%B³}ΟIγsΟJυBž¦Ž  š!© iCΎN g18O‘Π9{sμ@B΅ODΙΖϊ‚χ) ¬»wΥM»[¦<"e_w=ΠΡ s[„Ψ‹«8Β—FΠ&HžΗO«Μ}•ιΗFgΕ «Ο!ΕΨX>bJΛΌΡ=EμΔzτ$QύfηχΘ\§Ž‘Ψ|CθHy~B_£2i<ξGΨ—GΊ0f,žΉlη’TXίίCΈώ{e?tœWA­alDΔE2Χφ4\₯ζ…OCi ΔO‡€[aΤόˆχž}D/ψμ˜„ΨώBΩΨ~›Ω*^EWιιθhΨΦ#[5$ξJŒΫ€ωD― iLφΑΕYεώΜ—±<A«^Ύ€`L‚Α’…ˆΎB:ΧIcΗhς-RcjΓ–―c˜ Ο7ρ*ϋs‘½'„n=φω’’ΙΩγ1Αυ@ {h` ξRlς„ˆ7<ƒφEGuΙx€—’h šΧGV?½'Φ•O)$v +*=Ξο’¦σ Β“e4‘}ed‚ϊb³Οc―šUen qώςΔrΓiP‘[(±φcΰuvδε”7όˆθ©Έ ΰ1―Γ΄Ι!―Θ…²]s­σAVy7“Ž ‚τΞ:Σ‹XΆφO1ΪυΚναΥoR0[ΖΏψψΖ;¨Ϋ―@/!ϋΉΧ½΄„ Œ¨VΫ3R)g†ΘΙD‰ͺ•·"Α˜νJD“ζ‘šΖπ AΏ,qoœ€wšL»»ξτ'ε&ηdΩXΝ™ŽΔ2!KoΗΏέπ¨υyπ¬ΠŒ?;aCiu—ΰυξ 'JV€P$Ιk1Ψ> Νμωή:»F·MΉQ;s’œ§"ZrΫΌ©ΝΣ½aΰΫΨ΅P8ν}§ΜLm:³Ž,@¬…όpΥί™‘+—©εbΥ[Ήϋ“Η0*ˆZ€ R†“]m«½ΩN₯Gq)‚ΏΈλ&ωΰfjϋΘmBŠυM–­*Qν tΦ΅BτΆ"Ι₯ζwΠχέ$A΅Ÿξό©3{ŒsŽdίΰc/X[K@Nμ§f(ΫO²Kέaψ/Ά…Ά:λήγx(ΓΠ§EœΨ‘ΉցήTήߊΚWR’€ω WΧ%OSϋb,Δ&CC¨:»CI΅eτΣΪ‰aNeΔ[»˜?—ο½K֏[Zΐ΅ϋw2Tv€M)Ν2δef·Ζ‹˜v" €ͺ88QΦ4„…ΉCΨ+Fω΄­ΤMy˜HUQRaΓ™WrώfU£Σ’n€5S†·©JwYŸ―sΖμ‘”zkα½ΣŒμ ˜ρL8qPb;˜Yκ¨γ+“Ψ]ϋ恣UHEΆ,ƒ΅v37Ύ ϋX^QΞ«šF¨[ΣλBIΥ™‰RήIT‡gή8Β<Έχ ‚6‹Έ•ρ»BΧ‘Iή»ώ£†|‘Ž΅Ϊh£ 7y ΣB%”gΕεaQΖν¬ΐϊτ a8ϊ6nΚ`ΛR―ΌΗhLΗμΜΏαRΑZ¨_ ‰ͺž!Ώ{Xͺslf) £AΟ‰£χ₯~J₯¬ΩαΟeΰͺΉsSW±gΈΈ«Ο֟ ŒPp6FŒΗO5yσ€ΗXS 8RΕ₯kϋmŠFΙbΣ«1ɐO΅„‡_ ίcœ/'Q­&α}’€3Z€ύΤ§{χHδŒΏ²`κHύυ²jΰZ_k0!N蟫αU " Ν!©Ηqy‡P!ιcο]Οx69R"Oo;qΧ~›π‘^ωξΆΌ •žsΥ;.ݘUΥ0xkLΐΚdHΉ¬oΆQ†Τ+Ζώ€o‘X˜)j«ŒƒΙ,wLΝ  c.'Ιv#­$ό§Ο­g=ϊz–Q€vδJ€/nŠ…`·Μ5Žχ”3 Lη!>όώΚHω/LΧ¨²€+΅^9άn««;—φ%Ρb2εͺQbuγβ…#—–Q¬’<Λ\`§-»τmάƒ‰‘x yzβVΞσ§m`-rs …C4”υT«a]«\[η»Kψ ίΝ:Ίtzο$§¦’D.sΕ9E‰ς»B›#ŠΚ ΒΕ.yQ[.ί?­±!ϊfX)(Υ…f:XZΖώgή©Τc0OL3 ;οšn4;=©’hΩΣpˆ` ­θϊ£–‹·Phρ©TΚ γί Ή”η€—¦ ηrΛ|3/^7ι’3Σ{ ›Fρ!9W —ž\Ή‘C4S2ΊΊ˜L?^ͺ‘_ΗwύίcΎ°‘ˆ9Έ7sγ1‡:<Ψ±Ϋ<ΐjή?€dtŒάΖ½Εƒ~ΔqDh Φ([ε±1CPΈ)d»Š-όM²“^νŽ)†5Ποΐw₯Ώhy8UW’B|D~·ΡAu+y‚XQieυ’Š™Qν4.υω&ΧφCˆJ£l―£fv[l(Α5Υ-œ~=#]tΗΎon…U©Œc”[ ξάy7ιLFπΩ&ιφž°ιΓƒ7:gŒV_‰™·'n‹I`ƒ’‰@u‘HήFΉνΑRz5ΫF9&3Vv+26Ν>έΔά¨έ錘*―ξrΆ.Χ`wΖ5¦ΉW2…³ϋۚbO₯Wf·4q‹Κ”]ˆuΕJŽ8‘θ‹5ήO {t‡ίμ̚=’Ž’ψρ…,΁\€›¨U ?Ξ@8nΎ8ΆαΫ•ΡEηθŒ~_+φ™}‚)R7žCρ`p°B„§Χ²VλΥΊ^B©χΉ“ZΧΘC§ί!抬0ΫώΤ neΉw± -ΕΨσ[zmΐRζHjujnΕ© ΏΡSεq€ΰ6°-Α”]αΤΏ„ΐ Μ¦θSrF鹝ΙcœΩωΪ ͺ.±pU~!YΥ©|Dό…ΆΏP†r0Œ}/Ντ΄ Iƒ4(mΈ9+4 EΉ MFλΖnί’l”;3ΤκpM\=F―|3o/e”έϋ”=€ƒεB”›Oμ4ςΊ$~Ž9_œc56i]riœVρ«₯=Vž4΅K]·νG ΉHcuΚ6,Y…νΟ~Q«δL— w~ž0o;ζn!=rA ²Dt P1Ν: IŠžœΓ;―μΘ CΩƒΈU)eyxτwφK`Η›°UN6„ΟrۘύZσ’α|csΆ)T‚3:u+ͺΗξ―Ξ"²'ΞDϊmτ“@eRžˆ\Σσ“”η―o-X1ζχ7“«(hω«»²χŸWψwέλΉΝž6:Pέ!D"°ςΝ-ĜMސ›—γγΊ4csdꌿURˆ:ω–¨/T"ώφφgηέμ‘E#œ9)Λ_ζΓYXxsΟ|„t‹₯P21 νlςƒIέ[ΞlέΪ:vωƒcTBZΡ\‹ΫΤE8„—Uγύv vίΩΈv%ΪΕΛΛώΪ™‘ΡŸρλΡ:η61ΰ'w>•gΪ…s5+:J£E…!Dρ―с۔ɉکΒa’²Ά» ±›΅ρϊ{iήΰYy“μ(ΦΈ; "£ΗsάΣ·™Νμh}mμΠ†”„YιΟ)—Gͺ†€iV7€$Ÿ«TΪ>§΅KdV FΔζeόl#€])¦Ύ΄Œήή1KNŠ<^LΆωa©ήΕΒuπο™/τρ7ΨŽ%πw§ώBXŸˆγgι±½•A1rα‰;‘­B‡₯κ…ΣΔιiL:r z€8‹„Υ2ΝΙE€/dw₯[νU荚€vΪoΒo¨ΘμšΑΒΘ΄ύ˜€Η΄vrμ‚γάxΝL5E8$ΎΧΦ=/τZΘ›Ϋ ψ˜Εΐ€—±Εk)ΨvΎNΣοΛ{vG¦V™&…Ρž^?₯첩nπRV“"Γ^7΅ύ΅&„#θ­K"rέ'-j εΟEc¨J_xΨ_MΪ™8Σακy9wμ9ŒƒΗ€?Ό;qB*ΟwIι7Kь@[fSήρ{J©wX#c+ΐΧZ­ψ°AΝmΪΧπu Aή=Ÿ<άκ‘])‘·Km (§άίΧ)u.BˆΨ~ΫΟ^mF X]`ξV¦άvNΦg‚€$c€­~s ω€G"έeέb,„fQΨ½ί*?Ϋ^Ύnϊ*S w»e z™“-―~xπ© (₯˜Ι(ΈOΡTζΦΧ„?sοέͺŸiι{ι8,7}Ξ6eO΄ΏΟ‘ΈΣ„QLμsb«Τ&„cζΗΩ"2'„Ζ=#7–λ‘gΡκDΜδfοLψ%’Ό―n6e`καΐšŸ+Θ]š†½cΙη¬)=;οaj―Ł άm#n(XθV>²ΚΗυμς”;NˆLK§η“]xŠ;ιΓsΛRΊ_'ͺzˆ@$ίφΊW3Ÿν`“ P|kTEͺξ L(ŠŽ+/ K L ω9ωΚ›ΨWi92³HΎYσ]y]€ίέPOέVΞ"οTΪΨpFβίͺcˆΈ²FY€ΠζwΪΪo8»ΏΠs ՈεΝ(Α°φ gΔLI~τΊςʁΎ‘]Šώ‡υtΥ}zώΛܟ7μw΄ΥJ‘_`Άi'1˜‘ΘdΖBζbz‘1₯­€u›]`*›mπωτ‘Ν\;₯eIMγAAAƒkΤ*ˆiƒ/ΠΤ–χy΅j-SFΡ½8ͺ΄_E½ΛΚ}΅Ο6[†Λ½rΎDA̍ –@™_€ϊ•·‚SϊX₯9FΔ-·Ό²6ΈΒ½ϋJ”ΥΙΑ³7΅:w`Τόκ£"?v*Y „7 ±ΣΝψαΧ”ΖΰΪ‚tp―½.\·Ύa4‘ΥooA9ρΔsλ8]ιW¦ω(Έ“λŒF¨$5±Ι,Δn τ»–U·?)܎qnxx*"eš!Ψ©Ω€μ@e«¨-ξz‚dχRK<ω©ήο½ΎΫ“tΣΕyGY ψκ^ρ›Y+2ƒ.‘όγφUr1Nr›;Χέ­?/ 20ϊΈΑ/ΟXε3¬{4‰dθΐΩΡB―G Q 0cΆΙώΩ‰{πΎI(ΗmΛ“δΨ" Flͺ>ŒvAβΙτϊ|? œTW#«BHnbu”§–ζo‡BU4υΤ«Χ€ΌΦz1ε%eS|θG—%9Τξn|NTV*6NΗ¬<ƒeΕ(CΏŸ@΅χ'Θ6Ξx™1σΨΆV”Ά`τΌ&―bΐYhS*΅heτ >Uoΰηi{κι±’Δ· 3ΥyΓ•y‘Ζ„|‹ΠΖGV UΉ<Μ›θxθΞ»mάΜZφκ"Κξ+Mφm)c9+ΑYπ‰s«χ'νΑx‘ϊyg[8sS9Π―9Λi† ‘•ƒ™]‹Υ\}3°ΖŽˆ[`zλ/ζ`>•¨’‡>]‚ξη’hΐΞy[¦R4ItS<>>ψ‹Ξ»ζ™Cβ‡Ν²ΐš_Βh1’ΓΓ4Z­Όu ‘(ΌxιBOq6% Υm0šW&νΏ΅`Ϊ•53g>qς«Θ?Ž±oΫ/ϋΏA¬9€: Μ~HΟυυνzίψŸgY˜γ°]:Lσߐ9­”jΣ3t"ύ±žΒοβΞy‘ͺ=ΘψΨ8‡EQeΧNΫΆmcMΫΆmΫΆmΫΆmΫΆm½ΣΉ―΅Ώ"I₯ΐ”ξΡAήSΉ’ή}&&΅/΅ΑjλTQ³₯Μ(Ω[άΣ¨fOd €Dlk˜yϋ謲Σμ0ή€υ$ρ_—Q-(ζΖoC5¨šK'Χ) ΫδϊDwρΤΪ8j^$NKΈž£­©œΐ-egΔΉ*—>Ϋ€©-”άΩ Ϋ`ϋγcΓ±Gΰ°`™:}xΏ† £ΎΘ‘(δ!‚w£yžƒΣπ2 Γ’p4ΦΝΩ%χLιΔ_%½ ’‹ƒπ4β„Ku)ͺμΊ8νΤG―!‰Ε;DFTχEA#Ϊloψƒu2Ώ”'+ΰše₯10ΆαΓ₯Pτ›3³€Ž‡’aύΓ§Ό¬—₯ƒB)£ΜB«BcθGρœ`’8έΑͺDȜ~ηRNγNΤrΑ "©δ΅ΎpvΎμF/b¬FšFπFm%3%3€QDŒOŠY”ώ]L’ ΪΉΈ¦=°|JžUwΘ{yR—©)*–±™aνaŸJ§½0—lFI‘¨‘T€ΛΔ\ Ilnό‡ΫlPLά,θu βά½)ψύ9ŒΝ2δΓ\χγgdΟi3΅S‘˜ψ@‚D2ΤΣͺ˜ο'‘#χv§@œ3hrΰwkΜ@τ SWKϋZW⑨;χΙΕoε ―Θΐ]LžΓh͸ݟ|θYΚΑΑZΡ©ν&™{”™JU4ƒOT€šτόqXl:w±H¬ 0@ΆΥ Ά£Λν7Ά‚‡駣o)‹oΓ ΐ†5[^šaΧφΠηύ΄…`ΐ%₯?oΧ, ΕϋΧ».cfP}sζφΈ‚σNάθΟC@ΉΘΟ‘¨~‰1›T¬_κφ&δp2ώμ'τͺΓπ…PŒ &w§σ˜u©ŸΜeΎ&κέ} Όiθν՚’™Sθ­;…c–ω›ΏY=υ{£^½pθ#υΏχΫDdΝ•;πΊ\w—i¦ίŸ₯Χ4&!bβ~ƒ­ΑBμ0ό}„™λ`ΌΛγ‘Ψiaο°q‘I‚€g΄γ±ψ©\nA,Έ"ή₯&ίO€Ε;—ΎΚζί4¦vωEΪ!©5D΅~ήzlΰ}?­b“”£Ξ…§ Θ=SŠ.Σ θ ΡF‰ΐπM}wϊhάΏΖε6‹θΰyρ_ͺΊ>De,Κλ{©ƒ±ΐΉΝa<Λ7`_Ώ‘lvόΈt)ΗJ„—ψ«‡NΉ •X0e‚ŽXζ Σyo%Ÿέ u9Λ²―+ υ°έ³α“υ0ŠŒiω ΌΔψ¦‡L[‚£ΦAtηυΪ©΄ρPΊώΦC‚dπ䐬B··ΞR f=*f!₯HŒ$/q1μBilg(Ω€Œ)ŽΤ·6Ύ*Δ'Ό9/Ω>ΊΩ*~4@όΉiΉlDΝ_)3I rIςŠψ©Ϊs‘γ»ΪΚΖς²}¬KE鐒$jš–ΟGxΧ›Γκ?¨ŒiT…00‰J*‡₯/ΆΆE’·έIΧ`­ο}ΖBΏ|ηξœώΔ\Όžή;JDς€Gœ‹ΧPjωνΘŒ&eΖΝʚwΰ|νW|š”ι’³δl^δ*šΐΗ—u„Ώ<εωφKHž™CE¬EΫΩ‘SΏͺ’&ν)π0*ο»qεXNζa±& M½qT hK©ϊ±y–WI°]z9NΨ…ψ>cXΒρ α0†“|κŽrΙJbvϋΏr ΖlcΫμh–…Θ)f­ιέ_Υ%€˜΅Μ”Θέ”Ϊ™ςΒ₯΅θΗΉ ƒŸΈͺζ! ΜMκΜ­ϋΦAR u˜’Η0E‹Ώ‰Š¦¦}˜Ÿνq }7Oμ‘³‹b0CΦζ+ )ηΠ?R*Iΐc ζŠeABm#œΩ_M-³B–g–{μΊuΰSǏρΔcαΈh2 ˆ f›‘²i—±Pš•§ΧOΤ―*ταΕLΜ'Όz¦Iΐu7Oε‡/%έ:Θ#ψώ½|KŒμ{[Ζ ‘οΉ|τ}χ·Ξabυ IΞαβTΙγ…Dυ ›L3Ό‡ς=Wc‹ΎξYε”PB0ΒΐƒΥp °qχ^²k{ŒR9’YŒϋx„Ig pe« ‹ˆ΄lurt‰Έ€J…š%…ENeΗKΐ•β`8ΙQ³ά°oα+X*F#ZC’ΥfDnΘΚf„°[=¬ΦK’ α[Υϊο‰ωV‘{³ u§Γ†οLζσt<©Coρ΅τžΪΪ6d¦pB¨__ͺCd—,ΎZX4°Άb¨ΩαB„jζšw0o)Κθ–JA"YBmγEIΩjœ[2A£s–ƒΜ|!qg12©o ˜ΘHΈ8όœΎλ"ΏKζόr^yPΆ φt8Ρ@?&»½ιžQ>iڌ-„ζΚ‚$Γ‹,€AΖtB―›ϋΫZρ-³ό0eݝδΛrΊ1%π]^Ή7Αώ€γΰΑvpγ]XFΛ‡ώ&`»s ΉI¬έ<7‘†w‚Ej΄Ϊ¬wτ„ρ—:Ύ*{Ά82Ή’ΫΆΒj–΄}τvJ¦2ά@υzpέq–%!"nΎ5Π•S “tˆΊ]±ΦΞS–άρpηΝ’O\ΆdBb/>δwqξ‘7ΙB†‘]h gC04%%€₯XΘΟWΰFI+vγJΏAχq¦ατAΰ–Μ‹°rrM οr±m4P“`t0ašΪίζΣ¬š…†κ°Η!^BΑΣ; ΔΊΥάgwΦΉπΠ‘ΞΰiΥϘ† :\Rί£†ΝFrνι΅/₯l‹Ν UK³₯z&cφVΩ@QΩτά‰oŸςρ‰[lζΈM^Ι d&7\ΘτνEΊ§ζ=υ‡ξς#uή1o»Ε³-"5—mΎΗξ«4ΐ#^³ϋ|Be‘QΉόΝ_ Ξnlwΐ₯LzΆαΕsclηK©ΛΎ‡:ΨQf\g]yέ¦[f€Ɂ‘μ{GΌFωΡs‚},ηKetkd Έ5,,™LΖρa¦oDTP«“NφΜ·Ε}ΔίdyΩM«Iυ¨«όƒΥ]_-#Ψ΅{–GieΟτs|‹-ΐ¨Ζ …ΆβJœr€¨’‰3Ÿη2tt~ε³ΰ™” žjr-t\ΥΗ π0 嶺…n%Ιn χ j@Ώ‚σΐj-mQ*δφ°₯ςFCό/$­Πβu+_’ΐ̝%xΥ»βQ‘³bι-^6[± ,Qhχ‡ρ΄v$χΊ€3žžE9 /ϋΊή2Y<Υ/MΐΜ,EΥͺ·ΠnW[[ˆu‘ύpaKxuτ§„ ΗΝ+½‰œώCm­)˜ΡW}Ÿβ–«¦VγfO<ΦΎ%‘* 3€(9@ΟΥρ1:)LΫ¨)V±Β~νΡ)~@• pVw΅¦ΰΆ^‹ω)CDΐεΣB_Ήι†fηϊπήτ-ρ NK::œ™‘w=xjύ†p2ϊ‡¨ΨσΧreύˆ" '3χ˜Ό‰Z§œω;ϊ3:5ό|R τ–ω³@* Δ&Χm;e¬_P/OJά DI5=1K²jσ²uΈ/KjθΞ1Α Υ­ŒB_/@ˆ£Ο$UΟύάOzikj‹ͺR€ϊ²΅x’t¬–%4ύλ σ‚¨r2vν«3ΌΫqϋωϊ•Άj}κ§DσCxύKL U¨ν>Εu£λΉΑΑHΘžH‚ΪB*z6p±α0Š»%ƒt±Z”•βyΟΉS>ΗzVY‡ΔPr€»ά} ¦)H’4tΨ‚α[ͺάΩ"˜Žžγ‹ώλ²}gG¦Œw&ήΦ€ίj, _W  ­₯œφ PΞ_HzΊ9>6Τ΄)&q†Nήe±Υ\αŽ€Σ°5tT$ukY΄Ά)ΰoέιο‘TFН‰’΄ο²° @šŠK ^Aξΐή7rm š³LΪ”2,§ς….6BpΛΠ­©βŽzYχ‚ζΰ› 9 ݚ ζ8ΒiΪ½πP^½’ζς5Ύ|αAOόQ[υυšγχƒηM_€³ιsDΓ‹¨.όSQΈA75f?LF±6yq{ο…Χ i*wυ»•~ ΛΰTυ”ϋ˜F7Ÿ16†k΄ΰάPX0 v’Ž›Β~I‹Όy‹ρCπ(ΥΧT8a~G–S$j$ ÌSΉΔίΖLŽ ›»2o "‹G½δ;χΊ>’S³I v>šC£ `ψ…>ϊŠvf0!Z*ώnζt’˜‹έ°<ά0\ΖZB2ήδ\9ky9>%Xv'E³ψ\€nΰ}'HοK‰†ύΉ1Uρ0Λ^x0?HSι5qE XρΉC ‘·ύmŸΊt&ΏYΙΥΆeΜ —Βψ;‘δG? ³¨¦‡›Š–ΦYšD©—^ ΎΧ1*ήRΘzΧΙyWπšw“Xϋ‘4^Κ,ž ·ΙψΝβKΒ(σΕε€!΄βZVΦ[W¬ ‡όŽΙGZ¨ž«σ±ϊΉ—Θ;>//#] dΚϋihη160dΫdΌŠzY½Μεb„u„NΠΙX:•o(·R0$½λ=‘έP‚οYπΈΩ,8σΙ! ye°PFU/cΛe-3%—ζIΤχ²„“m b—ΕσΚΞD6ΐdrV.τ³†9,ΚJ@?—[ UfQ|!’όΡ„'σφ'œ0^hJ!’”"%Ξ,Ÿ‹ΏΐΎήNœς―P<ΙGχ·ϊδe€‘αvi€V·u―‘π!ΰ©*qύŸš—Ϋ;vMΊΆS½¦ΗΈ)VΞ˜ͺΞj|o(*ίƒ ζ’Q=‚vυ|‡έͺs RΌŠFhNλrΛ”rE­x„ΑZ “ψΖ£Δ ‘pDςΒ΅ΪΡ£1-€ηΊΧΒΫj.ίP<f©j¬˜ψxͺN‘θ6%_f +#±Ίdύκ5yΨ¨Š_[₯ď¨M,6βΪήic˜— θSUή·ρ …ίΣΕΚ―ώgΟ‡ Οψ°’;ϋ ήΩο…IIγΓ4.#ύαίό―–Κ}™U¬SπK«ΪΨ‘ΒG’^‡‘Δϋ†»M¬έ=oΆΔ3{›ί΄p•{φ!œ5Θyš*BKχoωΤ/ΎαδΞ`?1ϊ’­WΝD Ždσζλ-Θhΐ0-κzϋφξ›GΨ8DdΝbrγΑa‘‘άQ‹ΘƒΧ:œΎ™SG§d<’@έ£Ψ:”ΰlσ΅§Υ'BΝ1Tυ#δMTωlw­θs «n²F«™«‡jus› ˆ›šΣ‰²)%PΜ$:θ|†6&τχ8ϊΚΦ€•πBΧƒω’››ηβ³›‚•©kI­z£UΞm±7η"^―·rš@+y9ρc;5 ΰ¨)φB j€)$BsΟΕ™ύ°˜^Α«Hο§O’¦άšgφ½‘ f]•–kόΝ^½q—ZαH»§[*›+L¦Κš—*,Yψ5‘r†šΊTcφ₯‘;YSς“°ƒ-Υ`2P ά‡ΩAμQ|`_@0χοΣ)Βμ~ ŽωW†¬μλφوiδLt )-P8ϋΝ:H΄+vμΒ†x4ƒυΝ§=U‚WZAXκrψφς•>ε˜xKB†ΰ(†°¨΅η‘-kδ L2κz&Ψςψ(֝Ba ’φ8,8²}Θό`“ŠγW9ΗΡΟ(ΰ΅Od{μ,δ – :{~£]˜μe•qEϊfεή0:% °n8Ν]αΟΣ£aΏxϊ/ΰΌ%Žžς ΟΗAsΕμE΅OΦξCW+Ÿs΅+kΛύΨ5Έh„»ΌjΉCB­ϊ‘JbU‘SKFυ²•`ƒV‹]οžFέQz‹|k χƒ½±ŒL›§IϋœY^°λj|ϋ!i’‡„ΩIƒœ&Φ‘Σ’‘1KΑKΑ,Mα–ΰ8LΕN§Ιω#ΐ i¬ηΉ:xšΦaΰ&ΠΘή7r3“εσςݞSΥψ$tPκn MΏtΣΠσ ΝδΆ…ώ+%ƞΐΜbΎ‡Γˆ^ˆοτ–XεŸ5]=η?ΒΑtΩ”ϊΘFβ&Y@kzrΟ?Έ€ρα’#A½Wκ^TΝΠN’˜^Γ›…˜j1›σχλ­d·Ο₯­Α•iZ«Iμ€Oς Hϋ€.u);)g6υυεχ€ΥΣ@Α·A«;ύPΈ ΖρS–φbNIgΡ#&YάΞΓΙ͏u©Ό£T@Ι­ΰΡλUΩSy―^Ζ”βΏ©ΐΒ•ΥψF;,Κ0Wο &{ΌκchΛsώXα©π³ ˜ΪσΝ₯A\Υησ‘ϋxͺiε™ϊ"RΩΗμj<'Όg„AχO‘ω±­I^'ͺp}δjίΌ…‚ $3Ξ‹₯ ΆYαVψW§~τί?νΝ}XZ*™}_6##,€PΪ‡όθ(σ|kκˆOM:#Aο\F~˜m Έ‡Δω…L7M’qύ;"₯4Q֫󑔬&ΔΤgœu7―ε V%AΧ$œ1ͺ― #^εžDλΔ›xρ;Ξ«¬Š``₯έiQ&Σ­ηέ?qΫΚ|[ςΠέπΉ¨4  ρΚ·‰)x’TM*—π)’kσ,Ξ₯ε΄uΊX=ο³ά&”[*Ρξ΄•Ήq³¦ ©U-κXΕLΉŸγ·΄―Œ‘WŒ–ϊ<Ά* •y ͺ6l:T‘‡+m©z#μκ9™‡‘™¦η±a¨ŒYδqyπξι ?Ϋϋρ rΊΙŽ,–0+½;f8+sΫ”9 ^?(Ι5Ή/½Σu"p]‘ΕTŠρK|Ό•W~5y¬ nH<Μylό%™!ΘΛς₯²ϋDύτm w9Δf Ώ €L‹Šg8Α―βωΏηœ9.Ή«ΊjqΕ«W£Δk=_¨Μ·CΔΧ»&xρΎzh!ζŽϊs­ό \§€Dw4JΒiϋ¬UOx"SP”bΛ7b(‚ϊ-%=PψRΏλzxakΗ›Ρ+ ‘ζfιϋp7’w6WU…ΫΪωΦFcι*―»jί ΰλ_O&xH‚Γ`bάΩσ«h|₯ΦΥ nΠwΌ©Ϋδos3τT₯ηEP¬9{ 'ΰvΕ(k_G—Z€L!£*7˜ΉΪ3ή:1„Υ«LK7}ZΫΰ@ۏέϋ0νŽH!fΐd―Šτo«3$%TŠ2 JO(˜ W©&@»m#P\¨jΦJbeιoΑέ–οeο²l2Ml6ϊ$σ/)8ϋ]86 ΒΜΡ$ lw*₯_§.!ΚΝ‹!¦.Œ‰σ&ΏMLd­?έθYp(B€•±X/11νδ­VΏ#b*Ό½”Ν€kd4uUΑλnγš››ΜΙzχtΓ(ξ*TόA0ΑΥξ*΄mXd†α«{cYAŽ?³vΓ r\_Έ.)’Ω#=’LSε?η1œ’Ά)U Yq[s.D_Žπ$C@G"έΒ§.*¦ψyΐηΠβsς*EυΕΕίlΪΏ°vΛ;*.™^7;aυΜ&Œ%A33 \W }8HہLΙΩύ¨(β\YZpύŠ †ΰσx"Z@F^½κ`—HPKΗ.‘†“ΦKο…Ζ\βΤ@Dޜ•‚Œ˜rƒζͺοχ£7ύΘ»zΚ€<Οj–Ζ|…Œμή;\±[†Ύ+d"‰—θΟ\Σ$_YRέέμν‘±+4 ύ‡ΗχO£9ΌϊεσjŒ„ζpMλ/ίΒg΄Drδ2˜ΓϊΓΛ[¬›VγxδΛΤ’wc vϋί"‹ΫΌŠH½€|FΒ€β”IΠ§΄qΕ΅]P^ʚ, —4[Αe‹/Or*–™ζ«‚Ϋ]Ί{Zθ –n¬σ†|pυQΌΟ΄Ηiγ`¦πq2cΏ Ξ-ηŠ7"\£.TΨJ„ϊχ‘Εγ₯‡‚`5Ω)ΞΫ€ΘI’ΰΓέ‚€g»Ρ -Ω-ςZΎ$F³°*ΒSOπ<9“,ζ’ H8 Ϊ“†νfP—΅iGzέkοέΎφωσ­―<€Υ8Ι·Οοmk’7ow\VwUO;S„-ΰάo  »/C|tΙΧˆΟ;π3 ξώ,fωc°ˆ’N;Νͺ§3Pξώ₯Al$;}ΖωF)šxAδ]Ÿey`΄„*DαΏΓaΤ9λtž˜wANΛ^ΐΧVΐ₯ψΓmDlε|΅I•αΐγ0^“Š72'.pΆnyšSΛ¦ω3ίl[WΖv’?acω "°³H«aε#‡όΨ—?„©˜κ^ΟαΖ3΅±XπAU|­86ΆιύΒ€Π4Ihv@Φ¨ γ]ΚΌ WYV΅WΜ`Ρ•άΩf/s"‚μθ˜ybΛ―ι‡]7y˜«)₯ ˆ\ԏ :)Ή€΅Έ‘α~p~ΓŽ0! Φ.Ϋ1f·|ό/Ϋ/H€ΔLΚoŠˆ2&΄Ξ°ΝυΎŠΊ„Q­Oki‘ΝcίiS©9{Μ°=C‚Υ;Λ`MC?2Ϋhi>ZDP% s”Φς >Œ½υ›ˆH)3'ά=ͺ­³œ©0C C"£οΰŽ A"V ζVμ7 .šΐ‡p22΄λyVε©ΨgΨΠήcξ>ΚɐβXƒςχœΩ³ν^V,³šCΘ4^χ§676:Ž)˜]Ν―­°ξ<ͺρš:nΙρ0ΊTε¬(ŠgζΓQΈZύϋqzšf«ƒ?Qιγ$(@Ew!­•šμμhΘΦF—[Uœΰj πFΙMv-16T2Ο‰%jτͺb\΅$Ψu@ "&%Μ§φξ\Άzϊ¬4.šHD.x{]ΌQŽ ²f•δ_ŽŸ©ϊι.rL&~M#UΪEb «VΎUƒάBZηTρ7‰κακWn2ΐΟ,’­‘Ϊ1„Rδ¬5ςζ‹rυΡ·p“τΰ0ϊx›Π©ρΓG«˜Oi*ͺI%ΫLΰ[Sk\`.Š=ςΫJ‚QΈW}šcjΨΰξύ‹Jκ›`°{¬oΙ"W4&Οۏ௽š8ά’WG¨Κ%ΰ:τΡkŸυMvΦDςN‚fo ΐ” Νξά –—[+ΏtƒπoIͺ3Βα[ΑΒΖζ`κ­ΘaδΟσ³>·e’¦'ο&3°€2_qͺ‰χ–mΙ„‚8¦I³ΕΆΟA ΆXy§wŸδ“ΡΩE°DS”"ςΛηΩN{«ΣOAmooΟςŒ' R Y0>ΡpΞ“?ߟ˜{ίΕΎςΎ§wŒήœ‚.εΛ—ΉbηΏ~’`Πέ#Ρ†νΙϋΰnΚ)ΎΖA<†&ωhw¨ψ@}\•UξυκXΣ‡Zζͺπ€(g=yΧ ƒ­£©zt έ2  9Οθζ_oϋϊ8Ζ’λ—r ©sbα;Œoς€Tω[½―EOs―x~hXpIφεHΈszτΙΧΆΌο¨Ν$ ΉΆΨsS «XLΚ±2’;Ί­%η=i5΅3Ωm•ς©œV|20n–Cn†.Q΅y‡UζίVhސίNUθ?ΎepΘΛΩ–jDCΣΙ―‚ώŒ%ŸišZvzU[§α‘€X&mυ(-‡ͺ+e²f 8@JKi.O*8yΠ•ί›Uτ˜τ_ΤZ Q)χŒl4-ΙE«Ko3χ?H‹Ÿ,κK~Ρ‹η ΙAϋše'ioSmcψγτ1θΜ¨Εΐ=/²rΈΙ Ίaίlβ*_›£I\‘΄;nmLΏbHšiα½ ΞI vοFXCτ)X ˜Ν E²‘΅‡hΙD<ΏD₯‘°b ŒόEsέZό9ΘdϋŸκ€ΨkvοβΖeΑq£OsΕR~šθς‡„Ρώ#±,{ΌωQή±΄&ˆυ@‚ΣMΒZ@‡žƒ…ΏPF&ΥΆ”΅*%3fΙTγάΣ±ΕαΝCEKΖGŒΪ?K7eŒ;wgˆ τ΄²Έ~ΝcΈ §!ΠyΒTώ"'ά“˜Yάe”f6Ηχ;±z,?έFGc0ο­‚’–EQI­τΘ>lγ@–ΏΖ@Ίι:3…7έίέ―ΕReλβΞE&­ΎπΞΏSEˆz—‚˜€wΤ ₯¬” b₯/`.Ϊθ%ΰqšή+RgMjmPK άΌI­*[;™WIτz*πZΰyuB|––‘ “^”VΌFΞΞ―ΛΐAg΄E8™$―½Žω’€Ω%^ΙΩ':€€¬qžΩ~f„ Ϊcƒ@Ÿ+Pβ4ΐ©—¨― ΓΌΡ0eXœ!°ŒSφΪ!Ϋ€pώi‘†”p{Χ/έƒͺLΓ°7*C oίξ_©‘}CDNrGY+δ#Fb0ξ(ΤgμŽh'oĚβ‹-Pϋ·ΘΠ‰ϋl‹T$χζ ΑUΪ)£š9鬇l΄qΥ€•RMJmRΦ*ͺΞ=κ)q2†Ώ΅eIτΏ>.°x) ξ(#φ^;Zob;_k?Z΅§£‚ ϊέ™{σZΰο„ΝF₯Œl’!―ΠCƒ Pί4v<[Œλψf©ΖΚ‰kΕ)*Ϋΰϊ ²π(=GΦφό½’'\Ϋb›(Ί`G)NΞF δ½―tƒΈx°‡e;ξjMΚS wUιΐΚυ‹BθΫρ8Ex§6Ζΰομ€88“΄J„"Ϋ’ψΟ[,8λΝϊ½_!ώ8ͺγ ― ηx™φUΑΞ΅ŠžV jεk ΧE/Ξgωε0C_4Dΰ2«v!ϋη™αMaμτΕΎ“…m$±I8ηzΚϊK}C`˜jm‹K/ͺΧμ:H€ΌΡύΣ8Η”¬‘GΝ—cΧ;ΑA£““΄Υ¨άr@Ί΄[β„ oΖ₯)ΕT¦Κe8uߟjϊ«3gΓΓPSΖYκ«~.Uϊκiβynh]&κΠ-]Ν’’oέτΣe†³ΖΩΞ…Ιμ!Z7‘uΖPB’δgΘΎύ|nrαί₯δ†FͺΕσεdκ”‚€Whžq{6ά¬«X<ρ&pςΡt+Π< έ–φD,Ξ}±βyθZ‘@~R›EdύzhΡ ŸηrοηΠ€΅u6Νω5ψ«Ω³7μgMΒΕPQDDw‰ΎZž1ΫPtTq}7oί ?r>δ…%;j’ΩΐMΨ•φ(¦,~Έλχ―ωnX† Š^•^ς|ΠmZϋdύ»γΝ#t΅b-’²™φόer)ΗV,yImHS§š˜1UΚ 5h?6ΣN h[1|Μτa3M^Β3F@‘₯ΝΡvkˆ–IeqͺθWr—)‰?ΚτGŸ­ŒϊΑTͺψf°βυˆκθg"›ŸΖΓ!π3•ŸSD{…}u*iκΙ&%!ήaTW6ΈCt±"γz•Ž -clxε[7Ρα‚Τ.α6ΌΏΙ=i05|Βλ14·€€Δ#‹·YY³œPϋ‰ύκRWηEθZπ$?#\¬qklpί΄ζϋ–}ζ}ψj|–ΒHϋ‰ΈΟ Κš±Awp0ύ©9]gη £$CΓΠiMΨΨ² pθσ’ΌˆpZ 3ά₯<» –σι ΐΐΊŠ,η%w«ε1CάΣOVδz•1ρΩۊωΔ+'ΌYΪGσ”P£eGϋυ3}‰:’ ,ΰ-d{CM·x5½ n€ό~α ^mκκž‘'Ρ?x+dKζτH›Οχ€ωOά†Ι6x8ί]ίWΎ@ΝθγsΝ΄Μ?ΎΣΐ›P‡…gvgήƒ‰ν` I°ΓΆΓH§^hrδ½%ΪΛNJήkJΚg,’6Iψ‹Ξχ==·EΑ_Yœ)^mz6ΐl<ΫnU’%ΏάΟcWS$ή―HΪΤ_΄MzΕθM•Δ!―θ=A]ς Έ¦ΉΥΘE€VbC齝t]&?·»4μ{H ’K^ΐΊ§¨‚OJkꊍ‰„.dθjώS²•ƒ±bηNΜ§Ί‘m*šΩ)9[/-+.ƒ•@Υς N‹“όpϋSƒΕin闍g[P5RΚ‚,ΰ­ϋ€yϊfΜ™ΛςD™ β69R2γQW^Dtθ㏱£œ«ΘΔ[4€<ΗΛΎιΗ†sρΰž:wγΗš±~‹a+Qε%D]±J- Œu΄a €―ΈQ-ͺ₯ρίΝ!Ξo¬œE—΅Iΐ)ΥγrP€ζx’“[φͺΫmrΣ“h_HΜ’»ΚΈw /ς¬Έ…ğ.']ÝΒM>•ό φ‰b` Š‚Σ}ρiΞ{ ―ωdMOuW2Ή£9%…xDε7ΐ§g«™o½Ύ‹-ΪΫl»,ίΨΰμχWίE(ΨDΏ%D2―&! 6‰Ω€€ω™[Y»ΫΜAsj Σκέ)€˜:©OΕ…+©•b£νqYΊ΅₯Λf[C‘ڐΨΦΠ•Π­όάΗ„iΊ˜θfPͺΑ ‚>^ <ΨMX™›ͺΩ»δ%½ΥJbfμ…#J;!8‘[XwΨrjίŽν”\’e#ψϊf»~g}: ΅Ν΄OΓΉΜώ‹ΰδIΪΚILΊΔ9L5Α•Ϊ RžιqΩ0)4xς…4=š9?(G{,ξΘJ˜Μυ³xβ\\–ŸS!¦-ιΜ΅eώ¨VENyΔ¦αžΟΚ^^ΎLj4Ve`ηNΑŠόE&š(ξΎ½»63³ν—]ΰԏMiότφ½eks 31Ȑj’΅0N‡aiΦ4φ8lΥ] €eΞe•BwφΓ>‘ŠMίο2’v²΄«X›ύ\¨z€C΄)ξf¦Ϊ tΑΟFZΣΤ€‰pν|€@κO¦u5½W˜Uξx Yή"0 Έ–β”[$yXوηΖ nFΊb31‰χ³ Δ0¬hΚ““Vd‘ΌβV_4ϊ˜Ρ:φ­Εμφ<ρIφd~λ?‚Sχ=SC¬w‘¬G‚ΤPμΫΧξ€xςgΰ«²fJ”½3‹QΡKβ#΄ΪƒkΚΞI―ά­ϋ₯CΖ4}’©r9 ²ΡέΤΤDmŽώΑˆxAuΐ’ΜJl9°±σ TwφVZμw%οοc˜‚ΐ YΨ€bΨ¬Ο֐₯1CΦƒ7Z d,ρ Α?ΕΎŒ‰!ΐΆ“„™;,)ΪΝΡ2…–3—’Π”ςΩP }Oά@S₯JΓυV&KšLsMiŽ:tΕRe/z?·2L¬”Ζ|ΰ ΐ (€eΏ¨9νΧh¨&¦ΰk8žš[τΔ²;ιoTξ°TQN±fθ皚Π;­Ϋ@mΎhΫcΠζ+!Ž„η’έV‡Œ$!‡λ„cΩŒΈί–χšPVHυŠ3:œ7ΚβW‹πνsAE·#_ΓωlCΡͺ’%'=”ȁΆ’μ£3|`Dmg΄ΑLς^΅χρ39ίΞ_m֍4E/·j瀧΅ιαHT$’Βε§ZΥE;‹`Οξ*ο1g΅eφΓ; Ίϋμ8j‰V5ξn‡„ =Y•bmΕ+CΏQLΡΎ‚χ#²$m8„ΔπωE₯˜/ί\σΓ£qΛύd:f蓷ōγμνΫΉ΅KΏ^bχΝ#½l”Ύ9%[ϊwƒP“’'žΛύͺΗ΄Z‘ωπj΄2ϋκ<萒`•λ+›Ÿ‹ξΘΩb½ƒb aΙM½1ωΞSΞŁ‘Ύ£― ƒ31 Χΐάy‰<]•jψί½&΄³§γ ¬E˜\ΌΦ9BβmΕ6o5­&>³OIζo’ζσ#ιλ†VΏkSfτΘt` ₯ε˜J}ϊεx(·L ώΟ[ož˜ΒKKΝ£ζ΅F Έ¨ΕYΆ’λΧ†H€…5οΜL—S/ςTγ ~Ρ02»XEΦ\ ꇀBŽͺ!+¬Ÿ€”ωŠ•(ŸX/d%/T€`?šυ»ΔQΨόZ6 ΛΏœΦI’Žx7”!׊ƒΆZ±†Σ­‘VΣ€Δ‘N­˜΅@4ΠφžΣ–]¦ΖΤΞrξ.Μ9¬Ξ‘uvrΣ«P±φg $o΅xΖ5ψ«Σvχzθτg(IO&?σ‰ζρΆ?HcD¨H]:@Ψτ2ΕθΣιΌ7xωπΗͺX€«[©ΘβΡ,ΚW+?ˆ6‡] ύΗ‘8μγ2Α”4Θΰ%Σ‡AŽA6`v²Χi’VLΦυς+q‘~aώuΡ#“…aΠγβ«οZΛΎά3Ν5™]μ2Ψφ:Γ4œυP4 ύ iΉέSή;aΣρu―)4ωΩ}sΝs<Θe‹LHγؚmδaͺύ(Y RΰΎ³¦ρ­ЎY¬Βή1™$ΒFύ›„cθGβϋ½FHΐ$Cί’_Θfηi“ž•΅\δ<­g / ‹ΈΪv³ΒαΨC ΆlΌΒά(vΘΛ EΌ½άڌ³hdλO²ΡΑο€sD•Fvl\JMqΔ§Ψ~^ΑΪρ,ΛΑg+^?ιώ]‰Usiά­ΨTά玐¨ƒ0Γί3±Όφ@KήΈ’•‚π*dςrΘc½-Ϋ•Œ%)ΧΨΐξM3’m’YρnΠCρσΘ|C€KΒa&2 ϊΓ’Φx .MϋFtάڊRcXνm% ~­'Ty?Ež›¦Ϋχ7NŠν’ν QuΗYhΘAήΑ»~ή+±*μΠDSοLEκ%§Γ–ά¬(& ͺ?aΘRβ}έΏκΓ}βω@ᜟ`yί·/ΘΔ"Λε;ΰCWέ Η_…Άχ9fI9›‰[›ρ~η…zjλc—ιύ:x,BZΗqψ’εnΎΜψAη4Υ:½η]4g"D0M{f+ o‘pI!s«―‘-ζΖ­z5Θ½Ί>`I©7~ΥB@^|˜ŒγΕ³xΣ,›N§Ζέ‡~#*‹Δ›‰½^ωμγΒ|όqB¬‰xlζnύ£[Ρ‘ΨΩύ4ΈΔά\ηκόA[#™@σφ‹Ο™PGh¦ή˜0vαtCN@΅Ϋ“ν\]&±k–u[G1>w΄›ƒρ•Αnͺš6Ÿ­=‡Ζ^Ν>‘¦š[ χaG79zωuj[8‚eAτ!Χ²νΦϊ“+$@irΖ.μτ™Π搃₯?ηΣηηֈ‹eƒϊρώΗΗ~’‘ƒς€ϊϋΠ‹_Ξ¬{oμ#6!  Œ?%mMb#©ιΝ%ύό}]π+9—žƒ-²0H¨sΖ²–`A£Ή?ΒΣδ_ΥL…qΟΥ’Δΰε’•$DεFjŽ{UΔ—iνo}J˜ψP·ό›x?όœ/r›9§1+΅'^SΘ2ΊλΆ-‘Φ”μ£Κ±[›©Ε&Ω/Š4qυΔ,Δ(ή0Σ eυϋ˜ξφ‰ΌŒ1y–ΫΕAQN`*α»―’ ˆΔ›φMdn†Œ‹…Kp˜¨ƒLΦVVςλˆΦ½p'xlδάΓA*€f{» xcΘΤ²vœ’έ[Μϋ€ΜΈ:ΰβΛΨ5YpΏ·–Ή Ν‘Οη6z±jHθάsTΕ‰v$•[§s7¦Toλ ¬Y Τ _XzK*Ψ²³bBN^ˆŽw tW&\‹)Wίg˜[υ}WCΡz/;'aMFςϊv¬½2²ΟτwιΝμuŒ¦‹<6Ύ₯ΡζέVAνύιιΥά{=ϊ…Ο"Iςκ“q\Ν#o^Ω༟ͺη΄՚™²(Ε)Œ…Z ΐΕ™kƒR Ov#ρ/x²0H8Ϊο…ϊΣY;$!qАEΏ*ΏΊ_Dε=͘ˆΟgD«Τ’έƒO|2š¨NΡg.κ.Ρ l–ζ3Ϊή—z |ϋ—ωΈγεgc΄dξ_@2 +GδŽΤv p™΅γw†&u_ Β λX^$ν{Βα΅›6~\AJx'₯pŸΌωβkΦΚ₯l`ΐΐPΊ2₯Dκ°­EΒ3oΚ[C9`uπί:©ι*λμIΣwσŠ·m΄λmΩ‹ςc@θρK;p‚η“ΪΝ¨OλΪΡ<²œ5ώ˜:η »mΌ”*Μx’““ŽŸ0(4aϊNcORTΕ~Ύ7Kž%₯ME&t“—ωuωϊ{ΒAښyδπ'τoA\²8‘ιΚΞςεΎ §ŽΧΡμος†iγοΤ&ŽdΚRkζ0iπ+Pk^”-7A\(Μ]*θ#ZΫ?Δg!‹BRW ‰%+‚)pϋα΅Ύί·ΓΫπKIΪΏN”Σ}Γ‡=σξΈ*jλXίΑΆτΓσΦό ’Β¨ρΎΘYοM·΅Ν»{cNŽ¬‚”:`QN†yM‡ƒ‹ ¨2₯“ΧφΉΑlΗC‹Βφ―όEπ6-P7©Ν`φ»β»”r ‚_G’Έ£δ?ΈμΑσι& ξ#{n Θ6% %’v ΙuI―ψ<β™Ϊ&αΕί: €Χrλ™gdb=(žV'ήυ”1΅e·ζG/ωΆ³HαΤsΌG}Θ\•8Ι6:o#«―e1wnΟΨ­C%ιNτE|²7£χΦGxΛhy“ο=ΘΤ5ƒH:Xίi‡‚Ό–ΪQx½’ F³·Ήχήgϊ$v‹kΣ‘l ‘Α‘ŽIhΚΓ!h:ε{+‡^yGΩp.Š9qίxLrtΓSαŠ'=ΆΕK€/2^tfγ~²ΟhΘ/„ZΚI R$οεMEΥžΖυ΅σε\Κτ­ΙωOkΕn£LΛ& ήΓHaŠυ‚@ν kbΆ,₯ŽπŽIύ„8υ'τΉ&Arˆ(ΒΏR~ Z;Zψ6Ύ˜TΑ²ψΡFίM+wΏ& ͺYͺΑœ“₯yνyΰFΕ‘i|ίώχ’LΌ’6EQήdnΑ:ψ³²γρ„œ§Nœζz3φ!?p‚g ·’ΨΉ;†ξuσD?Ο+½}A΅ψ†Ž?LNςDγ’Ο0kyχlŒ‡pΔu"κΎ ½‹Οΐ„Y›ŽV€Τo“ea#ΥΌΝ ’ ##9έlRo#²:ΒsΟ5Eg‘* ½―&‰˜ZŽ_Π‘;eΊ"xSγβόZι`Q,ŏ Εά+—ΏΩX· ΝΗτ!8@••Β‰f(Z%ΖrŽgQxˆ*Wγ8ΎΡΑΏ‘>:ϊ·ŸͺLu1ε€±‰qΖ§ΤόΦ±ν=ΒZͺ!ZM³fτ%ά̈3 0띺΄Φ΄€¦Žϊv|&UŸ*†J—ΚΫΆŸ€kj§P!>ό)Ν€l ό ₯_6iŸQ<(=κŸ$΅cAsΡ"FfcQ[c…σyϊ+υoόS‘/퀜ΠOO£ΏaEυ¨Χ›ΕΩεαRl!₯@CQ#ώ$}‘ΣαŸZmsC“ν«θΛQLΐ¬~gΏε^§3~f.όΞFΞΙΠcΎΆ[_½1a,s*A Ή«$kiA'aΡ6ιξa₯}ωS[IΕ±Ÿœf$κjڞ―r”L`ύY‚΅Ε<κΕWš‰qˆβi§τ’ŠΈ6ϋz”Α1ςM‘₯ΓY+"L]΅―Ω‡gήo δη 9ς[TΞ κ €Ζsm²«Ή (γ§…M‘8Lˆ—/=cK(χΎ—–—N€ΌύŒ8‚ˆ?’q²=VΑΚǟΤΚ™5»53>$Vˆ‡e {,³»‹Υl¨Ζ΅ς΄ $Ρχ_― `³‘+!£/G ε²4pΈ*3D?¦§ζύ€ž 0τφ7V”JwC§νGug₯rYΕΙξ=χ–Ω«/ZeΊι;PΟ“Δgε#oθDα7,²%©]/oωφ֟iŠ‰© tβ‘”ο:π±ε7u‡Ίλ¬#chθM°_,†ΞχhbF<Ό0δ(nβ‘œψdήMΚ›n„=³ZΘ^4. }1Κ‘D6κΣΐ˜ΦKƒ OEŒ§fα9/i²Τkƒ‹hηΜΆžqM€|ν\ήΤ™ ŌC‹wP$ήE0 †ΌEήί΅:υ\―o]Mθ”υJΧ^˜u5Υ{‡ ™‚`~£Θ>7s&ΓΕΒή›ωWƒΕΤηΞp6ͺb9~_χɘgΰΌήδ½qΜ»•α*WΏ W›}]–¦³o[„ξe“œ°†Šή «ϋ. WαIL΅nVˆj£i‹Ί\Α-Œ+T?ZδζyƒΥψx!°Ρ ‡ς…ϊΌnΏ‘Υκ“Ω!ΎCB)PYΧ‰©Mπa†„©˜ ΓψU47Ÿaι žΊ‘.Ρ"΅Λ/ €n$Ή †ΤEc8^Ξ1Η*7,η«€£Ν±(¬κ)n£ͺ^Š*P./n–π#§Ώ«ΦdΌŽg^Α5ΒMΧΕΟψa ΑΨKΏsyΪHƒxΡ–$«Q‡€$œάtotφc―ši<Έ«Y†!·ΏͺωωΥΰDΧh?•‹Κ0¨oΈ!Š³λ+†¨©κΟ'δj…N#ύαϋnλ—”Ήξ‡Wύs2š¨bm3Ub`qf”–ζXόλ½``”gcS―ˆ$ΤXG(:²°‰χ‘~€ΐΌ¬Έ°―ό‡χιWΜ―R Χe½ͺ5ψΩ”ΘΜ—Ι¨gρzι{MΗ° ΄ΏŒζ`ο :“Λr©`χb“Α‰ψωΣ‘ηkVƒ4ϊH'Β°U+*Qκ·xΪυšQ±NͺUL{X4~]œLΦυζ…Ξ Π{Θπx!gΕζιž%―7Šγ@$Ο{αr«ϋΕ@ƒ’ύJμjŠν¬ΰχ τ²Χ]•\δU1Η’ΎΊu°Ό³3P"CάΛιή–ε[mLν€έΏ^ΌΈ©ΤΡώ2•Τ ΝBA9»Ϊ'μ{AŠΟΘ6λΠl–΄K4π’£³’u§lΧJΗB£:ͺu’2Φ‘χςα=lτr³Y˜m –l˜ζ‘g ΰ–Ÿ‚Rp<΅vΤ²‡ͺ€šίζΫ‡`•ͺzK=?΅^€Ά%yϊΞΰ…Μ+έ@"ΊP>ΈύιύΡ@Ηχήμ{—P ƒ)Θ A™ΚTΏ!·pwχŠ‘K6Rsί•G”ˆϋMLχϊZ&I…nZDηΆ…Ω‡­ $π^V@Κ^Uϊ μSΗ™g…”šΊ qΊƒΏ£¬™#ω[BΖtΒ s™ΉΪξά,ݟ8Ό/_y°χe kόΘί #³—$»β3­ξ|œ˜’ξœ/0ΘR1‰Rα ·Fu3τ³š •νY“zΤr‚σΙ)G lœμ5Y»IλŒlύ˜ΣΈ£±γRfiΖΌpΰς@ο@ΚΥ),ΜχΖ₯ΡjΤ‹ΐtτ’²γ°ίL,ΠAάΡφ+±Q?Ύ?3‡&v~ 4Ό·ŠΑ’ϊV΄ΜγuΧ”~ocˆ•œiΖζͺͺ„ˆ­ΛΞ"p•€ζŸIωr'qΤΔΈ(Ϋx3h·˜*Ιχχ?ΰ,ΈO9ž²QY> Oxj¬Aφ°h&¨2MŸŠύ&;δwEk«ΎΗ†-Ώš4•Šev–:Aa«&Z‡τ3&΅ƒ(ΧYβ<#€Ό‡Έ,!ΩΚSe,žΦR|υΫΚϊ(„mΤSΓγ3ΣZ=Λω›Mρ‘Ό\c+)=ΜFX­Ή*2ςΜ€&΄t²N Α&mdq?θ½mΆΈμj$GΠT†η–ώ!ցλoη§P힜§ξΥ{98Ϊλΰ‰uŽΉD†εέd*…=‘Νw<|Dže«SyΛχε8ϟ1-?±ŸΜ αyΡήίgŠϊ₯UΧ—•:ΐDDQKHΕ¨R²ΐ~cαΓ ΆλΝm’$Ρ/Z«GΚsOB±KM‰‹Β,’GtΞόWΫ–Iΰ}|΅/&)ωAͺχφŽƒ§‚N4”šΰΰΐ ³<ΙΣΣμ“ Eν‰`7•KF™ο`ΩwœγE‡―Zt€΅‹ δη²–<U9ΒΦχ+ΎžΧ*„’€ηγ6 Φ‘"Ή—k·F­v!?^DF§ζ0>šNΠkχFειτίρή»ΓU[ΞT£BNJHr5αΑ(ι/]|Φ75χrGεV3(ΑH’Β€πœΛντ’η_1)’ž„Ι _ͺς Nc—Ύ‹φΊφ―Tž8Ÿ?w3;E,9΅ πήH'œ¨°ΤΧeAοS 0šΤ6,Jπ먬[½‡5ΣΆiNτsΧΣΒ·«Tu―Ρ d?πHό.C›Ϊώ-b[M°Ξͺώ‘§Τ$“―;auΏ,ŽσUT&“ŽιΎ!έOšŠ:ƒ R7Qύγˆχ‹Φ >:sΗΦ*m—δΌωΠοΗ»`š`ΈΧ„άs…4…KUλ&jΤΥ•’Γ.‘η·2*L5C– νΪϊs—!‹\ΨvG"{ΰ₯Ψ!€qŒ΅ 4&tQιαΜκ— Ρlκ²κIj¦kŠx°ζ³šfψ ½+trΎw^dr α½Tδηδl*Ϊng€)d)6ζΦsχ ίΖXΒ5ψVρ4a'| +i¬Α»)œF€†e#y·kφβflΙ³1Λ@mΚΠμαω;‘1I2^zkLFή’@’ΘλVΣic\:Žρ·kYHθ“±:Z½ή½g§΅αε‡ΓkIvΉ£ω–ω½wΉ^”€\4'UΧΆ?s’Ÿ?€Έ½ΌŸΆ2λζ§e!@£RΏξψv)ξRH ύΩ?€2Ω5)φ)Ρυ:ΆCrlσgΣΔƒδ¬εˆ{š]βf–@ά–KΘy-kE°CΡ[J16I`ϋΘ‚3>ƒ’υAεΏ0ͺλnΦ₯§@ϊηη4ΙBHGΊΘ2΄"ίI9 )ψ€>½υ˜ΰfΧ!ΐc8_ϋ(†cΘιͺvXš)D―ZQνΥ0Iφ˜,Hζ΅<‘SyƒίμM›hh“Ϋk"ς·η²b;²bNK6Ψϋ4§KΫΘ53@2ή.²0ΣΊ•τI'²Ζι$ύ8#Ρυ’χΫΙ–!Τ¬^1ΓŒhΠΜϋύmίJα΅S―ifΤ1Sn41 „—|C”€“ŽΊΈŽQ€Βαa};΄νa^6ΤBˆ›ΧΦ4>ϊe,0Γ’ ‹e£Hώ ω ₯ΦΡΑ!—x‰Oει4[_‰ΎyΡσI1>6Œηƒζ-Κζ*/#ˆΣό#c-„§5S3AςΒ©j—!ζθ Χ°BΣΥ) ΅…:Τ^ΕNEΠίYA§IίΦΩ+‡ΗΊHΌY„­swϋht†Y…Šœΰ—rΚΛ4V8SΉ9lμ@έ.YςeAj"!α»=y‚?VΗλ¦ζύβH"=έϋσ,‘\‹Γ&KQyTρ£Έ=—EA¨0Νa“©;ΆΒ*π4ϊiύγ4m["c ύrƒDϋͺ€ςaΩeάψwJ/@"εΐKRο¬ΗaΪΉΎ……j›Clψ©= ͺΩKΓE{T,y5X[σM)ζ[KSrδ5+lί?­=†’!„«oέ{? #<ŒΘm„…ξΫθ„)ˆL?E†zVLŠSt€εΎηΙφ³ι„Π”+πPYξϊni½°‘9·Θƒ‰ΦƒwŒYI*αvdη`½Ο\œ›‡”›‚ γCKvE‚"59Β1s.ΙόžΗ?S? ₯ ·ΛMΊCψ·Εt·Q=oΜ‡θ/ξ<6B6 Χ[pee|$ˆ2i·ΙεMΘ-zXiΡ:—i“l΄θ wG£U¨=ί] Ljh!’ =ٟxΗ”ƒ”i˜\EψrΞNφοP½_MΝτkr«RJΣ;|/q±pUUας~΄ΨfΩ.ύ νΤ#`Τ Kq5ΑΤΏŒ|εσ$`_jƒmKκHπh?%'oϊe——€b/@}Ο ƒ Ύ΄κάo*ˆ?ΖΙδ›IF©Ρa΅ΕOiε΅]‡SVY1ͺDuϊdQ/† ;£οwΊΞΫ…!˜*₯­Έΐ₯ 4ϊίσΠυΪ4= !Y‘Γ_PΨ:6jΥ|VDtYί5ϋX.ͺ­†B°sΠ γυ‹hβLl‚bΎφqρ9“Ύ1ΒO]jΗH…ρύcϊέp‚vΓ¨€H"ζ7P,gyy›¨κHήΓ˜ {g@Ό½ΙWPV8­Kέ1ΖεnjF~ŽΛΓ ί/ή.΅fc1σ­| f= ΦnaχΕ™Χi#NΙ3}zudΔR›#!bβ1Q¬lΧα(‚ΜYš θοn—h°η{ΪτΟσtSzΝ{©ή/α¦Z%“žBfέz»ρ_θ”rΉ₯²j>[ZA~eGU•8mbο±I‘r3u„‹3Kwkτ-"λ·~²?ΤnΑOλM!m·RζσϋβζΈWϋ”}’(Π ΟFίβζ`&0G1%3]]YmΒ šH΅­«ρ0slΡ&ΩYι7ςπν‹²vλd†p*-€2~—'@Š Φ€O4‘J)ΕΠPσυg<ψOΤsδ°³%xΑ­ΥjN’ΕQΐa`3 aο G£β[ŒPy’Ω|ΡMŒόn·₯…ͺ˜o”OεC-yi£R8˜…tΖ pι`d=ڜΎΡ|ϋβHΗw$&Ÿ*ίΎυ>ΟΪHdΓ΄B\εbήkΤχΫ?ό%΅:€`ο©(v#mDTo‚WiYƒ6Ξά½7Xτ ₯«w„υ²M}™š μ# Ή‘@ ¬D–{©oψ€=2έ 6Λω<’Ώ'‘₯ΎΒλαŒ+ίΩƒt/0c\wŠ]Ν)z\Μ[¦i«T7υ―Ή»!δD_Κ‡ ž]ΰfΈžŸ°“>š;ˆ―ς?ζvώ¨DΨε Ÿ±lΦ‘ˆχι7νπ‘ωWή# ε7ΕΆw¬Eu(;π6aƒϋΆϊˆίτ?£Jt“Α!ΑVͺTIqδi §¨ZgΉ˜w¬ "₯Ε»=Κ χq7aƒ9ΨYα£};β»M^Χ@Μ#)Κ-_E#ŸΖr΄'μYχ’€~ _ω/kΛM:²ΔΥ’ˆ―kζΐ©ΊzΛΥYΚΔ d*ΓmBA?R]Δ‘Κ A^Tύ …H―°ϋ΄―7kΞz`‚±υ¦`έΊ-δΘΓX¦  ΊMœ ³κΫ»ι'λ ¦8¬r°-Κ\ςœ‡’zςΠ΄|HU²7”©ΉV#§«Ρ.} ŽΌσ­ΧLNΉ›a7ζ€εθμγΧο /jh*`Θ”Ž~<™1IE‹,ŠΫ±Δ)a‡ZΛΞα+ElPE―ͺΝZΊπ=xσLΟ,{Ί²‚ ­-ΚQΉ†ΣΚΚ—”Ψ―¦MΟ?ΖI“”τ^<e‡ωηέ<%‰Υc.¬πbΛ0]β:έ²*ρ+ИΈρO”aε5½Τ*Ξ>žO8zXο?­`'qsŠSα @V§ψ@Œ“ϊύ7£€ +γ'sο)³ο|šυ5n‘PiT-©Cšl)>ž$ΩFŸQ•^œA()Ϊ»‚2·]3 ‘ΒEŽZΊΝ€-a‘μƁͺ½8„*·"~Ρ{EœΡΗ\l*)Μx[’-νβ”b+ϋξW;ΨKiΔ―Βϋ7o^˜F*Ϋ₯ ϊlVχN¦ˆO/έΜ¬τ·H 3&Η‹2A”KWx3ύμ3“gqΏ~-‹7sp)t»•’‚όL»3ΗQ«H‰’¬|ύΧvxτDoPΓJ ΪOLσ‘Tˏ'90χͺuKœ”­~Q_ωπ˜)‘4“¬”ΟL]8³u€tϋžfϊ$‘xΞV2³Ž~ΜηQ]ώ{7bwp>jhέΩ6,ŒΠΞώb wgΌξi‘λZƒS‹YhߟŸϊ²i y›„ψĘξΰΩc žWόΠ‰=ώς'ήΘ‰H“VQΰ<ΰ nΡΘvOπa•΄E ρΚ‡—=JLjβΏ>Ρ/ŒΓΐ46’U5FNΖdγ ZιΕί‘ σΏp:tΚ$Ίq8¦†ΨI vM„yu·¨Ψy)Μ,ο,&ƒ,Βύ@‰ν2r²‘4Oνc07•8ίπΉΩ]37SΛ¦άUξ@~Έ³ετiΡtbΥE± >θέύΈrb?+ͺπ‰y#€y@k5 χ •;Κν€>&vεΔϋ$Ž`–§6^¨Buπ€ΠΙoπ"h©8]“²”λ«sYΏwΉqNϋ”s„m"L[ο‡ζIλEεwq–Q&!`M«΅’φ³i0°¬ƒ0βDΎ<ΐ(dΐgΉ%‘Δ|~ηϋE ΕθrΉ΅_%n'à ɜ§ήRˆ*‰#Εων qBι|¨½X,½¦’|X`ψή­©ιsΜY†ρ7ξ6'δ.y·υΜμθ£ϋKmνΫ,χυΦΕΓΡ“‘ΝΉ;zΧIQXΩΐοβ«6\Ε™ξ/ŠX£Ώ\­Χ\Œαΰ|rυΤΙmbH ρΪνβe§„W7&Β2:%:HBjσΤUm˜ ζ}ς$»ƒ##.Υ·Φ6’²PγΤ!6DˆNΣςLVD ‹57Γ±;yjM₯$-`ƒ)5„ͺٚ –·Ω‰©κ“Τ{„‡%…φΉ*£;Ο2žξDΤς{|A–Π–n΅F‹ae=>ž‘~p@ύΣ Eό&zΪ}9Oi΅ΎwΦβ7Ÿ:~τyG _ŠrεΚb¬\ό«ρΫέ"½OZv@ΔΥ¬ΈŽ8|HC ΐ.»Τ|U‚…,…|br8«:½»’4pœͺ’G&1έπ#i‘Έπ³&Όš5žΝΛιULφ“( ίMΘΣΔ»>₯)Δ]ž|SΊη«‘Ή¨ν-z£”κbή ¦γν9R3Žjφρ% ₯°ΰŸ<w ΕΎvtLΑΝΎξ£™₯R 9ωLφ3 ΄1S„Κj7χΪd‹H§­πUr$ΫΝϋ0{“'±>Ψ3ؚϋηχ‘Ψέ_Ωήυ«ž»y“.Š±έ)T²CΕ–°t€ώmιt Κ1υϋm }°v qmώ́VΜ‹ͺY'ŸΝΐžΉ„ž6J ΌίGΐ¨8Γ‘ ›υ"ωOfΏ›—buε!φΞΚ—vΩ7ƒŽk φΦׁo―ξ‚έT4/&K씝«Šψ›@K‹…; —έϊ˜ ψMG~y9ϋJΗ«Ιr―ŠΙ΅Ja«8<~(ή8|σΆψ«Υ]! WC|ΌA¨'§(Έ$›δΎ ξδΘΏ(ε¨c]ύζF…sΥ'ΪMUGΖ;b εxޝ’Πα*£ŸCJb›( Ω)ŸδθξΆDOKZx?$„Έ]υuΈOΗΥ6rήό νΏg?ϊ1†ΦΙx*’: ζRάϊά£:‹€ρ»"ΙpΝ*€Τ}šΦ:Œ’ŒwωΚ(όΛuXΘΛ2Τξ‘ίƒ3:–ӝJyPDλΊιμRΞΌfq?{ΤيΪF½’qeGΉ*‹Β}U.Ϋ‹.ή-Όv8Β™’ΧΩ³I&‡θΠvθcl4—hmbΞ¨ϋZ½ΪΏ^ΘΎ’ΗέΨΰΨxk”·φκΓI"ώtšηfœΖ_Νp―t‡fΝmmF³cσ¨‹SΒLΔϊΙGd-“bόΉΆ­‰yCΈeϊο3Wΐ4TΪΗψ–ΎΗ ηoΫPŠaΉΔφ V„"Kd©V πi•;yυ'Ÿ!ν£·1«Ω<.Q3ΫX*\ߐ»Ό}H:K›=Fgώ— θ’ Όφƒ­Γν>coI΄9‹Υΐ)sY,ώ˜΅RΞ/ΈVgN–¬ζ”Κ΄₯έUz +·LAl…ΘΨm9‘ϋ€ζo;"fΛGξ+]vGŒΗ 7λ >XL‹ΎͺNΪFAaΕlΆQ- ρήυ€χΤZΡ¬³š χΤC φύQ„ΨκΔβ{¬IέςNOΒύΔL#’ξ|'CΈmVϋ|ϋy,όž~uΤDœΣ5©(WTV”Λ‹₯Š+ʌCœ…UUSQ£7-£‡έ7|]|—€Gζ4¦Lnξ?²νφU1 MγΚςŽ ¦7χyΟηΰώ-Nό•Ξ1ΐ˜Wmž°nθˆvoΦUj>evԜϋX½ΰUςb«τ―φYΐKA# 7&•‰σ{¦ψ°ΈΈ€ΈΎW»5x"Θpj’ΆU\CΧτo|…Ϊη¦ψ‘γΓ,ζvf‰Χύ4δ°qY±ύδ:7˜λ„πΒ0υJΣzwzξ­ϊηh’;%”R£S؟€do…ΙaυUΪ]ωŠ¨δ:αRH\‚iθzρ ϋ”4y_P$ΐI ή΄.4)b§πߞm…a{‰SΠ‘` ήΜΎr‡WώArαr‚Ωoϋ>+γψμd=u‡?2PR9ζυRdmΎPvΓ°™\(¬D$TΝfI₯%ψy’WbΧΫλ&,+ή ΈS~œ§ ½ŽΞmB<£L,~™!PnωΑΆCΒ―ohLCI}επ’Ξί^LIΏΩς£3)}Ÿ=·ά$ΒZ+(~ς\N'ubˆεzGΞάO-Πβ{ΗΔΪκ’ΐ#P#ΊΏΩ>μu9ρΌΛ˜7}’έ­@!„αα*‘AN·π·ΌΫo‰QYΠ1μA{£{Ύ*ucό·3ԏVT9»Bωεݝ”jϊ“΄)'…²Zf/μκΒ1Xh¬_X›,½»]/ΞΣΉΟΚ6-MΏGΊM­Y–½&BcΘ!.ύ+š]LτσHœΰy›»‚KI΄Fƒ'ΡΊ@μ΄zΕΎ­Υτ'"EΝ0εUfŒ,@g/:›$+ω_%W€3£IͺΠγΥ§λ/G_™sώcπΏTη„tiή‘L’‚šuCjžžQrΤr(ΡΆήΗ ΝΦ/…hγJ~ν26rmH“ WU0σψ Ό„7‚­„l3‚φβHFGƒT-ݟgλ_e±Yό[‹<)ή³§ζnh€²u{€Δ†/·ήΆΐ±‰UΣ V€.τ ΐ &ͺ}bhŠX΅FΥ½,œΣ­Šί΅;@Œ’ λ«ΛγcD₯Θ YIΑίͺΖΩaγο4‘ΊJyΩ–φ@U;XOδήΒΕ) ίVΗiφHX„γkQc…―@›ΗίZ5U²DGβ‘Ir™6Ην£fU·•E½wH²ŠVZήΕ%šιΩXΆ~E!š<•=[n7tΗPyώ“|Σ€¨ŸŽ?A€φβθζ@γΝΈςqbKW©>Sž£D¨PUΌΑ;½c>qδœ•N ό·κBεχρ 1 ˆr…– y‘f°LW g2ΰ­ΔmΡH°Oc Ÿ™ο'c6κˆ>όDŸ)Ά·‘sWΏnρe•o―'Δ0(i —\κ#ͺœb—Ά<ˆV³άΎu5Ή…4a((ˆG{0΄Ξ(½cΎ>½Όδ z¦;%jόwG£ s¦qΕ,[χ`{’ρ x&'±α’ »OΕ\Θ>a:= z €ΚyMT[ɞy‹GΔηmZuf|ZMΣΜ‘#t2½ Qƒ_»FΔΖ€β &'? b9)‘€λ³dtQή’|ΥqΏ€€άΤ’T8Ζڌ}Ψ΄±ƒI,Ωͺ{%‡^xΤ+JanΫ&σ‘Αž'ή!?ίyδύD’zΉqέngfOΖ›¬yKΫ΄±Bs»+έΚ€D¬ ΧNŒ0=zαΦeθ ^•{£ψ §§‰ϊ’αΛƒ0ˆΝnΤKρ²Ώ mFΝ»sŠϋ&x–έ aKrŽEξ‰#D4βfž²²‡šΦσFιBσ\ΨΪ‹Ό kl$KPνcαεάY?ϊ­ˆΩžMθ†ΈΤ>ΛEd―HeMάh6Š™v…ρ`QjV­Ή΄iσΌŽΰ©Ϊ!#ΗΑ{ΏUγ£X’€zΎ¦Š2·υf¨²ω$&§ˆΈ4°όθd­aηΠΔ'Ω*J#ηdΕk‹e9W„9έΔ=AΜ/lΑš Εaσ]*³₯YϋIέCσ²zγrtdΦ ΈδχΨoŸ$ηλ5σ7ϋ›³K σu53ύsVp?wn<ά~°JRμϊ“Ίf’xΒ žO'ύ+fζΌF'‘DΔI­―Μ?ӈΠO cΎν#Φp+ ˆΈ[@(³ΊΖ_7ο¬O„—/Et ngφ¬,“+/ιΆ3QτΠ 0a,'‚1ένDЍˆΟqgž΄d`fb.˚Ό"£ΩIL£]?sΟύˆΘaςgj―™Q‰7›Zχέ7}ΟvJ1Μ\(›$φw[_eΚk]ŸνSm4UΎK₯όΤ&cχ…+¦Ί΅σ62¨ƒ›ϊ=Ί>]έXΰr…ZΒρW²ΈΜΑΰ…EmξVνψε?-Μ„r[θZ™h"ϋ£[;Δ¦* rύZͺyΰbσ@>ZΘύ)Dxθ4š1HŒŸ™”Dϋ;O^ΪΛ‰ͺkη;X€ύ­U—q²ΞM“3χjγŸ&xL|Ξ’INΧdΘ6PΝ͐φTpa%φ‡,Ξ'½2]Ογ=χΈόυβLd[4~{‹‰ΜkFmŒM߈/·}E₯Ρm1ρυ’ˆ Ήƒ{KzΆΓλBšΫxkΰ6<+ζ”/4ΞL Y… —2‘•³Μ›y_uΠ2εθfOώ¦ΎaποΧ,Α] Λ6)έNw$όϊ˜ hZwR πRν;‚ν%§&Ί“ηnέ Hu©ΧKa΄ΑB–¦;1tBqΞj%@ε³!#Hπ ςΔFm :δείOϊδΔj±7Ίfθ Οτ™ͺ—’hρΖߍυš4Φ]œ{l ΝΩΥS:z―‹Iάdžύˍ|…]šͺiΌK†Ώ3cγw«›†˜ήώF|Iϋ‹gT±2’Ε^(JZ£ΞΐaJΗ?4aΌ·΅oEφ‰ΏjΊ0Λ²ςe s~B²c'„ y (œΗ3h΄Ο›f ΕΠΡό:Ϋ ‡¨ Μ΅G)—Σ ž’!†β£+XdL’όΊξ!+c;τύI5ν8ί#£ψα_»vA;4Λ±L#)ΆyΕ•4Ό…hŠ‡›=_"Ξa4JσBΜ^γc ΅εEe9˜άόYzhΆi•Ψ?8ΝfHΧc ¨Ž‰Št»σŸΰ˜ώ βξl–ύ—έMœkQŠ₯-eΘ±­AŸ§ς¦λ‰ eZ¦²ž»7^z‡ύΛή‚%Nσγ3,Tψ†•EτιφLŽ€oΏ@ΰŠΥΩγeκ†]uΏκΫN ž#%αD.+O ί KΡ_+“Φ<.Υ퐂H LIbǞψtꨋ:όbtηνQ‹.fdρϋΥ‰ϋ«|š³ ½²κλ•ΗΒj,.σv‘Ζxΰ·φ†ώt₯yCΉ5TY9ŒV­r?ŒςΛS1υ†&ZΛΨVͺdΝΦ6ε>ϋ=(i3ώή“1Τƒ˜O"m λ•%E Žž6WW†)UώΒwZBΞNτŠj<ώΩΕ‘n]ή…ΧacΧ(Υ-σZ^K±’tΉf½|)[¬qξΖάΐi)€‰ςP±?N[ƒCΙ5/sςgΞg-ΌfdYEp/N'2u* (€ΌΓ-hνυΎ€HΗθΪM€ΰƒ”hen0ΙθAgΟώΞ0₯ΆŽσ»ϊ,©Θ°…φ,g’yM$ΉRšˆ_sπΘΨ/οC $δ QΫͺ&»>ŽnΎ"«ΟN€—Fπ³ρΪR 6xd(rβtcšZώwafŠ"zνXίπΆΔŒ2 "ΠΘ:Ο#PΠ³$ε)0¬Ξ?9IGou£9nϋF{Αΰε#2NŸhιι‹.gW΅'Ή_AžDœα«©χυ²½…†„W«cοΚ]_FgΑβρ— .ΦGΗ£ΉΌΙ8†ρη_‰O}©αλL‰³V0΅Άqœβ`0>G„ρ lDŸΒ“Μ©τ=ΒEhžΉ<ž§^Q§θ9U}΄ΞΡ€ίT`m£ΣέΠ8Šυ6=žŸΟψ-t‘Ο”Ρ¬?˜¦mΚ]6¨F›Έ!ېΊπŽ<"Δ5υͺcEL'[ΰΦ‰QΙ­wΞΆhΞΐ]_α4ͺY―φ>Π*‡ιˆΡhΰρXF’A}zΠbr·ΟdC»΅_2σ‰“8ή0iR¬»΄ ŚM‚FυN~5sX£¬L£λM₯–M…3‘•U ILψΎ³qΎ0υτ…v§ςk°]΅g―9;z2ŽJ…¦ΓUΊ³Γ 5γ₯Ί‘Ω]ήW!$ώΟνAφ_1―jϊ’σς±Σ'Κ§ΟΫ­PMσoN% €₯…ΧΒ.$Z%!₯Αψhΐε8,σ¬mήhsΓψD… /”ہΧΐ―±Φ εΓν¨τp½šg“{Λ~Δτ‘ρš ¦F%Ν‡zΎ΅v,œη°iξ7Krέy˜Uφ’΄s¦œΖΨ@Wθ€N]h)†ΕώΑΠ4“DŠmύ„‚4ΦZΨς6fνž k‹υΡΙD&ά ˆΊ“­Έ$Z¨”r|¬Ζ4W8!M›>Nƒ²¨ΏβΨττ2 X1ΌΈβŠΝ¬―UαΦχY8OχΤ•2δrν…Φ5΅lb‡8 7*Z½βd°Y>@;«`―•>OLZ`ύ±…Β؍Ajοΰ<ž=)εόθΉύΨsš»-DΪ2‘Έ>'D€γŽ€JŠh<ίΙ^˜ 7 β‰lI°Όμi…ƒΚΏ3νRKNΪ½ν«ƒ—ͺ{ˆ~P—Ή΄»ϊοhǎ1Ÿέ[!b±±:'n±7q‘)BmMΝΙEnΌ%ς#tξPw²HΜ$ςXΉΆeΤά€— εΈΩ4ΌA)³€ŠœΛyJλχ©Φ˜φ[ωΏ]ίAχFFΡxΑe:’{Β·[±pόΆmτΪQu~A³Λω}'ŠœaS3Ϊμ4Δτ²[Λ΄ƒ%)™" Z€ΌŒΨςρ Α+9Ρηm„Οθ1ΡQ¨_Ž0ΐΖή΄ά?ΖΦU vōÙ?i yΜmIφξ%v°c}(Φ©l•ο«C¦)4Cϊλ.»‘Ρ>gB³ εο^γϋ’dBά‘0Ιrc™>ύηΟ Η?Žg[5Θ±Χ±>SΔu~δ'ήΣ9V:fΘζ9#;#ίf$΄:JΚΝ–ΥhrΟΛξ©ΕΌN©Z„Ρ±αUΛ“z‹Ξ4ό%υ FκΠq *Γ‘°pXΎ:UϊΎν½+<,σββ μxK―ŒΤ―ˆ¨fmaΰQΪΰŽU>Σ7ΎΜJ ΜΩoγ₯Χ5όΑDμό!$τωl^eτο,ρΊqaΨ‚>ηŒD™"]={7y>ξ“μaχΉs†§-ο!ΎNSΌ=mB’‰ΐς„φΊ)λOΊΩϋ„1•W-»ίŽιβ©ΊƒTΩόRέ}ηmu\ |φͺVcx΅¨"F2-ωϋ$  —““ ­₯0Ηΰυ=γ ‰a8ρP―ΐbΨ$MKή±\ΆP¦‚i"ΊωόoΡΝ$Ώ~ ]ί£Œ’ eTŽuΰ`kŸabΣυΡVnBηV™+\½Ώγ „‚?DBΦΙ±,4&6ρπ \΄₯Η„.F`Gβ5k oΰpπ ΐ1ΐGΝό‘­ΜΜiφΏ|†)ν‰RIFdEWƒΡΩƒH†΄ͺŽG:ω=Q `«A„:Z:εW‰r<«£Ζο!ι:qnQjΚ™²•—5)ΉΫ !τΈR?†0V†ΡŽ»ψκ,ύΒ’@$鐷sΌ’ΊcMŸzƒOBΟv™H—›_*ήJo5¬ΕŸM!– RfLΌέojl„ύΕ΅)1kGΚTΌύΚΏQ _Φ~ΕΌcKΐί§Ώ#WΕ0εγ»ΚόΊψ£ύU‰ηΦ/ͺŠXt΅€δλ+«ΡΟ{U,ΖQgΉΒ C­³ύΖhΔ9ΏtΧ3ΥΝP<\*Οε™|h ¬Β―Γ=­€πθ똚ξΞ‡hΞ§±μœ달ό Ίμ·θœ‡%Œθ.”-[Mi­1Αš7)TΚϊ΅Ω Τ#§8ΨΨœrϋ“%όMζc/o x¬5͞¨”Φ=FœΊG·Ά`p—Οfυ;€δbžΏο9LΐeΝTyε^Ήν+F°\+γOΐΨ6H§Δ©ξe–˜sΦΥΞxKΏώ:ηΙjυˆ­9ΗήΖ3XΜ―YώΕΪς€Ι:l«ΡγΝΔ#ΠJQΈμ“G ιΊ‘Ο]¦ι`₯—d >‘αBp.a"BωZahΙ؎ˆΪΩλφ κ†!χ”%qUφ6β!†ζ[œ/ΛΪ”ΒπsOνk)\-ŸψΉΌ žΞAΖr‹Ζ>ˆζ;υ˜ΝhήΑ¨+‰Y;€'ΧxθdΊϋe!ώ{φιeΫ endstream endobj 23 0 obj << /Length1 1786 /Length2 46472 /Length3 0 /Length 47128 /Filter /FlateDecode >> stream xڜΈP$k“χ‹»; ΐΰξξξξ.»» ξξξξξξ 0Έ ΫΗywχ=gχξ7¨_Vf>™Y§’ΊΙˆUθ„L퍁βφv.tLτŒάU ¦„§PΡΘhccD§ 4w΅1r0Σ³12Α‘‘©ZΊΨo^pdκ@'gK{;ξϋ‹8\>-’F.…ΉrFžFN737€™‘‰σΏ흸VF&Φφpd"φžN–ζ.ά€Ÿ~ϊ2²Σύ0³wϊ«ΐ_U€.@»ΏΦu{„ι[΄τ_ωܝ­-iτrτy{χOΛ']F6τJ{;€1ΠΒΘΖ `oφ―œj*bΚ*-‡„²‚š’ =@έθbgd tL,ŒœŒL\>Έ?‹025šώ΅Ύ€‘@Υψω5²³ψ¬ε_±Ρq~vςW“Ÿ^ξ–.€Ώf.nοdPZΈΈ8p30˜}ZΜώ²Π¨>#%€v@§E|ζύΫ3!# ΣθlFotaψ &³3±·΅ΪΉ8Γ11L-M\>5·΄ƒcψ+•ͺ§ΐ0šύΛΉ8Yztι™Œ}ώ}¦χY·©½ηίξςŸS0¨((iΚkΡό_Dσogaa{€7'€Ž•‰ ΐΔΖΔ `bώ<ψώχ̊F–YγίΡRvfφŸzωL]ώ« ·€ς_b£όχTςφ.–&Ÿ³ύ¬žWΕθ2²1ώο’ωO‘|:0ύΡΚlAάΥΖζ_s¦όίόπ7²΅΄ρόί"ώ§§π_‘όΏ%Rωη=r1²±4²3·ω·ΙYάhͺhιbb03²ωμρ_v5;S “₯PΡήΩς―} cbdόΧT-,M¬ν€ΞΞΆΌdδlς)S3ϋΏhgϊ? ³3±7΅΄30³±ŒœœŒ<α?ΕΛΜΖπfX~fφψΌŸ΅0ΠΫΩ»|†\]|ΊΟp †ΐ ό—ι?ˆΐ ϊ7qΔώMŸž²€AξoϊŒ“7q2&fƒςίΔςΉ5ώ&VƒκίΔ`Pϋ7q}`τ7}`ό7}VfςoϊkŽ ¦@&πψY€ΩίΘόkfωΛŸ™?K²ψ7²~–dαι`΄ϋ‡Η§νŸρŸ•Ωό?ΣΫώŸ†Δ~ξUϋΰgiΐΟ΅ώŸ 9?o€Λ?πs>hλs!ΰgf―α“‹‰«“Σ§žώυΌψΤ±™ε§€@  άΪ²½ O°Ucpχc½ΠwΊΓ–!v΅ϋW"VΈΠ}z°AyλŠ/¬wœ)9’ŠSό΅:ΰrΰ ‘ρ_Δb8ά­Ή„{θ‘ΡΠCΠΐ$#πΞι›xεΉa(ηn‡π’qςNEƒJ £δι­τΠ璜8AOQέ:|I§t¦c7ΆF}4…ΔΩΥ„)dͺ΅ή_<•σ0›³:£kuK^AaίκζΙbΕ;Q'!τt·ϋ;§eκWΪ–—|םΏ‹n]!ξFΛJΓηŸ tVΤzΫΆS'έτtό\΄z_ΘΨψΤXζˆ6\5©©1)°s³¦‚N‡zTCΪ€ˆn41C^1ΕΈ?Oώ|π;ŽS~΅βα^-ξͺ™­΅ΖγA·’GΘΌΕžΈ~kXql»d")ςe)3XΫc“cˆŸ@Œ£aξ«?£ΒΟ“wφnΡsά―"~ŒνΊ•HA˜ \ΒtΈ"ΰλ}x#“έιΚ5ΌH,ϊΆj¬θ›Ρ–Y rθ•΄Ι+ Ϋht ΜJ$ώ`Φδ/‚ͺvS #φZbώBη"}-nΣΗvςN€Σ±'7'oξΘΉψ‘“ΏκΨΒ’*θlξa`T±uΩX’uΔ‘+2ΛVJ„Χυπ~}&μ•Ι©²€ABτ6 ‘“SΛBυ%sw~4:‘Œ·ΥΎ΅ρ€1…Π>*Y˜ώ¨`²‘b{ΤW ]LIV»ΥŸM„Α›#1—v„§όΖbΣ΄Γkz"'ν‘Ϊ>’ό‹±^Κώ¬σ^kΞ€έ‰Α|ρF₯Rσ¦NjtB!pΒΡI€#Δ£'O/Sκ‰nΨoΓ1|Fড¨Ω1L‰ΤΨχ³ά?“οu’”°PΈxvHP «Z ΛUoW ΖσΝe]ΎœA“ΠήUGq%5ϋexƒ`BAδ{s_Yιτ‘ΪͺŠι¦τu λI0,VF ΙΞΰDύK`Η»—b‚#Υ\qΘθ‚b‹“α-aIΫ8> ‘ΎΜ[ υ$ω¨7?‰έ⨟8˜™αΩ–·vSZ­gΪPάμ-–?g›/λ‘MHΖΧ›œLΒώέ«aΥ»ή’ωqn}[>ΒΡ©χ˜ŸϋΐωΟλύ4­5Χ‡G)qπ'ΔΨ2Ώ ‘„/τ6aΜ½Mώ>f8Qβμ»[5ŠMο“Οwœι _WyμHe.œϋΏEνJ?]*ZιΔ…>υTΈa™)±¬qΗέ] ΠΥΞPyqXΓ!pΨQ4³·ίwΊHƒΐ¬€B3|}hLŒœ(Ώwq!žίž‘ηΗ;ύBσpzƒεόuRhψηVsuγžΖ° •(‡iP’ŽFB―©b7ήZ lͺΚΐιοE0xΟ k‚CϊχS^ "Ό•φuœ²Ϊ$%™Z.šΑΕw2ά~-Ξφ¦Όυρ¨ιέ™v}χcό#©PίΫΝ’kƒαΪΙmIdβ‰Τ>jΌŠ΄ΨΚΔO—Ι~» f"i&ΧqzυB!άΌƒθβ#=d d«ς `qX₯ω‹½aJrA)Ήk°7GβΣ’t4Τ„λ–Ί­Ψ†œw€v’΅;/‘Άs+}ŸΩr’κzTϋΡcώ­₯V"P¨KPΉoœΡxCVnΞΓνΪ― £[/–›ΥΫΧJ¬δ«>.Σ―ε²·;)μT2TPιάH@Fϊΐˆο?rB’ŸΥJMϋ \§`YvJΪέΨή‰\yD寐²Nr­β›?w09ZξZ+ 0!T7 ‰@rC3r’4Θλ„³3U`$}`Π0Β⁞Ιcνœ”<ξŽM“|ΨO|Ψ=˜Ιλ©8]+ίg·ηTtΌΔ—2YŽ0žΝίκ@ΏΐίΰΦe§“%ΑΎΰ_ν&GΤ$­*l…+H š½Oέ$§PhβF†ΔN}OΣ›O\I―-[© |…l4ΨΚNJ0}yˆΆŽ,\Ϊsκ%Έ‘γMDœΆΞPο4±υ­„!Gώ=` μθ83ΔSj—ΥΠΈnέLΠβκ'xYΈi~u@\t£=±¨ΡόŠiΟ{#ΘκφJΝς:tΐYΰkq[…Σ{ΚE^™QxUηΫρ¦ D!K3α–¨ΑΨbM•žσ—ωzΝ‚χ!f“ΓV=ΰoOž~©†πŒεŠςΗΠAxί™ΠZΊ›·QσΑφέΩΈ5vqg9ΖQ‘ŠΌΦwE½μ7 ±η›ΏFHζπ{†L Θό)ΰ-iM—ƒD \μŒ’™fΰCJ™ωŠ`―ΡΟ—ŸAΝ©ΜtΥ~υ{T@8 ίιΑΠu+4υ9FόΛKzΊi=9}‘IΣΣ `5’ΟJ‘€QλΨ9[™Π³&|:ηͺ4 >΄²Ξ‹ύlΡζ Cώώ’{_S&½mYŒ—UR χΙZδP; 9Ί„λ• ζ¨\n_Χώ΄»ΙΨ†\ ίͺjθντΚ7ΜGdςIn#(i·™pwDΕΡ'7v.+L“[}gλΦδβΞ5ΠΦuž>)|Σe‡ϋGΙ?Ωϊ‹F ή^€F>v3XvŒ`Ξ£˜Ύ΄Ii`†Mw¨%Ωmž6Μ—u !~x›υn5ύ¬FXΎKy($DR“ŠΗ©χρqγήβSOΊΠ|Πb“Gε+[(o·5‘ϊξhJ…«MI mT™t9”Ό‘ *\VΠ*Τ­=ς`μœ*ςo~aΣθJ@lm—“βzχ™λΜΑ/‚'Υ/OΚΥκΩ;3κ“Φ‘FιϋtE04Ή•ζBΏ0RtΛFΏ{εΖI‹n‘ 'υ9Υ‚%άwπ£$ƒ %!k]‡Ι _ Eξ‡ΫΜαb%ΎγΚ·ςunͺΊ±PΥqd”±ςχF]W\4W€αΕ ½I± ‘}Do1ωλΖ:Ϊƒx”΄nωήκVΨΖαρOeξ&(χa/8½oωωP=ί~© VžŠ,Dm??(ρ.€Q½-iͺj NVΪR‡oΫAοΙθ{™ ·‚]{TUξ/qΌΰ§₯Š9?£³Z+»aKΥ’€σr¨κΊR‹½ ΝΗ/1Βό5ΌcCΙγΊΡn-2Ν9Ϋ‹°Žϊ-*μ¬Ζ`1ξόN΄ίq脧?}:>b ΡΝωǚ»©Μ‰h43FJ)žΨ~©£4SΊΧš©«U¦'’XΛ΅‚Τ±»jΖ90&OχΗ„πΌΝw`ʌeƎMλ±ιw™£α8τη}IΑjΗœŸgΤ^ΓpμΥO'0…<ίΨν&;+ όHWλεL~έqΎ&66OΎ =ΫΠ Žϋω,₯—Β # ζσ)9ζΆΆΌ”h³''xΐ3QS½KG4…Σ…j»ŽΥΩθ­U;¬υs”±­‹σ`ρB(hX‘ObEͺœ/YWφΑΉz„9\ΦΡ#…ΰ’ k`˜1uξMŒ3―a6υ-Κπ»H―L3 ¦†Žηυ ~―8*Ιβΰ20Θ³φΪ{9'θάO©„e΄ŠΒω"u -„χξmϊ6Ž±ΝΫΎ)φΞηί4<|Ω«jŸwh3·£’λ;η!θ} <Γ8 ˜ξv΅\Ωr \λνΓΛ•Ά ·vάiξλ oΉ§$%ΜΠ^Y@ΠξjΪh$Ϋ³qη~γhήpμιτ©ζ­Π2°š$όΊͺsΒ")ε=,§B#(ϋ‘΄ΥƒεΜnχ’ρΐΫJΓƎή$\Ε_·:qΗ¦¦™—ΐφšώ>^ΎΦ»³t™~j΄‹«Ώͺ°‘ϋΦ’ψΞ4ρU i:ψΝΨCAόΞ‚Α~NJt•¨':_MR•Z,‘#ζΤCΖ€M£ELPΦ₯8^Θ―ο!‚cW€M\’:€©ŒfΎΥπŽΕ―Bό΅Yο΄Τ΄6i¬“]¦ο=Ω£΄ͺœΎ9*¦φm¦‰ƒp³Φ£dmŸΐ–nη©j‡B­†Π%ΕΉ\x΅6ΖγWΩΡ+¬ΪθLΖkβΜΚ©RbsζφΜΕύˆI@ͺߊT»τ )’ρ²”Ρ·…—„] άw‘ί;oΜ±PgνI0Ϊ±mˆ£’'ώ(ojΦΚA°•–όΔΤΞ°·υΧΟ†‘s:<π%ϋt_ρY@ŸΧ*ž]c"¦ΩΈΛΝP,θεΙ!b šM~ζ{-Ÿ‰ΜΥ‘α*NsV_‰ΣΙφα™(Ρ„3·εε9ΫW$ƒDD«·Eβσ΄CΞΖς—νΎ•…δΩ³'P₯ŒίV ˆ †Β¨ΏΠt3ΙMj`υΫg-ΫjŸ΄Ώ7< (Imk7λ=wΆ>ρ‡ΌΡμoίξ½Ν±]‹œ―{{`m@o*­Ιϊv™Πbξ‰`gˆ‘ ς{Ϋ‹Ήκ·3‘κ’C„ΉΤ³$f°roaΘφ-:]~‰:ΠΉ<ƊΜσή· ½ebg4ΆβibTAFNθP†%P žϊXYTξΔ©YΌνF7υ tT$ηq‘1½OUT―Ch $E %‘νA‚¨ΫcΖ#t\/€Vπ»tαa"—”\Υ<”Ϋš“πw$*<˜b.ƒMššόp£οΣ2ε—μΘ“ω™vt#΄b-ƒ¦€©™š,‘‡“ά@ΛﳈΏ”,Ρ‘Η³pΎeƍ…”±&ΒυŽl]ΪΤ*(RΏ»΄²ι1ιE«Δ¨‰+O¦RιuF•–U¦ρ4©fP£Î€ΣήΊ'ώ&Όs€Žc…*ƒc ‘€Υ#Θ­} —œgo½²žF–ΑΗg_DTϊθ…5‘jώΙΒΆŸ΄ "w6αΣΟό5ZχHΒAL 7§ω’a ΖΤ 8σ™66Ρb.ͺΜkωιF*°7nσΆ‰ΩβCϊEŒΌΰ‘(7ψ**ΑlώžΈDΦ8άζ#³Ρ% »;0n4BpT¬pph£fw7ˆΧΛτΰ“jιpΟbϋu R$rνϊ3β2? ‘“#58όAŒ¬’C0£οζX‹€Κ8䁇^’6uHY}bE cC«Šίβ‡)G»eΔς\ϊ«Z!“Σ8ΈψYρ7ύς―’ΧεκFœΏB0:ύάEηη„–ΔdH-_HR¦%δtgnQ3dΏυͺŠΩν‘/β[Nπ†“$4SAίΖΓιΌ½h)¨—Œ^μΎdNDϋaωͺγΠπ½9(πNκӊ ]N[{ξ ωόTΦ©29±ίΙ(πN<³ψ•°ηˆ ŒνμCΡΨm|…κςK wκښΠ;Ί…>Ρωα%κͺƒšV­LϊwfHεqΠ£Δ`Θύd§¨w:ΛΉq@°¬Ρ_X^ξx$·ΑrΞ«ͺ6€¬Bκ…Ξ~Ύ‹0ΩƈόΘ§_όϋ7Rˆ&‘½½Ϊ±ΎS=`]BPλ›œ‚Ψ8dF9:‡Χ­TRΘIρΕ:2ڟ<Ιv€ΘφX7OŽ eΘ‘8‡«Ρχ#7h‚η£y›J™Λ₯½½ΉŸΝ/ψ8ͺ@ΏA·Ώg]γ_ωΝezϋ6ύΒύξΆφq€ΧΟ†rœˆuS`¬Ω”Ϋ6B]\‡άn΄ίž“^Υˆ­Σ†BίOεžo+‡ύ“’ψΙώβ<:˜Μbˆΰ‰J’H>{)šf;Ί—‚4…―ΛΘεIμΓ-#lΆζ€o{œ-QβJŠœΠR›ΧΚ»d‹θ§_yœ/)8”iξΡίΰσB)–1γζ-VL^³$m~έΐ%>’Ύ‰{§zόͺΟA퍬Κε^7ζ4˜½ύM: %0fη΄Ή©ώ$žT·<,κ=‰ϊ˜i—μ& ”TvΟy—+γ ’¦υN‹ρΈY¨’L }μqmΫΙΘK±γEΈ$κ™ΆϋeuˆΘΔΚxμxβ+έι}ωdŒ“Τϊ‘μιvhGeaώ^…§@IpχΡY¨«~u uΙ΅τ‹<7Οσo&›=ϊ“ψ~΄΅έ’δ)LjΛΪ~½—:Ζ,gΡjΰ…:ΦπbτXE¦ΡkΕ“?θSνΧΎ?©αL.ςΏNΉ•γrqΒΔ“uΉxΔ ’φ΄ Ε›1ΟPŸ±³5L»ο3““ΚΧ7ί•‹UΖ#bBΆΌ‹·tΛ3vσz εe~’~”Ϋά G½s<§ψ'ΗHZΟ‹yξ.:Ή§Y ­:μο‚rάδ½πΊβόΉΏƒ΅'¬₯PF7‰ΩΓT–Ξ‘―τΏkοΩX-Š0‰Bή΄ŠΰŠώ1ϊVH οΎλZPτπΓmσ§μΈVρ³"ύQŠ~I‘ό„ΞγταuΰnΈSοzΔH(Τͺ €FΎ ΅ά»-lIΔko€\CW{‰l’…'ui}ΦΧΣ"ΑZτΘo{_taδq%Οσ·  Μή²Λ΄9ΙXί54ϊŒdν­ω7Ρ©€μ¬υάΩω§ΪΎΈvIjHΉ–Ί©_ζ]ΙΗ_"ŽOPμQψ”„ΣŸ}"θΤ­ΖCΧψg“ŽFk'{— niΖ¬ΞΟΞ€ Sΰ‘“ z=‘•„ŠΓΜ¬ό.Λδ.Δ"Φϋ¨™#Όξξ° &ωΛΘζY%_œΈΩοE7α‘/ž»„¦t­)ž,q[ΜšR3›BKΗ,ξ+©\ΰ Ωy‹FωΘC©?“aΦs’‹πΞΑšZkao0ξxώ±ˆ0Vπu αO³οΥΐ’‘χ¨(J ftΖ"ƒ‰ΒEΦ`’³‹1Š‰AU–˜Ηυ<>$ˆΌ¦ Ε–KV=Κzlͺτ‹bΘ~Ϋ 39Νol­΅_QwAα₯#ncEŠkt8θΌ""ΎΎέ0v Fœ;π*t?λΘΞ‘–9,Έ‰AΤΦ \ρ ­Υ"JΜͺ0;ΪκRV€ͺo1ΐLτ0O•‚†Uχ9ί'cΈ;·Ι œ}Y/:ΛK뭍‘9FŒ½²›„ΨαB""R’Φ™ΦρΔΤΣΥ„{dμ·³RϊΝ)ώ:e]’Z>£Έˆ,‹‘[ήΣ%8g!½_tŠωIAΎ’o¦°sΈ Šž£5fRνkλΟbd-ρ5r;θ©ϋ@,…_·ΜY­ΒΰΗTθΓO₯NʷՍρ²žθ έRwo―彜ͺ™‡NνΪφΰ-‘; ˆ³`[³‹$λ κUΫΞSe[l8wθ˜8σΖΡΌsgΥΘ ΨkΨa:‹7–³/ι5FJΦ“Eͺp!•\z:Ι‰>P<^Ή G·ωήξή$½ϊe­«7f²;›q"Ӛ`θ½-9˜ πN‰oA­‡τΕˆˆzΩFhΰΌβ>εζn™#ŒIέ{€S”ΦΨΔM«Kx[„ρ"+ξˆΜ£'Wy7όT?‘xœα­Υ½2 Ύ#Ό°ΈjVrΕrτ₯+$ɚ₯ΫП– aΤ€|{%< ΫŸ3άτΦ.3ήVtŞ΅ώΚK*ŠξjS‰Œ¦δCˆ}pz;ω3Pκ[C1$ψIFe~©ρ6ϊΣαNTDύ~2†αt}₯μαrξoŸƒ!Φd(yGΓλΝ˜ Ρ²~£QΧΌΌ†κΔŸυξΛΎd;Σ4±ζυσΈΣ|Ϊ\ψΐ˜Vkκ8 QŸ-ck!$“πHHFσιΓj”ώɁik+’ve“ŸžρΗi‘6tα9JΣͺ—»Έ²cΘ›Κdλ €ηύwL{ΰξ¬Λ‘ω ΑQq½κοίΘ‡θΞΞ›rž'r{iC{Όθυ+?ƒmτ{~,Χ"±ΫO|{φΌκς3d0Ήι±ώ ΏΑΛχμήCFΝy­«+q?΄pΘ» AΨζΒχnΘ)λ«―F/4‘δXΓcΡ Ώe₯^‘l—’ύ³‹λς©Έ”σ24?λk>ΰ%ωΣ·]ξΜΆŒ1ρ7ΔHN§fŸΔQΩ”}E&igLƒΧZrŒδ—+N—τΓ₯ƒ—ΎωΠl–c/TζQ?΅ΈΦ‚`<Ψδξ„V ,7 υ@ΝtΝh΅„τγ _{“ξ›Ηώ’ΡƒFζχΜYO„όžNNοI_ρπ$²}ہ« _E—Ϋ1΄.σϊ¦=«Dζ:~δΔ™YΊ7},ώV>wPφ#M}e|ΕήΚcD6ΒΌΥ֚Q>Χd\_E±9MJΤΛE)ž“1κ˜ΌT ΩAv δe€b"˜ο)PΠΐ;ώΚ‹IhwΊ•Œν’ϊَδβΝ,v|ˆK΄Hμ’4sί…}©aν·¬Yυδ?―§Ω™€pοο•#;ΘPΎ₯;@Ryθ˜}9Ÿœ³σή0)τˆάΈŸgΜΎˆΗΦΡA.Ei‰8Z χ#β-YqΟTdͺΊTMJUXdΚd“½.ηΙ_ž'Υή±ΙΊk„ QΘΊΉφςmΞΦΝΒΩNϋpmς·ιΰYg’2Tw’{ΌΙΓμ@ΧM15(˜Aui6v¨rJ LιηΥkύΜš9aŒŒtF5­ZƒΛ€DL!ωaŽ¬‡6Λ)ύ•Ψ±™:υΓΑ!φŠp‘‰|-IfΓ7F°>žŽ HφkYh±UΉ»¦φ₯ }ͺ=Tˆˆφ7!vγψν‚BρνΫT`±QPΊ’ΩΟΛΣMψF%ΒDꭜτΛΈωΨύΪ,°FAΛg5έΞυίSχMVΑΧ΄‹lXf—Δ¦ΙuΙχ@±―”‘0μψ«Ει`ΉΉ63GZ G0kώΥ΄nω›/kΡ}Ό†”AΞ©Œϋξƒΰ&-ruΪxD άC°0θΞNπ‚$΄>Ψn!J_—ο9Λ· ”X1fv‹sm}βrΥƒψλ°ΒΧ‘²X<쐉Vρܟ™θ«3τؘεNώDs‘ι‡oͺΡ7L] σ‰`9™Hα >BB²½Δ@ƒCˆ>KŒ•@―›šhΎw\’αΦgI5•G-Ώ‘.[ης+$·[ƒv©ηqΦ–ψτ·*aπ$Ρ΄πΰVT΅F[ΤΑ fFhŸŠέ ’ύ˜aو²_Ϊ)†η_r;υυFŒάΐμ>Ž‹Ωs,λN£Ί>&Β֊χ–ž8Ϋ>σΒΓ²μϋƒN¨2CΉ.³e‰"Β­u―Ο`Έ‘ώnξ-ΖͺΈ8ql}κZλω5Ύσ:ξ†[ΫΪ:Θ2Λ,])™ΝblqTκrŒ‘θΈi~?M:Ύ² ΟϋtZٌ*››υȊˆ>QυͺφχQيδίΐmδ#‡—εω¦€.βπ3:ΜέΎύΦΧΦκΡi„ξQ΅ ϋŒB«œι†#Ε ϋή ηw)§bCŠ3π›κξςkJ’€0j>o }bθ ”Wσ-Μg-σˆτ, -pΒ‚Ιc“ΌΫG²Ϊο―4xt‡{RΏ0")ρΟ’›Γ²»`‹ϊtHΞiΌ¬Ψ\‚kE³κ2liΝ³‰q0Ο³B`ϋ»IsFέΛΝαΟ™z/΅˜yΤ’ t‚μ–<’H ΫdΠ!1©‡γΆ£μ€«Σ«/‹ ό)L΅ώ…4˜Gλγcc·Λ™ψωŠN¬Ό’€Dζφu;pš ŸΏξƒΙ©_)Ϊb•¦4¦Χν(ΪΰΥ‰ΝGΰ ΉηˆFg,/·ΘςG’σΰyΆώθ,ω6XΎ±YŒοˆ†κ3ΖFΠΘ’‹`Μ­Fš-7ίYŸ%JΔRε“ˆΊ$uΙ•p6α*οΚΩιΊ7sš ψw%Ω?Ž›ε6n·΅bk™yϋΫ+ϊZ“h–Ύγrΰ–τ]ρOLœΞΒ‡Fͺ°Jο˜θ―&2²*κpi*™Έ‚—’λΖ͞>½ͺφPxX‡Α“9ω9χ–·0bη Qi–š!κ‘ρέWfΔ¨‰―e"tΔ0·AΈW3 HΪη#ή€fY_ΞςsfE)k«<…Π’”cjΣΘΚΌ\β…Β ώžΧύ2΅j φπaΞπ0#¦±· ΉήKL <ΙX£άΟΨ=?·KΠKQτ{Η%κKYτθο=β‡Yχ’ΡŽcυ!ŒV’τΡίl©ΫφM}΄q^·ζSΚ-YΦ¦ΐ:l«γ½θa{\#ζŸΩ]χhE-θ1ΟΆΜP?…wD¬νΌ>ω@a% 2Γ£HMeσ6γc›?Ζα)Ηv’X΅ϋYS,ψž“ώ””εUYύtμΡͺŠ0$OΒΣYсβ’ΩaM¬+…­ώκ1e˜¨Έ€Ξ Ԗם­Lp‹Ν捼1€nh΅lόKΠυšβϊόνE0 Yύ„9T—Χ)`@B’Ξ²ˆa€&Uwh…#y0σκΉ[`c/!oykYμ'΅#dόΟrΐ„άΠπΖ‘³y€Ζ΅A“Ο^νC.EΩ4΅EIΗDο‚€/~s]ΪEΌχΘσKε>ьσ˜χ…½ΗrΥ‘*‘/ΈvΦχ"ωdχ|2ζœCρε­%Χχ;ƒjΘζAΗ"²I$wΚ–ή&FM―Σ]›sΡβ Iγ<t\3bςΩ΄φ3*]wϊ?:σœΡk{Ν_˜` ςΖTήτΓ©A”·άŠ]WHάφFΛ‡Ώ‘Λ(κ²PLκ2$vρΈBΉzXe΄ηΝ1ΓΛκ¦ι>(R\ΑŽ%gDH|»| λIiz(΍GΕbΓϋΊ7Bθjα·(βώLΝ&Τo: ¬½-ΜΊ’CΠ KΕΔ₯—ν™πΒ„L€}JR.ΜΆ©^Βμ_λώfχΊA6M K¦eCmr€›*†ί`iaN‡γ ­Α€p‡’¨clV>αSoŒhx£6¦UΒ {@#ϋnN]§ΰ6ޘΟΦWζn,ζΏ%hMQε s;2^’ŠΉͺ9ωυ5 £N€}Ϋ`e˜–Š" ΚρΣΊ) G—ΉH»’ήΩ„Ε?g—€W"ƒwNƒ§:Υ·^=dwΟ―¨­Βρ`Ξ ά‚›ͺŠό’ν1ωΘnΠ§ΗεΌΉΫg1ΣVύ\―ΘέBWiž€Δ’Yd`3a,Σ‹’.jTYF#j–Άtπxσ" w-=:³θf*£`O‡:Šž7ή8RΛ Ϊ1ŒOς½?i @ξͺkΪΑ:ΒΡfͺδPZΪ:P’™h€Ή‚΅ŠΈιEwΧΔΟ€ϋK†V2h ]KCŸύŽS³q#{ξ ε,¨ψΆπ“]κωε0χ°=ώ˜±—_Zh©΄κ»fHλI‡²ύ6rtψΫnθ(l-DγσŒΊ‹_šAˆΘ-‘d†ς:ŒγgΡ<ˆg­5 4βვĽ °|ν“ύήϊ QμΙUΛώφD¦ŠσG,š\©δ8!ΗZnπ³zJΚ‹ϊ ΟΛEΗMοYθW―Ύί­7™sG3oωύλ¦q›΄`eͺ. —_HSCξ0| ²oͺ.ݘ ̘ZJ [b’ΓεHœ"β»5w:šΨUTj―¨+Ώΐkmψ©#•_’t£Eρ ›-ΏœB“ˆ¦ZXH(ηYl•Ύ8Θ0|]‹ΩTΆŠΩΘτ¦γΞŠS‚ΈhωDγφ*U deJzβϋΝU@•xb8ž1AGOV Z*jLt‚'έaΚϋΦ/Ζ ‘3ΐΤ"α!xσϊ₯π‡G ŠΥΗ€Έθ)%©δ>TW3Ζ‹γ/S5¬·† °₯PΘT€ωψˀ׏οe?}4Œf•«Œžž'<ƟήΟb·”κΨTλχ«Ω›‘lΧN;γ©E-Α‹?}U­εDΨ‚Α’KΓ™z:…ZΣBwpfβςψ=/<>κάΪΕ9Q§•ŒP₯PhEΐ`˜Έ3b‹m€ξβ3Ι„K—Φ@©”+;ΦwΌkΟς\/DKΤν:œuό»Κˆc:₯΄r‡I ²΅/ΉR`=3JM φœ‘DΝ#-γΈ‘0|%­ΎMοM:Τ'Y\‰ϊnwΉ·n4Ξ½wΖ9ŽMb›ježή`Ώ1]¬σγPh…τΈΜ9-ί_·^σν₯Ίͺ=$-°<§&z[σ FͺεξΗΤ’΄jόκ- ޞ¨&ΑϋFšύaτ;#ϋx±f=ΐ:vrC•«}tšΨxG»WiϊΔJV@'ρ¬~bφ+ρVΧZdlόK[τ½’OΆPL~"‚ξΡ!s°Μ‘,έΈΣΞTfŽ6EE ³R|lDγKJ8{7ύΝ~ςZbxΘσ¦ΆŸΦ”p½UYθ»PΘP™ΣPj)sŸJZlY₯šΉQF—§σ#3~kψ‘vί‘» ƒΆ†Z½—žυ ]¬·ΣjJMΓcšάχWBρ v|νοtύ%b.―Ζ7kH(Ηέ ΅Ηφέ½Ψρρ~돽Π!³έΫ°ZΌ-…Δ£vϊ7lμθ{:yΚ€3i ΊgΣΫ[žΛτ?Qΐ?5aΟZΟ8Ζ>«XΟd²»‰˜pPΕ7Ι%Hgpn6yw|byŽ©Έ •S铁lβπhpΟ+m.^|έΞuψρSλ(y6ΣY‘™}<1…‚Ζqqc ρͺ³RΨέͺ@ύNŸΚr₯ κEE½εμύ-BΊ&ΑŽϋΧΝέμ«lΡ•†?²# ήϋ*Fδ φΊCœ~O·^0x^Ί²…NκςNΝx‰»°Δ*ρπίCεΘκζgΩ–Κg’€yή'A:sx GIz,?€©$΄έΨθβτ©U…OΒ‹Ksœ žΗNΏτΑ·μ½κOVhpΩeU­kΔ(O–!ΙX1PΫRΐaΒ©nS\ 0nœ=Dςuβo¬#αMΥ%_Ί*;R5 ƒθΝώDμλa'ΚοiΦs%2z«1™a”°aΟ`u{υ*Ί‰2˜/Ž(Y""&n₯^X]ΒΗL ¨gί§,tλGτΏιy{”pqDΝ_ ƒ„Œ ύΉ~φ‹Y/ΫΙ4 Υeυφ~ΌΎx•ξ ΎG χ-•Y‚9΅ΦνΛ&ϊqxnTmŎZaΗ•Η׎χ*rZΒΛ‡ Ϊlήη{#ατΌKί°—ΐγfSΨ9τ_ςΆ>φηΞz—[“ίV9”Ο«‘wΞ!ΚΥΊΧμυνΉ₯}d[₯χ?ΙE ωNcRHΨ™Λ'35a‹΅YνΖί {ι2ΎέX ?–ΕΪΑ˜†ώŒDΛώ£ΫΗNεΞ0Δ“o9?WœzγΘύθNρ‰λ«–»(αz¨œ+«&5Ψ¨λyύγzΨUͺMŠς8ΓBmΛ‘ Rϊ‘qκe€wΠ¬!¬n’ξ9ψΈΐpKΕ7 h3·—QΞGFaΚYtδYLR'‡Τ₯G>Υ’ˆWb§υ€a_’=˜‹fTeΪdοIoM{άha+Ηνς™ύ ώθ”qM’ϊn£±Bγδ00΅ƒ ΓΗΨbϊŒ>|›qvyΔΕvŸiωˆ·ˆEΖ“ϋΔ—Η‹αŠžΙΈΈ¨αwZ—κ·Κ,ίαώ’[zp]A τaΖΙO\δ[±a')ςΉBZJ{κ΄Ϋ!“™-hΪg«ΛΫ §[ΰ%ο REMy²Y±u¨ ζΝΥΡ©»”lšύ²[ΐD8u‚zF€o?ήWcϋ9/iΗZώ9ΨJ ΰNΏ\AYžŽ€μ 6#αR€h!aωΠ}?Ckv:ηφΘΆBLžšχ6 F΅OΧ_|Ex(„Xμ‘4Wό.κ5WΙ ν³s‡?ΙΧEwQ垰ݴgό3ΊςΪΆ,ω܍ο /%όύOTΊyώ‘χΟ6P“¬ΙdΏϋ/ΝKžh(ψ·‹ό8`-δ,φ#—mRH¦ΒΪkJ±•Ω³LΜ2θF§ω¬Ρ{.ώ»±k“°’ΕR4鑎 ;œŠνXΏ <0—Sn“½Ηάκ9ρΙ ύΕΟ1`λٻ՟}H―έdΙ’ΎU@――9›Ocƚ½£ ΟΓi“‚’3G}£δΨ·₯ς.•ϋQ΅~o^ƒjΉj–ΛόŒOR‡^ŽΟ'‘&θν˜9<»Δ«qfϋ±gͺͺωˆc{ιRσΌ $gΚcP(Μ½Ζ([ ‚Θ·Β¦nXZhˆr“μYΑFμσ=ΕοaΜFΡ.LρzόH%€}§Lΐ‚ϊ¨7Œ°Ι ?y?6v„ϊ%w@ Ρ‘„ΗΦγ―xxπκh<(r7ΨlαWώΠπ&˜326t§Ÿv„`” ˜(`Ÿdϋά«;dg-±E˜ΙΩF΅£GLΝ Τφ²’7 :Dφγ8W‡…}C.τA‹³}Vsχ]¦^„v kš’λ«ap Κ˜ΤβΡyPXτιPj:7 PM/pΖ¨^ύwΨ κΚp0ΫUψϋύΚQΕ‰YηŽκέ₯8Υ }Vx5Άφo’Z.«'΄²a<ύε*θP`υΨΕΎΩ 5$ώ4lΕ’r?R’€bΦ[8ͺγI―£ŒΉC½b&Τβ»Ψ!’†U;Νρ§tbtO|HΌύ‰Ξ99Ηέ¨'βWGαΎƒ½Y—žΤΒ7Εi“CΌΑ“φ“92‘δ-¨ΐ΄φΜίΤ¬ΗΛF0Όώ1`Oψί5‘ ςF^8σg$–ΉΠDœ»qNη‰·χQΡ7κμƒ6C@ƒfRΆfΔΑ³χΙ± 6Δthρ…ΌΚ‘Δ†{βʚ'#41ΎΒžΉ °βΒ@X‚ζΆƒ°-M9Έ)sU#ΰ ^ ιŠπΤ[ί¦ΖΪOΝς­Υ‰¨όmf ²œ‹N]ρ, σqEnΞ%x΄D―sζ΄'₯fε ξΚSΒν9ΆΑmΘ Δ5°˜tΐFά=LV\b΄Ή‘Ρ=k΅’U„>"έoh‚ΡgϋU"VTΈγNΆ5†ςΩ‡†ηUψ#¦Β‘. »μ8€~Τ,©§»ψΨΤω©(Yo83μ6*0žx} KIίΓίaφ°œ…=εΩ"Ύ°Κ³Δ.MΉ%΄fιpjF†W· Œ©c,„X²φŽx2Κ‘iϋ|ΛW‚+IηΎΫ>M©1Y Ϋή«^††¦¬±Δš›F£&X紃M KCΆ)™΄΄S\”Ήe?1Ξ#ΜqΟ)Ώ’„‘λlΞβΣΨ-XαΣ π|ΊF+Ρι”5zΑ;G½©2 $ξ4(ďƒŽΎΒφΈŒΟ4ο;―δG8Ό­ή €&7κψx‚ΟWZ°"¦KJ„σΪ²χ4$eΜ•ZS#fpZΉr?΅mχΪbI7=cΫσόΑ(Ά)Ιu3ԝ‹†­°Δ#R…ΘŠŽεp,ͺν PUpwuβq£φ FΆώ2;ψCΠθΛrϋ^~qGl—χ74œS&Ι›zΣ0׈Œ:†―(ψΎ *ψ³ϊ9«Λ %„”Ε-½?ŸκόEa‡λwٌ’ Θς&Ώ;\OίΠKF π Β›ζrwΦ²:-Ÿ¬υLγ2΅¨DM#¨₯zΊ6Ώκiͺ'Μtx<ΖίO\}«πoXƒ…“1“ΏnπεhkˆΔ‰ΐ§₯H½^‘ ^ω,ί™¨n%L‘Χoφ,N‡0¨Όeaθx3 Δ9GΑS‘^«ψ@ƒ₯6u~8η[žΔ,T\9ΌˆcζNΧυn΅#WH†Ξ!‘πNw²NcŠΩ€Πό˜ŠΧ&Rχ_%‚‘œš],ώHΒηzوzd©Υ½~²ΖΜ^±–y.,Ι½‚C9›”πW‰yΛ4RWΑ*ΏΊ¦ί"ITοΤ#craόdˆHΣz¬ε‹έd²ϋBΒχAίη‚·œUΒ’’0+šΧͺ0ςv₯](φEΝ ͺͺψᘟgοϋύ6ΎiYFvΩΆmΫΆmΫΆmΫΆmΫΨΛΆmλΞ}˜‡©:Ώt₯:ιξdBΓF9yΦ¬άχŒZΕoόΰNΎͺ«jΛ¦σ‘€ΐ qΉtžpΏ0"[ιν%&₯iMeμ£κˆ;ΫδnωΓΟ€ΡΚ1u.β4ϋdŠο±PχΐΓGώŠϋ 'pp6ό+|ΆiΩΜδ;,r6λόr#€%§qδ.nCαvπ'5UήΎο<δ°ξ‡ΨίG<―ύz‹Χ#4JbΙΩ₯Sύ{„X]cρ5τΉ#Α3ŒœŠΊ;.Ύ’²«ΥqNWΈΕΈJπFΆTY\ŒXŒΰwdχH˜ΐ‰„…Λψ―J€rΜ\Ψςσ)εkΤϊΌ8Β$ƒΑτΩ‘μRψ^sS9ή΄˜ήCύ¨bPuΓ2Α!ώN`ί }›‚Χ‡gΊaπ–ΏkόνK%•§lWΉΔύ†Υβb(›%„_‡με/ΡΜ΄L»7ς·β L£Q=G7£ί|’ΪRώ Σ™9‚ΐ oΔΥ°H£±lΛz†γί<‘ω»»•LΆ{¦C»t%΅Π°ϊx‹~wOΔCΨϋlYl9Θ–©0χJΝDߏuΝ_ΜΖΜ ηK½χ[`:0ŽΧνcΜ` ֐2)ΑχΙ' πݝGθΖF±’˜$ΚX°»ΫΈ«₯Oϊ@ΩΚη+Φ©L₯5Ωτ·.šώh·/XxΚEαΖ™Σ^ΐgΟψ±Lš=νFΟυ€€}TΝ'—k{ΦS%ŒGΡ'.‘y"y&ŸΉfΓ εߞΫz„k)‹?69‡—άΊΐΞμͺβH8ωΤΝ ήˆžμ·xzmφΐΎ•»8­-c•„ΩλΨβšή‹ω`b8φ z榃Œd‘”**R4ώ=") ;Σo:>zuzζ|‰Gϊήfl¦CŸ0ΠuO›ΟJ'&βιzUΉιγtϊhψe˜ Iα;±¬»Ήλ* ΖkΨΈ²™vλΉ8δ?,‹­3Ξ\[6 $2Κ§1Ίœ?ΙάKπΖUψrΛήQ+2sφjŽΞVω€ € ξΉ]ƒMλΤξέ\>ΡΟGd¦‰ϋ»ι—Θž‚η’• S5λΙˍΟ#‰~Cι`– I‚±› ΫΪ’$žχNR9β8δπ•—‰^pΉ|_(γ°/’Γ]#wΚΜξΡ9gςUγX­€ζX ό†$}oXυ“ΫΪΰMΰΡ ξd6J9uρz―΄3•φk8―ƒΖ›¬‹:Ε°C’;εQζzn2·’Ϋ —`hόzQ£έΩ HzE6[―œ– βΣ9Η1)…ΉN€8υι4Ϋm7vœ₯OH1›Η΅ΖΤsiΪΒ—ΜI₯θύf)θrυh”Ε; Σ‡#jbΜπ4`p‚`ΒaΣRy¬Λχ«že)ƒΗκ¦Κ&―W™PΫΝΑώœs΄fΙ9?JσβHΫe‡εmΙ~]ΏΠd   S{©kχy!0’+––£Zλ5_‘ε›E¬H³£γ{ΕHϋ¬C’½(σΩ½QEb½ρ€RQΑβ_q€ΖΞΩ—Ϊ a7cϋ^Ή¨A‰k/μ§V/6D₯±±¬'η±L¨ϋΠ…έφKXξΆ΅(‹@¬y§Ρο«H#ώ…)QğոjΟ χ40l˜₯6°ύϋ΄[­―§σYΗΊδ=b¬ͺΏΤIΠX ±›6x£E;uss²s •&?E3™ΰΎ-H Μ¨ƒΪίWή¦[Yξ<ΰ³ozΏξQΜW%ΣQ Έΰ˜A±Ν€’y17Ρ’χ)ΎFbκτynY%;­ Ύ’±…ΎiΖ (1·%ώ†KΈgQ; Ζωβ9*ί₯lΚ¬Χ Ζ Φ#ˆ”9€ΜY6ΌΉήBΐτσΑΟ/AεW[g•‘ΨKlšΞ‚λΊμ i”›ΦΚgs@£$Χ eYΉ4ώ” »\]nH Q_Ώϊ• 5s;o[ύθιͺπ―‡0@0'~š^”»BT¬5– zΜ&οvˆΪv\ρI6 ¬ίώ e¨w ±‹ΤRΝ"›Ž·(ηΊ“EΌB}[E\ˆ-ηŠΩ‘§₯Υξ*$uφM-gζ’Պ dΡ"“3_Ύ3ξυ’v‘’Η ύ&"#OŽ+:2pΣS½³WB<Ϊ «ΠΆ‹«9@Ώl0ρ­‚‡Iκή(‹,ίztΒΚG+ΝΚ½ωζΫ…ώ«ω£Œh¨V‡ijαsf4Uί@>₯ΛΎcυUώΟΠ_w»όKι7mςͺ\fZ9κί€ύCGs’ΣnEœΤφ><Τ 3Ο€ϋˆΈ•`(ίΒuΕeΫτ―α>W–¨[ko=ΜΝOW&λ-xΖEqι±w,^YY±I b‘/œύ69/ς?|ΚG΄yΛΰWG%τΚΜf!5Απ„τQχ¬<‡χΑͺ9 ΰ. ‘YŠ!Ίή[©ΐΌ"€ΧFU'!­·‡Z] l x zaAκ#»5#ZiθYγ οΕ]ž¨w˜ΫPΗΆ5πŸ}ϋ‰yKw돷›ΔΉοίρy΅¬R!rΒγZ”ώ<•eΊζki+ͺ gσ!Ο•8wSDΉ6Τ䦇a‘†}Τπ4R8$‚D¦6“τΫWΝDβKΖ&²0GΖΕΒ%α˜ιNŠΫz—zΜφ››32sΛξ,ωΐvP¬ Ονφo₯'--έ+Ρmψω蔇ε«_Ψ h" ΗWihw™ΖO~ηςΗqXή·OΏ·ΛρΠΘίΚζ vΌ6˜Βό! Ÿ·95VIφlΟ¬ƒ„‘Cͺu‰‹%W46ιiυ…O=\ΛιX'%eΞ― †nJ ‘q„Gθ€o{>~=1i ΠρšDž›ΰωxΖ_ΤΚς™¨4…σC]Ή’°YηεˆAΙπ5Ž½iu3G[ΉνgϊωΆΜgωMejŽkv%ŠΓΓE‚οtΥO£_M#™Ί½ΙηΦ—‘ή»;L ‰φp#dμκε ݞ§sϋΆΓ σΦ‘ΊM°o~kΝοϋ ΥsI²ΚΫο"LβیΊuΘΝε€ζ³άωή}5˜˜tΣ5a ά€ρζςΐZa*>ΣμgAχ»3~¦ζh¦r—Jυipƒ‰6k·Υ|½SΉΕ~Έ‰‰3μΨ+άBβ†Ί•ˆ‰ύ0χKΨ+«ˆI¨ν!υg4|ΔB?dU₯±l}ύ§‚CΚψ™stš-{)␳p°0/Šsgώ™ΎΦΨΝٍυ:Πί€F †yζήϊ«°«β%³4ΎάC Ά7ΆύGυiAc:‚Έr L‘ΫVΪΚ€[ˆX(ς—•žŸYΗ~™%[M<‘~΄n8ά‰A—JlΈEœώπT^ϊU[-?Ί°’g!Jžλ‹/œψ£@ 3W’³i%udϊ S]g6Pa΄Ϊ䏨Ry ›Ϊ7:Φ«’¬§ο²m§ξAΛπ@ϋfΟ7« !›w%ΐςΙI.K7iLΟ:γ«ώLλl!εœ!]WΣ%§ΦS[›Γ›Μo˜Β,Zν½‰;ΟoFΝ‡ƒKZ`d BhIΝf’eΩBΌS …±§‚p{}Β9m ΛΦ!WΫY^ίiw²9I<Λσ―κΒH@ayo“τ$π‹Œα<j”’Βϋ§Λ—>ΖσΨX aΎ9οP‘~€δŸžWRΞu™ B •ͺvtΰέΖΒάtϋwmΝλr›Η ‡{χRfdΫΐ†θf_ν η*Ο*Ο7κ%λW,^Π«.ͺτ°OZύ>”ΆΊ6 Ηя$φϋŒ‘€Iζ₯0`κσ»³_Ώ–³Ϋό`Sθ£l)^Œ%?ͺŸε₯ί…φT4œΞ_Σ\PLπjZΥΖ½Υ2ΔθYyα.’γΎ‘[q¦O•Y `³‘ππ9¬2‡vCz«n΄ε½™ι„ΜΎ›<\ssΆYΏύvΌΒͺϋ:Ž·bΘ/‚”ό/ΣL ςCβ`σ c-3Ά ^βpƒ‹Η-Ρ©+H!OZ₯™3‚‰χaaƒ@udlΣIΟH8 ΞˆnJΓՎ?C6†I·š-’&ΤK±b5wΣί π"aKB<™$jΑŽˆSΞ°\ Λ±6C$8n³ μΞη΄±YhO[ŽBΜ³ΣΞJν―ο"»η‘ JŸ1)ΧtΉρqŽ*ͺCD·½5>Ν¬ψ6λζ{Έ8Xά©Š6H)‘yC8Xaε'R„ e,εz™¨ΖΐKMΪvΖ\˜ζmά֏½E½sY„ 2Λ(½ΒηΠw »΅:™ίρO·]$*₯;qΩ¨ύl}—2€^UmŽ}ή±Ώo*Α˜³]2#SΒ1q:.9M»Φž‰£άnΥaP‰Ξc€ qМƒ·tWΪυέφό€ψ Ρ“šŸ4@ΆΎe(hΣ ]cύΪρΏ`6«χ'{›Υ*η”‚Ύf l[ΪΔh"‰cDŠˆΖ»+M?0b₯|w–ƒ΄ΗT€ι£’ΈΜ:«Ήώϋβ"²:>T|ψrpφ©ρΡ|H2eFο"ϋkΘ_Š1*ˆή³$0I%$‘ΗM΄[ΝΕφo»μλβΡΏΰ} `ΨHU–Š :ξ ΊχFΤώ•|QίĎήλ̍€Η&CZ<(#Ύ³p!U‹νβΡAΦ IwΨ{–£`D•ΝNaŽž@2d,+a°k˜‹Σ[J‚υJΟΨσ;?Α‘~‡Oλ>/ͺ'‡·κ‹Φ΄3Oγz¨S^²TΙ‘ΚβέiŠΎΥst§‰$\ΝE¬ΩΩO]fp_X…μxdž .Ώ^^ vŠ5#rΫτ$5¨ίλ0$0ˆŸ‘fXήkΎΒΣΈNκ@»`šγVΈGΟdTθU&PΪ«‘eˆΪŽ«œ­,N'±+‘BΊc.•%Ρ\‘΅iΎο<ƒež,­Eƒ%MfόΡη);°#0_gKϊ8ˆcŒ"α|Ψη°Az·’΄σqGη‚Ω„ž³ΰ!N˜βΰ•λqxEν°‡O™&κ,,΄άxŒh΄ΐ°!­ΥΩο­°΅1 '_^pvXΔΟqU8­DΜΤ‘W 7ο;R ΰ+‰~dpOΤψ"±΅ς ςqz€sŽJ|=lΛ|A& ͺΛ,LΪ£β[ž‚π‡ͺίοߝ T½’#J‘εL! †αϊBC.Pά Υx<Ει<όӞRjKx’TΕ—IΛ“Αit$ ‘r―σpισπγžύ —ru”_"ωY7”[Ήί‘ΣΎΉ(θ@X'ε&‘;“o/*ŽΒD‚`ƒUωb 5Χ„Ϊ½δΧΎΚ›6Β>©ι’”’ί|1}Γh/ŽY”£=kβ 3B!„YCΌ0ή³<Υ€l/,υΟEκΕ{TI”ηΆWp»z-£+Ϋμg©IΔ {Z·ιaεO·Ιˆ΅Y#YβPΈ 4vdJXΌ²δ hŒθθPΘΞςρI‰³³C‚€ΏIε½°(Δ3¬G›γQΓξ UχχκKΎ|§P4a•\‹£Ξ}Έ ¨ΧΊ(z ͺ ,ΜG΄ˆ& °ŒΨ?BΩu’X0ζμpOϊΡΙ@Ss^:ŸκΟF§kv=ζƒJάΝdQQγRŠΓ³hšD·Π³Ήgͺ*Ι9G.ž9Ό]ί,ΦΟΰπυΠ‘²Ώ!Ε€`Μώ™Ž;Ϋ/UΎΑAsCμ ₯δΘw½DΏo8ΎŸ+L*–©ρb―νηό1’7ΉαR‘χ—?Μ9N₯ΥSh¬fω°νΌgQ@Š ¦Ω¨ !†Ρžkψmβ‘~¨C}¦‹ΪΥή…KΕtΊΥ­V–JmΎ ‰JGφOœεξh:#f£ΰλL?nβ3‡ή₯D?’UyE ]#Ϋ]R|»χΣoύhΩk»pν'/τ1‘,ωŒ >恐…Z9D“U˜Dθ@ΚmzΔΐ€‚ΨΕ·†)fƒΪΘ”₯ΰΜΨή“„tλ λΕ Y‘ˆγ‘AΛfm©kk§g 0](ͺ~¨ea’y,E^쐧» ΙΎΦ‘cΑ;‘Φ>Lΐ{ˆπxO3X·"κΣD.€7z+υC8 λγ&›1ςΏΪlωe_·€‚€«EB΅:ξ³H…5˜™@α™•be ―ώ‹”FΪ'˜άwŸΦΖ‘Τ½ΨUKIjΩ3ώiœγy#[•dξγσNŠ<χοΎk*©3Ξ)mdω[π!ΐSGχUχ“³I„­4I„ΚRh2KΕθέΰDέ#UΚωŠοsφζšYυτ'6§υS]θ'2λ›αͺEΟΙ―ΘΉβ‹ξZ5…ˆ8Λ8Σ>0ώΦϊΥfΉM”—kš!ί–Σ!(όΥ$QeΒ³G)!Ή4τ°·?Wi«{t²I„DXτ²b¨%+θ—‹AόΓΩf-»}”Ο§ A¬ ³6έ­qΩ»Δ<ΫΐLQΌBwΣΧ»"†[Ϋh›[d4ŠŠ§”γEJE1ΗΒvWξ¨!‰?†ήύ›5Ρπι³Όr%lYίjšυ†ΊQA›9σs†l“^ψƒ<–·*kͺ)«έΞPAJύ—>Ημ}Α\4Ε6φΟ.VNήΰΚrώ܁­¦GBΤΩ>χθŸ‹Š;₯έμ(LB–£ΘνΐEٌόӈΆ.ΎDδqαηκΔT€ΟΖΫΎΗΒfJϋzφΊσ|O²yψ»<τΤEA"1uŽrΨτΎμ±ϊ­ Φχδw³§EGφΤυzqd³ {'μeζ£έM.Σ°³!?GŒΪT-qw1žυW‡ξ RqEg­fʁη?a™w/„‹³–OγS ϋAωΊ6²'#Aγ‘Aνι` ^bv” LύΜ—^ͺΜ‘US?κ«Ψ›Ω6Ξ°άΞ6Ρ’όvωRΎ»Z &ΧύVφΉNΜη(…ΙΏ<σTε!emωΧ$ΆS ΖP€zn°‡¬Bnt?Jΐ‚~&[βŸι}MVcŽψHNj3ΧάΛ*› μΛV⛍ξ”3ΨΫRχښ'Ό‘ž>)dθήEGŒ™ΌoJ­ŸΡΉtΜ6ΛΔaC,•"ήA½ŠΩΖΉα…΄Σ_ίψœ νgΌyω–ίŽβ«ο`R/ ι2πχk9cρ₯»~ο’ώΤΜOΫω΄ώ―lY€ηψΦ$ΏΖ/F₯ή Κΰδ˜‚χUKδ¬Μχ§€‹ΎΡœ _i³\nΘ!ΘF‰+ξl%’’ΔΛ-Mρ$=d#—J1KΡ―Μ«€Λ„h/ΰO{&ΥΏ¬oΒavό3ϋΒε[αζ#Fcœhε Λ3έYf;Ύ²",…ΤρL΅uΌΌ°™ωοXήΰA*Ύζ–UPj‰B«η|Yι{O―π lΏ@ώΆΘΓ³1E/Θ8τ²J—:ΟFɞœν©³ΑU˜šZKΫ†[³όθdtؐ…₯V6is‡Qκ€Ό%¦ z'!MΨ|uύ!ζ}μ™σby9ρ%&IΈ€4Θ­sΦzt‰ GpsiρωF!ΠkRpλώΑPψ z2fω&|jNbΤp%SZ.Ν>ΣόD­ΗsFfβžuu_²VΤD·½γ6Yψ"O™ ¨¬$6ŽGΚQΑ:I&άΊeγaοZU™υk›«άϊ\ΐ7kNΥΎY$Ξ³έjΩδß©ξ”ίNΓ£§νκl|›…2|.–{ιέ„N2πΘ›Ο½ψ·bsGa€ŠNυώƒR¬\η%)vψΐσrh=M %{ΏΔΒuκ₯εŒ>>šHΓφsG/ΔRμKήάύc\Γ­ό ήŸzVxΜθ2ζ γΜ€+o"ˆ$ρΞ¬z9μ΅νΙύ Ύ±―†‹)ΌˆͺjΖ½φψie{αŽ/Ϋψ‹τ1a²·4ym| ¬P©ΐ[U 1ŒCΒ)±ΖIZ:ΚwΔ7>Π z^Qνoe$ρ Qšu(σΫΖk~lοŎa‡ M«ι½‹.uεπύ·Ζh4»“†fο©Ώ―;+:§$ sηΈCΧp0xΙƒ'dΘτ]ƒ^[Μ”φmα€#tΌKa/Wb‹qιτ¬q εΤW3‹ yΛ#Œ²>VΙe'EuΦΙΠ–Z\4H€³κQφ_ΘςΟΌU?$U$υ’WΰΠP 9ςΆ΅αΐΦ8Λ@SΗΚƒ›₯ΫœPyΙΏΎp’ΟΗΥŸOΠA>Κbήeu<˜ϋ'έƒ4³_Ήτ"Β½[žΤ#w<ΠKΑ£\ΡάZwϋφ΅ Ο° πω§Ξ!‹Ž ΧΆΎ“™e˜q…0=ΡΧτ0~”œΏqν•ΐqL&eWQ eω?ςdΩπ!-€†΄£ύKyρκ? ?Θκ))gl ϋτMΑȚ?κIM§ΒΈ‹ίΆηwWiIr”ΚJ/ϋ‡@~ΫvZ–3Ρf 9`§‘X&}ΓoayG8ηU&δΑ»›XA»ΐξΣΙ-j›€…7πϊ}Ή{ –γΰχ:όNβΜMJ»di‚Ώ σ(―`-ΐη>cΎ'θ„Σ¨…?R&ι΄)x£‘>pPAV)©¦!ΛΆ‰}‘Œϊ¦!M!θyAνY" ™2[­YrJfό_Χ¨ϊ<βχšΉͺS(\βκBΘΔΔ1jΚΕύωS½sV―=¬φfmΣ{Λμ/ϊθΓ\-xΑJŒmΜ}Bώ2‰ΰΕMά°μΣs+†δ T­XœeUŠϊbύοΟγ>6&cυ„ χΙ–˜Φu±ν§ψ£>v:Dπ6ˆd\Јγλ_†¬ύ±3ΜΔ_ŒlΖΰ!Δμ³ψi©’Ψέ{δΨΚ’#ΆΩ9γP–kcSp­Ÿxβ©n‰5ˆZ±fZν~σ² 9–Λυ6•χ²ή\Ότ#€»<Γ§ι—Θ#Š]μάΡυ»–Μ"Μ_6£ΏR„ˍHCY½ίζ䍬ΕΠCεηVn±b]=k:^³TΦL΅Š:R’N‰ΐ*Υ“¬ϋ—¨ΡQ%<ΌϋšβΈύ—[£θiTNq1υZEπ*ς’+R’uΆμ ‚»ιˆ]|œΰΨ1¨‚ιPΘ±?΄3 s(v⏻`‡vσσbΗ|¬w/§¨s)¨±yPhδ’dΘ^‹Επ9»™z`tpΌ²`h··ό±yΌ9lεxeG₯νpˆ{Γ!yυαΉ*e²E8ΗΆxG±ΐ}MKzθ ­]©†HΎ?|δΈ};‘)οEφςα ―nΰΒͺo6ͺήlaλ€'r£Χ7˜Βs΅ϊΜήΞJμŸοφ€Lω8b Έ¨W/2b€αΪNIj‡SκsΖUyΑ―Rεcˆ―νr Υ*Q1ςšΆy§ΊUΗeο M7\™bΩσy½°λ:qsŠΪηΤΪ΅ μ‹” HŒΐ±9Κ`"ΗΈ”Έξ}Ζ2ΩΣβVKiκ Λ EΔ‚5δπ…σX™r™=>h³οT*Γ·AκεΕZžFf gΛN››TŠ”δ£8·ΞX„X^C‰‚< (˜U©ΙΟΤ:ΐ^ΨΥ mάch]l?e_Ζ^hό©)ΗΪ\ΑždΨ0YGΑ§‚ώd•{Ž σξεΐ,ψ²ξŽr Ώυϋ%ΰγCά₯/§!nRπOˆΩ–σ“P !υ•[Ν&R[μFpΞ…½ΪΚ T’fΆί[―™=|dύ‘FŒ{³ΞMMPν$}Υω8›4wΉ‘αŒ§­ς…·0nPhQ›η…ŽΦƒN›S ψκϋΪ-λ”{b!Σ.ͺμu8†Z€Κ”RθrT’™%•—’­3e€pΘci?Ϋτ0ί―·(DE`=]znc9ΙΖΌb [Ζg»άu’―Χ‹Ž’—b/’)ξ‰1sf§šΑ—ΔpNTOƒX?ŒΫnK?ΤP„g˜+ζΎ §2n {φ«#,ύo gFJ*W.―<z΄J«ΩϋA-‚CŸ}tΉΓ­ώ‚T:ϊεš™‚'Ϊ%ŒυΣΙΠ1€Α#Ί Ί·—Όβ> Μ΅΅Y2Vyj,’ΰ{“œeλ€»i₯U―rUe˜ζχž#ΝC’!υS±^•?,šεKq©Q~‘uαuΐ&ξ‘δYχkrψˆί±šϊ‘μfP>N/ͺ΄Bx»°ώζτβ%±q‰ΠyLDHΨ$Ύνav1'ԁσ"›šP…cBšΞΞcHηήδΌmQRΑœ ’?Ά$a_?Dmρ Υ”ηΦWeσχιP¬#.νB:„Ώθώ«rγ`Ϊ7’yDΉ Όεrš˜9r|ߌ€k³ΗΈ§ck€η;VβiTτ3Ή³σό£kŽHͺUύΕƒq# ΌόͺΕμ".Λ}N~ξvΈΑάί²’.fγχ‰ Eα_1± Q‰Dό—HΎJƒπOΡ"iυΫΓΗίΖJŠ8@m£Œy‚šΓa{kϋ‘Θυ0 X Ž˜- Ε™’rau‡h)DΌΏdοLHΦΏUΩW¦R쑧ΜΖ-ψΘ8D GΐΘzu,A΅ΰi+3]e&_ΑΩ ΣΣf­μ­§M:­Έv\Ά·B%?/fs΄¨WŒ‰#]ΟAϊ`΅ŠVΘ'CΤ}啐ηζOΧa6ε'3&›wΛPoιO΄“¦jIƒZ#:Α΅9Ψ3šΥΫ σι€γζι„ιφ°ΐβQ‰ξ©’Ότ¨(œMs8ΊΡn„±ΗΌ%ΔtΡΌΫλ σΗaqF“0`Κ\§“ε|ρhύϋξ“Κs €ΰ}Kc2ΧhMΫ~EΖμύ¨l τ(­’ŠμρJ9mδ}BΨ4ͺ3œύΧ]„ΑγΈό·¬S5yΦοΟ)^ŸbB3Τ{Ž€O°LfMͺi<ͺΊς<6ŽΙυAϋUŒG GΫ:.tB¨€ &ίςO³ H#q(’S7£Ά)œ…ϏΊπδθsγnΊ οΕ¨*>Ή ΣιUb¬έj˜Vm π$[A‹ΞAG‹Α•‚˜\ΚT]|20&ψڑfΌΰώ«κHΎœφkdς>ΖΡCΥΕΧ› ―]Ώ ͺΆΕ_ί¦Λ€Τό— fΖn)Y‚‹€”ŠTÝ-±ΛN-ƒQYžι‡Ζθƞ2sS₯™r`σΰB'πkuTnO3’²"΄ €Šμ1ίYβΖ#[ˆΓ’Άΰ ΝM>@Bώo–V·E^1«α … UTΘώ’²―Ϊg%€V λίΊͺ`Πv}f-­ΩmΞ <’γ¬aw΄•Ν£λΕ‹ΨEϋ£SCό‘½βθ ΅rT¨£•Ρy!„½ΪοW³Ι|Ξˆ 6±οΛ―sƒΗ'ιαΩ­ΰ¦Δ Ξή«Ιπ˜χ`%Ώ0}Χ=€ΚŽϊον jo’$ Ί+~‘Sϊ& ζύ„E±„ΐ΄ξ IžZι·ΘΜΩ%εζ@,—±ΦF!ύ#qϊ,[­Οκ4F͍a8]τΉ@3ΎΐζσIΌΫ*Šωžq°%˜Λ€Ν‘b΅ΐ…΍-θk°V…S+ ”QiwγΜFN¬ ,Ϊ Lϋίσ>»ύzόz€qΚcUκ=Εؚκ±c°œΤ£Drz­ΈΑ^_RώuA1ξ(― 4‘„ΥΦρZ}ΩΠΑ¦χωQ™ŒΫθο ‡tCr}끴dl9Ά°9χ°|žΈΌ5A©Œ#ήΥDvΕ^π\ϊoΔΛ9 °0φ›ΪEΤjŒCΓ+›Δ0–α`~šΈ*΄˜˜¬ XiΉξ‘DΚ5ǘΣAQ“G/Ϊ”.ϋ6έγ½9. ΖM.„”aF zΉ#Ό¦3*ΉθΩͺœTΞζ\Τhˆ""•ΰ ο5;½‘+'‡χΰ΅°L4r§kK9ύέώ:LJΐζaΘόΩΞDCMŽ¬/ΜͺfŸvI*¦J98Ι»ύΛ9‚Γ΅λΆ2 U֏1?Y˜ τΔ„΅ΰD@'*| Θηqχ1WΝbi―ωγQ16”‹Ιπ½Žζ~lrXtΣ(‘ψV ŠνMΝφ0"t—"?τΥ Τ1ͺR[sšΨ,fΰH$-«a1R0‡μkηέmƒΜA΅7%@†‡κρ°ž„ˆ yσ….~-½νΟnς =η}pmZ-ζΝ’;7’ζNŸKCνAv쏆νhBAϋ"‰–ΐΙ–¬ωvDFK}²‰w„σjv!7συ6Mξ '6ΰ‡τf’#$/±ΛmΠSٞ‘φPEe½Υ-₯φz†: Ζ°‹T|8\A2qrοœ‹o΄ˆ΄ξˆ†Šc|·…€ήtΡ… ύ*ηΤDa" 6Φ€S'…7e°B3»ͺlΆ1ˆΆγ»rΕΠ[½‹δEaˆΎ‚τβΑ«Ύ9θd8~¦R“8UΓAΊŸmθBΒε~Ώ/ψ^φBΟΐΜU!šgQ5#Γ]TG₯δY»žΎζΏ.zύsΉ’(HΓΨΫ]σΨΓ;  Ÿ™~L‹ώ'Ϊ—tΏU€ΑώΌοζ?κakΩn Ε6q~/*όΆΟΖ5ψ„’«‘+θγ5W.ϋW›nϊΨbypKχ&@{ ½Ίζv‘πq£ηΉ)8ή™uξ/œ8θί¦NοκtO-z5επυEt³ΠΖ0ΘΊI8GdΊbVεΡC!η―wΊLΓς$[,П&a;Φ‘ΰ†!!kΦ(š΅ZΈΟA§E/Χw} j‹Β†΄δκΒΎMΊΧ$*<ΐT>νρ%θU³CgΆφΚ–:Bœ =rχ†ΛΛ%Ώk•”φ9s˜_β”œV‘$CRΚε_‘EΖbj†›F}{’umŠ5&»‘5©ά,ΘL‘ ΞT sή…XnΗβ•κΠέFX³½2gUόlπqΥEϊΞȏ½.˜†Δeuύ\Ή*dύρˆΕη·:§ »†θΝ9{ŠpΔ;Έ­Χkθ\δύ«‚ΜΉΏMY¬₯¨ΧΦ#πbBŠ Ί” «a˜΅ΔŒ]ŽeήΫApLΆο&―­­%sε.gδ|1Ώι:0‘bxΜtS>ΨMΕI˜Ξ@νƒϋ’S»7šΨNΰ!œ#˜ρ΄ιŸ!£š’ΨC ίΔ+νx8‚’Ν#—ԏ·πΓ0Aͺ>ψύ‰.kτ a‘^tZΌωΑΜƒ·―g‘ΡΫώΚ–Ρiό]{ΨS1Ϋ0ΖH¨l°J],— žλίΜΚ$‡‰Ω7Γv€εVυ>Ή€yβϋΞΌΪ}£°/IF)6;Κ*ΰ(¦Ω˜t;Σd:OχτύΑ ‰n5ί‰LΚ†Œ‹ϊ-Lz“l“Az G7“‰Ώ,­ŠΰH ΈKέU$YυΰβPN§ulPD1/'Έώ ‹Υ‘rΎ3ήλπtqΘΰά ₯Β€d|ŸΪ ±±ΥŽ€΅˜ŒΒz»οD©ζDƒrΕlvΝ'²,=Ξo©οƒ 4β@λz3ϋw°Ώc)ttΩ`Uδzγ—Υϋ¬…Y›ηνχ6'f­Fπ+mΥ―­PΡ­,ͺ!n.Ÿιθ"›@/«΄σΏzŸ~¬;uVFTW|βήΕJ^MrΊΚπ6–VHpΔϊf‚D³r–•DŸiyw%CγGΚ1!nˆ°ž’5(ηyξ-c2SA”wG)6μ Ώtιm9l•PώA› ηJΧΠ+―”©λε΄υhΐ)%Frn¦ζΠΊ%τU߈Ε`W'Ή@ΉϊΓoΉv€zSz‰΄$ΎZώπ-ƒ-™¨!aJ{8‰@ی+¬eʁ°%O¨πf]»άzϋσ»9ήα]Ϊϊ˜€GυcΞξIH±›=TλN±’ξ…ρΩyί*-G©2A©dΉ¨~ζλLήέ9­+8 ElW3ΝβsΪ­»Ν/hΔMΘX!αcϊϋ·κΝα·“‘¦77§6ξξ±#€€ξŠTΒ¬ΣΞΝA.M-2 ‡3πΉ£Όκ‚ΪMhr€d#€aap£9 „–ͺ#¬Σΐσ6ΚσOπυΗΌ π* ―Νύ:­Xλ4ΗxΛVͺ?gΐ‘φ„‘¬ε3Ϋθε˜Ϊϋ ˜4:œ« ΗΗVw+ЍvΉP€K3F4ΩύDCx…Aό儝³΅΅CΨ‘ή|ϋeg?ΪFΎC4θΠIp':ŠΒΙάSE1ad"g€MιoΔBΦDϊbξΉ₯Tιp*ŠΚŸΥεμά₯](VUvά6΄`šΛΘ n?FίάT`Ι>ζ­MŒ$pΧΤءȍ…θ¦άƒš˜r›yρ‹ίeΫ?%¦sn©=+ΡQ¦‚Ώμ1NA=/]Fƒ³ΘγΚzΠTW†«ΰh²»·œ:a1Z΄φ͜˫οΆήηb\;9Ιφ<œŽ/ 6ΓΓκ8[1ϊā•Ζ‘―j5‘eφd{vΚW˜°μΣ)γOΨDˆω‰ΙρH½Sη6[š# (―ε6ζ‡D†E΄ζζŸAεΘΈ9ͺ27}HΤ~ΐc^ώ‘}1Ϊ8$u“Ή+ό=Ϊ~p¨wΛ,G2•ΩΊ±γςJ„΄Β!₯Ž’«uΝG‚μmΫΖΚyK Γin[yΦV§Ϊ½’/Ν,ΡI:μ? Ν±MY’I ε°"a  |š¦jgt³sΩ'σ]Z »·΅‹χΨ$Οζ·°*ΗΔ$Χή‹ g/–΄+υ4Ϋ>ύ‡Χwk柭vZ‚EL¨ ’φY§ΐ~cλΎΫ`R°–hφδA?‘Ά;d&3 Hj0Ξd!τ±©λ'Πη Φiϋ4…MnY‡ žJ3#lBC¬ψyΠε;ΈΣ„/ϋu.Ξς‹Γ>±π3Χ‘Εή‰‚;Έ'.ρχꍑπ[νΨ\%V{‹Ny}l«W)ςk5Λ\B¦›Φ žεK₯ ]8C`“2$± ωdω‚J’CIθ'±χVU§αΝΐ$V&~›dΌ†υΔ‘Ρυœι ΫjaΩžκY)ς] isΑΗ ³œ8Q@B ·¦b«ϊ*;ΕκΡχΎ&ηPΩΗƒ96LU•ID%“)DBmίc»A₯G ρqΚξ˜Τ‚b-g›ΓŸΐ_)2Α( «ϊ%ΤW\C2ŸK  F/Έ‘φ’]ό]Ίί+μ₯©€t«ΐtΡDCGbr ΓCtπzŒφ}’Z:JΕ€»t(Μ™ŒŠΜΆΐlLΏ§Ξ΅‘ ?rΦ¬Π_ΙTgE!oˆ p—/΄zΫΔπ*;e³}.°LΠ;ώ©7"ΡΆ46Ϋύ°Š½ΕΞ΄Š½u`Kζi£³‡Q$ΏαwΖΖkptR™ΧRΨwW‰“fˆF‘β°@—nΓόGͺœ΄.ΨOΌθJΥθV­H rΘhΣ2Ί`·_l΄Ϊό< £Α-Ύd€Λ f Ήςο`!I>΄MŠΒ+&&,稫c$Χ’άp£ͺΦΐNΊ'ΟΉΦ¦†’Y i=ΖU {xθΔΤvγ=ΦO;ϐEYŠQ畉ŒG턁‘l€MNω'}d_λ²8¨θdΐGΚJ†ς†SY³b°ςοδ΄X-Ζ©ΕUΠΜ i“lΈ0ΰΈ <²φΜ ©5½θ Dƒ+ψγγpeJ΅«"‘ρΧ²0―+’Βp[4 >ίΌ‘vΣή˜ή‘Ϊ3²Υm»ϋΟV[2R–§ςήΌ§ΟnVf$pΑ;zβCο6™b ¨E4[ՈOεΆGϋ,Ν"H#“ύϊ;Msσ_9ΔζπΗɞίp 5έΈžθσm#q1žFiΥTυ3¦n쳆ΰ^,OΥςœδΗΘqϊH) Χ+ο K°k¬P|.P ΑΪtχœ€A1 λώ¦CVέ¬Θ`žΰωϊHݏ˜Π| `Α•/ s$οπΛάΦ8Z`»ΞŒΰΙ‡ πi+JUmή›|ωM±μί3†23*σdΞI œͺή,Ώw¦OΣ-OΔœ:€ώ_[ηԜ ΓΩΨΖ$™ΨΆmΫΆ1±mΫΆmΫΆ“'Άmμ·W[΅υώŠ>ΥΥ 5’g`ѐ.QΗΕέyIb-€Εr˜ίϋǜa[ͺ„ Ω-ϊιŒΑο΄[₯+[„μΙυΜδΉέy ŒΏKόήDφ*%Q5Έ?’γΓ΅—cΞE‚Ζ―u7.”ͺˆTxΨγRζψhQAˆZ. TΛzδ£%cžŽθVΐFϋΠΔ“ŒνOmkΫο+‚+u‹₯†–w·ξ΅)Ξ?βδ‡ά]ώJc“Q¦ρευΌα4ΰ•Θ Φ /*ύcΜθyPΗ•—#J―β{ „ν%aϋWό¬f8¦y亁ΈO'6ΝΧdκ5}ή d·ζΫΜΙ:³¨ϊΈΡa’Άβ°βΨ’σzΟςU–1ΠhG”ΓΆQFž»Κκ„Ε%g'!š†+iΪͺ£`ΌΑkf{‚~Q %ηh07ͺΜDΆ{'Ι:²ϊ—ΰΤπα’už5-Maω&Iχ eš(ώ %Hkw˜Β2;ΎyŒTαΈ1Ν-wθνΡ("SΨ-δ%a!£ˆzΎ>‚¬„ΨΘ³Ζ[:N₯D”V=Σj­;Φύ•K§ŽΩͺτ’γFil#/ΟΈΪρ§΄η7χsRƒO j`b4ίάθωœΊy•|cUӝr[ζ1†ρ»ΞΥr°0$|Bi†Λ²φI΄,Ξed N\Ί[Ρ8sσH₯*b}IU?» Ξώ±ΏUt»ψε,©iMLLυ\"πηJVζ&˜Άΐlš5ΐD ζ“ΔŸ° ΎΘ›ψIc‘›3%ϋξ » φ›YΣp ΑKΡΘlέΠ°Fωλ=γθπjίxψš Ω€Τ‚ΦAdŠ_1›Ÿ9σ μ±'±I\ΦΌrφάγ^)hcΧΗ™ μΣHjΠ'ή3“έΛQ·Θm―šŠ"κ¬PτL‘!πΫ¦Ψ8―ι VΆ)!§ΞV ƒŽ©Μ΄K\©LεύγW;™s‡eŸε|χΰ—/6…‰–ηνƁgt14©Ηa0‘,v©“σSτεύ½ψFJ”Tm τΓO0@+–buί(Lg (Ά±R2˜Ά'{€βκ° Šž2ιEasc*Ψ;g₯Š.ΈS=τ€šŠI}{–=‹γ&ϋζ[°ΆhfY¨BΟH΄ώ4¨l‚§2Pεϋ]ŠΆ]±…BͺΥLξ¨υ²Eΐ|Ό4ΘΒFoί$fž{n[ξΒΗΌςνθ=Ό,Pu$ψ,‰Mm₯ώr5Œ?(Λ©ίJGœša€Θ₯· η™«UΑγκ=§ŠKη½ύ‡Š»'AŽφrςΞΌ.DSΧΠί£ZΝOε ηϊ)Y6ΒΪ£`MMdΐ\˜vΑ!ΊŸ ŽΏέmρ%RωρλΨαR‘pΫΫ ΌιžΜ]-Vϊ ©ίMεbnρ ˜“z/;ξ¦τ¦κζσo7υη7½ρ‰° K§Κέ0ϋb Μ·ΧvWΜqο˜ήv—‰²€Ν V&Υ} `ςciφ$’· njj,(R ³‡ρΥo{,:^Υbax§«4‹Π4τμφP`Ύ(υιαΗ»ΕnΓ=$|κΞε‡iζθ·=\]i q†‘-;ΉpέΐΩ¨{ω2₯― οΣρ5F^˾ω‘Y‘|jV|<9)ρχΎβζ|ΡτοΊ  όΚυˆA’7ρΏ rΑŽ…KTΘXΩ‘yƒΰχŸg—Κaα^" *ΞLxlŒ†.Oœ›)}Ψ§ό˜J/χxB¬_!d_©ξ5‡}_6ݏέ}(πεnρΞΣ0ρΪ›$ΛF\YXŸ›BDτV$ϊdΊw?‡zυς kIVd^y|v&z˜Φ ”αuΖψD2’(ηKαc6Ξ`UŽ;μΚLv3Σ­X££N_z1šcά„δxΣ–σq‘a©$ΌΞΘ•x¦ΗΛƒS)ά/ Ϋό@9ͺΜ[žN’N©ZάΑ 1GΕΠEχ΅œi|9G˜*ΐ8!œiύZ HπφΖFΛωτŠΫΏg.82tqξX„ΆMF¬NΪ! ΕΡ6Δqqμ’±» 3αήDjη$Ω&»³ίs3RΔα¨6}F-Τk§vpΉ—`π:`_Δc•₯cA ΦξΡΔοζŸω5”Yp|Άό/bΨ»/+ˆ%ΞkόeΖ³&œτΊΗ;cmdMΎ‘hboŸσΜjΐ ͺnλ/F#‰#ΒyΠ"gαMΙpΈ”Un‘e,ΏƒαoΒΰˆ΄2ΰ{†lš]XΑΰΙbΙ‡7έu°₯ΊBΠ„ΐόμ $?€]θF2$υΣ Š€”kȞqςΖθΡWεuc£Œο‘2»\ŒMvWϋ_±a9έ82ΐ`nQi”I±Υ₯%"ΑCjΙXρ­Ό9?oτ ΄΄j|U혎SS]Vz.Ώc¬QjΦ3mQεΦΆL:"Ky’4Ι2A0U\œ—νΑWY ΦS¨N\η}Ό˜h-Y―E 'J́.ωβ΅£Γ½ΝQ ¦‘F»‚‡ΆΣΩCH…Ή‘T£ΐ0¦]Λ‘#Α –~{Κ…Άη*β_uΞ*΄1tωc*[εψ”€Hξiό“˜H4nMοCχ 9.uΙzkύz$tτr?Y…Ή+ΖδLqcΰψ~ŸEEcΒNTZ‡a ώͺB>0„Ή6θωƒ˜ Θa[χή”΄ΞΔω“Όs>ύz.„Ά*+Ώ›' φkxΒA-Ί#c&u°7τόiλψ²υ RΪΜYΚλψBΐΣ=ΟΑΔ»ΰ 3YέFSyΆ ,ί93cΤΠΤ°e˜±Β֊ΘC&7γ.yΌ±―°K€^qΥyΦ)Σͺb€bΠ5uFvσ?ζcVΏ1ِ7ΆNƒH™,ZΔFΚς”1hDρ πƒWΊ4y/―ΙΌΔ,‹lπ°αAqχ q,^ΏEE*yΝ:sχπN‘½`F#άύ–ωυκΜ!§-ϊσ*" PΘUH "24ΎJ fuΞ.ΰkπΛa‚ύGͺqTΣͺΣNδI2΅Ι ΏΨψX]¬"/‘yesΨHΗΜI‰·‰4V¨αhρρΆπΐτNgΔΗUvƝΔω ίhQ˜’b-Ύb‹xλ*]ƒxΠt©P:۞kͺ½vœ·NMε™œ•¦ͺUŽQKυ,γœJl μ6ήg!oy`εΝ&ήŠ:GX\ZΚΰ ΪyΩ{₯b…2ίΆ-ιC¨[ΣΞw„άι―εRΒΊyΙ3ˆ₯eςηsπ—œς ,γϊ‚ξΎ,μ‘)­νgjφ―ΌΖ-δΏ΅&κ xbdX5UΧΖWϋA^bjK¬Ρok,δ έ…Ÿ;׌Βx M3—ςγ‡=ήlVψ’'v¬i·Πb^:Z œ$NΛO‘ΝI Hα4!ΫΛZVyσσ|R:ά89θSOƁΖά½0†Yυ΄ 5OήΨh”λπu©EŠΜκϋγ!„dO ‘ΝΏƒQ€OR·†]nu‡΅ΚM€»#M·fμ¬lŠΗψ‰>€~kΡ E ’˜I-L{Ξ]¨Z!»τΔ΄OyΡ L¨ͺΣΠjΕ-½y@p¨φpφJ«γQΧ€Β64r½yΏαάƒ>1 yάΐU]2Ώͺu d·ΑΜ#ΟΤ(μ’ƒ’+jd”7αη΄δ ͺΛlΡ •ΰt³kW}-Χ¦Υ‚{aΆ(DΥ€ΌρKXFNͺB•N\Δ­ 4GΏ 0Ϊ@Ÿ½ΡΩƒΒα>βκ Μ'κŽ?9-5dœy|"QŠ3Χ`tQk2M€„₯W3₯2Y]H­Qΰ^‘wH&[xŒδŒ?bk7'ΑτΝΈxάbZ•–FZ+“³3ς*lCkߝΑΐ₯y}6§»γK)†³΅Kυ¨l'|εgŸ’N\WGΔ—qσψ?&U+^>W =βޜZqνSlF­Εaχ9™ΪΡ€C²Œχ-ΩQ τφFAc£0ΈΉΎψ«Iαψ0―kΊ»Ι+Fύ3>”δΈ!JαΖDόšΗqEq=6;‚Δӏ!Ž%£BϋmŸlEƒ.ζ^qšLYΦ£ΒŸΛFUzSHΛϋ.»k £σ-τ*qψ€jΖΣ― Š˜Ό–Λ„LάΔΦ9PIώ,½‹zΌΒCw>ˆζFΚίVe»³hΈΧm©οm\#du=j²g¨Uδ¬q—œ*&΄Yc EEjOf Ÿ;λ Ν((υ Aή…ηo§ΡF¦ώ8¬|Ae°ΣΌ:IΚe\s8Ήϊ―:)ξ|hωb%bD©Ε‘Ϊ& #'Wqz§"=ΈD™Τ ‘‡έ-¦ηfJΡIzέjF o•φ’TςwPEφ5“&½‹ϋIlβγξΖχ{ŸίV]9ίCEζ―ΠΤ$kθ zS»¬(5ΆΓtz˜fέ­P벚PφΒέG‘F2Aˆ3ήΣGΓλuΠ‚2Γ„r‚Šy$x,ρ²ΛzJNw]fη:δ%e.GΘΊQqMΡΆΌ»g3dπΗTcEα㓦 yRs­ι…ΒόSP ;Θ§ο Γ1OmHh§seV‘'ˆεE%0θt‚»υTΨίρgρPΑ*{ΚR‰5/»ΪΎΙ3Ύ“…Ϋ­%λUA€~θυοfΫIλ―—<Ι« =UΒΒK/ίsfo›Π·έΈGΜφM«½Vuv³Έ?hI ζlνω+B‡Bχ%Qž΄t@Σ†ε>\6‘p"΅rΝ5›υŠ[ϊ]qΖhlΪιότZ1c[±ώοƒ1< »+/ŽΒ›ΆθΏγjzΏκ―Θ$ξ½n$pޟ9 2άa / fb-γrͺΩΣTT€ΩEn…lŠΥkMcŽι² A ,κkͺR֍*ο@¦&8§“}§<ΉδΔ°ζp6«‰ο4Χ8œ!ϊ{.₯φ„β1ί?Ω ΞΦ΄0Ν‹\S~·fχ?8u/Τωΰό"$RεPΞΥώΤ’,eo‰EΫΰZώ~κέKMxλ’u`‡Cwne9Y?X©’ša]ψX5#ηΌqκP0Νb³ŽΌ.…‡šύΪm8ω²q6ŽNQτ©?ύxμ‚«6CΰZΐ½HνΪ“Ϋ€΅Pqισmχi~΄ϋθί«‘)eT…#L΄e‚Ι9?ΪΊq…Ξΐ•’Ιιδ8.LM zΌΕμτ…ωFTΟxΔ¬-γτ<}°ΙΑ‰―›Sό2ΙρNρΚžΰ3,¬QϋδxΏKpYΟ€₯) }>8¬6›¨ϊέMΚ€‘Άjώωη¬NJ ’;Ψ'³CruιΧσ#ηZΤΙoΏΨqό|σ)m!Ζf²'₯λ-Θγ ΅Χ`t’΄6š˜oK=t°§~a9ΰe,ş™‰Μ`ΘCnŸζΞ: ­ΒI΅wlΥΩΚΔ,‡λΠ³ ’ϋHξρ }‡8—«m^΅Η†χ`Π9κΖ9_ŽdΊ­it„~d$γΘσ}ΦΎFώχ©'jΤσ B—‡tE3]2Šν/9Œη~%݈μθ„,²υ‘m‚­^΄&άsήξzΞHΕ#νꞽoΩ§»€+Ηrq©±(Π±UσΨYvԞμ]ΉBŠΛΔ†½Š2H\3~Τušχ`=0׎xO€Μδ· ŽΏ‰Δ›n{ΫMΝΔ=”sw<αφα{j{`LeΟκΨ J­τˊPΉ/€/qjeΛΏ#‰£Κ‚Ζ ?Ξ§!Ζ>uΆ|Sx@@]-€Έ Λͺ>–Κ( ξvΫ‘(γ”vΒW9:t~‘;ŠG–DΰQβ1 ΤΆ§ϊVΆ*ωΈΧŸ6^_Ήρj~‹1:™mΤ”¦C‘ωϋηR· d«Ζ“Ό†iG‘b iͺu†’ͺ“Z‘ ­ρzΘΖcΐ)œ8ΈŒ_(ύΜϋx|Εy ‘b#,΅‰Γ&‰’Hθΰͺ1…R}αJbH_ΜGn}ΜSόω |ξ“σ}ž•ΨV Εαo“ͺDα ΕΊ>ξνΚ.ƒ#$Y ˆ¦¦Kζί¨[†χψάΠ2~Δ„³ΎC¬~ „žΞ¨dF/ε!¨ςz1lόιߘH»φ―⬑ ϊΪpυǜ³^ώhώ³ΈoHKnψΪαFΞΑώ{φ!†9{Κr»‘/f… _y4Ό‚‚hλυ”W>λ\<˜χω*νοδ₯ͺχέ`3Š:,ΤΪ^Χ‹žΘk;μDίυΫι,ˆ9ωm‚CύEΰζ3ΩWηaύ‡5ϋc³iξq2ΥQ€;Ύ―8ιπPΜυTXϋπœ7·G&nΒ3uŸ=΄L(‰"Λα¦A Υ₯Š ˜Β΅σ]»1Ν_¨ }νΆϋΟ"Φl-S!β―5F­jΡΑΛnσs’Βνψ"˜xœgΗ=r<΅όŒ ―Ε[— ΊŒΜ,/T›ΕCg•’~ιΑ ݁‚)ρΙ/ŒsH\F$ΉSΓ »]S‡”₯GͺύaŠqom]RΨ8ΜO0œ\€ˆχ±Ί‘Oτβ)ρΖΡ VJƒΒ₯‹ηOΑ6Σ9xo­X¨™π’CŸ¦ͺ ιώΑ6Χή*7·“œŠΒζݜ¬’Ί\`ϊι jβ‡ε•Xυϊt–%<½Gό»KL“έjΜ`Qj€=BPΞkX—OΒ:wH¬TonξΔv;ΞB‘4_7³ lvͺηxΉ1@˜&^‚Ή2B™ό³yΖ]bΚ’i$7TΚ9Μ³…ΐ)Έ ؈„šοΥΕ TœR™O)Z¦νke›‚©ΤυΜύ†Gg˜Χͺy‹­κ©ΆΫG£‚‚5X{ΰψΊΟ5 Ÿefν]’Λ…I8 '“θ呟μωθΩ( |};›Ρ•=°zILG§pYΗ"r z‹¦BŽIχ›eVK Σk΄θc ΑNBΚ“ωράΗ> „F•štτ/‹`\0γusCρ0AN;n_I%Ύ` :ΣΖ€˜ZiΏ[›ό-Ωv=Ε£#σE–ΘΆ—&S‘ήC₯ο…T%~*ο|ΆΫ*5‘ΧzΩΰΠ.χ[QUP‘_r(­`D,°rλF]ΞΓΣ­Lβ΅΄δ;“Έυ9ύ¦6oρ ρΚ»Νπa₯΅dSš"jT©‡μzβΑύ3™iζ§υζ7kzyΩγȍi1r­Ν]>&άς³ΛM½Οz{Θ„Β ͺλ:ζގhμd!―Μb_@K±L>Z†ŽxυDήΔc–λό‡ˆͺ8>GΰΚcŽœr«ς³NΰkFK΅j±ΕΒ.h»ieb»ͺF„·¦Lj«°ΐ+έΓYwΑž3θΩ…ec…υΦ–ΐO՚Kη₯τΊφˆœΊ,ΞφY›κOνy η'νdPˆό.π&•–›ξlέpMΚ‹’Gλ3⟺zHΘ·@ύ *$hώN©CαhΒΌ…ΘeVw‹”“³s}΅gz’#,αφ΅=ΘފA ΅ zά£{Α,ί6τ ]g>₯Ί§4‹ZHXπ‡dα>TΚ KIΨγWΒ=½IΟX5}Έ ‹ΤΌ=Ϋi…ίΣΟ,šηe@vΉ―Π·_‘εj©0V* OnšF΅Χ)γTή δ<ζ“ο£4Λ„)Ψυ{ΡA‘Tβ‰Ÿh±±Β‚@BnϋΕ±6|ΎπCrΠΞΑ“θ\ψ»ϊiά Db^FŸ6°ŠCΆ£9Όν«Μ뛆۱dŸ\Έζ†:Ν’#°ΘΉF™„αζΘή—έώFη ΈŠ^―³Ξ|ΙZNjNCαΗ`ψΗ°Έn½Cx°B«~k₯7`b@ΐŠ’Ώ’Vθ#π $γH ,œˆωˆ/w€©άn―#2_κπ2†΅τ ιwΚ©“o•§΄Ζ%7aEž4ΏΑ‡8†ΓŸχ…ΘΒwUϋ3NήϊΫG¨Γƒ.€Ÿ:©—‡Φπͺ Λ‚αR•ζqDρƒh{€…,d]ά%Ÿϋ‡Ϊ"P ²*›bΆiρΛ)ڌ9ρϊ2¨ cTPmΫIά‘\"e[⍑_@K{θtψE'Ϋn;Tΰ’€±mυΘή1Jˆzψ>*Α`Toζθ]Γφ(Ρ)ύ–e$D•βΥμ8R np]θ~7ΐΉ΅X+ƒ©$‹.…=Fΰ[ΐš@xMoΨ€κ+dψjΊε_ 2‹_WgήwOΈΊLš΄τ‚‚‘Δy›IΏ›χΩlΓΪTι‚a"IwΈej”rΙ²ΞSΜμ"Ε―ΤαΈi,Ίϋ!¦™Ο†©ύEΨŒx20Q3ˆ5Θ(&F@αΎ>SA½΄ΆΔΞΜ’+=QEH7?&upƒSU8o½@λu·Ÿ9T{©»―Βnj όPν]@τQΗο0 ~Ÿ5NΐΒτIKχΒsΘθŽ*‰έ±Tξ„ ΅ϊeΕπ«ŸϊKΔ_APδa‚ΒΈ―vb3˜IšΕ3I@~Φ†Π λΞΓ1L8+jόXψG`Π(¦-μ₯ΟΚs |'M©΅±OŸΟQ€Ί 2xΙp²*‰φJ»±Τύ—mmg±bRΌ…WΙιZχ'Π€μI»ϋΪ^)‘±…cp4ΝB5Υk’ξμο—ε‚ϋ ƒ•Έˆγ„6{’Ό—«‘Β+sΖδΈ£8ΓΩ[š !ΙΉ3}jV *9ΚΩgeS f'?eC0Wδ*›FŽ³VW9wΒοjDg(‰½ΉσΈΦκ‰.‘θi.šΪ€Φ`]Žώ½¬³]Γt₯€Ρ»ΌΑ‡IΣ]§Ώ§¨ŸYιλnΦvLρf’γIJΚ/7ήeσ²δΈς’6"€@Ύ{iϋ¬ό†η}Ί]ίIlΈNμΘD¨$“‹9ζΡ)₯š^0v2ΥH£¨ny£τh…Y-ΨΌOφBψ0SυϊβS1Ζ#P³Eκΐ7VKGί™ώ3k kό˜ά"ώή.ΑχcςHψ ˜£*†‹+= DΈ ΉmϋΖeRq«€™˜Ψ†‹]­ϋ[Ορ­rۜy°lyt C4VMαπ―Γ8φRΥμ…OςΧ@>7Ÿi“9H΄Ή!?D3ο΄jΨ€Δί’γΡηAήbG†ϋζΝ~Xxy©ΣΙgU²Α(/ⴆ1s|›€ o€|ί‰k;Ώ,©χ9pμXx²E©’ͺΔ ¦ δΐϊ\KUι–4o›΄2”OΦO€ΟWzΜ³Σ|T­μέΌ8ΔΎcυ7‰Ϋ*δό&š@β%ƒŒμ*y+΄ζι] §θλto–0 ~;tΝlΝΈŸœ­Uυ΄0³ΛΥ,n/|ςίδΪ¦J%0ŽŒ’Ι<{!νzaΰΑuSΦ…³Ύ”ι”Q&ŸΥ!Ω Šφ˜>“σ՚UΙ» 5ρ’ήc‘Y voοΡJβtρVeΘπͺ1»βœΦPΝpΨSo.νi\Ϋ;VύF^…OΚ:>μTV]Λsο=u¬Έ.:1ΛMγ]^FψΘ`ϊ Δ΅hš:„¦"1 šˆ5•6„ΛΪ;Α8š3—N8­άܘΞKPB 9ΣΚΪ\3£‘¦'”,T₯Ρ@όD)žƒμΓ”`K‘ZοΊ &œ†ψ€εZ~ε-Η¨šŸ¨ΎMΫΦα„(kΜ"oAω[oϊ!Κ$Γ€½ώΖςxΊ—‡•VωrΗXxΞ+€Ÿ¦tr=’…ζως9ΌΥ&±G“ϊuP.kY‰Œ^$^²δ?)η+Ρά·εΫωEΘ6Υ/΅7Φ zΜ>Ώ[ρnOΙKυ _]ύr ­Ζτ•ιΊγ=DωܐtA{ϊψrn4Š›M’‘”Ϊ}\£Ί[½Ϊ.G]Γ«Ϋ0Ζ•0–κB"RT ΛAδ­ρΙη”ςͺNΨ©Ό½Ϊ±ΆYhψ+αα·€‚ξ«t†"’25–Ξk4r#»yF;±ΈoyΖw_9Ω=υ9š_Θbt N½΅xΟ°Σ’Ί²ΦLq <"Ά`uχωYA§X ΜTΰQQ—y'hv`‘xϋψΗ/Uj3˜ΝYXω―Δdh($ώ“ώΣ»ρ±ΥΣβλο­y’όϋ¦j˜ΓdVΐ3γHξft”)KπC ε'–Ή§«". ’bϋhA;Ώέš(φ ¨οΡΘT‰ΰd\clΊζΌδρΞ­mZv’φIŒj?5°Νρ?Wmι£2Ÿ<%Ν~aRƒΒ χtΗδψ{Π:ΨΧλλωμs”:jCSXŒSdnΨb›Σ5UϊΞ5·,θφMS΄›όYOόzš3•ι Ρ’μΘΓεEψΙ} R@o ϊ'2KέcΓJn›ΌrγuΟ2•υžxΣΫ”¬Cρ¦²ΘρΪ£qΩόΝ2ͺG針Ϊ―Ι‘ λ=πp8ΣΛ.sKων£ϊΚ.)ƒe³{2ΤŒϊ’α<ΔHμ]™N:“τΓ°"ρ+σΆύQc ι!ΓώΛ’YŽ¦Y 6šΙξΈχΛΫ:9cΡωvέΩΐA³“;ͺ€2N„+ρn_#ΩNSR₯YβθψΥ qŸΘ+vύχΆ$RΨ9x@μά<+NΤΔ Β>nO(Ÿ–ZΚρ'YΆΔLE»Ύ­ώνΎί΄βΧοƒόΎ笌ΙGΆMΌ9Ν8³Ζ}‘QFŽE ˜±’ύΐόiϋ_Al?-Š1υ­nΎΠώ@Ψ,§έ„ΛSΗJ;ΝKί/Ξѝf44*δiνšΟkΑΒ#‚ΨVσf‚;T]ώΥ―ίs}’>y€ͺŸα•Rfθ‘½λΟ­Ϊ[― WbΕ¨L‘h9ω»*spλnn_ƒ%6υ-μί:Β~ Ÿ"³jξφοh¨«vKνž<ά}?Ήο˜ϊο(‘Δ!^΅g±yγΣZΟ1cΣαD -C:2_ˆ ½#.*œ"ΤΙ­ q­’MΪ†i³$zΤ’_D9ˆ ₯J .ηΌγ«-jΌUpwΩyΌ‘Μ iΑO? šcΑ kln»MΑnjŸ·XψŸψIυδκCΤQΉΘ‰ζλφΦβϋΛL:μ;nσnυ‹–lεΖ—w.Π#!μIοΚΎJzθνpΐ"݌ΖkŸˆζ.C#{?β.ŒU%αžδfU€8Λ…ρ_H­™1ΎΪFρΆ―Ε―:m£ ρΠ/hMγ`;/Ή·EoζΩ4X’+‚εΫΘ4!Rϋλτ{²έ€»δΫqƒ½€άΒ4Qδƒr7gυΖ<_π½`ΆvL@ƒHΉζl·Ζ8Ψε،+°t,­ΉE~ΓΈ„»8s•Ύn'(φ’ڜ§qlθδ,Fεδκ†f-1c‚$―`Ρ#χ&^Ιc ςn€GwύŠZΤΗV‹1Ϊ”„(Δμ{΅Ό9gV«ƒšΌ. φYΞ άZϊ‘…Ψjθ ΩΏ|Z«-m†ίψΑ4#ξμ’ωƒΛO³£ΔΓ£(yΡ5‘ΊΊ ~†nofβ₯h–λυΒl$|αa~ͺΈ–zεvn:fixοΑ‘cσ5Ł(aΕΖt|³λ2&οA%N ͺΟoαira A«qτω,νπg~c‰B¬¨΄]n!²bπΪ/2₯ψ!.8ΌhƒΆ,qc%YrΧF§€¬·ŠŠŽBθB«)¨MοYδψCS)Όβρa,–kγ]pΩcfδη™ qUAtΛ`,%0εΩ¬Xΰ=ΒηκΑ+ΐz_OwΒρOα kzDqΩ0΅D½΅γΙcΧl!ΈβΨαIλυ\ue5ƒR\’%λt Ni­ ŽΨ·@°2μ}1/ΫΒω‚υg”¨=g=Άg§ΐ™ΟjRGΪ²RY_ί‘£ΝJΌώ„kώ3ςbcπU·ž<†GΥ%±sάΰτ?ξώηNοN ’εlŠ²½…ΚL›Œs™Σ=5¨€(﨡RUͺχ*ŽŸΗ N|z—WΝD―3QZ§Ν§=ͺΒς3X Λξς­Ϊ]·π?“΅―.„5 pΆΓŒΎ4ž‹ )k•ΌA° Ώθ ₯ΖmΈ B4¦ΫRΙP­Ά_ ŒKE‚(›+@«/Ι“»Œ«ή’ΨβOd.λΦ9Ζͺι2–#υδΎύv&z} ²E§Zdο ‰ώκδ!³ƒC†,<…Λj&•L%ξ₯HωΰΟη›Ÿ<׏bΌ’Τ Ÿ‘˜ͺ §WX΅rθzˆp΅ΥC.›ͺβΪ‚)F:ZQ(BB‘“b|ό ΏΒΧY-oašΈaTεH΅sΠi¦Ξ^˜—Υ³Mbκ£nšβγ+ihγΛΠͺ`ΟΎMΊΆ_c­ρ°{ΖܐCœuΈ±ψ‹M5{·3P./žB\ΙξΣ)Ύ¨ ‰εφ­ΖωηZˆCΟ ]η>Π}ελΣ§^mΥΟF OP3$ ΜH¨ξU֞‰:݁₯Υ‰¬c šΙ\Ϋ“VL½¦[ΕΙ{αΟYR>ωt½J{›’Ϊj˜η/ΣZ 1όP+ θ_˜sρζBgΉωΡΓmΕvQ2ϋˆgŒΤ†‡Ϊ>΅C·χΚW»ζΨ ί‘ι1F¦ΈΩ RD|ƒd_Οφ ±‡Σ·1­ςŒ k‡›l{mΣ>ί–eίΫ·—}Κƒg^π>Λͺδ­TΡ· όM:ΨM8¬eqD…δ­ΒΠμ©πΒψΕ"’9£P§ώOU?φ$"Ε"ϊξP֟Ύ«Š¨Χ'gΙΝή§ΆΌ}Σ²Kw€!bω€ωnG(τβΙCΓG½dy¨ΑψyFϋίγ”C―nk鬡–Nm7…$λžρβg©LΤo‘9b!»Η‘ijθξξ½sŠsόjη DX&—ρ–TωZ^}~6‚νϊDλVICƒ^~υCošιη»6–‚Ζξh%ƒώyε}ς­ŒΤ`ω…#†Ζ΄όۘ”ό€―‚ω¦Ψ)^σμ―ΫzΘΒΒSΊKhAQΨώ”&.ώRvέχ£FGΖ/˜Δ Ζ3Ž΅^8ͺi£τΐ}Š5“γΊŒZ΅dp_†,%υ‘r$eCΫοΓ²&€‘YΎΥί"k"Θτ!ͺςΉΜ[ύ[L‡.Χ°z£tλύΝΩ&φ¦0Φk=,«hΑα"†€³Μ‹HžW(ΆB‚]¨{q¦,q‘ Μcτχ’pχtΫ"?,―[εβb¨c™ )w±K―Όγ ¦GLƒe­ζφ‹[(z‘υœ$}γ\©ŒŠζ~mΚPeO₯CϊφμΖg”šΥ/3+†ο\ΖΩψa>k'iΦΓ.— _dνΣΖφΆNɞ•j’ –ιΏ–«jΥtΰ₯μΙπ·6λ°0Δ[»ΙU¬μξNΨΣ+SWΞ›¦Pž Ά—―Λ΅₯{Θ΅“―Šh=Ε¬œ‘a#:Œ₯–HXΎ‘’lθmεΏ'@ύ0nϋ?%ΏφΔeγKΫΦΒΝ:M˜Ρ0–!6zCŸ}\ESL‘ͺΣΈƒ`a-$¦»κ ¦ϋRCέΘιž5σ•NΚ! l>ΤF`|ΘcμΥ›.^ΐkύ6hzHΗ"ΊγžmΉ²dθΟ2Ϊ,@#›‚ΟόžΠΠVξƒšΊ΅τ­Ύώ‡LG]τ»8#uΓχΥή£«έIOΜ[{λšOΪ6ύ’}eΤRΖ:exYσ”2νKΦtΏΔ'ΣG™‘'Ϊ2jˆ₯ήπi‡}£πL6Οε5H"prͺπTŝΦyŸZXۈ’‘Λ3jι­₯ΟCDΟΗ όixy&deͺ‹"\'Ρ8 ^vε\9gg&’Σ-M±Κψχ"ŠŒπ”ΏŽ+fζΝ$ ύŒι{ΐΎ׊dY•ώvmδ endstream endobj 26 0 obj << /Producer (pdfTeX-1.40.21) /Creator (TeX) /CreationDate (D:20210318135639+01'00') /ModDate (D:20210318135639+01'00') /Trapped /False /PTEX.Fullbanner (This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2) >> endobj 12 0 obj << /Type /ObjStm /N 18 /First 127 /Length 1146 /Filter /FlateDecode >> stream xΪ­VΫnγ6}ΧWπ±ΕΒK‘") ‚ΖNœM›[c·›Βπƒ"3Ά Η2$ΉHφλ{†²|Λe»@0&gΘ™s9c2Ν„Μ0aΠc‘΄L2%5‹˜ŠBX™2–‰˜%~τΒP0 ‡Hβ@J&|Ra"­Η_’ρ/dR„K˜”ˆψ2²“Za ρg£ΰθ(ΰΓ—₯cό6Ί€χŠEνuΕ,pέόΞUΕͺΜ\xήpε&yΪ-žΩ(„Aiσ9Š32Ÿm„)±žςψιηe±Z‚ Ž‘Ξg[›ω€ΛtQ-iMφΒxoΐOέ?yζξΞ»Œ_°Ί\9Ώ*€―˜ε`™fŽωΨ„Έ]Ή€X~xφ\Ÿκ΄vА οƒ ;:bΌ/!^3«Ob„mŸ”χ}dαχ7»¬Yqρ$˜ΪxnΛ"Έšτ΄ΟψΠ=Χ4’υΨΈ!ΖψrϊψK– Ώ>KβψΈ1φNcοdΟ~V­™Ί­#K[‚4Ÿ Ιβη,Λιm؊6&Έ#jfBΛlΪ!‚Εωˆ£z―imX¬±RQ?QŠ% Ζ΄» Ll™F¬ΆY²6²γψ%°δΣ‘IΏΚš½u5šοbR Τbϋ\›‡ ϊΚχ=f£DΎΨηŒ₯ρΎDΨ1ωM¨š_”‘ΖžœNŠSρ„HlLΌiJ6Dι·gC^G%Pq²N[Ή™Ζ$˜E΅‘]+° %q±Ε©Q°–Φ Ζ}œ0#‚ψ?7Dm£ ₯ΩΔ·VX/Dλ$ƒυjJί'aΌb±}C17Šιψ•b^©΅b4¦Œ„‚ΖνάΆoαkQϊ€d‡ΦΓ?ήΉ¬¨²O]••ω².Κ¦―Σ'xΎ\έτϊ'ŸzWΓ‘ᘧӊ©fFΧί[Ε:’6'Ζ$wΦI•Ρ…|ΈΣε—Ogλ!%!_·hΐ/κtžg'‹ιά1ΤξιOΔΐΥ±^£" 1KKΊ-~β)ΟΈγ3>η_ς’WΌζ«Ÿ0ύAD²½ΏΟνλ―η§—Ÿ†ξώό₯ttaΟηi§[Μ'o₯νD(b|Xπu‘BnΙΖRχΘσYA2lΨj΅ΛφŒ_ράι)Ÿ€υ”ηΰ½Ψ2ίε-ŏπάό~ύΧ!ο;7]ΝΣςκZ°Ž‚ΌB ωŠΊέ₯‡r—Ί}oŸ­Ϊa’]ζ]~ φ—ΰΝoω}ΧψΠαϋOZ<ςΗ‚Μψμe9s ―ΜΣ6|şω·=…’Χ -²b’/¦€›?>:ϊhβΓ<α±λθ¨IήeΖ&\kX7όw{‹%ρ<Β–£²νΎŒͺCώL›ΟΏ_οZ=Τ~HFœ€nZ9υ}U{›L―LθηeU“žπΛt=£―ω€žUτώyσΘ|”όΓ‚9@"‘P]ο 1[$λ·Q»τζϊahί;ΣθΤ!:!wΡQUΆθΜ@Gω*zν­θ­πίς ’Wψ`{)j‘˜Ν’φ]‡)JF΄Μ endstream endobj 27 0 obj << /Type /XRef /Index [0 28] /Size 28 /W [1 3 1] /Root 25 0 R /Info 26 0 R /ID [<5741E671724B55667B88294A65D6929E> <5741E671724B55667B88294A65D6929E>] /Length 92 /Filter /FlateDecode >> stream xΪΙ=@@„α™υΟ²\Cγ*άΒ 4"K#z‰ŠΜΧ pytorch/docs/source/_figures/build.sh000077500000000000000000000002251444262304300202400ustar00rootroot00000000000000#!/bin/sh for filename in *.tex; do basename=$(basename $filename .tex) pdflatex "$basename.tex" pdf2svg "$basename.pdf" "$basename.svg" done pytorch/docs/source/_figures/graph.svg000066400000000000000000000235211444262304300204300ustar00rootroot00000000000000 pytorch/docs/source/_figures/graph.tex000066400000000000000000000005471444262304300204340ustar00rootroot00000000000000\documentclass{standalone} \usepackage{tikz} \begin{document} \begin{tikzpicture} \node[draw,circle,label= left:{$x_1=-1$}] (0) at (0, 0) {0}; \node[draw,circle,label=above:{$x_1=0$}] (1) at (1, 1) {1}; \node[draw,circle,label=right:{$x_1=1$}] (2) at (2, 0) {2}; \path[draw] (0) -- (1); \path[draw] (1) -- (2); \end{tikzpicture} \end{document} pytorch/docs/source/_figures/graphgym_design_space.png000066400000000000000000010020051444262304300236310ustar00rootroot00000000000000‰PNG  IHDRΦl¬?©Ω&iCCPkCGColorSpaceAdobeRGB1998(‘c``RH,(Θa``ΘΝ+) rwRˆˆŒR`ΒΐΝ Δ’ Š‰ΙΕŽ> @£QΑ·k Œ ϊ².Θ,LyΌ€+%΅8Hβμδ‚’Ζ [ΉΌ€Δξ²E’²Αμ vЁ@φ;Β>Vaί« r²?Ω|I`6Θ.Ύt[Δ†Ϊ ‚Ž)ωI© ίkZZZh’θ‚ $΅’D;ηTe¦g”(8C*UΑ3/YOGΑΘΐΘ€ξ՟ΑαΙ(v!†±9 ώKXώ ΔLzθ00πOEˆ©20θ30μ›“\ZT5†‘Ι˜“JRb˜ξ° pHYs.#.#x₯?v@IDATxμέΈΕΪΐρ—^€ƒ ˆ(’6,ΐUΑ.\±‚]°{m¨€½ ‚ ±TlΧ.°`/DQTΐK‘ϋωζέϋεpNNvv“l’έδ?Οsž“dggg³™d2;3εŠL € € € € € €XΚ[·²@@@@@@:ΦΈ@@@@@@π!@ǚ$’ € € € € € € @ΗΧ € € € € € €>θXσD@@@@@@θXγ@@@@@@@ΐ‡k>ˆ‚ € € € € € €k\ € € € € € €ψ cΝQ@@@@@@ ck@@@@@@t¬ω@"  € € € € € €t¬q € € € € € € ΰC€Ž5HDA@@@@@@€Ž5@@@@@@|Π±ζ‰( € € € € € €Π±Ζ5€ € € € € € €€:Φ| @@@@@@:ΦΈ@@@@@@π!@ǚ$’ € € € € € € @ΗΧ € € € € € €>θXσD@@@@@@θXγ@@@@@@@ΐ‡k>ˆ‚ € € € € € €k\ € € € € € €ψ cΝQ@@@@@@¨υλΧΛ?ό`έ‘Z΅j¦Mk6"/0i$ΩΈqcόΛΕΟ·ί~{iΨ°aρs ΐ‚ δΟ?L7gΚ•+K½zυ€~ύϊR½zυ@$‘Δwήy§lήΌ9ρFσj—.]dύχwέΞςM`ύ’%²nΡ"ΧΣ*WœΤhΩu»n˜fήWE–χUσΎͺΟϋΚjΘFςE€φ^Ύ”d8Οƒ6_8Λ%_sE{/Ί%K›/ΊeGΞƒ ½Ό))'PΘ„ΰ’Λο”f̘!-Z΄°žδ»ξ*?ό³5ˆhΠ ,^Ό8ώεβηΓ‡—‹.Ί¨ψ9HWΰώϋο—‹/Ύ8έdΚμ_΅jUiέΊ΅ΣΉ£<όη?ιl+£”ϊ •*U²vΒί|σΝ2xπΰΤΐžy%°τ›odΣΪ΅ΦsͺΪΈ±ΤΨi'kœ0oό妛δΧλ―wΝb9σžιinŒ²…7Lœ"ΛΝ-»šχΥ.Ό―l„lC ohοεMQ†ςDhσ…²Xς6S΄χ’[΄΄ω’[vΩΞ9ν=Ϊ{ΩΎκ8^IF¬•Τΰ1 €@Zk͏ψί}χσ7bΔiΤ¨‘\sΝ5Ώ©R₯JZi³3ψXψι§ςŁzξ°ΓyηIϋτŒG@@hοq Ϊ{α(rQΨt¬vω—:{ύρ{υκΥ₯^+ωδτΣO—:”|‰Η €€U`ήΌyrΙ%—Θ°aܑTgŸ}Ά”/ΟςžV46"¦ΐζ δ‡ .H3vGΘ7Ϊ{ωV’œΉ ½—ϋ2 …'@{―πʜ3§kα,—œδκΡG•%f­·pΐΠ±ζ†Γλ `˜5k–3jν35Φ5Ίξ2#πΗ}χɊ)S2“8©"€DV€φ^d‹ŽŒ#zΪ{‘/"2˜G΄χς¨09•H 0l ΕGζ@h <ώψγYΫ-Z δΜ ¬™=[~ΉρΖΜ€”@@p ½ηΓΛ$@{/ H’A :Φ@$ @ΊφΪ€όο@Lπ-πγe—Ι¦•+}Η'" € €A Πή R“΄(-@{―΄ΟΘ₯kΉΤηΨ €@ :TFŽY gΟi#ωοΏ/s^~93‰“* € €>hοω„"IΠήK‹¨dA€5Φ²€Μ!@( \uΥUΉsgΧ¬Ι²eΛdΡ’E2gΞ7nœ|ύχ’―ϋ ”ή½{K5όD'Žψδ“O¬ΎΫoΏ=N*°iέ:™|ΡEzφιvgχUuήWι³7 €„R€φ^(‹…6_8‹%ηΉ’½—zΠήKݎ=νt¬Ω}؊¬ΐ{μ!GuTRη?wξ\ΉξΊλδΙ'Ÿ΄vi’σζΝ“!C†Θ­·ήšΤ1 9ςPΘ§ΟΉ[~6LVN›f‰Α&7ϊΌ―άhx@ςX€φ^8 —6_8Λ%ΧΉ’½—z ΠήKݎ=νLiχa+ €@Ϋn»­θ‚Υcǎ•J•*yξyχέwΛ_ύε ΰ.°jζL™š ƒΊ|Υͺπ°Γάwd  € €IΠήK‹¨$@{/ H’A `F¬ š©δ–/_.Ÿ}φ™LŸ>έω›5k–4jΤHZΆl);ν΄Sρ *d* OW§“ϋβ‹/δ·ί~sώt4ΛW\!]Ίtρ}μ?ώψC>ύτSωο+‹/–%K–ΘͺU«dλ­·–νΆΫψOέτ―P.[JzυκΥςρΗΛηŸš?Ύ3aύϊυE 7–]vΩΕ©U«V­-;fαQο,dΣΧ!=τPyΰ€_Ώ~ΦψkΧ•—ΝšP_|±5žmcΛTλ˜_ύU~ύwηOλ0}ΌbΕ §Ξi€‰ΔώvάqGgzΝςεΓy_‹Φ»οΌσŽΜ45Ϊα©u¨ζΉmΫΆ¦MiέΊ΅T55a…πΉθΧyς%—Θζ5kΚDίι²Λdƒ)Γ¨„ ζ»Ξόwί•ζύ΄Φ\λ,JuλJυfΝ€ζ»Jγ=€B΅jQ9k>W›ο)›©sW™οuϊ·Ι”ŸN9©΅wί]j·kgέ?Άqσƍ²Θ|¦­˜:UΦ̞-λΜ{ΈbνΪRΝ|ηipΰRwο½₯\Ήr±θόίhκ<-£U¦ž[iώV™:Oo4už·š©σͺš?ύΏ•©?˜)…Λ…΄Ξ „DH Ÿ>ΧrύέΜV$ωτ}»δyήΫ’Αυ·Ε"“hοΡήΛδυ₯iηΣηbΊV΄χΜΝώQmσΡήσ½Π±ζί*˜kΜ'^:ϊƒfŊ+šΏώ[†.χάs,]ΊΤš‡]͏N·έv›τμΩΣO7κ:=?όp©x+W,υ<ώΙ½χή+―½φZ©—υX'žxb©Χ&Ož,Ά‘λ{ν΅—Σ±ΫIΏψκωέqΗΞΡ±ΧυΏΏδ΄“ρφΫo—·ήzKτ±ίΠΎ}{9ε”S䀓Nr~όφ»_TβeΓeβΔ‰rΠAYI΄³χΟ?”Ϊζ‡ΎdΒgœ!―ΌςŠuνˆmησΗGΝλM7έ$|πhgŽW¨R₯ŠhcA―o½F’ωa2Ϋο―sΙΥφsΞ9Η™κqƌΦ,Όϊκ«)u¬e³Lΐ:³¦Υ}χέηLe©_ϊ½βCσζΝεάsΟ•³Ξ:K4hΏΩϊΌιdΨh~Hw Χ^{­ 0ΐm³λλ―ΏώΊσY£7plήΌΩ5žΎ/tͺO=FμsJί§ϊ~u ;v”?όΠm³„εsΡ5ƒ!ίπ—ωμ›7zt™\V5­Μ:†?™TΒυ•L3ΣΒκbάEλΧ»f·R:τΤSeΧnΚζ½TxΫ€₯TnaσΎΪΩε}΅δ?‘ρΗηΆ«TkΪTόϊλβν+M'Τ―7ί,³ž^dΣ¦βΧγ4ψΗ?€₯Yη²ΡGΔoržλ]«Ώ›οN³_xAΦ/\˜0ŽΎXEo1ωίΑάδP.Ν›―t]‡?L§£#7ΊΤyΛΤyΥM·ƒ©σΆ7u^•$λ<Χc!ΛηZ&Ϋ{nόΩόnΖχmqΪΎΩhG₯ΝΗυηφΞΜάλ΄χD‚nοiie’ΝG{/sοƒL§L{/}αtΪ{zτBiσΡήKαZ+"ψ0£ΕŠ ±υΟtnYΣ3#¬ϋkϊ6lp0?DΥ©SΗ3~|žφί’/Ώόš'žx"ιt㏣ϯΉζš2Η™4i’5mσ#kρ>ΏόςK‘Iζ™gž)Žΐt™Q.Eζ‡^Χύε9ώ53r€¨[·nES¦L‰?D֞›ΡRΦs0«Ύσ’m—vΨΑšwυώχΏν;qΣ¦ME¦ΓΑšι φ•¦ι`,2΄E¦c̚^όuQςyηΝ‹¦M›ζλx)[οίJQ―©’η˜θq²ε–ΰ0Eƒφ<ŽΎΝH©D»'|-eŸΣXΤ’E ΟsKδ{MλΎ}ϋ™Q“ρΙ»>7YΦcή|σΝϋ&Ϊ`F₯υκΥ˚f,Ώ%›ν’~ψΑIς…^°ξ―ŸIΆ–ΟΕDy4#‹zτθaύ3λ&Ϊ5+―m\½Ίh¬©ƒΝ-/eώώ5ΚΙΓ€sΟ-³­dόI睗•Ό&:ΘΊ₯K‹ΎλΧ―θ5S?—Μ“Χγ1͚-ϊκ+'ɟoΌΡΊολ•*%:t©Χ^7ο+Ϋ1±Ό―|ς‰uί±Ν›kκν·½V‘‚5~|>~0 xΨƒ9oΌQτ–ω~Χφ|\‡EkΜϋ=Υ0ΫΤycMg;†ΧΆ7Lχ©σΦ&Qη₯š_φC UΪ{eΫΊ‰Ϊ{ρΎΉψn…οΫρNA΅ω²έήΣσs›λ/ώJϋίsΪ{Ι΅αγsΥήΣ|Ωζ£½_²eŸ‡ΉΝG{―¨(Χν=½b ‘ΝG{―lέΰη•pΞEe~½+τ £jŽ>ϊhgŠΊd-Ύ2w~Γάι¬Σy…9θt‡rˆΜ™3'ιlκH>M€£ωtΤH:AGgθh IχΤSO₯“TΞχΝ…‹Žζς oΏύΆW”RΫΏϋξ;ΡiZlΑΟqΏύφ[ΩgŸ}œ©M…hKΞΊM§Τ‘qρ£<­;ωؘΞ{ΐGς‘ˆ§OΟ|ΔήƒžM„\—©Žv<ςΘ#ΕtF9ΣςϊΙ³[­»Fε\£?ώψ£[΄Œ½Χίήfš81˜l0jNύ=wξάdwM9~.>΅NmFƒΩώ&L˜ς9₯»γT3Κku‚‘υφΫOššΩa:Νγg:ΙΜG5έ‚ΙΥΟkΜ΅ϋΉωž³ΘŒ°ŒJ˜vη2εκ«­£ΤΛofΏίΜZ”± ‹–7³lXΆ,φ’―Oš$Ÿ›ζ:Εf2a“©σΎ2uήSη­6SV¦6›:o–©σ>6ŸΛη ΞK'ομ‹@&rρΉ–‰σΘυw3―sΚ·οΫΉhο©±ŸΆW.Ϊ|\^ο€Μo§½ηnL{Οέ&Ρ–\}.†ΉΝG{/:ν=½¦£Ψζ£½—¨6ςkώ­²SΏόλτs~¦«sΛ”NvόρΗΛT³ζFƒ‘$'œp‚Μ6λ$Φ›i’>ψ`ΡΔ ƒNΫ’Sny΄₯•+?¬1cΖX§—‹?―±cΗΖΏTζΉΧq5 ]ŸOΧ‰ "θυqήy牝Drf°Τ߁d K‰΄jΥΚιΌρ:œ1κEr]¦ZfzέιυdΠ©R?όpg=³ Σ΅₯΅ΠL§7'€RΗ]`:Fτ³Κ65e,nΊ αs1Y#RP;]Κ³žΦξfΊΎd¦―-“F†_XgΏΟΝηψ ο{·¬˜αύ2ήL½&‰) έΚτλ‹ΜΝSΉ&εΓόΥ¬ίβχG§Lδ!Υ4sε¦MΩ}χέ­ΩΦΡgΙ”·6’lAΧΗΫe—]\£˜i|œόυ=t8σΜ3“:·γ§σpK3¬―kGΈWπκX C™κzjoΌρ†Χ©€΄]G~ώω)ν›ΚNϊ9č:šSΧφΜt(„ΟΕd Έθ"Ρ@ρ‘Ωι§K]31ΜabίΎiuͺΕΞm½ιάύσρΗcOCωίLί"zΎζξ’”σ·ΩŒΣςΦN.ΫΊl~°Ψ¬MϊίgžρU~7uή_ͺσ֚:οϋ,ΦyΎN˜HdQ >ΧΒπέΜ«ΘςνϋvΪ{κΆ6ןΧ՟έν΄χΌ½iοٍςαsΡ~†Ιo₯½'…φž–lTΫ|΄χ’_ΖοAΗZΌHž?>°\˜5’‘XFJ$€w’\wέu%^ρΠ¬Ε#·άr‹RˆωΗˆώX₯kύδόNMͺΓπ½:αlΗΣ;9u4€¦“‰ #IωΟ¦4Mk,?ιΌbiDι6Ϋlγ™][ΗZΚΤ¬ƒ&7άpƒηy4jΤΘΉ>ϊυλ'™8β1k±ω=€υ΅™ίέσιFΠγθτ·A³VfPIΉ¦“οŸ‹'ξ²aŽ™ΎsΑ{ο•ΩZ±fMΩ- eœΔ 5S&Κ{ID*ͺ6u„VΊA;ΈΦΝ›—n2Ξώ˜i΄½‚YM~υQηU1u^cσ™ΨάΤy-L·©σͺ›:ΟTz^‡9¦.Z™…:Ο3#D@ Q\ Γw3―bΛ·οΫΉnο©·­ +l΄ωΈώbΪαωO{φ^ΊWcΤ?Σ=ψύiοΕ‹„ϋyΫ|΄χ‚Ή¦*“ ©dJ R₯Jrμ±ΗJΧ]₯iΣ¦ΞH6]Οελ―Ώ?λΊθΪEό±τΥ»₯K,ώK―vF˜…KΔ*ύπj3ύPηΝK½ΈγŽ;–zξηI:Sσ=τΠC’?pΫBΥͺUEσzYK€eΛ–R―^=YΌx±˜Eε­·ή½ΣO§A³]sH§ŒJΘ΅‹Ng6·ŽτΣ9χo½υVORύΑ_§mq :½™Ο-<πΐ2qβD·ΝΕ―λ(»3Ξ8CΪΆmλt|Μ0λι:W<ςˆxuθxw›5onΊι¦βτ’yΞ{ ™γ„%Ÿ†–Ν$ eͺk‘­X±Β•Tλj½vt}΄³ώt3’Hο΅…'Ÿ|2£#ΐtηe—]fΛB©m͚5“Φ­[‹–α―f9][-iŠK%žβ“L}.¦˜¬οΆΡά`ςγ₯—&Χ2Ωή Γw3/~ΫwK―}ΓΈ=Χν=5 K›λ/|W(ν=qnΞ¦½Μ΅™©ΟΕ`r—ωThοΩ―Ν| €„(΄ωhο₯_ΚkΑ8f$ύΑ_;‚Ά7?œ” ±;Ε΄#γφΫo/Ή)αγΟ>ϋ¬LΗZσcŠώ• •+W.ω΄Μγ:Θ‘fϋLύπΤ/dϊW₯JgΔQυκΥKξΣO?-υ<ώI:uœNΗψi·έv[ΡΏŽ;:wΪuκΤI–.]Ώ{ρs΅₯k½F8ΰωΒL1ε&Ožμtnj±-xM©ΗqKC;δtΊ>―  :}^g±°Γ;8kχι:jΊνζ›oΆ §ΗΉΤόΈ]·nέXiχσHϋ 9H aΓ†žGΥŸ fΝ$5(ΒR¦ΪΩn ƒv:jέβθM Ϊ)ΥάόύnAoxΘδԊ:ZM§!ρ ΊVΒΠ‘Cε裏.5ΪNΛH•ΦF£Χ ^ΗHe{&?SΙO.φωΥtθ'ZWl+s#ΙN—\’‹,ω>ζ¬ηžσΥρSΙΤ«{>υ”lm¦‘­X£Fqϊ:Mσ¬gŸ•)ζζJ0J‘œωŽ΅—™‚q;³ΖYόϊw3ŸxB&}ΆοΣ©\ΏΎt7Nj·kWjŸΝf:λρ½zΙΌΡ£K½žθΙβ/Ώ”&ζ†*· wΙΪB+SηmonNq LΧΥΤyο™:ΟΦΩ7ΟΤyt¬Ή)ςz!dςs-Sν½°|7Kυϊˆκχν\·χΤ; m>ΏT―όΜξG{Oœ›Πiο₯eςs1ύάe'Ϊ{ΡlοιΥ₯6ν½`ήΟLŒcΰ©θ—V퀈οT+y !C†ΘΉηž[ς₯„u€V˜ƒŽ {ι₯—œŽ.Ν«Ž6ϊόΰσ“Ή3½—ω¨dψώϋοK>-σXG‹ΔwͺΕGΡ‡~xόΛ₯žGνΗ0ΈΔ:|KAΖ=ΡQk^α½Σ›•άΗvνόΠ…mα”SN]o‘d§ZΙψΪΑ¬Sώι4~Ά°άŒΌxπΑmQ|oKζ=ΰ;ΡEτs£fWMγCXΚTG3Ϊž{ξiΫμl«a:Ž7?ͺΫ‚Ž ΣΞ«L…'Μψ^‘ω±^GΨuοή½L€ώ₯7X|kFΗμεuΌ ιsΡ­Œ–ό³ό~Ο= 7·5υZydξ˜ΕgϊX­ŠΉΉ¦³Ή‰¦q₯:Υ4›Ϊ!Υ̌ΐοfjy¬ν™ΕΣς<”6°φ}ύu§#+ΎSMwn~ΦYΪLjn[ήάΡL ί©ζl«XQφ1ί§κν·Ÿ>΅†D³%wXξQηΥυQηi§¨v$ΪΒ SηmΞ`g;6ΫΘ΅@T?ΧΒςέ,Ως‹ϊχν0΄χΤάΦ‹•I&Ϋ|\1εpύ§½χΏς ½—ήuΥΟΕτΞΊτή΄χ’ΩήΣRŒZ›φ^ιχ^ͺΟθXKU.Γϋέ{ο½R«V-Ο£\uΥUžq–€8ŐgΒDΈγŽ;δ“O>‘ή½{ΛV[meMQGyΜ6SΩ‚kδ'θθ$[°f³ν—‹maqΡ2¬h~Τ³…ψιGγγώl~,΅ukϊΗw\όnΕΟ_|ρΕβΗ‰Τ][† –hS™ΧtԐΧT§^£λΚ$šΰ…dή vΔK~ZkΦ¬)s>a(SRΧ΅°―αΨΎ:kί}χuύΫc=$SϋΊ~›ή΄` ϊSσ­·ήΪΝ5:ڌŠ©V­š5^ εsΡfφΓHQ‚Nˆ†‡&MGh˜Γί¦£F§ ΄†ςε7ˌط…J¦.ίέw₯rƒΆh‘ΩΆΉ™§‘Ηˆ–W\!ε}ΌŸZ Μ¨{·PΑtΌmg>½ΒΊE‹\£l0uή&:oΎΗM0±Δ›™:―©σάώκ˜:oν_Ε’σ‚ˆκηZΎ›%{‘DύϋvXΪ{κžλ6Χ_²WvβΣήΫβL{o‹E²’ϊΉ˜μyΪβΣήϋŸNΤΪ{šλ(΅ωhοΩή…Ιm³ ž\ZΔH@Ώ”όΣ,ο'θ΄bΪI€λCΉ…eΛ–ΉmΚιλύϋχ—ζ"ΏA§€υΡά‚vΜι°q?αύχίχ-qΒβΐόΐ©k9Όk~μt }τ‘hη‰Ϋρ^έΊu³ώΰοΥi wX6ςΉώNCͺ# oΌρF·ΣqΦ9Τ) γ§,uέ!nC²οΈέ#σT―Q?Aοπ‹a(S]·QGjΩF’έύΞΪ€—˜©ψvήyηψΣ(~ΎΟ>ϋ8ΣΥΏΕΆ©ZcΩ8ˌœρυ‹«ο%½™ΑΟτ«±}RύŸ«ΟΕφf-,ΫΪ£z>^7€zΞρϋΝzώyYdnD‰εLgh[—Qlρqsω\§τ Ϋq„ΤΪm7―hΞφjΫm';™i}8ΠWό\FΪΖ|6zmXΟΤ‹<¦Όg¦΄φ ΅Mg•WΨΌnk” ¦ΞΣ5ΤuβΖvšnκΌ"3ύςŽ¦Ξ«i©στœ4λΘ@ ΄@>ΧJη"΅gaψn–LΞσαϋvXΪ{κžλ6Χ_2WφβήΫbM{o‹E2rωΉ–6ν½WL”Ϊ{šσ(΅ωhο•ΎΦyFΗZ:zΪχ0sηy’©‚ά§@ΆŽ΅Ν›7»νš³Χυ‡HΚ2™ _βΟ9ηœdv)wΜ9’+n2Ω0|ψpωπΓ“έ­Tό»οΎ[vΪi§R―₯ϋ$Χ.%σ―WΆŽ5νTgΦ…Ρu›―`Ά©Gt4Ξ‚ %[όΪATόΨΟ―©ξΦ―_/_}υ•h‡_²!•χ@²ΗK|ۚb%σί±–2ΥN΅–fύ*Qi :5¨ώιϊdZxΰ’ΣΥ7λ!…!Œ?ή3Ί(}2AoŽxψα‡eεGϊds‹›«ΟΕςf•Χhj·<ωϊ3Mꏗ_ž0Ι^(΅vέ5αΆ0½Έμ›o<³Σβόσ=γ”Œ ηkl^»ΆδΛ‘{\Υ¬ρκ'θ4˜^‘Άιμυ u|Δ±u¬•7u^ Sη­π¨σf˜ϊNj˜:OGMnmκΌϊ¦Ξ«’:ΟΛ‰νδR WŸkιžsXΎ›ω=TΎo‡±Ν¦φžΪηͺΝWןίk;lρhοΡήKχšΜεηbΪ|΄χ_AQiοiξ£Τζ£½—ψzKεU:ΦRQΛπ>»'ΉnH½zυ2œ£ΰ“Χ›λΦ­|Ÿβί-Σ§O—™3g:ŽϊƒΈΞK™†ΚοέTρ™›4i’ΌυΦ[ρ/'υόΊλK*~Π‘3αR2:RGoι(.· sξ'κXΣN·Ο>ϋΜm7ΡQC={φtέT₯S¦LqM#~ƒŸ‘hš^*k™~ğK.Ÿ/²L9Λ—ŽbŒω¦2έ͌’ρκX‹ΛΤ©SEτ‡½IbWΣι±ώϋ;]»v•f͚ŒfυΏ—§^οšΟdΒΆ¦Γ@ΟΟkݏdL·>wμ΅_ΜgΗΊyσbO‹W6SvξrύυΕΟΓό`©Η -•6t¦ΟHζ*Υ¬):‚Λk”W2if"neŸM:ε‰-h:zη¦W¨ΰ1΅Άξ_δqΣ•ŽτκX‹εc₯©οτoΊ©σL₯'5MPΟΤ%υΝίΦ¦Ξ«ž£:/–?ώ#F¨~y}—Pλ¨ߎz›/Σν=-γ\΅ω αϊSί(Ϊ{΄χ½n£ϊΉ˜ξyΗφ§½“(ύ?*ν=ΝuΤΪ|΄χJ_k©>£c-UΉ ‘dB… ’‰ŠΈ©tDΈe|γƍΞ((ύ’­gΪ²ΰsK'ί^Ο…‹Ž8κΡ£‡³F“›§ΫˆΆOΝτWk-£΄3¦ωΥ-xVΣύ>ψ`·έS~έΟq%δ{ QϊazΝOC+Ρ ~l³U¦Η{¬ΌόςΛI³jGΎvΘιίγ?ξt΄u4'œp‚³^`“&M’N3ΥζΝkέU3•ΝttΙ]d:Σk…πΉθζώχ?Θ#F$άάϊ–[€r: ·…νΕUζf[¨±γŽRΌL6Τοά9τk:­’ŸPή¬f ULηcPΑkf„mM7'…:Οά½δtΘi§άŸ¦ΞӎΆΊ¦ΞkbκΌνΜ©Υ²XηeE:dB ͺŸkaϊnζ§\ςύϋv.Ϊ{Ɜ6ןŸ«>7qhοΡήKχΚ‹κηbΊη­ϋΣή³+F‘½§g΅6ν=ϋuηwkςΏ`ψM™x) $ϊ9εΔBΊ£N_™nΠΡhƒ ’¦M›Κα‡.ƒ–Χ^{­ΰ;Υrνb›QΛ|Φ¬Yςƒω‘8>€3 €¦΅tιψ$³ς|αΒ…)'ˆχ@JΞΑNZζ^!ΡΊ^a*S"±W―^^§αΉ];ΪώσŸΘ₯—^κŒ\Σu ϋν7Ού‚ˆΰ΅ήfͺSVfc^!|.&*c½^ΎΧιΝZVρA§l~φΩρ/‡ςΉΕ΅iεJkήͺ™ΟςTB5FC₯ΒζtxΩvlbκΌm¨σ΄£m©©σ~4uήXSV_˜:oe–κ<Ϋω± \ Dυs-LίΝό”aΎ~ίΞu{OνsΡζγϊσsΥη&ν½-ξ΄χΆX$σ(ͺŸ‹Ιœc’Έ΄χ©”~φ^i€ž™›έν=7™δ^§c-9―¬ΔΦυ|ς=€ϊnΜεή{wήYn»ν6™—`z¬XΌBϋΫkšO2>Ψ:Φj›ι±Ž<ςΘψ]J=ΟUC+Υ΅₯}”:ω?ω裏vνΪ‰Ύw3Ή¦h›’UνS½&uΔZ¦C!|.&2œϋϊλ²Δ¬γ˜(μ~ί})πJ”V¦_Ϋ`¦gφ ©v¬UŽΰtΨ^aΩήήΤy[Ή.¬©σš:oœ©σ~7užΧt”aq dB ͺŸkaϋnζU6©~·ρJ7—ΫΓΠήΣσΟE›λ/—WžύΨ΄χϋΠήKμ’θΥ¨~.&:—d^£½η­E{ΟΫ(Υ΄χR•Ϋ²SAn±ΰQΌ:^ά²’_Lϊτι#Ο?Ό[ΟΧ΅Υ’E ωζ›o<γF%B˜\t*ΉγΜtO=φ˜+Ÿv¬ιhΓXψσΟ?5©bΟγsΜ1RΕcŠ,―Nƒψ4ƒzžjgHͺς­ttŠš?ώΨσpmΪ΄)'leͺΣSόψγN‡ώwά!ΪYDΠυuΫϋοΏ/ϊήΠΕ“ƒ›ŒxŠ?FόwρΫݞۦpuΫ‡Χύ $ZWMχΤ5΄΄cBΌΒίf}Q[˜?v¬όΗLϋW2΄0£δvλVς₯΄ϋιX«h¦N%Tτ±žX*ιζύ>–»cη^ΕΤy]M7ΥάΔ4ΝΤyEΥy›M§#Ψ˜:o?Sη₯2h,όGμ „ν»™ΧΩηΣχν0΅χΤ=m>?―+>7Ϋiοy»Σήσ6*Τ΄χΌKžφž·‘k 6ν=W9ίθXσMEΔ R½eΨ°a)uͺ鈊.]Ί8klιtnÇϫŽ΅°ΉθΤ ΆŽ΅ &ˆN‘ΈυΦ[;—•m΄šF8逓Φξ[λ± [&Φώυ—Ϋ&^· x4²b»V0u^kSη5575M5uή_¦ΞΫP7ίΤyΪ9ά2u^,όG`ΒψέΜv†ωτ};lν=uΟv›λΟv΅ηnν=φ΄χό[zLΪ{[Ϊ{[,’~δ£ΝG{/iΥR;Π±VŠƒ'a˜6mZ©QNΆΌΖ:Ρ8ΰΩsΟ==§&΄₯φmatQνΆΫNζΜ™“OGyι—ΚSO=ΥΩώή{ο%Œ§/6lΨPΊvνκΊ=ΆΑkNn]cα>3uZΠ!λKηl¦7jΤ(ΟΓι:‰:t(/ΜeͺSΡ>σΜ3Ξ¨5Ξρ•W^‘7ί|3ν΅ώt$η¦ScΧ]w-γ‘ξ zΧΆ­σ,ΥiuιXK·dς?Σw¬™=;%ˆΥtμ¦δ–μN5M·—©σ6›Qk L7ΗΤy™:oCšλ›ώlκ<νΘ­•:/Ωs$>x „ω»™wξ£#Œν=ΥΜv›λ/œΧ0ν½δΚ…φ^r^φν½h”“-—΄χl:ξΫθXs·aKΘFŽ)6l°ζͺ}ϋφςβ‹/J«V­¬ρΌI΄σ΅Χ^+ύϋχO΄Ιχk™ψ±<Χ.‰N^§²;ρΔε»ξJ΄ΩyM§ΌΣŽ5:bάΈqρŽ?ώx©P‘‚λφΨ―†Φ’E‹δ裏–­˜6,F–ρS§N•'Ÿ|σ8=zτH' eͺΣΰuΤQΟ^Λ'N”O?ύΤωϋβ‹/|Φ+yς:­βƒ>(χίΙ—y¬SYΪ:Φ&OžμŒΎΣ5 “ t¬%£U˜q+˜“εΝΘ§Νζϊv t¬ΉΙdζυr>ξ^Ltd}ΨΘΤyϊ·ΩΤyΛL·ΘΤyϊ·ΨΤy}ŒP.™^3LΧ.u^Ιγπ‚ˆΒw³tΟ4ŒmΎ0ΆχΤ9ΫmΎBΈώ½~³½?ν=Ϊ{ΩΎζ8^8hο…³\RiσΡήK,ιXKΞ‹Ψ9xξΉη¬Gίm·έδΓ??‹Tλ]Ι]—MΒrνβζ‘SƒΨ:Φt=)νΰόϊλ―eωςεnΙψšRwnάΈ±kΊA§lœ4i’tκΤɏΑ \sΝ5NΗ©WŠΊ†^’΅2Υ5Κφέw_ηοͺ«rΞ]§FΡk]§Œόᇝf™ΧόΖ+³£Η :*pΚ”)±τύ¨£G΅3ΫoψυΧ_εχίχx,PέLΙΌtΆ»…•ζ:ZΏl™TNrZί₯¦c‡‚@Šk%TήΤyυL§;›:O;Ϊ–˜:OΧNΣ)#—ϋ¬σώφ―δ±yŒΉˆΪw³T”ΒΨζ k{O}³Ωζ+„λ/•k6—ϋΠήΫ(΄χryrμ0 Πή Siό^lσΡήσ.ΣςήQˆ@ξ–™Ϋf͚eΝHΏ~ύ|uͺi":2#B˜]φΨcλΘA]—JGτΨ¦lήΌΉμ·ί~ΎŠJγyݍρΝ7ίψJ‹Hι άzλ­ςΊ5 ΄CόΰƒNxΐ0”©N[ϊν·ίZ,X0ΪΡ¦ΉΊ&Ϋχί/Ϊ‘Κ)§$Œ[ςEνXΣκƒΪιηt΄\2aπΰΑN§u2ϋ·0κνΏΏυΔ7―Y#5#Σ“ :%ኟ~JfβΖ\YE¦Ξ[jκ<Ϋί:—:O^ L§k²u5uή!¦Ξkβ£Ξӎ΅LΤy±Sε?'†οfΑM4R s{O³Ωζγϊ Χ5K{O„φ^ΈIr“[Ϊ{ΉυOxτm>Ϊ{ ₯R~‘k)Σ±c6ζΝλyΈύ=~΄‹%°ή¬’SδC»‹ήΑxύυΧ»RλtŸ|ς‰λvN«³,Ά³NsΧ¦MωρΗc/•ω―ΗϋΧΏώε;M£#κlαŸό§xMKbΫ?·iG’­άKž³ήεθVΖa)ӎ;Z;τšΊηž{JžVΒΗ-[Ά]ƒ@;•υZt :‚SDΡ5Ρ‚ Ίζ€WΠi,_~ωeιέ»·WTюκW_}Υ3R¨d¦εάΚ\7ινΩh9·PΡ£ŠYΛ²dΠγκ›―Ž³ι< ;&QGO:4θl’žψΔΤy¦s΅Π2ΪέGWΓ\»{™:O―Ώy–:o£©σ6θhΕ€λ<Χ`€,–οf)Ÿ@w {{OI³Υζγϊ ΟL{/qYΠήKμΒ«ξ΄χάmt ν=»Oͺ[iο₯*Wv?:ΦʚπŠ‹ΐΊuλ\Άdώε%K–xDG–ψ >ϊ¨―ιιό€•λ8aw9逓¬,Ίή_ύεΚ¨ϋ'<π@kΗΪG}$Χ]wά|σ͞Ι\ΉRΊwοnζV­ZΎF y,"θh‘vΫm·9£»όœŽ;ξθ¬Εg‹›λ2Υ΅#Άήzk™7ožk6uδ₯ί ˆGy€΅cMο|¬“δtx~Ž―kPΆmΫΦϊΡtΞ8γ §S―[·nɎ3FN;ν4Χνω²α·ί~“σΞ;Οz::*ρ†n°ΖIucS3ΪG ί›όΟxψaΧ$šš Ϊ'9RΡ51Λ††‡"fΑLk‡Νͺ?ώίοΎ[Z^~Ή%₯mšm>?šk„Τάnh(gκΌ*¦Ξ[g©σt-5ΏA³©σlkεLW)užί<Άψiοεϊ»Ω–άΖ£°·χ΄²ΩζγϊΛέuO{ϟ=ν=NaŠ•Λ6ν=χ+φž»ί-‰Ϊ|΄χόκω‹WΡ_4b! β6έY6lvΪi'ΟΓθτŽϋ쳏5ήψργεr?Ψιz\QawΡ;Άφή{ogdK"OΫ˜:Eΰξ»οžh7ΧΧ΄CΰώϋοwέnΉε§sΑΆŽ”6.Ύψbk§š¦uΔGHεΚ•υa^Ήd[“K;³΅ §OŸξŒ΅u’ΖU0?°?φΨcζwvσC»%„‘L΅3ΚΦ±6Ρ¬ρ4vμX9όπΓ-g²e“ΧˆΩ¦M›ΊŽβΫ’Jj.Έΰ9χάs­;―Z΅JŽ:κ(9ύτΣεΒ /”Φ­[;倝Νί}χΌςΚ+2bĈ‚˜ΊmŊ2nά8«—v°Όͺ›λz۞=eΗ(ǟΈBΦ››iv3ΣΙΊ…iwή)SΎΪm3―ϋH0-Hl·Z¦Ξ[hιX[fκΌω¦ΞΫΖgg[[OY-ƒu^μœψώό΄χΒπέΜίΩδG¬°·χT9›m>Ώ`―kΪ{σ€½WΈν=½hσS―Πή Ζ1ΠT\Ϊ|΄χ‚S¦c-8ΛΌOIMΦiοό|Ή£Q£F’S?,Z΄Θ5ιoΌQŽ=φΨ„ΣςiGΙk―½&—\r‰θT^aγƍ^QB±= .:5H*k›%;ZM €C‡ΣόpϋΖoXΛ§oίΎςρΗΛ•W^)Ί@yΙ ―zθ―pꩧzE‰τvν@ΙTΠ©":θ ΟδΓP¦zM}ΰ12¦W―^2`ΐηšͺ^½Ίλy=όσςΘ#ΈnΧ ~;謉ΈlμΣ§h]ιΥ ͺυ€ŽξΥ?AΧΠL¨‹~G»ž— \@§τκXS’ifδλRSλ(·ΊfZš»ξ*+ΜΤΌK'L…fδρΣ©CȜ€v€zoκΌ–¦Ξki>G+ZκΌY¦Ξ›αQηων Λά“2Δό΄χΒπέ,–ίBψ…φž–CΆΪ|\Α^υ΄χώηI{φ^°ο¬ΒMφ^4ʞφ^pεDΗZp–‘OIΧ‰²M51i$ηn4‘SΫ¬Ώ’#MtŠ,Z/Aο"Ξ·0{φlΡ©Ξ,ΊήZ͚5eΪ΄iςΛ/ΏΘγ?ξ9ύYΙt£±¦y»‹Ž ΣQ‚Ιώ ŸJǚzθ”l£GΆO; 6S£ιŸvhηΪ† œQW:*ΗOΠiπtJ?Bς‡z¨ 4ΘχŽΉ.S]oμj3:Fοds «W―v=½¦tδl³f͜?]'MλΥY³fΙ;οΌ#3fΜpK’ψu][0Sa«­ΆrF }τΡΎ‘υ‘mt©ο„ˆXπ Μ΄™M§Ν_7?(ΤB3RPpΉ{QΆ©σtDΰFK·ΙΤyΏšΟ[f΄©σͺ›:―šωΣutΔαSηΝ3uήju^“ Φy#Y"+T{/ΧίΝ"[)f<μν==­lΆωΈώRΌ²Έν=;6ν=»[£+@{/deηζ£½\9Ρ±œeδSυŽ~ύwΟσΠN‰… :ρΦ]λ?¨;wΆv¬ιqτΗλώύϋ§}H?£Ϊ>H@ „έeΫm·uF&yM§V’££‘?’¬δvΫγvνΪΙ}χέ']t‘-Zρ6ςΖΟ΄7Ε;˜;μ°ƒη”“%γσx‹€–Ο³Ο>+Ίv™ίλ2Υ5Φ½φZgDšWžuT—vμ¦t]Ώ.]Ί€Ί»―ύtšΗ~ύϊ9£Ρ|ν`‰€svλcŸώΉ%›Ψ"ΠΑŒ‚\όΥW²ήΤ½A„Jζ¦  >Φa βXω”F’ωφcη§k¬΅2uή3"Ν+θZlσ¨σ™:―A†λ<―s`;…$T{/ΧίΝ ©Μτ\ΓήήΣ•‹ΧΉD‘½§ηΝ6ןΧU“›ν΄χμξ΄χμ>lΝΪ{Ρ(GΪ{Α”kΑ8ζE*W\q…ΤͺU+΄ηR­Z5Ρυ‰t*ΚTƒžί›oΎ)#FŒπLblDΦp‰‚‹}W₯JOs #™t*‘tΓmfž‘C‡¦u½Δηa»νΆsFζμ΅Χ^ρ›xnΠ5ΗFŽιLΙZ΅jUKLϋ¦\–i%Σ  uG¦Κ^;lu·ζΝ›ΫΪͺλ¦ι:*Ϊ1–jΠuεξΉηgΥTΣ`ΏΒΠ5΅:Ύϊͺ”Kγσ\ετnH]x™‚€ΉΙΘΚ›:o_SηΥΙΠη]%Sη`κΌ­²TηΩΞ•m’@Πν½\~7+€r‹B{OΛ#Ϋm>ΏπΌ hοy—ν=o#bδν½”₯₯ΝG{/˜2’c-ΗΌHe›mΆ‘GΜσιόπi]CMΧ0u‚’ »μ²‹L˜0Atθ½~ρΣ©PlαΤSO•Χ^{Ν%4ΫΒξ’_"ύG¦£‚7nˆ­ώx0~όxΩm·έNOη‰ΚL_Φ¦M›΄Σ*”Τύ»ξ’™3gΚ駟Θiη²LuέΖΟ>ϋL.ΎψbqRŸΚIꚐ:’OίΗΩ Ϊ‰­#ΧτœtΚΏAΧ%6l˜όϋίνpΤΎΆ qΔ 46ŸΕϋ›5Έͺšι‚“ ε͍6{<υ”4=ε”dw%~LΐȊE©dκΌ.¦~haκωδΉτKEoB(yƒE… dϋν·wΚW;Τf̘!—_~ΉθλΌ:ΦτG<‰šiD»ύς‹μpήyRΌ’τjﱇψυΧ²ύi§ω‰Nœ4*˜:―©σΊ˜:o;3š<Q†:ϊ­ƒ©σΊ˜5k–ψ>–fΩ’ΘD{/—ίΝ’<ύHG{{OqύΆεόΖσS`\~”‹C{/5?Ϊ{©Ή±WώΠή YήK―ŒΚ™^읯λΧ―—iΣ¦Ι―Ώώ*K–,q¦ϋΪ΄i“3₯Ÿ~±Q_:zG„ΝUΠΞ²χίίΙ£ζSσ«‚ϊ#°ώvΨaΞ:BšΧB atY³fθμ+V¬p- ύ1ώόω’τϋοΏ—?όPƍη\/ .tς€υλΧw:t4£ŽP;βˆ#€E‹ωeC8rY¦‹-’1cΖΘ·ί~λtNύφΫoςχί‹^ο± Σ.κ4΄:•¨vξjγC ύθG=εΛ—K£Fœ‘i±σ‰―£€ί~ϋνψ—‹Ÿλ?νŒ$ `X·x±Μ5ΣCΞ{λ-Ye:oΧΝ+›ΝχΡVΝΌwj›χM3Σ™V‡QN6ƌo[gκΌω¦Ξ[fκΌe¦C~₯©σ6˜Ίbs‰:O;I+š:Ο)7Sηi™54u^-F|gΌ|8Ι dͺ½—ΛοfΙD5~Ϋ{j†6Χ_T―jχ|η²Liο‰3λν=χλ“-ώhοωsΚu,Ϊ{Ι•kΙy8ꨣδέwίuMα„N_|Ρu{¦6¬[·Ξι8Π©ρω!λ2έ°aƒ¬\Ήω₯7"δsΨyηE;ݐ!Cδκ«―vΫΜλ ›M·ΡΤyΜ 2zΧ#ˆΘυw³ψόπώψcωσΟ?]SΥi"uκNdF@ΧF{ςΙ'­‰λ’σΊ~šίπϊλ―{FΥ)0  € €ω/@›/˘3 ―ν½π– 9Cό`*Θό,WΞ P lήΌYώρΘ_|ᚯΚ­·ή꺝  žΐμΩ³₯iΣ¦ž‰ότΣO²Ϋn»yΖΣyφ{υκ%Ί>‹[ΠQ{sΝZYΊ~"@@ hσεoΩrfΡ ½r"— ?,(”?eΙ™ J―ΎϊJŽ;ξ8k§šmΦ―_ΏPζŸL!/Mš4‘­·ήΪσtϊτι#Ώώ»5žŸN5MΰΤSO₯SΝ*ΙF@@ ϊ΄ω’_†œAτhοEΏ 9ˆ–#Φ’U^δΠ τμΩStέ₯5kΦΘ²eΛdρβŞyξή½»Œ=Ϊ3@ =+―ΌR† ζ™H… δ˜cŽ‘½χή[Άέv[Ρ)"ώωg™8q’|ϋν·2}ϊtΟ44‚«ΨΊuk_q‰„ €  Ϊ|Ρ('rYx΄χ ―Μ9cȝkΉ³ηΘδ₯@‡δϋοΏOκά>όsιΤ©SRϋ’ψο+­Z΅’΅kΧ&Ώs’{tνΪU>όπΓ$χ": € €@Ψh󅽄Θ_‘ Πή+Τ’ηΌ@ L™ uŽ‰ΕtjΕ<@ ³͚5“|0³1©kηέ /Όργp@@π Πζ ‘Γό ½—εΘY €@4θX‹F9‘KςR@§›»ύφΫσςά8)Β*pΖgȍ7ή(εΚ•ΛH΅1χΑψ^]³@IDATZΟ-# Q@@ΠΠζ MQ‘ ½W Νi"€@ΞθXΛy W@ηήgŸ} €3G GΧ]wΌωζ›Όyσΐr ?šτοί_ΎωζiΪ΄i`ι’ € €@thσE·μΘythοE·μΘ9DG€Ž΅θ”9E ojΤ¨αŒT2dHޜ'‚@ΤΊwο.όρ‡ΌσΞ;’΅c,•P½zu9φΨceςδΙςπΓKΓ† SI†}@@ςH€6_&§IΪ{‘,62(WdB„ςKV@ δΗwœLœ8QΦ¬Y#›6m’Κ•+K­Z΅€^½zΆm[g„Zοή½ΧB~*d‚˜5k–Όψβ‹2}ϊt™3gŽΜž=ΫωΏpαBΡ― ΥͺUsήΗυλΧw:ΟφΫo?ιΦ­›μ»οΎΞϋΌ °8Y@@ €hσpαsκ‘ ½Ω’#γ R:ΦBZ0d @0¬_Ώήι$׎5 € € €@ώΠή˟²δL@ »t¬eΧ›£!€ € € € € €DT€5Φ"Zpd@@@@@@ »t¬eΧ›£!€ € € € € €DT€Ž΅ˆΩF@@@@@ΘkΩυζh € € € € € € c-’GΆ@@@@@@²+@ΗZv½9 € € € € € €@DθX‹hΑ‘m@@@@@@μ Π±–]oŽ† € € € € € Q:Φ"Zpd@@@@@@ »t¬eΧ›£!€ € € € € €DT€Ž΅ˆΩF@@@@@ΘkΩυζh € € € € € € c-’GΆ@@@@@@²+@ΗZv½9 € € € € € €@DθX‹hΑ‘m@@@@@@μ Π±–]oŽ† € € € € € Q:Φ"Zpd@@@@@@ »t¬eΧ›£!€ € € € € €DT€Ž΅ˆΩF@@@@@ΘkΩυζh € € € € € € c-’GΆ@@@@@@²+@ΗZv½9 € € € € € €@DθX‹hΑ‘m@@@@@@μ Π±–]oŽ† € € € € € Q:Φ"Zpd@@@@@@ »t¬eΧ›£!€ € € € € €DT€Ž΅ˆΩF@@@@@ΘkΩυζh € € € € € € c-’GΆ@@@@@@²+@ΗZv½9 € € € € € €@DθX‹hΑ‘m@@@@@@μ Π±–]oŽ† € € € € € Q:Φ"Zpd@`‹@χξέ₯^½zeώΎόςΛ-‘x„ € € € €€)P1ΝύΩ@œ ¬X±B–.]Z&6l(σ/ € € € € €© 0b-U9φC@@@@@((:Φ ͺΈ9Y@@@@@@TθXKUŽύ@@@@@@ J€Ž΅‚*nN@@@@@@ U:ΦR•c?@@@@ίoΏύΆΤ―_ΏΜίi§ζ; ""€ €Ή¨˜λ p|@@@@όXΏ~½,Y²€Μ‰X±’ΜkΌ€ €„U€ka-ς… € € € € € *:ΦBUd@@@@@@ ¬t¬…΅dΘ € € € € € €@¨θX Uq@@@@@@° Π±Φ’!_ € € € € € €‘ c-TΕAf@@@@@@Β*@ΗZXK†|!€ € € € € €„J€Ž΅P™A@@@@@«ka-ς… € € € € € *:ΦBUd@@@@@@ ¬t¬…΅dΘ € € € € € €@¨θX Uq@@@@@@° Π±Φ’!_ € € € € € €‘ c-TΕAf@@@@@@Β*@ΗZXK†|!€$°xρbΩ΄iS¨Ξxώόω‘Κ™A@@@@ά Π±–ϋ2  €@Α h§Ϊή{ο-Ο>ϋlΞ;Ψf̘!gŸ}Άάwί}_. € € € € €@i:ΦJ{π @  6”]»JίΎ}₯uλΦ9ι`‹u¨νΌσΞςΒ /ΘΏώυ―HpH@@@@³ka.ς†ΐ•W^)Υ«W—iΣ¦w°5*γ#ΨJv¨=ρΔ²qγF9όσE;ϋ € € € € €@I:ΦJjπ@ gΪ‘₯Z± l§žzͺ3‚-lΪ‘vΦYg‰ŽP‹u¨ι±΅sO;ω € € € € €@Όkρ"£ΥbόG@@ ±ΐ7ή(_|±Μ;7q„,½ͺj?ώΈμ·ί~R±bΕ,•Γ € €"εŠL?:eΪΥW_mzΖgΘwάaσΤSOΙ€¬qN;ν4:t¨5ŽΉtωε—[γτνΫWξΊλ.kœηž{N.½τRkœ“O>Yξ½χ^kœ_|Ρi\Ψ"xβ‰2|ψp[yι₯—δΒ /΄Ζιέ»·<πΐΦ8―ΌςJ©υͺEξΥ«—<τΠC‰6Ώφϊλ―Kώύ‹Ÿ'zΠ³gOyτΡGm*~νΝ7ί”sΞ9§ψy’=zτpF‰ΆΕ^{ϋν·εΜ3ό=Mψ裏½^mαέwί•ΣO?έEŽ8βyϊι§­qƎλ¬f‹tΨa‡‰^―ΆπΑΘ)§œb‹"έΊu“ηŸήgάΈqrI'Yγ|πΑ’Χ«-|ςΙ'rόρΗےȁθ\―ΆHŸ}φ™wάqΆ(ΉsgyυΥWKΕΡΞ΅aΓ†•zΝνI… œM›6m*EΛPm7nάXf[’κΧ―/ϋοΏΏŒ=:ΡζβΧƏ/έ»w/~žθΑ>ϋμ#z½ΪΒΔ‰εΘ#΄E‘=χάSƌcσέwίΙα‡nΣ‘Cyο½χ¬q~ψα9δC¬qΪ΅kη˜Ϊ"ύψ㏡kW[iΣ¦|τΡGΦ8S¦L‘ƒ:Θ§uλΦ’Χ«-όϊλ―₯K[iΥͺ•|ώωηΦ8Ϊ©Ϋ©S'kœ–-[Κ—_~iσΗ8?Ω"νΈγŽςυΧ_Ϋ’ˆvwμΨΡ§yσζ2aΒkœ?όSφή{okœf͚‰^―Ά0{φlΩc=lQd»νΆ“I“&YγθtνΫ··ΖiάΈ±θυj σηΟ—ΆmΫΪ’Θ6Ϋl#z½@@ΐΏΐτιӝοNϊ=Ό_Ώ~rΥUW9Ÿρ~Rxν΅ΧDΫ‘ρα˜cŽέζ'h‡šώpΫm·ΙΜ™3Χ\sŸ]‰ƒ €"ΐ-=0F"kΦ¬‘… ZOvεΚ•ΦνΊqνΪ΅žι¬X±Β3?ωYΎ|Ήg:ωšŸuλΦy:εγ' ςγ'ΏώΫ³άƒJgύϊυžΞ~ς“Νt–-[ζιγ'?™LG;Φ|πAY½z΅g^u¨ΕvςκŠΕ‹ύ_Όx±ψ9/mΜ{Υ‡~¦£ρ“ŽŸόhΗaΆς³dΙ’—λ ςγ' ς£Σz½ΦΌœ΅sΦ+ψ9―:uκx%#~ςS«V-Οt6oήμy^[m΅•g:~ςS΅jUΟtόδ§rεʁ€λ˜χLŒ € P,Π’E ηζB½™ρώϋοwn²Τ›(υF\½‰&S!ΎCM£ί½ΌnLΝT~H@ W€© ·μ9s@ ΄Ά΅ΦB›i2†y!°hΡ’œOm•œ €@^ 4¨xϊE½iqĈ’#ξ/Ίθ"™3gN ηj=φ˜μΌσΞΞ9₯ :‹M͚5cOω €dE€© ³ΒœΡb^#ͺU«&5jΤ°ž°ŸtτŽv―/Η:Μkd[!§£―‘dUͺT―Ρ …œŽŽˆ¨]»ΆυzΦ‘]^# 9m{ΈͺT©’$£‹’ο°ΓΎF­Y ΙΗF][ν†npbΊε§d2ιœWΙttδ’ΧΘ6]/’nέΊ%w+σΈΣΡGυκΥ+cRςIε5²­ΣΡb:ZΣΚ—/οάn‹υtτ{…Nϋ|ϋν·;S’ΎόςΛΆΣe €/pΦYg%œ‚_Ϋ™n#Ψ’™ 2Ρ΅’θ:ZM§Ζφϊν δ>ϋlΡλΥ΄C^―W[θίΏΏΌρΖΆ(ςΘ#HϞ=­qΞ;ο<§`‹τΰƒŠ^―ΆpΑˆ^―Ά kWφξέΫE.ΎψbΡλΥξ»ο>ΡλΥώυ―‰^―ΆpΟ=χΘΙ'Ÿl‹"Ίξε³Ο>ksΧ]wIŸ>}¬qτw ½^mαΞ;ο½^mαͺ«½^mA—ΠΩΩlaΰΐ’Χ«-άzλ­’Χ«- <ΨΉΔηζ›ovΦ΅ΕΉώϊλεα‡ΆEq–;ΡλΥnΌρFΡλΥ»ξ:ΡλΥ4Οz½Ϊ‚ž»Γj :]―W[Π‘κz½Ϊ‚–©^―‰‚.sρΧ_%Ϊ”τk“ήƒ@Θ’€~‘ϊΥ«W~TF«NJ‚DRΰ³Ο>ύAYƒ6:cΈ«―ΎZΊwοξόΙ#Ӑ€vφx}WˆMυjcΡ›½Φqφ›ŽΧ:³~ΣρZ'ΤOG„ή$©ηf ~ΣρκX *ν°ρ z^aʏv0yεΗmΪΔ’ηκ'?>~ *?%σοφX―ωDjίΟωΔΥNΈU«V9±Χ⻧d<}oQ^Ϊων•NPωρ“NPωρ“Nμf€’ρƒJGλT/η(ζΗΟyιθO―T:zSˆ—³Ÿόd3]+Ω+ψΙO6ΣYΎ|Ή§³Ÿόd35kΦx1‹ŸόψIGΏ‹y]‡A₯γυύQO:Šωρs^z3™—sPιθηΆW*?z,·σŽ΅ Bp)•#A@ „€NΣ¨`A­ΖToA«’Ρxώω狧Όή}χέ‹GL›6ΝΉs5šgE@ό θH*BzΪΩ•­ 3κ@ˆšί7’Vbφό2€έ‡­/ΐT\  K½ΣΔΟOΙδρŠ+‘C‡&³ q@ OΞ=χάⴇzΘΉ#QΧ—Υ ϋ~¦}ΛSN Θθh,―`Ϊ¦ρϊA#_ΣΡ‘]^#ΙtΊ―»xIΗ~o²Žσy₯kƒyuBE1ύά|ϊι§3^g¨ŸN—X­Z5λ±κΥ«η9β\G¬yΨΛf::²ΛkDZέΊu₯R₯JΦsΟΧttD–Χˆ΄:uκHεΚ•­>€cχΡ‘]^#jΧ-UͺT±:ηk::"Λk$Y­Z΅€jΥͺVŸ|MGGvy$«Y³¦gž―ιθˆ,―`5jΤ½άHΗξ?ͺ]Ώ›m»νΆ©~Χυϊf³/ΉŽ΅’ͺŽV»πΒ “ί‘=@@ `tΖ ™F‚κTSL:Φ φ’βΔ@h ΅Φk«E«άΙ-Ή8逓δ±Η“ƍη* @H <ΨsϊΕtNθςΛ/BŒ€ €δZ€Ž΅\—@DŽϊ駋N“€^σGδ”Θ&DP ˆQkŒV‹`Α“e@@Π drΤ£ΥB_όd@‚¨XPgΛΙ¦,0tθΠ”χeG@ (Ψ¨΅aΓ†₯”$£ΥRbc'@@| 訡Q£FΎΦ£Υ|ρ @²$ΐˆ΅,As@`΅Ζh΅`Κ€T@@H$‰QkŒVK$Νk € KF¬εRŸc#€$-κ¨5F«%MΝŒ@—.]D4Φ°Ϋn»Μys’δƒΐ?ό _ύ΅s*νΫ·—}χέ7N‹s@ AZc΄Z€/2 €@^ ”+2!/ό“BΘ[ Θ;μ «W―φ}ŽW\q…0­­o.""€  ύl0`€“W>λ#Qdd²@Ξ<σL9rdΪg«£Υf̘!5kΦL;-@@‚`*Θ $I@ k±Qk~Θh5ΏRΔC/7ΚOaΞ@4΅Ζh΅4πΩ@Ν›7ΛΉηžλόwήy©Π±% !€d[ΐΟ¨5F«e»T8ω+pΞ9ηH«V­œ\Άl™άrΛ-ω{²œ €i €3jΡjiβ³; €dT€Ž΅ŒςζOβO?ύtρIΦɟrεLˆΊ€Χ¨5F«E½„Ι?αΠ΅bnΏύφβL=ψΰƒ2}ϊτβη<@@©ŽZc΄ZiGž!€ €@ΈθX Wy„67W\qEρΙ•+W†6Ÿd  Oΐ6jΡj…w=pΖdZ gϞ©S'η0λΧ―—fϊ€ €@dR΅Ζh΅Θ7G@ `θX+˜’ζD@όp΅Ζh΅ό,oΞ 0 :΄8/½τ’L˜0‘ψ9@@ ΄@²£Φ­Vڏg € >:ΦΒW&δ@ ID£Φ­–$"Ρ(`Ο?\}τQηο—_~ρ”Ψwί}₯wοήNΌ’’"Ρ:ˆ€ΉhםτλΧΟωΣχ&Β'Μ¨5F«…―όΘ €” c­¬ ― €DL ~Τ£Υ"V€d <χάsΏηοΣO?υ•›!C†H₯J•œΈ5’Υ«WϋڏH ¬ΐ‘‡ZΌt―^½‚MœΤ@ 0Ώ£Φ­9 !€ €@*f0m’F pΐ2cƌ琬!œΐ¦M›Šw(Wœ΄oίΎψ9ȁx@Ž9ζ˜|8•Ό8‡wάQξ½χ^ΩsΟ=₯cǎyqNœ €™ˆZ9r€λ!­ζJΓ@™k!+²ƒ@Ά.\(ύυWΆΗqΘͺΐͺU«Dδ“ΐš5kςιtςβ\tΚY €ψΠQk£F’7&܁Ρj Yx@(_ΎΌ3mΌ&‘7βθX J’t@@@@ ΨF­1Z-!/"€ €@šΪ™φΘ#€™JΩέYc­¬ ― € € € €,ΰΆΦ£Υ†&9@Θ¨#Φ2Κ›?‰χνΫWV\ιœPΥͺUσηΔ8@@@Ȋ@’QkŒVΛ =A@θX 3Ÿ“Ίϋξ»σωτ87@@@Θ‚@όZkŒVΛ:‡@@@θX ”“ΔˆΎΐηŸ.z!@ 7\pΌρΖΉ98GEΐΘ‘#E;"δ“@₯J•dγƍ’λž >\FŒ‘O§ΗΉΈ@‡δν·ί.pNΘo:Φς»|9;’hΨ°‘l»νΆIοΗ €ΑT«V-˜„HΕ·@§NdσζΝNόΦ­[ϋήΟ-βG}$cƌ‘‘C‡ΊEαuH`ςδΙ2~όx'΅έwί]:vμPΚαIfΥͺU2wξάπdˆœ  @QQ‘Μ›7/ΐI ά 4mΪ4χ™  € c-£Ό$Ž € v>}ϊˆώ₯6mΪ$={φ,ΎCω¨£Ž’<0έdΩ,ο½χž 0ΐ‰qΕWδeǚετΩ„ € €9(ŸƒcrH@@ςN B…  AƒβσωυN| ŽυH:zμ‹ € €@πt¬oJŠ € €*pσΝ7Kl:ΟoΏύV^xα…•ΰ΄@ (]ƒŠ€ € €Ι θMj=φ˜σχΔO$Ÿ€Λt¬Ήΐπ2 € €Ι 4iD.½τβέ $λΦ­+~Ξ@@@Ȏ€§ή―_ΏβΏ ŽΚkAI’ € €Fΰͺ«rξ†[Έp‘̜9SFŒ!—_~96 €@JnSAž~ϊιrΛ-·€”&;!€€/πέwίI=Oˆ@"'@ΗZδŠ,7~φΩgeνΪ΅ΞΑϋτι#U«VΝMF8* € rZ΅jΙuΧ]']t‘“Σ[o½UΞ<σL©[·nΘsNφ@ ŒnSAn΅ΥV²έvΫ…1Λδ (Y³fΔyr’ €e˜ ²¬ ―$Π)Ξ9ηηoεΚ• bπ € €@L ώ²eKηι₯KE;Χ € € € }:Φ’_†œ € €@Θ*Uͺ$C† qrΥΎ}{9ςΘ#C–C²ƒ € € €©0d*jμƒ € 7_~ω₯όςΛ/ΞωtκΤIvΩe—@Ξ­W―^2zτh9ϊθ£Εm*·@D"°ΐξ»ο.gŸ}Ά#Π±cΗ–ΰΤ@@@ [t¬eKšγ € €„R`Τ¨QςΘ#8y{葇λXΣ»wοΚs&Sδ‹ΐa‡&ϊG@@@²%ΐTΩ’ζ8 € € € € € €‘`ΔZ€‹Μ#€ € € € € €Δ ”/_^Ξ:λ,ηe}T c-(IA@@@@@…€wώψγž—ΰΊθΟ "€ € € € € € F¬…§,B“SN9EV\ιδ±jΥͺ‘Ξ+™C@@@@@2!@ΗZ&Tσ0Ν{ο½7ϊSB@²/0cΖ 8p ΄nέZ½φΪμg€#"€ € € €@Κt¬₯Lǎ € €$'0aΒιάΉ³¬_Ώ^jΤ¨!ύϊυ“mΆΩ&ΉDˆ € € €9 c-gτ@@ p€lΨ°ΑΙΚ»ξšΡ,ν΅Χ^ͺU+ωρǝiΆ―Ώώzyψα‡3zLG Ÿτ½τΝ7ί8§ΨΆm[Ω{ο½σωt97@@@ωδ, € € 3Ύ}ϋΚO<αόύγΘh>Κ—//wήygρ1τΈΏώϊkρs €@rcǎ•³Ξ:Λω{ι₯—’Ϋ™Ψ € € €@ t¬₯€Ζ. € €€*pψα‡K·nݜέ7nά(W]uUͺI± € € €dY€Ž΅,ƒs8@@†*εΚ•s F-Ÿ}φ( € € €  @QQ‘<ωδ“ΞίΘ‘#K™Ž΅ΐ(I@@νΫ·—>}ϊGΎςΚ+EΏπ@@@@ Ν›7OφΩg“¨I…Ž΅ΐ(I@@·άr‹T­ZΥΩaΒ„ ςζ›oϊί™˜ € € € Š99*œΐσΟ?/λΦ­sς}ςΙ'K•*U"wd|θέ»·,X° Τ) 4H=τΠR―ωδρΗ—Q£F•JςΦ[o•N:•z' €ώš5k&—\r‰3%ΕυΧ_/Gu”Ώ‰… € € €9 c-gτΡ:°ώθ³hΡ"'Σέ»w§c-ZΕGnσP`όψρ2kΦ¬RgδpζR “™3g–YhρβΕ‰’ς €€O½)bΰΐR«V-Ÿ{ @@@r)@ΗZ.υ96 € €@A Τ¬Y³ ΟŸ“G@@@ jt¬E­ΔΘ/ € €@ _}υ•L:ΥIsύχ—V­Zš>‰!€@ζΪΆm+gœq†s€½χή;s"e@@@ΰθXγR@@(hgžyFyδΗΰ‘‡’c­ ―N>j‡~Έθ@@Θ–kΩ’ζ8 €@ΔφΨc9ν΄ΣJE³fΝJ=η  € € € € Οt¬εsιrn €@€Η{¬θ@@ 6mΪ$|°kr‡v˜ 8Πu{ͺ^~ωe1b„λξ/Ύψ’4nάΨu{οή½eΑ‚₯Ά4H=τΠR―ύδέwί•;ξΈ#νdλΥ«'»μ²K©ΏΪ΅k§n2 δΚ0™<@cο<ΰœ¨Ί>|vawι½J—ŽA±!* ˆ(Υ‚ Šί‹"Ε.’Ύ‚Ψ’β‹  Ε X± ’ ’H—*½—₯ξ~χŒfL&Ωd7Ι¦<ηχ3sηΦ'qIζΟ9ˆo©©©9‘γSRRΒΆ„΅°‘€#@€ „ΐΆmΫ€|ωςα됞 Δ μμl™5k–ί™ύρΗ2tθP)T¨ί:yΉ‘!€{ΰΐ€έΞ™3GΦ]λUηΊλσΊŽΔΕƍΞ;―c.\Xϊχο/<π€DK`+(†yeD;@€⏀Ši―ΎϊjΨ'žφι€ @Θ3}ϋφΙ°aΓ€V­ZςΙ'ŸδΉBH[Άl‘/Ύψ"¬KQO³―Ύϊ*¬}Ζ{gGŽ‘§Ÿ~ΪΚ3ϊΪk―‰ ž @€€;<ΦάΉPκ pι₯—Κž={¬"EŠ8ξr @€ .*ͺ=φΨcVwC†  ƒ¦α+0@ΙJΰν·ί–:„mωο½χžhJΜ—ΐ¦M›δκ«―–oΎωF^yεί ”@€ Β‚ <σΜ3AΥ£ Έvξά)»vνςZ`₯J•€hΡ’^ež‹?όSV­ZεΉu½>σΜ3sυDwΒ~χέw’7–.]*λΧ―υΤ¨\Ή²T©REΪΆm+_|±T―^έ«]^/tƒ€z̘1Γ έ£‘t233₯F–gΘ 'œ —\r‰T¨P!―Cψ΄ΣυθΓ ί] ΘιΘv’’βsΟ='ΚAEI§i"§ ¦’><Χό*φc*φνΫΧ«ͺ x΅kΧΆΚ¦OŸ.7άpƒg «„A±I@7¨§Ϊ}χέgMP_Uτ/V¬XlN˜YAˆ0νΫ·ΛΜ™3­οhωJ7%Κ­–ίώ ’½nΤϊϊλ―Cϊ―Ώώ’>ϊHFŽ)+V¬πi―ί!us˜nΓ @€ώ%@L™Yp@Q  ½{χ–=zψˆ=nΓλ.Ϊ‰'ΚρΗοεηVΧ­LCΨ4hΠΐςΨr՜m>,S§N΅ΌεnΏύvΩΏΏ³JΧ:ίzυκΙγ?ξ*ͺωλ`έΊu2|ψpiάΈ±Μž=Ϋ_΅ ΛGŒ!ηž{nPœƒξ”ŠH@κε©a―τhΤ¨QL¬PώT­Z՚‹zΉͺ—)ψπxuͺgηO?ύδ[’Έ$PΆlY/~Ӎ[φΝ^Ϊg8£„cŽΡκCΉλ―Ώ^Ύύφ[iΦ¬™Ο°έA7a€ @ήΦΌyp@$ a»tι"οΌσNΘ£¨Ψε—_R>Œ|ΠzX°{χξΗΣ.O<ρ„%°9C`κLw@λΓωόδνΠπ‘RΓ6ζΥ4\wή™Χ洃@RΈκͺ«D=Vυp†¬-(ꝦB»ΗΤ›@=]1@ΐ›€zn_sΝ5Φ.αΕ{ ‚€ kνΫ·χZ7>ϋl+Δδ„ ,ο4‘τ‘\©R₯μU­σ_ύU.Ίθ’ δ,Y²Δ΅ζ‘Σ°—ϊΠ^=Ρ4μŽξ°μ³Οδ駟ΆΌρœ«˜Χ±cΗ ΖuΆΥϋ·άr‹³˜k@ Ξθί₯γŽ;Κ΅ζJ|ΰβlL€@ήθ' k7ύn”_‘G=€Υ;ΛnIΑιΑfΏŸ,η*fφοίίgΉR]σ±a€ @ ΗΪΏ,8ƒ АŽšΓc't’*ρΤSOυy½jΕAƒy•ιΕ»οΎky‘ωά°|ύχrύχΫJώ>ΥoγǏ—–-[ϊάλΧ―Ÿ<όσ2tθPyφΩg½ξυΥWrο½χΚ£>κUξΌP‘lǎ^Ε-Z΄°Ζl€‰WΉ†xΣζ*ςύη? K€’’έΣmλΦ­2oή<ΛkΞ«q.BΞώ€¨yσζ©S'ΡW=T\,_Ύ|.½p(h… ²ςή¨§―šzγhˆΧ’%KτΤ€@D θχ‘nέΊΙM7έ$¦ΫcϊwP7jεΥτ{€~'΅›~²G΅ίKΆs μfšgνŒ3Ξp»E @ˆiϊέουΧ_·ζ˜’’"W^yeXζ‹°Œt@‘Έΰ‚ ¬ΔκΙεΟTδς*°ΩνΛ/Ώέ±\Ίti{qΞΉ†›t yι₯—Šζ[ΣπjώLησΜ3ΟXωΡ ΰUνΉηž³rLh¨6wξ\―[•*U²vE—(QΒ«άνBsΞ=φΨc–g›ύΎzΈω{Θa―g?_³fu©_,κ½WΈ0δΫqx!pώωη[ή«š+R½pΥβεcž€@~¨˜V\9ιΠ‘ƒ|όρΗ9]}πΑ’ίυ}‡Μ©μrβ ©›œτ°oHri–4ES=לΕ4B’ kŠ]…[όσO+·³z8κηP£ih>:}ΥM²]7Ώδfϊ[fρβΕ^ΥRSS₯]»v^e‘\hgήθ¦M›JŊƒκF=ΰ5„ξŒ3dνΪ΅’yττ©5jH­Z΅δ„N°ΌEσ’wΠm½ϊ;¦H‘"9sΣρ5‰nΤ°χGŽ±ςΘκ&@ €  ϊ=OSΆ¨ιΏΓk ώ@ρF@s7¨wVZZZSW1έU²eΛ–œΊϊ E„ωΛƒ€υυΗ°έκΤ©“«¨f――dκ₯6yςδœbύΑωδ“OZή#9…Άύ«a#ν¦ύ#ͺyΪτμΩΣGXϋα‡<·C~Υ]Ωέ»wΉ Ψ" ΐτ@ YxΌΤΤ›Μ.¬νέ»ΧΊΦ I‘šŠΊaΙnR aνo*Ί)«nέΊ’aΕν¦‚I’šζqΦ tcǎ•;wϊ,Sσ›:Ε±κΥ«Λ΅Χ^+Χ]w%Hω4ϊ§@ChžuΦY>·ωεΡ¨‘ΪκΥ«ε΄ΣNσj¦ίt Ή™†τΌλ»δΝ7ίτζ΄ν²eΛ¬.4tύΐ₯sηΞVύ6mΪδΦuΞ}ύΎ·oߜk=Ρίe΅kΧΆΚ¦OŸ.7άpƒx6Z…ζ?φίzž2^!@±N€νλ±ώΕΘόtηց¬Ωh¬ϋŒŒŒ™Σ€⍀ώX FTΣuιE ¦mμ¦!έLέ»Ÿzκ)Ÿ[šC-§šOSpΟ=χx kZηΓ?τ+¬©¨ζά=κnTύ‘!νy,ςϊC³W―^ˆjno,eˆCˆjqψ¦1e@ _<šζΉΥߞΜιO›ζEXs ιΦμ‘ΈsJ“εΛ—ϋ¬Ό~ύϊ>e‰P0qβDΧHΉ­mέΊu2|ψp+²††υ]B=ΙT@[°`W—ϊYΜ‹°¦νœ¦› ΥΫ,iԍ²{χξ@ΥrξιS§Nυ½υΦ[Esa‡ϊ[*§³N4”υwήι,ζΘ…ΐFγ΅½tδΘ\jε~;έxί–4©1J˜C_υHσ(χή¨(„5>AΈε–[Δσ [C!¬…J€€ƒ€†iΦ¬™£4π₯†ηqšηο‘³\s«9w”κ˜y -’aPΤ»ξηŸΞfΡ’EςΧ_Y‘`r 9QAlάΈq^Ε'žx’Χun*Μi˜Λόš†[5jT~»‘= @(a­T©RVΨ½)S¦δΜ㣏> ΫWΌxρœ²`NT³›ζΥό»jžρμχ“ρ\CΊymΉ}w>³fΝ²ΒBεGTΥΟ‘†…ΤPυn9œ•‘†¨wΦT¬ ΥήyηŸ&Ϊ Σq4”t^L=9Ÿxβ ωμ³ΟDyω ş[ίΊΙQ-7Jά‡€;&\λ6σ_8Μξ{œbREΫΏΏ4~ΰΆpΐ₯€$€°–”o;‹† P0tGe¨ζ–/ΐξΡeοOπ9퀓Nr}έΎ}{/aMΞ™3GΊuλζΣ‡†›Τ#?φυΧ_[yςΣ‡ΆΥ]³š#ƒ @ρH@σ.yLΓAΪ…΅ύϋχ[ž4I%XΣPyϊΞnΪ―ΗμγyΚ’ρΥ-Ή†‡lάΈqBαX²d‰¨7€έR¨Ήϋϊυλ'ΡUΏOoίΎέΚC¦ΉΧ4Š††q΄›nŠλΨ±£¨›ΫdΝσ P@τίjOςP=Ε hΚ »Ρ\`=ψWAΝcκ}Š°ζBΟRϋ$ǚXŠ›’’Z~Γzή·Xy}Ϊcj½§žzΚ§\=ςyͺω40χάs—°¦u>όπЅ΅^½z!ͺΉ¦ Q&PΤruΏήςJϋΞδ†ά½p‘Χ ―ε…Ζ+΅•‰ζA ‘€¦¦ζ„ŽΧσpΒZΈH (ά„kr„5Σ½φZ Φ«MœknΝ £‘Y4‰8 ƒΓυNΡ›ΐγ?ž³]|i8K €@’Π¨šUo½υVΞgΜiεΙ*[ΆlN™Ϋ‰SΜhέΊ΅¨χN"™ E ,λ’Τ+IΏ§&š© ζό73ΤόŒΚ{Χ]!‘ΡpΞΟ’ώvFXs‡΅?7Σάjκ g·3Ξ8ΓΚkf/ ζ\C¦ΆjΥJ~ώωηœκ‹-²ςρcΠcELΘΊQ£FS•:€@”1α\O7oίuθ ;mλπ›L¨e ‰F@Ÿ³½φΪka_ΒZΨ‘! 96^xα+)vΈζT³fMΧ4Ϋ5Χ\#φΤžŠκ©§?ΨυΗ©z₯ιΓΟAŽ%^!` θΝο’!ΊΤήxγ +$U‹-¬kώ@  h8H»°¦β††©ΣMMώlΩ²e2ώ|―ΫΪζŸ€z5iŽ­Ύ}ϋϊ―Ηw<ίΑ󳄯ΝιPΓΌwκΤI4|γΦ­[s†Vο9ύŒΦ―_?§Μy²~ύzΡόmv;φΨcύζs›5k–½ͺu~I'ω”[ΠΎ}{/aMΫΝ™3GΊuλTmΫΆ•Z΅jU—J€@τ€›M)Ηφο/σ†ά΄Išœlf&€°˜w! a<κaAˆ5*^ir{§ηΪYg% 6Œθtυ‡΅ξuŠjBgψπαrƒ‰U?€1@α" a«.0ΙΕ?ψΰΙΚΚ’!C†ΘτιΣΓΥ=ύ@ˆ9*L”)SFvξά™37υ $¬9=}tc‚†₯Γ| ([  TU«Vυ­δ%bQCh~όρΗAy™9qiΨFuŸώy―[κ΅vηwz•Ω/άΒ@φιΣΗ^Ελό›oΎρΊΦ υ:Λ«ι¦@§9σ·9οΫ―;ξ8ϋ%η•΄P@IDAT€@ (g„o7Ϋcς¬eOW La-0ξώCΐωε0€b‘@Ŋ}„5Ν=iamƌΦΞM;υD?~|HoBέωjs@ ωŒ1ΒzΐwτθQΡΏCzt4ω0@‰H@7Q]tΡE^aόΎψβ ΛΘί&g轓O>YόEˆgf%K– ι;§UCτ)7ύώ¬α15Ί‚n KfΣMrκEΆjΥ*Ÿc͚5($|0άt#žσيŠΏ„5§8¬γΦtώNΣА―Ύϊͺ³8¨kΚα΄Υ«W;‹ό^7iΔο=n@K „Ω€œf<ΧοΨα5‘έ&ο{aνΐ† ²ΗjVΕΉBζߏ޳GV›Ώ7;ηΝ“]Ζc<ΫlBnφδ“RΩlŽ Ζ΄ύ¦O>‘ΝζwM¦ΩΜpΐδ};š™)EkԐbΖϋ΅Œ‰Tύ’K$ƒΛΑΰ€N„ ¬E0έC€@τθnLύ!l7Φ"mn?`―»ξΊphβy €@°τa•†κzι₯—¬&κ΅vΞ9ηH82;κAˆυψ±ηΗMIκΡsγ7ϊ ―b‚σ;Z’†TqlμΨ±> (ȝΐwί}gmRωςΛ/eξάΉ!‡xΜ}„k¨°[·n]Y±bEN‘†*]Ήr₯hxG§ύυΧ_’σ³›  6 nίΎέ^έ:6l˜OY~ <ь‚ιCE_ ˆMκΕ]άόMΪωΣO^TΝΝ6ΑkΎ#LpΗ?”β΅k[Υ7™θσM΄žL³Αn·l±_Ίž0!(έu—¬yσMΙ6‘žΆΟ„ΝU[σΏΙ₯JηΞΐΤ/Χ¦³*ΧˆΤ¨Δ@€ 8ύτΣ}Fp ‡βSΙOΑ=χάcύψΥΐžγ7³{Λi .t‰Ϋ\|*9 48DŸ€>θΊβŠ+¬£Aƒџ@>F|ΰ¬0ΈΪΕ‚ ¬|kω莦ˆ;ϊo§ζΤcžΩ%6έ5§M›6Mς"Xiξ€Q£FY;Hu©›7o–F9‡°’Ž; ›5kζ,Κυϊσ…ƒ’Oΰκ«―–Χ_έ:άώŽDFΑX₯J4hΥ@=Υςςχ.ψΡ¨ Ψ#πΡGΙ•W^i&Lˆ½ 2£°ΠPΫ=zτπκsΦ¬Y²Ρ„ŠršSpS₯Z΅jΞj\'!²τ;½?AΛ‰DσΜkΈΜ›nΊIτσζΆ©ΞΩΖίυe—]ζsΛί<œβ°†κΌΔ„@σgΞ\Σώκε·a-ΏiΨ Ή~½Άε-υΜͺdˆΉWŽ-‹δŠττλφΊψΑeώυΧΛ‘έ»έn.3y¦—?ρ„%°ή΅+p]ξB ¨t @C@Ε¬vνΪΙΧ_3M4»Pί4!B±»οΎ[9B\uΥU’tœV§N+tŒ½|Ι’%ŠΈ¦9BTΔ „J@…5ύ›3tθP9ώψγCmN}@qE@Γ9zBΰκΔ5Ο€ ·άrKΞ:~ύwΡΓn½{χΆ_rž€4gΨ5Χ\#šOΝišΗOC-ž`rψ¨Wš~ΗχnΏœνƒΉVΟψ“N:Ι+?³† ΤΌe΅Lώ m0‘؜a ;tθ •+WφTρy-Uͺ”O™ΌπΒ ’k —%bžΒp±‘āν?όΰ;]γέ]²qcίr?%Frνί_?Υ\‹?ό°όqί}>χRΜί« &‘ζSΣ£P±b’iΌŒwόψ£όeΒ?;EΈέ&·δ&λ)Ÿ~*…ΜF Ρ"ΰϋt0Z#3 @ 4ǐ]XΣ!&Nœh%»6όΟ€I“¬6φιi‚χ&–·›ιƒlmc73Ψρ~4_»wοn=²χΑ9 `”(QBπΤ †u D  ›¨T\Ψdς±xLΏ‡Ω…5§§zτϋ½ΜΣ'―‰G@C=j(E§¨¦ž`Γ‡—Ln  *D|α:‡9sζx£βπν&€™Ηή}Ι2ήv Rλ©xVΌxqqz©7~ Όlφ18‡’ƒ€ Un‚˜Šj…¬i(FσΗ*§z©ζΝ₯r§N’―₯Ν‘jώ.₯;B8kεmί/άN;ΟI γMάzόx)Σ²₯§θίΧ~ύδψηŸ—ίΜfΒ•Ο>ϋoΉ9ΫϊΥWςΗ½χJΣGυ*ηJ@7ά{6Ϋkhπάώ= –ΒZ°€¨@ Ζ hΑ―Μ—‰pYΌ&@οl’ΨvνΪΥΛϋKw2k…»LrΫM¨}Έβfϊ#{ΐ€ςͺ‰νν΄'Lˆ·€βZOw΅:Ms"¨§ά&$‚ώΓνfš\|„‰E}λέΜ-΄‘[=Κ @€@2PDE²ηΝΓ5}ϋν·²ή„΄ς„zt†<γŒ3DCηbΙM`ƌ>‚–z’7qυ·B°l.6ύ©χδm·έ&φ~œΒšSVΑμ"γ‘‘›U¬XΡGX[Ίt)ΒZnก$"pΐxΔΞ1a•Ί„Qnlς7‡b™kΦό]έ<σ¨?x°41^h©.Q~μ}5Ή |“Žg Υ/½TZšηZ„½BE‹J‹gž‘υκΙ―ζٍέV<χœΤ3›‘‹πο½ 熀nTΡΠρjϊ<aΝBΑ @ΐC@γύλS!(^…5]Έqγ,±kΥͺU^8TθϊΤ„8έ„P/3=8 ,°ŽO>ωDœm΄ύρ«9ό™†e9υΤS}Β΅h8ΙΙ“'[bέq&NyΩ²eeνΪ΅VΎΆ)S¦ΘΗμn²vνΪR½zuΡ‡C[Όx±άkv^}φΩ’ω :™έ_ @HfšgΚ.¬ιNd"n½υV+•~²›†Δ π« ζ΄λ».$QMΫ―[·ΞΩMHΧ•*U’Ž;ZΏ< ΥƒM'Τ¨QΓΚψΝ7ίxnY―έΊu³ΌΡΌ ].Z΅jεσ{F…5 Hn‡Μ¦ή=ζίΖU/½$kMNΪlGΪ ₯SαΜ3₯šΩΈ’kcΌn«™(<ΑΨ“ΫzΏΙio·b&τnn’š½~έόGΆ˜εΜσeΑnω“OJΣ‘#=EΌB ’πX‹(ήΔι\€xΤκN.Mή‹AˆU*`©huήyηYω μσœ7ožθ¬56‘^~ωε€ΥuΗΛλζΛa‹-|BΛόdbŽ_qΕΫλMΥΤγP»°¦χzθ!λPΡMpc€ @ ™ θ†&υNS/5©—š kΞπάκαΦΓμΜΗ °pαBΊα.T[΄hQ¨M|κλnyύ½β1‡5β…~†έΒ@zvΪ{κϋ{Υυh?vS‘Ξf~/·σ{ξΉΗ'άτΤ©S­tΉ΅ε> ™Fά<ΘΌΚYfsn¦y†ptίΎΐ“1V6ϊιΐuόά­fž+ͺιίΊεO=εΣΣρ£GτTσi` ™ΏOvaMλlόπC„57X”E„€{,¬ˆ E§ρLΰζ›oΆ λΓα={φΔσR˜; $TϋΑ$γmέΊužW¬?xuχ¨ΏDΰφŽ5L€†γ,B}ϊˆζYΣ„ε]Ίt‘΄΄4Ο-^!@€4Ί‚F°›~g[½z΅°ΦΎ}{Ρπx–-[ζ‘Y³f>eΉθwώόΪ…^(š#ΥnRΝσκΉ§"²F―Ζ4ŸšΣ¦M›&y4%£F²’m¬\ΉzέΌy³4290@ ΰ¨·Ωnu'˜c―ρRΛUT3KQ//Ν‰ͺ₯š\τMΝί‰`m»Ι­¦s²[yYσ²…jeLJŽ2ΖKΧn{ΜΖΝ‡A Φ’A™1 @ @h.Ω³gΛ‹/Ύ(r%XӜiškα7ސ’%KΫL4ΜΠoΏύ&ηž{nmτξ(ΥΠ”šDΥσΐ§ΌIμ«»@5@€ ΰNΐήQwΑk>]gΨ;g=χή(MuL¨1§-Y²ΔYπϊ‹/ΎπΚε°r€›Ί―»#lΪχζσόωσελ―Ώφj©œύεˆφͺh.T(lםW±ώΏ‘!ρC5 kΘ.ξͺ«ΝK‡A‰A ω[€!λί~{žTm[)f6k[]—”;ι€`›ϋΤ«`6Ο8m‡ΩhƒA ψΧ0”€@˜ ¬ρ$ˆ sΏΊσ„# TΗ~/ΤϊφΆφsM­?σjώηΖo΄•Ξœ9Σ­4δ“°wǎ– ₯»@5Μ’ξΎΌμ²ΛDσ‘εΥτ»ζjΣΠ.ϊγψχί·Ž½{χZchήΝν¦ήpn?ξu\ a©†…όε—_DE8£σG²Φ½φΪk­CΟ1@ oΤcΤσ φ$σΓ~ύϊyλ([ιί" αμNχ\S‚@@Mš4ΙI@c°δ! ―k›PΪ«l9uuc”έTΠάT”€~w† 8q’τ 2§~_P1μθΡ£aͺΏ4œΌΗτ7Ο5Χ\#YYYž"λ5Ψ0žFC† ρηtϊ»*Ψ΅*'mc·"Ζ3eΰΐφ"Ξ!8%V¦ŒTΏτRidςΉ©Z5Ο«(ⳓmŽό‘:°Σλ,”Ι”jΪΤ§ϊ>Gώ6­°ϊ΅Χ$3ΔηhšsBΒϋLˆ‚„%€°–°o- ƒ ;’E‹JΧ]­Γ^©σΝ;‹y5}PtυΥWη΅9ν Όϊκ«2fΜ«Εhί?„5 ίΤ·o_ωςΛ/₯Aƒ–ΐΟσ>TηŸΎθ%' ω裏ϊ]ό9ηœ# Z¦!ucTΈlμΨ±ακ*nϊ‰$C7ρ]σ‘©7םwήimfs΅}ϋv1b„<ρΔ~E΅7Ί5 X¦a«šΪΊαΟc Lx7»΅iΣF4Δ}(¦ΏAτw†€τ˜Š½L$υκ|πΑύzΐιfΐˆ~7rš_Γίc€@Α(l’κh^³PLC6fT¨ ι&4rY“.£μ‰'JŠΩ„œ_+i68…b»~ύΥ§ϊr•ΛίŸŠ.ϋ–/χ)έoΒB;m΅ι›‹·œ³žύΊ±ω[‰°f'ΒΉ“Βš“Χ€ @ˆs₯K—ΆΒIι2ΤO…ΓώύϋΗωͺ˜> o—\rI@aΝ™‡Ν»uψ―f™‡vz„Γ4bA2 k‘dΨ‘C9υΤSε»οΎσz‹4δαδΙ“-AI#W”-[VΦ]kε›2eŠ‰ΒQ7ΐi΄‹oΏύ6§ŸΕζΑπ½ΖσC=Δ<(‚Θ€‘5.5#*Xω3υ`Λ‹7NTH΄{tj?*"j(z I―|z8p@TΠΣC#o8Ϋh;ύ馛nS (`*Ž#/Tδ Ε™ N[/¦β †ƒΦσ«ή_`M=ζ^~ωε`«S€€+£™™’eŽH[–Ωά€AΐN@7+υιΣΗ* 6g©½½ΏσT7(‡ @€β—ΐώσ©υO2ρΝ›7ΛΘ‘#γw1Μ€€_|±λŽ;J“Cƒ€€†2Τp“*Z…jϊPNσ¬ιΏ­]Ίt‘΄΄΄P»π©―β°ζŠt3 έXͺT)·[A•© φΓ?Hkφ-―¦Βߜ9sς5ΌŽM;@ ±Ω·/* BX‹ ζΈDΕ΄7ί|Σ:μΉMσ»<ΦςKφ€ @ˆAςπΓ‹>S{ςΙ'­0NΥͺU‹ΑΩ2%@y# Βšz 9-Ϊa γs»τ3£ΉΛnΎωf+,b ™κ.χΣN;M4\€=Ό£ζξ›:uͺ•·l_>λΏΣšϋΜiΧ]w³(δλ*UͺΘμΩ³-1ρΎϋξέhŒiΙAƒY‘*ƒ©O@ΉHσ³Q Ε /HjzznΝƒΎ_¬fMŸΊ LW^ιS¨ L«Vns‚°Ζ‡ (={φ”=Δ¨Υ‡4 @€@μΠRbJC>νίΏ_τ‘Z2ζμ‰ύwŠB y .\X²³³σ  iΣ¦ωj―»…’ΜmBkΦ¬Ι­JDξ_{ν΅’G"XA1TvuκΤ±ς‰iΈΔοΏ^~ύwλΠ‘Κ±FVώ1½΄›i˜Εί~ϋΝ ωΛ/ΏˆŠpΊy₯]»vnΥύ–i@§°Φ¨Q#KΠσΫ(„šΛνΖo΄BBϜ9Συ{Α† dǎRΡδkyλΊu\ύξ ΉζB΅Dϊl†ΊvκCΉPρ¬PρβrΤ±‘βYgIΙ† sο 5ͺœ~>ZΣξΦάΉPκ 0zτhG —€ @±N@ς5JΞ>ϋlkͺγƍ“Ϋn»MτA4@Hv;w=ςj΅MΞ΅«―Ύ:―Ν­vΛ—/χi―BXΈ­hΡ’΅kWλwίτ@ FΘίοΦφ.]qa-˜ΉQ‘ ΗZ¨Δ¨@€ 8"p–Ωκyh¨yVŽ=G³gͺ€ Δ&ΰάΘ\άxtδW¬Klb¬ˆWnαUXΓ πX‹Ηw9C€ „€ζYρ„Ό_Ώ~Ψϊ₯ŽΤkMυκΥ+–¦Ε\ oόρ‡ΜŸ?ίκ§qγΖ²eΛ|χI€’E`έΊu2mΪ4―αϊτι#eΚ”ρ*γ€@"(ϊιςΧ{οy-eΫ7ίHύΫoχ* φbΡ=χΘΪ Όͺ·5ω/KΓ‹ ‘!€°τ @€ '’!'H“&MD ‰FΰΓ?”!&!½Ϊ AƒΦν f=Hp/Ύψ’'yώύ|Υ,HVšOΝiΜζ‚έ‹I©«μϋσOYj6f:”Σe‘%€„Ι‰A  ʌ@€ @€ μίΏ_ƌγΕ£cǎΌys―2. $ ΝšI…vνΌ—“-Kώϋ_ο² έ}·—¨¦Mj^u•€Ə(|T „΅0@€ @€ @€ Έ8xπ W±^χμΩSΆnέκU~Ηwx]s@ ΡΤ'€}]λ&N”υοΎk/ xΎn$Ρ6vK-RDκh/βl#ގ?ή:&8B‡ζn~θΡ€ @€ @hȊ+J½zυdύϊυ2wξ\Q5»rΚ)Ύ}{{η€Ž@•Ξ₯Jײў_ςθQωΡδ‚npΧ]δΑ%%ΥέθΘή½ςλ€²ϊΥW}Έ4{β )~μ±>ε@ ++K4©Zͺωl]zι₯a‚°Œt@€ @€ _‡–όΡ:|€€ΘSO=εv‹2@ G ΥΈqςε 'ΘώU«ΌΦΆΤ„„άτι§RατΣ₯τρΗ[Gց²kΑλΨτΙ'>m΄ƒj½{Λ±7έδΥˆ4„΅HNώί{ο=ρ„.θΡ£‡ddd$ΘΚX @€ @(iiiςδ“OJλΦ­ fŒ @ ΚΛ–•Ά,ߟwžd^ν5ϊyσD`­€ρnωςΛΑV§ΒFΐέ―2lέΣQ’θΧ―Ÿε2©n“{φμI”e±@€ Τ–,YbεxYΉreRs`ρ€ ‚ pΜ1ΗΘ—_~)ύϋχ/ˆα€@(e±3ψAΚδcSAΛ/—vsζHZ©RΆN^x¬%ο{ΟΚ!@€’˜ΐ3Ο<#·ί~»9rD ., ΐ“ K‡ „•ΐ«&ΠζςŽ;€L™2rάqΗY‡ζ\Σƒ1@ΙH H•*rζμΩ²κ•WdΡ}χΙ‘Ν›ƒΒPΪ„‘¬?hΤS¬ ₯ψΧΫ„K@€ δ"0wξ\YΎ|Ή΅h Γ€Ή’ΑΪ΄ic‰jΊΦI“&ΙΐEΛ0āΖf·³'yΛ–-γiκΜH":t=0@‘¨}ν΅’G,X€ζ’R¨ΤΉρF©yε•²yζLΩ0uͺμ4‘ lΨ ‡Μf„ŒŠ₯h΅jR΄zu)Ρ¨‘ΤΈμ2)e6'`(hkύ0> @€@;v¬Œ3ƚΓθΡ£“FX;ωδ“₯W―^ςΞ;οHvvΆ ϊυλK±bΕaI¬€ @€ kΑJ§ˆ}Σ§O—C‡ΕώD™!‚ pΕWΘάΉs}jΎφΪkrI'ω”Sx$P΅jΥxœ6sŽ#-Z΄ˆ£Ω2U$μμlΧΕjŽˆΟ>ϋΜυ…€ y™™™‘„ @ & ¬ΕδΫΒ€ yuκΤ‰ό Œ(πη]Q³fMiΨ°a”fΑ0€@ΌhέΊ΅μڡ˚~½zυβuΜII`ρβΕςΛ/ΏXkoάΈ±$’@œ’’βϊήξάΉSτΐ @€ θ@X‹.oFƒ @ˆ1}ϋφ=0@ ώhΣ!C†X4hPB kρχ0c@€ $6ΤΔ^«ƒ @€ Δ/‘ γwEΜ€ @ρM΅ψ~˜= @€ $0‘ xΙ, € @a! ›Τ&Mšdυ₯ί«{χξ–~ΦΒ‚‘N @€ ΈŽ9"… σΣ!qίaV@€ @ ρdeeΙ%—\b-,55a-ρήbV@€ Ψ"°kΧ.yδ‘GdΪ΄i2ώ|ΙΘȈ­ 2$>}ϊΘ9ηœ“+e‰Ι@`ƌ2`ΐŸ₯κgόΩgŸυ)§ρH hΡ’ρ8mζ @!`Ϋi°’Ήκ”)SδΰΑƒ‚‹.Ίˆ‡*Ιόa`ν€ $ ™qςΙ'ΛβΕ‹­υ>χάsrϋν·'ΕΪY$b‰@Ω²eE ‰@`Ρ’EΛ(Y²€4jΤΘυ…€ @ Φ€ΖΪ„˜OlΈώϊλ-—Iu›ά³gOlN’YA€ „€ΖŸοίΏN?ό°μΨ±#ηš@€ @€@2@XKΖw5C€ @ 7ήx£Τ―_ίͺ©’šŠk @€ @ ™ ¬%σ»ΟΪ!@€δηŸ–I“&YNJ+ b#––fεXσi8ΘU«Vy.y…@ΠΠq½{χΆŽγ?ΎΐηΓ @€ Δ'@Ž΅ΔY! @€@/Ώό²Œ3Ζͺ1zτh©[·n€ΪΙw«GΆm[™={Ά•sχ»ξ’ργΗ'V“.ΈΰΡƒ @€ -x¬E‹4γ@€ @ N <φΨc93Ÿ8q’εε—Sΐ  @€ @ ‰ΰ±–Do6K… @€@^œrΚ)½{wyύχ%;;[|πA™2eJ^Ί’  @€ @ *RRR¬°ρ:XjjψόΜΦ’ςφ1 @€β›ΐˆ#dƌ―_?Ήϋξ»γ{1Μ€ @€ž€ŠioΏύvΨΧ‰°v€t@€ Δ#PΏ~}YΏ~½”*U*ρNJ @€ @A@X T²W»πΒ eϞ=†ŒŒŒdΗΑϊ!@€@R@TKΚ·EC€ @€€Βš §ώ ΌςΚ+ώor€ @€ @€ $πekKX,€ @€ @€ @ y ΰ±–Όο=+‡ 3φξέ+%J”ˆ™ωθD4όmΙ’%cjNLˆ V­ZI―^½¬ΞλΦ­™Aθˆ%K–Θ―Ώώjυέ°aCiήΌyDΖ‘S@€ @x¬yHπ @F`ǎrζ™gʌ3 lž·mΫ&wέu—<τΠCž"^!'pύυΧΛ€I“¬£C‡ ΎZ–Δ"0mΪ4ιέ»·uΌρΖ‰΅8V@€ Δ$„΅˜|[˜ δ"P£F i€‰tκΤIN9ε™>}zΤx΅Ϊ΅kΛγ?.·άrKΤηΐ€€β•@ff¦lΪ΄)^§ΟΌ!@€ @A@X !@ ’ξΌσNIOO—Ω³gΛΉηž+mΫΆŠΐ¦‚šŽ­‚Ϊ#<"–²oίΎ’b@ deeΙΈqγ€AƒrΓ 7Μ]@€ @€@ dggΛ;οΌcοΎϋnΨFFX J:‚ όP!K-ύπΓΨμ‚Ϊˆ#,AMΗVqO…6 €r'°hΡ"ΉφΪkeέΊu’!ωf͚•{#j@€ @€’@@7ƒzBΗ_|ρΕaa-l(ι€ςKΐγ΅fο'ά›?AΝ3&ήjΌBȝ@Σ¦MεςΛ/Ο©8xπ`с @€ @ Q ¬%κ;ζuιdu•ΤγΠ‘Caξξ όMΐι΅fη’_-7AMΗΒ[ΝNœs@Αxψα‡₯H‘"VεόQ&Mš\CjA€@\ΠΠι±fϋφν‹΅)1@Fΰ¨ΙœuψpTΗΧΝuGφοκ˜ ‚$PΈ gμψ! [·n΅&ΌeΛ©P‘BόLž™BqE@½ΦƎλWΔχl'Ÿ|² 6L:uκp}*¨=φΨcςάsΟε„{τΧo5d(‡ ΰŸ€nŠ0`€Œ9ͺtΧ]wI·nέ¬Ν ώ[q€β•€z'W«VMώσŸH©R₯ tS§N•'Ÿ|Rή})^ΌxΞ…Α!Ψ&iB—g=š3ΙT³1¬HεΚ9Χz²oΥ*YύΚ+²Ϋ„;ίηŸ’ωΧ_’^œ5σ*΄o/Υzτ’y΅Ινβΐ† ²gρb―jεLNωBlLΣ‡χμ‘Υ―Ύ*;ηΝ“]σηKφ‘#Μόm«œΛσm{hηNΩτρΗ²Α8Eμόωg9Έi“1ύ©₯•-+E«W—Šg%•Ο;O*sŽ€*dέΛο”η_“'Λ_S¦Θή%K¬quήιεΛK±c•βuλJI“‡Ή–y¦\¬fΝόg1ΪτΙ'²yΖ Ι\»Vlά(* 5ΏEŠΥͺ%eN8Aͺ_r‰d„ι™υΓrσΜ™²oωrΩ»l™υzhϋvk}ιfŒ’ J₯Ž₯ςΉηZŸό,0šcεgž΄υ&bΤdb΅x3αΚ…@ŊΦ\ΈPD†ΐΝ7ί,£GͺsΨt‡κΒ… }κ_zι₯ςΑδ*¨iCυV[nΎ0ιb €B#°kΧ.©k~<λf5}Θyλ­·†Φ ΅!£F’!C†X- $zA‘% ω5›5k&₯K—–Ϋn»ΝΪ\¬ΐ¦Xσ`ΪiΊ!Cοk*¨=πΐ2ί<€Ύϋξ»ε‘‡ Ά)υ $%π‰ω­ΐˆA«Ψ‘ƒœfD΅½ζYΐ’{ξ‘υ&R—ΨΔ7O]ϋkUσχͺΩOHρΪ΅νΕ~ΟWΑlΎ-ŸΌVμhD;OϋMΣ§ΛόnΜ5kΌϊhυΖRΣrέλ¦Ή8zΰ€,}τQYf6· S¬„Ί›ΏΥLž©””g—A]«χΗ½χΚJ}f“ +ν0Εχ>ŒPYΫΌΏ‡χjskΝ±r›K"ί?j>§… ν_–ššj>Ά μωY7‘ σCΆ€ nΉΦό €ln’šΦŸ0aBP’šΦΕ[M)`HNσΜΞPOΘλ•+W&'„|Zήk~`{Lpξ4?Ί1Dš@C³[ΈgϞΦΡΌyσHG€€!Π€IιΥ«—μΨ±Cξ»ο>©m.?ψΰƒ²{χξˆσQAνγ•pΡEY’ZΙ’%eΰΐ— Δ% ž^_Άjυ·Δχ ΖKλ3σwP=΅ςkKFŒοΗ“STΛ­ίέΏύ&Ÿ5n,‹οΏ?hQMϋά»t©Μ5gŸΎει–Ϋ8ΞϋΫfΟ–™ζ»ΧJ(QMΫ«φη /ȌzυδΟ—_vvπz•ρœiΔ@υζΛMT³Ζ2α/7˜'Ύ6 oΏ=δΠ”‹ΝΏe3Gβϊ‰ƒΥtPσ™YυKς™i·έ„ΕΦ’9V°s’^hΦBγEm@ˆ@ΉΦ"1<ΉΥ"A•>!?^2?„τ‘3ώΩ΅?³™ήtΣM–ΧšΞHE΅/Ύψ"v&ΗL–@Χ]εwή±Ž+Έ"aΧΙΒ kTPΣ]ίjΡ؜‚š‡‡†£,gΒ΄a€ςB`ƒ‰pσCχξr$čY&α³±gυώ——a­6κρ΅Θxq…j*ήΜjΧNφ›°•y5 ©ψUλΦ’SΑΪζΟ>“οŒ—ί‘Ν›ƒmβU/λΰAωεΖeν[oy•ϋ»PαiώυΧ‡όήXύeeΙrγU¨ΫaY#[gΔ΄?ΜΏmΑxnύ6ΏΎ3‘!wOκά,šcε6ξηΐί>pyoOK@€@Dδ–k-œƒβ­Nšτ$+έ€πίώWή4aZ4ίZc³‹ƒ Δ$ΰρZ{[CeύcMΓ‡"Σ‡σΥςΡyo5'!Pμ3y³ζšœ\vΟ«’ζϋkuγΡUάxWiώ5Ν#¦ω΅ΦΌφš•ΧΛ«γ©4ίE₯š6•²F€ Εvώτ“,Έε–PšXu5œŠ[>B Ωθ ωΎj\v™”:ξ8)R΅ͺ•“μΐϊυ²uΦ,Y3nœ§ΉΓ~ΈπBiχύχRΈD‰€sΩύΗ2ΫlfRAΡi₯[Ά”šW]%%κΧΝ=¦Ήιφ˜œk*ήν0†ΌΜ„V\π'•LξΈ@ΉΠ?ό°%ry΅5V²Βι§[ωΤ4§Z‘bΕ¬εΣO₯PF†³»œkο<#β9­όgH¦³ΈY›ζΩ;lΨ}+VX‡ζ_ΫτΡG^MŽO?νxEs,―ΙqvδX ;Δμk‰ωΎ²*Δ:Pr­εu-δVΛ+9ΪA qτλΧOƌc-Hσ;κ5@€@`Ώώ»hΦ,γΰfeΛ–υΨ‚Ν±HPσŒEn5 ^!`8s¬ΩΫ€-*'˜°ƒΥΠ–ς7ύ~– 1ψ§ω½π«ζv„‹Tξ¬ €°yάΜ-ΗZњ5½Β?–2K+±I_K›#ΥHιεΛKFŊ9]f›Ώ΅ί΄o/یPf· #’hς±U:ϋl{±ΧyΆ΄Τl±Ι―fώh{έ«fΦάΖ€ΡπgΊφ―Lnϋ]&|ΎέR‹‘_]ͺ™¨n¦σ]ρμ³ς»ρΚs ruM~ΞζΖ£ΜΝΆ‘o–΄œœK˜P‹­Η—2FΘs³£FτϋmθPYiΖtZύΑƒ₯©ΙGηΟΎ1μΆ:’mβ‹RLjdμ/T…B0νΦΖDS¨f<έ,šcΉŸŒeϊ=₯wοήΦΥΫ~€IaΑ@(Θ°`€@ˆPr­εu|ΌΥςJŽv€ @ΙLΰ8γ‘a”ύ™Ηƒ-”lώB>:Ηΐ[ΝI„k@ ―R'ΣΙ&_šz{Ή‰jΪojZšΤ5f–U¨ΧPκω΅&А99ΥRR€ώ!ώηŸ-α§ζε—[ΒZI#"ΩE5P…=§¨VΘxšφωηE5m›bΖilΒͺxθ4Ν'ΆΣ!šΩλ,κ)QM=ΗN1iώD5m―,λ  Ν{ΜޝuΎΞy*Ό9MΕ±Ÿ §¨¦^„Κȟ¨¦ύ2βh‹gž‘ζO?νμVV˜œp6nτ)Χ‚l#”n7ΉγμVΥxΉε&ͺiύcΊu“Ά&GŸlo.+MN97‹ζXnγ'k™ŠižœκαΥ”%ΒZ²~’X7 8 ι\kδV‹ƒS„ @ˆYχή{oN5“΄ lšΡΝώϊλ/9Α„υΊΘ<̜D~r«ΉQ€ Θ wά!•MΕ`L…€Ϊ}ϋϊTUΟ,υ ΥΪΌϋ45!ΤS ΞΦ€}/7avΦμρΗ₯TαΧk]sγβI΅Δ„sw3‚VQΚiυŒη^Ε3Οt»^Χ1y˜Λ7»4"ΧΆoΏ΅YηkŒœ†’΄[±:u€₯ύyΪλκy]“{³ͺΌμ¦snό΄ΞζιQWΑδ° Φ4$e•σΟχͺ}ΊY4ΗrŸ²π@X /Ο„ν­K—.£GλΘ“6a°0@ ΐDk o΅{[€ @ ¨ΧZO—‡΄nKSm’ρŒp³9sζ%¨i[ΌΥάRδ…@š Y[Ο„% Εfεφ²·Ωkrtν˜;Χ^”λΉŠtΥΊwΟ΅žVΨdr„iN8»ιάk^q…½(¨σΖaπ°Ιζ΄ |ΰ²RοkˆΚ†χάγ¬κχZ½εκΈ„ΪίaςΜΩ͍wœΣŽ7‘ϊƒΥ]Ds,ŸΑ);„΅°#MΜg\š=.“ϊ%ƒ -‘ςZΓ[-Zο γ@€ $2ϋLx1 ³-Γ[-Z€‰O † ;˜VΊtH -bršU>ο<Ÿ6ΫMn°`Mσ“55*Ψκ²ΞeSBΝ+―΄ΒέΙ?K5i"%Ν¦/3aέ<Θ֏:§UΉΰI ρΩpεε—Έϋ·ίΌΊV~{/φ*+or­iξΉPM½ΘΚ΄jεΥlΟ’E’iΌ£ζ&Ϊ©ΠxhηNgUΏΧΗα°ΣΚ•^‡[Ώne‘ΛοdΉ6ΡϋζΆ)Σ $Hx­α­–lŸ"Φ @€  ‘x­εw|ΌΥςKφ€€@(!μν*Ήˆ=Ξ<]φϊΞσrmΫJ±Z΅œΕ~―έD»Ϊ7ήθ·~n7ͺΉxo5˧َ~π)s†=τ©ΰRQ©’΄3yΜNϊλœΓι)θ6~Ή“Nrι-Έ’ νΫϋTάaΌ£ζφΨoD²oMϋP=ל};―£9–slΓOa-όLι€ΒL ά^kx«…ω ’;Δ9–-[Jw†Ec=6ΞW»ΣŸb’Βοw ‰»3ffρ@`ι₯ςή{οYΗΒ… γaΚΜ I Z^kx«%δΗ‡EA ΐT0Qy±Š.’Ν^σ$X+ετ Ππΰ–-²oωr―EkΦ )·šWcsQͺiSg‘h8K»άΊUφ­Xa/’“ RωθΌš Ι”·η(έ¬™W•mί|γu­N―3Ÿ  άΦΈΟ‘ΏM›«θηγΑgΚwύς‹L―[W~ΎϊjΩ0mšΩ·/ΐhΑ݊ζXΑ͈Zω!83b~z¦- @ ŒΤkmμΨ±rθΠ‘|χŠ·ZΎŠΐf·§Xdόψγr›Ι]ρ½ οςΠCΙέwί™θ5) L:U† b­}Π A2*„°JI ŒEC B<^k“&MŠΠδV‹X:†@’Θ0!3*VΜΣκ‹V―ξΣξΙ#¬•4ს=όαS΅h>e‘¨ΐγ΄CΫ·{Ή[δ˜c$­T)―zαΊΨυλ―>]ν1‘!W½ϊͺOy0N1RΫμ_½Ϊ΅i‹ηž“Ω&OΪQΗ&ΐlσόiΝk―YGJZš”=ρDΡπ”κy¦aαβΕ]ϋ TΝ±Νƒ{ω'€°–†τ@Q ΰρZmΧζΗπVΛ=ΪBΐ―ζG²Šjj#GŽ”λ―Ώ^*Ήό˜½gZ@€@,P―5Ν͞erυDΒπV‹Uϊ„@ςH/[6Ο‹/dr€2’ΚQ›Σα„΅Β!δ(sμς-¬Uμ³v§°ζΆžτ |Ϊ…«ΐ9Ύφ»xΨ°puoυsdΟΧώ*žy¦΄ύθ#WqΝΣ ϋπaΡpŸz,3ΏiThS/<Ν9wL·nR’~}OΥ€―Ρ+ΰDΈ™o„‚Μ7B:€ hG5ΌΥ’υn1 Ώ \sΝ5’ž j{̏Ωxΰοό€Š€Ηk-‹"·Z$¨'’›@Z>„5%—^œΐ¬ƒ½Γuα&p˧ǚ›ΧΩΡΜL―)ήΉΣλZ/ςκαηΣ‘£@ΗΞrŒο¨–Λ@ο‘ ^§υ•T8λ¬ ΖR‘mΫ·ίΚοC‡ΚΜ δλSO•υfsI0Ν±‚™O’ΧΙΞΞ–χίί:&OžΆε"¬… %A€@€ xΌΦς:ήjy%G;@y'P¨P!yτΡGs:xι₯—Dσba€ x"•k o΅Δϋ¬°"4όx¬ιܝVωκόρpσ²J/_ή_υ ΚlΪδS/­ti―²Γ»w{]λE~ΗυιπŸ‚pδ/σΧ·½<°¦υΚΆn-§ώΉ΄3^iΗήr‹­UΛή<ΰωv‘γΗ^½dξε—‹ 9ΉY4ΗΚm.‰~_=ι{τθa={φ Ϋr 6”t@Ρ Ÿ\kx«Eγb @Ύ:wξ,νM’χ/ΏόRŽ9"wάq‡΅cΠ·&%€ Ο<^kαΜ΅†·Z<"˜;b—ΐΡ|x˜ixF{H]₯Σƒ-\+/μ’Σ,σ―ΏςΥύυλ}Ϊ§•)γUV¨X1―k½pσžσ©”‡7:ν¦Ε /HjzzztoR¬fMχŽr'Ÿ,z΄xφYΡάo›g̐-ζwŒz¨qνΝΧ½υ–­VMššp‘ΑX4Η f>Τ žΒZπ¬’ΊζG&ΞμΑώΑι₯‹¨Χ@  xΌΦBΝ΅†·ZAΌ[Œ @ΰ_£F’Φf¨ξΰΤί}χœjB¦`€ Xk o΅Δϊ|°Δ C[Άδy*™kΧϊ΄”°ζζY—ΉfΟψ‘Έ sŽόinήiϋ]ΦΚΈώκͺxζΜY§u+š°Œ%6τΧ,*ε₯›7=κ$ΩGΚŽŸ–ΝΣ§Λ¦O?•νs战2§-3Ρ:T0Σάk‘X4Η e^Τu'@(Hw.”:\}υΥ9.“»sQζMΉ„ vyΙ΅†·ZΨί:„ V­ZΙ₯—^šΣfωqŠA€@βπx­…cex«…ƒ"}@n†YXΛ¨\Ωm˜|—₯;/νpκΥωκwŸϊ΄/}όρ^e•*y]λE~=ŸmnωΫφΖXψψβΎ\›6θή{₯Ω$xyή}·ˆ)wΪζ™3E!]Gs¬&Fεk9(8 x!ΰρZ vΎx«KŠz€"KΰΏύ―dddXƒθλFƒ Δ#\kx«%ήgƒA V0αlؐ§ιΈ >εΪΆΝS_Ή5R/&§p³oεJΙ2αΥσj¦MσiZΦFv+c„Ά”΄4{‘ρΰΦ­^eΑ\ή΅K¦˜οώ“SRrŽyΧ_οΥ΄ŒΩ„η47ΞΞ:y­!›<τ΄?ήg;~ϊΙ§,?Ρ+?σL¦ΆkΙτn³V@ D ―5ΌΥθg)ˆ_~ωΕ M¨α tΩ½!“ΆΛZ&ψH“o`šω1ΥW_IΩ²e“–  H7fGf͚…§SzςM ^kx«εϋm @ ž|γ'Ÿͺαz/;+KVŽνs―Β™gϊ”…£ p‰V(B{_GŒP΅ώwμEAŸοZΈPφόώ»W}Πʚνv+T΄¨”iΩ^d―}σMŸ²ά 6|πd:δU­ςyηy]—?ύt―k½ΨφΝ7>eΑ,Ίη™^·Χ±ϋ·ίΌšο]ΆΜλΎΦΟ Χκ½{Kρϊυ½ϊή΅`—ψΝ±Ό&ΒEΔ ¬E -C€@$ 롆·Z$ίϊ†@bxρΕ₯{χξΦ1έΔΛΗ"K`ΐ€rΑDvzO^x‘ΌώϋΦqε•W&ΝΊY(β@~½ΦπV‹‡w™9B Ύ ¬Ÿ4)δ¨π²Ο2v+\Ί΄«e―“ŸsΝ5ζ΄O=ε, κzέ„ >υͺvν*if N«₯‹³Hώ4’’ζLΕΦΏύΆWυ”Β…₯9ηx•Ή­Q=λv/ZδU/˜‹}f³δR“ίyΏρμσ7o–y5/V§Žε΅θ©£―[ΏύΦ«N°%}§•,)©f‹ζXž1y,„΅Θς₯w@ˆ `ΌΦπV‹ΰ@Χ€ @πC ?^kx«ωJ1 V›Ν¦ΊM3f„Τη’Gρ©_­gOIIάcφ:7ή(bΒ(ΪmǏ?ΚΆοΏ·εz~dο^Yσϊλ>υjυνλS¦΅oΈARΣ½ξixΖPς‡iݍ~θΥGωSO•΄R₯ΌΚJ›ΘΪ΅σ*3 ž,1‘δC΅E&ο™ΣCζUWy ]Ϊ§ _ςn›ΝηA½C5gθΗ2'žθΥE4Ης˜‹ˆˆάρ›2C€ώ&›ΧZaσ%IΕ7 € @ˆ>Όz­α­ύχŠ!¬ά|³d_Ÿλς³–yΧ]'»Mˆ?»iΕ†&μ`$­„ 3XΉsgŸ!ζ^r‰8ΓϊT²,Όύv9ΰXk©-€r§NΆZž©\YjτισoΑ?g?]q…μ4ατs³C;wΚΟΧ^λSΝm-Z©ώ!>uΧMœ(λί}Χ§ά_Α:γ…¨mμ–Z€ˆΤ8Π^”sξΐφ.^,kήx#η~0'[ΎόR:ςυ•ukΪO4Η fήΙR'ňžΠρϊ.CX Iϊ ! Β™ώ#ιfΝOί0@€ @ ϊς⡆·Zτί'F„@2Ψ·b…|m<¨vόό³_ ‡Άo—ο:v”ΥcΗϊΤ©uΝ5RΌvmŸςp45‘ SMή3»e]+_Ÿvš¨°Ȏ<(σϋυ“U/½δ]Ν.;«blϊL€ @ M@½ΦτAVV€’vx«ΩipDŒ@‘B"F΄QΛ\½ZΎ2FΞε‘((ΥΈ±¨ΘυλύŸWχGvν’οΞ=W*uθ e[΅²Ό’J›π†YFLSOΌ-&£z`eYγΥN/Ž½ε)ίΆ­OΉ½ £B9ήδ€ώ±G{±6ήhߘЍ₯š7—r'Ÿ,εN:I •(!{~]vΟ/¦Nσ‡ί«ή?ι½χ€pρβ^εφ‹VγΖΙ—'œ ϋW­²ΛRrΣ§ŸJ…ΣO]Ÿ΅Ζd—ρ ΤcΣ'Ÿψ΄Ρͺυξ-Ηήt“W_φ AYΠ Y:bΔΏΕζs±δα‡e“ Zλκ«₯DΓ†ΦQ΄zuΙ6b[¦ w̝+«^yEΆ|ώΉΧ𻀜Ν±œcs~)&Ω ·„ώ1θ1\eβΠξή½ΫZΙk―½&₯qp`‰,ˆΛ 0q@IDATc'›/qsζΜρYΑ—fΧΦ™ζ‹1@ ~fχζ˜1c¬*£ΝŽM½Ζ @‹Α$ž+7So΅UζajΉrεr«Κ}@!ψΔlΚ΅{O―[WŽ1bΡ2#VεΕ2ͺV•vFtΛΝ[mΥ«―Κ|G³VFθͺyωεyV~½ν6YρΤSyjkoTύKεΔ7ί θ­f―ΏόNZ žfωZO˜ ΥMψΚάlχΘχηg žΉΥ tΏ€#ΫύπƒO>7g›£™™ςYΣ¦²εJη-―k )©‚e eΐxΖ·ίϊδsσtΝ±L8ΡR&„€†ΥLMKΛΣ°Ρ+O€‘_k~Ρp‰Ma-±ί_V$„΅Δ{OYΡΏ6šγυκΥ“}ϋφY…γLrΝρ‹A 7£F’!C†XΥ™έΙzA±KΰΤSO•οMN"§6a³6™‡ΑδVs’αά„΅p ύd›(ιQ Β=‡¬Γ‡EΎ`BKζwμhŽ•ίΉ&s{Ά&σ»ΟΪ!@€ ͺ˜P/*Šxμž{ξ‘LσΓƒ Δ"Π«W/ΧΥ5»ώΥ\ΡP@ dˆj!#Λ΅z€ECTΣ‰Ds¬\NΏΦό’α @€ -ƒΨΤΦ­['O=υT΄†f@ˆš5kΊŽ€^Λ @7 A=Υ„υακa-\$ι€ @Θ3βΕ‹Λ°aΓrڏ1BΆnݚsΝ  $.τττΔ]+ƒ @ ΐd™Π¨]t‘utοή=lσ(ΆžθHS¦L‘Z΅j%ΔZX x$pηwΚ§Ÿ~SgΞΘ7λ»Nž~ϊiωγ?dχξέ2|ψpyζ™gςέ/@ ž Lœ8QFŽΟK`ξΘ!°sηΜsϋΙ_|!-[Ά΄qΈ%Π¬Y3yύυΧγvώL€r'€°–;#jŸ|ς‰:tΘbqήyη »ΙχcΡΈqciΠ Aβ.•AˆqeΛ–ρ2=DŽ@‘B…,‘kΧΦFŸSN9%rƒΡ3β„€znώςΛ/q2[¦ ΌΨ΅kŸσΌ‘£U ΘΘȈΑY1%@'„΅pLΰΎΌςʜP<[Άl‘ *$πjY @€@AΈΰ‚ δ7ސ^½z ¦ κ]`\@€ @πGaΝΚ!@€ !pωε—ΘΈ X$  Χ1@€ @ v ¬ΕΞ{ΑL @€ €ΐ‚ dΥͺUΦΘ-Z΄Ϊ΅kΐ,Θ zυκΙ…^h5=ξΈγςEΜ·III‰ω92A@€ $„΅dz·Y+ @€€Ρ£G˘1c¬r=οΧ―ŸO Ψ$Π­[7Ρƒ @€ -kΡ"Ν8€ @€  ψ yρΕΛΠ‘CCμκ€ . .”«Ί*\έΡO„ ΄ž0A²Μ₯PΡ’9ηœ@•ΒZ¨Δ¨@€ @ˆ‘ +T¨ -[ΆŒ,€œΪDη=c@…ΣN‹½I1#@ βτ»tΧ]­qRSSΓ6ΒZΨP @€ @€ @€@,P1mκΤ©aŸJψ$Ί°O!@€ όM`ΫΆmrλ­·ΚπαΓA@€ @(0x¬ϊψψάsΟ•έ»w[“NOO―Ι3[@€ Έ&πΗHΫΆmeΧ]R¬X1ιΫ·―T«V-ΧΔδ!@€ @ > ¬ΕηϋυYΏρΖQ“!@€  5j$uλΦ•yσζΙώύϋεΎϋξ“±cΗ€ @€ u„‚Œ:r„ @€B!  §G•ΣdάΈq²pαœkN @€ @Ρ"€ΗZ΄H3 @€@LhήΌΉ\pΑΦάjΧ“sdR"gu–tξάY>ώψcΙΚΚ’‘C‡Zη°In+V¬E‹YΤ«±I“&Ι „ΥC€ @'€ΗZΔ3 @€@,Έωζ›eΪ΄iΦ‘ye±Ψ%0rδH)T¨5ΑO>ωD>όσ؝,3‹ χί_Ίvνjϋί’2&ƒ@€ @ΙMa-ΉίV@€ Έ!Π΄iSΉϊκ«sζ;xπ`ΙΞΞΞΉζ€ @€ ik‘&L€ @€@Ψ >\Š+fυ7ώ|™0aBΨϊ¦#@€ @H,|πθρᇆmaδX J:‚ @€"Mΰ˜cŽ‘ΚO}dλΦ­VΏ[Άl‘ *„} :„ @€ @€ Olά(ΫfΝςšςΑλ―χΊζH,©‰΅V@€ @€ @€ Θΐc-2\ι€ @ N,\ΈPV―^mΝΆY³fR«V­8™9Σ„κΦ­+]Ίt±@4i € @€"Na-βˆ€ @ – <όσ2fΜkŠ£G–~ύϊΕςt™ `#Π½{wΡƒ @€ -„‚ŒiƁ @€"N`ίΎ}ƒ @€ @ y ΰ±–Όο=+‡@ΒΘ>zTΎ9λ¬|­'%%EΚ–•ŒŠ%έεΪΆ•Š¦ΟΒŊε«_ϋX:j”¬xζ―'Ož,eO<Ρ«,szυ’ƒ›7ϋt]½wo9ΆŸςΌά²EζτμιΥ΄H•*ζν·½ΚΈ€ π8r䈼ςΚ+2lΨ0ΛρΒ / οτ@€ @qE@Ÿωžώω֜SSΓηg†°W& ΈΘΞΞ–m³fΉέΚWYjF†TξάYŽ1BJ6h―Ύhό/#»vɁuλώ-0gYz]Gκbǜ9’Ήv­Oχ;ζΞ•Jηž+%Lž–όZΦ‘C>ŸΗ’δkΚ/VΪCΘ•ΐ#<"χέwŸUοŽ;ξ°~<.Μϝ\ΑQ€ @€@‚P1νΓ? ϋκΒ'Ρ…}jt@ ` ¨Ψ³ΑxR}ή΄©,Ίχή‚L£άΊUφ._ξu=p ΘΦΙ[-+3S„Ιc-y)²r@Kΰζ›o–₯K[“XΌx±ε½V°3bt@€ @HDk‰ψF`M:t°vύͺΫdzzzF KΔ.μΓ‡eΙCΙΒAƒbw’ΜlρƒΚΜϊυ½Ž?σσŽ… nž>]ΦN˜ Sa€ εΛ——;οΌ3§₯†„ά»woΞ5'€ @€ p 6J8(&AγǏO‚U²ΔD#pLRΈdΙ€ΛΚΞΚ’#»wΛα;­ά[{Μw1enΆόρΗ₯d“&RϋΪkέnS–(Οήy€IMuoό,°τ²τή‘ήι‚€ 6μbύDΕς©€μ’bAΐ‚½{£ƒτ&U:{gισζ#ϋΟd2IfgvwΚ{ž'μ$ΉΉΉχ7ΓL’sΞ{ʞ}ΆδMKK€Ωp $@$|ξΎϋn:t¨ό«²Ώ›7o–΄ηO<‘| 8c    Έ#pt~Y§υΩχ.\(ϋW­’«WΛΑυλ%o‰R bEΙ_‘‚ρ·D›6RNŸ]€δΞν9ΗτΕxΦem©x₯;t°n λυφΙ“e0¬VT՞ς•.mέςυ‘½{eσΟ?Λ–ί~3Κu€oΪ$ΗTI¨@εΚRPKi€5o.•ΈBς•*²P;œζ[’ukɝ?Ζ!8š‘#eΧμΩ²{Ξ9‘΅š½ό²”νΦ-£Σ‹ΈΎεχίe?”’–-3ώή±Cςj€_^k‘:u€LΧRVKΰύ’%6:ΦϋύεμH © 4 ipΚ$@$@$@$@ρ@`Ώ:ΠVΎώΊ¬yη#πΫ>ζΓ[ΆΘ>‚[,₯JFΰwΥ›n’‚κ exΆ5ιΜ3ƒvŸ9wk$h»Χ†kΦΘ„Άm›©£›ΞΑΛ5πmQώ²φ£δ„Ν1‡cχ«³ ΆφέwΟεΞ9Gjkϋ-[Ϋύό³IvszχhΪU”ζσΑΝͺV4η–[δΰΪ΅mmέ°n]“s‘Žc½:=εΔ λ.γuΊ:?±μ™7OΦρ…ˆ:<«ικι3ΔόeΛ΅η†Δ @ΗZbΌœ @” ¨r―^RΌU+™zώω²wρβ€žκΔΪ?dΦZ•ΔZYύΦ[REΘ–Τˆ& @rhΤ¨‘!yΩVΥIZ|Έϊκ«εe8« φkΤοc=&oΎωf|OŠ£I`εΚ•²ψδυZ͚5₯nέΊ!ΫrGl8v옜ιπΐΟm7ο―³’m_~ω₯Ό1CΩgŸ}ζκ”ΏτKe‹>θ΄Ϊ€€«Fͺg₯ύτΣOςάsΟE|Šzίƒ/ΦΕ¬Sqη>;Θ)†>‡Ηf$@$-ΦιοΝ ½~ύ= ΗΧ­/Σ@²6šEκ2Ɋͺ N«­κ«L9ΦpœέJuμθκάCϋΥoΏ-ίwŸ‘e?ήieY6~χlόώ{©Υ·―ΤΣ'y tjκ{ΫgŸ•Eιx?’δΚ άsr†<^ίΛΥz'ΫκΘ Η1²Oξˆ9t¬Εά[Β‘ āΒϊ`¦Γ_ΙϚΊ}LΚYm‹f΄QJ$Ύ_η.T(π=V)ΠΉšέΠISόsεαΟd|Ώ»= ψ#pΗwZbΘ₯³Ο?|ΖΓν4"φ駟–R™’I "‰=‹―ΏώZxΰc’χk=\Θβ‹ΐ όž0aBΘAΓqϊΰƒjπ··άUΘNvŒ1ΒυΌιιιGύ¦iΣ¦²³ΏEδ&ΝΘjΫ€ nΌ2{ώl§šuxf5Ee!ۍ+i͚YweΌ’Ž»Ž΅ϊκ4 ΧΦk`ŠέΠΏ›Α9΅ψΡGݚ„ή§Οh–Ώτ’lωγi―ΌR‹ έΦeΟΚaΓΒvͺΑι™ιqλXPvf²άΈ½7.Cζ'+ΖΗΗα‘ @Žΐu‰3Ξ:?t”i‰C žCν=σηn‰3KΞ„H€’‹ΐYg%έ΅Ά²!ώώϋo:Υ’λνηlŒΐV•f3fLTg…L³qγΖE΅Οxομ¨Φ—yε•W€ŽΦ‡yύχUι*Xκ*ήηΘρ“ @,Ψ»t©L½πB9~θPΐ°r( 5οΉGš«"ΎΟή°A:/X mΤ±ΤXΏ§‹5mΠ+Gwο68Ηl}™ +]y₯ˆŸY ’{΄ίpμ€Κ'ξΤΐ«aΌ.ΉΔΊ)ΰυ’§žrtN₯δΝ+₯;w–Sώϋ_iρι§rΊf§5QηW•nΕ‡άMϊƍ²ΧRΐΫvΓ…ΥqKQΨΤβΕ%Ν!ΒΙ~ΦjΈ(Ψπν·²O£uιφ fΕGΑ5€Jf©][ͺjQΦ‚Uͺ8u‘©m`΅φƒdϋδΙ‚ΊvˆŠ:‘?ς—+gpBTS%uzαβ2ΦD/ΪΆj$τΡ½{3Ί;vΰ€ΜS9œ3~ό1c[vΏ8¬²›υΗγθΡ²K₯)Αί#ήΗZ¬Έ΄Φ$)«Ÿ—2]ΊψϊΜXη`ό`_ ­-—;ώŒfG”Ιš‘# τέsζο#­]„‹PΣΜΟ³ΉŽΏΕ΅Έp•Ω4νΈ~nPθwϋ€I‚¬OΌ§(βk|†τ…}ρ!”$'p#€Οε!•AΚ£Y₯ζ "πό~CƒΫI€‹j‰υ~r6ΙM`”^Σ(φΤΤΤ¨€ψBkž$’εΧk·–zνεǐ™Ά}ϋφŒ%TfΪ;οΌ#=zτσ΅ξ4H€H ϊvͺ³ΕjyΛ”‘zΏœΗΗ3§Š άhπ`ω[3Ϋ¬ΆC3άBΥZ«¬rNŽ΅z?nνΒυ΅“ dε«r<AΑ3jΗαή½™nΈlηΦ,Έ&―Ύ*…kΥ’ωκ¬²Ϊ ­Zλή{ηCΦνn―κσΓΤ©‚ ΉϊšEηVϊϟΐjε5[ΞξT³ξ7_WΈθ")¨u»Ηžvšhϊ·ΉYVΎρk4²χ‚©Ξ=™Mˆ²Ρ ’c-{ίGžH Ξ8₯ΕΡȚplΏ:ΠVκ½9…Ύ²έ« Rπ­–_&¨γVUλ4¬\ΩΊ+ΰυ&uϊΜqHM·6š―ΡLV+₯QJν<δtΰΔ@4¨ θΦ›q,ˆΊ‘l5k- gOf )ύΘ[₯ηub•ŽHQΣ&M³[/€ΰŒAΊ½Ϋ™Ÿ±ΐ±Y_λοΜΧ¨"«Α©… NJ*%–vLkzό£XΛ΄0=|N†Θ6,{Tβ ‘n…ΥΑ Yˊ΅†?ΆιηŸeŽ2΄ZWuϊͺVΝΨ΄Y‹μΞΉεΙΈ=ΩπJ2Y Ÿν₯zaj΅3U¦‘X£FΖ¦Uo½%<σŒΠΎ­Ά_lp΄Αpό)EV[£-fX,μί_VιE¨]"mvλ²Y,S^pJ·ψψcγ"Φ<žI€H€H€βŸΐŽ;δwU‘8ηœs"žΜF Lˊe,‚*h€ήψργΓξaƒΚ‹ύ¨ΧQΟι5η UŽ[τ:p‘–p ~ ϋ€<€H€Hΐ~ή­χΜVC&š§šy ²ŸμŽ΅S§š»ƒώVΈψbAΦqux™†ΐokϋyΞuΰίe‡­ΌZΓΈ¬Κ―;‚₯νΟ V―ξιT³φ&[5Km£ؘ†ρ/Χ`ί†ϊΫ΅D֘rπ²½ZίΥΚ νKuθΰuXΖώ΄ζΝ₯œ:r6ύπCΖ6τIK,βͺ‰57Ξ&ŠώP_€Lb9rδH{fW$ې1e7?ζ1(tϊ›FΨ,ρEGG‘ΩΞώ™gK”?4|»-JΖή6Ϊλ8ίοZ[Ξ@'§šΣωNhF˜+œ(™±C*[0I~pΐ89Υ‚ϊΤ4ώΥzq”™\‘Z ½ΐ,~ϊιAέΜοΫWŽμΩ΄=«6ΐ™‹χ}‰Κ%„rͺ9{ί?Θ όϊK/ήΰΤΰ,’Θv§Z8ύΒQ:C£ΧζκCϋ΅½莣(πŒ^½ŒμDμGfΫΨSO•zΡμδT³χ±CΏŽΡ XdiH€H€H€β—@qΝΜΟmSo€d4 2Η-rPθ³”>LFƒCξf ›€NNDY9lR…€{5˜F$@$]pͺα9ŠΥΒqάΰ85Ϋε νA°Φώ!ΧΩF»9e‘ΩΫ`ݐ΄d`aΚg8•©@6τr ‚Ά[S ’70ΊξΓΫ» pXν ±Σ~œj8|§tΫ Κ9ακ­Α‘h.ΘΒ;"p:œ~Ω6vΠ±;οELδJ}X‹”I,»υα'’ΐQΥRΆ§~cήεΞ;ΟΧτ·M˜ ³΄>_η”S§ΗTqŠΚόνR Ύμ°-κDŸ¬υ+E—ƒσcκM«YCαHp9ρφκQ?³5»/RKΡtπfoΎ))6‰‘tδEΖTv؎ιΣe‚FAA&4³ΆY³ΠΖ΅haȍfΆd*.μΓH 34{nκ¦‡cΈXΗψ!ύ NVΉΙ½Z8;’νpΞρ‚5jlK$@$@±EŽ΅Npe5ΤM<€Χš‘š]²Ž”Υ S‘"1ΔΪρεTno§j@“έ~ωεϋ&“  DH±ζο½°‡t`†’GΒ|F 9H»‘φ—srΐ9υ‡Ύvhm5»2SΙφνJJψ9'Ϊ ϋ+Νφϋ„ηυY_Λ₯’Ι _xΑosGηΚ€„Δ\CkΕu[Ή2` Χ©θ{ΐl˜#(™#ΨyR X'pXΞΓ‘e/$‹ZVε|HΠμΥzdSUٞe“K#Tͺλ+j·!σ ΊΛ8WΊFƒ"Ν»οΚξΉsπ “grΧ]£crηΛ°ςUΰΌ;i;§M{zyYo>Υκ6­ΘΙ’ͺζΊωw:¨ώθ%{Ί;φkΦLͺ\w>εAͺ?²0G8q‚€42 …ZΛ¨S$ŸΟθ_Θμ_Ύά ͺ‰JΙΆm₯˜ξE1Y™E’± rΘ%ΙFdVΊβŠο#― ]½νTj¦¬rν΅RΒgν λ±~_C2NΝ£φμ8uψ•ΡχΏ²:‹P¬5ΕPχ ²˜pήΥ‹q»#,§jMŒz1Ž”ΖΊkζL™Χ§ίa‡l©ΞmcΗfμΗ…,καf‘€Κ›"Bo§:ρΉ·Š«9{–>»(ζŒBΞΈ@vί–ί~ ϊ< ΰ Υc―σΠCφΉN$@$@$Qv…^ΧA9Ε4xΒΙƒΊ_™΅υzύ„μ,«]¦Ρφj Χd783οΠϊΒ7ΪΦ6«2j²•Τ‡ΐ4  θ(€™LX"±mqBλf†c¨•Žg:‡U1Θ΄=zo΅<ο eυχΣ]°Fυάπ¬Βn%Z΅²oς½Ž’&¨;o5<* υΜόjΙγω›_sΚ< N2¨,΅Φ$‘”@ρ;Ά‹}t¬Εώ{Δ’ d3d‡ΝΌζš ώΉΤ©…z^~ 5―μN98Z¨³N«ΑQ‡I™ΞϊΡλ5cgΊfϊX3έpΡ§’½m)ψΑbΪ<-κjw¬ΥΡl«’Z{Κ͎«Δ+ŠΚڝjp†œ¦ΊΨφc¦ƒ )ω+^{Mjv“υXΜ}©:§Ώτ’Ϋi3φYjpβ5UYΙ²Œ8`:’fk=7+l†c ύΤΥΪrλ΅ ½uLpμ!―£Φ”s+p‹γ3c'΄YκΈ΄;Υςιgγ΄?4>Φ~ρ™)’NΥzQWWε— dΤ¦Γ8MCν΅Ω*νΣςΣOΝMΎώώ­uΞ¬ύΐ±‰‹oό…CυΟμrN[j₯:v”ζ#G:ή8,Кj¨f5djξΆΘ9¨RΕψL@£άn{Τ17]ΌνU'›Υ6hΤkV"|νFυc֞,hέ@Ώ«θgŽ–Έ M3WƒXšιο -ώ ΤΠ‡:gkΰ,Ω³Žβέό ¨όEϊ°μφΫo(E9ΘHk_ιυΎ¬vΉ^w£~M€΅>xt2όNΆ·άs8΅α6  ¬'€ί°ƒZλl“/ΦgαdQ—}ΥΠ‘‡"k­Ž‹j“ de-ϋΚΆOœ΄ΛžuΤΐeCΡ† ƒφξ·Υpj`Ω€gnα‚Σ‹θ1φ@`ΒZ³¦TVe7Τ¬+­Ορς*NΧl›@θXK 7“S!$€~D⸚^” σŽ+dmΥ¨X{φŽΟS€ˆ΄΄o83όΨNuΐX-―ώ(wΠθX?ΩC/ΉD T„ΡAvǚυ‘Ό†τžΥ‘ΎRԁr†f€•V§H(ƒ|b-uζαβlžFΈZm:t0΄ρk΅5Γ¨ώSOΉSU3τr«ΓυΔ¬ΆY3—ŽͺC&‹τέlΔ™€HVΓ¦§ΐρe[₯ηΫn‹θΚ]Έ°΄Υθι’!2 Ν!€hv_=uΤ,0»$ζzΝβΫ₯xNπk5Υ΄_dοαύˆΔ™XοΙ'Ξ]ŒΣΙPpΡoλBΘ‡Χ¨ΆΆϊ2Τ\ ·Τ‡l¨Εv<==γˆfCqep‘‘€Χ4@`„ώ?„ Σ ΥΫ4³˜–˜Ζj&νυ»m:γ—h‘φjΥͺ%ζD“hV—θuZb@]ο%JΘYšΝ:ί¦}―Qβ”ͺ‘³Λ@"˜‹½ζZfϊN„c ‹‰Μ΅$g5|g&ΊcmΏήGΐq 'β*}X»ZΥ$αˆΟaŊυθπ·,Β™o―hεeΎήΈq£ρ[cγo.½7κ ο™΅ΙZSψ°­.SC}ΰ\Ίti_]ξUε‹Ÿυο7½wϊηυ§*λ5sUΝζhχ ΘΝLέA§ωΒY›_ο―LΓωGjΐέl ’›£­G5Ϋζe­§άMωh$@#€²$PxΑ³)sΒ ^γ~݌.·*TδXϋςKWǚ£ €‹c γ·Ϋ^½φ^­3cΟ';8°fοŠΨάύΨδυ׍’%(]b5ΤΖ[ϋώϋΖ‚R"PεΜ%²άψι3)λΉψ:Ά Π±ΫοGG$χίΑΡ(δλCΧΒ•βΗ smΏˆ@&Z(Η€SŸ{φ r¬I.:˜‰m¨…΅R/μV«o_W§š΅}u&^«™UΦ1›΄νκL΄fΤY±Ώ.―QΙ žyΖΎΩq’ Τ wΠr!…Œ9dυ•jΧΞρ˜p6–>σLCb2‹VƒD!²χ F1›Ε(κ«7“vkτβ‹žN5λ1UoΈΑˆZC¦–Ր9ΨΚΆΝΊ?Τλ–zŒίΒΎ‘ϊ¨¨οSέBνΞ؎h/GΗZξάtψpΟ;ΘEφ€I­έu:Φ¬DψšH`€~'Ν:)#Σ_ώŸ„Y’I€²‡k0d“Ykϋτa#Φ3γL…α/ T³d at¬ύ ‘jκ=ΟL•·&‰jp ½χBοΌσŽμΪ΅+hš[΄ξ/1¬V©R%C2σ&UΡ€C*”ABσL½―°²¦›¨šIΈΆFο}Ϊͺ\ΎΥΰ¨ΓΌ ’žψέϋ裏‚s8v™ΚΐΑήΥχͺ€ϋ9ZNν[†!…‡]οή½~Μΰ€4ƒX~ύυWΉε–[2TΜ6[·n5_ς/ $-νκ4G”_¨Δc8ΰ THΏλχ―X‘qΨnutοW©ΓBͺ`7Τ2Γψ¬–¦Ξ€"ŒΚPςΔnKάΎ)’υ£κ¨χk–Χdήϊǝkf_'τzAπX–iΠ0mΌ,wώωRAŸqΉΙkš}πoόπŸFΏsδΘI€H Σπ£ˆτod1ω58ΥΑb5'}fλ~ϋkθ5ΫεφeΡ ΗFDώž”@3ǁsΧQ™GΏ†›pԎ³ΫNΫMΉ}Ώu½ήOXW=_—;)ϋdmMFplε΅E~B’υγ’i›΅VΘώ“7²fΏ©)\EεHΓ5'†(°nAc8#uͺaμυ|^8γ’άΙπ™JkΪΤiWΠ6Τ΄[4?φΎΉN$Ÿk&΅iŸiV―ύα±ΉI€r–€ιX»PkηΛ—/`0Θ*ʌ9Ι@šŽ5HOώG`Ή΅ξρI(§ΈΤέ‰gnψ¨U«–Ό¨ΧύNN΅Ps[§΅―(υTYΒƒL2'>‹™1§γ:κƒ_7ηΞσφΫoKm½VF¦˜=ΫΝiψχέwί χ©ZΗ[Ά†Σ1^۞}φYιή½{SΝλ8ξ'D'§Υ4½ž NsΤz‡ƒΖ―S­ˆƒ<’_^•5kΝnƒt2GH‡γΝciΠ³΅TˆΉ=Ϊ#Νάσ38Χڍ'₯‚$œŽ‡£ ζ ΅δΕοϊ½;^³œCqu:žΫ²†žY"Ϋά\’u:Φ’E’ύ $$ό(.Χ‡pΏjΤΞ<Ν:Γ‚—Α)Υ\3¬ RΓΓ1d½…λ §k[§yDΧ€†Ρc8ΊτΗΚj{lu―¬ϋ¬―ΣZ΄bY7yΎΞ―r,v;Eηc^•}iμIΆiτhYΝ7φSgz΅αμVεΪk%w&$ŽΉGp€iέ5\Ψω5ΤΥkψΒ ~›‡lW\£ΰά"Ψ¬‚΅“•ΣHYΏζTˆ8šŸΏγ`; Ψ&pΖgΘΕZ†ŒaΘBH€b€ιX+Z΄hF =s”?jτ8dϋΒ5»C±ΦŽ5λς™η ·ΟDkιC'δ2Ν&¨ ϋυ*1‰SŸC<€ƒ€a(»Ϊα᳓ƒ,ΤρΦν_ͺT›έœϊ·Ά€΅˜oΦΊΛ{φμ±nφυ™œ/iΝlΘ9ξΦς ™5Θlχs©έ”Ω~y Δ;<‹ω£n]±«Ξ„šW. 4Α³(΅Σο°.ZW=³V©W― Cž ‘Q €*Λ@E(”‘DGvXv8Φ0βΚΌ–ιθ NΟ}ϊH•Νυk;¦L‘ικ8‘Ώφ―~ϋ`»Θ »ŠX~ψα‡Θ;<Ω₯ £†’‘ āόͺŸ’ΗΗΧάΙ:kGq³‘― h+Urλ˜1rΊFοΉΙBͺ^]°DbΫƏχ₯Ιyp¬UΎΡμ«άΉηš/}EqW\hX/nyεΗ EΩ3ϊpό!­•MC1^H\nQΩ«ΝΧΊre΄ζGΈΞGkζk\hΩ­Ϊ­·Ϊ7ω^‡Œθ­ a΅mzΡνχ=-‘7ΞNN*k~^Cώΐ―‘F_ξ‚Ε]y ί¦Jv;’‰φ>ΈN$x5?Zƒ$PΧeœF βζκΌσΞKΌ‰rF$ΗπΣ4ΘA~ϋν·ζͺ‘=ƒZk¨εΧ •7mΪ΄€ζθΧ4λωΜmΙψwκΤ©AΣF”72³Ι–.]*Θ†₯΅’ΥZΜvΛ«΅΄Ϋ©|>κ©a)¨Χα4kfϊτις΅Φ)·;αζ«" xύ’JNs΄χo]Gfx}M#$°O³ƒgi9‡c*ql7Τ»Gp6j₯Υ'Ÿ1αo$υΟ­η)’ΩT¨gΎΣςΫΈK%(Q·Μϊ< ]₯€ν2x’ΏlYkw―S5(ΖɚΌρ†δΉEΛ’Y’ΓϘπά K}6₯ͺ-Z«r«Jw"Ωxžθ J_Πg”¨1OK>ž8'Ξd9“ΜθάΉsF„.Βh$Ϊλ[‘jΥ|΅Ζ©†ώΆ‰eλΘΏZwVΫ«7T35€ƒκK§8<Θ·Ά χ5’W’€΄FP,~δ‘pΟT{8’¬ΊΪθΞΘ2zs˜ Η™bνΏX&j 8Υ¬;žžnν6*―›j„εŸz1kuϊ€λσ"•ΚlςΚ+R…φ"Ό΄~[Ρ^ΰΒΫnϋτα_+₯ˆδ’Ž₯hΤ[€ι―‚aόΕC  ψ!I³[5€aθΠ‘Fs<πDΖAnϋχŸΞΨ†H KX³ˆΞW%<ψ·ΚΡ!ϋ,ǚS†)‰ XY2‘8θŽ'œjΰŸHφŠ^ΓοάΉ3`JlDέΝϊͺa΅ςεΛ 2φπLδ.U0γ NYλgt›ήSΝΦZΟΘμ²[E}ΪIk©ΩVΓg2ΗΞk· .Έ@ΥιdS4xο1­m7dibžΝš5³ο2œŠψmΔοβkϊΰΨjγ4ε½G}ώωη­›=_CJϊ NΛnέΊΞKΌΖσ₯’ͺψB#d"p\ƒGπ\)Θ©¦Χ’υUfΆšΦ"ΜWͺT–#€Υ±†"kνύkΪ¨υhφͺΥͺθqnηYξB…‚ž§‘–½_U·ώca_1ύώΒrΚύχ ž%ξΤΞΘF©pV:HL/ΣοO8ζP{–‚Γ»c^œE” @{…h±„Ίp‹ς)Ω d;<ΨΟ―7N•΄ˆy³7ί”³ώωG*œ”‹²^+^}ΥΊΙχ룍΄]orΰ΄[ͺϊΩsτ‚i’:±~Σh‘Ρqψ«FEΞΣ΄ώΓQΞΌ 5ΐ½‹ν‚Δb¨£ ΖQΪJ Π΅ϋlr€ ϋΠ©vΩJ½ιΔΕS$ζΔΏ€Kt?ηBζ έœ ΫۘλElΜναώΝΤ{j9I>€Λm«©bΩΝ—$@$ψ7·%Σߍš‘€Μ'dfΩν‰0λ ۏΕυš•a΅2zύ33°΄­΅υxΎ&D$€L'»C Ξ-?ύTκτοοΫ©]$†g_v•'»€]³ςšΑκeωώ_οΣgl‰hx–X’eK©«ΑΒοYu Ϊ šξ–ίΪΖ ρK€Ž΅ψ}ο8r ,&P@L-ΏψΒΡΉ†l₯ƒš΅δǐ6ΏPTΗk]—Tq‚/©7A‹tΫκ·ή’­ϊΓΊΩ² E?ύF£Ν[€&ϊΜ› ‘Qφ±§+fίSλ΅ξΉGŠΩ£:5if< :)³v؁Ď5I†pkyΒ¬­jξ‘:gΓ•ˆ°>€ 5&n' “β!"ί΄Ιϊ[M#ˆφίu«l#F™*ί©<»[Ήr₯@ŽΞjΦl5λφdz ‰Cd5]―uΖͺ©J€“ dǎ₯§ΚŒ'’V‰HZ ™h… Άnr}νΔΔ‰ŸΩ j{Βae5HKBύ«ͺ&φώKι=[χξέΰƒdΥͺUϋͺk°ΰΫoΏν;ϋL.²eUΐaχ²C κ€…X#ρ9•@Λγ§TCˆ>Έ™…dνVν¦›€’Φβ Ηό>“ Υ'‚rνjEpψΠοXΊ[@ΡΙjΘΆΚ£Ξ5/K;υΤ &‰κX³O’υŸ|RZh@½έvΪGμϋΉ_θX‹―χ‹£%Θfˆ\IΟ‡5˜Τn¨K]ͺχΔp.H€ώG`Οί‘(©uΓ΅½ͺΊ©A2ΐτ;dΓIιYGΘk― hjΕi>ΫmNΊPΗ:mGpϋ―ZƒέΊμΡZ•Yaϋ4πέzΌΆgξω9/2 ©½ΥvΟ›'‘fZϋγλœ%ΐk9˟g'ˆp4!*g¦ε[Νν"ισ³oΌQŽξέk=$δk8!P“ j+©DG)½¨• R‡Gφμ S^jά1Α†βqUS#7WΨ’4Q―’JΒΠμpΝισ)τΝ›ƒ†λ°ύaAΠ$Έ²ΐwή)XhΙEΩ?i]SZ|@δΕ`χk‹^x!Ύ'ΔΡ;ΐΧ=zΘΗœ±wU}@¬βͺαfv\‹-Ω;‰dpΝΣ‡uΡ4dAͺ/Ρ E{0ΕizŽχξέ»Γ9Đ>΄ρύ…Ίe^ζδ€s’—D?ΘB΄gΒ΅oίή¨kζuϋ~H¦žͺχ@4 ςΙ¨ΗWAΥ]όX~ Pεχ²Rl“LΰΈ±[ΡFμ›<ΧWkj€V^[sk°…΅ήžgΥκΫΧ¨·fν?Ώfb•Φz“~ υΤμΆqτhΩ£ί!EΓ,?±_ƒ¨ώΡλ»– ŒΉ°ΦŒΜ +¨Χι*‘|ά ΌMε‚ΓΝ(Δ؊θ‘PeZͺ*εbζ‰#ξ2c-ξίBN€H ;ΡΘN»α‚ΐΙφ-_.³nΈΑΡ©–’E\K¨$d>}ŒLΈ3΄Έ)jΉ―΅Χ:MŸ.MίxΓpͺ9υ›Ϋr;"wʒʊsΗcŸυ ’ZΟjpŽΝΛδω<ΕΖ‘ι‰₯―_txjZZΠ6n    x"`—ƒ„scΤ¨QSX¦³ζΜ™ΠΖήOΐNYM ΌοΎϋ’œͺΧ]w]ΐηO86~όx9κS‰Δμ·[·nωF«!{ŸQ7[―Χφφϊm5jԐ֭[;6ΑAU₯U«VŽmύlμΤ©SP³i*ηΧ0ΞͺΆϋ'ΏΗ² $*Τq·ΫΎ0³ξ·Ž#pTEjyτ™P[FιuΠοίΞmϊ]g΅*Χ\#)ΉόΉŠ©£°”=X³α–ͺšSΈ†2*V§Ž―’ίγYε BΏišn5ΤΕ;qόΈu“―ΧvιΗ΄09|„rŒ€Ώ 96<ž˜H€bƒ@^§Β«7AHιž‰BΡκ( 0•R©ΤSrΆή‘˜i“Χ^“κZŸ«lΧRXSΓ³κ‚ ` +NΩQ¦žΆCσ€ί-q8?νΆρΫo3uQ›Χ!ΒϊΰΙίφsψ]wrΜε³έΔϋν‹νH€H€H€H Vΐ1‘f ²gΩΗjΟτIQEΘΡ‚ €νν·ί.Λ5H°wοήΑ ’| ΤΦκuϊp-Π«W―°i@ΆΡΙ©‹¬57s’ΌJλu‡²‰RkΘ:Λ¬5t(Q`―ίζΦ7δΆi$@ŠΩœ6Ψ»ξ³ΟΉ¬νΠ μ©p†EPοέΪ}9H‹‹Ν‘TΩ§ €Ωχ)'ΜuόΕ<‘ηΧΦ}ρE”j9εή{ύv‘©vvΨ>­‰ΉφΓΓκkλΨ±rH3ί¬•*ZΞ0₯γΥ‡h₯ £E’ύ $4ƒ'‹·Z'‰B―vC Š½Z-E£]P΄4œ΄ρμ\všC€Žλάρu9­ρQρŠ+d½νΒYkNrn ς:8Ό¬YγvˆηΎΆBε8ΐιΒέ³#6    "W•.ΌπΒΏ1±ΏmΫΆ L sΨvΗΫ駟ž΅4‹¨΄TΈCdi!ƒͺ΄Brˆ¨«–ΜΆOƒ#‘EΆzυκ N΅C‡E„ςC‡ θΞί~ύϊl³ΨΓΨηζXΓψνiΘ‘#GΪ7ϋZ‡£ΥnkΒΈ_©¦δ›ύ\\'D$¦2«vC]3dsΥΦο8Ωα;δŸgŸ•e/½©–Ύi“Σ‘Ϋπ#_ωςN =6yαβ-[JQ%'·Žρμ€œΦ—άdΝ¬Sgΰt p©έΏΏ@(TάQ|ί}·¬qψξj€σ/€™»YiΥnΎYVimΚ–μδΩtYΗΊ=&Ή4XΒΝΆŽ'ΩkkjΆxΠ²Ÿΐ1ύά‘Ά6,—ΎX†Ρ± ŠμƒH α μSΉF»9ιBοvΈ‘©vΣMa9ΥpžƒZ€<; ιν)zApβΘ‘ŒΣΥΊk‡τEΈYNG΄Ξΐκl΄¦θWΥΉ7뭌ΎεEc-ΎE£\¬²™pH.Φ ¬ZaDNCέ <ΐ°ό¨ο_Ή(f›Ω,F'9\ΣH€H€H€H ή γη½χή˘δψΡs«*AΨ Ξ»“Α)cΘ~\<Γ9φΞ;οΔγΠs|Μ“UMυ6ΗjfΑŒ3Β–x gpμΦ¬YSV¬X‘q€JWκυ?δν†ZfŸΥΰ­S§ŽuSΐλϊΰέn?ώΈ}SDλ{}ΦΗIΰτ₯‘ (sΦYR’MΩaϋ ΙΓ *q\SJE5Ϋ3UnδgPΙΩ€ίUΦη-«U3j½oΧϊ_¦!³j‘ΦnD-΄γ PV³½½,EŸITΎςJY‡]«‚ ΆLΨ©ϊ›=V‰4`Αj¨qσ/Ώ₯PŒεxzΊμV‡–Ν?t Ž―xΩeRC³«³Ϊ eyŠΦξ…#3ΓτΉΝRU’Ϊ¬Ο‚ͺ^½Φοb,*U2ž©α9ήNύAν»­ώ‰Bς‡βEu&–ΰ³™&ρΎBΗZΌΏƒΩ4~\dBΓv¦F2@Ζ€FΙBEK7ύπCΠt‹8Dίνωϋο v%Ϋ΅ Ϊζ΅aoˆϊm^Η…»?·‚OkΦLvͺ”€Υώύθ#£X­u›Χλίp‘‡φeOF„xoϋσ—-+ ΅xξuZmω+―H)‡:Φ6ΦΧy΄ΰ.œk»-΅?Žͺƒr½FβΒ6\Ϋ­ŸΏ½ ΗiqB¦‘   @Όθ₯‹”,YRΆoߞ1d₯99Φμ™>ΘθΩ³gΖq|‘άΰ΄Ί[^{I1:Q‚<β‚ œvynCΦΪO<ΠcxΐA2ΝIΗ‡²ƒ –¬ΆH3χ²z|μŸb²΄Nϋΰω³I“ 2"»fΞ”YZΛΜΛΰTk§YQȌ²:Φpά'Ÿ4–όκπ9ΫA}Ι©οΚϊέΚ±–[λ°UΞ„.Ξƒς­Υ!8EŸ ΄e»ξž=[°ψ΅"š1Χ,·λ>ϊ¨@Šς€:6­†χ‹i¦„ΣξH3χγoq­u‰L7Zb`΅Δz?³l6—iD@χξݍe·>τ₯‘@2X2p 90gθq’ύΫηPw­¨FΊ„kˆpΙ.+wήyA§2RήmΡ5AlΦλC «A³Œ>όHT«zγΑΕx5‚iώ]w…5e§ΟΡ ΝˆΛŒ­ϋτΣ ΓΚ«Τ@j±bAΫΉH€H 4™z³Œΐ2 @lΘ£Χ——\rIΐ &L˜ ›€―μ2mΫΆ•Š+Λ•δ$GVέΊu};ΥςεΛgΘe’>o;Sϊ%ιTŸ-”sΟξ†Tη*IΚφοίjWT·Σ±Uœμ,I @Κ°Ή>χΣ*\«€u;jptΑͺUΟsL©!ΰΪ)xύ’¬IjΡ’™>$$;N*iόΒρΧAΛD2Žp'€@τVϊ{‘ίγΪ™vnN5dΆT]fU‰Β7Ϋgf¬eYSΛΕ@IDATkž‰H ΞΥ“9·ά"λ΄>šέͺ©S₯ˆƒG‘κΥe—¦~[mί₯‚4rΏΆUkE8Ιωω=>άvΥtŽK4’Ι*)ιΛ-Ώ.e»vυΥΪΪ³ϊJͺ΄Av^τψh!κΉιˆ2F£ΜŒθ€“};Υ8s;mu•.2"Γ,ŽLdnŸ2EJžq†Ϋ‘ϋ AΎV£ήμV•ΕηνHΈN$@! lήΌYξΉηωLλhVΧίτΕ‹ κ:ΡH€b‡δί|σ͌‘N}ϊτΙΨΆP3ψ±X Α’4@Ν°T υΤμ†ο{H-6WΩ2d₯αwΐ\ΰԍ†Υ][ZiζΒ4K]nHP’nYU}HnΪFUM±Λ@ž₯ςqeU9#” ρΰϋ7ވκoY•*UB ΫI€Β PIΟPΆaξώ#[TΡΥτωCI ©£r‘VyΗ|šΕ}ϊwίuˎEθ\‡σj‘ΚΪ %>"΅όεΚIΗΏώ2di&Ψα-[|uYLΏ!ɘE_'πh‡c' Έϋ[K~@YΘZsΝγP)tΚ)hπ`A°3-1 DηΚ 1ΩpV$@IJΰΈΦΫ5k–ΜVηΩ^} f·ά… I=›|‡ΩΊΠλ5Εjλτα\EŸ²3;Τ‘2υβ‹jnYϋςz[£)ν– έ ²†•5βiν»ο4›©ςmT;uΨάμπ]2KYΩ­¬ͺMtƒsΆ‹υΒ0³VX/Άΐjσ?t1C£QΟPΙ„’zSοΗώΎο>I_Ώ> iQuϊY/Ίvr…H ƒΐ"•ίύχ€LJ­gPIeShΙI ΏJΉόφΫotzΒ¨yƒ‡‘}ϋφMNq2k<τξv²~ˆ[έ‘8™‡ιƒ@‡ ηα¦}‘ΧίVǚ=Σ…κ)iJήΏ¨Ι)E»S ™`U₯δ 8,UͺT–Β¬Ž5œΞαϋτzή΄oΎωFŽ?nέd юΑBz―jΟ\C9~? δ D‚―±„kΞn£υΔP?m‡ΧξΡ ,œEν‚•+υΗΰτB['+§2‹Užv›ΚBξž;WDp4Λͺ”ώ^†c•τ„έ±VX³{K©C/†Zn,rν΅F χFuξR)H”_9Όs§δΣz‘7ζσB~΅ζΒ΅ΜΎ‘Ξ§` ΈoπάsΖσΎš})ΘCκΌ‘qvxλ֌ΓA4Iλ…΅V‡O¨Μ©νZlw–F|¦k‘V«AB ΖwX7%μλΪ>(pžFRυڐ©xάREŠΗλμιZΌΈ΄KέΆcͺε=_λC¬ΆDn°‘Q7l˜@ΏF$ΰNΰΥW_•š ¦on»ν6χΈ7a SιάG΄ΰ»ιL4h\―ΕΙΣvΞρ>18Kθ0‰χw1ΌρΓ ‚χ|θΠ‘Nš4IΦk€‘)υh—lίΎ½”Σc΄δ&€ΐ »C ™hŸθΣKUκΜ―ΑA‰!{ΩΡΦ~μŽ5»s³ /ΌΠσ΄₯υΑ΄έ±φήί±ζ‰Ž H G ”Σ`[,™΅BZs­^³Fbϋ5£Χnp„EΫ ±ˆ,xΛδ‚“σKΈ 0˜>ΗId΅¦hΏ‰Τk‰τnr.$@B^ hζJ}uΥr‰\/£ς%Tq‡:œ¬ΆH³š6¨ƒ€¦:@q“ͺ\α8Ω―EP7~ϋ­d•bD!ZDιX‹Πξ[²DιΓ>θ3CzΠ)© C2‘~VG ½"¨P­ZP#m¬–O#3›.Σm5+Žh6ΪDr*ΪΈ±”8ύt)‘²%Ή –½κ€ά5gŽ ΒHC)­]ϋ[©31ήό%ƒεΘPН€(.‹œc8s‡ζ8œ›σοΌ3ΰ°£ZΣr²Φ·Δηͺψ©§JšJΣ +οA}x΄U%8Χ~ψ‘\»6ΰ8¬ΤP9€’­[mη  w¨‘σšη¬X±BvhΰΛ3Ο<#ΟΩ~7έ{ΰ^ ¬&€:SVΗ²Lαˆ€Sυ―–θu³Υ I#ωσηAΈI%ΞΒqͺ‘ƒuΆ Β N=6”)SFΊͺδώOš‘b~Ȟ―¬nQ3pβΔ‰ζ.γοE]dd£ltX9UοV―^°Ž5  xX©†VƒZS•uΦώρ52˜•–ˆο¬Ώ9Ρ±ζ[‘ $9TTo e^Ρ:Θ:Mλ\ύ©|ΗlΊύHŸ₯ςŠ^§Z;Mαίͺ‹Υ±†γ–jΆ–όκt;ϋ€l™΅Ώ2]ΊH^užY3Ο°πΆm²ύδΝY©ΩOU‚²‰J^ΝC¦™ΕA ιݚže•ξχ²ζκd*’iϋΙdΘθ«.«l’α0¨©Ž°ύϊwŐ!‡Αα η¨]*2 ‘m’mύؚp•H€H Hi=­4ζƒxd4ή‘Ώ¬)7“@h£lΘNC–šiΘRƒc ²VC†Ϋ%Άΰ1λ~ΎNpΊΪ­]»vφMžλŽΤ λhu¬Α9ό΅'β3μ$y­J§ω1ΜύX N:«Μ€uŸΧλ‡~X>ύτΣ€fίip%jΠΡH€‡ΐA Ψ8ztΐ„P2$/U˜p%> @= -0ΘƒGΛ’ΧS΄FΔ~H€H †δudώœ₯)ρ^N5sΨΠTnώφΫ’»`As“οΏ•τΒ₯£ΦY+¨…«Λwž€„©Ηœ[kΓ °kf­†Fι7WιHd₯…k‘ή·°:gc HBx˜hΪΗ,sT™F$;Μ¬RsDΘψιίΏΏΨeομνΜφό›|ͺ;Θζ/Υ:ΫαΨ­‹ GU€VPƒ0/VΕ«MΡkyό֌?ήΊYQΥΏφpvΠϋY«α2±Γ5•Vυ «]wέu‚Ίt4 Δ!pτΐYu²ή΄9«2šέSLK‚ΠH€Bΰ―ah6άc!ΠQπξΩ³ΗΨy Δ8.²HΖΚΨ ι[„Ψ€Ϊω;τ&iΦ%ΓrT%"Q? …OQ/«²:£ΠΦΙʝ}Άα Ϋ¦²»ηΞQ'\X„σΟΝͺkΦRε^½d―ή(ξΣεπφν’§hQ#.­ys·C}_K•Σ9€ΗmΗν:ώCͺυH#σiM€B§œ"…O.ΕΥ‘§’_«―YX"΅*zƒ‰%λξP‹,œγ½ΪBN3ŸΕ4ύ\΄;Vvι{Ύι‡d‹ZGMΎτΝ›εΈfNδU§+κε!KυאݘωΝj7ή(X"΅hΌ§μέΡ0jiύB,4 ˆΘi!£’W5kΦ”½~OEk\μ‡HΰUZM%Τ­υ€>ωδ“ω+T0κΌΧΈγŽμ;)ΟDqN€Ž΅89|      HV‘kΩ-™¬όγqήψΜ,X°@Ίwοξ9|HrB6ς‰}τ‘”ΦΪD0Τξϋξ»ο€P‘Bž}x5€σΚΙnΊι&§Νam+WœόυΧ_2|ψp)S¦Œοc!#ωΙ'ŸΘ‡~(EŠρ}’ Δ6S΅~b½₯ζ=χHέ'ž–_}%]/–nšΩZR₯ri$@ώ P ?+Ά$      ˆ"›αΝνI ^”Qω{,4H6ψύƒα·ΊK—.Q™>kQΑΘNH€H€H€H€H€H€H€H€β‰ΐ9ηœ#X2k¨ΉvύυΧgφpγΈεΛ—GX΄­@rΑK΄ϋf$@$@$«Ž;fΘΖc|¨·Šυh₯ £A‘}     $ Ν›7Λν·ί.όσOΜ™% ΘΓ† θς’‘:λ:δ   @Τ 0c-κH³ΓργΗΛ‘#GŒΙAΦ Ez“gE$@$@$@$Θ>ώψcΉνΆΫrap°}σΝ7‰<]΍H€H ¬[·NFp†«ΊJΆq…H€H€H ΆΠ±[οG̎¦gϞ²mΫ6c|[·n•R₯JΕμX90    Θ*¨?³~£ϋQ£FΙδΙ“₯ ‹½gnφK$@ M`ψπαA’TwάqGBΟ™“#  D @)ΘDx9     l!pꩧJ―^½2ΞuύχgΌζ   πKΰΐ2bĈ€ζ]»v•Ζl㠐  @μ c-φήŽˆH€H€H€H  ,Y²DώψγcYΏ~}6ž™§ŠWO=υ”δΛ—ΟώΤ©Sε«―ΎŠΧ©ΔύΈ«U«&]Ίt1–Ϊ΅kΗύ|8 Δ$pθΠ‘€‰aέͺ dξ|葇̗όK$@$@$Γ(Γo‡F$@$@$@$υ† ’1>lΨ0£~V֟•gˆgU«V•;οΌSlL£_Ώ~£GΦ!΁7υK/,4 ˆeυκΥ“₯KK­Z΅A<3fΜd¬YνŒ3ΐN:Y7ρ5   @Œ`ΖZŒΎ1     @μθίΏΏ”(QΒΰςεΛurh$@$@$ΰDΰΘ‘#2}ϊtωδ“OdόψρAN΅””A H€H€H >Π±οGI$@$@$@$@$CŠ/. ΘΡΐeϞ=λ|A$@$@~€¦¦Κk―½&-Z΄πӜmH€H€H€Β €ΰS:ΎsηΞaιή”Rξ|Έ—H€H€H€H€H€ τιΣG^ύuΩΆm›τνΫ—RŽ”Έ‘H€H  *Θ_|!mΪ΄ Υ„ΫI€H€H€" +W.ωύχί#θΑωP:֜Ήp+      ΈΘ›7―|ώωηR­Z5£vŽkcξ$ HZ#GŽ”©S§ΚΝ;%--M4h`,¨Ή–'Ν%νƒ' ˆ[όυŽΫ·.{ήΎ}{Ω½{·qRΘΠH€H€H€H€H€„]ό  x8묳 H€H€H 1Π±–οc–Οβλ―ΏΞςsπ$@$@$@$@$@$@$@$@$@$@$@$ΛθX‹εw‡c# Π«W/)P @œ™§$ ₯K—        @Œ c-Fί‹rŠΐ¬Y³rκΤϊhΠ΄Ϊ΅kΗοΘ *ά―Š-―CηΈI€H€| cΝ'(6#     p T―^]V­Z%εΚ• χPΆ'ƒδi)QΛC’ΨΆm›γTJ•*%νΫ·wάǍ$@$@$@$YU7₯γs办€#k™}Gx      ψ @§šHlB$@$@$@$@$@Q&g€γ£mΡsΡE{dμH€H€H€H€H€H€H€H€H€H€H€H€bˆ3Φbθ͈呴iΣFvοήm 1555–‡Κ±‘           @– c-K°&^§(ΈN#           d&@)Θd~χ9w            ί˜±ζ’    $"Ϊ΅kKǎ©U¬X1§Θ9Ε(γǏΛΝ;₯dΙ’1:ΒΨΦΏώ++V¬0ZΉre©Y³fμš#$    ˆkΜX‹λ·ƒ'   ˆ”ΐ½χή+cǎ5–σΟ??ξx< ψ"πσΟ?K“&Mδκ«―φ՞œ |φΩg©S'c>|Έs#n%     (`ΖZa²+     π"€ +8q;& ,?όS:wξμuχ“      Δf¬Εΐ›ΐ!     $Θ^ύυώο+'NœΘXη       θ7nœ`?~|t:Τ^θX‹JvD$@$@$@$@$@ώ 8P ,h4ž3gŽ|όρΗώd+      _ bJǟyζ™ΎŽρӈŽ5?”Ψ†H€H€H€H€H€’H B… ‚ϊ~¦ 0@ΣΣΝUώ%     ˆQt¬ΕθkÚ2eŠ‘*‰tΙ#GŽΔΪπ8    ˆ;<π€”)SΖχΪ΅kεΥW_»9pΐ$@$@$@$@$@$lθXKΆw<“σνΡ£‡tμΨΡXvοޝΙ^x     €I H‘"ςψ㏛«ςΜ3ΟΘφνΫ3Φω‚H€H€H€H€H€H φΠ±{ο GD$@$@$@$@$$nΎωf©S§Ž1Ϋ]»vΙ¨Q£’dζœ&      Δ'<ρ9lŽšH€H€H€H€’C`Ω²e²aΓ£³Ϊ΅kKωςε£Σ1{!ςδΙ#Ο>ϋ¬‘­6xπ`iם£ΨΔ$PΉreιΠ‘ƒ±Z³fMs3’     @– c-ΛΠ²c    x πβ‹/ʈ#Œ‘6Ln»νΆx6ǘ@.ΌπBΑB ŸΐW\!Xh$@$@$@$@$@$](™]€y            Έ&ΐŒ΅Έ~ϋ8x H GŽ‘ΤΤԘšL,Ž)¦q0$@$@$@$@$@$@$@$@$ΓRRR€}ϋφΖsεŠ^žYτzŠax  @lX³fτμΩSζΜ™“γ=xπ  2Dyδ‘ @$@$@$@$@$@$@$@$@$9p¦?ήXƎ›ΉNŽ’cΝ 7‘  d/Z΅jI‘B…€yσζF™œp°™΅5jΘύχί/½{χΞ^<         @Μ c-ζί’Ψ`λΦ­”I€Mƚ\[lβ(H€"%ππΓKξάΉε»οΎΛV›Υ‘vΟ=χΘ¦M›δͺ«’SN9%)ρx         #@ΗZ‚½‘Y5Ρ£Gg€L+V,«NΓ~I€’˜Yph™–Υ6'‡Ξ ηœ|4  ˆ“'O–}ϋφΕΒP8      %@Η?$@$@1CΐΜZ³(ΪΆP5σœΜV3Iπ/  @NψηŸδβ‹/–ΆmΫΚ /Ό“CαΉI€H€H€H€H€H€,θX³ΐΰK  œ%`ΟZ³Ž&R›—C ηbΆš•8_“@ςΐwO»vνŒ₯B… Ι3qΞ4¦ L:UFeŒρΕ_”7ΖτxsjpλΦ­“‰'ΛͺU«rj)­[·vk"<ςˆL™2Ε΅Νΐ₯M›6m4i$Χ6?ώΈ‘αΦθ‰'ž0ήάΪ<ϊ裑C·&2hΠ 7nœkφvκΤΙ΅ΝSO=%cƌqmΣΏιάΉ³k›gžyFώψγΧ6=τœuΦYmž{ξ9ων·ί\Ϋΰ3ί­[7Χ6ε—_\ΫΰΰΩgŸνΪΚ ?ύτ“k|'œ{ξΉm^~ωeωα‡\Ϋΰ; ΧmnφΚ+―ΘθѣݚΎ3ρ,ΙΝ^{ν5ωφΫoݚȝwήi\·Ή5ΒoΑ7ί|γΦDξΈγCή­~›πŒΛΝnΏύvΑu››α·ΧΜnvλ­· Ϋάμ­·ή’Ο>ϋΜ­‰ά|σΝ‚λ67Γ΅Δ'Ÿ|βΦDzχξ-Έns3ά+αΊΝΝnΈαΑu[΄ŒŽ΅h‘d?$@$@Q#€‹[ό ;vΜ³O<  eΈ€ρkΜVσKŠνH€H€²‹¬ΰ‘Θ―Ώώjό&β!‰Χ‡μ[,žgνΪ΅‚Εj~%ƏοιΘ:zτ¨΅[ΗΧΘ|υrd9rΔρXλFH[ξΨ±ΓΊ)θ΅ί~ΆmΫt¬uΓαΓ‡­«Ž―ρΤΛuθΠ!Ηc­'Ož,6l°n zν·H€ΊYzzΊΫnc"{9ŽότσΧ_ΙΚ•+]Ο/C?Λ—/wmζεΖΑxΘξεπρΣΟ΄iΣdΡ’Eγρr$γ`τ³`ΑΧ~όμ„ΒΛQ³oί>ΟЏ—£&TО΅σ3fx:XφμΩc=Δρ5œ4`δf^ί+8ύΰ3δf»vνrΫmμƒσΙΛ1β§9ΰ»ΥΝΌφ8ύŒ;Φ­ΟοKŒχάΛ1βυ½‹~πτκΗ+€ΐo?^ίίθgώόωžγ‰V?p xœ²^|ΰ”φ2|gxυ'Ή—ωιώ½ Nk―ρΰΑΏ—α;Υ«Gx™Ÿ~Ήζ―n §ΎΧxπœΔΛ–,Yβ9//η Ξ᧟K/½Τk8Ζo ΧΌ.Ήδ’¨τƒ`(/Cέd―ρψQMAPˆW?^ΞKŒΥO?έ»wχš–qΝβ53@Π­3Νxυγε΄FΈσκ§cǎnC {₯ ΓFΖH€H€²š€™΅–Υη1ϋgΆšI‚I€H€b"θ‘½Cφ•Wδu¬ŸγΙ9)))9wrž™b”_ΔθΓaε(ώΏΘQό<9 @œH9‘§cη°I€H€˜’h yβ'Δ€u?ύœvΪiž’­Θ4σΚlσΣ2ΝΌ2ΫπϋQ¬X±€yΨWότΣΌysO YdšyeΆωι™f^™m,^ΌΈ}*λsηΞυΜlC]S/IΫhυƒΜ7―Μ6Θd–,Y2`φd¬ye€5nάΨSb7Zύ cΝKͺ2’₯K—ΆO%`™f^²± 6τ”όυΣδQΛ–-p~ϋ 2ΦΌ2·ύτƒL3―2όφ’ φΣž₯”/_ή>•€uΘΠzIϋι™f^ιΕυ’DφΣO:u€bŊσ°― kΫKZΩO?Θ4σʐ‡l°—D3žmyI4#ΈΌrεΚφ©¬G«\7ΫN€+Uφ’ŒF¦™Wζ͚5₯jΥͺφξΦ£Υ2ΦV―^Π·}₯FžΦφcάΦιXs£Γ}pqnή”α¦ΐλf0γ@Ύ  \wέuβUk-‚ξCρ° ”Έ‘‘  @,ΐM=nάρπ\PŸΓλaI,Ξƒc" Τ™q’ΰςr¬‘    ΔχPΉX)Η’£PŒ‚Aτ‰WAι,ON$0Β©΅–ΩI³ΆZfΙρ8  μ"€HΨAƒ™7ήx£gVv‹η!     d$@ΗZ2Ύλœ3  Δ ³ΦZVe­±ΆZœ|8L  ΉοΎϋHH€H€H€H€H€H ό― v „C   'ΘZσͺβtœŸmΜVσC‰mH ρ @g5„°xΥ=H|œ!       €:Φάθp  @Ž0³Φ’=f«E›(ϋ#ψ%0xπ`iם±|ϋν·ρ;ŽœH€H€H€H€H€H€H Λ Π±–εˆy  H dEΦ³Υ"}Wx<         $:Φ’ο=ηŒI€H ξD;kΩjqχΰ€I€H€H€H€H€H€H€H€H &Π±oA$@$ΰE šYkΜVσ’Νύ$@$@$@$@$@$@$@$@$@NθXs’Βm$@$@1G ZYkΜV‹Ή·–" ˆ€ΐΊuλδΩgŸ J$@$@$@$@$@$<α4fΫδ%ΠͺU+Ω΅k— 555yApζ$@9JYkό±;v,Σγ`ΆZ¦Ρρ@  #π裏ΚΰΑƒεΰΑƒ°aC9οΌσbl„      $f¬%ή{š%3ϊα‡d€IΖR¬X±,9;% /‘f­1[Ν‹0χ“  ā;wN5ŒωΑŒ(π$žζΝ±’     @N`ΖZNηΉI€H€Β&IΦ³ΥΒΖΝH€H€b˜2Φ>ψΰΩ³g,Z΄HFŽ)7ί|s 8λ†vώωηKzzΊλ F- pmΣ£G9pΰ€k›οΎϋN ,θΪζ’‹.’}ϋφΉΆ5j”.\Ψ΅Ν%—\bΌΏnΎώϊk)Z΄¨[ΉτK3HB5όςΛ/%---Τncϋε—_.;vμpmσωηŸK‰%\Ϋ\yε•²mΫ6Χ6Ÿ~ϊ©”*UΚ΅M―^½dλΦ­m vP¦LΧ6W_}΅lήΌΩ΅Ν‡~(εΚ•smsν΅ΧΚƍ]ΫΌώϋR‘BΧ6Χ_½¬_Ώή΅Ν»οΎ+•*Urmsγ7ΚΏώλΪζwή‘*UͺΈΆΉι¦›d͚5mήzλ-©V­šk›[nΉE¦M›ζ؁Όgu–Œ1BjΤ¨αΨΖάxΫm·ΙŠ+ΜUΗΏΓ† “Z΅j9ξ37ώη?‘eΛ–™«Žίxγ ApŸ›υιΣG–.]κΦD^ύu©S§Žk›»ξΊK/^μΪζΥW_•zυκΉΆιΫ·―,\ΈΠ΅Ν!C€Aƒmξ½χ^ωϋοΏ]ΫΌτK¨Q#Χ6χίΏΜ›7Ο΅ ²―›4iβΪ恐9sζΈΆyώωη₯Y³fmˆ2{φlΧ6X>υΤS]ΫτλΧOfΜιΪζ駟–-ZΈΆ0`€LŸ>έ΅Ν“O>)P’r³GyD¦NκΦD(­[·vmσΨcΙ”)S\Ϋ<ώψγ¦MΧ6O<ρ„œοΦΧ1νΪ΅sk"ƒ ’ &ΈΆΑ3‚:ΈΆyꩧdάΈqmπ^tμΨΡ΅Ν3Ο<#cƌqmƒΟΖ™gžιΪζΉηž“?ώψΓ΅ >«]ΊtqmσΒ /ΘoΏύζΪζΏύ―tνΪΥ΅Ν‹/Ύ(Ώός‹k›ϋξ»OΊwοξΪζε—_–Ÿ~ϊΙ΅Ν=χά#ηœsŽk›W^yEδαfwί}·§ZΎ3Ώώ{·nδΞ;ο” .Έΐ΅ ΎΓqύηfwάq‡\xα…nMΏ)Έώs³ΫoΏ].Ύψb·&2|ψpΑυŸ›έzλ­³gO·&ςζ›o ά χ—]v™[yϋν·ΧnΦ»woΉβŠ+ܚχ2Έώs³nΈApύηfο½χž‘vεΤ&W\ςλ―Ώ:ν {ka#γ$@$@9IΐΜZΓƒΔpŒΩjαΠb[H.5kΦΜx8PΎ|ωδš/ Ό N‘3fΈ6Ϋ½{·λ~μD?‘ζΑ^ί+h‡±x9Fόφƒcn†Μi/ƒσ ίAnζυ=‡cg͚ειΠπΣήs/‡ΖφνΫέ†k샓ϫ8"ΌlξάΉžύΐYκe~ϊΑΛπΛk^p xΩόωσ=ϋ“άΛότγ'¨ί=^σΒƒ/σΣ‚#ΌlΑ‚žγA°†—Α‰ξ5/{™Ÿ~Όœ"8½ΌΖγε€A?2πκΑG^ΆdΙΟ~Όœa8~K½ΖγεœσΫΟΉηžλ5-ωηŸ<ΗγεΕIp­α5//§¬ί~:wξμ9―εΛ—{Ž§S§Nžύ ψ&ΤΌΰX‹–E―§hˆύ  €D€ΑQŽ1[-ZlKΙEQœ¦δ5²Uh$Oυ[±bEcΘΘ”AΔ1H <)))αΐΦ$@$@$@$&^o„ ,Ζ›§œP‹ρ1rx$@$@$DΰΊλ3δ―‚v8l€O^2.‡r   Δ<HΒAς iAD{–-[6ζΗΝ"cνψργ]"ΒΥ+0k^™m‰Κ“Η]ό™f^™m~ϊA–‡Wfδ―RSS]ηφνΫKήΌy]ϋA¦™Wf›Ÿ~&Nœ(^™mΛ—/ŸλxότΣΆm[ɟ?Ώk?πΚlσΣ2ΦΌ2Ϋ Ÿζ%IŠl#―Œ΄3Ξ8ΓS’ΤO?…σΚrEƚWFšŸ~iφσΟ?rtφ7\ /wϊι§{J€’―Μ6Θζ)RΔ~š€ud‡yeΆωι™f^™m-[Άτ”lE¦™Wfd½κήϋιη΄ΣNσ”~E¦™Wf›Ÿ~iζ•ΩΩΕβΕ‹Ό?φ?ύ4oήάSŠk^™m~ϊAƚWfd)K–,iŸJΐΊŸ~š6mκ)‹Œ5/‰]Θm"Ϋέ͐±ζ%±λ§dšmΩ²ΕνTΈqcO©^?ύ@ŽΤλΊ™f^’Ώ~ϊA¦™WζvΓ† =₯ƒύτΉV/5 dšyIΧ―_ίS‚™f^™δ~ϊΑs/)cΘٚAa‘> ~ϊ©[·§$22ΦΦ­[κ4Ζv?ύ cΝ+CrΏ•+Wv=—Ÿ~jΧν)ьŒ΅΅kΧΊžΛO?ΘXσΚΨΗs΄ͺU«Ίž χ«W―vmyf/Ιh?ύ@m¦zυκηBΖΪͺU«2ΪΰΪήΜΰCƚס~Ɓ/θXσΔέ$@$@±IΈ σσƒ Τ³ ‘  @"€C έL‰5H+‘ΎP2ޘP·Κλ†=˜pŽ$ΛΎωζq’οB]B죑   @4 ΰ^Ι¬ηΗ³’a”‚ŒEφA$@$νΜZk^'fm5/BάO$@$οpƒψόσΟΣ@Δ?"o“Ε>ωδ£F"²]†š,Σ曐  Δ&?΅ΦX[-6ί;ŽŠH€H Ί oςΖo€OξΎϋξθvΞήH€H€H€H€H€H€2Π±–‚/ά@£ϊμXΌjΈυΓ}$@$M^YkˆJσF$@$@Ι@ΰφΫoχ¬Q“ 8G ¬%‹ΟbqLYϋ.°w   œ$@ΗZN£sŸsΞ9+^ΕkγhZ* @psœuιEΰ|£‘     D‡ΐ=χά##GŽŒ‰ [χμΩSΆnݝɱ   πA 6lB$@$@1KŽ38ϊ駀1"[ν©§ž ΨΖ     Θ\sΝ5ͺU+yςΙ' uˆk―½Vςδ‰μρ–?”?›65V‘G©χΔAƒά8z΄,~μ1cϋ’δƒM›dڞ=rU‹RΎ|ωΰφίŸΡ>cgJŠ”Σ{‡ϊƒel2_lϋ‰Ε>jfό-{φَν7ς‹,zδ‘Œvζ‹²έΊI}ec·ΝΏύ&‹Τ4ΚvνκΨ~Λ8Ά/έΉ³4pΈΟC§ώKŸy¦cϋ­cΗΚ’μΓ”R;Jƒ§ŸΪΎuόxYΤΏΏˆ2΄Z©φνΫo›8ρν­υuΙΆm₯Α3ΟΨΆŠl›4ΙΉ½ΦΠtjΏ]•φλgτ“bS‰Φ­Ϋο˜:5£½υδΕυ³άπΩg­›ŒΧ;¦O—…=΄½xΛ–ŽνwΜ) |ΠhoOΪi§9·Ÿ5Λ±΄ζΝ₯αsϝwל9ύ[w¦5kζΨήΪ†―I€H€’O ²+Ÿθ‡=’  @Ψ† "Ώώϊ«;v,γΨ«―ΎZNΣ›  xXΉr₯l‡s°š5kJΩ²e½α~  HZ-Υ±`Άυξέ;*Ά£κ Ϋ3ožΑŽ';Όs§Μœ;WΎΠ³N6HΥΏWU­κΤ\Žhϋέꌰ[Ρ μ›ŒυΓ;vΘ.uvΨ­pέΊφMί^)v+B1γϊΧ ;»ͺQΓΎΙXΗxvN›΄―`¨ω’½:μV R%ϋ&cύοψλ― }ωœ”h„ροPg–ές•.mίd¬£νκ,³[ή’%ν›ŒuτΏ]qvKMK³o2֍ώ'LΪ—»pα mΨ€φΫƍ Ϊ—»@ mΨ€ρlSη£έrεΛgίd¬ήΎ]Ά΄/%wξ mΨ€ώ·ͺσΤ―aό[=¨ω Λ=pΠNn  Θ2”‚Μ2΄μ˜H€H »Ψk­εΦ›7‰ΘμΟC$žώω ΙλQ£FΕΗ 9J  ΘAΜΓV­Z%p°Υ];Λ$"!ωxΓΛ/ ς¬―Ί^Ί`A #ρμΔ‰ψžSœDœ?Ύ?<=  Δ>f¬Εώ{Δ’@–ΈρΖeύϊυY7;%œ °ώŒΣ–)SFϊτι“±Ξ$R)šN:%ΒT8   8'`ΝZ3§b:Ψ’) ‡Ϊγ?$ϋŽs"[ν"]¬²{ΨN#   ¬&@ΗZVf$£&©$Γ²eΗήyΐWQ-|H…!H ι½J―"Ŏ¨`E±aΑ† XPĂ周όŸ`Α'vμ<Š Hο½χJB™•{ίΆ›Ϋϋo>Ÿεξ9{ʜο.ΙΝΞ™™ͺΤχdff’%ƒ ₯ε`- >"ŸŸOγǏ§σηΟΣ‹&Ή€|€†Χ¦IOO§φœOG€FΚ'ώπ ρZΣη9–™=a`+Ι fYx«%[ Αό©Κ”Λ€ώAyΫ 4€„%—BΛfœvνΪydι0¬y#ΐ"°~%ΜιΑƒ©tι$ϊŒŒŒΐRMmn»ν6’ ΰ{β΅Φ·o_š9s¦ιδf6Σ†ͺJΩϊω/Ξ#΅θΣOU΅ΖS‹·šρ j@ Όΐc3Όξ7V  ΰ<bΞEΩ‘C₯cDDIΩÚ'(b 0bD.1¬ΠσΟ?OΣ¦M 0-‘€@00Ά k–u© lbˆ3“¨V­hb™24‡#«pΞ³&šΊΑR#”Ί˜dsΏ΅ΚΧ\CέW©3²ύ3D΄­φW]EέV$ε֊©PA3·₯P©woκΆ|Ή₯hύ΄ΥΎβ•WRΧeΛ¬ν,'6ΫχμI]—.ύ§™J'[νSzτ .‹[†΅~Ϊlί­uY΄ΘΪΞrb«}….]¨‹άŸKbΙAf³}§NΤyΑKsλ§­φΙό³σόωΦv–›νΩ[ΉΣΌy–f|2'[νΛ·mKώόS۞K6Ϋ³αΈΣ8ά>‰ŸαŽsηjΫ— OΉ–-©γμΩΪφ%θSysκ0k–±½ηΩΠ  %PŠy6Tςΐ`6Θ—ίS§N)W%ΤCRR’–¨’X‘ ƒεnAO ϊΟώ― /=C† ‘Ι“'+£Oœ8‘€ P!πχίS~*"»ΪWρKζ-Z„Κς° p€μΧ<©¦xΰξΪ΅‹<9,ΖAβ‘υ™'=Φ"B–δς%y1ο|’F5/Ɛ     ΰ;w¦λ―Ώ^YφT>ά ³`Hp& ^kΎ”xF5_Η\   ˜κ Υ¨WS¦LQB…λϊ±nπ7±cΗ<}X+…ωA‚šΐ›oΎIΏύφΡΦJ6ΝΩ ΗΤ …ς ~!`/ך'•o΅gžyΖ“Cb,:Ή[·ͺ»ο6θXΏ>΅²“ŸΠΠ   .€aΝ%lθ‘K ΗA—0‘ψμ³Οό3qΟZ«V-jםB 555ŒI`ι‘J ^½ztύχ“„:‘—½97„B€€€'8’kΝσΐ[Ν1F°(ΚΛ£“&ωϊ.œ?Υ"Ϋ@IDATμKƒώ  4π—TΠά*(     ΰ #FŒ %K–(GΏ~ύΌ1ΖΏxι₯—(!!AΡcΓ† τ)v΄ϋύž@%―5o ήjή€‹±A@@œ!Ú3΄Π@@@@@‚€xcŠ922Rρ^C(Θ Ό‰Pœ€·s­Α[-ΐ¨  aD‘ Γθfc©     αK`Ψ°aΤΏjΨ°aΘ@ΘΜΜ€}ϋφ)λIKK£ŒŒŒYΑFΐβ΅&y=-πVσ4QŒ  αA T©RΦΤž …΅πx~°J0'RF5Ή_|ρ΅oί^9>ψΰƒ0ΏΓX>ψŸ€·ΌΦΰ­ζ{ @@@  ˆ1Ν’ϊaΡ’E[<Φ<†€€€€€€€@ψπ†ΧΌΥΒχyΒΚΝ $Φ―O]9?°^"γγυU(ƒ€x‰ k^jΓZ΅ŠŠŠŠ”e΅jՊ’’πθ„Ϊ=Ζz@@@@@@@ΐ]β΅ζΙpπVsχŽ ¨ˆbZr»v‘Ά,¬@‚ŠBAΥνςŸ²}ϊτQb‘Άγ_ά999ώS3ƒ€€€€€€,‹Χš'„·š'(b O€aΝΣD1€€€€€€€„1OεZƒ·Z?DX:€€0Φψζ@5οΨ»w/-[ΆL9Ž;ζύ 1 KΨχT *!Ox­Α[-oρ€@ΰ―5W kπV œ{M@Q^X²Δ XρΉs†:T€€-_ΎάŠΑς·ƒ΅ΒΕ„‚tΊ€€€€€@°(_Ύ<=όσΦeΌςΚ+tϊτik9ΠOξΈγkŽΔ`2:Wθž"`ρZsf"Χ#Ζ4      ¨J•*EοΌσuλΦMQρ‹/Ύ Ι·–‘‘¨*C/ "`ρZ›9s¦M­α­f .€€†@bύϊΤeΡ"M’ u¨πΦΌΓ£‚€€•ΐΦ1cθΨάΉΦ²³'Qρρ[Ή2•–£JJι…Κ5kζμ0h  `ƒ@5¨M›6ΚΥJ•*Ωh…j}]»v₯kΉ†N:₯Ω`T ύ{Ž‚€/ ˆΧZI†5x«ωςn``& ο*tθΜK€ξ  τ`X ϊ[ˆ€:ά­[){ΑͺW«₯έx#Υ|ΰJ¨SΗ£cc07Ο>ϋ,Ι š>}:%`Η;/(Ιk ήj^Ž!A@@ΌF9ΦΌ†ƒ€€χœέ½›vrΈ¦?š4‘-Όσ³ψά9οM†‘ΓžΐΉ¬,:³s§ζ(.({. ‘HF΅PΌ«X[ΉΦΰ­8χš€€€Ψ'5ϋŒΠ@–ΐ6¨m=šόη?ΤξΗ©l£F«+ ^ΞtχϋοkΠeαBͺΠ±£¦π5#GŽΠ”i+sΨμjΥͺωZΜ ΰ3―5x«9MA@@œ" Ή€-©""<ηgÚS·A@ΐ3bSS)‘^=»ƒ9C…ǏӹcΗHŒhΆ$½‰^quž7$‘1@@@ΐ΄iΣhĈΚRŸ~ϊizϋν·ΓaΩX#5}5x«υν„ς   ΠĘΆ|ωrλΓšΗ‘b@°O RοήΤϊ³Ομ7T΅8΅~=ωυW:ψΝ7tzέ:Υ•NΟ=ϊqmώ|J¨]Ϋp    ΑGΰ—_~‘I“&ŸβΠJ P±bE:ΞεeΧƍικ«―.‘5.@p¨Η›hǍη5₯s·m£Υχήk?‘ηm9eŠ‘  ž'Γšη™†δˆkΧ₯σηΟ+kkΡ’EEαΡ ΙE4rM›’uGŽ€}όeyrξ+΅>LΛ  ξΌ£Td€ϊΞA@@@‚ΐΎ}ϋθχίBΝ‘2Ψ'pαΒϊσΟ?ν7D "ΩΩΩ^ΥV"ۜX΄Θ0Gq~Ύ‘ώ#°μζ›•θCj κ=<₯φκ₯Β9€@π\PΙ ΅#pε•W’ΔB—#''Η±Nh ΰΈΖ}χQχ•+©tΥͺ†9N­^M»tω° P   NΨΙ‘§!    φ œ\ΆŒ²,Πi `X ϋˆU€„!ΈκΥ©ΣάΉS‘‚aυ›_|‘ςΩ{    ξ0mW]u5lؐ`\s—¦σύ/^Όθ|'τπΦΌ†ƒ€€χ H υ†―½f˜¨8/φNžl¨χw…„¦Έp)¬¬ΏuρΕόαΆ^_0Ε ΰ fm%{Λ!ω^ Z=φ͜9S ?’CQC|K@’@@@@@@‡eΞ½€&  ΰ Ή{Β:½aƒ¦ή©S©ώ¨Q6s­dfRξΦ­š>ΙνΫSdιΦΊσΉΉ΄ΗΙαπ’§Φ¬‘‹EEΤδέw)΅wok['…6φ(ηΙόωgΚY΅Š$δA']Ύ<•α0–{τ ΤΎ}©RϞ6υ΄5~ώΑƒt±ΈΨz9‚υ.šj-ΛIήή½΄ογιτζΝtvΟΕ‹/&9™Κ€§SJχξ”ήΏ?%Φ――ιγjΑΫλ•{%χL-I-[RtΉrκ*»η§·l‘sGŽhΪ%wθ@‘±±Φ:ύ³Qΐ¬υ"ΟΔ…ΒBk΅άΣ€ζΝ­eœ€@0xγ7hς₯Ν'N€!C†“ϊΠΌNΰΝ7ί€ΆmΫ’xN}χέw΄dΙjΟί      Ž`X Η»Ž5ƒ„R‘‘tΩγΣ6°©EŒ!GgΝ’ΚΊΙLŽπΞσ5χή«ΉΤ‹Oρ5j(uwΝPώώύš6ημxsΠφ·ή’όψμYM_KαόΙ“$‡wO uλRƒΡ£)}ΐrtWφ<~‘§6ψTδ\fΟV¦8Γ9`6Ώπβ—€2ΎΙΕΒcΗθ ©Žρmα6Uϊυ£&γΖYΧmΡΡΡO_­w;σά駡Ί,\H:vΤΤΩ+l;–|ώΉ¦YoφΦ‰ΛΘ°Φων7ZsύΦ²ΩΙzφ^P‹*;#ρΌ ΞA@ d΄iΣ†ποθ―ΎϊJYΣπαΓi!‚ψ†€­PΧ\s 6ψζ`0%°•|ϊι§M―y³R"ΧtώϋoΓQ †:T€€x‡ kήαŠQA@ΐ§nΌ‘Φ>ό0]Ty‰Η8›-ΓZI ncγΛfB=ζ,K½–Ξ²§˜3rfϋvZqλ­΄Ÿ >­Ώό’b’’œιi+r+οΌ“ŠNŸΦΤΫ*dΘAGϋ_jΓσ¦έpƒ­f¦υ°^SΕP   &πϊλ―Σ?ό@…ό]cΡ’E4ƒφγΝ)ώ–Κ•+SλΦ­5ͺU«ζouΌ2Ώ­MGΥ9ίξΥW_ν•91(€€€}LςΫοε~ 1 ₯tκδώ@@Β„€€}‘οΥ­Z΅ςΘͺ‘cΝ#1€ψ—@ ‡α“°Šz9Ήl™ΎΚny7‡AsΕ¨vbωrZΠ΅«ΣF5΅BGΩ‹nοŠΟcΟ9W$σ—_h)5ͺYζΈΐΉί–έtνϋδK•έΟ@X―]%Ρ@@ͺΒ‘sΙeΆˆC1 ZT©W/ͺvΫmTΆQ#*]₯ IήΆ‚C‡(kΑ%΄‘ή»-Γ8.½ώzκΊx19Κ"oΗZ1p &τcbƒT•=αβkΧVς―I^Ά3άngŸQώΪ΅ρ/Ϊ5>He7¦ςόΛΆ$ „υ–€Ÿ»Χ$4gΖέw[‡m.ηfSK*‡­TΙZ%¬!  ‘Mΰ‘ό oBΙα<ͺΫΩΫόώοθaφ˜‡€€€€€€@8pμMk8ΑZM 4kΦLωZ.F9ψ‚ήt T‚x@Ω¦M c‹'Vήξέ”Θ†GdΓ°aD.X›Κ˜©½{“|–γ#"&†btα..rϋUƒŒj±lDk=mUΊβ λxr"Ζ5Ρ§"ηδͺ?jm3†Άr~5υΌ’{m5ηψΊ|ϊtMί’ j]D™2Τς㏩*ΪJ±qO/2οžΙ“iύOh qϟ§lμ±nEΕΕι»)ε@Y―©rͺLι…δ°Θ:Ξα§7¬Υ{ξ9§σ»YΖΓ'€€@pHNN¦ηψηˆ#”όΜα—aX Ξ{ ­A@@@@\'ÚλμΒͺη\ΞΣle64UπόΙ“¦υf•ωϋχSΝ1‡λ N _{Ν·›¨²ΩϋL-‘σ½ΣPY;^LΫΈΉβ8'ΚκΑƒΥCΠ‘―Ύ’Φ!©eKM½½BDl,΅ϋρGJeO9[M—±g^ιΤTZΞmΔήjΉύΌΏ–*ά•εš|ΪzΥΊα@@ΌM@Β-ώΒ‘—οη 0wάq‡·§Γψ    n(ΚΛ£ƒ_MΉ›6)©7dcr>GŠαMSeΣ©tZšς™ά±#UξΫ—JEFڝ± 3“r·nΥΆγΝ9Eˆ«’Νy|/p>_΅HT‘؊ΥU6Ο%B’€96{Ά₯¨ΰΘ*ζ ηeψS禕χK²;6%Εζ%]0[srϋφYΊ΄΅›θ°oκTΚY½šNq©‹EͺΙ»ο*›Φ­p`X ‰ΫˆE€€Q4!2g k–ώ—χ₯s2{rρβEΪΙ_τδ_²kTSχ©~Ο=tδχίι0Ο«–m―ΏNmuuκλfηu9ηKIF5uŸτ›o¦Όq`/‡²RΛ> šΪJ j ΔυͺυΓ9€€x›@,o`Y ΫPγν91>€€€ΐδnΫFkξ»ο—ΞκΥS"·. "l H‹έ~Hϋ¦L‘σΚ[/…ΗŽΡq¬tΥͺTƒ7>WηgL6AΫ’ΒμlZΨ£‡αr΅k©G>sVΞξΫG :uvcC]o^ƒ=)8z”6sT…ύ_|Au†9ι+©CDdυ†§ž’ʜڒ.·Ov2ίΦ6Ϊ­Ήχ^e,Λ?½φμ‘ψ5”βΡY³h͐uΣϊ₯FηŽ·4Ηgˆ‘΅`)  Φ’Λ–5]ΏΩ—'Σ†—*ΕΨδˆQMšεœn–/(–1£Λ—§Œ;ο΄ώl α urxΖ :κ”ΦvQζύδ“Ά˜\iπςΛ© ϋx†P9Ήb…‘u ­Χ  *@\"Pw/Άβ<•rTtp7€K‘€€€€ΈI θΜΚ^ΈΠpHŽuXδ(@³9ίόNήψμΜ{‘ΞMΏυ•Wh.G Κ^²Δ2œαS<ΙʚΠ9Ή9Ϊ2°YΏ”nέJ4ξIί½œd§/13£še|Λ§€Ιόι'šΟžf’₯θμYΛ%·>·K‹ϋτ1Υܝš k}{ €8N 2>žΨΕΚΠΑ~K§v_oόφΫ–’έOω’¦—Œ»ξ’HΞqζ¬H(ΛΔF΄έ8›όΑΰ¨TγTΡεΚ9Ϊ\i'9ίR9ԁ^N,^¬―’@[―AAT€ΈD`δΘ‘΄rεJεθίΏΏKc €€€€€,Ž.°κξ»5i/œΥ«˜ΓG.ζw%%l3LΒ‚›Θ™ϋΠ·ίš™―n΄uΜZΓαΙ‹NŸVW;vΞο›vŽ§؜ΩΠm6ψξ‰i3M /0¬…ΧύΖjAB˜ΐ…sηˆ84£^bΩpδ¨Hlh‰;ν¨˜Ÿj<ψ £έ νoΊΙP'_•cyWκέΫ0Ε “YΆ^ƒ¨[*tι 7ςŽHE!‚7@_Ζ~Zr8ΔόΎ£οαΓtΕƍԑΣc4?žΚ5ojύΟiGZΔωλ‹ε}“‰T•œυQ-Zς4λŒœέΏŸN.[¦ι"ϊ¦•°ιqλk―Ρ–Q£4}€P*&†*^qΥ>œΪLŸNνΨ;­Ύ28I”I€§ΣλΧ+Όl­Ρ0"‡7h{τQ]-Šα@9ΦΒα.c aA@’²š‰xd9*eυc%t”Ρy;wjZ”ΙΘp*·š¦3$”€^$,££’₯‹£M5ν*vο)KαΜφ횺@\―FA@@@ εœ‡RVžššJιιιaHKμb#Ωω“'58$dc›/Ώ$‰€y_$ο*±!κ²Η£CίOΛ ΠxΊfeQΞκΥT7bλ₯ ο’χ)ΗψCsIΌΦΜήοh© ‡y^½TΉξ:²•ς$›# myι%}J¨__YgR‹†k4d5Ÿ06>σ νώΰΝυ¬yσhΛ‹/Rγ·ήΤ;Rp’Δήo)Ϋ΄)₯ςζmω,ΗGϊb*T°\Ζ§”βθ^-[ΆTfŽŒŒτ˜Z“²Η†Ε@  Ύ&P™i:₯3†΅Dέ—,Σ/UζnΩbΈ\¦„ΔΆ†Ζ&±•*j Oœ0Τ™Uˆg^¬‹Ή‘Κpr^½꾈Ϊzυϊ’   αHΰσΟ?·ζH|ο½χΒΦ    Vϊ|ρ1όž₯+§ΨΠΥ¬T'ιμ!ΦδwT5œšEτ±4’”zq6€YΘj·ίV)ηηΣJ™³ΈXs]ΌηΊ―ZE¦F΅K-%mI³χίW<τ4Ή°λΓΙΦ†u}[u9Ÿ½νaγM#Δ@'a,Ε°–ΘΖ>WίU©ηΑΉλ"Ψ«r?r,_Ύάυt=aXΣA@‚•€™a-ŠσEΕΕ9Ό€¨ΔD‡Ϋκ OΡmΓο4Χ‹£†΅˜ςευ].Grn9%Gͺ‡~‡W ­W₯*NA@@ μή½›8Β9 ”†   dλΥ£ΞόϋV΄όψγ [ Τυ4 ……tŠCͺE<Ρ’ΤU%ž›₯θ8Ήt©Ν>i7ήHΆQ-Ή›6Q.‡„tDΞ8@'tγΗ€€PjŸ>¦έχσ¦ͺ³{φhΕΥ¬I-ψωwτύ—0©―ŸfŒ l°Ϋω:g —³—^γ7ί€ˆ(t†[0·…a-˜οžu_Ο?”W³Ϋ―EEE>œS8JΰΤΪ΅†¦e užͺΠždά87=ΦΜάόe7’#ν†aMƏINΦL£GhλΥ(λLΑ$Ÿ3έΡ@@τNsΒψ'9gGƒ θΞ;‚}”A@@x‰X}Π Ν!©/œ‘¬ωσι’“ΡΚR{χ& ί¨–Σμ=wfΗu•α<Ÿίισ·ΕΥͺEΪ·7΄•Š,“γΙmۚΆu€2₯{wC³“Λ–κJͺHf]γͺW/© …(ΦBτΖbY αE σ§Ÿ ‰[£9΄aΕ=ΌΒΜC̚΄ΥΕYΝ s±Ί/gΆ†.,!4­>–ϊ|Žι­½a-ΠΦ«ΧΧΡ²YHKGϋ’„*ό3ΐς:+++T—‰u€W άύTsΎˆH„‹W_}Υ«σapϐ‹g9κΐξI“hΕm·9=¨„mL0ΐΠοη+IΜΒ@V»ύv›]²ώΫpMοufhPBEΩƍ WσtωΫ te5Υ .M/\ξ4Φ  ².ΣφΧ_7¬/ν†Όš4UΏI8»oŸAg*τ h₯oΉζΝ’€˜ίφ03¬Ε¦¦jΊΪz5Κ9X/Λ²#   %πΪk―ΡδΙ“•Κ‰ŸȐ!Ϊ(Ψ%Ε‰ΪǎKύ.%‚裏hθΠ‘T‹wC@@@@όK θΜ’lgχξ΅y—Ξe“΄>ΟΌ³Ϊfάqν™0AΣMΒ<Φ9RS§.˜†,Α°&ϊλ%—CCξ:U_νP9oηNC;gίk%6lhαA†΅πΈΟX%€@80}:ΩΎέ°Bo†”ΙΚ5mJ©ρ”ΛΫ½›.pΘ€~ΉζŠdώό³‘[ωΛ/7Τ™Upμ‚ΜL*]₯ŠΩελΜψ‰;ΏZm½’›Δ>wFNoΨ@ηOœp¦ Ϊ‚€€€ΓnΰM=8ΏλΒ… •|kΟ=χ}υΥWχGCϋr·m£5<`h˜Θžγ-ώο υ¨_Ω‹Ρ‘ί§¬ΏώR‡8βΡrΙνΪQόe—Qή]Φn§Φ¬!yOo²ΡJ"‰~jIβ|pςΫ’B“χ[_~ΩVs—κ‹rsκΕ!Ρ!αI‘ ΓσΎcΥ !B@βUoxκ)Γj4 J={κ=Y•πqM5h'Έ5Ϋq€jbστ}r7m\/ΕαΚ·i£©³Y`o¬#3gΪΌlλΒΕ h7{¨θ%₯[7M•ΏΧkφeν΄Ž—Fa“Βρ?4©E€€xŽΐΫͺδνί|σ -_Ύάsƒ›Œ”Κζ-Z΄PŽττt“¨Π" žGٜkJœ\΅*΄ŠΥΈL@ŒVΛnΎ™π†'‰p$yΜ5ͺ%š„GtT‘j졦[α MΓ@šτ·ŒWœŸOψπΆΈλΉηmύ0ΎkΦ°‘ΧrΈ6‚± kF&¨  ήY‹zυ"Cn±R₯¨9ΗΕ–Χή³n»ή{Ο₯i²η^ͺ\wE—+§―ΆY>Δ/πœ1ζικFρœIό’N/ώ\―>4₯θ–³z΅^E›e ΉσΟm^ΗπvΌ[ωf~‘#’ΒyR3ωϋŠ7ε»ξ²ζH|β‰'Ό9Ɓ€' †¬ΉυλΣa;ωΝ, ‰ˆ₯$ήΠ\󑇨3l{ς¦gW₯ͺI~6[†5Γ¦lŽˆTuΰ@›SεεΩΌζΙ 0¬y’f`ŒUΜ)tZΆl©­Ω+SβZ¬.O͎q@@\"Γ;-Vς‹$‰­—κƒSJ—.ϊj―”k>ψ ν7Žˆ69Ι;Σ³/¦ :Xͺμ~ʎ;3£Oυ{ο΅ΫWέΰΨ¬YttφlJeƒ££²ν7 MΣoΊ‰JEχžψs½₯u9ίDig kϋ?ϋŒ$ @@ΌMΰ ώέZ·n]zζ™g(αqΌγƒ€€€€Bΰ η [uΟ=TΜοXτR*&†Κ³Q!‰ eΩ+-fMŠηC>]M瑟#‘Ώ•oΫ–N.[f½”³bIή²ΈκΥ­u²Q\²•W’Ω{K§θ²e-§šΟfœΧ7‚Χζ)‰ΛΘπΤP'Δ ΐ°β7ΨSΛk€ εδδ(ΓIbr€€HlκΝ/ΎHŠw—Κ˜eΡ&¦bEjόΦ[–’Χ?κΤ‘Τ«’£Ώύ¦™kο2κΐqΌε˚#²aΨ0*8tHΣ΄l³f”Ϊ»·¦Ξ‘ΒΊ‡¦ΞσηS;α .œ?OkyGΦιuλ4ΓJψΙz/Ό ©³όΉή²όsX/§9qοώiΣ(γΞ;υ—4εlϋ°αΙ'5uΞ"y›^ ŽΥW‘    Λ8ΏΖ«―Ύ     >" ωξWr(EƒQ=ΑΎς ΥΰΌ|±MΐΫ"α Υ†5™OΌΦκπ{‹ώαNΑRT>3J) ΔxOΕ:Ο5‰,TR^6Ν$(€€ ΐBβA˜‘<ԟΘΛΚ·kσ1ˆluΰ…“Έ ŸηΌeηŽ§Ž—žΗ»ΤήajΥc*U’φΏόB1ΙΙκj―Ÿ7ζ\*’»Kλ:ΐšΟ±ΌΫ͘A»w·©C1―oύγΣ^}‚e gΙyΟΜΌΖlvι‚$ʝ߱#΅ύώ{*ίͺ•isIx»¬ʚ7Οp½:οQΓPo©πΧz“yΧW'ό=ΛΖU«°qUv£E–.Mι—ΒnY―ρ‰Δ ίχ駴qψπ}ωδ/ΥΔnπΞH™jΥ ΝχMJͺΣS;Ϋ  @@@@@@@ΐ!Η8zή UŠ#Ϊ|ω₯ιϋ[ƒŠΞ©zΛ-ΚΖ^uN7½aMR fUnΈΑξ΄±Ό™ό¬Ξ°vfϋvΦμ’Co€aΝT1&€Ψ!pjνZ’ΓS’ΐξφfΞ€x6ΌψZΚ6h xΙ­:T3uυιCβΞ/$9Pysca>{§Ÿ3GρΆΚίΏ_ΣO ΅}”*΄oo¨/±Be0Κη0σxΎ”nέ(νΖ)Ž e’«MŒ“¦R j…YY†αβΩ―‘ƒ§?Χ+ži[GΦκΝF²εόΕ΅tZ%s^›rΦ‘(7W z|ξ\*ΜΞΆΆβp\΅y—ΨΦ—_ΆΦ9r"‘!τ"^Š3«T‘DΎςε6Ύvmjόζ›ϊf(ƒ€€€€Λ$'Ζ’E‹h9‡?Κήςr;vŒbΩ›>£€ρχΛg[ή„”””δς\θ ΑB@Όs:™lJH–%@O/8ΕΈυRγΎϋœ2ͺIόƒυΓ8UŽεMί•8=ΗQŽbd1ψε ΨqΌi·ΰΘΚϊϋoΛ%ε3­_?Šbγš=IβwKϊ”(bXƒ€€?ΐ°ζκ˜@ό05aVŸΨζ? £šR}~yѝΓ-JFg$©MκΚ![LžμVΞR’›νΓ©Ε”)TΊjU»*ΔςŽΰό₯Qt.Ϋ¨‘έφϊώXo‡«Όbέ:ͺ9dE°Α«Da/΅>H½87_Αƒ•¦ΖK―;b5—ϋΓΖ=€€€x’€„||μ±Η¨‡xrΕ¨¦Χ%‹½κGsψl ΉiΣ&ύe·ΚωœΓΦSkn)b9u3QU  $o’ΒαΜΆmNΝpόΟ?I UξJT\œ’–C=Ξ ~η“Γh²ζΟWW“€Ό(α˜‰’ΏKΡ{k²7άΆΧ_׌ιHa3ΏΏRΥ€OΖ An½“rd^΄ πX {‰•€(1€Ιh"†‹±Ε“Ί%±χWηΏώ’Ξ!wδΧ_IθζσnίΞ‰q‘ €b*TPΒ-–αΠk’M †‰ή$λΚΰP1“έψ%ΫYΞ»VŠ M‰Ό‹I1€₯sθwγΠϋc½’sσ‰©1ο:ΒΉΞNπŽκsΜχ‡+―A‰Ή/LΛ_~Ή!ο^‡dΓ©ωΐTνΆΫ(—ΏœΛtΙίUΆ,ΕρΞκ$Ξ큒œ9s†ήyη’Pz άUlk\ Υw˜Γ!‹Tβœ’s \Δ4pΰ@ϊفy111Κ}>ΕΉ|s9§¬=ΈΦœwwΦ¬YΤ₯K{ΝΎ.γMηΠQ·rσ@“@Φ-ΠXA%‰ηΠ7ίh–$k佇#r‚σ™.ευΔ›]šl\Γ^σΠ# lΒΏ”ϊ!ΒA##`Xs„Ϊ€€8M@ NrΤη˜ΥΉΘq΄έ‰dιγκgDTUεwrψBό±^1°U0@9|±F™Cζ,Ο»Ιε€€€€€£$WΦ=ό2ε(oYΉr%ύΚ›oς”HΘJΙε–””δ©!=6N λζ±Eb BfΡΡ<πŽ,―%G뱈ΩfWΙG&ή\uGŽ΄ώώ΄΄·|r.ΣνœVcΗΈq6jGŽXš;ό)a%—ύΉΜLkŸΣ}G-²Έlƒκ*»η’β’ςuΧΡυ†6.ηt)uŸ{ŽŽcσ½SoςZψγ΄oκTΓ>ήRνΦη~Ιψμ³Ο€I“άǝY7o¬c‚@ΈΘέΎΦrΘ½$Τ­«€WΠΧ£8²9―¦ώ]©6¬ID δŽιΔ’Ešα%δαα3θ26(Iδžθς啨By»wSζ?‘ίΧ„DŒ«QƒΚpjμ… ­γœΩΊ•6Ώψ’’ νη4MνέΫzΝΦ‰D ͺƞέ;Ε`gC2ΔƒΝiυι§τGΝ9»w―¦χv6"ύο)…7γˆŸ9ιώ—γ(oτχ‘yCv­‡Œ…Ψ#Ú=BΈΨΈq#{γ άΈqcŠδŽp„€?ξβP?Ÿς‹ρ.[±b…ΝέΣJ#ό&L˜@?pΨn½ˆ—šΤχμΩS©Δ²x³‰‘KΌΫΊrώ1.©eΧ]$sZ<ΥΧ\=Cή ΞΗΎ}{W‡πZΏ@ΦΝk‹Ζΐ &Š8™ΧΣωΣ§Γ„–iF@6φΆζΠ‹4kFΕμ™₯–Ž°Šs™Ω1ͺuζΝOΗωPΦ€ίΆW_UΙgί—Σ}8"Υ8€-ΓZ$ηa“TH Ϋ³Ap1{ŽηsΪ΅œZ½šδpTΩc>‡€€³"œν€φαI {χξԜ­όrœόπCSΟ%ο̎Q€€˜Ι9Nτ2|ψpζέ“πŒ­Z΅"ρŽTΛο:κ‰'žPW9|ήΏϊϋοΏι'~Ω¨1jέΒωYjp-I λζ/&˜@όI Οώύ~›>ΎfMκΘΏk%Ϊ‰Ε‹ιτ¦MΚQΔ^Ϋ’?-Z5%˜½€­™Tζ0‹b “°£§Φ%ŽίMe8gn ‡[vFͺH›Ÿ{NΣ%‘~}JιΤISηjArΉΥδœƒrό؜9”ΙΏ£s8dAf&rΤ51Šή²n™WΒOJ9W€ΖΰΑ$„ kx@@@@šΐ«œ/`ςδΙ ƒ‰'*a½Β^$P–CώŒ5Š†ͺΜςΪk―Ρ`~AQžse@ΒƒΐΏώυ/ΓB«T©B―ΌςŠ‘ή݊ϋοΏί`X[Έp‘β-λπ’§MBMζrŽ#‹œ={–yδϊν·ί,U~ω dέόčIσςςΚMό2zϞ=ΚF€C‡Qrr2₯σ Ϊ΄΄4ε³cΗŽΤ—_>;’‡>“_ςnέΊU£•xjJN@WeΡ’ETXX¨ιήΈqcͺxΙλDsΑ€ Ο±ΊgϞM8g’δ&ΜΟΟ§jό½:‡ƒόR<%%Ε€wΙUfλƒwι₯­eώ©S§j~ ΎfΝ%¬κ»οΎK½{χΆΆΑ €€λ*_uΙαͺΔσ†‘ψ»οv΅»/oηNC1„yZ"9άx•λSOρ@ΐŒ kfTP      ΰςΛ”χί_yQ}ο”.Ε; !αA@ΌƒΏδΠKzXLLŒΎΪνςmΌ+]Ό$ΥyΥΔ±”s²ΈjΜ£Κλ―Ώn5[”OΈoΏύ–nζœ1ώ’@ΦΝ_LœΧβI;eΚΚΙΙ1t?vμ˜Α8&ήξ²Aΰ>k&)["Ϗ= —Χ²'H³fΝ υφ*φνΫGtb¨“5Ψ“£GsμAς‡~Σζ€οŽ;”!>ωδzꩧθ*~1/ν/Ώόr{C[―‹Αξή{ο΅–εDŒ”ΟΞY³fΡ<@ϋu^=βΡ )™@η’μΔ~½D!€ Κ@`7o\T‹„—ΜpΣX§η ΰ/ώšσ‚€€€€€„θθhϊμ³Οhσζ͊-))Ιe•8ŸFΣ¦M•C|XρκQΧωϊ<uσ5 gζ“gσnφ (..v¦›¦­„”°Π–ˆ—^Μ dϊ6feρΤ‹Ωψκ6cƌ!ρuΕ|αΒ7nΙΟοS§N©‡uκ\Βn9©>h  œφLšDόƒU£|- _ίνΝ›7Wωήι)aΝS$1€€€O ΄™>:Νk=Z}ώΉOηΗd    Ώ5½8ZNίןε&MšΠπαΓ *ˆΡ`ωςε†z_V²nΎδΰθ\ΫΆm£nΈAΙ½§ξS†συˆΑ^Β!Š'šN7nάHsωoργΗ+/θΤνε\ N½zυ2Œeiwλ­·’?Υ"yΧd\gD<½–-[¦ι"ϊφοί_S§.HNKΙq© ΓzΕW(Οσtώ맟~"yŽοΉη’Ό˜z‘έώfΌτνΜΚ+W€G}Τμκ@BŒ@{·ξΉ”ΗΪ²΄Jόσ±G€€@(@Ž΅PΈ‹>XC£FθδΙ“ΚLͺ ώ&’Λ'ΰo}0?€€€€€mωωω¦žΐ† Zͺ”Ο*UͺΌCΤc=Fί= 0@γι–••E«W―V<»4Ή yπΊsΏ?ώψCsIΌΦ$T€£"σκεΊλ35„I»Ε‹ΣK/½€οBυλΧWΦiΆƒȐ!4aΒ%Oα| ι;oή<’gμ­·ήΤΫ+H(Iω?b εΫ»wokH_1ςU¨PΑrŸ AB ψά9ŠŒ΅j+εε7έD…όσP-uŸ}V]Δ95ν6™ ^ ”χ&ωd‰EκNoꈱA@@@@@“ΐ±cΗ4Ζ‹–’—*X₯tι4Y·_Φ²vνZţɟλ dέόΙΕln1‚ͺEr7.\ΈΠ`TS·±œ‹‡Ψ;οΌc)Z?KΚ΅fΡΩpfa oηΌEf"Fm™SζRΌηV­ZEfF5Λ8βχώϋο›>Ο~ψ!9rΔΤ‘OKN΅R₯Jш#”ωΕ8'ϊ‰‘M }+Vth,4sωwω<ή(³‚ύέ­ύ–œLGgΞΤ(˜ά‘ƒ’PS‰1ΈρΝƒκ ΰ –!!!ΑCcLθ_ξ8ΠM@@@ ΰ œ8qΒTΗ`ίΈΩ£G%7Χ§Ÿ~ͺYŸxέ|σΝ”‘‘‘©χe!uσ%‡’ζ*,,$ m¨ρDsζoβ›Ψ+CŸγqι₯κ!5η7ήx#=όπΓ$/‹lΪ΄‰$$€–μɁH?~JJ υιΣΗ΄λη2Ϟ=šk5kΦ€?ώ˜βββ4υΆ ΒD6\Ϙ1ΓΪDτχέwιΝ7ί΄Φ9z"†Dαq@ξφν΄φ‘‡ λΦ₯ζΖΎ&pαόy:Ιaε06¦7}ο=ΣK¨`%ΓZ°ή9θ ^"0gΞ/ŒaA@@ 0 T­ZUΩ%-Ϊ!όP`ή#h ό²³³MαŠa­gϞ¦c9S9kΦ,ŠŒŒt¦‹ΝΆϊΧΏθ·ί~£γǏ[ΫδεεΡΠ‘C•|UΦJ?œ²n~ΐa˜RŒjb\SKΧ]ΥE»ης=BΎ?¨Ÿqυ³ @ς–IΨΖ―ΏώZsIΌΠ$Ό’=‘0/^Τ4»ε–[(::ZS'iχžΙΛlΙ‘ζ¨QΝ2θ /Ό 1¬Iύ―Ώώκ΄aM Ξ0ͺY¨ΊφY”›KYώiθ|>'ΗP‡ π7Rό³©)αΛ·iγoU0?x” kΕ‰Α@@@@‚€Ό(’ ΰ_ςψ«―Ύ" /6{φlŠχ―B˜έ£ly¬•+WΞιyτω©œ€;θ Œaι“Μ!―ΔsGβοηŸ¦~ψΑ―F„@ΦΝΒϟŸbΣ{ΆnέΪ)•Δ0wκΤ)§ϊΘ³’7¬‰—#†5³0ϊgΟ’ŒδVO8΅tιEΙk¦sδΌeΛ–ΤͺU+%|£₯ύζΝ›ιπαΓ”––f©*ρSB”ΎύφΫ%ΆΑEΠ!Pš6\ΞΉH+tμ:‹ΒJ@ΰΦπ(€€€€€€ψ@Ώ~ύ¬ή=βe3jΤ(Ώλω„FŒ‘4yϊι§b¬$X~Ύf+‡₯֚5kζ”vƒvͺ½„άΟ8o‹x>=š†™jΒ„ tΧ]w)aτ4|Xdέ|ˆΑ©©δηx‘νέ»ΧpˆQΝ–¦£“HψFy6Τ"aGŽ©œ›…,Ι°&ϊλEBCNeοWdηΝ†nΞΗ6lhθ π&ˆˆεχΉ§η€aΝΣD1 Ι]!(@@ H.‰ο―y±aΆ«Rίemμ9ntpƒ€όΡϋΦ[oQίΎ}•Q¦L™BO<ρΥηϐΰ'P’aΝΩΥΙ³αŒHž3_Ι“O>I_~ω%­Y³Ζ:eqq1=ψΰƒJΘΌΘΘHk½―OY7_³°5ί’E‹θχί§Ώώϊ‹V¬XαtˆG[γšΥ·k׎.»μ2Ϊ΅k—υ²<7»wο& ο¨Ιe&ϊ©EςΑΥ«WO]₯97ΛmψςΛ/kΪΈ[ΘΝΝux†DuXΙ h˜P·.uœ;Χ itΩ²†:T€€x‡ kήαr£nΩ²Εϊ…Rvω󏁐ƒλ§υκΥΛO3cZπ<χίίtPIΜέ­[7Σk¨<}ϊτ‘ž={\~a(9‹žyζkή΅ΐΣ9Cΐ“†5gζ•ηθΤ©SΞtq«­ό­όο›$•Τ,²jΥ*’͍’sΝ_ΘΊω‹‰e^1ZΙ½±ŠΡ^ύ)ω6nά¨rψ\ΌΦd3 ZD‹'Ίή, €τ·%ωωω$‡·Ε]Ο=oλŠγGsΞΖJW\ŠΛH©τ@IDATKÚ@@ h ΗZΠά***/§›6mͺ'9%@@@@@ΌA@ς«I.‘Ÿώ™,Xΰi0¦ HŽ΅jΥͺf•zήΙΓ'ω°|)’Γκ±Η3L)Q$ηš?%uσ1d‰g¬£F΅ΨΨX’H=τςσIΒΧΊ*fωΩlι‘)†ڜΪαOerΦlή\aπX α›‹₯€€€€€@°hήΌ9‰Ζ΄iΣΥΕkmΙ’%ΑΆ θkB kΧτΕ_h,[ΆLρNŒŠςΞλ‰Υ«WkζσUa̘1$!(Υω§$dήΠ‘CiƌΎRΓtž@ΦΝTa/VJΞ°{ξΉΗ4MBLL I¨Ε–-[*αε%OεπΤσZ—Cϊ‰w£ό?°ˆ„ ”η¦zυκ–*ΚΜΜ4„ΌςΚ+)55ΥΪFRΦFXΐ>ϊˆdmž’ŒŒ O …q@@@ hxη›kΠ,Š‚€€€„; dΙA’––FβUπ/W_}•Δ;£sηΞ$lΠ `fX5ΉΧ·ήz«W9oή<―ŒkoΠψψxΖΥW_­iϊγ?*ž˜Χ]w¦ή—…@ΦΝ—$L¨ρυΉΗΕμ•W^‘x€RRRΌ’θ 6¬Ι„β΅6lΨ0λάb€½pα‚΅,'%…”λb<“{­χ\λΡ£G‰yΩ€/@@@ dY2\qςς¬I“&ΚρΝ7ί„ψj±<β!9‹fϞM͚5 ₯‘₯]ΆrίΎσΞ;vϋΊΪ`Φ¬Yvu»ίUW]eͺOΌΦτΖ·'sr€@ΦΝΙ₯Έά\~Ύθ Zβ‰6}ϊtzξΉη6ͺ‰ΞΉε–[H里)³nΈΑξ΄+V4΄ΩΎ}»‘   Ξ€aΝ9^h       ΰ—]v™fΑΎ$P»vmκΨ±£aJ ΧψηŸκέ­ψοKλΧ―ww·ϊΏχή{TΎ|yΝβ₯χK/iκόQdέ|ΑΓμΩΈοΎϋθζ›ovjzwσζUͺT‰zυκ₯™S ~PκŽ9Bύ·ζzΏ~ύo4M₯IArκι†5=‘ΰ+η²qt!‡Υk~8ψA@ H ΐ°€7jƒ€€€€€@ΈoŒΖ+GεΚ•ΓGP¬ΩgŸ5ΥΣΣ!?%lήθΡ£Mηςe₯δΐ2[Ϋψργi͚5ΎTΕ0W λfPΦ 6l0Œ*αg•Ν›7;ΫΕΠ^ΦρβΕ‹τύχί+νΜΒ@ήuΧ]†1Μ*ΜΦ£7™υ³UχΒ /lzPβ] ρ-"ΞΧx|ξ\ΓqB•«Ο·a6lς»JŽM›6yLQΦ<†€€€€€ψ’ΐέwίMςr\u>"_κ€Ήœ# 9Η$ό^Δ»Μ“αx‡NK—.ΥOγ—ςΰΑƒIςΛ©₯ΈΈ˜{μ1u•_ΞY7oΩ±c‡a ³gΣΠHWρρΗλjœ/^ύυ” ιh iω΄\LOO§+ΈΒR,ρSς©ιεηŸ&WŒ{φμQŒΔ»wο&ΛqμΨ1ͺ_ΏΎ~ ”A@@ `Θw.KꇦM›zL/Φ<†€€€€€€”D T©R6=ΙξΌσN%―^Iύν]+,,TΒ,Ž7Ξ^SŸ]—5Ož<™bcc5sŠ‘ΒίΘΊy›M͚5 SlΫΆΝPWR…„0C•»G7ήx£f˜Ε‹+^σηΟΧΤΛ“ˆΗ^ηΙ‹D½QWΌα^ύu͘Žžώy’_j4h!?œϊ:ΞA@@ T 8φ›8TWu9L@v 5jΤH9"##        & ω‘6Ι$/νΕsg̘1tξά9u‡ΞgΜ©„}ε•WjοΛFυκΥ#1L’²nήδΥΌysΓπ_}υ•‘ΞVΕςεΛc˜μ„χ„˜…ƒΌηž{HšͺΕΡ0–>#FŒ°œZ?ezO8λE“ρ&Υ³)]Ί4=υΤS&­Q ‘E`/{&/`Οsυ‘΅pa@,2u @^T"Κ‹ccθ"ΰN ξΒ€₯€€€€€\Ξ/“˜˜š@W Όϋξ»΄jΥ*Z¦Λ TPP@£F’iΣ¦)‘―½φZͺ^½ΊΝiδYœβσλ―ΏΪ΅hΡ‚:vμH~ψ‘ᚯ+žyζΕ8αJ>oλ¨ΊI˜ΕyσζylωS¦L±ŽΥ²eKλΉεDςšΙ³4rδHo>39qβ;–Δ+–QνΘ‘#f]K¬“°UͺT‘ΜΜLk»uλΦYΟεδςΛ/§ hκμΊκ*ΊξΊλ4žu’χΝ7ίLΟ=χœbΘΆεwζΜzόρΗiκΤ©†idύ΅jΥ2Τ£@BΐΩ½{){ΑΝ² ³³5eY71ρΥΌ0¬ωŠ4ζp‹€b$w–Όž;w[c‘³ ΔΔΔΠ·ί~«„©3 ‰(ω―†ͺ=₯vνΪT©R%ε8uκmέΊ•ΆlΩB‡²ΉV­Zќ9shύϊυaX“5ϋί¦N:‘„γ $ Tέπ‹L9'GSW†~G²n8 8xζnΓ°ζŽ@@@@ H €₯₯)αEύδδδ ]ԁΠ'  νΫ·§σηΟ+‹•Ύ}ϋφ ύ…‡π «U«¦δJJJ’vνΪ‘„­ƒΏ₯C‡4dΘš8q’ΏU1ΜΘΊ”υ@…θ?όsjΦ¬‰g–ZV\I’Λ̞ˆQM<κδPΦ€ί«―ΎͺbtÜ#"α mε”}ϊ”ΠκŸKNRΒ&JψΔ/Ύψ‚*V¬¨\¨P‘ύτΣOow { Δxe&κfΧ©“ K–,‘I“&9eΘ–0’_~ω₯’ϋω-!6 ‘D  Μ4HsΔedΔY·€δE%Jq\οΐ μνΕΕbhΠ$Π­[7š?ΎaqύυΙ5€€€@θΓa|Δ€""/‰ωε*S¦ ₯€€Pjjjθ,τJ>όπC%Ϙ~a<ςH@δ Σλ割Ό¦X³f’MBμI½γǏSvv6I.°*Uͺ(_βυ%aφ½φZ%W›= ΰ —Έxρb%τ¨„•‘ςŒIΨRΙ?&F―š5kڜBς‘Ι3»vνZ’ηQŒΔ]»vuΚKL ΜβI§–ϊυλ+9Υuξžηηη+ΗΔ (‘ 333ιδΙ“Š±Pτ–uΛΌ~RrΝAŒΔPBλ₯mΫΆŠw ΎeXχψγΙ· ά8H„‹¦M›*ŸlΌ’ά»žδXσEŒ      ΰΓ† SςSΙK`Ι»f kφτΣOΣΫoΏν0‰w ˆABMΗηΜ1@(δpΉ$βe\Μ!G£xCŒ/Dζ*EΡΡΎ˜sψ€―Ÿ%. S1Φ‚ψζωRυN:QVV–2₯ΔΆ—όπΩθχζ›o]wέE'Nœπ‡ ˜@†ϘA‡ό‘ΞπύsGΕ’"Š©Pβ8o`όe—QbέΊTύή{).#Γ­%ζδΠQΞ‰˜Ι†θœU«”ΉŠrs•1£Λ—§2œ#°b”Ϊ·/UκΩ“J9ι ΗΉΟrΞD«pˆβŠΊόνbμΙ^΄ˆ=ΩΎς’βΌ<Šε|―₯9l2ηί«:`€’‹u;'ώš·$΅Ξ3Χ£3g±Ω³)Ÿsžpn1b–αά“qΥ«S‡m:p Εzπ]±§ž₯ž»u«uyόŒκ%‡sL^(,΄VΛσ“Δ95Υ"Ο[Ρ©Sκ*Š­T‰"9€ENs^ΞsǎYŠΚg$•“9η’«rrεJ²<Χ–19χeiΞ5kGts‡ƒΎ―2/‡V¬ΘωC]ω£f.γ”mܘb+VtuHΏτƒaΝ/Ψ1)€€€€€€€;Ύϊj>|89aΠ@‚ΐΩ³giςδΙ]zυκEM›6ΥΤ‘F@^μoyρEΪ-aL/E»Rλ(žtrδ¬X‘To;–jάwΥγίίbsFŠ hϋ[oΡή\SΜgΜδόΙ“$Ηι hΧψρ”ΐΖΌ£GS:Ή )ΌoΚΪφΪkž kύ.\°–ύπmxςIΚίΏίZg9Ι»”'Q nω{ŠΜΫ‚ζΡεΚYšΨότΧΌf °atσsΟΡώ/Ύ ‹*Γ“₯m‡$ΩΙ'΄α©§¨2秬Λν“/ΏάΔιOO?KG~ϋΦΨ Ύώ±Η4z¦tοNόSS·σw΄Ο_mΛδ΄λ―·Ά;Ξ}τc‰εUΜ1&9ΩΪΞΡρL]ΠΉ3]ΰg^-=Ωh­6¬9’›;δξB6Rλ₯ΗΪ΅TY3}΅έςΩ}ϋh;πh„ u½Υ†lΝΕΐ-DjΠ @@@@ΌO “w2nήΌY9Nςαΰ!€υΑs― )€ΐœ;wξ>“ςM7έddΉψμ³ΟZNρ I {ΙšΓ)cvψ‘©QΝLi1μωθ#š]»6νωχΏΝš˜ΦήΈ‘ζ6h@[_zΙ¦QΝ¬£x“­ΈυVZΒrΔpγŽ9CΛoΉ…–χοojT3ŒΝ^m‡ΎϊŠώd¨<7 ΎžwοΗΣ6Hξ›:ΥΤ¨¦_ηΕση)σ§Ÿh>{ιm6ŒŠl=υύΤe_>Kκy=u.^{₯taHΕcSΈΈ"OoTKξΠAρϊtežN,[FΩ *:hrαη•ΪΛ`5ν„&ΤχσεΌ[9όε–Q£τ*P©˜Jα°„’OMΙ–ψ0δŸaςRoN―_OKoΈ:όχΏkO]αΝgIΒ€ͺQ'ωώδς³«–Ta)ω,’Θ\yξpή?΅ˆΡPŒŒŽ„΅τ+δόΑΗζΜ±•ΟΞη&9ϋ\w9TeΟΟ΄m―ΏNMΩΣI/ωEώ*6Vθ 7±lDk=mUΊβ M1%²1₯"{{Φg‘x_mεθjƒ€δ^[ΝΖ­Λ§OΧτ-±ΐο,Ž« εΩγ΄ΙΏώE:v4νΆƒ―m|ϊiΓ΅Γ>Ο)ÚηΝfc㳩—„ϊυ©Ν—_RL 2d5Ÿ0A1Ίμζ{«–¬yσ”ά{9'ž-ρφ³”₯ Ιa‘u?n0¬ΥγΌpΆξ£₯Ÿ£ŸUΨ£1š=Ο³aΜ"ϊ4“mwήi©²ϋy˜sψIxM΅€έx#Ήj}ιm`X ΘΫxJΙΛnq“”#222π„F AOΰ%9PTTDσ7_ΨΤ΅;xŽxΟέΘ_š4iτL±7GωeΔCόΒN~ΏρΖαΆ|¬@ H\Ιω’$BΖ8}7Š=jϊs~Ωl…Œ-Ar ΓVΝΜ_~‘|Ξ“€–Λ\Γ<:*βIV“CκεδΚ•šͺ£œŸ+oΗM]tωςN…Τ³tn α urxΖ §½ΗZ~ς EpΈVGD<τ*sh@½feι«μ–½9―bΐ4ΙιΨ|βD‡j–HΈO½ωυW}•Rφε³dͺ€—*“9LhB½zšΡ/-š†\PΌΊtžƒ’»ΝίιS$\£„mΤ‹™šΎ”•0l,WKϊ-·8όIέ/PΞaX ”;ΰz,β››7oVŽςόK  ž& /ΐΔ %"ΙΌ½m`33¨ΙάςeEŒ|ΰ# !Τ&Mš€lΦΟ!‰<|‹€Ζ    ΜBΪIˆΒθΔD§4ά·―όρ­ιszγFMΩ,žx9β-¦ˆ e92Mb£FΪj=©χ’Σ6Π–*prNnΐ-«Ÿ“‡<η€aΝΫσžΰάjgt^G2gjοήZ”’ZΆ$ρSK.ΏOΞ?|X]₯œϋςY2Lξε 3―5Ι―ηˆϊϊkm3ώRύξ»΅u~*ΉΜg6žŸ–ζ΄0¬Ή„ @@ΌA@ Zκ]8ή0°Ω2¨YΦo5 |‚€€@πPΞ­[·VΟηπC/˜μœΎUAcψ3μΕ΅ˆΓ§ιuC‡ώ―‘—ΞNκBΉΙ4•―ΎΪιΩb+U’K–Pηωσ­Gmͺ1ψθ₯ηfwUMΒAfις·•4Ά£‘.ΥcΔκBAΚ΅BNƒαŒx{^3βuεͺ€tοnθzrΩ2cŸ%Γδ^P F:Γ±x`εε•8³„KΜfη΅Tθά™βkΥRWων\Œ­ΎQ-§Χ―'ω™T’δ:D'ψ»ZβxMΪ·WWέ9 kAwΛ 0€„.΅Χšz•ž0°Ω3¨Ι|πVSSΗ9„Κ•++α—$<σΓηΎc₯‘I@~—ΏσΞ;ΦΕM›6Φ­[g-γ@@‚ΐyΞ!zlΦ,Γ‘mbˆςδZΕΣ*oΧ.͐₯8|i₯^½4uŽΔx“ΒžQ–Cν vξψqΚΫΉS3”δ™r&·š¦3Κ6n¬―’3œ ΒQ‰―SΗΡ¦ΦvQ>O/ΠρφΌΩmPGοufhPB…η<]ώ6_>K%¨κ΅Kqό¬¦tλ¦ΏψμY::s¦¦N_8τΝ7œ|N.QrŠHΤtήΔ¦3οCu³0ΥnΏ]έ$(ΟΌ9(o”Π% ^k?όπ—Π~™[ lŸ}φ βΣΟ?<Υ¨Q£DbP›>}:3FɟVRcx«•DΧ@ t ΌόςΛ$@ 4tνΪ•ε°TΏpω0bΔšΕ/ !   Θέ²ΕΠΉtZIξ%O‹Ω\eͺUskρ’ΣKα‰ϊ*›ε˜δd›ΧΌyΑΫσžb#½δrhΘ½S§κ«*λ ’ιμΎ}šΎfχΧ[Ο’fb$/Z‡(W‹ Μ<'-mταO#JloιηΛό;ξ =&h¦”0υFŽΤΤ© ¦a aXS#Β9€€ΈOΐβ΅φύχίΫLo`+°±γkΜ94„“"os`ΌΥlβΖ:cǎ₯ߝŠ9ωϋμΩ³•£—‹;κƒnρP@@Ό@ΰόΙ“†Qυaα \¬(4™ΛmΓZjͺAg kδv3LθB…·η5c°ΥΓ›‹rs5+χε³€™Ψ‡…τώύiέ#P±*όγ‘ί~£b~YΊ΄A“3μ š³r₯¦^ŒpQl\ $In׎β/»Lγ½zjΝΚΫ½Ϋ4d₯δΧΣ‡·Lβ°ν‰υκ²\‘ ]Β†N  ή$ Ο΅fk.‹mωςε¦M^ύu‡ŒjΉ?ι£@@@ xœΰζ[yW΅Η9l”E6lHχή{―₯¨x­‰χ@@@ΐ5ηsr ΝrˆΉPafx‰sΣcΝΜ³˜σ±:*11Ž6υh;oΞ+λΏΰWvαά9MW_>Kš‰}XƒXڍ7jf,>s†$ך™ϊϊkCuΖέwκ‘BΙ!§SΔV8HΣ0μυζK‘τ’φΑrxjnΦάΏ x`v[/UvμΨAr@@@Β‹€CRϋτρω’Ν<ΖΞ8ΰ=bΚ—7Œ›ΏΏ‘Ξ™ 3Γ\lJŠ3C„\[1žE²wΏ:˜,R Bή́εΛgɟ7­{JepΨΓνœϋΧ"Eμ­wlΞͺrΝ5–*2„dƒš™·›΅CœH8H΅aMT―΅:Γ†Y΅;όΓDΊPμΒ#TδΩ»π¨ͺ΄γo @½ƒŠH‘&‚]Ρ΅!*φθZΦΆbY;+βͺϋ­»‚½°Vl€ "¨4ι‘χPCΎσ^™qn™d&™™ΜLώηyf3χάsΟ=η7γ’Μ;η=€‚L–W2Κσθέ»·?ΕΚf D£|{ΊGJ* «Φ‚}˜)V«EJ’~@@@’U jƒ©•7ΨεκpE†Gΐkη²eΑš‡TΏsΙW»Ϊ]ΊΈκ*[…Χ>yΫ,ˆ*C,ίKQH7»ΰW«UŽ/;Σ@j`3pε—«ƒ8¨hzΦY’RΕΎfΛ™™RƒΈN;-F™!X‹Œ#½ €DAΐ·j- ][]²Z-Z²τ‹ €‘– 2RύΣ €‘d› TJzΊ­©ΎΩ³aƒ­.”ƒ½[·Κ&ήϋfeŽο1σŠ+ό—ΦξάYLΊ,±>ΩaφJ-ޟMΛv"ΔƒΥγΖΉZΦ9μ0W]e«ΘξΦΝ5εhΦbω^rM.ΖY;JvχξΆ»κ{±xο^«nΛ¬Yβτnn@Ζ{ΡΰhƒγŽ³ SW°ωV±ξ^³F6|ϋ­ν|γΣO—*ϋχ?ΆHΠk ϊΒ1l@ ²DsΥ«Υ*Λ»ˆy"P²ΐZ³‘ωόωσ­Η–-[JnΜY@ ΖΡ^½γιp;@H«^]²=Τ5ϊ―½ζͺ+­b΅Ωgj_A­Yξ‰'ϊ«Τ¬)VpΝ_#Rh‚qΞ0§K|ΊΥμ_•Λ/Ά6$¬Σ£‡­2ΤλΧΟ5퍎€ˆ«A Ώήu—|ή¦ν±mξ\Ϋ±|/Ωn\AΝ/ΊΘvgέ›pύW_YuΞΥjΊΏ`γ3Ξ°΅ΧMi+ϋφΙͺwί΅ͺ<Σ@^x‘­y’XKτWρ#€I.­Uk¬VKς7ΣC {ξΉG:tθ`=ήxγ0€) € €•G aΐΎPΎY/yζ wu΅sO)M)Χΰ˜c|]Z?5ž³όώΔΞͺŽWŽλjΧθ”S$½vmW}e«πrΦUΫ~ύ5lŠ&έζ‚‘#e§Y]θ{μY·NjšΏ΅œ%–ο%η½c}άτœs\«=}iσήzΛ6œ&&Εb•ΜL[]Ό4:υTI3AπΐβŸ—#έe΅&M€ώΡG6MψηφD˜ ?&€ ŒΊjν]σ­—pY/Ι‚Υj%ιp@βEΰψ㏗±ΖΛψαL›6MόqΧ%=Μͺ‘›nΊΙUO‰(P―^½¨{ϋΒ…ςσ 7ΈξQ£m[9δώΟUɊ–W^)σ|ΠΆΪLΣΨ­ϋβ Ιu€… v_m»ζ£l§λυν+ιf₯N`iuΥU²θ±ΗΔ|ΰ―ήlώ?dγδΙR―O]iO ·o—ε―ΌβjΦβ²Λ\u•±’φΑKNώ²aβΔ?§oΜη.=Β\ψλwΪήΪ‘ΦJuμΕ₯υ±|/ιύ*²T5{6<ι$YύΑώa¬ώπCΩ`R>ξ\ΊΤ_§OZ\r‰ν8ž4¨«λVόχ΅Ιόχ©ι-mο'3‰ζf―Ή”ΤδZγE`-žίŒ @ΐπ­Z{Ηρ—²ς°Z­¬r\‡ €@| Τ­[W8ΰkP Μ^‘–­&Τ«―Ύ*ύλ_C½€ΒΪ΅kΧNτAA 222<kM›6•sΜ7ϊ) PΊΐ^³―ΩΪO?u5¬έ΅««.Υrs₯ΩyηΙς_΄uύ£ωΠΌοηŸ‹ξUR)0iΧg\z©«IξΐΊšζί>­_ϋρΗΆsΣΝWτωδΙ:θ [}°ƒ97ί,»σςl§³9DrΝW(΄»ν6W DS6>ν4irζ™!1­4+―œi S«U“vAΎ4Λχ’oif_?gΩmΆˆEin Φ Μή„?_wνΦ5Ϋ·+hl»8ŒƒH:47ι kŸ‘ΑΑβbۈšUpHέϊA‹~Ψή8G’$W˜0"τ —‘άkΥjqω3(@ΒΈΜ|Ϋ|ήΌyΦγΦ[o ιϊ§Ÿ~Zژ½?3fΌώϋ!]C#@ψCΰ G‘Œϊυm&έίwGi­&³8Ψψύχ2α°ΓdχΚ•΅"΅:v”ΦΧ^k«σd ¦š½έΛ+dβα‡ΛϊoΎ ¬v=/Ϊ³Gf]}΅,ύονηΜλ]LϊΚd[=cŸdxG M³‘Ii+EE2mπ`ωEW‘9‚$νtEΰLσϋΨτ³ΟΆ­.Τ6›‡5Z·ln{Λχ’ήΈz³fΆϋλΑ²^βΒBW}€+tΕZΊωBX`ΩϊΣO‡ά¬`‹E‰€ƒ¦­Ϊ¨‘mΨΫfΟΆΧ1έgx ­.–Eζ½μΫϊαΐŽƒk±|Ή €@™"΅jΥje~ Έ@€X΄h‘lܸњΛν·ί.ωΛ_€ŠGŠ’€˜,“@@ ΒšΦΛ³ΟΚ΄Aƒl=ο5«ΡΎ5)³:w–Ί½zIݞ=­ύ—ςωΕJ §©οœ«Xt¦žfΫ‡*5jΨϊςθ‡ρ|qμ)4+ΟΏ?αipμ±R§[7Ιξή]j›ΥrΕ&˜ΆΛ¬N[oM.7+Σw-_ξλΚ³΅Y­^―wo1OώθφKςYυθLMΈΐ€„\ϋΩg’Σ―Ÿel9οή-[MπDΊzyφ¨{…΅:΄DήXΎ—t ™­ZΉΖ£+"?5‘Zζ½VΥŒ5₯κA#FΈΪ•·"Υ¬Ψn:dˆ,yκ)οLšD]Υ‹I‡”΄4ifζe₯m 2ψζ ”ή2Θ<« ¬y²PιΠotjš-iζ?  P‘ΨkΥjρΚqO@βGΰ»ξ’M «-ζΐf_˜Q£FΙ΅AΎ)?£f$ €ď@³―!fψlύχ3`΄}fεΟΦ™3­ΗsΎ΄υΉη€V‡%6kca;~]~β [»}VšHgͺH[#ǁ6:;ϊq4©΄‡uκHo“bsς‰'ΚeΛlΎΧΤVY©5―m(%–ο₯Ηc­Ά,XΏή64MΛΈρΫo­Ί³ς2ZEgΑk$ή€I΄nmλ7ΝL:Θ`΅4³[³sΟ΅έ?YH™,―d”η1uκTΡ\€ϊ¨cώ–‚T„€oΥZYοΝj΅²Κq €Ι# _ΌγŽ;όΊώϋ%??ίΜ¨ΌΫ.¬Ό“gζ„) «‘Ίš4zΊκ,ά’©θΊ™eMCά[±σγΛ>(bVυ”©˜τ-―ΌRΊ½ς ) KΤ‚ΜgΐΩ=z”ΠͺδSdιΓ’ž•UrΓ€³±z/₯ιžo·άpηΨ>­kR"Φ HnΓU]‘vΘ>τP+₯«—f“N4œχ‚WρZWΖ7ŠΧι0.@d(Ο^k¬VKφwσC@ 4λ―Ώ^Z΄ha5^gφ……”?‘„V ΫLͺΊif EζΓ{ ρ.PΣ€ͺλmΧ9‡όίΕ|θ-̞PΗώφ›•φ/%„΄ΚιζΛϊLjΗc͊ρζ&Nι`φϊ:rΖ Ι0 œΛ¬ QΙ“εP³B=5„1†Υy6Φ°‘ ˜2ΕJχ™Ρ AΘ3¬mHv3FΊ›€iz­Z!_ηk«χRϋΫn“.ϊ^pμέηG΄z₯{LΟΞ–F§žν[Ϋϊ΄ƒT½J‹Λ/χͺNŠΊ”}¦$ΕL˜ €@₯lΎρςΞ;ο„5_]­6ΫδΦUo@ Pΰž{ξ‘7ήxΓͺΊοΎϋδœΏ9ΨΟ@ ρ^ύu9‡™&M¦…l£<‰§ΕˆˆŒΐ{ο½'ϊe7g9ύτΣEΟΕΊh@mžY΅šχφΫ’’ž.Η™k™Ν›Ηz܁€Ψcφ/]3nœl4A¬=kΦΘσΕ•ͺ&0S£];©ΉQΗ¬„J―]»άσέςΣO²ζ£dέψρ²kΕ Ω½v­›}Ώ2κΥ³φΙqšZ―αΙ'—šj²άƒIβŠvν’ufΏ:έo‹IρΉ{υj)ΨΌωcσ;Sυ¦M­Xšκ/«S§ˆIΔβ½TΈ}»δ›ΜlΫΝ£ΐΌw«˜v™ζKWΩ&@˜¬+¬Ό^ H9μX²DΖ·nm»…ΞΣΐ{E—’’"~Κ©fΥ«G’X‹„"} €ΔT`Μ9rΘ!‡˜Tξ‘7D€7ΣIq3@ˆ©€ώΡ½{w™i>(rι₯—ΚσΟ?Σ1p3*›@ΌΦjΎύ‘Z^u•ϊ쳕ν%aΎ$ΐΎβb<&Ν«ΙDM@°ίwœmΨ›UΰmoΌΡVWΡ ¬‘ ²"^Mξ‰ P.pχZΣΥjΊ"…‚ €ψτχƒ‘#GZ‡U«V•ϊυλϋNρ’TΐJωxΦYς•Ιb‘χΦ[β ͺ₯ddΘ{/&ιτ™I-RΦύΧ’Z…Ι!ΕΟj:y¬-.Ή„ ρJ2H@xΨ΅r₯¬6i`K³σΞ“ ³w\<ύ2];“ŽV‹¦‚ŒT‰\O‘ύ € ‚€ZΣ€Yi…Υj₯ q@Δ=z΄΄oίήzŒ1’LΉε–[ͺ•IŽ‹ˆ ΅Ξ­}ΤΌ‚j¬V‹Χ‘"€Δ―ΐM£μΨ·¬υ΅ΧΖΝ€5˜¦{)λcήΌy+Φ"FIG €±ΠτŽΊOCI{­±Z-Φ― χC@@ b¬=Τξ»Oς‚¬P «Υ5xžΫ-’Ÿ=φ-ͺΩΆ­t~β‰D˜cD$(άΉS–Œe›M³ΧZmσ…–d/Φ’ύŽΠό/^lΟEVo­[·–΄΄΄υL7 €@Ω|«Φ4%d°ΒήjΑd¨G@@ ΉJKωθœ-«Υœ"'‚ΐή­[eνΗ»†Ί»kWW €@€Šφμ‘4³/±―θρ΄3Ο”‚ύiΩ}υνoΏέχ4©XKκ—7r“λΩ³§ο‚υλΧ‡½wAδFBO €€]@gΑkGq{«ΩΉ8B@@ ιΒY‘8yV«jπ@ΰ_x T­__j˜Υ±»σςdστιRdV¬–Ί}ϊHύ# ¬JΪηΦ’φ₯eb €@εΠUk‡~Έ|χέw ίgΏP@@@ 9ΚPS V«%η{‚Y!€DG xο^Ω ~"€@iψΗ?€]»vΦc̘1₯5η<TββbyρΕeψπα•`ΆLΔΠ€Ϊ΄³Ξ’―L֊Ό·ίσΝΊ2 žΥjebγ"@\Υ7–~ί|#­―½Φu.™+X±–Μ―.sC [nΉEΦ¬YSB N!X»vνςΈiΣ¦2bΔ1OHkΉFϊ˜|ε@θ θžG}΄Μ™3G222δœsΞ‘Φ­[Gχ¦τŽ% Db…Zΰ X­¨ΑσD¨iφ7κύΡGaW©]ΫUG )n/Ό ›¦N•‚Ν›%=;[²:u²ΊηZj•Κfͺ|3ŽΤ;‰~HpqγΖΙΒ… | o+VΘλ―Ώξ}’ZT`ΰΐΦτ΅cΨ =:uκH[σΗΌ–œœœˆά¨ΎΩ”½φώ' δŽ;ξ7ήx#"}Σ „'ι€šήΥjὴŽ?tσoTΓ“NŠΏ1"HjΗ+ϊHΔ²hΡ"°};ψ+Κψ„ΐZαΈ @@@ b.ΏόrΡG€ΛΘ‘#₯wοήV·o½υ–άtΣMrΨa‡Eϊ6τ‡A’Pσ݊Υj> ~"€ όEEEΦΆ:ΣΤΤTΡγHk‘P€@@@€θΥ«— [ςL0ΫD³ΛήQiWšΎ·›”WΡ=c֍ο:U΄s§«N+φnΩ"λ>ϋΜuΠΜΕ«μ5ύ―ύτSΧ)νΗ«hύڏ?v*Ψ°ΑU§Ϊ~ΗΎ[Ω=zx·ίΊU֘_s–μnݜUΦ±φΏϊΓ]ηjwνκͺӊ½¦ΥοΏο:W»KWVšφ«ή{Οu.«sgWVθ{f•`₯–Buz¨ύ[ο/ΗΉZf ―ΐ—Ž?οΝ7­E‚BυuΟσH\3H TΗΏrμXW5Ϊ΅σOa~Ύ¬τΨξ@_/k.F*@¨ €Fύά )¦M›&š‹TΊ@@Hf6fΖΠ‘CύSόϋί.………ώcž €@δΦυ•δ™΄«Q ͺEnΈ%χδXTrcΞ"’@4Ξ!f € °bνO ž!€€Πύ$šΤ@Šxϊι§eΚ”)ssξŠ `Έϋξ»εε—_–­fεΒ<“–mτθΡrυΥWΫΪp€•S@ΣΔ&eI”ΐ`”D„ή<ΫΝΰ§όε/Vo5LΟΏ?‘žι@ $k%ιpJ( ιmΪ›μ@Šψδ“O¬Ε˜Ύ~ύϊ’+S΄ψR_ΗxάβT ''GnΏύv6l˜όΕ|p9`ΐ€8)ΓB ±ͺζ折=€©TΆNά½6I@ χ}₯).})@ƒ₯医׎!€I @*Θ$x™ € €@Ω0ΎϋR^2€μq%$₯ΐ7ήθίc­C‡I9G&…@E ΤλΣGLŸ.Ή'υ‘Dj₯Y€ϊ‰ϊ„ƒέ Ȋ―`ΝγžρΗέKΒ€@Κ$ΐŠ΅Κτj3W@@@°ͺU«&ύϊυ λ#€@ψuΊw—>}$›M€ν·{ο•΅f4J£ύiσœ}gtτωμ3g΅€Ωg>«S'ισι§f[8{JȌ νkx τώψγϋ―iω½G¨ύΧ4™gzηξ?;ΫU§5Ϊ΅“^~h ΌGz°φfuΟχίwυ¬}f«Vσ½χ\ύgιΏz‹rΨ;οΈϋ―]ΫU§Υ›7—Γή~ϋs―A• ν«5m*=ή|ΣΥWzφΥ΅ύo„ήγΖ}ΜwYY:­¨Φ¨‘tύυ?ΟνŸC• ν«š-;Ί½ϊκŸν=žeΤ­λQK €@4RΜ?žφί’qϊDΈΠt .tkώόω€‚t©PΔNΰάsΟ•±cΗΊnψΊωΓ[ΟQ@ώΨ²e‹lάΈΡͺΘ6ΦΦ«WοΟ“ώ8jc’c@HNkΙωΊ2+@@@( deeΙ?ώρο=τlήΌΩΜ¨8plΏ±ΧZΕ½Xά@ °– /ΓF@@*»@vvΆ΄jΥΚzΤ«W/¦W]u•΄kΧΞΊ§Υ4ΈFAψπΨLΚΦ’RDn™>]֘•n@HN%K–ˆο©X‹”$ύ € € €ΔTΰŠ+πο‘xϋν·ΗτήιιιςΟώΣOM©i!) _u{υϊ#E€؎?ήspμ΅ζΙB% € /PTT$­[·ΆmΫΆΨ|¬EŒ’Ž@@@*“ΐ Aƒ€wοήΦ”χμΩ#wάqGeš>sE ‘¬ΫgŸIΫζ~΅ζ@E€ΐZ(J΄‘ζΝ›ϋS¬€₯₯!‚ € € `}τQΏΓoΌ!Ώώϊ«˜' Αlμ΅―#B@ ^¬Ελ+gγš1c†?ΕJ:uβlt @@¨>}ϊΘgœ!x |ψᇱcNJwE°|Ά#&O–Η'›§N•΅ŸV4F@Κ)P₯rN›Y#€ € € Ρ£GKVV–έ#2žτ‚@,κ™t}M@m£IΉnόxΙ ²[,ΗΔ½@@ Ύ¬ΕχλΓθ@@’,““#-[Ά΄ξ’ŒS@pΘκνˆ? °ιƒ‚ €”&@`­4!Ξ#€ €$΅ΐƒ>(ϊ  € € € €₯ °ΗZiBœG@@@@@@ΐ°b· € € €@B lΫΆυDQ@IDATM6mΪd½vνΪBJΖ„|4 € €QHIIρoύšΉuf‘λ)*Σ¦S@@@ΌF%­Z΅²Γ‡χnD- € €TJ ¦-Y²Δzόώϋο3 °1J:B@@@ΐ.°zυj{G € € Π€‚Lθ—/vƒ_±b…Y7lήΌΉDrΩdμfΑˆ¬ΐOύ«l›3ΗΥiλk‘¦gŸνͺO֊₯£GΛςW_΅M―γCIΞα‡Ϋκ8MΟМh… €@Ό ¬\ΉRξΎϋn3fŒΜž=[:tθοCf| € € ‚΅h"΅kWΩ°aƒE±~ύzΙΙɁJ-°Λ|P²δ™gDŠ‹]ϋ +U`mη₯²q$›CΑƍΆcBΐ3t+Z"€ ΟC‡•>ϊΘβίώwωπΓγyΈŒ n½υV9ϊθ£ε„Nωšh5\jώyδ‘GdΨ°a¬Y³hέ†~@@›© m €‘ ,ν5Ο š^½iςdΩΑœ½‘¨ό­φ˜ΰωφE‹l’έ»Λίq%νΟJϊΒ3m@φ <πΐ’›₯k7nœLr|i3~ p'žx’θ£W―^ςΩgŸUΘψ5 vε•WJϋφνeΧ]Υ*δUΰ¦ €T^k•χ΅gζ Pε/½TβΥ+4π–`ežωπη‹vνl-3f$Ψ,βgΈxΖΟkΑH(M`Σ¦M²lΩ2λ±}ϋφšsI K—.rώωηϋΫκ*Ÿ}ϋφωy‚@’ uΤQrΈIϋώΓ?Δ<ΐP{ξΉη¬¦ξΊλD₯dά €  Φτ…cΨ Pq§L‘νση—8€D ¬•8!N"€I,pΗwHΛ–-­Ηk ψň$~i˜ /πΰƒJ΅jΥ¬yL›6Mήzλ­„Ÿ@@ξ½χ^?D,lΞ€Ϊή½{­ϋkπΊM›6ώ±π@ˆ…{¬ΕB™{ €@R ”ΆZM'»Γ€TΤ\½ή½“jξ^“©mφ`l~ΡEΆS™Ν›ΫŽ9]ΟΠ­h‰ €@Ό 47ΏέpΓ 2bΔk¨Θ?ύτΣ%###ή‡Ξψ(Q@χXΣUkί}χΏ/ΐΦ³gO+πVΪl}Ν•GξΏ:׀ӟ|IΦQ“Αƒ₯ΕΕ[Ο5 6|ψpyΙd ιa‚i·ξo―?RMͺΥώζό²_”—\p揧yοΎ+^»5>γ Οφ«>ψ@–½πŸύ˜ώ΅4:υTiyι₯ΦοΆΚμ›ΈάάΫYžrŠgϋΥϋŸ½ύ6<ωdiyΩeΞndΝΗΛ矷ΥkzΩ܁½Ϋϊ©,s΄Χ‹sΝ^x-/Ώά֏¬5i<ύk}ξρΗ{·?^–­Mώ,f< Ž=Φ³ύΊ/Ώ”₯ΞφζΚϊζ½ΣκŠ+ώμc³υ_-KΜ*Dg©oVHzΆζYκΡ>gΐie„:Λϊ‰½Ϋq„gϋ ί~+Kϋ_g7R―_?ΟφΏ^–Œεnoώ;ρΟFσžΧφΎ”ΑΎ λšΟZ]u•οΠsΣΤ©žύΧ5)Y½Ϊϋ/δ  €@T¬E…•N@ YŠLώώ•oΎi›^Fύϊ’m‚Kλ>άVΏβΥW+E`­‰ωΓT”ΘΰGzA¨YYY’E k²uκΤ‰ΛI6LF›—7nά(‹/–§Ÿ~ZnΌρΖΈ+ƒB ]΅vΜ1ΗΈ. 5ΐΦΐ\ΩΥwυΪ΅²φ“O¬£¬ƒ–ΐ€šoušΆοξk―?MjΥό $ΏGΐZσζΏ·5}δ?φ=©Ω‘ƒο©νη³Oφόr–šmΫ:«¬γ¦Υ&Έζ,™­Z9«¬cΟjΌs–κAΎ”¨γYύώϋΞζR­IWVhϋU&˜θ,U6tVYΗ:žUοΌγ:WΥό}λU΄}ήΫo»N₯›ου άιxς;λΕι΅k‹xΦtŸςΌ7ήpυ_₯fMΟΐšŽgεΨ±φiΥ«{²τυZωϊλφ©ιιžν΅―L4)iiήνΝψυ3―βX³ϊεWσβΒBΟ@™Άχ οΩγΩήΥ1 €@%Π­΄θ—τ‹o‘(€‚Œ„"} €@₯Xeώ°)άΊΥ6ί¦ηœ#Ν/ΈΐV§y&ΥOqA«ž @@Θ\eΎΥ―ΐλCWƒΕc©m>DΎϋξ»ύC{5Θ―ώΡ@ΙofŽΕΟ<#bς;‹¦žΤΗ–ύί†žπΓΦ& ζΈXσρΗ2ΛcΣθΐ~ΎώϊΐCΙ9ςHιχυΧΆΊE>*σzΘVΧΣύŸzͺΏn½ΉΖΩ—œh‚‚Μ ·μέΆM&υλ'Ε»wΫ.=ΖK ¬ν0΄ΕO>)˞^φOg)0ΑΏνΑF=_Ν Z^z©΄ΈόrΙlΦΜy‰8–žώ›z<)2& yDŽ!E;wz΄Ωk‚½ϊΨ6gŽόώοKMˆ=πΎϋ€‰ r…šΚE m―»ωοατ€ΐsή{οɜΏύMv™•žΞ²cΡ"«Jnso½ΥΊο‘Ο>ϋΗFζΞΖ#€ €Δ΅@8«Φ'’IQ—V˜ηΊ*m΅£Ξw¨νσLgΝο ΎL§_,τ*Ο8Cτ‹wΞ¬}#“i€f»vΞζΦ]•¦B·ΘτΨ«₯fΫΆ^Ν₯α_ώ"™-ZΈΞOɈr˜ω[ΔYj™oξ 'ΘaζoOg©d< Ž;N3ϋ‚;K°ρ48φXιρζ›ΞζA}κ}΄τxγ ύΆνš`γΡ/ύu3ΖΦV‚'Η|©―ϋλ―‡άΎήGHw“QF‹~ΠW‚υ_οπΓ₯Ϋ+―ψšωk_·OιφςΛώvΎ'ϊεW―’_2μϊKžΪ?¦`ύΧ1_JνϊΒ Άί,X†T €DT€ΐZD9ι ’U`Γwί‰/ΰ›cNώΆaΊB«‘ &iϊΗΐ²Β¬t+o`mγ”)2Υό‘§ŸPΛΎ‚YςτΣV©σώ#­J ’…ΪoYΪιjΎ97ί,ϋφξυ_«μVψ‘΄Έδ]¨Oτ:gPMΡ•zΑΚJσGέt „z%ƒ]£υΊ’oήύχΛΒύKϊ~ρ…Τ3ΕkΩ6wL1°λ ΌpŠ&›>dˆ,78κ³ΩΩα\nk[Έ}»Μ4H (‡T̐yζ΅ΩB-ΪφΔ /”CCό»FPΑ‚P^χLψφ&€,¨η9ίxko―W΄}’<_―ל:@Θ €F+zB’Wΐ΅ZΝL΅ΩΈ&¬+~œESαiΪΌ²–u_~)ί›o†T Ό—šϋιͺ«d…Η·ωΫEσyΥϊυ­”ƒΞ{δ½σŽ³*€c ’9Kσ‹/vVω7Lš$3τ|˜A5扦0œl&n aΓυΐλbυ|Σ΄i2Ι{Γ ͺŽoν§ŸΚ³g`Yί―Ί‡Ϋ”“O=¨psχ&πΉ――Q@wτPi~ΡEVΊ’ŒœΩi‚wω&’Gtυ­˜1ΜΎξ:i`ΖWΥ΄Υ’ιƒQ›ΝFβωζ~%Χ€·Τύ|₯ցϊž†ύSΗΊΖμχX4hΈwλΦ°RlΪ$λΜΚ±ΐ’―‡ξΥεUΤDWϋi€1°θ5­ΎZjwξl₯ˆΡ΄,Ϊχξ5kDχ^[φβ‹²υ§Ÿ/±φ’ϋή€L9Α¬bK«ZΥv.֞7Χ—|-4)2mΕ€ΚΡ/ΝΞ=W²:u²dξ5ϋν6ϋΙi°QΖΞ@œΜœjV^φŸS‹Ÿ|ΥΌν7Jύ\υ^­†•:Žo>ο1i5 _E―€δΊργEΣ†Rv―]+λ'L°5­n>H«ΤQΆ:ίA±Y·υηŸ}‡ΦO]‰NŠC―”ŸT›Ά;Δξ`ŸY%ΆθρΗ]7Τ•bYa€νΤΐdcΤ¦ΊΒ1ά’έ­›τ4+ύLͺΝΚA<"f½ΐ’«γu»ΐqς@ς ™ίχώmVΦo5_(Y΅j•<Ά?«@yϋεz*Rΐ·j-’c`΅Z$5ι @"!@`-Š• <³Ί+¬Gq­v©ΜL1V˜Tzb‚₯ι!’j6ΞVRΜ²b― …΄τŽ‘”ΥϋŸμ2ί¦,υκΙalΦ­«‘t1gΩόγΞͺ˜Χ5ιkšΌο₯xχnYσΡGUAŸη½ύΆ--£6Τ½Ϋt^Eƒjšn2°δτοxXκσκM›ŠΪ–=λΧVΨσ΅Ÿ}&;Lκ¨ΐ’{ό…“ZΣw큚QV½~Ψ«ΗΊšžš5”’bΎ5ΤΠ€uM·IA@ Ω233εώϋοχOσσ…“uλΦωy‚@’ DzΥ«ΥυΐΈ@H^kΙϋΪFtf]Ίtρ§XΩ°?RDoBgΔ™€ZφKQy­Ίr6ςJy¨Α „„RΌ#jz½t³wH8₯α‰'jΎ?Ϋ%ΫζΞ΅ΗϊΐΛOχΥ ₯δ½ω¦½™™[‹‹/ΆΧi@¬«y uΊwhQϊS]υ―© ½bͺo(«Εœ3ΧԘ΅ΜώrΆbΎHΞ ΗzfΟΎΪlλ’΄έΣΞYφXs’pŒ €—˜Uγφ[¨ϋΕέηρE—$:ΣJbHZc΅ZΏQ˜ € ,|ΙEOŠ‘#€‘Ψψν·²sρb[W΅:H²=ΤVηu {KU7cξ2+=‹€kdd₯―Tƒ O:©΄Λ\η«κ~bf?°b³‡‡―芦Š,ΝΞ?_~Υ•w+5ΰXΈc‡T©Q#θΠvˍŽ šλυλ'5Z·zMV­Dε)&N”}……ει"jΧnš<ΩΥwΛ«rΥ…Z‘i/ηύς‹­ω³[¨ο½PΣ”ή ͺ#€ž+ΨΈ1° Ο@HZέ@]Wͺ΄χΌώχΏrƒΩ+·}ϋφI;g&V9tΥΪ1ΗSξΙ²Z­ά„t„΅L˜~ζσ g gΛη΅#€„'@`-oΡBf*ρΊί—ΧϋDWI–§θ Gg)#ύo† HPHDaΓ†ωS^Ώͺ{kR@ Œ9ΏgμϋfΣο«τc<n‡@DΚ»Χ«Υ"ς2Π  €DA€kQ@₯KH|Όwί•’ύ«ό³1«Π&φιγ?,υI@ͺΓΐΆΛΝ‡Ά-/Ώ<°Κφ|―Y9α,99Ξͺ„>n2hΜΎφZ)2ι}eΝΗK‘ <¦U«ζ«ςάnVπmωρG±>Ρ ZΈ©.4•€BΫ`ΎΎyϊτΈMρh›hƒχIΉkΉΉ»…X+ΛήnR €@%θΦ­› 2DƘ/όdffΚ‚ €―I-NA ‘Κ³jΥj‰όΚ3v@’_€ΐZςΏΖΜΚ °ό₯—άWΙεΛέυaΦX{·-[&™fE”WΩ»e‹«Ϊk*W£ͺΠ€Xγ3ΞέsΞW4©{­5>ν4_•gή›oϊŸϋž4ΏψbίΣRξZ΅J~6ϋ•¬zηRΫ:θΎzωsη:«+όΨ+›YΞk^«"‹vν y©!·₯! €‘¨U«–4mΪΤκ*;;;]VXΓΝ ϊšζw€{ξΉG›l’A ¬{­±Z-^}ζ€ €@|¬\Ή?ίίώŠ2>!dαΈ ’W`ΗβΕ²aβΔθMΠ¬d[ρϊλAϋί»m›λ\F½zΊD―h~α…)KιL™ΩΊ΅δq„λz― νσΛBͺ₯V­*Ω=zH+“²ί€Ir̜9^έVx]a~Ύk ε}Ÿμ^»ΦΥgIϋΊS €@ŒΎϊjYaφEΥǝwήγ»Gφv-Μ—FEP-²¬τVΑΎUkα ƒΥjαhΡ@J(2 %š™/’λCߎTaΕZ€$ι’F`ω+―ˆIγ©Ij:ΘξΈΓ³»4“ώΗYΌV'9Ϋ$Ϊqύ£Ž’jζζ»Ύ5²ζ“’={$Ν·|eΫ―ΏΚ6Gp«…Y­–bRs–VΆ/Z$3.ΉΔΦΣ\˜bVWΥιή]²»v•,³*-³U+©aϊ3΅JόσXΕcΟ=]™Wž²;/Οuyz‚ϋί5!*@@b*ξͺ5V«Ετεαf (?ΎΜΊβ ΧΘkp€t}ξ9W= €‘ˆO#?gz,ƒ€¦"©Άί£ΤT:–KD`Ÿ ¨-ωeΧhΫή|³Τ2«žΚRvš΄σ|Πvιφyσd³Ω3L;Ξβ΅κh§ωv²•σ%ΝΟ?_<ό°j…f΅ήΊ/ΎF'Ÿμ―s₯45―Υnώ φ?).,”M½ς€γύχKΛ+―”ͺ Όw]F:Ξ)—;U©W`.‘\@T € €Δ\ΐ·jν»οΎ+υή¬V+•ˆH‘ΩFA·˜p–Β€=̝η8Fˆ¬΅Θz&mo³gΟNΪΉ116L˜ ;—. ¬]±Σρ‘‡l«¨l J9Ψg–/1i} Φ―·΅ΤUk^΅ͺ ΨΪιA$φvsuΝ.ΈΐXΣ!ι>h5gH]ιlΊΐ)­?^6πC`•€˜•h=ƌ‘&ƒΫκK:Π]<–   qΛSv.YβΊΌv—.:*@@G ΤUk¬V G•Ά € PQ,=ͺ(yξ‹q)°μ₯—\γj|ζ™eͺig)f…T“Aƒ\ύjΐΘ+h“m)ιιΆφΊ’kΟ† ΆΊPφnέ*˜΄Šο›U^ΎΗL”‘τ6Y;JΆcΥήκqγ€xο^λv[fΝ’ν Ψnέά€ ₯lύωgW³–—_VPM;ؐͺΥaVΤξάYΔΌ·‹ξθυž lSs΅w–:‡ζ¬β@@°|«ΦJΊˆΥj%ιp@βI€ΐZ<½Œ*T`o~Ύ¬zχ]Χš›UUε-MΞ>ΫΥEΑΊu²ξσΟ]υiΥ«Kφ‘‡ΊκWΌφš«΄ŠΥfΟ²}ΆfΉ'žh;θƒζ]d‚ξ'·ώ«―¬:ηj5έW¬ρgΨΪ;pξΛ¦νκυλ¬yΠϊ|³Η[<–*5kŠ\ \‘ €ζ½ύv@MθO·š}μςωΕvxλτθa«γd¨cR«6iΔzΤ4mQ@@ ςΊj­€Βj΅’t8‡ €ρ$@`-ž^ Ζ‚* )‹9Ι«·hQ¦`Œs"9G!U6tV‹¦ƒτ* φσ_ςΜ3’{ΐ…Sœϋ“i*ΔΗNQoΫτœs\+τςΜk‘%ο­·lχorΦYR%3ΣVμ`ϋΒ…SYμͺ+­bιθΡ₯5©°σšΣY~β gUHΗ+ǎu΅ktΚ)’^»Ά«ž ’MΰŸό§¬4«Sυ‘κQ@x(4i©Ÿ1Ώώμ±?žΖΙX(M €Uk¬V+Mσ € OΦβιΥ`, P‘Λ^|Ρufη')&byKJjͺg BM½·Χ€yt––W^))ΆjM‰Έξ‹/lu%hΫ5}dkR―o_I7«Ύβ©T5{…5<ι$ېVψ‘l0››;χ»kqΙ%Άv%ΤhΥΚuzϋόωΊ’*Φύ΅x₯G,ιšXžkuΥU&Χ¨ύύΉyΪ4Ω8yrXΓΠΝ―—ΏςŠλš—]ζͺ£@b'0Ωό›~ΠAΙ5Χ\#·ήzkμn̝ˆ’@°Uk¬V‹8έ&₯@­~'Ί]γψK‘IωB0)¨ΤΦ*υΛΟδ@ΐ'°}Ρ"Ωψν·ΎCΟf\Ή «­œ₯xΧ.Οτ“ΥrsEƒzΞς£IKΉε§ŸœΥγ‚-[dΖ₯—ΊκstΥω*Μ^lΞ²{νZgUTŽ›_x‘­ί³ŸάΟΧ]g««ΩΎ½ΤλΣΗVWAm³W³8SK:Οo2ͺ©šv²¨(°:δη±π¬Ωx½¦ΣΝ*ΐmsη†<Φ97ί,»σςlν³9Dr?ήVΗ €ρ&°έ|9$Οό¦m_VŠ·ρ†;žμμlYd~OΥ2~όxλn΄G žΌV­±Z-ž^!Ζ’Ί-€fΕq>ΌΆ”H„ω0F@ šΊ`Β·υƒώŒT!°)IϊA„πZ­“έ­›dx`Δζ₯«Εͺ5mκκ/X:ΘƒyD2κΧ·΅Χ}ΩΎ;ςΘW$mόώ{™pΨa²Ϋ€4 ,΅:v”ΦΧ^Xe{^½Y3Ϋ±,{α)6ι‡’]tΕZzέΊΆΫlu›_|±ν|iΩ]»Ίšθzσ‡/1₯fΑ¦M2χΆΫd’ βιže^eχš5^ΥΆΊXy4r€€š}ωΛ+dβα‡ΛϊoΎ ¬v=/Ϊ³Gf]}΅,ύονηΜ/]LΚ)]iIA@ ž4EbSσϋ•>xΰxj™ΖΦΡόώvYΐ ςΫΜο(ΕΕΕeκ‹‹ˆηͺ5V«ΕΛ+Γ8@H>TσΩ–oλ‡εΛ—Gl‚U"Φ!€ *°Ο|8±όε—]£δj5νάϊ†ΔΰΑςϋγΫξ΅aΒΩi!™Žΐ–¦Hμςμ³2mΠ [ϋ½f5Ϊ·ύϋKVηΞR·W/©Ϋ³§€™o¬ες‹l™5K4’ωΔΕvžοi‚JUjΤ°Υdz€N\ϋρΗςi£FRΛ«š _Άmε #/‹ΘσT“φ²ι!²δ©§Όϋ3:W΅y7ό³ΆΑ±ΗJ]ΜάdεΧ;ο”UοΏ/mnΈA²:u’τ:uDQ;/–Υ| k>ωDφψ/ΙlΩRͺ›λ6šΤ”Ύ²}ή<ωυξ»₯ώΡGK± Ny­μŠ•§5λ\α§AΑοO8AΤ‘Ž gwο.ΊŠOΗ»Λ|«½IͺAέ]ΏT΄ώλ_₯^οήΎις@*PΰXσoω ζΛNΊΧΪμΩ³εHσ%«φf%?DΘ5:֚μϊ7Ν;εŠ+Hδι0vl­[·–aΓ†Ωκ8@H.kΙυzFm6kΜκ ί7#6l(ι₯ ,ΊͺΗ\HK³=‘žcS“X3Λ§dΕλ―Λ·ίξΊ]“Šπ§Ÿ–ΩΊΜ΄σ•}惕­3gZ%ζ|i₯λsΟI­JlΦΰ˜c¬rλΧΫΪiZF_šΜσAN΄ŠΞ‚Φ48T=ΜεΪΊΪͺ»Ω7μ+“°Θ€‰ ,[~όQf˜΄š₯ ͺυ3ΟυζXΣλζ?ψ υΠUˆ'šΐœ³Δ³ „νψύwωύ‰'lΓΠ‘GυjΡggG?‘^K;@ˆΌ€ώ-¦A5_™4i’胂@2μ3γΌυΦ[Ι0ζ€€_ §ως+5?O@€ :’”/kδ'uπΑϋs‘n2i($“ΐς—^rMG9ΊΟY€‹0«ή’…«ΫfεP°zθPιjΎ₯¬«ΞΒ-šŽ°›ι»©Ωs«΄’V­š΄»ε–šEν|]“Ύ²fΰ_‹K.)Σ}k˜o κΞi™™a_ίΤμq7ΐμ³–i^―†'Ÿ,)ιιaυkΟΞf%δ&ΨgΎωΦ8ύΝ·…[^y₯t3ΑHR@ϊUx‚ € € € €€M ŒŸΎΩϊΰHX½f“ϋUο½ησV3Ή. ±BW­9Kώ―ΏΚ³-Xiaφ;φ·ί€‰Ή6₯J鋍5½a'“πΨ €ωωηλΦUίήμΫΡeΤ(מ]†QͺπJχ˜ž-N=΅ΜwlzφΩrτάΉΐ€E,΅˜ΰR½~ύ€ΟgŸIΧ^³_κ5UλΥ“^&ΕfZ ©4½ϊŽ΅g“ζςΘ3$gΐ―α­ΛξΡCϊOž,‡κkΒϋ+hGœ@@ˆ θŠ  €>όωσe’ΩΒω˜eΎ(IAˆ@Šω%ίccΠw©oφVΪ`ΑiYoΔ嘽Ÿ(‰- ϋ2,\ΈΠ5‰ωζ4φlp±ΔUލeΝΈq²ΡBφ˜Τ@{Φ­“ͺ Hvν€ζώG(I―]»Μγ.4©υ—υνζQ`ξW%+ΛZΉ•έ΅«€›η‰Zt΄MΖm›ΩN:Oέ?Mχ·ΣύΗt_½{Νωζ»cιRΡ=ρΆώτ“nšg₯§Μ1Π¨wI₯"<·˜1ωθ#Y7~Ό΅‡άn³‡Eρξέ’a‚„Ί_žfΤ•™Ί―΄4‘%͍s‘8χάseμΨ±Ž_7)cυ%ς[͞„;vμ°:Ξ6ΑόΜ2¬rό¨θB9r€άfΎ€ε³ς^“­<ωδ“rέuΧ%Ϋ΄˜ ΄š rκΤ©Q›ίfσeΚ fmg©mώ^?ʜ£ €D_ τeΡw@C@WOijΔ²¦G εVULΪΙ:έΊYPΪ'J›†Š>ΚZj˜=Χj˜Υƒα–ŠπΜ6B}tΈλ.pχ“ζΡ―Αώψϋί.£ΜŠM-Ο<σŒ\}υΥžδ € € €  @`-ƒ§ €$³{§%σ«Λά@’U X’™#}ΊœpΒ ¦MΧΉp+ d€IΆΛZ΄ha;φ:ˆη±y7Τ:―y]~ωε‘^N;@@ fΦbF͍@@@@ΐ) )φ·ΏΙ“O>)ϋφνsžϋxΓ† rί}χΙΈqγδΥW_•N:…έG° vνΪeν>ϋμ³`M*¬>žΗVa(܁$¨iΆ;θϋε—™₯geΉκ¨@ˆŽ΅θΈ+ € € €@”vξά)Ϋ·o·ξR£F Ρ%±4tΞ9ηXA°Fž‘‘a₯ξΥ½’σσσKk.Ί{χξςωηŸΛGQjϋPhcǎ•!C†„zIΜΪΕσΨb†ΐHrtMc~τΡI>K¦‡DF %%EκΧ―ou–––™NM/©λ‰Ž@@@ˆ‘ΐSO=%ΉΉΉΦγή{οαΉU$ΆlΩ"GuTΠ šξyώωηΛ{ο½'ΊΩξέ»­τŒΫΆm“Ν›7ΛμΩ³εϋŸhΊΐκΥ«{I―9ύτΣeώόωžηΛZ©)+uόρXβylρθŘ@@ yRSS­}sΧ­['«W―ŽΨD ¬EŒ’Ž@@@@ TΛ.»L¦Nκjί,ΎτKeΡ’EV*G ŒιΎiZο+ΩΩΩΉsg9ωδ“εΉηž³nόη?ύίHφ΅ΣŸ›6m’“N:IvμΨX]ηkΦ¬‘ΫoΏ½\}Dλβx[΄ζLΏ € €@,¬ΕR;ο΅qγFλ‚ϊ-ΑHδΌO` †Ž € € PN]m¨+ќEW©?^žώy+ν£σ|°γzυκYοΎϋN6lθjφϋοΏ‹ή3’EzS¦L‰d—λ+žΗ±I € PAΦ*>ΡnΫ‘C돚 ˆΩ( € € €”E@S8ή|σΝKυοΝ &Θ1Ηγ:jEϋφνελ―Ώφ ʍ9Ώ'_¨ύΆsξαW\\,W_}΅6«ηρ<Ά α¦ € €@¬E—@@@@ΐ.0lΨ0Ω³g­RχΏxσΝ7₯kΧΆϊ²xΰςΑΈ.έ°aƒŒ=ΪUjΕ}χέηjϊσΟ?Λc=ζͺuE<-Φ܁dΘ_°@Ύ;φXΧγ§kIφ©3?@ n¬ΕΝKΑ@@@*B`ϋφνώ”Χ»wοˆ!pO@ ̝;W>ύτSΧ|o½υV0`€«Ύ¬½{χ–nέΊΉ.δ“O\u‘V 4HN=υTWs j-]ΊΤUˊx[,Έ•A 0?_Φω₯λ±ι‡*Γτ™# Uβb @@ Έε–[dΤ¨QΦݟyζ+­W …Ϋ"€I/π―ύΛ5ΗFΙύχίοͺ/oΕW\!3fΜ°u£{°ιjΉͺU«ΪκC=Π}Ϊ4ΥdΎω`ΫWvξά)Χ^{­|όρΗΎͺ ωΟc«rάtǎΦ Κ_~ωE–,YbNσςς€nέΊ€IiάΈ±υ³oίΎrβ‰'JZZZ©w[½z΅Μ›7ΟΦNWjφοίίVΞΑχί/ΆK:θ ©_ΏΎ­.؁Ύ5Π­ϋX±BΦ¬Y#»vν’f͚I‹-¬€ηœsŽδδδλ"h½Χ|5ΰ]­Z55z^xAfΜ)³fΝ²ͺ>ώψγrόρΗϋΫπ@x °― cB@@@ ΙtΏξ1cΖΈf₯°ŒŒ W}y+Ξ=χ\ωϋίnΫWMƒS§N-s0Cƒ*Γ‡—λ»Ξ6<] χφΫoΛΰΑƒmυ±<ˆη±Ε‘<χ•‡O>ω€<όσ²eΛWWλΦ­sǚ6m*—^z©\~ωεV@ΚuΡώ }uΤQΣ?ύτ“rΘ!ϊ*–-[&‡~Έ­™κt₯•΅kΧΚwά!―½φš+0§Χ.\ΈΠκβΕ_”›nΊIh΅?μ°ΓJλΪ^v—]v™XŸh²eΛ–VέηŸ.W^y₯,_Ύά:φύΟϊυλ}Oω‰ €@ά €ΖνΘ € € €%dffZ+)t5E͚5KhΙ©xψφΫo]βW©RΕϊp=γ«U«–),,΄VΒψ~–g…Žσ³Q―^½\CΎρΖeΫΆmϊXVΔσΨbιP–{½ρΖΆm[ΡU•^A΅`}\ΉZq©{ϋM™2%X3Ρ•d^΄wήy'θ5%πΊNΣ©κj³’Šξ3ΨΎ}{k₯˜s΅›Χu{χξ•?όPt΅ΩΝ7ί,ΊB³Όεα‡–N8ΑT+oΏ\ €€—€ξ³λ{x/K΅²¨q  € € €@… hϊ=]έ {ξΉ§ΒΗΓJ˜4i’«Α‘‡j₯Τsˆγ ]τίώWΣΣm£\΅j•΅ͺΗVγƒx[Œ)ΒΊΎ7/Ύψb)** λΊΐƚ>RΣBjJΓ`εόσΟwς ΉyTθ Igρκ?°Ν< ΊB΄,ΰββbyμ±Η¬ΫΦ­[» λΉ¦έ6lXXΧΠ@² θΏνš"YΉΉΉeνΖu5  € € € i]±ζ,α€–s^[‘Η|°άzλ­!hΠ`Ϊ΄iϊXVΔσΨbικ½ζϟ/§vš΅χ^ΰ5Υ«W—Ώύνo’ιu%šNηΝ+_~ω₯όϋί–.]Ί6·žkΐιΈγŽsυεk8dΘΡΰg`Ρ}Χ΄ίpŠ¦Oόα‡l—θx d« ϋlΫJ7M55sζLke—νbs ϋΰyδ‘ςΥW_ΩNιͺ5MjΡϋ:Λ)§œβΣv“'Oφ\ΥΫ‘CkžΊrΤYΎϊjyꩧ¬} σŸΨNO˜0Aτ=φΘ#ΨκK;ΠT’ϊ߈―tξάYŽ?ώxџϊΠ _½zυ|§ωD έ<{l³T#€ΔBΐώ5™Xά‘{$€€~ƒΛ—b%0Θ–“aΠ € € €1X·n-ψΰ»ΉξK•¨₯Z΅j2jΤ(Χπϊι'kE“λD +βyl1dιV , 4οΎϋΞT lγ{+Δ}τQί‘gI{­y₯k 7€WΘσΞ;Οΐ'ΤΦ{:Σ\κκΉ3fˆWPΝw½‚ϋΏϋ?Οχσ“O>)kΦ¬ρ5 駴Ӓ’’"·έv›u Ξιψ4°¦>MΥEA@ ή¬Εϋ+Δψ@@@HpM›6yΞ ;;Ϋ³>Q*:κ(ko.ηxuΟ?_Αy.VΗρ<ΆX”vŸ‚‚ΡΤ†EW’Υ¬Y3°ͺΔηgžy¦λόΤ©S]uΎŠ3Ξ8C4`X~ωεΡ”‘”+Vˆ³œœ9α„؁šœ~ϊιΆΣ°{όρΗmu‘h qĈΊͺ3Τ±@/ŠΛKΰ5κ@@ˆ±€~xζK;€ίπ§ €D^`γƍž–%°vΜ1ΗxφNεηŸ.iiiα\΄νΏώυ/ωψ㏭,/ΎF;vμλ»ΞΪ―ΚWW?γylααΌ§Υ4ΈXϊχοxXκσ¦M›ZΏGΎΗ5γO°’ϋ–iΪΖ7ί|ΣΦDW‘izŊ¦ά·oŸ­ΩYg%ιιιΆ:=ΠvO<ρ„«^χP 5¨ζ»ψ»ξ’χίίwhύό裏¬™­²”ƒΑƒ‹) €$²΅D~υ; $Ώώϊ«•Ϋ¦›n’xH7«’λfΰwήyg’3 (I@S8y₯q*ιΞ!€„'lΕZνΪ΅ΓλΘ΄vξOvζg`’,}ψΡί_uεŽ3ΕίΈqγδ½χή«Π B<ΝηW‘?υ‹5/½τ’mέ»w·—v Ή­[·–ΦΜv^ί+Ξΐšβ %°ζ•ωήσέLχVs„;βˆ#¬}Ν|mBύΩ΅kWιΦ­›•ΎΡwώ·jΥ*iάΈ±―ͺğϊ¦‘#G–Ψ†“ €$‚© αUbŒ €@’ θ¦ΰšίΏeΛ–V0+Ψ/Ρfψβ‹/€oίΎr¬Ω:ί„Žφxι@H={φxUWv%CΡύ­Ž?ώxΧTnΈαΙΟΟwΥΗ²"žΗK―{iJΔ‹.ΊΘφwυϊΔ‰₯°°Π«ϋ uϊ^ΡτEWΟ-\Έ0°Κυϋ¬œ{ξΉaw‘iΟ>ϋlΧuΊj­€β•R§ΑΚ·ί~λ:₯«ΞΚZ:θ Χ₯ΞύΫ\ *:uκpΔΣ² μέΆMΦ›/…:›¦L)k—\‡$­@JJŠ•²YW©ϋΆ€ˆΔdI Eϊ@(·€²4 υΩgŸωlωΟ¬½)nΎω樀ˆΤ€šτ4˜X΄Ž‚ €ρ/ Ώ+άvΫmΦ@oΉεRŒΕρKμƒ ¬rΘ!aόK/ «½¦ά‹ΕΚ8]ύtί}χΙ­·ήjίSO=%^x‘•FΟv"†ρ<Ά2„u«νΫ·‹"[Ίt©λ‘A΅`«0C½‰¦oΤχF`Ρ4Γ† ¬²=χJYR`MΗο,šς…^pV‡tΌhΡ"W»p‚γš©„‚ €@,RSSeΓ† Ώ%΅ˆ“! €@Y4 ₯5_Ρ΄9Ί‚-Ά`5½/«Υ|ϊόD@"'PR`-ά»<όσa]’ϋœΕͺόνo³φž5k––EEErΥUWY{ψ¦₯₯D©½@IDATωλcύ$žΗk‹`χϋώϋοε“O>‘oΎωF¦OŸvŠΗ`ύzΥχκΥKΪ΄i#Ώώ»΄Ύo/^,šήΡYt/3_`Ρύΰ8ΰ€ΐ*Ϋs―ϋ‘ώa8©NkΥͺe €$ͺ© υ•‹ρΈ7oή,ϊ ™>"ΉΙsŒ§Αν@ Ξ|«ΦœΓτΨΚ»›3ε£σ>zι?4½ξA € PΩ"X ΗNχΎΪΊuk8—”«­Ξ{ξ9qΠt?α'Ÿ|²\}—χβx[yηVήλ5h5xπ`9όπΓ­/φι>f‘ξ›ζ•1Τρθͺ5g –+ €ΧυΎώvνΪ%ϊˆv)οΚ½hώ@ˆ†΅h¨&aŸνΫ·χη"Υύ ( €@΄J l•5ΐJ@MηΓj΅h½ͺτ‹ €•]@χXkΦ¬™‹ASκE³hκŸX9Tχ°Ίώϊλ]ΣΊϋξ»Eχ\«ΘΟc«( duθΠA‚΄œγͺZ΅ͺτθΡC†*“&M’9sζ8›„|μ΅?[°q8Σ@j τœsΞ z―X€?Υ›X ϊp@ ‰H™Δ/.SCQΐ·j-0%€sΎ›/Eδή½{M¬γόQξΌσNΧjžMeIA½`ΧP € š@ώύε΅Χ^³5ώᇬ•AUͺDη㉙3gΪx@4eΰώSϊ;μuΧ]'οΏ~¬†αyŸx›η€£X©{†]rΙ%’ϋ©9KFF†hͺΕ]»ŠJΣ}κ|H½_υKΜϊχώwΰ+š‚Rί7-Z΄πUΙκΥ«]i =φXΙΝΝυ·q>ΙΚΚrVYΗO?ύ΄θά"Uš7o©θ'DšνΪIŸΟ?w΅Nςš»R PnθόζZξaΡ €@eΠWI5Ÿ/ΐζL΅γ;οά8ήWου“Υj^*Τ!P9τέΎo[Χ¨QCτ›θ@Θ xΦtŚΔ2dHδohzœ0aBTϊ-­SύχD't’­ι| γƍ“SN9ΕVΛƒx[,4Υ£¦RtΥτo‹ϋοΏ_ΌςJΙΙΙ‰ϊt 5½‘Z»ωζ›ύχΦ mqq±XŸ””RΟkπL_kηΚ΅£Ž:ͺΔ}ΩτZJ| h-χΈγβ{ŒHrRA&ω Μτ@Dπ­Z uμΊ!|y «ΥΚ+Θυ$€~p₯{θγΕ_Lά‰0r@ Ξ ΰ9ΒG}Τ³>•Ÿ{¬κˆDΏ‘τ1pΰ@ΟT}ΊjΝΜ ₯ΏHΆ‰η±Erž%υ5~όxW@KW’;VξΈγŽƒj‘ξΕl,gu–8Wΐ9ΣA:Σ@jΐμ΄ΣN Φ₯ΏΎ~ύϊώηΎ' ,π=ε' €”Q€ΐZαΈ @ Ί± t±Z-Ί―%½#€ €*ΠΆm[ιΫ·― CΣ5~ύυΧϊςVh„ŸώΉΌέ”λϊ'žxBκΤ©cλCWιέsΟ=ΆΊŠ8ˆη±ΕΒΓλ½qωε—ΛΰΑƒΓΊ}yχΝkΠ ηX}€+ΨV¬Xac͚5ςν·ίΪΖtϊι§[«Ρl•Ί§ž³XsŠpŒ €@ψΦΒ7γ @„»j­α^3gΞΧ%ύϊυsΥ•Vρλ―Ώ–Φ€ΤσΞ΄Žϋφν“wί}ΧΊΞ+ δ…^XjŸΪΐk>Ξ ]HνotΧ]wI›6mlΉsη†Σm@ˆΉΐ¦M›D›7oŽΨ½ ¬EŒ’Ž@"-‹€«Υ"ύͺΡ €±Π”zΎ?”γ!ˆ»™'ξtΟ±ƒ>Ψ5]]φΦ[oΉκΛZ‘{νN:΅¬—GτΊK/½TtΉΐ’©Μ―Ώώϊΐͺ yΟc‹6ΘΒ… ]·πzoΊ9*Fν¨ πΤSO•š5kΪ.τ₯ƒτύτl€‰}τΡΎΓκ~jΞ’{ό•%ΈdΙ+HΌxρbρ=Φ­[':tpή‚c@βF@ηςmύΙ½S ¬ΕΝKΜ@@œ±X΅‹ΰs^#€ €•U %%%θJ² .Έ@tί«ς”‚‚+Νβc=Vžn"z­ΞyΤ¨QR΅jU[Ώ¨¨θΟc‹ΆM«V­\·˜?Ύ«€ Maͺͺς]q{ΖgΨΊ™γ›ΉsηήsΏ3Α™{ξ9@ RΒiψ‚·Z€ξ"ϊ?H~¨«Ίκ‚C{2i/ž;χέw8pΐtήWΑœ9s¨OŸ>tο½χϊͺρσέ»w'1LD£D³nαδ5`ΐSσoΎω¦©ΜSΑ’E‹4c˜¬„…X…ƒœ8q"IXS£Ψ ©_sΛ-·θ»_§κ η:i±#ή€* Ε{Γ 7XΤFQΈ Τ”•ΡΞO>1m»Ψ ˆ †‘ι½Δ:« ±>&θ t―5 j §Ρ.ΤΊ’=ˆ'S§N₯ό‘.\θ6¬ͺͺ*Ίλ»θ•W^ΡB%Ž?ž:tθΰVΗx°oί>’OβσΑOiϋ€#FΠSO=e:ι‚Ι“'kΖ‰@Βπ…[ΧhΥMΒ,~υΥW!ώ‹/Ύθj«°°Π΅―οH^3y–ώφ·Ώ‘§EΕ~φα‡&ρŠτdTΫΎ}»ή€ν_ ۘ››KΫΆms]³tιRΧΎμ 2„zφμιVζλ`άΈqtI'ΉyΦ‰ήgžy&έvΫmš!Ϋ“\yy9]{ν΅τK/™Ί‘ρwξάΩTŽH0¬%Β]ΖA@ Ζ ˆ,Τ†5x«ΕψCυA@@bš@jj*½ύφΫZ˜:«ˆ²ΈSrθΙΦ»wo*(( V­ZiΫή½{iΥͺU΄rεJ***ςΘaΠ AτΩgŸΡΟ?†5σσΟ?O#GŽ$ ΗM­ΊΝ;—d …ˆ‘ΜhX;φΨc5£λόωσݚΟΒχή{O3(Ι³Χ΄iSΪΌy³–WlΦ¬YτΡGΉ…DμΨ±#εηηΣΌyσ\νΘσyηwjΉΠΔϋςψγwσ΄“œœLηœsŽf°σTG<Ψ‘™3g’7lΨΰvΉε;kΤ¨Q$|²‰q[ z²‰¨z4pΦYgΡ•W^ιΦ@@@ ‘ΐ°–Hwc%―5x«ΕθΓ΅A $―‰Lš‰¨ωoΒΠš8D ]»vZ©I“&Ρ»οΎλ‘ΛςεΛI6DΌΔϋH 6Œ$l œ–Γ?œΈβ zφΩgVΕΤ4λfR6β₯υςΛ/SώύI<³ŒςΓ?δ2σ%bT:ٌ†5ΉξώϋοΧ61Ί‰aΎH8HOωε}εάsϡӌ©ŽΌηˆAπ„N 7Ίι§ŸH6»"sb †€€€@"@Ž΅DΎϋ;€ΔPΒΰ­C7ͺ‚@Θ–„u’-Π•ΰP]€€@\ΘΙΙΡ `Ο<σŒζμ SRR΄0}β©&F5Y4!F£h‘‡zˆΪΆm-κΈιΝΊΉ)’ e(α&Εh寜wήy$yΦ$Tι‰'žHςμ+Ί΄W―^–ΝHθΖƍ[ž³S(± ΠΰΑƒνT·¬#†? ίŒ– £@@bŒ k1vΓ .€$*έk-γ₯‘.ϊ  HtVNBBΚb‡ΌΌ<Ώqˆηδf“|’/LΝuδ‘GϊέfΈ.cβ“O>ζƒj7šu j`^.ώσŸ¬ει“ΕwΎDΒIJΨD Ÿψκ«―RΛ–-΅Kš7oNύο)33ΣW>Ο‹ρΚJ.Ήδ«bΏΚΪ΄iCί}χMŸ>έ/CΆ„‘|ύυΧ΅ά‡ΩΩΩ~υ‰Κ‘'Υ΅+Ξa:Υ­”ώ»zh@œ'Π€γzGW`oη™@(% «#%”O0"Μ’+   ϋ$wQee₯6ρJJOOύA)#xꩧ΄θ  QA`/‡ΠΥ·P)΅P‘D;  Žπζ΅&£b|ƒ€€€Δ'kΉ†dƒ€€@<hΫΆ-½υΦ[4bĈxΖ  Ž¨««ΣR?ˆ"III$Η‘ΦBAm€€8F@ gM›6₯R“&L0•‘@@@@@œ$ 9],X }ΓHŽΧή½{k[AA5lˆ©:'οM,τΕΟΙπ?4©šΒ܁Θΐ­#Γ9ζ{ΙΞΞ¦κκjmλ ΡD cǎ–†΅^½zE“šΠ@@@@@θΨcΥ6 @ˆ­ΝΈq\Šk@@BD†΅ŒχfΦ―_οCΔψ@b˜@γƍcX{¨         +`X‹•;=A Δ|πA*)) q«hœ!°vνZˎŸ~ϊizύχ-Ο‘bΐωηŸO………±¦6τΈ"ΓZ\έN μ˜9s&­Y³Ζώ¨ 1Hΰwή‰A­‘2X8μ°Γ`X³Ftizz:鞯©©©A·‡@@@@@@@β— kρ{o12¦NJ²A@@@@@@@@ΐΦ|Βyˆεœ αηλ3i’UP@ύ¦M3•£@@ τ`X =S΄      ΥΥΥTUU₯υ”––F²A@@@ ž ΤμέK;>όΠ4Δ*δc61A€4hΠΐ•ϊ!)))d@BΧRΘTBC       ΰ›ΐγ?N999ΪvΗwψΎ5@@@@@†€Σφς‚ΩJKKC6nx¬… %ψ pλ­·RΛ–-γc0€Δ W_}•/^ƒšCep8xπ e³σηΟ§oΌΡς A@ΒO`ΫΆmαο=€€D%Φ’ςΆDŸRεεε€Πeee‘ΈPBβ“ΐΔ‰©[·nρ98Œ @b€ΐ?όΓZ ά'¨ ‘"ΰιΫkΙ’%$@@@@@ ² 2²ΌcΆ·N:i±H7nL»vνŠΩq@q”<Φ%‡λ@@@@@@@ ΜτΘ!aξΝƒ€Δ¬‚6{ΆIΫΞ9 ˆ Φ"Γ½€€€€D)Puu΅¦]£F(%%%J5…Z ‰H†΅DΌλ3€x& ΄άOτ\g@@ΒN‘ ΀€€€D3k―½ΦςϊΕ_ŒfU‘€@X΅jMž<™ξΌσΞ5†     ΡOkя !€€€€€€@‚xν΅Χ詧ž’ XŽΈC‡4vμXκΪ΅«εy‚@4ψωηŸιε—_6©Ψ§Oš0a‚© ‹rsscQmθ   ΰΦό€…ͺ      ΡC 55•²²²4…’G± 4™?ΎΙ¨&“΄^x!Mœ8‘ΊwοDλΈœ%πξ»οZΦΔP|γ7:«zˆKεεεqιί‚w`X .p–€„r•-žd€Iτμ³Ο’ Ǐ―ΣΔC-999ž†‰±€€€€€@Ψ ΤΥΥQvvΆΦORRΙq(†΅PPD       ΰƒΐΑƒ©Aƒ^kvΨa4cΖ Ν¨ΦΌys―uq@@ ρ”―]KΏάpƒiΰ™]ΊPΏ©SMε(=ΦBΟ4.[ΜΜΜ€ͺͺ*mlΎ>γ     XΈp‘f,ϋκ«―hΩ²eΤ°‘χOqδš 4. P³w/mŸ=Ϋ4œΒBS @@ΒCΐϋΫ|xϊD«1H`Γ† 1¨5TglίΎ^yεΝ ΆrεJ—ό1xβ‰cμ€€€€€ΔΦbλ~A[(%PSSC|πfL›3gΥΦ֚4•rΦLXP    1C†΅˜ΉUP@@@@ HΒ^‹€€€„£ ΄   n»ν6š2eŠi΄YYYtΦYgΡ€I“hĈ¦σ(ˆI±£*4ΠxόρΗ©ΌΌ\Ϋ.ΎψβΠw€A†ΐΉηžλ«δ¦=z4͜9“$,δ‹/Ύ£š‹v@ : ΨΉ3:ƒV   QEkQu;  €€€€€€@¬8p ζqΐ€4aΒκ₯K¬zƒ@B¨ΰΌςΏ>ψ ₯4mJ}y$‘ƎΑƁ¬‚φώϋ&ΕSrrLe(ΦΒΓ­‚€€€€€„™€δ4«Φz‘P©©©aιqνΪ΅τϊλ―Σίώφ7Ÿ!cgϞΠ(€@θ 衍μYΪ aC:~ύϊΠw‚A ΔΔ€–;~|ˆ[Es  Ÿ$Š„žϊ!))tCΧR|rΗ¨@@@@@@’”ΐ΄iΣHς—Ιvϋν·‡TˊŠ -Œγ˜1c¨kΧtχέwΣ‡~>Π€€3Δ Άψ²Λθ³nέhΓσΟΣA6wΊβ jΤ¦3 ‘W°cšžϊ‘¬¬,d}ΐc-d(Ρ€€€€€€@¬˜7o½τKτφΫokαΖρ̘1ƒN9εcφAbˆ€ΡCMŒiΊ$₯§S·[nΡρ    ^ ΐ°ζNκφοίOΤuΧIύ~A@@@@@ Φ |ώωηtΥUWњ5kLCiΘ!βƍG—\r‰ι @’Ÿ€'ƒš9ΌΥtψ°C†5;”P‡:tθ@%%%‰ββbjΡ’¨€€€€€€@άhΥͺ•Ι¨Φ«W/š8q"]pΑΤΊuλΈ+‰Bΐ—AM8ΐ[-QžŒ@@BG†΅Π±DK      1J _Ώ~4hΠ Z»v-}φΩ4i$2dHŒŽjƒ@b°cPΣ Α[M'ίX!PΎnύrΓ &u³ ¨οc™ΚQ  z0¬…ž)Zˆ!ΥΥΥT[[«iœššJς  _$ϊFNN₯€€xΨoΌAωωω”Ξω– ±GΐƒšŒήj±w‘1Q͞=΄ύύχM(r Me(€π4‹VA@@@@ 6\sΝ5$9de{α…bCih  ΰ“@]]}πΑtϊι§SΫΆmι}‹IH΅‘]»Β¨¦BΑ1Δ1¨-Ύμ2ϊ¬[7Ϊπόσt°¦Ζ–ΦπV³… •@@@XŽ«Α!€€€€€€@μX΅j͘1ƒ^yεΪΆm›k R&F6€@όπΧCΝ8rx«i`@@@ΐ0¬ωC uA’–ΐ’«―¦²_~1ιΧωͺ«(Ο6•£@ΰwΨ;gO<₯ΧP‹‘#E؁¨$ ‘322θΰΑƒ΄|ωr>|8-X°ΐRΧ²²2-μ+Β½ZβA!ā` jϊ@α­¦“ΐ/€€Δ7ύϋχ»(ί‘ΦBAm€8J rΛϊνΩg‰κλMzδœ90¬™° \φs؜]s纎e§zΧ.·c€€€@΄ΈξΊλHΆ+VPοή½Mjζζζ…^H'N€ξέ»›Ξ£@ Ά„Β &#†·Zlέwh   ππ’φA$))‰δ8ΓZ((’ G lzυUK£š(΅ϋΫo©|έ:ΚκΕQΡ9„›ΐ’-‰΅±Ÿτό|JnΤΘX„}ˆK½zυ’!C†Π’E‹(55•Ζ―ΣƎKΙΙΙq9f ‰@¨ j:3x«ι$π‹2y~cθ¬Y&ΥSš41•‘@@ <`X ΧΈk5==έ•Δ»Aƒq7> (Ά lš9Σλ6³α­ηέw{­ƒ“ λVέw­β ·aŒž7šαV†ˆW·ήz+mΪ΄‰Ξ?|jήΌyΌγ„"jƒšΐƒ·ZB=Bq9ΨT6 ΅=ωδΈ›―AdO“oŽ:Κc΅ΦΗOέo»Νγω@O½ύ6­{κ)—yσMjΔ^ςΔ!ΓZβάλ F*¨ˆF»ΎϋŽΚW―φͺ k^ρΰ$€€€D-… Œ3hάΈqtI'yΥσΤSOυz'Ab‡@8 jϊθα­¦“ΐ/ΔΙ©ͺ¦20ŽbίΚ•Τmςdjboυߞ{ΞkΏuUUF5° `XK€›Œ!‚@<πε­&c―X»–Δל“ΩC@ά δRϋ‹.r+Μhίήν   IΫ·o§W^yE3¨­δ 2‘ œΤ—a-’:’/π§AM4–Ιφƒ55–ΚWρΏ=Λ-<]΅nM½zΘtMՎ΄B―oˆμ“Φͺυ~πASύΕΕ–ν§΅li]ŸC½»Ϊ—Φυ‘ΚΉ–νsžδ·ίnκ7΅Y3ΛϊΥ»w»Χ?Τ~JΣ¦ΤϋLνT—–Š;ξ0•KψA«ϊ5{χrƒ>zτ£†[Χ/+³nŸλχΊ~SΏ5ϋφΉΥw΅Ÿ•eYΏΆ’Β­Ύή`CΞ»cΥ~νώύ¦ϊGrFυβh!ͺΤUVŠ;οT‹)™#@YΦgCŒZ_ΪOβPώ½ξ½ΧΤN݁΄RΪ?tŸτ ώΏΛ_ͺ&όo5οΤκΨcCΖβΐΝTόΥW!k ΔΦβγ>b δΕuΛύŸΫΨSω£€  v~ς‰[ωfžœaΝ @@#wΪi$™€δc’°Χ" βυ8‘ŸŒέ95<ΡύΑhΖ΄9sζPmm­›2Ÿ}φmέΊ•ΪΆmλVŽψ ΙFΕ—]F9ΜΏ'ΓW(F*aδvςΏ'VRΓ†Mμ!«JfΧ–†΅Z6μl|ι%΅:Iώ++Γ—V…Lυ3:wφXΓσΟ›λwμhYΏΌœ6°W*ιΌhΞRfώΫτιjujΔ†+CYš~{ζsύΌ<ΛϊbΘϊνι§MυΣ8\ž§φΧ?ω€Ή>6­ _υ< †Β—‹eNΐͺΎΜ¬›6ΝΤ~j‹žλOjͺŸΒ†JO†²΅=fΟ†G«ϊυl([;eŠ©Ύ­ kυΥΥ΄ζΡGMυ› ÚBEζ‰BiX+ϊΟˆψί€ $° ±D`λ{οQ-―‚3JώΩgSϋ .0iϋEo½Eς" •ΐœŸp?OΙvΙ%—¨§q  f»Ωk"'fOγ…³gΟv3ͺeggΣ€I“hξάΉ0ͺ…ω> yp‚@KξτJήNac—‘ΒiTsb|Ζ>%„]L τwφφΕ:“Ή’zž©¨!σIP `IJΗ 1C@V3ͺ"F΅¬=΄„Τ²‚M—j‹±ύ£¨ν)§θEψˆΝxυAAsψ& …5zτhš8q"qΖ”ΙaΊ ρG )ι6ή:ΘΠbΡhΰIg%TŸ α"P±n-8”cTΌ%ϋZxΑ…«οhm·†μˆgjΞΟ¬TmΫF%ΌΈ *x¬©Dp  φoΪDΕ_|ᦫԚL)Ό²ΉΝŸώδvN6Ώϊͺ©  α# αVBΉR0|š’e§ ˆWZ»vνθΞί³fΝϊŠs™\Δ9@}Υ$ddη©‘M ιτ˜Π?€€w₯|ϊή$Ψή^Ξ‹s§4=WYΜέ]αXΏ/~κ_³gm›5KۊΏόR§PΏ’8’QΤ΄!Ζsώμka λλέ.‘π‘Ψ"ЈσWΚ¦§€…φπX E΄ q›8g)/7Ɛrλ;οΈι΅σvHΒεTyι ‘HxΙ¬­`ͺ,*"YՈCerœόμž=)wόxJβά=‘–pχ+«²φ­Zε¦v³αΓ)™'€‹$‹–œ{~ϊ‰φ.^Lyb«/Η o}όρz―ΏrύΞ‘²σΣO©rσf’dαbˆIηI΅Œ΄\yrΣBψnnž,ωδΕ~ίςεTρΫo΄Γ’ηE‰§σσˆσΕΘo³#¨Ν 'hIΥ=΅₯—«χ¨jΛύ”λWξ1ͺ|p40ΐu^vͺωCL ©* Χ%ΙΆ?"νμ`―ΠmοΏO{~ό‘p2wΙ#!"ύJRν–GE­y|­Ž9Ζ֍ύλά\eόΑΩςˆ#\‡²#αuv͟O[9~ω―ΏjŒλ˜}ηehΤ¦ Ι3œη?kΊΈ]ˆ°X»v-UπΏΕύϋχχΪ‡ΡΔΈ–”δίΪΟ©όξqΛ-·hmίtΣMτ¨Eώ―γ$€€£$8β7Ό΅ΰχΓΗΟ<“Vέw•+ί!Ž*θ­sOkήΑ9ΠΎs9Χ\ρ矻ΪήφίRη±KNKs•²£†Μκޝ8ίyσiΧ€8@ ™ ¨f‘R†΅P‘D; %°I Ι“μνΞ?ί₯ƒΈό7dΟ5}b_NHrΰ’·ί¦Nœ;X©γUΡΏ>τ­γΌ<²BΜ“ˆΑ€Λ5ΧP§+―€NB,ςž²’ͺXwΰPGv$RύngƒΧβ‹/vSι86er’l‘Ÿ|’%―dΟA£8ΒΙX¦ξW±ΑeΕm·Ρ&φ Ω­ΜӁΖφ 5QλΡbΜΚhί^?Τ^¦sΩ0 †ά̞nΑΦφ°wΦ"~™RupunΨ©ΪΊ•–ίz«ζΩ5lφlΚfƒ‚κig¨ξuΧ©~U₯V?ό0­ψΫίΤb[ΗψΓσ—o€Ϊ²2[υ%yΉ¬4ΫΖμ »Žzς Φ†>Δ<5θ·-oΎIί‹±·Ξ“j–εβyΆκή{i čΰIΘζμe­RΆl}Η^™βηˆ7ΩχηœC›^~™{ύuJετ@₯ΆΌœ~bC€Νm „ρ½)]°€Fr¨έXlλZTπJ`Ώ£½ΔήμbT+ηŸuω/κΨΔ‹qΪήΥτsψ#μ΅ΪξΌσ(ŸίΓe`Ϋύν·DΌ K1Ι7»’ΥW#z»Vκwς±0Ξx½Δόωv•ϊ―ΈΒΨ„Χ}14uζ vE«—ΏΨ­NRΏΛΥWΫ―Ο†,Y”jWΔπUpν΅v«ksχέSb@λΚ =νŠVŸ=§νŠΠΊρ’?»’Υ?δ‘mχšD©wΏΉΫržΉ%ό<Λ‚.E5&ΓZ‘DARŒ˜Eς·A@@†5<Άδ³EII‰VW’Š·ah6[  ˜ΌΥψ\» .0Τψ}WΌaTڄ‰“r™:™κΫ)Ψυέwτ{ΓyσR³jG sG’1ς€8Υ―ͺκz^¨QMΒΊ¬Όλ.΅I{ΗΌm-―.άΙ‘FσΚwO«@ΥƜβ&ɍœ0Αo£šQ aψ-‡MΕƟ&OEΕώξE‹4ύ$όi "‘@ΏβΌˆ#8h “:τ»O€’―Ώφ[1ώΐ†ΟΡ|mΕ‹ΤοΖp€$0Y|χ0/Ίο4ɍ¦JΓ† iΏ;‰@μ0Ψ6σB¬ΥχίOε!ς`“(²@¬qŸ>vΥA=(& y½%ΝB«cΥθͺΚ]ωfτ7́~½2»wojΜ[ ‹₯υvΏ‘N‘QΚiΔ0( ΕΛω½M~«ω›>΅ys/ΩluΩκΈγ¨υΨ±ZΊ £ώξG²/Ρ-)8μŒΩ©”#’[€Ÿ;<© k‰t·1Vˆ΅jN}Ή‘Π ygœaδϊ’XΫ‚Ξ%<ω#ΖΆwήι*²»³cσω£Ξ°[ΏVΒnHώ¨ •(!,φ.YB₯ίOΖΌWφο[~9ρWœκWΥsΟ?ΠR?V―_Ε‘­Œj 8\ 686),Τ6 σQΙ^~₯l΄‘^Υ³­μηŸiΑ)§ΠαΑSˆ ½_§Ένγ~ΡQBEžΣNΌ‚4§_?ΚξΡCΛ#'/°’[Nr―mδ IynŒ"α+δ™Λ^lκx%œb{1ή’… iίΚ•ϊ‘φۚ'2%_š.’χ/"!.ησG‹z81Žφ.a|䃣Qnφ’WΕ!4ΔΨ(FqΥ»M^δ°w©eU«?ς_§՚°‘Nk‹ΌŠu7s‘π’ƒ*»ΩΠΎ=+όY9¬ΆcHtw/”Pj½zυ’‰κϊ^όԚΓ₯A@@ b`kΟ‹‘Ϊ±ΫfΉZr°ρwYPΒί„²θo{³@@bŸ€ξ₯&ήd’χ[™»ΩΞΗy§ŸΩώέΟyΰwσΒj£HH‘ƒ†”ΖσώμG:E†,0_Ξ)6Δ‹OυΒ½ε{YΆ²₯KŸsγΕ§Ω#·'GΣiδη{\$ϋέΓ™‚CΪχ$N₯}"ύόxbθε0¬%ϊ€ρƒ@ŒΨΚy”ŒyΣD} ω(a5TIβ\mO;MΛoe<ˆa­ΎΆ–~δ‰!Υ¨&† ήœk­ΏpH( UJΩ%a#χσKŒˆπ‘Ζ6œκΧ¨ƒΎ/!+³³ρBŒ—ς+†Œ$6’Ι*'Uv±Αdεέw«Ε”ΕΖ₯ΑΌϊΤ‹ Pž~š–MžLλ•Έό%_}E+Ω0ΪΗ"½ή‰“ά$‡˜›1—•jάΏΏ6ΦΖ<Ιh1<‰ͺΥΡGkaOŠΨ˜Έˆ?Œα#«Ω[xΟO?™BBΆ=šdΣe)‡AQ kέωΕΉωˆz•όΚGď]d2ͺ₯ρXγP«2£Θ%jKΞοΦƒ=ecηW3>K’{ν'‹3„'Kl OˆBp4:”ϊrψLOγ•ΠšΛ,B³lεπ0¬Ω¦·kωίψΊCa[Ε»F’C@μΪ7ί|C9œ»ςlΞq2‰C© "7ͺύžQ@ QH„PΨ$ŒxΏ›jή'‰γ8% k"ΉΌΐUrΡΉŠ!)Ú|'ͺ¨|έ°ζgΊ{€Sdhίΰμυk•γ^ΥΝuΜcάΐ9!eaϋαŸ|b;η}$ϋ]JΑαTΚs€ŸιbM ΙΊ₯  Δ«G•φ^¨ΉŽ%­*²ΒQΌXό1–ˆ•QΔ3M<§$Ό•QMκ6=μ0:’]ν[ιOΖKmο;Υ―•‚•œE^™ή•c»ΛΈΔΈ%ΈΊV'δ6Š„]p{FC‘œ—œ r½₯QνP‘OX&°^ΗI³ΕΣΛ“8ΙM<’ΚccΨcJ5ͺλθϋςΒίwΚύΠυ«”sp`η7N Ύ‹½ΟŒ’Μžf#ΏψΒdT3Φ‘}ρjθΙ…j’w>'yΟΔ€ˆ΄α±a`τ  Πσ,+Εn‡gžy&½φΪk΄ί<}ϊtΥάθΰ@ ”tΫ1+VΠ ^Τ%Q’C^k]‹‹@’Š€nX“Χ­9‚QΆψ‘"ΠXfg‹βΡ* Š%ςŒˆήŸvΤ:bxZΜ‹JMΡ_ԊVΗ‡Rd|ΝyΨν~ÊαI’ωeT3τ-iP$ŠŽ’|I$ϋ]B™‚ΓΞψτρKΚ‘y±Κί…σzŠ˜ςuλτ¦όώτσγ·‚ v k vΓ1\ˆeβή]Β“ΰFIc—t‰£νIΔKFκ¨’ζ^SϏ%jTΐωΖ$Œ‘/Iεp”Γή{O Mη«ρΌSύu°ΪΒ+·ϊ<ς%±W‡/ΩτςΛ.o=½nη·Θ†•†μνgGΔp™Λ‰ˆRΟ»΅S§‹\ϋNr“ΨΪ{9\₯Q4Γ«!­Βš–.X`l±}™\΅β.žbύ3ف=ΪZ„o]ύΰƒ~­Ι A4”=ύμΔΝCpͺbό•°!ώΌDϋ­ .$ œ|πΞ >ΖΪcΕŸ;—Γ‹q ‘ ΰf`γoŽ@ lšΧθ ±Mΰ GŸΠEΒA₯ŽΣ‰lη\kώˆΜ=Iͺ£Ϋ5φg¬γkί[ŠŒ–ω₯λΝ7Σ`Žβ2μΏ₯ώ<ίԞΏ›ZDgSdΤ)ι'Τώ%M…D†Q₯9GΎ9ŒS€Œα1JΪ‰£ωίΑaΜ¨ο΄i–‹Β%EΕβΛ/W›q;Žd_±·]Ώž yQώ6€ΐ©FŽζœš#>\[΄3`€›ήr §ΰπΕSκκ)GΔਊ€ˆ‘yŽŽ”Σƒ£τΘO Ω KΩύŽ;μVΧκ9Υ―7%σxEz‡Χ΄#š†_ΜTƒ€]£š~m vΫyΒΟJœδ&F5u%X‹1c¬ΤτX–Ξ/^jHΝΕΕλGςΔφΠ¬ΰΗF‘<†ν9LͺΏSΒA*²• ΠvWήι—Κ ³ό}ΩΙΡΡζΔMU%ά&@@€?ΨW­’Ι†9Ÿ_4ž½ίεܟqΨέί…΄ŽFFΎ5Γ>Ι&ϋΔ" Ψψ]Tσ`σΧΐΖή²ϊ Ϋdq­.ΝDRvEυ>3ž³ΪΧΒ@*'τόjZq9ΦΌ₯Θ8‚ŽdÏ,•¨KΉ'D9=Ζ Ξ>Ž£tώλ_mΨ[λPŠ Σ CΑ’«2₯3ΐ‘Fύ5΅;ο<-Όcz^žΆH6—Ώ“ 8½Δα<Ο2”ίqΉQφpdž" ιA"Ω—¨ ‹ί­RpΙιXϊύσŸΤaΒj6l˜–χݘ~γ(φΌ“Εβϋίm$z ·Bεΐ[Κ‘~¬ΟXŽς4”Γ χ}τQ-RΟ0ζxΟ%ΑμdΉ.ώzΊΙuNRΕInb+δηΤΈI8PDΌήό5.ωΣ~0uΕ ­Šό ΪρS―“Π˜Ωœ_ΞMψe‡Ν΄+ς<ετνk·ΊVΟ*ΖΦόbˆΚ ρE ¬¬L :œΓ υdογπ€Ž„tΤEήΑ>ύτSύ0κ~―ηUΙUUUΪφη½…€$&Υΐ–Ω΅«-’?¨lεJ[uQ @ J π»Š.’¦£’ŠCζ~jψ}Η¨†ΈžΘ*(p].yΗύ'Rdˆ±QM)!9θ:ωπ<“q΅eO«αΌΐœ'½ά†Ήώ™g܎υƒHφ₯χιD §RŽ8ρόθœγεW’qθ©2Γ{0c„a-zΈ@ bv}σ ν_Ώή­Ώμ>}Όζθ+Kή₯τvντCΧ―ζη:ςΌ³Ϋ" _KŽ§μ―ˆgδ#³+NυλIΏf<α–Ρ‘ƒ§Σ¦r‰w­J³‘CΥ"ΫΗ-8¬§*jx9ο$·L^…ԁ ¨Ζ-ΩΟ°X%Όz,ΠΠ*ŸP‹±T•Ž6^ΜΥkτc«°—Vύ^_ύmyΔj‘Οc5 \P½k—ΟλP@β•ΐYgE—]v-Pήwrss5ο5ρb»<ˆλγ•Ζ tΫ±l,Δl> lmθ#*ωςK’ΰΡβΡ ’‡+ΆAΦZ·6υfυܘ**$·›§ΆP ρF`αΒ…4—Γ2ί|σΝ^‡&yΎ±π2χzN‚€@Œ0Ψ8ό£qαή–7ή wήIΩπ±; uAΐš@>‡ƒ4Φr.s1Žtœ4Ιϊ.-_³†φ.^μv^Ϊ F¬R›"ΓhXέ$EF:ηE3ŠUΨGYˆ<Sl ηθMιωωΖκAνG²/QT Α;₯ Sp„:εΘͺ{ο5|ŠSϏOΕPA#€Px@’žΐ¦™3Ν:rβIρς±½q8#+Ρr·)^OΖz5{φ΅ύ` kv Nυkl€’\Uη`S^/S½Ό’…[εΦ­΄πΜ3iξΘ‘τ+―F’€Αvj’;0₯ΖΒȜa‘»ΠέS76U—gΗ$qΘ2„‚@C΁)!πdKNNE“h!°}ϋvzτΡG©{™6ŒnΉεZ»v­#Ί SˆFΊνXŽ€QΘί™™Ώ«)ΉΦΨk  Z<₯(žYEφΥ›˜"%IZώFœH‘!ϊJt˜μή½Mͺο]²„>ι…~œ0$ΌamE…©ŽΏ‘μΛ_݌υMΑαdΚ§ž#7μ{&5Οlp@ HβXYI6aO6 ύΆΫ,»°šδOΙΙ±¬k§°Ξf.'§ϊ΅3;uBρrf§Υ° ܊ήy‡~βUpͺ7§ρ$₯₯Qγ~ύHBδŸs΅5Šήγψh«ρH†`€jΗΣεΑό}™CΨ$πΤSO‘lˆE555ZN΄3fМ9s¨ΆΆΦm3yβψ~L»1Α€ˆ­ΓEQ{»‰C@ŠQm3{­u―΅nέ@ Ζ Θ"ΜάSN!γBνβύ””PZ‹–£SΓ@6γEJv£Y6Θ…N€ΘΠuιΟί7ίqž΄:%/šxοmβά“²5HIΡζ"šMβyΦ‚ffκMΨώd_ή” G 'SŽ8ωόxγŒsΏ€a O€@TΨτςΛDΒ8†Jq ιΙ°fεQγO¨:UΗ†΅Ž;Υ―•.”Yι/ντζ ₯—‘ϊ’kΥo$ξ—Ξ¨œ½~œ8ΡΊ”+4ΰ{1ž5),€Ζμ•&Ι%|ό&±·L΄KC ο2ρΜ Fͺ,ςwͺ« ƒiΧ‚€@ΌψΧΏώEwpŒββbΣP³²²θ¬³Ξ’“9Η@@ΐš€j` ˆ}ΖΡhX“(2[9ΥH§Λ/7 NςΑ—)yβσ‚ )XΝG„;E†>Έ–GAΓ?όΠΈ¦Χ9Θ ΄$ŽlkyD3΄I¨Κ6γΗS[/™Υ΅«^Υλo$ϋR w «΄ΑD²Rη±Τρ|~Œz`ߚ@τΟδYλR nΪ²‚F•‚uYL`@IDATo€μ=Τb[Ηϋ9μ£⣜_ JψA3z¨XMςKψΙ@εΐΝΆ.uͺ_[ΚΩ¨$Ζ³d^εT§„hyΤQaΝ[ΰ$·z~IWΌΦ•—»βΥ°½8~vΗΛ.σΈ2Ξύ‚θ<²JςΜί‚ŒΚ0ηiυ`tRV  ΰ,1žj Ψγy{>ObΟι3Ξ8ƒ2Xqμμˆό～÷ι^zΚα\ύgˆ+@xά!6ω…€Δ–ΗCe₯zΧ.Χ€Δ+ΝΚ°f’ί₯‚‰¨γDŠ £ΞbπυΥW΄μΦ[©„=φ|‰ΪvΝ›§mΛ'O¦f‡NΧ_Oy6XD²/‡Μ'ό|ν΅΄•£ω+’‚cί²eΆ.s*εH4/9ΘΉΩ~{ξ9ͺVVv‹Χšx©b΅ ₯CΎ6υZ«γ=?ύdUl*sͺ_“"A€΅lIϋΓZω―Ώ†Χ°Φ΄©IγHά/ιt租jIƒ 4`O΄Α―ΏξW\v1ΠE£€Z„ΛCu0²·ίL—η `*C€€€5Sy%q‡¨nΜ^ΕqH³ œ/£ ηΝH$yμ±Η΄\r2ζ›nΊIΛ1—HγΗXABK@(@@⃀D†i{ϊ鴁=όu)™;—ͺ8'm£6mτ"νW ٜσ₯§ηεΉΥρχΐ©ͺžM¦Q_|A»,ΠR‘l›=›*m~Λοώφ[ZΔ[ώyηΡaτ…Η©Ύτ’σςm/[Wφφ—ΘN₯?ώ¨Ν‰νψψcΪ½p‘Ϋ‚Z½½5ψ‡f˜“άkώH(ϋr"‡S)G’ωωρηώΗsέΠ•ŒgJ΅mΫ–$©l%%% a' 1’Υό\ι:P † UΩ£GSšβφ/u$€•HLiUV?ψ ZδσxΥύχ»yΎωΊΐ©~}ιeχΌδSSE<”Κ0UpΘΐ_}”φ³‡”ΎIΎΊ,‹ΙD§Έ•―Y£W3ʚ }lxα5œ;muOΧM›B[#©4’{I”Β!Ν  hζq.‹‹/Ύ˜ΪπϋΙΔ‰i.‡)’jΎŽ…Q3ŽΣγΞ;iΜόω4–ΓDvΏύφί#Φ(οόμ3₯ΔΏΓ`ϋς”‚cΟ/tΏν6ΫyνύIΑαdͺ–Xx~ό{β«6<Φβλ~b4 76ZL$΅γ˜ΞΎβ9ΫΠ@Vy&­gχw£ˆΡ§†έίU7ο&ΠoO?m¬JşN₯?ό`™—ͭ⑃½όα©φgUΟXζTΏF‚ΩΟaO?YVςυΧ4Γ“„b”όκ«”ΩΨ[Α/u««έjΆη\2V!β–Ω©νωώ{7ΛW―&α`WŠ9™°UxD»Χ‡»ž$y^ϋΟΟφΊΊ*]΄ˆvqΜυζhW%e§ΆΌœ6½ό²RJԁ'•!  (ΆmΫ¦Ξfrθλ5‹3r(κDΑq‚āΎό~;pΰ@’<wέuυοί? ν{πUϊRΌœ‹ϊ§K/ΥΪ‘χΚΌΘ@•]<α»ιί֊%WΚ²eΛhGAHe―‹φ¬‡*»ΎϋŽ6ρΏ±ͺHˆ2«φΕKCύ•οO UnUΏ”ίύΥϊ—䲬ΟߏVυ%„~ΗI“T55Λϊ……–υχp”«ϊ’ΓΨͺύ=K–XΧηϋiU_Bίo|ι%“ž™Ώe}Ύ7Z}%oD±ͺ/αψ¬ΪΟξέΫΊ>/Ψ΄¬ί«—eύ}«VΡ†_Ττ7Ξ+ΘbM+}φρ·œUϋYμInYŸΣXΦηpΝVυ%œέΖCϊπD‡‹kfAuύuλ,ΫΟδάͺVν»ΔεŸ}Άϋœϊ—ZΑuΧ‘DR)ηgΓ(ε(TβDŠŒ`u—άr½xaΈόm―°y°PŠΏ}9‘‚ΓΙ”#±ψό„ςωˆφΆ`X‹φ;ύ@  Θ ¦UΘΔvηŸ2/[5tΥWVj‘ Տ ¦‡¦yΚ©:-δ\mΓ8aώPρ&ςσνΈq~y«I{Nυλm,ώžλzΛ-ξ†5n@ςΩ΅=εΫ‘Άp˜ΉΖ(IQΧn0Ήφβ&¬RΒ(’·έ»Ω@΅ΰ΄Σ,C.Ϋτ΄o•{°jΗOΥ*ΟκΪ•Zσ³ΌγΓݞ?Tθ#·°œn”ƒ_8h•a… œnΜπ­?^©‰Cˆ :R__―u&‰‘«*2ά½ρJ»]V+‹'ΕkνݚCA@@ Ϊœ|ςɚ1νέwί₯χή{Nαwϋ»οΎΫo[[Ψ1ϊΰΨCbγ‘Θ ’ίZύ&“jςΈαωηυ+¨9οα-›#μXI/ZpΛ­t¨RχYΆ/υŸ{ΞΤ”€)πT·gŸ5Χg/:«ϊ¬oΟΪ">₯‡6γΗ[ΧηφΧržtUZszΛφ₯ώ”)juj}Β –υχsΤ }§J«±c=Χ·Θ?ΥκΈγ<Φυα‡Υζ©ε1ΗXΧί°~΅ˆœΣ‚#ΕXΧΤp41‚±±Θψ *^jbXSΏα%―xΟυ„JœH‘!Ρtζσsj”>ό¬Ιs$Ÿ­‘”ςo©.²hܘg=’}‰N€ΰp2εˆϏ~―ρλ›@’ο*¨ ‘%`εΙ"«4χμ2E΄«ό|S{žΒAφζdΙd”J72—_Π$οW½!‘^ηΐ]΄š_^Ύa―­Ϋ·λΕ~ύ:Υ―_Jz©ά†0m8ΌŸ›πφ"~‘[.^h‡&²έΞ:―¦ŸΨƒI[!eπ’Σ}Ωk*³sg«Λ΄2'Έ5α•£ͺHŽ9ρΠ“p^žDrο-cδΧΌ2Wr–YI•η'½];Σ₯²bŸ¦, ϊpHΞ€τt·3•›7ΣΧ#GjΉέN(uΌ²xρW˜'8x…ζž”oR8Aΰ/ω‹+δυΏώυ/'T@Ÿ H@&’›6mͺ<‡Γΰ^Ξ^Α Ω[b9―ΨΏ‰σ]ΐ¨–€† 1B@Ό}Đ&"οΉb\ΆΣΨ ‘ςqχc%Mύ[ju.¦ΛΌ|CΔΔΈb\oίp1ΑJΊΘΏYb6J)ΏsνgƒΎjXkyδ‘d~Οx­?ϋVι"#ƒ£ιTqd=†ό–pψΛ@$[IΏ‘’ν9(’}‰ώN₯ΰp*εˆΟO ΟI’^ƒ¬D½σ7D)1΄θα=Œ*†[MΪΥ^¬,Vλ”|υνg#*Ν‡§~«σκφο§Ξ=—f7n¬F–\uύΐ+ΜηrLξΩΠ±βo£Z^α¨K>‡³τGœκΧ}ΥΔαW2:v4U“Υv_r˜”Ÿy•ΨF#‘Hv/X@Ώρ*Qαψ‡˜± ₯!/ĝ―ΌΤž±ΐ n­Ž=–š±‘U cω‡ΩΔα/Ε{±‚W–|σ mδ06 8|ΞGΉΉ΄†UΔGa՜TF‘Π+8ήy1?Ÿ;>ωΔxΚ΅//΄ͺˆgΩn_žGρ°\6y²ZΕοc1pχ±XΉ)FAYχν‰'JžhΩ6{Άφ·$+‹‹9¨¬tϋŒΓ X­ξ|υΥ$χ  ‰D --ξη0;―½φmηΣ§O§!ό €ΔρZ`ˆά.›ΤξΉηκΘοΘ³8Zˆ•ΓϊYG€δ+! %½ΑrΞΙ₯ζ;3Υ ž"Γ­ξ[ΰϊ+vSdHΚ 5²’δ&σΆ°Ω“.jθΗ&ΡΙ(‘μKϊ•ͺH $’rD=EŒZξι81Nεo»ί΄iήͺΰ€Δ  +:gΞ-wΪ}χέGΦΡ›\Ε I ΑΠ­’NDBΉB@"C@χZ“κ5vΕkM ΓgΈ`ߊ΄‡½¨¬ώ‡-ΥZrΞ΄c–-ΣBϋI>Έ]Φο EŽ5ρΚ“DΡόBΣ„sˆΘU‘zvΔ©~νθf§N£6mθφθΫΐ†§wέEΥ;wΪΉŒr8w™ŒΪθ΅in:’)‡ς’p–;?ώΨϋΩΈ$a»sΈΘΦΗοͺ+/σΓ8ԍ䒫«¨p•ϋΪιΖΉΪRš4ΡΒkΦsRφpKΦ» 'εϊλIΒ¨Ϊ1°φηDιbά†€€@¬(γ§'Ύ‰σY.ΰΕ#ͺΌΒ uΔΠ/*• ŽAβ…€'―5γψt›ξΑ6ς|%W|ζPεΎ^ŒiŸσ"ΖuK—^c#‡φWρﳇφ ωϋκrΞέ+’ΕαΖ­€)Ώkΰ0»ͺxͺ/-ύ9ο―*Y]»ͺEΪqSy§}†G δσΤΎδ οΤS¦Ά2=΄/ί£ύž|T?« ΐT&βΙя߱51θδ©ύŽ΄ΡΧ"r„§φσ=κ«F|αΞ2»tω½O忍ϋτΡrcλΕς,ˆxj?›½ΌϋL™’Wwύzj_’™HώgU2<δβΞκޝz«!νY'νK}‹žr}Λs둇Tu,CΧI₯LΎ½Τp€¬Uˆ­>K[Ψ¨τΙ7{ςΟ>›Φ¨Ο€αR™ —HŠŒ/ωoΪθ1'}IŠŒ^-¬φrώ9N HΒ¦vε#ή›H?ϋψoYbWοΪE ΩΛO<αδγ܏?o}¨η$Gήφ> ‰έ./°ςχ ©l$”δΟιe#jη7Κqlΐρφm/uΛV΄L‘alΗΞΎ•Ηπ’._ίφ2ος9Έχ³nή$‰£ίHΪu‘€ρš¦Όπ`4§«œjVΙΎΔXf•‚ΓJ/«2c ŽŸΨΐh%žRpHέ΅Ό Α*EŒU;j™ž"ζK^ha”BŽta•ΗΝX'Ϗ±οXߏ5=ό£„‘—γPˆυ_C(ZFqE@Œik)ΩΩ½ΫΤ+LωγΈ‚§ϊ DWυšδτtΚε°²EZ"ΙMsΩ7.ΰ!ΚκΓL/zήm˜•EMωEMΆH‰$G–­‡–ΠE’##Μ£NΏ ρD ?―ςΟΰ°ΏΊa-•'c$€ΣƎ /΅xΊΩ €€WvΌΦΌ6pθ€]£šT?ν΄Σ`T³u@L$Œδ.W₯υqΗΩ2ͺ©Χωs¬§Θ‹{x‘V "‹sΕSVζ5|‰Μ» }ηϊŽίQ«ŠŠΝ)ZZρ‚6ŸΒ^ΞΝ9Τζαrsπ«―Ί·λ)8d^Ξ_ΡSŽrˆΟGE ,Ό(₯MIύιΚ+ιˆE‹¨ΗwjQ­‚IιηΗ_.‰Vk‰vΗ1^Ώ μf{ [h”6ΌβGΌsό “²“_HŒϊ„8Β_c‘kί©~] ΔθΈΕθƒΪ ΰ  ‘Cd’Ψ.\H­9€sG9KεΌ.b`ƒ€€@’Ѝ]§žzjΨQΐ[-μˆΡA 8ΐωΚΧ[δλγ‹δݎg―)»a­8HξΏ`—6³%iΥ—§2IιςΛ©=‡DŠ 1Ζ f£\οG‘"NO"s&{ΨΨ&Ο‘ξρ%ι#ΔLΫ8o’pc\’ƒ‘§±E²/ ½9‚sΠ“‚£ ΟΙ‰NrΖ«)8<Q/—ϋΨα’‹΄ΝŸ”#’C"YEϊωQuΕρ`Xϋƒφ@@ΐ’@='8]Ι.ΫF‘$€…/Ό`,ςΉ/‰T«8¬QςN?έxθΆοTΏnJΔΰΈΕΰMƒΚ ΰ0g8œŠlΔ%°}ϋvz…WΟΰό+9Εuβe*‡‰ρ&0ͺy£ƒs ‰F@χZ 6ך7nΊΟ[œX!ι’λ½λ7Ία©―©!ɏζ+_›ΫE6"Ω—S)8Œ"™rDο7Ϗή/~ €%Ή°ΐ€Xhvψα$«wŒ²υ?‘JϋdWͺKKιgΥ`”€΄4jΝ!£<‰Sύz'VΚΑ-Vξτg ΤπDΒ{Όθε$Ξsю'nαήbTyν΅ΧHΞC@@μˆ„Ρ ήjφξj€€]β‘j£š§Ύ#Ω—'’‘)G’α.„FΦBΓ­€Δ1 5Π’c&₯fΟϊžc:WlΨ`,Άά—i~=lU)†ΈŽ—^κ56ΆSύZ"† Α-†nTμέ»—nΈαΚΛΛ#™€={6ΥΦΦΊ4ΙΚΚ’ρςΉ¬¬ΜU†θ% ‘Άλκκ΄Mφ! ΞΠ½ΦΒ‘A$ wαΠm‚€Δ> ŸΉκΎϋάΆ=K–ψ=0SΔψέ.πH@R?Θ֐ηxC%‘k)T‘ˆB]ΨΫlϋ‡AΓκυ]σζΡη={R—k―Υ oί9£Cͺ;p€ͺΆm£2ŽΣΌε7h;OΨΥs™Q²{χ¦>6 ;Υ―QΧXά·XΌkΠ@"C ##ƒ^εΔεΕΕΕeΒv'5Ÿ4iqΖ”@sWcΨ‰()S¦hή†ιM7έD’ϋ ΰ έψŽ\kπVs枒W"€‰ν§@ jΖ…”‘ k‘"‰v@βš€x¬ ϊχΏι‡σΟη£Φ»ΖZ_UEk8¬lvEΒJfƒ›ΔCφ%NυλK―h?nΡ~‡ €8G …CޜΟ?—jς"N:>aΒκΒ‰Ϊ!  ΑΠ½ΦB™kM7Ψ§ψ! ω«ϊ=ρ„i@i­Z™ΚP <=εH­!’…€ˆιyο½”ήΆ­­Ic«aTrŒ kŽ‘­Žλِ ‡V+/‘@»sΑΕΆτ꫉ BΠtΘΝ¨–ΩΉ³νλκΧΆ‚QZά’τΖ@-3yoMJςρώjώωΨ±cι˜cŽρY7Μκ’yˆ+Ί,”^kπV‹«Gƒ 1 I”€@dθ)GΆΝšεκPO3θ•W(³cGWΉΥŽ€ˆYΘ!θύMcΥΚ’‡€χ/ΞθΡš8L uλΦZ R‰CZRRβ°6θœ#Πωͺ«θ˜U«¨&s()»’Αa"{ή?ζπ‘ώΥτφκWο?VΑ-VοτTTTΠΜ™3i̘1tύυΧϋΌΈ3/p9ξΈγ`TσI @@ΐΊΧšWš―Π uζ3(ΘcvŽ|a=EΜ²[o₯Ÿ~JεkΦha#kφν£}ΏώJEοΎK Ο<“Ύ4ˆΚωΨ(vSΔ―Α~t€ΗZtέh ²»u£O?M½ΨP&9Τφ­^M•7~ή*·l‘†ΩΩΤ¨Mj”›K™Vͺνι§S³‘Cƒ™Sύ­ΈΓ €›Γ7έƒ€@ Μγ+/½τ½ύφΫT^^υ΄œW„Jž­ΤΤΤ0φŒ¦A@<Ѝa‘πZƒ·š'Κ(ˆ$€‰$νΨθ †΅ΨΈOΠ@ €6mJβiqͺίH3Τύ[¨‰’=ˆξZy-“²A’—@QQύ›σžŠ‡Ϊ^ͺΚή½{iΡ’E4rδHυŽA@"D@χZ &ךn ‹Κθ@@Ό@Κ―xξ$BA&ά-Η€A@@@ŒΌςJ’²²=χάsΖS؏2bΚ^ΰ·ί~»Ι¨Φ«W/ΝSm {Γ¨e7κ€$PΕΰ­–p   QO)G’ώELAx¬E 5:†€LԞ{MΪΙΙΙ‘³Ο>›&MšDC† ¦i\   bΑx­…Β0βα 9ˆϋ7o¦5S¦˜τΙh׎Ίήt“© ‘%€”#‘ε«­Α°«wzƒ€€€€@˜8q"-^Ό˜δWςχ€§§'  Y' “οIIΏbΡυsψp–€n $ΧΌΥœ½wθ=Ί ΨΉ“Φ?ρ„IɜΒBΦLTPα#€”#αcκ–λλλ]M†κ›†5Rμ€€€€€8A ŽζΜ™C;vμ ‹/ΎΨ« ={φ€Ο>ϋΜkœL7ρΚ|Ω ΡI ―5έ #‚V   ±D@Ύ56όέ &F59… ΗZ((’ Ώ ¬Z΅Š&OžLν8tΡψργιζ›o¦ͺͺ*ΏΫΑ  ΡI #ΌΥ’σ^B+?ΐ°φ μy! /Γϊζ₯N€€€€€WeeeτόσΟΣπαΓIΌΟώρΠΆmΫ΄kJKKiΦ¬Y^―ΗIΨ" {­ΩΡ:CœvQ@@@BI‘ CI3ŽΫΪΙρ›!       ΩχΜ3ΟPee₯©™άά\ΊπΒ iθΠ‘¦s(ˆ]Ί±ΜN5x«Εξ}†ζ   H`XK€»±‚€ OpάζΝ›Ϋ¨‰*  α πΛ/Ώ„£Y΄ QA %%ΕΝ¨–ššͺ…€œ8q";–’““£BO(  ZΊΧΪ’%K<6¬ΰάh@’†€n4σζ΅o΅¨Ή]P$Κ €΅jEΧ^εZB=ˆo ²Δχ1:+έΊu£5kΦXB€€@xν΅ΧθάsύBΝ όN`γƍԑCŸ8V―^Mέ»wχY@@β‹€L?’•Χšή–.]J}ϋφ―Ac4   Ž¨««£† χ/KJJ"9…$…’΄     xΆoίNS¦L‘^½zi^hVΉΣT*0ͺ©Dp  ‰A@χZ³-ΌΥ¬¨  @@@ Z ΐ°­wz€€€€@¨©©‘χή{N:ι$jΗ9>nΎωfZΉr%νέ»—ή}χέ(Τ*€D =ךQo7c=샀€€@0δ#TΓZ¨H’ˆc+V¬ nΈςςςH< fϞM΅΅΅gggΣΝ;]Η؁HxτΡGI>e#/@ Ί XΡΰ­έχ ځ€€@,HNN& G-›ρϋ5Ψ1ύ\2ΨVp=€@ΜΈζškhΧ]1§7+3fΜ Ιν£ΚE]D:uR‹q 1I9GbςΆΕ•oΌρM:ΥmL2A:jΤ(š4iqΖ”™™ιv ΑX΄h}τΡGΑ7„@ JΘΔVλΦ­iǎšF-[Ά€{ξΉ'J΄ƒ <όό|Ίδ’K‚oΘC ϋ7o¦΅=f:›Ξ‘Ίήx£©  ‘'ΓZθ™’Eˆ W_}uLθ %Aΐ/Ώό°6aΒ:βˆ#μ4:  >ΘΏ©<π€ΆOB@Κβ)λ₯‹+q@ bXϋϋίLΈ’šΐτιΣ£Z?(ώ:thX k8BΐΊΗ7©•SXډ β•ΐ†^ M―Όβ6Ό^ό­bδH·2€@Έΐ°.²qΦ¬ +))ΡFU\\L-Z΄ˆ³b8    ήˆM< † FΗs %%!ͺΌ7^8     ϋ7l ]sηΊ5^Θ\nωδzηw"7:π€δ2„€€€€€DΦ’η^@\qΕ$Χ²Εsž—’’"zπΑ©[·n4jΤ(š1c•——»ˆΏόςΛ}μ€€€€€€€X@Ž5k.(Έ! F3ρL«――7©W―^ΪΉ .Έΐt Ξπ rπΰΑ4nά8η„   ΆlΩB/ΎψbΔGŸΡυ:ΥΤoZ«V¦2€@ΌΘ),€φ]δ6ΌŒφνݎqα$ΓZ8ι’mˆ£G&γδ|NN}φΩZ8Θ!C†D†P@ΐO‘ εoχž{ξρtΚA@ΒL`Α‚ŽΦΔ€Vpέuašθ&wΪi$œ"ÚSδΡ/€€€€Dˆ@ǎι裏ΦzΟ΅Σψ#΄Q£Fκέ€@ψά|σΝ$@@@@@TuuuΤ°αοf°δδdͺ­­U«t ΓZ@Ψp€€€€8O@>>ώψc3f eeeyU裏>’””―up@@@@@@β‘€1ŠK°γK Ά\     Y«V­’Ι“'S;Ξ±qβ‰'[o½εSΥ|"BπIk>‘‚(.. ”••Ρύίь3θ»οΎsΣDΚ&MšδV†w•[ΆΠAŽϊ K‡GoΤΊ΅~¨ύVlΨ@_xΚV¬ ύΏύF•[·Rj³f”ž—G-Ž<’ςN?²{τp»&ΠƒΪŠ ΪΒοψϋ–/§ ξk?χ]YTδκ―QΫΆZΏΝFŒ 6'œ@ 8”]°Rϊ㏴σ³Ο¨bνZ*_³Fϋ­ή½›R›7§Τ-(»{wjuάqΤzμX­ο`ϊ W_Υ{φPνή½nͺIώΑδτt·2ύ@g«SƒΤςˆ#\‡²#ήL»ζΟ§­ω•ϊ«vκψώ€ρσΡ¨Mj6|8εωΟ”žŸοv] υΥΥ΄ƒ#”̝«υSΓόρσ•ΩΉ3eχμIΉγΗSRjj Mγš€a-B Ρ €€€€B`ύϊυtχέwΣψ―²²ΤDnn.9’κλλ)) )L€P   qD@ CkώωOӈ28’AΑυΧ›ΚQΰNΰ+6ŽT1C]Z{,ότSν°œ M+ξΈƒŠήy‡Θ`|““Υ;wR9G(ώβ ZΙurO=•ϊς}Θδ\ƁˆοΦ?υm|ρEͺa#‘*zΖςFlΠιΘ‹ι:\r ΙύφWΔΈ΄όΆΫ¨ˆ ylE2]^Ε=ΩΚ–.₯"‰ˆΑFΌŽ_L=ο½Χd|4]¬„»―΅S¦Πκpλuθ¬YΤφδ“έΚταμVŸ k§ςχ“.EοΎKΏπίOε¦Mz‘λW "bp[ΖΉ}σΨΈ6pϊtJΙΙqΥ±»SWUEΏ>τ­{β Λϋ·#Υ.Χ\CΌ’R7֊ίSŒͺ…lόνΐω³!Ξ€aΝξθ@@@@lDΛ―ΏώΊf8Σ/HεΥ‹γyγDώΛ+I% 3@@@@ ώ TνΨAλ¦N5 4§°†5ϋΫήŸ~Έΰͺε(vdΫ{οiGƒω=½ν)§ΨΉΔUgΛ›oχηŸo2ήΉ*xΨƒΰ*6r­yμ1ΑgΝΩHhWVέw­Ί~:ȞRΆ…‹ώυ/ΝΘvψ'ŸP³!Cl]ΙΎl)δ₯Rmy9ύΔΖΚ’·ίφRΛpŠ ’E|J, ‘_~ι—auΟβŴ謳4AC‹–»Uμ%ΉόΦ[iγK/Ρ°Ω³)»[7βBΛΊ(t†–΄:Γ½‚€€€D ιΌΪPΌ½d»βŠ+’D«?ΤhίΎ=}τΡZAώύiΪ΄i΄•?΄ήα•΄ϊӟ`Tϋφ@@@@@ΐoΫΨp±ΰ΄ΣlΥτκ9šΔΒ3Ξ –έHθΏ'LπΫ¨fl_Β~Λa!ΕPcGĐ·ς»ό3ͺΊωN‘μΛ b@»u|Ύγ|ΥΆj†^$dηl5†5œ6νξβPώσŽ:Κ–QΝx±„€œ;j•―[g,Ζ~€ΗZά¨    ΰ, Δ)©γ• Ύ<Ξ|πAzδ‘GhΰΐN©‰~A@@@@ ξTpŽ±οΟ>ΫΝΠ%9ςΟ9‡2 ΄ˆ~Sr‘mϊχΏ©rσfwό.ΏψςΛ©qŸ>Τtπ`χsΚΡΎΥ«i{·Υ8ΰv&‰σ‚uβ~9ύϊiΉΫ2:t ΙyV΅}»–{M w{—,q»Fς‹‰±k,λ–œ–ζvΞx }ώtι₯Ζ"mΏωθΡΤι²Λ(³kW-Z {κU°ρF6ΙΏΆγΓέ‘ώdœG.ZδVn<ˆd_Ζ~ݎΓF–|ύ΅ΫεMψΚ}ΘΰŸ 33iχΒ…΄kή<-D¦[E>ΨΝωΨ6°GY' ΎΖΊϋΨ8&χͺŽ½γTIαά}-Ωΰ¦υ—­έηοΏw W*aAΏε(%θ"ΓZtέh    ΆσGς«―ΎJ/ρ‡ΨυΛRc‡vXPΑA@@@@ ²ΔσH1pIήͺ|6΄5°Θ]άƒ½Ύ~{ξ9ϊωΊλά qkjθϋsΟ₯£87YÌ ½9ΣοΊΗ§šR·ςΖ‘BΒI6ξΥΛ­ΌηQnά»7΅βΘ’k«ˆσ{-βά^ΖάoΥ%%΄η§ŸΌ†„\rΥU&ƒΞŽΨΡ‰dFIΟΛ£ΖlP)ΈφZΪΚ‘.—ώυ―n₯=lπ‘όsyμ₯g%‘μΛͺΏΚ8€c1ui:t(υε›ΝGŒΠ‹ά~%όζ²›nr+“ƒ­ΜΓ›a­ΎΆ–~δ£ͺQ-™Ÿ“ήœk­η°ž*₯?ό …άΟyρDτO"ΖΤRΕΛ―!{¦ώρǚΑΤΚ¨&m5ε…•Gώψ#΅ζАΰHt=υC-:C%0¬…Š$Ϊ Ώός έpΓ ”Η«@Oγά ³9‡ƒρ…~ωςε΄“Γ{@@ό'pW7[ΐ   [σσ©{Ο¨[W~ί„N Ϋ­·RkΧgGςΞ<“:²§‘*λž|R{/QΛεΈΎΊšφώό³Ϋ)ρDk˜•εVζνΐΚS¬tΑ—μγE|’Ξ(-ƌ1zέoRXHmÎ΄i%‘μΛͺ@˚ DCΩ0™½}IŸόƒR[Άt«&ފžrΟ‰aS kͺ xφYjΑyΣ|IjΣ¦τμxeΧ†OH… ‘χή€ H“&U᳁"~Š(v±#ΨP,¨XP±aο Š€#½ƒtBo ε?g~7ί΄™-Ωμnžs]ΓΞΌσΦ{–μξ€€€€ΨŠ "šί %Yΐ¨ΗαΩ=±ΖO'ΚΎρΖ!ͺΠΑ€€€€ž@υλ―§’eΚx4°δ@«Ψ―Ÿ‘Ν‰Ε‹ eR ‚Xλ?ΤlξΟ―7«°ƒϊΎτβ—œ—άiηNWu{\ηΦ[©Ο͚ͬ_³²ΒΛνd=aΒͺ[·n1₯eƒ€€€€@pπΤsΜ5ϋ }ϊΈv ^υyΖ\'jΧ¦š,¨·¨ΨXΧiG―ΗψΑΌ|rD™­+ƒE²…έ»“§žkv δXvsqz>₯[7§U κΕθBAΚ ΅aAEή9a¦3₯GuGϋ‘%JP aΑEΒZp]Μ@Š-'8ŒΒΑƒιž{ξ Έΐ¦ΤΎηdŝ;w¦ξόE §Σ‡όΔιLIθmc—s‚π/8ΑyoΞΩΙαC`     ΑC ™=—Ό±“ϋg·mσ¦+·m$—lΖή½΄sκTZαAAι0¦B*mβa•ΖΟβύVF0‡ΫψjΛΧΉΊΪ'Τ―οΪuόZ"1ΡP7Ο$ϊR6‡λLη(%j‹gq5ΎF u‘γ}δYsŒ*`ρΛ>`¨1€€€όԎδ ”ΐ¦Τδ «˜ˆ|0βCΰΆΫn+yύΞ;οX.|αΒ…t3'*―T©έtΣMττΣO[ΦΗIΰ%Γ!ΝΌœΜXržινάΙ“ϊ"Ϋ㜳gι8‡άχωη΄uβDZ=j-δ‡ςf7h@3γβhV͚΄vτhr—ΛΛj€S¦PT|Ό‘Šδ{ΫϋΡG΄τ?‘Ÿ8ΗάόŽiΓΓΣ‘_υZh δX†yQ]œ­œ5Ο@½™…‘ΤΧqwη₯ η?”ϋN „ο] π΄zφμYΠ™K`{ώωη鑇" §λa¨„‚ΞT;"¨=ωδ“τΓ?KLsqOD>€@ρ!ΰ y-+6 {ššJρNρP“άhj[ΑΙΙ7r2ο¦&O‚ͺλa@@@@όA@BψύσΚ+†Dδ©Η``žˆfQΙ[“PŽQ ”«ςψ:οPX;Ύhϊε:φηŸt’SxβΡ“ωJΈΓ?LK.½”r32L›ζŸ?OΒRΆν|%’dI*ΧΎ=U0€ͺpΊŒR=»9–ιB<,ŒbΡ²°ΜL`-ιΓ{Ν[O·ΒZϊ%‚Ηή  AC@ν΅¦ž”K`σ5›;5υXπVSΣΐ>o"² ΰ“5ψιΐG}Τ ͺI~ΘI“&Qe~Κ     uψ0ύσK†mο'Ÿbψ°Γ±C`轞ς²³-e8@Λ ’œ‚b{§‰˜εTT+έ¬™eίξNŠΰΥeή³μ§6'f:t¨eέ»wΣ§Ÿ~ͺ©S“ΏΰνΩ³GS¦>p l‘κ›ήκz―px ɝ$ο-'&’ή^N όρΗ[V―ΞO έx㍖uφsx geΥ8TΗ0Nle‚ξƒ>°ͺBUͺT‘αΓ‡[Φ9ΐOΕM›6Ν²ŽxέHξ(+;tθ½χή{VU”άS#FŒ°¬s˜Ÿ²|χέw-λT¬X‘FŽiYηΘ‘#d—+%%EB¬::zτ(½ύφΫVU(99™n½υVΛ:Η9Aρ[o½eY§|ως4šYΩ ώqφζ›oZUQΔy[ΩICςΖoXU‘€€$ΊƒXYYZZ½ώϊλVUH§;οΌΣ²ΞιΣ§ι΅Χ^³¬“ΘΙ οΊλ.Λ:gΜ‘W_}Υ²N©R₯θ›P8g9—όοξLς¨Ιw S+ϋςY提΄ξΖC9€€g"""”‰ž΅BmψΉ6fVœ$ܟ: €ΤΥ{°ΉΪŸε{:+ωχD.Ρ[ίμ/{α…”Δχ:ω~^<ίKIΰM^#Y𧕻θ"’­ΎKγϋ”GfΟ¦£ŽR<Τrψχš•νη{@qU«R3ιΔ9–“ω²NIώ]«7½«?ouœΝχP`ήqMLŸΖϋΏ3ξΦψfbΎΌ]d㛏n+σΝ»‚zϊϊWΎ)ιΆ½λΔΤ©Smϋ5j”«ΊΫWΎΩjΫίDvΫήuβύχί·ν‡oΆΉͺ»}ε›μΆύάpΓ nΫ»NπMΫ~XΜpUwϋΚ’ˆm?,Έmο:Αbm?,>Ήͺ»}ύϊλ―mϋaQΝm{Χ ηlϋa±ΠUέν+‹ŽΆύ°κΆ½λ‹©Άύ\zι₯κn_ωΖm?,Z»mο:Αβ·m?½{χvUwϋ:{φlΫ~.Ήδ·ν]'ζΝkۏώοˆώ˜Ÿ8ρΉφVS¦΄`ΑΫΎ:wξ욾ΫΧE‹ΩφΣ‘C·ν]'–,YbΫO»vν\ΥέΎ._ΎάΆŸ /ΌΠm{Χ ~€ΒΆŸV­ZΉͺ»}εlϋα άΆwΰlϋα \ΥέΎςƒΆύ°°οΆ½λ‹ΈΆύ°°οͺξφuΫΆmΆύΤ«WΟm{Χ δmϋaάUέν+ ιΆύ° ξΆ½λ ΧΆύ°ΰμͺξφ•…kΫ~X(vΫήu‚EzC?,ŒΊN+―ηΝΛg‘_S†$~°Απ·JΎ έ~ϋ큜ΖΠpχ[­}ϋφΊšώ=<ρχίωόh³a›ΫΊ΅ Σή~αίj~s[Άτz₯§ΦΥτ%ύΞ7Ή_{ώ|ώŸόΎP«μGEεoyζ™ό,‹{­ϊΙιϋXΠ­›ΎŠWΗy99ωΗ—-ΛίόΤSωσ:v̟Ξsӏε:N>έ«1\ό1ΦΖG5Μ/•ογΉ3³ϊΗ.tWέmyκŒ†q7Žg¨δ? υ`¨η΄ΰ(ί«rρw½ξž6Νiσb]£[|—ζ¨4~cPό‹fO@<\!Vδ‰}ό—Ά8,3,ΦΘ7Ύ}^‡UJŸ;£π"Œ.&–βˆ@INά-9Φ`     αG ›£—xk™ϋφšΖp佉WΨI%―ΆφDkΗΡ„ŽK1ο΅² ₯ξΒ―ϋQQT];j4~fρτ&ω•…½?LrΌ^}υ՚δΑ"»όΜΑ<7Νb<<0[Χ£μ-Αaκ=μ ΥA 0–.]Jήί0˜ύrœ~ε\³tU«χί§Z6yΛυ«ΙΟΛ£9ό°uϊφνšSέXt*«{ˆηžίϟkj»σ•U·Ιm―/ϋ‡ψσqΙ₯—jŠ“»u£.œ#Mogy^‹ϊφΥ7{ξ9Ε<ι`6‹DκuJNΈιιωί9–kA›Ψ±cλ3ΟΈ•ΧφίOUώσM™λΐ¬ώŜW|§N*Ž^πΛΈBS·!Ο₯Ι„ š29ψƒ0V―Φ”wώγJιή]Sfw ΉΩ~α†Υ"΄i=mΥδΌ}0kς V‰sr(Hrε[³nekφŒP@@ ˆHxƞ={κθjo΅Bƒ€€€Xΰ¬4fΜš7ože=³“ΩΩΩτν·ί*›x‡‹ψ%’Œ·Ϋ2Ω΅Ο$Δ׊+¨/ί¨t™Αlžκ2 gΞω}ΥEδ$2B0ΟM³ΜΦ5‚=:`  PΨR9ZŽ§ΒZκ7ίhΔ&™c ŽΎ‘ΔΡ³τ&Β“ή/Έ@_d{Όϋ½χlλΈ*ΔΧ­xβεefΊŠˆsŠy%¬•Φ ˆ%9:HδΏB…tΘ± ;ε:v4k['NτXXΫžƒzQ-–φSŒ ϋb  !K@ΌΘDψ*Lρ    PtδΙα»ξΊ‹$―7’š~ζǎ£'Ÿ|’ΔkdγƍϊΣ>gς Jy¬ω4“ΖΑ<7“ι’@‚†ΐ‘Y³θ0ηAσΔΆ>ϋ¬‘zUφBŽ`½%°Θ₯·³[·κ‹,²§ΣΑ™3-λ¨OŠπ•€K7#ΉήΔΣΞSΣ‡~L₯η δXžΞ½(λΧ6Μ0όΡί'=OC%UAΪΪ΅΄συΧU%Ψ ΖιΑ23Μ@@˜@a _πVΓ[ @@@Š–€ˆAW^y%½Ξ7μ2UDGG“δ–u’GSV%p’›[οζλŠgρ Ψ/ΎψΒΧn ₯}0Ο­PŒNA@ΐOΦήve¦¦Ϊφ–wώ<­boΪΣ,x¨-‚CKH@3+£Έ€Ξώ/Ώ4«jZv‚s.εΟJŽagzή]‘^;»e νύδwΥMˏr˜ΙlΞA§Ά²:aMΞr,υ\‚y_8•7yX|ΩUWΡ©5kl§~ŠΏΗH8Ox«Ω’*’ ֊;pJ 0½Φ S΄sΊ>ΤπžΐK/½€„ϊ“p?ό°χ‘e‘8uκ”’ηv¦›'πE@»ώϊλiϊτι$ωΘ²²²”πŒ§OŸ¦“'OZΎ©ωγ?’„ Œ‹‹3]ƒ΄Ή‚s‘lυΠ3ΐ΄3U‘„¬”ω£σά‚‘ζ B }ΗšΟωΆN\ιˆδΊZΔyχpN6½I«„Z΅τΕΚqηΪہοΎ# hυP‰Œ·αΑi>‡ΜIKΣw‘g:dZ.…΅FŽ€φ\SΫͺ›o&Ι7&‘εΠΜKΤVcΌJϊ2Θ±΄ Ο?O"Ίͺ-sο^Zΐο΅}όŽΩuΘ>~œΆr>ΌΏΊv₯l‹λ«ξϋξ DEEΡy~ΏΛ&‘ΓύeΪYώκύ€€€€ ˆζο\kπVσγBW   PDς8œQNNŽ2ΊΏ‘ΡRŠε°7σΝ½₯K—ΦA7ρ Κg9ΜV…  η₯ ))Iٚ7oN—]v=Η7 ή}χ]zωε—NέθߘΌτK!.!!A}ΚλύC|£KΔά©S§zέGa5 ζΉ֚Ρ/'™ϋχӎΧ^3,9ΆjUͺwχέ†rΨΰ›ξ.O°Μ={h{%wλFUΨC,ž…²’œ7ν‹mΗ/¦c,4γpΓzK¨_Ÿšp,wV‘W/*ΗBΚ‰E‹4U6q.Π3fP]Ύn‰M›RΙ²e)“σ{¦οάIΏžύς εs>N—Ι|βΨsϋ8ηJs™x‘m?žR8?}‹ϋτq’2œΗ­ώύχΣ6ώŒ,0φzΫϊΜ3t˜½―%Ta©† •MϊΝgαAή_'9Ÿ¨δs;:w.±ςWΠTvŽKεΪ΅Σ”ΙA Η2 Δε;t ζ―ΎJβ©ΆάŒ ϊϋΊλhΥπαJ^>ρjΜ9s†2ψ=(‘"ΥΉρ€]΅‘CigŸ©»ΐΎJθfšΊ­ aΝ-œ.―΅ΏώϊΛoS‚·šίP’#π˜ΐoΌ‘x’ιŠ—šx¨]rΙ%ϊS–ΗεΛ—W„. +Ω•ŸπqIm;ΨAΖ|Ÿόχ—‰wγ7RΎilΜs 6V˜„¬Γ‡iϋ€I†i—a―(k,ΆβeV…Cσmα…‚Ί" ΙζΔb*W¦Nœ»,&%ΕmuΙ»vαΗΣά-(χμYM½S,’¬όο5ef"ͺuα9‰™ZX“Ί[YΤ“-–Ε±~,Μ©­ΡcΡώ―Ώ¦ λΤ&γΚζ²ΘΨXE˜Σ iσςZ–s—6zόqu‘f?ciςƒ:£G+l7<π€!¬c{֟X²DΩά-£Χ¨ΛΉh Β?ˆ+:Ytμ12€€€ό)„Α[Νπ¨    ~& !ο»ο>C―β6oz*ͺ©;jΠ ύρΗ¦žn“ψFτYέ Mu[»}½·›xLήzλ­^“vν σ|0Ο­0׍ΎA@ΐš²GWνΫoχΈ+ρ2κ#$Gη—Υ›Τƒ„‚,:φ@@ΐC"ˆωšk ήjBGuπ# |CιΧ_5τψ‡GκΖ9mόež±M›6΄’ozͺνΞWs?5ξ]Ε‘Β$4ω|ƒQm"j <˜jq˜’²`ž[Q1ΑΈ  ΰ)ZœοͺΖ 7ΠI|Φgξέ«δΌa£t“&Κ&BZΥ«―¦₯JyΪ½R?‘vmκğƒ’?νηm;½q£²ε°G΅δ9‹―^ΔϊυΧ“Τ5³Jύϊ)„«L[³†ˆEΈ8φHKfαƝŲΨφσΟ©ισΟS*{žΰ§ 2ϋΘ’|_b%I<λ”­n]JlΦLγ"K–tΧ­iy Ζ’œvVyντσ΄ΎΎ½λΈΚε—ΣΊΌssN^ε½T“C;Κvžsͺ₯oίNήUςΫΕT¬¨lςzj;wόΈϊPΩ· ?j¨ŒΏ€°ζw€θ@@ °ΈΌΦ|Ι΅o΅ΒΊ:θ@@@μ ΌτK†J•9?ΝSκΘί6rδHƒ°ΆpαBΕ[.&&Ζ«α$O›„š<Γ7Γ\–Α7$oηb?ό³«¨H^ƒynEΔ‡AΣΣΣʍ|Γ{Χ]΄{χnJMM₯rεΚQUΎy]₯Jε΅S§NԏopGιn€š }πΰAΪ²e‹ζ”xjJN@omΡ’Etξά9Mσf|#<Ε‘ƒΌEθžΝω‘φq^&ΙM˜™™IΥω¦nM9'€Χ^{-%''kΖpr`Ά^Όc9“ΛdόiΣ¦ΡͺU«hυκΥJXΥΙ“'SŸ>}\Uπ ' ‘«ρΓ²¦UκߟdσΦ$7\Β°a7Ρ¦Ύ.s‹:ΉόΏ$ kώ΄@ŽεΟyͺ―’œW6Ι‘§ΎˆΌzƒ°¦'ΨckεΡ@@|$ΰ‹ΧΌΥ|„ζ    ΰγό΄υηό΄ΌήD‹ŽŽΦϋ||έuΧΡC=€Ι«&"ΔR~Jί[1CD•‰'wή©™ŸxΒ}σΝ74hΠ My ‚ynδΰΛX" Mαppοs8ΈS§NΊ:žzq¬{˜ g/—#F(‚”‘ΡΏςώ—υΆ†½MZ΄h‘/Ά=ή³guξάYSO„:YƒfcΗ§^N/ΜIΫνμA!φΑИ1c¨?ίό—ϊνΪ΅Sʝό#‚έΝ7ί¬©*"₯Λ³sΦ¬Y4jΤ(Ϊ«»Y|τθQM ˆW“xιMΌ„` ΰ ρHσΤ+Ν›q€M ΗςvŽΑΨN<Υ&‘@K–-«.Β~€ @X 0p   ΰ_ΌΦΰ­ζ{΄_HΤύMόμ 7Χ ΓJσ“ΰf∯cέvΫmτΩgŸ)Ί/ 1)ž6‰~~β_=†έ~0ΟΝnξE}ώΛ/Ώ€λ9τZnnGSΩΏΏβq)ޘsζΜ!ρΚ23ρ$mνΪ΅šΣί~ϋ­Wš΄Σ›„So3+“<ƒχ±·Κi“|=fνΞ³'‹„?ύρΗ•0ͺ&L ψψx³ͺŽΛž{ξ9’\‹0οΔrΈΈ>θ]c΄ ·y„?ΤViΐJβPŸžX>‡ž<Β9ξΤV‘½₯%7 ¬θDέΠ@@Ό#ΰ@o5οX£€€3Ή9,Blr£ά,X`˜`«V­”z†A\ ^AοΌσ•Τεœ9pΰ€βΥS”SζΉ%»±ε½9ŒCͺy*ͺ©ϋ•π‘RBΊ3ξτf&ιλ˜‹‡€ήΜϊWΧQLqΩ²eš&2ί«,n΄>σΜ3τίΰΥ›άhμΩ³§ς~ώβ‹/ο4yίtΣM¦ή—λΦ­3ε₯οΧμψοΏ¦;ξΈΓμΚ@@ μ”λΨ‘JθΌΨ|χeςη‰S;wς$­Σ…ŸŽŒ‰‘Š—^κ΄ Τcβ}-›>r‚/p΄ŸζΎτ„Ά   @žeπV ΰ…ΑP    `B 33ΣΤ“§}ϋφ&΅C£hόψρT―^=ΝdΕ«η–[n‘œœMy ‚ynfa7žˆd'ωΖ₯Ϊ$d£ˆ@β‘%žl]tU\™š6mͺˆPwέu—ς~o3½¨μΨ1Z΅j•Ί»‚}ΙƒΧ½{χ‚c׎§^kίρY½ 8ΠT“z‹/¦Η\ί„5j€„4±π…^ k―½–€Ÿ[o½•¦M›F‡2δ”NζΝ›GςσΤΔΛXώΈ¬yσ抠χΙ'Ÿ(!27oή¬„SuΗ+€„2Hw’Λ­yžσwΈζJwSςͺΝηϟ,W‹=KrΈkXΡ€°V΄ό1:€€€—<ρZσD„σr:h    `AΰΘ‘#¦aφ7nlΡ*ΈOΕΖΖΫoΏm˜δš5k&Γ‰σάˆΑΡP+V¬ΠΤ«P‘-\Έš4i’)7;±_|ΡpJ<άά™YΈFO…5³0C‡5RDm³άqβ=·rεJ’p¬ξLΌΰ^{ν5Σχσg&Β›'&žvb’θAΞ&γ‹ 'σ‘M„Ύ””OΊD]juΩΫ,B:ϊ8ΖόΞί6<ό0ž=›ΞnίN6ςό™3tfΫ6J>– DΆiCgωXm₯ωfόwVτ ¬ύ5ΐ @@Ό$ΰD0ƒ·š—pΡ @@@όHΰΔ‰¦½%%%™–‡J‘<μ%Mzο —ˆ ?¨γ`ž[ ؍#!‘$΄‘ΪΔ­T©Rκ"Λύ«―ΎΪp~ι₯†2WΑ•W^iΘρ²‘½$$€Ϋ·oŸβe¦›œœL}ϋφUμόρΗ΄kΧ‚cΩ©]»6½χή{―)ww LΈβ Νiμ&Ož¬)sz BβσΟ?O%Ψ›ζ9ΜΤTΪπΠC†mΗλ―{ήZ€*ρXkσΡGΔq€5γδeeΡvώ;ΈΈOšΣ ύΐαβ°‘Ώ7lHΛω‘όw2/;[ΣFΒJΆε½Q~Μ¦ΐ'˜GΈP@@ ˜ΈΌΦΜςuΈζιD|sΥΕ+€€€€@α8~όΈiΗήk—\r‰i_žΚ5ΛΒΟ“φκΊ/½τύόσΟtτθΡ‚βττt%„ή?όPPV;Α<·’ΰ‘SD5}Ύ•]»κ«YW«VΚ—/OκχΈϊ½ oœΘ7F%άβW_}₯9%^hNΒ+JΘόό|MΫΑƒ+Ή&5…| υ^yε}1I5§’š«ρΈqγhƌCευ§Ÿ~R2M‘ΝΑ φΒqζ=,φάnβ±R¦ukο€@p¨ΞΒη9δπZΙ1©ϋϋνt¦eΫ΅SD΅„:uœ6A½B&a­£{Β% Β™$Ϋ63Iά.β @@@@ h ΈσX+S¦ŒΗ›;wΗmτ τΒ„ώΌ'ΗεΚ•SόπCMχ^x‘ζΨξ@„Ή΄΄4»jšσς^Ρ kβΕεDX3 ©οΉ“άjzOΈ‹/ΎΨ«!ρ„SΫέwίM½zυ’₯K«‹ΊΜs (“Α$$’lΎΨόωσ)''Η£.ϊpθ/ ίxμΨ±‚vβ=·σμΤ―_Ώ LΏ“Κ!υωΫκ°χB‡τU•γ ΚΫ·oo(sZΠ½{w°&ν–-[fι?™g͚5ݝF9„€„S‡ΨCX½€`Η L 4‡|lωΖΤδι§i?ΏΟlέJ™{φPo™ϋχS ώ[©ΕVL uλR YΞ‡ΏΩ€^Έ ¬._τ  ςγLŸA†•§*a    EO@<§ΜlίL aMΦ&αυš5kFͺ§Πχσ2 ‘χ*?‘^”Μs+J.ގ-’λμ—_~qδe¦§${\sΝ5τί`U›x­=ςΘ#κ"ΝΎYHNέ™YΘ|ρ:σΦδύ­7}ώ6ύyυqΣ¦MՇ؁"!άΉs‘Œ‹AAΐE ΊlY6Xh€°ΪΧ³`eωK @@@ ψxωε— nB3†ž}φΩβ!DV,!χΜL„΅-Z˜r[6|ψp·ηΜNHΘ=ρŒ+lΟ§'Ÿ|’xΰΝP"žάpΓ J=Ν‰σάˆΑ£‘Ξ=«<Ό·{χno{χξ%w^˜N‘πzaMΒ¬@šδ°Ίλ» CŽ?ž$ηZQZ0Ο­¨ΈˆΥ¨Q#r'hιη%O½·mΫ–FM , υλΧλ«8>ΎξΊλ uέΝCR„k―½ΦΠήUˆπ§2„5qΌ‚€'Yœ6Φ  aJ@ς tθΠAYέ©S§())IΩ―Q£F˜ΛΠ#Π΅kWϊτΣO5_Άl™n―D‰ΒΉ=±jΥ*Νx:˜0aIJuώ) ™wηwŒ35 Σq‚yn¦.ΔBΙvΣM7‘δSΣ[tt4I¨ΕΦ­[“x₯Iž:Χζ―χkƒ ¨}ϋφJ˜PΧψ‚Rή75kΦtΡΑƒ a {υκE-B%&&΄WοΌωζ›$kσ—α7—ΏH’P"P8ί\C‰ζ   ςFŽςkΐ@@@@ ά ˜ kβ±&ž8C† )”εΟ›7―Pϊ΅λ4!!DΐΈτK5UΏώ{š9s& 8PSΘƒ`ž[ 9H¨G ₯¨ΥΔ쩧ž’Q£FQrrr‘OIζ ³ΪΔkνΎϋξ+(‘6//―ΰXv¬Β@ΚyΟδZλ=Χzτθa™—MΪΒ@@@¬ €5œπnέΊ™φςβ‹/š–ϋ£pΦ¬YώθΖ«>ϊχοoͺOΌΦτbŽWψΠ(˜ηζΓ²Ξ>‰ύ5ϊ(Φ$?Τm·έf` 7νΕsgΒ„ ”m8oWπλ―ΏR³fΝ詧ž²«πσ 6$&‚Ρ‚yn…Ι«eΛ–†ξΏόςKC™»‚εΛ—+b˜< ο3 yΣM7‘„5U›Σ06>ψ k·ΰUΦ©χ„+8i²#ή€z6ςPƒx Γ@ˆύς -`ΟdυvΨΗEΐ@ Έ ”ξιav       αF`ςδΙ΄rεJZΆl™fiς4ρc=FŸ|ς‰*qΐ€T³fMMυΑ™3gHr<‰ΞO?ύ€>₯μ·jՊ:uκDS¦L1œ tΑC=€ˆή„α+μΉλά$ΜβΌyσόΆόχίΏ ―Φ­[μ»v$―™Ό—yδo>3;qβ=Ηω­Δ+¨vˆs`yjΆ±rεΚtπΰΑ‚¦kΧ-Ψ—vνΪQγƍ5evύϋχ§j<λdήƒ ’±cΗ*BΆ;Έ³gΟέwίMΣ¦M3 #λ―S§Ž‘ P Hή»γ h–ž=r€ζ αEΒZx]O¬@@@@@@‚ž@tt4}σΝ7J˜:³ˆ’κΞ;οTΆ¦M›R½zυ¨B… Κ–––F[Άl‘Ν›7SjjͺΫ΅ΆiΣ†ζΜ™CλΦ­ aMΦόξ»οRηΝIΒρ“λάπjΩόa"”©…΅^½z)’λ’E‹4έ‹gαŒ3AIή{eΛ–₯}ϋφ)yΕΎώ{ϊ…=SΤ!kΥͺEΥͺU£… τ#οΟργΗ+ΉΠΔϋ²OŸ>ηάνHhΫ!C†(‚»:βΑζ}ψα‡$Bβξέ»5ΝED”ό†]Ίt!ρΰ“MΔmτd/P}ι`πΰΑ4zτhM_88Ξ³ΧΔ$œgl₯J›FbNΒZ1`ω       PͺW―δ>|8MŸ>έν6nάH²ybβύ#ήGIIItΡE)yψD0(jλΨ±#έzλ­τΦ[oυT γσά “υCxi}όρΗΤ’E Ο,΅ύύχί$ΉΜμLD5ρ¨“M-¬I»§Ÿ~ZΩDtaΞ‰I8Hwω%ΫuΧ]η€CEμΧ―νΩ³Gs~ΥͺU$›S9ˆaEG@΄†μU (:‘E74F(ΞΚ”)£`oΎω¦βζ+‹’%K*aϊΔSMD5±˜˜Ρ(XμΩgŸ₯*UͺΛt4σζΉi&κ§ e(α&E΄ςΤ†J’gMB•^vΩe$ο=_MB—6iΔ΄ ݘ˜˜hzΞI‘bK—.₯ΆmΫ:©nZG„? ίκΛE€@ρ “žNϋΏϊŠΞpŽΗtΞ·˜ΑyΕC1Ί\9ΔXΞ-*―ε:u’JύϊQ‡Έ³³,ΟxfΛm5Ξ3•Βω½΅γ‹Qž.R@b³f“’β¨ΛσgΞΠα_₯#³gS&ηQ±8—#Δq>Λψš5)‰EβjΧ^K1ΙɎϊSW2[o9Ξ…MΖίΓyρNq~Ι4Ξ隟“CLžLϋτ)¨cΆsrεJ:2g₯sΘή³Ϋ·+―ηNœ θςε)šηZΊaCͺΠ»7UμΫWΉNf} ,|@X Ÿk‰•€€€€€€€€„ ΙςφΫokfή›oΠ6oή\S†π#ΞΪΞ)ShΟϋοΣωS§  υcΖP%ΞyΩ€λ—kΧN)wςΟ!μVί|³¦jo)8¦ΨαY³hυ¨Q”Ήw―rμϊ'ϋθQΧαUD΍<T=‰σ½κ-‹ΕOΩN―]K©_MΔ‚g-žC㧞’؊υΥq&"ΓdX€€€€€€€€=μμlΝεψκ«―¦cǎiΚ~ψaΝ1@ΐώ/Ώ€ΩυκΡ?/½d*ͺΉ[q‡ŽέΒΒΝοΣρ%KάU#ρ$K4ΠRΏύΦm«fν’»u³χ€Ώέο½Gs4P<ΕΜD5ύ˜ωœGχΰ?Π|φ6[ί}”Γψj[Ÿ{Ž³7™^T³κwΛ„ 4§Q#Jελd&ͺ™ΆΝΝ₯έοΌCΏs»Λ—›VAaθ€°ϊΧ+ωFxϋφνIς§uγε8ΜΫ―μe‘ΆŽ;RχξέΥEΨ3Η, •Γ†Η€υzeΉ>r1‡…<Ε! έYλ―7œ2Θ •L RΏωΖPjΦΏΊ’ˆS«GŽ€œΣ§ΥΕΞφσςθŸ—_VΆσiiΞژΤΪωΦ[΄ι‘GLΞΈ/Ρsσc™zΧΉoυΏ3β}Έˆ=­Νjc/Τ d¨]1Μ@@@@@@@@ d œgOŒεμΕ ›™EDDΠ+―Όbv e @qœg«ρΣOHΔVl(CAπ8³u+-½όrΚΣy°FΖΕQν[o₯2Ά4{ΨςΜ3Š8₯.“ύˆθhJξEΙ§&9Υ’βγ)σΐ:Ι|χA„;½nΒ«γoΏ™QίΏϊψΤίΣΪ;ξPΩξΛυYΕb ήΚ_|1ΥζP’ υλ+yΤΞ³X˜Ύc‡²Iώ΅Γ?¬i"Χfυ-·Pw7ο5•qR ¬…ΤεΒdA@@@@@@@Β•@Ι’%iςδΙΤΆmΫp]"Φε#Π=ϊ¨½ yQΨρκ«tώδIΝ4$dcΫΟ?§Δ&M4εrΝ›6₯ ={Rέ»ξ’Tž–_sΖΣν‡’=΅j•ηΠ‰z‹«Z•RΨφθάΉšSβ΅&‘"š^z«

‹kjσDΈ‘vqΥͺ)α Ο?^ΠMφΡ£ϋϊ Χ(aSΏϊJsJΌΠqxE;SΒ@²@‘ΆͺƒS€ΙN"dόc’'²ε[o9Υ\γ47N#¬IΉVž kU r$ͺI'Ω£No™œΏΞ«ΘΒΪi.Υ–ΓβŽSQQέϋΑIΒZp^Μ @@@@@@@@@@ ΜH>―Φ~¨YUΩ /ΤΫˆΧΫω΄4»jšσ ¬±—aΝ4 €IxIπηV;˞pj+ρΕJ^3u™“}ρώJjΣ†N©Δ3›6QζΗαsXdl,5›4ΙIU₯Ž™ψ%ΉΣ²ΘΝ^ΖN¬ηŠ“ Ύ ¬…ο΅ΕΚ@@@@@@@@@ˆ€ »ΨσGo±,2Τ=Z_Œγ $ΐ!eσŎ͟Oω99uQ±O’πηŽ+hwš½ηΞnίN₯κΧ/(Σοd¦¦rŽΕΧ©Cε;tΠWUŽ-X`(/ΧΎ½‘ΜiArχξaMڝ\ΆŒβ8Ÿ™+ΗσŒ―YΣIU₯Ž™χ`ΖΝ΄ηΡαΗoAǝ‘bΨ€°Ά— 'YΦ§Ÿ6,© {φ@X3` « ±˜ΙΉΞύς mvΎQΏx ΫXυškhΧohN₯²ΧZΓGΡ”©ΜΒ@V:T]E³ό―Ώ4Ηr ^gήZb³f†¦ιΊόm† ͺ‚DΞaι‰ΕT¨@₯Ήδ S[ښ54«n]ͺ>dIΞΊ”ž=©DB‚Ί φ‹kΕθbc©          ΑI ημY’l»wlιξgξέKyΩΩ>MΌ‡o4kœgΝJX3 i!¬Ιόυv†CCξž6M_μθ8ύŸ υ2φμ1”Ή+(έ€‰»SnΛ[L™BK8OZ.ηES›δΖΫϋΡGΚΑB₯„π”0—βε–Μ―ΪΤ΄Β{ΒZx__¬@@@@@@@@@@ H _΄HρB;φηŸtrΕ CπΊ"Cwκ˜9€€€€€€€€€€„­– D :w¦m'*"ΣΌi₯MΒ#:%P½Φτ&α ΝΜ4 €I{WΫάΜLΚγ­°ΝWΟ='σq­ΛΌy”ά£‡“κ$BΫρ… iγCќ h~§N䎫£Q)¨ @X κ˃Ɂ€€€€€Έ#πκ«―Ri~ YΆqγΖΉ«†r*"Έόή¨p#hι'CImΫRνΡ£©Λ‚tΙϊυϊ*Ž«]w‘;Θ2*Šͺ]{­‘½« '=έ΅[¨―Φde™y—Ήs©+{₯ΥΉγŠ«YΣρΊN,^LΛY8]ΑB€δΗƒ (~ϞaGΩό6 „‚τJt      Hη8ΟΕYΞE"–νcΞ‘@ΞΫ“±ά݈Y³f ½τKžt…Ί  ~$°ΗƒO~–βͺT‘ΖO=eθ2ΆreC ‚“ΐYΞΆς¦›(χίο0κYFDG+y»’Z·¦DφJ‹―]›x“ΧΘώΉ•_š½©ΚrΘΒ“Λ– }ŠCPJή²x•p”uπ ! d…^½(ΆbΕ‚vϊ’‰‰ϊ"εΈΕ›oR$―Ν__£†ΏΊrԏ„Π”­Ελ―+9πŽΜžMG9t§x¨εœ>mΩΗώΟ>£ΈͺU©‡‹„ R₯Jω}`όoτϋ΄Π!€€€€€€€@DD„)„EœοD6€€@ρ" Z£ργ‹Χ’Γh΅y99τ7{0D5φͺiΒ‚i­Q£(&9ΉΠW,α Υš (^kυο»―`μΣ§εεΛN ‹0r^Δ³¨„ΚΥy₯p8E«Όl6T¬Lσζ$[ύϋο§όά\:Ήr%™5‹φ±’ΛτΆύ…aNr―ΑΒƒBA†ΗuΔ*@@@@@@@@@@‚˜€x:ι­φDkχΕΤpμXΗ’štΎX΅ΑƒIΖU›>€> €f•/Ώ\έΔt?&%ΕP~vΫ6CY8D° Z];EμξΚ<υe―Ώ†>JΔεz;2gŽΎΗ!LΒZ_έTX»€oΒΎτKα°D¬ €xΣl0Α0N™Φ­!¬¨_AmRΏώZ3±ύ_~IU―ΎZSζξΰΔςε΄”Ε0ΚΝuWΕ£ςR-¬ `+oΊ‰ˆ…$΅UwΥ¦ώƒj…5>!λ¬rωεŽΧΊŸ9I΅EΖΖRύB~ΰH0΅°vvΛΪϋΙ'T“=§vτΟ?)›―ͺ­,„55Žίw&3‡ό2±p#pχέwΣιΣ§•νiN`&Δ¨ή$―™xs峨εΞΝ8AX¬šί±#ε€₯™VΛ:tΘ΄άͺPΒ6ΖT¬©ršs©­l»v”ΨΈ±ΊΘvΏRώTIο‰ΜbΰςAƒh£x‘ι„;u‡9gΟͺ›o¦Χ\C"τ©ν‚—_¦„:uΤE~ί―5r$EθB6Κ|$Χ[ήωσΆγ7–θΧΞή~Ά=‘‚?δρ{-11QΩΚ–-λ.•>ΰ±ζ7”θ@@@@@@ ’££I6€€€„ ½zQΉNθΔ’EšιJΘΓ3fP]~h(±iS*Ι@ζΎ}”Ξ9Ύ~=ϊεMHΔψZ΅(Z5:Κ%žU›ΖWr‘εegSΕ>}4c˜DDEQυ!Cθ¬άY ρ`σΒΪ|ψ!ύΙBbΖξݚΦΫXD<όΫo”ά₯ ‰ŸlyYY”Ζ‚žl‡ύΥΠF:¨:x0Υ=ZΣWaH(ΛϊχίO۞{ξέ³(Έυ™gθπ¬YTsΨ0*Υ°‘²Ι5Θg±-s~:Ι!>%χέΡΉs ‚`Γ±c© ”°ΐΑϊΜ™3Κΐ‘Γ™:™%„5'”P@@@@@@@@@@| yΚ.δœfs[΄ \φΜRΫ©Ώ¦•œΛΜΞDTλΒ^Qβ₯Φ€έVφΰ—-–Ÿ~,Μ9±κ°ΕyΨͺ_w“n u’Yμΐ‚ΰβ~ύ(sΟΝω΄U«H6§Vš=ζZ½ϋΣκ>ΧkτΨc$‘(3XΨT›\#Ω\&‘)EΔΤ{ΦΉΞΛkΩφν©Ρ㏫‹° 2 ."–        ΰœŒ ί:@kp@@BΆfΟ&­<΅jC‡R7Ξ³_³&UΊμ2Š(YΣ. υ“Z΅’MšΚ₯ *‡n,ΙaτΌ5 !ΩmιRJjΫΦΫ.H„ΏΛ–ω4OŠ‹£φί~K±U«Z6O;+Q-₯gOjΗ]€.΄€e§8ΰ±— “( 9ιι΄λ­·θ ‡Q“›έ0fqUͺP£'Ÿ4L1V—'ΛPAE η“άekn»ŽpXDK‹ˆ ς;SC©οSΎ<]τΓJή²\ώ;拉x΅‰Γκ­ζˆϊ"c+U’nK–(a7±'ΨΉ#GυQ†ΓHJHF UY&‚cwφN[?f ₯~σ εηδ8žFBύϊtΑ‹/Re}5Η= b°ˆΰ“Ϊ €Α>cΜ@@@@@@@@ δLŸ>Ίκ*ΓΌ―Έβ ’s6ΤvΎω&mŸ4‰Ξ=JέψjΩ6m= Œ PΜ Hώ΄‹Σ鍕-‡CDJξψκΥ•όc"z%Τν–R:η0;Ζa!ΣΦ¬!b.Ž½¬’»v₯²x‰₯οΪE³Ω“Nm₯5’^›7«‹|ήΟΝΜ€#sζΠAOq(Θ¬ƒιάΙ““’’Μσ?jχ@IDAT[Φ-γJψIΙ5,–Αa5SΩσμ{ίI(Θlsυr.Α}β…¨luλRb³fTΕΐH?xΛϊCyΉœ―ΔΏƒ’cMŽύaπXσEτ        τ‚šLΊ€ΥBβκa’ ~*υοO²yk œs-aΨ0o›+νωΗΠΎφ-·Κ|-‹βΕjž\"rΦΏο>ΝςσΟ' } •©ι!EΒZH].L@@@@@@@@ΐf‚š«ŸF?ξΪΕ+€;;9Ϊ’¨†bΊΏp܏4x₯…γ•uΆ&kΞ8‘€€€€€€€€@°Τd9πV Α‹Š)ƒψ@ζώύtpζLMΥ‡₯θ€$M@ώG ς»ΨΠ!πΪk―Q™2e”mόψρ‘3qΜ@ DPΫΖωΣfqn’>¨δQ3ήjfTP P\μš:•8ρ”fΉunΏ]sŒUQQQ”––¦l'9ŸŸΏ kώ"‰~@@@@@@J ;;›NŸ>­Œ™••Π±1€@π°σPSΟήjjΨ™Πξwή1L5ΆJͺ=j”‘ `E '#ƒv½ύΆ¦J…ή½©Lσζš2€@(HLLτϋτ!¬ω):4O5Χάΰ­ζ"ΧP!uπ myςIΓtΛ΄n aΝ@jΉό@RTLLA‘/Ώϊj:wμXA™μ4xψaΝ1@Œ ¬™ @@@@@@@@ Dx#¨Ιΰ­"Σπ ί7¦˜”J¨W²RSιδŠ”Λkj+Χ±#₯tο.Β>€€ k&PP        άΌΤ\«‚·š‹^AŠΌσηιδςεΚfΊήˆjώΚ+¦§P  %aMΛG        ALΐWAM–o΅ Ύΐ˜€@ΐ D”,IΝ'O¦²mΫ|l ‘HΒZ(^5Μ@@@@@@@Šj.dπVs‘ΐ+€@q'[₯ ΅ϋϊk*ί©SqGυƒ€cΦ£BE@π§ &s‡·Z ― Ζσ'ΨΚ•ΙL–rXh3mXΊ”ΝvΜνtεo})ώ\Ά²mόέ-ίβ»[2w+οnVΓξά]wέE#GŽTΦvλΓ‚@ Έ(,AMxΒ[­ΈΎ«°nβH@ξύ—/_^YΊ8\?~ά/.¬}υΥW$?~¬lΤ¨QτφΫo[UΑ9π G}”rrrάφ5aΒknιΧ Ξ{ξ9ϊβ‹/θΰΑƒn''"›l6l(¨σΤSOQΥͺUiΰΐtυΥWS= Ξa'0>LΟ>ϋ,}τΡGtκΤ)ΛAε3DD6΅½σΞ;΄`Αu‘fΏtιΦ4Dp(ψœ ιΠηδŠ”›•eΉ οSŠΓ΄„ͺν˜2…Ά<ώΈΫιKΒτΛω!+ΫΔίές-Ύ»5ζοnΦ¬†ίΉ˜˜’   SPsJlήά΅«yMί½›φΌχž¦LβjΦ€Ϊ ψκ“όΠψn}}y °F ͺ5b„Ίͺ²ŸΑšφ_½ΊiύΜύϋύΛ|ͺU3―ŸšJ»ίί0x7™Ν'σΐΪcR_r-ΥΊωfC?Yό;{7‡‘Σ[l₯Jζυω!Χ=&υc€ώπαϊn(‹ξωΰCy ?kV?›ο˜υ-λ›τ/χ˜φΟίΦΌι&ΓΈΩ|γu―Ι|’ωζ¬Y}y€hΟ‡ϊ‰ζ‡Ckf(—P|{Mκ—,[ΦΌ>N.¨―z(XψT2ΔΠ? @@ώŸ@~~~Α½FF2 Έ°†  ώ$pϊτizρΕiςδΙtφμY―»Nε!o½υ–²]zι₯4…oώΥ‚η¬Χ<=iΈ~ύzκΧ―Ι5€€@q$ptώ|ZΨ­›νkM-ί|ΣΆ*€€€@¨(ΑMl›4‰ΆσvŽ“Β΄γύeΪ}& _[ŸyΖp|η֚_[Ÿ~ΪPΏ{N› Yά~¨Soε:t0―ΟύoyςI}u*ΫΎ½y}ώ=eφKRΫΆ¦υ³XXΫόΨc†ώ“8š‹™°&Bάζqγ υΛpd³ϊ"ΔΙ1z+Σ²₯ΉπΕBά&ŽN’7BΝ„²,Ώρα‡υΥ©t³f¦υ³₯ώƒλsŽ)3‘,›…Ύ <`¬ίΈ±y}y€χΎϋ υK5lh*”‰0Έ~ΜCύφή7βX\ｆϊr½ ¬° @ „΅BGŒ@ ‹€„ /3»π²žŽσΟ?ӟώIρŒ1όE·$?A+<β₯ Q­πψ’gΟ HΈΡŒŒ · ‡ρ§­Z΅r{'@ΐyηΟΣΪΫoχ€ κ‚€€@ΨΣΎΌ]1kmόρΗπX—Κ›(<„UφΈ€€kΑs-0όπΓtύυΧϋδ₯f5œάT˜Ÿ~[΄hMŸ>J”ΐŸK+^ޞ›1c-]ΊΤΫζh…B@B‹žΰP.ξ¬S§NΦάΑAΉΗvΌϊ*ΩΈΡγvh  ‘N πoue!6‘€₯J°™„– K aΠ β–;]ρ$όΧRΒ†›…- Λχ) EL ²ˆΗΗπ  ΰ1ΙΓuΕWš¨¦žΠόΤδpŽΟΆ?šΤ‹ πΎ$;vl€GΕp  <$„Τf“OΑ3CΜ@@ @:w-¦ς–WxRΟΕMΐ)$Œήw* ‡+ττ}•Ν!7%¨l»¦Κ&€€@ @X eŒ ΰ7+W€[nΉ% BΧ'Ÿ|’„„τΫ"Π‘B`οή½΄eΛЁbK@ςjδϊ΄Ψ‚ΓΒA@Β†@―d6oίχξM-8ηu\υκ!³ΆψΗW]€™*&   ~Ϋ,ό)VaKΰψργtΥUWQvvΆGk¬T©UηhΙΙΙ΄{χnΪΎ};εprlOμ•W^‘Žœ„zΠ Až4C] r-μ¬J•*tΟ=χP£F(::ZΉφeΚ”±k†σ  τϞM©ί|τσΔA@@ ς"#©Ξ­·R-Ž²ϋύχiΫ³ΟRζΎ}…2tJ¦ύΖΧͺE'L0œ‹―QΓP&ρ5kRγCπ©+Έ₯ΌΡO¨«*ϋnλW«FΨ Ioq\nfqόΫ©η S‹~ξκΗrύ†γΖΊ’r3‹είΥ L"ŽΘΈfS±"5ΰάΕz“p…f’BυzΘpΚ]ύ©ΰƒΖϊ„„iלρ  ‘J ‚4(τλ―ΏNwέu—%―Q£FΡΫoΏmY'Aΐ$–ՁšόEYXpθΫ·/Νβ€ΦN,>>žn»ν6%[‹-4MΟ?OΫΆm£uλΦΡSόCΘ©ΧTύϊυiΣ¦MΘ·¦‘ιύΑ| „Ω΄κaŊtα…ZUQΞνηpj’ϝUΰ3IόCN”ηέV9ΦΎϊκ+’ΏZ–Τ›„ϊ—s.ΛγΟƒέο½G[Y`Λβοϊώ΄ŠύϋSǟφg—θ @@@ H δζζάˍδyδ؏5PD!K S§N!;χβ6ρίέ±¨ΦΎ}{ϊψ㏩Aƒ¦˜Jς»¦M›*Ϋε—_N?ό0‰θouσ[:O·iΣ¦‘ˆ0ί Ψy¬•-[–Ϊ΄iγh jnžΪtΤ•@  ΰs¦α†pΧΫ_|ΡcQ-„—λΧ©—Ηw7Ώς ‡Ξδ;άγ Άwήy'=‰Ό…αpY± HŽVQ‡”¬5b„ίΆΓΏόB'ωΎ²mΫ‚4€€€@˜ˆŠŠ²|xΪΫε#ǚ·δΠ@  žε'˜άĞ7ož[QMίG'Ι~υΥWι·ί~#άμLnΦdffΪUΓy:dY+…C`xšΈΩ²Cœ Ξap·>σŒa&‘±±T‘OC9 @¬ deeΡΙ“'•ΝΚ{έΊœV.­ΟŽΤβ7(ΦOΤm†¬—󿐇χ]›Ώ:‡Ηš IωqφηŸ_ύErψπαΓtκΤ)’0Q’ϋ§2Η†–ά?—^z)%&&Ϊτζέιόrώόω΄wο^’Sž*==δ¦sΥͺU 6 S'[ νΨ±c΄pαBΕ“GΌy„Ρύ³ϊβ‹/φΛ4NŸ>M , ;w*Ϋ>Ž±.ω²dυκΥ+xε9T¬(Χ$Χηgy!žBTήKuλΦ₯ .Έ€š5kFMš4‘XΎ±l&αψγΫiΙ{bζΜ™^­‘7'Λ~γ³?cr³S=πθ‡~ k―½V]μΡΎό_!O@ιοθΡ£Κ&‚όm‘Ϋ²5nܘϊυλG]Ίt‘˜˜Ζπ΅r ή§yyy–ΣΡ3X,Tο?|Nση`Q3…Ώ'Β(SŠϊZ8ηΖΘ3y@£ή˜1tžΏγ…ŠηοhςΔ™-[(‹ΏSd9B%ωNJδ‘)͟_•€¨ ϊ;ξ+Χ ώμN[³†ω;©lΉ| •°“z² ‡υ,£ ;νnΌ<ΞρzŒ[œΩΊ•29΄Y67+ΑΉCγψ;}r·nŠχ„?*Ιαί rω‡Ξς–ΞίΧε5ηΜeLΙ»#7©ε5Ώ&σwŽ`#ΰΨjή|3ν‘‘Ο=ηSˆΘΓό»ψδΚ•TΦa„Œ`γω€€€-knψύχίJξ₯9sζ<igrΣ[nΜΛΝφ!C†ψμe!sόEρΗ$Ωwj-[Ά€‘C‡*sΡΝ©I)«pU’γHF—‰ΰ8yςdzώωηι 0WΫ AƒΤ‡ΚΎv’Ÿˆ=%Jό[2--^{ν5e yΥΚD|˜8q"IH?OM”κΎΑαΞƏ―ˆ-fηƒuMfs1c†ΒSDJ+1CήǏq²c˜\ΧβΫoΏ₯›nΊΙ¬[₯LΒ.J˜ΖΒ΄^xΑQχ―Ός •γ„Αޚ\ο>ϊˆ$_—•-[ΆΜcaMΈψα‡τςΛ/ΣƍέvφμYΪ³grώΧ_Uκ'$$Π€h'ρΠ –χ©άό—χ•ΪD0΅2α`&`JθNω[§Ά"fΥͺUκ"ΝώuΧ]§ˆώšBEυGΖ½α†,g8wξ\jםeυIΙ3'οw&\}τQw§•œ„…ω9‘8PŸƒ"pO:U3Όό?΄2ω[£Ξ"uε3ΘμύκΛ猻9Υί™O°όMqΗ&ΨΛςχΊCόˆήbω;[Γ±ci?œμv|ρbΪΖ^δ‡gΟ¦|‹|h%9―euώ;Φψ‰'(šΏoωΛ~βΎDœrgψ³ΌξσF]χΔ₯΄μκ«ΥEšύ8Ξ­ΫmΙ’‚²³,DmαΟί}ŸNŒΏ \Ώ“ά΅+Υθ!ͺΔʙx*ώΓߝχργjάYŒ<΄Ηk¨Νa§#|xpLςψν`|ρŽΜaΤΜNσχ½ΕΧͺE΅o½•δΖuLr²ώ4ŽA@ Θ DρoΦ:·ίN5ωϋΏ―ΫφZλ`ςΉ\δ‹Δ@@@‚žΐ«A?ΝΐMPn¨α'†EL°Λ·€žU6xL6ΉAχώϋοΫ Iκφ}ρDΛ7Vή~ϋm’>=΅5ό$­lρϋ=z(!ξΔ ΙΞδ&ΥDυΝΨ-όΤλ%—\B©©©vέœ–NλΛΝβ«ω†‡x:±Ν›7“$:ξΨ±#Mš4IyuNκȚ­„΅s7Œ‚uMκ΅‹—ΝwάAί}χΊΨνΎΌηδΖϊΧ_­δ(kΞO` §ο ·ϋpBƁΙΞzυκ₯xŽΪΥ³:/Β’ˆ"ΤZΩςεΛ­NΞɍϋ{ξΉ‡Φ]k8η€@ώ.|ωε—ʍ|Ήž"~–α§ΫX°ΌOE,κ«―œLΉ Žό 0ksγ7„5ρ˜]½zuA[ύŽxύybEύΗΕΣd«ςXύ·\ΟΓξ3§°?'\σ τη xD›½Ο\σ1{]Β7έeS[:uL…5_>gΤύ»φ‹ςο‰Μ!Xώ¦Έxθ_Ε»QΎΗYY‡”άšVu γœx8‰·š™5凩JπCΑlηψoςFώ~Ήϋέwε`;Υσ\'?$uπϋο©-†•gξώ°ώ[–o!¬εY|w“ρEpΚ²ψ© ½ Ϋ(Xj5γθ²ΥgQ―·SΫφt_5l ;ΛfοΏ΅œSH8wdΐXŽΤΰ©₯NŸNx€2ψ–ΑΰF~Πb3η,«:x0]ΐωcψΑ €l‡ψώΝ)~8/©uλ`[ζ   δηCuVryϊ›oΎρHTSu‘μJΨΘFΌΎžώXn<‹Χ›xjΩέΰΤ·ΥΛ Pρ"O3ρ’ρ—‰Χ‰NE2OΗΑΛ.»Μ±¨¦ξ1?Aέ•Ÿ–P‡ΑdE΅&ΉVm9³SQMΝL ΉΞ¦°¨Mώ_ʍv;ΑΙfη%$cˆ€c%Θͺη!"{Ϟ=½ΥΤ}‰Π+ojUΒGϊӊκ}κΟ5ψ«―pωΏγ/žφγΛηD(|zΚßυCεΉ¨ώ¦Θ{HB[mž>α―kΈ•½Ό2vν2tWŽ§κm ˜MΒ<.θά™vΏσŽ#QM½–Lώ>ς?;Ζ^σ‘dΫΨ[^&'’šz]ΫΉέvώ¬vΩv¦–±G«QΝΥF^Σψ»Ζ_έ»+a6ΥεVϋΉecqώ΄όͺ«ΌΥΤ}η±Ήο“OθOώm”Ά~½ϊφA@ ¨ΈΆήζΆΕ”)$^ΰžr­yB uA@@\ ¬ύKBςΙΝjΙ;ε―€Ρ£GΣξΕ‰Ι sρ0qȟ&σ9r€_ϊοˆkΉΖ6Lž·σ—›±FΛIθMwcˆΨ1˜ŸέΚy+‚ΑŠjM"ΊˆHkЊΡΎ‰&ΧΓ©€dΥ—/η$|₯‰χVίΎ}νͺ9:ί°aCZΟ7\ޟf―β©"^vφζ›o*Dθφ§IθΦnέΊωνοUQ½OύΙΔ_}…Σ1ρ€_>'BαsΠώ*Odέψ›bΌϊNPƒEDPsΧηϜZ†1|,ΘζορwΤ3aŒν†Θ?ž–q¨πLΒ›ΫυY˜ηύ{η&E•΅αÐÐArI’"*fTT֜]sΐ€Š9‡]]uEA]sX]ET3(’$g’δζ?_­5[]]U·Ί»z¦Γwž§gΊ+œΊχνPUχάσ$7εφΫ“>ΔTΝrCύ΄š±>ΑΉ$mƒͺDό€Χ΅a9Šυ:ύg•£_"Λ?‘ζlVYζτk[ΥK€ά–H w”Ψ4{}―gž `ƒDσšε‹ά!А   DI€5₯ (P7,j»τKe΄Φr0Ω :8¨>i υŒ 'ΜMyŸ'žx"T_‚|­C €ί#ΈiΗϋxΏj₯g‚•UŸpά(‚‹ΘΎ4Ι"¦›s˜ΐΪρ:;»R₯J‘5eΟ=χ΄²N‘yκχ¨θŠς:π§*ίΤW΅SύήyωΖ²©S§ZuΧ’|–ΥηΤ―oeΉ<—Ύ;eΑ1•σD6œΛ‚)Ž™MΏ'h/S@!Φ&^w Θm-T°Žf—g²½ΰ‚”‚jvίΆι„ΤΒΙtΫ‘Χ‘θ³₯MΊ©»4s ο9]‰fΌΉΊκ»οdΑλ―»Η½ž₯ZΘn¦ΓΆ¨‚Α•§€‘ @6@€­­ͺ™›@€ ΅Φh$@$@$@$Ό¬a†<²° ”CφΥ)§œ(m™»‡z(‡/ρ‰š#’$k˜}ŽΊNι΄1cΖDζυrRΙ֊ͺ!eΡ'τ2 QjΨ•₯…aΈo†iβ£62EΣmΙ|ΤUΗ%™c†aΦo¦|χΒΆΧΉ]}wœ}+η©œ'²α€¦Lχ]¬Χ—TfF$@ΩB ‘[²Φ΄V=H€H€H€Β¨vΓ\έξΩgŸ•±cΗ»·Χ^{Ι%—\"]Ίt‘6z:Wkd@.ξωηŸSΰ’z¨‹τΐxgΰΐ²Loˆƒ¬J•*r›JΙ©΅Ϊ΅k'uλΦ•U«VΙB•ΤωX ξb¦Ώ©ζjmAξ2‹J"3챑 tϊι§[΅©š7one²‘& $ψΒΤFτήW_}%`Ζq†Xiτ Ωzύϊυ έγ-ZH§Nd·έv“i*7„Ϊj©Hq†>pΘ ρ>βsn2|/3Ιπ„©OW½zuλsŽZˆέΊu“ XυΫ0C}"ΘPš Ώ+§jν–Ν;›6 ΅>]ŸΣΆmΫΚw:λήiψέz=`ώ{μ!/Ώό²sλ9>³Q[}w’ζΖ_*牲<B6Χ]›n6lΨΰΫmœ;찘υψŒ§Γ²ωχ<υ›’ΦιπΉC'Oύz㍞Ϋί}·TΡ@G¦Ϊφuλ|ΫξΩζςε₯¦f|ΧkfΘ ›@!λΏAo½υVΠf‘Φ₯σsŠί…C9$¦Ÿκΐ πθή'ΖA„/rν»!š”]…9O”εy°™Tγα4“΄lΧ]΄ιΆlώ=›tώ¦€›}Tώ§ιδ―ΊbΥu’ΤξΧ_Υaβg‘žWΒ}*κ΅m·W_•Z‡­B%mςΒ7ί”)ˆŽΐ,Ω*sŸ”S‰ιύtςGS­qζ7ο₯—dόε—‡n|₯zυδΠ/ΎZ*1ν΄]zsΖ‚ϊ>&[υύχL―Ό Y‘AΦώ»€₯Nτ³ϊ:Q §NΏ χ|ιΠ‘XC05Ώw—ΛI€l16ΰ7CΗœη§"½7[«Ώ…ξίj{ώ'  ΘNΈ7΅ΛOD'ΰ§,δŠ ω©—ή½{ jΖ8ƒjΞbπνΎϋξΜ” ²u:λχΉηžσάΔ”•rΎΞ.uΥ܎Αρ'-,d©dΈύ"σνέwί΅yȚCΦίχzΓ?Yg'Ÿ‘Ι‚iΘjq՜ώώ3ΜΊκ*η"ΟηhW&Xiφι%θ1j†Aτ€“NŠ(Β8‹‘1΅Τ|±τL}ς X›φIΧϊͺ\“)xΠAΙΟ?TsΆ©B… _Ȋ5Ω{ο½'‹/6mΈΎ4?§ )£•Ήφέ)#Œ1‡Mδ<‘ηΑ˜Ξ¦ιEΆώžGΎ¦€Α:­…9λΙ'ρ4ΞΊθ΅eA„΅AγΑ‚y!κ‘UΦ‰U‡ι±Ζ'ŸTΓαŒj‘ΚG+‡ššΕ–M† ²ΘrΥЏVšΥια‡Cu©@'žuWYH―Ϊ=Χ ΧΣuυΊΐd^Z{Ÿuͺ’du4+ήdŠ"ˆdλUέ`—NΒ’‘ @6°l:~cΥ`Σϋ­*:aΩ2d­ω¨ esΩv   τΘλΐΪ;οΌH΅V­Zςψγnc―DƈIjΨ°aφζ%!sgͺΦGλ „1dΐYP6[Π~ξu¨ι4jΤ(9묳%QΪSO=%5kΦ4ΊΌυΦ[Ϋ„ Κ₯ΈAiυi–Φ½@p3Θ¬ΑηήNυΫς›ιμιͺU«ϊmR*ΛΓΌ‡πŠϊs˜lη 'θ%]θφ‡Αςz:{=Œαϋo’pE¦a˜\ΠρJλsΤ†²Z—‹ί²bi7‘σD6žν~¦σ6ž€K>¦ΨŸ‹‰}ϋJ±G’a―^X'·d²­Υ@ δMgϊYYX9ζŠz=}π§ŸJ₯ϊυέeΚέt²Z#CVj»›o–‚ΧIνnΉEκ«ͺ„Ÿa€·©^S›lλΚ•ž›lW ȝ*9dΛϋκ„‘K΅χ‰ϊ˜`0Ύ#Q¦ρ¦‡~ψAdG}΄ ¦Z"†zN6 2ξύκ8ν‡u₯ω95΅₯,Φηβw§,8ΪΗLτ<‘mηA»ŸιώŸ­Ώ'ΰR–Ώ)ϋh-¬ ϊxh&™€ΫΎύΆ¬ΤIHn+§Ηξβ“ΕζήΆ,_CvΠd»wœΤ Yί³jΣ¦²»ΚO½γ“ΫŒXΏ›Φ^42λp€¬4HΊΧUΙv“Υ€•ΙvmέκΉ †QCΝ+ˆkο0Ge₯‹UΏ­Κͺ䴟‘?Gh Ή\0L8ΒΔ£ Γω j A†ϋ4Θn—MΒ/j]§­>ο‘νϋ ­δ§’boσ/Ν$5ΥCΎ\eJM˜ΘΤίΌy³νΦσ?ξL“μ0‘ χ·Avι₯—ΖάzmϋΚ+―ΘFC€Rυ5r³^~ ΗΎ> ή φΉθ’‹€Pk'j›ι/ΌπBγ„Π7΄ώ!J%J`Bo½©²Ί¦{[ŒE „DA:ήtΕ“ΘΫzŽ3M€ΔX κΒ'b(]1`ΐ€˜]Ξ=χ\γ„DL]ι3ωΐv©}ΏΙ₯v€ν•6ή¨Ζi°m­Ο„‡³5³·aΓ†Ά[ΟPYΆl™η:{!&)γz)ΘώσŸΘ₯Kƒ6‘3Ο<Σ8f₯&“zT‡7nx¬>ψΐXvβτΣO·Κ•9¬ΑK“ΪΚi§&MυΪ!Θ>ΤχΚ4I΅Θέ’οnŸ(Ο`R;Β8&&&/X° hK5(H ;ςΙ'2oήΌ@?(ι1± C ι ρ2μ{Β 'XγjA~>ΥΙQs 5X‘zΤ¦M› 7‚2¦ς4Ηι΅e˜„LŒ 2¨yνΎϋξA›Θpt4sζΜΐmzιdΈv*ίd‡1cFΠ&‚šή(ωd(Ι2}ϊτ Mδ˜cŽ”u ²‘#GΚ4U²ž={Š©œΚ—_~)SUu"ȎRx¨§ΩW_}%S¦L ΪDŽ<ςHιlΈ§@r‚μˆ#Ž=΅ήs‘Ε―₯‡ZΩ4ρΉI†I˜ Ώ—A΅jbU‚8ΘPΫΚcA†1-“ ΚGα^9Θ'?~|Π&rπΑΛΎ†{—υ~JhA%―nUΡZ yAv N 4±Ž3FπH‹ι€V©šfjkGzs’φ6ιh`ΠF•:L¨z4ϊΤΛ„|†ΩX/&Šυ„h<Ά’|έιΗΈΏDλ΄― z£dτ―7'A.βΦι—'Π§žγφq/@Ÿ‚>Z;Ο½KΙλLλ“ΚcφύΤ“ZIϋΓ<Ρ ήb½Ατ«?ͺa\%΅ Ϊτώ`ήπ'ε;;έ}χέΖφ&ϋ 'φ@ίZ|³XβΊ•iŸSgοΌσΞΐ>i+ηζΟυ’%ΠΧ_ώςίύ3ρ»ƒσι³―ί>y­ΐw%Θ'>ΏA–ξσDΠ±Mλ’8:‘A¬΄&Ÿsσΐηɞg2ρχΝδί”ΐ7’Wn[»Άxh£FΕθ9Κύ˜xΓ -―ηrχ>ΞΧγ―Ύ:p¨Vώrι₯ν@›Š†MθpΫτυa•*~W¬hτ9X―ݜLάΟ Έvƒσε£Fξ‹?ψΐΨl0ζ쳍Ύ6ιuΊΙΆ­YcττޏθΤΙΈΏΝισφν‹'θ9}ά²r₯©iY»^'^ώŽγ|¨cTIΓθGF?π0ϊΡ n£U<0ϊΡ:ΙF?`0ϊΡ}£ˆgτ£υŸ~tέθGΎ~tΰΫθG¬~tΐΪθGš~p?tν…u—0™μύθ@ͺΙM±N°4ϊсT£H5ϊсK_?θ1ξosӁB_?φ (4ϊ7nœ½Ήο(΄όθτ_*ηο»Ώ½BK*ψξoχKψμΝ}›Ζ;ΰ+ΜύξΧνγϊύΧΑ]ίvΨ+tpΧθG­νΝ}λΰ·Ρκϋξo―ΠA}£Ÿ0χή¬0ϊΡ Œ}Xί„1ϊΡ •οώφ Rύ`μΡdZϊΓθGƒŠ&7ΕT4ϊΡ «Ρ]~4˜lτ£Ai£Ÿ0χjl7ϊΡIΖφθ$£aτ£“,Œ~tςˆΡN1ϊΡI1F?:)ΖθG'ϋύθd£ eτ£“‘Œ~hτsΝ5Χύθ£Ÿk―½ΦθG“Œ~Ώώz£Ÿ'Ÿ|θ§_Ώ~F?ͺΘgτsσΝ7ύ¨:‘яͺΜύh™'£Ÿ;ξΈΓθ瑇2ϊΉλ»bόhr†qŸ°€ʞΉ3ΡLO΄ΉΎJ֘"λΞΎ93ќ˝Οα+Ιf»a¦f―`ζ f "­‰’ox2.ΪΡvΣμ΅„Ί66Eς]›'<Ξ½iΌ.Ν>™>Σψ|bfA"ΦDυζ1“Ε4σ!Ÿ‰lζσ†ΩΆ˜‘lšqœΘq“έ³M‚ ²•˜%’ŒaΏ ™5;u&:fs`S’VšŸΣDΫVΫηβw§4Έy#έ牲>zυ9]Λ²υχ<ςύ7ε·{ξ‘­3Μ+5h ξ½7]™Hύ6Μ ¬€³υ!—˜ˆUΤ do™2Όρ™m+…”k†ΜeΑ²υLV>„΄zρ]Ύn9Έή0«ΨήyƒΞ†ΖcŽΚRk9)ΤλΌΊz}XO ts΅-μMω_ ”Ζ=A“ d™ΥΆ†H πz#ήdv1aΩπ½ΘΫΐΪςεˍo(Z£Ά0ΗΕ1wμΨ!_|ρ…•ςˆΐR1MiάQ·Υν/Ω€ Ϋίk21ΝΠIdσ2ΩΆ4ϋ€³AϋˆN“L—Hdr` m†δK’R#^}MuΩόωσ]@~Δ$Νγηΐ$=ύtOΖJσsšLϋ½O.~wΝΜΟ”η‰L<ϊυ;Λ³υχ,ςω7e­JyΜvΙXٟN=$• 2]φΆeύ£Aς§FΫΆRNk¬%jυTΚ$kV c•+nVΩ Έ³k%&ηψY•ήZ¬c ›NΜC@ω*1ˆ@[ ~6SΙ΄¦*/V΅Y³„]fʐy4I8†™”?&ιE“|#˜@ζΡ$½ζ2š9ˆΩ$߈!σh’Ν 3i2&ΙΔ0΅!σh’L4Ι@’_πc’:4Ι@ΒdMR‡aκ‚CζΡ$Qh’D{ σ¨™ˆxκka&$BζΡ$QΖdM…aλI;;)9HΚ9-Μ5dM“ύd Η‚ΣDT“ $όAζ±»AΨ$ ?y4Ι[…)]™G“l&Ԛ 2&9²0~ σh’k3Ι7’­y4ΙΠ…Ή‡†Μ£I^―EˆI''Ÿ|²Q6Π$‰~i¦™QV±•A~ σh:Κ«˜ ~LMς8dMŸ0~ σhϊώ˜δΡΘ<š~Lςπ™GΣXT?˜ m’σ SΦχγ¦σœIύΒxΈι:ΐτύ³ύ˜oL2π™GΣu›ιw~ σh*%c’„L(6™ιχϋCζ“δƒ,ŒŸC9Δxέ«ŠPA‡±ΦΑιΊΧt>#œK5«/πx&Hμ ΉH“Σy~ ιτt_…ν±Ό ¬™.Έ˜ΘΆ¦ b ŒCƒzυ¦‹ΥDŽΕΆ¦“WͺΗ0ŒRυ_ϋ—fŸLΊϋΙάά€Y˜‹Θt± sc‡cγ±4YϋυΧt㜠Λ0ϋͺ„_Σ—g»ΐ¦ye.~wŒΜΧ}η‰L>ϊv< +²υχ(ςυ73κ&\sθέQά'’–κΩ·ρl0ΤβΪΉaC`S«κŒψν\5Δΐ”ίΎyΏ< °Φμœsd±ΦβY’utR2ύ ―Φμw<~՚x tpfŸgŸ•†ϊ")3M;?ρΔ‘xξίΏ$~{μ±Hό<ςΘ#‘ψQωHό<όπΓ‘ψQŸHό$[oΨ}π Ίιξmƒ^ίQ–ς=š …©RnDεά#ργv‚ΑΓg΄&d’†zΤQXT΅δϊΧΏFѹ馛"ρ£a‘ψΉα†"ρ£ύ‘ψΉNkαEa}ϋφΒ¨Δ\$~΄LA$~01$ ΓĐ( C’0LθˆΒ.ΉδΑ#UCέNIϋ $Υwί=Ι½=vΣ ΫŠ#δ θž₯χ(AR”{s      δ4 9έ»€Ξ™‚»¦΄j—G}Μpƌ-Œ™΄o fB:A‹ι&νΓ΄cΨμ!“ž€)uG¬P!Ή―») 8ϊήΔzΤbΤςΣO?Ε.t½‚Tj³W\nz™Ξχ 3P¦ u*6Γ,[Ώ;eρφ%{žΘ–σ`i1Mηη}ΰοIzήI―Ίj8κg!0‡ΙΦjνά [6l˜ŒvΝFl£3€&YΏΧλXakjΤπΪΥΈ¬BˆZbF'ωΊA@ƐTVIυžΏώ*Σu‚ήŒG•bt…νZΆΏήx£,όs9θ“O²&@EίιƒH€H€H€H€H€ό$7ξη-‹–›΄kΣΥ―Α²Η<© 2‰ λ ν[€ŽώS‹§3°f .]άθ7˜j) ³+¨–„IOίο¦:?~ϋE΅ΪΌ¦ΐΪ7ί|Υα,=a€t£Ύ“ŸAkϊ}—άt£±<(hΏpαB?—ΖεaφM5[Θ؈ά —Ώ;ξ·Λλόγή&ΥΧɞ'²ε<˜*Ÿ°ϋσχ$,©μΨn§ΦC*lΆ(lΣάΉ‚‡ΣvӚQZ^O˜l‹‘«ίώ[ŠŠόVqΉ‰€!°†έΛλ΅H§ζ:aoΊJλ}τ‘쑁h:4Φ/ϋμ3+8ά."I±0Ηδ6$@$@$@$@$@$©ς6°fͺ:1O«lOΤζ–‚›1cFhr;ˆ†b‚(ς—lf@Τ}’ΏΜ €ΟCPπ,ٚ}™Xƒ<\ΦZ Σ§O7ςa―ϋDgcγdGy€ηjΆ‚k ,πά/ΜΒ0ϋ2°†dό6ΉϊέqφŸΛ ΐ»sΫ~Ξσ 7qώžxsατ#ΧΈyΡ’€²iώό€φγN‰(άcΩουΧe—f­-W9GΤ_+2DΆ―^˜#ΧΦS΅Ή5;vt­αK |#€Zh¨η΄iΣ2ο:$΄_}υUιΥ«W™Φ/sl   @©``ΝχΚ•+εΔO”κi–¬yε•W‚A)ΌwήyΗ40ω :Χe?ϊ*X›4i’¬ΥYΛ΅jΥJ¨³™X Σΰ—_~YUι£Tνίώ·Ρ‚ά^ΦΈqc Κ,+™ϊΘ„KFZ0Θ―έŸ–8Lόξ„ωŒeUΊ)$[Ον'―yτ¦Κίo.\š~ε5 Ύ@3ŸvmΩβ{0Φ|Ρ€mEΉkξƒ#ϋ°‘κΖc—^¬;VV~ύ΅υXυέw²cύzχ.―ρ™˜ϋάs²·¨ΣH€ς›€ΪΆm+W\q…άvΫme`Γψ#Q£Ύ}ϋφV[ςϋ]aοI€H€H€J“@Ai,“Že€†dκ6₯Ϋήzλ­ΐCtξάYFŽi ͺΑ fύΣς—@Χ];αΓ‡nγ^9mΪ4™5k–{q©ΎΖ,ΘύφΫΟxL̚L΅­sζΜ1f«‘!Gq„g{:θ ΟεφBΤXœ8q’ύ2‘a~<πΐ„|rγΘΔοN˜Œδ5kΦ„~ Sύn„>Pς<θ Ώ'ή\Έ΄tTSΉρ Ϋ ΧΫψ ²}­Φΐ-II֜G*ΠZ»uυ:a[o•ƒ?ύTNψύw9μΫo₯ύέwKΝ½χvnψ|m’Χ1NΉ’H λœwήyVvΨ€¬ΫuΧ]'‹/.•~ΰΎφ…^vνΪΙ•W^)˜zί}χ•Κ±y   ° δm` V¦™Ÿι¬W†7ƒ’¦,”>}ϊHXy7d$Ρς—@˜ Κs:Λ8»λ»€4κ2™ΪtύυΧ›6‘Ν›7[³Si/nMry˜™Ω£GΟφψe²97~β‰'œ/C=ΗoΡ·:ψd˜₯ [Zβ2ρ»ζw™Υa-Ρ zXΏ©nΗσ ?Aώžψ³αšτ¨«υMƒl—žsh–@"IΒυ“''² ·uπ¬넝տόψΨΊ|ΉΣKΙsΪκz¨U“­η„ rŒNΠkΦ»wΙzΏ'¬ϋ­ζr Υά&»ϊκ«}0Žγχή}χ]y葇€M›6¦C•¬#qιWχ­Δ ŸψΘΔοŽ}žπm΄ψώϋοε’K. ΪΔZ‡σC¦|—ݍεyΠMδ―ω{ς?Ως¬’J-WΧΩσ©‚ ;TΆΩΟ*θ1*7l³ǍΪκiΐΕ8›σμ³6λδύϋGέLϊS£Ίw½ππeχh―'Ÿτ]o―¨‘ŸέύήxΓϊό- ¨5»cέ:Ω“+im_ @~@ΦΪƒ>(3gΞ΄@ΨΆ_|1R‰H§δ£W™f«εχ琽'  ²"·5‡œ[P`νΛ/Ώ”{ξΉΗΊX4½A6l“N:)PŠf͚Ϋ1τw•`1€γΒ€©·Ζ'·Ι.¨ΕΧ₯K—ΐΟ4z„xΘΜ}τΡΎόμ³Οδ’‹.ς]_Ϊ+*i΄ξ½χ^γ‘ί~ϋmΑχρωηŸ—F·_½z΅Υ׏?ώΨΈm5؟qΖ‚ΰ™Ÿ!θv«Κ0‘–[˜:Zί|σ <ΨΟ]ΙrΘ Π’#‰ί05|fQWΡ”©ψΘ#ž›’£Ν^ΩzΔΐQΊΏ'ΙΖΰΞAv¨1Χ\―ρπHΕ&hΫηδλ’Ήbξ“`φΉ―³€ 9Ft¦H`ΐfγμΩ2λv7έΰιΏ«i­ΰš±FKž€—FΉ‚©ά l]ΊΤΧ1j©…5c·γ— ΐZ9Νt«¨΅•h$@$`g­ab£Σ’ °™j8&³ΥœδωœH€H€H 4 d€δ!CƒQ?ά`ΓΜτGfIΠ 9|"Ϋε/ω‹qΰςΈγŽlΫ}χέν§ΎΓΘ;Ž3Fn 1¨”EγΫΘ*}ϋφ5Άwγƍr‚±G eϋs`‚8ψ,cύŠ+ŒΎJsƒ~ύϊI˜ο ΪτΡGIǎ™^^r«ψΞΞΦΑϋΏ“}χέWΒΥΰΏ¦€²\MφŸόGzυκ%«V­ άτε—_–ctpΣ`Η$‡hΙΘ΄οNΥͺU₯₯‘ΖΡrΝlΑη(¨Φ‚jι $O:vΟ0ίιL<‚}iO§Ό~ύzωβ‹/a>S‰9·φ¨ΦΌΉ49υTc§&ί|³LΉσΞΐνf<φ˜ό¬AZŠ<€ ᱦNͺ ²5ZΧnΩ°aA›Δ¬Ϋ`Pΐ¨ͺŸ ― _ŒΎ ΘΘZC­3/³lO€›Ÿδ£Χ12ω:Χ«½\F$@$@$;22c 3πρH·uνΪUNΥAƒ?ό0πP\p|υΥW–[Ύ A-ΤΒ“Ήgr!›3ƒκδάύrϊι§{ζ#8πΑXΗ7Υ…BۘΡfz‡²ύωηŸ/ψΜvŸd9©†*+™ES'ΠišW"[μ­·ήΘφ…ω,#ΨpΫm·Ιν·ίnΙΎΆhΡΒͺWˆ€U1ψšˆpΐΦ ‘iθϋCΒ AΚ CFμώϋοoωάoΏύΏGΥ«W·τcu Yƒo¨$SC?i©ΘΔοά¦Ίˆ΅ΦMw•;^3 P+n=φ°ΞŸ“΅Ž³Ώύφ[j`ΌwΆžρ[iη0ΑTςχ$zά7U\Bz†J)―Φλ`dΉΥΡί£BΨ²^εΡWτ“¬Πsές‚:©Ά9χGΤ” 8F³ιΫέr‹΄SYν ΥͺωbZ¨Ys5γ?Θvϋӟ‚V—ι:ΘΟa@>ΘΈβ ©X±bΠ&ς―ύΛXsχςΛ/™0ιεπ₯—^²jOy­³—]vΩeRΉreϋ₯ηœΟ·lΩβΉΞ^xι₯—J•*Uμ—ž_ΡΊˆ¨Id˜xŠΙ=Aφκ«―Κ¦M›‚6‘‹/ΎXͺ|Φ°σk―½&˜πdPΠΐυqAƒ χαΈ—2\w›ξ0.PXXδΖΊ_Y 鋝qέ 5› ƒ GΠδ)μ‹ VmC)T2 Πdηž{₯h΄Ν;šyμ§4€λ[Λ‡`Γwύj γ> }BmιE‹yΉŠYΦ‘C?~ΌυˆYρΗ ”ΐxK‘L‚iB)Κ(4Πμΰ ΓΔIΣΔ«3Ο<ΣΊοςωvΣXRLŠ'2•Qΐ8“IαΚ)¦±…ΣN;M7nΤ-ΑδωΕ‹nƒ±Ή&Mšnƒ ΄¦ΟΖΙ'Ÿ,͚5 τƒ‰΅^p;A‘ͺΉN& 2”Y°`AΠ&rβ‰' Ζ"‚lθΠ‘β%sκάχ†¦‰—(m1oή<ηnqΟqίΨͺU«ΈεΞ‹˜;wsQάs$΄nέ:nΉsΑ0½œ3gŽsQάσ?ιu…{œΥ½κ…γχ!Θ0αAό ϋόσύ Θ‚5έγ9RfhmΪ ΓiΏIφ~˜ h*ιΣ³gOiίΎ½½‹ηŒ-‘4QuΤQ‚ίΜ ΓΈ·iό%H0‰=ΘP’eκΤ©A›Xη‚N:nƒR6S¦L ά¦GΉsηΐm06‡ρ‘ ΓΚEΞMAͺwΨχ°Γ³”Δ‚ό|§j¦‰žH4Ϊk―½‚άX₯A0d?έ{ｃ6‘~ψA&h½ε C™*ΣdώΡ£GΛΈqγ‚άXυP1φdˆ―ό’5€ƒ c`έΊu ΪD~{QŒ­z& šTnΌφσ\¦Α™R΅ώσŸ¨v]&―Žκ‡§Έ   t{4Q¬ƒ—ΕϊλΕyθύτ¦ΐλπΕϊ£dτ‘'σb•΅+Φ/p±ž$‹υdPότΣO«μŸq_'k΄έΟτ‚ΤθKΏl~»—λM“ΡΏžΌ~œθ…N O=©:7χ|A₯@ͺοΉfjŸτβ.°OΞΟDΟυΦ—Q:VhΦX©φŒτΖ·XOΔ‘»£YΕ:(‘P;ρ;”ΘoŠύήι ~`»2υsŠFίy睁Œτ„Ψ7ηJ½ τ₯™˜ΞΝ=ŸgΪwGoŒϋd’ϊ―θ=ΉΨ ΣužΘ”σ έOόΧ›©Pμ5 ½XZŠuP’X'58]”ΉνžώΨcΕs_|±xϊ£Z―‡΅nΚNJ―ΏφlG&,Τ@Žρϋ―cS5pbτ££ xύh ΒθGΥŒ~40`τƒσ•ιšAτ~pΞ3ωΡͺF?:€nτ£ίF?ΈW6΅G¬~t ΩθGˆ~t€ΨθGƒPF?: kτ£²F?: kτ£ F?:jτczJ{½πϋ₯κ%Ζ~ιΐœΡN˜4ϊΡΑB£,4ϊ 3.ƒ{toά5ΆGw~tπΫθGΏ~48`τ£Α£ zύhY £ ζύh0Ηθ'Μ8—»Œ~p―j2 βύhΣδ¦X'ύhΠΥθGƒ·F?”6ϊΡ ΄ΡͺύόωΟ6ϊΡΙF?:ΑθηΝ7ί4ϊΡIF?:yΔθγΝ¦ο»N1ϊΡΙ5F?:ωΘθG'ύθ$£UΩ2ϊ8p ΡPŒ~ `τs­^?›8#ζa2MΞ1ϊyςΙ'MnŠoΤ{S{όq£Ÿ›oΎΩθη1½o0Ω-·άbτσχΏέδ¦X“#Œ~~ψa£Ÿ»ξΊΛΧΖ_£²ŒΜXΣF©"·€ •‰‚Fa’i&’»ρ˜ΕρΜ3ΟΈ[―ΝΖ¬€ ΓLš(κ'…Ιj jΧeΜdκΣ§•–j‹ΕΗL ΜΜΘCfγθ…J©4 ³13Ξ4 ΖΩΜL†„‘q³χCΆ iΆ­½­ύΏiΣ¦’'nϋ%§H ΣΎ;˜ ιQΏΩΑ‰v™©‰žΏ=F2Ϋgβy³g͚eμΞ«φ,gS6Ρ™Οό=ρΓΕ₯B «fΆ―YŒΫ"’?­¨ΏiΫCΤ.•ΞeΩAόδQc­ύέwΛΝH3j±-Υkšd­‘Ξή――3zi$‹t€%»•³}Bb’χN9 ƒ# ΘΌήȚ·*TCσ>°JuUg „‡’–ΐFf@6?ΙHΈA~ΐ”nk:$nΆ!Ή‰?ƒ”„Ξ47ΚiψνΟεΩCΰΩgŸ•uλΦ $;R±kΉΖ’«Θ€ΐšΞ,HΎ@&BgM€=γΎ¨‰†H/OΤP«ιζJ‡Υ©SG Νa’IΗ±sΩg&}wΤE{ Ο“ͺAz¦V­ZYͺL³L<jFΈ@r$SŒΏ'™ςNδ_;΄ΩOεΪ~TY₯bƒΔž‰Ny•…ΫWΟ‰cTzŠ-έUΆs™Κ6­4LΦK娕TNm― ŸΜƒ‰eœ 2“ $φ…Μ£i²„³nΆίρ σh’L4Ι@Β7dqd&ωFμ ™G“Τa?gΔ}F™δ±/ό˜€Mςπ™GΣ$€0~ σhͺl’oD{ σhOΊΑk/Γ5™Ι ‰hšζ>~L’€Έ―0Ω9ηœ( ˆI‚Y3ΙΗ™Žv}οή½χ…Af’oΔΎy„€T…ρyFΘϋ™IΎϋΒj€™IΎϋBζΡ$Ζdq]d&ωFμ ™G“μ&š 2&Y<“ $Ž?&™>“ $ό@ζΡ$ch’„Ÿ02&HψΜ£©έ&ωFψΜ£ι} γςŒ¦Ο½IΎνΣχΠ$ί?yΔΔΥ γ2¦ί_”h0™fNϊŽΫϋ†ρ™G“ό²IΗƒ“¬΄I~ŽPΙ_Συ–ιχ~0gϊ32&3ύΞaLήΉsg «0~Ό`’.7ύ~£y4]―šδα2&YΔ0~ mςc:Ώ‘=(q…”σqϊA@sΠ A8D΄Uκ[X?™&ιlwώύ‹!뀄#y腁%ίθ<†ΧsΘk₯r\ΘΤiFM(y&Ώ4λtI|ΩύΝTι¨T$Ί2΅O6sύΑ/ΦΐXŸe½¨/ΦlŒβ τY|WJ[ ξώ?υΤSΕ¦χ0Ωο³^(«&―σp ?ΧΜbH&ΫΏύ λ Ϋ0–ΙŸΣL“‚΄yfΪw'Œd…ίgΛυ°XφŠu€/π³XVRΰž ηAϋύΗ€²$`ƒΈΊΧA²ΐΛLΏQA’ΓN™π{‚φdςo ΪG)HPˆή–θuζ`½F’`4­[ ’8λUώ,h;JAϊΛ€NΊι¦ΐ7vΫΊuΕ_ξ·_ ί φAλ>VIΓΥzΏB#  ,s_₯γ΅Ž5ƒλH€H€H€H †ΐŽ;JQ’”‚ žβ£WAωdͺ+*(’&βmβ‚™(h*~?ˆό"jffŸϋΈ˜u€‚|(¬Š ¦Ω˜Ρ‡BΆ΄ά'€™ΘvA‘Ξ0³l"‹S^Qjk–‰i&©i&Šν7U›ΨΚ Γ Έ¨ ٟ˜‘‰’›ͺ•Ÿ’[ψ‚Τ¬κ5ϋιNτψmA±ΣDήΣD‘οΫgΪw²§8?%cψŽ (r˜θΙψjŸL;bf₯Φ65ΞΨ‹ͺaόπχ$ %n“.υ:σΰ‘C₯Šf‹'jU«ΚΎ―Ύ*Ν5»€–½¦²Š……rΈ^σ΅ΡŒy1lδΗ½Ξbν‘ΕΘkλύ H€‚ ;Ξ”΄Ψuχέw_ΨMΉ    HωςεEk [’’’Ȉ0°ζB‰Α½I“&YA…]»ΊΦš_vλΦMFŒaIH…IωΆ=BšC‹ .FΓδ# …@ 3©ίA9HSς‡•$5j”hqKA0֙ލΘ  žjοΈι¦›¬P2ΦL©ύι&Ο?$!W‰”όd}ΰI|΅`­1΅?‘~υνΫΧͺΧ„ΰH2t ΏMZψΨϊm)ΦDϊ—«ΫfΚwG3žD3ͺεΣO?΅>γψ¬š rZ\4«3ιο„ιQ―Ο΄σ μZ¬^.V™ε0RQσπσΗί?2\žn Uζθί~“ΦW_-ετw)ŒΥR9«#~όQZͺδ-ύΚksoΠsΈNΐizφΩRN%­“΅Ϊ*iΦυΕεp½Ύ*Τk-  ˜ΰUL›₯΄“ !7E#   D ‘Δ~$²_Т吴AΎ―SΉ59r€5γƌ–^:΄ιqΡX―^=KηYbΈΐƒ±I_9 OΛ O>mΪ4λγbPΑ< /Œ?“†n˜cq›ό%°vνZ«>΄Υƒ‚QΘ†όδ“O|A!ΐ‹¬¬L1|?ρE ™]¨)€ΰ ϋ§Α Ώ‘Œ bϞ=₯qγΖiοj ]Γ† “―ΏώΪͺU€: vMd₯@―»ΎΦ2~υŸώτ'λFΓ<νη,eύέΑgAΆ… ZŸŸ•+W j Θ vό7iλgϊ[™iηA•Εœ‹q^Fh’C_΅q ˆσ12ΤΓhΧGɞΏ'Q€―Dl]΅J–ΌΎ,ύψcΩ¨rΆh­β]ϊ=AF[Uύ=ͺ₯“AZh0YN‰P~Ϋ­z~@ν΅5Ώό"kt’Υ†™3e»^νΪΌΉδ`’V¨YσΏο›N*Δ{ΦP'*ΥΤί4  $JΧGΈ‡™©Ώ7ι0¨1°–²τI$@$@$(Φ%¦Ϋc SΕ$\sΘ8¨Ί1ϊϋί.·έv[Ζ΅ΫΩ ‚#ψ€ΐGΦ'‚X™bnց.ΣEЏ–ύrα»“ύοB~φ€Ώ'ωωΎ³Χ$–ΐ.½.Ϊ±aƒ”―REεF# ( ΌρΖ‚Q&3>}ϊΨOωŸ²šΏ;Yύφ±ρ$@$@$@$@F΅Φ|πAcΝnS³Y[ΝDˆλI€H€H€Κ‚₯ Λ‚:Ii"°`ΑiίΎ½lΩ²%MGψŸΫž={ΚΘ‘#·€ΟH ‹ π»“Εo›N$@$Χϊχο/·άr‹Εΰζ›oΌ¦‘ d7ήxΓR9IΆ5ΘV>|x²»s?   ]»vY‰%@¬ϊΕ‹GB…R‘`€Θ -Z΄°κL€»5ήύϋίNχaθŸJΏ;₯†š"   ΘΘZkΧ]½eΆZθΈ#   ΐŠ‹‹K€γ—.]Φ"CIG$.ΉδΉώϋ₯\Ήrii#FŒUΟ--  SH~w–nI€H€H€H€ς’@*΅ΦX[-/?2μ4   d Φ²ζ­bCI <{ξΉG† "­Z΅ Ώ“aKά]yε•ςσΟ?KσζΝ [s5 d'~w²σ}c«I€H€H€H€2“@²YkΜVΛΜχ“­"  ψ/ΦψI %pI'ΙμΩ³eθΠ‘‚ηŒ%cΥͺU“ΣO?]&Mš$ƒ ’† &γ†ϋ@Φΰw'kή*6”H€H€H€H Γ $“΅Ζl΅ SΩ<   )§“Εδ@$ϋ.\(οΌσŽΜ™3Η*ΈhΡ"λŠ+?U«V•ΊuλJ½zυ¬ΰΩA$G}΄xΰR©R₯άΔ’€~w|ΐp1   dώύϋΛ-·άb΅δζ›oΌ¦‘ d;wJǎeζΜ™‘φΓ?ξGi$@$@$@$*\‡T¨PΑrSPP x…1°Eϊ ,&°mΫ6λ5 @xόξ„gΕ-I€H€H ]6nά(λΧ―·άW―^] Σu(ϊ%HΐoΌ!^x‘Ρ²Υ†n܎   @ ¬…‘ΔmH€H€H€H€H€H€H€H€2Š@Ψ¬5f«eΤΫΖƐ  @ΦHW`5Φ²ώ£Α       @ζSk΅Υ2χύcΛH€H€H€b P 2–_‘        DLΐ”΅Ζl΅ˆΣ   €E ¨¨Θϊ_\9iΤ¨Q$T˜± F:!       π#”΅Ζl5?j\N$@$@$*Ζ QΥΠf¬₯ϊp        #Ώ¬5f«Ρq    "ΐŒ΅ z3Ψ       ΘU^YkΜVΛΥw›ύ"  ά%P!w»Ζž‘      @v˜2eŠ|ύχΩέ ΆžvνΪ% 6”εΛ—[KχΩgyα…[π) d7|ΎO=υΤμξ[O$@$H€RxΈ’H€H€H€H€H€H€ʎΐ€δΊλ+»πΘ$@$@ θή½»Œ=:‘}Έ1  @v dv½_l-      ΐ(Mš4±>ψ Ή     @Ϊ P 2νˆy      tΨ°aƒYΧ­[—ŽC”ΉΟβββ2o@$@$@$@$@$π?ΜXϋ >#      Œ"P\ΉŒjC$@$@$@$@ωN€kωώ `I€H€H€H€H€H€H€H€H€H€H€H ΗμΪ΅Kš7onυͺ  @.\IX‹#      @τό€ ;uκ$‡rHτ€G  P–/_.C† ΅-7" (Έ–^²d‰upΦ’2Φ’"I?$@$@$@$@$@$@$1?)Θ#t™}$    (c ¬•ρΐΓ“      $GΑ4Τ’cΗ½H€H€H€H€H€’#ΐΐZrάΈ       @Švξά)Gu”―—^½zΙwάα»>Ωο½χž 0ΐwχwήyG7nμ»ώ¬³Ξ’εΛ—Η¬ΏσΞ;εΨcYυ‹O?ύT}τΡ”έΦ­[W:tθσ¨U«VΚ~qPV i#·%   / ¬yQα2       ΄(..–oΎωΖχ8Ώύφ›άzλ­RΎ|yίm’YρόσΟwΛ–-nnj# .ŒΩζςΛ/yŽK—. lw²Η¬P‘‚τνΫWξΏ~)­[Y1L–χ#  Θ>Έ†œ?ΎΥπrεΚEց‚Θ<Ρ        DH`Ŋςε—_FθQ¬L³Q£FEκ3۝νΨ±Cž~ϊiiίΎ½ΌφΪk‚€'H€H€H€r@‹-ζΝ›GΦΦ"CIG$@$@$@$@$@$@$@QψΏϋΏH]Ύώϋ JZ<eΛ–ΙΕ_,W\qEόJ.!   °P ’       Œ%0xπ`8p T¬X1’6Ύϋξ»‘ψΙ'UͺT‘8 Ts™ΆjΥͺ’‡_fΪK/½$§œrŠœtI‘όr#   Θ' ¬εΣ»ΝΎ’      @–ψύχίeĈrόρΗ§άς’’’΄Τ(KΉa)8h€‰|ύυΧ {X²d‰ :T}τQ™={vάώ}ϊτ‘)S¦HέΊuγΦq    δ3JAζσ»ΟΎ“      @†¨S§Ž ΠΌΣ’’ƒ„ δ]»œ₯~ύϊ1―σεr|όξ»ο€K—.qέ^Ίt©τλΧ/n9   @Ύ``-ί?μ?      d)ηŸ^Z΅je=ώφ·Ώei/Ψl7֎<ςȘΕC† ‘­[·Ζ,Kζ…[²}ϋφ‘C‡d\εΜ>5²2ήΊuλΧ§aΓ†Ε-γ   Θw ¬εϋ'€ύ'     ,%°nέ:™?ΎυX½zu–φ‚Νv@FΩ9ηœ³xνΪ΅’jgρβΕVv–ΣρΩgŸ—Αζ\Ÿ/ΟΜμΫ·o\w—-[fΥc‹[Α$@$@$@$ΗXΛγ7Ÿ]'      L#°sηN9ν΄Σ€bŊ1MKUς?ωΗψόσŸΜΐΪD:θ 6φ ΤY£‘    ό@…=ε3       ([Ϋ·o—ΊuλΚ1Η#Ÿ~ϊiIc>ώψcΩΌy³T­Z΅dY"Oά2;w<ά5Χρ™KΫB™kξμΟΙ“'Λα‡žK]λΛƍ[ηΝ+σζΝd8βsΨ΄iSA=:ό?δCδΈγŽ‹«ηPΙ΄iΣbVH=b–%ςβϋοΏ—mΫΆΕμ²ηž{Jƒ b–ω½XΏ~½|φΩgςωηŸΛΒ… uτπjήΌΉ΄lΩRφέw_+[4™Ίƒ^ύE°ΆJ•*%ΝΑρ_~ωe7nœŒ?^vμΨ!O>ω€τκΥ«d>!  ( ΰ:―M›6–Kœ‡ηΜ™‰{Φ"ΑH'$@$@$@$@$@$@$@Q@` †l2g`mΓ† Φλ3Ξ8#αΓ ˆπγ?ΖμHkΕR\9iΫΆ­Œ;φΏ ώψ‹€Ih —^zIΦ¬YΧΝεΛ—Ηǚ5k&—^z©\~ωεV@*n§?¬Z΅JŽ:κ¨ΈΥ&L½χή;nΉido=τИΝ0@ˆ>˜ ’žwάq‡Όωζ›q9μ;sζLΛΕ+―Ό"ύϊυ“γ?ήΪώ€0Ή.Y€έe—]VςO€DLΨπαΓ₯OŸ>²`Αλ΅ύgŊφSώ'  Θ @­ηPΞ›QkQ‘€ ¬!P¬2ίzάΰ$άtVΤٜ•uf`%}ΤΥ™x Τg…jΥqΓmC˜ΡΏΏΜώη?cΆςΘ#rϋν·‡9<·!œ&°T³°g<ϊhΚ}¬€Ω΄…:H }ΰ?kΥJΩ/ „'ΐΐZxVά’H G`¦ΒͺoΎ‰Ό7zΣΏ›Ξμλ¬7 …{μΉ|uΈC ΥoΡ›W§νr 8—ηγσM:CΥύyή¦³ci$@$@$@$­μΐZ͚5-Ω½?ό°€+C‡ΘφU―^½dY˜'ξϊl{ν΅—tЁH˜}Ό0~ryHzemA.3Χμ½Όψβ‹ͺ99ΰsYΘ―ΎϊJΊvνκ\Uς;―ΐ‚U‰Ϊ{ο½· όŽsΟ=χmβ»™œψΗ?dδΘ‘^΅’΄8p ƒjΎ”Ή"ίlQωUχύ{² œΉΔε*T6}ϋJΗϋοg€-Y ά$]ξ[‚ζζ$@$kμ)Lͺ/tfβΤ»οΊξm]ΉR6θ¬Tηcη–-YΡv6’H€H€H€H 7  ξ’mƒtΪ¦M›¬Lη2ΣsHε3&f3§_ηρb6Κ³£GŽλ1”::vμ·<›LŸ>] ιΜ„DP»οΖoΘ!B6tΙ’%‚ϊr,=ύτΣ²Ο>ϋΔu{­N<φΨcγ|Ωž{ξΉq²S¨»Ώ‰δݟa΄7({σaU±π ͺUͺTIzφμ)ύλ_είώ·•†ΰΧ%—\"f»m€IžΌάΫy½†¬θ΅Χ^롊ΛH€"$P¬ηΝΩϊ;5BkeΞν5Α„rΪ pά‹Ÿ„t`ΖZΊΘ/ @ή(VιŒι=$;΅t—ΗΟhΣtγ—Μβαί}'υ΄(7H€H€H€H€Κ‚€Sšο€“N²dθP³ ΩgηœsŽύψίKBΟ–ΔΞ¬±&VΙ+‚ Zͺ2€Ζ7¨”7@lυκΥ1G…4γΫoΏ-:uŠYήΈqcAΖQωΛ_δύχί·j9?£+u²βΈqγ<%!›6m*Gy€|ρΕ1~ρ™„TdXΓqέvςΙ'{Β°έ?ό χή{―{+KύτΚ°»κͺ«δΩgŸ•[o½Užyζ™˜}G%wλδΡΗ{,fΉι$(ί/dŠφκΥKπωκΥ«grΓυ$@! lΥzŠγ4wΥ·ίΚΎ*KαΈ?ι"ΐΐZΊΘ/ @Φh’EΠ+Ά»Xε0v¨6ύv-l½U Y―ΧΩ†z§ΰΉΟ,Υά/Τ³VZؚF$@$@$@$=+―ΌRΞ:λ,Λq²2eΡ·ŠS%ΰœiΙΗN8Aœ2xΓ† ³κEyeΧxΫ-ΉοΎϋΚξ»ο^²©sΰΏda=)**²2Ÿ–ͺD™ΫξWY±\³Ÿώ9¦K 6”οtra5b–{½@†Ψγ:y™mNC†›_­5Θ5zΦξ»ο>§‹ΐηΞΟΏ½aοή½ν§17λO―ΪqȞC½΅ @)²ΰώ©/ρύΈώϊλcό0@ϊυλ'5ŠYτ™v0d>"CYtT²ŽF$π?UͺHψί‚€gΘLCιϋ‘©iž[Οι%i|Κ)X'§ΠH€C€g³τp₯W ,$°§ή UoΥ*‘–oϋύwY¦7φ“tφ’W]«iZΤΊΕHAŊ ωεΖ$–@-jqΡE1›WkΡ"ζ5_  @@`%lp%WδbΏœ5τ²ΞΐΒ•.2dˆ\ ΧΩ&›3gŽ@ŽΞiΞl5ης|zώ»ήΗ@’π…^°δ·mΫΧύ#Ž8BΞ<σΜΈεΩΌύ„΄‘Ӑ‰&¨fο&ξΐš—Œ¦½ύ駟.Χ\s ΰeΫ”)S,ώv?{ΉΧ… ŠΫύϊυεOϊ“Χζςϊλ―ΛάΉscΦ΅nέΪTsξ&ΘR¬₯lCϋŸ|ςIyτΡGνE‘#Ch$@ρͺ4i"‡ύuό Γ’Ν*W»TλŽΞΤοδΖΩ³γΆί§ΤΣߚJuλΖ­γ Τ 0°–:Cz ΘcΈ@i~ήyR§{w­3Φφ[ ΝσηΛ‚7ή`ΦZ Ύˆ’@S½AΕƒF$@$@$@ΉJΰψ㏷6l(ι"²ΠΒ֜9{η\ ¬-Z΄Θ³ώ—έ_ηΤC fγƍΞΕqΟΛ—/oΥ‹[‘ε Ts{τθ‘P―š5kfΙ¬ΫV¬Xa?ϋΰ?dέ™“ψlB^Ρdt›ρθ1yΫ=υΤSq.QC-(S-n]pΧ]wΕΦ°Ν'Ÿ|’p` YΕ ͺyζ2H@U Θ΅Ύβ ++ν{­υΈξΧ_cnΥ,δ_5Λ΄Ϋ«―Ζ,η  h0° Gz!Θs5ΪΆ•*ρ™jθοtέ€.׌6ΚAζω„έ'   HšδιNQI«·ήz«ΔLj#¬:YuκΤ)YζυΔΜΨύΩ;ΉdMœ81.!+ 5°rΝPΟλUΧ σ~ϋν—P7Α{νΪ΅ νiFχgY\ak^Αaψσ2ΤVC&’Σ?όp«™sY˜ηLνΦ­›όςΛ/%›O:ΥͺΗΧDτΓX•ΈλίΏ˜MΉ @’ͺ¨<λašρφύ1ΗΘΗχξ °D#|'€ΙBξLξ(˜Dα„>H€H€D*Φͺ%u>8Ε†™3γ–q     @xƒt‚N™:η:ϋωL½?~Όύϊοφ³’/¬¬&ΤαΊι¦›r’‚ͺ©ŒΊσΰO"φ΅`οΠ:G‰X―^½ςNCφ>£AΆxρbAύ6§΅iΣΖ·žΫ7ί|γάΤzή]ΥU’΅#<2nΧ1cΖΔ-σ[€Ίs-[Άτ[Νε$@¨€“LΪτνηmλ²e²Υ‘]·@žh₯₯μGT]f`-*’τC$@J ΎΞt›—Φ΅{›\} )’›6εjχΨ/    R"€ΐDνΪ΅cŽζΊY©/ά™>εΚ•ΘΡβ €νΥW_-³fΝ’Λ.»,~ƒ<_‚ϋš Θ Aƒδ<-¨AΆΡ+¨‹¬΅ σ’μέ»·ο.ί~ϋmά:d%k{ξΉgά‰ΜϊοάΉsάώ\@$u5νeλ΅ΞZͺΆSk,ξΪΎ=U7ܟrŠ₯ sκνdgH€ʚ@-Ή”ς5j$Υ¬*)ΉHkGΰ"h£Ÿή4ožlΦ‹¨λVU%'Qΰλrˆ4:ξ8)§©Ν&ΫRT$λ [΄ƒΫ֌'»t°muζSν}φ±_ώί¬ώ–hλ%~(¦O̎*Φٜ•Tr₯šΞ¬¬’™…{μ!-υf½Z‹ΎY V ΄HχͺοΏΤ΅Ϋ€7½Ε;w $ΐ©AϞLλTiά8·‘n»Z%–«dΡF¬@#ώoΣ’ρ`SIg―Άo/ U}7-BŽχ5¬m[³FvΈ€h*7l(εU2Ιmξχί½>ΩΧΥtζOυ%•Ά―_/Λ>ϋL–ώΉlΦ:[TλU›7—j:«΅ΆJΟ4;η©μšΩ›lΉ   @v¨T©’œzκ©12~_~ω₯¬\Ή2.Θξ©;πvΰJ‹―Aνγ”υΒΒΒ„†ΘBUƒ ς˜C„TR>jψ!‹lžήwΉͺ‘>]*ωΖgŸ}6Ζ‚Ώ·ί~{Μ2η wpλ‚khΏΫ ωςΛ/»‡z@«ΫζλύVXλΤ©SΨMΉ @Šjθ˜ΖoΆ―^γiέδɞ“ΐc6rΌΐΓ²O?•’>²€%1³Cοίaπ_UλL68κ(ΩMΗ‘}t¨±(‡{Ο§QŽ—ΈΗ=’χςμζ%Φςςmg§I€E,n+μΠΑ½(πυF½‘›3`€Μι%Ω4nΫΆ|ΉlpΗ°ΎŠ^Ψ Ž[ΛΛ/—j˜π³₯C‡Κx-nd“ώς—˜ΥυUώγ0΄2\xύ¦…·ηhQlΡ€–ΫΆ©τk~ώΩZ5ύ‘G€•Ά΅½ήDβ’,YΫ©7·ΣξΏ_ζκq½XmΡ@€₯lνm₯½.ZΜ»B΅jΙ2αύrΗ²Xƒ€Zuχ P 2χίcφH  cΚm‰Φ½σŽ|Ύϋξ2λ‰'<EnίφkΜΐ™¦‘;Κ*—Ώ½MΊώγx#τζΑ@― šΧqP™ϋάsV_ηΎψ’Χ&Ζe[uvςwτ›ρχΏ‡c΅k—ΜΣc›QwJΓ¦=ψ ŒΠΐκb}_½‚jžmΠΐδΌ^‘Ίίο?ύδΉIΆ.œ§υ:Fθ…χ|1λTsχ«X₯&Š† ‘―UβW­σAYQ7!Ύ&  ό"p΄ΞŒ―§ΩώNsg₯Ωλά™>π;σΜ3νΥόŸη΄‚,葇*ϋΫί¬:faƒj^ςˆaq"kΝm~r^2^ϋΫώ6«ϊιΆT3χέ>ϊ'|&e·ΥhΧΞ½(ξ5&‰coΏj¦F²ζUŸΝ―ξΰ0€:ΟQ©t?Ϋ¨χ‚₯a ¬•eƒ'€!^J>…†Z‡˜ΠϋMVω‘ďϊί=Pζa”Κ #-ŒqΌ$ %n“ι(™ιοΫG$P+ -·–54§!eg²υZl΄ΦŒ@ ΙiZ'«υUW j·!σ ΅§p,Τ£Bν΅ω*—·vΒη.V½-dώ€Ylευ&ΠiHΣoΰέΆzΜA Ιi»i{Q£ΛΆB΅δeλtΏO>Yvẙ¬Υ΅«΄Έθ"ΑΜ(ΤΫ€Wθ#.ΆVλNε'^w4Τ‚τaλiMΈζ«FY‰#\OKjν½·ΤD; Δ VœυP)HwΆτΒq!‡^ι0τuœ‡δf½Γ—Φ}ϊHuε‚:jΫ5Θ΄Q₯`π@ύ΅e*Υι4ΤN―Η#£Κ\ƒ4CA sj”«—U¬UΛkqΙ²i*U‰ £ΫΚi­”ϊ‡fΥSCM΅ς*ΡΉYg―Φώ.QY'wnΦŒΐχδΰaΓβ>Ϋnί|M$@$ΦθD”Ή β¬RΩiZξ€œγ šέoΫNΝφG`βΪk―΅Ι½NΖΓigk½]  fΨ%—\"¨§ζ6ΤρƒΤβΎz]Š¬΄ΦZCΨ~ ¨…ν‘χcέ»w—1zf$(Q·¬₯ήλΩV€mnΘctBγnRρ5kΦ΄wωœ*… oQY.Φ)ŒŠ ύ@Yψέ=Φ‚Ζθ€ί˜V£ &K»ο½1n€πΝΟ;Ojj`υκQ'e,0)|Α«―Ζβ-7ZKRτψα©P£ά{ZΊΗK’χςμ’ΐ’Ή2 N ΘcΘ({ΑV Λ‰‘@ƒZ¨ηΖ MνΚΥΤ ΡώoΏ-5]ŸqAƒ ›†={J[­‡ΆX?ιƒS†q›Κ$"¨TO%τœV_;xΨ6ρϊλγkν΅žU½C±7ρόΏK%ϊΖͺŒ‰;¨V EΠχ{ύuiͺ²*N³λcuΈλ.™ύΜ32Ek«9χEί§«Λ^š•Ɯςβν£7‹u΅f—ΝΧ Ύq¨+ζͺύ–Ξΐ;]7λϋ $­]υ?\«ωGΰrw}/– l­Ϊptuιλ`QΣ€‹P‡d …‹GŸvš΄tξ_Ocm50κg«τ’ϊ7•“p[ γσ][ίΏ8Σ`ς>ZΨ}ς­·ΚύΌ8mε¨QV=Ώ=UϋF$@$@$zθ¬z–9€½ίΥ:΅Ξΐš;Σ§@)™Ÿw!υ)EwP ™`¨¬~W_'¦ΫΠg` ΗCpψ&•>·νƒ>ΠΉ‚±“Ϊ‚d ±‚gΥ«WwζΪQGX—Ν>&“ d/LPθ˜`bχA5ΏσΕϊσ‹NˆvΥ*λ˜Σ~oΌa9Ω~πcQ…:9 –δθ g‘Jƒšχ˜Ψlj―a’ρώ·½(ξΊΗK’χŠk8€‹k. |I$Ώ!ƒμͺ@Σμ*HΡ!p…¬±*…ΈIgψΈ­‚iξA\t„±Υ3`wj6#°ξ f3}/Ίj½5Ώ |ψ.―ٚ{σŸRCkNόΣfk=Ώέϋυ“*9σ9   @@A²guŽmίιuσbEoK=Ίe Χ‰exέ`γΚ۟λυΏ; …L΄·u²κ­…΅°΅Ψόό!{ςF½wsϊqΦάΑaΜNΥ{ “5hΠ .°6Ck ΅ΧšΨ4 ά$°E3\ΗθΈΠVYδŽ|ωΨάηŸ—Uš}ζ΄ςšivθ_”Lvs>Gݎ\«ΦΌy\© Ԙ_σΧΏZΚ4Ξ}π<ΗKά}δλΜ#€‰*νώ¨5ˆΙV3gΌ€‘ ¬E‚‘NH€rΐδ›oŽ€άK³lj΄mΚίmΫd­Jά9 ™hA©σΞmρAw`-NrΡ½S’―q!4GnΫύ†ƒjΞν[k}‚:ΚΩF\AgFsχσΖ„κόχΏ»{Ύ†δγd ΒmV‰ې1‡¬>HFiΦtfγΑw}YΦ ‰ΨHλΗ-ύδ“’]άr%+JαΙV "Cςӝ‡ ξ5‹­²ήΌϋΩΝ^„ ¨Σͺ©€Ž)¨ζάί…š₯V€jΫΐwΦ“Oʞ>j/β   <"€:SΞΐZ±N~C β½Eύ«iΣ¦ΕΠ€|$&ΉξΉ@δrUΆH$¨†}©δ~*ΦP'Q«kŸͺ<½mψ-\ΈPšλ 5j~ϋν·φ*λizοƒΰšΙΊuλ&σ\?X£‘ δ”Y―ηΊy*ΌP')λΈ’Ϋκλ€gΏ Ί8oβžΪm]žxΒTsξΣR₯u—κoΩΝΊu‰Ί»–a}—8ϋΜη™KŸχ9sζX D`-*‹ΞST-’ Θbε4 …a‘!ΦTs_%ˆΑqͺ6k&•κΥ‹9δΦ+b^Gυ’θγeσ‚1ξpμφ*σΦ0» ΅γάΆzμXχ"ίΧA3―Όvj€O·₯ƒΡjΤtsΩζoΎwΣΐPφΩ[;6mryMKH~bζ›;8φύή|SjuιβΫλBέC uŸ3ΥΌBBΤmΞΐ£{_“   δ6CTΆάΞN³{jg©AiΘp;C―gh$€ «ΫKb’έΤ©Sέn~ν–uΔ΅σϋͺ󒁼πΒ CΓ«?ξ ](Glt—^‡·Υ £ΞΗdU³ ‘ DGγ_μ³O¨Η•vόH'ΉΥ1˜oτ\Έΰ΅ΧβΖ“¬–ιΉo/-9βgΛ΄nωFWΦNΕ:u€…–9IΤΌΖfPζjOnΛ₯ρwίψ:0c-ήsφ˜H Š51Kε g«„€Qƒ H!(΅―ΦsZH&`ΘzσΊhIΐEθMQοΛmN:I*ͺόe"fΊ4@γ”<„δ`«½ώA/Uš4‰[Ό- ΑG/yC\T"πX©vνΈ6x-h£AG<ΚΪ Χι–†@› ΏΩD‹ΩοZ[mƒkΆx=•aΪ­W― έ<Χ!‹―ΆΞΎ]γZΧ θΘWυx_=p!   @Ξΐ$-Θι=ι˜mŒŸωͺN଩΅` G#/ι§.ΕόˆύK%ΝS΅SτZΊ†Ž;λ½AY—ψο4‘{j}ν0†zjnϋHU& μδͺένήΞύz*Oτοί_Ά92aΠζZ+™F$L΄^7qbtΥΖ’jν΅—―OԜw[ ΰ›Ζ―άϋΰuMύmΑσυS¦όo΅JοA‘j·άrK\pύ<υΤSCχœ°ΣͺhΉƒ~χΞmψœH lTTuœf:ΦΠAΗ†ͺ4nΨ”ΔΨ8kVΜ6UuyM•™LΦΌξΧ7xL\Θ–ρ’ω*±ι.bbƒšvυ“6ωεϊΜ%ΐΐZζΎ7l @†(§šΥΈ`i¦24xtΈο>™tέu²δƒbZŽ /dμ²‰ΪŽ 5Ψ6ijϋ±ρη8ΙοΪΊ5Q—)mB³nƒΔbŚ5έ‹ΣϊΊRέΊ‰ϋΘ0LάYπ{ ?ͺδΑNW]4d2BΤγƒδ'δ1k«ΎώΟ„@Ϋz-nώ“~žΕ5`€ŒΞ?ό0tύ@|nέfW~ωeχβP―έώΨi“Κ=ΡH€H€ς›ΐW\Q’υQ;€δr~Λ­ήCς±ΗσνΤΡG-υ\uˆ}7Ž`$G§Ίx)䄦Θ˜ŽΙp_•wκš!›λvUd€Δ¨—ύώϋοςΘ#Θ?ώρ½Dφͺ-]ΊΤkΧΐeml¬χ“EEE%ΫMtΙΑ Ϋ:&8Ψ}όρΗΛΙ'Ÿ“Y‡vŸuΦYVVηƒ>θ›iΚλ―Ώ^^φΈfGΫ΄iSV>!ˆ†@-«ΡTΏŸ‰XΊ+Χ―/•T긎*%alcTaΜk\§jσζavυέ3·mΣίN/Λ†ρ’ωϊθUΓ«?φ²ŽϊΫΚΐšM#?ώ3°–ο3{I$P PλιΩ‡ „;Έ6Uλk!5Ύj³fΖ–¬μ6d‘­όκ+YύσΟ₯&ρhl˜n°}υκΈΝ*ιΕ\i[EHqf°5Π™J κ\³›z|Ώ«&3U6dπ‘^]Y]£];{ΣRϋΏmΝ­7αΫυΏΣ ΄N ‚j¦™oΞ}Ό.’§iπ9JΫ±~}”ξθ‹H€H ˜Ζ€ZΎq5ωœsΞ ¬!πVšφJaγ…!Θ“΅t2<ζ˜cδC‘ου~Λi<¬5‘Pκ¬u‚κΤ©# UdŽΚΉ¨ΧΐŸκ½™³Ξ2%›ι}έwͺxbΫ4­-|·fŠ ΪVόΨ+D]ας:~f˜ `εgΘ`KΖ^Υή$:3:αAΔaΓ†ΙašU ><Ά¨Šzx|φ)Q]a@IDATΩgqϋ`?|—Ύϊj<₯‘ DLΑ±°Κ0Qz›ΗΈNʁ΅έv‹kšΧ˜6ΚΦρ’ΈrAή``-ο?@$%ΜΪgΠ +0ζ”„D$τ₯›k}*?ΫΌd‰L›Ή%bΥ~Ϋ;—ͺLήϊΙ“‹ςάpΑA*—A1ψ‚Κ•ΣΏ(βbρ0±<ωΆΫd₯Φb0m(ξ‹Η”[o•Ί,¨9η₯Unς•ΜzH›ώόη?‹—\C7­…pamηζΝ²KιΆΞΨLwθŸH€H€H 1]»v•v:iζΜ™q;VͺT‰2qTς{κ”Azqo••Gf–ΣƎ+¨ef2Ր•ˆ‡3°†ύzθ!끠sa r~5Τa;/ΰώ1Θ?‚ƒwάq2ί₯ς0nά8Α#¬!cξEΪΛaχηv$@™EΐkΒt΅3ΦΌTŒ0.ΰgΩ6^βΧ.Οoωέ}φžH€’'€@2ŽάΆ~κTχ’’Χ‹5˜6²C‡ΠA5–jkΊk5x˜μŽώυΧ_ι|²}έΊ8χ•JQ^'ξΰΎ’ ‡}ρ…τΠ¬΄6Χ^+U[Ά έbώIε ~Φ›mΤ[H·ύͺō—ώyάaΪkΆe3 žˆνpΥLdίDΆe`-Zά–H€H€r“ΐŸub—«5€™ΝθE&Ώ—AΚr“Z%j½{χΤYk©Χτ'žx’TTΕ‰T ΑαN:yΊtcΝ$χ­e φΧ{’d Ώ1cƀԎdΝύH€CΐKω%Υq-Λ–Ε5Φ€4”Mγ%qγ‚¬"€ ρYZWZώ$*cΖZT$ι‡H€ =tπΧωΦ6θϋ/*ρ±Σ5kξΚUͺde ΥVƒ­ΦΊ΅TΧώh1φΆς7 ^³J»]™~ΌΊ(xμύΜ3VΝ<°V¨Τ'²Σvx+ύYτΦ[RUλ›ν©r‘ι²y:Έ0ϋι§γά7ΦqΗˆ[nZΰ5[ ϋμύάsR Ÿι¨¬šX¦‘   δ7֐)δΆ–tŸ―3—>3¨]vΝ5ΧX²ˆA-…$瑇*‹tΚ;’vߐ!C¬ΊeSœT†ΰΥwά׌Λ/ΏΦκΈΦz•Σu&dΧΩη"Ι–ρ’˜Fσ ``-W‘ @2¦im΄][ΆΔξͺ³P_Κm^uΧjvιβήΜψzž£.-ktβ‰q‡‚|a’΅-ίΕψfΓ£ŽY–m/ͺiύV7ιl/ϋ±ς»ο’κF‘Φ7pZΕΒB)PFQΨT-Ύ^€ΕΦέΆ{Ώ~ς’Kά‹“zνυyΗ,ΆuI7ͺ¬ΔŒώύK˜‚νV-Ό^ΓΕ(©†r'  ¬&π/½j§“QπxτΡG³Ί/l< @ώ€“ΪvΊj,χνΫ7@°Η$@ΟήyΐIQžόΉήΈγξ88ΰŽή;‚€(Ψ°&φnμ&ϊ·%c7j4ΆX‰£DΕ{ο•ή‹ ½·γ€+ΐυϊž•½ΜΌ3»;³;[οχ|>ΓΞΌσΆωξ°7σ>-€ΜήΧ7<ω€_sΨώ曆v8:NRΫΆΊςhY/ΡM ΰ…k^ΰΰ€Ψ!Πΐω€°uα&~9R₯;'7Λ-•a’»bΝ΅ΉΧγ=ίOfnχ^p²ϋUW‘„ Ԋ„ψ“`VEκͺ^}β%•”•e΅‹ˆ¬'Š/ —©Ι£ΧάΤ€-²΄―†~Μ>τPKν|UΪΖa&Χώύο†jβ-8ψ‘G εώ΄eqžšΈ=εΦ<ψ ν.E¨z7v½ψbǍΆ'„  C ¬¬Œ$?‘l{ύM1‚‰€΄:UUUτ―ύKwέΗw •0m"’Η‘Σrh₯lώ|*™=[[δs_RLl}υUC½n—_n(‹†υΓ€Q^@±ζN€€MυυT:w.ύΘnξΫMrc%pŒόχήkΪ•Y2Χνo½eZΧ¬°”|ζž~:±™£ΩiŸe œγL•šβb΅HwœšŸO]X¨ΚΒ‹.’ς₯KΥbΓq]y9-bE£*ω$ Wϋ η±ͺ«X½šΆN›fkJ{~ψjΩσM+9(Φδ~Ylς€›9`b+3Iμ€τΉεCwrοxχ]CΉ§‚νoΏMκ‰ψΤTκΓήuˆ΅΅΅ΊiΚρ™gži0ΈυΦ[uυp Α Π†=ύΝΦ`œ{­” ΛoΊ‰jvμΠM1kΨ0ʟΗz‰?λ^ξωβΌp&¦”·p@’„ΐΖgž‘€œK³mβά`βYΆŸγH‹·V3+Χ<ΙΐΏύR;u2==b„‘|η{οΉΌzϊήvι-ˆά•λJKiνCΡΊΗχ¨T«Ω΅Λ]έγgZ—.†s[¦N%qΫχvP<›Δγ¬nϞ–φušoζ€I4φ³Ο¨έα‡·”kwJfΝ’Ej°†s hEςΟυŒ‘'έ―Ό’\‘1Z.Q”Y•œ­=χP<Ηχ&{~ό‘ζ0ΔΗSG΅LWΑχA5?μΞ=υTc˜Rξ»7? [QŠz₯½κΖ•;²²Tζ½K›7‚•ΐσΟ:‹ϊή~; Ό~ŠγρΝD,ί~Ύα’ϋQ•!|ίgτμ©γ@@@@"’ΐ6dkίΎ=υξέ›vπsω‚ H<Φ΄r8ΏCMβχ)€„‚ΐ`N· šͺ«[†«ήΆ~7Žϋΰjοεχ¨‘δ}}σ /΄΄uνπΦpNβι=?λ%ώ{ι/ GΡL ‰£HυλΧΟu lPΎš ΰ(֜ ˆ>@b‚ΐzQR9,9μ]οoτΨk‡c₯\XΚ '­H軝ό Σ‹T² r)όδ§’sK}ψ!νϊόs]hΌτξέ)­°J4ωΌΔSjε]wQϋ£¦&~θ1³’Χͺ³bμ VŠS ΏόeπΛί`%gIJ^ 琗σΟ8CΧΌž½Ρf°‚%‹Γ—δvεŽC œWΩ–/Yςk^/%,’œΓΚΔDφμ‹‘ˆ}nΎΩ₯ψlΉV&­yΰ*ώκ+κvΙ%Τ† Λ&ί™(e«YΡXΖ/Χ’+oΟwί'¬ki*;ύX •;z΄ΜξδώS½ΰ\}πχ±δŠ+μvg¨*_£ΩΓσΘ—_¦X\΅y³ΝZώΏQόε—”7~<‰η¦l’›pί²e­ψ‹/ m€ƒ‚³Ο¦žW_­λ      Ικω™>GΝLΔ ςI?σ™υ‡2πE ‹Χ|ΔhϊηλΣUmΨ·f<ΙzUΞΘ‘$^fχu^Wƒέ=œΪC’ςToέͺk'=―½–ڍk(w„c½Δίu/χœρύšyMΒΖ‹Δ{0ξφη*‘Xσ‡Ϊ€€€IΩΩ4ˆ=Κ\q«½ΤEΔ‘ϊ;v•od­”/\H‹8Ό’/₯ΪxφrO'­bMΪ­ao9ΩRYs+ζTιpΜ1”ΜΚ3­η™Τ©γ%%3fΈͺηy°R*ΰ”Γž}––‰§™FΤ̞Zϋ/vm›ψΌ/ροSfώΎͺEΥωώwίMΒ°Š‘Z‘οT6·HHCQzjωΉΟΉ?sX9)žnŠkœ@;ρ£}2{ŽeEπlΞαV½e‹χ}’+τr ΚήCψ~€€€€€@¬HβˆO<ρβΤ%^¬«ά°6(Š}Ιq.FΧ²Y•ΒσΞ£‘J?fmC½^ΘΊ—ΩόQnζρ—άgρ   `‹@|ZšΛBηXΆ„π₯Tsw,!νF°§RBzΊ»Θςg!η:›ΘVιέΊQΗ“O¦8aՎ$W{Wω+β94‚Cυ‰Χ™]wό‘lεTΘ1ΌcMψ>ΓyΔR Ό^šxhySͺ‰·αhVΠy Λιu€9)–p9av‹].Ό&Μ›GIYYrU˜€€€€F sηΞτηWώΏ ‹ ΄‘¬ΨΐΩμΚγίπμqΫύͺ«h$›E±Q; υzI λ^κόq nπXs“ΐ'€Ψ —˜HΙνΪQ2‡D”°ˆm9ԝ„ZΜγ0ˆςGΫžsεp¨Ώ₯Χ\C»9<žWα‡–v󺇋ԆwLαωφΡGόU••^»Πžμ{Λ-$v?sΘJmlmmoϋΪP,€$iνΞχί'ρXσ&’ΗNςΗυβpώ°ςΦw$Λ>δšΔήiΛτ'ΪρΞ;>ΉhηžΑ‰„‡<φ˜+םΆ<šχS;v€‰sζΈΒ]d>ΙΙgEδ–(»°υ@@@@’‘ΐT6FœΛ†feee”-ΡM8άΏl’s-‘ί-!  Nύy}©γI'Ρς?ώ‘φr4$«"ƳÞ~Ϊ• Δj©κυ’@Χ½μ\κΆqcRŸΘ₯υ\;@"’€δO+=›φs^2Ω8D€δβJg/‰k-ž;&ΉΡάSΙΉ¬δAhί₯D¬„Kc―)Qψεψπ’q¬YCΌΥ•”P"{‰'\6+6¬z Υr»]L%<Ϊ]»¨–•'):(ŠΪάdImΫΊ§Ϋ*>«8 ηφ<+ε—i )\&+Ξβ΅θΪzυ’¬ΑƒIB(ΔΫτ>Œ&œy7Ηe/bEp9‡ ­)*’:^dœ~rΏΚύή†Γƒv9|WŽΑhΊ6Μ@@ ΄}τQΊ…„DnfC 9Ž5™2e ]§δ?‘k9Q–Ž5Ι'5†ΓωΛ9D#r^KΪυ駴ϋλ―©šΧ2jŠ‹]ωΠΕΈάυΞΞkS’M’&9‘Φ#”λ%N¬{EγwΪΪηάΨΨΨbΔ"9ΦδΨ bΝ Šθ@@lhβζ’`²ͺ΄΄Ω=ͺƒ€€@« ή{97¬H{Εηq$X(ΦbνΕυ€Δ (Φbε›Δuψ"ΠάΤd)Μ£―~¬žΗz‰UR¨g…@°kπ7·Bu@@@ΐaβ‘Λ^iγBw   `J@”i²A@@@@‚CΐJξ4'GΖz‰“4ΡW°ψ™•0XΣAΏ             ™ΰ±™ί f            ΰ'„„Z³f«u\\œŸ½›A±fd‚('Π·o_Η―‘ GŠA@@@@@@@@@@@b‘k±ψ­βš@@@@@@@@@@@@'€PŽ#E‡       ΰ ζζfӎ֭[GoΎω¦ι9‚€Ÿΐϊυλƒ?FˆHP¬EδΧ‚I€€€€€€‘§$λ_ύ5Ι-„‚ -oŒ      ΰ—^z‰ϊχοοΪ}τQ‡zE7      ΰ™<Φ<³Α&PZZJkΦ¬qΝpχξέ}θάsΟuΎcτa °mΫ6š5k–aδ‚‚?~Ό‘ zχξΣƜA@ & HXuwθxQ¬9%P¬9Eύ€€€€€€€€Γ&OžL²A@ Όώϋ¦Š΅Ρ£GΣ›oΎ —ˆkhœSΡ΅XΈDΦKŠ΅ΦϋέγΚA@@@@@@@@@@@l€bΝ,Th½c­υ~χΈrˆj—_~9rΚ)khΧ]T_ &      X‹Žο ³Pδζζ’l„‚ iŒ            Υΰ±Υ_&             HHH U«V©ΕC±0Bt            iϊχοοψ” Xs):πD`ήYgQνξέΊΣύξΈƒς;NW†ƒΦK`σ‹/ΦiΣt>π卧+Γ€€€€€€€€€„ƒk᠎1A •(›7ͺ·mΣ]}νWθŽqΠΊ TmήL%Σ§λ Τ•”θŽq         α"ΕZΈΘc\[jχξ₯ϊςr]›΄ΒBJHMΥ•α@@@@@@@@‚EŠ΅`‘EΏ   ŽX}ύ΄ρι§u}9s&΅;β]@@@@@@@@@ XβƒΥ1ϊ`˜:u* 8Π΅=φΨcΑ }ƒ€€€€€€‹<Φp#€€D Ά#FPΧ‹/ΦΝ'½kWέ1@@@@ΐM „s±Z΅ΚuX\\μ.Ζ'€€€€€kAC‹ŽA@μ(8ύt’          ‘HŠ΅HόV0'Ώ 455ѐ!C\νγγγiωςε~χ₯mŚ–φA@@@@@@@@@@@’ž@ss3­\Ήu’XsJœλΙ©‘Ϋ««©©Ύήv»@ˆς’‘ͺ*.Π’Š<Φ’κλΒdAœ$PΰρνώϊkͺήΆjvν"yψHλ…»u£μ#¨πάs)%/Ο±a*+iϋK~ω…*7m’ͺΝ›©zΗJΞΝ₯΄‚JνάΩυ™{ΔΤρ„(.!ΑΦΨ5EEt`υj]›ά±c)!5΅₯L{ΛΤ©TΎx1ν[²„šhΘOPώδΙ-uάsk)ˆ‹£φ'Άʎ<4•ΜšE;ί{*Φu]G#__J~>₯vμH2nα9ηPZa‘·ƒΊςrjΨ·OW%₯CJHKΣ•ΉΒ5OχψκgS]ω%ν>έΕ£Ύ΄”Rω{ΝθΩ“2  N§œBρΙΙj³;uoxšl0ξmuΞ5Ϋ·†—ϋXΈ»%)'‡²‡wϊό Ηο€ΟI‘€€€D0W^y…N;ν4ΚΚʊˆY~τΡGtΘ!‡PΧ]#b>˜€@d¨ζχΙζΖΖ–ΙΕσ:I*―_h₯’ΧhΆΌψ"νgο–*^³©ήΉ³e½&o$*8γ Κμί_ΫΔηΎϊ^+ όY§ρ4¬£ώ9}ό1•/ZD΅ΕΕΤΐλ>"ς~,k2ν:Šςy©Γ1ΗΨ^gς4πάωΑ΄σΓ©bΝΧΈ²Ύ”ά₯σHF―^”Ω·/u»όrJwΰχ9ΤοξeΜrχ7ίPεϊυT±nλ³ŽΧxδϊ’y0³_?κpάq”όρu@’†€,ΤΓβ±εK©&ΥΜσEόΒφ{_-Ώι&ΏάΪ·Ώυ}έ»7­Η?Ό*]Tˆςΐ·š0˞N%s樧m―yθ!šΝ–4ͺRΝNG 4μ³i>Ώ [κ‡BvπυΟD’€ •„ržεμυχ'A•οΚ—΄†­έ~ΉυVϊqΤ(:ΐ^~.…₯v $Ν8ώή‘rok.Εn8~,M •@@@@ \ΖΖmΪ΄‘»οΎ›Ί³ίύχίOϋχοΩΜE‘6‚£‹œzκ©΄„ŸΕϊΧΏ†ll  {ΔΣ뇑#U€XXp/b/­otyjJΓίwρύ+VΈΦŠVίse₯šΜU’ -8ο<šsI$ 2»"kSί°§–ŽϋR@Ίϋ–΅·MΟ>λZΫτο»‹-}†ϊέ}5=ϋ†=eΛ’RM‚ο™Ν/Ό@ίr»ωσ-]—T εX–'…ŠΆ@±f *ƒD3ω£%ή/ ώΌτ±βc=[$‰‚­^ U草„\tΙ%–8Μϊ’πŠ³Ωe_8ώŠX­dλ @DΒdΞ9ωdΪρΞ;Ά»‘— Ω#NvΑv'„ržςP9“C*Hh;"³ΣǏ§Š μ4 J]οHΉ·νB Ηο€έ9’>€€€u²ΐΏ‚—d5€@π $³χx¬‰”••…DΑ&Α–ά!έ 5‘ ‡6ƒ€€€?Š>ω„ζž~Ίνu’&^™wζ™΄ε?ρgXWίΕEy3}ΒWj—΄(bπ+SVeχ·ί,φς«Ϋ½Ϋj]½¦ΪZZϊϋίΣΆΧ_Χ•{:υ»»―bƒ+FψfsCλYΚΪ](Η2›+ʜ!π«œ3}‘ˆX«9¬‘όTE\σXΑ!ωΤdKΰ0"C»ŒT$w˜ͺ„ΫσΟ4—-#ηžΓαχώτ“]6?ˆΙu€³₯jbF•Ξ›G%†₯†σΖ©RΚωΨ$B›‘ύΤ~|‡jžβq&ίI#{ρ©’Δ9σ$†Ή‹Kf¦λϋ,[°@vBFΕƒ0œβο½ͺ{[ΒTtΕτA)γϋλΐͺUξCΧg>‡j•<|n‘\vž$Ώžζ‚rg΄γ<²A@BK@”Ϊ>ψ m;κί­`{‚s7ϋ"Σμέ 5 ωΈTyG’6πVσDε ΎTrή¬瞫3„–χΙBφθΚΰ¨C’MςˆI~­­œ[RMm"žJKXQ”5x0επڈρχ]\"‰rK]«"o'ωΎΊœ>e D©:‘δ$“΅1ŠέϊςΛEœΟεuž ³gS"{!{“ύό>η7Ώ!Q(ͺVr\^|1΅ιΣΗ•{LrΣɚ(οΚζΞΥWg#‰eΧ]G&O¦ΞSζIBύξ.σ]l²VΥξΘ#©§SΙΰkK+( z6Τ―diΩ$ZρgŸι.‘ ρ垘δΕs-”cι&‡Η ΗšγHΡ!€€'αΚ±VΒ Σω‘κ¦ή†έ΄G½ρeσC€™ˆηӊΏό…6>σŒαtŸ?™s,ko²τškh“ΔœΦHΦ°a1³8l€7ΩΑJ½ωσZσ‘|-νΨkΞ“˜ΕξN“άvšœjY¬Λη‡ω₯X<+%ωjJϋφ-έr—΅œωu'gΜΒ‘-Ϋ±…¨™¬γs+L¬ΖεAοΞσζIΜΖ΅•cMι8Xσlβ„ΌΣωΪE«QΜβ\k’œWͺ”±’SBiΚƒ¦'ΑΚΗPεXσηސy‡γή–q—qœΗN+G²"ΧΣ}¨­ίν°   ±D@r­]ΝΉƒΝ$''ΗTΑf5ǚ/…šŒ)ήjΘ­fFe fΤkΪ:b-ωΞ YΡΗJ*Uš8MΘ¦ύ‹~ΎρFΓ:(αŽZΆŒy=ΐLœZ§iζHJ38§x +Κ΄’ΒJ΄C§M£G­-ΦνΛoͺx€­f#W* ΝΩΎζΡoΎ©)ΡοΚ΅xΨa΄oρbέ‰ψΤT:τΥW©ΰ¬³tεξ™ο^Oϋ…£'© Ή^œ£s(G…2“pΌ»Ο`v{•\νΓωo\V’y“T…ͺqωhŽτTΐfΚ±ΜΖoeΘ±ΦΏu\3€@ΐD9&!U•X MZ΄Θ£RMNΰ«aΌˆ?τ©§ σΨΐρ€ΕΓΜ›ˆ‡’V’Ω«f+|)Υ€Mη0ςΨcΪζύR?r­΅(Υ81mŸ[nq]·(»2·ΧœV©fT)θxΚ)4ώ‡Ό*3ϊp>ΊήΌ©²‡ΫΩ ₯©Ά·sΜynΰ{BUͺ%²gšx2φΊώzS₯šΜ=ηΠC]όσ9žy$ˆΏχF€άΫV†σwΐκQ@@@’€x­uιΕtΪn6»9ΨdρχΓ?tεP;ν΄ΣL½ΤάΒ[ΝMŸ ˆηhD‡ροŽx{™)Υ€οψ€$κΕ‘€\ ¨„έpβω΅ΥfHHήΕE±§*ΥΨΣlάwίyUͺΙdγx=hGqε‘*’O¬\Qšiλ¬ςIƒRM’?Ξiž”j^XφfγX³΅­ν¬ΘΕ›*αxw—”%κZ['Ž>εK©&sοΜ§ΖrŽ>¬»”œSΞLB9–Ωψ­΅,ΟΊCΗ/_Ύά1 FΌc]£#?­l=£z₯χθA‡πΓ„'k"uΦ’(ιΔ,΅"Φ6λ9Μ‰'iβδ¬ϋ8l€V\ ξυΪϊfΦ-7zmϋ£ί}—?ό0Ε'ϊ8›ϊŽao:Q:ϊQή%kΌΰ€~3[:Y‰7ν«o_ηƒ9OyŚ*ΓΩ;QŠϊ’dΆή=Œ­š$Ld€ˆ{#οm_Γυ;ΰk^8   ΡL@›kΝΣu¨ Άͺͺ*Σͺvjr«™bD!€€ϊήz+εst+"Š€ξ‘FρΜ’ί1»bυ]\ϊ6[ƒ’HBY^!¨σ‘θ8M<©Φph_3‘υlXJoφάk?q’Zlzάƒ=›sΨγM+΅l¨.iDT Η»»€›P=κς8‡U‘΄2γi5……»―PŽεŸΏΔ!Reθ#‚˜^P¬Ω‘…Ί QEΐυΰΑ–5ͺˆΔͺRΝέΆwΊw[>w}ϊiΛΎΊ#J55ᩝ?Μ_Za‘+D£ΆοΪ={΄‡–χεα―€ς*#Ψ K,΅¬ˆX'u<ωdCΥΊ½{ eNsžEl€ΖV—˜β…lέfU„a?“{Κj{'λΩ½7"νήφΕ"œΏΎζ†σ    νΌy­i―Ν­`ϋηš6“8²…/5m;x«ii`@ΐ_IlψΪ›ΓΪ‘ύ+I­TpŽ.5²‹φΌΩΎwρbŽŽ#9α΄"sοzΡEΪ"Kϋ$€"Π,ΊPΡ'ŸθR‹H3I%bg=CΌεz˜όφKͺ ­„λέ½Œ£Y©"ΉυμˆD%#~χ&Ζθ &†$‘ΛΞόQΧ?P¬ωΗ ­@’€@)η#«X½Z7SI<*ωΕμŠX ˆ”V¬\IΥ;wj‹ZφεAc'‡ΥnЎˆgΩƒ>€Δ½όθ£v›κ »ΆC†Κ½H\Ujƒ¬X φ<εSρF”‡E;"Ι}Σ{φ΄ΣΔρΊώά‘to[Ξί+σCˆfVΌΦ΄ΧWYY©=lΩί·o_ΛΎ―x«ω"„σ V tαImΫZ­ξͺ—Κ9ΝςO8ΑΠFή=­ŠέwρνQ•Ώϋ₯hBj;IO’©ΥpXF3²ωHI»‘Δ©0μHGα₯¬™μ_±BΧEΈήέΝ οeέ§Ό\7?o=Yq8yγFέfΦ―YY°Ζς6_œs†kΞpD/ H`―’ΠU¦˜;fŒί3Νγ$±ͺ”Ν›§ΉŽ3ΨR₯+N΄[+ΈμΘޟ~’ζ†;MLλζŽKιέΊ™ž³ShΥΝ_Ϋ§YξΆΊ’mΗχƒ=ΟΉs snΤQ†2_’Sςά…SόΉ7"ιήΆΒ.œΏVζ‡:    ν¬z­9uπVsŠ$ϊ°YΘM¬ƒ‰ΑΆš§Λ]ΧμΣξ»Έ™ϋοoΦ΅₯2³Τ#fοΞfιHΤ°‡VLιЁ&Μ™CγyΛ½©ž‚fγ‡b Ομ¨b%ΩL^΄λΉζ‹E(Ης5œœk3D J dΖ ΓΜT―3C/YƒΞVnΪd( €@\ί«Άn₯Ο?O l„τ6¦™Χ˜·ϊžΞeτιγι”ΗςΔ¬,ΓΉ¦šC™“Αœg-+Υπ .W]ύΊ„pηYsκή°rρΑΈ·­ŒΏV u@@@@ RΨυZ dήπV „Ϊ‚¨ς82?ήΔπΊbνZΛ]Ωy—” •λΧλϊNγ5;ΉΥtωΐl}KΒYjE’ UnΨ -’86ξ`1!ˆ’Lx»75"RΈήέEιgπΰγωξ[Ί”ΎκΥ‹]r IJΧκuz;εXήζsΞHt¦τ ‘G@rA©r€CCnž:U-Άt¬>ΘH£ͺ-[,΅ΥVj¨¨ ™[ΥζΝ-[εΑύjVͺ5ΥΦj«ΌŸιPbΞδάά€ηŠ‚9O±ZRΕΞ±ΪV†Γ)Nέξkυ½νΧΫg€ώx›3΁€X'πΞϊψ㏻\Β 7έt“υΖ¨  ΰρZ{πΑiΫΆmŽυiΦΌΥΜ¨  @ΐ)Ρ,ʎ•Ύ ΥκΚΚ ež μΌ‹X΅ΚΠMZ—.†2;’ΰQ₯΄TWd6njηΞ”db@­kθηA8ίέ‡M™Bs8OZ£’­™S΄l}εΧ—”D’βEˆη™(32l_m(Η²=94°EŠ5[ΈP@ š¨2χ՜dΦIi8pΐRw%³fΡΟ?§½œ”[Ϊ:βΡΐ\)ΡfμkOύJςΥh`ΞΣμAYϋ+ιaV¬9qo„σήΆΒ=’~¬Μu@@μΨΛΦΤ+ζθΨ΅k—½Ζ¨  ΰ·ΧΪΥW_νXŸjGπVS‰ΰ@ ΙΌΛKͺVͺ4jΌ˜κm(ΦμΌ‹›­C¬XΛΟ7 SߝΝ'9/ΟΠΞ©u|ι7Tkx’RdμgŸ™*ΧάΧΧ\_OξSΆu?L’h/<Ι9Χω΄Σ¨Ε(O‘Λ=w|‡BA‡+z3Ζκjjβ-ΨβΛ»¬zηNšwΦY4}ά8ZΛœςΨͺR-Σ$τd°―Η[ρΙΙήNGΜΉ`Ξ³ή$ym Š΅@†Γ =ξνHωηχ„±A@@@ T‚k ήj‘ϊ&1΄ΌΛ !5ZŽ―υ!©š)ΈτX3σ:“χg­˜­ψλα§νΧl?ήέEα5ώΗ)﨣̦h(E[ΙΜ™τΛ_ώBίτνK?qνxχ]C=³‚PŽe6>ʜ!Ś3Ρ €@„p"φ±•Kςφΰ$PΏνߟvZόΓŸ’BΩ£FQΆς?}:³|Ή•) N ¨š2tRΫΆ~Ο 1Θωζόž˜†ΡroGΒο€”8    1Cΐν΅Œ ‚·Z0¨’OhέρXrͺ‡U Š:Oί†Y€€δvνKNz4y εWΑ‰e]z)5r>5UβΨσKβ2gαJ›ή£eπ&ŸρœΉΜξ+υaΪΞμkMfν΄GέhΊ·ΝΎ/aͺ߁p|?@@@ΒI XΉΦΰ­Ξocƒ@lh Ώ½„gΤ†Bͺ›SΤMršI™@€fΗCσ€μl]YBzΊξXΜΌη •ό(ˆ΄wχάΓ#Ω†=σ Iξ·έ_M{8΅‹x¨5˜(΅—ΌύυΧ)­ €sΈH+Κ±¬Μ'눒Σ:>>ή9?3¬ΰΖβέ‚kp)ΟΤxΧ‚₯=»tgφλTBM ΄-T J΅„xί}Τύͺ«(%ˆq©ƒzq­ΌsυASpToέκ7•Ϊέ»ύnŽ†Ρvo‹=\Ώαψ~0&€€€„›€ΫkΝΙ\kπV χ·ŠρA 6 ΤνΩγχ…UoΫfh,Ś™g] λ2q3ŜΊNeζVerέ~Dς»{Ϋ‘CIΆ>7ίL͍TΆhνώκ+*ώςK*7ˆΛTYχΘ#.Ŝδ^³#‘ΛΞΌPל€s*:σώQ  a#`ϋΉbνΪ ΟG,YΚ䏫FβΨmτ›oRΏΫo·¬T%$²˜…v0{ Ά:λςΕ‹­VˆzΡxo‡λw "Ύ0L@@@Β@ΐι\kπV Γ—ˆ!A ¨uX±–’ŸjΙ&†ΩU[Ά4VΥ¦M†φm‡Χ•₯tθ ;–ƒ@z†5ΡπξΗσΉ£GS»ξ’ ³fΡρό=τ»γ".Weχ7ί¨EΆŽC9–­‰‘r (ΦZP`@ ΦdiΈ€P(ΦΔM\•ξW\AkَToίn§:ꆀ@NH+JR­Tΰ±VτΡGΪ"~?ονpύDό—‰ ‚€€€@Έ½ΦœθήjNPD fj8œbMQ‘Ω)ŸefkKΉcΗϊlηOρbR7•7R ΖΨEl˜J+Œ΄’ΝŠΆΈ€$m‘+ bνή½Ί2+υϋφΡ‡))τA\\ΛΆψΚ+uM£ρέ]B>όΫίhΤoθEΚ.4”RΚ±™gkj ΕZkϊΆq­ ΠΚ΄?ήpΕ%3fΚ¬¬ΌσNϊͺW/έΆΕ CσύΛ—ΚΜζb¨€XΉR)ΑaΈ $rŒρ,Nxͺ•ϊRW¬mm™•}ΙΝΘύhe §λDγ½mφ/ξVœfώ@@@@ š8ε΅o΅hϊΦ1Wˆ2œwiΧ_؞tsSm|ξ9C»Ό‰ eN$Άiγ E¨ν«U;ήyG[dy―Yψε]}Q εŒ₯+KHK£μCΡ•ΙΑΆΧ^3”ω*(ϊδj«ΣUΛ?αέq8ήέ+Φ­Σ­ρɚŸ?\ Ο>›2ϊτΡ]ΟΎeΛtΚΟPŽ₯›‚FŠ΅ ‘EΗ α& ωΤT«œύ~(¬*ΩM~ν£R[Ή7ɏզu’?–ͺ¨ΚυΌΩρζ_4+FY˜ δ~Έak|ΠPζ«`5[55GYΈΟhΌ·Γυ;ΰλϋΗyg\zι₯τ3G ν¦›nr¦Sτ 0'ΌΦΰ­πΧ€@|ΨρφΫ>jO‹β₯RYχIlΫΦT elν_‰Ω{ν†'Ÿτ«³νœ¦D•NΏω %ρ5¨ρδ“Υ"ΪΔJΕfVJΪ‘ύ―ΊDκpΜ1Ί2³k φ^z.―Eχ:Ÿ|ξ9S7/«™Κϊ`Rf&Εk"…r,«sF½ΐ@±?΄ˆ`mΩ³(oΒύ ωΏ?J•3Y΅ιzρΕΊ?’ξ2ψ³*kΦ¨E^χ|=™Ήζ{m„“!!Πν’K γμωφ[[nώbΉ΄ρ™g ύDzA4ήΫαϊˆτοσX!ΗyG†π3Ÿl;vŒ•ΛΒu€@LΤk ήj1qΰ"@ ’ μώκ+*ώϊk[s\σχΏκœy&ΕΕo™½ΗοOΔa΅R6>•Μž­-ςΉίPQA[_}ΥP―Ϋε—Κ€ ϋUWQ\r²ξœ„Α΄“?LκξϊτS]νŽ8‚’²²teαxwΕ—„ΌΤŠδ–―D»’†~Μ>τP]‘K70‚F xγƒ6et  Φ τΉεCενo½E;ή}ΧPξ©`;[0I­Δ§¦RŸ?ύI[Τ²―&|•jϋ–Κ&;₯όp4χτΣ‰MΞ’(άrψαΘ,DΑΌ3Ξ ς₯K}N―|ΙšuόρQη­&ξ{;c²«RS\¬ŽΓρ;`˜ @@@@ •Δk ήj­μfΑε‚@ ,»ζͺή±Γη šκλiρWΠ~6”ΥŠ„QμΗ©C‚)m8Μ`ώ‰'†XpξΉd–’ΔPρ`Αrφξ―Q5kΨ0ʟ<Ω΄Ij~>uΉΰΓΉ…]diύ£Όœ]v™‘½Ω΅H₯pΌ»« °ŠΥ«iλ΄i†9{+ΨσΓT«δλ“΅#UB9–:6Ž'ŚσLΡ#€€ ξp1[Ζ8΅©CwδŽμVXΝ?λ,ϊEΌΠΌX‘ˆ%ΜkΑ9ηϋΉλΊςψγ”Ρ³§Μ}=b„{·εsη{οΉ<εΌΉΛKΞ­¬ό‰C JΌl3©Ω΅Λ¬e!&0θα‡ I|«·n₯ιlu΅Γ*Θ·*΅%%΄ζ‘‡h{QΦFιχξ{;­K+m™:U·άP Βρ;`6”€€€@k#ΰ―ΧΌΥZ۝‚λπ¨ά°~βwω²E‹σΎ7ΦΦ’?ό6Ώπ‚ΆΉΛ n8‡vτζm7ψ‘G(Ή}{]»:N2s$―s%³fя£GSΝφνΊΆ™RΟϋ?]™ϋ οξέ―Ό’$4₯Vd-Pς«›­νhλΙώž€9κΊ#{/Φ"Ήn(ΗRηٚh+Δe[jΑ ή*+ύ]c΅κ€€CJ¦O'Ωv‹aς3ςε—ιVvUmή¬f-ηΕ*ώςKΚ?ήε‰#ή8M55$aϊd+ζ$Άjι €“’φΌϊj]_ΪƒΗKΉόPVΚZ‘p’;?ψ€zέpe DI99$A•ΓΉθΓiΧηŸλΒM¦σƒYZa!•hβ;‹εΜΚ»ξ’φGMMό`δΙͺH;.φ'ΠnμXϊΤS$–mZi¬ͺ’…ηŸO‹Ω"K’όΚ=ΥpΰUmΩβ ΩT]­­N…lω΅ύυΧue‘|ξ{[b’«RόΩgτE§N”9`₯πΓ~Fοή4˜Ÿͺ„ϊw@Η    Π Έ½Φφςώ€r·šJΗ A!ΐ‹νξHAΥόΞώ#{εMœH9‚¬ΗHΞ±rVΆIΈΕ½¬<©Ϋ»Χ0 φ$ΘωΣC!YόΞ+JŸ―»N7œfKTy_Ο9’Δ+Κ΅ΎΕkFβ‰·‡C1Š–«σΪkIΦ7ΌI ‡έώόσ4Ÿ£τh₯ž½ΡΔp8kθPΚ=μ0Κ3†Ϊ΄‘ΏόB©§θ£ˆcv9?† Ο32΄]ιφCύξ.!(ϋά|3­eCθaƒό5<@Ε*T΄ιΧΟ΅Ι]3RW³²°lΑg=ί}g0Ζοwϋν”ΛJEUB9–:vk?Κχ©ΣŚΣDΡ€@ΔHfΦXVZΝ>α’‡%­μ[Ό˜d³*²xΘΏν΅ΊXϊΚ1«ΏcwϊFφzΣJωΒ…΄ˆ]ζ}‰<čη7±|Ρ*Φ€έ~h“-• ŸΐŠ9HxˆrU”›+ώόgCXGQΠ–Ξ™γΪ<ΝNrτυΊώz£bM‰›ξ©}8ΚΓ}oKrc±”«Ϋ³Gwως‚S2c†«,­ζΜ$ΤΏfs@€€€΄Fβ΅φ 5n³ψξo΅Φx—ΰšA τΔΛ¬3+‹Φ±²Κ-’@“ΝŠ€°ηœK e ΒiE$*/Iαu₯C؈z‡ˆ”α’ώœjDζ ‘+νˆ›OΰΠ–‡Θ;΅叧>%,ⱫVΉR€¨yΙΜΪH ”AμxμΪ΅Τ•ΥνH¨ίέ%Η$φNO>+Χ¦½ *kwγΨπ:½kWν)ΣύPŽe::B ™Ε‘žΠ €DFΞ¦i}@IDATu΅›­‹$ζs9»¬ΧQ]Y™K‘•Ζ*7YΌΊpΎ,'–$Z)?ΌμηXΣ²5pˆH#­vΔIΌα2LςGΉ‘Vr~8 G°O’lςƒŸΜ1:9xΰΉϋΖ§³κ9§ZεΊuTS\슽’ŸO²Ιw-Κ4­H¨¬€ΥΚ1œG/“γwG‹„λή–CΦ¬‘ ήκJJ(1+Λεε™ΝŠπ$ή·"‘ώ°2'ԁX%π<8zΛ΅&ήj35ω₯c• @ <ΎΰwςΞε–Œ^½θΈυλέ‡ΤΔa wΎΎkHς‘Ity‡Ο8Π΅ΙΪPΑ™gR’[-ƒa§œΧ‰v}ϊ)νζ°”ΥrWΦ"Δ³*™ ΅%L₯x‹Iώ5‰ˆδ+ £ιΥς{ψ?vε «έ΅‹jwο¦6b%S›ƒ›¬Y9•'ΤοξUΜq―Χ”rδ+ )ΧΦXUεΒ#kΙΗ΅ρ=”5x°KŸ”d_KέPŽΥ2(v!Ś#Ρ €€Ψ'°ξ±Η\9Ϊ΄-Oβ‡Σδά\mφA@@<xε•WθρΗw½„­¨ψΗ?z¨‰bp¨γ@;w¦~ή5“oΨπρΞ© `π₯X Ƙακ³Ή©ΙV˜ΗpΝ3šΖmͺ―'QπY5δ δΪB9V σlνm ²΅ίΈ~°οE­HΘO •°F`7[όσΟmηΝΦ‘€@XH΅3Ξ8Γtμ\6,ƒRΝ A@ΐ6;ΉΣlwήJˆGZ(”j‚7”c΅―Σ‘ΛNt€t  ­„€„0’ZιxΚ)”Ν!=νˆDbήΝρ·΅’Β ι3N[„}ˆGqNβ^xΑp= 0”‘@@@"•k‘ϊΝ`^  I ‰CΨ¬Ίϋnέά$φϊN`lGv~π.Ύ»΄-π`Αk§_ԁH%δ!MΞΛ§ 4qhΤ#FΈΊ§Ε‹;2kŽ`D'  ­…@ξᇓ$«mΨΏΏε’wΎχ Έο>JγœV€¬Œ~Ύξ:]Υψ”Κ?ι$]@@@@`γƍP— ±@`ήΌy¦—!‘\?όπCΣs(h#Γαύ'L˜mΣΖ|A@ & HΔ¨eΛ–ΉMkN kN‘D?  ­‚@|b"΅η6EšϊςrZpΞ94rΪ4Κθήέ+Ι«6οτΣ©FΙΣύΚ+))3Σk[œh}>όsΊN1Θi}pΕ±N@n§vZ¬_&―•3f ΝεΨ%ΕZμ~·Έ2 θΕ‹[»>ϋŒšλλ[F(™9“Ύεά½nΈΑ₯xΛθΡƒ»u£ΖΪZͺ)*’ύ+VΠφ7ί€]Ÿ|BM\¦•ΜAƒhπ#h‹°          X‹ΐ/Sˆlβ±6ς•Whα…q¬f·4ΥΤΠΊ‡vmξ2_ŸVr+ά|UΕyVH@ΒΧ@@@@@@"‡€sA%#ηš0:.ηGÞy†(.Ξο±rF¦£–,‘ΆC†ψέ‚€€Δ6Έž5b› @@@@ΒCkααŽQA@b€@Οk‘φΗCžzŠΆΎό25VUYΊͺtΩνςΛ©ο-·P|R’₯6¨   `$pρΕΣ±Ηλ:Ρ‘Cc”€€€€€€€Γ Xs(Ίh]2ϋφ₯αό' όΫί\9Τ¬YCΥ[ΆPoΥΫ·Sbf&₯vμH©:QF―^ΤωŒ3(—“YC@@@' Κ΄XW¨y Ωƒ u†8Dτ  ΰRš1c†_mΡ(τ$ƒ6ί91„ώ;ΐˆ K X‹₯oΧ  6Ι99$lp’€§P'žx"M™2ΕΙ‘Π€€€ sηΞ₯±cΗΪhͺα$7n\8‡ΗΨ 1FŠ΅ϋBq9            ΠΪ $$$Π’%KΗŚγHΡ!€€€€€€€€€€€€@Έ >άρ)Δ;ή#:$ΕZ ~©Έ$η @±ζ]w©u%%Ίγh?h Χνίζ      N¨α¨Eκ:Gν•W:Ρ5ϊˆPP¬EθƒiD&’Ω³iρe—QŚ5ŽMpΟ7ίΠwC‡Rοo€ώ¬°KΜΘp¬ο`vT»w/Υ——λ†H+,€„ΤT]@@@@@@@@@@ V8T2Vˆΰ:@@ΐΟ>KΣǏ·¦T‹‹£δΌ£%W^ιι΄«όηλ―םΕέψοΏΧ•­μ1ZΓJD­ŒωπCκόΫίj‹Zφ·p¨L΅ώQ?Lm‡ qΥΩτοΣΪΏέ₯4liΔ;•¬`E›ˆ΄οΓa5ϋώε/Ÿœμ*“kjθ—Ϋo§MΟ>KM΅΅-εξQσu―{δΧƒλ¨Χ_w)˜άη=}ΪΉΖ]ŸNKΈΒSW~—χ»λ.xί}–ΪΧΣJζ°υ΅Χ¨Y£u7–"[Ω³rωŸώDO<‘ϊrύ\VpZ•]¬°[Β N­ΗJSχ=WΜαU—\uU³RT+΅{φh±      1E@ ’7N™B²ώQΟ†ΏͺΈΧp΄ε©l έύ²Λ¨―'€³1¬'©+)‘™Ge8}Τ₯ΤvΨ0CΉ―‚ͺ-[hϊΈqϊj¬¨›¬Dλ+όz­k’δ”΅#1θ¦ζfΓ₯Υ°ςSΆύΛ–ΡŽ·ί&b…§xΰ5ΥψέΠQKŠ΅¨ύκ0q`ΨΖ Uς&NtYΰ¨εN'¦§“xΔ‰RF+b#^fil™dUŠ>ώ˜^t5μίo©IΡPρ—_(Ξ)ΧωΤS-΅ w₯FV„-ζ‡H³~ΘΝBbΥ%žmsωZ°'Ÿ)e‹¬οΩck['‰ηV¬Θfφf\ΞJG«χJs}=qˆ’O>‘ή7ήHδή DΦ<τ­ΌνΆ@Ί@[(%ΠΘ`G™,ψΉ/g2{ίΞ‹YNΛ;­` /bz’·ήz‹:qτOrΦYgΡn6VΣΚeβ8ΆT¦|ΞFY?όpΐCδr4‡ώ΅B»΅εˆ ‘”p1 ε5b,πE`;½Ypα…ƒh_νΔΘz5+nΦύγt―S΄cΓk3O²,V ‰G+b„νbΝeΌ­νˆχe½Μ›rOͺGλΪΓj^σXύ·Ώ™!+ώwΘΟ6›_xΑ₯d;œˆν%―μE:(Φ"ύΒό@BN jΫ6W)Μ$.Ή*«ξΉ‡Ά±gžͺT“ΉŠWΦΐTΟnϋW¬pεSέό+Φ¬‘ Ό€ΧοΦ[Υύ;Ž‹#‰oξ·°‚ΛL|)ΊVsxL³DΉςƏwyε‰g^{£‰Η€πΕ€κΩΆŸY‹ηίαμρ˜’b6eε ²k―υx'@@@ΐQ2,δΏ'"ΩZ{φpXθο9΄ψ±ςά)O³όΡ©ξb’ŸΞmϋΤSO‘xκ‰Gάο~χ;Š“ηT€€@Pΰ΅y—VΣRΔsώωψ΅:”2Ω³XςK/Ɂ.k[8EΓ>㨉# œγy½Ημ½\ςΖ―`#¬HzYχ6«"ωμe-H+2ίΞgœ‘-νGλΪƒ|?‹M‘΄;ςHκΑi,2ϊτqεQ«ηHQ•6Έ6ΙΏ&ιD΄"ߍ¬³Mς°Φ¨­‹ύθ"ΕZt}_˜-€@ˆw˜*’\°σ°‘Ά·zœΐ "±0RTD™cEͺ41­εαf‡ω+dE[œ‰β¦?{*mbKݟ9œŸV'‘ώΔ+λ(ΆfzeερΓƒlnYΖωζTΕZ?Σӎ“θ[΄J5 90bκTΚθΡΓ0¬<8J~4­ˆ5Χ>φ8sKZΧ4œs¬u<ι$wQΛη~~hΟόπΓ¦VvrΘ§k]6DB~ΞεόxΪ‡cι§+Ζz]wΗ.KfΟ&Q0ͺ†Ϊ%¨©•?Ψη?]γŸyFΧt//L­βœnƒΦΊJ&‚R;χ,~qΘgOBω”—Ι‡'I€!   ­›ΐ9”·“Š΅χΨXHBPBŒŠ9χξ%lL7cΖ z‘ί%  Α!°κΛΚt‹±―Ό“‹Α―VR94qΦ AΤα裩{ηοΰΏcσ%ͺ’ζoYέή½TΞkf!!Σ ¨ύ€I΄η»ο΄έ’„u΄³Φ%ΖΆͺHδ₯€¬,΅ΨuΝkK―Ή†+*tΧ5όωη©+Ι΄"l³•χζu²œje―ΗHΞ5·”σ:£°.ΰ(QΨ!ΕZμ|—Έ‡HN3UδA#žΓ£„BYΡ£*ΦΜ’Χz›K<{φᇔο%ΗC|Rυbo!I€:Ÿ­—΄dΒ`+[Aυδp0‘,8Ξ΅(σΰEͺη8|t?Ώ:!oΏύΆέDL©lœ7š£7XρL+))iΩ$§™ΌτKτ[EΚ)§˜F€€@€T£ξdŽώ3aζLλΪα ΨClΘcΡς?ώQ[L₯sζ˜*Φ€R^0S¬ ψλ_u}x;0 ΩEΦ%L$šΧ$ύˆ°ΤJ'φ.T•jΪσξύΞlό,^†?pD%m€lРŚ›Rh?›ΨSs ―‰Δ³γ„’vBβθ}€€@,¨S,†δΪBι5“Δ‰ΔU±«XλΛa ½)Υ΄ύpβυξ—_-rνo`―$O/Ϊ†Κa((`O²ώœ ή“RΝ=₯.’44ΞΣ!ΦFž”jξ&b)&–]ͺXΉR- Ωq-[’Νa«0Υz*„‡±[Jϋφη²υΥW a0ΣΩΫΟ—RMΫ‘XΘuO9HŽΎυO<‘)±Ύ;š-·D ₯šuf¨   ­‰@)‡ΐϊ†Γ+9!EEEAΙQζΔάόν£sηΞτΣO?YΪf͚E«9ό—„ΨάΞ!Γ^xακΥ«—ιΠWq¨+ap–@S]I ­Θ{Ά―υ m}3%MΩάΉΪ*ΊύΞ§ŸNΩH+ZςM°"b\\ͺτŸœ—GωΗoΪ<šΧ$:“¬qh%oΒν‘Χ}I«‘FER#>yν'% k›:ή½9Υ9kN‘D? 1CΐL‰•”²λKΞΙ1Œe6'C₯ƒIάΎ·b΅δ©»\,”$ά₯V$ό€jA₯=ξ}«VUΩb%d"³<{ψp“3Ζ"Ι/§J-/F„Cš8Tη<ΆNSsΔ±†‘}ν5j;dˆΗiΙΓΔϊ'Ÿ4œΞΰή<Υ Έ wŠw}ϊ©‘ΜW(v ψ   n9ό<›ΐFPZ‘pNˆ„Λe­δρΒ`kQΘ]Ιωcf²‡Δ“gHΙaψ'ŽHp–€(՚YΉ¦;Ši—VXh0χΆN!α%l£*f^hj9v…TΌœ Ξ>›$’*ΡΎφPΖQŠT©fc;’ΟιFĈٽI †ͺ*;] n„€b-ΒΏ L@ τΜ”X‘T¬‰ε’ALr€κ,χώ€Άm=6-—xέω'œ`8WΚΉΈ"Qr;Œ2ϋυ³4΅d@iΨρΔ-΅—JβΖ―J]˜kΛ8ηœs X;x―e£'3ω…sπ@@@ΒO@ώ†UmέJ9_ό6°΅+bx\ΐχTρε΅f²Λ¨έ΄GϋΪCJ‡”i²^±oιRϊŠ΄]r q~ϋ†ΚΚ–kΖNλ#Ψϊ.W  ή $™„4 ι½ΟrgqWVk)1₯}{Ώ—ψάͺΤ…P‘¨ŽννΨSxGom΄ηRςσ)!%E[ΡϋXΑ9Ÿ­©‰™΄’ZP@‡}ψ!Y΅lS“#K_’¨xσΤ©Ϊn-οW_o¨[΅e‹‘ΜSAζΐžN‘@@,Έθ’‹hβΔ‰ΊΞκ9§l.?“{μ±τω矷\Τ'Ÿ|BΥΥΥ”ΖyJό5 δ ^4“ kΏ”°˜βΉV¦Ό¬X±‚ŽτΣxϟο)m*yqT·’DάΔ†Ž›9Šˆx8Κ}XΐΟά’N>8β:Cθ«9Νζ\TTD«•0μρlΰ7aΒ³κ–Κf͚EuJκ€ΑlμΦήβ;ΰθ‹/Ύ ―ΏώšΆmΫF’GOώOuι…Ίqψϋξ]ΌEύΙ;hv½’¬MΥxαΘψSωέcρβΕ΄dΙj`o›'žx‚&›DO±•@  4TTΌ»K4#χVypΏš•jM†EξΚα 7ύσŸ:rζΡ[Σ0^k±°φ0lΚšΓy•Όh’oλ+―Έ6‰n%!<%Ε†xΉ‰‘{bF†Ž-b—k±ϋέβΚ@ό$`ζ±*Wo‰m¦ΔΛ°θ]e6w«D9“ΐ‹›PzκY§Τ“€»ˆYΈOoύ©–Νήκ:}NbxΟεΓκ}Ο J’T“όxV₯΄ΤPuυ_j( € _–­Jbf¦Υͺ¨  ¦D™…š)š¨.Śˆx“ikΌΨ(Ηgœq†νλ%œ9stν$ €kΏb‘( ½ΨαΒ…ΏόW&±*’@›Β‹§/qˆr“ά9»wο6(Η Ω ρ²Λ.£+ΈΒ₯ςΔ¦€€„Ž:κ(Γι₯μρ0lΨ0CΉ―‚-lΐ6nά8]5QΤΙ5ψ ιyϋν·Σk―½fPΜIΫuΓΐύη?‘?q”Œ9΅ΤΝΡ1¬Š(μ.Ώόr]uQRv?ψ.ϋΥW_ΡUW]E[Y1 •=aΚ]­φA άJXiΎ‹Ύνύα*[°ΐvˆG;σΟεœυ’£rΓ†–fϋXΡ]Ι‘38Ό£*’G]槕lV&yΛ{ kνΩpkμgŸ™*Χά,šωyEBdΚΆξα‡Im’n£γ)§PgφΌoΣ§»*>c@| ^. @"Π†-5UΩΏ|95²%_°ΕS½~π±"V=Ϋ<υ₯z‚j εiœ@ΛγSRν"*Ϊ7³‡Ϊ^PͺX³Ζ0ί‘lιi'Ή±άΏM!Έ‡#υž1D€€€@Δp+ΦN=υTJQžϋό i­X“Π“_ ¬7‰HΠ'Fίzλ-κέ»7ύγ0UͺyΊ'ΆoίNχέw 0ΐ ¬ΥΆO23šά‹ώˆY;ρΨo3oςβ‹/RίΎ}]žbͺ·›Y;ωχΡG‘x›έtΣMT₯xk˜΅ρUφΠCΡρΗoPͺωj‡σ λDi5דּh:+ΝΧ>ψ KAc5oZ¦Ij«ΌΊ°Χš*žΒAš†4iοξ/–ΦDΉ6ώΗ)ΟΔHΒ}½ΪOQ΄•p.Χ_ώςϊ†wb/gO\΅ν° X‹Ξο ³"\“άς`SΞα*‚-’|ΦLĒȊβ±&ύ«VE*κ¬Μ9uΒιqΘυ-ηΩέ¦E•~wήI…’Ŏ„*φ7kvΎΤ0#ΰV¬eq” »§•ΟΨz\ΒφΩU!7”sφφοίίՍ{<»}ΖZ} }hζ΅%α2cM¦OŸN—pŽœ@”ͺrΚύ)! =Ι…&‹Οf 2Oν΅εοp¨6UΜϊΧΦΉώϋιΚ+―€ύϋχk‹-ν‹'ηγ?ξR°νγΘ*ώΚsΟ=G·έv›ΏΝΡb–€(\ΎεΏC;-*ΫΕΐ8{Τ(κqυΥ4žÎap₯Π$?›'! $η°χΆkk9Μ|όwίΡφJλyν΅”Ζas­JιμΩ4Ÿ§ ψoA΄KY½ΦH'α›ηΝ›ηΪζΝλΨT 1”θ@ VˆΫ6Ν7δ³·όvlm,i旍ϝCg€rN0+@¬mΙ§¦ )γ©lVζ€:ΞΨΜ–₯žzΚΠY''0€-d튧π™Γž}–μ†Ζτ6vzΧήNγ€€€ψ$ y—ά"α ?δπΧnοΙ΅&y ¬Š„Κ“­HΏnюη.kŸf‹MR<³bIΦp4ρ†¬Uޝ$wίώπr+]%ηX)‡R—šΏώz—βI”²Ϊ{tοή½΄˜£›HθDU x!u€Iτ{;hEξI;Š5Qx©ςΞΓ,^f2›½$ξΉηΓ)ρ”λ<δC ηD©ψΟώΣ₯|F1ψό‘C‘έuΧ]τΘ#Ϊy+P’Ϊ_’΄œόΦ-;ί_ώHΊ]Ÿ]MώΨPξT™wkP¨?TY•Z‡k)=ε¬Ξυ|¨‹κ3Ξ ‰Ν­•δh,ί‰Ϊb[ϋ)νΫκW¬]k(C€€€„“€6\R?ϋœtIΊιˆηŒΠvjHρ‘άZnΡ.ό»ΛZΣg{$ˆη“xf©rο½χͺEQΌ€½2΄Ÿ³g²ΗͺTΣΦqο §Η{Μ}Ψς9‡C„y³pfΚ^Oν₯ά, δ\`Ϊ€šσ*˘ZεŸTοΉEΌψk¦Tsw$^pO?ύ4‰ςQ•)S¦˜ή#j=ν±xΪ‰ˆaΰ-·άβ_”s2?·g`{“wm؁X# žNͺB+Ž=ΡFΏω&υ»ύvΛJ5QΠ"…gŸM2VΤpjH1ΦνΔ¬Ύ€5­=ˆρ}ξθΡԟ&̚EΗ³Χ_Ώ;ξψ5–jχ7ί(%8ŒfP¬Eσ·‡Ήƒ@WŽ·Ÿ`b™ωΛ­·RS]γγξηP {+F$­ΫΨH.Κ?qΥχGΜ,fωζόιm¬hΰ0sΩςTυ<”P‡±₯X a³GŽ4LΔμ{7TB€€€„€V±&ΓjΓ6ΚqMM }τΡG²λS6nάHŽN+Zo5mykΪ—‡βΥt ΏχtηΠσfa 'NœHgڈž όκψ]NBjE<ΡΪpΤ«bΖȟ»ΏΣO?Da₯ -)!!­Θ6Ž,’φŸΗή,ΗΌiσW_}•6mΪ€;ΧƒCΘ½Θ‘ζ­z “Σ― QΨ=ρΔΊ~­ˆ"ρa–¨,β[mz K$Œ *έ―Έ‚ 8—©ζ΄H tΰPΆZ…_ΥΑhF5ll!Ρt΄"ήVVŒ}[σΪƒ„|ψ·ΏΡ(φV₯LyQΟγ8Ί@±]ίf  "Ι99TΘ}ͺˆb­ΝπjfΗ«9©΄™ˆ[Ή-in¦]_|a«‰Tvεwγ€ͺδρΛ4$4dIΒAμ7I@|ΘσΟ;’ί―ηRP₯DyPVΟ{;^yητU―^ΊM”Δp’ΐ‰'žhP|¨^hžΖ3στ‰5Śδχ>|Έ₯MB;ŠIΒοΑω£_yεe“* loζ΅€Φ‹ΆcQͺ©Χ;aΒ[—QXXh_ΈΗKδ Χ(aU1»7Υ:r,a Ue³άΓIII†κRοΙ'Ÿ4”?ΗοzV•jξΖwς³Ύ*Ÿ~ϊ©Zδσψ,Vˆr π+³w~³wu_Ό¬\髊ΟσR'ό²σ`θYΣ0ΏϋΊ§³λ‰–΅‡Šuλtk²ζ‘zξyΊnmΉxf(†ςϋ–-£@= ΅c`?Ότώžα Fˆ"Π—“Loγ£Ml«Q‚esLϊŽJHm;ϋλΩκoΗΫošHn΅Ž'œ`(χU }uΏμ2_Υtηε!‘’΄’ΨΆ­λ:΅e؁•* ΘΔςΊχŸώDέ8©±"ωΤT‘π¦ϋω-+ω½AςρuξάΩ]δυ355•εw€ΐˆβF•¬!CΤ"ŸΗ›Ω 5PιΔ[ε=^›οMΒAφΎρFWΎ5m©μ‰ΥžσMZ‘h^{Hηg‰ΥΔ^ΊnΩΛα‚νzJΫL^Ρ΅%efR<πύέΏ]Π²Zύm₯οοbλξε#,7οή—ηδ«@+-³κo6―™U’9L=°›@‚š]ΓY|睲DφŽ»Aϋ‹φ­_΄kuVΆ±±ήΨνωoΎn=b„΄Π›ΣiuωΪC―ο|G–λ΅9»ξkvC΄•uμwσΝI―ϋΩφ›τ:ΫtܚzΗΕΠ²/P©Aβ1cΖ8.ΠχΑΚ&GΡ¬E‘H ΧΝt’ S§ΚΤ㏗ ĝλ~ύr³τWΏr υΪf‡.υΤFϊhXR"e:‘΅ΥgΆ@Ϊ^ύƒ,Y¨MχΎμ²d›$Νώώ"΁οκΘν&NtRϋ‹5PV€%&·iό-ϊ? ¨•λ…³ΙUΈν6κΈe«;νoώυ/yEΏYΆ•”rˆ©σ3Π‚j©Qχ—Χκ­τϊs4“°Άν ύ,Ι†ι©wτΛό½hΰn‹υ‹ό†W_•v Ά»θμas Ϊ»φΨπΚ+qϋΨώ]uRή>z‡3 @Θ€€eόΈkVŽΟ2z. ψώlΑ!(c(“γΝVί{όρΗ³uΈΌ:ΞTύ;οε—_–w4³`Φ¬Yi—xLΓ»}υoΛΟτ&ΘX³R₯Λ–-+οθo6—™Οέ,zΈώ½™¨m ¨¦rΛ-·$ΪΌFλwξά™ς~τ₯!€€W ƒ^[j£7Boυύ~Ϋt 6]H_½™΅…f{ι’vi·ώ±ξωηe½ώ[垎Αρ4νΦMΆθό_±f™UσuξF› Ν*-uΤlο°f7”ΫΝέK-`— υ¨αΌοuυΪƒ•²<τΪk@f5‰^_Ytϋν²α΅Χ€ηE9ΧώμϊŸ½UZ jοκΥRͺ±Ήο6½υ–ˆfώΉΫαz#s P²/`Yη±2ΖX‹ͺ5Œͺ#ϊAςY ©N:^ΛjΜψΚWdλτι Oυ€ώ‘±νƒΎžθ…"-½rδ―-½4σ­6Ν²ΜΊhΪΧΌ\@ ΚJ :Žέ₯4VkG[P,¬u8ξ8'xXCw³`έ–/J΄ ¨ΙοήΆΎ?·/Ί£5/θ£υKω4Ν²τgbnŸ=[μ‘j³ κGMusΆC@j,pœ~lΫΆ­lΩ²₯ΊΛJ ¬ω3},ΰC˜†€ XΠκ*½xVŠ1HΛΚ#Ν;7θ₯Πu–΅φσŸά³αΊ€rνAe mDmοή½bL·Ϊfξez|τ@ ΨΝ―Gk£·– Ν”u7»žτ‘ΞeΦ,¨6^―υXf”;°fϋ-₯νΡD>''¨”δοΏ»ώΫ’(°V¨σ°u―A \;F]ΎφΠ鍊ĦΑΨ£Mw³χΘ}έΟ¦Εpνψiξ}Zλ\—–ιFΛ/θBtωεΒΩ €qυψ/ι@Cτξ“Ɲ:Ε½^ΣέτΞ γτ’ΪΥbΗ ₯z_yel1εŸ–4ξ7BK@Ζ:,΄9΅τZέ°2gΜœC0Υ3²/ίtςς’-Rέ…ν@@ 4lΨPΞΦ›ΘάmςδΙ²> τ•Ώ δΈq〫ή,GCΐYύϊυK9¨ΦXη–Άr™6}ή>Υςώ5mAσ³% ξωƒΓVͺσόσΟOxθέ:w6΅l(sŒ|°i†κ΅% Z₯ΫΊι<‹už΅βž=₯Σ©§Jƒ’ΔΣ‘€Ϊw«!C€y‚­]uڈΪόΝ_W―=6m*#υMBΎ;X•všΫέ²Gh€Ž)Sά*ωρœŒ΅όx9 Θ’€έYΤKλ*wΣ24‹5+Μ™+-σ±aj‰ΘŽ'$½υ³ϊ?Ω(›έ{”Ξύfu»θ1ϋ4=Y³ ‘•~΄yΥ‚Κ&Ϋχ0½³²ν>ωα₯2 wG& ―₯/ΠDίϋ‰ši₯ ζλέXε7¦ΤIKύlYP΅Άs¦t06B@\VΞρ‘G©^S‘₯™,0ρ½ο}―zέΌyσΔξvž–¦!`s†]¬χΨ|jώfσψY©Ε‘ϊ]Χ²zkιϋΨΓ‚ΊQ΄Γ;LFjζB¬$•υi%(mή²žz‘<ΦΦιTώ2ΗkωΈŽ;Ζ6‰ϋΩ"ΑΝn<π€ΨΉEΥςqžΒ¨lθtμΊ’Ν]φΡWΘF’!iΣλκ…œ9Σ+j\·gηkΉBλΑ4uυΪƒ'i†Ϊ§?ϊ‘¬Ρ91έsωόΛ6ΝΚ@6¦3σͺωiςf9šoyΓΑ‰ €© 4lΦLŽΠω±μ±CΛ€¬ΧΉΕ6jΆΧ›ΤZƒΆo―ξΘΒλ@φE’ΉήΩY']·/BvχK&›eΐυψζ7eν?ώ!λτ‹–ΝοΆG`³ϊΩv'’=¬nwW-‰cηSΣΦϋK²;-’]ϊ(Χ< υ:»{*hR^ β₯2‡[l<ιnΫΟύσ΄4ζpο{~ˆ–‰±Gͺ-Υ1[½q{ΜfŸ›ΧΟ>+φΆΟΚ6-iσ–—–:eA­ͺΥ o¦Ÿ_+aŸ›t›}£ΚΚLχΨl €ω#0aΒ'Έ°aΓ†κ“zVοwΦό™>6Ÿe «Ήκν›“ΟJ)ϊƒj– v«ώ]w©ώ]Σ]»ŒϋΨ܁5; ‡―ΉζšκcC†«Τy—έ-YHΫΞ‚g%z§?sν˜cŽI:/›ϋΡ œύ ίN™N릱ώΐš]h§½(Z]½φ`Χς†?ύ΄ Έλ.Y£ί7Άjε+iΧώ*φμqhμ˜e!:K³…ήœaΥ© "Θ&ŒΒž>2#@`-3τŠυHΐώ‡iΓ~όγκ³ΠΉ³l1ΛLk€]«Yͺy7½;Χ™l˜k=l˜σΘδqθ;sθ΅;©Έ›*sΖτŒ ½€]h?~ΌΣq—.]’?=ζœ€A,HvύχWmŠ–k_³fMu©GΘ/}ιKI/ŒΡκ·ΐλ:—΄? e™hOλΣs΅ΤYͺΝt΅i–=yυΥW‹»`ΝΆ€ΩzƒfXk―seϋk‹/&°ΗλdN§œ"φ¨i+Ρ9ΧJ.Ί¨¦»;ϋν֌^³›p£nuυΪƒ9uέa.•ϋχK…VoͺM©Μ¨}ι/{Μ±–=kŽ„υH PλπΫB3¨VΈ9U@¨§;w–Q£F9Κ“ΥŸž©ͺͺ*‰"lώ«…:±»YωHŸ|ςIΒ%Zβ, šu°:€Τ~άA|+:tθ 'œp‚g­όV­Z嬳9ύο{^?σΜ3l4ΟΚ€…az³£ΏY`†„ ,{πAΟ&v£xZλ<ζα‚e€TΛΓ76ΕS"°–"›!€ € € €ΐΑ;vluvZl4±,5+ ιn–αvφΩg»WρΌž XΠΥίb―ώυΙ–ηϟŸμε”^σ—u΄ΰπίώwgί 2ίΤ²ν©΄ σριRι'ΆΝΟ~φ3ι«eΝ܏ΉZrŽ†ω%°WoXχβ‹ž“κ~α…ά,ξa―5―K € € € Γ tώ+§ηn–ρ³RηφΦ&Mš$V†ΐ’%Kβ·.lΕc=ΆIθ맟~Ί4σΝsmε ­Ε~Ζ:ιͺ•PŽ=φΨΨbŸ6ŸšΏ½¨Λk \Ύ|ΉάsΟ=²lΩ²κΗFS¨ŸΞΉDCόXώΠC"ž“κsε•že¨«v“ΥτιӝΗΤ©S#; k‘Q € € €dCΐ_ήΡ2~nΈαρ—½σo—±qŒάθέ»wάΐ-Z·.ي·ί~[,PUΫV\\,gu–§›iӦɜ9sδ½χήσ¬Ζ7Ύ!©]Ύ³@α„ <ϋΫοΖwάαY—ΚΒOϊS)//χlϊ­o}Kl^:δΐ={dωΓ{N¨ƒ–«m9hg ΤeXιxϋUKνΜQ~@@@@Z Œ9Rzυκειε駟φ,[ΐζ¦’!`GuTΔ_ώς—Έu‰V̜9Σ †Uψ²:mΆ>¨δΕ_,•••ž]S-ΫιΊλ‹=­ώiηιΟ„«~1ΰ‰e~ϊmš4i"?ϊяΆfԁŠ²2ΟPmyζ9ηHωζ͞υ‡ύδ'že@ ^€ΫLβMXƒ=emw@IDAT € € €9.`ε οΎϋξ„£<ξΈγ€mΫΆ _ϊ+ψξ»οFΦνγ?Y_u₯£L:4ŽΑζ5³lλ―Ώ^¬ΔhPΫΊu«άyης›ίόF+₯yK₯ΕΆ_Ώ~}μiΚ?­lcηΝeέΊuΥϋ|όρΗΥΟνɈ#€ώžua §œrŠœvΪižΜ:χΉηžλduώβΏH˜·kΧ.Ήκͺ«δ‰'žˆ;ŒŸ>}βΦ³κŽΐ›ϊοIc-\rΘ!²oΝ)5K*4cΝέڌ#ν΅Œ2 ’ XKξΓ« € € € ƒηŸ~ΐšΆLŸΒδΙ“ΕQ4 ςΤΗΐZ& ?ώx;v¬ψηW±’‡Ο=χœP0`€΄nέZV­ZεΜ+φόσΟΛΛ/Ώμ)‰h™’έΊu“)S¦TΏΥ .”oΌΡ™ ­L3@N<ρΔκΧ=±9_.Έΰ'`—hΛ`«Iϋγ(H\±b…gw "Ύϊκ«2~όx'ƒΟ²ψφνΫ'Π³Η+―Ό·u`ΏK—_~Ή§/@ ξ Tξί/₯š}kΐ¦οtａ/±ΌΦΌ,!€ € € €@2dˆzθ‘²dΙ’ΈΡ6jԈ2q*υ{…ΝSφδ“OΚΰΑƒΕ2³άνƒ>›Λ,¬YPΝ²να¬Ω~·έv›σ° ›ζRiV2Α‚šΝΓφ΅―}-θ₯Πu΄€ΰΙ'Ÿ,+Wτl?{φl±GͺΝ2ζ}τΡT7g;¨£ ŠŠdΠo+­‡―£gΐ°Θs¬eΧ›£!€ € €D$`sjΡ’EφΈώϋ#κ•nκ’ΐWΏϊΥΐαžpΒ ͺU«ΐΧXY¬”‘•›΄ UΊνΒ /›g­gϞrꩧJ‘^„m³ΰπG؍•nlΡ’Eΰk©¬΄€ΨŒ3dx-.’[ΰούχί―Υ8R+Ϋ €ΐΑh₯‹Œηιsε•w :$@ΖZz³* ΰώΜ3Rιš|Ά°iS&,#€ €y+°Fη™>}Ίs~VβV,°f™Bώ–ν2ώ㳜»φ™±ΉΛΈβ §,b²‘ZIΞqγΖ‰•‹t—w΄Ήϋ^xαgή²έ»w'λ"τ5 ^έpΓ qΫ]rΙ%qλ]Ρ©S'ηίH &ήtΣM²qγΖ”Ί°2’Χ^{­Sͺ2₯Ψκ„ΐ0?q«άΛKK₯Ho>i‘εoνas4$LP'ήD™3 ͺ΄εΜh € € €) άsΟ=rέuΧ9[ΫE`[Ξ·φϋί^Ύύοǝ֕zWΉ½FCš XΉΔiΣ¦ΙΌyσœ‡•ˆ΄RŽέ»wwζ ³ Wοή½ΐζ0³²}τ‘XkΧ2aΒ„΄²Δ–/_.–Iηnύϊυ“ ΈWΥϊωή½{ε7ήp‚V rέΊuRͺΧΫ·oοŒΫΞێkε'm9ZΈ€eŽ=:nΓ‘#G:Ω‚q/°@ oEηΝ[Ι‰ € € € €€*pΚ)§ˆ=jΪlΞ΅‹.Ί¨¦»;ϋ-]Ί4nΛ.»,n]mW4Υκ&§všσ¨m_μ €@]¨¬¬t²ΟmΌ6ίͺŽΤšž΅šΚ± € € € €΅xπΑ={—””Τ:Xηι@κ±€lŒ•Ž·ΐZT-Ίž’ύ € € € € η«W―–_|Ρs–^x‘΄Ήh € »Φrχ½ad € € € €y*πΠCIEE…ημlώD €䢁΅ά~ € € € €@ž μΩ³G~ψaΟYpΒ 2hΠ Ο:@@ χ¬εή{ˆ@@@@ςD ¬¬Μs&Ά|Ξ9ηΘζΝ›=λς“Ÿx–Y@@άh˜›ΓbT € € €$°ΉˆΖŽλlΤ΅kΧδσ* pϊχο/νΫ·—C9DΦ¬Y#³fΝΛXs·1cΖΘ€I“ά«xŽ €9*@`-Gί†… € € \ K—.b Λϋχο—™3g: q6hΠ@ξ½χή —X‡ €9(@)Θ|S € € € €@ώ Ι}χέ'Çϓε @Θ2Φςδδ4@@@@κŽ€eά>ϋμ³Υ%mλΞΘ) €uC °°P¦L™β Φ2Δ£jΦ’’€@@@@|O<ρ„Μ˜1CJKK₯U«V2`ΐηas5lΘ₯9‹ €D*›“9ΚNωΏw”šτ…@Ž μ[·NfžώAΥ·έ&νƏ?hǏκΐ+{L>κ)OwGά~»΄7Ξ³ξ`/Τ•ql'Ž€ ,ΎηωμwΏσ`Œzξ9i}τўu, € € P[γ?^μAC@ό °–ο#g@ @ΕΎ}²eςδΐΧ²±²|σζl&γΗΨ³bEœcω–-?nΊ¨+γLχΌΨLΨΎ]φ­^νιΊ²¬Μ³Μ € € € €~ –@@@@@@@ ^€ΐZΌ k@@@@@@ˆ d +ȁ‚’")ιΫ7­ͺͺ¬”=Λ—ΗνΣ€K)lΪ4n}²…%%Ι^ζ5@@j%πΜ3ΟΘ<ΰτqΑΘW\Q«ώΨ@@ °&ΔλΤa¦έΊΙ K—¦uε₯₯ς―6mβφ9ϊ©§€ύ1ΗΔ­―+Z*=Ύυ-Ο©χθαYΞ……Ί2Ξ\°b  € «uΎΜ)S¦8'3jΤ¨ό8)Ξ@@rZ€ΐZNΏ= rA λYg‰=r½Υ•qζΊ#γC@@@@ XK$Γz@@@@@@:)P©ΣM˜0Α{AAΌχή{‘œ΅Hι@@@θͺͺͺ;΅2˜o½υVΰk¬DΘΌΐ‚ 2Ž€ P+ϋ.+o΅¨΅¨$ι@z(P±w―4hΨP ŠŠ²vφφ₯Ȏ۰Έ8kΗδ@ €, ϊ…^{Π@@@ »Φ²λΝΡ¨—ϋΦ­“ zΞ½ΝθΡRΨ€IυΊύ;wΚΚ'žm³gΛφ9s€κΐψΫίJΗO¬ή&Ρ“»wΛκΏώUvΞ›'»—/—=+VΘή5k€Q›6΄kWi₯‹σ³ΝΨ±ιδ“₯Aaa’Χ—oΫ&ΆoχΌΦΈC)lΪΤ³.ΆClYτbHϋ‰«ν‰ΆL*kώwΩ΅x±3ή =Ζ;J“NΔ|Ί}υ«΄[7Ο~ΙκΚ8CeyΉlxυUΩΌιό•―HA£F‰v?hλν³»α•Wdγλ―ΛήU«dίϊυNΠ§iχξRά³§΄:TΊΎ4nΧ.ν1ξΥ;Ρ«**ͺχ+Πί™&ϊq·έϊy_ωΨc²cώ|Ω£Ÿ½kΧVφΫMš$]Ο>[šχληή₯ΖΟν3Άαε—eέ‹/ΚΆ?”² δ€žΏ΅’Φ­ΟkϋcŽ‘Žϊ{ΦαΈγώ]K40sXϋάs²φωηeΧ’EΞqν߈FmΫJ±~>Jϊφ•ζ‡&=λΏ€ΈGDέ€½ήώmωόΙ'ίΥ½+WʞΟ?wήϋ΅WΪ{¬t;οRiKυ+YtϋνžMGκώ.§ŸξY[°ρxΆΧΐΪ™ZΟ7ΦΦόγςιΥWǝ―½nΦ­Yΐmξ·tΥΰڐ‡’’–-υΙώSWΖι?‡Š}ϋdρ/)Ÿύξwοcl{ dτύΑ€χε—KQ‹Ξκη|A‘`κyρΕ±]2ϊsŸ•ζίpƒ|ώ§?I•ύm·H¬}ώ‡?Θ§?ϊ‘t:ε9L·o3b„Σ„Λοj€uŸ•b­ύρΗΛ8 ΰYΫ₯Ÿ•ω?ϋ™¬ωΫίD\Α7{-φΩί€ε‘θ6Ο+’Ÿw;V[o °ΖνΜ @ΌHT 2oNA@@ Ž DWT²Ž8ΓEά° αΣ4 ΓT έκΏόE^?δYϊλ_^ψN΄Ώ)κEι75jΛτι‰6ΛΘϊ»vΙLΝn™©D)―^h_£ηωφQG‰³Υ²9ΞmšψΦΐΞ{ΐpŸσ>ΝĚχ“ŸΘ»Γ‡ΛNΝςsš,έχ|ΎBxohΰΘ²,ƒ‚jώCWνί/λ΄TΣ{(ϋτškδ@‚ΰ”ΏDΛ–1φΞ°a Θψ‚jAϋ¬Σl―78ΒΙψ z=ΩΊsη:Ώ/ oΎ9aP-hΛĜuΑ2ύΛ_ ₯Ϋμχσ ΝΤ²ΰΉ?p˜¨/{/–?π€σoΓςGM΄Yυe›7ΛΝτ³`oΨgιH?+τXο}΄X°5ΥΆπΏ74“Π~Ηƒ‚jύθ{½β‘GδMέoλΜ™›°@ jzƒJώIpF € €䆁΅άxυRΐ²PζkfIΊΝJ~xΡE)_lκίΚ.NΣruΨΙF³ω ¦Ÿzͺ¬ωΏKϋpVΪςƒ―έS0νNRά!›γ΄ΐΙ-ΛKqˆNιΜΙγΗΛΟ>Ku—H·³€Θœο|GμΨ‘~Ώ„Yͺ™c`Ϋο+/šjgλ4KjΖYg₯}όJύ ΎΞ9²R3θRmΌ™7;Ώcτ» |Ψ¨ίγηjEk§σΚ²ο[4ς@ ΏώFCΘokωύώrv䬀•ΓsJψ}1B»hέQ/€ΫO 6h ’QΫΆqγ·Ή›l'wk‘½‡?ύ΄΄ΠRwξΦ€sgηΒ‡cuζζZ£Ž™:g™;(Qeίμ"y[Ν ΚXΣ ό›τΒx¬΅9Rj ΛΆcΗΖVy~.ΡΧζ^{­g-¬ΥΉ΄2XΛ8+ΏρΈ š£hω½ž:‡”ύ­Tƒ±VJΣ‚)Φ<ΖXή’A‘ZΡߚiY>ϋό΅ O\Σ`οQχί/sόcYvί}ž—7Ώϋ3Ψ‘:wYͺΝ²cΝ‚Ι6§œqh°ΛίϊipωΓΛ'?ό‘η3oe)giP읷«‘š΅* }¨,P­±ώNύΤSbΏSξfΏkΝ΅4f{-‘hΗ΅¬>›£Μύ;ns―Y@iΔ3ΟΈwυ<―Τ±Yv¦?¨VΠ€‰ύδ“υάs=ΫΗζ«λ§σΘ}¦Ύσ4Φ½―ύ[±θŽ;df ¦άŸ) β₯e%یΈλJ ΞΦωάάžΨ†a5 ήωΚGι<Š½5Hζn\kρΕδ‡\u•¬Υrž(tζwϋbC›[ΞζΨλͺ™ˆ4@ό8Co*³ |ψ‡Ξ7}Ά–Ζχ·ρz“š½FC@κ‚@ό•ΈΊ0jƈu^ zŽ1Ν9τΊλd’fX€‘‡^Te5nί>ξ<-sΙέuθ 4ΛΛTso{ήU€ψ«_Ε«Φ€μ[υΞi>ιτ•―ΘψwήIT³ξΥ γ!xτ΅MΊ_MΛϊΊ ]Μδ8-8j™\ξΦP3Σ,{«ο~T³m[λVφ9ι¨₯ϊ²έ¬D¦|όAΛ³1ΥΎd‘ΐξw2HΟΫί>ΣωΓ,«2έV –£ž^,k,(¨fύI_Ν uY……žCXισ$%!- ·E³Οά­P3ΝΖ½υV\PΝ½=·y`ϊkpΝ‚~ώfσ‰Y ;Q[zο½²έχΊeŽΡŒ4PΝέ‡Xπ)θχ{΅ς,P˜N;Lης›€άDA5뫧–£ώ§?Εu»αυΧε€–š jUš1θχ¦³^(υΥ‚φνrζ™2ZηΦS`ΟΛΛ4ψGC@@@ˆ(Τkb±ροκMξQ5kQIΤH`„f[yΧ]RΠ0<Ά²Ό\Άk =ws1!₯εάΫevΔ•sοασVΓ†ΙH͚³@KX³ c#_`Ρ2²1§R&Ηi ¬ωΫQ:ߞ•R kZ·–QšΉce"³Ω>Χl©X¦\μΈΕ½{Λ %ΚϊŠmϋiŸΥΞq7ΛZϊΫίΊW₯τά?΅”f*ΝR½4 Πί,Γ«J³ύΝ֍ɲ,cTώ}‚–{^|±t Θ€² ² fŸehτ·C4γύΔ‰ώՁ˽/Ώ\Zϋ2ΜΚ4pi%VSmφYζd’€₯»ΛlͺegέΝήΣDΑC›‡ΝQgϋ΅Σ9μRmVf΄“/°l}@@@@‚,3>φή"ύ΅Φ7cˆHΐ.ψw=묔{³ Z•Χά-‹Ά_S£Β_b²lΣ&w—{>T3„,‹(•fυ;zjά¦VΊ2Σ-“γ\§7{W­ςœ‚•ά릙W©63<\Ke«9&Ν€ς7 ¦T‹νk% ύmύ?ι_•tΉHƒ‹‡\}umό/φΏεgή7χϊ]:Χ—?Τ^ί ™ƒ6—›»Ω1{hωΞt[+ιkV0(σrέK/Iu&λϋΨοj:ο΅eΛΩ\‹ώfeDSmAcNΆo§“OŽ{9ΡΏ)₯6/›―ٜxι4ΛΨ΄ nμaϊ{φ€ΣΫ"€ € € €ΤB€ΐZ-πΨj.`s&yΟ=iu`Ω‡κΌFL§YΦ[ΠEύtϊ¨ΙΆmΏτ%i9p`Z»Ά0 nϋ² Φ2=N ͺψ›erY@$ΦCη*ξΣ']jΌνV[mΧΒ…žύΝΙζL·YΖ‘eΊΫΞωσeοΪ΅ξUIŸwΧ’”E-[&έƒ́Φ1 dηζo6G˜Ώυψζ7SΚ΄τοg%Z›ϋ?ΗZ–1(ƒΜζ σ7+IZ€eBΣiN ΛχyΪ1wnJ]΄><νίΣ&]ΊΔυ]ž Xˆ΅ί‰ςmΫβϊH΄’O\ΆΜσκ7Ρώ¬G@@@¨@xν΅ΪυΟή €@ @›Ρ£₯ΨWB-pCΧΚΝ°GmΪζχή“ͺjΣEφM΅”»σ 9ζΚ·lqoωσLsλŒqcnΜ1qλΒVXιP›oα­·†mZλΧ7ϋζ³یYγ~ΫMš$Ϋ|™K₯οΏ/M}e" έ,ΝX?4ΈVK‘Ί›3ί—–Zt· `[―Λ.so’Φs+ΏΊpή<Ο>fκ/iT’ΥΏ§“ mήΕιΣ₯²¬¬z ΛΈK₯%›S-Ρώώ XΫ.Q<θ½Ϋ£A²)ϊ™­{–QKC@@@ΘmkΉύώ0:ςV (+S'k₯ό¬όΰϊ—_–7ޘ©Γ$ν·δΠC“ΎτbΓ-βVWξΫ·.Κ™g™ύ%rv=zΤθ²5ΟΪ–;n|ώ¬³Έ ’¬hqδ‘q―ξ^Ύ\ΖΏυ–l1CVύωΟ²ξ₯—doŠ%A·N›&3υΡνΒ ε觞’ό›–cο.ΓA@@@\¨¬¬”cΎΈ±Ή  @ή~ϋνH†E`-F:AlμZΊT>Όψb©ΠωΤό­f„ΩœD­†•š•Vά»·”θΓ~4δŸ:ΏΧΑXΚ\ςzWY@ΐ&ύSΩ6hΜΆίΰ(³ϋR-kYQV–Κ°·±2ξ2Ά‘?ƒ­aΐœf–!Z›Άo͚Έέ‹Z΅ς¬+,.φ,ΫBPφ\άFy°’Ν¨QbΑχέηΜΉρυΧe“–·΅ ΅Gχ)―Φ€\Σ]εH-IC@@@π TUUΙ{ο½η¬΄ΐZT«ΝQIdT ςΐωΰλ_ͺΚ·ή*½.½T„9™2zyΦΉ?˜b§·χσΟk|–e7ΦxίTw΄ΰ™^2Ϋ·½ήιόπΓSν&²νΚ7mͺq_{uσΦ‚2βjσΩρ‚sώίΥ μ΄=γυ?ί–[jv­=½φZ©ͺ¨?”―½&^}UΆΎΎˆσ·%wίνζlξ5 € € € €@ζ’ Ρe~¬κ±€eq”Ϊ…eWk ™h#žyFΏα†”ƒj £ΩAEP°'ΥΡm›=;ΥMk΅]Π\_»/UŸ5έΉ,βΐZγŽ=Ciœή“byBOG…=Λ—»–ώσ΄εQGyΦ5ξΠΑ³l ΅ θΕuXΗV4Π›ڌ!ύnΌQ&L*'ιϋpψO*’λύmγoψW±Œ € € € !k‚₯[ˆV`ϋ'ŸΔuΨλ’K€λΉηΖ­OΆbοκΥΙ^ζ΅ 4;μ0±`¨»ν©EΖΪΊ^pw•±η­† ‹λϋ`ΦφiYΖ}λΦō'•Acn3z΄gWΛ–ςnv/[&΅ H―{ρEΟ1l‘΅Œά­•ΪΉW9eΛ6oφ¬KeaφνςΌΞ­hs)Ζ³ΏσTvΝιm¬δγ·έ&ß~:nœ₯|·Ž € € € €™ °–WzEˆv|ϊi\mǏ[Άbηόωa›πz†κ)ΟΚlnΆ-ώw [Φ~“ ΟYmŽ=g?“Χϊφυ·]Οϊ―ΈuΆ’Χ₯—JƒF<―YωΚtζ³mΧσŸž>ڎ+E-ZxΦŒ |YΙKw³Ή#-›0έζ}iuτΡιvΑφ €@ήtοή]&Nœθ<ϊτι“7ηΕ‰ € € €@ξ Δ_QΛέ±22¨Η-}€bυ_ώ’²ΘV Μ8λ,‘ŠŠ”χaΓθZk ¨΄βϋgŸ-Ϋ>ϊ(τ€ΫζΜ‘©'”΅l΅Ψ€½ξΊΨΣκŸφω[σ·ΏU/‡=Y­™fώΟlA“&rθ~Άkάλ_q…μ]³&n½Eεώύ2[η"ά‘ΑHw³rŒ‡kIΚ ΦLΛ v<ε”Έ—fiF@Κ%+uοO―ΉFφωΖΨbπ`ιxβ‰q}ۊ&;Jχ /Œ{νƒo|#₯ΟFωΆmςα·Ώ·ΠΉΔm”₯ώΨ… εσ§žJλθ›4+£Μ7Ϟύ^Ρ@@@@²#@`-;Ξj)ΠjθΠΈΦώύοNΦP²Rq6Χ\ ŠΌ§%m¨ Άoύϊ Υ¬Λΐ€»ξ μΈΫήΟ?—ΙšY΄JKZ0ΘίΚΆl‘Ewή)ΦΜ±²ƒπ~u@S'-aθi€yξΉ2Ο²Π’dYζΦl͚υΥ―ŠΦ5τt1π7Ώ‘’ά]Ώϋ³Οδ=υ*ύπCOξϋμO=αYωψγξΥΞσž_,%½zΕ­­8RΛUθΌgξΆwΥ*yoάΈΠ9ρ*ΚΚdΞwΏ++yΔ½»“w”–v Κ’‹mxδέwK£φνc‹ΞΟr-•9e€€s[¦N•wGŒ}«W{φm~ΔηΚ+=λζB―ο|G¬4₯»ΩgΓζέ ϊά»·³η›ή}W¦ϋ?‡šuχΩτοΘ2 € ‘ΐσΟ?/εΎ9m#μ>ν¦ie…Uϊ]•† €ψ «KΗΏ­ΥΜ’jή«;QυJ? €@ΔŽ?^Ϊh a«^@w7+­·φΉη€οUWI‹€¨uk±ΐnΏhώΑ·ώε—=₯χŠ5˜Π΄[7ΩβšΫΘ²Fζίx£΄?φX©Τ @’Œχqy^sΆ£GΛ ω±¬+w«Ψ³G>ψΪΧdΆf΅2D,KρΐΝ²gεJ§Tdεή½ξΝ₯›f7­ώσŸ=λ2Ή0쏔w4ΐ»gΕ Οaλq^}UڍοŒΩΖ]ΉoŸXΙJ{lxε•Έ}¬ƒη'} FΏ Δ².χͺΛ»š©ΤnβDι’٘φΩΆΉΛΆi°ΝΚ6nΦ Lωζ͞±ΪB‰f€‘sΤ%k-ϊχ r}ςύο{6³ΰ΄e ΪοcλaΓΔ2°œσΥίΛ Ϋ€₯-Λ₯ώΦη{ί{ο“΅ΖνΪΙQ=$35ƒΡέφk6šU[ $mF’6#GJa³f²sή<±,Ζu/Ό β nΪλ#5ψή°€ΔέΥA}neE½φZY¬Aβκ¦ΪE·ί.΄Δ§•JmvψαΞΓώͺ σ^ –Ξš%6?䦷ފ Ξ~Γ FƒŠ4@@ [ŸιM^‡κwΚτ{ΘΕzΓV#_9οlΓj?ωΟeΣ¦M2{φμl–γ € €@°ρQ7kQ‹dDΐ²\ŽΦωšήRršδnΫ>ψ@>ΤrqaΝγ5Ψ`Yξΐšν·H φh’³OζnΗ0ΚZΏn% bΞύοŽ+λhA©­Σ§;D²yΙϊώΰρ5ίά`‰φ―ΙϊF΄­Ϊi'Ÿ,Τr·νϊ‡Ό=RmΝ5p5δΡGSέΌz;Λ2λ’A§%τŠ5  Ω#•ΦΈsg«σz5φe…νΫWa–χΩ½χz^Ά9β6όλ_ΞΓσB’…n\ ƒ|ύ$ΪΌ« ?π€|l™f Ώͺcs^―‡΅‘κΫΌ_Ώ°Ν²ώzΏ›n+ ΊGƒξfŽΩ#Φ¬L¨ύŽΈ b―Ε~ΆΦcΏ›oŽ-ς@Ȋΐεϊ]ώnύ>ϊ]­Rp‡ήd–ν[, φϊσ[ @Θ¦₯ ³©Ν±@ VV2o¨fm§έe7MΤy֊{φ”N§žWŠ0νΩ‘Φ‡όπ‡2φΝ7₯YΑ+£ΧΦ[e¨–7¬ ˜/Ο2Ά2Ω,“kβŒjψπ¦ϋΧΏ.ή_ŠZ΄¨Q4Ϋ©w ΚZfΩ8Ν(KV? AΏύ­τ·μ6 lΧ¨i ³Χ₯—Κ0 Š'+ιοΫ―CŸxΒΙJσΏΆά΄{w¦YsέtNΈ\l…Zbs€ΞΝΧ€kΧ€Γ³s² še؎Π]―΄dNy@ˆ@ X»ξ‹9ˆ?ΧJ`³ Ά‡~8£%"- v’ΞΧ;V+™Δ‚jC΄Ε駟ΑYΡ € Ί@ ―”₯~ΆD’θ¦σT;wtΠrt‘M/κ·Υς|c΄Lίπ?ύ©:K§qΫΆ2JKΗζP‰ΈΠsΙΣ ΪkyΏγτύͺeΫsLΒ€§Ν»Υ[ƒ-ν§e;hIΔύ;vΔ©ψηηŠΫ ‚M:u’‰šQg% uθr-΅ŒδΡO?-GkΠ§¨yσ”χσoΨ@?ΧGύώχ2Dƒ‹–aΦλx‡h@z’–ˆ΄r©ιΆ~ZnΥφ΅’“ι4 >NΠ‹CτKM‚?Vρψ œ’™ώyΙ‚Ζae`θΣΗ/^,=4x™ΛΝJNμ4ΛδKεάάηb₯<νί―q”.ξΡΓύΟ@@¬ XΦZΧwaw€ν!ύžεlP;AηvΤb'z3Ωϋ1 ~"€ €@TiΛβρ8 ™€ΝŸΆUΘΪ‘σ,Ωγ€–ˆ΄y‰Š5cΕ²s,3¨€wο„ΗΫ­seY ½ν}$’ΑŠ¦šAN=­k‘”π`Ό’ΐ~Smχ’%²oΓg~©Ζ;Š=μ=΅`š»Y9½Yhu·γtΎΌζ:GUΆZ…ΞϋΆQ³ΐlŽ―mZ’pίΊuR^ZκqνσdŸGΛΘλsΗΥ$¨υŠžχ>c+ΦJϊφ•–.-J₯–G\ϋ8Η·yΝl>:sj~ΔΞÎΩυœs€‘Ξ7EΫ¦Ώ+λωOΩ¨ewl.C{Ÿ,³ͺ‘«­Ό€e‹Ωόk–eΖ²-[dύ‹/:sΗ•­_/e7Jc½cA¦f_<μχ6Σ‹QϊϋΨ£Žkτ³ΌU3!­€›Ν7h­‘f5Z¦σΠχΎΕ‘G:ΑΈ‚’"7,#€ €Yψυ―-Χκό±A­‡ήtύυΧΛ·uώdχlΠοgϋζΣ΅ύΟ<σL±ΧάΝj·άr‹Ό‘ί·ƒše«1·Z λ@@ ΣΦ2-L €Xς«_9s΄Ή;²`΅iγ^U§Ÿ‡Φrαδͺ*+Σ*σ˜ cΞυ1Tξί/΄­iΉΠ\??Ζ‡D)πΧΏώΥ)?g}~UoΈΉμ²Λ’잾@ ‰ΐ½¨·ήΘΈQo JΤόΆTka΅Ψ±ln5Κ@Ζ4ψ‰ €Ωh˜Νƒq,@’°,Ew³žV–]tζNΛξΘκξΡ,#¬΄Ίϋώ1rΘ€•Ÿ{ηwœƒvΤ,χώ:ͺ»7N§ M>ΒΤ©S₯"`ξVw?V‚Π—=ο~ݞ[0ΰ€f“'kcƌ‘†!σcNΧ’Σϋυ&‹d-ͺ~F-E!™Π34«:¬¬ί¨Q£φ½(Y[¬Σ(¬ΧŠ0ΙΪa‡&tz‰d-ͺ~–hυ’uZ(Y³ϋ;wξœlYͺU~Φ]›t›¨ϊ9δC€K—.IυΩgŸΙš5k’nΣW+Τt ™‡}Ω²e²ΪUΡ(¨Γ¨ϊ飕sΊ…L=²|ωrY₯x’5»ω¦»V6JΦV¬Xα|WHΆMTύτκΥKμ†ŸΘš•‚€!€  -Σ§W-ΈυVΟ£tΜ΄]YYYυr·nUZ,¦ϊ1γœsξ'ΧwπŸγk}ϋζϊ €YΈϋξ»mjƒ„½€:½πpXίz‘9΄½Πڏ^H νG/€†φ£.CϋΡ …‘ύθΎΠ~τΒ\h?z+΄½πڏ^0 νG/τ„φ³rεΚΠ~zφμڏ^4 νG/>…φ£ωBϋΡ‹s‘ύθEΩΠ~τbjh?zΡ:΄½hڏͺvοή]₯“C·ύ.%ϊ™Κη5ΡΎ±υμ =/ š…ŽUƒ“‘ύhP1΄ ‡φ£AΰΠ~4ΨڏΙCϋ™2eJh?z3Bh?Μ νgΒ„ ‘ύθM‘ύLš4)΄Ÿ7ί|3΄ŸγŽ;.΄ŸΧ_=΄ŸO<1΄ŸW^y%΄Ÿ“O>9΄ŸώσŸ‘ύœzκ©‘ύΌψβ‹‘ύhζgh?Ο=χ\h?Z5΄ŸΏύνo‘ύœ£ΫΗ~―ύ|φΩgCϋ9οΌσBϋyζ™gBϋΉΰ‚ BϋωσŸڏή\ΪΟ“O>ΪΟ7ΏωΝΠ~ώπ‡?„φsρΕ‡φσψ㏇φsΙ%—„φσΘ#„φsι₯—†φσΰƒ†φ£7s„φsύχ‡φsε•W†φσ»ίύ.΄Ÿόΰ‘ύά{ｑύ\}υΥ‘ύh‰ζΠ~ΉζšΠ~μϋuX»ξΊλBϋΉσΞ;ΓΊ©²Ρ‘ύά~ϋν‘ύότ§? νηV½e#cM…¦!€dG ²Ό\άt“η`6/ΨΠΗσ¬ [XϋάsžΉΗlϋs5„υΓλ € €@Ύ θ…ƒ|=5Ξ+‡Š‹‹C³ΦRnXvJ*}€² Ώ©(±M}ΰχ’Ύ½γœ/D!@`- Eϊ@HI •?ς#\εNΦώύοΦ[₯iHϊ|μεZΒγ“ο?Άθό,ΠR?ΏόeΟ:@@ ¬ΌΜψργžhXHΫΡΚ*†•Ξ +ίλ'¬Δ\XΩEλΗΚ3†•†K΅Ÿ°j5²C&mVζ1¬dYXΩE;€•g +–j?V(Y +ίhϋZ™Η°r@©τceΓΚ7…•o΄ρX?aε­,ˆΦ¬W νύBv‚Φ §!€ € KZ~J’Ν΅ΥX- eσ€Ρ@@ƒ)@`ν`κsl@  lzϋm™zIR΅Ώημ tbψΎW]%ν9FJτΞ—b½“€B'¦ί§“ΧξΠɝW?σŒ¬ι%©τMVί\“½†Lδξ9XY °VGή(†‰ €ΤsΛβ±»Χ7nܘQ‰ηυ=*£Η s@@ L€ΐZ˜―#€D.°Jƒd|ύλ’υAjΥ·••ό”)RK{δc#°–ο*η„ €δ§@¦³ΦΘVΛΟΟ g… €@](¨‹ƒfΜ €u[ ϋό§tcHύώdgΩZηΕ8fΜΌ ͺ%;w^C@@ Χl΅TζͺιΈoΎωζšξΚ~ € €@€Φ"ε€3@Tϊ\q…·p‘τ֟…)LΔλ·XKΜτΏν6'S­D'o₯!€ € €ΐΑ(ΦοτΧ]w]FbΩj”€Μ-"€ €@ (Y4vAˆV Ό΄Τ™Cmη’E²wεJΩ£½«WKΓζΝ₯I§N€sg)ιΫWΊœ}Ά΄92Ϊƒηpo›΅Μ₯{N9›Gν˜19 @£QΑ·k Œ ϊ².Θ,LyΌ€+%΅8Hβμδ‚’Ζ [ΉΌ€Δξ²E’²Αμ vЁ@φ;Β>Vaί« r²?Ω|I`6Θ.Ύt[Δ†Ϊ ‚Ž)ωI© ίkZZZh’θ‚ $΅’D;ηTe¦g”(8C*UΑ3/YOGΑΘΐΘ€ξ՟ΑαΙ(v!†±9 ώKXώ ΔLzθ00πOEˆ©20θ30μ›“\ZT5†‘Ι˜=`JcΜΕ‚Κ pHYs.#.#x₯?v@IDATx인5Ϊ€#X°‚" `ο’ vlˆbο]Δ²ΦΕήΦ^PΧ†}±WΕή Dμ‹ˆP\AD"ηŸoώ»sη$™™S¦œϋζyξ=3ιy“ΙdΎδK)8Fa @€ @€ @€ @€ @€ @€ @€ a2œ7²@€ @€ @€ @€ @€ @€ @€ —J±4@€ @€ @€ @€ @€ @€ @€ Μ@)6σUD!@€ @€ @€ @€ @€ @€ @€PŠ₯ @€ @€ @€ @€ @€ @€ @€ džJ±™―"2@€ @€ @€ @€ @€ @€ @€ €R,m€ @€ @€ @€ @€ @€ @€ @ σPŠΝ|‘A@€ @€ @€ @€ @€ @€ @€ ”bi€ @€ @€ @€ @€ @€ @€ @™'€Rl櫈 B€ @€ @€ @€ @€ @€ @€  K€ @€ @€ @€ @€ @€ @€ @Θ<”b3_Ed€ @€ @ iύυWI’ @€ @€ @€ @€ @! qΗ€ @exι₯—ΤΰΑƒλΕ³ϊκ««ΣN;­ž]΅o~ψαΥ·oίzΙ4nάXυιΣG-»μ²υμ³rσω矫χί_Mž]-ΏόςjΕWT­Z΅R­[·V]»vU+¬°BV²L> @Ή"πρΗ«aΓ†©>ϊHΙ{χΧ_uί·σηΟWmΪ΄Qk¬±†ZsΝ5έί΅Χ^[νΊλj‰%–ΘUΙ, @€ †H +rι ϋ<Κ©ƒeΰ¨ qγΖΉ²(]l:tP-[ΆΤ9aWEΜΛU.m½:\‰΅L`ξάΉjΤ¨Qξ{rƌjΑ‚jι₯—V«¬²ŠZύυkΉθ” €J$9rŸ+`%ΰ[«$l‚@ξ𬛫 ₯X3\ @€*@য়~RGq„ϊν·ίκΕφψγΧ»Oβ¦}ϋφjψπακΓ?¬—άΌyσTΏ~ύκΩ₯y3fΜuΓ 7¨—_~YMš4)4+’Ψ»Ν6Ϋ¨½χή[xΰͺmΫΆ‘aπΐ 'œ ξ»ο>οΆήο€Τ!‡RΟΞ»)5œ>OΏgu–‘Ρε—_^qQΔ[rΙ%ΛF΄Θ"‹ΈJτ-Z΄pOmΈα†ͺ{χξͺ[·njΉε–+;~"(&t[)Ξ6Q Θ)°=τΊνΆΫΤ§Ÿ~j φγ?*ω{ϋν·λό΄kΧN]tΡEκ裏V‹-ΆX=ω$pάqΗ©!C†h3ΔO¨Ξ;kέ°„ @€²M Ž\Z6!, EjΦ¬™ϊώϋο‹μΛ΅Θ‚œωSΉ΅XzψέvΫM}ύυΧΪƎ«5j€uKRd%]Ίt)9ΙE]ΤέΘS6σ” Η6ί|s΅ΥV[Ή²Id)υ±^xα…jΠ Aυ-{wΧ]w©γ?^λ†ee $=/ΧεQ΄υΚΆΩ$cKzœ”dΩ*•VΉγ*yχΛΌƒ·9§όnΌρΖj»νΆ«TsΜέΘ:’ύλ_J64•uA³σΞ;mΘτΓ= 4<qδ@Υ¦“Ή”±ά1ŠΗ‰΅G‰hΏIΛ}ψ֊V/Υφ%‡έȘ^6r1™Ν6ΫΜ=ΗδŽύ4€u©+΅ύŠgέΜ₯X3\ @€*@ wοήE ±ς'Κ›i˜λ―ΏήU υ§}Ηw¨ƒ>Xm»νΆ~λΔ―'Nœ¨.ΉδΥΏ%>Qψ:t¨ϋ'?Η™gž‰’NT€ άίΒ… νM·ΠΓUj8/|ž~ηΜ™£dbωσΟ?uΦeΫΕιl‰‰ΠMώFνž„)ˆBμω矯N?ύtΥ€I[pάbH£­ΔΜ"ή² ΉgϞκ½χή+‰Η„ ”`―ΉζuρΕ«^½z)Ω “O2A*;*κL΅ϊx]ZΨA€ @•%G.=}ϊtmβ²θ―Z& rjδOΥͺ]{Ό"W0}‡ΪC&λZNϋ°ς½-_}υ•zύυΧέΜ―ΆΪjκK/U‡~8²”d«“Τ š—Ce¨¬3I qR&A„dͺάχ¦Œ δoπΰΑu)mΏύφͺoίΎj“M6©³«ε Qά‘5$Ο=χ\-“²A€@•Δ‘U) υ’Ν‚άG2TΞΕ_ ΦωiΨ―σ"χ±—ΧΈ{μ15wξ\k°>ψ@}χέwjυΥW·ϊΓQ©†΄.•ϊ.Ÿ@ϊ[L–_b€ @Θ(9]λ™gž)ΚέW\Qd—”EΧ]•μκ7’ψwμ±Η*QfJΛ 6LuκΤI=πΐe €fϞ­Ξ;ο<7#F€Uœ\€+u. ΪΊ?ζa P«fΜι*ΕJψΛ/ΏΤj1)΄>ϋμ3χYͺB¬?ργΗ«cŽ9FνΎϋ!όyβ€ @€”Κ’\:X/Y•SσYκ=ς§RΙΥv8Qφ9ςΘ#ΥlΰnςYΫ₯₯tY'ΐΌ\ΦkˆόA²)φ¦›nκnς9i€šrΛ-· [σ΅L!T‡@ε@΅.χ‘šDφSφL¬ω$0pΰΐHδ‘G"ωΓ J±ΡYα€ @ iΣ¦©“O>Ή(Δ†nX€”Zδ©ΚgŸ}vQ cǎuw(/rHΐBa»wο^Qε4Ω}]βόβ‹/(A>“5j”>|ΈφoήΌyω,Ή†@ ’8/'dρΗ1Bαω% 'Δξ²Λ.Κ΄³}©%“Ϋ%ή°]KŸp€ @€ @–εΑRdMNΜ_%ξ‘?U‚bνΕ1fΜ΅Ϋn»©·ί~»φ G‰rA€yΉ\T™„²Ή·,°οΦ­›’qn­™_ΉρΖk΅x” €ͺH Λr † χ‘ͺEφSΕNΤΉ §Ώ~ψᇑςŠRl$Lx‚@,(ΕΖΒ…g@€ ¨Ξ9η5eΚ”"οguV‘];ν΄“κΨ±cQ²2Ρ"'Θ%iD‘FN,?~Ε“₯Ÿ=z(Ω}@@G`τθΡκ²Λ.Σ9aš#pόρΗ«Ι“'W₯\r²ΔW\Q•Έ‰€ @€’Θ²\:XŠ,Ι©ƒy«δ=ς§J¬ΈζΜ™£φΨcυΑΤN‘(I.0/—‹j"“€@€ΐΧ_­:θ €mνάΎόςΛ꧟~ͺQ@HŒ@–ε@ Eξ#•μ'±&OB$GΡU6Šϋδ“O2X ²ό@)6ΏuGΞ!@€@f Θ€Lώύ‹ςΧ¦Mu葇Ω§a‘SΞύλ―ΏΤ%—\’XvδΔΊΓ;L-\Έ04ΝF©vνΪ©Ν6ΫL­²Κ*j‘E #dςhχέwW ,ˆδO€@Γ#pΛ-·pͺtΓ«φWβ_|Q½ώϊλΦrwθΠA]xα…κώϋοwύΎωζ›jΐ€κςΛ/Wr}˜ΉξΊλάɞ0ΈC€ @€@uδA.,yδΤΑ}T―^½ΤRK-U€@ΎtμΨQνΈγŽ‘3-"hβ‰'Œaή}U("ŸBmŒd€,Ζ|ν΅ΧŒ9“wνAdtχδ½9ωRNZΓ† Σ&=sζLυŸόG­²Κ*Zw“e5ϊfSZ&ϋ$ϊ„΄ΫJ–ϊ€ίέέ΅\ήΊMLυ$φ’œ*Β]yžWZi%›ΧŠΊΙβKI[gV^yεH ±^XِBNV>ηœs<«zΏR>ιηΧXczφa7IχYxvK}'Κι2rβΜ”)Sάz•Ύ[ΪΣ"‹,†Ή"ξσζΝsίURΧ’¦€-yΘΣx€" ˆ€ @#P-Ή΄©˜"£™²Λ.«Z΅j₯_|q“ΧPϋ4εΤ‘™‹θ‘ΦεO‚! TDάΉς&2’W^yEν·ί~κΉηžΣζ]δ"όη?ΥσΟ?―u·Y&-S‘ΌˆœUδM"XqΕ•Θ—DI8+²‘H₯ZnΉεάy”΄OΙΒσ“§yΉ`›OR•%Ytƒξ>IY!ύτv•'ιS¨M[O^tΡEξΖΪΟ>ϋ¬ΆΘ)μqΠ§Ρvu™·Ν ̚5KΔ΅k€‰Ρ-ͺC–ϊ6‡¨ερόeyΞ. σPI?jύ]δ΅7~!5Υ”UϊΉΉΤ?²Ÿ¬=Ιη'‰wke&’WΩθΝdd-οgŸ}¦ueΪ8czm$-«]oYχED‘Έ·¬ΛHςVq(Ε†Β€ @ 2Y ;-λΜΞ;דּΆΪ?ή=)υ­·ήr•TE3hδΔTYpΠ½{wΥ³gΟXŠ¬’ΐ £GF«nΉε–XqE`±ΣκFŽiτ!B$ωπ³€BόήyηΦ]f%έ¨J±2#J΄/½τ’ϊχΏ­δ#ήdd±Η»ξͺŽ<ςH%;ΔΗ1χάsϊκ«―΄AnΊι¦:—οΏή]#‹fD Ζ3’Œ"υΏΞ:λΈ;ΙΫvŠύβ‹/κq7ς'qK;φ+qoΆΩfn^Ίς+' K›iWβζQς” La)\Ε«|Π½6ύKͺLι'i/Šθ"$zυΥW]%U]ΪoΌ±:πΐέ?ΩQΉ#‹«$½_|QΙd‘߈ξQG₯N:ι$WΙΙο–ηλE]Tν΅Χ^F₯X)ΫηŸV‰ [ΝΎ9Ι>!J}¦ΩV’θδq―_ςσώΟߏΙ‹«ΊΚέ€A”=#οžΞ;«3Οξfž›χ+§’?τΠCꩧžR“&Mς¬U›6mά°;μ°ƒϊϋί*±¨‘.ςΐΕδΙ“6»ν₯Λn"^νΎϋξF₯X‰BήYQ•b“μͺωμVϋ8cΖ 5`ΐχ]*§υ•œ—Xb w£ŽUW]ΥUr>βˆ#ΚZk ²aJΏ~ύάΊ²‘€ίHϋ•Ά|ψα‡+™ΤLJAן!@€ А TZ.mb9dΘuΫm·©7ήxΓUτόΙ7€Θ™χίΥ«W―Θί„^ψ΄δΤ^ϊ•ϊΝ‹όIΚ›5TΤ:xσΝ7ΥϋοΏoτ.upΚ)§Έ’=₯ΰ ™ύϋχwεA&₯™ΏY­(šG1IΚTDpυΥW«Αƒ»Κ°A™²—_Ω SζΒDΦuδξ»οvη`Ό8όΏŸl»νΆ~+γ΅,@|μ±Ηά?a9wξά:Ώ"Γ[o½υΤφΫo―δτ^Q”#ΚΚ"³Σ9ΉWN ™<>?y˜—Σ±NJ•„,ΊRm]8%-+¬•ώFΧΖ*iW©q{ｧxΰcΦN8α΅ι¦›έύΗI_§3²Ωα•W^©sJέN6ε4)ΕJζdξ9Š©fΫ-wN@Φ,ψΧKΨΚ$sώρƒ¬‘qw˜I’o+—ƒ½@-ΜΩUs*kγZ~…=[ΈC ͺ!ͺζs]+r©{d?Υ}J‘ϋ”ϊ­•₯wk’2JΦ Τ—i=’¬]εύ}χέW›δΣO?­nΏύv%k]☬Τ[5Η}qxˆί,}CζA†XΝ<Ζ­»Šωwv‚Β@€ @ "n½υΦ‚3PΥώ9‘Σpn 'žxba±ΕΣΖeJC읓C Χ]w]ΑωXŽ”ήi§¦MΓYxQp”p#ΕΧ“£p§MΣ+ΧA7Κ:Ξ"c܎’SΑ9ν°Ξ―ξbώόωG΅ ~½όΔωu&  Ξ yΊ¨΅vŽb‘1GΧΝ―3ρΉ-8 D Ξ"mZW\q…1­°2JΫš 6Ψ@Ÿ³ΛWΧoΌ±ΰ/Šό9J±u~‚IΧΑ±Η[”?‡sΊh0{uχ₯†«‹ΐΉp Ξβ cϊ^>όΏΞͺ‚<·Ξδ§?ͺHΧcƌ)8JK‘sœEDnΌŽ‚¬1Œ³»|€΄γxrvy6¦',₯ν8ΡΥωu$Xγuαλόκ.’θ›“μteτμl+IφŽ΄ΆM8› Έ(œ~G™ZλΗ\ΚυŽ;ξX~Ϋ3ΞΒ»‚³x.RXΙ‡sβΉ΄βΏΞ† Ζ|8 KJΟQ/8rhœ’‘q&Ω%ρμVσθlTRp•λ0ΨεήQΊ.\{ν΅gς2΄.lύŽ#ϋμ³ ςώΡ₯΄s6(Œ5*4]<@€ @•#P\:8χξ›5kζf9ν¬ΰlΖι›ΐ9α«ΰlWp6σ‰Uΐ€εΤ Yώ$cϋ¬΄\:ΨLίΠξό²•`8‘{;ʍΦvθ,Œ +ιήQR±¦3qβΔ’βu6“²Ζk“G{ &)S‘ϊp—œΣ`­ωφϊ ο†nXˆRžC9Δχ]wέεΫϊ+υΥ©S'c<ώ|‰άΟQ’rγΩΆίΝνl(iL3Νηǘ©‡¬ΝΛΩ&)JR]‰Ά.ω­Ά¬Πί”j­Ώρ—-Ξ΅Ώoπ_Wkœ$ο›,Άwοή‘³/σΖώ<ϋ―eμU)6κΫ·o¬€D¦οΟkπ:,Ύ$Ϊi<užΌœy{gΓn+Ο$ϋΆr9ψ ’η9»$ζ‘lοΞjίύυTΛο"9Ή†@Φ ”+ς—/©η:iΉ”1lŒRΛk€όi½;LγΣUCξSκ·VZ|€nό&i™†?νr―Ν̌γvYc2{φμΒRK-eτγlβ; iΧ[γ>g]jΎ!σ CL"±t…¨ ΕC4€ @(8»Lk?βœ]ς#Σqvs+ˆ‚Jpr'ξύV[mU˜>}zhΊ2ωoŠΫ9!04|)V[m5cš’ηDΎR’uΓΘ$˜©Τ?ώψΨω%TηTΨ’r–3ΉVŠR¬(zL‚Ώ&₯Ψ4κ ŽΑ΅Τp^?όpA ΩD½x]={ρ ’μ5~ϟ,zͺ₯XQςΚ₯ϋuN3b«»OͺoN²O¨+\ΰ"ΝΆ’t`›`— ’mΆΩΖΪf‚νΘΩUέ₯)am‹M‚αδ^ΡΛyͺ±ή­Δ«KΣ³{ύυΧλω―φM’ύ_RΟibΗ[SΚ;Q6?pN’·ΦW‡¦ίφνΫΎώϊkk•Ϊϊwί}·Ά`W—v«V­ ’\ @€’!P\Z7;QφpN˜*ˆ|ΫδΗdί­[7wαSTIΛ©«΅02ς'©Ϋ· ,ͺ€\:ΨLίΠ–L‹#e#Κ0ΕΜ«Ί*˜TΙχΥRŠuNλ³>KΗwœ5ΟIΚTD±}‹-Ά°ζΧτό{φΞ)η8k™J]ΌκE*r­ΈrvηdŸΒ;οΌS¨–Rl5Ÿ―άq³6/gλƒ’’G%-‹.·­'%+τΪV-φ7^Ωβώz}Zπ·šγ$Ϋ·:tˆTΩ0Qϊ»`Ύ½{Σ&Λ‘"x W…)±’so›4ibΜ»s:©.ˆk—TΫ5g’Ξ ”3ooSŠMΊo+—ƒΏ"σ:g—Τ<”νέYνρ»WO΅ό.ςΚΘ/ςB \9WΞ$Ÿλ€ε>RΖ°1J©J±Θ~ŠΧ$zmJ~Mγ‡VCξSκ·Vή­iΘ4όuUΞ΅τΛ,³ŒqΜξ}o8'Εύμ·ί~±³f½%5ξ(qΧ₯¦ω ™bRyŒέ +₯Ψ $@€ ΠΠ L™2₯ΠΈqcνG\ԝVeχ΅Ry“WΑί8 ΄ZδUw’§ΔeR\ ΤβAv& ζΣΏε–[ZBWΟINAλΡ£‡5oώ|†]GUn²}¨Λφaι˜άeWtξωM9“kq•b?ϋμ3λB]ΫJ«β <¦₯†“π·έv[Ιuλ―sQvš9A£FN«ΧεΧ³“qΐΏώυ―ΒόQδλΕ™d—δ³kšΨ‘0₯ΌeΗΜržzυ+‹†mο [ΏSΞf!²yCά“‘κ5n @€ H*!—φΎ‚Ώ²Αš(·ν£ήΛF?QMrκj-ŒΜƒόIκΔφ-XiΉt° ˜Ύ‘₯]ιGŠbφ:λ¬cm‡Qη`‚y1έWK)vΑ‚Vε^™£0™$e*’‡R;u}ƒΘΛl§–ΊxUςψρΗ[`κςγΩ΅iΣ¦pζ™gΫ•-Οi>?RξΈ&‹σr6†IΘ£E—ΣΦ“”JϋͺΥώ&ξ³γωχϊΰo5ΗIaοβ(›ζ>σΜ3Ζ>Nε+iΒΖUq•be|δνΏΏρΖ΅ΩO²νšΖ3QηΚ™·7)Ε¦Ρ·•ΛΑ_‘yœ³KrΚφξ λ3όΟOπZ·Δ_/ήu­Ώ‹Όrς <¨„HΚ™τs΄άGΚ6F)U)ΩOρšDανΣψ@ήAՐϋ”ϊ­•φ»5-™†WOεώ>ϊθ£Ζ1{»vν .t“Mk‚γο^6Β‰rθ?―iΥ[’γ>)oάu©aγΑj~CζA†˜Tύm5ΙλFΞC… @€@Ωž}φYε|Έkγq2hνƒ–χέwŸr”²‚Φ%ί;Β5|ψpkψ₯–ZJuνΪUλΗ9)V9‹4΄n₯Z6Μt=φ°ΊWΛΡ™xRƒXτ3gΞT|°ϊύχίKŽσΤSO-9¬³hG9;γ—Ύœ€Ž@9J―Κ.Ɗ&‹u«=; sΚYlΡ·έ›£θ¦φή{oεμϊkυxΖg(i“₯©GGWJΠΜ…qv˜7ζiΝ5ΧTΞβ:­{}³.#Iτ iΆ•,υΞD’ΊώϊλuΥ`΅“w¦³hKέ|σΝV&GG©ήxγ “sΙφŽΐY­½φΪΖπΞB uςΙ'«Φ­[«Γ;L=πΐΚY,gτ_ͺCύ_žέRί‰ηž{^•0žEν’’zα…J 'œS|J~JN”€€ @h€*!—6aΩζ!CLΞ‘φΞ©˜κΉηž υ'’–SGΚT žς.’"'!ƒŠŠVd“Ξ‰j̘1Ζ ‡~ΈΊι¦›ŒξYrp6V5Κ %Ÿ"οΥ™€e*ƒ Rwί}·.+υμ…ΧzχΊ‘—xΰJδ•6"Οr6y+)ΪI“&)g3Π’ΒΪeιωρς™Υy9/Αί$δQY’EΛ―»ORVH£«½]5ΗIϋοΏΏr6­Υ'μΨ:ŠϋF7ΟαΥW_υ.‹~=τΠ"»,YΨήϋ’Οε—_Ύ(»I·έ’ όΧ’Τ9S|qμ³Τ·U’C–ημ²0%m$‰ρG-Ώ‹β4YΔγμ樎9ζχ'ΚβςlΘΈ#μύyύ_VžέRή‰oΎω¦rNξ΅"•IΓ­ΆΪJΙd‚³Σͺ [„*Bή/Ώόg˜cΣ¦MΥφΫoο*ΨΚζλ­·žυύ$ρ9»~†E‹; @€ P&rεq“—M-[Ά4.f ΖwΙ%—(gWτ ΅φ>I9΅6eZΦ‚όIT[³Θ%l²Ν]vΩΕέάΛ[h'ξ΄όΪδς&L(ΚV2Ι€sςnQ>< ™£MEŽ$rξŸ~ϊI :T9'C{^Š~₯M½ώϊλEφεXH|Q6y·sςœVvRιMa₯џn%―‘ύό?ΝΰšΔRgUξSνwkVeQλρ—_~±>γ_ŸΤ¬Y3w‰)ξGyΔδΫΎZυ–₯qŸ JZߐy!ζ!¦zlŸδ±΄€@€ P›f̘Qpv4’•=Ϊ?ηΆΠ‚;»ˆiΓJœK/½tAάuΖ™΄/8’Ζ°έ»wΧ«gwυΥWΓ;“*υό–{γ(•Σ’²:‚·r“ˆ~§v²ζ©[·nηc^οuΧ]Wp–αΙΟ‚³›­6¬XξΎϋξΖ°^{rT Ξb˜’8Ώ‚Δοω ώ:‹= ΞιΕEα< g²ΝΦ9!Πσfόu”pŒαƒy‘{G©¬°ΦZkœΣ8 ΞΞ\υβM³Ž=φXc9ΑK½|ϊoJ χΠCΣFΞ.pG@γOΖ½~ηw\v:bη,€*̚5«(œXl½υΦΦ4;wξ\pv”.8Š|nxιSœEM…°:ρςβ(”kΣ-Η™ά΄ζΩQΠ+όϊλ―Φ?yf>όσΒγ?^pNΎ΄ΖΧ¦M›ΊςσtߜfŸf[ koΥꇝqΦΆ!ξΞΒΛΒμΩ³έ¦!}ͺ£τ_p”­αΌηΓ9©€ΰœ–YΧ¬d‹σrQVK•–,ΊΤΆž€¬°ΦϋΫsbs3υQ~ϋjŒ“dόδOΓ-sΣ¦M3f[ζ¦ύώύΧΞFΘΖp₯:„«ϊφν)jΘά»?ΏΑk)»Œύ&ιΆ+iΫΖ3Α<Λ½mž\β;ξΈγŒε~ςΙ'Ε‹Υ€Υ·U’Cήζμ’ž‡Šςξ¬Φψ½–ίEΦ Gd@₯δ@I>Χ~ŒIΚ}$έ°1J-―=’ς§υͺ!χ)υ[+->R7iΚ4$ύr£n»ΆnέΊh½κν·ίnτ/λnόρΗΘYJ£ή’χ ŒRΦ₯&ύ ™byŒά+μQv%Ε@€ @ ,ΞΞ›Ζ7gžHq;'‚γpvΕ²Ζ1|ψpcX4„™ϋοΏί~»νΆ Λ]>θ&‚<»―Ώώ:V|εzvNM³ζG&œέΞ­ΙΘβ›rͺM±8μCέ9Φ™2 ‰;έοψργMA I(ΕΆhΡ’pΫm·ή}£fΪuPŠA –n£62Φ—³+³±ΔαΫoΏ-8'Γ;§Ε…wv6ϊ—φ"BΆιΣ§…σ,l°^{KC)ΦK»ΏK,±DΑ9Qΐ+rΡo}sZ}Bšm%Ν>ΐ6ΑήΎ}ϋ‚sβwQ› ±wNΕ±>_2‘$“,:㜠o +γŠj˜ηŸΎΰœ(jM;κs%‚iyGέrΛ-‘ &έ%ύμΪ&v<¦Qή‰_}υ•΅~lcB™@rN$6†_wέu΅Ν*¬ίqN€Ρ†σ,Ο?|cšRvgGxΟ+Ώ€ @€@… TB.-YςΎ[LΏ={φ,ΘFj&sγ7Zγο’7S«—Ν€e*>ϊ¨‘½Θ[lfμΨ±Ζ°²ΩšΞ”²xυ­·ή2¦#ΟθAdœΧ%Ω‹.ΊΘ^β(G)ΆšΟŽa˜]Φζε$Ώa}P΅δQiΚ’KiλIΛ k½Ώ {VLξaοώj“d³Ξe–YΖΨ_ 0ΐ”εΒΝ7ίl 'Κ(•6aγ*Ω°QήΊ?‘ΏςΚ+…[o½5’’i—.]Š²ŸtΫ• ΨΖ3^›‰2'ΰΖ6'¦›fίVIy›³Kz*μέY­ρG­Ώ‹Όg_δ…@%δ@I?Χ~ΆIΚ}$έ°1ŠχΞΔ/²ŸΥ΄m|P ΉO)ίZ’Ϋ΄ή­iΛ4ώWS₯_m΅ΥVΖο Ω .h&Mšd][{Γ 7ƒοΣ¨·€Η}RψRΦ₯&ύ ™by46ά*;4r^d@€ ”EΐΩά~ΕW4ΊωœEώΫzΧΗ|½ϋΰ3!΄ͺ»wNυͺ»6]8'Η™œ”³“Ρ­η΄0k°•W^Ωκ^iGη΄;c”K.Ή€rvUŽ‘я88Ϋj=φ0ϊιίΏΏrN4Ί›š7ožzκ)εœBlς’N<ρD«»£dl [m‡•VZI9',*ητSε(ΰ*G‘S›d–λ@›α-σŸ(gA“6΄°qzτhεμΚntχ’”S{iVϋ7oς'αQMTήoΎω¦²Ν‰ˆ\ΖΩΰM­°Β Q’˜GAƚ'\B2“ [2=pΰ@5kΦ,cώΧXc %2±Λ/ΏΌθο„NPQ櫌‘ϋξΌσNί]ύKα+}–i^ΗΩΰT]qΕξœEύ•ΉKϋωΡ•"L6“τΌœ.~»jΚ£²*‹φ—ߝ€¬ώΖO>ϊu5ΗI2Om›Kp”χ•χ€Ι8J&§ͺΩ;μͺ5Χ\Sϋηlt‘vέuWuꩧ*GΉ3427ξ7i΄]ϊ¦λ¨s¦πqμ³ά·UŠCημ²4UΝρG­Ώ‹βϊθ£zvQnd!YΨQh΄)-ώόσΟQ’ͺΈωΘe)™Θ 3Yƒ°ΌΗqο½χŒή™•€ΒŒ(`›Œ( œ—MήU―^½\+£ΗAgm©Ά°YvΕ1ηD%|φή{okVΣ웃«fŸf[Ιj°ι¦›« ή}ǎλέϋoœ]@Υ:λ¬γ·ͺw휐£€Ÿ4y·WΛΘβYXβœNm\7m+8'N(QΊυ/άτβI£Λ³ηθœ\νa+ϊuN9 ]π+~dμΩ¬Y³’?ιΣmΟzQ‚ŽΕώϋοogH˜UW]U-ΏόςΊΰ]5Ϋ³1Q @€ Π@Ψd·Qεa¨Ξ=χ\λ7¬ώK/υ.΅ΏQ”bm²X[Y΅ ¦l™Wω“`«¦ *¬ZœkάoΡωσηk½Κβ6Ω0kJ|ΪΜ,mJ₯DΣz& ™Š(―˜Μ| D%σ!οΌσŽΥ“(5]rΙ%Ϊ?‘MTΒΨ6•M:mAzιK«a|~LεΙΪΌœ)Ÿž}5εQY•E{eώ&)+€Ώ v_νqa‡f̈(Ύ.X° Θ]6F:th‘½XΘ&ΖΞiœZ·]VϋΆJrΘβœ]–ζ‘ͺ9ώ¨υwQάη H›€M6U”δsδUKr)²Ÿψkσ"χ©ζ»5Λ2ΰ3«»·)°Κ3ΎΝ6Ϋθ‚Y7όqNΑV6.Ϊ5–Υͺ·,ϋ4Εg•δ7ddˆyΘc½ ,ρ₯ΨΑ € @ΰlB'›@η1(%»S›ώόώfϞ­F₯žyζχΤΘ 7άΠ=Νο'ξ΅M0&“εa»HΗIΟΆA&ι¦OŸ'Ί²όJΉl;ΞφμΩ3rό[m΅•uQDγšm·έ6RY”5³σΞ;»»ύ†ε+λu–8ξΆ]vΨa‡HQ‰’±ιΔ]aTvϊώϋορΚΞόQŒΤeΫX”Ό›όΘζO<ρ„uη:/¬©_{Ώ©Fίμ_«Ω'€ΥV²ά„΅{Q:4™υΧ_ίͺp*§λΨoΊ…¦΄J±—E/"¨–Τec Ω|#Ψ¦K‰WNŸ₯Μ I£Λ³υ(Ψ6шΘi²“'OVrΪ°ξOƍqŒMωۏm,YνφμΟΧ€ @hh*!—cΦ£G0/»|[ΪNkϋΆ¨΄œ:R‘Κπ”Wω“9Κχ§ψ “ˆŸΈf=φP3fΜ0“yΫFdΖ€rψε—_ŒΉ‘ΣMύ2£4d*"Χ²™ &¨k―½Φm'"Ϋi§άSa_ύu5sζL[ЊΉΩϊYňΏ6mΪDρΛOšΟ)£Yš—3εΡo_-yT–eΡώς{ΧIΛ ιo<ςρ~«=Nκή½»jΩ²₯6S"ΥΥ›ΜI›”αm €΅‰dΘRdίO?ύ΄’Σ€ύFΗΐs―Φό«Ώι7κœ€)|ϋ,χm•δ6ξLcΞ.KσPΥ4„wQœη ΏΘrε@I?ΧAf΅$χ‘²!ϋ Φpψ}^δ>Υz· ‘,Λ4ΒkP©½ν³Ο>JΦCιΜΎϋξ«³³³Ε[η)δ’Zυ–₯q_•δ7ddˆyΘcXFq_4Š'ό@€ @ΐF`βΔ‰Fg›@ΗθΏ“&MrOΫόδ“OΤΘ‘#Υ_|‘ͺqςg˜βΥl šΒΚαw·$&ώ~ψαχd3˜j]σΝ7E'kϊΣκΠ‘ƒ6τZό›ΛJΩΝ*¬^Ό Ι‰„Y3Q'³^•δjN_xα…ξΞωQӝΒθ…E¨΅ΦZΛ½•ΎB5uF&Œ7Ψ`“ΦNβ”SjΙH*‚ ~ύϊ©Ώύﱊ–TίΜT΅ϊ„4ΫJ–ϋΫIR7&uq³ΰ-ξa¦ ͺaiˆ»,ό““=δOϊ·ήzΛέA^Nϊ3fŒυiŠωηŸW>ϊ¨{­η'ιώΟK7ψ›Φ³υ8eΚλβΡN:‹Tυϋ¨mΉI“&ΖΌ kDƒ @€Κ&P-Ή΄—1‘E¬±Ζή­υWNΗΨzλ­Υ /Ό υg[πε“}TRNν₯YΝί<ʟ„GX=xΜͺ!—–ω›‘ΣΰL'NΨΒeΙΝ¦ΫΊukχ€/ΏiΘT6Ωd%σ8’lfDώόΖoΈβW”zwάqGwΣ4Yμ6ΏΞ}κΤ©κχίΧ9ΉŠR]ΊtΡΊι,W[m5%ς’Jš4ŸS9Βκ!Ιy9SύφΥ’GeYν/Ώw΄¬ώΖ#ύ7‰q’τ«²₯ΜeιΜ‹/ΎX΄™…œ «3²8ύΐΤ9eήN”εοΈγ΅ρΖε5Ά[”‰€EΤ9@°’n³ά·U’C^ζμš‡ͺΦψ£!Ό‹Jzπ ”+JϊΉ’ λ―ς&χ‘ς!ϋ Φ²ύ>/rŸ°Άκ•2l,λ2 ―\¦_YΓ,k‰Lζ€09Ή2fY«h:ΐFΦυιΣΗ>ŠC΅κM—vZγ>]^όvI~Cf]†(\ςGύ•zRl©δ@€ PGΐ6ΩRŠRμ+―Ό’ξΌσN%'ύυΧ_uιTλB>Πe§hΣμ"T‹{²˜)―aΚ΅"όˆΊ ΄)¨φΣ¦M3z]|ρΕλνΘnτθs°MΤΫςEQο2ŒU=Ο»‰ΊΓΊKκ ’Xe§^“…‘J( ωT?ή”œjΫΆm¬S!³¦+»Ϋ&?g͚iΡ–:υΤSΥΊλ«’μtί¬ΐjυ iΆ•ZνΚUj-7|°νDΉ—SH=τΠ:eVYψψα‡Ί;Ν‹²¬œ`Ί`Α‚(QΉJώ—g’ξΌt½ί΄Ÿέ¨οΔιΣ§{YΦώV«Π&φ_Ky_` @€ μ¨΄\:X¨ ‰Όpώ.=;ο7ŠRl’rj/_¦ί†*i|šκ!hε•Wͺž={ͺ•W^9θ”›{ΫΠΰwx2Q ΌύφΫλdDqΐŠμH”±δοΔOtλκΖo¬θF¨γƍ3fIΪιD] ۜŽΞ»,>?ayJr^. Γΰs%L?y“E'-+€Ώ‰ŠκϋIjœ$§»Ϊ”b―»ξΊz{υΥWλέ{7έΊuS₯¬[πΒ§ρ+§ϊ\vΩejύχ7Ξk¦ΡvΓXD‹'Š{–ϋΆ$9ΨX•;η%|ΪσPaοz›[CxΩʏ²H \9PΟua–δ>’7d?ΑJ>+rŸj½[³.Σk 0z‘η)lνl fRŠύφΫoέuI›oΎΉ10‡jΥ›?έ΄Η}ώΌ˜“ϊ†ΜΊ Qψδ!¦zŒcRlZψ… @Π°M6Δ™“]΅O:ι$ΥΏm:Υ΄”|š”bmε‹›'ΫΒ(‰λ?ωOά(Kφo›$“Iš(ώΔm qliωγπ_ΛΞMy5"θˆbl\²PQΚΥO”υ£Ζeς7oήΌ:'›’c»vνκόEΉXsΝ5£xKΜΟ]wέεN€ΫόσΟ?•L 4HέtΣMΚԏΙΖ·άr‹U0—fίμ/c΅ϊ„4ΫJCκόu™‡k ξΊλξŸδwζΜ™jȐ!κž{ξQ"δ- Ζbˆ°ZΩή‰ΤIχ^Ζ²ςμF}'†q’ L’6²A€ @Θ.“ΌCrG.m*aά…D6ώΝάLιyωNBNmΛƒΈ5Tω“”½Z2(‰»\#›α}φΩΜ―άΈ?wξ\υξ»ο“Ξ1„Ι ŒΕpπΛ”½`‡rˆ5j”Ίϊκ«#o’ζ…υ~eΘxΐU½ϋξ»Υξ»οξ9•υk“eŠ<+Ž‘“y+m²ψόdi^. οjΙ£ς&‹{ώ+-+ K/Jέ…ωΙ[VΫΈGΦζί6NΪrΛ-Υ*«¬’Ύώϋ’hG­da½œ|-fΒ„ n]δΡ±…ΡY55r7τ•rz;wV{ν΅W½ΤuωO«νκςβΩEπό—σ›εΎ-Iε0,'lVζ‘ͺ5ώ{Ύjε]TN ,’&P(ιηZΗ'©υ‰Ί΄ƒvΘ~‚DΏΟŠά§ZοΦ¬Λ4l-@ΦΧΙ<“Yj©₯Tοή½MΞ½|―ΨΜ#<’ΚQŠ­V½Iž³2ξ³ρσά’ό†Μ² Ρ㑇ϋΔRˆ•‰αξέ»+ω ,ΧΨN©\f™eʍΎ.όf›mVw­»°νˆ₯σ΄“IC9ΡΞτ7gΜΊ Mš4©»^ˆςO\c cK+n:yπΏάrΛEΚ¦‹§)r[[Z¦ψ*i/ΊΥ6ώ“₯mΒpY|ΗΨNeO’~₯ΎW_}uuα…Ί']Ϊ_²kΆ©ν€έ7'Α,ΝΆb{.Mubcb cKΛgnq7eH"―Λ8EN±όσΟU‡¬ΙΎσΞ;uξIχ’p–žέ¨οD›’±”) Ž’€ @ȁJΘ₯m₯•Sγ˜… ½Gέ€-)9΅1£1?Ε€UA―=φ˜:thcL.*QˆυΟQSž\š„,ΐ/Sφη§OŸ>jδΘ‘j›mΆρ[ΗΎž4i’«Τ4lΨ°ΨaulΚ’V˜|ΕηΤ©Sύ·5{₯yΉ4!ΫδΓ6Ή²)ΟΆ0Ά΄LρνΓΪr₯ϋ‡JΗ,άη­ΏΡ•Αo—Τ8Iζ d―ɈΜή3rR·ΞΘ©lr*SZFNό’“ιL²Ρ­œZ-ο Ω<\ήA2₯]FρSnΉMmΧoΤ9Sψ8φΆώΖΦO™°…±₯₯‹/IΊτ+egš³ΛCtFλ.]Ί(Qζ5) 0@]qΕnΉŒ‘Xϊφν«½φZ­-Z¨Ι“'ΧΉΙ½ΙΘͺ²ϋ˜M  kΫΝͺ!μ„δεΎ!Ձmθ=χάSU’¬ΊκͺuΨύΧu–½˜8qbΠΚzŸδ Ξ֌”θΈΦZk©#ύτΣΘJ±΅ χ6‚ς9di^Ξ—­Δ/σ&‹NZVHΏI&9N’>XπΚϋ/hδ)σό²‘’, Χ™C=Tg]vYi»iΑΜ[ί–§J§›Χy¨Έκ»(.'όC Iεʁ’~ƒlς.χ‘ς ϋ ΦjόϋZ•ϋD!‘W™Ζœ9sΤΣO?₯ˆeϋyδ‘G2₯›Χq_ZߐY’!šcςhΚ»ΙΎ‘Ι{@€ D% 'ΈšL”Ψd²j€IΪ(V\qE5tθP΅γŽ;jb%Π·ί~« Υ2,+―ΌrΤ¨"ω ξv tέuΧ)ω˜Žkž}φYe[Δ°ώϋΧ‹R^m Ώ&EŸz‘όχFv ;v¬ΞΙ΅Ϋ|σ͍n Ω‘!Ձ­ŸˆΣΦ’ΆYhhRz•C#FŒˆ•(ΟFυ)Β”<΅oίޚ²N)6νΎΩšα :¦ΩVR§Κt`Δ τ+»lšΜ»οΎkr ΅7->”€ώέΙ“ξςόμΪN2ωψγCλD69ϊθ£UϞ=΅ΊΎ.4R<@€ @™%`ϋή “ωF)”,ϊωπΓ£xuεΉΆ$£(Ε†εΉrκH‹α ωS X―"O…ž5Χ\S]yε•ξζQ―κK/U²Ιf^Μ…^¨ώόσOcv=φΨ"7Ϋ3^ ™rQ4"Ολέ»·zόρΗ•(„}φΩgκϊλ―W]»vUΩ—`}ςΙ'šγY‰’ΎνY9rd€?όsυύχίGς[ ž²2/—&Λ<Κ’“”ίΔoI“䀝™;wϋξ”1ی3ŠΌ,»μ²j=φ(²―‹,Άέ$Ωζ±oK’O%ΝΩεy*.“†ώ.ŠΛ ¨6Ϋ{/L¦βε-ΙηΪKΣϋ ΛcΦε>RΫχ¨Έλζγ{CΨL-Λ}lεφάς*Σxα…ΤοΏξ£ͺΏΟ<σLI놫•©{φlmΩYό²Λ.SΧ^{­Φ]g)qqΖ:'ΧY³fb±ίƒ,А…/Ώό²ίΊξϊ†nPr‚gsΗwOv“πΫn»m”hœŸ†TΦn½υVmWc“(ΔΆΪjκ›oΎΡ¦)')DQΦ~β‰'ΤΔ‰΅qδΙ²]»vΦμκ €έ7[3\AΗ4ΫJCκβT™n‚=Nψ _™˜yύχƒΦξύσΟ?―δRΫnκΊ€/Ψ2ϊ'δ’ξςόμΚ$€i\χΦ[o©£Ž:JWuvRφ|°ξή±όςΛ«‡~ΨoΕ5 @€ sεΚ₯£¦›n*:½Rώjϊτι:'Χ.ŠRlrjcfKt@ώT"8_°Αƒ«υΧ_ί΅iΣ¦:ηœsάΉŸ—ΊK‘ηύγ0ž&[η1}ϊτ±~“Λ’Θ^½zε4i™ΚW_}₯ž|ςΙ’|ˆΕk¬‘?όπzn"ΓκΨ±£ϋwφΩg»‡Šς₯IA^h+aD~"'ΟκΜC=€.Έΰ΅ΤRKιœλμDIY6k(&+σriςΞ£,:IY!ύMi­3Ιq’œφ*ύ›ΞΌψβ‹Κ΄9¦τΛrŠl­š€ΫnΦ8ζ±oΛΓ°όθζμς<Vή {-Ώ‹‚eεy P 9P’Οuiήε>Rd?ΑZw_«rŸ8ς(Σ0`@œ"–εW”oeΣΑ\V<• œηq_5Ώ!σ CΜC+ΥNQŠ­I⁠@ ˜€MθdθxΘL§ΔŠϋzλ­ηy3ώš]Œ6Ε][ΩΡDΎΕQ6½λ»Œaδ΄XΩ₯ϋώϋο·ξ/Θβ‡8ΐzJμ^{ν₯daIΠμ½χήF₯XY ! ³»νΆ[0X½{فχζ›ogηΏΩpΓ U”E_ώ0i_'Ή€‘ΤΑvΫmg¬V9©qȐ!ͺ[·nF?β “Ϊ]t‘֏œ*ύΪk―Υs[wέuJ±²“5Χ\£dq™ΙΘ)±ς,Φ‚ ›πΧ΅ω΄ϋζ$Ή§ΩVJd}ΣΪuΧ]Υ Aƒ‚Φξ½,X–E‹wί}·ρti]ΐ{ο½Wg]gηοσόΧuώ{Qώ/ΟΟξΪk―DTw/ύφ?ωOεW8sόοΕsΟ=΄ͺ»ίb‹-”nρF. @€ ά°Ιn£Θ₯£XΎ3DΑΝ&§’M>*;Λ‚―0“΄œ:,?qέ‘?Ε%Vμγ7g)J±"³0}λ‹δ 'œ δ›7‹F6…ΨΫoΏέš½=χάS»9i2•ωσ畐eαoP)6X(9αχ–[nQ;ν΄SΠΙ½_b‰%΄φq-…‹œ(¬3²‘ΫYgε27ΙAd1§ΘΪ’ΙΌ\šάσ&‹NRVHSZΛLrœ΄Κ*«¨­ΆΪJ»¦Μg›ζϋd!t-›€ΫnYζ­oΛ"ΓΈy2M%Ϋw“—NΉλ{Όx’ψ­εwQόH•&P 9P’Οu°όy—ϋHyύk5ή}­Ι}β•ώ}ηM¦!Ον«―Ύj,ͺ¬‹+—›7ož™žΙ<ςΘ#™QŠΝ󸯚ߐy!ζ!¦g }£Έπ@€  Q&jάΈqΠΪ½· tΌΛ-·œwYτkΪνΪσ(“mτn‹~uŠ^AOΆRςαZ sξΉη†*»ŠςΗ–[n©>ψΰ%)A#v²kyηΝ­§Υ-½τκΌσΞ wο{φμi=O”wM‹+$a·σΞ;«Ι“'kγΛΣO?έθ–UΫ©•ΞsC©QZ•φ¬3²pπΐTί}χΞΩ΅“TΗsŒemݟœ4ΆΆ'ΚάΫoΏ½qGύ?ώψCΙb¬O?ύ4m.οM}΄­0iχΝΆΌUΪ-ΝΆPϊ€8ufZ°'Ώ_y–uCx~xΰWH-»δ…™©S§ΊQΘ‰ο&#ώΕΞIχy~v;ξ8£rςŸώ©Δ]&tFήΧ^{­ΞΙ΅“ρ € @¨-εΚ₯£ΠXΈp‘»‘Ϋˆ#΄ήgϞ­d!ΎmΧ|‘ΝFωΦMCN­-T‰–ΘŸJg &§}^uΥUF2qςΙ'+i§I™³:t¨φOζlδδ@‘εΚβ³0…ΨE]Τ=ρV—€e*rJ―IquΒ„ nyuωτΫ-ΎψβώΫzΧ›nΊi½ϋRoD–)lLζΞ;οT‡rˆ{r­ίΟ”)S\%j™χiˆ&+σri²Ο›,:IY!ύMi-3Ιq’δπ°ΓΣfτ§Ÿ~RΊ±Z‹-Œh#Κ‘em7‹ˆςΦ·e‘‘-OΊο˜<ΟCΩΚͺs«εw‘ΌΨA λ*!JςΉςΜ»άGΚƒμ'X«εέgUξS^©μ‘σ&Σxβ‰'΄λv½RΚΪ!ΩΠ.Ξίƒ>¨dE“%ά_ύΥ䜨}ήΗ}Υϊ†Μƒ 1y¬TcF)ΆR$‰€ 4`² Ψf›m¦%πγ?ͺ™3gjέ.2ΰ ;n‰d¦!ΥΑ%—\bD*m­{χξͺώΚάΚ„Ί<벫š(£™Œn§gQze8“;v¬ϋ\]}υΥξI΅Ώ,θ’έόe·iΫNo¦8³j/‹ΛlF§xŸvίlΛo₯έl+ ©ˆZoΊ φ¨auώ–_~yuΚ)§θœκμdΑŒl2!'«{'ΗΎυΦ[κwήqOΐΣݏ:κ(w‡λ§žzͺ.œξB·E’ύ_žŸ]Ι’Y“yε•WάΕθoΎωfέ{Zϊ~Ϊ™ΖD+―Ό²»θΣ/φ€ @€@> ”+—ŽZjΩ„GδΔ"#}ώωη•ΘΌEn+ߊb/§“™L‡BO—τΒ¦!§φΔ/ς§JP,Ž£W―^*x’ˆί—Θ4ξ½χ^ΏUΥ―eƒΓvΨAϋ·Ο>ϋ¨3Ο₯/jΥͺ•»H3NZ΅δ7+σri2Ν›,:iY‘νΉͺτV^ϊ›(ν5©q’δEή?aγ žγϊχ‡ΝΣu’m7‹\ςΦ·e‘‘-OΊ9»<ΟCΩΚͺs«εw‘ΌΨA λ*!JϊΉφ3Ν»άGΚ6kθkόυυ:‹rŸ¨y/Ε_ήd"2™e—]VνΊλ&g«ύΎϋξkt—β堜,˜Όϋβ~FυŸ‡oϊ<δ±bmάΦb @€ P6gΧ£‚3HΥώ½ψβ‹Φψ'Nœ¨ ηΕη| >ψΰΒc=Vp>ψ ΞΞθgˆ5ŒΦΩ՚Ά8vλΦM—#Θ)όφΫo‘αΛρΰ|ΰjΣφς_ξoοή½C³7gΜ‚³8"R>œIB“&M"ω•Ό‡Υύξ»οnŒλ½χή Ν»xΨf›mŒq<τΠCΖ8liKή₯¬M›6-΄lΩ²ΰμ¬^Οl`LΧ9Ι΄ΘΏΝ"Ν:8φΨcεp”ƒΩ.5œ­ΎΌφξμΖ_X{ν΅ Ž`%R{sōω=ztAže/ξJ^sΝ5Ζ΄Kup„[ΦόJ?Χ8Jήn›6•ΏOŸ>EQ¦Ρ7۞Λjχ iΆ•΄ϊη΄c[ λΗΕPcXy·…™UW]Υ~πΰΑaΑc»;›:ΪΆmkLΣτlΔ΅w”ϋyKͺKγΩ­δ;Ρ9y₯ ο0φςžv&BύI<ΞFΖz©DΏc—†…ŒΓ€ @ˆD Ή΄—@ΨχG9ξύϊυσ’ ύMRNݐεOR•ψ΄}ηΫδΆohgCJm;q6‡²~6oήΌ0mΪ4mΨR,GŽiM―œgΒΦΩΠ°ΰlŠšEk/ΎJΙ” CΛξ(šœΥ Ξ†£ιƒΞ8γŒΒn»νf•g8'ΘΎύφΫ’²:'ΊΣsο‹ό{ς ―±ΖΖ°—RmςŒ4Ÿ―όεόfa^ K•G₯%‹.΅­'-+¬εώ¦Τη¦Τ~$JΈ8γ$/ΞfΞ‘ϋ>gcg/XUΓΖU}ϋφ­jϊy’mW³gΒζ—$|Π8§φλ5ΚΌhZ}[%9δiΞ.y¨JΌ;mΟ‰mό^Λο’ΰ³Θ=ς@ r €Ÿkk’rI3lŒελεέϋΝΛΪ#ΙoZοΫψ rŸRΏ΅β#u“¦LCjƍg]kηlž5ͺ"}τ‘qό+ίRΫn»mQ±HΊήχI9K]—*aƒ¦Zߐy!&Η ϋ€ξνGΕ8O€ @QΘi’—^z©Φ«œ0κ|iέΔQRqO>|ΈΦΟό‘…XχOλΑ±lΤ¨‘’%ƒfΦ¬YA«zχΞGΆ{2d=ΛήΘΙ”Νš5Σ9UΜNNΕάύΥk―½V±8½ˆdwυoΌΡ»5ώ:JΚς+g@ ›q>T"νξ.q8§Φz·₯“„Ϋκ«―nMFΚ:cΖ ΧO΅OmHu ;ΤΛ)¬γǏ7ςή_ύ΅Ρέο υxλ­·ϊ­κ]ΛnψΞDͺΊγŽ;κΩΗΉΩh£”3'Hζό:‹T‹-Œ§νΚNήA“fίΜKχiΆ•†ΤD©KέΣQΒΩόΘn“/Όπ‚{"΅μκ_ γ(󫁣NͺΛϋ³+§ΊžtIκζ›o6²yOΛ φaΖ™\TGqD˜7ά!@€ œ(G.Vd9uΚY`o<2,ΌœΜxΜ1Η„ysέΣ–SGΚdˆ'δO!€Κp–o[GΤ•mθ’qbΥE]€nΏύvs&νδ$‡~XE‘%%SP§œrŠzύυΧ•£jδ6vμX%qΜ 7ά Βζ$βΔ'§=8 Ϋ”£œ«œE΅q‚Ί~eξ«kΧΦrƎ4'²0/—&ͺΌΙ’“–ίDoIŽ“όΉ’S»_}υUΏ•φΊ]»vn?§u¬AΛ$Ϋnρε­oΛ"CSžtc΅ΌΟC™Κj²―εw‘©ΜΨC Λ*!JϊΉž΅ χ‘r ϋ •7΅(χ±QΚ‹LγΡG΅Κ…:θ [1­n›nΊ©jΣ¦š4i’Φί;οΌ£~%ί5išZχUλ22Δ<δ±ν»Q%"!@€ tιE΅jΥJ B”bÌ(‘,½τaފάε#Y”έΆήzλ"7±₯XgΗ"­›Xώϋίv6ι<ΨyuώK±%Y\Ρ³gΟR‚ΓtξάY9§ψΉΒ(£'Ÿƒ(Π>ϋμ³J&Œ*aœ]ΘΚRB¬DΒβΨc=ΒΌ$κήPκ`₯•VRΞ)ͺ}ϋφeσ•ED—ύτΣ1¦γ΅qγΖκŠ+P/½τ’jΪ΄i€L$)SΉ”l"ZΙE‡2_βœ*©¬q<9§Σͺ'žx"ς<·΄£ϋοΏ_ΙBπ†h²2/—&ϋΌΙ’“”ίDo™IŽ“όΉΪgŸ}Τ’K.ι·^Kί[­Ήm‚)[&ΩvS.ͺ1ωΌυmΖ‚dΜΑτε}*.ζZ}Εε€d@₯δ@I>ΧΒ­Vδ>Rd?B‘ς¦Vδ>QΙδA¦!γ“YnΉε”(ι—jdŒ΅χή{ƒΛ&ρ’”›“χq_΅Ύ!σ CΜC+ΡΖQŠ­Eβ€ @p• zυκ₯%1bΔˆΠ“Όdβμ_ϊW,₯™άyλ­·Τ‰'ž¨$ΌΙόρu'~ύ˜veaFR§Š‰b―μL}ηw‹ƒωΆέχξέ[½ώϋ±4ΘΞμ―ΌςŠ{r―-~››p;ν΄Σά“ςl tlq$εΆΣN;©£>:©δ"₯ΣPκ@žΧΟ?\zθ‘‘Έθ<Ιi³ο½χ^€έυ—]vYWyΆ{χξΊ¨Œv›l²‰’•ε­³ώϊλ‹!›œyζ™EξiυΝEIΘ"νΆPϊ€°κ4M°‡…‹βΎρΖ«Ρ£G»§™W"‰CvΦϋδ“O"½w“κςώμΚ’X9•EvD-ΥτθΡΓ=1@�€ @€@νhΤ¨‘*G.FζK/uΏ!Γόωέε„X‘³·ήz~kλuδΤΦ FtDώT ήΦ^{mw.ΔTN ΕKY4—E#§ΙpΐjΨ°aξ©Άqε2IΙT„(΅‹nœgΨΔ\6ηΌηž{LΞeΫο»οΎκƒ>P[nΉe€ΈDV6T•pΏώ»1Œτ­΅l²2/—&γ<Ι’“–ίDo™I“ό9’y9==Μ”3wVέ“l»Ye§Ύ-« ƒω2Ως>,gΨ}-Ώ‹Βʎ;²F Rr €ŸλZ‘ϋH{@φS§"οrŸR¨dY¦1rδH5jΤ(c±D‘u‰%–0ΊGqςیM)ΧnyχUσ22Δ<δ±ά6[ΫRΜrι€ @ c=Vλ_cˆςZ˜9ꨣ\₯’νΆΫΞκU”-eρΔΗ\wB¬(8šŒ,@’SufθΠ‘:k%&mΫΆΥΊUΓR&N8αυν·ί*™@…ί8F”QeΗ[Ω ^”‹Kύθή~ϋνΥΨ±cݝΪEΥHώ…™€σΝ7ΗRnŽšF5όΙnθ²Ηtp5 ‹³‘ԁ΄―GyΔUlίyηΓ°ΤΉ‹ ε±ΗSοΎϋn,%rΩύύ΅Χ^SrbBσζΝλβΣ]Θ‚˜“N:ΙM#‹/+v;μ°ƒ5+χέwŸZΈpa‘Ÿ4ϊζ’L$h‘v[i(}@‚UZ”T³fΝάň’Θ*Οzœχ™œnάqΗΉγ–[o½5Ξτ^Ψ€ϊΏΌ?»"˜–ΣweχΛUW]ΥΓϊ+‹;e³‘W_}U΅lΩ2Τ? @€ ό(W.m# rOQh™Zk­eσͺD¦ ~ρΕJNͺŠc²"§Ž“g_δO:*•³“Ή›Cζa PΉKŒI`ε[~›mΆq7F”9ƒI“&Ή§šΚf‡₯š€d*’Ώ 7άPΙ†―W]u•Ze•UbgYN/™Ζ /Όΰφ ±#ˆ sηΞξœœN-ύaϋφν•Μy¦U«VJΈί~ϋνξΤ^{νε:MŸ>έσRτ&C/ C‹¬ΜΛ₯‰.O²θ€e…τ7ΡZfRγ€`n;μ° U½ϋuΦYGuκΤ©ž]CΉI²νf•ižϊΆ¬2Œš―ΌΟCE-§η―–ίE^ω…@^TJ”δs]+ri#Θ~ͺχ€δEξSIY•i„)€tΠAecq«MΞ'Ž|ωε—e§S‰ς>ξ«ζ7ddˆyΘc9νtg—ΘlnYN© @€ Qh%Γ 9묳TίΎ}ƒΦΖ{9L”^nj£Ύωζ΅ΤRK)ΩaέuΧU={φT­[·6†κ0gΞΥ’E 5{φμ’ ’Dλ- (rLΐB†ι²γ”(τΚ©―?ό³š:uͺš2eŠ»˜Aς-m΄‘+lΪqΗ+ΒΔ_΄ωση»;Œ‹RΙgŸ}¦~ωε7ώω§›Ά€/‹+$m9mΕWτΟέυoΏύζ–O8˟΄%—\ύΫ`ƒ U’φΰ5€:7nœ«ό$»#Κ©₯Rb€]ɟ,"Ϊe—]Tǎ=<%Ξ›7ΟU–’έώψαχΉ’]±Ϊ΄i£DΨ&‚©τx“θ›υ)§c›v[iH}@:5ό©JΡGΉγYμ8aΒχ„yYΈ`ΑW-Š΄²€Pϊ"YΨ, e’¬&‰ώ/οΟ(μΛΈpπΰΑκΓ?tί2.™;w[/2.”Σ[φΫo?΅Ε[(YŒ… @€JΙ₯mΤDf+›χˆάVΎϊι'WΩM6W9­œ)rμΈ&ΛrκΈe©„ΌΓV‚qD#„LEr"r Q8ω‘l²&2ε3f¨™3gΊσ5Λ-·œ+?’S]D Kϊ£Jœ2‚ή—δYd'’¬oκ—d³P™)c»vνtN5k—…yΉ4αζI†¬ώ&ΌuVkœ€KYζnD^oRξ— .Έΰ]Πg—TΫΝ*Ψ<υmYe5_ m _Λο’¨uŽ?€M r j>ΧΘ}Š[KC{oΐΖF™†Nϊny|~“ϊ†”Ά›ubς·•£—ώ!@€¬όquπΑω₯6Y($;„gΕΘ)‘ϋΫߊ²#»~Λi©r"-€ @€ @€²M Orι IδΤA"άCΥ& ΑΙΖ§’ά4²Ω˜lŽš₯ωΌ`Ή‡ `Rμ—>μϋοΏw7v† @΅K Or δ>΅Ϋ)Y:i€Γ=ο©ς ™χ4ηΏ‘Ω @€ Δ'pΐΈ§ΉCΚ §O>ωdΠ:ΥϋΫoΏ]›ώω矏B¬– –€ @€ @€²G Orι =δΤA"άC6Ώύφ›šΎ!γ3ΛS”bσT[δ€ δ„ΐΑ¬ΦYg’άΚΞӟ}φY‘}ύϊυΣ&+ [l1­–€ @€ @€ MyKΙ!§α#°Χ^{½,X°@qΖJ_uFμwΩeυώϋολœ]»γŽ;Ξθ† 4 όϊλ―J3_}υΥΪl΄hΡBxΰZ7,!@¨=y!χ©½vG‰Κ#€L£<~„ŽG€oΘxΌςκ{‘‚cςšyς @€ ]―ΌςŠΪm·έŠ2xΐ¨'žx’Θ>I‹οΎϋΞUΪ•Ε~Σ±cGυρΗ£λ‡Β5 @€ @€ œΘ²\:ˆ9uχ€@r:’(~͜9Σθ½Y³fjΏύφS:tPΝ›7WcƌQ#FŒPŸ~ϊ©š3gŽ1œœ@ϋΓ?¨E]Τθ@I8ρΔΥoΌαφ]S§N5ž+yϊΗ?ώ‘Ήζš$³GZ€ €L Λr δ>)7’Ο$d™¬–šΚߐ5U‘ ƒRl$Lx‚ @(…@―^½ΤΓ?\/θ"‹,βNΌwκΤ©ž}’7Ί|5nάX}τΡGj“M6I2+€@€ @€ @$ “fA.,’.ŸΘ©ƒ”Έ‡tΞ?όͺ(~έvΫmκδ“OΦ%‰ TμΏώκ駟M»I“&jόψρj₯•V υ‹@€j‹€NΎ’9._Θ}j«νQš Σ(‘’ΰ2§ZςΥ¨– CY @€²Eΰζ›oV­Z΅ͺ—©B‘ .ΎψβzvIތ=Z 8°(Ι³Ο>…Ψ"*X@€ @€ @ȁ,Κ₯ƒ‘S‰pΔ!pυΥWW\yUNXD!6N-ΰȁN8…Ψ,Uy $H ‹r δ> 6’ΚdΉ«²šΜ0ߐ΅S­(ΕΦN]R@€ 9+¬°‚κίΏΏ’έΧόζ₯—^R~ψ‘ί*±λK.ΉD-\Έ°^zr:lŸ>}κΩq@€ @€ @ω#EΉt"rκ ξ!ΈδTΧΣN;-n°"Λ,³Œ’©Χ\sM‘€ς@ ]»vΜυη‘’Θ# @ J²(BξS₯Κ&ښ!€L£fͺ2—α2—ΥfΜ4J±F48@€ T‚ΐΞ;ο¬Ξ9眒¨.Ίθ’"»j[|ϊι§κ©§žͺ—Μ²Λ.« €_|ρzφά@€ @€ @€@> dI.$ˆœ:H„{@ Tr"—_~©N9εΥ΄iΣXшγ?^;Vwήy±Ββ€@ΘΖάGy€>|ΈZnΉε²%ς@€@J²$Bξ“R# Ωά@¦‘»*Λ}†ω†Μ}j °HΑ1Z,!@€ P!ύυ—ϊυΧ_‹bkΩ²e‘]5-ζΜ™£ώψγzI,±ΔL’Υ#Β  @€ @€ όȊ\:H9uχ€@%̞=[=χάs’λΔ‰•όύψγjΦ¬YJNNjήΌΉϋ»ήzλ©ξέ»«.]Ί¨ΖW"iβ€ P5Χ^{­zσΝ7ΥάΉs•,`nΫΆ­jίΎ½ϋ·Ε[¨N:U-m"† @ _²"Bξ“―vCn³A™F6κ‘rΑ7d-ΤbΌ2 Ύ!@€ @€ @€ @€ @€ @€ @€R Π(…4I€ @€ @€ @€ @€ @€ @€ @±  ž!@€ @€ @€ @€ @€ @€ @€ €RlΤI€ @€ @€ @€ @€ @€ @€ @ ”bcαΒ3 @€ @€ @€ @€ @€ @€ @€@PŠMƒ:iB€ @€ @€ @€ @€ @€ @€ Δ"€Rl,\x† @€ @€ @€ @€ @€ @€ @HƒJ±iP'M@€ @€ @€ @€ @€ @€ @€ XPŠ… Ο€ @€ @€ @€ @€ @€ @€ @i@)6 κ€ @€ @€ @€ @€ @€ @€ @€ ‹J±±pα€ @€ @€ @€ @€ @€ @€ @ (Ε¦A4!@€ @€ @€ @€ @€ @€ @€bX4–ob€ @€ @€ @€ @ΰΨ»0)Š΄γ/°ΐΒ.qΙ‚δ¬’ΔOF Ÿ9λ©x"f=†3sΐŒ€@$#9ηœ3συ[w37‘{vfv¦·gφ_Ο3Lwuuwυ―{—ν~«@@@@@΄ΘΝ͕ƍΛώύϋ₯eΛ–f΄ΨhU’D Ρ€ΨR₯J‰Οη3±ϋφν3ΣŊ‹Ά*Λ@@ FΈ‚b4h²Yύϊwή Ιc@@@@@h~ψ‘ 6,P€cǎrχέwζ™@@ŠšΐδΙ“eπΰΑΓ\Ήr`š Βτ]πτξ»οJωςεΓ³™G@@Hc½{χΚ9ηœrUͺT ™gR/ΠΉsg©S§ŽθΟd^^žδδδDέι‘C‡dλΦ­²gΟΡ X Ž-Y²€™ŽΊ" @@ fΈ‚bΓVJ—.-½{χŽygD@@@@@`Ϊ΄i!5kΦδYsˆ3 € PΤτ₯Έΰ”<Λ4!ϊ"exκΥ«WΎ£”„―Γ< € € €€·vνΪQAžD@Κ6l(υκΥ“ƒJρβΕM€k΄[·Nζϟ/ΫΆm“¬¬,9μ°ΪR¦L™h«° @βˆ+(6ΞmS@@@@@@@@@@@@΄Π@ΨX‚aυΰφνΫ'γǏ—™3gši †νάΉ“dg›–'ŸJ#€xV€ XϞ*† € € € € € € € € € € € ΰuύϋΘτιΣeĈοdΥͺUf”ΨϊυλΛ±ΗkΕ–Ž¨ώdχξέ’ίΡ’Οη3eŠ+&%J”ύ&!€ Ž:κΊ&ν4’lΩ²ϊ? ΨtΌ¨3 € € € € € € € € € € € €@‘ hpλœ9sδ½χή7±:bl­Z΅€wοή€I3lx%Χ―_/Ώύφ»lΨ°>|Q`^`χξέ+;wξ4C@TͺT©ΐr&@Hg:ϊz‡€}ϋφ¦³‡DλOPl’r¬‡ € € € € € € € € € € € PdvνΪea?ύτSωωηŸdǎ’——'ΗwœτθΡ̄g‡³aΓ3ζ™;wnΤ‘ς4ΐvΫΆm&p(''‡ X;Lς@|ΈL˜0AΆnέjbO8αιΣ§Τ¨QΓρX³³³ε°Γ“ύϋχ;ΕκH±›7oU6++KjΧ-Υ«Wwά& @π’€Οη“νΫ·[ωd~§ΔaKQ@@άΰ9ΚμΨΈΗŽΝ‰R € €ψhτKπ € €Λ—/—Ο?άΔnΨ°AΚ–-+;v”σΟ?_Z΅j%ΉΉΉ !€ €€‹ΕΊˆΝ@Θd… ʐ!CL/X›6m²=Τ_~ω%$ΏdΙ’rΥUWΙ Aƒ€Z΅j!Λς›8p Όωζ›ΆΕ4h 3gΞ”R₯JΩ.·Λ;v¬tοήέn‘LŸ>]Z΄ha»Μ+υ°­\”ΜϋοΏ_|πΑ(%μi/fεΚ•“ΌΌ<Σ«Y»vν€K—.§O)]Ί΄ύJE χΈγŽ“-[ΆDιwή)>ϊhD>ω h@½^§―Ύϊͺ8pΐv…•+WΚΈqγΛ*W,·έv›\ύυ±Kί½4mΪΤnQ‘δq](μμ@2H@G:σΜ3SvDuκΤνΐ„δ=Ϊ½wNΉFυκΥ“+VDΒΰΑƒεž{ξ‰ΘO$γύχί—$²j`} jΥͺζΣ¦Mσ…>CΠ RjάΈFR{l@ΜΨΊu«Œ9R~ώωgΡχcu0!ΆώΆmΫ"ύξdζžuŽ @ΐλΕ½^Aκ‡ €ήΨΌy³άtΣMΌysyλ­·Μ ¬5ήΏΏ<όσ°aCΡ†ž;vΔΊͺ:tȌ̩#Ξ„ζϟ/Ο>ϋlΜΫ‚>Ÿ/b;ώνFېWκ­ŽvΛ’ΥΫάvίœ¨η|Α‚ςλ―Ώη~ύϊ‰ŽΤ-xΡδ!`'°{χny葇ΜΘ―/½τ’c@¬έΊϊΐρξ»ο ŒςΙ'eίΎ}vΕΘC@@ εηΡξ΅έ|Vλι΅»οOEž>sΙΔ€VK—.΅ύθ3//$ξ±½p¨ €d‚@΄vΔdύ  NDpϊ}‘Ο’•’ρϋJ;ƒΦηγǏ—^xΑt  Α±ώωg²ͺΙvάΈFvM6 €DΠΏ±¦L™b‚b΅Κ%JHέΊuεΤSO5οΈι³xύ*ΪGƒj΅ν SΫL’π±@” ›2Z6Œ €@ζ h`dϋφνM`dA^φΣ`XRƒcψᇀΐiP>€(μδ•z€Ϊaωςεrε•WJ³fΝdΦ¬Y©ήΫΟP}@¨#6ί{ο½²}ϋφ„rύϊυrϋν·KίΎ}E·IB@@wxNέΉ¨<#ˆΐR·ΈΗvKšύ € € €€–,YbFη*]ΊtΔχŒ3ΌPΕ΄ΓΌyσ€cǎqwNžΦ&•ηΪO“E5@Z@ίc3fŒ,Z΄Θ ζR¬X1ΡϜ9sdψπαςα‡ζϋωτΣOeυκΥΕ¦υ•@ε@Ό&ε΅ Q@Hί]N?ύtΡΰ³d% bνέ»·|ωε—&0 Ϋ՞΅ $/ΏόrA6Sΰu½RHŒXΈp‘9wγƍ“&MšΔΈΕψΐ…^hzN–Ηwί}'gu–|ϋν·ΙΪ$ΫA@@ž8ΐe΅gA‡Ξd!p]θμ@@ U`ίΎ}Άϋg4.[–Έ3Υχ¦›n’΅kΧΚΓ?χϊ¬:ύΤΩ²e@T@ ωσΟ?Ν :J¬ £οI\Ή2& ύ{4''Η 8R³f-)^œqνb‚£ €ωπ?j>@,F@Hνi²gϞ1ΔζζζJ₯J•"7d“££XθθŽϊΠ  ιυΧ_—ιΣ§t3^ί+υ(πΔΈ5kΦH·nέdρβΕ1A1DΎώϊkωμ³Ο’KEœ@IDATRh{Υ«W—Φ­[K£FD·δ—40φ7ήΘ―Λ@@@ <'ˆ―¨=#ˆ]†’Ιΰ;™šl @@@ XΰΙ'Ÿ”Y³fg1 €d΄€Ž«ο©Υ«WΟ|κΦ­+U«V5yšΛGƒbK”Θ²F˜Νh*@W)ΦUnv† €@ϊ 8p@Ξ;ο<Ω²eKΤƒΡΡ(40²}ϋφΌysΣ»•Ύ$;eΚ™4i’Όψβ‹’#€Ψ₯;wΚΕ_,?ύτSzΕ:xπ άrΛ-2zτh»έΈ–η•zΔzΐΗsŒm°‘φp6oή<Σσk~ΫZ±b…\zι₯2vμΨόŠfΔςSN9EvμΨq,zν“bΠ‘Ga~–Ο=χ\)S¦LH1ύ}’A―/½τ’8υς|Χ]wΙ€$+‹ΫŸVF@βΠ‘+V¬Χ:α…kΦ¬žΕΌGhτΘ‰ IhΦ¬™ΤͺUΛq[ϊŒgσζΝf$΄hοƒθ{Χ\sŒ7Ξq[,@@2I@ƒ`/»μ23B¬κowΪτ½΅ϊυλYοΓ–ˆwuΚ#€ €€ƒo…;ΐ €φƒ–‰'Ϊ/΄rΛ—/oΤώφ·ΏE”ρχ”₯Λ(ύϊυ“Ι“'G”ΣŒ_ύUΎψβ 9σΜ3m—ΗšωΓ?˜νœqΖ±’’r^©G,χκ«―ΚQGεXtΓ† 2sζL9r€<φΨcrθΠ!Ϋ²ΪφωηŸψΪnάc™ο½χžΗj”^ΥY΄h‘L:ΥΆ]Ίt1ΧZΩ²em—·kΧNτΣ‘CΉόςΛEkΓΣΊuλDu„k € €$W€ηρ{zαA§NLΗWωΥώƒ>0£Ω•ΣNΣ΄ƒ"’·ΈΗφΦω 6 € ωΪ!ΙυΧ_ŸωZD6ΐ"zβ3τ°oΏύvΜ‘ίαi‡^?ώψ£\pΑ’ν¬vI;8ΧϋΟ Ψ-&@Θ(νȈΌ2κ”r0 €"P}Ίγ²όΈqeκ}~ΆΙ\Ža25Ω € € €@²)6Ω’l@ xρΕeηΝΆGX¬X16l˜4iΔvΉ]f©R₯L`ε7ί|c·Ψτ>ΊiΣ&‰ΦΠb»bXζόωσElo½υΦ°%ξΞz₯ΙϋΜv›—]v™/nίοΝ‡~h;ͺ逸ž**T0ΫΫΊu«ΌπΒ f€f­ΗκΥ«IγƍE?^xaΤQ„m+CζͺU«lKΥ¨QC6lh»Μ.σ¬³Ξ’Ϋn»Νn‘9Χv tδγαΓ‡G,*S¦Œqχ/XΏ~½θοΐiΣ¦ΙάΉsΝω>ϊθ£₯mΫΆζsμ±ΗJι₯ύΕγϊž2eŠŒ5J΄Gε₯K—Šžc ΦΦt½Φ;μ0sώΟ=χ\iΩ²e\Ϋ/¬?“ ¬ϋ[Άl™ιέY3υη«nέΊζ[―ΩsΞ9G²²βΏ]tγη8ό˜˜G@(<ž$nŸ‰ΟΥpλž(UΟβ=ξC‡ΙθΡ£mοΡu[ΩΩΩ¦3²x·λ/Ο=6χΨώko@H_m3²kw#*Wœœ~ϊιŽ§Fλ(εΔOmρ'/΄“€’mAΫ΄νΪ. 80­mcΪ)τ„ Lϋ‘ΆO._Ύ<Π>oΰŽ;L;L`AgŸ}Ά”,Yδh{Ά_Ϛ5ˌΆΉgΟ9όπΓM{άW\!:u Zσ“sζΜ‘wί}W&Nœ(+V¬ ,lέΊ΅h›U·nέΧϋί"§Rα―{q£mRΟ—^ÚV\iΎνώΡva ττ'mχΤwœ’[χ©NϋχJώGaΪηί|σMΫ*iPlίΎ}m—g¦κ ή‡N'ϋΎίιχ£ξKήκΧ――“ŽIGΫΥχΒ“ΎKGσαλΟ§κΪOΆap™F@@@ ιΦ Π1§1cΖψ¬ >ΥͺU‹y] "€ €@z X jΎκΥ«ώώ›@§­‡φ  ”ηΈ]+ΰv»—^z©ν:VC°m~Ŋ}VP›νΆ43όοœΰγ³#ΧσJ=+θ°`Π AΆNzάVПΓZΞΩVC«γφ¬ΡbW΄–8pΐgbF½Ύ‚Ο‡€ν»δ’K|ֈDQ·ΌPΟύΥW_ν³;뼝Ά}V£rπf§υϊ __ηοΌσNΗutυ"¬οΉηžσεζζΪοί¦lι{πΑ}ϊsh5.:–΅^pܟΥΘn»ήŒ3Μ:VƒΏΟι8όυ°n} π-Y²Δq?‰,P'>‚Ώ­Qˆ}{χξk“={φτ5oή<βΣ»woΫνXAΫΆϋΦίwšτϊΌοΎϋς=GV£°Οoi»#›L+ψΫwI'Ωξ?Ψ!xΪ ΄6׌^;ρ$+ΘΩΧΏŸώόoΟiΪzρΒχΜ3Οψ¬—7bڍ?Η1U„B € 6C† ω»Δκp'mκNE#ΐs‚Θ+!]ŸDΙrήύŸΥΰ{½Ο.Hrλž(UΟ¬­m’έή|σ͎λιs «#¬‚°šgΑηΙ?Ν=φΪ7ύ&άcθRcε|χέw!ΏκΤ©“‚½°ΙL°ϋΘ άΘ”Γγ8πœ€Υ©kΘοhίϊ­ν<ΙNƒvܟξΣκdΣq—Vΰ₯γΊΪž§χrΑ©0ΫIRΩΆ`χ{ήόΗΞ;οψςςς"Ό‚Ÿ:΅9΅ZpΫσοWΫ6΅-οΊλσ9΅ΫωΛZΟ¦Κ_W«3UŸΥyͺγΆύλYΘϊΆlΩβ_-κw*ύuΗNΗθΏ_JFΫ€ΆCϋ=žo«£*[›Tί§Z4ΫΦχ°­O"™V0Έν>ΤηυΧ_{“VπΆγφ΄Ν>ZJυ5ζίwͺξϋ~?ͺεG}δί½γχyηgkwΤQG9λ5’μk?U†ŽΚˆKΐκ\ βχΙβΕ‹γΪ…@ ύώόσOŸ5p‰yΧμλ―ΏNβ@Šœ€>'°:cςισ«]»šwτγ}G<Ν~)λΝ„ € X7ΥfδΞΰ<΄Žk5ΞϊgγώΦ‘et„P»ŽΪO^]ϋυλ±ŠΥψ'V hD~ͺ2ΌRT_ψv­@Ιπ¬ΐΌŽ~β”΄·dύυΪk―uΌΎΒΧ΅ώ 퍢E‹fdΝπεασΪk£Fδ_ϊ—_~ω₯Σ*QσυΪ΄’σ=GΪkŽ°ͺ#νΖ’¬†RΡΡe­€όXŠΚθH΅zΝ\sΝ5’? ±€‘#Gš‡΅GξXΧΡQdoΊι&Ρ ­FΠ¨»qγη8jXˆ € P(<'ˆ½¨=#ˆEΖ­{"―=xώωηΝ(TNFϊμΒκ$ΐiqLωάcΗΔd q»%@@ΐ]ΏύοfτO§½j{‘ΆΩ„§_~ωE>ωδ“πl3―£bΎυΦ[’••e»ά)3Uν$…έΆ m&_|±lάΈΡιΠ“ž―m4ηŸΎθ}œuϋΦKξςΠC™2›6m2mP|πAΤut‘Ž2ά½{χ|Ϋό ΫΏ°Ϋ&ν έΊO΅Ϋ·—σ¬ MΗκΥͺUΛq™[טΧξϋA<ΌCŸͺ† € € U€ Ψ¨<,D@Ώ€Υγ°2β»oίΎ&p*bAŒuλΦ5‘ώ©W―^Œ[ω_±Η\²³³—ρί©W_}U4pΝ­δ•zΈqΌVΟĎ»YΊt©ν²έ»wΛ 'œ ί~ϋ­νςό2W―^-gœq†όυΧ_ŽEΧ]k—·nέκX&ΏV/«ςΖoδW,ξεV/ΠςρΗΗ΅žΏχή{q­“_a Υ Οx’Ύ 0pΰΐxV‰Z6ZcκW\a‚ξσ ̌Ίƒ>ϊθ£ςςΛ/ΗΌφž={DΟ«^3Ρ’Ύ”bυ œο ΡΆ‘/J_uΥUωΉZ#ŒˆΥ›ΈX=‡Fۜγ2kT`±z•w¬«?ǎ•c € €@‘ πœ >ώ’τŒ ?·ξ‰ΌφLΰ«―Ύ29ωθ οΙΈΧζΫIΨ9Ÿ{lg– € €@αX#lšΆ)ύΆKΪIΡΏύοEpyΛ-·„δΟhΗΪf<)Uν$…έΆ°hΡ"ΉόςΛm‹γρ‰·¬5ͺ°cΠ²έΆ΄ύξΧ_•‹.ΊHζΝkWΔ6ΟέRτΤ)ΆΏΪ&ΓmάΊO ίo:ΜO™2Ε±š 6΄]ζΦ5ζ΅ϋ~[ gbθρDυ@@@’ •‡… € ΰˆφ²λΡGν/ζ‰o ²½νΆΫ"κ’£„Fk ŽX‘€^©G#¦ΥkΤ¨!•*U²-«ΑŒ,žt΄ΛΙ“'‡gΗ5―#›jΐŸέ§ΪCφ\ N#šjΨzŽτ€άάά¨ϋΥΐΜ+VD-ΟBm<|ι₯—βY%P6Ω#ΧjOΜ‰€oΎωF† –ΘͺλhΥ%J”ˆΘΧ =χίΏθK½=zτ'žxBτe»Πm7`ζ† δξ»οNhν§žzJζΝ›ηΈΝ&£ώθ?ώ|Ηύh/γΧ_½c@«γŠa τχΏnΗ.₯ϊηΨnŸδ!€ €xC€ηρ‡’τŒ ?7ξ‰ΌφL@ŸhηHϊlΚ.ιH_‰ήƒ†o{μp‘ΨζΉΗŽΝ‰R € ) ν` ,HψcΧΖ₯{iέΊuΤΏοΉηΩΆm[ BΪfσΗζƒ':tθ ·ί~{pVΎΣ©l')μΆ…K/½4αΞDσ…‹RΰΑŒ²4r‘ή?hۘΆΙΕ›΄ΣΧ]»vΩVΨώ^h› ‡qγ>5|Ÿι0―χΦ?όπƒmU5h_Ϋιν’טΧξϋνΌž‡‘ΧΟυC@@ΘO€ Ψό„XŽ €F`φμَϊb§Χ’6\vΨaΥ5j”θΘn%―ΤÍγ­^½Ίγn–/_²LΟ^{ν΅Όΰ1XGE=z΄Όω曦·ζβΕνt]Έp‘όσŸ ^έLφΫoŽtΪ³²Ύ€ £`Θ1CΆoίnzX>逓"Ά£Ϊθ7Nψ£m2NI_ΪΦ­['ώφ&•QŸχΫ₯μμlΣΦάΦdW.El@@@ΐ λEο˜Σ˜1cτ όΐΗz°σΊD@τ°Β= Σγƍsύΰ¬|mλcΛκςΞ;οΨ–i€‰ΟjΔ ”Σ‰πΏs‚qΦ¬Y!eƒgΌRΰ:Ε2=hΠ [=ξiΣ¦Ε²‰ˆ2Ηwœγ6­Ζ―ς½zυ²-k5~ϊήx㍐²ώ+ψΟg½Ψi»^ωςε}7nτ5ίΟ<σŒmΩΣN;-€\πŒΥͺO―ΰσTƒ‹†L[/ZΪsηw†”ΣυΥγτo7όΫ FυYΑΊ!λYΑˆΎγ?ήq6Β―λΰX½υ:―υωδ“O‚‹›iλ…ί…^踞ΥP±N’ύυ—Οjτwά—ΎΛ”)γ;ωδ“}/Όπ‚oεΚ•1Wc€IQχi½$β³Ρ}ΦK!Ϋ\΄h‘Ο ŽΊΥˆ²ŽΞX/°Ψc\μ[ΌxqDyΝ°zzχΥ―_ίv=kWΫu¬žε}Vΰ°ν:ΦKΎχί?b=½Ξτœ:Ώχή{!λΈρs²Cf@@ £¬—Cώξ8σΜ33κψŠΒΑπœ ς,§λ3‚Θ#ω_Žή;8έ#\}υΥ+γ”[χDn<Π{('«£##’χfVgWŽετΎ*Ϊ½|Œ¬ΕΈΗζΫξYYΔ…B†ηΎϋίuκΤρ\©wμώ²FuτN© &πΕ_„όŽvϊ;0‘ό'Ÿ|2ͺΦ”)S|ϊ\ίnΫΪΆ0ώ|_ψ=vpΩόΆοv;‰[m vΏ'ƒ]όΣΪΤ²eKί€|χέwŸΟ 8υνΨ±#pNβiΤ•¬ΞJmΟ•ϊόcοή½νλΔΌyσ|U«VΊžΆωhΉΰduΌγλ₯‹γzV'ΔΑΕΝ΄[ώΊ3·Ϋ&΅Νοώ­ύΫ­ϋT5©]»Άm­uqR’5j΄ν>Τδβ‹/φ½ςΚ+Žmg΅F4φ]~ωε>ύ[4άΡ?ovϋτΎΣ.ΉuΉqίνχγG}dwψ!yVP±­‘ΥΩAHΉΰ™D‘D―}7 ƒiH\@Οχφ;½_’ψ^XΌ.πηŸϊτύLkΠίΧ_νυκR?@"¬Ž|3gΞτuΦYΎ]»ϊή}χ݈ηd+ε“a?ά–υW3 @π X‚b½Δ矍ψŽe€X‘SGňχσκ«―Fμ/֌ .Έ@:tθQάj04£/F,HQ†Wκ‘’Γ l6ΈGέ@ζ'¬ Ε@–Ž\9bĈΐ|πΔ%—\"ϊ±KνΫ·wyα…BςS5γ•z€κψt»N&ΊLdύιΧ_υO†|k0ž6‚ζ—΄ήn„πFΆ-ZΘχί±9}1υα‡–GyDΪΆm+'t’œpΒ &°bŊ¦Α9|TЈ$˜±yσfя]κΦ­›TͺTΙnQ O_šΥσΟ?<WΠ mθΝ/Υ¬YΣ±Θή½{—%²@ΟΕτιΣε«―ΎνT―»bέΆŽΊsΰΐωμ³ΟŽΆε4jqpyΩG{i·λEZ{COΡRΓΛj½'MšdW§žzΓΧρΟΫν[—©―Ύ-½τKŽAΙ ”vλη8Z=Y† € Px<'(˜}QxFΰ$δΦ=Qa?ΠηN#6鳏‘#GJεΚ•˜’–Ο=φ(ΉΗώŸS € ~zφΖoˆΆIθί5±$νh΅ λ>’έNβ•Ά 2:thάνF±ΈG+­TΫnΚ•+g:£ ίFλΦ­Γ³Bζ[Άliή†η―΅MΊuŸrpF;x2dˆ8Δκ!Ήyφ}žΒˆ*cAB € € €@š ›f'Œκ"€ PΉΉΉ’­ϋφν‹Ψ½HνΨ±C΄ŒW“6ϊjaψˆŠ―ΌςŠh/ΐΪPθFςJ=Ru¬6lpάtπhΒsζΜ±-·gΟ3z―νΒ L ˜΄KΪ²nΓ?bm΄9]_ƒc5ΰP??ώΈiψΦ†ναΆ_Ώ~&H6Ω½C/Z΄Θκ&OχKγJfPlƒ ςέ­6Β»™Τ½OŸ>ζ£ΑΦ:Ί8λgυκΥqWE{Oμ±Ηδ»ξŠkέO<1¦ς¨l«#d―[·NͺU«u;ϊR‚φξۘ6mš,Y²D/^,σηΟ—ύϋχG]Χn‘ώ¬Z΅Κn‘γ¨ΆΑ…[΅j%ϊ‰–άϊ9ŽV–!€ €žΟ nŸιΟβJφ=‘ξ»°Ÿ 8ΔjέκΥ«y˜ζ₯2qΝ=ΆYY*―3Ά €@Q8φΨcεΘ#L˜ Ώηρώ kG―wήy§<τΠCώ,ΗoνεΊλs\λ‚d·“x₯mατΣO/ΠΊ±ϊ…—«P‘BxVΘ|ωςεmƒb›5kR.|&ΏŽ«όε½βοΕΆIΏ‘;χ©ώm§λχΚ•+₯sηΞrν΅Χš r»γpσ+μϋ~»γO·< ΣνŒQ_@@@p‚bΓE˜G@[½Bƒν’Ž|ιε X ΘΤ`ΈAƒ…T_{RΎυΦ[eĈ!ω©šρJ=Ru|NΧGΙ’%E{\φ§M›6ω'CΎ—-[&οΌσNH^Ό3λΧ―Œ6{ήyη™‘€'NœΣf4HvΦ¬Yζσμ³ΟšνhΨwί}w Π6¦ E)δ4z§’£ΐΖ’κΦ­K±˜ΛδεεΕ\Ά0 jύΞ=χ\σΡύΟ›7O&L˜ γǏ7½ λ9‹%έ{ο½rΪi§ΕυbJ¬=¨λ ΝNIΟΉ]P¬Ž~«AΎœΝ7ί$όκ΄O ΎΦλΩ.Υ¨QΓ.;ξ<·~Žγ+ € €Έ&ΐs‚‚Qgϊ3‚όtRyO€ϋφβ3ΏΙΟ?,ο½χžœώωώ,ΧΎΉΗŽŸš{μψΝX@’& ­^ύυφ}χέ'Ϊh΄Άm³ύχΏ”QP“έNβ•Ά…ƍ»rΎβέ‰Sg½Ϊ֚H ߞWό½Ϊ6™κϋΤDΞ‘ΧΦΡφΗηŸ^΄½ρž{ž›Χ˜—οϋ#`<š‘GO ΥB@@ˆY xΜ%)ˆ €EZΐ.¨Λβ4r§Ή~—)Sƌ‘½MΪ}ͺV­\<ιΣ·έv›ΨjPš€Ή•ΌRdο₯KEƒZνR­Z΅Bή5ˆ:Uiχξ݁M—(QBή~ϋmiΨ°a /ž‰ε˗ː!C€uλΦ&3žuΚF)6Φ ΨΰQwφO~¬½GΗ³ΝT–m€‰\|ρΕςςΛ/ΛΜ™3E{%~σΝ7₯GQw«£Zλ‹Η±&ν;++Ά>„?όpΗΝjύΒ“ŽβzόρΗΛ©§žj^^Id4ΨπmΟG ΎΞ―ςΰνD›vλη8ZX† € PΈ<'(Έ¦>#ΘO&ΥχDΊ/>vΉύφΫeǎΑY…2Ν=vώμάcηoD @pO@Ϋtžzꩨ;ΌςΚ+₯~ύϊQΛΔ²0ν$^i[πjPf,η₯ eΌβοΕΆI7ξS rξ’΅ξΠ‘CE―»Ο† Lΐύ¨Q£DŸWδδδ8ξVG¬ήΉsgΔr7―1―ίχGΰx0Cžͺ„ € € —A±qqQ@’+Π‘CΗƒe$NνIxςδɎŸTχΘ›-?ώΈν1h£N²ƒlwdez₯NυK4δΘ‘Ž«† :`鸁8hΎΑI―+ œΤž³K—.Ό(ζιΉsηΚqΗ'“&MŠy§‚ξ”b}vΛ–-N›(’ωt­A²ΰϊΡz-σΟ?c6Š§A~Χ]ŽΫ­Y³fΘ2mPΦί§ΏόςKHΎΣŒŽΎ₯/°όνos*‘­Α9Φ@߈†eΈωsΆkf@@<"ΐs‚‚ŸˆL}FMƍ{"ώ½φLΐ_/ύΦ΅#.―%ξ±#ΟχΨ‘&δ € €@α Όϊκ«Q+ ‰^½:j™X¦’Δ+m π[RψH±^ρχš½›χ©…}μeΛ–•Š+Ϊ~4XΌE‹½{wyβ‰'δχί7NΩΥY;ͺώϊλ―#Ή}yωΎ?Η£zτΔP-@@@˜bϊ(¦MQ@2Y@G5tjdύβ‹/ τ"Ÿ6ΜN˜0!ε|ύϊυ“ηŸ>b΄ΘΏώϊK^|ρΕ”οίΏ―ΤΓ_Ÿd|>άq3}ϊτ Y¦A~λΧ―ΙΣ™vνΪI§N"ςγΙΠFΌπ€/ βέ»woxQΗωΘE]$S¦L1ΝŽσY_ΐf›6mςΩ‚ˆŽΚ›IImgϞm{HΥ«Wύ̎<ςHωρΗMθΊέπ4uκΤπ,ΗymΧd»k*|%UΨ)Υ«W/dΡΐeŊ!yΑ3ΪΨ¬AΨΗsŒU[©xρβrΓ 7 L‡ΏΠ  Βχ(lM$+¨ΪνŸγΰc`@@ΐ<'HΞyΘΔgΡdάΈ' ήa?hΨ°‘84ϊτΣOΛe—]&M›6 r¦ΉΗεγ[bzͺΖ €xYΰΓ?4ν\ΡκΈiΣ&Ήϊκ«EΫo ’RΡNβ•Ά»Ά•‚XyuέπγτŠΏΧΌάΎOυΪρ;ΥGο§.Έΰyϋν·m‹h‡ύϋχYVΧXaίχ‡„Ν؍¦VΔ³^6τ•@@@π¬A±ž=5T @o θΛNiƌ2jΤ(9ω䓝ŠDΝΧ ΕTφΌσgžyFΪ·o/α#Š΄a8x±L{₯±Τ5Ώ2γǏ—oΏύΦΆXΙ’%M0iπBm ³KυλΧ—ηž{ΞnQRς4ΈQ[υ³}ϋv=nά8;v¬6όšίιœ9sδσΟ?—σΞ;/|QΜσϊ2¬SŠ5`3ƒb[΅jeΛ’ ­οΎϋν2§ΜΪ΅kKΫΆmε·ί~‹(²hΡ’ˆΌh:Jp΄Ρ―όλ.[ΆΜ?ς­£λ(;ώ΄xρbΗ—P΄ήό±tμΨΡ_<δ{ίΎ}!σΡf’]gvιѢ崬°~ŽκC> € €€ϋ<'Hžy&=#ˆ¦βΦ=‘] γ™@σζΝEŸ;\wέuςΡGETkώύ¦€οΏ>bY’Λ=ΆχΨ‰^A¬‡ €^X»v­\{ν΅1UQ;‰Υφ• /Ό0¦ςN…’έNBΫ‚“΄;ωψG:ζ}jdmΌ—£ν–NA±8ž σKΕ}V–σ«΅v4‡{lάΈ1<ΛΣσ©0ττS9@@@΄(žφGΐ € €€+ϊ2Y·nέχuΣM7™@CΗ t΄ŒAƒ9,M~φΡG-—^ziς7η½R8«Q\G[½ε–["ςύ:Jl΅jΥό³ζ[ƒ_νόa—’ΌrεΚΙ©§ž*=φ˜ žΤF;ν][Gθ)Uͺ”γ>c \uΪ€»Žϊi—&Nœh—’·{χn8’™ζ3Ϊσl… lβηŸΆΝΟ/SGŸΆKU«V΅ΛvΜΣQ€cIϊr‰]ͺ[·χΒ­AΥNI;p ˆΥuœκbΧ‘€βͺ«]Šε:»νΆΫD_ήtι%°I/ό*Γ € €@‘πœ yμ™ςŒ ?·ξ‰ς«‡Ο΄³’Ρ£G‹ή‡>ϊθ£’&Ω₯‘#GšΈμ–%’Η=φΤΈΗNδκa@πΊΐUW]%ρXέpΓ βΤ^λ±:΅M„―k; m αr©n£=αιν•ϋΤȚy#GΫ:’]P¬W±dέχWͺTΙιπeΛ–-ŽΛtΑΑƒeΦ¬YQΛxya² ½|ŒΤ @@@ ύμίΘOγβ@@ wέu—γVgϞ-ηœsŽlΪ΄Ι±Lψ π;묳dλΦ­α‹R:ΠC‰>Δ/μδ•z$κ°gΟιΫ·―툜ώm8Π?ψvMhΙ’%²rεΚ@9§‰εΛ—‹ŽΪώ τύχί7#|κ(ŸΑŸN:‰Φ;#ˆGΐ­{"/<ΏΟΣ{Λ*UͺΨri'a=φ˜ν²D2ΉΗŽήχΨ‰\U¬ƒ €…) mN¬”)SΖ΄+U­ZΥΆŠ_}υ•h@m’)Ων$΄-$z&’³^Qφί·oŸ-’[χ©Ά;OƒΜ’%KΖUK·1·ξϋsrr€T©RΆ:Κ°Σh±+V¬‹.ΊΘv=·3}· έ>^φ‡ € € P΄Š-Z監E@ ΐoΌρ†”-[Φq;kΧ•sΟ=Wzυκ%Ο?ΌόςΛ/&ΐjοή½’#ꨏΪpμ±Ηʟώ鸝T/Έι¦›€aΓ†©ήMΎΫχJ=‚+ͺ½Fk0kψ瀓N’ƍ›σ―ΡFM-^ΌΈΌϊκ«’ίαI·‘Ϋ²Kͺϋ·ΝU_ΥkΛiτ”3Ο<3°Ι£>:0>‘Α³N TZV{ΞΦΰ[»ΤΎ}{»μΈς4πΥιε 8Vύωxΰδ›oΎ‘ώσŸrΚ)§Θε—_ξxμqUΐƒ…υΌκ(Ίvi͚5ζχΙ•W^)ϊςΕΆmΫ"Šιο½ήŽ8β3fLΔrΝΠCtdκxΣ“O>)>ψ μΨ±#dU6Ϊﱚ5kFŸ6jΤ(dώi{Γ† ώِo½ξ ’#HήΡ‰ΈuOδΕg:zΧ?ώρGN Šu‘Θq%‡άcsνΏ4‚Ÿ•ωσψF@δ άuΧ]RΉrε‚ŸΑΧςŠ+plΧ6§#<΄c―<}γ7ΚκΥ«ƒ³βšNf; m qΡΈpψH±EΩι₯Άžnέ§Ϊξ< 2Ϊ΅κ>Ÿ/βάΊΖάΌοoέΊuΔqj†^SΪ!³vHμOϊNŒΎοpꩧŠvhΰ…δtν»iθκ€ ϋχο—­[·šυZνˆBΏύοκί':­ο–}ΊL:U>ϊθ#ΗQ9“«£Εάwί}rύυΧ‡WΝΜk`μ―Ώώj>Ά20SΟΑwά!C† q<: zՏ&AιπΓ—]»v™sΛΛ:*O5·ο΄@˜4Hžxβ sMU¬XΡη7pΪ­«?αθ‹)vIC―_5θΩ³§ιqXƒautΰηž{ΞqδVέ–^/vIƒώ/»μ2yε•Wμ›λO_ΤιΦ­›ι `ήΌy2nάΈˆΰ_ΚΧ]wΤ«WΟ?kΎSύs²3f@@<+ΐs‚δ/>#HήΡ‰Ή·Ϋ^²ο‰ΌϊLΰκ«―6͝;7‚A;»ωζ›eψπαΛβΝΰ›{l½fŸ•Ε{Q@bΠΏgυSΠή9§nOΫ ΄ΝΜ.Υ©SGξΌσN³hΰΐςβ‹/Κ΄iΣ"Šjg°Ϊρ¨Ž›HJf;‰ξŸΆ…DΞBbλ„ΕfΊΏΣˆžzάΪΞ5jΤ(ΙΛΛ3:k§T:νfΫ]bg±pΧΚΞΞv¬@xη»ώ‚nόŒ»yί――Ώώ»πBΎ΅M_Ϋ­΅]VG^2eŠh`¬Ϋ)‘kίMC·=Ψ €@f 軀۷o7ύθϋa:ΈΒϊυλMπ«ΐζζζšwτ]@½w™?ΎθCŸ“λ=“ώ_]’D‰ΜBαh@@ @Pl€‚ @ˆU@GzΥψ~ψα˜VΡϊ‰–΄1ABΌώϊλъ%uYίΎ}M@Ψ?όΤνΖ»1―Τ#ήz;•Χ@γh£Ÿθz:Β¦66>σΜ3Ά›Ρ‘?50–€#€6Μ—Χk©kΧ’½Δ…'•SGŒ'i€a“&MβYΕ±¬Ž†«Η§ββMmΫΆ5A™ρηυςz>t”Wm<Μ/ιυ‘ŸX“^Ϊ[zΌI_πξ^'L˜Σ&κΧ―/Ϊs{xjם½Υ.uόψρ’Ώ rrrLo…v£%‡oOηœu™σΎυΦ[Ž―;wξ”/ΏόR‹FMڈ«AΕαɍŸγπ}2 € ΰMž$οΌdΪ3‚`7ο‰ΌψL@GΨΡ—ϊτιΜ˜ώβ‹/dĈ«W―@^’άcsmχ¬,Ρλ‰υ@@ p–/_n:NqΪϋc=θœS;Φ6·O<ΡΆψΧ_-oΏύΆθ½[<)Ων$ΊoΪβ9Ι/›ΙώΪ9nι₯mΫΕ4xβε—_€jΰλζ}j`ηi4­Γa₯Ν.ΉuΉuίοχΑίflwΜ .ύ„' <Վ RΉφ΅Nn¦ϊψΩ> €@ζ θ»Sz_€]Lš4ΙLλ­$«QθGίiκΤ©“ CG‡Υ7>όπC©Z΅ͺθΘθ'œp‚4hΠ@’uφ‘Ή‚ €™/P<σ‘#D@T<τΠC2tθPΣ“jA·I'ΙΗ,­[·.θ¦β^駟φDo`^©Gά€A+h―j<ςˆ|ϊι§1]O=υ”\pΑA[ˆR{=ΥFόζΝ›G¬ά‘CSŸˆ dθ΅ωΒ /$°¦ύ*:2νΘ‘#M―±φ%μs5x<Ώ€cϋ5½Ÿ«υε_ `Mfλ@_φˆΦCΣώτzŽ7icϋΏώυ/3ϊpψΊ:ͺŽ-i jx@lεΚ•MΉέz«V­2A΄vΛjΧ->ψ έ’˜σtdγοΎϋN΄ΡΦ.₯ϊηΨnŸδ!€ €xS€ηΙ;/™πŒΐNΓΝ{"―>θέ»·c ‚šέxγQ;?²s΅Λγϋ?*άcΫ]δ!€ €@Ί\~ωε’vΪ₯.]ΊΘy睲H_φ>묳Bς‚g΄³ZmSˆ'%»ΔΏoΪό©ύΆ)Vχ˜©ώχ‚ρ$7οSγ©—WΚjηΤ+V΄­ŽjŠ]rγs뾈#Žλ―Ώήξ0£ζ5jΤΘΆΣα¨+%Έ0‘k_wε–a‚‡Εj €E\@G:uͺΌώϋςή{οΙθΡ£eΪ΄i²tιRΩΈq£ψŸύκΐ:š¬&νΔBYΠA*ώψγωόσΟ壏>2Ϋ±T£ˆsψ €d„A±q9@ GΰΊλ“oΏύΦτ΄•h Ύϊjp₯/λ΅lΩ2ΡΝ$Όή‘G)W^yeΒλ'kE―Τ#‘γΡF–ξݻˏ?ώ(wέu—85°†o[Λ½ϋξ»& ΊnέΊα‹σΧ =½ώΞ>ϋlΗ²·άr‹όύοm°K4΅iΣΖϊ&»Η8mdύε—_€_Ώ~1Uνψγ7ϊ΄—;§«½Σϊ…Ÿ››k£wήy§@ΏWτ8jΥͺez|VγςεΛ'th^x‘θ‹ύ±&έΟ7ί|#=zτp\EGλ‰'πχ¨£Ž2½8Πv›ϊ7Zΰλm·έfF"JδΪ¨Y³¦ (֞’?ΗNϋ&@@ΐ{<'HΞ9Iηgω ΈyOδΥgϊr°>K±KσζΝ ŠNFβ;R‘{μHr@@ΐ›Ϊωζ¨Q£l+§K>ϋ쳢˞xβ 3R¦έBQιͺ«²[δ˜—Švέm ŽδI]ΰΤ6”Ιώϊ3o»°›χ©I=Α.m¬I“&Ά{N~Ϊ/έΊΖάΊοτΡGε”SN±u°ΛΤχ τwΈv|μVJδΪΧΊΉeθ–ϋAΘ ρuΚ”)ζ=9tBŸ›οΨ±ΓΏV¨PAͺT©bA Ι]dυo €ΥιΥ«W›`Z9vΜ9™ΓQ € €@ˆ€}«{Hf@@gž={Κ¬Y³Lγ«ωΕ’΄±VGΌΤ‡/½τR`ΗΒŠΥϊ>πΐŽ=œΖr<Ι*γ•zD;mΐͺV­šh\Ÿ>}dπΰΑ²hΡ"Σ¨O _π>4¨U<騱4Rjpͺ6Ξ,X°@ΊuλΌ©ˆi­οΓ?lΚώίύŸνθ+ύ7C«†n²5hΠΐ©XςkΤ¨!ϊΰmόψρ’/λ~tΔ]Mϊs’#}φκΥKΎώ{;v¬θ ­[Άl±έ§Žθ™••e»,2υψυ% ½&ήzλ-Ρ^tγIϊπSΟΉφϊ§ο5Ρ†πW^yEκΤ©γX έΗ€dξάΉω^“z|Ϊ{αγ?.zΜRΩ²eE―Ω &Hύϊυ₯E‹R½zuΫβχίΏΜž=Ϋv™fj`¬φ€xβ‰':– ^ £έͺnσΈγŽ ^δ8ΚŸcǝ²@@ΐ“<'HΞiI‡g‰©›χD^}&ΠΊukΉθ’‹ω΄γ£•+W:.gχΨΡβ;ž«†² € PΨ‹/–ΫoΏέ±—\r‰΄mΫΦvΉΆ'hšSϊϊλ―MΫ‹Σr»όd·“οƒΆ…` χ§3јcŽΑ4žva7οSέ?Λίγ 'œΰΈ}ί#Zeͺ―1·ξϋυνωŸό§Dλμ[Ϋ© dΪώc}wΖ7Ξ‰\ϋΊ · γ<Š#€q^ߧӁ6mΪdή₯ΣχΗtΠύϋBίχΓƒbυ];εύΨclCŸΣ΁τέ«―ΎϊJΆmΫ±N§ζπ@H{bΦΎXBG:逓Ε5 B‡™'!€ €~™3gš‡ϊ­'θG’4 L?Ϊ«#cFk,πo‹ο’'°uλV™1c†θυ£mDΣ?W«V­jD5Vlιˌ‰$}h6yςdYΊt©,Y²Δ|―X±ΒŒ$zψᇋӸqcσ€LŽΉ:dθ騣₯J•ŠΨ½φςzΗwDδk΅gΌLKΪsŸ^ϊˆž3ύΦΟ²eΛL0»ϋ? 6”ζΝ›G 6 χΡλ‘]»vαΩf^―Γ;ΜLοΫ·OΎόςKωλ―ΏLcΊΞλHͺϊ0΅wοή’——g»h™ϊϋρ»οΎ3ΫΤνκ΅Χ n³ώ) ΦΧ4ΘZ±Χ­['6l05@[GΎτx]W7Υ?ΗΡLY† €ι# A_ϊ’˜?yζ™ςΩgŸωgωΞ ždΠΙLς‘Έ}O”Ο’L±9ξ±#H ”Α=vψXΩF`Ĉ!#Pισt}FBΐN@ΥgzΑIŸυ%ςœ2xL#€@Ρ(Μv’`mΪ‚5ܟΞD%LΫΒΦ―_o>»wο64k€c‡L EΈ΄Ϋχ©αϋΟδy717ξϋ΅=χ·ί~3ν»:ϊœΆνλhΊM›65ί SΨ)‘kί_g7 ύϋβbΨ΅k—δδ䄬 οΝΈ€RfH‘ΐΝ;M{αΠ‘CΝθ°ϊŽ $ °uΤQf$φιΣgȐ!˜‘`uΐ ΤCσΠ§υo}n̘1ςΣO?‰Ύϋ₯#Οκ;«ϊΎΜjπl:¦©S§Κ7ήhˆΠΐN;ν΄t< κŒ P„τd}WZΡg6W]u•œsΞ9ΆοΚΗΚ”ώCIΕz€”C@W4˜K?$ΠήSυα“~R‘όΑΩ©Ψv²Ά©ήͺT©βΈ9ρΣ.i@c&&…Ujκ§0“(koƒΙL:ς«φθξfΐαkΉΖ|R΅ίT§ͺήl@@ 5<'Hk&lΥν{’tx&κσΚ=vr…ΉΗN'[C@o €’$ψˆi[Φp:ύ58Ρί!r¬’nί§ΖZ―L(ηΖ5ζΖ}Ώvͺέ±cGσρκyIδΪχ‹†ώ}ρ €@Έ€΅j cΖόhbK–,iήCΥχ΅4θU;ψΧ‹Λ•Λ5ίαλ벊+JλΦ­E~Σ`ς>ϊH6nάhhuδYύάnŠπm1 €ι!@Plzœ'j‰ € !_ύ΅h/ŽvιŒ3ΐ₯KΫ-2y::ι¨Q£l—gjP¬νΑ’‰ € € € € € € € € E‹ΙμΩ³Μρh‡%=zτ4ƒkh±& z­[·΅n™9s¦Lœ8ΡΩΞ™3GΆlΩbfcέε@@ΐΫΕzϋόP;@@ xύυΧeψπαΆGυδ“OΚ­·ήΟΙ@IDATj»lώύ―_?σpΞ@―^½μ²ΙC@@@@@@@@@ΐ³;wξ”%K–˜wγŠ/.Mš4‘Άm6£ΏΖ[i5Ά^½z¦M™:uͺΐBbΧ­['U«V΅i6ή}P@ _ xαW € € Ptϊτιγx°C† ‘‘C‡ΚΆmΫeτaάwί}'}ϋφ•/Ύψ"<‘½αiΐ, @@@@@@@@@tΠ Ψ 6Vu΄Χ5j$|ώcK—.mΆΉgΟΩΎ}{ΒΫcE@πž#ΕzοœP#@@ 8ν΄Σ$++K8q”[·n•nΈΑ|4Π΅L™2²f͚ˆrα]t‘”-[6<›y@@@@@@@@@<- ονή½;PΗάά\Ρ€ΦD“Ž«οήι·&ŸΟ'Lts¬‡ €`€Xžͺ„ € ΉΥͺU“aΓ†I‰%’€ΘΖ{Μ1ΗΘΓ?u[,D@@@@@@@@@ΐ‹Ε‹7Mψλ¦#»ξίΏί?χχ‘C‡DίΏΣo ˆΥwυό£ΖΖ½1V@@O λΙΣB₯@@2Y _Ώ~ςή{ο…<ΘKδx;tθ ί~ϋ­hΟx$@@@@@@@@@M ;;[*UͺdXutΧU«VΙƍ> ͺ]Όx±μέ»ΧlC·_\Ή„·ΗŠ € ΰ=‚b½wN¨ € €@ΠΐΨ‰'Κ%—\"eΚ”‰λˆλΤ©#oΏύΆL˜0AςςςβZ—Β € € € € € € € € €xE@VkΥͺ%₯J•2Uš?Ύθ'ΡΡbW¬X!3fΜ ŽΥQh+V¬(UͺTρΚαR@H‚@VΆΑ&@@@ ΆmΫΚoΌ!O=υ” 6L¦M›&ϊ@nεΚ•ζϋΰΑƒ&θ΅rεΚR³fMιΪ΅«tλΦMΪ΄icΦ%°KV Π‡W^yeXξfsrrlσΙD@@@Hξ±#MΘA@Ό.ΐίp^?CΤ@@’!PΊti©_ΏΎΤ][–.]*Λ—/—~ψAͺV­*­[·Ž AG™ύώϋοeξάΉ&¨677Wš7o!εΛ—k;F@o λνσCν@@Š€@₯J•δΊλ+Gκ½C¬[·ΌόςΛή«5B@@@ ΝΈΗN³Fu@@K€ΏαΈ @@@ΐ Ŋ³‚bHηΝM@¬Žπϊλ―ΏJVV–θ²zυκE jυω|²{χnY²d‰όψγ2bΔΩ²e‹x’N:rΒ ΗKΙ’%½p¨Τ@’$@Pl’ Ω  € € € € € € € € € € € €@|yyy­[7™Πχ§*UͺHNNŽ–o@N`Σ¦M!u9r$ϚCD˜A·vμΨ!ΪήεOϊςL5ό³|#€ μΪ΅+d_kΧ ™g`πΏ§uY»vνύ$‰i@  6ˆΎXοOϊ}Ŋύ³|#€ €…&pθΠ‘ˆ}―Y³FκΥ«‘O™(PΌxq)S¦ŒΤ]یόͺ?>ŸΟ|όΗ«#Λj9ύ”(QBtž„ €™/WP¬3œtΘyŠž„dš€@ ½4؝”|}QT?$@ΒΠF. €™$ /žς¬9“Ξ(Η‚@ϊ hη‰ό>JίσGΝΘ$m‹'!ΰ$`w},Y²Δ©8ω €q hΗ9ΑηΔΉ:Ε@@” “±q<$ΰzυP•¨  €²@ρBή?»G@@@@@@@@@@@@@@ _ΈFŠΝwk@@@@@@@@@@@@@ FuλΦΙμΩ³₯X±b1½˜nG?ώQfK—.-eΚ”‘άά\©\Ή²δδδDίK@@ΐΣqΕ6hΠ δ`*T¨ οΌσNH3 €@: \qΕ²vνΪ@Υο½χ^9ζ˜cσL €€χ.ΈΰΩΎ}{ ’<ςˆ΄lΩ20ΟDb_~ω₯ΌφΪk•[΅j%C† Μ3 ΰŽΐ 7ά K–, μμΦ[o•γ?>0Ο €ι&πα‡Κ°aΓΥξΨ±£ά}χ݁y&@·F%C‡ μI“&ςδ“Oζ™@ά˜a@ έ¦M›Rεš5kς[ˆ3 ΰ–ΐϊυλCvU©R%~…ˆ0ƒn θ‹yΑ);;;x–iBΒΫDta―^½$///€3 €± |πΑ!7nΜ}Aˆ3 € PX»vνŠΨ5Ο "HΘȁ-[ΆΘτιΣEcT4ͺϊ|>“η63AψΛϊΏƒ™νψcυ™ŠŽ«q1ΫΉsgσ©^½ΊdeΕZΌ ¦@@ ψŸ»ΠΩ% € € € € € € € € € € € €€FsΥΐΥΘ‘C‡ ‹Βj°«~tδWMώΰW]ό1 ϋΥ2ΊΝύϋχˎ;D;Μ\Ήr₯,Y²D6nά(={φ4£Ζϊ·Ό¦@@ΐ›ΕzσΌP+@@@@@@@@@@@@ γκΤ©#ύϊυ“ 6Θ΄iΣL°ͺΑκ(―+V”Z΅jI•*U$''ΗΠ*ΘΎ}ϋDGTή΄i“¬Y³Ζ¬λ¨-WœΤ­[WrssεΰΑƒ¦μφνΫeσζΝ²mΫ6Ω½{·,X°ΐδλ~Ξ?|S6γ‘9@@2D€ Ψ 9‘ € € € € € € € € € € € n 40©Ÿ}φ™Μ˜1ΓTΏlΩ²];iΥͺ•4lΨPͺW―»sηNY·n,_Ύ\&Mš$“'O–­[·šΐY]ηδ“O6Σ{φμ1Α°«V­2A·S§N5£Ζ.]ΊTFeφΣ¦M›ΐ(΄ιζG}@(jΕ΅3Ξρ"€ € € € € € € € € € € €Πa5`uΒ„ f4WΆsηΞΏiΤ¨‘”)SΖ·+V,€Ζ>ŸOš6m*τzΔGH^^žŒ3F4XvΩ²e’••%:tC‡™cudY ~ύψγeόψρ’£Ηj`¬ξWΧ/UͺTΘφ™A@o λΝσB­@@@@@@@@@@@@ŒΠ€Ψ/ΏόRζΝ›'₯K—–N:ΙΉηžkF‰Υ€Y§€A²%K–4ŸΦ­[KΩ²9²{χn3ϊλ_ύeΎ5Ψ΅B… &ΰUƒkΫ·ooφ±qγFΡc΅ΌδΨΚ•+;νŠ|@π@qΥ…ͺ € € € € € € € € € € € €ύϋχ˜9ΙΔ‰EG~­_ΏΎτθΡCZΆl)ΡbΓy΄l£F ₯gϞR­Z5Ω±c‡Lš4I¦M›fΆλ/―A΄Gu”tνΪΥΛpΌ:Κ««&šΚ–-+yyU$++K4ΰΦiΨ5jJvvΆ+VL8 [Άl!(6QtΦC@ΐe‚b]gw € € € € € € € € € € € €"4#΅ϊ-4HU?‰&ΦΏ vϞ=Ά›ΚΞ.mF“υ"«eI € Ε¦Ηy’– € € € € € € € € € € € €@F ”(QBΚ”)cFiΥ`ΦuλΦ™Q[=Θ]»vY#Ο3£Ώκh±:r¬]r:B¬&έ―ΤX»²δ!€ €€·²ΌUjƒ € € € ΞΫ·o—#FȌ3d͚5²vνZΩΊu«T¬XQ*W,΅kΧ–.]ΊHηΝ₯\Ήrι|¨Τ@@@@@"+@{@‘=υ8 €IΠwjΦ¬)Ε‹gΜ·9sζΘ_ύ%yyU€tιRqνOGύλ―9²|ωrπš““#‡v˜ν6–,Y";vμ0Λ4xΆJ•*‘Φv'd"€ €@J>(φ΅Χ^“«―Ύ:aνΡ#//OͺW―.uλΦ•nέΊΙ©§ž*7Nx›EyEuΫΏΑ“O>)gŸ}vD~A3(oΎω¦νf4h 3gΞ”R₯bCxμΨ±½{wΫνMŸ>]Z΄ha»ŒL’)°bΕ ιΪ΅«ιν(x»}τ‘sΜ1ΑYi9­½*…,͚5Kτ¦“$‚WAQΈύφΫεγ?Ž8dύ»lβΔ‰IyΈtύχΫώ­ Π&L˜±o2(lV­Z™ΐ1=ώψγ©Z΅ͺ–o<#πΤSOΙsΟ=¨6Nh$©h >\^~ωe3fŒμΫ·/_ νιυΜ3Ού› ξuς=` Έ&ΰ…ϋ*/ΤΑ5π"΄£ργΗΛωηŸοΚ2D ΰΚΎR½“eΛ–ΙqΗ؍vŒ0lΨ°ΐ|aLπwwrΥρLg&oMΫs΄]ǟτΉΝ½χήλŸε@ ώοώOΎϋšλsφνΫ‡δ%23nά8Ήψβ‹CV=ωδ“εΥW_ Ιsc&ήΙτ6g7Ξ;ϋ@ΐ‹£Gω›:Φ:ζζζJ5ΜGίΧΠη²Ν›7uυŒ*Ηs»δœN7Ϊvξά)*TTXƒTφμΩ˜O§ »ΨΞVΣ¦Meοή½±+₯οˆj“~τχέρΗ/§œrŠι¬5¬hΖΟ¦ςy,ΧrΖ_> …* ³j,‡vΆ½mΫ6YΊτΩ»0(Š³=QAQΑ6l( ‚Rμ"¨ˆŠΨKˆ•hcοΨ’όΨ φ±5*φή+φ‚ DAc‚ΡϋηdΞΩΉΩ½έ»½½»wžηnwg§|σΞμΤ―LUΌθΫ1‡Ε\ ŽƒΥΧI“&‰Η\|χέw* ”w―³Ξ:E[x % ϋΖo(^-π/€χJΎ!_BGˆ D€δx³ƒ—hσ¨ΖΑb ~zΌώϋΕG!vέuWqΞ9ηVΕGP—P¬Φ ?₯x!ύυΧΠϊψγΕE]€˜kγ₯φί n΅ν)n^ GΒΈωζ›4ΩξΖolFqτΫθ+L?Ί"@|Ψš l:ΩύΚ?¬Βz\5ίΣυΧ_―4ΎΩιpΌ·αs^ΐΥ?όP$‡m΅or†Ϊ©Ω‡·iΣ&g’œ,€vΦΓ;L*%Ι}ܝwή©~{ν΅—¨₯²œ$2lyXW偆0|θ_9`Ί3ΗΏΚS*V6ŁωΐΔ­S§N5/ηέιVρLΟFN γ£ΉΞΕ3 D€Τ7ύϊυW^ye ·έv[*B±cƌ Μ#‘ ς«…«ΗωN£Ÿ9Χ’0O" Œd±“Πeεڍ9R¬΅ΦZβΈγŽC‡m*¦ξΫιVPΩ5λσs YΟΒ)lwρΪϊ·J…b‘ψE΅ƒ"7 0@ρ6“‘Ÿϋ±lΛΊ…ρJˆ€Πo―±Ζbƒ 6PΒ°˜ϋBaν―ΏΔΰΑƒ₯Α³N‚³ŽύœΓ‚ŠίIαΦ_•€+d pΎ†ω ”ΙΌςΚ+j\Yd‘E”B˜ξέ{¨y/ϊ2Œ7ΰ]Dδ9‡-ZˆŽ;*Ck>ΚΗ4‰ D€τ¨{‘Ψτ!ωoŠwέu—hϊΠC)­QΎςI’.„δ  Λε^ύuΡ­[7Χ«¦φ5j”»μ²Λ65,|}!fύψφΫo£GV ―Ό•‹ΘΣO?½H4%ΑB4P lbΓ–@’#λΨΥ ΕΒΪμ΄iΣ’gΞD€"@ˆ@lƎ+† Λ2lT’°ψζ›oͺΓ©φνΫGε»&D ֜ΝPΖ&lΊ,2 D€δŽΏώ*ˆΨϊΓ–)Ό!°ύφΫ+ n3fΜ(’K±8'¬Fp0wί}w1MάΐZΛN;νπγC8υxζ^Ύ!Dΐ˜³pΐβ†nP ›Υr¬l5Mž4jΝ6nΉ0§|ψα‡ΕΣO?-Ξ?|1|ψπΖ-,KFˆhΐ{7+Ν°j†26kϋeΉ‰ D ΏpόυW7ΔΦΆL™δ X[Ωc=,qie•8σ¬Fρε3Ο<#Π—˜n·έv­Z΅2½x‚@½ž9‡‡ήD€”ALϋQgͺsηΞuξακdŸώyΡ»woe«gΟžΪ›W"@€η8ψvΒ/Žƒ0ϊ;—ƒ‘‡ͺ„¨vέuWWϊ"@ˆ@N@ΏΏΡF‰m·έVρKC0vζΜ™β΅Χ^S‚²ΰ=XnΉεŠ;wŽ συΧ_‹ϋίκ~ώόωΚbψ*«¬’κΥ«WqΌύX’…α4ŒP’![ȊΐJmΤά:'0‘ "@ˆ DΰP(ΆLS€–“O>YάύeBςu^ΈξΊλĈ#DχξέσJ"ι"E 3Κέ|σΝΉŠ’Ω~·ΰ‚ *ζxӟ οΏ‘A|~Γ‚w͍ΐ_|‘΄±mΎωζ…!¬₯#D€"@ό π裏Š½φΪK™9€iκ’‹.1Z[·Ψb όΤSO°Xς—ΏόELŸ>½˜΄±žxβ‰βάsΟ-ϊρ†$E λͺ<А7†/@ίΎ}•Vις!…?~ΌΨd“MAoΊι&—ι*.³W >"@ˆ D€ °Ο>ϋ„bA”VV#λΪίG>tρh†3ηxH0h^}υU±φΪk‡η¦ί½ΐωλ“O>)ξ½χ^₯˜ΠŒ€χύϋχWώέΊu3_5ά=χν’W)Ο’cfΗ`»³‰χŒ³€‘#GΖ ,CΧwκΤ©βηœsŽ€ νΠξ·ί~’_Ώ~b©₯–ήΌ&D€m9!` Nˆ@E΄mΫVμ΄ΣN|Ε°φύωηŸ :ܲPΔ ΑUX…CΑV(Hΐο[l1±κͺ«*γh˜η.Ήδ’EZ0>@‘χ?ώ( μ«uλΦb½υΦΫoΏ½Xa…ŠαxCˆ D€δ†Š…¦hξp9Lz0™F,„1! sX$ΏυΦ[bέuΧ B#€Ί>ꨣ”V—“Iˆ€˜3gNΐΪ* ΩsΟ=Ε?ώρ":Πα‡ ΄βΛ:ΉΑ"ςž{ξ©j³'“ψd9sΜΠβφΥW_ ‚ΆY—@M1@ΔΝ‹/Ύ¨6Δte—]V@ D€"P=X“xΰ%±C‡Χ^{­ΈZ.` 6L)ύΑώ φf΄;οΌσDŸ>}J”ιθχΌrδa]•ΚαΔχΙΐ!:¬―Δq. V /Όpμψqς`"@ˆ D€"°†οΠ‘ƒ˜2eJ‘œ;οΌS\xα…EΥβ‹78ίΏϋξ»!aΩ₯―TRCWf9s.C" €°ΐK/­~묳Ž8όπΓ•p, L˜0ASΒc°Κύξ»ο μa4ͺγΎ]²šεy@2ΌΒB³έ…!“? Ίvνͺ~ƒ RΒώ& „f‘Έύ ]e°-W†c"(ά†`lϋφν• ΗϋοΏ―x§aΩuήΌyκχλ―Ώͺ„q~‘}œαa»Μ2ˈ΅ΦZKYœ…[π šξηŸV–Θ±—~EŒΨΫ€rψ1‘β= D€ό#°@ώI¬œΒkΉFmβA˝ύ{ζ™gΔoΌ‘΄…@γΗW\‘&8aΉ!>]ύ" 5Φo Hy3 0vμX₯ΕH—‡guVΰ°‹1—fd‡W"@ˆ@=#°γŽ;6–ξΊλa›ΈεƒzΣνΌσΞζ#ο‰ D€*8βˆ#Κ'ΤψGqύυΧΗˆ5³Ζ”ΨJ͎;ξ83ο‰ D€"@ˆ D€œ"a«½χή;@¬ΈΌπΒ ΏΈΰεψζ›oΑχΪk―ΐ™iΰ%πΜ9ˆA`«­Ά/½τ’_Šι&Nœ(FezρΎΙΰy@“7€:.>ϊ7X΄έm·έf{ρ™"@rˆ„\‘ kΫm·U–Ύ8ΰώΏΝ7ίLτμΩS¬±ΖbυΥWWΏ5Χ\SτθΡClΆΩf*Μώϋο/φέw_₯B―š5ό  pΐ{Pδ½ΔK˜ΑxOˆ D€ΤΑQΎφAβRK-%ώτ§?‰[o½54ωI“&…Ύγ‹ϊ@ΰΨcUšajAνΜ™3Ε'Ÿ|’„°g͚U ˜g pΓ 7¨άgŸ}ΔΚ+―,6έtΣ€Ν7ίxζ D Q€†bΐj`~Iκ Z衃6PlΡ"@ˆ Υ#€΅­½&–Υ‹/ΎΈbζTh\=ύτΣāω yΡ"@ˆ D€"@ˆ ωGΐŠΕΆςΚΈ₯p)ΖΉ)]<xζ'†"D@(Λ±=φ˜²¨eβqφΩg‹ύλ_¦ο›ž4iΕ7P±»tι":uκ(Ρ§Ÿ~xζ D€δ³ΆnέZ@θuΰΐβχΏ½vͺ„Y!χqΘ!‡¨ίΑ,† &†ͺΒl½υΦ*sC‘—νΐ—=`ΐ@±ύφΫK ±½•π-,ΜΊΒΪqωLˆ D€δ …ςENm©4h²NφŸό§„ιΣ§—ψEyΌωζ›βρΗΟ=χœ˜:uͺψχΏ­,@ΆkΧN΄oί^¬ΈβŠL£C† k―½vhRωφΫoΥ{hS sΪ…Vν:wξiωαΐ` k, σ³Ο>_ύ΅( ’C‡bΥUWUΧ 7άPμΎϋξ«m:J―S¦L·άr‹ΈοΎϋ”(F±±ξΊλ*ν-}ϊτλ―Ώ~₯Ι‡ΖϋψγΕ%—\"Ž>ϊθΠ0i½xλ­·Δ=χά#~ψaρΞ;ο”γ.²Θ"ΨB@g—]vQ“ο8yΟ™3'Txzνf̘!.ΌπBρςΛ/ ”νhΪ΄ijCa ˜‹ΊwΉέvΫM΄hΡB½BΫfοΏΎΐ†ΠάΉsΥδν V€zυκεJB|ψα‡ͺŽΗ'πν̟?_Υνzλ­§„Ββ9kO|ƒO=υT±΄XΜAϋϊ‰_|±ψZ•ρα[­Φ‘NΡ ΟΒ‘κj΅ΥVSΪ“:vμ¨Ϊ©­%©Ϊ<σήΌyβƒ>POžZŒrϋν·e‹/"nΠΝƒΪΑƒ Œ½Υ:τ‘Π˜Œ>m½eΛ–jΞ„9ζ2Υ8Œέ&LPγ6ζu³gΟVύ ΄—"νE]΄βδ}¦ ’²κ7ί{ο=…ύ?ό VZi%5Va“2ν IŸυ\q%ƌ˜U]Δ$§β`YΦΖzΜƒπύa]‹ΥaΞχ·–/ύƒ°ρˆσ”€ΗIο²Λ.˜sšξͺ«ͺz¬…¦ΦO]νΈΡ0ΞΡηX7€¬°Β JxΰςΛ//ŒΆϋ“N:©θηΖχœ2‹yqœr"LVη–šίϋΔ:ϋšΧσ“NπΆ½ρΖŠχλΡΆmΫ*ήΜΎ}ϋ*Ύ3lϋZœ;DΡθ»Οφύ G•Νη»M6ΩD˜qΠNπMAψ)ŽΛb<6ι¨ΥχnΥ½ο²fέWg…σ!͈xΓZ΅j₯~&?*φ₯α’ςŽa ˆ;4#ή,3 D€ΊB@Nb;)°…ΩCρ·μ²ΛƎλ+ΰΥW_]€Η€ χR1qΆr²δLοΐŒ•–<δ)τλΧΟ™†MŸ~ξΡ£GAZT)ȍ”’ξΈγΤwfβlήγ[’L€©p#I^Σ!iφ›aCί†1O χκOγ#‘ Χ^{mAͺ$€eαΒ–[nYόI%aI—ψϋ¬η’Μ"<0_ΥεΓ5Μ8νΊΈνΆΫ}?ϊ©τ%‚κW'Νρχς@Ί4 αγ£0k:0ίΣN*-*μ·ί~¬  _>ΏΣ@Fe°n4ΫΖ­·ήZ&†ίΧ§œrJ€©4Δk†lηαπ¦ΡΞΓSΎAkΟύ€‡` *ž°žΖZB€"©²©Υσ<Έ–sm}YY΅\s‚Žrλͺ,֜εhI׎΅£A«‡yž9ΆνΌσΞ>²©YšΨ›6Λ‡{©τ'zm’ž Ο1±Ψj«­μ,ΟW^yeq.§ηtRΑ`A2K9Γ'ρ¬dލτΣμ?³θ‹\˜€Y~₯xκψ΅Ύ^wέu6*zjMRέδέwίŽ=φΨ‚T&ΐPσRΩ`αδ“O.žΛ\ύυpΨόγμχ’ŽΏQτ₯I—+Ÿ΄ϋcWiφ ibλ’΅ό°—₯Ώ\₯BΈf(6ΛX!scσ<ΈΒd«Švι₯—Ϊ0ΪρΣO?(Ν'žx’$ )X; _}s½ΜwκρΜΩUΉiμ+θtΑ£ΧΈβ¬4‰“߁ψRΑbhτ΄φι|΅c“π40n΄}Š=χά3ΠHP²άάί{ο½:ΡΧJ‘ŸͺθoΞ#ΝyΞζ\|lŒœSΊO{^gίΞE‡―sΛZξ»Κ©ύςplΜvΉΠB iς ―Ύϊja§v*i»:<Ϊ΄΄W†Šqβήψ8w¨΄έζ4ϊ쨲ϋώ†£ςΆίI₯@cnS­ΓήƒnΈβ *Ξώ’ρ8«οuj–9ΝύΨΈm9«²ΪνΓW_mη“υ3p7λχRαJΦd0?"PΧ€ΧίRά9n ή-ποBNεΘ#‰€‰"@ˆ@$±y²Ν6Ϋ¬ mΖZŸE% Λ ±-0ΨhB±`ŒΆϊΉε˜šΚ κτ\Χ?ύιO%“­΄…b₯ε΅Βb‹-ZN]ΪOZ₯,Hk²‘0„•B±`ςΡiΕΉβp› QΞ¦ΡιJ‹€…=φΨΓ™psvΧιαZN(z•b‹τρ=½ϋξ»a€)Χᦦpθ`oTλχζ!¨½ρ‘ΓΰŠο ‰`/„αΐό‚:3ӊΊί`ƒ ”ΰ“ŠœƒΏZ ΕBHΤΔ릛n  ‚Ν)σ=°ΤA0ΜHfzεξ{φμYR_6σjTΔΧΞή΄FΌόQΏVWl^Ϋι2‰³75QΗa ZτίφΖͺΏλBr4+Ε勃„NΡG»h σλΦ­›:$°Σ²Ÿmz€EdμΨ±N|W~pΜBp‡B±vν₯σœ₯P,„Yα tΑlKI1aƘͺγCQ6½₯…τ’ήΕŠΕχœ€@_σδ“OΖŽ;ξˆύιςuΤQ±„Ϊ}¦Β₯έoΊ{ν΅Χ jy ή4φu£6RγΚmΎ³m糞γδo†IΚ\ε£.ΠχΫσΪ‘#Gšd–½‡ς ³. h#ΚωͺO0iÁΉΓΕ°νŠυύ-EabΏkf‘XΆs»5Ÿ«mηΑΤ’Ÿ°–0Ώmά_xα…Ρ‘<Ύ­χyp­ζΪΎϊ\]Υ΅^s‚[΄Usέ™Εš³  uaSaΟXWΧbŒΎœ]~ ΕΖCΪΗΈhζμcώcοEΖaΒ‚@ΤΜob‡v¨ϊ@D—Υ?‘Ge4iχŸYτEΊΌϊšvtΊ•ΰ©γζαJ‘ΨΚjϋ5ε κo‚}oΏύv‘‘Ψ΄ϋ={όΡ4ΊζΎΆRΪtΩιϋθν<ξΒΦ¦³™ž)ΫL΅]}Y]ηΖζypυ9$OΚ©m~C9$QB|p`α—r<:Ÿ}s½ΜwκρΜYןΎ¦΅― Σ³Χ#¦@•u΅Ο`?ϊθ£Πΰf;©d? ϋlǚπ΄0n΄}ŠfŠEΫΨqΗύ/ζΗq„ ӞSκvͺ―>ζΕqφνtώϊκσά¦'/<y8°±Ρ}8”&Ϋsך~P|aςΈΞΧΉƒ]ΠfξY‡Ρ—VŸ–Ύοo8,ί0›7# ‘X>0Ϋφ2Κ9_γ±έ|}οφό'ΝύX» amΩ竬f]ϊμ«Ν|jqO‘ΨZ Ξ<σ‚ r fϞ­[gΝϊIž›Ί~³δψŠί…3f(~xœH‹αΚ ψό‘^½: ΕΦkΝ‘n"@ˆΠP(V#a\Σ΄{Ψa‡ΐζb QξωηŸF4Σ)w‹{¦€UšB±°wC(ŒNhΡ0–>,v…₯ε?hΠ °¬””Pμ”)S”ΖN;}Xά?~Ό3έJ…ba•¬U«V•Ρ€]a΄`ΧᦎΝ [xAΏΓΥ<΅7>Μp° i>—»žψ>ΆΫn»Dρξ‰'žθ¬‡ZxΦR(ύ‡‰3κC¦»ζškaŒsIγ±αkζ§οΑhφ # ALW)ƒ‹½ι…΄]½žΡ΄αšΤ:εΪk―ˆ―ΈΝ2ΰψΫ›ͺ:_|Χ`Ψ‚Φ`νg_]‚φ•bzββƒ°p°Δg[λ2iākXγϋ=ϋμ³›PΘΏM6)ωΟ–X/6σtέƒ†¨CβμyS(6\±g)‹y4Ώ˜ν‡qRΝΈ{ν΅—Š AnΣΏœP,6ΐ`Ι݌χύi9aL0‚Ίϊ|—ψnMΖ ;ί^½z•Œ&>>ΣF>>ϊM“~άΓ3¬NΪez†R”+Έ"§\=ψg»\qžνΊbΞχYψvLΌ“(δΐzߘ?μΐw˜xβΫϊόσΟ «Ίj€6M§-λϋ[ŠΣΜ0Ν*Λv^~L©¦›m,Ξύ9ηœSςύD­aγ€Yi˜F˜g=Χφέη’.σ°ζ6Άθλνu§ο5g*Y;f5FGίΞ.?…bΛ#ξs\DξΎζ?φ^d9&,{Šob£φ₯Λ£ aŽŸH?jήν³τέιRϋ,ςH‚§¦)OW Ε&― Xρ‹ΪKΦλ,σ %f80ύΚYŠυΡοΩγI}&λƒ.³|υΗ:_}BΨj›υJ‘Ψf­ωΚΚν:76Οƒ+K΅ϊXPdbφ§θΑθΗ!ά2Λ,ˆ%šqœοΎΉζ;ΐΐΔΎ^ΜΝϊMs_A§k―G΄@•~_ξZPl’ύhΠα»#΄1n€}ŠfŠ½οΎϋ}ϊ“Ώύοh6NηkNifζk^gίΞ€ΓχΉ₯MO^x2ςp`cƒ>#lEjζψηΊΗ8>uκT³Zχ>Ομ²€N{ΟΪ&*ν>ΫL?‹oΨΜ/ξ½ΝΏvΖ7=„ƒ‚m³]€ί"Κωνvΰλ{·η?iξΗΪekΛv8_eΥuι»―ΦωΤκJ‘ΨZ!Ο|k‰x`δ Φδž}φΩd 0g;φωϊ»λ»+ζ 7ί|sΖx0žΐφ§ηϟ_ΛbU•7…b«‚‘‰ D P(ΦQ ΥΕbb3mΪ4eBή>œ1ΒΫn»­#η Wοή½‹g3~ϋ &OK( Cq-m•£ΪYΓ\R&ˆrya+Jσ`”P,h<ω䓝υ²υΦ[;‹P‰P,°΅5–+WΤ{˜sΓM–υ½λj‚Ϊfxl šΟqξΓξΚΕ]zι₯•Φž°ςfι_K‘Ψ<0€ωή{ο]RtXβ΅Ώ―?ωΟ%α’&ύ° Λά<π@―ιΣ§Ξ:λ¬’6l δζlz€—ΩW[m΅ΒΉηž«6`Qϋ±Η+œzκ©NΛ΅8Hτι(λ]›9©€zUφ8Žg8τW¦ΐ(© ν-Ž³΅Θc3 νάόVΚ ΕΪύββ{»ΰ‚ ”ΐζχί―ΖΡ/@hΧVŽ>λ‘Gq’Œςuξά9@ζ&Ψ€7™ƒΠχΏόςΛΧ<L†.η3mδη«ί4ΛM}6γπGΫƒ+”‘+Τ¬Ϋ›‡žφœ€œP¬Οz6Λ”δ>.s•οΊ°™0QοΎϋn¬’ΌτKφ:šηΕqφν4Yœ[Ϊτδ…'#η66XSškLπλ]tΡEJPζ‹/ΎPΚ.Ύψβ‚­„ νγd”σ}ξ`—4Ω{Φ&}>ϊl3}ίί°™W’{³~θ¬ΖAx¨eΛ–ώνK/ Mχxl·_ί»=Is?Φ.CX[ΆΓω*+*3‹Ύ:΄Ρdτ‚B±Νlrƒ”7`―|ΌΗsLΚx`Τ+±ΐ}ν΅Χ†%π9rd€ž~ύϊήλ‡1cΖΒuιE)KΠο]WΒuλΦ-/Μ -β'Ααγβƒ°θKMά1ζ™cΒΨΈΪB5hƒΎs9=:O|ηamΒO+¬°B€>Œ‰iZu±ι₯P¬H:ΟφΒ§P,4²i+@Ί­ακ·uX}E{lΧ]1ϊ,-L Ζ3½(‘X˜-Ύψβπ 'μ;Aώ°Pk dBhάΥζρΖxφ-ι²AψΦ€Μ§ΨΨ·Ο΄‘WύζI'(+p―ΐl …&>ϊ>J(Φw=Ϋuχ9.s•οΊΐβsΉε– ΰ ₯qΖ]ΈφιΣΗ-‹:0ρ„βMζA—]vYσk—σύ-Ής,ηgί·ήzkΉ(^ߟrΚ)E<+ϊ₯΄Ϋωo{ac 0―΄WR_tΠίΠ] Χ(σΰ,ηΪYτΉyYs’MΊ°΅ΧΎΧœqh0ΏŸΈkΗ,Ζh“.ŸχŠM†οqΡηόΗή‹ cΒΒ>Ί9ΞΰΚγ|(`0ΗOδΆίν»τέ‘•ω.ςˆ‹'ΒζΡQ(6Y­Œ1’δ[=όπΓCχ6 0ϋ0φχη(‘XίύJwό5ςM—ΟώεΘ’O@>•`‹xΝξlΑ" Ε6{‹ˆ.ΏλάΨ<ŽŽνονάΉsKζPΠΗΩJ/±ΗσΚrΞwߌόσ>ίΑ΄ήϜ}ν+Ψ둬„b“μG£eю}`άHϋΝ. >3{΁—ΛbNιs^œdί.‹sK=Ί.jΙ“‘‡σ€0lΐƒ½EΜ;\JοmϊΑc΅ΗδϋάΑU{ΟΪ,‹>[§ŸΕ7¬σJzMS(Φm~@ΜιlΎZ“Fί㱫ψψήνωOšϋ±2ΈΪ²+œ²’ώ²θ«ΝvR‹{ ΕΦuζYKpvž™€<ψ‘!Ԋ”_€ΟΧόΑOL}ψ0,™/_-˚$o Ε&A‹a‰ D `]+π»ξΊkΖ$oΉε–"Ÿ₯τŠ$mANΦΪE ΕκET₯WL€Κ Δ’όΆπ•ΞZ¦Β,"aq ¦VΦΌ†YDZf8σZ·Γά?όP°…t\lψc1o;’AR‡³―Ψ|tΉ(‘X—Έ4ΉBΐnŸ}φ Ν Υ0WN(ρ fӏgXΑ΄'·v7γA+’νΒ΄L!9°aζrΠ±5΅šy‘Μ.αΧα¦Oίcσoν΅ΧVΈBx›€ΨhΠΞήψΠρτ–9΄‘Ž3qβΔ‹/:ΌΎ’!œι° ²€ !ι<8{,ŽπUtίtΣMφ !b—Π)ςr΅εϋοΏ?ΨΠ΄…΅Π–s_}υU‰%UλΫ. ƒ‹kΣΛ΅9†<°)§ΫΫl³΅σΩΆή&(zΤQGΏχή{ιٞpKWlO\|Ύώϊλ‹ a}³M?ϊ[ΰοτΣO·ƒ©g=(ϋa‡ζ oz>ϋμ³œοΕ_4ƒ€zouΐžzj%k ±Bγ[9g+*v7νn»νΆ@{ŒŠ΅η ΠΙD^Ÿ~ϊιX}νΡG cA9‡…€Ν@κjγ>ӍΎϋΝΟ?ΌΓG}΄לqiΠίD’΅£ο1ZΣδϋJ‘Ψdϋ}Ξμ½H–Νxˆyϊ°}²d蕆ΆΗΟ0‘Xίύ'(σΩ!ύ,ΚOΠ“GG‘Ψψ΅‚3ŠEY$0O……ηrΡόοηΕqχν²:·tΡόkΝ“‘‡σ€0l=τPWΣ ψα,ΠlΟΈ‡&—ΛβάΑUמ5θσΩg#ύ,ΎaδS‰³Οό“ZŠΕœΌ‹'œpB Ώψ)zθ‘H²|Ηvΰγ{·η?iξΗΊΚΰjΛp>ʚU_Ωp2xI‘Ψ @fΉAύ”ΔoΌρΖJπότΰ‡ΟΜΊλ«~묳N?·ιΩ³§œΕ;άΓαρWΘ‚@ΈFž φ1ΧΞ’ΟεšΣ½ζτυ}ψ£}ΡΜt«GΐχΈXΛωϊv©x2φ£°?ΆOμι!‹ώ€ϋάΚͺ žͺ€Ιζœ5˜ϋ3R»ΎŒΪe)Ε>³dj/Ξ ΰ»ί3σJrο›.Ÿύ1ϋ„$5Ν°D€Tƒ€Tˆˆ.…u…TNπ³€’Υ’sw;;Ž~φέ7λ|ς|­χ3ηzέWˆΣ&βξΣωnΗ>1ζ>Eœ–Pa€‚ή‘φ~0^f5§τ9/2β!«sK >φ‰]ωDωΩυ_‹σ}Λ/ΏΌΚΥ\―~8 ”‚½ΏιΣ§žυC-Ο4 ζΥgŸΥ7l–§š{πvJ§²?πΑ?I Υ i(EœuΦYBZŠ-f ~;μ5n»νΆE?׍ορΨ•g-ΎχZνΗϊ(k-ϋjW}κΒξjΌ±+@ί. ‰^½z )ά*6ί|s!•oΑ(œβsǘή²ώύϋ i LH‘XώqΌ‚Όo½Tϊ-ΐƒGGˆ D€4nΙ«Ζ)_Ε%‘šF„1ϊυλWV(sΰΐ‹+ϋg3ςΨΔ„1”K­>vΠͺž₯6fg|UD1+`ƒ“H—+wheΗ‘Z]KλΜ0RΫ XsΝ5M―β=κ’‡‰νE]δΊΕ?L`8Nžwί}·3Κ#-ώ:ί™žaB³Σ€nψπαJΨ“ψJ/₯vTgt,(ΒΡ H&荍slJΊœ)¨ιzίΘ~€΄Μ£w!LΏέvΫ ˜)xωΏ,μlαYL…Υ₯Ζ Aƒ^Y ΕJΣθ‘8©ΡO ΜQΞΒ†P\Ψ7!5ό i]UύπΝΩBΗaωH άa―2ρGJ뇁Ό>ψΰΐsΉ[X >όπΓrΡΤϋγ?^HΨeΓ’Ε&ƒιΐψ@Gβ €οs1ν€fI!΅vκΗ’+ϊs\ΖζΖ­€NZΖ¦*Ψ‡eQiJΝυΧ:r€Y/!„o:αJλυ¦—σώ”SNψ†τOj¬. η3mdζ³ίDί6f̘@™Ž8βˆΐsΤΓ‰'ž»Ο’ž£hMγΟΊΠτm°Α%sτ;οΌSΏv^ωΟό1Xzι₯~x¨U`‹πrΞχ·T.Ύ/lηεΗW[‰ΫΞ]qΛωύόσρ Xϋfιšaμcν»Οεš3|Νιλϋπ9Fϋ’™ιV€οq±Vσ쩃ΡΟt#FŒ`ΗΎB-οώS—ΝηώWVeΠeα΅±ΐΩ•­Xgaη 683ΑΉT\η»ί‹K‡Ξ7]>ϋcφ vmς™_τξέ»d_έVjiηmΏοΠ‘ƒRΎj‡s=ϋξ›]yζΙ―ήϜλy_‘\;H²Oη³ϋƘϋεZBύΌ·…b]<%YΝ)}Ξ‹γΤΎ›¬Ξ-]τψΨ'vεεWλσ€0Ϊ°ΆŒ+ »Κ*«’qρ« kΙ P>ψξ³³ϊ†νrUϊόε—_Šwήy§μοέwί0ΜaςzθωΔ™ ¬™… Šι`pΈε–[J~(λ/Ώό’ƒ•½B(₯\^m΄‘3`V­ƒ₯XL„mΑ6,τ+q + [ΠΔΩ€ƒ°XS>¬Ε&qJ²Έ7­•W^94(κ‚™.­lQ‹—³7Q]aΥ–|±9£0΄υ;}΅-;C+Ώ-|’Γκ+4'™Ϊϋαo šιpaWXϋ„ΐ€ώٚ†Γβ₯α ΗX š.JxΪV±Ψ s°@λͺψA T\g ήƍ—V8lδšεΐΙ%•Ζ([Ποω矏ŠR|‡q1ƒRΣ™šMήvŸgίfX’6ΫΧΞ;ο\"|j†»·•–Έ¬Ν‡Ε…Zk­₯΄Ρι0θί}ύ¨˜ƒ™σMτAaσ 3lΉ{Ÿi#oŸύ&,I›}[«V­Δ&›lRΘΕχ+¬°‚ψWτˆΈΙ’ž#²Oε•ΟΊ0 ΄5ΌGΓˆgΏwΝΑVu€uXηϋ[ŠCΓψνsL|›΅›Δ½·…άΝ~;nΥ„k†y°ΉΆο>—kΞπ5g5ν½\\_}WΉ|ωΎvψžΥbώ₯’ϋΫί B1,@ΤZ Dωξ?uΑ}ξeU]^ &(-ΣmΧΆt¬ίΉ8[(g‘ΕŒη»ί3σJrο›.Ÿύ1ϋ„$5Ν°D€TƒζrφšeμΨ±Š Υ•.ψL₯—ƒψqη„ΎϋfΝyς«χ3ηzήW(ΧβξG#Ÿν8 Œνoή>+°±²ί‡%Ψρψμ›goΆΛjNιs^l—ΙυœεΉ₯+ϋΔ|’όj}F\β:[1“ΛpF­Ομ²ψξ³³ϊ†νrΥςϋ0¨bς“„Ργs<Λ3Λο½Φϋ±i—΅Φ}uXŸΚ˜3gŽ˜:uͺo?φ0–Γ ,­·ήzJΌθ]ΊtQ―Ψ―†|aΐ0x°aI<‡ΏύοUxπξ½ύφΫJvϋ&Οvε”2& D€"άζ σB]•t@3˜kƒN' ‘Flv@«„ΜΓ|WL†`σΦ[o5½#ο±QM%Π2mUΠΞ‰|>ώψγΠÞΘ+|‰Ι6o\.Μ ¬ΆGΏj­yΝ•^ǎ]ήjŠ‰θ‚ .θ|Χσ¬³Ξ8l³-ρ^}υΥͺ~Γ„6Γ‡ΰ ΄ΗΈœK Ϊ~‚|β‰'J^OŸ>]Ρ Ζ’8BŽ.-qβšaΚYϊ°―KHB@Qwθ~C +X4]”•X–b_|qaZP†ΰ:±Γœ-d…~±\}™iaqΉμ²Λš^™ήCSs΅{όρΗ•u\—ΰωK/½$¦M›¦ƒͺEν¦›nZ|Nz ‚θCΡ‡λ~ίk˜ξ€ιWc‰ιPŸhI„b16igb§ύ\WhηŽλlλ»i(:ˆ›7ΓΥ?`”Δ·-b?ψΰƒJh₯hΓΦ"­JάΔ‰Ρ°‘–Τ­³Ξ:Β΄ͺώcύυΧ/&ƒ1ϋτΣOΜ#{μ1΅9 ϊ°¬‰ƒ^Δ‰kmE'ξ3mGΤ΅š~Σ― 0eα…ŽΚδ]œ9'"eQΟ%ΔeμQM]˜€Bι¬k‡5πƒ¦EΫAι6§΅ΓΌ2LG­κ ΦߒƐΧhΨΞέψΔmηξΨΡΎ`‚C†vζ\RϋωΌ6Γ<ΨΗ\ΫwŸkα\sϊό ~KΫΧύ[Ό«7ͺ³žΐ*ΐ9ηœSs%k°’DRςπέšdϊΪΚ² fyxߘΨc>ΦΰIχ-βΫγ XmΏ'JΒTK—Οώ˜}B%5Κ8D€TŠδL(B€rK#ό³Ο>+Ύωζ›@VqΞM"ͺν›#’ω«F8sΆηυΆ―Υά§«¦g1χ)’ZBύΌƒp‘ιΪ·oo>ͺϋ¬ζ”>ηΕ%…rxΨߍΟsKGφΒΗ>±+Ÿ(ΏZŸ„Ρ–66΅>w°Λi·½΄ΗΕ¬Ύa»\•>CΑV~(Μ‰ΐϋkσ…"_π)]yε•β…^P<š0ϊ¨€%„6Β&CππβΣO? Υj‚I_V.£C”¦W`[­PμŠ+(N8α„S?ʏ£>Z<ςΘ#‰ΰˆ²δemΥΞ$ lΦ@SY\F°$šπl’<‡ΥS‹-’$S –^1@ƒήΐ"'ΎOΣA ΒΫε6<ΝMYφΩgJΫ‘+^όΉΒζΡ‚aψ¦΄ΐ&”άύNλΟΆΙ‘C‡Ζ.ΌyσΔ+―Ό’”$Όψβ‹ͺ~€+όσζ&Mš )IŸcF΄7zγhEDόΈΒϊf^Ό'• €Ν+hz»ωζ›Ut`<πΐb=φ$‡οŠ/΄kםθΧ―Ÿ~Œ}…El-€«#!€J'Ύώ{]]‘ι0?…•οέwί=0_„έkΑ{|kΠ^α`όβlΚωLΫ,ξΣξ7ν ŒwIέͺ«Z6JVυ\–€]&i«―ΎΊ€°6ΖFνΠ~‘EΦvΆυzX{·•# N-λ lξm—%ΛoΙΞ›ΟnΨΞ‹‹=¦ΈΠŠΫΞ]qΛωٚα³Šm†ypΪsν,ϊ\9Λ}9~ήϋ£ύPΚT} ΰc\Μrώƒ}1DΈά‘G©j©  teΡšεχ±•uΜςπΎ1°Χν•Μ;+‰4}τ{iΤ’Ί|υΗμ¨q¦Aˆ@ To“M6Q–Xt<œγΉ„bm₯—oΌ±S)ŸN'κκ£oŽΚ―ΦοαΜΉήχ’Ϊ@^ζ>Y`Μ}Š¨–P?οl‘Xϋ¬.Λ9₯―yqάΪ°Χ?6q‰sn–NΪϋΔaωDωΧϊ< Œ6(OΣΥϊάΑ.‹Ο>;ΛoΨ.W₯ΟδtK‡₯~έχή{OάuΧ]βΜ3ΟTόΌ:,όO:ι$qέuΧi―ΘkVσΚ,ΎχΌμΗ¦]ΦZχΥ‘ ˆ/‰¨πCΙψΚ!7€ω”=„d‘8ΒΰΣΓΥ拆"π݁JώΑs  «―ήQΚ#ό"ϊ‰"@ˆΘ -›nXΓbV]Β‘œ„₯B,Œ]šD`‘ ¦<9{αgVΞ"¨ΆΪϋV­ZU›D*ρ9ζqν΅Χ,X!a3CΩey.,c¬†Ή$‡QΒlI„bΫΆmFύsˆΐ 7άPBΥθΡ£Kόβx@€–¬O<ρDgp»0΅&9#δΜ‹Xh“Ζf₯vwήyg‰P,Α¦0βν»οΎ:JθBh—^z©8υΤSK„αΒ"ΩΦzΓΒωςOkSήξ7L Δa΄CAAR aiџΔAσ+-‹π`š±…baAϊ‡~(&·Λ.»TΤN]c;6ΝͺusηΞ-IΒΎδ1bDΙΌD†πΓ?¬~πΡ!‡" ψ%JͺΟ΄A‡―~Σ―–Zj)d—ΘΕ™ƒeYΟ‰ˆ― °―Ί°IΑ8l Εbv>Βίtˆηr΅¬ƒ$ Y|K.lθWŠΫyט !Τ‘ν—Φ³½–˜>}ΊRπP©’$›.Τ·­¨Œ7ZIV£Οƒ}Μ΅³θsν1άn'v=ηνΩχšΣgyΣ£}Κ΄ΣAΐχΈ˜Υό e¦Γ ϋ[pΠζ}Ψa‡‰Ϋn»Ν ’ω}ύ§Y(}QΦe0ΛΓϋΖDΐσ+aπŽ³n7Ρσέο™y%ΉχM—ώ˜}B’fX"@BΦ^ǍWLξή{οUŠZΆlYτCŸzχέwŸqS‰•Xί}s€ΐ=4Β™³=Η¨·}…¨ζd?ιψjΗYaΜ}Š¨Φw_ύu Ώ„-€υœΗΌ8nMΨߍ―sK=>φ‰]ω”σ³ϋγ¬Ο\τ(ΘNΣΥςάΑU»νΩυΰŠΧ/λo8.]i†ΓyΥΊλ«~묳ŽšW‚οW;πםwήy"κ›φ5kΜkVί{φc}”Υώ^’κΥΔέΌOΊWeΖε= ι#`τΓΪ-Ήδ’N9πΝaNe3fΜPŠNqΞd―Α`< ΚVƏ―ψή|σM±Ϋn»)Z―D€"@ˆ@}#°@}“Ÿ.υ`<Ωi§B8q’σέ·ί~+ ­4@,::θ  ίΞΆTfζ׌BM°TuΞ9η˜0ο!0›ΔΊo”ζͺ9sζΣ-wΕΤ•‡n›6ml―ΊxΆ!uAt•DBπΠ©2Ι€ΐ˜ΦΤ©S^Ά dΰeNlαV²ΫœΟ<σŒψ€‹°υΦ[ XˆŽr°>»α†Š#Ž8’δ€ΗŽνΆ°κ8fΜρΤSOΩ―3}Ζaƒι*έ·’¬‘λόšρ›Υeη΅6 0 p ρΠC)p&5ΆyVβ>όσJ’•-v.e,ΠBΕ+ƒε”ˆ`. «IΨ”‡»(η+mŸύ¦­}›If€Š—u=GΡRΝ;ŸuaΣ…ρΟx{ϋν·…}ΐτϊλ―‹)S¦£bLΩb‹-ŠΟζM=Ձ―oΙΔƒχᰝ—b6¦”†τγƒΉ³ιp(υΩgŸ™^Uέƒ –υσysΝάθσ`sν,ϊ\9«jφUEN{ŒŠFφŽ@VγbΦσŸQ£F•(zƒ2€ϋξ»Ο;¦QdΡΪω§½U‹2Ψeβsc!`·©JΦν8‡‹λ²κχβ£ΓeEWΪύ±]Ί<Υ^k½F©–~Ζ'Dΐ/Ψ«7χΑ»%—¦{φΩgΕ7ί|SτBψ!C†ŸγάdΥ7Η‘%Λ0rζœχ}ϋLΪWϋlΗYaΜ} _­#›tŸxβ‰’ŒμύΰZΜ)Ӟ—2Δ#«sKWφ>φ‰]ω”σ³λ?λσ}I±‰sv\ΛsW}φΩ΅ψ†]eΜΚsJ(7Φ°}τ‘ιΈχ92ϊίC6νJ#©_­φc}”΅–}uRάžx@ΑT€…gΥΫ°ΐ3‹wX³AŽΓεΐ‹Χ‘C±πΒ (I€4(g₯#D€"@ZŠ΅κrύυΧ·|~{4™ΌσΚZ—½AbΎοΪ΅«Ψ|σΝΕFm$Φ[o=Ρ­[7ƒhΌwΉ4€˜Μ…9Σ’ZX˜Fτ‡u9X…|ώωηΕΓ†Ηε—_π‹zXsΝ5C_ƒ)υΗE1―΅ΦZq’Pal7±3M!`½]MΡa•:νΓ³?όPΌρΖΒΥ‡Ω“υψνγ[@ͺ…ΐ xλΞ¦ΰ›mAdθΠ‘‘Υ„ {hš~λ­·α°8p Ψ`ƒ ΄Bv΅ΥV hœzηwq²~hίΎ} ΛJ-Yb‘o:”•Žδ 0ΐ@‘ΘuΧ]§Hƒ` ˜£χΪk/υŒC h—ΧB4a‚x:LΨբގ5τΩՎUQs”}~³gΟπϊι§Υύϊ*ہqB΄/½τ’θΤ©“ύΊψœvΪΎϋMΫϊ­-Έ_,XΔ ΄X—s΅¨ηr4%}ο».lz PcΠ Ao >Ž?ώψbPΣZ;<χάsO΅ζ)0nκ­ώ– (xΫΉœ¨1Ε#]_Χ ζՎ;¦’”_˜n“M6¦²(ΞƒMtβέgΡηrΝ―.|„J{ŒφA#ΣL¬ΗΕ,ζ?`TΈςΚ+Υή:φy°―ήΪ >\τνΫWΤJ`ύ§.«Ύ¦½U‹2θ²πژΨ υ*a¦si—uΏ‡&„Ιš4ϋcφ qk™αˆHך P ΰ€­τϋIόfέ7kΊσpm”3ηΌο+dqΆξ»g…±λ›―ζ,!ίY3Ρ`+-ΐό<¦«Υœ2Νy±Yž¨ϋ¬Ξ-£h¨υ»ZŸdUώΌ;ψμ³kυ gU—|ϊυλ'.Ήδ’ΐ+μAφκΥ+ΰ‡ίγqI†{δ}?Ά8ΨWW‚γ|#€yψ΄ΥWπΒΪ5όMY° Y π‚—α όj:πό-±ΔBΓyΒΊκͺUσšωπž"@ˆ¨Š΅°·-™―]ΪΓ&Ož`7ΓƒAΜα`βt9Lΐ|»(ΖTSγͺo:ς–ώθΡ£•uHlf˜Ξͺ1ύ]χQ«q;nXXLβ) ζBΎώύnΈα†@!Άί~{qξΉηόΚ=̘1CmΠ™mψζ›ov űƁ䒱sφkZ(€αM ΕΒΚ3¬Ji+Q–ΏξΒ /`ΰ7έώπ%_‰•3ίχΆ\₯uj3±Ορ]sLΏRπ­k‘X€¦- ΛΡθ΅Ϋm·έœβτϋ¨«έ_BC” ”³:•f’wΠb†όΰΐhh(@ŸχΣO?“ΓΑM>"PižΑCι%˜Z1'όε—_η{  α“ΈZφΝIθτΆQΜν1ͺ’9†|‘&φΐ°πν|·γ,1ζ>…οΦβ'ύ™3gϊjδ²λ»– Ψm)«΅²YκjηΕfZQχY[FΡPλw΅>Θͺόy;w°Ώ³4ΗE;νZ|ΓYΥ«ΞΒέˆ2ω~Ώϊκ+ύ:pυ=2Λψ‘φc+„}u%¨1Θ7‹.ΊΈT”ΥN‰ώΖ Δj*ΖKœ+i‘Xτργ +¬°BI±χ^€‡±F+θˆ D€ΖA`Ζ)J:%yε•WBr ;64<„Βb –I]Ξ\„»ή'ρƒ63hDqΉqγΖΉΌ~ΗsŒθ₯KΙ―wοށpυφ‹½ϋοΏUdC@ƞhλŸ|ςI}yE]ΓœΛA Yk§q½o?,4šΙAΫάsΟ=(ς°aΓ,>%ωA#',P›5\‡rφ¦fR!#,GUόέxγfΆ™έ2D΄lΩ²˜ζυυ‰'žί}χ]ρ„[Γϊ>Ϊ“MFΜ[nΉEΔˆΕ‘P-Ν€6uκΤΔδ@ΨnΚ”)xq.€D F@{§©‚ Z3·­Eα+u°2kχ'N¬4Ήͺざέwί]όύοW α € E•Ξ+IΫwΏioLVrΐχΤSO0r=䭞]4–σσ]όaΨ©Ž δΧΊuλŠ)Άv9X¬cp9MΈaΒΡai—K³ήή7Ϋ㦛n 0΅mΫV€ι―§­€κΈ` μ±Ητcρjo˜b3φν·ί.Ύ/wƒMΏ“O>Ήψ{λ­·ΚEρςXm·έvΕ΄a1ρ‘GQΟ6έΠ‘CΝΗ’{‘½χή{ΚcΗ…υύqβ¦Ζή”ΰƒDRΑXΤ+,μj­€°pNGςˆΐ‚ .¨΄kΪζΝ›'ξΉη1wξ\±T;lxoΆΩfϊ±’«έg&―‘ΑcδΘ‘κ‡ώΣ΄k‡ώ|€IϊQ)]π•vύζ†nX, n>ωδ“’ψ@λαΡΖqΎλ9 •†Ι’.\΄AΩ,1›Œα`5Φtq¬9δΉ|~§&NΌG€νόΏΨ”SΒτϋVϋl‡±Ο>ˆ²ΓD=γ0κ΄ΣN £”™Žσ`ψχΎϋ\;}9γΧM!Σ£Σ ‰i€‹@γb^ζ?£GmΪ΄)ˆ²ρ ¬©Š/3Έ±ϋ·΄Χ‡"€Ή…τkQWΉθΧΨs1(³zσΝ7c{‘P&[ΞeΡο•£Αυ> Ί|χΗμ\5K?"@|#€σ'[™%cΑΉ”^"|\—Eί—–¬Γ5™³=>₯½―Π’E‹@υ€ΪVΜ`<ΔU†nDI|›E;φ±YhξS˜hΤΗ=„μ/Έΰ‚±ΰλΥ«WΐO?ΨνΙΗZΩχΌX—%κšεΉe΅~WΛσ€¬Κn―ukΝcci‹vϊ>Ύα¬κ.n>6¨ΙΏ‘ΣΘb<Φyε隷ύΨ€Ψ°―NŠΓό#A[e•UŸ5b!λ€ρ |‚ΰ+@Ÿέ’EKe)gJˆήYπϚ5«XH„ϋφΫoΕ»οΎ«ββφ` «ΩψΦ‹ π†"@ˆ@"Oϊ§ n ρ<π€θΫ·o@© ϋ°οaAΡεμΕ΄摇 k‡σ α0f­φλ―Ώ#FŒ‹¦„MΒ„f7έtΣΠxυςbΉε–SB~ΥΠ;|ψpgt0ŸpΒ N«:΄ΰž~ϊιϊ±δ–vI@zΤ XdΩVV!ΨjΔΕ-Π»ξͺtfx—`?¬ΫŽ—_~Ή-ςήt©Tˆ72“˜/χΫoΏ@HγόόσΟͺΏ/ΊvνZΒ<―ίι+΄BΩύΫϊλ――_—½Ϊ‡τe#€ujjΥΖbK/M”Λ˜1cα!ΐ„ :"WlE`šΕhσ[†ΐ^\αφ°r³Ξ:Wθ/“ωΰΫ‚ μgœ‘~ Ϊι΄ΓάQΏΓ5IFΫϊ¦9τ™vύf=Xα0*Iί†›ΈΞw=Η₯£’pYΤE]{ν΅Wΰ•ŠΥWΌΔXb‡ DϊίCžλΐη·δΒ‚~₯°“rcJ)rΩψτξέ»ΔZ,˜D.Ήδ’Š ψΛ_ώR’XκΌσΞ+Y+q\ΔΎϋ\9+«—4cΩc―›υyΕ£Σ€‹i₯ƒ@γb^ζ?`tψΫίώŠ£φPS~πέ†‘›Φώ―UΒΚFϊF S§N%ΦU.»μ²Ψ…‚bΓ8–b³θχbmΜ‚.ίύ1ϋ£ByKˆ@¦ΨJτξΏ~Ε¨ EΣ¦³Γ™ο\χYτΝ|kνΧhgΞΎχ`Οvq”@g»Ϊι€ρœE;φ±χ)lDςϋ Ύ±–(©Εω\˜ @sJίσβ85’εΉezj¦–ηY•9oηΎϋμ,Ύα¬κ.n>q,Εf1Η₯7ΛpyۏMZvφΥIcx"P€ΧΌΌΰ„αœgŸ}V<ƒ"³ω―ΥW!…b—Ψ³ΦΒ―P¬ςΒ /(‹±P*Ϊ§žzJ1ζ·’u­λRIˆ D€Έhh‘Ψƒ>X 0 τ·ε–[Šξέ»+m!ΫoΏ}€κΰΑƒK΄ͺPL¨\ ‘Π0βr˜hEθ$ΌΠιGY<逓ΔB -€ƒ7ί|³60*VΗ`ιΕε )eο½χv½ͺ;Ώ#Ž8BΈ,Η-”lα$[o½΅³Lœ8Q@X9ΜZη6Ϋl#ϊχο―“jθkΨFz#ϊ™gžQ -³lΥ|KΛ,³ŒθΧ―Ÿ™œS˜υ― „»υΦ[Εwί}πs=ΐς¬©½ ·Ψb WΠLό €ϊlΰ[›9s¦φCΛX‰EΐvνΪ•τ‹°lΗAσΤ‘NΡ·›ξΪk― ΰ`Ύ³ο_}υUρτΣO½1FΨι_ς†δΝ7ί\ΐ¬vΨθΊβŠ+τ£ΊΪ‚³—1Ž?ώxΡ²eΛbθ &S˜ ψΒqƒωbMgόo°Αζkeα›sqΰ?όπΓbPh[k­΅ŠΟ>ΣΞ’ίD_dΧαE]ΠβW,¬uσϊλ―—΄+HΰΡw=2Kω!‹Ί#sSΡΖX0†•^ν°ώ2ΏUνo_σ\>Ώ%>»`;"ΘβF/_'ΩkΉ#ΏσΞ;ŠŸόa¦2dˆΟR˜ΛbNι{^V6Σ?ΛsK3ί<ήΧκ< +,ςvξΰ»ΟΞβΞͺξβζc·™={vIΤ,Ζγ’Ls⑧ύΨ€°―NŠΓϊ@`ι₯—–<Κ}o?ΎsΘb<όσŠB―Ί4ψπ‘όr°‹ύκ›nΊI`=FΰωΣϋΧƒ;φϊ°>P!•υŒΞGβς‡Φs9I; D V,P«Œ³ΘwάΈqJ˜ ] Ə_Vˆ![C½¦?L “wάQ1‰Ο;WMΈ>ύτS%(A\h΅ sΠbβr¦€†ύώΠCϊӟΔΘ‘#Ε)§œvΓ’‹™T§ρη?Y` z:κ(ΰuΧ]W@°ΧεW‡\―κΞ˜ΒϊL₯šh’9ΠΖV^ye5ρƁ(zφμ)Φ^{mρΡG9³EšηœsŽσ]#zb3―YœΝ Žo³W―^UίΆ` «zwέuWIšΰ6΅ύ‘‚ΰ54†9,ΡfM‡ΓX*΄„eMχΕ_x™ΔΓͺ.a΄Γ‘ωΡG­•ε›(₯: Κ` ’ΑΚ+―Τ―C―>ϊ¨€…^{b?› ψΒ}χj«­VΜ Ϊ­ ˆo2_7θΫΑ\i*`ffZFpή† vlŠλί*«¬²Τϋκ«―.ΖΣρ_~ωε€Μ%t}=ϋμ³!Ο γγξ»ο^,ΨόωσBϋΤKΓ’ύk¬‘ζCŌδ ζGoΌρ†ιUr~υ ƒRιτKl’ΩB± Ϊ]΅{χέwΪD‡qΔT€ωšiαΩgΪYυ›¨28@Ζ•0‡οㄆγ:ίυ—ŽJΒeU.ΪπξΉηžΕW9δβ3nl«V—ΖCžλΐη·d@ΐΫΨΞγ)zΥ§Oρχ|Π'μΏώJ‰ΦW_}xηz˜5k–;ν5˜’ΖFΞƒ]hFϋeΡηrΝ]qήV³vLsŒŽC+Γd‹@γbžζ?Xγ`νZ;Μυ1^„ν›λpi_³θ?]4§΅…΄kUWΉθΧ`/}†v؏ξΫ·―ˆRά …/ΨKΓy\—EΏ§ιH2ώfA—οώ8Λ>! ΆΊ>x%D ±°_~ωεΫο/C²θ›C²Žτ/ψ)Μψ8rxζμ{_ΑVΌsΑ(†ι°:%"πwdα²jΗΎ16±β>…‰FΎξqΎϊΪk―)Cλ­·ž€`¬ιΪΆm+.ΌπBΣ«δ>‹9₯οyqI‘B<²:· Ι>7ή΅<Θ „Ό;ψμ³³ψ†³ͺ·ΈωΨ–bΏώϋ’¨YΗ%ηΐ#Oϋ±•ΐΑΎΊΤ‡δΒφμΩCΐΨΦΥxΏŒQίYσ½ΆiΣZρ nΈα†η*°*‹ωνΨ±c †=ψΰƒβγ?V ΙΡΧuξάYςLoνδΞ7"€^͌3”υbπΠ "@ˆ€d‡ΫI3ς0cUόIς±γϊ (%‹τ˜΄₯u/'J)JώδΙ“ rΒIƒ΄βU ηΘ0&½rβεΜO2€ΛXιΘ‰\ iӦŎkbί£Ξε!N mσxΩqπ|Γ 7˜Αœχ’₯3.βK¦ gΙpλŒ#7oαΓ<·Ϊj+g:fYήύ°θ)ˆV6Ύ™VΤ½`ΝG )†ζsΗw„Ζ³_Hk^§#Ώœq9ζ;›ΐσΆΫnλŒ'5οΒΥκAj'Πwί}χ₯JŠ\”$H ΣN;­κ<€°PΑώξ€π3]iU5?ΪαškYxψα‡ί΅dd*ΘEaA ΛΒ/·άr…Ο?ά™Ά\lΒ"myPXxρΕ S¦L Δ“Μο%a₯p«3]—§<€)‰―Ώ©νΆΫΞΕι'7΅K‘Φ£ RΈ4^.’ ͺjAZ/,†—‹δ‚θ.>£+Clq|n»νΆ" )l­p·ϋM”KZΈ-΄oί>e±Η ›ŒyIάψΗ@~θλ}9yHΘλΉηžσ•UAnΊςΒχΧIΛ—Έ¨;yΈ+Ίάά)‰{ꩧƊ[i Ρ£Gς”υ•&UΟΗχέwί’0¦ϊέΦν«άθ6ƒ–άΫί‹Ώ%W~qόzτθΐ^jΆŒΝ[©Œ)@ζFRgU?sMΑvώΫή Ύ³°1\i;O£qΨcΉξ0nJΕ[jμΔψ-­ ПIλ…{ξΉ§ ›JθF\΄£8ϋφΈŽΈυ8ΞrEŸ›—5'ΪΆ-ΪHؚΝώZs&₯!ιΪΡ¦;Ν1ΪNΫΧ³TBKvήyg_YΥ$]yπ(Ϊαν·ί^-YŒ‹>η?φ^$φbΛ9©˜±Ώ3Ξ8£\΄Ψο㎟YτŸ.’Σꋐveˆ‹§«¬yπΓzί¨ώmΌρΖy +·4HMϋE¬4fR˜Ϊ{4χV±―Šs•-·ά²ή^·‡νfΡοΰ€γotωμQζ,ϊ„J°EΊ‚:―ΡίRρ.a!‘ΈΞ₯υ”Ππ΅~α:ί0Ϋ;φ*qYτΝ +Ι|Η5ΎHη•―$N#Ÿ9ϋάW°η{h{Λ/Ώ|σ}N*•έ°ŽΓ·Ι—cσGHεη%υ’=’΄Χ¬Ϊ±OŒΝςΰΎχ)€RΠβΌm䀓N²‹•‹η{ο½7@'hν₯K|Ra?©@]Ν+ΜΆmφΑΈ_uΥU RΑv¬2f1§τ9/ύvωΓφ³8·΄ιΙ+OF-ΞͺΕζψC ΓΦ hψΎΟμ²  †΅;Πγ³Ο->ω!@₯Ξ‚ΦJ“*Ζ³ω{zχξ]|gήd1ΫνΐΧχξs?Φ.CX[ΆΓω*+κ0‹ΎΪl+΅Έ—‚Tώ ΈƒwŽ42Σ§OWk6ΜAΆΨb‹ΒFmTΈφΪk Xk'Xžxβ‰Βn»νV« PNA*€(ώΐ ^πƒ'ϊ§Ÿfλ¨uyΏΕζ›o^8ζ tωFς6ΘUAJ+\|ρŊΗτηŸΞ7Ρ€Ž"ΰπάKλξi€°Ωf›nΉε–B΅}#¬έΕvΝ& A‘8ŒΖQœφ&š~†°„―τ³yS*Ϋε0™1Γ†έλ{3 iΡ4Vά°4₯΅–‚΄’f&Yro ηι΄β0΅Fαh wιŒΓ6ά’ ΕbCυ­ιu]£„b1ωΖdΪ/‰Ÿ΄pW"§ΛŠ«λpS§_―B±jYĚέϋŠu΄ΉΎΣJ°Ž!4a!—:t¨³‚ω @,]ίό0D9©•Ι™6Ϊ(J΅³7½π>j£WΗ3―8ΤΡmίΌΊΔΜxζ=΄μCL€΅ΔK€uƒ‚΄H«85σΠcΎWΣρΠG_zι₯f6κ>.6œŸθΠ4a£XjΑ*`“‡\Ά0!³(gΣγs“2ŠŽ8ο(₯δaς$ fJ0‚ιφk^ǍY8ϋ+J( A9‚«?DΩ©S§Β;μPšΕCŸΰΐ$Μ‘˜Μ›τγΒΥΨ CŸ‰ƒφαΓ‡€eη‹μ°&r9ŸigΥoB „]ν2—{Σ2„£ΝpQB±ΐΟg=»κ'Ž_¦™¬κ"Œήξέ»p֘£Ν&uΎλ ž.š}~Kόβψε](V·ƒj|p Άσί„2’ΖVi;/‚]Ε Φκ•τΩv‚ω+ΖλΈζΑYΟ΅}χΉ¨»<¬9A‡-Ϊ\’ugkΞJhH²vD9m—ζm§νγ™B±ρQΝb\τ9©„ Jƒ Η3 ”,Š=>’ΙΖΟ,ϊOνiτE:]ίe¨ε|D—±š«½wK‘Ψh4Ρ_„)Υϋ̘«Ω{u˜οαΜΝόΓ’³θχt)“ŒΏYΠε³?Φeφέ'θ|’`«γ4ϋJLΝo„B±Νή"’Λο:7Ξ³P,JΣ«W―@Χνgn•Ί,ϊfΠ–dΎγS(֞·ΓF:sφ΅―€=(TΣmΞΎ‚GσΫg56OJԚ$I;1Ϋ|VνyϊΒΨ,Ύ―·}ŠzŠ΅ΫngœyΊ”΄κΊt]}Ο)}Ξ‹“μΫeqniΣ“WžŒZœT‹M‘X΄sŸηvYπ–Ϋ³φΩgϋώ†]ύF?B±ϋμ³O`Œ_Ζ^Ϋe1ΫνΐΧχξs?Φ.CX[ΆΓω*+κ1‹ΎΪn/Y?S(6kΔ™_^€Ρ «Έκͺ« P²#)sη… ΡίHλ°JΩΖrμ=@€v“M6Q´Æ +άx㍊>/εͺ” ΕVŠ\φρζϟ_xε•W 03pΰΐΒΰΑƒ•‡0ήώμ)dŽD€Ϊ @‘Xξ>,ΕΆiΣ¦pΤQG&ΗΤ€ƒŠq 6„m2bγχΣO?UΞaaN?ύtGi j€t FΨι„|@0Φ΅™oΗ·ŸΑ€NJ]½ ΕμC9$°bc%‹ψ΄}Μ2vΌΈΟzΑδ<ΚΉ7uϊυ* a½<8ίB±vWZBηΊθλYgεL›@ΆεN'μŠ>Z8Λ9—υKfΪB±gžyfI™Αd•TS,ΐΪͺšfΧ²ΠΌ ν=0.«q±AΊφ¦ ςˆΪ‡ΦbXαrΡε!Y”‘œ³ιρΉIYŽ–rοmF;ZŠ-‡XΌχyŠΕG}tI{‡ζβr.©P,ƒuεΕ_Ό$Ώ¨o σ,lΒ…)τΠtN˜0A)"ˆJ+μ]Ÿ>}"η >ΣΞͺίNΠ”+εa8˜ώΠ€‡ΉΨα‡GΈΚg=λϊNrΛ4“e]Ψτ‡oΎω¦4Φ³Ο:ˆ‹§‹PŸί’+Ώr~Ν& <ΨΞγ)Υ΄σrν.ξ{0dΪVΏΜ~Ίά}Χ] /ΌπBάμTΈF˜Χbν³ΟEΕδaΝ :llΡ£ΦUˆcΊ4֜•Πdνh«οΣ£uΊΎŠM†lγ’―ωO%LX@–uν1{l`Œ­Φ%?}χŸς€Ρ™ιϊ,CRAη‘[―™―ŠmζΪO^vΧΉqή…b/ΏόrηΨqΕW$ΐˆ‘EߜdΎγS(ΆΡϜ}ξ+`ηήaσΫŠΰ°φ«ωΞ‡P,šsνωψΔι›ήφ)šQ(Κ‘τ‚6•8ίsJ_σβJφν|ž[Ϊτδ™'ν$Λσ€j±I*λσάΑ. Ζ–r{ΦΎϋlίίp%ύŠΝΓ•†₯ΨγŽ;.0–ϋO<ΡIžορΨnΎΎw{ΚΝβΈ8ϋ±vΒΪ²ΞWYΝrωμ«Ν|jqO‘ΨZ Ξψ δ•Τ^"δ’U₯#PKή—σBβ΅Χ^[nΉeΉ κ½άht’EZ@Œ§^Iζ !³*&Ωe—Uu{ΣM7 )œ;i)UHνkβΑ’‰8v<¬&Mš$€Ά’ΑR ]๚‡vΪIH¦’@”xΰY? ί“ΚT{“–a΅·σŠ4₯EkΥηHmIΞ0¦η‘‡*F%Π'ϊv{ο½·@YL'©Jp0ί»ξ₯EXρΞ;ο\£œ< (ί‡~(€uHύωΦ[o­ψΞ'6θ§οΎϋn!­! )΄[Μ3κF Τ iω[{μ±QΑψŽδ)d_BΧ{μQβ—†‡΄Δ-Ə/\yΊ—Φ™…d\R ‰(7O“–ΉUΪrNH…'δJό€uk!7K„ψŽœ3ψL;«~…ίe—]Τc‘ΤP-Πί™N ‹ή½{ © P<ώψγs1Ι”kΛ,³LΰΩυೞ]ω₯ε—e]Ψ4Λδ’qx½υΦλ»4Φs^λΐη· Rs΄,ζj.¨›₯»Κ^‰ί ¦L™"N>ωd±ΤRKΕNkΜcί}χ]էǎ(rœ­ίΒϊξsΉζό λJξͺ];¦έwURΖρ‡@σΏΌΝ°ΦΓΎ½ι°Η&…&L―Lξ}χŸB€΅₯ΣEtήΌ6ΨΛΐ™Ζy睹G’γ,lάΈqB –$"‹~%³ +‹ώ8‹>!)Ά‰"P—`~gŸiβ9ξ|X‘³θ›ΓςΞΏΜ}ξ+`{PrΐωB˜Γάε8oG8Ι€4U¬Ϊ±OŒm@ΈOa#RΫgπCI#bϋν·Ws`œ­aĈόl•8ίsΚ,ζΕq˝չe\zjηY•7oηΎϋlίίpVυV.Ÿ-ΆΨ’$ˆTF'€Pp‰VγqIΖ9ρΘΣ~l%°―5Ζ!ωCΰ‡fˆΧ_]HPρωηŸ i˜BρφƒΧ^Mqη€Ρ‹β\wȐ!η+؏FΏΉ‹$< ωC„Υ#RθKHŠt½?),[Ε!ΝD€\#π;ΘιΖ₯PjB S18Ο₯E¦β3o„ΒΒR;€ϊ^0Hk˜κ0§‘K_© EH-£β›oΎQΏ9sζ(‘ 0#HMζb‘E±£ž!\ϋ裏Š—^zI₯#5¨¨Ι"„ͺ€eX%8‹ΝPƒΡ%CυζνGyD¬θ:Βf„3πΫ`ƒ Δ6Ϋl#€ζΕd‰3΄:vμ(€ηbΪTFϋot‡…#Ϊι΄iΣϊ€φνۋΝ;«@q@R7cΖ !5v*aZ…!M€•w‘olvJ ΑBZ…Ÿ}φ™*ϋJ+­$6έtS!-±:…Φζϟ―ž!`Š `œΔψθrΎ±ΑΈπΤSO)†΄‰'ͺΎ4‚όzυκ%¬κΧE_½ϋ΅kΧN|χέwΕb@pPjΜ.>σ¦2€E% ―cχοί_ κηfΉbŒ–’…΄b'€~΅iωϊ |ϋΨDλ₯KEpL:UaŠ<τy mŒOšΉϊn₯Ά0g_•©Ο΄³θ7Ν²‘ŸΕα<Ύsτi_μΉ565o½υΦb40ΈtλΦ­ψ\ξΖg=—Λ»šχYΧhΕψe?ΪI+Jq~τšΧ:πω-ΕΕͺgϞj‘Γ£­C)H³8Άσϊ©iJAθσRΜ­±–šWΥ†΅0ϊp(;ΐzγ\ΩηΑ•b˜EŸΛ5ge΅SΝΪΡΧ]YI’cAAΜ)§œR ΄σΞ;+₯OεΟ€‘@IDATEή„"ΕΈ˜‡ωO(5~‘E黈P†41‚’Ž<°˜$ΞW0Ÿ‘‹‡ζzwήy§ΪΫΐ>3ζƒX³wκΤIμΆΫnJω‰fϊˆ—bi¨,ϊ½JΖί,θΚ’?φέ'T‚mi+hœΧ€1]»•W^YWθg^‰€‰ψ4le©8λƒh3»,ϊζfΖ7¬μ>ΦΘΛWΊP”gρ-AΚΫΐ»uuJ Γ°„–νΨΖΊ|υ΄O¦y(§ΥNRJΑυ3―ΡψžSf1/Ž.aπmη–ΑσωT‹σ€¬ΘγΉƒΟ>Ϋχ7œU½₯™O–γqšt3­ίh€Ύ βmήΛΙ“'‹1 ?ύ† οˆ@Ύ4Υ:†€…i΅FƒΒ‚΅Φκ" T,›xππW»';³ JkΈβπΓWΨuΤQb»νΆΛ0wf•~ψAάqΗb̘1ͺΎΠ&!Ό}Ψa‡ ΄iΘ“Π"@š( „¬εi§¦δ\>ψ`±ϋξ»—(ΣL‚ …b“ Ε°D€4,Ν*Ϋ°Κ‚5%ŠυSνŠυƒ+Sυƒ„¬ ΰ@»ύλ_%Œaϊ―•#­u ρΗU"­[·VΪq₯σ‡@³ ΕϊC֝2ΫΉϊ"oκ­ο’PlΎΫ©#Ν„…b›©ΆYV"o(›οϊΙuŠΝ["@lκmŸ‚B±v ςΉZxnY-‚ŒOˆπ@½τΥŠυί˜C>€q0­8γŒ3”²"(_6l˜R"WΞPX>Jΰ— ΕϊΕ7ΝΤ)›&šL‹FBΐ‡Pl|΅„$ΛBˆ D€"@ˆ@ξ€ukhFΣΗγjξ›5kVΐŠ¦ΆF¨Σβ5=nΉε–’@,Rέwί}bΣΓ—)εΆσ|Τ© D μ»’αΕΠD€"@ˆ D€"@ˆ€?ΈOα[¦œ <·ΜgζBˆ¨φΥΥ ΗΈD ̘1CΐzωΟ?¬„b!8³θ’‹J+±-ςG,)"D€"@rmoη’H D€"@ˆ   /Όpρ7zτθΨ <ωδ“›‘Ϊν΄ΣNbΈάΡx€y½μ²ΛΙ ><πΜ"Π°7B-² D ω`ίΥ|uΞ"@ˆ D€"@ˆΘ+ά§Θk͐Έπά2.R Gˆ¨μ«k‡=s&>€0μ?ώXLΊmΫΆF\pΑ’oˆ D€"`"@.q ή"@ˆ D€"P3Θϋ‚ .ΠμYΞΝ™3GsΜ1`»μ²Kΰ™ι πτΣO‹ργΗΫf›mDΧ]‹ΟΌ!€Ϋy#Τ"Λ@šφ]ΝWη,1 D€"@ˆ D€Ό"ΐ}ŠΌΦ ιJ‚Ο-“ Ε°D€Ϊ ΐΎΊ6Έ3W"ΰ EYD,±ΔβwΏϋϊΝ›7OYυ•Σ%D€"@κ ΕΦ²D€"@ˆ D !€€₯ι¦OŸ.FŽ)°Ιζ>όs±ΥV[‰I“&ƒ¬½φΪΚ―θΑ›Š˜={vΐϊξδΙ“Ε°aΓipΒ g>zC€νΌήjŒτ"Ψw±"@ˆ D€"@ˆ yA€ϋy© ‘6<·LQ¦GˆHφΥιcΚ‰@-hΣ¦θ₯‹² :Ύϊκ+ρα‡ŠoΏύΆ–d1o"@ˆ D Η,”cΪH D€"@ˆ M„@=Ď;ξ(ξ½χήb©Ο=χ\qχέw‹‘C‡ŠŽ;ŠUVYE̚5K|ρΕβΡGχέwŸ€₯Xν 5p̘1’E‹Ϊ‹Χ ;v¬>|Έ€ρwί}'>ώψcQ(Š©mΎωζb‹-Ά(>σ†Τ#lηυXk€™φ]lD€"@ˆ D€"@ˆ@^ΰ>E^j‚t€Ο-ΣF”ι"@G€}uϊ˜2E"PKZ΅j%zφμ)ϊχο―xΒ~όρGρδ“OŠΕ[LτνΫW]Zh!ŸΆ&‹k—4|’΄–"@ˆΘ Εf9s$D€"@ˆ Dΐ6o»ν6±νΆΫŠgžy¦β“O>§œrJρ9웣Χ_½θΦ­[Xϊ'D`ζΜ™βε—_.‰‘γΡ£G—ψΣƒΤ#lηυXk€™φ]lD€"@ˆ D€"@ˆ@^ΰ>E^j‚t€‰Ο-ΣD“i"@ό ΐΎΪL•Τe—]V 8PL7nœ²;wξ\1yςdeHaωε—K/½΄x—β ΉΒβ ~ά8΅Δy"@ˆ ρ Pl<œŠ"@ˆ D€ €₯WX=φΨcΕM7έ°•}χξέ•…Ψ]»Fγ»XqΕΕ?ωO±ξΊλ¦“ ωD€ν<ŸυBͺˆˆF€}W4>|Kˆ D€"@ˆ D€d‡χ)²Γš9ωC€η–ώ°eΚD€΄`_’L‡δ―ΎϊJ ΒB€uΡE3fΜ'NΣ¦MνΪ΅š]b‰%ΔΒ /\΄‡ςόη?’sηΞ†Plΐ†"@ˆ@ @‘Ψ:©(’Iˆ D€"@šΦ­[‹+―ΌRŒ5J]yδ1uκTρΕ_ˆ_~ωEΑ°Ψb‹‰φνΫ‹ˆ!C†ˆή½{sΣ2ε₯K±Γ;(܁χΪk―­~»οΎ»Xf™eR΍ΙΪ ΐv^ά™+ Υ!ΐΎ«:ό›"@ˆ D€"@ˆHξS€‡%SΚ'<·Μg½*"@ˆ€‰ϋj ށϊF`Κ”)ʈ¬ΐΚ5KV^gϞ­c§OŸψΓ’Z{…΅Ω>}ϊˆύφΫO,°ΐυ ©'D€"@ŠP(Άoˆ D€"@ˆΘmΫΆ#GŽT?Π­}Ѝ‹/ΎxžHmHZΦ[o=qο½χ6dΩX(" `;ΧHπJˆ@=!ΐΎ«žj‹΄"@ˆ D€"@ˆhlΈOΡΨυΛύ†Ο-Γ‚wD€Ό"ΐΎ:―5CΊˆ@|ζΜ™# ψ K°Άΰ+„cρƒΣΧΈ)ϟ?_ρΕ ΟpD€"@ˆ@} @‘Ψϊ¨'RIˆ D€"@š…ZH¬ΈβŠM D€"@ˆ D€"@ˆ D€"@ˆ D vπά²vΨ3g"@ˆ@\ΨWΗEŠαˆ@~hΣ¦θΪ΅«hΩ²eͺDΝ›7OtθΠ‘DΠ6ΥL˜ D€"9ŠΝrfHˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€t˜9s¦˜6mšX`IP¦‹­HoΑTΒͺ‹,²ˆX|ρΕE«V­RΛC'Τ±cG1bĈTιGΪΏός‹hם*ƒΞ‹W"@ˆ D ώ PlύΧ!K@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€4)Ÿ~ϊ©ΈξΊλD‹-RAΒ°Š…@,\l±ΕDλΦm€€i[±άrΛ‰•VZI¬ΌςΚbα…N%?€Ω―_ΏT²A9π£#D€"@ Ε6N]²$D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ@“!πυΧ_‹Η<–ΧB‘ Π)'(ͺ…IMk±°»ΔKˆ]»ŠM7έTτθΡC ΙV 7ςZh!Š·T‹#γ"@ˆh8kh–šf9‰ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€"@ˆ D€†D@ Ήώη?ZπU_ρΒ­ψi‡wΏώϊ« ‹+œNWόΖώΑzμgŸ}&>ωδρΥW_‰Αƒ‹₯–ZJ'λε ~ωε—"-ΘB΄Ίψ €&J:u ψρ"P@[‘ι:θ ΅ 2ύxOˆ@Ύΐf‡ι† ’6cL?ή'GI¦{ρΕ93α= D #¦M›ΘιΘ#§žzjΐD€"@κ ¬›ξ±ΗγZΓ„χD€d†ΐΜ™3y½ύφΫμˆπ¬C‘ι`aƒŽ„!`Ο§nƒ 6°BGˆ Ι°ΗέΛ/Ώ\ά~ϋνΙb "@ˆ D€€Œ€Άάi&ϋε—_Š:˜^Ό' ΥV[Mμ»οΎηn“&MRU[΅j%ΦXc Ρ³gOΡ±cG±Μ2ˈ֭[„b! ΨΙ“'‹7ήxCΕ… ,„J»tι"ϊτι£ωηŸ•@μτιΣΕψργΕΔ‰όp}πΑΥήzΧ]S« Ύ‚.μΫƒ6΄}Θ―@Ώ₯—^Z γBΨΒ³οΏΎxξΉηΔͺ«*Φ_}ΡΉsgΑ^:"@ˆ D 1H$‹I‹ι0y€Ι{:"@ˆ@£!€… υΐ_|QίΘ)υΨLβό/§•C²ˆh*Ύώϊk D€FAλ\k4Jm²D ΎΐYϋ£ϊCROœΕ?{ηEΥ…α#„4R $!zo›€"(  ½I"X(** "¨¨€ ‚’‚"(ι½C¨‘„„ ½‚=—†™Νn²›l6ν;Ο³Μ½wn}—έΜΞάο˜"`μΗεΛ—MUG9€€€…Ψq°‘σ` »@uΘ3ΕΖp4XΆcǎΙΘ―,ˆmίΎ½ŒΰZ·n]rww—bXvͺυΠCΙΊa•ο3π«ΈΈ8)žύϋοΏiηΝ2B+‹R}||¨vνΪ²>Χε&‰φΧ_₯HjHHˆlS½zu«8νb‘ξ‘#Ghοή½R¨ΛB\Ž«D±ε·—λ±ρ:ψήώΊuλ€P–#ΨrΤά¦M›J°¬„@@@Š‹D±EjεX €€€€€€€€€€€€€€€€€€€€€€€rΡΡΡRΚ‚P‰Φ―_ŸHuκΤQ#Γ["‹d9*,‹j½½½ΙΓΓC ]wμΨAάΧƍeXww!¦%)2υσσ#έ²@φΜ™3RP»{χn0`9;;«’[cγeWΖ}nίΎΦ]+£ΏΖΗΗKα. yK”(!›se^#ΏΨ8ΟληH²\ŸΣμθ†Σ;w–s’ρ€€€@‘!pͺ Θ, βA€E‘½uσζΝΔQTY΄ϊ裏Κ―vvvfC`Ρ)‹hόq*Wœ•²ΨυΔ‰B€zOν‡…΄…΅U«VRD{ηΞΊvν]Ώ~]ͺͺ•-H$%%Ρ–-[hωςε2BlBB•.]š*T¨@…ΆJ•*Rૈa•yύ...TΆlYyžηΓ‘fωεΪ·oŸΥ*uq(,ŠΛZ+γβD_½ŠΆi(”ΖMϋš’oΕͺs?Ό5ͺ]UΝ# Pπ Œ~g%%§ͺ}ϋŁT#¨’šG"gώώχ-]³Ym\;Έ2Mz‘ŸšG@@ΐ6&ΊˆFD«ƒ½8ΰIjΩΈΆšG@@ ΥvΡͺv¨Σn\§*{Ά—šG@lE`ϋΎc΄`εzuΈ*•|iκΨ‘j °c!—θ“+Υα<==Υ4 `HΐΝΝΝ°ˆζOKΎ™ΚQ  =/—ώA{ŽœV+vοΨ‚ϊtm§ζ‘ό"v'FLš₯ž…Š00$’’Jηϟ§ππp₯΅FΤ°aCΦ°nvyΡΦ­[—*UͺDtγΖ :|ψ0΅lΩRΤΚ}888P£Fθί₯ΘΘHJMM₯«W―Κ¨²Ωaμ|FF†Œ:»jΥ*)π½{χ.Ή»»Λˆ·΅jΥ’ΐΐ@)Ό]Όx±Œd«νƒEΊ-Z΄ „„D!¦=(η‘œœL!!!΄fΝͺZ΅ωϋϋε*‚­v<€A@@ςŸ€E’XΓ+φv₯¨K›¦ωΏ Μ@rIΰ­ίλz`A,ΎίtHO€―K’4³lVΏ΅hXKS‚dN\Ή©kVΦ£ ΎuDΫψθλ庁κΥ¨‚οcd@@ Sη.λ¦μ]ΞΫtD°›·γuCΉ»ΊΰϋHG[°+©tΟ0` `Š€½½}¦S›7$OwΧLε(ȞΐοΒy—ΦͺTτΕο-€A@@ς@rjZ¦±qΟ ))ΙRΛBR₯JΙΘͺ>>>9fΓQWΉ= d₯•E«ά·Φ8"-Χeγh­·oί–G©Zj±±±΄uλ)ŒεH―^^^ΤΊukκΡ£‡θ²ΉƒJ‘―a€XΎWΒB^W«V•Φ]KG%Ζ?~\D‹έ+˜zθ!b1lR’6dΙƒžœœœε8,RεWZZš<>¨a~κΚ•+΄sη.Ω‡““5oޜž~ϊijΠ *fΝJΤΚ‘k+WL,žεzεφΪ΅kC{χξ₯]»χ ΣˆΉ•Ln ₯τ»τΠθk g.ΦίIK’Kaρ"ͺςƒVιιtπdE$\$έ‰U2ƒΛΆ cW{κά²"Uτ-cF+T(Κ Š-Κο.Φ                       Pd °ˆ•E°JUŽΨŸc(‹[oήΌ)±,0eΑ©1KOΏ##Γς9Π*BZcu³*γρ.^ΌH‘‘‘B3X‚*V¬H;v”Ρ_³Βλ³L™2ΤͺU+Ϊ³gEEEΙH·W―^₯[·nε˜‡±qŠJYBΚ]:uα&₯ήΉK%Δ{ˁŒτŠΏ•*?ά#ΝBΞ°πDŠI‹γάα₯{τ•ss€ζ +δ²'4(  Š- ο"Φ                       Pμ8::‘···Ε²8υμΩ³ςΕeœ·Δξέ»G΅•_–±^^^R¬jΨΟυλΧ)!!AŽΑbΦ²eΛZ<χΙ}pTWΗ²ΆzυκT³fM)²5Ӝ<Ο£^½z΄cΗ)ŠML‚D1ΦΟΟ/Gσ3gΜΒZ‡w”(ρΠύ—…W λšσrή –Ζ0rYIyΞ²Οc^Ξ΅0φ-芈ΖBl\'9ƒX€&(·ΥϋF‡                        yDΐΩΩ‰*GIe»tιmΫΆnάΈ‘Fr5ghΔ²ΠuΛ–-!EΆnnnT­Z΅L’X°9rDF”eAlι₯e„WN[j)))+«νΦΗΗ‡<<<,νF­ΟΒΪςεΛ«‘kYά›””€žG@@@ π@€ΨΒb                       ŐΐύθͺΥ¨Q£F΄wο^uηΝ2Βk»vνΘίί_ŠVΉ‹V•θ±,@e!lFF†ŒΚΡaYLΛ/ͺΪΫΫS`` ŒΊΚmΉ> LωάργΗiΧ]tλΦ-M6 €|}}3‰gΝy;x|ξW1·T©άI]x<_~±)kVΖΐ@@@ pΘέ•Bα^;f                       …š@Ŋ©kΧtώόyŠŽŽ–QbύυW5ΆAƒT©R%ςφφ!WWUpΚΡ^’(&&Š._Ύ,…G₯¨¨()&-WuξάYD‘ lRSΣθΜ™Σ2B,‹oϞ=+Ε¬•΅uλΦδββ’#†vvvδθθ(Ηd1kBB%''K!oN:Ό{χdΐG6₯csBm@@@ `€(Ά`Ύ/˜€€€€€€€€€€€€€€€€€€€€€€€dK€E₯-Z΄ cǎΡƍιφνΫA±±±tβΔ ΕΥΫΫ›œ]DX;)@eQlR‹bcθϊυλtσζM–£«²Π΅cǎԩS'Uœ{›6oήBkΧώ)ϋεθ,8­S§΅jΥ*GQbyaΞΞΞδιι©ρκΥ«©+SOš‘`AνΉsηdτ[ΞlάέέΝh‰*   ……D±…εΒυΤS”‘‘A{φμ‘ΡbYτΚ"Ρππp*Y²$FK½wορ‹£ͺ²ΦΑΑAD”υ¦φνΫΣΣO?M>>>κH·nέ’3²―R₯JI±iυκΥι‰'ž ΐΐ@΅ž₯ Ž0Λ‘nœœθΝ;RΠΚβ^.³4ϊ,―#Ψ?~œRSSeT\ψrΤ[€€€@Ρ!QlΡy/±bJ f͚ԯ_?)Rέ½{7]»vβγγ₯0–#»²πUΖrΊD‰R8κκκJnnn²]Σ¦M©M›6 £Θ"SοηηGT­Z5jΩ²%5oή\DŸ΅ΧΥ΅$ΓΫ   ͺU«9r„’££iΣ¦MT¦LΩ?Ο‹η¬ΜΫTί tαΒZ½z΅\7―#Π6nάX nM΅C9€€€@α#Qlα{Ο0cΠ°³³£5jΘ―,=wξ?^ŠDoήΌIiii22,7βΘ±,fe+ `Ή~ΥͺU)88X dY«5©6k֌4h £ΈrݜDsΥφ©€yόφν;ΘωrdΫ£GΚ¨±,Δ­_ΏΎœObb’:wΌrt[ΛuX{ζΜΪ°a8p@–³Ψ–#ΩvθΠAŠy•±p(ό Š-όο!V            V%”B[φ‘Σ―PΤΝXŠ―ψΔdrsu&χ2.TΑ»,5«WƒšŠ—«³£UΗ. 1ΏKΧ"(TΌψx)μ₯§ί₯JΚSe?oy τχ!―’°\¬@@@@@@@@@ `1+GHm€‰ŒζzγΖ bAll¬ΈΟ+’ΖήΉsGΞΦΑΑ\\\ΘέݝΌΌΌΘΧΧWζYLjΜ*T¨@]Ίt‘‘e9z+·7ΞkgNGͺmΣ¦΅΄ξ§C‡QJJ cu]μZ1‡œKλʐ‚F`ωŸ[hμ΄ωκ΄&ŒκC―xFΝ#Ε…ΐ_ΫφΡβί7Ў')]l*ΚΞxcQ·φӘΑ=©QνͺΩU—η‹ς5#_kΏ7χGZ½i—Y,šΦ­N― }ŠΊ΄i‚MYfC%œΈ{7χΗsm@@@@@@ `q+‹cωΕ‘U322dTU”rž£Κ²°•λ™#uvv&~ε•V‘^½zIΡξ±cΗ€ 6$$„BCCe4[—Ε²α–£Ζ[·ŽJ—.-…Ύ·nέ’€€$I–ΧζηηG:u’/S"ίΌZϊΘ{Εζ=γ|Ά0Ύp„€@a%ΐ›”8bΆωξTΖΓ@@ΐΊbγiΑΚutϊΒΊp%\FcI»“N₯ΔΝΠΏςT©Υ ͺDΟυνFήε<¬;8zBBΰxΠΕΧΑŠU©θ£$s}δ{ΪΎΉCά7Θ5Vt  `|—[{O·½mC(η.‡Ρ€OΡφύΗ-šnώά²WΎzwiC³&=Ÿ­C”’xΝΘBίΉ‹§/–‘Τ΄ϋQΜyΰΔYόϊΗT½JEzeHOκΫ΅½9ΝP@@²$°xˆpr±‘.^χΘ―FΠ­Έςp‘ήΛ—₯ΪΑ•©“¨uγ:fmxΞr #'Ο^ΊFΏmάiδLξŠψ#ϋνdΦ’_θŽΞ<Œ6…~>ΤN£@@,&u“: ~]m·ό³·¨a­`5€€δ½²8”#ΓTsp°§–-[Κ=½–#Εr”[ŽόΚ/6^Ώ8B,G…εϋΝJ§Y$[©R%Ρ–E±-   Pτ@[τήS¬@@@@Š-m£)Ÿ/¦›±ρ™°μ’ΨόΓ― ;Ρ’_ΦΛ¨O£ϋu#{αυ      Ε•ΐ_[χΡΘ·f›6+F«ώήAΗB.Ρo_ΎGΎ^žYU΅θ vg~»RmSΑ»,M;TΝηw"Eb;›@,Κ©qΫ—ή›G»Ÿ¦Ω“FΛH9ν ν@@Š/3―ŠˆεKhσž#™ ܎K$~:…Vί.Hϊό©―R#+ tΞ_ΉN³­Κ4~n μ팊bγ’θ£o~ΞUχm›Φƒ(6WΡ@@ΐΐovRΜνΟ¬9j; @@ςƒ€β 4·Ž¬YtZ’D ›,ΑΥΥ•ΪΆmKεΛ—§ƒώύϋ₯0666VF‚εh·Κz”°φφφ2‚-· ’¦M›Jq­υ„Ϋdρ@@@ΐlŚ A@@@@ *ŽΫά4ϊχΐ ³§˜˜œJοΟϋ‘–―έJλN'7Wg³Ϋ’"€€€€€€Ώ‹HnΟOωœ22ξeZRύUθρΆSγ:U©|Ywrqr€°ΘΊz=’.‡Gꍻ€σ)mΓσ—―ΣΘΙ³hΝό©"κ@Iν©§oŠΔ«7νRΫW τ/P’XvΠe(ˆ-Y²υξ–κT­L•|Λ“Ώ―—ˆ ›FΜηΒ•pyάuψ$ρύ ­-ϋc39Ψ•’™žΣ#   Ω8yξ2uώŠOΌ9'»μ@ς‰QoΡΜ7GΡ ²«^`Ο_ΊQ`熉€_Ώώ³£ψ.+Θw,Mχ"oίΎM‘‘‘” £ͺ²@VΙZ2InS\9jΨ°‘͜ω999QέΊuΙίߟκΥ«G.\ kΧQTT”ŒΛQby^,Š΅ΑΚ”)CήήήT΅jUͺV­±–ΛXΜ (š Š-šο+V    ŊΐG_/Ο$ˆεΝ§Α~Ϋ½—‡]vy“O؍h›s‘a4ζƒ/hΙΜ Ίrd@@rF €πϋxΫ¦ΊΖ₯JZG ‘λ\Ψ"’ȍ~ησL‘κT«LΣΗ§–jg#ΐΟ›ZύΏό}€³©™ WPDΤ-΅ξΎc!τΑ—Kι}Ρ\‹5γ?;χΏώ£]Ϊ4‘kͺTAWΞ™fυkͺeWΓ£θ…w?'ζ₯΅οD½oKΧ«‘-F@@L`aθ3―LΝ$ˆuqv€¦u«S…ςeE”ΨβH²‘a7Τ~ξ€ί₯W§Ν'/jί¬ΎZ^˜Ϊυ¦yc  E—ΐŠuΫθψΩΠ’»@¬ @@ @`Aμ­[·θΤ©Stϊτi:{φ¬Μ§€€LjςeιξάΉC7–βΤ’yψμ_‰ώͺYYμΚ‘_=<<(88˜“„ –_ΙRτΛλα±₯K—&GGG)–…°iVιΓ΅’>€€€@α!Qlαy―,ž©£ƒ=U©θ›m»ΔδŠΊk΄ž«³yyΊ=§-twuΡf‘›Ψsδ4}Ήμέx kΡΌ)/S JΊrΞ¬ΫΎŸ&~²Β£nͺηΦmΫO V¬£Q}»ͺeH€€δŒ@iq?βΗO'ζ¬1Z€€€،@\B½ςαW™±ύžθ@ŸΏυ‚Y9μΰž¨‹pŠy؝#*ώ­ή¬AMκΪξαLk**׌ΌIkάτωΊυ΅lX‹–|2Jg1ΩY₯ ειΟo> Yί­’ίΠUŸσΓoτΣμΙΊ2d@@@ΐΡοΜ‘θ[qΊΣγ‡χ¦1ƒz«‹“ZΞ»6ν>LcΕ5€vŸΘKοΝ₯—FeέΛ¨usš`§/‹qsk«ώώ—nΔάV»έ―›šΦ&B "ΕΞ‘oKΫΫi«d›φυςΜΆ*€€€€9ώΪΊΖ~π•9UQ@@ς„G…]Ώ~=mΩ²…\ΉBqqqΔ’Vζ$J,O255•ΌΌΌrάήά…²p—―žβώκCͺ°•Ε±©–_lΌe=,εD°ζRF=(: Š-:οe¦•4­WφϊE¦rΓ‚ί6μ€ηήώΜ°Xζ{=֚>8Ϊθ9‚€€€€@A πξά%€x δωT­μGλM'»RΖξπf\φŒίΊί«t36^]ΒW?ύQ¬J ’N`ςμοθFτƒθΌ^ΈΞžτΌΕˆΨΑκO³'Ργ#&SRJͺŠξύyKŒŠbΥ …¬¬N ύξ]“ «†A@@@²$ΐΧf'Ξ†κκLχlΆΧkΌYwϊkΓ‰½η+v-"Zzηs0(ΚB…xdεΊνΊ%V π£^a± Vι€Vp½9ͺ±σ*Ε.^ +ΠίG)*RGΓ{uͺRο²9ZγΈgŸΦ‰bΣξ€KSλΖurԁ€3¬Τ-Άc‹†F±ΪJΝuμΠ§ιƒ/—ͺΕΏoάES^”γk΅£\&B.^₯WE$[ΕjW  yS^69―KβzC±ΰ€ JGΘS}³œΎώi­Ξ1TžˆΞA@@ΐ ‘‘‘΄nέ:bqι]ήο."¬ΊΉΉ‘ΏΏΏŒ²κδδ$¦–FUεH³Υ«WΟ$N5cJU £₯K—Rpp°―jΥͺRDδθθ1¬EDQ@@Š6ˆb‹φϋ›/«KIM“^ό ΞΖ‹OL’ωβΖЁg₯‡΄hαyτψΪoΙΓΝUVα¨]λΆνWͺλŽm›Φ₯?o]™afΗΑt9,°XτοBOthž©άX{3_Ώ}?±gΥC'Ο ‘D,%&§’Siςρς€ŠΎ^τxΫ¦ΤMτη+ς0Λ \ΉIΗΟ^’ WΒ©z•ŠfG `ΤώγgιZxE‰Ο¦½ψρ^ΡΗKF¬θoωDΠ@@ PΉxψo‚b%K– ΦMΜΫ,ϊ¨π†oh‡N£.mš# ŠžζληϋƒJI Δ5p …ƒ―…ω7kQ²c!—θ\h]QΈœD”¬†΅‚©^*…bEν½ŠχnXψΑžξοfdΘ{3Υ*ϋSVNοέ»wޏؠyσ6Ή89’w9jVΏ&•/λ^”ώ«b- P€ °0‹£](Žd\]œΘ[|†›Τ­žλϋ’ΚχΔή£!"ΪβmΊŸHΌ‰7Ύ³ΐ²Έœ[oίύχ…\ΊFα‘7)&6ŽΚΊ—!οrT₯’―Ό§dΝ7χ¬¬I}ε%EΏ¬'ώlhmΦ€ΡTΪ!wήώϋvk/Δ5Λδ΅‚χ¦]‡iTίJ6_ΦώŒž8wY·ž:U+λς–dψϊ¨TΙ’:v|ύe‰Y{}¦ΖΆΥ8<~NŸ!˜š;ΚA@ ¨8|κ<ρKk/|R›5™."ΙN›ρ59[؍h:xβ5­Wέd›Ό>‘ "ם0“’ΕΎ6Gqm²`Ϊxr°·39th˜Vλg²N€€€€5 DDέ‚ ֚@Ρ€€@ dˆη·gϞ₯]»v§9Ίj₯J•¨cǎT»vm(ΦΑψ^2 jsiΦάρxήqqqB/^$Ή†5jP`` Θϊψψlrqq6ι<ΙάρP@@@ p€(ΆpΏrφ I)4nϊ|£sSD±+Χo§·fG·γuυ΄ϋ/xƒ•©~M-[Q쏫7Ρovκϊη o€ΚNΛΥ?όΎ‘>]ΈRF 3μ„ΐπP~mίœ&~Ίˆϊ=с8*™›«³auδA X¨υψŠKH’ |Κy‘Υχ½ϊƊ Žo}φ=­ψk›Κ§}³ϊيb/_ΏAμνxύφ””¬ΆΥ&ψsήOlΎ&δZKΐoZφ«Γ›“l_¦ζ³K4ο=†‰Ɋm_6;ˍγJ=A@²&ΐΫ΅V‘|Y³7΅σF~χ2.Δ“KO °UΚp‚B€–Ÿ}}»β/bq’1c…λ½W†d{mm¬}ve?όΆAώžΧΦλΤ²‘Ψœ8NύώείΜνŒΧV‘ν?Ν6zmΎώί4eΞjέO>Bγžν%σk6ν¦ΉK~'Ε‹£j‹λώί!Ε•†ησ;oνχΚΏGΨ1X΅ΞΓTteΔίΙ3'σ<ώg?ό*~ΏmΧ9£P*wnՈή~q °UVŠ„ηα ϊbιjZ΄κoβΝA†Ζοα#ΝЌ7Ff{Η°-ς Φ%`ψωoT»*ύωΝrθ[q4ύλŸhυΖέFοΓπߝ– kΡγ†Q½κU,šXjΪϊβΗ5΄`ε:b‰¦ΜY8DxiPwωχΑNlΰ°ΔnΗ%Π狧Uμ Ρ™Ώ‹Ό<έhpΟΞ4Ί_7)”΅€oΓΊωuΟΚpΘƒ€9ψZrΩ[tUk‡…-ΥΦ•ε$SΞÍoΧ”ώά²WmΎG E±ζ\3Ξωα7ϊD<§a»wO/ΰe§)~­ϋ©ct€͟ϊΰώ­zβ‰ΌϊŒ†‰gYZ»x5\›΅(Ν‚δ^]ڐ6ڝ]©’fυ‘Wλ3γ›LΞ₯ ηΗZفӐ7fΠ ž€)/"KD†}*ω…"šΨΔO*Yy즉NΛ…<‡Π°ΊzJΔ]‘Θ$Š‹ΪΊŠ(κ½ΉK„ rau5Ού8JݟŸB“Fχ§±CŸΚΥΓ;΅c+$ςβ½²Υονo–Τ΄ϋΏίΆξ=JΟNϊTΎW¦πlΡίv:E›–Μ”U#Δίγ!oΞ€#§/˜j"£Π°@¦ΓΰΧiŜ·σuΓ­ΙIβ#Ϊϝ߿ωkΫ>zuΪWYώναοό]‡OΡcΟN’yS^’ή]ښEνψΩK4T|O°cΔμŒE»μ vŸΏσ5ΓΝ±Ν»Π˜ΎΡgMGYdΡοlq―‰ονώ4{²9έ­“Ÿχ¬ŒN…  £g.dΊκΡ)›VζŸώώγ7²­lΞ5#;0Υ~?vͺ=—.r˜²ΌόŒrTkZ`δC"R?grqr|PhAκΛwΗXPϋ~ΥΌ\Ÿv2y5Ž5Ÿ!hη‹4€»ΕυΈΦΪ7k :.Σ–›J·j\G'ŠεϋΣΗΟ—ϋ,Ώ¬—ψwˆbμ„k`χŽJΦθQ%–+T @€X£ P  `u£ϊtN‘6Ϊ/?ϋxαέύžC!€€δ””ŠŒŒ”ΟaYΛ’Ρ6mΪGV-,Ζs4h>}š._ΎL,πMLL”Ρcω&²*T €€ ¦zυκ‘――/Ή»»‹θ±.…e©˜'€€€€”°BθΜ&ΐ€Η~πUΔ²€’‡ΨPΛ›2sb‘bƒΥ Χ?¦s—Γrm@ XˆQ9z½<Υ"A,oxηM˜}σsΆ‚X-Δ+α‘ττKοΡOκ£hλ   PΈ p$­edά£έ‡Ok‹L¦άΌGwŽ£€WφχΦ•! ΐΙs—ιΕχζf%q€M/O∞ΖlώO“£ί!εΦΎωym&Al·φΝθϋoX΄Ρ2»y°8bΒΜY b΅}πχώ‡_-£Ύ―~(#“jΟεGΊ ΌWΦ\χŽƒ'hΐψ²Δ*γq΄·g'~*ΕgύΖMΛR«΄αc|b2ž2GsR΄ΕHƒδ3?Δ΅2¦oΗ%š5“τ»wiά΄ωtώΚυlλο?"ξΑΎkRΛΫE΄DC;sρ*u>IF7ϊšΒnθ£€”σ(CU*VΡωŒ{€šσΓo΄tΝfή ωΐξΤ fΥ )žε<θ‘Έι¬0Α@Š ²ξe¨eΓZ΄ϋΘ!μ+"Τ¦Ε3Ι·Ό>Š¬vΝ„ΣΤ/~ΤѐžrΙEΧ2  `E,.’tj£`±`αυΟΠΓυͺ{xe»— …Τ½c £γπ}§‘f]!ΠZύU¨³ψ>jT»*ωy—₯gCιΐ‰³΄\|άIΏK1·γισΕΏi›d›Ζ=«l‘B%pΐ@ΛΧ]Υ*Ό¨jέ;΅€κw†u.4Œ¦½\%ZAόζŸώΪp]^Νό?a‹ΟhύAΔNΆ΄ί9| ήiθέ κώH ςςt3œšUςΆXOΤVγ(Prς Ai‹#€7g.^ΛδΘͺžΈξ΅ΔψχΊ‘E݊%χ2Ά3eξbυΊŸησl―Η̊ϊzιZ„:}ήψXρώσζ1·ιΜ…+tJΌNŸΏ"λΤͺ(Ÿ+Χ ͺ$~”SΫ!     E&²··—«Ψ΅««+±΄0―ΑΣΣSΎjΥͺE5’ππpΊxρ";wŽ.]Ί$σ·oί¦€€$Ίv회Λ"ِΏΏ?q»š5k’···Œ’λζζFό, 6#πPIρNμ₯ωOόΏSλ• ‡J”Y΅ΐfΣΑ@  P” @[”ίέΈΆ=±BA›G0έiηΘUYPT©ρ†¬=bγGŸδ†vωz$})6kM~ΎΏα)δA Xˆ7ˆ“lΌΡκύ±CιΙΝ©|Yw“\ψεη‹םηˆTsήz<άξo’δ“,B―S­2 {ϊQκσx;8~:ρfL6ŽυϊŒoiίͺydW ϊ$ό EˆΐμΙ/P»γUΑX€πdίeΔdš4Ί=σx[U0ΖKζk·ΕΏo”‚Xεο—»8•Ξ•`Œϋ€€@^Ψ°σ i7χ΅jT›VΜy;“•―›Τu₯₯³& qθzύγoΤιΜ[Ί†žλۍ\MD”U+IΜ]ς»iO=Σ₯-}ρξΛΊοWνωά€A¬§XΟ_O₯b“’Φ8Ο/ΛrDΏγBЀ،+Δχ~»|sP•ίο•ΒΑZGΓςΛή½υΒzaΐ“ͺWaƒE²=…P₯η οGoT,!)Y&=ά\hΞδ©[‡fΚ)yδΓόώuϊ&±ΈE±΅[χB«ΐΐς™@lόƒθ°Ο AΧ{c†θώξ°@­}³ϊ4ΰΙGθρ“ˆ―ΏΫΈλόξ0uελεkιΖο qήάΐSψ^Ž‘±“›±CŸ’ή]ڈ¨ΥΣιΤ7²s=v„ΣFD΄ήRΪΟ[²:“0€Nχ‘΄ΞjW­Lύžθ@»w”"ڈ¨ϋχ¬”~²;βžUv„pΎ Έ£› Cg+Ίωœ©৊aφΈ?p†ΕΣrqv€'Δ½œΞήͺ@IDATeSf«Ο(ίηžςς LNdψ~ΒΜ4ρ“…Τ@ˆ“Ϊ7o@|ΔΧP.NŽ¦¦mvΉ­Φg«q”…ηδ‚G(Žψ{ΣЌ]#ΦΡζ݌ˆ_cnΕ ‡ώΪjyšώχΐ ι,K„ŸϋNΥWΙfyΤή7 }ΨyΧ›βoπΎc!YΆkέΈ}:q4!²l–˜p@@@@@ prr"///u_{JJ ₯ g₯ŽŽΉΏ™μμμˆ#Ης«vνΪ-±!!!tώόy •e·nέ’ΔΔD%–ΛOœ8AΎΎΎ2Jn½zυ(00P dΛ—//Y@ ›οfρ³„ΏΪ9–₯’eθήέd)΅sς$;‘Ώ―’Ν¬=)^„°Z°Φλ =€exΓo β Oα†7X98ΨYΦ‰kΉlΙήζ½σ’άXΜ8"o–ϊόνιŸο?&ή¨kΜ8j Gꁁ<  ˆΘ*xΣ¦fΠρ™ΚJΛ-§}΅Œ’RξGŠβ;‘ͺ<2D Εβο vC![9χςQΕa  PΠ Ή€›€ξθ=Ί“Ο ξΡQDΛφTO%Šh³'Ο_Vσζ&>ϋώΧL‚Ψώβχπ—οΙsΡΔ2!ξ5ΔjηΝQύβ]]Τ’Ϋq‰4λ»UΪj6Mηη{•— }eΘSτ :A¬2Gl3€§’Υ?χl&A¬RΑ±΄ƒ§)y>ξ=vF}@«-G@ pτfώ,k…€ΪΩpΩΣΖk‹ˆ―³œΎ +S21·γθsαlAkσίΕ¨ V[‡#8ρ½ gΗj1;QX ’UZ”θ.QΗ΅ΖbXŽ&ij9vΛ’OuS΄νM₯qΟΚ”·5βwžoyOΣN ΓzŒΝΡ–ŸΡv—"~cσΰ{βGΞ\$ΎΆξ=fͺΌ/ρΘΰΧiβ§ ι· ;‰£[ηΔl΅>[£0ΘΙ3₯-Ž  P Δ&YΈ’«}      …•‹_d„X^ΓΥ«W)""Ό°.G7o^[₯J•¨I“¦τδ“O!ChΤ¨QτΜ3ΟP»vν€h–#Γ²σǘ˜)Œέ²e -[ΆŒ.\H?ό3νΫ·Ο£uT‘Ι+ Œƒ³/ΉxΥ!G·*δδQ\ΛΥ‘βY Εζvτ  PL @[Lίψό^6GΈΌm)νϊy}%6>MQΌxƒ•vc“-ηΘJΆμ9jtHήπΫ_Μט5ͺL3ήiμ”ŒF°fσ£ηPŝΐμΙ£³άΰπ9~φύόΧ6%+E³„Η^sŒ½ϋ>+D·Z[½i—6‹4€€@"πxΫ¦΄gε\ιΌD»,H±GxώpψΤyR’Χ)uΪ6­G;—Ο‘‡λΥPŠpEΰ˜Ψ@―΅ξeο1’τ νω¨Ψ<θ’Ύ £iϋ4–ζ Σζ€;5Έg'š+œF±“«Ό΄ν¦¦υͺg;G>ymxo]½+Φ‰οϊ]™­2ωυ^εεϊ|„ΓS’Weάζυk*IυΨΌAMκΫ΅½š7–¨ +ΞΘΈ'œ‹eήΜ««„ €€Νpd¦‰ΟυΛvΌ k‘ΧG άΐX΄*.dα/ΊυͺR―ΗΪπ©l­²Ÿ=Χ―›ήχΏώ£ΛsfώOoͺW¬²Ÿ·Œ–δM½<έθ%!j3ΧpΟΚ\R¨WP h#Bσ]] g€S|σγ3ϊΞKƒ€0ΦΑ>kη―χξέ£γgCiαΚυτά۟QƒξΟS“§^€·?ϋ^Šs222L-K-·Υϊl5ŽΊ0MΒάgš&H‚€@±$`θθ‚ssKο!i-κV¬6›§iΎΧ [«Tτ₯ΟθŸρf5P]1v΄Μk΅Ζ‚_SϋPξ€ί₯ί ~―NΛΤNΫ      P”Ώ‚‚‚¨nέΊ"2εC2šκργΗιΝ;…aϊfΝ‘ –-[–ͺW―N-[Ά€ž={ˆ#hπΰΑΤ­[7YΖ‘a9jnrr²Œ&{ΰΐZ½z5ύσΟ?ψνgeTΚ=‡Θt*γݐ<ό[›οΓδμYJΪ9ˆ³ίσ“ϋρΡ€y»‹±ψpΔJ- 0ΌΧc2Κ*{-(Άϋπ)“SaΑnVφTηΦ\©‚Ρ*;ž4ZŽB(ΞX΄Τξαϊf!ψzωZg¦"βU€ΨΜh υ΅Ζ’¨Τ΄’σ_»6€A@ˆN]ΈBgΔΛ;y.”ΦmίGζl:΅€_Τk0ŒjΗNcΜ‰^ρϊΘgθόΖΕκ«w—ΆfOι£o–Λ Ϊ#„Γ™Ω“ž—Ο΄εy‘Ρϋq³»e§[Z‘nϊέ»b#εƒΝfwd…Šωρ^YaΪYvΑ4SG•†΅ΧΠΊΆ{Ψ°(Sή·|Yέ{Ηbbγ2ΥC€@ώΰ(Ρ,ŒΝΞxSE­ΰJΊj7ctyΞp俟 "Έ}ϊΡLυ²*θΩ©₯ξtτ­8:¦–ρψού[Νsbά³½LFˆΥU™‘}ΊJg†εΖςΈgeŒ Κ €h­Œ³“6[θΣωυeaμΡ?Ύ‘ί=MΫ\»|=’xΞ=_x—ZχGmΫ—eS[­ΟVγ.Φ’g†m‘βF ΦΐΉ”»ΈΥ\†Χώ1βZΫΖΧπs~ψU7Τ{― ΙΉVWA“αφ‘":¬‘U τ§Y“FΣ©u )|ΧΟteϋ2ρ7ϊkϊiφdjί,σsκœ /–1μy(t8RlΧ]‰£¦FFF¦M›θθΡc”šzί‰?―)*ζΰΰ@εΛ——Qb9ZlŸ>}¨oίΎΤΌysςςς’Qcy­wΕ‚¨¨(57B·Ή¨pΐ: "ρ9{¨„ΖΊ“ƒ«ŸxωPI{½C²‚8kΜ @ #ˆb γ»VˆηΜχ?~c€M6ΡZ‚‰…rΖ,¨’/ωy—3vJWΦ¨vU]^Ιμ>bZl«ΤΑŠ.Bki76r›Ζ΅«™ΫTΦ«Vٟ[cΈPKž!ΆE@Šψ€dέ’=ΚRːΒF &&†Ξž=+#¨–+WN EwνΪ%ΑΑΑ2‚¬››•.]Ϊ’ύόβ7“ΏΏ?uκΤIόζΚ_ω ^“““ιζΝ›tεΚΉΆΠΠPβGΗeΑο;wδ[ΗΒ`www9w D Caϋ?†ω‚€€@A"ΏW%‰ζbU*ϊΪdK‰K4Ϊδzd ­\Ώέθ9s oΖΖ›mΦάώP ;?37U³¨<1)E·άž/ΎK% C ιjdΞΔDύ‹Ί—ΉJ@@ -΅[χ’_Φλζp½΄`ΪΈLΧ`bƒ;FΰΧΠ§ΞO>§OͺmήqΦmίo‘'z΅1  G8*ήwΏNΓ'~ͺΖή»wφ9-_<4GΓkΡ°–Ιvlِ,ύύ½ηΘϊχΐ £«xϋ³οι‘ζ ‰£^ε΅ωy›'FΞ£œΑFΝΠ0½πJ[7/ΣΆz―ςr †}W4σχ›a;·œE¦1μyό#ΰ$6CXΣB―ι#8εδϋžηγει›Vlάƒ¨΄†NΜφk;4§ ξYi‰!]X °(VkW4bsmyaLΤΟ(;©θ/_ƒzt’h9Βυ¦έ‡ι· ;ιπ©σ™pΏ;w ±v„£˜­Φg«q”uiζ>CΠΆA@Š+ςe=tK7x.ͺ;™EFλ@’«•s/“EmλœϊvΕ_RΘͺτΦ°V05SMɚulVΏ&mZό@ΨjV£WκΦ‘υ{’ύΌv«ΪμΨ™Kt'=μE!€€€€€F, ]°`ρύΘψψx*Y²€Œ{νΪ5Ίqγ†Œ˜Κe– Q[΄hA:tΘQ,G³e±+G…ε5^Ύ|Y=rτ[Ι²–²ΰΧΛΛ‹*T¨@5jΤ ΐΐ@ ’*UͺH–υA@@ .ˆb ξ{S$gζY@7EΖ&ΕZγMHI½οmΖ}‘( 2s\ΟT“7βδΦDτf€€@Ρ!0w‰>’‘›«3-š>ž|Λ—Νr‘Ύ^žB8;žΪOΪΏ/“DΤΩΗΫ6•7‡³μ'A@ΐ†ΊΆ{˜–|2ήœΉ€ΒnD9956ο9"_4‹DD*_zΆWΤ££Y‘ψ²Š$r[8’βοΗ…βϋ5―ΝέU/Π0gΨ―;‘M憁΅Κ~ήj Γθ³Ρ–GΏ™}›’vΟJˆD±"ΠJ\ΣΪξΓ§¬&ŠέΈϋ°{~ΞβμhέMXΊ4[~Fη.ω.]‹PGοΩ©΅oV_Ν[šΰhΨŽ–½φ±Ϊ4Q8[Όp5œͺUφ—eΆZŸ­ΖQŠ€€@Žx 1©‘ρ}’Νu²αyΓΌα}>_V8”ΜK[τΛz]χύŸx„μET[›‹“£xΰEΧ"’Υ‘ ϊ¨'B@ R₯J4xπ`«GE刬+V€’Βb^‹a8@ǎ“QaΓΒΒ(::š’’’d€[Ž ΛBWOOO)†εˆ°Mš4‘€€ba,GŠ…6/ί!τ   ‹D±λύΐl4ΨsΈ)»›‘aκ”Z~+.QMg—p7±aΆR…ςτρλ#³kŽσ y@ ¨R]―χ¨A*fGΣ5ΞηLbrŠψAŽ¨Ρωό6`x"H@9…—goWŠ ~d·lŸrΔβ©ΫškΗs—Γ¨cΛ†Ω5ΕyΘWN₯¨s«ΖςΕaρλΆ}ΗhΓCτηζ=Δ’YΕΨ Τ›3ΘΘSJ™α‘ΏC|8žΊuh&O9ŠώgΌ9’ϊ½:M­ΚB€wη.‘Ω“ŸWΛ¬0ζ°*»1 7o²γ«‚dΦ~―,]~XJ υAς‚@`E}τΙ0‡ζŽiΉP»ΩίΠωG½΄ΤΜΩoψ›£0ί³²”κυkI‘j’ΖΉθ狧~έΪηzSoJΪl Šmχp=›Α³εgtοΡ3΄aη!umBΜ“Q,wTΏz΅?%q<δ’*Š΅Υϊl5Ž²FA@rFΐΛ3³xuρ³‰bCΓnθg‘¨±~u•r‘9sρ*ν>rZΧΓӏΆεm™I‘u΅fψ»B{i(θXΪ―_Ώ<™¦ΈYJ8ςΞK;ώ<-\ΈX O|Ώ™r±—…°εΛ—'ώQpp°Γ’££c^N }ƒ€€P% θΌ0- χ2¦#¬ςfί¬,CˆfC.]ΝͺŠξ\@…΄'ŽœΎ Ν"  `Cό°΅ŒA*φˆ_νΜσΏ γϊ0gΘ/w 6«°SŽ,n©Ή»κΕSMLeιzP@ xpsu¦Z—οŽ‘“λdڌψΤy£‘dZΝΤT±JY§–¨{ΗJV—¬ήH»D±Ό2C«9γFυ τΟ:"9}ζeάΎW–Ξ ΏG,%†ϊ yAΐpcΉ6“Ήγρ=α«αQΊκΪH±ΦÜΏCEιž•&2Ŋ@©R%©mΣΊΊ5sΔΣί7ξ•ε$Γ}DߊΣ55KwΚ[~F9²«ΦΨΙVnΝ·|Y*ν`o²[­ΟVγ˜\(N€€˜E€£«6¬¬«{ΰxˆ.Ÿ]f@υ±6M²k’«σ«ώή‘k_5ΐjW­¬+3's;ŽB„ΐVy™s-oΨoBb2±8­U¬¨Ν"      …Š€E•€Φ~ΉΊκο‡ζŽ {θΠ!Š‹‹“bXςσσ£ΖΣO'p΄σ@@¬E`·ŸnΩ{TνkϋfΤΘ`££zR“(γβLK?HU; ₯ρ=ΗΖΏ‘Yθ`ρIΣΜhrΪψαr‰I)κωqΣζΣφe³rt]­vb"qπΔ9JcΉ:›ηΙ5<ς&8ͺλγΫΪςϊ½Βο[Ώ£@ΐΪ‚*VΠuy64Lά{’ŠΎεuεYeώά²‡ξ gˆŠq”σ B<¦XPE½SΎg{LDW¬_£ŠR%Λcϊέ»2κz–•ώ² έ³2gΎ¨ΖΌ<Έ­χ€ξΤlq/υιG[S‰–;œβŽΨaιŒoWθϊd‡ jWΥ•εuΖVŸΡzΥiΕ_ΫΤεμ;B‘1·Ι»œ‡Zfi"6>‘ψ^…Φ ϋ³Υϊl5Žv­Hƒ€XN ΟγνtΟS £°fΧγΏϋυΟ1»΄iš]“\_·mŸ}ΟΞ9‹»]Μ{τ;sΤΎψ·Α±?Ώ‘›¦ΥΒlΗΟιο)ρ³δΰύo—lΊΐi+ΰίe,μυφφ–Qa+W,£ΒΦ]›|}}ΙΛΛ‹œ-ϊνΗΣ㈳l9½χ-γG gO΅ ά20‘’HΐΩ±4nϊTΦωΧΆύd*Z,oˆ}ι½ΉJU³Ž­Χ1YoΤ۟oˆ2eηΔf½ΗPΛΎc3½¦Ο_nͺΚAΜ P3¨’Φw«ώVœκN˜Θ°Ψ}ϊΧΛiΞΏΙΧζέ‡sΌqΏœ‡^xΕCnΎ76 ζέͺΈk¬Κ@@,'` Š{Ψuθ€Ežδ•Ÿά#ΝV―βoQ¨  yEΰΆΨ―\Λςq‘»8ΊTΝΰέΤΣMΆΥUΤd|½λΡΦ³fΪVŸΡGZ4ΤMϋŽΈήώvEΞωqg›Δ=lC 6ΈΎ΅Υϊl5Žαz‘°ŒΐSΆ’χ΅•V!—Ρ.αhΝ;|ϊ9sQ­κβTšςΩGU?εΊ:':·j€Λ››1άγΑΧςμ ΒcηΚZ«θŸeΔvm]€A@@@@@¬OΐΓΓƒZ΄hAέ»w—Qa‡J=zτ f͚Q`` qδXΚkΌ7))‰Ξ;G§N’vΜm‹z   ŸD±=*Φ3¬S­²Ρυ‡έˆ¦ώγ¦Sτ­8υ|ڝtΪsδ4υ}υCΊ!ίMDρδœΐ+Cžm˜Ί ’D±yYήIO—‚Xmεή]Ϊj³₯μν¨Œ‹“Νρ³—tyc™_ώώ—Nx6Ve  –xΈ^ έΖzξaΪόŸL:O1o|~ψΥ2]q­ ²·³Σ•! ωE aΝ`έΠ[χ3ϋ! Gι⍆Š9•v ͺ•ύ”¬EΗΟtΙeοΛeΘθ{udf叿ω™ψz>;»OK~ί¨«Φ½cK]ήV™Ό~―π{ΔVο$ΖΘ+ΌAaά³½tέΈf%ˆHζζΨαSηiΗΑpΨ™ΝΈaϊώψ»ς•!=uέ±Π•£6fgόΫΰ‹ΧdWM=_ξY©“Br@ΰΝQ}3΅zkφχ΄όΟ-™Κ³+XφΗfš΅h•Z‡ζ h`χŽΊ2[dlυ­ΰ—) ξWβ:yϋώc9Z&_ί²“G­5­[|5Q±ωœ­Φg«q΄λE@@ΐrμH«C³Ί†Ÿgf|›•ρώŠχ>_¬«©eγ<½?n%ΦΥΩ)Σ='έ„²Θp$uΓ{]?΅5‹ϊS8AΫΔ½6­½1²6‹4€€€€€(±±±tνΪ5έλAΠ§δδdΊzυͺŽa›œδΉΟ¨¨¨lkζXPP½πΒ Τ»wojΧΥ¬YSFŒΥ:I5wŒθθhΪ³gύφΫo΄pαBϊγ?ˆχQΐ@@@ŠREg)XIQ$πXλ&tDx'5fϋ‡PΝ.Γ©²Ÿ7•σt£γ!—Δ¦YΛ£ή(}OxŸΨXυΆ’Υ—―έ*½sχμ܊‚DΔ‡£Šρ¦*~XdΜκV €ΗΫ55v e f¨@/|’>_ό»Ϊβν9?PejP3H-3L$§¦Ρ3Ύ•Βuε Zs#Šε~xL~@ͺΨW?ύA]Ϊ6%CωΚyφΒόζΜJΗ"Dΰω)shσž#κŠθψΪoΥ|V‰ΕΏoΘ$ΒϋiΦdjZ―zVΝδΉΛΧoPηatυ^Τ“Ζ}JW† ₯J•€Ύ]Ϋ‹(,©Kf§)/Ώ?ΎxwL&Α¬ZI$xCЌoWˆͺΗ΅Ε4εεAΊ<2  Ÿ*x—%©υFτ-9ΣΠb!φτ£ΩN‹Η&&₯¨υκVΤEΣSO˜‘ΰLŸNM>;Q}ΘΕώΖ~ψ%mϊa&ρχ±5ν΅q=Ώ€>›ό<•(aܟ[xδMψΪG”’vGš£γΎ’OΧEΆx―π{D}«‘(€FυιJί­ϊ[½_/±OΏό>-l2‹ˆ­,σ¬ˆtŎ΅›ϊϋ<ގΔ}aC{q`wϊAόΊ·3Uόθρόϊeή;TΡ·Όau™gG c¦~)οχ­`€° έ³22E€Yš7¨Io½0@:˜6σΑ—΄ύΐqš:v•/λ=•)˜œBοˆ{Ά?ή€;Η‘ζψzΞZζ(œΌh―‘ω{ΑXT[~Fί3˜Ί‹οΕΕ&΄!oΜ€™FW›ŸRW{x5O½¨š'@ ˆ?Ό7­\Ώ]έΜΘ›žxξmšρΖHω`ΕΙQ]1{q:pβ½ϊαWΔQe΅6³έΐ₯­o,έλ±6:QμνΈDκ%6oςeφΨ_EˆζY(αΚu±Š#ž(b}?οrt=2ΖX·(+„SˆίΕRˆ1”2SΗ΄΄t][Η›εΜ±ŒŒ{™Ϊ¦8 Š3 Ου₯_ΩAME±υ –}^‘©―£vΧΣm°ηοι½GOΣ_.££g.*Mδ±ˆ£Ψΐ@@  ˜ ’w›>_λ#yΟΗ― WαψΕΠβ’€ΰΘ0ΒΤσύŸ0¬jQΎa­`ή» -ϊe½ΪξδΉΛτΕΥτͺA€>΅B.iŒ―ŸyύZη!,ΊΨuθL"oΖκF`±•ˆ’_–Χο~δΧ;‹qA¬E€#ΉΎσ@±ωό3΅KvˆΨeψ$ϊκύWˆΦπF|Εβ“hρ³ς>φ78on7Œ:«΄aΡoΚι½yJ‘ΌGτψΘ·hΞ[/ΠΓΒ!o’gc1έΉΠ0šπΙBbΡGVggkζXAΊgeΞ|QLΰΟ;%a§*Z[υχZ»uŸˆ¬iiiiΤ²eKzζ™g¬Ωm¦ΎΜu2¨mΘBΨψψx!—Ε°'Ož€S§NI!lBBράω₯uΎͺm4€€€@α%π`‡Iα]f^„ pτEΓΝζ,7PD‘δΘ9οΞ]bNu΅Ξ‡γž•‚ ήμ‘Sγ[_½χ U 4ύΐ(§}£GΎ₯ι“ ΟIοϊŠ‡+Žς1V_ωşχκβσ!Δ²!"rˆ±θΝ#ϋ<.EφΉεΗށ?ϋώWŸ7€)-Mmœ|uΨΣs;ΣΖ²άΞνA@HFƒ]ϊιDzfμΊˆˆ,”ύΞ‰ˆ£…σί‹ΫbƒΞ΅ˆh£79½EδΎ„€@A#0¨GGZ±n›τŸQ§Άβ―mτΗ¦έ$64Wς£Κώή/Δ°WΕwo~δλe­ μή‘ž|€…Ά(Gι·E±΅Bτ £~²πq­έœͺψε¨OΓFό[:L¬ƒIΫφ“/ώŽ(Dμ\δ”ˆ–«ό.ΠΆeΑG9ΛOΛλχ ΏGςσέΕΨ Φ"πTηΦ΄μ-ςϋ]ι“v’U{»RT;8€…Σ1ŽŽ~VVmP˜:v¨tL¦΄7_œΙΣivύptΪο>z]F$Λ.Ξƒ˜O K›¦τΣμI4b,b/ϋZγ‡·¦ΰr½—„Wγ·EkG₯ύξγΧ©ϋσStΒ+₯oc‘Dz‹HΌ9₯χη)Υp°2Ž Έϊ«χι =ρˆAτWŠ£Η Ρ{ ΧNα‘ hΦΔΡrcΆi(ψwꜷ^€―MΧE½HŽΤΚ―¬Œ7ΣOmxVUΜ>Η‘i§N#ߚ­Άa§4γ¦Ν§?Ώωΐ*ΨαΝ‡―£ΑoΜPή°W+ΔU‚£‚σuz©R% OΩ4ŸΧο~ΨτνΔ` yH`Ω¬I4rς¬L"»;ιwευΌ±kzžΟNyyeύœλρCޜI[χΝ΄’ WΓeτXν vtψωΫ/‘₯›ς Κ=+νZœΰΟ?iΣ€.Mϊt‘Ω“ Ηb'Ÿ m³ϊ5 OY%B'iςμοΜξΛVŸQ"-˜6^:’yoήΊ•iŽ|ύ~=2&SΉa‹PG<Σ…&=ί_=Ϋ°ηm΅>[cl(0ŸGLυ‹wΕuπ  ξ!N8Sγ—1kΣ€-ωd’Œo켡ʝ<―λŠ~²7·ΖQpyΘθ·?3zθJx$ρΛ”Mx/½1²©Σ(C lΩ²TΉreΨΣΞΞN7?ggg Κ“H±ΎΎΎVΩ ›°³Ψ5..Ž\ΉBgΜ‘ .‹aoάΈ!Ο±˜λρ³" ;99‘››ωϋϋS‹-ςuώ,UA@@Μ$Q¬™ P-8–v ŸηΌEσϊ“Ύ]±Ξ€PΥΥΩ‰žλΧ•Ζ λE₯…ΗmΊοψ3Gοή±unՈζ.YMσ~\)ʎa§φv4Όwl/αeΤΥπ4ς V ΐ‘Ÿv,Œ¦~±”Vo{ά7Φ}Cρ•7=’‘±Σ9.«W½ νψi6›ώ΅.ͺ‰a‡Όμ½1CθΡ֍ε)C\†υ‘Θή<³ρμέ \επ―Θ‘\‚" Θ!(*žx§iYv™•εΟΚ£S³΄L»ΤL­ΜLΝL³²M»ΜΌ5οϋδ9UnΟwϊο8³»ΐξ²»,πyzm;Η3ΟΜΌΧewgžοσύfυ~Νω|5]ˆOͺ°Ažƒ»8“  [ΰΞ}ό]xΡͺ τΙΚ΅ώFεsρχρ’‡Ύ«Yoμ Ψ›Ύ}›*Ȉ³Ψ~΅v=#~›£πwψ?Ώœ-۞ŠŠ+³Ι@‘=–Ώsσox[)–~­π{ΔV^i PΎ–ʝΦΧόΉƒf-ωžˆΑ**<ΘΓ―=KνC›UTUZ/]Sž&-]ύ}ΈμΗrόψΞR‘U‹Ζ^LŸ9«ΌΪΚ5«ςŽλ `¨ΐƒο A}#θ‹5ŠŸ7–<£έg`?κ!vo?iΔ}νυζšφ±{€AίbN„²φaΝχ(/½ϋΆίžω+‘Š²Α*ΉMσ@zκ‘»ιQq‚³ΙZ¬u~Φڏ‘ηz€  _€Ώ/oύζ#šΉυ}?χΦ-žnZcΨ Vΰ ΫDF¨σ±—(φeŠ‰O¦„ΛiδΫΨ“ZωKŸ­‚›J«1 ‰•^μ¦ύž˜Dg.ΔΛνpΰΖΠA}εyLԁK—―ˆΑYNŠ ‘εBz€HY0ό|Q3_ ςχ‘ΐ&τ€ψNΜAΆ^~ήΈ‹^˜ώ©|˜ θA+?˜,Οο;v–ώΪyPϊ͚ž)ϋ††4₯Φ!bšVι΄)Œ‘–~­π{ΔΘΥmNΰγ?5iΚ½·w£oηNΡΜβΉ–δξΓ§hσΏGDφΦΛ’σz6ScΟδΥ°E΄•γkΏ¦–Μμ\ϊM ž³γΐ ‘₯1ƒŠDFZώmΠ¬ii`ΈεΓΝQlεš•9Ξ₯6΅ρύo[iΒ¬%ς)wnΫ’ώYω<_[' ňϊ‡OE‰χθiρϋ:2DpIfNžt†οz‰χi1@Υb@“fM}­ΚΔχm’ΕoύΈΔ)›*Ώ§ΫΆ 0°ΖšοΡ«YΉtτμŠ»”B1—’₯λ<•›'}§ πmLώΎ^ΔΟβZr·πΦ•6΄ΦωYk?•©† lέ{Œ†½2K>ς€€JHHη1₯@jj*ωϊͺώFύσ55τΐΞJ§Ϊ<}9ν*ύψηvιšbbJΊτΔύ‚ό|Δgxˆ4CMμγΐχω»?WˆIψο3ΨN|ίoθαNω7†ψάε{]Ύβή2 ”cήY@k7ν–½2ϊa)pZ^€ @€ PEΧσ (°ογͺ½ΗΕΕω‘Τ.ϊΤΔjΞ\;H•ϋ·sΏ)KΎ—bΝ>±………”““CW\‘cǎ™3g₯`XΎ^–——'YhΒ`؁³δ6lؐڴiCaaa"Βr0―““ΩΩΩY‚Εjm?~œ&L˜ &Oš4‰ξ»ο>³νϋ\μUZόύ Κ/,&ώ [Έ)a'ύP;j‚λ;ΆόZαΨ  -ΐίQΟ?OΣ§O—>ίnjC>ϊ¨”έ]»‘σΦΓΠ#C=θΰ/Τ]Ϋ·’zV[d‘»« υθΨFzXdh0Z Xtφαρ€‡Ρ›q<ύπ]flMA€€9ψμΐή]€‡9ΪC€lI  ‰7r§-’E…9©Xϊ΅Βο‹ώΣCγ€€•xCKwηpF=t—τ°τiΩΚ5+KŸ'Ϊ―ŽΤ«s[ιakgΜχm8x™ΖkΎG9(³«Z³Xλό¬΅kΪa_€j’@“Ζ ι•ΡΤΔS+χœ\κΧ£ϋnο^n¬„ @€ P8¨³’ΐNξg―(kŽs-š^»vMΚΐjFυύfggSJJ %&&J3111σεΛ—‰CSψ88(–3ΒΆnݚZΆl)e…ε`Ψΐΐ@ruuΥTΕ3 @5TA±5τ…ΕiA€ @€ @€ @€ @€ @€ Pσ ˆ3¬ΊΉΉ™εd9θ”³²rV7V½ύφΫ-t{ύϊuΚΘȐ²ΖEFF©S§ˆ3"'%%IϋΥdΏε `GGG)–Ο³¨¨ˆ:wξL?ώΈΛ™b-lL4@ΰxΠώϋ_Ρΰ94Z₯Š­R~μ€ @€ @ΐ3­²D³ͺ6oοή‘ϊF΄W-Γ  @€ @€ @€ @ΐΪΙΙΙRkΏ~ύΘΩΩΉR»η@TJ=qβmάΈ‘<==ιΆΫn3[Π)1εζζJYa9φάΉs%eˆMMM₯7nΧΡdΏυππ ___ι8ψήo“&MΘΗΗ‡Ϊ΄i#gJJ ;wN €MHH žOOO—‚_yž_y[ΞvΛΗ~αB4mΫΆ‚ƒƒ¨qγΖδνν--·ϊ a‡€ X]A±V'Η!@€ @€ @€ @€ @€ @€@Εœ1•ƒX9Τ”΄iSκΧ―]Ήr…N:EEEEtμΨ1Z»v-έ{ο½δλλ[f³ΌO'''jΨ°‘ZfEWp «›››τ¦V­ZQϞ=ι₯KuAȞ“¦/_Ύ,τrP,?μννιτιS”‘‘N~~~ΤΌys)σ,Ÿ Οs,²Ηšψ’`3@€@50*(φμΩ³ͺSΊšK]§Z†@ΥQ )5]uΨ“ζ,₯·λ­T-Γ  `ΫΩΉΧTψά۟P½*ΑLu5`&'οΊκ,ž<ο*Μ@°Ž@• Վޞ5}πΕΥ2Μ@€ͺ“@–ΦoΈϋOΰ·Fuzq¬¨AΉΧo¨ΞζtΤEό=R‰`°–ΐυόΥRSSUσ˜€RΰκΥ«ΚYiϊŽ‘“E&;εX@ €]ΝVUZΉv­ίΌG΅ 3€ @ *JKu 9s&’Τ.“'O‘#G©΄΄Δ€ηΰΠΒΒB)‹*gXεL±ΩΩΩτχίSZZ•Ωnqq‰τonȐ₯l³eV4Γ >N///ιΡ²e κάΉ3ρu²ΘΘHβX– .Prr²„͐Ο!33“bbb€ _άεs •d9 -ΘςrȚαB€ 0*(Ά @}#ŽΏh_LΒΝ8z=q(€€™dd™©%4T•@JZfUνΊFο7Ώ ίjτ+Œ“ƒͺ‹@zf6ρ€ PS8ךkΚ«‰σ€@υ(£γγοQυ~ qτ¨)ά1e θϋχq)%­¬κX@F π€ΜΪƒ2ΩͺC€ ‹ [¬m4l»œέuΓ†υTΩן3ΔjΪΰl«PΊgΟ:zτh™'ŸŸŸO=zτ€Αƒ(³Ž%V888JΩ^9H6((˜ΊwοNΡΡΡ"sμy)8–3Ι&&&JtωΌ8P–KJJ’²αr@l`` ΅lΩ’:vμHŒ΅Δ …6!@U$`TPl#v @€ @€ @€ @€ @€ @€jΐυλΧ₯VΞφZ™Β°š‡¦7nH₯šyνgŠΝΝΝ‘›7u3kΧ΅Δ<―››«τhΤ¨………IΩm9;μ™3gιβΕ8βYŠεce£ττtΙ+>>ž8 ΟvιA±–xΠ& @ Š[EπΨ- @€ @€ @€ @€ @€ @(OΐΝΝMʚͺΙςZ^]s―γ@Ss7kR{υλΧ'~pφΨ€€@)lJJ q€l\\œ”E6!!ςςς€ΰX~ζγ猲UΤk‰b#@€*0*(6$$DΥ ‹«;½σαηͺe˜ PζN{…f€Ι‡>rμ«ΦΎ³< @ΐφޝό<έΈ~M>Π±ί‘ΰ­εyL˜&πτΗΪοδ[„Ά₯gΗΏ)Οc€¬#πιœ7θrR‚Ό³a£ΗQ§ˆής<& @ΥM`λΖ_iσΏΘ‡έ6Ό+3QžΗ k Ϊ»ƒΦ~₯Ό»€ΰζτβδ™ς<& XK ςΜ ZΉdΌ»† ΚӘ€€Ά@ƒ ΄‰ΎKΙΕΥMg9@€@ΕkV.’γ‡χΙoΏ{0 ͺD0@ΐ:ΛΞQν(4¬ώ«D0@ΥM ϊόiΥ!7jμƒΟ6•f k d]MWνΚΝݏT"˜¬%`oο ΪU½zυTσ˜€RΐQτΥΠ.=n»ƒx ˜ZΫσ€ ΨϊΧ:U΅€ ζψ] Α  @U%γΊΞqΝ@‡€V,—?«βd9ΓjέΊum*(VιΐΗΖΓόπυυ% NJJ’‚b9ƒμ±cΗ€Μ²uτO)Β4 K ˆΏ£ΕΧ¨T 2€Ry"αX,Ύ#έ,žŠΏν7KKΔς*ΜΛͺόNj} uΘNά·²―§Ž³²5£‚bmνΰq<€ @€ @€ @€ @€ @€ @ ¦ p–V[ΚΤj«ΞlδκκJ‘‘‘BYYYA₯₯₯R`―­7Ž ¨Ω₯ΕE”vξ]O»Dul(λvuU/))₯τψΛTx-›κˆ80φ¦8™‚¬ J9Ύ“κ$6ζ«λωΩΒqΧ±«CήΝ¨qX[8œ2A±e` @€ @€ @€ @€ @€ @€@uppp2ΔφμΩ“8Ϋ-2ΕV§WΗ š%PZRBWcŽQVΜ ‘}Σ©f\œMiιMΚΜΜ§βλyR@¬‹•‚` ―eRΖωdη∠ΨJΎ.uμμ©4,A±•tΔζ€ @€ @€ @€ @€ @€ @0X€a‘aΧ`.T„,& ς˜ŠŒΥ7‹‹ιf;‹ν₯Ά4ΜA±$oj…Ύς₯₯%TZRT[(,sžΒρ¦½°ΌYj™φΝΨ*2ŚMA€ @€ @€ @€ @€ @€ ͺ(ΑWΩΩΩ”——'b‡Ddq ‹xSΈΎ‹‹ ωωω‘‚ΉŒ±C]@·D>S¨/=n-Δ” ΜX–c‰ Ζ}™p5yώgϊΩϊi"(ΦΦ_! @€ @€ @€ @€ @€ @¨@ °°)..Ž’££)==ςσσ‰ƒdM Š-**’ΠΠPzϊι§[=VC€ `=ΕZΟ{‚ @€ @€ @€ @€ @€ @fΰΦγǏӖ-[θδΙ“”ššJ7nά09 –·ηl³£F"³3„ @¦ (Φ5l@€ @€ @€ @€ @€ @€l@€³ΐ^Όx‘V―^M‡’μμl*))‘ŽΜΞΎκΤ©#=Œ9Tn“3Μς€ Ψ’‚bmιΥΐ±@€ @€ @€ @€ @€ @€ #8vǎ΄~ΚΙΙ‘`ΙΕΕ…άέέΙΙ©Υ­ϋ_p¬ΝRAAPkΜ>P€ +€ XcΕP€ @€ @€ @€ @€ @€ ؈@\\mΪ΄‰ςςς€V„νΠ‘………QΓ† ΙΩΩ™μν)))%oοΖ&mk#48 @€j €ρίjj N € @€ @€ @€ @€ @€ T7ΒΒBŠŒŒ€sηΞI‡^Ώ~}Šˆˆ #FP³fΝD–X'²³«+ΛΦ©cάΩέΌySd˜­+=ŒΫ΅!@€€εk9[΄ @€ @€ @€ j)pύZ.νί½b£ΞRFz*]ΝH£kΉ9δκή€άx’·΅οܝΪwŠ g·jyŽ–Ώlύ5Γρ™W >6ŠΜ~ƒνΫaTΓόΎΩρΟocΰύCιυŸT8 JMόΞȁΎ<ˆν_}F…ωž>~ˆ¦Ύτ$5kJO<3=8ΜΰmQ€ P–ΐ©c鷟Ύ‘„‹Ρ5ς쬫ezχυ£ζ‘mιΎ‡§Nέϊȝ₯ΛjΗΛΏώ|žAm”΅/Ώΐ`ΊwΘπ²V΅όγ™“ιΟuίΛΫ|φΝoΤ)’·< @€€)Ή"iχΦ?iλΖ_ιόιγğ»šβήΐ“‚BZR`³–βukz`θHrvqΥ¬Ζ3 @ΐluκΤ‘οΛσυWoooςςBr³£!@€lNA±6χ’ΰ€ @€ S6nXC‹ηN£¬Μ &JΔHψ—βc€ΗޝΠΪVΠS"λΗ:8:κΤΗ@€ @€@mΨΉεšώ곕 Xa«ΝόBQgO§_­#/ο&fγγ€έ‹?”Ϋσφρ£—^Ož―ꉂότά°;).:δCαmg½1ŽŽήK“E`qέΊuMn B€@νˆ½pŽ–Μ›NϋwoΥAΘΙΞ$~DGž‘MΏύDAΝiΪά₯Τ¦}gΊζZΐABΛ?{ΏRΝuνΩΟ,A±[Ewe@l₯ C€ώ_ΰ―υ«iώ¬©t£Œδ#όΩΛƒUπƒΛ/ί}IoΝYL#zύ x‚ ˜GΐΑΑ‘άέέU±ΌΪ<Άh€ Ϋ@P¬mΎ.8*@€ @ΐΒΒš<φtd.ƒ·β“K>žA|£ςσο7’›{ƒ·EE@€ @€@MΨϊΧ―4sΚ*))Ρ9₯Π°ΤηŽ{¨mxjθε#e³I½œH—γ)ωRΌ”‡ R`6ιZψυodoož[‘YW3h›dΡ”ΰζ‘6ϋΩάwtb9¨υϋ₯­Ϋ’― ωϊPA~>±gνKˆ½@GξΡι8ύΗΪοˆ;°½:ν#Νιβ€ `ΐ…σ§ιεQP^nŽAυω3|ά“χΡ€w抬u# ΪΖΨJI ±Ζnb‘ϊ)I—θ£-6… Ϊ)ΐAFοM}ώωύg£.'%ΠΛ£‹›ΗΈW§“½ƒƒQΫ£2 @ ,gςσσ“ΫΛΘΘ ΜΜLrqq)k,‡ @ΥZΐ&Šb£Ο«Dβ€ΪŸΎύœyyΣώ―€8k@0»€££#΅hΡ‚‚ƒƒ)!!’’’(22’όύύ©N:fί„ @U-€ Ψͺ~,Ό†βΒ‰³‹«Ι{iμέΔδm±! @€¬!pβπ>Z³r±jW­Ϋu’7g/’–­UΛyfχΦΏhώμ)t%εVgέ][€_Ύ[FCŸ|^§>@€€qNNυθƒEί·jC€ X] 7'›ή{ΌN@μ=C†ΣΤχ>•3 ”w`œ –3ΛυA Ο»S5Υjρ[=ΌsΊνŽ{uš¨)ίΉ#3)KΗ½θύΟV‘r±ήι&"ƒμ’o o–Ξ£‹>TΥαΦs?_­Z†@€@YοΎ>†23T«G}•ϊ%rqu——σgΧώ][θύwΖΣΥτ+ςςΩS_€o6μ&ΟFς2sLp°²pVZGG'ε’ §ϋψUX§Ό ό9«=Eyυ±€ P‘ΐωΣΗhωgο«ͺρoΐIoΟ₯~ο67V­»~-W 5‹Φ­^AόY¬)+?ˆξΌοςiβ―Y„g@€€ΙψΪ²eK8p ύπΓG;wξ€Φ­[Kdk2-6„ [Ζ{ΈυϋJsˆB#Q;ž[Γ_η bο;ξRΓΟ§@€ P›Ο›ΊyŠ–~Ώ‘μτ²pgάvbdήƒ{SVf†\gΝΧK+k`€ @€jΊΐΒχί€τ+—U§Ι―Ώ;ίθΜάαwξ’Υ4φρA"ϋΝ5ΉΝ%σfθ Š•+Tσ‰δΔxU@‘ƒƒ#M›»Τ €XΝ©sΗι§Ζ½NGξ‘cΥ,¦ΓϋvJ–υ]δe˜€ θΨ'‚\ϝ:ͺZυΪ΄y4δO©–ρ w‚ξΩo -ƒ2<'΄ΠdOΝHO₯ΎϊŒΖ½:Cg›Κ,Pf₯ukΰA¦2Ν½ν‰#ϋιλΟη½6€ ”'πλš•:Lρ€Νƒ¦w3Ξά>ρν©]§nτξδ[ƒ4ηίΈN ?x‹fϊ΅ήν°€ `¬@Γ† ιΞ;ο€Σ§OΣ‰'hοή½Τ€IκΩ³§λζζF<Π!c‚dyPΎŽY·n]c υ!@fΰθblZ‡Ί PΛ‘± λP·Žψ¬S<£/kφγ@ƒU/€ Ψͺ pVΰ/ε<ςšr$TKμΎΈ¨¨Μ KμmB€ Ϊ*ΐί»’ΝTώψ©³+ό.Ɲu'ΌωΎκ¦cJς%i$}ν‘{Uc€ @€ P8@εοί~TI`³–τΚ[ΥJΩ@σVaττ‹ShρGΣδΕ—βcˆχεΨL^V“&.h]“hΩ¦=yϋš–αgδσUA±……tNdκά­OM"ΓΉ@€€V.™«j΅Ηmwθ ˆUV nNO>χ -ύd¦ΌxλΖ_ι…IΣMώ. 7€˜ΰοšάB3i•ηܜlš9eŒ΄ φ_€$[eΨ  ΤLdέφχΥΙ½<ε½2b•Š¬°ΫΕΆ»Άώ)/ήΉωwΊ–—cρώŒς1@5ZΰƍδδδD]»v₯ΤΤTJNN¦υλΧStt΄”1ΦΫΫ›\]]ΙA$ΰ WCKii)yzzRXX[ldψv†Άz€ c8(Φέ©.ωΈΩSj=‹Ο)ΗΊδλζ@υνΕηbα¬Φu[­_>Ϋ9ψΘ³'θΠήtόπ^JΑ9YYtMŸ6πlH^}©±…΄Yk¦–­­vΰ±ΞΡΖ k(1>Vz$]ΊH…ωδT―>y‹cjμλGΝZΠύ?A­Ϋu2ιΈxΔσέΫ6Ύ]›ιμΙ#bDπ41rwΥ«οL^ήΎΤΔ/zD·Ό_Μ71iΨ5Y€GCβ’©Y‹Φg ΰ ¨SΗε€Κο;{Gςυ   Ρq+ΈyhM&ΓΉA€€B .ϊ<ρg‚¦πˆ„†vν%FΓΧ.gΔχΉ>ύi/Ζ< ›ΰ  qώϋ­›Kωω7€οΐ!-ΫP°Θ–ΝΏGkRαkρ1QΔΏκ;;‹ίο)4,ΌZœgM{­¦_ƒQœφb¨€€˜όšSPσVT^§VΎAzϊψ!i›τ΄Trvq׊|(Όsjθε]“ώ©β\ P£80λΔα}€HΖΕ͝‰χp»Ž•Ύζ©ω;qςθ~Ίšv…r²3©IΣ jF-Z΅%Ώ€`£:fθ{!x°Dώνΐί³¦‹λ֍ȧIS  !GG}›˜Ό Χ¬L¦Γ†VXϋΓ βχ†²Ό>γΡaͺžr‘ΡӜgιό™TR\,oΛχO†>y+ŽΌ’ &Μύ½pώ΄κ,Z„ΆUΝ3ή₯ΥY”vόύ˘bξσ+kίΦΪοίΤ{e;–C¨iάGΚ2lΤ8εl™Σ‰L²ΛΜ’ƒFS“ιΜ‰Γχό272rE’CSxk–fLύW₯]ςgμτΎ w_³ο$ΦtΐΎ @ΐΌϋwo‘ϊjZ­ομJ‘ ύρg^VΕr;1QηΔ5σξš&ρ @0Y >>ž~ώωgΡ‡šΤίƒαΐΨΜΜL:~ό89‹ϋν...δ(υ΅  €:tθ@‘‘‘b;σήW1ωd±! ΤZ:"¬«‚mονLυDŸαk…₯ΤΘΉ.…4¬'e‹­΅0΅πΔ[ _tsž2ί\Y:=:z`·ήfyd4ΝMπ͏‰;άΘ“τΘΟ™u„QεdefΠ— ηΠο?+ίΐQ/†yDR~π±―_³R Š}lΔΊλG•U˜.))‘ ?}M+—|$eΣΘη Ζ=΄o-˜=•ξ2œ8s™›{νκ˜‡@ά· ε‰QxΉp ψO•¦s²³θ³ή’Χ₯β½ϊWΛΑν+ΟΑ艑s5›ͺž9#Ώί† mΆŽςάρζΙϋ{Κϋ᧛'ΘσM Ώ·»4h€¦ήΧλw•Ϋq\SΟ€ PΎwlWŒΕΠNν.ξδΦΐƒrΕg’¦j&ρ @ΐζ8ΐςΫeŸΠ/«–'ι+uΔPxœ…λΕΧή­π»΅Ύν+ZΆώΗ―iαϋoͺͺυμ;fΜϋRώϋΛΏ‡G=ΤWUη›_wιύnώ―dκ³ΉοΘuο{θq9f’4Ο#­ΏόSβ XνΒ£Φ6Χ&Ύυ‘Mv1χkeί#<θΧ}½nuNε€·ίwχ9Λϋφ‹O€lr‘]ψίΐ˜‰οHΧ|4λŠE Μ_}FΏώŠR“5‹εg~ »υ@“ήž+‚kƒδε˜€¬/ ύώoή™ϋ‡t ™i΄μΣΩ"ϋΕz½Χaψs§cΧ^τς”YΤJ X`L)ƒςί‰΅ί}I|=·¬RίΩ…†?υ"|~Ω‹‘ˍ)9Y™΄jωΪόΗZJΏrYgSΟFiπ£#ιQq]ΨCΚV¦TΥ5«Κ3Ά­½ό]ςΟuί©xΐŽ½TΛL™ρlθE· Έ—vόσ›Ό9wΦŠ5δ;γͺ/HΧ‚Ή‘›₯r{γf-tϊŽ)T„ ›HJJ’‚b9ψ΅HάΧε X~δεεQnξύmω>-ίία‡!…WΜΟ ©DZn ₯lΆbw~ζk»Ζ—έ*Φ@€€Άϋk_·5υp"/WG*,)₯zφuI$%;ρρ¦X{kΜΧ$Γξ€Φ€3ΖΉ˜M`λΖυ4sΚΥˆΥ†4y†ΜyƒβD‡…Χ¦Ν3ψK΅!msr}γεz;i•Χίθ™9e,]5Ÿ|vByUιϊ΅<šφκ3΄Χ–rλi―άΈ~5όw½ϋρr³t.Ρnσ°EΞΞ¬ι°_XX("w zyτƒz;#–w_>O /ͺ `)&κ,-λΧύ°œ¦Μ\@]z¨;δ—·²ΦρTΝypΊ"+’1₯Ht4Sn/~υ³9κB€@}ou€ε*tΓ 8Υ«_Ζ·§§₯¨by`Θ­ ˜‚ `C :οέW₯μyεoεšψwρ°Q/Π “¦@TVϋkΕχλω³¦¨VχΩ΅•±Ό’!I‘„—•usŒ_+λfe^εκ΄xήtZύΥ"iZίΈ½ "cιΛ£ gΗΏ)ύŽ7τ¦ΎφΜΉΜ―•΅~(³8ˆίr\ξΩFoΏς΄Έ’P"ΓΩߎΪC+~ήJA"[1Ώ9~;υί H\G»πkΘΧUž~δvšχΕOfλp«½ΜC† (ίΕ…ΏοΪς'}πΞ„r?{ψοΏŸ~½9{έuPƒvuφ€ψ;1RΚ<[Ρ΄ϋ• mΗ?ΏΣ³R›φ+ΪDZΏχVšσΦKT^–EϊεωΊν‡KVΤΎJUyΝJίρ`*8ϊΈΞ=”ϋ‡>YΡf―Ÿ΅`e…u ωΞΘS•Ÿ΄U+.ΦΈCSί’οΡ‘νhύ€ΩƒΙςw\gWy™1o½ΏΨ˜κR]KžŸς`,΅sήCP/¦!ԁγβϋΈ²tλέ_ΈLΉΌ¬ιNέz«‚by@œ oΜ1K?ŽΔψ8Υn­›p1šζ‹AΛ5₯CמφΡΤΕ3 @ "΄υ AΝZU΄‰j=Jα!”Rq9ΙπAωUa€  %ΐχ`ΉŸ&θU{Π=Ύ.klαmΰY±;q¦ή½{χJ™zοΈγ A?΄ŠεP€€ιφ"ΦΝΙNτUϋoΐώ[lό'ιϋΗ–U/`Wυ‡€#¨Ž'Žμ§―=kt@¬ς\7ˆμ2s§O”:Λ*—Wfš;8͘όΌNgcΪ\ϊΙL1BϊχenΒA/lt@¬¦ΑŒτTšϊ₯Y„gΤ*ΞΚρΚ³Λ?ΤΉζςΟή£(ύXkg3šπτCεΎ§ iu @ΐv8“²”””ˆNω{•‹Κœή±ιVΦΔYΐύ‚Λ¬€ͺJΰΒωΣ4kκ :AI<‚«—wr=υ•Ώωœ^y?q0QeΛOί.Υ ˆνwηύ4{ΑΧFu΄¬π8ΔΕΙOή{½ά€Xeόw‹ωοΡkc#ΞLZΥΕ^+sπΐc―{Όά€XΝώ8ΫΫΫ―Œ–‚Ο&ύ_Ή±šmψ™³κΎ+ε”t«¬i@ΐ:Ϋ7mήΣ9Ω™퐳H8νJˆ»PaύSΗŠΑ—˟mϊΉ‰½pŽΖ Ώ[d_Xα>οΫIS^|Όά€Xe#WDgΖ‰Ο>" ,‘\^Ρ4YU$„υΆ*pZΌ΅KDΟΫ΅UϋykΌGΫvθͺrβOώ{Βίq,]¬q~|ΦڏΖΛ”{šmρ @ Ά πu‘“G՟λ½ϋίmCηn·©κsŸ‹άœlΥ2Sg΅M Ά|¦Ψ"Ρωϋέמ'ΎNΑΕΕՍήω`)ρu4@€€98ᆲ4φi’œ­pš―…ggf¨κYγ3R΅CΜ@€@hΤΘ‹zυκ%=zφμIέ»wW=zτθAΖ>Έ~„……!σir8+##ƒΆlΩBΛ—/§oΎω†bccΛ¨Ε€ `.€-ύ@XΝ³ΉΪF;ΥC™b«ΗλdsGΉτ“wΝΜϊϋ/«hψΣ/‘9.μπMωw_SfG'{jΤΨ‡œ]₯Wεu ζ\½n'o_{Q4ςμ εΖ,ΰΞ“_.2¨l#7χΖlŠΊ¨φs§O’ΤδDΥyπˆA!";_°jΉffΥ²ωτΗΪο4³3wώΤ8jέΆ…΄ “ήΧgN&~lέψ+qη#Mα|ί½ΟΝ"ώ~ΜΑ±\ίΤ²ζλ%΄hξ;ͺΝά3„¦}ψ…4"¬jE%g~ίϋo\Ο“ZqtͺG>6Šξό˜΄ ™΄―ΛI—hώΏιηU_¨~ψw­q% }βΉJι›ΫΒkeϊΡλnY ~SqΖatγΒEΘΦν:JσœεqνχΛUƒ~ΕEGπ{»©+λΩw υμ7š·j+*J€ Ώ?~ϋΉκw!‚Χ―«iθ“ΟKmγ€@Υ πη <ΘΧ[Ή4ρ€‡†?C< ―_SJž>ˆžyω ΊϋΗTdω{Ωρέ”yΡ|N°F}1pJeΖj–(Ξ°%=;ώQe­λΡ›>ϊβG`TώnάΞ#‚>\ό½ύZϊώ«9ΎZ$‚žWύ&Φ¬«θ™³ρqπ‘²άυΐ£τ֜ΕͺΏ―Κυ•™ΦΔr@ΤgίόN!-ΥΑyžƒIŸy„8 ISΎZτ‘τwΏͺŸͺκΧJγ`g†ε‡ƒƒ#=ΚΫ4lΤ ͺ›§ν:DΠ€AChόθ‰;αjΚ΅Ό\i½'Myoρο7eαΓμόΜ£wΠŘHyΥΞΝ (VΦΐͺV€―ιjΚc#_ q―ΝP}ξppl·^ύιΎ‡§±ΓΦ”}»6K;ΚΊώΒ5p@MαŽρό™Β hδζΙg'Πΐϋ†”qΓE‡ω:Λs½9o§.έϋͺ iΆaΕBΐ€GΕ΅α'ΟTG‹ΠΆtοαβ<ž ·&Œ&Β5¦ΰš•1Z¨kkW.'©©Ήt°nέΊͺeΆ0Ψ¬%ρƒΛ Ο}ͺCrvq“-«*f¬υuΧΉ_˜4]g™ΔψXϊδ½Χiώ¬)ΪΆ#uλݟ:Fτ"ώεμβͺ8RΣ&­u~ΦڏFΑ”{šmρ @ 6 d€έϊ.9Šϋ§šeε=»ΉλΏff€›eΐ e¦Ψ&Mƒ(>ξ‚ψ|œ,²Ϋ(S·>4yΖΗFχ#αΎ,«Ε :šΒƒ{έ%(€ s π5nS ί«ζAε”₯½½4r¦!@¦ xzzR·n݈$5gαΟ0Ύ§Γ”[μ’––FόσύυΧ_)Ε ˜ΫΦ1@€€RίL”5pz-τΣͺ₯F?6ύώS™ΗνΥ»Ž;:­\·“žxvΌΤ)ŠGπμΠ΅'=ςψ³τωχIτmxφΤQ}‹^ΖA±ϊ g$yκ…Ι:9;ν§_ύ*²S6Χ·Ϊ»Cgωκ•‹t–iΌ9ϋ3©σρCßY ϊIͺ¦Ύχ)-[σi²hκjž^΅Œ8› j‹δβ'nz.i+=,²Œ”Λu—-˜-2E]γI©DˆN—³~£·Σ£¦N—}ι£₯kΘ©^}Ν"Ίœ$2­_-Οc€jŽ―›ωΙ Uζ•TΡΑwΞ›/Ρ}Biά“χ΄WŸ‘ηw κήL ΛΛΝ‘x`—OΎό™5φ‘—a€€­Dž9:ά±$Κ+ ‘Κ–ΝC’Οί $*o[εΊoΏψD' φή‡§·ί_bρ ‰—ό «<6ώ½`Ε:rSd-ΙΙΞ€o–ΞSV³κtUΎV–<Ρ'D@Ϊπ§^Τ{ƒ“3Ά?ρΜx½»?uΆN@¬¦"Vγΰ4e9ytΏήAΕ”u0 XW€³7Ÿ:«ΜΟΞ ϋξΗΛUΕ߳ϝ>¦Z¦™Ι™XΏƒ-(Λ;.Υ«¬ΓY’f/όV dγ"/ζA~ω^7#νΥτ+:ƒΎτϊ{4Ad“,λσ3,Ό }υΛ6ΥgŠΌ£r&pΝͺ¬²yώή€, -”ΙTΉkO[σ=ΚY‚8ˆ__αkβηΕίEώn=ιΩ‘u‰§ιOσgO₯-­·υmVα2kŸ΅φ£9aSξ!hΆΕ3 Ϊ(›sk@Νω»‰AͺŒ)υκ;λ *œy5͘&Κ¬›˜+―γοκO?r{…±ΌΑ±ƒ¨oACs©¨°Pn£Ό ώ½1kκ8ΉŠ_SzυuΠ‘Ό€ *ΰγψsNY|t9::)a€ `²cςύќn±κ—₯Pό^‰‰‘υλΧΣΊuλθμΩ³T\\lφ€dυ^1@€€RA±J L$ ΜΒ’ά οχ–ψΚ#vw,c”΄ Š¬.ΚφŒ>WFpνMΝͺi›ΏόΌΝ¬κ™³ΒjnΎσŠ€„8:πο6UΝ w ζ‡Ύ¦}gšτφ\}«€ŒΫ6mΠ» !P“&Οψ€άξšs:{’6*YΉγ4 lHι,F~θO©ͺnέψ«j3€ Psϊ Έ‡Ύc?έ#2=) wτεQη·m\O<ˆŠ&{¦fς톩}§nšEx† `S‘gN¨Ž§τf©j^ί ίάόΨHβLšGjŠ:˜Ύν”ΛΈΣα²Og+ΫΎ1k‘Εovυ½γ>jΧ1B΅o}3œwτΨWU«~ωώKμ€ͺ œ©ͺΧΚ‚§$ŒπA―/—»‹π.=tΦσ²AΣY\Π"4L9K%%%€ £ͺ€@ΐͺœuϋ™—¦VΈOΡΛ»‰ͺgωΣWΎY0x MiΥ&\dΥmVSGσμ,²žΡΜJΟΏY©šη™5ί,‘ΒΒyΉŸΨŽƒ{+*ž" pψθ+ͺ&―Η5+™ΥT@;€ΖΩΥ­šž‰ώÊχθΨIΣ€ΐ؊:2—––ίg[+Ύ·Ξxν9zgzμξ.΄πƒ·ι¨θΝί‰**Φ:?kνGίωzAίΆX@ 6 hŽζ{©œΕάΨΒ׏”εj†y‚b“βγδfydώTx]9ψr]QQ!­Xτ!½6v˜ΞvΚzši$“oΉp?i,%€ `+«Ύ\@‹?š¦:ΎvυΒ«ΣUΛ0@°UC]Ϊκ±›σΈŠŠŠ())‰vξάI+W€ό‘’££₯ d'''Uμ9χ‹Ά @Π@P¬ –T Π½χ-2―j?FŽQwDΥnζbL€φ"iώƍλz—»0;λͺήM8[ξ fΙ7@΄+ρyl=š¨σX·ν€j΄–c‡φho*ΟWΤAμŽ{’‘ΑL_9zp·ΎΕX+ΐAK½n7θό~ϊv©κβύ<)e™5hcQ‰;ν+Λ)UP―\„i@¨A1Qg(6κ¬Qgtαά)Ϊ΅υ/ƒ:Υ0*C0“€½½ƒͺ₯λΧ”!γ)ρ[χ―}Ρςγϋ‡ͺΪ)ofωΒχ₯N‡Κ:†^wΎκw²r½9§~όƒ›»χ‘'TAΊΕβT’’Γ₯Α ™‘bUΌVf8μr›xL •Υ9U³!gνΥ.Ψ\Qiμγ§zνΈ~–Θκ‚Ψ†g‰ζΐ؊ w6oήͺͺZV¦ξuZ|πuί«κ=8l΄jΎ’™ƒ†¨ͺdŠŽϊΚkΞάΙώΧΥκ@Ω‘ΟO,3C¬ͺ113τΙη€Α$΄—λ›Η5+}*XVx{equ­Y#UυεΐΨ΅[OΠΘ1“ΚDVι™NΎt‘~ϊφs?ϊAω`Ϊ΅εOΝ*½ΟΦ:?kνGϋ$Ή‡ ½-ζ!Τ6ν XνΰVC=άά=TU33*ϋœΏŸg€§ͺΪε™ΰζ‘bδOhΓΞ³΄ύψeΪ|8~Ωr‚ζ~ΎZάCξ―SΘώ]΄ϊ«E:Λ• ~ωnνΫ΅Y^Δάy@€€-€₯&Σ”Ÿ /ζΏ§κΕΧΥޚ½¨ΒkπΆp8@¨ή|ΖΠfΔΑ―ψYPP@999”œœLǏ—΅=0–ccciΓ† ΄|ωrΪΈq#₯¦¦RσζΝ©K—.ΤΈqγκύG@¨fφΥμxqΈ6  ="y‡T\\LηΟ£U_Μ§²2Ή–·½1λ‚›·¦ƒ{ΆλέdΥ²ωďТ©KΫ¨SD)λ wμ²³³#1΅’rκΨA½U‚š“O½λ” ΓΒ;Σ₯‹ΡΚEτρC{u–aj²@Ÿώƒ >½8­`ϊΆα] ή–+…΄"ξ ­ šηL!qΡη©΅ψ[€@5G€/8~4cύφσ·ežοΛΝΙΦYϟ fO₯=Ϋ7ΡΜOVΤρ_§,€ `ANέzΣ‘};δ=Ω£^χ8½6ν#ςl&/7Χ*ΕΏŸ•ε±ciόꬱΚυζœζ !ΖdοζΏοmΪw¦3'Λ‡‘t)ŽZ……Λσ֚°φkeσ n^αnœ]\uκψ5 Y¦owϊQs œ¦lΣ€€iέϋάaπ† ½Tu΅;ζσΚτ+—© †\―^}gƒ³Δj6jΪV 2»œ” YD'Žμ—:Υσ‚qνυΖυqΐ«&8Ά,^― Œεn###ƒΞŸ?OήήήΤΎ}{)#jYΫΧτεlsβΔ Ϊ΄i“λββBνΪ΅£»οΎ›κΧ―O?ύτ%$άΊUΣ=p~€ ͺ@PlUΏ5dhΖYΉ’D¦­θσ§‰;&ργR| •ˆ/ΕΦ(‘tx½7υε"½Σ1βfŒΎr5ύŠ”ΉΙ©žΎΥX'Π¬EkƒΟ‰; )‹)ΫCZΆ!e¦ηρ· A±JULC¨ώΤͺΛƒž<ωάκέοnβ€Ξrw-/G|?₯·ύM«W.RuΚ?΄w;Νzc}ΈX½ͺϊλΰ κ.Πλφ»iΕβUΏ­ωoΦπ{»QϋΞέ©g߁bΰ§ή Pφφ•»Μ³δγςοe₯›)ίΓ•Ϋ3Ϊ¦ƒAW)ΫlήU+(φ’r΅Υ¦­ωZYλ€|D‡TC ·ς RM±DΐΆ¦m6[yςυ~ }β9ι»Ός¬§"μ&Ήχ?Φ~G|kΨ¨q:ηo­σ³Φ~΄OИ{ΪΫb€@mpuk :eξ°lJαμ?Κ’=πr‘ΣΧ―εIΧ«ηεŠNΤ9Τ[\γzmϊΗ~GΏγž!dο`Oo%οŠϋœΜŸ5…ΎX½I^Ζ<πΌמŒ ₯ε|_`ϊG_ˆνΥAΎͺ0@°‚ΐQρ›οΣ9oPŒθO©]š…Πœ…ίχgB XB€οΫ^Ύ|™Ž9JgϞ‘˜˜ΚΝΝ%ώΝΘAεMΠ¬&8–·α€Zˆˆ¨pϋςΪ)λ²²²€,ΊœΆcǎԻwoκάΉ3ΕΗΗΧκ€αšςϊβ< @ z T·dυ:ΧZy΄/NžIƌκ―ArrRwd,W>σγnίΠޝ¨:θ*λYkϊΞϋ‘u?¬ 3'oe‡©hίqΡ‘"kd$ύΌκ ςφυ£{‡ΞόxE™uΜ΅‚Ž[ά=ͺ6IKMVΝ[sΖ―•΅ΞΗΞΎhΩZkίΨ P΅όώ―lφqν3Π°€―[›R΄.έΈ~MnF;πΦ”ΐ~C‚bqΝJ&ΗD5ΠΞώΖχGψϋ Ή²ͺ•””‡TΔεϋ8–.Υι=ΚA;ό6ςi€š%σfόh „σύςOUA±Φ:?kνGsΚgCο!(·Α4 Ϊ* =€gd5₯δίΠ ŠΥ€Μ”6+»Νa£UA±ό]%.ϊ<΅ —šή΅εOZΏf₯Ό›ήύ‰Α0GΛσ˜€ XS€³ιύψΝη΄lΑ,*,Tg`ηγˆθ՟&½ύ!7·ζaa_€ P 8xυΐ΄aΓJNN–2»r '_›Υ\ŸUf‹ε{Bš’ΙΛΟ\—Ÿω~‘››yzzRϋφνε64ΫΤΆgΆδ¬°ααα’K½zυdφ2₯(έΛ۞―»k Ώ†œΕΧ\₯€€TΌήš‡ΉZE;°Œ@)‡!”ή$~O˜σ}`™£½Υj‰xΟ–ˆχx»I_o­ΑlP@|¦ΩΩ‰χ™Ω?oJ€χ©Ÿ™ϊ€«OΛΚΘΚΜ η†έE†v2ukΰAύοLΩ™Wiη–?Κm»²+ωΒΡͺίφΠͺe €ŽEE…F7yIŒΜύβˆϋθ‹Υ›¨u»N·Ά7ρΛκ­ʞΊ)ώX @j4|o‰’οβ³%φ£έ¦²“¦φ:ΜC€€i±ΞιdRσΚ;ΔjφζΩΠ‹&Όρ>MυΝ"J½œ$œωϊΘΛ0@ΐzυ»‹Ί‰N[ώZGΫΔ`N‡EVΧΒ‚ό2νR| }φΑΫτυ’θγe?SXx—2λj―x~Β[tCΩZ6_^΅mγzΊλΎ‘ΐςB LΈš[]u$υ껨ζ­=cΝΧΚΠs3εχFϊ5Tυ Pσ,ρώΏ’5`v–oC΅ƒμϋή ‡rTS>S”?”m)§kΪ5+εΉaΊφ΄χ>vόσ›|Β|γ1υr"ω6“—Ufbηζ?ΔΐͺOΛMΤ§ώήKυ-=­ΊΎG»τθKσ—―₯g‡έ©ψ5ςμI©γšζο“΅ΞΟZϋ‘‘`€Lhδε­Ϊ †ΤИAn Δυ₯쬫ͺvΜ‘)VΥ  3ΝZθώuξ€»bΡr«ό&’g?ιΊ™Ό°Œ‰λΧςTkŽμίMi©ς2;»Ί4`Πƒς<& @ d^M§ΩoΎHϋwmΡ©ά<”Ζ½6ƒψΎ  @ΐ)))΄yσfβgΎξλΰΰ@^^^R6S*EFFg“utt€i—έ&₯§§Σ•+WθΪ΅kuIϊμή½;υξέ‡Ϊ΄i Xρϋ“„ΝY^½JgϞ₯¬¬¬2›εί½qqqkΔΛǏ—>˜#¨ˆΫNJΝ£Τψxρο£”κΨ–ψ¨ΜƒΕ Y@#iZΈ΄ά &΄ΔίΆΪ»;™.FΊV“Σ:TR˜O—ΟΔΡυ”«T§.Βψ΄_V“ηΖšγo’ΙΗQ7¬#“ΊεG…ΏΝfΛ―QRRεδδˆ₯Š€Lτ•}έπnͺψ,}JΌ3‘ά€XΎ˜Σ‘kO kίYܐθ@Ν[…‰(q;š?{ͺήC3χˆΟNNυθΩ—§°Q/ΠΏΫ7ŽMΏΡΑ=ΫΕ5ΓdωΛύ{SΗΡWk··ΗΕMŒbΞΑΪ%΄mG¬²Ϊu*šwu3>NEmb=ͺ»€vΖώ!7ϋΣoHό’«Τ©6kQ©νMέ8/7ΫΤM± ”!wAg 5¦tθΪC§ϊΕΨ(BP¬ @6 ΐ™»=8LzpfΨcχΠΡƒ»ι聉;κ»H”›“M―Ώ0œ–ώπ75 )χ,ψ·ϋδŸΠCGw’ά*p“γεmζ͜LΊυ&Wwy™Ή'Š νΩwκeufXw18WUKΏVƞ~+†ϊ€€ΉΌΌ}UM^M»’š7t&ύJŠͺͺ_Σ yΎ±Ο­Y^˜–ͺ’•+–3q%%©œ΅­ͺiΧ¬*ϊΨΙ(»Ρx ±k˟4πΎG€JeuͺεJίΊ5 iΩ-b  `Œ@ Θό¬,¬ΪΆƒΑΩ”ΫVυ48ΜǏ@ζHΎ―jΠΑΑ‘‚όΐ«±/Ή‹ΑO”§βc#©Ηmw¨ΪΖ  [¨'n<υμ7Pzπ±qπλ‘½;hίΞh»ΘϊΕA³šΒ<}όήλ"σΤ/šE:Οό7tΖΗ_RΏ;ο—Φ9Υ«OߞK“Η“λrΎΕM§Χί½•AV^i¦‰,1’Ί±E;€‰ΫR1χkeμΉαχˆ±b¨XB i z`ΞHiJΡΞ\θ,7€Ύ–dΚ>€¨ZWLΤ€kVŠΣΒd-ΰΑ>9k«2›όwΛ?₯{† ―tgέRΪΏ[‡3·Y«Xσ=zβΘ>Ϊ+Ύk ΡtλΥ_3kshXΈΞvœ-Vk­σ³Φ~tN @ΐ(†λΤ?}μ QA±‰ qͺ68ΥSO»ͺJVšQΎς.›j}η·a`7€ ½±ΞΡkc†Q‘ΘΊ,ƒIc'N#ψ€ `MόόΊxρ’h“œΡτφΫo§‘C‡J±|’ŸŸO'Ož”‚^9@–-[Ά €œ€ΐXΞ$Ϋ©S' ’ߝbccιΘ‘Γ’~G)Έ–3Ο’˜W€_ΞΪλμμ\fΓΛυόύύΙήޞ\\\€mΚάΐΘ"!05pw‘›(ζΰΐΌΤΤ+"+s:ρ κΎΎMΘέέ]ΌΟΠ—ά|ΖΥο_,ΏwΝ[h.Ζjގ°Ι’““₯Aψspp'₯xτj~v6sψfβζοό™ΖŸg•O.‰ X›ω—R=„ƒDΛ* V¬•oΚλ««›Ι‹λρ(–.Ξ.nͺΞΒ9ΩYRg᝛£][’bρ%^_Ήpώ”ΛΑ―gNΦ©vξΤQeXT^€oΆΊΊΉS^nŽάΨ₯‹1Υ2(–/„£@€€ωŠ‹ΥίακŠI¦Œ΄λ&2 *ƒbέl,˜Κόrh¨‰<0Τ€AJ oΎOοL|Zόξέ.ŸκΉSG€Α‘ψf‰Ύή₯‡«Yί³οΤΨ;ψ(Šφ?/!”„$$½χ^€ͺ φWΕή°W{Γ^°wΊ*R€χ‘HO Tύ —»δ.Ή»\Θοα³άΞμμΜμ77»{»σ{žγeΥ‹τ,ωeΞΧrρ˜‰₯G?Kž+WŽ;‘ΟΊ=kQlIq­χχtΊ¬+gϋΛί#Ξcy w³ŠVnŒφδh{p‘œg*8ά’Ά>7D”²*aΕϊšb«ψ…τΜΚΦρ1―rΐKΖn½šœ &Δ•Ώ-K΄{½²κΘLO5UΡ­·ηD±ž£ˆόc΄ΝΩjY h«Χ¨YdR΅^―§ŽΟSνθΗΕO  ¨¦MšmΣ‘«η,μΥD±—Œ½α woΫ`*ΫoΘHSΊ4‰LΝιY¦6ιS·ΪΎu$¨Q¨žtθ3/7[ΰ Νhα-Z“\'  r#'SOέ“ι·wOΞ|_]nύbΓ$@$@•›ΐ)Νq6’ŽΒq!ζO΅hΡBϊυ막ρ‚-` jEΎdee)!QvvΆ„„„¨{55u€―――ΡžΡ„Fί~ϋ­ΔΕΕ)lλΦ­₯mΫΆͺœ₯B”™@£FT΄^όŠ3ό=ςσσ•'iiήKβΧΥ«ΧȊ+Τx9r„ŠΨ|>r¬χφ›=#ŠDΰMωšzβ„,X°@9qˆŒl#£FRŽ*qTΆΎΒ"œrΰ^Ž9Κκpƒ’ΨΚφ *γρ&k‘bm"rι^ͺmmG^τ‘({›Κœη’yςσΧ©§ŠζecΦW‹46š ƒ!κλ°K/SΛΑύ»δΞkFΙ™3†ηWBήΞΪ„_΄cm˜ΐ•z<©D‘ήρδDΝλzυξRMσnŒhP€3H ΐΉΕψ7ϊp”tν5ΐa"ψρπυΗojή­N©} ΈαΆi‚¨WΞZΥͺfWˆόš‘vB–XΥΆ«K,Γ$@$@ΞΐM£!*bR|Œ„6ifΜ.v½@σάk}Ϋ²uϋbχαF π][7Θζυ+-Ν 6JMt΄dΨYs™—ίVFυ‰‚έ ΓΩ „aαζ(zvͺ°dίχψ‹²yέJΙΟΛ±δ½ςΜύςΥ‚5₯Ί―ΆTbgΞ¨Πœ[9b/>°ΗT΄sχΎ¦΄'ξώ[ρχˆ'ώŠlƒHΐ¬«1Ρ‡$%1^‚CΓnφ/-ϊ9ν‰nˆrή08DO‰…籇’vK뢝,eŠ[γD£C‰βΚzΣ3«βϊΙm$P)½Η$ŠEYκ 5©¦Έ}νmΓ3ΣΩο½lΪ ‡ m5±Ž'ΝSc΄e›ςΫ’-‡ΆwΗfIO=.ώ ƒ,yΞΐ©«u”‘«ϊ§k/Nf}ΎP…·/²QΛΐ$―NZd"Κb‚Ι³―jk7ζ‘@₯'ΠΌeΣ؞Γη2αš[ΎψAΘώ鬙Ž­Ϊt”©χ=iI;³Rί? HqΌŒν=`X‘|c.Ϊ«—/6fqH€HΐEšvνΨ²Ξ)Qμ1-š7~λοCτ6―Σΰ' @yΐoπo>yˍ³…gΕb8‡jΡͺ­ΰ7±nxθξ¬6’[ο}\ήyι ΛΧΞ~ΉσΑg-yZΨiƒφ Ώ•±Ώψ@ bΝΏa°ΣΒ_}ί²|ΊϋoΕί#eωλp_ o €Θ‡Ζ!Ϊ+ χΰσΎTξzx†Γέ[ΎtΎ©μˆq“”χP=³qΣfβS«Άζ˜0OΟ’ωίΟ–Η_˜eI·‚ϊO€$WΔ²Ν›žYY:Εp’&κ"Zμvƒο2ζ~ϋ‰LΊώv'k;_ό‹χ_UŽXŒ;cœWΥ’x<5FρlxΦΛ>o.ΤξΧΑοΆž.υαnZ»ΌΘΎΦŽmζoYυΗ/ρ){χ%Αύ|I†η‹ζ|e*6tδxSΪS w­ψ{ΔSIΆC$ΰ.xuέΤi¦κϋδεf›ςμ%ΰδaΗζ΅–Ν˜pύmζϊpΌζζ{-e°‡iˆΪX’α·ΑχŸΏ[R1ΛvozfeιWH ώ{Χ£Eφ‚Θsι‚ο‹δ—”±xή·ςεG―›Šυμ7DΖLΈΦ”牄§Ζh“f-₯mΗn¦CϊαΛχ΅¨Σ™ςMΰώΦϊ™vϋΞ=ŽŒζ©γσT;Ζcγ:  €σΰ@Ό‡vΝ5Ϊ皣 άγggΞΘϋ―M7ι3hΈKž#jzΣζ­LuΆπSΊΈœvlέ°ΚTδ¦;1₯™   ς"°|ιSӘϋ7ˆσLL˜  (xcUΧζ?U³α°°f͚¨„˜ΉΉΉ’ššj ( χήΗΗGŸΫB<‹Θ²4   πΕzΛ_’‚τ£E«66{Zpϊ”1>F JΈ~ό€b£Δž;wΦfΞdϊΥ­§yοlmsx½zdwωZ‹³ώ―?$ϊp”Δj"ΨΏ-ΤDΟΛ-“†Ων_d‡.¦:oΎϋ1Sژΐ$‘‰Γ;ΛΫ3—Ήί}*?~ωΜ|ςΉftoΥ¦±¬Ύή²M0τR=ΙO +MΒ#δκξ4εΒΣρΑύ»LyΦ‰Σ§ςεΥιDιΊAΠzɘ‰z²TŸ­΅ΘFΓ8?ͺE΄gπΒόΖσ|AkOEΞξ‘ΫdTŸΛrΕΠΞ’ŸΏ²μ§Χ±oΧV‡φΗ΅UίGόφ³wΪ—…HΰB$€Ή#Η_m:΄ΜτTyρ‰»ŠuŠ‚0!hφ{―Θ֍™φΏcΪ3¦4$@$PžƒC‘ZuΓοYάK8bψšŸ—c)ΪJϋύYZο’Ψοαgί°xΦC₯ψέSχͺ—_–F\΄‚H΄o<°ΝozπΔώΐ-Ο#t«EΗΕΔύς0Oό­ψ{€<ώ²l“Hΐ•&^{« “nΉ9ΩrΝ$3#MΟ²ω‰HWέv•iRˆqWIHγ¦EΚOΎι.“xμŒζΔπξΖIJb|‘²z1ΜΠ~ηχŒG/«zΫ3+½_ό$g ZΫΤϋώtͺοwxώ•‘vBΟ²ϋ™Ÿ—+―gΜ2­ŸΚΟ“‡oΏZvmΫ`ΙGdχ›m8ΜΠ dŸΜ” Γ;™–ί6;ΤΛβ³GίΑΖ€ΐΖ{―>]μσμA,œŽ­ƒζ ³Ό‘ϋΑu  ?—Μ5˜t½ύ`!¦‚L  €› @[«V-K+ڜώKZ_©Q£†¨(–Ψž‘‘iσ½?"[Φ«WΟ2Ο ??_+›αΤsO½M~’   €;TuG₯¬σΒ%ΩΎ‹ΊΉEΔkƒ ηρ{DŸωGΫΟ’ŽX‘6AΐφΨσ³δΞλFΛ9͍΅eΜOŠy!c]ιΡW\#xαo4L™tένςσ7³-λaΜύφKΊΈL֝ώκΗ.ρ΄Z\;άFΐ ·?(hΡVS'©CΑd§;―-ӞzE{ωy™Τͺνk9Dœ›p.Β,D•5Ϊγ/Ύ+ YN―_s²³dΑ³εΣY3-yX)λ€ <ΈbςΝ2οϋΟ,υώ}pŸŠͺwύΤ,yZΑDIά?γψq―­Dp>σκ3Hzš9ςΔV ƒυ’tχߊΏG<ώ'eƒ$@.&€H·?πŒLθί ξφξΫ&§_ώPΪhΧLΔΧ ’Ω½;·¨η<Ζί˜„aοΪS£¦άvSςΒγ:YΓ3’Ϋ―ΉTρΆ`2=§Αΰ('&ϊΌωΒ£Ρ ΟΆαlΝσ¦gVŽτ—eHΐD]Nˆ;V$:,~k―ώs±φ;yˆτ2RE{kΰίPˆeS’β•#Ώ–ύ"―Φφ€vΏΨ:»Τι FζΊΠζ[Ϊψ½Xs„ΨPs$S΅ZU“COΡΞ=ϊΚΔk§šή Ah„σΠ|νΎόRΝ™WοΓMNΟ»c‘ΨcGdӚε²q͟EΨ`?<«°ež:>O΅cλ™G$@$ΰ8Ό3…#5ύΎ{~?ϋ]IŠ•©χ?i™χ‡4»5!μ_Ό/ϋwo350ωΖ»ŠDw58χΟ99ž”`Μ*ςΞΚΈΟξ7―]!ρ±Ρ–l8Ζ{Τ‰ΧM•ΘΆΕ―^}΅ ΞΧ΄r+~_(_hQn‘6Ϊ=Ύ`LrH€H€ʍήIΓ©‘Ρޞω˜όGϋW›rσ=2|Τe©‚ϋ’  € ¬ΏΏΏϊMΗΧˆκš••%ΎΎΞ±&”ΣE±g4Η‘ηΛ!"¬Ρποd`ψΔ|ci$@$@$@$ΰ-ͺzKG؏ŠAžόoΌσa™ύξΛv;lkP-’λ₯γΆ)&M;‘"²•6bή‘vΊ«IUοΏVφθ^‘νeΪΣ―κU›>οyμΙΜLSβ Σ'˜άυτΛHx ΫΡm¨ŠEIΰ‚'ΰS«Ά<ψΜkς„&ΊΧ_€BTα+–Π&ΝΤXJ;ž,ˆbK?αš[eΠπ1ef…ʈ:c© ΐτ‰–φ&N^§‰S™Xf©„+$@$@₯&PΗ―Όόώw2νΦ+M!”zξα©ͺ^Lz k¦&θ`.ΪZ›@pb5¦I€Όΐ˜ ΧΚo‹~”=Ϋ7YΊƒτΚ?IXx i,Bγr51lrRœμΨΌNpΏl4Τ1ψ’qΖ¬R­cε_Λ~5‰QΏόΰ5|ρ­-KU§υNψŒs5ž-lέ°J-8G‡†©{ύθCϋ-Ώ ŒϋB00υΎ§ŒY_wχߊΏG<ώ'eƒ$@n 0lΤε²xώwκόWη,wh’ΥjΥͺK‹Φν€±φ¬ΡΡ!X΅uο~Χ#3”c2}λΟγ&©k'"Lι'kί~>zV˜ζ18$L9Ι1ŠmΡώ#Ο½₯’ΖκϋχιMΟ¬Šλ'·‘€#yφMεάBX£αΎŽ§°8jKO½τ`Ό»ΰπ cχŠΊAtŠ6τΛdΖηΧ‘φδ½χ±NSWόΆM[,jΟvΑƒhΏ^ύš“Η:Ϊ1$Θ©ό\K9[+»φRmmCž§ŽΟSνΨ;Nζ“  8F““§Ώϊ‘άpΩ@ΓpέΰΌ ž£ϋΥk ‰š# [ΦΊ]g#WšŸ6Gδ•Ώ—ΫΎDΰΔM· «— XHγ¦R§nύσοx­žgιε―Ήω^iΫ±›žδ'   ”+„Έ£EΪ·Ι)ΰ@~SH€H€H ¬ j |b^~tt΄ZBCCMUc{pp°όη?QbΧ””IJJRyΖ‚¨#55UE‘Εϋ}1–α:    ”'σξ;Κ³l»ΒΈ~κ4ι¨ELuΤZ΄j+³η¬”1―³Ή &Ί~υΡλ6·9›yυwΚu·ή/ˆΒZZkΩ¦ƒΌψΞWRΓNψ€―ΟΏυΉ•ΒΛ9^8½ώΡO2dΔψv‘ϋ‘@₯#€H―|πƒšθc}πxy»~Υοr(j·MAμδ›ξ’{yήz·R₯•φ…·ΎP—lU`Λ)"L½οIAΫΦ>Μ# (;DœυεBAC[†θ1ΈN$'ΖٜTί«Pωψ‡?ΤΔ [ϋ3H€Κ“~ƒ>φό;Φ΄…©§O "΅όm‘|£9nYπγη*Β”΅ “ιο{ά5ΦT‘ D¦½Χͺ.8₯yω™ϋmž_¨ΪR’Xό&G4AέΰθΡK0±Dw”£oΓg>ƒΥ}zΥͺελϋΝέ+ώ1ώΥΉN$P‘ ΌςΑwθ₯E‘°πŒά·S–/―&Ε[ bqž½σΑgεͺλο(²―1ε^ύΰ{ݘ――#r,/±pbψψ ³Ž1oyfεLŸY–lΐΐ{άwΒρ_i χrο|±Πε‚X½?WέPόψΧΛ韞£"=χΖgκ½œΛΪ2άΏONTη·β±‘Bό3λ‹EšˆΦίVU–Zδ§―­SΝ[l§ Œzγ“9rΧΓ3“Ί\e­Ϊv”―•}‡[%&€½ϊαj2ΞMηώ9Wlyn$ (HΝ“ύg?/—ήώΒαh…~šϊ§_ώP]/F$@ήJq/|λ½O8τϋΗΤ(T¦Ώφ‰ΌΝ›ΞeJ{¬Γ΄\=ϋ™ο…Ε’\Wœ|πέRΑοτβ ‚8¬zkφ<·Lά,m{Ϋάύ·βο{δ™O$P‘ΰ9ιLmbw•LΉω‡ΚγΩτkšcB<*ir?žα|ςΓ2Ήdμ•Υm]Θ[žYYχ‹i( DΎŸ»|—ŠŸŽ"0cL₯=7Εxu—MΈζVε€"TGΝ“c[²InŸφŒvnύ™­Ύ7oΩF|ϊ5Y΄zΏ8ϊLΫSΗη©vl±a  €γπ,eφά•2tδxΑΌŽβ¬ŽΙυΖΫ’Χ?ώΙCΰβφwt[מύεηeΫεnΝ‰q=-ς»#†gCΈΗ\    x[‘b½©μ   „……IϞ=ΥoœœY»v­όπΓ²zυj%~-,,T’ViΥͺ•–žž.λΦ­“-[ΆHff¦δεε ’Η_Ώ^ε>}Z•σσσ“† mλHžH (ΌΛΐ‚g4˜GN#p=Œ-Œ1RΰXs=ߊPγ4Eτ9ΪΡU«VΙΠ‘C-ΕρΐzρΊC–4W*Œ΄²iνr‰=zDbQ‘a‚5–fΪΛϋa—^^ξΡΆΰuϋΠώ]’’/)‰Ϊ’}ž8ž$΅}λh„o UwύΆ΄7ˆ>}8JŽ9 G΅εX΄6.΄‘…10(DΑΌμ)iVεϋyΟOΊ€«ΐ­n/Ώ―E¬Π’“Ό“@b|Œ¬[ΉTφlί,ˆu*?OόI`p¨`bό ‹Η&4ΊΫN€$Κ†ΥΚqνά’‘žͺΞyMš΅”¦ΪΎΛω ξξλ—ΐ¨>¦(/ο½X:uοσo•ŠΐΟ_$³^~²/;ΐΣ8Ό™ΐΉsηdΛΊ•ΚιJ|l΄ΔΗDKrRœ4 –¦Ν[IΈΆ4mΡJ:tξ)~υκ{σ‘°o$`!pγεεοCϋ-ιg΄(N—hιi•~Ϋnέψ—$iχΔ‰ 1Ϊg¬dhχΔψνΪ€™„4Χ~γ6—Α—Œ‚Όέ–ύ:Gfέύ·βοoωK³₯%πε‡―ΛgοΎdΩ}ΰ°Ρ2σέ―-iT§eΗ¦΅Ϊσ•e‚{χLνωΚΩ³g₯Ύφ\Ϋwξ.}°psΔtgθdŸΜ”UΛ~Ρ’Γώ%8w""-~4VΧΛqJΐWΪηΒΦύπ–gVΦύbΊx‹η}+/?}Ÿ₯Pێέδ“—Y•u₯πΜΩΏ{›lίΌVύΎ>™‘¦ž½αL1F[·λ$½ϋSχ‘žδ„w2qΪ;)Œ9ΌδFdΌˆΦνJtγΙ1šu2C’φμΠ’ςΥΞoG%A;Η%Δ“œ¬“κž68$L½£ΒgϋΞ=€ƒφ<Ή¬ζ©γσT;eεQχί΄v…ϊθ#θy:u’nέΊ‰€₯₯ΙΆmΫdώύ*‚,ή£`§Ÿ~ZE˜υbεΦ΅]»vΙk―½&Ϋ·oWŒ^|ρE1bDΉυ‡ —/Όο\±b…,]ΊTdΒ„ ΆmΫςν[' Œ€pβπγ?ΚαΓ‡₯_Ώ~2nά8Zε!P΅ς*ΤΰΙΤεSάQ΅KκDd,ξ6ί:~IΥb‘‘ ΈŸ’9_uÝjqkφ[€χ²«n΄_€[H€H€Κ…<>υt±ZΚ₯l”H€άH‘­ΗNΌΞ-xWΥprRQΈϋoΕί#ήυ]eoH€JG FšnΏw‡#œρ“nPKιzιψ^ήςΜΚρ³$ Ψ'P­zuιά£―Zμ—*Ÿ-x'ρ2gΜ“c’4DWυ€yκψ<Վ'Ω±- Έ ΐiΔυš¨Ο›Μ§Vm8|΄7u‰}!      ŠζLuμΨQ‰1—-[&ΙΙΙ’‘‘!ΉΉΉ*‘ba υιΣG6lΨ ;wξ”μμlΩΊu«8p@9",((PcΟhΞ!ˆmΦ¬™ 8P f/(`n8ˆ΄œˆYη†°Jo €±Ψ];5ΦjΤ¨!!!!ήΠ-φ.(Έ>Α©ΓΕ_,έ»w—FI͚5/¨cδΑ”L€’Ψ’±                        x- Θψργ•€uέΊurτθQ9uκ”ηU­z^:R₯J‰ˆˆPΡ+!ˆEταΌΌ<%žΕ6]ΤY­Z5 –K/½Tzφμ)ϊΡl¨9šτχχWΌjΧMa–mL•&b=aƒ‚‚”°œc§όιy &P«V-iΣ¦όσΟ?‚λǚ‡^ΠE±^πG`H€H€H€H€H€H€H€H€\KΰΓ7žsm…6jλΩoˆtλ=ΠΖf‘              €g θ‘]Ǎ'­Z΅’mΫΆ)Ρ+D°ˆX©’ιυλΧO‰aW¬X!ΡΡΡJ{ξά9%βCΩƍ«±cƌQ‚O}_~%ρπ„ dΠ A11xΣάCίΡττt%φΦά₯i ίρzυκΉ-2"4Ÿ±@ΌŠλ5jԐš5k–Jˆ§Χ‹θ›'Nœ;vΘ–-[dύϊυ!Χ_½‡”Ν‘@ρπ]E”γvνΪ1Šqρ¨Έ•\F‚ς–-[Ϊ­Χ£C‡ ’9γΊg ˆ4ήΎ}{»ϋp ”–E±₯%ΗύH€H€H€H€H€H€H€H€ΚD Iσ–2ιϊ;,uDDΆ³¬s…H€H€H€H€H€H€H€H€H€H€H€H€H€H€H (3gΞ¨(| rόψqΙΝΝ] ˆ(~M›6•ΠΠP©U«Vѝνδ@h‹¨~‡–uλΦ©H111’žžκ‚Έ‰FήHΐίMΤ©Χ«βΨ1NτΕ•,τφ\Υ–«λsε±² ›€ώέ3₯ρ{mΜ·^ΧχΥΛW©RΕδωϊψΓ' Χ>½<ς°Oqζly½.γ~Θ³ξ‡^ŽŸž%@Q¬gy³5        ˆlΧY°ΠH€H€H€H€H€H€H€H€H€H€H€H€H€H€H€Š'pφμYIII‘={φΘ¦M›δοΏ–ΤΤTA> ‘ω4h mΪ΄‘]»JŸ>}₯aΓ%ή)fˆleΧ]*2μξέ»U;ηΝ+qίβκε6¨h %''G’““%--M­#c‘Z΅j*sύϊυ%88X[ Q3Λb{ˆΜŒφ @‡0mU­ZUΥ€ΪρχχW‘ Kj ϋŸΖ%P`l!".Νσl₯=ίΆH$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$`EβœψψxYΊt©¬\ΉR%‚7φŒαN\\œDEE)Ρ¨Q£”Nͺg, ±\fζI9tθ όυΧjΩΌy“‘^="„FΎΎΎΖέΈN|ί!x;|ψˆμάΉC Ξ!T…   ΐ"„CτeDcŽˆˆ:(Ρ^ύϊ ΄θ”η£V: Fooοή½ΪΈΫ¬ΪCΤηΌΌ<ΥvApΧΌysι₯‹tκΤIόόόl ΥQϊ  D|†ˆπΤ©S–ϊ ˜G}θ7„€8†š5k:Ϊe–#·ΐwΧ―yσζΙΑƒΥwξΘκիՁ(‚ξΆmΫΚ€I“Τ΅iρβΕ²eΛ΅>vμXiΥͺ•M‘+κΗψZΏ~½όςΛ/J|~Ι%—¨ρ`-’EYh!žέΉs§Ίžβϊ›™™©ΉΕ⺈ώEFFͺσ>Ρ7[ΧZ·‚«δ•S[ΙΏ<|                       ο%pόψ ωγ?dΡ’EρD²0ˆηͺT©’„8Θƒh‘.!ŠƒΈΫƌ£"ُBΏ˜˜q"! ͺƒP‘PέΊu₯eΛ–2hΠ ›"#c}\'ŠL 77WV­Z%+V¬}ϋφ)!,„oˆ² ‘Drˆd ΡωώύϋUΔfD“=zŒ6Vl‹Uν1γnαΒ…jμaάΫƒ˜e ~Gg]1Ž‡^DxΞ ¨’yDΒDtMœΠw,θ;κ;zτ¨:>ˆ/½τRιΦ­…±φώHΜχ(|10¦’tŒE\σΰδce𽆳€­BU”‡`Χ>”±g¨c]Ώ†―yϊ>Ψ?;;G9‰ΐωccη}LΑqκπ}Γ‚±Ω―_?=Χ]šgPλΞl…H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€œ"±λ];•`/))Iν D«ˆόˆHCΰƒνz4;ˆ^!Œˆh©E²λ¦DwΨΉ={φ(έΆmΫ”ψ":κA„JD§μΩ³§τιΣGZ·nMQ¬’ΓΌΎ―e AŽˆ­sηΞUŸ¨―M›6*:+’Β"š*Κ`|AT ‘*Ζ„r—#βjχξέ•ψΤ>ζ=zL΅‡ρ‡θ³ˆήΪΎ}{5–!Όƒΰ‚;l?vμ˜lίΎ]ε5nάX Y!tΥ ύ@τhDΏ―S§Ž:αSSSU[λ"%’»ξάΉS‰gq 0”Γ6Dˆ…8‚ΪΘΈqγԘG}Π’Τ·cΗyΗ !„†8Œ{G:rŒ,Ce!€λΎ―Έ>ỉˆΖŸγ:…²ΈF¨¨²eiΗz_Œ'ήW¬X„θh―K—.*J3ϊ ‹Μ‘η“ˆΰΎ|ωr%ΕxkΪ΄©ΪG?oX·Α΄k PλZž¬H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€\B ..NEy„(Β5ˆ„&Nœ¨"A€c³!Ϊd£FdώόωJΤ1’MŽ3F‹YUΘ—ŸώY‰λ±β#μρ_‹-€oίΎjiΫΆ­ΨΉδX Έ±±±ςρΗ›Ζ€½¦ x QbrDRΥ---MRRRΤχέΟΟOFŽ©„₯ΊπT/!-ΆCdΊyσf%ŠΕ˜ŒWβT½\qŸoˆμš ΨΒ9DbξΪ΅›ςΌΠUίν|X !Ξ½ϊκ«•ΠΦ(ΚCί!Ψƒ¨)μ6lΨ ƒ ’~ύϊ1΄žŸ^AίyD-5j”\|ρΕJΌkΖ*’€λcΐ•ΕΨ‚Θ}λΦmJHŽΘΜγǏWN"Π¦ΡΠ>Ε[KˆΎ:œNDFFͺΎΛsέ=(ŠuWΦJ$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@e"””¨’ζAP‡hˆ ‘P```1 D:C‡•S§N)!-’hBDtδΘ%ήƒ Ρ"!ŒC}A(Ψ«W/%†mם ς,„4πfψŽβϋ½pαB‡Ί‰θŽΓBόmΕBl Q¨yȐ!v£ΎBh a)"¬b,bA˜ŠΊ5τυθbvDΎ™!“Ft3fqH€*$γΙ‰¦~Ώ:ύ™υς“¦<&H€Ό›@NN–©ƒΣΊEͺΧ¨iΚcΒyy9Ω¦φξΨΜϋ?&H€Hΐ3N€˜οWί}ω)ωμέ—<Σ8[!  7ΘΙ:iͺuΛ†Uό­a"Β €§δη昚:rpΟG&"L xŠΐιSω¦¦0™‹Fφ`R΅έƘΕu Έ d€Έ ŽƒA•™@Ϊ‰”Κ|ψn;φ‚‚ΣΌs]VL$@ŽΘΜH,4  ΈP@ΒgΝΚ_“ΗA›@α™ž*φŸ½' †&`ΠHΐ[ߏ”€x{Ε™O$@$ΰ$άμ,ΑB#   o$ρ­ς€ˆντιΣJ1l“&Mμ bu:ˆ0&»vνˆvΆnέͺ’Ψιυ@ˆˆ˜ύϋχW‚;€h$P‘@܍οψΤ©SΥxpD$αg£FL‡ ρΔ΅XPgjͺ6/K«BΰΡ(’?"‚$Υ₯₯₯©(’’’=ˆήl9005`•€ΨB^D‘\·n&ΊK–εΛ—«•ΔB” AŽ λΘƒ’;£‘έ„„xΥGˆάΡΟ₯K—ͺ:‹cλHBB‚κ?Κeee)‘,ŽŸFή@γJbξ)G OΕbΑpέάΎ}»+_φ ϋαά€σ ΧXD`Ζ3|Oυέ^ί*CΎS’ΨΚ„ΗH$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@ή@ΑΝ ΠP9Ÿ»1Npp°!κ’[B€a"RŽ9Rϊφν«‚ξ”ˆ“ΌΔh~~~J\j/j²£έ†84&&F8 E_=¨„―ΊA ‘Œ#=Κ²>‹ŸΪk»fΝšΙ•W^)ιξή½[ΟC‡Itt΄βΈ°@ΐ Ψ#""€]»vΌys‹ΠmC¬‹ΎΑ ΚCDhG’S‚ΞX°„χ¨Ο‘}νσIΐUπ=DτΥϊυλ—θΒUmbLθγνc\EEE©sBIm`_}Α~Φ"Ms?ŠbέϘ-                       €Σtρ„:Ρ9"bEYD€DLˆω‚Ψž={ΚδΙ“₯K—.j»Σα$ΰeJ#L5ΔkGŽ‘ Θ† T€XŒ]˜‡ˆ•K!!!*b+"ͺξέ»W ζ06KcΎΎΎj,BτΊiΣ&UWRR’j‘j±ž˜˜(ˆH»mΫ6ΥnχξέeΒ„ ©ƒhβ\?DΑΆoί^m+‰ Ž †r8.œWτ<΅‘@9ΐwΪ::rY»„ο8ΎσΆΖςpΔ‚uŒ‰:HέΊum–·ξ‹>~01Λ*Τ·ŸiΫœΕΒ«€Ρό|kΙΟέkΜβ:  THΌψ‘€fœ΄τ}Ϊ'HΧv--i x?Ϋž~[ςςΟ{;BoŸΊσ‰lζύχςώΎf«|»h…₯—ν"Βερ;Ά€ΉB$@$ΰOΌ>[β’S-έ9e¬τνΦΞ’ζ   T4 —­—Ή¬΅t»[ϋ–ςΐM,i €§¬ήΌ[>ύω7KsΝ›4’χέ`Is…H€iξ'ΰ”(ΦϊΕJυjUeδ€ξο%[  7x²ζ¦ ˆεωΝ„„ πzΈ/Ι3τ²W§HιΣ₯­!‡«₯!›xά΄›}?žMD˜  Οxι£L uŒlΞσ±‰$@$@ΐώΓ1¦.Τη΅ΝD„  OHΟΜ65U―Ž/ΟG&"L xŠ@΅‹Μ―ξ]νέSΗΑvΦ@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$ΰ BtΙnέΊ‰u 4/θ»@εFccǎ*Δ‘M›6•+―ΌRΔ†‰O‘~A(Q,΄ΆBόζŒaD™ŠŠR‘`±?Δκ‘‘‘JŒA„ΑΑΑ¬Y3%€Ε'Ϊ;t萀€€¨Ύ’mˆψ €…6++K-Π’ΟFΫ/ˆ!>rδˆΤͺUKΥΡ»wo‹Π™γaY(OˆΘ *ΖDAAElnέ/] Ž1q¬u$W€ u”…Π|Χ]§O °+r…vοή½ͺ,ϊ-Ζ4"@ΣάO€’Xχ3f $@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$ΰqώ!„wνΠH€Ξ(,,TβTDz„@β·¦MΟGˆ΅ΕQ\Ή5))ImΖΨΒβ¨Al‡}-Z€DhBXˆ`!ΚΥ =D­Δφ–-[ͺνΕBΜΪ„ ΆuλΦj[bb’Š€ ±kχξέU”[½.γ'φ?zτ¨όςΛ/²uλVA”Ϊ‹/ΎX‰ψŒεΈNήNBX,pψ ‹b!XΗXFž΅AδzμΨ1%*Η80Ž7”Ε΅’ΨpMh‘«>Φ1ξ0Nτ²Ζzu‘ϋ₯KeωςεͺŒΏ6mΪPkεΖuήΡΈ.«&                       ς$ΑOυκΥΛ³ l›ΌŽΔ§Ηa|@(‡ˆ¬'NœPQ'­;›­„€Ώύφ›*§ Μ‘C›γΖΖΖΚί-7n”˜˜•g]„|ΙΙΙ*B,ΆA(λοο―ϊ ! ’RvνΪUE‹…pϋφν²dΙUΗc4€QΔ{φfdd(-D€ΞFΌ5ΦΛu(/ˆζŒ1€1Œ1―Δζˆk4Œ#ˆΑW\©DπΖmϊ:Ζ’.wξάYΒΒΒTΤYŒQŒχ¨Θ±zY|B‹σΕ† ΤΖyc°AƒͺcY»ΐΏάΧk&                       ('Ίˆ―œšg³$ΰu ¦kήΌΉμΨ±C²²²$..Nζϟ/ P‘±b“'³dώ}²zυjΩΉs§ΰAά A,xΦT{Š1’IξΪ΅K” ‘(ϋτ髉ρ«¨ΞάA`ΡμŠ+”€ϋ"2l«V­,Q.½rΠ AJ΄·mΫ69~όΈŠB‹ώτξέ[ΔπθgJJŠj υA `=€gϞΕΪϋƒ1ίk θQb1κΧ――ΎϋΕBŽHΚMš4QίkŒ#ˆ[!^έ²e‹Šψj/Ί3ΖJ§N΄±ΣG lssseνΪ΅ŠΑΠ‘C%<Ό™½ΩGՏ1qω²eΛTτgDcΗψμΧ―ŸΤ­[ΧkΉ]h£(φBϋ‹ςxH€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€Š%μ±cΗdύϊυ’““£D₯QQQ¨Q#%ͺ;s¦P233T€ID†D΄VlKLLTβXD‘Δ~ζA¨W’Υ«WO V!x…ΘBΦ₯K—*a+κEύΕ’NˆtαυΆiΣFF₯D±zΚB8~όx ΒXτkήΌy²gΟ DΣ„pQb>¬±θ„ΉW\q…ι:o½M~’€·€(<22R‰Ψ!φF4gˆX“’’4yM`^]#€Ι5((HΆ†+‘9"ΛZΖΖΰπαΓ΄q’D΄Ώ’nDsFYΨ!ͺ… cuΧͺUK"""dμΨ±Ύ}{‹hέΊ~¦]O€’XΧ3e$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$ΰrŽΨl•±•ηςΞ±BπˆF]e±7NΜ9#ΡΡΡZTΨ“²uλVU=’Αbά b,Δ₯ˆ*Ϋ­[7©]»ΆΜ™3G ξ Δƒ`.44Τ!1œvβΔ‰ͺD―„πnίΎ}*ς$„yΊ!r%¬M›6•!C†ΘΰΑƒ₯Aƒϊfυ‰Ύτοί_‰υύυdffͺˆΆο‘ΰ…Oˆq ;v”‘#GJΧ]•˜ΦT!$ΰB«ΦέE¬—]v™Ϊ„Θ­ΏBŽυjΥͺ©1‰1‚ο=ΖD­ΞbΌΫ2ŒΉ6mΪΚUW]₯D΄šCψ ϋΔz|Bh‹±Γ‡WQ—ύύύmUΛ<7 (ΦM`Y-                        ΈŠΐΩ³g%77ΧMV½Ώ;wNE‡ΤΕpΗεεεI~~ΎŠrgk?[yΨ">ˆιP/Κ“Dn-Z΄PbT|—±^Φο%’<φκΥ[‰^!(…0"UD’„(ΆFR·n]%β7DfEtΨΨΨX PbΡŒ9ŒτBΉvνΪ©~’¨ΓhH·mΫVΥ‹H—ά₯€€(A.ΔzΊWl©·kOp1,"ΏB0{θΠ!9rδˆςA°‹sŽβZDŽE]h³Y³fEϊeμ#ΧI 4π]ΓΈθΠ‘ƒΊV!ς*ΖOI†qƒ±ˆρ…:BBBΤχΉ€ρ ±:ΖΖΪΔΖXΒΕ8Bgb± ί}DfΖϊ©S§€I“&&‘«ήGŸš*β+ΖDσSˆΪŒσΖ'ϊ‘m`` :a¬#R,·%υWoƒŸ!@Q¬k8²                       p [eωςεJd„΄-ƒ(bΦύϋχ+a,"ΫA΄·rεJiΨ°‘ΙΩΪΟV„~5!Β„J4(O’=Zϊφν«D¨‰#7–ΆouλϊIοή½• /--M|χaˆ αD₯ΨA Αέ”)S’SŒ Œπ`HwκΤI5θ ”³% „ˆb=_!hΝΚΚRγb\ΪΕΎζ‘ν’Ž}„8bWD³EΔ[΄γΐΎθ7ϊfΎΎΎͺ―ό\Mγ’S΅ρ]ΖχίΙ’ ίQ\Ÿ&Mš€Δ΄Ψ/,,L™ώ9Ω^ψnC‹ς]ΊtQγRGΪB€‹1„1…qwσΝ7«λ Ζ„φΆ ωˆΠŒ± a,’/γ: ‘9±¨γγ uS k‹’ϋσ(Šu?cΆ@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@₯"  ]cbb䧟~²πμUQ,ζζζ ŽDBX`z”YGφ3–ADLˆλΠ>H€H€H€Ό…E±ής—`?H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€\JΰοΨ$Y³uͺ³G‡Φ‘u3—ΦΟʊ8›(k·ξUυμ(-š8)­hMΜ! ͺU« ’XΒt‘«Jxΰ?΄W£F ΄T±š8{φ¬)Cl Α°.@R₯Šϊ{αΣU14"ώΒτv\U7λ!  OНkθΧΜ‹.ΊH]/Λr_CQ¬'ώrεΤΖνΒΔ{fx€υgξΎVξΎvΌGΪb#$@$ΰj§L“μά|K΅ΛΏzEκΧ΅€ΉB$@$P± j SΣ³΄‡³ΥΔΏž_Ε>φžH€Κ‘@BJͺŒ™ϊ΄₯½:EΚΗΟίoI—e%tτtŸ©Šυ?½-΅}jšς˜   o'wκ΄4r₯›U/ͺ"Iλ²€ΉB$ΰ<ιο|%‹VltzΗ‹΄ Gυλϊͺg˜x;¬OιΧ­Τ¬Qέ‘Ί>Ÿϋ»Μϊz‘CeKSΘ·VMYχγΫEvν5ρn)8sΆHΎ#Υ«U•υκˆ]? ¨'}»΄“α}»hκ8²;ː  \ΐΞ=§M0ΊΘ«Ž“†OddI€φάή]}σD^•! °IΧƒΫŸy[vˆ–ͺΪ€Ϋ-σή³YξBΘτ¦χ-ΈΖΟxοΙΝ?-]Ϊ΄?ΎxY\)Ίώ^< p†@DD„ :T‰0ΩΟe!ΖlΣ¦ Ογƒ YYY''NœPλ………J¬ ρpƒ €qγΖ$ΎΎΎJπSΦΏΪY½zφχ?+>>>tYλεώ$@$@$ΰIp(QPP ™mΫΆ,Η–Φ(Š--Ή ²θxΒtΕΆ'Ϊb$@$ΰj˜άoŞ;η™s§«ƒυ‘  œ'•“'KΪ, –―—]QΡ’‘•cAƒΙ°-›6–ˆ¦!¦EΉξ²αβ[ΛΗ²+$@$@φ œΥ<βήY·ζaΑϊj™?ρ\ΑX7*䳆2ce$@$@εDΐψ\ώάΚ©l–. ψ]o}―θθαΕ&WEWlά)Ÿό΄D dŸΌcŠ\;~X‰ΈπΜΈ΄ν:?ίΪΆŸGΔ'§Κ™Β‰bΡξΡψdKσί,\ŽsH―N2σΑ2*… WH€HΰΒ'°eΟAωjΑŸ—$Gγ’Υsr< τ—vα2yμιί­½š|δ)'³sεӟ—JΤί±‚h}Έfœ)T…‘κ:…ηφS―-AυKΥ-O΄QͺŽq' ¨„’O€Λλ²ωo=)]ΪFXž\™=ηw%ˆE›SΖ•&!žlή£myΣϋ8hΊe(yϋΛω²S$Γω4  θΡ£‡΄hΡ’\ή£γϊR»vm 1΅?]~~Ύ:tH6lΨ QQQ’’’"'Ož][½zu ¦M›Jǎ₯gϞΌyσŸG—τ­ˆ—Ω³?S’θZ΅j Ϊ‘‘  @E"€ϋ Μ₯ΘΛΛSχW]u•΄jՊ’؊τGd_I€H€H€H€H€άGΰΗΕ«δ±Χ?Sήfm΅’™•+˜ „† 8ο=s·τνΪΞVqζ‘   ”@ΰpL‚ΌϊΙΟ–R!Aώ2γΎ,i   €7H?™-Σ^ϊH–oά!ŸΟ|ΘmQιΌι˜ρr’ΰS¦ΙŒϋo”›'Žτ¦ξ±/$@$@.&p :Nžυ΅:χ[WηγXφ‰•Ÿ[­‰PΙ‡3ξ—(ύ΄δ/yF‹ώŽk±΅Α[΄&άΕ²lέv™=η7yπζ+εΆ«GKυjΥ¬‹ΫM{’ »s  @σ—­“΄ΜΟϋˆZ^–’–)/~τ½jQbΈιŠςθF©Ϋ¬θΟΰοΊfœ|¦½“G΄Ψ>ό^Fι-6(5ξH$@•™@`` 4lΨ°άόη?τŠΘv›7o–%K–Θ¦M›$;;[‰zΆšφϋbδ!ͺλdϞ=+S¦LQ"Ω²όρ\Ύ Ε’=   TΈnαz–ššͺœEδζζ–ΉλŒ[f„¬€H€H€H€H€H Ό ΰǝΣgɜίΧ8Υ•Έ€2ώŽιjrΝτ{“jUωΙ)€,L$@$Pι €k“šjΡΩukΥ¬1E±: ~’   Έ…€OκΈQ1“Ώ΄y@Ή§NIvNΎδ:m³KΪ"Ώϊ‰ΌυΔ6·{S&žUT―ζΨσ ˆŠqΟ–!ΡW?•ΐueμΠ>ΆŠ0H€H ‚Ψw8FΖέώ΄ ΪΉ#κ˜[Ÿ”WΉU‹’>ά‘]œ.ƒλΟδ^”5[χ:Ό/D3Ο½ϋό 9Αόν³™R·NνbχυDΕv€I€H€l˜χΗZ›ωžΞ|Ns‘›wJ5;iΤ kT±’ΔVτgπΖh±ψ;<υΦ2{ζƒžώ°= Έ`P˜Z~JΜMCdΨΉsηΚ–-[δτιΣ*:uT|}}}U׌Œ %†EΩΈΈ8ωύχίΕΗΗGnΌρF©[·”φo,’υV©RE,;w.?l™H€H€JAŽ|eΑ‚A,"Ÿ—φΊ¨7οΨT½4?I€H€H€H€H€Hΐ Όύε|»‚Ψ¦!A£c+%x…'έCGγM‘dρΠς£K =ΉχϊΛ½πθΨ% Έπ \€½ΈΉt`ӁΒc;H€H€H€H€¬ tkίJ~ψœuΆΝtfVŽlΨ%o9Ovˆ6•ωfαr™zΥhiΣ’‰)_Oά}νx™ͺE¨+«ν=tT.ΏσY9SxΦTΥ“Ηšφίr₯LϋοD{›‹δCœœ*¬Ϋ&ο~³PE4ΊλΩwe@χRΟΟטΝu   Nΰh|²\yοŒ"‚XίΪ>£Ck τΧ 9‚H²ΗΞG–Α!γϊt‹Jγΰ†2ΈW'—Sxι£Šb/Ί¨ŠD4 U‘jΦ―+1‰ΗU”Ψ„”TSϋ‡%Θ=ΟΏ'_Ώϊ¨)ί:α‰6¬ΫdšH€H x?-ύKφ:V|!lέΊηε2&ΪV†wΑήψΎΕ-vΡς rΝΨ‘2΄O|Ψ   ΈŽ@VVΆόρΗ²cΗ%ˆˆˆ &H·nέ$44ΤΉ5??_<(K—.•Υ«WKzzΊΖΆkΧN³Ρ"Κ–NΎQ,–ͺšŎ;ʈ#)Φu^ΦD$@$ΰη4ηΎ‡–+V¨k)"­—ΥJwU-k«άί#ΰ)»yX£ΫΚΝ?%'OΪ,W§v-i¨yŒ.ΙκΥα‹σ’q;    €{μŒϊ[^ϊψGSεπŠχΚ÷Ș!½‹άΟζhh_ψΰ;ω|ξ驪ƒ―6G&Œ ‘A¦Ί˜  p?šΪ3Œo^Μύ ±    JEiFι%Γϊv‘»Ÿ{Οα ήϊbž|ςΒ6™T­z‘6Α¨lŽZ2NζΘνΟΌSD;jPOyδΦI6Ϋ-kfmŸšΊy˜Z.ξΧMΈλYIΝΘ²T›Ί@0AύΆ«ΗXςΈB$@$Pρ άφτΫ¦σ=ŽNξќ<Τρ­e9@8‰\Ύa‡άχΒ¦y"w=;KΦόπ–ψΧσ³”-λΚƝQςώwΏ˜ͺι¦…ΌϋΜέiΓ)ΕΥ[δ±Χ>“€ι–}έύӟ–Κ­W²δW<Ρ†±=“  ”L`ΙͺΝrίσ”\Π%žxcΆ₯•Ρƒ{iNB,ι uΕί·ΰ·ω5γ†ΙΗ?.QΨΥχλ~|[jT―v‘ώx\$@$@ ƒΘΆmΫδδΙ“*β+D©cǎ•:uκ¨θ­ϊ!#*lοή½UTWb7lΨ III²lٟ«WoM[G/κτ'œ|`0Y   T$Εβϊ₯_Ο\ΡwŠb]AΡKλθΡ±΅l™χ^‰½›ΏlL}κ-›ε&Œθ/―?v›ΝmΜ$    o πΕΌ?δŸώ1uεέgξ’«F 6ει‰:šg|f{jχ˘(€&…>ωζςε+λYό$      Έ`Rπg3§ΙŽ¨#—tΒrD4±Ž» /voyς ‰Χ’Ά­U³Ζςώ³χͺΎΖ|w¬# ξ’gHί«ξ3UwƒŚ0A$@šDpi΄Χ*7MaΜRλ˜p§ Ώ~όΌ\|γ£–Θ²Η5Gκο}³H¦ίs]‘}J›1}ΦΧ&Η”-ΓCeιμ™RM›ΌkΛΰ4Qmϋ_}Ώ€ŸΜΆωΰϋ_μŠb=Ρ†₯#\! °I 95CŽΔ$Κ¦]Q2χχ΅‚θεή`Ώά(;D[Ίrϋd:²ΐ(‡•›&Œ΄ˆb΅ώӟ—Κݚσ   T»wο–ΈΈ85G- @ϊφν«Δ±ΆϊΡEIdd€tθΠAφξέ+²oί^ΙΟΟ?Ώ‹bm΅Ε<  ¨Hΰ΄Ρ•VΕ••±.¨  Ϟ­ ‡Ιc$    ABΦ_Vl0υυωϋo΄+ˆ5DTΨQƒ{³dρͺM’“›oΚc‚H€H€H€H€H€H€HΰΒ Π·k;Ӂ€h“·ρlΑ6γ½oeΝΦ½¦ͺύ΄H}ίΌφ¨ΐa—§ "άfƒMΝΕ&7₯™  ¨Ψ^ύτgΣ λΣΕ¦ ΦX¨E“Ήο†+ŒY²ΰΟυ&«i£“ Μ«Ψ{θ˜i―ΈΙ V/Ψ°A]™ωΰυ€ϊ„ƒ cΤs}£'ΪΠΫβ'  @Q/}όƒ4tt}«\qΧ³‚λ‘·bαPyζ‡?X:έ‘U3ιέΉ%ΝΟ@”ށ=:X~χλ…nϋ=ni„+$@$@$ΰBp2"Ύ}{iΤ¨Q±΅Χ¨QCUΔXά›deeΙιΣ§‹έ‡I€H€H€œ#`Ϋύ’su°4 ˜œ&ΐλ›-»ξ²α–μμά<ωπϋΕ²uο!υ@,Uσ<Ίgρ'RΏξy(πόΉτ―-–ςΖ< idΜ*²ΎvΫ^‰I(ϊRΏ~]_3€w‘ςΆ2ςN–ίVoxVέΎοˆφ’ε€δζŸ–Z5kHpΓΦ¨‘\:°‡ŒΦκk€₯i$@Ξΐδ›=‡ŽΚί±IΊy˜ΐ°#†—œ[φ’xΝ« mlV―VMΒ‚ < c’H€H rX‘έ§αώL7ίZ5εΏGκΙ?οΉφ²"χœQΡ±«_Š–H€* όώΖύvt\²KH–S§ΟHkνž9R‹…{gόΖ½ΠlχΑ£rψX‚μΈΤς©)]ΪFHΗΘζζXOœ‘ΏΆμ–=ΪqOΛ”Β³η€iH zAŽ§J•ωΊ;SX(‡Žjl’O¨(ixώΨ ž4 l =ΫπWߍ:šXƒF$PyˆŽSηƒ,νzθ/αΪsΧΠ ·D0tφ9Pyœ³‘!˜[ηφΫκ­r$6Ρ΅k s-™¬”Η³)Ίεφ"ξš“…(φΊσ(Ό›˜σΫjΉαςKά~’Ζϊψψ»OqΡšχΐϋΏŒŒ Ω·oŸμΨ±CE†:44TΊuλ¦ΎGˆ"L# p Šb]Γ‘΅8A`φεgί΅9ιΗ‰jάV47”άςΔ›*:¬3ΐ³ψͺ;΅I½Σ€―ζџF$pž&YΒγ<¬@ϋ1ƒ€iόΣ%%5C₯ύοΩsδΟηΚ/=uDύ+ΟΌσ•Μžσ»Όυδj2‘q{iΦOώΟrΨμEηœͺΗsΐŽ¨F$@$PvIΗΣM•`ςΎ3†Ι5υύ΄θLΩ–έ…œF$@Δ‘½ςqgTΦΗ…ςpψtύΓ―ΘSΖΚ3w_+Υ΄8²Οζό&½φ™©Ί‘Ί›±Ψˆ~#W!οŸώΑGΛ՜qΛΒ›8μΩY_Λ{šΐ˞‘Ύ½‡ΙΈΫŸ‘Ηo›,χέpΉ[’ ΪkΏ€όΝ»ȍΎfΣ9—υΎπp?υ©·”xΡ‡3JtΠ΅πΟυrη³³Jό-…v0Ο8°L5Hޟ~ONp&fό}cέ?γ6DΉ₯‘ xš0σΦ'ί”Ν˜=;“(χΏψ‘|₯9œϋξ3’¦ λΧmw²=ΞΆΜρ\}Ξ2φ³<―!ί.Z.½ό‰œΥΞ‘φ ΧC8avέΓς‰φœωRm’ Hΐ΅Άμ>hͺΠՎ 35ΡνυΌb™ΰ«7vΩπ~κώSO{ϊ3&Ι|Ξτ―η°0ΙΣ}e{$@$@ΞΨ 9°1Ϊΰ^:ΗχλΦή$ŠΕ3₯™Σώ[δY€± GΦC‚όMΕ’N€Λ)Ν)šOΝ’'~¦€eš±¨¨YγF¦ϊπDEe  €…ΐ­“FΙ₯ƒzZΖ<ƒΉcϊ;Ζ,­ΓωδΖ]Q–φΰ,³~έ:–tI+ξ|6URΫϊφ²<ƒwζ}‹ή>=1' οŸ‘^‹w6xGΥ<¬θuήΨ7“  €7@„OG£|βZAdTT”œ>}Z h»wοnS‹y (oo~Ž‚XˆqiΞΧβΨ:_cρ{ 2μΝ;εΟ?”δδdυwƒχΨΨXIMM•κΥ«KxxΈΓί₯β[sl+ΪΗB#o&€ί18–‡α‹1BηεAΏμmΊn–cΩϋΒ*x»οωΌV‹—0γ΅Ι±»-Υ_γΈiεΪ‡^–ί?‰žόKE;UɚvΒέ3œΔβΙ΄—>RΡ’œa«MτΉβgeΦΣw ’GΡH€H€.<˜Hc΄F “%ηh/dΣOš£ΔΈzBl‰` p1D¦‚(LDx€‡Ιοω§NKvn~‘V?όώWΩΊχΜ{oΊ bVYνγkbΏ0U3zp/ωlζ4— oρ`τΡW?•Ωs7΅e/qξά?ςΒίΙ†ϋεϋ7žΠΌΈ^d―¨ΗςWmΪ₯„ΙΊwt놫jžCm‰¨π·ΎϋΉwε›Χ³ήΕ’ή΄λ€]AlMΝ#{m2„ΕΦίTπσΥ*Zολέf©+$@›ΐƝQ2yΪLΑΔDG ΪΛοœ.Χ]v±#Ε‹”qφ9§ΟYžΊ†ΐQ$DƎwiΧς5ZΤτΖΑ ݍεH€J pB{‡³ΙJ;V‹jο*ΓΛϊ[5Η%qVΞΆΪ· —YΟάεͺfJUΟ!«HzMBKUw" π.ΈφlήcvψgdΞXξνMNΖp½ΜΚΙS‘Z©ΗΊldσ0SžΗlΨεPΤτ_Wl4ν‹η"ፃLyHx’"2ƒH€HΐB ƒm‹-ƒΓ ς2.qέΡ­GΗH}΅ΔOO?›*±C(ΰι9Y½;·±ˆbqx«6ο¦(Φg6A$@$ΰ9xχtπΰAYΌx±ΔΔΔ¨y!!!2dΘ›ΡeOž<)PEνυB­cǎ)­―――½bΜ·"pφμY%LMOOWΒXOήdΓ† J«‹qρό’fόύόό€S§NR«V-θi₯ΆAƒδQ!՟‚I(–Ζ DγX<νηŒΗΰΰ`5>‹ν(7z%Šb½ςΟrαvκžοKΎ&<υV{όΩ₯ΔκΗ„‰ΕS˜)+Ύ~­ΔH-ϊ>Οήyΐ7U½oό₯{O --PJΩ{ΚAάώ]ˆ{£Έ·ΘOά{NPDˆ"SY²χ¦tPΊχHώο{0!7MΣ€ΝlŸΓη’;Ξ=γ›ζήδάσΌ^A )Έ‡Ε­G3ŽkΊ,})­[q„ίxΝ~ΓΖ+³Ώ«!ˆgΡΣ­WL ή]ΪSWŽκx$ύ8­ΫΊ[-΅QœI ιξ™oS|σh5Έa^A@ ›ω^pιΉg{Σ―{GγΊ-+›vξ«1ΐΤ£SŠ-§"€x$ϋOyπΐqΔΐžtοuΡ@ŽΎ-ή’dBΚΎΓιτΚμoιηελŒ}Y·e7½ΓβΨ{8C[s~ Η^ύDS„Έc½σԝ‘Ξωαw**)Su‰Θσ*n‰ƒ‘όΎΑ«8€KίϋrΙΔNCZΊϊ_šύέ/4•£Ω»3Ι{vΣc―jάΔd²§φΉfςκΠ6‘^όθ0ζΩΆη =ύζηκ½3΄yρ_kιMvΘ½υΞ7μ2Ύf²³Šοͺ¨¬2ξ‹ ₯w]CCϊv£Φ -ŒAE»pιΕiχ£Ζό³Ώϋ•Ξ5ˆF μeά7aτκτίδVΙ;σΉΖc­ZΖΜ{¦hΆXp+B·GjsAμπ=ιΒ±§SΫΔ8ςα‡>ξΨGπΔzΉ&HΪΌλ=φΚl΅nοφŒ9λše­ΝΈ‡Θ$Μ[ŸxέΨ Ή?‰“ŠˆδRΪ΄’ƒG3h 3ώˆƒ;ˆΈ!Ι8σσ|M―>r‹a^A@ΰHz– ֘WPd,%™―{7_~žq»‘+3ήϊ‚ώδΙΌ¦I|ϊάύ*Јι~WΛ5F‚©˜¦ Η7έΔ:€€€—Ψ±οHοχ=;Ϋ7Ύ-ξyζ)+'―Α’ΨΨ¨§+%;εΤwΗΣoΠ’Ož£;¨-ν=tŒžzγ3Να«&Ž¦°`Ν>ΩpE5*ŏ'°dΥMεو-ΙcS΅΅Λ•cπž“5˜Ώ˜¦?Vo€λ.kΊ λ   ΰ΅DΨuτθQϊϊλ―ιοΏ&q ŽŽ¦Ρ£GS=Œσ5L;˜‘‘A?όπνΩ³§V—Bm•””¨%<<άτt¬[! bΨ?\Fϋχογ\z+9q¨™ͺ£ΈΈ˜23³,ΊΣ–––ϊυλ)--cύxΞ’#κ΅VΖIχΛΤΤT>|8%&&Zˌc ΰ6ςΉYΎ|9νΪ΅‹*YΜξτ«τ”?²ς©•$bυΠ‘COξΐ^E’X―z»ΌΏ±βDΰ©iΩ?›kˆξLΫ*UEPמ'*₯³#ΩjvZωbα" 9η`Z&½Ι€Ίι2Σ"°Mž€ ΚN±’d²ϊ“w^Mη1HΉUΥ'ƒΟyυ“ωšΓβ.υΚΓ7S4O(2$™Ψ-“ ―™4†.7‚`ΧƒΛ’L>ΌwΦ{τχ7―;Τ‘ΚP7^A@άG`( zκ›$2ί <Ιά4‰Ι<ΊΌιq¬ƒ€€§ψuΕ:Ϊ$έΨLΉN~ωΚ#5„¨ς]ΊpϊόΕYϊ+έϋμ»Ζs^gε —Œ§π°γ>{V^ϋt>O^ό\sΚE<ιώΗo³ψG“±A¬~xη)κάΎ¦Ω–EΔ²ήώ€w2ΜzKΊˆ?Θο w₯Χ?[ œZ υΛo₯OŸ»―FPQΙ2zH_ΊΘ~ZφαzαΓo” Cά€MΣΚ ΫΘTψΡ!9‘Ύ}ύqjWsς©L$•ίSœ5”Ξ½αΪ±ο°±¨6οˆbE¨+‹€ΥQΪ±ž°Π`:—η!xWωϊ,xC ζλΝ'721MDbΏXO7?ώͺr‰2Œ±Ξ΅εΥήq g]³¬΅ΥUχ N Χθϋ―Ώ˜n»r"ψ›%άΔ!cΩΓθβ;žΦmό™HτhCP γIX¨“@yE%επ5ο0‹aα)σύ© H―>z«ζσXg‘V2H Fω^gš|}}θΓέCξte•ο‚AM"ψ―qό=’sFš6λ  ^J@Ύs›'Ση§ζΗ,mGFΤt˜ΙΞΙ§ŽΙI–²Ϋ΅ο₯‡n¦WL3oΛδ`ec―{ˆΌρRωž+γυŸΜM bM„…Y ήζŠ:μκ42ƒ€Έ€Δ4MΆŠbέ16eΪNΣuWΑ»cNVr-Οδ«xΜKςu[ΥoV?Le6ύΐ:€€€χη9ϋχο§ ΠoΏύFΉΉΉΚ}P„VγǏ§Ϊ^EH+ΈΦάE+§’δχ3’mΣΣiσζMtψπajΖΟιX"jΫ‰ Θ₯ΧλX{ς{Žy1ςήPqq 2ΘρΜs9n[ΟΚBFΔΈ;v„(ΦqhQ’ Θη’°°ώύχ_Εξδωm~όImΖAX‰Ε’ψΒ—Σj^)**ŠN;ν4UΏΕμΨι±πKcߚΖί0™$“1{±—L(*)ε Κ©Ι@&πζœοk­ςužq;΄˜&q$»š'‹^zη Κa‡σ$47_vF΄gžΫ ΠΤ~Œ΅mGs^| ΖduK<žyk—žt}’γ#Oλ₯&lZΚkΨ7|@šϋΚΓtΩ]Ο…±‡ΩjξΒ?θͺ Ξ2dΓ+€€@'π6;!XΏUCαjΎO˜NPΧΔ€xM;χkZyω„Q5±š Όqεωg‹}Ν N°Α­{’y΄nσσ,mΏόρ·τΜΫ_h]ΖΏŸΕεΞ\°©Ι䀍9,π5ĚNίΎρ8 ˜|›Q(š›_Δ}†žβ€=ξH2αδmͺešfN›RCkz\D³/>x“rZ1^ ω‘Ιvκeζ³qϋ^ΣSι±Ϋ΄(ˆ5Ν$aqΎξ‘»ΧnΩe\Η €€wHη eοΞύQΣψOŸΏŸΞΤ[³ΟtcΜ°~τλΗΟι—έ­qœ6Νcmέήq w^³\qΉvςΩV'σKp‚7Ώ†1oC³άΫϋvλ`Ψ…W& Ώεγ_l•…ar­₯L]8`ΚλέF½»΄·tΨξ}Ϋχ’;ž~³Ζyςστώ=jμwφŽͺͺj:t,Svq‘α;£Τ+ίΙί›q7…s $ο'WxΚέΠ›h "WΓ1K―(BD(¦²sOΣ±tŽ­ϋRΫΆ’fήCΣfΎMΗYh+)-3›n{κ zτΥΩJx›Π"V±Ψs0d|Γ4I ‡yόΜ7ŽΙΤ–\QGmuc?€€€ηΨΙΑ±Ω…!ΕΕFQλ„–†M«―ξ›²Ϊ0'tǜ¬`ώξΡ«K ­ίΊGυLζ„I`Ξ†Γv""   ΰaD8ZVV¦’ςΚπ,ΚπjksEdBΑΑŽ'-//§}ϋφΡΒ… iρβΕ”­ΚοΧ―Mœ8‘RRRjmZ\\œΝζδδΤκ+γΊ‡’ΉsηΦZΤ$PQQAςވ±’―oωIπp'*νDa§+fκρš1μiζΛRqό^φ8ηU_MϊŠ<ͺ¬Θ§rώΜ$πDrύΡΏ\Ϋ«ω5Δί—š7#‘•ΙGΚi‰/eD%:2\+$H’χ€(Φϋή³FΡβΛY`ϊμτ©$Xυb·f½7ΟΘGΤ­C[ϊ?)Χ•ZΔD²σνhzνΣSnd+YbM+‘ΧΟΰ CΆ€žfβΪ#Η¬<΄±₯@δp;gίΥ^oΞ1Πͺ ΦΠΰφmZ©±”ΎZlΨeχ«­γ@¦]yΝrΕ=€%Oΐ|δζΛλd'Α$€€w3$™Θ Q¬^Aΰk’ΧSΉjI€‘Ϋ<ŸztLy°{DpzΥτYTR¦PrΙψ‘tγ₯η̧֣|πυb~ή΄ͺζ³=29@άqΕ…Ο$0 &γ룇τ5;›  ήJ —οC¦Iž₯Κ½Ξή¦qTΟΚΙ³·ˆZσ>€χξBπXΜΌ0ζ“@eoΪiά6_> '½ΑA,ZΕΙ„UλΙuXoŽ‚€x ν,Š5MmγM7­»klΚj£œxНs²Ϊ°PΩ Š•.Κσ ˆbψf£h&Eΐ0.θμησ†ZRRBYYYtδΘ:vμ˜r“Έ³ϊlk›$ΏˆLLύϋχ·υ΄Zσ‰ΓαϊυλiΙ’%΄bΕ :~όΈΡqpΒ„ Τ·o_d²²–KC† ­Υ]TN“χsΛ–-4ώόZJΑnKδ}vzς%ίΐX ŽνΙٜη+Ξ¬UΕG©ΊT‚΄XΧρ ΎO Eu&ΏΞSχ|Kύ²eŸΊœ* vQue!ωψϊ8=ˆ½-mB°D@>£ςY•EDλ>Τ₯…?5ς!ΛžΛ–J±sOakΖΏŒβjΚaelΉr¦mfυZmg ΘξBΕΊ6ͺ:I` GğuίυNŸkοUΆΥšύ/­υ˜Έΰ¬a4λέ/iοαc5ς­X·’ΨT°£©‡’#Ϊ6ϋv01ύΑzOoΛ‚%[“L&7(νD@IDATΕΚƒΥ²ςŠz= Ά΅Nδπlι<‘|ϊ¬χθηελ4 •Χβ€Ω π6~~Ϊ‡)σύI“ΞFώBΟJΊwκE$K}ΣޝK/~ψζτλ8HΔ’k¬³ΣuŽ³Ή ΤυψkŸςC₯“!Δ εΐΡtκΩ©φθ¬6nGFω­σΥOΛ4g\4n„Ν#DΜ Εn£[˜ )οζΛΟ#q&”dσo‘όΒbM9Ψπnr½ωξ—εšNάx™νB­»™LŸ-XBε•š2lΩ°gΘ]Χ,WάCΏψ g—+[RR\s(Vάb‘@G@Ζ‡o}βuΜε…ϋo QƒϋΤ»pω>yΓ£/ΣΑ΄LM}8 ͺ­Α5'Φ²‘Ε"WY’:₯΄¦Ÿ½W9ς5€œ   ΰYςXXjš’XάZŸͺΉΧdηκZŸ²,³]ΥwπbOΪΊϋύ΄μo5amς°‘LWΤa¨ ―  žKΰ„™ΫyTD¨ΝuΧΨ”Ν tpFwΞΙ2wΆ7uχup7Q€49EEEJ<CαααδηηύRΔnΨ°A Nwοή­D§Oq”1ZΣ9ΖΆΌα’_ΔWQQQ ΕJ9ΉΉΉ΄vνZϊϊλ―iλΦ­J¬J#FŒ Ι“'SχξέIΆ­%_,Ϋ0_-((Θ%s/¬΅Υێœœ!κ7?ΖΆΰWη‰bEγͺcgVeF[‹ήΥΗΗ—|ό£ΙWΉΦZΞ:†r³Κjζΐ3"ͺφ!:ζ 9¦V”υ# c'Ητ$Σήbƒ|)>¬±Ι³“žΚ«š©Ο±α„ά/mƒtbƒPt= x7zv§Ή‡€8™ΈjB¬==¬-iϋ6 ”Θ“κJ©ί’(vΥΖΪΕΆu•‰γ ΠX ŒeQ¬­iχ£š¬ύΊuΤlΧ΅Ρ19‰bΨ *'ΏPe•Ι›;χ‘ή]ΪΧu*Žƒ€42πγo‘oΝ‘’’2MοB‚ιΗo§Α}Ίjφc@Ό‘ΐ°~έiΩ?›M_ΑΡ΅/Ÿφ?zž'ό·K²="Ί±VδΪ*ά¦ιΖKΗΣ3Σ¦˜ξrΪΊ?L΄Α%ΦΠ™θΩ·[*­Ϋ²Ϋ°‹ΝtΉ(6=+§†ΐlΜ°~Ζ6Υ΅"χ­?dΜζΗΛΜΣψ‘§™ο²i{ώ―+lΚ‡L ήA #;WγJ-ΞQύ»Ϋ>ΖίφsPSqO5MβΘ|ρ3θΡ[ξΌϊΣC6―?σφ\ZΊϊ_M~iΫ§Οέgs0ΝΙNάΨ{(ή›·ˆfΧjσλŒ«EΡ  N&PP\’©!:Bϋ]RsΠΚF€ΏvꐏcœΙύψž½KŸ.ψ­ΦΪeœΖRp.yΎϋΐ ͺ—ΞΌ‡$Ÿ₯δŠ:,Υ‹}  žIΰx.‹LRTΈν#ά16eT—―ΊsNVt”φ;‹£rΈ&*"°wο^š;w.υιΣ‡:vμHqqqΛ™ΑΑΤJΫ›RYY©Δ¦sζΜ‘Ν›7“ˆa ξ°ς{Π°Ψ^βɜ"Š±m}“Τ›‘‘AΛ–-£Ε‹«Ά‰s­°:t(]rΙ%ΤΉsg!«£’Τ‰dΐΐΐ„α:Φ©²η€ θ^s^…}₯ZΙmsќQ†^τ6Ÿ`₯Zq_υΊ“>›"ˆm ωZzŠέ^N@Œ”€…±r•«bͺψ;Λ‡ΔAC”5ΙΗO>Υ\ WΗυΐ%Ά$/Ϊ‘Ωφ’†£©ήI`Μ°ώ6;žΈ²‡»j…w†ΊΛΚ+ι–Η_3lΦϊΊmοA‹Η$r7\)-’ΑΞ&L 3G€·5™‹Ν{uN±υTcΎ.νΫΠJ7θ#ιYΕι`@š„=τβG΄έB$z™œώ OV•ϋ€4"ͺœυή—TΕΑ ιΟΏ7ΡiήN§υκLβ.:¬_7κΥΉ=zk]e ωνy}ςυΟθυΟΤ8₯g=Ύ»Χ(ΔΖ=;·£`p`OκΗΑ­LE±ŽfΨsΊCςHΣΦΗ μck’a >֐$‘s3Žηaώ€–΄,ZΆv3­ήΈ½!Εβ\#pΠμΧ— ¬;ˆ›w!1ž#'Χ#Ω3TWρΞΈfΉκΊUΛΊΊg<.’eΣTUεΤΈ¦UaΌ†€|gz“ƒ[Ω“vq°ΔΟΎ_B|΅Xσ]ω™·Ώ αzΈ»Ϊ“~ψ}5½ϊ‰60Œνō5‘e¬=EΥ™7,$ˆBmp )EΕ₯TRV^£Μj§=ϋ»_iΝΏ;θ»7Ÿ ο"€x?Θ°M'*Ω%§>©œηE˜¦Ψ膍7ΚQ«Ή 6˜Ώοήyυ$sz?jί¦•ΊΗI ‹}GιηΏΦ›Ÿ― κ#γZ·>ρ}ώ⃆b5―¨CS!6@@<šΐ 3Ql€N±uuΜcSuΥιΜγξœ“%¦Ι\Μlz λ  φΘΜΜ€E‹ΡφνΫ)11‘RSS•@ΆmΫΆJ°ι5Nx"=vμύςΛ/΄jΥ*ε +4δ9yHH²mO’{ΊΕI·>IΪuόψqΕωϋοΏ§ƒ*ΗZ 5Š&MšD:uςΞυaΰ-ηΘ߈ Eυ¦―`™AlηξΘhΎ$ΓλΙ-Ηύ/’BvQζ>³r\ EtˆžJ@Zε›aεΡΈαρΈ³>!ͺ\ώ―RVψ3βΛΒq|F<υ―£ξvA[7#δp {"β;°Ϊ:‹ΚΛ/²˜'-3›ΎZΌΜβ1[wžΘ+°ΙmΦΦςΌ€­“(ET.xLΣΔ[WΡ8LχΥ΅.N¦)+GτΦA@q}yμΥOθΗ?ΦΤθ˜DΎΏγͺ θk&yœ{KΖb€ΨA [‡dϊˆ'βOyΰΝdy°#BGƒΨQ\²ΕiTD²gιCυω½Ύzγϊkν‹­{δεiΤ > mZ,άlgb\s³=uo67›άyΰhzέ'98Η‘΄LM‰ρ-b4ێިΰ(Ίλ·ξ! ρχ¦$υΝΘ¦ϊN˜utϋP€€σ˜‹bγΨωΥήΤΊž’X[ǁΜΫγͺk–«ξ!mνŚ³ΐ6€€ctβ`3ξΎ–&Fάς—–©‚ε{ς3ή’ί?}ξΏξuΧ·sίaΊύ©7jd|vϊuˆ¦Kύ έ!γΣ¦\hs1ςύn·qα5τΚμο”[‚αδ,žρΦzνΡ[ »π   ΰΕZΖjΏΫ˜=΅΅k₯ε›9w8iHjwι8ΑΡBCCI\RE|ͺΧ‰@”…’δΫ4ή€―ΧKΝ“.œήκνθΏ ”η™D+Α||}”TΌœ…μβΰκΜ$.±ε_PO>κ:αHgogΆeΧ$QlM&ΨγDζQΎœX•]EηΩ•ίžΜ₯eΪHφœ‹Ό Π Ψ“iο‘΄έ—AΩ†¦r³‡Ί -烀x€|ϋ‹…$n/εΪχΪ‘§υ’YΣ§ͺ(τžΧz΄@N@&~ϊόύtίsο³θρΈΕΕ=κχΥΥB/_θΪΙcιΞ?“ΒB‚-žcΎΣš˜2—ƒO=Θn ̜f~šΓ·λ3©%Ϊ, ω±¬oW]š»ΣΖF7|Β©₯:«ωaΗ< υοΞ«xΘR~Ω'Nd‘Hmy°@ΐ{0α›O:³₯'IρΝmΙV#­γ@†]}ΝrΕ=D"PΗ:@T``„W†θΫ­=3νZΊλ™·mί{ˆƒΗμPŽ±Ζ΅¬δ³ΨθΚι³j|WΊϊ‚³θκ ΖΤr–kwK4힝RΤ₯}Ί™…EβkHίόό=uηΥTŸϋ‘ Ό‚€x±ZGΧv\­O*5swΔΕkŸ.Π4%2<”>δq’ΊΥ8hΨϋΟL£α—O#ΣgΓ2Ξ4nψΝ„mWΤ‘ι6@@<ž€˜W˜&ΉΨ“\=6eOΫ™Χέs²ΜηžΘƒΑ#ί_” Π΄ ΔΖΖRοή½);;› (77—rrr”8vέΊuΤ’E κΠ‘ƒr2mΧ%$$(¬#‰£¦%iΏzΣ”πTŽgœq†žŠΘWΪ,Ž―φ:ΕΊ(ηΫ› ±βΖ+’c\Š φ’‹.’!C†P|||Σ\Ϊ ΞMωE4-B71ΧT]Uωœt£tS“\T-«ύX¬«c*=‰ Ά>ο.j,ͺu £Ύ¦V³Z΅Όκ€(ΦW>»Nΰ#sͺͺEΛByή ΉΗ y'ˆb½σ}σΪV‡‡Ϊ6±ΦΥdέ„Σ’ŽΏT ΨO ύxŽύ'Ωp†DstG*.9ι<ΰŽΊQ'€4%ΩΉωtΫ“oΠ’Ujt»c»$zβφ«hΜ°~5Ža€46r­;cP/ϊξΧτύ’UΚΡ΅ΜJ€˜}‡ΣIά]Ÿΰ+ϊϊ΅GIΔφ€‡oΎ\ ĉʐ,YI“Ηž&,φ9γΥήI-σɞ!ό ΔΥιhΊV°l> ΔνIΛΜ¦»ηYΪ²ϋ@Ε΅mG½Ί€ΠΉg ’δ€xsΝύužƒ ήA 3;WΣЈ0ϋθHΰg'w\³\q‘νφ$W°Ά§=Θ ‘ΐEγFΠ#―ΜΦ Ω}ΰH’Xq&ΈρΡWΘ<ΈΙi½:Σ³|Λ“8γώ³y'}πΥ)§ΎŠΚ*Ϊs0τμδ‰MF›@@μ `ξ+Ž―U<£ΛΟΟΧζRdΌ('ΏP“Ώ‘N±βXΎaΫM™άrE‚XΓ Ν£#iζ΄)4υα— »H~/Hπ·Φ -Υ>WΤa¬+  ^Cΐ|μ_~Ηٚά16ek۝ΟΣζd:Ί‹(@š,Ί^{ν΅΄}ϋvεbšžžN²ˆ{lff&eeeΡ‘C‡H²qqqΤΉsg%κηUΘFGG+‘§όtwͺ΄½’’B‰_Ε!vΒ„ Jτΰςζ•––_-§_ύUΉΕJΊwοN“'O¦‘#G*v.o*΄J@DΛ"΄J½ŽS«Y$ͺg1΄Οξ¬Vβε±^Οsε«Λ”Ψ’X|“Π$ Ήη„‡‡S@@ ‹bΛ”ƒ«8Ε²q¬ΣT±U,·-㏉žU±ς(Vσ–xΥD±^υvycλ₯>=/)΅]€Fζ‘β€Ξή]ΪSλS½ρœΘzL23žŒhΒγ΄$ΎόΝfφ¬ϋE)΅M’[¨ζ»₯^T  M‰ΐŠυ[ιΖG^¦L3gq=Δb­+ΩύPΉ@@ ©‡¨KΞ©q†]ΙΧΙλxαΧΝ»πΐ^M“8_]vχLZόαLJiP'*˜|ριΚ‰£•Πtώ―+ιΠ±LγyΣg½GΓϊv£π0ϋ£¬ ©cΕ’+x§¨‰”¦y’#ΓL7]²nξΊ"μ™d²Ο͏½ZC+G κ­Ζ<:³sXrb΅iΥRγΌuχAG6eΈ™€Έ?›¦Όν„wΣc΅­Kπg&w]³ΌυβΜχeƒ@S ΰO)DΎG «΅Ύ>ϋή—5‚p%΄Œ‘ŸNςέΫSΣιύ{hD±Ξƒμ"Q¬§Ύch€ΨN .6ͺFfτ΄είϊΆ&ΙožbY”ڐdΙ}nHŸv9ΘBώέŒ’XWΤaWƒ‘@@ΐ#˜@3όP[#έ56U[{œ½ίέs²ς ΄ΟC$ €€€cΔΔΔ(f=HD₯‡¦έ»w+!μ‘#G”ƒ¬ˆ;‹‹‹•ΰtώύτχί“8―¦¦¦ͺE²‰‰‰JL(N¬ξšk•——G%%% Œ΄CœXEλA¬4bσζ-τΣO'beErr2Mœ8‘¬άH…«΅$ iΊ‹§΅Ά5Φc2ŸΕ τ.η bΊͺ"ς£ΨΖΪέSύ³˜]buΊJ ΰώƒΕ1 <•€|V£’’Ψ±5J‹J©€RGΥ| `{Ν™mθZΝρ“JΨ)–Xx’>'(EΈ€η>‘u TιYό¬ͺͺΩ―ΊŽ”“_TGŽS‡£j™ό*“BŸ½wꩌXpφmZiκͺζo½;§ΨAXs²›7ŠJJ9r‰νΡ'έά\T ^I@’Β_zΧ3dξ‚xΥΔ³θΡ[― hΖ"€@S&ΒQΆΟΪO-ΒA˜ώ½‰~]ΉžώΎšD4kH4κΎηή§o^Μ°Λβk€Ώ½?c?γ4u\"yΟΊoͺΊNΘ8žCΏφ)½τΠM†]΅δͺJΜ'|J°,W'σί1H“  Ž@„Y`‚΄Μvχ1ύΈΦmΦξκ8Α]Χ,o½‡Τ‡Al `ξB“žeύڸ菿ι₯ΎΡ”,βΪOž»ŸZZ$i2Ίyc(©1OΗsςΜwa@@ΐ ΄ˆ©) ωgσ.»D±ζθ­Z₯rνΑ#ΑL“Œ!™ƒ˜·΄ί<šΗυΓ(ΧdήΗξƒGιΜ!}TvWΤa©]Ψ  ΰΩΜΗΎσ l›?θ±)wΡ4Ώ/»zN–Ή±@σθw‘@½  ΠθˆπRp²ˆΈ΅K—.J΄™‘‘A{φμQξ±;vμ Pnn.8qB-€ 6PΛ–-)%%Ε(•σΕAV„œ4Ζ’7¦ͺͺŠηέžΤ ˆ`ͺkΧn-‰ΣξοΏ/‘M›6)ηZ·†„„PAA-_Ύάb@rΣ?.Ρ„½βΠ+ύAr>ƒ(ΦΟϟΚ**IW™ΟŽ±:jfE£βόV9ΏεŠΛΰfz†SDDώ朏54€€άc$¨C@`s¬ξb«ςGΥ©©’o/Rι}Τυ<2²ζ8«S€ΒF’X‡‘DAŽ&Zk‘uΉ§Θ—ΰϋΧzΎωΆ­βhέ–έζ»iγφ½5φa€€kΘΓV™΄YPt2“ΤΊχπ1―ΕξΨkϋυΘ5tQ €4.Ε₯e4ε4‚ΨπΠzγρΫhόΘ“B­ΖΥcτ@N@&†œ?zˆZώwΟšςΰ‹J$k(yΓΆ=κΑ΅[ƒzw1 b 獗&œ9˜~`‘­!}Ίΰ7š<φt²4ߐ§!―ζW[Κ2„΅KͺΫΧ–rνΙc>ιΔή~δζΗίώb¬²UΛXΊτά3ŒΫ Xc\—•sF€wžΊΣ¦‡…Ε§~‡i Α€€WˆγΙδ¦ιhΖqΣM›Φ—―ΫbSΎϊfrΧ5Λήk―τΟξ!υεŒσ@N8x4γΤ―΅k]ϋχΑέμ"{λ“―iςΛΖKήD}»¦ΦΨοi;’8LXH•”›fTΜx+  ^E €'αφα{‘ιΌ†΅›wEγ†Ϋ܏Υf΅Ξ>½ΏΝηΦ–±ŠέWL“Dχυ΅ΒoTΈVkΤΜu˜φλ  ήA@~˜&[bέ56eΪVW»{NVΎ™XN±|χQ€@S" ΟϊCCCΥ"BΜN:‘Έ―ξάΉ“vνΪ₯„±"Ž=vμ˜rŽ•c"”=tθύσΟ?ΤΆm[κΦ­›rgmםr'?qju…¨388X9«Š T’«κ΅τ7’––FλΧ―W"Xι{ee₯β7wξά:±†ςDD›˜˜D-Z΄d~†½xu&ΪΕΖΖςߎ?ιy.Ί’€Ψx² €Jͺͺ,`Q‘^ ΙEμgmξO‚.z8ΉΚgU«Έ­%όŸvtΡ±Π³l«nK*ψώ’‚I„Rdd”c+Ai.#Q¬ΛP£"{ XsσΪΗΒ8kiΕϊm!΅ΌrlHίτν/Λkd;’~œ$:xO,΅–dςT±ΙdCή€?JNŒ7lβ@ΐN2Iάτ!ˆKOοίΓζRδΗπΛkt½’/υӝLζ.Άθ什eJ”Δ¬y6Ήό΅v³-U €€@= ˆ›αžCiΖ³Ε©eή+Ρi½Ίχa@šUΆΡ5»z°e’~DX(}ώΒΤaτΥTZ^‘Ξ—ΰ4ϋ€Ϋνβ!'?3mŠjGQq©±-w?σ6-›σb½Ύ‹ ©eE‚\r]α‘Α΅δΠξ>Ζ‰[vΠμ”qW§Tώ½cš„Ή΄«G§v¦»k]_Έt Ν|gρψ —Œ7Šbε·ΠNvQ7M—Ÿ7Κ懓‡e™žŠu/' “δM“8*‰K†ω$AΣ<¦λβ%‚0g%w^³Όυβ¬χε‚@S!°kΚδ±]ΣΤ©]’ι¦q½ ¨˜œώ¬FP*oΌt<]2~€1Ÿ§―„…kϊ`ψήοινFϋ@@κ&pρΈšη©«ΜDu•πΧ?Ϊη˜cOPΧ)u7wš-)+'ω.ΙφΉΐαΩψD·ΙΖΊ]Q‡±2¬€€x NνZkΪ*Α%λJξ›ͺ«mΞ<ξΞ9Yω& ΗX8Ε:σ­FΩ   ˆΰHD™²4oޜϊυλGǏWξ±Yq‹ΝΚΚ’R%ώάΆmνέ»—D+ξ±νΫ·WΞ³²."[ά:°ΈΔ|rΏΞΜΜTbTgΦY[g„•a₯M"΄4€ββbΓͺΥW‹―€ΥΥ"χ:)ς΅z:„€Ahΐ§ΊjvŠ-`§Ψ*jζ#σΑοϋ γΏ3]ykύNŠb£££ ŠuΘ_ qω¬FGGσ}%D9Δ–Vκ¨ΌŠ?£ΞQ±σε˜Jωr\Ζ‹_ΣO:¬Χnθθ¬~£\ǐ+:x$Πΰ Ze•|Τ¦EώCχ] ‰³y’Ι­·>Q3b·y>ΣνaύΊ›njΦ―δešΦͺ-šmΘ„°QWMΧ8“ςM3Œή›q·a― v辍ζ!ξGίόLS/gσDn»›NοΙ“ΛΎωr;[q2»₯¨„2aύΜ!}¬–'?dόγo«yp@@ ώδ»ί—‹ώ4 ή{ϊnbD° Π”δ²ΐι•Ωί»\YYe“(VN  .©mIb IΞ―OJhCέx=τGΖΣE`ϋάϋ_Ργ·_iά稕ͺκjϊmεz’ίΰΆ€7η|―kˆgEsΧV[Κihž„–1„+†ƒqε‹ϊθ۟ιε‡n6n[[ωώχUšΓ£M~›H™¦N`’±WηφšόΦ6,Yiν0Žxξ¬ƒ€x' Ζ ₯G^ω˜'ΦκTvrπ‡•8mhίnuvhΓφ½΄qΗ>c>qwDΰ° μ+Χo΅K»ƒƒ}Ι€aC§ΩN)I†MrEΖΚ°  ΰ5Μοcςά€δΞ±©ΊΪζΜγξœ“%ϋLS‹θHΣM¬ƒ€8™€8Κά_ϋχοOβ„zψπaΪΏΏZΔA6##C9ȊHVD‘›6mRωΕ5V²²€¦¦*w?Ȋ¨Ι‘I\E€»nέ:%ΤέΈq# 4ˆβγmΈδ¨φˆxςδΙTVVVoq‘ΏΏ?΅nέΪαœΥΗΖZŽνδo=Γ'E±%€―ζΰξΎΚ²±φ™ͺΛIWΕ`ώEEE²Ψ0¦ΡvkdΞ―mˆ`Wc_?Δ²[,{:胝ӿjώl²K¬ΌŠ;­`ϋ"’wpμ·οd€V{0ξ“-ΆξhΖqΊμξ™t<'ίxΌΌ’’VsΤΣKξšAΩΉΖύΆ¬ΘδΧΣϋ[ΖωwM›ωŽq˜iyβœ2υα—,“|γΩ• @ ώξΈκ }/»Dπϋj› ¬ΰ¨L¦‚X9ιΒ±Γm:ΧR&qŒ ΡΪΌkΏfΫΖΧ?E[v°tϋ@@@ΰ»ίVhJ‘ο_γΟΐw0 l€4}Ί€jϊϊΗί›T΄QΝΞZ6ͺYXΊϋΰQγѐ @ꐜhάΆwεΊ‹Ζ²3EsΪ›s~ M;λώ­9ΙƍgߝGς t"―€>›&Ϋ„3‡hΆ]΅!ƒΊ²xΨ4}³ψ/ΚΙ«;rύμΌΜΔΙ%˜ΕnCL&»ΖD…s„Zν°_ZFΆiU΅/ϋg‹Œ7Τz@Ό€ŸŸ/Mš*y²ΤN~nτέ―+謫ο§ωΏ­Τd πχ£GnΉB³O6~Zφ=ΑWšύ8ζΣηξ'™0νm)4Xλ›W`›(ΙΫϊ‰φ‚€@S%pρ9#4]_Α¬O½ρΉfŸιFqi!†VqΠqCςχσ£ϋnΈΨ°Yγ57ΏzOΈI³˜ W 'IπˆKΞiΨT―πόΆ'_―S+c ³ήϋR L xμΆΣ”ηŠ:4b@@ΐ+ˆΘdpοΖΆ•XΧ#έ=6ell-+Ξƒwל,ωn:οK~—«ΕΜ€$Ψ   ΰ@………JΠ*bS»ώπΓ΄jΥ*΅O„°ςMlϋρoFqά7=yξ/.²ώϋ/-Z΄ˆ>ϋμ3ϊκ«―hϞ=*―£š'χυ^½zΡ¨Q£XΨCGŽQν[Ώ~½r­¨8υάίQuΦVŽτ_άpΓΒΒκ½Θω ™ƒQ[Ϋ°ί:qLLLT.ϊͺ ͺ`γ5½ŽOjfύD―<Κ}U‘’€?§•’D†pΐτΚ7³Ι5Zq-[²(6D΄ZP#S$μ⫧\._―oFaμ$-χ7$ο%ΰη½MGΛ›qωzφ½yV»z0-“d1O‘α‘u>P1=gPο.tγ₯γιέy‹LwΧε!Νϋ_ώdάΆΆ"ƒeο>}—&R©΅ό8 P;iS.€―/£τ¬““4e‚υΉ7ωμΣι―΅[ŒηΙΙ·=Iί~% θΡ‰RZ'DUή{(MΉO}φύv«ͺRωγšSZ¦mΞLΖ °β± ‹JIήC* ΄}€₯ΌΌRs”!ƒώΆ€j{cZ―œW[Θ!Oc% nζnάΎπ! °ΦγۜHβƒ ήHΰώλ/α1o›~ο³οςwδ]τμ=ΧQ8Š1OβΠρΡ7?ΣΜwζj9"Θ@Ÿ©4εΒ±ταΧ‹eoέ}ήψ|έuΝdγ>G­Μωαwυ[ θΩΙX¬8‚­\Ώ¦1—ΜyΖύ²rύΕηP|sχ nΚƒΌΗn»’.Ÿ6ΣΨ.atζyfΪκΧ½EE„©ce,˜ύaιjzθŏ(―ΰΤwΡ1‘τ,>2O2‰dηώ#Ζέ_.ϊ“’x¬δώ.αHω‘Ζύς SϊΜώφWZ°δ€HDΪe=HΖqε(iϊϋΛP@RΌ6b’L2zΰωhςΨΣ)EΪς€P^‘@άK »dάΙAL+mδ c£―šN·_uυλցΊvhKω,’’1•/ώN ~[EU<Φ"I„`„ΜYΙUΧ¬ΪΪοχΪϊ‚ύ Πˆ‹υ¨+ο­³«₯etΰh†ρZfι„YΣ―'q5Oχόοσ]<†W­Ζƒk¨ηωn΅~Α[υ<ΫΎΣΒC΅ΏŽe°―δπhω™ι§MΔ°†τϊg xE=|σ$’IΩΉω΄š…°orπuόLΥ4έrΕꘜdΊK³^Ν“žςψ€i2Ÿ0έ/λ2ώπν/ΛIΖρ iρ_kiκkhΔΐž$.·†$ΟXΧό»ž~sŽF(#Η/;χ :cΦ Wφ»’© @@ΐ» ιΫ•~Y±ΞΨθΥξ ³–ά=6e­mΞƒwǜ¬-ό Džw 1›r2Γ+€€€s”””P~~>εζζ*ηΧΝ›7+μ±cΗ(++‹Dl*ΟŠε™±Έ΅ŠVœc»vνͺάcχξέKϋχο§'NPAAmέΊ•δ\)WœTSSS5Α³λΫ yŽ-.Ÿβ`+m\Ύ|9ύσΟ?ͺ}²OΪ#ξ~" ’ΆΪ“šρ䲨¨(8hΪΝKσŠ 4))‰E±(Ύ˜tεώ"νΘ/Χ³³2Q‹bcccXŠr5\α\MυΩE Kϋ6tύ%ηΨ,F5ή.)ž™4†νSΓ.›^gά}­z(σ™γ˜M'—I"¨ΎυΔΤ±]νŒμ)yA ©Οη￁šώœqRΆ ŽήΙΒWYδσή‰?oβl"Ύε©yšzρ8‘}C“ˆp_ώψ[?‹'ΥίϊΔλͺθΪ&ˆήuΝ$7'qίBpG2jf.’­‘Α†Ω  @Ό•ΐ&}ωӟ<‰p‡± "†όaΙ*jΟ SΫ$RrR°φpϊq5YtςœtΕ„3ιΌQƒη7dε‘›/§YΘi*F}ώƒ―ωϋω N¦υ­G~很XλΟΏ7©%Žβ΄fqƒ$ΩƎΉ¦OC=£‡τ剑—6έφ:fX?Ί”'wΞ3ω½ ‚΄Kξš‘Ϊ”Ϊ¦Η>Τ³ #³F?δαή{3ξΆ(:}θ¦Λhρ²΅šΰ<μ‹…POv– Aό;eΧ%Ώk IΚ|λ‰ΫYΜό³TΛ~ωΥcό Τ½c² N4υ’q†μjkλ„t„ίCϊ…Φ²Hš8z(}0sšα^AάH@y°ΛtμSϏΌό±ΥVυeΑμ@6πΞά­ζkΘAW]³ΜΫθνχσώ`š q·ΫΌλ@ƒΊλλλC/=x“ϊξk© Kθ$°©°Ηyφμ+―¬9–mΟωφδ•@²¦iΛύTPT¬ –bzλ  ήE@~ΛΏύԝ4βςi”ΓŽ†΄pι’Eξ1‘α*X„α˜ιkο.νιžλ.4έΥΰu©σσ ‹ξ|š'–Λ“ί 7>ϊŠΪŽΰnς¬7—ƒΙΈ‚Lx6O2Ζ#σ8,%WΤa©^μπlCϊvΣ4PB˜Žikώ·α±)Km1ί'A$œ5οŽ9Yλ6οtqτ>šml€€8‡@ii)egg«εΰΑƒ΄oί>:pΰ₯₯₯ΡΡ£G©˜Ε‚"•E„°"§φνΫS‡(%%EΉmϊϋϋSf&›Yq»vν’5kΦ¨rD ϋΗ(Υ[o½U‰g₯¬†$ωΈaΓΪ½{·j“΄+''G cE”› D­"œ•vΩ*Œ•r%ο˜1chψπα i"Ξυς^ΗΕΕ©Ώι\ώ;ΥU²‹jU)»φxAλνlb3Šne>λ~u‘νΆ~6μ¬ ΩAΐ‘D+βT©f¦Vθ©”£ΚxaΓξ'¦ ­ζωW…•:*©Τ“―€r"—Ο ’χ€(Φ{ί»&ΣςΗnύ?Ϊ8~_½Ρ¦>χθ؎>yξ>MT›NδLςό§ξ’sFœF½ϊIH§u•#]>ϊί½*ͺi]yq@ΐvcO@_Ότ ]χΰ‹$žL“Dϊ—₯Άt+G5~δΦ+j;lΧ~‰άΡ³χ„›Σ<Ό5bΙ1εBvG’I§·>yR8kΘ‹W†°vύoxι(@Ό“€ό}εα[θŠ{f²Ψ‘G K₯XFHe±–NλΥ™fή3ΕZ»Ž‰;­8žN}ψ%γy"°Όϋ™·iα»O;$B¬Ι™qΧ5tετYΖ 92ΑTˆk¬όΏq#‘οφ~~Ύζ‡ά²ύΪ#·?·ε³KjΤ/YK),4˜žΎσ:½K‡•ΐαM·^|ΗΣT!aK…Ε%ΚΦ°mϊ*BΩY2nψxH\† IΞ“ΙCCΝ&Ιρ›/;zι#CVΌ‚x(yΰ)cŸηr`‚ι³ή£γ6ƒ9]§ί›1M 3νVpc‹ƒ΅«Y¦ύh χΣώ`@ΐ6cOοOίq•Γ‚΄ΨV«{sΕ5Φ4@~ΌϊΙzΤAcηšΒ±  ΰ -bθΫ7§«ξ›₯ \%Ιηΰh²XJςΣη ώλθ4€ƒλ,xλIšώμ»΄qΗΎΕ•Π¦ϋkμ7μ5Έ7½ψΐJΤkΨgώκŠ:ΜλΔ6€€€g 2~nΚ°†Η΅λJξ›ͺ«]†γΞƒwυœ¬jˆbϋΊ‰W ΡεεεΚ VD¬"|q©ˆIe]aΕεU’"Da©8ŠΆuλΦΤ±cGΓvδυ$εάƎ“†$bTqi8p ΅mΫ–ζΜ™£D²β<ϋΧ_ΡΔ‰•ˆΆ‘’ΨκκjZ΅jύόσΟJτ*Β^)³ŠƒfddPzϊΙωςόKϊ`k2ˆbΫ΅kQ¬­ΠΌ<ŸήΔ-φΰΑ,-Q’Qί€pŽxyΟ΄ΝΧ³ν₯ύV©Ή0βΊγy9ښ°Ž' ΧwΉΧˆψ!ώ¬–ςά²ά2wϊ:PΛ.±ΥΝ(»TOUόy Rυ9ΎC(ΡeμσŠwY³Pœ"Μ_ζ½ς0=uηΥ”ίβΤ³΅πΠ½tρ‡3© »Α4$M8s0­ώκUš>υb ¬{‚W`€?έ|ωy΄ξ»7!ˆmxœ Vˆ‹ΣςΉ/+—!+ٌ‡ϊpDγ―^}„žδk‡88;*υμ”BΛΏx‰FžΦΛj‘β4ςΕK©Ι¦ςEΝ’3•Υp@@ NŽž{Υ™@@  HeGXωξ,‘ΝmωM+hγšΣ»OίE?Ύ7ƒ$2Έ#ΣΔ³†ƒzkŠ'ۏΎύE³―!£χ‘Eο?C(ΛZ’ρ f%“D1ΩΣZέΦŽΙƒΊ—Ί™ζΎό»ωΆ²–UύΎΉρρ΄~ώ[tεΔΡVσλם–ροKBVΣe‚ΠuμώΊϊ«Χ” VŽ];ωl›Η8¦^(*"LΝ―˜χΚ#ΞγΞJβBϋΫ'ΟΡΗœ¬CΫD›ͺ‰Ž £·žΌƒŸυ>ΚΞxuΟύpE65™@@<‚€ˆbΞ1ΠΨ  Ή―–ΰ“ΖLΌβ±)Σ6ΤΆξμ1xWΝΙqΣΚ [έLi@2Ώ @@ΐqδZ+nͺΫΆm£?ό“ζϟO~ψ!}όρΗ΄pαBZ½z΅r‰-,,TBΉpvδKMM₯‘C‡ωηŸOΧ^{-M™2…&L˜@ƒRβVSA¬‘₯ʁςμ³Ο¦=z(Q­Τ-Ž±"ΌuΔ|Y―ŠΠVDΌ²”•4Τ‘g2'W^ Ο§₯n{G΄Ρΐ―žM@\†εο\žθ«ΚIWžΓ‚pg7ΪξΦ5γ>U"ψ?vW¦6mΪP` γƒ Ωέ4œ6{KBB+v¦*=ε”²ϋ±#?ͺl:[Αεe•TQ5ΏΚ΅Aœ€ χ›‰lF ζlk šγ|“Ζ #Y•d‚Aφ?ί:ͺ8UŽ|y½…έExΊ~λΪsπ(;ΎδςENOνy²¨Lψ•WΣΙΎ—Ÿ7Šd©o1ξύ7\ΒužKΫχ¦ϋΣN^v8ͺ"γΔF³₯|ΛX> ' γ詞4™ΆΎ}Ζy ΰ ϋ—~ζ°beϋ3§±σλε΄xΩZ’‰τY“m9Lξ΄gΠΤή6ΚΓΧo^ŒŽež _yΠ‘τγμ’GςπX&°wHN€ωΨ4ϊ”8±Θ‚Τ8Μaηβϊ¦x±,υIr―sτ=Ά>νΐ9 ΰ)ξ»ώ’ @@ &š6εBΊhάpϊσοΝt0-“©%C9¨ΆŠ‹₯v‰ρΤ61ŽRΪ$ΠyόZ~[KɜΏΎίEΎ~νQkE+!n}Λ6, ψόεfϊΣ²ΤχτΜμ\%ψν”’DSZΣCH`+OMg νG²lή΅Ÿ–ήHiό›γD^Ε7QοS{ž ­C²wΨΪ™pϊύ;O)ΑΓfv_ΩΉΝΘ¦±‘κ7”ίΔΦόύγ‡$€lυΖμΪ²O=DlΣ*Ξ’3­ ί}νd֎₯½‡Ž‘ΈΉKp’ψΡͺ½ΆΆω@\G :2\9yKUUΥ΄›Η[sς ©E4Gg‡©ΘπPMcJΛ΄’Ψζ<6j)Ω;Ζb^†+YζuΚΆ#ο!\’!χ΄χfάΝξΌw[j&φ@“%πϊc·‘,LœγΚκ4u[ω₯fΫς³!Χ&G΄e€€Έ†@LT8}8σJ?žC_.ϊƒΆν9€Ζς ‹ώ{–G½Ί€π˜Ρ»ζ84ηί Ή—œ7j°(-]σ―›Ψ$φρΒa~Ξ*γς[@žοvLNRΟYε7‹½ΙuΨΫ&䁦J@γ Ήo4”Ϋ΅“Ξ¦―~Zf,ζΫ_–Ϋτ\Ω]cSΖ†Φ²bΟ|}Η¦œ5'Λ΄Kό½‰ηWεwέ~εDγ:V@@C ---ZD{φμ‘#GŽ(‘j^^UTT¨ d>Ύ8β‰ŸΈΌ&''SηΝI\Se_ll¬ΈΪښθθhjίΎ=­Y³†Dh+BΣόό|5ΟήΦ2jΛ'm•6φνΫ—]/'y18Ŋ ©i RK!‘aTΘΙΊς ‡Zl9OΖPϊvλ [ς#€€gπσσ₯©m­6f7 4M±Q–E±¦y²ξŠk–₯φ5†{ˆ₯~a€€€€@Σ#ΐΑnξΊf²Gu\&˜Υη»Ύ=pEφ΄yA@άC`²οΣ5•6nί«πι‚%4νΪ YPγkSƒκsΏ²w<έ¦†˜erΕΌ3ηd}:7c’β[Π₯ηŽ4nc@@ΐ18@³gΟ¦ΚΚJ*/g‹TE'σο£’’”0°S§NΤ‘C%˜KHHP’ qΛ“<υIβDiI]β>λˆ$BΨ‘#G:MΨ'B`€¦AΐΧΧG ί’’’hηΞνT]ΑΖlεΉδγߘD±zξSUWζ“/cc›SλΦ­λύΉnθ₯§{ˆˆTE¬zŒƒ<δ—λ¨˜cwϋΥο₯νa5;D–驨ROώώ~$bbb΄™°εu ŠυΊ· Ϋ TVUi’rΛ$=['7•”Vv—2$ΔΆˆ‰4lβ@@@@@@@@<Žΐτ©ΣεΣfͺve°ƒϊΟΛΧΉg ςΈv6•>–E‹ZkμξW_@2F‰  ΰX%%%”‘‘A‘μ²'Ž{β +Ž¨"ŽGΧ]»ͺE„@"„u„‹€7ςGo" χ€ˆˆuίΪ½g7Wθ(‡…±QAά¨ͺX‰ϋPY݌έguT©ΣSPh0Ι=K; y7όͺτξχ­p§ήψΜA%Υ^ΜΘΣzΫμΌZ{)8 φΈπΆ§hε†mΖ“žΎλΊωςσŒΫΦVώZ»EE7δ9gδ@‡L’0”‡Wp41ΓϊΡ°~έiΕϊ­ͺ菾ω’XGCΆ£ΌχΎ\d Ϊ—ל˜0ʎ³‘@@ΐVβ*N­­Z΅’6mΪ(aœ8ÊΈTφ………Qp0[ξ90‰λμ\ DΈβΈ'nœΆf­«"\…x΅.J8n ˆΛgaι₯”Ÿ—OΥe9€―.£f~C8ͺΧUPUi&±],E„·P`|vlωΛ@O" χω¬Κ}$0 JʊιxI%Gψ’‹Zυ i,Ÿ\©ΧΣ‰’jͺΦ…σύPFψϋϋ7€Tœλ Šυ€7Mp?Χ>]ΰτF„πξπ=œ^*Sg κ­ΕΎυΕ4υβquΊ1”–•Σγ―~bZ;Ž ΨπHΟέw= Ώ|UUW“~ΫΈ}/υιšκ‘mm̍ΚΝ/€Ο,1vqΖέΧP&Ÿy`@I@[Ο=χ\κή½»Ċΰ'**J‰ŒκώθY,T1k=”OώD”+‚&W€jΎΏ———SEE… ξͺΣ±Κ‰“‘_qt„+ϊƒ:œK@’±ψΦ‚-μͺšΓϊΡςρ γŠOών8·Ξ,]ΟύΙ%]e>½λ(.>žΪ΅k‡Ώ}g"GΩN# TπU\Θ"V=U°³«ŸOΓ¬bΉvž%Κ«Π“Ž’εή(υ@<ξ΄·eCλ2Τ¨@@@@@@@@ΐ“ ŒΨΣιΝk›ητ:P€˜8spšρΦγξτ¬zζ­/θ‘›/«uZzΦ ΊφΑθΐΡ γyRZΣπφ`@@@@@@@@ΐc tl—D·]y>½2ϋ;ΥΖη?ψŠΎxι!mocmΨϋ_ύDΕ₯eͺ{ΌοΌQƒkWΡ/p;δδdš:u*ΕΕΕ…°UUUTZZJ•••jŸ%‘¨bσσσ©€€D ICCCΥ«-"Y09Ϊ}ΦHιKQQεζζ±cΗθΔ‰TVV¦Δ±·€€εˆ­O‘‘‘j[²HM›€όMtνΪ•vμΨAU…TU–MΎ!‰υƒ{ IϊιΨ?S\buΕLνΫ·W‚?i#v‘jσζΝ)–Η³22¨°BG…•z b«Ψ†Θb«δ>W^M%\V€ŸΏΊ?Θ5Α–ϋœΝGV7€(Φ ΠQ%€€€€€€€€€ηψφΗ=―Qh€8€@χŽΙ4nψZόΧZcio|ώ=ύψΗΊμά3(9)ž’β›SqIedηΥΟΛΧRY9‡Kύ/ψΣϋ3ξΣ]֐―         ΰnχ^w}Ώd• όφλŠυp‹uρRX\Jο}ω“ͺ5ΐߏfMŸκβ :hZD*‹ˆ|Δ9U„£βε %$ΔSϞ=)$$Δ"”-[ΆΠςεΛ)†…H:u’:P<;NŠϋ«§€ΜΜ,ΪΎ}mέΊ•]ο@IDAT:€±J+Ξ±’Δ!VάjΓΒΒTϋΕ-WϊέΉsg‚ΚSήIχ΄CΣ" £œœ<ͺ*Ο¦€ͺbjζΑ ςR·XV κ«Κ¨Ίμ8ιυεόωŽ€ΤΤTϊάΊηέF­ήJ@ξ_αμ›”ΤšvοήΕ"Vε—ι¨y`3v?nΖpϋ“ˆi+«υt’TOεΥΝ(84ΪΆmλ2gsϋ[Œ3μ!ΰ9ίRμi5ς‚€€€€€€€€€€ΨD@½Μ4ΊδΞ΄rΓ6γ9Σ2ιοΞ3nΧΆ@―=z+uMm[[μ# γZ/y•E£’’hνΪ΅ͺγƍS.Ί–άr eΰ΅ρΏ‰V­Z)±]nn Isx9A>‘¬,υ~σί»τ‘Ί<|Ω)Y„μςω•Ύ"€7Ώ]Ήη΄nΔA¨΄ΌŒrJu”ιKΎυμL¨‚γ&δ•VSί3$8Drr2Δγυδιi§AλiοΪ          & ?ρAzβ΅Oθ˟–i\`­U%BXqˆν”ΪZ6$0b`OΊτά3hޏάb]χ‰Kμ[_,TΆiΥ’ξΊz’λ*GM  Π„ ˆ φίURΕ>|˜ͺͺͺ(66–JKK-’©¨¨`ηΜ‘iqq1eee)αiFF;V Jƒƒƒέ"΄“Ά‹3μ?ώHώω'I› BX·ŠxJάlE+‚XΠ–——«>K>ιKvvΆβPRRΒύGνΪ%«όa`g£& "iqRέ³g•–RUi&ω…΅ζΏm―μ·NWEΥΕi€cΗۈ°Υ·ˆqΎEο%¨μβ|ž•YFωε:*ͺ Šg|ί€Œ‚ =ί/Tp„ζΝ›Cλ½"š–C«Α O&Π!9‘nΊμ\c»wH6c@@ΐ:πΠ`zρΑ›θα›― ΩίύBΏ―ήHGSFvqAHP Ε·ˆ‘3NλEŒF§υκμ–IΦ{RΏ£Έ‡ΤΞo'0γkh$‹c%ω±hΙω ŠŠιΩ{―Suγg9Α<ξˆ  ΰ|iiiτΥW_)‡XΉŠλžΈβYsGρh‹-”+¬œ“——§‘ζζζ’” DΞο€I "rQοΌyσhΩ²eTXX¨ϊΪ+’¦˜˜ 3Š›D›ŸŸ―ڝ™™©ΔΎ"ϊ=vμ-X°@•>i$%Έ2© «M„€ό ‹(ΆeΛ–tθπ!ΕfPuE!ωDρί–·ΩΕ²SreU•g―πg!‘Ϊ·o―>―MδνD7)Ή_EGG³[lkΨ —­Ήe:Š ς₯ϊx WςGϋxI5•Vι(8$H}N侈Τ8@Ϋ8ήGτ@@@@@@@šή]Ϊ“,H  υ'NΣ¦\¨)₯ͺͺšŽηδQ8G Α~&ξ!ώ‘y        ΰ$QatαΨαN*ΕZ"ΧΜ-Α>p"qy]΅j­^½ZΉΎŠ°¨U«VJμΪ·o_%΅T½ˆ^O?ύtΝ½{χ–-[”“fQQmΪ΄‰.\HIIIμ°ΪΞ₯TE»rεJZΎ|ΉκΈΑΖΗΗSχξέIϊ“’’’ΪμοΐΞ―>F§Ψ²²2:qβmήΌYυeǎJδ+.³Ώώ»κΛψργΙίίίμkδδoYώvŽ9BΥε9€+ΟfQ¬Έ«ΦGnη>X:]5ιΚ²yΙeΑΊΏκSBB‚K?£ξλ=jnΜ$˜ƒ;Hζ{Ξƍ¨°Œ(‡…±muΎΔ—z»Se5Ρ‰R=•T7£‘‘Τ‘C—y°»Ρ8ΑfΕڌ A@@@@@@@@@@ ρπσσ₯„–±―cθ€€€€€€€€€€€4€Ε‹+§T‰kδω矯€βŒi‘„ŸŸŸΤΕΕΕ©ΌύϊυSεˆ;«mώϋoκΩ³'΅iΣΖθΘj± ξχZ³ώυΧ_ΚνU±βxήyηΡΰΑƒ•°522ŠΫcΩ^cΕTϊ²tιRϊν·ί”{μ‘C‡”‹n―^½”ΘׁMFQ^B@†;vμHώ»‘rsr©ͺ$ό‚ˆόBXPκ`eW•QUY&ιueMέΊuSΞΙήΡ΄j'`p8oΓΧόππ*fχοΌj*©ς£π@ώΪaκ¬ηΜzΚgQ­ž|(66–]v/«½—8β(Ε:Š$Κ¨ͺͺR.―βμ*‚-ZΠθΡ£•(6&&¦Ξ–ˆθ4""B-β.B‡¦νΫ·ΣργΗ•0vΒ„ ΅ kλ¬ΐΞ ŸυλΧ«ϊεΤPvχ7nMš4IυMDSΦ’ΈίŠˆVœ3γγ¨’’‚~ϊι'±¬8ΘnΫΆ ’Xkρ1q§Ψ6mΪ*Gαͺβ4E΄'_Ώ`ξ΅υΏ+ΟΑ’c‡ΨΕfS3VΘ&%΅¦ΆmΫRp°τ ΌŸ€|Nε>&Χπ]ω”Η’Φ‚2EϊΪ£‰eqv™-©¦"Ζ©ϋBm"ΌŸZΣμA=Μƒ›&(τ@@@@@@@@@@@@@@@@@@@@@@@<‰@ii)9r„δU’8a:΄^Ξ‘Τ΅kW8p ‘T322(;;Ϋe].(( P^^žrτλάΉ35ŠΔρΆ.A¬i#Ε·}ϋ?~<‹cγΥ!ωξίΏ_υΛ4/Φ›qEξΠ‘²HNW™OU₯YμΈZα5τUT]‘Mϊς|ςeρ`—.](::ΪΟ†Χt m’δ:/ΣνΪ΅#6x₯"Εζςb—"–ΙUU³(–Ε΄%Uz β` "—  H‡D±η½DO@@@@@@@@@@@@@@@@@@@@@@@š’’εθκγsR"B’6mΪΤ›@XXuλΦM9OκΩj―ΈΈX‰beέI„«'NœP·"l4h%&&Υ«jWuκΤ‰DX+|Δ5VDΎΒ ©i'δΤΤTεB)’οJq‹­,`ή૧jnk΅yυ•μ¦ΩR‰ΰε3‹…€\·εs*"Φ*­"v‹ΥS…Ξφ{άJ*υJL[ΝzZqCOLL$qFGj< Šm<ο%z                       Π„θt:*//W=αͺ䂃CκM@DC±±±δΟ.”"$᠈I]• ©¬¬ŒEzUΏCB‚λ]}dd$%%%)'M[εηη]uλ](NτZς7œœ¬Δ€>>Ν¨Ίό»Ε²σͺŽm%=\«γ6κΚ²¨²<›?Ÿ~Τ½{wjΥͺ•ϊœx킆ƒ€rWgqŒ­Φ7£Όj*ΰΫdmLzΚ.“sτδχ_Y1116ž‹lήB’Xoy§ΠN0! "?qT5$qA­¨8)’5μ³ηUDΆ"J•W¦ŠHΦ•ξz•••Jˆ+ύ’Ϊ ΡŸ΄= @‰b₯Όκκj΅Θ:RΣ$E]»v£¨¨hW—RuIΊzε?Œm ©ͺ$ƒXΕΛΞ—‘Τ£GξC”·Mϊ€ "ˆΡ·ž?—yε:ΚηΕVέz•(Υ)·Ψ€ΐ@ε Gεϊ½ž|Φ©o>6΄rϋφνš\9ω…Τ‚[4ϋ° ήH -3[Σμi3ί‘G‚>ΦμΓ€€gΘ/,Φ4πϊG^’ Δ@j‚’Mlή‰ο"ΨΧHΛ:‘©θ‘—gΣ³οΞΣμΓ€€x<³ίpΩ„ίήτ’­ Ј–”jz³uχA\4D° ΰ*%eΪIŠ™™™ͺυx!œœœ­>σͺι<9qΡk€Α°ΐρœ|MΏύ…όΆR³    ΰ:ΎF΅”Μ‡H `J ((H‰‡Dΐ*Bcǎ‘Œ/ΥW,'’Τƒ*χY)S₯"&2ˆTMλvΖΊ8šŠpΕ9V„¬¦ϋμ©WœnE(lH©©ιΏ₯Ν;QλΦm(oΛfv‹Νβ%|όC=Šžtμ›MΥΗY$ތڴi£ξ¦‚xm<vϋMxx8%$$Π¦M›¨Έ’J cu:―[Ό^Q­§Ε–±ŽΆyp°Χ²8©q°K[^}'_΄¦αa\γϊ“@o@„@Φ‰<€πrΗs½ΌžΩό²ς |σΜ·­hb²sσI$h,D‚±ζΖςn’ ΰέ*d’ž}yχ›ˆΦƒ@#! “ό@ 6–ώ>Žd―-;φƒ€ΨI@2›eΆ³d§q˜'ΥΔΔD£Πsχξέ΄yσfjίΎ½ΖAΦόΌΪΆ³²²hΓ† JH*Β€ΘΘHjΩ²emΩΎ_ΔΌ"ΒAŒƒˆΉΫi§¦ΔΉυ©,77>lt‡•ώ³H ©iˆUΒΨύϋχQii1U•₯“_HKjζ㉂ιf€Χ•³Al‹bK(<,„:tθH111MϋMDο5 ^―ξAy9”Λ Χ2ώT‡RΗχŽ’J=Tp ώ'ŸυζΝ›7ΘqΌQƒφβΞ!L¨Ώyh:€€€€€€€€€€€€€€€€€€€€€€€@Σ% Β‘ΤΤTJIIQnιιι΄tιRΪ²e ‹ύJm#"T9wΙ’%Κ™OcEpΫΉsηz»ΞΪ\ΉIF/‰Wϊ%’Ψ5k֐}- 39ΝβjYY­_ΏžvνΪ₯DΆβΨͺU+ˆb-jZ;}||¨wοήJtGTMUΕiΚ-ΦJW“7cxY.U—f’―Ž†©S§ŽυŠ»Ίύ¨κC@γββΤgT„ΉμόZXIδCz«ΕΙ½,―¬š ΚυδΟΞΰβͺmυτNΕzηϋ†Vƒ€€€€€€€€€€€€€€€€€€€€€€€%''ӈ#”HNΔ¬7n€Ήsηςε+hίΎ}”““CJ*‚!Γ"ξΓΕΕΕtμΨ1%’ύα‡hαΒ…tβΔ εͺ'b’!C†ΈT|A]ΊtQb(ι‹΄Ρ’E΄gΟ*//·ιέΦιtTTT€8όπΓχ$ξ·gqΤ‡M[!@ΫΆΙόχЁ‚ƒ‚؁5Ÿ*Y«ΧUσߊg±ΡWU(AlUy>+AlΫΆm=«‘h 8˜€\§%@‚\·E ›W^ME,t­KΈ^‘kΖyυTZ₯Wχ€€$—vp0g…@¦ΑΪ3%rˆi ‹ Gg½mΊ λ  ΰ•ž{μ.Κ9qάΨφ«nΊ‡ΊφθkάΖ €€ηxrϊ TZRllθMw?JΙ©ΫX©Kӏί~n<9΅S7šzΗCΖm¬€€Έ†ΐ«3€τ΄ΓΖΚ.Ήζκ3`¨q+   ΰm~_<Ÿ~ϋρc³»υμOWήx·q+  ΰ*kWύIίΞyίX]λδφtλτ§ŒΫXWΨ΅m}όΦsΖκbbbŒλXs‘‘‘ζ»xξΖ;^c?v€€ΤM`ήΗoΠΏλV3Ž<{p‰q+    ΰ.ετΨέS4Υ‹ƒ&X" γI"Š5Έͺ² v9=z”¦]ΩE6™έφ(,,L B›5kΖΞ«:™–ύ?{ηEΡ†ρ—ήH…BB―JοUAQDEμ½WEEQρ“*Š»Ψ•ή{/CHHHο=ωζάcχξ’ά%—\ξςΜοwΉΩΩΩ™εnοfηyJKK£θθhιΖʎ¬ρρρΔ’RvΦ8p uθΠA d·:Κ©sηΞ΄{χn)Φe·[‹³pόλΣ§5oޜόύύ…8ΠU:γͺϋΐύΞΙΙ‘Ž·μϋχίӞ={ˆΕΏnnnΤΎ}ΑkKΥΜκςΆ««‹ψŸhO'Nœο!Š!gοHrpφXκΥ 4₯₯%TR˜!ϊvAτ¨HΌΓ€3΄—ζkΕ „NTΎNqΖΊˆΟϋόό)ŒΝ/q 'γ§eA{^a)₯ WΩB!Žυοv›ελ’ύ0K«c…m„ϋ ΌΑώ¨`D uŽΐ<7w͘Y‹Ο7 d@ Φprr¦\Ί*ŠmίΉ;uμ³Φχ»Άw0.ζΌ¦‹>~ψ|ΤA@j†ΐ’yΣ5'jΩ¦#>5D°5§OΡtΩ?°m"Ȁԁ΄”$Ν©ΌΌ}πy€!‚ €@Mpttœ 448Ρ#ΐ @υSΎƒ©ΔΤϊ\S¬ύγGM΅Ππζψ] !‚ €€€€΅δεζœsHPπjΣ¦ ]wέu!566–X{ψπa)xυρρ‘‚X77WrqΉ"&eΦΌΌ<)"MOO—βXΕM–΅#έ»w§#FHqlMƒnΪ΄υνΫ—φξέ+]ni͚5tξά9κΨ±£ΖF ‘T Κ*―ΕΕΕr,,κεq=z”Ξž=+˘Odd€h³HΥτxpΎΪK€EΦμΊzρb,•δ'SI^Υwj&4±΅D[R$ϊ”,œb‰ΧJ7k!Ν α΅—0z•'ΐs၃4Δ]Μ‘d!vΝ’W'γοOαƒNΩ,Š-(‘ XΛΧ?Ύ Ω³D±φ7|Œ@@@@@@@@@@@@@@@@@@@@@@@l›€‡‡ 6LˆεhγΖ tμΨ1)zMMM•ΒR ƒXDΚ|μͺΚBRa­Ηy~πώΰΰ`)HεΆX0h 1‘‡‡;υξέ›N:%Ε°III :sζŒ9±Π‰έ_œœΛΟΟ—Bବ,]]‹±£F’]»;Ρ"€B€Εvμ{θΠ!JNI₯‚Μhrt ¦zŽn΅ΐ,Ά”Š ³…KμE*-Ξ'F!Τ²eK)Tϊg°g|ύa§X·^ŒΉ D±”UTJ^.$δ―†ΒΨq=ΛΜ/Ž²₯Tί‘…„„ΎA¨=σͺkcΓΥΌ½β/€€€€€€€€€€€€€€€€€€€€€€€€έ ₯›o#œ/ΓhέΊu)5!!AΊΗ*XΓ²– JYtδββB~~~FΧ^{- 2Dn+.¬ΦΥΈqc?~Όtƒέ°a±ϋ- _/]Ί$·Ήί,pUϊXTTDόΰ±qοsww§¦M›θΡ£iθΠ‘Zγ…¬εηδ•-ZP“&M(MΘ‹„SlQ~’ψ ’;λΊK–—ΨRα^[”s‰κ9Τ—βnx+BπZŽέ*ΰΟω€€)Œεχ@z^‰p‚Jλ“ΠΌ€‚b’.±9’Ž‡§ ρuΔΣΣΣ  μƒD±φρ:b                       uœ€ΏΏΏpYν#„±αtϊτiι¬zωςebΗ؜œp”E£,†e‡YvβcQ »P²ˆ”Ϋ`1’΅‹to½υV ’έ»wΛρ$''SAAΞεVψςxœ₯ –Ε°<¦-ZRχξέ¨W―^ΔZϋΕ¬ΕηgAxλΦ­₯ˆ<'W8³fΕP}·@ͺW_ΈΕ ?Jk₯Bρ~Ν‰£β’ςρφ’β]kuη'ΐNί(Α ςr…0Ά” Όˆ\λk»ΓοΤά"’L‘Œ-Žιμ&ΞΧ2Ύ. Ω'ˆbνσuΕ¨@@@@@@@@@@@@@@@@@@@@@@@κ wŠŠŠ’κΪ΅+eggSZZštŒ-,,$v‹e‘‹b½ΌΌ€xΘΝΝM:κΥ&Jζ²ϋνπαΓ©M›6Rˎ±‰‰‰RδΛΞ±<N<O±ΐ‘έYPΛ ,¨e§@W!€1ό>hΧνΫ·Ξ!y±p‹-ΞK£zξ.VϋΏ)!ᆙ—BEΉ Β―Ά”‚‚ƒε{ΐΥΥΕΨPvK€°Έ•…±Ο§δΌ"Κ-t$7‘ˆόΟτόΚΨE&« „rK…‹l=yΰλ’ύ€(Φ~_[Œ @@@@@@@@@@@@@@@@@@@@@@@ `A)‹AωΑ©ΈΈXηΚy‰rϊυλ[MψΗύ¨(q?}}}ΙΫΫ[Š\sss)%%EŠ|YΛ±\‡ΕΌμΛuωΑY~@ [aμη‘`!:νΠ‘EGŸ§β‚t)Œut‚:G‘Φ΄[¬pPQ^<fI‡ΜΦ­Z wcό?γί΅Ξΰλ;ϋ aμΉ³g„θΥQ8Šχ€|[^ vP"ς™ωD₯βΊζ(E±μ‹dΏ Š΅ίΧ#)~e¬Ή©¨¨ˆΨ©ΟΪ‰ϋξαα!μΘ"_vΌ-ύΟ*…,žͺ }΅6+œί|,οΤ©mάΈQΈ§Q±ph-v OΦpf-₯’‚LΩ‡z₯Τ¨QSΩ7gg'σ†#@ΐΖ πη:‹[Δη>λ`3 ˆr„(ΆΈΤA8Β^\‘PΕζSnq)9»9K!-_3μ—€υΏ™Ψ/[Œ @@@@@@@@@@@@@@@@@@@@@@@¬B@*ΒΡς:ΑuψΑΗ°ΰ”XӈY[ —J&Υ–ΔXˆ_kΛ«aύΰ§ΠΠPjΩ²%νάΉ“ σSΙ©ΰ2•Ίωw֚ύί/)Ξ%6CΊGDDPσζΝν4FfΰΟ{777β`ΞΞ”/OYω%T(D±.b'ώ[P"\b Ε5Ll{ ‘{@@ ǎD±vόβbh                       u‡@ff&%''KAk^^κάTQ`,'~ζΧg!lJJ 9sšόόό)22Rˆ‘œnV‹mΣ©¨¨P ―ˆ™\\\ΘΫΫ[—ga.ϋ Τ1λ$TfGPwwχ jaw]$ΐέΊu£cǎSFfζ$£Gͺηδ%p°GeΝ€Β,*ΚΊ(ή”L:tβؚ9;Ξ΅€“““x/Rƒή”‘–BiB›_Hδζ*ή&β­ΙοΞ\‘ΟευκΥ—‚Xv—E²oΕΪχλ‹Ρ€€€€€€€€€€€€€€€€€€€€€€Ψ9“ΖΗΗΣώύϋιΠ‘CMωωωυUq- ²ŸE«ŠKlVV–lE‚ŠpΆ¬γM-g‘ξζΝ›€ΰΆ~ύϊ²έππp0`€NΛ}αώŸ}ϊPώύu§α²]»v‹zΚ:ΊUάΰ19±”››D±UdiΟ‡ϋϊϊJgVegηf ·ΨΛTί#ŒκWέ)ΉBnlž\œOEΩ±β½›GΎΎΤͺU+‚γe…δPΑΞ πg7 ]ΩUœ΅°YE₯”_tuΠβγr‹Ω-V|Ξ;:ˆχŽ/yxx\­€-»$Q¬]Ύ¬€€€€€€€€€€€€€€€€€€€€ΐ…σ§iοφMD»N])ͺ5œαͺϋΏβΒΉS΄wΗfyš.=ϋShΣζΥ}J΄ už@aa!8p€V­Z%±μμͺ<‡TρƒσΚ>vNεΔΟΌOΙ;::Rƒ ΘΥΥ•’’’δ>K@.**’ΨΨX:wξœNΛνsŸ”ΔΫ‰‰—uu”ςͺ>s»ςΌA9 @@ ²&ŽξKYYΊΓ—|»–|ύtyl€€Ψ6"1α›’|™œ]\ΙΗΧίΆƒήƒ€€ ΔΗ]€Ηο‘λA‡ΞέιχιςUΩΘΛΝ‘;GφΤ4ρΕ/ΫΘΝϋ4P¨υrs²ιϊnMuύ¬_ί‘ΦΌ€Λc@ΐ| ޟBλΦ¬2ϋΐϊυΙ«―˜λυ o#©{ŸΑΤΉ[orσ¦€ŸΎ^F_,ύΠ”ͺ•ͺγξξiτάψaέ¨ 0ΏRm:99S?9αА:wνM=ϊ!oίJ΅‡ƒ@@μ‡/8m‹'yρ5ΟέϋψTkίr²3)1αyxzΙο¦~°ŸW#ΰkΞ[/>L=@υ…˜`εκέv ₯6έoαkόΗΌEΉ9YΤͺ]gZ΄ς/‹‰©μφΔΐ@@ ŠhυκΥ-―, jΨ°‘|xzz‹fYˆšššJΞΞΞF©5??ŸΨe–Ϋ(((Η³φΪk―₯ž={R›6m,φۍϋεεε%Kό[•―ΥΖάόΨΝ•EM–ώ=Λ’X?”E€GXI/ΖP~~*•δΖΉ9\‘—ulUΛKJŠ„Kl4d’§²8—ίΗH $―|έrssΧ΄\Κ*,‘’’RrοΛ|ρœQp%Έ‚·ΈΖψϋc­p]ψŸ(Φ^ερ%”'ρ«#εˆ k₯’β+ΡiτΟ_"’_ €€% $\ΊHY™WE±%ΕΕ–lm€Τ0ΜŒtΪΌφwZ»ϊ':qδ₯§₯θzΰ-Γ†GDQX³(ŠˆjE#ΗήMξžΊύΨ(›@qq%a¬’RΒ"”Ν*?s4VuΫά : l•O€@@@  πβ %Υ«‡y&…žA ²ΣR Ύ+šάΦΕhYuΗζ΅τ݊…R,ϊΠ3―э·άUαB\€¨ΥδσšPΡέΓΛh­ψΈq#Ώΐθ>S /FŸΥUϋυϋrœέz€§_™Wl€€€ύ8ΌύςνgΔx|mΰyržoLΝ[Ά₯cξΑ"ϊθάj‚HFzύπε":sς8±sχ«  _Š“‚CΒεu*"ͺ5έ:αaςlTι.q›«ΎYNλϊEž#%鲦­&α2pΔ¨[ο¦Φν―ΡμC@@ΐrβθή[κœυΙJ)ŒΤΤΰƏ_-‘‚X>%_‡„ΥΰΩkφT΅ι~ h{Χƒ΄bΡqί~?ύψυR{ηƒ5 g:D€―§OŸ¦νΫ·Λ{ν,κ ₯λ―ΏžΪΆmK~~~”M_~ω%mΫΆΨe―£F’’Ψ¬¬,ιήzδΘΪ·oΕΕΕι„±,Š ™BΜj‰ΔΨ!C†P‡d?ŠΕšέV­ZiζlΉ]»vA]dKœ—Ϋ`N<>”G€)YzθΠ!ω~(Ξ­NE™TΟΩ»ΌΓͺΌ―€ ‹Šsβ¨^i‘­7‘Φ­[  [•ΫE `ψΪΐ"q~f¦ Ql‘ ±:QΎΈ-Ÿ™_"LcΔu M+D±uφ₯ΗΐA@@@@ΐ~pΏ·'=JύϊYƒΊ{žΌg”X\σ=φόδθδdΦρ¨   uO2―[ύ³CΣζ-!ŠΥΡΐ€€€@upqu£ ΰΠ2›ζ9‚άœlΚΚL—ΟΖ*nZϋ;}0υzyκ\c»kUΟU8‰Ε/¦$±;ž±Δε³ί~‰όόiΐu£ŒUA€€€8uβ=9q€Έf˜4’>vΧzξυχhδΨ &cn%ΎώΌψΘν΄wΗ&“eΡΜΗΌIσ_&/ο&ϋιΗο AμΥΐ]¦τΟ?RͺlΝYς˜š u@@ΐ φ½΅«―κΗ³ή€œμLy‚F£ ²OV_/*ί²­ΟΑ«έbωu˜χξdš:{YεΰH2 °ΰυ₯K •+5oޜzχξ-^•ƒΨm2""‚X€šžž.E―)β~opp°tdeα,‹iY΄Κ"ΫoΏύ†.^ΌHΏώ;EEEQΛ–--"d±!‹š*J „ΣχGq¨Už+:¬ύ@u°3- «ΧΨ²κ£Όnΰ7ώ?mΡ’%;v\Όg2„06‘κ»Υχ—–ζSJŠs„πφ2•瑧τ΅oߞ<…Έ @ΰ Ύ~ψϋϋ Q¬‡Α–P°‰-*ΰbwžPΕζŠmͺη@|ύΰ’ύ€(Φώ_cŒ@@@@μžΐŠEsΚΔ7 §ΆΊΚ€ΡgNΩΣ'4N²<Αωνη#€†tηOΩ=+ @j#1iΩgΠ0MΧ8b;€€€€θhΣαZšΏ|•~±Ρ|FZ*νί³•V,šK'ŽμΧΤawΈ['χΰXA¬»‡΅λΤ……PzZ ±“lμ…sΊαςυiζ”g¨QpκΦk άR‹ηM7Δςβ΅°f‘Β©6’|ύ(.&š.DŸ¦„Έ‹šΣž?σ/͘ό$MŸΉ¦άXζη•ŸΔ‡6₯k»χ₯­;ΠΕ gιδ±CthNβ@JbΑξο?~YmΒ`εδ#vλξ!³¨‘.ά χ[Τn±λώ\E#n^Gέϋ ²ϊ: φF //O ]·ΥˆˆζR«'οkΤ¨‘Ω±£lRR’½²(VI\§I“&4tθu:WΩΓ‡ӎ;d{N5d8ΐλΉRSS);;[ŠbY¨ΛYώMYΩ”œœ,…ΐiiiRμΨ¦Mιͺ[Ωφpœύ`!9 ΜΓΒΒιΠΑT˜GŽyaδθα.oYUlii1•δ₯RQ^’hΉDž“bM ši―Fβ]'~Χ±Σ·§§‡0„u€Ž@»pΞ;τγΧKuΡΉΒ§{Ÿ†ŒΈ…5ΡΤG@@ ϊ ΈΈΈ»}Qύ'Β@@@@κv€ι?δFκΡwM{υqΓ=ƒψ|αlzsΦb£LE~T%±ψhκKbϋNχ=ώRUš.σX7»Yd+ωθΩ(=}ίJMNΤΥΟΛΝ‘?W­€Ϋξ~DW† Ϋ'π–Έή¨?οyDTαŽϋž OoέyQρŽMЌן’”€ΛΊςi“§ΟVm&_]YU7ξΩN+?] i¦U»Ξτ괏DPŠVšrΞl^ϋΝ™φ2]ŽΣνcwχοΏXDcΛ TqδΰnϊΰνuΗπίψ`α·Τ’MM9gφοήJ“Ÿš¨ ΑβέΑΓΖTi}Α‰P u”ΐΖ~“Χ™Ϊ0όΉΣ_Ρu£ŸψmΦ4R—·ΧΪxΏ…›Όε.ϊvΕ'ϋμw^’Ώl{}0.° vAewWNόΫΟΟΟ—|| 㱘ˆ]φΟ?OO|¬ΎΨ444”:vμH,ˆe1ιhμΨ±Β”ΐ©FΖΗn7n€}ϋφΙsvξά™X%ΧΜ½{χJΧ[vΙνΪ΅«tΝ… gΌœ6{^ƒ$œ’#ιθΡΓT’ŸN%)Dξbώ€žeΏΛ”–‰φ©TœƒΕ~,Ϊ6φΆY˜θ8Xˆί»γχ†³³–PN‘Έώ9Χ§,!Š>±δαξ.]b«κ.n‘ξ’™j&P΅;ΉΥά94_uOΏ2CL܏zCh@@@@j)ŸDxžœU'^TsΓMγΤEΊmŽŒμk3©]ηnτΦ‹W_xQθΌw'Σ΄—λκb@@@@@@@ΐφ π’ΰ©,₯γ‡φΡ₯Ψ ΊbκJΌhkΚsχKΧVυ9š6oI―Ώϋ±Œf­.―ŽmvΑeWέ»Fφ4ΏφŸ ŠΥA@l›ΐv!r=~xŸf/L™E£oΏWSΖ^ ΖA|ώ=8nˆΞY69)ΎZ6Ÿ{ώMƒc*[°`Φšΐ”α-θ“/W“c ¨9hD»N]iΒ¨ή”–š¬;νΚε—+ŠύtΑ{šσψψΠΌO–"t¨6:wνM‹Ώω›&ŽιG|_€ „ώfΉ«ͺb@@ΐI—/QτΩStpοvϊλ·ο‰έΛkCΪπΧ/ΔΑ••„ΐ@ λ<―NΛυί±Χ]λΌ8+€€`ρ VYΛ‰έ% XΩ9/00Έ>»°&& ήΗ¨ρΈΊΊP‹-ˆλ§€€Π₯K—€8Ά¦D€ΌŒ±?ώψ£μžνΥ«W•D±,ή²e‹bX(U999UjOΝ ΫφK€…δΤ°a#Šοƒβά*φ!GqίAHΠ-6π’Β αΛσ!…BΈήPΎά…Έ @@K€―_ΎΎΎδκβBΕ9ω”™/b]K)[<σ*bOOι,= 9{%ΰ`―ΓΈ@ΐώž•al—EΛŠΔ—e$κ'ΐ VΦύΉJs’'_~»LA¬ΊβPα ΫoπumόϋΧωΎ¨9)2       5B “Α¨/ήVΔ0κrKl2ϋ-Ϊ»c“¦)O/oš1…p‘σ”Wg†EΈ!aΝ4§ˆ½x^“G@@ΐΆ |ϊρ{šτθ;Ψ¨ V])΄isΊλΑgΤE΄vυOFBk*™˜α5'Τ~j΄2±JEvx}ϊΥJV>ΗΗΕΈΰ*N=@;·¬S²rQwy‚X₯"_oΎγ~%+ŸΪ«Ι#  P>%σfΠΠ.a4z@;zϊΎ1΄L)¨-‚XΡ,ϊpΊnQ­ΫSΗk{θςΨ¨yμ{m~Ί΅t^΅ύם  PΗ°ΦΓΓC7jveρ§~b‘kΓ† ₯3l^^žΌ«ΗN|ΔΟ,He-;ΖΠκŸΓy=qΰAξ?WυΌΩΩ9Δ±ά§όό|***²DWΡ†`εππpŠŒŒ€’βR*ΜBς‚4!‡eωeRii •δ^¦βόdρžs’.±μPΛο$-~_°S¬‹«‰·a6;ŊφΜBρ§΄žtŠυφφΦ„œέ€S¬έΎ΄5;°€έΫ6Ё=Ϋ(AܐΘHK_~3©―Q`£` ˆ΅7SDT«λάΩSΗiυͺ•r'έbcΞSA~žόl(ϊLaΝ"ιΖ›ο€Vν:Wͺ_Ή9Ω΄yέjΪΎιoβ›$)I‰”›“EnξΠ0ˆ‡QοA7Π€‘7Š|γJ€=ˆ»M'ΕgΘ…s§e΄^ŽlJ⩇μ’QύSΕϋΞΡΩ‰‚‚C)ΌYρ"$ΊA`Ηζδw/e΄nξž4fΌv!‹²ΟΨσχ?I›ΦώΩuζδqκpMwM2  `Λ²23ΔχνSW~_8KyyΉς;3»F5όϋΥήΟSDŸ9Iό{ΓMDΞlΥξjΩ¦ƒΝŒ5_Μ]π< VJNL7㇄QphS Φν―©τ͏Bqσσό™εo©K±ΡΒy%…όΔ‚ΣΐF)@<š‰ίSž˜Ά·χΖεΰ9ΤK±(+#]Ξ—7iJ‡T‹ƒ‘Ήσ@ΦψΜͺ‰kH|l 8v€.]€΄”djΰ#ζΡΕ\2;Vω6*οεΒ>*ΰίϋ«ώZΣJ–¦jιοΔώ}ύιΝyxαΨ›³–‹j:΅νΨ…Ψ‰GIi)ITP/ά"\”"<ƒ€Ψ(^£ /ζ7ρ1“F3F8Ι.šϋ±pˆSBάE:zptk5©r*;}‚ΤΑΔyι5έϊ”sΔΥ]½„“­~:*ΖΩgΰ ϊΕ΄bαMYŸΓĚ”Φš²²2―E_-―Ϋ}ϊί#Ίml€€TL Qβus΅1m^χ‡Ό/’τmδΨ Κ¦YΟ֘›2«ƒΥTΉΊΦdρw%x»Β―ωυ;ΊιΆ‰Υ4 4  u€‹pΜσσσΣ‰GYΐšššJϊN“j§XΞ&%%I±¨›››šΪ<–H\—U-•ψ·θεΛ—e•ί₯κΆYΌΚ"V>7 bΩ­φΤ©S” £Ί~yΫ||\\9rDχ˜§BpX5μSπά>»RFEEў={(― —Š„€ΥΡ]θBκσz›ͺŠcλ ±­h3/‰J sΘΓΗ‹:vμH 4PΊ€gώμζχ$_χŠJ„i’PΖζKq¬ΠΔ 9ˆ`‡g€ΊA’ΨΊρ:WΫ(ωζΚ'sή¦};7=GΧζ'Jϊμ“(²e[Ίρ–»θ–;¬–…U|.ž4Y#άχŸW~*E±·Mx˜y«Ίj™Ϋόe{Υ·Λιӏί7•”ΗΞB\~μήΎζN›DΓF'Ž~κε/)e‚Ε»#0ͺ_kΉ¨’Ζ‹ϋΎύkŸcFzΝw²+ƒξΪk U$Šeqϋ§"Ί$O`gge*‡jž›·h#ίo£Ηέc±ΕLΌhσ{κΞS_DΏϊ{Ο]Ύ’ρΓ»Λ J½ε?o"Ž‚ˆ  P5ΟΡ4Π³³t6‹4 €ΐ _ ŠΥ`E@ΐF °φσE³ιϋ‹δ‚wcΓΰΙ{vΖxό…·*ό.nμxSΚ~ώf9Ν›ρͺ¦jΟ~C… `19‰h΅œψ7τΔ1ύ4u>ϋi“Ροσ[D`ͺωｫ;bΜtχΓΟΙ<»‡ΉδCbA“~βIΡζbNβΩΙ3kνη<ΏfΛ>šIΏώ°’άΕDa"Π}ΏDƒnmςΊΤδDϊω›Oι'1’’tY.Ο‘ΉέQ·ήMν;wΣ•«7V,ž+—qY‰ˆXͺN,ΈΤ)XW4ΰϊQ4eζ'Ί<6@jί½μ#ϊύΗ/₯ VΏWα"hΒνχTeޜϋ-ϊύ©ξ5YΌ‚οAπΪMN|Ο’XύWy¨<΅6nάXŠ„Xdzξά9ω Ρ4ͺ8Ες= Ύ‡ΞΒΤK—.»Rͺ―“ga-»©*χ4JJ8Άς±UMάζργΒόjυj£b[ΓΙ“'₯£-o;vŒ–-[&Η§τΗΤ>πρiΒπλΜ™3r,{’fNy†~Ag/ώ^|6%Ρ~Υbϋ„XΦX²Δ<₯?³Τύ΄ζ5δΧοWΠ¬©/”;ŸΞΧC°xίΨAτ–ΡgΠ0uχ±  `‡φοΤ΄bιΐ…,Ί}U|·Vψ*'4lt…χž”ΊΥρ­iΦ/ ‘ΙΒ$́Ȁ€Τ:DuκΦ{ YŸρ»υֈbΧύω3=ύΚtƒΉυ9LΩ ΦTKλ0ψϊΘB˜ŠRRbΌFΛυCΒ" ;qτ€f~"8΄)qfS/ 2ά<‘”©m£€Τc…!Fί2ζ.x†ΧαY#qπΙC{·λNέΌE[ςφρΥε+Ϊ¨ΞΉ©ŠΞ­μ―ΚΌ9χ[”σρsM¬ΙβϋOν:uΥΉΕς=ΎGΥ$άπ:―ξΆmhΞ4@IDATA@L#ΐ XL/^”Π-[Άθ„|J+Ξ"puΓ† …hΘMά?.€ΨΨXιΐΪΎ}{/[\Ϗ=JΉΉΉς7"·ο*~ΣYκ;&σρρ‘θθh)xεk˜±ΆΉŒχ±p—UIΌf€|^vύΤwΡ­JΫ8ΦΎ π+‹ΞY¬#ζΫKς…(6κ»šώ=³,B₯₯ΕTΜ.±BλβμHαααδοο_Vu”ƒ@'ΐοGOOΟ‚@ε—RuδΥ«_O|Ά»ασ½ύ—8Τ‘±b¨$ppο±ψςrπTtΊUΒ)ζ½7ž•_T+ͺkκ~v:yσΕ‡ΜΔͺΫdφT鐠.Soσš''Ž2[«΄‘œ”@“žΈ“xaΤE,„}恛ΝΔςPŽ¬Ώdώ “q+\ΩέυιϋΖ”ϋžVκβ@@ΐ6 p@u lΤX­p;';“S“5υ,½ VΣ82  PΨ™Šϊ‚Xž hؘ<½Όφβ›ΟώGί}cΉ₯F,£πΫΟ?1Δφr#M›»ά¬Ε™e4΅Xά›ύφKε b―V&∢ ηΌM/<|›Œj«ήg­ν][ΧΡ³άR¦ ΆΎ£ρΈv§Εk=νΥΗΛν6Οα”%ˆuvq₯†b‘ͺ³³‹Ρ6ώόε)ΰ2Ί… 6IΰΰžντΤ½cΚΔͺΖΞ¦Oί;š2 Pο3eΫάy Μͺ‘kΘΪ?~’"cSLςτ;―ΧψάT…=ͺώ 5½&«γ΅=4ƒΪ΅m½& €€@Υ„††Rχξέ₯»jzz:­_ΏžV\I›6m’βSΑ²Θ400Z΄h‘sŠεύ;wξ€””ιΪΚβSΤnΫΆςσσεZvέγ㌠W+Σkn§U«VtύυΧ“―――Ό~+’Uυ³ΆΊ¬2Ϋό»”…½,l4hυθΡCζ•φρ `5‹©½½}„Έ:Ÿ σβ©T< ^E‡–³_[œ'±bύbip·τ¦ˆˆ)Z/η μ:M€―μzΞώ\Ο/*₯Œό*z'G'!ˆυΔη{ϊ1Ύ’ΐP+Gΰ“ΩoYDΜΚΡρΗίχYBxΐ_pίzιaβ›4ΖGσl$?δβγbΚ]δΛ.]{  CC šš3mύ{μ AΉ9‘ξΕGΗΣοΦUθζbN»¨ Ά@ΰ½7ž£½}μΠ*’ώ…„55:„‹ζΠo?|‘ΩΗ‹ωΗίϋ΅jΫ™"Z΄‹cθθΑ=ς±vυOΔΞ$Jb>ΧοήgR„g;!pϋ=Ρ°Ρ·λFΣcέΆ)'Ž4ψ^Ϋ’MSE¨•Xμϊ³χj\2Ίφ@χ<ϊ΅ ?x2;X]ˆ>M+Ν₯­λԍ…ΏS³8–λW%­\ώ1}τήλš&ΨkΚΜ…šθ²š •Μό*~+δζdΙ£YδyΣmιzα`ΪLžλRl mέπ'}·b‘fΞ`η–uτσ7ŸG³·fβΧlκKhάΔx±ηˆ1wΠθqχRxD”pοu¦ψΨ tκί#΄pφΫ}FΧε-λVΣן.Ώ Ε±Ι‰Wsθκ{y7 '_~‡:uνMAΑ‘ςF'οLA"6όυ+}Ε"R‹&8¨Ω€‘#ε<‰Θ n’¦‘-e6ϊΜIZ³Κλ_M\Cx‘ \•ΦL^{’Z΅§&M#„ϋωyαd~ˆ~όz±Έ’xyωή§Io¨α@  βΕwB֘™ž¦k…?χΖM|T—―κΖΒΉoΣn½ΕΌ |όhΖόδκζ^Υζ+}<;οœ:qXsόu#oΥδ‘°MgO'ύοχ-Ϋt4k0‘-ΫΤOMΎLή | ΚΝ)πρυ§N]zс=W…°3&?IKΎϋ‡Λ™+Έpώ4ύO2W§Qcο&wOu‘ά>¨rδ}—ΉΌά)ΔMΈt‘RS‰η("’ZSΣζ-ΙΝέΓ =€€Ψ›ώΡ „ο˜’¬17UVΏjrΎ¦Χdρχuβ{57Ώ_]„m*`Ρκΐ₯ΐ•έbωρΓ?Π]»θΖo€#FH—TΧ»wo:rδεεεΡξέ»e@ιnέΊ ΑŸ·tde‘μ™3g€XΥΕΕEΊWςq–LμV;dΘΊpα‚<;ΒrbΑλbbb())Is    rz₯ž)}αΆ8€Έ£ΈΟΒ^χνΫW:θšr<κ€€B€ΧΫDD4§FQjj*•δ&RIA69a^ee±όΏ\œŸAEBλΰPš4iB,nη[$² (Ÿι,‚Ν/. €œb~γΒΐ™<<άtk‘Κn{μ…D±φςJ–1ŽΥ«Ύ¦£v—±·μ AtΗύO­ΐΡΉοίet»ΝLxπYŠjӞƒδβ‚Σ'/ί~FgN3zΜρCϋ,"Š=vh/νέ±Ιθ9ξrέ6αaςπΌκ†Γ7Sf½υνΫΉΩΰώ"½kλzΊρ–»4ϋφlίh ΜSWθ=πκΡg0…6mNμZΖ‘IρK£_ΎγbΞΣΚε θΑ§^U7m°kΙ‰ρ΄M8ΕrβΕκOΌ4U.φ hXζΈ“Δ1_,Ρ.ώcw©—§Ξ%o_έqΌ°;JDά=ξΊA,€ι±;t‹Κyρα¬·ž§―Ψ%“;ιŽΑ€€€νθ,=•M<©τιΗοkg’~{Md@@ –ΨΊα/Ί}VΧKώœ|α7BTώ.έΞ§+Ν\π₯†.—ί—•ƒΎZφέ:α!ΝoheŸ)Ο_.™'/Ύ₯©Κ‹ξ'O_ εjvV1£bYp0³_ΕβF­ ηω1κΦ»ιΩϋo!w)iΩG3ιϊ‘·Y5`Υ—KηKAͺ'ώ­ΔΒ ύ >,’βGΟ~CιMα@ΆyνΚ!R4Ε‚Ύ™§NϋwoΥ?Β#ZМ%ί Ζ‹UωχΤΰacθρ #ˆΦ*ιȁ]QlX³(⧃ΎΫ•jςΩέΓKώΞΣ" P+|!>ŸY―$W7š>οsƒΟ›Ž]z ‘ΡΆΡΫ/?"]’ΨΉΤάdξcy/αn’ŸR““DϊΕfη_š:‡ξέOΌ-9)ΉγΊΙWδ|ŒΌζωϊUb] hWώpΞeo;qτ€¦OΚ5χr|¬ΦnϊηwM[κΚ-Zwχͺ―ιuΚUΧΑ6€€€mΨ΅u¦γν;™&Š΅Ζά”¦£ͺLMΝΑ[cMV[δšοΙΙσΪMώ͊υ\ͺl‚€@πο¬:ΠΰΑƒi͚5”˜˜(έ_Ω5ΆcǎΔN±œXΪ³gOΪΌy3:tˆ²²²hϞ=R˜Κ‚’άά\)ϊγϊ|ƒ…z,’eW>K'vn½νΆΫθςεΛΊυφ,δsύυΧ²œΕO,fea/‹vΝΕςρ,μυττ”΄~~~k(,=.΄gψ2(¨…„„ΠΩ³g…yS€’ƒ‹XΛ^IkiI1• ­hΛΕΕIŠΟYt‹ P>Ύ6yxxΘίω”’_J%b~‘'°«3„εε󳧽ΕΪΣ«id,ωli£Ÿ©fPΤ\Έ.–%Š=°ϋj$Oυ<1ρι nξσ"*ΎΉ>QάμΈ{A}ˆά>vxŸtp1Ψaf‹b%^,zο£/μbwΪ—ύDw οqXQ*ξήΆΑ@ϋυ§)» ž_6Ÿ† u>zΌt¨yααq”ž–’ή%·Ώ[±ˆΖέύ¨FΨgP  `G”’ΑMΒιέ_,V76ΤEs§iœ»φHΣζ}f¬ͺμΪύθύOV‹ά»Ρɟ?όό΅\―«ˆ :M€χοΪ’apΣ­-Ά VΣ02  PCώΥ[ 8βζ;+Ό™3rμϊμ“(QwβΔ#§O%ώ=onϊ|αlZτα4Νaό[™]ξτ›šJΘΜόΈόί,š»τG7¬‹N(š‘ž*Ζ>K,‚|Ϋ=0Ώ ^pΒΊκτΜ«3 jκύ.B4ϋ’H±ΣŠβt–•)έΏτέ`Ž‹9uzτΉ)F±κ:pθ12εΉϋtΕG*pMw06@j$€μΫΟ?ΡτeΖόΟ©[οAš2u¦WλhΡΚΏιξ›ϊˆ…κ]&m›;dΝΟ¬šΈ†ŒΉύΎ2σ3PN0yΖΗ‚χΥΐ?,bζk{›ךΔ•@ ΰίςύ;”Ώ(DY\kŒ ;Δ½:ύ#jΥΆ“±έf—qPΨ―š}όΕ©d ‘M‘XX|ιb΄ ΪΊκ»Οtίy`όόχ ·=/³Η‰@@jΜŒ4ƒNy5ΈTΨ`§‘Αk=ΤΨUΥ‰ΧdL½”fΎρ,₯&_i3αR,Mυ ϊhζλΒ±΅4jLμθ}φ$ρό†:q ‡χ?ω†ό_χ9†:5χ ϊν{]ίAW]·9hΪSχŽ&vβγy‘†A!ϊU°AgO ΛρWξupχύQPH¨I#±ζά”I¬†JΦX“ŁϊZ΅ιDGν‘#ΚΝΙ¦Γ"0gU‚aW4  6MΐΧΧ—FŽIμΒΊcΗ:}ϊ΄VT bYπΚ‰Ε³‘‘‘4fΜΚΞΞ¦σηΟKΗΨΈΈ8)8e1?X\Δ­Γ† £ξέ»WΈφ 2ΰψ,xˆˆΠΞ’Ψ 6Θ>° •…ˆ]Ίt!΄š›x,yμ­BV%ρΔθ‹o~ dΛ}Ύ¦[sϋ½š:kW€Ι#  Pw ¬X<—Ό?E 8΄)=ϊόš2d@@ΐΦό{τ ¦Λ%₯%šΌ± ίόuΫέδ-M*α¨anϊτγχ ±,ΈeκΔφ<‚ΪuκZa—Ω!χžGžΧΤϋώΛΕR¬)¬‘Μωοκωπ)#[Ά5Πe¬+Ύώ4jμݚ]ϋwmΥδ9sόΘ~]G^οΪ{ ._ήFΛ64»Ωa @ΐΆ ,ΡϋΌι+ζrΛΔ*£ mΪ\\#&*ΩJ=›:d­Ο¬šΈ†ψ4€‡ž™\!?ž/Sά­”Κ— Z‘δρ  p…‹^Λ{”ΕΙYΉγΎ')ͺUϋ²ͺ˜UΞN¬―<9ςrs4Η »ιvΊνξG4e•ΝόπΥΊgLΏ Gχ₯›ϊ·₯ƒhόπnτΕ’5‚Xvŝ$Ύ›χμ7€²]Αq  ΅ŒϋR'Ύ—ΚΑ΄ΜM<€N) XΥe•έξ3h}ωΫ&Š«χύΠΎ΄NΈ­st}Al—žύισU[¨}gγξ~ΌΦ’ΈΈXΧ$Ο=ρΌΨΫ/?bΦ<ΟΊ?WΡ½7 c»ΊΖ±  `3Ξ:¦ιkphΈ&_^ΖZsSευ©:χYsM–ΎPY?uuŽmƒ€@] ΐΏ’’’θζ›o¦»οΎ›FM½zυ’"XEΛXL4`ΐιΪ΅kW)6e'X~(ͺ-[Ά€Q£FΙ6ͺΣ½’ΕͺΚΉ•g5΅nέZ:Τ²«-χ…]b•ύζ<σx ˆ­ ύΥ?F~±(6 ΐŸ¨Dά§ΘΏ,ξUδŠ—­W)³Wυ„³ep›ΝK"tvdfρwu――)³?Ψ6D€?Σω:ƁJΔΈB‘ˆem˜RŽχ‘ ½˜Uμ*Β]T`]<Ό»phΥΆ³ΑΠ\7 L]pώΜΏκ¬n;Wo‘€n‡™ƜXΉ vΛ]8χΊυ‡ˆι'v?Ήσώ§τ‹ež?•΄·αβReίύOLR6>>†–|τ.Ŝ?m°ίΝ4zά=ε({%ΐ7>»φ`πΨΑD-NΏρ–»ˆ]fMMΌ_νΊtXάXΝΟΟ«ΤΝ`SΟ‰z  ΅›» Κϊ"m]§¦£ό½oς΄ΘΝέCSŽ €ΨGG'M—WΌ’†Ώ…œώ‹ψͺΩ©Κά+~σ£²iΙΌ΄\8ͺΣΝγο§g_›)'ΥεΥ±}σχ›άμπ1wŠΐoˆIΡ+‚avBΉ}ŽZθ AMn°’ω·Ξš_ΎΥ}έΘΫLΎΑΡ³ίPϊrι<έρΌhδφ{Σεyγφ‰‰`AχΙ2ΎΖ™Ί0633]Σ2 ΆM€?oώώύΝ ΜjMxθϊυ»ΟEρ|M¦dΜ™²ΦgVM\CΖήω yxz›‚Œ5nB—b/θꦧ¦θΆ± Pub~ψW£₯βžΝ o|@έϋ ͺt£ό}ς­€Έ˜σš68Xκ &wΤXF&%ι2ρ£*©YdKzgξr hQ•fp,€€@-# /ŠΥ·šΪ]/oΝ΅&59ΙΤCMͺwζδQ:+œΥΝI§Ž¦MkσΠM?e€kη.κΧw”Χxεή2/|»~Τ8)ͺhњ ςσeN{”6ώύ+ef\=žΧš°λϋ¬…Ϊyύs"  Pϋ €₯h―a|35YknΚΤώYΊž5ΧdιgQ»ϋZzœh@κ"ώ]Δ’ ΧωϋϋSΫΆm)>>^8P†J±«š οΏώϊλ©iΣ¦tτθQŠ•Ξ±,ž ”’ΤvνΪQγƍkδΎΏΊoό[°oίΎΤ¦MyϋΚBX$°&^cΘξɍΣΙ“§¨¨0‹JςS¨ΎsΡ­«Ί“Šϋ(\-‹‹„6Mk3ΘYϋΒΒΒ¨Q#C­KΕm‘Τ=ό^dQ,_ο8‰KŸΤœ°HΦΤ΅Quš}Ž’Xϋ|]«uT·NxΨδφ‹D€ξGχӊ…s¨,'W“« bΣζ­hΧΦυFk­X4‡ψΡ²m'ΊΆG_κά΅tρςn Ώ(sΔԊαύ»ŒV o.*…έ§.lΣα£’Ψ»·©«aμž@Ÿ7˜<ΖszbϊΆ5ωXΘ |ψψ‘"šηΕ›ηNŸΒώNf΅ƒΚ  ΆO€#Ζ΄r-œσεζdiδκζN“§D»τΤ”# ΆH s·ή΄{ϋ]ΧY(ωcwΠ Sή§°fΊrKnp *ώΝ­N·Mx„žzešΊ¨ΪΆΕ„fY!ΖNΚs,T8zpnwlLΝ‹b9PƒΎΐ¬χ€λt}ͺhƒ―[Ώn>‘«Ζ‹?υSΏ!#τ‹LΚ―ύγG“κ‘€€mHNŒΧΈR³Kbێ]Lξ|@`΅ξLϋvn6ω₯’9σ@ΦψΜͺ©kH1‡ljβΉ,uRζ΅Τe؁ΊN€ rΧς/όΚΚΚ‚—4Ίxώ,%'%hͺ³ψόω‡n₯Gž›Bw=π΄fŸ©™ΕN£[Φiͺsί¦Οϋ¬ΦέpΏpξ4}»b!=ότkδνγ«ι32  ΆK ;3CΣyο¦ Τςb1urp0g§ϊHν6_ίσ9ϊEΩ)+ρ<Z ͺΤγοΑs§M.ΧΠΤΩK‰λ©S–^@―ΒΒέnžχyιΝΩΥZλ M·>²Ξχ?I/=:^I;«;fΗζ΅Δξμ·άρ€   ΆG U_kΖ΅ΡsSΦ$lΝ5Yϊσ_ϊ―›5Ήΰά  `λrrrθβΕ‹R<κλλKμΆͺΈOςΨΤfQΚXύόό¨{χξΤΎ}{JNNΦ‰bY„κεε₯)υkκ™ϋ.…‚|Nc}―©Ύΰ<  &ΐβμf͚Ρή½{)%-ƒŠsβΙΙ£ ‘ƒ‹ΊZΕΫ%ωT’'‚γ–Rƒ )88˜<<`θQ18ԁ+Χ~/κΟkrήΝ κˆαjΉΊ4zŒΥbx'Gφ<)’už>qDF²η1Ρg„%|‘ΕΞS^C-MptωχθβΗWKηΛ¦82v—žhΠ £₯HΆΌ/Μe9έδΡΫ“-―krίqΤXβθήp4FeφJ Yd+“‡sώŒ¦neœ›"’Z“Ϊι9^|6A«ΑŠ €Ψ=}BφατWθŒ‘HτMΒ#ΔbΥΟ‰―H  `z Έž–.˜©ω-Ύ{Ϋz?Ό΅Ώ¦;±»hη½eΠ(%Z^UΖύρoκ~c«Ϋ©Μwwυρζl·lέ‘L v₯n³m‡.z’Ψσκέ5²£=§@#³œ»xΓΧ?°J}ew³€Λ—Δ³jκ8$΄’αιφ;»h_›b% @@K€ƒ!Nž±@[XAŽƒ%ώϊύBμ²Xσ]y‘π₯G?jΥs-hw―_³ŠV,ž«)d‘=»±6 Φ”W5γζξ)’]WΌ¦”J)';‹ςrs NΙΑΒV}³œνέA.ϋ‰XΌ‹  `ϋ<½΄BQZ^™”/\TΥΙΗ/@­τ6‹Zυ±<‡sΧƒOSοώΧShΣζδ&qΩ"EŒ¨nYχ'}ύιGš ><―Ε.ο3|©ιΎ0–7ΧΊΙέΓΛΨnYΦ4’­ό›ξΣ—ΤΞtϋΰ-ΊιΆ{¬Άΰ»Μc€€€Ιͺβ[ΡIͺcnͺ’sVη~kΙΦ”fa—ϊκδ†ΆA@ 6ΰΐDηϟ§―ΎϊRήŠŠŠ’Ν;S‹-€Ty}wpp OOž‡t—N{\—ΛΚ[W_^{–ΨΗγ),,"ΎORύΰυ–X3a‰±’ Ϋ"ΐ’»ππ¦δγγ#„δ)T”—(_s©Ύ£ΈΗgrœ±R*)Θ‘Βό$!Š%)^gQluό―Ϋ]τL#ΐοΎf±«8_/”ΔοOW •ϊxΆ}ΕΪώkXξq*uο3Έά:ΖvΊθ-Ό1V‡'z8φ/ί}FΫ6ώ₯YD`¬~u— q ύψΥR:zθͺΣKEηvΓV'ΐFκ¬IΫAΑ¦‹:Υ š:€>†·kκ3«¦!›„λy&ΐ"ύ§&½CC†‘§οCΉ9Ω²ό=yΖkOΡοΦ‹ZMIgO ι“Ÿ4¨ϊμδ™ΤA’±tΊλ§hβ#Ο›ά,Ώ;{κ8mσ!+ΝΡ,`qπB!~εy&·‡Š  ΅—€ώwϋ¬ŒτJuΆ ?WsœoΥE±|bηUujίΉ½9k‰ΑΪOoԘ7‘¦Ύό(νΫΉYwθΦυζ΅PίΑΓueυΚXe:yΪGε b•ΨUwάΔGiώΜΧ•"X"φΒY³‚–ιΖ€€@­ ΐkπΤΙΛ»r.κ557₯ξkMn[{M–w_ΝpSS°–K¨$”tόψqZ½ϊO)¬ ‘b!ŚšX[[‘ο9Š―PΤkNŸΉ]S± mhhεξ™s>Τ΅?ό>  FΡœΈΈ0K\Σ©Ύ³·iƒΒΩ’’b!€Ν όtωFUŸ1­¨ΆO€?ΗΩ ƒπηΊςΩΞygΫ F`2ˆbMFe›…Χ+Σ]M%»ˆLyξ~±ˆr§©‡T{=VωΏφξΗτόΓ·‘ώΒRSNΞ@—2‹Φώω3Mžώ‘tŽU§/¬Sο«κvΎήM¦ͺΆ‡γA 6(λ₯~Ÿ/œ;­_D<)[ΥT ιΈͺναxΪG€ΰ~σوέ^ ΄ξΉ·]{ €η^›)£ΠΧΎή£G  PuΌ@pΖότΑΫ/RBάE£ ²{ԎΝkεƒθ oN£ΗίK#o™ z=FΏ°<1eFz*Ν™>‰¦~°T0‹η½Ό΅(¦œΐΫΗOS-1!N“―‰ŒΎS¬ΎPΧR}`W°Ύ\LKζΏkx¨¬s°K‹")«ΚAl‡€ώ獗XόmnjT‰|S灔ώΤτgVM\CψΖΈ―6ƒ2^<ƒΤ<6βΨi4sΚ3Ί“Ÿ9yŒ8ΰΛ΅Β1Ά’”)ΔF―XBGχΥάž+ζ™ϊ &ŠrΓž†N°,ξšξ}5η-/3rμέrΎD=Β$Ψ• @@ΐ6 €¦ §-Uςτ6Q˜πί15=7₯κjnZ{MV½{4©zb慁“€€ΘΚΚ’N±™™™ΔNy999RkMmAΒ 7mΪD›7oΆ¨£+―+cΝΑ˜1c Š­-/Ά φƒ*YΘzψπaΚΙ-n±Ιδθ!Ί—Vl[„ψΌ(OiΣ„86ŸΌΌ<)((ˆ<<δA@,G 5%‰¦½ϊ8νΨτA£M›·€Η^x“zυΏΞ` @@ΐήπg]7ΰŸ?~€u"ψΣαθZŸWζ0c’ΟΠόw_£εΏO,ϊŽX,`NzθιΙ”+„ΆμD₯€u«¦λFŒ• •²κxφ¬„(6?/GΣW·šΏΉ£ιƒΎPW³³’™„K±4ι‰;ιΤρΓΆ,\ [ G–ώCo€Π¦bξgh…Η €€mHNLΠtΤΛΛό`Ό@ Ί“5>³jβΒ7δΜI5Αڜώ .Ψ#λGέ&]αr²3uΓ;wζί E±Όλ­—’Ψ ηtΗρ»Γ²KlmLC†ίL‡χο’AR”ώρ½³ ηN‡Uφγ@@ΐvψ4Τt6?/—ΨH1˜¦’^&_Μ₯§₯hJ«κˎεΗνΥ΄ωπ3―W(ˆUπυ  §_™Ao<ΏRDό{ƒΏ…\qρq7"Š5WΜʁαΨM^έWΕλ[@@l“@Ύ^ ‡Σ紬17e-Κ΅mM–«pxB¨:|a“œœ,]Uωή»N²“₯™·)ͺή ΅ΐσ±ηΝ£;wJα“%šε6ω> 4νΦ­›%šDu”‹ρΨm˜…¬YΩ—©$?…JK„nΕ„Ϋ‚ΒΣ’J‹s„.Vt~"8Yγƍ₯X»ŽβΔ°Aΐl|S°ΚύxxΐοΝΪδznφΐp€Ω Š5xχυ§ΛΔ²Ψ c—žΤ¦ύ5" uGjή’ό`™3m’QxζΊmDUθββJ<9‰ΖM|”Ά¬_MΦόB»Ά7K ΛQίޞτ-ϋa½°Οv•­{5π%λ'^0ΪSWύb³ςž•XˆfΦ Pl;]«Q™φαgβƒ ΏΤκm‡5‹Τ+©™lVfz͜g:L`ί-τΦ Qr’vΡ?Gš}Pˆ΅FŽ€Ι£:ό‘ƒ@]$ΐQ7ά4N>Ψv­΄oΧfΪ·s <~HήμΡηΒΞW/=:ž>ωκOj‘ΏΫ Ο‰/Ύ9[~ΖςΒΛ΅B„w1ZWoΦΤ©s·ήΒ½ΓΌhμΊLΨ(,0? VΒ%­3¬5\²τ˜f φ–L|C]Ατ±Ξbž£{ŸAΤJΜg°sX°ΐ6 Σ8Ÿ:qΔ’]A[ V&ΐξΟκΔnήζ&>SΙZŸYΆz ©ΞΧmƒ@] ΰμμBMšΙοΔΚxΟ QlEiιGοα lLοΜ]Nόέ»Ά¦k…cή_.Φt/.ζ|•οmiD@@ΐ*όœχr|,qΰ+SΣe!6ΥOΎUtŠεΰ ϊ‰Χ˜“:vιaPύόΩ“:Q¬§—α\SΣΘ–ΗTTΐΧΥ’Ψ賆}―¨ μ¨=τ ι~(«§Φš›*«?Υ]nν5Y™iš!r@ $ͺ`! τα§ς\υ–­ΧΧτWVΰ€₯ψ™ΫπρρΊ± Ή\9­χ²ΪΕ™yM{Γ† ΙΧΧ—¨€(C8ΪχΑ=;hž­rπ‰#ϋ.ώU7-n¨lϊηw:βY\ΦYΎ‘τμδwΥ‡b@ΐΒΔ HuβΆ-Ϋv49‚°ϊXkoηdgΙζΦξΞ φL€#ΟΏππ8ΔQ·ήM<;…Ό}|νyψ€THΐΥ͝zφ*\™Ε―»·m ν’υύB,šU„ϊΰν—hΞ’ο•"£ΟNNΞτζ‹©ε~W7zφ΅χθΕGΖικ']ΎD ήƒ^zλͺƒ¬n§…6*!Τβ’κδ-aΥtΝ“pι’E»πΝg£{ΆiΪd§°η§Μ"ž3A¨;τ$θ}šB"ωrΌ)Υ*]ΗZŸYΆz ©4h  # οB“˜pI·ΟΨΖΖ~£Ο>ω@³‹Ε΅Ση}F~z.}šJ΅ Σ©koƒ^€$'”‘@@ΐφψωtϊˆp7G{QΟ=(8”|΄kp’r βbLγj<‡ΪΤΌΐŁAΔσ5κ >Ρg₯}Λ3s0L! Q§wt4?H…ώ1ά&€€€νπςφΡt^}ΡμΠΛXknJ―5–ΥΏ?QΣk²2υŒpύ­±—'°s,ˆŒŒ$///ΚΛΛ£Λ—/ΣΩ³g₯#*‹„l-±¨©U«V4hΠ ³LΨ Ά¨¨H2HII!~dffJέ@PPuκΤ‰©uλΦΆ†ύ­EXXνννMaaaβ}vZbσ¨$Oˆ\λ•ˆ^–oώT*ΦΓ—¦ ·Ψƒ2€TίlεΏY™ΊΖbΟ±IQ, ΅@@ͺ@nN6½φΜ½A¬‡§MžΎ€ϊ Q}'FΛ  `ΓX9膛δγιWgΠλΟή'~'―׍θψα½ς¦9U·C΅ΡαΪ:A¬Rά³ίxύM΄~Ν*₯ˆ~ωξszγXκld1ΎR6*#ξŚβŠ[….=”HΤIΏOκ}ΖΆ3R駕Λt»ƒ‚iψθρΊό!vV§~ƒGΠ”χš1'+S}(ΆAlœ@@ Φ=*!Ξ|ώޝ›«•‚΅>³lυR­/:B φΒyΝHΩ9Ά¬Δ.²οLzά`χ‹oΝ¦Φν―1(―mτΥΝέ“rs²t]ΛΟΛΥmc@@ΐv 8‰E^|-R―Y8,D±ΧΌΥδAΤ ¨Υ{ΰ &[VΕ’"νϊ‹ϊŽŽf-`VΪυΧ0΅˜ΙK/¨YΛ6hη–uJuΊP —Χ˜θ3Ίγy£y‹6š<2  ΆE@? $Ο£›’¬57eJίͺ£Ž΅ΧdeŠ΅›κ§X5 lƒ€@ε °0¨cǎԯ_?ΪΊu+₯§§Λη-ZH(‹fY@ΔkΚ[P^*{\ym–΅ΕMC‡₯.]Ί˜Υ_FεηηSFFΕΔΔΠΏώKϋχο§θθh)”  PhhhY§F9˜DΐΥΥ•ΒΓΓ…Ο•rςς©¨ •κ‹νς f …Kl†PΠ »…„„£cύςΓ^|Mb—tΎΆ)βΕ`²ϋˆbνώ%Άμ/ §Xc‰£z–'ˆεcΜ:fμP‹”ύύϋτνηŸ΄εPρζ}ΆJDΠ~Αΰƒ‡–GΠcw§‚‚|ƒγΥB^Ž€ΝηΡOρq1”˜W‘P/αR¬fΑŽ“ˆ$ΪTΙβ@@E€?[Τ7qϜΔ2%±ψτδρCšͺΊτδk"£Ο–™Ÿ:~˜’Z·7ιτώώ•Ο›«{넇‰ώΕςo‘³§NθφρΖπ›ο0IΛuγbΟπ>$Ϋ#Πͺ}gM§ccΞ‹…εiΒuI뜑©€ΚΔ η(„UW²ζg–­^Cͺλ΅@» PWœ;}‚’“4ΓmΪΌ₯&―dψ;Ϊ+OL0ΈΏsΫ„GhΨM·+Υjύ³»‡ž(6?―Φχ0ΐυB«ΎŸz@OδZQ+{vlTι=πzMΎ2}§ΩΌάŠΏCBΚ B‘ž|q­_ŸΥ²¦Z‹65’Ψs•ψέrαάiM›Ν’ZiςΘ€€Ψ¦‘Ϊίvιi)ΐšsSv+XsMVœb«ρ•EΣ u™―eήθΡ£)77—h°ΕΥξξξβ=—L₯…Βͺ~ς,‘‰-).’Ψtͺ'ήWόž„@[ƒ0™€β ΛΧ ΎNqޜk›Ι'BΕZM ‚OέZέwtΞ ΤQ8Υ§η›ιε₯ν&ύ_εΥ7wG SίθQμΠήrέhZ΅νD-Ϋv$Ž˜ͺŸΨVIΧtλ£l<ΏρΒƒ4oΩΟδ(~ KΌhμΎ±ƒ4ξeJ½!Γo¦7g-V²xPˆˆj­yoψυ2ΊεΞM^ΘΝBvυ"ρ­;ΠCOOVΑτM_ƒΚ'Ε‚υ}”« ψKΦΖ~Sa@@ΐ‚XΨ΄zΥJ]‹όχY‹ ˆΥΑ€@]"ΐΏΩW,š£rQaI’X>€ƒI±†Vί$ͺL hؘ|κϊpΖ«ΊΓY`»tΑLzμω7ue–Ϊ(.*’mbށ_›’V~ϊ1©ΖζT)ρ―)η*―N`£``‹m)ιΗ―—ΛSη*ΩrŸΧyՍ—+φθ;DWŸΫT;ρŽVbξΓΤ΄nυΟ¦VE= )Ž;‹Οω‚PόόΓW‹ιήG_4©χΖ‚št ‰•¬ω™e«ΧΡ’€@–-xΟ`O»N] ΚxnwκK“Ύƒά΅=ϊΡc/ΎePΏ6Έ»{P²ͺƒJ0U6A@l”ΐ`12ζk ‡ΐΑφοήZξ e¨ΌΖβΔ‘ύJV:‹wξ[—―μFhΈa΅}»Ά˜%Š=wκ8•””θΊΐ‹ΪτƒX Έn€f.ŒΗoΪvμ’;ΌMόNi)IΊ*|!"’Xl€€€ ΠΏŽ•΅ΦQ=4kΞM©ϋQΣΫΦ\“•™‘.œb58*HMM₯ΨΨXςσσ“βWnίΎ]Ί₯’ΏΏ?y{{;\š#*χδYD:fΜy\•:YσoH~π8yΜ 4"ΩεΛ—K‘0»Ε6k֌’’’j 78…½ΰ1ώβGJJŠ»fSq‘KΉΓ--)₯β*)ΚσR)4gρ7€@ε8::κ‚5°c,_ېκ‡Ί5\ŒΆͺBŒGEaΡgZͺϊvϊΥ3]Š½@S'=r΅@o«Tu#Co—ΙYŽZVZόα4ŸHQΧeW„σgOͺ‹tΫ­Ϋ]uQ`—²*νέAο½ωq΄RύΔγγωt‹Ξτχχr£~ς Έσ§4bσ˜σ§iύš_LβSXP@όώW§λD€δΚ&vd-u:yμ :ktϋ―_Ώ“nOFw’@@ Κώωγ'Mόέͺ?Ύ_i˜  Pw΄jΧI3Ψ][ΧΛHxšΒ22)υό™«Ώ]έά)<’ς7€nΎγj©χ[}ε§ θ_ΎC—ΡΕr‹—ΜŸAό ’Δs«ΎϋLSmΠ 7iς5•α…–<9IsΊΏ~ϋžL‰\ΏK8οήΎAw¬‹«uκΪK—oΰλ/oςι ΔFό₯‹κl™Ϋ»·m ν›ώ.s?v€ΨΎ4xΨMΗΏϋ|‘pΩΞ”Λπ’ςŸV.3ΆΛbeΦώΜ²ΕkˆΕΰ£!¨cψsοΧ¬_³J3r̈†Ε³Ϋ6ώ₯©Ϋ8$Œ¦Ξ^JόΩjKI?Έm^^-u}(‡‹HΊφ¨©ΑΧ0ξP^*(Θ§οΏ‘©³rΞ?UMν:w#/οšfU°nC]™ϋΎpΞ;κ"ŠΑάτϋΖsOY8H[Ecηϊ\g™ΰ¦N7ήrΉ{x©‹°   `c"[΅Σ¬ν‰λφΤAŒ ΗΪsSgοNγͺκ=Ž3“}o–¦iΪt_)₯₯ JiYd+Š"*ˆ’ ςxŠΈ"⊒’ΰ²<Κ" U(P–RJ[ΊPšnIΪμΝμ[ϋΞΤ;ΜdI&i2σ=0Ήwξv>37“άΉΏϋοj›cά‘<'«γΕ§¦Μ8z0ΊΜ:@°ΨΉs§όυ―• 6H]έαοκλλ%??_Φ―_/―Ύϊͺ<όσ²bΕ S9υ yόρΗύΊi΅ΩU«VυϊΉ:‘υ{y ž~ϊι2~όxs³¬[·ΞVυηοΗ‘Ψ'Άihθ{KƒΧcΗζš ^€jk”ƒ­5fγ>ΊΘ—ο–FΨη΅Φ™`l‹©Ύ#VOJβX„―Cτ. ϋŸ‚Υ€Ί6½δ‚½―…)†ƒ‘Ψαπ* ‘mœ4uF—[£W”ώίk/•ΧσΌ †κΑ€}…{DΏΨψάy'υX%Ά½½­Λe229%՜°0­ΛYήo\ό±ως7S-ηU’Ϋ·JΎ ΑώΗT=ωΓ/Ύ/W^tj·Ϋ7ύθΉ>ΛΌβ«Ύ'«z?ωόS—Ož6G~ω£[䉇(ώε^ωΡ­ΧΙ₯goΧι=­σXθœtΚǝAξ@ ƒ@ξψΙrρεΧψŒΥ+oΫ²Αg\ǁ¦ΖΉλφDCιNΣ@λŸψ€3Ψ§ϋi¦²΄wΣύ|—ΉJqwM―Βό³οOwO3~ άa*Dœ΅h²ηΆόΠ?ύΨ_=σ9ΛΨΌa­_ϊΩκΜγά?τ§_ω5/!ͺ+ω„OΧ.ϊ\χcρ™@ FŽΚ­κ4ύϋWχπ§ιί΄ υ΅žI§šΏWƒ†ž‘<ΠyoϊξΟ"ΞTŠυn΅ͺ½yŒ 0Μ>vξ§}z°ήTeύέΟοπη=ΠΨP/7]}±lxχMΟθHSEαŠkoφ w|p ΊJ.<νŸΫ~ι\uζΡ‹G|μΌ‹A{―Η:~ψ­k{Ό ΉN¨Η ξΝO|.¦γΏςοθ]§vρυ·υύuςΣn.hξL¨Ώά}Ǎ²γΓ-Ξ(βνκwΟ<@zRτ1σyΆ΅φ@ηυθ„GϊΨ”gc»y0PΗΰΤ9Yϊ9ΌmσGη}ιίεΗ.\άMο ¨€a?όπCΩ³gΤΤΤΨοζ Dθ…²mXVŸ«φϋ¦U0kkk‡υχΐ9992uκT’ͺ¨¨ΌΌθ $&ΔΛ¨Q£|Ξ« /=z‹@4λœΫ¦Η¬Ž§…@dψt•žC`Ί©œͺ?8τγŽMO’ΊεΊΛDO 8hžΧ+‹ϊΣŹŽ`΄ύώ―εšΛΞ6ΏPt>ΉV«¬άΧΝ2έ­ϋμε—ŠόρnΗΜ;^.ΊμjyμΑί{φ<Φ/ržxθ>ΟpOτΔΫΫοϊC§«™φ4Ο!Ž—_}£όΛT[-/-²έΧ“5―ωμΩςΫ~"§|μ|s΅ήD‹ώlŸEw~ϋλ’Ue½Ϋ-?ΌG2Fz ψριg_(λή~Ν3Ÿž@zύΛεšΏ+³ζ,1γ&ڟ}»wΘ3&|πμJkλαjUYΩ9RZΌΟ3/†·@C]­θλο΄*…;ΟuΌo1UΕ½ηΥηυ Ώ?M?_;ΞΫdTΡW­φ—χΑ&ŸξςG+ζΏώ΄KΈN΄J ŽW|υfωΙwχlΊžΰ·ΕόŽ|ύ­?–„ΔdΟxηAνyκ‘ϋ叿ώ‘3Κήγ"z aωg'ώ'Ο²wlΫ,ΰωάU7xΖλΑsO>dηVύύάiZL/XsΧwnύ₯Ξh{ΙΟ~I22GωŒΜ= ϋ•o—ωΚg<«U£k.ϋ„|ύ–ΙΜΩσD/¦Mη\υ―gδ—?ΎΕηβ^#3ε†ΫξτΜο<8φΈ“dχŽmΞ Όπτ£ζΟTω’ρΡ‹9M «ΟŠόY^6Σ¦Ϋε|9ͺΓ₯E…Ά’€χί_:^[Vφ˜Γώϋ―ždτ‹ά,§›‹ešv€Ή"ͺwXΫgb@`Π&O;J>{εΧΕϋΒJZφŠOž*—\y5{ΎL2•—4$UΈg§<χΛΛ/πkΥcΎz²jƒυ3«»νŽŸ!έυ…ρ„ƒ€ώόϊβ…ΛzνͺVCέWΈΫ󳬫Ύρν»D«Ώvlw™0MΗΦΦΦ*7˜γΑΑjϊ»Υc/½¬ΕυΈœ„Dί+Ν——>ήήγL<‰ 0lτ;S½0š†aφχϋο‘’Β|Ήκϊ[=η=TUVΘF„}δΟΏ•-ίu&΅χŸωό΅2nβTŸqήνΫΝρ½ή£:}gεύδMΐvεsOˆΗwΪκ—_Ο~b‘|υζοΛόE'‹VΉušž_²ρέ·δχζΒζvΈ@ςY\" O<Ε™Τη^/ŠΌϊεηΝί/O{Ζ?σψίμwΖzleϊ¬9žcbzlE»όφξΫεύuo{¦ΧW\w‹€ŽHχΗ €ΐπ˜³ΰD[8ΓΩϊλή=6ΦS;Η¦zΪΆ<$ΞΙΚΫΆIτΌ§3‘tΌ“σχ € $$$ΘψργΝEbŸΉ‡9΄ΊκpοiEόŒ @ΤοΒKKKmΠ7ΨV=0ςT DGG˘1cμ>W__mς°36>έ6ΕδΆ0σ6sΌ"I4¬MCΎ θΕωτ'm†ΥaZx πŠ‡Χλέοήκ‰ŸΏζ&ΉžΞ'[: οκδ¨$sηΗΟ½ΈΛ0iEY‰ Ω:)}g9ήuΜ|Ήϊ†οΘoΪυBYήδι³DO„θͺ]χΏ?ͺͺ ϋNWΟϋ3N―²ϊν;ο•ρ“ΊnλΟ2˜pΠŸ7~η§ς-ΊwNΚΦƒ£|Υ[Nξ»/U”ΫΎ» δ_xι—δδΣ>Ρo2ύBY«Nξρ,«²’L~pΛαjΆέ z™9α_«9iυ- €@πφμκ΄°Ž!ΩNψ1’§κ|~ΜΞ$ €ΐψΔ…Ÿ΅αGοϋ4 ωςΏž–±γ'™κ‘“%gμ©3aΨβ’yoΝjŸ“tγuKΟ87(ύΠ/W½τ¬Oυ/χώT–žώ‰N•LϋΊBύΫΊΔ„6υxΔΪ7_±·τŒ,•3Φ^΄f§©ϊαό-α½Žγ—œ&W}ύ6οQGδρ 'Ÿ!Z1φ…xΦ―ωωζ—/²Γcυ‚]&ΈͺŽŽύΠ»wάύΗ.C§_ϊΪ·μI‘ήηΡ‹|ύσ©‡eκŒΩφ’Aε%6ό¦Χ8M—y۝Ώ“3af½θ6ύ;λ‚e³D—θίE^˜Α$@IDATr₯3Ή=uΤθ±φ5pFώŸ ZλMΫ)?_Ύχ³ΓηΉG##pΥυ·I©©š­'¦;M/pϝ=,Τ€ώ¬9 ε±ΏύΞ™-θχƒυ3«γ†χϐŽύapΠκvΫ?xΏ_έΥο€nϊξΟνοΎ]-¨« Ρι…νΌƒ=]ΝΘΈΑ¬B˜œβ³iΫΝEΖτb&ήKρ™€@a% Λί~Χοεςσ—ˆ^0άi«^zΖ—yΖVAM6•ΞχμvžςΉŸvΤΡPL0[’ωμΉσ·Λ7Ύτ)s‘-S%εΏMΉγ¦«μ~ιq*½ ¬ΫΡ wulzŒηΊ›Πq΄ΟπMίύ…˜‹ϋhΰΥi»ς>―ρ;8zΜ8‰1'?ηοΚλtlE'˜3Ήΰβ/:³r 0Μζ.8Α§zAοcΪ>OώwΰH›κj[:ŽΣ‹H Τ1ψ#qN–σ½ƒΣΟγO:ΝyΘ= €@&Nœ(W_}΅§r^i‘k4Ϋίsύƒ΅=}YŽώΝ©ί·khJkΠ·₯ΕΏb&}Y󄇀“IOO—€€$©¬ͺ2…εZΝ{¬ϋΎ:Τ.[λΜ¦Rlb’deeu?1Ο €@ϊ™€!X§2lΗαgζɐˆ ™žΠ‘AψάUί0'˜κtεΜξ6@+ όψ7™“S» ΕκI«ύύέ’W νo»ψσΧΘσ%Ο?ώφϋN'υϊ»μ)3Ž–ός/ζŠ±]Ξ’?4΅Βλ’SΟ’ί˜nΗ«‘v9“ΧHύςηϋΏψ³½ς©Χh"€@‹—}L~rο#ςo|Qτ„'ο¦_ήvχNχ™/\+_ΎώΫή³τω±^Ήfύκεηϊ|yλ,°«‹hu€«Ύ~«όπ–kΙΈG’@QΑž -‰Ε €‘# ³ώοχ%7ε)ΜίιιX³©’₯'zŸθyλΑμc³J½Fυλ‘V§ύš©ΚqϋWx–£Λ;M5Ϋίών9ρIΟ“}x &=9ς–―~Φ†7uz‚eΗͺ°ή‹^°h©ύέ~¨\!π–όΪ¨}φ‰½7Σ>Φ€lW->!Ιτϋϋ’W°οͺι‰₯·ώψ^ΉΡœ|ΪΪΪ♀ΎΦV†υŒπz–1RώΗ„CŸςqΡ yŸœ’σiΕ–Ή {Νqψα§/ŠόκΗίκ4ž 0΄Ÿ(9Ω\˜ΰgί»IόΉŒώŒΡπ½^$Μ»ΕΖΖ{φϋρ`ώΜςήΨPψ ρξ@ΐ?ΝρζkΏωέ ]€ΕΏ΅Ω©3}O¬ΡΏώΣ―δΛ7ηΨω‘νkGPŒ‘ΩςΛϋOn1Φ€©w«5GΣ[WMηΏσ7Š^ό7Ψm֜ςλΏ¬»οψ¦h₯χŽM/Ππα֍G{†[|ŠάtϋΟm¨Χ3²‹zΖ½ώΣ~—όφλι4EΡήόNγœzΡ΄šσC†σ‰έN_ΈG8,0Ε\RŸ;eΨψouπœŽΤ±©Ά₯«qy ~°ΟΙΪ΄αŸ.©>Γ €τO ;;[Ξ;οΌ |ο½%"uΒFήγ‡Σㆆ[Φ ΖjάξΓΥ‡S?ΨΦ‘% ηθΔΕΕΙΘ‘#eοή½ΆPœ©WΩΓFšpv{³DGGIJJŠ Σφ01O!€@/Q¦X‘w*ηyoV€Oςυ Ι₯λ/ΏΊ)ΉζΖοJ\|b·}Œ‰“σ/ώ‚όᑉ^ysΒδi2"=³Λιψν]ΆΒc—O0R±Π/πϋΧ»vέξΚ_kφΗχ<(~r•Ήιψ^ΧΊμΜσδοΟ½-_Έζ$Ί›­χB’£cDPύγΕub½axŒ€Ÿ‹Μ’{z΅­2δΟ,ΣgΝ•ŸέχΈ\{ΣχD«3«M9[ώΆβuYpΒ²©'Uήυ»Glˆ^6΅lοqzžD\ «J±/…9@ΠΘ5•Eυwg½²Ή?―ͺ@VvŽάώΣϋδ·ζB½2x0Ϋ©¦RθΒ}ΦJΆO=ϊ@ΠV£'Iήϋπσ’ΫχΤ²srΝ…ͺ_άδ€œμΩΣΊ{zNƒj7ο—ςΣί=j*ϊNξiRϋχΝE—]-½΄NΞωδe=N{μΒΕςΧ―™ λ‰=N§'ι•ς6Η94«Mιhx؟vα₯_²~Π@`θ ,=ύyψΩ·D–L›yŒDEEwΪθ£η.΄υ€z=¦ΫρB`©iιζιοˆΑϊ™Υq;‡ϋgHΗώ0Œ θwDzя‰SfΘ‰KΟ”λo½Syώω‰©Z—;aΚG†Α£Ήσ;>ψΰ)o½Ί2 zO@πΠγ"χ?ρ²œς±σz y&₯€Κη―ώ¦άύ‡ΨΰΠ@)M7UhτΨΏΝ…ΙμχηorΚωφΏ³ίυŽΚλΧ¦ιōϊϋΘmζaξ­ι4ίϊΡoδ.saf=·…† :zn£»pš^@³ΠTο­©cS½m—>?ΠΗΰλœ,­2Έώ7<]3n’θωU4@ΰ θη τbccƒzΣeFGwώ>)x[>°Kjkk“-[ΆΘΆmΫl΅X]›γチ¨ν KŠΊodddτz,ζ£m?dχS­0;œχ«ϊΓ#ŽŒ€ζ3τ末€ŸϊXΗΡΒG€J±!πZ_ω΅[DoƒΩ’Μ‡χ%W\';οΣςφλ–ό]y’Ώ;OτJ0£²ΗΘs‚Α©Ώΐ犝ϊΓεΩΧ·ω½™O½²Ιοi;N¨_`|σ;wΛe_ΊA>ά²Α^ ΅d_‘½/+-’„Δ$sΒο˜ήrdμΈI2{ήρΤ/GψκΝ6μΊsϋVٝχμ2·έ;?ƒ!©i’™5Ϊ„`—ˆ8ˆ««vμ;Γ 5ΧμΪ&ι μίϋΩύΆςλκ—Ÿ7«ΧΨκOZ=V―t?rTŽθ4Zρ$ƒ¦n£~ωϊ‹?=!e%ϋδMsP©ΉΪrεώrϋ3OO¦gn³ΜΙ£ϊΛ•Σ΄Β΄ήh‘!π“{ήηŽ|κ²/‹ήϊƎŸ$«·ξοΛ¬Μƒ@H |ρΪ›Eo4@ΞϊwϋεWί(gžs‘¬}k•ξ‘}{χ˜ϋ|©4'€θίͺ9ΉΜE¬Ζ›Ώ‰'Κ3Ξιυ@½€T_ωωŸθΌ‘^c4ˆΫΧe;‹Ρ“,υBWZΝτ΅όΣΨ+ϋΛKd€ όN0'UL˜<]N0a½hΥPm‹N>]τΆ}λϋςφκΨΏ9jͺ*Νɜ£DOΡΧjς΄Y6άαoτo”{ώϊŒ ™τQΩΈξ-sle£΄΅·Ω‹žΝλ’2­Tώά—Ώ!ΛM°ΆΐΪgήozυE=>3yΪQώn*Σ!€ΐ $§Ž0•ΌhΧ¨'δοΪ.5Υ•2ΒΟΤ}W+-y·&SQΠ»ιqΟZ ΗX:.c0~fu\§σ3€ΏŸiί5•yυFCΎυΓ{DoƒΩVΎ›?˜«σYΧ+‹}†ƒ1 Ώcφχχν`lΛ@x”Τ4ωήϐŠ²by~Ε£²σΓ-φ8€VŠiŽeΙ•iζΚ™η|* στo…ώ|–,=γ\9ιΤ³εΥ/Ϋc…ω;m@©Έ¨@22GΙΈ‰SeΌΉ›4UŽž³Pτo–@›ž“’簜vΦryoνjysΥK²―`·9VfŽ{ΔJκˆ s1²Irœ©H7ϋΨγν±‹@ΧΑτ €ώ θΟρώ|nψ·–>ΎψΜ?<¬όη~}―|€ŽMy6΄›ƒο뱩:'Λ»KοΌρŠT™σ«œvι_sr €@'Ν$477‹~HΣω΄iEXΏNςσσε™gž‘ΒΒB›uHHHάά\B±ΐ2m·Z©2++ΛΜλvΒ>‘οQ}jΦ συ6Ο#€@Χή•aυΨ wv£λ9j„bCνδώθΆΟΊΰ’A^««ΣJ7zθ–˜”,ǘP­ήh 0πz2ώ§/ΏΖή~mέ―ACΈηϊσέOΐ3 € €a½ KoΥDπ&}υΗΜ_$zΞmκΜΩ’·`6'pΘ2#Ν—7σŽ_boώΜ§ΗGfΞgoώLΟ4 04τ‹’ISgφΈ1ω;·ϋ<Ÿ:"ψ•b½W0?³ΌΧη<…Ο§/ά#€ €„·€^μζsWέ0€τ€΄Ύόh'τxΖΒ–Ω[ σ2= €@h̚³@f}¬|°ι=Ϋ‘gΠ\ΨρFΏ/ˆΠ—Ο«@§χEz0ŽΑδ9YΟ<ώWO·³F‘Ÿw±g˜ €tΠΰΰζΝ›eχξέ½y5L[__/eeeΆJμφνΫmHVCˆγƍ“£Ž:ŠπTGt†ϋ$ ί΅fffΪί5υ½Χ[•J}^«Šν73!ΰПηzsφ9}L(ΦΓ6Ε†ΝKMG@@@@@ΒQ ­΅Υ^ωΪ點€η|9δŒλξΎ‘ΎNv|ΈΩσ΄bG€gz†y€ € € € €ΐPψΒ57Ι|ε3v³΄‚ϊκW^₯§Ÿ3Τ63lΆ§x_¬~ωO/»ςzΡc”4@ΰ 466JEEEΠ«OjΥΥΨΨXβσχϋ₯ώφ¬½½]^}΅¬\ω’ίΆpΦ©ΑDέζVσύXCCƒ8p@tyΪΣΣε„NιΣ§;“s@Ώ’““mΠ΅ΆΆΆΧeιϋ3..ΞξO½NΜ Π­€~9!Xη³I‡ΗέΞΘ!%@(6€^N:ƒ € € € ΠWίύ쎾Ξκχ| O\ζwεUΏΚ„ €@/7\y‘¬_ϋ†gͺλnώΎ|ϊςk<Γ==XχφkφΔgš“N=+θ'S8Λζ@@@@`œpς2wαbYΞj»Έ§y€Pl0`ϋΈŒΗΌΟsΡΎ¬μ9{ω₯}\³!€τ$PTT$Ο>ϋlΐ!ž–©ΟiΥU­zΞ9η}Ωέ­[ƒƒεεe²cΗ‰κΓ…:†’t8--MN<ρDYΆl™ Ηv·nΖ#ˆ€Ύ·44ž””dCιΘξψώσ^ž>§‘Xž†ύπ«ϋVD„» dξa'@(vΨ½dl0 € € € €ΐ@<|―b±>ˌ'λ#Β †ΐΒΕ§ψ„bύΛοδΒKΎΤk5†ζ¦FωΝOΏγ³‰'SQΓΗƒ@@@@‘)pγ·ο’Λ/X"ν&Θ£~ΫΆy½LŸ5whnloՁκ*yφ‰ΏyzxέΝ?¨θhΟ0@‚'PXX(?ώΈΔΔΔo‘fIΝΝΝrάqΗΙYg5h‘X퀆"##ϋ΄N GιΌj‘šš*cƌ‘Ω³gΫPμ΄iΣz -………€†bυ}¦οY ΕvΧ4μ­!ο””€νn:Ζ#€€C±nw„™Qo΄p .―4ύD@@@θQ`Α’₯=>Œ'sƌΖbX ΐρ‹O•?όβϋžyΚK‹δΏϊ\υ΅[»=M§Ήνϊ/ΘΎ‚έžω&Lž&σ_βζ € € € €CU`ό€irΙ―“οϋ…έΔ~{—άυ»G†κζ†μv=ρπ}ΨPoϋ·πΔe²τŒsCΆ―t 8MMM²wοή€ΓvήU-WΪ;(Η8jll”ͺͺ*OΥοΑθ§Ϋν–… φ+«αΓxsΡꌌ ™8q’Lš4ΙuΩ4‚% ϋI΄Ήΰ‡]υ½₯••{jΦΦ*±Ό{Rβ9zΠΟ'½iΣΐωαΟ,·Ήο}^¦B±‘σZ@@@@~όβώ'ϋ17³"€CW`ςτY²ψ”Λκ—_πlδ#όF^]ωœœuώg$'w‚de±'§U”˚7^6ΣΎ(-ΝMžι££cδ»w©Χ겞x€ € € € €ΐψόWΎ)/ΏΈΒ^ψνΝW_’Zμ Ώ υ΅ςψCχΩ΅FEEΛ7n»k·€Υ!€α% Υ*΅"ͺήϋΫ4HΤΪΪjƒ|ζkii±5\4bΔ7nœ­Ά:sζLOψΘίeχg: .[ΆLŽ?ώψ>WuuB½ŽU½§!0ŠMNNφλ½κLλ„ωb{X&α" ?睦½Ρά‡Έ‘Ψι € € € € €@x θ@wόμOςΝ«.’υkίπ`ξ‘?έσcΟpw’cbε[?ΌG&MΩέ$ŒG@@@@`Θ Δ˜γZ7έώ3ΉώŠεvΫ¨;Έ/Ργή'΅5Υv₯—^ρ53nβΰnkCΒL`Β„ rε•WΪκͺώv]C±†Υ*³•••RPP »vν’ύϋχΫ©.σ€“N’±cΗ΄\ΧίΣtZMSo4†Ί€5Ϋ[υWέίt¦κ}fϋ .χΖ6 ―Κΰn‘ΨΑυfm € € € € € Ί€žψ“{–ίώτvyαιψTνic4ϋέ»(&Oοi2žC@@@@`H Μ_t²|όόΟΘ +ͺΕήK€Ubύλ½v…Ω9ΉrΩUΧήΚY ¦Z%φ’‹. ΈχN0φΐ²oί>Y»v­¬\ΉRŠŠŠ$//O-Z$ΉΉΉƒZ)6ΰN0GP 22Έ5§ϋ“wH―γfi₯Ψ„„φ§Ž0 # €ξgή!؎Γ.ŽΙ‡©‘ΨaϊΒ±Ω € € € €@8 δNœ"}ξ+žOž~”η1@zˆOH’›ΎϋsΉκϊΫdΕ£–5«#%E…RQV"΄3ΗΖΕKΖΘQ²π„erκYΛeφ±ΗυψΕmΟkZΟς2΄^Ά@@@,λnώ,8a©]ΫΝi³ƒα^k‚U7άϊ»ͺΙӎ’˜ΨΈΑX-λ@ΒZ@+PjΨ―-55U²³³%''Η.β±Η“?όPώωΟŠVŒ>}ϊψΞHC‡ ΨΨ(ΝΝΝφ;.ύžKQj ΕΨΨX‰·ΥnϋκΑ|" ο9­Ϋ[sφSο0_oσπ<t/ΰ„Нϋξ§δ™Pΰ―ϋP|Uι € € € ’ӏš#z£!€τ] %5M.ΏϊF{Σ₯΄΅΅IΥώ2IHL–ψ„ΔΎ/xˆΟΙgȁΨ<@@@@`€’SRεŒO|r€–Ξb»ΘΚΞΑΌ+Ζ!€C\@C}cǎ•3Ξ8C6nά(kΦ¬‘wί}W^{ν5™4i²DGχϊ¨.Ά··Kee₯TTTΘξέ»m%ΫςςrŽΥοΊ4d¨ΑΔ€€$5j”Œ?^F-ιιι6 ;PΫΕrΠ0žV€Υχ_oMχ1 lΣ@ .ΟBt?$λα›„bΓζ₯¦£ € € € € €tΠ+fgfξόc@@@@@@ΒJ@CEΉΉΉ2oή<Ω΄i“TWWΫ`μςεΛ%33σˆXh6//Oή{ο=Ω±c‡μΫ·Oͺͺͺ䀩LήΪΪjoZΣ &jΥ[ Ζj?f͚eoγƍσ«’ηι +φϊή‹‹‹³αμž:£οӘ˜˜ž&α9πK@C°~MΘD!,@(6„_\Ί† € € € € € € € € € € €ψ+ /§M›f+_juΦR{ΛΘΘΤJ|”½{χΚͺU«l΅Ϊν۷ېVΥζ]P«Εκ­ΎΎ^φοί/»vν’υλΧΛΪ΅keξάΉς‰O|Bf̘! ώ20~ hΨή*Εj5c Οκ‹i Π?ΓXί`¬χgB–ΞάΓE€Ÿ¦Γε•b;@@@@@@@@@@@@Ππ^||ΌhUK 555Ω0κ²Σ’:$εεεςτΣOΛK/½dΓ±---v: &&&Ϊ€«†u{΅΅΅΅›mm”ΊΊ:»Ν²sηNθmhh­v;gΜ^Γ‹vaόƒ€ŸΊψ[V«Ικ΄4Ύ‘Ψΰ›υ%Šκ―Ϋ‡ € € € € € € € € € € € ‚€V\­§"«R«ΆjPu°BGZρυ­·ή’ηž{NŠŠŠμΊ΅η¨Q£$77W²³³%33Σ†w5«­ΉΉΩlw Σ–Ιξέ»eίΎ}RUU%΅΅΅ςϊλ―Ϋ Z)vΦ¬YƒΦ»aόςtΥ*°=5§RlOΣπψ/ ΥbΝΗ-ŒΕ†ρ‹OΧ@@@@@@@@@@@@G@©οΏΎθ½†`΅2«†I+«αΫνΫ·ΛΚ•+₯  ΐnVrr²œx≲pαq2mΪTIII±ΨΘΘ(SyΣ©Ϋfƒ±Ίέ………²eΛ†ύΰƒδΐςΖoHVV–δδδHZZšΣ]ξθ·€³5«ϋˆΎ»jϊ\LLΜ νG]mγ%έΥ4K _B±αϋΪΣs@@@@@@@@@@@ζΔ ΄’«wxΟ™ΏΞVhΥ*­ZyU[jjͺŒ9rΠΒ|­­­²vνZΜΥυk…ΨΕ‹Λg?ϋY™:uͺ θκψžΪΔ‰eφμΩ2fΜy葇dǎRQQ!―½φšΜ›7Ο.―§ωy@΄ ¬S±Έ»ωt ˜VΈΌ»ν`<‘*ΰύ™ͺ}€_Ύ…b·nέκ3χκJΉθΜy>γ@†£@iρ>ŸΝΎλφδΧwήκ3ŽΪ΅΅5>xϋ7―”θ˜XŸq .P_{ΐg¦Mο­αχ?@Α(+ρύ}υž;o“?έσγΑY9kA@`jkͺ}–ϊΞ›―𷆏 0X u΅>«ΚΫΆ™ŸG>" €ΐ` 456ψ¬ͺ΄΄Τg˜Ό*++½νγ+>yŠΈάξNだл@ΥώrŸ‰žzτωΟ OωŒc@8μ΄Ϊ’’?~|§ρŒ@ ΆΆΦVU 4p§!"}―΅΄΄HMMμή½ΫVWΝΟΟ·γ5:eΚIOΟ4δκκjΙΛΛ=β6Η;&Ož,\pΜš5«Χΰ‘³‘PΤ ο©§ž*rί}χΩP¬SAvΡ’EvΩΞτά#Π Όκ{΅§¦ϋ¦Ύ/έG{Z&Ο!ήz1ˆπχήŠuτα ι/?E…{œAξ@¨¬( ™ΎΠΒU ’¬$\»> ύnnnβχΏfα €€U•ε’7 €‘" !Ž5‡Κ«I?ή­-Νό<ή/![@Θ΄··‡L_θHπΊz”E,SΊ5’7 € 0ΪΪΪ†βf±MC@@Γ¬ώσŸ%::Ϊο­Ρ<ˆw(VΓ¨ΕΕΕRUUeC²τӊ«Z₯5&&ΖοεφwΒ²²2ΩΏΏθ1 εžxβ‰2sζLΏ±ήλΧ*·K–,‘W_}Υ†lΝχ’Œmhh€€$οIyŒ@Ÿt_Ρ›^{ͺVΩ[pΆΟΐŒ„™@Η0lOϋ]˜Ρ„Uw Ε†• E@@@@@@@@@@@†Έ@yyΉ¬\Ή†HέT 9αX'x­-³²²δΜ3Ο”yσζ™*˜@ΫηιλκκlhU†"Τ@l¬ΩΩΩ2i$Y³f­~«qϋ΅Μ>wŽCRΐ ΕφΦ9'8ΫΫt<½ ώάςžJ/τ@ /B±αυzΣ[@@@@@@@@@@@! ³jυΣ@BA:Υ¦χΞ-66Fβββ%''ΗVˆ=묳D«­fλLHH°U8ϋΊ πMLLτTρloΧΰΤ‘Ύ.ŽωθR@ί·=5έΗz›¦§ωy>p.δπΡ…£@@‘Ψ‰'ϊ₯€€Θƒ>θ3Ž@`8 lάΈQš››=›>eΚ1b„g˜ 0τή{ο=qP¦[;cΖ β„—­€€D κ3‘^Έ‹γς>$ € €ΐΠͺŸΗ :Š0μLžU’tB±‘‘Qζ’[I’••%#GŽμΣ²œνιΟ½{μΨ±¦bmœ ΓY³Fζϟ/cƌιΣbKKK%//ΟVΡΥ}H—­Ι€!,'Œηάw·άޞοn>Ζ#€€―€ξKZέ{Ÿ €RΊο…b‡k'Ωn@@@@@@@@@@@@ 4Θzζ™g†DΧRSSeΑ‚²iΣ&ωΰƒdνΪ΅ςζ›oΚgœ)))ΙΆΪ«?Υ°Tuu΅Όϊκ«²}ϋvς4i’sΜ1I”ΖC¦ρ_ ½½½Η‰» ρυ8O"€@NV/젏‡d΅Šl³ρd πIB/&]A@@@@@@@@@@@T@ƒxzΣАή:6}ΎΎ^šššlυԘ˜˜.§λ8_†u=5έΞΉsηΚ²eΛ€¨¨H΄λΣO?-ΡΡΡ2oή<[Ι6**ͺΛνt–έbη{ϋν·νΌŽΝΞ΢˜9sfO«η9Π@¬σήλnf}Ύ­­­ΧιΊ›Ÿρ π‘€ξOχΉƒυ³Eo?λ>š“G‘$@(6”^Mϊ‚ € € € € € € € € € € €a-ΰ„BΛΛΛ%99Yrssm¨΄+”;wΚζΝ›%33SƎ+£GΆσt’νjώ@Ζ΅ΆΆJqq±477wju–₯!Γ &ΨνΡj±[·n•Η{LφμΩ#Η{¬ Ζ&&&Š†xέn·MηΡpoCCƒ Σ›J³οΛ+―Ό";vμ°ύ™?Ύ Ϋ&$$8«α θ{OίΫ½5Ž†Αp*ΕκΊ Ιg-,e( ŠΚ―Ϋ† € € € € € € € € € € €ψ)°oί>ΩΈq£¬[·NςσσmuΥ‹.ΊHΣΣ;-ACE[Άl‘Ώύο'γƍ“Ω³gΛ’E‹dςδΙβrΉ:ΝӟZ•φΙ'Ÿ΄`##»³h WC†ξΥΠ«ήλvj φέwί•Q£FΩo||ΌhΥX->ΫΦΦ*uuu’Uaχξέk§­¨¨ "jxV³οΌσŽ^ƒ΅4‚% ϋQo‘XFίΗ«[kXα$ ϋ‘ξSΞώ€ΫΫΫΒ‰€Ύξ‹€@@@@@@@@@@@@`Θ h8hχξέςόσΟΛͺU«lUU­šͺΑΦ³Ο>»ΛP¬F5hΊ}ϋv[Ή5//Οj5L{ξΉηΪ€l0ƒ±Ί=«W―–?ό°ΫΚ΅‘κšmmmRRR"₯₯₯vήθθh˜uΆOϋ―ΣhπΠ j VΆ”}λ­·d͚5’’’b«Νv\ΓτUΐyί9½–£ΟipΆ§iΊšq ΠYΰpΆέŒΥύJ?+τ~ ͺœwήΖ B±CαU`@@@@@@@@@@@@> μίΏ_V¬X!/½τ’h΅X κi(΄ΆΆΦ>ξj±‹•ŒŒ innΆ·’’"ωχΏ-ZΥU+¬N:Υ.§«ωϋ2NƒKΊm½—œπ N§ύΠ¦χNψI·―cs–ι„aη5$«•bυyν' ` θ{Zίc=5}ίκ{Πy_χ4-Ο!€@οϊ9β4ηsΑζ><Ε†ΗλL/@@@@@@@@@@@@ 4θωξ»οΚΚ•+m V»˜šš*ΉΉΉ2wξ\IHHθ²Χ3gŽ ŠΪ Z΅²²Rήxγ 1b„|α _‘#GφbνrFFEEΙψργνΨΘΘΑ³heA­*›žžήa«D οΖΣp^oak}566Ϊj–}_s"€€ 8!X'dϋ—ξ‡Ξ0Jα!0ΈΏE„‡)½D@@@@@@@@@@@E@«»Ύψβ‹ž@¬V~]Άl™,X°ΐVzΥpkWM+Αj(v€IRVV&λΧ―·Ub7mΪ$UUUςΪk―ΙμΩ³εΜ3ΟτTkνj9ώŽKLL”sΟ=W8`ͺώΞ¬ι΄RμŒ3‚΅8–ƒ€ЊΛNX§ZqG λιtήΥ-;NΓ0ψ' !X­Π¬ϋ•ξsΞ°s3U¨Š •W’~ € € € € € € € € € € € €@X h(//Oήzλ-JJJ’Ε‹Λe—]&£G ΎvΧ4L€Udυ¦Υ`ǎ+)))²~ΙΟΟ­»zυjY²d‰h ΅Ώ-66VN<ρD`κο²ϊ2Ώφw°+Τφe;™gx h•Ψ†††7Ϊ ΕφVQΆΗ…π$XݟΌ+Γκη †Ξiα%@(6Ό^oz‹ € € € € € € € € € € €!" *5ΐͺ•]£’’lΥΧΣN;M&L˜P].—hEΩN8AΦ­['ΕΕΕRWW'»vν’ΚΚΚ „b5”ΪSH7  fb†€€vν­R¬Ύχ5Δ§ϋ”Sέrl>›€ΐ°Π}¨₯₯ΕξKϊΨ ΕκcZψΈΒ§«τ@@@@@@@@@@@ϊϊz))) έi›:uͺLŸ>½ΟLMM•yσζΩκ±ΊΜΪΪZ)//·α£>/”Q αι>Ψ[•ΚŽ‘Ψε [ Š€‚Υ{mνννv$;(όCf%„b‡ΜKΑ† € € € € € € € € € € € €ώ hυI εiθNAZν5%%Εt˜2::Z²³³mΥY}J«ρις u€b# ϋ_MMM―ϋ‡ξ?œΥ}Iη‘!€@ίt_ςΕκgŸ†b›››{έϋΆFζͺ‘CuΓΨ.@@@@@@@@@@@@žœ@¬N₯; ίΕΖΖφ»›)@`ΔΣJ±Z©²§¦Α= ¬λ>@(Ά')žC wέߚššμΎ€AXέ§t_€…—‘Ψπz½ι- € € € € € € € € € € €„ˆ€Vnjc+Ίjθ3//OΆmΫ&ΉΉΉv\ έΤ€ίϊυλ₯ΎΎήΞͺΫŒŒŒ@ΣετZzύχεƒ>θθhΤεk I·]›>ή΅k—Όωζ›vš.Τ‡‘Ί\­x{άqΗυanfA ³€S­²ΊΊΪΎo{ͺ«Ο9Z žΣ@ οΊο577{φ;έ§=‡θϋ’™s8 ώ­a8m1ۊ € € € € € € € € € € € Z)vΒ„‰’mA{φμ‘—_~Yφξέp@¨Nήzλ-Y»v­ ι²Η΄P¬Ύ\Zy6**ΚηΎγΛ¨Yg>7gyNψΆγ:F /Z­²ͺͺΚ†σz›_ƒΩZ)Φ œχ6=Ο#€@g ΔjΦ;λTŽΥηhα#@₯Ψπy­ι) € € € € € € € € € € €„€V œ8q‚,Z΄Hž{ξ9[-o͚52zτhYΆl™Œ7Nβββl°T§υf©!=½΅΅΅IEE…lΨ°AV¬X!ΕΕΕV(++K.\΄Š­²={Ά€₯₯Ωͺ­Ίξ©S§ωl“nίδΙ“ε„N°Σσ₯@¬VΥ₯!  ΰiΐ΅ΆΆΦξG½uz­f©!Zέ?i Π7 Δ:—uΏΟ­Δ¬γτσŽ„bΓγu¦— € € € € € € € € € € € ‚™™™²tιRY·nμΫ·O*++εΩgŸ΅Υb9ζ;v¬ ’&$$Ψp¬†χ4«Α"­\YZZ*;vμwήyGςςςμsρρρ²`Α™?~ΠΒ©‰‰‰²|ωr$ΤmΠ0Szz†x‡ ΅*μ’%KdΪ΄ižυκtύmΊ άͺ `hoώύvρwy vtޏώΞΗt pX@χ; —;‘X«ϋ“~žι>„bΓ硦§ € € € € € € € € € € € bQQQrμ±ΗΚi§&/Ύψ’”——Ϋ λk―½&›7o–‘#GšπiΊ$%%Ωͺ―ΥP¬‹4«•a5H[SScΗλtsζΜ‘σΟ?_F4-­;sζL[ΥΟ©X«Xη±³"έV­φko” Vs»έΑZΛ s ειΎ¦ϋ’?MίηuuuvίΤ`οEԘ_ ΐjuf­ «ΝωόΠa­ά¬‰ …‡@π~3/z‰ € € € € € € € € € € € ) ’jˆUC―ΌςŠ­«Α!½iυX –jΟ ιtΜkoo·χτ^§INN–… ΪeiΠV·Αl½-O·kλΦ­²~ύzn:ϊθ£eόψρ6λl{0·‡e!ΠW αω„Ό{Z–Ύ΅RlYY™ ŠνI‹ηθZ@?«4\‘tη3A?7œP¬>vΖw½Ζ†Š‘ΨPy%ι € € € € € € € € € € €„₯€†€&Ož,^x‘deeΙͺU«dχξݞπPSS“ οi`ΘiZ₯UƒyZ‘U«Έfggۊ³gœq†Μž=ΫV•u¦¬{ ^½Zž}φYIMM΅ΫqΩe—ΙΤ©S : Φ‹ΐzzΠύHχ©Rpυ+„α–ζζf[•Y+Zκ>GCΐτbΊ47·˜Ο―‰r‰΄47RΧ°,-|Ε†ΟkMO@@@@@@@@@@@@ D4˜7qβDΙΘΘ°ΩΌΌ<ΙΟΟ—’’Δkll΄•a5Π§X­Ϊš””dC΄99cLπtŠΜœ9Sƌc«Ζ & ξάΉS***€ͺͺΚn‡wEΐ#±M¬ŽΜ«6οΡJS9B"Μώ$‡Ϊ;NφΡ° ΔFD%Θ‘Φz©3Υ›υ}ž––φΡσΛΆ™Ο²‘₯z[C(6τ^Sz„ € € € € € € € € € € €a* !׊Š Ρͺ«ΙΙΙ’U`£’’:…4`΄cΗΩΆm›dffJNNŽdeeΩ0ν‘’Σͺ°©©©6°«ΚΚΚleMέ>CAΐμ6ΤΤ$ω&Ϋ*1ζύ,Xu&V7Ϊε6Υd“Ν]”45ΧΩχ΅VnΦΰ: όΠΟ„κκ*[υάqP££$):Bb##€Ύ½MjkkMθΌή^LΑΏ%2Υp ;œ_=Ά@@@@@@@@@@@ψ―@ii©|πΑ²nέ:SΙr·Μ™3G–/_.iiiŒ4»iΣ&yδ‘Gl8oΒ„ vϊyσζΩκ²ZIv°›x.\(;wξ”ΒΒB{ϊλ―Ϋpozzϊ`oλC “€:///—v‹r'˜j±‰fΊˆNΣ:#τ™3+2Iښκlh]«ΕN™2Ε™„{θE@«λ…κM <!)1ι’δΨijˆŠΥ‹Ap…^ CδiB±!ςB @@@@@@@@@@@_ ‘\ΉR^yεΙΛΛ³Uσ4ΐwζ™gvŠΥ€‘σΆnέ*6L»aΓΩ³gœuΦY2}ϊt[±u0Eγγγεδ“O6Υ«eŊRYY)O=υ”θΆΞš5K4«ΥoγββMεΫHq»έmž}΅―4ϊ*Πάά,ωωωΆ"eDD”ΈcFHDdŒΔš"²έ΄Š—©*ΡT&Uζ}½kΧ.™`έ·4hϋK[[«DΕΗ‰+:U6ο7Λκ9αŠ”ˆ¨dqEΖJKs₯—ΪΚ–#GŽ x;˜pΠ «ξ{ϋM%Ψ6sA‡S6)&B’##l86*ςΉ˜Cƒh₯Ψ¦¦&SM<.܈¿„bΓξ%§Γ € € € € € € € € € € € *u±1b„œtI²pαBΡΚ¨ιιι]vUƒ₯Η{¬Lœ8ΡNί{ο=yωε—eΫΆm6XτŸόΗ„j’SN9Ε„S]].#Ψ#uϋ5œϋΖoHdd€h5NmFΤ “V±νKΣεj(V+κθ«€΄5«Υ(E\β²!Χx9ΨRΩλ"#tŽθ$SY6MΪ›«M½LŠŠŠlεγ@Βέ½ˆ A­[\\,U&λŽ8()Qn‰5XSX\£"ΜΝ%εΝ¦’lΕ~ϋω₯UΕi‘-@(6΄__z‡ € € € € € € € € € € €!* aΡΌΌ<[YUCC rΒ '؊¨ jyέξ4$šœœloZUvΒ„ ’ΥaοΏ~Ω»w―μΩ³G^ύ5YΌψΔA«Ί§αUν“φE·Oƒ»ΡΡΡύ~υt™.—Ϋ.―ί ca+ !ν;wJ]]DΈ£%2~€ΈΜ½?MλΘΊάρ—)­΅{l°Vχ³£Ž:Šχ₯?€LΦϊ™ ŸI5ΥΥeͺœˆu‹)k?ίβ"]’kB±‡€’ΌάΧ Ε†ώΫ…PlθΏΖτ@@@@@@@@@@@BP@Cz»wο6U'Λm°nόψρrϊι§Λ΄iΣκ­Ϋ„Œ222dΙ’%²~ύz»< ώνΨ±C*++EC³ƒΡt;ζΝ+‡+»šΔS›Λ!S¦L βYT8 θ{²ΆΆV lΥβ¨ΨTSω5Γ€ςLeMΌφΪL΅βΘX‰ˆL5ΪXSΉ^ ₯Ϊ„ό233{› g­!ςϊ†zIvIjΌΛTŒ5ebΝΎk‘i&$!-¦x₯”””Ψ‹+θ…h‘+@(6t_[z† € € € € € € € € € € €!,P_ί`@N§OŸ.3fΜpΎOOO—ωση› ±―Ϋ`MM”••ΙθΡ£»­8πJz˜AC±‹/–9sζτ0UΰOivΚ€m%άΐηfDΪΫΫm vώύ&‡!ξθβŽJ²ο+?S±vZWL’©›!- ΨΩ@zw±G άtί+--•ββb9d§˜l² ΖΊ5nB±Qζ>5.BβMJ²Ύ±Iφ™π¬~v1"άιBΊ„bCϊε₯s € € € € € € € € € € € ͺ--Ν6Όͺρ΄’₯†ZΣϊάέθθhSvŒθ½.³΅΅U΄bμ`6νƒή‚έΤG›ή@ Άnθ/―©©IςςςH―™'@IDATδΐjq›J―‘ρ&(£YkšΎύL:ΫεŽ—ΘΨ,ŠΝ·v­;uκT‰‹‹σo9L…@˜ ΄΄΄H~~Ύ­`ε–τx·$EGΨ@μw+I2ΙΨԘσyΥ.…&«ΪΤTS•ωπΒL,<ΊKΰπxι% € € € € € € € € € € €„˜€~τζ> τ?τ3©¨hŸη‚i±‘2Β„_MΑX[)V£ΥbcΜpšŸΥ&κdϋφνφσcτθΡ~‰Ι†›•b‡Ϋ+Ζφ"€ € € € € € € € € € € €F 11A4τγrΉμ-//Oτζ]ε5¨Θ† €±±Ρ.#))I222YDΏ¦Υπα{ο½'O>ω€¬X±BΦ―_o«iφg‘;vμ”—^zIV­Z%οΏΎΤΧΧχgqΜ†Ί?l7ϋ•V‹uΉγ$*.G"\±}–pE&Hd\ΆYF€”••‚Ιϋ²ΟžΜΊu¦:μnHί·oŸΉP‚[Ζ$šΰkŒ-λΣi—©˜žž`ͺΘΖΉδΰ‘ƒ²cΗ€-τ\ΰΑgbBB€J±!ρ2 @@@Pψφ·Ώ-ΫΆmλ²[<π€θέ5ύRΰΡGνςιλ»N–,Yεs:R―vλ­·vωό₯KεΪk―νς9F"€ € € € € € € € Ύ@LLŒLœ8IFŽi+ΔjxHßӧO·aΩς·iπoνΪ΅ςΞ;οΨPlll¬δζζJzzΊΏ‹πk:'°ΫΥΆιs΅΅΅’o΅B­’4(hsΦΡΦΦ&555άάl«zκ}KKK ‹cϊ0ΠχRqq±δηη›ύ’I’β³Ε—iRy}―SθrGI€©λŠJ4Λ¬‘;wšj˜E2eΚ”0–¦λψ h•ΨRSYΟilhl S vT’[β΅J¬ο€ζη»Hj”HvR€Τ4KΉω ΩΎύC™B"’“Ε“.­-΅²wo‘ έNš4Ιξ«ΎΣ2„@x θη™ώό/(( @ŽNŒ΄X­ŠήυΕ’έΆRlFΌKφΦ΅Ϊ0ν1Η#©©©I„2ΤήEΌ’‘φŠ@@@@@@@@@@@Q£FΙ)§œ"6l°Α½²²2yꩧLΠn―Μ™3ΗVŒMKK³αΦ¨¨(ŽΥ ¬VMΥΠ©G5τ·fΝΡ‹±k%U ΒΝ;W.\hH‡ͺΑΥ`«VΕ|ύυΧνϊ;.Sƒ¬ZνVΓ«:­n—φEƒ½4W—₯ά-[ΆΨΗΊΜδδδ †|Ω&¦žΦJ•Ί―DΈγ$2~€Έά½;χόΈ"γmΕΩΦϊb―³ϋΕρΗ/‰‰‰'g a& ŸQZ=ωƒmΫ€¦ΊZ’]2&9JβM5ΨξšΛ\π!-.B²έRΠf+<ψα‡’““C΅ΨξΠ†ρxB±ΓψΕcΣ@@@@@@@@@@@@ Ό’££mEΧ%K–ΘΏύo[υU±555²qγFΙΚΚ’ΜΜLΆΣi΅i਱±QͺMΨH+`jΆͺͺJZ[[mυ¨£Ž’ .ΈΐVš59£ 5­©‘? *εεεy*Έ:+Π0«n—ŠΥ~”——Ϋagξu9zΣ€oSS“ §€€Θ”)S$!!ΑŸE0 6L½gΟXՐud|ΊΈ’ΣΔ”Œ5:ϋ%€ϋ‚;6K"cS₯Ή‘ΨSV«ΕBΫ―Ε33ΓZ@nλEςL ]ΆKV²[F%Έ%μ7έΦg6ŸU±¦ŠμX3νξšv©2•f΅ϊ„ dĈT ΦοˆΞO(Ά³ c@@@@@@@@@@@@a#0rδHYΎ|Ή »jV ’jΰUƒ±ωωωioΪ! ά9•TΫΪΪlpTC²D·b5«Ub΅²l0›[C ,m¦ ncΗ n‡6έF­f«7}hΣεκΝνvΫ~ι:-ZδqtyL~uuuφ}ZZ\,&Α*Qρ£Εe*ΉF˜χcΰoΙ€ŠMwL¦΄5VΨ`Ί†΅ͺ₯Vj¦!@Χ 5hυσκκ*IŠŽq)‘’lξ{knw„dΖ»MUY·¨h“ΒΒB{qˆ©S§Š^:„bCη΅€' € € € € € € € € € € € †ώœ9s¦\|ρΕ2jΤ(yυΥWmH«jπ΅ΎΎή'Xͺ!S'0ͺΑW Γj5ΩcŽ9FΞ<σLZI­{ϊι§Ϋκ›Z-Φ»ιvi ·ΆΆΦnŸ†˜ϊRαOϋ¦αZ­Œ«ΛΠ VΥ` 4˜WlΒ°ήnik—ΨψTqΕd˜"±&ŠΥ‡vηu—;Jάq£ΔΥΈΟ„Ώ+εέwί΅ϋΰΨ±c;…Ε;ΟΟBS@ώk…Χέ¦Js„ΩG'FΚθ€H1yWΏκ3ΗΊE&¦Έ₯΄]Jšμ>ΌέTœ=φΨc©BoB±!τb@@@@@@@@@@@O­ˆ:mΪ4IKK“ργΗ‹VΜΟί#₯₯eΆ"«d5θ§ΑS Œj6!!A΄ΚμθΡ£νΌsηΞ•Ι“'KLLΜ€"jθοSŸΊHφνΫk·ΙY™V¬}φΩgeύϊυ6Ό4eΚYΊt©$%%ω„zι»ΊΧ@¬ψΥPljjͺθϊ4,μΚ·]­Ÿq‘! ϋˎ;€  @άf_‰Šiξ΅€V«μw™ΨΓH‡\βŠM—HS-Ά₯ΉΚVuήΉs§dee ψ>―½5­ ΎΗ„aίyηinj΄X—δš*±)1¦ΒΉŸuΉW‹ΝMŽ”κζ{‘ œηζζΪΟ;ύ|  B±Γ5€ € € €!(πΖoτΉW·έv›θ­/νψγχϋ „Ύ,Ÿy@@@@@@@@N@ƒ±r=ωδ“EZιR+―VWWKCCƒ΄ΆΆΪsC4ͺΑW­Ϊšžžn£6$«Οιt:†huyΑιΊηΜ™#³fe14¨«MC±ˆΥ[dd€Œ7NN;ν4[-Φ™ΖNΨΛ?Ξφz‡c5LCΐ}ίΙΦ­[₯^ά1©βŽΟWT°«'›€zdΌDšj±νΕΤTcίϋnΟΞΞϊ~ηOί™#% ?uΏ{λ­·$ί„ΡγLiΨ±Ι.•θΆUbύ ΕκφǚΔδ8S-Ά’Ρ-{λše«ΉHDΚ5ž‹,8ŸGͺ―¬·„bϋoΘ@@@@@@@@@@@@!! αO­«·¬¬‘&hzΠ†M5pΤΦΦfC±8Υ­N«χzӐPSS“TTTHII‰Όχή{68ϋιO:θVu]±±ZΦ·"­nγxSεvζΜ™’U^΅²_JJŠ ο \6",t?Π*±Ϋ·o·…a£β²Δnk +h^ο\ΊΊβ2Ε3RZ›kΜzσdΧ]’!uέh„ƒ€^τ N6mΪdƒα­Ν’c*ΔNL’δθΐ+»ΊΜgLFΌK&¦DΙS-Ά’ΌB΄P‰VEŸ5k–ωό vΐ=^₯‘ΥGB±Cλυ`k@@@@@@@@@@@@ Έ\|uχj­­­΅aΨ={φΘϋοΏoΓ€ΫΆm“ιΣ§Λ'?ωΙ^ηΚƚ…hXvι₯’•25,¨&ΒKΑe9ώh8―¬¬L>0•%¨·VrMΘ1yΨx3{p±‡·ΗΣ#“$*>KΪκχΨ ΝΊκΎ§ΑXα ΠbƒθkL5Χͺκ*IŠrΙ$ˆΝJ0Ÿagb-YŒ™1'Ε%ΥΝn©)k•}ϋφΙ›oΎi+O˜0Α~Ζ„ƒm¨φ‘Pl¨Ύ²τ @@@@@@@@@@@θF ΅΅Υa5(€U1·lΩ";wξ”ΒΒBΜkhh±cΗv3χΐŒΦ μŒ3μ­/khll Wi%άψψx²νΛr˜'|τύ£bσςςδΰA‘θ„‘βŽ%.SMy`Ϊ!‰ΠͺΝ±£Δ›.­υEvύPOMM΅οεY/KE`hh…p}Ώ―^½Ϊήǘ]mbͺ[Ζ™@k΄y¬™ΨΎΔΡΝ5$ΕT™`Β΅e νRp Ε^τAχ«ΔΔD9r$ŸCγ-Π§­ Ϋ'6fB@@@@@@@@@@@†—€VΑ¬――·σφξέ+[·n•M›6Ω ¬VΗΤP©6­Ψκ܎tšdbss³έ ­«Ϋεέτωββb)((°>΅111’™™)ΩΩΩ2fΜ;μ=θJ@χššΩΌy³ †W‰h•Ψψq™JΫL΅Ψθ$³±ΦT)ϋχο·ϋεΔ‰%--m`WΝ8‚ϊ󻀀Δbυ³HΪ[eL²[¦§GI‚©«­/Xg>·ωΌΘ4EžuyυζBU΅δέwί΅U˜/^,ΙΙΙvό3όΕΏΧŒ-F@@@@@@@@@@@όΠ ŸVΏ¬¬¬”’’"Ω΅k— άι½kO«ΖjΨT+΅κ-::Z²²²DCy:|$Z[[› ΉjAέφœœ™3gŽΟ¦θ4ωωωςΟ>/λΧΏg+άj_έ¦ςfJJŠL:U–.]*G}΄­Ίι33tΠχ“VMΦ}£­νD%e˜*±Ωακk­Κ+θvπp΅ΨΘψ\qΧν•–ζ[-vηΞ]φ}{€φΑn7—'‚€~6UTTΘΫoΏ-6lΖϊZ`uΙτŒ(Ι0χΠ§5j 6ΚμΏc’άΠzP6–2ŸyςΞ;οΨ@μ±Ηk«ŠχiαΜtDΕQ~VŽ € € € € € € € € € € €W@ΓF2V›€΄΄T4XΊmΫ6[VkI+ΖΆ··{Β°$Υͺy£GΆΤ)S¦ˆ†tό`7 Άj[ KiUΏΊΊ:YΆl™Μž=Ϋ³=ΪG υ>χάsςΜ3ΟHyω³wΰqη―κN½[]r/r7ΖŒ)!ό!@ @HBIH%!Τ τΠC„‘„ΠMwΑ6ξ–lΛV΅zο]ςΎ1wάI'λdK§;ιη9έξμμμΜoχvW»σ~o•ξ£­"$ΜΝΝΥ.²β{δ‘G"44Τ±˜ί$Џ@cc£WoF}έ~—X“-&[dΏr#“‘άbm°„&‘­£Zn·oί¦„έS΅ΐ{dΆΙZI`tΘω[\™εόΎfΝ¨©B¨Ι€©Q€*«Y‡+‰Έ6ΤLQu7v{j{΄π}εͺU Aff¦1\Ϋc=Ύ!@Q¬o8s+$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$0’z{{΅0΄N‰ϊΔAUD‘»wοΖ]»P^^TΫΫΫuΔΦl6k—IκΨι•ψMŽ|2Ν0…€ΐR‚ΦΆ*μά±C·iρβΕ#Ώqn|D@ΞγMMMΨ²e >ϊθc””μEˆi¦Δ˜1)Κ ›yψ±Ž. FD3c,θV#ςκ;°KŠ0«k†8£O™2AAͺSΐ (6`vJ$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@ύ ˆλžΈ«–••)lΔaR~ωωωZΫΪΪͺ₯".•$" „„Lž<Σ§OΧί2-BΨ°°°Q‰γλΌ‡O>ωD9έvhA¬ˆ•dZΔTŽT\\ŒΥ«Wkρ―τKD±3gΞΔ¬Y™jh1πφνΫ!βΩ 6h\ιk||Ό£ ~“€& Ώq#–γ©VDΨ¦X‚“•sk΄ZξA¬cW˜ƒ’Π’ͺŽχ:”•—aύϊυ˜1c†ώ]:Κπ›•€γ·ΆiΣ&|ψα‡(RΑl_ΉΈΞŒ±"Βj€\₯†ύW§*•z‚θ΅*α{ςš΅0Wκ ]™εΪΘ(Š ŒύΔV’                       €ŠŠ˜OΔ°;•λέε,™­Ε°"•ε’ͺ"HΧΤiΣ¦α”SNQΩJ;!!!Z(λVω(̈θ5++K»»ŠΧb± 11 .Δ₯K.―ύΚG¦E›‘‘³Ξ: G}΄nωΦ­[ρϊλ―a͚/ umάΈQ³‘(vv¬ŸoRε—ΨnuΜyό;Κς›‰€œ«Ε!vσfqˆύH.RΚFq‡kA”2jU§ςαΔΊ@RϊWΔ0#Ξ‚φž}(iιΠΒXΉΦΘΆ'Ož›Ž±.Θόv’’XΏέ5l                        ΈθξξΦξ§5ΚΡ²  ΐ)Aluu5ΪΫΫ!eD€d³Ω`·Ϋͺσλκκτ2qT=ν΄ΣtΎˆό%uuuiQlQQ‘ςFEEiρξI'„””-’•ΆŠΘuϞ=ΊΏβς'iωςεXΆl™ΡΚ|DD„£  RŸ‡wοή­Λ9Φ‘rL㛀όNĝψ‹/ΎΠsƒΡ±Ι0XΓG Œr=Š€Εžˆήφz}Œ―[·V»XΚ‘€όΞ›°iΣF|φΩgΘΛΝƒΕ°“"͘-‚X›Έ}FT+άā֬Ά“jDwœΈΒv‘΄Ή[TƒΊ–tuu+Ηρ£κ–.νdœE±ƒ3b                        5½½ϋ΄Ά₯₯%%%Zάιp„Α§Έο‰+¬8­šΝf„‡‡#,, iii˜:u*DL·vνZύ‘Nω VΪU_ί€ΒΒBέιΗόωσqςΙ'cΚ”)Zΰ*e$kQ¬Γ WDΏ‡~8βββφPƒƒƒ1oή<νŠ+εEH+ί"–eL$ Blω=υτ` ‰ƒIΉΔ" ο“QΉΕš‚S`j«DWs‰Ύ‹θύ°Γσ Ggίᙀ\—$ Γ–-ϋb‹Š aήΧ…΄ζ)Η֘`,ϋcψ€›βk3‘n€Imq_o'*›±yΣ&t¨λƒˆsεš#%Δ…œΙ? PλŸϋ…­"                      ηΔρ΅₯₯E‰Eλ΅tηΝءk—ŽŠσ©>3fΜΠ’ž‰'"##C—WYGΧ>L55Υ¨­­UβΔ-ώ[Ίt)RSSέ±ξΌΌ<ν’+ύΎKEάΧυ6::Z­Ÿ¦ΛHYf577Sλ;”Ϊ$.±7nD:ξŒ;,!ι0‰K¬:FG+ΏM#LA10‡¦’»½F»ΕnήΌYΔ1™B½Q:XΈΩ!A¬όΖ6lΨ€Ο>…ω°)‡ΨŒHf*‡Ψψ”>ΥηςsωyΫΝ€*an/¬ΘξTŽ±MΨΆm› .±OQ˜5k–Agρ!οvŸ¬@Q¬O0s#$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$04βl*b’¬¬,ν+σ"μ!¬GΕMU\`Eό9yςddffκοττt$$$h§;“Ι„€‰γ­}%I»₯βjλšDd•——ŠŠ -bWq”}“Τβμ»CDά·ηΗ'‡Kl^^.Ί{za ‡Y9΄Œ"·Mαψ>ΝVΥ–D˜νeθlΚǎΩΪυXDο}γsο±ΧώN@ΞΝ"ˆ]Ήr%V­^­\Ξ‹¬~ZS”uf¬U9Δ*ρχ(bάDk2hΗZ“Αͺb»PΤ†­ΫΆͺ`Νڝ}Μ9ˆuΒ !@Q¬μ Ά„H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€œΦ]‹—_~{χξΥΞu"2—H‡#μ„ ΄#μ΄iΣ °β")βΠΎΞv²^ $qΖ•τQΪl³Ωœ‚VGϋΕ97??ΟΙΓj΅bφμΩΪΡΟQΖρ-uˆˆΦ‘„ ]64ψ-Ώ+q‰mllRξ°!°„MQn±‘Κ%ΦΨ•06Φπ τ΄U:έbE(>uκTh Ϋ@θμμDQQΦ―_5kΦ ΌΌ A&L2azŒ16u|¦ φ«–οSΏυ`ε›Ί& ¬ζnΤvhGv ’ΠΠΠ€Ε‹λ dΙPλ?ϋ‚-!                       'q†ΝΝΝΥΞ°"ΘΑλ€I“΄vΦ¬YΘΘΘ€cΕΙNΔ‘žΔωΦUΠΫάά¬Ε±ύͺ¬¬T’Ψ|%ecbb4O‚₯ŽŽΤΤΤ8…±v»}LprεΑιƒ# ‚7qa—Ψή}&X•C¬%8YΉΔϊ…"VujΏ[¬ΡžKh2:σ”[tΆn݊ΔΔD"πƒ#Α΅H`x ΄ΆΆ"''Ÿώ9²³w ¦±v%ˆ6a†Δ†Y• Φ_~fͺλ2"ΘlDj˜VεkQmΛ«οAAAv‹mRΧ‘#–.Ejj*]š‡χP9€Ϊ(Š=$|\™H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€F†€««©C*bΨ)S¦`ζΜ™Ϊ688xΜΈŸŠnHHˆŠë8yŠxQς$IžΈ–––κya""αΈΈ8=ίχOUU•69œr#"" Ό˜H °°›7oVNΚ%6JΉΔN‚Αbσ;0Fs(Μ!“ΠΥZΪΪ„¬¬,̘1™™™~ΧV6h|σl]]>FW­Z…;w’³½ ρ!L‹6+—X B­u½ς?N"Œ5+₯ξ„`%ΨUΒ]«2…Ν―λA• Β°fυj΄(aμR%Œ—ζ±tΝυΏ=α}‹†$ŠΝΞΞv«Y’eȍ  :’’tww;»!ρŸ'N@@Hh===ΞΆJΤ3›Νώ1u60@&υ?'Žζ SaΛD$@$ΰ[eeeθμμtnT"{:τ;39A$@$@D ‘‘υυυΞΛs˜^R; q‚H€F€€D——χ]Ž„„„Η,ΏI€Hΐg$jΊkͺ¨¨pε4 Έ¨­­u›—™Γ; žœPϊd  @?}―»<ς^zι₯~ε˜A$@$@$@Ύ& ΒΏΎ©ΌΌ\»aφΝηόΨ& ’XωΏ_ΖHΚG\RΧ―_―Ε’"κΛΘΘ@rr²Σ-Vžu‹PΤULH„d\Œ8Ώ:ϊΌqγFqΔ˜8q’ξ†τ‹/Ύ@SS“Ί;ΕAΧ5ΙoHήHΩΌΌ<]VήKΚ{Ž/u%5>§ΫΪΪπε—_’€d―ψ±Β’ “=^Α΅žΘγό'f˜‚γ`±+·Ψζ<- ί²e‹ώνs ™μ§ρή’.Θ396W+©/θιξ@‚™ΞŒ3#=\ bΕ‚U~b~œD+"ή “v³Ή΅=¨­QΧέ/υ8—––L›6M]s’a6+ε,Σ¨’(Vlγ]“άPεζζΊfqšH€Ζ0‘ 6»3 ?y@$&  Ρ% /Ή˜H€H€H`,€<ςa" Ρ& Αωξk΄χ·O$ \ƒ@’ τ%ΰιψ(((θ[Œσ$@$@I@yΉσ:ΘjΈ    ŒW˜Ω+υK"ψwΊ={φhΑ‘ΈπΙ=«ΜoέΊβ|*fιιιZ¨#©2­Ε±~Ω©4Jϊ#ζm"Œ­Ζ¦M›πίώW3‘¬ˆd?ύτS-rαo||<Δ9WΔ±Ž$AθΔIVDoΌρAΖ΅ΛŸˆ†™Ζ/LΛϋ  6 ΅₯ F›œ†fΐd±ϋ-£9ζΠ4twΦ(Ax­Ξ›7O Β,Οowۘo˜8ΓΚyXΞΉrM’sξφmΫP’ΞΏ6γ>LŠ2cr€)‘FΨD Ιd0"ZΜͺΝ‘vΦt‘΄₯Ϋ·o‡ˆbE§°hΡ"€¦¦κkƒC€toΜ4sH’Ψ1Σkv„H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€όœ€bΕ=5;;[ςσσ!±νννZ$+ΑΥΕ uσζΝΊœˆbEH+BΡ΄΄4$%%igT.‚{¬ˆ[El$β*q…•ˆbwμΨ‘EΎ"ft”·X,a`ff¦s/ŠhxŊZ0,Ϊ½{χκ tv»σηΟΗΜ™3e91> ˆHzνΪ΅(.ή -°†€ΐl‹Q–J΄§D~ώ—Dxh„9dΜνΚ-Ά«I‰ςŠ±nέ:ˆΠ[πL$0Z$Hƒ/ίΤ]9J ی£`R”3£-ˆ΄`5όΜyZ_ιw#,FE`WίΑ΅(¬οDNNͺͺͺυgι%ϊzKΗζAxŽΠbŠbG,«%                       C!… hΨ£Ž:J rD *βXˆŠ›jGG•ƒd“ŒnΩ²)))ΪquΪ΄i˜wΆvΡμ©Ές§g9η9A$@Ύ"πιΪ-xςεwœ››”–ˆ[.±sž$@$ΰ+[vζαξ'_vnŽ‘ύ(8α@DDDΏάGoΉa!φ~ωΜ  œΐΓΟύk6e; ~ϋGΰœoγœη    ŒŽΞ.όμwχΈm^άB™Ζ'³Ω y&†δδd,^ΌΘΚΚΒXNJ8I„±"mkkΣΣ{φμΑš5k Z &h‡YΑ}JΨΣΣ©Ϋί’Έξ-[Ά eeeψψγO”Π·L ₯Ν"p•6O™2'Ÿ|²r•]W!’ΥjΡ ZZZ΄PP˜Νž=§Ÿ~ΊFλ.Ϋγ#rόˆσ°ΈZVT”ΓdQΗRhLAQͺb ιο=#ŒΆhX‚SΡΣQ§œ*+±aΓν-boΫ>:ΖωfδΊ!ξ°˜A½7+y£š2φ"!ŒiQVd„aUb@ψU Ά;ε¬`V׍x›ΊώĚ΅H6― ΥmνΨ΅s§ΎΦΚyEœΘ%…\sό5ΰΔ`} ΄εCΊ{ιϋbΕj1γ€£ZŸΩ^ θGΰzΫ?άςDΛσ›ΐ€ίϋ’—V.™7G,˜ε’ΓΙƒ!PXRαΆZLT8ϏnD8C$@Ύ!pΗc/ΊmhξŒI<»α  @ ΘΚ)pkς„Ψ(^Ϋ܈p†HΐWjκέ6Κσ‘ΐ ψŠ€Εδώκ^φ1‘ΐ@¬VεZΡ'}cιDG†υΙε,  €7^SΑ»\Σ€ΤDώ_ΰ „Σ$@$@$@£F ΅½£ίΆωΜ ’q—!’7»έ?αααΪΆΆΆV‹“ ΅‹¬•Δ]U\cE+0‰p΄»»[3“z***΄cf||}Ίώ Λ9‰F’€ό†$X8-oίΎ]gθhkE¬z₯‘aFjΈqv#‚” vL%ΥqΌ°”θˆ 2  ΎΕΝ=(/)Α' :0ΕΌyσ0kΦ,HΉV1,χ7k#»-ΦN$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$pˆDp#Ÿθθhν)βΧββbν+Ξ±ΫΆmΣΣ"ˆ!“8§ŠV§&“ " |ϊ駑‘‘‘EuβΌ qiυΆ΄155U‹‹;l‘vψμκκm“>‹0Φ“θHDΓ'žx’r]qϜ8q’Ψz 4vˆ»€«’’"νlYSS£Εkθdε©LbIΐg€Ρ KX:ΊΫ*QSSυλΧ#)) "§Ce€”\qY««Σ6n܈mJ[―ζ­†n€„š0=ƊdυbLFC@HΜ½A‡Y°™ Wί‘ΆnμνFuc=6oތ²ςr-^Έp‘ΎΆϊS°‰ƒι―Ώ―CQ¬Ώο!ΆH€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€ ‚OωˆS€ˆ[E#Nͺβ+N’999ΪΕ‘‘ϋ]ˆΕ)φγ?Φλ€€€hͺd§M›†I“&!""B;«šΝ£+;‘«΄OD"Μџˆ{J"”]Ίt©.'LF»ύ΅“ωΎ%Πήގ­[·"7wz{χ)Al"LΑIκ8‘γ{Ÿosˆ[3-0Ϊ`²Η£»ΉyyyΪ΅3#c’r|Άbν\ά Θ5Cάa7lΨ …ŸE{χ’SΉΓFΨ ˜nEF” ΑX”PTδευkrο«7s?ιkŒέ»r`‘l^}φ6v’¬Έκz[¦3,Dff&βββ΄S»7u³ΜΠŒξέΙΠΪΚ$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@ˆ`TD‘β*‡8v―1νΪ΅K‹cσσσQ]] q±“L‹s¦8ˊkrr22”8VV'OžŒττt= !ͺ‡¦ {–·Ϋ—rβvΛD½½½ί€8\655Γ$N«‘“Τw¨*˜>“9ΦπΙθν¬S}jΒ–-[0gΞ̘1C Β}η7 ,ξnε€ͺ»wοΦΗ—\#κκκ¬ά`'EšaAj˜ AJ «Τ°ςK Μ_ΣΠ I?₯Ο‘ƒfdD΄½…]¨jlΠό²²R˜BΓr'vq@g>ΕKΦD$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@~A@œRΕυU>"˜·ΏββbdggkΩR-ˆmnnFOOͺͺͺPYY‰¬¬,-ΕΡG‹/Ύψ€ξ¬~ΡY6‚ Π‚uλΦ‘€€D)Ω,0‡¦(Ql₯#Ϋ`0)§Ϋd˜CΚΠQ—£ϊVŠM›6iΗηΠPϋ2‘ΐΑGnω͈±a·ͺO‰r@ννlG|° iα&LŠ2#ΞnRŽ©ϋ·1^Δ°}‰JΏMFƒβb@¨uΏ{l₯%-έ(//ΣΑ'ΚΛ+ΤυwΆvΧs›ΝΦ·Ξ$ŠbW#                       @ .°QQQΪύ533S Eτ΄sηNμΩ³Κ°θκκβXq׌ˆˆΠ‚Y‹EΩbgΪ;vhΑ•ΈΪŠψJ„·Σ¦MsΊ[Š«§΄KΪ#e†3I}iiiΪ w8λe]ώK@ά.sss΅`TŽ?³=^‰bΣ`0ΩU£XΚ§~&SΜΑiθn«D{G½0ŠSμόωσ)bχίC―[ΦήήΟύβ»~ύzˆ³x£r>΅›φ!Q a'FY¨„±β«LΉ™Το1XΉΖŠsnˆΩˆ˜ζ4t‘Ί­C]c³QYU©Rˆk¬8°Λ΅x8©ŽΝΧoŠbΗλžgΏI€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€H€Ζ …ŠKΈΥ%&&j1κβΕ‹QXX¨Εͺ999!lyy9ššš UΘW’:_xα½=³Ω¬λ_Ίt)¦L™’~ϋ-e›+V¬ΐΗ )3\IΈF₯β:ηœs(Š.¨~^μσϊϊz¬Y³FΉ$—*!¬M‰H•»jP¬jΉX+μ F˜mq°†$££ΎEχQqEt>μ’rΩ$ΣΨ$ Nας[μ–-[΄›Έ8œz:g3"%άŒΙ‘fΔ؍ΚΦ Ž­±ΙαP{eUlβ‚‘\cMˆΆ”0Ά… =¨TΧΤΖF”©kλœΩ³1wξ\} γοτ ίΒ!4‚«’                        ψ†€ˆPΕ­N>©©©‡Ικκj-XqlAAΚΪνβ¨9 €]λ’““1}ϊt,Y²DΉΕV`Σ¦°X,Γ*χWωˆk«γ»o΄¬oYoη₯NΗΗΫuX.° ˆΘ{͚/PW[³ί%V bMA1έ©~­7κ>YBΠΣQ‰ͺκ*ˆ[¬όŽcccιBُ3$(Aee₯T AD@.nήAΖ^€D˜05‚δ0"•S¬ω+1,±zώNe~£„Ε‘j"^‰Š£”kl~]7jj«±©Ύ">.))Αόωσ΅[zdd€ΎFΈf.u%@Q¬+ N“                       ΐ8%`³Ω qpMKK…8¬Šxu8’ˆo-Z„ &θ:₯ξΩ³g» χDΈ*‚ΎcŽ9fΨΆ+mGDƒΑ ϊ”6]a~N@\b³²²“³ ½J‘f΅ΗΑœ€DgΓs,ϋOχ•[¬I Χƒaj‰EWS1v+7Jqcq»823‘€+Êψ΅  P>Ψ¦ά…««ͺΠΣَX%€M‰°hwΨΔ‚”Vιaιλ ΠΛi- VΧaΌ!V3’l&6t‘¬©΅JΐώΕ_hq¬cgΜ©―O!!!ΗzΙx¬Ν½μ6‹‘                        x" βΤ¨¨(O‹:/,, guΪΪڜuΘ6d[Ž$Σ"ˆ±μp'Ε&%% w΅¬Ο TTTbΓ† hll€Ιl‡9$&K”2Z΅v¬ω]*α’9Lυ1 έmΥ¨««Γ–-[žžNΈ›£Υ$ ΠΪڊRlWBΨμμlμΩ³m­°Y€‰ΡV€…›jRNΜ_–ΗΪ―ΕΧό…ŸΡh@˜Ε€I‘ϋe·*‘l •0Ά’₯yyyWλάά\,\ΈP…ˆΧΑ)|έΦ@ΫE±ΆΗΨ^             Pjλ›°§¨εUu±Ϋπ#          ψ-‹Ε‚©S§jΧVG#E+bUG’ω””$&&Ίε;–κ·Ιd:Τ*ΈΎŸΨο»]‹ΝΊ{”KlX,!)0eߏE‰ί>%ό5cŸrΒ5—£«9_9δζ`ǎ ΈΞΟV4OΔ°Zx)ΒΫΆmΣΜϊ†z˜{;‘¨D°iaF€FX€~/&%²ώϊ”μƒŽ“M(¦AŠmœΝ€P%BŽΆ›PΤΨ‰’ζ^Τ6Τ)gλT)ΗήΒΒB̚5 Σ¦MCxx8δΊιz'΄Όκ&E±^a ΜBŸ­ί†³/»Ε'ΏρWηαWηξ“mq#$@$ΰ–Ÿ{›[MYρ̝ˆŠpΞs‚H€Hΐ ttv‘΄²ΖΩΐΔΈhΨ‚¬ΞyN  €?(.―Β©Ώψƒ³)KζΝΐγ·^αœη  €hikΗΔγ~δlœΩdDιͺ—œσœ =cι™n·ΑsΗγ/βΩΧ?@]C³ͺE x)[ΝsŽ'H€H€œδΪa6ϋΧ€ίήή^TΦ6 62|ΔΪΦΦށ΅Π`»“'H€H€H€H€H€H€F—€{ΜκYζ`I„«―F‰Λ"P^^ŽM›6’±‘FK°ΕN…Ρ:Pρ1’o€Ρϊ:=νZ\·uλVν:™‘‘1FϊΘn •@WW—vΞΟΟΧbΨmΚ!Ά¦ZŸνξDxΠ>LŠΆ )̌ ΑFΨΜκRΒΝ±(*·‘*Ώί5UΑlΚΈ:B]NΩ§Ε±ΕM(+)FMM-d!33©©©—u^ϋο•Αο&ϊ―Ü" /|‘$r  Œ'2@έUΫΣγ›σνxbΜΎ’ ΐHψ͝Oΰ…7?rVΖc·ΰ¨…™ΞyN  €?θξιόίαH“R“ό& πsΟε{AΧΟχ›7žŒ•gΊβ {ή5ΖΊ­;ΗΣξc_I€H€†H@ΟΌφr‹J‘WT¦œšŠ€ψdNΙΐN;ΛΝφ©Γ@}c3ž|ωmdο)ĞBΥ½e –f5Θ9=9“Σ’0sr~ρ½S0!6jˆ=ή_|λ<Ό±b5V¬ήˆ‚β HΠI‘α‘HMˆCζ΄ όμμ“°`Φ”ύ+π/   Œ2€ωΈ]γ¬ύΕϋηυΗIƒ$@$@$ΰ=q6”ˆΊzΤ;τNυ€|‹ƒ‘8η9„³2m΅ZδΣu½ο Kϊ’€Ό«ΫΊu Π«d.V»ΈΔŠλ°  &F³ϊ]O€Ωž„¦\%¬ΛCvv6’““υof$ΆΙ:ύ—@[[›vέ΄iΆoΟBqq1:ΪZ¬\JS’MΘPbΨdυ ± b=λDe˜oφ§p6+ζQ6£βΏ±φ Δχ`O}Κ››‘›—‹ΚΚJνθ;ώ|Μ›7O»>Λ΅ŽιkΕ~Ν‚S$@$@$@$@$@cœΐλ¬rĎρξ²{$@$0¦δγ'^vφ)iB nΉόΗΞyN    ΐx πΌKAμxΨΡμ#  $ΉEΈωΑgραšMύjwqωdν.ΔΛο|ͺD¨‰xτ–+°ΠΡ—ήϊ7>π jκϋ΅Kƒε*α|ή_ΉOύϋ\ύ³οβ’οŸ«ΤμMͺRŽ³7ή4ώύξg‹‹ W>Ϋrςρ―}ŒΓηΞΐέΧ^ˆΜ©Λ3“H€H€•ΐ«ο―DuέΧΧ=qmg"  πŽ@kk+P[[‹²²2””” ΎΎ---hooΧ’X©I„°"‚΅ΩlˆŽŽF\\RRR…ˆˆνͺ'΄Lγ@EE…ΕnΩοk ƒ%4³}ά€0˜‚UŸΣΠέZͺBλ•(6 ³gΟFZZΪΈa0ή;ΪΩΩ©άFk°sηN嘼 ;wνB}]¬&’ΒU€:%„M‹0"Z 2%OΕ°Ύ?jΜeD«STˆΕ„°  ν½(iθFuSƒ3oGeU•Ύ.\Έ'NΤΧ8oΧ}ί#ίo‘’Xί3ηI€H€H€H€H€Fΐή²J\uΗc£°en’H€H`8Τ¨Α#――Xε¬jΪΔŠb48A$@$@$@$0^ΌυΙZ·†…γη県ɩ‰nωœ! Άηΰۍͭ^u^D¨§^x=ξϊν…8οτΌZg¨…Δ φWή†ΟΦoσzΥζΦvόρ‘βE%^}ηo·#",δ€λ–VΤΰŒKoΞ³,θ²PœtO»θxαήίcιό™.K8I$@$@ΓCΰ?ο}><±  GD [ZZͺE\»”€«¨¨H»δ‰°«©©ΙMλΐ"N±"Œ Cdd$bcc΅#ζΤ©S1kΦ,$%%iΑ,Ε±bc[œ„ΏόςK吚ž^εΔš¨œSTΗEψ琠mF£&{Όr‹GwΛ^μΩ“‹-[Ά 11‘n±c{Χ£»[ )««Υωs/Άmۊ;v ΌΌ ƞ.L1")D b#ΝΚ‘Τ»RJά€ρρ«ποΏ`5fD€­ j…H;H@IDATεΧu‘²uͺ+Κ±Z‡ rmΛΜΜΤΧΊππpν˜ξί½ΩΦQ;²|Y;     €‡]ίψ€Χƒό Ιl        τ#PXZι–'.wgŸ΄ά-3$@$@γ@ήή2|χΧ·τ{bΗβ9Σ‘£\b› N²ωΕεN@]έΈβΆG‘’‡c—ΜsζΧď½ΨOk21%=Y;ΥΖEE  €B»Δ—WΉm6'Ώ—έϊW<{Χ΅nω3βΊχέΛoυ(ˆ‰ΔμiˆŒ@NA1vενE[G§suŸ}Ω-xνα›±xξtg>'H€H€HΰP Ότφ'ΨΊ+P«αϊ$@$@$0nΘΈ.qmέΊ«W―ΦίεεεZΫΫ«T*νΫ·O‰· ϊΫŒ,GD͊«¬”ηΨ &`ζΜ™X°`–,YŠττ4ΠUΟ•ά؝υ† ”Ϋp½r‡€54¦qδλΨ³FK(Μ!©θξ¨QNΛuΪmrρβΕHH0ΣX# ηΘΖΖF-šάΌy‹Γf£hο^t΅·!<HŒ0+±₯‰J+N€Fšhϋν!`6‘AFε«άcΥΎ*mιEQcͺZΫ°gχn%r.Gnn.ζΝ«Ε±ρρρΦΧ?ΏνΤ6Œ’Ψ„;ΪUΫƒ¬˜δETθfυ㨬©χΨ\‰.αq™kfdX¨λ,§I€H€H€H€Hΐ―άσχW°vΛNΏjC$@$@$@$@$@$@$@C! n{ M-n«Μ›9Ωmž3$@$@γ“ΐEΈU΅ nΏκ‚³qΩy§#,4Ψ™/δV¬ήˆΛτˆΫ8‘_ήό >{ρ>ΔD†;ΛκĚMΩxψωΊU³@]·ΊρW˜19Ν-_fήώt»ϋo(­¬q.{ϋ“uxς₯·qαχΎεΜsxϊ΅χ΅ΨΥ5oϊ€T<~ΛZλšί₯œ2ξϊUάϋχ@¦%΅+‘μoξz={7ŒF5βŒ‰H€H€‘ΐ[―Εε·>rˆ΅pu  ?δΤββb|τΡGψψ㏕£ε-p•|ΌΚ·όΏf±X`΅Z΅°UζEό*ΛΕQD±ςq­δ·΅΅‘°°ΗΩΌΌ<œzκ©Z$+²Lc—€;wξDQAzχ€έRc•¦Iuz<ωaŠˆά¨άb'ΐζfνœ+Ώ―θθhύ[»GΑψλ™œϋ$(@VV–?gaχξ%„G±Ι‘<Ξ°Ia&-΄΄ͺŸΒψρLΜcΑq¦²(εr¬έ„Pλώο½Mέ(lθFMcΪΧΝΚI½J_?E+ξθςΫχτρ–(ŠΓ{\"YϋΟ_να«ο―Δ/nΈΟcΉ³ώoώrέE—1“H€H€H€H€ΐ›wΰ/O½MeI€H€H€H€H€H€H€H`@WΧρΡ‘³œ& ‡DδΊ){[ΟοΎφψιYη–'328ψ›G-Β›ίŠoώδZ§³l… €ώΧΎ›.ϋQΏu6γ¦Ÿusπ™š‘Œ·ŸΊ³η‘Jί:ζpνj»μϋW ¦ΎΡΉΩG^ψ―GQlcs ξzβ%g9™8υΈ₯xτΏ†έ֐³lχ7??'½wΑu—\IΫs πμλ+π“3OΤσόC$@$@C!PVU‹έ%ψbs6^yχsξεC©eI€H€H`ΌακoΌχή{O‹{DΤ(BXqu‹‹ΣŽ―₆ΠΠP-Ž¬c₯l{{;Z[[΅C’ˆΐΔAOꑬδηηηCœC«ͺͺpώωηkg=YŸilΗαmΫΆ©ΰ’M0…Α’£9dœͺ 0YΓ•SnzΪ+΄sξζΝ›1i$$%%Ν`φͺN»†Š;²‘侞L°‘aAJΈ±6ΐn6¨η‚ϋ₯αΡε8Δp]VΪv«λ]Ί†™ΥΎ4boc7Š›{Τ΅T‹Ÿ%°Δ¬Y³0{φl-Ž΅ΫνΧΟCi°η'Ν‡R#Χ%1N@"†τ’fŒwέ#   €# ξ)ίψ€Ž (Ÿ’–„Φφ·HσΧ)6˜H€H€H€H€H€H€H€Ύ"`΅πu/ οξzςe7ί8bGA¬k‘ΙκYωε?>·>όœ3ϋ΅VαΖ_§…³ΞΜƒœqΫvε»­}Ϋ•?t¬E\tnΏϊˆσ­#ν-«.Έ²Μ5=πΜλ¨mhrf……« οΏπ(ˆuRs¦OΤ}Ώϋo_s»ηο¦(Φ§I€H€%pΗγ/β±ώ‡–ΆφAΛ²   €gβθϊώϋογΝ7ίDii©ώTΔ ˜1c†ωL™2iii‘eΕOΚHρ¬8$Š8VκαknξμΨ±S‹ΔD+ΒΨΪΪZ|ςΙ'ZT+βΪiΣ¦ynsš€ β»cΗ9:`Ά)—Tε” Q•©ceό%εk4Γ¨8˜m±Ϊ-vϋφν˜?>bcγ”[,Εα|LH@9oŠΠyγƍ(((DWgΒ”«hzŒιαfεk„M‰a•α¨φIΏ‚@ήǎΆΛ~“Λ^ΈΥˆ`'76؊ J›[ߍ²Ζν ,ΗBnn.Ž:κ(}ύ” γ%ί’:Ž~6%2¨gžΞ8Α™-Q;U†ΦoΫ₯#€U©Θ£[χ’"Βt‰όωφ'λœε]'–/žƒτdu“r€τω—ΫPP\Ρ―DTD¨ŽΪo‡ yhυΞ§λ ‘U7lί­^΄Τ£Ή΅Α*ͺhB\4RγpςςΕ8EEMTσL$@ΐ–yΘΙ/FQi‚ν6,˜5sgLΏ›αβSXR­»ς°§°Σ'₯B’–δΕλΊ­»°·΄•κ·lU‘žRβ ‰§MLlu.' PWέρŠΛ«tλΝκ‘θc·^‘Σά ½a³I€H πΘώrߞ[T†όβ2΄΅wbΊΊž19Mί‡ΛΧΎL]]Ψ•§ώ_)«T³TjΧqΦJŒVΧΗθΆ……r“Ϊ;:ρΙΊ-Ψͺώ?ͺ¨CWw“βυ³Œ‰) ϊ$Η‹ΊCή+  C$°#·HŸΤ9;)>κΉkς„Ψaί·iC}¦γ«σΆk;}ρlk―ΊΙvŠΛ«Q]Χ€hυL\ž7΅0b£\›Γi8d#}?6’χ=£q Έ?>g:„+C=/ _1πΥvΫη\N$@ώN`cΦnΘΗ5]ϊΓΣ\gœΎ@9Ιήφθ Ξ ’ς,ύΛm9XϋšΠη,πΥ„ φΞTΏ½?ζgX2ofίΕύζgό3ˆ³Ÿ€„Ψhlxύ=]ί،λοϋ^zλ=/Ž]2ο€’Ψ‚’rHtδw>]¦–Vηzr^ψώ)Ηβ'κΕοpΚ—EG~οrη¦DŒUψισΞωΑ&–ž}φ~%δ’²Ÿ>/¦€' Ά—“  |Eΰ…7?‚\ŸιΊ‹Ύ‡ω3';fωM$@$0‚D|qί?^Ε/½…ŽΞ.[2 θΝΏ>ΐ{ϊϋŸ~׎ή^χ˜…Œ'yΩχuϋψ#πθ-_ί;¨ ωίώιWίΓ?ώσ*UP’=ȊӿyΞWA½Ÿ;c bζKΏο|β%<χΖ‡ 35=ΏΉπœqΒ‘Ξ(ΆV:ΔOΏϊΎ~ζβΊΪ G.Δ“·]©ƒΉζsšH`όΫΓΟ½ηΥ=³ θ›$ˆΨ/ψm|χδc b«U³p·‹ϋΤe?:ί8rAίΥ0\ΟtFκΌν«g[ύΐ|•±1{ξΗ νΑ}“\% δΧό Σ2Δ­/΍ΐpݏ ΄Υ‘Όο©sΐ@}ρ&4Ÿ3ΏυρZάόΠ³Ί™žΞKΤsdG²(ΧΨΥ/=ΰ˜Άσ²Tθ+ΎΪŽ'H€H ΐ |ώεv·„ΫpΔ‚YnyΝH`°yΚ5uӎ\g‘W?X9<’ΨΌ½Ξ:eBπHΠbo’΄+2<ςnΨ‘ΊΊΊ“ϊ[OK 6ΧtΡ‹]g8mSΟ€f¨ΰΛςΏŽ#}Έf(ŠuΠΰ7      ŒyΞΩΡсχή{yyy:XSHH23gγΜ3Ώƒ₯K—"::Z;Γ₯β$©?±±±HNNΦΒΪύοZ[SSƒ5kΦ`ξάΉX²dΙPͺfY?' ΗTAA>žDb I€1(^Ω+Κ³ˆώοδόΌ;ΓΪ<£Ω£=&[‰r‹-DΞξέ())ΡΏ1q^f rŒ‹#ΆˆόE »aΓ½;Užˆ$ΣΓ­H4#ΦnDˆeΏ;lo`t­}6SΞΏv%Š"ϋΪ‚8› ωέ(oξUξΑ%hllΤΗΙβΕ‹1sζLε «œ‘ƒ”8v  ’ΕΠΞ+M}M½DωεΝyτγ}lnmΓΟ―v‡J{ώυΏρ±zIςδmWαHџ‰Ζ"f%zΟ/.wvM%έόΰ³ψ«H9P’—’Ϋrςρν‹oΔο.ϊ$ς ς(ΙΐLΗΐωεδ$)·¨§_rΚ«jZ­_ώ=OύχόύtφyIΪ·`φžBάψΐ3xκίοβΎλ/Ρƒϋ–9˜ω}κ)G?dύnSϐͺ‘Ύ»/υ1‘  xG@Γ\wχߜ…eЯώsž$@$@#G@\sηγύaυέ’<°•€SηζN\rξiΈρWηΑ’^RΉ&yYαzOμΊL¦]—‰«§τϊ«pιΝϊ¬Ϋ¦ώ‘οεsΞ·ŽΑΓ7]vΐ]\··vΛόδΪ»=Χr-'Σ» Kπ‹ξӁ…ήxτ–a °υ·Ώγvύ“mtτaΔ & 'MJ˜yαυχ*qS…3―οΔξ‚\qΫ£xF|ε‘Q­‚ ¬άπυ`{9GzJΓρLg$ΟΫΎzΆε‰Νso¬ΐ5~έκΪ6P’k£aόƏ~ƒ'Τsζ“—{?¨ :™?> ηύ˜'‚#yί3’ηO}ρ&o΄Ÿ3K°GΧης}ΫμΊ¬ούτpœ—e{Ύbΰ«ντeΘy d«]Dc—ΜχZ|*εZ4ΫM+χ·_uΧΟC€O)iBŒ[vie ΪTπq»M$•WΧΉ b₯ψΔ”D·΅ UpΧηBςξ9#e‚[™Αf&₯%Ί‹bWoΤο±[ΛI€H€H@\xΞ·pς1‡{„!Ο .Ήιλ€E 1“H€H€Ζ1°wο^¬[·ννν°(Qό3πƒ|Gy€rΆ ;d:‘‘‘Κ5vΔΆKΕ}ι₯—΄ ,''[·nΥ.zV«υ·Γ όƒ@ss+v+±gyy Jθi MSn±ακω†΄ot[aP,"”[lΊ[JQUQ‘YMœ8Q>F·mάΊ7δ&’ώ;wbΣ¦MΨ³gλλ”l&F)gΨp ’CM³`ώJηLŁ7d·Œμ_³Ρ€p΅ΟνQ@”ΝŠ’ΰ4v)ΧΨFμT±Υ5΅:Xΐ~ΧΨύΏχ±(„W†ΘL$ΰ;έως[ρ[A¬Ό„9]‰φV¨—“*”ΫΜyΧό9Ε³:Χ!€# υ½λΙ b];ΥΣΣ‹?=ς<ΎwşΠ=ΐ€uΧςŽι2%„=λW·x-ˆξ·=‚;—WίΫ),­ΐ™ΏΌβ,ΘD$@$Έ:ΥC€‹ώpZΥ½€°`<ϊΗ_»_ΰbΛI€H`δlΟ)ΠΤΊ†―|φξasz9€qeAΟχC4·Άλ-Θ€ςσΟψ¦vSš˜’³ 5²WEθ}χσ/ρΔKoA^ :Gk6γιWίΓΟUΤDoΥw<Φo`lT8&₯&©¨ΐ ύͺΈιWράΊεΛό_ώπۘ?s2fMIΗή²*|Ή=GΔ}@"ζ;•·?Š„Ψ(R;˜π›H€‰ΐmΌΰvOwχ΅"%!.ΊΐΆ’ @@hR‘Ο/ψέ_ά\:Ž9|.ωΩwqψάιpD›«SΒͺά’2άττ ŽΞ~Ή-)qμΥͺΌ#}ϋ„#1ύ«A”2ΨρφΗ^t,‚ΦΫ―Ύΐmή9£&*”«ˆκμκvfG„…ΰOWόG.ΜDjbœ3`BM}#ήόθ ύΏ‹λ Κ§_}§ΏΗ>ΟYGί ιχΕ7> LŽe2XσάӎΗOΞ:SΣ“Υ L3ŠΤ Y» pλΓΟιώ;ΚΎσΩz<όάψεy§;²†όύΘσō<γΆή'…ΗnΉ\oζΆ€3$@γ–@“zn)ŽΦβTαš–/ž‹³O:ιΙ`Tα’e ψ:5°^Ξ‹’ΆξΚǍχ?­§‡ϊg(Οt|uήvνƒ/žmΙΰ_ήόuy–%N"³§e`RZ ŠΛ±M1ώϋ+οj'qGϋδ9σέϋ7ΈαRGΏI`P#q?ζΊΡ‘Όοs€kίωσγΞuRβΠ_|ο„}ε(+‚RωHZιώL!4ĎSΥ₯U³Pίψ΅cνԌdόη‘›4‘ΐK¨)|η›GαΤ_ά€ΉEΞjΧmέu@QμC|"ͺu$ωΏεΩ»~Ϋ/Ȏˆ δsΒ‘ ΅ανOΧ9VΑ_žz—œ{šS€λ\ΰΕΔƒΟΎ¦‹>ηVς»κΪχΧ›~ε"»-δ ΐΈ%π€:_Έž―μκ|υŒ‡σΥjຜ“ή_Ή—άτ€v‰瑦‘>ΣρΥyΫ΅ΎxΆ%Ϋ›Υ½πόκGgΈΉ”K`Ά₯σgβΜ[†s~}«[€ŸwUΰqr–pm;§Iΐ‘ΈsέΞHήχŒΖ9ΐ΅o}§ύι9³ϊrϋjSbΧ$Α)tOμZv¨ηe_1πΥv\YpšH€Ζ Ύ’Nι—λϋPoϊαAόZ]Ϋ€i)ή¬~ΐ2χώώσΓ«œΤ*Tε“~φ{όξ’ο«χΔΛέξseάΕ3―} ±„Ϋά¨968m’»m•jsK[;BμήθΪ£‚ΆυM΅υMŸυ-Ηy      ƒ' ο>Δ₯RλδˆˆSE;R)88sζ́|744h—Ψͺͺͺ‘Ϊλυ1ΦΦVδζζjΑ Αψ•#ͺQžτϊΈ%ώ½9£)¦`$Ή­---Z˜~Ψa‡!"BΔLώD@Ξ‘"b\Ώ~=Φ]‹άΌΡ<}Ω²‡ŸcΐΝ=τ‡_κΑΊœ}’;ί?υ8‘½ό"τ”ΔSάo˜H`<x^ fο+ˆuνwtdώσΧ›ά" Χ54㞿ΏβZΜγ΄Cxžž4+žΎ?SΏΓ±RΑm<―_x:*;vΙ<=ΐσ@/€—/žƒοΏ2Τ‘Š”Λν‹o~μ˜ε7  @¨kΐ₯7?θl© ½λ·:η9A$@$0²ΆμΜsΫΐΉί>ΎŸ Φ­€šωΡιίPξ ΡΞlq/άάT‡#mΚήγV͍ΏϊΡ ƒ%xŽ<£pMλ·νru›–όͺ Xιφ«.θ'ˆu].’Ω{~w±ΫGM-­Ψ–SΰZΜ«ιϋώρŸ~‚Ψ¨gί|™ΫΐR―*c! 1M L{όΕΉυρΩ»―=ΰωκΔe‹πΎzώ)nΧ“†ϊLΗηνϊ1’ΟΆΫόιY§σYέ­Λ%@ΓΓ7]ζ˜Υί"bξ{}u+ΐθC ορ2œχc#}ί3šη€>υμX|Ξ<Τσ²―ψj;žφ3σH€H Π Τ7} ΜΡ—("WΗ2OίΑΆ ~A‚«λΎώεioσ¦€'αo·_Έθ―W–TTγW·ό3NΊ§\x=~ώϋ{qβO―Γ€γΟׁΣ›[Υ‡« i/?ψLPdϊ¦ΔΈh„…»e‹ΛμPRnQς„†RΛ’      xO@_υυυθξξVځ ΔΔČΈ[ett4’’’΄HάΕq‘il¨¬¬Dž φφξS.±Ρκ§cΗFο†―βšlΆ%ΒhT.ΝνΪ]·ΊΊzψ6ΐš†…€–wοލ?όοΎϋvŠ΅τv`R”σ'X17ފ ‘ΚY‰EιCΛΖYIΐΠΗƒJΫΝF€‡1O/ςI5’£Ή›7oΖΫoΏO>ωX»E‹c{ 'Šb}hϋΟ=νx|ςVύλ~<ςΗ_γ: τOWώtHQ;‡³λωΕεψhΝfUΚ`¨φzJ gMΑΏύΉ§EΌo|ΈΖγ2f’ΐX"π­cΗβΉΣν’ˆR―Ύΰl·rOΎτ6šΤoΕ›tοο/: πVκΨΊ+zλgu"r½ηΊ‹œσšXΆh6~ͺ·ιυ«\g9M$@$ΰη.Si*U€yIςηQuŸͺBb1‘  ψ„ΐ–ΉnΫΩ§^8 –$κλΟ8QΉ˜„:?28r8’«°Β¬Άsœ ˜γMš;c’[±½₯Gˆ½σ‰ΑΥ½$sj:ΞSBίΑ’ =Œ܊­Ϊ°έm~°™»φ2n{τ·b?Ru>¨{I 2& p%πηΗέΟWί:ζp·tΎkΣ“Σ’pώwΎιq™·™ή<Σ‘Ί|qήφΤf_<Ϋ’o7\r§Ν»εIΠΉ΄€x·ΌΚ·yΐΐŒδύΨHίχŒΦ9ΐΟ±ώœΩ›σ²―ψj;žφ3σH€H`,¨ktΕΚ»Q Ζ5Τ$Οe\Seνώημy;}ςςΕXσςƒ:πΈk@y햝χ±³v«wΖ_‹a₯άςΕs±ςΕϋqψά«ΉMO›θξ"τΨ oΊ-?ΠΜ+VΓΣ3¨Z>?6.#     a!ΠΫΫ qχ”@~f³ααα#τYΔ·²Iνννϊ3,a%£J@Φ{χξEYY™*‘`PΌΖΒ@—Ψ~;F εLΆH%ŒFLΚ‰΄ωωω†LώA@‚lΫΆ ο½χ>ϊθ#T”—!άά‹©QfΜO°`Ίϊ 2ΰ+ƒXh4[α·Μ&bmFdΖ™° ή‚τ#,½ΪYϋCu|­X±{φμΡΞΡ~Ϋ /Ζr^@b‘α%pŠˆΐ —B"ŽϊKZ½1kΐ¦ˆ`χ@ι;ί\†)jp˜§΄ςΛ‘ hυTσHΐί όμμ“½n’β]gw©(OωΕeƒ//K9|πμ)ΗG΄{©τ‡j0zzς„Aλwθ;x]^ΔΆwt:σ›H€Hΐ H …VmtΆπΧηG,˜εœη  ΐΘ0›Mn‘€5]]nyžfωωw±ϋƒgœŸ³OZξ©Ψσ.9χ4<~λϊστ]ΏρzPhCS‹WΫ’=^~ϋS·²ί=ω·yάφ™9αΘ…n9«6 όlΒ­ šΉγρqη/ΉeLωΉW9ΠJ`& p% η«Wίϋά5 ύΰT·ωΝ\ρ“³0»ιΦ“eή>Σ‘²#}ή–mxJΎxΆuα9ίB˜rΉς&₯Lˆu+F§*7œ„ΐHݏωβΎg΄ΞžŽεηΜޞ—}ΕΐWΫρ΄Ÿ™G$@c@½–Ί¦Θ>βVΧešŽs.Y]Ϋp βC^–΅§;Τg(i{N>ήώtνfΚ»gΧτŠϊΏgΫ|Χ,ΣΥuΈζΞΗ=.λ{?ε±3I€H€H€H€H€H€™€ΈΔΚ³g €-‚Υ‘~Ο-β[«Υͺ·#Ϋξςb,Γ!w’Œ8ΆΆ6νͺΩ©œOζ`%ϊTnΐF₯QQΗ“F³b€„Γ&;ΪΪΪ΅ NDβL£O@ά«7lΨ€>ψ›6mBC}βCŒ˜gFf¬‰Ακ\iζx ΡίSΥƒRŒ†XŒH 3i—α™1JXmΩ‡šͺJ|ρΕΪ‘8+++ E˜k—°΅N@œPώό›ŸψλP9‰πΝSšœ–ˆδ>ƒ<•[˜9{ŠJϋ-Z½Ιϋ­ύVf ‹ϊ'ρp/\b]‘ͺ 3§ΰΛm9Ž,W`ξτIΞyO')Q¬7)'ΏΨ­Ψ’ΜinσƒΝLΛHA΄r΄uDξθμΒΞΌ½˜?sς`«r9  ΐ(ΘΪ]€›|ΖΩ9o_wΡχœσœ  π e‹fγΣu[[©œOΟ½κά}ν/01%Α™ο«‰SŽ]rP›zνύ•^­WVY ωŸΑ5Έl‘λμ§%xÎwξ,c6y»ξO<ϋŸ~ΥΉžL\τύSpΫUΈεq†H€ΚΥ€oWWkqŽ:lΆχΟLb£τσŸΟ" ·Οt€­#}ήvπpύφΥ³­I©‰›=ΰt”z6εšκθTεŠƒΣƒ©ϋ1_άχŒΖ9` œcω9³·ηe_1πΥvΪΧΜ' @'ΠΨΗ]5*άύ^ΫώY-ξC‡ŒΓd7!ƒ›―Ύγq<ϋϊ6Eή{ P&οk―ϋΛSxχσ/ρΤνW£―pW*<Œoβω~€fe^Άχν‹oΔ]Ώ½ί=ΩsΠ΅Vmΐξ{βTλ)ΕτΉχT†y$@$@$@$@$@$@‡F@ώwJω–$ΒΨ‘Nbθ#ΒXI²mΊcŽ4qίΤ_[[‹]»v‘WJfk”rŠ2ŒU |³ύ@ۊAΑ1ZcΤ' ]mε(**DCC‚ƒƒύNίhlΆ½βœ]WW‡5kΦ`εΚ•Ψ[\ SO§rυ4)1¬‰!&[ԞSΗ5λƒ₯<ΎΧ“γFΥδXRί‘AFδΦv‘΄‘N °›››!Α"ζΝ‹ΠΠΠ€ƒεώd;ΰšΟ—ζ΅cŠ/ϋ–SPμqsν]Έτ¦=.sΝΜΪSΰ:뜬©Χ.“2Ќ‰Ζ"Ή3&Β>DΧηEJDξ*ŠΝ/.͌I©ƒ–‘}ΕισfXlλ©™“Σ°ΚΕ=zoY%E±ž@1H€ό„@[{.Όα>εDΨ­[dWχ]έrdp;  €o ˆ TάK»Υ $Gϊdν,9ϋ2,™7⌺lQ&ζ͘¬^6όK-Gϊ–ΛεUu(RχόϊSR‰OΧoŚMΩ­β–Ÿ_βώΏΜ„˜HH o“DΊ•ΰaCI|θŸx蟯χ[eξAόοΣ―f ŒY}ž½,œ5eΘΞ―Ι qΕΗΫg:ήT~¨ηmOΫπΥ³­Τ€xO›χ˜ΧχYrww―ΗrΜ$OFκ~l4ξ{ϊφo$Ξ}·α˜ΛΟ™½=/ϋŠ―ΆγΨ·ό& ±F "4Ψ­K]jπΑ€5.Β5ΕD νy…λΊΣ"jν+ˆ•gψ—ψLœxτ"LNKBˆέ†¦ζVδξ-Γ»Ÿ­ΗΓΟ½αΤGž-ύςζρά=Ώs­ZOΛ³•{»ίψρo!χ ’š”Pψ’›ΐ“/Ώ9Σ20mb ΒB‚u δυ[waύΆ]Ίœό‘mΫƒPεβŒξ\Ξ       ‘' ۍ΄KμΘχβλ-8„Ύ’3–ϊυuύgJ„Ν(--Υ¬ΝA0ZTΐ°―ΔΦώΣRj‰Ρ¬8E »­J‹1‹ŠŠοϋWkΗvkδaχηŸŽ>ϊ•••Z΄81Κ‚YΚΡ3!T‰ωUπ:ŠaΗφqΰ‹ήΙ1dV^ Ρ6ƒY›a5ΐTέ‰β†lWN±βΊέΩΩ‰… ",,, _)ξ‹#ˆΫpJD|ηJ>˜¨ hIE5^~ηΣCjAM}£Wn³‡΄L£Dΐ'εΎM‹νσ5ΏΈ¬o‘~σή ΌzsK›ΫΊg\zŒe©Ύ©Ε­t₯ΛKP·œ! π 7>π \]En½ς§˜’žδmc#H€H`ΌȜšΏω\pέ_ά„±2(Q„¦±i° ¬#.©"’ύΖ‘ ΰ‹g]]Ψ°}7Δ½v햝(,©@qy5v°¨μ[©Γ5%ΔE»ΞϋτšM;πΩϊmλ½αΎΰψ₯ †,²υX3I€ΖΎ’Ψ Κωu¨)υ E±ή<ΣρΤ–‘8o{ڎ―žm₯A멝Μ#o ŒΤύ˜―ο{|uπΔu¬?gφζΌμ+ΎΪŽ§ύΜ< +βcάονϋΌητΆŸmnEc‡AϊΏΏΐS~Η­ήΓηΞΐ“·]ΩoόD˜χΟ9Y~|ζ‰*pωψόΛνΞuΕ-φνOΧα[ΗξΜsLΜ™>Q;Γ^οίΡΡω΅ΈwcΦn§ƒ¬£lίοGoΉ—ιa·μh:ΕΊρΰ      €wD +‚Άμμl-.Z°`½[™₯Š@kk+ςσσ΅ΛhΆ+χS%Š5ΡΘl0˜ƒζ ht›‚ΠΡсΌΌ<Μ›7’ΨΑΐ σrK%’ξΥ«WγΣΟ>CEy"lFLŽ2cF΄±Α"ˆ₯;μ0cχΥνSΊ›r‹MUbkƒVC7ŠΊ§Ξ₯έｧάεϊ0ΒXŠbΗύaν[ώϊ‘Ύ©yΔ@΄΅»Ώ@± ±b‘aC·Hκσ"±΄²fΠ–{#kέSX―Ts¨©£ΟKΰC­λ“  ·>Y‹όη=g…'}~’Μ0‘  ŒœψμέΧβ·w=©D§Uͺ\Ύ?\³Ip”+H"~zΦI8οτo 4ΨξqƒΝ”¨ŠS0οxό_ύ‚θ T§Έ„΄΄΅΄Ψ™ŸίΗy1&jdD$ΰ­SΑΎ~§άWώvϋUΞφq‚H€ςϋˆψ#Γ‡ώ<'%!ΦQݐΎ½y¦γZαHž·]·γ˜φΕ³-£Ρˆ˜a8ΪΜoŒΐH܏ωκΎΗΧηO,ΗϊsfoΞΛΎbΰ«νxΪΟΜ# ±B .ΖέΡ΅Q9LjSΟj\Σp<γxπΩΧ]«DDXžRΟ-γcάςϋΞ$ͺ cOήv–Ÿ{\ίυΚs“—/φ8Kή Ήp–r”}›²χτ­²ί|\tξΎφ8Rm“g*Ž$m4›MŽY~“      €Χš››±jΥ*όλ_"Ν«―Ύš’X―ι\Aa.’XρΡ4ZB΅ϋιΑΥ4ΎΦ2¨w—+!q{[ υ1k·οxρEu轇XΔ~¦±•εεW‚ΨYqL‰4#JM›(ˆ:Tαq΅Š06Δ³ΙZΥγΠέ΅]((,Ċ+΄0φ°ΓCxψȎƒσͺ±^’(Φ H,2|ΒBόσb©‚³ŒXκέΧ;bu³bmςbp¨©οKΥ`›m¨Ux,_VUλ1P3Ε•`΄RKλΰρG«mά.  ψ;ŸxΙΩ ƒϊημ˜ΓηαΥχW:σšθλ,ώΉrέ«¨s7©?§Ÿp€sž$@$@C#pβ²E8nιώsς+Τ\užm?@°™ά’2ˆΣιέ{~πX˜9uh tIE5Ξ»ϊΟΨ–“?@‰―³Σ“&`ήΜI8υΈ₯ΘHIΐ‰?Ήφλ…L—Ή‹~}μϊKΞΥΒέϋŸ~ΥΩΊΧW¬ΒY'­ˆ:39A$@Š€λ½ ϊσ‰n=i€Ο۞Ϊο‹g[κ_•!%_°RƒX8 χύ˜/ξ{FγΰiηŽΕη̞ϊy <_1πΥvΤW.# @'ΠΧ)V_»»{†$μ”g6΅ Mn(Υ)vGnQ?—Φ.ύα ‚XG#b£"pϋUΰηΧίλΘ‚ά+HΆΤΔxgžλΔ΄ŒΌϋΤν:ζ³wΰM•νΐJ@Λn)-³μ½AΩ ˆ|€(Šσο7ξ½pΰώάχEPQ– "Θή{(έ{—υοΧο„œ6i6ISzΏΧrφψ5'$'η>Οʍ;dϋή8Ω8ANœ8}½F·φ­δΌ‘ύΤΝΩΖnδΌvΛnλEH„ƒΐibφP€ (@ P€ ώ'-kΧ•ŸώYWŠυχχ—ΌΌ<ϊΈY ''G>¬"±§€–oˆΤVΑX6gjImΊͺͺn+L‘ΤΤdΙΘΘΠ8\ƒΙζ>ό γΜΜLY³fόυΧ_’’œ$‘~΅€CC?]!Ά-Q—―²Qΐ­F0Άi0Ψ~rB ؟qB‡δΤ ’]»κg·nˆ ΞP¬ Ήη<ωeΎU]Œ-SΥ2³^Λsχ­₯w—v–ώŠtΤ«ΐEfYη‘@U—?0Š,­[X=Χ| ‰lbXR[έ"εγ™χͺΦk+w›‘εŸΙEsdεςK±‹(Ή Pΰ °˜‡Ότa…φτΕΏ3ΝΰοΗP¬I„= Κ/ΰηλ+SΖΥT†]±n«ό΅V=Τσζ]τ]εJ.5+'O.ΉσY .`lΥ¬δθrυŸrͺ‘ΏψH±ͺ[$ϋχοΧŒŒνΤ©“>ηκΝϋΓw-oώλΤπmσU?XΨkΗΥSG-=+ΧΡ$–ρυν„ςpaμsw_k™Ž €YΐV˜άxDF μαhVΛxά7E³ΧVn4Ÿ;Ϊ·›½I9œ (@ P€ ά$€ lrr²μέ»Wόόόά²άD ‘Ώ‚‚ŒuE`αέττtYΉr₯̞=[6oή,……… ΓΊε/Xz‘EE…ϊusμ˜*ξTΛO||C₯–ύμIι%Ττ!΅›Ί~_Uˆ,..ΗŽE6χ ΰ}#..NWˆ=tθ°Φ:)­ΓT 6άWκ¨—²Z=±ξύpιfΌή|}T06ψ”ͺλ/…ΗOΙΑμBΩ΅k—„……Iγƍ₯iΣ¦ζ™Ό¬‘X/ϋƒpsN Τ―knŽͺ―ΰn.;χ:½0]ΡMdν–έ₯¦Ϊ°}o©a@ œ(p==Ζ~WΙ /[6―\(cMψqyfηζƒdο‘£Υ6»c―σοa–f(@ P€ ͺB©9P?ž½λjΉϊ—tHΦΨtTιΐ‰ΰΚό5χUΖβτσΨ‘}ε'oά‘ΥQΛQwΑt¦•Ό˜³Όί2²rd֜ί,«ŠP7τΉxά0KɎώέ;X±ΖΈ‘{Κψdž νΣΉ‹δ‚sΞ’Aͺb,(@ "’u;’˜bέλTχς΅[œš’yβ}ΫΦΆ•χ½ΛpΧΉ-[ΫΗap—@y?ΉϋsOU½Ψς=ΣΞ3ΫΪGGΓt+«ύ΅n›)WΦ΄7°gG™σΫςR“NH‘„δ4‡‘:\<•—_Xj~_‰‰τξd|©ζ ”CaςœΌ γΤ\GU%’-»˜¦Ερηͺ†‹³¬τE°τ¬ή]œ^<>pΎ2kŽ₯RΎpOΏ ˜*ό|ΝΝ’ͺ@rZ¦4nPίαβ–­ΩμpN@ P ¦ |ωςƒκΔ¨ΊΫ[9ΫuΏ"ΫχΖYζzγΡ[τ…CΖ€Κ„°Œeπ™ @Mψ{ύ6YΊj£e·Ην'=ΥE™ŽZݐ`ωόΕϋ₯νΘ+₯ ¨XOŽέμ?œPξ "ΖΊπΉ~η>σMf¦ž?ά©@,–qθh²±¨2ŸΫ¨οΦ Ϋο;]b[ZΆΫ=ι*y杯,㯟r^™‘XΛ„%:žž~΅ΆΟUί͌vηΣo˟_ΌTαο2ΖrψL œΈHήΊ‘jSfvΤ―«ξώλDCε¨έŽ81eΕ&ρΤϋΆ­­σΆs[ΆΆ‘Γ(଀;?ΉσsOUΎΨ³υΆσΜφΆΣΓ=eΰ©υΈΣŠΛ¦(P՝;Δτϋθί%BŽΆoΩjσo’ηœΥΗΡ,ΗγΖΦ-Ώ°Hπ=€es篝(Tη‰βJœ£ιΤ6Ζz±rΕ½3%;ητΝΝ^}ψ&§―ΟψγŸM’š‘mY^Ÿ±δάoή–™ΨA P€ (@ P€•ΐ5ZǏΧιφμΩS©g;Ϊœ‹ΖΊ*Σ°Œ£GΚΪ΅ku₯ΗυλΧλ[xxΈΰΐ[RRReVΑy@UήΤΤTδ¬νοo(V½žΨœΐMκ}T₯Ψ`Šυ“c…ΉΊκ1ͺξzsπΝ™=σζip€M›6Ιϊυλ€@›†Τ–XˆmTKύ=X!Φ›v5bΫΤkΠ·v-iμ#±α~’w¬XRUX7­ˆŽŽ–N:ΉυθΚϋVffΞKw Χ ʎΩψϊσ«εήλ¦ξf^²!twσγ―•\fΰ^νŽGhβ‡·ΧΫbk"\6όŠ{?Κ”l“F–χfάYr0ϋ)pΖWw4Z΄bΰ΅ξL{σ‹MΗtSU©½ΈͺuhέΒτ£οG³•k/:Χι‹ίŽ·Ύ(½«Ί˜ύ‘›¦Vxσ†™πΕ‚p‘όˆ=Κ\&Ύ8τϋ?eNΓ‘ ( εΊ€ΖΪ+0ΐίΊW’"Kl«(Σ0φP€ @ω2TΈκՏΏ·ΜtμΨq§B±˜οΛΪD *Δ σW΄%$§Kn‰WukίΪιΕΝ]ΌΒ©i›5W7Ρ W7ΣJ·Lќ_ε•o²τ—Υργ’ΏM£G:ψž`šΨͺ§Y£pyπ†KδΑ—?² E¨ψωχΏ•Ηn½ά2Œ @Νθ¬.Η{­qώΟ|»@ξΎφB§Pήωκ'§¦«θDžzίΆ΅}ήvnΛΦ6rœpηη1w~ξ©Κχ{ΆήvžΩήvΊsΈ§ <΅wZqΩ ͺZ`βθAςπ«³T%š“zSvξ?,+ΤΝΛυt\υtύφ½²aΗ>Λ.„κŠ[T°£UT³RsX·΅\ητw¨žYW(A₯ΩΨVΝMΛMS‘Φ6ν΄ [Ίr£\=ωKYŸύΈΨ4zόπ¦~φP€ (@ P€ €gπέ\7λξVΩB ΨF„ΪfϞ-ΫΆmΣ›)Γ‡—ϊυλΛoΏύ¦+ήΊ{?Έ|Ρ†333U‘XU)Ά–Ώ wςFWεz]Τͺ->΅U˜ΈΆŸ—S’››£MΛ΅ Nμ΄ή;ΣΣeΓ† :8δλ#­κϋIdˆψ1λ΄#'t―ώ£^›QuER jΛΞΤγ'λΦ­SΑΨ uξ¦λξέΚKχ)=ˆC(ΰ=ΫΕΨܘ#‰)rɝΟHJz–e|‘ͺΆRέυtΚ3$15Γ2ά™„ςΞκm;»jγ™ώΜ;–‹Ζ¬—‡κ1Χ>τ²Νq˜ξηl§ζ΅7Q€ΏŸΤ 2ήΌkΏ©ίVΟwΏ.“-»ΝumMΗa (@ P€πژ6εwUmΓΩ­N¨›νμ>xΔ2P`€΄‰΄τ—·#Ό~¨ΰBIλŸ˜jέk·ϋΟ՛ԍΦΫo=?˜= Β¨Φmφ‚e’ž™c=ΘfχοͺͺξŸV•Xκ¨°Ϊ@'.V΅Ή05πš Ο‘nν[™FΏωΕ<Ω΄ΣρχΣLμ‘ΞH_ίΪ2aΤ ΣΎ½ϋΝOκ§+L›FZυlPΙϚσ›ΥΧwzκ}Ϋή–{ΣΉ-{ΫΘαpFΐŸΗάωΉ§ͺίlΩzΫyf[Ϋθξaž2πΤzάνΕεS€¨Jά€wXΏξ¦MxAέ(ΛΡ‚©·Ή@IDATy\_ρψ?1Ν7r`/ρχσ3 «HOίνKέΰόι·Ώ”¬œ<§‡mŸρΦ¦i;ΆŽ.΅mΤMΦ¬Ϋς΅[¬{νv#4<ι*Λxάδς #-ύμ (@ P€ (@χ ψͺ’Y“ΆlΩRZ΅j₯­[·w?°.¬„XΛΫπ}υΘ‘#ΊZlXX˜ 8P¦N*“'O–]»ŠΏͺXΚζTέΜΞΞ–“X―Τς τ̊Ϙ΅¨j±ΚMj¨sH“ΌΌ<‡η“Ξ˜έχπŽΰΊ¨ΘώύϋεΔΙ8ΘG’λϊJ€ϊ ˆΘVyŠΌ!–ΗPZ"ΤOΒ}Τ5%y‚JξρρGΌφύAFlπ^1ƒ{›ͺ>ZoικΝ;₯Γ9WKLdi^O6« =‹+QIζΎλ/–εkΆ^…₯ϋ«Ÿ~—ů׏΅Vw5Εzpgοώ%ψ±ΘVλ₯œ;€­QF3Jˆξ™ωΎͺ†t£έŠ¬¨ΰ|ι]ΟJUEeT(ΉνΚ‰.΅h!Σ.=_ώϋΙ–ε>όκΗΣΌ©tο`Ώ2T~a‘Ϊ‡χAw£!ΜZΩP,–…υ.[sϊΨ·Ύœ'ηœέGp|[ ?ΒήϋόϋΆFq˜ άψθ«²dεΛΥ Ν?½gι/«γ“–ϊ!Λ—”>]c˚M;Ÿ(£ΊΟ4έ-—MΫ]|,™Vΐ P€ (ΰ„@D“TU,MLω·jκφ½qς‰Ί)ΞU“F;œίΉsσN³ΊΔΆT‘ΦΪ₯ζ«£Β²Φ λN%οζŠοm£#•QŒ6λϋί€w—vF―Νη₯κσέΥΎTj\Y‘^|ήPyK…Ouα;Ο€[—ο^{T©sΆZ’Ί‘Χm3ή4> ‡βhXŽx½x 2ϊ„ΔέtoWλYόρσ‚@(P³nΊdœ|­ήo–‘•+ozL>}ρ~AΕi[mΝζ]2υgLΥ™lMWΩaž|ίΆ΅­ήtnΛΦφqœpχη1w}ξ©κχ[ΎήxžΩΦvΊs˜§ <΅wZqΩ ΌAΰ’±Cτυ ΖΆό₯ͺ²>ωΖηςΨ­—ƒLΟy…rωέΟΙίκ¦γFσS$ί{ύEFo©ηŒ¬vω=¦α“Ο9Kžv©izpbΚΨ‘ςή7?[Ζα†η·<ρΊΌρΨ­₯³–‰TΞΓΜ|οΣΝΔ0ώΡ[.³žLwχθΨFΔκ&>Ώ-_«FμYjZcͺγήππ+F―~Ύϋš +u^Ζ΄0φP€ (@ P€ €CΠΠPΉΰ‚ •>}|Μ7Όv8s%'01θΤ©SΉ—„kš4i"gu–DEEI―^½,Ϋ¬¬Σ…ΎΚ½`ΞPn’’’+›"ΡYKέΰ«bQ*Νň‘σ–κυ\ΛG…‰ΥσργΗ$??ίωy9eΉ`‹@lFF† Βͺ@l=?©z΅†bΛEYjβ“'OIα‰S*l\KόΤ₯Q«Wv©ι8ΐ9Ό}Υ{B£ΰZκ£*ƞ”„„]1ΆeΛ– ςήΦŠυΆΏ·Η$pή°~ςά{_›†•μ9Ÿ$x”lυBƒΎΣ(ζνί½ƒάpρyςξΧ§˜±^&~€y›_¬ΩνΖEο>uG©;•Ϊ#(PΝΎ˜·Dβ“RεΎλ¦˜‚}¨8²bέ6UmωmIJΛ4νεu•¦ ΓLΓ\Ρ3ύκΙςν‚?%!ωί‹ρq‘όΈλ–™χ\+9PB‚κXVƒ;―¬Ω²[ξ˜ρ– ͺ¬u{ύ‘[€qƒςί ΚzθΎ`ΜY¦P,.:½ΰ–'τΠ}ΊΔJ+΄GΈ~o\Ό€ϋُ‹-Θ& ΅kΙe²Ώκrr K£Ÿ‹ŠŽ™ζŠǎw8&8qβd©y T¨› (@ xƒΎά©>ϋνξηήUŸ·wΙsw]#‘κ¦3%ͺƒ|4ϋWy杯L£nTΑ-[­yΣ†¦ΑΩΉωr ΘκLΊp7Y#Ψ5ΈwgKP3}σσR_'Έοϊ)R7$Ψ²όΰ…Σ|§Ύ―TΆαBΠ«'Ÿ#~·ΐ²(lΛŸΟ•;Ίΐ2Œ @ΝθΤ6FίPΙϊFbvμ“‘Wά#·^1Qzuj+ΫFKVvž>Gςεό%2wΡίr\;ACoάTΜ]ΝSοΫφΆί›ΞmΩΫF§€3ξό<ζΞΟ=Uύ`ΛΦΫΞ3ΫΪFw󔁧Φγn/.Ÿ @U LPΏ~ͺnN†0¬Ρ^lΊŽ"QΊιRΑMΠR3²d₯ ΒΎ©nπ΅VύFjέ¦]:^ΪΕ4·dκ>‘Ξ•QηH¬›υω λαθΖ9˜9Ώ-—΄ΜlΛ¨ΛΦΘΐ‹n“'οΈJ†τν*¨rk4ό^ΊjγvyκΝ/d£ϊbέ.7L†υ7WΓΕψ Ο"οϋ‹lήu@OŽ››_~χL}­ΖωΓϋ—Ί‘Ϊ—σ— ΞWYίl½]Λζrιω#¬WΗn P€ (@ P€p³@``  χTΥ{@YΎήvžΉ¬muΧ8Oxj=ξrβr)@ xƒΞ/Όύδν2dκtIW]6ι*Α7―ͺΞM$£LΟέ;΄–»™lVΩ¬σσο— oJrσ ,‹ΓwyUχΧU7QΓo·κζa8ίc}±¨1~{ΖuΆ*Ύώθ-2βŠ{-7τΑh―~ΰE «"ύ»uΨVQ²]ύf·YίπΛΓ ŠΏ{ν]ΩΦΦς9Œ (@ P€ ά#€€ 7kν-Ά;22²ΤM˜ΝWΦx|ΖuΦΆΎ[Οg}-6 ǝ,Δb½Œ3₯V*€uςδ ε&rς”ς(ΖωTVͺ¨RΥ)O(ΠΧ‡`ΓκΤ©#!!!ϊo•››«‚§ΗΤΰrςXžœςQ‘ll|U5υZΦΫϊΏΧ^SΕbkΧViM6— ΐ699YΣQμ«–„¨?ύ±S'%³°j_―.ΫΑ*ZΠ U!6!χ„μS‡~a­@ρχ”"u=Zb^±Δ†Χ–π:¨‹π7[Eτ{«ͺ'θ[K…bKjjͺ~ Χ­[·"‹³ΜƒγΑ™σ,38ΡQώ[\8±PNBW yώ^Σ]PšQM„«οTRΤ³“1ΘτόΪΓΣΔO]ˆωΩάΕ¦αθA@ΦV #Oέ~•œΥ»‹­Ρ†ŠΌ¨P{νC/[ζG¨χΞ§ί–ωο>εη,+`(Pmp‘<Ξ}ŽSAω{fΎ')ιY·ύ,Uyϋ½Συ Ώ¬'θoέ[ιnOΏoμ­ηΆŒνγ3Κ+ΰ‰ΟcξψάSUοŽ|½ι<³£muΧxOxj=ξrβr)@ xƒ@³Fα2ηΗδŠ{gšn(†mΛR„αa«α3§/ά/Έ‘―«[usΉo=!χ¨κ¬JTΕΊ²sσeΣΞύvW;|@wyιώt¨ΧήDΪΖΘ‹χ_/Ύτ‘Ύ‘³1°¨L‹‡­Φ(Όž|ζc:kk<‡Q€ (@ P€ l ΰ<Ό+[VV–!KB}Ž«Χ±Ju²UH‘ΨΌc§$#ηΈδΩΏΏόσΟ?Ίβq…ψΏ™πήrθΠ!]u7w(..τϋ C±•ω‹p^ΤQ?Ύ|ύκCςφ—σε½o~±T ’λ/+w^u¨‹²ΦU|σƏ £υ”Χ>+―6·Tεš’Kπχ“«'Ÿ#ΣοuΗΡΠ’£ΩO3Z`ψ€ςσϋO달·μ>`w_[¨κ±ίz…ΰψςDC5Ϊε_½"OΎρΉΜ]ΌΒα*{¨; ? ͺIa\έΊΖΆ’ε_Ύ,w>σŽ¦koω¨N#T\@³Ύ³“½y8œ (@ P€ή$ΠFU„Εηπ7>ϋQ^ž5Ηαχil;ͺryAξyώ=ωιχUκBYϋ~Υ―"Χ\ˆk& wpWCΪEŸ<―*Φ”gήώJφΔΕ;\ͺΌβ¦_;Δᴘΰ2υ½ehΏnςΐ‹Ϊ Α ςχσ•«&‘;Υ΅Ζ²Q€ (@ P€ ͺR 11QζΝ›§Gφ·Žπ, ,ώρΗ:dW“ƒ‹™™ͺ€S’ tͺͺƒR+GjωΚ?£eέ~ͺͺbέΠ iΤ¨‘τμΩSΊvνͺn›"‰j‘)©κzΛ,UΪqκδqU±φ˜ egΛ/Ώό"υκρ\‰« 8NρzMKKΣΐl•έ>’„΅TuˆΫΥ{κ™ε‘Ύ² I½π-6J¨oΠ΄IIII–ν[“₯ 'KίHΐ3[uζ­Ξ*­*Εκ½κ€z­’:7ήΏ‚ƒ½’{Œγ!//OX*TWΆΥR ΥΫλΜ‚~ύw>|Έe†aueηo³,ύμ €»πr]·uμ9xDU’ΜPNIλ‚‹<π¬Γ°.ވμά<ΩΎχΰⰝκ±λΐFo ^ΘΩ}ΊΚ`uχTwά9ΥΕ»ΒΕ•!ΠkΒ4‰;ͺ?aθ©>ι~ΑΙΩΜί-X&7=φ_Λΐσ‡χ—YΟέcιGε€_ώ\­ο>œ”š‘/nmΥ\Ϊ·ŠRAΟήnύAΥ²6:Ζ'Κ‚?Χθ =“U%[Tm0LΓΡ*¬‹‹AHυD;š”& W¬ΣF)ι™‚œΫ¨χ―Ά1‘κ‚ΣXύ…ΥΫq&£ν¨+w}6*a θΡΡθεsήύϊ'yθεӟχπξœΝF P€žrιtΩΆ'Ξ²wžΌ]&ŸsΆ₯ŸπFΓ Ι*XͺξvŸ$qϊ‘¨ΎΛgJD“2²©DG6‘V-šΙωκσ8nˆUž†οκ{γŽΚ#‰:dΪ΄Q˜ 2HpΫwΣ\€>ƒoV•Gvξ?¬nΆ•*ΤΣίϊBeX[λ?¦ξH·rΓU±dŸώ!₯ED]ΥΥ™ &7οΪ/KWnxυω?-3[š6 ΧϋΪ:ͺ™ήNT₯e£@MxI2p7ΪΨ‘}εΣηο3zωμAγΗOΘnuΎ5=+G…Υ“¦ͺΒ”q“c3n|τU™ύλr£W–©›tlmιwu‡»ή·«λΉ-Wϋryf/ζ-‘ΫgΌeΨ³S[Y8λ9Kuξpηη1ΓΕŸ{άυ`lsEŸ½ιΩ{φ¦σΜφΆΡέΓ=eΰ©υΈΫ‹Λ§(P•ΝΤΝnξΈκ‚ͺά„RλF5›Šό?_jAe ΰ΅eΰp(@ P€ (@ xnzvήyηΙ€t Ρή«pBD>>>:t‰0QMo hΉΪη?πwB@zΥͺUΊϊκ!CdάΈqrμΨ1― Z‡6]mΒε!Θιš@!-€'Ž³μμlY²d‰>|XΊtι"ƒ UšOͺrΜ|M»φΥβ O,T>Ηί°G•.(ΗP¬kΞ\(@ P€ (@ P€ (@ Pΐ«PύΪϊG ύα-7Ώ@ΆZU¬G Φ™ͺΩ^ΐ‘(@ P€ (@ P€ (@ P€ €— 4hΠ@†ͺCBφ6 ΏιA"t;ϋŸ½εqΈϋϊJNN–?ώψC‡b»wο.ύϋχ/σομώ­β(pfο‰x_DeΨ+VH^^žΰύtψπαRΏ~}Σ΅gΖ^Ÿ9{a\·‚Ώ#‚ΝΎκΪ•Κ΄ΚΝ]™5s^ P€ (@ P€ (@ P€ ά.0ω–'eΕϊm–υ²`ΩΛJίψόGωιχUrΙΈaΣΌ©4oΪPςς %15C–ά(Ώ._#…EΕ–ιόύδύwŠŸ/V° °ƒ (@ P€ (@ P€ (@ P€ (@ PΐγΌzΕγδ\!(@ P€ (@ P€ (@ Pΐs΅jΥ’χŸž.SnŸ!+Φo³¬ψ`|’<ϋξΧ–~{ώςΪ#7KΗ6Ρφ&αp P€ (@ P€ (@ P€ (@ P€ (@ xDΐΗ#kαJ(@ P€ (@ P€ (@ P€¨2[?ιΉrβ(A·³ AΨ%Ÿώώ7Y²rƒNH‘ΔΤ 9yς€ήψ ΐiΪ(\†υλ&UΆ_·φ‚J³gBγΉ­3α―Θ} (@ P€ (@ P€ (@ P€ €χ ΰ7Vγαύ[Λ-€@υΐ1ζγγ£ΥsΈΥ•`(Ά2zœ— @ θή‘΅ΰΑF P€ (@ P€ (P}Βλ‡Κτ«'λφβψρ’’ž)‘!AT§ϊξ˜ƒ-ηΉ-@M P€ (@ P€ (@ P€ (PaV+¬0έ5#Bz7–vνb₯aΓ|Fνw†ή €γΜίί_ZΆl)………¬Y3©]»Ά7l·Αƒ Εz›«’(@ P€ (@ P€ (@ x›€―omiΦΈ·m·‡ (@ P€ (@ P€ (@ P€^/€p–ŸŸŸJ@@ƒY^sο"˜Χ±cG†Ελ!""Β½+δ)PπΎ[―^=9r€τξέ[gxf«Y ΕΦ¬Ώ7χ– (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ Pΐ5’Ρ£Gλ $B-Z΄pΑRΉˆκ*€°^TT”ι‘›Υ+«λ_’Ϋνν¨Βά΅kW9qβ„>Ξ|}‘τφΏ™«·qW‹ry (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€gΌ@“&Mdόψρrόψq½―j±ΉGΰδΙ“’““#ΕΕΕrκΤ© ―•}ρwςχχ―π2ʚ±¨¨H tXλ φΚγΈκ"pμΨ1} οwΨnTx ͺΠ1‚γŠΗ–+ώ Υs ΕVΟΏ·š (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P  P 4$$€ · ζ¬:??_.\(ϋχοΧ‘XTb­H8Aζ‘C‡JΛ–-]Ž‡ͺ•Ϋ·o—•+Wκ`l³fΝdάΈqζςuqπ€Ž΅””™?ΎdddΨ]5ŽKλVΦ1Š@k«V­dȐ! AλΩΨM‡ Ε:$β (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ @U ΚςεΛeŊ‚ͺ±>>>R2€ηΜΆΕΖΖJǎέŠΕvνάΉSζΝ›'ιιιz=ό1λΜ_†Σx³Β­xM/X°@:€?[Ϋ‹ιp αψΔqj―‘’ςΰΑƒ₯{χξ ΕΪCβp» ΕΪ₯α P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P ͺΆ+**X?.αααΊuλrc±Œ˜˜ vΫξ;vLo#BΌΨ^# θΆrΑπ€uκΤΡ•]ƒ‚‚l†]ρZOKK“ψψxΑ1(‘‘‘R―^=›[ˆPlDD„Ψρ(K \‘X”πΆnιY9{β4λAμ¦(P-β“RMΫ=ύ™wδαΐY¦aμ‘Ό[ +'Ο΄Χ=ό²ϊϋ›†±§όΩΉω¦™VoήΙΟ&φP€πŒ@|ršiEΏς±<χξΧ¦aμ‘(@ T'ΜίαώX΅‰ί5ͺӐΫJ3H 'Ώΐ΄7[wδϋ‘I„= €§ς ‹L«JJJ2υ³‡ΦΈΙ6βŠ{€vmϋwά/9=ϋ)@ Pΰ΄@JzΦιΥ5kΞo2wΡ Σ0φP€ (@ͺ8yςT©Υ&&&κ0W©@ P † κ$'Nœθθh™6mšΕ"μκl ‘ζΝ›;;yΉ§3ͺcΫZ‘jΆε^)g €›π:nάΈ‰\vΩe’ŸŸo3ŒŽ μί-³gτÆ7nœtξάΩζΦαAε°°p›γ9e ”+‹;X7|Π>Ογ¬MΨM œΙi™gƎp/(PƒS2jπή»oΧ ‹ŠωωΟ}Ό\2(@§R3²6 P€ ΐ™"€Ο5Ÿ)Mξͺ·@±ϊq–οGΥϋoΘ­§ΐ™"€ ΊΨ(`OΐΦλγpbŠ½Ι9œ Κ)€2—Ό)s9ΑΙ)@ P€ €Ϋ6a£(@ΡΥW¦0`€ΝŠ•4’\/"={φ΄»`δqΪ΅k ‚κ¨ΫΎ}{4hέy8‚(W(Ά’+α| (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ \!€Π]y*Δ:»NcΉΆ–mTuvYΞLgk}XρpfΦΣΈzyΦΛf7ΚΐkοδΙ“–IJφ[FΨθ(9-ͺΘβ°nΖpŒΓ Γ0OY ΫVžι±,cΖ|f½θg«†b«Ζk₯(@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ Ό@Ήόό|III‘¬¬,ΙΝΝ•cǎ鐟―――JέΊ‘Έqcυ\Wόύύ+΅Υ¨ώυ`}™™™RXX('Nœ¬«N@A%ά† Jhh¨ζheX^NNŽ$''KFF†^‚|¨Ϊ’—.AAAΓƒŽΦΕρp₯^§8ήvμΨ‘ ΌN;t蠎‡’””€«Οβυ,Χ¬Y3Vέ΅k·€₯₯‰ŸŸ―4oή\ZΆl©_οΆΆ ΗΧ‘C‡$>>^ΣXFll¬έιρ~PPP O¬#//O° 8žκΤ©£ΟFιγΓΨ·Μά&Ά“\{Ϋƒ–~vP€ €gώϋΜ’Θ²²)WM“}YϊΩA P€¨nKό ‹~šmΩμN]{Λε7άiιg(@O ¬ωϋ™σΕϋ–ΥEΕ΄–›οyΟ P€žΨ΅m“ΜzλyΛκpQμ Τ«W―Τ¨GfΎ#Α!‘₯†s(@ 8ψzΦ²qνJΛ„Cnj—sΖO±τ³ƒ (@ T•@qq‘FŒPl»vνdβΔ‰ϊxόι§ω²aΓ}ΌŒ3F’’’l†\±|_+V¬_ύU‡b‡*­[·Ά9=ŽϋΈΈ8ύ~°{χn9|ψ°~?(..ΦaxγψΔό={φlS`` »xΈ\;ε Ε–όaΕOέΥ`π°sμ,šƒ)@ TΧκ™6XΎΏ™HΨC―πσσ—9Šν£Ÿtλ=ΐλ·ΫΫ7πθღM¬ސο&φP€πŒΐ―=cZQlΗn|?6‰°‡ ͺ›ΐޝ[M›ά Qώίfa(ΰ)ΜτTΣͺBλΦηϋ‘I„= €§|}ύL«βΕ&φ”°U’Y#€^}†©KP±— €SK~ωή4]Ttk~/0‰°‡ (@ͺ(,Θ/΅jž3(EΒ *-€άͺU«dΑ‚²~ύz]±Γ@υρρΡAU„UΡPΕUbΧ­['ωΟdĈ:¬Zž@ σϋν·‚+ͺΕbωX*N’:-֍uΥ―__ξ2dˆΈ–\Β»όσΜ;WΆmΫ¦Γ{FEKLk,a[„pϋχο/ηŸΎΘbl¨j„VqΜ! Ύ}ϋv}L!4ώύχίΛƍu ۈΧ0ͺ!6L¨όŠκ²¨¦ά½{w]=ΦήΎ Њ*ΚX.Ž YQ ΆdΓvΰ}`αΒ…ϊΈJMMΜ‹mΔρ‚gPΉΉ¨¨PW…ΕqˆcΎY³f:,Žn„Θ1½ρμh+Œy0­Χ=Β―Ν"°Žγ Υ–q\φκΥK"""$$$D Η€QAφχί—ŒŒ ”νΧ―ŸZλh;8ή5 ΕΊΖ‘K‘(@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ ά,€p*ͺ"ΰ‰ng‚suκΤΡΑScz„ήώψγ]ΉαΡ6mΪΘE]$·! ‡jγa:b—,Y"‹-!8„βΪCΕI„OiXͺΓY³F‡λΆ›4ι]ε2<"ΈZVCΰ­I“&Ώt3¦EΕΗƒκ^„Dǎ+ηž{„……ι@¬1‚£½!T‹J’γ"(»oί>Ξu&‹mΜΝΝΥgρŒ}θ₯‹œ}φYϋυ! ˆ@.B΄˜AX„[QΙΦΕ"΄χηŸκ-‚zέΊu“©S§Jοή½u΅JΜo4l?*lb;Q7>>^‡{τθ‘M°-lπ#„Žc Ηβ!C$**JΛβ8 ΦYWn/Ž­7θ ±8ξPvΒ„ 2bΔKΥec}x/ΐϋŽ[T‰Ε{Βƍu΅ιU}9((Ș”Οn8ύηΖ•pΡ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ *+ΰηη' ςυΧ_;΅(T_νΠ‘ƒΊ‘ϊ£ΡP]΅S§N:€Š`θθΡ£uΨ­d₯Gτ#θΦΊuk] rλΦ­ΊbdvvΆ pλlCxn/‚΅ΰ…„„θ°«±^iΩ²₯Œ?^―λnΥͺ•₯Κ-ζΗώ―[·Nt;ϋμ³eΐ€6yΓ"δ7lΨ0ΑΆ#P'¨r‰ͺ² ²QΐŒc―ΧqγΖIll¬εua8fέΡTGΕΧύϋχλγ!σ‘C‡Jddd©JΚœ#‹:ͺE£ϊ2ͺΗ"‹χ†bέρ*½L†bK›p(@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (ΰ…’"ΘΩΌyσR5[›‹0jΣ¦MK…Υ5j€ClύϊυΣαΦ-Z˜–‡υ Bkaa‘š””€C­ζ!Όgp΅΅^λaΌ"όŠ`B½˜wύϊυ‚*ΰaέθF .88X‡d;vμ¨+Ϋ"„‡ΠQΡ!ί={φHbb’η’zlγƍuΥΚ΄΄4λ՚Ί±/aa tȁ\TΈD…L†bMLμ©b„ΒQqtΝ¨~\ž Ί1oYΟ8žP…φΠ‘CΊ4*ΐβ8Ε±Ή½†ρΗβ˜ΕqwτθQ½Te6ξφζεπΚ 0[yC. (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€π€B‘N›6Ν©΅!ΐͺCvΦ3 LŠJx ‘ZdVV–θ@ϊ322$))YPthAR,‘·ς†σ° ¨|ΩΎ}{]UU)Qmv͚5‚ ]tt΄ήFT…EΨ!Z=ьP,Ž4„]7oή¬#ˆnk;p"˜ŽιΡ𾁊±^Φ|Ά–Εaε`(Άόfœƒ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€¨„Ψ4h C† )-οζ ΐ†*‘‡–}ϋφIBB‚dffκ°*£x ΑS„ίπ@pΑς6TˆmםLš4IτϊΔΓϊΒΓ>‘z-²"ψΫ²eK–E₯[£R&֝žž+Ψb°K–,qͺ:%φX4TΐEΕX#δ«ς T±*γXπT΅U8ƍcαVTqήΈq£C ;›£a~ΥρώΔP¬CΊJOΐPl₯ Ή P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ PΐS£αQ™†ωQquαΒ…ΊZλ‘#Gt¨ α6„ΪP©=TkEHUwνΪe©ΆZ‘u‡……Ιˆ#€I“&z{χξU•h“tUJσvοή-{φμΡλΕ΄ΓφκΥKƌ£²ώώώz΅΄"|‡*²¨@ΫΊukΑ8gLŒi°O˜ΗSαÊxqžš%€Χ"^Σ8ξ\ٌ »ρΪ/ΉlΕnΕx¬Η*Φ:jX.B΅xFuηππpOŽΠ\4ž‘XAr1 (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ @υHMMՁΨ~ψAŽ=ͺ«·6oή\VQ©‘T£z+Β©Μ}ςΙ':[™=DΈΆoίΎΊlJJŠ Œ'ΤUcQ6--Mβγγυvαλž8q’‹u£j¬τCΐφςΛ/ΧUfν…ώlm/Βς±Qΐ›πΊF0Οj8.Œ ΙΆ–‰γ Ax¬!ψ &θͺΞΞn¦C%hλŠΞΆΦΓa`(Φu–\(@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (@ P€ (P V―^-‹-ΥbQvτθΡΏA06((ΘR)έxdddθ0jy‚§Φ ˜U%ρŒΠ_dds½ΆmΫJVV–~ Z,B°ϋχο—mΫΆΙ–-[t0φχίL₯ƒ{υκΥΣΫ‡ͺΆδEGGKϋφν­WWͺλ-.F5Μ“:ˆmΐƒή&ΰL՘Ζ8ΚΪTU.((ΠΗ^Ιιp  *l@@€…~_]Ίt± +9Ρκ²FΕf„j1―±]Ζ4|vC±ξqεR)@ P€ (@ P€ (@ P€ (@ ΈU +3]Ψ+©Ι R'(DϊŸ5Β­λγΒ)@ P€ (@ P€ (@ P€gŠ‚tόσμΫ·OΩZ΄h!S§N•V­Zι°iΙύΔτΥ!΄Š ͺB–§a~„jQ ΟΠuκΤIWw <ΖΕtyyyz=νb}{χξΥ•d8 «]"x‡p,‚|F˜!ΪvνΪ•rE€vγΖMjYUbQΥ²{χξβοο_ž]ᴨrF‘T“8vl5 Ηk?))Ι`΅žΛjΤ¨‘Τ­[W/Σ">pΰΐ2C±¨<»gΟ=-Ά›cccuuic­…έΣΝP¬{\=ΊΤOί}Y>ΰ5·¬³I³Hωlή ·,ΫΡB/ΩMR–šμš[ξ—+oΌ«Τp (ΰI+'œ%ΉΉΩ–U~πν ohιg(@ x―@qq‘ιsfΓ&ΝΤ—Φ@οέ`n(@ ΤHΔ£Gδζ+Ξ³μ{Χύδ±ή³τ³ƒ ΌS  ?OΖτ±l\νΪΎςϋ¦K?;(@ͺ8“Ξιβφ^{Fζ}χ©dgeXP}ΥΕ7lJ΄τ³ƒ (`ΰ__οΊDŸ}Τ…bξ>?ŸŸ—#)I ’.,­ζφυζ|¦(@š+p\]ˆž–,ώκ7θϊa j.χœ (@ T„θ’““uΠΡf͚IλΦ­ν†ΰŠ‹‹eϋφ풐ποοΪ‘κ«³ Σ9rDΎώ{€Γ:―Ήζ C(ΟhΣ…„„θΚ΄;vΤΫ…P,ΞρκPB¬ΐ"Θ‡νΙΙΙ‘΅kΧJίΎ}₯aCΫΧ΄cύ‡’ωσηΙͺU«τ:† "XC±†>Ÿ«ƒŽ¦PΑAvΌ/ β3›gŠυŒ3ΧB P€ (@ xΐ’sMX/Ψ$n(@ 8)··|ψζLΛԍ›DΘ-χ>eιg(@ P€ j‚ΐ_}Δ@lMψCs)@ TP`žςΦ‹ΙͺεKJ-ΥΕρΨ»k›ό6ο[‰Šn->ŽΫƒ‘ρ‡Κγw_';Ά¬/΅M€Π*Ϋ6―Υa’ΟίU¦^s›LΊδj ¬ds{ωαKωπη$)!ήή$:(‹ πόٟΙΘ±“䁧_gυX»ZA P€Ž~™ϋ•Ό2γ~ΛΝJN{q³ γΖF³?_zζMιήg`ΙIΩO P€ (Pώώ~-uλΦΥ•VγββtΥώύϋλJ©"LŠ°,±+V¬Π!VT₯Dψ Wt[ί4·¬έ@πU${τθ![ΆlΡ•-ϊλ/A%JTxΕ8ύΈCEΨƒΚ₯Kυ3*YΆmΫVW‡5Β³Ι6LPEU-ό›3gŽπφμΩSοƒŸŸŸή‡€€$YΏ~,X°@RSSup―W―^‚}΅U³¬ύΰ8 T΅Ž%ΌώΫ·o―Γ¬š£ μ/Ώό’Ι¨¨(–-,,RΥ‘γtεΏώ[ΐ­«2[οŽ½=zι*Ξ8Fpά/^ΌD?Ηeπΰ³!tT¦5ήΆmΫ¦Cς¨ΎŒyq|8P­—Λnχ 0λ>[.™ (@ P€π"ΔψΓςΒγwzΡqS(@ P <™ιi²Tέάΐh1­cŠ50ψL P€ (Pc–-ώΩ΄―Α!‘rΑ₯Χ ͺή±Q€ @ΝΨ³s«άzεω’›“νΔαΈ}2ν²σdϊ#Ο«ͺu—;5Oy'ZΏϊ/ΉoΪ₯vƒBΆ–—™‘¦ƒ½?…ΌΝ"§«ιΝzλyˆ=}C5[Λ.9lρ/ίKFzͺΌςΑ}!kΙρμ§(@ Ψ@Hα©ϋo’…σΏ³7‰Να ρ‡δΦ«ΖΛ…—ί(ΣξzL|U@ (@ P κ¬0` μΨ±CFΧλͺ¬³gΟ–νΫ·[ͺ¨Φš––¦ƒr ϊΒ«Ν‘:+ͺPζεε銫ŽφλkΠ Ξ!d‹Κ•¨09wξ\ΩΊu«DFFκ°*BΆ999ΊZ%―Θbά¨Q£tΛACP•bǍ§ƒ΄ΖbΉ™™™ΊΪ%ͺΟΦ©SGŠ‹©ΐμaΩ΅k— ‹ ,*bN˜0AW©΅t΄?Oͺ@(Άcǎ‚8Ž%TU^²d‰ΔΗΗλk@@ :Ž²uXY„Ώϋυλ§ΓνΆ*<γxŠŽŽ•–qœ¬]»VŽ—ŸώEΟχ©eFοψ<‘«EγΨ<¨‚λψ~ˆjΟ¨Πά»woSEΩͺτ© λf(Ά&ό•Ή (@ P€¨αΈ3Σ“χέΰτΕ@5œ‹»O P€ (@ P€ €— $ΔΗ™ΆμG_”Ργ&›†±‡ jž*ŸNΏnr©sΰAΑ‘Ή{oiά4R²2Σ•dγ°;V,3½CšD4—Ύ‡Y†»’#-%I»λZ›Ψ:A!ΆCgiή’• t@mΒ°Φ-nnu£ΛιςΨ οYΆΩ=χλY6±Q1«ίYCW9rhΏμήΎY6oψGN?nYΞΊUΛ\wƒ-+b(@ œQŸ½χŠέ@lDσhιΤ½ ψKάΎέ²οNΣ‡Έ`ϊΫOί– Λ₯ΧήvFΉpg(@ P€ €'p- ͺ³"Τ†ξΚΆξέ»ιP)yληύ@IDAT¨=zTGCxU|†C?­¨ςŠΚ²σζΝΣαSρP)²Y³fzlζΑφλmΔ2[·n-“'OΦU+ΌCθ•.NEX„^Π*PΆkΧNbGŽ©«ΏZ//$$D†ͺΓ―ΐ’zezzΊ,[ΆL/Ϋml–&;wΦϋά§Oψ¬1ΩνγυfΌώqό8Σ0Ÿq|γύe5ΌΆ#""dβΔ‰ϊ=Αr¨»zυju<βX}L!TŽͺΚ΄―Z΅JWSΆυώJ°:u’)S¦HxxΈ¬Y³FŸλΦ­ΣΑy£J3φ ϋ…`.*Ď1B/ΏaΓ†em2ΗΉX€‘XƒzΫβΒΥ‰“ ΰ oV£ΖΝ*„uΟ>ϋlβΫ²e‹^ͺeζζζκ° –‡u5oή\:tθ «Δ"‹u°Qΐ•8FρϊΖλΏ^½z:΄Š ©£†Χ1*Ήβ5­ηΓr0Ό¬†izυκ%Xζ¨–Œ*―: ‹`+Ž!?¨(‹qΨ6ŒΗ8[ΛΗqΫ·o_iάΈ±>φP9a6Gehc›4iͺ±mτ²Qys[Λ+kϋ9r ΕVΞΟληΎύgΥ‰ϋ ^Ώά@ P€ (@ ΈK`ΣΊUςρΫ/Ίkρ\.(@ P€ (@ P€πˆΐ)wΔkΠΨ#λζJ(@ Pΐ{VͺλŽ-λMx·ͺ$>αβ3 C.Κ0d”ΌωιOrέ”‘–Κ²i©IςεG―Λ΄»/5OE$'ΖΛ_K˜f9v’<όμ[β«.-Ω0 ΫΫ­χΉώβ1¦jzΖ–Šυζσ¦€Qύπ†ςΪ¬Ή²Mϋ’«Ρύ=ϊ ’χΏY$WN<[ ςυ0„η~σ±ۜ‰)@ P€V?¨ ε%+=ψτrΞ¦XMuΊ•Ϋο|x¦tξΡWžΈηzΛό?τΪsΙΣύΨ2Œ (@ P€e ¬6~όxιΧ―Ÿ€Š*‘‘!7jΤ(]A•b&E(α7TjEπ9„RэρW\q…Ι‘rdΛ–--ΑWΜcT“EπU„jK6„ψ€CΨ6!!AW£ΔτFθa\„μ°^<ΐ-«A>„lˆMMMΥ[TΥΕ6a<–‡υ! ‹alp΅Ž‡nݺɍ7ή¨ΩxέαψpΤpγψΊψβ‹u΅WΜλΤρmc[Άl%ύϋχΧσκPlhh¨^.–c ΑΫλ―Ώ^W€EHάήq…γλΗ4°'''KVV–žۊγΗSΣ¦ΝT8Ώ‘^—£}δxΧ 0λzS.Ρ‹pΧSάyΝϊN¨ξΨ\άΕΤΦ9ξX—I P€ (`_ΥSžΌοˏ’Q1m€¨0_’ΪŸ‰c(@ P€ (@ P€ @5ΐχl (P³f½υΌ  Y#lb­'ŠŠi-—]w‡Όσς“–ΑKό 7MΜ© Ν,3Ωιψε‡―LAΥNέzΛ£ΟΏλπbKYo»†Μ|τΛ’χξάjι.Ω±sΫFωη―₯–ΑΈ˜³¬@¬1ad‹–2iκ5ςε‡―ƒdϋζu–nvP€ μ ΘΊτΧM£Qέ^ ΦzΒQͺ*μοjήeK~Ά ώsΡ|ΙΛΝvϋυŒ–²ƒ (@ TsΥPΑΡΊΉ"‹ε!δ†π^LLŒυβ-έΦλA`oπΰΑ6Ηαό*SΆjΥΚζxΛΐu X‡².Άšυzm·†u#€‹‡­ε•gYΦΛe7œΐk°δqδΜλΣ ²,*Ν™ωŒiQ αT­. 5•FM"$ΊU[U΅v’΄jkϋnœξ@ΪΏg‡,ψρk9·_?β”β’B ¬#Υ65j!-ZΆ‘q“.•φ{Th ςσdΉΊΓιΚe‹τ$ι©)ϊ₯u‚€aγ¦,’… ~Ž 5Nυ7«Π:8ͺ“ήβφν–£Gβ€NP:ΆzJlΗ‚cΒU Λή­Φsθΐ^}‡ί³FŒuΈh„Υ·l\- ρ‡$C§Ύώ~4"J’[Ά•˜Φ±ηη (P=^xόNυωτˆήψΪκT½πΉ9ΩκsϋžΏ—Ϊ―ξ@Y ?·jΫAbZ΅sιχgTŽ©»xά·K/Hˆ“ΜŒt oΠH·h& Υ£₯ϊnΰŠi©s8O‚ο-i)Iϊξ€Ν"[HDTŒDͺG‡.=^κΜώp P€ΐ9Tœ/ΙU7”Αωˆζ1€Y€[~Ό)ο9O½o[;zβάVbόaΩΉ}£$%‘Μτ4©W_GWη’Q±ͺA£&Φ›Γn TZΐݟΗάωΉ§*ށ{γyf?G›mw|yί—± Oxj=vq8‚ @5@³d˜sΚ•ΣœΪϊ‰ͺ2λ{―Ξ°άTη·mZ+»χqjώ²&Z΅|±iτΈ .sϊ\H―ώ§/|ΓB23$5%Q_{bZ¨κωμέWLƒ;W]“Α4Μ^Ο°ΡγM‘Ψ½»μ‡oν-ƒΓ)@ P ζ ΰΈ‚ό\ΛŽΧ ‘‰—\cιwΤ1υš[M‘XLΏoχιΪ³Ÿ£Y9ž (@ Pΰξ‘9»|GΣ9_ςZήιKΞ_²ίΥΛ+Ή|φS ,ŠΎώ*:ŸυΆ8Z†£ρΦΛ2Ί+21/Ÿέ'ΐP¬ϋlkΔ’ργΚ;―<%λYnsqg4#„€ >yη%iΫIπƒΗ—^η– «°ό(ςώkΟΘόο>΅ό€ƒαF+RαŽΫ§ΨφΉ_Ο‘Ψ‹.ΏAFŸ‘1Y™Ο'NœΏύXf½υ‚d€₯”šϋn„qΧ¬όC^}ϊ~9wΒ%κ¦OKhέz₯¦η TΏTόυη±lκy§Κ7LΧύΈ βόWpα`Ι¦οx£Žύ;šιΤIάρgwΠbb9Έ πΫ…λυ"³³2ευηawc}“²B±ΓΟzσy^EέY1ǘΝτάΊ]G}|N˜r•K/ΚΗE—`YΒX‹Φ²τ;κΈdl?}£cΊη.“ͺΒ!(@ 8'πσχ_˜ξ{Ν-χKϋNݝ›™SQ€ @₯Ύψτ½—eφgοIqq‘Νeαd!ͺrά|χv?Σφώ«ϊσ<pςΤIΣrnή=Β2lθ˜ρςθΜw,ύΦψή>χ›YςƒϊώŸžšl=Κԍ›h ?g‚ŒΏπ ιΓ|gQΣ„vz°ί½1SζΟωL]“gg*Q7θj+Wί|―^Ύ/Ή²ΝύζcyνΩM‹pφ(yόΕχŏUΔL.μ‘@M@ˆν«ή|fF Άd‹V7.Έψͺi2fόEβ―ΒVΦ¬Τω£]zΝm‚κS%›«ΞιΈλ}ΫSηΆJΊύ;Ά¬W?Ύ"؎’w§Ε4ψΏ±w!rΗCΟͺ›LΆ3fγ3*$ΰͺΟcφVξΞΟ=ξz°·/Ξ ―ΚσΜ.ώIήzρq½™Άή;.{ϊs«ŸŸΏ|>oΛ.Ήκ} τ”§ΦcAb(@j.°~υ_¦=@0§{aφzpc°v»ΙΞ­,“,ώε{—„b“γ-ΛDGοCLύeυD4όŸvμX±e2άX7d·nΈn^nέ&\|•uo™έ±κχœ/1šOm£“Ο (@»‡ξ30d€>fXFOΛ6₯o܏30[GQ€ (@ P€ lΈφͺ;+ΰ 3W`Ι‚ΉrΣecνbννωή]ΫδΥgŸΌΫζ…?φζsv8BSΖτ’ΥE 'Oš/˜-kψ‘ηΙϋn”ΟU¨ΟQΛΟΛ•ϋnž*/?u―Ν@¬½ωΜύJδ69}A‚½i9œή(€Χ~ό‘–ͺ*‘½ωβcςθτ«mb1Ηβž[δΦ+Ο\ΤnλΒLg4TtΖ…σ>ώύ±σ°:©|Ε›±Ζτφž?~ϋE}Μύ:ﻁXΜ»oχvyC…}±όu«–Ω[\Ή‡c?Oο‡Ϊ΅_εiΗ,Z(ΈςΜΞi)@ ΤhCχΚ+κ¦$FλΦ{€\vννF/Ÿ)@ PΐΈaΞE£{κJφ±X=>/γfRάzΉΌ>σa]qͺδf8~άς™•’J6λΟΫǏ—ιqώbπς‘ «–ˆΕ΄Έ‰Ύ»ίtιΉ2γώiΏ»`£m^ πνgο”ˆΕτ¨žϋψέΧΙΥ“‡IŽͺΜθͺ6ηΛδΕ'Α§Ο a ΔΊ ˜Λ‘ΐ"€σ —$Όώ¬Ν@,v3nn™ωθrσεγτϋTFZͺlPۏ4U₯ΙVsΕ9wΎo{κά–-›ω³?“Υ/Λ—όbχό߈,^=yΈΞΪZ‡QΐW~³΅>w~ξqη{€­}qfXUŸgΖ͍σςGΥM K6cžΪoνŠχe,ΠSžZ ‰= ͺΉΐFuλΦW(ΟM±zτd=»ΊΡδ\»ŸWM–Ρƒί†ΣR’,Sΰ†`šDXϊuΰ ցXLί€Yd©Ωvnۨρ#"’b7TvΆa»FŽdy ?η?ΞΞΚι(@ P  €$5ν}tΛςέΨ 7₯¨ήΠ΄ [7­3Mΐ P€ (@ P€ J 0[Š„œΨ΄n•ΊxσZΑΕ©m­>؝•ώAΕzύΈΈυρ{/3όf=½­ξw^~RWH°5Γpμ­WŽ—UΛΫ›€Μαi©Ir-—κ Λʜ#)PΤ…z‡£²ˆ3 –ίUΥ₯οΎα"9^ŽχΤδΉγΪI‚gg~h}ξ‘Ϋυŝ%0-k~Tv½ύκ‰eΎ”5?ΗQ€ €wΰξπOά}½δλ  •Gž{G\]‰Ο;φ–[A Pΐ»φμάͺΒ€7IvV†iΓπά°q3 ­knτ|σΙΫrσγ†Iι}Ζω l­οώΈi„έ;Έγζ:Έ‘—3mυŠ₯rη΅Ψ½qVm__›‹Ω«Όž~πf›γΚ;πΫOί‘WfάgšmΘΘqςτ«—λbXΣΨC œq›Φ”Ϋώo’Ψ UΩΪYT6½ύ&HvζΏ7F³5MYΓΚ{NΗSοΫ–mφΠΉ­%Ώό CΖΞOΗ9θL“Δ£G,›Κ 8+ΰξΟcξόάγρχ¨gβyζςΎ/{ΚΐSλqπ'ηh P€ΥNΏ½n^ΏΪ΄έƒ†1υ;κιΩχ,Σ$Έζ’²7πΒy˜ko{PnΈσύΈυΎβkη܈iελΩΌ~•ip` ›‘ΪΝ꼏u6zΌΤͺUΛz»)@ PΰΩ; 8+ͺχΏ₯sιXΊ»AΊADP »»ΕξΕB%DR@@ΊKΊ›₯»KτχŸηΰŒ3wηξή»μ½Όη󙝙3'Ÿ9svξ™σ=―Hv»wyE±  +γǎȑϋlξΨ'3&.ΏτϋσlΞ™>Q† όVΌώφŠφ Θ€I€¬šXΌd‰-Q*^ύϊ| ΏYes;&ΰ_{Λ=R©jm)S‘Šι–/ž/lΖ υXqE°Oψ‹šΆv'‘ΗJ@ (%FτωθUΟϋΪ£½ή•ΒE‹₯‘k1•€Pi—9žψ…ŽϊZΚΝw?&Υj7 /ΌΠTξπΑ²eσ:ιΧηC™1ιw§ΒΌ›#Ž%Όν°ΠQͺ\EsΊyύωκγΧνKFΠϊΐΣσ‹#X$aQ*· –ίΰLΔ¬UΏ‰υΏ‘Έ³`ΒAkΚδq#eπ}dΣϊΥN,θΥ²]g3FΰxPο—ŸΈΛ,’e_B{ιΧI—ξ·HΙ2ε%CΖL²sΫY»z™|ωώ+·y½ΤXμν§Φο•€‹c|χ™|ςφσNš΄Ύ€‹τzλΛ°&zΠ% cGΛ³ή$τ[nW―Q iwi7Αͺw¬ZΎP–.œgυ‹#L°5+—Hο€>Ζ?‘γpΖt’Υo»Λ±-„ \m[’΄+_©Ί+UΖ²ώΈIΦZŒ‡τΖX·Γ1†όέηοΘS―|d{ι^ $J οcξL#ωή“}€»n~Η©eœ±«}gŠxζ―Ώ¬EkWg\ΎΧΫ_8ΕΏπ‚³ο܎GΐA8ύ2Q£Ε ZωΰΠS% ”@š'°aνΚxοχ«Τ «^ε*V‰ώΐΎέ’+wLYΌ`–}hφΕ¬oΗnΗΒ™ ηΝ”];ΆΚύ{€ 5vT¦|e)UΆ’dΝ–έT•€PJ@ „Lΰš›ο‘KΊ\γ„―V³žsΚΑͺε‹γΝ»¬P₯F(Q5ŒPJ@ (% ”€PJ@ (%ΰ" ’XŒτx8fxYΎh^ΨUΛ_°°\Χσ~ίxsgN²&EyW΅bρ₯ΗνKω*Υ%ΒrδΠAY·fΉωh±~Ν ;˜gΏrɟΙ"Š]±d,˜=Υ“Ά}σώ§δκwJφY€Ω²iΌϋcςηœiv0gΐvξŒI©Ϋ ŽσgM‰'΄shͺƒ4lΪFŠ—*+¬ ΗΚ€£†όo ‹8Xcπέ§r»΅Bͺ:%V Ψ‚Ψά1y₯χχ#­ˆ•ζΞ3’ο=)ΡΈλxœšΖ™YθΛ^μ+°?ΐ!c‚οΔξz…Ϋ/G‹A΄ςq³Πc% ”@z!ΐ’.·λ{hΰ5ΏσœΉβ‹_μΫk-ςUΑ/xDύψ½ς؝έeΥ²…N>Œ³άωΠsΞΉϋ`ΥςEξS)[œοήΉΝ+yY¦ώ1Κωώλ hT¨\Γ|ws‘ΧRn`8=WJ@ (%H€oIuϋίδΫΟήρDg‘ΝεΌσ­<τD (% ”€PJ@ (% ”€π%p――z¦³§ώ!ƒϊ}φ6vδ   Y+iϊ9„cί™"Χίφ€4hάΚ¬°ΙͺnΧέ&Ÿ4ΪωwΕ?½’tŽ(ΦΟ1Ωτ–»w&ΤΪa°NϋΡ7C-K“em/Ο~ήΜɞsNϊϋI~χYύ”€Piΐ‘ΓγU!gξ ŽwΡΗ#KΦlρ Ζͺj$γ?,ΒΓ’3&•‘ƒϋΙλΟή/ΧvlΰΔ2vς»_Ο9δZ€Œς±Ύ'ϋm°άΠΉ‰Œ·φ‹IΈλΔbΞάEz=r« ’U§”€PJ ~Ή,œ;έ“ΥεWέ$ΑΖ‹<υD (% ”€PJ@ (% ”€πPQ¬‡ž„BΐmιΡΎy›ŽA…―Xo©d•΄΅.Λ‘ξτΒ=œ0aǟœΤΩηξύύίI»Nέά^Ξ1Vaν xnΫ²QζLŸθ\w0α”ΝΟU^GyξmΏKΒǞ‰c‡ϋ^SO%V4os©T«U?Ρβb κζ»υ„όΣWζ9πx9yܚ4h‰60θšKdŒKȊΘυρί ζ{^§ASΉβš[<Χ&Œκ9Χ% ”€Hέ^ζ>ΩΏw·)$οy½ήόBrδΜ•Ί ­₯SJ@ €#«—/φΤζŸύγ9χ;Ήπ 岫o”\Φ„M{Ϋ•LWΊ¬Š°zύ&­όŠΟ―b•ΏΫβ<ηξ“Ύ½ίτL°,W±ͺtκvƒ;ˆοqž|δ²+oτ\[8w†η<±“o?{[ϊ|τš'"γ§_ύ8Ig= ι‰PιŽΐΧύU3k,·A“Φ‰Φ³x©²V?m™=ʘΙG£ίφ«F4ΖΆXΰνŽ‡žυΛήγΗΨ—mέΚΎ°{ΧvϋPχJ Q‘|‹τ{OJυ~PΣϋ8s(ύr΄D+Ώϋ¬~J@ (τ@€EˆέŽo£,ΖcLΖνφ(vοξςΔέΧΚύ7]&OήsΌΥλ!=τg9rθ?‘/ χ>]ψνβη˜kρχί;—.ΈΰkA™ΕςΚ“w…όύ™Θ.·tm)~VwΔυ@ (% ”@2θχΥ‡ςι;½<)-^Jξ~τŸž(% ”€PJ@ (% ”€P‘ΘZ0 ₯ώ#p‘5YͺRΥΪyό{Τ²}ηx~nMλW»Oγ'Ž;Ηηrpθΰ~ίθXΛύςΓWεͺξπ΅.‰˜λ{>ΰ1…νX₯4˜λyίSΑ.6――?ySβ6­‹ξΟΉΣ€Kχ›γω«‡H+Ί^Χ3δ’vΌβzk€χω石“γΟόυ—lέΌQ*L:L°iλK€~㖁ήρΞ±xβ³3 ³‘:&»­<-ύsŽœ:u2ICΝSΓ)% ”@ςόc™5uΌ“Ψ ·=(5λ5rΞυ@ (% "O C†ŒžLΖ  ν:v“Œ™Ά0x‹υ»œ-Ήέ57έc-|s«I6kΆμ!ΏΧ9r(€’πΫcμˆAž°ν;_² ΅QσvςSߏψ¬ΝΝχ8η |ύρςέοz‚t½Ά§<όά[βΛπΠ% Ξ[τWγGύκ©Υ7ήε9Oθ€ΗΙΘ_~Σ§O%ΜχZ¨c:DŽtΏν[@Λ3c[W^»dΟΪ‚=…Š“ΫΆ8Ε=tΐάΫ  JΐE RοcΡxοI©>ΐ…Ο9LΟγΜ‘φΛΡb­|œ›«J@ (tF P(n ΅Ί9sΕ8 NηΐΎ½‘FXΈ=Φβ0³¬Ή_v΅―υΌΓ‡Όγ7^˜A^}ϊη[1"Ω‹/λ.Υk72*ΛιS§dÚ²nυr™2~€9ό_|暼ράύςξ—ήqžˆUNVJ@ (σŠΣή}ωq™1ιwO½Λφ΅O„οꔀPJ@ (%NŸ>-‡ΆζL‡ώ=χ6CϘ1£dΙ’E²f͚UI•ω3oΎΗŽ3εƒWΆlΩ${φμ)6Ηƒo?'Ož”ƒ:σϊ3X ΞηΘ‘Γ”+) IσΔ‰¦φbf΄‹˜˜ΣN’’¦Ζ‰3gΞΘ!k̍~€{I{ȝ;·yΆέ₯ΰή8pΐι/2YσΒrεΚeΝΛΚμv^Γ &°‘_„Iž[Εͺ΅€nΓfR»~ScΩ2gάf²*+¦&ζ–.œλ€xΙ²R¨H¬ο5·g•u|E±‹ζΝtΣc%¦d°^Κωͺγ™ΓzςςΕσ(ΫβBΕΆκΰ„Oθ`c€ψΎjΊ w­d™ ’;&―Ψ"{&{n\·ΚZ VΌ°κ‘”€P©‡“XXtΑvΌσ%Άh‰VχJ@ (%|j7h"σfMvDδω„eνγ±^οHl‰Ž΄š·½4IYM=$€xLd ˆ5iΩ>€Έbρ†‘ΣV9α™ΌŠcα/Ζ8άξκwΙO{­ΖΊ―λ±Pη7}{vz¬Zg²,GU­Y/d(ω –jΦψϟs¦…ΗΨ4Δ1ΒGΊίΆΛδήGkl«˜5†ͺclΚνμq*·Ÿ+`"υ>χž”θ‚qLΟγΜ‘φΛΡb­|‚έkυWJ@ €uǎφT!WξΟy¨'LΔt» .ψoρp·4Yt ²Cϊχ•ϋ1ίpέω XΤμ―ΏN;—ωύΔ‹οKωΚΥ?κ4hjΞ―λyΏ±T»uσηϊμiδןΏ–nΧέζψιPJ@ (s!ΐdθ‘Ύ‘/?xUN?κI*KΦlςμλŸθ"Ο*z’”€PJ@ €$={φΘό!λΦ­ y!pΚk‹a=(P@J–,)EŠ‘όωσ§J1"PΔkψEΛύe‘š0a‚,]ΊΤˆζ ,( 6’ΊuλD΅ξϊςΎΚύ1b„ΗΒ‘nέΊu₯iΣ¦ρ„ξΈΑŽ^½Zƍ'GŽ1m bŊrΩe—±m°xκŸ: ˆ=z΄¬_Ώή΄Σ|ωςI§N€\Ήrž?~\†.[Άœ]θ™gΎcǎR’D OΈτrB»FμP1;‹ρΉΟΦ­[eδΘ‘²wο^ΣξaΦ΅kΧ$ ΜέιλqΒB›m—pzU ˜I ¬ͺΉfεRY·j™YɞΥμγ6[’%Œ†«˜ˆ₯IΚ°zω"³ύά··)Rιr₯^£–ΊC#’Mθε&˜₯ΫΣ§OΚ+OݝhΧ[b ?·οnλεJ-Qϊ±QΏΤO bεšŠ¨ά]“ͺ5κˆb7Ή/ϋ—.WΙΧ?Π3nΣzWbh==)SΎ²Έ-Cο΄ϊ2Εϊ‘R?% ”@κ pκδ yρ±ΫŞπΒ₯ήωR˜άN (% ’K qΛ‹₯ο§oyΖζ͜$Χvl Υλ\$XF­]Ώ‰Y°ŠΥSΪ1hΉwχk #NxοίΎm³ΜŸ5EϟRΡΆΕmς„Λ—Ώ°ΠN¨Ž1ˆ<ω „ά„ϋμ½ΕΣpGLΚow|=VJ }ΨΆe“§‚U¬ β™2…·J+ΦK“βBΣ %νsν·ύςˆΦΨV‘Ψβ~Ωϋϊe XAχοΏ£3Ύξ[υLs"υ>–ο=π#ΡζaŸ§ηqζPϋεh1ˆV>φ½Υ½PJ ½Θ‘3·§J,Zžh&&oώ€$rœΒ±%δ»‘SΝ€ΟϋφΘk|fχŽm2iμpcΙ՝ΠZkΚύ7]&Ÿ4Z²ηΘε\:rψ sμ>(T΄˜|τΝΙ–=§ΫΫs\’T9ι3`Όά|E3Ω½s»sνσχ^’Λ―ΎΩXΑp<υ@ (% ”@όi-ϊΡλOΛzk>e +V²ŒΌώρΒό$uJ@ (% ”€H-°p8cΖ ³ Ύ‚•‘9Ά•XΔ”X”,V,VjΦ¬%7–²eΛ&IX,ΏsρG΄†XmΩ²eFΘKω’9o…±— Θ¨Q£ŒΨ°L™2R¨Pa©S§φΉTλœβΒ$..N~ϋν79zτ¨§Α'βΎB… IυκήΕΖBΙΜ"¬ά½{·Ή°nίΎ½ŠbC˜‚a|fΜ)³fΝΖ‹/.υλ׏'ŠE\>eΚYΈp‘)qΥͺUMΈτ&Š… uε9A\Έpa©T©R—{^ΟΧ /s̏;β φΐ-]¬—tοjž>'Pχ[bΥ)% ”@κ%ΠϋνηΕ½xΙύOΎ*LhQ§”€PΡ'PbUyυƒoεΉ‡oρŒ0~€ΠΤ›²z-ΛJjΓfmΝƐH»ΏNŸ–Kȟs§Ι’?ηΚφ­›d׎­rΖZ3©n{€(6ΑΒIM*€x‹Μ’³§ϊ†ύψΝg­ρŸΦa‹l}SO% mq=uΚW η<”“ΒECuΊΣ eLΗή>ŽDΏm§νήGkl«H±’ξlυX DŒ@€ήΗ’ύή­>ΐοF€χqζPϊεh1ˆV>~χYύ”€Pι…@ΰ»ύΡΓ‡’T΅Σ§NxβΕ䉬(–Ezxo t7έυ¨,[4OήyρˆhΓΪ•2ψΗ―„λΆϋ?ρ·ζςμkŸ$(ˆ΅γcU·ϋMwK﷞·½δδ‰γ²mΛ†°=s"λPJ@ (‹FD>y»—L?2Œ3Ιυ·=(=ξxΘwnbΌκ‘”€PJ@ ((`^‚F6„`8¬©fΚ”Ι·„!PO[σ1ΣamAΨΪ΅λŒ€ +‰+£)>υ+,V-7nάh„~“'O–zυκΙE]υrΑΜf|–σ?~ōΊeA‰Θ™ύ’%KŒU[ŽΓu€g·'ΪΗ껍΢ӳύ»?aΣ›£ξϋφν“+VΘ΄iӌΘqw… Α%ΠήΩpΑΨ₯7N©‘>*ŠM w!‚e(X8ΦZU,4 ‹αK*½ιiM$N΄ˆ†e₯‘ηήόL½σj œ JΖ¬ϊέοΚ„ί‡Ι³―b,ΗΊγ εάΧΞυψTΐG¦sMOγ+hΘ™+ό—έ\1y=ΕΫ³λΏΥw=\'Α>jΊ‚Θ–λά§ζ˜I5ηκN[«σ¨SJ@ (ΤI`κ£dΨ€oΒ5iΥAΊtΏΩ9Χ% ”€ˆ>fm:Κ½ϋΙ{―<.»Άoυ- gO›`6‘§₯xΙ²εΪ[€s·Φ€ΕΎq’κΙ γ―?}%_χ~3ή":Α̚-»œ8~,ΨeΗ?ΠbZΰ@Nΐd:HHΐ{ψΠωΰυ§δεχϊ&SnšŒPι‰@`•Σšό+”„…ΡΘ#”1wY"Ωo»σ±£1ΆΕjΪ1yΌ‹ΔΩωλ^ D‚@$ήΗϋ‘H½χD»πγŸήΗ™Cι—£Ε ZωψέgυSJ@ €yςπTεθΡÞσPONžΕ,rj:ΙZ­ϊςφηδφξmΕύ­wΠ_+‹­α²ηˆo ‘m‹š…\ŒΞWήhƌάγ@Χ­RQlΘ5 PJ@ Ψ˜>πϋΟ₯Ο‡―Z’ψσŒκ7n%<χ–/U֎’{% ”€PJ@ €*"7„£₯J•ŠWNή}Ψ!8=tθlίΎέXS<|ψ°±4ΚmG$66Φ€/‘(xPN,<4Ȉb9.W.ε nΈσ%558o™Dφοί/“&M’Z΅ΞZό &ŒVvwz‘Z––ϊGŸ@(χη η<Ϋ~Ρ/qdrΔΊσ¬Y³eĈα²zυjΑβk‹-‚φevύΩ»ΉD¦tšͺ›€Šbέ4τ8$μ³>@΄—P„l$Θ$«Vν/“CφΛ”?~ )€bΰ¨ίˆ―Ο‡ςΣΧY/[§ΓN*nΣ:Ή·Η₯ςe±R©šΛ$ύΏwΨ †αΊF”4Hj$# ’ΨS'{ͺ’%kς˜…ίc‰υ#αό«#‘_šξ°~ΧΥO (%pΎθϋΙ›~ŒΧoΤBώ=Δρ vpόΨQΟ₯³§ΙΎ=»Ώ .ΈPZwΈά9Χ% ”€@γν₯5Αƒ>y’΅πΤ|ΛΊιιS'ƒ&·y½τ~σ9ωξ³wδ½>ΏH•uƒ† ηΒΫδ©ϋ—΅+—&­¨eΑ―bΥZ’]'‰-^Κχh—hœΫγpRž8f˜΄ΏτJiΪϊΗO”€PpΏλrž3gψ‹œΩqˆ)ι~Ϋ―άΡΫβ·J8.¬Γ)†M›’ϋ},ο=)Ρψέέτ8ΞμWΟ„ό’Ε Zω$TW½¦”€Hλςε/θ©Β©“'Œe†p,°œ²ΖlάοI'–b=™ωœ°(ΟƒO½./݈um+ŽQ+@̈L±¬;rδH)R€ˆ”/_^Tܚod‹Μσ\―^}Ι—/ŸιŠ/.yςδ‰`ŽΡOϊ„΅ˆΰ’E eΑ‚‚@6kΦ¬A Ασ‹f͚™>ωUͺT‘Œ3£’€Šb“εy“›Ο?˜  –ΑœšυI•κu€B•šRΆBσπƒΧžςeΚͺΤΎƒxfΜEn»)³JθτIcdςΨ2wΖ€°²ό3ε©{δ›_' ιαrζΞ#‚“fY­τ\\Ž$LD;—ό4H.Y?fΒu»vx-ΓζJ‚uΏ<±Œνv¬0τΪGί[ ι„7ρНΗ%J§άŠ@G ,Žž+% ”€‹ΐ™3gœ3Ϊ>|ύiη<œƒο>ΗlΌςδ]ρ±™¬ίψ5m-•¬ίςeΚW–’–ΆHl …Z&X†β'ˆ΅ΚiΗ@κγ/Ύ/―μ!Lt` ·oέμdϋξˏKνM,k)Ή?=PJ@ `Ϋν°.cr_$]4ϊmΏς§¦±-Ώς©Ÿ8Ιω>ιχž”κόψ¦ΗqfΏz&δ-ΡΚ'‘Ίκ5% ”@Z'7‘xUΨ½s›°ψW¨n·΅¨X Λ“‚–bν²TͺξZΐό_Ονq›QlŽœρΗ>J•«hGyΟΒλnQμζ kCŽ«•€PJ@ ό9wΊΌτΨ²oο 0C%wL^ΉέZά’±|ζ/©SJ@ (% ”@Z#€¨ ‘V^Cq•*U’%JΘ‘#GdάΈqƊμ]»dƌ¨Q#)]Ίt(Ι${ζ"hc£Nα.f›μJ ΒρπμΩ³ ±cΑ‚ή…Ω@5΄ˆ$#GΉςΚnΦbώ'Μ3•9sfc:‚YF=iΎ];vLΠ•%Φoπ›ώοΖotϊšlΩ² ›ΊΘPQl䧫˜h9}βί:(TT^ωπ©VΣ_ z& V[}3 Ρ3§eΑς’Λ―1ΫρcGdρόΩ²pώ 3wΥ²…ΎpέIoή°F¦ώ1JΪ]ΪΝxξρAιαg³RζNC•@z'p0 "ωλvΉ,ΑyrΈΦK·γ%€b՚Bί”V ©ƒ:% ”€PJ@ (%Φ dɚM΅hg6κ‚ψuήΜΙ2kΚ8™4n„ š΅‹Q½χΚςΑΧƒm―$ν~Ή,š?Σ·mΗςh―w…ρ‚δpΏAvνؚΙM#cΖLςβ{_I‹ΆL˜ΜY²ΚΓΟ½-ίΥέ‰³wχωτδ‰—ώ³ λ\Τ% Ξ[Bω]c3‘€Ω·{g(Α’&ύΆ_αRΣΨ–_ωΤO $s}‹τ{OJυ~|λšΦΗ™ύꘘ_΄D+ŸΔκ«Χ•€Pi™@ή|βΩΒΉa‰b·nΩθI£pQΛ²ƒOΊž@‰œΞ·i½}9Νν?κWτΓρ’½ΰ.Ώ.™3gρ\Γκk›KΊ˜0χ\ίΡϊ'pΚ†··Vύ&B>nηφ8Ω³k{’Φ(wνΨζYΕΤN'c¦Μ[Ό”}ͺ{%¦,_<_°ΔŒπ<Η$ρ5+—x‚ΦͺΧΨs~.'τEn‘ϊϊ5+€ΞEΝBN’ς?|ωΎυ"xΒΔαΕε¦;,0%Ε,Μ‹Ιώ½»%oώ‚‰&7Φ”DΓh% ”ΐωNΰνΟ~φ}‡KŒΛ‹έ.ό°έ3―"•«ΥΆOΝjgΞ‰(% ”@HΝ›)sfLtΒ6oΣ1ή€Lη’λ€…mήότGιΨ¨œœϊχ=όθλ£Θζ (ΎpEKπχϊ kWyΒtμz0hŠΫΎΝμ#0n`ω(χΪ•K₯|εκA}Ο')_}όΊsνͺwŠtΉΦ9υΰΑ§_“9Σ'šίfvœ·z=$ߝšδί2v:ΊWJ }¨TύΏw]j΄-n“>tP# Εm³,GmZΏ:” I ­~Ϋ―p©mlΛ―Œκ§B%Ιχ±HΎχ€dŒmjgVΞHϊG‹A΄ς‰$+M[ (%.ξ|•ηϋθ’‘kbε›?{ͺ'H“V{Ξ“rR’t9™1鿘›Χ―ωο$Δ#ζW:ΈίΊ\…ͺžσ Ujš1Ϋsc~·lٸΎnφ₯ΛWςœλ‰PJ@ (?οΏς„lΩΈΦΉ”ΙšχΞ₯F‹?=PJ@ (% ”ΐωH GŽ’;wnΑ’$s¦κ?~<$|/@ …P‘8ϋφν3{ό‹!ΊΝ–-›ΔΔĘ-kΦ¬–F$³oΪΰHƒ΄Ž=κˆ™ŽίλΆδ‰uG &v£ˆ ;&{φμ‘#GŽρ‰›/_>Ι™3§)e Η‘'u;vμΈ% >dκL>”ŸϊQW„ͺ Υ5œό ‹%JHνΪ΅εΰΑƒ²cΗSο R₯J™ς$”ΖΉ^ƒ5,Ž[‹Πμέ»ΧάC„6kXΐ<‹5―?S¦ΰ†ρμrΠώΈ0fΞχ9K–,¦ >α_‘B…LΫ₯Ρ~iwάwqΉ,ΰ'βΐˆ4μτ‰—/_~Ι›7)c`[ΰžvh{΄m»^΄κŞ<‚΅C»NξύιΣΗϊrI“rQnς²ΛLΚ›7―©#υ uξ”;ŸΐcPψR^€ώά'ΫQ?ψφ\ι“6샱‘ݐuGˆΟ½αœgG½α φœήΒQVΚL:ά'8ڎ6DŸ‚?qi#€c/κGXψŽrrώ01ή„'?ΚKΉχοίoŽΙ—φš'O§Ώ εώQΚIš8Κj—ƒϋEύ>ςοσuΜΤ™|h#v;ɜ™… ΟΔO ΒλuΣB΄Œ%pψΠίτ³eΟαλo{Κ2^ŽX­"εςδΝοωΠγΞgΕ’RΫ΄s•ͺΦ’ŠUkŠίŠ©nλ°u4 –„Ό` +>ώf˜³:Λ€±[―lνYΞN¬mΗςβ»_Ω§ΊWiŠΐίΦ?Η™ΦσM;Ε ψφ39γzAΘW p’'ΊϋεW¦|eO_0€7νϊΫ}‘°ΣBψξž”^‘r ΉγΑgνΛaοσX/·n5IΎa³6ήžs^H¦όρ›ΗOO”€PJ >Ψ₯γ{†ΰΈΨA‘ΨRΊœNt QJ@ %ΐxAΏ>8ΧΟόu:$Q,XΘ +ό~·{`Ρφ uΟΒU'Žυ―dύξΥM3,€  5 d‘Ÿν†το+OΎό‘}šΰ~οΓ=Χ6kλ9υ$Α"rϋOΛGo<γDATάχΣ·δžG_tότ@ (σ—@ΉŠΥ$“ΥמΆ>ˆαΨϊσWrΛݏ‡Εo1"†(ZύΆ_qRΫΨ–_ΥO „J ’οc‘|οIΙ> ΫΤ6Ξ¬œ‘τƒhεIVšΆPJ ₯ ΄±ΎΣφ~λ9ΗjΐΖu«dαΌ Ξ‘°ΛΜbΓ«–-΄O%kΆR»^πΉNΐDJ–.ο 1wζdk1―£ΦBΛ Ο+qGϊsξ4χ©UΆμR4`±ρ–ν;{Ζ£¨ ίT­YΟ7ΨΙΤ?FΙΑύ{ΛLX+£ί z ”€Pώ00fψη"?^x· b"z ”€PJ@ œο¨ΩABΔqœb‘”mηΝF8ˆ€‹ω#Η™!Dt[Όxq©V­šΩrεΚO,·uλ6™8q‚=nήΌΩΑ( ±+VΘΧ_mŽ‰΅mΫVκΥ«g,'β2β/[ΆLΦ―_oΚ…πςR[[Ίti©S§Ž°§Œ‘8ή#©Ϋφν;dήΌΉ²zυjGKΉlAiV«V]ͺW―f}‘€Τ07nlξΕΠ‘Cθ±ε΄iΣLύš6mςœόpΛ`³^΄h‘lάΈΡ”?ΩCD{wkΦ¬e±.eΪB°|˜‹ΏeΛ™4i’•ΦnK˜S.Ίθ"©Z΅ͺ,]ΊΤψΗΕΕ™Ά°±H‘"R±bEi€‰.\XV\iΒ .Œ•FI™2eL˜={ΆiGnQ,νq+iΠžΚ–-kΪ壴λY³f™φ„ψΑ&m’φ‹ ·f͚RΏ~}#L€m$δ¨βL8­Z΅JΆmΫfΪ:Ο ιΪbUΪ"‚ΘbŊ™raΙ™Ίω BΚ/πΟΐΘ‘#+Π€ΩΊukσ\–ό—/_.S§N5ΟΔ„‘ι»Ο©C³fΝ€FΎΟ(eΩΎ}»Ιg•{bσ₯νΐŠg‰g5ώόRΎ|yΣo”²Dށ}μζΝ+ώω§ΧrΟθ{pˆ€©–―iΤ™ΆΒ}&όǎkξ υεΉνΤ©“τΊλγ>†"ΨΕ‹›ηŸz:tΘ΄.ιοΘ‹vEŸG[q‹έιqL{>}Ίiγ”½`Α‚½{wΣΦh/δ΅vν:σLΠ^c‹bi'΄CΪ }Abν00ο”:WQlJ‘O£ωΖ–(γ[rDŸμ“˜<ωβ]ί±m‹ΌόΤ]ρόΤͺ@IDATmύΫQΨηIΩ³h0χΥG―Ι[Ÿυ—œΉrϋΑ*Β¦ ώ+“Ί-†±:'œγύψB’KΜ–·_|DνυŽ™ΜλΈϊΏπθmΞ€3χ5Ž›·νθ₯ηJ MψΊχΒjΗ­Ό 9ϊˆαΏ|ο ΊΓεžσs=ΉώΆdμΘAŽπ6nΣ:™4v„± XΪY/2τnΧήZYω\+Abω ‹QΆ[³bq’’Ψq#1¦μ8ΊWJ@ (% ”€P©@₯j΅Ιχž”ξόΨ¦ΆqfΏ2FΪ/Z ’•O€yiϊJ@ (”$ΐβυ›΄’ΩSpŠρΝ§oΛΗίKp\ζτιSςι;/8q8hΤ’m’ί|=‚œΤ¬ΫΘL΄²'±xΩΘ_ϋ™ί A’xΌY4γ“·žχψ±ΨyΰD,Ζ“¦ΦsΟ ψ~Ž•›ί{ε ΏKκ—ŠΌτĝ2gΪ§DY¬•h†L\κœ't0|Πχςε―x‚Όύy©V«ΎΗΟοd[ά&Ήύj―±λzή/7άφ _pυSJ@ (% ”@Τ,kD§{wο0y_³BxοιύζDΛΐοιγǎ8α*XΏ•έ™φ…Μ«i’pXž-aY$Α2Šν†ψNͺΥLψ}kΞτ‰ςόΓ·ΪQœ=ys—tΉVxΗ·σβ7Οƒ·^!|5Xςδ+ΰmߞ]–ΦϋώΦ°i›x‹lωFβ ―Η_|OnοήΞάfΑξΧƒ&œσ*A²Uo% ξ7ένΕbQς[ΊΘ›Ÿτ Ίhΐ²Eσ䉻―u>žEͺΊΡμ·ύꐚƢόΚ§~J T‘~‹Τ{OJχ~|Sγ8³_9#ι-ΡΚ'’¬4m% ”@j Πα²ξQμΒΉΣεσχ_’{}Ρ·x'Ž“§ξ»A͟ι\Ο`MΦκyο“ΞyΰΑαƒδ–+[zΌΫwΊJξ|θ9',8ήωͺeψΐοœkˆ\­)mrυΑT'πΞν[­EΗ{Κ‘ƒϋΈZ“Ώξ{ϋmΝΎxΝ-χΚΛΦw;Ϋ­X²@ή±4π™7‚Ž΅œ±&Έ}πΪ“²nυr;šYdύΞ‡½B\η’(% ”€p?j°λLύίζ ¬'J@ (% ”€HΗΏΝœ9Στ˜―€˜ !’,?Η\ `ΝpQs@ρ±‘b6C셐 ΡβAΔ…ˆΚΘ·GF΄hηC¬@be’<€Ωs?˜_a[ˆδ›{ή αeŽ=ڈΘΠ’7eA G\Ž)e‘Μf±¨Š₯JΚΣΎ}ϋ ΒX'"HκŒ΄°ΜˆΕFΚD™©+Β?,_Ϊ"Μ.]Ί$h!ΥfΤ=: —^z©όόσΟ&ψ!ά+Wœ\}υΥ¦¬0>WG½"\2eŠξ‘ ΓΓfUKΨro`h6ˆ’±lΙ=tίGΚFϊ4ΩNœ8i¬ύΪάΙ‡{I^ά_ΒΠζ1rΨπgC‰΅`±pΰΑ7ρΙ1%βWΔ’XŸE|HX·n 6̈D© "lφάcΒΨmΛͺΓ £X t”…τ† bD±<88!ΪEΨh?/” f€IΩ(?eΓ’κ-·άj=›ε‚κ>σ <‡+εΑQΞmG¨#,α γPΌ`h§kΗ |FΙ kXfEœ‹¨ž8p€›ύŒ’eƒ¬I›ηQ("Vβ΄iΣΖcέ™4θ7ˆ C6u’0ζžΫmΗnsvxκ‹#_ό~τ΄#Ϊ=o¬ώβOš`)ιR>ξνžϋGΐσrΙ%—+΅nA―;Κ@ΉΩ“Β[Δχδ…`Ϋ%ψπ<ΡVxΖΠψ!μζ9LνNE±©ύ₯²ςUͺVΫωGX4&I=}Α’Λ?ΦCLΟξ(ύΓ=ꕏεž—Κίv$ξψ|0ισα«n―D/νz½π1ήνjΦm(WχΈKυϋΒννΨ·GΨΗ9OθAΰ o™,+­&”^SΡ πΫ―?ΚΫ€η}Oz„}ǏDžoχzXφνέε)Κ•7ά.ω φψ%ΗΙMw=jY‹ύEXAΗDω{nΈTyξ-iέ‘‹Η /τ]o> `UΦνž~­·δΝ_Πν•€γv—vσˆb™tϊPΟζ#4"HDττ—[6“–h`δΰ~Φ ΗΩ—ΔBEb Χ$e¬‘"JΰψΡ#Β½΄έ©S'νΓDχ§­°ξΈDΰγ{(Ž―qOZ/λꔀPJ@ (%π{ΰ­^9Eyχ₯GeΉυΎύΠ³oHφΉϋΰΘαC2΄_ωκγΧm/³69²P‘bžpG­U3?xυIig-fSΐ²›!cGΨUη’fŽP•Hc†°&7ΖΘ­VYΗv *ς›eΨΐoeβ˜aΖΫ ΆΓμΪg‰vz~KΨΧ{χ£/јν·nΥ2k|’“<ψτλR₯F]a!/οŒ“ΗŽίxΪ³8βΩ‡Ÿ{ӎžδ=c6]―ν)Ώώό΅“eωω›ήrγ;~z ”ΐωI \ΕͺfA%χBbXνyeΉξΆϋ₯jzfAΑ#‡Zc$λε7k±Γ‰Φb‚φX+cΎ,*)­~;XωSΣΨV°2ͺΏ…@$ίΗ"ωή“}€ΫΤ6ΞμWΖHϋE‹A΄ς‰4/M_ (%’ψΚβdˆamχsίή²=n³άρΠ³ΞΌ‡ϋχΚbKΫΫOeωβωvP³Ώφζ{₯d™ ?χΙίό-»,ΑͺΫ~³r_»ΝZp|άΘΑΞκŒΑ|όζ³ΒΒβ­.Ύ\ψ’―@!… ZΫ·n’q#~‘ŸΏύDψ–ζv7[ί± ληXδxϊΔΡΦο—αΞεΏό`Ύ36S©Z-g\Š±ΖJ>}χY²`Άžƒžχ?-1yςyότD (% ”@ ƒφΙΪ•ήΣ?|ύ)³πC`ΨpΞY ½mΗαDΡ°J@ (% ”€HQόΞgγ7ύΙ“§,Ρζcεu̘1Ž°«‘XNEθ爏ˆkœce!+S¦Œ±δH|Δ[Άh±‚0,²"ξCƒ‚ˆ a\͚5₯eΛ–Žψ.ώόF4†G^OΖ‘gι₯UKD{ˆΞ8w Ϋˆ‘&ΒCΔπ¬ “XšD<‰¨ab5ΔΉΔCΨΘάtΒ_|ρΕFXwΎ· rΓΒ%eB€‡₯M6¬ΐ"ŒC4‹E[Κ #ςΪdYzΔeεΚ•₯^½zI4–Η}ΛΞ¨ζ΅nέΪ0±HΚ½AΈ‡P¦XΑ„ΑΉ8[ΨH`Π‡¨‘6€Π•Άƒ°’ϊcMKΒp‘-ΐQ!e#χΕΎvΉμsφ°Ά­±¦H«·΄9DΗ΄₯ HΡ’E1#ρΨ(Ϋ΄iΣLaq•ϋP°`A#’€Ρ+B~,«rOyžιθηh‡<£Τ‹φC{£ŸρsΑΈ–Ό±&Œ]ϊψπΜSfΪu΅Σ₯ Τ8ε ]a©–Ύ†2ΠΏΧ:;κΟ=ηω’ΡξxΆμΎ†4Ν>( m•6Λ9Ο=a5jοΉ Μ/₯Ο3€t4΄E Hl ΉωžΗ₯oοΰ“5ύ&Gε΄&€^rΩ5Ύb½»wšŽˆ‡κ\\՚υδ‡{Ι§οτ:—dLάr•ͺΙ#ΟΏν›ΞύO½ju({eόoήUί|ρd•ΥηίόLJ•υ_ω$H4υV©Žmx§5Aœη~ήΜIfΛ—ΏŽ-n„žλ­Υuy t ›·•;|.Π;YΞ³fΛ.φzGž±Dϊvή|8EψΚ[’΄yφφξΪa&Ιϋ ψ»]»΄hΫ)YΚΓGθϊ|`}tήδ€·οnyυι{Μy°I₯=¬IλXΒj–:% ”€PJ@ (%tκvƒŸΊ'"F8vΈ±R’t9‰-^ZŽZbΨ۷ȟs¦Η›δH-Ϋ_ζ[έiΪ'mŸη…g„<‰‹’zψ *έυNκ±-.Ε²2urΧΓ/Mρ,!FG‹# D™X+…ŸέΦΛ½Β²0Μx¦Ι‡gƈU9'>Ξ3VV‰ƒpŸΌΈg°f+Y²€yξˆƒ‘,bR΅X•₯όά{ϊV­ZΖ°¦oΗΑ‚6ˆάΕΣh;ˆΠΙ›g Α9yƒΓdόψρFxM[]±bΉιsθ»jΥͺε° , νžπτiΈαCφrοιkΰ@J[>|Έ©/ύli3ΉyS³Λš §eKn΄>8Μ›99ήΚ™ΑJ[ΆByγ“-Ρά _Q,bΊοΏxWn½χΙ`I„μΝΝχΘaΛ*μΐΎˆ7±6ΤDΚW.―~ψυπfρBƒ…Χζm:Κ'–7p5TίH.ϜΉrΛ+|+υ΅pωκ‘H›˜8x“―ΚΣχέ`½4œ2•`2e UXwνκ7j)―ZΟ/]‘rM[u·>λ/½ΉΥZ}ψ˜'>φϋΰKΐkoΉWξ|θyOœs9Ι–=‡©ο}7]fY—:/)Ώ…°tuΗƒΟΚkOί/Όz(% ”€PJ@ (ΤJ€ίΛO½ς‘zγ™xώL¨|φΟδΡΫ―²ΗO;—=b,Γ:ƒΌω Κ/Ύ/M[_",’c‹b BΌΕσgIνϊM]1ώ;|ϊՏΝoœ‘ƒϋύηωοY?—-{NλχΤ+‚e΄δrXδ}ΐ²‚ςΒ£=$ω­φ¦eΑχΣ~Kt Ω‰€J@ €K|(b\³…%”οεΗ%”ΙvτQˆχYπΛν²dΙζ>=ηγhχΫvSλΨ–]>έ+p Dγ},ο=)Υ$Ζ753'VΦH]ƒhε)NšPJ 5Θ_°ˆ|Ψwˆλ7Κ|ž=mBΌδΆ[’Ϋ`ŽΕœ_³ζ‡œλ"ξΑW% ”€H_ΆoΩ”Ύ*€΅QJ@ (% ”@Οn!κσsˆ­γ«"ΔΑ‚Τ:+–Αβ#„D܈ψŒτ6"€Eˆι'ΦΓj,V1w!ŠC¬‰h ’mEΡΕ"*c£|I[ 6D­ˆ[ύ„^2’C[lGˆκΦ¨QÈtέυ!-ΚΝχΔnˆ)Φ?Χ¬Yλ\sΗ±Iaβ5Χ\#₯,adΰψΒ>κƒυPκ‹x΄·lΩl¬XbΉ2’ΞΒΡe@˜9{φl#δ># MŠγ>pΟ¦OŸnΥg‹Iq`ύϊ kΔсχ'&ζ¬5XxΣΞ "’FΛ1χ)1 ω"ώλΪ΅«4lΨΠ#,†-χΐ_ΊλE~Œ!ζ~U―^݈ͺν0΄WΪ'mš{OZ΄=,“νρ*‚GΫqα—Υ“ƒœ)qb’apψσ|!"&,ε!nύϊυ@qo|—Σδ/D šαβh«ˆvˆGΚΡ~ΩBqŽyvπ"DεyBPŽp*lm‡Θ>V\y&Σ#žν₯‹±ψΑrBV„ΕπΐΑžˆUΉΏά{ξχ/ιΠwΨ‚[»L‘ξΉ§M±Λ1ιpοθ/[ZBpwϋ MΪύmš8XΚ>vμΈΘ?cΩΦ―Ώ΄ΛDέθ+hο—]v™»ΡΗ ξΝnι]¨3 lθΥ>‹vΪ©evIƒΤR-Gš ΐQί‘rΟ£/JΦlgWΣπ+xζ,Y₯Λ5·Θ—ύΗ +o–.WQX ΤΟ}σιΫΖb£ί΅pόθο|ψy4vΎΙϋΒ0DwˆaίθέOΎύu²eΉ¦T’ΩΆΊψrωω·ΩΦκ¦OH¦ Zw"™2e&οό} bέ`τ8ΝΈ¨ikωμ§ΡΒ3”Γτ+|#τύ5"Uσnd}ΐόaψtc-*πšί9« ΏΧηΉχρ—kΞΙι*T©!? ›&υ·J0Y&bΎύy3A•ώμοώN0Ό^TJ@ (% ”€P©@ Λ**οαXϋ ε·2ε/T$V^x§|jM^ΔJTBΫυ·›dό‘γΥiΠTΎ6Υ²6 Όδ9GœŠυן¬ίψbqŒg° O¨Žς<ως‡ςΞη,«Έ σ{γκwΙ q €σ•=BΝ"δpm,‹Ή šx{`½wθ€oBNC*%Ύ ΄lΧY~9ΛτE«Τ΄>ΆdŠWακ΅ΘγΦBLͺgL7pQ―˜ΌωβΕ9WhφΫֱ-wυX „C οc‘zοI©> 1Ά©iœ9±²Fκz΄D+ŸHqt•€P©ίkϋž(­;\ž¨Θ3gξΉωΗδέ/ c#‘rm:^aƊκ6lrŒε|;dŠY<Ε3Bq,TόΞε9k‘4Β‰9Β<σϊ'ςΆ΅Π2s[Τ)% ”€…€Ÿ₯ΨPβi% ”€PJ@ €EXΔϊ©ί† qδόωσ "@t&XϋDά…Θ‘ΉΠΑβ?„}Ϊ†!hD—ΐ‹΄°΄‰υBΗ‘B:Άsq ·n3υBxGΊˆΡιR6u~ŽrpέΆT‹o)ϋχο3b;Ώ8€¨+˜έ"5;Mt?tDΓ ΐŒΗz½+=nXV/_dVBέΉ-ΞμwοΪ.Ωsδ΄&έϋw‹•β%ΛJΊ Mη,M?>ŽτΌοI#v]Ώf…l\»R6XΫΖυ«Y’Abςζ—…ŠZ"ΨζΒ„ŽH¬κW.υSΡ&P©j-#(ΗrΤ £¬gm«μΫ³S Z“ΫK[BΟε*Iγ–[/‹‘™O}ΞΖd©Bά—ίλk,ΏNŸ8Ϊ²p=ΗX±ΕzlΎ…€ ΅21a°‚ 5n Η—Ύ,»wn“™SΖ[V¦γdΎ=¦Ÿ,QΊΌ”΄ΆjΦ„S^,l‡υ6u©ΐ[ŸύœδB]ΥγNaKŠ+^ͺ¬L_±/)Q5ŽP.}Œsι‘PJ@ $7Ζ nΊλQΉΈσΥ2oΦdΩ·IΆmέdν7Λώ½»ΜοδΨ₯­΄JYΏΗΛHΛφCž|Θ€πw>b,…l±Ζ!ΆYi3`Η8@ΉŠUγU…wνήߏYΦ;ψκ‹Ν‚\»vl•Όω JΒEΡW3K8ωρμDΚ²xΑ,k\ΑΘϋŒYπ«n"V]΅h'lkV,‘ΩΣ'˜χCφ[ε+,Ε¬ΊRίr« Vis±Φ‚]I}χ{«Α‰%―Χ•€8Ο δŠΙcY–~ΝPΰƒΖζ kδΠΑύ’ΗΟ€Ouλθ€eυΫνχτsαŽ—¦ν~ΫΞ?9ΗΆXΰ!©ύ7εyΡ²ΜΛ¦N œ HΎΩεJΞχ;ΝHχIν£"9ΞlΧ=Τ=ο­ατ1I­s`y’Ε ZωΦOΟ•€Pι‰@ςςϋίΘήέ;dτ°²~υra,K±­Ή EŠ•ŠΦ·έ‹;_Φό~+„σ?ΘΝ”9,Έ³qέjλ·ΗjΩdύώΨbΝ1aŸ!ƒeΑϊώe6kή |°ΰpRsR˜ΓΆcWωsήt™9yœl۲яΪmύd‘˜<ωM>5k#5κ4Lp’\RςΧ8J@ (%ώ άzο“Β¦N (% ”€P靿± *d, Œ!ΔB€‡° {Ξ±B9bΔΩ΅k—΄iΣƈ>-+’&βΦ«ΊΚ„CWΚΈΉ…{ΑΫ"1DnvΩψή™ 2XZnΚΎfΝc•t™Ο˜Qjb–±DΩ²τˆΥZ»|%K–tΚηΞng#°LΘQ¬S"Š…'εJpByΊ―1G§tιFŒ‡…M,yΒ «¦γƍ3χ ‘aΈŽφ‚θš:‘χ²^½z†G`[ L›πX’Ες%VYν! ]Άl™9Fθη`‡±nέ: ±~aύΈ§X\ΕZ0eυs„A€J[ΰΎΑ .mΘί’λΩT¨oήΌyŒΥPΪ1νƒvΔf;Ν;—yž"šεωDˡĜ].;,ΒSςJiG[^²d‰ :ԈEaFΏ€΄S§NΎ‚_ψΫBqΔα°¦ν­»λkΔΝΜ3£Ώ€u΄8ViΧ―_oς₯<ۈιψ&δΈo΄+Ϊό”)SŒΈVXΖΒ.Φky.όu¦―α9…U0GΗ³N[!?žλbΑςKNΕ&'Νσ0-&qvΌβΊT[s¬Ν°EΪ±:iMKTΛ¦N œΟjΦk$l©ΝΕZ“Ι»ίtΩRΊlq»tΏ9₯‹‘ω+% ”€PJ@ (ˆ`a˜HXB₯ΰό―R£ΩB©ˆ-Ϊ%¬‹X1 Η’‰—Ι›Iΐi§‘{% ”@΄πΡ§l…* f·yύΟυ˜<Ιo)֝A΄ϋm;οΤ:Άe—OχJ \‘|³Λ‰χž”κμ:Ϋ§¦qζ`eŒ΄΄D+ŸHστ•€P)I€ΕnnΌγα”,‚'o&S•)_Ιlž 8aL§AγVf‹@ςš€PJ@ (% ”€PJ@ (σ‚@Σ¦MΨ q”ŸCP†Ρ–·nέf ΎΦΑBΕ;v‘b—.]ŒxΟ-Ϊβq"/„WˆτeΩ‚Hς$}{Ο1"0Δ³TΗωΩaύΚͺε@μIpˆΓΆb₯11‡Ψ¬jΥͺF¨Θ1‚7˜~"4ΚLΪΆθ8±΄ω–KxΫ!‹–<© βK¬T"b…?‚M¬#’ΓΒe\Ήμβ%Ίηž>vξάiXΣΘΑ©m7±Dl±)V3I‹4iƒ€‰xΥnξtπC8I™aŽ#.eKΜ*λ\hΪ1u’M’"O»]Λ“ςΠώ±€Š³Ϋ»;|Λ(`:Φ<©*ULΫ'Ν9r8Aˆc?v|γ~!œdo;ž%Β€€£ ρ# Ÿ;wyΎiGˆD/½τR)e‰δύ¬Κ—/o¬ΎΪΒVβΩχάέgΨ<Θ «§ˆRi#ΤGXϋΨ/―δτ#ϊΔέ»w+υ -"βΆΛžP~΄)ž5,Kca™~Š:mήΌΩτΑΪΟ r “>mpγθgθgS» οiNν΅Ρς)% ”€PJ@ (% ”€PJ@ (% ”€Pg¬|Τ±“ΔCωΈBψγǎʺΥΛμ¨–u₯|’'_η\”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€H?ψŽΨ AAλώΖθ‘->C8…ψ..n«Μž=KƏo„gΐ&Nœh„‘XBΔ*’Ϋ!Eμi >_‘B7Δ―Ύ°Šθ–γC‡α#βG¬-ΖώήIY‚•ΣgBΗΤƒ2»E±”9˜Ψ,0-„y‘ŠCͺεΙ“ΈΨ–<Ο±Ω.9κj§κapΫΆm wξ)χ,χ+”Ν›7χϋ₯Oωχοί'XωΔqα†8Q^(Ž8VoΪ΄Ιv~IΩΧ;&%ΆΖQJ@ (% ”€PJ@ (% ”€PJ@ (% R5‡oλ& ηΝpΚx“―Hχ›ξqΞ:X0{ͺg₯Ψfm:†τΡ$‘4υšPJ@ (% ”€PJ@ (% ”€PJ@ (% ”€P©—"ΐP¬€Ϊ5@ΰUͺTI#ξPd,,ώ‘Φ1$VOyfΆoίnΔ∌yV€"ΌdH±9mΪm5£„Κ–\Χxή)σ¨Q£dκΤ©–hw―a[­ZU#ˆ­]»Ά ‡’βlβ#’ί²e‹uΌΟz†:}‡-¦§]Φξ;BI;ΉΓΐŸϋF;ε>#RE¬N{Μ”)³%¦Ογ΄Cξ#iς,sδΓ³e?'ΑΒαοΧώR²­$TVχ5ΕΊiθ±PJ@ (% ”€PJ@ (% ”€PJ@ (tF AΣΦQμ€ο>—nΧέ.XŒMȝ:yB>y§—'H‹vώ¬=τD (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (4K \1’Α%JaΫόωσeŊFˆ‡uΖ HλΦ­%{φμδ6oή,³fΝ’9sζ˜c,:"CDΖf—ƒ="CDaˆ(Α.Ήι#.£Lδƒ˜ ‹Œ‘XX · gEq†-EΓskΦ¬):t«›πβήαθ9TίιΣ§ŒH‘ ΑΞα Nm‘+χΗn+΄»½ψΑ’|~Β?Ώ°n?β„[>wόδ:¦mb)ζ³gΟ6bP,φ"ϊ΄…―Τίf!֘³+Νf‰k_K2…›ΟχΨ±cε?ώDσάGϊ]»ΙE5 YK}–,YbX,_Ύάˆ„ρCό 'ΪŽϊFΈχωση7~0³―(ό‘\vϋ<ϋό_`¬»†"V΅‹wα…τIY=m˜ϋNΪΑ\ji»ΑΚ—ώ*ŠMŠš†PJ@ (% ”€PJ@ (% ”€PJ@ (TJ aΣ6ςε―8₯Ϋ³k»|ωΡ«rΗΟJFλ¦Ÿ#Μsέ"ΫΆlt.—.WQκ5lξœλPJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJˆΟJ–,i,ΓbMτθΡ£F°΅iΣ&ΛμN)[Ά¬ QύυWcMΛ—X΅΄ΕŽˆ0³fΝ*9rδ0kΓ,XΠθcώψ㏎΅Q'Ρs< όΆhޟc’ι*:Φƒ7―[·NƏoD~Xκ7nœ”)SΖάχPΈ%‡0“4μ-Θ”‹ϋ³λcο͟\αy^x†ΖŒcx―Y³Ζ<[”‹g‹΄ά„η<+l "EŠ˜zc•QlJΊύϋχ›g}ψπα‚0•²SΎvνΪIΛ–-Ψ=”ςaΡyζΜ™ςΫoΏΙ’E‹ŒeXΪμ~ƒΎρ<Φ±ΘJ>X~^΅j•i»‘δ“œaόΪeφσ–―₯υηΞ>ρΌΟ+ŞW·[+«”€PJ@ (% ”€PJ@ (% ”€Pηr•ͺIΣΦ—Θτ‰cœͺχζ™2ώ7ιΨεZ‰-QZ )&'Ž“½»wȜ­°ΏΛιS'π™2e–ίύ:Qλ²N=PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJΰΌ"€¨΅T©RF€GΕ}aΑ«Ά(!d $£G–Γ‡1,Β5Δ•Ε‹7VcbbŒ(–="?,="tΓ mώύ“•+‚IΫ2,εC„ˆψ–γPα°PιΦ†/-…α^ro°‹θyυκΥ†–:Ζ":„AbBΏΜ™ΟŠ©;܎?nΡα°ΰώψlXrE™αk8ω¦DXκwΰΐca+½X[₯ώ<₯K—6Btγ<'ά{Ÿ?λ8·lΪ΄ΩXU%”rY§M›fDπˆδi#±±±FK{βΉΕamKΤ 0ύΐ‰',kΈΩ€X±b¦ο€ύlcΩ°¨J9R’ ZF°kηm·_ΥύσΟΛ*πQσŒΩq°€›%KfϋτΌά«(φΌΌνZi% ”€PJ@ (% ”€PJ@ (% ”€8_πQι₯χΎ–ΗξΈZΞ›αT{{ά&ωΊχΞy°ƒL™³Θ3―υ–²ͺ ’ώJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ œηψ.‰8Ρ-ŒD(yζΜ‡ ΗXzόγ?ŒΨ°ˆ-±ΩΈqc)T¨φ!REdk M9Ζ!ξCTζωΉŒΒ8 DtδAyί! γΨΞ7‘δŽ;f χXρNšϊ#ΤCθ‡h-=9ξI͚5₯yσζFθŒυQ„‰Σ§O7Φ8%δΈΧ9rδ4BFΒqίϋΌΠžΩΰkα [tœœiK–,‘aΓ†!+ΟΦ[›5k&:u2bΦPς£.άσ &˜τh„φIΉθ[`@y8ΥΛΜΤGθkΉοη³SQμω|χ΅ξJ@ (% ”€PJ@ (% ”€PJ@ (%p^Θl [ίϊμ'ωτdΜπ+° @ϋβ»_Iιr• ¦Χ”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€8Ο φBΈΖ‡θ !B5ΫaαsΜ9FT‰BΛϊυλΛε—_n¬^" &ŽDΠΆ~G ,œW¨{ĜX™Δš#">6¬ΫbΙ11‘'uά΅k— >\βββŒψ­Z΅j΄iScΑ2Τ2€•pXέDЈ΅ΨI“&‘!¬¦L™"… 6χ>Ψ}ΑŸψX3ΕΑA,bΟ*Uͺ‘_bhXKELK|6‡XE(˜ήbΘeΛ–ΙζΝ› k¬βV\Ωz^ΊXε €uη9„/¬RΒ!ΰ€μX„^Ίt©±ΦΚ½jΨ°‘\|qcι6Ρ9eηΎo²¬Μ\Ή<ŸΔγ™½δ’K€}ϋφ¦ pύΪm„g!*ιΈΓp-ŽraΩα=}\θ+°’…μΔeDΌuλV#€₯ότ›€™)SΒBτΔNλΧU›Φο –_ (%E%Κ”—«oΌΫΙ±\₯ͺΞ±(% ”€PJ@ (% ”€PJ@ €nΩ²η”Η_|_ξxθ96ΰ[™3}‚μά'{wοt>Lg±VΝ_°°4hάJΪtμ*5κ\δω(”Ίk˜pιtl+a>zU (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ œ 'NΚκΥ«αia-ό:rδˆl͈ۢΤ‹aν±vνΪF‹ΰ/!‡0ŒτΩΫΒ6„Ά70&Π?παkι₯(A,BBDˆp)_BŽΌΈMž<ΩEmmƒ Š–f―qΟ*T¨ :t0υ]³fΉ«V­’;v’η`άρG‹x±τιΣ§M[A0Y―^½D±άϋεΛ—‘'ιΩi"LŽφH[΄ΉΎ°ŠZ₯Je#ŒL¨Ξ΄Mž5DΛ+a•Πσ’PZα^#„ΣˆΕg̘až)DηU«V•.]ΊXϋ*¦ „š.BRΪΦR©ν{^§N#6M(Κ‚XQ)βXβΓ#‚XΚ…€—>+Ψԁvΐ—ς”/_>Q17εη_Ώήτ}ԁm±bΕB²d›΄~-αi½vZ~% ”€HV•ͺΦ6uJ@ (% ”€PJ@ (% ”€Pi—@rΣ]šZπιΐΎέ’=G.Ι–=GΪ­X"%Χ±­Dιe% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (%DX΄\°`Ύc’drδΘa|cm‡¨Q˜-C0†@‘WB +V¬Ήsηa˜6!‘β76۝ΑΕ· ‰‘!Β5Δsδ΅vνZY²d‰(P Aρ"ίΕ‹ Δ³ΕzXqL―kŸˆXΫΆmkD~Άu]ξ­}_ƒΥ‘1¬"φD€Έ`Αc-˜v‚p2˜ƒ/β[, Ϋ"Q¬WͺTΙXΝ /­ϊΓ’φM]9>Ϋ~Ε0Jˆυ%,’KX!ξΆύάιΟό‘̈>‡ "S§N5ωσŒsŸΊuλ&΅jΥJTlX,ΚMϋbnŽηΪξ;ΓΊΟ‰GύηϟoΚd_ßφΜΉϋŽseFRΌxq)WœιΙwηΝ¦/C$ŒH<˜#o„ΐ³gΟ–-[Ά8ΟWΡ’E8έα‹Ÿήύώ―‘ήk―υSJ@ (% ”€PJ@ (% ”€PJ@ (%pžΰ#LBEΣ΅ φ<ΏΕZ}% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (%2[–Ψρ"5D[³fΝ’Αƒ‘$ώˆCΛ–-+5·XD‘Y²d1’6„gώ°ώΊoί>Gτη.(i!Ί\΄h‘ :T°LŠŸ-vE`f εάρΘί-$<Μ“'OΉƒ™c!Zk€‰#€E°9~όx#Œ%XΈηX”EΠ‰%LκCΰV³fMɝ;·;xΊ:†ΦφνΫKέΊuΕζ2ς«4BΖjΥͺI5L»ΰώΑz̘1F0h @γ"€¦όφΫoFΛ=ηcα·qγΖ‰ZM L/-œΓ^©«έ†α…@’η Πq`΅nέ:9r€^rN\Η~ρΣIκ9ic uĈςϋοΏA*u /θΪ΅«΄lΩb)g8υ§ο°ŸiΪ }βuω~Ž6BYζŽ;Φήm”“λΑΪ¬έ¦I—°δ,¬_ήn?ʎœ~!ώόζbzϊL,L#υ»'ψνΩ³G¦OŸ.&LC‡™v@ίBZ%K–tξ«;ΏσιX-ŞOw[λͺ”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ”€PJ@ (% ‚ΨΆm›&$γ»Γ‡Λ† eζΜFΈŠ`Α‚¬Ys©R₯Š'~ˆΉ°ΒŠ° $β0¬΄b}4oήΌF˜Š 1*‚[D€γƍ3ΦρCdGώΆHΠΞӝΧ²fΝjΒβ0mΧ]FX‰ΕJ\ƌ™,1ef“NLLŒ΄jΥΚXd\Ήr₯)‚W½ˆΡ°ςˆsς;xπ θŽ=ZΦ―_oDmXΖ₯•+W6ω™Lιx"HνΠ‘ƒΒ Η}‰΅ΪF3YΆl™±ήI˜7ožΧΆiΣζ_Φ1ΦyΓΦ΄„S¦L1βFΪV|[΄haΪεI9œi[Ά8›gqKK`ΚsC;ηy8uŠvy@βββ¬ηq¦a…•TΨΨΟ Ον—η+Μȏ{4jΤ(Ω½{·yΆ°,ŒETž{DͺΫ·oχή?Ϊ BXϊκX¬X1Σ―.Žηα-BY¦o‘] vEtŠΨτΟ?4νρ,Ž:Swx ’'l # ςe+,Τ"B&?Ϋ4’Y¬ί†κΈYιΰC›η>!>zτ¨±”L{ζ>γΈN~K—.5‚^ž/ξε‚%Ϗ6Τ2€Ηp*ŠMwUλ€ώΏ½»ϋ±«,τ8Ύ@ *PhA–B_"%Όΰ@M MLόΌσΦΏΑ;ώ ΈΠ{#&† `Β[„¬€`-Θ‹§‚‡³Ώk²6Σ‘3έciOΗύy’νμ5{­ηεσ‡ž΅_ΘΈ: Σ.ŸϊΤ)γŽΩwNηώχΏΆoί>|Ώψβ‹W_Άαϋž?=»ξΌσΞΡΏ―«ϋΩgŸϋτΣOυ΅“μ4G=[ΪΉg]σΥ³μ²Λ.λθλ±5o8ˆπC‘ΨpRt‰ @€ @€ @€ @€ @€ @€ΐ±(„΅ϊU¬ΐηze:·`Ω΄λd;¨BmΗΖk―½vΈνΆΫλΌ«―ΎzψήχΎ7όβΏΓ΄…Γ ΅³γ΄CcmΌ}χέwΗπΨ΄ k;-l+D[°ΰjαΐώσύY nΫ!].œϊ­o}k<Ώ]!;χΧΏώυΈγlΗwέuΧ°sη…σkγΦ[osνΔωόσϏΆ'žxb θ­υ« \ΊϊΩξ‘΅S vϞ=σέi§ΞL^}>YμkΧU¦λ»j㳦ΊVΧ½ρ4sΟ=wΎm6ϋ†‡~x οMuόŠ•οLΧε•ιC=4†Ÿ NNΦ³.ΘΩϊ(ΰٜu~αΒvυmvνΊπ°αΐΥγ:RΧΟY4Έ9]?΅³ϊλτΩF_7:Ώ0eαΛΦj%»η mό|τΡGΗv«?Ÿ‚ιy˜,πS!Ψ|p ΥζY¨΄]Ϋ±wmΩ¨/ΣΉΣ9χ~φΏγG½ΫίφΟξγWΗΎ΄›j†έ'ν\ιΈσZG*σΝo~s ―ν>λΉ}\}kz_Έ΄΅Ρ51£ž΅UφΚ+―vοή=>7 Λ֏νYR{u©ž/ωΛγχ3―ά ώ·. ίώΰ?˜­½]σΛjgυkώΑͺ7=Ϋ ƒΧvAάΦϊ΄vAηώP@‘ήζ6sλ9ΣqmφliΝ_ύυ ςNΝΤ‡Κ‘ϊ2Ώφk}«½ιϊ΅ŸŸhΗB±'ڌθ @€ @€ @€ @€ @€ @€c,PͺX­ήW DΚZ―<+¬Χu…΅Ϊ©²`Y‘³vωlWΥvςœκ[]OΧξΨ±cψώχΏ?ΜΪ΄@[α―BnΣn£]Ϋ« \Έa…Αjγ‘Gw,όWˆ«`Ϋ_ϊΖpώωη―njμ[;3fϋΓώ0WΦNΑΦv}ϋνΏΟv…όhgΧvΦ,€xφΩ猻P>υΤ“Γ«―Ύ:ο[6«ΗίΞΆν–zγ7ŽΎW—ΞmGΥΎ_›™mΫφ©Υ§¬ϋΎρw~AΌκι}AΡ£-S§>MντύΝ–ΦΑΧΎvωVΜΉ qυυύΖ½^Σ\ή{ο½cΘυwΏϋέ8GYD, ›u₯1°, ۝£σnΝUΟαJΧδΥ Φ—EJυυšvΞΎΆT¦ω™Ξ_trκšϊ9s혺nΊιζ™ι)Γo~σΠ¦lΧΨ°?+ӜΆ,Dێͺ…Σ―ΈβŠqNΪ™΄ {ν½4Ϋ}ΉzχΛ4?΅™Sύ˜ΒΗkϋ1΅3ΉvœλΙ'΄ž«―z ˜ΧV₯y¬―›)]Sϋέ3έγ•ζ‘qυYχΗ{μΐ{ί―ύλάξε•υrΥ,»gμkλσυΧ_λνYΣΊ­§Υc-\ίsμ;ίωΞx^;΅fR€΅Ύ(.ΔΪχΊnšΓζqZk“λκ1χ½v›m§μv„m—Ϋ^œ[οΥ[ ·Œd€’ ƒIDATΉ‘ΉΏθ’‹ΖΠ}kΎ~MksuέΣϋζ£~΄–¦55}v€―ΝλTχ’λχHuλϏώix¬{¨~ @€ @€ @€ @€ @€ @€>QΒYνΎXπτpAυ›|― n°kGΖByÎT ¬΅[lΎΒz…τ †Ζ-dV(«g«sΧlGΖΒ`αj·Πν~τ£ω€…%O:ιγαΘϊΡN“?ώρΗ mαΩiχΘκωϊΧΏ>ΘΦφ΅ ο-·ά<\rΙξα駟C„ιΊΆ \a΅ΣO?cΚ―ž΅₯ΉΈγŽ;Ζ>τY;οΨqξxZυ΄Sτέwί=‡©ώ΅u,r<…b§ηΚTWλ©€|χSχo!ΧΒΨ…κλwσ—CŸΧ—ΦΥ…ωΜρœoϋŽ1LέάVWc¬­Υ₯ΆzŠΝκ™gžΫθš>λ{=—Ί΅U8»σΡqŸm΄Φ2μΦΨκίή½{Ηΰ}!έΒΓΥΫυ=W«»ωkηΪΦJϞυJΧΠοή(LΌή½α5S]ΣΌ’τ¬ιu’—“fX‡Ξή=Ύώϋ‡Ÿόδ'σ3Ζ‡κνœ{C€­*π—WφΝώͺΒσξŸύωΓ§?sθ_i™θ '€ΐ«~yψpφ³SωόŽΩ?.gΘRŽNΰέΩ_ΐzλΝΏΝ+ιηΜl8Ύ―ΏφηαƒΩT§rΦΩ_>3ϋAšB€ΆͺΐΫo½9Όύχ7ηέοη0ύ«/ušΧυ܏TΟρώ|S‘؟όηΓOϊΣγέGν @€ @€ @€ @€₯θ¦_žYšAθΒνœΡ_’W @€ @`ω~ϋΫί7άpΓς ܈ ψ(”5lΑ½EB`›©ώΪι5ΥίΧEΚtνtn;Ӟ|ςbΧNΧψΊ˜ΐΗ­?ω΅°XONμ³Φ:uΏ,ΊžOμ‘mwk¦{{s΅¬φτlšžΥ?½ΦΏjσŸ¬Η²Ξηfδ>Šuoζ*η @€ @€ @€ @€ @€ @€ @ΰ( ™υǏe9šνΡ\{,ΗτŸX7λΕf•ΣŠΣ±v8Ο¦Fr¬Η±ΨͺΪZgΌ΅Ί«· @€ @€ @€ @€ @€ @€ @€Λ(°©bοΎϋξαΐs§mΫΆ W\qΕόΨlU'žxb8xπΰΌϋ—^zιpΦYg͏½!@ΰΔxόρΗ‡>ψ`ήΡ―~υ«Γ™gž9?φζίxν΅Χ†—_~y~ρφνΫ‡―|ε+σco @ΰψ<υΤSΓ{ο½7olχξέΓ9ηœ3?φ†l5W^yeΨ·oίΌΫύ¦ŸΗ(8ήoΌρΖ°wοήy³gœqΖpωε—Ϗ½!@€ΐρxλ­·†ηž{nޜŸoΟ)Ό9Œΐ]»†Ÿύμg‡|²gϞα”S6υ+ ‡\ο€Λ,πΒ / ϋχοŸœώωΓ\0?φ† π%π―ύkxμ±ΗiήΟ/αp@€ °€'}8+K:vΓ&@€ @€ @€ @€ @€ @€ @€Ψ"'o‘~κ& @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ Ε.ρδ: @€ @€ @€ @€ @€ @€ @`«Εn•™O @€ @€ @€ @€ @€ @€ @€ΐ όjG!t±œ|OIENDB`‚pytorch/docs/source/_figures/graphgym_results.png000066400000000000000000017327001444262304300227220ustar00rootroot00000000000000‰PNG  IHDRΣƒœ’Υφ ΌiCCPICC ProfileH‰•—TSΩ†Ο½7„–Š”Π›τRBh‘K! $”‚Š QΑ± "‚ΚˆŽŠ(8@Ζ‚X°0(φ>Au,ΨPσ.ς3ο­χήzϋsΟ·vφΩ{Ÿ³ξYλ䏱8V OT ‰ d€€¦1πΏ ϊ¨'€αp₯bfll$@mbώ»½Ώ ±ωΊέXύΏš/εΕ’œΑ“rσP>‚9W,)©Bύ¦σ Δc܎2M‚6ˆrΟg³|Œ3Ζωέχ˜„82‡#Ι€LCύŒBnš‡μŠ²£ˆ'‘ΜCُ+ΰ 3yΚSσςζŽρU”­2ώ’'λo93”99œ,%οε»‚„Rq.gΑyΫςre5,ΠAHΒβΖκ‘gv'gn„’EΡ1,δχ4ΖYXβs₯¬΄ ζq‚"”ks£#'8SΒVζ)`'L0_?Α’ΉqΚZ™s‚9’ΙΊ²œD₯_ΐg+σ ’'ΈP˜=ΑœψˆΙ–/‘Ε)ϋη‹B'λ†(χž'ύΛ~…lεΪAB˜rοœΙώω"ζdNiŠ²7?(x2&Q/.TΦηΖ*γωΉ‘JΏ΄0^ΉΆύ 'ΧΖ*Ο0›;Α@’p ψσ ƚgΝ/³ &z«ψ Άˆk?•αμθμΐΨήΏί=ˆ~iW‚ή_žB‘8>ι‹4ΰΘ HO&}–'PΥΰBW&)χaΖ^X@j€t!0Vΐ8wΰ@01 €‚Ω€  Hΐ<°,e ¬›@-¨;Α^pmΰ88 ΞƒΛΰ*Έ ξ9/ΐ0xF!ΒCˆ ιBF9d 9Cž EBqP*”eA"H-‚–CP%T 퀑Ÿ‘cΠiθ"Τέ…ϊ‘!θ τF`2Lƒ ` Ψφ„™pœΟ‚³ΰ|Έ.…ΧΒ5pΌn…OΓ—α›°~ QAθˆ1b‡x",$IC2 ²)Gͺ‘€ι@Ί‘λˆy‰|Βΰ0T c‡ρΑ„a1\L>f f ¦³ӊ9‹ΉŽιΗ cΎa)X}¬-ΦΛΖ¦`³°σ°eΨjμnμQμ9μMμ φ=‡£γ,qΈ0\*.··· Χ‚λΔυαp#x<^o‹χΕΗΰ9ψ|~ ~?ώώ~‘ B0"8Bi‘„PMΨG8IΈFxJ%ͺΝ‰ήΔ"Έ€ΈŽΈ‹ΨAΌB$Ž’4H–$_R)›΄ŒTCj&#= ½UQQ1QρR™"T)V©Q9¨rA₯_εY“lCf‘g’eδ΅δ=δNς]ς[ …bA  €Q (k)”3”G”ͺTU{UΆ*Ou©jj«κ5ΥWjD5s5¦Ϊl΅"΅j΅ΓjWΤ^ͺΥ-ΤYκυ%κuκΗΤo«hP5œ4b4ς4ΦhμΣΈ¨ρL―i‘¬ΙΣ,Υά©yFs€ŠPM©,*—ΊœΊ‹zŽ:HΓΡ,ilZ6­‚v€ΦKΦΤrΥJš―U§uBKNGθt6=—ΎŽ~ˆ~‹ώYΫ@›©ΝΧ^­έ¬}MϋƒΞΎNΉN‹ΞMΟΊ έ`έέ ΊmΊυ0z6zΣυζιmΧ;§χr mŠΟξ”ς)‡¦άΣ‡υmτγτκοΤοΡ1045l18cπn`˜mXexpȈjδg$4ͺ2:eτœ‘Ε`2r5Œ³Œac}γ0c™ργ^γQK“D“““‡¦$SOΣLΣ*Σ.Σa3#³(³EfMfχΜ‰ζžζσΝζέζ,,-’-VZ΄Y<³Τ±d[Y6Y>°’Xω[ε[5Xέ°ΖY{ZηXo³ΎjΫΈΩlκlΨΒΆξΆBΫmΆ}S±S½¦Š¦6L½mGΆcΪΪ5ΩυΫΣν#νKμΫμ_9˜9€9lpθvψζθζ˜λΈΛρΎ“¦SΈS‰S‡Σggsσ ŠKˆΛR—v—ΧΆ|Χνwά¨nQn+έΊάΎΊ{ΈKά›έ‡<Μ<=Άzάφ€yΖzρΌΰ…υ τZκuάλ“·»wχ!ο?}μ|r|φω<›f9?mΧ΄__Žο_ΉΓ/έοG?ΉΏ±?ΗΏΑq€i/`wΐS¦53›ΉŸω*Π1Px4πΛ›΅˜Υ„…•υk'Χ? 1 Ι i u ]Ϊ† ‹Ϋv›mΐζ²ΩΓαα‹ΓΟF#β#j#GΪDJ";’ΰ¨π¨Q’Ν£EΡm1 †³1ζa¬el~μ/ΣqΣc§ΧMη·(;ž?'~_όϋ„ΐ„u χ­e‰]IjI3““>$%W&ΛSR§\NΥK¦Ά§αΣ’v§Μž±iΖΰL·™e3oΝ²œ5ΦΕΩz³sgŸ˜£6‡3ηp:6=9}_ϊN §3’ΑΞؚ1Μeq7s_πxUΌ!Ύ/Ώ’4Σ7³2σY–oΦΖ¬!Ώ ZπRΘΦ _g‡eΧgΘ‰ΙΩ“£ΘMΞmΙ#δ₯ηiŠrDgηΝ?·Ol+.Λσ½σ7εK"$»₯t–΄½€†Š‘™•l…¬ΏΠ―°πγΌ€y‡ηkΜΝοY`³`υ‚§E!E?-Δ,δ.μZdΌhΩ’ώΕΜΕ;–@K2–t-5]ZΊt°8΄xο2²œeΏ–8–T–Ό[žΌΌ£Τ ΄Έt`E芦2Υ2IΩν•>+λWaV WυvY½eυ·r^ω₯ NJκŠ/kΈk.ύΰτCΝŠ΅™k{ΧΉ―ΫΎ·^΄ώΦ {+5*‹*6Fml­bT•W½Ϋ4gΣΕjΧκϊΝ€Ν²ΝςšΘšφ-f[ΦoωR+¨½YXΧ²Uλκ­ΆρΆ]Ϋ°½Ήή Ύ’ώσΒομέΡΪ`ΡP½·³pη“]I»Ίςό©q·ήξŠέ_χˆφΘχΖν=ΫθΡΨΈOίΊ&ΈIΦ4΄ζώ«‚΄7Ϋ5οh‘·TeŸœώσ­C‡Ί{n>b~dλQκΡςV¨uAλp› MޞΪήw,όXW‡OΗΡ_μΩsάψxέ ­λN’N–žTœ*:5)ξ|y:λτ@לϋgRΞά8;ύl﹈sΞ‡œ?ΣΝμ>uΑχΒρ‹ή]ςΌΤvΩύrk[ΟΡ_έ~=ΪλήΫzΕγJϋU―«}ΣϊN^σΏvϊzΠυσ7Ψ7.ߌΎΩw+ρ֝Ϋ3oΛοπξ<»›{χυ½Β{£χ‹`”?TXύHQΓoΦΏ΅Θέε'ϊƒϊ{Η?Ύ?ΐxρ»τχ/ƒ₯O(OͺŸ=m|ζόμψPΘΠΥη3žΎΏ}Yφ‡Ζ[_Y½:ςgΐŸ=Γ)Γƒ―%―oΦΌΥ}»ηλ»‘Ψ‘Gοσޏ~(¨ϋqο'ΟOݟ“??χ₯ζ«υ׎oί(ς 1GΒω.tΐ™™ΌΩ%*ͺ‰I3Ζ5τwƒΖuw‰Ηuφws Ζ€«€ƒθ°(Fs0&ƒμβ’4i¦‹σx.2ͺ&±Š·ΰ;ψ*Q(F·)_Qݎά 3\»ύGӌoε¬ͺμλp(b'€ IΎ% pHYs  šœiTXtXML:com.adobe.xmp 1798 4518 1 ΄-d–@IDATxμέ€εωψρηv―WzG‘Ž ˆ]…Ώ… b¬1F4Fc—δg‹-Q£&ΖΔ h쉽+Š"PE‘ήα¨w\έϋΏΟ{ΌλήqwμΑννΜήw’ew§Ύσ™σΩwžyή€ 3 € € € € € € € €H@@@@@@@@¨H’ώšΜ§zBŸ€€$Ρ € €4¬€»φvοnν\ƒ; ή@@hXšΪΐu \ƒ7¬3kCZ€ϊ»‘EY4e$S©ΝS”J΅³€™hζ‰rsΜ† € €@“ΨΩυυΞ¦7y@@@¨§ΐΞ±w6½ž›cv@ΨYύΌ³ι T Vƒ$ŒΑ4QJw‚ …B²dΙY³f”••Ω₯SSS₯S§N];ϋέΝεͺ™ @@κ(..–E‹Ιƍν5x ττtΩsΟ=₯E‹u,Ι$@@؁-[ΆΘβΕ‹eλΦ­R^^.z ž““#]Ίt‘μμμ]Y%Λ €4‚@aa‘mCΩΌysΈώΞΚΚ²υwnnn#”€M €‰#@7OQK³jΥ*yφΩgνkΪ΄iU–όΕ/~!Ώώυ―Eίυ€δ–©2_@@@ *w==oή<™0a‚<υΤSΆ1(rα³Ξ:KΞ;ο<9ςΘ#%9™Ÿ6‘6|F@@`Wτ:|ϊτιςδ“OΚ<°Γ*.»μ29χάseΐ€΄ο Γ@ Ύ³gΟ–ργΗΛύχί/%%%U σ»ί]`ΫP=τ*γω‚ P»™ij·±S\#~~~Ύόε/‘ϋξ»Ozχξ-\pτμΩΣFu~ϊι§ς·ΏύΝΞ裏Ϊ©Ο€ € €υpΧΰ ,1cΖΘ«―Ύ*C†  žΡŒϊ”μ;οΌ#zν­Γk―½&#FŒ¨†X@@¬€»μ³ΟμuχO?ύ$§všœpΒ ²e+“©}΅<χάsςϊλ―ˁ(γƍ“ύφۏ€ώ~@8 Έϊ{Φ¬YrώωηΫ€Hm#9υΤSmλΧ―——^zIžώyΫ¦ςτΣOΛGAύηγΖζ@ΐ<Ύεqš2eŠ €:t¨άpΓ 2hΠ III±Kλ“°½zυ’«―ΎZώόη?KΏ~ύδ ƒβD₯-³!€ € ΰ\#vντίώΧœ}φΩςχ²οΎϋΪτς:―^ƒο΅Χ^rύυΧΛέwί-ύϋχ·Bny·>ή@@@ nw ½iΣ&yό‰ΗEi΄­ϋβ‹/Ά]«Ί₯;μ0{ kƎ+·ί~»4k֌vpΔ; ΠΘώΦ.ω&Nœhi4ƒΨε—_!έΊu•€€$["½§Ω΅kW›ΰ‘‡²χ4[·nMύέΘΗ‹Ν!€€HŸRΗ1s'‘7Κ+―Όbη<σΜ3娣Ž²4‘PHτ₯?N9ε5j”,[ΆLf̘aηu'©:6Α$@@¨A@―«υ©)4Ό¬kφG{ ^’ΆmΫΚ駟.μ>yςdY²d‰W―α@@@ zw ­™!Η>2Φf2ψΥ―~eitš^ƒλ{=DέuΠkυ•+WFΏζDhpW/\ΈPώωΟΪυkύέ½{7ϋ9dΪOtžΞ;Kεψξ2i${/SgpΛΫ™ω@`‚iv ΩqDyyΉ žιΣ§νβIηΠژοfΣΣΓQϊšv^—a@@@]Πλi Z?φΨcΓΧΩαkp©|²*77Χ>Y₯[Π§°tpΧηφ € € €@Τz½­έͺŽ9RΪ·o^NΫΑέ Χ-ZΘΐeέΊuRPPž‡ €ρΠϊ[³ωκCGZλ υv ιηϊ[ΫXzχξm§Ήϊ›6ΛΑ? €@­tσT+ΝΟ ρz‚ΉνΆΫ€°°PτΗ‚ϊ"r(++“+VΨQiii #'‡lTΙ@ ΑšΪΕ·kHJ°ΓΘξ €@πJξΚ‘]8?^JKK₯M›6φ¨TΏΧλσE‹ΩiάƒΦΝnΤΣ–„@ Α\—ΰ»ή=κφ0@ ΌR‡»λl½;mΪ4{=έΌys+\½ŒΔώ駟ΪiΪ9PGGjπY zέ―}uυ·ΙΌώϊλ6@mm(ZΟ›7ΟΥΥί΄‘ΔλΘ±]ˆ—@}λo‚i’8R£}Φ5LŸ>]ή|σM;Kχξέν{δI¨Ύ¦m1 @ΐΤνή8”S %%₯ΚΣ°n/#―±΅ύχί·™k\c‘›Oί©§#5ψŒ$†u{bGφΌ) κ.£Ad υ\oΨ–””Θ{ο½g'pΒ ²eΛΘΩΈώ’Α@ ρ48¦cǎ;lΠΥί:aΚ”)6˜fπΰΑα{ž‘ΧΦ‘ŸwX#@&*@0M”^O8np'—fώ§Ÿ~²}jŸ„ηŸΎvΨanVϋ)κ·mΫVcv]‡NΧ€AZeaΎ €>p]Ϋεδδ쐙ΛΕί₯"j¦ζΧΊ›ϊ{—Y<  ΧΈZ‡kΰJvvΆJT΅ξά]»ο:ΧW_}%wί}·]ΰάsΟΝd£ƒ›·Τ4τΫqΥΡ}ΦλπδδδπόΥηα; ΰu͐«υ˜^ƒ{yˆ¬»]9]]νΎGϋ^lκuMIOύ­σ!€€΄^Τ:\W233=WDWo»ΊZΏ»Οo½υ–ά~ϋνΆΜ§œrJ8ψ]§λ>ΡξΉ#J@ ‘\ύ­+YYY ΅Ϊ[O]υχ| ψΗ?μΆN=υΤpΰ€«ί5XR―΅k΄ EΧ­χ1έό5ΝΗ8@ΐΛ.&CΫΐSSS£.*Α4QREž ά Iož.[ΆLξΊλ.yι₯—LΤg'ΉπΒ ES`Ίytυ?όπƒάzλ­φΗ„.γ¦ι:υ䀁6ϊΓIob0 €~ΠΊL»ΩΠF ‡zHΪ΅kgλΉΘzΣoϋTWy] ₯¦Δ<묳€K—.φ‡„«ΫλZ–i €€ΧτΪtΙ’%ς‹_όBnΈαΟFžK΄žuίΏώϊkΉώϊλeƌΆμgŸ}ΆhcVδ<™'ώaΣ³ΝMf­»έΰΞ[ΪP”“-Α@ΥξYέ|Ό#€^Πϊ[»š4hόυ―΅Α%^+―»>vuwυςEΦΩΥ§UͺI )`»5j” 6ΜƒVŸο €€΄ζ›oμυχo~σΟ9²ήŽ¬«ί}χ]Ήϊκ«eΓ† ςΗ?ώQN<ρD{}ξζωξ»οδΆΫn³υ³ΓžΜσ©IΪΐ=w)μ’€ΦίzΏOο^zι₯»΄ŽX.T[ύύΙ'ŸΘύίΙ‚ lB ¦ΡΰtWk™΄ηϋξ»OZ΄h±ΓuΆΆύkΐ€ή€V@ΐo~ΤϊP»Δ;πΐm{q4uΑ4υ<Ϊ‘'—9sζΨF«I“&IΧ]είώ·μΏώvα ζ›ΙμΉηž6pFŠC_z²Z½z΅θς#FŒ°λˆlθ―gјˆ›€φ§½iΣ¦.΄γV FΨ°ώ€˜5k–tξάYΊuλVΩHΤΫe €@C θuiAaΌψβ‹φf¬^ŸzuˆΌΧsŽώθSUC‡΅Aλz-9ξGFF†τθΩΣ>iή7³‹`@~όρGyϋν·Eƒp΄{(­Σݏ*―P.@ΐ h}₯Ϊΰ½Ο>ϋΈΡžz¬“5›γ–-[Β7Wυ 0m€Χ'[#η«sτ•$’Ω!7nάhH’ώSŒ‰ ΰA­σ΄=XƒQζϟoLς`1ΓEru΄Φ½zΣαΪk―΅7‘5+δ%—^"ΉΉΉUΪB΄ \8 mάνσΚ•+ν:N>ωd›Q’6π03@ΐZ—ι΅λζΝ›εΫoΏ΅™½\lWk}¬7ήx£θCIšQμͺ«²ν n·ϊΐ―^½μuvd­υΉξ³.I'ΩdžwΛςŽxYΐ]._Ύ\Φ¬YcہλS^‚iκ‘εN.ϊiευ€Wz‚Ήώϋeψπαvmn>Χ ί£G›ώRΗ»A?돧™3gΚ„ lCΎžΘτ$Δ€ψE@λ9­·ώόη?‹Ft6‰ [¦_Ν€ ƒώˆ8οΌσͺd=°ψπ°€»Υ,‹ŸφΉ4kΦΜ>1κΕ"»kkmΔΧF } vξάΉ6+Αίώφ7ΩoΏύl±έ΅·Ϋ‡Κ‘‡*ςσ%ΈmμΧkπ—_~پƌc—'˜Ζ©ρŽ~Π†όuλΧΙ’E‹$//ΟsΑ€ήΦ>΄ΝCΫM΄[/ΎψΒd±lk8΄}τΡατς;uί~ ’:\π» δπ#'r§pΜ€^pΧΰ³gΟ–Η{Μ£hω\½ι΅²κυ΅f#xξΉηDhtΈΰ‚ l†Θ=φΨc‡rο½χή’ΧηΊ?npϋ¬%'NœhΧ£Y1ΛΚM0»FI2 €>pu™f₯7n\ΈώφbΡέ9E{Εψίώgλm lΥ―~%7ί|³τ4Ήy"Λ―™:κ¨ΘQφ³>ˆ₯η¬wήyΗf'Σϋ‘΄‘μΐΔ𰀫á{ϊι§Γχφ’-2Α4QJΉ“‹ž$^yε‰―OθΣ°·άr‹qΔvMnΎκ«­ž&HηΣΑwΪ(Ζ€ψIΐΥ_‘λ~*ξ–UΣ«υχξJ²<ΔC@λ. Š΄Χ°lΜvΧΦϊτΧΔIεβ‹.ΆLηœsŽ½οέ»w@:“­›“jΎΆv];ΉΊ[ίέω,ǁm"€υHV¦e―οr±žίΥΫڝ΅6Ί_vΩev“šW?λMΩW_}UžώyΫXΟώSϊχο_k]^[y5˘Τί΅ 1Ό(ΰΪƒέ5¨Λ¨eruΉv'ψπΓΫL:ώΊλ³έštθΠ!<Ž\[·W}Ÿέt=1 €~¨^—y΅άώ^»v­όη?±ν&ZΦΛ/Ώά>˜TS €Ϋ—ΊΪ·]{‰;q ξΤxG? ΈλΠϊ–™«Φ(Δά HΣ(?ϋμ³βϊ²=γŒ3l6ΔΤΑΝΕ*ΓσšŸͺΜξNΚξδTe"_@ Έ:ΟΥ[*Zl‹’ΥvδΓSΫ«qη@ύ[~֎ #PSΧ₯^z:Τ•Q»σxΰlVHέ{νηϋŠ+°Ω ά<ΡͺΈωέ5Έ~ΧΑv=Μ‡ΔKΐΥW.έΥcρ*OmΫύπΓmπŒ>ωϊ§?ύIŽ?ξxiΡ²…ν¦ι’‹.²έd?ώψγrί}χΙίώwiΩ²eέuρφkp—μΐν·σ¨­ŒGΌ$ΰκ¬Θ:LΛη₯vWFΝ`yΗwΘƒ>h ο½χ^Ρ€v͈ζζ‰ΖΦΝ[}ŸέψhΦΑ< €@Ό\εκ²x—§¦ν»2j Νίοϊ»όγΨΩξΌσNωνo+­Z΅ͺwύ­+ˆt€έ¬nΨ°ΑΎζΜ™#ΪFRΧ0i$Ϋk]σ0 @@„–ζ:ώ \cOFF†\ύυ’žž~ ͺϊSUΊ_Ÿ¬κΩ³§€€€Τ±f&!€ € €@mšiFŸ’΄Δ©i©R2 ΜPΣ5ΈŽ+((ξέ»ΫλρΪΦΙx@Ζp ϋ‘[ΤΊ]λμM7ΙψργeώόωrΚ)§Ψμ‘σmΪ΄ΙfΦuΈA?λω`αΒ…vTδ47ο €»' YΓφΪk/›™&²k>ΧFΉvΫ^\d3ΣDŽη3 €@γ hύ­½eάxӍ’’œ~ΘΏΆϊ[»τΫsΟ=Ώ lπ‘Α4Q΄αΓ‡οπTU]‹Ež "?Χ΅ Σ@@ό!yk½Ψ3½Ωzϊι§Λi§υ8QS1# sκu²ž;΅nΧ¬>φhψIΩσΞ;Οވ<·~ωε—’ν05 šυF‡Θωkšq €Ρ Έ:;//O.Ήδ’zΥ±nYχύV™@`w\έ« τΊΊ>ΧΘnΩέ-Λ#€‰.@0M”G˜K”PΜ† €$ΈΧ…w€±nΧ΄σ<ςˆ\wέuvSϊω¨£Ž oΦΥω]Ίt±]m»οfž;wόωΟN˜LdΞ&Œΐ@ΐώυ@Q( €@=¨ΏλΕ¬ €@”ΣD Εl € €¨€vλΉjΥ*iΦ¬™}‘‚ €; Έ@mΤ_Ίt©άwί}ςόΓΞψπΓΛΩgŸ-iiiα4τn ΪmŸΎj>ύτΣ„ ¦α†GMG™q € € €xC ΰbP @@Ό/ ύP?ύτΣ2rδHΉε–[lP–Ϊ=5ού= „ €Δ^ΐθ–fϞ-W_}΅ €Ρμ2O<ρ„Œ=Z4}δ|‘₯Š<―κηP(d'λy8‘†βββπΎ%~±/ € € €$‚Α4‰pΩ@@˜ Έ›zkΦ¬‘Χ^{MZ΅j%?ώΈΜ›7/¦Ϋeε €ψMΐΘθϋδΙ“ε’K.‘gžyFφέw_ωίώ'ΏϊΥ―jΜHΉŸ‘["?GΞγχΟS¦L‘›nΊΙmήΌΩήπϋΎQ~@@@A€nžα(² € €" OƒAINN–Ž;JIII£l— € ΰόςαGΚgœ!ˆzτΡGΛ­·ή*‡v˜έ pυ>%UΞιΦυr›Ρν·Ο<ϊθ£’έV-X°@φΩg9δC˜ΖEGi ^πj]šnH/‚³³³Eϋ8-++“’’"ϋ]wί-S•‚o € € € € € ΠLG…Mq·Ωg@@Z|Lγ‚b4 γάΉsm¦ŸώΉ,^ΌX>ώψcyΰdτθΡ²bΕ yδ‘GdΠ A2pΰ@ΙΝΝ% ¦–?F#€ € € € € € € €MYΐ·Α4.Fežzκ)ΉζškΒΗ±GRPP`ϋ:Υ‘›7o–ΫoΏέNΏβŠ+δςΛ/—]»Pγ Π4άΉ£ϊήκxH_W]†ο € € € € €@γ ΈΆZέ"ν΅ηΞ–@vπe0»Ίnέ:Ήλ»δž{ξ‘vνΪΩnœϊφν+ωωω2ώ|џt(+-³ούϋχ—{ο½WΆmΫ&ύλ_₯U«VΤXώA[ΐ7܏/ν ЍΣs…οΖ%Ά{‡ € € € €xSΐ΅Υz³t” @ ) ψ2˜FO€₯₯₯ςΜ3ΟΨ@šN:ΙΉηž+§œrŠ΄nέZξΈγω裏ΒΑ4½ϋτ–Ο>ϋL}τQ›±fμΨ±2dΘ9σΜ3Γ7P›Ag_@¦$ΰdΚΚΚdφμΩ2mΪ4Y°`lέΊUΣ€SηΞr葇Κ ™™™α ›¦dΔΎ"€ € € € €^X½z΅,[ΆL:›vΫ6mΪx‘H”@ ‰ ψ.˜Ζέ]Ίt©Œ?ήΆ+―ΌR~ϋίKVV–ύžž‘nί]τjzzΊrΘ!²Η{ȍ7ή(?ώψ£|πΑ2jΤ(Ρi  €‰)ΰΞ%%%6σœsΞ οh‡D» tΓίώφ7Ήΰ‚ €Y³fΤ8ή@@@@@ Έv\ν‘βΞ;ο”{ο»WΖ\9F»ξ:z™ˆ±=«G¨] ²€Ϊ§{vΚΒ… eϊτι2hΠ 9α„l žl5σ@E¨’JΉuΌΎΪ·o/Ώόε/ν΄~ψA–/_n?λ4@ΔpA•oΎω¦h Ν^{ν%ϊΧΏδΫoΏ•3fΘΌyσδ±Η“Θ5Χ\#O>ω€Ν|ζ–K<φ@@@@π¦€f€yχέwεΈc“·ήz+|Ο›₯₯T €‰.ΰ»Μ4ξ§žPu8p hvtš›nGlGΗΉ€™>}ϊΘa‡&sζΜ‘5kΦH·nέ"gε3 €@‚h½―υΏ¦}α…μ^iΐΜyη')))φ{»vν€G’››+§žzͺ<ψΰƒrμ±ΗJϞ=ΙN“ μ € € € €ψC@˜ΟΛΛ“ςςrϋί@ˆ—€o3Σ”––Z³ΜΜLINŽ>&(H0΄'βx‘³]@Ψ Έ JΝBφΤSOΩ j63 € …B6XFίu8θ ƒdψπα’YΛΦ]kΗ™œfφ@@@@@Gΐφ@a”Τχš oœR°@D’Bρ˜–fΠaΡ’E²eΛIKK gŸ©©¨.CΑ’%KlχP}ϋφ΅Q­5ΝΛ8@ ΈZmΪ΄‘ϋί’Α”ϊYζ¦λw ²ΤΰLάΣfϋ@@@@@@@ i ψ.˜ΖΕtιΕ©gŸ}V~σ›ίȐ!CμQ—eΐF__zuΫΆmςΞ;οHQQ‘νή©sηΞn6ή@L@ƒe΄ώοΤ©“\tΡE6ž tp4Ωk>ψΰ;Νkκ4ά»ύβΎwο¦σŽ € ΰ]wύηέR2@@@@/ψ.˜Ζ‘i0Νi§&Ο=χœάύ’““#p€νΎC3 θ ο.ϋΐ¦M›δ…^I“&Ωin—Ρ/ξ¦ͺΐ? € #ΰκw}wη·sz#Eƒk6oή,Ο<σŒΝrvΖgHǎν,ξόαΦα–sοšb'`»ZσPokZΝXU`·cΑΌV^S홁 [±ϋ eΝ ΰWΪνόΊ?”»nΚψψΟΫu/»©Z=;Χ°―γuΣΫƒϊυcόΝθRP@@@β!ΰ»`mψFνͺcτθΡ6˜fφμΩrα…ΚW\!{ο½·¬]»ΦZY³FζΝ›'«V­’—_~YyδiίΎ½μΏώrάqΗΪyt]4¦ΕγOm"€ρpuΏf,{ςΙ'εž{ξ±…9묳μωΕM/((Ÿ~ϊ)άυ“Ξ€ΣRSSεΗŒί°eš€€νjΝC7o\ΧoΙ½|,X¨"$Α@ew ©2σU84μ" ΰkΝT›’’²C ΅―wŠΒΧ*ΰ΅ΰRwi“œ\ωπ—\_ƒ•Χ"Wj=–L@@@@ ±|L£@.ψε裏–ργΗΛ9ηœ#νΪ΅³έ=ιτ~ύϊIίΎ}εΕ_”'žxBΎϋξ;iΫΆ­$''Ϋ.žΖŽk»yŠ\—~f@H|(³uλVyτΡGe̘1v§|πA6l˜ύμζΡξŸτœRΧ σ2 €@Γ l(Μ—ς’πu_Γo‘~k¬ΌΡ”΅[WKyEΉ½ρ•‘’!ZΞόmλlН»F­ίš~ξ δ“•–#Y©Ω ΏrֈψX`κΤ©ςόσΟK―ž½δΜ_)͚5³υΉWκoΣzΆθkσ ₯Έ΄Ό2»ŠG.Ϋ&pfεκ-ζΪ!dέ2Rdέ†BYΉΆΐvMλLι …*$'+MςrΘΘι…cB@@@_ΐ—Α4Κ€7443€vΙ‘Ωf4@ζ₯—^²]v|σΝ7U$³³³eυκΥ6{Ν\`3ΣT™/ €MBΐΙ¬[»NΖ=:Nώτ§?ΩύΎλ»δμ³Ο–τττ*7T4sςδΙRZZjoθλςξό³hΡ"9χάs=s£ΏI@v²Ιh0Θ›ίOζ­V“3νwρέωΚ»oI$Ω²¦PΆ•šn*€yN ™ΌΰM™Ÿύ•”—‡Άη«qϜǧā€ l.]/'φϊ΅άe`•:->%b« €@|άυŸROœ8Qή}χ]Ήχή{₯_~rψα‡Η·pl=¦ 2ώ•οdΪχλ%/#ΩΓΖ9šΖmήd³+Ϋ–/›ΆΫk‡fΝ²εωw”7ΎΝ–J“ψ^NHŠΉζY²‘DΥSF ξΞυDLRY9 € € €€W|L£OŽΉšΓ‡·™h.Ίθ"›…fεΚ•RXXhk233₯S§NΏιΥ«—΄hΡΒ«Η‚r!€ΔH@Ο:θΉC»gΊϋξ»εα‡ΆγώύΐΏεμߜ-xιnΆΨ 柼Ό<2dˆϋZε}Α‚UΎσN@‹uέJ…ήMrwŸnυ[Sεug’)FυϋpϊݎΣ"z€k†κe¬ίΎ27 ˜LS\\,]Ίt±Τ7n΄;JVšΔ<ή‘{0ΧζL.qοΡȘ sA‘ε©:hωtœNΧ)φŸͺ³4β7½€ Ϊ ŸFά(›B@@@ΐcΎ ¦qŽξF‹vσ€―‘C‡Ϊ΄ΈnΌ6Œpζ7ή-OΓ™“ΰH\WΧώωηrΛ-·ΘoΌ!mΪ΄‘ϋοΏ_N:ι$›ιLΟnΎH‰κ獐yZ4 JIIIδl|FΠΫGϊί£ύŸωβ…ΰ[?h™τξφAoyU–±κx7=οΆά+<@|-πε_Κc=&x =g|ϊι§2eΚ»O‘4ϊYΟ‡~ΈΧΧ;Mα@@@@@@@`—|Lγn|ΆnέZξ½χ^™1c†|ωε—²zυj›‘ζάsΟ•]»Ϊ¦%%%²fΝωμ³ΟdκΤ©§OΙΜΜ΄Yh4πfΓ† ςέwίΩ,7`sόρΗΛϋοΏ/GuT•ξ vY—@β& η =|πΑΆ Σ§O}M˜0‘Ξ27ΞXΊσM33@@@@@@Nΐ—Α4³Η{ΘΰΑƒeςδΙςύχίΛΥW_-§všŸ››k»q*//—‚‚YΏ~½ ¨Ήϋξ»eζΜ™ςΫσ~+'žt’ ˜Ρ šiΣ¦ΙΏξω—Ν^3vμXι»o_iέ¦55 χηΞ!€@SΠ¬eηŸΎœ~ϊιαnόj ’ΡρzήθΠ‘Cxή¦ζΕώ"€ € €MYΐuά” Ψw@@@JίΣΈ΅ωωωςΐΘK/½$ωΛ_d̘1’““SεΈκMΤΌΌ<ϋl5mΪ΄±7γ'C†±™j:vμhί›7o.—\r‰<ϋμ³rωε—LSE’/ €€?΄+Ώ^½zω³π”@@@ Qτ!‹²²2Ϋ%|jjj£n›!€ € €xK ΰ­βD_š~ψA}τQ4hΝ8 4‘PΘf“Ρ€7θg―ƒΞ{ΕWΘΧ_-oΏύΆ9νΈγŽ³σύψγvšή„e@π·€;h}Ν+ςβο=§τ € € €ΡΈίK—.•{ξΉGξΌσNΫ5|4Λ2 € € °«ϊ[Δk/·/I’δ>šχΚψ SZO•7’€1ωθ»Μ4{ŽŸ~ϊΙ‚qΔΉsgϋY§ΉιNˍΣ?Β΄΄4›…F§Ν;Χvη‘’’bgΥ 6ύϋχ·Ÿυ‡sqq±dddΠΥ“α@ΐΏξ<ΰί= δ € € €±pν‰~ψ‘\uΥUΆM0 JϞ=EΫ΅]ΡΝΛr°n@@hZ^ύ ₯Ό’ά -cr 2¦"Τ΄’‘ψ.˜ΖύηSPP`?jWNυω#sσ–””ΈUΩΔϊΓ8++ˎΣtϊ#™@@@@WΐΚθƒu›6m’HvvΆhΫcQQ‘ ¦Iά½gΟ@@@ ^ΕeE’_°~{¬Cόc\x„φή“_Έ^²²-Miy‰¬άΌ\B™₯&YIΘ”7^bΥ·›$-³ZKJ°2Π§ϊΤ†ψξΫ`Ν$£Γ¬Y³dύϊυ©S§ZŸq?Š΅{E‹Ωε233ΓA8`£Α56l°Στ³>}Β€ € € € €‰/ νƒ°΄΄Τξ°{(/ρχž=D@@Ζp± «7―”Ηfά#iΣ[Ξφn”« 5ng{ψ`ΫΕΣ©§ž*S§N•ιΣΏ”/MΆ7tμΨQ(‡z¨ 6Lϊφνkƒd\°LΛ–­δΖo”ύχί_Ž<ςHiΡ’…]”@'Θ; € € € €@Γ DΜθLj― · Φ„ PoFxΊΉήeb@@Έ ψ>˜Fƒb΄[§³Ξ:KŽ?ώxY·nJyyΉ šIMM•ζ-šKΫ6m% ZθPEΘ΄STΆT{μ±rόˆγ%9XIα‚lβvDΨ0 € € € €@ lΟ<γe'Lšμ€ΝŽ\Ή³DΥ$ΰAg—@ΐƒ‘΅­‹ŸΡξ’ΆOˆœξΑβS$@@Fπ}0:…B!ΛΥΌysΡΧΞ†€6Vl4ΨF ’сŒ4–@@@@T ςφl hΪ’ΜΫPY‘i*–@0Ν<ςe‚j€Μl[Έ JΞΚ@ΐW.°₯1 ­΅νφG»9}ψΆ’’²6a51/Bμ·σ]` € €@Β ψ:˜Ζe‘©γcͺ= Ί)++“­[·ΚΪ΅k₯G6[[OυωωŽ € € € €@ƒ ˜‡Ή‚©Y’ΣρPΩ²ό IΝι$iΉΝ έΖΌ…ά`{Ê@έpmσ‹–o”}°@2Rƒα_w{塬@kά@ (Λ.μ¬4)7χ ‚Α€Όψξςρw₯φαέX»hφ›² iΩ,]N?Ά·έn-Εd4 € €@œ|Lγ.²50fΜ9²~ύzΫ½“ΛT¦Ρο:Ύ€€D6oή,3fΜ°σ<ρΔ6˜¦z0Nœ ›G@@@H`άϋKV›½ΝΝάI ˜ζ9Σ%9Yiψ€³k P«€Λ³yk±<σι ι’—"e!bΣC“{ΖΣ”nΨ`οT¦¨I’σς%yE¦Ωve&όZ ½&iώΦ’μ»gŽœvΜn¬ˆE@@b.ΰΛ`H³tιyπΑ‡δΞ;ο¬7ΤρΗΏCΠM½WΒ € € € €μ‚@0%Cϋ7KΖτŽρ.”ŒE@Ζ’€cnŠ4ΛJ6™bb½}L“”­…A)p›2UqNzP2²SLΥ\nΖΖ&7ΩMIO IvΊΉ5›MΈ=β@@`7|LγiΆmΫ&=φH3`ΐY²d‰ ŽiΡ’…δζζš'{’ŸŸ/7n΄έ:uθΨAΪΆn+!Uή±cG9ξΈγ$9Ωw»Ώ›‡›Ε@@vI ςWdƒgδη]Z1 !€ €@“ˆmΪ…&Λʎ#€€?τ'W™ ’1=.™`šΘ`±Ψ ¦©0χ ͺ»ά|ΧmWΟ|_uέϋVa~Cκ6Bœv’₯@@Fπm4Ι’E‹dάΈq²Ο>ϋHff¦άvΫm―_?™?ΎŒ=Z4σΜΝ7ί,yyy²|ωrωψγM›₯Y³frε•WΚ/ωKpΣΖl@@ΐοΥ‚fζqi|₯ύΣο–ς#€@Spμ4uφ@@/ Δ2ΈΝ‹ϋK™@Ό%ΰΫ`šΕ‹ˊ+dέΊuςςΛ/Λ±Ηke;wξ,C‡•ιΣ§K―^½€Gvόˆ#dύχ—σΟ?_^{ν5šy&777l•žž.:t°ί5{z²Υ€O3Ψόλ_’)S¦Θ™gž):? € €Ρ ΄Ϋ§•΄ιΩBAs#Φόί>Aγ†₯hΛΖ| €@|*δΛΕΣdνΦΥ’0™»l†šψ•Θζ₯1h}½aνFΩ\΄ΙΞ΄Θi)ί,J‚sΛ+λpSΔxWγII&λYY‘tmΩSφnίΧΘ¨ΏΏΆŒ € ΰV|υ•δ8_’ΣΣ€"T{ζG7Ύλš“Ν½ΌE Jjz†„ΚΛ%5#C™{yYΛ–HYΉfŽΡ`μ+ΜφR23₯λQΓ$˜ͺΏe@0η&Ώ"Έ ˜”” ƒαέΠΦΦ­[ΫFƒh"Ÿ@λΦ­› 0@ζΜ™#Λ–-“ξέ»ΫΖ3]Ž@@ θΩ’1sΖi§Ε`@ΐkZ%~ΏfΆΜ\ύ™d₯˜Μ]‘ΈΡUΡϊsΏhS©–˜ψ$IOI“ωληΘ–Ε+M+‹ο`R²δ―–_~eƒiμω%ή…ŠλΡcγ € €qΠ es!­4 Ÿž(©νΪIEYYL‹0Ϋ[SPhΊΧΪϋvΑδdYύΕη’υέ7rξ±( l™ψ3;v”.ƒ‡HPL0ΝφύΕζX' €€|LγϊDlΦ¬™U^·nJ^^^8p¦M›6všv΅uλΦ*™k2L$«f¨ωζ›oD—Υ`@@κ%ΰξΠΦk!fF¦"`3VmίΩ¦φΰFFr¦4Kk)ΙY¦Α;ΎΑ4ξο-)$Ι)Ϋ$`ΙuΠ†ψ΄dΣ6Ϊ<LγζΧ{0)(!σΏ”δΤxν"€ € π³ΐφΠ5#€6oa³·όšf€Ρnτ«A―‚υwT…Ή—9θwνς)ς…Θι ςΩl7TϋXuΫ ²nV‚ ΰkΚΗ²|Έ νΫ·—‘#GΚΜ™3eάΈq6ΣLAAέ“V­ZΙ‘G)Σ§O—?όΠfΡ@š-[ΆΘ[o½%sηΞ•]»Jλ6•έAωpχ›t‘υ’)¦NMZ—G@@έΠ,¨σηΟ·ΏEww],$€€F²yε΅=n‡Ά¦*νO*―Ί1 € € @άϋM ƒ™`»β»`χ„ŸΣός—Ώ΄‡γ½χί—C9DώχΏ—μχΦ­[ΛqΗg?ί}χέςΐΘδΙ“εώϋ﷟΅‹¨ž={ŠΛnγΦ™`Η6awGΗ,a/;† € €€/άMΨU«VΙ­·ή*ΗsŒ<σΜ3RΓ§7} E‘@ΠQy%mο.œ"I€‰ρΩώTΊω¬γ*‹kšP=R^[[*ώAπƒ€»AμήύPfʈ4Άχ~[œνE+ΰΛnžτ’C£:ρΔεφΫo—?ύιOv7lΘ·KΣΣε„NηžN–-]&7έt“‹Žοή½»,[ΆLN;ν4IKK³σ˜νŸKόηΣ†θ₯K—J™I+ΈΗ{Hj*ύΙΗ¨P@@p?όπƒLœ8Qϊχο/―ΏώΊ}ΠCqΏcέ|Ό#€MU ¬¨PJΆl°m{^0ΠD/I$)).‘‚ΝLI·A5妜…k–JrNŽ Œ y"†EΟ%ΰ“ήΜdΫΦΰ@Ο pΙσ‡ˆ"€@œτώβΕ‹ν}όN:Iΐ΅3 ΰ_ΣθΕ‡ώxΤ 3—]v™tΠAςΚ+―Hοή½νqΆχή{ΛC>$wάq‡ΌόςΛ’E ΙΟΟν κΉηž“!C†xεPŽ(\ΓσŒ3δΪk―•Ν›7Λ_ώς1bDK3  € € Π8₯₯₯];Ϋψ£έ kmd²&@IDAT£ €€φκ€)ΩΌdžότܟMJG©•šψw[dr Ki($ωsΦJznž©Γ“dλ²ω2ΩΏIVjPBqηAνB%…’Ω±τ:γjI2η@ο lΪ΄I>ωδϋ€πα‡.­Z΅ς~‘)! ЈoΌρ†MŒ‘=Κhύφۏ’џMΥ-ΰΛ`·KΪH™-Γ† ³5ΙΙUwG»~zψα‡εςΛ/—Υ«WΫy{υκ%]»vΧhΈυρξ]  l4L³hΡ"{Αω駟ΚQG%Kο:J† € €@“pΏ]τχ&Ώ9›Τ‘gg@ ^¦;%ΐb«lΥkαΩΖɘbTT˜'`#cfμg} ―ςοD0&')žvCΥ »ΟΚ@b,πΞ;ο؞t3ωΟdτθΡφ‘πo–Υ#€žpν%šcκΤ©’c²AΚ5ΛΎ4Σ/Y½<}ψšTαͺFŸψ`Χέ\σΒ /H=δΰƒ–ζΝ›‡Kω˜>¨―κƒ[Oυρ|χž€;VΪΕ“Ύ΄BMII± Σ`Γ€ € €xE@Ώ0 €μDΐD¦θ*jv2oŒ''%Uv'/ζ½Κ`{Q2₯΄eΥivD•Yβς%ήQ=qΩi6ŠψKΐέΣ(**²xΰΆϋ}PxΫΆm’™™iοoDήΛςΧRZ@ a4q†Αδ ν‘Fοϋκ½ΰκ 4fk¬ϊ ψ.˜Ζνβχίo»xοο½χž}τΡ5^|Έ†L½(‰όΜEŠ“τΟ»;fZ‰†Lκ]ά8μ%E@@@šΈ€ <Τ•j,ρb©#δ³₯ΤПψΫƒyβ]Œψ"°uˆ³€»Η’Ε m>Ίƒ€€€ΔzιηHΓθΦΐ\ €@β θΉDλŊP… ’Ροœ_χxϋqΟ|LγώZ°`€¦¦JΏ~ύ€C‡΅Ϊ»ωu†ΘΟ΅.ΐ@@@@@@*άc©Βυ@γή£^@&&`kβΔήΔΠkΫ]$ζ¬­x5ήwΑ4Fi4šW»ϋΡ—š­D#{£Έπ‹VŠω@@@@@@ ) lέΊUζΜ™c»,κΥ«—€₯₯5eφ@ΔЌ˜Pγ^‰»§uξYτ‘'u¦ρ&ΊΘέώύϋۍN›6M¦OŸn?ƒA›}Fƒd’y5^©Ω € € € € € ΰO½7σςΛ/Λ!‡"'žx’L:ΥgγΟ½’Τ € PE DcΊΰ*―€€y7έp%Υ#‘I•υωό‹ο2ΣΈl2ϋ쳏ΌςΚ+ςχΏ]nΊι&)**’ƒ:Hš7o.™™™6[›·¦c€lΣΣkšΔ8@@@@@@ Ι h°Œήk)((Ω³gΛG!λΧ―—ΉsηΚ‘G)ΙΙΎ»ΝΤδ) €Τ& ±4&~FrΪdJZ^ͺl[S,φk%iY)h}w•γ.άΦ¬Y#Λ—/—ξέ»ΛΊuλδάsΟ•6mΪΘΰΑƒν»ΚΤL£A4ΕΕΕ±cG3f z΅ύ—Βx@@@@@@ΐ„B!Ρϋ+εε咽轧ƒ»ocΏπ €ψVΐΔΟJΘd€iΦ!Gφ?§·o-±ŸA“‘F#mšΨ°KΑ4 uTSΐK]Ηΐ]”­Z΅J.Ίθ"ΩsΟ=mpL—.]€€€DžώωΊO6l˜\qΕαο|@@Ό(ΰ}ΌX6Κ„ € € €MG@Ϋ(τε‚hܞΧχ>[Žw@π¦€vοΤ’sžˆ ΡnžτΥ‡] ¦iθ £zέ$2LνΚιΨc΅έ:i$΄Z.χ²#jψG#§ ₯g―ž6ŠΊ†Y… €xF ‘―»<³c@@@|'@;…οF@`—Be!Ρž±=Fc—Vδγ…κLSVV&3gΞ”mΫΆΥ;E/²4˜%55U²³³₯mΫΆ’——WcwL΅™’vR·nέ䩧žͺmΆŽΧ4„ϊb@@Ό* Χ]šyOƒˆ@@@@@@F04M4†&Μu0ΛS\T$Χ\s|πΑα•ΤχC§Nd―½φ²―C=TFŒ!{μ±G½ϊΥLII‘–-[ΦwΣ̏ €xZΐ]s­_Ώ^&L˜ σζΝ“ΣO?]μιrS8@@@@@@ Q’¦q;¬©|²²²€wοή’““c³ΚlΩ²EΎώ{›ι%ΣL3Q1RXX(ΪύR―^½lφ™R)..–9sζΘ²eΛμλγ?–'Ÿ|RN8αωη?)šmΖέ@rΫ«λ]ηuCdjAΉžΘi:δ4·<ο € €€—fϞ-cƌ‘>}ϊΨk—ύχίί^{qγ₯£DY@@@@@@ ’¦q)ιιιrγ7Κ›oΎ)'N΄OKsΜ1rξΉηJΧ]mΰŒBmάΈQ/^,}τ‘Όϊκ«½{w6l˜\yε•²uλV;ύΣO?•3fΘ+―Ό"­[·–;ξΈΓΎΧη&‘+Χ† dΙ’%²`ΑY±b…θχAƒɐ!CDŸμΦντθΡCΊtιb€κ³D<πμ €xSΐ]Ϋh°²/wμΨΡvυTPP`ƒiΌYjJ… € € € € €$Ž@ΤΑ4ΊΛ€’œœ,Ή9ΉςΙ'ŸΨ@š»οΎ[Ξ<σLiΫΆ­*2:Yg%“&M’?ώρrΠAΙ°αΓd―.{Ω¬5τςτΣOΫ¬4=φ˜νξiΤ¨QUΦQΫ  …dκΤ©ςΤΔ§δΕ^΄A::MΗ?ψΰƒ6˜fέΊu’?Gq„ ζ9r€€¦¦’‘¦6\Ζ#€ €@άά5ŽΥhfΏ²²²Κ²4υΞIγvDΨ0 € € € € €45ͺΡ/uμ½»±SRR"ONxRή}χ]ΉφΪkε·Ώύ­΄oί>H£σιK½ Τ];Ήΰ‚ 䦛n²3/<‚νξ) JηΝεΒ /@σΥW_Ω'―u9·ŽΪŠ€σθΝ₯ηž{NŽ<ςHϋΘX›F»–κίΏΏ],%%ΕΎλ“έ:lήΌYN9ε™0aBΤΫ± zόŸ%πKœτ/*2[ΗBFtœφsBDξy@V@ΟmξzΘΎs*HΨc͎!€ € € € € €€·’¦qΕ^Άl™L™2Eš7o.'žx’νΦI³ΐΈAƒ\τε nΙΞΞΆYgtάk―½&Λ—/·“uZ^^ž >ά~_΄h‘δηηΫΟΡόσΑΨ¬8{챇 :T}τQΡ£ `O T–C§ί{ο½6ΞΎϋξ+ηŸΎ|ρΕΡlΒσŒwBΏΜqΤ#™ž–,ΙΑ@ePΉ‘4Ÿuœš)‘ "›²;Μ? € +@Ÿ°‡–C@@@@@ŸΤ«›'έ'νšιλ―Ώ–Γ;LΊtιbw³›>. %77ΧΉh NAA]ΞύΣ©S'ϋQ»c***r£k|Χ'³u{kΦ¬νJ-ΛυΧ_/}ϋφ΅ί_~ωeϋHͺŒjΣ¦\rΙ%vϊ 7ά`§½ρΖv9͐γη‘`[©,\ΆΑΟ»°σ²›ΐύ;¬H«Φm•¬ΜTΣέX@6l*’™sWIvVŽνΦΛFάμ|mΎšCƒˆΚMφfΉι²Gϋ<_•Β"€ € € € € € € ΰGzΣhš΄΄4ΩΊu«ξ|Ÿ΅K Yh4«ΥƒoάwχngΪΙ? .΄]ΰ죃EQ@¨w0M»vν€OŸ>2uκTωόσΟ₯kΧ68FƒlͺΓhΐ‡vΏ£Γwί}gƒW>ψ`ŒιΉtιRϋ΅uλΦ’™™9i‡Οn‹/ΆΣFŽiΛ£_"·Ή .γ‚O=τPΩgŸ}δ‡~°Α=Ί?~΄k£φΩΙ’jμ$ζ₯EeΓ…9†&^fmr’ΫμD"Ι&Π€UV²S’Νρ5 ΨΌ4ϋ¨;ž–ZοΩόόgMΩ@@@†p?™νƒz›œ¨ΠAΫ™SL&δ ΙTˆ>ΰ”bΪ5M3TΆ8qδ@π€ήΣΣKww?Π»%₯d €‰.Pο`ν’iΐ€6˜ζ‘‡’.]ΊΨξ’\ΠL$˜;Ρ}όρΗςΐΨI‡~Έh·K:hΆΝσφΫoΫο˜ΣΌysϋΩ-kΏΤπΟ¦M›μ؎;JvvvTΛθLΝ›7“V­ZΙ΄iΣl·AvA£&εζG}Ή 41ιYμ|οNέE7έΩŒν  ΪΠaχ{ϋψΚg†κ^…ο¦š}- οŸοJO@@ˆ™€>0α©!ςΗJΈ`ϊf{9Ν»—JlVΩ ’paϊƒm³°Ώ›EB₯ζa }pAχίόήβ.yBzv¨΄~Φl4ύ;δΘ“!x[YHέ3ORΝC]jEΜ„ώ°Gώ->%G Qτ2½Τ<ΌŸJI¨\ζb<εθ² €€’¦ΡΰmΤΜ1£F²Α1λΧ―—?όαrρΕKώύmŒNΧω΄¨uλΦΙΜ™3νΌϊ]‡“O>Yςςς€¨¨Hf~=S^όο‹ςώϋοΫit€€€ΨεwL£]Mι A5ΕΕΕ’ššjΏομŸυλσe͚5²Χ^{IzzϊΞfχΕτπz/΅Μ6Έ\m ΟnΌξ|^R™έ ί7e… € €ψW`gΏ}ΟΆ¦‚&s¦iό­’$œb?&mΟΪΪθεbƒΆ!5+EZvΟ“ ΆJvΗ Ιk—]Kc|π—$Gκ) υ€ΖΦunž!ηάΡΆΕh–Ϊdκ ΙμψRΐ] ˜wώ£χε€Π‰%Π*=Cz˜θgmή,ϋ™{‰-ΣΣ<υPBbi³7 €;ˆ:˜&rE”±cΗΚ\ {챇όζ7Ώ‘ή½{Λ~ϋν'ΉΉΉΆ‘jΓ† ςΥW_ΙO?ύ$νΫ·—•+WΚSO=%ΪΝ’ϊύάσΞ•ΉsηΪοW]u•ΝpcΏΤρκh£©f€ΡaΚ”)’έDν½χήζGoΘn»ϊβΪ•ΛœσΕ_Θχί/#FŒ:TŸ•ο € € €•š¬ωkYQΡ†΅RZΈΩ«xα\§†Α`PΆlά$εΫ l½œb²όoZ'[–ώhΞΙ¦-%†ΥtΤ‡Ν^2„$%#»IeχΪ§ ΜXο`5ъΖ‘hΐŒΎ48₯Θ€]Σ Α1šΑΖe…Ρω#Ν$£/7ΈυΉοu½λΌΜ£mζΝ›'>ψ Lš4Iž~ϊi4h,[ΆΜο|τΡG²xρbωμ³Οδ½χή“}χέΧvρtο½χΚΑ\Χ&˜† € € ΰeΧͺͺοUnz Τp?eΗ1?OkΜO±k°oΜ½Ψυm5υύίu9–D@ 1’ΣσΜ 2“‘ΖΧxαζbb*{g―&ψΆ|½jšd₯δ˜`sΜc5˜ί'―S^’ΉkKVzΆ YΈαGyκλG$9έd«ΡΝΗθGB0)(ŠΧΘΙ½GΛa] ίΟ‹Υξ6τzcΔΠΕd}>™ Ά­[·Jvvvψώ΅w£~EΦϋσζ‡πκο˚©$9§½©Κ귎XΜmŠ₯Ο#m)™ξ±ΧI’ιΩ&#―Ή¬Ÿύ±Μ/ύVLOΝ•§ε8WΪdYΑZi9ΰιϊ‹ίΖB‚uz\`—‚iά>EΐhЌΠ諦‘ΆH?dSΫτΪΦ₯Λ΅iΣFΉζ۽ӝwήig8q’μΉηž’E yσΝ7mwR­[·ΆΣΎύφ[yΰlF τ‰,MΫa € ΠΨ\Ÿ4Ά8ΫCί Ψ!SzHcρΐΤ^bK‰{+ƒΗΏq‡  €Ύ(//·O0ϋΰΨΫ&{‡^VVqΎcηŠΑ{Μ&Θ$%Ιd‰IJ5Wζ± ¦©0Jĝ”’€ωŸϋ ŸuΫΙ¦ft¬ώβμ~ΝφΝFhβ­΅°°Pž}φYyχέwε˜cŽ‘ΣN;M222^±{’ΔdΝMJ6}š…­<ώΑ4ΆΔ”-)hΚβΚh*Λ$S?&M΅Ž ŸŸγχG¬έN©›hΦa†&)°KΑ4βQ±ββb۝Reƒ\Ν†‘2:ŸVNšΊI‡Θi5/]σX]N#΅»§nΈAN8αyύχeΪ΄i²tιRΩ΄i“4oή\φΪk/›gπΰΑ2tθPΩ{ο½mV›Θ}¨y ŒE@Yΐ?yΩ € € P§@ii©|όρΗ2}ϊt9ςΘ#mΖq— ½Ξ™ˆ@T± gˆjγ»?ΣΟ‘@»Ώϊ!bΫφ~K}—―οόα›­υ]°ϊό•/ΪΕHlύΫͺΊ»ΕπέαͺΣΎ,±ήΏ†/1kD – .”»ξΊKςςςδž{ξ‘Γ;LzφμΛMzkέΆJ0ΨΊΫKυCMeΡrVΦ‘ρGΤςΥTΖψ—Œ4Ž@½ƒi\ΚΊuλdκΤ©φ΅rεJΡ5άRΫ ς@³fΝ›¦OŸ>6^`7"cݏ&ν_R+½ύφΫOΆlΩbStiš. ΈΡT]9996{Mjͺιή nμώA@Ό"`.˜L<Ύ-Mdΐ±~ΦΎΆuˆoGπ € € € wΡ’ErέuΧΙόωσeςδΙ2vμXιάΉ3mΌ xΜΩ₯]h°“Ϊ·]ΩaΪ#μSω&Χ’ήˆΥ—ι­ rœΏΪ**o‰ΖώΖ¨†ΚμΈ3ΎάqJνG`W¦ΈνμΚ²,ƒ@b hfM››+ϊY_ ρ¨­.¬m|ΌΚλ΅ςΔΛ‘in·^Α4ξΜκΥ«εΆΫn³A1-[Ά΄])Υ‡oΫΆm•³λߞ»‹r ›7o}΅kίN’ƒ•»‘oτ₯έ?Υ4ΨeυU#\`Φ΄}Ζ!€ €€G΄εΒ\l[ΏJV}ρŽIι.‰β|al6―%˜ΐ™΅³Ώ•μΦνl9‹7ηˏώ'ΕΝς€ά/ΗRΖ\ΐYC‘‡/©ΉΝ·―煝Gώ( € €xW ??_΄‹§Γ?\τσ† l0wKLΙh ΣMGY‘ή Žνoq š@š­›€’ΌΔ΄I˜ šδ •ΩqI©ζ³i#pΕfοM2$`ΆΛ€μŠ€ήΦλ χβ^ρ(² MOΐέ9ŠjΟ]ΕςΪk―Ω@š8ΐ>pςΙ'KǎE3ΏΈyjZ‘f’ΡŒ1υ§C]σΦ΄ΌŽsΑ4ϊDΒ¨Q£δςΛ/·YiΊuλ&Ψγ—%'r‘Ÿέ|Ό#€ €@ΣΠ€mj*-Ψ,λ>Z’4D3μΕ9˜FK`Š Α4[o“”΄43ΖtckΗ6|σΎ2ƒ&˜FϋŽ΅£γψ)€ ¦ I›C$UŒ € € C’½‰χC1άCV@tξIαΪεςγ w›ΐΣv`›3bΧ¦0νm-.“-K’ Ή”‘Χ\Φ;MζN\.)¦ #v[6ν7d)+X+]NΈRšwοkš#τ!£@tXΜ…DhύΙ€ΤG κ`w¦έ;½ϊκ«6pEƒbžώyιΧ―Ÿ€™›=AӍS4?h233m£™·Ά)-+•Ÿ~ϊIώπ‡?Ψξ₯~sφΩ2Θτ™«έ=νΉηžΆ{'·,•£“ΰ@"’L orv;IJΝ ‰iσOδ–kώμ~Ξi#U0m½i ΪlfΤ¬z f6—`– \ΆO{ΥΌ|γΥ†2}κL{hΐj±sζΝ+wήy§}όρ2tθP۟n=€U«VαrTS]Φ'ίm 6₯OΑ“ŽΝ'b"€ € € € €4Ό€>q£‘,ϊŠΝπσšu[?oΓf£Ρm»qξύηYζ“έ†>ΜΓ€ €4@½ƒi²²²€Y³ζ’‘‘!‘έ5Ή —Ζ,~NNŽμ³Ο>φuςΙ'Λω&°fΦ¬Y6¨ζ7ήχjΫΆ­{μ±6[Ν€€{χξ’jϊυd𗀦sLJN7εo6AQqoΊΊ0γ*,–ξ―£Ii@@@@@@@π¦@½ƒi:wξ,ύϋχ—Χ^{M.\(¨βϊͺmμ€šΘνfggΛΑ,p€wάq2oή<™6mš|ψα‡ςή{οΙψργνλΐ”£Ž:ΚvO₯ο]foώψ‘T&¬=)(Ωmφ–²’¦ΤΙνp€•²=˜Ζϋ@@@@@@@@@ΐλQΣΈ “Φ­[Λ1Ηcƒiή~ϋmιΧ―_8CΫYδβΎWwλͺ>ΎΎί#ΧγΆ E»€Χ‘G)£G–όQ¦L™bƒjΦ―_/ϋίm°NOI1Α >0™g*B’šέFZχa“Ρ€ρΑq£ˆ € € € € €ΐgο<μ*κύΛΆτNHH(IHB"½H/‘<°R„'(*ο―OAA°°Š’ΒΓ ¨ €’€’ΐι"B¨!½mΉωΜξοrφξέέ{·ž{οw`sΞ=gfΞΜgζόΞΜo~3#" " " " "PZ 6¦![¬TUUΩGaσζΝ³3Ο<ΣƎkψΐβ‘- ΈŸ4rιk€ΡŸ_SSc€i£6²‘#GΖ΄ 80&iύϊυΩuϊ:z^7PΎlχ£θ­MX»‘>(iEΣ`€R__oσηΟ·iΣ¦Ω;μ`'Ÿ|²]}υΥΆί~ϋΩ&›lbƒ Κ³δ’!ϋΨΦ[omcƌiΧθΗΓθ(" " " " " " ]" …4»„MD@D@D@D@D@D@D@D@D  6¦ρΔbΌΒΦN―¬[»Φ›šβ /Ϋm·UWW·k BΈΪΪZ{ΰ¬Ξ£λΦ‘Ur^}υUϋΧΏώe=τέtΣM6wξάlœΫn»­pΐvπΑGš‰'j5š,ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€τ<Ζ3ΝΫΣφΨ‘ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@(ؘΖW€a›€ο|η;φo|£]Γ™ŽžΙ4ΩδΙ“£³#Ήχ|…Œh.½τR»ηž{βVNkƒa»c=ΦφίΫuΧ]γvTC‡υ[qk)Ρ•η{XE@D@D@D@D@D@D K a9OΓ?3yŒl²at"" " " " " " " " "JΣ`ΈβΞ OX}fζΜ™~Ή[G³˜HܘfΙ’%φ­o}+r³aΓ†Έ…[9ν»οΎ6}ϊt›0aB6Ϊd>όbWžνauVZΊΟUα↦ŒUSš†p­Ί*aeΣ*€~ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€€•@‡Ζ4i68ρ΄}ϊӟΆ½φΪΛvί}wΫ|σΝ[­–γF4ψMž§΅0”.H  γ«ψuΖj5,λ4tτ˜0r m1ΌΞžZΎήέd˜\ϋ£’έι++₯HD@D@D@D@D@D@D@D@Ϊ#Π1M}}½­_Ώ>†«©©1ΆwΒa”Β–JMMMρwWώiΟ=χœ}μc3ς&'" " " i!€± «T‡QΧuNX,°&¬l Cš΄”’!" m ϋGk +L>Θ>½ΫD{{M½M5Θκͺ«’όnBWD@D@D@D@D@ʝύ„8+*[¦L•{–•Ώ 'ΰγ΄iΑ€^•χtωX1iγϊ;Η–Ν—ϊυίdϋ5!zΈˆ@$ΠΖ’#™ͺͺ*{ϊι§νŠ+°!C†ΨŒ3μΨcΧ1°ΉόςΛν7ή°ΪΪΪ(| e‰X·nxβ‰6yςδ6‚«x\ˆŒ7ΞψΓΉ`ζόN ΕΖΖF[Όx±Ν;ΧnΌρFϋΕ/~aϊΠ‡μψγ/δ‘ς#" " " }G€^°¦Ω4 ΐNQk/­¬·ύ7bΓV‡φMί%COθΠ%F4Θρ`[ώ$Ό»FR‘D@D@D@D@D ΄ Π7ˆcUAΟƒ§yά*K;[J½tHΐΗq;τΤ‡7yq΅5θV›ί>Ž΅5Lg “΅5sδΠίPL^V’‘ύ]z~’@cΏΙ .?όαγΟΟ~φ³φ‘|$Σ444ΨυΧ_oϋίέkΑΗΊΊ:Ϋ°aƒqΔ6yςδ–†C‹+8–w<ΊΠΛ5 qA½tιΈΝwάaϋΫߌγ΄iΣl̘1qeNŒ:ώ%@§• &ŽlŸέc’­ZΧh›„-CͺΒ u$ϊ·lτt(”@CXY,Ί Σ»ήγ-τiς'" " =C mΊRγzίdΓ4Κψ3mι₯/§/²€t.N ˆͺͺΪ*6nˆΥΏέ%Χ°qaΧ†Ϊ°kƒ<^9Κ7ϋMφΚ«Βξ&αώσovη˜έRΏΎ2μV΅UΝzkVW—΄ΨtϋρV74Θ­0RΌΡ΄Ρa€8€X:ž΄›W$τ–4#V―­·9Ώ|ΨήZΉ!nyŒD»`ΨSUmλ—>oƒΦCŸ¦0^]c—ώιI«»s­eš0¦ιί6μ‚}-YΩ`ηfgΫfΖΖΡ°†Ι•εκΘυfp0:άaβ0{όΝ΅ΆωπZ› Y,ˆ9Jε›ϋr-ΥςΜWcο€l·έv6ΑόPQΨ ΑƒβJ. žργΗGš]νπŽ΄zzYIηΉηž³{ξΉ'Bsλ­·+αLž<Ω0™0a‚pΒ ΆΗ{ΨφΫoo'NŒ+νĈτˆ€ˆ€ˆ€€@=N:κ9€°t”$HϊCΎUo:R€Tˆ€ˆ€ˆ€”"†Ζz[²β•£: ΝϊΧώΜ *ΰš0Ψ΅dωΛVW30€-tΤ5ΆxΩ"Ή|˜5„™δύίUjNAƚlά°ρ6€nh »ώOY–ž]Yϊ_Z€˜w€S7€Φ0¨‰’5Θ±–α­'ZIξ`χ€ΥaaJ‹€ij`H;αͺƒΕFΧRςΙƘfp`––τ$Pυϊι.›΄ΝFŠXγ†ΥΕUjRR,½žw= ύΪ5¦a5ώr0›nΊiξε>ω4’ρ.^Ό8vΓΆS_|qΌ!ΠQψ)ΗHŒ>Ι~WΗ΅ΒβZ¬―%ΩΠ‚ Sz\:€#εΝ-’']αR&CK¨Ÿ]LΫbυ{JϊߏΊ`I΄EΨή Χ˜ŽεŒbZτ@ 1cqΓνΛtZΦ―[R˜ιΧέ_ξΡγΑgύϊυvηwΪV[me[l±EPX{xžγΟZΆlY\…₯ρ­·έj·ά|K|μθΡ£νν·ίŽi{τΡGνӟώ΄}ο{ί³6Ϊ¨U²’qΆΊ‘" " kX}«¦¦])N" " " "ΠoΌυζ›oΪΉηžkΧ\sΝš5ΛώψΗ?Ζ£ίο·κΑ" " " %G€‘›ΊͺAΦXΥΆ¨ŽzΫ4drΝ€ΥΩ€0Γ½Άͺ. Ί NΝ[?eoφΛΙ€°AU˜ν^†Ώ*o¬_λ‘ύK Όƒ¬žPŽΌΈ–Cσύ›Ÿ@’ωΡθjy@> !Z“›šl›M‹ϋ΅$HK ²ίΠ²―ScθΧ*¨‡·O έ‘b7\ρ ]·v]uΥUvΔGؘ1c:4ˆ!Œoε΄hΡ’¨δύς—Ώl̞,ԘΖΣΑΐφ’%KμΑ΄»ξΊΛ.ΌπΒV:V 9πΐνC±ΉsηΪWΏϊU1bDΦΖ ˆΟγτΌι(" I?ό°±Ϊ†tP4Πs˜τ§sθOL$ΰούομ+a\#'" " " ]%ΐPRό/θtω―ί]HBΎtΔk™¦Όχϊ>Ν€†™δ)ΰΥχ™Χ+„›,00XьTm‹Χ6›¬\ŒΫ¨r" " "Π=ZΉ«{όΊχ΄kLΣΡ#σ›ίŠΪ“O>Ω† ΦΖ †AgF+¬Fsί}χEγ—;ξΈ££hσή[±b…½πΒ qις?ύιOq€ΜhΜ4Ϊ±Ηkϋ쳏νΉηž6mΪ4:t¨½υΦ[1.ΡφΞ­Ξυδ}ˆ.Š€T<7–aε«‹.Ί(aΔχΐΨN;νTρ|@D@D@D ΌΟΕδ9(;QΟάv¨>^αŸTL_&!mΣXve‘ U<šκΆΟΤΡρ΅ZWm;m6²ωTυ―ψΊ!" " "PŠ6¦aΐ™ν”Ύώυ―G#•Ο|ζ3qˆφ#ΐ.\hW_}u\%fƌΆzυjΫyηmψπα‘gg+ΔΧW\v’pπΑΫ~ϋνg³gώ4γƍk΅ΪΜ† ’ήu." XΎ|Ή­\Ή2ΚVΤZΊtiΑaεQD@D@D@ϊš}&œϋϊωzžˆ€ˆ€ˆ€ˆ€ˆ@e λ1iΤ ;r‡ q<&,Jϊ!•‘wεRD@D@D 2 mLSWWgl©Δ*1_ωΚWlπΰΑvβ‰'Ζ£ΤΧΧΗΥh~τ£ٍ7ήhΣ§O·gŸ}ΦΞ8γ ;ώψγmκΤ©‘v!Ζ4‹/Ž~χΨcΫwί}γ*4»μ²‹?>[b<Χ•ΗΔι[Ke=θDD@Š$€Q+ΣΘ‰€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€4ΟT΅μΕ!CՁr'P°1 F*¬`Lσίύίq•™ψΗφωΟήXRƒΆXb5škΉ&ΪlΉε–‘ί¬Y³μ‚ .ˆΖ0C† )Š)ΫDΉcΕβ9rd|ζ˜1clΰΐΡ ڍiπ›<χ°:Š€ˆ@±$KŠ%&₯J YΧ;3t-Υ<*έ" " " " " " " " " " " " "P(‚iˆΠ j&Nœh_ϋΪΧ¬ΆΆΦΞ=χ\;ε”SlΠ AΆΥV[ΩO~ςγ°j͟mΚ”)Ά`Α;ηœsμ£ύ¨Mš4)¦‰»BκXa†νœζΝkϋίνώϋοq`@³ηž{Ϊ‘‡j»ξΊ«Νœ93n=oꁒϊ].*Ry%P”1 ytƒšqγΖΩi§³A͜9sμ₯—^²M7έ4^ΓΘεWΏϊU4vΑΠΖg½;`χ|ΐΨΦι駟Ž[Gύγˆ†5wάq‡ρ‡ϋδ'?kxζ[lΡΚX'lύcΔθ¨λΦ­³ωσηGΓXΎ£ΥΥΥ’seSD@D@D@D@D@D@D@D@D@D@D@D@ΪθΠ˜Ζ `Ϊ³ΈΝΖol§Ÿ~ΊΥΤΤΨYge»Ύϋέφζ[oΩE]dGy€qΗ6PnDS¬Q z¬jΓίμΩ³ν˜cŽ±Η{ΜnΏύvϋΛ_ώb‹-²Λ/Ώ<ώαηΈγŽ³5kΦdŸ[SݜE7↧%z?" " "PαnΎωfϋ—ΎdS§N+Ξν°ΓΡVίΛ ―ΚΎˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€T(i:Dσ{¬Pσ•―|Ε† b_|±-Y²$ž=:‹”νšΊγά¨ÚιΣ§ΗΏ8ΐN<ρD»χή{Υj»ξ:[ΌxqάVŠYυψ{ζ™gβκ5œO˜0!ύx:Š5κρp:Š€ˆ€ˆ@9ποΰκΥ«γ·”ο$«ΐ±΅βφΫo/ΓΣr(dεAD@D@D@D@D@D@D@D@D@D@D@D KΪΣψΰΪ₯KγΆM¬:γΖ,ωžΐ½ΊΪ:Ϋ}χένρΗ·xΐ>ρ‰OΨϊυλm―½φŠ«΍oX₯†™οΓ‡/xΦ»‡%O Ζ:όνΈγŽvτΡGΫ©§žW«ΉφΪkmήΌy6xπ`«««3ŒnήϋήχΖΏm·έ6.'ρΚ‰€ˆ€ˆ@%hhhˆΩζ»Θ·™ί­ P+‘‹ς," " " " " " " " " " " "PΉΪ5¦yꩧlο½χΆύφΫϘ΅ή‘Ϋ°aƒΫx\°t11b„ν΄ΣNφο|Η6Ωdc57Π©­­΅ϋξ»/έl³Ν6Ωλŝ{ύ`ά¨†ίγǏ<χΨc΅ϋοΏίξΎϋn»μ²Λbπ|ΠΨΒ‚΄uΤQΡπgί}χ†8Ήρλ·ˆ€ˆ€Tδw#šψ{@%P^E@D@D@D@D@D@D@D@D@D@Κ›Jo)ΎΛ»Œ•;θymŒiόΘΰξΌσNΏTτ‘-Ÿς9;ί½B―ωΰώ1¬α­¦f̘;μ0ϋάη>gόη?νΖoŒ[X,_Ύ<―Ήζγώο{ΓΐGND 3ΑR.ό_ Ξ υΪζ΅γDΉΚ$Ύ’ٌ·d½θDD@D@D@D@D@D@D@D@D@D@J‡@G3I›2Œ…Ό£/L(₯" }M 1©°ͺ̜9sβJ3lΝ„Œιάεzzg`ΆY> ˆΫ?m΄ΡF1*Vηρvμƒxψσ@ΞGŽWžΩ~ϋννπΓ·ϋίqU k-Z€ν+:Fͺ»"C€FΞ₯2ϋιΩ« +XΉΛdšl@UUό9`@σΡοι(" " " " " " " "Π\kΣΟ3D@D@D@Κ‘ΐ€κZΛ4­YkŠͺ΄ΨB9–³ς$=M ]cš™3gΪWΏϊ՞~^«ψzʘΖ#MΖ‡a U…pΆzο{ίk³gΟΆγŽ;.Φ<χάsLGN4¬]mΦ£΅ͺψ^uβ½do³άϊo‡UhLR];ΘκW―΄Ζ5+¬‘Ύ‘μ Šj‡ ·ͺ5 KΆ+α" " " " " " "P–˜Iޜ1&lΚ‰€ˆ€ˆ€ˆ@1/:Ξ†MάΗΦΎ9Ο†ŒΫΖj‡Œ-λρbψΘ―ˆ@ϋΪΣ΄ο΅ν_ Əm}δΏ‚ΡKπ%Ώ―ξ]Οhιeyϊl[m΅Uό[·nV§ιb…-Λή½ϊπφϊύΧZΝπqfe™sΦΥͺ`―,[k kW…F”ΩΠ1cν•»―³§_ΎΣššWή*;MΜP0’ͺ_mSό’ Ÿ΄e\ε«·etYV"eJD@D@D@D@D@D@D g  ·`+† –©ͺ` Νϊ™ž}ˆb(_΄#ΒδιšΑ6zκlΉΩna"υ°+AΨ₯€10·Ψ-_Κ™ˆ@7t٘lυc7Ρ«A=}€9VVΧ4hP―>S‘‹@9hjh°Ζ΅Λm@νPΛ4†ZΚι ύΧ³Μ_h@…‘Mυλ¬a]X™‡ΥjΚ±QεΖ4V„Υ ₯*Λͺ­L‰€ˆ€t‹@Τ«Π0H‘#MΜΚφIž4Ί43™Β©£ˆ€ˆ€ˆ€I`ΰπΊ0θUe«­³“‡Ϊ aZU·H„ς." " "t;UU5V5hD‹ˆ€ˆ@ηΊlLΓ σϊ0θΌxρb[Ύ|Ή5…ζh¬¬Ήmσd7hΑߌ3lΤ¨QΡΏ_o .π,©q{¬’,{ρέ ϋIFKέ” &υό–ρ±`gΧ’Ηkρa€ͺͺεZ8ς°² ―Ÿƒ²Μ[ΤE"" "PΑ0ZIΫG²9MVΪ ίπG_¬¦ΊωwUXmOND@D@D@Jž@h45flψΖCl›£Ά΄·_Zn㦏±!£‡-ΊΡΩ”|•Ύ&)Σ‰Σ}ΝQρ !Π%c΅=φ˜]sΝ5vχέwΫ½χή[G}΄_ŒiŠJ€<‹€$„U[h``,׎Α\ΒsIžF{– ne,δZξFΩCIf²½D(ίrm/ΛΊ." " …XΎr½­\½>Μ^ͺfPuu΅-y½ykJšguu5φϊ5φκ›kœ¬$θν˜BsΪσώ0ϊi ]£F ²aCκzώŠQD@D@D " ν° 37² 3Ζ†½žB›L+λVDΉ+“" " "Π;ϊ__;ωR¬" ½A (cŒhX‘⩧ž²όγ6wξ\Ϋl³Ν NΧΈqγμ7ήJθ–Aι‚CΚ£ˆ€ˆ€ˆ€ˆ€ˆ€τοϋάσοEφΣ럡ΝGΧY}nΌ+UUeV, †3ΝFΟ#† Άίήό¬έπpM`jlΥΟϊ‘Ϊ0Ψυ² φγ·΅=wΪ,φϋreΎ«-z’ˆ€ˆ€ˆ@oΘΠ„ΆO l†{;»Š_D@D@D@D@D Š2¦AρΩΠΠ`Χ^{m4€a»&ώN>ωd;v¬ΥΦvΌ_-αλλλmόψρ1λR€¦ ( " " }B ΩŽΤG@ϋδ‘>„τΔ8ΙIV\ˆ*μ<άa4}zSν†>Ε­‡‰€8 #«Γ*nΘΜw6Uς›ύpŒ2Ν‚;ΜΛΞ&€3~W…?ώ‹ ΞήνŸ“`KcΥύσh=UD@D@D \ τ³±pΉbUΎD@D@D@D@D@Ϊ(ؘΖgf.Y²ΔξΉηžΣ>ϋμc_ϋΪΧlΛ-·lsW4(V$y(  Β¦iϊ\szΜΦV·l]’‰[qTWWγΨζ‘O}§Λ’κ)" έ$€ΌK•ΉrYžΝbsbγj’Ωkύ|&ύœ=^D@D@D@D@D@D@D@D@D@ !P°1GΆhΡ"[Έp‘ <؎;ξΈ.x|:Š€ˆ€ˆ@%xυΝUΆlΕz«λΌ³@?f=¬^PUm+W.·•«7XuΨ.dΨΠ:{ύ­Υφτό7C:«[Άeμίi Β6…mU†©³MΗοG^z΄ˆ@₯@d»Ψφc³hNI›τp!όΕ•iR²A:Ύyύ_bJˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@i(ژ†mžζϟoϋν·ŸΝœ93ζΦW­)­¬+΅" " "ΠΨ !·έ’]tσ›<ΊΞκΩγ= K•†5Άκ•VSSe£F³»ώύŠ=τΖΏΒ@l*63±š`|τΪΚz;|·Mμδξ”’‘α4ž " " " " " " " " " " " "Π[Š6¦3fŒm΅ΥVΆnέ:Ϋ°aCo₯KρŠ€ˆ@Θ΅uh1ˆθίDιι%I eaŒB6Re#UYCJŒi ¨²Ζϊ*[Σ²Ζ±€sΔΐ`Hξ₯ΑUW °Ζ†*««i™†D) " " " " " " " " " " " " eM hcšΝ7ίάvΪi'»ςΚ+mήΌyΖop…Ί-ƒv…ϊ—?θKy‡μσ^μΛR_Εσ@IDATTιY%O |&ΓnEΩΏ4δ'l8ΣγiρjήΔ…Β?λΌWŽZ˜ΡΜθ•t(RΚ!Pπ΄s7€1b„~ψα‘Πώπ[Έp‘q―˜ΏΚΑ«œŠ€”"νÎ7­·“I‰aA)ςTšί!.ƒL³½Lnέζwξ΅w²Π/gιβΦ/τP>$PΤΚ4¬>ƒΡΜΎϋξkgu–}ύλ_I=ꨣβΦOƒΞΥ΄—β=z΄ΥΥΥ΅η₯€'Wδqƒ£’Κ€+" °"¦ΓVΫΘA5φΔΫλmT85Έ¦΅Qx•—α’ίeU¬ΚŒˆ@ό€BVE@Κ–€dxΩ­2&"Pζ$ΏΛΌ€•=²& ^ΦΕ«Μ‰€τ"’Œil\·nέrΛ-q['ŒjώϊΧΏΪε—_nΫn»­M˜0Α† bUUUmΆ}"luu΅½φΪkvΙ%—Ψ¬Y³ΒΦMV5 ΰΕqzCΧ’ζγ£Ψ±S(H#VΏVWc‡n½‘m6²Ξ65Θ6=ΨΓήκP˜L+'" "n.ΏΧ}ω嗍έEƏŸξD+u" "2ΣxΊY]ζCϊνΈγŽqΰaŒ+DB#{ŊζF'…„ρη¦ιθ ϊϊz»βŠ+쀓N²‡zΘvΩekj «ν¨3‘¦βRZD ha!šhηΧ"ފ―ι&ΰ2γΞoϋΫvύυΧΫ#—ίAžsΞ9vξΉηΪ /Ό`S§N­lωœ=ι>lΕ " =LΐeψσΟ?oGy€νΊλφ³ŸύΜjjŠVKυpʝˆ€ˆ@G\~/~e±}ϊ}:Κν«Ί*Nοχ: ―{" " ύOΰΑ΄½φΪ+κQΎτ₯/υ‚”(!k-|pqΠ Avꩧ–Œ~­˜όΘ9rd •πΕ<«7όΊ± ΗoΌ1*ςyω’H/δ4’Ί*μΩ΅|ωrϋωΟn_|±ν½χή•-Γ5φšήJ«”‰€DQ~ŽΑϊš5kμΚ+―Œ  3fTΆό†Nμ‚A> Ί₯o¦>‰^tp] ϊV€ωιOW:ήgŸ}•P₯FD@D €όfUφΉπμ―ύ«}τ£mεO?D@D@GNϋ›ΏyσζΩψØH-Ύ²RŠD@O `cΟ Β—νš*Υρ±a6μuΧ]ggœqFάκjΡ’E•ŠCω’!;!΅K–,‰³`ιD̜93‡–L&z#‘>ξΎο6€ešύBoς¨]ω›+ν’K.‰νoΆ‘m^Ήƒ™&«4ΒFMω[ύϊ<4rs8lBθ§i€£rλ…r.ι#πζ[oΪ=wίbώχΏΫvΫmg?ώΈVK_Q)E" "Њΐ²eΛμΎϋξ‹+ϊ²:ϋΦ[omK—.•όnEI?D@D }ƒeEΘΫo»έζœ=ǞyζΫeη]lΑ‚’αι+.₯HD tɘΖg‡’?Ά{B8ηs\ΗZ½ΎΎήΦ­[g―Ύϊͺ=τΠCΆώϋΫ”)Sb8·Μ>mΧώρΨ!‡“uΨa‡Ω>πΈΌ%J)iγͺτˆ€ˆ@o`ΐυŠ+°Ο}ξsρQμ ;mΪ΄ΈMΘ°aΓzϋρ)ŸΨ°ŠAυ@2n[Ώβ%«<Ώ™ΎiUαFFΝ3‰Sš%KD `πxν΅ΧΪ'>ρ‰˜Ϋ/|α ΆσΞ;ΫωηŸ_Ήο<‹ΘλαΆ³aΟ v΅!@³l—ό|ˆ€τtAθHήzλ-;{ΞΩφ£ύ(>lΜ9јύ+_ωŠt(½‡_1‹€ˆ@— Έό^±b…]pΑqUv";σΜ3mΤ¨QvΚ)§H~w™Š€ˆ@οpΞN§Ÿ~zΤ₯πΔK/½4όλαI†χn(v2%Π%cX o½υV{ψα‡γλ(Ί}vrΥŒmܘ†₯Ω1¦!Μ”4¦aYbάΉηžk9ζ#V”Χ7άpCΌΦΡ?|Δψ/ΉsΧpΎ4¦Ž~[ξu§Yuυ€C¦{τ^@Ў’Υ½”h%‘|Γ cΞ3ύ°”jUP‚6Q‡C8Ζ“”qRrΊCc‰–ςmj©q‘žeZ–cο,f”δΘ1—aωολϋ€‹4bLsηwΪξ»ο j>τ‘Ε•Ζ^xα?a|‡Ιjb†3šθ/ζ5,ζβyφ£Λσ|‘ρ]l^) ως0ωόφυ5Ά6ώ]6pΔΔ8Λ*MMΎZO"γ}°–η!všλh¨§ΤѐήζkέLu£Ί:δ΅%>"myV7c ΰXdH¦›ί•ξ`EΉΚƒςŽο ΗδKXA„»’ΥH‹κάς’P.2mΛ…ϋΥαp<‹°aGν<ΆΗÐ.κ} Vc[ΎΣN;͎<ςH{ν΅Χμ™gŸ±‰›LμπΉΔ‘ΜžωΜ·ίη˜+Η©αΥ! ΄Ώb[,O|& oFC—’w%Ά0Zκ’―–ƒΜε»Σ=GΤέ¦ wψ,x|~μ^μ•lΩz_J+Λ±λαMjώ^S[ώΊce…Œοo‚YGόZ΅ΑS\υW\ iŽ:κ(;α„βΔͺ›nΊ)lRo”[ωςΞ5 mp—ΥρwΘoxδΊl{‚pΘFϊΤ)f•›ώώώ «ψν |qθή‘·έHρPŽαHq Η›?]*œΒ¨ΆΘV€m¨ΣUϊ™=Γn@ψΔr‘νΛ‡2κ™Έ Λ[)ϋB?Λ;Cφw¦ύ689υrσcΪrΟʐΧ]w½ύο·O~ς“qrι½χή“I[Ί=λNNτ<ζ“ύ”=w¬Νmoͺ_ΉΦΐ˜―i˜πΛ„χΟω΄ΗΆΝυ6κ4b<-w›#nγ΅τ/΄Όc!#Q7Ž± (nEΞΨVE§ΔΗ§…ΡάKfΆžeΏεωεRgYjΦiR_#ΆβλjgHΝ}―gΤ‘π^†tΕΊ"‰¨+I]Bj’Hi€ ŒA;“’N:ι$;ζ˜clΧ]w΅‹.Ί(ϊΜΧnφ(Ÿοπ:€?Ώί~Ό*Θ5βαύ€Ɵ cGšu(Νμšλ^`X6u[‡όT/Gδh”₯*˜ŽŠ£Υ½ζwƒΊάόŽψ{ΠΚS~„Ψ‚ή±E7㎯LˆI/M!8›uPΝ¬’z‚|εΓ΅ξΘπ’ŒixΒφ₯—^²SO=ΥΏώz7nœa$ƒ8p`4œρDsmόψρ1Lm-3%›]ςά―•ΒρΰƒŽΛOŸ>=ζuαΒ…Y’Ž3ώΛ763œ3ιH‘”ο « Τ†JΐέπΑΟη©,―‘ΣΆ ³rΛ*SμΐR[;0ζv@ο7°ΞκjͺB:‚`o¬œZΦ¬ϋό™QρdGS• X_Ձb$_ϊΨjƒΏ(λb|ω|υΟ5ο Τ…:Μj4|³ΆΨb‹˜ί$Κη’‡αd>qCœΈBεwmέΐ(―«jκB#)MοQHKuΥΤ Ήinƌ₯δŸκ€Ώ*Θšں0“ΉJ¬Ε?υω6 †i`Ϊ:r "«LΨφ²Ή-Sε[†Ό3/t$jΦ†r―Žς% ™wDχ[ΐ&Œ6Φ΅ΘτκNΚ…Ά(νΟl{5%$]~ΣΡω―ϊ―hH3yΚd#?υ<€Σ”vτ]rΩν²Ό£AΊ(Ώƒμς»§”&Ύ(IΉΥƒ²¨4δχΫ²5ƒ!‘ήCŽΊ;0Ȉ8*KΆ%Ώ &; ΌGU_MψΖΆ¨zθγ: ΘδwuδwΑβω] ς»Yw™n€m§‘]./‹}^_ψ3fŒ=πΐ±ύν[d3Ωͺ3Χ‘όvyRˆό mΘZڐθLB%Ν dxΊd7εΣ¬‹HΘοΦE BΗΎ ΄υ2jηuφNψ}V€klͺŽς»Ί¦YVτ€  ]_SϊχU‘5‰bΏ‚΄ŒΞΈ«Η* υΉ!”M‹ž·³6ΈΛ0/?ΧΉwυω=nĈq{ΥM7έΤ&Mš£wJGΟκX~7λP<οωΥΡ7 ϊ dv‹]RGλα{}'QmT…IL5‘QΪc8― ΕfjP`V : ‡ΒwƒοXέ₯sEXηmΥΐή ΰό~9³mΛεRGy§}ZΚ .θΞςΜΥι(h7ουέ{‰²±)|cλ‚ό© νl\Ύ±—σ)lΎeε²ϋŸό§Ν˜1#*Ζ7¦»2άu(žŽt(±=δκ¨*tI•E7kDΟοΓΊUD’›υΰUANυΌ…d οςΐP&ΝmπTL΄ϊΞ+:”†ΐm`°sθ¨­ΣΥρ^EJxŽΪί…SΜͺƒA?‚λLβνΠbΫ]EΣ9†2ώσŸ£!͎;ξh/Ώόrœ!Š0}β‰'’2‡™£$œY£7ί|sTœP<πΐΈ5’d›ΨΒφŽΟY³f΅ŠΈΠ—fΥͺUQιŸ42βΖ ϋβ0Μyϊι§γμΫBΈΎ64^^~u…­]ΊΐZτ• ώZ-(γΣ§rjUEΊύ£:τκWΧΫΛ/­±'ŸlŒ–έŽ΄˜B=μK^xΡ–ΎΉΑͺΧ¬ ΎbE1•–ίζx!ŸAQΥ“ƒΓι₯ŒiΦΪΪyOΩπ·ΧYcC}ΆρέišΓKΨήI,ή1¨IΚΉNΓφ‘>’»μ²K|’+© ‘΅k׍€¬Œΰώ£όρ=χάsQQΖ΄O>ωd‡ο'F Ό` Λ—Ψͺ†Ϊ M—΄&OΘVςXC«=EvΦ6Ψ’E+μ‰y™>žpςŒzε+ lΡkƒ1 aΣU.€(΄ˆ‚BΫλ ΧSαΐ«λδHΐIε³θΩΕφΦςεAa³>”LΊYHeΓ€FgτRD$Ua {uΓJ{ι9²|„Υ7ζ—ιδΉM{‡όζZΪκi€Ο#}8XH:ί|σΝΈΝl/qΠoyρΕc\΄ΕQ‘XJϊγf3£j[πΒKΦΈb‘­lJŸό&Μ(j ΖΞΘKڎir€§iU½=χl¬[fκίωžv9ΤΣ ΏΧΌρ²-|m5šτ.GΥ›ί‘ί;€λΫdMΦΪΠΊΧlυΗ6oO±@ρ³θΩ—ν­·—Ϋκκ Az§Q~7·yR'ΏΓΐυκ +mώ†6nύΣ‘½ή—<=^drkΩ²eq;m—=U†=ΛScwΫm·%ιδΊίλθ9oΏύΆ½ςΚ+­Όž6ύσΟ?―?όσ6rδΘΌςΫ"_[όœ­{™­ZΓ€YΊΪzδ©!τ X 6ς»1Θο/4ΩΣcΧ]νžξcΘsu0pzϋ…gmρ[υAΗ°"•:†¦ΖΠ ΕyOδΫλd·Θοϊυ6(σ†­}ό±ΨGθ‰> ²›AΧλ–Ϋλ/ΌiλΧΗΥΈΣ¦Μg₯€Έ"OΪΚ%,Ζ4kΧΨSuOΩڍꭾ‘mΫ’ςwYζϊ`_ε=-υΜΣ1tθΠ.ΙoΆ‡ZόςβV δ™qΟσ‚ Ϊθΐ#—`ΨπΜό7­aΩ[΅‘¦Oυ&¬’€ž&φ€¬λμ]ηλλƒ.­zˆ=φΨΰΠ/c•©ΞBεάͺB›xΥ« mαλk‚ώ}m±‘δΔYΰOΚ+Σ²’qΦ0»ΐ°]φdTγΪz[ύΤ³6zCθ—ΥηΗ ‰₯gΩ₯+lMu}ŸΆU[˜!oσ4ρ IzQ~ͺΒsVmXtΟΫ°•£ΪΥ t)%σ_|Ι^ +VΥΠχjΙGgαΊ{?Κό χi³η3jιnόmΒΗolƒ yτθ£VΖ7©λξ8― † ζ/ˆ—Π‡§ΝΉ g2)ξH{!}τC,œΰρxx~3Œ‘<»w¬[ΏΞκ7΄Υ5 9ΊrΥz[ώΖ|[³¦>θ™›WόρΈptJ_ΙόbςLŸ aMƒ=σΤ °~£žΡ‘΄$€χθ•ω νmΖωVΣOWξ—oK‘…ƒAϋΊu6ςΕ—mωΌy…†κΤ_”+aβΗΛΛΪ[σ—[mΥΪπ%GξtAypq”Η}ΠN*4[θK`Θ7υρΑΫFΓ6ύλφυ(θƒi‡βΌ ^θ³xPA.ΐŠώΒ€iζΓώpfΨ°a™χΌη=™;ξΈ#σϊλ―g‚!Hζcϋ₯œΉε–[2Aω‘ "™ϋξ»/sβ‰'Ζλ'Ÿ|rΌND_AO‰'=™9¦ˆό½ο}ο‹y ΫWΕk~žΗ°­UτύuŸΑ›ΙLΫr’Xͺ>υjΨtζ»26xT―>£εr1)⏔όƒLφ?’tΟ=χΔςέk―½2Α &¦εΆŸ}φي©3¦mšΩtβπŠΙoΙΌ£7ΞLœ1Kε’ΒoήV“ge[ωΌ3ŸϊΤ§‚=TCJ$vλdΈμφvφγ?ž;vlfΛ-·Μ„ΑΤVς;ςόcΕΌ;οš5%3€ϋνμ’‘)” mΨ…vdlO–BZ+,3Ά˜™m•|@? “β/Uη.Γ½}}Ν5ΧDφaΩωLX₯¦UZέΟίώφ·²)Ÿ6²!η}›5srfΜ(ΙοΞ8υυύA›NΌ:½bκa_σνΞσfM Ίšœχ¨”Ÿώω­δ`š~δΚοΏόε/‘ύqΗ— Ϋ@Ε€Ίάφγύχί_²εσYSK6νύρŒΫrF¦vζbV„<Ϊlμ™ΙγUϊ©―Sή•^™­o}+;ώ—&ΩνiqΞoΞΟ;οΌψΎώπ‡?t/mŽΏόε/+杖ΜOgcγi326Ω¬bκa1ς°?ύ’?AŸiθιgίuΧ]QΊΉ@ΜΉPΤΚ4!±q…–Vg΅•N8ΑfϞΝεh•Έηž{ΪW\aμ‹=jΤ¨ψ·Ωf›̜ΊπΒ m‡v°όγρw XB`-’œ)<α'NŒϋΜζl€Χϊυλƒ…ζόφ»ίύή^{ύuϋΖ7`Γ‡dΉΦ²εˆ€?σX9υ¬yΆϊKaΆϊWΏψE;υ”S,ZΔzΧγ;vόq'ΩμΩϋeWΞIWmμωJ4εΫΝz§ίΩλ―ΏfŸϊΦΧ£ή©©εΫΪͺRφσ_z2±g”ΎΞQ¬/ΘπΉΞ΅ο~χ»vξyηr’>,˜²ΥίBtT6Τ}ξΣ^cν·ίΎ'‹°Gγς|σΎ2 Ϊ[ιj ό.D^Μ³z"³<―&¬ξ2wξ£ΝuλάσbέΪV=aΥ°ΎpΞΉ«ΟκSf”ehs¬\ΉΒ.ωί‹mσ°ϊGž›XΙJΜrΛ1ΆSB]b“ίώφ7Ά:Œ}χΌ3mπΑQίάQ&7ξό.₯zF¦4/¬ώ Vμ?ψ±czΨξζ£P†=‡vς{«­Άκ³τšξ€ΏŽςτ—<ίyηνκ«―Ž:’δuΞΓ€ Π‘tΦχˆM‰PŽisi¨[Ε0ιJvŸ~3:KŒίoωΆ,+―ώμ’‹μ wοl‡vXΛϋE=κ›o‹'§³cΕ”‹λIώΫo»έnΈ±y|bbΨώ³‘'Vξ t±χC: ω¦"Η\Ξ# -Ο’iXJ8ΜΤ·]vή%Ζπ0:ΠΙ“'σΣΒ*-qK§Ρ£GΗ{(ηΓͺ5q―Υίώφͺx­žxq MhŒΈΔώρΌ±7ψα‡^b©Oorυ―…χ 3πR=pŸ^‚JYgΨrνΤ`L³k:βΘ#;σϋyΈόΛs«$.yϊYΎςΠC-‰4w'‘lΣuι₯—†ΞιL;ϊθ£»•Βφ0·–.΅‹/ΉΔήχΎχΫΤ©S{8vEΧU|'ΎxκγηG‘οDW9φv8φη―άJFŒiφΨcυ9RTΨ?ώ8€η=ο}―± ²\:̟?ίN>ωσqϋΈ£Ž<*‰R*Ϊ˜2eŠρWξŽmcO9ε a;ڝγφιεžίRΚίCQοT/½SΚ mΤΘQ1E‡|ˆm·έv)K]e'Ηu(“Β ϊΪRs£FŒI>䐃U· (<Œyγ™6}zΠ!}€€ςŒνيψΘ kfb΅\ΗƏi<πΐμΆs‡ΠέξΨvΫmΏrwͺ[ι,aΖ'~Ζ'fγ9O€«Œ–Ύ΅4Ϊw”γψ„·a;#^΄1 0XνŒέhl\†ψΓX%χδ“OΖ™BΣψ=f όρvε•WΖύυ0¦©$—J‹Ώ+κ†[XΒ“•~pόφzVbYRrSF€zΕl―[qfwH£ήίΒ ͺίΕr.―σΤuΞ‘§όρ”sΎ ―ΡύλΣΏ{€Βε‘Ύyύ[&ώΞ”λw’άdxΉΛ±r―ύϋΆwοιIωνςBς»{L»Ϊίžϊ±\δ„δwwkHί†χϊ˜”€ \κcίμΩ§ΉόŽύ"ιzn7bσwΖe·ΛεΫ*ލΚΡΝ ε^·Ί‰§MpηΕχ‹sήAήG&Yσ[.?ήq˜Α Nώύ³όΌΰ‚^e}CCXy5Έ4Κ|Ιοόe˜Φ«Ήu«1Εu+­ {#]Ιrαœw?Oτνβγτώ!].§Q›³bεwΡΖ4XΝ²JΨ“;ϋαχDbLΓμΟ'žx"ΣψuŽlύΑ 5Έ—_~9‹Ml T’TR^{£ˆ’ΉΞ™rτσ\?ϊ-Επz–―>ε»VLάς[Ί*Ήμ+9οi©±ωΚ ί΅΄€·άΣ‘οDi•pΉΏ+Ήυ1™ίδyi•Zι§ΦΛΕs’,‹δΉίΧ±o΄W.*“Ύα_μSΚ½\Ό>ζζ3χw±άδΏ{Ό\’±x™pτσδ}χ?•K—An Κ₯LΚ%ΉεΣΣΏ“œ’η=ύœRΟΏ-ν±iοz©ζ·'νΜ<π Ξ:ρΚ’θ•“rη›[·’»•{ή{₯ΒτQ€*›>έΑcς•AΎkDQς·ŠήX–%°X¦ϊΘΕ8V’a―ΐ_|Ρ–,Yβ—γA…΅nΥͺUρ¨D PΙ3y^hxωP]λ 5…)u^ούXκω)·τ«\U’*t•‡R#i" ω¦xg5έt₯J©©X‰A‘ŠeΒŒ'εvς<…IU’D@D ’²2yžŠΔ₯$β…‚hi'ε²Λύέ…˜€ dΫΰΩ“HDu+}ΓΛďιKae§¨Λ₯hcš7ή8Μ¬]»ΦnΉεcL–[;Φ6ί|σxώΨcΕ£ί{ν΅Χμ‘G‰ΧϊGΊB€•ε#εD ―TβΗ‘―Ψκ9ι!PSS£Ω–ι)ŽlJ$²(R}’rJuρ(q"Π§$ϊw—¦rκ2:μͺ½΅›QR&;ub/Χ» Pe$ ΊU|E©xfŒ“ωXYρ‘BD §$W;κ©8OΟΠ·₯g8φt,j'™lLγ°† bϋξ»o,‹ίύξw6gΜΈJΝϊυλ- X΅wΝ5Χ؝wή·‚zύυΧνκ«―ΆΉsηΖ{ΎέS›e΅βέϊΗΉ”VͺK;΅r½ωζ›Ϊ‹΅΄‹1Υ©wΩ€=lS]L=–8ud›QΎπΒσΖ·\.]6Τoˆ rΉ”ΤUnjΌ<τθΏ:PWW§AόΎϊ§ο]ά₯’'C@ς!έυΐΛΗΏ―ιNmy§’u)-»X{}¬――/οΒ.ΑάIο”ΞB󢎿;ιLeω§ͺε·κVρυ–] •r…X½z΅-_ΎΌπ•κ³₯δύ\?f¬εF₯rι‘|—£ /κVΑ¨ϊΛ£Ζ'ϊ‹|ΗΟuύI%·Αk:FΤϊ. Ά³gΟΆ£Ž:Κ½φZ›7o^ό»ψβ‹mβΔ‰ΆΟ>ϋΔγΒ… ν΄ΣN³wά1>όϋίΆAƒΩ`[nΉeλˆKτ<6l¨μA.oΜZ„ΤŸξ|°τ>ϊθ£ν}ο{_4ήβΉέ‰―ΠtΛ_Ί[ο<υΥ?―K&Lˆςν]οzW κΧ=Λƒ€όΛ±σοyσ#%ΦQύ6l˜]zιeq…Ήφκ;qŎk’οΪQœεQKz>‘c`ι3†ϋξ³―]wέu6nά8κθeμGnvg«τ#ΘW.άΘ}όw!ί /?z|G|°ώιπ ‘Όu₯wΧ")ύP^—θƒΡ/ΫvΫm;Ν”ΧuΒzψNΙCQœλ.»μΛyαŽ6¬Ψ;ώ9ς=ε»:yςδhψDΫ¦j@ΑsŒς&Ϊϋ&*ΫΌxΪ\t^nΪΖC\p9ΑΚΟόγmϊτιλΙ"z§cŽ9Ζ>ψΑڐ‘CbΒ½ΜJ'ε™3f;'Nš¨rι‡"φ~M9­b›6mš]uΥUΆΙ&›¨nuP·\2Ωϊμ³ΟΆ‘#GΖΆ­Ϋ5Y­}x°ϋθG?'ž31D-;πl±Ερ½τ0xWεΊNΐίΣ΄λQ<Ό3.s:Λu‘a<Ύ)S¦ΨοϋΨ'$ξ4Τ-Ζz;Λsoά¨+aΪK{!γν…ΥυΒt΅žu6>QΨΣ›}ϊΎg_ψ-ʘ†Ψ(lY‘f³Ν6³σΟ??ΎωΦM[o½΅wήy±qπςΛ/ΫΓ?€ΨŠο?ωΟΝW¦ιOΑΠpYςu£6ŠQ±όT%ΊΎn$Sg|e€Jδ­<7θ­zη2‰χϊΘ#ξ2'P[[Q–τVκ„|§½.ϋΡΣ‘ΌΗ5Ώ"δύοΏ{ksτp±s‘Σwυ{mιB+Ξ ζΞ=ιΑο'―qΎΓ;Δ?Ώž › “Όξ~“χύšŽm 8§BΊŸΞΎ'Oσ0Ι''ο'―λΌ-ς§Ί¦Ϊvί}w£QNς»“ά·Ήνu‹ΑΨd»Ε―· .p―£ϋωΒθZΧ`”ν†ΩΔPΙu΅k{6”ΧϋΡ£GΫα‡žΌ'§Ι²•|Ο’mχΔuGΓ‡oΧOΉίπϊΘVιώπ‡Λ=»%—?Κgο½χ.Ήt—s‚ύ™ΏψsηΧύ·Ž½K€ΪšZ5jTο>¨bη›ν΅LόMNώ₯nι›ή~A0aϊ½ο}oΦƒήΕ,Š6'^h/2Ι<ιΔ-I£ω<ω^bLß;½—N’kGvΑ άl(ά΅Xz?T²oUθΣ γογΪόΉσλώ»?ŽiIC±ιΐ±arωzψΞΖ'rΓιwρœu±!;Ÿ(6BίΧbγνmΒκιΡ>ΝkΦ¬±‡z(ΙμΏώΖΐ$ŽFϋίν†n°gžy&6>·Ωf;βˆ#b‡΄\φƊsεΚ•qV,Š J³*fUž§žzΚΦ­[Χ©r˜—‹3”EΙΞIU.덺χτΣOΫ₯K£QW²υ€“²&€QυŽϊ‡ΰMŠ°δyςΓΰυo䨑6}ΪτNλkYT沨C«V­Š Ύ#F”Εv!ώ}f΅Vˆc YƒΒ2€T¬σ8ωήa$Λ·Ÿw ε J)”₯3δΓ·ήz+Άhρ]œ4iRVž%eW!ρSΖΔΙ‘ψi‹Œ?ή\Ηω³ ‰«’ύ,[ΆΜ/^Ϋ4”ΑΠ‘Ccύ¦#ΧU†΄(ήΪ@”7+"PήΈΖ[Ιεδyg6μΚU+γμ!dP₯΅;Bτ]ž}φΩ؎ήtΣΝ‚brσBƒΚ_ $ίσηž{ΞΨώ#ή©S§v1Fλ ”ΛόωσνΥW_5VκNYΟ{E›’0ΆL–{O€·ά⠝J{ύmp9(†€dl1΄zΦ/ί<ϊ΄½‘ίΕφ]{65]-Y‡Ψrη•W^‰ΫΡg#oέν£w=e₯9ΎρΖF;Χuώκ›΅-GΪ‰θx¨gθmhƒΊή¦mˆΚΎ―E‹ٚ΅k‚βΔb›VΎβ§Χ½Κ¦T|ξαŠGŸΛ itτθ[‘NϊVcƌι4™ΌW„aΜhζΜ™|¦­!#λς:Νx{`L…^0d:c»μθ1θζžώω¨c" crι&πK/Ωππώ γύρπΜ'Ÿ|2nyΘͺcΎπJΊ©΅€.$ΎK.|ψσ†K^ϋν·3αe /dΒ E^ΊXzΌŒί|σΝLXΎc¬‚Ύφ΅―e<|Wr„zζsŸϋ\|ޏόγD‘0%JΐλMPz\ίrλζ'?ωΙLhŒE_‰βP²E ]Αˆ5σνo; ]Z½+ŸωΜg2χέw_& ¦Ά6χ†Ώ'=φXζτ/ž Ζ­βόβΏ˜ +ΠεΣοg”t™+Έ"fo΅bΘο_ώς—ξγά"ŠvOηΝ›ωΖ7Ύ‘ ΟVq†₯ƒ3aΛ—LX0†-&ΞvVΖ7xΰΜg?ϋΩV 'Ožœωζ7Ώ™ Ζ›]bHΈο}ο{™`Lή*ή}θC™ίόζ7Ωv±Κ¦Œ+VŠ²ζυ,(Ο2| h]zι₯]ͺΫ)ΚVI$ΕΩsD^Γ>,‰_”¬/‰Œ–X")ʁςψαΨ₯Τ{Ω₯g&¬œγΊλ»b\Ε΄΅Ίτpθ~£_3'Ÿ|rό†'υ‡aΥήLXY?Œ γ3ϊθ¨η’O‹σ:˜ͺΣ&ΰεωβ‹/f‚M,{tΨ8Ώ—›=ΏŒω2a£†ϊΣQ˜ά8π›Ύγ­·έš +Weώς—Ώτy’œ#:δSO=5r [jΖtψ½άDωuΖ]ΗΆν0Lnϊέ7Ό¬ψΦπΝΩ~ϋν3ώžτ‡ή"`eψώΡξB?^J ½‰B†²VJXϋοαθ’–θ~Ž–}Μ]ϊ‘ΊX>r?ξίγ*΅#yΐ•z>ΊΒ<³§,–ŸΗΌ½ϋέοŽ3νσΕ…ί†ϊ›9kf·Yyς₯ις=OΧʏ€Ώc¬pυΥWΗٟΜςηΔj—­δnΎωf{β‰'μΠC΅wή9ήƒD¬ΑΊvςδɚ½^~U£[9*ξίΣΠι°/}ιKφη?Ω>πΔsfc»ψβ‹γ_θ±}φΩ§Σo°ΗΙJPŸόηνψG\†žeiy§ξΈγŽΈΥ#Χƒ1ˆm·έvΖΩ­Β*Ρΐ°ͺίPo‘hΑ4ΞͺψΦ·Ύ—ͺ]°`ύμg?‹² φώί‹3Dœ}Ύ,ϋ½`ΰd'žxb\!πƒό eC\M…™φlΕωΫίώΦ.Ίθ"ϋΤ§>W&ΚW%_ -ΫΈͺΙ?ωOϋΨΗ>Ϋ2ψΗγΦAΜfΉι¦›μϋίΎ=ϊθ£vΑΔYΞΎ=n~ρΗ·`hf·έv›½ο}ο‹eΐ· ΛϋŸόδ'ρύδ}€όψŽΙuΌqή>θZ,• Vގf6š\οHΦO_ɍ―δϊ‡€Λižξς·»K SΖΎDΈfb^’α…³’OHR—ίή6ϋΫίώϋj°ύΟόOΫs=­n`ΡG§χΧΏώ5φΏώυ―Η~“mˆ4•G₯Ε9’‡ύΑ~W0HλJύΑΘλΛ=χάcŸψΔ'βΚ Η{¬ν΅Χ^± ϊοΫ~ρ‹_D½ΝυΧ_o‡vXEχkzN8!κφΓD$;θ ƒ’ŽΜu)Ώύο£^¦¬ξγαϊ£ŒKρ™πrηο°NΛ‘Ύ[%3ήXLߊ•ΨYq#­ωΚΗΧλοςε+μ’‹/‰;½|ωΛ_Ž^ύ^Ύp½yΝW υmy;{Ό}·šξφ«;{–ξw«ς†I£&ζφϋ¬ζ‹+΅~:5¦ρΧQ°dŠ°ύwR'‹“ϋψεΘίͺ°ΔύaFθ―ύkc-;ΤΝC©₯»'ΣΛ A…Ε:―?ω8Άwz…σϋΙgΆW§άoΎηx\ωξ΅_ς™:ο[tΎ9ζ˜6 +%‘[nΉ₯ ϊθεέ[eά^ΌΥΕ6Σ…^%ΰυ WΛ‘{]Γƒ i0Κ8ν΄Σβ’˜<žmΩn‘FyXΥΛ¦OŸ—Gυ°ΉIτλ,yŒ±3a&M4ρ#ŒuXrσ¬³ΞŠFa†·”L9 γ]wίΩλoƒΣμΩ³£B€eKχΫoΏΨ6ϊΒΎ`³fΝ²ψΘ‰εŸ˞ώκWΏŠ†4ŸΒην ŸB4φ >³uΒξ»οna%Έζgn·½ν΅χ^eΡζz‡DχΜγk―½faE‚ΨI§\PΪ°EχΩ%*I΄YαΩΡ–O'[xCš0C/L†œ” ν₯wάΡ>ύιOG5ΞwέuW•M‹³Τδ·Χ‘άμΆwέύuυ~{αΒΜ“΅·§ύ9:ζ'ΠG|·w―½λΜϋrοh—sκΙwέγβψ‘|Δ0vEΞ;)κΈlρηρpŽq&KbwζΪΛggαΚυ~’a)ζ1_y&―%Ο=ωqλΈφ˜΄.ͺ°ς±H^Kž;š|Χ:»ΧQΒvtΏ£{=—pΈφꁇ-₯c{,z3―ωž™Ό–ΓΚΟ°B c@.—βΕ<ΠΒ( χΡό¨M:5†UΫPώΉQΞSO?eM™fγζhaϋœψ‘δ™‚B,όΗw€Μ [ΖU™ Wސ±}’x½Mΰν‡|ο;χΌ]βχύ·#nχqωβεΊϋγΌ#—|f>ΩΌ„΄•»K²€―³υ|;Σ²χπŸ?/χΊί/wφΞ‘|:kιΟsιŒS§χΓ{Θ{δρΡ¦A)O{&Χyš<=qΉiŠ:HΩ°αΉΦοΗ ϊ§δxσςδθεLΩz½Ižη»ŸΜΈϋυ°ξŸ£ΧΏηαΈN]Ξύ.ψ}?r?_/΅#yπΌ: ηδόΘSΐ’-ƒ|χσε;7>ΒαόΊ—‡υίΙϋώ¬ά°Əξίή―y>ΈξΟπp₯|ΜΝ'ωm/―\χ:Ϋ^ž οu!駱ι]Pξ3ρηΧ’ηΞ>ysΉς&ΰu•{ΓΡH†UJ1€‘ΟF=€?M_Ϊ'Œ†­ lυκΥL9½Ÿ]-i‡ΓXq₯bV^άm·έΊ]Y†σzΆ¨‰+:³j-::τ .{0ζώΘGŽ‰ϊδZΨ6΄,Y“)κΤύχίƒ°RΐŒz†. ύ#γ-\Υ€N‡6£?ύΣ\fπ^pξοGœΏ/Iωš<χ”{<όζI~ϋ™ψΖj6₯ωΚ*ΞΫ<'7 ψρη»?zό~{ιv=ωΒ$γσsλiδ7Οα~χΫ•cΎτΧύ^nΌΙη'Ο=­^~¬Δ»JΏ:ΧP?”“?‹£sL^KΖΟu~{Ί8χ0~{ωΆάλΔηΞΣξΏύθםGς™ψI>ΟΟ=M~ίγκλ#ιΑyz8ΆΗ‰|ςηώ9&yε†KΖ‰ί$~ηΊ|iΑ_O>+7,Ώ“χcZνϋ|ώϋγZ§+Σ°ŒΛΞa%›λ˜ωφ³έΆΔGeYξAˆ½χήΫXήΞ -7>ύ^OΒΎzq–-¨/Μgy'•|Ωͺ‡ ϋoυ ?aDcΛκ’,IΖΐ2ΦΚΔ—œUξqrdkΒ1pL'‰p“'OŽ³ϋHσν„όεo/Ίής•Χ RεεΝ‡ΰςΛ/7ΆΊa₯ :Κ,Ɋ£FŠΣΜ.ev έlΛφέΜ†χώιO²λ».6δ12H>ƒsIzθ‘hQα! κ"LV!nOS ¬D ›£t(1pAήα¨c~€ώνVDΉφΪkcύίwί}cύΟWύ}r£ΆB&ζ:±eΖ"Τwβτ°Ή~+υχ«―Ύ(δŸUάΰƒ°؏{ξΉgόξΐρΏώλΏlζΜ™Q™Ί5­°9[:*πf6FqαπΓ_«₯©­£Š~+υ/βXΘgP$™πύg›& —~ψaΫvΫmσΦo‡h?`LƒCωκqFO ΆΨav3ΠƌύzΉΖϊ§¬όίύ_4rd6«χƒPάSWXiνρφΨc¨tπz΄aΓ†Εύ™gž—Μe–†y΄Q,Ί}*Ϊ΄aY’žv43ƒ¬Pεώό[.υUš0{γ7β¬"κΎ§ΗΓάyηvΕWΔΆωχ—έδΟ/ύV€’-ψΩΟ~Φ-Zϋ³ά‡+«²7.Κ\˜3Cφτ¦:Κ”ΰσ―sΜ=ςœŸώτ§†‘$†v„υ0~ΔX•4Mί\Ψ³„΅ίˍ―Σf¦£wXΆlYl3#‘Ή(ε¨χΤΪοΘbŒƒQ€ΓG¦ξ’³`0χ?όπVό™eΞ}Ϊ5¬Ζͺ`΄‡ΉζšΈβ%ΖΖΞ”#}IΚ–£χ ‘OΙjeΌg|wsε7}FV(Γ8–w“UQώσΣpΉ_ŠεT©iφz±tιh,K}ePŒχϋΦ[o³Ν‘ΕΌ—”1u‹~ίΒ… νξ»οŽο6³d©Η΄“©ΣΎd½Ηή‚mU™ΐΖ`Ώ©K΄X΅πέ»ΎΫ&Œ½‘nRηhσSŸqοyΟ{μύο<χϋΤ;Ύ΄s0CoΗσύ~τœς<­΄u1ΪGoΔ uςΟκΜςΗάι‹ 7‚+zϊωΌ”K’»oνζqsd{Tδr9Aϋ™o'}v «“z d8δ#ό;„{ήθλs½Όωώζκ^yε•ψ%>τ ”1[ζς|Ύ#GŽŒuƒ°…θΏR^|Ωδ!“u=Ϋπ EŽ"O“yEwό»ίύ.ΚpΆ‘wηά‘­¬Γƒuβΐρ τΥFr¦΄Y^~ωεX_˜­Ξ*dΌKΔΕ»Ζ}ΎοΈI“&ΕmD(3ΉΚ!@{ην κΓDϊo“C» ηΊjΎύ•ξό}δ;sΥUWΕΎ2ωG?A“§ίΡ=Σ6₯=ˆ;τΠC³mΐ$£Ν6Ϋ4ΆρακmΔάΆe%Υ9ΎΓτ…ωNψ–Δ΄·ύύδΝw‡_Χ§U£RΛ«Λ Κ =}9d*eG‰6Χ»ξfS¦NξP~ο+Ρ£O‡ά‘Μχφ3[©£Χ8ύτΣ³“Kymd=:ϊΥ΄iW&‘­Gϋt$§›#mDτ|΄=iΏQ7ρOΈiΣ¦Εοa= y₯_Ki‹Π—€ύG›‡΄Ζτ•ΠGΫ]0ρΣ–a! VœŸτχ¬€Ϊ]G;“Ά}[ϊ'Όcδ6λ.»μuΚ­τΗyΏ‘ΜH'[ϋρ mL䀟ΞΔ^ϊE§œrJl—mίύߍeƒ0„₯¬ˆoΔIΆΗž{Δ~2mxηηGΚ’6ϊΌyσ’Ž†ΘΙ­·ή:¦9α~a9ήϊ7ϋΤ'?ex`6ξ‡Ό£Ϋ₯ήΠ7ςo<εtΞ9ηD=«ϊσϋd8ιBΏC;°{τΡGνή{οuϋΤžΟώpτ}hkSO©kΤλΗol;νΈSdK½ƒΊΎIŒ2Ξ‰~‰v :oVfΗQ.p‡u˜~ύXτ\JΡίΪ~‡0Ύ>xH–}2Ο΄Ρ)3ΪύΤ _GΗε}£€δΉ—Οc!Κ€s Qίύξwη}^2|Ÿ‡„ΆqA ΕkA©— •…e&tT2‘3Ολjλ2a08žeU&Vτ:«ρž™P©2a !ϊ  3a`(ž‡ΑƒLP@4ϋkyNό‘J†€ΧΠ™ΝΑΛ5͘~κχΫϋΓ“‡ΚΙLPfΖπΤ±|aΉΘLxq³lΒ Γ5όœ½NœA‘ ŠλΌρΰΌσΞΛEG γi ½‘Ӟ J‹Όα‚ Μ„X&|,[…Λ>X'ύNΐΛ2ϋLPΊΔrτϊ˜/qAgΒςsΡ_PBfΒ΄UΩ‡­’2Θ>―α££qωΖž–θŒ~Ύυ­oe봧%4²2Gyd6Ληž{n&(dcΌ&ώΠ?"ΠEΤΟΠ(Œu.l»”­“Χ³Π ‰~ΒκΦAδνWΏϊΥθ?)o=N?"#©ΫaΠ1eρ²?ΟύTβΡ„ŒΘ't΄²ν%ΏηΗ ΜΝ„Ζkτw}χG\I™S,ΏΠθΜΌλ][Ηψψ6βόYρGγψΓ₯Θ((Ϋ%”σΡOΨW6Σ^™xœ‘S˜ ³£0Πγ ›LPΨg‚QE<βGr…N¬a›ΫΜχΏύxξm?ƒά¬l€LθgN=υΤθ7 †FΩκ~9ϊ;:ΔκnPΈδ7 θΖ6.νwΔM_Œx‚ΒΫ/g,Xν«!³pΉυ΄.lf’qf#)ρygaK™Λ.»,ƒμvώ#†Θ%WΜi0¨Ž~ό^ς”k™ ΜΙ…Q– ρσ}Ζrş2™`Θ―‡•γ’Ώηe@™. Δg–,YΚOφ%vβy„gPZe'9rV‹υώ>Ρ6 ŠΤ˜cg>Ω8x§Όοζχƒ1”EΡOP Ε°sζΜ‰ΏΟ?όVτψ†γ—l|Ήi …2A› ƒ\χ<ΠΖ ŸΌai'•Kωe3_'^W)gt_A‘υ^ζΉυγ/ωK†ώd0πΘ[Έσxƒ’6ΆΙλΧγFF ›ά‘Σπ6:z:ϊ8σ¦›nΚΖGΫΏη%(Ύ3Αθ<ŒΩ2a΅Μ }η’<ή|σ-™0H“Aχ˜ΌξηAy1xΌυ υ™oΌ±]9”Μ™«―Ύ: %[…£DCΌΟαycΗŽΝ„‰7ρ;κΟ JεLPήgψ6‡A(Ο=mΙγ™gž™ ƒό­žYjεηy¦<‚!q»œh…΅l^/ΌπΒθ—6·ΗΑM?OΆ³oΏύφ,Ϊά0 F”±] “Ϊ<3 ’E5νρ$o?η~`Λ¦%ΉNΚ–uΊŒ±ΪδΡλ\°υyΏμν~/Ϋd²‹œM0jŒlΒ@v|ψΗ?ΖίίωΞw²m4χΫΕG•l0Οw˜θ۞Œ«Ή€oνϊϋΫeξΉϋž mΉwδ<<ΒDΎX—ΠΉΣίMΊ` ύώ†­θγ-ητ§σtπ²‘]ŒbΉϊw7y€M•ό#—Γτζ³™‘MF[ΠÎs½οV˜ˆοΉ>ƒ€Αx#^CV]rΙ%Ω°GϊϋΌ—ξ<έΔMŸίγM†αœώb0.‰Α< c₯ε•pŒ₯&Ϋξή'e¬ρst ψ#žξΊ†Ζ†¨—ΘM»¦_ΓؘχΡι3ΞΕ}ςξŽό1~:hΠΐxώ³·Wί †βmΒ`/@›Œς₯wΠAE?ώlŽaΡƒΜ―~υ«6νmΖŒέ ιίΟΓͺοY½ i$έΑˆ*ΖΟEyΩ ππ―Ε_˜Δ―£ύΘΒύ$΄ώ-yέΟτφ„?/™†ή:G&&υUž?R–”ονn‡ΙΧ™`Τ• Ζζρw˜°”MmαŽΖ3 lΞϋα]‘Ωήψϊ”ΙS2τI ώO:ι€ψ|τfξθg£Ÿ,c–ϋpΏύuΜ»VH!Sq΅fEb5‡…Ϊi§fA©hsΞ­²˜q‹EΦΗX>a!EZτ\@IDATˆuχe—^fώπ‡£₯KXb}„œ…ŽQv}ž#WΪΌ ±ΔQΈΦή_¨θρ^x‘£5`ψE 3¬± cφυεμ³ΟŽ«w„Š…j§‚r)ΞςΖz+C¬‰‹}6±@έaΗβ}, “kTfraaΚ~ΈXΙ.Υ8 =fΕPwqžίd:/-”!”ΤU,}™EA³³½“–€¬ ƒΏθrΔ•[ΙϋΡλvhΕYsl‚E.u λbc%‹΅κ—Ώόε8»–YΜͺS₯U–Zκ.4|£u?ηX ·W―˜αŠc&VΒΉΠΘ‰³«ρΓlΙ\ηΟήhμFρVη‘A•λ­βcɍc†«£εsXk3k·lyσ,ΉφΚ?°χ?~‡g΄όf4Φε_«n­X±<Ξ€Ε‚ΧQ|ΡC…ύΓLf—γ|˜Χι$ ŸωΔlXd6.Ÿ?σžaζ~™­:"qε–$ffdπ-πχ―½ΈˆOτ 0+”3±ČΪ¨΄ h[²κ³xΎϋέοZdΟfΨe³GΌΒΆcΜ¦bΕ«γρ›>ZPNdΫΎ΄£Y]™$΄qyvgŽΆΞ¦af$.ΫξΙ δ³•XQ§œWΊBV2Cό² 3˜xoYιδ”SO‰+NΡ7‘―;?Μ~Ζ±οB<Θ`2S‹•‰ qΜΒy}Ι†ii{z?‹Ϊ+Ÿl˜8AξΑύ}Α0cΑ€,Ξπ‚#«μ±$ού8τ„aΕ³όΫκΩ₯‰c˜/.ΘΚwΏO9ςaΔE]α;ƒ•²FχD™q-ΣΔλ1‘%ψω#Ο΄±ƒi,‹€ž «&ΕΌO[ˆ>'a\3c8ŸΓO0TŠ·ό»ΘΨ’w¦LhρρΞQΠΕ}τΡρ½ ώ°­+2œw–ϋAρΫν”#ϊΎ<'Yζω’k₯O€νθϋθ£'oŒΌΰΕ!χΙ_/“aΛύάίoδ}šΙ“'Ηέ `ιί§rgPLώƒΘϊnθΙθ#„ΑδΈ’Γ!b{ο³w\”ώ«T²Cξβρ¬B€γ»ΘͺcΘsϊX΄ƒψΠ6“T²«“EΟϊ'•Ό\©γμ ΐ–ηήG’ύC;U)Σ`ˆWžΛ͈ΡŸ1ήΰmλmΆŽ}1t¬ ‚£-ΐJsΈdϋθΘ‘[¬NJxžΛσο½ηή؎d•‘O~β“±Ώ#hω‡1oϊό΄ηΞ8γŒlΫ‚6#ν<ڌ¬‚CϊΙ+zYt}δ•±κμΣ‘]Ϋυa|’Ύ$«Tβ‡φ?r“p΄[h‡π­‘Δ³h£œpΒ ΙδuξܞœχdΤKπ½C―„Žˆτφ`uτ)ΘŽp6ϋΐ0μ@Ζ]·n}\ύˆΐžρSœ—ηδUΡqΠΎε7«ς|˜1ζKδ“φΟΕr“ώR0j2’v=a(Cڍ”Ι―ύλ臕ŠάΉ­N₯ΕιΔ>φxuUσ86·ΌΎΠ6„;μ>Κ€ϊΚ ˜θΤΠWΠ†Όΰ‚ b۝4Q–θyiCR_pδ‘7—ϊCτˆΤ_μ}κϊV@’ 3ςΘΚͺδƒq!ϊ‹—όό’X·ΡΉΰ‚Ρg|Ι ο%άΡ[ς‡ώzΘϋ”°JuV8O ο+άΰ/LψΘΎcτS‘Χ―,y%Κv_ΙέΓ%γΰύαΩ¬BŒ£ό9Gg“τoφγ?Ν'‘G‘N*34.©ΰlAΕΓƒ,ŸΕǍ aq :μΌ ή°ά&ΕΣ…š?#ρX–(*:Žε€¨Αβ1θΧΫ Κœ—˜εΡq($ƒ5i\ΊaΔ2ήI‡"‘€1 Κ…`˜nI~Ž@π£,aiWWŽrŸν2>l`ΓNJϋďΔΡ¨Ex'Γ±u\ ,H+ΒvςδΙYακŸ’$@₯ Β9Ζξ|πι0£$υV– kΟGέΖ!7Y6 aΟΗƒΖβsΗG›:DŽ{,s†‚T2Ρ ιΨUΤU”θ8–ΚlΟΡΈd Bό&I^‘•>ΰοRΏ—τ?tXs£cΪŽή™d˜J8χ΄1πΡL>‡ό‰Τα&q>ώΘωΗοΉό‘,ig1ƒ£νFƒ˜ktjς•[N”χ“š³φŽ_>~ΖΘxςωυψΰΝΰ/ίΪ$t&(³όγΡθiΪΞ^–ωβΣ΅'ΰο(νdϊC( \Π&ζ]Ε°‹₯|YZ•;œ·1¨3΄Qιwy;…ώwpτΡh''λΚ:Ι΄‹Q4 Δπepc œHgC}s›>ηVφ§ηΕΣ–½Q†'δC}ΆGδ]¦½†± ’†™X1Χ”ΑΗ>φ±l™r‘ώFx θaΰΟ6”³σΛ‡Λ™vδ‡pθNΙΏέ<‡ίďF=ξxΧ©7τKz,ζΉΙχΦύλ˜^”ϊ Tx]·AŠQR’ €‘ cP ¬n3C]D†‡Mb“Aτ ‹^^d—]~Yτƒr½C͚9+l7’‰a0˜fϊ κ/Ζ:a΅Υ'uŽΊΗ ίήόςMα›SΚu΄σΗ;L»φ¬πΝ›xβΰŠγεŸ–Γg+Md.ΞeL0ͺΐЍΎ9°ψΆβψ†ςmΞΥ!Kψφ2Π€”ΈΠ}a0ƒ#  ήNδ¦ΘΎω RπδΞσ¬Β Ε2[ΗΉCηŠ.‚τ  +FωRͺe‡ £_Ϊ6 ŒΊ.˜ό’α}@ωτSOΗ~)ά\.ϋΡΩ$|s³…eB»Ϋu0”mxΆυζώΙ'Ÿυ0>Α„ϋΘa 81Τδ»”,Σάgιwω ήΈσo1ΧβφΌαΖ― <βήsΘ{’±W©Ύ1έόΗσŽnγ_\e!Ετ1ό^7Q6Α½>ΡφΓ!W+C7€|ς~m`ϊiLΐa€™ΘJfιyη[!+†4ag‚Vυ‚οζO~ςγπ ώTΤ{z˜Vžτ#UX¬q=νRΈη7mY&4Σv¦‡ŽƒοrΡ^ΐaˆϋΉ“> N.ΌΰΒh”ΖuΪ τΓp>F΄όCΑ}0 °}‘nσ|&aΜ†ρmΪφ΄  +¦ΔX0Ζe‹NoFό± ο5αθσ3–ΞD\Ύv½χui“ND6Π>’ΝΒ3]ΧΘď\1.ό³ l»ƒC_δύ†gŒ#ιoc4β,ργ²μΦΫnmWΖЍ ηπώˆϋχΆΌ“ŽίΨΐ‡w]‰;ϊΦΔΓΈ0²qcςŽμDЏb¬–οŒ;ΪΪL§‰q ρrΖ_n<,GΖ9ΰœΤ©ΈΪ₯΄ϋ©'nΨ #C:0("θxέ`‡ϋΤ&PΠG€zM>·7Ξikc„Δδ ϊ%Ι­ΓΠ₯P&΄m3_ΉrU4Vd<ȍ‡ΆΨ|‹V ΠabA1Όρ~,i§ΎΣWy1Τ'κqbDδ:Ύc8d6“‹ΌΝ5ϊBθίιγ„•˜£ž„ώ©ΧOίAϊH8Œ”θ‡?εγu1ήμηΪ5¦‘2ςrS©Tϊ€ J/ km,Άθ΄Ÿ iΨoΝg‰0Pΐμ*’WΞ4HζIηΕ 3‹ΰΐ(Ε SΪ‹Aθ '>XtN˜}2λ]³ΪA1ΙΛ†Ciιu'^Θσ 1$X½aœγ9^J„3ΕQˆΊ_.ΌΔtπΉG]O:^ΨΩA!CX> ΤqOSŸΞK€Λ ¬˜™9μBήζΥ·ά»`§α„ΐ§1„₯¬+q’ώ™5Ε3©§Τ)fͺzγ#ιOη"P d±Οfqηυ2ίbd ™|€_:=§7n’χύάŠψυƐίΣΡβJp \Ϊ{ΧΉξεw8Άη7SΚΉB4τ‘Ct˜ΉsΙ'ƒ1ωκCΎΈ*εΖΏnόεu8_ήύυ;Ÿ?†R fθhiυμ'LšΞ" :’>KΩΏKΙψt^ϊΌ\QΒ£ MvyΗ £½ΐ¬ϊN΄G“ςeJoŸψ‘φJ):ςaΫ‘6KΪΧ\gFŠ'\;2¦)}=›Κ†>ƒr NSŽn4ΒΒ•”Gy (ι£’½η»ξ§Ψc±νbγο/p††ˆ(Qžωj`€ivθ‡1# …3Uρ’ c¬l@Ω ·`€šχ#ut„Cfσ>qŸ2#ϊ­9—ω(§xz οSς|”‡ r9ͺUί‘η0pΞΉOρ6χ0pΓρf Κt₯EχGΐe:ςάυήfγ7ίxκ JHΌ»ά§ ΰD±‰ŽΝΫΓ†³σΞ=/UFΘuΤ9Χ?`ΠηuΚύQί1κaiΪ~(•Qβ30ΟεΎ§ΫΓ•q€ ΘΆ‰1bόΑΓ Έ£Ϋ»/&Ή£ΛDoιύŒ*ψ@ωΜͺ%Ήz Κ Ρ =²Ž|—)kϋ.’<θγ>Jfoϊ}Β#—ˆžξσ}Η ΗΚE”·wβΕϋ‡²Α!\ -œΜ«ϋ0hƈ·mΊ\OXΆ„|F“|χ(;žƒQ ε‡ξ§±)ε„ρΚΪ }ϋυK¬”άδΦ7κ Χθ‹3˜„acŝwΩΉƒ˜*λzύσ~x^0£―α.—§_―δ#rΗχ€ ΕΤ+Ύc>`>;΄sω^PΧ0Τg`™±―‹•ΖΞλ\_N?ŒΆνs .˜ΜΧ]wƒδρ±έΕ[ήέJγU*ω₯\ωΦβX1Γ ϊ|ήη’έζ(>6ƒ_oσςνf2zκƒυήf.τ]Α€‚Υ¨™,AϋΛγMΜh pΨ‘‡Ecϊ‹τυ&OžlΤC i»Σξ§Ν— GZ1¦Α8ν;ϊ±,:q4+Πδs΄[q«V―ΚκŠωΤ§σΒšΏθΉœ=:JΪΚ΄™]A»Œ>l=ΟtG[c~ΪΛL`ά½3α MΟΗΨ„o)ϊOΒΕΏΠ£ύM۝νr_A–:ƒn&΄!±A€­Λˆ½φή+φυιOϋ*γ~ΟΣή•#qpΐ±ŸξΊ_\ΖQhg’6― 0"}8&Φ$ωΕ‹½ψ—-ύtƒ”§·΅Ι Ί*κ!mΰαΓΓκΑΑΤΣθυΜu&θ=X†ώcΖ³ΰ}Y~ΣΗ!žΓ€^ aΈΟbΤ₯Οή™Θ]ϋΏ³»Ωl6Ωά!άlΈDEEEDQAEό‹ ˆΗσy!Š’ς@P‡Š κCρδPDΎσŒΓ ο3CYa›Ύ£ΎP§™Ÿ…p_ Τ.Ό§]”¦T|…+vK°€ΞD‘™qOΉΈ‘g1ρa.ΨΌy]Ÿ)Χ6κO•7m€’c&Ϊ%Α—™ΖvΎ`πď ©Φ•.έ5 ΤGw}Ωξε„zƟϊ#ΚεŠζR’I&“Π ψΠ$JYΓH ·τε AΎPZςΟψΒ)‘ΉGΠLϊ™?¨mΤnΌΧ\*@œTςΓ°’AO `p“b‘(#—Ϊœ‘…bΖ/Ϊ™†Βˆρ %:μξe§˜ Yxα Κ k5ΟΈΠξ6εh9`· C6\°SEJGζ“ΌCfWΏŠΫ[σ€΄5ύ”~¨8Z8‘Μ’rεHοj΄-κϋά£X–r…3@όXήˆ. Π§θ δ8hω"ί eQNιΔO0Έ„ήQ~²+ΊΗ˜= 2ϋC=ώD<0Vμͺ^(β‘½ϊ» Ζ»Œ^„?ζΰύ”…“R|\FOβγτsxψm…JG‡n,˜jΜ•)[¦uGΕυ œ’5₯―"ΚšM[ΏθϊΨ| -²»•MLxμ/eξ’Ε‰jjW /Θτθι—±ρͺhΣχΐ³ψ7iψ/ οŠε―8ιΪΈ έ‘ΖgŽήΰ(6ϊ5Η9@ΗπΕi\,”™κ~ΛΝ·Έ‘&‹ΰC8½+Ίy߈Ι3?2₯/š§tx xΈdSk1ΎΨ1ΟbN€FFG˜Q+ΒπqΌb€CδyΌR`DŒA¬πάxh΄:ΠΘX8]8Šγg³ :tΪπ< α1’ΌθxΜ7^ <Σv²c Œy¬Γkq1p­' oJ7ΡWτ†ˆa“¬ΝZ8ϊ+ΙΪΜ3%χͺ¨—κ[x₯Oj BBΆΖ˜MkυΜ‡δ ˜ω―₯Z%‘mή‘Χ‘ay{ώ9²FG#= xρτΉΎΛZ>΄Ω1£Γ «hΖJΚ¬2N4.«ΙΏ€1&~t8*¨ΊXτPq¬lY@'U\i4gΖΡόZŒσ)MzŸΕΏπξπhΕόΐ³^6G”£†œςaXϊςϊΒaQkυAtfGc žΤ8VWΐΒ+†5(ԘǑ 0„GIΔόύσE-Œ³ΐˆXηwž+•˜ VωI‘TΨ―xFIΖΩγτ½o}λ[ξ1‡w€Cωˆ"9%ί’όΛ#ΙAŠ§tιZŸόPXz΅/ry!£9Βΰϋ(1&Γ(ΊΖΈ’ BσD€ŸSΎ€e<@¦ga Θζ[lΫγ°ΌΜGyŠ}WyΤΫU2uaΉΛα=Ž+\ {(„Ω ;ΰ±DΈδŠή“Œ£1Φγ™Άΐ³•‹{kόs©zHq§#m=8¦όπPvwuΝ6CΗ3¦UHΰ₯ΜWδύk¬uF)ΟΘ0άFc(s0Κ$»!Mn<]¨|ιΪ\€φ δ5€˜C,ΰwάq>wSœζΒLΆΆͺ;σ<°(Œ‘‹u@aαYaΊfsjΎ_δDpαyΘ±nŒ`P³Ϋξ»Ή1 c 2ι„χfΑšπ„+†4kq,|[ΐ\―ΏΊuΤQ.λ3Ž 5Ύ„“zΉ’‹Δ(Ο&xhβΐS?}ƒ+s>Φ Ε7Έ’‹CΎeΝ# ŽήΓ(F+ΥΤ£ ιc ΣA[ZίFF„ž0˜KΖsΎ8-ρψg\ΟƒUf<"'re-ρŸwό3<όΠΓ‡Ή"k¨{|³πˆ7tΒo ή1(^Ÿ0NχΜ©5‘^Gd²vŠAΧ0wΟΛR«όςσsΙg*—R.cž§LΗ…ΌΎρςKZtl0‘­?ΏΜϊ>†LθΩͺrͺ eŠHNžO―ζ{W8oϊ†jŽH’oΡΆΘΪΒΏή^ΕCΡ32'Ε£'–˜—² @8δ)9}LλŠ―«hŠk!ύ°›τ<π Œ&Ω„ZOyMυ΅€1* ²[ZΧYcXΑβ±v–I¨"5paΊτܝh‘%O©6W8WpΞ eη<ΐ’’XΡ1@vww»bγ3ŸωLΙό 1ˆ‘ gΫᒝ εW¬•ω@pΩ3G<(!¨ΡI iVΟ0pW₯S] ːžλ •(ΌEγ9Ω(_Α|x>dέ BB^μP4£tL’XxΥβ€ή―Λ%έ% TŽXEΨο*Μ«b¬†±Φ‚RaΡ"ύCVΩXς‚βI!ΰ€Ÿ…q›ω™υrΚ‘bΠΧίηJή‰'”jCβχά‚4΄1“Lv*£˜Κ₯σMτΓβ”&mτ  ~΄pN»ΐ³KΕ#œ>%ΉΈ°/7mΓ…@ή₯ ¬ŒΗζ‹ŠS2“‘>ΕbG­°‰ά…Μ¦ΉqΨ€ peΣΊ$ⲈΪl8T}E?xVќBψΤ•ωίFnδΈCΗΪJ©…{Τ€?ΒΊΌψ#σσ_Πr^MXDg±oπ­xΔτSS έhCtcxRΔhώΜ­0$η!<΅°‘‚Σ&N9ε”Όχ?²!q[/Δ`€9bΒΊ!s>ΡG%•†vJΕ'\ο$kͺ’w|‹pN…Obpΰh‚Σ^α‘λ‘‘Yζ½”?Nλ &ΰ‡2bΜΔ:^ΧΨԍ19IΝpΩe—ΉΑΈlψΚWΎβFMGy€{kcžŒŽπΗ/vOΌ‘ŒP”Χ4k#០.ΘΫύΓ&Dδo ΧρbΒ€ρ+=Π(bePq29½™Ββ«·§αKJ)¬€\™0Ή7 p,2 :ι-Xΰ Οa^ΎΚΙΪΤƒ–“O>9|ύλ_χt8LΧr΄}½ ŽψŽ·Σ:Tyόr?δθίcΔΔ&5ΦL 3Κ@½XΣ Νβ4žΙ”4¦‰ΛUs«BT˜I†4"`½‹σJχ‰΅5IχεjŠ΅1†4H0?tjn8ΰŽ¨$?h?(YωΓϊΕJΤΫώ~[ΈςŠ+έ…/ƒ7L@‹b\‰/†­Ό΅Ή1έ‘ ?‹±AX±Eq' ΤΔϊ7^ΌŠΣ§ϋ„ρΒ€ψ#“sH€Ε₯RVΖϊ‘Όlΐπ 5yυ€θ‡έšqe|£rEњξVmΐ˜>˜΄p$  wBΚΪ5kσ !₯v;Ηιξ»χΎpΣΝ7ω8Ά‡M.Η”§pΟX*ƒή%Θb@ψΟ’oyŽ…+hγ3€*’OτΎπJžΘΑ(d€¦οWχ2@£}FΚ³πιΉ>1@ί/ΕC΅λ‰Ι©&Έ’•b}—wr/‹‘‹―cς,χMς׎Vη±–­–Σ_%λ1†r¬ηx42 r Cζ&π6j v.šƒξQ–π‡‘ ;NΩΑwν5ΧΊχPv€±«Š ΰŸy#ΐβ²6 5άΌΛλ€Ξ’GY‰lĎFq›šμS)°Γ‹?ϊFSŒ5°#¦ο:ς]aηξμJΡbύ΄ο€x͍ 20€aήψ‘}ΘιΎΞβό†>"Ω½”Ό@θαPl#ί£8=υΤSΓ—Ώόe.”mγŽIκ«βγ(χ«αγŒΧgu–}τΡn@ƒά§φβΚΌHΝͺΕ΅xb΅ιj1ΎtΘΜ ΩΛ‚Αo>Ψη”(α―Ύϊjί<‡ηtqq;0ς>c―«Χh.Ώ)q_!]”H‚β,Νxβ‰nΰ…lΗ"γ>χŠΣਨ¨zšΗ ™… ‘γΌςJ?*“wΰθȈ¸ΝπŒ‘Η₯K]^•ώX4₯+ΌN›wΠ%”λ›_Μ59‹{Ί2Fζgq]ρ‰— 61 vCΏ ?ΰ㎡ΦΦΆΉ‹xLΑσ <Cr­Ο!w!'pβΊ;Ž·ζ(^t Θaτ—‘ά±i#Υ^gŠΕcξ‰όƜ”ωe^hW9œώ+ύ aκΏlΚc> ½"2ž°ξΨέέ>ς‘ΈΡ r=υaωŠf #ŸΙζΜό!oQ<££§\p‚c8A]Τfί0Η&>F”e0FΒטcΓΏ·T=h#γŒ3ςΏxlbη/JH<ή Έ&žψ—GL?~\h@Ώ£ΉQ@c¨?4Ω8ΰφŸ_ψsΗtmsT;]Q‹W34d~©Ep(ϊE7wγ_oΜartγa~a$Ϊδπ؜ρPεΠΛα)‰ώΜυ°Γσ>­ώK\Ρ`sb+[ktτkΎYT-όc'9cΪΆϋΩύΡ;fƒΠ˜ΧlψcS•ŽΪΖ ›q†±"ζk,ΖƒS ΫΰSχ›g,βc$°9IύTψR?DΟܚω°tgž(ύΤΤ^θΤ8Bω•ΦQ£ζδŒm9² ŒνŽa ϊΙ½Μ Ϊ~ ΰ‘†9›Γβ€ΚAd1 #σLζ”š §η6–’»“‡tϊ#|ξyΟ{ž; `^(™Ήž)ΉυiΚhέΡr?Š£0•]Ο•^•έ£œPVD_C‰α†”“ρόHž%.먴 πv0oΗ@(,―ό(?ŽEu0ω+Ζαc>–_o]jkΈΊ+t¬υτf½»#kΣίχ0C*θ+€Ά Ι‘Θπ’εΔΑσ m)έ aΥB^Ž¬6αΔ‡6™«pd-8₯=΅Ρ•°ήžπ†HΦVŒ\ί!xa\<π@»0 “1 αΠψίsτγ•vΑΈŠτ¨ FQΞ°ΆD{ιϊ8ΊN›adS΄«zρύZυ5΅U–L‡ΕνπPX•Y₯θ Ž:C“Β”{uήA;<£ ΖΠ¦[l€!€4·έv«+¬Ο=χ\_D.€AΎ-F*‹9:1‚-ƒ Ψ0T:4W`νψ±}Μ]~ŽT•']λ’ ±άΙΠ…„”χ2άς—ΉνXK³pN:Ρω³h†RWy πμsY€KΒΐ¨1O{Σ›ήθ;΄αi(!‘YΡ­,ώω»W‘Kρ_πHρiWL(Pk²ηyΝ5Χψ1zβθ…y(―f»ŠŸ Xj1σŸQ²σNxdr©3ƒ“δ¦–8»ΤNΒ!Κ>vι³€Θb:γω)_v7aΤΏΑ]*.¬Γ€h”°(Ή~τ£Ήρ 1Τ.ΰά’T}Υ«^•Ÿ„(—ΈΟ¨­˜”Κc}W΄qžά3I‡ΎLByΘ#Aγa@΄= ;’1ξι£L~ ± Žι(Eʏή(zΨύΙ†Ί± OΐXόsŸϋœ»Iυ؏λY׎yOM('IΛ=,6κȝRy(―FΏ‚{ά‹£,`Α~㞱\1ΰ¨ ΅e)Όρžk2'51ΐηq θΟϋΧ;Ž… ?πΈREFq݈§ρcEπ   BρJ߁ώδhΐΈ±Ο(oδ+?,ςƒςξγxψΖ7Ύα;‹DΙΟ)}ΑΓ"¨>Εβ¦°„rNΊ¦kΘ‰Θξ(­1˜Η@W΄€οsμ:Žΐϋnpσ `Μχ‡?ό‘άη›φς²2ǁ3'‡π'>Α\>Ξ±π6BQ­{!^Ž"Ή‘χ@Gy(~£^UOMp½εΣΏμ ŠΥWόX»½΅ˆΚψΚ‚ ρϊΗ¨‰yΊš £Ε4JEΑΰ‹ω‹NMΔUϊ9hxϊνχ%‹ͺμjΧξwvΐΗa Λ@/„ΌΌ`~ΦΘΉQpPi= θ žΖψ @_Z¨_#kxΨi§ςΗΫ4šρόHx_s €+ΈbΎˆήΨ°ΐq4,Ψ’wΡ&iO”~jθ₯Ο;ο<Ÿ_‘?Τζ*°ΦλXWF†Ÿζ=χνΏί~―wŒΠ8–: |$`‘#df6Ρyœ–ερψ ?πΘA\ »»Ϋ bΠƒ HΏ.ZΔπWσPθYFΗΗ#©NžYξ= ²;kG”Cr―%ƒr―:β‰wΥΗ‘pΒ ·@ή-Μ“2`tΓΊ>uΠ·ωN\>x:σv€£αg…yωΛ‚π€Aς1>[rx§ Π{ o3ƐœΘΣ| /?·ώ-»ρΜ_δ~(£Κ¦Ν‰Yc’ ίc€ πΚ‰ό_i$ΐ@†Ή Η€ΙP,8JΦZŸΦ’›pZ"hV†μρό…όHΓΌSϊqΒ1 ₯e%mͺ8(Qτ(A±€€³0`b†βc&Ιt0”L1M):€eΒ' δ‡2<‹ŽΉηŸΎ/ba–”ςήΐΰ…ΗσOϊΣΞh±ΚDψ‡I`M 3@ξΕΉΧχΉOP[¨”E;₯ΪΒ€€gδ }1aω ½Hΐδ­a­Λ`ΐ9‚ (4΅ƒΑ‰όpˆKo&λiK•Ε#€Ÿ„2ν`ΕϋΞwi†:Ο‚±˜„ „Χ,ŒYάcΡ  / P3ΉΑ}=φπ±Κ“+,‡1 €ξ1:dE3 z„hΒr€@QΪ2EnšWκΫτuv΄cYO{ΐƒŸX$=ηœs―γ?₯c<ΔΠƒځΕsήΑSΨK€{„ή=Ν2Ÿρ• i°Pπ^£γŒRΫ '=πxόρΗϋ„ϊFAω―‡c `l va’‚ρ.2vœ ƒ0ˆ8ρΔύΌ1žΪuΧ]]1Λ’}†~ˆΗy±Iν"μ5ξΉτ΄ΣN »οΎ»σ:(0Td׈ ±ΐΒHr ωA³@K,μ»οΎ.›`άο€ηίϊΦ·ςΒΠZaήΠυΒE ]ξAΡΛπxx=q)'žΘωΘaέΌ;ϋά`Ώ’'γj©ŸvwwϋŽzμ°CY`G Š(Β5ΐυ°άι’—ςPC;ψ=c7‹9μ΄’1ΊΈ­–Φ―!xΰ―ΔΖ‘=k{Ό?_|·TΊFGΏΖXΜʜ{ FΠ³ζΐ\@Χ£±£q&ZaŒΎΠc½η=ορ9}±ƒ Μ§τaZόoFœ9jψGm‚.Ο3¬½!s!Ο2–³~‚œ„Ό‹Οu΄):κB~KŸ™5«Σ.- d^tu’σJΙ‹θ1θ_¬5"³±πΌ>£ut δ…Ξ–ο⡆1C·zI'ΉlNωSΠb˜ƒ|C&?κ£Ή!ΖθΨ1tη;θ £ώθρBXΜS‘A$‡ ΏG&ς¬ZMSƒ{ςf]•ρ V”9‰2QπΙ¦xFj―°^Š>CxT‚ΒπXΒ1ΦX™Ώ³ΆΐΪί’μͺ4*/ψξΆ9 γ-ΐ|]83Μ·Α-p„ιΚTWpΚρ^ΠߐŽ€v–AΊZΎ%½ΕζςxΘ\9ε‡θoΩάF:Ψx>εΟύ¨ήqχ’9KΎ·r?Lΰψ%ϊCtΜgΐ²6sΚρz [kΗ΄Ήdmp(£tϊ›Ί ΡΕ‹{AΒaΜŸ5LΌΔΠVΜ+1xgΜΗ0 @7sψα‡ϋΓ-πK_Β TνEίΓ‹πΑμ}…τκ·’h’ώǜL›‘5ς€©ήˆΪΚ²Ά‚ «°?Ϋ))czΖ|XœψΑvrδγ™URό*έ7(ŒΈ½f6xdLqονoƒ‡ι]©ͺλ½1²Œ Jž:+ό3£–ŒζαζA!cΦmž₯uΆŒ'ξαΖTσŸ!_³Νlœη¬——ςΆΑ;c‹ΕΓΚI~6gLAU2-VdμaιςN7SŽΡ” Φ;'ΡΫΡ¬K–‹67Ε£Η3‘a½xΚΟvg|ÁEιš°…TgL=£4ΚΜΖ2&dM =š!MΖHŽ>  όlW«ηUŒ7Ft¨άΫ‚_ž˜!uΖY<άR†ε)~!>™βzX¨³SgΜ0·$ΎΰέΆΩ(_+蘾S ·f̘±Εo %ωtζΩΜΓΊζΜΙpΠ^ΒλΌ#Ÿ§ν0χχΌΣϋXO:GOΓώ¨ΫBP†ω7΄f‹c[€ΛΨ"?›αUΖšόό¬< yύ  <I£¬²*ψ΄³Œ-\{.͎+՟qȌ^?Εψ›ΤdlΑ±ιq&|!'‹ΧΓ—-žgΊκͺΤGΩW'3™ΪΤ6Idl±Ώd Νˆ9ƒ\ 0§ /œ³φŒN΄-ΘYf„αα­nžO<ŒυK­3*­ΜEν$’“_ΗηΡy˜1MΖ [ςίS”Qs?’RώrυD7Γ–|έ$— Ofμ1μ;脐WΕρ‡*~ΐ—yβΞΨΖΛayΗυ€?1.ΜXΰ½mτρ0Ύ­οKVε=ϊh€φ•ɌŸ=Œ;Ά+cΥ3fL“A*y<ώΆy]Ο ΏK&ΣwHkΖ!%ΛΜ\ΆU:Ύg†@ŠσΧ=ϊσX›Aφ$Lς"ι˜_Ζ|CλΟ”cphΈ„mξΙγ)Ζ‰VyzδwΝΙΘw2ΐŒ›2fVO¬“"η ΐ‘ ‹Ο\0/03ΎφNψγjΖήŽά›Α{ΎΏ’Ά ?cή™KκPXwg% o™‘” =#γ”Ότmθ7ngε1UΧ·FYaέ*͈Α]ψΘZΘ*δ–NVynέꈝ_XΉYe<¬Τ–v“e­Uͺ )|μΰ,oΌw°ΓCΦƒ#ε*ϊΑRΞ:«ομΑeš 8nεˆEΗλ`₯ νaŠ»Ώ±~#=ορ £s ‘7Β±ψΎΠ,FΩΩΕ9‹Xžb5ˆΕ%V/zρ‹Β’ –x‰O:¬ΰ°TΔz‹όjži\Υ³;€ΆFV£}ρ`@Ϋ°TρUŽ%m7Β r‡ψψΐξWή1>°ΛžO‘ΟΨέ€;M3šρ6"\}Š+}ƒ]βμnωΛ_b»rY‰qŠέδ‘4Κ7]κwτo<ΧΑko0:λ5™‚~ -ΐCΩ-z@v•œ’χ‘F΄ o>ό°Γ]€vMQλ;œ ΧΝ6ί,μς’]œo‹~IGήμΚ’'eΛ·ωΦglΗ$2žτ£αQŒ μΔS‚¦ϋNͺFžΗ1―a7=c^ל.‘Θ―Β=8Ε ¦ΖEζμ.c^Αξ;[ψφέ?μ„W»’<™6μΖmπž1š£…ΰμVcξΓ7πRΠ&”­p—žΏ¬ΓΥZgΧ² 23»Τΐ‰Ζ2p)―’q5ιSf$γξ„Α<5pΗ._txΣχt%.ν€,/οmj+d%Ž~Aζ‘m™BσΜ)ιπqϊ‹βΣVŒ΄7ε.”1›tΔOP έ GΪ™έzΡW<&ΡG‹ιFΪg΄;/E—‘τΘrΗ}Œσϊ>sFθ z_@Wθ<ΰ-Θ„zkuxω¦@/"e }·ν…§ΠwΨ=‹—+δ™zψ;/‘­+1ή‘…yΗΈZ€Ηsm£ώiΚZο‡o΄γqιΣκλŒsΰ~NšΒ9ό„Γx!Cφ†Ηπ ϊ68Fή'?Ϊ θx e,叝‘΄‡dϊΒ²R>xω4―GΎa'2ψ@ΟΖΨF»ΡVx[BŠω/tLύΩ½Χ>ΌΡVΔaʋNa:'ΩΑŒŒ3{φ¬Ϊί#;zΗ=ιyOΫΣN Πž )x&Ί h"¦ °ΐ34HOcwyΊ€Gγ] Ohτ₯YϊјΜ O\Œυ¬oΞΈΑΡέπ|hŒ°Bl&Θb5;Ξj>Τ¦Œϋθάχ‘aΡ)K>’½‘ΉΗθWΫπ™χΞ’zތυΘSΘχή{―ΣΊOd+ζπΘ„’χH§>ΕψΟܐyή΅vΌσ; λ!oηΧ‡ΜɈN‹π2ΎΕΌY†u ΦΎΙ yω4¦EςΓSc ² G QfΚΣm²9ίf½ ω™0t4Θ!xF&7θxG;ώ_δKyY/gԁΊQδVxδ*Β™ŸγY„Ήo!€_<ωΰ%Ωο>Θίθ ι³…iνψ&:tIxτ‘OΞwΐ e­θΚΟ]θ\8γ1Έ$>΄D΅/u…nΠΉ=η9;X{ύΡy-sxΪωqκμ₯6ΗBφ@Ÿς:+δt0/;ΒτςΘ£ΕΪ‚ϊ“ΩVερ„“πνΗ²6tMΠ/Qψ₯h[8B'A™9b:Γ $Έ…ΏβΩ pΌmeηA›ξ`8}ι/ušEž‡NιΗxY%_ΦΡγXŸ€6ЏOΪ™Ύ"¬ …2ξ°γ~2‚Φp #/€ψgžqfΈυΆ[]Ɯ—xΌŸl{’ŸiVˆu«φδAhL^Pθ‘τ₯£ΐ(x§“Œx ϋ$£#¬TΕH‹»( Β-/*_Ί­a Π~Π„hΆ―βΆG(βLP˜4ηγοr΄¦pΎ):UXœe  Ξ³0^αs₯ι”wΊΦhΛJι±]Φ$¦ ΡELK’CΒζ€T&ξγτqώΌK00ˆιŠ{θ‰φΚυ‰8Oά;š₯~Eyz€τ㇀ψ‚x aβ…qxσΖ;₯η]ά.ά«­y§< γρœ`8b<γΧv~¬άVΖqH-|s₯]$‹(η8Ύβςn€~¨τιZίˆΫΌοœήuδ»ό8±Ο|ϊ3azϋτΌŒZŽΚρda'ώa1_‰ωEa<ρ•BڍγΕy‰ŸπžpΚ­0•₯‘ΤSΌ€zΖ|WυŒqEXŒ―±β^ί&_αYe",Οηz‡—qέ©—κΟ}Oϊ_HΛ…οK΅#ιω‹Σ~'nΫR}VνΓ΅Τ·x§ώΣhmΎ›*igΪΈKO˜xŒθYΈŒι_ωŠvWό8ΞGιΉbΜΔ4ξuτS o…Ε~ŠυΏRι qFα™η8^α=ν ˆΣ0_ΜκN^ͺJΟ•8”CνΏ«·{κˆΎmw―U0[ Βΐ Ηγ^ρΉ~…Cβkkβzϋ™ϊ…#Ώ”ρ½ΧwK½'n1"Ρι”ΖoμGίs±tΌS<] σς™€W^f WyΉb|Δ±mζωŏ­Ζˆ5ώΎκ^ψύΒgΚΓ7₯©|Δ‹ΣΔρυ}ΚGفΈ>qΈΏ¬ς}ό-₯Ÿ¨«p’z¨n|OνΒ}/Ζ‘β+nα»RkυΕζ—qZΎ©ΌΉqΗ#Οΰœ?βpU>-­ΗϊC-@Isa `EgnͺK–λ%¬™Μ5x°£OJΖ‹_`α9ρ»ZΏ/UfαK\λυςQΧΈCT“gŒ§ry{WΨΙψnœΟΕ‹7Ϊtδ• Ά0P =–’ΈF1m‹_ŒI§γ9xΖSxΊNΚριRΌ}βJ319ΗtΕ}1ڍγ¨Εβι]ί…—uΆcŠ²έη_€Η@ŒCŠα»0ρJρήΕρΉ/–ga<ž Η@ŒΗJωu9|“{œg©Έqœα%JOΕ0POό;n[)Oί„ @IDATCΉ3έώλ«qκ_,N!^*MS―_‰γϋ>ο›a§)υ,V1Κ΅WaΌJp_ŒUR¦Έ|υtγ¨XέU—8a#αd€χδQ¬= ΏSŠβx•|‹8ΝΠΐk1(ΕΓK…ΛcͺΓ*iηRτBΩ‹₯'LPŽώ σ-|ŽσQ~\ζŠα-#χ…ψˆί—J_ˆ³RyΔρ οK₯ρŏΆα ίRε(,k#΄uŠqΕ3F.Ε ŽWx_ ΏΕΒHWoc}_¬ά)¬Ύ1M£₯ϊΥΤ–Ύ˜L5΅%ͺ―ΗΌ­ΤXΗ©’OMIb\”Β%‹γMMIλγ«₯tέ“)ƒΗmΕ})ώ«x•Δ)ΖsŠε«<‰―:—‹·ͺ*¬X:ή)ž„yωΒpYp XΩγ΄₯Ύ“M]έoœ/)KεΗ+V>*ΧBΉ«XΎŠOΪrΈσζˆΣRžRΗ‹οK₯)UΞΒϊΔί#ί±ΌσΟϋΈΎδ[¬n„ΗρβϋΒψεή‘ .β΄Δ+Μ[iγx#Ε!n±o)―©Ί–4¦QpΙƒ[)ˆP„―wT χˆΈtΈJ.€Η/ξy•ŠWλαjt¬°„κ& Lq*© ρΙ‹4Υ€«$ο'a a a`²0€Υ/< ^X‹Ό,ζΝ”UΦ­”•Αy4eNό{²¨+}'a a`’1_ŒεΩ‰ώ^5ωςoν2 ζϋ Θg4c@%u μΖΎΙίD}OίJΧ„„ζΑ€xJ-*›h…˜‡‹ŠοJ‡RmkMΆL)~Β@Β@Β@΅€?ςW«2xΉϊpΔΌΌZύOβί尚ή% $ Τ πΪ€ΧξZηί;hΣepσΒ,#ΧXN―χyYήςΠΖ‘JMvκΕά(Φύ „ /“]fψ^Œ{ϊΐΨ ;(ΜOΒΐa:ε9Άš\ϘFΜ„³Ξ8γJη/W½θ<έέݞΎ5^yOT> –²rύΧΏώΏώz?―‹σΪ@6η\rΎg]κΜί‘ΚHΛΉoΗ|ώ,lΎ‘ a a a ž0Δπ’ρ–Nςs(Ε7k‘* eδŒPŽ1δœL„ΈΕ‹ϋ‘œΣΙَ•»ΩqΘxπ©O}*,\΄0΄Ooχ1’’τ)NΒ@Β@Β@­aΰ±Η œ·ϋ| ͺ‰ΔDΧCό›cb93ύΊλσσw9c›³{9šσtuΞr%εQžW\qE8묳ςΌŸπρΞ!~υ«_ωωΒ­mΆ3{ό?3ήΕNω% $ Τ ΑŸxβ ?ύγx~ƒO­_ό–2’[BςΰƒzρΠ‘pήύ»ξš?Λ}€rKGžμg?λg―KO3RΪτ>a a a –0 ήuί}χ…Γ?<ΌωΝo₯β-‹x:ς8μΡƒπƒt™\οŠ&΄@ιΐo»νΆπΙO~2tΫΊ !—*C OHHO ΰ\ΰ€ο|η;Η3ΫqΝλpψ 7άΠω ·ΗΒ/Ϋ5μς’]Βόωσ3&U?όαΓOϊΣ°Ωf›U}JγP'κάrΛ-α£ύhhoOΊϋ‰nΖuΖχ§Ÿ~:HvꩧϊZ‘JLtRώ ε0o@.emπδ“O;ν΄S^N-—Žw%iX`δo"‘RF=‘e¨$o!ψŠ+ ώπ‡|΅ΥΞκzΕ+φπΕΨ .Έΐ³yΓήΰ Ι;μ°CΕ‹—]vYΨsΟ=Γ†.ρ³•Νr§’b₯8   5ƒ›oΎ9¬Z΅ͺβγώ&«ΰRκ Δ}ύλ_wΕ ίήyηΓ‚ όόTSY=σΜ3―Ώbδ’±jlΟl?ωΟݐnI42κRŒ„„ΪΑ< Γn S.ΌπB7―%>&ώΝΨrξΉηΊ²μ1ΡY²dIψΓώΎφ΅―Ή2θίψFxυ«_]•!γ5ϋοΏ˜={ΆοZοΉ FφGy€ηΝχ$ $ $ ŒΰW†μg? GqDΝΙ‘βα,£€:ηœsΌκ°£ΘŽu(ŸϋάηŽ;ξ8²%'ƒ3.όϊΧΏφΉsηN―vJω$ $ $ bώΘBΌ }πk_ϋΪΒ(5χ,ž~Ο=χ„/~ρ‹ΞƒΡ§°0V ¬Y³ΖΜχή{o_sHͺΑ^Š›000Υf³ζ/~ρ 7*œκ2~_όš όŸψΔ'Βε—_ξQφΪk―ΠΧΧΞ?|ώΠ‡>dλœφ ώJS˜Wασ£>.Ήδ’ΐhgggΝΛΰsζΜΙ;­\Ή²ζζK…ψm”gθ Ο@ŒυΜYΡ9B;γ­ok|₯zLΔΓq’rε•W†N8‘ͺ―gL§&σ‰€zκ8LnΏύφπϊΧοηVJx"xΛ[ήβƒ VLXΦ£ΔρΊΊΊΒi§V±gŽΎήυw…<°κ‰ΘD΄OΚ3a a a R ΐΟQ€ PΑ+χZΚŸΎθ’‹άώqΗ} Όόε/ ³fΝςέ±XΦγQ‡‰Δ~πƒ€g6ρώ’uΙΩ;vttψλΓ;Μw“ρ   υ‚ψ<›UXδγι₯–dt•―.μz}Α ^ΰžsP‘a2ŽAγ§?ύιpΘ!‡Έqδ _ψΒςό;j hL䍁€ΎEK·   5‰bQ±0Ykό„ΑO{z{Βyηη†4(ݏ>ϊhη·”qη›ίό¦Ώ›7o^ψ—Ύ-ZTž‡ηdpιPέ‘νQΠ&ώ]“dš •00PΘΰ3fΜp]2‹’Φ2H7‚ρΟwΎσ_|ΨrΛ-ƒτ!Υ”]όϋ½ο}―ιΈ¨jςHq¦ βίxάΐ˜fώΌωSU”’ίΏΖ Ζμ°ζxΤ» Ϋ>k[Χίίzλ­αμ³Ο_ωΚW|σcϋ˜Ζ(mьsΘμΐ‰'žθށ“ žCLΊ$ $ Τΰsν3ΪΓUΏΎΚεYΙ₯•Ύ¬1M³+$4ˆ00°σ‰…R,6;ξΈa^{φΨcF©uηϋήtΠ›*mΗ$€ΪΖ«κ)rΒ@Β@Βΐb€YE:Ÿ©*kρp”υΗsŒ§ε“ƒ:(Ÿ ²„αϊρ»ίύnΈφΪkݘ¦šρE†9 κ]sΊς;tͺα}YWροϋοΏ?œrΚ)ώ)δoŒuž-μ(βαί(ƒΎϊj_€­΄Š‡Q ^ψK000PO¨ΥsηΕΓoΊρ¦€ΧΓ—έwί=^ŒαλlZB?τΠCΓ«^υͺόϋ’79Ο4βίΘίΘαό%HHH¨7 Θ°[ΕMHH˜l dŒW·Lkq^^«γFλΣ ¬βHU^KyαΥoΌqΐ»Ηφα.ΏRΚ„ρ Hό»RΜ₯x  ΅€Ιΰ’AΕΓγωE-”“2P6Κ…Gφ£Ž:Κ=ς>ωΟόγσξ£)«τ1‰{)MΒ@ΒΐTa@ό[άtΣMξ‘Χσ+Πu—ϋ†κŸxx9,₯w …XN•μZo5+OΖ4΄8Κ\ΐ?ςΘ#>Аβ‰ Ha\΅°+σ LΟΊΖyΕχδW― ΊWωΑ ηρβΚΑzώόωξVt"Ύ3^eNωΤΖ›^TΓFι·ͺO5Wκ^‹υW™6Ϋl3?"dνΪ΅yWΔςl@=ρ*vΗwx•7Ϊh£υκRH3zΦUΈχτ\/ΧΒz¨άυZ•Ώή―©]¦Kα^%’opήςŠ•+B‡λ-X=©\ΊzιO•ξD.&κ*|Α“=φX?ΖDδcώ .–ecΩ›°bmBω띾EόψžηzΥ#.k=Φ#.½ίkΥ)΅01yΧrνQ¬uΡFΉύ7ω²ΦθΨwάΡ•ψxϊ•ޘ‡ίwί}αψ‡7Γ‚ †5G±v[&d“€‡<.†ε2υλΚ<φ²ΤjΗ^³©Ο‘ΪvJm1ymVMΫΤK»¨N*―“‡ΥΚΎD9)ΫC=N;ν΄ΐ1OΧΗρ‚@ΜΟ sTg]NώSYΒς¨\ρu*Λ—£ο…Ώ„£Κ['α¬2\ OΕb'z+†•αaœ’ΑfRmςΌI_‚w ižυ¬gyΩ^Kαžχ…οSΊα%¨Ν§Βς+e=Υ§Xωλ1¬’v‘^©m&uΛ΅xηύO<ϊϊϊόxR<ϔfβJ;u9'cš2ΈW…8φΫoΏa1υ’αώρΗ7ί|³ΗΑˁ'%R|=λ=;n λ<Φz²[ tW^yeΐ₯5^‚<π@7ͺkή)}σ``Όι²y0Wί5EΡΕ„aΚ {ο½7ότ§? \pAxγί˜pβνάL£ροΒϊQηSΤ.SΧ#αώξ»ο?ϋΩΟ|‡ζl>ψ`WTŒ”nκjTί_―…ό™γ.»μ2wMόΚWΎ2μ½χή^Ρrό›j§ΆΆΖ˜©>υέʍUϊΤ&΅Υž©=¦=ΰΗπژߒ€Γ[δwήΞ;ο<Χ‘|μcΛ{Qi‹΅›dπ8?Ε―Υk±zΤjY›Ή\©j·υSΫLMΫHžForα…†σΟ??œyζ™a—]v ΏωΝoΌPεΪ¦πψw­ιΐ Λ95Ψί―&όUίv g•α,α©2<•Š±ζœ κΣwΦYgyR $ρRC9άΧ“ ΧIχεκ¦8ι:ωHν2ω8/όbΉ6ΐ©§ρ\tΡE /|α ]Ύι¦›fΣπύ‘Ež„fD"ώ”ΒŒ~όγ‡Λ/Ώ<μ΄ΣNαE/z‘GΣϋžž·ά’ 4^ςŽ£G–-[–λ7 τΓB5u§3ζφŽU_;OΘ1­αΪί]ΎϊΥ―Θ“³ΥΉξ»οΎnΌN- ϊoX R‘ž G”Ϋa1ͺ€D5ϊ*ή¬΅νot΄RXθοU£=k―0Ώτ1žΈONΞΟ<σŒ(‚@υλκꚜ€―ΕΖxSΒx%&^94FIY4a 3ΰό©O!ύΌsά·Ώύνπη?9,Y²$όϋίvEJ…mΆΩΖyEΆ+₯ώ4ζ†ˆ2FΏύνo}!ȁε·έvΫ γί$gΜg²y‘ΘΗ+ΐs½s ₯αΠ+uI2ΙΤΆ&ςΒ3+ž k›ώώWZΜΩαsJΖΨ+6§Ά΄ΝρuψžQΔΣι7k{Φ†ιmΣ‡΅ύο)±ρ^s`hβj)ΎΔΈG¦>υΤSÝwέύυᩧž ψΐΒqΗ8Š6P;!ρ>ζΡά#ƒ?όπΓWθqΞYςœγ"Ξ;§Ίpz4yq ή`U6MEš ήΘθ)„Ÿ8«t?v —ύ²€Μξ‘Α…wžΣ8;v|W“rsr΅A±΄΄ ;λεΕ°Xœ6: \sΝ5αύο8ςΘ#Γ›ίόfΟΉ³0‡•GqΕS_zπΑ4εWιΕvΉ Δ{αΛΘnεθOišνŠl…Œn’|[YλΣwΠΏΑ«’ΌYgτMxV―’xΦ+ΧˆΜXk†y*e­\ι—π]€ϋώσŸξa YœυKϊμι_>έy;²5qΕη Qτ^zVX+Γ3E=CΔ+ΌεΞx4εφ‘y^P|}ϋίΈnžή֚ΐG[LZυ‘‡nΈ‘γοΰϊŠ+°ΕπΏ»rΙΏ=Ϊδ$4ŒvήφΆ·\R«.£Ν2₯«- ΐΨΩAσλ_:Μ0Ζ?4T=―(¬<`p ?l»νΆήΗΤέb©φž™0πsΞ·,πϋϋϋάΛ|E5τΒDΟ‡~xΐΰ†φEΘ£έigŽšΈ―7βa^½zυ*―J£Α!7xϋΫίξWΕ­·:Φcy…khτ‡?όa }˜μ=cΟo²©-²Lp@xιK_ZΥ«ι2 χγ© Ω­­΅mX†'¬²Ε½G}Δϋ|‚Ι6|αλ_zΨ`ρΓβϋ14θ² ύ c`}§¦‘QΓ…c‚ΗXώ‹_ό"tww{I­˜άq€ή!cξwΏ ―ύλΓ–[nινD›΄εΌyσόžψυͺ#G\±SxΘ 8¦΅L³Ε¦~?Rε­o}«+Θ―ήκWΟε½ε–[άsbζ 3ͺyΚvmΌρ&‘Γ jX˜έkΟ=Γk^󚀞ΰFν£0¦pDE«Νσxφƒ=ζΞ™λ²œβφ›L«sϊ²`‚ρΑ€δfrΣQ!π_(CQΨcΉhΡ’œ1IVΟςΧΏώ5ΌϊΥ―φ#όh#Ά“Ÿt(βιγSρΛE4…1η~τ#3Ό½Λ·2™μΔ—yZ¬.½Ά`„žcΎαdή<›ƒX]‘Ησ`qˆάnΚδC9ďWώω8ιfT‘-ύ˜ο΅š‘αΐγ=fΊ»~“γμ`“ ½γΫmg|β-o sζ7U!R’’PΫ0‡‡/jΉΛnj›8‘δŸ—ΌδΕα ƒζ +Ž—ξ+Η€πΟ¦/~ρ‹.ccτˆ>@Ž.J‡ΧΖRΔ[[[¬™œjύ‡χβίΕΪ²Xž†,π«_ώ*\χϋλŠλ¬)― s³mΑωο|§γ@υ›¨2ΥSΎ΄- ςΏύο}‘m·έv ―}νk}―§zLvYِσ“ŸόΔηk¬= o&Ίή Θ{θν™σguχλd'xOŸq1—BF„―$(ρ]bάsΟ=α»Ά)}iχR_δf‘ϋ‘r#›wή9?Ζ’ζ—Ώό₯―m½υΦ>ΏΖ°›[m΅U^V*ύυΪy£>Ζf*dŠGyΨt}ΓΧ*‘σz{ϋΒsŸϋ\ίl›Φw&§ύ˜Λ±QŽυέφφ>2κ#ΆqCšE‹{Aΰ ›ΨzΝaΆ>“τͺγΧ6κΜΡαΜ][¦΅ λίπ„GLN{ΐƒIƒρ ίeυ‘}ȏαˆHŽ°βGΨΡ‘†5<ςˆ/LFΉUžΙΈR―kω­-ΚN m38Cm _%­υΉι-ύ¬Z{'$7§CλŒwίg;‹3a΄ύ2—K -»Η= aLΣhm2†¨λ€qύΫίώn4Εμ–›™Blhΐ•˜c©ά4`ž^έλμŒO²cΑζδ₯E½ο}Ο'ϊ(π―Ίκͺ€{ω‹/ώ…ο–=φΨc}‚@‰Ψ-‚"ˆέŠ1fE>Š’z†{ξ½Η'©‹6ΨΨΨθt―Κ΄L_`ƒ *ž7Τ‡κΉžυVvhλςΛf΄΅„Άιma0Σ~ΰή°μαGέ χΪiŒš˜Veς€b0†ΒŒ9Ω:ω/!j q&΄I 6™θL6Vχ¬ w?ΜnŠ|lo£L_Ζwίӟ€onF{ΗͺΣN;ΝέΜ³Ϋc™“N:)\aΗ€žpΒ ΞΛαΫβ[H~α _Θ]λ»μΐ` #œzμK*3u` ΫbιΆF±­‘ν _¬ήΣ 6ΐU‚ΙΕt‡RαϊλΊ7ή Ψ3όϋρο5…]KXφΨςΠaJˆ—½μeΎ˜$:άR6ΗΧ„ΫΗl1όβ‹.‹7ZZΪaβ6—dC½ΝQŸzζΙπΤΪ'!5τ¬μ μ˜ίÎόL‹γK'βY±Γ·ΨΡFΫpdβ)§œβΚ;ξ9ΒyΐΈώνŠ… ;·έv[ΝσoŒΉ~ωΛΛCK«)§Ο΄±Ιj“Σ=δλ• Σ§ „ΗžXybυp†τΣΓƒχίžόη»1π™Ο#݌ l»ΘψΔ¬™ζbΖΊΕ•΅CvoJ¦Gο»Λd=ϋ„Ι{+zΒέwwΨgŸ}ܘfLN‰GΔΐύχίΎϊκ0wΑάΠΪ1\ΉObok—ε>m†’}aύχΟΛ|©ŸŒˆή’4~²‘oΧš[t'θH™ Cΰί ηl:zίϋή—σΨ™5’!ό›δο©ΚA9Ρο\ύΫ«ΓΏξΌ+tvaŒO…μ’ςλ΄Π:­?άsχΏό(YŒφdΫόa„ˆ!Ν?οΈƒ@§ƒ—ΏόελmlH8[‡3Φ„0ΊoxΨΫk―½’ρvD κ›Μρρ"~›ιο71YΠv:λšΖξ3άa”·ϋξ»»ήVι’¬m„ρeNΡΐh]6rτόΟ„ΣO?έΨhoΨC^Ηs»<Φ(;ς‚†uԟΒkύ*AΟzι₯—„‹64ύw{ΆΨ9ΎoU ½k²y8><­οLl«ͺM‹oψΛ_ΒM7ώ5tobl­nΠΪ€7Σϊ{{Βΐͺ§|Μξ±Ήϋ 7άφIzΥqm΅†f?ϊαΒv;<+dΪΦΙvϊXΛ€m*°ΪΘ’t?υυ%:QθGΑβώߝ^ρŠWΈ1ή5ΚuTΖ4bΔ δq˜LςίΗB9ƒBŠΈTQPΤ0HΏδ%/ρ3ΊE₯β8|‹ ξ.ρ4σŸύΟπ‘ψHQΑ±»»;|ψΓΞγJωπ}, L5ώΚΥy4οœηtΝ=ӆ΍w5ΤΦuκς΄žhnkό]X½ΜtπlwCΫ¬0―{»Ξ΄,ΧY+W—?±ΝΓȊ;ΓΌ•'Kϊκ‘W7)PάvΩ²}Ÿ΅ lΨi;Λl§ ƒA΅`Êο$μ3’»ψŽε‘§}Ž+ͺΝ'ş< h¬β‹μ‚ΕΒ^°ύφΫ;οFασя~Τ•%ΩΗ‚΅-»°βτάΓΏο0Ε υΜΏΫZ§‡ysg‡Y‹Ÿ¦/|±ΡώΠΔίμ²*Ρ΄d -¦±o7ƒ€mζΆŸ αξ§ϋΓs—Μ½ζΥ€ρ5ΑΔa€ώέΥΩZ„°tύmλ₯} YΧώ·˜qΣΚ‡Χ„{|8L›Žlbƒ΅UͺΑ°δ…σΒΖ/Xh|#’uΜΠχΡ›—‡g^™_œΈ’7nΞ1e'TΜΏŸύμgϋ‚λ?ψA_tέoΏύΒ‹_όβΌΌΞYΎ/xΑ †!G<όK.υE‚zζαxT\Έ +Μ^²ShνΪ<¬^v]˜ΥΎvX}ΣΓδb=³kNxεVσΒ£+mΧξύkΓϋv˜ζ̜~Πo‹ζζ#Α$` 'ήΣΏgwt…ΕΫΜKvšZg˜‡1γγώΕCaΑ]a³]6`M¨ΰ†ΜςαΙ&‘€ ρpΪ‚VyvDQ' <Š!ε+_ /zρ‹Βζ›mξ<|‡vXΟγ―ψ7 QguVΝΛΰΤyV§tnf,y‰ ηδ.-f0½ϊ‘°φώKBΗό,Η…GrԁοŠ‡ω+K²9ψ˜πv2OU[t΅†}·1Πxz‚ wexͺw0΄ύ’0―£5¬ιΟ„«ώoex°§΅ζioΠ4%YbΈ1g֜°xΫaγ.°ή1ΌΠvk‡Β—†NΫΛs‚±aΒke†κθ:Ύτ₯/ωζ":’»Ž«€}tCόYύ#ωˆη£ ™Nq,Δ~ωΛ_‰Άš1£=̞»8tmΎ—Νν0BtΚπή§ώ>υ`M”Ux¬₯«UdΊ5h:Hύ―|λΠ―˜' χΘλEωΝω3ξŒΞž°υ·‹m=ΓΌ‘Φόš{λj υK6Jςtww»Nώ­o}ΛΧ7ŸϋάηΩfRΖ€₯έξΥ=½aνL]l@Ώ™»ŽΕ˜¦Χ‹ŒΉŸ3ƐW(Τ2.šΉl’α@cW&μjΐχ‚XλcL+„ Σ“O±0ε_/WψτΠ yi2«όiνσCf°ΧΈ-Šuη]"%˜\ ΐ—M0΅QΙι —ν6ά‰ίLnišλkτϋA;κ¬Υ&σ§ϋΌZβ Ζ4½«ϋθλΊύΓΖ’Φ™-‘cA»υ§œ¬#ΡΕğ!Λo]‚ζΒηxΤΆΧΖό›φΒΛ†ρ]t‘ΟQδ¦Xί.L―πFh\ηχχχ„L«•OŸc|Γjg‹’(ΑδcΐΔΑ¬±΅«³mΝ|₯5Γ3f9oΖ΄0Χώ-z,I““_δ¦ψ␡6 󦇢™­‘o•mΊ1γ<Υt,°Φnƒ=Φygcω§)°3ρ•,€wρpAπ …›~ŒinΌρΖpί½χΉ1JU˜v€p½―•«—ίx²Q›λ,Μ΅†ΛΨΩς™Œaφ4ΉΒd‹[ˆœa²Έ„Ž\%π†μG —ΒG­Τ·^ΛΡj²Δ7Œ,2yNΖ4mx²²φX`QZψŠVfI΄':„“ζιΘεΣl—μ “Λρ‘Z‘ξΡ·’]ΜΗΪ1υ‘±΅ τΩτyςΙ'»7šξξn?z˜΅ή‘Wγψ'€£‰ΩЊqΐ’ —„E Ν‹Dq{΄ψ…θe ά2ζΫPd2tWŽςxofΪ ;Ϊ­'Ρ•]ΥΆx+}τ:έ–Α²“!HΩ«lί sm»ΐ<5XGn<ΎΟx4'^U>—ζ~[Ψ/ εΛM6ΩΔepŒiΰγ?όΠ°MK…ιλ›ZχΙΦ 9Ξ&ιΆ–h‹‰V-tί&έυ·ϊΖ8ββ±4Αδ`}ν_œgsσ l­ŽSN8Κ}zKΖεrŒιΪM&ν†ψΙ©I}Hί ιG²:“HΤφφaCRΖuέτμ[τβ3ζ΄eυΰΣ`μΤΪaN{ΜπΙψw#BΕΖ4¨8ڈγ0Ϋ™eΈ€cχη–[n9l!Y z’―Κ„›ΘOΙpΙ%—ψΉ}4n±τ-Wά7sμ ­#Ύuχέw»»xŒi€o|γωsΥg€Όx_MάJς«΅8μžagέ=cΓξK{ΘH^6ͺρΘYΪμ.”Β Pp˜ …AΓ¨σ·€Φη‡z|ρ‘£/7„e'’΅†ΫTžΡa`ΊMϊΩ9±¦o τΪ€·ώ¦EξΘ*Κ9;>Έ³Λ§Χ2Αe%|€’όS€qΗγnΦQ± ₯\αΨΐ± œϋŠ1 ΗΌ Θ‘(*ΖŠ…{‘')CΞ‰u«{σH“αάϋΛΨ=;A΄«‰Η$)}  ¨DyŒ²ΐx#Ζ5°ώ &π&Ε°;ΝVTψaN@Dί睋)Ÿ4nΝΐby6.ypTGA9И *ƞ&qΗΫc^™ΰίπεBώΜΙ\xςΙ'ν¬λή¦qωάΦ†!Έ 3]i²0Fα­†#θ8ΑT`ΐ”o9=`ScΫΆ(kόΔύφgWv! OEI›ε›μŽj5CHρτ!SΞΉ1gtkcΟβί-fά 6lMT=‘—ρζ‹Ϋhζ`x{„φEΒ3:›έvί-όώΊίŽ§°l’ΑΉΦPΗ6Σ5υo†O;ΙuV°³:dX¬5]FNp>n†U%ds¬{Œ?|nhm…’Ύ9Ξψ†δnΎ†<«θ·WΔi·±7νZv(–#4ΟάgȐο28ς΅΅β4=UžΕΨ›τ1Ε0X]˜x/Ζ4θʁƒ>Έh&ΟyΞsΒ_μhάψ_|q8ΰ€¬}ŠλQK…Ν|©+2cr^Ϊ‘·Ϋl³Ύζ(΅\—F)›·‰ΙzΘί¦:u9ΫUͺVAΘyΌίnϊŒ¦g W₯Ν ŒKψ5ύŒΠW·΅·Y·² 9έw.kΦM άΨtN‡’՝g…s“Ω-=|ΊQη­£š£Ψζl=ώv± /ίm77¬Ωn»νΒ¦›n: Y0π‰δΔΔ―Ίκ*7€ΑM0Η|ΐΒ₯β#-ήQPˁ<ς|ςŸ ]|‘qζαΰƒφ—•ω…’r™5ψ;α›]f³Βΐγe' ω—Ώ©τΠ’Œφiοό=YŽ2Σ2ƒ=Ύ1{φ¬κΚ–bΧŒFfvb(5Ν `lyΓΙ¦jŠ±ˆ\ ζl`Gši@―d4G)Ε»1yέλ^ήώφ·‡SO=Υ=«16ΡfŠΓβ+η‹2θλέzΨͺšhΦΛ‘f0šAΜ,Ό”:ΩΔ’}FGZŒΒVb,₯]†26‘ˍk6‡ph±π"Α)ΔrρOΣχΑρΠ`_6‚!|{;ΐψ ‘_πœ•I,…3ωΓΈ¦έ\ˆ³  ψ{ΏK?ε0 ώ{ύυΧ‡Γ?,ΌδΕ/ρ#=˜gςo %yδώΕZδ@2©?4؏πΣޞυͺθΔfuD14}z‡ύjŠΧ`Xššκ`Μ„ hjΉυ ΐ"l[{Y%ΠzRΐΈa@Ό–ω9mΒB«ƒρρalΫτŒAΝtΫAΥjm”`μΐ0ύ‚ ~ήύξ£Βg?ϋΩπρζ>o²ΩEx9²Ζ’{Η:Ο^°ΐΞX4PœυJ‘k¬zαοΘνΆΑ¨ΧpαŒAΔΥωψƒ> °ΏBΦAKbθ‹e‚ρΗ€·“-& Ω¦XΝΐΐνŠ\ΗξxΖήΔΗ5ώΓΡ1π ϋo{Θ6“Ν_wσΗ‘=υΒT—Z½’WΒηΏ`GSšœi„/zηžφΐόχΏύΐqͺGy€oΕθ½Ύ λeΗ;6ŒW„εΌ˜š@a&Ν™Α#²[‚ᐾBγ5t‘ϊίp{Šqί‹Ϋ¬aΰ%««΅•uλ›‹…iyŸπ'Œ¬»ͺ?Z΅*όΧύ—{ΓΐGαΌ\1t)Θί:†Uqβk£ΰ―Ύ3Ϊ;L–ΛΙεYI"[UπΪΞ‘C­­I—·Dήc0Νι&lTGc(ΞΗΉgl^LΞλπSPx<νO1Σ<­#o3α±y§xoΉώονEμΖΫΝ‘wΙX{κHΠ‰(οTζY1wβPP}τΡa‹-Άύλ_ßoΈΑN;ν4wφͺW½Κ½Υ<ϋΩΟ‹/Κ έTf (/εOΜψe1ΏγŽ;†/~ρ‹a›mΆUΕΚ₯οp΄Σ‰'žθ†4Έ“Μg>γDΩήrSΤ%X‡:LΧs7?π€u s‹ΩVλ*Ά€άΏΖ £πX” 110kΦl;κ©-¬bχ’ρ# £­νΦφ†ωσζ»Ra΄ω€tƒΎ­νH/3Ž0=όπΓݘή θzΣM7…Λ.»Μa9γPzhΠ”Ž(>|Η«ΧcšΎΠiΖ4jE\M ]ΆΫβί ΉύΚ1*vα Ey»Δ³‹z¨P•‘ώ@ΏXΝ‘g6akmΕHAC‘Μ&쀝ёνgu„†š)*όϋ™§Ÿρέο|η;έh_|[Χ[nΉ%\sΝ5^f”Θσ2Έ©™ŠLPAXdΒφιΙΩ―ροιvά†’ ¦2Α΄£ŠRδ؁νΈt₯‚hwjJΨ\_Χ,’ £a r—μƒZ Y Əή΄%j„›Ρέ2Οb§λ’EYό§?ύ©{,ΰhmι\o1ΈωΓώΰ$χΩgŸ°Ωf›ξƒ5š ™ ―·Οχ@“Νp8§€#‹³s–…–ΔΗnjφŠ2@ _4οΒ₯Ο4쀝i}MόEΧRiRxy μ²Λ.ΏBοωυ―Ξ9ηΧ½Ώχ½οuοŠ[/Έg£ς²{sώ¬d―ΜΌ³žΑ0„N†ŒΓ±“Υ£1Ž#?+Ζ»ziϋΒΊLτ³πΒ•±Cύ(#Ε1οx2ڊ=Δ)¦‹D6C{Β«ή₯λΊυVŒΧeΏ~νk_λzpαάqLίού½ΰ/p’zί¨WΙε+ϊ‘ΛΝ3$zUχΈβ‘£Χ7–&Κ‰§υ_d<δjN‘Πή—Β―ήgςwG«’fμΟLp ΰ©3;Β ­q"MΓbΔϋ†ΏόΗ`ΰŽ.kͺΆρΌρo _ύκWΓ·ΏύmΏΚν# šσyΜ1Η„ ~pAΈύφΫ‘…?ΌΈ <Œ`EΉlΩ2ϊοʐ†ό+ύ+U 8Xi’ϊρvΪi§pΒ '„½χήΫ… ”υ@ό­Rω5S8BαB;/7c‹Q™ڟφΞυКC»Yϊl‚ΎΚΞmŸλg·Χ\SΖ„ϊ'0Χ Ό€Wτ–€+Π΄°Ϊ7 ψ‹-ΜH ¨<Ÿs’0 ;ZWσ?Σ?ΓΈΕΒ+žhάΙ_}υΥξρ€ηO}κSωqDι oT@)ΟΈžχΒaγσM Pό³λ2ΑΤ`€1”E(Ό0q`Ž‡Ÿq”#λLP¬vΨ±AςG+ΉŽ(lœ`R’gGΧfxόΔ'>α‰QΪ³θŠΧΕx9묳GΎr -όh–±˜E& hŒ`m.ξqc ΒL |aΖ>π_c°π^;ϋ ޞ”ηŽšIωA1„1 ­΄AQžξνcΖhfόΘK% ΌΔΌ££A'tήyη…;οΊΣvš²Τt(O=υTψΙO~β^#ω“oΌ±ΈQx8΄4sf§Ρ–Ν;ΠLŽπ:ΖΉσΘ ΖπδN“ύϊO°σ΅ΔΗyΕΞΩ™xoI0ρθμμ4žάzs‘E˜8Όεη¬Ξ4ΎN|“¬χ )ιάΦ‹P£-6ΑfθΗ€ϊq{Ehΐ+MΞ“/oUΧ‚˜Mωί„j}‡ϋF»'ͺAΑZGJψ*ŽiΡVV΅.N–ύΫ±‹ξ­―%ΟοΧαˆ;x1ύ#υ₯K—†ώπ‡αάsΟ 8 €φΓρΚΞfΣΣN?ΝtΠAMqL6G―Ξ4ΉbЌ§Χ—Λ‘%Φ:\p8MMδ΄ŠNk΅­­Ήώ$+Rδ?IΏΟΛίiΣo/γu£u/74›e^›ϊ0¦ΙιΑ Ϊ’’o’#χvσrΣ¨ύhΤ+2X9ΎψΕ/;οΌs€ιwάqαό£+Ή/Ήδ’π³ŸύΜp·Ξ{ΌΉΌθE/ΚΏ€ΐ=–ΑO Ιξ-Ί=[…ι;cΉήqΗα›ίόfxξsŸλg ώωΟφσcmwaόꀂΫm· x 3κ7–Ί₯άS™Ό,Y²Δ’‘Ύ•E_£`œ―BfπΣoεάz›-ς;Ύj΄ΐ©Xcΐΐ|s]ˆςλι[t²|Šθb*Ν΄«ϊ2aωΪΎ°ΡF6μ Q1Bj0"ΌΜ±sο{ίϋ|Β€;βk―½&vΨαξΚςΎϋξsώNρ?ώρ‡w½λ]MΕ»QP"4υ›ƒ)ϊ˜@t.vεR60ύN6p?Š1† QπΣέ;ίjTt²q\κ{Θ/³LθΣftiψ³Fi’±‰Δ€M gšW4ϊY‚Κ1 ώ :Βάc81ΝW\αsΞχ&μk_ϋšgzμ±Η:Ÿ©όkυSσ‹vΫνΡΦf;>μh ώX°aJy/hΖ9ΘT·&Έgύ%:‘Β‹„m ή ;μ=J£“ƒpΝQ{Ω]VH%ΐ^gzΧτ$Σ—@Q5Αβαm΄‘ΛΧπk6cύε/ x2fσ_ϊWψΑ~ΰΩώχwΨoΏύ\o"ώVΝχj5.›8:Mθ_f<Ϊ,lG1ϋΔλ†ξε2&& ϋΝ6ύζcO ³Λ°V2ΙΫyΜ1ΕmSlθι³rσ§ρ/IΚ±π ŒdVφ=cΚy3l˜ŸΉ΄R(gΟNGoβo’Ÿ]Ό;ΚDpσ§ίΟ0έΰΗ}3¦1―Ό›1£ΣδΆΪΥh#JͺΚ ό1ΏeΜΠQ– 4θ/ΰHrNš§Η²{›ύ Ψ [Θ^ 7<ΔA{i>U’+Nο8ϋμ³Γλ^χΊpόρΗϋš-λΈ¬erΘ―~υ+γoναόσΏολ»δ¦΄ΕsΠv7¦™9"ΔO0v ˆΎΩPυ½ο}/όβΏ—^ziψόη?ο™Ο›7ύ#1’Ωm·έ˜Zκ²ώΎ£ΏΌΎ:άΒ[π:-cπ‘`ό1O¦0οa}lΝρ‰BvΑ+ΌbtM{$˜x Π6sζΞ Λ}ζ¨ζ=l&G)ζFΧ\ τšβή jΠ §Εό‰oΎ Οx‰¬W˜f;VΠ…p\κpcθ E‘iό0¦1½-6?΄ƒΫšό‡Ύ dŒβ/ΣTF±1MβWΓq&Ή±ΥΦ‘§>ΌFEQΈ'5Π^Β_„œ‚[ΝKΡ‘άpΓ §^xaΈόςΛ=&ΞπΚΎώϋ»“ι]ΤΩ5Μ#rωlŒάϋΦδψ~N0βΚΨ‘ͺƒ&o£KI2ήδ69†­Ά™΄Ητί9΅wΎ΄}ω{c3φ€7$  ™5{–­—>i"‘uˆΒIPŸ$IcΣ©tΝ™έ°m5κ`Μ`a\σςχΚWΎwψίvΫmΞ΄±zδΈ$¬!7½ιMaΟ=χtΕ G3!ĊΩ+―rW}Ÿ…ΟΝHΐ3y±β2•Λ‡‰±ς+ΧhΤCuΉΈL*ΨΑΞX \άbίj€°Ε‹.ζΪ’§Γt\Π·`ΩΉŽVj§V¦ώUaεςΗ TK‹΅ST’‘0€’lσ–υδ²{쨧‘°¨³₯œ^½hvP0ϋ-ΦΪρδځнd‘+xŠFN5ψ0c†—GuTxύλ_N;ν4W!$Ο›;/,ή`±—΅šq¨&*7ΖBpώwWל°β1™²Š€~;ςΣΖ0x!ΠΜγΨΡ;κδ­mYcšΥXύν"”φϋΒμ47²uΖ)αˆπSΖcƒ{ΚQƒ5ηΔφ­ξ ³7š•ŒΣF‰HρodΚ#Μπρ5―yMψΒΎΰόΫXμψFΌ`ΝΔΏ₯ΜηHšYΘΪx-0cΓ  Δ·$¦δΗηφεώœ&Hz›"†ΗΧΪ1Šϋfν–`r0gtΜ+Χ./kΛ€'1ψ~lL“ϊΡψ΄Όy»νΆ K—. o}λ[ΓΪ΅ώeή8zώόyώ‘Fβα’”φ³fwΩBΏ-8#gW ΜaμX¬ASξwΞ2ώž<ΣTΒ ΐ'ζΜι2£»ηέβΫ±ζŠ0 ΫWφφ»/)σ+@μ8DAΦ›g<βίχΈΑŒOŠΟΥHv‹gšΎ΅ύaξΌΉώΕFβ%γ€ΒqΝBΌ ο·ήz«χŒΩ½ΧN@f*'ήfπφ… m‹λɌi0΄ιθΐ3Mδiύ˜M²YcΦ@ƒjΝ€©REeΑβQΒeY4EΤV£%ΖΨ~σ 2dςb :F7}ceύϋ˜ΞΰΧΫoΏ}8ςΘ#Coo―Σ ςivΝnΆ'MŸSc…dεr3rά(lux°Χ 'YΟM}srΫύU‡©ΈΦ63γ#ϟ°νX…όmc6m˜`ό1€k–m}dΩΓΗΥl|γhmdς.[gkΤΉΈjςbfΣΧΧόq? α ΕΤςεΛσmΧώžυ¬gΉ7›·ΏύνaσΝ7Ο ωˆenΔμχ1ΕϊΑ>χΉΟΉE%–—•*)₯Œ.σ™Όηrq»υ10Η8ΆάκYαώG 3Χ„i­Έ‘/˜ωŸl’CŒEυ™Υά“Ά³wAX²αΎŒ¬&Ή0ιsŒ<:l΅΄;άyη†εfL³Έ“IρpαΌ’"pΎ2ƒόΓ+Φ†-7ίΜv²uU’,Ε™B h2"/Ε@cJ±w†iŽ₯>πΦΗ¦Hr…}_ROqƒ·š+[όλ³²-ΉΩ„f噦ιu"›E2,ς#»ε{ν?vΐŽh2Œ.±θοYeΖ4ζι&MϊFƒΙlρo΄μƒfνLPg›AδJ[Θ Ψ+ƒ€*F!“Ζq’θ}ρLƒ"ΘΥs€ςχ1;†’Îαš•f½ς“ψC™iLύΛm/«+䇫ηT?CfΐŠαMZ4VΖη*Ξ8ΈΕ[¬—©ϊ‚Ζαυ"Τqγή2Β4σ.cήΓμp‡*jcqmΞ̐Ο4σ7XhJΙqUίUQ–ΖŽŠμΗ&=υΖ¦‹ƒ5…±‰πΔκ^χbΓQ@#mqLM(<|αΒ…‘wΙε=Eδrλ&ΘλΦv΄a‚ΙΑ42’™œ/ŽWΰϝf€8`G€’qyMŸρ“ ϊΜc/G$c‘FWd%ϊ'c8¬ωȘ&ρEaiψΌ€# _Γc₯'π„± †4.»3Ÿ’Sf=T ‡φ”Η€ϊ!ύ“MόB#Λΰq]… χΎ‘σL“±΅ΐ,U/k›gΩϋ“‹<>vΨ‘ckμ(§\—Vφ˜>c§B η’^Um:,bz50¦Gτ¬0½"z,λHέλϊΘY3Ε%9θ]έ6ή°+ίV#€¬»Χ£šΓl―ψ‰'žχίΈρΖßώτ§πέο~WΡόŠ‡—χΏύaλ­·χή{oΈκΧW…Λ.Ώ,œpΒ žζ”SNqγ1σa‰‹<θ»σηΟ‡z¨-ί>όα»™—Ύτ₯c€Κ °€,ή ,ZΌΘ@εŽςΔuŽί»'bω‹ΫΘa΄ΛV[onΗ…aξ¦ΰoΟξX¨:£ j΅#žΦ†΅Ο, K·zvX0?·«’rvQ;ΥI%)‰υG¬]α?Λ~Ό*<2†Qa;+™~Ψ ΘΖVIΧτgΒύO­{=ocΫ•έQ9,^z¨9 ˆΔΟc£ΉfεΫ(A0`G–Σ`Pc{ebŠ©₯>“„^›LLΟ±*NB’ν°π‡7¦±°=+{ύ˜'CyΥΓ„J7dΦ³’g˜1ψβ€keήŽ/ζ9³fu…§ŸμIΖ4ΓQ3eO9Ο@@N7ώΑς »©ϊμyq{›Υ—=ƒ}Κ ΨdFiŠr¨ίvHqše)2ΉΕQςμT Cc ‡‡Ηό›ά΄0"ώ>Ά/Τnj<ž ogϊ1xdβΙ΄R0Ό™lήoξθ-κΆέ΄xT)ζ*‰'ΪsΩΟ 1ž^ΫcΌΊxJT Ϊ¬Θy¦i‘M-PyO•BG‹α•ΆY°pΙε=Y#χΈϋΨ=G²φ­1CšEsύ˜bΎΟρGϋύ”< }ΡΏΪͺ|ŠΪ{ _fan omΞ3Mq™Π0dΦgv₯#?Š΅† τOΡs’dR SλΒθ+ɘf>ΚέAOτHy’α γπ8έδϋ•a|ΡOΖG Ydp―lτ.0pœ˜‘:Ίo'40c^~ΫΪΪ³FπQšt;ρ˜mΤِ΄Ϊ„pS™ φΞQ*ίΆ Šx;±?ΖlƝγ‡ϊΐϊ@Π™ΰ‘N‘}γ―+ϊ!>Ζ4θΐηn77ΣΔX’ λλν wήugΈε–[Βοϋπ£ύ(¬\iξ²rΐ9|oxΓόψ%Žs☠gZΖΘίς–·ψy}§Ÿ~ΊŸ‘ΝyΪύθGΛΐ(ίψΚΉΏύνoΓf›mξ5#άΏγ>ΐυ»Τ‘ώχΏ»‡œ7Ύρq–Γξ)o\ηa/ΣΓzPϋ’Έά|σ-l†zŸ­Yάj»βzُ€ρ΄ζιΓ;½άά-όSŽ5!σŠΤbV0x%ι0ΑόρΥ‘w`z˜ήββSΕedœΑBώ‰΅ƒa•Ή‚&?ο@β£qJ#ͺ4˜ΒLρΗDη›α ’¬ΟΨξWhΌΛ  šiη«χΪ‘{?Š ^ψ;h«δυ”Œi 13~ΟΝΚΏΡ9a3ΤΏ:΄V)TψXgΚώώž•ΆQl‰ρdL3~™Ν ُΕMΌlάτ?VUόCίβ>ΎΪŒάηηh˜Ϊω™Ν+[ežiqς”]`±λ4Σγ οξ]ΩoΈ8?GJ2βΔ· ό\<}βΏ6ώ_М…£ή΅ΟQ™₯έθ*6Ψίcύ}‘σπ|xΊΙcΐ ŒΨ€Œό›ζ»yΤ”ΌaΌύΥs*YΑqz± ψι·q ξ–ycΣι'¨’΅4>fuέnΤfΊ“¬ΗHcώFgCfτ›3wŽ#Vύ΄r,§˜£Εcq§ιIVšAάί5,―1ΓihΈ+g5Ϊο€t₯1@ŸΐΓ£u…0`λΓ’t²αo¬ΑΨPΊϊ©5Ύi»QΧJGeΞ΅zυκπΰƒ†ΫoΏ=\wέuαϊλ―7έtΣ0ώΏχΒn»ν8›o³Ν6sλ1ECZ²dIxΫΫήζ-όη?ύŠw nGiΚ]οΎϋξπέ/8ŠΘ#θv½+gƒvc‚‰Αΐ–[.5·¬KBߊeaζάgΫΉ<1kœ˜οV–k–-}Τθ­?l΅ΥΦ>p’V‚Feω€Xυ€ (4wΨjiΈοι§M!Φζw˜ΖHj¨ˆΠJ;Πi[l±y₯ΙRΌ„šΑ€xFτ‰~ί‘eΖ4ζ‘Ε€»€―™6OAβvΑΝZ3¦™νCζ΄μΒ¬M $6V&―¦’=1&kοl7«όΣ­ζf•ŠΡτ™Ρf›Ιΐ±— &3Lα3Λ” ζ΅ 3°Ϊ?‘Œi&Σ•ηΙBΒάΩ³Μ`ی3LE7έΌΒFzlvŽ)‰δšΈςSΜQaΐΖKw݌›x/pο3%2γ˜ψ=‹δ‹KDMΑ a@2ήzαΥƒfLcbXΪ*f.Oτ†ž5ΛΓ’%ζωG’ъš ’Hj'ΖN΄‚=Ά£r™oX3Ms>ΎΚ 7ζΚ™Œi*ΒμΨ#©mζΝ›Zμhβώ5ζ)DŠ›œXoΗ%ύ†‹7Is€±£Όirmq΄›?3Cζ½nΨ\ΟΐΠ ΅&tΪNψdΘXœ4Π%M³ ‹[$Úβ1S¨0ΐψ-ϊcρ2Aq `ΤtΜ±`ϋΧpΜο$ Tƒυ;tsm­ζU‹Hί4} ›^ζ₯##«AλΈΔň£Σζλ+W―ρΝλžiNΞ£‰V›YψeZ§t—Μ„΅‘›υ―Θ2άas‘’ΙΦ½°ΆbCι@―ι½Μ( ‘‘βZ‰ι`€rΦYg…³Ο>;,[Ά,O―xΕ+ΒώϋοvΩe—Πέέν tŠ Α”ΑPΒοΨa½νΆΫz΄εΛ—Έ(]%WŽtϊ―“OΆ]g&›W) μ\±bEθή’»$)^•Ψ|σΝ͘ͺ;άϋΐΓaFΚΠaΗ(ω`Um¬ςΓFΟ φ‡U‡φιναΩΟήΞS‰Φ+Μ"E«3 ,\΄(lΉ΄;όΞ³7e“S-ώ'ΊžBAθœαžiLi {2ΖcN lњ“Π$3ft„Ε›.²ζβ”τ­3LΡeνP Πwϊμ¬ΨιΆ‹œII‚„‰ΐ†,φυbLƒgšf›HόΫΡ0ι?πο.kƒήώg|‘Ν8Ζ4k—t™1M‡y5H0yΐ˜Ο4ŠXiš/ΕgοM =ͺ³αΫ‹––Z½―Σ³/žΫcΧρ‚±Αf1K|6ΓO 'η–Θ! !„%XςC0±°cΐΫxΖcό={Oο«€Φήσ”Tu·Τ-uKκnι֜½k½υ>Uο­[·žΊυ£;ctΈt­I(&τxβ„^ΐ{3ν Tκ˜ςΠ+ŒΩΊΉκ<Θ4]Ɛ¬²= _±ηΔ܎ΆfΨ`†ΞmⰚXeτLλ‡ αb%­ύω₯>†Oμٝ Α YM½œίΗ¬$IΖg€cg»’œrΐ¨‡s#ΰ€Ψλλ0 ŠŒwΡΆΦ}žiΚγρμ=?χ+λ uήZ,Δ²žiψvΪG-^Η™σaάΦ”$™$JΜρeͺK1μsR]'dΗM.€!Σΐ³=Ρ$•q ό˜!½{`ƒ%±CSy ‘†^gFΖΝ"RJF«σ7έΌsΦkPyKW=Okmk5s\Zp’ϊ„ρSΨWƒ«’Ώ£ΌΙ4Δh4*χάσΣ4‘ζΐrϋν· C:uvv^`‘Ε[‡lσ•»Oς‹ίο7Ω“Σήήn•χο­·ή*·άr˜λ©ΐεJφΩφΌeJΝ?nΟλoαX,ihΨΕ•ςδ“__^‚ά…ΧoαOΟ紏ĹŽLJΘί/;/Ϊk<"εs§^³>°m’«S·nΩ"?ώΕ½2Άε¦¦Ί΄Ms©7£TΑXQF°*j`:,οΫ£“€K¦ηΧ,μŸi(ζ`!ŠΥ²DΕ άσͺ'΅Pi\ΕlεΛC½†d²ΧT,ήμ':ΊΪΝJWιλαΑl1½2kiΠaD@ΖαΔyKk’L£ΖΕ¬HιΑe `Ϋ*WΘOΟ4r ΐxͺYF–zK‘ΰΈ’„¦ΠΨ˜Τ ΄K=*%’Ζ <ΣxA¦QοbE:Οlψ­ C}'ο3΄Μρ;1yΔ1IžλIΙ4 θfQ`›jnn‘1?H x€¬cˆ tC\δh©πž›@HjocGz5¦κ‹@ΆœSIΒ“KΊZ›αb>ŽΙ»YqrΠΟ”ͺ*’Ϋ§Γ1C–l„;zM₯Gΐš< „qꐙɐ!Ο€υrΘχ8Θ5Σ#~ikoK“i¬ŽTϊκΦ-©Ο›z΄― !―N―ή„zJzR§•„δšHh z<–¦Ό`hۚ[γ™s)βi.6σχ2ϋξ‚ν σ3«‚}c C?LO40ΔݝΈ‘\ΓΆF/ʁε ΐ…n”ύq’iΨΎΠΞfΡ°h.Α…ušΚ‹@-ω’^ΞHHφWπ›lb°(‰ύMsS’θ”)Oνuϊ»|,žνmνΖ$…ΪOQgη=΄7x€₯ΥΘζ½ό­½; &ΣPydΌζ;ξΈC^ϊ—Κ₯—^*τ>Β ›Hαuζ/f.πxΝ«_ύjyΩΛ^fˆ8Φeη›ψAY%.ί{τΊ#ΐ:g½μΫ·†#°‹α›†‡‡%Βκ…s•Ζζ„xΙ4’eήc2ЊŽΫΚΕ—\)'Ο>–η嘹νΔΨ„.£ψ₯­Fb[kXt–ΡιSpΦ.{φμ1ΚΆ6Ί%Τg–{:;δόΤ \9';jΘ–j‘”l ρ4ˆY³sΗ#γ(c¬ά+mΙ5wE xΨώ’F€ζΦNL2MΒ@€A©IΩψΕƒyΩ9Y2ν—(g8 «†ͺeCšχΤΉΒυπ“‡Ν Ψ₯ϊ†ΜŒ©a@΅5+eΙθo†q_Ιi™ιv± \ 1›2‚mW ώΕFΈ°όˆΏdšΣ‘¨8U¨ς€:f03dzT^Ό¨κDtΙg²4υl^;3GU/ΖE(ΡΪΜ’` Aχ†$0ΝΜΜΘΑƒehhHnΉε3YIjωε/)ό»α†δŠ+0αPμ€P&<Ζd‡/G©ΘΜσ™gž‘{ο½W{μQ9rδ)Ž*˜0[Ϊε’‹.’ύϋχ›ςr_Sι°uفΥέ—Β;ΝαCJΣΜ ΤΉωA%‡ΐ₯{ϊ9Σ8…OΣΓOΓsΞ²uλΦ%nΠΣ•„cd^qΩ>ωοώ… }β5‹½#;rNˆψ#³rlΨ/W_²Ζ΄.s‹mλ‹έ―灡ŠͺMM-ƒ‘ζŽ6LΜͺχ“Υ/K¦±εP2E’tΏV–sA2 Ι0±LΎψHζρΗ/6"={z”άP ~zyaX2 1pΒI w…αWμ«9qΞUτAi<΅τL@xO‹Ϋœ+φ35ΏμX―Ρg ς1i’ }J°£zΜό½Φ0Œ’+Π5)Ε@ΐκ΄Ÿ‘L ŒΑh2„Ϋ¬€l™&—ΦMΫUvdΓ§ˆΗΨ—6£}ζt?Θˆ€€ ΰΆ1ζƒΉ €U.c²ulvτΏ’#`ρuΊœf’% Ο@τD3; ΟΒ”έψ"~i0ΑΒI&+χ‹^Ν°"p9A¦Α7=<1ς"'νΡ°ΜΗ?k&W8oΏχŠ`…/eΗ$=θβΒόΐΜ”Q΄οhʎΙ2uhW$Ο°'ζgΙ_ζΙ•ςL“ύn=ͺδF€d™Ξ.9qͺߐ&MλBˆ§phΪ8`° “¬’;'=S,Ψ΄΅΅Β;dTBTΒS‰u0ƒ}.JκlοR»ͺ¦DΏœƒθνλ•ώƒηŒάΥδHΛίΕik,DxLΨΐ7τl¬θqkΑdΫρχχχΛΧΎφ5yϋίoΘ)ίώχΣ+o­ΐ9tθάuΧ]˜€k’·Ύυ­ςξwΏ[vΐ“ƒΝΓV―ηΏε$›•ϋξ»WώόΟο’|pAVgϞ•Γ‡ΛέwίmΞ½π…/”|δ#rΝ5Χθ xZΕ;ΐϊ‘‚MΣπ}πWNJCσEθ«Vs’6©†Εƒύ™™ύ)’ίΪΆέβ! 9­%XΏl“œδΈ ―/ύϋ€LΣ-;α+/ „ϋ'ΐΞ|vdZ^qν&4ίOΫΝZͺe-K‘ΠΘοΕDΣδτlHq¬Άά,υΚΖ/Ζ’_OγϋΟL=‡$k΄*ϊ5Γ4”ι&^,β'‡a€Οk‘Ύ;iξα@"4³ΏGψiRŠ€• œ€Ρ'‹ύΖ€Šύ<Νoq¬.Hό91˜9ά ˆx9@Ψ·H\ΊPW )γ―½~ρ\υl1hτ5ƒΌ!ΣXk36Ϋ ;ΕβX“¦Φ&qb΅’&E XPNΣ@άΣΣ-ώ©σ†L“Ω=oD@¦™0nΝ9~eRΩ±8j…ž΅x²/₯μFŒΜΠ,4ζs…<ΓΖι€q‘(―μzڊI–!ξt=Ÿ”έΘ“q'2 Γ³jR Eΐν&™ΖΰΪ“%ά2LΰΓ“9=XΩ[hήΥp=ν”‘ό6U.ζW㜻"fΤψ§);$Λ°}‘<Γ”œΕ™.Ιxœv1MŠΐr mŽ £# ϐδnW",3ώC¦aϋ²v–εδ―χ†€Εšzήμ'ᙆ‘vκߐlϊiFν\…a[θΥ 1ΊΜ½ΏΈΧx¦AW…ΔšΈ0Κ•'―ΐ[€YP x%ΧUAdΫΐ Iζ+_ωŠω±Η“H$bπ΄Χpu$+‰ώώο^Ž;&π•Pc.\Ζvπ{/ΌΡάzλ­Ζσ ³Ήωζ› ƒaψ|†{”xΐxΣ9zτ¨<οyΟ“_όβrΰΐ”ΛΐΎ[vμΨ){χ^*GŽΧΜ°Τz7€+K… /σ½Υ¬V«αΉIύΡaάΗΧ–iΞoρD +g§αŌƒΦ%“ΥY—ΞA―PςG€+f)+’QxBImη·^Ylh€σ’‘ˆρn`C„Ζ Cςx-Ξk*/τLS‡o$―μSη$Θk† ‰–―΅Ή’ ?sή[wΚ‚€Υα|‘κv7!jSPjαm&i”œΧ”(9™›@xμΦVz,Θt· 7=°8Δ—“ζaΔx ӝΝF©[XS3±„ρNΣhΒΖ-ž—ž->τΙ:βŠW#ΗQ9q„λOG€± αYρhR E€ς^δ"aLͺ2Mκ«ΗG‡§—ΛmHvΜΧΚσBŸQΙΧ[’iς«eΆ!Ξ•q¬@Ϋ·5eG€žiΨHž±‰}q Έ)™Ζ"’Ώ… `mq†LΟ4!x~„"AαnΘ“³ΠΟ[ZZL–*ο AveΧ}θΫ5¦―υ`QR!bXˆD {dzͺ™†]₯€ιJ&h¬ Ε•έm žό66lθ•±Σ&σ™Ο˜ŠeΉΆͺqm~φzύ]9Ά=΄ΆΆΘMΟ}<π˟ISοY©υt#σ|?Θ•—cn Σ„Ζdrθ¨άtΣΝ¦Μ=―{Υ€'8/ή}‘<ωΔγβίζ•FΗY‘νύΩZ#0¬Eό? m»w'Γέ©άΘ†–[O Aγ1σόc˜5Ψ―~ ZβŒ•14Έ¨gšΧ‹Ε» «UNi σ™ςΧXΈ’«¦ΰ™&0>#έέέΈ7ΉϊΜκD₯ }B5!@Ή@ΓΓο*™fukήΚΦ I34ϊ’\ΗU4ώ2„ΚςΧ‘! ϋΌXyq:%Πmύ$0IN2MG«z+νTψSm­‘Α-ν}hƒΣRO2M-IΆl‹%χfUfkk«ΚŽΕ *Β9Κf끂†ϋω΅‚ ’ ½ΧΤ`q–¦ς `ϋΥφvN’Τ§=FΦ`Ήr4wςέ›ϊ:Μ‚ƒς”HŸR X]ŒGό¦Iœ™MD9bΓλ™‘τ…,žsΑ6X ―\’w°dβi·Kς  Ι”φzU2ΝšφLƒ~ΧφΗ L°G±ο±‰±―&E P¨SPVuvvHp#‘“SφΗ£AimλSG…Z„λΩγ2‘ΌΪΧή/4qCj―£ »2 KRL~χΆ7θEE ₯₯U|nΨΑ£ψ<@tJeσyHή#ύ#ι!‡ύœΥασΉw=]“χ(0Λόc³ύ—ΎT>πΘσŸό4[;³As0Ί‘TήωΞwʟώ韚{>ϋΩΟB‹ΩYαφYχέwŸ°ΓΙ?’jήώφ·ΛΆmیa™BώΡ°yΙ%—˜pSϊΠ‡ŒW‰/~ρ‹ς裏°$zϋbΨΆ³wο^ΩΌu·„'ŽΛld c6?«-–C1Οq’ {§OI,«ΊΪLWκ^Lδ*%/+7H ΫB ]9ΐCΎ{>Β ©qr$˜³“!Ω»kblvV $ϊUŠ€ύ8ΙDOr$ppί2e|Ϋkͺ’U}mχ2IM¬Φ•ͺΚS-t3κmτΟ4ρŒΙΧ₯žΞΑWΚ†&#½₯3ΥκBKέ―ηB ό¦¬ όΆΫ…ζ‘ΧŽ9)―ΝψYΣΛlΏΖ3φ«ΕΕ:Ÿάκκλ€ ;’πLCyN94m ίŠ"45#Μϊ^MŠ@±°bΚζ–֐i`§™€er1;HrTš@˜κ}v₯l±Κ₯ω,D€²™„TZΒ ΨΝ― ΚQLδYΒΝΒτH) ηBςD~5AΟAH ω D q=se)Λ‘yWόξωMSβΞ2άG:Ao™†“{\h€);V§ε―έΞ~₯%σΙ4»iʎν‘$ΝΔ°J‰βω‚φ`φ=Π©ΤΣgvάτθβX›œ$J‡Σ“”ϋρ‹€₯ν1WεΩβ8–β,Ηΰν-ΝβΔ o‹Τα£η˜=έ{`:2MC‘­ΓR”Aσaxμž-= e,Bš?*Ί€m*”Ο »¨“ΧƒψNβS₯ΦUήdƒώ;uκ”0DWμώŸσzγΊ‡ A.€xŽΖΔ½θEς²—½L&''εΔ‰6»eΪηyβ‰'L>o|γεε/yšε›-s–‡ΞWΏϊΥB2ΣSO=e&UXfClεΗ,τtνλdrψIΜ <γ‚sΰ―4‘I0εž”½_P`ΙOg₯7¬k(?Ψ.wοήmB6™„Q&wί`ή•ψ0Θ4GΏΙ5W]%>„Βa²νΫμθŠΐ:D€ΖGωi5f˜,2Iλπ•*’Θ4 pΕΚ+κ/`ΤΦ¬ΊUε~ ~=`Τ‡α¦’+_“VΦ₯KΑΎ‚dšΰ،lΨά›žtΡΎbiμτŠε!ΐq ε=pr['”–‡c1οβΨΧ’i(< ™Ί''jilΠT^Ί²ΖΉΘ τ}„t2–9Ί?*d“δ!Xšš|J¦)oΥTΝΣ¨S47“Π…P"&ΜΣ―N τŒP—A¦±6°₯οΦ+–ƒ'θ(L˜'d@Ξ&z¨5Ζ αΖΥίr!ΐo¨­£ žh@ŠΔ2ez¦aHUΊ“οκ†7Ι”—Υ·ΛU#•σΙ4ψΨ“dš€Qp–ψΏΤ«JSvμχf³_₯GmίΝΕQ΄·ΡΦf·μp<Λ>™α^*—rφ0ώ2 t#MŠΐrpΑλXKΫγ‘fžΙbα ττVυL³\@‹pm'­XΨβGA I’iH€ ƒs8:1eψ+O‹πHΝ" t<ΠΩέ!Ψ³­gŠΰEtςx8!3aΩ°6pΧ\^©ΊAΑ½ΟΔΔ„<φΨcrΩe—ΑϋΛΦ48Ή²ΗΙJΊψβ‹ΝυcccιϋΜΖ ώ#9ηΜ™3&‡λ―Ώή(%ό°r­άΆS\q|Ϋm·™ϋHξ™ž†aI?JCΡ#œΗ@]=έσ‘X(Ω·2€jpϊ6μ9™c¦–Ds―4A–αwLΓmx3…Ν"#'ΒώjRŠΗαp،Υ蕆 !4eG€sw$͐ƒO=Н’sJ•Š\Αdvhœh …Bζ/_`8Ψ€βΐ”‹θ’o^™ΧQ ƒζP _Ύ‰΄££Γ\Ξw!+XSι '}—]‰Ϋυ43Œo’“‚ Ε/ žA―4± ˆ<'πa7Ι•W^i£†β£½^rlCόω«Ψ/Η†§d"4 7rθζ5Gξ’dΔJΦc#~Ή|Ο.Ω°aΓzyE-§"43qΒ‰J¬¦ΥG€u‘©/Q‡β1;΄Ώ«_Κ*Ε•ίΔΖMerhJb3θˆθ/Θʟ σ…0O6ίΚBKίf- ΐΙN2Ρ8KωΝ %M«‹ q¬ Ž1h\Hz¦c^έ’UηΣω}0LN8N―°²ϊ>ͺΘx¦‘ρ‡ny­&E XΨΎί ™ΠϏΡπ4Φφ,Φ!9M 1VΛΊ\ i’{±Κ₯ωdG€r›uF eCfβ^Ί8C i*?ΤuΊΊ:ef Ά[Ξ ~¨ŸG@’δBIMŠ@‘X;0e3=€Qήέ€Š&W Τ47΅¨^½°CώΪνE.―ϊSœΏΚ$Σp[SvhγxŠαΨς³$©†ϋ<ži'˞ƒUr#@[ŸΟΧ,±ΘŒΤ ΜS$β7‹ΰ¬]<χz¦TPχ‘?φ‡Aš†ŽΗω9z¨ Fβ‰EN8fΠT:μ˜•ž”Ι—Γι@ σJ΄cΞ”6p.(mς%#yδuο:Ό¨`2Moo―\qΕςΛ_ώ2^)—βΔγ–@300 Ώώυ― D ΕT e‹]18o~ϋΟd!y†!«˜xΏΞCmεηˆ9±§!σ–[ž'Sγύ’D™S-*Oš ΙψΐSrÍϕ-[Ά”η‘ϊ”5‡€mˆ| ”‹$ΚE@˜‹`ΌΞ‘ΧΐLӈΊα\…¦ς Π‰0O E¦ι#Λ ΘÞΒoΧR.SW―τ>-οή‡@pC#ΐK^ςΫC=$υW%ΏϊΥ―dppΠ¬ˆ΄xR‘š˜˜C‡Ιg>σωτ§?mN½ε-oI“_μ΅Λω΅εαΗtΥUW™,ΎύνoΛΧΎφ5™šš2Ηk²ύ Ι—Ύτ%γ)‡λΕπLΑΔχΣT:XL ‘sέυ7ΘτΔ9‰ϋO'­˜%ψ’^ρμxh+̏Κeϋ―JΒJχΆšσzA`γ¦M²kϋV„zς##Θ4σ ΞVK6l?:†(ν;ΰ™†I允A«8ρG$ΣθκΧ΅Z­V§Y«ε«΄rωཬwSBSΒ›|Œ5f)U‘ iN‡twwH΄Ώ¨΄–‘ο£,ŽϋTzCI€©a A0QXβκβwλΩR @― ‘qx C9™[έ(–ών靂)[νΠTH9^Λ•7šVκεXΖ)1„yŠƒ₯D€΅­%½’|U ₯]χx< Ζ’f–¬9jnΣ±θŒρLcΙ4λώEKπτψΟ€dšόΐF£Μ„=·Ήπ\Sv8)K½)Ɛ—θ{I–$™†ϋ<^k 3ΩοΥ£Š@.μΌ€ΣιΒΈΟ2 ZΔκ ‹ξΉ―iu0c%|ϋ ‘M]<Ž.y&ύ΄ΞS”Ίv¬έΊ³³ ϊ|}TΔKaœΰ—•]Σ'$85$7ίό˜pSΔYλΌ’[ΫβοfλΎ^©ΆmΫ&|B†ΰ–¬ΣCM2±Σ¨««‘Ι™„τO…eχŽνΫ‘žGVΟGμ'l˜υψ•Tf«ΘfΎΩ~4σΈn—†¦μμκ”gΟ=cάΙ§;‡γ@"Ž_δKιS“"P²Ι‹r/&aR]ί°η9DD½DI¦iP2Myk¦ϊžfuνYxœΙ+q d’Ό&JSι <Π΄v 7‚–ή‰Ξ€hBMx&"͝ΝJ¦Y»UΆ.JFΫ‡ατόA’©©wΓ4 2MdμAΰRω›³I¦α„Ώκ™&'Lι$ΠLOOΆΡXT‚A„˜Ρ”4™³ι6ΜgŒ’Ψ§>U“ϊ.­]?k&zPȁ@=Hκϋ%βΡ™FΤƒl¬Κy˜γuF‰ΰ;§JΣ*ˆ5qirΉ!7•LSΊhnio£GΒώδB€:Χ‹: pA)e³΅WςΌEAdvRƒn’οΌσNΔ—k”~πƒFΉdΘ§l‰ •*z€yίϋήgB< P[žΆΦ6SžG}Tόq‘‡šo|γ& WsBƒμ߁yςΙ'…‘‘ϊϋϋεε/Ή!απ|±Κ” =Ά­[·‚Τr«όθ?Ώ#½W€L“ ³΅πΚ•δ­©—Ωи̌“-Ϋφ€ΙSZη+ΕΆ2ξ§Ύϋ’‹Œ9zŸΩΫ>·ƒ¦ιr Ž MΘ/ΈέΘΏΚxs} E€“HIb«%Π8  c›¦ΥG s…뉫oxŒΫšJ‹€5ΚPί퀋˃£ύ’„_ΤQ ώπtDΊz» ΑΌ΄%ΥάE`-"`Ι4vb!Ε©Q²κ*T‰4LΤsΫΌ†8γŒ'CΉ˜τLƒIY—Ϋe<πšcϊŸ"Ph†šZ@”ΟΚdžΥΙά|πZω5Vfg£.±ξ¨&`ΰΧ΄:Έ0™Τκ™& ½Ϋνδρ‚ τ@Ι ¦±©Q"τLƒπάα™°4"„3Ωz,yAτ…υ³¦¦f‰Ÿ„Μ₯η0Κή°ΡΩ`·f²ϊœΩΡXΫνάΦ΄8ŒΰΐΠNΝmΝβp;Δοχ›>ΕΪβΏ»ΊΞRžsb=N2 tujE΄Ρ“\c'ά« }Ϋb#Τ οζ :žΥ%ŠύΝ/( ωνΗR+‘ψCoTGΆ’NzεJ I½§·[ŸO.*­Ια]c¦„ΏΞΐ[€?"=]}ΖγΣJž½ξ-˜LΓ—bΓ¦2ΙΨγ ωD² Ω΅ηϟ7ΚΟxz…α$„˜γρR¦}ϋφ Γ<½κU―2hŽ;&CCC&δΛΣΫΫkB?]zι₯²~Σ«R\ΚY<ο½{χΚW^+<ςik»Tκ|[8n)rB[Λ4<~TΪΪ»εϊλ―3Ζl­χ"ΓΌΞ³c(½»vH©γπBγClzxΤ³’ U†š3²{ΣιLΣ€T"Φ νv%Ύηzz'λi˜²†aκRί>Λη/=žΰsγ7‚»,'ΛHBMfyœeβ}+-›-‹ώ.€ΕœaΒnΌρ9ςΏή'ρ©g₯Ξ» k‰8lKgŸqΔ.Ψρΐ™}F^όβ—ΛΆmΫ2Ξλf΅#`Ώ{Κ…=υtίCΏA¬δNιρ&;mΨΜ$›•gG²eϋ.ι…·-MŠ@%"`TRω§iυ ώΒ^ΆΟd½π˜ΊP.OέXέΠ’i" ΣδΓ §qβό€!ps@nσ)O©υ)Š€"°ΰΈ3ΣΰC9ΐ}«w…2V[θ™Ζέΰ6!0O†1bΦ Γ<9]N Rm’Μο›΄ΑξΔΖΗφ—2g/OβJΨFμ}Ω―Σ£ΕD ³πjμs&%"Ω;!Γ8 ΣΗt£ΌLΣΰiρΐ(ΒͺΒ– rmΌš• @"ƒΟ2M$(5P(±Ό ΅¦ΓJ¨ώ–αH4bΖΊ΄%Ρ늎{³γdΫΟδδ€ 8}υβΈebbΒΨwΩAΣ\8nr2 =Σpn'©6%Ι4IοsοΠ=E 0~•i61ί₯΅‰–‹^]LŒ¬L}λΜ—£!kθ₯έΚΡb>Oσš‹€Εؐi:»dκό΄$"–Ξ8χΪ {π$xήΩ{{{€4›Χ…λ*gkΩdBi$\ ’LGFGλΜe΅pe!3ŸΜηΩγφΧ–ή&Kω–±<τάΚ 7‘».‘σƒΗΔΡΎOjܝ–t9ί|Qθs(r±"ξΪHΤ™ΗδΖηάhΫͺΰŠee_Οφΐ ΟM›6ΙL$&#pOΆ΅™lΨdΗ=ψ€cΑˆ\Ώq#ΘxΙ˜Ι™²¦²ΡΡ·«κAΤ`»V2ΝΪ©q€H¦±δ&ώ€―.”Λ[G$ΣP_$™Ζψζγ©bd$³ φε,cΕ"”«Α%­­­Wθ¦" TμOωGΣ&ΚΥ-ε₯gΚζhd&)ΛνPUƒg§žk@Άaz*ύTΓΣϊ[ή‹‡HΚ«“H$’^ΔV 8­ζ;r‚[\υI‘!ΒqLΔ 2Ν4―AR9Qώš’Ν†δ™θXԐ Ω―jX†ςΧC₯<Ρ~Ü@%™&@?Ϋdš8=Σx}i2M₯Όs±ίΓφO™‹~tB:7Κ–LγjvˆkΪ)c“†„”ϋŽκ=ΓοӁo3Ζ0O„ϊΊ σδ€No‘“Άί±9 ) @Ή…ffd>Ϋ‘•]:gXˆ%ΈΤ,:Έ`B1O`5ι·^°³diΫζ?k‡"bΎ•y7π0ŽΗΓ ™ςKwOwUθMΛ&ΣάsηΞ™ΠNThΘ4ZxΉ‚šξώ8)ĐKχάs|όγ†Z²cμ΅‹ύ2ΡΡQs ? †²‚Žl^Ɲδ`*³ Ζ-nΖ8Γγ>Λ@b½€h*Ά.zπ‘έtΣsεστw΄α¬ΤΈΪP~‘EH¨ίΩΠ°L ?%ΧxŽμά±£™j•Š@B8mίΠ-g'Γ²―«WλΠ~π²ΓpW†HO²eΛfcΠ¬ΤχΧχͺnj1`b?Κ?M«‰3Τ}¨λP·α`/(™¦ΜUCύ°­³ 1`v ¬|‚]Μ ξH£ ‘ί½©Λ„B]pPͺ@ΐŽsͺβeΧΙKrεΉ ±‘Θ4ηΚS9θϊ0QvΣ3]‘ΈN^I‹ΉΞ wAΣφjσ4ΑΑΓn-VΝ†BX›•¦"@]{„ΰ€·kΗBΟ4l豆ΆO^«rΎ΄υ‘-wΚqΚκX†}κδ°χiX†lHι±BΰXΫηk”p!xθΊqx¦ρ47VΕ€P!Xe^ΛΉ’i¬νΘzΠ΅Ά •‘™h‘Y‘ߘ˜€gΨ\-ΠI'œ2~f\Β 8©δΫO=d~ 7κNœ±%†τLcη³έ§Η|˜/£¬,Λχ~½ψπ7sω0ΉšΟ ω•ή©xNSωhτ6JsG“„¦αφ K&Φκ%/ΏΑ‘ΓΥΰE ’k³ή°0‹υx$Ο‘|ςΥ… 'w~τ£ΙίζοεήϋξΝϊήBI—΄ Oε_ώεΒƒ9ŽΨg’ΈσgφgfεΛπ‰O~Bz{zΝ]ίϋήχδΫίώΆq©Ÿ^ρ“#ΏΜΓŒ?~\>ψΑΚ‹^τ"Σζ ΣΜλu»Έ°nΩN½φΉϋξ‹$8rXΌM;€Ζb>Ό•‘j¨h%$6}JΒΑiΉώϊ«Δ΅Ž‹[•’ΫFxžΩ—dgΖ$šπJ f τΥγπTEΗ±eσσͺV&UΚ{λ{(D€D &Κd•‘ŠUϋΟβO2 T _IΉC£Αττ΄’iΚ\3\ρΊ‘―WN œΔ*XξQΤGZGIιx<°€Α‰tt΅/Λc™_M§(%B€cΰωϊ"χω§iuฟξβcπ89Kί€Y³Šϋ΅˜™I…ΐ¨₯[y*K$nκκ]‘:cl2M₯D€vD.£χ―ƒήΕ.<›^'Ό ΤΧΚΨΨψω~αJέ*V6s,d&W#qI`΅'₯•Y Δ«'O««y½^‰ΜŒΰΕa‡Ny¦ihhW2M–¦`1³‹¦­ύˆΆ G°Τ!β699az7Ι4‡Œœ‰KΙ4ΩP£ά·žiΜδzrŠ]" ΣπΈ&E`₯ΤTπdJ±Y­ϋΩ‡p\^‡ο?©†sQ)-βx.ŽΑj•΅ŸλkςISk<΄'Ι4sF©—§N}<ŒJλ–3waNe˜Λ+§eυ@>ψ Ός•―”;wΚΦ­[εδΙ“z›α„άγ?žn䜠ξ†Η6|z“‘»|~ΙŠΦ*jτ<σέο~7}ιG?ϊQ³ΝσύύύΖγ 'œ¨ΐε›ΊΊΊŒ·.ΡT~lύ³ 8p½όθ?Cη€eΟ ƒ―ξ“g1 OœM›·Λώύ—›œυ2M`|F6mΫ*MMΙ°€ε+©>IPΦ ”Ω™ΚqcΚ*<ΦJ©+»œ„u8ŸζͺsΌ«±ΠAnƒ0oΘ4 ΪΨ1Ae#‘o·Πψ“M™d²€,%’¬ι†dNΎΕ²ZSi°φ!κέ££cΩθ‘"£zΈΩ2ΧQ/£c£B ₯)ζšŽ…8‰Job‰X3u’IX).—[jλΰ 6•z¦‰βΗό‰Άš²#@›e‘]”eCTgΏZΪrn»½ 6Τa‘TΡt>j~λ°ύ2uχh$xsl0ψ cΏNΙ4σ!Σύe!0Χλ#۝¦ΥEΐF½wέ$ŽΝΠύΨίβύσ?3B’l1Ϋ$ή|νk_3Ζ©—Ώόες'ς'†PΓjΘ\{ «―{έλŒONτΩΨΈ<ίΧΧ'/}ιK…δ*σMΦ3MsssΎ·θuEF€mˆΚχuΧ]'wηχ%6uRj}[±Κ„F¦εz§‘u†Ρΐ<*―zΗ»₯§§Η”άΆ§"Ώ†fWPQολΫ χΓ03ŠΛ¦ζzifehzFΆΕΥ*šJEΐΚF»²¨Rίs=½ϊ‰ ¦ΆNΖΗα z˜¦#`Ώ ’i¨_ϊ‡±2œ.εsiΞP=0ξΗf€εΚ–tψP›OιK¬OP΅‚=“f~Φ8΄VΚXmεΰx³ΎΎΞxψΘ4–ΞHρ¨N˜U[‹(οϋςϋ§^WΛO΅n<ά°Ή),”†Lγ†·”σΉZSiΰ$gΰ τ4%Ι4σ§T\ Ψt6Ίd„›‘‘Ω΄i“z¨)m•,ȝγTΚrΚl!Ήn™γ#MŠΐJ¨)ΛέЊPX΄B Ι4nwƒκ‹›ι™†—qt!σ1‹d]Q§(―8 η³©Ι)ihqcν/d™»NœπN3†>…)sΜPQ¬ΰe(οζ)™ χeϋ<Ξd±MžΧΒˆc h“Ρ+²2μϊ[Έ˜” ’HjηH‰ŒίΓαˆ’iΚQΟ οΓ 2Ν ŸΤΉYσGμ·Έ 4ˆHK[³ΈAL†”kJ η»b€yψπac ψγ?ώcΉσΞ;₯³³+mςΔOΘΗ>φ1Ήθ’‹δΉΟ}ω^ρŠWΘM7έ$wάq‡άwί}ς{Ώχ{†L“o§g VΏπ…/€ΛεmΌ0±ύΪΧΎV^υͺW-Kω`ώ™Δœτt£¬lίΎ].ΏβZyβΠaxΉLj<}ψH/tlω_6½Dύ™:!­m]rΥUW₯έΆ=域^Y-°mlΨ°Af0:πμ'BθΞNδς}J¦©–†P₯οΙφoε£ύ­R(Vυ΅­nΔ•K4Έ4y\β+?YδΑFξΥ›^ωͺˆυΑ ֎Ξ™:λλ`„Μšhά Βύ(™τΐH‚¦&E@¨.¬ ·dNόeΛτVS]Θ¬ώΫ2 V'h8₯!(%Κ±έ 9I΄Ρ€ϋύΣ{ςŒ‘7q‘P³hJ6ΞΨ4jλ$πθ"d@IDATšξ7ςΖ’7ιΙe"`λhzjJŽ:+—οί$ξTΈg Œ¬ Θ ³šž&—όf`XΞΓ36ν“šΚ]Κ3Ωp}vΏό%Ρ'VτxδnπAGˆHέl co’i0WΟ49«™dšΨ,κRŽ•L“*s‚6œ© iΊH¦©wδνͺGθΐ$™fρ»«σl-H3Qzϋδλ£#f_LBmsΥ‰ŠΎu±H.tš½Ν*ν\ναΕBΈπ|h§½„vj{Τ½u†¬I’¦#`Ϋcc£{ξΑ dšω‰GΨύ“L™¦£Ή zSuiRŽ“ζC’{Ÿ!•NŸ>-Ν-ΝΖ ‰4lθτΪ°~s㣏>j<ΥΠ€H―/$ΠΠkΝύχί/ς/CΒ”™΄γΐ5ίD]ζΫΏΜ οΔηΨs…ό’HSH9ς-―^—φ#mkk“kΉVBQIψΟBK"ϋ89PΞ/§ΉWΝ†Fdrθ˜\{ΰΩΌyσά“Ί§ΜCΐΚ€ήήΔ`MΘt ;ΔSžiNƒLCΟF”3LΆΝΞΛBwu€mΧό΅Ϋλϊ…ΦyαIP•ΆΖΑ’%#š`p‘sΫααaσvZOε«d_£Oά­.‰ΐ}%cΒ.0άS]AŸ Γl!Σhύ”―~τIŠΐZC€Ζθ¬HΝύ₯tk­Ό•\[΄%Τb%°1e˜ ΜΐζΡǚ™σJFCίm΅πϋ2Žπ@.O+t† Ιh„9 USS+΅N―Δ± ›žP˜΄ζk‡-¦ƒCƒ2Œzκπΐ[tok0*d„ύ >§œ‘ZcΒ<ΥΓΠC2’ν³#GLŒ†y$ώΟ)D'Žιχh ΡVˆ &Υ:Mί‹₯φW˜―ήΎ<¨_°Oα‡ξBœ'ͺ| χΤ₯<œ%Σ,ΧεήE½¨d†V…9G?/Q(Γv2 ’M‹~½ ί(™fFv—«jŽ=*W^q₯lΫΆΝ6Ώ$Φ0ΡC έΨ2ΡΝ+ΰ–[nYβωΤ§>%gΜ1η ύoβžμSεα‡–ο~χ»ςψγ/PξηίΓ}«¨ΠΓΞ]wέ%ώοžώ(y^Sy°˜_yε·q»ΜŒ‡g™ aLyΖΦξαJ‚ι“h{㕆!Βψ UΈς„²Š/knj–Φf™ ΗLˆ§It>0+Iφb²m΅Š!WW2  δόΰ΄y]βεκX<³ΩI˜†††ΚP}D&―GΪzZ%δ D<—a’%sΜh+PC’ALžγ=ΣhRκE€αψh γΒ&‚H¨Ρ0}«Χ&αALX'[L Q δ·uoΟθ―"PLΏ žGH‡VΎΙ4:cλΘύZ7j>Ψ424¬Ί_n€–ΖڈhΌꩧeww«4»hΐ§fw!±.蘰DŒϊgΞ5ςœφ%΅\ΐ©l[¨‹ωΆ½ωϋe+K<ΘΆqb<Ϟ[ο0pΞΒ ςL"•οσ”‰Ριtͺn°HΕRΟ Ξ„Δ ™θwΏ †yZ-‘€ί/Sc Σ€”Y2Dšz`GoΔ•ς- ΐ2N3M,#bqrβ'p5)+A€m) Γ€W/nσ 2”¦ΥC€} •6ΐcΓ“ξΆ„ K’i2―ΥmE@PJ…@+cOž;/­σΤθ¨AˆYi‡ΑA eα6-)ΈTΟΧ|η"Πθm”¦Φ& ƒL3 7–sτ^ΚAGCIOτ–Θ€†1ƒώ§T4φpυΗ8$j8Α€©ΕΆYmUuh¬Ά6Φ‡IΖ”ήλΦ}ΠΪ(–’B]ΛΌΙŒΐƒηŸ•z7Ζ”X›4\,φ’Ό―FκœMΠωDα EuΏΕπZΩ9?κη χ΅4JΘλV3b"•5k€ΡY#u·ΘΣGiΨΥ•AΎΌ»Q φ;ΰd4“έ_^†zΧRπ[Hκ2³ΖCΦΉsη„\άaυœ₯ςXηΣdx&˜Γ;μʜσΠ”8<ΣX23žiXL­);@P¦GύβlL’iκAά€gš)„qL{ΘΘ~kΥ΅ΊΗNΡ”g‚ΐ~˜ήj2ϋζͺG_Έh0€Sm½Χ„ !¬ͺ¦ς#`unΦΗKΝ N D:x¨ΑψΌ䚑ρ‰4™Ζ^_ώ’VΧ©ϋ1 Ι2q̏^H”ΔL)=ηβ‘Έ‰ζA]ʜΑ½•œ’Λε xΓ¦&Ÿ\tΡE26>nΒ5eήΚπJΧ_½<πΐ Ξ‘xΓσLvU5+¦Δθν&σžG˜ψ ΊΜvMζυά&ϋχ駟6χΕβpγ¨JŸΑb΅ώ£0dœΪkX4( ’iΈ* o5Ι~ΜΈefHΖž–›o~žtuu­Φ+ιsΧ!”%ݝ2 I¦“3i‡‡φφφuψ6ZdE`y¨rΊ<܊yΧπΘ°ψajΕ c1œ‘ˆΟY+MX ¦½i]υάyqαkn”π4βΕ’o€}ua‚ ΖapN†8]xQjA€€j’ 8ΈŠ’Ζί dšBΗΐΥ‚[)ί“˜άΣlšδΣμΦI)Ρ―ήΌkS CΉ~©ΓΚΧΪzΜσάΫε„Ά2ψξ&‘šqLͺϋεlΩ'ϊϟ—S τ`’³ „3w—a«€ΥΆdi†.Ύ©Ω#OFΖFG—ύ<½qy$νΌπI‚ͺY΅ ―γX1«©D Ν³oo}λ[ςΞwΎΣ,V܈EozΣ›δ‹_ό’±©W‚L2d§+I€šοmC<•έuŸm s($Πx°Wΐc^jβdjšƒΫΏ†yrz Π0LU„ΗjtΑcM =V ߜ—_ξNjz‡γ¨(°γ.ψŸdΰrσΦϋ ΐ9`ŽΧkjλ₯ΖαΑ<2¦!Γ4­¬±‰IΨΌ]ΖvB8mα>RNŽŒK8E~Z½VΧ“Ωwy<Ή€L“βrπς™a ˜< Fw2;ώ_’2TΐKφυυɞ={δϋίΎϊ¨Ή΄§»Η°­–ΊOΟ—νΫΆΙεϋΘ±gKkΗ~©q#tX†[ΏEKcΥl,(‘©βσ΅Λε—_f@Άύ,z―žT€Ι4cNΟBώΙ$<Τ΄tφJGG‡β£T4vΰn+ϊeΧπΛQ?‘ŽtβΔI㕆‘¦B³`εC>ΑΏ|_«WŽχŸ7ϊ”υ~²†_§bŠζσω€γ“£f01—η‹ΚM'E=ΑύeƒΗ­:eΕΤόϊy•έk«8ξεκ*†’ρΧCΙ4t]¬iuH#)ϋ©˜Μ’°’4)ED€mŽzεΑΣO?%Mm›ΰ―<ιΙ9ΏΗΰώ:§xš7ΘΉ³§ell6ŽBξΟο)Υ|λ‡ι)xΦζ$Κζ–„ε³Bb.2¬OΚς^_½„ ΰθΡ£²^ms―Φ½R $ΣΔ@LCθ[’iP'\©©ψXωΕvώ‘}HώνίώM6mΪ$o|γΝδΚOφ›δ’κηυN―ΤΤƒ0€yG§ΟG«Όϋ$Σ ŽΙφ‡ σD88‡75-n‡Œ¦Hμͺw—―^H¦‘gš 3i[IzΓ‹‡yFάJ¦YP3Ά±rςζ²Λ.3dš―ύλF‘ΎςΚ+Ν ’η6oήlξ½ώϋαδζ΄7†~ϊιOjΞ΅ΐΣ“bfg‘μ³Ι€gEώε_ώ圫{{{₯ aX~ψaaΘ§|“~Σso2“θω–'ίόυΊό°uLΒεϋ―G~Pšαa¦Ξ'‰ί2Œ!³‘I™>Š6Ί_ΆnݚτΚͺFΐΆ?—Λ%­­­2…AαddVό‘(VŸ5!N Χΰc―«j°τε+φ™ω’kό₯¬C¬X’išΰΞ²½‘NΞNˆ„Q?П74ΉδOž”ρ±±5ώ6•Q<+σ©{6ϊΌ9ŸςL Žαω’ύƒ+d9ˆˆ†cΪήͺΖΧΚ¨~} E`ΩΠD2!γpqΠ€‰ΏzŒSόJ¦Y6¦ΕΈ‘ύ,$v2+ϋΓ_ό%Χ½γ)š‡"0ΚƒcǞ—·Mjœ ™\Α7χͺω{l”Έ“m=rκ©»αYyRΆlΑa΄c«ŸΜΏKχσGΐβΘΙ<ΞzΩ’L-uΊŒ°™9BŒKŸΟa<1>|X^|ϋν & mΫΊb(”(&μλ±JΉaefQO₯ψ [Ό)»ΎϊΥ―"Ν›ίόfyΧ»ή%—^z©iσύύύςΝo~SήϋήχΚG>ς³¨vǎλV>ΥΥΥK½Γ!³ΑXŠLS#6\AρŒ x¦ …Ν€gi$=D"αΚxΉ"ΏεΤ4Β—Έ}.x¨K†€e?g,θΧIό\x³ΣΝHTγώ3οxΖ%Ί©δ…Ηι΄8\˜λ©σΰ.zρ›…v8―Œτ’#ΐ18όΗΫΙ™ΑaΉbwG;;>x~στN³±΅IθE’ΔD†`΄zʊ,ŠPΧζΒQZQ2“•Ξρ(Ό³γ„ΛνΚ<]ΡΫy¦ac%;ϋΆΫn“ϋ―:tHήφΆ·Ιοώο›_NB_sΝ5°ώπ‡†ψBBΝππ°|γίk= μΆmΫ •ΟζD7•xΖhe˜&*·<Ξ}>ƒsnΉε“w‹Η)Ω ΈψE/z‘άpύ ζ5J\-E½ΑΦhΎ¦V‰Lž”†ζ‹Π§±™ζ2:₯>g~Ή08Εη0ΐφΓ+Νεi"—ΦkQ«©b3³ν .§τϋ£υ”kkιŽ[“"PΉp₯ΏώiZ=ΖΖΗδ4Όμuΐνo bi³6’Τ ΓK§·^¦°–z•Y«Wκy2υΖFτ τ ρΠάXπ¬Fq0d0»ΫΊ”LS=MCίTȊ€ρLƒI(†8žMČg1z¦™žφCv#D…†¦²"`υc7γΖδγiΌc²zPς¨ώ―σ0ώ žwΫΕp%πδFΟF»«YJίΖωZ‡ΤΊΫΰ}#*§NΒ‚‘}i―ΠΕ+auη4€ϊyτΠaΩΣΣ ½;ινd1DΪje{»W?υ”œ9}Zvμά©:ωb€ρ'€ΓΥ»AzBΘ-ΚuN¬h*.vŒΙ±ϋΨΗLζ΄ϋ_}υΥιΩPO=τ|ο{ίώ’L³^S–ΏΧΧ;   =Ao«G %Σ,^› ±6 2 ½y91ΩI2 Ιnš"@ΩεΗΐ  $ΣPΰPΐY‚W4Ž4eG€‹˜¬Ζ€<šμ8ιΡΒ Ytzjδ\„_­s‹ΣεΑώ΄ ύD»Ÿν ΛU―^v,>59)ƒS~,*ν¬+5αUωέ{"νn9 8= ‘L£©΄ΨφOΙ4ρ!U9€%Ρ†Ξ-xm΅€‚Θ4fΎτ₯/ΙΫίώvsˆŠτήπ³}γ7rΝΏώλΏ¦Ίυ"sΕWΘ‘#OΙ>πΩ΅k—Ν*ο_Kˆ`©O|βι8τ¬ΰΟ}ξs† Ο2Όγο0Ο•˜?>―Χ«Ή@ZΕγ[·n‘]ν•gNœWxTjz.hN9Λ…Αt<(19ΆdίΎΛΜ•VδΌMO(σ`(Ϋ-ƒώ˜Μΐ•Ι4VώΜ»TwŠAΐφ™όΥ ₯Υ­ΦS§NΛΩσςœ-Mβ΄+—L‘j0°¨‘}½mς$Œ›·‚Ψl&jQg*£J[gt1lΥ 5Ε,™&iΧI>FDΊΎŒ†£²‘+—₯^—φ%4χŠF@eΐΪͺ^C¦Αλ…CRVΠ; ™fΪμ«!ΈόυEέ†œ Λί ΣΗϊHδπDQώ’κ++—ώœ«]}›ΊΡΨ°@ΓΪ©r›«ζ@Pγl’–Ž-ςψγΙ-Ο»Eš0VUΗˆ–΅cλη™gž‘cηδyΆ‰ x3§XȚxΧ\ΦΣ(Ÿϋε19~όΈ!Σd½Xg8‘βΐ,K=@n+™¦θ0§Η•}}}Ζ#=½7νΫ·Ο<ΘΪ 86’gϊžΨj‘X/λ9qαD=ΌΣΜ&f _γR Ε@Ι4‹Χh:ω$Cγ%υέhLΙ4ΩPcH,R²«†!κpϋ zμG€a(Κv[Υ£ΎΈDβV«ažͺΎ]z¦™œ—ϊ¦n|“ βtϋdϋόNiχΣT>¬>~~`@ΪœΠ³I―IŽΦ9<§nήβqΙ©3gAΗšΚƒ€!Σ`ά:˘[LΙjIn§ώg(Φ—Λ©dš9ΐΨ6pά ζλ_z‘rύνoΫΈt΄™qίχΎχ™[H¨±ιΡG•?όΓ?”?ψƒ?HNH ϋΑΨkςύ₯œΜΔΨ­L[·nM+σ™ηu{} `ΫeΧ_½<ςΫ_ŠoΓ δ%hT¨Ήο“4&ΑδO‰Σ²ς«€·Χή3χZέS–B€²Œž«¦ΰ "Šž›“ΥΆm.u―žWΦ+ΰ3Ρ@F™ͺ©όX9sφΜι—Mϋ:γ¬ςJΈΪ«έU'[<ςΔ“G0ψ›L{`+i«ο‰^„ϋ£Ρ>Ƙ°σΤcΓȏΐ3 ϋ V6jRκC€rœ}¨ί1`J Άλa žœš2ξ£9 ₯©<$Η‰ ΥξΫί97¬ςΙ4ό›C>Οβzχ5eA ₯CΕcμΛU!1D,6Χƒk–»«ςυΟx ©w DυPxπScγP£);œ€₯u’όΝM“"°lΨPfΝ€L3:r^z:–Ωϊθ ΣΐCΈz‰Z6΄+Ί1"ζYxgoiΖ²4?uΪΐ=ŽiήwxxD¨·wtt˜σϊ_ι0!΅@–!a&%„<”ήΫYWgυΨΐ ^:kˆœl~ρ‹_lHtE//6]rΙ%ς©O}JξΈγαJήsΡE™ψͺ$ΒXαe―/τ7Sωfή―zΥ«δ…/|az…vζω\yΫχΰωΜν\Χλρς ΐΊγmϞ½Φ(±ΐy©kΩ ϋάx%΅§1υ‡τlxLFž‘+ίς±„+­Ϋpι% Ω ωƒ+NΆGKδΆΆ§%ΐΣΣλ»ΚŒΏv{έΎΜ:,Έ•/Œ₯}zΣ†―txjS†)ΐIΪ\θέ Χ産Ο"\Λ[2ρ:|εuUdΚ~κΉP5@¦™g$€nΒϊ{ΛπLΔτ ‰ͺI('”!I“C9ŸͺΟΚDΐΚρ0V—2qK#VΉΑDΫ'_½00LAΖ±Β»Ή₯eΕγαΜgλφp"#I¦™Kd’›}ˆx„Ρ™'Ϋ—ΞR―Pr"`εΑΘȈ=ϊ΄ψΪ7#ΔWΊζ0hδΞ ―α~ήΫ-c§ŽΛΙ“'UχΛ‰Uα':둃OΘ„mΪΰ«7σ›Ζ"Ό +š‰“½lύmB=]άΫ"Ώϊυoεwαv~/Θ4šJΙ4A’išα™ΖΝqR­pu9Η™$—€ςŸΐ±OR·Δ„ŠTΒ}z©‘\#‘μλ_Ί οτ‘|DnΈρŠ΅—ρΧnΟGk-†h`ϋ©%™^i ‰ε·ο=ΏόΥΎo=Poͺn0Oo^μέ,Α¦Ϊ1šώ”QxobxΊvjV¬¦s ΫΗ’1Τ£Ο|Μϋψ©Ή3άmb:ήM#£Λ@ΐφK@¦&ΞIŸΓ‹οяw22xΜ„ybΆV_Ζ#τ–°8“ˆyζμ9i†·θwV`ΏR‡oΏ­!ρBSΒΠΉŒX£©<Πγ:͌\Tš5ρ]¨£jςζW0™†F¨ώο6 ‘4ΣΩΩiά;f‚ΚFΟγΟ{ήσ䦛n2J΄UDν‡’y}‘ΫVψΩϋΪΫΫ…6Ν?oλοϊA€žeφ^r…<}μYqv]-5nΦoŽ—xHΒS'€gΓ6ΦmζE‹ΡΦΦbZb!ΐ S7ΌoMa°ΓA%Σ+ΝGX‹°ogb›ηŸ¦ΥA`ttT;xHvu5Kά[ΞΧYΉiS“Γͺ89ΓΥ°ͺσ”§ iϊm<«gœαϊ2<–F_c:„¨ΦMyκFŸbΘ₯'Ϋσϊ[Bο46ο·Έ±¨9A^+p“Η!ύΣ~㝆dMεEΐi`¨«eeΐΰc‡•t·Ο‰ Θ(o}TΛΣεθSKχŽηa&ΘCk$^ν―„ΠPυή^„³ ›°BΟyΞsTχ+Ώ,ΧR?γxηΰγΛθΔ”μήέ y]kVΐ.Z;θjΉJvG«S> ½Έχξέ«u’ηb’š ?(­n―™”v"Ι<ΞŞj+.βvcq7^θο½χ^‘WšΆΦ6ωγwώ±Όυ­oM.:ΐΗaοακρs šeΪ˜½ά=z΄Έ-Bn΅΅πLƒΏΩx $FJΫ9Œ"d_‘Y°n[Aaa+ΆaΊ˜SίωΛ|)Ά}’fjΤΠΰΒ:qώύŽΡOγjΛ­ύuψ `|5ƒ_tΏše#`υΏΑΑihDψU.ά―s"δšO†‡NKaž4•.D’6Γ<Պ Νpr’$ΊvBβE’rΎΏίΞκε/iu=1ιq•0b"’i(–•L“Šέ΄ τππ°όυ_΅ό⿐OϊΣςGτGF!Άηy=5χ™,˜vΏ ž+t8eŸeœΗ,K 7lΨ`ΒCeΎC·λ%%@ΐΆ™ιΝθ±G’Dh n\Ϋπ4~žΙv•~4ΫΫ[, Ργrρή=²uλΦτiέP E€ži8Θηΰ2!ΣλV‘yιυŠΐzAΐ’i’ƒy]½Zυv!žNΛ/έ`tk‰.Ξ€Tw'=^°υ1Θ8vμΈΠ“ Γ ©ώRΊ³Ψz<ΰŒqB&‡ ϊκˆƒˆP dϊ n΄χ•dš³"pΆ7Ά;+ΛνͺΡ WθVΉΰΚω1L8ωάNiΰŠSˆ !›ΰžψΙΑ)’―\eΡη\@ †Π χ“LcϋUŽ+ ™υΓI~Gšb @›Η’O?υ4²ƒΗ7B<O»Τ±SŠ]^b›„D‡-ΔΣΤe&‘)_ΪΫΪTΟΘ Ώ…YύŒvΝ‡~σΩάξ“MΎΊ€₯ p_ οεΦζBe{‹S:| ςkδρxΙ¦Ϋy›7―ΣT|Β ”ω'ύεj–:Μ¨Ί0 ‚i2MρŸ¨9f"066&CCCΖγόζΝ›εη?Ήœxφ„ά}χέςšΧΌFhΓ₯ά£>ϊδ“O/φ™χΟίΞΉyώ…eΨ―Εl=Σ$βAIΜ B`ο‘)7¬λLz>1ŠE‰Έ4@Ω₯νBΣBΨ7ΔA1m ΊΠΨ:v΅ΐϋά…ω³…wVχΆ¦v|—Η γμ^u yhg©nXτν—‰€ΥΡθюvWoSι χΦƒ€[됄b²s“Λ|”ήV $ߎ"ΜΦU= πBuαf~λ$Σtx’‘=陆φ.%»^ΐ¨”[΅ΉLYΧ‚€κ)©Λ]πζgΏ³R–k΅σ.Ψ3 Γ6Ρ­#SWW—™tζφ|A³Τ>ο)VϊήχΎ'wήy§8pΐΈωΜ•/•=~tQ„oα6'Ι<(Ό•―|eΫτx™ΰ‡G%sχξέXένD¨§~©kڞݲ‘²v$B# Mƒ€³ΫL,²ΘσΫ`™_C·Nΰͺ&·Ϋmd z$Φ01°NN­ΣJΥb/‰ϋFΚ^φόΣT^Ψ_Γ‡‰-ϋΰjž“―sΰΨΗ%δͺ‘έ]MςΔ‘§ep` έη•·ΔΥχ4φ ΄άΔA¦AmqsN⃱dΩhRΚ‰ε7WκΠm>‰Υ0€-'Ύ…>‹+δ‡FΗdΘ3ηBgVXšέυ2:5fb}š§^ΏrβπJa¦Ύ“ζ΄Κ!ΞΙ2lγφMŠ@1°2ΨοχΛc="ήζ RλΑ~Ή όΎΆmrθΰΓFχ#™FΣΚ8ο=φ„Ότ’pŸςJq°Tβ%ΤΗΫqΟšεǏΑύ¨!Σ,u―ž_‘pD¦F§₯ξώk]΅βlpˆ¬μw5•kWνλλ“―|ε+Ζ>Fb͏~τ#yΟ{ή#?ώρnσΞwΎ3=β΅ίϊΦ·ŒύέήOuΥ'N7†ΑVΊb”3ΛhΚɏ;•”Lc‘ΘύK΄(1^06Ξ}Wž±} ~ΩΔbΑΈ$ΐΆ§ν€αF5εF€jϊ d‡ +Ζο4χ₯zFΘ κ gA¦q64§ΌFΒΆο‘ήζnxT;kμ)쫬>ŸW¦zъ Η™axφνέ²4>zΚΙΜoήjz›δΘ4τ|ΧΪΪͺυ³"Δσ»™αT©e¨F n΄ηͺIg*˜LCν`₯Ή‚‰J±Q<@šύ@!ΧfΟaξΡΙΙIsΰ‘‡š{"ΛŸmέ°Ά₯Œ4BkZ{lΫΆMΪΪ:$g|‚4›–Im$©™|ά1Ω³'ιbW;½΅WŸλ₯DdΈr2”r„;λρJΥφυRƒZΞε ΐΥΪLlχΪ'.Αείcϋ«q%<ύ΄tyέΧΈP=γ`‚δ/“okqΙ:#'α™oΌΈi*=–L“ˆ’lΖΪ@ed$Φ#Ι—‡#γ¨n*₯G€γ²8ΌnΠθ“άŽ§u—?]Ÿ00V» ‚L³o{+H44ϊαžH¦ιG8N°k*?ΤsH˜©e₯pψhDx3 5Q(Η5)ΕBΰτιΣrςΔ3βςmΔΔ?B»e]Φ·ΤΣRϊF½[κΌ=Ώ„wšci{ΦRwλω…Π8ƒUξσ?#ν>μjΓβ0Ψ9L΄σIΤΗ K.jsΘ½ΞzyΰWΘΎ}— =άZ½>Ÿ|τšΒΰjςι!Ώ8C±+$ΣLO*™¦0σΏ:³OδwΓ±! zμ|ΫΫήfΖ=όύΫΏύ[yρ‹_,—\r‰ω6mΪ$―{έλ²>μq„W{ίϋήWΠB֌Šx“@ό³οΜχε“~ΣفΆψdžΝv,σ|΅nΣ–L―*‰YΜ=±―‘*jΊwξ\hkՊΟRοΝ™ σM.u‘žWς@€ΊΔΩ³§Εαl’‡7)γβnhΑρ3i2MYι%+D€ί5ΗθύηϋMΈΐNx α’RHF“(&Ξ w5›[½rβΤaH(’i4•T@; “&ϋ«O5‡««[8[³•Ήvξά)ό{>O<ρ„άzλ­iφωά+Λ·wΣso’όΗ4ε°Κοό§srαθY‡ΜyΖΦ½φΪk…μω«―ΎzώεΊΏhii•½—\*ώϊaρD& =Α©, ΏXΖ΅ J dšΦΆ.ΩΈ±o ”\‹°ž`gN#e†ΟηΣ ©υ\™ZφΌ°ώΪνΌoΦ ‹‚ΐqθ'GŽ=#ΧonFˆ§άμoΞιi¬•]έ-ςΠCΏ–λΏήΘ*5τ₯rfb½”qYD2ƒ’ir‘£ΗK… B4B°ν‘LΓmU&” ρΕσε"< xΘ Αδy"ƒži|: E0Z<=[ ™ž›κ`€K¦,‡α;†~Š¦Θ6*ΓKUej'Ÿ|ςˆŒŽ Κ†Kn€%f·e“΅ΨV1ΡΫΠ#Mm›ε·Ώύ΅Όΰ·™E*η kZ―g‘s?τπ#rQ§OΆ6;LΥΠ`ŸwΒΕΜ«―±N.έΠ"?Ώο~ΉύφΫεb 4ϋMM‘t5; )²d&’i&>‘αŸ4‹{fΞlχLό₯yΣM7Ιώύϋα…λ1³Θƒd›μχfχΉ—„•΅θ—ο:}ν*λωΗνϋTϋo&ωΘba1³ϋϊ›D€λ1N ΕΓ€Ύ!ό3 sjΘ’ά-…r$‰XξkτŒ"Pccγπn2.ή΍2=@6ŠΓε3žihO!9ZSi°:m''O–->³ptώS©vxΪ³Χη’Ÿ>}R&A¦ΡT ΑnB’ρ"ΟK¬&])o2 AaCgh§Ώψ‹Ώ0Δ²ΟΙHΙK^bβ£rš υRH΅Τ5‹TΣ‚S·έz›πo©ΔΥh$ΡΌωΝo–O~ς“ΖE%ΛΎuλVsk1Λ΄TYτ|nl=ψ|rΩe—Λέ?όŽ΄o…α™ŽΜ˜l špΤ/ώ‰~Ήz%b½%εΞ]Ο(Ω°νŽΏΦ3 u`“/=ZY0τ'›”LSήΊ₯Μ‘Q‘ξζ&&eηε]βσa΅‘+-, ν/έή:Ω fώoaΈ4dš…Wκ‘b"@ύΦΊ{Ÿ/λ:2υ[^§I('$ΣpŒΓφ7œ1rΌœΟΧg%°zδΘθ¨YQε)2€ω=λˊžφ@nŽŒ zֈ€–~# ‡VŸςL“S2„«ΐΓAΔpž²΄mh½”Ύ>*υ ΆνΠωSGŽΐƒFH0]x]«¦E±[ άb^­»]\½ΠŸΒΚΩs‰­ž —„nή”Σ”ϋΏ+£γς²[ΰa&7}ήνι]Φ"υρF„ΊΈΓ-Ÿ‘ίόζ7²gο^CH_¨EC€cT†jμτ`•l’ιπ:dlψΌ0ό“ύώŠφΠ*ΈXŽŒŒ`1곘TτoX΄‹Y]ΗBC]fσζΝ†LCOLφϋk―]‹Ώω”QΫUφšc{ 6™IΙ4™h\Ψf¨ —Λ)A,Κ1dvλ€Žo~Tν°ΚΆ•ωΞosΩΧcŠ@6H `0ήgšθ5’Ηπ1’TS}bό”ιϋΪΫΫΝ΅ϊ_ι ™ζιγΟΚ–/Θ4surΪΓΩΛ8AHlk¨Cν:oNΙe—_ϊvι«&―'Μ&?«Ό­”‹ ²ψ³γ †—£α3Νϋί~ω§ϊ'ΉρΖeλΦ­iOΩNήO†ίk^σ‘ΫΗlΧ,Ψ|:S>›Fζ––ΉξΊλδΟόΟΝ€χ]οz—Xw”™τrΚ’χ»jaΣ¦Νβt!δNxρGB˜Ρ•‘P―$ΣLž‘νΫ_&MΝMζ€ΦcFΊYl;4 Pΐ_%ΣŸ^ΌΞ°²2‹š’³έ³ΧT¬4>>.€³··M:½X} 9ŸΓ½P^°Š~k«K>= wGΗΐeΝΟ\ΘV·–@€ΖΑΕ&ΏY—$Ϋ¨1l υtΡΘτL )™¦θ˜αXšLS+Αp܈fNΊ""…τ €ΘΠπˆ°Ξ(O4•κ6œpu;άXa•²ΜρρθoλA¦1d›Τ„lωJ₯OͺT†††`+{Dš;wΰγ§}‚΄Ί¦:§Έš6Κω#Θ“O6d«Η―0ηͺΈέκάύύύς?Ώϊ•μξm•-ΝuFηŽAHΣ¬TH’qŸ~“·5ΧΛ–6Ÿ<ˆσ/|Ρ‹€···hvΞBΚS©ΧΪz£ŒΗjrw“žiΰQ „gC½LMJ(JΎ~uX©˜γ½8ιxΟ=χΘήπyΛ[ή"ŸψΔ'€££Γ,±I(ΈX•^λ™Έπ–ΙΦ™ΩYσρK^Όα¨œ[‰Ά~i'MPΑM%•S‹Ζά_βblŽΓΒo4ΤCρ)N2ΣalsοΠ="ΐΆΖΏΜ4?σœn+‹"υƒΠΡ#τVJ2 •?Ά1Μ5Φ»ΫdrδˆτχŸ‡Ž½§ο}–žΜΡΡ9Ρ? Χμίd<Π°[™<₯@›»F6aaSO=/Py#ΌΒ ρ}°>ζJβ y֘ eυ…+*o+o2U˜hό{χ»ί χ²Ώ5ξ·mΫf&ήΎσοΐMV~n«oΉε–’’i p–€qρΕΛ;ήρΉλ»Μ»LSH>•ΧΦΦΩΊhmm‘ν;φΚδΤVybΕ =ΡΨO˜ΔšDDb‘¬"l”ΎΎ>C’qnνύkλ­΄4λΆNˆ*™f=Τ––±XΠ@Ι~ž“MkΡυr±ήs­ζΣίNyZιk’Vw-Όδ6κSW­‡œΪ7η’ΉςωωΟ~:Τ₯φ₯©eŽΩ7PΞ†1ΏυLSμ5ΧΕƒ†@"Χ Ά•Ή8^₯<Λώs+ηλa$oD˜§IU)·Αאv―[‡‡Qoš₯¬‰…yσ»‡ΒRηπ&'1X)όƒu¨uυ'=Σ,ΌS(ω#`υƒγǟΑĐτtΐjWOPΏΐ4œΎΙƊPžnLΌ5ΐό΄ψύ~γ†ήΪι Ι­š―=tθ 9ςμ)y5ŒφmξΊEuξΕp’as’f₯μe|ς>rD:dΘ4‹έ§η–‡@šLΣθ2ažŒμ†g ‡§^&R.νͺσε=Aο²X›ωΝ;Ν‘Ÿώτ§†TsΫm·₯W‚s\Δ…²?ωΟ ™ζζ›o–­[·Ϊ,ΦΝ―™CεΗΌHR›Dκκζ.vεXXΣBˆ•Ηλ‘θ lmμ8Πζθ‘&½ φ—[ φ QKαχg“ձ씐=ΏŠ@ΎP—‚gοZ„_­1dwΘ°Y,&­…ώ° `2`gcγJ‹[χ[ίττ΄<πΐƒέμ•ένΝ‰9”ΤΌζ²p  ‘λω]-πNΣΡ,χόμgrυ5ΧoΨφ™ΛΚXoZ€eψΨθ˜8=¨7,<ΰX©d¨†ζsά@ά¨ FΐŽOΉυΞ;ο”Ώϋ»Ώ“Ο}ξsfŒsΝ5Χb‡Λ¬άΑώ?ωδ'?iς§ΗwzŸg²χ›uςΏWώe¦ωϋ™ηtsΟ©…& Φ»Žƒ-s‰ '~£3Xΐ–ςLC‡u±PΜi< έjΚ σ­¦5ψΌnΡ‹τ\/η~ΟΓC‘§©dχ†2”ύπx›aY‘_ΣΆ’δΐ7κn'NžŸ«^ΊΌτχ˜=±{f¨F§<:0bOΦ^φ;τθJΰ|ƒ!¨wΪ·ΨΏ'Γ9ΝΥ“Μ3  £Τy:ΤJžΏΦοΝ›Lcↆωτ§?m ΅φX!/Ip;;;Ν-TΎŠ‘ ­0^Ο²“ΙΖΔU=:yXŒš(^Άm1n=‰Z>π2Ÿ―`βΣƒ4<=$ΫχξI“iμ½Ε+ζTM°ύX2 •§Ϊy«-ͺ }ΧκA “LCeISι°Ί ϋ‡–f·C679ΜT ƒ€tαqΊΎτaBpg[ƒ|υ7ΟΚ©S§EΙ4  *κk0䀏±ωΊ'χy\ ˜E…]3[«οZ2 uζ€_Ι4‹@V²SVžϋ!ΟΗΗ'€ΥƒU¦ ΟΠΦkeΆDIŸ«N†ϋΗ$˜ƒ–¬@šρΞ2L2 ˆ3fΩ[Κ΅ž>j% ›0P nόΩ{ΐΈŽ*kψJκn΅€ΦnmΆlK^b;«ΘF0 d “|$B˜&Μ0 ϋ2„ΆŸ„υ ;$@ ;ΩΗI;ήwy“΅―-΅–œzΊ­§V·Τέκ–ΥέU‰άέοΥ«χή©ͺ[·nΊΧ°Δ‰€Κ£ΗŽΙΞ];Δ_Z/ωώEΈ:Šα1Ξ2'³QžŒΑIo±ψKδπΆ§εΐώ²jεΚΙ,φΫ¬Π+ΝγO?#5UIŒφ΄ϋRF'›xν( i,υΘΊΪ€<φμγ~žaεmJ-œ«ΆƒLγ­„ ™ΝΚσ€Lγ/ρIH¬#£# 6ωR{Σ-ϊ%εYd½›nΊ zM§άvΫmςΫίώVώώο^JKKe;<£>τΠC‘οϋrΥUWe$‰F«˜ο«2ά}ŒίUίΦγφΣA@ηΖΔM“]|V$¦~—@i@††elΔxHͺ ŽοœΓΩ4Θ>Ι΄ΕΨdHξξi9|@ŠJ–αS0¬š Η’M½?vΤD^©ž6.${_{έT΄oΣ³ΰα–#πή[h<=rΚδ6"ςγ'Ηβ†RŸ<°η$κπ°œ}φΩS3Ω_)C@==Žaξi†ψ&K΄₯0ε:Rάd‡}ΌxρbύΉ >Un™Ώρμι’‰δΖ―΄iα!`„%βNDYΈβΣς|(4 Ο4υυdΠ…Wƒ™ωD\%ΑŽ“žΈϋ³Ι"νX2Ν©«αC‡Ι–m/ΘυeRwε“ζ¨ΨΟΔ< υ΄ΖϋΖΚ€<σΜ3rΑΟ:1‰}΅=“ 4F#ΡhYΔέqwmΗ ΕΔ~ΞƒƒNh'Ά?»A`~0u„:ρWΔ υδ&Fmq)b=°ΪΩΥ«{%ƒθηgΦJŽkš³”0σiκδ~ΈΈY»¨PώΊwTž|ς Ω°aƒΥΙg†-α³ i9Βpεβ2„yrˆP^„πΌJ2Ν(dΌ ŒΦwΒ7°LA€vVbΩάά,_όβε΅―}­όωΟ–§Ÿ~Ϊ,2^½Ϊgθ§3Ο<ΣlFΛdμωμόΣΔοΊhŸΙο₯ο“ŽO΍i/UάΨf,™f*Δ„‰Έ„6Ψt<Σ°a’0O5Εd*lα_샆“ύ“νMΫ\8£ύb˜΄§σ€·—dš=xϊZi&ttΣMΡΦς Ε¨•#G!_―LcSz8tθ AX­5•EŽ‡vά.–ۍSκKΌΖkΠQx’|ΰ8dSϊ02Εs(›Τ&ξ7q F.$Υ™tά›Θ•• “iˆΒ\TΛ‰ΥπππΔ »žX±\0$³ώ7Ώωά7¬LkΦ¬1Κ ί+ΥΟϋiμ™ΩΠϊ yK°X?‘폿I¦±Bt6Dνωx`γd‡¬JΆ;ΫβAΝζΙtΤ;ΗHύžιο”)Ο`~ΩΡrL^ΏfΑu%½Ξ8κhμ7ΰω1Œ}KΛ dyE±<ϋόVcHnωΤ¦τ  cC€βΎ›%ΣΈΡ°ίη Ι°»τΊ©δΘωΊΏ½Ο$=0Ό΅wuKs½„G]ΤsΞSn°›žφ†Ž‡Lc睓Ψ₯λ›Oι΅ ˆEŒ|ΖuA%`5‰Ζ!z9Α–0<4œΗ°εf9j·`ˆ§έ{vΛ(”Ή‚bl@Λγκώޞ` e αζϋ± Ζώ];wH[[›PχΣϋΟυΩ|}KK‹<ψΘ£ς’₯5²˜^iRϊ²γ²$ΰ‘Σλ+εΡǟρΰu²z J)Βέέ]Πo°Ω©ΨcŒϊ,ά2Χο‘NŒ©Ή΄#6₯ΐΞP˜Ϊ^-Z$W_}΅\~ωεΖξNyΓ9=Χ0Δ(S¦Λ ˜E#*ο“K Cζ₯γόGuWΪIιΡ›u―m‚νΒ¦©(6₯e Σte,δ{ŒgšΎaρ—ϋ ΡfκUΉύKΫϋ`>ϋ€kΐΦ~™ΫΩ·O|γW„ή#Ηαϊ$ί[ΥK↕Cx¦ρUΙΡ]O‚4Ϊ›θ-lώ$8rδ¨8vB^±¬ a²Ν4gΖ)S)Βh/©Θ!x¦aΘΚΚʌΧ?’€mή.1Ί5Υ"#‡]‚Ψύ΄«ΰΏ\’ΝI‘i‚d,“-Γ}έMΛ ;ΡL“(žγŽMξΰnmm5ΚE]$η!±M Nή^Œm.Z»αΒ―. ΖΚ³pίΞ>ΩBC€mŒ2ŠςΒ’iZνΨηI$€2QžΞ4Ž¦λώΉVŽUœlΒΏ3/’Ϊβ|)€ό‹‡Mΐ¨…•`ΖΡ„POχμEH,ͺθX˜kxΞΗϋ`¨Δήhχcrμ°FΔhθΨcιB€s2ξΆΟƒ·X… Ɏ› l:5τυυJ+ΒKΡζ4e<’uH…?_ΦΥɏŸ>(ΫΆm5d½w<εΨ<ΡP »{zΰ}d„v’ά¦ήM2§Π#]wI˜΄)υ·–——O»žΣzš–!Cπ="‚"gΘ«ΜΫcrΞλφLΓοv#blψ‹‹ŠVΠƒ£aο4#C£ΖΓ>Χ;˜2½Ε~ϋδΞ°’LΓqVΣ8leμ―6YE€6οƒ ”B<ωJ#.G›Ή&Ί;ΫΧργΗFθ,Ϋ'#PJΥOΚ:=CH-?Τ€(M―ΎΓT¨§τψΙ;²^ΈωtiU‰μA¨Ϋnx&ΐ¦τ!ΐΊa].@FKζͺ-—Φ‘&Σp·Ν/~ρ cδ’ΒL)r€£bΕέ’d΅766†•ς™Κˆηάξέ»ε‡?όa&ω08Γ}—μέ»O.})t?θ6MG@ε,ΙλOmΪ$Υ%…\Ο³θψFηž~IRG(γI”\Uι‘ζEeςΘ£!ΧF!ω@Ÿ!©‚νEazH¦αf§"z¦™Π»αiŒžjΪ  υKEyE8Ώύ’:ˆ7Ϋrds‘Η3ν7m όγϋhϋΚ₯…‘dκ++h΄›vΔΆaΙ43£Xθ/”’ςbΑaϊΊq£‡'k?ˆŽέ(Ι4Πo&Dι›£c #E‘EΡK°G-N?£\ηzτ‘ƒΰ}¦\ς}$‡FΣΌπΈVR%4›,θ}Λ¦Τ"ΐώΛϊ Νκΰ‘Γ²ͺ¦ ›D=ΓάΙυΥ}gΪUŠ1―E8ν‡·΄H<Ϊ”^žά$Β‘·C]±.Ω—rIgš™ γI;…ΟπCΉχΎ{]gϋzΙ%—€”L³dΙΉςΚ+ α"ζŽκ*¨`.„”””ΘΊuλdγƍ²v-βδΩ΄ ΰΰUWW'ΐ ud¬OvVΖ1$αΖ&‹@ͺ œ`Ϋ⧝؀ U[ΞBF€c;ΗyΆy~·)½8 §Θξέ{€»·:Q­Α œ]`bξW…EΫ‹Jeη]&ΤSMM©K½Gzί$wJW2–ρ†Η9^Ψ1#wΪΔBxSξ°κλ™Ζ=ΨγθΒτΒΙdeΐόՐbέ…Ϋ2_‘ηκόDνAό€71’l˜‹\x°i~Βƒ“Ώ¦ΠρJγΎ%ν陆6Y’A@eΐ» |=RPTcH/‰)vρή‹½ήr)­Z*{Rͺ !ΜiQ{]Ό₯δRΎ={φΘ³[_7–Ι"θΝ±tΉd11fdθευυ΄ΊΊDžί±:ώnyρ‹_œl‘φΊϊ0_βΒͺz¦ανχΎb―tϋ’‘GhΆ)}¨œKίN]Ι΄ύρOί‘2‚ΏmŠ@>]’iΤvΔοŠ_μ«rχL‘ΏHJJ‹%48"c Σp7?ΫΙ4·θν‚ Ήά„ΐά‰Δ7›,‰"ΐpΨτ8γχ™ιEv***f”)ΉŽm*ޟςš:}08$…ί$™†ΥΓ1ŸτL32<*‘K¦IζΉZui’[ ‹ ŠΰBήΘ€T{¦!Ί(‹%KeΧΞmΖΣΙ46MG€ύŸΔΕΫ·Λ±Y·ΎVŠa°Ÿ° OΏ`GXΣ^(σkΙ}»ŽΛσ[ΆΘΉηžkυΒ9`κΎ”DαqΘr†v2‰–{Θo/<Υ AΎ»νΟξλμw‹@<PO°dšxšΜΓ9/7b1QζχXsδΙ«rχΧ,Š° ˜dšqς΄0>ΓdΙ4±ΫΗο|Ά+d1"Ÿν ν.Ltγ ›,q"p’΅ΊB·xΛ0τ M±UirZI6όGψ!†ΦυkΝe?S‡@6:vBΦ¬‘bx¦‘η“fμΧyπœ/‹JόpΈpΠθψv}?uuY–ψiBdσ€Φ‘+#=2Q^ηJŠ›Q’JC$έrΛ-ςΉΟ}.Ό¨ ,*S$ΪμΒNιίώφ·ςλ_Z.»μ2Ήώϊλ₯©©Ι\’eF»>‘cΙv5~&r/›wώΠφα/τ›0NΘ¨°«βΔσά‘[UU%l—6YζŠ€ΚΆ3N¨ΩΖ΄Ξ΅l{ύΒA€υ™rΉž‰˜Ψφ)W)gΥ8‰•ύ=7TΞtuuΚ>Δzm(/Bˆ'ΗHhΙlΙ~ΠςΉΆw`ΠΔ¦Η½\nΟ‰bo~b:SŸ`½²πΟ¦τ!`εχTl)―Ή€δ!™Ζx¦Ι3$ •3SsΫ_ιD€uΡ•Ε…XPuάΆ ΚkΪ*±‹ΎuΖ…A’ilJ?™&ˆEWο„1ΘΡΝΏ¨γ™f($#‘ά1₯υιwˆ&Ώ™+Σu•·­0>|PŠ΅ 7–M %GΠ`‘oδyό TKϋ‘'±3σ¨¬\Ή2%₯gS!Z/ννmςΨOΘٍ‹€Ά˜:%³#RϊΎ(’ηϊ’|iΙ}λΆΒ$w}–”ή/‡ γΌ”c&ωžB.ΨJ™ ΧεΑεΖ_υΚ—C°ΨWM!lcόΣρˆ}6—†’’XqήK¬˜μxf2dlbΓͺ±‘cΐŽ^φ΅έΝ\BξœU<ΨIžα¨ΝDΜψ[ϋf€Η'—ύ7ΪOέηgχ±lώήz’UΰΡ·ͺ!8٘"UA£Ηy0_¬€·»ikk3k‘ΩŒΙ©x7mwGCŒK…ί ƒmΪ¨vφ˜6.ΥΘ_Vθ‘Γ-G0w™ ΑΌVˍy©=‘0£κ™†ύ%F2^ΦP&oŒ<Ωv8n2Ύ8θi&ήtώωηΛΕ_l_όβeΓ† ²bŊx/;Ÿ ξΞCΓ™gATξάηγ.άf9~ΞD˜Ÿ'ΛΞ»pΒΐ‰{[»μΨ³O^Τ01b“qVΛω—j‹ €μό;wΚe—_nC‡€‘ιhpλ•‘·ažXς%2―ύŠ{,|sU~‡B\@κ;¬@¦α4—O(bOyΓήζŽ.ζu‚LSΟ3τ@™ΈΦ@—Δε~―BΈ Φ“MιE@ευzέσΐπfM½-+…Ί⯏YΟ4 Kͺ?sE~·΄΄Θ±£‡₯lρωXνA7Ά―΄Hb– ςna₯€HΉoί^ΉτK¬ώ£α=zLΆξά#—¨–*μf₯WΗlΒ1J™ύ0ΛdΩ•0ς―©)•Ÿί&­'NX‘³C3‡ΚpΚoŽ™Ζ»δ΅“œώευ#ΜΜπHxL₯'ΖΌ‰=an{aΫγ1›βGΐτ=νžρ_–39ΉfQˆΏžΰ -8hg3¬—ύΨM@Ι4ZΤ¨@¨Δš€’ib_iΟ(:ŽΖυΌζΟΖO}χŽŽvx›ι‡ξLΒ;[•qεzeΞ σΕ㙦ύiΪ]ημΧΤ"0ο?ΗΔ εΉœ$iN­Ξω'φ8΅*-Μ‡§_―œ8yRzA΄φΓλ—MιAΐ¬±hΒQncΌΦ ^θΕ&WRΒdCaΛΏYz»ζζfΉξΊλδ»ίύ|κSŸ2ξNW―^mΚP‘6kYqd`Yƒ²oο>ΩΊu«μέ»ΧΔΞ₯Μψ”Υ‹ͺ…;ΆΧ­['K—. ³¨Sω q<¦Ν’T0ΙΨv·;Φ'Φ$Ϊ(™ΖΦeΰΪK¦!ΐvd”JJ0|·)σ0υ9Q—ά9ΚέkH‘|aΨBz9sηΛό·žω τ]έΖ#βA’"Ι4Lšgζ’μΩD ^ΔtΚφ#'δͺΣΦJάZR­JNڌKMIά]zdΖ+ I¦Άξ©•Ωσr,Π±!VnΝλΌ=žξΆά†ζ ‘Γ‰ευ?Κo―Χ›“mžςΊdšΒκfC¦αμΜ„ΰ5%9‰’\-Ω«X]έ!β5‘>’M• GJAθθnο3£-jσƒuC¦Aό-ξžrκ¨=q y¬C&«ϋRς[~wtvH7ϊˆ1ΜxΚo†r.,,Μ ωέΪzR:ڎHΥͺ:}hg‘Fϊ”@.„dšό‚"xΓ9d0ε8hΣ$ڏwΓSφΨΨ¨ ©κg“h²yςͺΉ}γμΙ²ΗβR #lάn„ύ:γΜ3­L™¬F~;dšB΄yΤ‘–ω]Ο4τ*F;°MdpΫCXΗ.»`?3šΔˆšŒύhς§ΆŸpŒφaŽ0Ϊ Πγ“v]Λ°@ME€ν‰}žhTuη'7όΩΎ9«XΏΒ:8ϊeλΙ©6p―Ρ†B<Γωb”Ηٞ:::wυ8Ω›2œ|ExI),3lθIά¦τ @ς9Γnq£=Σ0™α#σUtωE%>yα[ΉΆΓΉ€M©E@ηPԍΈHσqv£|ΤkΟδ8Ÿ­?“"Σ$²X ΚΥͺU«Lˆ§ο}ο{†δB2M*ŸiǎςΛ_ώRn½υVγŽ+rΗ VξƍΝ³\uΥU†ˆ‘ G*±>eΡΘΖX…YΧZg¬O*\P±Ι"*΄ΉWΆyΜ¦ΜD@εEGG‡όε/‘»ξΊK~ψΓ†_ζoφoεΥ―~΅Όξu―“ΖΖΖ°| gΘς/:Q₯εw.&ι؝ε―>ο―§m‘ήφb'ρŠ€ΩΑΚΦP;e΅U€ˆ†ϋ2Ώ†λγǏmσώr9pΓ™t ­[;V€Ά!(݈©|ο½χʟόgωρ&"\yε•F~S§onnNνΝpiŠ ‰}½]RTWλ’γ™fav)Λgj― ψΥ2φΡX ,Ήœ cRΝQeu¦+œμμν— w¦Ϊ4/°?  ΓYΙ4˜GbΌ’pŒ! θ]h‹ξω¨@›ζŒ€Κ)Ψ|πA£όη?7db~Ωe— e8εχš5kζ|ΏSQ€Ύ#ΙZG‘ΒβJa€οxMίSεyJΜύ¨χuvvaQ€&ηζ0³‘ΛzΩ·Ώψ!|#$*-Fά7©²yΆλ“;Ÿ' ΈΧ"„όΫ±c§±?ZoŸΙ!©Wqή4Π? > Υ7!ΒižρψΰQ84Š15¨Ων§E a¨ΓρOηq”νJ°MΈ°Ή€Ρ―˜ρ;Ω’i<„Β€θ‰AˆxqCΏ+–ΡKΘ­£Δƒ}Аi&Φ€8†{`³ΤΎiρŠέ&΄=΅··ΛŸοϊ³άυ绌 E―Έϊ꫍ ε΅―}­,^Ό8k۟β000]ΉC Kͺ α]ηζŠ‡σ ω•]Ρ[ ο twχ˜ΓΆME)Ώ¨Ÿ³mcn€η’IΗ,^λŒς³ϊ`χαvSFΜ μ‰9# zQ,2 oΐMILΜ›+))2M2ΰpηu5„1ΈrbJ…@RΑψτΣOˍ7ή(O=υ”H€Y΄h‘ρDγσωŒΫΟ'+™i?&ΤoϋΫε_υ_εŸωŸ₯ΎΎ>kσΒYπΙ4lC4†j»aέΣ0ΑsLΪ²ΰuν+œbΨΖάmcΫΧ)”$o―υvnΆjπ+_ωŠ)ιϊλ―—††’ ³Ώϋέοδž{ξ‘Ο|ζ3fάΠλ’ΌmF]FΓ‡*I\|εδ”νί¦τ!088({φμ•Ε%ˆχΚXγΈ—£ƒ&|S^˝Ά ₯~Ω|€UN`Q%S₯~ωyΌ€γ(b%ΝλΌ=ž*‡9ΡώΪΧΎ&Ÿώτ§MΧ^{­ρ0ΩΣΣ#ψΓδOϊ“όώχΏ—[nΉEΦ―_Ÿγυ„Όa7φΰ@vj{AΐŽν|‡‚F¦«q61ΤmξdΰΨΩƒω§ΏΌ4ΊgˆΪJ@¦ιΐ’ Ι6ΝΤi†°λiŠχΥ1υΡ.^x7¦ώ3ŠEK¦™sΕ¨όξνν•oϋΫΖξΒBiƒijj2΄ξΏ~ωπ‡?,·ί~»|ϋϋΆ\pώ+ΏI’‡˜@y”2?ή4Ά0gpΩx©‹@ήj₯νδ9qβΈ!ΣΜ½μμ(AΫ½@ϊ'Γ<±­Yy?/χ/ΕζΘ‘#ςωΟ^ΎωΝošΣ7άpƒΤΥΥΒ·Ύυ-ωΝo~#o{ΫΫδ“Ÿό€Π‚^η.+[ΎΒ{Γ<ω‹+¨85:Φ»εAχϊJŒώFg=?ΖB*ργΪΖΊΊΊ€ ϊqM Π±›$PT>„Νφΐξ2§] \i³&Š€³N„.Π'΅£©₯’ NR'Θ•”™†?ή€…δ–C‡™ΛR΅SRΛ>zτ¨|όγ‡γ°TTT˜έMozΣ›ΜB)]ζQ9aΰ3Π{Νm·έfŽqa΅ͺͺJ>ψ‘†pΌοdσΝ?$D‘4ΓHχ‡ J¦™§²w΄X:””ΏψΕ/ ‘†ήgnΎωfYΞ9R oWdD“Xσε/Ω0υΙΚη‚m.…ΙαD•ŠˆW"γόBo ρω†°Ψ½ΰ!©*ς!F,Œ*Xt‰₯œΞφόΤΘ`GF¨'‡«K†Q°)=ΈυhwΠ>νœ=–8Δ“²θŽ;ξ0rω₯/}©|μc“ /ΌP€1€Ρ(τΥ―~Υ„’%Y’ΤνuŽψ]3㠝Ξr γ΄,»¨ς@(πϊΝΈΖE%΁²‡…T[œ£Œ‚ˆQCΉk-&όˆ”Υ$ΣΑŸραώAΣ~Γ'ν—΄"06ƒ<Ι4ΐ>š§¨ΐη1‘BΜ"žΖφΤT ‰κάΐtτnλ_ώς—ωM]σΐBc>Ι’_ωΏbdy¦nr Κ± Ο4ˆ;Γ”c–BΑοOQ•΄Ά<έΟΊ’UΘΈ{A¦ΉxΒ‚cρŒ—Ρ2¦θuyή£#+Šd+άΟ·΄6zIŠn‘“Ε„2+ˆ1֏?ΖhοͺH†}ς•xq>hφi7΄ς;'›Ιœ^ΪM aAlC§lŠ1γ|Cšω=—Σb#3σ™dm>3—š=gU~“Lƒy.Ι3τ⌠bB’%ΣΜ^לλςӟώΤiήπ†7ςϊΩgŸm";°―^wέuς₯/}ΙΪiηšjiii֎Ÿ|ηn„mς–2z,=έθqΠ―ύ%•&?ηχ”qΪ.gGίζˆnΆθκξ•Ζbxμ’Λ©8Υ?ζ/… ½“XS΄ω}œEΪl3 `ΖtΘ†ΙžœΩš^ώmτr”‘KγMv]6xHœ‘JΧΎά9ΚDoLFsIΪYΈ«ιό£pŸWώ½β―3Ξ8CV\)MMM†eI#Ξ[ής3˜ΠπeKqη™ΝϘǘλσΜε]μ΅3#ΐΑK'ƚ“υE’”*οzά~ZζŠ@4EIεΝ\ΛΆΧΟ*ΣI΄€Χ¦w½λ]rρΕK bY§\lδΨΐγŒϋυ―]ύτΣ…„κό?ϊяdίΎ}&―^o~dρ?4 Κ8Άj“’—ο-4€ξδ³i~ ±f/3IgΌl·<_‚ω1λΞ¦ωA€aB‘a³#Ψ0W]ƒΦ ‰64<3Μ“MsC€ς—m^Ah£azϋίo6<•••™tœΧŸvΪiF&A’αŸžώωΉέψ^Νώ|²υBΞ`~™μ(vΣeAτGd+œQΣ‹r•?i Λ₯§λdؘ%cN’W˜έΗNJ-Θ$Έh.έišό¦Gb}2' ύ ₯>9Ρή!­ΑjΣάΕ¦Ί ϊΧD¨>-Ž`k†ι#Ή˜€›,Ι @ϋI·:γ§υ΄IcQgβB&Χ|ˆ[έ6:f<Κ…FΚ2z§ ' ύ–΄FcΚ—1θ’τ*ι^œZQά“TCO5”χΦ^9πνŸtf@ολLο|η;ε’‹.’l&e_eΤ‡sΟ=ΧΨPΈ¦F‡΄™3ιυζGύCyήΣέ…GEΐ€~%’λuFσƒ]Ε[žή3d ζUϊ₯»―φp/ϊ΄cϚP5αΪ q?:Φq J =f ²2ԍMjΏŒ2ό%ΚΔ?αϊqίI¦a5q|sϜάΩ²ν{žiΨH·nέj”$*M±-3/]ή>χάsΖΐMwbK—.5—ΉΙς9PyΫΌy³)ŽD™k―½Φ :°2&½†ξΛhΠΩ΅k—ΖN^ρ„ƒˆM z α_d{cδq›,©F€²ΓέήT~€ϊ>ΆΌτ! uFrwΖ^½ZΞ<σLsCŽ”ϊΩάάlH6ϋΏkΘΜδ?χ”§ΎdbΐΙ½Ύo.1ŠOϊtkIƒSi!'΄±Ÿ‚ηhn‘rJΒLμ„]™˜P4”Γmk‡ρβVR02Lλ5φ΅φL<Η™°TyOY6Ομ(ž”ίΏϊΥ―Μ αΔ€r[?—/_.ηŸΎlΫΆ-ηδχππn±S›qΠFσ |†4‚ΑΘ¦ωE`uΑv«dΚoώEΚoΚςj,όqΑAΫωό>iξݍ8 $=ϊNJqδΊSOΗ“-Γ<Ω”ŠοŽίίa {ρ‹_lτn꘴ΉhΫoll” .Έ@όqikkK͍OA)΄y JΐS„ΫH?εΡ(ςΈέ/ ’•½‰<9SBώߍ^JPΒL΄gpΒ―Ž;žiΨ―bt«hΧfκ1ΞτγJ*9ΠξsŸ“xΐx˜‘r-1?'=d7r§ήΊuλdηΝςύο_hτfJE#§‘¦₯₯Ε”wΩe—…‰4\$LΌŸ‹©ΉΉYΎϊj3hμίΏί(} ₯οy­ύ}j`}ςOλ‡Ÿϊ=Z]ŸΪ§΅wΟt΄mi;ΛτχΙΥηΧ1fŊf"ΑρJ'ϋzN?©\sS,]“Η‡GΗe4 „ m/ΚΤlΈ Άa£{v./±ρΔ¦#ΐώ‘}ƒίU/qίFeŠϋ˜ύž8Š#η >ϊ¨1¦©gK=§Ÿ”ίjΌuΛoήΥ‘oŽξ―OΑcΊ˜«Η2νΣύξω0ψθ"lb€`gφˆ]Pš·*ΥΊ œ‰²˜ ―Γ e ŽŒ œ†ωΝ–\Μ₯A¨ΔKC°³«2²ΝΞΫƒηЍΨη9ζϋ ShbPG‡@&3ίσa‘·ϊOj…φ †lϊύΏ‡ρΊOHšaŠœ·S7Wωy.–ό¦‹(©yβΉ—ίί‡B`ldˆ'κiS‡θ7@>σŽ£ƒ“ Α§–ρ1zFπΞ^ΐ„:8βNQq•ΑŸΟΊ’'¦lƒ΄[ž<Ω&u₯EβOΠ+ εt/τξaoπΏτγ;Γ²Ζ•X7( :~9ˆ<$Σ°Ϋ `q‘75ϋκ‚Γ†‡B γGΈπ§ρQ€1•„IλYΜ@bIφQʌ²2s5Η#ώζq’ilšŽ@GG‡x=Θó—³uA~ϊΫ'~„λb$‡xA΄εΐBύŒd\›¦#ΐφDΌθ‰†’ŸqEO5<Ξ0P6MG@upn$ύλ_jτΜHΈΪ9wUx<:8οΖ|Τ³2&Mθ#°qχχχH969v“ήμ€|O‘ ˜1 cή5ƒ”„Έj¨+³k2_±Α€Άo¦!ΪΓGƌݜ2@§τ%˜σ°Žh ΄)=¨gš‰}{QoΒ9ν(Ή’β&Σ( štF·½ό‹7Ρ₯ο­·ή*ozΣ›ΒFTΘΗ[F¬|ά­ΐΔ{0ΝT.Οι{Ι’%&?δŒ ΜSηΦ?¬7­;χ›λqχ1ϋέ"0W(2NIœλKgαυ”L$J2΄“;ι9=FΧςχή{―p\¨ͺͺ2‡΅DζΥkb$ίs š©Ύ'σΟ¦Τ# Ә2 @!fl¦‘p³ε`πΠΑΩΫ’J_žά{`ΐl\^UΡξ˜x Δa§SvˆΩ EκλΞ]"λ‘FYΉΞψLZΏξΌφ{r(–Δ–n‰έIΟι±^x!μΖΨ-Ώ™O4―ϋ3ŒγœΓ¨WŠkFGƒ95™uΧι©όnL:)‹C yυŠ„nς μϊ¦†c—βœ‘‘ΏΊΔ0½F_T/ΛΪff»ήžŸDΐ„κYŒαQΜτ~’k˜ŒΉάTζΕθλΕ\—vΞ«lšŽ€κ•••rι₯—NΙ ηt uΛ–-ςΘ#ΘΪ΅kΓ6p½€y5ΏΣO―7σtMzΨꏆΛΰͺκΉ?Ω°™žιq’tέΨ€ξϋ(ƐxdV »F·fιΤΑΓώ}ΧΎ~lHΚ“γ£rמ>ΉΌΉuz6uΤy^S"ue(eSzp<ΣΔ°s 9ΑΚ@Κ₯Ύ7™F…( Οώπ‡Νξ*˜³-ΈqX^^n<Σ455…¨΅ΌΉV7Λ^Ά|™)fϞ=αSUθ"ΛΧγTώφξέkN3τ=6-\Μn|<^ͺΪΝΒ}Sϋd Μπs6·žΧ>Γμh=ΊeˆŽτNφ½ο}ΟςΦ·ΎU–-sΖNhiœή·oŸ1θ]xΗBŽ9Ω–ˆb•mήgμyκ4H䳑dWې£—δŽΐθˆL=ΚvF’S!BΈQ.«=ij©Ώˆ6 υΟ·ΙΣ'‡₯άλθΰ;»F€κθ Όf΅:yμ:q—FΫr‘Χ/έ ΈBήΫ”<† ωl φQπΟΨά€°PδAςoj―°­Ω ήSΫΑlZa2 ζMhO^ΘzHΛ8^ ϊΌ³₯„stŒ-(ίϊΦ·δͺ«’Λ.»ΜΈξΥΕH-ƒωfJZ4Ίός—Ώ4 άξόμsΜ©{b[`ΰ„‚»ν_ϊρΗ7D_.†Y]$½mΔ-›ž{ξ9Ήε–[δ'?ω‰444BΝ«_ύjΣ§άωψDΤοω-);R·Œ–w‘γΌ ŠJψρθ•ŠΗςΐ¦ωE@Ϋ‘cV˜zo=ΖZamqw%λI―ΡΟ©WΩ_©D@ΙρĚr"œτ«9N”α3φKŠPΉLμwξάi πίωΞwLψя~dΌsή₯ωτΆt9ιv^Ο•––š― ₯ο˜Εjjπhoβϋ’ΜΟ9Ι¦MO―'^μΈ&#ώvȍ#πΰ5xΖ{ο\ΘG[θ/”6ΗΠ›γ\GE4ίpr} s1²Ν!‚‡°ΩΎmJ.Ί8ϊϋΨΤrL_aύΊΖΤ©9μ/‹ΐΜΠφΗΠή^Ÿ“mxZΑœKs³„ίMΕNΗ\u <ΡωM‹dG{PJΓty…_xb―!Σ¬^½zκ…9ώ«――OφξΩ+Սβ)‚79¬ΐύYY]‰ά|Τxω!™Ζ¦I¨·χa―}ΡRΉπ> Δ C’κυ“WΨoΡpλ֏>ϊ¨!όιO2θαχ’K.‰v™¬]»ΦόE;ΙπOL›z’<ηsκ<ƒ 6‹™ψJT, Ο{ͺΣΩ¨3·=‘$KJ†•ϊωŒυ1QΎΆ4§ζθ© δ ΘΚΠq)ΙG±—Ε@€²ƒφRΞΜ~έ(ωX/zΞ†yŠPδ‘Η{LώνίώMxΰsκ oxC˜Lc&:8J·$Ϋ<ϋμ³Ζ w^yε• 5tv* >Ζ‘|ίϋήωαίτ~CVecc£©l.’ΖJgu–!φtwwΛ{ήσγ>–‘=Vφψό"ΐI QΪΆxw Lvl;Α™ίΊΘφ»qΐ“i0±Ξ%WeΩ\·:‰ ΜΈηž{δ#ωˆlΩ²Ε„j‘wƒK/½ΤΌΎζscΑcξΔί”EΩ"{(Kέ (ίΝύΫύξφϋάΠφEςq‚U ’yMή`j³3Η£rςs<ΔdbqgkŠŠ Ιx² ϋ-Υ°Ή8ΒDΒΆ¦HY‘ΗνηάΠ~ΓR8οψχw‘1ˆ^()ΏIxdrηΣ;F«cώlΨmκ†θ² $όGwΔ=έlDrιΣΪ±©P··zΕ4pRr€:zΉΟ ›ˆcŸ©6xΓ)­qB'Yžϊ΅έHš’*‰Zˆ!Θ°r8‡Š• —ΩOΈA’c.τ™„<Σ¨€έΊu«Όγο0”€‹4‚Έ…+ΩΘΧ\s!Τ0δk^σcηξ|γa1j54Ce7NΟ5όΣΔέJ¬@7[•Ώωt Μ΄dΙσωμζ ύgA!@·ˆlC‘†Ά7i†Ύ½ ^Ζ>Μ‚G€Κ&εΫ—[†,ψ·»(GθJώ¦ΊIευ―½όΗό‡œ{ξΉζ:ΝYH,γ~δρΘλ2ε7ίCΗG>³Ž·™ςό™ψœ%#cB\ŸbNόuP‡?eŒφee₯ΨYηKΌ{Ε¬Έe„~œ0πΈM©C@qζΈό›ίόF>όαƒγΛ_ώr¦ο’‹.27Σ|‘wžINΟt.²œ…φ[ίΧλυΑOΖ(„Ωr=πP`η6σ_cΜ‰™F!’I=ƝTCΨ‘Κ9M.ζΏ&bίQν Σrθu£υ4-=0*§hβ.Ψ›oΎYΆmΫ&ηwžp7,mB”Γš/ςΡd4σ:I+-ςͺS󻀀7Ζ³;Ÿ <†σώΈšο¦―7Ϋυš²8Ψ#ΐ€ξ ·ΩŠΛ¦σϊώ”±UUU58 ςϊΔ·yi:γ†ΐs²ή ΛΒl}ΆlΒ{>ή…αΊΌ ӌMlB06}Τ₯©Jτ›1x²ή™ζ£&²σ£ϊϊϋ°ΠŠω^ˆa,ΑCM?ŽιΊC¬q*;‰ώVŠ#lΫ±Cšk+€ΚΟ0Ίτ RU”'kWΛOm’+_s₯ΤΤΤΖί£ί!ϋŽͺΜ§'ŸΎž>Y‚ςτHcˆ`Ύ2,ZZ-;wμ4Θ9f)ΞΩ‡FboΔuφM/Φρ(λιΩΣΛ±Ώƒ°λr³΅M±ΠvDιΝχ]οz—Ιό–·ΌΕ8F8ϋμ³ΝoΝY’ΆέΘγό=ΣΉhωOι± στβ’2θ!θΪhMy±<βx₯…>ο/+†M%VΎSϊVsΪͺJα=’6’ΟΡΝY₯Ό†ή‹°IΥΪ½Χ(ƒ‰χΤ0O“χΣͺΛ‘} yuύ4–\™Ό2³ΏMeΐΜς.˜T&ϋΫίΚή½{M₯ο~χ»BΧκ$Ή0©Q‹q$ΟΠHςΤSOwμτTσ?σ?&€FmmbŠΥβΕ‹‡ήƒΟΑŠ‰+‘Φi83f͈Kτ}XΆM .‚3”Weee˜«8w 0Ε4.μW³O·ΐ lQ2 t6e.:ˆ³I€yη;ίi^†žΞΈ(KχρξρDσΟψΖΤ&²(qάζ˜Νwη«|Ν’Χ\P―Β0A υΤ?D³ΐά›$Ι4­½ƒR…q²ήi˜l=Rς[NhξcVQTRϋΙ6LœI€yλ[ίj ghُ~τ£Ζ³»ψ=ΧΪΌΔ9†ψ w‚<ΚpˆΈ³Φγς˜”Ϊ±₯E" νΞ!1b1ߜ’ΒΥ$ύπ"ΖΊΣyσ”<φGΪˆNsΒνfͺ³΄=Mφ¬ςψό£όΝίόyα7ΏωΝΖ;Νgœa~« Χ~4+*:ΈYLŸ5σόeL„Ε&vrczŒώ‡<\οΓΘδ–ΰ@‡”••  βΖ1ώGΛȜādš:ΨtφΚ ΜσΣΡΩ6iδο†ŽοET”W mέ$Ρ”&ϊΗJ_‘Χ„FqΌΣπλ“:"Βά’ ŠΞvCHΫKΜG_o/Ό ž )hTls Ν£d Σ$G•M/μ”7œ^oΘ4$+’HNbΝrΏ<ΆwŸ>tؐi&―Κέo΄CΎπΒ X»Ο“@½?Ό(9‚§iͺ—VΘs›·ΘΙΆ“†š»HM}sσŒ;†ΏδΈJYOΙ‡q544aΨ¦^a)ͺop]ƒαTίύξw›SόΰεCϊp ”y4i~ύMŸj£#α’$@2ΝΌΣ€0ιP΄& χK“L”@I½Ρ%έ§μχΤ ΰ+τIΌNqs)7A<Ζ—&š-υμή!x,.6ϊ_|Ϋ\‰"`ΦCαyFC9E½š!ζ™WΙ4QσeΡΑΈΙ4*\[[[ε‘G1άxγςφ·Ώ}FαRRR"7n4‹Σΐ5;3ΉˆI2M+Ά}»ΐ—ήΆ_^V.₯ΠkΊƒŽλyΊFžœΞ&xoL<ΖPoœˆtbχ-Ϋx€·K΄Ωc ΐώ3‘x~Ζ<1/Ά'fB€!cI€aΫώ?οό?ςΑηƒΖΔkr]~ϋ±£‡;§ς`τA ”Ρ‘!ͺ₯ϊ±Mσ‹1g{u‡‘h6!c8ŠV„aέ©Ž’λνx~kΚήm>ΰXΘvύΔOΘυΧ_/τL»Γ¬655…![Ϊ~qQ1ή±2ΠΙ±λ5Ÿ‘ gΠΒ$ϋ$Κ±P?l x¦)IΆ ¬ΎzCŒυƒ#…7°h‚9ΕπΰvH[Hj+Κ…Ο`Srθ"η©>κ;ά9€nEŒ΅wQ֌cΑδ)Ο&w7{U"ΐϋΎ>i<ΤγΎ—u¬Ρϋν‚&ΝƒμXθψλxύά³Ο"ά<ύ#ΔS!^Ξ@Έ=Θ Ϋ\ι“χ†δΩηž“ ηn08.τχJΧσ©Δ Ϋ_Ψ.ε ₯R|4cΎΰρH Xφ=vXvοΪ-kN[ct'Ν“ΛŸ$ΣpqΦ[%e>9ΗτL322j<›φŸi°―²ύέqΗ†HΣΠΠ οyΟ{Œ œkjLڟ§]œ₯ΈvXVZ!½ Ι€™λ°7T‘˜ςΞφπϊ;24 2³ζ8ε΄ύ‘όώ")‡WΟ>θo£°ΔΝ¦qn?‚k:†0Aθ<„γbΚ΅6ν ‘ΎΉ i}β$L +ίξΫB0“,šKdgTrƒ0ΛwΖrά΅k—!ΆΠ# '-31τΓhp'ΣΑƒΝg" Λ£Ρ`"Χ”OOζe±l›6½Ψ)YOόΝ‘ρGΓ‘žφkΨ§Λ(W؞hŒα§abfΐsΫGœŽ€Κ ΊUύΒΎ€έ‰ufA–Fύ xιοο7;8QΣ?zΐ"ƒ?—’’iˆ8'2>ηV©zW?<Η,]²»VCƒ«TP“ΥDp©ρ„Π60"՘TTNξ€MΥσΪrT¦ΔΒcΆσ±³Η§# X«εWΏϊUγ™π΅―}­Όϋ†w"$ε7ε΅Κn~ζ’όV……~|8—qΌΣŒ†‚ ڐΣ›Tڏψαf¦†q'Z’¬f»¦ƒ„Ύ!’ibΡςΪcιB€΅%Ε8%§=lηΤ#Ή λίψ†τττΘe—]f ωFŽ&Ώ3Ωe?ΙquυKdx6 ξz₯b—΄fΘΜΉ?2Τ%ε5†¬ηU9•ž Χ,["G{2 —$ΣD—Π)„χ ‘hOPκU‡ΙΏ)ΌCΞU€0 †< 2γ™°"ΡΟ`–αΑ9o]ύη\ΣHΙ Βϋ~ooμ :Π3 ©! Υx=…fΌβM”Ψ•’f`!:/k;yR6mή,Λ*JdYΉΗι†x ϋfΨ[ ‚M=B£< ΒM[[»yS½6_;%Όoί>yώιmR·žJ€η ͺ"ΰ³tI‘ψKύ3»1 9ηφ†Lƒπ!μ"ߐkFFGd`δe›¦! ύΡD>ϋΩΟ ‰4χw'τμKb{,xΆ{αβζϋςŠJξ7ϊσ4ΰά _ φ―Ξόή}~Ot˜Q^ΐ搱P$Ζ•‘v° *A0ΧI¦±D긐K<„ξ(ˆ‹LΡΓ<™Κ0ςΩ„yšΰj$~£Μ»"nΟ4ϊj4|PΈβŠ+Β.θfZtΣsμ(βLτ$’hb9Z–Q<- VώΈ{l¬μρt! †8.l3.+ΓV¨bΐ{ς;Ι4f1…Ζ›,sE€νŠ“c‚p™νJε\ρZΘΧλ˜ρΠCjΛ–-“ˆοόΙO~μ>aʝψ›²ζ½ο}―pΡ6WR$™†m_±Λ ζϋ=i^Ή’Y^ؾ͐iκPΗΤ°’ΰΓPαFΝ}ΓR[UφΈ”`16ϋ,plΰίl}ƒ„L›ζŽ€βLo˜wήy§ Ι§ΔHΞE"½gQn„—†’ΧΏώυΣδϋܟha–ΰυzΔ_\jΒ; a#Υ°BΎXοTσ__$I²έ†°`λNJmδ“;XpΚΔ―-B!$…MσMρQ›3΄7L«Ή(ων‘ΈxςΙ'ε'?ω‰@sμΨ1CŒΤ…"•ρ,ˆ ενXt{Σ›ή4«ΧγΈn| 2ΡNΡ°x‰hΩ„ŽŽPOβ›Σσ(h­τ·΅!„xͺͺ1ή{Σs―Μ.΅^H›–-•};·Αθ^&₯ήHɜΪχ£‘¬οCψŽ£έƒtΪ i¨―77q·ωΤή5ϋKch\‘_‡ϋ&’YFj›52 ΙΕJ¦±Xg›HεrΑΎ§§[ς=X―( ·s,Xγ9]Χ.l›rnΛΩΊs·\±ΆNΚ  ‹1ž >‘Ϋ/υtNC™ά±e֍φζ¬]Bmœ³nΪ΄IŠΛόX\„f―*ΓΨHεΐiώυWω€fe₯lέ²UφμΩ#gžyf\6WYυUϋšC¦3δΚΨΘό<ζ‰ή€ˆ-“ζΟ*ζπ2Δƒφ¨ϋοΏ_6“ψψΆmΫδŸψDLxGG‡Ό}ο—W]ρͺ9άya_J=½’²J†χ2ϊ3ZNΜGΥ‘Α.©ωFΙ4ΆΕ„+©$v•#Dmg[ΌŸ8cH<±Φh6οΕΨS ›ΛaR™k~ΨR†‰‹ΰc‡yβœsIΌΙ[xΒd  ζζfaœL­5LΖl5EΓ 4ˈ7i‡ΰύλO'HΧ^{­,Z΄ΘC#ϋ3Ο<#ň•–HΕqα΄»»[ΌςJΉπΒ mη‹·Rζ9Ϋۚ†qΫ³mp±€ηΤ0§ημ§E QTΦπ“Š;eM@Ώ%Σ$ δΛΟ]τl°qγF!ᎲbηΝQYη$‹²ώ΅=,°ΧIΛγp,δ»σωΗο‘ ΥiΉqΚI1.B8€Υ«Vɞ“έ3썓Γν‘Όc…φXwPjΥbμ’dŠ²ΧāΐL:¦1\p{¬M)C€x9rD^ϊ—Ο4t΅N##eV€Aƒϊ =g°Ζ>–νIό=/\W:ρΏG0vαΕ)_¬ Ÿ@‚1H2!X„B0(Dz@Θ‡ό§WšA°λΕβσNΊzŸ§Ν½;ζΗΆ9` n"εJξ‘”š7V]‡ϊχ%—\bˆΤΗ©cS6EβLωM²$wΛκ8K9™/5O—ΪRτΉ3²±q©τέυ;©™ύ]ΧϊS{G- 叆d¨―U–,―“ΊΊ:=a?€ΦKyE…,™ζ‰M›₯cpT‘κ’ξ―Lύ#‰P—­hΑ•^Λlš ^Œ—EΧΥΫΧςπDIθPψ`Ό‡‡rŒ»΄Zέg.(ηξ΅άΌΩίΧ#ΎΚeΠΝΨΘͺΟγυ›πO£˜`μΚΥ€γ17£=’,T\9­+…Pt{ »"ΰ7JŠ<ω²ηΚ@~γ†Ά3ΟύθGΝυό‡nΚH¦αy2}ιi Ω΄nέ:C¦Iφz{]z’γǏ/4Ϊ¦τŽ\τ₯ i«H)"φsP–Ρ˜ΛΕ:~ZΟ4sEτΤ^Oύ?ψAωΐ>~Θ±*|_(cΈ 6SwώLώο¨dΆ}ώρύΥ©y2ω=Ϊ³«^Σ°x±Tb@ƒϋЈρŸi„Šk1ν•:ƒ£60,/:c‰!MΛ`ΜΦ›Φ]¬Βhc²)5PέxγrΓ 7„ œI&±~(σ©ζJ’gš@ \B#Γ0ό8F0‹lš<Π+°;jυΰ#™&βψ›;―@¦©DH>ώlš?bymj@IDAT\Ά;˜,’„ηο52ζN”ΥΧbσΣΫίώφπ3Ο&Ώ=g\gš)oΈΐς…cǝϊ†zθΕ‘nΙΓPΊΣψHŸ„‚έR_Ώ>Ό33“p›κ£±bΕJam΄τŒΘj,ςΊΊ|ʁesAωD?τςώ ¬]³Φ΄εΩτΗ”?H–Hݎ2‡Ϋ’‹.•Hkσ(ΌΩvΨ 7AνR?ψφrμΎ5;©bΝkNΝΣg]g“ ¬ͺΩςd?J©ΓDdx¦γΏ€±^b–Θ‘“GΕ_•.ΜΒ(–•`‘o»`Έ”ζ¦fSΎΥύ’Γ\SS+ηž}¦lΩςŒt²Μ灾–˜Ύ½δˆ£(’ηΰΈl?ή%]x‘Τ€`eSς¨Lΰ&H. υΑσΊ”Σ£π/0…«z΅ς#¬ŒMdθ뙆s<Βπ˜ΦE2MΘ5τLΣ“3‹CΡ°ΣqεθΡ£ςΨcOΘͺΒ8Ρ‹s΄άΞ1zb\Qα‘εΥeς(’Όβ―@κΕ9gŸΰ¦ς͏=#k66KQ΅oF@μΒψQŽK>ΘIεM)x’8?.\paNo€MlpŘκC˜VΆ7Υ¦HΓ@`£Δξƒ<“ˆώΝό:ής{Ά%•c΄Ώ•Ο3^τ;Ξύ¦&φGi‚ΖΎbuΈ©θ€ςΫ[ œuŒ HB'i} €Σ rumm%62YοAρ`—Lυ4C’L¬AŸKm$ΪΠC$]PvηBJˆL£€¬‡˜ψΗςŽwΌ;=V„έ―Ώς•―4Μ=.δpzΫΆm&ώ5σμ?°ί„bzύλ_o„΄ 3-sΆOξlZΉrεŒΩaΐΠΑ3Ζ8™ΝƒGŒW^Π‡΅>g”˜Ώ£±άΨή¨άσΏΫd˜  )Ηέ)lK$ΤtΒk„M™€Κ’Μ~‹τ>=I‰Δ‰ΚΗQ‹YϊπVlύώ"£CέύΰCr΄/$p‰LCAΌ‰“ ξ€ν“p7\Ι7/3’x―·ωC€:ϋ‡©?ΙEAέ6šž‘ΝώLν/ \’SY•L32άφLSŠPC6Ν?;Λ·ϋΠP‹z0ŒOZΚΝΓpCw_υα|iY Œy6Χ|Φ¬²„Φ ΤY"γπ|>¦ήkVά3υΕ\Ο­οX WγK—ΙώƒΛΨp7Η\©ϊj¬–&πΡ&Υ5 πΤ₯ͺπ¬*Gλ…ž _τ’sεΧwί/ΗϋjdI)Όψ$ oΗ e=5΄τ†€;’uπΖP6aδΧg‰·,›o*_I@ΒW„JdbRl€ J2Mρʚ¨ΩfΙI΄_φ7N2 γŒΗ9ŽΕ  2MOOwΞ,EkŠΡ¦M›δ™ν;ε-η.Ε nάAŒ!GI―**sκςλ-;eλΦη ™&ZωΩvLεΧΓ|π)*υKYS Έ΅π¦ͺ—+€Ο΄@Kj ₯nM΅<ωψSfSϊgœaZΣΙΒŠ7ηχΓ[|DΚ44…ŒmΞ  a§Χπl΅" α˜Σ+εR»‰(FZ)†ί„Hu¨4n!ζ΅ρΡ ΞχJΥEf^oΩ6_όh?――«3ŒgώΈ ι ŽItΏυ―[€u &Ϋήγ‚.LZ?ŒA‚LȞŽγrt£n†r‡L“󀍓 r2σΦ·ΎUώτ§?M‰yχέwΛ/~ρ ωΩΟ~&wήy§!juoEH¨|ΰfB©£bζuϊ§™ψ›χknn–o~σ›α…oΝΟ§–e?¬Ώ––L`2Mδς§β½YΣΓ#㲫-( p]ζYg™bυRq\-ƒ:ΆγΝ Ί6Θ4fŠjΊ\ΠΓ3!Ϋ ”Mdθλο3›!ς<τnΔ5lžΐχx¦ιλλ5‹IΙ”›ιΧ¨μ’§ψ‡~X*rZ₯W|ΰ#ΉW΄ΔnΙώι…Ό¦Ϊ O6₯rί}χKΒe±k™ΡΝ¦c‡’Ώ>πΤ¬ͺ–ΕEζ½ΓΪ@ψΛΔpΉΔπHυšr9y΄Mžyζ™π†βlΒ%ήw!™†„€ζV$ΣPΆΕΝd₯˜fĜ‹)WΪUΌψΩ|Ρ¨ι½’’ d™Ί΅C¦iQΖC½mΣ/ ‹—έ"zIφh*¨¬ͺ”Š@ θ#2# mΫ1†ΧνθrΧ Λβ†cKw΄_Sˆ‰4†LƒAŸb82™C”Ν`–Π»ZρψF°vŸ )aΟ4ͺΡXπκWΏZΞ<σLω§ϊ'yα…δΔ‰&΄γd‡‚Š…sΞ9ΗμΎζ$‡ƒΛH&Ή―Σr¨Ψ=χάs¦ΈηŸ~Š°sηOζ~φšS‡€Φ/ 2τLγρqrLξ—Ϋe–³ςΰ΅΄VN?nH7ŒΗͺמΊ§·wΞd虆^WΥ–IOΎ΄AΖX―G™\£φΩγA€γ³&χw=f?ΣƒCX65"$@O<Μ”HubΒ˜šDMj :ΥρΎ6„ήϋή…Ικ?†”ώCέBΙ4Ρ &ζ+ΖΔΧ&‹@šΠ>Nϋ}h¨Ο σ„ ­Κ4?‚-~­ ŽτLΣΣ” ˆ·§aΘρL3²Νˆ,E©g½ήΊPHΞN±Pžή>Η©A@ν+aϋ’)x4Ψ&ž1σ°κ˜ςΘpτL³bεUᝳV–DΪιΟΛ—/—³O?]ž|z“΄!”F‘žR™TΖν•ύ²~ΓΉαΠφ©ΌO.—Ε0Nlγ£πΘN~»“½>oΨ3Mψœύ’2(ߘ’ɘ™Ξ₯μ\P?νIΙ/πΛh^wcΓΖ:―_‚ FφφΒKΦ6r5νή½[ž~~›\άT%5%Ž­b&0 vi£h,υΘΊΪ€<…kwμΨ.ηŸΑLWeΕ9φφ‰gž}FϊΊϋ]ΡŠ p ―7›z‰k ~i\[/ά€l|ωFi\˜“λ$Στ’_–`n₯›>’kJ1εyζΣM$YΡ€μK€ΪM`{=±${$α##Γ±Ž‡ϊ`Wι…7­I2M΄±/­š#…Σ²lqƒ΄w—ts.I8κFTTp3iμ-΄₯3’„ΞΑ’^d&€ρL‚Œ‡ Z€΅jpΠ„yςΘ Θ£#Ήa Oj© k„‘•ψwΕWζ,ΑfCζNI†Hq t4p’-:::Lγ8 »?μ@št?YΆ΅΅Ι‘ƒ€€1Ve»n1ΉQ£&ηJa ^ŽνT>,$ΣΨdH•U”'}Α 4—{₯΅­@ZOΆ™TB,Tν5™‚€hH”΅m}ώj­’’BΞ‡λωŸόκW1X!5Ε ΣΔq{cBKβΝΑ!©«Νi§™+Σ‘kΕρHYŸ…±Ή5δd¬—%ΩΖ†yŠ…Ž=ž(³‹a Αr=d–L“€g.“r—cgΌ±φ%4ŽpA—πχ Ό„ ΈYG¬;›ζΥυcίΥρ‚ϋΌ=c˜:Έ_Ά|•΄χžΌΗδω*`}€f)f.'φY”ξHί1Ψ½*€Ή©Ι’cƒeΖDž¦ΜέpξΉϋαGε@wH–—!Όν Χ%zŠeq1~χˆμ;Ω%7^xaΨ.9»άIτnΉ™Ÿ›5‰%Γ:Ή-ϊaΟ48oSκpΟ+ωsMn{Ώ;ŸžΟ„O’>θƒ)dGVc‚ΆVΰ-’`O0ΌaΨdΚ‘L‘§ž|ΜoO«ς‰ή»Ύ4– εqΪ* φam΅_ξΫuB6=΅IΦ―ί \ΜΞΦ€}€k›Ÿή,εu₯RΊ‹υ†Mο`λέ fD aƒl.“=’ν/l7dš\CΈΨƒ~Y 2 V lΪHQ.RΙ4±0΅Η-Š€φ!ΪΌ—`ύϊΡΗ5hDκ5Λδ'ΌKϋ;Ν:7ηκΪ―'3Ψo©B€6¬Fiž:΄ίΨH0μΜp>9Ω Ο4~_‘,‚H›‡ΙŠ”ŏkϊφˆ:2ύcœži‚ƒCΖ^žΎ'Z8%'eΕs “Γ 0Pω\±b…!ΠΈ ƒΟ>ϋ¬ υDο5MMMΖkŒϋΪTΐΐ…¨Ε‹›’ψ,A,‚ϋύΞή‹Ή'[ΡξΙF‘Β5Ϊy{lώΠvBΟCϋφΌ ΛΞzβ―΅‹ρo4&2OEzJΣg¦ xžρψ —Lxκ5:œC¦ιΗϊCεZR|ZZZδi„Z±( K° ζ^\‘μ›1ΗŁ9£‘^Α6ΛUsΒxμ²gΌ6ƒO8°_žyβYY}iΘ1XŒ'Σ2žΩςA@*o*‘‚ΗσMŸ»τ₯—ŠΏΠŸsvhφΛnτ»e˜Sq|e[ƒ©Ε|3M±ΗkΞ3ŸMx ά‘ΠΨΈTzΫΐΚpH”“Χ² ”d/ςJuuυδ)ϋ-₯¨ „Dχ††z9ΠΩ+ΰǘ>λF–i'§-₯c`XV.]ŒPηŽWφXΧΨγsC€dš!„ΣςΧ8ži’–Ζα § ΰ½f°0ζ)ΫΗyΖΝI() T ωΛ_ʍ7ή(7άpƒœ<ΙΈΠΨ‘C.σ0=ύτΣrυΥWΛuΧ]'ίϋήχ„ έTͺ΄œ²²2Ήζškδe/{™ΉΗ―ύk£μς9˜'Α»ΈSp¦?νΘζ‘ν? Φλpοή}P˜P…ΥpUˆΆεš<σIΩΦ<₯R±hΉμΪΉC’Η9μ΄AσΓώcˆN€aΒΨPRož,*σΛqkŽ=g 6›E 3P",?-™f~λp άχ?m₯μlAL8I˜P£b<ˆ3ΎΡ p€wDvτΙϊ³'=σY&ls<Σ `}‘ΧΌR.ΚΩ½{χΚS/μ’υ ₯R^X`αͺΣκg΄ϊζ9’+όω²fQ‘lίPφνΫ-kΦcϋ…Bρτ¬x =RΪX$ΐΜD ›ν-]`WΓγBs•<Ώe«?~|Ά+³κΌκΰΓΐ±½»€x¨χ˜U!Θ%όΗίΕ^t‚lC― 6YβA@Ϋ–£7Β3 οa}BΏ’P°[κ–Α¦b‰ρΰšlΦΧjΥΜ#B:½ΟD<€=„εd_P$PHφφφΊ8 ‚L3²1ΌyšEŠˆ‹0n‘"ži0Φυwυ[Ο4™ŸͺPΣΝ·Ώύm£<3ώπΑƒΓ™‘Jˆ\˜f’πήχΎΧ°ΨoΎωfoTΛ2’όG•ΩeΛ–ΙΫήφ6ξιέο~·ά{ο½rΑHSS“ΠsMΨ¨ŒŠfηs'.Œ0ΞRρLξ²νχδΠzΰ€ΝOo’ΚΪfμκ’‹δS+ΪF@o©# ԝΫL° δp·W‰‘Y»1ΡΫPW$•ώ<© ψδΰ–#rŒdšsΟ΅Y²λ™fώ«VuzΧ[{Ϊjyμ±Η€{¨ 1 Υϋ™¨Ι a²q¬oTΊCr:vΫRΧΡρ3φ•φL²P―₯Α†υFœ#S§δŸMyA€ΝΒΐM¦±αƒζω˜7αbwk«,Ž” σĐ%v}L Σq‚ΖχQ„ΖΡqΧ}#Νяh$CžΘ:sη΅ί-ρ Πά܏¦Ke¨ηW 56ΑhΟ₯³δ‘φ2Ψ*ΑΎvθ~gXΑ,ˆρ΄κmΟ>ϋlωωο {αfy<9&1ΉΎ·v) ΚλןcΩo8χαڏ|δΓrήyη›>7±rCkgg§ά}χές΅―}MήχΎχ™cπqυ@―;έέ]βυ—Κ8Ι4fŽyΝMΉžbΖb’k€Ω…_Šφ9n f΄%•₯―4>,˜…°I3|<‰£­Θ2„.ͺ**”gΰαζ%/y‰ρΟυ™”G13k›6KEcΉzŽν)Ός‘ψν-ρHB==yϋxίo‘¦εM™EJž•gZ»{ΔΏ΄Ρ!ΣLΐH8I¦)‚έ« d’n˜”άά’•θ$™ΊϊεξΟΘXωοgϊ): ~2ΔSƒCg΄)ύTVUJCeΉœ„·™‘1―—‡ΡǞ!x°9Φ= /ͺ«•ΒkΫ”>ƒΏrb܏R3³ς €pΣs¬ΧJΣχD §δ„Β<)…d²ΉxSSS#ϋΨΗ€ΞΌ• (~Ύρo4y~υ«_@Ώτ₯/·~$ΌΠyͺθ%(š¬rαqQΧ¦ωC@ΓτΕΊ#λΚx<±dšXΩγρ#ΐ^Λ–5ΙΞ½‡Δ―ω X!CPήζ”@ζ–ΡΑ“β…ΎΎrΥJSZͺljsz΄ ΉψΜ³Ξ’ §―‘珝σ ±yžN`NΆfhVfŠΆQ„”εK’S†ΐ‘QI;Œρ^ ς λŒDšq> ΡfΆΙ“ΉΚώ*SΓFO4LϊΠ‡ŒΝ\un`lnn6βk―½V~πƒΘΖ—o”eK—₯ΜΞΟ³Ξ5Ι4]π$²‹ͺ\aϋ’ΌvΘ4‘Γ@ε^˜'βΪΡΡ!O=ύŒ¬ͺ-—š"xŽ7 Ξ<_b%šKκiqY‘l߱Ӑ°θBΫX|%eNΘ±–γR{V₯–yΝ}\O―ƒΛχΐσΚ"ŸTΤ—Λ–-[δE/z‘™[d+fn|TΎp?ύΡψeΨ°i{b˜§"x΄θ@ž ϊ/“^g~Ψ,3 @[Imύb9ήή%Ec cτΖ ΠΒπ;ΨΧ&Υ‹Φΐ뉝«ΟcΚNUW/’jΪBϊΈ Γι·ΰ4ο„]ƒΓ& ν.LΆOΗ*G‚s9Ψ/ΦEI˜!ώΣŽ‘θ^ΰcŸύ&KΆΧIάaž8pSΆpP»σΞ; 8gœq†|ύλ_—λ―Ώ^*+xΠ ?—.]*―zի䳟ύ¬Όσο4ΧΠ$=Ω€21|Τ#<"χάs‰+I&žD"“>s<ΧΨ<ιG@λcǎΧΫ-ήΐbt\nΜIMdboFΗ-ƒk©BΩΆm«έ ‘ύ=+ls”q;wν‰Ζ#uΕΨ•‚TŠXκkλ+eΗΝmCˆΝŠ£ΝΉθβ+‰4–L3υht+άnνš5Ψ₯ε“# Ә_8ftUσΟΤη1£τ±Έ¦ΩΣΪ-§―ϋΩ{8ΉŽ*kόNOη4έ=9k”“eK6Ζ Αδ΄ όa–]vΙlϊΑώ—°| ,ϋaLΖk›`ο²`γˆ18+X²•s–FyrΤδ(}ηTOzZέ3έ3=#©»ΚΦtχ{υκ½wͺκΦ­[·Ξ](ω#4csš_©D€‹γtͺ€!?Vβ8f¦Αv “ 3ˆ7Έ™€νΟ8ΣΜ π1n₯i’Γ<ι¬}X΅eψZΦ“τyσ9=gφ=Ηs ͺΨ€(Š‘'^ݍΉΖό0Δ@@·/:Λ-\΄»[[ε,C=₯$4Τ‘.ιν8%σ-5z^JJΞ„Btέpΰ—_.'AE~²ƒΞΟz»ΰΤP`YϋΪεΝ7\/y¦^¦fœ«9We=2ŸNg1¦žΑ?›»™qΞ€Τ  u“SώΚ+―νώd~γ³Ε±’6η&[nhmmiMΝΜ@)ϊϋΒ ΅₯a ~3:ššiγ π 4Γ?F?˜2˜τuκGš©ΑFΓF8ΤϋμβΑCv½Θ^F€ψO§Θοϊ38m©ΊδT]}ΚΧ‚FοsΏhωΓ΅¨Σm§ΕWμV‹±K-HŽ‚5ϊe4+Ο;C ΄ΦΞ;… A™’4Ž /ηΗBΉvΞ›˜τhΝ_<%ϋ¦Id`¨ε<„κλiWΞ3a©6ζ©Ώ·]ESΡsuέ&“Ή‡Ι›8Τ™ƒˆ(ΣάΥv¦‰dFŠaνPΕƒφ&χaq Μ―1izθνλ•n°)gšH―Ζ·Ν ·Θ­BƒςtΊχ›€˜iκέ»wσ«rΉ »;˜¨TFƒΕcόG§2Ώ<χάsςΪk―©X™dIUΊύφΫeξάΉ£tͺ‰–ΛηeŒΤΕK'z‰Ι7ΝθvDΈ}ϋφΓΈ‰™nμΈ§•“ZευύάΓd9‚βςΛώύϋ€Ά;"ΚcΆΙsW˜o0ΊΝ‘’σΐƒrΫ₯Θk…‘F8K*nY°Z›šΜ‚΅i4i‹€6PQ ž³@ΪΎόEπb0ξΟ›U!5mυ1ΰ‘χˆΓFŒΙјUƒO}ƒΓ2{φl₯π5’υ°‹ΰΥζh4¦n2C™kΈ`k’A`& e8۞f©™Ιϋ›{C€»γως‚uB32 p=θT‘ηΜχιC€υ0³6 +έgp`Π8ΣL_5dDΙlk\`^Όx‘ φuΙ™ΎfΙ>]w P tJGσqYpΣu£NFχK O=η§3Ν?τ¨;=$ sν‚u^%—+%œK[₯¨­χA‡3Ν0l+šΞ­ϊ^Ι”iςŽϋ•ZLε$ˆΐzγWό³X°*ƜiόΝΩxh™RZZ*wί}·ψsόc6Ος<Ϋ8λEλ4άxpΙ€‘φΫΫ#m­5R0,†˜9‘Όlmβt‡› l¨ίeBυβΔ θDΓRθΙ³ƒ„«| d aβ_²tEgαo†z*ΛqΘ;NJ}}=³§e’ž©ή8Έ‚φπβ/(όΞψmα’$ς½@I π›Θ¦h‡=ΪrJέΑ:Υώ2ΙI“r„Ξ4^8Λ8€a!L\Hη˜Λφι@τΩgΞΉŒύΐ˜4.Z1|S0’Ύn8ΌŸΑF uΒ9—l’°‡ηδT;·@srJθϊΐ)&γΐ‰΅©4f‘”£8A΅££oXœ 9#Ξ4Fߎ‰ΨΤ†;…b™‚σ’Υ ύ›Β—γTt w±ΐαΦp)F;Ž…œ ₯sJΪ™†‘wΆlΩ"·άr‹\Ž‰(SΌΖ«;σ”••Ι΅Χ^«œiκΰμΐy^˜δzΓσίTSͺžgͺΟa©oh=»wˆ/· nΗ9±;­ =΄ΙYˆgλΫ‘έOIυαΓΚ™ΖΰhHC‡ɞ‡dE‰Œ4aSf„₯ ημθ—½{χͺ]9FN$ƒͺΙ{© @C?Νdtζk“ˆeK—Θoέ/§η₯F+₯«ΖPXiR ΑώΤι°ΈdΜ9κγιb36ιyGΝLCCͺ6 λ7εoφ2Χpς`’A`&ΰ"'¬l{όεωL>ƒΉWHgšσ1Ι’‘E@½πt~sd: ‘ς9¦ωEGb¬φ0Ο0ŽL2L†ί,.%ݍbκ‘,›6 Ά­ e7ΰugTˆ§μl‹TUΝ6NIcΎ ’²R–/š/‡ZεšRŠαŒ v“1 ž”ΝΪ ŠΞ…δ„]έΫΠ)Χ\q™bζžΰRsz’¨ϊ‰Σ}xnΤ b’ε›ΛΞ! ΗΚY³fΙg?ϋΩs'πMŸΣ‰cλΦ­2ώ|ρω ηFRτ\‰‡υ1ύ©σ^ˆO‹ZBˆΆ–0y`Oφΰω†±˜Šπ27>ΪΙι;iZ›…Ξ4œ―gJbqν‡aΤςβλha.―0Ί+v <&‘y‘!¨ύʎz%.~™&„ΕΈ3ΐ·x§›σ±bΏ`‡†ϊΙ)ςI6ή—Iγ€aΎώ΄/Έ†ΠΑξ!΅λ?:#Cj8v0" ¨zHεftύ,Ϋ§Ζ‘›CZαLγΖάΦ Ϋδ Ϊ`Β°Ί]'mψΫ =ΘF²ΆΆ6ZΧΨ//ΆΪΌxŸ‡aRύ~?XνD†ϋπ Rͺ£bΓ τu²…ηΐΉΓ€™A€›99‹PΨݐ‹n„Ή‹—8ΦtΒΉΓ {¬©£x(Mύ8υiΪιΤθ+p«N,•ψΗJά*Fg§Ο‘ΖxΚπtίX@(©ΔŽΐπ“JP’‰J;Ιt$R»ΡA§³³SΟg›θ32ΑƒΥ`2ο2ο`Κ‹@υαj9yό 8s*a|Β.ΖGŠ•XΧY;B=•"Φ­[vμΨ‘‡ΨζxΞ$ƒΐx°p‘tœeκZΫdAžSœΐΉήΑWgš€Λ!Ϋ·o#cΖ+Ӝ3\*h©' όΤί/•wΈ”ŸSS\€­ͺͺBθ¦Aiκ‘‘ Όσ&zγo… ρTίΩ/•%%Β“¦{Ÿ‘σ.·έΕH¬K.ΔRG6Ι 0“P~hfύ}&οoξuψ/ήΌ’Ž”γγ1\+Ν|K4Q6«ΐQ…rͺHRƒl[Άτa±μ +Ι$ƒΐ…B2Αbιl=†H†hˆ­;$uμ νοͺ—`(¬ΜaGκ€ΟπΜΤӘȚ½bω²ΏMš{(”!’¬₯£,λΑY‹°QKκ&77WέCίKύ0R‚€ οYW¬$VΗ\=§U͟iE€xΣ^pςδIΉοΎϋΤ½ήωΞwJQQΡθ}Ω’igο‹E’nPu7ΞΟ"ΔSΓVjίϋs‘ 6n”ΦύbƒCMsSœiΈθšώIχ#ΞiΉΞβ‚nδ!}εηHŸSς?O`cΟ³‡»…kŸΝ}gδ©CηMΆr­,±cώ\œγQ»Φι˜Δ€οu.ί₯ύοΕPgnμΞΟfc–"uΠξ¦^9ϊrœhΰΈ…Εγγ+€«·Q n»7ΒW‡2Κ”q…Άy:ΙΈ±~h.[ΫΪekg—ΰχ–ŽNΩή~m. ν&­#Ξ4ι؞.νήpq?½ΗγUcΨΩ!Κ"%Ρ(Π'{•3M€cθΕύ&—ώΣQΡΉiσοžΑ‘A&ΖkA$*™Ϊfv:Σψ|Ζα)LS>€Ηe†x’σ«'Dgγ°λΗ₯˜ Υf…#­Γc‡ΎΤœ›K“v¦Ι εΚ•W^)«W―Vαš€;¨ϊ=šθΈΒ Υ§>§~$ψ‡ΧDώγe?ώΈΌη=žzjΤxΙ‰mΌ€ο»aΓY°`άύΚω‚ωυΉxךγΣ‡±§ ₯“ΣΊukΕ(–,w ΄N:a p±n―:4Ξ#ΤS°`ž¬}]ZΣGΖ‚ΐ›ΊΏ7€ i-BΠ--Ν“bœ "F‰\—E.+ςΛ֝»δ0ΨkL2€#ځ†2XOΗχΌ˜ί©€΄Dζ–ΙρΣˆΥ‹ΝΛQ6υθ4Κπx+θ-k:zδ2„©dά_“¦†xb q,Vβ9:άΠ0k’A`&ΰ’(Ψφτwޟm€™E€NMό§υΛθ»s6ΓΊ JΡηΝοιA€ς›Ξ[Œ>JΙΒJe{oO/*ŒCδτΤBf”ͺε. Γ³fUI_W‹œ8—gοŸB‚ς΅ Ζσ”ƒ\„YΏ~½b¦ v6ώ[·nlήΌY1Ω¨xΊtƒ¨οK‡‘ΊΊZ,εΛΘι‘–mπΜΒ<¬5g;e°}ΏXΞτHKk;œiz¦ϋ±.ͺς‰Mt"―Γ ω6ςhΊίαη˜h6ΤφI5Bά±K²ϋνl”ά|2ΪEΥ…”ˆ\‹σ9l φ–t+σ½ΊΊΊα€&Ÿhv`ΔπMΝϋ;δtuW λ<Ρ+M»O«sZ)ΠπΓκ›˜©ΓfR’3M36Ήζ€y¦Œ4›ΰ@3@>€ΠΟΝ`ͺ’Œε-­­jcl&αcήuκxOSSΣθ" i‰β%]Fw7w ‰’.βˆYŠ‡ΨΜ?~β„μΨΎέ•bqRλχXΧgΉ[‘žμΎriͺ>(;αψ0{φl³0.rζ€F`Οξέ²χπyλ‚B :- x§¬<0Ό/-pΚκκFΩΈq£,Β3σSΦiY’Λ1ŸKέ–υη₯ϊ—ςs—••KyI±œ¨=.CΎ1N}£ο5’ڜξ?+{ΪεKp¦q»έF4}_zz`Dώ€°”ήσg3xžϊ€I™D€…Hg=χšΙg0χ #ΐz ώz> 3k£‘™ήίά-̝τΩv,L`ΡU―ΉRnΣD'›,μJοκθ–aξM2L=G¬¬¬Ÿ?bš±ϋη‘H* “IΤ9‚Έ―Yϊ»›eαΒ7©Ή(K2²d2xŠ IS 'φcX$ο+GΈp{Ψœpi”PΫ‘ΧtHYQ‘0|”IΣ‡ ϋ 9“=²CVυ |· άYΕΈPmτ" eνξ?ώρε»ίύΊαώηΚήπ†17ηFΪλ»nΜ±θρt₯θ|Συ›Ξu΅§ΐnξSzΫΩaΞυΈμΟ"σΑ‰}ψŒE9MΧs\ŒεžAΘK:Ϋπώjme€{©~‡‘l†°«ΈαΜF=Šϊ§Θd\Αoε`ρbόmE_‚½1"ϋ%ω•ο5„6e&jΈΑLΏ eTok?š1 «’Œ=½-mΐ“ŽέZ?e: I0ΣXoisiομ|85Σnόcˆ' ΩρΩ#6:Ν*-`­‘γ—YXΧ­Μ|&‚™|Υ|Ξ“`HͺOžP›bL8ζDPL]ΦΧ† 9ΔKc†PWd°a~;μ_&Mtβ¬―­G΄K0`ƒuFχ“˜wD½ΠαΖgšΣ'OΓiΆSΘКΞ)ag=QΟA,³ΫίϊVyΰ#L^^žόε_ώ₯”••)a€σiΠHIχΒ /Θ~πuθ _ψ‚TUUιΣ }F*νΗ“5ϊ ρΔT[[+›6mRηυ5Ρ7ΰq 4ΉvPΓ=ύτΣκ4…¨14ZΥΤΤΘΆmΫδ‘GQ±‰ŸψγΈšhκ²’A•»}Ώώυ―+šΒBΔ1…ΧoXaΙVF:ΗόώχO HέΥb—Ξ{WWWΛb0L0QΡ'/“VτΥσgΖΠν‘‘‘μΔ, C<©ήΚϊ§Χκ§D½f9Bβ+X ;ΆmB–εΛ—λ³ζΣ νQλ6n–λͺς€Τjώˆ\lut¬ρcπXZΰ’‡ΆΧΘk §3gŽZΈŠΘjΎ.ihάb’,6ιΒ @ƒUYY©τa–Π;|Ύ UΓCŒ%ΫΨΥ/‹*Λ+Φ§Φθ.Σ_g4ΈΒ¨&ηt“π}ΩmΈFζp9@οά₯v™…ι―s‡spξE9N#5“Φ«Οε0ί¦7λ@§ρξΗσf¬‡Njλ±Q99Β dϋd¬©%e;+(Λ™Χ$ƒ@* ΓFqq©μ?tB… ΙBHκ1Ν„nB½ tx@†ϋZαœS °ƒ1…—γ°‘¨άζO$ΤΡ-\(|α%Ε0;”YTEyΡήwV4ΆΛ?ΒH–k=DήΛ|O \0[†ΝI=‡νΆ?ονn›bvΰynτ4)΅θ6ΝOΪφΙFσ裏ͺ]ΗχάsΌύοW!t>}wΦEΌϊΠ,zŒΦΧΜτg#lΟ==]βΙAΘdνμnZαGΑ;gΩύΨHαB8¨:N&S6KPF’‘«oP9Ν¨.Ή˜Fœ~˜mˆϊUDάbMSF2…Ί½nlŠjAx&Žέ $d‹ #Ž#ξ77‚gK†r¦ۘΗγR‘b! ­HΌΐ»»³K©’0Γ%ΠΨLϋ•¨Θ€π7Θ|8nΡ™Π€™CΐϋUl·Ν±Ο@_VιbΚΝ±ΩΝ―) ΠάΆή!DS#ΩΦ&JΘB›>°H’½ιBλx=ςTΞ'5k$T’KKKεŽ;ξ— ‹Λ=$/Ύψ’ϊΗAž.4ς;)•u’# ΩiH­hλ<ρ>υ}Ι~σΉΟNήχήχ©Sη///W Jœ@:uJžπsοή½JωΗ;ή‘ŒΟΙ>Χ„70’F`?v)μΩ΅Y ηά„‰ vχ(χ·‹Aή,«KlώΉb³lΨ€<ˆ]ΝΆhκ6A 3$›nςtΒsΑ³uE‘ΟΨ}‰O„γ»²yrμX֚E!yφΉηεšk―UmK—“!°™ΧLC΄‚£ψ©Ώ§αλ^΄―€eIqq±r¦9 γ 't‘j+gC8α”4uφI tŸœœ°#πEϋbiτ`]έ]ΚhΟ±4ΰI4ΆΒΣΙξ²Kgg§…i„4Ι 0SθΉ’–ι3u_sŸ± «Ε–xυΐγΜcΖΩ±ΈMχ―ΎN5ΜvΉ 7;'Αρkδe;)υ)ΓM2€κ³fUΙΞ]»« 7ύ¨π!η`b·ά€'τu5Ι¬ͺω’£7Ρ+ΚδB—§ΞL«ͺͺ@VZαΐΞέX[I8±JΘYΧ5 ε‘0E%|­Ι89ΪΫΫ±@ gLΨ¬+φ#|’VΎ΅§SZΫZe6ώ3)uθΎΒΙ6/ς/²sηNΉςΚ+εΞ;ο”oΌQm€Μ§οΞcό/2‘•ƒNί Λr!ϋ?S}C~ϊ°qΏ'ՍJ?ƒiԞƒΠbN„ƒͺ‘A,,Ρa"Φϋκ«.υO Η/n8ξ–^ Θ³X\Γ6‰W ‡οhθμ‘P0ˆu$§*Cίk^”—pέ! Hύ‘Z„f“Ζ+Q΄­――WN4˜!lΌλΩ—Ψ•œ›  •Σ=%εLC04° ,PτŽ·ί~»¬\ΉR9ΣθpKΜ§iήωΞwΚM7έ$tXΡL0Ί ζK6έ|ΣΝ*ΔLR™3­Aˆ–U`ΗΉνΆΫδZ,rO”θMN%˜»…Ίκ*ε”ξŠΙD˜\Θσ{ξϊ^½zΈ^sΊOζ”—™‚6Ή ΪNH†rn‚²9mJŽs>Dϋ w—;°–:zScϊŠ“ΏŠ²Œ‹ŽO=υ”|ϊӟ’††FeΧφ·Ώ-+V¬Pλ:ŠΎ ―^;“Ej“‹g|mΔϋ τ‰Ε‘#Γƒpπ‰^@₯-:ۍΝ9hΪ*n°ΘfB’K»ΔYΤc[/6[ε„υ&κOI₯‘ΎΚΦtΚα C!Εk7I•}‘e¦m8Ώ0α»ΐ*ƒΕaΎ;ž#b<°:yώq6ΕΎΣ*άL<†§σ―J#d¦ι’3 ν„’1…*/>œg>:ߘdH­ŸsΞωx˜‘μά•Y«bΐ£ΞaΜ!ΐϊΰ¦Q«ς–9ΎZLZρŊ|}ύC( Lš¨χΡΏ# {H9X—g(ο³σοΖqN%|’ΕΖξ±#,f½rTNgb‹€i4tT~ΈƒϊŸψ„Όλ]οR@“Ϊ^Η$χΈ=β„Η1•%2ΩhΗž§aΞ,Ι$-τθύя~tΜ₯---ς»ίύN.»μ2ωΜg>3)γ—.LΑζnj"°gΟΩΊe½xrη‹ΕƒO‘UwΠΈO4’aA Zl^±εΜE|·gdΣ¦Ν*Œ—©ίΈΐeά =q;pπ€<ώΔRY’E6qΐ€ΞEμXM­‹γϊ’\›\]™+ΟΎπ²\qως–·ΎuRς&γ@7/|Ρ#ΐ…=φΑΑ!εPsΡ?pš> i―ƒ>/B9Φ2‡†‚ΘΔa± †¨A) ¬¦Φ­"σ˜ο©Gΰ t κΈ‚ …ΥEf$ zΐ€•'‹06‡UN·Γ™fΘ,”+ŒΜƒ@†!ΐΞό§Γmy}8sΨ­ %†άΰ“fƒ q·»!Ώω…ƒ©–ί#aƒlΟFύύ@Οfζ Ν]έ† ‹@“έ‡0O¦ ΏlxT ’Kg‡{₯Ώχ΄Zάt»έκbcίHΓθ܏W*JK€Άz?XfόβWZwbuΓZ„l©οθ•y‹ηgšhpSό›‹«u΅u’SδSαVΓfBl‚ΒdΙ°£oτΓΡ£!ΕwΝμβ΄ γ¦Υ?ϋΐ°ΐxF>υ©OΙ—Ώόe™;wξ(8—šοE]­‘‘^¬v²ΆγΖφ΅χ‹«vw΄5ΒQ Ι<ϊςiώ₯ λ8KΆ4vA/Jžq•hRV’Ψ¦s@Šƒ9j K7Ψtp8R„1°wPϊ;Ε{ΖuNΥΫ΄βC€|Œ03Ψ;$½m}ΚI¨2%Ρ&Ωη'6Ξ»ΑbΥ›W4t”θς>8eqΡ—ωωs#C¦gή3)Hώΐ°‘–lΘ΅‘y!ΫY–Υ6Όt›yzRxN5sOoX±gΓΏx‰ύžΞ6vμLjννΓζ%co‡ΥdkύˆvͺϊΊzqzνpXϋn„υξρKΞvX$Pδ—šS5ʞ’Ξ¬Ρλ4γ#q– f’§,YΘ Cv˜·Όε-rέυΧ)ouξΖαb½šΘ`σ₯/}I:€γβD*CG}η;ίQ!€β•ΗgΥΟΛ<l£Ε»ΦŸ>XlKœΜ¬]»Ζe8[ζΑŽΤlρ…ιΨ§ωΠ&‘}f{+Ε,–uk_U^qΜYc―5Ώ2έήh(ό±G₯¦©Y/sK©Š8›N x"°`§)rJΐ~XŽ9’0m+pζπ%ƒ€φΎΔ,³cώΒU›bΝ=pοΐ ηD‡­ΈRQλκ„>ƒqΣλσ©ρσΒ=mζάω 9t¦‘‘&ΜLΦ5¬# ΒrΗl{[» Α)Ν$ƒΐΜ#pnn6σχ6w$4>πibι†#‘rΞ3M+ήΎ^89žΖb˜]-Β†o?Ί@IDAT!Γω,C‡XΰLΓE[}ν΄> )<#πz=’_P!C˜L2΄ ΅ΉdσƒΜ6\Τ …BFχKΎqςͺΊ³ lπaWλ)γdsŠbƒΜ4§ϋ%ˆΞπLzQUύ0¦Œ€rf@)mνmR}πˆδϋ‹X€ŒΆΓŽγΛχΚΙ“'₯£ύ )ςΌ:`ώ$…ρc[©©‘;ξΈC_σ7σ7ςΥ―~U9€Ύm˜―՜:)ξœbLβl5ψ)ŸρΜ4YYdCΚ+M}F„.`CΡυ[^Q!>ΘΆ†.8‡@ήE’Γ|‰$4#εxx²!ͺ „e¦c˜•‚ΉΘ›γ•¦^ψΟ’MiΠτη/ΟVΘU}νƒΩά#‹/nTOχ€ργό¨,dE`ω±c]‡c@<θXT/@ϋ$k7ν3ιr/σ~SC€Ίν{YV b Λ‹Υ Ά²Μι\it:Β4!CΠi0{³?»H=3NβxβK8Χ.LXζq€šβ)ΚΤښZθά~l…ۍέγWΪ/B–_‘GŽ:†: χ‘)>ΚE{ω€™iψFz²ΘKγBƒ>΁‹Ν―"”ΚΛ/Ώ,O€‚‰μ1*…εUψϋώΎμό/ρΩτ3 !Ξ) œΪkΥ ΆρP›Ωγl'λΦ–PΡ"±ΈΑJ3ι„'-‹# žΌΕr`ίs²eΛV°'½s΄ LΊhsα%€– cσμ³Οȟ^Y-oš[(W:”gϋD[ΈΙ\S°ΚM^ωΓΎ:yŒXŸύάηΤn4]ώ% ’yψŒE@‡f€c#iMΊ0Ψΰ|μσΈ₯Ώα°hBgšΡΔα ?hΜΒ€†xγœό™4hΉΞώΡΥΩΪJ› η€ξ9R7Θn…³ε™!«΄o3iΣY)¦μσ`;₯n“ #„Ω§η=ηe6¦s9ΗζAn6£ ~¦ž¦΅:Tα”έ­-mβ Ί ΏaR‹ηWŽ«L6°ΦΠΤΦΪ¦τ R›d˜*ΤΥςς &£Sœgαh›…vE£d2 ωΟ€Ω†r%Η “¦†€–»N§KrΐςΨΪέ§˜Xj25ΣηvŠ?ΚΠΆΕ©=™Ή:&ˆ»\›k[dξmε£Ξ4<ΗDٝW”}{φIss³Ιά€©!ΐ>Bέ’vό^xAΌςJU Γ=ΡNΐσ:ξO܌sϋν·Λ²eΛΤ΅ϊψԞ$υW‡η³ ‡–cG‰+w˜G)—cY±7 ŽΆv6֝Εαz‚ΎmΕzB&€`0(K.CϋwKkŸGΚ|dΑ›ŸS‘&„Y›{†₯Ά£O–-_š–Μ4‘ ̚-‘ΌtΦυΘπΐ™°71K&Aυ΅ JΣ‘YφιeβΓ*¦‹΅O%σjενΖͺρσά.±ΓFO[Kt’|’³M˜ϊjR„s/St^σΫ ‰Χ„[Αf”f£,+ΥΠΐΠΞ ΘΑqΐG>aτ“f3θΛd—’3Žq«ˆ²ΐο΄JO/ζφΈΖ€ιA ₯₯YŽ<&‹ί>W9Σ„εp AΟ£άhtY₯ ΙžΒ”c‰zΉώ£±¬­­•φnρδ;•νd˜Μt#:ΡxΔλ›ψΑY}ΈZ½ζΪΈΜΖ+ηR87)g6z&₯Αζ$ηΨ±cΥ³Vy―SρφϋόR\ΪD€Γ‡Λ?ψAω»Ώϋ;)EμM&}­ϊ1Ι?ΊrρoέΊuςτΣO«SŸψΔ'FΛί»w―|ε+_‘?ώρcξΒΠP|žόόό΄V†ΗΌτEτCΧΓ~­|ε%tΈΩbρΟBΓΐCŽ,Nξq)iαξ.wξB°ΣΌ"6l”χΌηέ£mbr嚫.Ut[ΫΎ}»ά{Ο=b³Ϋδ-•.)υΩΒ Φ |wf£βξ³g˛˝›+=ϊ˜€²τmo{»r*Τχb~“ —l³\lβ'w½jŠμTŒΡ—Βϋ_LΟH%•ΞΙcL#Τ£ςe‘©Ÿ™«9φΞӝβͺ²ƒ0θ³rX :ρ7fwVLξhμ}HŸ6ŸιD€Ntάp8*l*ΏΣ±Ζ€™G€Ž4=0[’,Zdΰp‡HBjχ|¦ΖgΎΒwΤϊx7BΧ­•εW/V;¦Ξ’έ-rγe9AΆ{σάrΌυΣ€œi.Τ³›ϋ¦”Λάδ58pTΩ'Ζ*‰Ύ'P†ϊ$:Ίg$œP’Wš|γ#ΐy 'aGυρσFŸΥטρ6™Τώλΐφ­Ϋ%PβwΎCΝWGupΘξ,Κn„αφεzd+©iΌηΖ“&‡€ž_Rn}λ[ί’―}νk£sN}.VΙsSiηuTγ;POζϊ„Η_ YφNͺ‘ 1—F„ώpϋ‹επ‘]v¦9/kš`}&1gΞl) ψ₯Ίm@ζ…¬Βp₯ ©Θypœw§ͺΥ3xFNž§έ!άΠLό΅~η²Kς°ξd£[qε yα©₯»ΎDόeΙ…hb9ƒ]CTέ*K._¬œσ ˆ.’'‰‡nll”ξή>Ι±YΕۊj+QΧ³ιΡ ΦŽφ€¬ol;ΚA Ί¨œζ§Aΰ|Ϊ°aβΘαƒρWv€ΤήΉΜ Yο ”ΘργΗTΨί8u₯£Ό:• wδBtΦΑyi^ΐ₯6pž•ΤHŒΓˆƒ#y°·b ―――SvWMμu‰ω™$Ί3|֞={₯h~ΨFυόڈ_0υ(wζ½ωΩΏoΏ²KR—ΤeΗΏς;“”3 `ŠΘ9ΨΡYε₯—^’|PŽ=*EEERVV&§N’…9 ε»ίύ.˜Aή —_~ΉΠSW§Θrτ±d?Y‘=τΠ†ιSŸϊ”|μcS‹‚τPύωΟiV¬X‘v50½ζ91pƒξ‹_ό’ ύΔγ&Ν Ί3ΡΫsυκ5RW{\Κ/PΨT΅³΄σqmρm³ύ³³t'Ϊη‹ςΖ7^mΨiΖΕ,=OκΆΖ σ/ρ ihοχ- Ι’όΙYΨ29Ύ—‚ζτ­³\ς8‚¬ί{ο/T,ΫoΊ)m'ˆιΩ:2ϋ­tίΰB,©ˆr·)²ι Κ'³‘šΉ·'…4 |θyKy ‡u™Ή'Λμ;uwwIG[‡–Ž0Σΐ;Ÿ‘ΣHEΩ\ΰ¦}$Fl*τ\} σiˆ‡Γ‘•“2›σ½“ΚΘοxˆ₯φΈΖ™΄Π5ˆ/π ’I"&j1‘₯…Οžzb»446©Xίάυ«―OνS™΄ό%Uww[―8ό˜2Œ‘ί€ŠuΐPž|—yν”’›6Rυ].· τŽ8Σ$_*εΗΩαAe―2!>“Η/‘+¨ΗΕRΊc]¬Ί>ύ[7ŸSB@Λπ­ΫΆJc]£,Ί|ΦΉ0ΈFΦ°§όφΉ$P–#ΫΆl““X˜™3gŽ[§„Ύ(‡€E‹)’Φ·ϊϊϊεΠΑβτδŠΕΙυ θ£Z‘/ΜΕό3jΑΥζΚ•†κŠ± ͺͺ*2SZ―¨¨”+–,–έΥεΊ2§Έ= VŠΩγ @{F{°μkμ”«–-‘’βπFλ8Ω/ωΓΊ}Ν™;GBυTΫ#y½~ Ϋ|=ΰJF›¦nYπ–Š‰Wjy˜`)—\6Ύ™Vaƒ~ŽΣ!~8ΊN4΄rŽΕ|^θYǏ—«°ζhΦ/ΉͺŸρ>…°‘υuΗ₯rΩU#!ώΠιΨΨΘNFΗΙ@ΉάΏ 6ρ‘3IΣ‹uΆκš:Ή΅bΆ νΦ­Ζγψχτc._ςΑχΰ˜ZΓHw¦ΈψHLΟ: οήΉGΕ~qδ`sν& άJu!t%^ΓΝI'Ÿ„ΓS=|ΣΣΙ1r_VLx¨π8ώγ]†kΊοΎϋδŸώιŸδζ›o²ΌpWfhόΐ> Xθ`σπrΛ-·Œq€‰y³$κη"ΫΔη?yYΈp‘Ί?υ9†vβsy¦²²R^yεϋυ­o}«”——ΛψCυ.Ό΅Ύ&ΙΗ0Ω§€ΐΑƒεε—^όΕ’ε©@Iͺ N‘D})ΚΑ`hqζΓ+n±μέ½U6mΪ¬N¦»"ͺ0Ÿα>ΝϊVŒ4!Πo›”EՁ’-y€ΨJ1¦Θܐ]ή1Ϋ#V»UξγΝZΔ•fβ=ζb“ΝfW†³:Δ)¦ƒI3ΐππbΐφ‰=μΖ"…ΤΘw;v„QxυΑαΙ€™A »»GΪO‹Γ fšσΒ<©κPκ‹ ;%θ‘ΊύτΜ<˜Ή‹AP†kH²"Ρ±Ζ€™G€‘ΰͺ‘Β _rΰ―MΦκ‹:‘΅Ψω]tΛ©Ίz΅ £Ο™ΟΤ"@œΊ8ϋΕΤInU0ΌΛ‰Wy[‚ύ”²έrHΟιCΪκΘΘ΄½!ϊmdΓp†Q/R($ƒΜ™a΅HdXP’mβΌάέz€aœΗμρ.‘!VΘ¦Τν…}Ό‹ΝΉDΠvnπXϊzρ=—β;BFΛpόΆΑΙ&APjŽΧΘλ―Ώ6[κ>˜ΘύLžΨ°Έq5ΡΊήb—vρexˆS§Ž‹έf¦‰nSc'Α^E§«Ν%dUgJχφ₯ߏ •K—.–ζξ~„kR*ΤΉal\ΰ”=΄Ύ{XNΆuΙ’Ε‹% fvΕEΕrΕU—Kcu³ 34υήγύQH"Ϋ„<}ͺG1Ρ-]Ί4#μΚZnp3Θ‘£GΕ )H†Υρ›—:ƒ i8β†4“ ±`£L£{Σ¦β •H– Ž2Œ«6*ΥŽ“Ω8ήΧΧ#{χν5ΊD,0SpLΧGooί‰Οi“"8j’½wBζΌGώφoVž|ςI£Ό<(£PŒ―EΆ—wΏϋέjWwΊ³,ώKeΟ΅~ύz΅“¬7>ϊ¨r¬‘‘J]»Τ9ξΤότ§?­nΈαω—Ύ„ψŸ+ΤΞ:™4³°ξ¨,­Z΅‹NMβ-a₯aD{κDό²1μHΎ`±όιOΟͺΆΙ«Rέ'zs~ζΠς‹Ž4wύμg²yχ>ΉevPήTζ' ζ©hil©KσθPγ“ώΑ!ωιOͺΒΝι{›v6συnξ˜<\ˆ­«=%OžΨ]9q⸐ιΐ€™G`r€ύt‡8mΩp¨ Λ)=$Rƒ’Μqbχ<'}tl6if ΩΩΠg›b/8Λ—TŽΝΒΗ,0Σ€ιΙ$ƒΐL!ΐφFγ™iΪΪΪ”ΑLέΫάηtdΪŽ^§„³½’ ΜAΩΝί.Δ/ρήc'θtΣφΊΜ‘ƒ‡Δ_ΰ;%•^Ξʈ“Έ »wΙQξ3]>XζpBp—4…3 u·Ι$:ιbI[ΩΩ²Q–I©A€ΆBʈl(ptt€hH΄†˜Χ«??ξ>ΡλRσδ™S mΉΫ6l—’Εy Ž‡M™»™’e8ΑΚ_ι–PiP±^ΣΝdδχΤΪ mΆZ†%ς©νσS»λτ_Mϋ`ss£8|`J±¬βlΑkΆΝπP‡2FΟfa.»lΪAΆœ8=(πυˆHѝρά)žaή#­ύςϋdρ’%α±0ΕkBηξxαΏιφΟΝεάΰMVΔξFlP›Έy©‡§@ˆ§ΞΊn)­(‘ͺͺͺ R3ψ­ΏΓ/A¬ϊ­# Hγܟ°ι&€ωοq\G›&“‘ϋ σ'555Ή aϋΚΔβ.bcA±rΜβ)Q‘žΦ­];jΣ3m*˜)8ΤΤΤ,›·ν%Ε!τc‹Z£‹ͺŽΡ»ΠΦΚκrΣ–β·KS[»ΜηΦQ¦ρ Η/ΆσύϋχcȐx ]Κ1&Ρρ‹Ζλ6ΫΆHnξ«>R­Φύ§ρ±/XΡγΞΖ &'˜ γtύχΛίύί+†™o~σ›j;Ÿš)ίψΖ7δ駟–σώΠH:[ξΒa"UeYZΉP'¦ψG 3ޏα[˜>ς‘(g#—ηλΦ­Sχ½μ²ΛkσρZ²ΌγοΰOΕZΑE)έxΤAσgΪΠuΗNΊς•—$'žX|³1†qKPΣLθι(mΑNγ.Oή9t`·¬Yσͺρ.M»K7“n_μΟ{φμ‘ŸΑΑeǁΓrΫΌ<8€š Ρ±μ0“ycΞ%yŸε`ΊyΧ\ΏτΓ(wηίΫKJφ™2TΝ53@kk›Τœ:,v…X=Επ"U‘žfϊ9Μύ:W4ΐΰΫ•Š(}6"§wά8λƒ £†EφκX&M”αœL4·4‹§Π-ΩN,œDVHδ­qΒζ%qΆEMΌΝξ¨HpΜχι@€ν“‰›8_γό§1§ ³Ψt =q™'±“§†ά|M<0τ0$h€ΈΔό“αŸ ±σͺ Έσ‡2\ΧγΔw09’E€†υ£ΥΗΔtbΡΜJˆ#ΔyΊP}€5>Z}tΤ™&Ω{šόH8/εΏ©τsΊyhY’ηΉ‘χ0ί“C@cΘq“Ξ¨t€δnX&sψΧψ)ηΙNΓ2Œ>>>VɜΥύ₯;Žicφηz%8Χ‹=r¨€1 ω#₯’θdγcpΑβ=pTΩ€yv*ύ.™g6y/ ΨΨWΙXΧΫΣ Ζ:,¦fqVMε`ό”eσ‰Λ_,Ǐ«Ν)•YΛ’ρ―ΌτΟ–•—ΙΒ9UrΌ2³oX9 †‹U­6δ=kζTV`“5μύt›X°pΈά.ιΛΜ™axƒ‹†ŸlŸ}νƒ'Φ,W_sυhˆ§ €M½"C<Υb[κυˆNΘ ΌΈωJΏΜΪ υd’A μ[t &CsS8Uύ.dleJ™,GσΖΩ²ί.EΞ«ώ$Ώώυ―ελ_Ί\wέuRTT€&“4ζκrθΑ>“>'γp1Ρa†IOfιLσΜ3Ο¨gΉξϊλUψ)žη³ΡΩ§ͺͺŠ?Υ’!i Mš~ˆ=Ϋwϋ½πΒ ΥΩ*ŽΌΛΕβΘAΕ(Χ„?Ίr–U²σ•ΫσΟ?§vς&Ί¦ψ†¦Έ ˆ€n_|„M›6A~έ)»±KψνσsεΦJ§x1@G/NOεq9Άp°ΰŽΆεEvy8©V«ΫNηψ=]οu)—ΛΨΌΤaΌv Œ§Ξ˜΅rά(¦-£‹}FƜΣThL{v«±‘Qό…^ΕJ3fΞΝ›DL2²Qo§΄c§„a5˜j ˜λA€ς‚σ²­vaœc̔r©Λ£ΖIλ]›…Οb2X‘x:nG'Κτθ’ˊƒ²;ξ;a΄`ς&:Ώω=5Ά²£½CάΉ.ΙΆcΑ,Φ Š[@R‹±dσδΉδΔ‘“ͺ/MνξζκLF@χiΚhκΪ–l„, ΒC.$„|Ή`uB+‘Lʎύΐηdςr˜L·A·]ZΰίevΔ'„y"™΄νδΥ΅―ΚΆMΫ₯pižψΚάJFGκάcΚBΏ²ΐ#*4ί'ΉeAyκOa~{PeΡ}qL~σ#γΠν€ Ž‡`ϋπ…ΚΒ!ž€Δ„bΊCVΆ L³δΔρCrτ豌ΐOہ‚Α¬X~Ήμͺiƒ νωΏ>―=Υ9,ΗβιͺΛΕησ«‹t™—piη(/+—ΚΩUί-ύƒΚ&󍨠Σ3 °μiκGΈΡ^YΌh±be»MwΜτϋΡI­γi‰ αE9ωΖK<Λ|₯Θ Ξ—ά€`’A -ϋΩΎV|ErrΑJγ-G§γl=Ί‘ΏYμbΛ™§Ξ―^½ZF°κr’Λ7Ώ“C@Λ4²:n€s“ ›HglαOΕ‘‡5ƐPσς<²mΧΕ€2ΑeζτθΆMߊ=;φJώœΨ`ϊiΜ―,φ,ΈrS°Β/ϋwυΧο²Kρ₯ǘ€AάΉs§άz뭊u† ³gΟ–όγ*ŒΙο{ωφ·Ώ-o{ΫΫ”CŠΥJjDΠψΑPΐπJΊŒ1OΣή“»J˜HoΞ€czΉq·&ΣΥoxƒΈ0ΘκŽΛc€b’cΎF0¦Ά―—^|^‚Εˆʁ,ξΜxͺBν1ΖγΦ•»DjNV+κWΆLOΫ‹λzέ·ιΐςβ‹/ΚΰΨΨΪ.οœ’7•;RΚHωζΠιF‡še…ωΐ|―ψaXϋ―{!>π€ Ή •―™”‘ΟhΎ’Πm±££SΆnέ"ޜΔΚσF.d£U؟1ΖΡΨ\ˆίz<ͺΑ$Ο…O0 0ENο(k†°:λ„ΚU˜£ΌψΝ’Ή‚iZε£‘ΎAν':eξ’Ωj£z|tΣηŒnΨ Β~UβχKmΕ’ΨψοΙkiƒΩκΙν–jD¬ΰΪe&τΛρ‘1g#Πνa-’–­ήuΑΕΨ̟‹ώH†ο²ν*ΛU(Α’%ςΪΪU°•o‹,Ξ|Oϋφν•u6aƒQŽy9>$PπΘΈCgφŘΫwΐ!‡Ρhθ λ9RL–(H‚Β€B`Φ6‹Ώ μ``λUu£‹D]~ήO tvO‘sΧ,Ω½{·Ϊ ΒϊI§wF―w½qζΓώ°όκWΏ’ο}ο{ςΕ/~QΌςJΕςΒέ1ό§@ω}&€’b§½œc‘‰ ΟE‡ ¦²²2υίyœ•HGV*S^^ξ¨#Ž:`ώL lĞΤQd₯ΙFΗ²;UMbβ d)yHe²Σΐ»4_F₯g”Α›EΟt{MΙλ˜BΖ ΐ:Œl[ΏƒΦ?ψ!,Ξȟ- Β‘Ζ ZύΤ2Œyόΰ° y•–ζΣ‘Ζ'sχρΧΏ{XξΊλgBΚJΆ}ώ3m.=σϋB"p TΕ‡ξh‹-v0„ΩόβΟ›#;wlo~fk†α„Ž‚žΦ…ΡωDm̜‚f<˜<ϋ²λΘq„zjWiδΚτΥ4λj@λwˆ1ZΒ“Šσ'”"jj> Ω£E]gΏ,Β5ωω±²¦ε1Ά3Κ2§Σ)‹/–¦c-2f:ϋ71Ππ0κ;#m'Ϊ₯²ͺRŠ‹‹υٌψlnn–­;vIU((^Θρ3lHΐ$ήτH'Φ>lŽŸ ₯bWΖυˌhSxI=γΪοΣ|Rr η jΕνP*ΪΨy +;Y6l’ .§Ϋ'xςΒ°“ΊoŸw‰90ZΟ£Συ³.ŒΕWΉ`;IΞ‘7¬πgΛεeΉςΚΥΚφ•πC˜ŒcΠ}„τΦ­]'…s ĝO“˜²²χθδE½ϊςΑg4ϊ0ι{ι<—ςg\gm„εŽ Rd2€ΣO<‘ΌΎHΕσtZΡL­€κίΣύ α%]₯nC4NΎ˜Έ›a͚5κϋΫίώφQg²Π0νΩ³GνVΰχͺͺΩβρxTΕΞτσσώ™–6oή,――[ Kaψ/Kb3Y€Π₯Ήγ!LRŠ1Ι\bΌ'‹ηΕsŒΩoιEyΟέwΛ<ψTζδΟΰΠΒΠKVxωkG—ι|r=pPοŸ΄ΙϋηΊδUωςšΧδG?ό‘lFΨ)͈”NƒΘtbjʞ>Ψgθ4»uλVζ»ΔξΗ8 ω,«1dηΘIPοΖ8ΙdΖΕι«–¬εASS³ƒ3M‘Ο1ΚLΓσZΆπsς…žψpΆιTŽzΌ>Zγu&₯Nφš›AWi—l0Jθϊ-Γ¬xα»i¬i’.,ž™d˜nŽ;.-ΝuΨ=’/w‰8=!ΜΫΞQͺžΧ^§ϋ2¬|=–ξΫ·O1gž$"ΖWΚmιΑD‰ |ηζΤ ΏM<ˆ+”1L¦ )ϋC¦ˆ#υš[θ ub=ΔM8ι ΪΕ‰qψθΡ££N‘¦nβ"fNL€Yi~Οfχ(={‚μqNΓΝΛκRz;ms&M ΚmΞΙ)·σΌN)πΐ¦‰ΎkY%ޝ8ί'3 υqn;Œρƒƒγ„ςˆW9 Œe½pΓγSO=%«_Z#₯—J.XiβΪGcΗ:c†ΤτΛƒε#=τμΒ+“‘γΡ€eΞo]χΤ·ΆoΫ]9:sα φ|Ϊ“m>qxK`'9’q!eJΰάQRT$§Ϊ{dŽ!Tqγ!ΗsύpΈ©;έ#³gU¨Ψ™ΣΪΞɚάά\Ι- IoΫ€œ€E:hΐ‹`‚υ§―{@9p}Š)SμpΤΉZ[€NHn¬Ω±mΡΑcm¬ΔγΔ†v~ζ/r»δxM-Β°ˆ•έΛ@΄^ΑΠ~=φ8½[Δ‘Ε+Νx+Dn`=B oα²wΧVω#BGr“ۜO2”½ς87½ψΚjYQž§œbβtσΈχ 3ύfΛ….iοκ‘5Η5ύ1Sδe\`¦p’¦ΆFvΑ‘1PζS›Ζuΰ>μ#ΏUB³rδπΓ`;4Α—ήισœitγ[°`|aR*++ε™gž‘;οΌS>ϋΩΟΚ 7ά ΨiξΊλ.Ω„Εaτ"?Ή8§Λ˜N8΄ch§εΛ—«[‘Κ“ΟυψγΛ~πОŸTΗΙ”ΠγΞηž{NΎωΝo*―U*8‹-šΞG5e=5`λΙιΙ+Xi²¬Ψ₯5#Tψa(ρUInρB„˜ϊΣθnPSA—&ΊMQζΠ‘ξŽο~WžΕ€| XώlΎG‚ζ—*ψxjt½9χbΔ |η·Όkq‘†RνψŽ<yτΚΤ2(bΣU¦άρΠνŽ! ^]³Œ]³`PΩύbC§LςŠηΘ³ϋλκκTQϊšρΚ5禆ΐΡ£Gδ Ψfζζ{•ΓΜ0”PΪVb₯\W–,)Κθaιθιλ/Τ±Vv†††•q~”’9ͺb” rŸΜ4ŒΫΡ%§ΓψκΉΝ}Σκ\€έΌy“Έ} žΗ[0=zσ€©f‡Ϊ‘ή\ψ·Σγ"wS„gYI{Ο›ZG( ˆ?OW»‹S δ½ΕζVvΆφφΣ*“žKΖΉΒŽƒ€–³ά€·NE>—”ϊΒ!λγ\χ0Λ’,_ 6Ϊυ6Ž†^Υχˆ{‘9qΊ=ΏώϊλςΫίόV *ς€θͺ Ψ=θ34ζ$š0ΜRΦη”»₯τΊ<θγrίύχ©q–χ0u“(ι™Άƒ`δuωKΑΘLNSvΓN’ gΚp2°=ιΆ›žˆ…ψn―WrsCΩ7¨Xgx,j:ΜC#Np A΄‘£G ςσIγ_%έ&ά=TTZ(½ν}2 ¦ˆ qΣ@2ΟfI^~^Ϊ·+ ±";σφ;Δk³K œiΈAΛκx˜ιγΜgŏb¬ :ΑξΌsηŽΡhϊζ3σΠ²™‘Icύk«$XΌL,ώy#`P/‘ΏBgΟƚe¨h.Φq•υλ׏w97ΊNΈ)ΰΩgŸ•aŒ+ βΓfEŸΕPΖ)“2`aΘ*W”…δ™η_P2„Ω΅μηRs*=^‘p€λt—*Όbu`D]$’ŒσΎβΪl05gϋ₯­©]­½kbσς^’Ξ³ψH6Ύ’’ωηώgή‰»Ύφ΅―Ι²eΛΤkβρε/ώβ/TžGyD펦§wNNŽΪι‘ρ˜‰†|γ7Κη?yΕ6σ•―|EώόΟ\^|αEΕTρw½Kn½υ6υ8\tΏοΎϋ„L5ά=Bο¨ϋ·}/݈τ³›ΟΤ! ΫΥ«―•½{ΆΑΨV&"ππOZjNκ±ΤΜ“&ŸΨCKaŒΔ ϊ’‰©9),/μEZ¦°MqγΑoη?₯;Pί½¨,.)Q v•&˜³LΛΛπžTr ά\α”?_ŸΗ!?Ήηηrο=w+ΩΔλ~1-a 5ΔA€νŽ‹5k^•ΊΪcΨ…½@Ρ+yΜ]WΞ©«―›1σΨdŽ¦+Z"sθφm[Δœ…Ψήyj‘Φβ©%±_ΙnνΌ«σ§+ϊ½Ψ/{UΛά\·δ!ΎtXvGKοπSjaQžSj[ΫdΗφνj Π›S.τϋ\κχΧΊuυαj0E ‰;ΟƒνΤ»Η;ζa8Ώ’Žξ;6κP<ώUζ¬Aΰ|΄άνξ鑆ϊ0LŽ8Σ$m€rgΨ0hΛjk ‡5?ŽζH2μίΏ_V•*ΘνŒΖμϋI%ΘΞυιLSΙPOΠαΉPoRςθΎB¦ΆϋοΏ_v§”\ŽπΓ;Eί’u +wξBΏ”]]${·ο“ί>π[΅ TΡ—˜ί鍀žΣ‘ͺΆ°π‚n/|Η§σΊΣ›―Φ›pΊ §3ŠΤQ}p¨‚’Ϋ7t&.rZΝꇀμ/±ΫΉΙ‘:˜>›ΞH{7†zςϊ½2Π30‘C7Η !ω ωΌ™α|€ϋMmmμEί,ςz$ߝŽΟα˜Θ·B‡!έ² ΒMM™Σ/Α&σΥˆi-ΒΦ<ψΐΨυ)εlCg”_ϊΧςαX>τ‘Ι=χά#~Ώ_Ε“γB“6 ²’tE©)ψ£Ÿ“¬3ϊ―*ώο.+V¬P,4V›U9Ψ|lεε%„d³ΩΤ?~η΅d±ωΨΗ>jVN “U€YŠI‰  ±=ŽπO=υG δ–IvoΉ;kJ=4‘»GεA5[|³°ΠuΉ¬Yυ’ΌnΌK£ΊxjΒΎΛ…η-[6˝ίϋžός”Š<Ώ|xq@nͺtΒ³5[…BIv@Nυ›λža¦(tΘGϊδϊyEς§•kδ[ίϊ–<‡Πyt@δϋ0ι~’κη0ε"ΠνŒ”ΧO?ύ€ η#nμ| ŠT8 ”εΡώ`dφ“ΚY–eθkωέ€Τ  1₯cΰϊeYY αpJ+‰dόΙ.„šrHkG'vΖ™p\©©Ψ₯4#όΦYl‹%γŒJ±ΤEVΜH]Ω°°β»PΝΙΕΒkt=«λΝƒΐ`[’ή@Vš•+Wba΅IΉ ₯ί3Γp†Μ•@αBymέͺΡP}SΈΉ4ΊΘ:±Œ6μ†\˜k!γO)Ο ͺ~Œυ½¬<_žƒs½a‹ƒtr‡΅¬e½lC(†xr8+uΒλΙ0ζ Ω%§ΐ§vR™°ΐ‰ΑgrE@ΟοZ ί φΓ€Ε±,2 œ“c―οœi†Υfw+c$©ζ™t{οJsξZn“ ž‘^ZΑN3?δ„άNliε\IηΎ1ΤΣ\„χ@ˆ'ʜtΫyyξM§η›:7ύ쟩¬•7JpŽoœ±t‚g‘>S΄‘‹ HΕςYυΒ*ωνo;ΚlϊΞ¦Ρi]Χ υ±ΞΛޜ"ΕζΘπΦΙΙc₯$€ƒyήΩ‡ ’tjΟ€”mΛ^˜Nγ%•Κ.ΧZ25q-̚Mž#ώδΰ<#,\VΜ)2)νέ»WφTWΛό@ŽΈΑώήnMA>qb.φq/ΪΩό`@φ`rΚ3)sP2 ƒςώΏψΉd£?Ω ―ƒά/Bc ·„ΡAΫΚφW‰§ψil8…5ο{•γ4u|=Ά$\Vg$Vtpβ˜ω[87‡όςΖbΨ¨, 'ρ γP4'h“λ«reυϊMςΚ+/«zΡs°xΧ™γc8qς„l~}‹ΜΟG6oŽY‰‰ΰ±ι_Έ–aW96 USπ‘#GτΩ΄ψŒιLΓ7ӍžHQ·pαBωΘG>’B(=φΨcςӟώTnΉεC>=σΜ³JYbh¨'žψ=Œ‹€ΎΎ~΄1G–™*τ΄+++Sμ9œ$1ΜΩrξΈγY²dɘ[½ρoTl; “Ν& >ߘŒζGΚ`1†4ώ§N†wθΥ\Ιd)y"°.ΐe -έ²[žyϊεό₯ΫQJna I9jπE;b=q"ΜpIίώ”{χΙ­σ δƒ <²^€xœͺάOεΓ+ε~€ΐ $οBΨ§χ-Κ—ξžnωξ~(Ώϊε/GLL%ς¦¬Xθ~DνGyTz{:Ş y숀9¦ζsF²Gά•Ή4ԝ'Ÿό£έ΄ΡX¨¦ζX5 {†™ BαΔOJA‹a¬β!ψθΑΖ*A0^q’Ȑ›L¬c“R‡EΨ_,Ω–°3 Ƙρα·y¬ γβv`ΑΖ$ƒΐt °oί~θΥ/IŽ3„惀V²›αSmtXΗοηŸ{^H₯kdχtΤ@ΈΜcǎΚμZŠM%ήsΖοΨ’βάQξ9ΙƒγέΒ<· Ί‹°L¬+“¦Žw₯2F··ΐ f;ΖΖDmBag(’ς*C²aΈΈ[Λ$ƒ@²°/“I†l%n?μΓ7ω„ –SWWkΪδδTWΦΤΦΘZ„Z:šOν`ž„ϊΜK(±Λα9 {6ΒΪΫ'“ΡΗ γώΡσRnΈ»ηή{€φhT\W$yKrΰη0Ε±—3ά“ΓoS,7Ε εχ>!>Šω/œ-^4nΥ€εΙS§NΙΆ-ğ?GĞ£tζδ_Κ›lwήresάΉk§rnΟ”φ4Π? lά,―‡jQŠ5R•‡Ξ‹™*©Πω5›› θχZξυg€ύ@ΛΞSιάκσG)B5ΩΡΆtJ¨WΊGΨαΈTα‚σ<ώΨ4Ψgδ|Bπ₯[&έh[ύωΟ.§[Δ]|-bfγU“jY#Πp£)Βϋ…KNΙUrpNωοώ!!SδώTۈ“0u>¨7GλδΊ2―”Γ–Mv™xcΙDχεu\ϋscΜUEv©ΘΘ£=6ΒπΪLw&Β-ς<Ϋ0νέ;ΆοPc•!R­`YNΥ’‚ŠΓΉ}Λ–-p,ΕΖΏΖΒΘgΌXΏ«΅šρŽ/ͺ…„nŒ€κ»φΪk峟ύ¬ό‹ΑΓΒ—Ώόe™=»JD₯α_ώε—kΉFΎπ…/(η›uλΦ)Η½[C—5ή½£Ο‘νFύSή„cΟς8½ž-Z€|ΌςJρx<αό#ΧρžW_}΅|ς“ŸT‘ψ[3θŒ-ΝόJΊžνi°δ•,KΞΫ.«‘φ‡ΚΈ¬Θ+[φUΔ©YΞψ?ZŸ!Σω~ό#9΄ηΜΊΎX —;˜Z*J¨ ³ˆšβFHΕ²ς€tn‘<ςΠ£ςΠΓGγ©b{ ]O}‰›OΔκ)ΕζJ2ΦQ'+ŒYsδI `Ύ¬}446ς`Ϊ'Ϊb[ΫΫΔ–·m"ά²”]#θ‹.6Br•IiΦΘ ΠΦ&NΨΘz¨U€XοΟfjΓXΒLm­m±²€ε±“'OΚZD X˜›'!D— ³J5±(4xU."Q,ΚΟ—u(―nΔ±5*›ω™Ζh½‚μ'?ώρOδΤΙ#β+»ι—Žθd#m+) ΨoaGC{vή•,]ΉΏŒ7Ώ”††cOK½~°rε+ςτσ/ΚUyrE]²!τ&S#‘·€˜ ^ ρ›+άΦΥ#όφ7¦n"AŠσ]Ηά8ΊφΥ΅’_™›uœ+&sk ΕN ”ϊα4Ή]NͺQ…θ{O¦Δ‹εš iτƒ²hE”ΗψςVΠό‘…ζmo{›όΗό‡’Ξ$γˍ7ή¨.#­έ·Œ!’˜‡Ξ7τl„Β©;”.?‘O–§ώ©pη`Y<ω|ϊμθ5#η#iΗ»N_o>'€Μ¨ώύσάςΑeΕΚ‰ζήω#Ήϋξ»…Ηϊ½M[LY“o"t›’3ΜσΟ?/=ϊ° fKvξ PCa!S=W+& »O¬Ή˜4δ+₯”‹†L쓦}žCk²ί4†GΑYŠr€ΜΗHZ ιΟπΡΡΏτΒχΐ΅²δ,κκ5μ°% J,=hτ"σ%iΘHΠΤΨ$^°GpAv’§ŒIΏ]zΊ{„αL2€-ΗΩϟ~ϊiε(γ/ΊB,άiΕ†§t\°ΣXs―_°@~σ›ί¨έ&#ΜG •έG"DΣKrYY.!­ΏαΒυηD·’'›Νς’yuσ6Ω³gχD—˜σ ΐ1}δΠΑC248€ 82$j©C^γ+q+fΟλώ7ΑνΝiƒΐΘx ΤΩvg:5™xz€‘Ι9ήζΗ…ΩGδΚ•†Ϊ£ΨΫ4ήζ\ t¦ΡxΥͺ5RςΚ’\œ#†ΠΧMtˆUʝτ ‚V™S”g±ΙΜX¬f=VLTF&Χ˜PV>|X~ςΣŸΘ‘έ‡eΦ΅₯Rτ†1’Sj²έ$ˆ¬ Ά4ζ—ί’/…³ςδA„πwŽndΟ―sόE@Χ-ϋ=wE;=ya6GΥΐ”@NςεTƒ‚s€Wμώr„ύ¨‘½Ψ4Κ”sp!#Τ€u΅Θο‚ύ5ώ"(».ΆCŠyΚ!g;˜ΕHφ‡v„‡N‰'Χr9:vΗis< ΰ9pΊ½αTΝ)΅Ω*αύe'POƒ»ν6ΗOFό³ΝΡ ‡‘ž*=n©‡ύ…ε2ιϋ¨ζOZ" ϋ•Φ+ώλΏξ–cGφcCΕ5°‡,WŽ0aM-e2 Χ)ΫΈ_² •PΙyuΝK –ψ•χx_ύ “)=―ΡΈΠ™υώϋG…wΊ ›ΛΨ€˜ͺ5<ŠO¨α²΄ΐ!ΧU`£ιΖ-Ψ ότ¨m\?C:γ<•w#ρΕ±ΓΗ$4Λ/N0σΖ¦&st΅A©`aHιϊ»w§­kdΥ&yXτ €¦έnW¬0d†ΉυΦ[εsŸϋœpαmυκΥpVxF l£‡ kΘjSPP Ž.g’' CΖQ-™xΝμΩ³Ει Η@g|―vμŒt”™¨<]wιͺ|—DŸ'‘ςMž0;wξ’΅k^‘@!XiΌ³p0Ž9c€…Γ,„šr,—cϋŸU!¨ΒΜof“.,Ί²/2΄Μ°ΡPnl€Ν’\Ή4 vςrG"ؚ&«Ντ›κηΤ»Ψ/³¨…”uNyaΥj(ύ»δύο{―άxΣMJFςω43ύ¬ζ~遀n?ό\³fάwί―Δαφ©Έ±Ω|4°xŽa9iρ‹½ΰι­Yό_¨qχͺ«Rc₯.;=šΩ·ΠΨq—Χz잩mn“wΞ­/ -”œΔOα“TtηΒxΏ΄8(«Χ“[nΎY-^ldF|ΰ’>Cfš†ΪF){C(šγG ΘL3808Ί[Ρθ–£™/“@@Λ ^ΚyΥ#ό‘³ΰ g«+JŽ‡e7Cυ9 ή Η_†CΝo%;ώ**ʍ|˜ώΡ—°?Sv―A]Τ΅΄Κ-ΛK3e7R~L”(Εi0ζ‚Δ<›l­uƒιυ9YΌx κ*ΧΤΣDΖ8―ϋ ²χμή#sςΔ ΗΖDκcLqΌ μnε\—²_p,ΘΙΙ1υ2(σc"Ž;Ζ©fqζ]Ž6EC­qδΈ.+[±ΫΈ|ωh“‡Ϋ27Ά™”Z۝{ΓΆςΛJ₯Πƒ…;B›XcrιkψIf›R8Γ_Vθ“ηwοSLΊoϋۍ]q bhύΆV.tήϋσ{εDυI©#MΡ•AΕμx6U«+ΡχΖtΧWμ–ς1χE₯=OdύθG>*yyycž-κRσ3 Ψ³g―Ψ·SB׎86Rλ^œμ rΝ [¬υd…Nζμ7Ώω͊?²'[κŞ0X€φ>"ο^R€ezcgP‘ΑώΟM?"Π‘ž„³ά©S'₯¬ !i3 ι6@Ηξ`Aσ†<βΔζ'2Σ χ3ΰΰCEηb1ψΐΎƒp\ͺQk^ΊΌt‚NΏ7ΩoBxΔ|DΎ(G˜'ŽΣtŠ™LΟdk$ΣE…Λ%9X'ά°aƒΪδŒ|O§Ζυ.Ί-ρπ.„χΊχή{εδρjΙ)Ch'0ɐQ&Ύ½;ͺ°q’UΒnžM–Ι‚λ%„‰βkλVͺH'ύןςς°…Eh}sάβ2ΰ€†XόΝώ―tΓvrϋœTδΨ&­{Η‚ς‚ͺ#m*Χ•‚Ό'OύύͺNn{Λ[L}ΔMΧ ύ,ΆnέͺδnN…ςΰτ94Ξ8£¬ρq½"υβ+u‰/ί§δύuΧ]'ι —a*šZ’ ±‹ϋ?@IDATΠΒ‚ΒDη”λ―Ώ^Ύτ₯/)Ά†ϊΜg>£Ψcx^ηηχ‰©Ι™Θ²tιRυoΙ’%Ψ¦Sδ=ψνrωε—Λ 7ά0šGηοσΆΫnSΧ‘ZΫ€Τ" ;'½±Ήƒ6 lBΦ ©θ}„X§“Q‘RωŒP`Τ²ψηI¨p<υΤ“BšY&ݎSy7SΦΔwέn˜›υqο=χΘ7Ώσε0Œ‘οYR,Zθ•Ed£Αd$,n.xKšψΝΞΟ‘|²Τ0¬Λ{ΑRσ‘Λ %λΜ όψž_ȝw~O…}αb–—¦]ž£92>Ί?ρ“γά]?» γ0Νβ7! Θ,\ΜN4Ύ,&MΉ5°±f―—ĜύΙO~’vvρΞl›¦]‰Ι': ? Ά ΛΚςdv ›λwαj™ HΦU.Ώ#[–ΊδdC“ͺc-3L½L`‚§;;;€ώxƒΈ‚Žf€>~—QύΒι·)F²Ϊ˜d˜ ZŽ³ŒΧΑ@υ«_Α!†89f;ιK§¦lG¨ΎœEβ-ΊRφοέGΚϋTΈ]#·§R”»ΔVTθŸ_Y)σ 2»x•έ‘wΧΊ`%Ώ’Ψ/―ί€ΒDζ1ί“G ΆΆVφmί/Α ?vŽ[aϋ€άž@pGίυμπZᐔέΫχ¨ΎΓ,Ίώ£³›ίHτάνψρcrΊ½Eμž"΄AJ‰°όˆΜ›ψwΨ.œyβB¨§}{χH__ŸΊΤ΄Ι‰Τ‘Y …AŸ\–g;¨¦R#ϊΞ,Þmc€]B~·Ί‡b§Αq}o7S?‰ƒξάDpηwΚI8T½©DJήB‘¬:IŠκ„ρDΉd¨ρ•Έ€β–)^P(O=ώ”²UΧΦ֘9mΒ@^:u›£¬άΆm›š—YύUXΩΑβͺrgžΪ»d9!žlή΄~4$Y)i »»»Υ"˜ m³‘³: ;•#bœώͺd"vUζX%ΰq¨¬‡LšƒΤΥΧΙΚWVIAUΈσ±§kl,ΔpD>1TtήΌ€Tο­–½ϋφͺ¦€ε¦ϊ‘fΘόΈG‹CA bS³jo“|GΒΘλU¨§PH^ƒ“›^ί™d‘沋-Ÿψ˜tΚΊσΞοΛΙΥβ/Gh§ό«αΐ‹}*/ΕV‡GdΎI…|zύ΅•ς£ύX1ν±―f’ŒI]7…ύΘ#ΛΖ]{ε†ŠY†πNœ’§z€T5ƒB‹0wΏ­Β%~K~ ›Ω0β0ρyL::mnx}£ΝΟWΟΞΟ3•#J.CYp†R8/W6ΎΆqT_šJΉΓ΅Κφ—ͺΡ=ͺξ<*ΔΣ7Ύρ y葇δ»ξΏί―n©σ{‘Κ<―ρGTςπ0g^"tάH&1t C_˜”ZΈΛ΄qγFΩΈ~­ψ –JΆ·3–Ρ?΅χN¬4Jp„1qTθ©ΞŽ6yι₯—ώ{ί&g‘œ]»“σΜΞlΞ»ΚYBBBΔ#ݝ/™‹>_°ύΫg?Άωwp€ΘpHd‰ œsΞ9ηΌΚ9ΔΎ=ΫΛHh΅;»“wZΪ ί|_wuuwuwuΥ[ΚC%=֏ƒ‘ΌKΛ ςώΰΑςΙ°aς»ί^FOž&Sρϋν²€o™Mr h‘NρhŸ/½τ2Q.@Žχjνθ|χ*€i «ψŠΊΚ({ί››fέΏ™o:…ΗςŽσκŒιΣeο‘#0πΆKΚοP{ rΏΎ‰-`Ε‘DG(—<«Lž‘¦^’7Φ΄•ΓM΄ΉRΪ{a`»pΑ\ρε΅‚,φ£~]ϋRcmrˆΙ[ƒΙΓ0„^©φ{™Κh2εXˆ0€keΪμΉrSQ–δΧSvR_[α5K»<―L5[R)ΌΓZ˜²Ž2dQφξ¬’¬fπM΅ΚtGNQxD‘Χ8‹m’[ž#'L‡3ΥϊμIΠίXςˆ‡μsηΞ"\”Mƌυ5ϋLcŸ·Β°΅!£˜ζcοLδέ&κbϊ%%8 ϋΧDv}ϊι§ετ©γβ)Ύ‘ΊaCMΧ‘«4σ£AGŒ9·Iͺ‡lΪΈZž}v€2θΡt₯˜ —wšl›αΓ?—Qγ&Κ-eΩr{QΠ)^ι»ΓΝ΄χ³e˜w3 έYκSη/ ΄ΤΫ²sηΞ΄ Έ†”‰Lk‘39Έχ xΛΉ›4:oΤvυšr˜η6“Ν ΄Ι…σeΕΚ “¦αΪG’ε{T°aC™’…ˆΛε"ΚπO§Πϋτ΅kίυ=Y°.ε@ΰwώ…δά~ϋν ΞΛ†I8Γ*Δheέ²eΛk‹Mo΄π$lqm7PiZcBCX¨Ίœ„K4…βΝi!S§L’>}ϊ(t£psJίί0θΎΒ1}ξάΉ―B:A U’ν‘οvΜ“Ά€Ώw#τ εΊ:¬hXQ ϋη1]―"xΠςP†h5σφž—FŒ’ Ι£> ψΨήB9Θ€ω–°•JWθώΑM*‘Α† ώrΨ)Φό^@™iΪΈxΒ€ͺΧJ‰χBa$/†qe˜δtΥ0ΰ9qβ€ώ*<ž.7§SΠΌβΒ~βδ)2Χ'­ ηh«‘eAY¨ŸΩ2ΌίΕi@—΄bŸΜ„AMyyy:da}XΛ=Ί}8'–Τ›‡Ν²-ΌTήΟψθτLζZ:"NS*ϊŠΏ•+xΐAζ5HH‡Ο1κ]¨Ž΄‚5&„ϊZ½pόυΰSςσόGΉ₯ϋ-Ašͺ|Γ/!ύD’p@Ÿ#¬ZΉJŽ> ΩΝ;Α²8"–[”η™’aΛ^s™6mŠάqGoΙΟΟO™ρΗξΡ£Geδˆrƒ§#Β_ΊΰΘΠvΑā«?W_ͺ~γ-.s†t‚ρΪκύ'eΖ?η,ŸΟ—2<ΊΊΖΰD΅άΨΆ}›Œ=V…Yβk†ΠDAγbzIΛ>Ό_Α= δΆΙ’υ3ΦΛ θuΎϋέ呂LΪ‡ΡΉ08j‰ΠΆy0¨QΌ»Β½†6Θ3› [ Θ4μŸζ₯ Β §Sκp@3ξωFŒ)C?"6»KΕ}Εΰm‰ρbˆœ―gΑ…D&’ndΐ ΖgtΘώ½σδτόδ§?“Ύ}ϋ4YύΈnrnϊτιςαΗΓ€YΎ_ξ,±*ΗΡFΊnγPΚRkΫ9Χ,Η/xdόΖ-*ΔΤ/~ωΛtHΟjŽι6: φ… ‰θȞ;Ξ{n0O]—ΫυΌXέ(ξ";τ3NYΎtΉά=WNNN͜YϜκΆ¨Σ„ΦP/bΩ`Lϊ{θ=7ϊ¬ο'£ψΓΦάκFc†•ͺIΧYΣι|τ{Ν½ψ ―ιχΠߟ΁E€Z[΅b1”·a£xcΥτj±αωFξΙΰ$˜av‹ !¨NΪ’ŒΚΚΚΔ‰ΈZΘDΌtNš‘ς€^ϋ[·l‘ρΖΛhX­š±ψ½«UžtΝ5IΌϋC˜©χθΊDβ]Χ‹J=Β¦ΆLt”;•€Q]ΈχŒ xιUx7,’‡~X…¦³",SΊF‚ϋ©“‡ξœΛθΙ2tθ07v€ψsJΔ”{«2 Φ–σ±ξuυ©?ο…A ”ψ†¬vbΛ4ΚΕ δΥW_FΘΕCςΰHv6 iΤ—τΛΧ8 ωΓm’‘ν=rT~Π©@0ˆ‘Ž%œVaζlIˆ iε7I›|―ŒŸ8InιΡCΪ·oŸn‹―q?Ό .\=»χˆρήMvXκW+ΘκΜχeš2Φ)φ”“'OͺΨ:ŸKߐζ8ΐ~ΖD9N%ΡΨ±ca9θV±φ! ”D%‘ςί«Kbΰwœe{*χ6Ι‚ _±l‘ xξ„ότ'?‘.]ΰΉ…rκέ·eMσEσˆFvS!»?!χVJ–Fk [Ψ‚Κ“‡° ψY{ȁ6';w†’!7έ6aςu φΫ·nΗα…[,τ°ͺ吧Ξl9t0)Ϋόρδ»dυΚ5 „†‘ι”ζ@}8@4„½»·"œΜ½p(Τ|£pΉH4ΐΈ7W!nέ²H—ΤΎΉ>τ4Υ{8ΗMlϊ΄i²fΛvωvΗBθΆζΎ•,GΡ1¦{‘S>_½M¦N*?ψΑš€{θZ†{RΒό1FςΛσ€ΰV0W τ»žωΛDω₯‹Εm–‚~1bί½ΰ <‹?ώρ•s4υΌKeE†ΊνŽc­ΆhΡB1Ϋ°&J:‘chMφŽϋZΊΨ‰ 1{:R*»vΜςΚZeLC™“μIσοβ₯‹2aόx™‰0€}[ζΑ㟑xBΩά³ΤV_o›ωŒrs±y,’v@_|μ±ΗΔ`U}χΥ΅ež`Χu}Έw>|Έμίs@Zτ+Eθ kν†4!uP½μ€ΑίΪ-ΗΆω‘Ο₯Σ‘^'UΗEζ)ξ,t-ΗbšL$l*ΙCζγ4₯ιυ@^ƒV­Z!Τ}xŽQ©ΒοTͺ‡cμC»vνBδ“‘2yβρηWŠ)ΠθλεμlηHτ¦Ί8W=0ΐΰο,.ƒUΞX(/Ύπœ0ΚC}C9B“f¦T˜κβHθοwo!Μ’Ϋe—Ύ0€)žœg]ў΅ °"$aO ᜸‰Σg)#Ξa}ηp8Rnξ ε{8Ÿ7¬ί λΧ¬ ―wθΎNυΊ—C】·σZϋeυŒΥ²~ΓzeLS―ητ¦˜Ν( œυω©›ΧŒ˜0kώ !Ÿ«―sΑΖ? σΠο₯+AΫ6.d±MΘO’Lš8A\ή\Lj@₯1˜A'΅L\΄ΊJΕ“ΧF¦Lž «W―N@"S‡$έGt?6t¨όΏύ ‡!M;@‡ώ°SΆάSnƒ2 a2b1ι& wυδO‹έ,χBΌΗο·σJίVω²τίι ψϊλ²iΣ&E2yH~jY˜(υHΣ{θqEt6ƌ}κ©Ώbs1Z²‹ΪŠΉ°Ÿέ•Υ‹W.θ²Œ φN†|2"δ“΅ψnρJεγ!ο"„Τ‹«πU\ηt¬»ύΧY#ΓG–NEi C ²eΒMͺUͺεEχB„£8y sοD…Ί§εCΈy¦οr€Ζ4U{ͺΔ e» !λΫ@ά3ΰ΅gχYeߞ}rϊΜi•aƒuΣ d8 ε8Η.‘«^} Όχξ›bwgΑΫͺŒ±ž¦·UXJ" Τΐ{Κ£Κ¬²^θΧ;0G<%£FΊJV€ewέ]²{Β”i‘ΨA#Ϊ£ή’‘ΦΜ΅/ρUψ‘%ˆ+>sΖLXˆ•Ώ¦S]Π|ΪεΜ‘ͺΓβ+s.Ÿ‡fu=yƒί)ˁŒι.Tτρ“* οΦeέΰΙτOM˜μt‘¬0[α΅j  ΣPn7¦3†0N@žμrΩ°~-BIR?€ewB>jΎp2|δ(i‹5w ǘ°θŽPk„”laΈ·G­ ²d #7oξΩ5-W=’_XWŽώ1<ΜσΟ?/γFNφΕRvwdU†‰―)ε ††4yέ²€Ω=…b0δ₯η_–χ?x_ˆΖAϊY—¦Τvͺ]RθeΛ–Νπ‚^(ξœVπXσ’Ρ#ΩιΠΟα`dDΨUΞ¨aH¦dξ3zό²Σ¦N“>όHZδΦB‹8`θq΅σύ0οe8ϋΫp¨Y$ήχ>¬Π6™·_όœμIσŒF"cǍ• £'JQ§<Ιn‡0ϋμ³2}κ8Ι)ι,ζ‚»Δθ‘›ύ(†4Ίή°_¨yΐΰk'Φ"θΗ³‹εγίU‘Ή·}‰};Ζρήu­ΗšϋυΏύMΞ^Ί,χ–;eŽιI0{Οέfƒτ)ΆJΗ²ωlΤX₯χb試΅΅λO'ƒeΛ–Ι‰#'ŠΠ£tΧQm$ KBΊzJ(Λ K—.Mϊ|QG¦©­s]Rv&zύϊπ‘τδ³ΫνπΦ1 CLΩΖλυͺλ|FηΑΟι9hΑΔψ—kW/ƒ7V?Ι΄ηrUˆB‚νΉ"‘iΊ pΔΌ%lώυ2sζLιΠ‘CΪj1μ ΙC9φ†W£%dθ’•«ΥζκΫνs₯ΰς=PjpŸ«h±Χ„T+βu}uύKϊ !Cš₯fίyωtμxY²tB?=’Ή(ۘ4#NP:Γ„ζ€nwŽ«cǎΙxAύψ#x«ž“¬’ž*4S&οθ!ΥυΠ=¬!Υβ³PαŸή]»d›=²jεbόξοϋGΦ«Fv²•NAθΆβZe2<³.Απ©[> /a8§Η{CxΕ–₯ž¦9<Ώ:α€w΄rkΟ[₯G[’]ϊ™jp<:yZrJ}Α ]ξΜω‹Ζ46―Eφo? gNŸ ζXΟηΣ Πτ8@ΩΐDyΙΝν’%KdȐ!²yΣρvc ›μωΈƒχ5€#‘γ θΩέΔirΑ{ θb/Ώ ΫwμG€zΗPLZN©/ιΕΝ’L„±βe”wl½J«Ξ0ΆͺΌν±^‰0§cǍ—›ΊvUαΈtω Ο=΅ŸΤό‘Μ^·n½x <πΆΠ/£ξυΫΧε…Πό|›RόА¨rD‡Τe^χΉτΕ&Λέ/¨›Z»Ζ4vΏdΪΈOγol’‡>Εθ›»!€fΙώύϋ₯   ±§δσΊ-¨s˜ϋθ3εψ½ο~―&”šW"¦αΖΠmE§ˆ₯Π™]Ύ|QŒ2Μη@vΖZΈα‹‚λ—KƒIW Ή,ƒΡΓ·hΡβϊ7&ΑUΝ;’ΊhΡ"yηwΔa·)d\§Q­yΓYSi™X΄ρ»Jmςι†KςΖ Aβ:@·›oVϋžΠ2“€E_#1”~ž# ~ˆδΐp3―‹aδ£γšυIPμψ[ΊεάΡK²~ΦκεG‘€€3¦ΡO^μ€x6,Z !EλD.ψ—ΊBIΈlώρ ―eΛ–)/ΜΗ‡h_ΣmSU΅O)-άΎ|„ie&FΎlŒ63Ϊ”3„e°‰7·•Μž9Uϊ!ΆζM7έ‹‚SΎ φ &.€ψyύϊυ€°Οήιb·šεž6ωΠφyπό':'=uπ­ΙΎrΎaβώ‹0uτr+„aM+ΏYζο9#O½π²ά%ΑΊ ―Ž˜ICυ%ύ’²Έv\­짟~&σζL“¬œ2ρbsAωkΐβ(’Š£κžΙƒY[@Lω½Δ‹ƒ‚sΛ Ο?£ Ž k©I‘t¦lc„Y1ZOœ>SΊ”ψτq˜νvΆε¦τές¬²ιΰ)3f,βC·Uα…2αk,»α­ :xθ \B« Θz`rυTvΓgƒ?b=Έb3ζ4“ΝˆPhΥe’!¦SšΧr t|ξέ»WΖΑˆbτ¨αΚΛ5»βN Ρ΄‡’‘ejδΈ^\›S]ίωN°&7f΅ ιMφ₯2~μ… πwίωŽτ„žž|L‘t© ιY°`L™5Gn) H ŠE2i9žη4¨!CWΐksΦ,)//O+λΙh’’^ΎF­|߁ŒHX:an5{L’]ξ—5«‚‘žJKKλIQϊΆ¦Κ­[·ΙΞΫĞ…ΠpΨ Ϊ4Tv‡r2hΏθνfΓ† JΏ₯Χ-‘w¦?9@αΈ)S₯ ŒΝ58ͺe]TX‘‘τm`Ωλό±“&+R―ή½S~^Υλ†'N¨• ωHΜ³Tφ-’μΆ4VAθ=nX"1"Υz€…†δ’Ώ-0ψ™3}ž=αg?ύ™tλΦ­F•kA^%Γ+Γ‹Νœ ½H^K(Σΰάρ„Ξƒά 8š^€jΟrY±b…2ΐ""~²%=~IχΌyσδΥW^‘σ0Š{HRΤ?~m\ΟqΜΫΈcύήmμ–“ς ςώηωgιΪ5ΉΗV(ΟfΟ™ DΡΧŎ°EE½β„AεUςZΞΤΩ/Γ‚Θˆy]Όrιδe™1y¦υώψG?V‡Α‘εΦ™_‚ά iζΈ‡Πa-rs₯ΤbAΈr ω‘sΥ³;Υ«6d΅ ε6‹”ωύ@‰«Bη’LMG½2JίW„ΆηγO?ϋL&Œ%žΈ z!F ,…yΦy#Ιπ*{ο΅>ΟΘλ-Y–,9²o±<σΜSς­o}GξΏ>jˆω†Φ+Όrσn]ŸC‡)cΙEΨ/ίΣ2Gn)°¨qkCΝ%Ά —œepJΏ§ Ɯ/Λ AoˆΗν†γiO}[“|ί²a±7ν}Λ#§3©ƒ“ΤΛνθ4ΦmWa***’VΟ•Tš}=HιFhrzΑ?ώψγ2xπ`eX“••%~L–D’qc€π3―mΫΆM†"”Μ―~υ+±ϋ&e `qΧlŠκhστΟar`Ŋεπ–Z%Ž@Ι΄f'ΐδVW fΉ‚²ΛθiI»RXΌN«A8bΏK§†q@Y*8Ή~ŽΠoiΜδι!~Πή/ύΛνRθΌ:€“^Ž4¬ΤΤzŠΌ`δB ΛJ˜T«| ©ϋΪΙ†Ν›εύαOjΡB9ΗD^§ϋ¬bEΚΎ„Ž«#GŽΘgŸ.ψύοeιβy’SΦM,Εχ S[šΒ j› ΚMΐZmbτw{Ι}β/h+S§Œ“?όα2aΒ…Ηώ˜ξ“`•ΰΧDδrΩ-rsžE\@0‰„^YΙ ΘΖ%οV„ΈδσΙ\ΐΪ2±άt ŸG‘K—/)Εz‡υ_ pΩ`β;Η%U{«€uQ–Να·Cͺ>‘ϋϋ5©Έώλ_Ÿ–‘½‡°NΉβ,ν/ƜΧ4–”A•†ήS&ΐ"Κ{«P!}ς y±­―]Kh:²>―e7 5Ǝ#6 ήΩMΨzD21;*Ϋg›₯5B„Œ  Π™šz;(&Τς’ηΈΝηpζδqζb]‚υr£·o\ζ ‘iι-vΙΖU›αeΈ·*—Σψj½EhχΓ‡ͺΎ„Pu`M$… vybsfΛ„¦>Δ¦΄ŒPl¨αεΒ~ WSχΰ²[₯{ΎEGbΝύUI_€fY47‘Λn)°ŠΕl’αΓ‡+!”Šν€λΔϊ-|'ήψ–Έ.©Έ§@!40œUΛ_g]ά―PήΫ³­RŒƒπfwΛ‘}‡δ „ΨƒΉŸϊηTmΏΈ3>Β°˜V­Z 9Όrhf''t\ ώ¦nˆΘ ;ΚΛt‹'Pγr!˜τΈˆH1QΞ„΄jΎQκ₯—^’γgΟΙ=ε.ιΩ™YΝΣ†’ΑF]±~¦AΝa„₯πμe€Ξ<“ql]Λ³_xI\Ύ"E·ΔΧ¬:”](ΓΒιzΈ—*< BMήζ—’Άy2ϊσ1ςξ»οΚργΗ“’_Ί)T„2nŽ3»,SdQi4»Ιj’έδ JEsœ ^ΏAάκίΣο‰Ο=Ύˆd΄|ω ₯<CšαΠ7·D8%θΊαtڏ†\o(‚7θ4F ; Λ±:²δέwΚΛ0 άΎ}»Κ8ε]mΡνDΤ ‘,Ση-”εΪΟώ΅°€΅ε½λAY Ν '~„œ’ٍ׭[›”r΄±œbί#rδ%Kžjw‘]‘3Ft›Z‘\.AΟΕ2M({ωςε*b og?JΆ”4Θ4zΣαν·ί–ίόζ7Άm[ΕoΒ·φιΣGŠ‹‹•₯τ榇χΣϊ•ή„„άeθ§ϊη’•«VΚύΏWθ]¦σNΆΖK$z5_˜π†Nw@‘"`dW‚ίΈDΊφœψΎ.0¦ξΌΉ3εξ»ϋI§N"]P“ΘOχ kN¬Λε8Θs–.—6…ΩrwΗ<εζ…²›‡—Ήgψ—NΧηy£€„JυΓ»‹a]οΏ ŸU‡~zψαo(τ­@FlHΊΤ—τKs@·'Η_Ή± aιE³Ε—[!Ύ„ur7Wπ’A#V9š£ yW+¦ ΞbΙ΄x$ΫQ$篐Ο<)‹zί%>ψ€΄kΧNL&x‰W/’HSK¬37ƒ3g̐+VΛ}@δͺτ™”ό‹7(KΉiΉ J―υ‡]2bδHiƒuRYYYZ4 ΓΡPνΚW”1Š£ΕέW½ΖS°E8€΅Ήmκ`Ζ4 ;šNi„ΚΑ/.ˆ‘c& ΔΔ c€ γό}$Σ ctx4©qƒΘj‰ΔΘL.ddί,N{X”1£>Aˆ©ΕςπΓ:ŒδWk ew°wTΚξ»[εK1P!ΞΑύkdή™εΈ†x= ςΑς*?nœ’αάΏκ5@dJK­\¨Z»f­:xp‘M5™ΩΨ6bυdΟΖA|ΆKιΊvλ*fSZž§Vj|mτψdΈ±υPΊ³Š$ΓβGΖτ]‹B/:4ϋ€Ι ΤΌ„ιΫ ‡²νjέψ€Nœ«8ΏN›>]–έ ΆΝ—rx¦FB$Τ‡KZ–—zŒrK‰W&¬Z«Π΄Ώυ­o₯ά<ͺϋ=ί©`π²nΕz)λ λDtφΩ¨Ψ1Τ§1ΒΉ‡Γ ΐf»Q nΞ«Ο,{η’W_zUφ \Γ£<*Ω€Ω6ε5Q8lε½Ί?ž:uJ–-Γa‘Εύn Ϊ H1_O«^ :w¦•!ΜΚeΫΦE²χ9ΩΑυs,λήΠ²4ΟkΚ”Ι8kyGΑk=Ψά£ŒΗ3±jμq΅’‰ γέadHγœIΫOΙsΟ=τ§ŸΘwή©φɚ–†Φ%VΟi:šw ¨ „~π₯”φΞ:"ΡD#@ ε”ΏΆ,‹ήPŽ5^9ϊόύώ^ςςς’f_ ωΕ³"kωζ«αΓ 8³‹4*ζ<›ΐˆό+QΞ2«E9­άŒΠbtΊΧτθ{Σο‰ΓΆ ηWžίNΊίΗ}€ ’Sy§dϊΪAGβSr78Π(])qΰ%¦Ρ½>J»BSι–g–WPτhγdι!¬S<“ζωfrΏΌ—‹ά©€‚pB?=ϋk*ν·Ύυm₯X5"mh{Δ“φtٍγ@h;r\?AFό\ŒπθΘ­ΈM2 Δ`―†0ŽŠ²¨6ϊkz₯Ϊ,]C›Oξ•²pΑlΜΣKδ±ΗΎ%}ϋφ‘‚‚•Ih]jΛ5•λϊξΐjΤθΡRπHG,ζΝίvƒ9R› ζCΩPδ2ΘΝ….Ύj›Rή—””|mm”Jόd]t[Q‘xδθ1Α³Ψl7„½΄g>fΔθ΅8LπŒ> TΆ₯SšΊ‘D:‘ρωgŸ}*Ǐ”¬ό6ιΤAuεiΕ;΅ŒεηH&δ eOŒή ξJϊ$Η^(g―”—_z!tΘ#<";v«΅i†~νEOϋ @‰Ιρ:₯S ‘7 I–‘²g αH:Αqʌ™jŸΛpžιτuθ6ΪΏΏlΩΌN€ZΎ~cC―pψAž;ςlβΙwΚͺ«δτ©ΣiέACωΩž;tθ0–‰+«\ΔDοt*Υ#™ %Οκ-•=«Λƍ՞/}°δ±– D’QyeO­Ή-\sچC:‰­Ξ½z¬χΧτΘθΡc”“VσζΝSfoyΝ5ξμΩ³ΰν;PΞ<'·Αζ&o5t|5·cΕψΖ6.θδ9X&œ²ΫΈΕ‚=UΥ‹ ς‰B&c˜κ’™tύϊωΘs`Ϟ=²bΩ"qZV5ru­Δ!BϋΊΚ°ί›#+W”[nιž:dέ‡OŸ>³£δΝwί“€Οƒι-·ΘΉH _ζAΉhR5ΐ£LΪύεσ/Θq8@?πΐΚρYΣ­ΦjlΎš>κ &Lœ ―ΏφΊ8]N)Ή#W²[»ƒ‹ψHu7Κ#l8@Μ*κP^ύGORγπ³PŽα¬¦©±u‹φσ›α@2gx·J6PΫHw4σΟ’hK–ΜšΏN~[UhΜh–™Ξ»ανΗ»vν’aŸ|"ƍv‘Ψ ΦΙ yŽυοW†‘‘L §·φ'5]pΦ·ηΑΎP˜|rμΐyζι§dϞ(yΗ(*L‘υ=ΟΔϊ%”ζ©S¦Θ;οΎ/εω~Ή»Τ&yp(„]rΔtά‘¨9% 9‰Žvκ’O&lΪ)ΌΎόΛω?MnOΏ~έzΩΌv«tύv;1Βx\…ΕŽ“λΚƒσφa&θΘ}eNY7m“lΪΌIhL“™‘TA“TM“Β˜FΤ5kΦ¨π4p Β`„Pχͺ'/αr€Λ3nFιMιžE)ƒχΫΏM¬\-«Χ¬“o>ϊˆάέΏΏςZ`ώzœςs:%t»q\RƜٳe$”ΓU{Ά#x+1ωΫK¦«žΣ”· :•Σ ψXΦ“«#„}Β?ƒ³!ώ²$ΛY*¬”χί}θpσείxH)—<xμ ιΊΕ’Κx”ΕΆ£ΧόŒιΣeλž}ςX‡Ιwε *υ#άTz£@cu~etuaΉnj*όŽDS©xτΘQ„«t(…zpv«_ΞͺIρ!νΝaxμπ1aισΑ‹&š8ξ˜(ψGxξE‹+Cγ λVŠ7P"Ω-ξShŽκΰχΗΜ ²Z)”—dfί$v ‹Yύ°ξ―6ΊDcj*²„νEt"Šέ౎…Rcπh’°Eh¨γΆdJ ¬¬ΩO―ΌIŠ<9› οUG γ…ΠΩ;·ξζ}Κ‚!ž"ΨHΒf΄»;Ο!Ukφ d‡RΌ…A^ϊΦ&ΐΚ ¦­ˆCκδ ΙΝΛΕω* ΥP-k#ΒtHάfΪ²Εdu+t3YœNg“—Z>wόψqrΰΨ ωNΗ|ΙΥkξˆπΏ~™°Ε)†θ@Τ«Τ)­Ψ'γƍ•Ÿόηp:H~€1Νk’€9B>ώπcq{]όξbργ@ΩύD’‡uͺ΅%Ωxh;όw‰ŠΦE€€_Έ\ΐpσΗ?ώ‰tοή] Τγ[ύZσK3°-.cέFΟS§Ž‰§Ό β’θHԜER½JV„χΙiG’Ε@Ζ@ΚΚΚΎ~S‚\Ρύ–ό’αΡG~(γ&O•J†φ+΅ύ:ˆΎi)Ή™¦KΎU!κNήiΧί|Kφ!šΐ7q.SXX¨Έ€iL–]EV†.Ÿ ύT²²½Ε,†vŒdWC^”₯v?BΠυΞVϊ…₯σ—΅c€ότ'?UNlΗD䙦ιβ:eφSN8h-Ζ ω-T–«›Β†ršέsΝ—‡dB 3ΊΡ‚5]Ό7βΟέ* ’Όϋή{²mΛ:Ι-ι,tΦ4ΨσƒDΖLG!ž€ήL8μg@Ηβΐa9ΌDήοM¬έ7₯ζ{ "‘Ηo]\Xδή7ίz[r>aΪ€轉„H£ιWς‹:3’zZδΜ₯€Μ˜·@όΏόF.„Σ}P?“JοΊnDν£]…ΣggΠΑθ Λ'L!±I\XΓ¨ΙDœ’–y2wφ\ΉύΆΫ…zrMcli|)IaLCαBeζΌωσ„Šτ²fΟΎ}ϋ*ιLΌοzIΞ lώλΏώK:€P3¦ΐŠξΎϋξK+(―ΗΈ0―Q‘Ή`ήlρζΆ’ +QiΈόN¦„ώC/]³G,@§Ω»nΰΌW)cšΪϊV2Υ.š΄† >’@ΡΣzΒ”ι€Rε±vΉΚ#ΜΛXΩͺ<$ΈώH&…©—·β!^4?K°pɢʐٻΟΙ χ+”šο~ο{ʐ‘οBΫ)υ8’Z5mEΩΓ1‹—,ΝHYΎl‘Έ<9’Σβ^Ιτ΄@˜lά9&ΔΖ’zdƒ*ŒΎΆ@(+h*= νςώ²½2±εεεPήΫRšΧ‘δ#Γ‡9tTμ>ŒiΐUŽ±0cΒZ]f9²ϋ€PιVTTfιΫ“zό³τ`$’&QΕ΅ΛjwψxY!€ŒΥ.ΆFΩ_-KcΖ%9PŒ! ' ^W₯\:²B¦M”±ωžϋ€_Ώ»”€^‡Φ/f€Ζ¨ ]·έ»wΛxΘξΚ\―΄˜ΰI‹ƒš¬cΩ" Ψ₯Θ+“¦ΟTΞ·τθ£Ϊ'O1μ‹ΤlG¨›³§Ξ‰ σ^&ܝͺ#’›Lb|:eΣΜ ν–Ήz$·”F›μ‹TR±ˆή¨Π⏆ιθίυδ T ΌΩ*aXJΣ€SD£3a’tΊWϋ ΐ~#έ uρE D‡΄ ˜4ζ— 8¬ξΡγV΅/―λΩDώ]ϏDm2d°Œ=Qς+s₯ʞR‡b΄ςt5Γ£ΐ4fΕ₯/‚aNή=χ€<υδSς³ψ™ά{ο½5aŒΣσA˜f–Ί_Þkƌi0lκ"Πr£?ςΡΡ©C69atU*Ϋ׍‘mΫΆIYYY˜5ˆΝνšO,(:οΌύΆ,ί°YΊ"<Η]@  ε%S4†0σdφ|oΩθ…ρψ$ί}6v‚:ŸωιΟ~¦ 8¦BiΕνqKš΄wο^ 0Ό#3'ΟBhή)B&=ktQbeCΟήξ“Σ Ϋην'Ÿ|R~ώ?—;zί‘Pˆ5qcΤ ζYќ €™Χ#E0¨Q΄ήΰώHό€š₯6πΰ°vBLQn§₯#ΑέΘε‘ϋ- Ǝ+~8ύY$»ς.10¬“wλ³Εh °ΘΥελ9‘^Μ5Hΐ „¨ΚE(οΛ"ΔΝ.yόρΗ@–4ΎžGb]Ρt_Ώ^^{υUΉ‚Ev2»4Ο’‘d{‚6‘j ΠηœӧΔ"'/d"hd²εΫχw /C#Ρ h±dΡRΙmž­ŒΔU{E"γϊζF`±‘―Έ€ΏnΝzΩ[΅WΣΤ7‹DΉ;ό„Nd4•‹αΙΙτλ_Zn»ν6υYδm`ψ›^Œρώόό|ω˜™/^,œΨΣ©α o―"cΙ’₯rαβy1βRybΥ,“žw쟀τΟ”L *Έ|…²pώ|&ŒtθΎ{š»D=ϝ;οΩ‰ς—?E&Bρί­hΠςτ“–΅kWIvYq”= F'l,%«_mdq•Δ ΟuHlΘξ&ςΔ_ά‡Θsε‰'ώ(οVqϊ'“ξ—κK ½θv€aΖDΖ9qqΒν’e3DU©―zš e–Qn* ΘΔ©ΣΠwΦ¦g£_•³ηΞ˜ζˆΨά– 2 Χ’ZΨΦ·xάOdš .ΚΡcGΥSμλι”ϊ•γ4†δα*Q1Ÿ‚_²^0…ξΚ‡Δ˜s ΠA†(™ΟώQ]6!‘lpWˆΉπ.ρ7Fζ^ϊΡ;ςη?Y>>\x€Ζ€e7λ›jIΧmφ[φT%ΖƒεΰZ6Ϊ­ΔόΉfvΓ(€ΠihΘAη"hΊRί ©ξw ηΌnν:§‚ ωΙ©Δ?2₯βNJ9†;œΧ™4 κKϊ₯Ιr@χ*)·oۊΗeyt:#ζ †¦69rεΰώέκp/:e%OlΚG’ΏMz5 φ»B~ΊqHCrNΛrχ7Zώτe“2hΞΙ*Λ5Ÿ7!\Η‹/Ύ(“ΖN–²ΞERΦ/WΌ £·€¬|*$7LΘD›Μνμ“Š» ΔξΆ«Π.CpΰΗώ–¬m™ Νs½:lίΎCvlί‚υ@δ°WΙΛθwHξ ’ ƒt»;[hΠΗυ?ϋF"%=~ϟ?έΔyꩧdΓΆmΏE@¨΄+C?£4W(*Šΰψ`₯MϊΑ0e Βuώ4‘‘4&ΒΨ<#Oh|4ΰΉ2gϊ)ν\ %}rΔ]bHMΖUΛ"3B¨δuΛ’ΚΎ… ωψ₯^’OαΘJāDΰΧ΅, MΤi.ΪΘ1ΠΨ †ΏD‹‰A7S€°Κ«D(}X£-…ƒbh›^Koϊ{μ8ΐv`b‘‘Ϊ 7ސwή$›Gœ%w‹1»+τp0WN£±£+ς%UK<ΤΓ`υ‹1χ6„ΉΉKN>%ΟCž|όρPu昈γχZ^θ±ΓpεC–½Π—φ)qIΫl†DS]ϋ@‚}'<7σAί§Δ&Νς2ψÏdήΌyŠRΆA*&]/†ΐ%jΊ+{Hθ΅βbψŽNb00w:δάΩsΒ°SLšΖdαR Ӑ™„kέ²e‹βλ­·ή*‹Epm‘Ύ)T8uξάYzχξ-3gΞΖYoΧ]}³IίΒ-L>,+V,›3[y΅7,ΙΈ“†π„‚Š^Βv_²Χ -.ΩηιjθΆηΈβϊτΣOdΔθ±βr:δΡvy9Φ†8ΰ†Œ©9-]Ν“x~γΌ―/y€³Ύ»Μ ε؜NέuV^ϋ=4¬‘Ηψ#…ΞE:uϋΕ“ζtΩWs ΄Mˆd1eΚT„$ό‡ϊU’•ίVLΠ G0€δ«ΪX$κΘͺ¦ 4ί#ΛL[’κΒ‘eςρΰ·Jo<ςΘ#j.Φ΅‘<Έš;ΙϋQSgΝ†Wj–΄Θͺ“‚κD«ε˜/e.»δZeεήcΒXΆmΪΆ{ @ΛG³'θώwαό9Έη έ’ψK¬3΅p £p΍@Γ_„1 •νι”ϊ`a›A’iNœ8 λqPr_^ρψ;Β`›rœα?Bεx΄$BΈ|'¨ΧΒπΛΜκzKb㑃ΛεΥ—žEΨ£rύ`mά£ ApKKΔϋu]¨Ψ›2ušgΉ€΅Ÿˆb±A₯ ε Q‰0}ξ|Ήλ»$NʝΰgΚΧυ«7bί FxTγδ<’Ί05"0$L8ΐ”fΙ&xpΣΙ‡a·)́P0؝ΫΔSxsΠ±HΝ Ρνθ<Έp&BImΨ°Aι*τάJSSϋΜΓΞ I―Κ…ξEmTΌWΝ€j­8K&L؝wΦ8Ζ›Άϊ–―ηEφ1ςψ•W_‘ͺϋ€’G±δwΟ‹ΙZα.__B£qλƒŽDΓΦμ6@°v‘fΆQήλ! ΚγPό~Z· ή‡‘§–λΧ―“ ηΟ‰ΙU γpwp­΅]·&ςή]‡ΝY€Γ‘5rΰΐ)..Nˆ~:~©cϊτ“Oδ“α#ΔƒG[€šk Γ=pjΡUŠΑ;Λ"`ΐέeΉΘΊ™2qϋ)yφΉηjΓc}SΌ^oάx¨ωFcΩ³gΑ1bœ8Oήχ Ω±s‡όψG?ϊ Ι:V$Œjλ6£‘θπΟ?—ισI_„κΉΉΐ,&¬b=„AϊU·RG@ZKϊ½ b”PBP"€F6!0C²‘θΧί%Φ Ύ¦<ΖΞcύiΦ¬Y5žΊκBϊEq€FO§O{ΐ’‚ς:‚ΜQCαKΘsxQ‘hλϊmʘ&‚%€³Jrh™± aαŽ; F¦Β΅’ ^2ΩΜhq)‡3:ž5Υ€η_*ΧaΓα°EΙΟx‘θΆ ,' Τ‹t£‘Λa• @„ ­μ7Ι0gjώ²N .§Ÿ~Zμ= ε½ €πVΥ†4Ίβ©φΞ†D;r‰γ.qHiί)λX$cFŽ“ƒp ˜ϊdiΟTkΦG#Ž«Mpb±Ή`hh!ςc,;Π9 ±ωJpn±CΦU{[Η›ί‘γ—‡V/Ύπ‚|0μS©ΜσΙwZ»₯[ΎEι c©Ÿε‰^ο˜‡΄|»•GΚs}ςΞΰε₯_Pσ›žcu;‡>­ΟšoΤ |γ£gž —°§o~w±ά⏽!¨šTποΎJPΑς€€]‘Œ9^žώy…j~iςχΞ°gΛpˆΫΜ㏑ϋ)½_οΖ(\SμBy>£ Nm>Y΄b«ΆG‘€t–υε€[<ܟ‡ΓgžyFvlέ(9ε·‹)Žjη|ŸSɐFs=’ϊ¬ŒD.θ'9₯eξ¬)J―²nέΊΰΌ-Αt*Z̘1C†}6\Ί”ζH―b+ΦΪ±ΥokN6ζrμm g©;€¬³λΐ!ωdΨ°”DΦύˆλ#Κb‡ί.6?χ¨dBπ-ζ―(Χ„’ώ2ŸlήΈYˆμšl)iiFh_Υ>),,T±iΛhBΏϊ7‘‰BΌ‘IwL-XšO2>§'A]³γ0ΌmPϊ^ΉŒ*QD%kBh0@°y³+`L³RˆΌ“——§&΄¦ΨΦ‘­¨ΫFœHίFœέ£'OΛύm ε– < ‚pΚρڐ…ΪT?sδ)υ-&©xιή[‘)ωxŸ OΟΏ(UUU Δνv7ω>oΗϋκΝ;eάΈρ2~άh%>s+n£―-]²Υ‚[Ύ€\eJFΩ šΥΑ,ΤZ€ΎΟ@θ'§£D¬ΗVΛόy3dω²%ςπ#Ιέwχ“άά\UΛPή¨ Iτ’i§υχτYs€c‘W*aΨ«ΔBμΌ}η\‹,ίύ₯L:UZ΄h‘ ’5}±’'YΚΡσ;Γ29ό@±Β˜†]7Μ π~ϋ‹ΡfT2–½\Η†ƒ¦f‘Mςvφc&έnρ`‚K€‡™‹>vτ¨Q²zεqyσ$ΏΥύ’ CΘLBΝ«Ξ€χ₯‰žͺi€$ζLGΎZ{\β8ΆαwηBω‘@IDAT.{άΥοΉοΎϋ€²²R΅…ζI’ΧξzτiΪ<Tš©,Ο/-|F1Η•FΣWΟ©Άy™»p±<%Hλ6mšόڍΌΡγžή|D³ϋ­5ύOσ.’ο™PZ}€±ΖΩάξ=»…(·š†H–Σ”ςJήX~k™Α0l[6mϊΠLnτLŒS-ο[ΚυžΗό‡PO6wΎμΩ½S9q§ιΉή©~mΙ,].½›εH)Φά\"$ΒLKhPSβb$φs ΛοΊ j=ιώΔχω{ψβΏΈψ…TάQ(Ω”1#ΖHvQ@ {ΔW~…u`\εž–Eh<†™2χ5ˆΕc’ε‹W(Δψ_ώς—rΣM7Υ¬‰γ5FY. βΙm΅*t"Kʆ]ƒύΫ GS’Ә2 ²σoπ‡g‚M9ΕC†λ±Ε~Aγη·ή|SŒ‘ής~pΊlƒ³Dς+wu'OΙaŸμΉ’‘Χ[²R{λζ…ςΧΏώU~ρ‹_H=β>~CY―ۍΡb">― †4@{’γ ΦDTq%[β ΖύΞMyfΩw& “fΜRQyτΡ”ΤΩœdΗ֝’ΧΑ/&θΈΥψG»‘L"β˜€Hη-tΚΚQεΠαCRRR’T](–¦ΪfΜΩsgŊI˜pDMœΨΖΰΑ–žDΒΝS ”x-NΒ₯7Zχ3VύβE ”α‰ <’\D«°˜δΛΡ  μX=%PŒ­Q‹˜ΰ…θ>ΟΆcnj‘g<Δ…‹ς`«,Ή«Μͺ ix€ΛΤψQΜ'ύΪ0όc{ΠZΈ{ΎYkι–β€Kή|χ=yϋ­·”‘εWCe`Γ(Kœ§tŽ— Χεs<Ργœh4Ÿ}:DμήBρVά/¦άž’‰ΆJJ€[5qx%ΥτSΎBΩdp‹9Ώ·dUή‡XτyχνΧ•g‘4X2φΝΠv?žΒΛMˆKkTΆ 0„± ³d ΠitΈΜπΪ­iέΝΎΗuΝD9¨žΔΎZγ˜•)ϋtƒ*ε‡τ8‹—όΦmΙς©€f˜Χώφ7yκ/OΘζ-›$»όVq”?(†ΐM*t¨bGΚqΚn$Κn 7\₯π€ΊauξΓ|U‚υΰpωύο/ΓΟπ7ΊM’Ή――_ΏAV¬Ϋ ­sPΕ•&Θμ―Φp šuΜ΅Ιƒ©»r…rΡ<Φχ6ΥwΚΥ½{φβ@3SlYPΐ²«6FhΧΑHΣΨ}vx΅ξPuܞώΉ°§J_>yς$<ΒΧ‰έSw;'ζϊhβ’£γ ›h•VwΖΐβ‘v§δΟz-@ώσ°Ψ ΓΊφ@₯αΑY4[ \f’iΛ6‹Ωd”ΩsζΘ™3gTOδΉRΟ… ΚsΟ?§ iJξΘ•œN^9І«^"„Λ“€½ŸΓ ›Ο‚ΓuΏ”u)”ΕσΛoΌ!ϋφνKψ6ίυΈΣύ#yF"’ƒ=²‡498%C8U΅ ˆeEY0¦tΚeΣ¦j?ΙzΕcŒλ6" Έ±cειgž•ƒπLΏaξ)·)CšDΫκV"M4ςΉ―Β&χ·Κ–}0ϊΛ“Oͺƒw:ρ²ΟE“Ÿšo Υϋ*Βٍ6R ς]zWφ=0€!‘ $χ(‹Έ6₯‘OEο"9rπ(tŠOΙ4 #ΗSŸ¦Ϋˆˆ«Χ¬‘B„XΚ·  Χ.1L,Ά…1•ηtΘΚU«β:6cXυΕ±KΗuί#GŽΒψzY,v§Ψ‹ξƒΏliH“@ƒλ†ά‹Δθ™Π­dšέbΘΎY<%½€f}\-σl€ϊ₯hΛ»ϊΦ‚tp.αΰφͺύr[‘S*fξ EীL$δ+δȞ…f…ˆ6δϦ1ˆ€eXRVξ:Dο?°_φo; ξB»rλ& |7}Ωζ·¨POΫ·m―ιοΧ!=!/%Ό1 ‚B½z!ςC/ΕεsθD°~»B Π* νή½πΧyΨT»Sgδ!p―ΰΙκX.6 i―"AχyN’Γ†•^yUς²<ςX ·B€±`eŽ΅OόGηUd7ω/lnJΩwΫ@χps‡t«,ΟΗNΧqπGω•(‹΄X6–ξΟTΎ2¦υε/4κKτ©Π 3ςόϋwδ©'Ÿϋ%α,*H l(ΰΙ„v0₯Θͺ Ό2 ˆ•™ΥN¬Εχ"|Θ²eσωӟώ b!«e5Ώ’ί2‘-ΦίD₯ι ΜbWμ=_Θe*-ία/Ξ@΅!ĈΖΧ”η²ΪZYχ3*žN?!‡ΉQ!CΈz…σ“2Κ!2 ιhωMθmΚPz7Ε2ιΎΒqDΓ«Ο>ϋΖ$Ώ“i“ΗIV~kρV>$¦ΌΫΦ$jRFŽ+©’꓁C ƒ·5dw„°ΊW.\Ό"φ<ͺž–E‹ΗUΫ˜Ύΐφ’Ζ Ρ[©6ω ›yυn΄1%„,ΉNDƒφπŠ―©#π”eύP}ib/Ίξ„,ήΏo?Β ;ΰ εy*,,N@φ€ $#ΠηΨ~\λ₯B:rδ¨ΨΏή~~¬mν Ρ>€TͺΦUœ;wJφa.Τsc*π3ά:lέΊUfΞ[ Š²€Κ Λp3ˆΑύ€©Θ χB§ΩΉcG JmxZΞΒΑγ«―Ύ*W.]‘;ς$!½30/&rΓkΩΐ'Υs“ o‡AMΧ"™=}ŽΌχή{rόψρ”Φ­h9Γzς όJT λnM -ΦΦεƒA£Ε«Χ¬ug‘;Ψ)pψ cJ›»™λ°G8‘ΌΓΛFσ„ˆiC–/Ό$>—MΎΩΚ+½Š–ΊDΥΫ²ΥH›Ζz½Bδ±V>qQδ…—^–?ϊH…NζZ]Λ§π8SχέΜ›ΞG/Ύψ’Μ›Ή@šυ(“’>Ω@€q§ufA"%±ΫA]hDhκ‚nYRΩ―δeΘσΟ½ £ΗŒV{δhς«.6ΠΩdtb₯.M &^s3Λu όR—Kvΐθn#ΒΑ5εDC "yQŽΗ"i™DΝ§Ÿ~&ƒΎ*YΩ…b)μ}ΔΤ ŠφΊ95 ·Œΰ¦qΌ!ΠYœEwΒ¨ΰKyξΉ2}ϊτšu}΄δ]8Τ.:α˜‰“€[ih.8+…ž$^γ9Ίot―šoP‰b· hm9~φœŒΒτιΣ§Rb§ΗΡ³βΙ›οFhV“(‰wΫρœΒβ5‹·ΐ­δ1ΧpL‰ΠΧoΤgτo oLs=Γ™PζςsXΥΓ=4͌ϊΎσY.Hx υρΗK~~>b’­Χ8)5•D0mίΎM.]ΌτΰΙQJ$Š„­‚1Αδΐ‚zsΜk‰ΊΟSΐ9Rήώ`ˆTζδ•vi Lτ΅τ v‰D"=M‹κΙA6ΠςΆ± €§Ηm•92yζycΠ ₯‰η&+֍€ϋ3 iώς—ΏΘC=$‡ΕζpJ—Ν:―—Ζ€dΔgCΕ(Gi1Β*=ΣΒ„zy“ΚƒŠu Ξ™™Ά€Δγ*»G¬Ώ|πή›JqΛEŸžkΘ»dHšήΥP>ο?|TZfYn)SYΎΗΊ5ƒ3™H)ΠiΚό.Y²l™2#“…Ÿ±nsΦQ6˜¬Ψ$BΩؐ€ΪΜηaƒF94ϊΠΘ4 Ι/ύΜWΠ2”†½ 7@k1Ε’OλςYήϊυλAΓ‹P½I†ΈΏΝξSa_1ΈΛ‚KκΡ°Ά‘‰=œϋ!xSύΕVvŸd—φOV˟ό':t˜π \ΛΓΠ§υ³ξ?4ΠZ±z4Λv#Df|χ5δ4 ’³yί sΙ–»c!ˆόχwC2ϊ Ϊ0J-{Ό°ΘvŸM­ΩΏΣ)|h] C›>ώψγςΗ?ώ±Q!·Γ§ ²OhWU΅ύ` ιΧ°΅Cƒ(C¨'·―HvΓ‹žk &-Λ”_’=€ωΟ04†n„*ΌΉη’(h‡΄,§{i‘5ΓR]Ύ|)!ηI½Φα8%βΚ‰£'€ΈgŽΪ"„YS7€Ρ=z‡Ψ—YςoΙ’Ž…2yόd1b„Zσ³o¦ΪX€όf½xϋ―ϊ―ς›ίόFΟ : Δ»tZΨ ­Ω–DHΣ(½A¬₯'Š­ύόω3@ ξOt—‰Ε»»4φύχδύ‡I‹’lyΈ™(‡πΗ¨’±fM•η1έΙHk'ΠόX §”ζψδθ ώ@h$ιρ₯ϋο¦M›δΕ—^”u«ΦKΕmΕRc9[–[Ί›TBωΙΆ„J-Ø!6n)»+FeN89 „ξ~„ΔΡ'”~fϋ𼌺Ό“@a+°Ω„SΈ—ΔrM ©Δa—ƒΫ`€«ΗJ\ŠS‘Ϊψqωςε’››« @£MŠζ3Χh&LΑΌ+ώœ °χ£ :E[(ψ)Ϊτ$^ώΑ¬Ϋ}mŚ› }υ78>/Z΄H‘Λρ€ε"yΖ!,—ΙhξΤm`τ“:-FξΆφ›€s‘O&Ν}ΥΚUρ`wΤΚ$ζΦΝ[V z;„WRc2j₯Υ#c0œ4Π έκ±Θ»’Ξρ4†;ύz0΄·α:<“B©P‘`©ο_f΅rƒy4D ι I4i’<ρΔŠj.šbb½yˆm6-Ω`ΕPΌ–H‘o†z²8π@ά]γ9§'”Θ—–Έ9r¬°­'Nœ(ίxSšδTd™Τf¬)/}·ΥO==πήν‡°\=« j>ϊπC₯€e;§rfέ΄ §u.½Η—T_Ώ>Η"w•εθ±4kφl8ώY6¬_#9=Ε\ΨOŒξJόŽiΉΕ re'nNΥΤ™αCŒπL°™'·όfY06Œž†}bJ†ώ©Η¬KaΈβ΅™₯rΊ ΈΜŒδ.u?>x+΅ΛuΚ’΅%Ρ=aUcΗρε „$‘dŒf`W«—ΙΥέ4\²xΈk²™δμ™³MΪ 7\Ύ]ο~-§)h˜4lΨ0ωνo«bλΓΩλ=ΙkZ†S)IeχKΝ•œnbΊ–Ρί ϊŽήΐŽI’£Wυ|e°η‹h<ΎŠ{Εζτ#lί@xU=•ηŸ–Ρ&©±ωsΝΏxΓi ΟοCY±y P‰—αΡ·f횚΅Lcλšμϟ8yBV»Jzx1G{’έ²Ž;‘ Ρ“±€ŸγŸΤίΝΰ΅Χ^“±=qωε€ί{°^4Βϋ!L3-žj—!€fwη@W±K… Še›Ζ»,=§PΙΏ†)Ε>‡”C1ΛDΉ™¨©θ4ν ό2uΖ „\"3ιΊ$Ν€…k-g 6T6Ω$Ε7ηIN{oڐζš›Τ!»B¨AΘ§’Φ2ψύ!2sΦLu'ω˜ ‰}‚”ίD28p £$’ό£ڏ}WΠa°‹ δ^άμΑa¨]hŒ¦ΗS,ϊ.‹ˆoοΌύΆ >Jn*Ο•‡šAoλ κmιΖ” ½’4’Vr΄t)DΧn΄ίO†”χή}7’Pšo;€ΖΠNΫ6lƒ!M‘tΟ³‰lHώ¨€™“ΥΒ%%w",UOήψ&tψbΊo ?™(+ˆWμυŠ‡π8΅SΧγυByμΕ9`.BNmΫΎ]9O‘MoΌθŠEΉΊŽ<%ςσΟ?―Š…³—žη Όεk@Ήσϊƒ:ƒ«ΜΧ£<Ύ}#mpγ2X½°¬χΆ[ήm³_ j4pƒnΓη_fzΑ²•rs‘WΚΰ€™θ™αpœ§ˆwΓω΅kφTGΗ³~:W²οΖ“οαΤγFχž9{Fvnί…πμ6D”BXΌ•[$ωD<ϋ΄\„aCVεέbΚΏC2)˜’EM)¦UΟW4ˆ4XΔΰkƒP…χ(ƒΘΛζΛ“˜λh©ω—¨2GΣGΩΈzυj)π:* F,z­)Ž5$r9ʏ< JΠ­*ΌσIT~6€Ž yζΤΙSrόΰIμΥ€’ 4Š¨Άe:ζRzP<~RŽ?Φ’›μ3”αόγaΥ³Ο>«ΕΌ8`I…%/Χ ‡a ”‘g-†ΌV•βš6š R{ Ә­θcv7H·M c•χ„kΨΈm‡΄ΘvˆFΠ”Υ3S¬H¨W9€‰ϊk"•y-²}χ^uΨ^―‡cxΗ)ΣΌyσdά¨ρRΤ._r:yΔOd΅UMDζƐ?_+ ό ]±#Η*y³βυΉZηCΩΎc»Ί5ζi-Ώ©C'*δο~χ;)**Β8KœΞ°>90“Νͺβ{Μ‘cXV" ε>eΔπ΅>… ‘λΨO±OωlτXιZ‘'χ•[25eSβ΄Xž½jZyΠY#D’kw(A͈Qς œ*"‘ΟΠ|#Ϊ›oΎ©ŒK{JώΝY@ήΗ‘#}¦5!7"6~Σt‚_Ύ ‡”τΞ_ΆO zSf͚s O:%›€LSˆO ±t%ͺ‹τΊ«Η!`7d"ά’[61‡εM%ιy}ΕςΚ‰!„˜τυhρAΟ4^’#«Θ@¦œ[€HS¦Φ°ΑruǍΙ’/ωΐ3ƒL₯Oqδw—CφΚΥ§Ρ~5?cU#φ:!Ϛ5[œpm‡¨68 κS‹XΡνrη! *}ŊPH –.—΅™*ι8BNž>y!ž€œGgQ ΓDH`ΌΝkΕΪν Ξά“Λ˜Ζ˜ό ‡ΰᨀq8Z˜RιΑU¦ϊ $BoΫΊM†.ύί­žνή½»zoͺ/4¦ΩΏΖ4ex’8£2- “iΔ†, U‡ͺjϊK$rN–<τζ‚0κŸ`Svϋ”=Έ©I§δδ 4¨ β­ œ ΐσμiΩ²₯tθΠAΙΔh/¬γΑ9P(μƒ>PVήeeejAΘΙh'ΝΟιΣ¦£μΧΔαt‹­ΰv„i†] ”“I΅S&·Π;Α‹LdΊˆπυg«ζ*ΕKνΡ£G4 Xή{vο‘u;χJΏ^ΝΕΘ]*°8ξβ’P0•ρV―έ€­€rΓj΅Ζ…œD/”λΓσD•QaCH0|8aLƒp‘0¦9ƒόΎΈLm\:5”DƒahΌ€φάsΟ‰ίοWH•Όν€ΧB\σ²όΟ?&ΩΝ‚aωœ%˜GBexάFz΄ΩPGώ¬7T+\:cέ,Ή·‰Οβ—U δ™gž‘_ϊΧrϋν·G]iW‘uώLOό«VKi–S²qΰ©Ώ Π€TZΩa,RιwΚjΐτ‰c@Ι™ψΝ.uς3Z7pMEπ#Gˆ# ¨4<δvβ<c‹Λ$gNœQfΡ.2•ςη‘Ε‚ " C·kΧ2}m½υ0‰Θ =7œ†ΖU&+CΰΔrmEKΕ¨CŽ:!'±ΆkJ‰r€m°‡4ϋUή9ΖγAΣΫ΅6…kΓRYΞcύB―}κυ>±Φ‡bτƒξΣ»€tτωηŸK ί/y]½Pΐ›°=‹ηf&F hh1θpWΐO±]ς»dύ€­2mκ4ωαXƒͺήΠ¬α9/^¬B~}τΡGBg$·λ!ρ€Q’<;wkOPΒΠ q<ΈΟ€ώΒdqΘα#‡ΥA€Λ劺~| N‹ˆoΓ>!7UδΛ=@’Ξqε2‘dI²&’Ξ:δ%¦©:ι\ωψΣΟ₯  @ξ»ώΛO-οΈ—5j”Μ›9Ošχ,—όn>ε1ŸΜ2€žr§”τϊRΆNΎ(οΌσŽδδKΛ-£ήu?;†p¨ϋpζΦΎ€XlphΔ · ϋχtφLƒψ-Y C·“ΨΧK1”αM ΡXlζΜ™jONYή₯K0ΐ1ΝDΉDcŒ‘ΈΏj—*ϊ@]ω̝m<{D4kݘΌΩS΅AM[„ε9*Λ–Μ—iΣ¦Ι£>ͺƒ“{8ΟjωΈaΓYΊb₯΄Οs+}ςNFIr/{€ϋόv0Œž³ν ,CψX†“§γžζE’T₯†L½>’νuFfœ$R"}tˆbΈP†¦d4'Χ£%±8y ΅’‡L<Τ_Ήr₯R ]ηφ°.ρΰ˜Iη]ΧΓσζΞ“ω—QεχιΣG|πA5!ρΉdiπΊκXίί΅09ƒ‰ρΐώ=@ΐ„eNpΣ’*’• #T^lΜN!6λρϊ²'eξΣύz6BL›»@μX*` [Ζ”wLΛΚTιΝΧvJΦ‹ό„šΎ­Θ!.―’)@L‘r$VžώΧΝο—€ ΐ° ?ϊяT1?ωΟ•ρΠ Aƒ’Y¬Κ[ΛK"ŽΙl…!M/1xšW—Νω-zZ’τzΥ;aˆ€_;±ƒ;§vMW <*MJKKrA«Ϋ™p©„uφΨΝ’ƒΓX"\„‚5^J,r“ϋdzλζ»Μ²VίDθΛΞfhΖtΊ–Acšσ@ΏeCΟg«‡³2¦Aθ‘ η.Ddέz-­Mε;Ϋ„JΞoϋΫͺΚ”γέΊuS5wΡN\ ]²ΠΘ‘£dθΗƒ₯ ¬ƒa,b°ηapQ~'’ η\R]6 "MNΙπw'δψ™½³Υ:Q(έ`—‡C;χξ•v›ŠΞξOŽ²o³|aƒafΛ"C–™¦%‘ΌΒIk,Rθ\{μθ1qψμΨ‚[)šmEIƒ³8Y€,ψ…2™Iο•Τ—τΛUΠmΕqυςΛ/Λώοͺίι”Dƒ^‹S'OT‡2fΡΈKŽaΒό”a°ƒ™5ŠΘ–·’tί’‘Φv •ω]ⳏόψ"μ7f e9ir€4ζ€εΓ₯]γAΧιΖ9DχW=Μ_ ›Φn–Φύ*ΕUh.u’)`R- εD’ τdbΝοoν–œ­Ω2uΚTιέ»·4kΦ,!Ϊ6\λώHC"vόΫΏύ›Κ‚οάCώΟόOBΙoΌ_Ί„ύCθςW­Z%“&Žo~[1d΅Sς1ρ‰΄ ΰΌuG‘v1fuΧ©½ΠAΑ@†εε卒uα΄3ϋΫpγΖ ²uAωvΫΦβ€cOΚΪU«.‘‰›ηΈeεκ5Κώ€ηɞΈ>ϊ:Kœιu»Ζ½Nδ7tνfΜ·—.\J:€°„7¦ΡJB ¬Gο½χ^…H©FgΘ(ε¦>i),ΣΈΠ’"θϋίΎ&Z6εD€‡cGͺ$1Lƒ§N‘N˜L¨y€%“žπΥ%e_‡kΥΏι{ԍ}aΤ‡LbweZν˜π4™-Γa‰^ό0<ϋy1 ";ε μ”ω—pnΔ M4Z;£u/ΦΨbP‹œΠ€/E*Ζvlε7IΗ’,™=oΎτνΫW:vκT―Ešξ#‰ΜM#=Nθ ΛΕΠoϋ[eΩ½mΫ6ωχwΉυΦ[Γ«;ψVŸ€Λ'4# Ν?%YεwK¦ˆ4Wυ¬xχ²Lll¨xΒOϊΤ0Šχ€Κ{† q]>#UΫ¦Ιη@†ϋΥ/©ζ_ΝΫ(RΠ ¬f;ŒiΚ€l`Ηuϊa΅nθΝό p*b}ƒ)(“τ·κ‹uΎq¬η8MrzΗQ!6½ΒΣιλ Wί0ήΘ °ZoΡκ/l~MΧ|W­σepυJ].Ρm.^ΈXcΈ¨ύVW‰τλ΅·Ύ―wΝ+CN™2E‘ΑίίωΞwΤ‘!ωDηˆf4Μ™3Wπ”΄V‘ω G7δΉΰšŽM’—7:[!-˜2Ό Β·θ9±kšΌυΦ[ Ω3??_Ι3½žΏ^uβuνΰΑ@ Έ(Yv—:g«†Υ²Έ9TL°Ž\.χ)AΩΝΊ…•gυIβB<οl‡UφUνSνL7ΥDYM”5³έ9K…ω ωω{mβuJ6έ6ͺM8)†ήZέv‘—ΚG=Œωe’˜Œ”MΤ'h9qΥύ±ώv§Š4¦‘ώ„{ ³χοί_…<% ΏVb=ΡHδyhŠgΠλφΜ™Σb43d+κSCΫΥ4ΦΠ‹ΛΧϊΫUŸ3jΙCίğυaϘQΆ*Β’πB°}η.)ς:z—>¬«ƒk΅RrνψW\] Κo•/^T_{s­Ή^ύΫή†‰BΠΌ²œγ#/ΖΉqNz Ρƒ•α]§ψ*a Oέ+@'N”΅!Η?ΥΩΟΏΔ&K₯†6x„yN$ ‹Η$9m|²μ“΅²|ωreLSŸ1Y«Lˆ0αfΗ1φκ+―ΘΝ7w—Κΰ> .TΩΔ"TφθΥ}–λΞΙ™7 ¨Ίΰ#€υ£μ<ΑρΝƒc>Έj«Ξ(ψ†{κ•TΉ™ΛNΜ »]υz7iPΧ5iD9|ϊ¬ά]™‡9“2€aΆυ€Ό~ 3΅ΜR™sώͺ_]Υίκ—}]wAIΉΧ$·•8dθΚ}2qβD)+/Ϋ Qσ™γƍ…±G¦δtφŠ-Λ¬P¦ͺΕ~]δ4ψχkyF^©ι>‚LS+_ΘΚ¬–N)ܝ'“'N‘^½z)‡]W Ε\bΒΟ† «NΥH_π/8ŒΙ/Ά‰ώγ0Σχ`2¨5Α[H鱑FGͺψͺ-ꝍ*»!ΟΥΏ„ΖߩۘNt4€yόρΗε?ψτμΩ³Ζ°&RuΠeiͺυw©CΉ½€ΩΧ ‘P!ϊ“pz„Ξ9οθ;μ[π Sϊv¬°zj€ι ΙΞIΦ€ΨνdοΪΡ²hΡ")‡¬‹T›…”τ΅ΊύΈ>έΈq“΄ΘυIN’iŠEυr_Λ»Φ d=ς ν¬k°¨ΰU–ΝαzO­ω5ΰ‡/PP+²μ2nΓ69„Ύ©`Lsκtph²S”²U±:,}΅JΒchΆ ϋ Ϋ‚ω©wΌ^Os½‚ψ${”Ί-ͺ³x†BJκt»ήs‰r-ρi8R“•Πt‘Ll|6–Σ ŽT—P’ˆ5­aωΜpS»wοVJ΄ϊΠT[gΰυkSmχ^{_ΌΎ‡wϊΤi𠝟Ζ°ΐW $,Ψl'‘μ©ξΜ΄ζ#iυ— x­VΔEͺάjΚy„ω%Όj-π4;yς„\€‚=Q;m¨Π~Ή0ΏKΦ¬—{[ηK 5Ω$TθG˜ΥΧ’Χο§.~)Ο\7`γπ@₯qMJ&Τ‹"ΘgΝ”v™²a“¬_Ώ^:tμ¨ͺ:ΦoTήΪgnto¬Σr‚‡ρ!999B^ίΌy³"GίSmαπyθόBy²vν:YΌpŽδ”ί"™ήΦΈς’žρξ[ZΖ~qA.9„… Π`¬8œΖ3Έ$ͺ+1ΈNή€>E¬ς½gΘ¬Σδ.ΜΕνΫ·WΤ·mb@mM4Ί¬Ϊw@ό@¦qa1Ξ© zhυ^sӍ>¨{«η;ς‰‡τθα€ε;σ»’ξ ώ^Χ+Η9=aύ6£œ…,½ωuJDjΪbυ:V©ε‘¦˜μΤY5Θ5•χΥˍ―h£ό Ά―ϊ½ϊ —Ν₯žAc]Ίˆ<•ΓWO%Β§Do{-K©¬ΙO~’Όbιαk„‡‘*y]ίs#~†Φ3΄­υ37ΊΖό<€Όω¬β|χLga΅"(QCΨΜ^:-—Οš’K2€¨¨:ͺ꘺–1~gWςYνΕ}ω΄¬Y=UζΝ+ίόζ79šΤΊ‘>νmΚC۟1₯i ᳕χbθoυ‘ƒ’‚kF5ό«`”ΫJi€ψ{Xsς ζ›!Nhf;mBoΟ‹/%΄ai}ψ՘{ζ‰ŠڊρΏ)›©ΐQΌmΒλJ~γ%―ƒύΑό‚$ΰ²zίϋ|H4΄:ΝrφΜΩcš`&ρy ·ΖƒJ’R͚5KΚΛΛ•N‡4p­Obύ΄œ¨«‘χΦ'οΖάJ 1^DΝ Τ‡pπεζΦ‰Ώ«ήv2Π_ρϋ—ڐζz·Q( υ08!B…Θ²Ζλg‰ͺς6τπ%%% –ε7D~ΧEwcψΓ½χY Ε™`ΐ’‘aB7 όΌ‘Œ ₯­†{υ<ÎN#-:»σΡu >ΐ‚ƒΩL£MΙd:ŸκԘ:κΒ|ΣύBΣ(AŽι.†²Ω¦dbΧΔ,Ϊκ³HngŸl[ΆSˆFB”P:†GΊ?’d'ΟΩzˆqΔΠJA5©>Υ»’ϊϋY°l—/‘§—Ό^σW“sνΘ&a=ԋѐJΣ«ίΥM)ς’ǝXf͚-mΪ΄qB’Μ¬O ε ?3©Ά‘=ΧΛG?GCž•+–Š'§ΉdΨ‰πΦWςτzOΕπ«RέΏΌψΩ;@;‹’οoz"ιδF@E@Š ‚”€ Š(X@°’^₯κ‡X¨ϊ* *’‚"Š ψki‘B‘z €ήžo~sοdοsO½υ”δžηœgϋμμμΜμμξ¬°xΑLσΓm6N³ΐ(ΌkΥΉ(«Γ ³u yϋψ0j…Υό4Ο]vΩ%¬ΌςΚΊνœ χ~©pR}βι©aμθεŠΆfDί1έΔ½ \cšΜΞxςΓ>yΦj?ρ€Υ~X5‡¬φ6s2~σ’VΌIkcΥΧΧρΤ‡<]±φ 1gδαͺŽW¨P:`Rχν}Αϊ„xl,5ι ²ΝΨ‰ŽΓ—αuΔΗάΤΧN„Ÿξ–UχΞ4j ‘«―n»@:bωΥΘbP:yσλw>^œ¦^ΎΗudPŽ|ΫΚ6¦L°Y<Χvιφ:€;σP³Ώ…vNε’₯σν7·}WOXb »ͺ²₯Ά ΌšQvο9=Ό:? ̞?ΌξJaƒΥFωΔڌͦ7Ή†ζm6‰­·ΪJα)SΔYœ“aΉ6‹Nΰcπ0~λ]ΉtΖ€¬hΰέ1°ΐUςmΏΞ?σρβί>8Εw†.7Ζωe°…Εϊ₯ŒwΟzα?aΙμgΉ #ί±Q΅ΪΖ¦η ³*–ΗM΅ΕyδJmαυ>ψ ;ΣΔψν―z”+GυaηΕ SΞWΆaœβ5k‘έ,[#αE‹Lΐ4_»Ct‘ω6 1ώk;:μψΉΖƒqψ[„¬\…’0Š7Ϋnϋ8΅ο2μ©ΞQΤ–όγο,” 6ό/ž/‹§yζ™©fΐΈ?¬²φfaπΘΡΦΡπο΅ νυ'tϊZ˜υόm&‡Ο΄#Π‡‡•Ζ~( [a\―—Τ½ #昲όΪaωήα›vΩy—°βJ+₯UΛΑΊWr΅©βΎ†7R+ζœΩΖguΚA΅y‡σEF,Γ°#o°ΡοBγ CνŠΗyζ?π–.βκΏZ2΅ΈV%§ΊεFχY4βϊטeCFΫ»hΡbw[ΊΐpκόΧοΝ:;ό{ž]AfF$U2tHX²Π~ΫΓάαιm^-šžό4؜>lΔπN‹‰qςEχΥo• β»~χUyέΙWuΒ™f»νΆσ, ΫZκ«ηuΙ τγ;‚ΫŠΜNd¬ΖΆΑ‚16’ΑζΔ/pΗυ+YPcxc|{ψΠΓΏpώ|€€ Ψ~`λΣΕa]ιFŽόΞFŽ ³/6νNQ”~0μ›3^΅y»Γ±LmκNž=M-Ύ˜3kv΅β:F‡Ζ[Ν!άi΄§…τ4½α›ίΜημ4’[_ ΩBΫΡjW²Άm»VXΑœ©p€πzΒί·Ÿ¬ϊŽ1+…©OOυ“Ip¦ΙIυ·ž μ)Ίzš^υΑ&Τό›ϊ¨­ω§Κͺ΅ΞJi0Ž†³E˜lA¬Φ.έΑΝζkڟ;ε ΅yu‰ύ9ΜζΧ₯ ,sγΗKMΡ«Jr³n6Δ!&΄αPΐi§€κκ?zι#Ξ“ΎϊœΛ>˜ΙΤΣ ζ…EΜAΖΨ¬Υφg§ΰŸ$ύ7Ή3D›]σy—9ˆΌψβ ξL#ΊͺTfŒ76Χ-ΆydωΥμ:;dΑ>δwξ€…}Αhu±­+8zΜN0Κt·ΕsŒo ±7Ό4άυΐ“,_(]k½Υν¦…)ξΰΗΓ½Užς€/pΨΰdšΕFs Τ₯Κ‚@#nΎ₯Αy } Η‚…fbv]ήΟ±<Ω_-yB§8κ #/«Ÿκ«g•ΥλΒΛjM_m9=‰§:qς]|ϊN¨&oεAάψ;Ά0@cNO½ηtφž{2¬³ΑνιλΖNMmΓΫ[Ο› ε³ϋΝ K›ΓΘ:ۘΝΒ6M••αIΫO°”5sΠ=&<υ„9ηbc6gšΈ/ϊ’&κOΖΘ«ΣήcΧ^),Ά)₯;vνZκ‡­cŽΝ΄oΑB»ΆΡγψaΫmn1ϋ‡‡ΩKgž΅δ\}\κ0Μθ{ΕQΓΓλΣ^wyΏ―ρ]}νͺ©:σDGeΜμ~WW΅œ“2Ϋϋ$ΨΉΤώΝ›ΏΘζ`λΛwδˆ‘n_Ylv˜₯‰•WΗΆŽ gψ([71›Žt+Υ½ͺ|Ί‰όU†h½ΦlΖ™†† ±΅6²šψBdΉΈ”―:€πJiˆKξ(ΌυΦ[]hRΒpš:uͺ; ±S“]Mμ²―\}*L8 όΙ“'›ƒK/=yGΘ†N‘“ϊ€ZŒΗ‘Kf„±λŒ ‡r¨_ΝΕQuχO~ ,}Ή mγx}Q4ƒ,³/³ή<ςίπχΏέ @깏zΪ ’]žΠζbΫmwϋγ/…ΙOχ4ηϊNΟ‚έ«s—†?ΎG8φ3ŸSμžΔίώφτpχ³Σm’¨Ξ{»Ύ[XΌvLwΆ6ζΎ΅ άgΗs»*ΡyL'O>ω€σG‹—4°o©/lTwͺHώ7jΗήtΣMΟ5ζ ƒVΈη~όψρ~6Ξ:ΰ qτΎ~Η·‡Ωsζ‡ΕOέi;₯ξσ°zψπ«l–Μ άό]a~Ψ=pΖφόοSΆΨgWG;ΌΦeƒΟ›Ώ8ά|σ͎kζaυΣ@ΧOxbq‡γ ߜφZX4kH8ϊέͺ49}± ρo[!zΘ‘aύυΧχqyΝ5Χ„K§Μ63U»qΏ–Μ‘ϊωg³pA˜τŸ˜ΑΡrŒL}L0ΙΆ0¨ )Ζόλo˜β4ci˜ρ֌eXaH›¨±ΨcηَX N°qyΠϊkΞ½³ΒKS_AΓ0&°,Ωόι‹lg^ΈβŠ+όͺPx€xΙ²XσvC³Σ§Ow# Μυ2¦ς‘^όγΏπ§~ΛǍsΪΧhαUiψΝρϋ\uΥUW™ŒχˆŸ2‘Ο›ί8π͝» ΌϊSΑ΄j«C“γϋωϋ0svΞL ŸύτξαŸΨΓvέN;γΜ°dθΚf$¬v!‘ο+‰}±BΑεŸΟsXuΥU]ηΎϋΎ₯KMPŽP^l§;άτΘ αξvY:a‰ς™ΉˆcΝG†ΓŽ8―Τ»δ’KΒΏώυαΚ‡f‡αƒjηίε<άΊτΝ™σ—όνoσcν+Ιj%ͺΩ°―ι/ψ.WO<ωδaϊΜ7ΓΌ«pZŠoΎuό;,²Σ;f-τ«£χέw_?‘π΄ίœfΗgΏ^yξ5ΫD,y,5'«Y3g…)S¦„ /ΌΠO«ˆ>p€Gξxη;ίY?2[jœ *ωꩧž*\'B$ε ήΞUέΧ_}xΦΦ”ώΡ_€©dή΄ΎHπόΓ2σύ ±'Όh;Ÿdp Η-uε_1›‰W:ϊ-α܁;ˆΪΔ\Ι)ίOΫζ-9±Œ“{Ίk§,δΪ~ψαπ–$Ύψ‘λόdqc‚Q­*|5ži”™^wπΑ‡~πƒακ«―6έϋ‚0{ͺΩρΓ]ΥργB1†­E3Γ’y³Γ΅¦»?msYwΫXΘ²ΘΪ€_NΠa²φOΌœjΞΓ6―Ψλ>²ΕΑπΫΣΊωϋ·tPΰJŸΣΟψmxϊ•Ÿ¦Ώi1jΐ7jIf™<6ܜž.Ώό »5ΰΗƒθ―\–qhtφόΩΞοž}Δ&ΰ>†EΣ³Π6ΝΦ4ρυ ίύξwα‘Gv/™e%χΞP]ηN›¦½τfΈθ’‹Β*«¬g4 =2ΦY+βτ—žzΪUͺΑ½±ΐΖ7 Ά_ΫΏ’κΖ‰ΓyώsΈνΕωαn+Úεb=ͺ†ωζxΙ‰Œ™ .Έ jΊ‰ €†ΰί―ΌbΆρο8N=}y…ψq₯ϊ=πΐoN@yΐsοΉϋ·O^~ω偓oβω-ŽχίG5§δΑαεη αεηk䣕jΨ³παζ,™-x=|ύ«_6ښ&ήx}ψσ_.΄ΣiV Ν‘Ί^ΐIΎΞ›.ωη?ύ„ΟΎΤyΤθ7¬wΜ{kfxψΉv*toΙΨ₯1 —Xl:Τ4›[Ά²yxχ¬Oœ~Ζα k²y¦/ω$5£sl{ήμyΆ†3Ιψƒm»1\€—jΗω 4¨Ύ<±yΞ±5έ‡ܜ­…έaœΖΧ—ΜΚΒrζ`}Θ‘‡tZŸxδ:λ°aΖ΅kΙ·ƒΙcWαšb6Ÿχ—mK4oΓ hLUΫo εL3Π„Kωό ρΥ"™ vŸ}φ)ύ”SN)žCXu••ΓΈ±{»ΰ‘ΓΠ¦Βwd½χ½ο hO>ωδπsNŒͺΑ\ŒFλs²ιΨεt'zκΏφΪkϋξWx΅ή©žπ#Ϊ] ΧqD½πΘΕγ›« qό<ϊθ£Γ‘¦‰οͺ½υτΜχE=ՍΊτΌΏqd²IΟΰμ³Ο§žzjΈτK}—άB;YωžB½γ°§νλIϊrύ‡3Η›'pΨsΟ=ΓI'vί}χͺd‘zΕ9νεz›VXΑwxφwυ–vϋν·χ9½ξQšΰνœ,†1³”£‡RiϊλύR39"L6ƒΧ{7Ψ όδ'?qηΞbsVΥ©R9Εϊ’Ršώ ο­Ύ&hŠ-?φ±…‹/ΎΨyψBd%ΫΡΧPΟxμφU“G­ύE|vΕ?φΨcaΓ 7 ‡~x8α„C8‘šΊΖ©‡> M8apJK9>Φ΅φυϋFxέb‹-μŠ=Ϋ͚ΒαίΊ[} Ϋ]+ͺ9, 0z|όρΗΓ'?ωΙpβ‰'†O}κSΆ£g"Εͺν)άΥ–ͺqcwχε[ά.ƒχ§έ©žϊ­opZγΥ†R·aΓ‡…λ―»>|ϊӟφS;/°ΣCζ/°λzA7ͺΆ΅Δ«Τ·Ψˆ‘ΑW_}υZ²­ΫΈj/Ά4vόζmΰ„Γ»ααΜΗ:υ § κ)έ:m™=ώϊλo(ΠΦ…ώΝηWNbθ '=Ι£RڞΆΏRώωpΚγ:,ω±!ν±Ηαψ㏇z¨E]fΗΘ§λ«ίέΕqβΣΰSΠ?xβT;NϊΚWΎR•‘·pΧ]\ΥZ~oΰ–k4‡›žΞ)V˜ž{Ξ9η„sΟ;ΟΦΠzGŸκ).ΔΏe―Gυύ•“κŠαH·jŒ΄+θΛρΗή ΎΒ‘dπ˜Άώl›$ΨάΔu`υ πΠ5ˆώ£ΜbλGΨiΏ4·τ_/U.©·ϋ±λϊΔ²+σ*Χ¦tŒbΌ¦tμή Αf ?[o½υ<Γj된izesαžY&ώ½ƒΝ6ΫΜ †zφN)—„e@Έ‘yΑcHΠΊ5jTxΟ{ήΣτΰxΞ &„7ήΨ "Mίΰj |Ψd“MΒΈqγ¨ζΝ]U Ϊmmmiž¨γζ'ώš06π‹€sΤOoΗό›λϊΤ0ώτϊ¨>j–jc€[ψkvΠbςX»κ'―υƒΩ›τ¬Ÿš΅vM^}₯ύδ ρΩΈ˜ ώ0€ƒ>l0mUΧs\5ωράνΨ*T‡-·ά―ηά|σΝέ©¬ΊT­‹ΣΖε»ίύξΦED?ΆΟfqς,‡ΆD[ε°3pai}bΰp_©dΩNšm}§#Ωψ+α 9ΣTΒP/„Σ!όsν^Θ―U³€¨Ω!Δ.<Η΄ /ΏΫ‘Uρ“Ϊέs 0VΩέ}АhΜΡPΥ‡‘ͺŠά ‘όZ6Ψ4!07 άNσμŠƒζα³τco{07!ϊϊΌIΜ{βCβKτO΅B^ŸW° Π<‘~awΠ σtΥl΄ε2ΈΝνΝ ’GρΡe3Πc£χσ¨ϊCύ“ψχΐφj±ρ4<Βv*ΦλΙ ένυ¦ι›=ŠO4“ivš¬Μ—E1NlϊE}T:u ιO 4ΪXI΄UuδρEjΖ`£υ{m­ξΨ1―_@Β[qάJNΞ„/ΡZρTιmo` ΩiR΄5¨γŠΘD[½A5=Ο#?·Π/ο꣞—’rθ Τzͺz’g£₯νΩ埍ΦΪTί¦ΐ€-‹4έΞ»¦θ˜&n„&τ&nbjZΒC’/'@IDAT@AHM¨HH¨iž¨g)EΒ@³b ρƒfνΩΤ„>Δ@“:ξχ!Ζϊ<λdwκsw«π»•8%J(ƒD[eS"¨J ’κΧΠY’΅ͺΡ•"& τ:όΩϊ,”qχ1ζ–ξγ.₯μ[ $gšβ7 @=D`7’³ wρ’ΕιψΘnΰ.%©ZΡ3σκπΦ¨±RΆχGΞΒ_ύZΒFνΜ&¬·ψP6­‘›€~Ρ3ρ‘ώοΞ‘C‡¦ΘhκΩ½‘JŒ1 ~ ώΠ3Ž“Ύ<RΏ |h¬ |M 7aHξΞ–¦«1 Κ™μNeΡ3`‰wκ›ΎΰD[΅w±lH΅§lέΜ-\O— $'u\¦ρ)Ή©Bκ\IoΏΞΚΡ$Z3kx‚ϊΐ€ζ–4ήλ£?T ρaύnΕgr¦ιa―#%θ? 0ُ5*ΌύνoO (ύ‡φ–+‰:€ϋk4Y8šφ#ρρφcyWZi₯0rδH;ϋ+m­'b1b„χ w'¨ €ybΰϋ‚{zQ²„ z6|XBG`@r£δHυOT-UΑ0ΐ|ŠΌΓόš``0 1’ηΐΤb`KΥ"ΖΠ!νzg’ǁν|ιΙξ”ΗHύό֜št£ν“fδί‰Άj£)NXa…άF_[Κ֎͚ΖςΛ/ŸΦ4ͺ$K=“½²JΔ•ˆ&ή­g‰hΝύΊΓδ-š:¬]O΄UέΞά’Φ'κ£/ς΅Hλf{Ν#%ύ 2‚Τ–ͺybwgςν Ξ`¨ϋοΏX΄h‘ Ÿ`²'ω5OO΄VKΊCwΒP9zQΨ:λ¬zθ‘°Κ*«x2½WιΩhζ~-5FςmΦοW\1œφ›ί„Qo{[IžZmžΝA}ӊR8€4υEΎδ}μcαž{ξ kΉ¦εγ•Κ3/ŸoϊέΥβQx­fž(•'%+ŸΞ΅HΏͺΕψC&Z—jϋΈqγΒƒ>V_}υŠ8‰ιRι=Qϊθ5 ―ΫoΏ½Λ“π €χΒΏβτZ‘)£ŠΞ™O™WGνiθ…UΜ€D„Τ―%SβuβίΛ3fΜηοxΗ;–½LίΠθ.Z˜μNޝ+°Ι&›ΈΜƒμτ”wΞ=ύͺ„α[ΟJρ)ό}ο{_˜2eJhkkσj7c{£?„BN:ι€φ YIΖ­ˆZdEζ–o|γ~2 Žέ Jc@tφžχΌΗΗ₯τ©)RH5hΌ;ΊU΅iδ4³ΑΈmϊ]ΊKg•Φ'j©…h†4κϋZTάAVρt†U7±CΗλ―Ώξ»bYt›-B&HHHHh ̝;7ΌρΖΠΆΪjaΨ°ΦΨΡί]‘©\―φEžεΚkΔ°J8ͺžos5ρ«‰“Ο·Υ~χŽͺΙ³š8­Φ΅΄wώόωaΪ΄ižώ­@jΙ#ΕMHHH ̜93̘1Γΰΰαd@Œ₯Rκo½υV˜>}ΊΫNpβΦΒl}ΤΆZT£U§ΆRSμVΓ@54TMœVΑ[5Έ¨&N«ΰ«Φvb‡sγCrδ{)~Β@Β@«c 9Σ΄:t£ύZζΜ™.»μ²0kΦ,?.»’Œ£ψίυw… &tΫ`Ζu,W]uUxβ‰'Β–[nΆέvΫn΄ %id τAw8BĊ;t©?h1£½ΠίΨ±cΓ.»μRΈ‘Θ ¨DΧεςHa …—^z)ά{ο½αΝ7ίτ*,·άraΓ 7 μꀦΕΗγϊU’ω©ΟL L~ °p€ο¦›nƏοΏ‹ειι£€oζΝΙ“'‡gžy&,\ΈΠΨU ΩΩU ‡Εϊ%ŽχόσΟ{žτ5οqμ]ύυ½Ώ“ƒAyβy>ώψγΎ3œ9†ωcε•Wφ~Y{ν΅KfB: ž'”'ο_yεο―Ήώ`ΌΠίοœ a 71PŒσωΓwΏώz—£·ήzλ°ΥV[ε;ωtιwχ0οpπΓ‡Ν6Ϋ,μ°ΓέΛ4₯κ  SήrΛ-αώϋο›oΎy·ϊCόΉθškqžρ<¬·ήzi\υJ/΅N&y^Ρ:-―ο–¦~©ΟώIύRŸύh΅Ž<πτΣO»ˆ“':'†΄Ωι4œTΓ‚s‚θΞΘ·7άpƒΫ›°ω§« Ϋq&:γΧ³Ο>λ'¬ΰΜή°Ϋ`ŸΫh£άFΗ-ρζ‰qΎ°αΌΗΈ”kγ7NτΥ€d ώΗΆyυΥW»ν™“0°‘),ίt½Ηζvν΅Χ†Ω³g‡έvΫΝΧ–O£ί„ b›žήυχ“C°γ3ο`Ώΰ*υoΌ1<ςΘ#a§vr»E)<κύΌyσΒΔ‰έΖDNTXwΫ Ύ©‡~ιnκ5tvΗwΈœΓzD-}Υ[ύΒΘΪ.λϊΠ‡άS―ψκR/CB‚`ΐ:?γ―•@ν΅‰*3γ#3OΥ?όα{„/sˆΘμHD/ο”_žJhoωΆŠξlq²jzΛΣ¦՜ΩnvΗ₯ςkyΔΆ8 ƒf’S23|3S8ΊŒS:³Ώόε/™9tΤ4ΰ»W^yeΆέvΫuΙΣc³K.Ή$³ښςl%²}™1 ;ꨣΊΰ>υνo;3›ͺp¨όΐωε—_ž™ƒ`—<ν~ΩμψγΟž{ξΉͺςl₯ώP[%Ή1'\xα…™K»ΰρ£ύhfFΑšy„)'™-šg{ξΉg—<ι>ΊκώV}Σ³+šwmaο½ί0γRvΰ:]žuΦY^€Βz―΄”S1 €ηοϋŽϋύί­™―Λ3½λ>θN8Αϋγ§?ύi·2Ψyα…2Ϋΰαy™ƒŽη…<– 4ΐώJΗJ!    υ†ρńυVΏjκ£ΊΫΖΠμψCI»φ‘‡š™“H5YΆdα›Γή{ονrΠ‘G™W@α-‰œ¨ύsηΝΝ.ΎψβlΗwtΕvκ΅ΦZ+;ωδ“3ΦW€VΖ™Ϊn‹ωnc΄Φ.ψ2‡—ί_|ρΕ–Η—# ΰYέHή§IDS§NΝΦ]w]οΫoΏέΛTXΎzΨcθεΞ;ο,›&ŸG=ό†ξΟ>ϋloƒ9τ{•„GΫ$’~ψα^μ€€Βς•{s~ΚΎϊΥ―zš‹.Ί¨lš|ιw`@}EώκWΏςΎΊϋξ»½π°[`3όη?ουψσŸά?Hθ₯R†Ϊ$ž he9ΌΟρ’ΎοΎϋόžTΧWαul‹wα‹_όb0‡š`BKQ>έ ϋ°εAΑ½Ωi όδ'? ώπ‡}χ;la0|ϊӟφΓΨ1Π,ψτΖφΒ‡ϊ…έ4'Ÿ|J8ν΄ί„O|βaύχw/pvΑύρ §œrŠŸςFqm€•«»Θ 8ξΈγόΤ-v됧Γ±Ηλ|‘'Η·V“gΉςš-ΜHΫη…K/½4μ³Ο>cΚO=υTχŠg7†9*…Ÿύμg~Ϊ€)πa›mΆ©ˆCαψΊλ 3ΰŸΡθvwœqΖΑn}wc”]'JΧl8ξλφ4Ώ‘oΤ7ώΏΧw=OmΤoΒυNqyΖαz_,οV6~δOGRΕ_α|/‡°f΅UνΤoΪ¦wjg¦wω8zΟ‰n@~—sœG±΄•Β•#?γ6ΖνˆρΗ‰ί+~5αvŽe}±΄Κ·ΥžŠ υ©κ―ίκ?½ΧοJኧg>Ύήw'ί8―|zεΫ(O΅EνΠoΥ_ου»RΈβ陏Οϋ|žŠ?‹₯+—Vρ•·~Ηy*,~Χθί«mg―JΕΡ{₯ΣoαNου»RΈβ₯gW δqΩ5F}Ώ‘οi§5žwήyα°Γ cƌ Ώόε/]ίΓ^Θ)²θ觝všŸΈb œeOF¨οχMνb<^pΑΑPΨ\Fz cΰ;τΕ]Ύς•―Έ-:c>²#Ά;μίωΞwά¦}θa‡†!ƒ‡τVρ —θ榛nr# ΐ–‚mŸΣ’°₯όξwΏ ΗsŒŸVΓΣ6‘ΉΌ­΄ Χθ¨π@γ*?‚‚|Xγα’§žzͺ [Ε¨Šσ@·dΰέ_ψΒΒ£=ΪΙΞΗU9ρ»bεΗeι{>M5ιŠ₯Ι§#υβDN:h  0Ρ Εκ#œ({xγ `άε‘XC‡υ9’Έ²w(β«ύVΈήλwώ™―πbιβΈΕΒI«8qΈήχρ»ψ=߁Jαν±ϊο3_JV[βZΌώϊ΄pώωηϋ+ΖY)(–q‹ε™Ο£XΪbιΈκΘΫ%y—Ο£Xzβ $gšΐz•‰`΄™‘£ k’Τ Pxœ/Ηϋ*[ah΅δEΪrιΚ…©άτμ_ T;ξŽ#YΉΚ€9VZ(F3ΕήΕωU /F»J_.Lq3a Z ˆžΈrοOϊ“'³SI‚ν€ 8Ÿ\+ƒ³Nζ…>ψΑστwώWŒ–•'ΧΤόώχΏχΠβ,}½Ξ:λŽΪĈ“ –a@4Š‘”ρ`ΨΪk―½ Κϊ{ίϋή0z₯Ρα¨οεΖTœ‡‘w₯]–[ϋ7½η8βΣ~σIŸ2ζΈΒ ΰj—5Χ\Σ距wήΩm<0}4%Δhœh$ίΠbογtΔΟV€-^*lI‡γ{^Ž.U†ΚͺxΝπΜ·5›6–ΒoΉ0α\ΊŒpU,…ρ¬ΗmΔοΕΖ€Ϊ‡UΒC5α[β~ιK_ Ÿϋάη ΛΈLΕΡ»ψΧ‰χόΖΙP3NεΣy‚τΡPˆi¬T꽞ω{―wqώq:…λ]©x ΟΗΧϋF}Ζν-Υ6½Χ3ίΦbοyΔω+]wΓH_*m\N±ϊ(mOυiΤg©vkk5ν.'>[}BΉ„• ΟΏoTΌ§zWΖΧ±‘eόψραΏψ…oRΡ&:Ϋ»ίύn_ΰa³›,ΎωΝoΊΓCεœ[+ΖΝ7ίμΤU_ψ~πΑ}ΓHka xkΕϋ ³oηΫ~-NGŸόδ' φθ {φ;Ρ'μΈΣŽa“χυόz”β5ͺο·ΒW§ώυ―υΚ²Ιn&(,Ξƒ/αΏυ­oΉ ‡M}v2d}7,Υ€υq±y6ž£IΐoυέΏt|(½“n7vμX·‘’‹±‘ Θ—₯΄Εή‹Ο;Υ-Ÿ¦\X©rH(ΌύWϋ'σlΊ8΄ΕΚl]Ϋg5mPŽΔ•>«gV¬NΨ;q ΄m λŠ§'ykwwή—J—EœbP,NόNuŒί‘ήλY*ΌX™}ωŽϊωϊ¨LΥWΏGŒξ›mω zσφ‘O£χz– /W—ba:£Z:SωύLΞ4έ ^Ύƒ&9ŒΓ₯0M%,η;i¦Ο˜.Xθ {Π`σ(6ά…J μJhŠ2 Κxt2aάΥΚeΕρΘ3ώύΦ[o—ΙWޘqΌJuIα}˜Ρ7¬νͺŽ‚β=ΖqT#ϊ•ΏΧ^{-̝;ם π€δ7iπΊE™Α Ξb5qW]uΥ’χ±r‡'΄†q’²˜2 q-rBaŠ§tι™0ΠS  ¨φ·ΏuGrh™q­ΑKΩΡqΘ!‡œbμ8?w¦ΛyώσŸτ΄œ΄Α=Γ#?σ™ΟψNlN»!Oœi*εYΌf ΣŸφΖ4Ώ?œΆq2iθp…ΣΖ~ϋνηχ…rΗοG>ς‘Š'ΙΌόςΛ>ΏmΏύφa=φp₯K} ?$Ov<ΩuΑŽ]φ>+Ά°ΧLψ₯-’Qhφ?ωγŸρΑ)N’oζ½φή+ά0ρ†pΞ9ηxœrΖεΙ)PΧX?~χ»ί Ÿύμgέ‘Fσ2 Žlμ¨zςΙ' J~-uOq 8ϊ"°CžŒ¬’{ί“Π™vbˆgπDANΑω Ω!–SHOW|02sΖΜ0gξηωδ‹l’]AΔ+Π.re@Θ4ͺSœ™‡Ί3ΰœΩ¬ΰΈxύ΅-ΝŒ_ϊ<‚WvάσΰΕOtάσ.ŽχQ)|ΗUδPτΡβψδΖoxΏ³ƒ§Zύ(Ξ£žΏƒ7ΙΜUψ ?†Φ΅°Ε{ϊ<1–DίjαΣ¦M œ,F‘6Π8εFΖ#:#}¦wqŸ©oπoκBρxˆD)Β₯…~4ΦUF!]ϊP ςd1₯Ÿ‘!θίθy’· cμ όAΠ§ ‰~bšƒΗB—’λΠͺhGq)—ω€&ε΄ΛoΕAW₯n”GΉΚƒ8πdΖvŒwζUψ’π.Ό.πͺ~Μ‘«¬‚ώέ>n‹α]v υrv α—²¨8¦nΈfώ‰γΒ»ΰaτΔν ―αmΞσ`/_ρD φ‘Άƒop€qΐoΖ t ζAΰžΫWΤtAh‚±F<Ύ‡ΘMτΏ }βΗ‡ΣgΤˆλΗOί›βœ&‡όuΠAωΖƞx/c}=gšI“&…ύL_GφH4²lœ`+ΒΙ§†υΧ_?ΰL#ό6ΕΤήΡ|fs6šf'°{όOtδ[œFΰc ΞφkGT‘πςώχΏ.·μ°Γ>ζ–,΅ Υ6/‚―Ν7ίΘ Θr2䍼ΑsΨπaž7αGΏωF1|„Ϋ0G―F– ,dhι•ΚŸt@ό;–-‘9κMΉωxqdud‘Β˜:$¬ΈΒŠl-ΤaϊταΕ_πΊ’γΐφxyωΕ ¬α#x£>Ψ9 Fε²'Ω―(Ώ>ΝδY‹ŽΌŒ NΆ+ΒΐΛ²–™d_Β‘ΟθopH>Κί:>xO\dCυ<Ό#ίΗŽΙ›ώF—,Η‘ΌH©·ς 2ΊδO»ΨΚt7ΪιΤσro[ŸκŠn":††D qy}ύ=ΖY^—‘Π+ΐ9@\πΘζOΥΫΛ Ψ–WωΡFβοzOΌΨB<βπ;_δlκΔρόEξCαδ½ςώ 9=}@~¦g@Πή|…jΑψγ؟β O<ρD°»ΡΒ‘‡κB»ιwΨ~_π=χάs}PqK=5Π`°\{Α‰ μφ&―νΆΫ.|ϋΫίϋΫί\QŠΛW:˜=W8œtI~%ΚΪk―Ϊ~πƒ6™0IGόυGO3OΒ‡βƒ(Μ]‚«hΕKΟΰ΄ Νπg”Tp„ΗuΖγΞ/ΔΉώϋ­SΪiψƒϊ eΩξgφ N!’GΏiL‘pb΄ε )žυ«HM…ϋΏ mmmα’K.1Ευ9?Ε Ϊ@.ΐي£½zθ!o³θyφ7vΊc›°»ξΊΛ,Fžΰt°8>γόϊλ―Ηw¬ŸxDΎœDΕNΫ[o½ΥΚ·bqΠΓΩoάΈqnΐ$Œ>J£-\χ‡,Υl qΜΒΚΎϋμλr!σ*}Ηb ψη8ρηŸήΗ,FζRπ3ϊΓN;νδΗ‘Oœ8ΡηXϋrΈBfδΚDpK~€κ’>ΰJ]ϊ†έ™ γ8ώ£?h+23²ωή{ονccΣM7ρλ‘±}τQo:NτOϊΧΏ -–| Ϋ]νΞ©`Zά.αΟίϋήχ<|Μ‘Œ‚βgΪθ‹O<Ρ\‰ƒΔNXvcxςΎeRπ„ΛϊŒω—EτQ°³ύGfτލ‡ΡFΣ#Ž8" g?ώψγaςδΙ>v‘ƒ™Ο<π@ίMρΊ„ws=α,φ0χ³(›·-@K/½τ’;―C£\§ ?!έΧΎφ5?e1"ϊƒ8δngΥπ~Ε“z?ξΈγ<ό„N(,8¨-Π ͺ+ œ‚ …ά{ο½>–‘…±υφχΏέy.<§fμ8œΤHΏόΟόιίμΔ“…'μ@̙̝̕Œuτϊ9ΎŒωxœŽ…xi…o!Γ“vΟ=χτy’ŠG1wŽέϋΞάτ7;ν9QώΟΓΘ͊ͺΪί}U¬Žΰ >L;‘ywί}w§λ|ΰΎψΞiͺ΄•xΖ²Ol_!LΈ`‘ƒ]ΟΠ=vWά·ώzŽ{βpMH8UΊαΔ'†?ΣηΧ\s­ΧΣAYΘ&œy›ϊΖυQώιٜΐa‹+“αΟθg€μ’9-bb7A6H°LVbžα€\ζ#τΖ6 =:α*Έ#θΥW_ν¨`AN­Ξg:ψ< „Μ?ϊяΜV΄‰Ηme>ΔΨγ9T ξ\Ο*œ0ΟKΖΗn£χ‰ήκτ:6λΘeœrŽ¬‹#2²˜1J΅Ί μ₯\—Ž,p¨]Σ‡^°‰pŸΟνγvλ>ςίB6Θ!γƎσ“¨Ρ‘Ρλ)Ÿ΅Cτ=d:œΘ_ό_呃βχγΈ +’9›χΤ[锆Ά²)9Crύ~_ΩΟν?’λ©$i~ϋΫ3|Σ%ŠΘ.πδJ䑞€κƒξqε•Wϊ)bΘΖk―cmπΞ^ΏΛ.»ΜPhC)P>ΰ½|bCu©%bΌβHY—ΠWΨΌ•ΡΑ3˜mγΫάΊό…^ΟΖ0αPi)“υμbZW“ΌΜζD}‹Ίq2;r$rx δ°Φ‡‡LŠμ(ΐ6@:Ϊ†Ύw‘]Νχε/Ωεϋέ>Ά›Σk‡\kˆ< β$‰ή€\*ۍςνλ'8ƒR7t™έvΫΝϋ€ω˜› °q0.θ[ φ ڝ1wγΰ½ ?Ϊή±§pΐ§@ΏΠq₯υuxΆΦΧ‘― ζ>Ζ+z²e”Κ&'PΑ/}…œ¨K₯ο―χƒϋ« TNsc@ƒ§ΎWjρΏύo7>c$Β βΎZ 0& Rϋ™χΏM©ό(Ε†;ί0>qUΖ'm ,(κά›Θ‘c0N£ι`ψ7a&Γ1„vθaPqΒ  ˜p‰_/Χ‘>Ί™Θ:κ(7Έ`xcA #KΌπ\θσ{Ή Υ{=y0‘ab’οΉΒ„°ΐqžΎιί~“šβτΚ;=jΕt‹±ΐ Z,Ϊε‡ΰXΙ „ σόσν‚^δx€ωΌεdƒ†Π$Ί^†'UpΓb«v ‡zbΐ“£ σR% »'ήωΞwϊE†+„’3Ξ8ΓwhκΚ»Τ'νXX—⍨?ΪcΪ±—C†ΊrΑoβ.^ΎσXαω'}Η\Ββ) e`œGDΉAή`–χΜ3ͺK>Ÿτ»y0 yCΓwμˆodQYδos,:Κ.‹<1=HΗ°„†ph‹±" ›b΄ΰ4+œrάiΧd_θqN‚ζΰAyΪΞc˜²γςσαόVͺ‰[,}£Ό£}΄•IΑ8ι γ†ΗΩωΞfγF‰ 6Ψΐu˜-·άϋ}ΐξ(p_ ·ΒM©xzΟSί•¦QŸj<‘y :gGΊΰg>σYŸ7‘—ΡΕ }ζΩΗ+:%}}ι₯—Ίaίδ‡q_ΐx¦ώθ£ι0atBŐΘieŸϊΤ§ά8E•­τιY ΟΰιœΜ…}›Ό1§ΞaΈΕπŠ!φb3ΒBΠ.`xΞ•πΰ1’;ŒΛ8„`l&?μΠΞwΡ'#±σΏO0‡-€y…Ή Oh‘D”‰Η’ς?υW™ΉA>„wΟ;ο<ί^qe“xΕa0HX$`‘‚kSι#ψ8}ΔFŸΗLEθ,̝πrp¬€Π—ΘmΘνΨ‡pΞ ΎϊŒ ζVμLΟ‘μVτ!;\Α;|‹EfψψD<ζqˆ‡α€ M`ƒΑσ°L}fͺ—§iξ*T“±BίΠNπˆΞ]ƒ’0ΰσΫDΤ^= ™E_¦'AτΝk¬ισψgμ ϋ`g€_sπcœ!闏}μ£ουύ γ™y[ @qήQΡιk“aη6Ζ"ΟRπ‚9ΑΜοr„(·Uή‹²ΚxCΗΕ Q:O«ΰ‘švβ3 Nωؐ]YΠFFόιώΤO«ε€\dOl9‰χ?½gXl•Μ!¬α@sΰ‡9Ϋ ιΐFmΤπse5tΤΘqDΣθMΨ?˜—ιSd4dd;x²ΨΉΐΗB©φBΨδˆΟzzΥOν;§ˆ †ΨΙ/€Κε»Λ~φDWDήC7ΓώŠΜ‡l‚Ύ‡L‡žέAo€ηɘεΐwάџ’e°έ²„χΜ#8Ÿ?6γΠVδzψ$r%ςŽΘυؐΟqΔ°‰S5ϊ.tNœπΧ]w]E©ω©6ΰΐΟΑ)ύ'c­½2–ΰO8aGbά©PpO>gžy¦χΞPθδ₯4΄=δΕάΐό ~Α32ϊό?ήηΘg8TΒρ…Mη*Ÿυ:π̚vθe―½φr§,6ΈPφ-·ήR(’tJ«g!0ϊΒΊF— Ρ›λΑ²9}3 4ŠmώΔǚ3α8Ϋ"ϋcΧEŸ@ΧΣfΝrεGUιρWΦY?Η)8Φe΅±]±¦±ŽŽ‚½”οΠ,vD6kiMύ…ώ…ξΙF€4τ=ϊ2ΦΧ±±CgΨ:―θUθOΠP GΠσ&ψDΗΒΎ‰ά.gέb4ζ™υχ‡U>AΒ@M°βρmRΙlbΒΜl@ω;3ήd6˜‹ώ‘NimςΜLaυ΄flΞlWJ¦΄ζαž™Π”™ΡÍYκGή6ρtyo ΛίΩ Ν̐•Ω φ:˜±;3’#ρp›@2;"Πσ#/σσχ6f6Ιe6h= ξ̌R™Mπns_¨HϊR7MΩi/Ω—Ύτ%ο/ϊ΅Πο6{Ϋϋގ>wyΓ ¨Υ -™hΔV—=h-²«d]sΟl103…2³I2όή =ώΊ²λφ2[Κla03cEf H™-Κe¦{ΩΜ ’Qδ[[Τχχ’έ)Ή|§vτχΆΘΕ+ΟΛΏt|0Pž)ΕΩ’Ε‹β ¦ψ.ΑΜ8γύe»i23Τef$Λ̐ζrΌ³sΟ=Χqa‹s} Ξ‘/‘!ι [˜ρp3ΌΈ ‚#ΝψηοΝψVθςΣ\kΖ Η₯κ"ΩS²%ω"οŠγ?μCuuIτ=Ϊ‘o[`ΝΜpγψ²…ς̌9.“CƒΘηŒ@yΡG„™q/3γzf;ͺ<\8D·$ά M>ΖD/δ2½ΐvffdΛΜ –Ρ))ίv}eΆΛΚΣΠ‡Τ0Γυω§2dςcόR?ΪΑŸνˆΜΜPθavbMAU™ιYߍ‘Σ™ρ47nœχ₯98ο4G[ηπi;ΒΪι:ΰ7iΰΛΘyŒ;¦:ΫΑlπ\ωM<9γΑršΆ 9μ²· ηΓηπΙώΤλ…η_π Ϋ°”™ƒ‡ΧQ2₯4Sx7g4η©_ŒasRwΌΒCΜ‘1³“3sšπ6Γw‘·Α;<…~2C·ΫžμtGyŸΫΑΗ‘—λβγΜ“Άΐ‘™Αήσ£ΗpJš“HvΕWxŸΠ_πrζsrΝlΡΤγ`S˜!;³E„}˜a9£m€eήΏςŠ+3sςtȍ κ7[p^l Dς² m…ζΝ1,³Ε,o«-ZdΆpδΝ΅E—·―(OΖ“ϊ…>Ψ‰}.+!kΡ7θCΆ(δ4fœΨƐΜ <ܜΌΏ‘;1,³§ό½-(dΠ 2UFzΆ  ΖΎM!/@Ώ­·ίv»γΕ}ž/™‘s‡Εχ-τ‘vΛN†νל@ φjh*ώC…GJΫBθ*4UmG΅…ΜœΨ\ξ2gXŸ§ν$l­΅ΦΚΜρ"ΓΖΖΌ› 10€lΝ3ίJG‚§"§!i<˜³‰7yʜOόύ€IρwΘζpμοX«Λλ‚ΘYθζδ₯|Hh'b8-‘†MΕ6’:OG^d=H²½d h‘ϊ—4¬5Ϊ¦i—η™#ή|sz]šSŒ‡³ Cj½Ιεϊ)ΊάC^Θzθ’θ΅δ‰M%–λΡ΄¦…}:h>ςΜkόΠxb €ς°# # sQκŠόΛΊαv:‰—€ξk'‰ψ;sHχw€Q?Η›ΜφόKϊL¬{Hφ5‡'ΟΟNαrΉ‹ςi;γxΒ„ F?hL#‹™Σ†Ώ7Œ‚€pϊݜ9< ‰ΎΘΟNτρχζθW―ΐW±/Œ?ήΦC6ΟΠωτ;8°ΣZ2κJ½Θό#Λ#³bZ~ω·gζΨαφπGΈmš+θj¬o οΚ»/ž*99›:Ϋ-^gp‹¬mϋ™έdήόvYΉ[ν5gN―λρς4ψ±λœΌΏŸ„ρ}bWd •΄κgΥΕ6#ω{Ζk²cΠΣ?ώωΜ6@tJGžΟκ/pŽ^@ώŒ9τ…z„‘VΑ έΖ€Ώί­Ζ. vkΪ κ”—ΌΖˆ‡GŸa‡Ψx¦α‘ΚΡO6@ά› ―ŽƒΒc“ΣkluOFΌύJqvž˜Κ½μψ·)z”Χ  ,t―Θ§νdv•¦`ϋŽ ax=βM‡·»ΐπDΕλiΥSqΣ³ρ0 ΊΔγ―Rv"Ι“’“#lςρFει9n)t M˜αžΎΌΗS–έ‰μ`xβέ‰G2ήxτβŒwr‚„žbkΒΉg£cŠE—qήx„³;uφμ·άΛ8Σw₯3,̝3Χ_Γ …ωŽ8•ƒz$hΗ€ψŒϊ…yE§ϋδq 0‘Ό(žσiȟώL@υ]ŠΓΞ§ΆΆΆΒΥEͺ‹ΒΣ3Έ'>4 ˆ†‹αE»ιG[L)άη\, ώΪ”?Ώβƒγ69zy‚£Z9ρΞ/?™9§ΨΈςι£i0` §~Νc<η#³²ΛcͺνΤβD;3Jψ 'ρx…ζS8y‘χΠ,r2'έp\*»K8YWΐυbΘΧΗ©Ϊ’’ΛΖf$*π ōŸf―‰–ό.ΎS2B°+Y͌;Ύ£ž“gθ δBϊ‹lΰύAΈ¦ΩŒgv¨ΑλΝΠγ§Φ‡ο˜«½e±ξωΧ,=s*‡-€ϊN5ξμΠω mvSΑWΑ?<Ψ θ~€2»κΠγCŒ dΖ»ͺΨ)Η-ςeΗ*§B'Yπcδ€R2>;ɏ:‘SrG7ΐ<‘]‚τ3»Τ“4w<Θi†28YB;ΌHΛ %μϊγev†Ν™=ΗσMΌμ4@oΨΨ=ΚnDθ€~„p )}Œύ‚0τ@ήΞ΅¨ζψθ»2 ηψwx2»dαΨ( χ6Ϊ° @sΨ#ˆ‹­…γΜμ”„~Ι›“=8•†Σ”ΰ=π#δ 3~ϊŽΎwΩΩάΠ΄f,ΌΓ“9ζΐ £G·γό³3™έλŒqτlvL‚#Ϊ OαΤ vBrꏂύtΖ!οH_ΜΔΨGώημPε7»ˆα \ρ°³“P˜/( ΎΕι τ3t€½Λt<.Δ!.ύ͎Nϊ[ς'<ι#»~$<ώΔγΎ“ώ…]AτCϋ h;ΧRB“œ„μ€'ψ0;e‘cΖσ+ψG€žώ£ΚpΛUzΘΨμFw%ζ h@v?vΗ² έ‹pvƒ³Ϋ˜]Ϋԝ ΊHΠόˆιLcŒΜ‘%Ζ:§ˆ Γ±{Ϊ%β6?†:·PmηΚΊ“~~’Βs9Α˜0π– ’θ`ώβšHζNICV…ο1³λ}ό1ΗΙ~ܞSk~rΣ§" g#ίπ'ΰtζ^t_0!|+NzΦΜ’pŠ'L Ώ Σ8)yx欙EO¦>|˜ŸVΑ‰ΌœΒβqvΕωΔ|œ<σΏU2s'qΘ,Θ δPNIAξΰ4NQαD8ΡL°E1y&ŠžWΉ`<γHi&2θ/¦)ΚΕ ƒeΐ.ΜQΖ4§zυ€)mkbPN`’ηb˜€Ώ!L!œΒgΛωιZ›Jy’ρ©G‚ΞPΏΐJqή ΗΔ―• :̍}€(‚(gηž{ΟΛ,ͺΐgtΖτ„ϋ8†ϊ@aΥΠ7Š&€1Εψ†‰7„ν·kΏ–„χΜ (|\Ήΐ‘χ,. 0&h^ hΜ£ŒΚα>©χΈήgdRθ&V,Q4QΦ‘ChR‹QΔXΌεΪ(ŒK—΄ηKώπy XdδxaυXΈMP˜+qτ`ρ ΓΣδΙ“χ™cΔA _ΰ'τ)ίI‹βOΏ±hΗQΖ,Δ(ΌΊtŽEΎΝΰž‰ρqœ΅p1γτ^ήαLΓ xOz1b€Δ˜ΓΒ δ…‘q@΅™£i9.€n‹QŒ2έ₯S2^‘γσs:cΐЊ#M<Ξi—ΚΕ±Ϊ’ž[.)¬ώ0@_G/GΥ’γ¬ιgœe0`ΚΙψ%Fkψ0†JΙ†ρ}κ―ΓΌΉσόξzςRJ'§^d ρŁ?qό9Μ#\„±ޏΡϊm£Ϊ •Š―Ί6ΤΣPρΟρπ,`΄ƒρ^α±.GαΌc˜‡OοlΪβͺ!ψ8²qσ|œE ΰγτ±_Α‹XΧΐT―ΏΌ£_Ι…ώ#ϊΐN€#ΪΑžB²@Ό8%m˜―rbαώΞ&"dρJϊ~σχ΅4”a)\ΒηαϋΘEψž’'‹,Δ`“€Lp-•pρct(½―₯ό·11§7h\§X,δΚΐNJ*Θ‚ω4ΩςΪk nh;<{»νΪwg#Ζ“@cPΏΣ3Έ XHFF`‘0^Sΐ6ΐΊ…,μψπ/-Ά"­©Νl0AώΗ φ?ψΛ\Μ8£βδf'ΖΉs‹ΪΜ·’ΟDsυ‰ζ\6"\‰ŠόΔ8`ξ₯Ο‘ΫŽ=φXοGΙUΔE.CΦf…υa£<ן"Χ©ί«α=̌7δcδjζzΚβIωθv8Σ@{Θ‹Ψ\_p’A^‰ν8€£LΪΕ΅Uθό΄‰wŒgθ“«¨δ4$ΊV}₯k"#Ζ2αόρSι= Ψ–t;MΔλ ^‘ΏhŽω8ρPΩ|g:z5ΘΔ΄KΈ ^5u#_ڊŽŒ,γ\2 o³ μsΠ oY##.}€O=y/2:Žˆ€η¨¦.±ΜΆZθPߐ?λήύ«C"§LώpVb]Ν_τρ‡dm`΅ΩΈ‡^$Y›uϊ>~βώ-|‡μL–†–Yϋ†fD§…t…|Fnw8Η–.ΐnΒZ(zΜGν*U⩟‰Γo’χτkμ@ŠΏΟ³ €θ΄θ \ƒˆ# ςύG|$gš:ιˆF­ԎZσ{₯1βΘ@»WΌΓƒ‰T^€0GŒŠμΑ#ƒ?F& €ς,‡#<»Ω™Λξ];JΫƒ˜…;¦Λ'6„Χ˜&J$€z‹™πζ@]&_κOΠh3ƒ'“4@Ώσ§‰ΈΪ²CXτΞ†ςΝ$αMB “RΌ‘šPͺ {MΜώ2}$ tΠ­xJG) γ,t―ψ₯β8•ς$ςDHJΠΒ!σ‡ζ—Ξ1Ϊωζ:ζUρ”|Όψwœ ρ‰aΨkGeΊRˆc( €©&︜fύ­Š£oαIaτK₯ωΎFιΐY—°Ϋlέ~ZH¬ˆΩ΅H ˆrRΝsθΤ‚ŠΚlVœ·b»Τ§Θ©rόi #3=ώ˜Λ26ρΗρ~Π$Ž€]eμΪ°ΒΨF&ΡβΚ<;vdX@υρι£$Τ7ΰΠoπ'\r‚ ΈWζ3cgΐ έ†ΓF·ΣΧiBΡ! œ!1ΰaD½κͺ«<t8ŒAόcΝ_›nΦΎ±έ…8¦00ΰ 7²‹UΞOόΖ‰†τ…ΠηΠYΜT‡<Š˜/ΉΛž‹πh6U`Tc “7ΖR i€Œh|7SΒ":ί)C΄#£2˜ζβ$h, ˆnDqν‘+.ΗGύV<с~cLί6ήbΑ‰‡wψ <]:D¦ƒ§Η 1ΑΆ N+CΦcμ`ŒlFΉoυΥڝTb<0GŠkG&α1ξσxΗώ`wͺ†Σ7ε`ζ~b7(;j™k±‰‘γ3/Σw€δ?ΎSρ9ΫδίΛqŠtŠKœFh> ?΅« |1„œKωƒ7ΓOγ~ͺ¦Ερ2Θυ[μ~Μ+<Ωe’ŽLI‡=P²ΗkPp_NW‹γ§οΝ…Ρtπβ /†ΣN?-ΨU ήHwΠ±5~k₯ΧfΑ”ΪυΥWϋόrψ·/œπD q£οz6 jmσ8``ž–#ζΒ™Γ9=‹ΝP8nbΗiEΌ1i7Ξ,ξc[?όσ}γˆζv™qͺΗιΫ˜K™_˜γ•ΎΦ>Jρϋτ ²‹τΘΖπT»ŽΪ’‘Ψ4Ύm|³φwΈ?¦V|g'4τ:d/dmΩTͺ­ΉΖεευ@…±ΙŸMLv]³ΛpΘ’娣hPρyR?ςSžΤ£­­Ν«†,ζ›oϊ©’ΨΘΊfCipT‰εO”ϋPyΉΧ5ύD‡Ε‘§{œ™X»EW ]θ½k™<ΆJ‡3ˆ2–Šσ$zφψργ}}TN#π0ιηJSκ)~‡^ƒΪ†ύ”ΊΰLΓΙ.ΰYŽM'NτΏcqe#€bš‰σ―ε»κBΏΛAEοβ|X»“CO.=€7κ—Wι;²Ά]­θvNΖε` ζXΦFg₯τΟΈήωό£ό±Žc'’’›’7‘3αЏ Ρ/λ ψΣψT9<Α 2xΌ.Λά8a«‘ΰς«΄;'‘Nωxuς‘Όκ€#΅b0:”αZ€iwΊ9sd2OΗx1!Λgp1XQΤ1z3a΄ό 8†%7&cvͺΐ&7σ‹YδKϋ˜4 &0ήzΠ^ΡτQ5ˆΔ˜ΕΰK&.΅ Αϋΐ) `',Š&ΒSž^8ιcμ:c]„΅€ή‘Ez$ t νΪU‡QmΗbˆΗΨ‹£« βπ8Dίτ£9βπψ;”ŒŒeπ)šΔ£ˆ£έ,’1oHa‹σJί› πg”ύP>M-œ63νυiΞ#4ώ‰‹q!?n‘#ΙΘE +)Θ€γ;Š~[‡1Η] ΞΫ_tχ#’}Ί›E#€ΣŽκ ξΰαΒ=ϊ' <Θw2ΗνΑaBN₯ψ~ΏδχAEBšχΠ1c‚]jΜ{v·vΈβŠ+|לZΝ.5ώΨEΘΌ¬oΞg;ΡχΈ*PΞeŽˆΗ’:&‹δΫdτyο/ΛΗ©}ΞNWƎŒŽŒ1ΖHNγT( “1IρJ=)#Aγc ’< Iάbό—w8b°S#2§!Ε€ΑC©τΙ8Œ΄πzζNJΑ™DFΤ8n³|1rDΡ¦h\εηΩ|dυ‹ψ8Ζ_ηγ†K…) Ό›λζΎS†ζΫoΏΓ–ϊ—;ˆhƒ–ΡΙτNOΥ³˜C‡β4ϊStŽάΜug,±HŒm„ίψ:ΊMq]ΈQx©'ωΖς΄9nϊέX¨dpƒΎEγΆήΥ $ΠΗ/5ιβ¨ΐΞβειE4Eωςqβ|Σχ„Rέ pK8Β(_Jΰ˜9s†/βYΚΠ«² O(&£³Βxͺl ‘Ε+lανd7ΒάyνŽNyt଀Σ΄x[ͺ•–4­,’ΣϊΟbwσ+Β°ςVΪτlτ΅3ΕMΗψQ‹¦rz)4Π? »NŠΝ/κS=œx$?Εe¦οΝ‡ΛΈe8rVDy”| +FG€ΡBά~ζ4ǝbωτ„I~εςζ{-Π‰Vs2Q-ω4Rά<ώιQ0&pδ4}“Ύϊ“φsα^†#ή׌{ϊ םϊ·Ip/\0‡ιžoŒηœκ€žΘI£θόaΌγhotΔ•Vν›#0Θ²K γΡv½ 282 ϊ s:&‹8Σ°ΣxίΖοσg'|ϊ›φΥ ~Ο1β8Μ°$tJΚΔΑ†?6q|χ»ίužeQ{Η‰Σχ–Α€hƒ;αΉNžMαΆκj«†΅Η¬ν D2q‹wΎ‚(ς€'aμΗπΰ8Ι5h—W’iL^*>Ž!ΎmΔ—ΰœψ§ Ψ%<‰ΔΙΛτŽ|@)όΧϊή3k ΅ζ‹_ό’;Er2'ŠΑίΉ cϋωγ*ΔγŽ;Ξw!k\”j*v•N²H©ˆzί1‡ςSύ§ τL€ήDπYΖ-ςŒgžy¦/@I–M·"ΦΤφ[o½Υε)Nd3*ψΑ 's!s±χŒ9ϋ3V™ΣΈŽ#ŽΧ 8T{±Ο!Ÿ"W ω'σΈll⩉Η5 2…/δN¦p&–#›τcΕγN:ΑA“ψΙ™¦ώ >D^ΒIωίψ†λ{θXθ}8œ p²3›βu :c]NW;έzλ-αφΫoΏώυ―Γf›nζΧΐ@ΥB±5₯Ε1P6θŽϊΚή‚CO©qI5¦ΙKr=§PcΫaƒΞ3πψ#r"rτ έφ΅Z―Ί?>qāƒ8=1Oτ‡sΞ9ΗΈ]λ蕆φlρώ-Β 'œΰ§θΠG\Uc”tŒjπO~ΰ·T\lΪΒ=vwlŸψΔ'ά±β™g°΄Λ\W…£¬χRo1JεO;ΥΘ‘qϋcϋŽςθσN+Tώ‹κ‰ύƒS~qζemδ9Nα,Œ3,sQ9<αΤΒψǜrC³>ŽΣfvθ˜ωά«|j©>£ŸKργ4νιΪGWRk&δ^ΐζκPœiκ±WΈNΕGΎ9 \Όˆ1rξ³Ο>nŒΔKQŽŠ_n€+Žž0^O Φ\“γ·Χ μκ„1,ƒγΗuΛ‰Ιᕉc`-?π‹ΕOο₯hŒ Α©Σi‘1†Ύ5»ύšΈ΅LL€Ž”sC‡οœr„ΐMiΙΗIΏͺΕtŒ‘GŽμDΨ‡§σ­W_mΏκΑ=^ΰ‹Η‚βC›L V(“σy’ΰ8oΘ << E?δŎsJ#'°Νž5Ϋ±!κ OΰΈOœ1„C"ζϋEρnρΗΨΗ€Q˜Π Πg@άΧQ7Ρ.@IDAT oΥ§hε š8­ υβη(ΏŒ3Εσ/ΉœΪp¦DΡδήW@εΕίO@|L¨ΏHM‹œKρPΡ|;αB‹y€7ΛΡ‘έ‹π- ζγ²3ŠΉ@Ž`ωpύ†N%δΛΤxΠιYž¦k΅”US?αrE?` ǎJqγcύ!~_ξ{)Cvϋ8?i"ά3&Πί =dhώp„αΖv a CodΡΊm\›Σ2qŒ€€αŽξύΜ±LΞ«Ί£ƒ²αtƒΚαŸ0ι”Μ›ŽψγXyœŠ10bΈ?ύτΣ}χ:Ξ6μ Μ™Je€π„azΒ‘ϋΡ1? Ϋ~hΫ.«\ΠX•[Q5|\| Η>ΰWΏϊ•σ%«/xOm4PyΌo5ΐ Ξξnl*ΨDψc79²Χf± αρΠ.ͺζBι²zΗΫ s+PnΛΡEΝ₯ŽτQbΎˆŒΑόŽΞχΡ]ν”―ύΠοΘ0ŽWSM>ΘΙ«ΨΝε€\ΰ}¦G,°ρ‰ΎΑE>γͺΝNφΤ&ΔI₯&!CrΕδΙ“ '[Š¦τDƏmšo*ε݌αΰDzk'ςP%»L±tιέΐ`€ΎD·CοCώBOγy›Ω‘‡κ›ι9)“S‰‡ mίDœΟώσξx‚­mί}χuη›sώxŽ_o“·”k! ©C1»υΐΩW›όΙGφuζκ―M„iό2vq $ξvΫmηΌGœ84§™rcΊˆ©‡μ{ ¨§t[x7›PψCζ‚W³^ΚΪμ/ωΛpφΩg»#moΗIƒSBΰSLΐq±ybγΚU[:ΉNΠΟ§A.”ψglΗ6μ^όmΎωϋ}NΑF€Σς#Ž8ΓβL£:Sκ›/‡χθσ€Wίς‘!–΅iτλ Π&Ξλθ-§žzͺ;&Ι9άΔOeΎζtΦ“œ ƒV¬/f:²3VˆƒϊΌsj>vψvΨΑŸf-χ¨œB†|-ΰ„Λ5~τ/t†^yΔωδχΑYx*»΅0 ΑΚeη!€Ou˜+¬&&2&Οb όx²«…c·9ώK@^L¬0xŒxΣ:aA†Vp"@ΚS™,£XqΗ RΉž }4%iw" ₯ο’Ovœ=ρψ…Ά‹&XΓ{γP|ZLSΠΒ‹ΰό㠍ς(dšΎ$ Ԉ ƒI“& +θΑΓξ]wέεοα½(#aροHΗ;v„μD’Σ γ@c…δž{ξρ8 bAΛ_ΆπNyFv€¦<8şΰ‰~Ύ0° @°“XW&Ζ}Βχΐ;J §­Iρ ΟΤ/ΔεΪ „U€έ@β3ŽχΊ·ƒx¦_8e·œxh,hlΔύγμ%OW8qβς€_4qnΰΈTGb-θδϋΨ#€†Η€Ζt$§ρPϊγ'pπPψ…h%ίxβ“΄»α{7τ`ΨYNόζ4 dchR2uΜ#ˆ# OΝ ‹Κ"Ύκ‰€•A}9ΎmΌ£αΪλ­¨?`TƒJο?rΪω&…Έτ%ύ‚1ˆΎΔ›rΙφ'΄ΠΑΉ#€ lΠ™ΆΆ67ϊ°ϋ `ρuΡβvγ FτGζ8dn F\«%G'τ6€ΉU2ΚΆΫn[؍UŽί²CΝηώράBŸQ' θŽμ£n€ΜΛεηΣGΒ@’ΙpΘΫm»Σ>c_Ό~¬kΚ0NŠθΙŽή“~v’ηŽ!ς[ίϊ–Σ/ ™Ε1”S–βηͺΡ²?%c„—nΞγ9”9;|όςΛ.w\!Χ±Σ`(|€γO8FώVυ£'h±6³q* Ξ‡2¬ƒŒθΰμtv@Ζxπ,Y„>Π'œVʁXρΚ=₯Ÿu¨iε’¦°ΐc[1 @θαΏ8ωΎ­.YΊΔΗ5ίωΣo΄m"s ‹Ζ\­£“Ω,Δί΅φwΨa‡ω‚)r βΘ_΅5ηΝΈ‰_βπΝIψφ4d]dϋ˜žXϋν˜…Kth­qΖ1­Ϋpzz2xΠ© ϋ2γ0œ?΅Oqsΰ1 žΙ’>ΫqŠ‘NΛάNίβpΖiW\γL{cšλuCρ~ηlΠ›#θsN§ΑRΝX!y`'焁κΐoθ gκ‚ž Θ.‹.ϐτ(`γWNύώχΏw۟Φ:‘λΙ Ή†r”GœΎ Gόπ‡”.~WΝwε Ο9ΟnύΏΊώ‡< gCϊ4sz.ύ?Šyϊ+΅_}ŚνgμjfNtDζζΦΚςυ-V7εŽ<δ$3N}Ε. ˜?Α?ώx?ρ‘wΜΙl0ΐh=χ(Ψ]%CΖΘ°—cCρt΄ΖwNCB&%nw‘ Gv7ƒ^Lm2pD[Ό€½½τ/τŽƒWUκκqα‰ώPŸΔUysϊ›…«³qΘAV'ω‰‘oΩ.cωœ1Ι‰aΘώθ ΡΏΩpŠΑ~¬©ŠC7¨χ—ΎτEί\Εf(θBσ€hΪ#πGηQ:ΐ•IΕ774X0b\ DvλΌη¦Η=ŠθςH,†εΗsρ’Ε.Lq<†Sε§8,h!Xή€ZΩΡ#Ε©I•ς)“?v*±σeΐ‘Λ κƒzX=Ρ Ζx9˜74}`°G9„FΊ!tα) p—$^ ΠŽhŠ|Ψ=ˆQ‰“VW–Qι£Χ0€3ΝΧΏώuΞαi,<‰§A{KŽQC‹SΠ/BJ Β³vϋ©Rο~Ο»}1! AΎ¨±€δƒ²vŠΧ“£v ΔS| 9Σp„¦”'π0g‘ΰμΐθ>AΏ0/2W)Op`Œη6ψGι‘―ω#ΧQPFη<° @{Ζι£ ΄rΤ,Žΰ§\”MηΧ¦ΌcΘΗ3>Ξ5:ZœΏ(Ύτ }CρΊΗψ5aΒ_\a 2WqΘΒΐ₯—^Ξ9ϋWnP°)/™ζ%Jυ-TŽY…oŠ‡BG8jΛœ]XΠ rƒΖz)ΜlΆωfNk8Ιp₯;αvžxBŸΘ!MXμΓP”Κ Αu‚ώUOx29GήΤ―•ά³@ψμ3ΟΊα~«1ΞΠkΞ=χ\ηΝτ1†Œr€· Η8]ΑW”Έf‘ƒ!q–,^Ά¨X.ΟzΣΈΐψΒΏ]θΗzCc½h»θ=QΞΑ8‘ky›ω~Κ"€3šŽ±ƒs!οΥœW¨2Ο½ ΰΔΧ‰…r2t₯ΪS.ί–0P π #½kΔ{‘ρΨ-ΚψΗŠΞβ€γ+‹yτ‡ϊ ~Νi gœq†ο€$ž ω|obοΓ-έ'ζ₯ΨηΔKσ WΨ_tςψ…‡3`aήθ Όjώh•~jεvB£τ7c˜SΌXΔαϊ Ζ.Όp~`㝧ώZFΐ“N`–Ώψ·NΧeŒbΟΐξ.gρV£7h:Γ,'θ‹ΕtΡΆ7Φ0;wί}χπ| ΥPενΎX(g@of1YHψΈEχΐm[‡ƒ»ΏHu‰xrξΥW_ν} έKv’Β¬±hsWœ"Λ"»ο€δ žlŽΰκ$€Σ+X ―†7C[Ψ4Π Ÿ1η P`“\ζγϋgΝaΐ™€: qΑΆΣ":!φ>'tuΥωF2ŽΪJ;Ω΄gbl±]ΉIΊ.φ{@ωω>„Ϊ .q$Ίΰ‚ άaHm ;δ)pO°ζ‰uRϊ₯KΪν>ΰώζθ>N݁oU”Α‰ΰΨGp>Tp„Ν•φβ(…£ ΐ|Γ{ΒΕF@ί!ƒ«^O?υtΑρΙΉ˜6‚©Ÿ‘+‘#±“F h†qE»ξΊ³}γ4ΈžpΒNθTvΎCx :¬@Π6Ό—|βΎ‚χ²‘ ­ΚΑΞ‚c2΄C]Ωγ΄θΈπs`?ΣΏΤδ ¨=<7ήψ}ᨣŽςχ?όα Ξ¦ώ’N>ΊοŠU' HΥX ˆΡkTS­„D&˜$QžyΗΒƒ―4/rΐ#ŒΛάq§ Η½qμ _L| D(F˜G σΐΚD€αšΑώK/Ϊ]ŒΊ0Ζ‚ΓOϊSϋ󟼰+]ΜΒG]a`θ°a^Ÿ˜6ŠU° O}‹Δ άΘ$ρ‘m>δ;b1ΪCμZΕΐ¦Ό 5”Eξ Δ`„γ‹Q7ρΠ$ZΓ#£yp΄Y‚„žbΊ…Ύ0` ˜ΰxI'ΉpŽ—9ήπμvAΰ‘⏀ΕAζΨcŽuΕϊĘΔ"ΟΥW[=|πΑNΟ4ΠτΗwx@ΠB¨ΕS`'¬W#hρp>PPϊΈ˜'|ξ©μfΑΟϋ]wέΥ1ζιΒ Œ‰‹ΣΗgόΦ»Χu…²@JŸ‘˜± ƒ ‚* B5<†yc—κβ]β͍’¬}η;ίqΗ$œŽπΦΗρ ž―, „a,Q8~ vpϊ‡g5”>ŽD₯Oqδ₯™Xΐh ΰ‘ίκ»υ„³Vx2ήϋa«πeŒœ.…!G9$Λ(ΘyyF΄‹Ρΰ裏7›£§TAgμJaA ƒN,€Πͺ”gKžVήΠ5tΞ1Δ\Ή@~„α$ΙσφΫο°yγ»n\ByvΉΗκΨΜ ƒπM[ω0°±ϋ–yύηζ^ψ Όg$xπγΈ°σžτ’υ$? C8R8ΌbΒ1bψˆpδϋύψeξŽΧξ―ΈNž¨?Ψ΅‡Ž₯c£1¨1 QΒX`ΐ˜Ξ½— €3ϊ‡έ¨<‘a˜#ε,CtJœΡε˜Ή"J'’Ζ»ΙΤ €eή„c,’N©Όqμ9묳|‘χΈγŽ[–§υ­tβRύΓ{κ%Ϊ’¬‡Jύ¬π˜¦βVζϋŸMdjΠ#α1œbΗk HΟG6Α&`ΖI²ΔP8y‰ίGy€o‚§γ Œ=c§}z¦uό‘ΩRυG/Šα4Β»#Σ!_3ΓgΐχΧΎφΥΠΦ6ލΗΘΠπ&€kΈ΄ΐΎΞΨu―ΘΩθZπθŒτ£Žƒ₯ζsζχ˜ϋ1 WΝΣ•βΥs84wΘ!‡ψ&"d_6±±Y Ύ‹Β’ ΆhUς0C°Α@ϋΰΉΪg‘‡cκ™o±^g'Γ©/…μ†ΠIώ=αΌc\S―ΊαθVθΕwI^44θcψ)Ž lj.‘ 7δ.4`>Έ\9‚, ‡‘†F@7+ΟRώ4W5ςΌS­΅„q"4z6l π?]›‚œΚ©ε67μi-‹³ތN$|‘Ϋ"oλͺœgΜ Mζhl6_ώς—―·,Ξj!Δˆ / oΉθKτζcŽ9Ζ7ag@ξeΑžηX«Πω˜«‘«Π‰ρdςB†ΨsΟ=]n†αΖ·΅…εmώχ(/8πp Ξ4¬Η »#°^ΘfgδdΑ‘<œ¨k‹lƒ&ΡMΩ¬ΑU:¬]βtΝΊNμ8vQ7Κ°bSΑNˆάΗ†*ddJ‰ΆΣ&€'νFW°Φ-uŠŸΤπAžΘ“Ψ˜)uVœωΨΔF+pΜˆ@NΓ–„=Crh^ώπΞφulΨA‘΅ΆP* ΈDFγDΖ,λ΄ |βΰƒΣ <}_ύfmϋ²φώϋοοΈ₯ίd·@ΧΑ¦‘“Φ‘36¬oBΠτψργ½ iψ,τLΏ€sx/γΌj“)rTŒμ-ŒΦ7 !pskHŒSγ4ΚΙ”δIϊRz3φsζDτ\AgΤšV™}…Σͺς΅Š$H¨ Ζ=Ύ1°Μœ?³ Ιί)¬T† 7ƒEf»¬2Xžž<β?Sž33bήΩ©1ž₯ ΆΜ§όύ)Ώ<₯S1Ά£(3£HfΜ«NyΪΐσςŒYw©§ζ錑tIg“Nf™BΊ€λTxϊ1`M™0“™ρΕϋPτX¬RΠέ ιρl‘©KεgΚtfBfBFΊ°‰ »θ’‹ύ½9dJ£§ d™yκf&tIk‹™ k™ƒΌl₯ιR‘τ"a 0A(3&3!Ώ ν™ •A»&΄{ΞvL’?ዦ z|[tΝΘmςΫύΜ„.y’Ξœ2Κ;₯ρι£Nž{ξω̌™-ZwΑ#|Δ +βƒ³ώπ‡B\;ΥΓΥ?π2SΘ2sμ+Δќ·J+fΆpŸΩΒwκ… °]™-ΦfΆ€ΪΆ Ω†ΒXИ°“B\S\ ύ¦pdς4εO}c»i2S2Sβ ι*T170Gθ{sjΛlQ³ =Ψυ™-(e¦pZ &>ιD_…ΐθ Όΐœi23ˆtΙ—΄fŒrή-ώLRΚ1…Ϋγ›ΔsέB“ζΤ“ΩΒ`—ό[Μ`εο™_ΰ?ΝΒƒ9Λeζ€αm΅+„:αHm&9Ng¦δwΑΈgμ›³cΖ <ΎΞ |…psβw½ΗΦ»δ‡NΔLΎf(̐wΥΧ4ΰ‡κ>iΖ­Μ³Ί΄6ΫβC†ŽFι˜ίηό5Ί% 9ή­pp(PzžζŒκq0Ϋ€‘™“pf†¦BΚ Ωή2τ9³evΚ™Η5γ§ΏV=”'οΙc5ΧΘΜ εq¦|³>1 ~’Ÿ'L˜ΰύˆό¦px­9ΎyΈν`-4FαΆH›ΩB™‡ίtΣM/5£efΖ_/:Σσˆ#ΘΜ92³Ε8·“><ε+6Κ2§Κ̌β‡y¦iNm±•™9§x[Jα]ύ‚NVιŸ~ϊιΜ μαŒOΒ튧‚mIψΤΣzΌoΰΟΚ“§S_rή΅E³\”™αίΛD―lN6…zPnœ§ψ…-ΎxθȜo;Ερ ς!άCwί;ϊ{™9ͺΪ.όΆ΅΅e'žxbf $…v’ŽΉ΅˜‹LÜ|ψα‡{^Ά‘€ ϊDως /αυ‘‡)„K7"LᢘT‡+½IM…Ρ&ύ,š©φi' 8.”GS!¦ƒ>.‘Ϋlα8αΚ0 ZAFE'4G‘.tg‹‹™9ve ${šπe έ™-ΘdxŒšcBΖάj‹γ½@΅)‹ώΒ:12šΦlβ>廝Ί™cyΑΖ0ΥτqδΒb}\4‚<¬…U“ήκΙfί=πœΓ󔝝xΊαΙ†§)§Δp »lΠc~ΑsχLΣ.E•‡g'Χ5α΅h Μw'αΩΙ.Ž―§Œbu$w7²³Χ΄·ŒxΏιΨΩbιTnzΤ/Π‹Mp~΅ ΄Θ_1€†π|††πZՎΐ8ς„† Qδά[Nθ/YήAόΆΙ8Nξί‘#ŽŒ&»‰ΌqρΐΕs—1€r<0}$ τ"πώΕ˜–ΠΌ•“ΊΖƒwtcL†£Ω RΜΫοϋ§<fΚιyβΝ ΔσHτμh(ϊ!άΐSLΐτIπ<Χnjα~Ρχy)ž2α vΊμΎaWCάwΔ£ίLπΣ(Lyπ]™šG΅Wω₯gg Δψf—ςs ΐNNΣυ8qJv.βu΄ι&›†M7Ϋ΄ηΙ)$?τp˜1s†Λ/ΜΘ7xύqάBιKSaΐ ΛΎΛηX; ׌-.°kω”"mΆ› =ΐ؍„ό€Œ‚¬RΨi…¬‚Ώλ\Ή»J!-q4DθhŒΗJΙ΅€aŒCsœ <β΄ι{ϋΥΥπ=μ£ΘT₯vψ·bzA·CΞgήα=σ'Yp2Ηm5<©½1˜7˜?›xMaηbMv.a¬±žΘŒd;δfx+²ςμ΄™ŽωΩ;–έcήFž@^βΔ:Φ°s gΗrρyΗIœhΓi&œl‚ŒΎ ‚^MΕΊιb€‘ΝΩΐu~κ<‰\.=5¦]Φ9‘]τIκ ½"{(>mC7DΦDv`OΐΆΛ‰9¬]!χp:v£8Ε―εYΜf„œ‰άΜΈ€[s` O<ωDΨfλmΌΞω²9‘—:‚?μ%τs*r,υ•ΔI윜B²1吖ώl§σ΅ΦšnsEΧ‰AεA#ΨNΘ\ς\b#‘ δΙbGdζφΥW_Γl*›z}ΡΥθ{θJς"ύ…žŽ φΐ›)  ΰέΨΑz—Β  iN€œ–-υO΄z£έ’΅‘tUζdθUq©tfKψPΰxΜΪcΒvΫnηΈ"/pΗυP„ΡτMς‰BŒ=ΪΞ΅kτiœ7σ}ΝΊΌl“θ<ΨΦc:†Ύ9 έV§:)/ήzΛ­αωžwήΟf<(Ό?π[ͺŒδLS 3ι}Ÿb /ˆΏš<‹Ε)φ.ίψjβδӀߍžτy5i«‰ΣΨL΅( T’­Jαέ©w_δٝzΤsšJ8ͺ^¬m•T /–g«½«„£JαΕπUMšjβΛ;½k, Θ™†γi9VΖ–|+ΊK•HΡΟ—”W7}―Ξΰ]-N«ΧŒx―¦νΥΔιMάTS^5qz³N)―„rΘΣcώwΉ΄­V žςqςΏ[•Ϊ^ ςqςΏ+•‘Βκ•ψα@5ϊZ½·΅7κW _”QMœή¨KΚ£w0PU1 Ϊ}z₯P•)gΪζJ‘Όγ† S|ύΦ³ΤϋjΓo Ÿ•Ϊ@έͺ‰Sm”N0\o‡σL8Tŏʽ‹Γω^Mœ|šfϋ] ͺ‰Σψ¬TN₯πFι›‘RΡTΟϊΔα³w'€šU½ΐ; ‹¨Θϊ$a†]Q4 8@D‘ƒD@PΆ lY|x"KXΒDQL$« BΨ!$ΰ {€οΥ―ξύ©ΫΣί]fξLξΜΤ™ωnwΧrκΤιͺS§NͺlDρKΪ6πΈNΕ―Ά(ΟgpŒΓ=―FΛTΎGΏίhίαjοl=ι’ΏM›ZŠkΓ“Άwί¦Iœ«τγ>QμDXŠ¦ΪCΫvVk+βΪ΄cNεmΣ·ρΉoγƒo—ηΔοoΧ–‹κ*ΌM7ζψqή6}—ϋ6>ψΖqyNόΎ~M}¦κnβ“f\Χ6ΟTš]6[φMI›Όmzχ‰wΏ(Έ[›k½Η]oσ΅H[iㅍŸ“.yΗi„Σ$νΎpΧwLσzγΗ<Θs›ΏΕ->iξyQziΖρmώδkΓ‚7Χυ€IΪ­~M=S§1½β“&qžWKίΖ΅yή†§λξΖ·ΈϊύώΑ΄‰υΦfQΫZoώύ=]ΛΟΞ«•o;όhyΤ¦H|v ί‡«ρ+id>νKuΟϋZτNΥ%iR―6ν8Nš©ψ6,xrmq8£}vί>'«πExε[-=œmόΈά6.ε»ξ¬VxΗυhiΣ'ύT|&Ν"pΪΙψ$Έ©2Άο˜ζΆΌ©xφφ΅;ΣμmŽοgεM5φTq­όSρSam™kΕ·iΫϋ]ΝΧβθχ.ΦϋΧ›Nm₯]ήr`=iΪτύΎs`³8°ΡΆ·žτλI³YτοΟx6ΒΗυ€]Ošύ™Ÿ›U·ςq=ιΧ“f³θοxΆ&ͺΑ`Ω¦Lgκ©Eςεy΅λFΒ³VϊΥβW‹[Ζ}-n½υ\oΊΤ­τ«Ε―όϋςuWκ·Vž=ίς{wΛjqυϋ­Λέ}Ο‹ς/ 'Ζργη€k―λIΣ¦ίΚχkΥeOǏy³«ε­•O9λI3¦g«?o΄N«₯_·(<ΌΩέψΰιΧύ‡k΅‰ύ§¦{§&ŸkσΉσhm΅):ΏZnμχy§k₯ŠoΓ²Ÿ0ŸΊ KxΒV»n$-υΊ:±ϋȘsοs―­ά9Π9Π9Π9Π9Π9Π9Π9Π9Π9Π9Π9Π9Π9°s Ϋ|/ά—Χνa.{ι›Λώ™§Νεη>Aί/ƒΙž&zo•³§λΡρwl„½έo„[=mηΐ…ǁ½=&^x5½pK&»\ΌpίA/ύΒα@w€ΉpψΎ¨Τ}]uYΊθΝφπ΁΁΁ύŸ} Ψίq―aη@η@η@η@η@ηΐζq 6ίΝΓΨ1m„έΆn­L»UmW[••άΫ3Oύdš=ΓΧ½‚UΓ%>ύιOχ_© Β/zΡ‹Φc΅v΅« ?σ3?3\ϊ—ή+t*δ{ίϋ^=Nνΰƒwί+υώΒΎ0ΌϊΥ―Ύσο 7ΊΡ†[ής–›ZŸ”sήyη ―{έ놳Ξ:«ςψV·ΊΥπ Ώπ s~OšΌŸύμg‡ΧΏώυΓ7ΏωΝαzΧ»ήp›Ϋάf*y[ζ@ψφo|£ΆιΟ}ξsΓp…αNwΊΣΌ½ϊΞβρΗ?|ζ3g WΉΚαΓνoϋΪ“w+1­Ύy‘‹\dYο~χ»‡·Ώύν5쨣ŽΆmΫΆj{Z‘Ή?tt,δΐwΏϋέαΝo~σπя~tψ‘ω‘Ι~e<<δC†‹_όβU\γΧ~υ«—Όδ%+ή­(KVxŒπŽ.v±‹νƒ”w’;:v—‘―ωΘG†όΗ¬sσ“»άε.ΓψcRfοn™Sωιft΄±~φοώοΓI'Tu7:ϋu―{έ½Fӝ«…΅²4|=χάs‡ΧΌζ5Γ9ηœ3\υͺWn{ΫΫ?ϊ£?Ίeλ°Zύz\η@ηΐζr r‚œ{Λ[ήRυΰρ―ςχ'ς'»œΨ\vοly‡η»ί.~±‹Χr~φΩgWϋΨχ?ύΣ?=όφovΧΉχΘ›θH§8πoφoΓ)§œRΫΛ\›~ηi£Sω„%ώΏώλΏͺνυ—ΈΔπνo{ΈΪΥ>άξv·]8―_„―‡wl€mŸύ•³‡^ΒpΦ—Ο{½λV;φV‘q=t€ίϊΦ·ͺ=ώK_ϊ°}ϋφ?o7Π%Ώ\=Mηΐξpΰ­o}λπρΓωηŸ?άύξwΨs{ϋΫŽφΌ[Ϊ·uυΨΡΣήιLoxΓj_ψ­ίϊ­α7ΈΑnυ…ΰ5€ΧΊ;ξwχwŸμJ|ψεY?dΟ;ΰΠ©²ϋk]MšρˆG¬»z&5w»Ϋέκwλ8…μiψΔ'>1Όγοξq{ &EγΞ·Ρς“λ_ϊππ‡?ΌfκSŸ:άβ·Ψ#ΗθsΦyνk_;°ΑΗ>φ±α―ϊ―‡Λ_ώςkΦ…Qϋ‘}hΝχΨΗ>Ά4,δvXxώπUxqΔΓ­o}λΉ3ύΧ­ όο|ηsSΎ½Έ:Φ½krNτ_σšΧœ·ν—œφ xέ™fοΎ›^ΪώǁŒ ΉχΌη=ΓӞφ΄uWς*WΉJν‡G}tu”δh|λFΙοζ}ο{ίΐΙ”2ΎΞ΅kΪttl)ЁΏύοW‡ι?σ?ŸΣΖ‘Ζ‚Λή€―~υ«ΓΏόΛΏ ?σ??w–Q.ΪΞ8γŒα1yL%ƒ1‚3ΝV²τ½ο}οπωΟ‘ΜηξΊ‚<ξΏψΕΓϋίώα^χΊWΥ‘9Σtθθ8°9½–qτE/zΡπ‚Ό`ΐλ\η:ΓMozΣωsΏΩšΘ;dLΧ»ή5x—wΈΓVkΧ³žυ¬²ιθ3ΥΕNΒπžΌ+χ‡ΞMζ»ο±Η»λυ―ύu9ΣΔFzςΙ'{άγζ8ξsŸϋ[ΰoVgšy`ΏιΨ9pή7Ξ^φ²—[Υ{λ\ƒJΪύΎTΞό6·rœ»λ]οZηœi:tμ Πg>ψΑΞηλ6Υwgš½Αω^Ζ…ΕϊΎ΅EN-7Ύρ«NZΎς•― ~τ£λ#»ϋΥfΐ·Ύύ­αo|c]“΅fiΌβLΣΒΧΎφ΅Ί ΝzŠ.$θΞ4ϋΑΫ>τΠCλb½@;Pnr“›¬¨•ΑΖ€[όψΗ?^Bx˜ύΣ?ύSν&8{b‚ΝXώ¦7½iψύίύα†7ΌαΰŽMe_υuΒΞi§Vtφ”" ο―xΕr ΚU†Ÿϊ©ŸͺŠ#vίϋήwΝΕ@4βο>πΟ¦πb?Eb‘υr—»ά`—άa‡ΆŸ9Ϋ»UΆ"Ψ•σ—ω—uΧE΄}ŒCP@x‡ΞΞΝαyΙΌqgǎΓ•―|εzΚ@J°SŽ ± ϊαxxΫΫή6Ός•―žχΌη 'œpBUΙξ»ΟΘ½χώ_1½lxΠƒ4<ς‘¬Ξ4»½cθθΨ—8y`‡‹Σω,ΰrτp α‰'ž8yδ‘UοKΊΝ¬[p:ηρ|5 XŒC«Ÿ΅χγtΦ³“9ΛsΜ1uΑ*υjιαΘ»ΕwθθθN?ύτ’λΎ[Ν·νμ΅xΝψΪO± —Άή5²žαϊ…/|aέHgsW ρiΗΛ>„CύΊ78`Σ&0'_8ώžzκ©Γ/ύ/­κΤ•φkΎhγ»+'kστK]κRϋ€ΓΑήΰw/cίγΐΧΎφυJ΄΅’}τmύ°Χtpί,Ό―ΎΛ}‘nφ^ϊͺ6xζ™g‡—΅χΕχΨi^Ρ‹œ<ΓIή&kψchυόM΅]RΦ[mΦ „.σIŽΟ+v —΄ϋσ΅Kύΰνj°&˜Jx?ψΑήiΒ‘fρΠ‚–Ι7‹‡vπϋδΣf5όΰ±3†χœIΡε.wΩ=2™gT¦ˆ)wOΌRe82έ (?χs?WOΔωε_ώεϊ馡ʍΪTΑΆ'*ΉΕp†ηηŸύm‡ς~t9AΒi5—½μe«"΅ΕHώΟω?u@αH5wςZžcxΆϊ%/yI­—Žΐ<έV«L§§s`γ@d-ƒ³OΔρ’–>6ŠfXώ ϋβΏXUκ―ώͺΚvιBΆΩŸ άΗXΈιδς˜ωΛ_^ρϊόV‡ΞΞ—oνfτ™Nϊ狃v3ΚξIΘQΈ6dLP^ξ9ψwάqu|πYά­™k84²4‹V-œIΦi~DΏμςΆεNΏο8p9ηOηžϋ΅rZξuΚηSVΙ'Ϊ€ΤaksΐΒ‘1΄Ÿ)Μϋe«ςt†o’rμzβ·vν:uϋ:θ*€™-oΫΆmΓ?σ?ΧΝ•Y|_­ŽœgœΘ6Ν–μgL‡Ξύδσ_όΕ_ΤMΞ>΄―Κeύ\΅.uϋύαυ:μ#`Ηύς—Ώ\©­Άέ}„ξNfηΐF9`#G6ž© Ώt}Ά+²Ψ16 œBσΗόΗυδh:YlJΡσlŽγHc,³F{ Aw¦ΩΟή8…,ΏΆjQΤVοyΟ{VΗŸΡ1η˜π«Ώϊ«mςχι,ƒΗ}ΒΫ0α\:e+ι]§ς$^ώΔO…Y8?KοίΑ]Π™ƒ#8CίF―—]žπTΟ}ξs‡'?ωΙυeμ.ώΠ‰ΰš Σ-MΚ—/yΖχς­–6ρΑΪπ„΅e$ύΤ5ιΔ%―ϋEαβVωΌίCκΫ⧙z–―₯-iZ#__wΖΟαίTψTXη*ό~Sy¦Β’|MZαcšΫ°q>Οςϊ…–©4λIw™Λ\¦f5~§€ΠS#ϋŸΞΞύ’‘!>Ρκτ>°}ϋφαf7»Yu`όόη?_?[Αωq½2Ξ@ςL…I“ςζ ,4&mβ„s˜χkni’.e΅iΪϋ€&νTzi_o–, Oρq8Š,Mœ+C=§Π1¬—†΅Κ‡WΏΤ-yΔ₯œ„εY\‡ΞΞ ‡ι―guΦπΞwΎ³Α‘ύηιOz=ΙΔFΞ4λν³γ>>υœrαœΚΙ#~Žq\ΙοŒΛMx7V3/ψ“τ-ξ6Μ½Ÿψ6M‹.ι[:ŸΈEy₯»Β0p ekIOΧζ΅˜*k*lŒKΏΤ1y€  ΛσGήΏ9`Ž}­k]«.vΎϋέο>ϊяΞO²˜ͺΉvςƒώ žζm†M‹–¦ςSΩjXښ‡΄·„εyaζ‘΄SωΫ°E8’Ώ-k*l­ό㲂c>…g*mΒΠεΎ>/Ϋ1¦pKχ©ΟT˜ψBh€74Α‘rά§¬6Lψ$­Έ6ύ’π1ŽEιήβηυ<•.aβΫόΉ§ΏOιθΑWήRAΌ”WžΰKώ€γχ<γόν³ϋφΉ-c Χ†΅e΄8…KΪΠ΄]5νˆom9ΑηΪβZ­όδΣΡζiοΫτΉOYI'8ΪΈ6jχSy§Β¦p$έ-mΨ8―|5o3^WκηΥπόΠ:ܐ§m™‘#tλ2~Nϊ)mX›/4ηwή\ο‡'e^υͺWόZ'M%m›¦½―i9*ŽσΞοόΞο΄ΙVάg}]ςƒ\Ϋ{e―«όε:­•Ά΄ώ\ΰq°ˆι$l.Ϊ† sν ₯ϊ|Ξ―ύΪ― v7š|η(Ψq£Mώ4ώq|Βƒ;Τ'“Ή?ψΰCκў€Σ%½«²‚/y„·a‘gymTτN }λH#ApΜσο”k}>“uψα‡WC“Ο=ΩEΛ!)΄―Λtͺΰ­I•gΧEτ‹ΓSW<γϋ<'m[¦8ΌβSfΒ—R].ΟSΧΏό-_Κiγvη>υΫΘoL[ΚoiL˜kώ$>εgΐΡ‹^¬F',iΧΊ.βίjεgθΐΠβjΣΧΈIΣ―ϋ4ŠΨ)R¬VaQ~±‹]lΈΫέξ6<μa«Ÿ~²Πλ$@I©reΩQ3}§&ZN'l γtβΫ°EtI£Όΰlσ€Œ„-Β‘t‘?ιžgΧΥpˆoiIώρuœ.εJwΡ‹]t^†Εlς0εηρυη΁΁ύgœqΖπŠWΌ’VΜ)0N¦9βˆ#κ³OψδXM&%>2Δ5Π†΅8’Ε™Ζό';φSWxΧ#εMΪ©τ‘)tŽΛZ-\ZγUPΥ%eΘ·^X†Άό€γ•&εŠKž–†„σφη΁΁ ŸϊΤ§κgMQ`‘›γ` ₯τ^'ƒE}Ώχρ©ηVNTΔΝeLεiΓά/Β1Žσhq$¬-o*­°Υκ O‹#x]Ϋςά/’YΪGK‡8\cZ’ΦΈΕ‘dσΞjεΥ„£?Α²Ϊθ6,ιΪxχ΄e&k ayξΧ‹Ÿύμgλ)―τ<πώχΏΏždΡΆ‘p$νμ쯜]ͺ…ίβ·Ά‡kP–aκuꏼik-ξ„Ιό»’žEωή–•2Ϊ°€K\{—΄-΄α»‚#ωS†>;Ά“·ε%]ς%.Ο«Ρ‘τ‹ςOɍ6}ΚZ«œE΄Β«Ρ»žόp­FGΚ‘.°ΥΚNΪρUο'ΝοΜ/ž9όχy=œ½σŽ2WΊ•†ΓŠsΙaWΎrν<Ιγ“S>ρ₯/}©#|nEΛ[Ν±Κ΅kςΚ%H΄qV1ρςσyΤI:NΨΆm[5z‡ώšΉω#ΏŸς|»ν³ŸϋμπσΎQ?$―cŒM΅«@ŒNφg6ΨΡεΣ Ύ |½λ]o^—”ΡΦνκnw.^¨ΏGxόΠ“Ύ-Γ ;ΝWΆ—I§έΆ&¬ή:_γΧ¨t{—π{?ŽYεΟhζσ`/œΊγtιύ" G‹Ÿόδ'λαή#ϊχα˜yνΐω֞rΰ8ό*‡—ύΙΛ.B1Žή±c_}+3G;Nοœ£μυ\£6Zώr6ΓW|Τρη"‡^dΈάε/WΛτN.q‰K¬x'Ž:“6Gj7ŸψΔ'ͺ2`Τ–]αΕt γυΩ–vqϊwγη3cΒΌ7eΛ7ώΆ}pψ΄Ϊ‚ΤA₯Ml«ιΠ„oκσο|§φCνΕΟnΙ«μϋ–όhV­Š>E6’―·½νmλicϊ–#¦ow»ΫΩp™ΪοΞύΪΉu Τ§υy|ͺ„lΤ~υ«Χqfά'υ;rɏθΒ[σ οFOΖ9Οc<ΑΧ―{–ι{ϊι;ήρŽZ˜~{νk_»Κ_Ό―|ε+λQέNhΌΙMn² ΰr₯SšW’#δΩΛήBgUyΐζΐV n δ=7²ƒŒ£g’μ OΓAη%§ΙΕp#“•CφΙG―όΨΗ>V‹a―qΤyND έγy5ή&3γ…ί"[KhG³yώ§?ύι*gΩ"Π†fε‘™ό%έ Νp„ϊ0Ύ±γπΰ!™J£%6–δ1NΘγD|Ζ„—: δ%k§p‡χπbΗ’'ψμΨύβΎXΗ€ΨΜ-ͺ}fy š’έζή=^²Ÿΐ2~|¦Ό+εz§κ χ ><Π·Θ}œzκ©΅ύκΗiƒch?―{έλ†;vT;«yφ"mTYtνRί£σθ?ϊ_t €γ“Ÿή"Ώy#?Ω!―φ«_!ψτ‡”­κ›ϊΉδ§ΐ—τΑ“gελ3δ€~¨.x&Ω„<£Ο%OpΈ&LžΘRυ!Λδ%KΙ_xιnπβ Η€υ'ΓπR²£Υ«ιΙ)sœ΅gς-οηΗ/ωγΓφmK6kςΜ8BΎ‘WdšΙτ)HΩhΎŽoΥVϋο‡^d‰ox槞Ι|y6ζ ΙϋΓ=<οO9I/Ύ½?λ+g ŸάR~τ›WΠω½sνF}Ζ@>+ΟUΥ78α@=νNΫ0ήxwβ7ς­β“·-'aΪ’w‰Ώl/ή%ݜ~}^ Ύ±‡·Έvυ>4£υ sγ]>Ϋ>{Ghjω܎οζ%δˆψEΎ©/ώ«  ξ•Ÿ~Šx₯ΫVϊ‚wΥ–/=ίϋΠΆΌ«Γ?¬ΪούΏƒSVε·"Ώ1[π±ŽoΡ+ιϊ±·8m’<ΘΏRύ$υAcO.wωΊ¨N‹κ£ξή…~•z«_ζlδΤw ωN§0F+‹Ό4¦'³VŒ‘§‹δ_x‘>κ@¦ͺSt…+^ρŠ‡2Π›τ-Oά']pψis€Ύ‘ž~νZΉj+ΦΰΡρΌςWzaψL–+ƒ<ΥfΘ^i]₯Ac]i{?ψώ†ŸΈT‘w₯ΝΛλύ£ρΚ8‡nΌ`ίhφnπΒ;–Ξ˜ŠGž·m³n9­ΓΘΏ4V}θ>ίͺcΆχ*ί–‡ΒΌϋ(ŠRZ)/ίΎž•gΩpφ„'>aV„^ ?υωwχw5}δ³bΌž§wΌE˜•έ£³²ΐXΣΒ?ώ]χΊΧ­ΈJgͺ8ό)kvΤQGΥ΄ΕP3+JΖμjWΏϊ<ο3žρŒYι0στnŠσΗμ!yΘ¬±yΊ”υ³?ϋ³³η<η9³!kžΤ―LfE1©ιζoώfVŒΩύΡν”žςY18Χό©_}XεO©Ο=ξqŠΧU]ŸόηΧη’D͊αiV„ά άΙ‹&<@CqΌ™σVβ€q-G.Wφ±σ6<ΑρήχΎwžGΤΏξxΗ;ΞΓ‚§(W³G?ϊΡ³’hT©GπυkηΐΎΒ΄έ2‘ž=α O¨m½LfϊHόT}WΦ™€ί·yε!'oΘٌ sΥΟΕH2ϋ§ϊ§Υ¦Ιύ―ϊ―ΟΚιf;ςŸς”§Τ|δP14ΝΗμδΟ•œ,Gσ§ΨΧΤΛΨρΜg>sV>-·-Ɯϋάη>³w½λ]³2Ω¨ωεKή’˜ΟnpƒΤ|δΉzΔηJnf쑏|ς“ŸͺιΠZ£_e²QΓΛ)@5]dW}θ::φ+DNΠν~χ.Ώ[ϋώέο~χΉ^_&γU>ΡWΙ#ι@ςM1ƒnSΫgδIdLΕhTeh«;Ύφ΅―­ιθed‘k1^Τ0ς˜¬―yΝkζψθ‚€ήOη…W~εWfΕ¨QΓΗτε™ŽZήτ¦7­H+ﳞυ¬Ωo|γyš€Ν΅|ΚtφΟόΟsy Aq¦œ]ώr—―yΠοW #sΕ΄–Cή'>ζeθ+‹ 3όPNΚl―·ΊΥ­feq}fδKήb™•SάjΎG=κQunρΏχžcΦNΈπ΅œΰYηΑSφ?{黟ωΜgfε³N΅Ÿ²Ή|ύkKsRsω²ΐSÏ=φΨ:G\ς…Π<ΣO8α„ΩŽ;vκσμBoxΓͺ>ہ9o 8<Σ™πp'Ε€:+Ÿ[-zδ'g|δ#kόύοωZ^qδ<ΉL=Š…@IDATuόρΗW9VŒγ5όε/Ή€s`2f~Ψa;•Y;gμS‹l-ARχ«M)e΅Χ'>ρ‰³|δ#³Η?ώρούήοΝΘKΠΦνtό_ϊ₯_Ϊ‰Žΰ3g?逓VΨΛΨ Ÿ1 v0αΕQͺ–ε=Η.ψπ‡?|f^B‡1 ~γ7~cŽ3Έ]ΛΖ±Ωίύί―h Ι[#ζφΆ§ΫΦ駟>ϋλΏώλYlL-žίόΝί¬6s„΄+}X;0ΗcγL›p–ΖaΟ•–λΣ₯]i‡žτOtR‘ί˜·,τΜΛIyδέ ^π‚ΙΉvΚf‡<ϊθ£wΚ GqΘ™=θAšΫ;“'τλ ϋ·;+Ÿ€šΜŸς₯Ι—«yχΫήφΆΩ}ο{ίYY›ΔA6³ΥιΧ-ŽφžώΚ™~Ÿϊ»–“wgdΗ[ίϊΦ9ώ²·.τjϊ]Y<ž§ ²qtφœg?gn“M*’Uώ$qΓ{„O}ΚfΞΩӞφ΄Κ10>±#!ΈΨ§ΩeΨlΚβυN8”QgdςX_σv φΪΤ±½ήϊΦ·žπΊVΘΐΠ‡χΖΆΫζΙ=;ρΓφπYqπ¬Y€OΉδeΪ‹ρΆSηΰ€ίx2e«—F»œz)G›ΣφΙΑ™λΝo~σΩΫίώΆΩ‰'ž8s’Ώ>LόIΌ1δΞwΎsΝoύΕz hλc<}φ³Ÿ=+_D˜—rΕη7Ύρ³vŒ€+φ3ΌZd§-ζ?ΑG7 πš‡…Φ€Λ5c]YH^‘ΟCY°―Άxi΅YςμwΈΓΌ,αχ,v:φC@Η8ζ˜cζsΛ”αΚήφϋΏϋ³“ΛΈ›Wθ―™ϋŸ sΐϊ`ƚΨ{ΓΫυ Ι&mΘ»΄6Ύ·ά³5±ŸϋάΚυΈΨ€{υ«_=YlΚ)Ξ!³Ϋίώφ?Ή<ΨBχΈΗ͊3αN4Πw•oMgm_¦ >χΉΟέ GtΔδ™$ΆΑ­]“!ζςαC{%›ΨΔWΐςt₯Ου1Η“o‘ηΕ/~qMk}52Ν(υcΫ‘VΏ(LTΊ2°vOfӍ‚Ο|¦-«>”?‘E=2gΤΪρ i·β΅ŸLSήπώ9ΚJxΟMAYάΚ"_*“θzΔz•F:χ",ΤP:tβΙθΣP<ϊJγ;ŠP«ž~E(UOι’HU―7^d₯3UOE^pΌά.RΌΰ}ϋ֎S€‘­t ‘(χυΫΞvΈ€›έμfCι„wιΤΥ»(&υZ:Χ HžΛEΥο—ΑlΎόy~ίW~Yψ¬žye`™{k††Zθ:¨;oΌΫέξvC υ˜"(κυΕΡfέtα3OΙ2X EŸSPBͺG>Λ²ΐYOτ)ƐΊ££(ύυ3SΙ/ΣΑRσz7eΐŠπ«ΟŽΞ/JωP”ΗκΉΙ;Ϊ{δ­θΫΕEΉ­Ÿ2)‚±–WΧα_υ_«gdk]π·°ͺ'c™¨VoΛ’¬T―Ξ²ΰPί»ϊσΘFΐWοΉ8TΥw&μ–·Όeέ‡wEe’β²ξ₯-Ύσ”Svͺ—|‹€χ#h½ky‚ςηΚ³ohςξxσΛΓΛ“weδη;3΄[ϊθͺ“‰§λ\ϊΧ―:ύσ}ο_=A’Œ+F‹Š?ς‰~XŒΐνφ²σ%;9tI/#λ…Ρ“’«Ρ•θΚt)»«α΅Γg δ!=}€>oŒβŠqa(‹ C1tΤ0'B;Θcs ΊΉΣ"Ιmε+ŠcKMK–^_w‘Q9tZ²”^ήΚg;ιΈ~2Z>u/ ŸƒωD zy€±ΝΨR*κΨτ'ς'U74Eή§,<2—0>α‘ω ύό??ϊŸΓ‰'Xwn™/Ωi~§;έiž?εφkη@ηΐžη€’€œ cΩcη! ι~Ο{ήσj_υΩΣbΤ¬qωξDv†²7Χ Νχɜ²Ψ[?M–›Γ’‘NŠ‰ώ WdH1κΧΟ«Ϊy Ψ |†Lρΐ²˜^eα–ηςμI-ΐÞAΖ#ςPŒά5šŽœΫŠ.`Oyΰ0|ψ#©AlEτύΜ«ρΕ§^ρΘόžΤ9Wiπ¦,VeA²ΜΞDV³Qύg©οΕ/Ά΄σs,Ρμ„Ivσ|°£œΐ‘ήκ§ξΖ;PΥ‡}Ÿθφ€Ύl¬3Η`“°““άΕ‡χ½ο}s\ZΌ7ήDV ³[˜ƜΏ8Ω e{ήa1œΕx_ε?{ϋrΪ1 ογΛeηiΩπUνlΪ9―Μ~πƒ΅ζ*e±΄Ϊ}ŠcM-―Ω9@w^):‡~HΟΡ&θgtϊXϊUμ”βτ—,ϊ y3ϊ’>ΐ–EΗΔ¦YG«¦Οϊ}θΓŽ}̱՞§<‡ώšΉ΅9>["Ϊιal½τ›“K}Eэ虑—” Τ8©Όl>©Άΐ²HZνΫτΉβ(Se]Š}0ω§lΐ±Hc—yq ¨6MύP.ŽΨυ”˜Ά=φ1εd±WV:Θ2ƒάu²ώŒΊ(){¨ΊƒΠβJ&‘λϊ,ˆ|C'<ήΝ19¦κΧτΗ²ˆ6Ο_3¬γOτpτЍ9Φ œ L6G`ΛG³k;Ή2t’=μlΥ€,"—νˆχήπ£8΄W|d²ρ ,BW<μΐΕΡ¦Ύ?8²vpΡ‹\tψςY_Nxύ ΥΖΟ^Ν.lŒΓΟδ7žX―l$xεŠ6vuxϊ3ž^y₯8΅¬ΈΣ”‘Έό+uPgΠΚgΟβΘsmΠX@ώϋ‘phseΡ\Ϊ.ΉlDδ=ΪdόSŸϊΤ‘8ˆVΫϋ”~βD‰²0]Ϋ΅φ΅£Œ?Ϊq’š·‰Šd7ώ ―ΚβhνΦg€~ƒ_ΈΑπ#—ψ‘:ίaƒ·NaL,-σωŽόψcΚ{Τξ€ΥήZΘ»0*ŽΉ5ΚX»>›ΩȌΉk.xΜgcœσ3ŽYΛjϋ¨³Ω?πœ&^›6&’#Ίy…m»l‚Iύz!qΐ»!_φ— χ½ί}+δIq„ύΩ£[7%ŸΚ¦ΒΊφΓ /ο_ςŽυMsφE§Σh#Φ=AΩ Se{Ϊ±6n½/φΊάΐXD>+”wž2<σΟ¬ς\ή€Ί²ŒΑ:.ˆΪΈ°mΫφΆΪŸΠC$?Ω‡>£nl(κͺŒ²ωΎŽΑlΡdI…BΪΊ~ #Ν+ΘH}šœ€η«YeόLΩςdήΰK Κ’³θ˜δ]μdΕ£ύ¨αK_ώPΩk_œύ°π₯Ё'δA&_К±MψΫKωί(r‚œΖgσkΛ‘γŠϊθӁΫάζ6σρ”ŽΏŸ9OΩ ?Χ5ς^δ#§όΰΟη+τλγ‡}ιΛ:³uEγ­χG~dŒ Γιώ€μ±Θv–υεh“$L;€Ϋ{‡Ϊ99oiiJ=Ε“KΰϊΧΏ^₯Λ½xΆ&σ΄yα‘ΛΈG–‚Γ;ΌŽ7N΄Ρ_’[‘Υ8f=3ς΄fhώΠ;πyδ‘΅ΌΠΦ$Ϋz·…Θϋ(JΗ”α;ΫV< KλšρpΆγΠ.”φΚ{ΈΎκŝSVJ#―^lSΥ/†ΟŠ―tβYQώgeTwŠ%ξ²)Šκ¬(u'"g'΄ΨJg,^›Ÿ«'πXD―c;ŸJG©žƒ‘|ro48αF{Υ£›wΊ€πΨέT„p-§Όz’M$κξΜ™kόŸΉ Ώ2y­σj”ίiλ…ΠΘ;:<³Λˆ—(Šτ¬«™έLθS _ς γE¨μφdšΔ%Ζρ2tjoj» ΌΏbT©’E¨Ξ=‹Α{Ζ«5εΈΪ= Ώόx^„zυ>δhΗΐϋ"Τ*­ή«6c7―qεΩ lGn”fe;χŠ΅‹€§‘WΕQγΚ½κΛ ‚‡.Gο)Ζ”ͺΗ:…Ό!CΙ5Ί™š“Rθ—d*]^^ϊΔΫρcξ"tPzΈx?;ydgv‹ΣN²0τ‘k9ΝΛ[ezdY©xιίN§Q.έZΉŒvz©έI~tνoλΫ5?ώIoχΩ―-ος, 5^ΘυΘl²6αβΜΟ²σ[ύΝi’ηGv^Ω==9ό“ΰ³iπ'σsOσŒ-ζ€Ε0UΗ8ε8α4§‹†OKΨϊί΁΁=Ɂτ7§Πϊ­ΣBτQψφD.;QǐtٍhžΜΆcLnΠυŠρ½žCouκWN¦uRr¬,xUέυšΧΊζΜH§DG6‡ύˆΝlQΡα2Ά”žM.ΣaՁ uzŒθμΐܘaΞl~MGgk2^gθ!―θΚ²Λ>r3γτΕα§β(μz²$ΒAξg–ω‰5hQvŒV·wbΘγOYHΆαΖegΜaΠj,dƒωξw–μ#ΚRοbԟŸlΰc˜1 όAkN>c/ cŒz²αsYΨ¨οΠ`'rqτ―§ΩΖφ–φχ‡N68Ψ‹\혎N―<§Š8ι‡}ΠΟ\„mOhκΧύ‹ϊ{Yx©ν‚­IzόςiMμN±Ϋ₯δΚͺ-Ρ΄sνΕάV˜~@Wι“tF}‰>₯½“MτrD_ θœe°β iςA_‡_9N"ΤώεχӏŠ#M•-e½κ•Βt%yΙ :‘~ΨκsβιX™Η«ύϊΊ>H&KΝρρF?TŽ{ςUά"[t±9^φr—­§ΐK¦ψ‘}NB'™Bžͺ Z[ZΨvν.Η«βΌRuγVFzxΈ}yαğij8q-π‡š υ>ΘbrƒŒeGv"€wJΦ’―Χ.τΪ1―ΰΩΓΦIw/ r yο tqN·`£U>μ,ςΗͺΩuŠ”-Ζ»²@YOBbγ‰m&εγ{HYΌe8…,σ 8”ε*Ÿ]Β#ο‘,šΦΣdΌ c Z3Φh_ΖGς4vv+ξΙlψυ γž‘ΫΪ‹±Άl «φ~§χhμD±g·xΰCβ½3|PŽ~„/6mc‚8s‰βUyΨφσ1Όρ½Φ8π> β=NKžAΚΟΏΩΉ₯ƒ/s.i£Χ)h5Ά™‹xWδ ]=Ξ|K~φωΨγεχήΜ'Ε₯Ν:MΓz‚1—Μ8½ŒuhΙόŽn£\φ3όφ;³θFΦC¬™yom»N=”ΧacΠw΅]ο§8FΦΜ‘―kaJ:λ.ς{/N‹1'Λϊ$ΫfττeSbw†C_γχκ9'q‘±­LΡΏANΡV_XN8! Ι$?²XŸΝΪ›υ―±\D“z ±8=Τy7Ϋ+Yo,"§Ζ4UB›?‰Χ?b―!#Μ¬©κ3βΘXνY}ΩbΨ¨½:₯/] ΧΈ"-ΉC>“hΚi†δYόy/Φ₯ΤΗψιGΩj0Ο@Y‘ΞδtήKlΨιΓΚA‡~ηKμξ‘_ΡBXWΔw屏θΖks Ί†ώϊδBμϊΪƒ~RοΨά‹¬ρžΩVœΞFUΉΌo<7N£ΓΨ9υ_E’«8’ΤwKΎ“ŸςG§‰ώ$―5ΏΎγ―pσ(8[HΡ;=}~³ Ι―D'οΙ^όGŽ›Λ….οΐαΤ@ρp“• eΦ‡ς‡ΜΞ)€ΕYͺςv*]o₯+o£ϋ()U:%eΏΣξX~wZρ+ήΘUHλ„P Ίΰ >|ζˆ"0tΤ) gB“0π‚ΰΡYsψ]ξrηŠ€t:YŸΒ‘x$―ψ€Αδ7ΏEνˆ”*Žΐd! §ς9ΪD`%―«…Λ(΄Žή+cmΪφ>΄SΞ4θ7±P6: 4„SΞ4Αk$”8―h xS€^5Š‡#PήX½Fˆ™hαΟG>²t€d”Ό[@)₯l†ŒI‰« ΚομΈΦΪ“IΚgŠ?†Sί}jU|՝bVιy„ϋ™L΄Jm‹‡ςbΰ’@S*#œ₯qΏš3ψΛΞδω€ΌΕ½θm1Μy_1ξKοY νο8ε‹ΠΤΆ–tŽΡC+πq©<Ÿr¦NiΟ€˜"$]ΪR‹ƒaΝ’rt|εΝϋŒ3 e’φυε<€Γ$ ο1xΪ΄ύΎs`+s m–BšqΕ>“«Δ/ͺCβc`Ρ'žσ7Ο™χ?†aq5Ωj!ωΙKr˜l+» ζΞrmZχΖ1GJ’πΆ‹ΆdfœiΘλ²›gΕ'€‚KŸ7‘4Q€ƒaΎ•1"(ƒΑέ€a mzΨaWŠ0έΑ$)`²`’?Ύ.’hiiN)Μ”y ΑόύΪ9Π9°s ²‘Qƒό,;–λB\tρΘ, œ˜Λ°ϊΩ± Α%zwθH@ɞ1yΓΤ­ΚΔ*‡„3ψŒ!G ΛΫΚe†…8‘XΈaD©_ΡΙ/ω—¬Ÿ7~s8ύž[7i\”Β‡²σ΅ΤΪxcγ<ŒΥche6=·-‡1ŒAɘ`^θL~qeˆ7Ύd•Φ{ ,’2B΅Ίtπ‹}š…qΙά“^ Β§€λΧ΁΁=ǁτ7cΞδΫMΒs%ΟΘ‹Σδ Δ' »…8v σKςd ,0I“Eδ8Σ]δΒKΚB`π·ΈΠcρ‹ρZΪϊYͺεΝL‘QtμΨ4θ»S@g΅ΐYτκΧΌz*Yu44Ÿ­%ΖθΤ?ŸG‡ΓΈΓ1q τ\φ"‹ κoΑ#㜴δ(:όZcυ‹­ΖBŽˆYdH:vΉ8ΛLΩ Vs¦1wQΎ1€ κτe]=Έs5^gπί‚³η€EYŸA_Ωω:{θC:w>JΧsΟύZuΦa‡ΐ7³@ΖόϊΠμwhi²Έ•O_²9E―JίΒσήΘŸb³ΕώšE ΦyCzϊ˜EςŠ •ΣΞ˜χfQP»oηξd "œέn ΠΑΦ(ΝΨ 1gΜχΩX§ NβςΗ)-ι’ˊ#3A«“²Ή‚ΘΜΨϋθjάZ}/ψθb±Χ[t΄™6Ί_ίΚΘ©τ ­dqu¦)οŒL!γ,ΌΑ»ee[ΗŸQ ΐaΡRΈ_ΪWβΫ«vHFIG/¨‡ͺΒΙWςl ^ωΚγηε΄ο‡m? ―μ΄S νΖaΖ§ 2žJΛN”Έ)g}$cg>Y4.Γά… Χώ½χΆΖλAdυœΜUZή‰±RVlΕy_mϊφ>ρ«9ΣθœΆΰΧ²>βqoΎΒξd!V›ΐ›€5‰,°rX³` R~‘-ΚίΖγ_+Œ?Ζ²E}Τ"²O‘”“δͺ^Σ{™_Ζ™?ͩⴚr\Ρq,²`ξ6~]„sΗ)z΄iϋύϊ8°«Ξ4αΉ6֎ετŒ…Ϋܜ6mΝOlΔΪ ΅Α„·iΨV}RLϊ_Ϊ]2cŸuύrJGbPΎGτκ”₯ ·ΙScbVβ'ώ$žƒ#\μ$Φγτ‘1 1N{FΧΧΝύ…‘5›ξΗωρžγ$›„΄pι#›=…³‘uqFM|qΊ‘–Σž:€­3Νε’—όζ2τsφ™VoWŸlβ·ΦΊ¨ΘCž“#h‰})ε)[Έy˜qTύΗ`ά c€σkηX­œ‹ΥζouŒΝβρ4Žxiό‘+ΧVΟhσs¦Ι:xλLS‘”?κ‰Vmwjλ‘ͺηοόωx“ͺOψ"g{~~+Ϋ€!‡Λ„‘΄κ§JΏZ’ƒU/“ύ:&t.+"gΰ#―|:ΓQcδ„cBW ΚD£~RΞ½ς}ΒΙ±νŽe,;~κ52 ²JZG?θAς‚ήP”ZΑ›© έ#χδγ˜o›RXG9Π9°e9P&πυˆY}ίρέt[2!ϊ2}Μ°ΕX^9–>ΩQœ6ζσ†²ΘRυrp,׊Ρs(“ͺC•ΕίyπE'kυ³δOyγ«ψbœͺz³Έb΄¨ΗŽ»‡'υ0>ΠΡΑ­σΦυ3Ÿυ‘όρω£rΪY=ώ—ά.σr0u€CCxεC1ΆΥγΠα@ΛОψ੉GΚΒK=ςήΈƒ?Εα{ΕΈ"―_Ω}<άσθ{η~νάω秦tΨb¬z¨w:‚C}Λ‚{ύŒ(šιω`5ϊFδφΗ΁΁έδ@•OΕ.K/Ν|Υ\WΈωr`[™λ—…§Ϊ?}– >[XqΕΈ\νeaΆ&IŸ bcπi Gͺ‘,ϊ 'Ÿ|r f?1 ₯Oη²Ά§rjA—Ρ2<8͏ΛBq=Žœ¬δ”ό‘…έΝΖ 6›ίΈεoΤt‰χΰήζŽmg3δ8ZΡ˜O³YI/_[wGΟϋHYx¨z}ΝΠόq<~ΩSς7Η7¦΄Ίpψ OY©c@Y̟ΛΞΠ<!.ρM‘+nΕg‚VκPœeVΤ#υ)NJΓ=οyΟZwσcGθ+£ΔRϋ)ψΨ”nυ›KGßό—Ύτ₯κ€Ζnc{‡“7ψΕT₯>Uˆͺ,² G΄²@T™C·σ‰sΕC^ϊόζ˜kea½ΪΈ΅36Τ²Y°&Iυ =š’3μv€ξ–~Y!œ-S[nA:t”ΕταίίϋουΣCζԁθDΚdo!t°½d‘Ο~Δ‹+‹λυΧΚ/qωIvŸΘιϋx羜8PΓΙ ς’^ΩφCχt±²Π_ϋβ·ΏύŸY€§lZσž­ 6…πIξΙαβψ0μ^κΣυ=šΧ —*Ÿž SŠCύδDκκͺrτS œβΐPοΕγUqͺŸ0"'π0γšw‹_@ΪΌ»βΈ³Σ»n~σ[ΤΟ€ω”OΩ$UΓό‰Μv―έβ} θ λ}bΙ»gWf«]/X_Χ;3–‚Τ/΄η’Ή]ˆ,~-­ώE^Γ‘oK[ήk½3ŸlΟώk>Ά™ ίψTWq¨οRύS‡Π’<Ÿζ’°Ws ­5‰#—?₯Eχψΰ—ζ„ioζsΕi ζ±dξ`?G_1–• j”ς«r¬Gu£jΩ>Γ‚€%GπΣ:”O§Θ—Ÿ΄ξΡ’½™’s-ˆ·’κ³Ε‘¦~ͺFυθpαq€ )ŠλΪU9}¨β}₯¦ \zΧ―ύZν΄Έ#Μί™D6οΧx’Χ½φθsbμΜΦΰ"›τΏθ¨tΠβh7_Ο” G>c;  GkkγφsFω̝~Ίθοum¬δ§­ˆš?βќqΕΪ\ϊHΛ2ώΝ~,ž,O}Ύπω/ΤΟE“)βτ°"‘IΎγοP?sJηυω3}l lAτjkV x"Γυλτ{Άoρ-ΠΥΝΜo¦Φš₯5Φx'Ζ<5.π³8ϊTzΥη&7]βϋn•©ΎζμOΐΈ% ž*Ώ8(ΥpmΕη’Τ?c‰tp'ό*χρ¬8ΧτωΫ•ηά‡I3ΎzοδŽ6Cw‰NΊΌλΠWœbjvγNτ@τ™€‚θžΚ/}βπΰίAM·»trFοrΪMύ–i9ρ€*Μ―, N•ny‹§x¦δΰκ"Ό­ξ·mΫV£²³¬.tμ1ψ€ …@[hρΈΐGιφ~Ώ°φέξυ ΄iπ4ιΞ”αjς‚&Α& οΜ½4ڎ‰eDš‡4» h; #€M†)νκΔΡ‡b€–Τ L,Lβ…y οΐ={_‘LXMžυ­@- εΘ} ŸΊ2ΐ™β‘9ΎoK<θώɟΌl5”1u:ϋ+gΧ S‹—2W<ύη N‹G:ο0ύŽ‘΅­K‹§ίw(hΗ1ύ#r@ύukώέθ†Υ8%ŒlΣυ-ς,‹ BΠΚ™°όGί4AΠ‡-\ŸΛš6I󒄦 "O(νUί“eό‡ƒœ‹Α’ 8ϊ裫άKΎδuΥη-2Δ1V}8Ϊ–γFηe¬ hτ°¨n£d+p)7ΏqΊώά9Π9°p ς±™³ΈY1ΎX¨‘EΉΏρ―ήΈ:’₯ŒN œυƒ9θ*ίΘ:@.2<€V· Nϊ9F†›;„–ΔΛ—ϋ\…­Κ4A]ΝΌ€u β]vUKΫ…—mE/;ΌͺIζ‚±nΜΘΐ°–…™V/ ΉΚο>?Ο.P?R―#θέΐ;hZœψŽψε#†›ήδ¦ΥYΫΒ•1E}/ Vσζvά z½z1€+#€O@-l²„/γ1 μn­ς_™έsΩΉŒ¬Ε›{΄΄²y9i½$œ οoϋΫΥΕX ΰΟIoΓΦdΡβ_ψQχΜ–thγθυ«--`Žρΐϋ›ϋq=„u808pΕ+\±κl6yXT’γΨTRΫRι&μΙt'ύΛ&Κ9\Π…ζAnΨΛΜoι_ΩθyΪ—6˜φ›6]v΅Χ~M‘Ρ…τk2ƒ-γŽ;桐iN.₯κ·-(G“ΐUN-›RTu[y…Ž!lάc@=Œ¬tTΠΚrBŸΓ›Θe«ŸότΕ8Η‘mκ Δ₯ξ5 όA“…ΔrκmΥ©ξŠ'YHcΏζ1ΕKε–Σ«ΧϊΕaο |hqvx’Ε^xSΏδΫΆm[m/Ϊ Ϊ”Cv³q―ΗV›5 ²ZyxPΏ«ΎΔ+‹·ε3₯U7χžρ‰γΠ>2xίΩ]Ψ’½³rΒ@΅ύΰ­v#.kε€Ωζόσ€7Ϊ€1Λ8½h¬Α―Π֎mΩpgΪsΪ„ΉZ&lμυ^7 ΰΆ8žωNΦΪw, ZΩΤΌ[`m©΅Α »Vα‡[kVφoοn{»ί.ρ#—UΑψn¬ϊRΦ6ΰβΤ³uΊΒε·p基r΅Ωq"ΠΗ“ζόεφ£^υ ύέZšΝλεΊκμe­EίPΎ6ΘΌ*χ‚€_/€/ΠUΛ‰cՁPXlΎm[ΥΏΘαϜρ™ΉνΧΊ•ƒJW­ΰύο(ϊ³ ‰ε¬Ί6¨-jp™GΖ²K΄λ„μdΌrΘqύzJώ*ΘxG璏’θŠDΌ1‹žGξ5₯[K;΄βZ³FI·ŽΎeœΪ^œ±mžρς§}’§cΰΈ?<,Ν=~ͺτ;kΧεt­κμΟy§]{$‹ιΔlFζ%α'ΪσŽ”»­ŒhŽ3G[7ςΨΉuνΘφΆ.’QΖΐξ9€Όπ‚]Š“‰yΣψύԌεw~δ‘7^τ’ΧυZrHS―lTυŽ3f†ς‡·t λτ›b»"ζŠe,Ρξ8mY+Πζ΅›Όsσ¨ΨΦΜέ΄ΑΔ•·ŽΌ α|A|ꐫLδρ‘Ν¬œ~3”OFUY™wΙ‡ φAλαc}k§‚·X@w¦Ωb/dWΙ!H($&ι:Θjΐλ‘AΈiYi(εσ UΉNΗ‘Ξο}χ{ΓΩ_=»:˘ψ˜άkτXΚgz†ςyˆ: πt“w#―?l-%-~ -'5ΒqQω²$΄ Ώ6’τ gxα!_Žυ«J/ΟΚr\fΨ‹Κ2`šΌυŠς'}κ$.χx”Α₯΅;W<₯‰Βε.»΄cbŒg)Υψ QnSΖ8 α₯bœFΪ©0αΚ68ψδ κ&+΄JσΕ G©0Ω‘dfΠ’³ <0x=όα―ύƒ²στg<½Nί–'Μ « νΪΊvο' ΎwΎσ+ήY‹cwξ)ΐ –vŸ₯k•ϊ―χ α|ΝD,aγkruχλΠ9p s€β`΄iϋߏώθΥΙΏρŠœΛŠvL)ƒ™Θ_ΑiRγ—ρ€Ό‰3M˜ˆd|KXπδYAΆF†Wδ+ D›Μ€E8Π"# <©ΫZ2aΎZXΣ9Π9Π9P8@o2‰fπ ‘θ ­œ‰ΞΈ½gθŠε¨γαήχΎw]Ι€=ς SΙ½Υδύ5:¬τkΙ2iΒ²­€ρA‚ΡΑI”vc1ς ~ hc}2FWιΤϋ›ίψζπΥsΎZυbz₯Ζϋδ‹³LžΗWο4―yά²qgώάo::φ(8C0ς3j²9pN!Ι†Θ.Οζ»ίτuΖ\Ž7‘oΎbT6—6wŸ‚Θdrήό“ Fmv @ž¦όδkΣ’QΡo§β&C;HXd*ϋƒqΕ‚œ… φ’Δ·ε™3[p·(f!.»ύιζ‘£Ζ0r-Όkσ»§kΛΟ8<.CςΠ;8―,œ3aat· ‡fΊhMΩγ²6ϊlQβsΛcΐa‡>_πΣΌή!ΎΩd‘vαέ·` ΘBažϋΨςά―Ά΄ƒœCw*ŸrͺςH<) $cοLή1ΗθE9ΙΒΪιΜkΩ‘Ν±΅eWscν˜|’cZ`2―ηQ>ΣPOp’Ξ΅―uνj/DOΪ}Ϋο―w½λWϋ"ϊšΙENΦδ&G4Ά6^6Υ6―:yfKζέ±9Ί§’φίρŽwΜϊZYΐ^>ΚCxȞΑρ™<‰==iΙΐΉ%2²-'ι\…GF:-ΔΈπ”UόI/Ω@ΒΪ{ν%²ί"\dqθjm΅x•5 z4Ύ±eΰ[N /y•sδ͏¬'Œη΄'οLύ-tZ<φ£πKήΠΙNΓl˜γg›4=sڐίϋ—.6ΨΆlε―ͺνgyξ1N–ΜoΪ΅ z~μPΪmωάμ8λŠηθτpl&„_uΎSƍ΄gο©ΞwΚόη‹ε O΄#ό’FW2Βkύ’Σ™vIo9ν§ΥM»υt΄α Ί@«}όάΟύ|u†I>νEί1>igζ“Ζ³z*OΙΧ‚1™LΘX…>‹ςν:Ι· >γzΒ‚ByκJp’ƒy.‡Uύ’|€:sJΥχl4&―Θe…G--ύ~οsΐ{ˆ<ωςY_Ξ=η‚΅mΑ_―”'9>G ο•i<γΜ–φHNs΄ν†vΟτnzŸρ‰#χWΎrV‘ΣK§G‰oAΉδ/ωDŽόMšΠEv€μΔ­χͺ/d\!CΘ@0…Oίiυ?ε«Ol5‹μ ι7ψ;ƒΉωΠBœc¦ΖΆ€CCϊξ”Η>OΦ%]ςΆWx@+O ‘§ΖΨl€ššC¨SΥMΎpf•ύΖ!uB»v”y!γ~[~ξΙυΘvay§‰ίΘ5οΜ;δLDΧ!GΝƒ2‚Ο|'σ»œJ£άϊžβ=Ώ‘‚KΪΌcυ·)’|jͺ–AW#KCϋΉ ψ0Χ)»nρ?έ™f‹Ώ ’η8§υE1Εg#8ΓΨE_ΎXΏγΡ›ΖλJx˜πΨ*-Οξ(ζ:‘Ή+©ΣΔ²O[—ΠΨ¦m…N>ΏΏ`Μ›mΖ ZD£]\„“Ι˜I‚#Τ ŠΦQ$ʟ°8™hE¨F°LΡ–ΑΚ€@X·@¨ƒC>΅MλΏV+K“‡΅HΧ~Œ Ω7όΓ†wžςΞ:π·ιά[h0Ρΐ³–3Ζ5~N1(›δΐ jε;–Γ―οψυqςω3:(> eΐtPŠy’r£―x›hQŠπ> θ©ΛTY™ˆŠ£όŒΣzNۚΚίΓ::–8 ολ/ν8—>(ΕAε_ϊRϊέΈΏ‘λϊ!ΘdΉ>,ψsPρζŽ\§pgLl“Sώ΅SςC418‘%œ6cBκ7–΅yVƒŸ #ά °eU «&ꑝ$Θr‘.e@οcΌφ[ Š€Ι¨<Η@γ>‚‹dPd˜΄‘sξwΘ™xp©Ο―όΚ‰cζHŒYl¦cf—0ƒ@θΔ‘‘Œ7ŸbT{Λ[ή²β8ρΠeQ†žaΩάHΩ›ΚΞψfΜX‹'βΓc‹-™g΄΄7ΦΒΣ¦οχ{‡‘9τΉΩn1ι₯/}iύ-’‚Ό²θCΆYˆΡs,?Φ²»˜ΗZ;Σ΄sηEΊmh#[Φ²I»š½":1=ψΈrϊΔF ‹Μ±ΧΘ™ΈyBζcΩθ™ύ†s£Άρ` Ζ‹6Μ Ξ]vnͺcK‘g5Ππ]bΟjΗV‹‹γz†³ΗΈΰœ`aά‚‘p‹ΜΪέΙ’: οeΑzάξ΄γΘmτoφoΧΝT6ΗΦ†ΚŽMσω5zΉζΤ?`‘Οj§Ύ£—ξ„ΏK_ϊRu#₯y:[:`w΄SXDβ ΑFl#+{o[§Τ?,&‘ –Ζΐζ؜x²4ςTŽ"ρΉ’Υ"’Ύ ’“ΊΧ·£WΛoCθF v†υζIύ-δ­&m‚…;²Η=_<σ‹ΓΙ'\ωf-Γbτπ-ukγπcΫΆmυσZtmc"° ˜ΣL8Α8‚ƒ”“Q’ΟΛ«|'i/Ϊ GCόu²RN¦Όκφ«ΦΟίyχNqΣδέxλΡ[ΌττΨ‘΄‰φτΉ©²`JΏŸJΏή0<6πOlΎ6ƒjΞGζ;Ϊe κ₯?⟢ΐI…=ί‹zq:Λ©47-Ÿ^qG@{1ίac£»X X/ EΏŠJ>ύ'zOϋ^¦Ρ₯<ωΟŽ+Ί†6aγˆ°ˆMjδGƒ΄§”Ρ―{ŸUž”΅ϊΨλ^χΊͺ“Mι^ν²NΣR™χO–άχΎχ­οΕ‘Ο§κβˆβdΏ΄θΫY―ŽJζ8Υf#Ύžς'|H}¬7/“Z\‘AδE»‘UšΘύ΅hhροC—ώ<–1γ΄γgεg€Ο}ξsΗI>“}™;ΰitcΑZž•n=ρigδ“ΉΆhNg/mš£Π~΅γZYΧ†oδžlvΚ‘“ Ωυ|z3|  ’εt%τ}I.vgš΄„} ­‘‘kKΆΖιg‘Ο1N9ΆΦQ'¬DΘξwΏϋUoaωΣ 8?ψmίΎ½ΕρΘΣ-ωΪ²ΦΊ}‘i΅τI»Zš #]θ7`<τ‘­ΗΔ1‚8ξΣ„Π‘‰NKŸ|©Ž΅ €tΙ»VžEρλΙΏšΖψ΅ƒŽSπ `"J8S$ρκg―σ³Γ‘9΄zmθόfΊΥΟξίΐ΄Ο`φΌη=―ξ½`ώMƒ>oαΙβ$ΓΎΉp>{’Η0ΞΰΩ0Ζ‘gzιΤBk+Γ£s&ΟΤu=iV“M‘ΑŒΩμ4οΠ6•GZqΉζΤΰΤ}SujιΖ<ƒς,μZl3~Œδ~Ζ;؍•iŸωΜgΦρ iwχŠ†Τ;|Y gψ ΟjσžΥpτΈΞpΐΌΪ"2ηΞ3»Ι'Hp°ΆhGκδYλͺ:M†CΛZύZί€—q*ž΅kΞμ{ΨΓꂧΕ&2Πβ'ωΙ€³ »&‘όΚ•ŸmΠ睜JΒ†Θ!ΖυMozs΅žqΖgλ)&l†π¨ΏΌϊ‡μΗ=φqΓ+_uΑͺE&ύŸ<ΨΆlsΔ'uιΏξΙ₯ΜωΙ%τL<ʌμrίBžΙΪG>ς‘Uχ†«-+ι₯Œ–“¦&ΟΤuΝτ…Dε’ήϋb§A§ Ώ€£El΅α›vdξa-c ΤϘψΈΗ=κδœG½sλœ«,|rή°ωΨi”ΟyΞsV,ταϋm~η6υSˆNVΡV8zΈZD=ύ3§ΧωΫσӞφ΄αψΧtĜ’s£aψ”ρʘ—w»OΪΜfΚw4hKγωŽώΝq¦λΜ{θ'Nφ1η[Ω2—¨6&ήόζ7―2ΒΖek Nβη/aœ‡Τ§­  G}t§-’ςω»υιπ°ε[‹· w>s” c<ਛ%΄? Θν8(sΨp"–ΆΗYOή΄ο1ήώΌη8Ύλη>3€mc‡'m³ύl/k??ύ3Υα‹Ή¬­šΣs¦βh§Οk—œKrςύΫŒί9§c‰u¦EνΝΪ©kΪκ=cάSi…i筞Ύ(]Β•…>tnαΪŽΝLΫƒοΎ<’Mom\ΚT_οΏσMλΐΪ¦Ιύ’k‹{£<γ .λ°6zΡo8!ήεwο2ψό¦q‹|œύvuL—λ9e“ωœΡΨτ8²—9Υ‰=qˆτωFλ­mΎϊ°Εtgš-ώ‚6›Ό4jxγ-IhR ΞNβpε!—"jpΠΐyδK<9Ž+Βb#τΖQ9ρά[”ί€P6 RvΚ_”ώΒgαωμθGπšΧΌfΨ±cGε[<\ ε5γqoo'8‹κεΈΠŠhώΩΩΠΆ¨ΈzW“N 3―~ 'at{ηyhς-Β»ΡpψΰηΐδHG'Ρ€'=ρI• 9IΣβ–‡²cw γ‘ ΕΫ'V(χ+οΜ0$ΖΓ΄Ε³+χνϋ6hΡ/@_„-·£–“ϋ©΄ΙΣ―ΣˆLpr‚ ₯‡΄]t“°`n`ςρ—ZΪωe§1u5χE'АΖΈί#+ΔG†«c 8眯VΕ>F·OκN¦εˆG8‚›΄ Hλ7&ί-γt‡ΞΞΞ–8w ¦t+²(²§Mλž,a|Άθ³D ‘ΫΆm«Ι'Ω9u¬JΜ՜μ4oˆξΆ2Εƞ”ƒf3ίuf$W/rΣ‰;1–Y,Ήξυ»ςχ½ο}ΥhOΏdΤ`€Ο ‘1uI=7ƒξB¦ΛΌcZ Τ5c‹:g°φj ˆ‘άά¦ζΙγ“6&cGΖ€Π\ύz`s@[#Cθ<ΐ\[_`_ζΔ8’DΖΤ€ΡŸ΄»V—!CœϊΒfάί’=ωςœk«_‘I~6–rz³POδτσŠWΌ’:M<ιIO,–ϋ$œΑ«―κΣ~hΡg8ʐ>AE‡΅Pκδrύ^§Ÿ:…#ΕwNγ³·oί>?Ω6τ‘]δΚ8₯ΗώN.Iy—΄‘UΚ‹Νž.Ϊ•‘ΣΆΨ@8-ψ-β%ΌΙ›2…΅ρσϋe‘ΈšmCZc” N€}3‹»ςq4dkΖ:j°gŽ yΎρΜ`kCϋξΨJΘ`?§Π8•ΔI3ϋΨρWvνΏψΕ/žπ„'Μmχp’΅Δό,j‹ΪYο” vπG<βu=Εά¬Ζۚ`7ώh‹ΪPχEφ£ρΝo}³ήFΆ'|w―ϊ΄6 pJ Gθϋx…FοΘ{ 2ΦΥ°¦ΰ΄3λ>‘ΔαŽάΰp ―ώΗϋ£ΦΟζeύͺν@ί0ΞΉΧNΠ±Ρ±?4΄Έ§ξΫtτ?'?pζ!GΞ('N9…Ž,ΙIVμŒκζΒέΓ6Ÿ‘‹δ‰Οqq€aσύƒ?ψƒ*O89Χ~ΘМLΣΎ3χΑg|Ѝ>Iίτώ³I›ν€œj!'tp*}κSŸZΗΑEνTΎΆlΟ)Ϋύf€ρ5γŠϊͺχjz’ρ’|–O€§Λ»δ$ωΖγΧZyχt<™”υZN'dΊΣS½Ÿρ» }™?c§NKK:Wc_NΖΚΈΧΖoτ>νS;ΣώΨΧ8τqζLγP㝸#Žψ•Š~3Η(Έ΄ψΣVΙBΞ4ΚΝiLβm΄Ϋhχfϊ₯•α½Yb/λBη@:H ‚#‚’s1‘HΉ7°P2£€¦SΒCa£ΰ]o j“zBΓIθCp2ρπδ¨6qmžElŒs³ŸΓŠ¨c@yύ„ *Ώ H©‹ς U“§€Œλ<t3`ςEΉl‘?Žιzλ[ήZΚ1^Š~DyΞ.[u’\ͺKϋw΅:x…&³Ι‹vνΘ²?ΌοΦ:iΪ2Βc’›˜ioaJ―ΑΜ€:MPμVΙίβΜ}κšη΅°υM“Ή…ΈZ*W›H›I;[«Œί9Π9pZ™ΐ[šςO3&΅τ r,Ύ3 ΈΞΟ,MθMŠζυΡ^›τkr„" μD‹‘ΫsX€ΓMΒΔŒŸLΰ"C"»μΈ"oΑαd7cœ±‰‘‚ΌδKξΙΦEB&Cg-λ6r²"θ::HD{G9ι2Ζ{ d ŠΞμ:ώ‰ηœΞαƒ\tό} β&ΪΡιS²&at(‹5v3†GWڌ—Α@ξΘcYΟaήξJpϋΫί~Έ—=+—œΞΞ - Λ΅ΝŒ™IOήησ’ψΆHfK; +Χoηυ¦ΫfήΑ(eLβ(^β`ΙωΗ’XβcQB;ttlYθ§τ²“N>©Κ›ZΨrθsS²W9DΖΪ‰ONισ'tR5ˆ ηΔ±MAδlμ‡œ-X°`Ηδ Γ²<­œΛ=Ο€ VΦρ'ι[ݚΝΚψ€Κ’O«™žμ,Œ)=Y LΒsoψ”SN©Φ„%\c!0ΆΩΉξdg²•σdτlρlYψFSpˆΫ`Ι&Ό·p α™_ͺβ½ϋŒ»CCΟ{`sΐ²ω΅Ύ₯Ÿ±:±°Ή‘3ΡαqŠŽha°ΕΡΑγ>ν™% η‹YYΰΣΙφΏΨΔα‘“‘•œ[ώμΟώ¬žrmžM‡²Hͺ/£.sx})8ΙPshNBœ ,\q΄ η:ρ$:•«OIΩΨjξNWur‡Όd1y>˜ŸΗ1QXϊ+»jœœˆ’> ρυaω9kq—3CδWβc7` sG‘)^ Γ οŽ9ΖU3―ς'vQvύ)»Dhη„Ϋ†ω@h€ΧΫψθύϋd]ϊ†7Όa΅wŒωvζ—Ξœ;„/ΰ&ί9)΅λή{υŽ;ͺνΨ)λδεΆ²Θ>™LΖ£C»‰}^ςQϋvJ‘M₯ρQΗ tq.Ιϋ”vΣaY%ΧF9°m,γp[χή£>xΪ'–ϊNΪοfΠoζ;ϊ•Άν€cΘΤ|GN?M_*ΟΔα«w£Ν8ύI[ΌΏ’λd©ρ‰ΖT|Πχ½'}ΐ5υΣ?΄mNkωU3ρG~υΠ3_…ƒ¬’οS ύέ@IDAT؜ϋ‡<€:ΤA₯­HΉ³ΡςΦ §G―“ΖΞ4δ0yοσϋΨOΙαΌm¨Υ™ήEί$—ΐ‰'XεŒ1Ζ»vβ gΠζ%“3žI«/.QεΥ§;dY¦=o&!ιOδ\ϊNKsξΝ1|ϊΘIKδ²ώΆ­ΘKz+@#Σ(Ώ0ωmΈϊ^l%Π£3£ ?ΐ” f<Τ¦Nϋδi•o?ψΑ¦Z6ωŒΥƞ|΅$|„3χδˆ“Χœ–ΔitΜ;iwθ5ζw€ΌS?όπ₯“ajΐ&15–ƒSO=΅κ™G*;ŽΦ›\μG·΄Z²Η‹ιμ-θˆιŒΉ_uJΚ―ι ά€πΛ@AIΦ°)™ΜΓE„ƒ1ΕqMۊΠLψFκΙ€οεςŠƒd’gˆΕ@»€žυ¬gUArŸη λŠτRτrΜCͺp2‚QΎC›t@ύ‰“#O Φΰ‘&?iL’—h°§¬·€NŒκK†eΉ¦^yvMώ6¬ίwtμΫˆ|"ΏN<ρΔ*3-,duQΏO>ιξp‡;T™Gχ5ιtΉΜbδiu€Θ‹žŽgŸ2'έpšΟ8 ,bΨΥ–]΄ŒeζM©‹4 ­*ΐ΅uνΉQ₯[…'ŒiζVδ0y›1(š$Ί§hOώZΘΔ»ΎbdwŠΞ©ο>΅¦‚Kήό‚Ϋ‚pε£—vθθΨϊˆ, cΎε_ήRε>r $~\“Θzp(-g!—ΝˆŽKΧ=±8ΩΠΙ ²•δςk_ϋΪjcϊώ²Q=e±_ΐMŸ΅σΏ]qd#ωΓΠnnΝΑ»θʁР³~3I 3Rot]N‹!οœ™M―ΆΙΨlγΏη–fΟlktezsK3ϋHζτxΘξ¦Ζ{Ά c@ξkβε?x9πZV‰6ιέΪtγέ 9M ΠκάψŸmω² ψŠ¨ιΨΨͺ² bήΝY=ƒMϊΔzΠm+σtϊ ½DŸ<―Ψ΅΅Λ©~M~)“^ΙΉ%σXz LθAς“iϊ¦Ey:[›|Θ'‹ω>Σ’:Ι+Ž!σόe~­/mίΎ½.ΞZ΄wώωί«ε5gΗΆoΆv'ςˆ,u―œο—Ε8:fl䑐θYΈ%λӟεSvΫ—-ζ‡Ω‘Νσ[ ο8τΰ₯χρε³Ύ\uφΤχ€ν‘l”ηQzΤ\ο•f= rΨ-’ΧGώ΄8θοd)@[N,bS·θJŽs>ς^*ί }‘³ς­Ψ`,ζ΄|KΨzՁLσω”›ϊΊΒαΌ'ς›]8E[ΥnR‡6―w†FτΙΏΆ“r*’MώSF„Š‘CΊz’αYœW~Kƒ~ဋΌqκ¨wρOpγ•yLNšΐCΆΫ‘=™#jOϊ–4-d¬ΡίowΫΫΥ(ςBۈ£ml]π¦-±}]`;ΏΚˆΔη]yvτ1큾`½IΘ΅>LόIΌv°cǎZζΫήφφyJρ~κ₯-:}ΏΙ ύpjn:ΟάoΦŁΌƒφΎ/Ί1yΒ^JΎZ“$Θ—|Ϊ‡6ε]‘'ΪHΏž΄;6S›ΘG5c\֌8λDO>WvSc“8›πΙ0ΥNΩ,`ά9κGΝυٚa7€―Ε :λLΖ–@ψβ θΊ/|α 맩υKΆ}Hεc“η2ΈΫϊ$?9a`œΤ_#ηSή…}EwΦ2Υ‡ήLNλ9B¦VύώšΧͺΊΖωη/œ₯Θγ 'ΪΦΙqΜSφz';Νšδf:lψΖΰœJγ™.q‘‹\΄Άύ΄α» p©£~ζδ>`₯_DοβlΉΥήύzλݝiΦΛ©}$Nκ4ή©‘FQ` 1Iwt%p¨‘όΞ<Ζ(Βƒ ~›F‘”Œ½Σ₯‹KYˆϊL‚ 8:εα€²)>eΑλxLΖtBΜŒρμ'έVƒίδΖu ΌΙ€:C³Α“·Άχa°9α„κ`Ί»βeΨρ’&M{<3Xρ…qm•x₯Φ#Ύ}§κΕCŸ«ΑSϋ’€μn}δ7iΰΙi 7ω}ϊӟ1Η4„ΟΒAŒΡN! θΆyά£S;uT4Οev]–˜ΜNk?όα’Β‘r–“μtA•»r5Ψι_Όϊ_ύκWUc:΄Ÿ±’ό3φQΠ »Λ݈κϋ0τcH_Č?Ύg.Ž‚盨YτΆH:–[ίϊΦΥψOn›ˆ #ρ~Ζ5‹Ό&Dτ,‡Φ”G9o’‡šδη c<`ϊΨςB3 ςWyŒζθ`Τ0¦έίύ]K’ί•\1nŽ;ξΈϊl!!uGΉh!ΑSN&U‘Ixc‚‰Τ”R7q€ΜT>Θ΅>τ?ϋ’2vVCRQƒ, fΎ°¨ί/9 /9͐_ dέΞΥBLθ?ΗόN:άτOsΖ)»1d ΟE“gMΙη* ωΖ §υccv’gδgx Ω#άi§}bξά8ž[1’7Θvz$'œ[3 oηV1βΜιOBΐ’„…‹β/zρ‹ζ‹©p¨:bΰσιƒΔPφΗ΁΁-Ζ}Υܚ!ŸL>Ϋ§”E}9αζ“œG8u˜―›’{δ; ½ς_ŠAΨ)°αΘmyύ8Œ˜Ÿ γ|Q (";ΝYΙ-ς—ϋ©O-mΒ Ά ϊ&;…ArpWφξwΏ{Kθχμ_ζΨΡa•ιžœQ?Žη7ΉΙMκŠrιΠY axφ³Ÿ]ν‘ΧΥxςζ7½Ήξ5ތiΞ€'θΘ{μnΟώσ«Œ‡Χόΐ»lΑ}:Ά·Π¦ίK£^ -˜[GΈφj³Σqe^°mΫΆjο0~wθΨθζ””ym΅Λ,¦€-V–Ή±Σΐ΅§JΏΎότ•qΏ¦KΡ£'‘'±sΫ}oα^Iސ]ΙO¦eNΏrή —ταςeCͺE֜Δ1Ο_dŠzXόdΦο9\‘|V°γš―+~tΚYκž|δ IfΧΆϊΆς€\7χ·μ„ΊrζΨα£ςmΚf›θ½θπNΨ£}ΊŠ\δΨπͺWNΫΙ>ΌŒ{|NΤMYpl£U­²Sσ±?Ίτ‹_τβj[ ηvμΨ1Χyaθ΄0§^uόΑ·ς“Χυ{E{ŸtΨ¬Οiμ#Ζ8οΔβhd`‹C8@©+΄φa έΰ!9Κρέ;nσzΘ’ ˜“w%νφbοN{ͺHφΰmΒΙg‡z‘ϊΎήψΖ7ΦΎώΊŒ½ζXΖ“Μ36ƒ,όΆυG­ΨΤΑx¬σžΩ¦€‘GϋӁV ΏY_Ζhk&h焧™ΩόρξυK_ϊ2U'ΡΆ΅Ό«΄ψ^π‚ΜΧΒ8II³Π6όΘ‡W•O·qBmωwΟφ,ν¨Ÿ*Ώ.O§Ν»ŽAΫΞΪw0Ύ—V»ε1±grπΚ5O²CF2Ώ?ΏΘαΘ2΄m€έA8Η‰8χb£ -ςY½ΫέξVεqΖη2$:’C»Ύώβ‰\mΩ»{ŸΊ˜σΙ­γŠ¨Ν†WτΠqΙΩ8βΕαœlύ½ίϋ½κhΓΞ±>N¨Α!Ώυ+sγ*§yŒ‡sXΕ–1O³nπΐz­wb КτT}²Ύξ}γ=»žοP§Ν|υœ―VγCΛ2Ν<„νŠ\ΛIFπ₯}» οz@:t°E—7·ΣžΌ›-ΆΧƒ―Mθ&‘ί KΊ΄+²u[™S˜CϊτyΠd ΐŸqήΰΨͺΧ%©³U©λt­‹ι ”:‡~ŠZZfΘKΚ²nΐ πs‚‘DR xͺλψQ0(Υ ”ΰ'(ΜΏ‰>εσιOz!’0ιH&,Κq€“o±|πAu0!@x¨qκ‘ό?ε)O©Ž%„1!bΠ0°™μSϊx?2TGH)GΗkΛ¬B“2w’?Χ)β σ=ξq:!ρ>(ό Ν+κ^χΊWQΊ^U…©#!νTΘq…ήEœα‡Q.Ž"1θ\@ΓAυ=PZΫ2.ˆΏΰ.υ_+ΏψΕ.Θ8q·Χ²,Χλ H|ΠFœΓYKzUΖ=οT² EΫ‰NQ‘Q{Ι}βV”½hP8ζ˜cκb‰ Ϊ„MΫ2θŒqΘ&άβŽοφ1~iηΪ™‰Ήv¦}R& ŒŠ”;PΌ?»ά€v O;Θ4χιXUνYωϊ\U¨—ωU-ΡΔٍmqŸ·θ3žρΜ2ȝ]‚ζ $ ―Wτ1@xη>+&ςΡ%Ϊ₯Ε›‰Βš‚Γ—\›¨~Ϋ9°Os€Ρ…Ό1AΧ7€vξžΜ1 7L\ΘcγŒoš3ΠιGQώΒ„,`ζΉ½Β+=yANΗΘ=Ÿ‘„ο(Ζ “2‰L°iΧ‚ώLNΪΑ4Κ$Γ|ΆŠ΅0ucωW‹!ΒQΤδW &nv0ΐ―Ύδ˜O72 ‘ 5xΐ3ΔΩ!¬ r\]Τ‰,³£‡ρ‰U6fΑ:r,ψ<·r.αΉ¦.δœMcN‘ /m†A§1Γ’™ΟΎ‚Wlδ1]V=ΙΨπσˆ_>’žŽό'ς'΅\Ί΅Ή7ŸΝ!4»§cε6ώ0$›ΰ£ΕmΊ4ƒ;ynA‘ΝΛb΅qΘt΄έ[`KP/8FaΑ—,ψƒΦq$νΑϋ τ “sXΦμa«z±°=θA•/lšθΔNd6Ήw pΰϋοrη*‹ιΜμ μ΅l tFο€ŒVkΖ‹­d6ž΅r‘Ε½θ>ο-SΩχΊΧ½‹Ύ½J/Η‹WΌόΓΫήώΆjC%«2w€Ο»Ε7ό€+γ5:ψ‚Nό€Ϋ‚θγη}γ;]Χ» ΫZΧ°Š­š>nΎ‘­Xλ`Ÿη°γ]βŒΔCoΦ.πΓ\>zΈ6bqšJΦ“«­Η{IΜ΅»όOΑTάr’ρι#ρΦΈ`}mψΔe|Β7νšn`N₯_°SΫD[ΖZχ‘Ρ5χxŽ—ψ’½s>ΥF”―ίZDFςwζ;ςzΪχΎΰ4Ρ―Ωκ΅Ρ'<α σψ”ο*ΎδΣJΪΗ}ƒ}ŽμΡ^ΌgcηqΕY€C»š΅'έΰ ^|Ξ}ο~[Ρ³΄!NLΪτ“žτ€:OU.ω`Δφ₯,υ6†όφξ@Žͺάώ3K&Ϋd…,d!«˜ «»²$ˆ ‚Y—χ‚‚μ»χ^ΩD‹HD6QAd .Kd#$¬Ω“™ιχόΞδijšξ™Ξd2Σ=σ-˜twΥ©S§>Υ}ͺκΤS§τ[ςυ‹#ωg½τ=“£κRSιΨ+ω›ΘΜPίw}tήγKΥ­ΚCΏΥ ϊΎθ·|’Χ~”Ύ7–…@ε£:Χnθ˜PΫVΧΒΤ> c*υd«efΪφϊΎh™‡vX¬cυΡΎDυ«Ϊ ΄ ΅Χ*?Υ)ϊ λ8ξ?Β1œ~λϊΝϋϊκχ₯ί|c_}o/ΏόςΈŒv!(Ou€ΑŽήf΄uκΨ)Ά­¨Q[‹Ϊ ΤvμuœΚpάqΗΕkfڟθϊ¨νFu€ά@ͺίΏφ)ΪG©MBϋSYψο!όςβ1x>ǎΎώ^ώδϊ{}š—ν½Ολ―J#[_¨>U[·Ά…φω:vΡϊθw―Am%Ϊ©χKΥ:ηΠυ\ Ύ>ͺwN=υΤΨώ£s #ιxCη!J£}ŽωκZ£άt€i>h»«nΥώHί+Υ‡:Οφ;δ 2Χ7θΨ@ί]?Πr΅?PP’οί’λοωψnΩ¦i›iΠοIη#j»SY“εςωδ%νΧ5Nη:σγ"OηΛ-ŠΧ€ΘP€αΗKΤZ‘ T*\¬‹ουΉ‘Ώp° s*ό(c>ž_8ΈH]sΝ5qώ°Σˆ―αδ!5~όψTPHηNJRα@4*Ώ8.T._Κ+$Εiᇛ Q©P‘€Bεœ }Z=\XL…N:ίp‚‘ ;”TΨρ€Ηi]BΕ“ ;Άτ|αb\*μΏO;Α”ΦUe•}*DΔΧ;ΏΟ*έ8OΨ!ΕΧp’•.£§QFα$+υ•°ΞΎ½΄ώα9ΪiW*ΰ¬ewc₯ΣwAC8(ˆ―ϊΗ—v\©°£‹Λ tz|:αΊ7αˆtYIλyψkΨQ€·™Κ ΣΩ„Σ”Ύ[*O¨LS‘rN…ƒ„ΈMΓ‰\:οp²– ;Ψτηp˜Ξ#μtSaG§…˜TΨi₯§…j*μΤβ΄P§.Z§…ƒ 8N~αΐ<NΎy»aΆΧp2ž.8ιŽσ„“ζψͺοl8ΉM…“οt^a'‘ Q£©0Η…·T8эepχ§Ÿ~:N ;Α”ΆΏΎΛ?8αιεxYUžpp]g^}Πo"ωύlΡ0άΝ— Nι²θwNHβόΪ6ΎόI/MJ§Ρχ#sπνΦβvQ9ΒI@Jίu >=s>>#P¨ώ #©Πυrόώ«.Θφ›Ο6Nuc8ˆM…ήΈŠžŸΏ† Οt^^WωοΝM<­κ¨Π“N―ε…ƒήψ[Σ>0Ή|ΥΕ^ϊόΪg…͘.4X₯λΝ§}˜κSΥ«ž~»αΐ;kΉ΅Ώ 鴚'dΖ²„F²τxΥU!HΆΞoίΛ΄ΣυqhŠσ¨~JΦ‹‘±&]ζp˜ uΚ#“ˆηΥrCΰO*\ΐM…‹ΞΗ+΄―7ΒIuό½‡ ‚ρ:4|Ζ΅σιΉVΥ§«.WέζυœŽ»|Πρ¬κAŸ.(ΔΟΙ:IΣBο_©ΠΈgσϊ:\°L…φThhM…†ͺx¬­cόΠhΣi9ž―κ} ^¦ψaέ?§s₯UαRαξ­85™ήί«υcWΣiΎΠΨŽ1O… 0ιe† #©pΑ7–Mi΄_ΰω„†¨TΈš^nΈ_hΘ‰ιB£[<ΧΌ‘Α?½ρυ—GΈp“^ži§c]νoτΩT?_σελsxžwL.ΐ¦΄Όdωβ‡u„ή)yι\Gƒ—c]^@  όwͺφλψ,σΨ6Χ"}~ΥW:ΞσcM― 5ŸκP΅Sxώ™―αV¬Γόx1\ΜM/ΞσΧω¦«gΞ―ΟαBwlχ Νq9!x%.Ξ§σ °ιz³Ύ6 _^€©s^­}€φα"h*4π¦Χ%\€H…›}βr|^Υ±ϊΕ—\œN›Yn΅;„ΐςτyux”HΊώT†³f͊υ¬ζ ΨR‘α;–Auo²ΝBΗΚΚ'ΘΔe…‹±<αρρ5\pL…‹vqšŽέu<9SαΒDœεθœCΛ •SΪ—jπω—-]–ϊΥ―~Ug=Tχ«ήNδλ¦Ά·Πθ_g~9ψςΥ.¨6! ξ?¬ϋ'y.nŠcΩ$…ZίϋpƒJϊ». ₯·Ή?Τ–.ZΗ4j―Σw/yλιΤΆζηΜαfΉτοί§+cW„€€τςBπB¬›Τvθmyϊ.λXKν“4Ώη‘zΔΫ±•.\,‹ΏI΅y‡ŸtΎš¦γ™δwWοCt4ϊέ©ΞρiΈ¨šž¦6ΓΩ³f§—―7:ηVΎή§ΆrύUφΠΛEœ¦v}κXNiυ;Χο[ƒ—EηΧ:ώΣτ\αβ~Jϋ-KΗΛαΒ]<τAΏOΥ%žGΈ(Λ’Ί>YGκΈΟΫ›έ1f–γO£λ ~μͺΆ o[Χς΄ 2ίuœ™Y·hΓΛXFχMΆΥͺέYωiά 7ܐ €γgεlKVθν,ΎΎ:ŽΦΆ IΣš¦οͺΪK5ΈΉΞAΤ†ασ† ‰ΡJσ'Ηkϋ…@Ξ:ίΉ™α\Cλ―y/Έΰ‚:ΣΤ¦«s&MS;ΉΫ9­Φ΅ˆΣ©φk^>₯ 7K₯΄­ΌŒ™―Ϊo^zι₯qΊΚωΟώ3.Ζ—αΛΜ|υιΉΪ•^ΫΝΟίόBΗςρk*Ά±ΞΣτΤηHΉΈ΄φ{~έΐ΅_Χΰλν3{UΧdώFCπDάG«NHš„€€τwΔηΧwVϋP₯ΣwJυ…Ÿž|―uΦ5‘džZηΜλgšŽΣη¦ΙΌbζό“—€άτ‘ηϊ΄χ*½‹TU­υΊΖy}βux²Π9ΊΆYθA&.Oυ΄Ž‘5$·ŸΏΙ1)oύΞ2Ώ£Ύ’>OΈ1&hbzΝ£?Υ‹ΊΦ£Ά §ί“~ΫΎlŸ_γΥoBwφιχ Ώ°ΏMWD©ŒΥMΆΦOΫAƒΦAώt7‘Κ¨hZmΟ\ƒœIͺI]Ξ«.Σέ’:_WΟ :žΤ9΄zCV=žών‡{=€όΥ»‹ŽMu\«ΆΥ{ͺηΆίaϋXfΏκNO ͺ›’uŽΓU.ΥαΪ'δ|}΄/yΉŽ?Υ#²κU½Χ cΠ0{΄U½—y—€—YυιΞ°έwΫ=Άo(ν'΄Ž:ΠώHeτcl7ψςe―ε„Μ˜VuΏκE/ƒLΥہΪG΄P»‚ξP•‡φQj+ς}™κUυ&£m#;έU.ΔΆ"­lt,vΥΥ^΅Λ©+;ΫΙ'ŸΛ‘}€ξθΦ~Vwb«­JϋMνΤφ$7mΏψέ σkΠ>@m?Ϊo©­Πχ­qbΖ?*³Ξw΄’ίƒŒd|lEϊέ† ¬xŒβΫ\ίAκ·¬γ₯Ρo^η·j‹Κτ[χ+\@ŠνΧΩΎgϊ]θΌTm¨ϊ½¨½ΜΫPuN.ΊΗϊHǎήζνεΠςτ[W΅ζΡy­z½πί€~Χϊ-¨NΣ1¬~ ^ύτ^u€ΪαΤˊΞρ“ηϊ:ΦΣοHΏiηj5Ÿ!ž=Οͺ½Tm3fΌ痏Ϊ4]u©ŽΥδ¨ί©ŽϋTίͺ.Ρz(?MS½₯ž­T(/΅Qͺ–­²h=ΥC†κM_€…Ά‡zΠϊΛB–rΥ a”κI­Ϊ`5$η#ψΗΣ«nT{°\ΥS£κd *§Ž]Υ φ]·ˆuΟΊuUΉ΅|₯W―Βj_ΥΎIλ¬ΊKΫJϋ)΅ηͺ̚_m0ΪͺΎΥχ)„RΕ:[υ­ΦC=ι|E=¦ϋ ϊXεP¨νΰΗίξ¦s]#QύΊb―&n₯}›ΚΎW¨Η΅~jΗΧΰΫήΏΧΪ_&― (Ά™ζ ,Ώ4MΣ}P9τΔνw’Χ6άW―rΎ'ΎΠώBΧPτ;Π:i9^f­ŸV} ½ͺ Ωڐ΅ ο©7 ΅­ι·‘ί’·iœΞAτέΦοWίI}·΅υS ίξZ½Χ~OΏ j‡΅*mc ΎήρúϚGΏ%υz‘ν¬m₯ƒt¬‘γ ΪΞϊΞi;ι\N¦Ύ\MWΎΪ‡*–₯ν“9xω΄Ξ!@8Φ:7S]βλ¬yTi_¨ίͺκ?ύ’ΛΚΜ—Ου Θ]ίՏϊ]Ηφ^υ˜χE['₯ΧwόεΙ/§ΟνΛΚΚcoeͺσΤ+ˆκΥγϊΣoBϋ ύŽBHzϋ«žΥoIν™jΝ6¨~ΠyΊž4 Aϋύ^²moώθœZ=jYzˆŽ/½^Τo\ΏY/λϋ£ίPζ ο“–£cϊύdΞ—ωΩΛ£γjυζ€zPϋX#ϊ±­¬UNσͺ•ζΊι;Ά΅wθw υρ}³Ά™ώT—Λ7³ [e±£zzSϋ‡~ϋΉύ&Υζ‘γQΥχΙAΣTkΏ©υΡΊe4Ύ[χnqY§>]wΜ«y”Ο_ώ‹=ςΧp½6lΧδ΅9ίΧ«Nӟ?hπεΉιΰP«§ε₯ύŽzbσΆ+•_ΦΊ~ zQε•©οsTθό!AΖύκ1 κ)Gυ[EEϋXΏh]υ]Θ6ψ6Ύ^mFͺ{΅φύyΆy΄ΏφQ‚.ΣερuΞ5Τγ§Κ₯}¬gΥσ¨_¦η­}–Άy,Šϋ.•%[:O_θ―%asT9…^tΚηjΐЁGC›R_zύ ΄#Q•kH~ιuς«`”kήM{mj ¬sR£eλ\?v,ιΐ%9¨‘WŽTνxtbδJ«ρjόYΌd±½?χύτσ¬U9«BςƒΝdΩΤH J@mͺd΅μ\ƒι€*Uε₯uΙwPώš_ΛΣόͺάΌbΛ™‡~U‘ΎVγ†vjZ?Ήψzd›OΫQ;g5ŒkΗ‘yt ­„δ4$Χίσ―ώ4(LO§ςk[ΙYφΉΚ’.ς”_ν©όd’2*/-Oφ™&Zwπ©ϋ6ΩkΗ>`@PχIοδ΅Ξςρ|T&­·ςWY5=s›Ι@ΫCΣτ]ΦσΌ%Y—V΄«°AƒΕ@FΝ£γ?νG4^N“υ«—CΛQ}­²¨Dο΅:ΦUΉ7Ζ%Σ{Ω”Ώφ!ΉΚ¦t|Ÿ χΪ_Τ·ίTΨpύf½έBΉe;ΟΜg)ϊύͺŽ1~λή‘œΧλ-SΗ|ή `]|Υ1ͺ.«α6³.ρ|΄ΥυͺGUWιœUΗζj;RχκΊͺG‰ͺΡΧλΑ|Ϋ4|Ύl-Cu§ŽΙu~­ϊRλ₯zOA3ΟΗ}ώΜΧΜ2«ξTύ¦ hΊΐ‹uz€ž”΅>Ύ|ε£ΊSmbjgπzW ψjσΆ•SΗΩzΥ:kΌ―»—EεΧ ₯“—Y cz­“ΆΏ^U&mmC’eΡφΥcdtn }ŸΚͺ<΄έΌν ™^yθbtθέ&ξ;U&ί/{ώΙΧδώTλ r2΄nδ6Ο΅ίχv*}·ΒΩρΡ™*ώΠχXΏKΥcΙοq2½κ΅‘κw­c'ε«v2 κΉΎ§™ίγδόZ†~O:R{›ςΣχUυ‚~ ήNΜ#ω^Ώ#Ο›7?ό¦—ΗzLη֚_ηλ*w2}rΩϊύΞ mί‹>Xh©šTό=i>•Ϋ/_ ―π˜ m&σΡ{•[uŠΦGΛΤό:–S=­ϊHkχ A šΞ½³•IσΚ@2ΡvπvPΥ‘ΩΞν3Λ‘όμΛΠ:(ΐB7ͺ Ϊ/hϋh›ΝΗΓϊή(•KΫNƒΟλω%?{[­Ža5ΘΩΫG½Χ2΅ξ²™»%σ‘Ώκcύ©^ΧΌ:WμϋΌdz/‹^u²Μ΅.žŸ‚|§ ͺί‡ΗαθζμϊŸOi΄mu‘yήϋσLύR ίΆΪΩτέѐLοŸe§ο‘Λ-ΧχίηΥ«~?j³S υz{۟κDΟ;W}πOƒΎmL˜‘@ΫQΏ=ίfš¬ί―Ύ§ΪfΊˆΧ£Gχ/Υ'ϊi™™υ‰gοω©.S=§›ΗƒΏΎφΪ¬'>Ÿ^}^½κUυο’ΕK¬ͺΊ*ΦͺΫτ}υ}Br^½ΧwTuΏζOΦu™ιςύμεQzύ>uŒ¨:RίηΨ~Ϊzχͺ H¦ΝΜ_u€κHοΚXƒκ€ϊ~wJ#k­†ϊΪbT―©nΡr2λSSΩͺO΅ ₯Λ§>Už™ΗΪWϋ±F²Jω&w«κν7”—>kΫκΨΫ‚iσI“Μ³%ί7TVMאλϋΧReo¨ά*WS₯ih›k9 •ƒι΄vU5τΞ΅\¨{0"Ρ΅žν.ί.ω,§‘ΊvCσhh~•5Ÿ4ΎNΌ"€υ 4TŸhΊ††κΎϊ–ΡΤӚͺΜ εΣPΉσ±Ι'MCΛa:΄]πΤέϋ ΆΣE7]Λ6¨V½ΐ¨χ5šλ.[έq©A Ύ:¦Uύ’;1(‹TΙΑλ7Ρθ.M]ŒS·Ί}CΟ·Ύ<’iό½ΩΤΖd‹«ρ:[c²φ5κ!!<& .B*UζδΕ7Ο3W4]ΓΖήo峜|δZΖ#Π\ ύ¦TŽϊΤ7ΝΧ!WΧΠΠο5ΫόωΜ[_]δR‚Ί€₯@:Υ±νεφW¨.έπ @Π…^ο)'Yφlετ<ό5Ÿ4žV―ž^<˜F½,¨ξ!α†š\ƒΟ—9=Χψd:₯ѐ\·δt½Ο'Ÿϊε3>i2ΛΥΨΟΎ,υœ‘?%©7ΙΜWAgzzΐ5Χ\{΄P/ ς|2Σ―οη|ςi(¦kΠ«φΉκΩB½ι¦lυZ§cˆ†ςπωλϋ.δ›Fικς-‹ςh¨<υ-‡iM'ΠΫLyhΠ«Ύ§ ,PοKκΙEυœ‚Β3³­AΎeiW}Λ«―ΜυMσυΟ'§mιΧ|ΚΪPš†¦kσIΣω._λ’A―ϊ ( R½…©ΏxΠ ½μσϊζΎ9_!„€Qσ-L}•’ςπιΉςΥτli|œ—CŸ³ε‘-ζΙ–Vγ3Σkœ†dϊ\iΦ']Μ4Λ?ω.'9kΆuΟUFŸ\Nf^ž&9ήίϋ|υ₯QZO§χΉζ“¦‘Ό”w2₯OΙe'Σε3^ωδ3Ory ½χεϊk2会ž-σ΄ώ93―t>-9ΞηΧ«χtΙi>έΣdNΣgŸ―Ύ4ΙtzίPZ₯a@ Πό»Ÿo9σύή{Ύω¦W:Ÿ'[YςΙGσ{:5œω{ΟOŸ3Ηω4υιΉΚ²‘y$ηχeψ2½ zΥ8Ÿž9>ω™χ PόΙίzΆϊ ί5Μ•η™œžΜΣ§'Η%ίgΞ—LοΣ’γ’σ&ί{Zk(½OOΞ“ΜΛ§k\2MrΌ¦ιsrΊΖiπtΙi>6EνΏ>.™.9]ο=Mζx}NΞW_ΊdΪ†e[γ@ qλσΝ΅ΟC―jΥ…~'Έz88υΤSγ¬κ©D]λϋΕέΑͺ‹ΆΊσTy{mZϋˆ?%Φ]Άzά‚‘‘ΗD©Ϋrέ‘ιy¨žΠŸ‚nžαωH£ωtΧ§•Ελ/ŸΖϋ8½Ο5xšδ|Ι΄šξi’γuμ­nέoΈα†ά£ΐΏγΛ£ωτ^@•+mΩςe1PH4žFyzΊdώΙχΙε'Λ™οι“Σ}œ§KNσqžF―>.™.9=™&sΌ>'ησΌ²₯K¦m(]ω_|ώύ¨o›{­]tω€ρy“i“bšξi’γύ½OΛ5ΏyŸΗ_}|Cσz:ŸO―>‘y}žΜtͺou—ϊqΗσR Ÿ-z{/CΑͺO5¨Ζ\ž>sY^Mχ4>±―ž—5™O}Λρως-£§σω|9ώΩ§ϋψδ놔Cωψ2’yκ}r™™iκ›–Μ'W:]ΘΧ#f4<ψPΈXyΘ‘ρ½;ϋς„₯ΰυ€ ο­&&ΞσŸ\eΠ쾜dšdΆšž-Kζ‘qκέΒΑ¦‹±κΕ!ίAση*‡/'ΉάdΎΙωr₯ρτ>=9OσWOγŸymΌ@}Ξυεšάώ>W^šξi”§§KŽσχώͺ Gθw₯Η{©Φ|>=WΩ|Ί/#[:O“mšΟW_šlσεηωxΎ™ι4έΣdNΣgŸΦΨω}>Ο'Ϋ24ΞΣι}fΪϊ¦)½ω€σΌ“i}~½jΊ§IŽOΎχιΝ#s>ΟOΛπiΙqΙe'ί{Zk(}>i“i’yzήώͺG˜+FPΫnϋν’E*ΚχΣεfϋr‘ύ ϊε)6&Ÿ|JΣΠτd Χ'­ζΛ7}Ύι’eIΎoμόλ;ίϊ¦χ2ζ;_>ιςI£ε6”‘ι •½ΎωsMΛ5ή—•οk>ωlhš|ζWyσM—ΉnωΞ—oΊΜόωŒ@‘ l¬οtcςmΜίω<Ί ίqΗc–―ΏώΊιώo}λ[ιΗΗ¨'u_ΤSOΕtzτμa»μΊKΌ{>[9<ίlΣ’γJWίτϊ¦­Ο2’i“οσΝ_σ¬OΪδ2x_Όωlσ¦JγJωδηi³½nΘόΝ1or‘ž½4|ε+_‰ιΡ£sΔθ0ΡγτΘ9υvΫ³GO[όαbΫ5Τ§>=™g$Χψdš¦z―e5fyωΞΣPΊ†¦7΄ž™ΏΎyκ›–,Ktκ=m=φˆIœxοDσ’Ά~ߞκ ξ{ίϋžύθG?2¬ž|ςΙ¦‹ϊ―ύь3μ‘G1=ΆqΑ‚vφΩgΗΰ›ϊsoΎ©ΎΟ\Ύ|Ή]~ωε¦WOθρ Ϊ·N˜0!>ΎͺωJΔ’Θ.ΰίUυ\¨γbυς€cA=TΓ!‡=]φ\‹@λΠρ‡Ξ uΣ…φG :SMίψΖ7βω`±6JΒ Τ>ΘͺυnCΦ @( έ™«ηX_xα…±ΌoΎωfμ–Yw’y7ϊE±"@@ U¨!TZΈϊκ«cΠK•:χάsν?ψ :4kχφχΪΩuΧ_{±Ι•‡ε  »νΆ[ RΙ•ncχΖ^‹_{ν΅1P&Χ2?όp;ν΄Σb .3 €Cΐλ₯E‹Ωο{;χηηZͺ&ϋeυ¬₯Ης©~κ₯KΦ:yc”Qyz9΅οPέxλ­·ΖE©w”\ϋ‡U–Φž―[+ΐS=¨ύκWΏŠ=ed[o\yζ™1θUZτy³₯mΞq^Ž₯K—Ζcˆ?ύιOιΕ«{¨·O—žΘšYΐΏƒϊ­uΤQu–ώ‡?ό!Ÿg>κ³N"> ΠΚΤΪ)§œRg-\£ήΕ<΄ΞΔ"ϋ@0M‘m0Š‹ €­W@A3κJ^w ι$L`›l² ­w“³f € €@Q̚5+φ‚ »άu¬ͺ pυ€8|ψpΣ£Gτ¨‰Ž;Φ{άͺ^g ’‹ͺΚOw k=NJωl½υΦ֫׏ˆ*•Y½=h½Uζ+VĞwΛ¬υξMΕ€llΏ˜«@=ZOQ―aκαKΑ|ͺ‹T—Ž92Ύͺ<>ΟΖ.[fώzlΟδΙ“γc…Τ3Ϊ6΄Οh©ςd–―΅|vO΅%ιϋ ήŠ΄ΏZΌxq΄VοϊN( E=κΕ~}_τθΔ'Ÿ|2ήH¦ή τψͺΆ–MΕzΉ€ΦfΜ{&\Έpaμ‘p—]v±v؁^iŠ|ϋRόΖ ψoCηxz<ίηŸn:O;vlμY¬5H‡`šΖG˜@@@@V+ΰ €Ύ‚Ίˆ»jΥͺx‘Nε*++}R½I“ω蒟i£ ΐ:uŠψ”Q2]:γz“,‹—YόΤcd‘–Ή…¨X,4“@²^"U—κ1#ΊXΥΎ}ϋ:TΚLΫLEd1Ν,ΉυΘ/ν«5TTTΔύ•)3­/”WυP£2vνΪ5©ΠΛ[(n”£yt¬?„«ήΥΐw΅y·K+LίθόPη‰­i ˜¦5mMΦ@Š^@'`ϊΣ ±ΦΑ]τ†@@Ϊ°@ςψ4“!ί  ε‘| νΨ·˜Ή}ψŒ­O Wέ”k|K¨,^[½ρ·€[gξGsίψ%Ϊ°%¨ά™λ²a927.ω½ΜόΌαK Z‡@kϋmLΣ:Ύ—¬ € € € €U@ £Ι‘±Ί’ω46d9šγ}1–Ή9\X΄œ@²^R)Š₯>m9±ΦΏδbώNxΩω·ώοi1―‘O΅|W‹yKRφ¦πίFkό]LΣΤίςC@@@@@@@(Z’-9G@@@@@@@@ ‰¦ibP²C@@@@@@@(^‚iŠwΫQr@@@@@@@@& ˜¦‰AΙ@@@@@@@@ x¦)ήmGΙ@@@@@@@@šX€`š&%;@@@@@@@@β ˜¦x·%G@@@@@@@hb‚iš”μ@@@@@@@@ŠW€`šβέv”@@@@@@@@ ‰¦ibP²C@@@@@@@(^‚iŠwΫQr@@@@@@@@& ˜¦‰AΙ@@@@@@@@ x¦)ήmGΙ@@@@@@@@šX€`š&%;@@@@@@@@β(/ή’·\ΙS©”ι/9”””˜ώ@ W [ύ­R‡ξ6£d €€¨Ώω €Ε+­ηψ»x·'%GΆ#@ύέvΆ5kŠ­K@υ·υ΅γά%xEς( •iέ¨όηms)EΥPΐL>iΪ+Œ€@CυsCΣ `( Π&ͺŸšή&ΡXi@ ς©ŸσIS«B@6%OݜOš6…ΖΚ"€"ΠPύάΠτY ŠŒΑ4yn ίΑTWWΫάΉsmρβΕVUU稨°XίΎ}γgO›gΦ$Ch&₯K—Ϊœ9slΩeV]Sm₯₯₯Φ΅kWOD|@hΥΙͺ·οΈγ»ρΖΏT†ΣO?έN8αΫf›mΎ4 €-' ΐ™—^zΙξΊλ.»ι¦›κDA&L°ψ°Ρ£GӘ_G‡ €@Λ x{Θ²ωΰƒΪέwίm=φXνΆΫnvτΡGΫQGτ}ž:‰ψ€ Π¬^+ύ°?όαφψγΧ)Γ{μaΗsŒyδ‘Φ½{wŽΑλθπh9―Γί~ϋνΨ†’vu Ž;ξ8;ώψγmχέ 5fΗ@IDATw·²²²δ$ή#€δ gš0>Ϊw@ό±ύόη?·›oΎΩFŒa'Ÿ|² >ΥψόΌ"€΄œ€ΧαΟ?|lθQP€ν>θ`λΉIO[όα‡φΗ{ο΅ΏύνoΆύφΫ›κπQ£FΡΤr›Œ%#€Qΐλog~ψ·B―Kb£ύAd=zτ°Cύ­w]œέuΧ]ν–[n±­ΆΪŠϊ›ο ΠΒ^―\Ή~σ›ίΨ™gžKtώωηێ;ξ{’yύυΧνΚ+―4]¬=οΌσ쬳βnέΊQ‡·πΆcρ ΠΆΌώ^Ύ|ΉέpΓ φӟώΤΪ·ooηœsNl/Q ϋk―½f^xalΏψβ‹νŒ3Ξ°ΚΚJκοΆύΥaν@ Όχέwν?σ?ν/ω‹ν΅Χ^±EOΥΠqχ£>ΫΎUάΏύοΆί~ϋ@Ι) Pψe„‘π‹Ωr%τ*΅£9ϋμ³γF>‡~x ͺωκWΏj_ϋΪΧμ+_ωŠ=ρΔφΚ+―ĝ”ωδ;°–+=KFΪ€ΧΑκΞςWΏϊ•ύ󟌍Aηž{ιNXDͺ'5κ+νΓ?lzlߘ1cθa¬ν~mXs(―ΏΜ~ΕW˜"Τώ³ŸύΜƎ[§ώ^»vmμυ C‡ΆΛ.»˜^}ώXŠ€΄I΅£¨W±ο|η;¦6“[n½ΕΎάχmλ­·Ά-ΆΨ"Ά‘¨eΚ”)±χ_ΥίJ§ΑΫ`Ϊ$+΄°€κΰ^xΑΎχ½οΩ–[n{†T/^«ύdηwŽυχψΗΤzέJτψ‡@ ΩΌ dΥͺUρF#u §h\zι₯ρz¦]κζ#ΥίκQLmδK–,‰Χ1»vνJJ³o1ˆΕ&PZlnΞςj'€aρβΕvί}χΕχ:™ΨsΟ=­]»v¦nηυ§‚kΤ=ρ¬Y³b”~LΜ? €-.0}ϊτxWμΠ‘CνΨc΅Ζ“ΥίͺηΥp―Ζ! κΒXΟg@h9”Υƒ«ώVaΓ† ‹½Šιn*ΥΫ^«·H=βIƒ"?ψΰƒψž@ZN@bUO?ύτΣ±‡rˆ4ξ λάΉsΊ EΊφˆ#Žˆi&MšdUΥU΄άfcΙ €@¬ƒUλ_Šjλ7nœuκΤ)]λύh‡vXLσβ‹/Zuu54|@˜7oΎέύ±$ ¦QMiIiΊίl³ΝbΟνκέW½όϊ# ό:h¬E@ R€`š<6KuMu žQΎ’85θδBέ[ϊ]S;v΄ΑƒΗiK—.Σ}ZΙ? €-" ϊzȐ!‘Α~_S―a>¨χ“…ž={ΪN;νƒ'Υ₯1 €@Λ ψ]­j˜Wcϐ³ώήd“Mb/cκΚxŊ-Wh–Œ Π1ΆŽ΅;WvΆm·έ6>&ΔΗω1Έ^δα³Ο>³΅kΦΖχ~|?π €@³ ¨.++‹£GŽ½χfΦߚΎωζ›Ηr©ώ^³fM|Oύέ¬›Š…!€_¨ Χ1ΥF’Ηc4(N―IΥ½ŽΩ₯K—xΓ’&z8Χ1ΏDΙ@ Ž@yO|¨#ΰ;‘^›φ²«―ΎΪτΜoνŒ4¨α'9θΔaήΌyq”ž'›9Š€ο@ 5 xέΩλθεΨ~»νγUϊά­[·X,Ÿζe\Άl™ιŽX ͺΓ“υwRƒχ К2λΖ–ZW/‡Ί‘Wέ¬γκ\υ·‚Ψ_ύuλΥ«WlμΟ,3ux¦Ÿ@ 5 x½Y(λ¦zϋδ“OΆο~χ»¦γ5$Λ¨χͺŸ½G1υTS^^Ϋ<εΣ e]( °1’uγΖ\NΎy'λοlΗί*―nXZ΄hQΜRυ·zoΧΰλΒρwδΰh^ο΅τͺz9t3ιwήiUUUΦ«w―X,ΏN©ΊYιV¬XŸ‘‰ήξΣ4Ž:\  ΠΪΌήΜw= ¦ΙCJχ½{χ7ε«―Ύf<ςHL£G‰hHξ„ΦwΓΔ ψ@`ƒ:tμ`z4Hζ :Z' ”|όρΗγδργΗΫ¦›nZ')υw> €Ν& žͺΏώχΏΗςμ³Ο>¦^Ζ2κπL>#€_@uo=β_ζΌ—ίχή›™nCΡcWύb¬Swgͺρh|κοοΜ°G}4hδΘ‘ι`H/!uΈKπŠ4―€Ž§Υ»oζΰmΰΒ ΟΫΏύoΫb‹-ΎΈζYςΕΤα_Xπp‚i\’ΧdD¦οPΌhζΜ™vν΅ΏΆ… Ζ»―vήyη:Ή)tΥͺUY£:•‡Ί°WΐŽG‰Φ™™ €@¨ŽT]§»’τ νBΌχϊ[Ÿύ½ž{ώωηΗ"qΔι“M§ώ.΄-Iy@ ©υw<μΤ±ΈκοΏώυ―vήyηECφιΣ'Ύχϊ}υκΥιnη3έtό­uΧ1Έ§ΟLΓg@ ΠΌ.Swνή³K‘”ُΏU―g½ εγ?Ά[oύMμ=Rν'»οΎ{,Ά£«ύdνΪΪΗ>e―3υw¦ Ÿ@ ˜T—iPύ­ϊ¬†ϊκοΕ‹ΫM7ίdΟ=χœν΅Χ^Άηž{Ζ’{ύ­z^7,ι5sP΅±¨ύ»ΠΦ9³¬|Fr ¨.Sΐ•ΚΚΚ\ΙZlΌΧα~όνŸU —^zΙ~ύλ_Η²όρ6tHm§ώ¨m=}Cν(Ω­³κvsxήΩ1(T ©Ώ ¦Ιs«&wΎΑϋοΏoW\q…=πΐ6bΔ;ρΔ㉐§QφΣ¦M³ .Έ έSMœ– J₯%q瀓 ]|Nή‰•g±H†„€κΓ7ήxΓ~ρ‹_ΨqΗWeJ"³χΟO<ρ„{ξΉ±.ώοΕgΚj>ΥΣJσΦ[oΩE]³ηΜϊ{ŊρΔ©Π.^$םχ €@}jΘ~ωεΙαxφr;ꨣυ_}σ4η4――΅L―›υ^=όόη?9ηœsŽxΰ]'ΝΔ‰νwΏϋ]|L«_°ˆ‰Β?zΌŸ†Ξ;Σ%ψŠM@υ£Ό?ύτS»ύφίΩWΎ2ΌNΨ듬ΏUΥα:gPy―½φZ»κͺ«c©ži4hΥΧψΓμž{ψΛΌ Kύ©ψŠX@u.VZΉΚn½νVΛ‘<ΞmιUΛUςΙ'vέuΧ₯/Δκ|Ψ°auŠ«M/½τΨ^Η‡$λp¨Wο“ώh‘:3σ(ΟΚ5ΛΎύνoΫΩgŸ]p7Θ'λπδΎεΥW_m(“'OΆΓ;̎>ϊh«h_QgστΣOΫ•W^{—LΦίΪ,Ί†©sΚ.•VVZXA E𡑈 PRL³ž0Ήzλ­)ααϋγΎO(Άέvۘ£vZJ«‘’’"žiδγ5M_u’‘»j?όπtš8 €E ΰu™Uf̘ž»Ί’ KνuΈώς—ΏΔ ¦OŸn'œp‚rΚ)_ †τϊΫηΣ«―³ΧίκΝfΠ A±α?yΒRΠΪΌ€ΧeͺΏηΜ™{Q,d―‡U?τΠC±ΡjφμΩvI'ُόγΨθi|=t—˜.Π*`&³!H HsζΜ‰A8J—9έσΰ(TwN™2%vΣ^]]‹ŽTΝο,-”r«nΦ ςκψωϊλ―·Λ.»,ŽSΐγΎϋξί'λπ]»Ζ6έt”¬Ÿ•ζε—_Ά Ψ~ϋνλώδτ˜ €. ϊP7#=ωΤ“±‘P‹λυ²Κ«›IHsυΥ΅wέu—ν½χή±θ^ΟλƒΪΊυˆέ0ͺχš¦?π+˜ςΕ_΄Ρ£G§ƒ( uέ) M@bΥφύη?ΩφίlI fœΧα*λ€H½ͺά^xalΛN¦Q:;ͺ %σ\λύΞ;οΨSO=oΒκέ»wμiLϋ@ ΠόXTAݍ­ΏKB&΅-…ΎΆP>ίΉθU 8Ώόε/MQ ͍7ήhίψΖ7r–R ψ υw”δ(&?U]Ίϋ€IΆωζ›Χ©/ a}’υ². ͺξ½γŽ;LΑŽj?ςΘ#cΐL2—;WΚ{ο½gΓ‡·ϊ―²ΣO?=}Γ©ΟΗ+ PΘͺοt=oήΌyvΐΨχΎχ=ϋιOZmΑ^7뱩j·ž0a‚ι|A=ϊͺ>χ2½5_ζ1Έ( RmFͺϋ_{ν΅8?νΰ™z|FBπϊ[7e*\Χυ”ŠυΉ–GΟ4ynέδH6κΒMπ_ϊΧγ#@vΩe—˜“§ΛΜVΡ›ΙAι4ψ3b}zy›$ιΔ{(lΜΊ¬PKλuσόωσγEu;¬AQω§vZΌsΚΣdƒΧΟ>>s}ΊN¨@bΘ¬Λ ΅ά^7λnΨnΈ!^tPYυXAυ(Φ·oߜtRδΗΪ™λ§ijόριΤα™B|FbXŸΖŸζ^―Ώ΅άgŸ}Φ.ΎψbΣ#VuΡX½Œ7.έkAζzδͺΏ5^zυwsoQ–‡M) ΊlΥͺUM™e“εευ·^Ÿyζ™Ψƒύ‘yΥ»˜."λΨΩΣe.ΨΫHrΟUΗg¦η3 Pˆ:Ν|Œt!•ΣλζΟ?άΤ‹˜iT^έTϊ³ŸύΜΆΨb‹œυw}υ³Χν~ W₯g@ŠE@υ–‚ UOοΐ•Ώ<Δ|€“έ λ½(ϊTΟχΦHƒ§Λ#ΛtZίhΙWvBω’ Aΐλ½d¦rR=ζeΤ…ΨK.ΉΔnΉε–H§Fόc=ΦΊuλ–“σ1υό2ΧY‘ϋ~b‘O>€AZR ³.kΙ²δZΆ—qN`Ώ4ά=uλ­·Ζ€ΊVΗα]ΊtΩ ϊ[™%λςBΪwε2a< ΰuc‘KxϊψγΫO~ςS{μ±GxTφ₯ΆΫn»Εβη».žN―ϊӐ|υeΕ όƒ¨€Χe*žΧa…XT―SΥ»‚Χg͚ΙwΡE٘1cb‘“λΠ:x[IΆu^Ÿ|ZΣ@)ΰυUΆΊlc.w}ςφ2~όρΗρf€σΟ??ΞΟΝ€Ω–γλκϋ₯ρqΎ¬lσ1($ϊŽEσ-'Α4 HωNAΡJκ†ψ€“NŠs(’sΒlΘΰ!ιΟ΄SΡFΡkrΣΐ"˜Œ΄B«ϋΌŸ;wn ~T=€—Ϋϋ[;2t+―gΑ*:_εMσጠ6¨1λјyZΝ†gE@&πϊ{Μ9φ³Π‹˜ΪUg+ fόψρΦ‘C‡υΏ©§›x#‘΄ˆ@1Υeκ‰F=¨N?τΠCγ£²·Ϋn»tJxΖ΅₯jjΕZίzω4mxŠl @²KΎίΐl›|vΥΩ €Qb;μ°Ψ;ΝθΡ£Χ»ώΦόΎώͺq>dηΣxEΘ_ΐΫPΤ#n ½ΰ‚ βΜzΜͺ•έ½{χF΅‘(_@ ˜šβx“`š<ΏώχΏc ΝΠ‘Cγ]Uz¦V§Nβάɞšb£δY$’!€Τ#ΰ'+V¬°ΫoΏ=DŽ;6ή »ηž{¦ητξ)γz§L»πh)―Ώ—/_{S Ν^{ν­Ίϋ‹Eύ¦ΰ  P0^‡Ώωζ›ρρΨ•••φΓώΠΞ:λ,λίΏ,g² E5  €-/ΰυχλ―Ώ ΅kW;ιΗ'ΩžωŸρΡͺ*!υwΛo'J€dΠuΙͺκ*›8qb €Ωzλ­νάsΟ΅ο„›I½ξ¦ %›γ@†¦©ΗΘO".\h7έt“ <Ψ† f[m΅•-X°ΐV―^ήy6κ™F]Ξ0ΰKΣ< ― €Ν'0yςδx•κξ]vΩΕzφμioΏύv<Α(-)­SΥϋ›mΆ™υθΡ£Ξx> €4ΏΐΛ/Ώl—…Η;mΉε–±KyΥΝͺΏ½G±d‰TχλΧ/ήm•Ο{@ζPCώš5kμΎϋξ³ιΣ§Η:|§v²•+WΪΤ©Sca’7!©ώnίΎ½ 8Π***š·°, @ - ΊYmέώσŸM½CŽ5ΚvΨ~S€ϋ”©S¬4ό— €τϊ{σΝ7·vνΪ₯σα  €@σ ¨>V>㝱Wš―~υ«¦ή Υ1ΐ»οΎkzκ†ΤxΙtsΣM7΅>}ϊψ(^@rL“&9ϊΕ_΄?ύιOqΤμΩ³M]Χ7œzκ©±α_A5Ύ#«/=Σ@šVΐλή₯K—¦λο)S¦˜ώΉζšzφπΓΫ7ΏωMκοz•˜ˆl―Ώ?ϋμ3»χή{γBtρUW\qE½ }τΡGӏI^¨­w&&"€4™€ΧαΣ¦M‹½‰)γW_}Վ>ϊθz—1f̘xρVA‘žG½30@ IΌξ}γ7μ’K.‰yΏςΚ+vΤQGΥ»έ°τΰƒZοή½©Ώλ•b" °ρΌύγΙ'Ÿ4Υγt#]wέUοBoΎωζψ¨Μ@›zgb" Π¦©g£ϋN¨sηΞ±K΄Ž;¦Ÿ ›m£τ«V­²‘#GZy9΄υΠ2 hEΩ+—Ώό₯©+Ko ςϊ=YˆX‡―^{¦IŽη= €@σ TΧTۈ#ςΏW―Yξ~ΎωKΛ@’:ώΕ/~{œΡρΈ†’’Π‘Aέg:)ξ”νΥ«—uθΠ!™ο@Z@@νέ™υw6p―ΏΥ» €@Λ θ˜[ΗΤΙ:\ΗΪϊΛT―«Χ±αΓ‡gž™žΟ €@[ β#oΐ~ϋνgϋξ»o)k“$wPΙχyg@B@ πΊ·[·n¦ήΒD“οΰσϊkΎσ‘@`ΓΌξνΩ£§M˜0ϊ{ΓIΙh6―ΓG’ο‚}>₯OΎΟw~!€l˜€Χ½;μ°ƒmΏύφygζσi†δϋΌ3 ! €@“(@ζ;ίωŽyδ‘yηG½7 @  L“η€KžP$C P€:Ό7 EBς ώΞ‰$ €@ PΰF‘H €@Τίy ‘(Pκπέ0 ŠZ ΄¨KOα@@@@@@@@hB‚iš“¬@@@@@@@@Š[€`šβή~”@@@@@@@@  ¦iBL²B@@@@@@@(n‚iŠ{ϋQz@@@@@@@@& ˜¦ 1Ι @@@@@@@@ Έ¦)ξνGι@@@@@@@@šP€`š&Δ$+@@@@@@@@β ˜¦Έ·₯G@@@@@@@hB‚iš“¬@@@@@@@@Š[€`šβή~”@@@@@@@@  Κ›0―‚Θ*•JΥ)GIIIό¬ρώΎN> € € € € € € € € °N UΣ(PF₯₯₯9fHγ6ΥπύG@@@@@@@Θ&PτΑ4ήγŒdjjjμΣO?΅5kΦXuu΅UVVZ·nέ¬ͺͺΚV­Z? ΑηΙΒ8@@@@@@@@Ά+PΤΑ4³zυj{ϋν·ν•W^±_|ΡζΝk/½τ’ύΟόwάqΆ`ΑϋΝo~c{챇νΊλΦ₯KjΪξwž5G@@@ZTΐΫ΄ό΅E ΓΒ@@@@ΰK₯_S$#Ό±aώόω1hf›mΆ±N8ΑnΉε{ο½χμ“O>‰=Τhu>όs»μ²Λμ€°σΟ?ίfϞ₯<@@@@hNΉΏ6η²Y € € €4,P”=Σx ΝβΕ‹νΚ+―΄λ»Ξϊτι{œ5j”}όρΗ6cƌ0#‚΅kΧF‰m·έ6ήθ‘O—^z©υμΩ“jώŽ@@@@ ‰τxςI“&ΩϋοΏo;ν΄“ 6¬‰r&@@@h*’μ™Fwν¬Y»Ζξ½χήH3pΰ@ϋρlχίΏέpΓ 6zτθθSVV_GŽi/<‚ν°Γ6xπ»ωζ›ν©§žͺ5,i*JςA@@@@ »€χ¬G•{μ±φέο~7Άc­\Ή2ϋ ŒE@@@(ΊžijR5VZRj³gΝΆ[o½5yζ™vI'Y§Nβη:ΔWο*WŸΗμ2Ζn>0φDsϋν³b0Ν7ΏωMkίΎ}‹α³`@@@@Ά%°hΡ"«¬¬΄<Π,XUή±cGzOn[_Φ@@@ ΐŠgš«νJfφμΩφΖoΨξ»οn|p €Q MUUUl|HΊλΞύυοίߎ8βˆ8iϊτι±ΑBόΞ δ<ΌG@@@@`c¨-J!׍`΄Km aςD@@@`ΓŠ/˜&42h˜?~|έuΧ]­_Ώ~ρ½z¬ρήhβˆu$Ǎ1ΒvάqGS0Ν’%K’Ιx € € € €ΐF¨©©‰ΛP M²έj£/˜ € € € €@^ELγk΅vνΪψVv*/ΟiUeeeVZZjήhαωρŠ € € € € € € € €@ΡΣtνΪ5n==ξiι₯ρ}}έβϊ΄9sζΨ€I“lθΠ‘ζyπ5@@@@@@@@@@ELγA1ƒ Š[pβΔ?Ω”)Sβ{u‹›μqFiυ§qκfŊφΔOΔ΄Γ‡·ώύϋΗχόƒ € € € € € € € €€Š.˜Ζ7ΫΰΑƒνˆ#Ž°ε˗ٍ7ήh―ΎϊjœΤ]»8£z€“lHσΩgŸΩέwίmwήygL7vμX«¬¬Œοy6udΰ@@@@@@@@ Ν ”›€_ΤΫΜf›mf'œp‚έwί}1ζδ“OΆΣN;ΝFŽiK–,‰«υαβνν·ίΆE,²ξΐξΊλNλΦ­»ν΄ΣNΆί~ϋΕ4Κ‹`šbϋP^@@@@@@@@`γ]0<ψελ_ΊύξwΏ³γ?ήϊτιcΗ{lT5j”ιοΟχύΩnνν6uκΤ|³rεͺLcvυΥWې!CbZΟ+~ΰ@@@@@@@@6-P”Α4ΪbκQ¦’’Ž9ζλΧ―Ÿύζ7Ώ±‡~8>ιΝ7ί¬³Q»tιb .4υ^sβ‰'ΪvΫmWg:@@@@@@@@@ΡΣψγžΪ΅kΩ4zτh;ε”Sμ­·ή²E‹ي+b`MηΝmΐ€¦ι#FŒ°=z°ε@@@@@@@@Θ*P΄Α4Ύ6κ‘FCίΎ}γίή{οm555±ηWΠMiiiϊΡPž^Σ4π˜§ZώE@@@@ Q`εΚ•±½·S§N…X<Κ„ € €­P ¨ƒi²Βh\YYYΞM•m°Ι6-gFL@@@@@Ψ(j³U{ν΄iΣμ–[n±ͺͺ΅φƒ~`Ϋn·] ¬‘-w£°“) €­Vΐ―k9Žh΅›™C IŠ6˜¦ͺͺΚV­ZU§²KV‚Ω”T1ϊI˜¦«ΗήTTTdKΞ8@@@@@P[nuu΅=ωδ“vνuΧΪf}7³~ύϊΫ–[mΫs“νΌ-P<‰ €E&@M‘m0Š‹@]0α€ έ‘pΪi§ΕG;­]»6R†σ«\SZϋX§0F€IUς±OJX^^OΈΊvιj}ϊφ±Ρ£GΫ6ΫncCΡd@@@@@π@΅ϋκ†Κm·ΩΦΪ·oo«W―6γζΘΪ0,@"XΈp‘͜9Σϊχοoƒ ͺΣaC―EG(PtΑ4n±bΕ {ζ™gl“M6‰Α1νΪ΅‹A3 ,π$ιΧΝ;Ϋ¦›njšG'_K—.MOSΟ4ΊΓa§v² /ΌΠφέwίz•ž‘7 € € € € €U@w‘+€FνΈάQΎQ©Ι@V'ΰΊŸ|ς‰]~ωεvέuΧΩQGeΧ\sM ͺρι­nΕY!h’ ¦ρ¦>}ϊΨχΫ«―Ύj“'O6’9α„lΘΠ!VΩΉΦ¬Yc|ψ½τβKφάsΟΩΘ‘#­²²ΆέvΫπŒέ*ϋτΣOν­·ή²3fΨ’%Kμΐ΄§Ÿ~ΪφάsΟ:ƒji2A@@@@h΄€.x1 € €λ+πΑΔλΙtM:ΥζϟƒiΦ7#€@Ϋ(Κ`4 <Ψφή{οψΜ\=ςιœsΞ±ργΗΫζ›on]»v]}ͺΗ™εΛ—ΫG}dΟ?χΌύΟ―Ηήxγ SΐΝ!‡f>ϋμ3{ώωηνϊ믏½άάtΣMΆε–[Z―^½¨i[ΏΦ@@@@@@Z™€wθΠ!vΔ GGΦΤΤ΄²5du@`c]0w·₯žd½φZ{δ‘Gμ’K.± &Δ^g’HεεεΦ­[·ψ7tθPλΥ»—pΐφ»ίύΞφΩgΫb‹-bΤ‘‚gzτθa'žx’έ{ο½vϊι§L“„δ= € € € € € € P„zς‰ΠψkEFf(mζε5Ωβ¦OŸn·ί~»ν±ΗvΔGΔ@šκšκΨ›L²»O½χJQoϊε/i“&M²'žx"¦υi»νΎ›zθ‘±|3gΌΣJK‹–§ΙœΙ@@@@@@(V”ρ¨Θbέv”–(ΊhEj˜5kV|;v¬ 80Ύ/-)5Mχ4©χ ŠQPΊοR/4ή~ϋmS—^0Σ³GOΫn»νβ΄yσζΩͺU«βϋd`NΑ? € € € € € € € €­V θ‚i|K,_Ύ<ΎΥ£œ’Α3>=Χ«§]³fM:‰f”OηΝγΈͺͺͺ|“Nΐ@@@@@@@@6!P΄Α4έΊu‹θ7ή°>ϊ(ΎΟΥ‹ŒΖ+ˆF=Ρ̞=;¦νΤ©S|Υ?šΆzυjϋψγγΈΚΚJ+++KOη  € € € € € € € €@Ϋ(Ί`˜2dHάBϋΫίμο»­]»6ύΘ¦ΜMη½ΡLž<Ωξ81Nήb‹-bo4žί‚ μ_ϊWœ¦ΌΫ·o{§ρy3σδ3 € € € €@α ¨Ν§PώjbYj­Ό-Κε4Mƒ^jΣ΅|Ή½^F^@@@hkεΕΊΒ †9ρΔνΦ[o΅«ΊΚV¬Xa‡?άϊmΦ―ξcŸBCΔ²εΛμΩgŸ΅k―½Φ> ½ΟtοήέƎW] οΏΎέqΗφΪk―Εi†šžF0M±~C(7 € € €΄eBjΣ)Ρ†ˆ˜΅+/³ššΪ•±}»Ϊή‘KKΧ%(€V8%) Š€ € € Π&Š.˜F €ιΩ³§ύθG?Š½ΙθρLηœsŽύγ°ν·ίήϊφνk;v΄5kΦΨ’%KlƌφΧΏώΥ:tθ`sζΜ±ίώχ6bĈΈΑ'Mšd?=ϋl[Έh‘ΝŸ?ί.Έΰ>|xœVH.mςΫΙJ#€ € € €X±j­Ν™Y:€₯Y4ν,΅±3±WεΩ >΅.ΫΗόΧ¬­Άi3—ΨΗΛ+B€MMν2[8’E‹―Α>•*lPΏΪΗ¬7-Ή!€ € € PψEL#RrΩqΗc2—_~Ή=τΠCφπΓΗΏ\μ#GŽ΄{ξΉΗΎυ­oY»vνb²7ήxΓώz­Ρπ“ŸόΔN8α„ˆ£€_NœΘ? € € € €-ΰν9~΄ΒΞΊi²uˆΎΤޘU/)΅ͺ%³¬kΧΞ±8Ÿ/_cυΗ·¬€ύ‚πœ§uΑ4-\Π²ΠCΞβeUvπŽ}μ¬οοhΤaN Λ°x@@@Ά$P”Α4Ι ΄σΞ;ۍ7ήhγǏ·^xΑ^~ωe›%λZ':θ ;δCΣ<Θ¦E· G@@@h΄@*ƒT‡ΰ•juφR>@,V¦$”'³(5‘pα©J΅e̜Ψh ˜1ΔΠT…2©Œ@@@Ά*PτΑ4ΪpώLiΝθ―‘‘4t©λCEEE|λ τHγ2Ό"€ € € €Ε+Ž) ˜ωdFΣ¨xq|!QΉ˜…T:Κ‚ € € €ΐF(κ`οE&γA1ΩδtSUUeΛ–-³%K–ΨπαΓc4žOΆy‡ € € € € € € € ΠvŠ6˜Ζ`3uκTϋ裏lŊ1X&3 FŸD³zυj[Ίt©½όςΛ¦in½υΦL“ Ζi;›ž5E@ Vΐχ©ώŠ  € € € € € €mY (ƒiόbίάΉsν†n°«Ίj½·α·Ύυ­υž‡@hTκ―­qY'@@@@@@ς(Ί`€Q/4·ί~{ €Ωn»νL5šΆΙ&›X—.]¬΄΄4>ΚισΟ?uκίΏΏυκΥ+ΊθύA{₯ΙŠt €΄6ߧΞ=Ϋ^xαΣώqηwΆφνΫ›₯ΒΪ–΄Ά5f}@@@@@@hX ψ‚iΒΥ½’πίΜ™3νšk±Q£FYee»μ²Λl«­Ά²ιΣ§ΫψC;ψΰƒνΒ /Œ5σηΟ·gžyΖnΎωfλΥ»—yζ™vψα‡Η€›†‰H €@λPO4zβm·έf—^z©uοήݞzκ)SjjέώΆu9k… € € € € € €@n’ ¦Q †9sηΨςεΛνwή±‡zΘφί8^wΥο³Ο>6mΪ49r€ 2$Ž7nœmΏύφφ£ύΘώϊΧΏΪ^{νe}ϋφ½ΩπX‹HΔ? €mHΐ{₯ωτΣOmΑ‚q?:oή<[Έpa ¦iC¬* € € πs₯τή € € €΄Ib[k|Y΄pQ,ϊψργλ\τλέ»·m½υΦφή{οΩΫoΏΣΤΤΤXΧ]νΘ#΄ &ΨώοΪ?ώρ8Νσ+6Κ‹ €@SxCρͺU«LϋKΨ?Ί―|Y@ΏϋςTΖ € €³ηBΕΌυ(; € € ΠtELγ«ΗRh>|x ”ρρ;v΄Ν6Ϋ,~œ5kV|---»uλfκ‘FΓ³Ο>kΊpΘ€ €@[PP€†d0M[7aύ¨O@Xό―ΎtLC@(.κšj[²d‰­\Ή²Έ ήD₯Υγ~ εΏ/V©6ύ‹ΟuίJyU@@h]ELΣ‘C‡Έ%Ϊ΅kgeeeι­’ ½zυŠŸ?ψΰƒτ…AΏ«dΨ°aΆε–[Ϊ”)Sβc-”Π/"¦3α  € €Ytά¨€νgžyΖζϟŸ%£@@ŠUΰ‘²O<ΡΊκ*[Όxq\ΆΤnBΖ ηΏΠΖ«‘΄€,²Ρ„­6ήuŸc€{α”8–—@@@ υ”ΫͺθV'JκeFƒξY±bEό¬;κΥ Σ,\ΈΠ–/_n]Ίt‰3šO=ΧtοήέfΎ7Σ>ώψc:th±P^@@fπcΠ ΨΟώs»ηž{μΤSO΅‹.ΊΘzτθ‘>Φlζb±8@@ πγ<΅Nœ81ή|χΘ#Ψ>ϋμ“ncάΐEΕμk«ΧΨκͺΥ14₯ ¬ž^4³rυJ[[½ΪΚJTSή―΅e«—Zͺ]MΌ‰Rα?…0¨ΌeVQήΎŠC@@@  Š/˜&œ˜θ$iΰΐqυυ―ΩάΉsmΤ¨Q1F#ύ1O―ΌςŠ}ψα‡1˜ΖmtW‰‚l*ΪW€{­iG²@@hσζΝ³·ήzΛ>ψ`{υΥWM=!*˜†@@ Έτh'έ€Χ΅kΧΨ«u[yΤ“M_4Υž~uiΧέjRΥ-Ύ1υΠ$ΟT―©Ά©‹¦[—Ž]­΄¬Δήϋθ»cςMVή‘άR5j'nωA½η,]ϋ©ν3τ ΫeΘΪ·ό&‘ €-$ γŠπΨ‰·P2ŸΒXΫICͺ¦ξD•UCA•9IΗ?α(’ ¦ρ» 03nά8ϋΫίώf·έv›ύπ‡?Œ½ΜtξάΩ6έtS3fŒ½ψ⋦`›~ύϊΕi–.]j=φXμšο½χŽι d;P @@"Π Ά7ΊzυjkίΎ½UW·ό…†"`£ˆ € €@Α θΒ…Žυt|·vνΪx!£ΰ έ„¬©²₯k>΅R++Œ`]L Owͺ^[ckk|{„ΰšPΞεk–ZyιΊ`šΈΨTZRνͺB―9  €΄e‚ ‰Η ΅ ν+ΚⱞΆOMΘ­hW{‰\ΑΤ3¬;Q˜Oβ A %Š/˜f]8^ώύνΫίώv ¦yόρΗνΞ;ο΄λ―ΏήŽ9ζλΫ·―rΘ!1˜ζ²Λ.‹Ο8ώΪΧΎfΟ=χœέxγΦ5<φiĈ6`ΐ€h―ʝ@@|Τγ‘½r™i@@βπ;”‹£΄M[JΧ–•Ά³²β736νΦ/·ThMΆ₯5!ΐ'|¨½y<–­,”…ή`RαΪW!΄κͺgš²’v‘Όt1nύΈI €@“Μ]π™}πΡ2+/+Œ}b<|Αeeεφή»Ζ^uΌΧ±C;›φήb+―ό0RWΕƒŽB8¦¨ͺNΩЁέm“ξb?\Ύn’―%™ °AEL£΅UE§HΑΓ;ΜΤΥώyητlγX vμh‡z¨έuΧ]φΙ'Ÿ„ιηۚ5«­²² σtΔGXEEELΟE ϊ13 € ΠfBη΅mvέYq@@V&Ϊ\΅R(ΗΈ±4‘LΙ#ξXΊuε”~rZΛmuf±\-W –Œ €@K θΪ¬΅>χΪ<›0q†ιzuΚxRKNΛ ―₯Λη… šΨ+M‡Žvχγ3­ζ₯τ“ωό§(§y*Βγ,Ήx=υ}m]0M­i ‡E"€@B (ƒiT!«bξή½»qΖΆσΞ;ΫΓ?{›ρi[nΉ₯ύφ·Ώ΅+―Όzθ!λΩ³§)ΨfΥͺUvί}χΩn»ν–`ΰ- € € € € € € €@c:„G)νΪ«½υλήΞΤΛJ! y- ½Θ­ ½ο-XW$ΕΏφθ\f•=ΤιBuAτΚW‚iφ`zC!t•S2 ΠΒELγfz~±z›Ωwί}MqjםOŠ―»μ²‹έtΣMvϊι§Ϋ|`]ΊTΪ[|Υ† ’Λbz₯©CΖ@@@@@@@`½€²&Ρ(fmΣ¨»’’”U':ΚQœŠ>«œκΌ‘PΩ1 €@a ]0wΆpαB»ώϋCpΜ1¦GiΩd€Μf›mfϊΛ<ŸΜρ|F@@@@@@X_υSxC!–)S©ʘYf>#ΠΪΒΓΰŠs˜:uͺvΪiΆώϋΫk―½W"[τ ΖωψδϋdΐMq Pj@@@@@@@@¦(Ίži<fζΜ™ΡBwΚΦσŒCyz}NΎχιΌ"€ € € € € € € € ΰELγ―¨¨ˆoΛΛΛ­]»vρ}MM•–ζίΩΑ5Ι+ € €­€ϊΦΏ@@@@@šD Θ“&Yά†gβlΪf›mbf/Όπ‚½ςκ+ρ}YYYμ}FA2ωόmxiΘ@@haiZx°x@ _E?2 € € € €@1]Ο4ή›Μ¨Q£μ‘‡²+ΈΒ.Ίπ"[½j΅ν°ΓΦ£GλάΉsΊ·š\A=ΨtθΠ!ΧdΖ#€ € €/ › Φ]kήsgΑ˜"€Τφ&F $ί@haOϊ5·. ‹G@  Š.˜Ζwξ‹-²yσζΩ[la/½τ’}ϋί·>}ϊ؞{ξi½zυŠ2ΩD³jΥ*έx^Nu: €|Y QΑ4λσεΕΦӘΖ^=Κιλ_z썦ͺͺ*f¨rω_έ%|ρIΛ—-·­ΆήŠξλΎ`α € €™ΐgŸ}f[o½΅ 0ΐV―^mϊ¬Ηή2 €Ά@ϋn¬²žΆfι«μ·£•Utq6ιξi »π”@Z…€žψπΐؐ!CμΘ#δƌV±UY @¦Xο`όίύŸ­\Ήr½£Uθ’`EΊvιΕzχξmέΊu[― Υγξ½χήFy€Β@νΚΫYyωz―~£–ΗL € € €ΐΖΠωΩΪ΅kΓ5XžuΏ1|ΙhZΠσLͺΖΚCo4› ίίjjΦZYy‡¦]Ή!€ €9όΌqωςεvΗwΨc=f³f͊½œξ±Η9ζb4 €mW οhίΙέΎύτ§?΅§žzͺΡjŒΡŠx3fŒι‘KηΛΘ'cδl²Ι&ω$%  € € Π*tΕ€ Pd €,-³²R5Λ΅Αz<¬A­΅φ₯αΐΜ}jϊs‘•7|kšΊΧπ"ϋQ\@ XΆl™ιOΓ‡·:Δ^N•%ϋ— „ev@V'w0―ΉNvυˆ₯#FΔήe΄s]Ίt©M›6-φτ±cǘT=Χθ.Ι―~υ«±χέ-©Η§NjοΏ~ό{φΩgcτλα‡nW]u• 6l½j'΅a‰ΙΌΖλΟΗωkzΣ|― € € € €4Ÿ@A…”4ίjλ1νΝ·΄†—Κ£‘4τbνmˆ©ΠƒPiYm³iIθe›@Φ$ΰϋ;]·[³fMbχΕu΅Φ΄Ύ¬  €Θ;˜ΖwŠR=οΌσμΡG΅»οΎΫήyηΫύνψ㏷Ÿ½σp£8Ϋ{*Χ|έ½χ†qSL΅ 8„l:PB„H€@Ύ?IH πε#ΤPLο‘ƦؘŽ1ΖΖ6ξέΎβrU'ι?ΟθFή“₯;ιNe%=cλΆΝΞΌσ›έwggή}gȐ!ΪpΒΤΤΤΘκΥ«ε½χή“W_}U[Έyδ‘rΕW¬^q|ήΌy²`ΑyρΕ₯[·nrσΝ7KEEE+C˜φ fδͺ–5kΦΘςεΛeγƍ‚mΈ₯;όπΓ₯²²R>όsΑΤPΤ «±M{yπ8   t–€6τT‰Ψ‘ΣXwΩ>ΎTΟάΦ%ƒœ>"ΨG^t··τq·˜[$@$@$@$@$@iE ~ΫinlPήyΤ GΘϋH* œΚ`fΧΖMβkφhάω]€Ύr£ΤmZ-^ŸΟοqV6έz‹3—S„Y™pH€H ctŸehaΗ’βY$@$@$q’6¦AΙρPuΉ\RR\"~ψ‘6€ωΏϋ?9ύτΣ₯gϞβωRρΟ<σLyςΙ'εͺ«’ύφΫOŽ<κHϋ¬6Ψ±F™0a‚|υΥWΪ`ϋα ‘ͺͺJ¦OŸ.?ό°>2΄—>‘H€H€H “Φlά.;k›ΤΚ΄“iuϊτ`»uλ6©klΦΟνβ’|œKVVŠžiRlΙ‚μ›½~ιZV ½»i‘ψ‡H€H€H€H€H  8ΤtJ."ΙρΩΐ˜F‘„—œ§šΞΙςώ“£ε„Œ)ƒ TvΛGβCwEΠ4Ύ(:     ΫC‚†}†—v ΅1vέΊuςώϋοKyyΉόδ'?Ρ†4>εξΤx₯ ½ΐၦ¨¨H{1Ν›oΎ)§œrŠžn ΗΚΚΚδ‰ά{ @IDATˆ#ŽΠ^dV­Z₯ ^`€Σfΐ;¬zgœ5k–6ŠQ¦p‚‡œΡ£GΛ?ωOmLcdΒρΫn»MΡμ΅Χ^rήyηΙ¨Q£dςδΙmfΓƒ$@$@$xl½4kΉ<χΡFι]βΦΖ!ρH·Σiδ8ΔίX#uΥυʘF€¬΄‹Όόή*y}ρηΚ:ΦΧιδγ‘@Λ!‹·5Κυ'•S΄`γ•i  @0t‘}YŒ„E'ΐ’ς8­~ψg‹QŒ€¬ΆBh9# k1)     DK€οηΡ’bΌTˆΩ˜fΓ† ςε—_jC”Aƒi™ΫΊΘυόΗ*VII‰Œ?^ή}χ]©««kUΦΎ}ϋκνΚΚJihhhu,tΓx’Ω΄i“<ψΰƒϊπ!‡"Χ]wŒ3FoΏόςΛziŒizτ葧w7nœ\ύυϊΨ[o½%p@ΠHοδ  H<·CϊΉ€ΌΠ)^;|وώWeLγQΏϊ–ρŒΝWr–*Ε―vΆμO’¨’u©―C;ō―DH€H€H€H “ΪκΏθd<H &ʌ¦Ε»uL§%4r¨‘ŠΪV2κW§„ζeβF<Ϋq‹R~F#    0vξά)‹-’ΌΌ<S ,Hΐ.b™‚\Ę:)Τ(&l‘Z^τΰ…f͚5z §ΠΞ+³m–aΣ ΩΉbΕ yώωη†8—\rIАΖγρς²Όœ;Nνηηό\Β4P[·nΥλφ{y·JΟu L%έc~™ZF–k7τwzΥ―rψb‹Ÿ’Η§δ ΅λΑcϋΥΜJΆΣ§lΆ‡“œέ•Ι΅€ΐ°…mώ©{φh7vγH@§ΫAj%/  @x›7o–υλΧοΡo>6χ’  ؁€žα {θo>lπΥ‡ P   Ψ °;³ŸΎΦ—^zI;ρ˜:uͺ|πΑ:Η=ϊ`,“'Hb6¦ΑL˜JiώόωςρΗλta#3(l–ΦιŸ`QV]]-ϋοΏeθΘBθή½»κυHŒΑΝκΥ«u”c=VOΩ„ δkΌΡXΟΗ98†ε Γ‡—οΎϋNΰe‡H€RE:ΙόR%σM.»oGn΄?Ή΄Ήd±“D© y* iŸJ_;”'§Γ©Ϋ•XGϋΫ9κΈΦιvZΙΛ@$@$°›τ5Β‚ δK%ηŸΎΜ;Wο3Ζ’zƒH€H€lIjήΟρ3:έ–‚R(   ϋΠο…¦ΟΜ,ν+n¦KfΪtpά±xρb9τΠC₯ώzόΎΉΉYχ³f:–/=Δ<ΝSΏ~ύdβΔ‰Ίγιή{ο•Aƒik±HF,ΐ+²»οΎ[ΑΤJέΊuΣλπ³eΛyϋν·υφΰΑƒ₯ΌΌ\―c0’­—OπLSTT€ΧΫ;‘>Œv`άγρ4ισψ‡H€RA ¦¦F{Θ‚‘bqqq*D`ž$@$`K^ŸWv4ΤΨͺ£ƒ­.—KΆΧW‹Λιώg`@S]W%Ϋͺ₯YyFl»υš<Τ2.Ξ/‘\gnς2eN$@$`cθ+€ΫyσζΙμΩs€wοήςή{ο驟αyxΡτ'ΨΈˆH€2“€jΨ:ΥTΔ…ω΄Ν#ΚΒ] ΞW. ά3³Θ,   @bδΰƒ8ŸjOδδŠ_/UOΖ’Ω¦H π(S5Ž9`@cφ!π]έα2•’6¦AΗ.ZxŽ9ρΔ΅qLee₯\yε•rΡEiγYρ0 ΤΆmΫΣ)ΑΫ'œp‚4nhh/ΏόRž{ξ9yχέwυ±8PάnwT7Gnn`p`ϋφνΨΨ¨§‰΄σ§ͺͺJ6mΪ$0 ΚΛΛo'6ΗJ€Š-VbŒŸmΜ=O]·ί~»<ρΔςσŸ\.Ύψb­ΝρlγΒς’ ΐ`ώhj›vΚ£ŸύCκ=υA0©&ΩN‡lώR σΊh―ŒN‡C^^τ΄|°³TmCφΤψΤ{kεœ‰Λ C£jW§^jJ@$@‰#`ΪΧθ˜Γ†4ψ Ζ5˜&šs±'Ž=S& xΐW =T[\υM«6wο1έι‘&`™   dύ‘\~™δ•”ϊ­ŸI^ΩhqvΧύfY„Α–E5φxw7F4FP~τbHp™JQΣX…<蠃䁐 /ΌP  ηœsŽžϊiόψρRRR’•ŠΏψβ Y±b…ξ¬ΪΈq£<ώψγϊΛ/€…©Ξ>ϋlωώϋοuόkΉFyΈ9КMΨuӏ4οΏΎ¬]»VφΪk―ΰTS‘'«6œϋΙ'Ÿh™`Τo ρ%@Ε_žL-s @o½ψβ‹2dΘyζ™gδΈγŽΣz,sKΜ’‘ @τT“MΌκ ―ί«OBΞ!G½Τω_FhyTΗ>Ά!':φS”<ύβΩ¬ “R.  0οͺ0¨!   @zΐ«@^Qž ™άO hsΫζύ =RJ  ΘrͺΓL5(œξι:όhiκ;IάωεκW’φ£ŸOg  b2¦1Φaωωως³ŸύL£`ͺ§5kΦθωΜ0§Yh€§Lν― ‡~Έφ<ƒ8πZ³|ωr6l˜œuΦYΪ»Mii©6¬1\‘iY·(γƍ“O?ύTώύολyΤ0M ¦› =ίLAυωηŸΛ#<’“;v¬žξ ‘ρuώιx Β—}¨w ΘΰQ Ζ‡Π?Π]ψ*–H€H @―°ΠυžjΤFŠΡ:ναυEΛc‘ېӗcŒlR'«α9H€H€H€H€H SΠx&Sj’ε °;θ[έ«‘ς~»“’|ιJΐ•_*ξ‚2m\γ·~0—’ά$@ '“1 €Αΐ¨0–€'…}φΩG,X §s‚·™šš§¬¬Lϊτι#<q&½ΐΰλ]‡κΰΗ±Χ_],0ŒΑτQϊAεCzΠ ArήyηΙW\!χίΏΤΦΦΚ)§œ"={φL!…€―Νvνά%UΥUςΩgŸΙ=χά#[ΆlΡǎ>ϊhmxKžϊDώ‰H`φμΩςδ“OΚπαΓεάsΟΥΖJδd9θRθ(ά#ό26Λ/ŸH "Έ` ³Œ5ρ”ΠΩ‘’`[οŊξq ‘xΡL:g ­nφqI$@$@$@$@$@$@$@ €>ι@0K³Ψή}Όe?$iΰ]:ΨurdZYY ΈˆΩ˜Ήƒ,1έ~0N ŒZ°Ζ1π`cΌΒ`πAŸeОdϊχο―χαO΄F&οάά\9ν΄ΣδΫoΏΥSN͜9SO#uΨa‡ιiŸ0ϊœ9sdεŠ•2γωςΞ;οȘ1cdι₯rΧ]wΙΎϋξΜ›+'`κmηΝςτΣOΛΗ,>ψ Lš4IŽ8∎'Μ3BΐΤWBg’10z1ζy    €­ °έmλκ‘p$@$@$@$@$–€ί§¦ ΄‡’Φ;ρ1ΓιT3Ϋ4cPN—E{·Υεφ‰OMΉέbV“ΦεlKψ5»…Δl+ e4Φ}FW/ G Π!c#‡΅ƒF30 Α/\€LΈ`“#wŽ1¨šλ―Ώ^ΚΛΛε–[nΡQόqνιϋΰω¦ͺͺ*8Σ’E‹δψ‡žV SYε—χEO ΎΎ^0χ|=dτθΡγ„Xκ5ϊά3VΈΦρ3Ζm±žΟψ$@$@$@$@$@Ρΐ{Ό:UG5     @P³*¬yηi©έ°\ξ|ݟžRΗ(b`Œ rg4ΦlΥF%Ε=ϋΘΖωoΚw»+cΨ„Λ‹1#{EWEΒ;š―±VϊuΆχΖρA{Υ₯! Ϋθ1Υ₯±±QQ`_€`5¨@Ό‚‚‚`§’υX€σΓνΧ>e%;`ΐωύο/ΗΌΜš5Kζϟ―=ΣlίΎ]Ίvν*C† Ρ^p?όp™:uͺ6τΐUΦ2„KŸϋb#`κƏ‡F4±αKHlsWWWΛk―½&Έ'Ž9ζ}Oΐ='Ε !ΙDI€H€H€H€H 6ί|σΌωζ›2lΨ0™6mZΜΣΗ–c“     Δƒ@CΥF©]»Pœ%Κ{‹/IΪ. ‡2,©―o_³GΛ†šΆo‘ΪυυjάΜvβΖM ‡SΌu›ΕΫtJKšj87bLˆH€Hb6¦1τΫΆm“Ήsηκίƍ΅\ΐE 0Άp¨_uMάyη2bĈN΄/EEErΠAΙΔ‰΅G”]»v ~Θ³ΈΈXpΌ€€D05‚)C$YΉΏcΐΑ,;– ϊ7ά§gŸ}ΆNΣ°]y啃2     ψ0]±uuuzκΫ;ξΈCΰυƒ>ύφΫ/~1%     „ΘqεŠ#―Hy¦)Tγ‘Η»’y’…1†4jό Axλt‹3·P΄cš$Ι‘μl`Lγo.QεVΣ<1  @”b2¦1F(›7o–?ωΟrχέwkο/•••QfˆΦΠΠ WLz±œl΁— L%Τ»wο —xΌΑS … ζ\γE%\@, 1‰ΞΐuŽ©·6mΪ$γƍΣχκ–-[”Υy½6¦1χCgςΰΉ$@$@$@$@$@-πqjƒγ]Ÿόπ‡?Τ^S·nUξΣH€H€H€H€H€μOmzΏ7`H“Ζ4Ο‘sZ MjtΩ1μ_Q±Kθχ«’f`ΖN‚g  ΔB &cc„‚)c`H³οΎϋΚ²eΛ䀓N’Ύ}ϋjΟ/&N8!p ωπƒΠVάpηcŸ1X±b…{μ±rν΅Χʁ(C‡•ŠŠŠΰiΖKŽΙK³ŒΔΘPζ>1Λ&O“455ρΘΠϊf±H€H€H€HΠφc°Ό{’ύipH€H€H€H€H€H€H€H€H€@ΤΖ4fP_Ψ½ϊκ«ΪΛ¦NzξΉη΄Χ Έ―Ζ܊Ρ¬jRΡč„Ζ6l«―ΎZΰιζœsΞ‘C9D&Ož,ΤS;™s!»‘ίμγ’2™ΐχΗU2ΉΊY6   ,#`A*{ ε’{w0m@μgH-cΨd–©•†Ή“  DK€ΠhI1      d>¨i ŠΥ«WΛͺU«S;]tΡΕ2mΪ4s()KΣAά«g/ΉόςΛε­·ή’%K–Θ£>ͺ£F’Osγ7Κ\ ŸώΉ|πΑςϊλ―kڛoΎYπϋρ,S§N•ƒ>X†½ι˜σiTJ–Ϋ$@$@$@v"`Ϊ,v’‰²@6€] ΆUΧΙ―/QEχλΑ53ΰ–j‡SV}·Rzt+Ρ^8wΥ{δρΧΎ•²Šj1Sή¦RFcΛΣ¬,~N;z”τιQDo‘©¬ζM$@$‘žYψψΘ₯,U†©~iςϊΕ©Άu›ΌΕX5b<@$@$@$@$@$Π1m΄΅9–ί1€<+Ύb6¦ι₯‹”••KAAX§kJΕ€Oqq±μ½χήϊwI'Ι…^(_}υ•ΌφΪkz**ΧΰΧ»woωᨽΥLœ8Q† &n·;Ύ$™    Δ‘ΐΝ;eξάΉRWW§§²μΡ£’γΘ—I‘@ϋ`6“# ^™συq˜FYΧ„zj?ΕPΖ4M[w¨χυJ§:š›}ςΕ*q8Υ6\Σ€6ΐ ƒ“;šόrό”‘©†Ή“  @;Π‡_’ο’ρ½ΊΘ‹ΛjdtYžτ-ΝΧΟΨvNεa    °?6ŒVR#| ίM΄f‘m~7Λ±ΤΘ Γϋ@—ώκ.UB0ί”ˆΩ˜¦ώ2~όxm°²rεJιΩ³g°C9Ω5ζ†BΎ0¬Ωύeί}χ•cŽ9FΈβ ™7ožΜ™3Gf͚%<ςˆώ!ΌΥΐΈfΚ”)/LR†Ÿ“   @kΖβm˜γ?^Όχή{΅Ρ°uκΚΦgq‹H QrԌ±ε….5­RπΥ9QYEŸκHΘQΖ4Ϋs² λκ}HΏ©Aΐά.0±ΗΥaοΓΥ(G»œH€H€H€H€•lQΠ¦mΪY#ώ{§δΈ‹[>ΐJ½• $€#γ:HCe­φFγvΈ€vΝ·²qφ*q©cFώ”ςW2yk7IΕΔιʘfR Ρ”JΔΜ“L &cΙ`PgŊ2lΨ0ύ΅τ₯—^*O=υ”LQS&υξέ[`¬ɘƜ) ϊτι(κL™Νΐ’5ΟΪ]΅²lω2ωόσΟεύχί—™3g³(,(ΤΖ@ηœsŽΐΨ†H€H€²“€iŠΞ0απΓ2ζήz»ŽSq]„R7ν]/©ˆy’    @ΠνρΠFy•ŸE%   Θ<˜*άU_r\ωͺp¦?΅εΔ„CΩΈšš%ΗΉNu7ι± ‡»@\E½ΔεTγ*RΚ›ζŠ]Ž«@Κ{%CvˆΪšΔ|!―£oΈαωϊλ―₯ώ²χή{Λ–-[䦛nŠšΰA7ccDSWW'kΧΥsf͚%|π,\Έ0(ΣΤ©SεΘ#ΤS;AζςςςˆF?Α“ΈB$ΦƒΠx$§όq«9κ‰!ππW £S `3`Ž%ώ™ω"ΝρΤ,΅°6‘%52;Χΐ}‘γt«bͺkηQΏΥpu·\£˜'ΔχNfΧKG$@$@™E Πζ΅W-Ÿίhƒ·΄oL»τM[<υ5ΰfϊ9R/%    ΘTμυΛԚeΉH Œ;©ώρΒ,Β―‘ί§η+ δυξξkVΫ-•jiΥΘ‰ψ½Š–3(&W²ˆ@ΤΖ4† :u0΅ŒWκλΕ«.·Ϋ-γƍ§SYg©γαφΓΜ'Ÿ|’γλ8ᣆ;=μΎ&O“lή΄Y>ϋμ3ξoΌ‘|LdΘtΔGΘ΄iΣd―½φ<•H ;τQ'MQi™qά.w°Σχn5$¦Κ³O°;ϋ0Ι0IΤ΅ηtw‘’žd{cΈ‹GH^i?›4§3Œ5‹C$@$@YA Ε((σζΝΣS9Υ+ΓΞ8㠁'šύχί_†*]Ίμv½dύβΜ”ΙœΗ% @fΨ°}μ¨―gŽΣ6ΖΠA.—SVmϋ^Π j\N—¬άΆLς·9₯ΉΩΫ!ݚˆšΓΧ»}ΛϊKq~‰pθˆΞN„\L3ώŠϋL”‚ΓΤΌ€yΚΈ¦ζΰ*ζğ4S$  Μ%΄·«φχFΥ·O<Π‡ΠΤδ‘m»6K—Ό"q;άRS_-‹7~#]Š υGBvhλϊU,ϝ/+†Ψζ s―X–ŒH€H€H€²zϋΠnΟu9t{³Ω'Ϋ=~ΙwŒjԦڟ}\Xb  ˆD Mcšp†'π>3jΤ¨HιΕ΄Ώ#Ζ,ƘfύϊυzΊ)Κΐζ¨£Ž’)S¦Θ‘‡*Γ‡—^½ze±–ΓμμHήζάT.Γ•%΅ςWVΉΠ i ™­ϋS)ηξΌ•tl ξΖ‘αkŸ¬ž+sΦΌ&₯Ή]Υ¨p gƒ nŽ5Χcυχ»ΤKKžv7cš.{Y>φΌ£άΕ!BͺεΔΛ“Cv5WΛE“ΣΖ4ϊσή<©˜„ε9Ss +τΛ4?EIf&L$@$@Mο~xΟ^_½Fξϋβf)Ον©½ΐ »ήA9€‘ο·n”’‚"e@œ#kjVΘΣ WΎ#θ12•r’ύέΠ\'ƒΛFΙ…ώZ#₯RζM$@$@$@$™T³]Ίδ:ε°Α₯RΏΜ'ϋ•δΚΰn…viΆg&t–ŠH€H m ΄iLcgƒ#Ϋ\ “'OΦ?xΌ1ϋQ#Ɛϋ¬λi[[JpkωμP3ΆξVž6Œl>5υ— Ύm(―˜Q†δΘuζJ©»›ΉK΄ΡJςrn#'υΆβpͺŽr—™2`|–οκ"%ξrρ9μaώουOyτA§~ €άΒ§ ¨<~Œ0₯ή’+Ea$`j°ΈE5šΆO‚3dς$@$LkΡ‘ŒtKέέ₯Ψ]ΦbL“bρaΛ£„σε¨)ͺsΆιΎψΠqζΈτ{‚ΫνRNω`ΠnJyρQJnN8Υ@     $˜ΐώΛdLοbΙU~ΒSWΗ'8S&O$@$@iF ’1Ηγ‘ΖΖF]—Λ%˜ή F)πƒ‰Ž†‚Β=?yGΞ7ƒ0œ™?ΎŒ1BΚΛΛƒIE2š1ηUVVΚ’E‹dεΚ•rζ™gͺιV""¦i—•ΊTΥ4ˆC ΰΐzΨ.Α‘ΌmΫΊSš<ˆUVΝ…Ή²­¦^6lέθ”΄ θυ©RΘV^Έ–γ*–ͺU‰*Zΰ«P\(ζžΠεUB·m$³ξΆN@η5Κκ―6€ρiCΈΦzΗΓ…ΉΤΟ «WΙ¬λΑT¬η@Ζΐ=Šμ™g*€v)%fž$+<ͺr”Α°OΉ@vζͺe“OHςή‰•#γ“@ά¨φSλUάRξXBhίAQ„΄σvo£mή±€q–Φ^ hƒγ]νHσK„μ1§©Έ£ύf:'¬cZSό°Ž© Κ7dΰέ%Υ‚€σ&   H*’<§nΫι=%©˜   ΄C`KΙ8”΅Ζ’%KdζΜ™RXX¨ VN?ύt½6?ό°lέΊUάnwLΏ0hijj’Ÿόηc #—vδ 6ρ{φμ)ψ!˜ΑgΓΫΦ΄½^―¬[·N,X ―Ύϊͺ<ψΰƒ2cΖ 9γŒ3‚ιΪy~Ue«ͺδšΎΎ₯niΖ—s©FυyΆ·q»l­ͺΧ}ΗέΊ–ΚΛsVΘ»Λζ£rR¦xœΙ­ι΅Υrα1Cdϊ΄Q­Έ`Δ΅—„β•H@‡;W ς9•Q“2~Γύ¦88\n Φ³"˜kΠF…5·Nk‘Βοm'Ω[v”)Ω ˜  μ&€GJ‘r}\>¬X*Ώέ.½&u•Β2e€ ui―†ΐn‘ΉF™Lΐnmp%ΪΫζ½νπ5₯'BŽqk₯·2Oϊ΄$ν"©‘Γ,3a I€H€H€H€RKΐ†]η©ΒάI€H€H „ΐΖ4ζ8<ΈόνoΣ›_|±œzκ©Ϊ˜¦ΉΉY^zι%yχέwMԘ—'Ÿ|r‡i¬™Yh¬4¦γ²ͺͺJ–.]*³gΟ–ώχΏ2gΞ>|Έ‰S€c€ωƒω₯\~έW¬:ϊΰΑB"α+ΏέγIΪTC „ΤŽ0ε(9œZ#Oόθω•±V}εFe«oM©-ηξR Υ<žfiάQ%|ε*\Οxκ€vΣjq—”ŒΜ΄υΣξ“R΄†i”χ«Š^ΪΨ'EB0[  h›€zΔc‚<{Βp©Ÿ  k Δ] ¦aοSΫμx”D ‘z‹2κ―WmZ›‰+]€χΜΪΝ›ΔΧμΡ₯vt‘F%gύΦυΚS¦¦TTο.ξ.%’[ΌΫΛk‚ͺ‰Ι’        @ZˆhL―30>)++ ~YC•ή½{λύω(ŠŽσΠ±ο0Α©•:iw€4­F4F LατΙ'ŸΘΫoΏ-=τ6žιΥ«—@ήeΛ–ΙΰΑƒeΚ”)iiPƒΝΟޟQ‰πΚ¨L«© δ`Ώ©›T-wKGy0p¦ ΩάP+˟?i­V_ͺϋFΥ€ͺ€|Q(©W­T.RezŠΪM+eωs·JQS»9#ŽX |ψ< ’ί}Œ:σ:qζ¨tŒτK’g‘  @’ –ηK—Š‚@;Τή]`¦Ov$ Ϊβkg?+Ϋ—Ό/Ξ‚25‡OꧧD+Φ©ήΆΦzΔ³³ZS+,―½*K7½'^₯/RήώVRαΕ³cτ9κι;ωGZ—™RμXΥ”‰H€H€H€H€H€H€H€H€RA`cΣ‰6nά8Y±r…6’Θ/ΘžΘ_ϊW‡š€›jtF`όΓGΏ 4_BV#/<ιΐPζƒ>ΠF40€ΑU”Υ«W+§N9οΌσdςδΙ2~όxm„ι¬-€6l8π³GbΔAη°Y7K;Θ YΤδ_ E•:GέJeHƒŸ *G™™©²*Ή ‹Άj2E‡¬ΚνΌΊr’ΖRp;”GŸœττ•Br̚H€H …όΚZ^ωτS4ŏb`Φ$zxŸtδΪΰb‡φdKΫΫaή;”‚€‘=δTmpΡϋm 4τ»‹š’Ξ¦=©ΏŒ(         „#ј¦ΈΈXπ 0@ιΧ―_θξ€l[hL†λΧ―—ΉsηΚ¬Y³δΠ»‡ "'NΗ#&L[o½U`4`ΐm`cΞε’²‘@+Ϋ;ΐx‚S°Αΐ‚xP  ϋP,>΅μ_M”0σ ΰ>4?cΎ’ΚR½`–Κ’F‹£·ΥŸΐ ΈΨ·;FJδ…,–Œ­λ–έ\%        εŸA}Xj»ζδTΜΖ4&[cΈbΆ ―rŸέΨΠ¨R|>_»ΰL:0Βill”χί_F%ύϋχ Ό9ίκ…¦¦¦&θ…ζΏύ―ΌυΦ[ZLLγTU₯¦΄QήsΎϊκ+ωΕ/~!7ήx£tλΦΝCηi6L™Μ6—$Πq¦ϋK³ήρΤβs¦’#œ—½ΟΘh–ρΙ±C©@ˆΡ!Ωy  Δ™@67JγŒ’Ι‘ d@RωŸlΥΎM]ΑM“Φ,&+-ώ1wοTΆΪHΎΐ6α•ό‚3G       t βž“t@”4½^op£€ej£Œ"Σ„š˜Α†ϊyςΙ'eϊτιγ³?\™pΜL₯΄vνZyϊι§εκ«―–… FmLcδΐ΄R6lO>ωDή{ο=Ήχή{•g<ΟqΔrτΡGΛ_|!Χ]w””” iŒαIϜΗ%    Ψ‡ύΨ§.(          @’ ˜^qλ’5‰¦>}cϋQ[[+oΎω¦¬Z΅JŽ:κ(= Pψ32{oDcšΆŠύψγΛΆmΫδK/•’’’= j†+πF3oή<Ήλ»^db ;vμοΏ^ΠΌςΚ+2{φμVIœqΖr葇ΚΑ,C‡ΥsΆlΩ’γ@c-eŒzZΜ    °γ)ΡjόkmWATλ1‰NQH€lC€] Ά© B$@$@$@$@$`ϊp:ϋ:‘§‘  €˜Έ"ΝjΩbbjŸΎiΑ-ΡB.Z΄Hf̘‘³ωΥ―~%·άr‹ΆΓ@%›Ϊ&1ΣPyyΉφόβt:ε—Ώό₯φcΐ™%ΘY³Fžzκ)ΉφΪkeĈ ¦š’ΠyrΘ!rόρΗΛώϋο/#GŽΤSOιƒόC$@$@$@QPήνΌΚˆΖ§άHͺŸ]Œiπ/Τͺί―δSkτΛ„εP0N2WΤΰ=xω1ε @ς ΰCuε—τ•#O“Ζ]›€K·‘Κ›•r΅Κ‡δΧs$    80c'»vν’eΛ–Ι~πY·n¬X±B&Ož,/a   L# =±·ŠέZ™V»ι_ž˜ŒiP\cPΣ½{wΉκͺ«4Τόυ―Υ^cϊυλ§χpΐς―ύK»ΐ‘±XŠΦF'’ώwάq²ί~ϋΙ’%KdξάΉ2gΞmX3kΦ,ΑαόσΟΧ–Ω˜ώiΠΐA­ŒuLΎ¦!ͺOΰ     Hkκ#5½SQ―1RδίK½¨©Žu? άΊJ)<     (Λ€α ΌφϋΌ6Ύη     δhӘ¦­\ φθΡC~σ›ίˆΛε’›nΊIΞ`.Οόγ2cΖ Α β#šXZœjZƒΎ}ϋκί”)SδτΣO— hCšW_}UΦ―_/=τώΑη§?ύ©μΨ±#˜/dC0F@f]οδ     Hgζ“x―β΄k6©Iγ>Μ,ba‹γΩ€Š(    ؜€›ρΪΘ‹Ν‘Q<   ˆ;6iŒLΈ\Ν1Μ\sΝ5RXX(χέwŸlΪ΄I €’’"xZgέš†# k†pqxΑ½ΥΌτKΪεαΝ7ί,ƒ qΎϋξ;νΙfΨ°a«W/mτc„ŠΥ¨ΗœΗ%  Δ›UβM”ι‘  @6`‹Β΅<ΑdΖT‡eΥςQŠέL%νήΓ5        0φ0¦1F&UUUzΪ&xv1Ζ,ζ$λΗrss倐/ΏόR>ϋμ39χάsΕγρθy<α•Αί ώ!C€¨¨H§kφ[Σ ]·Ζ1²”——ΛΎϋξ+ϋ쳏œrΚ)rΕWθ韞}φYωφΫo₯° PςςςdκΤ©rΜ1ΗΘΡGHƍ«~γηZΣ ΝΫ±ΨύΕεξ΅XΣ`|ΘB9j¨EυαϋُŸ…•Ο"“   d&œΰϋ3ή9Υδ[βWΨδΣRψ§εmUU Ϊΰ¨        ˆD ’1ΝβΕ‹εC‘Γ?\jkk#―χ755ι)`θRZZ*'N”ίώχ‚i—ŒWt$Β0η›oΎ‘?ωXF­;υW{m¦ήϊ 1‚1F5Ψ†Χό&Mš$§všΜŸ?_ή}yψα‡υΙό±ΌρΖڸ怓N’Δ>φε'»&Zη]ε.tΙ¨–!‡τ•Ό’ά–ŽόΦρΈE$B[SΩI₯& τ%€WίC‡VΘή}Š₯ΐν”"e°αUνΏέoΗι[Ά΄•\ΑχϋόR1°Tφ;{Œξ;Θ/Ξ φ!€mΉ(8        @ΒD4¦5j”\{ν΅ Λ Η˘ΖiMΟx«Α4S={φ”cŽ9F¦L™"gu–6¬Y΅jUάσ7rdΧ½’~qΊ ₯¨χΎ²cm½Έ»Œ–ότ€’β ƒ±Ned6¦W‘LήZ/;›Όrΰ€RΙw+·σ©Mqν¨μUΈςœβΞ/ tβ³NR_'”€lF€ƒ6«ŠC$@QΠνpeΐή£(O5ωΰE5Κ-)ςKσt>0a        T@ΏώΫζ ¬–γΔ$ΤφΒpNl2€Ysφ0¦‰%3γ Ζ,£=ΰ­π£=/–xΦ<Œ|˜ FBψ544ˆΣ©\u0č@qο RΨu˜š‡>WΉΠΞΗ§*νΔ^ΐq>yŸκΉοWV ΠW [€Œ7Ψ™o£ΚΖ3ψΌMlT)…ROΐͺT“I΅™?έ¨N½x”€H€H JτF%¨$G£M’3;       °΄!ˆu@ l¬$ξ΄Θβt8Υ83Ζϊ1Œ™#.gΐ¬$Ρ6I,mTYuؘ*–YF•c "ω 3ώ9r’Ÿ―Œ=βJœ]ω% :ŽψΕ•mgƒa‘2’ώγΗ—!™ s-¦εΐdI€Œτ6¦εCπ¨ΆjZ‡ώv΄LΣ8ΒΏ$@$@v'ΐfžέkˆς‘   €σΡlΊΙ­ΌψΤΖ1gΆ]$Ρk©agjΩX.  Ψ“ΐφϊiφy΄±ΚžGS³'0>‘#;ΆK>œg¨Πμo–ΚΪmβnPS™{}zΜ95ν™kIA™Έ6yΩ3Α=N†²~ύzΩΎ}»ψ|Ύ@#/Π’ ɍΐ@+ρঀ€DΗ7ϋχ8!;—ΆπJ@ΪL²…½ΡΨςRΐm™Ι―YΆ„N‘H€H ƒ ―]ΚpfX·B)^³KvxΌ2ΊG‘jΈ¦ιcΏRΑς4      ΄&Μ±„T€2†$n—Sdƒ>]―Χ―<μγkpγeŸ½©¨ζI$@$@ρ&―/―|σ¬¬ͺY&yΚhΕ.FΓŸ@›d͊MR^\‘‹½£‘Fžϊκ!ι²Ί@Ω„(‡+ρ†Ρτΰ@rž·ίe½¨gΒμN:dLƒΚόϊλ―ε駟–>ψ@ζΝSΏωζ3fLΒ “0ŒgvΈ}β\$&G$@$@I$€')<ьξU,WβO3¦μΛΧ T4dH€H€H€H€H€H€H ΫΤ5x€ͺ¦Aym…η–L-½ςͺ―¦PΨ^³Cκ›uY‹ s₯zGƒ¬έΈ]Cα3°μͺ#ίθζ+οκέΚ 3°€,   „'ƒ―ί«–˜)•ΒΗLξήxΙW²™Q΄½ΰ0Ελσͺgvκ#0°ΦΖGFΐβ‰Ι˜BAΈΕ‹Λ9ηœ# ,ώύϋG-^yyΉTWW υYŒH$@$@$@ΩGΐ©¬ƒ* uƒΥ«¨©o’f_°Δ$@$@$@$@$@$@©%ΰSc5&±lU•\σΐ·Τ­¦CΘΠ7d+Η!>O­TΫ‘ΌΡ8€{·RygώωtγΗ8¨Œ$ %³Φͺ~«λšeΚΈrΩ™ϋθ―Μ“™?σ"  T0³κ–Κh6ε!ΠΖ Μτ£ V,ςhΥ3Ϋ4G,+–XΙYΥM!όIB›(&c@jnn–ηž{NŒ1Bπ»τK₯kΧβv»Ϋ%δυz₯{χξ:c    O YΉsF`“)<ξ%     Θb2ΏΜ,5ϊπ1MΰΚh±ΤΔΟ-Ψ“Yγ*Ž5••.iΰC’Μ*j‘H€H€"hq·xή›§~„ΈIέ YBδi‘u·!ΗwHψZ0ηΰJⲌژΖx₯ΩΈq£|ψα‡Z’C=T~χ»ίΙΠ‘C;$!i:„'‘   d ΡdIE³˜$@$@$@$@$@$@νΐ;²ω…Žο΄{r:Dh)œώς;Dήέc)™hj’ΎWeΟΔ’…T#7I€H€H€Œ@ΤΖ4¦\kΧ•5kΦHAAœuΦY6€1ιqI$@$@$@$@$@$@$@$@$@$@$@mΐΗΗψ(z£Ϋ:)­Žι†•8 ^‡Ν7Y;ρ1w¦—1Y,™   @όΔ<ω¦yZ±b…tΠA2jΤ(- : $@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$@$ξb6¦©¨¨‘#GJCCƒ455₯{ω)?                  ΔlL3`ΐΩgŸ}dξάΉ²hΡ’`BΪ \ρEρ žΔ                °¨irrr΄Ψ%%%rβ‰'κυηŸ^Φ¬Y#8ΛΟFε§($@$@$@$@$@$@$@$@$@$@$@$`#ρ DQH€H€H€H λΈb)1ΌΞΐhζ°Γ“›nΊI»ξ:}ϊΙ'Ÿ¬§~*((ΥDJi”——Knnn€(i΅ε1Α™m.I€H€μMΐθpκo{Χ₯# p¨ΓΓQα> °?κoϋΧ%$ p¨ΏΓQIΠ>=ζ ΖrjΌίCc ?Ψ$ˆ8“%Œ&@ύΡΥΛΒ‘@r μ6‹HnΎ)Μ-&c 6644Θ›oΎ©§s‚QΝλ―Ώ.?ό°Œ;Vzυκ%………βp8τqkΉp.φWWWΛ}χέ'Γ‡Χqy t–ίZ?\' l#@žm5Ξς’ d £ΏΩΟ”e9H€²…υwΆΤ4ΛI$‰ŒΟΔ²Ω«LΚXFΫΠ¨a›—ψ›kΕο«“gžΪvͺc>%. jμUg”†μM€ϊΫήυCιHΐή03Q‹„Xšυ,³οΪ˜Ζ(άζζfωίύ_Y°`tλΦMœN§6’YΆl™,\Έ°Ν:ο₯‹ΤΦΦjƒD4iΆy’MΩwξά)›7o–ή½{ ΚΗ@$@$0»qγFρx<―_?mπ™’SJ Θ^¦ έ½vνZ))-‘n]»e/–œH€„€ΡίMMMZ——•KE׊4‘žb’ @v0:ΌͺͺJjjj€oίΎ’—§Œ;D@}ΐλΚ“βήϋˆ―Ή^œΉΕRΨu˜Κˌ`%([&K$qŒώ¬¬”νΫ·Sg\ ³@$h°πU5ΥQ«~ΏZ’9’eM’¨iLuΐxζψ㏗‰'ŠΫνz ‰ζΛPx¦11:‰ζ“―–ζ„N ™Ξ”_]ϊ+ωτΣOe€Iβσω8 k§Κ’,$@$BΐθπuλΦΙ/ωKνQνΡGLUhŽ…œΒM  °κθχή{OŽ:κ(ώ>ϋ쳩ΏmP?H€"°κοΩ³gΛΡG-Ο<σŒœrΚ)Τί‘ q?  Ψ„€Ρα»vν’[nΉEn½υVωώϋοeȐ!Τቬ#εΏ°ΫpΙ/ βpͺnz₯I$o¦MHΐ§―Oφζ‹;Ο©Ϋ ΙΣΖ‰FmLc _ςσσεΧΏώ΅ΐCΩKωΠ/++Σ§tδόXςŠw\ȎŒ‚Ό^―ΌςΚ+rγŸoΤΩ`?  €} =g<ŠέqΗνzͺŸύμgYΩ°oMQ2 Ψ“tΈyw˜?Ύάxc  nφνyχ  ؁€ιΔ‡,σζΝ κo;ΘFH€H€"@ϋ?τƒΧΥΥΙc=¦ iFŒΑ>”ΘΨβ~ΔαΚWiκOΑγž6$ΘLVύ CH|„CšΡ£GSgf•³T$ͺωQήΏDΩ[vnͺ“ϋχ’άB5ešCDmLcjΦεεεf3«–(;~°δ|ξΉηt'PŸ>}τ lV`aI€H ˜W|AuΧ]wΙ#<"={φTΕ²πιŸ†υG‘I€²›t8:„ή~ϋmωӟώ$p1€} $@$@φ%€―aαΑχ­·ή’?όαz@ΦΎR2  CΐτƒoΫΆMzθ!ωέο~'{ο½·φ:oβp™ |ίIeζA™Dΐθο­[·Κ?ωOΉώϊλeμΨ±l‡gR%³,$h˜ΥI›εηΚ¨iƒΕ―<ΣΈr³Σ¨c6¦ΑIΖ²Ρ(eμ‹6˜oœ›n‘±±Q–,Y"Ο>ϋ¬άtΣMRTT$½zυJ·bP^ ΘJΥΥΥϊkΨϋξ»O^{ν5ύ±iΣ&υLσe%šH€…aW­Z%oΌρ†\sΝ5ΊθΐLρ)'  d-θο•+WΚ«―Ύ*W_}΅x<™Υ ΧφΚ{‚ήPi!Mk‚\o›€F†zx W'm'ώ(pPg¨]—•π‘Ή7 π άΰΝΧaκo˜Έ±ν‚ Τs nΜ½[*ΩϊάD_ί N7aΏαΊΔ1;Μ ―bΗ{œœΑωΪ¨sΜ9ZD§ΣQΤΐ5Τϊz -«Ω†ρhΈA]8bσ+½ˆΈήΝ93ζέτ΅ύ YΆlΩ"ί}χΐΓA[!΄μΨFΏ τ5ΆξŽ–ƒ: su.ϋΖ5Δ όƒ:Ζ}Έ~βψά‰…ŸQ±ΠJΓΈκY©U΅ΊΞŒ>BΫ;' ›Ϊ+ΡeΡκ­φ‹χqΘ…g Ζ,?ύδ=z΄ώˆtϊτιΪΙΑ·ί~+#GŽl3Ϋΐύ…Ϋ:˜gφšςciτΊ5v nK›]Ε1ρ­qΈžZsΠ…ψƒΎ\«>θ¦Ξ•^žΓΊ>ZΙοYםΙ&“ΟΥχ†yFAoθΒvžξάG:}TΎN»σιfr]XΛθπ2ϊ(ΘϋΠΕ²£!&cd„Κ]½z΅όϊΧΏ–—^zIΊwο.˜{!//OΉzτ(v?€zφμ©Οq»έAc¬λΠIH$yα•?|›››«έΝ› jK 0Γ/\@:fιtDΤΝΝΛWξ”άβp©_ψδΒe‘δ}Έν':NΏΈέyš‡Υΰ«³€ςT½δΊͺ^œϊ‘„‡C”rT±8%Ονw~Ύ>)ΗbŒe*£εζεͺyόœz.?_';—"f’‘Κ΄ΙΥ¬κFιu„ΆtŽfty^ΰ˜Bii©όλ_’~ύϋIί>}΅HΡΈΆŒI·a”ƒG€Λ«τP8•[dΩύ˜΄ž²O£ΰωκ9œ«\Ί”Ηܟ ΡΘQϊΫλ θo§j— ˜ϋ?Ίڎ•ŸŸ§u·Λα tμ·Ό¦΄}‚τ·Σ«κF1DˆUλ“lφgΜήctg>:ƒΊtι’§yςDα^>’Η~Έ¨G>ηΪΠαˆγTϊΫ‘Œ@ZΜ•lFβΨO‡γΟ?§κΘ5ν Έ†WPƒ~hCζ)W³NΥ·Ό‚Ζ+‡ŒM'Η©t«β–—Πίρ|74ά3¦ ϊ‘HΖBCΑ _|J»σ\βRούν=[.³ίxΘΨ±c΅ώήk―½€°°PˆΥ™68:ΐL™£Σ߁>”€ŠΧζ¦ΑlυΧ~ϊx ΡβVο0Ξ–wτέZ±ΣT:ϊ»ϊmp6Α£gˆJΐ%£ ™{ΐΡF?bτ b’ΕΩ‡Bύ=#9ύNΥζ τσΆυl5mQέ‘oSΟ;ο<GšΑCλwŠuλΦi]ήœξωeYθqΓ'\ΌΆOΎρΆJΠ‘cφ|Nt€$m“ΠηN[‡γ3*”Hζlkέδ/ήƒZήOmτD*8Ξ5:,RœTμ7ϊνε ρ ωΝo+ƒΦ²Z΅J‹dβD’Η#Ε1cΊf ωD ƒ τη*Cθ0;{κΦΐ8ft}U±RΝWΧnΊώsT‰nPr3j„9ͺ½έ¬yτ‘`ό2Π|‹O#Ξ₯ž;n•ξCQy\D-ZVGD θΛXι™ό–ρεΆτ³Ρί‘τ\[0c2¦A0yε•W΄!ΝΔ‰ iLye ΎXΒΒ‚ΓΝώσ8p ~@az§γŽ;Nϊυλ§eBgQ:…1cΖμ!n4Πwξά©=ωX­ž°ŽŠƒE?ζ_²d‰Ά Mqss]ςέ²-βέΎJv5»Τ–ύB³YΙCΣΩ²°νp*‘Ό»<²z₯G} Χ€ »ΰA’“Β¨ςΒ’ΣS»SVn¬oΓ.₯HqKΩ―n|ήf]X(ύΠλ«“:yΊκΠτ6Inc₯Τ«y˜ΉΚ FqG@9W._-•[vH½Λ«Bφλ‘Σ–Εͺ _˜ΆΣ΅ƒΑΨzo,Ξ],υέ<βin {ντSnP—evΉΞŒΕΕΕrΐθK 2cΏ9ΦΦυΆcΗύœΓ9&`έͺΏW¬X!½{χΦ^Ψ"₯‰³7]*M5•²«Ρ΄.7i¦z‰2e•ώVΐQf§¨ώ~©¬―τˆ³n‡Ϊπ°—κϊ°ζo[ύ6‘§Qςύ[₯~αΧq{ώΑΆΡ;ΆΛ–ο·I£·Q΅+`xΉϋ΄ςIΥzkύέΩE|K‘Μ‡₯ΙW/ίζ~+»*κΕγυ„ΥwΈΠ bΪ’FΗWšΞ₯ft*ή%π3²›cf_Έε† €²²rŽ.œ/k‹ŠŠSHΑψ_o…¦‰x.ΥΙ°qλ.©«Z‘ί\Π~T»mπ^€/ŒΠήu@@ˆΣΨμ—Ε‹ ₯¦T=+½{pξ°ΈͺΜkWo”][λΕ‘§ΪQ6Ευγχ©6Έaρθμ0/λ‰JΟz뚀zΩJιY΄D|κϊWcΡ‘u”9Χ•+Λ6/‘ͺ•;Υ@ΉzVˆ Ÿ­>―~ͺΰyc§. τx|Mβθ²Iζ.TχsK'g„Ϊ€ΞΪ΄i“ΐh<š|"$“έFŸ4Hπ³sΜΊΟΊŽλϊ»ΊΊz}cΫΆm“‚‚ωζ›E«WdύνVϊ{έ&υ.X΅Rš[μψl¦ΎuΏτ·KιοφΈX%cέ‘djlh–MλjkΌ―Ζ1W₯―Χ¬έ,΅Πί°;P μπ _ι [κoJΙΧ ξΒξΫΕίμι΄Η½εVςίmX*ΥJ7ΩT{Q/*ΨMC&e&΅Ν;eΉw…”Φv—fΥΧΦ}>s΄A7I€a§€τϋ#ΰϊ@@yΪ*β444Ό’5)ΓwW__jΌηΒΣ ΚŽ6x,i€Cίx,eŠ&.Π7«±¨`W<£Ι<Ιqϊά‰₯,6FΕR”hββή²ν;K4ˆ5Žz7σ56Jέ·K”φ‹Χ³[W΅—XΑ: ϊΝnνo#?d4ϊΫΘhτ±‰i‰v6ή/¬ρMΉ‘ΓΠ‡C«ž7ιAg9•Z½r…xw¬—>ε`Α†γ˜h{½-}(ͺΝk§€~ŸΗόnIδω{JcSΫ퉨dWε…ƒ†]›ΦȚ-ujLν){΅ΏQŽ€> ΄υΚΐΔVAλŽz)^΅AvΔΨ†i―πψΎnΩF©άΉ]άΞzΫυC~θΤΪ‡VύΠ^Ω}}( ₯… ₯[QΥnRχŒn9νΞq`»²~ύz=Χ‘>p•HTAΑρT‡΄€“Nς«ΞgΡGνŸ={Ά_uFϋ•!ˆŒ3ΞΐθσΝ7ύUUU~ΥxφΟ›7ΟΑθύ—^z©Ϊ_­Σ1ιE•ΉM"AfυβζW ΅DΚΖ?mΪ4]6εϊRο3Η°aΚψΙ'Ÿθ8`“ιΏQ#ϊ»wΥζ¦_ΦtͺKWŸώξCG°NlxŽ²WFΥ‹š?[λB;ώN6?Θ§Ό«iφηž{_MW¨E6zΫ,ρ K§{½3²jύ]‘ωΟ©Ξ0JΕΉξΎΤί©ΰMž™¦Ώο½χήVzPoΨθUCg_}υΥZ?Ϝ93’άjjΦ¬Πα%]Δ?zΤΰ¬(k4χ¦βτ9Ϊ/₯έX76kƒχ+ΰ{hFΥ‹š’ZλBk„]T8ϊPLΫZM½κW^Ž5{5kXύ­<ΧψoΊι¦ŒͺŸHz©’ ϊ{PV”5»ξο ύ]Bύm·ϊιίu PΟ!uΟ ?ΊnΑ΄ΏΝ³E}DλοΪ΅«_Δψ—/_Ε5ϊέ,ΡW¦τΛ¨:Jυ}0bxί>Εdj³φdͺ―‹xζίg„zζUτΰ5γ5vΙ%—Η νΏΏϊκ+]Ώ£FΪCCv£ΓŸ{ξΉ¬Ήφ=DMͺΛ~πxκ’x€ΥuΘp?ϊΒγ‘ӈίυ].έό#ŽΚ˜zιˆώŽΩΌkνΪ΅Ϊˊκ‘Ÿύμg2eΚuMŠ”——λ―ώŸ|ςI'lγΧΏύƒ΅β]wέ%&LsΞ9G[/κΣθ,΄₯“Ύ£ΌoίΎςόσΟkλ-c΅₯žQڊ«–°uuϊΛ*x:h+ΰ\­φ²•TO%,^a΅ϊ—›ώ*'NΏPŽ<ςH%©=ε5uΠλ؎©r’blSΐ²ρΕ^ί―Ώ>ψ ”–•)χαψΚΚ^WRόλ%pmΪZ”@ζžωZysψγώ(·άr‹Œ1BY<0ΨMζΆκeΑqΥΉ’Ώ?~ΌέΔΚcΚ™£ x†A[ΟΑ:¬pώ†Λzγβ²­t­i΄/™Η “KY¦Γ*χ/ΉYN:ωςƒd“ώΆ«žΨ­Ώ_xώY£ΎώΈυχKqqIφθo΄x’ΏU“vΫθ{FMK°ΰ«ς§?ύInύί[eΨ°a‚)‡Ζ’&JT½.2ΚbΥίϋμ³OΈhΆΩYcΥ£xOyꩧt;5΄ x—QFΪ;M[.@q^¬ω†ζ•ˆm}{¨?}ξάεφΫο–Ϋnϋ» 4°MOi‰%Ϊ4ΫΊ£M#4ž-λFέ[nυn΄~ΓzωσnKΞ:]¦L™ͺ#γ%‘mπΠzLΖ6ήLq ϊ”‡gž}F{§ϊŸ›~―ž­ΕΊ›ˆλ³3εŠVžϊzεΜγ‘nέΊιμ’=―3²Εzφ0§ξ‹hτϊΤ`lΨ~#­ΏΥύTΤ₯HλΐΆ΅₯ŽhžΔζΜyO{όI₯ΓοϊS·U™I8†λ_,U^5nΌαΙ―Ξώ©~ψϋκo0QΧQ<ƒ=ο™έϊϋ‰'—ZΥ‡ϊ§ύƒϊYmΧKŠjiΏ^ ΏαQ§θ* ύ‚6žυΪΡ΄L9bΉ.ΚTŸγwά‘ϋΊMη#-”:ύίπ0Φ‘€΄πDiŸpGR·Ο9(§S}•/›χάsLΪο0ωΙOŽΧΥ‘xίϊφ)x‹$ζΪK•`±\σ©’±³ω’Œf<ηΦ›ώ"Ώ˜q‚nZlΩp—E§«ΓqF?ΒφνΫ΅χάT_«αδΓ>#W΄Χ²‰?vο±ςτΣO‡mƒγΉ…ΎpxψΕ΅ƒ΄Νy‘rD›oθy‰ή†Gš\ε₯\„ΚΥWV»ξ:Ωgί}ΕΫβ©Υ.ΟsFβΫQNv­u1©iΑœΪΰ‘ώSϊτν#ΣgάjρTn—š w½ U{ΦMΛ•¨πϚ5K^~είrη]w μš=qπ˜ΤΡ 9yJΞφΪӝΡί1ΣΤΤΤΘ₯KeΎ“΄a δΖ *¨qΏώΩgŸ‰ςΨ"%%%ϊ¦u:ϋμ³εώϋοΧW8†}m)άH|x¦7mΚ·lΩ2ΉόŠ+¨ΏmXcsηΞΥS½Ν˜1£ΓΖ6,–)έυ·‘Ζ4Η{¬]1§\˜NζΎϋξ•Ρ£GΙ)§œ’vςS`ϋ0γ9Τ΄n3N>ΩώSΒ„9j€ΰ—ιαλ―ΏΦΖ4‡z¨¨ΩW2½ΈiSΎ;v菚‡ ‘žy§¦άΩ hUe•άίύςγc~¬?zΙ†2‡–1fcΐΐς²k·‚`Σ`.mXΐΌy˜#Ζ4ζΎΪ=묳δρΗΧ_ΐgΓΰ€Ά=-Λ¬v|eC‡Ύ0@€qB&—Y0 ώΰώC}xΥ˜°Όmn©Xΰ™{3 Š‘q"š{Ζά+Φ₯ωr' iΧV&λ²HΧb&—9]ξ-£Ώ‘―QζKύΪŒtΟdJ½P§φϊκHξFWΰ\κ‰ŽŒ9‘τŸ­ρgkŠζ~ΙΔg+υw¬WCκγ£Ξp-"P§Ύ> ΡίΦϊ0ϋν!aφJϋυͺΏ3αٚiϊ›χL|ξS£Π_ˆu\ϋψ‘Ο0ϋψPb*!`1σΜ3}ΣΌΎb£ši:<Σλ?uύšiuΫ•œΪΨΦz1Ο;θ$Μv“ιΧdjΙG—;ξ σŒ0ΛLΈgb½ηc6¦Ζ^½zI“rgoΆyii‰Ά\ΔWpϋ₯C‹&Φ`Ί XΌpΐY₯ b­ *Mώl*aΥM…`Κj–z'$€yΠ Bλ"άΎ€ Θ Y/ir „ή?i"vTb=a"›²š₯ΩΟer ˜z WΨnr%dnα°^ΒQIνΎL―£+ eky­λζ8—Ι!©^X'Ια)k½˜Ί°.Νz€σΉ?Ή²­>¬ε΅'—:s3z"΄B·I*Ή"Υ‹‘‚υcHΨgΙ:I|]qβgzΟΙτξXω2]·DjS ά™^φŽ]Ι=ΛZΦϊ°'W"ζf„«ƒpϋLόL\:b-TϞ=΅ϋπωσηΛΊuλZŽyΊG(L‹-RSΚl  L΄΄’ΖNΈ²eΘf²»©Μ)Xš—$T™…n§yρ(> @P/Δb’°Φ‹u=Y1I  @=a―Jd}Ψ«>Œ4¬C‚K[`§Jj«‡όSΛ?BξΤίΐpwΦ0χ‚Yf88žWœLŒH AψΜKΨN&›υ³1M=dδΘ‘ΪσΜ›oΎ)˜ΗΜLKcšθjψςΛ/υΓŸf_^^^'«Š§Ϋ™@6ήHvΘ†:[4{ΰ=cΟzΙF©x-Ϊ―ΦQ'.W̎νW “ˆχJ†Uhš‡ΧazTλΙ~υ„:α»‘ύκ%%’~°w­³~μW?θc6ύΜφ“Ž‘@bΈέnέ―τ$’˜l˜j–ΰ3/Λ/€,->―{ϋU<ήΣY/φ«Φ‰HΤΖ4Vaa‘vΨaΊ6ŸyζΉωζ›eώGσ₯±±Q0 mžzκ)yο½χτTP[ΆlΡΫ_|ρ…>f nŒ[-½3Mπ%nwΕ™ϊΔ` φ"ΠΠΠ kΧ΅—P”FΟw ˜c!uΜσ-u€>g£Ώ=Oκ…‘­`ΪΜ ·ΏVGΈ‘JfŽXsο€R–lΝΫθn³ΜFζϊ3SοšνldaΗ2›ϊΰ³Υ~΅ƒgkee%η_OQΥd³ήEnτƒΡβΩ94·“DΐβY9šϊIRξΜ& ΅΅΅²}ϋφ(b2J’δζζ 5Qp#€ϋΩgŸιqγI$B4ξ&0mŽηt_ڜ„φ7~ό  PeζΊ7}{iS‘.(κeλΦ­‚±L{0οEζή±—tΙ‘&¦O JwΚ”)rςΙ'ΛsΟ=§§tΒ΄NχέwŸτιΣG>ψ`)**’υλΧΛ•W^)ϋ쳏lΫΆM`Hc›iӦɐ!C’SΊη0"BΘζ‹Θ`ξΪ΅«Όψβ‹2pΰ@Νƒd†Lκ–ζΊ<α„δπΓ—.]Ί€Nζ$`tιΠ‘CεΙ'ŸΤΊyΟ%eΕά¦1”LmšIχξέ΅ώŽιΈ•΅ ΜZΟI|-ev¦.¬₯ ε‹gλ /Ό Γ‡ΧΡp‚Yκ υ'τ<³ŸΛŽhοΪΗΗ0§všόρ‚…PΦ`­_~¦«c=Ά“9πtΨ“€ΉΖβ5žξ:ή3WξIŒcNœ8QΚΚΚ2jΊDΣf0Χs{|Ν³ Ζ4§žzj{Ρυρ`aϊWΜ±¨ΚπHVακΓz܊ºΖ4ΗsLπ°υXpgˊ9Iχ˜γ‘ηq;@ΐά ‘x~ΊoEE‚ΣόΜ1½αO&ΦKŽ*xΐ\,B‘Γν6°κκκδΣO?ΥF2S§NΜቀ/ήyηyυ•WeΙwK4ά½χή[¦OŸ;2ŝ,ρwξά©-ςΡ!dΗΞ pυ—¨}ζΊψξ»οdΧ]2nάΈΰ5‘¨<™ξžL=ཱྀk4γyFoh, sΑWοΰŸ„°²†[ω 6\hβΑ‚―bϋφνμXN˜LΈΈ „›hC–””„ν€nuBo˜λ<Ύωζm(JC―δWΈ©δ Ο~FγλKθ -˜5nς%ΝΎ o΄ύΦ­]'UΥUΊύύsθp¬›xΩG(Ή%g΄5ρΕΛΆρ.‘\)’—›ΉjjjdρβΕ2hΠ ­Μ~#‰u|Φ­['x_Γ~Ό§ΐs >r`θ<Τ<±’-Άt4>8Aηdh°Φ Ϊ€›6mοΟ؏χg|Ωƒs¬qCΣαvtπκS9™kΟΟhΎ 2όQΏx§…nΑ4ΦΩώžωθbρJ{μ§ΏMύ£ν…)’·ΧlŸί§οӞ={ ~&^4%6qρaٚ5ktφᚲ^—&žIΣΊ ]Ž6!Ϊ θK«θZ!ύϊφΣ!Φxζ\λΗ΄8x~τλΧO¬q2eŒΠΆ:‡at‹)ζΡb8DSVkάΝ›7 ~Έ&&ρQ!t·5^ΈtΝq<'.\¨užίxfd{0lPWϊ^SΧ'φαΎΐ}f>Z4ρbεeΞCϊί~ϋ­N_ž·7hk>ΩχΒΞ];΅Α6ϊΑ³ωΓxΥ?Χ₯K—κι'ρ\`?LΌΘ2dX²d‰TUVIŸΎ}Ο;{Πϊ[c’’)ϊΫ<οΡχΎlθP̘L<μmΫa<νΕφΪv6ύ+ζύmBσ‘@hžΩΊwppBΐσ Ϊ^` VhηYλŒΜ6ϊ­Π&ΔΈφ‘ύŽ~Vτ·† ζ<δόΠΟ‚u΄KL κΙΔ —F6οƒύfΒ;κˆ#φθΗ2άΜ{1ϊG°|ΡV7^±M<ΓΊ~/άoΘKΏG©Ύ²ώκ]΄½{Ν€eΛ₯*`‡‚Ί8Þgέ_]]ν_Ύ|Ήϊ}οWΐΓΖηΞτ'€:7υ:hόκτ«—υ`Ž₯Iν_ΓZ=@ό3gΞτϘ1Ζr­~ηŸΎ7ήπ+e© dΞ±ιWBΓΜ_ύuyηΧͺNTGŸͺ«ςϟ?ί―:IΣ· ”<- ˜λΒ+—ϊΪΌλ»²,ι,΄©eιδ‘GόΚΣ[+=]~α…ϊί|σMΏκΦE5η€sΉΣAvΓY8ωοΎϋnΏšv«Uέ`ϋΞ;οτ«Ώt(eL3ζϊSƒqϊ:ƒ.ψχΏ­KaŽaΓΊώΡGωϋΫίϊUD«kUM ubˆ‘@Λ«―κLπΟ™3ΗΡE΅b‹zΉδ’KόοΎϋn+mκΜσŸψΡm›+οώΗ{,ψΎlΞ‰QΒ¬nx©N‡~θΏόςΛχ`|ξΉηϊί~ϋν6ίL:h―CΧ£žΤWq~5@―ٚγY :Γ nκWuϊωoΏύvΠκ::γŒ3όjš ΏΡŸ&~[XLœ•+Wϊ•gg˜1cZ₯‰wΒY³fω‘W¬Αœ§:“ύO=υ”_y$hužš:έΠCωρΏ`β[Σ°ξGϊhΫγš~τΡGC£₯ύ6ޟ‘—Cί±•Η_ώςŠ+t#q²0q ^{ν5駟ފ½ςΘαϋίξG›ΑΔ·¦Ί₯—^i Bλ:τΌlΨ6Μΐ,ΥWς­ŸuΦYΊΟ$–{ΝΚ ι›ο± ”9½˜η9=τ‘βy―fDΡ…0Η°aΦΡΆC[^Mχ«γ">~hσ·ΧΆSFΡώίόζ7~eΤήκά_ϊΧώΟ>ϋ,½ΐ%PΪ ψΥM{Όσ ύ|ΣM7ω—-[¦s7ub–Κ ΤώίσTγΙ¦^°DŸ ήοM<#ΊΩΖΨσΏώυ/QGΥκΌγŽ;ΞΔOψΥΗ­ς3ηgσ°›υξ¬ 3ε,₯'οΕwάqΗοΕΈ‡p/…Ά­ΝyhΓΏόςΛώ“O>9˜κSM₯ΣΫ²eK«ό©>`QΤn(&ΔΊmφ…&i?2ιBώ읠\EΥΗ'½χ„jz”.„"‚ΤOŠ)ͺ€ ₯£€ (‚(M@ͺHG₯‚ˆ (R‘‡’ΌΊίωΝsί}»ϋφ½Ό}o˜δνέ½wfξΜ™™3gNΪY¨­iœΤβοdΫ_~ωεΜήϋμ퓄ɍizΆΗΥ捙9υΤS½ΜΣ•)vτZζΏψEA²₯ίώφ·=[Α:4ΤΞ]ΟαΐΜ¨_Pn©_ψDθ9 Τ; g\\1Xe•U|,^zι₯=Χ ρM9>ϊύ½ο}Οϋΐ¬Ά3Ηw\^:P>BΟ`@kλμΩ³s Z‹R9γŒ3|£ώ£ύ(·±`5h―˜ςυL-λσ-’ί΅ŠkΪ%ΰa€yj;}ΐ0ΘΧφ‡z(3aB6»λό<ΰ~ϋν—γ5ξΌσΞϊ4‹Πjαϊž{ξΙαqί}χΝ\|ρΕ™_ώς—™ƒ:(w_ψU”§Χ]w]ξω·Ύυ-ο—K.Ή$ƒN< τΔ<ήΌ–Κ»U›¬ΒU²oPͺΣ7]tQfο½³ϋSπ ―(OIΊ‡Ρ {(‹bΰ{§hL“ΔRχ|Χϊλž­υ=}!"ceέuΧ͜}φΩxb)‡ΈΟ~PžBoΦs„…ΪλmΊι¦;&έy»ύχί?7χ»εƒξC#x§EHrώΑ)§œ’1o@ΏΞ9η΄bRehύ€ώPgΚΐγœZ ’ΐέΩcŸ{ξΉ;*ΥοaΥ‘ρ‹p"όc΄ͺ28β(dψ# π€ς¨ αžίΠ;†ΡΛ’κݘFΈw2|άpΓ 3¬ŒwŒ^„wνw”Gψ-vMβώ©§žΚl³Ν6^Š΄ΐΏX9ρYq Π'ι—β₯ΥοSα°Ή₯9'Γ…~Eˆ¨& hCQ†CΓ£<±r{°Vθ·Φ{ψ*x6aΰ h\&ΏΛΈ™1ΚΎ¬o§2,βMŽ§΄SWάι‚=¦ μΨ› F$Ιwω:ωž0Θ°SJœ`ψ“Ÿό€ŽwάqΗΜτιΣ3κC‹ά™ @>Œ7―ho€3<ςˆηIŽ_ψΊψΗώ>‹„•9ύτӝφœxβ‰~~Ζ鱫ΖΩ΅Ψ}κ+ϊyκΤ©Ύη}ςΙ'Ϋα8dΰQϋbxuφEάγοΖo΄ΙΦSθœ•†=s ;ΉΘοΣG”¨NνK©Μ;Σ€€ζ€Ÿλ>Wκω'’A°ς±ζΣs?ͺτ3°xUZθ­  ο©hLΣ³έͺΉFΤ–΅xSαΡ&@˜‡GΤπÝΡψΧΏώ叔Wιβ΅ϋ0 άBχ€YhhiΡΰMτ‹€vΪ)c‘ι½ΚΫ}΅‰%E ΄ΗXρ[ˆΎc…8νρTΞ;šλΠlθχ΄iΣ<‚A’~[hDίT𜠊…,χ*)o9λWοeƒc”Oΰž žŠΔΖΓΞZφη(oΣΌr½γ/ΆΡ0€7ΚWΖ‚Π @σ_W;–&³ήzλy"kW―β•όvF6IωΚίώφ7—Κ­ΌυFΤ~pŽsΈΓΈ…½ζO<ΡG8σ‘†}1ϋυ's‚y6lΨ0?†ύ- ~aΕώ{!9‹Ρ?Ψ(β*ϋξj„Ύ†”’ΐΩbœqfƒ>˜d0 Ώ`Α„ ΑηημYΓΫ•Α=ΞΒ83ρœlΣιωtΏ]¦x£κ0`BzζνLψΜX ΰ˜¨ΊΖUQ…™oΜUΘΆyΝΝ9˜/SΡœM˜0!ΨβvέeΧ0cΖ Ÿ$&o„ς`@Έ}μ±ΗόΌξm·έ6X84?σ‘~±Ε,Π/ΠGζτ•yπό̟ώτ'—W˜0.X?']cŸσΌΩ«~ρ‹_ &\ ¬·€πΛ•rLΠLqL˜Μ¨Υ}y|XΠ80Γ“ά>zί}χ f,”»“‘#Gϊά5!z0£Επ—ΏόΕη½φy…axΑ .ό1Ό;δcwšqσt©*˜αL€Ύπfpξ½χ^ ίg^šΑ ;Ό.v|ŸΆώ܌+}Lkΰ&λς:‹’,ϊG°(KΞSz†ϋ`ΎώωΟφύ5τuτθΡή7ΰΡŒΟƒEržΝ„Ύώ(ΰΊa‘bƒ9;–Οϋώ§Ο Μ9ΠΛ€Τ·Ϋ·CΜΨ*˜Α₯Λ]Lp&MšTμ΅uσLs Y‡»x»™πњkτk₯y±:-.uQε[΄μ`Š5_WΝH'Lž<Ωί?"Ί Œ3h…θΏx Ιgοz―žρ;™‡ί|Oƒςzιυ,™—|π ΌKυεžή­|Ιςσ=O–ΏW.4ΤΏ\dί§kŸΞ“ δgό€AiΈŸ~W›qlc-BΔ@Ή0ŸΜζ™g‹Hθ{rψ‡4hΌ2VΕΫΩ±Μa§ΫΙχΰο!ߐ!C‚lΈ<Œη€3§‡<―WИ6GŒ`M.a#ωu/ΘK„_ϊ…ύh|v_ΌK0£©`ΖOαα‡vy ύΒΊρΐŸζ δσ…½™EFυ~‘Μ$7(Ÿύ7PMύ?‰¨|ίΝσ9XxΆ`žνC0Μ:/˜“7šN‘ρ\ω!ΰXίΤ |°°Lyδv…Η‹υ1†ϊ8˜·“Χ’ΙΔ@θΐ˜ˆΠσ@X#Bdg²ϋΣAa2ΛA_ά¨Κ1By1°υΦ[» EΕ ύ‘>αΝτ ~„ˆrc@4ά,σÚkι―CΉΚ&%› 1YεK,!Μ‹2§|΅γΆΌτ{ΈΡ"τ[Ο–Ώu'4 Ωπ;(ΨDΰ^@:ψ MΫ•&^#JΕ€h4›R¨Κ@(ŸœF‡…FΠJ vΨa‡O(˜lJ%ψuΖ/†Ό@ςΉΕk[ ˆΨ‘!Dgv”–'x9έI@ΨgQRό C}ΚCΉκ?Ρv”Œμ±"O˜ΔbώοΒ8³HžθΛΫ~ٍψ‘Ϋτ0”c_ύΐόΒηόσΟΛ,³L°°Ε‘Ε%^fό¨^ 0ƒ‘¨&άpΓ .μ]{νuΌAβ³Έ²‡ƒ&Γ@Ÿ-LyXb‰%rs8ΰ΅φ[ίΟ­qΛΓΊΟώσnpύ&έΐ]ΐŒ1 €a—„—ŒUΥ ‘4FπζρΜ6g€GαμθϏaBfΖ7οΰy΅ƒpοlŽήœM6ΩΔ…ξz¦+tƒ$σ~τΎ€δÁC―1JΆΨb‹άΎ=‰{Œt,Žγ„ύʏP™gyΘ›Pΰ\ˆΑ_Ύχϊ‹κδC8Bq‚RΔ"Οψ|£ωךkΘNμ¨.—[τ7Vώ4Ίt#VŒqN`‹1²y1gω¦χιvΗί½ƒζ1J¦§Ÿ~Ζ ηž ₯0 ‡e/MfμYT€`©stš’^š…CFΦΘf-’€+—΄—LΆJc#kθλΙΜ™3έθ…rʐўς {Α˜“ύΖήδe ΐI„ufωε—wp‹ΔηειΉEOτηπ₯››x’Ύ%λΏW ›¬=šΖtS`’e?‚a0ΊφŒ €±ΘwςXΔ]WzbŠ‘/<c ~Θ’ 8Ώ‘Μ˘ΗαCγώhύυΦλ­ΏήB™Hε 'Φ€†0 ϊ½B)ΐγβ4Ÿυ`Yv’·ƒ——N†Ή ώ#o'Tήƒ|Ŏ’βkXzι₯<({Q ΌωžάƒΆI\Γ?XΣt¬+€ϊK8f=ΟΠ₯gm›>}z`ΏžςγΌ]Ίι¦›άQ€55Œ΅5½Τzλ—£kΙ>…ΟθΝε'ίUλίΥ8 ΈA™ρΦΜ‹ Ψ―‚7@ιθ‹Ξξοω§Ν3ϊsΨ°‘Α"«ΊAsΟ"9‚\ζgχ°›mΆ™ΛeΤΊb3@™vTΐQEjͺƒΏ¬‚?ςJ‡„8 i,Δ’{Κ°x2ΈQή 00¬ΈŠΑΒΆω=;'Ξ1!…ό0nxΰΙΛΰ…γHbςΥ‚€ ξΏ^­ZcS£oNΩ°²xΑ€1€hœΡ³]£Ή„€ήΞ›ίώχs‹WzgžΚ#qϊyΟΦΌ~ή3‘f(ΤolΎΩΔΐx‚…ΠσϊΑRliObζ•1iαύK6Ϋ1ΪIOφ@φ]šηίπx°3{ή@š>cόˆp@ΰ–~ξβG·c\9Œ?>7‰ϋΧ_έ#‚π žοq@}λ?βGΔ@'1€°›(vž°σŒ9’Χ9$t‘Ϊ·!$"‚θ–[n&OžμΙ’’HB„ O!Ρ—lAρ³3˜8q’+τ’yD  Ο–ρτΑ£Jϋ]„=8 ψAΨ$iƒΎ£ΌvηS¨ψΓψ‘μk0@a>yΩɞ‘oŒ}¦Θ"v¦Aσ‘“νΨ>w"­ϊ'/ώ ˆœj"χ5j€70έb1§™ηv$G֘ƎŒ7W²6Q>EB& €τL0ά^ϊίK: EIdhΆθΉ+?Ρ°0Ϊ@ΈŒ@™υB@”%x| Ψ[2½VijαŠ‚ΟG€ω n4§Υ>’τN™2Ε"ΟL?W:]‰Ϊ¦ύϊFy„{π‰Βœˆ8ΰEΆž±Ώ‚Φc˜έg_?cΖt/^iτz»ͺύ’Eap€1 gΒΙ„Ε²Ώ(JΟ¬Ώ‘Η0Fe/… aŒΝ€tω•ŸG €1 :oMδ)œŽ uγ†]Πεƒ/aljL&σbψ…@z]·θ`ˆx@ΤΙBΐxG,~Α‘‹΅‚¨δG}t»μb"ΟΠΠ|OΤD8«6EWΊ΅ό›ρrχέw"ˆΆ¦Ϋ‹s<†oδω£E΄?φΈγΒ³Ο>›Nncι“M?ιFεD0_xρ…pΖιg;ώC·sWψ"φ€Ζqξaό1Ѝ€.b­εD†Q£GεδqΙΧ0·πv2Rο o‡aΪΜxς|< εO:Υ_‹±0uKώd}jρ;νpΐŽ}0|2 gώΓ>ΐ'Fx€a?ωΏ…φJΒοK/½δ‘π„ΰš>Ε1xω–χ2τ‘χbόΔψ`ίΖ:X―Ζ4š¬Ψrΐo~ψαŽφœκ/ᏫφΕΎ€£> Ώθ0˜OΨ{0ξΩΓr•£CZξ¨|Μ#œˆΚʚ ]Ο’u¨Τοyih^tΏϋέοܐεƒœP›(Ϋ@ ΜW6ωxyg‹0 3sσΝ7η~ƒΔxξΨ9gn‘Τ‘•Š˜X―ΒΠ1|„G‚±ΦΔbάDθ= `“ΟR–YτzMFO“'Ov‚ηcωϋ  ˜C„˜cρg“‹! °Ϊ<ζ˜crF7škΚ―έ‰˜UΒΚC3 xw’ ŒΠ;@y MΞ’ΡΠo6@z˜/_ΌΧ}HΣph723„ω'‚ψέ€yχΥ –T/Πڏ(1ΨxούχŠ &εέΓΎmΦ›³|?vꩧζ6ΒΰAλ~v¬V’ΑFηGTš B$MφΔ8`Τ.zNίε3Βαν’νRœsΑ ήΜ€Ζ‚ˆ1=¦˜Β―oxj …ια¦ρxΓXFΡf’ψǘ6‚ς)TVΌΦ˜gπΏΜCy—Όj©ζ"Ζ²Μqό(클!ς0ΎtœΒa•‘纎7ήηύλ―½ξ΄„ρ‡Qΐ˜–λŠΰSŠL+γ>”βλC’~©ŒjΏb`NΏ!‡”>έ&πŽ°@VΩΨΨ`ϋΩ#³’}"zέxζ™g<a!άcτH?L‘GiQfΆœύγƒŸ––…Ρ =S~€o E³1ψJGsSŸ <ΔCΚ£(Ρ\+„6ε‘O0p`cDΔ½m!¬ΕϋΑ€θ΄§aΖέπΤΠYzί}χΉa;Ρω1²#’γ“1Q ^ή8’@+ ΡΚRN^˜}ρK' iˆƒq ŸππΘnˆ’uε•WΊ±=t„NΘρχβσQVaLAΤ=’m17ΠΡp< υE&‰{θΡΧpšζ™DM‹PYИ€0ˆb]‚—έΜdΚτρΌωσΒC>.ΌπΒpΔGψψΒαΐ`t[3φpΪ ’Κφ3Θ4Ž=φ8€žG<3γ‘θ_2€ΑπœρΟ8f»ώϋύ=DΑ‰1P. h½‡ήm^ Ω΄ΗρB­E–ŠαΉψ•₯σ©§—ƒ‚e)²ό}¨Œ7$<ΖICΐϋ,»`‘X@kκJ°φ!―. _ςΖ§?ݘ&_:ξA“π§q œλΚ})  1π…Έ§ΟψK‚ς%οΥΫwΝ ’[k‘¨ŽΣψ` šGι4ιίκs_οK§‰Ώ#-‚ŠύQχ“Α(>ƒ r9JAΖ+`bHaŒŽΪ&GC£8 μqJˆ³Ϋn»΅IB^ψ˜MA‰|‘΅@ŽΜqŒŒ1¨Χ|ƒfβ‚S4GbƒNG΄Šηθr0Ί@v¬c5βΌjƒώŠψrω…^πc;θC­MTnΉ©Λ9Lτ ¦ΰ1Β"b€! ς ’? Eώ_ hρ#+d μUN9ω”°Κͺ«pہΠl Ε"D ”“Λ œwA»ψWΔΫ!†ΏH‚θ^š·Γ˜†ρ/ΓxM'σŠ&Š…'„&Χ+ˆ^§Βψ`ΒΆ βΖπΟ_3α—υUψ•γλ§ή—Ξί@œq΅Ζ@‘΄ιΌ΅φ>>cΟ=χ »ο±»ο/γωφ™ΰXΖ4μ{ϋ›ΑZ> Ο΄/―.­Ό~K/ τFpΜiŒέΨΜkr=h4…HaLe„-Ίύί~ϋνέΠ―ρ$ +Όΰ‚ \™ /“h‘9PHJ)™L‡Α"RŠώQτ4ΤaύυΧw'?θΟΉB&'"ξ¨ΙrγχήΕ€Φ|­S8ΔcΔΕ>BJaα΄©uΓ]Ž‡ΔhΎCšd²wΑhλͺ«― σζfKi)k-z>gω•WYΩΏ+/γx«­Άς±Œ{\=ρ£LΠZOρkœΖ=oν+Δγ§y;½CϋΝ΅τ;ψΝψH«9α7κμCψ¦Ω’Q|ψαΏϊ‘\uQ“A@IDATΓtΠAaλ­·ζΆ§ΡΊ$ϊƒΤ‡ž±W~“ω’οMeΝυ§ŸΧςoυΖ³Θ€oϋΫaβbΩ£S“sAγ4ΰXγήχ’Ζkδƒδœ‘cA±Ή¦~"Ÿϊ”τΙwη{O₯έ+Έγˆ0³ΖšY;™D4B),­1¦ΉξΊλά²ε<‹,^LY(©+ ±>‹†M„jψ‹ΦκκΘν}b!ϊOΝσΞ;Ο•±Τ―„ΜzxΛ»ξΊNS±ψ$|κa‡ζG>± '½ζWοΥ2ΎΉΦ1ΗXeφ°ΦTϊ―0<ΡΛo„lnΈ‘WΌ–$•Ψ3š/¦φΪk/ί 'RQƒΞ=χ\χBDȊΗaδ}+±«§No’…jq¦τR‰°ΑωΠ€ςAOŒ8ξσ™§ŸqcšBεΖϋ…1 \’Μ3ΞtZ€œγqŒs ΒiKˆδ©>’DΎcHCϊ+Ν˜#‘8Β‚#vέuΧ•¦0Ϊ >QΌψβ‹•fΛ­²aμ5φρ°ήoΏύœΗ†>c<ƒ1 ιρCq@ˆ|ζHZήQπ₯ρAΝa@r.›| δ™„ΈJŸ/ξ1ׁbeRΟΗ*ScΊXΎt]”‡χι{±ό€«vΞ„ΓBνIφ™pS(­žƒ»bψ+V&eΛ[θέυr_γΌŽx–Δq©ΈQ•š>¦‹(WDΫWΔ}t&σbΔKp”„4ΐg©ΐοYΐ€AτAσ€ϋE‡Ρ ϋ|½‹ϋη‡Ξ`> k!²Ž6ΌΘ5D aoΚϋ4gH―rV_}υ\δή©y§(D#‘α=Ο•OŠcΚCΉ*κ§Ιfτ΄Ι&›ψQ_Œφ"+­ΌRXaω}›ŽF οΛx%ΡdΚβOύLΔ£iMsΗ έ#9φ?JίfOοFZΣDˆ('4ώ5>yW1ΩhΎτωκ'šΧΚ›/½ξ©ΠΝR+_­^ΑxΰΚq‚DΗ"GΚ±ηV„|_Βa>œθYΙ@ωH―ηωςͺO“ισ₯«΅{ΰ„6c€„Μιό£GΓ€VΟ ΅™|šΒ_‘΄zτΊJΟ}ޟΞW,}₯=+hL#‹-<Ϋ™Β€ž γ+'D,[:κ(?Η“ϋκ εα^„ˆˆςb@󎷠TΑ‘η·qΎ/ύϋe£EΕΉYήΎPιτ |Λ›{D˜@‚œP›„ϊΗ;₯G3y"D D Τ&’τϋ駟§IδθτO 6iΙt΅‰‰ΚiUš~Γ²‘ϊΪ?Ξ ΗΒO/’ DE ZΚF’rZkRν œvθ86©ŒCα₯–Z*L›6-όωΟ}ώΏaλmΆΞy…T{ϋ{’ώΒc>^οaDH½•σ¨·έvΫά‘ΈΙΌoΪΩλ<yζ™^m"¦qŒΖ€Sω=Ρ¦Zx‡Κ/Ž<{δ‘GΒζd@ΔΞK.ΉΔηΊΚ'ά'―Ιο΅€―Ψ†φ€―Q@(+% K§$Œ"!%in2Ζ |‚”•(? Νm" P¦ΛέL! H‰Ιύt]Tž|ŠB 9¬G=|WxqŽ Η…@x’Ο’{πdzαU}†‚\^–J§4άgœΠυ—Εka ΰ‰ͺΉΖψ—β©“ΉίτPh%ΣΗοεΖ€ζ?G‘ƒγ“žώy—αΑ[0P` =}"b€θ•₯―μυ“ χa$ΓNΡ€l™!2^xI₯$GΓ¦Χ •Ε|Γ°»°~έ³.hG‘όρ~ο`@Ό‘‰χ_tΡE.3"J#ΐ˜ *)œm΄‘GD‚ΦκX`8E[U–ZΒxΑΐ+ (d1ΦΘ›Π1θYαwςεŒχ"ΚƒΡQρvβ;’oSš4o'Ί(žE’ΙΌš;β3α q#™žΎ Ÿΰ'NdΐΑŸ£O:ι$—* 'si}Κ‡_αMψ…. ΏβυLi“Wή£rυžδσZώ.>γΖSO=ΥΎΠ3J—@Ϋ5~Σί“Όz±}1ΈΥ~H{Q]KkΜ/Ν΅j鏂Ζ4 8€Ε΅oΏόα|τœ+‚CΈgb`ύͺόɎ!]„ˆˆςb ·(ΩkˆŜdγE8QΌ/‰€"βηgyϋ"Yz1\#π‡ 0Σ^σc@xΜχTΖeL6/ΡΧ_έy9ŽΣ”—#αzc$ `ΠA€+φά±%%_άγ)+EΉπ­ΒΈ/' MΖήθΌσΟsΊ§ν―.ωUhnjφ~B€„q$πκ«―ϊ‹ΜhέΖΦn8Νs "‰<6θP~ P’zV/W”Θ‹-ΆXxꩧΗωΪ mΗ}&ά¦i„ςςœωŽ²ZψtίaΈƒA@”+ θ •©²λω*ά€+³ΫΝ5›€Η(_ έ«FˆθM ˆ χ βχgœαΥΩl³Ν܁Ρ<¦L™ξΏ~w0N+‡(ƒΏ|ΐ˜gΝHχ˜D«ΩgŸ}ό<Θg?ϋYηGpζΰH)ψΚ}χέΧή=5™=χrψKƒξ©mιηϊΏΖz―•€" υθΰƒvƒ«'žxΒ °½φZD@1ΚWΎ’SZ‹&ϋ8n‹1owcގύέ+―ΌR2oΑx–^FFωκΖڐ6-‡Ι—ΎVοi=Α€τςΛ/wΪϊο|Η#Γκˆq΅ŸžΪOΙ€ZΟ’Wα™«π+ή™VژTυ Lν”^Ο’εΧΪw΅>š~€_ΐθˆS…ΨW²ώ3ΆαΓΧμƒo½υVtGZNRτ Ϊ‹Ξ5ό΅ A°φY])΄’.ΪG±“a”ψ’JΖ4Ɋ#8(j(τ[ίϊV4€)„¨x?b 0 ‚ΙBrΧ]w…C=Τ…lςάc•(]T)ΎΒ0ΐΒ&Ο•ΥV[­έρ’£δΆ,8χ>"/b b ~0 Ί £Šυψ7Ύρ 7Tζμιώπ‡aν΅Χvd(]ύ`¦χ[ £δ“Oz°žζ‹—β>β/BΔ@Ob›XŒΉπI‚θόˆxτ±M4 >$™'~o‹h3(XQ–`Δ›Ύ#xC(„r|o|Μ1Η„|0`ττ³Ÿύ̣אG}”LΏ—†".ŸΪΠ΅ΰ5Θ<Θ΄fG3”FWΦXΖ €b€29φSΗz)]ΌΆΕ€ŒN5׈lήn-ΘBpLd…RζZΫ7Ε_έ‹Ρœ&0€Ωxγέ`†+γ₯Ό@Τ@yƌγ{GΖΊ”zž ρνfHFL6#ν#Ž8"μΌσΞnΟϋ€|Bi τ3'hΚι,¨žι|Ετ@ι΄ρwοcΕ'1¬!Ό+έrΛ-αT‹L€XΛ΄‡Α€ucŒtΊ$~εDC>φYΘΎΘΛώ> ³fΝς½<|'†8υraߊ/πӟώ4oσ1ΔdLΛH˜cΚ·άrΛ\”1‹$_Q꟏MΆ‚όeuΦΙΣ0Χpζ½…ζυ’\€ΉΖ8¨¦}TV‚*l,Β•s<ρˆ θwΌF D ”Μ;X, 9r‡ΝΧ…^θ†4<ž|η/Bω0 ό"ΐυ―νžΗ7ί|³χϊŠCτ’Ε&¦ žϊςυ@,9b :0­MΧŽ ιEπνŽτ»gϋS4ΊΜζ{·έvs£jΑ3ύρ‹~l4\ήθ’οώ ~D ”c( ·Ϋn;Ζ_€oLvh,#όD9 ΰA+#Ώ?ςb@ΈγΚ1œ{챇{ψ  ΛJΓo" "ΌΖΰfβY‘3}„ γΰ0€ΩpΓ ΓUW]εΗφΡ‘ΆƒΉΡ–πΆ₯0vcŸ5Uσ‘ ‘: …Κ*ύρϋ™gž Ώόε/ύŠΡΗ’1oPκF¨= ˆ†’$’Ώ IΞ<SΜOΖ€sΐqyRHωΤ‡ht`¬a`PΌWFzάΗαBή–ππ~ΠŽυˆVπ›:ώε/ ›mΆY.ͺŽΥϊU}†q’ ά0v$š Ο„cπ€<δ‰'Ÿp”ΕPψΤτ‘0-'mšžΌωΦ›αž{ξqCL₯l'S₯Ɵ`@ύΖ\ΓXΉŠ =נÊa)NόFόˆθa ˆ^ L†2ΘZ9Κ#v萎 ΝΏπΩ(]Ι;rΔHηΉ©6ΞuάOΣ)NŠˆ§χ‘^<;~ΘXP4ʐ†΄2ͺ$}2/Ώ#Τ60z!š1ŽWR˜2ΆpπYnΉεœ7ƈ•UWqώψυ™―ϋ}°B„F Ζƍώψ͘‚¦,Ρmφ™βi Ο€OcxmρI(s#D T ’ΌέΏώυ/ΏΕy»Ε½κΠ[’πςοΎϋOς™ΘŸYΐ«c˜ Θš;ž‘Ζ?DΰιN;ν47€aϊΥ―~;μ0ί3΅΄ZDdΡνiV^ieΏ~u”z2žƒC 9ΔBƒ† ζι‘qΘTpŠέαΚεϋ‹/Ύθλ(ιd|ΟύZ3βŒJ²]™["·†/Ήώϊλ]V‚α.< y9:ώŽ;ξΘνaι;π«}ρ /Όΰ¨Δ°XFd›˜ΫG©_s~FΖΎŒ΅₯Ϊ`‘iD°fnΦG€ξU2b}#ͺ5ΒHΈxŒ18›‹P„ss"ΖUώ ~”ΰ˜~a3³ι¦›ϊ;ξ»οΎœ0ŸΎ‹‹€§Ža Œυ„{ξΉ§[gwάqQL‚§‘~χΚ@`#Α&Έϋξ»}­΅”+›ŽUΌΣ"Βk―³Άo@΄.ϋΝψ1P& ˆO@‘'^γ†nhcTΐ8~ψaσkυΥWχ{QΨιh(ψ!^£ώςΠCα±Ησ<’€›nΚr”ƒΖw(=<¦N?ύtWΒ|υ«_ —]vYΨΜΰκ—dκOΟ?Šb@ΈB1K8bΰΚ+―Μ)€ΐ+Ν-Νd j'F:‘œEI‹ΑΒZk­•ϋγ7sCk/Š2λRu4IΡJΕ‡U‡#Ζbΐm·έζήtCά{φΉgsΗ³qδQQXηΙΟ^ )ΐ%ή¬Dd€Ηe²LœŒK€±Ι3Κ’l~μ1œα=Κ‹‘/u0’¬G' π„'%σ€φΒGΒΖ5ΰ£#ψkΙD<‘}Πgό%ϋJ•ωΫίώ6' W™xέίuη]Ξ %BήΛG*;^Ϋc@8b^`ΠΠ?ΰTψεΒχ«―Ύš―~œ ‚~υs,ί\σΔρ#b Œ`άAS !“τž±Νsψμ;L9ιΰ73ž Bˆ {5ζΫΘQjόP@Ω”•ΦΚ„?αBE¨ 0ξ0ΈύήχΎΉζ?nCtVX`L΄ΆdjG αΖ4x€(Jc‘|\‰ž„Q1ΚN9Q(€žΛΠWγ˜ϋŒ’8Ιqμ7βGΔ@/`€qΝX„·Γ0ΐhΰΩg³NpΏiήnΩe³‘iHΟ‘ΝπύδS4Sεγ9NΧΩ3δ+βλiόΣVπΜϊΘ1Φ8€°ŸΖXβ€ΘΡ‘~}ϋy:Šή€Ώ©ΛM χηςTœŽ Y€ΗF¬ZιιΠΨΛ£;ώψΗ?:ΝRΉ\9rμ°Γ9'žΥ:¨Œ{dΘ2’ςΎγŒΐρδp~ωΏ9ζ Hξ‹ Ψ@τΌδΈΗ™#€ cΖˆ‘#rs½FΔ€ςb<¬~a›ΫGY<³j~κνΤde5 ϊI„c­ΠυΙ΄t&ιΎόε/ϋ†Ά#’‘NM–ΏW?θWB΅r.'‚ ˆU=ZdφFOjήβ qΞ9ηxH\"Eν»ο~ROd_ς«q,<ρ€‹s²Ό½~ +§ gb€Θό Ο}τΡpώωη;-ΕΓ…pςlΐΥ―ε­],=b ‹˜V60°Πo¬εΗ€ζ9^„ΐDA‚χΠ~FΏΩ b7I»ωύFΈιwyϋΪMΰqˆγ₯—^κ)Oιx_ΐŸuΦYH'’ΠN;ξδ‚ΣΈΆ–·κ©tΖJXΌœc4­Ζ)W j %(Y‰vΗ^ƒγ/4Nα Ο>ϋμ°ΩcŽ8N;IΒ‘Œ)(°ηΏμ‘αοP–όη?W\q… -•‹k+‚₯ .Έΐ4φί°ςΚ+η₯ν”C$ ϊ,BΊγΦo h4Β™%–˜d(ήΘEA€GΚ9xν²v²Ξb7ύ]§<Όuι7„JΜϊ™g υ‹ρΪl9γ…'γ ›Π±Φ³_Ζ£οόσΞwΕΑμ2ΜSΖ΄ωΖo _|±Σ„‘π ”ΙΈ€.ΰqΙΌ&’ ‘δϊΧΏ†σΞ;Ο…Η(ΒvΩe—\>Κ%Ηά0~EQC_ Έ’¬BψΗ(|α‰ γΠ~”!b  h¬@W ‡ ίππmπ„Π‹―ύλN; )TsT‚ΦΖ/Ό8| ω )πδ&p¬ΞΘP.­»ξΊ^5φύI@ψΞϊ=‡Φ1Gΰi˜CδE!ΆύφΫη†8λ‘θ’,"¬R_m’-—_~Ή+=QHκˆ ΅χ™<γ†:l°Ύ΅c«8w„Ό ΉBχΰ) ₯(&‰€ eMdέaOΑž‘ύ‘Τˆ6@Gϊy± ‹9ύU€%θ/㠞ƒHΘ5 ―(£χΫoΏ0yςdo52mct†πLπBμ=1κbό“1ΞXD‘!b ά€f1~‘…θVη• πL4-ΙΫ½eχΠαι3¦‡«―ΊΊ o_Ι^8cžyMf=Ž½&FωΙw–»ν•R>mF—π΅―}͍Π'°ώA/X/ΩΗ€/ύAŸ _ΰ:šˆ±ϋ-p ƒw'β֏όcΏΟΊ§ύ:k"V ?μΧψέjp0τ@.sΙ%—x=Ž<ςȜ±‡ΖC₯ΰœυ`(τ8ΊμφΫnχ5ƒˆ{8([€ώ‘ˆœŸή#χb˜ΰ/h?Ό@ΉΘ$‘»ΐƒ /§_θ·ιζxƞψT3Ea>žyζ™~4Z1>ή ­΄«p0Ÿ6ο Μ‘3tPΖΌ6i’?ŒaΜ₯³;ω(~―# Ψdσ֚0ΒΗƒ…ΑΞ̞ύίΣ³:BG7U86ηψgξšΠ'c5ώΫ6N#lTψŸy―ω}[P2Ά˜Ε~*s©Lθ“±rθ§δŸ)X2žΎΜ΅‰ΕG ΄Ε€Ζ§)2 ΕΗδ/~ρ‹Ά‰β―²a@ψ·˜γޘό1›ι· ϋ=)O2Ά)ρz©Œ²U2μ0EHƎ€hC·EΓms± [Ζ6†±OβxιV h~Ϋf7cΡN|όY€‚vγLϊ¦(―qΡEEΪΡ…~g½9+sβ‰'ζ₯Π‹<“1Εiξ &˜ΞX”O}7E{zaŠ6Ό9tŸό"c‚†vύ›+0~i‡υ)Η3v BΑΎ1AΖsEq+Yˆ)Όϊ“Ήθ=ώ#~Τ$θΦzζ©Φχδuο½χΘΤΫ±/vΤQGyzSΦgL©δΟ5^‘™‘GΖ‹yΛ4WΖVmς(― ν3&άΜ›Za9mςωΔ‡Κ‘Žšf$–HQέ_Υ>3œΙνa’ύ₯οζ)Ÿ1!»7Vyΐž#Γx¦η¬₯ζQ›K£΄\Ν‹2cΚΛŒΖ€gN}¨S‚gLΑβε˜TΖ>©”υω“yΝΐ)/ŽM0Ÿ±£F9Β3<φAδιΝΰ $žΫ v<½9+eΜθΑΛSίΤ'ζc«»‚3@ΙΡ$MΠχ½ΎΆWƎΈπ1gΡ 23^›‘{²@δ―J›ΌrŸ΅€{?ϋΩΟry³¦@Μ›‡΄ζqž1ΓʌECπ4f4™Λk 0ΏgψmdΎšO’μ- ‘€ΪΙχ+―Όσ›1E:$ŸϋψΡλ O,ΒWƌ· ŽδΜ¬iΖ€)A ζ±hš3ΐςςΠ 7π+‡~xήw1†%οF.!b άMb‡ώΊhΚz­žρCί_±}ΉEŒΜ;~σρvΚGωΘΘ“t;ωέ jJβIʍž._ψA―hЎ3Ο°ώ‰&™‘LNiFH36υtθ/πΖμO’8M~Gφ’ή_)―agΜ 4o^ ϊ‘“Λ¨ΚWΟWα>[ϊa3dr”θ?ψN?™‘d^όšρTΖ wΫδS~φΛπΙ~Τw;-cN‚ΉyιTΡGkHQ°Σ-Ι°πΒΊ #X€ΓZK#kΏ’δ½b]Οήy‘R#7±6Γ"m™₯—1kΒ~5ͺΚo†ζ%σԈ•[v2'ˆΉNΊΜ?<(±ηb;έ›ώ.Ύ OeΌ°ΈΕ ΐΒΟWSζϋμˆ ΛΎ /YxlS^†ι敇 1Ƙ€ΜtDRCψ <+96oXΌ4ρˆe¬CGLι―Rz½7}₯Τqχέwwϊ”~^­Ώi78†ζžpΒ fœ=6^°τ%sόB›ι'  Ωα /Η^\ 2'ΫZŠΗδ6Ϋlγ^―xXB›Ήξιή‘ *#yχδaΟώͺ£ΎJζ­εοΰ…γωΰ‘²£MΡλQπgcϊ%‰_ϊp­Ο­Μ¨ΝηN)΄œsσwFˆθ Dΰ₯‘ηΙE4™q=!κΰFΣ6 -ΝY Q!ίο}—3Ž‘ՊΌΠ)h τœρΞ=φ—DnδΑΣ›hhD<#k‘ρΊ²Ρ†…I™:―Ρq8ήϋlή±ΟΊ]‚ΟI―[΄ΊΊΗ{Τ}4-α³R―τλΡΟCf,ΰΌcΚψ":ΡξDOY»Ψ'"׀ׁ/A—@Ζ"Wθ2γ“=# 9@Δ₯SN9Ε£i½%ΊQ­Ν©Ϊσ1φο"z Œwθ${7hW4~§Ψ‘μ‰pBDψEθτ4o§|DNQδ%"Mύ};t–γ£Ψ“Fs,ύώZ ύ`]2ƒ&η·Α ώΪ€ύδ˜&μXƒp?ΗQΨμo΄²W‚!B†ΕΎ)~Ρ£ιZφάsΟΉ 4‰#›9~>hWΏ[ίΠxφ†τŸϊ#‰p¦}1Gζ&χΕDbΞhΎ Ώ\ι'"wj9 ž‡ΉΖΤ\SZεMΎ»’Ώχ±.άQZMaΜXX„4Ž³lΈ0q…h―Œ…ΨΏ₯γ;R2Gω+θ#ώ"τ ’4~β7 ωΒχd~GˆθI ˆ†DΪΠ“XΟΎKΈ/υΝ±JΕTχ€KΣfm)ΎΏ“NΧ=o₯D d1ΐΈcŒ›Ι1˜δ5β8νžQTΏi^Ž·qτIzΡQ-TŽhJGιγσ…(Τ7]ϋZ“ι΄κΙ1D‹“sW㠝†tΙρ’oΉP²6χ±ͺiΒ‹:>Ω°jlO²ώέρΌ”β!οŠe΄Η@dXΫγ€’ξ$ι†65ιϊ%Σ€ŸΕί‹†HΏ;Ζ_€!γ¨\)"ξΛ…Ωξ)7M›΅yH—žN—~w…ψμb΄½λo«Μœ…Ζ]²ΆΙ1y$fΊη{gρKϊΈ7κάwTJgϋ¦XyqM.†Ξ?«ϊC΄2ΝM§!]±ρ’LΟχRΛLζ+Tξ§Σq/δ{oΎtΥx/‰ƒRΧ=ς£Ν])³Byjύ~Ώ΄5ίψL§)”"ξ‹a§΄g…θ7Ή‹=+­τκJ•“ωΖ¬Z“οY2o>:•o¬v”Gο+”·#WμyΎ2υΎx­ $Η΅Κ7φΈ―tD/²γXάaώ˜cŽρΘDΙ<(Ky䏼Iτ0’ρ%σ«ξυλΫήΨ<ŽGWE}’•Ί_Q•οDe|ς­Ο·ΐ–χν—ž~‹ξRίΠίHΏ;Ζ{L11Pω¨tϊ- ŠNs$šh:k€ξ)}GWρςΚίQϊψˆοŽ¨ 0ο{꘍ .Έ p¬ή·Ώύm?’Œύ0ϊΏϋξ»/\zι₯Ž’νΆΫΌn†΄‰FPΈκL‘ ƒ»J”ΣŽ$ —μC΄WuN¦)₯νD™hmɞ\ix)‹i""*]₯ίνŽy}χέwΓώχ??˜{έ”ΕY’,ΪzWw•]ΞrTWΞ½ηό6Ξ»~ν΅Χœ©ΐR—3(§M›VςΉκ0Ξθ;αΔΒ”ΙS|ρνN\—±μˆˆˆaΪ5cΖ ?Γ|·έvΣ£ŠΈŠ~sF#^όAΏ‘ΓDWγŒΝ6Ϊ(w>uG•nCΏO8ΑΟfιwG˜‹Ο#"* ’ί―ΎϊjΨύΓN;νT±ό9ϋΞoβ‰'ό|kΞp_~ωε~s$-ηόŠήw„g]tΡEαΑ “&MŠτ»#„Εη‰hxΓό†0ηΣ9MxŠC^*μΙ½τKNoŸzκ)|ΜYφΠoδ'k―½v>|xΙυ†ΩΟ~}τQ?¦ 'Ϋί111Π]€~Ο›7/466†3Ο<ΣeΙ•HΏ‘ρ ώη?ήyη—‘L˜0!|ξsŸ ›l²‰S NΤ6φίώχ=2ΖΰΑƒ#^ ςbšˆnΒt‡9ψχΏ=Όώϋa̘1aΣM7uγ8N¦xΰόMK,±D 2 ό™ŒίΊ© 5S ΈδΘ¬Ν6Ϋ̍’ψ]I šϋΦ[o…‡~ΨϋœSE0„\j©₯Βη?ω°αη7 γ'Œ/‰Wy·έv›\±η Ν܏111PU0rύΦ›o9ύ>δΫ‡8UbΪE¦ρ‡9毜 w•σέQΆ I7ηœsŽ»ΑΈΰώ¦›nrζ‚3%Ο>ϋμ€@_y:zάΉsΓέΌ;lΉε–ŽoE©ι(_|111P €ΦaΡ>gΜpΧ]w…mΆΩ¦ͺ•«ƒhρoΌΞ;ο<γ!ΞΎώϊλ=νV[mΞ:λ¬°ΦZkuŽ~ί}wψ—Ύƍ—‹2–{yό111PΑύΖ νψCΨyη+ΆΆ€N0γEŒ!-ΆΨΒ½ηΊκ*}ϊ駻kμΨ±%ΡpڎQεΝ7ίΎς•― sPΠVΛήΔ?""κΠ¬—_~9<ώψγဓNr|X¬Θ`ώώƒŒ >ϊhWΔ"ΐΒΎ8fΰΚ+―τ:’DΖ3zτθΡ%ΣοιΣ§d0‘~WL7NJD D tΠο^x!<ωδ“9ϊέΙ"ʞό±Η ίήχßώτ'JwœˆnΈα½ώϊλ‡sΟ=Χ Ϋ;ͺŒδ2D·ΏξΊλ\nΎΒ +8έο(o|11Π}@vΉΓ;ψ\fΫΠΠΰσšo~σ›ώωΚ\ENχΗmqn ƒΰν–[nqηJΡ·Ά){ο—κΓsΪi§εδήώΧ\s_Ώρo„O<э"•Ηωΐθ Ωcc«¨Η,‚¬ψ(b b ’0 ϊΝϊvλ­·f Β±μ„耝1M²…Ό Pm 1υmjjςΕC _?όpf€%=–ϊσΩD Με/`BJYˆπΚ<πΐπε/ΩίSmψ)Η‰eF D T s(!Ÿ~ϊιpΗw„Q£FUTΕEΏ―½φZ7€Α»ϊΡ#υΖ+γŠ+„9%ͺΑ…^θΡj:CΏQlΉΥ–‘±‘1n4+ͺχce""Ša IΏoΏύv§Εχτ3ΡαW^yΕ=X1€αlχ―}νk£eαΝ‰‰‘ϋΙ'Ÿ0€9ΰ€<ͺfGuem ύͺ«κ†πxgQ^δΑ;Β\|11PI€f]vΩenL_ T‚!θ7iŽ?ώx7€αΊηž{zτ"1`$ωσŸάι;‘"χΩgŸ’ι7Š ΰqdŠτ»’Fd¬KΔ@Δ@)€vC'/ΎψβπφΫo»\’”|=‘FτώϋΨcŽ ωΛ_Β‘‡κό7Ζ/}ϊτ΅(φ/…ίόζ7NÏ8βNzεΝWOρΨ¬U D3>θ ƒœχ&_„ˆˆˆjΐτ 6sζΜπόσΟ»LAτ­κ/:όΙ'Ÿψƒιξ»ο>ψΰ°Ζk8~ξΉηΒ―~υ+—…#ΗΰYΎςk'8:Α‹£”€φ«{|11Pί€ΖqάαtsΜα΄ d₯_Ei:[X-v‡Žσ@tμ±ΗzΩ,`ψ"ψΜg>;μ°@”Ι{ο½wΨuΧ]υΈ€λΠ‘C†52))SL111P!ΐ"ΠΊQ ^ξͺ«ωΟΒqΗη―ϋξwΏλ‘zχškιτϋΏύ―G(€~οΈγŽz\ϊ=xΠ`+)CL111PAHΣοJ©šΦ’ž!Θίe—]4₯«@^u„€FPΔΩξڈώ+]‘kΖΌρFŒyπBŠχ#"*9#SΞV 8ύ6έ(‘ΰ9JΩ2ŽΗU†θ–·‡R–ˆc₯Πo­ ΘiPDŠ0―Υ†θ7 ΟJΡXŒΨαΏΏϊΥ―†£Ž:*Lž<9WΝuΧ]ΧHί}χ]Rsί}χŒi”7—0Ο`ΐŽ2!b b b 1€ YB₯ό5NGπΦΎ`H#Ψh£όΈ§Η{ΜΣ`Π^j”vΡydHΠqώ"D D D T _π’]’Ζ4])°Φςh‘@HσΓώΠ…σl8αέ M:Υwe3€²Έκ^Xόˆˆˆ¨`  °„Ϊΰ7PIt c—όΰ^Ο΅Χ^Ϋλ'š+ϊ½άrΛ„@„1ν,¨¬HΏ;‹Ή˜>b b 71¦ί½Y—B年Λ/ΏΌ΄oΈα†ΉΘa€gέαΘρίDŠ$ΪAg€«΅%ΛΟGήΜΕ΄•€ψmρ‘•PŸdZ3­\Ε γU‘—hŸ¦ίΣ_^2ύζ+Ϊι·0―Ul”U*ν‚F£dPΊΚF4œ΅gΙ%— Σ¦Msc’Ψΐƒ£T%M)² yy pα/‹½‚δœΥ<ο•ŠTΑKΑrdd 9Κ©Έϊrόψρα”SNqωΙ*«¬βdΝ‘ώ³CΓ§L™^|ρE.ΣΩHP©λXgΫΣG D Τ>rτ{-μJ‹£1MXΣ"ΔCdΒ—)z ‹§ΰ7ήD6[l1έΞ]Σ̈~λͺ„ΌOοΤ½JΎ¦λŸ―ΥԞ|υ―ζ{ωϊ'φGοφhΎ>IΦ¨ΪϊGνQ½uMΆ©·Ύ«.+­΄’G¦)FΏ s 6> j£ξλ·ΊΟϋτNέ«ΤkΊξΥ³ZΪΥQ;ͺνyΊŸb?τ^¦ϋ’XMͺ₯ŸΤ¦J/u#δζ›ožCΉκ‹! ΐYΰΐj«­0žL‚Ϊ¨{(a9…ϋιg”«²•ΎR―ιΊ«g΅΄©XͺυYΊŸb_τnO¦ϋ#]›jνŸJ­7²’ν·ί>lΉΕ–feŸΕΆκš¦ίŸYγ3aθ°θ·ΡmςgZΫ«.Έ―²ΣύΪΏ;kέQ§Jjow΄§’Κ(Τη=ΣK…πί•·WtŸ]¬ΤϊQ/"Š­Ύϊκa½υΦΛ‘>]_ύF©ŠB5 ι~Δΐ²_Ÿ~mψoεΧ‘Μ_OίΣΈZ”ΆηpΊ(…ΔΌU…E?Œœ ?ϊθ#?Ύ# ξuGΉU…Θ«,†μ묳ŽG_ߝΤcβˆτΤSOy«ΣQΒςυ=χ’γ"Ik’ί+ωΪ•―ΎΥž|u―…{Ι~Š}QY=ͺΎ©η~‰Ζ4“Xn6,—ƒp@D‘yωε—Γ΅Χ^ξΉηžpΰΊηU.Ρ‚/ιAFYη,V3€ΫUΝm©ΕΊΗώ©Ό^};}’~όρΗvήχ<΄ΌΞO ‹TΣtŸΥύN·ImΧΚΒ@μ§ΚιΨ½ΫƒΞU€ Ζ‘οΏΎ{Νώδ'?ρg„'ΦPκ/]•C€ϋΠς΄ΰ_ι*ύšnW₯Χ·^λϋ©²z>φGοτΗΰ!νι7GƒpόΣOΞ=Χ+΅ί~ϋ…I‹Oςοκ']Ukύ†v£ 0OΟ+ιͺΊVRb]JΗ@μΏqUŽ”εΐjηΚ„ΖnΊι¦ώ—Ξ ν₯…ϋίφΗ8”¦ϊHχ#†4@΅Λΐ½έό‘ΖU7‹«q ,κψΑ9ό¦›n όη?Ø1cΗpn²Ι&ΎWqΤΥtσ ΥΠ[Ρ\~£Ηœ={ΆΈτKΓ;οΌΞ<σΜ@€v@cIΧ$‚tO†9ΙgΥς]m¨–ϊΦk=c?UnΟΗΎ !Σtr|jγ€φ§?ύ©[qώ󉆙―Ο ϋοΏ8ώψγ=²Q<χή{―ΐžη(`ZͺhΫΌyσr n²-΄‘3€Σ^ΒΙ4ρ{y1@ίΠGIb‡A˜˜©ςΎ=–žš››ΫτI2ηφU3sšlK₯}]Ζγβ\ή?ύτΣαψG˜5k–B}ŒΝ£Qζ Ϋ€Β΅Ϊι7!˜?;·S`q\φΞhžkύΤΠΠΰτ:>|ΔπΠ―oVΩ;5ͺί·&ΧSϊ‚yΔϊ Ν†.πωžςŒp €{}Πphωέwί–Zj©pωε—»-JVŠοωΰƒ<»~« π¬yTxˆ9sζ8>Š}ΠFIG€bιγ³ξΕ€―΅ζεΙΨ£/ˆlš4 λή·Ε:ΒύΑšZ "―S ;]Ζψ˜ rήyη9]Ζ‰cϊ~ύλ_‡wά1g #z°Ÿ}“~« xς?όΠΧ[/ΈΒ>h/4ΊΨ~oa•ϋ„†Ζ†ΠhΌήPΫ£gyΌφ‘w¦Ο~'x'=ŒΣiβοa@΄‚q†§>|Œ"θN;“‹'Μэ|yEΛ‹ρpδgΝUTρ|εΔ{Ε1“τ—ΤΙ{ΠπK.ΉΔ !‚ŒΙƒΒ£Iθ €ΌΜ₯Χ_]·βu’ϋΜ"όΖ}hW±Wέω’2£’[bγ…9Λ:χ›ίό&όώχΏχc˜ŸηŸΎΛ7Φ\sM;Κ¨δbI¨Ήχ B[·'†Σί_}υΥpφΩg‡·ί~;άqΗΎFr ΤΑμ2ŠζΓ#+I² p•NSΙΏοπyήNΓI‹­QΜψ…*―~Τ›υn/OŒ?x† &δΖeοΦ*Ύ=)‡¬η=QΝΣt…˜k±θΜ΄`ρΈξΊλ<΄όK,αYΩτ=σΜ3ξ !fsΒρOˆ€Ύ}ϋΨ†’Ε u;v¬ίΟΨTΰ‡Xۍ7ήθŠθώ ”ͺ.8m6¦ αwήΩΟHχ5©s<˜Š‹ΧN`@ύΓ½ωζ›Γ›oΎiεΩ©ήΦXc°έvΫ9³ ΄(>&ν„g’ 0gššΜ˜ΖΚIŠJϋڜA˜ΊρΖ‡-ΆΨ"ΤtΟ₯faσpρΕ;s†Υ Αƒ\hύφί&4UŸύη? ;μ°ƒ AΧπq幎„β{₯ƒΪΓ:uϋν·ϋΪ•¦ΫνΪ`ƒ”(MΆΑΰ¬τ/~ρ‹Ρ―’Ί†ϊŠqzΓ 7ΈηH_[cdl΅ΥVaΓ 7Œ›‰ξG{ήΥƒ‘έ¬«(Ί ™Ζ¬Yo„e—Y6'| |9|ή\K/½tŽŽδ-<ήμDwΙΔΉήτΗ +¬ΰk%Š’ιΣ§;]#=iΜ‘C9Δ.oΖ‘σsBŸ8}›1e_Ξ:”Ο@}Υ²·{μ±GσπTK·5Χήzλ­pύυΧ»‘qΦ ­΄VB7eLσκ+―ψJŠα#F…>ύ³ΚgΥΦ·°­Ma„Ο~φ³YfήΈ°μΦV©β΅œ˜9s¦/@Ǐ ύϋbaΒGŸ6˜²ιΕ°Ω6Λ2ε¬@,;‡Ρ λ0Β[y΅•B‹ύKB?½ϋΦϋ‘Ω„£λ―Ώ~»)Ι΄ρ{Χ0 ~@ φ·ΏύΝι7Βk,§Ÿ~ΊΟχ»οΎ{Θ aΟ‘ί=7„δ‡v'ι7Ρmͺ πδ₯}Νΐίΐ‘ή–bυοg›ŒgΏζ›"Š#°0Πz[,_|Άθΰ c#K,>1 μΧ'Όρφ{n΄±φΪkGcΘEGo§JΐPγΖn c'Œ }€Ÿiνϊeϊ‡^~>dϊΆfω›–>αΝ™o…Ο|ζ3nLΣ©—ΔΔb@4ό _ψ‚Ρ‡0ηωηŸ—]vY8γŒ3Βο~χ;FƒΡ)@ΠwƒάP)ΠD•«g•|%Η"ΰ?q)œη«/όv?—3^ύ_€f`L‘η0€ξΦ[o Ÿ|τa9tpψ΄‘)1­_&όο•ι‘Εξaδή8oŽE€ί؆9Π1w^ι­―žϊ͘>#\yε•a­΅Φr'ΊΗάυΚθΠEχͺ§5Υ_Sαη$"Z‘k†_Η '€z)x/”>ΩVΌ εYs°ςΚ+»’ρΠC 't’—‰wy_|τΡG·aŠρ”ΐ»―Bχ6ώ ΅Ήψύ>aό8³f΅\4q½“+ίίϋwτΑσVD\vŠγ±ι ’τᳬι‘ίλ1Αϋυ?ψAXoύυŠ+¬θ4Eμ±Η[3τ›ΝΟθQ#B4> YrS£Ϋ} ΩqΨ―v«­Q™ΖΝπζψnσ<ώ(;Œ„‡‘γ k-5:,;z`ΈΝ<|ϊχΟ Λώςψ‚vgΞ‰«Œ ‹Ξ jϊ›βυΉΒΜϋίSwX"Œž<<΄4Ά†Ωύ$4Ξir/―vΔ]Ζ@’~Sˆ"Bς}ςδΙu&K―oΉε­M<”·I€η†gŸpΥUW%Uμwη! uƌΖ.gήρF™±'α·ΫVέ9ρ^‹}bΡ”’―-rzθΧΰ‘ΓΓψA!|qκΘπ7> /}’U$υΠλγkR>Μφ?ΓCXvΪΔΠ¨ρ7Ά¦x&6~άZξlυΉ•Κ."’τΪ»δ’KζJ„~‘†ˆ/pΫm·ythό*ήn_ϊ—rιωβτΫh7B< Φ* |T٘Z|ρ aΤλ„Ύ#§Ϊ"Aΐ‚ŠC«ήy"4}πt΅μΆ‘°I66›hhϋ¦-ΰΝCΓ{fδ>Η³:’w΅/$ή)†π9hΘΠ°ό€±α3“F†;_ώ$l0qXXcΠpucsάC^7=c―3~βΈ°ΒΛ„ΑγμΘ…΄Ϊ‹·y‘1ωΙΛχΜ ­ΖwOέzΩΠH–ž·1}ΒΫO}2 έT©:/F<(‘›n½υΆpΤQίυD@Α2_4N’™Fk(δ;΄ύ‹‚ρΓώ°Ξ±Ί°ω¬yγǍ CΖL ίΐ$3Œζ,²0Uρo¬%Νοώ3Lœ;;τ3ƒ₯υƒ’šΈDX}‰Qaƒ%¦·„~ Ώρ+oΪn~α— ’Ψn…9M­a­Ε‡M–gσΦnΆ%°ΛδΙυvcxgΞόά~Όh†ψ°lύ•ΎΊΜ@ΔT’όbXΓρΩDnΗžηδ#Θ:λ¬Σ¦n8”α€€Α8G΄ͺά6‰*ψλ΄vΤ€5?_!τι7 4<#̟ω‡0|±ΓΐqfΠί2?̝ωP:`~·€Ά«ΖΈΒ˜‹ώQBqΏS}Ž‘4Άμέ=jSeT«ΗkΡi.‹°>xbιLtΟ4‘ξι–`DΓy­œΫJ˜ΰΞ!1¦ιΤV₯S›Θ3ˆπ”E)‹΅γtσbF˜Ÿ„d~³wL>ͺΚούΜ+ͺ₯₯ΙΜf,Τ²)f³¬?œ–))ϊ˜U½+DΊ=Ζ[³ ™gςΔaύΜ{Ωϊ&ΣβΜm>ω\ΟΧ°ΎήHτ·h sLˆί7 `aΌΈ+φνΟ¬α8;ώΙΑ4Dθ^ $ι/%‹~sε(4Ο·£=Ο±c,]μ§nν£4νIoϊfΉε¦ΊΒcŽq! nj’ŽΌιόTAP5Ϊ@{ωξόΒ “?ΨΌX‘ιΞ{/lρ“ZϋŸa|Ίς.|Ώ•ΰ£ΠΖιλΌx³)ιΏ=Ÿ76G2}ZΒΰ1O@IDATΓ ‘ύsν}Μ ‡£pν[ΟWΖߘ¦?iϊ ›mΆqcΒyΏoG!c*FΏΣeV Aύlž77Ϊ±‚ύ†šŒd‚m:ˆFšl9­0Ziϊ˜qdŸΑ–cšΌιαΝͨڞ7ΩΡΐŒλHSΊ·χQj£μg}6μ8ζέfΫ‘qƒΜ׎½°΅{_Kk‹C―ϣǍN3ώڝ τ·6Ϋ<°νkΖΌΙ‡˜±Νλ lx–ΣωΪEzΔί8GΪ’Έ³ΏD‹‰&ƒάόΈγŽ ΫQ·Κΰ0Šγ¨δ+;½Ξw/_ήzΉΗΎ±/²u‹ŠšΣ†Ρy­•₯ΰ ;=ŒοΪβ2ΗRςΕ4΅§qΖ ΄(w‹™±Έ‰–vΤ:ΖΨγ[,Ϊ.6‹_0žŒΜϊΪ·ψpŒνFv/ΊkιŒ‡ΑŸH{Ρ‘£8ΚΚύ4­mα:aΒ7\ǘζίώ·&± uKηΧ>)}Ώάνθς©7²‘Lί!Ζo7^ΒΌ^ζ~hχζΟ>ΜyπLσ§6†™D6KψέU»XŽ0ΐΎΫΖ(GΞΡosΊ*mΌφ<θώκJ6¦a3hρ‚†θςŠc`  Ώ+q.χ WˆDsμ±Η†»οΎΫλB=¨ buΐšŠ#Yw΅!}%¬ϊΫoΏν B₯SΚ–b‹9"Τ`LΓPštyͺ{ϊ~5ύΖλΕ¬ ρ]Hd΅·υ&cΈAh”±E «xyWSΫj‘}Œ‘& t“q³Ν¦ 7γrgl±\LAΟφ°ζ{¦|5E+F4ϊΛΦΔ60 ξ1_’Π§ϋϊά‹~³™KΣo½IτϋΥW_­iϊ°Ώfhvk£νšY»Ω(€ΐπΖ݌)i1ΎΑ š‘η0€!*Λ)†MFΓ[­O0β+zfυχ&hΡή²t~ΈΥζ†Ν›πμϋ3[h‘ί}}­?<•£Ε-&ώδγOΒ;οΌγ ΦΡ£GηζsAλ+Ζ3όχ§Ÿ~Z°*JΟ΅±ΘŸ ‚ά0γHϋƒX$€φ9nWζEθY 06‰@ΪdτΫh…‘p§ Ππ=ζ όukΓBώ»nΠΧ¦-F»™3Ρ›»{ϋAΫΗNΏί‹O\<ŒmΖ" hU’~+ΪΨG}Tτ›0δ8[A›3h΄}oK’³ˆΆ!˜•›Ψsc(2ΘO<}{šξ λd墘AMά3vοx₯4­±F΄αΏ‘ؐŠϋ —£δλJςFXt @|ώoΝϊιΌ΅Σκ,Φ!n\g[Ω™ ž%ŸέΚυO«σδμ{Ky/zj¬ρΜΘΑϊӟ†³Ο>Ϋ[xΙ'…C>€αz)MWy\#,Δ|ϊΑCCC‹ΙfμΟι<8*…ΨX2_GlMh΅ΌC,*"t,B``Αψ ΏG bNX-‘gM3ρ)ΦΑψΡs§ΫΊœΦΪΨΒ 7];(L>φ=ŸZ]†ZΔ“ΖEθ @g9½ƒcΐuλθ7ραͺΖ34kΦ¬œ\ΟWΡπτύjωώ’Ώ…Ό6–Ή Τc/h|;rΓŽŒη=δeΜ!_MήλωΕ7‚ϊ Ι_ψχ:#]ΪYΞ‡3­ωΫ`ƒυνԍݰ†H,„ξMnXΕΒΛ1πUζ<ΰ†4²ζϋίΎ+L W¦4ω†>Οh Ηx&/‘‡wΩe—pδ‘G†“O>9 ΜΧBβG6™{Δό†ω~vekή’Jγ?jμ#$³,6Ζ7Ψ,2ΏΪ'χϋ™– q„žΗϋ¦4o±ρΛ† E,ƒZ$‘Θτ|‡Ψ‘IΝσMPŠL•ιb>cμ;Χζ¦ζ0ΐϊ'Ή@υJEkθ₯Π`Βΐο΅Χ^ξ=Ε1|RŠ6ηθ·LΎώϊλής±cΖϊUij>ζd–ή쳂Ǽνs¦(+–l5οΐΠϋ*‹δ·]Ut&ΦSdΘώg¬§’ίΊVQ“ͺΆͺŽw[S]XΟάpΎ 9NΛ³ίαy݈Ψϋj›[ύύ`φΞΧκWΏrZΎΣN;Ω|0£%„°rfY€#ϋt kφIΫΟjŸ7bͺ¦¬œέy:'Ϊ‰Ζ’\‚v²}Yͺ'Σ#_gP·ΞΟ ‘έrAGT Ύ€ ΰ«ϋέ2ΗπŒχ6ΗιyΖ²πζύ«F΄ix/ύύ&ΚΜ‰'ž.Ώόςpϋν·‡vΨ!Η“F0{φl:nάΈ0b„EZΤ7ΊHΦk·Υnφ{.1½sN£σTKό„Sr˜‰τišžΝOΫΩΩ77Ξ³}εΰܞ±Rq’§ΥyKψ€#˜Ρ œ‘\‘½ °‡’]ΞY¨?+²eΥU)x»Aƒ- 2X‘s΄š¦ΨGnz,3/ψ»mΣ‰Θͺ) ΄Πι³Ί°Ρ»΅=CΛ1Ψ^x‘Wˆλ~ϋν玫JSjM5Οt-5_­§ƒ 12ΜύȎΑ ²Οk2 ή܈.ŒδοL [oZš¬œQΆOθ’š§π;β“ŠΔΐκθςγQΓ‡…yF7‘a¦ΔΡ“₯§6Ξΰ@ςA§–;{/#–:Μ5Yγp3πŠΠ³]ž7o^Έΰ‚ œΗς°Γs£=ΧG€gŸ}Φυ’ΘQjpVt§76}ŒXώ²ΰ<»sΩ&Σ[ ‹Υ³xν9 °og|\£1MΟαΎ£7Α―¨oθ'­Cε«΅η%sYbx9ΞθCρσυώρ„Ώύ1ϋΙO~β!Φ‰V³ώϊλϋGDτ!\eι~W”E9~’ΛΥΔΩ«Ηz_ΎzI`/WBΘcT³α†ϊϋ) ΟCΰρΗO<ρDXc5Βδe'ϋ½Zώ@˜? @3ΦΐγΚΨ΅G‚¨ΝxΔ’&2PΒHΟ^!lΞd…Af¨a\-#M±‚‡}„žΗτ³iž…«3];0~ΉΑŽ 1O ±]Ίx£d ˆρ"\Β’a@4ØQ£Fyaχέw_Ψ|σΝsΖ2JƒΗμƒ>θi8λ[‚ =_΄šTVnΪ”¨iΟ©Ό=7ΪmΡΗlc2ŽΗΌh*γM=5@―Oψ θC„žΗλι γcšνθDΙ#’΅ ΊΌyTΊ&±υο’½8Ι8ζή{ου#±υ[iή|σΝpύχϋːεΤύ2dˆ+59Ž©ΛΠ%μiwΔ”¦MsΒΰ! iΊT^ΜΤ8ΐΓ5Y$Y—7.H ΕώΫ³h˜ΡiέpCτ€΅sˆ)`[lŸΪj @sFf½;Σ 1νxΦ„r7_ϊφχXšΝ1on˜0nLά‡ΆGPMίΑQnΜ¨‘aΜw=}o5Š!‰KΒΗσ›ΓΘα£ΒΠ‘C{«:uϋ^­§ΰ,@p€vά!L™<%§Ηd½%šοC=δiƒΕ½–φγ‡W0΄±ΏΞP0U|PF €iάh#vF1^ZΡΠι(ΘAΏΤ+tZ3β` ±pΌμ²ΛόŠrΈηž{όΈ₯wή9|ϋΫίΧ]w+'ηΜ™γΟA<(.σ ²’)O}³ό† H“«7˜q!Ά£ς/’Κz ₯oέζ8‘fΫϋBCΔί•υε±πv€Ž°mœGψn{œb}ό8λ'Δ5ΆϊΊtCτ{ϋν·χό7άpCΈι¦›ΪΠοW^y%\zι₯αϊλ―«Ίͺ;f1wͺ~:Μζ½ κ‘Ρ>ΰΊ„Β™ˆΨΡ`Κ€ώ>^ITνψ*ΠΠ^Ή †^` ο ©`U…ύΓf9½DQΎaώ6<45ΨQfωxξR^ OnύΧbNJ¬ΉQY Ϊ¦ΡΗΙ”)SΡ}wΫm77€Až@{τΧΆ„ψ«3€Œ±ˆσ?Άu£ΡhLgr“Φ2ΨzΣ83HΗ|gΡWει‘Ӎ3:|8―ΡΦ­"‘§{¨¬•³Νt€E^$Ϊ`U§._#Ύ`_ψΒI—^r©ΛM ίΓΚΪθlώσŸ;Žφή{o7h'―Φ€ZCr:’HgiΘL4ώ^0κΝΘ)οo[xAΖ ŒWxσZ“εΕhχ–N gV?Υ³\«ΛξqΓ† sοώuΦYΗ#΅~ψαα‘GρΏ;οΌΣ…%LˆθB$—6Ϊ(¬»ξΊ~ TrC£NX”‰ΑD`ς&›ξωEψ ^Τ‘c?Ύϋέοϊ‚ƒ0Ÿ¨<μρ˜}ι₯—B|ଳΠ{챇oάΘ·(νς+ψ΅ΝξGσ,%Β’ώŽRλ‘΅½₯i~8rP΄Fξ₯>Δy°)QΜCΑ[sA>Βό!ƒ‡tΫόθ₯¦UνkYx†ξΒΞϋ͚cσ%-B#θcwΡ―ͺET7U\τ›s`ψΞaζλ3]hύήzλ­ύμXθχUW]εoDXΔz³V«τ›±…²΅ε]£Χ…’:r΄cšΉnLΓψjy]σVΘψh>ϊF/ ί(Ρ9^1BΟbΊ ]pΑΎρ7yΑn] #¨‘//†Ί|O©kΉ&|νk_sƒG„ϋDΔ¨oͺ‡~Ψ±„0&²P«tŠv 6‘©S‹™…Mΰ§)p[šη‡αƒ‡ΖρΈK=yΑpՍiŒn#TnΆώ‰Z ›z²Dܘƒx4Ν΅9ΝIT„Ӝ£PžΔγψu00mΪ4η³χέwίπΝo~Σ•±ΠιωvΌ*iώώχΏ‡ΥW_έ…ωΥNΏ΅wΰ˜šαΓΗ―­#;Ί,rKaήFΡφΦf;βiπ°0ΤΖk„ξΕ€―±ΖkC·Œ4aiό_RŽ©gρΪ}`Ÿ:bδw0j5Ίά€Ύ7›L₯ْی=2±v‰’c³fΝr™>ΞͺD7xξΉηΒyηηIIYs¦©©) xχέwosœv'_]wΙΕo@W&LX,ΜύΔδ3&Ww.ςΓ`.,ίό9ο{9π2κΓ‡γΜ럟Μ7™{`k;δ³3㧛ΠΕ+Μqτs]Ξ’ε€£vΣkb1`Ϊ-_nΉεΒ)§œ^}υUΧU>φΨcaγ7φ½'|άzλ­^:ΞM7έ΄ƒR«1ϋπώύ˜ΞΚBoaΛΑ™ΠŒi\ξκ“iαΣψ­g0! ϊ}δβIΎ£gjί’Ζt™Šcρ1ΊW/Χ.ομ!Μz€Iώ‡Α Lχόq'ΜΟ<σŒ‡…$4$‚’=χά3l²ρ&a…W0FoBρbΪ;‹xgή₯θξΞΤB„€η’‹. ·έv›‡I;ηœsό}|Πf χD§Ήδ’Krτϋμ³Οφ—0ˆ€φε/9lΆΩfnόWΛτΊΝ†›3΅³a„;ΐ΅ΡςΖy^†΅ ιΧAψΈ0ΐΨj <Œζ½ 2 ’}rVxέπ’XDI`=%μ|“ Y¬χ²lp›όΠφAƒ†ψΡ mΔ]Β€ψoπ‘:‘‡oΏύvχ’ϊέο~ηe~ώσŸ?ψΑΡ€ˆνΛͺ(SVJdš«už?Ž1MγΌ0xμγΓc„Θžξ^Ξc'd£E«Rd±&Š†v=έΩχ!„CΉΤbϋ λτ¬iεH»?tX Gί]=$ϊΠm―½φ K/½΄Σοk―½6όφ·ΏυΧ`h?Ž e΅ΥVλWχZ9ΪsΠζΡc'ΨήnnΘq|&Έ‡&wfˌ:š>΅ρ<άeN½ΦΨ~1Šm gίC¬4Xχ)’gκημ“ψΩ]Πώ›£=η}lό6HΤ Ω_Ε?!ξζt@ήf;fdτb£γώ΅8Ζ >%ϋ-·άβΟχΏ”°Ε|ΗwΜΕZ,m|ΆŒ{xœΏη`BΑyΆ^Ψ@ξΜΈ·β2&{ŸkΗόŒ?!'΄j!žkρ›ϊ—½osšΒ3$ΜdμθNŽŸξΒ$ψcs.j0žš1ό;Bο`@γ#t˜œ¦ε΄ΣNσ ahsμ±Η:Ž<:€uΈwj\ώ·8ΐ l[pHr:›š(ό4ω7rςAΞοuϊ0—ς7’ή€œρȘ¦š]ρM„¦ΰ,ΰm½B—W6fK.Ή€GŸΩ|σΝΓ·Ύυ­πτΣO;Ρ&λ'žx’'εh(—›l²‰‡egcΪΒ­χ³Ω’πεΡG Ό—υΞ5Q™†RΥς^mx•—7‘έδ9eNž<Ω―u5oή<ŸΨΌo̘1Ήσ;ӎB陼ϋ·‘†ζ¦ωb“ΔtΖΒSrΜΣ0‹€ "X mͺ₯:ΒT±yυvc³ „šL! S‹±W4¦ιžΖzψH ]laζ‰LcΑκ²1Ϊ<Ώ%τ7εKτ$)Oί@—§N=τP>3wξ\§ιΠο±ζΙΑPλτz<Β"ͺ5›ΐΟ'ΦΘ…+xχ64YΩH΄˜PGηΧ:~θυ―D6j4ΌΙΒωΟ3~³…cBόήιΦΜαΓ‡…χζjΡ·‰υ–›GΆΖBΫGΨΪ;ԎR‹Π=M Β0ςCΎm{ Β χρ5s±‰ίιž7V~)πqΦt?κ£pmρʞc΄{‰ΘWFRٞ°ΦΒƒcΔή`όΖΰΛ㞨l(ο°ΰ¬‘ŒEκDA›d|μ;‘i0²!ryΡYRΡo Ήq8ϊάη>η|82!Ρο‰FΏ;+*ι彘hπ ΑΆ―ήώ`ΞIwρ~« FΫm¬Ž1²[Y›―f\βxddΫ Ωαχ FΜ¦=§eμvΡ]xœ!#Μ€έŒaZ1hκ‚.‹H‘‰lό„ρqένdŸ1€΅ΦZ+ΰ«ίl~ ΙΦι;xΙSry:ωξzN>zτθ0zΜ&œŒ0EoΙŸuΑ ‚?΄΅t˜υΑG£ζS=㴞Ϊ>ތiΨc̞gcaL<²ŠςcCλεl“kιφS*’Xωρ^ΚΦ\sΝ°βŠ+z”ί†œq‚G˜ΐΰ‰Σ6€z ¬SθdšLWYˆJΣμςοΎύΊΐ€*8ή/π’™$iκaŒ–Œ€^HHΏp„<ύΰ<–:39”v«­Άr+x,,9vŠγ;˜x…˜}Ձk)i”žφρN”―Λ,³ŒnηPI<δΦΰp$/%?xΘ(WtδKο•D6@ ίΤ0/kLcJΩ!C³ ρ?ε{{,9‰ψψ™·?zΣ•ΙgϊN΄1Ό`Ÿ½7Žτ¨Ξ†―Τ›ΊΥjνΛH³H³Ϋγm ή±Ωχ$€Oΐδ$89Y$ΔΙ K8 πqY „|α7–ĐσcL~Β’˜Ε0Άρ6c{Ζ³hv͌vυΎιžjUO¦%u·z“ϊ–­ιξw©ͺχ©zoέΊυΤ½-ν-ΊΨbA)Σ§•_μχ$Ρσoqj„wΒ>£^!1\™]¨FΑ[ Γ”!Σ Θ‚ά°8ζ»\•κΩ$GΗ w“ Iτ4ΖvΣT}ΨχI”αΌŸϊφβdΌΐ;uGΏψΌώ.+w(ΏΜb! '+Ϋ_«Ώ©»uuχΘ‰3₯ €tc#YυΓ`ΎHwη‰Ψ Ό°™λR,W]LƒgΐώJB4 Q.$CŠ€_Oνqύ,VVŒΦ;/Α„ ƒνhΑbΦ…¦X±@Ύ&”ιΡ`ΜxBηά_SρΠΣ4U^ΏzϋαU|\\ ΕέδAΑτΞ΄’βˆσσ lθ§Χx©N΅”z@ΐκOύύβΑ:ΥxˆΆφΪ$ “†ΠCΟΚάhΟdλX›i©Ά 8/ΉΫ>v Ύδ‚ut QH†@(bgΝ— S ΫΠ3;77jͺ" Γm―Τέlίt:ΚJ]¨β­―’Ψ.v Ÿ¬‘υλ’z$;΄MΈV莏Λ±cΗδΡGz‰ω—Ύd/3Ÿ―~υ«εέο~·μΨ±CŽ9"χέwŸ —τΑ~P{μ1aΨ$Ζd΅/ΜE7ηωaΛ₯’OBγήyηΖ΍7ή(άιdΩlyn7υNa§7ΓM‘‘YH²e²Žό‰†9σΙ™[$‹IE4ŠΕ#sfQφ‚žΰυ€aDφ,™ΖήΧΥτ-Ξ$Σ0@a'Έ» %£~BΌX2MM+Ϊ€…³MΪΪό21vŽλ\ΩDyB.Έl…χM•A€ο†ΏU~[OjQ„nΚx¦YΚ> 9Ξ]N‰0ϊd5LHezεΉRNS†ΗC) Γ¦1R=W-W%Ο8œγ‰/~žš°ΰΊxEάξιΩ-eΈŽ±•i •α\ Q;VNu©4ŸΚΪΦoΉΉΠRχλρ°:81§ΌŽΖ2 ŒΚτT£;4KΓ΅w‘(Γφ‘Η3…_˜ΆΣ«ρ'ίM•A ‘δ7=ktƒL ώtΑFBΊLΈ"lT"2#Ψh‡ΡT~ZξΘ(ΌΡXσΕΩΆL…Ϊ+Λ_»υŸ# υέ½]ƒ7HθΥN―ΓxΌ`k\ϋš‘L37³ΐμɐiμψΌτέz&cοΞ±ϋηžΛχψ*Ζω)μΨΐΐ€τυ ΘΑ£§Ε‡…ά&ΖΆzΉ,ŒΙ&)1„x"‰sΓ† Λ]­ηΦ)`η‡]xδŠΓψοq,α}Ί‚ΟO=š‘Ι'ΰ'€q΄Ώ――‚₯iΦΕ ΠH:ψrΈΠ¦M@j&_¨l \τbθΩ©D̐§››uϊrx–ϋœ]['iƒλΤChS‘-VSν°Ί?sΙ4όnΟΥv΅)Ή$2M.XtφόσΟΛγ?.?ώρM μΩY,-€«ΊJήψΖ7šπK$ΡPΉγύ|)ξΈγκӟώ΄‰ΗJW’ŒΩΗ—¦˜tο½χΚψCγm`ttT>ϊяJ4š10S)5ω-ΘΕά|Ή;ίΎ§…χ³ŽΕLŠω φEΟΝ³QΎ+.u"4J$t<Σδ(ϋƈ2 1¦ ΤT}¬!Ÿ‘H€a˜§ΩhΒμΊd›Σί«_ϋυY"VΈ}>ž…3ώαέ!™†,ι6μ8ΧT9μψՈςΫΎσ KΑ…dš₯mdθ˜t%ŸΘ+Χ2šσbφ­;`Ο'RβχΈp‘1ΫΆ\|ώ ά ΑΰtO}ΎΔΠ!$ΣΠUκ=ω*Ο±F–αAκ ΤρΑG9ΟΌ=G/>Ϊ-rΥϋ΄:x8–!Σp– ²ύσHMΥEΐΘfΠ’Π·³Κš‹΅ 'ή­^ψ*έ"Ωώo&?•.­ϊωΫηγζ nφJ Σ|*,MK+ΫEV6΄d–Βf#˜šΚ‡€Υ­¦œdzcβΏ1|§gΘOM•E€!ωžιΰ± θΥ@Ώˆα’—ςu£²ΠDΨlšdΆm+[συ•;ε™•iλλΙκλi,ΖΤΧ7 ΚΎgž£™9irƏ ΆvŒ ‰Θ€lΨ~y6ΤV}=₯Φ¦PΗέ±uDN}^¦’iΩΰw†¬bΔηͺͺHΉλ'IΛΨLDvmۚ ΄ͺŒυζ²!`eM#ΪΑν³{αy:z:1±‘δC—ή}θ{mθΣΊ–™’JγZ3Χρmˆ'έ¨XiΔ ΛŸ6EΞ=Ω.|§Τ3MaΈe― …BrκΤ)QφΝƒ>h<ΛΨ ΨΡίρŽwΚ‹^t‹\ύυ&|“έΕΑkμd†Dz”yφΩgM^ŒΙJο6$άΨklžΛ}>|XΎψΕ/ζ½dll,οq΄„›dF5M%!XΨε7ovΗ.˜vΔ&qŒ;¦Τˆ_΄«Ύ‰Βδ N±Έ³Š›ιΟGγβ‡gMΥEΐ*oxΈ36:ƒ‡ΰ¨Ϋ‡ΔC 3@ΆΡ€T;rA―ΕΧnb›α}&ο4›“‰ΨΘ4mψSΉQ‰6Y.OΛI¦…GϋڍǓ’8ΠΛ£η–AΐΚn.Œw€LƒΛψ<ΠdνϊF€g2 gz­Ι%ΣΨϋ—)BO)E#@#PGg7t;iHfwz Bμ™ šE*C΄!©]ϋaΡ―κ;ΦR€L†Ύ—„сžΖ΄-VmI7ΫφΰBmI옡’›e.ΦRοκιlh£PIΰκMK"ΠΥՍM^pηΔΩFnVsXςžBN€±Θš‚₯ΏƒιΟΌGεJ!Θ~ 3Ω†LΓeξI‚ΈκΒΞX/Bhͺ, ΙΧέΣ-ΑΗ2žiψζόαb$c³ ιήΨ‰y”zͺlkiξεB€6σM›6€‰Mw±I‘ΦAd]ΨΈ‘ŽM™ΠΓ##πt­ž˜ΚΥ&k!;ώs½νΚ={δηΏ|a–:e¨ΝiΒΨ…Κς¨μ­sρyyώά”άςςΖ«oY2ΦL2!ΰrΡs¬fnl*…šΌaž±Y7SG™€/2Ž‡.—[‰Έ!Υh; `….ηxC;€΅­π{£¦‚We,XτDCO2ŸωΜgδδΙ“YάθωΦ[o•7Όα ΒK[ΆlΉΘυ,οg"ψό³ΏΉΈΜΠNLSSSΒό‹M7έt“|όoώFΌXt"CͺΠΔrnnN†‡‡ ½E―[„cvtn0ή ŒΰW € Όp’&ΫΜύ§*π=3q½‘!„`(¦‡šv°;­HΫ€*ΝpQ!”9$Σ€αώ2S b€μŠΕNYΊ•ζn8MŠ@EX˜-Π»IgεvPœ&&7½ΑΩ₯%–Μο0ΓpœŒΞI[@HœΤT](/Z}Υ‡σ.8ό$u˜”wζWέϊ5Ji+Ω]]š 2MΎgOA¦§ΰEH=δCG•7BBφττΙΡS“Ζθήδτ]’½υLCJšjƒ€ \|…a™€v’#uƒAmΪ‚₯Z2 υm£ζpΕ^£ΑΈ vχQνj¨%―μόΊ――σp?Hι3β‚‘«e―ζ9ηγ!δ’ΑΑ tΒΥd¨χ^‚νš)Ψ£πFΣ {J σφ4ζDέΨωoͺ/ΉI¬ژhξ‚χλX(nȏΞ€!―ΣΓ˜ΓƒUW¨=τ‹JηΦN³ˆ[ΩZh=0ΣE|<2edqΎp|ΕaΞ ™>Ÿ€gšΘ4τZ ²½8 »š²Βεφ˜°NΝ€=†Bz²ξh™»š5Uξ.LO›γξq”‡vX6Y%†ΚτΠdDz³ΆΘeοΥ“Š@ ΠΧΧ'}ύ‘§Lˆ žή Y~Ι€z <Ω€p} Πe<όΧΙ£h5j€ΧβΩ,£ΣQΩ;ΰ‘€ήΡMVŸ«­ͺι‰($ Ω{x!ž6o4^–V›―ή―”ΞΓ}πXG―4τvαν`/ζ›Š!ΩΜΓYCΖ‘ΪPΚέ +η—ρLs!Μ“΅―|§^Q)μ{ΐ9ΏσO=Ӂ6…O__?Β8½C^ϊΧΛW\!›7oΎΘψdγgρZώY ω}qβ±7ΏωΝς«Ώϊ«†ˆC― Lω]|―ύΝK_.‹Fu?IΦhοθ’σ3³°FbQ–1ΣΔ IWχϊEΖν“]Δ­nν΄4.ΔrG՜!Σ€€ !ž{©˜wL‘,ν0Δ‘iK2`₯£Λyώ¦KcžΣ€T’ Ϋ;:εΤΉΉŒά^bŠ=oΒυE ‘†J“¦κ `e3υΆ ψ\χz/ΘοκΤDKΙE ΕΫ"νπD―σΦ2ΕyχBJ!>y<’NΌ[LFΎηΡ{.ΧE d8q₯Α4FΌοt‚ >‘f™Χ°™‘¦©6p—¨c' ντitrμθΡT8遞=6{ kOiqΆdτmz‡ M†7UŽΉšr @ΩΫΣΣ+ΗΟΜ€–&m\9JCQ…˜₯ͺLmά»{6ΐΣ’Κφ’ ,βbΚ †]N§RΨ”Δέφ Τ€ΩΡΕMI™*Ed§—–€@OwOfW2Β5QŸ^1QαeT‡Π`Αρτlλ6WΌW/Pꁄ£άΉσ2yδ—OŠoŽτXM‰WL󰻇gOΛΆν»‘ΗθΈ°R qΈd¬Λvl—'žψ₯·ΆI»›δΟδgΠ‘ζšΓΡ‰ lέyΉ —!WΝB(/΄³ϋ`φ“ i&7*)ό Oάͺοε‚SΥο\£.NnχWϊ ³žix‘’iV„λΒΒ;;τ_ύΥ_ΙΠΠΠ%“»p`]X§0εw»0dΟηΗs###Ή‡Jϊn'YΆ¬™™—h4j^@ΎŒ4bφττdw(,Ύ§€‚ό&Ζ”ξhΗ’μΩcΠΣR¦­90%ΰρ€.θJSυ°οš%Σd<Σ€$€φRχΔΥo–hΫ„ΖOΙ‘ ži’ Ǎ|’ŒΥ€TC‚„ά=φ :&ϊavΦ–Š‰’ρl’$ΓυΩqέ^‘Ÿ•Gΐκ-τHωΑE>%WχΕ%X½ƒ]Κέ}p;L€;Όΰ:yΑpOr ε8Ι«τ>¦I¨Ά/’άΨ2Mhφ Wr̀萙 ΅ƒΠΥkΘ•¨‹ζΉ240P§&dur&«œƒ^QNΨ$­ž>š ΧGU‡mbMx*bΒS[£}ΧΚYΎζΥX°ΏmΪ΄E>$σρiirC7XΓ@Ψ@Ώ ɝη·lάbBp—^Rœ‹sƒίδΔ"˜;ΞΕ$‚°pLηι%ZΔ-v|$aΨ]':‹γ Ίφ²Ωπ&|Ζ±I)JΐγyWvΓ§Κτ <ϊo}"`ϋ'I˜ΫΆo—άχ-ι…2‘Ύ•+œ JpκΈl}ιΝΖfΓl~+ί¬W¬l{sάΪΎ}›<πσ‡ετΖ,Ώγ+_垷IΞS2E―Ϋ΅:J«)ΚΦ­rεjΊ@α°?’L3ΟΝH©ά0OP0/dψ§t""­ώœˆΣxα…θ•«Fΐ’iΘ) έKm૆΄lΠVΒvaβ\·QSΡ~J Φξέ» ^‘HDžzκ)9wξœ ³ΔΑ› & ©|Πό1|Σή½{‘ΚNΜE π£χ’RΫά<:$<π€<ώψγςμ³ΟΚ©S§L)ΖIgάƝ;wšΊ0,=κh* ΫNT’ΐ6Ž­t•Ζ6€a?“NΰλΕNnM΅Cΐξ‚ b‡@m~ŸjΧ¦dN’© $ΒPΰŒΌ„gς£s1cδ·F:ϋŽΥΈΊZό:Bΐφ)’2Ψγ±9ΓΘΟ??ΐ¨œB˜2½wΑQ׏bυλF‘¬|Ά›•u]ωuZ9ΆEGw‡ΔB iΟ4τΌ˜ΡwS ΧΔα]¬£―£‘ϊλ΄ιλ걬lΰξΥζf‡Ω‘ΪΔΈίYAŽ~ χΔqθαCCύf1ͺ *CcΨf4:X2MAPWjΙ4‘™ˆ€0/"œž!Ψ½k½γΪw¬*―•Y3X}™ύmxdD"ίύ/iΰΑw΅ Ά5Δ™”ΠΜ|α%ξΟeξ§ΝΕ–σ°q: —žiΒ±8μ(ώ†6/YΩOuuuI[ΐ0OqΘkxςΐ“tdu+Φ-θσKy4d‡O#ΔΣyq‚τ·yσ΅ΤU‹V·2Ά\vωε¦ŸIΘΥ}nXNh–§Ž,!‰§–/ŠsϞ=FφΪ:•§ΝE(Ζ3MŠži°sQBO†M%o’ώΆv㏧ΝηEΧιJ#Ќˆ'ΤΕ­g΅Wρ•σ·œ Ξo)Ϋ™ψ½QSΡd; ž>}ZΎόε/Λϋί~Ήζškδή{οΝ‘μ€eίΎ}ς—ω—f²»Ώϋ»ςξwΏ[ΆmΫf€·Χx~/U@Ωϊπ%»ώϋM}~ρ‹_\ž'Ož”ύϋχΛwΎσsŽ!ͺ>όαΛ ^π‚KΥ…#ΐ—‡»°ΣˆέΝέRhLi‚§ƒ8\Οwt\m ΟM―,7$;q§<=ΣD°­ͺ³ΥΧΠ―άψ–’ίŸΡxŸΖ€ƒσ›<„αbžž³θΔΚ΅RςΧ{• ™‹d2ολyZσ$γ™ςΌ³³ΛœΥ~™€ βΔΑ²ΏIζΠ‰D…_&{CBλl—“ Ω`=Σΰ݁ §Kωx(.έ:Ζ.ƒ‘ž*Το::$Ÿ·ρ0†…¦ »Kš@¦‰Efα½f—΄€Μ‘©6δ’i8OU2MmΪΑΪΨτL3‡π”Ω<>=<O‘^Ώ7kΗ¨M-΅Τυ„υeΞη6acQŒŽπ0°―4κΖ||˜¦e`` +Ol^OψΥϊY¨ks±% €:^ΨGb ,pΓΓ/Ξiͺ<œσ ΘθΨ‰Γہ―Ε‰W |uqmbπΘΰ٘ήm4)k αα-2°a“gNŠ«ϋ„z‚§χ₯Ζ„$‰Ηΰ‰²λ,[Ν£ͺ½f­΅xyλΛ γΓ‡δΔΜ€ΜΔΌγsˆN]ή’2ΉQ*;1NN#άυθtTv o’‘+Q”橬 +Ήy" 2Ν%žihΤ#™žiΪZΫ`ΣSΚͺ_ΕΝΤα¨os3?Υ3Ν*ΐ,Χ­φxE8Ώ56”…Ή.³·D›r΅ς)ŠLc…ΟΨؘ!ΙόΫΏύ› χτΔOH<žq‘e―a§ηΒ;ώ?ύΣ?ΙσΟ?/ŸϊΤ§ςjJΛ~πƒΘ«^υ*Ω΅k—Ικ%πΔz¦ΑAμ’ΕkdΒ>Eƒqκ‚'%/Τ‘¨d‚ν]Π@¦1]š9₯ƒ,72!αΉσf!ΙΞΕšΊyTΚΚozr₯μΆς[uΌκ7‘ΕœDβθLΜθίδž1Áψ°Ω€ς]“"P¬Q±ύ­`rzJœπ*#ΝτΨ»’{|5X0πG'ΡWί?ο"=V&h»$ω‘Ί7mXI4Y(—V³r\m[e{‰lˆσ„Ÿ=ψl&4vΌΌαΪΕ³ΥΕ·‚$τqv5χυeBδXωΏψZύ­Τ#ύύύ†ˆωΔSϋ₯Κfg&Dθ%u…lš~ψ¬Œl2λ2—\£.„gΏεΖδωΚΘωmν 2ME3œ'M€L³τ”ΌωΧn1D+Z¦f¬ΆΆ3p€ΝGΖvbσ’ dšd<,­=νjΣ³ΈΤΰΣ†yβf$~W2M ‘ˆ"KuŽRDuwiQd;iόήχΎ'$ΠΛΜk_ϋZyΣ›ή$ƒƒƒζαμ5μπwάq‡!ͺάwί}ς…/|Αx…‘'˜χ½ο}f‚ΊΪ‰½ttT>φ±Ιππ°!Ε|φ³Ÿ•WΎς•²ŒXΦΓ&~κΫίώΆ|ϊӟ–ΉΉ9ωόη?o8$„ΨόμυϊYννζBC¦er>–ilΪ k­°\τͺJ @C„»—c £cIρa1PΙ4•@Ίπ<©τ`gμδ±qcΜO{3»b»6t5…η€W*₯#@‚Œ¬ba °e„‰ά^’hγυ"”dΉ¦Ϊ @N™H$LΌXkΔ―Mm³T«Χ’”Π†w!‚…Ψ4<Σd&ήi¬΄Dg££I¬šJ#ΐyKGG—œš&›Kā~ΗnIyžŠI$xNΊ±‘€ςCη7•nόω“LΓΏ`0hΘxYCΠJ«ω³Σ£e@ΐ„'ξmΓβtnΥΕ #B2 ‰6šr `Š”Σ›Άl•c§ΞI δ²8θa ΄°l ―¦ΠΙ;`γ2™¨l/ Λ•ξ’mΙ4τπ Νٔδυy/²-”ž/«ssμ’ΰDαnR†'ΌbnœΎΒs<˜„!ΏΚτΣ κ Ϋχ9ά½ϋ2ωΕ/~†Θ~μλΟSM$Ν8?.ΡΠΈμΪ}dV«ΉΞζ“η&=΄Ž`»S'pCv^ŽPOXƒ–ΡΩ„lνtŠΠKT=VD ΕΒ‹›ΘΡ)’²ϋ²έΖ’ϊɊΠιUFΐέiΣsΊ`ILC―_œ—[{8~'cAimB?VΟ4Un’lqτ~B» tπ3kCΙ^‘_Ϊ"°TΞKjΕΑijjJΎϋέοšο •τ|@^φ²—W¨<˜«ΌΡ`uε•WΚύΡɟόɟ˜{ΎτΕ/Ι(Θ/εHΆ¬Ÿόδ'ςΐ˜"ŸψΔ'δο|§ŒŒŒ˜ί| νΣ+ΈBήσžχB_HoθYGSρΨ>h˜P5σI„yΒΰΓΠ nw‹ΊΛ.²ίA2™‰ m: -—ο€%ΣΨχ§μ…j†Λ"@έΜΗ$A bΓƒx\ jΫ, ‘ž\Vn2 δC „™‹-ϋλ9Ηδ"’MOο%¬οΥήJyΝ?ξŽε§H¨ŒX-²ΕίOμ[Xg’Ν—k•’OEΰ)$ οKρΠκ% @/Ωγ‘)τEi˜Œ1ˆξ‰ƒ ηΆ©žA₯fΆx2:7ε7υo»ΑΓ.΄Χ¬b \°Ο 4Ψm›γ΅ΑœXΔΰΕΐΧκΥ…Χξ•zt΄††6JhfŒ1}Q …t)‰zy ς~α<ΊM˜§RrΡ{–GΐκΦΤχ(³)»™μ"€§ΫΉΤςΉιΩR Ύ΄ίnΨ°AfΟΞJ2‚°ΔΆ–Θ”§Ω~ θη±`LϊϋΡ†ΊΆ\zΈN°}ǎXΠ Ι|tsΞKΖŒ%΄»γ|hφœ!ίΈέ.C¦¨ΣGΣjU-ΓΓςΒ+/“§ΟΜΙ,<‘ƒK“ΗΚ] ’iBzΰ|XΆo‘gOΉ‹Πό² `ηί\iƒwί4Ό°gl(θΔζ α‹BΟ4ACN$©WSm Ν„ΊΈ’ijƒΏ–Ί2“ilVǎ“ƒ·™o}λΫΜ$‡—–šTς…Πk^σ!ωζψ‰γrδΘ›]ɟΆΌP($ϋχο7ωΌυ­o•_ω•_1ΖJ–›/ρ8™ˆoyΛ[δu―{ΉδΉηž3/©@η»O- ϊΝΠЈ-ΘA(…A‰ρΉ‹ΙΆ•ω‘TBάYeί†hœΠT}¨’12Mw·Dη*ξŒ₯!Ώ½3 ži2ιΏU@€ažΈs…ήgζ"δ"# z+ά^Ζ#³Bc€zΪ¨B“δ-‚ς‚œHLccσ^¬+ŽΙ4ΖXIf ϋ ‚=…°Ρι˜ΰ6ΦNΌU§¬xs4dΆ_΅΄`ρϊw v.²f@I8¦™•φŽ^iό ‰Q½<΄Ϋγ7(Ώ)T~ΧΎeΈΡ§½#`ΌΡp}Šzx<œ?ήžΣ€” rΊΔ— ’„œN'ζυ Œ€Ό…γ¬V₯γ3 ΜΚΖ‘MfnŸχR=XrΙ4¦-Αδΰ1ώiͺ<ΦvH‚ΊΏΣMGIγ’‹ΆKΏB|OΘYKIx&Šπj}ΖYωΪj Š@ωθοο“Α[ΡχΟA­£oΣ~kǏ̘@oπ‰π9ινΫhΦdX z^ΠΤΈXέƒ!ξΔ&ς“Σa9±B0_Šε“sI9>’«φ\ž%ϋΪΊT¨hΝV(ΞΝΫό°‘ΐΐ<½ϋζΨΑηα=•Œ=›6X&νΛ†ͺώΓ5K§aC‘ύDm(U…_ +’WΥ§§§'—«Ί ή_†M.K {œ»ΰχμΙ°T'&ΐ°F²η̏™™™‘'N˜»oΊι&3Ιεl)Β€=ήΣΣcBAρΖ£GšOόn'oό©0X\’p›ΗŽ«yX&Iͺα"-4Υ},™†}ί.ςΥΆV ZΊ±eΘ4”‡‘)V-C¦aψξbΦ€T.ΔφττΓΐΟ4i,Δr&l§Ϊψή2M,<Ο4½†€Z:i—"`Ι4ΤMH¦ΡT[¨Χ4Γ ™υLγΐΛK7»2žΰj[C-½Qp8šα‘ Λ!‘έΘoφGΊ'ž5^klΦFΑ€^žΣΞo)³]N%ΣΤK»°τΦΩ†Ϊθ\ ―ΚΌ‘έΙxJΈ1D uυΤRλ§.ύ E»±Θ?iδσ‚Β]τΞƒLCOd›· ’53°²¦θΜτ†e –ΘΞ Ή`Hω dšea+ΫIΫ―½˜«lκGU„Wρ1g½+oYœΚ¦bπ91?Υξ•&=Έধ;wKpκ8&™τ"Μޝ›°Œƒ ¬αι“²sχ„uν2'­η…ά+υ{c!@›Η«kΉ¬f†·ι•ΕgI ±W2οΡ™€L…brν΅Χš±SΧΤJ‚SoͺndόΨ—€3ξ¬XX§1ΪήΒ%n°°ΊH•ͺ₯Ε‹9Χ.6ŽdšΜw‡β£ΤE“iΨ‘)X’Ρ¨ω+τi8 ZοΜ£\‰ξWΓa°΅‘:ΰĺ°½X$dβ³π%ΥTœ¨ϊΑμLsΰ“3 &§€λ™¦΄\υr @EΪ’i80©Q‘¨–žeΫ‘£³Cf±Σ$C¦‰Β3 wΕͺ±Υ;‹C€;ζϋϊϋ±Λ5x©λωyΊ“OHhξ,7½Ω°VΉ-$½z5ИOγ=e‡έ±šότή ώL†Lύ1 Ο4tM͂\₯[yO‹;ς΄΄Rτ.E p(il‡‡ή™[;ŒB‰Θ tπœ/l^Tx©ze1PvΫYυ,V r•»Φi oG°8KٝŠ¦'ƒφŽvΘσΜb•κ:•ΓΏ‘rΆzΓΰΰ ™§’Ο’ΦKžmΞcΛ–-Y½°”lτžε°νFωmm—ΌCωΛγV‰³^ŸWΊzΊŒΌNΓf’•Ν•ψ <ύτΣζY8ι΄Οά‡γ1; “‡~؜ξΗSΎ{Μ‰"ώ!9€»Ι˜ΞŒYρN–I%3‘HΘθθ¨ΉΎ=Π‹T+"·τN,φutva0B,px3 [4Ά‹z¦Y³jα ΔΆ°ŒNυlP-δσ—ce^«―^ ° ‹έ°)ΠωΫ›$EKv‰p~ότhωπ`ΡΧΧ/1,Ίb2ζΒ&„nLΡsRL:::Œςjϋ½B?«ƒ€^ ,™†Ÿšj‹υNζ’1ΈL¦1ŸZ4>‘„ΈsΘ4ϊΎΤΆ₯tc<‡'šyΖό6 raϋΡiCh·σ#Υ)jΣ#¨ƒ“LΓΉ0?ω[Sm°ο7΄ϊ[M˜'.ΞR–'bɌ£&E άt#΄ogg7ϊΪ4d5l%ιΫ…”ΉAϋJl!Κϊ@rο67©žQv₯_C™aεs‘WΒάί₯η¬wŠ€Ολ3’³π~ yMΉS֜ƒΤΛι)’^v•Iί“,@ϊe `ϋμζΝ›±ΩŽiΞ‘#ΓύGvμΐ β|*2 ΛΝͺ_‘Ά­F5ν8ՍH 7^½μ;5)c!Ϊ- σΚΞ½* ΅@IDATkΕΌπwy›’^»αυLfΆ%ζ¬·)E€sr_«k1πϊEΟ4φΕΐ;2ŸŠ]Ϗ΅M΅C€2„^˜ν:¦ΪPjΧωJVŸYΘ‡Ν’Η†††δΚ+―4ηοΎϋn"‰;˜™ΐ!‘HDΎυ­oΙO~ςκixxxΙό‹=Α]™###ζΆοϋ&\Λ₯²οΟζΟΠPίόζ7ΝΟ­ΫΆf½¨δ{{~ζGΐ…>Ί’O‚DCƒ~“_†ΡT[ΈθGo'4δ³o+™¦ΆνaK§Χ1Χ‚0ψ$#ΨUŽφ‘_·!ύ¬vŒ£\θο€χ™σΖ£&,•Ÿii‚βσwdΗΖJΥGσ]Ɗ΅$ϋΉόzΆ’Π«A3&v6Μ“ρL}39ξi‘—>%‹ΧΌƒ€5΄s4oΘ4X ₯‘i> o›ΠΗI€±]sB© ”ΫJ¦© τy ₯žMςz*‰…(x0HΖψ—4ΊŽκςB¦W‰€Χλ•ααmΞ= }»`5\ό‚^{,"ι€1ΣƒV_‡Δ$<<>?μŠW_sΩ8 ύpυΨj•E€kbœ &b–LcΛΓہ΅ΜfθέΤ?4ΥΞqψGyΒωΉΞΡkΧyK.Χ@’7σ΅q°ΰ-rΆ#S¨Όξu―3OGO3ηόy葇δμΩ³ SPΙΛ$§§§eίΎ}ςψςΙO~œΈγŽ;„,λΥ&[ Βk0p3έ{ο½ςοώο2;;k^<ϋ.ώd]οΊλ.yόρǍρμ²Λ.3χλΐo`(ϊπξ“„ΖtO%Ϋ42 5Υ:|G,£“m₯©φ°Ϊ:Ϋ°{ΖuνD rΥθuތyΧ9_η\‘I׈+Σ+εΚΆβŸΕίΆέJχιω*!pα•©RυWLI1n½υVω‹Ώψ ωΔ'>aΝΎπyΫΫή&;wξ4B‡=ΚιS§δ>x‹—©©)yΓή ΏύΫΏmς‘ΘδΧ~νΧL}μ5'ΏΑ`Π„¦Ίηž{䁐σηΟΛ‡>τ‘¬§rΥ§ώšΉ²5’‘˜]’iζ“ ’ΐiΤh\Yά Ιm` ωμίό©vXC2―Νο2M3f#”Ušj"ΠΪκ“Ά@BΕ™έρ͈»@¦iƒ‹l†yT;ΈΛ?κLόΤT[ yΈ#€w& ΒŒRh4 B£ΏxΧΆ–Zz£ ΠW{Β‡ ζ·š&‡ z8Β!@§ΗHΥΓkίΨœƒ*™¦φmΑP7» ±›–2<Ηβ+ΒτY"d9νυρΔZ‹Z!ΐΎΖώΔM-ΓΓΓ2;}J6ΐNΒcΕ&’$#Α BΑφU;§,6/½~y, Ÿ;†rc4U†Φ’Όζ+“„'±₯]*ΰN_A¦ID’εGΩ-Υ«¨–€”+ƒ’{ΫΆςΨcH:6)Ž–žΜ;€χb>>#ΡΠΈμΌιjC,#φλ%+Ϋ‡€μΕΖσ<τs9:‘6GΩ5ΔκΨLRœ›‘;ίτζ¬ΝΠ–½^°ΤηX8aSεΪKŠ‘W œ:ΩˆnB{ Ύ›Ήβϊ{τ5χD:/―Σ&+~*[§Rz΅ŠZY§Pag¦λπ;οΌΣΈ™%…M†|Κ—¬ρνo»Όο}οCLΟ-&r ²Ά>½½½ςηώηςΔOΘsΟ='_ύκW‡š;vH˜ΪœL…B!“gžyξ»δˆ>Ώώζ_—w½λ]e'χδΓa=#Aƒ HΙDζ F|ώΞ5@¬ηη―Λg3Κγ d»“Jc룡ψΞΠύp»bΉ3Ά d ΗPmΣH΅ΰ$‘³{ J³βΔΔAšέ™Η™†ξθ{{fΌl$LκνY)ΓνXj?뭎TŽ‘νp9??oB„49h½Ÿ‡OJ‹Η―οFκ uπ¬$wu€L36Ξ„λΓ¬nžΖζΣ)ΣO렊 _+·©ηiͺ| #Γ<ΡΫA:™ ΅Z΅ΣZ¬'¬Έ»»ϊAϊά¬4Γ}‘ή{)7HϊΒ}!=°yY™²žpͺ·g‘·mn δkېΗH°ΡT]hΗ5oΌˆ-—x ۊd――:Ή’i–ΓKΟΥ7μΛ”?ΫΆm“Π̘=Σδ²Ι2pzό˜μΪ΅Ϋ, [9UίO₯΅«&ΆO0ΤS'6>™ŽΛή„€v5 λ–:+α*Ν‘ΔΌA§φΔ«―ΎΪθ&ΆΜj>§–₯”‚€:B³ή}αω1#[›3ΊΓ<Αώκυj˜§RpΥ{RuOQd>w.ε½ο}―ΌψΕ/6ažHR9yς€LNNš‰ Y°F|Ρ‹^$7άpƒτRξΦ’r€’π•―|Eθ%ηΟέ%αPX&&&ς6wΡƒ ½δ0δTΉλ”·ΠuxΠboˆl 0;aΔO#Μi­±Η^·!¨ΫGΚ•mlΫΗm›ΤmΕ€bN—Σ„‰ΗCƘΟvQ2Mƒ4~=&Iν]™ΖƒΠNvZΝIE 1dΫΪz³»΅UŽΧ¦α(,φ*ΏkΣΉ₯:œγ‰οάψ™ ™Ζ ƒΌ€° K‚€ ²]“"P-8Ÿ‘χ°“gc½•‘v±ιξ‰ι™†σ0&«™ϊOU0² Ήi΄•εU­„v -- Σ ]’Qxš μ@v»3apυ}Ή.=° μ;O»Θΐΰ,τO‰ †zq’ΠΕ°5Ή3φ% Β’*I8”χ΄₯iͺφύ'i†ρΈ˜mΟE"σΣΆ«=§Ÿ•Cΐγφ˜±3/bΉ\‚Ό%β•JΖvΥγ1οKήkτ "°€jΨά+νύΠWΰ™ϊ½4{ οΣN3…~ξCθΏ ζi¬μZ¦U¬2ƒƒƒς’“ϋό‰Όxs«΄΅―ήΣ15—ρpZž:=…Ό―G8²U~*-NXτξλ€G_κδBϋG€ρ;γ•ΠλUBξκ^έέΣμΈfΏ―.G½»œ°M=•dυη’ΰq!Ž!ŸH–™•3gΞΘάܜΑ”n³θ†;q¬7ŒJ~ΥUWΙΗ>φ1ΉύφΫεΩgŸ•ƒšPNάYΒ] T$ŠκΚ+―4μY%X'―ξ5`ϋ²½“q Dι˜!Σ΄ΰ·βΊ:\Λq7ΫΐΎ―ΜOΫ€¨>'XΠ ΑξŒεŽε\ƒέκKЁ•0d„™>=-­πd `P\π0Ρ¦3;~―œ›^Q ¬ gή*Ώ+pqyRv·ϊ[%u&Πa’λyz6πvψΠF…ξ6/\½ZΘ‡c~νω1γ¦0 ₯αiΜ’iςέ§Ηͺƒ€™χͺ­‘:`PŠσς:ζG©2ΟοΝκ::Ξ€^R4~Ώ_zzϊεΔΨ€ψθ RŠΩρ 2M€wμ’ν[ ΣΨΎ\tEτ†‚ˆΕbΖΆI[a!αZ0? ψ<ζXφ.ΧT2DΗ&ΨI>CZ’Fϋ4tςDJ\~—’Ϋ3hιΏki˜¨Λoή!g§Ξ‹›V›{–I,t^6ο”"0©ξb`Πr`Ÿ ΐ>tΝή½ς_ϋκi#B=­ΖY&{&₯ρθL\¦CQΉkpvΞ©ύ0§τk]#@Ϊᙆ:9 zΖ†‡Ν₯όνρpƒά2u]?Θ:­eΈ™”Q6l€uϊΈϊXk’Θ4|N”ΆC³ƒ3τς%ΎL«\m>‹σ²ΗωΙΕΑΫn»-Kπαd˜υdI¨Ι­#[%c΅u3Ψ€sbGΓd2Ά<βΚ]$Ί‹ΎφΒΎΆΫί΅―YcΧ€4B=₯"\„ΝΌCͺ°5vŸ¨ΕΣ·΅΅™Ιoβπhv!–Ά›tαϊ° KJ¨EΛh™υŠe7υΜάΝΟ“LkTdš” βU½§^nΦΛ CΏ- ‰(62οb$CΖ@ p©{β:iσάΉ¦κΰυΡ(Φ$Ιμ©dJZ|0¨BΆkR*…υνξ9D}ېi–b,fί₯/ζΰυ·UC―Vͺεˍxτpν†Ηͺl]ΗΌt΄zql„ y‹«ΰO’ΨωΆΜƒΤ„—.iα•J#~ εΉχiRΦ:΄Ε m”#£?•6z¦Α¦ ’ζ#³gd绀η5)+!ΐMη{Άo•}cΣ²·^M[šM¨§•ξΛwK?„ώ|h2&½Β0RLv](ί=zL¨l?ε\ΠI2 <ш“^h04k™)ρCίnΦyaM›Œφ“T*eτ9~ςO“"PO¬j -Ι…&Έoσ91 ƒ%a`σ±yΩLμράΊπ;•Οώώ~Ω°aƒωΜ%π^^cο΅yιgi0ΤSšž < χςά-»ΈJΛYο*\ƒ½5δ3ŸŒ!’”υžr"@ΩΓp!τf0£ίkΘΟm»r–©y)+›9‰hΓBl2l ‹Λ…”DΈΎd†LΣΌš­+6?ύ,223PΩP:ŒεΊ“²»n_S θ: B ΄hz¨αŽYΫφέ*Wyš"ΫΟHΒ₯ΗƒXd† ώhlˆΑ{R@wTεΗ¬N#Κπ4@ž"j•‹”αόσΐS€£YΙ4y C«DΐΚjΊ“οθμDHμΔ4Βb/&,²vΓΐ?8>ŸŠ@ηIwO―ΩΆΚ*ιν @2ΝΩ³g1ŽΊ%‡ΧΡ2M‹Œ;'αpΈ€τ’r!`ή!ΎxΜk±\ΖΌνE’»}χ–»\Ο)υŠ€νΏάΐ100 ρΘ4ά.Α %ίlzŠ'p|ƒ΄-„t΅ΧΧλσh½jƒ€νŒΠpΝΥWΙ±©œšΛ„.₯F4K5C ₯δΰω9ΉωϊλΜZ[)yι=Š@-0s?n;£]…Ϋόf” l~šͺ€΅•Π~’L&AxΒt%ΣTΏ!΄Δ(™²ΟN~κΤ)ΪiffΖttΫρsKeΗgάa;)½ώϋεγΈμή½»(φ*σ ‡‰JΓGρΕbšžž61ŒΉΘ‘[Ζ͝uYeΒά„ψ›/)w Ρ­©tˆ½aob0š©†m³οsΧ;KEΐBl Ύ)ž4ΥΆ‡•Wlϋ»ζΣ 4μƒ ΐƒŒ.8ΖOϊι™f1΅‘ΐ©“‡M³mπΗd?λ€j Y ’ι–nδΉϋ’<³ K2 diΐΧ€TΚφΗΦVΘ4“Δ8ίΠΑΣɘψZΫ ‡+9 ν°R4q¬ε§ξͺZ ­κœ§§²2„φ΄8AJciR*…ϋW'B§RCHƒs‘t^έΖγˆνFγΛͺ˜ΏƒL“ˆ‡>}GΦφU©zjΎhۜ„έ±·»K’±”΄ΉšΔ‡ΛOŸ8)α΄5U μά'ϋN,S2―‘Ν „šμ}Λ\§zG€6ΒΎΎ>lΒΓ‚b|VšAšO'f1–ΜK/Βώι‚o½·`νλGYHδ/xά}ο·ž));»A"‡ΌΜX˜ «#―₯έ#ϋΗρΩ”œœ˜‘›nΊΙ¬£± φUMŠΐZA€ιΑŽ‘±|ͺέ8€Ηυ6]λv΄<+Σ†B>€&E ž(ŠLcIvδoϋΫςOψOrχη}Ψ\ΘΟ—>ς‘˜Γ6Ώ|ΧΨcφwώμΟώΜ„β@ύww²ll¦o|γς΅―}Ν°bω’šΈ£σΘ‘#ς‘}H^ωΚWšI—*…’wρuΔ­»ϊζgF!5J^ŒO­~ρύ±†|K¬©U]΄ά \$gf166ς…«τ›"PYΨ3;šθ!)g§,<$± KοnLvlm4χ|Pn[]Κ~ζ»NU+³©Ϋx0ωfhξ~5F{x#±†;˜›9Χ€T’»Π ‘Ɓ@„τa'«έpPΕͺhQyΰœ™rƒŸ*ΓσTƒCOXl₯‡§1ΥΚψTG‹l:ήηαU ³„…οθ„ρρ}’ “&G»ΔΞ= ½Β)NF\ƒpδ€ί@'OΔ‚fαTCWΆ³X9pvG'ΤΉN„Β˜ˆ&Ε2 εΖΉΙs2>1.[·m«lE4χ,\P9+γmΖP 'Ω+Ύ,nv6Έš °ψύ­¬%¬ύ…dκϊ©Ψ¬Έ`gOγΣαh6cΒZz­km‘«vn•ƒγγrύ`‹τ΅:$ϋEΑ —’ΫΙT4-ϋΞεΊΛwΙΘΦ­ί*υ„@†LƒυKθ™Dv7Λ%A¦iΧυΜ7Χ0γρΈ±gΡ†ΒοšzB (2­ψΟ~φ3yΣ›ή$Ϋ·o—ααa5§ΌςJ#tž|ςΙ¬Αpγƍ&Δ'B]ΖM‘ένn'¬6ί|ŸV‰ŒD"rΟ=χd/ωΨΗ>fΎσόιΣ§εΎϋξC,i_QWΙζ>ώΌpχ‰¦Υ!p‘-©e˜ΙŽ­.o½»tψήΡ+ @|Wt*ΛrήΙΆ a¦!σ?ΙΠ8RΞ"4/E  πLN ζˆΓΝΔΎΙο0Φ―5™£ϊo  ηS1dαUwέK½†!BΦ ―ˆ‘ί&Β%dyά&Υ,ϊYiΈΈκr·’?&ΰ†ς"#Οθ}•.[σ_ΚnκέΤΑc±XV–/—ž­4™ $2 $ό5suz&E ’γ|:‰0Aμ| ί&ηŽIβόΟL?”fθαS?χ°4{p­ ž#™@ŒEfWfK¦Q#M%ώεύθΡQρΐ»Ϋ@«SNΕA”ιρ9Δ †Νθθ1Ήώϊ”€W πσδΙEΎ Νt£ΐ΄€Ή$γ-’^ω‰8ζLΊ‹9˜ώ»Φ ™†žR±9τΕβsfΞΙ΅ &Κ,Φz+WώΆo°Ώμ½ζjωΚϋ 9κ0dš’J₯½7œ %εΘψœΌιu/2k{ΜΓ–QT~z±"PC¨Os3Iν™ !;8Θ»0μ‘h£ΞjΨ8(švoΪNΈ9ŒŸΊŽYΫφΠ/Eΰ‚εs±J;2=Αp‘Η^ϋΪΧΚ«^υ*³`©O}ʐSξΊλ.Ω²e‹ EβΝ—Ώόe3ΐώιŸώ©Όη=ο‘žž“w!ƒ½Ζ‹ΈuΏρΏ‘l'›ΗΠАΌώυ―7.ν’ΣE•_β_L,Ήg‰Λτp°/d'Ίό#Y@ϋPjφΑχΑ’ihŒΰϋ«©φdΌM€ΜE†!‘ΖξXΆ2―φ΅Τ4Λ ™žhŒδ¦{KiZύνβ^g‡z}FN$(Η)ŠΡoκυyΦz½ΨԍΌζZηέt'B4ΰ«ό^λ-Όφκο‰ΛνnA™ ξ¨β֌χ΅χ<λ‘ΖvώCύΫ‚Έ)D ‘uΤΊ™©jΖ~Šj©wΘ:j›uV+6μκΖζ„U₯„)>Ϋ=Β¬:rΊžΓ'ι(Ž!”ΈΫΝό1=$š”ncά·ω™ τŸ²!`qeψψ#GJ‡Χ-ύΎf!t;π;ΕοvΘ‘C‡Œ§1Kl*[4£Όp ₯™±Ω½@z΄&ΗEW›ΓΈΔιΑΒK4†±Ww1/‚HQΈVA2ζl,„π”a aρz[₯³³s>‘V»Ϊp|γ˜uεWŠΏΏ+‡&crYΛΨ1 ­ ΥζlG§ΠMšδšk1yΪ±³Π|τ:E  =›)Hž1 F=FΨHγ7½M«M―Ά­Δ΅KΪN(·hO‰FιΥSS½  οζΕ6ΖΩ³geώύ277'ψ‡(wήy§q1 ε—Ώό₯όέίύμάΉSn»ν6³Ψπ†7ΌAn½υVyΗ;ή!<π€Όε-o1dšB]ΫH›6m’Ο~φ³ΩκΆϊ[³ίοΈγΉύφΫKxΜΏ₯….•Q›΄„/lO.&5a—,1 „¬τ–2"@Γ}±½νXHšjί† iΖ7Ί#ζϋ£‹,΅o—Fͺƒ[[ύ vΎ"$&)ϊνβrΊ ’Ί|V²π9™ ’ŒόΪ7ہ;U(Γω™DύρΈΥYk_S­A£ @cFc Βblži¬λβFΑ ^Ÿ3ΫƒΊxf‡}½ΦΆqκed7Ε7V0eΕ» nεyγΐ OZeό ―ΣpŸyΞτ?(¨τ ~‚ΤŽNΉP;όNCŒ³ή"­ώ^εκ7Lq‡–gž?$/ΪΨςL³Δ!˜:=Ν²Ή«U:,'Nœ­o‘mQωnŽ„Žνp9V,Œ”\>§„§"fμ]ρ½@Xp1±―@&Ÿ•ωθΈ$A¦θ0Δω5P}­b!°kχnΩ>ΌEŸ–ι¨OΊΚΠjΛU“* 8N ΔΣ±ι˜lή0 ΫwμXξ=§Τ5†L‚{’žiLβΆ i ;!sΥ3Mm›voΪN ™&B2:¨m‹,*½cΡ-λν'†ΔβC*?~\Ϊ;ڍ7Ί‹γD²΅΅Uζ‰ι‰'ž0;ρ(€Θ€&ζӟώ΄όψΗ?6^jHΔα‚C1†+’@φ;{lbx§ cϋΒωάk—ϋN7Jό°H–ώiΘdu"ΞwΪ&‘HΥΎ₯—¬w.‡ ³³Œ«› σDΓΎ¦Ϊ#@T†LCF΄†ΰͺ}«4f ό ¦šρ8½  bB‘JDΕο@y-šoۘ Vπ©-™†E(™¦‚@‘΅!Μp±h‘ž ŠQ/-tQμt‚LcŒA ?–R#PΩΠ]]F‘PΨ„¦ΑŽs_K¦!ωNSν°σ~š(ΟΉ ±›Dνj%―eΌΨΌεI}ϊ6wΏ6 ·z/‘2Ί ϋ »ϋόέ I*Α} ¨ΚrΨΚ‚ž“Ρσ“²½³EZ]MCS₯`λτbσΛε}~yϊΠQ9πάse)S3YΚj.¦@L‹‹ΎfχGή[p†U’iά>—„fƒΡ]Μy‘ƒk.&φƒ<“ˆGd>>)ρXωϋαAΨ½φFk\Œ>’ιΝθΊ\+£sr|Δέ¬²ςΌ„rψ\8-ϋΟLΙΝ7\oBOΦδa΄PE  p}¬Ω¬I[ς:2…gΌMγœ}gΚP”fQ΄{Σam($U«S€@¬δ-Ω±£’…ΤwήE“iΈπΰAΉvο΅222bžΞξΚ`Μ1–:ž+ρΆν’™ΑI=ޝ&Gvy#ΌeΦHmλ¦₯―_hσωZM(>‰:ΝNΛ™ž ΧzΥΧΪnμiλΪ>™΅YOΘΓ?"W vI_+ΌΗ.4ΣΈ +6œΥκ‘Ηά$ΉΨbο­ν¬―-¦\<™›™“–Ά˜stšEϊ·}ϊf΄‘Ϋο’©ρi ƒΜͺIXΛΨΕ\Ϊcz±Ξ†₯)>Ο ωνΧCMŠ@‘PRαψ`,)'@¦‰₯°iεD"M,9/'f’O¦eΟ{LtζiϋιΚΉθŠ@ύ `Ž˜Έή$Τ€a₯³23η2ύͺT*¨^Ÿί„‘qŸξζ“πLγƒ·9%ΣTώ ΘĈšΆΰ$Βz6XρF½ bP·1p.’ «_ςΈŽ«ƒ]3^Œ1ˆή±³Κl:@Δ<»­4Υz£α*ΆΗΣΙΙIS!;wͺmν·tC¦πn†Χ †έ‰Η:gmάξP΅'§π‚L3ŸN@VcGΈ5d-WΚt„ξkiρΑώΆΰ-REϋrˆ­κ½<υάΉ¬·UΊΌΝpΠΐ›Γ) 5½>ˆ6ςΠ/‘ΡΡΡU•₯7―Œ ¨7}^;K<£ς›Π>ΤΏ:Ϋ2Mp"(s ³jRΦ:μΧ΄Υ0@$Œ>˜ΕgPzzzζ Άw$Υ)Χz+W·ώƒCCrγU{δΰxPfΆ‰cάb‘š[#tAΜ)EB‰yΩnN½|'ΒnΛ½DΏ+kkΛ£νδBΒwκJ¦ΉI•ΏΩ G΄3ΒΗ7ςH¬‘­UΗ»*7ˆ·$E“i6ΩΉs§Lbž;7uttΒΜ³Ο>{α^&oŠ‰ιμΩ³ζ³Ψ/θν&χ;λ™X dωΙ½žί91;pΰ€Ήχπ˜¦ v$hΈΘŽw€LΟ4όm•μgiΉλ]«A€οhP:ΰ·«Υ-SΣ3hέ©³LWs―eΪrA< Ο4N/ΑRiu]·`υή’ ršΩ)Ηδ!γΩ ™ˆˆΧλΣ]°%£ZΎ9‘ΰxJs(Κ’ξt\-ΖΕδDο©d*γυƒϊ-ζάτlΠΟ|)ΘqMŠ@΅`ίk¦•ΣΨ‚ eΠ„ίΕΞ±ͺ]οF(Dšθα-iυ8εόωσ˜£κ|³Φm‹#Μ*ဧώE‹]ν΅n•υ_> χ-žΘi²žŒΰ^βα©dΠφ•€ΨbŒΙΌΠš—ΈI‰ΗLŽ—΄<ψ“αι­YvvΉ±)γu†­`ZΧωέM²³Ϋ#³‘ˆ<όσŸΉ‘cm‘€qy0”ρsγΪΟ4 Κd½Ψ ω]0ήϊϋZΝxK―ΆO> τΨZAΐΞχž'πN’ΖύΰŒtvt’Ά=ΏVžGλY;¬,δ&χ+φ\&‡ΟΝΚD$c»XJ€ζΦφ<=1’έX ά°aƒ9eσΜ½NΏ+kk+ΙΡΓsΎgXΏ΅δz=ηMγΈΞ?11‘vπ:jn•ύ?[l{ ΙΊ{χnΉχή{婧ž+u«ιδΜ‡Œι‘‘ωιO*‡’]»veŒ»Ή£š^i˜ΜŽ0σ­°μ$χΘ‘#ς―ϊ―fRΔΖ³Κ#λ100 σ?#'Ož\1Sή{ζΜyβ‰'Μ΅ύύύXLτ­xŸ^p1Ά]؞ά9β‚‘Gš1ΩΕΞ©0~“Τ€©6XαΖA'…vΰNͺ`“[ŽΑ}qHέήΦ¦QX*4ΘCz š›šC\ο βBΫ€αbS]ΧΥYΉd’iHœ™‚LC™ …5Ο4TZΥ3Mν{ε¨_ΝΝΞ Ι5~ΏΏφkΠΜck2Γτ9œbύγ“ί“ IZ½ΤκG “>vΐ”‰,X€π‘‹­Ul€Τy-+Z½‚Ι4ΦEο2W]u•!Σ|ε+_‘M›6Ι΅Χ^k.τL³yσfSαx@n½υV±‘Ÿθ­†a˜˜ΊΊΊΜg‘°-›e΅BqόΘG>bΛ<{μ1aΘ§bΣm·έfΘ4…Φ§ΨόΧϋυ „¬ aηΓ<59ᆡΩi~“Δ‘©ΆL“D;τ·"4Θ43ƒk₯ύ½vmO όΦ©iqyβjuωIƒ’4)ΥBΐμ”mρ‚ΠΕπά1?R@Đi8ΩΦT¨χPF3L“$UΊ·δ$BSυ°γ%'tqx{γ’V†Δ@W°‚E2’$y&E Φh/¬] XYAξΜ™1ινh“‘Φfu9δτΙ4Ic²ΧΥ¦[2I©ΦΖ;d*––Π B(‚$i’R·cTψΙz‘šLH>K~\‘L“δυΌΟΪΓVΈEO€•ΓτδϋΏϋ?Η<όΪώ Έ›ΝβanV \Pμ†GΩλ[ε “Έ_ΆlΩ’2=¨2|·};“gΞΟJKž―αEl›dINZ§Ψ€ΤΪε•ΗOΒ;sTH¦Ρ€¬YtnpβΊ uHφi.2jRŠF€Dτ©;wH_w·œ˜†δΦohwZœYF€šssρ΄œœMH7ΦΉ™^“"°φΘxσ]0θ­ύΗYO`urFq8}ϊ΄ΩΨΛc＝Ιz¦Yͺ°(j₯Œ™Šά+^ρ ωπ‡?,ϋφν“wΎσς[Ώυ[ςŽwΎCΊ:» ±†Έ|λ[ί2‹q/ΑŽ ²Θώγ?ώ#Χsdd€hθX6iΏσ;Ώ#'Nœ0a€ΈΠΗγ§N2eΠcΛcβq;3ώαq.V“έMςΟ«_ύjΉρΖΝΩ|Χηή«ίσ#„’ ~N<Σ8ύXXr™ίκ™&?^Υ:ΚΎNoP)X~|0Ϋtz6(ΣΨ)Λ€ύ½Z-q‘‹9Yg<³Σͺ₯͍w&hΌN0Έ”όΊ“~SΚƒΗCΖΓTŠaž‡Si’i8ήZb—ν·ε)QsY ϋώΗb1’λ…{xB™œœ22b₯ϋυ|ε ^Γvq@lB›ps9]Ο7γ;Ι4ͺχT{Ν9?iθz™ΕYZά3.‹ω›rDSν žw σΥΎ@+_›₯Νλ–SηΗe ‹ƒκ ΅vνΒ’Γ w‹…z0HΈψވ υΔOυκD4U λ²ά¬ϊZd9υpΥΕ ¬ψλyδΉοώΛu[zeΈφΕ%²ΰ°JA»Ίœrν–>ω―o}ΫΨ>―½φKά‘‡KA€}‹)£GG₯ ž‚\~˜¬v3ϋ\yœz 6//ΒB~ώ΄ρθΩΫΣ{ιΕzDX#X„‹‰τLΛ9(m…΄ΣhRJE λvW_qΉ<όπΟεl8 0I_:ξaΌγXΘHΒSΡ΄žΚ΅{_ έέ=₯«χ)uƒ@kdΖVΒžMψύƒφ<΅£dA©ϊκŒ$Σ܌h Iœ?wή¬1ΫsU―”x1Ή―ΝΕgζWQd‹ΚuΧ]'Ÿϋάηδ]οz—9΄qγFyΫΫή&)rΛ-·˜οwί}·πο£ύ¨Ήfοή½rπΰAωΰ?(ΫΆm³Yόi #`ΒώΓ?όƒQ$­Aβ3ŸωŒ)η7σ7εχ~οχΜαε„σ’BJETΓXάK^Ha733m<Σ`Άk<ΣΜΝΝΒe6b‚kͺ:μϋμγ4δŸ™&ΰuI; ΖαD³ljo•ΡcΗδΊλ―ΧΎ_υ–‘n–ρ61p[›Ϋ°Λ Sd,ΔΆZdvΑλε’&E Zp₯A& 7ςfA‹° σδrΉ³dšjΥEΛΉξž?‹‰C―ΏEΪ[œςδα“AΘ'&+η/ΎCUzc»8ΰe’‰NΔH¦ΑnΩf¬0tΘrΊg₯λ¦ω7& BCCfž€F zι‘`P=&—wC~{š€³Υ-OŸ–“Ψ2„y³¦Ϊ @FΈ[¨γB;δ· σ€ ΪK“"Pi2rz)ͺΖR₯Ϋ bjΉ\ ‘RŽ[=š›ώώλ›ί?BΎΏpΐ#~x₯I@·Λ‡6坣tΡ;Ν@‹<>(ύ­–;wb‘[Γ •‹ο±νBΐ‘ηI[―_ά•ΘUέ|mrΡύΈ†χ–ζKηΝΞfυ’pBϊc"`Χ.¨«T£dš5ڐ5Ά΅EΣήόD—ψά7Ώ#ηΓ½²Ρ»yκΖc$‘’LσάΨ€όφž=Ω‰E?€Œ>cΎκ?ŠΐšC cχ†~­9::ϋuφ>΅ιe©ϊ—©ι)γ,Γϋ7Τq6ŠρΚ666fτνͺWH ΌΕο¨ix—.ΌM—ά°E¦±ƒ/•7’g†††δk_ϋšqυfΓ9‘)φώχΏίvyΞ¦Η\~χߐ]8xΫ‰’=_θ'›Ηΰφ†b–{X?«„Γ<ΡΫ‰žiš-˜Δb™4dzΡTΖΗΗaD8#›:|‚u? €P3πΙ‘Γ‡ D²RίΓΪ<ΡΪ.ΥbMγ wώξVC¦aˆoΐƒwhZCΈ¬ν&^“΅'γ›šωTBšθfcž‘ž Σ₯¬Ιg―χJ“΄qŠaBόιB¨ΎŸΓσΙ€z«i³‘L‘Ιεvd\ΞcW I‘”γl―Ε‹šVV oH€Ιi°]Ζ$S)έQUλΖ§Αg^;»6H+dD_«KΞAΟ£ΫbM΅C€2š› *ΔΩ BM³Έa¬c˜&]¨]Ϋ4BΙ$?RNγŸ"7³Γ° ŠΈK/];'gΘ†wzlsςϊΛ7Θp χ™υ”₯oΖ^²«Ϋ-ΧΑkΚ~ς\MJ―~ΝkT~,‹Zq'I8zψ¨ψ:½ž±Y’eΈ·ΣΟΜ)γ͜χ¨\/9½¦ž ­ΖλυMτ°―dšzn­ϊ›6oή,›Ί:δΜ\B:[άBGΫ¬vΒοtΪƒtt&&›»;eΣζΜΊ›Ν£ΎŸTk§,m%τξΫΤ„ ‘νυψΞί)θκμγšjƒΐιS§α¬aFFΊ}r>””ψ|Ζ[ΠΙ“'M…T§«M»ΨRωn,HC,Š^-³—Κάk0iΌι¦›Μp7…+ΈB>υ©OβΜa,άσξΨΰqao₯ΠΟ\ΑΖΌoΏύvyΥ«^eβˆΪ†])/ϋΌ.χϋJχιωK`hƒΙ‰σβnߍ ‘A@ͺ‰Νƌ$αJSm +ω£Ηε΅»ϊΰΊ±IΊ`4ξυΉδΠ‘£ΖΜ] šͺί—±±³βmGθμ~3;¨ΰ™f|€’iͺί _b3fΙN§ ό žiH°±!ž @Ζ inl•ΑΦfiχϋLXK:΄}jΣ04Ξ‡Γ㙆²›ό3†λc˜§Ž“Τ I¨&”)!«$€(Ώ~Uϋb5[αBYvNΙω― «z|ΝβΔ8ΫνuJ‹ΣaχHͺζBΘjηΓJΥo…"ΐχ" ob")Ώ-.3gΥ±΅PυΊRΰϋΞ>–1άžC ’ρdΊφ G­°+}τQω―ώ§\Ύ±GΫΰ6φzž±‹‰ωrα96Eζπ7 yδΨl‡όΫΏˆEΖΝ²»φUηC­ψc'OΔF£ΪΪ+Žμ cχ_aX―Α₯-n¨DMrόΔq‰'ββ†·UMŠΐZF€› sΙ4$ΧhRJAΐΞSzz{eοε»dτΜQΩάΡ)nΜUHœ1|_dlΔ)~D“σrx<$Ϋ‡7ΛππH)Eκ=Š@έ!@›^†ΰNWΣ6[ ν(Τε4ΥnHššž–[Ά΄ΛL$ ”©Η „ΟfT.S›v±cG*ΙΉlF!禄FME“ihψя~„Ε7§™0φbnooΏ? ιK_*·ήz«Ϊ.ό”c‚iΞΪέݍ؍έφgΆa³τKE…‚Ψ‘‘Έ2ojvb‘Ι+aόž›PSυ°οwΏžŸ“ώΦA(Η΄?4I›G~qjBNΓΝ|υ+§%£Ξb@ΆΐΫ„Ζ!†i `Ης‰σY2M9δ€B­‚½P!McBYώ.»΅›”Iϋb!(VζšsηΞb,I % }>‡΄{=Xˆ=‰IDνγΥΆ© μΛζΚΙΓ·8αΉ©Ω 2 ŒN$E:@¦ ‡"f‘lΩ τ€"Pf’θ“ «Ϊ έmPOΒΠ dš2]DvτRuψΘx€GHfͺέy±«Ώ]|^¦ΰaL=©h.΅ΊL(2!H£c3tlκ‹Ξ-ύΖJΔ”LYΒϋκ™°²ΰvΈ~υ«wƒXη[‡Όπζ4!œh^i …אP³σχ[7ϊδžύη䞯]Λ²q°ΐGY——Y;Φ±Ρc&$Ÿ―Χkl»l·e“m8\ζB[v ΆΙ‰c'djrΚΨΌl»/›‡žTκΪjHΒζB"?•LS§ ΅ͺeelww—l–ο?»aœ°žGΊ¬΄εηd$%ηζbrΫήΝΩ57›‡½^?΅†m'œχ‘°žIπΆݜΆρ$"o¬¨s¬΅σϊZΟιΖƒ H½>§<Μ¬πΚ Ÿ˜˜06{}?ΦΊ¬^86ο Ξxϋ]—OΉςCL¦±•αIώζoώF~ψΓΚ'?ωIωƒ?ψƒKvΧqp΅Β‡€&ϋ»οθθ¨y©lY+?vζ Φ…F zO!±ΐ>c‘χλuh€4Ψ9½8σ.ΒEef:C¦©D›+φω°}˜Bν™gŸ•}pΫHν4/nΉοΉ ;~\φ"NͺΆM~+y”ή Ζύ‹oΠ-N/Ύ`τ’@’β$Ό 1i»T²4ο\ΨΧ8v’LŠε™]³τn ©6Ψχ$ˆ3Ν0ιw€tΧ YΡαuΙ10ς£‘¨΄΄pΌΥTm(§ηfA¦AΨ-’hζ±[‹»_πp›Ž™POΥ“–ΧΨp‘5Ι0}Xp₯BΡμpc»’iͺί/¬~„ι#GGesW«t€¬A‡UOv~ϊδώη Γ°*™¦ϊνΓƒsAΘπ9γ’αωHjηί4ζ¬φ±νX›j©λz–‘‘Ψ zNθθΝ H’DM—τšV‡€}·i'Ήηž{ήι€Όώ²~ΩέΝ0B΄–žΈΘ"WχΉεΔ\—|'?•‘‘yλΫήfζUΆ¬ΒsΤ+-fτήvμΨ1γAΜΧβ#l%σάόΊ\±Apžy0„_η¦v9όπac#V;―φ­΅Ž€%Σp Q2ΝZoΝϊ¨ΏσΕ-π¨F ΰ™`2C&Ν‘±φλΩ6%R²}Ϋ63ΆΥGν΅Šΐκΰ:0½Σ‚;³2„]Ν4/ΨQ¨Thͺ6\;πό!ΩېM’ΐά)€ΝIό­ςσ#£rφμY΅‘T»QΚ³kscΞυ#ώV2MA·Υ42υυ!| ˜ΡLXσ£€ίφΊr|~γߐ;οΌSnΈα†e2h$γBέόΞΠTO=υ”ά{ο½ςΖ7Ύ±Ui¨©λΊΪ΄M#–J9Nώ<=Σ€Έa ™ψΝcšj‡u•γ Ξ8@‚μς:Δ‡ID/ΌY†ώΕ…Ϊ„ΜΤT} ™ffV|nx¦Α;‚΅-ΌBπnΰ”H<ΡΠŠκ·†–Hh JΖ£PΏ±y$H~Ζβ13ίQ„jƒΐργΗδ©CGενΧٝ‚Nέ{CˆΣh―˜ΠΗvU›Z6f©Ax›… χφyπ€LƒρΥ‰POΣπDωI¨σπό‡lΆΔΗL9vΉ*_©4δgΌρ>ύ™ ©cιkΉϋΜeϊΟ",QƒοωwΎσΉχΏΏ-/ά+7#T“^ͺΰh°(TΩΌΗ]πΕ›Zd"%_Ήϋ«f“ήΛ^ώςKl£‹ͺ£?—A€;:,έ›;Œ—³Ζ΅Μυζ”}% —“ ιοχΚδΨ”±­\~ωεΪ+α§ηλ«'rαŠήhhΏ₯Pm…uΩ\k¦Rv<ά²e BU»@¦ΑϊΨ’1fκ$διx8%1|ΩΊΥ<Ÿ½wΝ<¬VTΘƒ@ήgНά@Jcuk0wIx':»¦κ#@2Ν“Ο”—mν’Θ4a(Ϊ›}M²©έ#_;'g}γκ«―~Ε΄DƒηP\緞|ΉξΜduσ£Aώ)z΅Œ Q‰c²αHLα Zθ_Ή±΅€‡~Xž|ςΙ%ž~ϊiyξΉηŒςΩbά°².v'ZΉλΥ(ωMMMr=Iš@’1F†Ÿΐ·™™ŒΛμFΑ‘žžσ8<Ϝ:{†{‰ιmTΌ£mΨ»£ΏCžΑ{0/Sšͺ‡€`Ο;o3x rSΔ ’“―£ENΫ_Ϋ£Ρ{ΦΪ}~ΪjH α¦`~’\ΓdεΪΪ}2­y-ΰζψξξn™‹υΧ…D‘낝I€e"—-ƒύΣΣsαύ¦¬q’I:X€_&’iL’ž; ~Σ¦§kΔΥm`;–E˜μψ ½Ϊ’;DρΩΥ$;ΊεiΨPΈ1FuΊκΆ-dF¦±dr1U·Ξha™>·oί.όγ:’S^Ž$¦Τ2έzΫ­ςΟόΟ¦K5$¬©C‡Ι·ΏύmcΔΌωζ›εψεZ„»ΡTd虆$šf'Θ4˜¨ŠΓ#-­έ&dM K~ΏίΌ`V@–V’ήUc’Ζ’±8œn3ψΠΠΓΏ6·tϊδΫwPΖΎΙΠƍ…d«Χ”†ΚcNβ¬[%ΐ—Ÿ ™LθVό§I¨$VVπΣx¦1μϋΟ4ζX1VΛJΦ΄1σ“Cπ vUŸίL$Θ|ήc}(Ιζ„Ω‘μΘNώ£Z<5ušΘ „u‚Ϊ66'ήπnΟ4qŒΉ‘p¨Υ2γ¦8…MM™Z†‰EηΜn«†₯fΞEΐŸώμaΩ³‘Σ„η³k€EΫ‘nΏμφ9=zTv_v™ΞͺάRαPΨx+ΨΪ>„yk3δyϊ·[B'p|r ^wϋ«\#-Q -*‚8σŽV,HΑ`OΩ`K¨ΫώρΛέΨ•ςšΧΌFθގ©œuZ©.kύΌ}a’Ρ(Θ4“βrϋ`Ηχe!μ "™fόό9ƒ7•4M•GΐΆ ™‚$»u#$H?Β™~hΧύiŎΩ}ϋφΛή½ΧAXωΪi D€ƒΙ4\„Νx¦αA0½Nρ΄ΉδόΩqCϊλμΠ.Ϊc*‹€•”tii&Ω"yŒΦHM΅B` ‘œφ?-ΏΊσrρ•Οω\;&#=mrŒ}†€θξξ1z™ΆUυZ‰:$ε8]Θ3™y7^㙆dœ˜΄M ϊO°},Š°ͺNWrœ!ϊΈπι ύTSυ঍ΡΣgδ΅»€^Εβ‰Γ)eC‰\ήη“όδ ρ^C2mΗκΧ΄1Kœƒχ™ΠTXά~Ψ"ΰ5¨ ›q= ӜŸ›6;­}κj @Γ=]Η;a'1z·-t){½Q0ΊΟ-‘Π€Κt‹W‘ŸvΎsτθQωΏχ_d €Ή_ΫΥ%{zάΖ³r‘Ω-y9eω 6xd.Ρ)sπ|ώsŸ“?ό£?2ΆS[‡%oΦΩω ½»ν{jŸtuˆ―ΧS2|u\>§ψϋΌrό‘1³ aΓ† %ε₯7)υˆΓύiRJEΐΞ=Z[[ΝFωύθGfqΪzγ`ο’WδP’IΞLΜΚK6o–@sM${―ω‘(‚@IDATk ΣVbΒd›g ‹ΒΟ4± tυπ[­¦΅:27“|ώ Ά{₯λ˜)ΐΉ"AΒ:CͺΑ«μώηΗεΘα#²qγ&΄W΅j¨εXΈ‰ΗH¦αΣIF©Y ωΪkε³`2 αΐIπψΗ8eτLσώχΏ_ώε_ώEnΉε6ŒUΊ΄/Δb )΄ής–·˜xΒK]³ψž•~3Ÿ•λNI?7ήx£|ΰψΗςήχΎW.ƒ1“εͺΟJuYOηιΦirrB<­]и؝Π0ϊΈ½νrξܘp·¦κ"pζΜiy Fˆ«ڍ!ίΌhμXC7σƒ―<Vη›ήτ&£kί―lY|ι’Žažό]>qz3‹±,™}og‹L0η7oΪ\Ω iξ @ξDΓ#ΗΡΜ_FVX‰ΡπPU +'8n>ƒέ³#=遆;ƒΘΘ`±^ ž|φ€LNL2MΥ*§ ™z¦1ο ηΩΰ0ΈZθ™„mυL£=₯Šp>ΕΒ“»%C¦αn*†a(%ΣT±!PΗT.–3ά0eφΦNWΦ3$GS³Έ΅ocΌΣ ΌΘcΏό₯Όδ₯/•.V‘©Œ #•”‰‰ ρχ·ΒΫ7RΪΚγwK8V2MΪ‘‘‹ I€6š £3 ΔΛͺΫ |αϊt²˜˜rυχ…τc ¬^MΓο—Ώτ%ΩwπˆΌ|{·Όpΐ#.Μ½i _ώ%ςΝw˜3(Ί€ΏeȍέόέςΓ‡~.½½½ςφίύ]α‚₯­KΎ{υΨh˜?„Ε―W|έƒΫ…³…£lχoπAGŠΚα#‡εϊλ―Χw§pψτJE@XηpLβΊ=¨qώ²X·ΰΨΘ I nύύύζZΗΦy§h€Η³ύϊ¨\sΝ5222bΘ5_ϊΧevvΆ ζ|Ε+^QV2ˆ…nYSW^y₯ ρD/;|’iŠΙ§²αšΩΩ9™?Ο|H’Jc ‚Βs£OβŊȞ‡9¨ ©Ω3ΪώΛ]±§'¦δ»»€Υρh`‘§1Ÿ±Α‡Aζx ;΄x­†9«^“QFŽ“Ά>ςrΛςhπiiχ˜Θx©^•΄$EΐŒ}γ³Ι ŒμύRMf  ώςρ'd[O»tyαmΓΦ‘‡šAμ¦ώΑ1=vLvμάYΝj5tYv|%!rdz¦!Y›VxΞ S"WqCχ’κ=Ό“±F°Pδr/„Y³‹!A" ‚sšͺƒ€m“'OΩ½£/ ­Σ¬Ιw)Θ ΚτΛϊςπ/Ÿγγ$Σhͺ,Ά}6„„φΆžVi6:8ζ¦°ΞΡ3d$•™ΩS+ο+[+Ν½Ρ ±1mμ$ Ә”£w/†QΗaΨχόμ}|cΕ΅ώqQ±ε^Χλ-φφ]v© $jB /•4!οGBB’—ςςOΒ^Hy)ΏχΘƒ„R $”Ό$FhΨe)Ϋ{_—u—«dϋ}GY–%Y²%―eΝ€W½sηΞύfξ™3gΎ9'―L¨Ψ?ζ½oƒ'ǟέwΘ-/Θ›WΙYσŽϊt2‰4¬ۊ2Ύζσζ»Δ뫐ύ­n4|6ƒ·©SόO‘y9wνά%M Ν2ο΄ͺιqβΧdέέ^9΄ο°xΚσΥ@ΔγΪ>Δy₯.ιο…‘Ÿa ¬1?>lmd! &ϋΡΒΒ~Žž°ίRΐΎύϋδHw΅E.)!DςžœDΤ ~l₯Η-ᐓ<+'RέcΛ§g.†)Ÿοv™s…δΏoPzΊ­7Ύ±ˆΩ_©D€žiz{{$Χ™‡Ϋ@‘@¨'z1πϋϊu—a*οmΛEΐΘα-Ψ)υκΞ=²ΊΊ@ŠΰωdD<3RίγN«₯₯Nρ‘νΦ­_guπ :©?Π/MM’C\.ΪGΧ (Ώα)2žίΈΣΚ’ΠRί™v#h7’gl:RΕ! tŽˆΝJW†Άz_ ! ηΧĝ›~θ!yψχΙΙ +εΝ έRB2t@u‹―°rQOgΩ՞\Ή€Ξ-ΛηVΚ=χύ\{μ± Ξnl ›YΩ^Δ†^dΊΪ»€hϊ Υ—`ΰ:Ϊ‘’LΆnή΄―L’${‰E`F `δڌ¨Œ­Δ¬A€λzUUUCιχLœg’LΓσ6Yfκ™f cΣ–§aH¦ΑŸσEφ}›¦£X[·~ƒ¬™W.εyΠGnM½«M₯Θ•%ΗΝ)’—^έ$‡šΎΚΩ; q¦:z>ΌGz*ς€₯ΉUZ©&SܞiŒ!€$”ο}ο{*\Μ±D#σ•‰Δ–d€ΙNJΉ(ΒDBˆ–“k‰ΞNio="sVΐΕΌ’iΈΖ|‡»¨Š€‘±Q žl_™\­2χͺƒΘF –•ζГ£†b>Α„Q k-Ψ1‹p!•ΕςΪk―Λ%—΄hΈ;A ’”τ/¦·ΑPίzΈU–ΧΘ4΄Έ!±œh/w‘ αњΤψηv»ƒΖ@Νd±€ΐ»VΈΡ`Γ۟©C€r‚m±iΣ&]h]Xβ”’uΑoδΆ”λ5EnΩ²}»†ιvΧΚξΤ΅IxΙ γDΌIž`Δ2T!ΐιεΐιqθΞWNΞVΤ&‹@ͺiz%Χα(wΌ!CBΡ£Η›R€‘Ώά™σ4Β/™½αω θ=T7ΓκΒ’\©-ρΘKλΦCΏ4©ήZSΔι{v!’Lγ.ΔΨͺdšaΙΖΈ› jQU<­Ά¨ ΟDWΕιΫͺιSsz¦‘g»¬gb•†LwΈςa―κR²W²μg‰U"}r™Μ°>ϊˆόςΑίΘκUr!ˆ4ε ΞωΉω8T0'ωΡX4e\xΌΊΈ^€Χ_.wίs―†zΊπΒ Υώiκ˜δ[§mq陦xN‘ΈA:e;™q3‘‡γu 9ςs””³α‘Ν°I6H]]]BΕΨ́™†ί›,ΙD€d™ςςr,NVϋ…ιc΄KΡ¦QZZͺη“yO[–EΰX#ΐΎMΟ4Ω9 XAi Ά>Ÿr ‰λΘeΗΊΞ³ωώϋχο—-»χΚ%+η(ιM0jCAσpƒ½΄/(rˆoxP6n܈θ2«¬Νuš;EKΛQιjσJA•[1‘jήT½Ms5fΔνβ&ӘΪr?wξ\σsF|š…κD*Σ’ΗŸώτ'½„ξ‘ΈpmSόs2Mΐ1ΟSΑm}ΛΔΘΒΆ‚`§¬ξ΄›”¨Δxœ6₯σlή΄Yš`ΠΏtΙBqa° Ÿo‘ΩΠn υ”-+ͺ εωΧ6Ι‘C‡•L“ΊΪΩ’Ω>άνzπΐΔ/GAnΠ@4’ΊI6 ϋΕΥ…Š†¬L²ύfΊ+'Π#ΥNc5Σ…?οjHήΊm»TΈeNA©©βGƒ¨œΰ$€e!ΪΎOΆ",Ι46Ml#υ<ƒΖz¦α­Ρ.ΩpEšWμVr6pH¦1m:=΅³wΙD”LƒΪ<)”•εΐ_ŽzΝDLŽΥ3οΨ±CΦΏΎIήΈ ²†8ΘΚνΠΔί„ϊΓζ—δΙΡζ£ϊήd’«βΤ£oοπlGΟ4 ’i`ΨΟuδi8uΞ%­~½/½‹ŸωσŸε{ο“:/©ΟΉ%7ΰ‘&\0G/ngŒ¬_Tꐷ.Κ—ίξ’;ξΈC <yΣYgaŒž†JLΊφΗξBΊνίΆi›”/,…)›ΒhWœlβLέΌά)ΨdΆuΛVY{ΚZ±6ΙΙj―;ΦXΉq¬[`vέίτ'†ΏcΘY’ Μ1σ€\ο±ke ϋ9›πωήHa7 쎣½ΊμΤ!vΘ¦Τ#@™CΌ_‡ΧugN6B<9$&­‘ψ c*0<œ%5p °rN™<σμsrΑ[.JxΝ2Ί˜ΜφGR`;qŒhΔϊ>7"‘π>θƒ’]›Ξ8?=–ˆ’ϊ°q6)Χ0μ¬Sω‹³nqgγNξd`¨¦Xά5Θ†ζԝwή)O>ω€ήcŊΊ[ΔΎ„ρANœ˜θ:— ζό"ir@F9Ξ7*›d~ššƒς›λτbϋO0Ές؏y2# Ή`“„܍­G7σ ‹ΠοχiΈžWžC.±_§€€iƁܷoΏ”{Δ‰…–1mƒqˆ.ζyξЁCΊ ;…[ΪK-"`ϊe R*Π°0ξRyΣ?',Νf˜*fΒΠp䈼²q³¬±NΓ„PCI΄)CLΘ"(―ޞ^َ•‘ &ΏύLζ]!Φ\ sΈΚ2d!/ φ.x;` YΚz›,Ӂ' σ”“K2MŽή2 F ,kϊ@`²ΊΒ²Œqα₯—^TyΌ¬Μ%nƒ’­rluαόb,²VyδωηŸΧ9,Λ1²&e•Νπ‚ΫZΫ`PO>dl ¬yπψv΄ dš‘χ&Γ‘²ŸΤ3 ή¬lΚλ"ζ½pA Λπ>Φoΐ”ω6EGΐŒwΟ=χœάωγKEI‘\°06GD‚cτ’’s†ς~9ΘoY˜/C—οΈσŽ έΕΚϋρ>rXš7‹§ ‘ψΰEH1ŠχU /ŽΧόJ·aN΅ρυA’±Ε>,ϋΫ"`ΘTΈYžži"%κŒ*aIΝ)―=fH'ΰ­thȝ<`?Q…ΐίΤϊ1_΄)΅]¬!ž6mΩ"sŠς€똣iTδ·!΄ =L.,qΙφ={eοΎ½£Yν·”!`Ϊ‰ώΉVY6ΏD7%εbsXy}©μΫ»OC³&oΚ*3ƒ ν©qW‹ Υ©€ΔΑ8™ρ£>*ΖξΊ ‹edΰ9†tzωε—Υ‘Λε’3ΟM Mβtΐu1<Σ„-Άds1–kπ=hΩdH%fό₯,ΠοjΨgœΨ‘¦°ρ;•uΙτ²M[΄Γ»ήsΟΏ +ΰΊrZΓ5ͺ($ICy­+Λ—ΏlΫ)‡@’$™Ζ¦Τ#Πίί§žg\Έƒ…‘_E8?π1Μ“3Ο!^,vω°ΣΕ&‹ΐt 0θ‡gšžnπgJau0žiΨ?s€―7ΰ™f:κ‘©χκίΥ΄Ζ…¬­Cœομ¨ϊ7q’\§ΜΘ‡žNΨ; ›<Ž?αυ&ε06SαOψΉ[ΫZΥ@κΜkώΘA[`γν袞!FΥ^Τσ‚ži σDΓ‘²·Sϋo<χΛ΄@&[zώήΆ λ˜ΠΫ—”Μ7t6Θη4΅¬&cοLKIMAŽΤ{d3ΌΩ΄Ά΄HyEE`=Γη+ΙΏΘ9΄χ¬\²X=ΣdӞUξ–ύ/V/Λ™Ά|¬5)Ψf’ΓΞώΣ{*Oόν‰ΉcŸ:ηœs’J¦a£]rΙ%BrŒ¬ΒH Ν—•^qςΈγŽΦƒ!žlš$tv΄‹³€cγSc heΉπXΣΥqDŽm‘ΊΊ:=cI.ζ½μΑnω—Φ­“₯2ΏΘΈͺ‹~/ΊŸwaρoy…[~ϋΪ>ΐN>ωδˆJtτRμ™Dh>Ϊ,=Pͺ—bBbbΓ ™ΖƒΈή‡ΦZΖ₯fX6ϋΣ"D†Ζy.β₯Κ›¦††#ς&«+σ₯2?'’KzΆŠ²{>ΌΰCΆ`—DH;ΡK}[Ρέk7«.x aX'}EψΞ QH¦qδ劷ɫ“ΒΤΧΖήΑ"'π²Α0OٞjΘoθ~Α0O}Φ°ΒNbτo†X}ς©§dQe‰,*ΑFΘ?ΪA‡Σ(χg311τΘ‚²yiύzΉθβ‹…σYSn ‡ύ7™t ­HXΟ‘]ˆ“!m¨, )’ ‘$άΨdHT`β‚T/ŽΩΩDO3d$Aμ»ee;•Lγ΅až"ed& κχasέkΫwΙ[WTΚ)sΰEΨ_σˆW§ώ EŒ±»œQ BΝ@₯<υΟuRύΐύrν΅*V# rψΰa)¬φ覣€’=ΕφαλύάƒPO½έ}rΰΰ‹υ!΅—[,³κΔΨJ'τ(`lIόΤqΗ©»ΠkMΩ€€ΡΩ―•L“λτϋ»wV67γ7=ΧΨ”:L[Πkμϊ—7Θx]―Θ'U†Zsτω=M»²eeu‘όύ•Χ„ށI¦±)u˜±αΐ  εˆ Ήž‰‘Bά₯N':¨ “7u΅™9%ΗM¦1U&8ωx!A"y%ΰ²ΦœE·‡ΰ!«•D “–-[¦!dΦ¬Y£‡ςσΧ' θ}θCς| ¨˜ϋ².‘Ι܏ŸyΥ ½Ξ~Œ@ Œ$WΉx‘’B³˜²"ΧΟ4ݝ`©΅E.ΐM{χ³c—,*Ν“J1=bŽv4V6`¬•5ε²aΓ«rΡEK…euFClΚΗ[@*λνιΥW4¬FRH¦qΒλACCƒ%ΣLq[@<pœ€+Λ,Θμ€Τ ZΓο±ς‰O₯gY£›lίΆ]:ι]li©ΈΐΚSaτ©γμ°ΜC¨Έ2:θ‘Œϊ–u[žϊNΑI7Ι4Ž*μl—ΐ+Γχ†(΄—Ý ±W=0¦Ύ6φΘi.ΠΒ3‡‘#*Ž}1„Lc1J=»wο’ 7ΛI5…ΊS*šWΘК¨a^κΞ’γqέƒ/o†‡ΘνJ¦ ΝgΏ'κ9ͺ~;A|€Oχp`ξ”λΚ•φΆv=`ΖεδΥΐ–”‰°ρ}§gg’ir0›^xf’„²r•LΣΧΗΠΩΦΈŠ˜1Ζί‡~Xώψδ3rώŠZ9sžKw·Ζ#CΛKΥwΆ4λRΟ’η-pI'5>ό(6ΦΚ•W^i73ŸƒJγ‘&YΆ²Θ"ρm#γ]…uKΟp۝6΄y’@Ϊό‹ΐ¬F€€Nα™υ3κΐCΓCΊ ݝηžΥΟn.σ’iΰ€t²ΡD/νπLƒVίE%•ίΓΛϊ–{δμEeοΎXƒm‘π{“œž»λ<θuήή>aˆ¨γV―F»ΕΊ*Όϋ;QhG‘χOwS 4λ³$―Μ₯€uκπ™–β&ӘΞιv»ε;ίωŽ|γί3ΰ†ǁ˜DvξG}Tzθ!9όσεϊλ―—ΊΊ:ΝnΚ Ώ6Ρί“%Ƙ x’χ3JFψuΙzžπrgΪoσœtσδχϋ0α₯{4―ΰΚ ?9ˆA_O]0^vΜ΄G˜5υ1m± ͺššεΒΊ…βΰŽω žΝ5ˆͺ¬ΞΕ²nσV%pLcSr0mΔχ₯Ώ―dx5oσ“± ο b„ ijlY§ Ž›2Ζ^`MH2<“1V2 ‘Lāψ λ “EΫ^ σ~χυυɎpož› ―3 k όœLD˜PΎ3ΤΣΚBΩΎk·μΩ³GŽ?ώx++b„s${;»₯rAq`-Le3 Fƒdη€@μΜo»WμN–$ ½+ΏG±aŸπυI4ͺXπ^Ρ#uρΦΘ—Ρ+μ·d!ΐ±’ΖΈΧ^} Ίeτgz ƒΨŽ+1σΧΓΛXU‘GώωΟεΤSOSΓ΅m·Έ Œ+“Α’D.’Ί`bœο1Ν„q6dξΆ’§!sM\7°™B€ΨFJ³]'—^Δra3 ΚκH@D:Y“•γyaψ&ΆHΩ3ι˜ι7Ο>ϋ,Θ)Θρ «ε¬yN)p ΄<ζΩ‘tθc…ΥωAΤ©²†„š–ήJωω/) ,€μ?5cεŽiΓ†ΖιjνO•φ £“ΐ„"ζ9ΉyΩRZ[${vν‘6lς+..>VέΐήΧ"`°Μ8hΗO΄‰#ΧxΣ Οa§‘tp3§Ϋ³L₯ΎœΏΣσLv.’k膐*›ŠƒυL‚I Ύš~·eΛflβTr Tχ€%‚ύ;Ό σcqe±¬[2œ\$λε1’€ώζZŁ±^ι”ΌRμ(5ΚΌ§zg§gIζ!_$SRάdÎΠΣLΌι΄ΣN“3Ο鐟G?’μ\μ²Rώ`¨ †‡†fŸξΜ4l¦£ύ ¦^―‹°;€;*kΰΩφ38‘‰­Eο΅`u>ebΫΆm²Ϊ²:Stdφ«₯Σϋ ΫFνΙ!ŠΫ#Χ ·iikmΆ«ρβ•’Je`‘東2|΄ρ‰‰’iΤΣ:δH紞iF1šŽoΝΝM²yλ6»FDˆ|0uΰ!ΚΖ“]^‘'Ψ΄Cφ%N2M©Aΐȍ™Ζ+5rΜ»CvΝ’=8ΟY ε’­MΙGΐ΄ƒ•ί£Ψκ)νp· ΩM=‚až0α iF³ΪoIDΐτΕ£GΚ+―Ύ*uε…R‹ψݘωθΌs"ύ›Uajj 1―1ς…—ΦΙ;eΡ’EI¬©-Κ @B'<Σδ•διͺ±lš2 d8 7¦}Ν΅φ39ΔΒ5ΦΉδάύΨ–bΘ49NΥpOiAΙSΝΡσΩΨ=Ϋοχ+!gL– ώaϊΜ]»δ~R ςέ ©ΈeζΩjoΎƒε>9" Aš?w~ΎόfK«όςΏPBMuuuΖΙΣ†έ €:xH « tc_¬­Lψ†LΨ°ΐ{m.BηTδΛζ »TΖOxΝ`°X2 @Ž}`Hss%Σΐr©Ζ 蕆Ηmš>šαYlΓ֝ςŽηKτbmΘdΣδΒ£A%ŒΓ5ΕyκΡ††hΗcΗΨ6™Π/ή6―δΈ°hό5…Œ΅Ω 8Ρ;Md‹MΙEΐΘοA¬P~cΚ-όΩοΉ•‰ς» ?t0lτo†y‚ΰ9+Η“ΫΓJ;xπ€ΌψΪ&ΉpΕ)†aaxsϊobσδcV=vψl8Ψ ›7mR2Ξ‘β-Δζ‹ ξ–κμμRχΔAωΝ+GT’Ωι ‘ y([¨Ϊ”ŒόfΏ¦Β0ќo2Ρ»pYY™ξ~ζo“—ίgSπ!$_δδΒΨK¦ϋ]ά²Φ4i‘ƒτ`>iΣh?αΖ“‡~σΩ}θˆ\y\΅,-s(I1nh˜lώ5UNiκ)‘?` ωΓο/ο{ϋΥ3[߁XP3\νώ}ϋ₯΄¦Xul:GMfRΆ;‹Ρ7°8άΨΠπcΥƞ³X,™υ5κΒάΨiΙ4ιέBu nαxλ9›σΟΡΞ6Ν—ήO$͍ΝRΏ¬φ΄a8k^}³₯˜dšWΙ4S½WΊ\?)2 Χx }ΚLK—.•ψΓς“Ÿό$ΠΩλλ“Šλ³‘n~υ«_Ι½χή+ΨέΎ;˜ηwž|δ#‘·Ύυ­RXX8αΐaβž{ξ9yμ±Ηδ‘GΡΠ |zήΉτKε²Λ.˘έαdsz1c™fΌ+'b6 Wσ.w‰’Λ:“\>%΅sOPa·ξ; W?O!ξtMΔQŠ€Z DϋGšlΒΪΪΪ ί‰ ͺdO‡!@’_kk«δcG¬Ζ;―?)*!Η\pΦμm‡βφHΉμ±I `dψ(lόγυοΑ –τήχΎWέ^rΙ%JΊ4ωƒfρ>k€Lƒqa)Pίζ)ή±~C”G#ΎΔš‘šθ)¬ ,{pi»j£έž‹ΆE O.©,’M[ΆκĘ€›R‡'έ=ν½Σ½‡žœκmπ†Ω˜Ldƒ˜ΪƒPLφέ B3ε/ΔςΠ‘CAωύ,ž™τΎχ½O.Ύψb•α U™Iς{  ΐ& Ίx°;’XΓomJ μ77oή¬‘Uηa7• »X$ΘH5a›Ρ΄^ ΘΟγ<φ‚ /— ξs!γ­ ‰„ΪδŽυυχIWG—–!„ΑΎ‘“%†~r€(I²Ij“ !=ΉΪΝξ«L?^·nόαΗaΝώ©½rεJΉόςΛΥ&C{ ’³±οSθλλ‘lxΆ leI¬Ν陆Ά—ΎΎ™Ζ`šX)³/χ+6ȟώφ”œ4―LNͺ’M\ T(ϋݐCkη8eW[Ήόί“S±Ωp͚ΜdyΠNr`οA)Y䁗^nκ+Ÿ“ΡF.xb¦†Ί$ί%Κ›,‹€E 2C pύΖκΓ‘ρI‡£FŸ¦‡όΏύνoͺήπΜ:)ΧCiCα:f]]έ¬ΤΏ#΅Γ ‘,“ƒεpθcαz»M©E δ^ΫΉG>tϊ"l. 8`kPGŽ•xžφ“ΉΠλΌ½}²ΐ~»&`I8wλΔm m²Ίb±n>P¬(7Η™%xΛ?²§AZΫZ“p§τ)bRdšΙ<βςr.€λg2 f€xρΕε£ύ¨¬_Ώ^ζΜ™£Y– IEO4ŽΓ =ΦΌη=ο‘/}ιKςρ\٘¦­XΘ?ζ8œ»ξΊKσστ;ίωN-ƒδœΏύοςΕ/~QξΈγωιO*ηž{ξ¬ˆ $q7ΓUˆρ'R7‚ˆ«3ΧεΑ.?x¦ف­ύ:EΜϋ³wί>ΙΓβ½0ΔΙ4ρ$¨Χ‹•{δi1Ž(™&žλmžψ Ώ­­]ά….|’iΤݜωμ^μΥΕρψο`sFCΐΘ,2–oΉεα‚»,nΈααΒ+½Pv.ΒyμΛ_ώ²ΤΤΤΜZŽρ iΠωΈ²„ Ωψ6ΜS8RΙmϊf__ΏlΫ±CjK‘§μ¦œ/Ββߜ—ό}ίaiΕNoκ>6%3Φφa²–Ο3$D†Ο»yא‘r°8B}Ρ΄mςk“Y%wΰύ ΞNdii©|μcSo$ΞίyηςσŸ\>ρ‰OΘΎπ}Μu³­„θTy‘ži(π»¬λ›²έ.`IΦΏ¦oυα=}γ&©)ΜGœοΐωΟ«lzε•Wδ[ίϊ–ώQ?—ω—Y靃φ’>m]ω.•ΡSql‰Α³δϊfvΟΘ4}Ιj–΄-ΗΌΣτύˆYNgœZγ’"x—J”Πx¬@`Ϋ²΄έΌ‘6O~±‘]ώϊΧΏ¨νLτάΩΥ)mGΫ₯κ„Rx¦Α|tŒ€NN+91N;=ΨΊ‚dΣ—’s[ŠEΐ"`˜ΠφAo°$χΫydzΆ©ίθ ςϋίΎ|νk_ΣΉφΪkeήΌyςπ‘‡R’ϋοαο;ίωŽ¬^½:#lXτLCJΐ ΡΕGoΏ>Ÿ?#°8½›ς…ύ“Ό€Μ«=ά”$βcxOΣ±*†¦b6Fε˜ϋωώύ΅?—””Ψ6‹…Ϋ$Ξ±­H,k‚ΪIœπζΔ5g΄uuz©1ϊ5ν²™DVΔ‚˜bvόx“β$³μίΏ_/K–λdS6Λ%™…^Έ3ϋΚ+―”«―ΎZ κ$ρP ƒuΰ?zΗ!Γ– γ@~ς“ŸœqϋK/)‘†δœ[o½UwΐιΛzΰΐΉηž{δ«_ύͺ–Ιxχ ,˜Υ/2Ι4έέ Σ8GάGšυ‚d“λΜι¦K_ΐxϋŒΝ?\(a/tεJ%<0ρνŒg b^.ΘζAΞ+rΚ–†£r΄…‡mJ2ά=ήΡΦ!ωσwĚΫјοΘΛ•ώή•WζΈύœfŒ ΌΊοΎϋ”HC"εg>σαnXNyŽži‚πG?ϊ‘ΤΥΥι’,ΗsύδξžWq8';_]\BYβxp}v>A&`p¬[Šδ‹½π V" λl“xdxδEy><ξΑυΑƒeΥqΗ‘γΉςX?qzήΏ;Βι9L½Dx„,Œ₯t%ίr”}o"”ΰ!ƒ!ߏ»οΎ[‰4ψΐδSŸϊ”¬X±B\όώϊΧΏ.?όα5„+Iς³έ³‡yΟ}κ}†δ.’!‰ίiβΌ‡Φ=·A&ΉŸ\ΫπΜ…ώ\†xΡ Lƒ‘΄¦ΞΠ~‹Ajυ+Ϋ5Ι4¦3Ϋ/SBΐ£hW{ΌΜS2ΝΨik <—ϊ7I“τ’`ΣΤ0}˜ΆΪZ˜(§i£1‘Uiδέο~'φo&Χ\sΪOήτ¦7Mύζ3€ƒC4υυvK<Μ¨2Αϊe!t6IξττkSn {ζ…ε-+ηJ} CΔ§;ΟόP]]4XQξΰY牧žΑ†Όσ䀓NΚ8ωΟ τζΔ‚ŠYdIj;‘spΐαΞU[u£L$-%S[˜Eΐ"0« ώΒ9€%Σ€g3s,εZ 3\χdtnL:υΤS₯  @Γ­2rυςψΗͺίvΫmΊiΙθιωδΧz^Α©dΓ³―jŽTΘ4Ρ¦ΨΨΘσ6%Σ·Ία˜aη]π²^;7ΉΠ*lˆΨ7FVziΟ‡νu~©Gφ`=”{’ilJ¦­h‘έ«dnΡΒ»: Ιr°™‘dn±>tX7%QΎ˜k“W›™W¨υ5ΊQ0ΗϋGβ ΡΣO?-d<2͝;W? πTλΐΔ²ϊΧΏͺρόΫίώΆ|ε+_‘7ΏωΝr–/^,uuu²dΙ9α„δ]οz—ά~ϋνκΞl>Œ•7ί|³Όϊκ«ZNx}Lθμμ Φύϊλ―W²wΖR± )§ΎΎ^»ξ:uQόΔOΘ?ώρ-o6Γ~==έJ–α.Ψρ‰†ΙμΤv)ι†FL›’@?ˆ Mχ‘+₯b“I R[θ°CΡΆ|·Βί‡Ι”ιΧΠΓٜmjΔ!{3’δγAˆ΄\΄£―o@"^kδΏΫ49θ•ζα‡ΦN ρwςΙ'K^Bna|’—š³Ο>[=°τx@]@OξNιwίs²σF ς#ο>'LΆ`IεΏ ―Χ ΟUε`{3ΤSDω‘ΜIΉΟΟ#Pp­ΜŽR™w€‹¬7ΨψjM#ΤPύ•ώΈΨeΫ" ΐ±}ϋvΉώϋU~SΧ>ρΔΥν΄‘ίηœsŽάx㍚›#Nφ2%ωύ>5=ŒΥΓAκΒΒ+η^ΖE±νΙοt)άΤ*s‹αrΐˆ„DοΔ‘–κ;‰”½ΠΏΉ1„ΙΘύa™2|Ό­έκ],„{¦εjΫAχαB+=Ε‘ΐgΣΤ0}ψΩgŸzζnX’Χi’ύ„v’jh›‘wa&zϋ₯Ό2ΧN½3£ΏŸ!šΊA0dš‰%Ζhx¦aHΠL'ӘΎΡ…Pγτ8ν‚·ΐ5•N5¨f-ΙkοQό“WfhITI¦,™ςδ9yΠΪ&/ƒ Δq{ΆυΠηύnž³ΟN―Ž‘a‘y"~gγ(ψ%bVd†ςs8±{ΉK½kGΝlOX,‹@Π»i²6Β[H§3οηœ’‘U™HZ§gΘΒBF–Θ΅Sz’ωΧύW]3%‘†Ί:“Ή^ΜΒ†A2b˜'°,B Lΐ6q’ΰmJ½XΗά‡Ν€UXƒ,€ώ§}ŽM1&EΦο΄Εp*ΧUΒKϋ‡tΝyΜ₯φG ]€©±Y<•J]š4ΣTόΞπ¬ω°‡564j¨μ€έx†”°g£6nά¨»ω9°F΄<†+0ΞHV‘W†  g\¦šX>N¦ι"˜‰;>ψΑκβ(Y˜Lf’Ζοζš₯K—ͺηmΫΆ ½ΚΌόςΛjœΖΊε=c‰‹°άρΚςΝσσΥΥΥ²vνZu“Ζ°!ζΌ^4 αDŸ.Ξ³A–ζ.Ψ1ψ_”{Xq’Ρ¨}ΐFcšςΣ—b'Ok[›Tx\βq[>Ά-&Ύ›Κ•†9Wk–v·ΞΔΈΕ•CίQYΨΥξEΨ3,|£F @£₯0+M蕝‹E΄ƒiηΡ\φ["ό(“7mΪ€$Jz€a22Ϊ|._Ύ\ε;ΗΖ•ΝœΔNΟ4ΓYŽρŒΈ ΣΒkΣ΄ ΐ°]πœWZU„±”|bμ’%KŠ± Kn 9μΛΡτ˜iyY|bΛE¬\„PΟ4l&ώAΥ4Ώ³!ΐωG²1ί!›¦†@¨όή‡p–ŒηM9ΝdδΆω\ΎbΉθIfηΞ”LIœgiG ²ύ. ‹Τƒ88Ÿ)hLοsRζ6wtI•§Fΐ‘›bxΘrΪ–ɞ={tŽ)ήρ¦«ΥH<λma¨Ύlυ€0δΗ»BnΎ3ŒŸo 7γ ’d|RΡ3 ν$ ω>>rΫ)§œ’yθς›ΗHx7c€žHσ.‘―Χ }¦7ξγΠΨ=&‘•πLCύΒ&ΑΆAžρ%YS[&s@ΐ08%6S6ηD(”mAύ|Ώψ=™‰εΥζΘκy•ς<Σ%—^ͺ„³ΩΤcαΕημ@HxzvtΐΛ›«AyŽk]l ώO㠏™†Šv3d#™¦§ΉGηω$ςΩd°X,ΡΰzWθzZτœφΜLBΐθmX'zτΡG΅jt,ΐdl'ζsαΒ…rϊι§Λ.x a~¦Ωήζ|vΪI²²]P# P¨eD‰qΐp56₯ΪU|Q_菒iT₯sCκΰLό—_ΩB&ρ»˜b¬―΅"CΛI·ο|†!x¦Q9 +'‰VΊ=KΊΥΧτO*=πzRδ*SB]"<&Ί·τ`eP?£Δ65e§&3΅ΎΤρ(#r(Á9bύ½ƒ’νy¦ΫwT4Ί§ΐ”Ϋ³A6λΆ0}˜ΖŸ'Ÿ|R‰μ ΝΚ*£ω›7ξΎ’·ƒπsl‹πφ`Ω‘Ž³¬tJjθ‘ά’iP{ό¦gΘAŠ¬!(ιΝΙΎΓ9Pv;!ŠHΞ@ΠΑ[Π°ΰ]ΡC5ΤSIΎSc‘˜2„d›¦Ž€‘#ύ·šο? %ΒΔιRΰ[ΰ_£Ι4>‘ΞnSr <ζnΨKA¨««ΣBM›„ήΑ„ΦbώPINσzŽΙαr=΄Ό™φ²ΈΏ―’ځͺAΟ†‘>ΨcΥ60±Ο’+zŒe” ™œLί!ι°­Λ+σJ>ŒπΤ›γ‚3^πP]Θwϋ†ΕK/tΌyηπρ–G>ΦEK ŒΣu%nybη^% /ήq‘φYψχtχ¨l΁§eM9JƒRVϋzUηkD]œ!²£ε7ρ:'ΒιΆ!άCΥΫd°X,Ρ l6cnτ\φΜLDΐ΄ΫΒ εωηŸW!Χn™Μ9σΙ9-ΧB™ŒώΝy) Δ‘tpζγ΅ι€ƒ³Ξ‘‰uχΓσ ξτϊ8δοΖ3ΑIΐ ½K“D“mm(‘€₯ΰ»ΧΫ%έΰ ”‰sσPݚꨃSΏ@“τβ;Γb‡ΪΙ™‡s€"·Sν2¬¦ιΧόnSr ]€£΅C W«IΌ>ΑΔοά¨DοΎm:3Κ†7™Ζ EΑ³1ΙtGvεΕ›Έ€yΧ]w©χξœf9Ιξθf‘4VΉFπ³σζΝΣκs‰5Π ivL™η5χ0ΟABŽρ^Γ °Ή?͟Ή6τ“^n1шεΓΒR–N^iŠπ|vϊ!τž‚"䲇¦€@7ΌxΓ°²Pω‘ša’β©,Ρ«M>dΫ° ΛχΑ¦δ"0@#>Œτꕆ£~Τ†‚\„‘.Ζ€~Ό_f!ΜΘ™δΦjφ—fδtee₯0Hx"œ4πsέΊuBοΜΗ0~LζΌ)'όϊt•ί‘ΟΑgδŽY TΑ±* ³τVcΣτ Π ޏ’|wθzΙD6α‘Š Γ#ˆyNρΒέ‹rH¦±)ωπ=α"V6ڈςϋΘ†9ϊZ§8«r䟭Ί@#=jνšόάεξαH;ˆ|ζη‹/Ύ(­­­ς–·ΌE αΖΤ"šΞγ$ίPŸMΧΔΠ!šBΙ48 έ›2άθιϊ|3­ήμkμ7τœΡvΥn€X˘Š]ΜΗ0W@ΆxΉ’wΛΩ”\hr"”’²‘ŠF»’πΞwΚ†&ŽΠδŽQΗζF$³)΄Κ^ž§M‰„I&ξ5Ί΅Μg衴㐠?Σε·©;IZ4ΧCŸf’οY0ψγ?Ÿ ›­s³}°EΊ°#΅^i 1O„f<η)—75υΛϋz€D—νm>yf_·ΌiAΎή7κ4>žΒ#δ‘w³*ΘΐC܈x,Ζ"†Μb2叹p೓͙ÝRΡ_Μ\aΖ.ν4^¬ˆφx3αψ<τ‘/»Έ[>tŠ³rΌ„]—λΫn,ΘΪΕ—8ΑK έΛΣkAΨzΧψ(fΠ(9PΈ˜2ӍΔγ`ζ126T†›Ϊ2΄Σ~πύωφ·Ώ]έkσ•†|’ό6ΧσxΈόζ±tM|Ύσ$ΠhηγΏ`ι³ορ\$ΌυYgj½I8υ‘ œ”¬δˆ˜¨ΎΜΖE7&!τJ3α;–'Βk2ηΝ; ο YωΉβmθ•Ά]]Ί6‘Y|˜μ5¬k•ͺΥ₯’Cύ ς1ΏMΙCΐΰI½τKςύο_oFœ!e™Fή¦ΰΚσπDΟ–ŒžΙθ*7Bςœ‘ϊ n‘§μχ©#@bI—ϊσΨέT“*β‚ς9xA’§W!›’‹εr vγ^–ΡϋP β;eή«Ρ3φΫT0r(T~σe2Ι§ΏύνoεΧΏώ΅zŽ<γŒ3τVζnΰjΧΘΠk™η:4‘ύf*υNζ΅$sε8< C$λŸ`ατ6ΖdΙΊΆ§g^:,¦»q$βγΥΦ<ρόΓrXn Ό<ΉΏGα… }ΰ­>u¨OζδΚκͺ<w0?Š§ΐ8σ°,> οMb0ϋwxŸ³¨΄ΛΖg%Α…ςW=<ς "5(°‘^stk‡΄mν’ά‚ΐ;сΕgI,<«eΔn%LΒΨO;°M‹€Eΐ"ŽA΄Z8:FιpΖθΣ‘:…Ρ1Hή½ηž{τ1nΌρF©――Χο&occ£κΪζ·y^3,“)ߜK‡Ο!θ$d t€HΣ"ÝP@œ2μλ’!„ ‚²aϋ}Š’ΆkΚnκ7‰ͺ΅ϊ~4Ξ?χΚφvΏŽ„€Zί< Uπ0{φBκίΜΛD―6±ϋFΚcί΄)Ή¨ŽŽf2^ΨΗMbyŽ^BўlΧLIq“i $Σ\}υΥζgŸFp'|aΨF 3Ξ6 4ίώφ·εWΏϊ•2/—-[Μ*άyΉξυΧ_—»οΎ[σ|ΙJ%VέΜu¦`ζ₯ˆžAξΌσNωΦ·Ύ₯§θʘžiBο»}ϋφqSŽω ΝoŽΝΔΟQ—o48JŠ²QZg.d0g^#ξBsΨοSE@έψChq Šά ρ݁;‘rπΗΕΨtι‹ρ=ΩΜΘΕώoή‡h5 m?cΠ·m ­Δ‡Κpƒ+mΩ²En½υV!‘†Dš«ΊJΗ*”ρn6\(w,)-ΒφΈEΐ"` `τ0κ6₯/Ρtp©wΣN²|ωrωΰ?¨NŒ…OόΜ3Ο¨}%ΦӇ揕ofγ†#Τh!•ΙK/;8ΐοπI]Œύ>=Ÿmf!­6\ΗdP—Mόι™ζPΧ δA/FςŸι@IDATf;ρΥΎθΰζwΰΙΚ°z](ŠΙύQ PdΈ, ½‹±±„ΪΑCΟΟΖο “i‚*Lv\.:šίΜΓοG›α‚ͺ«[‹½|δΈ~IΒ?¬οsώωηΛWΏϊUως—Ώ,εεες‰O|BN8α] ­oΙΊΎςΚ+rΫm·Ιk―½&?ϊяδτ7œ΅ Ο­ŠζΎά9ςΏϋΏςΕ/~Q³’TsΑŒ»Œžo{μ1Θζ$Λ 1iǎBhΌχ6Χ«Ο Ι-ψ=X™±‚Β‘ΒNΣΨS#νΗT`ΏαŸy²ε±?²­rψOΆ{]TFίοθ-zΖ|½.jΡφD‚ωΝΛ8HCΧ–ΗΌμ)―CσΤΥΥM(Ώ¬ΒŒΚΞgΥIςˆ‘ž}/“*M™n°Ÿ†2rœ;]'“hθηDΔΚ‹Ι 5F.π“}3i` Øτ±υΠ A™Ηm›…δ%Σ,‘ή ©χ?ώψγr2B"|ώσŸΧπ­‘y˜ο’‹.R#=eς\hz葇„ξΣ΅FŸ'Ls釣ηCŸΪ~Ÿ2xΡ5"eΑ” ΐrΨ=sοƒ‚Δ¦Τ  ΨFn5„ήŸμQͺ‘Οτ ρ›ίόF>ϊяJΌ¨Ρ[ωƒφ•ϋΏ θ\:ξjύύάsΟi‘ωggώΏΔ"°ΩŸΊC8Y ΗFζ³™DιWC΅]£Ϊ‘0fWσCΑgλ˜z,TUδqζ3|“ί~&QΒSh‹Œ-Ÿνΐj™½NŠLc& ϋΫίΤύξuΧ]§D–Π—αιgž–ΫoΏ]C‘π²zυjνδΜcʘZ“—.Ζθ8|‚Ισά™Δ*\ ύγ(λΧ―WdXVTTθymθ&˜ξΘ8 lΫΆMήφΆ· λtoV__―/r¬z™Ί0Γ~όΟό|η;ίΡG ©ζ½ο}oD7$]rΙ%΅¦¦FΗΊoΔ ρAi²aδ†ίΫ ˜γ"Ϊτβΰ”­Fηc\εYy{φ}ϋ?ς?‰$Ά=β.αƒ'?Οm’DŒ3/wBQ~θ+λš‘φγΒ,e]ΊΙ…X6ΞN’ΘΣO?-7ί|³όύο29a` ŝ2ΪΘιπg0ΗCσ‡η™ιΏ‰ wΊͺg ξθ{Œ#ΛsL·™ώ,ιZ?Κq}f§g6.$τϋ‡€δ\–cSκ ώ+ޚX9μΉΙ `δε7ητHCCIμ4E’ίΌηό‹”θΩζω矏t*-ŽϊZ€ώιXZυ)-››’Έ±ˆ‘»Mž`;ps"%*IŠŸι‰ΟΔΠ½$ΓDθuW_=usz ΉŠΥΔ…ΜžμEEEJvι‚̘ǿΛIyάΘ›•Ζɍ$ά‹MμυΑ›ΎγΩ2IρYυy ‚φυΔΥK΄Q&nŠΰύ»Νm°X2 .NsaRmJoBυEκΌκqύχΛόωσε‡?ό‘\xα…ϊ€‘ωx`ρβΕϊιιϋϊϊτpΊκ+|Φh:9Ÿ‰ηy"=½=6U°]gΪΏΗ%ͺs87±RHομΓψΝιΪΗ= ;ΰp š)@f\Ήa΄ρΟ€υˆI‘iί /Ό ού?yꩧΝ+―Ό2H¦1¬02iπ¦'O(¨I*I€““Ζ… ΖFώψΗ?χŠτϜ9sΤ½ Πϋ ]–EK4ͺ―\ΉRΙ5Χ_½’tκλλ£eΧγ‘Λ«―Ύͺ†{ξ¨r»έζ‰ξε].Χ#Phα‚˜‹\,'Ϋ7wN:r` BΉ™)žbœ„ΟΙ—Οτ‰t|Φ™\ηόΌ<Ι‡AΏg `ΠgΈ§qΝΗ `κE@π’βB%˜Εq‰Ν’TόΐWΉg\Η¨ ƒΘ›λΘ΅οΝX%rΪΘp*ώ$T~φ³Ÿ•ύϋχΛ[ίϊV•ηΖ΅|€2g›ό}F>'ΙY˜0› EVvΞƒ}h~ϋ=ωδAŽ;±8Ϋη!EΖ} Έό‡ΐθμσɜό<Ι¦Τ#@½4b9¬sΏˆμΑΙ"`δwoo―<ϊθ£*Ώιž˜^ Ύώυ―G%Λπ~αςΫΤAe&™“˜kgΚ'Ν QS”n5Ώ=7œοqΑ³₯»F 3rΖ}ω˜Œl&ŽΎ=(ˆzψ\ Β7ŽŒΉΐώ˜9c‡Έπυ½€d‡AσΫθ@6%#Γ©wΣ&dΒcσσ#ωΘ8"Ή{$ωM™M; A~³Λ‘Cέ6–π'―η3GΒ$αΒψΪ”jjζΐ <Β.!δζ²²TΙΩr“o―D ¦\jΑ³pQ‘αβ™Μ;“H9ιš7'—τΜNρĞ"ξΖ$Tr|εύl²X,θPVrήMΟz6₯/‘ΊΔ“O>©^}Ή‰Άοο~χ»rξΉηκšCh>σ΄<žxŒzX:‡Υα3ž#ΊιΩΓ±°Ώ'@>lΧμGύπ,8NΫΦn7ΎοEΊΧ10ΗςδηΫ΅³H%α9Ωπΐ°£ ]Β^ΪOaca{ζ!o¦€„Θ4(4ZlάΈQήώχ«a€ƒμYg5nžFƒ₯K—jμ=jΈ€yι₯—*ωζœsΞ‰{‚H" “1,Λ wΈπnhhώ™TXX¨I£ƒI—άCΑ]LΖ«wΖJ湙穧ž–Ο}ξ³ςK/ΙͺU«Τ3 wΕ²ŒΠ|αε…{Μoσž¦f{8œ.μ’τ£ͺΔ8μβθ ε§Λz;!©HE0Ί‘―wτyΥ»ŒƒΝίΈ¨e!c? R]X]Ό¨ νκLEU3²Lͺ”a.Ό+|M»bcAΔ{γλυ‰‹5F6₯«œˆυ€ΣyΞΘζ.Ήλ»Δμ†ύπ‡?¬‹²Λ–-SωΝ:EΒ:ό˜ωm>§σY’}/bπLCΓb@ŽΊ’OD˜$»f³Ώ<Σ/ ±(KRdH‘‰"Ξ DgŸ_wιζεε+h³‘_Ξ”Φ7mΔϊdc!‹Ώƒή`Γ‹ΏρŠθυ3ε™­ΓΞΞNωΙO~"ώοpΓ 7Θ§?ύiY²dIBς;τωCΫ*τxΊ|tΑπŽΘΪ³#¦ΛS€O=M!™¦΄΄TΪzϊ•σΑ"KΌ”ε>9©ͺ›bΟK§x³ŒΌά…ΉMΧδw„χedξ4ˆρ”z·Ρ½3¨<΄‘ατvΛ-·hx'ήζΗ?ώ±Όϋέο–‚‚•αζύ ­B€c‘ηΣβ»κΩ$ώŒ’Φ―7g“π”αdΣ—-Z¬φ‹Γ]~ΡŽ ΆŒovβ0Ή"υƒ(λ̍Pτ°sς©²΄XjkkΗΤbΆΰ;ξ„|ζf’! Χ剓ΠCπδΙΝJl²X,Θp¬εΒhwwwΪnώŽόd™uΤθLŒρπΓΛ'?ωIaď‹/ΎXn½υV9ν΄Σ“/X:x¬sαε€λoβbSj(,,'6œw«Slf έl ϊ™θ\Η,))ъFλΛ©yŠΜ(•:zaQψϊύΠΣaυrb>kšΝ4„ Α~,({ =Πη±ξ™!)!k&I,άΚpI £DC…1I.Lt™HΆ y†ΔO‰žjh?ξΈγ4όR"}.viΠΓ λ·`Cγό |7ΣΰZJ >ΟD“V3Pόυ―•k―½VŸƒΠΎϊΥ―Ž€Xl"Ο4R΄ό ‘ΡΦ™—†Ιˆƒ €‡ŽαAάωΊΣ/-t†WΊ²²RJKŠεθΑič(Τ¦¨OΒ7ΆƒP“·WΞ‚‡'λέ *T‰Ÿ ΜΑ«@ΓM~Q쀄ΡOν©KΓ,΄E·Oί/kЏRBLφz½ς£ύH>ωΟλυ_ώς—ΥΫί‘ΠρΔδOθ&iœ™ΟΛqptΗ,ΜυpEO‚Ν0-›6₯ςςreΣw`"ΐu>ˆŒΈΕKWΊΊ/++ ’ŽγΊΨfŠ £ρ“ ΘΌ§ςŒθ–|—ψgΘίΌΉ>›ΩLγ0ς˜DHz·€άfϊΚWΎ"$Σ0”+σ˜dς›ί³ύs4†ρΘ“αLΈΐφΏδχΣΗ(·‹±«ͺ₯Χ/ƒCΉΐzςχ⡝ Σιθ–… ¨ώΗlϋMSs₯i/nΠUΠΑi o.Š‘AτN‡.μšλνηΤ0ψoέΊU‰ϊӟt³ΥχΎχ=΅qΌ4f˜—)ž~Ož©Υ―Eΐ"`Hκg΄Ÿ¦k$…„v–fθΓJbΏζškτ)ι ς?ώγ?€ΎΎ~Œ>Ι12tλιl2‹MςΡ¦ήWUQ[v6& K± ΧXΘ€š8»l7μ'ά¬ΐMN6₯’iΚ*Λ€γυτ6˜ŽlN’c†AθΦέR^YeΓσ—Ύτ%%™Ό‘5ηfγ'ΓΦpΌΏ΅CI3cŸ‘VJˆ7iύR_`wωhΚΏLγnΎͺŠJΩ³}›tbQ΅ΐ‘ΈΫZš.9€νmυ*±Μz7˜rσ 0νĐh…%…0†@c_„x\ThGΈHΛ‚‹4kθ 9…/lzJcX>ih€εη΅Ί ΫB-Ω΄Σn“Ά—AςΓ}kVֈgΰE2 6ƒͺΡ¦ν£₯MΕ«ͺͺ2€PZΌ:)pΐ°NΉ+©Ό@Φ>„ψΓκ ͙܏ca•¬s4*ιΔ!ΚDά3·™ΟΆErP'Ž~ϋ|πA%0”λη>χ9ωΠ‡>$œ0e2ΦODadŠ_‘8w0M‰Τύ;ήͺΚJεHgΏ ΦΈ%7Ή©6”ε~ŒΓ­½ƒπL3€ψυ tcŠ™FΊΖK’Σσ*ςF•!ˆω~„•‘ϊ7ZΆ9μ¬ύ9 LfHΎ―}νkB"Νyη'7ί|³Ϊ†L‘f#–ω=?I‚ ŒUα/Ξ§εeιΤΩUΎΗyΩlΛfΖ{n„xΓή ίzιeΩΥV*UyΤ»οiYgnhΪά2 ½~9ωδ“₯‡Ν»“~O”xΩ¦ωpΟΟΝD4Θ§$±o@Ζϋ@~uΓ(οg“Eΐ"`°θοο§[P7kkkz5±)}ψΛ_ώ"$pΠC;ναLF§Jί§›žš[œ’3½ό.œW+[_o/Φ"K@¦™Ψ>ZκΟέΈ₯g@ΧΞΦEmJ .θΝ•²mοQθιXΛ, i)΄ƒΏ‘Κ»ϊ₯ͺ*£6$Ατ—X:pΰ€lίΎ]‰-τHCREh(₯πΜ9N rϞ=š-‘ΔI%Λ‚1B?ω=τΗ'σ‡2’%ΜDΆ££CξΌσN ouΖgȍ7ή(‹/–žžύc,IσΗcTFf{’ϋ¦ό|Œ’=x“"­,α­Ζb¬ΏOς=–L“ͺώ%w^ν\ @ƒ7œΦaCρ&BtάΨν—2μ°₯qŠ)Ϊ;oΉ6ίX('‹JŠdFΊFΣV‚QgLbΓαUςaqά™η”<΄“m …aRμθZžžΔζϟ―‘ίσžχΐ˜ζVΧ₯Fv‡~’H’)‰~υLC2 RVvxϊΠW1>Ϊ”zŠζ©^ΑZή­ οH3Lxc—·τ°―fΙόyσ4Ώισ^l3$„υU.Ζς0ρΓFZœηqͺC$ΣΨ4uL_ήψϊF Η7}όƒό Όλ]οΉ]O‡ΚmσδΙLIŒ_ΜDY0šπH΄Ι„EκΡηžήo  OΤTWΚ‘Ž~sπ⇉„xjΓfcvc—Οώφ>9~ΡB5vΖs­Ν“ΉΠk K=β§;ι0Υ†MGyC―.—SυΓΔJ·Ή#!`μ<<ςˆόβΏPΟΔ ΣχΖ7ΎQe7m&Fnσ“ΏάΘώHe¦γ1>I01ά’ΖυXΔ“eΝ6|βzψL|~bA2͚%υςό.i„.LY:f( Ήf&~5ς BU½ΈΏ]Ξ\{²œrΚ)3±ͺ)­Ϋ² ‘ή`’A>UHύ|ži_αΌΛ&‹€Eΐ"` ΠήήΊEΈžΑ9ugg—…)Ν0z"#ƒόΧύ—zνΈμ²ΛΤB%6bΣ†Σ;^·^ˆΖ64ί›’‹€Α”›βΥΧΛή–Žο2‰©πΏ,½ aξμ•ϊσ₯Dt›Rƒ@>œhTΟ©–ζ.υβK›η0όγβΗϊEW³WIzτ8ΔdΪYΜ σD hθΨ½{·\tΡEΒ°L±€2ηψ²$έΕ%šXŽ)K[,Ρ’εC>nΨ°A~πƒ¨ΧŽζζf ΖLζMΡ\@!{χ-oy‹\ύυΊλ!<Ι›ξŸάΥΑ6υυ{Ζ=Ž ŠίΧ‹ r>llβqMρ€ι[uuuΡ;Α«SΌ‰9ι …^iφ΄υΘ²΅R  6%2oK½₯§IΓ5ύ½ί; •»θ’ m™ΣŠν6Y¦Š€O}!&:Ϊ4ZZŽκmLή©ήΣ^ŸzL[1’½BrCυρo|γJ€ ΥΏYΆ3#0ΤΫίώvkΗJ}eτ¨ηr³9υψfo<­cs/6mίq o0Ÿmα{[Ίδ gΝ—’’’ŒΖ4oδ 2τ‚ίو°ΊζΈΫ M)Ύn.qfnz¦Ι”yLΒdY__/‡KŒ'‹‰’–ΦΝ–ˆkMΣΌί}χέ DοZήο{] U‘αΫΝbTŒ ©\8½δ’KδτΣO~Μ%¦σ9rDwΏΜΑΞqφΒpΚȜ7ω9Ρ•1ˌ€Œ˜|ιόiž™δϋύ݈Kγ‹όHπLγΩ¦°°/•#r{tΚPhU¨ΡΰυIF`ά@C¨£ ξΰŎؽ-=²jΝ’ αm‚KμιΰϋR\R,‡w„Γ¦θΓΕΨΈͺΛΛwC%x+›= 2οχΓ£Ϊ…^¨Ζ|ŽE»vν Ž%‘Ω9&p¬ΙοbζΉ£žγZVΤx2ΐΐ&Ω9 Ν8[Η1σμ3α“:N.Βΐ-[ΆΊαalHq,¦ό†‘GŠμ’ίίή#kN8ςΒξ²Le{RχQOŒ$Σ€cΖ \xβ-e8σ])<ύΤ§χοί―~Ž£ΙοΖΖFΝ“Ψ]7·Γ8‹ε_H’\ΟΒ<%άX’Ε~ΖtΒρΗΛ½Ώ~DθY`y9ήϋΛd~6έ―_Ϊ@Š_ΊtipΗΌ•! ‚%»Α‘rΉ ›@ϊnU vπςa@η±~θίξR· !gpIΒ'e2ν3ά„ΕΞά%Λχ'™ϊzqq±ζKΒ­gT|ζ@JTJ˜Ηΐuψ?XŒ9œ‘ŸΔ“οφΩηœ-Ο<σ ΌΣ‘Ίβ\Y 9 °u΄™{ΒΦŸΌx MΞ:}­œ[d¦%ӎ₯₯₯‚ŠΠΛ“ϊŠ ”%Έ—αόl²€3re:?­ϋLB ₯₯Eu3£3kέ Š©«q͏ϊ[Έή6“κoλ2ΪqΉ6Ι ’ciίή±c‡Άγ˜vΖ₯œ#mΩ²E½Ešqy|‰³ηŸŸ±ž5£Ωστ3ηI*γΈse_GŸ¬©tH~nΘά|‚j^ή„θ0ΝΚΌωσ‚zm· €Kπ΄yGΘϋπ”x€·΅_JόžΰšΟΌ™ͺ“"‘šoΧ’’RY²d±¬ύui‚χΐJlξMΡZ'°dсM₯‡@¦YX;Wΐα‡M©E v^­TVW\yXθ&Έn42Mϋ‘.Y½|΅†‚Jm-fVι Ν$(ThΉτKε–[n‘ŽŽωΨΗ>&Ÿώτ§ε΄ΣNΣΕb lΎp»ΊΊ46ίwά!7nΤ'Ώζšk€ΆΆVΏ›—($&―a˜'&3!žψϋ]οz—\vΩe:ψ“5kαΉ‰™Έ¦¬Xׁ:Qyζ|¬²Lžt€Η™,TL*φΟsΏGrΎ‚až˜ˆa&ΰxώιω—˜r1eιeR—{°ΈΊ²Β©ϊθΓO@q Ώ 2:λΑ ΏΐB©j5φ{zŽΘBξŒyJ˜­žFV?Ϊc Η'%s‹-$‰aεwt0β‰^ ²²©°S›cxŸξB‰~₯=“ ̘HωΐΙğŸ~V±ΈJR$ΔΖΈDΉΞυρμ’ίδ•KΙ"x³)υЈδσaΧl’Γ?‘ι@>Σ©―ΥμΏƒ•ίΡۘ‘œΙh|ψι6ΜStδ¦v†ο7ϋ₯sή5kΦΘ#x\vΆΘ’’άHόLj7£x§c”]ƒ²εH«Όσ=οS7άΜlεGDΘ¦t€v’i4@η‰°H;ˆ°‰ƒ0κΌjJ7³AΐΚpΚdΌμͺΤEPμΖ ύη‰,'Q<£—˜ήgŒζ†·Λ―ΈB6oή,Οξά-ΥπNsz;nY<(°υύόλ ‡ώΉχ¨\pΞYrΡΕ«Μa»fͺμ§ύΈfξΩ}xg€LCηΧIN>Ψ_z±a©ΊΊZqΞdΌ“ ­-.…˜~J ΧW¨Λπ“”}ζ| «`‹žε΄΅΅)™&œ,ΓΎE‚ C@ρϊ±Mι…Ϋ0‘”Q:uςψd‰t’$ζε:&7τ=ό[9„ ώ+˝J8ά‚sδώΛΆ‘wΗύνέςΞσΧ`~MΆΝ’Ψ8#EL‹±φ_;oμ}aτwϊ”τΞζ@»υ!LW Φ“ΝaI~mfV‰ ‘iLΥιζg?ϋ™Όύο‡K₯EAΧa\pωRx½^Ω΄i“z£az₯a(¦·½νm“šΐPq\eΑΘ  •ͺέO™!LΗϊ4J;=mδ‚,3δλ ˆ80 ₯ζ! až¬β Οdœ[Άl™,fwKƒt-Θ—2Έš‘Ζ%¬Α΄τΙΖC-ςή«Ξ‚[ŠdTΕ–†@θϋβr;₯»"κxmΊ|κ‚˜1 Νn+{ΔO‹atΠϊϊϊ²BΙ4ΘΒWNŽSϊϊΪ-™&:lI=cdΔςε+$ΊI‘k*Jš Ώ§L ΨaΩOWo<{©P^0Ω~0$ύξη"LώΈi&R‚ΏQ%Σ`Χ m‹HMξ˜Ε2:nτ>£)\αΓož³ΨEΗ.YgVΑ΄dαΩ~΄]NE¨§ μͺŠΐΥs;š†θ€„φMM½²lώ\9ξΈγ&5?S°ύ1σp±Ά ΠΟσ2*ߜP;ܸӊ† sΝΈΒμI!©xݎŸc6{…vΌ%Ž %Ιςl @LˆGMMόλuΧΙ7ΏωMybo·;³eŒς3 )ΦuCCΏ<±§CV-ͺ“χΌχ½ΊΩ†Η3υ=aKz<5Δoxε%ΣP>Gα­>Α‡|Γ°ΏψΕιrdœϊ‘²Ωg($Οτφφ*™†Ÿόm“E LΣΠΠ }+\ΏΰΪΟ9rDκκκ’q;[Ζ4"ΙzΕ4Βlo5 Œή[__/5•εΊΑί;ΰ–R·qΣYƒ§~胑ε6#΅vχΙ Η―Sή$ͺb/‰r<Φ-”'ŸxJzZI£†^―†₯ί±τ/σα(ΣδΩ|…vRͺο~χ»ερΗ—U«V―ύΛ_ώ">ψ <πΐςϋί^ 4MYφ31JKA¦Α’Ι4t<H#Ÿ†Lγ)P#Ab%Ϋάρ"`ή£R9ι„γΑΞ쑃`ςΈi‘Heα4!‘}ρ„mφd„ΠΜχΕ”ι:{lς0w1χwbW,·#GHτZCΟ4Up‘fΘ4²ΩC€!ΠΫK2 ή{x£Ρ„pOΩ IφφtΓ…pΦ@ξn E€!0λΰBq{Ώ΄!δ=ΠD’ϋ;|βνχΙq«Ww΄†–eΏ'‘F ξktΚH₯k˜'4NͺHέ‘ξie:4)D’’3ΩφbڎᘫηTKgc—ψ{γθdί”Ρžohό,L¦όπ<ΦΫή'΅u΅Jά“Ηώ°€$Οtww«7’iΈΚ&‹@2hmmUΟ4΄=‡κόΞTžg¨'›,‹@²(//—΅'ž φVlτfζ}9ύH’.ΜγwΆtΛΙ+— jΨ”ZΜΈP_W/½ σΆω6qh”LrΙ4L&ojk43JOΨ3 '=ˆ.ί.†[ΥXΘΉρΖΥ΅jcc£†vβ£qRΔώ%K–h`z§1 fβ”(‘Χ±όM—s―½φšΕΟΠΈ‘ω½—Ν?1$IεεyΔολ9»΅³Έˆ& N‡0q₯gš 5qi6Ηdΰ»@ε—£ξzΰ7κςŒ‘žh΄‰”ΨB<ηΕ ΄Ή©KΦ,["Λ—/”ΥK"2[γSˆdOυ!VdŸ·_γy[2MΑ·ECΐŒ½½=ͺ˜fε€( VGiΰ™¦³MΔζΆiϊ ··3Οx£άύ‹ϋ₯‘»HͺυΤS₯gΐ/;Ϋ}Π±£Ω©ο αόAΔϊήxΈUΞ;ηl1ΖiS^Κ*œα³²ρn¨gš,ψ1 ˆΓ°cπ”zκ ΙcΏZ&‹mP>ίΌ>‚ŒΛPOA­.‘³@¦ΞΕΖ5lZ‰…“Kδ.ιœ—ς“ϊ0νN Gα9o’φΆΘ³ϋ₯Κσd‘Ÿ,&}spSz“|jΏlΨΧ$οΌςmrωWLΩ[χdλ4“―γF’²ιnΗkͺ0TN9ihˆ™ϊαω·²²R½ΐ±`;ήN^[ΐ4 `Φ)ΈΦΦΦ¦²£½½=H¦±ΊΚ44Β,Ύϋ=ΣΈœPΦζQ1œΚζe,™S”'›·οΤώgΞΫO‹@:#@ΉJέ<šΐώΟyψᇕ΅m„c6ŒάMλeΩ”8J¦Α.’„sμ`μφώ^μžκ—’1‘·Ώ‹½"ψN‘ΐΆrΕJ(ΓNΩ w[‘Ό!‰fHφ vt}m,ͺ―η6Οp`’B2 =ΟψzB<~ Q(ƒ‘τvφɜΪj„OΛ›βέμ偉 άπz»0&ƒ@“ぱžirr]:–N\ŠΝ‘ ŒBCοJx)Ψ‹ΙDKο –\mBΩMUe πνπ·d~­’i’qo[F|°b‘iπ:©~K}Σ&‹ΐt 0ΔΨLΊ@Λ/T((+L8»θJTR™Μœv ΌŠ-œ[#;°Ψ†0B‘b~SŽ“ ΙXί―6φH}uΌ‘ΉέγU*λž)e›6*(D˜'Ό+$Ξ@Xν1„…vΪ¬¨¬(ΚZQ¦ΐcŸ3ΕΠ&ΥίΧ―dH€Ιέ ϊG6Β±4CFΩκi|ίΛΛΛεCΧ^+'X*Oοi—μ"ΣE¨‘tα@Ο3ϋδΉέΝJξyΟ{ίτ`τώΘO’yGΛΛΚαρΊL=ΣΠ&’“ž©Β€6 Y²Ώc@άω.‘χ›,ιˆΧ^H|ΰ<”^…ιQ„Ικ*Ÿ½χλΈΞ†Άχ‚E€U¬’HJT—eIΆδΈ$ŽK’Ov’?vlΉ%qŠσ?Ά“όω)Ά;±\’Θ’l‰κ’(J’Ψ{ｐ{ ‚A$ΏχέY.QwέIΜ<$vχήΉsο}gξΉgΜσž±7―Ÿk>pΰ€\@π\Χ΄ή˜f¬.Α·0Σδ#νxŠUWW«νζέuύτŸΉ’ž#@έΌ³’uI=ԟΥ7Ϋ{‡3žŒ=μλ-RsUMΡωnK”GœΆ3ν³kT—»nŸb‡h6‡ήέέZζs²¨¨H.œ½ Νυ­‚ίΖΪF)).Q™‰ψœθΊέ΅y3μOΙβ―Α‘ςφλ_ZΎψΕ/ΚΦ­[eι₯q…Ž @ §ϊϊz…Ρ±cΗδ›ίό¦όδ'?νj­ΪŸ€;ͺ²²RΎόε/«ΘΎ―ύλςέο~W~σ›ίΘμY³dΥͺUB§υύ:YΏ~ύ5Wƒ’ϋΘ‘#κ:ΣqMͺ‘›όΖ=šζ)Wš/œƒ%²w«*|—. ίρE …ΓΚ‘‰›υqΡ:ζo&2t¨”Γ˜ΏΉ4ΦγQlWΈ©y·=#n+yXΌ5%sθqο°;€ °@9Μ΄b‘ΌmίИOgšΌό=γ3ύΡ³!C\ΊΤU½γΞ4<ΜH'ŠΧΦVμ7%+]ζ#χέ+Ϋΐ6sπ»΅δnϊcLρtΌ^&€Mrΰ Aͺ‚‘νqJχR[AΑT«Ϊ@GciŠ[šZ%€T¬:M_ΊΟmΪλΏΠ&v):ΘLsΥρ1U<†γ››α‚TO¦tŽe)υ4FΊ~υ«)₯ω™{ΰΌl>Tτ8¬sΙάy›©μα9h`mΕJδκ£Ν²Ξ<F”?ϋσ?‡α?―s2•“ά„uiW€„GΝ0Π«’u/ξ—cαBέEρψ<Š™†Mq›) qβΔ °ηΉεŠ-€±’γǏ)ίθŽ7B/^ΧΘqCέdίΎ}’Ά»ς`4΅x’hΤοΛbŸM|N›ΚFΡ•Βυw—ζŠ #ΐqάυXF s·u:nΫlM ­ Ε:fΨη•Cηš₯λbL‘ͺε?頎Νr ͺO“;'N4 ν©Aέ«ΪΊŸ¬ΰ {€ρΔi8Ω$Ή₯aΙΝΝνUΫ7κΑ˜Υ'_΄ησΌyσδ{ίϋžŒ9RMJ~ψΓΆ£¨ζ ϊsŸϋœͺ3yςd•ζι駟–ΘO<όJώμΧ$Νλ―Ώ.ΕΕΕΚ#jώόωςβ‹/v~@›=dΨωόη?ίf«ωΩ|˜θΟ§ρό)(a L8πJkΦ\2ιI@Wν™}ιA€ύqϋψqςώ‡Σεh}@ΒND`’im8βg ^FαI،D¨·`Ρ–©Ω؟f2–ž>θ¨ύΌJύ©E‰¦λq’Bεΰ2:¦ρΜ‰ŒΖ ΚνΡ»Ν§A c΄΄ Υςn[,HρΔ±’cƒ3Ν‘r#Ω QΙό§Ζšyc^o‘&,ςΉm\ΐω!'荒‘IΎ8j”w2Uζ ©# ίΈ©iŽ0€‚e8Ω6  .Α‘Ζ8Σ$’’™οԝ)»Ή8~όxy}ΚTΩ{¦En‰ΨΕΖ½i‰ΐOFzrΓΕΉc↉03έa«LΒ9©b=Π5Π)—Ρ)­ˆ† ω#i΅OθS˜Οώ φL›J'υ€jJ½@h@ΖS/Ώ€&¦z2₯k΄\…HΧ―}νkς£DfV?’μ‡EJkΉάuK©οe»TΩ·žl–9ϋλ€(’ΏKwyyΉ±·t§Ά?ΡȈW:6Ά ν΅HBGΗ%»‰Μ4MXœρ@ώ“ύΣƒΐ‚€Ά 0ΕΣΑšρϊC"Ξ|ρNΚΑƒUͺ'Ώίoδʍ‘ΧΙuκqΕ χ]{φHA²Ρ}-kž~ρ³Πk•BŸKvοΩ«2M˜1wτ₯ΉŒž!ΐg€6§Σ©ΖrΫV,XœΉŒ €nΪe~χ%M7VΆoZ/c‹άβdΔ\ΰ©S;πϋ4²k4Φ6Ιȁ”Nέ›σ™c{†ƒaΏœ‡# ²#…`•μ§Ξ4ΧZ_»ΐ“‡Φ@²L›6MΥ…Εœ_όβˆϊψͺ„Α@Β’'Dό€γΜΗ?ώqyςΙ'ε/ώβ/ΤώI“&)ε?T›jkοώœ[εHo†ΡΈͺφ‚”#εЈαΓS?™9"e΄¬ ι\–#d¦Q)οDΟe875€q" )'žDΛΣ”Oh0t€|ŸŸ;W'v7Œ3 ώ«Gύ9Ι6“ˆ.€ΜΠ”Τ1#†Λ‘³HrΞv\,yγ#Z³AΙ·ΐα†Eχe†.Ε4Ϋβέ©\ŽΛsNωL1d&€Qš–ίjθ1ˆCš[šUJΜ_…9ƒF€,3o»UΆ= ͺbΚn½'ΊΈΚOL³ZφȊ’"€κ‘*9~§L~γ‰)΅.ρ%J/§XΉ‚yQΛΕV,ΆzŒ3Ε|¦ ¦e:Άf:9Š•q8Ωζ!4(ΧαLckdcΓy¨Ζ±dΡc½‰wή)_Ϊ_J]}ƒΜΎ ‡Ο΅¨t{™ΠΤΨ&ε~ΥΩ™u Aμθ·'žψ†ΠfΚ©ώ¨φφο?t¦!3M3˜iZcμK οΠ‘ƒγ/5_BŠνf ‚y, φ¨sA / 3ΝώύUβtϋ%Η],.|8P₯œiϊςΊΜΉol˜=bΝζν2$ί/AgŽ4_¦γκ=ρ+’=κ–ϋφ+&›«5Μ7ƒΐ…ΐ•˜ώΝy7i:Ι΄­ϋMZΥΜχ―ξϊŒ5Rž>''/cσŠbyΤW@ ύ¬χŸ<'£†ί}±Pο2ŸYD€Y4θΔΫpκj% +ΦΝ,^Βusͺ”Γc:€R;ρθ 3fΜu3Zθ$ή·ρ?£˜jƌ²bΕ ©ͺͺ’A1ZλΔϊ=ύώΨcΚΠ‘C„F2ž/ΩΒχς?ŽΒCkJΟp8œβυεΑ1τΕW8ρςΡΑ¦μ4>¦"κ_xωU™ο΄Κg[$θ²Ζν#]·”ά^v+ΦεxC«Μ…ΣΞΙ3ηδ»ίωΆάuΧέΙ5Џk鱜Ÿ—/-ZTΠQlΝ«W¨p ΄ ?˜:*opžq–μšζΰΎB€)Ήͺ»³@,ξ"|δθ‘ν°αœ²„˜bHΚEΛνΪΉSΞ66Ji HΌ°I“½ž¦i]ψ•k.δ[vΘδϊσ²gχnΉγŽ;Œ¬A2Ÿ7TΤ0°/!ͺΉΉ™dQ]Z}IψC½Ρ8Σ$’α―Z2d¨”δ©ΐ°X©Η'œ›ί›‘κ–,ξd›eΡ6υΓόΙ8Α@PHpψp½š†*ƒqGυώΦ)[Œ˜―sέΊuςΠCΙX,φ°θΑίΆηΑ,++“{ξΉG9Σ9rDUMάίφΨT~ί‰ˆώοmIΧυτφ:n€γ;"ωόp @t,yTsΈθ 1‡΄Od¦ρ"…)ΩE Q=τΦ\Άr…ΤHȍ†β+1εψ„;wAΎƒ?ΠXΜΨΟlιyL@Z4u63Υ-ψJZc›έ¦"c3{5¦uƒ@gjOŸOΑmP”3*Ζ$iμNΏœ9{F₯"_gοyƒez Ξt^ͺ¨¨«ΕͺςΒ Ϋ”!£‘ωŠμƒ7ώCgLοΩMk]!@Γ'ޝ½/s¬tj€1J₯ήιͺ%³Ο Π{(+a…πΖ-i¨MD“ Yν.i:ί(αlcJVΠοH¦±ΫνR}ΆYΖ:Δ™ zέCB”κΊͺδ2©’uHg2%+ޏNBωμt:ΰΈ~ Œͺθ|£pϊJΆ'DŒσp?YΊΥ‹MM`v›/s|ΔΨK΅ΐΎb±»εμ™γΠCΘHfJ2PΆR6S&ϊ3ŸQμΨ3.‘"O‘|€ά₯δsz£έ©ΩιΎΰ¨·τp³l©9)φΗ_‡yD2ιχC»Ν…£Εia΄«Λλ‚CM4θ(‡‘Θ½,­ˆpn†ƒ£i³d/Α4‡χ 5Hρtξ¬J‡Γ\†=WσMg5rΛ-Γ:—φΙŚ“^Χθw™©—,[&£ŠΓRΗ¨ °Ν₯Cόςέ†Χ¨δ{,2¦4"«Χ•?ϊ¨JΙ§Ϋjs”ωiΈnΠσνV0<M“cΒKΈjΦ£=ΟΘ%€’…―C† ‘ €D=@&6—AŠ‰zΤ ϊόπXv Ÿd?S@IDAT#ƒ²Π)mNA=š~˜j’ψ3¬ΏκΦI§y0zιρ“(ΩBtΥ@•ψP$ΫBηυκκκδθΡ£Β…A^[wYQω»α]K5•{ᱦDΰƒCAw™ižhΔD˜ΨΓJɎPL¬ΙI²)™E@+^―O(υM-r©žT@ ¦96΅­¬(C†U“/΅?³—fZ!ΰ°;οΨ'ΝpœaT¬ξ3>­`°Ϊ¬βv§'GΈέ ΠLσtςXΨ\ΉΡhΩΈœΆˆΓ–:8ΣЁ֔μ#ΐTO£WJ5Ζ .”ό>τpg.4«šΖ?ϋ}Β ΅ŽbiwvΘpϊΣ‘†tά,qωή²Ω`θ=€ή¦ ΟΑΐ£dΌ€™Ζ v1¦illˆo6_²ƒ@Iq±Œ6Tjΐ"¦τ‘ύςMp’ήu’AF FΥ‘ΩΉ s–8”ΙtZ§ώΝ©ͺz±r/Ύ_BήoκΰtΈ1Ε Nhoj…QήbΓόŽ«R)*pΦ…œη°=w.jο2:Fr@'Ϊ:˜Fωρ/}IΖ" «λdϋΙhTrr­t]‹½z ηX¬Y–ξ―•‡ο»[>ϋ‡h&»†-Ύ7Z$vιεζ‡Uͺ§Λx_φθqΡ­ςaAim‚ξήΠ’ ώΪmž(6ζου€§ϋφ1₯S£Ψ<`‘Aπ“Ν“/­ΠYͺͺφ›5Œλ» ―Ϋ«ΫΏ€lί½W†έ’ο΅*'vΪ UάΑ%θΜa—EηzdΛΞ=R}ΰΐu{_ζΒ Ι ΠgD Rd_QDν 3 νy&@=6™Ψ’ίut¨§6…ϊ ύJ‘Y£ΆSϊӁ†ώzn₯ϋ―o¨οΚ²3M$7"&LE‹©tMΌτγυΎ³gΟ*ΗΦ‡±p‡’χ©Iώα1‰yΨ;οΌ#ŸA΄Ι|W(»rήΠηeΚ©[nΉE~ϋΫίΖ’;{vMI^ϊMY“R‡Ɲ+Μί›΅"“Ών7¨bδGzΧM‰ΒutS1œ+T–Η`<`, ν̘:Rί$#UφΫάvQ4ϊζ/=iέ ‘Ώt‰Ίcp)”I—Λ物 X΄œR?Μƒ@Π Ο©S§0AnE΄«³eF₯π$ψF»;Œ4O§δδΙ“i<³i*Y AΫ< ΄Dͺkδ""θιͺJy†3^iI‰jΖȈdΡμ]=ύΌ¨‰wŒ}¦£- AΆ`?Σ‡šΎι!³-θ±Υ„qΖ +Ψ ’Μ4lς²άjχΒΡζ¬44ΉΖ”l  CΘω=aό8Ωv΄VΞ61Ϊ>φjΕ'"Χ5ΚPD^•”D AZΎdγϋϋ9ΨG8%@@Gέ?ΤΙi<ΥύΨί±2χŸ>hk¦3ƒ”δq)”Xuυaq‰ iYO>₯t #;’Η‘XρέIΦΗ/}ιΛΒ`ΧΕ‡.@―nmG#Ÿ|«Ρš'VYjΞ΅Κ’ƒ RςΛύρ+; Οiϊ)yDtδ‚ΑW1ΣPŸΑΘΗGoΚ%ΨΎ.")t΅3MoΪ3Η²€Φυ錹wο€δŠΕΖ3=ŏ?_mgͺ']?ΧfΞqγ"ΐχA7nΨ υ`7œλή_­Qχ v€%l R=ρΧΖM›TζέvγŽƒώ~ε΄ΥY°ξrMšμ(|΄3 7LΙ|‡wΡ‰©­|α>229B±fηͺΜY4Ί?¨G‡aλ¬MΤ­υ>]·Ώ|¦μL3 |€r@αKψέwίEΞ3jΠkΐψ$*sτ,‹/–©S§*J ΚΚJ΅­'ΨQ‰ΩFmm­¬εύx>,±^βw}MQztQχ #‰{rMύωbΙΡΕRΔI/ΠΐΗ€Νfο·V_ ­0@ό>―œΒβk Δμ>°MHΝισ2Ξ4ŒΠ2%»πεΓ¨+vΤκxΒι/!‹ΟnL1d ύ~ζϋο"KBy•°Ηؐ(((ΐυduζ‚R8Α™FνΛΤU™v5”Σ,^ŸOyέׁšόX«θ{wς|³‘°#ί”μ#@ΗΟ[θ#τGΫbu‚5 ηΡƒΫΦ3Ώ ι@ φzΠt;ά!θw Žλp¦‘ƒdΣ…ϊ8cg:ΞgΪθŠΜ}˜ΞsΠΰAJFp‘ΆΐšμδθωVΘσKŠ–?lΠM³fwϊΰι²JIŒΞŠ½k)ΚωΦε»WEYu$άΣw ¦₯~„€Φηh+km…3r\ηhK‘Δt 5Nα8ιpδΔργ&Z6ΫV?~Ό|α?+{ŽΥʚcα°ͺΆ•RψMω~r}Υ‘‹rΌΆNόȈ#T z €Π\ΏΚˆW2τΆ4_tΤP.ƒΉlΝ^―GΙy=ξM›ζXƒ@Ά8rδˆlή΄NyCή›;ŒΉx—ψπ{λζurδΘαl]Š9Ο Ž€–}Η‘C¬]·^*σ`SBŠ'ͺΓ˜5’Κ1ξSUψ­RφΚ†1mΊΝh σΧ p}# u2ΪΑsΐL#9\wi3ϊρ“k2tΈι”‘ϊϊΎΝϊ긎I[ e‹ξ/ήϋ„Α Γ†ƒ>ηm·†Ύιθ⹞ γΞ4dκ―%f}νώφυ@&p~όγςκ«―*F˜<€"ψ(SΛΚΚβŽ,‰­1ΣμΩ³ε§?ύ©Ϊόψ㏫¨Δ:έ}ΧΣMTWW_σPρX¦xbαK}͚5ρ‰’Ύf΅3φ‡mρA€Ξ΄iΣΤVώζSRG@)WZΓβα|Ε-m^L©7oŽθ!΅ °œ>~†ϋ+’η‘–|Eš―Θ±sŠΩ@ ΎŽž“žΦΦ ΔΪ֏8Ε|b}:=aΏϋM1dN"ͺφν·/_ΐŒΣ%Κk8­ΒPγςζͺχ*'4nκwq¦―Ν΄0 ³ψΦΑH±Υ&gISž'yωθ3S²†ε2½ο9syœp¦±πu y}υψ;κLc•spΊΉtω’Ρ+―ΒcΎe:ŒΗ3gN‹έSAf±¨ ηx½‚t œ:uR‚:2Jdΰ’L“1 ε–ςR9pΆI&;ΕgΟ¬ηΙ)0Σ΄ ψ`Π 8Ϋ ˜wj °,}}¦₯l|^ζ°b'Ζw«έ’θ£/caέƒ@:8}ϊ΄[ Žx)Œ±Έž9"RνNŸ;vT΅gδzj½€ήxO2hε±O|BΆmΫ&+vμ”ΑA›ŒΚw"MSjνιΪX+=ό <ψΰG»΄EκγΜg{Θξθp:δr=XE†rۊy”ΣιJCk¦ ƒ@v ΌbΩΉs§Τž>&JξG¬ΨΝ\₯KlήRijZ$›6m–‘#GΦ₯μtΛMq2­άΌM>;v€DάΨ,–o:ΈCEŠγ Λ*·•ψε΅υ;e;ޟ\3Ε p#"ΠΠΠ; ²”lKν Χ†i+7ižΪc“ι-τ7(//Gΰΐ „m% x! m,¦τ Ί/|ϊ₯mœΟ Ώ³θ}}se}sΦ€izθα‡εχχeϊτιΚIfαΒ…ςΐ(γ ?½•ΞŸ?/‡–  {λ­·$‹?όOΗ>=›ϋίώχe#ΌaΩ–¦βω˜‰Ξ1S¦ΌΛμMž{ίΎ}JρdΕaΓ†©kβw­Ές»)έ#@‘Φz©žtΠ)-θŸζh`χ͘iFΐTAεΚdIΥ9ί|Y <eΊ;ƒΌΡ6›U"y3ΦӌyςΝΡi¦ƒΪ±m”O¦2Y < _%œihœαΈγ Œ?›Kόα2ΩΏΏJH1LgS²‹@AA>rωζ(~μ4η‘n8?ιTpΩ½šώy6­«Rί<}κ΄ΈΑLcƒαI=/δρΗl|0:ΑΡζ,(·©™bΘ$g¬pκΔ)>Ξ4±Θ*IΚr‹C|ΑB±Jƒ]MΙ……ERZ\$5ϋvΓ2 AΘ…sπ¦9v±Jl'=)ΩG€rΉ©ρ‚xσ\0 Ζ”n\F¨%l+؜šβ‘VΩΏ:sΖ› ­;ΠvEζd»Γ#κΪq[I*wΥ5θLcsΐ™AdtrχxθoJ*h‡š蟁-sνζ­`§i–A» §A›taUŠ‘“—dυΡ †AωӟώŒ2,λώOΊ1SQ!@fq2^3θ(_χJω+Wΐ:†ΥbΪ‹M1άhωΑΰί₯K—(a œg”1&£rά…’[8T.˜'?όb³ΥΗέχh1»θ±AeK—I~ΐ#CC6qβ%–ΜkSK•κ)d—β W–-_.wήu—b(ΠmgχŽΜΩ =C€γ΅Ύώt ΨNΰ˜·ι%4G'gkf3ΖΐΙπWό“†$tΦ…ΆΨ’’“]C‡Ÿd’3 ƒϊσ\4%g=eŠψΗjqν7ސ9sζ¨τψ'-U.—KMτuΣω…μ4wάq‡2τ¦β΄’ΟK΅oηΫςΏˆΆ<₯›V:.ψ:t(Ύ½»/Ϋ·o—Ο}ξsς D§p‚e„dwˆ΅ίΟώnΎΨ$n!`DhZψ$mZkΛEΠ΄Ά¨ƒRιοφg1[’E@γΜΕΦb°œ:A₯vβv&κ°λΖ>mΘ7c>YdΣS/ζήdδ•z\šΝΑ6¦K£sš)L! eΔΑšΉ€ u8?c± Ε% 0ϊΙ fƒͺ½;€ξœρΟT‡tΠ–Λ‘άφ"2iαΞσ?ή§ωypˆLIuλΰ fSͺ΄@Χa„„Σγ€οS[^»$ΖίP%sAΆQθ—ZΑ)„ωŸξΛTΟgκΊC€σ †σ§Δ†”|Κ™†Τοͺ ™ ŒCn_Dͺμ:Σh―»6Νώή! ί―ŒaδΤζ­[εΪ !3[Θρϊ R9l’'ξέ™ΜΡ=Aΰ"ζ«M.bΡά =<ζLΓi+ΦXnϋ5†Σž΄oŽ1΄E€ŒvLΓα ‘Ύ»“YΆjΫ #ΊΊΓ –Όj°ΣS©Σ;ͺiΆ%‡ΐΈqγδя~Df.X(γ‹œrkώU£}W-¨€άˆϊ8ΙξΪΩu΄VώŸσG2|ψπ5ϋΊA@₯+ΗB±ν‘ίYνηδDΫΣ©Π;¨b6K6mΪ$;Άm–`Ι±Έςπ` Οe±8Cβ ’κ½σeύϊυςΙO~ΟML―Ή.οΖ\Τυ€ΐž={dŚu2’,φpŒ‘ίbTξfμ`7ΧJΌ]”εhγ3{χΚ„ ‡Ϋ2Χ`HΪζΈnl±’™†ŒuΧκεάOgΦΡΞ4I7n*φύώβΊ>ΩΩiΏ …Bͺ=ξc_Π§ 7v/SϊmΧ¦―Ώ³π{-)―ΘθAN&˜Ÿύμgς裏ʂ ”3N·D01ΓBΕξΑ”ίϋ½ί‹3Αθ6T…<ψ‘e€IXά«‹G,^ΌXȎσΘ#Θέwίέm‹ŽŒŒD"rϋν·+7=0Ί=ΨTΈ ΅συuβ-ζCDgͺψΠΆΠjs@žWΫ̟μ!ΐ±ΜΕV²ΟAJ§‹1ήbΚ»ΠάΊΰα³wAζLqθHΓΘW›ωi£‹ΪG™h΅fΞ6/^ŒΧ7_ ™B ͺͺ Ž[—ΔκFΚ _b ‘2pR£Ώ‹΄ η£NͺΓ† ΝΤ₯˜v;Aΐ‡Ι„Γn“ Κ‘ζŠ\hnU“ ]Ω]ͺ“SšΝ @Ηΰ#‡ŠΣηgΐ–mηΝ₯ηί°o’•ΖεwΚ™Ϊ³rΆξlΏφΤοF³Ή—θΉ ϋ!ΓΉ•±4}ΪJYNgH·Ψ]ΉrόΠ*•ΦΦPqχψη<“²ԁUŒ₯2γΨΉ rΡcJφ c°Š€rΓ{F?2TwβΙαuΘYΘn£ƒgΏ_nζ3rΜΨΏGάαh°•Δ•†ΤοšL…vo)œ7ΚA5Κ,ž¦cLw»’t΄>τΜYΈXΆžh’‘a»ŠοI-:@h%g›.Λ› Θ}χέ―μ/ϊ=έ];f{Θ Σ }Ϋ›ˆšγh»}Υ€Ά¨ΓΙ<ζ΄!ψ―9©cL%ƒ@_" εί ΰθGo_{pHΤiώ ζ)0²slbσWΒq>€ΦbξΌσNΙΛΛS \Ζ>Π—=x}ž›c‚Μ+ΐ(s NΎŸV.nΪžα £ίi]]9λΠρΖ }yxžCVh•εΛ–Ι˜1c ϋiWΐ™}Χ$~`ͺl:¨ηΨΘπHM!&WυΥBΎZ°ŸkΞ,F¦j`2ϋΙχm(t’IΔ\Λ/fΈΡi…χgφͺLλm`ι’ψ]oλ/Ÿ);Σh`8Π‹‹‹εΟόΟεSŸϊ”Π‘†TX4πr …τR’Γ ™lt$5χΣ‘…”@©ύ°0SE%ζ’~ύυΧΥΛό‰'žΈζΑK¬ΧΥwέ~WuΜΎφΠiκάΉRbΗ‹ˆ©ž`ΔW―#,ΔΪ@i|ιX Ύν±ΛτŸΧ'g.ΆH ψ:§ 4σŠŽΛ€lΙ4όΆO#}}]½8J˜γ%€ͺΠίμΘCΫ* κXσΜt‘ΩΨ ΄ζ|ύy€oΪΪy"›θέM…HhžΐβΞƒƒ€K˜[ω£}ΠΘρ«ΠdεeuΐηΕ"l”™¦ςήϊΪ™&+aN’ΰdΊφT­δωζ Μ4 žI,κΩBh—'δ–“ΞΚΙ'₯€Έ$±ŠωnHυυυ²sΗvρ†Š‘<0M_Τa#~ί­p¦‘ŠAYλ­·ω'σ_¨Ώε"•}ΩΙ,Ζy>δ8Ψ"Γ0Ή\d¨0s#Bώh}šφ‚K—/‰)ωΈr›-t„„£dυΆ£Β΄<¦…ΐρΗετ‰ΓRQ|/TmΝLΣΖhŸΤΙ(E°x v‚¦ rΞ4\θΟFΜ€`릃οΎc‚lή΄QŽ•zd ^tW¦ξrπ\«l;vFώδ~_***Ί;Τμο²@’5Υη¦ίKG‘cŽΜc:ΐOΏΡΆiΓ )ΘJ³vυ2 •˜ΝΪκωtœwˆ?€lΪ°LΦ¬Y ΆύΗ2u9¦έm;xπ ,^Ά\F‡₯2€ΐΞψ A’7Ιε»Œ( Ι’₯Λδ˜2dˆ₯lMGS­O ΐΉsgΘ άκΖ΅PΏN(”΅—x|A€i=‘‚0Œν5Ÿ,|₯½;‘ύ„§Tσu8q˜ΎΘBts φ‹ώTίρ)R ΅§όΉκR”βύj#£4½RΙ Cv˜}μcrΟ=χΘψργΥΔ’Ž4tΆ™?Ύόγ?ώ£2μςΈDπω»§…©£ž|ςI•Bͺ³6xΔσΡΡv[gǚννΠύςδ)δ²σþÝbuα\Cgζ g )ΩC@;ΞXmV …ώ3λΡσ·ΐ₯άJ;;ΪLΙZφ4bώΤρΣ0ΪΫue5`Χΐ¦υ0¦Fk…rυ€ΞήΥ™3υ7N`b°oο.ρεVΔX ΄ΰΦH¨Q©(…Ξ€μΫ·OQ]r―ΛΊ¦ωΜ\$ρΒρ±+²MπujΑ»TS)š~Θξ΅|δΘ8:^Wΐ₯R‚Δ_ͺ¨Lω-Ps,v8ΣδΊδτ±Zα3fŠA Sœ€³VMυ>8δ«”|Q#»^xŠΙo8C2κŠ”ήdΕ3%»x‘κ‰rϊ"θi¨ƒ“™FEu*gvέWΩ½¦ώ~Ά'N(‡Ÿ©‰FLυ±Bv; —7žoDHm‡Ιά „Qΰ;wμO @r’#ͺ,Δ]OΟαˆ)£θ$©#f>˜:˜ΪŽΕ(ΧξΏ_v8#Gκ/)LkΤϋš ΣΦ#¦©E&‚BώΩΆ)=C€‹\ugΟ)ηΪIΤΛ³—p’5ιk$όŒ.Τ³~1GeΚrΚ¦q6ν8•ΉΕΊE±M‚‘~Β™†,Β‘β ΘΌ/‡ΰd©e[v؜εF@@Ώ“Φ­]+U`ΩSΰ‘0ƒθπ―GT2χ‘FŽ »,r[‘[φ=&«V­„ͺU»dΪ0u }…€Φ•™~•λ-6a‚ζZGΕ¨l0Φ8έ>9~μΈεyΝϊψΎΊώώt^s€CΏο]―u&n3ί³‹ί)ϊ½’Οάίixί=v¦αΑ@r:§Πi"QȐ©fσζΝς«_ύJΎυ­oΙΓ?¬ΎΣΈΐ’XWmθ៏όγςύο_|πΑψ5%6₯B^/'QόΝE*}ύ‰uΝχξΠύFVšššjρ‡}­Ό:υ±˜ΨbC΄μQδ om:Σθγt-σ™΄ γ§cώšNΜ†gΖ~f°ο¬U7sož=Q'NςiάI,d;pϊœ˜@ŸŽGƚg&!σ=]>|Djμ@Ίš²(½₯Švj3¦AN’ήΘ ©Ϊ·[˜Κ”μ"@ΉAοϋΛθ ¦ aj㍟ύ>ΰ>,Mp¦ρδ"Š…Ρ²΄i&˜  o, >ζώ¦σMjςMω­e?k›bθ-z<νή½KZ0Ÿ±y 1;Ž€Οq€ΕWΗΙ½B'£SτΆΊ?^cμ€Υ[!ψI–HδΉffΥυΊoΡΤHœΣ)’)V\£ςΐ™‘ΆN―S<€N§Ÿ΅tœΫ΄ΡΏΠΟv#6Άmۊ4baJUν½Υc8 xΐΐο —Λξ[εt­qόκ1” VTVΚΠβB©9wQZ ΛA6ΔΜ% ΅ύΚ:¬»χTƒŒ9 L„`‰3₯אμΤΡS°“8cΤθ^;μ+d#c€v¦ΡΟh―7 „Η$υ§Μž=[6¬[%ώΒ±bρ•A %:θRP]«·XΌc₯jοv8ΰ|ˆ4iΝͺ3Ζ5NύϋSκΏsηΝ—AωA:‰χ\GΘQ$σύ7$l“ΫΚ εƒiΣaS¬VUυΉ::Ξl3\/ΠY±ξŒ8άA f°M·Υψ(om>qΈ|rμΨ•zςzΉφώrtN§½ΔΘ”λ·ΗMί@|τΆ{4ˆtNΡ‹<Ǐ—… ΚSO=%ŸόηεΫίώΆlΩ²EFŽ©N§ινΉυρ‰ν1 SI%*¦τԞ>}Ί<όσ*%ΤϊυλΥ‚5χ%Ÿxœωή5t¦aϊ—Ρ―v*C½‚rŋÏP†ΛCxYES=uέšΩ›nθ%^ΌΣ6hΌζͺΨ 9“βΌ5ζΠ–ξsšφ:F€r†œκğη»‹_±gE}ΰ™±:-.€Ίϋp<Ϊ°γΦΜVƒ@κπ=ΗqHγώΊuk%.E€lΖ!σ΅D¨Α‰‰"ž 9~τdύ>uRΆaJφ £2!§α‚Ξ‘όmJvΠΊ!υJR#[0Θ‘’ΡΞΐOλŠΝk•He85cΣνDk˜Ώž! ΗeψΝ;AOμ+i΄2oΒ† ¨3δ8ΣlΓx<ίkΎdΜ‹Yb’A}ςνiή‘ –¬ύΡΟM-Φ”@‘LLKœp θ7!»TWWFΥhΜמ#Pƒρvθ`5X UQTq θI‘\‡ΰ%›§@NŸ>v²šž4dŽiƒST)Υ΅ `δ\©M…~²Κ8ΣΤΰ˜Κςr‰€₯›ΕΘwCp‘‹,½°υ*fš·ΔΞΐ<3ΏM9ηΠ)^tφ¦Ys¬A “0yΚ”w$œ?@,αQŠ}ζZ…%ρμPd˜Φμ4ΑόΑXλψ@Φ‚}Δƒ€F€ο$κΑΛ—-“M»vΛν%~)π0P+*"u½d?)VylNŠ·z€Μ+ΑN΅W%ρςLφD¦žA C0εoέΩZ€M @gγ|0ΡY‘Ο ˜–μ~dΩ1ΐΡΓp¦Ή˜‘+1Νv†3ΫΠ·@Οαλiβ€Δmζ»A /θ‘3 ΅Ψ|Aσ4ϊf›<χάsςΝo~S>ϊя*Λy/"#ιLσΜ3ΟHq,z#N^ΫαbΗΌyσT©)S¦¨λΣΧΈ}ϋvωϊΧΏ.Ÿόδ'είψ†<ώψγ2aΒωΕ/~cΔiuΌΫp£ž“4σ‡€Wg.rΤ€&M½ˆx7ΤΞlH‡P oΞ£†Ρ :˜Nenδ‰vΐFΏt\ψέ ›,Ƙ’΄l‘σΩφν;$PθS ²qC>ζ¬csY•‘ώ=rζμ™μ\œ9KΏC€†ΚΥ«V ²urmΣψΫ•s†E,Έσ*eΣ¦ ΘŽŽK=¦ϋxYΎαˈNk‚nC‡Hψq`RAωέ¬"ϊS–oη†=Ÿ™½{φJ^E8Ά §m‘‡uΙξ±IlίΊ#ώΌ΄­j~zƒ@MMlέ²Iyƒ {#ͺͺΓ‚1jύ»·T1[mέΊΥtˆSζ6ΆΔd΅•σdœ†rΌΐ:2ήΘπΜaŸΨ²ΦWΘΞ΄sΫ.‰”‡™8f Σvθΰ ]|ωΩ_΅ί09%hΎ§Œ€~Άw ΕΣΩ3'Α„T y %Χ…φθεξqyreγΖ J'Τηλuσύ¬›ίο—|8Ü«ΰE0ˆi± μYm0ΡΪ?λ›/#}ί%ΙΟΟWtτmͺšŸ) ΐΎ #™N•0ι(μ'˜άA'CŽ Σ;˜bΈή žΒgΰθΡ£ϊ}Cš/^gΑνbuΑNsΝBoΫ+W“O¨ϋaqδO›Ν*/Ώ'N3ΟqΛbΗδ΅DΞ rjοΎ}ρqs“Cs]ܞ~&N<%‚Θ7³ίρ‘υaE–‹³uƘυΎͺ­= Zξ=ΚXοΚζ€Œ[μπΘXαLγ‰ΈπΜΤ!Ϊ°:λΧgNΨ?ΨΉs—œ:y †J¦xςrΦ‹ΎΟΫ!@&ήp₯lΩΌ^ŽΐΘcJφ`—Ίϊqΐ8ζ³# …έ&υυυΠo&}Ω»œ~{&κΉ;6ξ”Π€―D4Χ.Ά΄}d°A9Σ`‘ΦWΰ–γ‡Ž+FΔ~ šΉρ΄#@ݎs0=²_œΑŠ}¦sC;d;œ!Γ…ΓdιEpn?žφk2 ΆG@Ο‘8DBaεIyα€ρϊή©š.΄?ΘlΙ:¨g_Υ>©;Y'Ύb0{@ίNτ#¦DvΘ<°ŠUνRΜO» ΣπM€6ώrΡ~ϋΆmβτ„ΔβbŠ'šά¨oχΆ Ϊc ΏνH!eRψυO8:"Φησ"ίe8Σ$ΧKμΙ ŠΕΖ‚cΙΞlJo kLΥή*€ΌΆ‹;ŒT|,½}dp|‚ΚΌy9S{V9λD6 Χzm…σϋΧ^]6mX)β b ΓΆlvvΝHν$žΒ;δΐώ]2i+B6>=oθμ(³ύζF@ΟGV―^-[φμ“qΕ`₯3#ΜI¬Žβ¨ΌΕ96‘ %>ΩRUυΏ₯ρυ·ŽŽ1Ϋ }‰€–³$`8,*w6Žζ6ŠΖ6Γ`¬Ξ \ΈΠ¨œyνϊyκΛϋθ/ηζϊόe)v„y»΄\ύsŸΧέ:ΣPπhΓ3Ÿ?^εψΒ /ΘwΏϋ]ΕBσδ“O s›έzλ­’››+ŸΒηεύχί—Χ^{Mώϊ―Zzθ‘ΈMΊPΠΧΕ”M_ωΚWdΤ¨QΚy‡Ž2:Βƍ ·ρ‚ dΙ’%κzΚΚΚδΏϋΏ•ašΧ€ΫKΧυ݌νhŒh$fp« ‘―ΠΜηΈ"`?(Γ"μf₯ΠŒ3?"t0ηt5ΌΠ ^ρbφ2žcΨρ%Χe“† Mr‘φ,½ 2•ύσ »@­yκψI –ϊΰι ΡΫFo£.η-rJτλΧoPl[¦ϊηXIχ]S.p,‘iΛ–Ν0,"W2sp#Β΅έ@ŒŸ<:ΉΘAͺ'»―Tš0™Ψ΄q“ΪkΖe€Œ~‘aν¨Eέv‹ψ!3<»z—ΆΆgšŒkœγœΊδΝ;”1ήSΐ(2 Δ’γEλ-BT­Χ›ΪγDΜχž" u»“'O"€aD ‡JŽ§Έ Ž Η ’ͺœJ9vl6[·τττζΈT€¨“Φπ ƒ[•Έp Π>ί ‡ι3g•~Η&uΏ¦ΪΌ©Ÿ_.*-]ΌTJ‡‹;?ΆH›ΨdΊοX™G.65ΛΆνΫβ:Έn#±ΊωnθCH)³qΓj€ή O:2ˆuΕΩ]kz?ε:τy€ψsηFκΰ½°ΕlΧ;Νg/ΰΒ Υ8₯ί%َkζDIΦI5-ccλRRϊσ@΄xGΊv'mtΊm05+άΞ5(ΖžΟτY§ˆ™YD@E.πNyχ]™;{ΊDJokdζN\I²ο ŒiΆZsΗΰψq²bΩByγ7…φzŽuύŒeρΦΜ©ϊέηdϋš3gT€•ftΎ,™QΡΪΫΛ£xζgΨin-‹Θ΄gΔ³θsχφζxƒ@Ί mυΰΑρc’26ΚΡ„L0^”β€ΐ‹ ζ‚MF:ίcΎdeΆΆΆt¬«Qψ˜§…ΏKg‚DLO8˜‰Ξ(ωŸ)ŸωΜgδ«_ύͺΌχή{ͺ%:Аq†ΗpπΝ_|ϊӟ–Αƒ+'nO7θϊΊΦ‰ζε5L:U₯t²ΫνjdΛ–¨ρΈNώς/R|πAΉοΎϋδοώξο”γΟώύϋ•cPŸŽΖπδ€ \·vŒDΓ$Τ’ν Œψ90ω#ƒeΛ¦΅R ZzS²‡@ccƒμBZŠΏKB.‹2Yπ,η"jžiŸŽ9O’½«κg24κΔY‰Kc=£€Ϊ:Σ°‹΅AD lΩ΄E˜Ϋ›%έr³υ‚ΉcΐαΓGdνΚνΑΫQ†8½―γOj«9bρ–¨θΪuλΦΗȎJλV­Ϋ¨ˆcΘω‚ˆšχ»μŠH§ 1²!­°_Σ˜Ζ– ‹kΧ—ά² \¨Σύ Ξ[θoΔƒTOΫU*Q6¬Ϋ»ζ$ζ‡A EΆι`ΧΞ­β ”œnιίi ²"Z΅BΙοεΛW(ΖPΚ3S>Εκ—ΰ„GγΦΤΑYœΠϋΚB>DΈSA))6iͺχ}`HέΊi›bs“ΡnΧNΡ86ΉσR<Ό@V―XmŒ§½ΐ»?ͺυ·m:«――›—,ˆ~….Χ~Πυ)΄φ_›©ZρΉiΣ&³`ΪŽαϋΆK'œ―YSI¨Χφ+™y,ηψ¦τ={φHΝώƒͺ £tι)PyΔ¦›`Q@±Žι” ιjΏχwnZθpόρ}AF&2θΏρΪ+H«=@lω‘Ά ˜W8“¬4b=8ΠΫ}8ώ  “χή}[&O~GΙ'£χχΏΖ>§½hΡΒ…²Μθw—ω₯ΘkC€mς£ͺ+ΤԈC[dΊ_„©H£·`Α|5žΝxλ 9³―/ )āύϋΔνc$R9΅ΧΝ£²”‹6:ΒÎΒy;ε΄)ΩC€ύD'ӎd ·™χh¦?λQ _'}ΑΚ ")Ε_|ρEω«Ώϊ+Εθς―ϊ―ρ‚ργΗΛ?σ?+šόΰjBo΅Ze*›eT:_w:φξΫ«ξΰώ菔³ŒΓ:cηςεΛΥ>^ηθ1£Υw;pΰ@ωΤ§>₯~οέ»W6;zXUσ'Ž1’bΆLuu§Εhζ™*„žσZ°Ε^BΜ5θ ϊεŠz©5ξ˜oΠ|ΙkΚPϊωj!–&<* >GŽ )Ɏ;εTŒF?KΉΣ¨B Ί¦ZV._%ΕΓ ’ΤΕͺCΪ€ƒm€ž Ιακ#xΖVΗeg›šζ§A %τ»m3XΒN?¨˜ ’)žεv'MΨCΗΘ‚‘Κ1’ K¦d:­Zπn A.„ΰLv;δΠ‘£Θί[— 0gχχνή«aiŒΗγΠaΡΪ+8`¬Κš+»·οΑ»vG‡υΝFƒ@²θ9η; a΅Ϋ]b RQ¨PμΊn†ςΫ™'Α‚²vυR°lλΊΎΩΫ+΄>ΝΕ:2SDΌNεIγ5Y"K‚nΩ]΅_95υκDζΰnΠΟ qΛW,RF‡‘2–ϋ6±p:Ψ8ύvɐ#5G…ΜΆ¦RA@? >[Ήr%"_KΕ;H΄t#«S9ε>™r‹n‘Υ«–I XJ =]Yχφ§kΟH.δ5Σρ]-‰ί£[υ~ϊ!K¬°€²Ώ΅“ϋΥcΝ·dΠršΞL;vμΐΘΎ"ώR7œl:te»#½|Pl\³Q‰+%s}¦ŽA θqΟZ2ωΏπΒσˆ½π0Ύƒu2%G}…μ—ΔβΞ[Α=’[X)oΌώŠΌσŽq¨Ρυ—Oύn£#ωŒ™³d$˜cFε₯•FγHΝwαp°ΣŒ*‰ΘŒYsdΧ]z·ω4\wP_;uς(Φ\@€΅Κ+Χ¬aΆΉ\«[ΌΑ€m―¦neΡΟV›šζgš υΒD~η\ήθΫi;…ζt°otmŒB37MΥvΞ4Z@pb9oή~v·M1_hlBϊω-ΚVfž—vP™ Y@€σ|Ž==Όχήϋς»ίΎ ^_PE›rΎμΝϋ"ϊ~°z‹ΕŽφ‚‘byŒ7&εS:φ::Η΅ζϟ'O–Ϋ‹=’η±ͺ΅€tj l‹λ 0ίO,vΛιΊz¬ΞUμχFΎ^GΒ\J\?ζœ0l3κζΪΒΉž†€S›S<ΑbΩ_΅ηͺŽ—NεΔτK;΄μbΠΏvΦ`%Ύ7N§bΘ7ϊv;Ψ²ΆAΛuψџ‚…ώύ΅tκLΓφGyDώε_ώE±Ά 4HώτOTžyζ™2eŠόΫΏύ›<φΨcŠαEΙ…«­ƒˆ― ’ΛsjzU>`,Ί“α ©<'Nœ(n·ϋšΙSssԁ†Ξ?ϊ˜ ^κ ί41"ή -’cGjΔ•;τg4uΔn@e“»_\ycδ䉣2wξΌΈ‡‘^0ΌαAΉŽn@cΚΤ kΧ­“²\Ÿ”ψmQσ]N͘β©"ƒ‚Σ’ζ¨h›±Ÿ™NΤύQUU%sΰ©_8$OB½γ­ζΏWĝλ”Β‘Ή²mγvY‡~4Ε 6oή$»wmOhXΊMφŒρξ “%‹ζ σ/›’Y˜ζmX€†ψ…{*kAL" ΌξΩ½[±ώٝ™>Π²›ΖύΥ«V«EX¦nR/SZε;(4(Ε κ0]HΙΘBYΊx)r3›…σ86ζKJp,ς9η\† —GlΑ[biC:½Ϋ6ΥΕ-žε„³Ξ8λΦ­o[ΙόNμ+tμή ‡u8°†>η†g™juτο υΡIο’aλLκν›ΡΟ «˜žϊΜΙ³’7<V>{ΗΞμlB=*WΔ“ο”βΫςeγκM² μ"jϊΞƒ@Wθ1wώ|ƒ¬‚Σω•+9β’A̎Γ:ΦΊj―λ}h‹ hςŠΐΒΌLŽ?ΡϊKΧΗ›½‰PGΫ²―Z*ΓnρΑι‘ςΊ»'žͺ $‡εϋd#”Ž"}Ά)=G€ΞΒG#Ί_υΌyΥ‘€ΩfΚΦ`Λfΐ%‹y^zͺ96Uτ{‚‹Otpyξ-ž@DΕŠ50#2Υ&;eυ–Β‘ζAρη–Κk―Ύ$/Oš€„©§šqί1l7ΓVέ·”©ΣgΞ–ΡΕažkΗ$m£λ˜8b©"σΝΨ²°Μž»σΜ΅ͺŽΎ–k0? }„ηη T΄ΐIFΩΒ;½j€pZ·8αˆ›‡uΟ*9q"ͺ3€&’ӣ̎ή! εeθτδσω”½›­rΧϊΑy]λo½;›9Ί§π9ͺE}@΄’mρ{+νœi4tš`Ήηž{„ι“žώyωΙO~"ίωΞwdΒ„  Υΰf'ώgQ¦IT &ρ‡)ό~ΏͺI†z±ρšΦ―Œι4 ,J³ΛνΌN‘½ƒ%‰¨‡Sύ0:D@ ·έHy0sΖ‡*¨K`0:C¨³NΗve쩐ό’2wΞL‘“–)™E` 0^Ήi«Œ-φK;jΔ§l£xcO Ω±@ΕΛ– ιYtͺζO― ž”34δϞ=[ŽΥ—ό‘amΥ%Φ₯V‡EroρK€(Œgm¦RΨV—ΗυϊŠM7#z{xι₯x7Ϊ°;Β€žΔΙνΆH@r`<[l>, –ϊsgeυκ5j<ͺw~Ϋκζw―Иafہƒ2"Qςv‹κ­BŸUΚs²aγ¦xͺM#zw‡λ>ΨΊu‹Tνά/yC‚βΒ"ψe†_ρeΪMΉŒ•.ΝζτρZτΧu„n·›ΓΝnƒ@=fV¬X)«V,CΜ(, –)™ΥκβU»ψB—rΒq{|2}ϊ‡J~°m#?Ί€-Ε]ΛXԞΏp‘Œ)Λ—"/Όγ"γŠ”ϊμRGχΥp”QgΠΗ₯x:S= ΆnΫ*Σޟ&%# %<ΔΧ½ψ¦f±ΘpΏδ•δ"ύΒρ:¦Ÿ’άT‘}UϋdΩE’[ΎοV€ZvvzœΩΡkθLΓtqšƒ ²ιΌqŽλ$0%ϋθηˆ~"Μ € UΘ"€χeͺϊφŒNuͺ$zNοΖ’#"ί}χ]5Ω‘GχΣiEΡ/Εζ<}b8:Λ°,_Ύ\MŠω}ηΝH“²˜_ΓNIi‰ϊN:uπ~X(^―7>©S͟8μWΎ8¨˜½ήϋPVœ‘[aΔΙUƜΠϊQcΩiμ‘Ρ3ΝΚ0IO6£\ΕαMΛέGœ MΗ³Z Κx‘;aˆΊΊEOƒDͺ'‹Œ-τΘΠ>.˜?_9|˜ώHK7΄kdΛ–-pˆ™%Ε -‚!Ώ»°υΘ @KŠΗεΛ-»ΠŸΣ²fC*μδͺ•K%X8\r\ω8”3„T κγΐ(Ψ@ξ,,2§°τΕ{?•+Ώ‘κj,©”’nXQ”ω-ΒμBL’λ²Θ¨BΏ¬Ω² iχάH·vΓ\«ξκ) ± @š¦ύΘa'πΪ]‘>ΜΗ²žΗΡ_Λ–.Oόtϋέ5cφτX‘¬ύFQ?,ΆR.p~ cΆί₯lΔ ΒdWfδ|ά§ΩΤ?ΰψzYσώσgdφΜχ•Γ₯½ψ!±A>e»L2XͺΡ-6O‘ΨΐP“[zR^ΞΕω.ΤSYŒ^“ Ž7^:wΞ[΄Dn+CΒ1–ϊ ίՎA!›ά1 $KW―“5ΈS Χ b9Xs@ΌαrΨR˜ΡƒΆ›’6w7œivξάΟ†bt†n@λεξύϋχ«ugϊ$bMR ²Υƒ™iΜϋ«—@χππζζf΅d΅D³ϋΙ]ϋŽτ°Ιφ0¬\[΄²wΛπ[ΐDσ΄”——Λ‡~(O?ύ΄|γߐϋξ»O±ΣόžΝτκ―««CΤ]΄ψl©=;VέΔϋοΏ/ΏϊΥ/eςδΙς³Ÿύ,Ύxqχέw+&>Œ\ ™>}ΊJSΕk/,,”‘#G^ ‚ωu λ ‚]fΊ„ŠαHŠ:1kΒ5΅ΫώˆΎ€¬ώA('«W.–Y³fΗ=;dΫ#ΝοδΠ}΄tΙYΉa³L( (:ω˜ ?ή{ƒF aP¬Η”FdΖμ9²}ϋφψ~σ₯χpL³?(kθ°wιb«Œ#+Mτς‰§E'Y°pα—’a…ςαϋΚƍk˜ο€ΰ8€Β³¬4τ"Ά‡ΒΈοΕάαŽOF~λΣ n, Φƒ(Ψ½»·ΘΪ΅Ρd<‡)ιE`žχUλ7*Η™BδΈ¦§‘Βm₯|Δ.§ y°η›΄‰ι…ύšΦ˜†eύκυR0"’œ•’μPW Sφ9€xLΎlX΅Q₯uΉζζ‡A  ΄AΉ=gΞ\%s½ωΠ½½Ε=“ί€+;5Hθ$ͺidρ’Ε²dΑ)/A€XMΒfmE<:ψ|™:ε=ΓͺšξύΉŠ~~χξέ+‹-„ŒYΝR* |ς3Q”’‘δύωΓeΓΊ•§[2q’›M-―0Θ`ΑKP°Η"ˆ…L3Τ·»+ͺ ΰgέ\—ΗΊεΔΩ:™?oΎ ~ιΌΨ~έ΄•Μ…Žγ@ϊΓΘ-±ΟT}…“:₯z\ΰ(Y€4~#σeύͺ q‡‚€Ž7• =@@sΪbζΝ›'?ϊяdσ¦5’_y·Ψ‹?‚TLΠι32ΰ/φ²X‘ͺΔVtΏδΊΑ8;δΙ'ŸΔzȌk$υ{,ρHσύΖA@5¦@ωkb~―[n/r U8ΦηpΤF2UΨ6C<Όx‡N(tJΨο‘χp d‘ NnΖV¦7ν¦‚Sz2e“ΓύΞθΚΈΪα“{Z ZμAqxσΰ„S­Ψ8R9Ÿ©›”΄{ΡkGφ™ΆrƒΞ4@@9κ1πΤ”ΎA€ŽθGWΐ!ξ€KΙy!¨mŸυΝfο¬:Σ„’βωϋΏ;yα…δƒ>όΰrλ­·ͺ+›:uͺόΝίό<ώψγς·ϋ·ςφΫo«όs€JeΚ%―YΩτ―}νkBˆόΗWΎπ…/(c&Ÿύμg3 /œ'οη“Ÿό$’GφƒkΊ―Ρ£G«ϋΧ¬~˜? έdςyύυΧ$)kξmx±ΐ0™”ςU―rln7VΒωςΪ‰Σ«œ{€ξ#F;LσΖΐ‚°Œ+tˆCys^ΫΌκ "n«ά*ΖΖζywΚCΕx-L=ώ₯e'ΝSί›*+—’  —μKޜŠN"±;ŒΘX8α\n½‚ΌΚo(ΞLHzά5ύξΐDΉ@&™`ώP0ΛTΔpˆMRB‚#NJ6JΈ`°LpšRžΨ„>WJΝ™ΚΧ  e‡rψ…q?#Ș|‡Έΰ@£8(Πe4άW" φΆ’,DšΎU+W^Σ†ωΡ;ϋ`Α‚ uΒΈοWi?’RwOΎ²ΐ€Bj‘Ό²\™9s¦’φ62<$σ½;hP˜2e²„ ‡Š•¬4;ΑΰJΉ(ΕB¬ˆPu卓Σ'©ΉηmfL¦ f»ˆ!< !7Vΐ‘}bY4e¦Γq3ίΊ€YΟƒƒδ„bT;)³ ˜ΤτA;H{΄Aλ"ΫΆmC …7$R†»ΗF૝ΡUλ|TΠQξˆSŠn8Ž{υ΅W…‘W¦ŸΊξΣzη}‹-–ΗAƎΖϋ?ΐΙ\’ƒ§ΨΑIΡ΅V8Ήά~°Σ,PAnfœ&‡'Y„ζ z|YHEΤ'wΤΥZ|3ž`Dž]F–DdR3λσWk™o]!°vνZY»rr<χΊ”ύ#ώμκΐžμΓγhƒ³Si{όn₯—sΞež—ž€iŽι ­‹plqΡοΥW_“|ϊΗrώόy U€¦π>•¦/υΰ¦ΞΪΩ>₯Τΐvο[Α=(³ˆωοzZ~ϋΫίΕΜsΠ~7Ζvφη! ¬]΅y›άUΚ9ζ!™Ό=Η)Ψ侊 ¬ΫΊ]ζ͝ ΞδΉMہΞΠ::eοξέ»Δ*AŠ§°²iwvŒΪΞM{‹Ε&N)cŸPkΗάΚgΝ”τ" ί™\£ηš~ςΐKsΕ‹q”±΅@IDATu˜eζ8—ΘNcJί ΐ>:V¨" Ή•A©©ͺ‘Υϊζbϊψ¬νœix=Z™β'™[>υ©OΙχΏ}yσΝ7ε΅Χ^SŽ+%%%J ΌτKςΕ/~QύξΉηυ³š‘F₯B›όϊ!IΧ=λλŒD"ςΓώPώε_ώEƍ'‘PHπfϋΦ·Ύ%O=υ””••©sΫνvα>ˆΟ>ϋ¬|ωΛ_VίΣ}mιΊΗΎl‡˜γC‡Λο~χ’\h<'Β‰π’/JΡ@D%ž^ρωβ(Όxη@y­TUUΕΗZ_ήη|nέGτBνΥWε8"£>Rξ•ŸMEZuφ§hxžCξ˜+ –―T5Ζ ίϋ‘ΐη…eΩ²₯2εν)R2΄P9Δ؜©GZQ} Nω]E²uΓVyύΧ…QtZξυώjM 7+Z.qήΌωrξμi€ζΓB¬’œ§qΏ'%*Η-ˆnrG†KΥΎHSAgŽ«Ξ³=iΥՏψ\SošΘ΅•·`16 A»r ‘Tαφ£~&»Δε°ΛΫ`αcΪM#;Š6lX/«–­–’Qω’ŒϋqKTTΎ'u6>.XυζcAΤυ›Χn–U«’ΞOϊ=‘T;¦RΏC@Λoζ#~γ7q—Ε‘?A,.¦VνM|aΤ’jCYnΙ(DƒΟE‹+|s2Άzuσwϋ‹…rγυ7ί’α`}œ€hΠΆiVY‡5ΙN3Άΐ!·ΑΙξ½g¨τΙάgδQθyI|nh›8W{NŠοΘ‰'υZΚo\Jx OΚ&Κ–u[εw&_Εέσ+5Gތ0@kΞμ™’W‚tͺt\W¬ΝQِΉϋε@Eδ,Ψ όE£dυͺ₯`\›ΉΣέ-k9Acύ+Xΰ.Κ Κ=₯.€ΰCτ>Ί ˆv[t~ɝμ4χ—ΉΔf³ΚΛ/Ώ$ΥΥΥF/οEέΤqή !ŸD†ƒ•¦Ά’.NΣ~—z\ˆΑd%·Țεkα€ΆPوΝϋ·=\fKκplλρΝ9=ƒGτ£Λ›―Ώ$‘όβ­xTlph·0ΕH―τωT―MI+εoΝ#wΥϋSίR,5ΛΰXΨ—[ΌSntνΪ΅K^Cζ-`ΏΧΙ<$“wΕQγΐ$glLΛ dκϋƒ}Yτ5ͺζA Λœ9sFΆlή(άJ R?d2φp*…"J+δγΎ}ϋϋ`–/½_œŽ:W­Z%°φ<¦Θ';˜jΠMZη&\jsΒΎ|ωr³.–Ε‘‘υŽπ=ΣTί€Φ)C°“œ9Z'{vοQλύM—ξΠ™&:P£Γ–ΐρΏΗγ‘αΓ‡ΛŸόɟΘOϊS•Jι™gž‘‡zHu#•ΕY³f)g—Λ₯φσaΰDIƒ―Αεοtέζ€ΛΜ+―ΌƒΧ;Κρη'?ω‰ΊζΔϋΉσΞ;εωηŸ‡‡ψ«ςυ―]9ήθλKΧ5έ νhL˜s~€I²mΛZρ߁ΘΨΈ½N‡M··n Oαrθ` “T*.φa:ΗD·q“TΠ}ΤΨΨ(“Α΅pΕjΉ§<,·‚•†OoWOΥξ)q(…ϋ·'Λ"XL(Rώ£Η0_0Ώƒ!Ÿ}Ewδ"7hκhmKΤ’iD'œ‚Ϋα<²H¦M&sζΞQŠ†ι£d4uφ€mΙβ.cϋ’κ l‹!Η± Q°ƒΔ.‚£ΞD4U•τψo{„ω<Lηφ€WdHQΤΌN€uΊ–r^ΙutAΨiξ” ;vΛL°Ψ$ΐ’?›©™ˆ€~Ÿr²ΝT”nŸKχΕz••¦«·jbk±ο¨nσSξP?˜‘ˆ‚O?jž—π2›|ΏΣ?Qξ›7–@αX±ψ+Σ€$ŒG90"Y#γΕˆ€½γ΅hΊ'ξJγά, {C4‘εΖργΗΰΠώš4€νρήR+‡φφo\Κp.Ψ’ς#X|υx\ς2τŝ;wͺϋ5}Π³nΧύ@#S>―[Ή^ΚΖ*ήΊœuvJθΰ\ά-’Eςξ;ο ΛτΉL_u\ΪNyMζγ%K–ΚΉΊSβ+MPΙΪφ ؐΖ-ΆΰpqΊ|Κw άfžΨkύμ?~\^mλΘιZ€4Ρϋ=™¦λ3P3‰Ήώ}εΩ²§J½W™JήτFθΪOβΒE“E‹Ι–΅[₯d\ΎψŠέQ„ς:“…Άθυy#ύp&ΘΕϋβ©Ϊ_₯Ξhdz&ΏωΫΦς…γϋ4d Χ#žzκIΩ½k«D*ξGι#°Ÿ †\ ±Χ6™T1εΓ…χΟκ/{Ι#’WyΏ9|Pžzςί0ύzœAΥΘT±ν»ϊzά1(–L0ΗjΟʝpδ<€7ο΅žά‘žγδcŽsαTύyΕΐΙΰ>3¦z‚¨9&]ΤΤ”“'Šέ[SΆ'¦£'Χz˜&ύ‘JΩ΅sdϋiuΡ’Γ.™ZKˆΞ_°vπˆ ZfpLί£ΆWΚ–΄\GŽŒ( ΚUkβιΚ“9‡©“Ž;*«W–’aβΚ΅#ΐΪ!%# eΩeJΰYt¦ηŒΧw+έzEπΕ§_~ŸΟ'wί}·|γߐίόζ7*ίπ?όΓ?ZΦ­['MMMHυχrΧ]wΙ·Ώύmε|Cg:ΦΠ@¬ΫKNΌΤ6ž„Ό.n·Ωl2rδHεΰ3~όxε?ϋYoβΔ‰ς•―|E˜ΪI—κuάμυ‰ ϋˆNŒŠ]΄`–䗏‡ρύ6l_πκΚRŠPPΗBld¬δ–NεKη+‡šΊΊs=)^ΐMS]χ xLΓ6yκϋrkyΎάW[€Εveΰ>d’|PŸ?Rα–Pΐ/Ώ“ΩŠ+F=}Fo€SΌέŒF{φŸ•ΪγgdΐύHο4ι8_ξIA'‘Ωΐ‰|„ΕwζJ!<ό_ϊνΛ²xΙb՚飞€Ϊ?ŽαΨΰ’‚ω €ιΒy±‡) zίθ`μJ2t‡”’γΚ_ή(Ω΅c“¬„Σ, ΟiJκhΩQSS£"Z[ΡΔGΚτb,ίΕΧΆIqΒ¨Ÿ‰p„S‘ΙοΎ'³gΟ†Α$j c{¦€Ž€ΏK–.ΑBμ:)›/ώΆΖύ6ΡέiψΈ`βGv›’±y²sΣ.aϋΤKυωΊkΒμοŸ,MχδΙoJ^ρ-bAŠΤζφN‹ρ] J±ϊΚΰΨ~»Τž>&“„@γΗ€‘ɏ7-»™&b‚Φoί)© Ι(€ηλJ SŠΠ‘f`Ψ!UψεΠΙZ8Τ\₯Ϊ7}|°¦ξΪ>œώ‘L~s²”,FŠ¦°Ψ±hͺtπE·Ί>*θ(βŠο K~qž ΖYΆ|YtwΫ—³Ϊjώτ'τ³Ί}ϋv° Ξ–`©ϊΖ&ΰΩΕΤ@ Rψω GΛΖυ«dΝκΥύ©’ΊW-'θ0=ιε—e ¨Œ¦Ε¦ΌξMoρX«%Gξ(vȝ•ω2sώByϋ­· “U=£Ÿ™­[·£·₯tX±DFΐΑΜ@=6/vpžN7ΕδΊ;¬‘·EδπώΓJ0ι;ΕΛμθŽi-_h{!;zŸ{φ€ Jpΰcb/Ό7!­›Δ@μ“;/Xq,Ξ Ψ ο_εcβΟ-‘I/=§ύΉfΓ΅=OΥΟlŸ\9ilΪ΄I¦Νœ-·–ε)ηNφtoήkIŸΈMEuNœ|L'”…eΦΌ²Ξ0ζ΅AΙόΜΪΆ±}ϋ6Θi8‚e&yζH dΘΚ»O<Α²wχΆΈΓ@ΆΏ?œGΏkHΔ±uίΉ΅Θ+‹4ŒΩ{WrΚέ #xΨi‘Ρ`ύvΨm2cϊt!ρƒξγώ€U_ί#SΩΨ[-‘Α•BΫ§ΝΘΠ μΪΆ[6oήάΧ——υσwλL£―ˆƒTtn£bEη•ŠŠ yμ±ΗδίύίΫ ™axΰuS)M:U₯ˆb'žxB˜ ŠΉC{2θ™2JύWέϊΚ’‹xܞx}zoό˜Ψ~^“.¬ίΩqΊNϋΤ“*ΡL!ρξ”·$·θ±είΛέŠyͺu—©νkή’_v«Μ™υvήP9d{2&ϊ[π~u‘€Ž4/ƒΙ ,?$VΊ€ΠΛτNΙMΡΨTxoGα'zεβ₯ΛςΛ_όBQ’ς<¦?ˆBχEχεΪο° ²cΣN)»½PςGAγJ%¬ϋ6:­ΑΓ±λΗβzιy`ώ²Ι―υkY³v:€#™Χi[fGΏ@€γ‘…Ζύ… ζΒΈ? H1‘Π›Α¨αƒƒ‡Ε.DΑ†ςΚ3ΚΑƒΥN}n]Σ|v@’μxΊΣ¦{δ~δ™ζblg…r›2žŽ―τHΘο–^|QV©”[Fnw†[WΫυΈε8žφΑ4 „`άεό5¬4]΅ΠΕ>τΩiHa_80_&Ώ5Yφ‚1ŠEŸ·‹£Ν~„€€~‹}N§Sμ`„Dj½τ8h0£σdš ••Λ*νδ₯―CΧ6Ÿν FΤΏ˜Φψ-€@ώ`ΦΉ}@nΜ‘Qχ8„²Ί³Β71#nΗβλ}C²lέFεPΣΣωqgηιΫ΅Ό)Λ^|ώE‰ηJι½LGιμΉ#Ž:8:+8ΐ+₯χ䩬 "2΄υ ώϋ©eζΜ™#uu΅βŠάc₯αή•H7nGHb /€‚άŒ,ΉŠ±ξ«sηΞΑVω δυ\Ή£"" ;τ³ξδυΥ–:ώ¦υς0tƏUΈdx Xίή~Gή}wŠ˜ΪW1Σύpώόy™ϊή{pΆQ o ‹GΙj<3Ωzdp* ^ΐΤσKFΘ‡L7ιΡv“ω–zLS9rδˆέν #l–’Αχ‹³μQ±†GA>»Πjo%M ΦmU*7X$†=Η†χ†£τQ)ϊ0RΤUΙ<υoςΒ /ͺtul†χΖϋ4εϊC@?.(Ώ™Κ%²;‹rE™³%R‘α9ɈΓkΈAIv€dαz αΜXJDΚ|Ο4ZnΡ‰š™œξRfΣ¦’Β“AΩ‡€&‹§@@Ύν%™ΎώώΠΎξ£ύϋχˌ™3₯2?¨œνXC»¬l-WQ€ΟΊ² Ωd|ζ:p\§s‡)™E@Ώg> ¦) K¬λ9θΪΈΓCόpΘυ«ΰή'Nτ+9Ÿ΄3MΫ.Ζ+‚ΛLiBV˜Η\ώλΏώK–.]*ρ‘mΈθΟBλo~σ›rτhκi!h¬ΨΆm›ϊO!Fg]jΦ(O(½?ιOx(ƒŠΞ’fέnϋΤ ±ΜTY―Ώ*α‚Jx­Γ›žΖό΄δv…€7Ό+$Φ‚{$R2)Š^S 8œ\%«λQ§ϋθβΕfωpΪ4°ΙΌ S?9Θ#!GΚtŽœQSΰGϊ₯ρ‚όΟ―-«cQm¦?’λFuΣ‘fΕ’•2`\‘M‰ QV=Žˆm{ZtYnΚξΝ—–¦ωŸωŸΈχg—[m‘κΟΏ΅|¨――W MSS£8"cb¬4ϊiο-B”αpŠΔ;Α všU»s~‘΄ΖXηzΫz9^χY ˜ζcζ‚Erοΐ<ΉμbN[t1Ά3,(³i¨ 5ύ#•^‘‹λ³Ο>7Ήέro'^—.΅ͺU»φKρmyβ-pAUΘ»7E=.H•Š(Ψ‚±a9uμ΄Μ ³Y`ι ¨7ί±ZΠΰσ:̏­oΡD±ψΚ•Όνύ@LΔ,*ΓIsΜtO‘’ΑŠgΡ’ΕͺŸS:G@χί³o’―ήyο‰HΠ‡Κ]ΒΕΤdΔf=/ŒbAχnΘώισΚο~ϋ"h¨OšΉPηπΗχ°taͺi:ΉΠN/~ΠΛχ(½ͺn°Ν'O•;Μ/ξ)PΜ“d €Σ›yΧΆͺώάΌy ΧηInΙ(Θλ ˆj>έWΗfv ‘LΏ½ΌP|H ΈcΫFYΎ|…²kυwyε5Σ.Ρiύ]8Nά: ¬Ό.ΙE*Šdδu2}¨ez’4θ–ŠΒΌόΚkp¨y7Ξς(³’iσf«£ε%—.X*₯c ”^ιΩΩ|dψΈ γ]°›MΘ;‚CߞόΆr†ΠΧx³aoξ'½θg™γ…²… ±O=υ”Όύζ$qŠΐFσ ±Β~N„θŒώzr€ΡXθwΥe8νGΔVp—*O|‘ςώΤ·δΙ'Ÿ”i°5ΣN‘ί#ϊΎu ζ³oΠύB6‘ΕH½1 Μ ’MPϋδ9²@Β+›ά…΄δ+7l’εΈFύ>ξ“‹2'ν·0ΕΣΞ[ΐͺ>)žΌJ_N κΧNOžlΤωσ κp#SEρΪϊZ0 ά‡N“½5‡δή~)ςFuσkkG-a­0€{±Ζ6Π)‘ _ 0ϋΡέΪ’•žίΊh·fνͺέϋ₯πֈxΐΔ}΄΅ψŠ\`tΟ“-ΆΚL8DΡχ£ΏτGitχ(ύ"Χ₯°°Pξ½χ^•κιΧX˜Ÿ1c†b₯±ΔXat}έFWŸL]ΐˆa¦fβQ£FΙ‰“W`^}εU;v¬άwί}ρ:ΊnWŸ<όˆ:ŽŽ?ύ½θ…ΒlΚ”)ςκ€—$[ZΚϋ0(ΑK'“Ξb‘ΈΓΓΣ†φσ‹‡bςρŠr¨‘ΣTyψRsΊˆΡ;οL–ωίη$πΚgϋdXnΟg₯μβBlπ,œXβ”ΗδδΩ:•šΖΣ †vtp=iΛ2sΪ,)Y(%χDΔ Zψ΄>202Ρϋ“l7e0ζsAφΏό…lέΊ5.Ϋ] ΩΠoΨ ΗΣ…σηH¨pDt16νΖ}5 ‘ϋ{¨"`Dš)kjήZθ·ΰ'qγZvΠ Ž4SgΜ–ρΡΕؠӚ”S$νΞ”ί·@nτΙ©sυςΜΟ.+WTW`δv*zΌξDΔΚΤw§Jα |°ΘψUΤjΪΦΓbς;<Δ'ΕΘ1;cΪ Ω²eKrhjέτhy@zψχή{_Ξ›)α’qˆfχ»£4PΏ€ ύΠ-›Ό†+=°θη"'Ύ‘›Τ}Ε(Pκ}“ήxK…δSƒάRβ·©τ©”ΛΙΦγB.pΈ°;±²@ή›9W^xώ7*θΔΘπnPŒMjϋ_ώκ—XιU $§—΄τ“NcCR~w‘μΪ²[₯te~wΣOiGϋΊoPΛ:Τ͜9ς²ιTGJŽΓ§δjT;Λφm }€b§.ΑHμoˁΥκ"ϊ«<ΧύD]ϋ·xΏ½‰΄¨#JΒ`γEκM8½€Λ‘Fχ4E’vt‚œŠ©ωΒo§œ.ϋ{ΐ˜ƒdυz σž0 Ζ…„ρi΅—θΞθξΕk T€ulBl]ΏMΩ«™>έΘτξΐλΏϋυ8ζajIε>VοŸώηδπαCR0δaΕFcσ£…l4ρ@ԘΒr]BΗ‡!ΚRc gιΗ€`θeο+δ<ŽtΏ₯YΖ™ΩY1ZŒZ²Μ 1εBΞ]βœsΉKή»»$/ΰ8&Ι,3Ι"K²d133£ΕΜhΩ²,½ολΩή¬IήΥΞΞό³»mk‡ΏΊͺ»ΊΊΊκ«>8|δ(^κχ“€ϋΤr{@¦.X>8²;5„(+‘θΣ›šΔfβ€κ`‰@'Gq΄›[“¦ήyCδxRqOύDz%ϊ’eJ&E ŽΉqΈ΅^ K>}ƒ^θ‹qcΗΦfT]†·VU‚εΖ~6ωM² -οF|z,.H%‰±j%B”š―U: :gαΐx™xΤ¨Ωq¨[Φ^'Ό8`Η€œϋS‰|Α†1鍁ΟMC©3'P}’ώ&:M\aνάΊ—A§ΡZZu=Pd„Ωuμ ό&³ΫGŒ‡ΆE^ά\OBTΉc­ήV d[Bϊ*ς(3%^d@!kΒί=8μœτg bψβδΘ’~SpͺvjαfΊ0 –A–:Œ•ωͺle›ν':j[Νε€Υ™Σ§OΗΠ‘ƒΰΙ©ΟR¨mιΈaφT…>εa‰O.2(8ΗιhϊπÏ Zg­ ώm^š9*•@έ-tΥwή~CG|†FΉH“ŸVJ­°ΚŠΡΚφι€·s]ŸA¨QΙU‹|"JjuΔίΛƒ©<n»Χ ZLΑυ,―Ϊ$•Q/Χίώ~₯_IP”―'}­ΣPΤ!«—Α›o½YψTiζ†ο–-[Ž9³§!5‹εδ/ ιZ^’Ο…I»|Ο­΄‰YΎ\8ε—ϊ'}­+osžŽ;- <ΈyΧηΏtΉ~‘kΧ%κΙ]u‰μKΗϋΒGτΤT›ΟΚBΙGΓG ΗΙ#'‘έ֍δμ„ΐΪΪε’N™/Q§GΡ―βmΞrO tώɈ2ΪeΎUϋ΄–₯>6kοΨ±Γ…ύΏίσY*՝ΫΙuξ@”—ˆ’τψmwyW«JΛZ(’“Š‹ώ’ΘΨTD{Ϊ ©ψx‹Ϊ±ŒΖόωO ϊΫ¦L±υυX=hJj―W>h,ͺάΜΜ™3°vΛ6tΘKζΪΖ Qϊ/ψQΘ›HΠZ˜Εσ…$lΪ±Ϋ”ΔTβˆh―m΅¨JH?©ι\qΡ’πd5\ΝΧ2φTJΥ…Ψ΄" Ο{ΣζΝζΪ΅γΨ°αšώX»PΑ€ƒ‰ςND•UyΈodΏ_εͺ­‹e :°Tvs":ŒŸ‹Μ/¬ά―ςσڏΚΑ²λ»’–ή{ο=ΔΗΗ!»}βΉΏ) ‚§ τ\₯΅³Ϋf0*Κ|·¦$η)ε1ΰ­¬b±Z4B”Ρ?ΫΚ~ΟΎwι£ύNFF†ΙšΤkύKII1_Υσ]»šρρρΖ¨ΈτWzΙΓF‘|4lΨπJ_©φο[E¦ ξΐƒλ8Œο…DŒιŽθΔ<φΏ*7ώΩ•H䛬ξπP–#† Β—_}‰ψώχαv»K!Υ^Wθ •>VTν€aΪμωhšηFf²±Μ‡Ξό*Ϋ΄`I‘†³‰Œ6œ°ν ϊΎτŠqί{ί}HOO7ςΠ½μΌΤσšΦ¬LΨτ.!ω'›Œ<δwχ°”‡Ώ4‡rΰ[‰’ιψΙjnœΆ»ζ@Ώ~ύπΟόΟhΩ²₯Ή§₯/πΤ^18°tι2̝=ιtμψϋ‘ZΚ€Όƒ—ߍˆ":M=€zλ0;w:tθΐrkΌήΎΗΛΞKefκ0vɚυθΘΓΣ›Ji*κά·z;†ΆL'BΖrc1qϋόυ™gπ“·άr+ _–»Π¦€J”εzFο• υyσηaκ€i(l™‹4–Σ«ͺ¦ΝŸΠim?ΦΜl;wί}O­lͺŠαap];7•έͺjbb2bΌDŠ!δzΰƒ /ΗικρŒ¦H=KΝ‘S# ?¦ώHJJͺΥd™•‘Έ'η€tχβUkΡΆΨkμp3λΟk9,οy‰τ3ι~O"jκD >Ϊ‡Ω+Vγˆτψ㏣]»vFG”₯γ’KΤΈ—ZΟ΄Ž* |χφ=(μ’_Λ4“›U΅mε₯ε0r1π)»c† g/2?ΩΟ‘““σ­±Rγ„RC:lηα;I)t •†eσ‚£³―ΖhΏ]™ZΎMDΪ‹Ž;?—₯ϋjΏŸ•νηFιλwήΑ"fv“‰^τ›dRΟ}]…•N—ͺ“ξΒέ ξ›‰‘#?c0ΝόΓ~ˆό|%ͺ}{m1oTΣ?Χ’Λ”©S0kΚ,ΆΞ#dŠY3Kς‘θ»ΡιŸ‹¬v8uπ ’…2Y-;………΅ϊ<2qΰ=ΛκωώfϞΓŸa׎-DXͺ O~/ϊ[Š―5€3?ΰ LΑdŠ&3#"™P›Ηΐ 7άIΕ8wd%F}:„Csqǝw’{χξP%5ρG­ΦΟ`Ψ”?vLnζ‘ώθ1cQŸe[³μ‰c+κKͺJ‚52tΆp]f 6ubτΨqhΟ=Νu¬(aϋP•χ―½v-φνΫΟOkΐO±i~m|έ~½U>ΙΆŽ¦>τ"!ΩƒuD€οέ«RSSkΗqωψ­oΩΉ/ UΫX±a3ξhšΕJ¬μ@VSeίϊΙ•e€σΟ¬€tΝ‹Ηΰg°ώrsσŒ}m―­›ΦΎ(7ΚςOIϋͺ4tόΠqΤ퓇t’α—,ωίΎε‘^Ÿ>ξS^l›±ΧόζΙ'ŸDύϊυΝχΚ^σΫ? οWULS–%Φ°Υμλ²ίΉΪsϋύΜΜLόΰ?(ύͺalS0ΚJ•6M>νώφV‰—Σ^―τ>±οΩΗ²ŸUηηvPΛ!τΡG0aά(dζ7aΝTΥxΝfΧε ¨κ&£]%Ÿh3€'3"†%FβKVόθG?’BΜ«±‹”•2–/[fZΦlΩJ‡= bα£C(4Vš6’xηΚu,O§τ€Qψ0φΔC?ŒββbσuK›ύmMy΄ύͺ•`Šu«@šΌn €ρ2&:νΣ£cPΓ:ίςD휷―4 ¦m[f³—8­jš>«)cπrύ΄γςπα#OΘωθΨxD§WΔΉ―“€ΞΪyΙzΉ›κϋŒ‘ά˜ΔyšcοΪ‘˜6m*κΤ)F\ηΧόΪ1θgœε… ’“ψ.ϋϋŽΓ υάθ–‡τxmXιΰŠ6ύFzΫ Τδ0 Fz{[$^{γ-R³X ©7’R"YSD£αάJθg"XͺΚ”~Š'ςQ,ΟY7ϋΏŒΉ ͺΉγN{·5Κκ­εW8χ<h·|ώςΛ/1eςdμάίo“O„Λς• f5’΄j―Τ6;φΗ$–d©ί A­~ ¦ jΰ½μΊ½nέZœ=ϋ’ςrθζv‘ώ½dΕYBŸ₯žR=u°rΕRΰ٘‚ij[Ε8`υ—§MŠŸA«‚L΄Λ"ς}ΨBωυ―§ΊYX―xƒ ΌFCΰt)HΑ„ Ÿ›ΐœΗim2Ψ9φέXωθ›B€yνυΧ°‡IKE]sΰiB0E:Y•½§Ud4Ρ{‰ΐώΝ—°iΞfΌϊΪ«Ζ–nܘ•ͺιωd•ΣX[…f__ΛcΩύή [ο_ϊΩ•os₯ΟkΒϋ–’‰ΠNή}χ=“κ+lΕ@šN̈υ–Ω‰#4Ψ£2‘έ^Φό^8„–σӟ>ŽF",έA’(d·±ύ”|T²eόψρ8h0Ρ‹Έ£±rb‘Β` §ZbΩt=«λ»]ζ>3™…9yΦ|Ξ±ςC΄uκάΉF°XΉ¨ήλoΎ%σ—’ EŽΏ΄³šͺ<Ζ šB’Ο6Šc »MΊ©YΈ{ώA<χάsxβ‰' RWu]°, j/ΟA.§“Η[D=žPηΎ‚hΎα>G₯Iη‰œξεΩlH E!‚YKžœΖ,/5ΡΆΆjΥκςΦ°w­ΞΠ|<|ψ0©GγPΔΈbpg£ ΄ΝbΠ"‘¦*«ΛΛκmB¦Q7LٍΓGr}pέuΧU[CΆ’ΓΚΚε«―ΎΒ„ρ°eέV4Ό±)yU 9OA] °“|„ΙlνΖϊρ[Νϊr¨:θ²tU΄?΅ί?XY«<νΗΔη›Χ"£¨+ΦάpBΙΦ FίdTΪ=ΡDΕΙ8wš> ‹AxJVώͺiΝΚG}WiŽΟFΒΠα#ΰŠuαž&΄¦Σ'‘Ί;4–·β²Φ‚DWΊΔ›ΐš©;Ξΰω—^†²Oο»~f]嚯[ϊμo«ϋcΩώ ΕιΥW_ΕޝϋPά5Χ(H1h± fΊ¨lŸΛ؜ ˜7k>Ξ~yOόό‰Ϊ¬Έj<ν8ά½{·)ƒœκΞαaj#šΝ1F‡~W~­Μδ*’ΪΩ;0iβ8’Σt0W–ώk½ͺ“gϋ&Δι τ›Όα‡άžDγΖS UϊΊ²ΆvEϋouzƒ΄oey©΄8flފ§žzΚ$ήpΓ ˆ‰Q&nυL>°ύR€πυΩ―QΤ3‹εˆ˜la\Mx\wύεžRqf6Əž9oκsS΄Ζͺšπ™·²”³zυjŒ7σηΝDt΄ YuΊ Š KB) Ξ―ϋΛ€ΊΩΚκχ"μ›J?EzΫ Š5qiλ±sηJ<χ̟ΡΓυΈνΆΫ Ώ Ψ΄ό« c%”}\·nΖOž‚ω^4rΗΘ5IΆN…Bh™ŸIΣf’gžhΓΠΪVˁͺΰ€ΥAgh*ω"*šAς‹›€Q"oUψ³JvZ4“U³qdο*lή΄ ˆΊQ}$Ω²₯Kρž½ι虇Œ’ΔŠˆFž2p:ηΖβπYF€Lo&όήχΈV39²šΪցΰεaω₯„„,(dΟ3'Ύ@q\’ςΜ‘ϋ¨+Ί'KΜωcrˆτ¨yρω‚­xζ&…1ΐ©S'žM1fΓήγrχΗχ‚LHζX!XεuςδISŠFQγgϞ5τYBB\.TbJΘ6ii„€–₯Αf―HΊΒαZeϋ-Ψ¦7ήx“Nόup΄§u^ όΩ΅,2•ν½)9σΣXbκz€EΖaγ†% x?ϋΩ?‘¦ n”•FϊlΜ€©Θσ€ž8Ν½.Δ°”GUϊ ό3ΔogΎώ–βHΈΉaœ±ύž‘<|`»)’2\jeiμ(pβοm€;ΆlΩΒ9σΦ¬X‹‚ΦΩΘνμζ¦’u«R —cJΙ‚ΕLά,BΫGΊ"°kΦAΌπό P¦‚t΅FΔεWύή³γσΠ‘C,΅4 Iι&‹ΘΔP§~χ†ϊδ«ψϊθj|sfΩ(Δ€7CtJ!ΧKκ^­IoΖΕ$ιqβΠη˜:m7i‚Έ{5ΞXΉθQ!¬;cώB4ΘΞΐ <(mΘ`E‘€JuX½-šŠYώο.κ73Ιζ―ί„]ω !²˜Π d©YϊΜ‹ϊGχΓXg7§Q¦‰ΆΧ&!Ί<‚kΈ ξswda„ΙhAΈcAf[ϋ΄†Š£ΖtΫΞ=s9³fL†· K-΅ΰ᳦Κ₯·Ν.Ώ.JΜEŒ―3’ΏšdΠr2hη5₯>·4ϊNΌžν«₯»‡‚σ’~v:zδΗ£±Ηeΐ₯»Λςΐ˜v<ΈT{k[f°vψTF3;vμΐÏ7š€θJ]ΆύΨΪ…JzσΝ7qŽAw&£ˆΆ—ͺmHW”Ϊ‡%KΟΥΨ£―θwBυU@Ξ‘³i0prrsΠƒ{ΥŞΎρ™•‹ψuμΨ1Sε`πΐ!ΫΡ(ξ™ o3Σ^-ΖAΘTˆb`TΡ{•ψΏkώ<χμsxμͽo„φ‘jφžφ§αϊvΑ4e/hτΩ³gcΡ’EXΈp!–,YrY9¨ήΌ"¦υ―[·nh@h9눬I“ΜςN„9sζ%vϊδfΓvA΄»k½&qd—ηπυ²lΐ›š ¨αbν눴˜xF~.ΐ³Ο>‹όΗ4‡MΥUn’šΖ£Xζ>z ³•·μήƒE^t!tξ³LŸΡ)šΕWj9 ψ η.ΒωγkΈGΰxωςΎ:{€ΟoAtr~9Φ^ηb„ΙVJυ5ΒΜιSΠ­k7΄oίFŽ=;ί$‘NLgpΡ Ζ=u†Pσ>t"²Xv²ΏDŸ"κ―&‘+Iξ»ή—ήφQμζβ(δ&Fa:χΟ½τ*Φ²Ζο=,έ°aC³ΦXZΏλzΥιsΫgm>cΝ{β"“ΡφρΑ ŠΤF#–λΉ―u:Žν:α#†£a£†ΘςeΥΘ9SΖΧwυŎ?=Nœ8 >z™Ή εν`lίΠ–ς‹,3U‰9p|ΗtΌσφ;ψΝo~œœœj?6­l€»eC͘1Γ B:~έλgšΜ'! hε¬J;\k‚]κ€Η„Θ<ΦŸΉy ώίˆο‘ζζ[nΗγ1B³t[ V§Η²}Σ^5B;|‚ˆ49π΅"’R(i,ƒeƒs H—+°>2 XΖ€šΟ½hjκΤ©SνηŒeEMx΄cQNΰџDΊ―Ύ)ρa‘ ͺĚ« g/²όTΎ9ž3{:}(=θλZ™ :ξ·V&"LΑΡBT[Έr ZxΈ‡–XR³ϊΤΌΑιt­±Q‘θΐ Zw|$¦νrџς9°δΡGΝ!΄H+Ϋ§°[Ϊ½ιάΉs0|θpδ4τQg3a. K©V†zιs””‚D°άΣ„ν2d(ώε_ώιιιΥF.•aQuύ­ζœš³{φμaΐδLŒ3šo{žΥήό^ ’a’QL’ ”:4»kB+ι'ϋMŸO‘A²w§ΦΓ9–~;fζΞ™EΫτvτΌ‘'Š •a捸cy[8U•}΄λƒΛ'Ϙ‰Vyιβ‹šF±SG£θ4«Η=Mλά4L™9έΊw£M]€ΧΆZT vξά…έ;7 ΈΥ£LMπλξJΝ’‹ &t#>9›ˆ±›°gΟ^ΦΪ吞Υ]ϋφν£oιmžqξΕm έ&ΰ½?ΏβW€[d_ηq/~cQFlΊ€7 ‘–š†–<ϋ·χ½βjπem=_Λ’hŸ|ς fOƒœϊY& &­NΟ„Θεςn€$~W₯·ε§qρΜcοΒ#θΚΚΨ€ϋξ»[ηb8ΫaLc'ƒ’q•eστΣO› E΅ ZP™†² "ψŸJ)θ7Š,ίΈq£ )..ΖΏϋΏγΑ„5μ5«ϋ<²ύ\₯κgχύUWž”ίΡΝ9IœQΜHC=’plž6HŠJΐΩύsΡφ Žςπλφj `e£1¨ œΰδ?ωt$RS’qg–uΚVΉ₯(³Pp­¨” PΡqnτ!oͺ¬.•ρ$Db.ϊ V¬Δϊ ρύο?Š^½zWK΄+=ͺφΊ²aO::έHΓςJΡρp•,X”κF33a׌CxύΥώΤυύŠ‰Σg£ 34υ %3rβUΦΥ*Σε’ˆξΗ{΅ΛŽe`M$fοva"³©WZο?φ(Ί2°Έ&’ΤXƒ]ώΉΣη‘^§"€%rœJx’`ο£ρ’’Ÿ€œV^¬™Ύ3yp½ο=ΦŠ pΪάb©ΰmί{noQ!;0‹CθηΩΏ` Β+±’Ž" QΜΎM9wkVΝŐ‘CρψO~bτEuΥηeϋ₯ΓςaΤέγ&OCuχύM½hεs!ΛΆ¬SUKΙ^_zάΛ@φήE hOtcζ³ζV‘1 ½9ΛχYΨ\IΤκ·+I7œή/+%ΜΌψβ‹ψβΤ(μše4‘\ίJ’₯CΧ-ιr ΙPΓΰ&«ζ―6ϋ…_>ωKΧ(tB¨ϊ;Ϋ±( ²=rΩM:²ΜZKU«žŽςίAσ‚9ψ¦.wήDίΟD–ϋlnPšmΚ=η}ΣφA>ΑΣ§γ½χίΗΡ3_ wƒ,tΛ—Ώ"η•²Κfυi({!δ‹Ž€Žhμe9VξΧΣb³0kαbl'κΨγ?Ž:T‹d?+›νΫ·cΐΗΣίΘ$‚φˆw³œ©5'δrƒA€δΘdδξυaΦΤYhBdΎx Z­­—λzM}ώUυΔ‰XΘ έaΏΘο^Όπ5{ͺ’NšιϊgίγΣr[Oά`°mzvsΜ™==YRHυ1kB³ϊB}=pΰ€qμρ)ΡhN£gΓl΄g@d^r θΤ£AεR 2SeŸT‹VswΑ3ύ^Β²eΛpϋw yσ戊ŠϊΦzT]efε€λOXή)³ΠƒΜ©&±S&0μα8PyAίξΕ§  mήό:cΧZ:s£Ϊ«8…V*ωΞ;ο’¬ pe²†pbΥξ7|mσPRl4ˊΗ"Κέή―Ο`όΨQπzΌ φzΠΐΫ~„’Κ@έΫφEkιΡ£G1kΦ, g™έ‡Ž sύ"‰Ή ΊVΗ`ιξ²}“4δ$ββ!t‘Ζ³΄θά5k±αΒ}χά…ή hΟΚ&›νyΖl?τ(ΪΧ^{ _Ÿϋ…έxhsα†ι 1'Σ™2r6Ε0θ)§c†‘mΣΒΝxώ…ηρΔϟ(υUΨ>…±Xj<ι›6mΒ΄)“‘Ր(Eη' ›H–ξKρ5ΗΕσ±`Αά|σΝΖοΰpβ―HžGΩJ@5j$†³¬SfZ2ξoμfβ?πρkΞΙPωMDΌUW ςΙ’?εφΊ Θεγτν'πηΏ<‡ΉΖήzλ­₯Ι~ΊŽέη]ιšN{ίΚG =ς9ξώ|τ.D*_.ΪM‰ΣˆΆτhΚ”θr‘Ÿ=ϊσpA(c*7oϋfΏ^ϋΎ°²ΤόX]Ύ|…)=³pΑ,$§ωΥΰFϊΘλ3`R‡°΄ώΪYΎ} εβ7²\c"’“ˆpίΒ ‘ΉΆΰτ‘΅θϋό_0«]gάB}ΦΆM›jWΪ!0<ΌΆ«¬c0ύ΄ΩsΠ6?…©τί\ΫeBς« ΄ισ“£Πšˆ:3ζ-`i°5ΈώϊλCBKνM«7Ž=B½Ύ‰©Ή~=Gg[d ’@HΕΗΔΙMfόκ|Ω)Υ›«οεΛ™3g¬1ŸM˜„Žuόθ‘Iτƒκ8MzPΎš–Lώ?y.6νΕ{οΎƒ'~ρΟ΅₯—ΛˆΝΚCΆΞŸ—―XŽ1£Η`ΡάΕπpMixk2κ³L0}+₯Yb*ΙξOΞJ€λ†$ηΕcί²#xώιη±°ηά~ϋζόXσGΝV†\Η? ›` \ V°iZΆlijΩξwΏ3ε”9 €Ρ”mϊ’­δYΎ|93AίCύzυcNh6Ώύνo«M–NΩ~ΫηvPd]ζ`βžμΊpωΊ’‘‘šδs<’–j=jŠ6–aΐ‹ΠlΡσ0bΨ :t?a–lnnΎ–ΝΚEΔ+Ψk4ΛM ώ)’pgΣl’ Έΰαα§όάͺ8Β!$‰hΑ‹g°F‡ά8δΠž“μΒΤ9σ±šΑi=φ˜™‹α^ ΟΚFεΆ&Nœˆ7ϊΏΚ₯θΦ dt₯Y©P$=η΅ΤΒD‚‹°xΙ¬HŒψδSœ–,[ŽGϊΊχθŸΟgˆ°}ͺŠBwUΫ/ιυ '`χΆ=hv{=$η¨ι²Μ ‰fΚ\D‚7ΩmάX5b#Ǝ‹BBc'& )GAΑ&*X―yχ±ςΤ~䣏`ϋΦ πΤΉQi(g;ΒΖ²|€§2Ύ:NΘ36»vνRm„geς kv IlΨ°O0qΪ,δϋ2pΡhΌ’Bέύ ηb°uwY&—ΥΉ„ΛΝ`‰ά$ζξ9‹Wί~ί ₯Fϋβ˜lΆoe―.ΟΛ@šΎ}ϋμŽDA·LΦνN3ΘŽ €±L5ϊ¦τ«j„n°uφ6τ{±ώυWŠΖ8Ο-ρ΅ίΕ­ΕBφUι·“'!³αυˆpq?ηόwέ½2ŸΣŽ ΪdtZ#Djδ qeηYeξΜί–₯Ye> ΝΒ«Πͺ( =ˆFS/¨œœ‹Ζ9_Vq“ΘrάK€‰ΖΩεyqpΗE`ϊ/ѝχ°•Αήα\φΙΪ­έγFC^³lxš₯pkͺ²Β촃εbDGϊ4­“²β ρΖ1;i 2Ι—Ϊ'•ƒεuνWΖ+?S‘Θ ‰pάΈρ˜8aŒ±Ÿ2 Ϋ™±"yX‘ڍ ŽΧσ‘b²Μ~§hdΌ‡%P2T€ΈΤυ<{YƒΛ—°μӍΈγŽ;XάΨ$οˆZ+‡PQŽχ՘•cΞάΉτ)]0ˆρ<Œjžh΄θ\#‘θΝ=1XΉ‡eΘιΏlΕR,BQ―m΅$Άnέ†£‡Β]ΤΘ_β)Pagr©,-³.V¬X†#Gξ λ3Ι@ςόkY=―δτ±cΗ` K~6-ΘDo–aΝ(A΄«Θ₯Ώ­θk£_ψ£X&ΫtΞ‰Ε™s阺x9ΉOψΩΟ^€nmԊ^?άΏoe‘ώΛο΅uλVLš< £Gφ‰ŽFŽωπ2Y));ήψ%i―+ FΎY­Σ‘HίΪ‘U‰XΌ`)–,ZŠ;οΎ}ϊτ1>F+Kk8π<,‚i,CUςώηL–—"˜>ϋμ3ΦZμΦ]­Ιi ‚OUΆ|ώύq~Μyγ S4μ}χέηόΝέΥ:x™Ο,Ο4(΅Qx—ADΛ—.„― ’)Ÿΐƒ3-%+—Ή„ή²κυ’“rικ‰LΦ*kΰψρ ¨ω1š6mjθ΄ύuΡW%AtͺI.BNšOD Α‰ήό9šηΉΝag}BaΉθˆ΅Α€– W½p>΄ U»Q@₯xG=’€F§ώ3}_Δr’άΛω€ΰ6 =ocH¬Τmμ8R–Šχ>π1ά<ϽލŒΊΙ~=αΰ@u^‹_’΅β{z‰Έ…ΩΣζΰ4!AϊΣ¬…œ―Τθp֏νΌZΖ Ρλ—#«~oFή •&ΠΡ@ι€ΚCš― ζΟ›…nΈνΪ΅uγ@MY=ηB:™Θ“cΖO4ΘP75Ι%τΛ y‰{‘ΐ Φ‡]DT770Zgζρ0Άo7 rί½χή‹6ΜΌT ‡šΥ…ζE5ϊ³zυjL;ΉM²ΞΘ{.Ζ~οNE†y ψΑuT¨³iuQΠ"ΣXV¦}ϋφ&(5P·¨½Ž38 }-gΒθΡ£±`ήLx Ϊ R%V•ΰHG='ᐣ²γλˆψsSˆ¦σQξnϋά}θΧ―/q-Z„‘$jI΅°crΓ†,Η8τ†LFͺc?vψ£%έγ}HΙl†Υ«ΠωΏ½{χv8έOž•ΓιΣ§™@0q~š:ϋ–¦ΉDSY'’ά,α υD£μr©΄fΜ€u“ώΩDšΙ²O;Xφιϋί>:=œ2ž­Œ”ΐ¨”Tw ²Ϊ₯'z σ?:ŽΖQ:ύ>ym|X1{%Λ€MΒ#ωxm„9΄nFΧ.] ŠhέjzξδΑ4b’"¨x°~SOψ•W_AΟ= oΏ‹Ωφs•2Π!ΏΚ;IρmΫΆΝlŽnΉεk›“VΪΚφCε¬^zιeμέ³ξFΫ3Λ4*VΩTvAqφυχΧΜ@–/aΔ{V7dΔ$cσΖωxφΩgMΙ.>ΩIηδ WV.»vνΒhƒ3BHΊ³i–9|M—£ξ(Ρ7εwΠΏcGŒhTD|G:±tπ0—(53,ΖΪuλπΰƒΊ±OXΑzZω¨ν@Φs>d82‹ΌΘοζA α‰Kš-‚ΞωrήPΣ…>-Tωέ=fa\Ά`^8ύ~ωδ/MνΘr^©φk倫‡ΖΔργ ILΗEJ}ze™•^ͺΫƒMΌΰ &&½Žξ[mΚ34kΦΤΌZšƒMU οWΆB™Pmp•θΩΊχΛMGBͺ<Ÿ"γi: YLΌ°z»ƒ} R’°,#shXᩧqSλ»Ρ Aƒ°XSΛ+[+3ŒŸ0ηΏ>ΟR!iˆ₯œΎa(/Qϊž¦ M1WRŒΩΘΪr cΉαT魌ŒŒ·α«λΒρ»sζΜΑΑ‘ξ«C[Ό-ΥuRυuy8(cη"Žλ"ήw ·NΒ€?&Ζ―Β6ΓΗκ9w–,^Œ‘C‡bρͺ΅¨γKΗm RΠΔƒDfΠKO:1fC>’fšΚή\…z hŸΉϋ,>`ΖΧR΄?πΐ¦΅M4±}.ω™cΚ9•‡ε/sοͺύQAΟLΈ”>Šz K§6Ι‡G)"GΠΨ6cϊ2 ζΧφk£ΫJz-]Ο;&…<8kφ,sθšΣ¬;QiR»Λφ…Ž(–νcΙWμ*€&Ώ‰PdmΚ~ΫiΟ-ςο‰s Χ QcΗ#Η›g ©Wφφߠ❬.ε­₯U{ισΫXF;'Ι‹)ΫOβ/Ο<‹‡Έχ0Π===έρ²²rlό(μΩΎo©KτΗ£­·/Œ^—θq_ΛTœΪ—Idμ&ΈφΠχrΜrφ{vlJ‡(YnήΌyά·ŠMV!=³ήFw!*©˜z]69'c©ΕΞPgχΟ9Τ•π‹ό‹ˆt!*₯‘ δτ€ΤΓW‡—γ“A@eΙοΉη^ε2ά‘ΜƒΕw;~e‡,\Έ{Ýυ ²™“Ο Δν­΄Οjž‡9[™ω(ΏG8Ž^©o΅ο‡–vΘ?«`šΔυz‰ °.'j™€ d»)xXe | ©X~̜9oΎυ6<ι)Έ₯8Eτ½Šo–Hiu]ιΕTξΓo bεΩ―½N”D­7 ˆŽ#‡'₯?ͺ†Ol­T eΡU'Mœ„][w#³ŽM﬋΄ΊI £?\Ω₯*„SrMέ#Š²ρ4I1eŸ2θ{Ϋ·τ0ήyν]"•MGŸ›ϊ˜ š¬¬¬R©ΨΎ”Ύα 'Ž¦±Μ;rδΣρ©φ/ς/θά©³yn?7/πΗ"}W-''ΗL&!O¨τΣφνΫMΝ~Vo[~θqώόxγώ8~μ0ς»šz¦ͺkϊ·MB8u3ΖΉ"x‘‹{η3γ―g€‘Š’ QeϋﴞYΊ„F£q,ώ²΅λΡΊ0Yz£^ MUκ°@σD:ΡΏF /…N –*δγμ]§ρλobλHή}ΟέhΨ°‘γQj¬|Ž=Š?€O‡BΡ%„H“Δ2 ₯§fb^OΎ€Ψ9CΘωωΫπ"!ηφ³Ÿ›ϊ„UxλΪK‰«V­fΉΓ•πuf°a&Η©ŽižCtšΖ˜7wnf¦Ό }Γ½IG¨ΙžΜNΖebκœyΘu§βώζYhN§Ύ›εωτU§κς²zΫ— £¨·£°hWCd”5kΧα;ξ@W©^/‘ŽΨ¬~4/Βπ΅ε„š7kršϋVμGΰ ywdΪpΐ€δ' ·U&–Ξ^fœe ςΆt‡œΖZ™vξl!zΘ‡~Wλg2ΞCΑΫ φkΎ|~(t“H–£j„τάƒX΄`Əo„‡~(¬2΄­4§Μ>iDŒbθ―ΏΉ€eAΑ…:ΜTϊ ΏVeŽž@ MkL a {]π&F1ˆΣ…Ή;χγ)ΒήuΫ-Έυ–[Q·^=£Gluο@_Η§lB•ιz:ΦΞNξ† \0 οͺG§·Z£xΐŸΩ"Νl―Ός ž|ςIcΫώ:½+΅τω9°sηNΪ³³‘žΕύ,‘ Β«q@ΖΠ‘fJfS,]βG§‘ίΔιΝΞ= ZϋσE+V£EQ&Ί³C΄Λ…{šηͺέ… Ύja GgΥΉ’](5νιΚ#τόœ=q˜@X΅•t~=φΨ£θΥ«·cΡ)¬ŒŽ?ΞΓπΩπ1(Ό.yέάHΜdτj0VύΚ ²+ύΦL—‹D=ˆFN'7’(Ÿm³wβ…Ύ/ΰ7ΏώΩΐΪΎ_ι΅ο;VfgΜα‘ϋ1ˆf֏•ζ< Ά³2΄kΠΡiΟ GWjΨ·‹/BγΖάΜθt2L›εΉΘ:žAγAμ9Ϊ έλϋˆΞ‡’s±ͺC©wro-mΫ.Ζ6ρΈ¨·£Ρˆ›œι;NαωW^7χέ?d+YG—ϊnr΄²S δ„‰Μfiˆf©«‹Ά–’:Βf‘ ܌?Έ>Ε8{ΕwŸΟWm7{!dwΠnmǞ2΅G…½»·Β[οFfΐ’NΎκκς²AΪ‚ΘD‹r·AΖΩ#:d Ρ«κ›=”νcy―ŠοY…F³`Α| 4ΛΦobiUΊεΗ›’wqD‘>4R± 2ΔVΰž"SτΚoβe‰ž…ρφ}Ξξ/ρΙg㈼³Τ=υθΩ³΄²εEn”―ΪuE΄}ϋφ5₯ ΊϋxΛ@5ωAΓD.†^*½.»[5ΒUžκσi»ΝΎό7Ώω9wͺ, ύ΅όb,±[—ριΰώέπ5Ό½•&ά€`ιγ‰Ωv-1™ΰ²1’θXvκ8΄t ™ZΩ”o½υ<ƒ> 3q=} i%Ύ“p“Δ•¦–†šΦ H>iΒIOB$fΠN:wIϊϋ§ŸύΜ¬Ή•–7WΊV°ί·τυ[‰Šqρ±π΅Ry'— H +έ]–yγ|­ΰϋΌ>,žΗΉ3wξΌσΞ°Ϋ •νVMxnΗ€ζΛΙ“§0iς$’ σ—tΚnΖΣV ŒΜαΪΜ£sΠZ]4‰“€+£ΝΟΧΘ87"cΫ#9©qGV°ψL,_ΆχήwδΏ«š•›yQϋΗpΐΪΗ«ιSί±w?Ο²‘^²ώ‰ΓαΦΜ¨ bMcIΤΖDQΈtΉΦš`ΫΧpλS-½Ξβΐž={qδΠ^εuT¨ μl‰ ―;!5;Άo3OŠ‹‹kυΉluΈ‚W_c©ζgΏΔ= RqνΪ`6I[6u.ύΫ7Ρ?rζλ ΌχΑ‡¦\ω 7τͺv6œε»Υ‘BςTrδΙ“±iνf&ƒ€‘Ω­υ‘^Ώ$ˆ†‡¦€S¨ςϊ( ‹„“θ‹3λιυqdSv.ί‰ΎOχCΣVMΠ§OƒΆμρ(θΩίl_νλP>–?%”Tςήr‚*²J-1±ςΕΊF=fμ©ΙΡξΝ*9y&g€–€|d4ηfA4Υa£`L0*@Β«₯5A|nO$$%γυΧ_Γh’|ύυΧF9Š‘n₯2Ήπ‰ΐφΉgραΠa(τ¦ΰα¦ιθQgi€θΥ»Δϋ―¬ΏvΑύς…RS7^§ƒΐoπό‹/£ΧM p«ΰ ΡjetβΔ ΌώϋύιX΅ΜG~iβόJ^_ Χ¦…ŠΣ>š_›twΝΑ‘=‡ρβK/bύϊυŽ™+αΚήP½cΗ,Z8iY A'…ίaJŠμ½©Π輊JΚγ&#Λ”P†€šSζ½₯τ»E―Υ²δΠσŸώ„†|‚œ΄<<·ΧM`Ή€h£Ώ₯ΛΓI‹V-?’;…™° †|”&½gcj~‡?βκEeΖJwλ_ΈΙΠΚxεͺ•X±h%²›{‘˜Mέξ‡U³‡τ‘lΕ !ΡΛˆfΜ€]½‘‰Ρ™!%°φζΧΔ+;ιΙΗ!#§9"Ӛς΄L…šyαdH|Γ,,/\ήΆ†θ™ΐB'λ««D£2:ί{χ]ό‰εμvμٍ;›εΰfΝ7£sΗE‡‚βκΤΒIϋ)φΣ,`Ξ1Έ§aξ». gΏό}‘/^ν5|ώωηώώ9P‡[9 ν­ώrδ-ΕρΈ£± ώΧΏ>ƒ1cΖdL§ιKΟΜY3±aΥFd·πΎ]εΨΫ©ΏοnxΌ# !ιέMSX’3 Ÿ±D»ͺΤ¬ŽŽΤ*­ώΠγƍΔϊϊΏϊ"ΞρΠΘSΈr{³|iu uy-MŒ`ψΑ¦\Έrz"£ξ- 6Nΐ{οτΗ ΄S…<¦$i«Kͺ˜¨°ΉΌΥ1Z UΖ5).Ζ”’-6½ψ{BE;cΜ‰Tƒμ΄D,Y²gΨΗΪVˁΚp@ϊCθαλΧ―Cr:ƒ%] “ δlαυtΈΒRvρ©ωΨΎm“ ¦© έΥε·vνU`υϋｇ‘wq Zϊˆ€%Ι˞ŠεΝEρ ζŒη™q£oΔs«_Γ™Άβ―ΪΎ}{ϊ‡?ό―τ}GŽF“λ’Ι½EΘnŸA_3ƒΛ4„Ε5ΟόΟƒύ·δή’EζXbV<ς:ΊΡδΎ"4κU{φ±Lφ_ϋαO<{7~œ)υkHΦX*9£ 6ɗޏαΨαΡd`ΙωT§N“%]Yͺ£’’J£ €r­ν|­Χ©μομ’Σ^›—+ρ9έιΈηA«ΉΈΌ‘œ-•νaΩί«~ƒ;Šύ‹e PΔ~Β‚χΝτρφΫo yY! ΙDˆJ“&NΔGΔiBΐέ8²cΰ‘“„_1 Ύ:IER‘DΗXΎ*7) ³w»πιψID–Ϊ‚όΰP½i!JYι7‘hφώ’ΡΰΑƒ1fδΨD)o£%CA[@οΙ¦Ύ(CV5ʐέ:“²―ΌŒϋΧ3A…–½o˜\L}W³:Τιd‹NΡΌrεJœ>uIy…5ΐ(Gθx 8κgW*’ΌM°nΝDB`nCf&ΛP…Q+;'T Rh4£Η:6wρ Ά΅O%ό&TΈ”Ήϋ΅Ι¦Φ£ƒ2‰rPΉ{Ξβ}ΒΜ/c9Μ|;v2¨€eys₯k:α}K§œP3¦Ο@l‘ΰ$›ΰBΗιwς_‡ ’Ό* Σ§OG§Nΰv»CΎV:A–W£ArVs’·cOΊC™Ϊ )ˆφ΄‘ο…%Nε«υΙ‘ŸiŒ&#%»%֞Ε1:=τ=GρέςΝς_ϋ;ΑΫΚΖ[²zοJ4š²₯UυιΎpn–~ΩίΚφμΚ(*»:—(‘ŸMœl²>{τQt&²« ¨-BέgK‡‚}^cΠΟΙc'QΤ=›΅΄S5‘mRs¨ΙΌφϋK0ά~„ΪίBC[;gή~ηmόβ‰_˜RŠ–Χ~“πώ₯ϊ―ζ$ύ]–£ ςZ·f9λΛwCDlš±oΓOcp Κ.J ’eΞξ_ ‘ν4kΦΜq|·σAh‚ƒΔ°Q£QμΛ T{<Έcο~”z¨M]ΣΎ"‰Aξέσ•x‰IΫΞΰ₯W_cιφcΈ‡0υJ ΄Ό %, 07v<DKfωϊyμΖ"”Vφήfκ\DRHΫz±jΤ̘9<όH@όΡ•%/ΤΏwšώΆγQͺΣιΐςͺG€―Λ«¦_‡¨Υ· ’ΡL«ΖŠ$Τƒγ[χ/α3χ?BI‹ΞΰΪ“…¬δUX±|6mΪ€Gy7ήx£ctΫ·ΘΡ [.EIMKV¬D³μT’–±œx@ƒ‚ί9™~^"μ4ΜLƚ ±aγ΄aY;‡ƒOUΝ»£Στwe$`Ǎt–Ν›θσ#–’MΝ#Πzžƒ7Κe’ŒΞž=evμύ+Σ‡pώ­νΙ“'ρρ€X°|%z7Θ$z{,’yφΚ“hέ[(X_ΐΔϊ£xηwŒ΅ΈΈ8lυΝ₯sWΘωͺΈ3i$¬[±ι9©hz“‰&εJεcV€»’7Ωθ)Q™ΑkΝϊH"˜–œ›`@ δ³?Άω4vفgτkΫ7έt“Aι΄H5—ς‘2d\ΛoiM†OκˆΚ;•–vͺΔ ΣF₯ Τ¨s-Ν* §8€ύϊλ―³œ qΩέHΣ d‹ GU%˜u-Μ©ςί¨?œpόZ¬.HJI' π›˜9sfH£‚ΧRTfΏW^E|L3ͺ<θΝΊ}™ €±•ƒͺ›T$vυΙ.˜ux {‘nk”‰„ά{κ©Ώ0Zr”AƒΌ± PΏ f³2:wξœ)Ώ0dΠδ5ΚFξυΥ+Ζ2΅D(Ρ ¨Ιl™†ΒŽ9ΨΆq»1&4NC) Kb(ν8pŠ.Ψω"CuΡ’…H$Όddb>&KΟ;₯‘–ΘX:­˜—d3…μ.γ̎ ι‡Ω,™ψ—§žΒP:τf₯βαfθYΗ² ˆ$»ΝκZ Ήν‚lμX²ΧeΊπ@ƒD€fa8O=ύ |τ‘ΙΎ°σŎG§ŒΊ+Ρ±vέZ,YΈΎΖ:ΔYΊOkΫα+ύ(Ψο“Ρ₯@Ξ¬f,™·ΤdT›ŒpΌŸΖ£“N‘_τQsκΤiΨ΅c3Ό-EHy“ΝδΈΑWiΒθ0–zecΣ<,G6ΊtŒ:Iˆρ_λδΘ‘#ρgڝ›·nΓ-M²p_Γ$SΦ.†Ž'ϊΚ#‰«}GRRΏT*€.νο;iίΥΔ‘/>Ed9Έ>숡ΨΚI‡ζqmω|γVδuΘBζui˜RΧ”―H‡­2cq‘Ω Ό©xλ½χ1`ΐGF§;Awˆ­Ξ›?Οψ²sΣήaBRu@₯±§t!ϊμz>L7‘ΩΙ)sΗ’ΜG;_’Ώ-=Bϋθ£θχΒsψ’ϋxwήˆρuυΫΰ2*XX=Œ‹`J;Pχ²όηYtΌΡΎλYF­ΕƒW^ξ‡w‰ ©CA'θΆ@υΈ2Χњ§ΆmλVlΪ³Ÿθ“qHrqΤr^™ ;ේ…±΄υ{β±mίlXΏΑΨ"NΡ'`Q•’`υeuγ·φΈφοεž+ƒj%‰<δd©Š¦ΛF'"5£ΐ Ώž>}Ζά₯¦ΪG:Sˆΐθ “ΠΊΐƒ.y±H νκW’’Ύ”o»{Q6lݎ!L²’Ÿ(ך²sWφΞd–±|ϊι§Ρχ™~Ψ»ήPŒFw"·“IΩρ&@Ε·k„:yα(‘M΄Ϊ šΌλ=¦/ {ržλ Dκό«©pόψ1#ΏPΚΠρΘ46"W²WsŠΒ·ƒψK’?~iii&ΫΟOeπώZ:”9υΚ+―P‰}ƒΔόX©QΙ\±κ#x4οNšqμJ©‹XuuŸy^FBB:tθ£LctΛ–-R~&πΪeρπ5–DΠ!‰Κ-ΡΑcUοdϋ§Ύͺλ ¬UθKŒΒ”_ΰεΧή0²>ψ½gέ«–ΰ‡|„œbtu#ΡΗƒV{ΒdžUων(ω’˜ιζk›Ž L{[:o>’SξŸώρŸœœ’ωRεύΎΒ ΜΪΒΉ*ύ­CO!@8e}ΉΙ₯ooήΌ™°’[ŸΡ˜1+NΝ”εΑ+!εS2Š°pΑ\ά}χ]ΘΟWΰ³›]SUšj䧟bΨΘQˆg&=Νsъh4%uͺ««—ξΦ¦GλͺΤn,ŠBΤhΜΪύ%ή4kΦ¬1c-[΅*έ€8qήX9ΚF[²x Ύ<ϋ<RMω9VΗύEΗb"’{<‹\Χ]w’’’j”n.†°ςΥ!Ž'²ω΄†9‘YΪ6mڌργΖ°4@=Ϊ¨ hψ±ΌSΈ’ΖϊˆHκυx_μ_? ™STTδ΄Λ{!™όψcŒ™4u³άθQ@‡mŒ ΄&žt]ulΆ_Z£Rβ"Ρ­ ήΤ ŸΎ3²D‘P`~ψ£‘AŽI6Λ³`ςΒήSI2Ÿώ s¦ΟAφπ΅L%’utu8)(ΛPMZΩ ωSδC‰‰‰©Κ[^υΪ–!$½ϋξ{˜7g<9νmΟ€£<ξWΛΪߚY΅-t(αΕσΤqˆq_‡H–‘υΕ/’ι“Η)mΤϊ‘#ΡwΆcZΎ ‰8Π4—Ωτt’8–$ETκϊ΅?Ύ",O:…3gΞ„\_‘Πkψ@ϋSDnO%™Ώ,λ5\€\?αd°N\²—Αn[˜(ών1οΤΌfΗ“Ξ£?π1ŠδwɏEKdZτφPrEΊRΆ΄ΚzwΞuαψWŒ<EΕΕxΰ Š5αΠ,ŸewΚξZ²t ΫΖbΩόεHΝLA£κ"ƒ%²Πn‘h„ˆk‹pZ0¬‰ΐŸ‚jR M (w£Ωp›ΧnƟ~χgtθ·έv+Z΅jψψx#BΛ£`ΙΣρΘ46"·,CΔ$?·όN@½ΘΏ²ΧΊ–ηVH:€4h²³³±nέ:s©kEΉ© Šΰ~ύpκΔ$fwBTjuF€Ή”SώΩ¦ΏQD¨‰Νμ@ψͺsδΗϋ¦¦±”MιxΉτ§UτzνΪ΅θϋΒ XD8Ζψ˜βTΣ:Eͺθ֎»¬δ"˜²€[2³κ~–°h”λΖΐO†γν·ί2NΤ`ΛΘΞ_ΥQΐ,έΦOΜ锁dFnV;ώ₯#Bα tρ\5yM³0aΜDL6΅Fm\.0ͺHk‹œΆΏϊΥ―πρΠ‘»Z°υ…Ήi9h¨mb0Ν1Β»Xσ‘Š‰-YΝ§NψC:΅ŽI@LR6:€­ΫΆ9°ο€A<ήFZ_epκ‡C†‘Θ“‚ο±δDwD*FϊLΝ/ σκφW}Σ?«»•|oύάL”š ŸoΕSy&L€{‚­Ώ+Κλ}¬Ϋ;oΞ”Ι“'›;—ύ,€|η-vξΨΑ Χ]—9΄g«η€ΓKξNž8„Ο™ν.Ώ•ίɐ*ϊ‚δ.„”σ.αΧ'Θ…ΤU7;¦ŠnοθΛϊ5₯_—η³tί­Ερh]˜‰OF~fΖt .¦Ωq³tιRμΫ΅ξ©ˆOwUOnLŸ잚‚ωσ ρΑizK4Ue³σUΘ]ύhk΅iΣ:ΐT /,=›6mΒ‹/Ύ„…σgΑ[Ψ1ΩL.%’X„Κ:•†₯2k8‚nΪqYJΜEtvOx ;cνšθΫχ“Όc>¬α”τ·jνzδ₯Ζ™(4μνΊΦ¬a'δgJcΠ]wV­Ϋ€c%hyaέ/‡o|ΰ΄·Ύψβ <Η½`ηΝCͺΏΕ.k8pgΎ8…hω]Œξη xγΰΥYB4ύάLj=@4&ΥΔfΧ_©(‰ιχ=YN[ šί%Ϋύ³Ζ¬6\nRY »Kn,ŠX>vΨπα₯gψ‘°]*ΒΛg%‘θ¬Y ωγΣΨΈn#κv)DΓ; ίΕƒδœ0B’ω.”,tςD²ΚKr^ς»y RMqϋ<¬ZΎ ϊßэώ₯HΩΑή9>˜ζRk ΩςL*χ$†UδŸ-₯4«t/½ΗΥ^ہ¬οL$ŒΥο{σu μ`6K‡ΒO† Γκ•KšΫQMΩ/EΧiα(Α$,dχς–BδIΞiΟΡM tlΆ`L*ΙCMAU €ΩΊkzΧχΰ&:=ά¬mͺ¨L΅š$ύ}VχυOeŸ\Τ±^#B§™²\4sΡςΠώ*νΌδύ!C°wη>δwς!­Ž2«βŽΌ&‘₯ΈT²Ϋgΐ›γfω–±~½? 0rWΤ7ύ‹€q}Œuη_{ν5|πΑ’ΪιύΆτiμ~Nδ«ψ”LDΔr£§.γ² £ˆ€™―Ώ>‡­Μ†Χ»–u7˜γeŊ„Jό f-X„ ²x™hΚ‚¨l†Ω“˜ίKλ•Τ’ϊν5(5ρΈ§1œ¨ΝŸ{‘†Ρφ₯#Ώ<μJάΎέϋ‘šŸΔ1–ζ鲓b’ΓLŽ΄Β$œ:y«W―1έtϊΌ),ριAρB4Wόη?›Λ αΒIm;a§L™OvcD$pΜIOk6…{“A’+1i qϊΤqΜ›7/δ²v}\½zώJ¨ΫtΚφbέ΄7σιΠηeƒ:yκWΕΘ°ύύ†χQ‡χ)ŽΓν,»Ίχΰ!<ϋΜ3˜;w±‰4Ÿ,«‚Ž²Χ΄χ‘ν―5$‚{ωμΆ*±glΣj-$τA*³¬rΫz±gϋ¨`ξ`Κ ¬<‚ωΌ¬ώ^Άlώτ§?™Ϋ;M[YlΨ°γ3†%K}%:<˜άͺš{I'DΈžY«W­2™ΰΊ“—UsΧ+_Վ Μ2t(ΖN™ŽΞu³pcaΨ£!έeυΨ•―R3>‘-ž“›λΔ£yA&a²R•ά§1ΜfΗ‹z—,Y‚Δ΄€•FΡ›ΥΑΤΉ”™κχq Κf)Φ• W1±eΣ₯ίͺΦ―%s|§dŠ‘œ―ϋΏ ωΦƒ™LZ–ΙVle`ΰK/½ŒVΓ]Ψ ΡY]ˆvβ‘Π¬>Έσ£,΅Ο―ƁΉPNBZŽφuFFAμέ³“5ύΒζ σj=ΌΦΟ¬NίΑύδqΤgΩ0>š¨_ΧzA‡ύN’—_0ž₯ž| €?zςˆφ8ŒΚκCŽΥίς+αg0Λάόα0λu¨τw Ή«σΨ#G›qC΄βͺΆ‹θηŽ"Ϊγ9ϊΉ΅§­‰MzJγgζΜ˜AίyηΒtγ7/œ¦«4¬ݍϋπc§Ώΐh’γ:Ω—mmmρYcLσφ·ΏύfL™‰ά¦>4Ύ«έ½Ζ― ρ™VL³XȜγΧΐ΄β$φτχέWΟƒρŸΗοχ{γSRΩ+»vZήωR5z wΕή*F‘obΤή½{MΉeWδŸ’Ύo‘x³Λ| αhΓ?œ‘lωŸ AΡ…²-fω‚1Ÿ„›P–‘-yv)ˆ£šκ2ζ!KT ωΠξάλ0iΒhfΟ1ⱓͺ*dυ­ά‹/bχΑΓΈ±nš7KrEϊAΥI‘]Υ}ύX64o«‡.υ²0~Κ |@‘`Χ,T6ϋ¬i³‘ί2ξFɈΞ€ΣVόkΰsΉBaΘ)”œΟ Y7N;…OGŽ ΊΚMo€Ύ(= Ϊ¬½ΘΉͺƒX‘T₯~ι₯—QΠ¦λ‘œ^ΐSχ*¬[zΗΚ–zZ 臲Ω11©‡~‚–E>τbi'£»5Wj‚N(§Δ ΩβΉά—(Ψ¨ Λƒχ™²˜k°š»ζEώΘ»bΙJd6Θ`ιa—ˆπO• έ?˜·QΧ’Έ†j?—‡εΛ—ϋ܎α`Š{©Ÿ œ‘Ξxχέwρ³Ÿύ M›6…ΛEΉ‡ iΌ‹&ΑΧ_οΟςΧ›–ΧQ™™Ε,»[4΅ $’Ή†[RN”WDt’Όν’Ϋ‡ξΑ«―ΎjΠ‚kΚ»γv±t™‚Χ²’\Υr=”kΖΛΔcI·mίNψηυεxQϋή΅sΐκoΐΎCΐŸόδ'hήΌ9’c’―ύ’ω₯΅}qψπ!Δ'yΉXškv•6­‰$ϊ/˜’γΊ—ψ\Sšε»Κ+Ž3υrΌhγ‹AQDœš»¨!! 5#κz»‚tLœ:£Τ†všά¬#ΊΦ¬Yƒgž}οΎρ\ ΡhrGί”…τzΙΖ.­–A4— €dj©―QD4ΛhbxΠδΆΊΈy―φ{ ύ^μT”š°ΣžͺΗϊμ³ΟšΊXv_Κηοz-%§,δT€ι7Κr1b„qβ+°GΠ–Αnvb&ύ&ŒGlœκΆbDw σ$§ζ(ρoσžύ¦“H¨hΦIL=!Ε>-[Ά@NNŽq2z³²Π· Ν6oί‰›˜υΩ™πa± Аӣ­©ίΗe^id ₯G5{Ρ‘ΥyΖ±faAAξ»ώRΤ©Λό΄oYY)˜N%J’Σ’ΰmžŠ.HfͺiӦĚH'*Onσ,̚>]»tA·nέ+Νk§^@υ%UVΰ­·ή2‘½r©L^8Eδ;v{woEVƒιt`π€£iΰ‡ΈξΪΉέ ₯€prP³zA‰/ΏτNςεΞιh“EΞl'Τzu»€m8@:ΜΕ›ρQ>|<ψ$%&ύ-ϋΜ ΝΚT%Άo߁$oό%%žμ.Κ „^‰ΓμcwaΆnيCt δ\ιΫ5κ}AΙ«$ˆ²©ΆΣιΦ‘CSfH2u³γN BH‰MHCdb>Ι*;{BMe ξΟώpέ‰`6V’§ 6­cΚ…δζζββΊ†εΉ9oΎρvνg d}7:ζΔ"V‹—fršDqS’υM3fόη'ρζ›o"-5 5ͺ’}RY"¬Όdwi§g§ΑέXŽ ¨šc‡kϊpsGΔ$νA~~“&MB½zυ”$”Μκ{ ΰνΏώψΗ?š@šŽ;–Žz~θMϋ™xŸ’]’¨ϊ5¦ς ’ΰψŠŒ%~ŽXΛ=ΠWώΧp;Φ4ό¦’x ^σ…)9ŸŸŸΌΌΌ ιωΣ$?ωEO?Γδ€3x΅:ϋSδΟS"RΌ;ήβt,[ΌΗ8n|ί–Rυ|%4ν• dΚƒdddΏήv³ΊCAJp]Ύl!²κvA”§δβ΄Θ’€Z lΉTξ~f‚ωΡΰά-‘zQ¨Ζs9ήγΙ'ΙΙΙAΣo•λGΰ~­³§ύL"Wΐ‰L…’|^6bΰn+Y«ΚM:ϋΆq LP‰ΪXD@ϊ{ΐ€¦<ŸΧλ5Α‘‘Π߁νΩί¦υ@η»ςΏ°6+?°#μoί ά3]›URp―k: ¬iMv ΞP,˜Ν¬Κρ@‹\cΏs ;φόSΊSώμDό΄ΝŠΓŠέ.L›:-Z΄0 ΦΆ΅,-Bn>}:ή{χ=œ:~ ΊΑΣ4 ™±†Η₯6wuYΚΓxυ•ΣOφΈ‹ϋE_‹4$Q–©«“°lα2VoǏόcživ3k‰εey.]Ρο8>˜†E9h0ψ'ͺ:wψπa¬\ΉdW΄³—~ΏaΓ†ζ­o¨ΚΣf͚…_ώς—f!κΣ§nΊι&BZΝ4? tFyθ-˜…μzέ™D‡½Y/B=“ $?TtπΎŠlgi‘O3¬[=ž™#+L0MUΘΘ^Sγ`Μ”iθΩ(ιďq"M¨₯Rv<Λ‘j• ιI§Ω±³ι0pp>ΆjΥͺΚ7JkΦΑβ9KΠβφ&Hʎ7χ Ωpc$œPLMΚ!†rπ4IΑήu ’S»vνKƒνψ.+Ώp{nPΥ2UΝ―ύkΣ…_ύκWp{άψίωί°Š"?xπΗK€?λJ%ύpˆ|ω1Q2ΐ˜›ΰΖξs ‚‰Ύkerωίο]K‡6^*χ%Dš;›xў:<šž UŠΉy œ’Ώ-+d1Ρ‘sιΖ{~ˆ¬μlτθΡγΚCϋR„»wμBzέT&¨}Œœ΄-iWΎ»™6C§YjN2ΦŽί‚Η™9nL<Ώ=|εWΟOμ<ՁΫoϋ[-ŸΚΓηŸ‚v_Έp‘#tΈέ―=sγϊ5HΜ*‚ν jωΨϋcσ^‘.D&d#Ζ•€υλΦ£}»vˆcpΏ•W•χVe”r~6j­Zƒ[šδ cΛΐ†;)˜=˜’Ήί52€Ώ΅Ύ΅c@ΝΉoRπιš=ΘΓ1ΩDB[ †όV­Z‰΅+Φ‘qοΊHΜδ˜1‹νΥ(―’ΟB%˜’)š’ŸQ”fN›‰[o½͚5«’Ž†ξ²vύτS(πN¨Hι ,qWΉ³‚qΪΗ}b{F Ρ)olν`θΦ«¨¨7URC{@Ά©…bΏ§{~M„¬Ι &ΫNϋϋ{-rLH¨ΤA Ψ¬λ4̈A§‚TŒ]½ςQ>όπΓA‘£ΣΚ_ΕRaξό4`¬~‡ς>Rrή'ϊβ±kΡμή³ΩάΤ„6zτh<πΐ¦«?όαю6Ÿj4BΥ–.]Jϋo2σš ΚݚΎ‘DκκκžXjω]MΦ£o Ώ•*„šΘŒVpŸ;‰ΙΗ m»Άθs#“ΘδDΝκX%βΩWfΣxψkšΔ_Ψ $Ί"Κ€šC‡ΰ ϋœΒ=ΎεΏΣ΅+Γ<Œβ>ω‘‡2—yόρΗ™lή$STζΊNϋν—_~…γǎ‘:ΩεƒΎH&Άj T"UM³Ά―ς‘O6υ3SΡΜλ’Α θt%_H^rZζ¦cΞ’%ΈkληΙhŒ!σχ°ΌU ΝxgΌςς+HKOCύ>ω&)&.Ϊp9b[ΚυˆƒΜΟ"Ί3 ?.‰tLψέ½ΰ ρsατμΡ³Jƒ3L£@΅ΨΨXάrΛ-HL€‘ &d™„„fU—οšB4PDηο~χ;<φΨχ́)S¦ˆšς]ΖN. /βBlB ’Rκ2kŠYΧ0΄2Λw‘€K3‰Ρ™‘ώΓ]#²(:Ψ#£IΎ€Ίσ0wξlt!βFjjJ@9+ eGϊιHΤΛφ  i’yhη”ˆL«ί’5.ψΏ"g΅x„j””;pž'y¬Ϊ΅ -?ργΗ‘nέΊr…εmιχ+ωΔ^Ο?o!Υ›‚΄’sΐz‘=‘ڜ龦Ύ!ε"ƒέΜψΰ}Ν™ΙlξKW˜Hʐ΅<«(M‘θGyh”‘«ΓΧN:αΏψ…YOζΝk~*Ψb§7;F:DΏ>K֘O’ڎ› Ξj«ΜούΧπίƒΩ†₯ΜΠ3ϋͺ‚Χ·Ώcf@$i=yό ˈ2WΆ})½MžΨρ­R@S˜!?gρ2τiœc₯/c²ε΄„βΌΈƒO5§θoΡ₯€Θ’΄ά¨ΜΨ3_šLΕϊυλ›C2Λ[Ct(KdοφύΘιԌs†uΖνpθP.택 %oπ:v,—~Rι{ψ7’7–KηΞ²σ!fk—’SÞ(zθΠ‘xβ‰'π裏’qγΖF§;… V>B4Ψ·g3κεvε!¬E«J*5θxwWRiϋ[§€UfιΩΡN[›ΑBΙyX»f•Ρο ¦ vS)·QcΗ£eA&:f+ΖY6ΈΤ·Π`€.π_ZΑζίξ'υ§Γκ(Φ†όΪ{Ζ‹i³η’CϋφΈυΆΫώφΕ?³λ‚2Χp/eκ^«ώυξB±ήΚώ6όC‚άL0;‘νUϋ{NJ=¦$‘‚iμšdrͺόv›6m64O>ω$yδ4"SO=eξ[‘>Ϋq€Ηͺl§OŸΒ±#{)eΈΚ]fο§SΩΖkιrφR|nFb‰ž°W7Ϊ’μχμΧό¨‹© š>™ώ€=zΔθ¦`Ο?+ΗmΫΆbΖμ9hš“&ξΘ“PϊP,‡ό|’^βRN~ώψυ·΅%/κ#ƒϊWχ”ΚŠ‘ώj‘ιΒ*O “af3 ½;“Ηr)OιΣͺ§L%H·mݎψτ8’?ƘϋV#Ύ-sΫGΏTτVΙ3}·*Ί_BσYΰŽ3`BΫjΧΆ]₯ω]ΥϊΜ0¬΄WVv΄š|)αΪcύŸϋΰq« ^ω›€€yu­}ΆγϋθΡcŸ½σ”³¨ΪR€J¨‘HGϊ‡± MŠPTζEE@Q>₯}HGι½# ;‘—P’ήχ~ηζΩΜ}³»wχήmww&Ήϋ–©ο33gΜ9s&\wέuaα‘‹„~n‘}λ¦HS‹Ί/σj’ˆ¬Ν!φίω|ή΅ΰۚcTnΔzaΔΔ·Γ•W\Φ^k­¦’1Τ°’σIΓ/3‹Τ‹ϊΫΰΨέ>“O°Θ ­¨ƒ)ιhMN[iVŸΥ¦Κ­vlω ηZb¨m°5YνϋC™&Ήž# ZΙ’ΌΦ,O;ν4WͺΑ@Α!‡d\ ηΉ5.…ωί93Lš8!τ[Ďyš·&YΌTjΡΊZ۞Η7Ν£ΨΌ‰ι¬=ϋ+{g–YDΉ˜q’υe•G±[ωϊ‚Sόκ›o…­ΝZίH›Σφx: ΄N{@ ηΧ‹žύ₯ύxθ‘Β+r~,―ΊXΏπΠk93ΦρdΨpÍςm Βδͺ|^S»σ;ΓΩgώ1Œ9"¬°ΝRa±•;t°ήΥ17©nΆ•¦ΆΡmϊŒŸπΠQU•&Qπ–χ[˜ΰθ§₯6XΤN2ξήΈϋύpΪ©§…†OϊΣ5λ“Ν―L3¨aΊτΌσΞ«ΰσRQεc6':Ο{ΛŽ;ξψC„ΐ% Μο•γ*!ͺΕΒς^Ž Μγ}Τ„Ψ+‡…/ΩρΪ±Š ½BΧαj3ΛάμiaφδBŸ~œ/>Β2mΰ΅AΥwπRaΰπεΝDδK €2 ΎξYζO\}μ±πΌν~ίgC3±;œΤΎΌAՐ->ε`ΡσέΙ³Β,V2β2Π„/QSΚF©Λ3π0PRΎ5L³i…ήyίƒαΛ_ήΩwf ίϊd9…%-Mάι7˜ε9z±0x S@³Ίς<(P܌‰3Γτgš j ώzΧ $ƒ<˜Rӈ•‡‡gn{ΤϞF™\ͺΡWkΑ,QΒ4ρŠ+θύξͺ•‹C5ΫmόΚf…³nϋτιo²ύ…- WqŒ2ξ=ΪΌΈτ댞O'²­΄ΉVκˆ14Α-(χ4Μ„φγς‰θ›ς/κx£z"KΞZΏρζ›ΓͺK-Ά0‹C°*f’hxž~O4ϊm³“QC›‹~[νŽ|ΪbωaαΪ§Η„Ηl‡ ŽziTύΖu;αγ aϊ€ιΖ Πe Ά †Ϊwe.?q™Χ sM˜)e<ž‘©τ₯NsξΚ²ιˆjεΔjΨΠ‘Cv„pŒV£p­πjε Yp†ž³Γ9ςεΈ·μ}?φ‹ίwu―xΤΣoΌnηuΫβBα.`q«=i]eξiηœιγΒάY“m2ΉΈLΈι‘΅pσh;ΜC?;Vuΰbα­7_1AdΛl η·ςΌU-²·4…7‚ζ»οΊΛϊ[Ξ­B.aΗv6 ύζΣ!SfεΒ{“g†av”‚&[jU5dZ†ƒNQ„αV&¬i>ύφ πΐƒ†-|γAǎP’ιŠ7*#«|ΥΩ8Ϋ5ψό3Ο‡%VšB&Ν—ςΤ΄‹δKΡqCv(πLyΊYη˜k–Luœ9^ηzA™gθ2ƒύxd Π΄Αƒ;,f–ƒ½0ε«Έof·–-„AΏΧ]w]?Ύ ½μΚι»*Ε£«t ω+/ό¦L™fLϋΨq;vΈζζRVZN5ήκΛΎ₯7―ύ3V,„<Ε«ŽΓψςͺv2"qγΛ}ŒX(°Ι„Eَ|룄γόK/…§ίx;ω™Υύ Ύέid5`ξFΞΎQ=φR΄;” ύΟώγ‘ϋF9/Ÿkiγ7YvxΈπ‘g].)eΚUNŸ©΄όqέ!SωxόΗaΨJFΓl§¬»πΨ%Λ"°σWέX,ο#uΔ7ηΌo•L­{ždI6ΦH-2ΐΞ>2K `βωΞ»–›x―#Ž%ή€ŽοΫwί}}ρ•,²Ϋ―oΏ.Ϋ™Ύ3i|fT}ΌW]σa37ψΛ=o‹xΟ=σDΎΜ†‘οΠΥaΝ;“Έ"τκ«}·)ΜηζL³§|h·lCΰβ4Ζ^ύU… oυGΆΊξ;d™0hρ5νΘγ»Β³Ο=ΧpCαςV­ϊΑ ;šgόΗΓhŽAbΏ„eΉ«j ·ΔhE.γΰS,Ζ<6(#ς«IK›χ!&’ ‹ξ^°MwX¦‘znΆk³—O<ΗfaλμΠož+±J'χrΊΧUοuE?uκ€°(G»»<%‘‘ΝkhqΑœvΞχσ1άζƒ.Γλΰώ:βΡˆoοζρπlΰοkG=qŒ2)ΖΖVwq}yυΥ0n䰈₯]αoΆΝczD+ Z§GΐΞ°gWZώ0οΒ³Ώγ}…ŽxZ‡\ΑdΔΘB^~εŸƒchCί§ώSaςέηϋϊλ―‡KνˆΓ!Γ+n»€YYο8ZΉš=g¦MΨQ&±t{>Λ»–ML(λ@3ΉΩ±Ψ:Ÿ~v Œ*Dž$ΕΡošyωνV[" 7aπ4[`k'ϊ}kΓή›κ–ΌΦ90l»jsΡo#£aεΕ„E‡ ύχΏaG³ ΏΦ(§ΊEρl’ e†nνΟ§l,œ;¦Ήϋ%ƒV.ďd΅‡Εΐ>Ζγδlιo;Sμ„ISf°|μ―§ω`B―Yl»π(–N±χJAΗ<-=κ{Πψ f,܎:χB˜ψΦƒΆ»oNΘυ3±Β–6±]©pυσ4ΞΚ„Iύ Σμ»'Ϊ’,Ž>Qk'ΌΗ›‚ζύ~8|Β”!—6eρΖΨ6 oΛ’μΨI³ΓΝ/Ž οΑηY;Ψzτπ°ώ¨!†#dγέl£+‹Ϊ‘rkZ$<ϋΒ‹¦φ†+;ίj–P}kϊ΄ιvόήΫaΩ ŒΗ*Ν CΒ‰k5s+’ ψ³fΜ o=:6|πΜ?bj±O £7³#Λ†Πg-P΅»jβ8Iw:ί7,6jΡπή»ο…If ešrρΓ ίY5τ•Κˆ’>rβΓυ\μͺψςΧ3ίΛΟ=₯ίΩ΄yF±iА‘ήWpZή!ΣPθž\ησΗΆτοΝΪTγνm>ŒΒA6G_(Ψf-xΜ‚ΎΉ'9vΔ՘Ρ1vφ΅YŽΙ“«^>J±πUω€ρσvΤΧjK#LΡΠX4“‘Τ₯ϋ.˜½…DπG9ϊΪXΖ s€™s¬>ŒΟcΖ…θ™J£εΔZΑ¦hΏπ€~Lσ©M?εsZ•MuGϊ“Mžg Ιΐ:NΦ±Q¨ZΠOOΧz‹αn=?̞5ΫΪ νΩ<ζ,+όw{! wΊm²ΎΎΆψΧ¬«±].ΖFοJ]ž+΄¬ic^_S˜5sΆ΅*;NΔθӌεΛgXςΎ~2x@0}Ζ”0kv‡’uάΧ«όΉUKNυ-Σ}Υ―bB”oΣM7υcϊ]nq]Δχ’αΊ6 ε ›0sςl–Έ;δΨŘ·μemΟεΰF_ΰGϋXcœ=gF<ά„zΞϋΟς6šœζZD(ΨΏ@η…θ―Έ#Ρb?ϊ>ψγχί?Œ:؎jλλΌq5xxδ'πΩΞ{Ο°za|³ws…άΞƒϊ]μo2βe]Ψ”ƒ?rk{(Σθϋz’vwβΖω>ϊθ#αεη^ λi­0|Ή!υ_³,τnυ>φΩq᭏ Ν5KκKτ+o΅¬[l¬%]¨HΩwŒΰΈΘςCΒrλ-Ώε‰π„-ΎύηΆ_@>)Ίύβ/Ζ?›n±η^£LΓt‡ψϋπμϋrΐ#•AΰgΣ‰Ÿ KΊoΏύΆŸe ώΉ0`@3fLXtΡEݟcH0†+Tž8οgΜ¬ωΗ“¦…ιo=ϊΨ±,gœ{ύξΩ•›9.|φΣλ‡o|σΈπ²)DόαΜ?ΨΉ—cŒ!‚Πυ”Τuο[P Θ͚¦~<9άbηqτΡ‡f¨Šΰ[I.ԝŽ:{τΡGΓl€o}ϊΝ`2iΦΐšΒΑΜ0&xΪΠ₯ΒO~~΄™a]<œσ—sΓί­Ό‹ΐbMG[kta&‡ &LΟ™’υΔ%ν_ύ§§ε‹ϋΝsΆ«aΤ‰αΝηg‡Μ΄uΟ‡αJKΗΧΪΤ€o0υ™aΏ―νΆάjK;ψΆpΙ‡aΛ 6 B“ˆJSξIxπž5yv˜>{zžVa}+¦YΕ_΄°‘s8+\!ζ όQ[(§j˜pΎοΎϋ:aΣo˜»lg<;΄*βͺύν* ωb†pďΓμηο9μφ€ £ΩΗ¨Δ\ΣβηEΞε(•?όατ0ώ½ηœΖφ˜~wΠα‰&πšα8"ΈgόΤ79`uώQ; ž8ΎqΪδ‰αΩΧηΨ"8μzcΖ«,’ίΣ‡ ?ύωΓ⋏gώρ£ί‡E˜ŠyΆlΌz?CΡfΞ]Θ0œfmσ‰pώω燑C‡Vm¬­τ{Τθ‹=τ -€LΟίσJθ˜νθλ©€έ»ζLΜ…6Α>考 œgϋχϋ{xεώΧΓ«ΫΨ…Bΐτΐ9Mž2ΫxŒIΞc@gΚ₯Ι•d ^1ύŽΦ*I§^a±MŸrιμΏνHΤlΑN»…ΤFΈrt+ΗΗ^yε•aĈΞƒ—›nόΝ€]ƒωYhšdΌαŒξ±Υ8Ž=ͺ Mιhb …ώ¦00χAψ‡hΚΘ놫ΌάŽ=ϊgθcΦgΝ“β²V󞺘;ύύ0Γ”ώiζφŸ5<§pέΑ±œ²5Eωχί{/ΜΤ7\7Ν#j„u9εΚ†anτξΤΉa£Οl~ΆΟήα9ΫE}ΦYg†ηήύΨ$B¦Ϋ$ee ζ}+η{γ§Ψ1LΛΓ’*έ_ /}sSΔΜ΄γΓΏvΨa— ŸsΞΩaΪ›ΆΎχφ †[)ήί€ ΆijϊΗ…_|1\z©Y1™’ζ:έω–ήΌΕΓ?f't̚>5άψδΑŒ“Έ.EOΏΩΚυc'ηΒ'Χί0|χ»ίυ5Χ?ؘ7fμ;α’Iγœ–έKΑ؊!„·ν”ˆ™Nr+&£Fjν¬₯±φ”λ;7Όρβ[vtό»U‘-wHW!½?’3Ε“qύΓ;τH³t΅jΈδβKΒ]Χάa ΎL­c-¬«”jκo >}ΖΗV–~9?bš KΠ00β:φ‘oτYpύΛ)c―R¦Ρ‡—σa΅Cώόr%Žc$8[Ά”;逓Jy7΅Ÿιe„eW>·υ&Ύ <Ν"“ΑnψS?­9Š~ύΦΌ wΤ<‡nd0xρpΐΧΎβ³ F,υλώvΧc!Œ{―‰Υ> g~ŸsΞ9΅Ο¨α9τ›mΆ±1sΓΈμθΫέύξcΓυΧ>i%«Ώ2M Η=wίώφ·ΏΕ―*ΊŸl¦ώšΥi ©„†3‘νŠ~Ÿxβ‰ ψdπWΑ±Τ;j₯•ΒN;νδΜŒΔΜYsΒΏο½!Œ›0Ηj²rΥH§iΤ~„θF©όΖnnΡKZmΡόowώΗΜZνFbυ‰rυΥWΧ'£Jry ’ΐ]‡]8 ~z=WžC)zςώGo}ή¨wυ™hςω¦€Tο²­G~•ζα–€*δΑ9ZΣΖ₯‚’ήκ±ΝΜ[lχy3/<Λi8¦_}ρρπζK―„©uό¨{ο½―ŽΉυ†¬ϊ‡εW^1lΆνŽ>7šiBΦYΆΠyϊ6vηL‘¦ έƒΦWΪΑ 6ΎΑίθ°¬`‚¬ άqα}αγ0(ΚFΰpύ Χχ8[,6ΰ*υ8γ2@ψZ pλξ»οΗsLΙΤQŠϋwν’Ά”³γ‘·ΨjGβ!Πcžςκk―…·^ΆΝ%₯dψU*•ΙkΓΕ_\₯ΤZ)™ΑaυΥ–χ’^¦OŸfϜώ~—Ν&oΚEΉ 'σχ¦ό¨…Zrρ%ΓΚϋ¬βG”!οk²­>όίπήϋυ—mΑ_^xα…JΩ½Wsj¬έ½RuΔr9Έ-a!»*Χ±ιϊB{1Ηbν_ϊΧbήeΌ―-.£5b*%a©%‡-η§aό„ρ¦Έ?Ν–‰ϋ„ϋo87LθΨΓVσ24:ϊΪ\Πθb΄pώύΓ*«¬vΨ~ϋŽυSΙ™BΛ•w>dz\υγο5ž―·:”jšΥUJΏ5‡€>sΔ_)χ£ύ¨”wSϋ™Ρ§°ΚšϊάdœY»e3ΙΔI¦hώ؍Άα£²’£ ίξ|ψ’K/6Ψhγ0aΒWΘνksΘ;|4LψE©κ»[nΎΉϊ‰φ4Ε&Λτ ‡/μ²½)NwΉc_;§οΙž ―½υjOΏΆ&ρο»>WΊο*qϊ₯W)ΣTϊq―…Ϋ5ΦXΓwΏfHψ£ΑΙωzX©‘Ζg9εΞ¦UNœZ…‘,hƒbe‡#z;μ°pόρΗϋjGhύ5ƒS}T»,ΝTΏ­CσΕ&κγšk wήygxv=‚OΦ­ύ5£«U=ρ­ΝRWκ3wάqGXsΝ5}WΣ%—ώέvΨL_ΐY#κ¨;u€Π­όFΗW-Ό„ΑΪk―]uϊέ“2V» ³»r ?Ί ί‡~xψε/~aEό…λσW›Jמ`PΝΈΥΖ³ze›OΏ;ξΈpν΅Χ” χή{o£ίΆs± ιw₯ΨΦͺnEΏo½υΦ°λ»ϊŽJΞ΅­%ύuΏ~cUe™e̜ΆΉZηWi]φ$<·>χΉΟεwΕΖii»«υΝ΅jwqΉγ{t‹PΆ»μ²ΛάΒœky½οPΗdw=]΅0¬€ΜυΖ»ά²Q.ζv[:κ¨π«_:œψ+’κ[/ε–™p΅ΓΖΧc«bSΐ7›πŒ±υλ_ϋšν¨Β~γκ¦'Ψ#K`'ο +¬ΰΥέ“΄*i/΅ ΛB9;€ΏψΕ/l—΅ ίϊžz΄Sd$0ΪΠiδO>ωδpΥUW…λΜΚΧ «Ov‹ΧΓ5²­Τηξ`ΘnδϊnkζG}t8ι€ί†ί―d±;IΦ%N=λ²1uΧAΏ‘Ÿzθ‘~4υW\α‹}sΜβr#θw50‡~Γ‡‹―Kc©a&Βdi[ ƒžρ}z§lω^h8ςox₯J]cΪ_₯₯¬NxΎ•cΐήyηm?ŠYΐόcί(€΅‰ζ₯HΥω~₯’mCzίͺWζwΞ$ΤαicΕΦs:ΚPΏVΦΫκ쐁sΪ„,pΥ‘Κjš…κ`λ­·v †zŽ3…3—8p [xˆύqߝ1ΑV4Π­ο°&σ»ίύ.όΙ6vσ]…ΎΉΤwUΎTZ½Ε―;˜—υmΦ§ϊΨ‡ 6Ψ |ωΛ_Ώ°υ ψΎZΡ£fͺΏšαZψε:χάsΓ駟ξk7έvcA>―Ό”jͺTέ‚52p¬CVJΏ“2Mνλ. 6ΜΝXΦ!«†fΑwβXε/ΉζA`½υΦsΞυΧ_ί½ζ)Y{—δ;’ χΙO~2¬ΆΪjν F“~ύπαΓΫ‚~³˜ŒKτ»ω"τ›I“ Μ‡&ΧŒΫa!(Ρο樏b₯`q’U(Š}#οΕwC/>ρ‰O” šόκˆ@<Ά²(›\σ ΐ˜Κ±„­8Ά–5O ”. ί°μ²Λϊ_鐽ί7¦ί+―Όrο ωqΤO’ίΝU©›l²I ―lΈα†έRΖh―ι\šV ί|όO’ou۞<-±ΔΘ°ω曻ܐΉgr j#ΦsͺhοN£gωku‡‚.nέuΧυΞ­ώ½½εϋ°˜½Λ.»„uΦ]' ίJyˆη­+Έbσ¬Ž%IΚ4um§ή ]Φ](ψ6v±»‡–j«sw±ͺw<&γμT€Nt–'ehΤξz³ζ§>C½ΰ8cG]υv εονίΰ•1ο§Υi™Ϊ’Ϊ`’ίqν7φ>¦ίŒ―όι,ήΖ–¬½sΟφυDΏ›³]΄: uvR΄"?ќ-ͺΌReι„κ‡έΨΙ5xl…~·ΨšxπΖΆ­ξδžθwwP«mœDΏk‹oOR‡Ζ±Α€1•?θ7–MZal­tWlOp¬WάvΰΑkeLΨAM»gΞI{Α/Ή„@OPƒžβhc©οV†j«ρί|}«·΅{ΙςtMke•΅ύZ„f|ƒQ'3ν˜μΩΛά§oσjv…iBλ#pω>ΣΛΧ1»CΏ“2M… §;Α»S1έΙ§Qq„pšκ{Ή&Χx¨Υ…}ΜΔΉξ_Βφ+ϊŒκ Ύͺ΅*ΝωΕԍκ§9KΨ³Rj‹­ώΝ=C¬Ύ±cz v˜κ§ΎuΝ­PŸ! υΧW6^zn ν_bϊ©=6¦½)Χbt"Ρ!ΤΨkLT'ρ»Ζ–.ε.Ϊ₯NψN\ώjG?1WO1Μ΅cžpωϊ˜W?’)UΚUΔυΧ‘ξ.]u’κ₯gυε'I>°ν)φ|ΤΖD[ΥΆRߝQ;ή©΄κ·K Wν\Χ΄VΦΨ/DT'ͺ£Ζ–0εzΠUO„š‘<© €ΎbScθ©cΟΗ―oίΎa©₯–ςs‹ηΏMwΝ€ΐ AƒΒ"‹,’&ΌΝPQϊυο8iΎ“\γHΜ@΄ΑαΓ‡‡6"RΞ`ΐ/„ΪeγJ”rŽ}PύΔ~ιΎq „ΆπΒ ;ί“„ɍ©Ρ.]SŠζΘUΌwΏΎ|…”$›£tνW α―ϊPύ€ΆΪ€… Όξ:λ†'Ÿ|2¬°Β ώ•‰žΤ·²…·υΝ½9rΣ·5*<ψΰƒyϊ ύ_s”΄½J‘₯ί{ν΅W2dH{Πδ_ϋΙO~ιχθΡ£½€©ΏΤΏΒ„Ήυ/AύsŒiΉλΫ·Ϊj«πτΣO‡ε—_ή Ε{…Ν–Rq²οΣse”ΒW3ΆR/‹/ΎΈ'^,ŽrV<=§kχ(†³πEζ€μςfA'?姑υS˜tν>RbjVlγϊΏ²'ε-”&ιm·έvN'–]vΩ8«ό}‘xς¬€©8ύJμnΌ|Ζ-|cf1|‘ίώχ}·²/²X€ba•žςθ*œΒ§kiΐ±Ωixι/hN_΅SJ§Άͺλb‹-Ξ>ϋl—θ]s~E*UoC@ν‰υœ;ξΈ#/μξwjΗέM+Ε«>ͺoRŽο«ŸSύSTΫ+η»f₯•V O=ωTXzΤeXyd+½μϋv}.†x”ƒσσO<э5ΨΑΜ]ΒΨΣόΊΜ …»Rυ‡ωόη?~ψα°τ}¦T<`Sά,„]ΕΛ†o¦η…μ£’!ΩnΦΈ>ψΐ5ςΡG3+Ή„@3 @·ξŠ0•¦Ύ₯•ΚPζε„i%Lω-“&N γ'ŒwmΙ%—Μ …Y¦”wB PN˜„dυθ χό«_’φMː}τQ˜:uͺ[ήkEξrΫW9αΚ ΣΎ­«τ——ƒ]‘0…ήes*'L6Nzž@9ψU+Μό\Σ]₯@Ώ?όπΓ0mΪ4_΄Αzj3ΉΪHWώ…Ύ₯;qH§«x]ω«,ε†SψήvνκϋΊςοΞχ–›fΉαΊS†ή§+lΊςοκϋ{Ώ«τΫΥϊτιNΓω~d(Ιϊ^»Ά„τέ „@oCϊΝ:&λE­DΏΛο» WΜΏΨϋΈώΛ ‡oΕϋZbP,νbο…oWώ —…θ ΏbώΕή+—ό―I™¦k₯–IαΨυ7\Ζ}4Ξ-‘`‚U~½π³zu‘§š€ςΫMυΚ+―ψbΜfΦZk­°Ϊj«ΜΨ₯Ί©_ kΜ)ΎπΒ α™gž S¦Lq&–oλ»n@+§°υ+]Κ©έ€v³Γμ½χή W]uUΐκΖ–[nΩξ°4μϋQxꩧ«―ΎΪ‰~―½φΪNΏ1©˜θDcͺηγ??ώxxγ7ΒΜ™3έτθrΛ-Φ[o½οδjΜ₯\{κοcƌ ϊΧΏΒ6Ϋlγ|ƒήs•“2υk―½ζΦ“&L˜ΰ;ΈαΙΧXcI< Πͺμ*Όηζζ†—^|Ι­Lš4Ιy9ŽJΫO|βώ¬°ͺκeφγŸ!<όσaςδΙNΗΩY˜6fTVqθoϊ %hϋΌΧ5Χ\Ση@?WύEΟJ‡ρχ¦o #—X"μ΄Ση}7Έό6][”5{μ±πξ»οζ-`¬ΎΪκauΧ ύϋχοφώϋο‡όη?kά.i›…h²Ϊυ©O…7ήΈ«h½Ζ_ί7gΜπάsΟωE-ζ5KXέpΓ ύθŠJ>HiΒηAK˜»³E=±“4eq¬TΊJε1ζWΘa6ί|σ΄# lθkΜA™Β«¬ΎΊυ΅uΦ)Ψ/ˆ Έ,ύφ—σ~„ύΔ‰Γ΅Χ^λ<ϊŽ;ξθύ'—ξΝ€νυζ›ovZ³ΡF”Γ”Σξ›α[Rz/=mcΔΏρΖΓK/½6Ωdοz/©δ½Ι²ί~ϋνpυΥW{ΫΫtΣMΰΥiŸπ F@Φ*ή~k²ΠήBΌβΌ4<92Aσψρ΄–Σy‹Ίΰx<ζ.βŸΑ ήΌ˜#ήϋ_ί4G½b‘ k˜ΤS\€‘gδ*π’―Ώώz`>0ΐζJ+­Ό²Λh‘(\±|Ϋν½π·ώσŸ.—Ϊa‡|~)Ώ“qγΖ9ΎοΌσŽγΛQOπκ¬š+ ζQτ΅_|1?Zf™eΒlΠΙςgœW―Έ·LXηΞρΧΞ.ώ~3 ΘΜ6g„.gΝa‰ύΫ§z|»°6ζ5wΤQGy]PρŸΌάi§–3!aͺŸzTŠε‘zσSO=5·Κ*«tͺκg‹-ΆΘ]rΙ%9ΜR½Τ±^¨ΥO²mΊlŒΩτ2ΡφN>ωdo›gœqFΣ•³Υ €vh‹¬Ήώπ‡ Πθ„ αsΤM_ΣlβœϋΞwΎS°nΎυ­oεyδ‘Ά§'υ¬Ρoυzζ]·ΌlŠ!m °Ή#Ž8Β۟ ŠΌρ·λή^sϋίsŸώτ§h«Ά›ϋΓώxΐnT π5αYξάsΟΝ™έψš€(χη?97~όψ|ύ(žν̝~ϊιΉUW]ux&ΚύθG?Κ½ϊκ«ωxέ(bΫFΖ¦Ψ”»ΰ‚ rσ?³Ζ¦δ”ϋγΨ©n²€)ΖΨ#<Σ0ΑRŽΊΓΙ?/=WŽXκ―ςΨ΅aBΏάώϋοΏ@²#r§œrJΞΓ^€JΪƒ₯šΫc=H“9!4Ί‚SšΊšΠ2wμ±Η.žπύΏ—cΞΗσ‡θGγ4IγΗy睅θέ·Β‰ΎΧΏώ5·ςΚ+/€Υn»ν–»ώϋΛώP₯ ψύοŸ³M/ €iGhηžxβ‰’i {Ζdh?uo‹8%γ΅“'|υ~ϋν·ΎΆ¨εΨwW^₯:δ ?φ‡rHΞ629Όςo'¬«ύ­`˜p¬ͺΆΨ˜ϋΑ~ΰmyar ήˆγή‘‡κν˜5‡δšV’ίβ΅ΰ£?ώxo{ωΛ_ψxŒ=|ΕώοVΜΫA£oΈα†œm8υ<ΰ+τgJκ9S˜Ξ͘1c|›³TΏTΒΧΟsΧ]w]Žω³π‰―»O~χέw,ΐ]wέ•Ϋi§ˆg Ή‹.Ί(?S^ΊŽ3&χΓ# ΛΟφ³ŸεLΑ¦S<H?.cWՏέT'ζΕΘΊNW;~׏ƎΫ)žκ…ω2λKΆqlΈ€g:ΕλMU‚vVr n# NBwήygΞvιδ˜όξΎϋξ%…•έΞ0E,Š€κβ­·ήΚνΊλN¬μ,»ά?ώρœν‚σEΎ3Ο<3ΗB Δο—ΏόeΞvmM/yTΥ o~σΗήv.η`π¨,kΆΛΝύΞ?όŒZr Z# ΆI>Ά{;wϊi§ημ¨o‡š€ΤΊ )ύTΠο/~ρ‹^_ψΒrW\qEžN ˆ_~ωεέ¦4 γλΣzT7L>σ™Ο8ώ_ϊΧsΆλ*Η"ΌΟχΏ}oV?rL6pŠWŸR¦\ZΈ َνάOϊΣ‹ΉπpΧ_½r†ΆϋΓ 3`@WžFΙKΌ†9Ξ:λ,ۊΈΥϊ›ώτ§?y,»μ²ΞΏ/ό 6πw`Y5Κc;γσ>BηœsŽΧ‰ν Κ]|ρΕyΕ'h γ.[·ώ2ύEa*νόα©Ο>ϋμά£>š{ψίϋ’¬Ys? a«w\©K1ΛTΡέέ’…MM‡€κε”Ο}ξs^ίϋήχr·έv[ΞvHζ.ΌπœνXυχΏύνoΛ’ŸJΣ¬₯δPζ’M~ψαΉ{ο½Χit@ο‘+Πoœβ‘́ ρΰ ―Ήζη;laξ+_ωŠΏG!+ČΣΰEΜN8Α|^rΙ₯Όn?¬> ζšŒ°ώ°Γsœldξ©'ΛηϘ‹³ΨφΘ·ώοώΟ±Ώε–[rίώφ·ύύΆΫn›³]͎£κŒ‡ψž Π₯!C†xΚΦξό»π±]ͺΉ­ΆΪΚq9ψΰƒslΚ£―! ±]ΘώώwΏϋ]Εx)}t.»μ2W°BaυΨcΆΐBLΛt‚τ! C€φ–ύλͺ0…ΒΓΓ@§‘9ŒΕ\6ž …—WΉψ]όΎ\…KΧζD [Ώ…κ8[ςlωΗου.ΎΖώΚ‡vόσŸάΫq’'Ζh₯ϋj# 6GΊ,ΪϊΧΏΞ+Θ  €‹ΓπσvlΌ`Ž<―o§4nΊι&oΧΠhςz葇|.―5ή£lΣξŽ _`Α²+xqaoJd“NΚPώχΏσkb?ώρ] žy<2³dγι±ω\αU/́€θΑ:4u/‰rΣφΫoορΥ¦Ν¨σ[¦°C†5zτθ›ΑΜbiΉ…c2pΝ‹ΏϋέοζnΏνvΗ—>FυxnΕε:cϊŒά―~υ+χ7+Qn8€v`ψrlH sΫ—_~ΉS<θ?I™¦TR³‘N—^z©ο΄DΘΏβŠ+ϊb`Ό+³ΩΏ‘UΚΑƒ!€0!dƒXΕα Z’X§! šŸ8Κ8lΊ―Β&Μζ3€ˆ ˜:νš‚©λ­JuK©Τ; .ΏλU;νK qκ]ΎvΙšΐb tβ«_ύj^)CίύΎζκkr,˜yœ& ’1 —ΥC@Ψ²(Ž@ŸΊω§ΟϝγZ°AQU bq˜tŸθΠxΉ<Π۟&­vT‡'§6ͺ+‚Ϊ)ŠΣ(άΔ π,βΟ ΌΈ9s;¬“ωCϊ)ˆ€πŁ™qvόX€‹ϋ:aμ( ψΨQB9;$ŸVΐa< S€>φίΌΐ‡ΕΦδΚG@X>πΐŽ1ΦQΑ˜φ.G?ΐju€%C8ōοά™Ιn_€Ea §ξX"ρ O―B€6£±ώ€Θ½ω曝Κϋν·ημH&oGχέwŸϋΕm(¬χ(S°+’Ά‡pXŠ/„%?ζεΠk6#Ι‰ζˆH£Θk¦γ䝾μΉηžž.t]ωu dO=υ΄[δ ρψΜ9[Αι›Qbf~m&Ηs`¦χ|#xƒ»ποΚ¬βŠžPί(TŎvqΠAyš,¨Δγ@^Q»€Q’€ (σ΅»2 ΡφO:ι$Η,Ν¬ ]ξ–[oqZΝXJ]ΰT7y@YΦ€€°vΜ1Η$eš"x₯Χ•#ΠU{,ζ_μ=tAJXΡΛ:β‹KΨR~₯όOΧBωfί₯ηζC€ϊ+V‡”Ά_©8…ΒΗ_]ΜΪΞ:γ]’'ƈ₯ϋZ @;ΔͺχGtX%Aω™Ά‡•RœΪ©βν˜+’ΐ»RΌ<…¬Kbα7ή”ŽΕ;ΗΘ—M‘ΘqΚ3Ξ£Uου­|; -`eG6ΕΎΞΝΑ‹αο§Ν*\cΎšΎrΜα1€R4υϋό Ο»—ςd-šτX”₯ΕE±^U‹mDSΨvΉ 7Ϊ΄”Z°~OŸ/œΒpE™ |ΩHπΖot‚‰>€Ήε=χάγ~šΓ²ΦL<κμž{:["ΒRd›έQšοTˆ=τ±K.!Π-¬“ΣΖ ?ωΙOΒ^{νεηΧ™ΖšŸAhνΉ[i¦HέC@x›S0KžΘ.»μβης`DΠ8§p»νΆ ¦θaLψδΧRηM{€τΣm„­ σόœτύφέ/l½υΦή_¨›p…MνόzΞ•΅έγzΔ©^»yŠ˜(€€Ϊ缚@=˜&w0€°σΞ;8ˆ‘^ΥΥ‡#lAΔ³2λbΑvΔ; ˆιχ6Ϋnl’ΰοm‘ΦÊΖΤ²ŒνžΆY œίk“Œ@έΨ1κ†3–7ίbs‡‰³—maΕοU·νŽ_ϊώΚP»1C0AdΨ{ｃY0&œπ󼳩:@»³E?χΆ#ƒΒŽ;ξ ηα5ΰmU°Ε]?»šΐ ΩΏδΚCΐ¬Ζ„‘#GΫ™l·“Ÿ fNDpΦΊ)Eϋ³x9κ垻ορ ¨“­>ύi+ڎΗϊ¬ΰΫqœnΒ'ΏW[π‡τ³jϋ&Θ Ά«Πύmα<0₯½ cϊYλ ΏψΕ/Β~ϋνW49?D=ΪΡ0NχΣ»\-χBύŒϊ7«sώ}Π]SΠπ±žώ³£σ‚ όή6I¦w)χ° Α,“Ξ‡‡XrΙ%½]’¦hς>ϋμ χ¦ γi›ΐ8Π¦o½υV6~0K~O›¦Όf%ί–ΝbΗ'ώ8xJSά0`–iƒ)ήψχΈg‹όΠ7Αy΄™ΊΩf›mΌNΐ,ΐΫβ‚)2:f ₯θΧ‹ž˜ΰ>˜ΨΓνcνΐLϊϋ½€]&μmΖο…=τΫ„ϋΑ‚)ΣΈάž>Ήω²W_}5˜…%‡Δ6x?Sξυw[fλ`;‰},₯―َΧ.ϋ©;ϊ°Y€ς?SDσ1ΩM? *"ύAŽgΚΆΑΓΫo½νWh/ο‹ρΌgμ€ζ3·δJ»EN«±([LΡ&βΒ[*.ρ‘1j,ΚΖ›4qR°E«` •yΊh–Κ<_Κ ­R9ΉBγδOΪv|^ή?›vznβφa‹Πή©c³Ζΰ₯~γφ•C}Σ†©Ϊαi[πEπ#r9ߏasGσW;&/Ϊ$σψΉ4»ιZMԎiΫ¦4γΌπ©§ώΞylΫΨΏ@Vjλ1oχΝo~3lΎy‡μ+ގMa'\~ωε>Ώ΄cˆ‚[γύΎeα…φ9γ7Ύρ`›œ'] mς‚υEΦ#αΉY[6lX~.Vf9Φ±Ž«―ΉΪi χπζ¦Dκλe_ώς—Γ"‹,’ΗWsxx|³œξ»·c b /‰Ε”BœΞ‰—4E_«&+―ΌΗΟ,-Δ―]œϊ<5}‡?δ&Θ²m³Aυ1θΏ)!ω{ΦόΝJ~'|ΝΊd0kCξηwϊXΑΩ9ΟΈ=χΪΣϊΪ~O½ΠlI`ξ‹£ώpΚΧšό')Σ4y5cρΤΐaΤX€…θ}ιK_r?žΧ^{­ՌίΠͺeb`@XƒPŽ›πΟ€Ύ hόαϊχοΜ‚ίΓ3ΩKφ @e@9θΰƒ|q‹z‘Ύ˜8γ¦Ω€£Œ:J.!PkΜ*AψΪΧΎRΪ’`;ΓlΰΩ6“«/Πθ†ΟΌR ή«>˜Lΐμβ`R™ό%W;„»ΜL₯/l1iΐ‰†kl2ΉCΠd¦όσγmνJ–RnXXe1‰…:ΫaŽ=φΨόb¨ΪfŒƒν0 ΆγΚ_±€€ΆIX…‡c|ό Η;ςΙΓΚ/N+έwF@‘ cζƒ]Ι¬—δIxŒpBΌœψ;hΕK.α ͺfY( <ΨιυΓώ8L8ζWΠχδΚGα?υ‚Bπ[l‘oο^oσΨΖN”ΘΨbΦ…e›pπΞ΄»bN4[?QΎΣά?$ίΑF:γΜάΆ/NqΦΫoΫ!Θm€nΰΣ ³²δ|#‚kΝ<ΜΌ….³hμ˜*OΣvύΉR Α8•Νzικ š‹p?πΥwBX4aα\pςσ‡Μυ*μ‰go\ψ§ ApΟ&a―d¨s”±Pπƒήœxβ‰y…ŽRω*~+_Uo,ΎšΕΆπΩΟ~6?η›Ύ}:” ιk›mΆ™CA8J9₯Kfnk–γ‚Y— ΆƒάϋqΕ•J'ω%ΚA€Ε ;Π’=τΠ`G„ε–_.ΐ{Ϋ‘eαΌσΞs™x6-Ϊ)›aΰιQG1^y: fИ¬#}ε=h ΓΔ΅ᾐLzfΩ,/PΈγΞ;ΒθΡ£ƒYΎue6ν XΙΨΑ‚% Ζ’‰(™…sχgΣ ‘Π.mΎΜ–+=7j(ΌΐΏ˜u#W*eΡΰ#<‘QŒΡψ£8Μ5Μ²€σΖπ=΄aEΝ „·G;2 œ`ύc=ήΌ ΄γ3Ξ8ΓΫ1y‘°ΞΪ±hmΪ^έm€UKŸΆώϋ»ςŠY τ΅Iρξjλρ·ΓΫ©-3W슷Σ?σCθ+Ž~‚r<ŽvΆnG¨ζyuϊŠe”A΄Ψ#΄ΑrPΖ6ΦΎ_|qbpNΌ@Α†yΟ{οΎηŠ€Όc]Η¦/θ Nψ‚αΠ‘C}Γο³$'aν™Ή εΰ%…=~Œyf]t~]αΫνjΗ 6ήΩQΒ>—avžν7cƌqω7ν>ή ^žEsXΖwqŒ7κk›ώΟ¦ΞΫΐ7Oyΐ›0n°yAσ(ΥYo¨ŽUάήPTΖ¦C€ŽΓΔ 5&Β *vŽ —3&–MWπ, ~Gu”Bœό Lά3π°KΗ`—„ΔEΝθ/μŽΓ©.Έ‡Ρ‚QcL‚O1ͺ;Β%—¨6hΤqΔnι„]Ψ0Q₯“«/κηL8ΑȊ~χY¨O§ΒL25 €ΗAΏ]膀bήb‘{€Ÿš @A‹ ŸΓΒ9“yaqπC,ŽαT·ώ~"€²4UvˆJx'%ž‚[v Ψ=aB‹rΝ /š΅€3έ’ ‹X¨RΌ8­t_ϊ=–ibΗ;3Οο ,j³ΘΞ"Bc+‹{ΌƒŽΰbΪΐ=u!‹ #FŒpϊξΣOY°ΠΚ’6ƒΰ΅βΨQˆ>߁Ϣ³Φ]Θ‰uζ=…Ϊ>‹(š‘Β Κ*@ Τ«PΤ‚½ν“ο«Ωƒ?[Ι)P”β\6LόΜnms;ψ6œςσϋ‘’4BaLIάGγ‚ΫηJξΪ£πŠβBb”IP"@ιF|#y±ˆΖxΐψ“ΎΠjŽq‘oΗI!Qί(œθσvd›Ώ¦>ΐB<œΒr] ^±vƒ[Ϊ•" eπ(€Ο|$αό§6}Z‡{φΩg}<ΐ"1Κ(š1wEΡ’qή‡4°bΖβ4cΰ}χέN>ωd·΄`ΗηΉuV}Ώ'š~Ž€κω~πƒx™°ΊρySoΚωΘ'Pξ’^Q8`œ€M1Ÿ<ϋμ³}ν€HXcΔβs,ΠΎ5ηT;"yb5e+ς₯£P†Εk;ςΪΫJ λ7ι'!PΰΓ ‹XνFΎAΫ„Άfϊ ΄–Ή!NŠΩ°1o‡U/˜γkŽύ…ΗSšΔΧ½ΦpP(ccŒΦ~²y΄β³x;¬σ2ξρ,yΆότέπ(Δ Ίπ…ύ5Pp`¨q0Oψ2’O]1o’n•INλ E?δcG­Ζ1ςοlG§v ^Ώ:ρWΘώνΎ†ξΚΆͺ―Ϊ1ŽΝ’aΙ_υƒΌ|‘9"‹€θklrbώ^Θ1Oe…5IκΎ½Πά¬Pάfx—”iš‘zYΤi ri br1£₯wιZ?¨κEP‘œΩιΜ@‚Γ N„ΥOΝgυΔξ&―0˜6C({Θ!‡ψ$ˆ'ΥIΝͺ‘ν GPΒq,τα i#·=Xuζ±+ϊ}χϋQP(NR8¬Χ˜έ‰:–΄}³Ši8‚Iv© ΙN?ό8RA.Ρπφm'=ύrΡhβ(g1Εa‰ͺΰ$s^χŸhζάqμ,€m.utœ{Μϋ±s½]žIsj§12εέΗt!‚9„,hΐ_#€FΈ‡ΐZψ"€Θ !ΘMώ˜ΎξΊλΌμό―ΆP^ΙΪ7”Xxe΅€bDΐ”]΄( ΐ‰λ:Ξn^6‰°€?ΒΣδZυA)ΔρΕΐ΅$Τ±,ΕNm Λε_θ Νf±ΗάNidΓβ‡p₯” Q¦Ρ‚ΒcριΩxΠp-ΔG9%y”kΑf±ό³ιφ¦gπBy…$vrŒ›ζ"θeω•κ>ΙQBΚ·ƒ › =ΧBΙ;＝O“΄³h©ΈΤe+bŸΕ­«gαRJ8ΖIα€ψΒ %°DvΕBm)§8lξƒ–ΗiŠΦ—ŠŸό]! φ Ώρϋίήi–[ΰω–YvŸ‹CgQD8υΤS]ŽΆ;Ϊ«‘A,`±zpάqΗXt’?p•νΞWYhΧS§MuΛ2Q 8ΘpQήaά‚ίG ~‹",ffωMv~£ŽΌk[τ-β‘€ƒ,Ε?ζ(a₯˜q9%VYYc;ΎΤΟTΎtm,j“l΄B‘†:€Ικ†Ϊν…όiŸΤ#VΨ„K}π‡?tω‹’πΜGhkk―³ΆoΎΠW6Κ‡qς„Npε,x"žι ΌcόL.!P+D‡t΄2J΄8θoA9ΙΌ‚ΠF‘ί›l²IžGPZσ‚8νœΟΫ½γ4–5NY’,€p ΈβΗαo€ˆ&ΏvΉRβ·γo†VQ7πrW]u•{1ξ,1r Ώ—4(['JGψB_„―xCς̎{J‡4U&…—Ÿnυ«Ζ ζ€ύλ_Γέχάν–xΧ^kmWΦEΦcΒ=ν^sK6u¨Ÿe±’Oΐ{?ωδ“yΛ4τ5ςD’z‰Σ' ψδ68ϊ%γΌ»ΚκMό“”išΈrš½h0πR€‘£A³€ΩΏ‘ΛρΑ©.βgά)§œβ¦raΈe6ΊqhΖo’NTL`9MŽ…q&Τπqͺ?ω§kB ΪHΠ(†%΅Ήj#\yz’ͺ‹ψ™•μ Γ±λ’8…υ‡τSsΐΕ3LfcώΗ;v²˜#Ε‡š$eΠςh·Άxμb,eΊqγ;L«ΎτβKαΊk―sΑVοΊ#t@˜Ξn!Ξ6†Άμ·ί~E'ΖΕςJο;ϊ»h3fώ₯@6…P9« £π„ΝV½rά 3,b`ΊZ‚#ŽΒϊ‹τ³ΒGB8vΊq Β¬±‹ε ŽYe.;sq,\—?γρΓΊq΅cMi/izΡ’ ψ‚ Η\¬ΨXήίŽš«I˜›D}wΦμYy₯„‰ΕΪ΄ž4Yl•Ža†–ΈΒ(₯Γ‚§Ζ —±/:ϋ΅=u@GP.ΉT‘οN„/¦('\…%x΅³ Γ{΅ڍŽ]₯ξ₯ΐ€vP¨<νόŽΆ ]Ζ!„ΖYLάkαͺX_+‡w`S}ωCϊITψ>,Χq<Ό κλX ƒ'δ˜P ι΄]Ϊ1<8²XY†a·<ŽφΏνΆΫ:]ΒBu֍7ή-eΑΧμσ΅}\ΙOω16°X…”iΰ'±β€Ε+₯…UrŽ‚‚Η”C‰‚#ŸΎυ­o9ΟΚ&YmδΈ'<Κ4(k0χ%]ε­pιΪ8DίΰcqX`wΏΖBΖ=Ϊ §΄7ΪυGxTqίώφ·½ˆΣΞh£Xΐbm:vXeBŽRνU–󙿐>–š’KΤΡ9ρΉτΪ_Φ©Ÿˆ·ƒ―Ξ*( ύ ζνH›4eOό]œώΔŸ SŒΟŒγ΅κ½κ@˜ ”%°”%y6Φ°P‚+όpͺΣBΨ_h“πe\ΒΙ―P<Φ©•Β ΧΚο¨ κEI6‘tŽr₯\V πΤIΜ«g璊Λ|S}FsXυ΅B<ΎΪEάΧθ_šG)έfΏ&ešf―‘^R>uˆ^Rά–.f\Έψΰgžy&{μ±>‘γ<ί<Π'vq˜–¦I>NυΓΔ›Ι);ώœ•Œ–#AΏ,±«›]αRdRΌtmθ“R`AΐW¬Ρo΅θΤ•ΰoω4‹ !A?ώΣ&q”ΧUY”.e)DΛ‹}‡'ή?|7_ΉYapόyΒ‰:Ζ±|·ƒbi‚«φqιΎ8΅θkΩάZ½έgΏ7=Χ΅)ψ cθ:vB~’ΫΪA/,ιθ8ψ –’ŸγDPΔA&¨τπCΉ~’΄VYe^εύ•†Y(Τ’€œχ³υΦ[¬βD mβ“PΐΕrόɟ?kŠBςš—dΊ4Τ υ$ε¬ζ’Nƒη₯βΏχή{»UFž©wΒG›Ca >ƒzǏ+2 δηŸ~ΐr¦V PZΗmΏύφH£v₯dŽ™Δ2Š6ϋΫίΗδ€'c€°ζΚΒΐ;μΰGp΄*»΄‘Ω(=±+{χέwϋ™₯&Kͺ+υ υιΤκ]Y…JzΤ­ΪQ)A8‚XνˆTψb ΝU˜R\„•ό± *Αb+«²”#°,VΎVx^τoκDBίBί%?ζΦͺ Η;ΥqŠΥtBΨ—“f±ΌΪρ=}M2ŽΪwΉ}­qLίάΨi’:4žƒΫ( eώσŸxαΤ~9Z‚γ@q(΅ΛΕ<τeκ¬ΓΒθΡ£ύ5Gpμ»Έ±6ςΞ;οΈ"7–υPž ?Κ%§τQϊ‘•'ή)ŒΖŽ[SŸΔ_ρt-F•OΊ6Υό,e᱃υσυΦ[Ο¨h«΄9]CI9¦'Λθάk\TšΔAΑ e΅—>ψΐΫαΥ&Ή'pXύ <ΈŽ7~›~uA@ν0ΞLΚβνJρ΄υ˜·SP\”δ‹9Εƒf«? ΫοΑό°yΡE…Ÿώτ§n” ώXΏ’•Β/ρκ1>ͺSα+ΟΗρD§βΈΊWBυ:έ@IDATΊστΎΥ―ͺx6oαΨ +Ν³Ϊ8χΒSχjχ]υα«πΊς>[/*|…κtΐ€ώΚAώΝξ’2M³ΧP*_B ˆ@AœP Αl'Œ0‹(μΦΔ¬grυG ¨ΘzβA]²hρ³`~ϋν·»ε‰ΨllύK›rL$@LΏ±x€‰]θ7 +„p$ΉΖ Σp§ίΆ ‹˜(ύκWΏκfΈ±0ρΘ#ψ’8“DΥgcJœrm'$(ηh €8Ϊ_<1f—)< Κ4XRAπ.z;aΥ“oι€0ΜÎrΚ4Έόn>ei…S\vsΜ1Ξ£sΔηYgε;’©«D3*―νΚF‰FΚMͺ₯Ζ10μ(G™†E/Ϊ?Φ›p|°ο0ΧB΄[φΜκΨe˜MSi§kοE@}Α΄Η.| ³_FΡqP’ΉΩ0JΪͺγΪI)Ώ8ω± -PψΎσ”#yŸ-‹hdω@ρβtΫα‘ϊ’‹.κ Υ²P•ύnπcuVΜr p–,‹h1Fah2ϋO»i7}γJžιk,ΈβθK`\¨Nΐ861_I)lB ¨Γ?\yε•αφΫn 7›uΨ±Ωš;hΙψργ}ΎXjάΠ€ΈΚΕH¬θ!ΧewμP–€χnSzaH)^Ÿ>ΟβτΉOJYDšγ~‚6Β܎R¬qΌ/|.r,\ώσŸw…rژθ*γg±vΑ˜–mπ:RJ4\yθJ;RŸ―Φ%ίtM4’1ρvβ;β’ˆΦΖΌέ°αÜ.ΟΤ¦gΜμ8V(Ž+Ύ^ό8›hŠυ«8^+ί OζΫXνf3?Ž#†8Ϊ’0ΰ§y|1>žΈβγQ ΎŒm8aο™ΖE₯«π™ -ω(|™+rΌΦΕ_N;ν4·ΖΌ\ϋαΑ5Ηα~nΎ6››sŒ5”:† ‹xš«©>k4Κ‚χ5ϊWo›G%ešl¦η„@/G@D‚Ζdλ°Γσ3nQΤ@p/Ÿ ΧΛ?·WŸAˆ?ΦΒΥ$ˆ+“aΜΒ²λ„?™ŽνU™ ›HtΡeOΞ>ΗΚ Μ.¦ς“cμȐω 䚸u;³±"˜0c’€ όωΗδ‡PIŠNLƒYΌM.!PkD7$Όwμ(oΡ xόqL~‰‹Sώ~J"€0 ŒΣ£%x?zτh硟~ϊiηε”α #,†wάqΎs«V˜tnuͺ‘UήUxiΗ-tXm]W₯„FΒPκπα‡W_}΅[Ίΰ‚ \Θ$₯ΒBΗαßzκ)·:†ί·ΏύνΐQ‹Ι΅΄#Š£Fςc}­δ₯Ϊ|ΦΏp2ϊCθBu ©J#KΟ‡ιm΅e $ΩMˆΐ±ƒWόπΓέKNJΛ£PόVxΗ""VθΟ`\ȁ γ'₯Έ‘₯Z1Ζ₯<”iξζ-Ζt<η± γΘ–x™U†‘Ο HƒC,βŽ8βίUΞXνV=ψΰƒα+_ωJ§έζε !Θ·•r₯}KΕL~΅F@uˆ%™ƒ:(|eηΓ fω’#Ÿ^xαŸs S⏅ldK τb1')UχΩΆJ{Φψ™υ‹Ώ3ێcΏtŸh΄]h4ό’x;Ρu• ?ΟΫ]—’!|&NJΤ _₯hFΨxώ‡i‡{aϊΜ3ΟψF~ζΨΈ³Ο>ΫeΪΜΕ†χΠ0ΙL™ΧΔ~ψΛ‰gάΎlhΔ1ΆR§Rβΰa Uέ(ΌόΧκ«xgŸuΆΟ/±ΘK½@Ώ%ΟFf=yδ0`ΰ€pς)'pΑεRίωΞw|m|θΕ”bˆ•>ζW:’Ύߎ•Ύb} Ÿω6Žρ+^#υ—Mώ“”išΌ‚Rρ•  A3(Οΰ8zβπΓw"₯0bΌ+I?…­ aΠžέ$Χςε/ΩϘ•_œβμ9³} Gx/‘\μŸξ ΦE@4‰Ηƒ°ΓGπΑ"Β$ϊ]Ÿv Όœ²Σ‹Ι‹©˜α–_\ –˜Θ1ωN.!POX΄ε˜!%pPώj―L\΅ψŠΰ’Oί>$Ρ!UόŠ0ΰΖo wάq‡ι΄Λ.»Έ "!28kρπEΑΛ^{νετkV˜<.dΉ&›fz.γ# Ku A§ΪΌb²3K%–XΒ…<s†ΠΑ’”‘O}QίπβΤ%Gr‘Τ@έ%Χz¨­H!k€Μέ$܍Ώ˜γ:ή~ϋm·$%ε,όIC.¦§qI3+DƏτp+°b^Iεΰ{ξΉΗ…,Θ©ΌΜχόq­|¦]ΰ)lP€‹p’οƒ?Z‘ϊΙΦ 8”€8ήOxχˆ~ 3(@αμ“fLC’ ι6B@u’ΎΖ‚o©ΎΞ(™iα„€²υ%Ÿn5C€>‡Εiφή{οπ£ύ(ΐGΔs>θ@ΌxDK΄+|‚ΪRΘοpEš-ΆΨΒyFŽ•²₯ΧΈΘΏ««hb6œhbφ}zn"¬έ`-^žuYψΫvΫmό Η?1w‘u,Τ,½T‡bc&‹Ππ’Νϊ2hovLe! z(o½ϋξ» κWΕ§d砝¦‡„@}ƒ·Ϋ|σΝΓλ―Ώ^6oGQ™ΟΛβ/ό: 7τ5΅y₯Ϊk―ω—ΆM5R€—Ÿ{΄ΙίΜ|œρρ6³ήΆϊκ«»uuhX 7Α‘H<Γ«#oE#.ΖWJ1(“2£8(…₯«+ο}τQ·ΰ֎›Υ©uΦ]ΗΗδ ΜmΔ3€‘¬ΤpζoϊF‚υΧ_ίρΧ<9cŒΪ5aU?ΜEαε·Ϋn;σο°ΚG_ƒwGξ•d'žGiΫ›ζQTH.!θΥˆ˜Aδώς—Ώδi~ωΛ_†£>Ϊi ”„ƒHωŸ™πJφ0ΑfgfόQͺ‘ŽΤTΊϊ©§½0«¬²J~ΗVνK—rH$@LΏcEš“N:)όπ‡?tα–~'Ήϊ φ?ωΙOά‚G8ΙAΓU,ΈΘ‰v¬Cί“KΤ΅1 ·ήzkϊ'4PΔντwή ζqΎΠ·_Ύ ϋΛτSx9„5€1•+α€x9°Ζ±ε ›$€†όλ_ {ξΉ§γ½οΎϋ†_ύκWH£ψ1^΄Ι£ ( ¬Ž£}ƒwΆν³( Ά0mΗv wέu—+JήtσMαΦ[oυΌR˜:α„ΒK/½dGCmδυ₯!EΙ΅’‘΄ „½(ΜΡΧεԞPΚΈώϋύ5y pďx'ή`5Φp:pΣM7Ή°Q~€I„‰X@Α­ΏΑϊωΝXξΠ±ΜώχΏέŸπŠΗ Ϊ'Š`»οΎ{Xj©₯™Η‰8rΌΏύφΫ½½‰ncu€τp²ξαν8j(Ωΰ—\B Ω€·γΈ_ŽΈ™« {vu^pώŽ.Ψ“·τ›τS΅OψjΧpωnjγuΖ΄μΤ'ŽΎ¦]ΰψaIΎοΝsr z!€β |8vjΛ’Lί>τ†#  Χ,"²p‹ΌOΌˆŽσd~έη½θγΗEύŸoΊ#mθrDΦρΘ§xΠ­λϋ§/†± {sΜuO]œOΣJρk‹νƒ±ξ.SΏμ²Λ:Ι(ΘΙ"(Ό‡¬8²ΈŠR?ξ’K.ρ ΠfαJ;;οΌσάκZΌψIϋίΪ>qΘ=XŒ…‘ύΣ&5Οδ=.΅#‡!ύ4Ϊ΄x;ρΩ(@§Λαν(> hX”…7Aή7qβΔ<FΡƒ~ΔsΡΝ6Ϋ¬Α_\μΑœ‘q0ίήc=œN ;Α§± Z£?ήcυϋΰƒv^:ΖXJXώΧΉζšπη?9`1Eό9ybeά₯—]šW¨!E­±‘pΘ<_eυmπέζ&—w˜Μ ψγΩWκ‚±‚pΌΣΡͺΜ-wάqGG‰1ε^κ|Α‘Ν'š‹’ ό„tΰΩQ8ΖΑγΣΧxOΌ>ΆΝόšχΈ/~ρ‹>Φp―vΒ}³»ΎΗ›kφB¦ςυhψh—A¬Πό—Ύ”:φ¦NΡ;Πξ\J ŒC Ά6f3thw"t`G2άcBJψδj‡νŸ]k`ΝBΦ-Ζ„-lBRή3±ΑZ ¦ώΩ]ˆ"ΤώϋοŸΠ¦ΎS»zI)wF€ΙŒ/“q„Γ&ξ*=UΡoLWώόη?χqSτ›…Ϊbτ˜0Ό‰~W»Fζ§ύ₯~˜„`Ɵ݈X— ˜$Θ<%“;¬LΰΈΧD=ΡοωX¦»ž!@[‚ŸcΚΞ›―ύλ.Ψ!U΅3XBa ―°žv `Š’Η‰'žθAΠ!Šί³Άvla/Η`=˜ϊηΑΌ5 ΏψΕ/|ε88”gHPoΠ,W±θŠ  α7t$ζΝI>ώƒ>pEνjSή­pΟΎŒXœΧΛ/Ώά•Xπ‚~³(€Πτ·Ώύ­grϊ駻"~…œΖdΜYτb‘₯κZ~…β₯w½κZ‰•#κώŠ+p% h(Bs‡θίX9ωίύί°Ϋn»yί§ν!θeαŸw,π#θEΑ!ma$»`™ο‘€°3;=φX§΄K,ILΊμΤ£\ΠΓH“ΕY„Γ'Ÿ|²+Ž ¨σ½ο}ΟΣ,Φ>I‹―»ŒΏ'. ­ LΙwα˜_£ŒΔ»'Α ?κ kgžy¦—(E3FR/ψ³σž9χ―ύλ°ι¦›ϊLÏ:C±Ž…q¬ΣPgΰΛ±Zμ²EΙσ‘GYπˆ Α‘‹ζμάdŒΨω+;»…ŠvζέΥΧO©#ϊύšΝb,Šΐ̇臧vZΠ‘Š` ί Β;ΖVψmϊX±ΆO Ο±<ϊι›Œ­8K.!Pj+ŒΘOα3₯’ΜΝfᏢΙψ@ΫF!—ρ:MBΉ†gθ9ςΡzΪ9‹QŒ/π(–!Gίx㍽hδ Ξ"ύDcrCθΠα‡ξτύΜΣNWYfA4υ‘ΧWΡ=Ζ*ΪΚ€Μ#¨'δDo½ύVΈνΦΫ|ξ/Œ¬>ƒv€\ƒ1ΪΘ\Dm›v|β/OtώΕsΪΘ~ϋνFνM†7‘£X ρ–u ζ:jΗ΄U,)ΙΚ[s –JΡͺ@³ΰs9ϊ ?c”ρMo]Ă퇂z1ήh#Όf¬@?bξΞϊ'GαHω NyϊCόπ½Ϊ؏%5ψ-”<Α:Β5ώ­‚Ώ¦ +ΘΉPœ~1οΖJ γ-γΌ;c,ι [Δc\¦N‘CΔƒWdƒό=σ‚Γ;Μ‘s8ΕoƒjρOd¬Θ:X+cμΰ™5 ”.œΰ-#2ΗΒςžxζ4¨“°YΛ»ξš?ς’x€D ―iE_cΞ‹B&F~πƒψ|KεΙ–³iŸ­ΐΙ%zŒ€6OΓ½4gΚ49λ”ώN~=Ξ$%PalγO”σg‚ˆœ -R=EΆ:ͺ3=³9;σxϊ±InΎΚÐκ£:°§TΚD@νΣ„’9cfΌm£Zfμ¬Z£Ή](EΗm1=MΑ©«U–”Ng„―MrvμVΞ& ΤΥ>ϋμ“³IEΞΊ<²βtN)=%*G@mΙ”,rΆ“Ϋ۞ ?=!ωρ {[¬ΝΩΒnΎ!Y šEšœ :r&pͺΌ mCψš€'½6τψ21BΞYςh™UšΒ•’νψ™ΠΒγ+Ο|bι¦ ΒΙ΄r&ΘΜΩ"Τ˜›`5g–ErΆ°]0 ½4ΕbΏ5e4OΓvΌεθ{8εγι§e°Eʜν\  ™’sξΤSOΝ™"Œ»Ϊƒ qsGq„‡gŽ—υ'°{ΫoίύHša ©9SΨ锦6ΑeΞ” sΆ€΅@ά£Ž:ͺKZ‘t(£Yeς4ΜκŽηΥ ?ϊ>θ²)Ώε˜Ogιλ^{ν•{πσŸ«8¦$“kΚ‘yόεΟXΚά܏ςα”Ά)Lζl!2Ÿf‘₯c ξΉN8ήΣ€>₯ρw>Z`xΠA-€―νΟ1΅YlΗθψ•yρθαM1΅¬yόυffλsΆΐβι¨nζ—$έ%J# 6­6%―œ)π-Πnig§œrJ+ZράsΟε¦ύšςJAΕfrΆΉΞγύξwΏΛǁgΉζΪkr¦π’OSis΅Eœ)jζLαΕύ5? [\τwΆ`ΥiŒŸcJžξΟάΒ9=OΎSίJYΙΓ0SίΙΧH“έt,ΓδLΡ3χο|Ηλ+nά›BKΞ”JsŒi8Υ―)yζL‘t8΄SΘΑŸv‚S»‘Π^ ΅IS¨qY ρ~ϋί{Ότ“¨%jΟπΉfqΙΫ,ν'ΏψžuJψ‹Jy;h12υ‹ΈŸΩ&φ΄·ε€Β˜Ή΄#Ζ₯«{ΖΈnLΖΗ›l<<ό’š²@άο~χ»9γωC/ωYˆrΙ%z„Νν3vmc†˜]±˜nEΓP~=Κ E. v==ύΤΣ‘Ώ™$ξ…΄.y‡ζ.Z†˜ί’Ή²2IΊ…€κƒΘμ@AƒYκνYŽεr³σζ‡νVf)RB `™m~4Ρ&ηŒϊδκ‡ηKCβ–₯θ7;+ΠΜOτ»~uDNμ„e—λ›o½f͜εψμcχ f2q‰†; ι§J¨=±–;μ1E™°ζšk.ΠΦ–+Ό† ρΧ (X²’ΒϋαΦOYĘ±Λ kμ†β=sψ8ψ9¬(,cͺŽh!θz1ΪnΒίρcB9§'J£¬Β΅y aΕΜΩ‘MŸΑ±›Ύ†΅]9₯Γ.^x"vVaΡ Y¦ι ΉΦς§_s<»πΰαΝθίk­΅–οώW;α«±ΓNXh.R8Ξ£P{Α* ib}J}}υΥW3š±†Σ„8MΥ3ιc…ΐ„Ε>‡Ύ°«z^(βfi0~°+Ϊ±› ΧŸ…e§ο3ξa-Ζ.aY‡e ϊ8–c>υ©O9νVZΊROΠ’_z1LŸ6=o½z˜ZŽ#},Αγo°ΑΎSΏ-Σd1Γς°ξA_c^C_Γzc₯κ‚x]Ιnc¬vΒΛΠΧΊša͏Ώ•VZΙε^²‘-GzNt…€Ϊ"rThς³"Γ=ύγsΦ\kΝ0eς?Φ€q«.π…ŠKXОϋφλVYy·μN;₯/HΗaΗ=q mΌgw8;ς±Ο=Η/ΦYg§gτΚI8,’¨―(]Ζ, Sΰ85dψς£œ”…2Α?υPi^‡5dπ-Œω΄K,j¬ΆΪjn΅>KOωΪ(u _bJ‹a萑>F1nνΉηžΞΧ`•‹Υ΄ œaΌ₯M"‘m`…‚φˆL‹ων~#Ή„@= Νίe±όO{U›₯½Κ‰Ύ9oχό<ήΞΪ/}…9b9ΌΦUhί¦ι}‹΄{,€β”‡ςlυ«Ύ^kUŒQΒΌ―‹?4«i‘₯4ΐ Z„Υ,,¦POΜsy[+SάΌŒΦx{ΖRuΒ|‡±OαZ½>Κύ>αΞπ` €OωΕi1Ύ0‡Ε’™ζΕΠ|xu°ΞΖΡ3aΡ`Ό Ž«Ύ†ηήκ’2Mo­ΉTξ„@B Χ! ₯TΑΛ S*~ςK$ Ϊ Π}ξΚΏ6₯J©&:#ΠU;μΚΏsjι)‹@9ψ•&›nzξ9]αή•ΟKRhΊj']ωΒ «8Εό‹½W]ω+\«_»Β‘+Bψt§+Bi¦wθ Γό;§–ž΅G 'm²'qke)‡ήŒ@9mKaP Aα…£ΗPτ’’Uόύψ³Π29Gž’ΐ«ψΊΖαΣ}B · ΠUϋ-ε_ʏοοΚΏ·`Τ¨rv…_1bου]ω+\ΊF +όŠω{―\ΊςWΈfΌφkΖB₯2υ^θ rFΤsΊΦrsKυT.R='¬‹Υώ ΣσάR ΚˆΫfj‹•γΧ“1φ妓κ¨\€ͺNx«+ωW'·”JB`AβΆW¬½ι}6NIώρ»t_>Β―Ύ CͺΕΒ•Κ1Ž_*\ςλŒ€p+†Ήό;Η*ό§QIΌΒ©₯·½ κ;ΈμψjqψBώέI“|•VœΎΚƒŸόυΤ5N£’x₯l?}OόqΩδΏΛ†Ν†Ρs6œΏžK]γ4*‰W*ΝVπ‹›ψ›π+„UΎœχ•†ΟΖOΟ ΅ΉΈ]uε―8ΊŠ‹_ό^a•vμ§w\.φ/υ.ŽΛ½β)Nμ/?ήςΓ¦ϋΖ"@ύΔυ•-M\XG·£ έ’ΡυΧ_μؚNΑQΆΑRKXά‹*”_Ά ΗOχ Z!·ΙbmOογ°qyδΏΣ=~₯β•Š«4ZωZ ›RίcΦ]|•F‘όρ“©r΄³_Œ[!¬τ.'ΌJα[*ρε―΄zΣ5)Στ¦Ϊκeν͝‘ΐΫeώ]BΤR=5E5€B@ ΅Ν ΤιUΒΎN@W!›TWU1%Ρ-*i{•„νVaΪψΰpτΡG‡³Ξ:+p|GuτλΧ/pTΛΝ7ίŽ=φX€έvΫ͏c5›~φY{/tMΤ JΪ^%aγςv7^œF«ήW›ž€Ρ“Έ­Z'ε|WΉΈ•.›gwγeΣi¦ηtΜS3ΥF*KB !H$ „@B !H$ „@B !H$ „@B ‡H)ζƒ>§žzj8逓 ¦ΈΕ[„γŽ;.l»νΆ‘oίΎΓ€— „@B !hG’2M;Φzϊζ„@B !H$ „@B !H$ „@B !H$Ϊ bΗ6΄mφ±R¨Α ΝO<ž}φΩ0~όψ0gΜ0xπΰ°Β +„6Ϊ(¬΄JŽL.˜Uϋ—\B !H$fΉΟ\"!P5V4ΙTθ{Σ»„@B !ΠΫHtΊ·ΥX*oB !H€@ΏR;H$ „@B !HΤψοdόε„©_‰SNΥF [ΏiNVm„Sz „@B !Πͺ$ešV­Ω*W–Ωͺrς)Ή„@B !θΝN£›½|=€>EM$  P―ρ‘^ωτŠ5!H$Mc)«Εψ¦ώˆTΈ„@"εƒgΜ&L˜¦OŸξh 80,Ίθ’«\6ŽήWϋZ―|ͺ]ξޜ^)Zžκ£7Χl*{B !Hԁ€LSKt› m1Aγƍ —]v™3Λύϊυσ’šχιΣ'τοί? :4Œ=:¬Ύϊκa™e–ρπJ« >+‘MP›£ύ^qΕaμΨ±a•UV {챇·Σ6!}f› pύχ‡;ξΈ# 4ΘΏΈζη40,ΉΔ’N£W^yε°πΒ ϋ.BqšΎY³fΉΰUcP³”+•£q@γiσχή{o2dHΨsΟ=έ΄°hγJ–rN$ΚE@ύ3αόη?Ξ3x7`ΐΦβŸkΉfXd‘EΚΝͺΗαX8`žS¨l•&ο~ν΅WΓ₯—^ζζΡ·Ψb‹°υΦ[WšTΝΓϝ;7\yε•αΉηž K.ΉdΨ}χέΓΘ‘#›žo¨90)ƒ„@/E`ƌα_ϊWxα…ό Ύώ―‡–_!υι.κStϋωηŸ7Ί}©/οΈγŽaƒ 6HΨu]Φ›6ΘxŠό0Ή„@B Ήνƒ3fLxδ‘GΒK/½^ύυπΡGyαQ€YqΕΓj«­6άpߎάFqkύ…³gΟφ,²²’[nΉ%<όπΓN―!+3ΦRΐkΤ¨Qa·έv #FŒ¨.΅Ζ=₯ŸhUD»zθ‘pλ­·ΊΌ‘ώ;Βπ'Wˆ§ΑŸχ_ωΚW|=Jι*N­ΚηΑ ·έv›ΛP>ωΟ‡O~ς“U‘?JŸγήnΊι¦€Ό~›mΆ Θ3’λέP·W_}uxςΙ'ΓRK-εk¨IφS½:νΠͺ¨^z)₯&E`Κ”)αόcxꩧΚ.!Lμg?ϋΩpψα‡ϋ᷈mΩ‰€€ * 0uκΤpΑ„x |σ›ί »μ²‹ oR{¬Έ)‰¦@€ΆΜύη?yEεα,γ}χέ7μΏώM­„πζ›o†oΌ1μ΄ΣNaΉε–KcIE΅άΊiχ,ξžpΒ ώ‘ΫoΏ½·γΦύβτe ΦEΰέwί ?ώρΛώ@χΏώυ―‡C9$¬½φΪeΗλN@hΝ3Ο<}τQΟ3+¨οnš(ε|d ίΗsŒ'qζ™g6₯2 ίί}χ…3Ξ8#|κSŸ β¨$—Hτ.θΛΠή("Ÿ~ϊιώ_όβ“2MU)όX<οΉΞ:λΈ2MΡSC`Μ9α±Η ($νΊλΎ8%\@ „@σ! ώΙE”ήΟ>ϋlWN)UυΦ[/tΠAaηwK/½tΝe7(υά~ϋνY@,+’μ,{μ±^άvΨ‘ξ2/ίοΉηžpΞ9η„m·έ6 €‰2Mr „@ο@ ;rφB_ΆΩf›Ή2M!ΏZΎCRλ(πWMχφΫoηe8¬»‘L£q£šω€΄κ‡γJd§œrŠ·Φa’μ§zψ'ešκaΩΤ)‘E‰EΙ“'ϋmC„ΨY­L&Η(.0Ψ` „έNt:κΧΎφ5·ˆΠΤš Χ’Π~™TᘸTcGqK•>ͺW# “Ίλ―Ώ~€Q_c5 Ρ0΅μ¬GΜ€ώε—_ΗΌο.b‘ +5ΝΖψήuΧ]α§?ύ©+Γ½ςΚ+½ΊŽR᫏ΐΰΑƒ=ΡuΧ]ΧwΏU?‡”bB !PؑΎV,₯џΗbΗΨΔNφ?ώΨ•ϋΓΞ=χ\_Ύτ’KΓ:λS“ρ‹1σͺ« {ο½wΨgŸ}\™&.WOοϋυν[l±0wΞά zΦΣ4kŸΕwά„'8Ξ΅Θ#₯™HΤζΒΓ‡Λ/Ώ|@ŽΥΚδΚG –Εχε§Πž!GΞ;οΌπ½ο}/|ϋίχ…φφD"}uB w ΉΠoΌ~ϋΫίΊ" VΩ……¬r-»μ²>ΰƒ »ξ:W˜;πΐ]ΑεΈγŽs₯w₯Uν/Ης Jι“&Mr«ΩτY££ pΜ%p¬§$—Hτ.°„{I'ω<]<΄„Ν5Π=hς‰Ρv:Ηη>χ9η©Ekΰ·ηζζ†~}ϋ;­lΔ—«Μδ­rU³qϊ1½­f)­ϊ#0lΨ0Ο”ρΎΕ₯ϊ—¨urLΚ4­S—]~ DρΥW_ ›oΎy8ΰ€άΜ7D3VL€0σ7%!Π0'ώπΓPΌo‹οζGjla“Ξ_ώς·tΛ‘&8ωu3ι–Ž¦ρ…XŽΗΒqΞ'NIογ°άλ½pŠγΘKl8”Ήžk‘8ς'ΌόυNΟψ%—HTϊ|οΕ_μŠ4+¬°‚+Vyδ‘nΥL9ͺOςΜb-Gίύμg?sK“^x‘σθ«―ΎΊ3Β:ϊ;Wέ«?+­bΟΚ“+acYkŠ―t Η‘#όΕ.N_ay§ψ„ΓΔΟά+\©0„ΓΑCs΄SΦ)?₯•υ/φL<凉ӑ‘wŸ½;·ν¨κΔHA•1 Πά§"ΪπWργ6HdPšylDfA dPŒΚ,&ˆŒŠΐGi‰¨ˆώmμφOL2(„I„€Qrώλ[χύvκξ·ΟΉηΎχ //{½wϞͺV­Z»j­U«VΥξσδ\`ΐΨG6Ζ‘λ䙏3.M·cΗ΄κχχΗžγτKη’&ΈosηΉοΊ–ϋύ½> όυ΄–ύ6X―ΟΏ πΩίίό˜ŒρεΪ³Πΰ| ¦ζžΌΞύ9ί Χώώ^7χ§ξεYŽcΖyr-ύ&4N₯ŸΊ—ςW“§/sκސΏψ8†qϊρυ8}“ΞuʟΊ—τGϋq[zν΅œλwϊFΐΓΊ£–Ρθs"[…ώWυW-*]zωu &Ιqκ<ψSfŸ.Οr/ir`ŽpυtΑόΑ—λUε‡I―€u δ^Γ“>ϋ}ΪUεIz=žΰp\‡£ΟΏqϋ7ωSFl|ή§Ν³ά[W~_Άt>―{λθOžƒ9*?ΈBGϋ£gγ6Σ?οΟ“v2ύ~]5UΦΤ½―σΰvΜυΤΉg«κ›rΏRΌnΔΝ?_¬z‡ΉΟΩα[Φ­ξ·…ο’6m£ήίKΊώy;ίo„υi“&χϋό@šl=ο^kτŸ½‚όdrϊžςάKYπυΧ==}Y ΟHwεyŸ?χϊr¦ΚroUY=ηSψsoŽC₯αPς‡Π™:υυΩ„}Ύ½œΛWV‘kŠχ΄—΄™>ίΤy;Ža7JZΗyƒWϋSFξΩͺrΖωV•%έ 3.nτν³?G—kYϋ±}¬νf³κ3€ΙŸΎ’Ύ“:ζ~pχχΫΞ>@]ξr—ΫΡσΜQYΑΧ—“{IΣη™:οΣηyξ₯>ΉΏ—γnτυ4oŠWžΘΙMSε€^©gΚΟ΅cδΩ|œ90sΰΘΰΐ&²e·ώ» ‡n"€ιν5ׁδ>]ξ­JΏέ)#Η1λπK»*ž€>y>OhΩνZ“?x“o|Zsί±‡άοο9O:γΈΰ O׍λR·1Ξώ:ΈΖεΝΧ3f·ώξοώ}ήΙnv7xΜcΣiƁεγ~yΝk^³νΠ ˆΕξ]Ο}ξsk>ΰ½;ˆ’'²'z<“.ΟΘ8r½N¦τysήΣ°›Μ OR–Όut&ΝΤ1ε«Γ¦}ΉΑΉνςττ'_Ž½Μ½Ύ ηύuςΜΗ™—f€?ε^δ:ΗάοϊSίηΖ}l|\ΉοΨCξ»—΄ύσuηςξE,ώψ1ΡΣΫ_»ΏΏgΙλΘ½<Χ'ι6=††ΰνσε^τΟr.MhMξ‚Γqιϋό‡‚”λ<{kιθθL½Ζω]―Βq8κςŽ–γΌ3ΝΡς&χXΎOeΝs+Oύλ/Ύπ…/,>ϋΩΟΫ—9ΦΎΟ-ΠζkΏφkΫVcΆ΄=£oΪ~QτΊθw;Ω€σ9~δ#iΎΎοm§[ΔϊήχΦΦΦ°mcOΡΆΫ½ώη^ …!n»2ΈΞ:λ¬Ά}›:pͺ+Ο3ƒ„)ξ#ωh «ΕβzΧ»^Z>½ΒΡ―Ύςβ ΪC+žΩΆήdΆ‰mίSε°?ξΈγ”ikΚ€Ÿ*ω>τα->tξ‡ΪΦΝ„έΥv΅ΕVρΗί―xΕp§όκώ‘}¨ρΩ{S_κα(M‡†ώΪ€†:xO>»ή§νεολ<Ξώωη·ΌΚΗ x}Ού„Nhy­Dž*?xφϊ΅'τόγ_hκo h+N΄khΙƒfτZ αݦ͎iIzοηάsΟmρεZΧΊV;ΟsΚ‡SGΌyxϊτ­ξΒwŸ#°}¨φΗϋίώvO~}κψŽoΫβ£Ο €½9ΧVτΩΌ+νf†£“ή?ˆŒρi> τvζ¨=j/ŽΪ=Y’~α8Υ7§ς“ \pA“-d¨ΌΪ\/›τ;}‘>Ё>φχχν3‚κk ΄ιiŸ‡}ύ~ωΒΕ΅―sνΦΦΥΕ–œΚBΏώψMίτM‹­’•p'opZ΅C\pώM‘ΟςϋdήΙΗa€oχόHΩΚ§χΰΥΏΤΑίΊν~Cƒ>N7D6ΰ‡|ς+χκWΏzΘάqLώ¦J& αӟώtKƒd»?²:iw ¨ u‘ΧδυyauYγN•|^ψεj7ΫΊΟσώρ;ς†Iϊqω{|Žt›Ά€|ί4ΖΏ΄;¬ί*9o2£_Υ;…cΥ½π‚Ύ'ΠG(“έHŽϋΛΨ&ιαs.½Όδ yεˆ‡δύr™cκ=Έ€ιrθwΆt―3R9ΪΛ.ψΥ7zΞω 3fYH%«"[š­[αWΏΖ ϋz;ͺd;‰½CξΒΑ¦Š/ƒύBŽNΙvΕΎ$_zΥ«φ99"ί·έΰΫWΏΖΥ›Μ"ϋθ6?ϊΩ{>gΞ@F‘Ωž‘=‘ΩθK]c£Myύ˜Ά›Žmζˆ/π«9GΐUφwΚ’―Ξ>ϋμF£rΩsςρ'·Σi>ΧE&ίϋ{ΑM¦[Έ–W]φovρΑ~°ΡΕζΌ~•sεAΙƒŸΚdS«Oτ©άΨ—xF—τό’Oρί΅ΰύ§ςΡcέ#Ώτή+ъΎγŽ;~±΅u½fΟφο¦―ί|>s`ζΐΑs }\=σΜ3›\$מς”§΄ών9Ω· χɁϋήχΎ‹;έιNM&Ώε-£>Ϋ}‹&ε%CŒq/wωΛ-Ύiί75™Aφπ΅tΉO֍}¦h ΘD΄ςŸ―ˆ!'ε5Φ¦OΘηδSv›”‡|ƒ^Ω*;ε“ΉςDΪu!»jΙ§LŸw!Χ"ΣΙK2,—ΠJǐ•ρa‘ϋd©ό|ζxΆ ΐ5Φ9ξΙOΞ≲Πg\ΔΎ ΝgΠCςEw£GZΊ‘lζγΒ3cuƒk†™3Ά9>›γ˜/«ξKgά¬―κs|ϊœΐEŸΊ&?ϊ>—~ͺ?³ΉΘΝψ2Θ2%ΆWd^ς„&ς€]›•€ΟsxΘ/}ž a³ΖΘ>‹|,ίΰ!+ΡΖ~––Ό μSΟΰ%ΛΩΔ|xφ?6ߌ:°‘αWώn61yE>£μŽL%£ΤKYτ <±ΑSΧMŽ‘[ZtpD―\―lΠk]σΐyχΩΟ5|ΚΤfΩ ½~νΘ?zO‹Q3Ε¨ΞjWnYί?ε"]ήϋήχ^–°mχσ|Μ‚ά/cxYsYlYFτ²„μοΩΟ~vΓχΐ>pY‘KGψσWί<]VgP—YΦgH–?ς#?2€IZΗϋάη>ΛwΌγΛCCζ NBσΛ_ώς†‡~θ‡–gυΛΪΦrYΚΰ€2μΗ~lωηφgΛRjCyΑQŸ΄ρρς…/|αp]ΒΎΏχ½ο¨ͺΚς-oyΛςΗόΗ‡t}έ*’Ώα؍ο₯€—OxΒΟϋόΞ½‡g<γΛμ½NB3κ›ΆKυηu}‡;άaωΊΧ½nYΖώΚό΅Νώ²vΉ˜Μ_“}Λ€OΟjωσ“ςk³|ζ3ŸΉ,%8™_ϋQ~)ΟεΟΤ1Έ΅ί»ένn o}ΓryώΟp$M)γ₯6‰ΧSυwOέΌ§¨ Ε•q0΄ΙϊΌΐΌν’»Ζ£ΰΧ6@ΪήΎρo\ήύξw$­γ]οzΧε›ήτ¦‘|e€œRΠΛϊvqΛ‡ΟοyΟ{ΊJ©΅ϋ΅%λπώ*ΘfωΠ‡ξq[Φ@IΎγ˜²Ιώϊφψ’^Jήώψ“?ω“ΛwΏϋέ;ςφπΣ e䐿V™-εW~eIN‚”ΩŸΛOv–‘{@ώό-ο{ί“šώνεEπœΑω­}άς–·< ―:άψΖ7^ώς/ςdω 5?ή-ΎΐSƒΞ%ΊS>ΫAϋΌη=ο9YΎΌυΙ³ε―ύΪ―-k{ΘοδWυW‡<εLάρ¬]ΤOΚ9§d}"­₯?υΤSY›t59±¬Οͺ-k"gΐ™wˆΪM Τv”άϊ³΄υ­ζΦnΙψδuTnκ^ƒήe­ώ[ήθΉΡŽ4IΟ†ͺ-·7ξ?‘>ΞψJq ν\«Αtk·τ’ϋ«ώΠ’g,AΎώιŸώi#1ψr¬Α{“ιSς>ύβ¦7½iλδE@Όύνoί葃θ’r ύΚx€τόΕ_όΕς§ϊ§—δbpη¨’W E^²Η’ΖsΊιδ“Oξε™c­nc“ΎάFΔ.?‘±γ˟ϋΉŸ[–ΣκόlE㐷½νmƒ]š|hύ™Ÿω™–η~ΰ–ΡΧ8΄ό©Ÿϊ©pυ4OΑόA£ώ”ίΏώλΏΎ€ƒΗyΨͺδη_υ_οRΣωρ́™λ8ώΖfgΖf«‚ΨΜλpL=ϋ?ϋ,O)Ϋ‡Ώaά+Θ|ωτ§?}‘žώό]οzΧςQzT³ηΖ8\Χ€f“C±wϊόεμ^~χ+ϋU―zΥς­o}λςšΧΊf»Ξ8˜έ<ώψN:ι€₯|Ζ8lαq™l3φΏύΫEΎ‡π§>/8€76οi‘[άβ˚Ψ]ώΡύΡςζ7Ώω>ε°[ΩžΡ =/r^“΅ΛΗ>φ±“γυG>ς‘Kc›½θE 7ΩίYς7Β&~ςό/ς/Ίψ8^φ²— Χα^ΚΏ|Α ^°ΌΥ­n5€K}rόαΎΝŸΌΖ17»ΩΝZžθΣΎ­ΰ]½>Νψ*ψαBλο’ωgζΐ́ƒζ@ϊ!ybόΞφe“G Οא4dl}&΄ΝΡ½ΟγΕ/~q“ ·Ύυ­›¬4^Ÿς5πOɟόɎrkB²ι0ς€L‰―θJWΪΆΏΙΝΨτ+HKf¦δΜγψFƒcM*/O;ν΄IΩΖrΞ~?/]υ?ρ€γΧyΝk^3ψOΓ64ί•ςΥ5ršόͺOΤ€GΊuόj©CΚ ›ω-Ύη{Ύη€Όl~Ά-ξmωςS Ϊ©w’ςθ³ΧΏώυΓuτ=9Γ́™q }\Œύω°‡=l˜Ό(εEgΙΓ/˜>ΗΟGφε:6Y0ητ[Ώυ[mN.ιΖGv&ίD·lΚJy―|ε+όl=ηgœqF{f‰Ÿ@Ÿ‹³“_ϊ—.ύϋόΞΩΉ‘'φ\μec~ΟΜσ™ζg ’>GΎœη<η9KΎΜΰo'ݏΉΔSN9eYιδ'c^ϊ’—Άgκ@.ΒΥ‘NΓ/<τцΎώ8ε/I^φn‘Ά||μΪŸί9]g½’ό1ϋO΄•ŸωŸ_Vε8θœϊ€β½ΪηNΑWΒ‘E«‰8³:3 ² ”‘»xΡ‹_άΎΓ*J± α }ΓήpˆŽΝXEMP΅<"ΨαŒrš,^ρŠW΄ΏR‹{ήλžmχ–x?‘Ktϊι§Φ’œ? C Μa)’ππεXhεΦδσ’„l«oψRFgΛSJbQΚcQΞαvν“*v0Ή—™"+Ρ^ΒΏEηIXΑ-‚oJΩ,Κ9ίώΚ‰Ό¨IΩE)šΗ qύόٟύΩ’”wγ{7ΉΙMεΈoсο{ίϋοϋλχ-JX·OΉΤ$E‹ώ+υΪΆ—³κ ”ή’βA·ΈΧ½ξΥή›U@κϊζ7ΏΉύΥ`dQOm5|ή³Lž£xΧεdiuyYmΧ‘Ζ u¨IΓVΏ–>Ρ°žύζoώfΛ_Ζ οίJ+ ^ϋΪΧ.jΠ²ΐλrœ,j’‘εk‰βgy™‹Ϊ―vjΗ [–zW 4‹οψŽοhQήފΆ€O—•²¨ Ι–V[¬IΪΕ;ήρŽφ\ϋπ~Ζ œRH‹wΎσνΡ]ξr—α»’Ϊ‹vPΖΙ’œˆCΦ\΅•’8kΠ4όαˆŒM›“ Ώ@Mr΄νX+` ­|ωj•ΗΎ­}­Ned,*ψ¬΅EιΛpj«ΡИφύ;Ώσ;Χ59U+H~π€φ&ί —Lh3ώξ΄ΡΰΪ™θdηdo Μ9xΕjcwΏΗ=Z»*C±}CϋmόΗM&>υ©O]<δ!iΈδמτρίϋ½ί[TfΓKφ•sΉ₯Ρζυk}+²© ΞΦζε -’΅Ι!τ’‘δ™ΘεΘΩ†xƒ8ε_™Φ―•OV‘7"ΖΙΡηχ»ίύZDyM6:RΕ ‹.CG ~g•ό‰-ϊ]?³’€.ϊO¦5γΊέ‹rΞ7ΪΙ²<.ΗEy―- @IDATΣε°’¬AΚ€γžόδ'ςΡjV:G]ΠZ†λ’ Ϋφ=rΊπΔOά‘Ÿός^θΰΉmšyŒε`iŸbτΛY6Θ<³ΒλAzΠ’&ZΊΧ{gΧͺ<ϋύί½!/orώΠ.ώUΏϋͺΖOιιwΫ=“Iή?Ϊ¬ώ*‡W³~ρ±ιύδ—η`@~νÊϊWϋVIατ―Υ[V\Ρ±tΰ#ρˆΆ2¬&l—ύšmϋ„.°:YZmζ{Ώχ{ΫΚ‘)ϊ¬ΰbο€r”5Y+¨Ϋ―¬ηηΆλΫάζ6‹ €i+΄£šͺέ?λYΟjΫΛ{ψνίώνM±ρ­0"ΩeV6Y={ϊι§/ΚAήϊ…TVZQDΎ[9κxΣ›άtρ—οώΛΦΗαD'ϋŽ>ψΡύQ·°•Η&gΘ:v6ΉMζŽwW ‡ΨΫl<ς„¬"ΡIώ¨S9™Ϊ*1ύΌ_υ•2§ŽαωT“ιθϊJ==cg’ ΖQεkεχψbc²ςnΘ&v0½Dn‘ωi\[mg5¬wΑn­ΙΫΑ^Eη–Bφ‘o/Θ_«Ν<Γ[Ÿ00^9qΏžk‰ηŸ™3.DΆ°aΨPΖϋ ϊmΝ§bg^Ά0…ΎOλίΙYBFς›ŒMΩΦδ;Έ&ςš=fLB>ς”Cwvρ5yfη(χς‰f―²α¬Ζd+Οnƒΐ₯<5š\³ύΩ§l[>²›^ ߍ›Υέ ·ΛrήΤΟ±ΗΫNٚδ•:ƒψ’ψ6j’€π=μHV?:νS eΪXAώrV7;T}ψj·ι:Š>P½BhχΞΨσΖΐ;ͺ Φφbσ[Ιʞɚ–¦-j⢍‹ΤA;0Ά9ϋμsεxouŒNT=μnυeW³KυΖό―…P힟Œ=Όγ‘ό‘± 5]-_ΪӐq>™90sΰ 9`Wφ'ΓΘώۍ£GΩ·dΏ10 ίΉΛ—ΑΆ|«VͺσΡ’K΅x©Ι&γΨZp;ΘίψŠΨ–Ζΐπέπ†7j²ˆωΖoΊZAυ“ω 6*;³ψ4ωχƒ?xσΚsl“Ώμώ6)ϋ“ΌfΣWOσ—³‘ΥΟ νlι^Ÿ)K^ ‘MΖφψλHΧ€aCΛ§.θsNΟ*ypΠY|E΅°έηΗζo1f‰―†o*€©ιθmυ[C-eΩY^υπ΁1 ί0 ?Α,gζŸ™‡Δ}ŽΕφaκw€lδ{±½*θnΑGΚώς±γΛ`+ς)“Χ$ήδH-΄μ)y"kΙΥΘEχAζ?Ι r˜ pγΗ&&ŸΜςKπ-ΓI8αfί¦¬φ`3ηvŽaϋ€ΜuΡ lOΌ0'G–σ »~όβ·ε›'_ϋ±fZώlφΌaψ$θ΄Λ·)DΆρ ‘χd* Ω£lzσζΙψJόρ[¨ίΏΉΥς^΅<‘χ8Ή¨ώζ_"ηkaWΣ9ή?VmvΠόρή-nhhHκ‡­olΒwΜ£τ~|γ*m£μ7šΙύ@p‘αԚGηγΖV|Ρ|3tΛK_ϊ’ͺΟo퐾ƒy’η=οy­ξžeΌTB-y$sΑζ%Œίπ=εσuίώ·o:Μ;­…j‹λT[Ή°ΪŽΆύΏΰ…/XΌφu―mυπ^“wLΗQu]•œα(ζ@"ΓφΊ3 –”l+(kΠLΒΆ]Β*x«3Άϋίυ]ίՎvΝ¨ŽΆ,»E+Z© ¬π/αXU/*άVšΪ½@δ_9i–ό1¬h­‰«–7τ·‹]~’Ά„c++јV•³Ύ•SŽ‚Άzώ―ψΆΛ@˜m₯’τ}YBo¨[96ڎ&v0ιY‚ΎνfPΚ¨Q$βDjŠTΥYN£Ά‹hυrJ΄UZe`·tVleΗ„”WΆBNJήrT΄DmzVž‘οIOz2οορ¨w Π†Ϋσrz,Ρ/zQ}Ρ`%¨©Ω‘G΄©²Γ3»έΤ$π²z‹·C‚χcG 4¨w)£Ά’A]E4¦lΗDβ–S¬Ρ^ŽσΆ’@~ο·lÝΥV¬μUί>»˜ψ }«v¦‘EKΉ΄ϊ—ΑV#ΐoΧ ε‹Ό΅ͺΜ*ΆRΚ- Φ.JΩ…Ρ’xSΞΉΆEή‹g txY™VƒΕaΥ‚τ₯Œ [‘k—₯R°­ξΪΈHdmΎ&΄ΫϋΕΓΧΌζ΅;p£Χͺ]Ο§”!:Χj« π”#¬₯«€«ΆcHΪΫEνϋΛ„΅ώd·ΈAκΡ.ζŸ#ŠήΝΤΞ4»i'€Τ7ωSFΥ°s•|Ϊv­Ρ7΅K+1k0ίϊcC΄5žk{VσDΆΐQΖw‹ΤΧ?ΚθZVPί ΄+mS€u Z~ύθeX·Ύ•Hn+m"GΛ άxυlϊ£ΎPƒ„ΦΆΛ!ΠΚ+CyyN­2ŸΙ¬šT]V Ϋ°ΓHV€ν—»Υ·ŒΟ–¦Œλe9τ[=Θ;OYΥ¬ς΅J\Inε(ΩB&Z)ε܊,zμ‘θ?šυ2­d»ΥδqτŸΎk₯UΚ@o»Qe‡xΚtƒέαΰ―ΐΝ&”ίΣέ@Ά‘ωυΝ.(δ~ΨEŒ¬πη)ŸΜΡ*p¨Υ/ωΙ~2Νϋ―AWΣgdͺΆεύ[fe-ϊ΅Ÿr˜΅¬αwπ¬:βΣΤΞ4αέ/]m΅}Jž*_ύ­ «Α[£ίj ΄zi³’¬ ϋag΅1}εάo+Ύ”E‡¦ ΐ£έyΏή»ByΪ―6‡Vh°qςτ΄Τ_ΰŽ]’/ι{V‚³Ψ€­"]ewšΨκΫ·oόΆΒΑΈ>νζό3sΰ«Δ΄?ϊˆ¬†Ι=}ğώ#ωDv²i²“’Ύcu=^NŽ†O΄CΉ GdE†θŸvI³Γ ώ θ2ηϊWΚr„ƒ]Θ> °ΡΠ­_ΑΑ%_ύiμΰRŽ’–‚LZ³3 9JW”#­Ι‰>?όv׊΄S#Χ7τδ˜ηdΉέ―Π¨cΘrz†,'ίΙA4ZΤΛ1c–θb»υ„GεθoυΔϊ«s"­šŠŒ³Z8«ΪΠι<»ή]v₯cλDϐVSY]‹Ώp‚Τ―]Μ?3fμʁτ²Ιx°ΎM.k vΗnˆ‚‡ m…)?ΫΪn!δj|ž—·Ω δOΏΓqΚ’ΖN#δ΄#ωΞΎE#ωΔ'Αζ±42l‘ƒΌδG±Ϋ :Ψ̐CVγΓiά*Έe…h2Φ­€Žf'EζXέΛ&€”©N ώzAή 2lΆΏg‘%Ύ$> iΘnγ :~G; Ψω&φ―ςAxς‰O~’ρΚxƒ mLΟ^Δ²Ρϋ"Wα§wΠjηA|‘₯]Lόδyt½ cAΆ$ϋ?‡--Yogεα/9MΎΗ¦G?‡vΘqτ¨wxώ₯ϋ₯ΖCΎ>tΒcΜΰZYI‡Tϊάsο²&Ϋ»£―’Oι»-g§Gο“R―v1Μ˜9pH §τE|B_ψβE»tν1_Iππ‡μrΰ>YΘΕΧ@ώΔ­ίσ]9lSΎξ̐•|Πd!8<π³|{͐)žEžf·zͺ·=η#—— t4V 7Θ42‡lΆc–τ*_w•VNMθ6:cC[υO‡ΑaΎBώ½ν@Ž6@ςΟ³‘Տݬ|ΈωΤιΧ€ρ½1=c#“lq>τγ‘1GMΎ7σMύMΚG;}…q>’Ρ„Σp)s–±αό|Ό΄s }ν³Χiψΰψ'Ω—|Α||ϊ©>GζΨ=&vMζΚΨzό„ϊλΨ—AΖκΓδ$»)γτΘΐW½ϊUMF˜›¬ΐφκΠ%?R‡<ΰΝώγe³²Ηω”ΟOΐONόθ%Γα1'Ρ?c»?¦/FΌ£›od™U>] -Ϋ΄ΗAžζ+ |%v»d“…ώΨ…§μίExjL€vwΥ.&~ςœLΞWEΘBσov F#Yοh>ΔΞdτZΡΒ 6>ŸΊρUτ `,’όZή!ώΈkΧp¨[|#ΑΓoMχy―μaώͺ³χ―ΰΰη5—A7 •―zZ€ΛξkΖEό-πj?mΌσρ΅χ―έΐAΧ…gτ}£Žy/όφρ§ΛO—α~£ΡΧeϊΉm8s”vΨ7WύN?³¨—6eμ &€䝴‹£πGΔΠ G1€7 ¦α¬Υ1} 9‚Nηg؞SŽœ ¦±υ―ΙΛLτ·D]:Š¦"ۚΈίύNήΡ9“Φ‘aœ-g9»γδNy}Ϊ©σ€‹ΔΔ-αe’j r Ea¨K”΄ςŽ?iB‡ƒδJ_{₯₯ ‚)ΐN#‡›άδ¦MqN₯γΠ&€₯λvhŽs£’@Ϋδl?™| i,ωύňFƒ€!οŠ%ŒδΛ‘P₯X+Β³mIy∁—0ήo(! iƒy}>…C€nΑ%/έ»*ΠDΡ$Νψγ`Š0 Ύpϊ3Q%–η9ͺ?εΒΑ/Κ+@)z?p0^(9ς“.u0©ΙOΟπR{γΔς)©'Ι—#rL ΰΈΜ@Μs&ƒ4}₯Vg,φ1?»cϊ…ώ†V |L£tϊ±­@₯ωξw£Ού©΄ξΟpρsΐ»Ω$˜FŸaψθΖϊ<£Ψ»ζyΟ‚6*ΚΊF&Φϊ6ίט3•QΜ€ͺhγΑq63&9¦ΐ„ρ„“uqT'-²g,qηΩ&ΗΘ }½"οΫD|Ύ1Φz<κ­?2(9|ϋ~¦oΠ/™ήhβœqo dL‚τ8n p¦©+Ύs–£ι”'Ÿ²γσ@q(3P}ΚiͺώήWο€2ΰκf ―­0έΟ»νιEP6 †τ§ŒΫπ2Ο :bh›δ48Τ¬_-Ώ‰„)θe0G]ήΙ­γόθ™ ¦‘ŽLt‚vο‹α?ށΟLIηΝ uΝΦ³Ϊ‘= m9ͺ³ΊKcP“ϋκύFΎΧJΫ–ό£½hηiy‘!Α4d΅Z`ROK›OΪΜ :΄€ΜΰB«s0N“΄σqζΐWƒileφ '‡ ½³ύwχv$‹ΨHl0ν—Σ˜Œ|€+ψΨ;½₯5€OŸHΪ9a(gμ³ΰ!K8Eΰα°ΚΡν9]ΆͺŸspdrΥΆΑœT€l’— ‘+M^LώΠK’ν›ΘΚΤΑ8Β’yΖɟηΚ#C²xAΊ~<@N¦Ž>—^φω§hv―VΈ:Ξ§x{!ς-’9 Οθ'ΑίgοχKCW%˜†Ομ#tৡ{\ύ9ž˜$U.ΩΩ™JŸzφυeCσ£τ6=θ&ο₯ΕCc—ΰ0>1g;ώŽ>φχζσ™—f€Ol0†τΉψδΖό$ψΐυ[ώ½){U6"»˜ν$mό±Y-ζsŸ―1>yτϋΛzΜΗY΄42Œ-Κ§ Or‚ŸLdχyΆ*˜F@[3‘γ2,ΈΟόjp„Η Θ!_]¦ζμΨ…אgLΓ^tΒ/Nžη¨C³ωΊΪAΈΥΧ§ϊβΣρœmΞodސ^1ΏIς;zτ£Z ΎυsΖΖ ρ―›Ϋ£ƒΖzΊΤ<€q taŸ.6>OΏPJ~ }ΖrpdΑ–w,v£’»Άsn9΅[Ϋ° ΏηΤ\3½ /ύ4ΪE·Η…WyχSyŽ†{ΫϋΰW­g8ϊ9PΉm•U»}bΐvWξeλ¬jΠm;܊4[” h[6}ΝeΏfρ₯Rۊ·A_ ιΓ±R>‹Š°>k °UXuΰφ χ”{ςΙχΫώ<Ρώr³}”έ΄|ξΐvSεXΤδΧ’6Ϊ_)g―Η2΄ε¬mΫRΚ›Ί’mΆ_· z ςΫveΆ΅*!Պ±½—mJω-~τž?ΪΆbσ ”M«?ΊύU@Λπ錊l[εJWjmΨ"LyεHiΫ —m[€—Qέ>#aKΆάνSςΩ6쎡m»mΖ”•-άΰ(ΨΆj«hΚEωT}Ύ,KΫβήφΛ ‚‚₯α―λš hŸj)a½(ΑίΆX.ΗMK‡7 FΫ^Ξ{ΛvqξΛ_J©}6¨œ&m‹΄~ϋ7΄xΥ§#C;[=ΤΝ6fΆ ŸSΏ–ω ΄γΘ΅Ο³ΨΎ­ψήA‹κoΆΑ/G۞Ψ3τ•RoνΔvpΪ~E·­§=WwωkΥΐ°Ms9ΦΪΆ™žƒRrν“1ΞOͺντΚyΤςΉΞ;p^ƒΜΆuh,΅νΤ*h­ρΥ3 -žT°Δβ'ή’Ρ•w£s™c½Ή‡&όΆ^ΐ3۝ں―&±ΪΦΝΪ€†#ŸΪOœmϋYο³oϊ¦φ[“mKΏ X”Ίm₯Xε•Σίj έΆΨυ™½ –δ`jίΆ…%{οxΗ;ΆΫςΥ`½΅#ρ>Bκλιϋy¦]—ήΆ²΅}bί―Λ@lyΰ=@½ϊtνζ?ϊ»v_ŽάΦΎ}ΆbΜ£šΠlνίφ˜HΣxE~z~žX[ύκ/‘ν K]kPΤd™m‘•']_–ϊΰΎΥA“-―yνkνΗΫ j5P+Φ;²…$ΩΧγp:<Ώς•<|> >‰A¦Ωn±œμ­-Θή9G—νΙΊ₯ψmΛLς\½΄Pƒ–Ζ»±Ό λ½2Ϊ{ŽΌG^ϊ¨ŒτΆ-¦:‡9§GΚ ήξ—“«Ρ#o9?ΐ4i_ψ¦‚ž₯c?ΠΪC9ΞΪ–šτ€χρ_oϋ_WΊςΞOμιO5!ήΆc΅5fh/GZkΫπ\ŸζσI10~5ήΆΖ΄­½Ά ΩrŸΎλΑg | Eπ€ΌΟθš(o[iκί‘%όΦαWfήU4dσΌ‹‘d™T|Γ'CV‘£ “?ε^lmm ν9ι΅λψ·­Ωo‘Ÿžχmή6ρ΅b³e#―θH>C₯―ιcτtιΗε(μpv“ONΟSŽ#έ\Ξ•fOV°Π [“†Ύρ‰©rώμȏ}œœͺΰΧ¦—* ΆΩztΨ&ΩFF³ϋΘsη);u©`Ρ&=λυήΊ2ΰφΟΨΗΡ€fΫΧΔkϋŒ–qS9eκ=\gΰχMοΊ2ΎŒ\Λ{ »|κƒd3ΰ»d†™3ΎΊHŸ$§mιΝvΣΏ}~ƒMy‚ͺœ—3΅}v‚mΜΞΌe}Φ”} nπ­7hΫΚ“­ξ²Ά—ΞχνΫ·8±μέ3k¬]Žζa¬ά2ΤyvΑω4{½E?|Ήόd“g‘w“α΅ }ΖΆoFΎ~œrϊ7Τd'ΉΉW ›|šΟΆύ :Τ‡Lϊ―†ϊΜμgΫ§fΓΧrL·mμι8> γ3ηcž°Η}†Κνhμλ·WZk‚ΆαͺɎ–UYα™#{Τφ5Π>_ΒWΣΣ#“tή)}ύ R7Χ΁όι1ηΪNM5‘ηΌΪ>χη<ωγcM΅Ο*Φ„‹[Ν^πi“ήΤΜ?3f4jG©!/@nΕήlpBžjβlπδ»•΄OIΉ7–•tKMl.j·}6Π'σŒq₯ρUc_Ρ&r‘μ‘Ο*θΌΩθ‘mπ©3ϋTύωVΠΑWB§2|jŠ¬GcMX6™άMό4[y}e„§ν~gCσ= ΏΘœ Ω«\χθΑΨΠ7¬²γ»ΝsEDvžXΊ“ mώΑgRΤ•N ΰ_<]žΟ|ΰƒΊε/iηγ́™‡Ξ}Ξxά'α"SΖ}Ž­j^†mμσ7όš‘{‘mŽlD>2Dξ…JφΟ(U Κ`§ΙO† Uωδ Ηχ ό₯ζ<Ρ "?#gkαύΰΗ&CΥ…}¨,ώΐŸkόΟΦL~χΗ.τ)ώ™η»AxΙΛΗΨΧαwd`δ)ŸŽΟ˜ςGψ -{•Ÿ’yθ•“Λ}•rΜχς΅ίλžχZ<οΉΟkΩ|Ύ”HΝ}W«Σέοqχφinφm_/΄hΖτ€ωG~:KZ>_ΐΗ”O͎qΐkž_ωΰθ²1#˜οΎφuκσL…7υ@?Ώuδ΄lόΠΊ9ϊωΌšƒρ,c?iΰvkχšΖ'ο0s )#ΈŽΆγeΆ ΝυYΝECgDžzκ©«ξ"-d’`\„K€€VΰΠ&8αςύ·­r–χt~y{B1LξSΕΉ££χyϋσ©ŽΚ1€LΒΎ(Ž€εά tjΛ³V†ΙΞyHOTTβ‚S"ƒ *v” PRVž9Rj‚=jƒζό1!Œ‚ρ}οϋλ–T}―UA /+4sπ3ŽΥ)2€2{gOZ‚ϊQ?‹& zLΐ™―HΦ†ƒεΠΑ;Ζ~Eu6ZΠΣOX§ϋ„=`ΐϋζŸ€χ°ΰ#πΝCυ;œΰ=iS΅ͺќ^Ο;i΄C ] ’4»χΤΕGVpsθh&#’l(Dƒ+ΰYΪ^i«ΐ»‘˜r½7ΗώŒ‘~π¬2¬Ά'uZ’ϊa€qώepG‡N«ˆβ604Ρ_Q³ν}eb_όΝpΙγ€Α>9± xΗ΅sF3ŽΘΌ΄;νRδFΪ₯λΘηiŸd‡> `2rΪΏφv’δ@E#·Ύ#ΩΐP"gΘ"ŽΧBG/ Σ–₯›’‘ϟσ>{ϊ―Ύθ[Χh}ι―q&ΛOFήϊ6εΘ¨`΄ΠΕ©Ξ±“~ ‡΄αƒSpΠίΙ/Π—•τψΐιF}sU`Šί‚βje@ΛGΦFΏυuŠ¬ί9=cυδΘχ.9bzy:[ΒϊAŸ ?Έΰ>Ž|οΞύLV\ΤΆξ­­!δ3έ ΆFς"|’·ΤΓ ‰l7`d\λΈkU»ΈF£—ΜΙ²!; ?Κ7ΰTβ8Β§πͺœψή%ύ­ηž{ξ ΫρD‘νZ&θ'οBpΡ[d)ΐguθ΅»^»TG4₯ ο`»˜¦Gυ|\„ΘκΰsDΟΈ.œohι†έ»Α·έ`qόqΗmU;8άΊ΄§o>Ÿ9p(Π>Ι.:DΰΘHAΩμkΞ`Žί‰LHΖτ Ά)<Jδb/;ƒ.φ΄2ϋ.2 —ξε:ΟMΘΖYΑώβTˆ|H9ŽϊνΙεP‰³"ύ8iؘϊpœ)'ΟՁ\™ŒΘ³έŽψŠ6A“΅κͺιcŽtv=]ΘnΖώκ‘zζήψθΉΐ±M Τ=ϋZρΫΖδ™T;3zTzΆul`ŽBΆh`‘uΓ;Α―8ΰθ)ό’Ÿ6‘18ηγ́™‡ϊ/Ϋ+°‰ΣΣ#?Œ1nv³›7;Š]ΝvJϊ›ώΐMίϋ}ίΫςΖ†νε#Ή_+N›}κέƒ²Θ;‘%lχΠΠ§œlΆn_ζ˜ύΩώ„μSrXZΆτΑ»9ΐΛq>–ιdύ·|Λ77›/Ξfe¨+_ ύdb3ΟςλBg°Q•Δ³W:αδίƒ‹=]Στ}jάΖiΨϋΥiή­w>· υΣΣο<ΧαMττ‚\ρΞ»φύ½§ΚdœgdFž§|ύš€/Υδ₯1€1ržη_ΞsμΛXuNƐ'ΧΌζΆύ+oŸŸ¬ζΛβ' ΣΩω Oη<φ;Ωy•c_vlζΗ„ MO²›ΙώπSN9ε›Χ€oόuό8Ρ{hιΛuŽ·ρşY©ό ϋφνΘΒsώ¦κ Ζ2~H<ŸΜ˜9pΘΠΏψτΉŒϋ_¬>>KΆW/—B@ό"δ<μΰ±ν•΄λŽlήψ}₯—Ε³ΰ…— qšuψΥΑ_κ9Ÿ<䧱²ŸoSs€ΝΗ—ϊό9ηoΰ?ͺO[hVόκΑ·Β§/Ζ―šl‘©ΡQ|[[Ϋώc²³ψΨΐόΠ΅ϋP{δ^ψuΜb;(‰όbNΩάΌu6ΐ«ς₯ΩζΚO=! .ώ.ϊ³vj=ΠSψάΟ3šKΐ›δKήρτΕ/~qΓ?iω±³ΘJžδ O~₯ΥΖτŒΊ½ω^ΎΪ°ΕτšΉ„ΜΏjw}ΫKώ£ω8ΣΝowT7FηΦΘOΑtV&™ηΌ„ NØ」γWΊtΊώ\Ί8qƒ+G†·ΙD8tΚϊE3&ΗΚA'V6Α£λ¬’'Ή]8&£txΟΠCPsΊŒ# Άi{€Cή8œτ„~θ‚σδΙΡ€„γ’θςvsOxΖΉ£,y•/Π㬳>ΨR»ξlΗ9:B—k“Ύ€2°βˆρŒΆ”ΫΦΌ" WΒRΎ8²Άφm-j«ή”Β)e’P &k«γΑ1yΚ1‘Lј¨τ>(ωj+–—αO£Ν@ nΉ¦Ȑϊδ[“M‘ dΦ©ε¨ο€6ƒTΤ6­θώS‘›φou³™έΑ¦τRέ”Ε(…7ύΔύ)θλΓ)!Bœ|_^XΜ;vgŽ>˜$O’Ÿ|&kΙRΖ¬ϋ1Ό“Ά?Ζ%ƒ”θ,²xWιΣ)£=Ψ㽎υ(£τε·½νmm΅η”l‡m‚; ”oB™ώ£ΟΙq°‚|Θ+;p‘Νœά‚~Θ όD:ΒcxΙ+ΝκQ ·]€ΙδΌiπ;›Ύ–xƒ2O½ι,φύ©mϋsΞYEΎλ = EίP_D&‡ρI›Ν{ΰΔ·r$P2uΐKο@ziκ“~;ψ“²ΰ’Wΰ’+•Γήύg {Tσδ‡3@C—°L ςя½+}’.₯σBηψζ3.&Πc{‘EΧA}*©¬[dUΫG@Ml(ύʟΆΞΆμϋ½sς€¬fŸ Ξ “Ώι{¬ΐ!Η2.X•-ώΐT9τ`dΛς Ί}*Qw.εΰ«•Kl>“•δΛ‹^τ’6ώόNF‘δ>>Ζ&XGKWΜp™rnΙR4δ+|Ϊ―žΦV΄ χŸ|Όδ2zθvΘ\U&ύBFbŸƒUιχ1fΜ8ΜHŸ‹ΑΨX 0›jJ₯xς€M[,,ςΔ?αΚ΅ΎnΗ6ω:ˆ.ώ"φh₯μρ‘]ΏΒψYΙΆ©±χΊz%ύΤ1cc0vZhdׁΏzϊӟjχΩlWώΊmϋ8|oφ―ρM=οΣNžw4ѝκ<θƒŸώ’?ΞϋΤy‹O~β“Ν_}ʟΑωΟΦ%ΣL>…o|nz->γ ώ»ϊΌί€_JzοZΩ[[[Ν§Αޞaζΐ́ΓǁA~JΎ}ŒΜΩ+π+Θ™ΰˆΜμ½ΚΧ”||!‚JΘγ_žκ‘½ΒΆνσΠΥγŒlΔ“Θν>σώ:y§ξεΩψ9KίΩ…†Ξ€λ¬“ θF Ρ.ν―όΚ―ŒQΧθ€―{ψtzό ©k|>Ÿ90sΰπr`θsίψ ƒ Όͺ„uΎ ΆΈ`>Jr2γγUΈVέ'_'ύŒϋ3E‘ ›ϊ!ϊ²Θψ,”οοη?2 cσΗž4Ο’Ο1ΌTFΖΗ;mΟφωϊsώώssΟτ―WπŸ„Ž>-ŒOψLψ<Μ«ΡK ¨LZ~Ψ,>Ν½ώHŸ$8ˆνO.›§1ζ0ΆΚΖέ ΌwοNπ ˜K0§ Œ½vγύx.‘eάc\_}?ηύ˜+mίπƒίŸoΪ»5ΐWg‘€MΜ˜C2—Š—@ΎΌΟΰ?s0ΝΡψVWΤ‰ΐ TLΜ™¨ε€ΠQΖ }|½[gθ κqΡΙ f`³³Α¦`p Ϋ•­ώq0 ΑΆNπΑCΪ9Z{ˆΠBθŸ9χ<Β)ƒŠqΊ\Ppx’‰Ά>°c,Ό“·?†Ό§C}S ΄ƒλ²Η^Ά­’δνˆαΣθγ˜±Š?OΚ0αJ˜ &εώΑ*U΄&W9΅ΔόX5 ”†°|ΒΙΞ c@?%₯]…w8‡ΗϋKV)|VΔQdυmυ†R”kOΏ ύΚD.…Όn4¦‰Γi‡γhΏCŒj»‘Ϋ$8甉+ΩτmΧώ#Ε CόP`δΣ1_YτπΛΏόΛmβlϊτG@nΖ#£΅©u ?ΗψΦ¦Mδύ‚EvΩ9mΪΡ+_ωΚφ§ν™Δ3ΙG6˜$“'τ¬*;v^©ο>―JΦ’ΚmŸHύBλͺrΤ' d{ςΑ“~ υ: e8β] Ιέx'=\δή9f’ΦσUΊD~C_ήΌ/}ιK·mψ œ‚μDΊΏπ…/\˜΄&Ϋ­DΞjdςB‡­.Ι εΙηέ>³Βx'oϊ8ȁ: ςθί?ός¦-α!όΘ΅ŠΝ'”Θy<ϊΗ:ΌOΓ©obί§ΙI“ψΪ4:νHaΕ9%¦}hx·Ψ!Ϋχg„'ν©η^Ή6™oε]]jRΓΠ—­Ž°cϊdf?ϊω0sΰˆαω©M―Ά”MΑpϊ£UHvΔ΄ΣcϊόΡ+ΞCΏsΰέ­«Ηvͺι_s€|<ζl:ίΌ?ΐΞ°Ή…Ή~(cΚΤkγΡqw¦9:ήγžk‘Ξε˜σ‰ΖŸ0υΌOKΰ­K£Σ voyΤ£Υ aiΒά§^Lt›δpΝ)φάη>·Mς ΈQeω½―'>ρ‰ν>G ωmγO©™ 1)kΣ*&ΰ坒3τ:^fΉν1“—²1H; €Δ䩉IΒέωu―ϋŸ*Pθ7[@Χͺm”E|š€€$¬°ΫŽ‹•&€vŒWκœw‡u4Ρρπ‡?Ό)˜Uu“ΟŸw g0τΕύŽEψ‚»Ϊύ䝫£ΰ%»α5ž 2©b₯Ά†Πσδ'?Ή­ς[ES‡~>=Β8°ͺ 3mA{ρΧƒ|Ή§½%mŸ¦?χ<ŽΎύΙΛ@PcrN?Žl°€Uζœ3ΟxΖ3ڟ‰>;'mb¬)Ηd*ΉΓθMωθςŒALNμ5*]ώΎ>«hYΗ[8δSM@ΊΎα;όtΦ¦|IŸk:Ξκ¦­­­&7r?ιS_G΄}χ²₯ζΓφ°ςΒ*aςΒ»8ωΰ?ΈΙq²)ΈΔ'Ÿ|r3’ ­Z%Ϋε(ψΠ`ΕΞ/d§ ͺU<M›ν”tϊŒŸς0ΦOΈφ ‹Ž?‘­\ Ι@Ÿ;[ψa5Α4&y΄cV―|ΓΧsψΑ˜ΗxαάΕίρsy€Ρ†Γ7mz ¦ςJη>άΚxμcΫϊ…*œqv|0ΨI“λvα ‹­ Pζ*ΌS4Μχf|58°Mj³€žnyάγΧΩ›ύ7½ ΤΩbΡ‚œυ]A€ώ¬^ϊΟω†‹Ώ(ytςύΆwέlˆφπ:CΣΊ¬I»*Ν&8δέ Οπκο‚θτ ŸZΌύοoΆ•^0ύl'26:™άqφΧπ£ΙΉΟ½&8ί"v1Hš>_΍σρΠ5ϋγέ«’yzΡ128m>Γ́™?ΨBd0yΚ™<%Ÿτ6Š?ηl ­:Οxζ3ύη•΄Lώ°Ώ,δ²r‘]Τ§>u;Ν„|Hζέμσ€Ϋq€RΦΰά‘φ+|Ρ;­ 2ζiδ©cψΞχ ϋ]"-ΫΈ 7SΏƒ τΘsΟθΎ Ί”έoαΡΎ}ϋšyόγΏ«γŽUΐGtꩧΆέήΘύU΄y¦M™ˆOdΞωώ́™{γ?(Ώ'cΌk™‰Ώέ0EvXao. ‡ŸyίΦΦYυο©>ή'ΤΧ#Ϋχβιq¬;ί„†uωεYxe•Ώά"(rΝξαw»Ϋέo’fͺ“¦ds³‘+Αͺ΄x-nMΌΟ.N>(†™—6μ&ϋ"¨ παY„_†9(siv"g{mΉ―^tΡ*ω–zGΞεz/GeΡs7ΉΙMΪ—7Ό›UεeL“ΉηM…zzΰ½ΗvΎΎM(g7ΈπΒm²tαz|κp(kτeιͺώ‡ΚD?Ήqά£ύθζ›6οψ·σ·‹χώΥφΧ?ήϊΦ·ΆωRcΙΗ=ξηΛVΈθ³Α‡Rώ‘œwσYž#Ή3m{ζΐ&q²3N”ΤGŽ „a°›θgΜ[q*ΊοΒϊξtϊ?£4ŒΎŒώœS˜0aΈPV"Γ9‘‚ ?¦κ†‚Œd ΣG&’q!Κ•h‚ѝΐ΅H8Β۞Α7uΟ³^`sxωσΩ τΪ₯ζάZuj΅Ω›ήό¦ΕUΏρͺmr“05αgπŠ˜4ΡαSέ$­ό&.νŒ€Η‚<Σx§x0~o‘uέ±os‚}p™`pγDϋλιK€~κ›2B?‡ŸΟ˜Ά{Ί­vΦ~Α‰'žxΐpXz/š|’ΖNE«ΪkΚJΩ{=φω•γΟJ?ωή“?Ϋ jpίVlŒ }ή½–;§Ώx8°¬{ŸdSδ“@/ƒnA`«@ϋ±5!/‘Οι+δ §Έ?»K‘ ‚h΄7Asς ξ"τΑu ^ŒC;šΨνi¨cκٟ'j?Ο’?ςχό Ξo[Μ»O.Σ%ύͺ#ΩΓxUY}^²;ς;2³ƒ1ˆϋΌ9—Ÿςθ…OρςνUΆχυάασάϋγύ‘‚νfχŽuA5ή}ςΣS[ε΄ρηSHœEθLεέ τ «τ 5§έ‘Kβ‡r­°k="`Ζd0ύt•―ΏJ[ι›φ‰υ<ΪΎΪΡ‹ž`όΣ&δε‘›€@Rο¬οsΡ›τχ|„_yΞΦ±"ySH}Ό{<τη;Μxίi^ϊΧ·vA?™τ樣{zΊ7-oN7sΰββ@ϊŠ6ίZ‰nI[Φή?Hσ‡<€Ω5ΙΖ^F}ΰ«Uε`&γβt(kŸKΫΑk~ϋΰ­ΚO»ρ½Oε°ΞŸΊΉ₯sχΟ=ZhksΑ aٝfέόΞΖ…α η`š#ό}₯Θλ;Ε‘–18œΕΑΙJ90’)yz+Χ„G‹x’ίαZ7™²"πrσœξ=€,Α-&§š²ΒͺέXγ™έ„TV|ZOh·έtΘ«.Κ£8¬‰BužQ.hζ”"t{ƒŸΒW+Ύ8¬ΰ‚?”‘o †ΞO„žό³ 8@x7΄χΕ‰"?Ϊ8Ηύ™ΔΔ#“w-œ0ς(Χ»“/|Δηώ¨pϊ?ιIOj‚Χ–œτ (jβ‡bΛ§>RΌ³’Φ§/zH»0Q-J δr?iΓcΚ΅YƏQΒΰwΌΓ0ιώβ#ch’œ‡ iΪΕώJξάR<ΪΈϊ@mΎƒC~mΝΡYYΚ‘tMhψ%  ΅3kΕςνoϋKΝ–kαΥ₯ρ˜6§}i»€ΓˆΣVΞΓ›\{nΗ²ΜŒ ΦΘς’l€Χ ]»”αO‚vζ[šπθ7SΑ4γ>‡Έvsτl꧟Πί.ڍϊ)ωϋ™O¦υG%9>–όCϊΡIπq4₯\»ƒΨQ OςΌΟ&]θa8λ—€A‡•‰NA!ϋφνہ#ψΘζη?ωm2”ϊΐ>°Ρ­Ÿ;ΒŠ[Ε+2νCϊP›€ ‡Ιvr^rΟ±φ6@^p¨ϋ#Ÿ–X Α@\γ=j όή?:ΙE<@ƒgώμyžω4™rΘNA »ρΊUl͏Y¬’&„ρDϋλοΡ£’δΨ§s.ΘΛ{Τ>ΙjŽώθδ[ίϊV‹•Γ ή%žΠ‘ψ˜ͺ›w@/*?ο@ϋΫL0}ςΌO.>χΩΟ5y~½­λ±”ZΫΥ'ύα5ž'ΐΨΐŎ9Ρε›–5§›9pqs@?яταή±Bn¦Ωy… _μŽvŸϋή§}Ϊ¨§vs‚ιΨz«d@Ÿ―?…vψ„^z'Ω}έ²Št*ΟαΌG–“slm²<tδΌIk&ΒΩΪ?χs?·ψ†ZD–ΫiOWρΔ}2”ό²£ '½EΞ<τ‘mγ€uAψ@Η’ϋόθ φAψ5ζΐ@ϊ…,Ηw²r†™3.\ϋ„‹|$δϋs•έBV“³Ζύ&Ψ€μ#23ΥΖ yΜφ€A_#yΙuςˆ¬`“»wX‘l#π†\š1=6%ρ„-JNvžC’O³ #Ψe=}τiό%Ζ~ξ­„ξ‘ttIόnΖ<δ½`4„Ž—I>ω΄)NϊŒΧϊtσώ™{γ€ώ¦OςSZQ.˜†Lzνk_ΫμV6$y£οMAž±©ίψΖ76[―Γξ°Y($_δƒnΩΉ7ξοπ#³e}J.„-Γ%π'„‘{δ¦Ε.|―μβΛon^…—SΥ$σμRαΣΫτΏήΐ;ζ‘ό|Ζt―w8ΫΠS\οΝΈψ9ώΛφϊ»χοA|­'Χnίc;Gχgœp}I‡π€_ΐΑ4Ι°±― 2οΌOΧ|?|Ξ|{ήχΞ¦³ψΊAhιρΡYΖ0δ)YνoœΞ|ρωυIλ±: xΥ% ο·φΟ•(Γ8ΘKΆ―ƒ/Χ4τ£ρΘ)»8σ9v)_ηcαGφ‰n6>±Ρ|(ν/ψφ΅KΊ‰ξφήάv½σΗΔχΘG>²ω°})ƒ\§Λ’KψΟ΄υt ―ΤLώΕρ!‚@g6(ηLP<Χ±σ—τΎα,€SψνoϋP‘ƒ„'Γ^ήqy Ϊρ]oΑ'sα™ ¬u'%hrΐŒ`χ Jσ<œGŒl“˜„ a›˜ΘpΔ‘Οπγΰ¨ζΔ7ιJ°©< h"{Ÿί9ΐN ό9Op3ώ9ΠέΓϋLH$Ώ£t&‰š$ςR>€^xΟ¨έ €GWςS‚ρΚ€…„G{UŽ­°ΡΟχ‚Αασ]νe wxη((ί¨TGJ-χG( :I<“άόf7oNΖΤΡ=ACωl”wl’%ΟΓƒΠ"½]<πΒ.:§«θΆ‡€C a8τ‘qΫVwŽPNΝŸϊ’e†K ŒV`BRπˆ|rMΕ‰B>kη&M’1ήt/ωΛ[ ]98EAΪ΅£?Ž‘ γΎ­‚ΫΕAώpμrΤ2vp†žΤ‡1I–}a*°g·β?ώΈ”ΖωM/₯ξΚȟr•O—Όα o‚[PeΐΘΜg,τΥ|ͺHΩΑα8GΘvΑδΌχ§ [e˜ξοώnK#m__η@~Ρχ&2³ΚUA^=”ο]™Œ₯§ΙO²Β_δ»o£ͺƒg|B€η·²εΗ#Ξ5:I[p@t!\Ϊ2~*/΄+_ϋ5° λψ }2e8FτχrO *d‚`τsχΤ!Χt%9-°=8=JAp“εV“η«d9½!ΐUšΟΦ 'ΊΧAκ hœό’_ΪΘψ/‰Ζο189yΨΖ{v6cƒ>½`@VΡ)ψMNφi[‚ωgζΐ́ƒζ@d}›<ί-6φ›™dM`xμy#OΙYŸ εΓ0ΐ&l0ŽdΧΖΨlΦδΛ1c ιΨvδΕΡl»π-έφΆ·mφ°έnΈ1ΎDφΪ±—όe?»•ο0“4& ŒΐΨ¦F“6@ǐλ»ωRΈB»φj ’nͺMο ΕψƒN5q‚«]Μ?3fψωjΩΫv-΄ƒ‘@Ά^ιsύŸgδ±•ζ‚ŸυMš©>κ3£ϊy7ϊΓψ9Ύ>₯ŒΩ{οψΪ½#2Wz}FϊήχΎw“γδ?€Ήυ—¦ηmΟ;| kωΩωάL„‚>}xȞ7YΚ†ζΧIΪ#‘?3M3flχγψ2πƒlqΚφbC³γ;ξεΔ%—τ=Μ‡ςϋ€^6ΰoή7Νvά·oί0ώhΦόD .Œ ͺŒχΌη=-W_NΟW6ͺwb^Ž―>4Θ${™lŸςαG°c³πޜ²9ΐ?b|άΪ:ΘσΘχΘφ~δ£Νe`μΝΜcυΐWB²D_Ÿΰ°pΧξΒζx©_ξς—kε΄Μ‡πcn=|Kρ›σλ€Ϊ²ΉpsAή3_”ωGχv˜ƒiŽφ7|1ΦO£,`0 6NΤ> † π.!0Ÿ½"Iš½TΗδ£4+qR!iΟ§¬4η4·±θA4ο8ξzΧ»Ά€‚‹°%8RVŽ„,'3˜ ¬7ΚTs«[mOπ™π~υ«_έ’5Ρ™όŽ’8 ~8ΌN<ρΔζ0G+`p³UΞ+“±‘‘Οο\]}ΏUz€?—=φ²-„Γΐ!ΰ„σ$ωM`pΚ[΅αQρ”ώΙk’$Ώz‚s?tξ°Β˜r<‘τθ’dLοrΰS@™α:Dƒj{&Y<σΞ3Yά2ԏτyvqΪ€: σω¦(6iAΪ‹6N˜DΑ ^Π¦gαAh1θyΩΛ^ΦMςkγΞ^@ί0ΡaΒΓJ@N―ΰw ΏMRΰ‡Άnf"z†K.΅I“‹υdk/{q"ν˜‘D6hδ‹` πO_&νΙ–8_δΥΎύ‘idC $Ϊ£> Ȑτ»”έμα‡qΖΠ$Ώ9xzY'ƒ’ψ%/yIΓzϋίΏρ`Σ"BΧρǟPwj»Ιθ·t ΉΡχiηœΔ&VmŸ.˜ΒηΤ²*S™ΣdΗlhBTωƒƒ\Šœ‘'»k™δUΣ!tΜ ς[i& `ΈηΠ/ΚΗ/z e;F^.‰ $" €,τ-Tο0ŸOέ‘­δͺ γτΝ0s`ζΐαα@Ζ“}ίλeDžΟOκƒvΨb²-ΘcA=ημ/;ς©ΐcάkόΘVηl?ςη£ϋhKŽdχ™΅ΰηyΟ{^σCd~¨vβααάαΓ‚7κ„''–―ζZΧΊf“‹d*;:6¦tlU'>ΗLvf|tψ¨Ή“ς’?e‘ΠXŸ tβ§0^4c›΄5ν"ΰ>PϊρSŸωT[ː>{'0IDAT`βΔ81νA:η&φιncT`μ™±Gp΅σΌ™ΕύˆΜαsεα §Ÿς”§,~γ7~£6J3υΗΆζί8ε”SZεK}άγΧςGŽυD±ωšΘ4œϊ:ϋΠ$Ÿ…P€<»΄υ–¬ˆΜ(οHƒ =j$‘Q½~φgΆρθ9/ —{αIŽž“³lhό¦ιŠ,ΰh[ ›l"T8¨M†ι|œΐ <γάΛ5Αb‚“ƒΨ6‹Ά.Φ©Eν‰B<£&69ŽξwΏϋ΅UωpG ;ΣΑωZέΚ†6Ρψπ‡?Ό)>•Η>φ±‹sΟ9wρC·ό‘π`’S€ŒΟŸ¨αησ &ΉCΌ†0%4oΣ&t+9:ξr—»4EΒE8yΖΓαυμg?» 58 ŠπΤ'Ÿχ A),ŠΑ€>Pπ$4ΐaχAA&ΥM"ΐG)xΏ")9ΑπΜ$€Υ±&Ηǝο|ηΘcXύ Ψ<pƒ(G+|Ž)ŸBI;iΘΦό$]Ž’ ’1h‘l9oπN}•Νp!ΠΥέ`‚I}΄Ή8­ςΰΛ9€'œJήš)₯LŽ††)L<”ΧΉ6—οaβιΰg£νV‘«μϋuΆ;†”₯?Y=μ=[ςŒg<£mΥμέ_ΉΪΰgŠ^eyiΓvcΐ«Ύ}ρΟΧ?ςŽ•’ΰ1ιfe §M‘œϊ>'γ!ΛΠ0Ζΐg€1&ΣV€% Θ ύ‹“C{bP)Gίm­ΓαS ε€r3ˆHy&Ι²cͺ­'οΤΡΐBίedΫ-…C^π›@ ψM¨2ΒΡfρδ“NnΗΤΞπ'Ηq9r ²δ²ƒ£Ÿ,ΠΗΙtγ£RέΙ ;θ›ϊΉ- gΚ³žυ¬ΕΙ΅ŠW9₯œΫ-ΗD±`8“·δ;όδF>|κ$xΥ‘ΌυŽ)[6>›§ΏΓ―­Ϋ«ΗVήή‰mχΙ4?τ!—Αύλύ“œ dΉeζxŸψlΥ(cέ=όΖ?2R |Š)€vcƒŸΌ‡8η“…lχΙmυ3˜1ΐΠ~’W8–Ψκ‚η ƒ―ΰužψ§>Iu…Λ_‘½Γϋάη>CiKT?ςβŸw€_&Χ΅υη>χΉmq§;ήiqύo½~KGίjδ»|hΤNπ;ν xυ…Π’{) Ο½[N8eŸzκ©ΝVΤ™^¦ΏN™θ`Ώθί "κλάσqζΐW“Ϊ 6OΏ˜¬Λg8΅ηΦξ«:g§™HcW’_dγόΗ—Ϋ'†΄υτ: °±‹ληlyŽ&²KΠ°ΎΙ¦”/}€žκAίJ·Οφ‘§Ρ” 'YΝ%λΙTφyx›[ίfqλ^§MΨίδϋh·u*P%4τεχηyžclκ<ιΨϊτ7}NΆ°ύξ{ίϋ6yK—’ΝκF—D–sΆ'˜ξΰκυ.=ζ¬d<Ός5τ]›<=mήy,8‘ρNΰπιο—­Ξϋ±Ε­ϋηoλfmƒžN°ζT}yσώ™ΣΠwτGς=Οϋ₯RstNηΨήέΠηΰΘiv{…ϞHΑ`cΌΙf(.˜ϊΕ/~q»“{˜D¦ς5Xgπsžύœ&Mά²ΩΩ‰ό ‚F²?ΙpΆCξ"›v“ yŽώ±M\9ͺω˜<ΉŸ#VHVΒͺ|Ι0¦9ι- zΦ³~©ΙBrΡφηY˜saρε+½f|Εv§W.ΣτΪ$”Ώ Mψ”ε]x7‚’θ@ς―yΝkšα£I½β/ nΈΨΖ|"ρεΛB-Έψ~ŒΟθSmΔΈƒύl,ΉU68^πƒLUž5ιΣμιyx˜2ηγ́™ǁτ%}” Λo­Ορ#²k;Ϋωpυyφ£ώ)ψΩ؞ίyvD^ω|z˜¬γ%ψ­υuώ%v»έsΗxΠ[• ΞΏ›>Ύ’Τγr—»όTρν^dΦͺlώM‘ΧKΙrtΖKzΨ}>cγ2ΆO<~}Ων7*ΪψδΔ Ότ>ψ&δ‘―,.ΰΑCοΔd,_”ττ(Ώ 9 —z+'~”©2Sφ|œ90s`š‘{ι?λςΔ–ωΞκΗΐά’±―~ͺΟ“mζ€ψ2ψυο,V—ž,θ!t5_bυχςŒάΚyΌ?Νύ=ηςΕ^^eGNσφΧΑίΫζxAϘ#γ_gw?κQj>j>}e›#cσe} ιq―;O½•c>’„έλΣΧ>a͟Jf’―tŸωY!lz/|yg­œ²₯5Η†.r–­ϊνu}aΥ‡ύkΜd—ή"“ϋ€|0Ÿkle>ό­ςŒΉŒŒ9J‹„ω^ψ`δQZ€ςωXθn8Ÿπ„'4=ŠGlqs ρCςωga΄όαΛnϊQΪ)`;ΰ%[ήEB9–%—%¨†<ε@^ΦΰdYƒ…αž2Kΐϋ”YŠ΄½›ΠVΞ©εοxΗeE4ξΐQ‚xY;μ ©„ﲄπŽt₯Z}k—ƒχK™,Λ16δOω΅Ή,Α>€Ε§Ϊ™aY©>‘«œC˚ˆς;©ΑΡ²ΆH[Α'ΛrΖ-K)μΈ_“ΏίΠV“†Λ;άα ϊk"΅•ο§„ΛrN΅ge€΄£2Ρ]Js(»”χ²&\–₯°Ϋ½7ΏιΝ GΚp‘σrπ-Ρ©εplΗR”€οσΤΐiYJy(OήrX΅6^“Σ;ξ?ϊя^–c©α+C©΅ω|Άtήy)οΚ }ϊσ}ο{œ·ύ‘Ϋ.οq»ПjΥΧRϋΙί.ζŸ#ŠήMMΖﴍΎƒ}gΙGV”q>ΰΥ.ΛΑΡϊ‡£λόUPΝ²œ±CΉΑQ’˚8Υg“oδiXΓ}2½Œ©ς—C¦₯)£«₯/‡ikλ›κ‘τ‘2—e˜-ΛΑΏ$»Λyίπ–‘άd]9…ZΤ©&Y‡wœΊΤ$nΛλyML.ˁ<Π;$“€―@“eλ xΛψoΧds ‚†ϋ€ξΑδcΤ i•ψGž†e /+Xb 'ωαλ2Bώ}ϋφ ωΙ›šMρCΘHuM9ŽεμiςyŒ·œ]K<κ‘&Qvδ­AΚπώ―:UΠθ@cΥΉw«έΑQ‘οΛl IΡ‘g΅sDK£Ν²Пr½‹š@ξΥ bY“AΠ~V0mΛ[ΑŒνX‘‘ΜU'5΄dS€LοžN’#―sέλχιœre§Μδ ijP5ζωyη}r©ίχ|$wρΓ±Ώ―n½­Œd£4–γ§Sžνύ\A²ΛΘΕΧΤ9»Ά&ΙRΩ΄lΫ>m^`‹{ΞΦagΟ0s`ζΐή9yΐ^7ΆΥ§*Θ{GίλϋαΤ9{DNΊfKυi3–fsηΎ1tνώ8Z*π|šm_zΨ¨pφ~rψQϋeXM,,Λ‰=ΰͺIΧA†W`ωŽρȐ¨NκΣΣƒMmά_ΑνqΖ .€kΡt„gIΗο‘zUP»GC>cη<«…HνYςυιΨΌloiωA)ίXΎv¬pgŽεάnΎ‰ κniπΛΏ/#8ΗΗ”Α'|λτOML ~†šPhyΘq6uΖSπΤ§Ό–5ιΎδGrΝ·‘²Πΐ§@ΗzFŸΚKΦΧdÎtΖtJh«€―f3«cMΨ χΩΡi}9γϊΞΧ3f:τ΅±O€/ǘVί¬4CίΤwkβ³μσ3‡Ύ=ξ£d…tς|ςΙΛ ”lΧόΐ|α|F‘χΈΗ=–5yΨ*1Ζ“q2[”L%›*0{YA†-}ζΰŠ―r9>ΫΪ‘`yφ9gοΐŸrδΙΈΔx‘—ηα*ΏΓiΏzΪ@oτώΎ¬ΙΜφ ―"§+ °έC7·© Νn―Εͺ)Ίμι,ρˆG e«glhc–πΠΡ$ΎšΤƒΜ8‰Ο‰žœaζΐ́υˆ|0XΑ ­Ÿ‘™;™Κ<|Δ±½ΩΉŸ<ΉfVpGΓ&ΠΟƒςe°·"ƒΝ[υces’‘½/Q9ό yV‹ͺ )ίEΞ=‹ύgΞ΄V‹#Ζι ς%ΎeΤ"¨φ¬ ώsΞ9gω|qΐœ%ŸKtFθ+¨piΎΟ}σ˜ ŠJ}Ήγσ€α?ͺE¨C}ΰ"›Ω»cŸ?ϋ€ΎΚ¬’ ΅0uΐΕ;ž_υώΗσΧ5³άpβyο£ΉΪΥ―Φζθ‘ 8π3E·΅Œυ“ϊ˜―¨…^CZυα §cιžπΠ\‚w0Ζ3μ9Œ³@π:OkΡΣ€Ηϋ}φΩmL—rΝEhΓτnΏ}8’£ψgή™¦ήό₯D±Y9d• m’1x(uy&ΪΟͺρ©θmΈEΒUi‘—₯˜Ϊj»Βˆΰ³{†Υ‘ΐΚwњ%DΪ pΛ—HΊ–h?%D[΄£θG‘‚o―UW"βν¨PΒ«}Β£ -?e9Q~ε°oΡ‚"4Wτ‰:T'[³ΩΙ@„£ϊά­~>bεΦΈnα‘`V€‹hΣ›ήΤ’THI<ͺAΘ’”ξ°λ@ςŠό+aΦvψΑ_Ϋ™‰r­SΛ/rQ”¦zϋ„ˆ¨ΧΤΉ%¨«“¬"iEj βvνΉΆƒ[Χ*έ;ίΉVδο_-œς­P}θCΪ>c%Ώˆω| Q~νF-ύ"qΗεK· πΨΚν νipΈ_ ¬αΗ/+ΰD€|³ ’z—βl>cUέ|\άόoΎƒ–ΤGΤ­hžώτ§7>Ψ-’œ ηψ'y΄ΡͺψμX…g…†]ˆμΤΖΡbU^ΟΡ’ξ‰ΞΥGSΗΎΌ”ε“ΟzΦ37ΏωΝΪͺn‘ΌoωΫu–ή»τWCkΪG_Vs>?r8 J[_υΎDϊ €­ΐiΎ`eΥ>d•Ο#‘md’ΆbeY₯νΫ YHιΫώΎ8λƒg΅hτ‘§dJ9=šLM€uί~m·hχ;΄Xρj—.+•ΚiΠΪϊ^κ©οκWεμm2ϊŒΪ-KΏΧηΘ‰ Ά…]؁μD²%ciς·&υΪΨ•έE†RžώmΗ<;c±ΣΨψμr@&ؐˆVvμ»j>ε°₯ڌoΩΚύXΏ=μ~.³ΈLΫ‹ίΘx9²,΄tIۘΖxήκ쀓ηdΩ€ξc»Œξ!Kω’b—&_τŒΝ?\πΥ=c°ίψ»gώι;ψ_h+‘Ιfε²_ωΡβ+p•μ ξρ ψVt!΄°kΉΡλέ›Πs€Šνɞ'Λίt-ΎX}Λ‡%?YίvΨαoa§:ͺ?~y½{™βƒc7σ3ο\™lqvσΆώ?4Ώ_C<Μ˜9°’‘υvtεSf?ί“ΓΣ’©v7η·%χΩkVσω§δ¬gππcŸ\»οVΰKσE““ιμtΆygΌ‚'8•i^«χε#­β―ς–ž,₯θΌΘόφ ~Œ ψ?θ*Ÿ…rRΤu• MWΊς•šN§¦χύ3nQŽρ|δ™Ό‰ -Ÿ]ΔΘΌΘzrσ„½8΅v€₯―ΘΜή†ζΟ GΩΠόύd-ΪΒ;υu­~ήَήfΜ،ϊΗ~ω΄.·’ΕζΜ•(³ΗL.ΫΑΖ\=K‡yΓΜΡςλF?Άκ'υ‘χμψ–Ήvy|ωΖΖWζΨπτSθ_έΌ;Έ£·‚Ώ?β3½ΧΏγ”―]ΩΡ2v½Ο>f.Β;5‘£Œ!ιCο=4τemη—©Jn{ϊΆšΝυΩΑΖΓΝΰ˜€Υ©zαΆ#ρ†ν&xΰ‘$Φ ΊΎ3ω<GΆΑ· ]ωtnθ}ϊ Ιςψ€ 8‚…Β"δ(’Š¨kΞs)!A0κθβψ…‚q•Ρ=¦•b2‰WΡκΝ©A`Q€›Τ-ufPΫκƒ `C±dr0iΫΓϊι―½υυ u ε'D Oίnt?κn[5ωM6HK€+Ÿ€zΟ=>Ξ1Ž:h‘žPΗ‡UτwΕOžVtύβί>Ώέ~)sνM½@ΚβΦ8π΄M‚ί9ψΣπ}Žή½ϋc₯|”;η‘ ZJ˜‚Κ³)"ϋgκm¨{Κ@‡w°Šς3&ΤCzιRΗqy)K=lΝμ]ygςj«“ΚΣvAρΜΧGΘCΐ»Σ†ϊwOφκ—Άχuή·•ΘΈ>ύΈlς-²E?’–L‰lXgΌk›&8Ι5ν–£\°KΪθΈ¬ώ:ύ•c§oΩ S›Ύ.πO½”²fοΠό―ŸϋΧΕ|y{ky2`U?CCψaBφӟϊτ —ιTƒ}M]Vι’‡s}œ\@/ž\ε*τΡυ†‰Υ”'m χπNψHρπSuD9ώOΙf8’ί‘ΌP>=₯||"—ΘθL4$}ΚΟqΠ-τ‘Εηνσ ―Ί§ό&KY”{τίΔ–πΌΤ£§γSŸ:―&g>άδά1Η³ΈΪU―ΆƒχhμρΠtEΑG&sΞΨ<⑏X<υ)Om|Μσ>ΟψΟ½ƒθgyθGν wΌσιΧϊœ:βσxβ&ι{ΠΙV_εΗί½θςΰœ3ΎZ`?κ‡Ϊρ: /ύ飫d΅όιd™ΕΎγδ!λΩπd½ώNΎ’ξ©q‰>(“џ؈œ>dNΚR±]ΗώSžηqDLΙI²T…C}Vι5ΟιΞƒ‹…FG²œlΐuήD–‡?ψ‡?x@―{g{y―όu…γ˜ν€ Π:3ψάh+»τ:υŽ;Ύμ+Ξvι^y=§Ÿ90Ε}NίΥΗφ ½­%oϊ―c³3k\Ι±m§^₯ΩyΧΌζ5Jξ΄«Ζε’ƒϊΎ?rN9±cχ4™σΕnΛΛΘTφε8λΛ‰½GΦ+cΥψoΆu…%#―ΈC6£½τΗΧ^©Ζ\ΜθΗcSφd©#hV?iWZΰ₯Ο”Ifϋ#/O h›>•j!ρEήΙ*|ΉŸ:Βk,·N(­d3}ͺlΌζ3 γ£€K½ΤOž)ΛKŸj'Ρ§cΠ†€‰>υ\`Žv±ΫψiŒkΎž90sΰΠ8ΠΛςθ3Ÿ-?bλΩΊžιλμ]}3²ΉΟ3.έη/Θ-`‚Οd»τάϊμ;>qΖ―δυ:\d™I:~M@Nπ‘“½ΜΫχ±mΩ₯Κs9$ςαKέΖιŒπeJ·|ώ_kŒρο_jς.64߈<{…ΨαΚ=_Ό‹}τc‹ωΔΏ4LηgΐΓUώ:²ZύΘξuυΫ+sϊ™—τΆœ1Ώ@έ€L!—ΨTλϊ\ϊΆΎΙ·ΐφ"•3φeHΛφ‚›νΥΛΊή.%ΏbS£3r+φάSΠΛΑ±½Ψϋ<‹¬η‚σ/X|‘μvψΗeχεΰΫP]Ρ'­”ό›δέ­ό½π»O»IΩ»₯ρ< ΦΟ-rV„.e.ZySλΪΈ2vΓ:Φ7Α‘:νFΟΊrζg—|lvK³i{›jk›δ]ΗeΖ&γP0]8i|ΫT0ΝTypj™==›ΰΪ-η`½ž­Γ±ξ™Ό`]šuΟΆsoηw>Eγ¦ω§ςΑGΉpΗ΄#»ˆΥVš‹ΪžΉ­(8倌CεAθέ Ο&4·ŠΟ?3Ž`μΦή7iη‡+ 6ν†k7zΏ¬ήΎΤΑq7rΈiέ„7›€9άtΝψfΜ؝›Κ–Ure·ό—†ΎpJΫ™Λ€¨•ΔVžφύΫΰ˜g“ΪUψiO{Ϊ’>ΗΧ&‚«O{¨η›ΰ<\iΠz8qjέηό3fls@Ώ«dωvͺυ~€ιƒiμ€`Ε*X'Φ=[…οh»Ώ 6}wGoζϊΜΈ$s`ΣΎ½›LΎ€ς@ύν„&¨Εn>‚.ΗAπα‘E²vJ¬Oo·ΕvR¬”η›ς@z°Ž§γ4)£¦±#±€A+S0Ζ±*Ν::δIΩSωσάqžέp¬Β½ΙύMpo’f“².)iVοuI©ΑLηΖΠΈλ:aμvά+Ύ”Ωη—‘4γϋsέwζ©2•΅ΌδY—¦§KΊδιοη|_pση:ΟϋωΘϋ<œο.Έ‚{Š I3υΜ=ΟwΛΏ Gξσηώͺ2Χέ—·ε/΅Ά¬cX‡»§c]ΊΰLš>_ž9zž4ύύώ<ΟΗW:κΡΣܟ+νΙϋ|›œχxΗ8rέ§ιqz>•&χB[τ B4ΆΔτΩ–M!87‘e gςΟTχς<ι§I“tSΟη{3..¬k·S4ν֎σ|ήΎ/τi’/eΊξŸχχsξ˜|Siϋη}η}ϊΰ§ΩKΊ©Όξw_ή8mŒοχyϊ4ύύqžuΧ=ιr½_¬Γ;?›90s`3¬λkλ0LυΓά[‡3i¦p{Άiή>]sΥύqy›¦“o]Ϊ<λiHYλž%ΝξΛλ³W§žzjΫfήNΙY”γ™ςR¦ΟθΪ~쫬ΜMšΰΫνZ₯ ή©•&χ’6ωWέ—nό,ys Ž©΄y6…cέ³ΰvLΊ?|>_υψΗ?Ύ}5δ•―|eϋΤΆ΄ώλ?©ΝךoΘανIγy“/4rΖ/4o―ƒχ}”t“†Ρ~[#5 ¦†}>880λλλ6{ΝO¬‘¬΅DIίŒΦΕΡkœ<’„χζΕ{Κ%χ{΅œQΝ&ίvoYβ„UΌ¨α£†σ–%θ}’trΕΙ΅>(ί(λή;½(yj(Ζo­4£ΖΝ.ΧzΫF §xaa“nσ–ΗŸ†Ξ;υmΏ ¦!Ω·ΆΆμpυ_CΜλf³ΦύϊՎ΄©)s?Ώt€ΡΤδIς₯%|Ύ0ωΆ{Λ'¬7ο@ xΕό]&M;WΌ\λ₯ΆΝ«%\X˜\Ϋr­χζη}τ’Ζ‹S6Β"U$ΏŸΈq’„ “k[υAm6ι6o>ώ4ά=΄žžL£Σ¬Ϊι5Υ’»Ο¦‘5ΊΩόόΌΡρΗ_?μytmMψτ€ήΌƒήϋΛ&l7Ύξ9x?‡Ε ΪVH\—^ά4ℏ6JWΦ¬Ώ™&λ-\…υssjώ:~μUψˆQewpyσΣΜΝΝΩ9r―――ΝΩΩ™JfzzΪΈŒd ŠEΠόΩϊέΊύ|‘²!ٌθ†ΕΞΎν©y‡uγBσεκI‡‘‘[KwΘH•© € €@ϊϊϊμPυš6zqqΡvζ6ΝΝΝζξξΞ>₯«σSM ₯k›››φ‚ŠΜυ«26Y#€@dΪ?jνƒ)ϊΫ:ξΓM‘3!  €@ͺΝδδ€=^[[3'''fttΤ455™‡‡sxxhΟƒu]φςς¬¬¬Ψ)‘TιRŸ+?u|Χύd½² $@gš Φ₯Z@'σZ4”˜žFgA Ÿ@]]ωφν›Ωέέ} :>>n&&&Œnΐr³υ•…7”Mΰιιɞx«ΊΝ‚@˜€φۍF―b_°ΤΉFΛςς²iοh·οKύšΝ”@@2% sJw~όψΡΜΜΜΨW=i»±±aω+«p—––ŒFKΤΒ5ΏŸ@ R~έ2WWWΆxO\w―Τv’\ €@9ά9qcc£™šš2 fvvΦv˜Ωήή~S€/_Ύ˜……366φ:jM©―Υκz±:τh΄œΣΣS{^ώ¦ ¬¨zš—?Ψώ­z2!ΰ.>h‡w~~nΤA’··Χ΄΄΄d’~T’xκt΅··gφχχM}}½ικκ2:kθ9χ½*^aξ7xssc;I>??›OŸ>™ξξξ°hl«rχ½Ρ“Ύ ±χ2€Ύ†Ρ²#Ž•ϊ³*oͺ €T…€;ΥC^GGGφ‘=Έ£‡4¬½¦νθθ°ΧZ[[­‰‹S@TR/p||l...μȍK#o±K}³R@ (Ώ6:nh&ˆοίΏΫ)Oυΐ£Ξ‰ΫΫΫMgg§ikk³y—ϊXβςΣH9ζφφΦ>X?00`ο„DS+@gšΤ6GχΠΑσώώή&©³ZάΥ~ΰ?@T θ7 ₯―‘Ζtƒ@@b ψ―#θ:ƒFΩTgξ>ΨΏ\ήώ°n=― €@Z؏₯₯₯(' PZρsβϊ“Ϋϋ Π™ζ}=I­΄“Φ?-ΊXΑΣηΠ(),‚`ŸΒ*Pd2%ΐΎ=SΝYΆΚ°o/=#€ €U#ΰ½&ε―tΨ6X>#€•&ΐ΅™JkΚƒT@ΨyoΨΆRΧθυΨVcLmMm©³'ΏΠ™&D@ 4•t/MΙΘΎϋφμ·15D@*Uΐ‹Ίςρΐ—“ΰ@@ Z8'––Ξf=ιL“Νv₯V € € € € € € € € ―(Q@@@@@@@@²)@gšlΆ+΅B@@@@@@@H @gšhDA@@@@@@@Θ¦ΐ™ΧάjZxωIENDB`‚pytorch/docs/source/_figures/hg_example.svg000066400000000000000000001520031444262304300214360ustar00rootroot00000000000000 pytorch/docs/source/_figures/hg_example.tex000066400000000000000000000017021444262304300214360ustar00rootroot00000000000000\documentclass{standalone} \usepackage{tikz} \begin{document} \begin{tikzpicture} \node[draw,rectangle, align=center] (0) at (0, 0) {\textbf{Author}\\ $1,134,649$ nodes}; \node[draw,rectangle, align=center] (1) at (4, 2) {\textbf{Paper}\\ $736,389$ nodes}; \node[draw,rectangle, align=center] (2) at (8, 0) {\textbf{Institution}\\ $8,740$ nodes}; \node[draw,rectangle, align=center] (3) at (4, 4) {\textbf{Field of Study}\\ $59,965$ nodes}; \path[->,>=stealth] (0) edge [above left] node[align=center] {\textbf{writes}\\$7,145,660$ edges} (1.south); \path[->,>=stealth] (0) edge [below] node[align=center] {\textbf{affiliated with}\\$1,043,998$ edges} (2); \path[->,>=stealth,every loop/.style={looseness=3}] (1) edge [out=350, in=10, loop, right] node[align=center] {\textbf{cites}\\$5,416,271$ edges} (1); \path[->,>=stealth] (1) edge [left] node[align=center] {\textbf{has topic}\\$7,505,078$ edges} (3); \end{tikzpicture} \end{document} pytorch/docs/source/_figures/remote_1.png000066400000000000000000000754031444262304300210350ustar00rootroot00000000000000‰PNG  IHDR…2ΣβbiCCPICC Profile(‘mΑKQΖ?Λt‰„ˆ{ ΓΒΦ@ΊfQ‡ΕŠ²ΫΊk₯φΨέo‚ΞAΤύ:wˆ H(Šα9φR²ΝΣj΅šΗ0?>ζ{oή]‚ΒXΑ  X²ŒΤόŒΈ–^}u0Œ  *ͺΙⲜ€|ΧΞ°οΰαυvŒίu]~]ζbϋO―υήσΪγίώŽπg5S₯ϊA)©Μ°O„Xή΅η=β~ƒ†">β¬·ψŒs¦ΕΥfΟr*A|Cά§ζ•,ρ3q8Σ¦λm\,μ¨_3πι­΄²Du€r³˜C’Žb˜ΐhG{&›žΆΑP† θΘΓ"wœ†4βE” bab J‰οϊχ]-{DGι)Ώ«mΎΥ xεj#[τiΰ2ΝCωΩ¬ΗφšΉ¨Τβ@θ9vœ·Uΐ5Ηy―8NγθΎ'―ύ Ÿe"M7ΣVeXIfMM*‡i’†D … ASCIIScreenshotΙΰΞΦiTXtXML:com.adobe.xmp 256 901 Screenshot r;’­@IDATx인EΦ†‹ά\Βvq‚ύΈΧ`Α]Χΰξ‹C‚BΑYœ ,$Yάέ5ΈΣ½‡ΤΠw2>=s§{Ύσ<χΞLwιΫ3UuͺN/ςβ$" " " " " " mI K[ΦZ•0R υE6& ₯°Ύͺ." " " " " R υ6& ₯°Ύͺ." " " " " R υ6& ₯°Ύͺ." " " " " R υ6& ₯°Ύͺ." " " " " R υ6& ₯°Ύͺ." " " " " R υ6& ₯°Ύͺ." " " " " R υ6&Π΅λͺ‹@I=τγ6l˜½vλΦΝ­ΈβŠwοήφ9ΌΪύ/ς’Βr«Θ"Π0(‚Ηw\N!ΜΟΕp̘1vω˜cŽqΗ{l~}ΤR˜šG₯‚6ƒ  a5Βjαƒ>XM…–! ₯°e… ΩVZi₯’«ƒ•”M‹ξ•PRV# G3­φDTžN!ΐ !f£΅ &₯2#­•žβ‰€ˆ€ˆ€ˆ€t&­v&}εέPY%LB΄Η0 ŠJCD@D@D@D ™€6“ΆςjIυšζWJf€ωDτYD@D@D@D • Θ|΄•ŸŽΚΦp¬Φc6Z¨€2#-DEΧD@D@D@D@Z•€”ΒV}2*—ˆ€ˆ€ˆ€ˆ€ˆ€4€”Β&@V­K€UBœΔ$)α ϋ$ΣTZ" " " " "Π(R EV馂ΐΓ?œ;ˆ>©=:©€”Žˆ€ˆ€ˆ€ˆ€4œ€”Β†#V" " " " " "ΠΊ€Άξ³Qɚ@`ΕWLά|΄wοήM(Ή²dH)L†£RI11cΖ€Έτ*Ίˆ€ˆ€ˆ€ˆ€ΤG@Ja}ό;ε±ͺΗφH )…iyR*gC 6B1lHa•¨ˆ€ˆ€ˆ@S4βγ¦\™ˆ@ΊΦOΡD 3XΩKκ{’’™™―†*"" "Π&wάqVΫBc‚`tμ±ΗΆ U³έŒοΏάϊv·ΫSW};θΩ³§}ζxŠzepπΰΑυ$‘Έ" " " M&ΐPx‡vpo™Π½ύӌΞΝΎs3-νάΌΫ9Χ}.η&žΖ=όΏέΓΓ.±’iς·ΙHΩ5…ΐx‘—¦δ€LD Ε Π)„YΒZŠϊΰƒj•°pŠ#" " @ ¬ΪΚΰ"ύœλζΐR2ζUΧν³ϋܘχŸw¬j]₯,έK)…i{b*oC Τ’ͺchθ#Qβ" " "Πγ7ޟŠ a5ςΦΞ½=άi2Έh Ϋκδh¦ΥŸΚΧT(…ρFΎ[·nγδΏ&…p<Ί " " -O€ώΎ[^ΞU«R3ΜKύͺβJ+­ΤςυTE RZ)¬””Β΅ΜIψ»λ»άΘ‘#έoΏύfζ‘xΠ‘HD@D@D@E€ώΫΆ‹Tb2Z’jέ^;ί-<Η΄6™\"˜n‰@*H)LΕcR!;‹ΐ /ΌΰΦ[o=Χ΅kWχκ«―ΊO?ύΤM7έtUε+" " "P'3νΉΞŸ+~υ€εχΊQύ;XΥ“œβŠ@gωhgWή-MΰΩgŸu묳Ž›i¦™άM7έde}ζ™gZΊΜ*œˆ€ˆ€ˆ@qζT†Ϋ˜€Φ+ή„Τ\šυ¦§ψ"Π‰€v"|eέΊFεΦ^{m7Ϋl³Ή;οΌΣυκΥΛυθΡΓI)lέg¦’‰€ˆ€ˆ@9I+pcΎϋΕΥ{€UΉ2λΎ4ƒ€”ΒfPV©"πτΣO›B8Χ\s™B8ε”SZω]tQ)…©z’*¬ˆ€ˆ€ Pξθ‰QŠ^κ6§σέΟEo놀…@Χ΄T嬜@˜ gξΗ(ΆΚŸ€8#FΨΒ…ZΘέ~ϋνnβ‰'ΞF)4hPξ³ήˆ€ˆ€ˆ€€‹ΐ°αχzΟ‘s&WhΈύθ§ϋ'—žRN" G3ΎΩζΌi•I\Η(τψ㏛BΈΔK˜Bˆs™Έ`FΚΓ?όΠφΖοι½ˆ€ˆ€ˆ@λΰ‰‡FXΫQ…ͺ7φΜΒ(Š έΥ5H ™¦ζQ•.(\X,Y8kύq ’? <ςΘ#f2ΊτK›ΙhΎBH¨ϋΏ³ΐΪWψ'3ύ΄ΐzͺΫδ&ZlYa%ŠS‰u)…>©l1Ε΅ςθΡ£«J’x(†¬.Ά»pX=NehΤοΈγγYˆΙŒ3Ξθzφμ©}……ΰθšˆ€ˆ€€€}ύ˜χŸO¬€έ~{Η…m:‰%ͺ„D H)μθIf‰b‡Œ3¦¦dY] {kJ ε‘ξ»ο>SΧXc 7lΨ°²΅‘³™²ˆ@D@D@ZŸg & I*˜ GIˆ@Ν€֌σ#&±ΚΧ­[·ŠΝN;ΏΖΙ–ΰξ»ο6…°OŸ>ΉsΛε€R8rδΘrΑt_D@D@D  °Rˆo―[ό~BKb|Έ)„oΌ±2dH)Dξ‘~πΑξ½χήλp]D@D@D@A€1Κ\]ŠαX3¦`¦£Ϊ*₯”$ ₯°$žφΉY‰ιdVhάvΫmnέuΧu[nΉ₯»ξΊλͺͺ–œΝT…KE@D@D % ˜2‡ )Κ]΅BΌ·‡ΫŠ£)—ΥΖWxhAR [π‘TR€$V γωTλ¨&7MοoΎωf·ώϊλ»νΆΫΞ]}υΥU}Ϊi§usΜ1‡œΝTMND@D@D u Μ™bθ•»ͺC”HozJάv³²j§§’4‚€”ΒFPMišYw8sγ7ΊΎ}ϋΊwάΡ <Έζ§$g35£SDh(ux οέσ{ηάύO尐ωΏn―o+„Δ‘BΨ2QIˆ@Χ„Q2M$uε­(―Ώώz·Ε[ΈwήΩ]|ρΕueRxζ™gΦ•†"‹€ˆ€ˆ€t>V ωCΙ3+,VΗΚ3ΜαΎύδυπΡ-μΓ=ψž©ΟΡ›LΠJa¦g}•Ιͺ²‰™( αn»νV·Ba”ΒO>ωΔ½υΦ[υWl– €RE‘ύ1^@~ώκ][Ide{ΌJD «€fυΙͺ^FΰŠ+pΫl³Ϋk―½ά…^˜9›I£–#pꩧΊ‹.ΊΘΆšόςΛ/gϞΆ’ΨrUD aγω™Μ¬ƒ‡•πše{ορΖ/α―‚³Y°ΔνΔ/»μ2·ΣN;Ή}χέΧ 0 Ρ’Μ;οΌnƒ 6p'Ÿ|r’ι*1HΠηb‚'48逓άGαN9ε·Λ.»Έ©§žΪ :Τό€Ή^*»TB υ+…tF+­΄’CIβ•Ώ³Ο>ΫμΒΉΖ_•CuΎ₯Ώή—\r‰)„ϋοΏβ !9ΛΩLiώΊ+"]‘ίΝο{γ}nϋέμ>QΥ ΗΌ)„§Ÿ~Ί;δC\χξέέάsΟ-oγϊz΄ T+…A ³”α©q {\P³¦&}XjιΕω7ϋ=f’ΜπtΠA sƒR8rδΘfWMω‰€ˆ@§@Ω£ο₯ίΝο{CΑpΦΑŸΓ@D―­N€1ύϋχwx`ΈšΞ‘Π›6 J₯Ž%―X‡”ά‚’˜₯NŠ•Β$W ³yŸwήyn=φp‡v˜;ν΄ΣςΏ ‰}f_α_|α^{ν΅ΔTB" "ΠͺθoQΝ;c……$lΦ&d+¬Ί‚₯ˆΐ‘Gi«„ηœsŽΫoΏύ:”\ΐpθCΖ €rO!L=ΒlP” ΠIΧΓ‚ΈYαΑŠ0 : ό 'œP/–’ρΏϋξ;7ΕSΈλ»ΞmΎωζ%Γκ¦ˆ€€@½ύη'3%"ΠJ=τP‡c™σΟ?ί&”σΛΖYLΟ1Ηω·υY#ΧK˜P[xα…]·nέ܊+h‹@I.-4Žf$^‰<$œγΤυη;¨4U»hYαQ βgAόΉΖ‘™υιΥ«WδΝL²€Ou’κνgθ£ό€¦hϊΊ!A€ώ›ο¦χ4Z4ϋoΎωΖΒψ³Ž‹†Ρ ¨‡@Ό}-€ί„k΄‘Φ]π8™‰ƒ«GΚZU+βeAό,Ÿ5Ϊ~u°©ΥΩn»ν’•W^Ή©y*3h&Zϋ—B}tVϊœfςW^!Π―_?7 4¨lσΟ?δ}” §"P-ZΪΧFΆ£©2­Χ|ΕwR$kζ,,=³δΜrsΨG―pΈΞ΄R9^Σ. qψα‡;άH³°™rξΉηΊ£Ž:ͺ λf–Cy‰€ˆ@£¨ίmY₯ΫY8¦Šύƒ—_~ΉΫ~ϋνΛΓO»χίί=πΐeΓ*€TJ žΆ•ρ;:L%ιΣ”^5ζΣP/”B”=Ό|…ΟΨ'σ%ʊ2|β‰'šBΘ~€f+„|p6συΧ_»—^z) _•QD@ͺ"P©3·ͺU`θD{ξΉ§)„W^yeE !E•³™N|`Ν§]υms|b=iΕγvhεχ¨|‘Υ΄VfPIΩς9M5ΥTvλξ»ο^IτΤ„A‘§gœq†;ΰ€:₯άtΘ3Ο<γ8Μ^"" Y"Π₯Ά;2KΜT—Φ%°Ϋn»ΉΊkΉΖmΉε–” `Ζ‹/Ώό²›gžy*Ž§€"PˆνjmkhK“lOS΅Rˆωc’2zτθ$“kΙ΄ώφ·ΏΉwί}·%ΛVk‘0ΩD!0`@§)„”}β‰'ΆΥW”B‰ˆ€ˆ€ˆ@kΨyηM!Όα†ͺR©M|Έ5k§R₯…Κ`½«„ρΊ&™ι¦F)|ψᇲw+ m=ώ€Zνύ¬³Ξš)₯ύƒ˜²Ÿ3ΩΞ™•tφPώ" "Ј-Yοsυ,”nνvάqGηΚΈ‘C‡ΊM7έ΄κ„&™d·ΠB ™UPΥ‘AL Ι655J!ηt$½RΘsJrΩ΅ΑΟ½¦δY)|ο½χjŠΫj‘9δ‡c™ .Έΐν΅Χ^-Q<”B­ΆΔ£P!D@&υώ1a\J l»νΆζPζ–[nq}ϋφ­Ή„š"Ζ΄ϊD[j”B:§€χβx%λ’σQž;ν΄ΣΜό£•φG²ΧΰϋοΏwΟ=χ\ΦΏJͺŸˆ@›hΔd,ΞΠ$"Π [m΅•»κͺ«άm·έζ6Ψ`ƒΊ²Τp]ψ9%R£6‚ηϊλ―߈d[*MΜGΣΎRΈί~ϋ98½»τKέ.»μR|ι(p+¬ΥΒ–z,*Œˆ@1›@±”„”%°ωζ›;φήyηOŸ>eΓ— ΐπwί}ηžώωrAu_ŠHΤ3d2lΨ°πΆξΧΤ(…2c©νY³RψΗ€v_αή{ονΞ>ϋl7xπ`ΗΎ€V“]»Ϊ&t)…­φdTV$ ΎΌŸJΆΚ΄Ι&›8ΜEQΧZk­D*Η0’Ύ>œJ$AIn­KRΏ$j€cΒƒeΦ…•B$«…{챇;οΌσΜόƒΓc[U΄Χ UŸŒΚ%"PϊIώ’t„τκ)“βŠ@)nΈ‘>|Έ)„«―Ύz© Uέ›`‚ 4\1.D€6°•%5η˜μGHb£f»μk`₯αXŠe—]Φή§αf’—\r‰»φΪkέ[lΡEΖ¬„•L‰ˆ€d“±˜Θ'!INμ&Q₯‘-λ­·žΓ<ΒF Ύ5œόχ%˜SςΚ)μcFx~x†ΙΧ Ί©Ηα%ι#%I}¦KuΥιόΠ¬ξΥλ †A<.†ΫA0oœe–YR΅RΈΣN;™B8dȐ–WωΡQόόσΟnΤ¨Qνπ•RE@ڌ@Κ\’—6Γ―κ–!πϋοΏ»uΦYΗ=ϊθ£ S)}½ΜGΛ<Œ o£2ΩΔ9{ό±Mˆk,ϊπΗ5ξgΡ’/I…°BάKRHΝϊχο_σ škι~ύυW·ΤRKΩ—ξκ«―VZ¦ΙιφΫoο.»μ2wΣM79φ€Aθ(0-Qg‘†§₯2Š€TK€ϋz”:βfqpW-G…Ožΐ/Ώόb α“O>i αrΛ-—|&cSdA᧟~²•ž†e E0^ΥbŠ b–ΪŽ€λBΫΜ_R’J₯Μ\VΣI…8wέu—{φΩgΝζ|ꩧvΫl³­€}τΡξwήIŠkK₯“–μyW\q…»υΦ[έFmΤR ΛF3ˆειΎˆ@š 0˜‘ί­ΤR'μC$N‘4sTΩ“#πγ?Ί΅Χ^Ϋ¬t0]zι₯“KΌ@JτσX_iΈœ .±ΈΘ"‹ΨŠ`ΑsAP Y5$~€έ₯\}“L‹ΌR©H‘“*eϋtLΌBCΒjΤk―½ζ8άτ’‹.r={φ4sΕϋοΏ?ΛΔ++…­ξh†}ƒΧ\s»ύφΫ] ‘³9rd&Ύ/ͺ„ˆ€"@Š™|‘>7KΚcEϊέBiκšΤBΰΫoΏ5…π₯—^²Β%–X’–dͺŠƒb’ ΰͺuŒrΗ~ΊZ…Ζ,ΊK‘6΄ΪΊ‘F\―©6~‘πγωF;*t#­Χβ3 ΥΒΒl‘³πžxβ ‡™{Ϋψ›h’‰ŠΓΚ}ξΉη:Ύ„_|ρEKΦcΣM75χΡ(„˜χ¦QψξΰS–.]R=Χ’Fό*³ˆ@' Οe?ϋοΏΏ WmίΫIEWΆ)%€©!Neή~ϋm›H^h‘…šV“=χάΣV GŒΡ΄<³cP”Βzλ&›XδΙ‚0Ι@Š™Ξ–ͺ#ml#8dnτ ¨πW h‘{œƒχψγΫfε\ΠργŸnΊι\Ώ~ύR}`)ζ£_~ω₯cf­Υ€oίΎξΆΫn³™Ύ΄*„0eφΝξ2+i΅o˜Κ#"ΠHτ·œ wΠAΉ³Ξ:ΛΜλ™ŸnoLncι…υ&£ΝT!Ο‚ύ|ΖΦSϊ₯bβ¨^…’<‘ f„οP₯ζψρϊλΗψ΅€ήgN)L •/ΏόrχρΗ»C=Τ”–Xΐ63ί|σΝIdΡΤ4βΗR45γ2™m°Αξž{}΅ΥV+Ί΅oΣ1a>%₯°΅Ÿ“J'"Π§vš Φ™\ύκ«―“‰RmkŸ~ϊ©}ΗxE!μΥ«WΣy0όΫoΏ©――‚<Š\’2lΨ°$“λΤ΄Xν‹―ψ…½ΨΕ …BΨH₯8sζ£Ε@Φ{}θΠ‘fZzχέw»9ηœ3gZ:ν΄ΣΦ›tΓγφΩgnϊι§w8Ωi…Υ8fGϊτιc+²¬†si’Α,Ώόςnήyηu_|qƒsRς" "Πz˜H]rΙ%Νr"¨­GT% >όπC3ύξ»οΜd”qXg ΐ p»ξΊkg!Uωb&™΄dq₯ε™?Ξk|α…cχ°χ°V ΘjΉK)¬’Ψώχ?SΩ{ψΓ?δ”ΓF{½ͺ²˜γŸ|ςΙΝ΄‡}o)Β^άG³‡pΩe—νΜβ$šχ~ϋνgŠV ΕͺΔD@RDΰp«Ίͺ;ςΘ#έ 'œ’’«¨­JSQΖ μΩgάπχΏ½S‹Š5ΩόσΟοΨ©εHKζP Y]CQΚͺ0VυΥWMIlfe>Z%mΜΟ9ηΣΰy}ϊι§έ2Λ,γVXa;N‘Κδšœ}…ν”σ}8`f˜~dI!δA²Χ€t\hG«¬²Š9ž9ρΔέυΧ_ߎTη ΰL†=„μΩgάΠΩ !U“Κ0+_’κ tΦωβR «VcI'u{ο½·ΉΧΕ€tΖgtΌ>ΣL3Ω ι[o½Ucʍ‰ΦY_°P›οΏήΒηž{Ξφ₯–Z*άΚΜ+’•ΒΜψΰƒκ#ΧCJaπς£bΦπ―ύΛΌ–<Ψ^qμ²Θ"‹ΈσΞ;ΟύψγωQšφ™/Xg˜r ;& ΜτΑ"ΛBg‘Cμ³ό„U7J p~+ƒ{Ξϋ•ˆ@₯^|ρE³,κή½»°Δj%Αλ)~dTώ©4jί_#=p–―UγC°ƒΌϋΟ,–ƒ”ΒŒ$ίn·έvξ‘G±s15ΐΤ”31«ΑYM³₯3V ρœΔBΌ†‘rφcΦE{ ²ώ„U?J Μ=χάŐ#žΞ<σΜJ£)\`‹ Ι3Μ0ƒ7΅’hΈς§(Ε°ς€/$ 9H³s€6ψ»‚<•’ uΤQŽύ‡˜Τ¬΅ΦZŽc.š%|Α°GAk†ΰš…ƒfQ1­laΰΩgŸ5Ο΄νP_ΥQD@JθΫ·―\}ΰZW*¬ξ΅7Ρ£G›BΘx…qΓΤSOέ²@4\ω£IϊΨ±pLCε%H_Θ 'œΠ|”h₯0}Ο’sžα!‡b.foΊι&Χ₯K·Ι&›Έ9ζ˜Γ|ςΙξ“O>©(Z…Y‡f|Α°f¦οΫoΏ΅†}žyζ©΅Ψ©‹GGΘ¬$uNh£>Ϊϊ;ΜHίύε’dΣL―δŒ83•§šjͺ–ΐX}uζΆ –+¦ž,†$%Y7 œ·7cΜςγU+…qMzΏΡFΉαΓ‡;Μ$8Δ₯›ωvΨΑ=φΨc )E°OnτRτ;οΌc ;Η20Σ‡ΫN‚Ήϋ €ΆΣSW]E@ΚΐŒ―8ž‘ˆ@œηcY„EγœΛ΄ΊhΈΊ'ΤΏκ" έ«„‘κŒΫ=fy…W)…D'Ό²Y™ ¦₯ηŸΎ™.ΏόςŽƒQΩƒ‘€L<ρΔ¦x6rΦαΝ7ί΄†rΣ°Ο>ϋμIV!5ii―Aj• *"Π$8ζ@1ΌοΎϋ\Ώ~ύš”«²iuO<ρ„M$γ„Žqc•4PLrhΈ²§ΕΎΒz:VΗoΌΆ±6ΠJaeί­Μ…šh’‰ά{μaΛ½χήλ˜`6uϊι§w‡v˜{ύυΧ©s#g^{ν5kΨ© {0WM€ΰ)K$œa”²b«Έ" "ΠPK.Ή€Ω4`ΐΫkίΠΜ”xΛxτΡGmάΐΉΕŒ&˜`‚–/sΌ€šŽΣ(³ΟZΓ=χάΣΞ>=ϋμ³mŒΌυΦ[»|°|¦)Α8Z+…)~€I}΅ΥVsΧ]w{Ϋα°Ϋn»Ή+―Όlμ7ήxc³³―'FΝ:ΌόςΛΦ°O9ε”Φ°Ο2Λ,υ3υqι(p©ΝωŒΏ°―•ΒώσŸnĈέΠ»Ά"ΐ€ž=„8!a; ~&τυZ)¬ξ©‘ςμY9¬DΨ‹HxŽu;ύτΣΝ²nΰΐŽ…ˆ•W^Ω1ΡΔ™¨œι—5a!KΒ~ψ‘iUKί―°ih:7£Ωf›Νόρvp%ŠαηŸn (Η:0SςέwίU]@–έQV’”ηŸήΚ…#fϊp#έξBG¨³hχo‚κ/"PˆΐYgεV_}u³ˆΑ!™€½άύΆΥ„IπaΓ†₯ΆςX½πΒ ζT/΅•θ„‚£’θ±jΘ_Ύ‚ˆ"Œ5ͺΓύρΗίν²Λ.Ž}¨€1Χ\sΉ]wέՎ|ΓΓρK/½Τ 5jL–Αβ‘ΫΎςK>^δ%’>·&~oΑξj™qε―؁π=τγοα‡ΆΧx­ψΑ7Α΅zrΒσΦzλ­g¦’·έv›9X‰ηΡΞο1ύΕΫμΠΞTw(H/€Μς/»μ²nȐ!ΓθbφάsΟ=ζ`oύυΧw7ήxcͺ+ΘΦΌ₯V³ς•κ 7ΈπŒW›V+xΌcc(Ζ₯ŒyM³p*N(ωΝ0‰ΦA)”€‹€?ϋ/:ν΄Σ"οι…>ς_–θϊλ―οP ?ϋbχόFh{%\±?ΒV+~,šyζ™#―XF_ύu΅Ρ3~Ν5׌ΆΨb‹ΜΧSZ ψs{­_ς“΅&‘x)"pΗwDήL4Ϊ|σΝSTκEυVRΡgœQ:ξ6€ί~­ΊκͺΦxg@‘79ΎόςΛ¦εŸtF“L2I4hΠ €“-šžΜG›’z'› ΊtΠAŽ½|·ήz«ΓΉ‹odΝΫ'³#ΜΌwάq–ι˜1cΚfNXLK+]1|ꩧΜdο[˜Œ²—P‘€œΝtδ‘O" "O`5Φpgžy¦υ=œί+Ι.¬‰Φ]w]η'KΝoBVj*g3­υ$ γα˜s/WZi%wΔG˜i)&¦Œ]Σ&μ+l¦ω¨”Β΄}CςΚ‹ -vνœˆΛοZχ ’²|_JώσŸΨ^φ6’N:ι€₯‚·ν=:ŠW_}ΥωͺΆe Š‹€ˆ@9ϋοΏΏΡ‹Ηm&:%Ω#pΛ-·8Ζ*Ϋn»­»ϊκ«3UAM·ζγd vΑ˜£Ώ’λ8ϊsuœΣ\sΝ5­Yθ₯jΆR)…B/Ν7ί|nŠ)¦°’W²:X¬ŽΜ¬“Η{ΜΒΕ_άBV(%… ΘΩLa.Ί*" ω˜ΜΔςDΫη“Iη‘C‡Ϊ„υ;μΰΈβŠτW(―τυ―ΌςŠϋκ«―ςξθc+ΐ’mΏύφsΟ=χœσζΛvΆ$ΗYτθΡÜά4sZ)¬…šβ˜ωM0­‡±R qVƒϋhLSqέ΅kΧz²Ι|\ΌΗΞ4ΣLnδΘ‘™―«*(" υ@1Δyž%Ω ΰ}ΈM6ΩΔΞ—γωfQ4œž§ΊΞ:λΈ›oΎΩ”xCV«mΉε–ξhɊ4κ(Ήb•ΥJa12)»ž„BH•YeΔ„4nFʏ…eχΫoΏέφ¦ O§—ΞBΗRt ze*"2σΟ?ΏmΈδ’KμΨ₯”_ΕΝ#€‰ϋ9o™sδ²*={φ4‘šNΟζ‹SN9Ε}ϊι§Ξ;q±sΑ½s·Ψb‹Ή /ΌΠύςΛ/-SV ›y€½”Β–yτ­SV ƒ’yο½χšΙ¨χ¦iNmZ§”­_ν5hύg€Š€΄Ν6ΫΜuΤQfξΕYv’tΰleVbφΪk/d§³•—Zΐ•³j₯8XΔ9#ϋ ±†λΥ«—Ϋc=Μ1 {ρΥΡΩΒJαO?ύδ8ž’"₯°”œG₯^C+-FΨ“xΧ]w™BΘqy†«”ή_αθ(ή|σM›ϊλͺή‰€ˆ€#pόρΗ» 7άΠkόq±`Ίή’.Ώόr·έvΫΉ}χέם{ξΉ-ZΚd‹₯ ΰdyvFj+¬°‚ˆ€ˆ@ILr’b­Β¬½€υ 0NΐIg(sφd;‰&€³ω΄7έtSwχέwΫnυΥW7gΊι¦3ΗIΟΦ αΨ7ςlΦΎΒρ8ΦΎšŠ‘πΕΟ³£ρ.¦sΜ1±šτΆz(εPΔ±­=z΄ύύπΓV0–ΣZh!·πΒ ηώ¦vΪκ έ&16Ψ`Χ₯KσxΥ&UV5E@D 4G%˜ua’(iMηŸΎν<μ°ΓάI'Τš…lp©˜ή}χέέα‡ήΰœ”|gψώϋοέ₯—^jVxJΖδ”=‰œΏΩHΑyuΦYΜΖJ)d08 ©΄dR +%U{8ΒBΗHԞ’3…oΤ¨QΉ$^|ρŜ‚ȏeΟMή·‚’FIœ;α„Μ»Υ;οΌ#" " UΨgŸ}JΗ“O>iή«Œΰ &pΞ9ηΨώΑ#<ϊ»gΧ²Ι3<ώψγΛBΛ>‘d †Φ œΓ‰Ι)Κ!¦Σ³Ο>{²ωΤΨcΝ1p7ήxcβiη'X±RX‹BΟμΑt½{χŽ_ϋ„4B)¬D™GΡ "J"ο±ΓFX= ŠbxΕεx»ΙwήiΞz>ψΰ7σΜ3·[υU_¨›ΐ*«¬bϋ΅ΨΖ0Ι$“ԝžH†+p€Ξ±I( oΏύv;ch»Ί³—4¬β!“SDΜM“œ6Ρφ11Φh©H)¬W!ΔΔΕΕPΈΦMR*Q εΗ&ά  ΣΣηž{Ξ‚™‡•Δ (ς:ΡDJ*Χh$fœqFwΫm·Ή>}ϊd’Nͺ„ˆ€4“ν%—\‘^{ν΅ΝΜZy!pϊι§Ϋ~O”!Žiw>|Έyͺό裏¬ΟowνX<ο21πΨc™Ÿ•°zXοD{tωΓ3j£₯"₯0)…£VE£Ρ²~½Š{>ƒ*·šζGοπωηŸΗτ…ρΗ΄p ,°ΐ8Κβ4ΣLΣ!4ψϋίnϋaψώKD@D@ͺ'έ'žx’y¬>…φŠό „Wj΅V[όφς*qŽγS0#ΌγŽ;Μ:HLΪ—ΐγ?n«‡(‰“O>Ή™•’ .Έΰ‚5A2dˆγ WΞ+lψ" ŽfJ‰Θβˆ&±?ΏZX*;έ«ƒ€_uKδ9ρΜ›!Ο?|tυΥWGx`΄Ϊj«EήΓRόή.;ςvΤ‘ίΓέzλ­‘Ÿ)nF‘’GίΎ}#ΏJؐ΄•¨ˆ€΄ SO=ΥϊϊIaωc6o©•λWΓXž>ή+ε–ΟB‘€χ₯`c–ŽWυ©] xK±Θ;^Šζ˜cϋΝ¬΅ΦZ‘ίƒX5Ώ§Ϊβ3^n΄Έr„F$‰W§z£remχϋ(άυ>§Ξ~>o½υ–)~ε3ς·#”ΓP'”F”Gος:Ίζšk’^x!œFΑο'LEYUHheΫl³M4υΤSG―½φZ+³ιe£χ«€Ήώ2τ›…^ƒ’XmOΏLzgœqFΣλ—† 7Ϊh£h½υΦKCQUΖ&@D)δχƒ’ΘΈ₯±„ίςΒ‹-]τχ\νo·P>…•4m„L|…=I#ΤcFΪͺΟ†aσχ)ϊUFΓΗ.…φ)N8ᄍΐ[Sšχή{―[c5Žyώφ·ΏΥ”†"‰€ˆ€8;Lšύ…˜eqF―ΔΉzϊ}?Έ΄ψε8}τΡζ]΄ώnΏύφ+Ό-οsΗ\ΰήύΆ¬Ώ*]žΐ³Ο>›sLσέwίΉvΨΑΣ,»μ²9θ]έzτrcΊΞζ\χΉώL΄ΫΨΧ1―:χ•σš{5q'O%•Βzšbhp,?κ X8]―_ͺψY’•€TiΗPIZΝƒmupdχ€ΚuΫmΎkq…ΡΟ.7£hγδσΌ±ή|σΝζZxœΊ " "P1Ϊό%–XΒS ΒΫYΒ ²εϊ#Ž8ΒΞ<χάsν<ΒzςΚrά{ξΉΗ­ΉζšvΠx=²\UΥ­NψΤΐ) žKΡ‰–[n9σ°όςΛ/ϋsήΦqnφuΛηπΦΞ½=ϋμ³qφ)Ϊ,§;Ωd“u0; ΚβLP{"q¦“ίœο8·P"" " οΨΓfΨ9Ψ3ύv’€Ž Μβ«…ύϊυs pƒ ²‰δF―₯ pt{Ε8BE"•°ίq­ aΘ`¬)iό7nUσZR)du‰ΥΒ$₯œνz’y)­Ώxmξ₯—^rO?ύ΄)†4Xt’ή _Ϊό6ήω¦§|ζœE$μO ―(‹έ»w―ˆΪΏύo;xω7ή03Φ‰ί^yό-0ΩΓρΜόσΟοn»νΆ3t·γ3ϊ•ϋ쳏cΰΰΑƒνŒέ akxU0΅=묳μάΒ†g¦ RO ·JΈ…υΧΕ―ΦkFΪ΅ώR(…V'ΰ]w;Ύ‰mh₯¬ll]j©₯lφOήΫώzz“L2‰[zι₯νο―«Ξ=χάs”E§Εy ςό£ƒω) 㬳Ώnο]tQ{9r€)…Ή†`lHο"<·"“  šD $τ*" γ`u‡ (4˜“β²$>‘˜d}χΨcwα…Ί«ΊΚΆ™$™v;€E_E[TόΉ…νPeΥ±6g•0 ™}ΏΐΠΏ”J’rϋ e:ZΧσͺ)2f xΎδH„ ¬€1»ΚώΒvχή˜Tσϊζ›oŽ³O1πa†:(Š¬(Ξ3ΟIO¦«―ΎΊ;θ ƒ 7ŽkΎώ{χψ㏻SO=5gZ«γ¦ΈY©˜‹@+ψο릙f;Μϊ?ωO+IehcX½„­Ώώϊk¦I¬ΈβŠ © 59ꂃΆ8ΰΫRΑφŠXΐαŸΰΜ3Οt<π€«ΦB€<ΡϋΏsΟ<σLΚk‘⧑‡ήΧ3ž/«eε°Αϋεb‹-ζX­βLŽί½žδ·Bt’(„SL1EΑΈσξΣ§™‘†σό ΤŚL:逎 Vλ~θšX©—’β'M€‰‹.ΊΘΎη˜I³§ ‡Jθ ¬²Ÿ‹­Y–zΗd…ؐζK,aΗO0v`B“ί7η2ΡΉΦZkΩ~9œ©¬Ίκͺv&ίμ³Οn«bLZb9ΆQJΏ]±―PJa»<νκ‰ΩgB2ζύηλJ©¬ωhzΒΜ6Ϋl¦²zH/FΨrΛ-mlΑ>Ή,&s“`X―BX¬ ?ύτS‡ΥΔ°²ΘΉΐϋƒ²Fφ)§]¨梘ΪJD [4ύ‘s¬Φ+^)μέσϋΊ ͺ^)¬·Μεβϋν·9E‘ΧρΪWΩΔ+)M`΅ΥVsΜΐ1KZ°3Ÿ86ZR$;νxIκ™ Š§ΣͺοψανMiΥ‡+)°D(΄RΨ₯KχΗΨV笱ƱΨz+Ι“Τluως›έ³#Ÿ΄L˜€ΞYΒ€vςΉ7?Ε€,Qšέ§°O1(ˆΑ υ³Ο>3€μSŒ+ŠΌΫίώΦ|ΈuδΈέvΫΉχίίΖ³u$£¨'#rVa=«…cW λέ^ΤrJaώσΗaAXEόχΏmή8b!$:,Έ*–]³―3†@‚BhΪθ‹ρΥDήsΖ"Β’Aάμ%K«€d§v²£•‚Wύ€U:Ω&Ζύqf ;˜žίs#&w2­ΖΏ6,ε‡D^ΓΩ9(ˆΜδπΚ>ΊVΎΑ찐)HόZ˜)8α„ά駟ξ>ώψc3[H’.p[j©₯ά΄ΣNkό’H³έ0[qί%΅―0<οvγ¨ϊŠ€ˆ@œ€MΈΥ£†Δςά’ΣkkΐΛiώ>EΆ!“M6Y‡Ε 4Φr,Ϊ\ΰ>ψ`ΗΉ…¨•“_H˜Ψ ―v1αm£ζs{χέw;(‰(RΣM7]‡UDά"w† D„Ω½JσGQ8p ν#d`’ςτΣO›ΒΌηž{:ρ•DΞךώω;(‰Μ 5ZlΓΌohjYU’|Μv5bίδW\αΆί~{wΡEΉ]wέ΅Ρ2—~nSgΝ΄JX'@E&`ΠLx„Α^!Σp_―• M=ϋ›ά˜9χ¬Η³χϋ “rλOVο³GΰΝ7ίgŸ"ΦgȌ3ΞhŠbXY<ρΔΝ$Պϊόj‰)|½€V@πύχίﰊψα‡Ίi¦™&§ ’(2cT$‘†ό9ΠX{ν΅sg –3εΡλ_j}Ξκώb¨w"ЊτΫNώ©`χΑ8ϊ\^γC‡­μLΒJ‹5v_‘Μσ+¦pqx»Οί§ψβ‹/ΖƒΤτ^}MΨ©FR k7rδΘJβoΏύf«βNk¦œrΚͺRNΪCe£:Ά>ϊΘlδ9ρ¦›nͺͺŽ ό'j˜£τοοg°ύ*΅DD 5 Tσ›.T~ίνζQ˜At\Ι+€ό}ωε—pΝ2Λ,nζ™gvΌ>φΔχω€‹&·R(₯°k}¨ŸΐGαN:ι€ΊjδDέ…S™" ₯°ΞΗ‰Bί‹ˆΒˆ¬°Β Ή•Δr¦–υ( U‘‘ΘύχίοV[m5wΤQGΉγ?ΎPφΊVž{‘³€Βvf₯ VRAD  `*ωδ¦θ‘μΕΏόΟ]ΊtΙΕ1sόΡ:·HΏά΅Ίήx₯π˜νs΄ΛH‚@“ύXd1&h·I£$ψ+κ t­>ŠbΔ tνΪΥ­±ΖφΗu:Α $^rΙ%vŽHχξέs "«‰sΞ9g<‰Δίӈ0Pi”B±κͺ«ΊΈύφΫϜρlΆΩf‰Χ‘Μ|4ς™΅OΥQ:ƒ@p(Soή(9²π¨·l!>ό )zρkα°π‡³sƒ’·ΰ‚ Ί΅ΦZ+χ9(}΅lEΐaΟθΧ½£™Q½―oϋΓ έbυ¦’ψ"`ςϋχZ±ΰt±Ζ΅TΌjh₯°Z5„ΕΖ<8¬AYΔ‰Νάsϝ;%+©E(" a£g–vήygwέuΧΉ#Fδά-‡όυ*" Y'ΐΐ/©Ά»3Wθ—βŠ]XΝ +~α5ΈηηΉβν:˜rε.3ΧΗό†| $Ϋ -+…έζͺ?IQ?C₯#Δ*a.1¦cΊx~zߞ€6ρΉsΜEXEδ•c0%˜Œ5ͺQΙη]f™eάοΏξ8G"" νDΐL½r’”4’έΖ}~PκŠ)~μο‹Λ 3̐3η,€θqmꩧŽGι”χ‹,ΎŒ_-τΗΤkBͺύ„ςό²œiJa#Ϊ†,σWέj# ₯°6n‰ΔϊψγMIάύ]~§œDΝ0Ezωε—ν`ϋΎ}ϋΊΛ.»,‰b+ H€~TΊv›3Π ­θε+~?ώψcŽε$“L2Žιf‘UΎ &˜ §•ίδV {S»Γ™± aVφtΆςσj—²%iAgViۏ£χ"P ν)¬†VΒa9Οf«­Άrƒ J\)ΔII3džyζq—^z©Ϋd“MΜ„W"" Y'€B!]Ξ8 «{…?}ςΙ'²ŸnΊιr«{a‹Bώ*G)eI0©£―3ήξή„΄Z31―:ηχΚ©W–ΎΩ­ mίy‰4Š€”ΒF‘­"]Ϋ0οχ²‘8²ρΖ;fΖ8ΰs<ƒγ‰ˆ€d™@#”Β)¦˜Β­ΎϊκŽUΐ M4QNΩCΙcPVχΒ+Χ ΧŽBίσ§ηώ՝[8v…ž€!$ °G8­cΊ$9(­tωh Α²t†)Μ¦›nκžxβ s<Σ£Gk£€D@D ΅4’έ¦†}ϊτq»νΆ[Ξ‰ +€’ςPμlΕSR«†¬Ύε=ϊWV₯–g«ΥhTΫ σΡκžƒBWO@JaυΜc‘Eqx*MJ:£ρψφΫomᬳΞκξΉηž€ͺ’tD@D % 4bOagLθ΅$ά Ε`ŐWΜI»ω# ΖΌζΊM>‘σώσv™ΥA™Œώ‰HC mCgŒλCG©Ά*)…-ςd’œYκΜΩOŽ§Xzι₯]Ώ~ύάYgΥ"tU Hž@“y”PΏϊŸύiP 1- B߈ JD ‘’n:s\ΧHNJ»΅tm­β΄oiΒΜe½g^uvΓ±ΤRKΉK.ΉΔq†a―^½άŽ;ξΨΎU5Θ4Z]/DΚJ):•ίRL W7 IDATƒ£XVΞK!“'°ώϊλ'jύ•| •’ŒK@+…γ2ιΤ+υœ{E'Ψθλ+…ΓJα€lα’K.Yi4…ΤHΒƒJwφ€^jΐ« "υŒηβΥS»§‘χ$ ₯°‘tkL;·aΎŠψ­ΨhΰEΧιl?ωδ“WQt¨₯½.T³VlΓ •SΧD@*#Τ€‘LΚ+γ­PυRX?Γ†€Pι@cα…vύϋχoIS™χή{ΟΟ,Ώόςξ†nh'%*" M ‰ύCψuφSTώ"<JΗrΕrΦdQ12ΊήR A5Α4™iβaΓ<ηή°…³ 1ε―•εξ»οvk­΅–y„;ϊθ£[Ή¨*›ˆ€ΤD ή όjΒH" ΅(†ŒνhZ}Œ—Š BVL@JaΕ¨°Vgžy¦;πΐέΠ‘C]ίΎ}kMFρD@D ₯ T;ψΣΐ―₯§ '‰ˆ·ω‡ΫΗ?k’(QμJ¬ R «€₯ ΅Ψa‡ά-·άbϋ ηž{ξΪRLhaρ_Ό˜ρAΧ5π‹ΣΡ{h΄H­Ώ¬ΰϊ—YR 3ϋh[«bμ—Α ιLΰόρΦ*œJ#"  Θ7ύ'yAD&a†AD@D@Zˆ€”ΒzY/ΚσΟ?oŠαΦ[oν˜υκͺ~" " " " " ]RQJ28ΜώK/u_|±;ηœs2Q'UBD@D@D@D@N kΪ+ ς§‹ΐζ›oξ^xα·οΎϋΊωηŸί­²Κ*ιͺ€J+" " " " # σь=Π΄TgΓ 7t£F2Η33Μ0CZŠ­rŠ€ˆ€ˆ€ˆ€ˆ@ζH)Μά#MG…Ύϊκ+Ϋ_8ηœsΊαΓ‡§£Π*₯ˆ€ˆ€ˆ€ˆ€d€φf𑦑Jέ»ww—]v™»σΞ;έΑœ†"«Œ" " " " "IR 3ωXΣQ©ε–[Ξ]xα…ξτΣOwW^ye: ­RŠ€ˆ€ˆ€ˆ€ˆ@ΖΘ|4c4ΥΩ{ο½M9|ςΙ'έ’‹.šΖ*¨Μ" " " " "Zς>šΪG—ώ‚‡Γ9Ώp’I&q‹-Ά˜κΌβŠ+ZεŽ=φΨτWR5' •Β@Y,Κ`Ώ~ύάθΡ£]·nέܘ1cŠVσ˜cŽqR‹βΡ ¨›€”ΒΊ*j ΰwάqΥDqR «Β₯ΐ" " " " "P)…UαRΰz¬΄JŽUΒZεΑ4σZγ+žˆ€ˆ€ˆ€ˆ€ˆΐΈδ}t\&Ί¬Φ£R€jWP %)" " " " ™# ₯0s΄υ*„2˜„BG:Ϊ_ΨzΟW%H7™¦ϋω₯’τυšζWRf€ωDτYD@D@D@D@j' •ΒΪΩ)f…κ5gƒ·$Σ‹§­χ" " " " "ЎΪϊœB” ώ~ψa;#ή½{;•—ΜΟ!isOŽ―ΰyID@D@D@D@D@’!ΠΆJ!Κ ϋά “”EFG"Τχe+ΔΈžKkXOΊŠ+" " " " νH νφ’π…ƒΣ«yΰR KΣϊφΫoέ|0Ξߐ!CάgŸ}V:r w£(ͺ!–’ˆ€ˆ€ˆ€ˆ€ˆ€δh+₯0¬ζC¨τ3f₯89i7ωπΓ;({ωŸQΏωζ›–]»ΊYf™ΕΝ<σΜξυΧ_—R˜#£7" " " " "ΠzΪJ)oΌρκ~YZ1όξ»οLΩ+€δ‘θ…λόρGŽ¦ (|αΕ/ΌŠΰL3Ν” "~φΩg۞ΝάΕ:ί΄«r^'6E‚ΪfO!ΚIΒC”ώZY>ϊ補RΜ:ƒ’>Ηχζuι%·Ί‡r·ΔKtψΏΙ'ŸΌͺjΓ)‰3 γ™'@ρkz/" " " " "PΆP λ5£e₯,(†ρλΝzΓ?”5εDωϋν·ίrEšrΚ);¬ζ-Ίθ’>‡ΥΎ\½hma>š€RΎptςΙ'Ÿ4猯π}ωε—‘φVπ‚bWθ3Jag ‡ΧsάG|e²žς4‚}=εQ\H3ΆX)lΔΉvx1­Τ„τΗGΩ‹+zΑœσΧ_Ν}—¦˜b sΤ”Ό…^ΈΓg£bφΩκΒ>LΓ$„΄$" " " " " Ιh‹•Β$Μδ#Η )J!Η-₯ŽΧ|eΟŸώy‡θ8b Κ^±>ΜT³$I¬ΦfΙΙO–ž­κ"" " " "n™W YΡKj•*ώ¨'™d‡WΟώ9wy²Ι&+θœ%ψρž#ΪQjU Q1=•Ωh;~kTgFhOν$ͺK-΅”Ϋzλ­;(SO=u)g7 ”B€Zo€˜ΞΆγωKD@D@D@D@D Α2ΏRΏF˜Κ”±φo&«·(†Ό’°2Θ½`¦[(œ‰€ˆ€ˆ€ˆ€ˆ€ΤO -”BΜG‹) ΅"”²R+ΉΏβρLΒsΑ&’(„œCΘ~Mώ$" " " " " %Π6Ja’G"πH΄Ώ­±_L₯." " " " "Π­žA0υLκŒ<Š£cx(JBD@D@D@D@D %΄ΕJ!€kυ|Yθ)i•°]H#ΆX)δΑ &±GM«„iόš«Μ" " " " " Ε΄R€z:β£\JD@D@D@D@D@D +ΪJ)d₯ΣΟj•ΓpNžΒ¬|νU@ ­”ΒPi”»RŠ!Η"!ά¨Q£1= iκUD@D@D@D@D@Z…@Ϋ8š)<¬<Β霼b΄t]D@D@D@D@D kΪ^)ΜΪU}D@D@D@D@D@D mi>Z …,R˜ε§«Ί‰€ˆ€ˆ€ˆ€ˆ€ˆ@R ΛmΘ2)…Y~Ίͺ›ˆ€ˆ€ˆ€ˆ€ˆ€”! ₯° ݁,R˜ε§«Ί‰€ˆ€ˆ€ˆ€ˆ€ˆ@R ΛmΘ2)…Y~Ίͺ›ˆ€ˆ€ˆ€ˆ€ˆ€”! ₯° ݁,R˜ε§«Ί‰€ˆ€ˆ€ˆ€ˆ€ˆ@R ΛmΘ2)…Y~Ίͺ›ˆ€ˆ€ˆ€ˆ€ˆ€”! ₯° ݁,R˜ε§«Ί‰€ˆ€ˆ€ˆ€ˆ€ˆ@R ΛmΘ2)…Y~Ίͺ›ˆ€ˆ€ˆ€ˆ€ˆ€”! ₯° ݁,h₯πϋοΏwώχΏέyηηξΉηχΕ_4όΉΎψβ‹ξ©§žjx>₯2xι₯—J–α›oΎqC† qηž{ϋόσΟK%₯{ (χ<ΘBIˆ€ˆ@Kˆ’ΘύχΏu—\r‰»ςΚ+έ³Ο>Ϋ©εϊ裏¬―΅Ÿ~ϊ©»λ»j^2ή£>κή|σΝ’ašyσχίwwάq‡3fL3³­8―V/_ΕiRΐ|^=φ˜{γ7š”»²I ΆP χΨc7υΤS»]wέΥέrΛ-n‡vpϋίέ΅Χ^ΫΠηtι₯—Ίc=Ά‘y”Kό²Λ.+Y†Ν6ΫΜνΎϋξnψπαŽF£ωυΧ_έ–[nιώχΏΥ“LΣβžqΖnΰΐMˏŒΚ=¦F™‰€ˆ@ƒ0αΈώϊλ»%–Xyζ™ξ˜cŽq -΄Ϋj«­ά?ώΨ \K'ϋψ㏻Ν7ίΌt wQpϋφν["Dν·φέw_wσΝ7מ@Β1ψαΧ§Oχκ«―&œr2ΙύτΣO-]Ύdj™\*ωΌϊυληnΌρΖδ2PJ™ y₯π€pwί}·£3xν΅Χά<ΰ>όπCwδ‘GΊmΆΩΖeβIΦX‰‡zΘV a4Γ 3ԘʟѾϋξ;wέuΧΥ•F3#_uΥUŽ†R"" ",Aƒ9VΏή~ϋmχςΛ/»·ήzΛ=ύτΣ6mφd\²5Sj"~C‡u»μ²Kϊ+’$J k’©΅Xb£Frgu–c™|±ΕλPΊƒ:ΘΣIΝ9眦’8Ξ;οΌΦi1ΓΉόςΛ›9ΗM7έdJΧ^xaKλΑt¬ŽM0ΑffρόΓmΈα†n¦™fκWθ'™d›e\`:܏@±BYΑΜpI'uΛ.»¬Ν††₯ώ‡~Ψ­Ύϊκξκ«―vό±[m΅Υά:묓K‚ΩΩ[o½Υ9­ΈβŠΉλωoΎύφ[wΡE™RDšM4‘•ν?ώ°TfD§vZ›U…ORε;ηœs,ΨΰΑƒέ[lαΊuλf ω>ϋμ’;”PΜT7ήxγ’ΜΏώϊkwωε—Ϋ}ςήqΗέ”SNii`ŽΔσxδ‘Gάd“Mfu\sΝ5sιηΏΑ|—π(<7&ΊvνκΏώz‡)“=zτΘΝώbtύχΫ3…σΚ+―lIώόσώϊ1K ·\Πm½υΦ―JŸG~ΉυYD@L`τθΡξώοάl³Ν–«Ζ’‹.κŽ;ξΈάgή”κSθσhλi‡―ΉζΗ6M7έΤΪ_,}θηhWYeΧ₯Kλ#ι—WXaλ#'œpB·ΑΈ^½zuΘ3ώπl)‘oYk­΅\οή½s·iίιΨz‚uQ~ߞ 8φΝ{ο½ηθ0ScŽ9άΆΫnkύ(·ιΣθ7`€0v0`€υ#!]Κ@OΏE™±j’Ažyζwϋν·[Y·Ϋn;wί}χΩ€²‘/«²XCM<ρΔ6ΰŸ|ςΙm<'VgI“x€ωΛ/ΏΈ³Ο>ΫVnaIHŸ·κͺ«†μμ•ΎοόσΟw―ΌςŠυ‘Τ‰>)U_ χ1}.c—ΩgŸέψ„ΙhΜ‹—\rIΗΆΖ$\ίyηsόHŠ±Τm·έfεη{PN(3VPŒ9'-½τφ=!Χ°άyηwl ΈύφΫۘ…{ZnΉε"―δD^ιώφ·ΏEΎa|G7N~Ύ‘‰?όπΘU#oςiixσΡΘw ƁψΎsŒΌβyEΞβ—+°rϋΖ(:βˆ#"―Ϊ3ρ³Ε9ρΔ£Yf™%:μ°Γ"ςσΚ’=«™Œύΰ;–Θ7Z‘7ŽΌoδ;ιh½υΦ³»~‹±τ`tΕWΨ5ƒ7l½Ymδ;>{άόκ«―¬nήόΙ{₯7*Η«šηaΠ?ȁnΈ!ςΨΘO F~b6ςJΠ85+Χ§„>oώωη·tό$iδν/ύ€7‹όdk$<}}ω^{νe}¨ŸπŒΌ2byηχIτ³τq^ω²vŸφ΄ΣNΛ•σΰƒΆϋ€΅ΤRKYJ~…Δ+6πΓΡI'dε£^ι΅ΰΤΑ+Ή¨\§Ώ1b„][d‘E"Ζ sΟ=·υߌ]¨/ύ β•Aψ‰ΙΘOHZXΖ'wήy§έ§oςJ_Δ«WΖ#―Fm΄‘υ‘τ₯»νΆ›‡ˆψ K˟~qέuΧ΅ώ”ώςτΣOοpŸώ–ώΠo3‰¦šjͺΘ―.Ωύrυ΅@±^΅gKϊoκIΪ~ΒΦBQfώ(?γξS6―¬Ψ}ΏΏΡϊd?‘lε-όž|ςΙX.½…+υaΓXgζ™gΆq!όͺuδ·E^)Š9δw1nςJœ%ΰMy#žΧ¬³ΞyΕ͞Q!Ύ₯Ύ?εψPWΏ`ΟΜO” žƒŸˆ°2xSb{ή+­΄RDyλψΙ‘zyQ―“O>ΩςbμC}Ος{%~7a¬κWϋν»Γ—ίμxvό>$Ω"ΐlHfeΞΟ v¨Ÿ7!5%ΗόDόω™'»‚BγΝ(sαύž³hΟ=χΜ}¦qB)γ:” ?s’ γWsq‚βgΕμ>JJ‘_iΚ…Ώρ+TJŠ·εΟ]¦§‘@θπ(£ŸiΛέχ³eΉϋ\pύC'D~4¬…”ιωY1ϋθg(­ #iPАrεϋςΛ/­|~†ΨΒηwΐ\ΜW σ™£P’D‡†‘‘Ζ‹† ‘cFΒΐγΒ / ;Όzθ‘‘ŸΕΚ]C™GΩ ƒκ:iξ‘`ϋUΦ\x?g<όlcN)$Ν εxΥςΤo1‰^xα+7e ‚ςXL)€Œά ύ7e₯ίς«½₯%%(~eΖςχV-ΦN3Ν46ΉΚrρΕΫύΈRθW›r}'©LD{η>!ŠM0σ zKœά}ςbβΫ―tζξΗώ(L‚#εκ›KtμyΏ2—»όάsΟYωΓ %'ώ\ύŠa‡ϋŒ½όΆŸ\|&sω^S η›oΎθŸόg.Ό_U΅±Κ2Š;ωΦŒν˜€φ–D%Œ±Z˜ΰη"JaœoΉοO9>δΚsΖ |G)ϋν·_όŒ3©―χUaΧκε•―’π3†CΘ%:Œu™\@9 βχ™ZYβߍpO―ι&iσQίψγeΣ?―Dψί–3σ ?;—σΞιfvb7ύ?φ#bvθgœΜœΣ7bS 6`ΑΤ―Θ…n5Φ°ύ‹α‚οr{υ03υ?d3Ήχ㯘ΐ`&ƒΙ&ό‘'ζA0ΐ¬5ζ•μΫ@0­π€™Uς™όXζ'Js߁›ΉB‰γ|g&:εΚβUϊšΟœ2ΐ³ώΉ$|cεόͺ­}φ+}Ž²ΊbκƒΉEœ.’ƒY0ω ζ#~8$χsLn0Ÿ β•qGω|§“3፛J”γUοσεΠ«ˆ€€‘¦ŠόΡ‡`’‡#•Π.{eβ>%τyl ?Œo™πƒWη­Yrxθ#邐ŸW ¬ ΧxύΟώcΞΥΨσΘ6“N?™jf©~€μΌ’ηΌυέ㟷ޱ­Ή ±7˜χ±eΒ+#f²Jθ‚©e,hΡ·Τ+˜³…rbrŠΗtϊ± Ό§Š‹WͺrΧΌRacΜύ09e_§W ;Œ_ˆgΙv?€΅ϊζŒK‚ψIYηWωŒS΅υΕD¦˜RΜ&‘ψxŠςalƒ°3LL9)_Ά πέ*$Œ[ΘΓ―δεncLcΣ±Ÿ.°φŠ°clιmž#[hβη[ξϋ«rίΎ#‘ ^Ω6GLŒE0Ϋτ ©ίBώŒsψ`‚Λ–$)^€ΕXσQ˜χΒa δ9μ=x6~ς>χYo²CΰOγδμΤ§CM؏‡‚χt†ΗI ώό,L‡π(‘tA°Χ§aΖC'οΩ3θg«Βm{Ε&;.tPήΤ0w‰F!.όΨΨ§PHhψψa’x`/Ξ>?:…Έp e/8B#ίΏΑ΅x}ψ\J°±§|8δ 4X~¦ΥVRΎότCΩΒu?ήΪk>sΚ@§ςη•(4ζtHμdαΡGνzφμiNƒ:$:φiΜP /χϋ9hPQΪγί‡|½9J^<;ώβΞhβΟΏ―zŸG(›^E@D MhηΩχŽS7„A.ύ-iή\ϝrΚ)ΦV§Π‡‘ A ƒWε+Fxζι“ιsΩ«ΪoϊS”¦Πί  β₯œvŸφ›}m„ Rͺ?υ+yŽ=(*ψ1@υ–:–vˆοσϋBΒψ•£Τ^ιΟ«°‰Χ‹z2‘—x„‚‹ΛΰύΤSO΅=tτ…ωΓ0 λbcΖυ¨€ΎρΌxζήδΠ1ΐΈŒρYΎ°G/HP¦Ι‹gΔΗ7Œƒβc‘WΎw<σb{IQ°ΟΕ…±D±Ύ>„‹σ-χύ©„ߏΈ„ρ#i#εʘ/ςŠ§Εη0Ά„? yό»Η}Κ*ɁL+…(4b~?`Α'η—ν ^χή{okTι4˜ες¦€Άα7Τ±Joθ™ ŽhB:•Ύώλ_² Τ4hlŽφvπφ9ž_©΄Θ—όγŸA_/τžΝκt@œεΘj*ή<ΐ)ΈJΛx°™τβΛRBθˆCώΌΙ†ΩLoήc|yŸ|ς‰•‰Ξ&ΜhΕΣf6“ϊpώ3ΥήΘ6†‡p‘¬4Ύ„α/›Ώiœ½ΩHΈΤα΅―zŸG‡ΜτAD@RB€%eX‘Λ&瘘£­΄OΙO£ΤgΪloϊ™ BŸΘͺ I\hΏ)‡ίγ–λo°Paά€"Eϋ’οJυ§„Γ:Δο_·U0βRGΞhDςϋΓB}aάͺ‡Ύ‹ρ}Š΄τ_AΈWh‚3άgς”BŽ”@IE1D©ΚOΰΤ%«w8‘«d SΎ!M^qΌΓ„7ω΄Ucϊυ ζ—'/ΌΗβ 0>ΎaΕ Ε·°ͺ―ψ³CDI¦,<{&ϊγrο½χνλγαΒϋrߟJψΔΏOΤ…g ϋ  ΖΛΘw‰1L±ρH(―ΥςŠΗΝΟ€ ίΏψwΥ]ψK²G ΣJ! ž8iθρŏή;2± Άη―ΔΕ1³‘ΜρcδK'nj&€A˜AΑƒΧ0‘ρΕΓe-B~4ς‘‘D1τ{$:δW*]<ŠQL!‚Ωe©xαήφΫooJJ4³ͺ€…*κE[|¬ή‘„Σ±Ρ±0λΙlθΰΑƒ!¦+}QJόλΘθЈ‹Ω3α¬$”A”uΜPPθ¨'3¦¬8ζ JΠž!lƒΉ)ρπNFgΙJ!¦©|GθΠ1=‘qf–»”γUοσ(”§‰€ˆ@°βζƒYΏΙΩ΅LvŸαωλωεϊ”ZλΙD!Κ!+wXέΰ13_°>auΘ;a±φžΎ—-#~ΟΊ­–α “>Œ~?€ESL(ϋ½W¦l„ΙFϊ§ΠߐΗΠίΠaι’ΏΚI_ΛΆ’&ΚΞ ½±tBΡC™ΰX­R[Ζ.Aq|ύυםw"cQP8ƒpž2yς|PYM,5. ρΚΥ7„γ•M=Ώaƒxσ_‹γ»TͺΎ!NxΕ£;C½Ω‘y²Δ©ί.cΜΓs ½ΈΔλGύ½„\YqΒB™γε‹ΗυΚ·9 Ο–qW-ί?œΞρŒHΦΑλ*p4Ø1.8š‰σε^ΉοO)>Τ'‚8”?ϏιAΌ%”έgœψF.·ΑžY:fΊΰS©0ΓIΨΣYH˜1c5³’4yξ¬ώa>”ΟƒΩ.V}£›Λ†RΚ_ RΌHΈήη‘+œήˆ€ˆ@Κ°BDΏQ¬=/Χ§TZ]VΡX…dε…4iΧγm{±tΨCGί͞ΊBΒ +o•ŽŠH›Ύˆ•ΏbiΡΡ_δχW¬laΪϊ₯†ύy˜ “b}%֚­G§Ϊˆ€ˆ€ˆ€ˆ€ˆ€ˆ@ύ΄RX?C₯ " " " " " ©% ₯0΅Nϊ ΄•£™ϊq%›Ψ“Ψ¨=xπτ.­?ZΑωγ%κ*#γ –Uσ½ΕQΒ ›Έa\lΓu8CΛ»dΡr―ΨΝγ‘ {ϊBΦ9Kg5œY$θœKΛ9Ατ μΉ“t<²γΩ”σ³$ώΈΟh]–žͺκ2ο"ΈmΕoΆσVš ?ΏBηΜ$YοŠΩΞς›‚kNφν·ίΆσsΌr-ΎψβvƐW"Πj.M―0Z>œkΘ™;όy£v6•? 6w&ηE­΄JΉxρ7œ»δ:Ž_Κ½ΟOί+‚–g=q¦a\8Ês ύα»φΗΩ?ή“©ύ]pΑρ φž³œˆC˜Η|œϋ~#Έεε½nsσ‘όʘŒsSD@D@šB€Ά›6œ³ΣβηΝ՚ω›oΎω‰ΎZ£·u<ο%΅θΩΘώhˆh‹-ΆˆF:Fώ؏hΐ€ +wώψ°a)a(A ­ΝG9χ‡σcš%ΝΞΟΆλόα»ζΆΊ–:rΚ+―μό‘ΊŽsšžzκ)χώϋο» /ΌΠωCM¨βΒͺ$αψc΅ξξ»οvώ0awώωηΗƒΥό>€?fΜ›‰<φΨcέ 'œΰ.»μ²iξ½χήξΕ_΄ΏW^yΕ}ώωηn=φ°σtp£—‡zΘΞPΔ«υ*$Έ‹2dΘ8·ΏώϊΤx§πΊ " !€Υ –)œΉΗΡ υŠWZάΣO?]o2ŠŸG€3―»ξΊΌ«ϊfE] h[σΡ'žxΒ=σΜ3£ώ_Ώ~Ζ‡ƒ`QΈξWΖάV[mΥα@zβp@W΄έvΫmηξ»ο>·Ϊj«Y‡D―ΎϊͺΕG1Β|bΛ-·΄ψΕς#Ξ-·άβPN0ŸD9‰Ÿ‰T,=β <Ψς >n₯wΩe—ΊώφΫo¦”‘&ŠΝ•W^ι–\rIS–0…ΰΰχwήΩ©'½|9ϋμ³­žœIŠΕ4”:‘Rζ₯—^:?Zξσκ«―n ©ϋΎϋξ›»žΔL98ΰ;δ—g·νΆΫ:ΜJ ¦³ϋν·Ÿ;χάs&FkΉf.ηξPNκ΄ήzλ9?h¦©ΉώΝ&›lβό*£†ΞΎβ€aΎ'~%ΤLtγαυ^D@D 9˜|dϋύ_%tΫl³›qΖ‹φΕ‘T~5ΠέtΣMξ­·ή²ΎpύυΧ· Τχή{Οϊd}ι1.@‘Ω`ƒ BtwΥUW™©*}*ΫΠοv§v²1A©ώ›ΙιΓ_xαλŸI{Θ₯σΰƒΪ6ΞΥCy`;Δ†nΨα|»bυ!ΚF]ΩΒΦΆ€„~°Τ=βΧ]{ν΅ξΏύ―6ί|s7ηœsrΛ„ΓΡo½υV7rδHK7\/τzΞ9ηΨeκνW mόΒφ Ζ&τ͌I6έtSΗΑθΕ„η{ΜT™ψ^vΩe]Ÿ>}rΑ9¨F~υΨωΥ=·ξΊλvSΑΰώϋο·ν <σx^₯žW.ƒ±oΎώϊkwωε—»Χ^{ΝxμΈγŽΉ-*ΕΎ”‰οκΗlΗ90n‘ΞΕΖ‡₯Ζ£0dŒYlό—_^}J΄νJ! ώΟ?l -?n„‡Α²/½h{ν΅—)|μΝChhθψ‘³7ŽYIV£ΔE]Τβ³ηnŸ}φq4:H‘όΈŽβuΦYgΩ=VΤX™£sCJ₯Η}˜Ν6ΫΜέxγΦhΣaΔ…ΖρΰƒvοΌσŽ]&/”ŸγŽ;ΞφŠ+0₯.Τ/—χ#FŒ°ΓIƒBΏ‚uΨa‡Ε/σžΖVq%wœ@u^`― s)Ήϊκ«­Α^a…rΑxξ7ί|³1Ye•U |Ύπ`šοGΎ#(¦¬’JD@D@:‡V)τ―τ›LζςZοdRs‘…²}ξL˜2ˆη`xV‰Oš(K€GˆC&F,b˜ψ<δCμμΐΟ>ϋ¬lPΨs˜=ύωΣηΖ»νΆ›υUτΗΔcB+€T}ΈI'ΩΔ(}γΖ L†—»‡BΘ^{Ζ읇}“Κ έ:λ¬γŽ8βΓ0ωŠΒVLθ«φ†ρΧςΛ/ŸKŸώ« BBό\Πa₯ΓΈ‚qXίΎ}έyηgΑ)/“ΰLv3Νs™oΎωάW_}eχ Η$Ύίcγ"ςBIDΚ}g,ΠΨXQQΖ œ5Θ€1ί'ž;Rθϋΐχε•0LT3!Νx…Ζ‡εΖ£εΖc‹ͺ¨Ž€oψΪVΌω†Ωκο;Ϋ η΅―Lεφ“ωΖ&ςΞM"o²˜»ρΕΫύ;οΌΣy')ѻοό‘W#―xΨ΅x~\`O‘oΨ#ςF|Γnιy“Oϋ\.=ίY|ΚVHΌRfι‘.B~^)Κυ&–vί›Ιδ…7€ιgα"―ό…KE_Þ?ίqDŒ.Ίθ’ΘwΡ<σΜωŽ$ς3z·ή=…‘ρ‚ψΖΩκ³?Π―ΈF~&Τώΰλ;ύΘΟlFΧ\sM‘Wn->όŠ[„x?nϊ>Ϋλ Ώ ^)Œόj }τΦ;–~ΌΌ^i ω­BΎόqˆω ςΘ+φΉΤσ"@|O‘_ ΆρM(£Ÿ8ˆψ>0ΖC }ΨKΙX"ΘsΟ=ge γ―ψψ°άx”4ʍB>zjΆ·«N―ΜDθ7ήxΓfpΒΚ•Βs&39,ω³ΒτΕ_t0S`Φ‡Ω(„™#ΩΎx<:t¨c f(Φ PI_ΜΦϊVζ0Cτt[±a…°VaEΚ;9³θ•”+V ΩΆ°φΪk۞χ|oΪρ²`Zχ捇UΖ&HΉϊ~Ψ'ΙΚc—V©{Αγx|O=f―Ο>ϋ¬™Σ’ή2Λ,c&©” $ΖPπ¬Dΰω'γ… Œ•θ›ρ_0묳†ΛφΚψ k*,‚ίςκΡ£‡έ§

JϊNϊ/ΆR„=i(&Υφ₯~5§Ceγρ+)¦(?˜nΏύφΆ'Ε%­pΤF\8Ξ*˜E–«[=‡ψB‡"Dxψα‡;ορΫΆ»G=˜¬i§`Κϊγ?ΪD:ύz\BΏVμ=ιs>LψΖγ2aΘ %σYž›_15sTΒ‘HφκΥ+%χž{μί‹αΒΝJžWΛ+α™dŽsa’=>ž‹ˆγW•mλα(7ω&Ι„CBy ±‰sΙΟγΟΨϊ/΅h{₯0(DαΗGγfaΨK€ ύ‘‡j«]4†(α>+E4Œˆ7[΄WμγQD‚ΰ”ε!HΘ/|.φZizΕβ'q³χΨ+ΐŒ\\±Εeϋή\Άβ¬ΰπξ»ο:φςΕgDIŸ†.¬\Vœ θΝflC±ΈtϊήlΤΒxσ ‡Σ„Ξ˜:±Ι?.άχ¦‘ξƒ>°½ρ{¬²BΘ6ΫKD@D@Z‹@%}'JQp JΟ^ϋΈ%KΌ―ΖG{δγφκΗ―…χεΚΐδ1+Wόη?Ν36Φ:8CI+¦2ή LΑ2ˆ'φΔ!εκƒJXVρPfq ΓΚ!{qbSμ“­μWd/^°δΑ1gϋ²ΊG<φ7Ζ«œΈεNό^xΨ2ξβ¬ίΈψ£lε1ŒΙβχόQXΆw₯0LΘγ˜&<7ΈηOΤβ€±γρΩ[‰PVεSO=Υ>—ΏY .(˜qε¬zζO„πxΉG1'/Κ‹0ξABΩyηΒηbγQξID Ί4"Ρ΄€ΙJJJ˜AΒ€g/x₯‘>ώψγMYΑ‘V”PBPτhLŽ<ςΘ\UYCi ρ€qΰ‡Žs:žΰΘ%ž_.b‘7•€W$jb—QnρRFG‚C–`ϊΐ +|0§¬Fό…6»†Ι ΥY}₯£σ{ Ν₯œΣ8\Ž³βGγŠ™ +<“RBgΰχœΨΖnžνy6x©Λž!κƒςΏΟ=& p:PL)dΦΕQ"" "Π|•τ8a"’Ύg)(„8aΒ‘―Fyΐ‰ύ…ίofΚf“ΔC9A±+&εΚ€BΓd3Κy’ n0ι,”. ΞEOL?ϊ°rυaΕ‘>œΌHƒ:’•ΊΗ &«VŒ…˜Μ…}ω£hαδ…k8±#M,iβ&”ωυ`+Κ%–:8›ΑK+γ/”TΜRQ@qΐτσ…z2)Jϊ\ςFp2Γψ N„ΓΜε— `L-±ό JΗS1†clGΉΚίβeA™gˆGWAΎŒƒεR<,οyή(ΚΤaόŽN eQ8KG-‘Όxo₯ή¨‹€o ΪV|γq»y3ŒΘ7ΆaΩΟΔΩαλ^™ˆόLMŽAGΎ1° ιΎρ6',Δυ3vΖο9Œ6ήxcΫΜμΫΘΫΆG—_~y.~~~…―χ atΟ=χT”Ο<σΜ\ϊωoΨόMω‚ƒ–BωΕ˟ŸΟΎ#ˆΌ©ˆ`OXώΌyFδ•£\πΰh&δ“»Qΰ Ntόlœ9±!-œχxε.ς^Π^ ι‡όa„σ6—ϋŽ±Cc¬ΰΤg6ŒΌΩiDωΌ’yE)γ―·8‘₯^™‹όΓΘ+Φφ ½Rmq6γMDmŒAΉp&Δ―pΪΈ €<{ž9c€άσ‚―WΊ,,ŽeΌΥ™2{λ‘Θολ΄{όΓΡLώχΑϋš°q‹7ϋ΅ςαd‡1"εφξE5a ΐ({—žŸ(Y]σΖΠ61žlψηjiŽ―βΡ9ώ«όb$‰ΐ;_eΎύ!}ljΔ©ΑFŠFj4§Ÿy½v–›9k°Τ†–6՞³‘‘jTm>Ψ€λg―7ς=ϊύ[yς¬xΎ‘΅6’7’υnj„Ά%šνΝΛaή-S~ 0>;›1jdΛCΟRŸγΕγ3ͺ™Άφβγ…³|ΗηκπL™$Ӂ8·“Ξο³\Ν$Ν€]½Ÿf#{OεŸΣoΧΊ·x§ύx¬ΊήΑzΓj.ψΈ˜«Ω³γΉΕΥ«γ*eΦ}#›ο/¬­>ν»<«O³›ε­sΊj―γύΕ1½NqΜ3©z­)gυ:‹Ε£ΟΌž{Ό*πρΒWΐn_±p.‘–RtbTϋνZ~Ϊ—ΆžύƒΏRΆ{  @€³ΐ±S8_σ˜%πsΡφ_•όΛi[λα;‚ΉcejS΄α?llo‰l Π ]³^V ˜)\©«l @€l.`¦pσR= @€¬Π)\©«l @€l. SΈy© @€V θΤU6 @€6Π)άΌT @€+t Wκ*› @€› θnή@ͺG€ @€•:…+u•M€ @€Νt 7o Υ#@€ @€ΐJΒ•ΊΚ&@€ @€ΐζ:…›7κ @€ @`₯€NαJ]e @€ @`sΒΝHυ @€ °Rΰ•«b_ƒΡ(IENDB`‚pytorch/docs/source/_figures/remote_2.png000066400000000000000000001544271444262304300210420ustar00rootroot00000000000000‰PNG  IHDR™―ΣΆBbiCCPICC Profile(‘mΑKQΖ?Λt‰„ˆ{ ΓΒΦ@ΊfQ‡ΕŠ²ΫΊk₯φΨέo‚ΞAΤύ:wˆ H(Šα9φR²ΝΣj΅šΗ0?>ζ{oή]‚ΒXΑ  X²ŒΤόŒΈ–^}u0Œ  *ͺΙⲜ€|ΧΞ°οΰαυvŒίu]~]ζbϋO―υήσΪγίώŽπg5S₯ϊA)©Μ°O„Xή΅η=β~ƒ†">β¬·ψŒs¦ΕΥfΟr*A|Cά§ζ•,ρ3q8Σ¦λm\,μ¨_3πι­΄²Du€r³˜C’Žb˜ΐhG{&›žΆΑP† θΘΓ"wœ†4βE” bab J‰οϊχ]-{DGι)Ώ«mΎΥ xεj#[τiΰ2ΝCωΩ¬ΗφšΉ¨Τβ@θ9vœ·Uΐ5Ηy―8NγθΎ'―ύ Ÿe"M7ΣVeXIfMM*‡i’†D ™ ASCIIScreenshotpϋfΧiTXtXML:com.adobe.xmp 285 1177 Screenshot 3τ#Σ@IDATx일EΧ‡§΄Έ΅8/V xяRΌXqw§ΌΈKq(ξ^Š;www}±bŊ—’ZΨožΉ½›άM²I6Ή›δ~M³;;;ςlξξμ™sΞt ¬‰ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€TA`Ό*ΞΥ©" " " " " " " " "ΰHΙ€‚ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@Υ€dͺ‘ ’IΏͺ HΙT5B " " " " " " " " %“~" " " " " " " " U’©j„*@D@D@D@D@D@D@D@D [΅~ύA΅Εθ|hN4‘Ώ[ΥΓ¬šς;v¬3fl{]Ϊ6ΫΎrvΜΈ=›‘==.βΫΣΖε° ΉiœΨ₯cšM±Ι€K\b‡\J¨7ͺG?#ΏϊΪόϋοΏυn·κΜ˜sŽž™m›oΨoΏnΎώζ;ΏΫίqΚ§6}T¦Κνε)ͺΪΐО·=­==<9˜›Φ–7Lk?Υ•¦G6"›δv’£r—‘ν+g'Ώ΅ωayώ¨MhO£ψφ„Έτimeδ!lkwLbμuqEt¨i\CK§»ν΅ΥέΦ,·ν{–وl†yΓ΄HqΧ νπΈά‘“Ϊ6sΣ §εT{m"EΫΜv―ν'F€@Ί½Ήη·w"…iSD@R!ΠΕZ!Ue†$S*ΧA…ˆ€ˆ€ˆ€4ρΖΛvT,™ά °ύ?wrwΫ†‰mΫΓ‘ΘζΈ+΄&e\†œΤπ$‡ƝοOΙMoΫ‹KΛ©Δ—6,›Ών_XΆOWRιt;Dξ˜Χ–3tŽMsΝΛ-Αν΅ηΪΠΎ[N^Wxϋων»v+ξ°ΘΈ΄œœO{1γN ΟOž·ύ»Υ‘˜φ„φ|γjmΟNΒ8±νΞΝθγx“1&«/HίΔ)žΪτ„Ήj*·ΧώŸ+­}·=oΈΥ~0//»mΉr³DΪ³ΈΌmGΒΣ8jKS}qΡR€΅•2°θ—+έί^ΛΈ-ϋ՞6Œimιυ­ΆΥΉ'·Υ—H+bΫ’r q{yyΫΖ΅ΟΥ2έα©mΉ»qiνeδζmKο˜F=γRΓƒ‘΄φβΖεΚΝΫ~J{z\Z€»—w\{¨ύ|¬XVlJΒqY;ζtτ_QU+™Š–ƒ" " " " " "ΠIβRmŠ«\•”ΫΛΣh΅₯ΡπφΌνiνιαΡΘΑά΄ByΫ;ζ(υΒƒ‘΄φβΖ΅l\¦ά/›«=‘½˜φ΄H1mysq؊MlϋΧΆλUκ%W &½]γ7ώφ€φτ‚iπ?O%ΔΓFXJΈ‘“ZQήΆ’Ζ˜σ•³Σ^·ί²‡Ηεhοm‡΄qύιή–{ΎΝ{½Hξs\ϊ*D ©’σ ηuGΓ*œ*«ύ?—ž»Ϋ¦μjϋŸ‚ύ©…Y˜Ζ᜝q™#©9‡svΪλφeΨΓέΊv53L?―΄δwΥξr%kPθI­A:‘iͺR2I NρδTQy ©Ά4Ί«¨ŠK/žΦ^FXRd#²9ŽWD)Ψ~ͺmE˜3§Iaz‡ΓγrΎrv"%Ά§Ϋ ΫB¦im ΝCΨVF¬ΠΆ:·P—7φΪΈ|Ή™Ϋ“ΪΣ ¦΅5Νώ—·νΰD6Ξd9"K¦rh)―ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@,l ˆm²E@D@D@D@D@D@D@D@²F@J¦¬]΅GD@D@D@D@D@D@D@€”L xΡΤdΘ)™²vEΤh@R25ΰES“E@D@D@D@D@D@D@D k€dΚΪQ{D@D@D@D@D@D@D@D  HΙΤ€MM¬’)kWDν$ %S^45YD@D@D@D@D@D@D@²F [š ϊίώg.Ήδ’œ"§˜b σŸόΗ¬²Κ*fΘ9vύυΧ›§žzΚτνΫΧ 0 ηX©?ώψΓόωηŸfͺ©¦*šυΥW_5—^z©ιΡ£‡9ρΔ]ށΞίsΟ=;΄©haE&mO‘"ά‘—^zΙ\qΕfši¦1Ηw\Ρμ·άr‹γχΞ;ο˜I'ΤΜ=χάfχέw7³Ο>{‡σ>όs3Λ,³tHW‚ˆ€ˆ€ˆ@kΈόςΛΝΛ/Ώ\΄σ+­΄’Ωd“MŠζ)v0nV,Ώ?φλ―Ώšƒ:Θν2b\Tkωξ»οΜ‘Gιͺ9τΠCΝ¬³Ξšz•A˜/ΏόΜ4ΣL©—­E@D@D μΓ.5Ήωζ›Ϋ©ΨO·nέ‚8 ψχίΓϊvΫm7—w›mΆ Σ’lXεT0σΜ3O>ωdΙμ7ήx£«Γ*XΒΌVαδξ½χή0­šrΪSͺžkΉΖ΅mŽ9ζ(˜†ύο]Ύ|ήγ?~πΐ„ηځL°υΦ[v¦iCD@D@D@ΆΨb‹Ψ±Dtl±οΎϋV*n–€ΐoΎω&lΫG}”δ”ͺσ|πΑaV9VuyωX…^°δ’KΗsLώ!ν‹€ˆ€ˆ@ΣHΥ’)ͺ5»ςΚ+₯d˜‰ΊγŽ;Μc=fN?ύtg9δ­–6Ψ`Σ³gΟ²­‰¬ΔόσΟ?Ρκ n/΄ΠBζ”SN1XTΥJΚiOmxε•W ³]»vuΦN+¬°‚ωδ“OΜΡGmμΙμΈγŽΛ%δ²Λ.3VqeV\qΕ4ͺV" " " MBΐNφ™ΥW_έυζγ?6VωαΆΟ8γŒΠrhώωη―ͺ·•ŽΓ&›l27~£ς©§žΊͺ6dεδC9ΔΌπΒ f5ΦΘJ“ΤH@Ν”LΦ:Ɍ7^[Θ§½φΪΛlΉε–χ8°(e¬e“™d’Iœ^^p;»α†ΜπαݒΚZτ83mο‡;Κ+δξ»ο6?ό³YguΜUW]eώώϋo³ΡF9₯ Ϋ›nΊ©™`‚ :Ταλβϋ‘G1?ώΈ™nΊιΜΖoš/££}™h’‰άφύχίoΎψβ ³Ψb‹™ϋΏsξqν!σΘ‘#‚ν³Ο>3σΝ7ŸYύυMχξέ]9ώ?”C·ί~»ω駟Μͺ«κ“‹~?όσξ8¦Φ|°γΌΤRK™Ι'Ÿά{μ±ΞuξΗtŠ'R¦Ω°[k­΅œϋ"iτρ™gž1Ο>ϋ¬cΤ―_?³ψβ‹sΘΙ°aΓ uΝ5Χ\lΪΩ§O³ήzλΉγƒί¨Q£ Χ v†nάΩm_Ÿ~ϊ©ΉυΦ[έuκίΏΏsι»σΞ;o³Ν6 σβjωπΓ›ί~ϋΝ,Ίθ’Ž•ύ„™΄!" " "ε–[ΞπApΧχJ¦ 7άΠMϊŠ ±p'cf­•Μ‡~θΖσΞ;―αyοΗmqγ°›nΊΙΗΈˆ1•΅ΐv“‡Œη^xaW-cB ~<πτΣO›χή{Ο…Y |ΖŒo–]vΩŠ›ίݍ;iγΖ‡„#°Φΰf«­ΆrcPWx‰’ΦΙXπž{ξ1ΈώςΛ/fΖg4L¦φκΥΛΥΐ1Ζ…c3ΖdL vι₯$CϊΐX•°Λ/ΏΌΉνΆΫ c4X3Ύ†±ΪΑψ˜qγBΕ‚ .θκρyFm¬χyϋν·]™(};}}‹€ˆ€ˆ@ΕμΓ(5‰ΊΛYK£œr­2#4AΆKw,ί]ΞϊΒΦ²)Μg;εΆνƒ/°ΚηjηΣό·XΈ²Ό œ}˜†η?τΠCAœ™ΆΟkcA…y)ΟΞ”/Ύψ’+Σl_&Ϋ^μƒΨ₯uΤQEΫc•/―Η—ccΑoΌα‹ }τΡΐΒzμ@#Xb‰%ά~1w9;ΠΟ± ™ΐΞ8v –λ7l\0ŸoνB0 ·±²rŽSΏU†.ƒvΗi“΅sΫK/½΄;ΊλΛ9Χ—Ϊi§ΉγόG;ύy§|«ψsηqM½œzꩁ΅ΚΚ)Ο*Ό«ΌςYτ-" " "PCŒό³άN€εΤδΗ3ωc,;‘τξέ;<ϟΟ3ώ«―ΎrečΓμδ•;Η*YržΈό?ψΰƒξΌ8wΉvΪɝgFΡ”S/c/Τ=Ο<σδ· Ÿ€ςi£Ulω¬9ίqξrIκ$ŒUεΤG=N8aπζ›oΊ:¬r¨Γρ±cΗI^{ν΅ξ\;X…YN9”λ…±²΅\Ο9N;Φ^{mŸ%°Ÿ΅NΛΙ3ρΔŒλ$" " "Ί­.7Ϋl³Ωη\›XŽίΜωΖΕnĈΞΪw/¬|pc–‡ ΎΙγg΄Κxψα‡η”eΞ AƒάμΑ*‹ e\p+›™³~ψΑΞ.vNτX‘φ0C΄ύφΫ;λ$Ύ±ψΩvΫm MXu!v`a0SǚˆΐηV!f¬‚ΖΌϋξ»Ρ*b·abc2Ήc―½φš!9AΥaΜ,$e#Μ 1‰0Ϋ;Μή©6ΜώaΉ4tθPΧ.;2C† 鼝ΩM¬‹ΰjc3Έ€λXL!ΜΔYΕ™Ωa‡άΎULΉoάwέuW7›‡•VJΈKήwί}ξΈΟΎ Χ‘™EΜσŸxβ ΗΓΖΫ2gžy¦Ο¦oN&?ΖΊϊκ«έΈK",x¬‚ΘYΟ`ycγ^–l-ξVβ¬¨°š3fŒΉπΒ Kž‡…4γ©Χ_=η0~ρrΔG˜χίίY§3ΖΑrκ­·ήrεϋ<ε~«“Ύc=ΔB7ŒγΎύφ[7^ΒڊcοzΘψ̏gΛaˆ‹»πν‡aΖRΘρΗο,ž°tgΡ,Π±Τg gdύχ7,c' ݘ‹±γVΖ€XΥKD@D@D jihͺ|Ε,™¬ϋV8kβgKς-™¬RΑε±~ψ³[Φ]-°Š_|ψν­^’Ώύ,[~€ΚΈ4Ÿ—ϊ½Dgπ¬9sΔ’ΙŸ›ί«Hqύ°.v5™vΩ迟Ac¦ Λ#{ρά‹/φαοŠY2ωΌvπXΧ΅€(_ίkΉf`j.›„ΈcΜly±!—f•uAΤJΛ*Δ\ΊuνsY½%νΖΚ,_μ Κ%Y—ΈΛ.κ¦ΟH΄VqδψoσΝ7wωΌ%“]9Ζν/³Μ2a;riv¦5κΧ6ώ½ΦΪ'΅[D@D ωDΗA…,™ςΗXP°+ύΊ±ŽUTVαΨ•nέ3όδ“OvβΖaή’ΙN† ­‚ĝg'¦\Z1K¦θ˜!j)O[i§Φ•ecUΊ}ώ³Κ+—ΖX₯K¦buZΧ4W6یΛμ¦G†•ΫXyε•]Ύόΐί₯zK&ΖcΦ Π•†Η€u“sεyλ/Ή‘Q«.¬ζ­[‘λ3ηFςΨΙΏ°yXH‘f'Γ4mˆ€ˆ€ˆ@₯Ζ³•ΊˆχC§2βχΔ Φ>Xδ—‡«p30λ»nΘ:ξΌh~χI…D^όμϋXSEΕΒ w“ΗB aFŽΎΪΑŽ‹EδΟΕ’ΙΧAl*;p ˏΆ#L,°A,‚ͺ3σdαlΦBXb0!ΔWb†Λ‹5§v›ΔΓ’ν^ˆΉΏt03„Μΐ[P‘ί³"Bpr,™Ό`…ϊY@8ραw€XsχΠ*+zNΦ·±Ζ².†ΞςŽ€λΜ,bω…•ϋXΐρ±ƒί¬wE큐@ώ‹g> ‘0Ncβ1†@όx <9f£§]όΕ‹_ ε―ΏώςIΏγΞ#3ρ8‰νi'ΖάΉXXyΙψτ€ίΕκ\d‘E\l$ϊLœ(¬‚ζœsNgNL§bRCΖlΔ3E°κ'8:3«¨rρ­ΨχqΆΨΖrλƒufώΊӏ»ˆο„0>•ˆ€ˆ€ˆ@΅jψ;ΏavΩV—ΔC“θ8aυLŸοΊλ.gfβe&ΘΣO?}7Έ2ό7ξX~šW„uα³–N9Y£ LŠK ngm9μ°Γ:dG1ƒ)5‚kƒ!L¬―tq;ώ# 9ζΡΰΎθ’‹ υ1ˆΰƒi6J';#ιL‘㊘yζ™]rTρG‚\(Ξ–…§ζ3΅³‹l‚‹p.μΪ@ΰr/‚{‚b§Ωΰφ1юŠg…r‘@œω{q"p€sϋ‹ΆΧ+£il£t‚%.›Θ:όρnτ'œp‚a2ꀓNrΑ·qΗΒmΏŠυ)°:I~_V±σPΒΠΖ4ΈYλ!wZώψΓ—•τ»XLᎢηž{š[nΉΕ­¨L(Ζ΄χ&X9y Ρ6PV”ΗS‘lBΩζ…Pcl?ζbEXώͺ}>θ:Η%" " "P)6@₯g'8% >όΦ]ΘεΆΓGςOΗ7χέww/ή—]v™a-ŠΞ:λ,—-κΫο²Qε/ΛσϋΕΎYΝ„X@ˆ/ίΊž9w”Z~@ΰ•Qί½[‰ƒό~&ˆm_§oH³β +vμ·ί~nε:”-(oP(Ω€”ΞΚ‡σY‘ aPΰΫα όNJ%('X%…ΨE΄EοOO %$Ώm€yΛ"Vta ˆ° 1 ¨eϋΎ ΆκDΑΔ ±°€"Ύβ*Δ πΚDβ6Ρ7¬Ÿˆ–6F8Ÿ}φq¬°ζ’m3Μ0ƒαz4‚`½sβJ•#ĞΰwΖωΘ2όρ«œ!ΦυέΕ‡΄rဝtFV[m5W­uΥsρ6Q΄\|ρΕ5kΚ /ΌΰΖ06„/“ΥέΖ~~ΡσσγEŽ§Ε…“ŸΠc<ηω£LΒ}ο½χ6=­ε˜·Γ²‰ρ)ιŒs§q\"" " Υ¨™‰ˆˆaΆμ2( N<ρΔ‚mζ8nrΜ΄0„’‡%V‘UVY%<k¬pBι6Α+r£pιή½» ΌM(:l\!W¦Ξ˜o³Ν6.9J)fˆς%=˜Œcm„K&Ι(0Gf›™I'ΤΝna‰΄Λ.»Έ€ΨΟ=χ\hα”_GtŸΑ”^+Xχ3”VXXyŠo"NΫ,›Px|πΑa›|‘†Y5ΛΨ’8’ο₯%(‘°.Β‹ΰήπσƒ7)\sf±pΒMμœsΞqνe—o%ΆσΞ;»€ηΈΖ18ΒŠΎQv#Yψ|έތ^«€ΫΈΡωΏ“€η(Ÿˆ€ˆ€t&Ζ X }φΩ.Τγ$ΖMˆ“tFϋ°ψf²’IN\ϊyΎϊ±]-ΪChGԍαXΈ†p~ςˆI7&ɘ|£mδN“!nψLό]uΥU7=Ζьu“1vd|ΝΨ™± &”cŒ·ξΏ~cceš=φΨ£xT¦ˆ€ˆ@«°ήΤ$ψΫrtA­λU`]‘Zό:*ή&ŸUβ„ΙVιXI$› Ϊ ¬UP˜Η„.$ΐ$βƒy[K 0q'£y­ΥŒkυXe `ι °q«ΐqΗνΪυΑΞψΈ}9{‰k]ΑΝΉΆt—ί,άώ¨Q£όi.x#}·³O.΅„ ¬™΅ΫΆ’0_άKβZ…LΘŽτΛΊœΤνελ―Ώ²Νq걫¬ΈCΦΒΜυΗχΟZΣvPXΣnjΰ“ž/v%Όΐ˜œrΚ)]@MuS¬Έμ,ιKpK|Zb`/.¨$y¬΅WX€ε l £ƒΦΝ0<žυ ›Κυ›~UϊαΪΐ@"" " A Iΰοό1–U(v²*°OξωΧΏΐ?νΚΉqγ0ψΫ*Cž{Ε[Μ₯ όΝB%^μ^ψμυ‹­pΜZ΄ΆaΑZk­Ψ‰Ό0ŸU€ωΣsΎ­ΕQ˜ΗƊtΗ¬ΎK+U' {XO`'YΓ2μ„c`έτΒ:¬β)`\ΜXραΗ$aθ3>ŠŠ΅PreΩ0a2ymμ&—n-§;IΨ•ρΒγŒΝX.΄ƒ[Έn/—ˆ€ˆ€ˆ@ΊPˆ}ΘdNpίΒΊ…ψEΜΌδ‹UΦάј‘ͺVpΝΒΒλ’8ΑŒk‘hœ’ό|…Ϊƒ» q0AΖ”9N¬RΘπ‘ŽrΛ*Ϊ‡εntq‚KΰλzȌV\iΦHήϊ,ŒΈ4~:ΤMμƒόΎΡo– ζϊ`‘δŒcΥ„΅˜πΉΌhΉX@aEy"ΜTb…”–ΨAͺ›ΥL«<•#" " ΅&€₯2γΖRYEΑ5 χ=,t,δ ΠΝ8ˆfj%Έ©Yε‘sω'ΖeΎ€k'Ζ…Ρ˜“i3ΔβŸq]\|›ϊ QBψvλ[D@D Ϋ2«dΚ66΅. όύ‰΅DπC=Τ™ΞŸyζ™Nι†I·Σ•€¬¬ζ±Λ/»LsΫ«ΪvΪYWγƒδW[–΁V$ΐB"Έή£<±–Χξ›Π Έ­`]ό˜Π‘ˆ€ˆ€ˆ€Τ†€”L΅αͺR-b l»νΆβ2`ωʁqVUŽUρͺ]±&ΏΟ5.Μo¦φE@D@D “°ή&ψ7A·£Bl$¬œ\pΑh²ΆE@D@D@R$ %SŠ0UTGΈ=ΠΧGO²βœ_}―cξΖKρ+¦ΩrΉΜ₯ISe‰€ˆ€΄"BΨ—0\ΨpO# ·_α­™¨Ο" " "PR2Υƒ²κhJiΗcς€dς$τ-" " " " " "ΠHβ#Q7RΤVh",±ŒςJ"" " " " " " F@J¦F»bjoS`΅@‰ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€4"ΉΛ5βUS›3C 1™ψ;3—W (ƒ€,™Κ€₯¬"O`ΡEΝOΎˆ€ˆ€ˆ€ˆ€ˆ€ˆ€΄$)™Zς²«ΣiXwέuΣ*ʌ7ήxζΰƒN­<$" " " " " " υ$ wΉzV]MI`0οΌσN*}“«\*Uˆˆ€ˆ€ˆ€ˆ€ˆ€ˆ@'%S'@W•ΝEΰΌσΞK₯CGuT*ε¨h&όσα“?!ϋοΏ&ξζθΡ£ηυΛ)Ÿφ3Ɵšψ;iIσεWœΟ,xΦφΛιg>σBΏ“¬υ±ΩΫ#%S³_aυ―ζVXaS­‚hΰΐζ裏y[Uˆ€ˆ€ˆ€ˆ€4YgΥL?ύτζψγwMΏι¦›ΜΒ /l¦™f³Ωf›™?ώψ£h—žzκ)CIεΧ_5[l±…™a†ΜsΜaξ½χή’§’YguL9“Οƒ6xDΜ4ΣLfΐ€¦Π*ΣxL,Ήδ’¦G¦oίΎζ‘G)Ϊπύχί7Ϋl³™e–YΜK,a.Έΰ¨Γ7 gžyζπ³Κ*«tΘSλ„j™s θΓd“Mf† Rλζͺό"€d*G‡D )D•*š:θ sϊι§'­JωD@D@D@D@D εάwί}fΠ Aζ“O>1ϋ쳏SθΌχή{ζχί7‡~xAΧ\sYύυΛ²2:δCΜL`>ϊθ#sΖg8…Σ·ί~[ΗW_}e6Ψ`sύχǏKόίώgN>ωdσΰƒš>ψΐ|ώωη怓NκΕΙFmdXlhδΘ‘ζœsΞqŠ#ΪUJ=τP3Ϋl³Ή>\{ν΅ζΘ#4Γ† ‹=ν±Η3—\r‰y葇άη’‹.ŠΝWΛΔj™ί–k±ζškΦ²™*;)™@RHBEΣγ?nZh‘’Ω'Ÿ|r—‡ό§œrJΙόΚ " " " " "` VLύϋχ7Λ.»¬™nΊιΜ1Ηc†‹ζ†npΚ›3Ο<3φx‘Δ+ΈΒM 3fGΤ―_?Wo\ώΥV[ΝΜ=χάf½υΦ‹;›Φ΅kWsυΥW‡–7(‘†ή!/Κ$Qa‘ƒ%ΣK/]²κ―Ώώr–OxKL8α„fΉζ2+­΄’Ήε–[:ԁς +*ϊ1όσ»ΟœsΞΩ!_­jΝΌΦνWων€djg‘-¨šsoΌρ†{(1Γ'}ϊτ1ϋοΏΏσ)'ΏDD@D@D@D@D #F8₯‰Οέ«W/σΓ?˜ί~ϋΝ'…ί«Ίͺyλ­·œ»X˜Xbγ›oΎ1ύ·™}φΩÜΤA½q‚Πi§ζ,ŸβŽΗ₯ακ‡’ ΟΩgŸmP†νΆΫn²β8ΡD™.]Ί„Ηp ŒSH…μŠ₯Λ.»Μ)šHΗ «©ϋΏ‹fsΫ―½φš™rΚ)΅J;”M~ψa‡|΅L¨σZΆ_e琒)—‡φD X5ρΐΈώϊλ]yάθ1‰%πήK/½d8.(Š™ξέ»‡'M1ΕnESΎL5ΥT7ͺr„ςQΊD;ΤW>ε’ͺTž}φYƒ Š$‚Vη q˜Pϊ0Ak ΦOœσύχίηg-ΈRjΓ 74(܈•/?ό³³Δ:βˆ#\Ω°]wέuΝΨ±cσ³ΦlΏžΜkΦ (ο/.PΞ+―Ό’«ŠXQ΄Ρ «½=χάs~ΧjoZΜÊ΄Ρ©Ίujνͺ<³Πh#|œš€4–ώ䁁ί4˝zYl±Εά&.s«―ΎΊOΦ·ˆ€ˆ€ˆ€ˆ€ˆ@XmΏύφsΑΌ§žzjΣ»wosώωη»p[y啝ežyζ)Zͺ_αλ |!–*Š¬—X Žqύ¬³Ξκ²±ϋœ½šίGΑB[>ώψ㜠β_sΝ5ΓU©Qδΰ.vΫm·9·Ώό>Tοˆc=ΦtλΦΝπΎα­Œ υ·:βΐςρl¨—ΰβμGϋ@œ&VΌ&9 .‚³‚R¬ίk―½\œ§Γ;Œ]sΞ9ηΈsn½υV·OΌ©γŽ;.TDj/™Σ`ξ*՝N ‹ Dtz+Τ€L υέwί]Π­k‘Eq7[­ŽΏ\\pΩ}χέ́θfTςs13A|¦Γ?<φE@D@D@D@D αΤkbz¦™frΦKK,±DΘ…–9Ρ ΰαΑΈŸ]tΡEζ„N(˜ϋΫoΏuρŒ’Aΐ fŽ9ΐδσΰΑƒΝΔOsΤΈΰΪχJ£ΨLγόρGC σ¨$ιC4±m,ͺ~ϊ駁ΜKυ‘X™ωΗ’΄·ZζX‘a‰Eμ*I琻\ηpΟT­<P aJY,nΠλ―Ώξς,΄ΠBΞΒ)SθδΖΰ‡‚ιΠCU0Ρ<\ζό»“/”ͺ¨ŠΞΈa‘xαΫ»dω΄Eϋ,mj1)uΜσΟ?ΆΥ›+U0Qqˆ°μ)&XυTͺ`ΒΪ‰w¬B &κΕ2)‰‚‰Όω &’τ|I‹­ό•ς’τ!IΩ>O’φVΚΫ¬΄°“t.Y2u.N―Κ₯J䨣ŽͺΙ€’Άtζ9ΜNμ»οΎ†e?1-$»μ²ΛBsΡBω”." " " " Y#ΐΔ4ο |›žkΣcξΆ&vχ=jΈ1?Ω2β^“ζ»ξ]Ώώϊ«Yzι₯έ*km•θh'€’ΙΏΧ?«oίΎν΅UWR2ΥwΆ*«FΑδ{‚n+Ηk:σΜ3Νΐ=DYbν΅Χ6_~ω₯a‰ˆ€ˆ€ˆ€ˆ€4π½…μVΑδK…oNέΏ{،ϊbX’qr‘b”."Πx€djΌk–Z‹+5ύΜo@«†υb%‹ƒ>Ψα4hP>–ϋ,ƒŠr‰ΈW(›$" " " " "u‘‚ λ₯ΩΛΓ~r³jjυ‰ι¬_c΅O$ ˜Lil ²9δTZ‹’ ³ΥV““N:Ι)˜XΪ4‰‚ >3Μ0ƒ[‘Bq™ZνΧ’ώŠ€ˆ€ˆ€ˆ@γp.H•(˜θ2J)kυδݘ—‚Z."”€”LII5Q>ό¨ύς‘Υv +&π9ίμj kσ?ώxΓ20$Πw9’ΰίεΠR^Ξ$€Sχ™{—oΑm΄u―γ]²$" ΝO@J¦ζΏΖzΨJ ‘―2‡#–K§Ÿ~Ί9θ ƒΚ. %S±όΚ.P'ˆ€ˆ€ˆ€ˆ€ˆ@ 0ξΕiΤ΄ύ«+ψM‹ξΧ4ΌΊtΆˆ@Ζ HΙ”ρ T‹ζύρΗ©›–ϋ]κ K±@”K‹/ΎxS.΅ΉϊꫧώKœrΚ)S/³3 άk―½ΜΰΑƒΝ•W^ivΨa‡T›BποoΎωΖ|ςΙ'©–«Βͺ'ΐ**ώο_ίmχAqύΪΥίeT‚ˆ€4ξ“MhΜ¨kπˆ{Σ+K%‰€d–@·ΜΆ¬Ζ CΉ΄ώϋ›Χ^{­`MX›πΑω¨£ŽjJ…SΑΞ—y`…V(σŒμfί}χέΝ\`†jΆήzλΤκέωmΝ>ϋμ©—―+'ΐί:Ώε~ύϊU^ˆΞhJά;4Σσ)/”:%"ΖC―x«•Z‰FχΠYͺ(Θ*–T2a­Pn ;ςτΣOΞΊ%«3i» qΔn΅΄€ηΛwΰ6Ν·ΈK.ΉΔ\wέuf‹-Ά(Φ튏uοήέΜ;οΌNΉρΖW\ŽN¬ Tά#$" "ΰ  \*wάΰΟΥ·ˆ€4*ήά½oZλ2Χ}ξκΊρΙ=ξ|Κ”ˆ€47ρš»{{W‰‚Ι—rΞ9η˜%–XΒο6τχqΗη\Σ聱›Ap‹CΑtγ7ΦLΑδ9)ψ·'‘o,@!δ”BŸ€ΰζf]επ ‘ˆ€4?–S2U;ωςΛ/›Aƒ5Ε/γΌσΞ«ΊΝς°0`€ΉόςΛΝ­·ήj6έtΣͺΉ”*@ΑΏKqΞ&πψγΫΈLΦ’iœ%REν±η’¬’₯xEτt’4–R2ν΄ΣN©\ γ?ήΕeH₯°N,„›}5J’f‰S΅Ν6ۘ«ΊΚάqΗfΓ 7¬Λ!ψχ?ό`>ψ Ε`ŠuiΉ*h%NΡDΠξJMœ#+¦VϊΉ¨―"`ZFΙ„ζœ₯θΣ’C9$­’:΅ΈA`vΩe—ΔνXd‘E ›f˜ ξ΅Χ^kξΎϋn³ήzλ%fPmΖhποjΛω" " " " "P+αΔt9Š&kύΤύλ5aΟὁ2$" ­A e”LόρGͺWτΕ_L΅ΌΞ.μΒ /tVM}ϋφ-Ψ#X/±"_3<(p‹»ε–[Μ½χήkΦ^{ν‚ύŁΙ&›ΜτξέΫEω*SD@D@D@D@"ΐδ²SυόݘΗwk³jΒJ W:/lσyν,χY€Χ4n2»ή|υ-"Pš@·Yš#Η}χέ—zGšm)cQλ€e—]Φ|ωε—.VQ³=6Ϊh#§\βwΡΏΤI Tπο$””GD@D@D@D xΰΓϋΒ“O>Ω>λ¦\δVZi₯Nν–Lπ}χέw;΅ͺ\D@D@D@D@Κ!0fΜ³ζškšαΓ‡›½φΪΛ|ϋν·M71]εh'ΠJ&L6k!ύϊυ«E±™)s–YfqmωόσΟ3Σ¦J2vμXχ |ξΉηœS‚Wz5užˆ€ˆ€ˆ€ˆ@g=z΄Wοsγκ7ή،5ΚΌχή{Υ$Υ+"!-‘dΚο†j –LΘgŸ}ΦPνΞoμŸώι„―ΌςŠ{.³Μ2ωY:e’‰&2¬Τ'—ΉNΑ―JE@D@D@D@Κ$πΫoΏ™΅ΦZΛŠ„Π,€‰Σ2!*»49–Q2-Ύψβ©_ΚhμΤ Ο@=zτ0SL1…idK¦ίέ)˜ήzλ-χ Δopγj~ΕΔιΒ /¬‰ΣBΰ”.-F e”Lhά't»ΌΥwk¦F΅dϊε—_άƒ_qb0υιΣ§z )— ΰί)Uq" " " " ©`1 LΌ`Α„5~T˜8•u~”ˆΆE u ΄Œ’‰ΈLX΅€%GuTZEeΊβ25’%ΑΚyŽ1Β=QζdQx σ»ΔJ"" " " " Y#@PoΖΥ_ύ΅W/Έΰ‚š(λόH” -K ₯”Li)†Zh!Σμrώ/’-™Ύϋξ;χ όςΛ/έƒλ•Uαά₯KΝόdυ©]" " " "ΠΒΎϊκ+7fχώϋο7σΟ?, Ζ΄Δk6lXμq%Š€΄–Q2qIQ œΏαjδΠC­ζτ†:·Ρ,™˜aa¦ε‡~p ¦X ΣΌ»uλζ‚%²:‡DD@D@D@D@²Bΰ‹/Ύpγκ?ώψÍ«ηž{ξ‚MΣΔiA4: -G ₯”L\έ^xΑΔ™x&½ςΔuΪb‹-Μζ›on~ψဧ5lΎF²d9r€{2‹B ¦y睷!ΈσP–{C\ͺčd_bœwήy‰Ο‰fόϋοΏMϞ=ΓΟE]~γ7́hζ›o>³J+™Αƒ‡Η’nTΫ>_O5mΉώϊλΝrΛ-gΈΗ¬½φΪζŠ+πΕVτ]M[nΏύvΣΏ3ϋμ³›-·άάpΓ ΅!ξ€Χ_έΜ1Η†ώ&‘—_~ΩlΈα††ό\sΝεΆ_{ν΅$§f"€§žzͺa‘ξmLμͺώK/™uΦYΗύξωm`}όΧ_…ύΪk―½ΒΏ‰UW]5LΧ†ˆ€ˆ@ϊ9ΑΔνΨ±c‚iΞ9η,Z §Š5Z‘Š@Λh9%W–μα‡^ΦEfΕ„Ηά™^vΩe.Nƒ\ΡηŸΎωσΟ?Λ*―Q2σψοΏf>ψ7AyςrN0B^ΞEτ@n”+•¬Ό£„FιQι}!σ駟:εΔώϋο­'mΩe—5O=υ”Ωi§Μl³Νfφέw_³Ο>ϋ$kœΝ•Fϋ¨¬šΆ ΤA™σΟ?˜=φΨΓ}οΈγŽ¦R+ΡjΪrΚ)§8EΞ―ΏώκΪςύχί»‰%Υ Χ«­Ά2Ÿ|ς‰{v”*ο±ΗsΚ™gžyΖ-Ν‚l³hΑOϊΘ=‡ΊvνκΖΥL|%Mœ&‘€<"ΠμΛLˊUK,±D`/sΞgΌρΖΛΩ·³©±Œμ‹^`Γ.οd“MΨ½ΰΝ7ߌΝΫ¨‰φ!γϊg_t2ΫΪhέβ{)°/›™mg‘†½ςΚ+Ž±u™+”Eιu"ΐ½ Πί{’&<χάs΅0 lœ-wMO?ύτ$§uΘc•ξ|ϋBsΜ*ΆkΨφ0}Π AΎχί?L+΄‘Vϋ(ΏΆXE„cd]—sši­w; ؘ9ιIv*m‹₯ &œpΒΐZ·cƌ «:묳ž<ςH˜VΙΖή{οL2Ι$ξZZeKΙ"zχξXkΩΐŒζ΅ΑV]­2:LΛκΖ‰'žθ~‹ο½χ^ΨΔG}Τυ’K. Σβ6vήygΗΚ*ωΒΓηœsŽ;χž{ξ ΣΨθΧ―Ÿ»ηη$Φx‡ρχΎ%" "ΠΜOXkΪΐ*Œ«Π/««Φς:°aIΚ:G™E@š@KZ2yέ!³¦/Ύψ’±—ΥY)Ω—Kgžo_ΪάΎLΊcΔrŠΜωqρ 3πwί}·!Θ4³­·έv[ά) —†%‚₯Pεƒ>pLφEΡΝ΄ψφf±­…ΪΔ¬3σr™+D¨1Ÿ}φYge„₯P1Χ(«T4·άr‹±Š„²:6|ψpσΠC™mΆΩ&'ά»ξκ‚ΗcaYL’Ά―XώX5mωζ›oœuΚqΗη‹sίή$ν·ίΞI/΅SM[`ΒυΪzλ­ fώ^°²" 5.|>ψ s—<ωδ“}±EΏqσΕς wμΙ'Ÿ<Μ;ν΄Σ:‹ V Δͺ4Λ2dΘc@fžyζ ›‰K'ΛVΙ¦εoπ ži¦™ ¬¦žzκπ0Ώ —C‰ˆ€ˆ@ν Όϋξ»n\έ½{w7žqΖΛͺ”1-Ο2έ·ΛΒ¦Μ"ΠtΪGΥMΧ΅ς:„Β‰O%2ύτΣ›Γ;Μ}nΎωfsωε—›6ΪΘ™ϋο°Γ†/ (Όx1ψόσΟ3Χ|;[ξβwτθΡΓάuΧ]f†fΘ\“6ˆ‡²‚'₯•Ν|˜”[Λ%³Λ.»–ϊ-$ΔiΊςΚ+ JkΉR([‡t”S‹Dε?ω™y晍yŒ&wΨNΪΎ'Ζ$TΣΪ{ζ™gζ”ŠςδŽ;ξ0ΔΌ[|ρΕsŽ•Ϊ©¦-4EςούΣM7‘ΎμRmΘ?ΞΒΫoΏ½‹G΄όςΛηŽέ·Φ°₯yΎX +KΊ΅Κ?œ™}άΧX%hΎX‹asηwζ'‡ϋ(τŽ<ςΘpίoψΙWΪ`\‚«3Š%&Ξ£J€5G'N‰M)hMΩ±6θυΨd“MάςžhπΧXc Γl=/,(š¬«JCφŠ—›¬Y2aνΐ,χ4ΣLγfZYΑ²,™ςΟ#l4ŽχΫo?cέ£Β΄Έ7ή؜tIεt9βWόζσeͺ©¦r•ωιΡύ€ν‹žSh»ΪΆψrQς`‰E¬e `?V‰εH5m! 7B½Qa bεš+βe1±o­UIYΧ\sSZn»νΆ•œ^·sJ]‡ŸώΉd\&ίX,ƒy9aςζμ³Οv‚ώ˜ΎE@D@'ΐ{ ž¬*MlΣJLΎUŠ5κIθ[Z—€”L5Ίφ ·ρ$Μwί}ηΎ±RYf™e ³ΪX14’πΐΙ’%•Q0aaΕƒ0ξ₯»‘ψV@ΞΊ;L£qΝb{ΔrΘ!e[7βN…`Ή—/˜΅ϋ—όόc΅ΨO«-yf`K°m„ΩΈN•#Υ΄W.ΈΪΨAZκΗsŒΫ.¬:X²r_B94ΑΔeIœF›vΫm7§„4(zβΚͺΜXκ:P<ΟΓR‚2 WCάΠ±pΒ:Š₯³%" " ΅!ΐ$'γjVγωŘ’‘u~5τt4)™j|±j`Ωe– ΎώϋKnXή°ΒέΗ\γT_|–,™PΦρ d©q¬β^Έ«οqύKΰΜJ[²fͺ?ϋF©k%$ξ…ϋχί―Z‘Q‡΄ΪΒ X0‘H χΖ /Ό°œ¦˜jΪΒΉ—^z©³Xb4VλΓΊ ¦υΦ[ΟL1Εe΅…ΥxXιοψγ76˜xYηζgΎα†ά¬² ΎκžΔmΛ²”Ί΄=‰mΚ)§tΓXΉ‘°cυ?~τ °β6γκωηŸί)˜pέVΌu~Ή“FΥΦ«σE@²C@J¦:^ –bΎι¦› /"ΈΟ1{Ξ¬Α¦›njxΰ:Ά€Όͺ²bΙτςΛ/»!Ad™i)χ°Ό^Χ77–oO<±”LυΕήP΅#ωρΗ;΄›΄jg;Z$!νΆ XΐzˆΏb3•#ΥΆeύυΧ7XGΪ•ΡŒ]αΞWϋ Lˍ₯gW4vυ™§A±yΛ Έ7ςα‡ζt’ϋA—ωΫ¨—TΣ¬:QΆδχ%ΑFγ‚_λW5m‘\άΌP€γŽvΦYg™ 6ΨΐΉμ½ω曆€ΥεƒjV†Γ%+>^iφΜ3ΟΈύRχQ”mΔφ"€6“ε(fΚikΪyΉ~Μ€η_Wκ!mΉζrJΔΈzq—€ί(“ςeΎωζsΟV%”ˆ€ˆ€€C€ ΖΥ,&ΒΈ:‰₯iš5qš””ς‰@󐒩“―ν€ΜSO=eXJ%Γή{ονf­ωfv= Β Yq™x9γAΨ§Oχ ,70p&iƒΛ”$―ς΄ξΈN]ύυ9gΥ.ba…S/©¦-Ώός‹sΒr(*O>ω€s.wuΉjΪ‚ςά».GΫrέuΧ9ΕFΉΑΆqα%6Vτσψ㏻’YQt\ς Ι\`Ž>ϊh§dΊκͺ«Rτͺ3ΝτΝ7ίά­τ(Α퍗™bΏO”S'Ÿ|²{ώα6μ –e,žΑd‡DD@D zO<ρ„W'–ηV-V.՘Άϊλ€D ‘ HΙ”‘«‡[ξsΌ„ 4ΘΝ`cΩ„‹ƒμΞ?ΈοŒζxρDΑDΠt„ΈP4«π²LΜ©Ώώ»Y»¨~Y(ΈΦqΕω€0=φ˜Α-λύχί7Έ`νΉηžf³Ν6sχ ώ±Ηλκ(Χ*ˆσ‰GϋŠύ½WΣ–e—]Φ¬°Β .06νΔb χI'ΤYyϊ~ΤΊ-ύ&°φE]δξΏ0EΡ³Η{8Λ&ξ;^’΄ΕηMςΝύ Ξ^iΘ½ŸΥΤ`€"ϊς?ήeŒλΞΉ·ί~{Ιͺx†πΙ—BιΔΙΚϟ΄Ύ⃱6χ2\1Ψfυΐ#<2lB~ί'šh"sΰΊΏ‰ώχΏ.N1©h Š&,Μ$" " Υxδ‘GάΈš8„wέuWυ(gΟ‰ˆ@khή7φ½žΔρ±<Ψ£x"f.vΔqβSξηΥ’ MΌψ{ι¬ΆŽΈσy™^guLώύC—“Έ|Ν’ΖΑ‡}©₯–j–n©y>ύτS7πϊσΟ?σŽ”ήέi§\PjUΌxγ’ΔίΘgœ‘s²―#jQ’“‘Θ+Ό10,ΥΎJΫ΅kW§8'@φQGε>4 ¦[nΉΕΉUωζΥΊ-ΤsΠAΉξvάqGW-χZw0ŽJΆDΟ)ΆMl&8£\Bp5j”ΫΖͺ'N>ψ`§€Β½Œs“¬Φ†/NŠ₯ηΟj'­xyτc»νΆsJ0‚•―ΌςΚ.˜{ΤΕ9Ώο΄ήό6τ}υΥW»&γ>‰ )‰ˆ€ˆ@uXΉΕ?ŸZO`3¦e•ν1cƘ¬/\QU-"KΐFώ—dœ€}μ Y`c}°Άw`πu±«k«νΛCpΔGΤ­Nϋ μl~`],κVg*βΫΈ\YhJKΆΏ/«ψθτΎ[ϋ[·K±m±ƒΆΰ½χή ¬Υ[μρjmόœΐΊ΅%*¦šΆPΗλ―ΏX%FΑΊκΥσ'>|xΑvp œΆ-(…ƒvΒ!°A)””¬ˆrλ³+§v•Ύd…GrYk­ΰν·ίμ€F$5w³_Ώ~Α ,›Xγ=λφθώ&ω–ˆ€ˆ@£ΰya'kω\—¦6Μέ3ŸώωΊΤ§JD@²E@ξr±ͺ·l%²6« 1k}ήyη™·ήzΛΰG+eλ!ΈΜΥΛ’‰™p\δXBόΦ[o­Gχ2S‡Μ‹3s)2έάFη™gžšΜήx㍆Oΰ@IDAT`Ϊ°N"Υ΄…:ŠΫZœΤ³-Δύ!8u!)·-…ΚI#έή ±κκeρXI}³Ο>{E+|–₯΄},ΐ4x© hewί}·Yk­΅œU(Θυ;ΰΖZa΄U‡d€”LΩ»&[DœŽέwίέΉS=τΠCnŽϋ/G¬ŒTIό•‚•ε`ΐ_ΐίχάsS0α"ΘK]«‰%ΆΪ/ή_άΗp½ι¦›ŠgLρ(λ H³ jKόUΐ}™{e=β3€œZοϊ 5ŸUYω{ΘΚ’…Ϊ©tΘ bχαNJ₯Χ^{m]›₯1m]q«2Θ)™2u9’7†6³#FŒ0»ξΊ«‹aΑΚSm΄‘[-yIΙrΦΓ’‰U²ˆ/³υΦ[VwjEΑ’‰X)•ΔiE^ΝΪgbΣˆ!qn YϊΤ’,B€R! ’ΆΔ_¬„zυκ°©υ―Pfši&χχΐ3bΛ-·,”Mι" " –U¬μΉύφΫ‡±ξκ FΦωυ€­ΊD [Ίe«9jMΉX›šψ :Τ Η$;¬œX©'©ΫK±ΊkmΙ„[άΖoμΖ^yε•ΕšΤΗx #˜/·άrMέWu0\ΠllΒtDZ+ς‘ˆ€ˆ€'€7‹&°¨Ε%—\Ru@§E^υŠ@'°/3pμŒ;ӁU}μ,tAšΦ-Ξ•}ΠAΜ£mμ8°+α¨3~Ε~Γ…šc­•‚ρΖ/X~ωεs²άyηξ7o_ŒsKνXCμyΦz#°νΐ.1hΠ ΐΊΆVA¦εo€έ>_ώsΟ=Xeš«vΦ΅Φ*ϋۚψ“L2I`-~Κ>—ͺm‹Urv0μΪ@_μ JEνθέ»w0逓Φέ5<ίΎ46&^`cS„i₯6lL°-VQS*{&ŽŸxβ‰ξ·`γ†ν±“ξ·l-eΓ΄Έ ŽΓ݊ώρΗKXD₯_Ώ~}i&Υ|ΫZωΊφρ-ι|ό}Ϊ%ΡΓ†ΨUj«ά ¬R6°€UwάqGxάopo΅ ‹ΐ*+kρΨx•U6όύ—Ξ=τΠCέ½ή*1 fη^l-7άgΨ°a]άί’ΘK/½X‹sχ[+u/΄Jέ`ι₯—ΊvνXε|Ιβ+ιϋ=χάX₯n°α†–,Ώ¬‚+°aάύτΆΫn &Ÿ|ς€ηYΎΖ1ώ>νApψα‡m΄Q˜φ[«ίpΏY6θχMϊ›e±§U²ά7΅MD ž€,™μέYbœ‰-3ƒΥ Mq2dΘΓlΨa‡fN9ε”Έ,J‹P ΔŒΨ$†Φώσηm.Vέ»wcšqμ•W^1·άrKΩ1·˜•}葇\,³‰&š(¬‹Dβ§α†ZHΚi_‘2ςӟ}φYgU…υP΅ξ\ΈDαφwI'9Λ―όΊJν§Ρ«\4#GŽt–2₯κ+t‹#ϋ"ηfήν O˜ ΛD¬z°Νwε3E6|πAηzˆΕD# χy«2¬(δλX¬ pΗ.&Μΐσχ‚Υ‚—=z«(p•>Mί"GΰςΛ/w1{Z7c\•Ή/:΄CV,Άήzkχ›βΎIΌJά©Έ‡”¬FωΆ/τE³β&Kh>Δ°Δ*’{[)αώΓί€ˆsΦΜ₯ςόρf‰%–p–L₯ςrΌάΎcQ…ε A€;KdΝ½™ϋιlΰξ/V‘Ψ‘9ά_¬RΙYCvλΦ͍5i{3 ήX1cΑΗo!Λ˜Vq™²|…Τ6HŸ€”Lι3mΈqαΐE!-Ιw›;ϋμ³έ ΕZ\˜N8!­jšΊΜ‹Q*Ψ™ό¦ξg³t7ŸΟ?ά 0 §KvΫ#ˆ/˜ΆγzπΩgŸω€Dί(§Ύ}ϋζδGΉ…;^…€œφ*#?έΞ»ExιΚoS~ήbϋΔλΩ~ϋ흋Ι{μQ,kΑcΥΆ%q„PΤM3Ν4λ)u— ”υω ^2yΩ$V—΅x+ZΜ?όΰxΰRl-γŠζΝAάΧpδ₯7_H+φϋόωηŸέύ&„—akAap‘Δ՝DŠΰ·Bό3/Έ›‘–/(~’.Eόνγ’…{s)aUή»οΎΫΉc•ΚΛq―€AQΒκΎ₯dꩧvΏ}\EK ρ0Y58‰ςΚ—Unί mpΔGά²;C¬u’Α–‘½Ί¬’L>”.άoP0%Ή¦ΎάFϋζ7L=φΨΔ1T;³Œi_}υΥΞl‚κ¨3β£έ:7FΥuf‰J½ψ”Σ²¨5Κ+VΊ@ρΔΓP’Œ€|Ψ“qΚz.~Μ–³’’—7ήΨ½L?ύτ>)Ρ7/,Hœdͺ©¦2_ύu’r’™βΪ=^l›ΐΞόm[·bΩJc6₯χ”E•H5maGVΩ$h*Ιk!Χ\s{သRB;°|β…Ά‘€ΤοE’uσŒνΦ XBρ"ΉΞ:λΈ—F,RxΡ”ˆ@1ό†°υ2ΕSΆΕδγ?vqΟxaOr?Ž»χ+Ÿ˜|,€²ψβ‹ΛΓBEu)Αb’UΔP^E­ZK=^ͺο,6b,ϊ슞_m)ρ¦x†z)t]Qrc Λ€ Φς~ψ‘±ώ΄¦ϊΖΒk7&m™ΌmaLΛDΓO?ύΤΝUE@R %S ½nϊI\8Κι'ΦQX°/K(²$Ι 0sHPR™'g–΅œ ―ΎϊjÊXΌ8{!x1:Q$”#>ψ3.DωΒΛ•ΙΟ?VhΏPϋ ε―E:χ–]& «·b©E=ΕΚάsΟ=]aOͺXΆŠ±°AYd—zιΑεηΎϋξ3(₯&˜`‚ŠλμŒKύ>iS‘ψ^Ι„/Ά'&XΊ£bxΩ-ΗZ£3ϊ:;ŸχE€5l,Ί`ΊΜ2ΛΈ•:+΅ ,T6ι6B…»·Υ’l¬Q¬kYξ£TIϊ_ͺο”… χFΚGƒ2Ύ˜5b1•ΛΏ¦”SθΊΪ8xŽΙΑμV<Ζ‚·:άΉ›IxnσAΡDψ‰FMœ6Κ•R;E =έ+J%5*JV“+ΥWK¬²ΔΛ/Τ’ς πP–yqωά:ϋ 'Ό0Θeu°3Ο<3•&a­„°:[Ύΰr–T)Q«φε·)Ι>«±’–iZS&©›<7ήx£±¦Bƒ—”΄ε†np.”Δhaε<^ Φ\(Yˆ­Ακj&₯~Ÿτ§ΠoΤ[*‡Lή:n©₯–r«s‘δ₯Χηk46joν Μ2Λ,ξ^ΒoαΎ3β¦mΑ“O>ιβχœsΞ9fΛ-·ŒJmΫπ6ί½«'΅BΗτΕ_,‘μb.Ε Ο&φ°”-&IϊŽΛ/ΟX"(―xξμ²Λ.† Δz mά°υ C++Nζ Θδυ2έtΣ¬'a“ΔUџ—εo¬}™°εyΙn#‰Ÿ8eL‹ ΄DD ω tͺ%+>ΈRρa[R_΅`ΞΛ" &;+―ΌrΕKΧ—DφjSπομ]“R-"φ‹Q01θ'πl₯.`ωuα€ΔΕι"-κ*’ί―eϋ|ε|»Θε\?Κ9/ΌΌ`aD(ΓωsA±QMΰm~ΌΐΪ՟ q_JY_‹,|[p@μͺJ.-ξΪ» ψ―Το“&βϊ'ΔCάλLμσ,!EbžID ϊΔ»ΓZΛ8,$‰}‡ΫΛ8„Έa,o‚ χXbζραވ Έ!OΉςόσΟ;‹ήoΌα\εςΛ”ν•C>o’οhμΚxζέwί ?άϋνJ¦‘‚©PJυέχa5ΦΛ¦£ΨργΕb}`ƒρ /ďB1ζ……((ΧKώΎOǐΏ»j₯s™₯ξ₯΄!“]ΕΟmγŠΜ}’ϋVdX@.²Θ"†8WΝ (ΩQ0α^ήh &Ο_ΑΏ= }‹@k¨»’‰‡fΎΜHΈβŠξƒω'q|Ψ'J'Icΰϊρ2qΧ]wΉŒΜ^Ϋ₯…έŒ>/^v‰ΩΔfݍI VcΙΔleΉnPιΤRΚ%@Ό^^n½υVsξΉη:% i‰Η•!*Μ0γΞ@μbRλφ«;ξνα3o—―ViΌ ± /CΈ©ω/2+bιT‰π<#^ρLpύ*€\‰–‚‰˜DΔqρmAω…π‚LŠ³*ΌθO&χI{I#(3 Ε8α·ΝߊяζAΑΔu*7N΄ m7?ΐΪ«Ab ±ͺ!ξ–Θ#<*œΈ7σ7R w,!ˆ6Β$³ε J― ζ\”(½{χξP eGWP쐑@B΄²„Ι…ϊPͺοω} ΜΫ(Φώ†™\τΚNέa‡\r_ ξΡU†Ω.€Π'χ†žΦ*‰Kβb΅„°νWU#Ψ7nd(–pΏζή}σΝ7ϋ*ϊ%ή(FYα°QEΦωzεΤn¨€ΥψΧM¬p`›™ψc—Ό―[ΫZΉ’rIΌV‘䐾ω曁KΨYϊΐϊϋ2^ά5°ƒ†ΐ><;8 μj[­|:τέΞό9V6FK‡cJ¨ ~ίvΆ°’Β7έtΣΐZ-VΙTΡωΡ“ώόσOwνω»ρb•"}rG>oλξεςZw¬hr‡ν4Ϋ—_ψ'Ÿ|βΪ`­ςάν΅Χά9ΦτΏ`žJT–h=VξΪeWwŒ&'ή>όσέωVΙ”ψœB_ύuWΦΕ_\(K¦νjTU”Φ}3l—΅,¬ER`c₯„iqλ»n`•u1 ۀ߁U^}ϊτ ΣΨ°1Ξ‚X '­Φ;<ΣΈ?ψg[­λSωΕ πχi•–2YΧ€ϋg₯rΥUW=τP₯§—<ϋΈU>—ΜWM†Fλƒu x?(&Φ’+€])±J€ΌωbŽ΅ ΝOΞό>χMξ;ΦJ/σm-Υ@kmηϊb'NKeΥq& P7K&;Ht&―φf™Xμ`έΝn2‹!©ZΖ lfσ™q".«½{k\IX5—Vbv |L‘q!ž ζΠ­*3Ν4“!Ξ„‚g`_J\QVΕ"ζ3Ρn^°dF/ΞβΓη‰ϋΖ3y\p­"+ξgέl³Νrg»:ˆ­”Σ>ΪΝΉVι׌²ΣςΫβ ΐE‰›νχyκΥ__±ο$mΑκ`¬Δw"Hτ7ΰ·} Z\^ΰ|ύυΧ«6φםs½υEl¦q‰T ͺ^(}ΥUWν?i}ΈˆΜ*Œά} ·Ά±BΐέΓK\ίY%‰ηΟ¬6ψ}°„:–LΡK}ϊ¬'§Η uΥΔΰα7]¨Α—›Φ7–’XφΤR­ΈΐαΊ\Lˆ±”Δ2«Gςz±οiξ7‚Ux£ Ο{¬Ό.ΌπBc'έ­ωΪΛ3 Q¬Ρh” MI [=zΕK΅h©Έ*άθxPHjCwL›qIKVXa…‚EaΚΜΗ›²“‘—3;kο>Δ2ΐμά›Sめ 4Ÿ…^8άf0Ρμ’ΈLq…½’Ερ#ςwQΏΚ/D(νl{~Ά’ϋ,oOVUΔfΰ…Š—r”ΈQρuψ αε΄ΟZUάΎhόv~[|ΊW2Y‹ŸΤα»^mιPqLB’Άά›ψ.H!χV?β%˜ψJό*q‡% 0ηZ­-ΪόBΟήbιωAΨ“Φ7οΌσΊηΫm·S‚Χ^’’±–βϊŽ‹A‰ φEik­2Β,Ρ~i»΅ k….?LV₯%ΈΉΦRQki΄>ΰΊXKα7Β8·oίΎ΅¬&Υ²Ή†„— ΆX­•’©6ΌHa~β%SάΔG‘SuHD  tΑ«–νζe¨ΪYHΌ @}¬ŒZΆ·ΛNγEΉ‘`².Ρ€ŠΆyAφŠ'Š%3ZQ…“ίζ₯»™?|Ρ™Ϊfκ_ΦϊΒοŠΰšόMt¦`νBΠS,–ςH΄‹|,QΦζ–M£έΔ5a HŒ Ξ΅%ώ `εƒ’žΰΕυrλC)GΌ›JξΙΌ’„υ±WςϋΗ3ΕΛ«ΧK°¨fΒ‹gυKD@D φή{o3dΘsΕW8₯{-κθ¬2Χ_}· ή ζ&PsSjLΰΗΒ†ε`δip—ώ’jΚ¦΅Τz &zΙΜ7+3ρ‰ 3π^ιΔ7«ͺ`݁\Φ+œΌε“˜-£QΆ1/ΖΝ„ΐΞ~υ¦Fi»ΪY;XρΝ3Ο<5©€€©όΝdAΑ€ΆΔ_b”+g)λzH%υ±j_₯uw©΄ '" F‹g\@m,Sj’ΡΪ_ͺ½Œi/»μ²RΩt\D  ŒWΛ>€m Π¨ΛvΦ’qZeΗYKTRvZ ¦bu³Š«  #> Δ·afdσΝ77X°|νΪk―νb‘œaΆŸ•GxiΕ₯©QD>μr₯jΣNV8cιθ›nΊ©6Δ”J°θ*I1Yκ–€ΆΔ£&ζΣ=χάγ,…βs€›Zοϊ ΅ž•hω{ΐ’U"" ΝD`—]vq ¦k―½Ά)L\+Ζ΄XζυΥWΝtιԁ5΅dΒΒ$MQ°Έ4iv, 7‘j,Ο8Ώ³,ΝζœsNΓ'ηΙ0ΖxςO>N VήΙΗ{βΫΦL'¦L?ύτ‹β―{GψΛΞ ρGκ%ω–ƒυͺ7΅%ŽŠqχ„ψ#΅I­Ζ*)ΝΣƒΈO|X$B"" Ν@wd,|˜΅+Α6C—bϋ8e"X""ΠΌj“)ΙJε’U<„r‰•——DόΑίzλ­²NΜB›$ fε#οjΗlΈίΆΛήbε»Ϊ‘xšrΚ)“]³<oΌ±‹CuΧ]wΥ¬άF +1™t=D@²E@1™²u=ΤhΫoΏ½ΉςΚ+ –Λm΄Q³t«`?ˆ%9ΐ.π]q΄`fhX5³db@&i<X"σWΗ$VM,«]ž=λ=Ζνc™e–qŸh[£ 'OwάqGNœ§¨΅ΫυŒΔΜΟΉηžmΆE@D@D@D@˜‘†jnΏύvCPμV­šά WY}cj¦dͺ\VQByΥY.Y΅θSVΛDΙδ9ί{ο½.ώΗ{ο½gP0λ‚ϋΗ}ž¬φ#i»°`βΓΫ^ˆσU>]pΑζ‹/Ύp‡Q2E­žP<± V-%Ώ?ϋ쳂«-Υ’^•)" " " ">-·ά\ύυ.b+ΉŽ1¦m€Ιιτ―ΌJΦ ΠPJ&\š$υ#ΰH|5ΚόρΗNΡAœ #Ž8’~ 通zυκeψDΝ—Ώώϊλœ8OΜ>tI…SL1Elœ'άπͺοΓN\¦BKzWSΎΞ¨Ν6ΫΜ-Vsί}χ™5ΦX£>•f€Ζ΄#GŽ4ŸώΉ[œ'#ΝR3D@R&PέΫo‘ΖxE‘,ͺUΉ5¦ENΒzηK/5t9ε”SΜ Aƒœio‹t?§›3Μ0ƒY}υΥΝΑμf ή}χ]σλ―Ώš§Ÿ~Ϊ°ςΦLΟ?Όa•€Xίρ½Γ;˜!C†˜§žzΚόςΛ/9e–ΪΑjlΉζ2…ίcέη?₯ΚρΪ`VrΉε–sΚ@f&―ΈβŠŠ*όϋοΏMϞ=Γ«%zΑΊξΐ4σΝ7ŸYi₯•ΜΰΑƒύ‘Δί(±Ύ«v61Km©†=ΚbV-#ΐ5³Λ7άpCb–q_~ωe·χώvYΰ΅Χ^‹ΛΪ!mύχwAώ τύœxβ‰ςf1«ΛSO=Υ,Ήδ’ξή7pΰ@C,Όre«­Ά2[l±EΞi{ν΅Wψ7±κͺ«ζӎˆ€4βl’‘L\?qZhLΫΧPm(M ¦–L}ϊτ1 ΆΣ)™€™¬¬σΟ?ίL;ν΄†ψKF~ϋν·€wοήξ*Y)Ν›k²Ι&3Λ.»¬ϋψ^AγjΗΛψm·έζ,ΒΘΓΛg~œ§b«%εϋ°£TBΡυK/ω*sΎQN<ρΔ.ΆVΞνԌŠ K-΅”Ωc=œβc†Z»%­œίΛόΈβŠ.NχR„4~kσΟ?ΏΩi§\€ώ}χέΧ|όρΗ‰•Mύυ—Ω|σΝέοσΟ?LΪ€ω²Τ–jΨ£8?δCLίΎ}έu{葇œrχTώŽΚ•Η{Μ)¬P£(αZ^wέu†kψΘ#˜bΟ0ς’Пd’I:¬žφγ?–Ϋ”Ίη;v¬AΉΚʞό όφΫoNΉώάsΟΈpOJ"Δ„YΎ ΙZk­εž9(μQfID@D ‘wια‡v ¦•W^Ή‘šžZ[ύΔ)Φω­‡*5x*H‰€ΤΦLμJpe‘ΪgΏύφ«Y[Up<Ρ£Gvu΅ΐf°n‹‰¬ΕF˜¦d¬Β!Έω曃Γ?<°/LΑΜ3Οώ}ΨεΉƒ5Χ\38μ°Γ‚›nΊ) ―kL7έtS‹.ΊhxN©Ώ/kα‹ΠwπŒώΦœβ²Ψ—λΐZVQ‘sŠ΅` ΊuλότΣO9ι₯v¬ςΗ]γόλg­7»2Kΐί₯kYXεoπώϋοϋ€‚ίφeί΅“ότυτΣO/˜·Τ¬΄₯φ#FŒ&œpΒ`Α Έ―y9묳λζX₯OJόm•ο]X‹ΕπœoΏύΦΥc•ΕaZά†η‹΅\‹;œω4kmε~‹6~_ΨΦG}Τυι’K. Σ mX«Ωΐ*–\~~£lΗIΏ~ύ‚X ξPΝόx†o‰ˆ€”K€ϋΩδ“OΨIΒrOmΊόΦ]Ѝw›cκˆ@H fξrφΖΝΨbν’–¬»ξΊi₯r Ψ5ξ]X1y±/Νζ²Λ.sVj»νΆ›OΦwΈΞ`*}όρΗ»`κψ€3#Ωτž{ξiμΔωιoΊι¦ΞΪ©{χξξο<ϋ’κ,[’ΊέМ3Ο<ΣYŸΘ]R;XnΰFyάqΗεTb•†λ¬ΌΌςΚ+n©βrέ&±ˆΒΚf›mΆ1M4‘/ΞμΊλξσ7YLž}φYg…%eΥH–ΪRϋό>Γ[o½΅αΎζ‹4¬6ΛuwΔrλ0άΌψ[φ‚%θj«­ζ,Οώύχ_ŸάαΫm{w‚2ž€…‘U™yζ™'l).σΞ;―±J¦0-nƒλ€…žUβ8ξŠ?GIi" FΰŸώ1Œžyζ7ΦγΩκΒ3Nξr­ώ+P›@ϋ¨ΊF=eΠ‰ΛG΅rδ‘Gu3¨Ά|OW9L֊&'+«ρR‹;ˆQv ’œ ΪIL79>ΡΰΔyzύυΧΓ γχάsOβςβ2β~²‚ ΰΞG’>VD‘” Δ]`EΖΕ_<³|πA‡Γ,ZρΒ /ΈX]Εΰo~‚ &p«w’ΰΗEn‘…r/(\Ώ, Κς―ΎϊΚ)AσΫΉΔK˜;οΌ3?9gίN}Ή•KqQ„½γ9x΄#"P#„@ψ~ςΙ'έΆWU{ByΞ9aL&βN/i‹ΛΔΡ'Ÿ|βb!Š‰ˆ@σ¨Ή’‰Α»u?1ΌδV"Φ•Απ²FΜ^Μπaže–Y*)Jη”I`θΠ‘ΖΊo|9`Ά ‚±’hJC™Xf›6;Vζƒ`UQ­P˜Ž€ΆP\`©Ζ€Ί΅™{ο½Χpσ‚%–¬XŽxΕΠ4ΣLΣα΄©¦šΚ°ςa1!3Ÿ4$Km‰φ§ϋh^Ά­λ‘Kβ‘ρςΦ[oΉ•oς•λώxΉίΧ\s³D$NQ1AΙΔί:=,€eΟ?q§Xx «Rκ7ρσΟ?»Ώ‡Bq™°Ξ;ϋμ³³Ϊ=΅KD Ι Tβέ„oΛqΕ@Έ³ν'θ°°,WX‘™Ε'η?cΚ-«Ως{k]¬™XpC""Π|Ζ«G—˜ @ΡT‰°„TJI”Jωηεο³κa‡ζ鸏ΰ^@ pbb|Χ:φ™±~ρΕσOΧώ8¬j΄ΰ‚ —#ΈΦρAΑη—β.η|ενH ±΄…ΏIΊ~ωε§`εήVͺAAPΘr3š·Τ62άΏώϊœ¬¬Ϊ…+œzI–ΪR {ΈXΏ~ψα9θ»ξ:ƒ{φΆΫn›“žd‡ϋ')(™ΊκͺΔ.b(`XYm§v2,yν…kΛ5fdΞ9ητΙ™όή|σΝέs7κ‰’Kzώ>3 GθTi(˜|ςΛb’Ο#\δζ›o>ŸUί0Ž@ώ-hBΦz¨%dĈ}a¬ϋB`gƒ‰ΒXWˆΐŠƒK.Ή$°/‚-Α‘T'mbΗΖΎ(•ΚZπΈ΅j ¬Ϋc`ƒ²Μ£Ιπ;MϋΣ§OŸd•·`.X[wή²{>vμΨΐ*Κ»"X`ŸuΏ ¬ o`u †–Iω6A`­›Β΄ό?όΣ]w»"`Ξ‘‹/ΎΨ₯[F`Š·ήzk`cα›mΆYN>Ϊ@Ρz£όίω駟MvΫV ιΞύτΣO;‹&d₯-小γrΐΦ'°±™Sλ*Ψ•Ξ«xŠv7HΒΕΞdΦ’Η]skΩζΞαΌθ‡k‹XχbΗΩ*΄Βzh ΏAžSΟ>ϋlπK/6φ``γ3Ήkν3rέΉΎ·έv›O*ψmγ;|ς₯Pzώύ;δOZŸ \ξΪj'sϋβΰϊΐ³ΐβΨ‰Ÿ° q}ŽΫ°ρ±‚΅Χ^;?ΩνσŒ±+šΖ«U’΅°vΧ†o‰ˆ@cπΏiŽ§ό½ΐ.X·Έΐ.’|τΡG¦“[k'΄ƒ<°“[‘κE@jA ›½αΆ„ΠυΏύ―ϋΠaάυ°pzμ±Ηάͺ=,m­Ξ k'βaϊίjΒ,<ξ=»ξΊkΕ]Ηϊ«2,š†Zq9:±6–Zj©ΪάΒ₯vνΪΥΕςΩgŸ}\Μ9βΞ!X0έrΛ-ΞUΘγ±Κηl• >)ρ7V.₯ΖRΧ.\”°$&]T|Q‹’θρbΫ¬†‡Ϋr©φe₯-•°rΑzˆεͺq½F¬BΔμΉηžŽq”S.χφ+|ςΙΡΣΓνƒ>ΨL8α„.Άœ£«δ›c\O¬©ž]̌ ά uœ›d΅6άώβ€X:AΗ£’΄ΎyηΧ8ίn»νάjzVιꞩΎΗ’Ψ ρςϋξι[D@&ηή–F<xσβ>ΝύZ’œ€‚'g₯œ"Πpj‘Ήj΄2­;B`ύ§¬˜°Ρ}°L8ξΈγ‚ηž{ΡΊTQ{ml7£μ±ΗVt~τ$θΨ1΄/MΡdm—I`‘E ώwYνχ‘GYf+Z';l+±dŠ²ζςUZφΕ<š\Φv!K&_Ș1cœΥΝίν“RύΆρsϊ‘D²Τ–jΨc…TΘςΛs(‡‹?§’o+VAv…Β‚§[%Ύ{n̐ςrλΓ:Ψ*DSnE[q²dͺ V*MK€ηz΅c§όσνdj`λΞrίΊ7-»ZvΜ.δؘ…΅¬Be‹€t–±d²‡‚b]#œ/5ώΤΘηŸ°tb•΅AƒΉμ°pςŸ¬ΗΖπυ³7|c΅……‚%DΎΠW¬ˆ«T­° ΰ 'œΰ‚ιb!ΖL€|ΦUΖYY”fα3XΎWR;“O>Ή©υŠ2έΊusΛΪΧ’7ήx£!°7ύH"YjK5짛n:Γ§”Λ₯P9I±Β*¨ 6Μ<σΜ3ζ΄ΣN+”%ΥτJκca‰ˆ€dqνb qŸc5λ™fš)νβ[’<β2ύόσΟΖNͺ(ŽUK\qu²•tAΉΥJ€―΄rZf™e”Z›, „­Ϋ_βΥέPDΏh^$°bŠφŒψLK.Ή€‹Ο„’NR^flΰάΔΧ³TιXεI²O€ψΔς2υΤSϋ͚/½τ5―#ijK<©z+μλ]_|―S,aαŇ؁HB€±γγ4…2Yν•˜~|μ‚ζφΫo7ΔΦC˜|φ '¬›ύv~Μ»4ΫΤheƒ1.“ωR25ΪΥS{E 8Y2ηSςθΘ‘#Ι[:±\s=B…VN6ŽGΙrȐΤz©P]΄υ‘G)tΈβt(š&>dȐŠΛiΥύ zύ—SqŠY±d*ήJ¨7–%S½Ι«>¨žγc¬τύΒ Υ–Θ’,τAΉωb㛆Š§¨ /&’ 'Ώ5oˆό>ΥrŸE7μj»ζΙ'Ÿ¬e5*[D ΞdΙT%pLφ·έv[χ‘((ή­Ž ΰκb§ %N΅$ξqΕκ1bD±ΓΓ Μ.n ΰVπ«f床Ρΐ'2[† W΅Χ——#‰ˆ€ˆ€ˆ€΄ΖQՎ‘’ΌŠ)«X™zΩe—uŸθ9^αδΏo½υVˆ<σΜ3Oε.x­ Ό#άpΓ ­ΠUυQZŠ€”L)_no;pΰ@3Γ+œ°tΊΰ‚ \m<|ΌΒiΉε–K₯QךJ ό裏Μ!‡b -½]iΉœΗ’ΦoΏύΆΩm·έœ’)­~WΣ¦F:ΧΟ–U:HBIΕ K"" " " "P?.KZŠ?ˆζΗ‚7;―x:οΌσ “Ψœγ­Ψ—gD΄΅ήFΙτϋοΏ›·ήzΛ,Έΰ‚΅Nε‹€Τ‰€άεκšjμrΤ9Δ?ϋμ3c—ξ NX;[Q¨PSyΠUͺ|ˆ+³–ξ¬–ΖCυΕ_tn…qυ+­0r―5ƒ“³Ο>[ ¦ΒHsŽΘ].‡vD@Ɛ»œ~ "ΠΨΚ?λm-α]Α+ΌκύχίwΝαΑ+«’Κ§Z6/Ζ’Ϊc,b1ΑΈUϊ˜–ˆ€4)™:ρ:Ύωζ›‘k–N£Gv«y+'Ύ§šjͺ’-μΫ·―yι₯—JζK’ΛŸ~ϊι$ΩΛΞσυΧ_»ψLϊ#@’€2₯Kπ]wέu ω$Ι HΙ”œ•rŠ@+’©•ΆϊΪ¬X­N1w·R}―₯‚©PέΏώϊk¨xŠ* PΠtλΦ-ΗΪΙ+Ÿ&Ÿ|ςBΕe.½OŸ>f©₯–2ηœsNζΪ¦‰€TF@J¦ΚΈ₯~™q­σΔ½ˆΥ•Ό©_Ώ~±υ¦=ƒ1ί|σ™wήy'Ά4ιη*«¬b?όpsόρΗ§QdK—ΑΛοηΏ[J…—’©Bp:Mšœ€”LM~Υ½– ΰŽ+ι,ρTQNee•ύ7ΗΥΞ+Ÿ~ωεΧ=<#ΌΒΙ»έeuUNb΅β.χμ³ΟVritŽˆ@ (&SF. /·(^ψ ί|σM¨p:t¨9ξΈγ ³(œ|ρωηŸί)ξΒ»οΎ›v‘9εΡVω 0zοή½Νζ›ožs\;ε@©$ΕRyΜ”[D@D@D ΅0VBI„•wΉa&PΤ`Ε”oΌρ Vλ|’2|ψπε«:ωε—.ΛΜ3Οά!ΞS―^½’§wΚ6q™ΌςΚN©[•Š€Τ†€”L΅αZu©ΣO?½Ωb‹-ά‡Β† Z:zθ‘.HήsΜQ³•κ’–1Uw&¦€½χήΫυi‡vpΐμ/’’D@D@D@D@R%ΰC ”R4yλ₯Ξp‘«΄ΓsΟύν˜$ΥΥ†/, Ύ ‹³8 ,‹»mp[$Έ·…,ά%xHpY}Β‚ .θΩfx!ŒΨ(ςͺΎχή{^€ώς<Ν2Λ,ΩΝ M2χψγW-ΫμTUΧJΊHdͺ Wkž{ξΉΓsΟ=Χ°Ξα۝Άώϊλ{Ζ?ύιOž|•UVιΞζΥ–ˆ€ˆ€ˆ€ˆ€ˆ@Όζoγ7NΧΎσΞ;iΈβΣW\‘ŽύΫίώΆCž'< ͺ"ΣΩgŸναz$5GPβοή{ουQύΉ’ρ"<ŽΈΓγ:}Š€΄‰L­qΊΤ‹)§œ2ΛLΌFΨΪk―݈jκͺγΠCu‘l›mΆρόLΌ‘‰€ˆ€ˆ€ˆ€ˆ€΄i§6π·ϊꫧϋβ‹/Rα Qˆπβ“O>ΩGτgœq:„ΪρR{Β 'τν™Χ9rdΈι¦›όεsLΌž6πλD βr M‘„>E u('Sλ‹.υ€‘I³ήutΨΧ_[l±Π―_Ώpϋν·Ž.¨Ν^N@9™zω  έ x¨YnΉεό‘©»ΒΙ+tE‹E@D mότΣOi˜ΒSόγž›kΉRριΰƒ3Μ0CxυΥWλήΏξ°¨ξŽiθ₯δΙΤCΚ.•–—Œˆ€ˆ€ˆ@§ 4J`’ω€ΰξTnΓΝ7ί<0j^LtώτΣO‡VX!L6Ωd(ŒΧ^{-·Eιμ‡~fuΦΐvE ǁ™gž9L3Ν4αΘ#¬Έyl‰f™zκ©}Τ½7ί|³bΩ슒ϋ@BwΌΞHΔ>ηœs†SN9%[M‘ι?ώρaΓ 7¬XφΆΫn Œ.ή·oί°Ζk„;οΌ³bΩfΈςΚ+=GΨSLα£)Vς¬γxo°ΑžG&±ΏΗΌ/εκυfŠ›¬‡0±)Ωe—]sI%“wϊ7χάs'‹/ΎxbβRKοωϋίΔFHμKάύTηzΎ'|od" "%ΐο%Χ‡VέΜφYΣ" "Πκβ΅5ϋœ•iBšr·Ι&›$Η{¬γ4€Δ^'§vZb"DrΠA% ,°@Eԏ=φXb Ξύ7δΙ'Ÿ¬X.»ΒΔ―συΧ_O,YzbUrύυΧg‹ψτ7ί|“˜0“˜Π”X’υδ€HLpκP.Ώ ž}Ψ~ϋ퓁&o½υ–Ω¨€ ύ+j6Z`2φΨc'λ­·^ΩMΎώϋΔ„΄δζ›oNΜ3(1οqί'ϊ؝φŸόΗϋqύχ'|πAb‚b²Η{”ν‚ΔžrΘ!Ξό–[nIΖόΔ’Ω{Yο$ΛnΧ›Κ“Ιd=ΥπN² `xα…άΥ.δώωμ³Ο†|°ε“γM7έtžœ•FΎεθ©Η[ϋ%" " " " νBΟ£FΪηŸHL9rdψμ³ΟΒΞ;οL\πh7ήx#π|•·wήyΗ=sφέwίPwΛ\ΜQ Μ8γŒξY³γŽ;†Ώύνoωκ=ΪΓΔ―°J+…>}ϊs"/Ώόr‡rωυ색Da·έv ύϋχχ?’L.»μ²|•eηρΰΪk―½|ϋ²l!^A›mΆYXm΅Υ£<―³Ξ:nΖ³jw^Lp\rΙ%ΓTSMεϞε˜ ‡·ΥŸώτ'gΞ€UŒ’Θ±–ύ€D¦±θρSν8βΒΚ+―N8αΏ€#6ΙD@D@D@D@D@zΒ°iς}CP"τ Ac½f˜!°Ή‹([nΉe~UΥyκšmΆΩ2΄W~Bδε#¬λ’K. ζΙδβGΊa… κ*Ίύϊυs1%VE[E„,Κo·έvαΰƒv>qϋό§yy…γŽ;.]lžD.ž†ΦVŽω'Ÿ|ΜC©€ˆyμΧΔOΜ‹)l±Ε. 0 €\oŸ‘ΘΤΫΟ€6Ψπ­·ή:œ!Oe" " " " " νM―#ΌmΝšπXΙ c“L2I@ΘΫxγ&šh’όβšσω6*Υ+²°Ί`αtΟ!Ϊ¬eωϊ)_© D,F|葇Β]wέεUωά–εΪ;ηœsΒΈγŽλ"XΉυε–YΘZΨtΣM]tB@λNΛ3VξΈ²άBߜ……@’oςbez³IdκΝGΏφ·Q†;ΕES&" " " " " ε43zƒΰyAyt7ΚςmΤͺίςνFΚ1bDΨ}χέΓ3Οβ9!„Ύϋξ»αΛ/Ώ /½τ«ΛΪO<–Xb χΔ"±»-Οx«U»XGΨ ‰ΦIIsα…vw—[Ί=‰L-}xΤΉ,„&FB2dHv±¦E@D@D@D@D@ڌ€ Άπ7S`’³δ&ʎΰΖ0φ6XQ˜Ρς'5ΚςmΠ^Ήϊ‰πΈψβ‹Σfρ¨aD:ΌkͺYΎώjϋΐ(ߌšf‰Ώ#Ωύψγa–Yf©V}xε•W|δ=rI1ͺΜ£>vΨa‡²ΫαΡΖθr€HΩuΧ]Λ–iφΒ<˜#0ε=Γΰρ—ΏόΕ9Π'ΒηΘγT‹y³ϋίjυKdj΅#’ώT$€Jώωη‡σΞ;―SΓgV¬X+D@D@D@D@D@ڞΐ2Λ,ΣΤ} ~Υ-\~ωε2uΦYgyΈ9Ž°‡~Ψ=wjuβ½χήσ΄rε;γŒ3ΒG}δ ₯yφA¬Αlδ³πΐψ49ŸHΚ“Ž“#ˆάB$Ηq£ΌΥ³§žzjΨoΏύΌŠ/Ύψ•» «΄«Ίͺ<ΕΰSόΩ|žT;†1fχ:l΄Ά@;$&|’?Δ,¬>°Ž€α6ϊ“n„ ’Σ)Ϊwί}ξΎϋξ`#Ψω">³σ±Ÿ6‚`°‘νά Œν’"σQ£Fωvδ£"y‚y&ΕίΉζš–PΛ;ۍ$2u#l5ΥuoΌq°!#ƒ )ξΌσΞW¨D@D@D@D@D@Ί@³½Žš±Cc=Ά „‚α9tξΉη’-πhΑH}γ7ΦlϊΆΫn ψΓΚ–#-αw$“^xα…ΓςΛ/Φ[o=/ΛσO?±Ž‘Δ©‘‹Ά- ^,§ΌΥ³δΖ%ΜmφΩg‹.Ί¨·±Β +x•Υφ!ίfv>»§Ÿ~z@ΌBX#d-ώ!ψ`•φuˆoŒάAO@IDATνΎϋξσΡιβ¬ΉζšeΧU[ΨΩΎP'7 $¨$‡77Όδ¨‡_­ωuά4α†^tΈίΗάlςV’ŸηyηΧ…υX _uKΞsCΗωInήBςqΔG„ 'œ°ώ–γ† }ΌqηΖ·ύξ2Ύ'Λ-·œη]hΗ‡’ξβ€vD@D ^όή:΄ήΝ*–oΖ£-ΏIσΟ?‰ Αo;βΓέwΖθ'£Ξ1*\%Γ£gόρΗχΪ•ΚΔε'ά·Œ9ζ|H¨›{Ε•VZ)+ω¬gθΛ€“NšŽͺGEEφ‘€ΑNΜΤΪ‡z«δ>σ”SNqεΆεO¦|rχre9ώx6‘ΰ<ω°πϋτΣOγ’^ωωΏ³°WξΎvΊ] ΰ’HŒ¬·λTΏ«ΰG›XpβκyƒDΒEŒeΔ}󦆷& ™‹W_½"o€x[C}ΡΉZΚ­λJ_PΙp‘~ϋν·KώpA―ΧΊnnμo΅’–!s/Ίθ’π»ίύΞέΤλνKΆ<7ΝψΗ?:$Ν–ΙNγΒ½Ψb‹Ήόκ«―ψΓ%žγΘΡκφΣO?…5ΦXΓoήΦZk­°ςΚ+ϋήW\Ρo؊φΏ7xπ}ΰ ,b–LD@D ύ π;Μon#¬9žbΏƒ#L*bNg&κΰEy ;‹uζ?:²F~}v1)+0±Ž‘Ρͺ…Φ³τ…—±Y+²Ως™΅υΤI2oΞ5„»Jο"ΫO1Ε%Ηη±Ησ—•κξUΛνaF&mIΐΎπΒKl”ƒΆμΏ:έz8Ÿμe΄wΜbΖύάΆκ’Ύ 4(1α)±7,ιr MμG?1α(]VmΒΌ€’…Z(™`‚ Ό—_~ΉZρŠλΊsoφΆ‹φΉb'~]ΡΩΎXN€Δn°’₯—^Ί€‰αΓ‡—ε_R¨ΚŒ g‰‰-^Η†ι"6Ο<σ$ζρ“Ψ,iqsΧNμ†'YpΑΣe­:ažq~.Z"΄‹6ά±sψλ_š.«4Q”›έ0'–£―R5MYn#Ηψ~π)h,xε>¬³ΝΌ³œ;~h^ύέqΥΦ£ΨΛB?OŽ9ζ˜΅_Ωy2Ω•L֞ζςμ³Οφ²#+΄ηή¨Χ=F>ψ ο α['žxbΈα†αoΡσδMCΉΦc&yxΠζ›o^2#wπf)& ¬U§έ„yIΔνΊ—ηŸ>τλΧ/ϋ kxγπ‹cX―αfM\Ξvϊι§/ToΧHHIξ\―£‘o ά―qkoe#GoLΙO<sΜ1G0‘).*ϋΩYne+ΣBΆ"@2χ'5ΆΗ#ͺYF>$κ'ΤJ&•VΟy₯W*Χ–Kdκ GΉ‡ξ#α#<̐ 0πρΕn‡°’zHzύn‘τπ(ΞEΆIHȈ„ Ec΄62E=†8…‘t1kˆ$$ž$–1β£e HαβΫYλj_ „CO?ύ΄Wː΅<ςH§ΊΣ•ΎpSJΗ­ΆΪͺ€mΒωΘ=Π™\UφΆΗE+Δ―|½%δf&šh"ς7/ΖΞΔͺΌ|ŠΡ:Kψω Έηe΅ΞΟΞrΛ·₯yφ$ΐ=|g„&ΆαŎLD uHdjc‘ž$€ˆ„ηj1ΔΏ’ „‡wry°ŒυόXΙD »0 ΉπͺcHY†ΕSζΟώ³§JRiŒ„ΧG}tέI₯©+'M6ΩdιΘ^¨Μ?<§ΑΘεΔ±]±τ1!„&ββαc!―aΰΐžc­^oτ₯D/!xΥkδŠ;ωδ“έCͺήmΛ•'‰6ϋΨκ£–Τ:ŒC"ΝJΦhn•ΪΡrΦ%ΐ½;χ •Δ¦l^!.ι~Ώu§zΦ{ Hdκ½ΗΎ-χœ$LjHE Α)Sd;•Ξ`T0ŒP+Ξ=’cx¨`Ρ …δΕϋοΏ ͺ#œ cτ³ΌqΣςσλβ<ήUτιψ㏏‹:ύΩ•Ύ Δ!6΄άrχ„Ο>ϋΜ…bE_xα…ž4ΊžŽu₯/εΪ6l˜τΒHfΥ’e–ΫΆΡΛ,ŸQΨi§‚εΠ Œ0ΨΚVλ8Πw„W™ˆ€ˆ€Τ"Ε&D$'ώ•θ„eQ|²L&"ΠzΖj½.©G"Π‘ήKό°^S1Jϊ!ͺ‡šΚv†@™φή{ο’Ν ½ΒKQ₯+†·φν·ίv¨†ΩΖgœΛγ‚+ΈΒΓψ©¬ή‘δcΩΟτ…4D&<–βˆ)Œžvζ™gϊˆs„‹1ς]QλJ_²mΠŽ^H UL>­Ρi—_~Ή‡Ϋ 0 άzλ­©h9:ϋT­νZǁm«£ΥκΦ:ήIIBRο<φΪλφ& ‘©½_―ι=υ LY8x?IhΚΡt£ ΆI~€…^Έ€jΔ'ςυιΣ§dy½3ԍ}ϊι§6eYΉ0: ώόσΟξ 3ΣL3…λ―ΏήXsαΩΔ:Ό«ŠZgϋλ^^qžΟi§6ΜŸ!‚yCIΈZλj_hƒΌG›mΆYΈςΚ+]h:ξΈγ ·_€υ–Aθ"1;7ΦΧ]w]˜tIλ­’ΫΛΧ:t¨φ£ΫΑ©AθaΖμaϋ£έι:θ ΟίΥ]#ο‹LšE1 ‘$+Žΰu„'ž:]5’{c1·S¬6HΊ<ί|σΕE%ŸxθŒ5ΦXœ8\pAϊχ裏z9FΊΓΣ©λl_hƒQqςϋ°τϊλ―{a–a­~u₯/ԍWΙΩmxβpϊι§{8a=νΧκ_½λ ΅DT'Τm·έΦ6Β !›„†ζ+ϋΟ²Ωf›-Œ?ώψυβPy6# ‘©ΝXoλ.arGuTCv{ĈJΨ’ͺ$Oqδγ?ξ &9=‰ΚΈ•―£Φά…†Χ-gx ‘―)GΒmŒ‘Λž|ςΙr›V\ΦΩΎPα|φέwίpΔG”ΤO(ίΫoΏ]2_I 3]ι U2άν·ίξ^L;οΌs…VΊgρYgεΧ(D&Θ·[xΩΰΑƒΓ 7άPΙ1eTΓJηgχU+" " " " έE@"Sw‘V;"pΗw€ “;UAn£oΌ1·D³"Πu1Sήc).ϊL$³D)ηρQ­'xΧΰwχέw{(CΒ_{ν΅„ήoΌqXe•UΝphγ•W^I—ΐsmίz뭊›t₯/„€1’bΩ‘‡ž{ξ9χΨΩn»ν<©yV|jv_’Έ4묳DhΪΛώ1be4XΓ…π΅FΨΝ7ίμυEΡπΐτœYδͺΚφ#N5Κ›§/œΩs#φ½ςAƒu(_΄=rZ‘Œ$ξ¬=τO“‡‹c-ΏοqΉ>E@D@D@D@ڟ€r2uαβeƒρΙΓ a)Λ/ΏΌΏ}ζAŠ?YאC2|d"Πhδcbτ8Dˆ¬±œΡΰ1’1Ίηαχίώ2dˆΖ†PuI'B”Φ\sΝpΒ '”ΤΫ(—$Ό€`™Ό²Šτ―³}!t‘f·έv Gy€Α!ξ–[n Ρšέ—(π Ψ•σ˜$—VτnΒS .!θ-σVΏF†άδPΦ9Νȍ’ΰ;½a+Ξ·VψΞβ±Βˆt$Ξ H‡Ÿ~ϊ)ΌφΪkaζ™gnΚ¨c„εαΩ3ρΔΧ<μ]ι ΓήΏρΖ>ς^₯ΆΊ«/5wΤ ΰmEΨΧj«­V€xSΛtw_κm!΄ΡΖ „/ςu—ρ2IHtm΅#" " "ΠΞΖl翎Ύs£Ιƒ(# E«$0±ž$υŒΪλ쭟xwππ|ί}χ•„WτVΪo(Go ΩgŸ½)IΐI¦]IτΙχ§+}!Qτ<σΜS±­ξμK~Ώςσδ"Τot[wχ₯3ν1ba3¦ΡΝ^ν‹€ˆ€ˆ€ˆ€T' pΉκ|JΦβ‘„h„ΥΒuμ±Ηώͺ‰Q% υΠΒLήyηc4¬rΣΌn¦‘두7α2huŒόFX‘im΄Q·t·ώ‘Ur—΅R_&œpΒpΣM7ΉwN·ˆ*tw_Ί»½J»>lΨ0Ώ~?υΤSaΪi§­TLΛE@D@D@D@F#‰Lαg¦‚›”Cάΰ xΦͺ€@ΟΪƒ`”Žςσ¬Λζ !? ρo†8έ―_?Ÿ&12ω<i1qn#λT]"Πh}ϊτρ„ή±ήΙ'Ÿ^EJ‹iJ*#" " " " " " Ν# ‘©ΫN8‘@©bEφί A€+•h™ +D!ςtΥHD}τΡGw΅m/" " " " " " " -G@"SΑCBξŸEY€`ιςΕιL&" " " " " " " =‘€ΧqT{μ±°Χ^{…“N:©Ž­‚€vΪi§)L.j*,£‡ΐΠ‘CΓ½χή;zW«" " " " " "ΠΖϊΨ(]‡·q»½λ+―Ό²·Yλ!tάqΗ ?όs8ΰ€Β5Χ\fœqΖn﫨ŸΐcŒQFΪBD Ηΰw\·L=ώ0kE@D@D@ΊH`ŒΔ¬‹uτΪΝΉΩόζ›oΒ?ωΟπΜ3Ο8‡9ηœ3L0Αa5ΦpΟ%%ω‡v\D@D@D@D@D@D@z‰L½κpkgE@D@D@D@D@D@D@D 9”ψ»9\U«ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€τ*™zΥαΦΊ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@sHdjWΥ*" " " " " " " ½Š€D¦^uΈ΅³" " " " " " " "Π™šΓU΅Š€ˆ€ˆ€ˆ€ˆ@&ΐ Υ]tQΨ~ϋνΓςΛ/† Ξ>ϋμπγ?ΆΜ~ί~ϋνaχέwοrž|ςΙ°ΣN;…―ΎϊͺΛuUͺΰν·ίφ6^{ν΅JEZbω9ηœN>ωδ.χ…sε”SN)\Ο»οΎλ|^yε•ΒΫtΆ #ˆsΌGŒΡΩ*JΆ+Χχ>ψ -σΞ;οx{―ΎϊjΊL"Π$2΅λ‘SΏE@D@D@D@D`45jTXguΒ6Ϋlzθ‘0ύτΣ‡G}Τ”—[nΉπρǏ¦ž•6ϋο;œwήy₯ ;1‡πƒ(ςέwίubλb›ΐŒ6ήύbŒ¦Rόη?Í7ήΨεΦo»νΆpΣM7ηΣO?u>6Ν6Ž3Η’Q‚_ΎοχάsO˜cŽ9έψδ“OΌ½χή{/]¦ hWc΅kΗΥoΡC‘α†nΧ\sMXo½υNππŒWΣ™gž=τΠtΉ&D OΰψGΐ¨Ν=χάαλ―Ώγ?~ΡMZ¦\ΎοΟ?|ψβ‹/Z¦κˆ4’€D¦FT]" " " " "Π F4φΨc‡•VZ©doρbΒ»ιΫoΏ-YώK/…»οΎΫ=C¦žzκ°ΤRK…Ε[,-sρΕϋ²Ο?<άrΛ-a’I& k―½v˜a†ψ[e•UΒW\πp‘? }ϊτIλΞOότΣOακ«―O=υT˜bŠ)Βj«­ζœsΞ’bi”Yb‰%JΦUšΑ[ε²Λ. ―ΏώzθίΏXqΕβφΡG…λ―Ώήϋ?ε”S¦UЏi¦™&,Ήδ’ι2„—+―Ό2Œ92Μ>ϋμaΓ 7 M4QΊž ϊuλ­·†±ΖΛEΎ7ί|3όόσΟή&QΧ]w]XύυΓωηŸ&t°ΡF…ίόζ7>β…τβ‹/†ρΖ/Μ?όΞyŒ1ΖπϊοΈγŽ0ξΈγ†Ύ}ϋzcŽ9¦σ™kΉJΪg―›«Ί*|ψα‡~,W_}υλιPΈΜ‚x p,ΰ-ξόη›oΎ°ΑψyΖzΞ x­±Ζή?Ž=b͚k½φΪπΜ3Οψ9²ι¦›†xŽΔz?iο—_~ K/½tΰ<Ν!žχή{oψΧΏώfuVη˜]ŸŸF ›qΖΓβ‹/žΊδ’KόΈgλζ|0`@˜yζ™ΣΎΓλΑtν―ύkITΖq₯―ύϊυσ}ε; Ά"`1™ˆ€ˆ€ˆ€ˆ€ˆ€&pΧ]wα‚’ <8±‡ύͺΫ™Π‘˜ ‘,°ΐ‰‰‰‰L‰‰‰εφI·ϋνo›˜¨”L>ω䉉3‰y«$‚—œvΪi‰ ΎŒu,3‘ΐ·³|>‰ 6Ι<σΜ“¬ΌςΚ‰‰ή§­ΆΪ*­χΨcυνγ{’…^81*1q#YvΩe“qΖ'Ήΰ‚ b‘„2φ`Ÿ˜Ψ’˜—–O›γu[΄\vΒ—dΊι¦Kfši¦δψCByα‹Ό˜…ϊφ=φXv³ΔD¨d‡vπe–χΙΛΜ;οΌ‰‰KΞƒ~ZXUb9{νΨ'Ψ³­ 1‰‰H>m’’—1ΠΧΓŽ&%φ•<ώψγ‰ Z‰‰#ΙΖoœ˜°ζε6Ωd“΄ξuΧ]7YpΑ“©¦š*Ω|σΝ½~Ϊ²ά[%eΰc"Hb\ZϞ{ξ™–)2ΑρζXG;ζ˜cΌ―ΤΛy2Ωd“%Ώϋέο’ΘάD$ο― AΎ Ηζ&Π%&Šy]3ψ™h«M8ΨŽ;ϋΗΉhωΓυœO&TyϊCΉxΌMΌLΛe'8/ΨχhoΌρ†oΟρˆFΏ9ΟM M²}7aΏ}β4ΐΔΔ5ί~‘Eρ}α|ζΨq,U¬RŸ"ΠPPe" " " " " "PKώμb Λ<μ#JόύοO,irZεΆIΜ[(ΩvΫmΣe<Τ# ρ0 ‘‰ro½υ–/Š’Œyπ$–£Θ—Y~%g†Θ@Ϋόγ}žζ-εφζiβΛς"“%)wΡ Q Ϊα‡žL8ᄉ%ήφE–kΚ…“(μ|ωε—©(ΈmόD £K Ήpa^5>χ§ˆΘ„Θσύχίϋv–Ϊ…ϊQ’ΡΡGŠm΄ ‡ΌΘd;‰y”%ζ½δΫ":!XeΨ!„°" uEqŒc5tθPfͺbΆ>|ΈΟσc(bήTι²ZY‘  Aν°ΓK~ψαί”γSκΖ²B σρψοΏώ‰yD±ΘϋM-ί“Οs0° ‘ŠeζYζ‹8yδŸgŸ·άrK/SIdBt3/²Δ<|σHJ&žxbgižJΎμΒ /τeΛ|ίO?ύt/λν_™Μ3*α;ƒΑ›~r¬e"ΠN”ψΫΎΉ2ϊ0j›‰5>B˜y)yŽ&B•L< „Ηa„d=χάs%£‘‘ΨšP#Ÿ²FΨafu°­ytΒλ0σPρ08gΝ<‡YB•=cTΉrf"X $*·‡6ct<=ί|σΝ>ηwσ.ςp%κ0ρ μΊλεͺK—ςƈd&Xy(u“³κK/M˝ DŒ5ŒΑ­·ήΪΓΑ˜'œ Ϋn»νΠ4Βa•7σΰρόE&ΰω*BΌξ»ο>±c#噧Œο³‰Lιζ„š­΅ΦZ>obRΨoΏύ<¬νώϋοOΛpœb²ΓώχΏi™z&α#¬οΰƒNΓ㦝vΪ0hΠ Ι3α§bu›CΥbςtψFή°xΌ —#<’K ¦„μ-Ίθ’>Ο>pΐ>]鑁Τg}^„:8N&΄yΘ IjΎκͺ«¦Η²R]ΩεΧx, ±#L1~—²ε4-­L@"S+υMD@D@D@D@Z˜€y0…έvΫΝr‘3ρˆ|JΡΜ3ΕΕςΧXT°2‘ŽBY³¬t–\@ζ]―’±Œ#frξ 6!’²Œάcœσ¬cΖz¬ΧΞ2­§]•F(ύ7²fΥ%" " " " "Π# |™„ΨY’M“\ο }s/ Όρπ°9ί/₯]vΩ%υ‰€’χNœ/ς‰K^˜ΐ+Ο’ΌΡΜrΉ'P~=B e.š²E‹μ²μ4‚ΔρΗοž8x΅XΈ–{1‘°ό?ωOZ”DΧYΛzΕεˆkY#I7l,tΜ€#Ίδ­\ςI7c5Λ;”ΞΗ ΛO'ύΑŠQκ²πΪΒ3ηuΑ.Η€Ρώͺe΄œGΎ―xΡWD,{,žxβ‰ΤˆuŒβ†ΐSŽ)λaμœς<Žγ”ފΆiΉΏόΨ!RΕγMψœεsro:κAτŒ£½Εz9‡j"ΗαάsΟυP;„2F΄Dέώέ tσ©œEaŒ;™τ4™zΪΥώˆ€ˆ€ˆ€ˆ€ˆ@“ "‡χ†%©φ|K ΞΠλxΘqΔaΗwtaΐFΛςP%†Ί'T΅=φΨ# €XRκ†τ’œL„&!ΰμ΄ΣNξBn›rFΞΕΝϋi‰•]dΩb‹-\€!ηf‰€]$B±δί.H U3ΒιΞ8γŒpάqΗ†¨ώωη=B’αXˆ:Gu”稲$ζΟ’rβ 9π„Α³ηΤSO x2rΘ!ή…¨b#ͺyˆΉ‹πrB¬¨fˆi6’ηm+­@@‰Ώ[α(¨" " " " "Π¦`²#ΓUڍόΘY•Κuf9WΜ{Tdϋ"}.R¦\[ε’ŽΗrΤY„’C~Δ3κΐΫqο<Η­H>MrqF7+bŒWΔ[:k„ φ6όσ—­‚€Ϊ:xKε qœ@—/Wολ―Ώ>¬ΏώϊaΕW Ο=χ\C;όΑ4΄ΎΡ]ΩoΌvΪi'Ο!E_nΏύφ°ϋξ»WνΦ¨Q£I½£ν½χήα–[n‰³]ϊ¬ΕwŸ}φρΔέ]jDwŠΐgŸ}φΪk―°άrΛ…<°Suh#$2Nϊj[D@D@D@D@ڐΓΣoΈα†αΏύoψέο~Ί2ά}~χΧZk­pΞ9ηδ·υό‡~Ξ>ϋμΐ'φο;œwήyχ hήyη O?ύtZζΒ / #FŒHη;;qόρΗ‡Ν6Ϋ¬κζ]tQxβ‰'ͺ–ΡΚζΨsΟ=ύόŸmΆΩΒsΜќFT«4‘ΐXM¬[U‹€ˆ€ˆ€ˆ€ˆ€τ@Ο?|ψ駟ΒΕ_άπ‘Θ~ψα°πΒ χ@jΕwιγ?yΝ0FvcΉjφΦ[o…±Η»Z­k§žz*¬±ΖαάsΟmR ͺVšK@"SsωͺvθQξΏώ4”κΆΫnσ°UVYΕχρ…^·ήzkψ裏ΒόσΟ6Ψ`ƒ0ΦX₯lΨc…χή{/Μ8γŒώ@Ν'vΑ„οΏή½h.Ήδ’°Ε[ΪΐbLΏώϊλαΞ;οtœρΗ?όσŸ M4«<²τωZ}ϊτΣOΓe—]ζuφοίίΓζž{nί–Χ^{­O―·ήzι²rΥφ«\ωJΛ`wΥUWωκoΌ1όπΓή§X/¨›oΎ9Œ9ζ˜a₯•Vκ Θ½τKαξ»ο―½φZ˜zκ©ΓRK-[l1ί–ωΟΒ7ί|ώϊΧΏϊρωνo«N?―Ύϊκ0Ο<σ„\Π—Υb”nψλΔwάΖoΌ0Λ,³Β*ρxϋύοΦ]wέ’’•΄…°2ΕS„ΥV[­ƒhωέwί…ϋξ»/<ψΰƒγ²κͺ«†I&™€€ž{ξΉ'<πΐα—_~ K/½΄‡še ΌόςΛ~?ωδ“0`ΐ€°Ξ:λ”xίΥZO?‡WΏύφΫ€—ΡFm”žsΟ<σL€;β(^`σΝ7_˜xβ‰ν <8Ϋ•€ΘΗ1ψΓώP²œzώχΏ‡χί?pL8>±Ÿ„N^~ωεή>Η”σ{ΉζJ·―vώ§…4!έE ‘‰€ˆ€ˆ€ˆ€ˆ€ˆ@A§vZ²ΐ $φΌ’˜Θ‘vΨaΎ₯…Έ%γŒ3N2ύτΣ'φ`ΨΓpbΒBb9fš·έvΫΔΔ‡dΩe—MΦ\sΝdΒ 'L&˜`‚δΙ'Ÿτ2φP˜Mbρ^ MTHL|Hλ`ΒDoί„*_N]ΤI]Τiβ”/―Υ'ΆŸnΊι’™fš)±‡ώΔ„1oΚ+―τνωgα€ώ—.(3QkΏLXπώšΈζ[{μ±ήΧ2U%/Ύψbb—7q(6l˜3αΑϋΑη Aƒ’©¦š*ιΣ§OrΝ5׀՜ώωΞ—γc"ˆƒ1Ζ#v葇&ΣL3M2ωδ“;SΛ•n›`ύα‡ξ‹Š0ΚnΛ΄‰I‰‰=IίΎ}\’E]ΤχΗςR₯EMLKL”ρ}ΰψrό8LhL˘π’˜X˜°+―Όrbαc^Ζ„Ά΄ΜV[mεuSνr~ 2$]o‚›Σ…Zȏ1Ηv& y™Zλ9©{²Ι&KΦ^{νdΙ%—τ>›€–˜ζuœpΒ ‰ ₯‰ ?~ώ1mα‘ήοΒζ™gφΊL\r|GN?ύτΈ©—ʝiMˆ@7έΨVΩ¦ΎόςΛΔΨ%tHmΊι¦ ].:έi–\-±7jς»ξJ,v6±€‡ώcxτΡGϋΕ _™©ωE=v~δΘ‘ΙŽ;ξ˜p|ΫΥμmAϊγΪΘ}ψϊ믝ΕΉ7²ZΥ%" " " έF€ϋ$D&Œ1σΜπ‡ϋ]wέ5±P,_fy…όaή\ϋόγ?ξΫόνoσyώρPb‰¦ΣeζΡ’ :4/*2ΡσόρφΏψβ‹B}B0γ‘ώ«―ΎJΫC°β™$šyΩ$όU²"ϋUΘD;Ο>ϋ¬³Ίχή{Σf—,χUbJΎμΗtρρCπ€‚W4σμIΜ#ΙE’ΈΜΌΛK*gΛ~fE¦"Œς• φp<²ΟWόγ]쉬·ί~{Ϊ²" ΒΗγν·ίφ*‹ΕΜΣ(mb™e–I–_~yŸΗ?ώαν :FϋΧΏώεΛΜƒΚYrϊ„m’Y2υdΦYgMΨ/¬ΦϊγŽ;Ξ…­W_}5V‘ B²7έt“/Cdb~ί}χυσΟ<ΏΞAσ²KEB Z‚|/—=i₯ΏN €ρ=Š†°…hE1–ο±ΗΞΌτΌηlώόΫλSΊ›ΐhMόMβ;\ IˆΛ,‰Νp‡όΛ_ώβ‰ξμΛgί•ξ±Ξ&;ꨣά}Υ.4ξri?”ΑήS¨K’εαΒΩ›·αžKrCŽg».ΟW\qEΓ»ΨΐH6ϊ όιOςΡ^ρ%·ώϋ7Όƒω‘bή@*¬5βLšh™*ψŽΫƒLΛτ'ίΒ8/‘θ5_·y xέv³Ÿ_Υ”ωήt^5 *'@Θ“‰ΠΜλ„Ω~oΜθsά'c„^™ LΐρyΚ0zšy»”Œ’ζ+;ρΟ<¨<τŽ02Β©ŠτΙ 3qΓCΆθΣ 7ά.½τ΄„>ρWɚ½_Ωv 3―_D"aS„ia„§1ΚίΙ'Ÿμσό#τΚŠ.ρ-Β(m03AHαoΡ-s„2ŒΠ0ΞŽάω3αɟΔμe~Ψf›m<<ΝΨ?–Β™gžι³'ΒίeŒu.7ηœs¦χρτŸί<•Ή¦&tRΟw΅Λ.»x΅Φ›ΰγχν„ώaμC ΧˎώΗ:{ΙξaŒ„»Q†~±ŸΡL`υγGc£-XμΆΫnΎŸqσ„ς~\wέuq‘s$—S<Σšn&0ΪD&Svƒ©ξ«J +Γxžzκ©Αά==Φ”‹!”ZIιΊ™WIsΔ&#(q2΅έ/x¦œ{|9B‚)ΩiyβwΩg™ˆ@k nž\ δΘρ½m€•)¦‘υ7’"#Ξ4’V©£Ybr£φ―™’twΎ θ‰#E5κ«θ)ΜΛΓοΫΉ‡η!7ώuΦY~oLΎ–ρ’™dF€3Ο›D’λFάσ[Θ[ Ξ"}B@C€ OΣN;m0gJ*«2ΣμύΚ6ίG α*y©Kώ&³Ε_<°Žς=τP—ψv–QΏ~ύ²]χη> œπb>ζ…Šη ŸlC^%Ž/AΘ½ŝXefŸ}vŸ₯ω”Μ.=稇<\¬ΓŽ8βηPcahžkΚB½E֏;ξΈΞ|RZ’ιη0ΫfΟ[ϊ€`–5Ξuξ'-ΚΓχ QŒ¨γΝ‚^šά3rŸ“€.G‘ϋΘ¬YJ†μ¬OΣF֊τ A‚,|°ίs―ˆ`ι<<93λk"Ngχ«Vέωυ•Μrωsϋm!sΞ—¨^ˆγΤYλ,£j}₯˜₯ Ϋm·]‡!Εσ2k<!Π/κα>ίrNe‹ψ4ž]u‘ŒώΝ7ίτη#ξχqΰ‚δς΅Φs~ΰ<€`„`ΕyKΫ–ΫΙλΨίό>[XŸ [x_ρΌτΞ;ο„-·ά2nRσ3&³GΛΟ–<7D˟qΉ>E Ϋ ΨΙ9ZŒ8`Ϊ΅rΙԈ1•άŽϊ‰_^ΦΙ‘xŽe’ΪΦccϋ”!©Ÿ•ιΕ²ρ“ά:vaKΜE6±‡Δ„‚Έͺδ“aϋυ8Y’Ϊ™ ―·1uΐc‘‰&6˜8kνYo"–Ηͺ“ΔŽDxE’δ‘ηΚN&N+ΛMIΘW$Ρύ!9]Ψ{ΣαΣ$M€}bρ±"}>ςΘ#½b¬ι›Ή˜zόx%Τ[τψΣ/ψM’>ϋaHˆαŽf7DkM@Žρμζfλ±LΉΟ­j$δ1δμ"i`dca:^₯ύPϊ1†ί ΞO’ώζ7ΏIcΕ)X«­ZI Λυ_Λ’Ή ͺ1±έ\x.¦σΖΉjω^ΖBνόόŒyμΝmr˜%Fε8“wΒD^ߜsΨΔΩ’ͺΈΔσƒΞλy%μ¦7!η›½΅JΛΧκSZπΧ YAΗsH”λP4»σλ™½LlΨά„|ΡΈ¦rΘ!ž##» λνe€'`΅·Ύ$ ΅‡ί”ο>ΙF-,Ρ`2Ÿ5rTΠ–yμd—L›ΨžΨΫHO Λώ“§ƒνςΖυΚ'“½Hυ땍ΐU‚Δ`ψχ“…&v$ͺ”ζlb™=xϋ}nLΤΝ2ξ‡ω͌ƽΙ’³F>U~SΉφc\cΈ‡ΟZ‘>Y(“”ΫΞDˆμβŠΣEφ«ήœL1wΟΡxNα>$k&Žω½#ΛΈI>οη"‹,βΙΨγv–ͺΔο‰γ|ΉΟlN¦Ξ0β^>ζMŠυs½₯―ΌςŠ/2OO Χσiα”Ι&›l’Διά·oΆΩfΩ"žΏ—ϋe~CɝEμμύΏ/δ{ŠΉΏψ}ΰw$klΗ³!Vk=9­, -»ΉŽ²/1I9Ώ/$-/gόΆΈ›!_‘DsŒ‚ ΓƒF²ο(ͺΕ‡—"}f?³7Ζ’πγSξa’vκ9ώœ'<\bζ™ε‚"¬1κ1―6OΪψx@C4cΏ*Y‘Dƒ<˜pξ=ςΘ#^ ?[nΉ₯sράFθΒθ?›˜°H[΅’zεϊׁ@ώ‘ΐ― j‰|΅F©4R B57ΰYCψΘ^ŸΚ‰Ν”―Υ§lLsώ1 Χ,Ύˆš–Ο!‰ YΉ β»ΐw‘k$ΧDPSœκL³ 7gΤ‡q›‡hɈ*¬γΌη&ώ3Β β)υgo~yπ§ξr‚WnΈ>ΧŠ‹ˆφEΔδ"}ŽύŠŸ΅Ž=εψ]ΰšΟˆ4άΤ":s™wήyΣ0γ7•DιΨfφ“kl{{κǍΨόθ<”™‘ΐΛF„’_„yσΌυώ ’aEΪΚφιr#E©‡s΄ΦΘD΅^NyQs˜eG€b;Ž—yZWm*ΏŸšήF /2qΒoχNάγp‹ΘΓύ/’9†ΘΞ5Α!€€Ε|XΖ}L4Ύ{ό^q/ˆρ‚2Ό΄ !4φόΆ°¬šΘT€OˆΤΓo1χ­ˆ;$ηχŠί@ŒϋSώ*Y‘ύͺWdβΎ•~ρbŠ$ΰX-‘‰mψ-ΰE/Γί}χ]§ξw’™GŽ1xQ₯Az²"SF±ξψYDd²PJίGPβe/έΈξsΔ±Γ8‡Έο@βΨ#TYΈ\byq}=ΏO¬ηw‹—”α>Šί²˜›ΔέΌμ@EΤ€ΏwˆmX­υζiε μ97x‘Ασ}€k᭚ΘDR|Ύœ³Ότ¨eY‘)φ!ΝΌξύΈ„CΔ/^a™ƒώ΅Ρ"2ρvœ/%‹¬‘Ÿώπ(Ꮛ$Δω’fƒθ)ΐr.¦<!ΤDγ‚lacqΦ?Ή¨’0Κdί ° Α…ΛZFϋόΈlΎωζ~SJ½(ΞΡc‰νσ"o^)Η(lYγA„ερM*£|Ώθi’ψΗ~㝇'ΝΦYiš‹?ΓΓf-ΎΩ† Q.Ϊ'žxbΆˆΠG.ξ΄ΟE1%kxP&ŠLEϊŒ0ŏ }ΰ‘­ˆ=ώΩaL©—Ÿxl㈠ّP)ΓΠ»ΥD&~ΜΜ½Ϋo@βqΰ“mΓ ¬¬!6ΐ&ŠL^Ϋρ4Š^^qdn˜±"’tΎmζ‹ˆψE^pƒΚu1Ύ0`DTŽ!Η:ŠLEΪ*ΧGŽ_v€¨"υπEΫΥ^„Τz9QδEIφ²#Rqνεw+{A9ΔU™ˆΐΘ‹L,ΕC―q¨ωώr_‡§M4^τρ=β>™‡fξ[U ϋI_^ ψ –S"ΒίηX/Χ¦θ΅SMd*'ΚΠΨcύά—Ελλ‰vΰ―’Ω―zE&ξ)ΉvΣ§θΕUKd’ܟΓΎό!(!ΚΑ2z1ϊσ,γΪWΞ²"λk1ΚΧQDdbyΞΰ< /ˆύ<Ηe_FQ†σk2eψγΕSΌ· ]KΩαžB¬γyhΰΐ% Ό€·\Ώώςš2<ίXΐτy£Φz<Ϊ=#SΌ’¬,₯Kϊ»PMd’ˆ©ωί–—³ΌΘΔ}ßόgηCy d©?η,κΘTŽ€–.£Edbg7ψrd‘)qσ|z0ΎΈ(σρζ{€nΠ―θή‰Θ”νbλ+ύ’RΤΈψs#΅¬+κ?νΔ‘9³επ0"τ0rΜ j”‹""PΡ>#ς ΪΠ&ήdϊP͊,CκγA}ΐβ/70Y#ό’šΘϋZξXπvγ- ξάYγA–ΐ(2αA—/ƒΐ‡ψE¦"mρƐοTμ’?HωΠ‘l_4ύαrΌBόΘΪ„ωΈA‰ή’lχψ}B„ΖΊ"2εΕζ"}ςF3πβΊ„ψΙw”›βΌεE&Ύ‡άˆe Ρ †%D‘)ϋ ΎυdίΉAŒœχœ£ECh—λs5‘8Š²ΥDϋ"brgϋ\δΨG‘)ώ&°_xσΐ"ΎΥ,"J³]ήψ­¬φβφ΅^p]β¦5οaŒw}Œ[΅ΪΚχ-ΞηE¦"υΤzBέΥ^NyQBܘs­ΞZ|αΒύΏM0”‰€'€@kΉ.+nΐοΟλ―Ώ^υ»E™ό½^ΩλhΕΚ¬¨Υ'6A°*χ’“ίwďZVdΏjΥ‘_OΏγ‹ όΊjσό&—ϋΫp]‹ΡqY‘ΟJŒŠl[© }αw΅ύδήo§θΉS.ž;ͺEq° ηΏΫ•¬ΪzΞ ^ΐtΖpx@πλŠΑ€” Υϊί•ϊ΅­4ŠΐhKόMΒ4x< ³½©°{ΨΰCS2ΊC4SΜγ€’HΏhφE/œάΞΔ§Έ™Ζdφf;]ž­;]XeΒ„ O\Ξπ¨Y#Αž=XyC απ²λ™Ά~_dαYž$1ΏΎR7Όh΅$yωΊͺΝWΫηlσuΨ θ‰ζH”g­ΑΔ²’"φΐ’Ξν³ε*ς| 1jη!9Ν#(μ‘σVΟρ―ΆŸφ¦Ζ«ζXΨ›”΄™μ>€ 3μW­DƒΤgcB©'6UΑΩ~,β¬šεߍΈ°H[‹jI c]ϊμH€α€«%ώft’8rI~λψ=αc9Α„QΏ.ΨMŠέLy»)ΘoVχ|~€„"}Κ7b‚f`θ\FWaPϊΞ(Ÿζ1LΝχy’bš‡WΙ:ΞO˜‚½•M—σ³υtžώa–GΑΏN°ήΔΣόβŠσŒ$“5ΰΣQaθ#–―γj‘€>ςJό>η92Ό2`νsΡcΟ΅’aΊ£±Xμ› 9ΑDΎΈΪ?Qή•ΜnφΣΡy,€ C1φΙnάύ:‡ΌŽ…Ψχhœί\‡Lh‹ό“γL’S¬H[^°ΖΏzκ)wά©>2cXhΔ°‡`/'όž€dΒ žΠ•yμ-y:Ϊ”‰¨~άΜ£ΐΕfοjμͺV‹@―%ΐοKό},{σœ,·*]F™ψŒš”?ηλω¬Υ'κͺτύΆΠά²Ώeωφ‹μW~›ZσΥ8VΫ6{—/ΛoχŽυZ%FυΦ“-O_,€;»¨Γ4Ώ…ωQζς…<ͺ–Υ:οͺ­7ο―ZΥ—¬ηώ?{ζ’_²ΎήδΛλ­CεE ;Œ6‘ΙήΚ χπjDšόˆyΏtxπΞα ko‚ΉTs#MWσΐ¨cd¬’1l΅Ή<Λ‡Υύiž0~σΝƒΦ’‹.Z²=ύΗβCO q†i{ϋκσζκ7Θ>cΚ0&ς@ΗCΓΌ²ήB<„’e>‹τΩή ϋθ–cΖGΣcDAFqΰBŠPWNdͺηψΗ~•ϋDΔΨV’eχ!.Λ~2ϊ…»~Πμ-½―B@²°55…ΎS7£¬ΔγN!σ–ΘV,7”ο#ηlΓΨημ9\€-FE၌awwΨaγŸΊd]#PDδ+:ςHΉžpήd-ŠΰΩey‘ΌHŸ²ΫΗi SπrΞwΔ\ΞaFωAά5Ο½X,ύδζΆœeΕpΞέxώR–ώaˆn ³›·μΆωuεζ³uηΧΗπj’½½ χΝͺ‰ΙνsΡcΟ΅?ώδχω"’t~»Ψηj/(C»yΑ; 5Τζοl™"mεϋWnΎžzͺwκφr +wώΖ%^ΐώεΏ_άΘs\Ή2+ί^Υ3ΪT¬[Ÿ" νO€αξγ½^ϋοφ »XϊN =žς#[wW?Ԏt7Κ―G›ά†ω4wtΏq³pŒ€Χb†Εw»P‚8ΓΝ%Γ>V2ήΌS†zΈ‰Dψ@d±°χ~ΙnGn-\Δ‡π΄ά.bδo,³ΫԚΆ….-»μ²^7Γa2T=⏹ω»π/&άΐsSΛώ!Œ!’Xh„Ώ)enμ-žΨλΑk šRΞ~π@h!Vώ¦ΦFvro©#F€’oΔf5Jh Λ½βŒ-!Ά{5 ’±ί˜…žωz<»Μ…9XXH‰F™Z}f-tΔ…&ΌAx D|΄δ}ξ-Dy«ηψη·ΝΞ#Μΐœ‡4KLιΓ›βύΏlΩμ4ϋΝ1έrΛ-]Hbί9lτ$—(ː±€<Ψγ)`Ήl‚ε"ΙV,7™—αAŸ}†#βP֊΄Cf~ΧJ’G’^Ϊ"±l%+’ΎΤJ4Gύ$Ž Θ1G‰‰Ώ‹τ™„Ψ$uŒΙΘΩΑ0Ϊφ€^i %7,wl³9™¨άD>?Ž$„‰‰I€]-'ΫΥJ4H’ΦRυš¨ι£L‘s*ζd’ #Z1κϋnΔ>κ εcbeΚΤj‹}¨–Ԑ:QσPφ?δŠαϋZΝΘGΓ1#R₯‘KŠŒΗϊβg‘cON&σŽ‰›ψ'ίYΪgδŒQuμ…„`BHb!Γ~P&{½πΒ™1Η_΅ΡyΈ–˜ΗpBŽ%Fή!ΙΈ‰Oή~ΌN3j }„Ήψ8Ψ†φ9ί°"meΊ–NZΨFΙHQEκ)ςΕq'ρ*#’§ƒDΔμCͺš€μ$j­wDχ|Ώ«6•ξœ&D@D@D@D@Rxb΄ŒρΖ¨D±jΙν²(΄aoα;ΣDΝmxγΑ₯š±Y‰²Z%‘ α&Ά¦l’QF€£-D¨J–½†Η2y‘ΈˆhΟχ­š˜\€Ο±ρ³Θ±/"2Q_Q:Ά?9―j½x l­—”AŒbT%oα3Z%Η‡λ?V΄-/œω—)ͺH=ED¦Z/'ŠΌ(©4"O½#)evW“" " " "Πk ŒΑžΫ d6’Ϊ e£…υθύΤΞ΅'σjρP yΜΓ;μA8˜θΜ“ΐC‹:“”±=I΄G― Ή%L7&ŒΟχšΒΟLΌΙ―JηɍC’br‘E#!N„6Υk΅ϊT>ϊ@X*!EyγgΔϊωs>’OΗDŸό&Uη ―#lkFKςj"UΥ²]] {BWIxΓσuς#D±άΎΗ²ιs‘cλ―υI¨0nφυ„tsάl4'?χ²ηVΆ­X†²<BσLά,9/Ήδ’`žf}ΛŒλα<―ΤVΆ]¦m΄7 οj=ωz™71ΙΟ±,qlΉsΜD­`/eό˜ΧΓ”ΊΝ{Οχ‘Π9™ˆ€ˆ€ˆ€ˆ@u™ͺσΡZh:`μMz°Π#OΟƒ‰πI0Λ(Œ2θδ„Bθ$Ι5b’ νΧ Δ. νŽ.¨ 6'ΠάWΚ-ηφΫo/;:Y‹tOέθε –νx_žδ€$A—ΐΤΛO νΎt3²[¨­{ͺ1 ^Jxά]sΝ5έά5'"Π.Π„p,ΌΆΠ ­Έ_ ¦cΉ`}PŒl,_žΖ‚g)λ-l8»ΊαΣx ΣΞ+―ΌπΊQ‘ετϋΤFΤ₯:D@z6^αΙΤ³‘φND@D@Gΰ…^π‘=πΉ|X]γZRM" νL1„Q-έ‘‘’-Η^:\{;ν£1Š€εΌ,ι?£(3ϊπB -δϋˆGg3_’6aήyη –1,³Μ2-‡7οΎϋξΐHΣ2¨F`¬j+΅ND@D@D wΐ‹‰?™ˆ€T#€Ψ@>΄‹/ΎΈΗ]3l4ΦπΠC…K/½ΤC‡«qΠ:R™JyhND@D@D@D@D  ϋοΏ?Ψh˜^‚ΠΙϊρˆZ|ρΕΣt ώaΉε–K—]vΩež»Ο"Œ>0`Α{ο½ηK¬±ΖώΙΊrνl΄ΡFώ za#ΖΊΗf%Ο$ωδ_gƒ˜Έψ‚ΰτΓ?ψ2Kšνε6ςω(2Y§δΫoΏM>ύτΣδ’‹.JlΔ<ŒΌŒ%άN&žxb•>όπC_vα…ϊ²οΏ>‘DŸΏύνoΎŽζΝδmο³Ο>Ύ¬\;¬Xgu\(‰βΠ—_~™Μ=χά^_VdšqΖ8ΘλŠ"Σ‚ .˜Π>φκ«―&0ΫoΏ½Οι"ΒΨώϋοŸς1ο$oϋΎϋξ+™ΜΜ…°(ήΡHwqϊξ»ο”&ϊF«C=4εCrρ'žx"VQ±Χ]wέ°Ϋn»…Γ?<.έΕ‰δοΏ~01,˜w˜·Ož>ϊ$"$2‘€2" " " " "  Ξ Ύ˜WRIΔ!σ οΎϋnΊ|¦™fJ§-Λσ1! D‘―ˆόNδ"7’yEΦ™’oC'D „(ςY¨\8p ηϊωηŸΣz™ΘΆƒ0υυΧ_‡™gžΉ€Μl³ΝV2OΎ&ğ((Ε•Φ'ύ“ΊΙWD’π"}"WBβZΦΘ15逓¦‹Ž:κ(w~ψαa»8YΨ^Ίi§l"Ο)»NΣ" "% ‘)KCΣ" " " " " u 6"N9³άBΑΒΛUyΗςy2ξ‘#G a K-΅TXqΕ]p²P2©V[m5ߞQί–\rΙ`‘xa§vςδίvηIΊΣ~ΘΆcω‘‚εuκΠΗό±d3™η% °Y{ρΕ]ˆBδ*'Ά·Αl>ˆ–΅?ωΟαϊλ―x>rΚ)ΩU‘;8M>ωδή¦…Κ•΄ηT²R3" "! ‘)C“" " " " " υ°άFΑrΉψ“έšQΡ=./dΛ žΌώϊλαάsΟ ‹.Ί¨Ϊ†ΘΔHlgu–2½}eo Ό›}›gžyά›ˆνσžLΩ6Ψ~ΦYgυQι²Λ}τΡt«Yˆ^)/]a–w);£ΨgΌ˜Šτ –ΏΚŊ,‘‡ξxβ‰q‘ hˆl–|<tΠAαε—_NΧu§…^ΨΫc€Ό¬e9e—kZD@ς$2ε‰h^D@D@D@D@D .Ϋm·]ΐ †\IˆCx5~ϊιαΚ+― »μ²KΥΊ l΄Ά` Ύέƒ‰Β6zΧGή£*ΗςΕ[,X’π0|ψpΟq„G‘ψζSήϋ†ςY#ΟΠωηŸ,‘v°δίξ)”x𚲀ׁώδΝFy ¬gΏN=υΤ@»‡rˆ+§ 7άΠCυ,‘v°Qς\Œ³DεΎύ!CςΝ…c=ΦχŸ°ΐ˜Οͺ;8‘―Κ΅‡=χάӏǛoΎvί}χ€X˜΅«―ΎΪCρ<“‰€ˆ@–€D¦, M‹€ˆ€ˆ€ˆ€ˆ€ΤM€$Ϊ$Φ5j”‹4̟tIαŒ3ΞΫn»mΝϊπ!1φ +¬ΰeρZnΉε<čuΡk²_HLηyŸHΰύΨcyρX6ΉΝ6Ϋ„‘C‡†sΞ9ΗΓλ† ζ c9ς1• •c=^EkΉf`Ώ˜ΖΓ*Š_EϊD融η]$ρξίΏΈζšk\τ’ΞΌ±μμ³ΟφP<8FλN6Ÿ³G΄‘φΒ<vήyηΨ΄ό‚xπΖ’‰€ˆ@–ΐ ?—] iθ,F{Γ«ˆΡεšeδ2"™ψ 3ΜΰBT=νπψσΦ[o…ι§ŸΎd[Βύ«›rΚ)ΛVG8ž=΄Ω§OŸeŠφ >δ‘ͺTO‡Š»° hŸΚ5Aψ !~Ν<ŽεΪΥ2φ& ‘©½Ÿz/" " " " " " " -A@αr-qΤ ho™Ϊϋψ©χ" " " " " " " "Π$2΅ΔaP'D@D@D@D@D@D@D@D ½ Hdjογ§ή‹€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@KΘΤ‡Aφ& ‘©½Ÿz/" " " " " " " -A@"SKuBD@D@D@D@D@D@D@Ϊ›€D¦φ>~κ½ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€΄‰L-qΤ ho™Ϊϋψ©χ" " " " " " " "Π$2΅ΔaP'D@D@D@D@D@D@D@D ½ Hdjογ§ή‹€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@KΘΤ‡Aφ& ‘©½Ÿz/" " " " " " " -A@"SKuBD@D@D@D@D@D@D@Ϊ›€D¦φ>~κ½ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€΄‰L-qΤ ho™Ϊϋψ©χ" " " " " " " "Π$2΅ΔaP'D@D@D@D@D@D@D@D ½ Hdjογ§ή‹€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ€ˆ@Kψ?Υ #oΘΕ½bIENDB`‚pytorch/docs/source/_figures/remote_3.png000066400000000000000000001753421444262304300210420ustar00rootroot00000000000000‰PNG  IHDR ‘ζT€1sRGBΞιPeXIfMM*‡i&    ‘aiD‡YiTXtXML:com.adobe.xmp 1 ^α@IDATxμ|ΥΥ‡·’¨(v  E%f·*ΆbwwcΜΏέvƒέŠέέΙ[ATD%Άσά}~½ήΕ…»1Άσ~½ž{ς{βsςήέmEEQΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡΡMΩΕMΉs±oΡΡMΧmΪ΄ω™ήΝέt{{-Π`=~όψy¬ΆXQ΄@΄@΄@C[€KCeCΧλ‹hŠΘg-•4EΔ>E D D D D D ή-k*2~όW“ubZ΄@^ˆεe˜9Z Z 1Z ΖK ž›ŸsΔο=4Ζ‘‹mš‘,ΟΗ3TΗbc£’š•β'šΥpΗΞF D D D D L½β₯aκmŸŒˆˆˆˆhVˆ—†f5ά±³ΡΡΡΡΡSoxi˜zΫΕ'£’’’’š•β₯‘Y wμl΄@΄@΄@΄@΄ΐΤ[ ^¦ήvρΙhhhhhfexihVΓ;-----0υˆ—†©·]|2Z Z`Ζ³@7šάΦƒvυΤό9(· ͺcω­Θ·2l“Έ-κψ\CdλA%[ΐJPθΏΩ³eΞΣ[«Σ€Φ0'¬0³ZR}Tέ™B­‚s– MN³ΔΘhΌ-ΧRή&«υ/Λ«ήζKβ?Ζ½œƒΎγtr :?~…—ΑιΥΰ π]άwΰ;q ε<Ά>XŸq _ΉώlΥΔ›χ πγqλά ΊΓŠ0LΦ/‡ΑGΰϊ}<ΨΥΪ0l›m>fƒλΑr>ƒC 6i―ΐ›€ε[LS ΐέπXΧ­ΠΤ;p(Ψζ‘Pέ§9~p”Γϋμ‡73.ƍλ™²Υ›ˆαEψ΄έΟέƒ?Œν©ψΟ…7@ζWΏοτ‚šBۜ‰lΗmϊ}ώΆpŒνf΄υ•ƒ—­AΰΨ©žπ|W€γ˜λ“ηΡ  mσ1 ΌΨΨ†ΐgνΣΚ fΗΫ²}φ8 jTk©ΖrŠς(hJͺ„ηΰ€[γŸZΞƒ–Ω}j ¨γsΗ'υœ»38Φ{¨CΑπ²PwP—ΨIͺt_‹ HNΈ*kZŠΝÝ`Xq z€g]°ωΚM―cΎε“?΅”O±Ν:o6=CΉ†7M ζ&_–’ΏϊΜηFξ3Α °-xqpVφπ$μώˆΑη…piπΐtΌυœ’nψ\‡¦O7ϊ½ΑΝ²žΣ.…Ξp`Ύχ`ψΎƒ9a0―σωlX‡Ώΐ23½?Τ$/0Φ”΄[α¬$ΟμΈ^΄΄‘ϊ Ξϋ -ώ†Y [㈸"‰sRφΧ²…P—‚ύšΰΒΝ₯±Dξ .Θ  9I‡ΐp4X_ΆΜs08n7ΓL`ιηZVŽ—c­}ξƒm ΘΝ’t†ΣB$yΫ!Ψ͍θ@°Ν‚ ΒΈa ‡2PύΑ…k}¦Ν…η\xμΠκM[”υΦΒ―ΰνΑzOmv>Έ‰¨{‘%Žγ<Υt-πA΅χ}\׎s;[#SΞYΓC ά΄έ[“\ίjφ*'σ\ͺΈΚωΧ끄6ηω:°0”ΓŽK"ίΑδ$qTβΝΈ:‘3γχΐμ'%Ό‰λσ5ιc—Κ‘a^βŽχΰ½ΐύςDθ ΏBΊΪ*ΘΆ¦ΣBΌΟιΧ~ν`θ“°ξ©Π²υK*ΒηΪBxΞρv(ε~τžτX…ψΰzfŒ ܟ vν3–σδκ›mι‘-γΏlŸύ ςŒHχ!Δ[¦ηΖ°Ψ‡Iκϊe„xν–.?έ琿ώάKΒΩNw‚ΖΆ!΄χ¨$nY\υ,8ΘN*uψΜΦ°AβΏ7θ[4k»-8_FΒλ Σ\/κD0lωΞcύ¬j0Φ§qAΞsΧHο$Βuπ#”'aiŸν‘„ογa±$|)ιξŸ›$~χ’ r<οƒν·§ΐͺ zAzŸΘDς’½ΎmCDβΊ~_Mό‘>‰_ΗυΆOώ Χ΅”ρξKμͺ>„5Αυό -„ΗΊΝ“Φz΄_Παx \Ϋžs¬ΟK₯€Κ$lϋ2x ΤΣ°}ΖWυr1ΞΖ0 6¬ŠΚΌšο°$ό0ξ‰ίvν™ψu|v, ξ7³ƒšώλOkyξΟ³$‘sγN‚Ε‘Ό AιπSDξp‡‚{_Κc-εόΘΉΖΒkIόιΗdεY+ _„+AnNζΣampΒ΄μ‰ATNεΚχ^’Σƒ`ώTP§ΰIΉ«αŸ[e:io'ΥŐ-λgΰ€ͺ°‰Xv˜p‘­ˆ[0‘ΆvmŠFβ[tJ菋Ωφ7€+ΐz΅γ|P A.κ ϋgίrΙΊ‚\n¦Nΰξp=½ˆΗ: ‰ϋm*μsέ Χsfϋ—D_γΊT'—ν z!ρο°|σBΊί3jΟλNΠΏ*˜yuά|ώ;˜œ‰©Z°ι>$ΡΡiBpΞΊΎ½§#»θœΈ_O½MάΞpΜ a£Η;Eηγ;–ΞΗ`¨«Ž&γ£π Œχ@ηζa \'Κrχλzށ1ΰš»ž/ *½Χœ@xΈ'ΉΉ.¬Oέ ΙήR^»`}ψάϊΐ: ά―wχQχ9ΐ|uΈ_Xλp8Ό ϋΐεπΨοη‘ju$ξ Wΐ«°ΌαΉέπ{qœ –_Υι,χχ=m;μ·Ά™ Φ†Ε τΫ=ΧΆ:f§ΐM`Ϊ8Άƒoΰx q·Ηυ2­QIΔΈϊwΫκΒ›SΦ{3Μΐ1{ ¦–+©ϊ‚ά •Ώ4²ΝI.8α:Γηπ„Ζ›Q8‚a]ΈΏΓBUΙzύ; ι:€*k@ Ψή„l]FDGX Πgαp€Ϊ1‚Θe ΄£g’Ι…Ϊ1‘ˆŸΐi5¨©$OΡ“ψ\H+Γ+Sb«>Ν8p)8!Ο„k@Ή†6f"κψ²ω΄“}{v5 l ?ΘRe*ό=~© Ae?ΧΎ*:σj]#SαlοD˜?ŒΩ@όεp1\ Ίύ~Ÿ?ξ΅$΄ΗΓΉγܟ”ΈΞΗ¨¦mΧιήβΰ<^&ΓK0<ργd~Τυ΅žDβ~]ΐυ0Ίσ:hM<›Γ8pΝ:§>…6kχΓ―@Άž'Βς·§?Α΅μ^¨n™Α9κΑ9˜πy \³Κ=u x`݈Ηυ½ΨΆaπ1¨Cΐvη’{PWθsΑS°7Έ―ͺΝ`g°-WΑYφΜ}π… μpˆί γbί‡Αΰ~bΛΓvΰώ4BρNΡϋψNŸͺχcχŸσΥ–AηαqoŸ zΑW lŸc;Nυ8WδWXlΧπ°άη@GPG‚u[ΞcΰάΨf†΅ Œιώψ-GΫνΑϊΣϊ‘ΐͺΠ†]a1ψ μΓ δœααψ7€~π.¬α<Β[ΟΚγ# κ`Ÿ™£I­ˆΣXεŒΔo§ηάηΞo‡^>ε@šφ(τN…χ4άΆΧΓΕKˆ‡ΓŸpψΜ,ΰs.ΚΆό–­N›3ΎΏΨžΧΐzf…―αl8άΎ„ωΐταπxβ.ˆλ ππ€³-Ά!΄ΓrΤ³ΰai;ŠΑvάͺ=XΏy²υοΒΖ°΄CΑ°zΆ‚­Α>_Φν‘Ί(xΘΫ&•Š­ŠJΰ8°οαgπΒΰσ^~,ί±8ώ†΄¬―hkεsΆo[Ψ>ƒΑη6ΛYΚΰT0―²}δ^Ζf†§ΐώO€ύΐΓύ0xμw_°m?ΑΑμπ |«ΓXp¬GΓή° «@ ƒ^°<οΓpυ’<ΦR½Τί Ξ6υλΎwLjŠφmb}Z‚ώΈOzvDeY `k)ŸΫGVb0Z Ϋqtvds Η΅Tψ‘Ž6-ΌMc‰ΝΣω¬₯’ζi’Ψλhhhhhh|-Π2ίbώh©΄ΐSω\|,Z Z Z Z ‘X ~ΠH"6#Z Z Ρ[ΐο1,ž΄ς(ά-uρIr½9Rς΅”ξχ Ά­%OuΙΛ‘ΎT]Ώλδχ‹Τ0Ζχο—•ϊΕγ|δw›ΦΞηzΘΫƒ2―IΚυ»Q»NCλσlx“~+ώ0¦‘ΘισhΌ4L»ΗZ£’f< \G“;&Ν~χ£Δ_]|’\oNoJφ‹Θ5ι%ύ²υΤθ;ΊΏ–Σεϋ%dϋl΅#’gvd-ανIί½–<υ<'ψε}εΜίΚψςqŒόw‹δΡ»pΗ$ώΞ 7ŸΖpou~Œ]_ίb_”²GΒR0NU]|UjύΌ€Ψoα±ŠοLΪΞp| yjJ:ŸΔσΰ›j2yΫέόν‰ΰWΘ΅Iψ%ΖΣᨫړΡώUΤυ˜/Z ž,°εzΨΟώ&Ξ»p­‡gΥ$ΰoj=“ψ=Έ>‡ ΐƒΧηΝϋ-'WΌιξ―Ύ ο…αOXXλύ†ΐDPΎΓί&ΐπ%¨™` ,χAΎςyΫδΊΌά­ΛΡv(Ϋ+ώVQmέ ¬Ϋύδ&°/Ω²_αW‘γχ7™²ΥˆΝax€¦νI0σιΙjΈŽΩ:πΜ –Ϋ^ƒ!—:ΉΜ^zn€ρ°Ψή2˜ξ…p 0Ώ}έώ‚λa4Τ€-It.Ή'šίηZƒ6Άέ†ο†OaCPΆ_»₯5νκεmΚye{WEαUx¦»J¦{ ώݍ§αλCΫSθi9 .>GΦ‚F­JiK΄Δi+lew1η6DΊXλͺVdόZΤυ˜/Z -°,e{Hx8za8Žux!.‡±ΰaά ΤUp.Μ -!­έ¬›ŽHόιxν`¬ Ok£ά a$μaoΪPπ’αΑσ,ψ&Gyΐz•Γ…ΰ%’6νLΛT—€—“ΡΠ ,Ϋ5Ϊ‚ cπh3λκRpΟϊμχ  ά³7Ξψͺ^.ΐα9Θnγ:Δ=?Β8Π&Υν;$e4――ΓB`Ωgΐ©-/‡―@1| ξνƒς0·Y‘ΌtΨuά ŸΒόπdb²5qύAi{νυ 8Wgεψhλφ2π2΄lKΔ’Πμ——Œ―ΰx0My‘yLϋn„0Žx©ςω5 ΊΰδXœ₯!hI<σ€cK˜Z†@_˜&€†ΜΦκDhόΝa₯¬ΔV„Wƒν k*Νz€ ’ψσqσΈ x“sεŠ':#ΫdN„Φ™˜ͺλ3lόz`Z‘€}]0‰n°―KΐUp0Τ$λ·,΅88‘΅—eΫ7U –—–φšzA˜΄ŽΑ¦°&8Vͺ-„ς]\n¦=Α>ΟA?βιΈΪn+X8—φB Φ†`'6°4;εΉ–š}¦¦ΓyΨΤy 'υΈ?ύ„Ms―P¦»Gμaύ«f|U//ଛ„}wyPβΟΏ,iίCXk–ύτχ$/A秓ΐ«ΈBkςJθ?A+P Αdp]Χ€+H<2Ι0Χ½A•ΐx°νρ573xH-kΒ ^5χL ϋΖeψ5φ²/ξ%λΓK žίqŽgp€άέπߚ„-ζTš—ƒΏΐύ+­ΕΈΏΉGy‰PΪ1h<·'‘Έλ…\ΗΒ1θoƒ:N…9@»y)P-Α1rœχ…Φ άχmΗ¬BΪόΞηΟ=δώnΏζ…Αr>) νζ±–2“€υΫΙχa?θOÁ Ž‚‡ΰ Ψ>…RPΖέ˜·@'m.=@€ι½αB“©ώwAγ― Α œŒnƒΐ6¬Nͺ΅ΑƒΡ δσΉβ‰Ξό,λ\'‘“ΑzΒ!9 Γ°œΆM΅€{αΕΏ¨½aUΈΦυ5άΩρ¦ 'ΒUΟ ΛW—‚‡΅‡ο/° ¬ σC/p“Έ l³‡ύipXžƒο&•―Fρ@8€oΓΏœ³Γr έ½ΕΫ§Ι΄6žWα°$b_άYΪqQυM"lο1°gΦι‹Γςΰ¦s-Έ@ν£α ΗCΫν³Α™Π ά(?ΫiYΪ/*Z !,Π.UΙΌψ†ία9Έ\KnΪϊ‹!(½ŽB\]ά±dš'+£αpΨΈ?δ―Dn/$‰sγΆםώ χΧVΎͺήλ)h;˜\ΧΩϊ™ˆ•`ΨξKό8’m—ΔΨίΡJ­:μέSžJβ½|Μ”•';˜mKσ·…_²2zžxΞΘ{° Assΐ>™φS0ν퐐εώFΨΌ%φΌ]ρ{Ixv„'ΰwp,kXΣsΔ1už†~U7V5•YοivΌN!/»†=(=.L½'x£ ‹θuό±:y8©‰ΰε`yxž₯»ΰTHΧλA9΅²όGS?ŒΕTΨƒO'’·ΚUΐΙ|Έχ‡…aA°ΌgAΉ Χ“§rΩΡ‰;vΗs{ i #ΠΎΫφ)„‚wŠ‚°­Ά9-ϋΧάTμί‘ΰ[ͺSWFˆ$ƒγη’Z! G'Z !,Π›Jϊ€—ΦAΰ~α:ττ€v_ZΦ‚κ1/K@φ…;WΌeΊ6v΅>Έ~άmKZŽ₯ϋΜ;0Έ¦ΗR;ΆνtΨη cπ΄η€σδ~Θ%Λ—ƒ“Δ qνͺ„ηαw8μŸυOmΥ ΄MϋςΦΰYθή}| ŸCMkΤ”‘>Σlh!4Bn‚Οΰ48Š!HƒiXε€H¨qιg §•6φΜ$όΫ‚“ΩA<Ž†΄&€yϊθtϋΌό™*£"ε^ϋωœœ0·όΆ%]^M}%kNε²£‡€“Ο…ϊ8i$ΠΆ‚QΰΒνΩ²AΩύ5ήϊ½ˆ…ώι.Ω‹—¨)ς’ξ· –νψEE 4”>€’#αSŠuοηΒK°+ κYpώΉΞ<,6…αkPΉβέ+?Ξε„rςl9±ζΚψώύ²:Αχ’(' JPvX{Ώ»† ΈnjΛΐap έ‰ΗΊW’ŽnPΆ΅Μs(νέΦ˜T°­ €d•0ΜŽ³y•}!l\“Wk©ΙΫ’PΜΓ¦nΔOͺήFZŽ‡έ΄m‘YG#Ž«¦!#‰_ͺš΄fΗZšr£žVΓ\OwCW˜<Έ=πΌ@T'o©»Α#ΰ-j2ό ΥigvEαx(‡!p,sΐ Π<΄²ενφ"π2ΰT]όAdAxNƒšd[Ό~ήπm‹}΄oGΒ`}–ηAt>ž1ΰA>΅r .ϋlέD„›¦₯J˜Ϊ.{‘όEάΫΰ%ΐ19z”€ΟyΡψƒ½`"€e_rθ;€Ιr(…ΛαˆŠˆ˜v S„{ΥχΣ^T,!Z`,ΟνƒjZCiββΤY jΙ=–τ…|χšΦLJaζtd5~λ’lUo>λ/ϋZΒmI/ί±gΛKΟ¬Y‘₯„Oʊ«ΰ"Ί`-{ωͺ-Ÿ””Bώ=E~ͺΤO)ψl³Sžk©ΩΩgj:œ‡Mύ΄kώ©©#>Σd,0=qoΛ₯φDfZš+_“Λc-Υlƒ‚Ts5uI —†ΊδσlI£γG‹3βΘձ͍h-ձō?[΄iγ£ΨΒΓω¬%ίύΝΪ—Fώ2#4t*ΫxΧT>‹ˆˆˆˆh0 Μ(—---05πcιΎYώMψ9ψ&+ΎPΑΥ(Θ7:~Ί!՝ΚϊΑ;πl £ΐΆψ=£ —΄ΡϊpGΔ:Ζyžψ)ΏΣ4½UˆώΤW–¦ΰvΰwπ΄Ω^ΰr‡b0ΎΡͺΆŸI7Ϊ†Η†E D D ΤΡ~/ιZ(Mθ‚»|«@}h3 νUΧRζ£€ϋσ{ΏΤμ›-ϋ:φšΎ;ζ‘u8L‹^ζαy§₯€>»e]Pΐς Y”σΒω‘φ?IŸΛBˆΗ;*ŸŸsΈ{~q)όŒί_οk[ ςύb}|Ί‚rK³°ϊ”ÜυYA5eϋEΤ΅‘;ψεΧ2X/Η­:i£ϊΆI¨»3Ϋh»²/ΖΪ­4nšαK΅Ή}nͺ5ΧT·Ή±?˜‡M}gχwŽώψ[<7fΕ{ΑΘuπωήi™Ώξ;~9¨žRΘώR΄ιξαך ηRησ~ψRψ·ψ—Θυ0qξ©₯½>ˆš"Ώξα›KΆ/;νwβΒΎύLho/B/HζRpΘ–λ2ύ%VΗ —ΝΜ“ώΒϊR„νZ¦;Ύ΅Ι}΅r΅Η2ν!˜‘m*χl™ί>ζDžž+8Η¦ uώQTnε±–rb VP(°ξξ¦d}&Ι~5ξauτ_9]<·₯bΖΰο˜ Κλfΰb΅ΝO'|‡;:C}Θ[τ‰υQp-e>Kϊ‹ΰνψfπcΠΰyΨͺ“‹u\u‰uŒ?Ÿ|jΘkšcπ%ά οΓΗ°έƒη#γά=‰[%5ώQ©πΆψ§ZΣq-Mu›ϋƒyΨ΄ΊKΓυτρͺ€Ÿnκ―&8?n:΅3”Γ£πό ~ώΰI2lu-’ρUύ…ΐGπkΒ0œ{Ξ1ηžrώZχ³π άŐ­υ‰H?Ώw’ΑΊ+a8Xφίπ2¬Σή ŽŸw­|=  Έg©6p?Όo€ν —ζ·ρί–χ<%pL‚ΐƒ=­΅ Έ&m“ΟΚίZxlγ+π=tυ:ΈΏ8&?Αΰ1°|νί ”ν»žƒOα PιKΓΜ„‡Β`Ή/—‘\:ƒΘa886«‚²ΪΔ}Ξz‚ ½ρ€ΗΓρQ³ΐ½ΰ˜ΪΟΗΑΛ…σΒω±-hr8B<ήΜΏψ Χώj»£ ή”ΗZͺΉ +¨ζjr₯¦/ Ήλη;†ρuΝ< ωαYkλT.€[α†T\SπjΟ“ŽΈΈVnΐN}K]«©ΟŸa–ΓI νƒ\ά†AY*\χ!\ΔΡt\Kic,$›.Mϋ'Γυ)žΒ?u'œœρU͟aψΛ_ ¨~ΰZΟηΰšΩZ‚σt4τ5όΑΓόP­ΰjXΐ@J:>ΏBηησ‹ΑL`ΫtΥwϊχώήΰ‘μ:ς9΅x€uσ«ΣΑ½+Θν$ΰ₯αΜΔΪ# Ž›έ^“ξλQ Γu_Υύ”ψuμοIΨΓέzΥκ`ΏV1€Ό@;½B#‘—ƒwaX μ§:ξ‚bΘώ]“ρύϋΕ}γ/pΜΥ°gΖWuiΈ8ρ[φά¬η'ύ^Ώγaž£a(XΊ v€C@›+myjΖχοxϋ΅co{Ό¨Μ„ ξδ±–¦tfZα`ΒWπ<„£7$Σ^ƒA#Ί™΄“qCΘΦαD H"»ΰ>Π:Βd-KΒΦν­n?PAkx\„χAܝρΏ#αvpρ*T^†Qp8ςUhvΙƒ-qΟoΪαX²φΗφΨζ›Α ²<\A†ΓbΩΏ}PNΜA`Χ‚p?|Ϊέ8Υ†€υΫ6mŸK‹©­Βσλ$™ξΑ՞w‚65ŸνΩ .†U@•ΑKΰ89Ζ Α<π0ŠΗ±ΐL μΏγχΨΈ: ζ†aIΘΦΑDΨ―“AΫ9ίnIΆΓ RmZ‘ ―Φ–)¦Ο0pN ‡g =Μ]Α9œγΞ ηΩ ΠΦ„•αψΤ0—ρεχςΩ'AO°μMΑΊφΛλ ΟΑ.`^«α{HkξeƒΟο ΏƒΟΧE}Θτό”dφπΪ-ρg]HΛηΫκAvh@ΔΊŽ…°38ΰNͺαopΒO„ΥΑό†Oƒ-‘| w€²lΫ²8π.rσΥE;’Ιϊεd8u, φՍH{μNΤa`›έΔF€φj ^8–‡ ΓΆG-2ΎͺΠ΄γFπά .8σμ ·€ d[pBڎ>`;BZΕlΟSΰσ{ΐMΠϊƒvtΌΧ„―Αςοƒε`.˜—£A[Ύ Ϊΐώ8ήj{ΠV½ΑΆ,'ƒ²œMΐ±*ƒ}Α8Σ]4>χ dΛΎ]Ixܜįˆ;ƒ“RψlZ Μ Ϊ2ͺiX ‚n Α°!όWB‡•k―<Α9|Έ>'CP%žΏC·8εo•ςg{Η'Ϋς§ΰ^ΧΜύΰώπ)„υŽ7£\ϟJJ8P“lΥ:>ο²ξ­i™Η΅]žΰ^ΰz J?―=6yξUάσήƒ½αcp―Ψ†Γ’π:XOZŽQPv!^7=Ž³}JΛώ|ε Οΰ3Ai ϋαή³'¨!ΰή”]v³μ±Pžβό£ΐ΄t»kšd"ŸΣ“¦Δόσ£²TΤτρfO–©m…‡Μ*ΰDXΪΑt9žα7ΈϊB^9i\8λB.u%Ί$ρά2ψ'd/p‘9ψ³AuκOΒυΰδu`Ž‡UaP–ν ;^„μ‹ Q9Υ›Ψ2ύ` xԢ𠬂σ—ύφ.P‚υζ+7Έ`!X΄G8_€εwΠN„ΕA[€΄ΊXށ2˜>‡ .Z‡LΆφU»f|Όh‹η`X ΄Εt ž_ΑΆ}΅ΩΏ„<ΐώ« M‰VΗ<[ΞΛ΄£χa|:2ϊ›Œ\σΑOΘ\ƒκ5ψ<(Δ9·ΌΞΣ0έ‡ζεήβ ―ΰK-z›tΧ”σίznMΑƒΒ79χKΧΫΣ°2€εΊtM»fΒσ›ΰχS½I&χΊp€m€ί}7-mΡ‚-&ΰ_>‘ΏΆk1[?ϋ|&X·ω΄νNpt‚ϋŸkΨzσUYς€u― /&α༎Ηώ†ώ8Ξ+ϋΛ.)fΕ?<ƒηDυI<ιz,Ϋ=χn°ό‡`π‚‘­Γ3x3o OΦS‹~&ύπόP΍/Αy8έΥ²@-π ΉƁξ/Hλ§TΐΓΐI”Φ–„μMάΌsΑ/zRϊΏ“μJXΐΐIY •KNΰΟS .ώρ0[g?‚&㱬Ίh29YμίKΰδqQ(Ϋο$π@zyΗ†\Ϋ>:N{¬ΥiL’ΰ†6 \AŸβΡVNθm`;p’ΫηυΑKEPη?#Ρηλ’vdJ“7mΑΒ©‡΅…Άž=χXʟmTRN―v‘0ήn¬aΜܝ“ΚKΓpSIζ ΟΤ”/¦ΝΈphOΐ‘ΰαΨ{ΐxX― ίΑiΰAτ$,ažη~άCάWœο5ι{O„gΐϊΌh| ο{Ω#Πlƒ{ސΦŽŸΏlm|κrΰ>M>3Ϋξ~雉] ­S<–#l ύ 6iΣΑύε£TfχΉ;Α3b1(λ~AgΠή­aAΘW‡ρΐr°hŸ{aIrόμ·mϋ6­ [ξ[ΐsπ*¬BΠ!xz€νuΏ ΰΎjώα°6Έ·ZΦΉπlφ\θ}`'¨M‘αv°ŽžΰX|[y|9ΒCϋToΗοSΧΑ©_Υ‹ώ‹’πoΈΛ&ώb\’Q7…Ίœ‹‚ωΫ€šœψ€ Ρ §‚JXζsέN„ΑΤ “Ε2/λ Κ‡ψ΄»λ m0ΝΕ<z@.œτβ\πVΠ~…YÁύ(ƒ!ho<φY]φCΉPμƒςpv‚Ξk сΈέ`#°^₯½ˆGHΙEλσσ§βΐο„WΆmκγΣεσΈ–Ώ>Ό—Δι,χƒνκ83γ«zΡVφMΉ`-#(ώ†Hl.½@δY ΪΤφvϋλψ―΅ιY2μQ[¦|σXKωΫ¬σζaSχŒ°Σ6kI  Β\Ÿ ΏsxH―e‚™yμΎδΕ fε3›AO0.¬“%ρ»ΆΥΠ*γϋη₯―Ο9χӚΐζ`܏ͺS) ΩΟ;ΗΛ@W­ ‘ η½{KλΦώ,Dh#σΩ5aC°ΟA–Σ6pΣαv„Λ@7[‹α~· ΄H%Ϊϋαώ`Ή¦+νΪk[V72‘y΅―zΦχ΅• Θώτ ά™ΐ³Ερ ͺ“Ά³œΰήτ*žuV‘)wόφΓ½&-ΫΎ6ψ¬mPΞ‹ΠώNψ;‰ρ†έ‹£κSy¬₯š›‘GA‡P»ΰ NGπ:¨λ` Ηΐ·ΰRΏΑ›°ά Oƒ¦‘žu5–ρU}ηα1ό;ΓPΈ:€εxθ8)½¬ό Šζx6Έxl‡δ`| ηΐπœκrυe‡νÁFf)Χ₯Α,ηΒ'`;6†οα`°ŒΐA]Oƒύz&—'·}9mυ"¨κ. ¦YήΛ° œ_‚ ίϊμχ> ½FΑʐ­Λˆxv…σ`΄υ̝ρεΎ4΄ Νg‡ΐΐΔ8 }¨ξπ#œ{Β‰‹Sγ₯Αq ₯­‰ —ΐξ`»΅·φ*ΞP §ΓIYt%dϋmgAjk)΄#Ί΅X`:Ϊ΄‚¦Ν^KσbrΓXΰ ͺιέUyi(k€z¦Kω¬₯–jαE”σ3,εΰmqs(εα<f†^πy ¬Ο‚ϋΫp&¨λΑƒS „m`qΈn𚰠Έw.`}^<¬=[Α~πx(τ„ΑΓωxΤ ΰ%$(~Θ‰!!εŽΑ?²Σ<œ<θ€a}°Ε°.Ό j_°_ΑgΩr΅WP›ΊμŸ:<„ƒΌτƒnπ¬cΑ>n kΑdΨ΄uΆ΄S_°QΰσΏ€Ϊ΄Ÿ:,_φΥr-_Ϋv€αQh{²ίŽ‰—ΌΑ1{Τπ~ΖWυ’oO”ύΚ₯׈\Ά†…α[Ψήυ' Τ“CΞέ η¨ύ DD7Z ΛCg―υ¬,1Ψ@ΈŸz|sPίz€ ’žϊξGύ–ŸΟν£†–\GΪώΥ€{8{hDύΫ^’ΌΜD5 h-5k¦Ρ¦…±c,%Z Ÿ΅Tζϊ„:Ύ©¦žηˆ»š΄ζύ|΄KsώΨχhhhΠωά>fΐξΕ&G 4˜βZ*Ό©£M oΣ¬!μ|£šΈςYK ρIC7wμ^΄@΄@΄@΅(%e`’ΊξΛ‰z9ƒ©ΨοPΥ¦;Ιπψ€` (”ΖP’lω%³#σ―Mή5’όηβΪφBΚοIω½³B«œ ϊΕλB70]^Ό4€­ύΡΡΡ…΅€_|ξι—uλr`ΆSWš_„φ‹ΤG€_>]Ρ@ΏUhl—†ΗιλlΥτw'βŸ†Y³ύ {ΰψ$ήί― Ώy‘Deœ‘Ό.Dά†Ϋ3ρgΛΟE;βύF½“2h~<η€Ώ 0# ¨5ž§ Wyή²U“f~λk κE#ΒΗ“Wα_14*Ά!Z`*,ΰΑ}ψDώ֏ϋ†Ϊό-Λ5Ό%́Η5Υ€+ΰh^άcΜkΌ_d6ΰ»_σ{ m Κ½Χ Δ#p„xΌ™Ώy³ίΨ7­(λ9 |f¬΅Ιzϋ}!€χWΧΆ{₯mΎ:‚Ί΄}wΟάΪBp΄<χΆk!ύEφνkCΫ½9Ν‡ηxόν«šδ§–αxhguΤ Ο1!Έα. {ÚIάΌΈŽ…ύΣnαΌkƒδ$ώFάκ>E™›4mf_«΄|ζ°μS`˜œ'–΄žΠφώψ΅ύΣ¦ΉΤΘ›ΐrOƒ0^ύπ;—ƒGΑ~yΦΆΣωΈ ¨φ`ΌΏΩζ|™κWωόœ£@-™@9νͺ)Λ7τΜJw1Ž‡#“ψΑΈ)-'φ$p‘hΈΙΰ„Jk9eΰβ―© zƒJOψ„Ώ†ΛΑ…δδr1ƒΕ@­ ?CYZW[}d™ξϊh;΅'„M%_κnι°–κήΈ4g6]ž.ϊϋΰγ‘λ%X Χ²λά5;Φυ,œi/ΐƒΑώO”«‚eτ“;ΰFpΓΏ<$άΘΧƒ—AϊWλξ=κπ{Δ~0f‚α1θ kρ΅½³Άϋαa΄L€pϊ&ΝΓ|qΨΚΑ6―ξ@;Πvφ« ώ‚γ +ΨΟ[Aν ο€ύΡŸΐF ^…³`iΈ*`ΘΦχDΨVΗh'°^χΦ=αaςRsx$彧ΐBp.Œ†ΎΰΈ~[ƒ ΆC;ξΪ~.ΘΦsDόΨξΛΐvϋΜ|πΨFνr%άJ›n—ρUΩψ=/Ά„aXή… Κ‘ΨσοKΑx”—Η{SΠ>o‚ωΤM ΆΕ>~ €ωώΗx¨Vy¬₯jΛΘ$δYP': N'w+P«Αbΰΐ›ζΒ 2ωDpR;‘€Ήτ‘―ΐΆ©Dσ¬™Δ€»yβΞfxήOλΰ~rΈ}ˆsB€UBΰWX6‰|ΧΙ”­«‰Ψ&‰άΧΝ’6εͺ/ϋ™MˆΠvn s$‰ΊΖ»ΐMsς·€ ŽxŽ„c‘kˆΔΥ6N&ŸΡζ-Α mψhp²*νδ„;¬ΛΙκDT³Α^ΰ3ύ‘”υXΆΧ4Ÿ rρΖ;αΌM_y₯¦oτ0›Ίiž–ͺrAόξ5³Β`pˆηFp­»I»ŽJήΔuξn Ÿ@s{Ι$ΰ9žLΒϋγ^—ψΧΓ —†‘ψWJβuφ†‡“π/ΈέΏΞ8XάC?„~ΰ΄ύ΅ιs2€χά‘„χ….0ƒ„—p]»ͺ¬Cyx{ •Azο\—π« ΄}(M8ΧΊ–Ÿ/53όΩϊžˆ°ΗšvμmΑύw~˜ Μφ"ΌSdώm’ΠΉΈNI©Ϊ―έ{ΪΓDθ₯ Oΰξ iu"ΰ8΄H"=«œΑ}ΜgJ{˜ Ά‡‡@υ…Η3Ύ’’αΈ‡Ci‚e„9RŽΏ7BPk<Ώuž —BΠζxžηήdpΎ”&܏kωέΐΉγ_£ςXK΅VcM$:‘žƒ³a]A#μ}ΰ&p²hDΓoΐΠξ…Λ‘ΊΞ9A\4Χ@Xœx3— ΛΨ,ovp2Όi­H ΔΉPƒ?'ψs₯[§m{ƒ2Ψ²΅G*ΒrΎ€Tœήa|*.έΆTτ―“MΈ–©e/ΪΤ‹Τυ° ¬{ƒ“εψ?p=;Α3p*Tΐσΰ$Ώ \Œw€“ΣςK![ƒˆp»x_€·ΑIkά°'lŽύέπ1\ ‡ΐ­π< Ÿƒ‹W›žQΡυmχwR•|‡ί5ΰόWε™Χͺ—oq\[푃 η~eγ9zΓHψŒ«IΦρU*C9ώωRα1)DόΆεZ°ΝŸΫΐΓΑύ5έ.‚’ϋ¦} *O< γjχΤ ΏπxΥ€Ρ©DΫϊiyΆeλTΊ{€υk«Κ$ή:ΎOόΉœ―S‘ίΰwor»άsΛα ¨© ’3JΫπobZ€γ: .‡΄μKZΆΫΛN°‡ιΆGYΖ’ΰΎδ~κεζΈlχŽ0”ΟlHδ\IΛyšžΆYΫY–*ΟΌV½„yj;m£u¦₯ΥO` ¦–*Ι ½ K•·uΚ?Ώ]yHzθypvΪy#8E>γύμΔvΖυ&η!΅!8±ϊ€ΖL/‚™:]`Κ²ΰκdΊgZΖY·Ζ_ή…_@y`t5λχ'Ρ&Φ–œτAΪ’ΊKL;φƒπ#XΆύΨ†V°Ψχ‡ΐzƒcΰπΒ‘lλ±πŒδ Ύ ώΫΑ<Ξ‡`1x—;ΰWڏυνœDά‰λb:+ k§Ώ‹`°ό₯αD°έOΒ-Πpγν”ͺhαΔο†ͺ©r2―ζ+ΧX\jSp­.•δzs-. ξ»ΓφP“,Ώ#όd²^ΧQMZŽDχ؁}π€²žΛ :Ή§Ί„½΅=ώΟ@›xipςpRΑ;_ώ/–g»ξN] wnpί²Nl·™`^¨NΪΠK—&/d|U—·Σπۏ!I\.§8™£m;Kΐ±—D:Ɵ$ώΰŒΔ³ Ψ^ηνw?S–‘ΒήξΨΗo"ά΁ΥaP>s\o-ιygœyzκI43ηγΧI8_Ϋ”Γχ`ί9 hkp₯ϋ-,KΓΑ0"A[»α͌‘z.€7αmp‘§ΗŽΰΏdέζ ²^ΗίE FV9™W₯φwsΪ'φ`\T΄@CXΐΝzΨά”έΈΓ_ …υ`c8/ΏξO—CWΨΎlύA„kΎΚΰp˜”k ;¬m ₯+ρ_ «‚υΊgW“άOnί€, sΑ' μS©ž,YΗωΰζAf]κCp/ΉΊφS«ΛxπL°Ÿ«Α=ΰž£ έ/.m―ΫͺΣY$h/Η€ ά j8Μ«Γ}KΪz}p«N_‘π8h―°άξSiΉ=φΛv»G:Ζκ6p άc‚ϋΫDP–}, …?AYΞ Ψ|φN°ώ΄n%°.8Χ–ηΫσPj l½α4Έ~‚»a0ψΜΦp΄†lmIΔ~Ω‘ω†[ζϋ@5ω"~SΨή‡magͺ ž”ϋ ώΉRamΦT8ν]‰ΐ]0'εBp# ƒΗ@χ}ΖχΟΛ:xLoΓΒ°8‰si>";ΐkY‰+Ύ ‰+Η]ŠΑ~=›Πχcπ°8ι~„šκΛΎθ„g~ΓΣJ "‰τPΧvjŽ*'σj{L ΆαΈ”˜ε…ov"\‡Γaπ3|5Ι6uΚΚ`½c“ΈΠΞtΫ 7…})ˆŠ¨o ΌEξOΐ6p?Έ‰yhμ­`;xΤ†px(x8χ†Ÿΰkx‚ΞΒsΈ‰Ώύΰ`pO»z‚‡sώAPg€ϋƒΟΉv¬IPΆηŒ―κ%„=DΪΐ‰0Ž†πLόsA9€u ?Αg>…ƒΰ P›kΐΈ2ψΤ`˜¨έΏ‚λψr― α«π›η@πΉSΑv«-ΰ°Οξγξ!ιώΜθv^Ÿ€ΣAΫhοίAΉΧjΛYΰ/Θ%χΌύa)pO΅ίAoΰ™”ΰη‚ϋgΔ—-Ο1ml»­ϋl°ν?ΐͺΰάΨ,{SzcumˆΐυB`Ÿχ‚ Έ₯MΖ€Ά·Ο–kyΟΓωd>Ϋ:?˜g¨ΰP8 ,c#ψΪΓέΠpΚγΛ—ͺ`€vψŸ#eόώ_ΥKΟD°ΆͺŠΞtΪ‰ασΩϊ™ˆξIδϋΈ/ΐjIψ;ά\Γ+%aφKΨ# χΕύ J³°-jc0=-Ӝ€‹'‘­qέ@ΞƒΆ0¬―Βυ œh@i§•«Ύtz1wΐ§Ί‚ΦώΉIh―m@―e|UΦ›ψηΫ 8‰Υ»Π+γ«Ί€ΉIΒ.Λ을΅ϋZΠ ή€5aI ΑΖ{α-αHΈ‚BΨˍγrI‚cώ^βολBhΚc-5i;²s²ι`Ϊ΄o!Ϋ5Κς ”묩jf:φ6„=€©φ³Ί~L‚—‚zQ>kɍΎς†wxΐύWΒ!ΰMΫƒ'άρN OΐΏx;?ξ'E€εmΙ2>H"½0xγΥ]Ɓ‘ΊΌ…yΓλŸƒΖͺΨη§ €΄2~Ϋ½<šŠΧΫ>ΛRΓΪπ?πγd6ύ6Έ”‡°‡kv=ί·:ٞμϊBšx?π ήl£ξ«Π% kCν!„ ΨEψہ7^Ηΰ8Τ·πWΖWuΑ:S0ž…« 4ρΫm9ΒsŸΰίmo΅€@IDAT­ 6€Iπ+XNPλξ 7νς’±¨>°,=’Κύ±AT΄@΄@΄@΄@΄@ƒX ^ΔΜυRΙxJ}©^JŽ…F D D D D δ°@Ό4δ0JŒŠˆˆˆˆˆψ―β₯αΏ6‰1ΡΡΡΡΡΡ9,/ 9Œ£’’’’’’ώkxiψ―MbL΄@΄@΄@΄@΄@΄@ ΔKC£Δ¨hhhhhhZ ^ώk“------ΓρΓ(1*Z Z Z Z Z ZΰΏˆ—†Ϊ$ΖD D D D D D δ°@Ό4δ0JŒŠˆˆˆˆˆψ―β₯αΏ6‰1ΡΡΡΡΡΡ9,/ 9Œ£’’’’’’ώkxiψ―MbL΄@΄@΄@΄@΄@΄@ ΔKC£Δ¨hhhhhhZ ^ώk“------ΓρΓ(1*Z Z Z ΐ˜…ςΆ,p™±Έh·@Ό44ΈΙc…ΡΡΣΩ½©<ΪΠ#ΌΥeέ‰„wͺKl€ψm©γŠzͺ§φ©§¦Εb m¦ti(Ζ8«Ϊ@±Όhh&ggιΡkyτͺoysemAdGψ,WbΕέJ=ίΥS]ΣjŸzjV,Ά>,Π²> ₯Μ2Ψ φƒΊh^2ύT—Œ5δΩ„΄_s€οEάόΰιx¦Esσπ/P1-…4Π³3Q‹ζ²K5!V-Π(,Π•Vl[χς επl¦΅‚Kα{P;B™žDίβ^ψέSŽΛοE$Χ'ξ…wAZ½lγΰSh―ΐPΓy0fƒgΰqp/Ϋ*Αz‡Α›0 /BoPΓΑXΘύΰ(˜|φt0Omš• G€uΪΟ1p!¨ LO"/&W…ŠΥ`2h»‹Αr”흏Βn ύΟΫ»μΦgž‹ ΆKΟ’δف ΰψ .€I°xN\ŸΑ°\ AΤ΄Z M›6ap§¦¨“ςx(ŸΌΥ{nŽ„^ΔυΛ?-QNnψŒ ξ4E5-0ki:·ΎqVŸ‡M=tΟ ΥήΙψώϋβΑzbVt]χ§Sx.Τ‘."{ošΔ³S6Δ?$Φ;n?‘PsV9y=-γea(xρQ󀏠axJ‹γξjq½τHε90εΧ[}ΚHσM[ΠΚx Δ½χxh ³€φo τ^lTπ²Ql @ϊ©©<₯Βi»[_umO=τ½y¬₯Μ@Κ"τΎ ϋtV‘σφVνāpΓέ yΓχζδM°+xS½rέW'ώyHk3ΦιMΣ‰σ%άΚΙ8:ΑŸΰ;‹_@9A}n0m0όj°ά₯Αr'‚}Ωœ„Φ18yŸα ΆΛ<v„Eα n4a1ΘnQ5jR΅‘φρφ<lοR°΄ƒ°ΰ―Γ? ”‹ za0eΖ€ p‘.kΐ π9¨`5¨„ΑβρV«ΓIyJΑM{ \Ύp^–γ&2; ε =£’¦ΕkσπνP‘ς1ξπΔ―ΣφמkΐωX΅ “σt^ψVγB=x‹Φ‡Ητ€Τm©πΓψέ[²u2#’Θ±‰kΫ\+ξ5ΦΉ8€5„ΐΔ$Β½v<ƒkυ-ψΤH°έuΡƒdrύn Ϊθ.¨‹Φ#Σ±©Œ―ΰο› λ-‡Στ χS΅.ΜΗHΤ·#|"jp―M₯½‹@*<xΒ`ˆ*”ςΈ}ΈP<½A«-ΰ匯κe!/₯ˆŒοŸ—“ώρώΛηd_:‰ρ`υ€6.[g‘+ήIή3;3α3 Go›=ΌZ'a¬EΏ ΣΌiνF }:"ε?.εΧ›έ―S‰;V’ N‡e3Ύͺ…œnO]­s1)-“Τp=Μƒμcφ" iβY* Έρ\wϋ$Ξ±΄|Ϋ¬vγΤΜ ν;¨E­HVIς-‚{EβΧ9 ζO…χΔ_—rS4^ok©ρv’‘΅,›IΣΧΙjώ$\Œ{Μ”JΟ^·Ωαυ`<^€ƒ< ηiΉgeke"|c47žΑ!Έq;dΕt?ςMIPΊm^FŽ ‰λ:Jη΅ΏΩyCΎlΧ=;h6<AΨ/O—k8θ<ξiω‰OZΉž]‰ [₯3εαwά0•ίρΘσLβlΙ©|ΝΪ›ΗZΚωQΪΤ―έ?&ߍϋAβΧ ύΑ βd— Ό5ͺ’TΘ ΰ‚μιΕM°Θwځyλ"Λ›ΌΔ”—ΫΊ.¨q°<ΨV'ΰ‚P(yΉrΒΎΪδ°ΪcN(ϋσ>¬uΡh2δwΑ P›:“Α:΄›ςώ3ΠΣ@J·αw,}`›•Ÿ:Xgxx v€Ίθa2½œd…ϋ4ψ)ΊvΟψͺΖΈ=ώI8:ΡΣbgyxs‡ŸσΥ‹²λΞKΓ$πέΉo…R˜‚ώΔγ|T CΈ˜{πόj$Z\ƒιύbEΒaέΰ’Wπyaχ Θ¦0~‚ /σ€u–B[²½γ‘ΆΫbώ χ³#ΑK’{Νƒ ά_d6ΘΎ»χx¨Φ¦ΣΘΠ-ΙTλ^;9 λ؞πFKk u=Ψ†6`»Χ|P)Ϋ€+ζQ―‚{KΨ—=ΐφ͏η7π\ΘΦ†DμΑp5€ε|ΈnHGFέ,Π²nΩjΝε z€εD :Ο΅ NΪ£BB-ž ΓY­ν‰”YCΨ ο‚KΛτγ$ΒΆ9iO” ¦P²?gf{ΌL€ε%&΄'ŸΛ?ˆHΗqIΨ†ΓsP“΄ιΈ¬ vΣϊ6HόΩΆϋ‹x/uΡ¬Y™ OHβά|7³Ν .hŒU xΐ »‚ΟγpxΈέƒαp} ]ΰ5P—€Οzΐ}wΊ<ψ½`Ό»€οφo΅Xn.DδΠ³Œ‚št.‰ΛΒ¦ΰΑώ0 ‚΄v˜~€;A€σΰ@pΟΉ¬[Ν}ΐ6ιͺ!μGβίϊΓxx~ 1xΞ‡'BDΚ„ίςηΗΩ=*-ŸΩΎHGF,ΗG6N€–I΅έpίIό%Έ§&~΅ΐI–Φ‰Z§"Nό.,‡r±?ι|‹>ͺ““ΌgŽD/!>ΤO§$pzˆΔν χ¦Βz·†©ΈΠV£loΠ6xaγ³Γ!―‹»Ίφ„<Ήά>Dξ˜J°{g…]ΜAΎ pΡΛ0¨ΩΑp°΅qΫƒOΆφ!’w*rVόΛ§Βz€[²β,Ϋ b°>Ϋξf”V['ΐiιΘ¦ΰΟc-5…ξ6Hf›z)¨‹ά/χ­KΖZςΈ.Χ¬%OSO>€zY¨I;“蛬¨Δω¬%jeAγǏ―1OκaΨ]ΐΫΊ·ΝίΑ›η₯ΠΊƒe½›ψ_Δ}”―‡‘7ΒpΌιxΛύ:ΐ™ No›Φ•-Ϋβ»…Iΰ»YΛ³\5³€‡ΩΈώ„`sπ™ΑK7έ! Ό΄`[ίoΑͺl•ρU]p6ΕXώΡ0Œϊ¨Ώ¦φ˜^<Τmλl`?+Α ‘ώ ύπxα²ΎKΊΜg[\`ζ΅~ΗΙt΅;Έ¨΄ωDx„ xεσΧΑwω.a9Ψ6Dΰjη#ΰ)X>‘ΰ»ΐ΄ΌH\§#gtžkiFοnƒ΄·©λΡ=ΑuαΊ» \—S+ΧΈλN½·e|ΝλΕKšo’ƁϋΜ3V> ­ά.‡(,ΟZς―VωTm!υ›°Ε?RΏUΔ dpiπUΌ„ ?mhRšΦ gοhΣ‚ ™λr»Jςςᛇ¨&l|ΦRΛάρΒ0γ ΰα4ΥcψnΘOCjKΰπΣάQ ε--PΏπ“Δπ)jύΦKo˜Ρ?ihƒPC'ό²Ξ Υ€?JόΛΥ€ΕθFf|nς¬ιΆ9Ρ¦vhbΓf0 l-YΠ ΦχΨάhFiΈ– ?,Ρ¦…·i,±yZ Ÿ΅δ—q’’’’’’’’j΅@Ό4Τj’˜!Z Z Z Z Z Z@ ΔKCœΡΡΡΡΡΡu²@Ό4ΤΙL1S΄@΄@΄@΄@΄@΄ΐŒώ+—q£’š―Fησζk¦ΨσhZ-0ΊΦuΙd]¬σD ΤnΈ–j·QΜQP TΠf¬Βό Ήώ΅Ν¨z°@m§αgꜻκEF 47 ŒζO²ϋΗ«’’κΫ‹SJΪ?”Φ՟Nο ώΕΰ¨hhhhhhj,ΰχΤΎ€ζό7v§ώϝ¨hhz΄ΐ±”ν?πŠŠˆ˜q-p MχΒΠ\/ ^šό‡‰‹BT=X ώφΔ΄Υ¨ΦTδδ\±©t&φ#Z ™Z ύφΗΚΝU«ΠqγξΘζj€ϊξwΌ4L½…ύ'K€nSΠ tb΅¦Π‘Ψ‡hfl?θϋ™0Ύ™Ϊΐ7?χ7ΣΎΗn7r μJϋόp`#og]›ΧŒΤ5sΜ--Πθ,КύσΒωu Σ χ©f₯†©*Φ-Ÿž$»—†7ς{¬Ρ枟–ω»Ί5ώFM£m}lX΄@΄€wxb†š‘9:Ρηo!ξaυ8ψρΗSg\$±|ώzSSψ…_ς°4DE D ΜxπΣΒ‘I³ΏŸρš?Ν-φGBsύθ4°.ΔKC]¬τίC3žΉλm’Ζ…ο„pcns]ΪΆ;™†Τ%cΜ--Π¨,Π‡ΦΌή¨ZΤ°™ƒκ~…¦²7¬υbm f¦φ=€₯°άη f½XQ΄@΄@‘,ΰor_¨ΒfΐrΠζψEξpΰšc“?₯ΣΛ4‘Žϋ"Η»9~‰ͺ‰ aμF3΅@9ύnΞίGΊ‘ώϋ§££κΩρ; Σnΰ¦τ=Ώ@τ"ΔΏΧ0νσ"–-ΠPXžŠώ†ͺΒFVίηΨόdT=[ ^¦έΐMι 5šZ¦}„c эΫιߚhά-­ŸΦω&§Ύ©Ÿβc©Ρ…΅@Sϋΐκ˜η•Βš(–--Pπ­\ε7φ’Ο‘'5φFΖφE  4΅οΜLΗ~‡ψ-δ0ΒэhΌX‚¦χ‘ζͺιxΟζΪω†ξwόρΔ΄[Ό©}ΰ/Lς.Δ?Ε:νs#–-PίπGχ‚ϋPs”—¦Yα­ζΨωιΡηxi(ŒΥ›ΪχšZ 3Κ±”hΖgζώ} “ΏjΩ\/MoFΖΥΙMν{}ιυCuκyΜ--0½,°0-§WA½OΣ†A;b’ς²@Sϋ€%o ΔO’ςš1s΄@ƒZ`?jά 56ΚΪќ±ΰώΥ@ˆ‡Ba έΤΎΰ»iΞ,¦03#–-Phξ?šπ?ipj ΔKCα έΤΎΠΤϊSΈ‘Ž%e,Π·ΫmώA©ί2·υ.~SΫΝ»ήΊjψnίeo-έ’Ϋ͝aΫη’M+ΐc°m Υ€M¨(ώƒͺ†²vROΌ4ΞΰMν/)ΖKCαζF“+iσξ·νT\\Τ5έ±Ύ]oݍhέβώ|‡ίŠ‹‹λίύ¦φ!Πnί·-Ζεε’B—Κ«œ\ΌΛδ’{…p#rύΰπg#jSC6₯•­uϊ+}—ΉcYζɞ6pΣεo™§oχ[Ο`ξάEάy\l1~³εn]¨o·ΫΧίXΔ:Ϋ·_·;ϊδӞͺ޺BΏn·šΟ3ω䍗†|¬Us^ΩU ©Ψ4^jοf›ΊU―;f),:Άε¬%—h?Qpσ-*.>'Ϋ(•%ΕU΅<#;ΎPaΚηGhΕΛͺΌ¨œζώ£‰5«Οΰ‡ΪΖlPΡ ’Κ—”LšΘ§•Ε-&”<†3KeQεω̝?ΈΨ>³αβ΅Ύο­mΏ-*ͺXΊχΫάΗ…XgeŝσiLIII‡Š’βuσy&ŸΌΝω[·ωΨ©.yΣίπ/΄Νθϊ„΄…α»½3±ύ…³ΐ„?*Άη/ ½tηK2οr[Ο2σΏJ*‹ŠUϊί§h¦%ŠŸ˜ψqεu[-sΗw~0ΰϋ) xϊuΏν©’Šβ—*Š+w-ͺ¬ά·eqΕ;“ŠZ^JΛQώόέΙCίίζŸιΧυΆώ•ΕE'β]€zή*¬άΏUλ’Ÿ'N¨Ό†ΝΏmίξ·έ?μέm|χ=EΌ“|„OC«¬,Ϊ›gg§άΛ†½·ν©fΰeί’βΚ“¨ΓωύN‹“χΏϋνν?£3M,©8ϊϋρ;|ίMωN΄χ†ωŠ*[_JΨΏaςyNϊή6w–•=έrΟπӎuψK_—{ζ.iυΧΌΜιV%Ε•=++‹ίφή€76οvϋF%ΕEσMhυΐЏϋφ“–—Μ[Ω’dvζυRL‘χξyw›7UyΰS‡^4{‰ΚŠΚΏΝ”Mm΅όsL˜0ycζί²ΆΚυl‘βšΚ»βBΩcZΛiJοΞ3 ƒd„i5L|ΎιX €²x#L?ΟhΨ{Ϋ< ‡Wψ—D₯;ο0™wwΟOjQΉώΏp˜/\YRΉXeqIYEλΚη*ŠZάΗζϊe-PTQ΄πJήυυδrΡ΅²ΈςͺŠ’Κ]L+)ͺ|œΝρΑ’NEγΈ μΞΑώbφ…!SWqΡBπ+·š½€GIIQ‰cό~ώτ{ε΅•{Xυƒ“'·xΘßCΑΓdΙ–³•t.jQ9€6¬Ϊ]Y9ΣΝ„?β™“‹K6γςrώf]oι>ΧΟίΣ·Κncη™ιρΕz’Ώη›]—Τ­/m@Α/ƒΏ9Π\UηKsuζΘP uΟ»;|Νψ16™ CQ₯nξ@τ<Ε·όΒτeίδYVπ’k8hΣεοoΓeς ζΫrLά-*+^ ŸHΌέ­Λύ|Ύ|@eEQ)οπw%ί3>7y¦ΏW›\Tς`qQΕΈŒ0η*†χλvϋ½$mΔΌά°²ε„—˜7Ε-'”¬VY\όXQEεi”1—–{ϊv½}‡Pwpύ‘JeQΕΕE ?{g'G‘φαξž•ΈApI 7‚k‚8ξξzΑ9δ°»ΒΑ"ˆaA\d7 $@—•ιώžμτ¦3¬ομξlo½Ώί3]]]ϊvOΥΫΥΥΥ\Χτλ:ζ9“Ρ‘Ÿη~͈ɑΆgμyA©>1FCru&£Aš [}’{Άλij4xνy χCΉ«o[ί»žΧ‘Έπ4¨#§dšoη§mA‡ΌνΔμΑχ)άΔΩCf²οΉφ‰εq§hOœ2shlΥΏ‰9ƒΰX‹θ|Ww‡₯‹mΤˆΘΔ™CΎ%ΰΒ›;:ΫV‡3eΚ¬‘_*ςΔYCfӨŊ?φ ΑξKΗπˆβLϊfθ" ‘’†™0‡9tΜέθq£;Σy|±~–cΚέgχΛgc+mŸ­2Vι§­τ«Aκϋ£ ½Υ₯Qςμςθ–Nt Կܹ뱚q²θΈ₯;~₯Ε¨—€53?βmv³dηi‡†“Ÿ?£K·Ϊ–Σ7#γ’ INz{§ί€Yƒoρς "ά22Ψ·[™ιœΈf―ΓpxƒG ΏLΞ|qμyΓ8ΈΓ€ŽΟ'Άέ*cε±€q?ςlšΨˆ˜ί9k'μ‹K¬Hζα“³‡ήIšŽφύΊfυJΛwΞΓϊΓωl ’K >2©š~ŒΡ\Ε†­“ [}’{Άλojθΰ5 Z^Ι£S5’‰h|cwΛ Χ6ηΨ B’.b 4Ζ`hNGώ§ο―0[εΨ&Ώβ]ΆY8εα6ƒ±ΌιΕBό鐗e5'ρU~ς^&wFΤkΙ¦ΐ΅ήXp)ЁQρyl(Ωq§ShΰyΞKσš€Ιh~Iή¦“žήZ™œδtλRrεe8·ΧγιtΞ[δn,Πγγ"Ρδ\«½Οuω1νΐΑ5·ƒP­Š#0‡#oΉiι‹ϋvϋ‰kEnPΆX!’^ttώ\χ~=s26|‘HφZ7’-i-(mΓΟΆV3J{Τ7N6F#i2~πσ‘3ΞΛ—σΟV[˘m#ƒDϋ'βΙ@v¬hξD§L˟ηj€―#έ\Χb$Ί嫰±ΗΆΥτcŒ†δ*68 Ή)ΧNjΊxV‹­ν^;%0Ή¦œθ°γ±Β–ε-ι–ΆkύVZψhZΑίNC­~8Ω=lΫϋŽQ mΡp±]_hIηΎ³±bCΚ~ψςn=Ϋέ,==&ξ.Άν. ƒ™λx±:–yλΦjΊ2ο'6ΡHτEέQΖξ*mο7='tw°υΪπŒω": ] 7ΩvœS―δPœ ₯κ³ςM₯$Λm4ŒœqnΟkΠΘmΰW@§˜Χπ1ηϊiΞΧACUaΈξ˜θkMωzιnδάtΧΩχ!FϊηGάOθœΥΞ±x40;q―βρT/?ŸΒ­­ό‹„λI# Nί³υ4ÝGλΦ}?Ǎʽ”GfgψΈ³—ΧπEΫ±Φρ(20<ΟΝτγUΗΦ ΙΥ*ν©%+ο€δ&[k©ε’σLΠΔ/#F1 8–ύ‰γz=Λ«ŽΨΔFξθJ ςŒaΛΈΫΙΜφ©<χ=‡»©Η ΏkZFδωŒ¨3–?Φ<Γ}’WΠΞN·"―φiΝvD5αvοίeΜ₯₯₯<ΆήΞΟώVδρ΄^ΝΟ‹ΎΞώσ“g ύ‰ywΡ°ίΐ3ε«ϊv{;wœύwΤ’3h΄ν;Χy‘;ΝσtŒŽγf숟-/Ί’θ‘Ύ]Η\Ašηό0Κχ†β±Β Ηυ‘;IŸt&&)­Ί˜ŒŒΚΞ0½|…J-τμmώΘnΟ6ζόΎΖP“–γMΦ:βτ6Ouڜ»±63Hϋw}(s²WE6ζNΜ2Γβ%¦Ž{³αΟξ9sŸ|yζ°Ω¬ŒS•O4cmα‚vΚ#ΆΥηΔ/μ¦ Μ«Ή˜ΝkΣ§χ)28’ω?Ε―™]`w‰=6c‡ΌΏ°3r™GαMα1\Ζ―ζε0šq&‡«MŒΡ|Υ†mH?lυIώ―w)ΪγyN|lb΅έtO νAΨΜUHλ`}τ—›4‹DEΎ?wκ—σ&Ε­<؎Nψ›ά {Ÿ1h•†€σ6μΓ\„ΆkoΟφέΥ+ή„Ωƒ?’‘Ώ‚W/σύtό-w_`πwAβώ›Ω§sσξKŸΣτξ@ηOξ8ΟS8ΝEH³ά>kΘάˍN˜,cΖύνŒ|\Μ]θ6άx.w½hOM¬›2kX6oεΔΑx[3is8Ο­_S ο#vμΡFαnΥ~Υ^ŸυΩhΠuχ6”ϋρΧΑTΫu$ŽΥΠΛEκ[ίMd ;F~Κ· €―g½Γ„Δ―yeρ0ί―ΈνΐY’‘ξά[άρM~›&λnς3.£ͺi@C±Ι|ŽY΅T=Ά&^u«z2&££*h@CΟυωx$υWΫZaΡcξθ–ρψnYm“<€΄μŠŽιuJήΌP݌Τc ό“Ίί’ϊΊ\’ϊ˜ͺ Τ5 ΘXΡPŸε*m}V@*Τέ 9WΟYΫδΑ°Υ§zΞΊIΥh ϊ4 G“ͺ/ω:‘rΉ_΅¬΅1…4h@―­€°e»S—…ϊ§Ρ@•4ΐk‹Οk‚Z•I±Θz&^Γ̚©‰υUκμHΛΐžY­ϋvΝκ––N-ΥΞΗ2 €Wmʜ]žj/‘E·ξ=Ύ`ω%†ΛG,άtŸΉήŸο\ME’ή(c[–Ηε+“ήΗ–γ>I™Ο,˜λʈ=§o·qΗS†‡XŸχŸδ§g‚fεΘf“³‡<ρμΫ0xσΣ~7jΫΝ[,I½v+κĊ~Ξ€χ)KW·qr7θϋΟbŒ|„¨‡u# φ°Ύ„;•ύ?‰σί+θ°eϊΊί—ΎC]fγ_ΊdυΚq­΄ΆΎ²¨°8΄2ίΚεΏΏƒ3CjΊ^OΕX8drŽu±cΉΗχrΚ4™oDςη9Χ£·ύψfΐσ”ηΰt/ς&I}]·Œ†Ϋ ΎΚΦT|7ψ ¦ΐ΅υγY,δHή[pΞ>hΎfύ`­ͺU"Y"kΫjQ.½K1x_α\Ϊ|Βϊ.βœΙω_ΟΉž‘Υ›$ϊžDΎ}’'ΧϋŽΛ­&ΟΜδNΫλίmά½Δƒpλ ΐάhFtXl₯ΤXΜΤω1'ͺο\ψlυεP³)‡­>5«½ΝΝαŽ›’­tΌ}%οAΦΥ?άuςΧΈiΉX*χ*­‚Θ—'Ÿ‘Ν»‡†―θqwʟNΞr3ί` q΄~δξόYβc`Ψ Ρ‹0Δͺl[oMΚϊΒ=Oƒx=FΑAUδΫ,ӌ5ΐΔ “»!Ώ?ω\¨οKΓΕέ}E°/²’γ`:ω‘ϊV…–’&ν“)Ογ‚ςL䞼¨¬4Ξ2pε”œΑοc}d₯ežͺ:₯»φν€»WυΘ™»sz§ΧΩ^ύ»d‘OΆs γ ©νψΟνHό'Χ–‘žrbFͺο”|Kώ<€%Υ—M₯<ƒœ4q­ ¬­±\MFΥͺτ¦φKωkέΓ£V$‡Q€%QΟΣzMή]£Ύ’»¬&¬Tχΰ.ΟSƒΌΙXδ bƒλ9?kίΞ‹ζziN€θ˜·iCFζ:4Ίώ±ψΆ#λόw"Ÿ…ŒL;–ύΰ½/ˆ-§‹C[Κϋ‹:ό\'NΊν\ΓδΚϋ)ΛVάέEhŒ_T‰νDŠ^Uddΰ+/šϋo!tΝ·\Υ§XΤ'&+^_pςκ˜‹»Kκ«Sα!k£Ν‡'ˆ―~χ’ςF̌Fέ]γαό:ϊvEαςΦΙΏλ¦σ#\ή¦Χ'#–{›k;ηηΉΛ­yŒΨ8›-†Υ—Π―Cw˜δ™*’GEη³Ζ εYήΜ95vΛΖphιδˆ‘πm|yikΜAŸχν:ξC7-r§w_“όk‰kM†μι…εΆ1’­'fvλxŠ΅_Ϋϊ‰eǏ]Ώή{1³‘Ά4·9£qc&σa¬{“Έ(XRUl’š°I,¦@Τ‘G%Ύ†}ΒPS‡ΈV[ ΣΣ"Χ΄X³a |.¦ΑλΏ’iƒΛθO¦3ύ»γZ7Σ€uv-ϋό Ξhθ°6‰c³²~±b·π½#…_Ζ\ζοkΛ'WΣpΎΙέZ[¦KμΎeϊͺœ`Έ˜ΫΆŠRLβmqσ—1œKvJ³’ΓIcWe=p4..=Ίœ±ο Xξ4ξψίΞ³’½ Ϋo=Ιfu˜ωKx,’ςNσΛͺmΤvvη_₯‚Γε3ΘlήK —αEΎŒ…³5'€PΠΑΪuvή1ιiΞΆdψ4# O–θΦHΖD?l=ά6 Ξ}@Ζk­£GqaT,ζrb7Ok|οΨΦσψ>„•‰QΑ'Φݘ‘QθοάΞkŠAΫ\sbσ\ϋ{.δ)20ωκŽρΧέPF$~ΠΗΩ6ΕK—1ͺχ\„mH?lυ©ή³_R§q:)Ώ :~]γΖiιž3q1“ώryΜΐG™ψ N'竍 ¦1tp9ΥΙ¨h•h@‡ι+‚šhθy‘kΨLΓs#/ΣΠϋΌ4x{£¬Ψ¨―_'β.μω2|λ‚—r†€±5 ρ—rλSΕ4Ό<ώkY¦{ρou#οj’!_²<™ύ’–—ΞΆ|ρ^‘Ό‡ψKχλ6nhΔs?iΌnm©—‡™±εqέ³Άw£ΟΆ­ƒΈϋ<\ χν2fpΔσ>΅š1Q4AΟεΚ]ίμQΟΙ|‚ŽkCώBύΒ’Β{ϊ ¬}Χίm\―w1rυu²½‚ΜŸπ?3‡O¬sQ_ΖpUJN:«5_T;γH‚tgΆΆ…0ΜΠΆ:‹V°ωP.FŒ‚ΰNώfζ7aα5AΚΈ΅wτο0Ιp‡ΚΔ―£qτex¦Ž–ΏͺΕVτ#2’j\4–)ψυ₯o†§ΜχyŒw$ΖC'Fζφœ9χ­ֈX»8°WVσόΌ¨-aΉΪpΌ-ŸK[»yγΥΰcx'?Ν΅_φη.hR―“ηϝ|K>Qq―'+Ύ‘πkΰ}ͺ¨‹>,ΒπšΥ#,•1υ¨> ΘhΠkdΙΘAFΚ~NFZu$ ½Ζ§ΡθυU4¬?―ΎV>UλmOT™ ې~ΨκSύW@}Ν!Ν}Ζs½Η’QύΜΖΊγtb#’‘^HγHΚ¨I“Α‰šu ΨI-’Mh€ΕˆΡ@½ΐqΤφΥψ4κ2:Dυ1U1HE Œ’PΜƒ¨Χ"£ιΰz­Sωz©ΝΠ0cXFu˜n-—gTΪh f4F6ΐφ5“]Jζ’Ί/ƒΏΌm’’₯­G… KG–Κ§L!—@·T.dΚ¦Yί™°c☠FFεΧΐ!Υ„c΅υUτζˆ΅ŠΦW€j½ΡP3g&lσ>FmԌκL.FυNυ}A'πΐΜg¨w—Ύ©°―°ΝΈŠŠΕή·χ+hΆFFIΡ€^3δ}ΎxYE &imζυW¦ζυ]a› UκfΤχ“jκo4P ΠJ‚³ͺ!έΊ”€FbkΤ₯BΧ—²šΗ5s¦Γ6ΰ+Τ¦;‘¦5£>“‹Ρ@½Ρ€MLͺ7΅-Ύ’ζUΛβυb|λ™&Pί!!ͺσ{ΤεπΥΗTΕh 4 ₯«{₯BAj© z<σ μRKω›lΛΠ€i(CAI<¬Ιϋ'1½ΪN*l“;k[Ÿ&£N¨@o&ΥηG{Q?‘IE £‘ζΞJΨ:Ω°Υ§ζ““Ρ@ρ€·y4aޚ(ώκ0ΎυNΤx „e@Kκ’E«Μβ+(ΑˆΡ@4 ΅FC}”mβ•ώ†m˜FdλγΉ4uN’Β6@³Ό{&Q?&)£ϊͺ©ψοP_πQχβ:0#ΰ("U₯Ύ^ ΅u>τΖΑv γ! Jθμηa¨Œ©ƒΡ@-jΰtςΦΘέδZ,Cmf­‰βͺΏζ3LͺΝ‚˜ΌKΧ€±θJΧO²†m@Ψκ“μσm3(―τͺεΔςa8-θt%lΒΊ™* TZa›°+šψ±Ϊ0Œ€Φ°4jW_Ε£ββΆϊͺSo£’4ΆyΏRQ=‹4b4`4P9 œE΄±•‹šX2f‚ύNρSjNP͟ ° ι‡­>5E˜λ»κϋ£ υp"ΈΪ1’Ί0!kώά΄"ΛCα₯šΟΊZrΤwο5!ς΅jIέ$j4~ h€n4δ…Ώͺ%ΦpGτ)l#)-Ωi€f5 yΣ ,Cϊ{S—Η‘'1(Q5ZΖA3Ρ­D ™FεΐςυλΧoY&Ph4¦yιœ•ΥΠ,4gΗT€Z4€Ρ ηΦFŒŒͺ Ϊώ™9 U8yUˆ¦yωθAkελsΩFŒŒŒŒB¬c4ΤΞΙ “Ρ †­>΅sU˜\ŒŒR\Ζh¨ΆNΆΚυιΧuμγύ:gυžŽ~]Ζ‰ΡΛΎ]Ζ\έ―ΛΈAΑ0A7aύΊŽωGΠΏΌξ½²šχλ6ζ°ς†OVΈώ&lΡ·ΫΨUφέƚ›d)Φ€c4`4t £!ι*-W‚_J"Γ2ΰκ’OΪ¦A…ε„γζawγI³}γGVGjΫΦSμkΩν˜δmlώˆe{ΧμœΥΔχσ·Gv{Ά1ξ έ4―󀜑·ωώΩζηzgSŽ“*'a½Œό#mΟs;8[NΘςi24i   T‡*ΥΘWGAκYšΑyo† ξZ3~1θΥKΝo¨8žwƒεxχ#yy#=Λ~‰ΞτίυΗδr7ώv~Δ=Ώ}’³{Yπ,zή|ηœŸωηΦ;όΡbωo'[–½‡e[s[¬ήπτ¨EglTœγ»ξœζ9ύ<ΧΪή³­™yšjΨθΟΦηΡyΫ­ϊuwbΤŽΞ‹D#='ΝόΌΗqνξ“s†ŽξίeμΒύuΌγxύθύI9C^οΫmΜ>ŽεœDλ=ΧynΚμA οΔ£;D½Θ`κΧ”7Πߜ4kΘfη»πΈ7ΨΆμfyσ’—εFTvtϋTΚΧΪ±Ό·&ΜϊͺΒΏΏ‰όαΊξ±ΟϋΠχΧ1#FFF5‘3PZ.>*ιŸl­ωVͺ>Ηχ½%ϊήιν#Σύ’χο2ξL”]νxφίΟίzž7™cΓόύ­́ϋnΟv¬E'3·ες₯c,Ϋ>ž#Σ‰ΣsU³†<ζπμΎ]³ΊF\ηΛ΅—{Žχ.ώY™Vί‰dδRŽŒn¬Α X–ζ:Υ8ΕΟGϋtμΓ΅‘q’λxΟ6΄›ΙΘ Ξhl–oθδ—Fwϊ έ²Ϊέξωf–3²ύŒAσ)ι?Ϊ―λΈ~~šΪζΉiλρ_FΦ;–ύσ€Ž/μlΫφžk§YŽύ™kΫwη&…΅»ΏηΉ1#Ζ΅¬100¨Q £‘FΥ½Yf•κd7K!΅v*UŸ΄|kOΊαyγΗŠͺ:ΜcΨΥ³½Λs7\V\υ2ΌH6ώ{œΫλρτΰρ472†}wbφΰg "ξvžεφI:C'Ξ2Ρs=ΟΪn@η¬ύΫυlΫ6qΦΰΗ&g}ΡΆΌg0B:ŽzΠ_Έ–7bΞ χ‚iη¦cqRφ«'Ο:cαf½|bΐ&ζ ύηΩO8{izfΖV%ΔΎgφό +zDΊkoφψαεΩƒ~$ώ‡„YDόρΡ΄Θω”g#·LΚ<#ˆUςΌλnχZ¦ΚqB}†\Ορ,ν100¨I $ΕhΠ²»Ύ°‹_p=ΦΦΐYίο„žcΆλίνΩ­όύTΨͺL§·yΊAEΚ’ΊςL]«:VUͺ4 ͺ™WCόJ –η°Ψ½’¨<Άυ,δψΜ‘=<ΫνΝyπŸ=h-nχ·ά¦%žk΄'Ντ|Η}ƒΗΣ„βΈk›ζ«7,πΌΘώψ} ?ξΒ—ω?ΐ ˆk}Ψ۝Ž†’ΌμΨc‹&zDA¬»ρxα›–E­Θ?ςμθfΖN ˜#fF4dΕdβΜ!ίβ°3›¬ΪΆΠΗ ζ[θe~ŒŒjHe6–ε)G~ΎϋHΤMMκcvϋυ^zήηŽη>?ϝӷsV'₯A‹ΝγιŒQt›7ΎεΙ šΒ8QϋΥ•MThΆzAž{Ρˆ$)8 ΙΥz )Ξm›Š”„αχΥDΪ4±Ρ³ΦΜ‘t΄7“Ξι°5ilx^ιΖοΈΣZ­Ν]₯ύβΕ•n`AΜ‚~λ­ά©+š4$-ο0ΧuNζΨ.m=Θ…ω—kΕ“r€ωισ(cσI«Ά‰_Ιΐε~^ŽΣΟrςΡ„Mζ%Lΐ{Ϋ΅ϊ0‚Π”Š‡ύ4‹Ϋb0¬ ­"γZ8—‘»6ϊg,Όησ-. γg4`4`4Pm¨²Ρΐδ¬ύi ›iζϋ ³ΪΡΰ]žαμACy φΑHΗρnQι_™9h ›ύ»Ž«ρΩιΥ¦½ͺ'\Ή»σͺη[])TΈ>iNΑ<ξτwσ ΔDΑΏω―ν9—γύ€œΑGϊΗ4Z­°‹όIΎpΫbυFΚagφο2ζRvυλ2¦7χ έτVLHΤ2Ζ‹4 ΐ„Ιm™σp>…!ΕΛeδ£ΝΐžY­#Μ/`τ Ϋρ³v’σί†pgΗΒσΓ Δ8άͺ(ΝΡHsέΙ–›>$Χ.ΨγηΒMœ=8“ŠQO#%% fΖ£sNθ2ΊΫΐύ²f6H»›ΐoΎΎΰδΥ%F2ŒŒŒjHEwR•ΝΟ±+\ΟΥσdkΩs~ΘξΦϋψ'έ&LK{?AΗσSΓΞ¨Δ}ΗrΟeBΨδξ9soapΉ8™†ώ*Ž΅ Υά&f›ͺχυ=ΗmGcέvδ3χάχ6“aαυŒ£f°m;-ΟνCή‚†ϋ³hzγK_žqόϊ`ϊwΝڏηΰ*CkΚψ=7šӁ}―²εΝsoηVt0α—QΎ³4· nu²ΗC©wŸUHΏ¦£ͺ>ΐ εΝψ₯o†Η°ώj½EπRΞ°y›Εs1ˆX_ύΈοώuž7“ά‚Μ|'cΓϋς”AΑΧQL$Ό—λκ Ο±—8–ΫRΞ°Ÿ0ξ}ΌΧ±³\ΟϊcΰN¬…“c#`ήΈΧ1†ηx·cΈœΗ„Μ‘Lj|7Q «~]ΤΆώdBΩ‡QΗΩϋεμA i€Ο£ΐ—Ρ wΒhΈŠτsηΐH^Ϊ2/=w>ϊGλμάa ¬τΓΟωw¨;b4\͐π 4ΚjέϊΫΨ¬zξ^'ζ Ύ_·±_[w“ίζbόδ &ΑΡhŸJgrmœy]Ώια"̞ΑιΞ±σ­V˝¦N‹2\κΧ· ΫΆΔύv¨B©uO σ$θΥΛr σω€;ΡaήPV$O8—§ψ―4–ήO hΝ|>΄Sή6GΧΠ>³ΰ=¨)Ρρ0Π“y/Iύ7:ΗSM•7Ι ήμζ%~<›CId10ΊU«’σ}*<e=Z;œ0ΜGŠ½Άέ’m²ΎzΉS<έ·Ψ&Κ.x¨ΙΨMGb€*μ;ΔuAσ‘zύZΥ₯‚£ͺg˜‚*Uiiυϋ/Ϋ94”X“ΆΖς’#ΉZ>εΞP# q‘+Ά¬…Ή^Gί§pλθΞ½Kžγήbelμ™ΦΑ9”Ξϋ“)Ωƒζ§{NΫ4kγ:F‘³ή‘»78Šν½₯0…ωΫσΙλ…7³O]—‰|Β+n[ϊΑ…7yφ 9Σ§χ)ΰξ~Žυσi±"Η²ωΕσ’Ά†²™`#ϋΫ~έ΅SgF'ϊaX<2~Ζ U2\¨ΑSŠ“$YH:ϊS΅IRz΅Μ7 -4―HA<ϋ)F‰(k‚©Vˆδ||n †Šh·N†½™RλZΧS Σa,Τ”Θ`ϋ%1Γ!€Ε¨VlžΞel€%pT‡(ύ«#α ¦!ό(ρWšΜΑq° μg@²DφΥ%$6=‘Q Η+λέ›ˆΕ#οΒφάΈ»Ξoͺd4x‘ˆ›¨Ν_˜8kΨ[ΝΧ¬FGDlƒΘσr#vt³4)gΠ ]`νy[πžϋsσάΕύΊfυ:Άλ -™tψ2(§3“ώ,%A‡―N6&ΈŠƒ0\μ2­o²['EΒ’?όžm―β`s_[Χr[±iN‡tZ–59Ν*Θ'Η±Ιi~Ο–“LωˆΔHf‚΅˜C󱻧 5ΆΌ‘—ο8Γ "K}\Ζ¨b7½Ρυ΅X?“uυk@o3iΔι]Π$Π¦π&θρ`β€ε’ £K]K™ΟΖw’³΄cΑ°*OI°€ΙΈϋΔQGv9¨£Τ]uPJΛ'.θ.OœΝΪ½`dά›΅Ιcς/ι©sUš””¦β$–wWόtΞΟΒ H”M¦7?’˜–T†JIeW]܈ZηΓT(”Tι’¨oΒέ|}«?‚Rš^3(RRͺd4XK©U+ζΔa₯ΌγxVόž_ΣuK17ΝΪΰϋΡ)o΅νί‹φqΔ_©ΫZοΣ3μΏ3–ΓRξiι^δd:m[#x'ώTΗq?δ¬OL0™’έžΧΙ?Θσνsό}m™σ°€ΝGœE^gl•±κ\ ‹7Mυc’žΧMα$X& ]Iϋ “Ρ ₯|ΎΓ‰šΩ§Jί%ŠF•η’”Ψ¨«p)ΈŒOέmφ5όg‚†­³A"£b=¨эΒΝ ΉtΓ ΡΝΥqVυZPψ}@ς"όΟΐXΓ 8ιηo° Αα  ΚοPyn…ŠˆςVόε0άί‚Ϊ„οα|Ύ[ϋ·ΑΏδ@”ηψ>€ν Q”ώBψΑhϋ| Š›»d ΠJ€3Aeybν<[ιλ%P}}β,’AΈ~₯;ΊΘ·ΠqΏΌ*σNπ7ΈŽ΅ΗΑϋ yž€ο`LΏγ>χ\Ύ€ΗA’>βPtόL(N²γžσΨj΄A7ΐ[Εύ΄Ρ~+8¦ƒt5 t-Ÿs@ω©ΊVo]7Χ/@ή?ό“Έ3@² t^Už₯p'€œ8U)Qό±ΔΒΨ0> ­sς¦ρښ9 ³JήΠΌˆ+e>sΒκΧcb φ[n“ΆZŠ/’ˆλ­bήΑ£Μ1ΈŒωg3‚p0―Ό½νEμo™IήCd ζ°Hπ"5ρ”’Κpή±< Ώ«Χ11Αρ^Λv7;[e¬ΤŸcν7]Ϋ?ίΏΫΈΣώΘk1Ž«n_ξ€ΔJ Ό}eΈ²?ΤxtQRΓRF)J<¬ >,# ͺδΗ ?³£Κh HΧΒΆπ¨! »ΓzΨN‚‘°'|κˆ2ΐ5Ϊ}ΰ'ΈΤ@«#Rά+Α—Ά8”Ÿ:­_ΰ)Ψ‚²;£AKφ5θγA# ΎτΒq)ΘΏ’2‹ͺ›ΪΖqp2t…γαP§4 δ/‘iŠΉŸf+=Θ°κ]@ννc”­Ψyτ’ŽLqw‰Κ>8ΎύΫ€δ!˜»Κ§τOI#P»Υf@PΆ`ηqΎ•ίtπEuAιιΌΚ ˜ :η―ΐ(»β!θšh JGex Ξ“!χ(θό ΅©ν@y•$έγTΗ/K χί—νΠ€•W"γΰ7P₯'h:—7ΑD8‚’Έ2€tννιΈ|ω‡όχ€Ώƒκ›RR%£A5aΉ™ζ%@IDAT”θ1rk>AΎε-o!φt<λ!&#κ€Š›w41¦Œœq^Ύο₯νΔΩCfbΔΛ|ζΜRίΑvœ³¦d2yζΰ7θ°ΟeNBgŽ5#ν£y+βμΩ½:5F’S/’?C‘Ψφ–%nSV΅ΠRΐηď°ρžγŸΆ£cKb§κ:†AςŒfΜSy’{³oP΄εIΗ«Ήšιδ[z•4Υ©Scζ60WΓ9‚ςΌ cI5FΊ€τ ƒ|B%φ„΄T¬Μρέ²Ϊφν:ξπŠ•Ν³eL+«XόCkΎΞ‰=^PΓe€Pf£X©ώ«ϋΐd8–:ο7‘?θNUΧZπEw™ŸΑΧqΫΨκΖ@ rΛΈŸ6.\JλΘε]+ ΰ[P™Τ‘ͺSP§δΛΏp(Οί£ΫτxΨƒΨζ‚:ΉΣ`/XκDΥήΠvΖ •ƒΩ.εu,‚ 8ͺΟ‘@sW$JWν’§+| ‡δ]Pέ$Ο@>•φjPΊCΰψψ2‡Κ§Na„Ξ‡τ2²A"CΔoούς*œΒK_1b~¦ΰ·”Ξ,P[Ή7¨œm@i©žΛ 7HY° @e¨ͺΜ&9ρDΎb«2HŽ₯οΑθΊφJ•νPxΥι1ˆυŸl%*·δXM΅“J"₯WIx«ΰ‘ό¨ϋ& κ}Z ψ՜α+HπM‰κz‹‹ηEg~‰Ώάj^ϋb3™œ=ψe<„/cγ)΄Hx¬πΊΏ£α~ήί'Ο^ŸιοϋΫxΪ±έψ°χγώ±ΰ6ώ* žκ]Τ_Β~0΅:2¨α4u^BP½RJ"–·/Χ“)ΤΫε-Ψΐγόyή(^ΠΤZ CR…W–Οχ •η»€&\7λB±―€ηΰŸqΤΠΎΗƒ:85Κκ@^uP»‚a_ΤYHτί롃ΘμPΥΙζι’P"Γ!(Mβ;νΩ6»G±] -γϋΏΕ·•Ωτ$’Ϊ·Ζΰ—ΣOηκ$—ƒκŸIbœeψŽΖ₯!ΫΔt΅οwΤkύ€l₯ υ BiϋzΔi½2|ρυ%½œχό‰ν+°1Ύ―T{,ILSmΕΩ<' —(ΑτtΞ$:/ΑςΙOΧ‹ϊuΒΑ8~YρυLΟ/ƒK<eeЈΑσ<ŠλλJn]ΣeιFαjTtqTIτκ$žΘ›οžAB+)±ώ]Fc‘Ω_²ζΏώδ5&<φΐZφτ§KeωΒνSSΉ(ΫG„=Ύ,+Ž&»¦GœζΡH“₯Αω #˜'3£[§#NΤ›τΝΠE‰ιπ*νŽNAζz="γ­‹Œότθv‰i(NΏcΪ€ηGVΗ5%&ίΧG’ "φΖ)³†ύ Rψ8m£ξ*Υ–*ΗuΟΪή‰$¦‘…’6F­&½²η,¦NEΎ±4Ϊl{•ΩΚέΌνΣζ+[4ΨF«@uςγΨnώϊ‰eΜ(΅°©}PσιΠ NuTGδg ‰ ιžΘvB6•΅'Γ(θ ’™°]ΜUψσMάύΫ›`8μ 2ŒΫ‚€ pcmΟφpΠυΏ ξWΪζlΆ‚—aGΨ*ξֈˆΚφ?x$2.ƒξπwdƒττŒv=ΰLπχε§0 ρ;μLάϋΔύΆf«ΊD Β,Π9PΌ?ΐOλ"ά*_’¨μΒ—έpt?ΏΞΈ}ΓKiφ?ΝΈ/μγ,·δr x~uπΓηπ¨ύy$ͺoyDυnK‘#θ”%—ΧV2tJκπ₯ƒƒαΘνǍy€ϊOy”Rf&gωoYτ6aD ##k4ΓΚeφ1Ρό† r)€V,5šj„*­XšΓΒ_λbš«ο’λΊ±φΖ•ϊΣ€Ξγψ†―HFSΎϊ αΦζcšσυ|J»FΖΏωΈΤ}Αςh΅Goz…u:ώdA±-x₯sώV™«†κq•ζΏXΡάλ1tΏΟŸ₯°ƒQ‹ά䳊‰΅οx»5Ÿ°Ψ•G/NΞ|±Φ0iΎόχQω.Ι²~αΫ۝ΰŒ9qΚΜ‘_3aΈγ<λζΪό‰kKΪ+ωθΫmάρ+=ο?θdnf£΄φ€ΧπΏυψbinδU-<ΖγϋΡέ\>huεRγ&Qg{ Œ5°Ύ|†γIΨ6ΐcp34Ɏ…› ύF ­Ό†¦ †όk ²?n„“a˜ κX|ρΟAΜΙΉ@Λ?³θΩ™m˜Σ£ϊ@Βάj»|>‘œpμωSφƒΩo%ΪOnΚDέgmΧ:žτŽ²σ2zα}“Ύό-HS£ύΙΡ+·ΨΊO―Ψsό–ŠBΉ•:ιΩπιqtξu=­5ώj ―+ τ₯ ΞHξά‡Η\…?'²Ή:°οβξ €’OAιΌ­dhfPG NˆΣ3^εHΤ9h_p’<ŠΗpΈ%Žςjοƒ/β8p(‰ u0Wi'.2T @u}$7‚:/ΙYΐ-f€\ΙV·/*»:²ypΘ ’ΌΝ ψ€σω Q'·.ζ*ώηrΌ₯g•I†’Κ•« < ίƒtι·d`θΌKΤ^sκaJά­τβο_‹[zTZ‚ς)€΅ #BΧƒΚΎθzI•Iη+? #ζ˜ύΰXX _ΒΕΰKpc<₯§―AεΪΦΓ+ ²}_Αω ™=@z} :ƒτ$‘~‹Ή χ‡jΟ™V{Y›œПI ….&]u]Q5vmAυ*I^cyο©άMΏΒDΣWσ64Pι0/‘ƒί_ߎΠΫ3€D:.s‘d€Gή•‹8?ΫόΑγoρ0>ιύΚ: γΑ’­3VeΙΝ£‰ίθxΏtλͺHlλ0 •%ŒpάόMΜΣKgsH4’Ζς^Ξ„ΉΓΛ›ΕΖ&°΅˜b%κΉoλ€ΒS—š™†¬"z0eŸ_cm•±ϊ…₯y͟AΒG²vς£η)ŽΚΞΘΓ›*W$sέ^LΦuMθG™ϊy…σΗΦ2˜pc[ΣXφϊ-oo-[ϊ:uΏ;Έͺͺ ™|K}Dq²OαΛtίΑvIΐ­Ζ?(jΰE―:έ ,ggzΠ£˜}^Gn‰š.G1’ŽM”%κ0Eq2»Ο\ό&γ?+ΑOΖIΠ@ –±#EΊ|>Ρ“ύχŠρKτšπψ%ΰΦhr`ίwͺ³τ;Lι_HdΜ%qΏ€²o$Rμό{0‘Έ[Fγτ€Ώ Ž`ΩdPIVΔ‰ν³―ϊ§§©~Ά2|)I―ωβΫΔύ„Γ΅³λΤNΆΙΛ•W!‡Σ!\’Όk-%] ²τΓ"eΦγΰRή°9†Nq1#7g6\ύ*ΟΠϋ³άυ?Β?:ΓΆάIά•Ο *eνš<5”xΣΏZvΎŒŽ4hΈ#gœ«F H0@θΗ7“t,E|°ͺ>―ΜΊ€oΧ1wσΕΚΟϋuσ>Ί††;Λ-λμΌΗZψ@FΟΨED< γφ΄Ψ5Λ—7­ΧIϋπ₯ΉΦ1γ]o^0Κr}4šφ:xƒΗ©έˆχ]Ωs²gvξΰ1Αγ‰1Μ±pΈζ½Ά•wNΉ d   $A΅:ΐ=Ψa0Lρ\υ“μŽη~¦έξ΅ΜHήΊiτϋΠΕLΈ«hD§ϊ2[~ †it0šYΌ+ω•Ύhp/`ϋγžγ-a˜yϋt+ςΙΐ^YΝ5‘ŒπSΉι[Ι6ŸΩ‹N“ααΗK‘ν"ΚAu¬Ά)RžͺCv†%%„!xcΤqpNža{*“/WXLΗx™c9S7XyGΉy –qFκ“ wy}Απ5rg4ΆίNsΑrKμόΜΏ»ω GΕv΄oηNηϊt$e’δΰΫεΏήʝ‚rO™5τΛυvξn<'x„Ι‡χπζDgυR–4'ρŽβŒŒβλ«½]Χi?ώ“1(Oϋξσ²υΜ3&ώΎ:cƒ—ο xήιNϋΨbe„bfιvGΙdΜ‘L†άŸEΖΖ)–KΘ±σχ–NxqϋCFπͺf,ŽΛ§Ή‰CYl±fCχΏvΣ₯ω10H= pW{‚!ΐχ_f―R)Έ³œλ8Ξ9n4Ί=-ώε“s†μ'Ν*w26,αNkOFnεŽπ3υtŒ;―'Ή[›ΡήΎ’!ΰ₯4¨‡0;6š±‰OέΨ·%ψΊ₯λ―/$κcXι΅Dδ”F ‰:Ž—αω*SUŠς6‘5όωzU©L\-c£ω”ω}cFe0q’«Ϊώ€o΅y–γϊιΏF;;Β_ŒDόͺ[x\›Zωλ‘Ω)qͺ»ž/A_x Τζ-‡ΔΙ›x%EΎ#•=‘;\ύ!r ‰θνŠlψθFι¨V©νQ­Ž4H³|ͺϊG_ΓX0kxΛ,2kw:ϋ―|xΓ?ߎz»αΧήρΌ’‹‹8±pΡωyΫr¬•±&Σa,€~΄fίo±ς-žUΘ―²uψ–e’ϋ―ŠlLΕΞδ#κΐPzh€φκγΩ·d¦εζ†F“¦"Υ­­Ι δΓ Yώ΅!4iV¨ΆYυ—ͺ[ΑPΠHδΥplΥ%ΚGϊΜ½†›,ΡθερΔΆdΫ4Y §r:i΅]8Οel QXΕ‘…ΔΗ­ψξΓJζ7¬`ξCΛMQμΨ‰Š€g¬qσ\7ί‹φτη1θQ«ΖGε7Μhξ9Χ6^»ξ’•MŒQr{C/Cλί”NJΈΤɞ%IN!TŸZΡ1‹Dέ’œ*˜TjA‡‘ηPC―QͺA.€­`-h2γ<Π܏ θnR―,N‚k‘9θnV―2λXΠ»ρ»Αgπ8xΰ‹ς<>„. twž°4rΆξ"ΨζΓ\ψ ŠnvpKΆ‡Λ 5|ƒξN‡:1!ΡΎξTuW,9ϊƒξΐ•ηο ΡMΕι γ‘ <ƒ`E|ϋ&Ϋ‰pμ ’7`lΜUψŸd”7Άnğl•ώ―ρc6Ϋσ`_XχΒzΨτψPFΕbxV‚t―‰Ό:‰ Εi KΰP}nΙΕ ΄Τρž Ώΐ8!Ž Έ§Ae•θ|Nƒ³€§Š±σͺ»όDQp l Yƒ­pο_A_Ψ'‚Ξυspτεϋ | ’†p΄‡α_Π v†α σξ‹τ¦ϊŽ‰{DΨ^χ@m’ρ’$g£ ?εč:―±₯ω *-Ξ`“Ξ{p_0χ!‹ω—ι‘FΑ6^αΘ?cΠ*ΒLcΓ­ϊ°ή‡ΟwάΟ rέΓΉ™Ηβ~σχ΄τŒ³ζ½Ν(Ζ\š ]t©&j4t!ΆH΅‚U²<ϊΣιΟ•^Ιψ&ZύΣ@ͺ¬{ŒuZGn5ξκ|t\ΩΛ ι ›/Αέ0ώ[P8N§nT…“αŠΊfώhJ_͍ Ή€Žg‚:§ž(κ<–ΑΠόΞUŒ_΄ίΙίa{)< π¨\κπT?uœBΗOΙ P4 «ŽYΤ1 0―€ς=$·ΐ΅ πj7•Ύκ+ι jƒΤ‰ +λ@υP'$HΓφ1Χζ?»oŒ./όω[Όbλ=|Γv5¨νϋΞ„;`Ό /‚κ-ΉFΐPe©SŠΞσThͺΓE έIv…³cBγ`$ξy σ#£E:Λ‚·γΫ½ΩJ€σN :·•m!Hw9π;ψ²‡t­kB"‘0T!_™rΧΈπΪΨ―ΜSΨθgμοOΚ΄€εtOγμ?ΖcY‹³˜π7ζ 䍰FŒόΊK‡œŒsςc£φیUό©4˜86œΙsanÏV$}ρ;qΦ.0ΎRΨ5kίΜ†iί²œo.A?ΜΫP  $ΥD Δ°Όžj…«DyΤ|=ασJΔ7QκŸΆ Κjp3ΰ=Ψ ΤF¨sP‡ς&Hτ_Ή*ζ*ό™ΝζΎψώ‡l?‚‰ x2"Τ‰HdΘj_r LuR%‰:αΫγΥ™ώ Τ!žέa1¨“9Š“Φx6…Ÿα\he‰G€ΣαϊΒA0†Ρ ΡΝΕπ˜«πηY6ΕχΥΞιΏχH§ίA[nTώσ@ŸΪ›p0Hζ‚oH)ή9ςDTf 2] κ<%ŸΔ‰ν~Τ†)Žβ«>ΣAy~Ϋ&?‰Ϊ‰™ £αΈ dpHv αlPΉ―…ΠqΉUuϊΎτΔ±( ε«t—Bq’…ηρJSyϋΧWάΚWqχ…­!>hέJΘ…ρe>Ž9pΘΨ8žΠHZmΦ„χΝυg(’ΰ~1_·Œ…Γhp±₯obGl&ρ΄™glΗφ&ζΔή²Ές―ΗRΞGέΰ±R€,_ύٌ ”₯wΐXx.‡EΠ ξuκ lπE£/y8ΤI’ΰ‚:Ι·…›Ψο"~w€`:±Ÿ`‡ γEι(u ‹ΑuΠΕΙΙxͺ.Wƒ:”ΫαEH”`6pP³/κ TΝα}ί“mbž?Žΰα'Ψ|‘~”¦/ͺǎ N0xΧ¬rψ}ΔΈοuˆ?Δέ£Ω–$Joxͺφύβœmρ|T~_>τlύσ‘sͺzΘ( Št΄ό|—γΕ‰τβKA0ίOΩWz?ƒ ‰ς|>ζ*ωηi ΊvAhΔ#…¦B!¨ˆίΙ† *±*„­>a9/©Zu*@ΨΤH€£@―n4Τ!]ΑŽΦο$π.UtΗθΛ8~…βwLλZhΎ¨³K”4€£Ζ°=$ŠS½U7Υγ 8V€κ’sβΛΏpϋ€ΞΗ‹π_x Žt$Q:bŸΔ}ωΊ ²©σΧρ…πožd|Ύ ’a^ΜUψs;ΥMaŽoG²υ@ΧΛN τ:‚ 6ΙC ƒf?‘Lo,ϋα0b4PνΠ…«;Œ°ˆώŒ§†₯2¦•Σ€₯©\Μ€ΕR'ϊR’R{Štt7,i‹AHuŠΚQ uZφΛλTgίΌΌMΈJi@FȈJΕ,#RmΚ(Ÿ9\;ΠέΠ‰ K; ’ϊθŽζΩ0TΖΤΑh 蚞 Ω »Ο  ‘ρꔏIό5ΠθEΓ8Ή0’:ΠΘΛ,p WκΛ”$μΠPΨ―!ͺdOκ’?’‘z¬ΪΎCBυjΠ['ρθn½3l‘Δ4ΛJ*B€φ°KYK9Ύ3ΗΤ©© θόhτ©Z$ώGΥR/“hΥ5π#IμZυdR"5t+‘eJ”Ζ’V4`»ZQ»Ι4d¨ν‘±6Sχ‚ς‡τS·„ε/Y” ΊέΏόQLH££££TΣ€1RνŒl*O˜ŒΥ*lυΩt¦Œ«Ά5°Πμ~‰& vŠΉκ֏Κί£ŠEή²ŠρƒΡυΊδΣΠ$θ™‚ξ+(ΣV°-\š‚ε ]‘ŒΡΊ§4llΨκ“ΊWNύ+YŒ†AœΆξU8uηχUˆ_\TιUSY€p2dάμ›Κ KٌѐΊg2›’ιύκ°ΜψŒΊ¨*ΟB+3RΟ5°#υο κ‚ŽήΠ-θY‚[o‡νz,¦· JMh<τ†OiαZqΌ7lr«£²AeΥdHεγw²šΤ‚†€&b*œβχ†βώۚDw0”VŽ]8ήόυ2qοΝ 8ΪΠ“}₯₯|}ΡDPν«¬rK”—κΎ'¨>τHq•vΏΜͺWIβ§#ϋzπΓJ½AηΟΥΏ)΄ιί―³ΚuρΓνŽA ϊ&Š^y½&ξι‡Χ„ςή ύ ¨]ν -`(.=Όcηͺ7[­5αK:jθšι JˈΡ@ΚhΰMJrlΚ”¦κ™Aζn κz¬“)T`ΧυTP€†ΗΑ<Ί†δ?žΙِs.NtξFZ—AΗ~‚6(ύρψž7ΰgπ;SœEΧο0fΕ9“­†ρ5ΙχkX]ad|_εœŠ#Q=T¦/αIψΙxЍΒΫq±•a’(·ΰρ(νΕp>¨3•^ζ€FdΌŸU>2 $Ϊ—wp;t‚@:zOΊ“ψ&€_ΥΓ―Ο:˜ mΩWύ_€©π (Lx”φ(u~%wŽMε;Eψξ‰Ά_€ΒŽ•WDη΅ μŠ#ΉVυœ+ 3HΞ‚_@׎τ΅φƒDŠΗo όζΒ3ΰ€ςZ :ίOΐpԘTΰTce2₯Žn¦(ϊ„E¦"W…₯2¦Σ@9;ujδύŽ{/άjΈ·u& Ω6Ζ\Ε κŒΥ ₯ΕΓ¨aWZ‰r/~Η­cκœϊΙ‘ 2Ž‰ϋ5f«ŽγLh έΥNu–’‘ ζ*4–ΰnί—qσiά=žν“q·6ŸΐΐΐΎοTΗΈo|§;ΫsβnuΔΕέΧ² v|0Ϋωq·ςy6ξΦζ]8?°―NϋšΐΎœο€:QΙφ Όε<]Cϋi' q+\ΈŸΞ₯Œ= ω λO–ƒ]Κ―Z₯œ£j+ƒ‡ͺΆ LΒUΐGΔώG•RH­ΘSœApjΛ”&…4°'eΡ]άρ2}ΑvXά}[uN»ƒξ,Σ‘$Qy°ήuhJ+Qnΐ£/ά J·$Ά‹Νβώo²•¬ubAω0Ύ³‚­:auζνAO$k:¬•yƁŽKτπeε›(£πP‡ψ>Ό £!QφΑCΗΌψIlΗΐρ}Ώ¬ΪUηΌG’Ξ°AΜ΅ιη9œOt:Œ‚D‘nώζƒΒ(?V’ΑΥ°.ψ’Ρ‰όΤω~₯Dξ̘«πη56~<ινΑΐ±βœŸ<Ζ­:uƒί`HTΎμ˜kσ]+α›Έw.[ΥIΖΕ\Π…Œ7‰φΧC{azρ-·ΠW΄ŽVP(5Žu΄ό‰Εώύ=ΝΎΡ@@Ί;Nl—Ϊα§ΞL΅ξtί‚s‘4QΗΞ`XΓM(Sρ8fƒ:=ύηT† ¨³Rόέ ›§Dw―*ηN ΞνŠί‘Λ/’ΰϋΘ³ Ήγ`"θΞχH,―‹€/§ΒθY’N0(Αy&ΞulG@PF±£z= κ8ΥΡ&4 ρk —€ς’ΡύοΣ`\ AρυζϋIe‰τV–‚Η₯‰ό‚†ˆόΟ£όtΎ―AνϋϊS˜ ¨<% † …;Q1‘¨Tˆ*±†Ί| jόΒ ?Q 5XκŒ §έ!φ€νβd«h_ψΤ©M…QϋURvΗΖƒ:¬ έw„ ¨±ο —Βsπ;(έ`‡ΛnldΰΆgiiGΙQŒμƒίΟπwΠ#έέJό4{γφ;\=ž˜κ€Κ#ͺλ7 NYFΠΨΤρ) ΥG’:+muΤNΊ ^­ωύaz₯y8ε5vvƒg@†Šςk AΡ(ΠΠΉΊ &τ}(ώέπΘθπu³άr,!ύxͺΫ;厹)ΰLœ*»_ΏύqwέtΈΘ%£―1θΪ“Θ}H―Υ«]ΜU81SΓ· ΄Ϋ@¨%-Τ΅ Gεt‘Ÿ†£:±?žκ³ $υ1ΥH~"Ή›α3P#ίΞuz·ΑΛ »Ρ ° Ϊ@qςž2ΤΙ.ύ!(jμΥͺs v‰oΫ²M”sπΠέ½ σ?Cqύ»ψί ―B^#m@’^• FB¬Ϊžΐb՚ͺΥ# ΤφȌ4Τ½‹-lσΦq τŞuοT˜ TH6‘5ρ׈Ρ@Υ€1κζ© Ϋ~ΨκS7―ͺϊ]κΞTf8―ΥП΄ΫΖΣΏœν‘՘—IΪh Z4`Œ†jQk΅'ΆN6lυ©φ ΐdt Μ&Ε[ :ϋι-‘Oβ%νAq·Ω Τ £‘ΜͺΝ ϊ9{έ!,σd4¨AΥπ­£i` CsŠŽG€ί†5Δ}μ Η@τ…mA t—Ώœ}Φ&―7Ϋ hNNoΠc‡ 8Ά‚DιΗ΄DOφoμ“pLυRO…v Η΄+ C Q?­π'€ΚTΩƒ@»ΐŽ ύμΎDp]β*›tέ5Ύo6υH:ωFκžΒ6ΰWNΑjh_χN…)q5h@ρ9π¨Κ_ZΰψΜύώ’p „F0τΥ«A’γΟΒpPΊΊΦnƒςˆ:guΒ_BKΠ(ίvΚXi —ΐΏ@†―Œ•D9Χ’ΨΕ5Du³6›yz²Vΐ”;i˜AJ€ξdϋƒ†ΨG€δΨ >ωϋpZ~βtϋ~K|G|ϋ3Ϋ­a~‚pWwΰϋΑΝAOά‰ώ»ψEΒψ»©vϊΎlίίΞa»,uώ*—ά2‚’†e =τƒLπΛ'#θ (3ƒ;₯ΈKΚυ 4ƒΖΠ΄’«££:€5–jXƒwu¨ψ)j|JkΌΑxΤ- Tΰιμ„š]ΘΎ†κ%#bΏ…?ilžμϋ£iΈ<ξοϋiW°:U‰:ΏGAnP‚α}‡pΓι?ΧΔ?ΘvG83°tΆeη’ Gΐ­ΗzlβΛν84Š²3œΎμ†#htH?Αύyμo¬G*OPšwpΕ•I4|₯nι'X>6θ1Μέ̟]#5₯ όͺ₯HϊΣ©›ψ•bϋσζΥΝ*lVjέΝθ™©1d€ώj@Cκκ˜t¨έΏdάςΣ±(œ SA541 4:0 6B?˜cΰ(8ΈrA’‘w₯§»hέ΅{pδƒΚ’°Ί»ΧΘƒx τΏ;"Ž:TuΦ*σΑ—3pόΣί lυψδUΈ”n+xV‚ώΧΰq²z‚F^ΎE*›ς’[uX’ϋ@†C¨μ2ή₯νΛ)8N‚'AϊρeŽΓαT~TΆΰq6l_€τcΔhΐh ŽiΰYΚ{V+siΕ}ƒjΰ„P•ΈCRΗU’TζNWΖ@U€’yκ¦L£$eIIι–δο§Wš~¦¬ψ~:ώΆ<ϊΉ½ιϊι›m4P‰Qrέ”DYέ¦Ζ•ŠψˆBŠ«d™TŸύ+ΧD ŸάRͺTΡ;έA€ΥN.%Ν²U4Ο Ž:””~Iι–δο§Sš~¦¬ψ~:Ϊ_?‰ΖFΗNΖ΄ΐˆΡ€Ρ@Τ@Κ<Ώ–»€"χζ€ #!Τ@-ί!©£k;…P΅Ι¨’―Ÿ‹ILFD›zΫjΙQΖμrΨΊZR7‰    TJ‘JΕ2‘RQϊŸFΎJΕΒU’LGw¦σ*ΧD1000¨ £‘”ZKInEΎϊΨΣδZΚ?ΩΩj όφ Οώ.Nvβ&=£££££ϊ¬Νψ6D x†Ί,‚u!ͺ“©ŠΡ€Ρ€Ρ@Φ€y{’NŸΎΝ ―o6h’YXζhύ 6hρ*#FFFFFFIΦΐ+€Χ?ΙiΦVr2hυΟ0Ό R[:4ω   $Uf€!©κ¬υΔΒ΄ΎΎΰΩ6ΤΊVMŒŒŒŒŒB¨½qπIΘκuYΘκcͺc4`4`4Pg5ς‹;™a*vmεηη[ιιι‹dB    Τ ˜ΕJ;S΅ύE―ΚfŽ     Τ€j»O4sjςl›ΌŒŒŒŒŒκ°ŒΡP‡Ož)ΊΡ€Ρ€Ρ€Ρ€Ρ@Mjΐ 5©m“—Ρ€Ρ€Ρ€Ρ€Ρ@Φ€1κπΙ3E70000¨I £‘&΅mς20000¨Γ0FC>y¦θFFFFF5©c4Τ€ΆM^FFFFFuXΖh¨Γ'Οέhΐhΐhΐhΐh &5`Œ†šΤΆΙΛhΐhΐhΐhΐh kΐ uψδ™’ $QZ{ΌUΣ«ι€ΘPŸ†7b4PœτΙ„6QάAγW~ „ΙhθD΅υε 3ΩRNu¨ΑΌΉœaƒΑ^dgNΠ£ά—“¦κΥ+‰iEZ'“ήήψΝ/Ζ?ΥΌ’@}β…Ίƒ­HuQ™ΔBͺ“Ό(Iι©l=“”V²“QΛ2τUΤ1Π2Ω™»χCύŠ:υAΉ`ϋδ»ο/#‘fΏ’Œ0Ϋp|Maκαm)μϊ$8B:ŸΓ§pN’¬·Ι„ΙhπOβg8F€ΑF0Ž²d<*sAιβήΉ¬ΔλΠρΉ”υ”:PήK(c˜τ^•οK€β ΏŠ¦%ƒ[Γ;XCαο$ŸΜ2ς’Ρπ(άZFΈΪ>ό Π]o«Κ(TwŽΧ…dΥ¨΅Γ;’σn°=ό§ΦJ’ŒΓj4άΒωΉ t'ZΧ‚DCS#ΰmx ƒt蝑%< ²Lu‘=Σ` •5μΟ€Ή‚CΐΚwrόΨHΆmΐ—σpΌ O€ξ"‚²;cAυPέTΙPP:'ƒκ·TTΆ$‚ΚΥώΑΖZw|½Ar"ΌGα}9‡ξψTΆ› Ÿέ"9Χ« 4‚ŸξΔƒ‰ρcΕιύŽνgΑ‘ i €τ|θJ”ΏΚρ.< 8ΩΟ{Aρ§ΐπew…ΞǍ λHR’.{”ΦνΠEi¨œοΐσ λO²=όΤωIͺΧώπ(=]KΚWρ ϊά’Ž0 ξNh ’ΰΠ¨˜M”«ρΠ9σ₯]wͺoπ–”ώq„ τ|6tε'z‚DαTŽ‘π\ΎάCωϊ’ύΆ :6„`;H]σoΓhθΎθZκ[ψνα>²ΰ5^s»±Hoχ@sπe]§Oƒn|Ρυ%½ι¦’ώ“UϋGAεΈόr\{'Πυ)‘ϋΏ θ|κZπe(Ž© σΌΦϋ°―λIΗΞ_ΊβPέ”ηέΠ₯ ·ƒκϊτ_t>υQ|ιΞΏζΑύ7n€ο@ηFηύ1Ψ$‡~0T>•³89Ο—@ρΟΠ&xξZŽΙ©ςάι πΞ έjτVηuGhƒκπψe;χi ’Žι|λšΤu7#©¦ |Ρ«eχΰ‘„:|ΓώΊΈŸώty ‹( ^ 3a%l]κhΐ0fA>ψΞ"ω—Ÿ~‘gάq[ΕϋτgΫs 1μQxτgUoDeRΩ¦ƒ•Yϋϊ³ͺa] _ΓΏ`=θ‚—θ‚WšΏΐW  Ό$Ρρ“Š9Έ7~σγώο³=1ξnΐvlCAυΨφ‹»ϋ²•ήώ€ή°¨>7@’\€Η,8zƒt} HδV<ιη4Ξ•aHo;ΐ σ7Oε?$ϊƒΛXhΓαgh ‰ς1jΫ€κ¬kahΏΒy°+L„BIώ*«ς8v‡ΐ$¨œŠ+yΖ@K@ω©Tuή•FψφΞΞͺ’αΓe*&,‘bw/baΌŠ…-bv‘―‚έύΪAΨέ"&ˆ¨`‹*ΚR‚(Ά€|ΟsχŒw{]vώΏίsΟ̜sζΜό'οέ{aμΆύυ0 κΒ™`?XΌχk8 wΐPη€yζ“Vm"ίA‹$± GΛr,Xί‡Α~V\ώ½9oώϊΆ Ψ—_ϋαCPφ…ŸΑλ:ΑΫ`yΥ'°i&TψbάϋΧϋ»χΤ‡[αQθφ_ΛΊ2=@ τ₯ζq$Ό›ƒyιqKh ΣΰΠ7}jΑ^πX]α{σΛ „],½§XŽΦPœ\„ΜΓ~Ψ7ΉΑΊ}§€ν|) Υ ,‡}{I§AθΆ΅νσάk £ώ*λ< l›΅Αώs(NJυ΄μαvΘΦ…$τ―ΩΎΊ`ΩGΑ:`;½ηƒz ή…αt°ή·e{υ2Œ ΐςΩ‡W‡εa&(σΠίνΐϋ‡ΐ© >ΛaΩ,£€UˆγήΉ+V€ίαn8l‰p&Ψ–WΐG`ύϋήΑ°˜‡}ΘπQΰΌ°ΐU†5q—­J<° Ωΰs! κPώαL΅‘ Ψ!›COπz„z &gB…/NDβυΐkpΩ2_³“ψ ŽvΊe“ψaΝΗc]p - [€“Βη ,Λχ°„δ€ΰ}λΑΕIx5ŽκZψΌΦθuΫCI*Ν¦α2y<ΙΘ‰εΉ$μΐ9ςœθ<Χ~†S‘ 4‚\zŸΔ©ζ="‰γΨ9un"a}ΚΦC$τH/βxkφΰ„«OΝ`6x^‚εt²Θ–“m’ΆσŒδx0Ηg ΘΙΜk‹J?œsNΚy ν8:I-–σ2°Ν-—ΟΙKpSϋ›ύ©(= ύnΒζ%¨ΰΥL¨pCπHφ`]¬ƒu?€\Κ#ΡMC‹ΒΰαΨ6…γ ¨ό{sξ~z—@χ$βϊ! ŸΕρΞ$μaνώ aŸd|Γ$ς#Η Μ‚ /αaŽ§CP_nr²5’„=R‰ή ƒtϋΦ&ξb°< = θrΞ/φΫtsΘKΈ£>§-9ι}ΆYΐvW{‹U^BkŽΆKΨ―jgx3*|±Oδƒύk, jEp,*Λvδ%μΝq ¨ `½:‚χΪo²u% –· ,@­yαΫ!΄οK„υVΩίηB˜&ό(¨—aΏL¨πΕr:vg3 “2σο5„σlΗ0W~Aψ*θ–?Œ‚kB_%±¦-‹G₯―eBΌ˜·u=,_Π ½’ˆ>˜Ηͺ2¬‰σ₯\Ή:Θ|yΠBΜΤΞ7ά8lΧBKƒΠI"—μπΗΒoΰΰUE][xφί―mHšί&§Β3σˆ;ΨΫΑhha4#l'w‚Qa€^Ω4ΆππχλŠ‡*ο‹™NΘN–υ θΚ β@ΨΝH"ο/π8Ξƒ/Α…πHΛ6q² * Π"D8ΏMϊJγ»mτ'v=™ΧCZζ™­φ$τΗΔ;P+ΑΕj2M% !WzWmχΰΑΜ$ίΨ@"σό œΣ νm™•œ˜ΝΡΎΈ;L=΅lΩΚφΤϋ,_π5LšΩχ5!Αv ²l“B„γΧ ;qL·Yvώί€ξρ\hCύO—w|κ:ΛdΉs)}O8οΒhϊ₯!!9†qcτgh™€gB™L·˜—Ο/€ ΫΕ1«oz1‚,»ύΔρΉ\i½ŽΆŸvΞqn%|fΊί|Hά…0-Λ”ξwΦiθ>όΚ1P7*―k^?‰{°©]αLxΜ«7<iGδ\Έ,γ8\ o€uaΜη ϋ±²?¨ο ™1©χA!ΐQΏΧNΕ κ͐NΟιFΊe~ΜΏp/'Ϋ>ΜΛžuqq멊λמOΧΓψ"―Π©ՊξHΕZΓΐ€‚·qό άρ:Θ ΐIBΩαCpp9H.;α©pxMY4ž‹·ŸηZΤ8paέNoASPN¦B#ψV‡ Ο©06Ο ϊ=*xό…ϋŸ‚#ΐA»'(Νλΰ>#hUp-υΑz-'Γ•ΰ€’–χΫ.£“DΓNQΆ™F†n8œX~H2χD˜p’€Μ€δ„h[Œο “Š‹ζζdYΘ•nέ¬ΧΆ κΐ^0ΑH’P.Σνͺ3x‹DΊ.‡_V_Αςί ΩςΉ[§Ϋ+Cπ5gκ²L=m· λΚnZK8ŠΛίΎQTώœšG+₯b­ _‡‹%ηjqtΜ…Ό§' ϋsό* oΙΡ²Y—ΰiH ǐOˆ{τ^ΗPmί τmbζQؘ›l8 @Ω L&± /–ctNέӝπ§°N*Νφ±]ƒšpΡΌ<$δ8šχσpqr΍‡›Ηh[Ψμ/ϋΑΝπ€e»Nƒφπ"< G‚ciΠΫ !xDποΉΥpqJχ‰6\Xu±ε ΞK—γθ\jYφ…:°78Ηί Ε)έΜ»uΦΕΖmvΎ6벚u€,jκA…ΖΓxlό‚r'.Ψwƒ’1(Ύ‡αpΫΞ―W.βΉδDο3ΣlA܍†{άΧΓ(x,‹:n'ސ„‚Εk]τ‚ϊ˜ Β%ΰΎ ,oυRη€αŽFr(Ÿ΄ƒSl˜γŸg>Α/qbΉΆΰpͺ^88|ΩΊ•½Ω Ά'3ΣΚ’_ΉΈ t(ζ&g7=–iMpRyœ,ӚEDV†Uΰ:hK‚“£ΤΡΠ lCλYTΊ>ω,ϋŒe»2 Ι1h=³\ϊΆx_}ΘΦo$X––ΰδ}>X.₯m`GπώΝ‘ψό[ΐΎ6ŠΣWœτ毞5ΰXυ­MΈΌωsλ<²ot‡MΑώ{3(Λy$θυeφΒzξυΰΈ ց]αQhA«ψ DJq|œk|ζI o7Βx0Λζψο »ΐα ~ƒώpΈ!ήƒ& ϊ@Ύ2Hί›χΪolgq.²ώŽ§έΰXNΫηZΘξΟ$Ν#ΗΦ‰Π ¬£:vμ“W€u·ύ[Α§-ϋ—ύΙη­‹Γ— Λ€χιAO}“`©Υ—+7…ξ 8¦u‘γ`O°o:woŽ[η½° yπ”EΟq±u? τζJ°^Γ!*‡ur€U©€zυκυ5kΦy₯(”JS`ηιΗΐΧ ^ƒζ°άγΰg0ύ πά\°S~­‘!Ψ‘jΑηΰ=i9€'AA/ΟƒΛιA8~wAΩQGƒƒRΩYG‚ƒ² L†σa Ÿ?œDV'½^πψλσ4ό Kιριi­@ΔI*/Ε„?'lΛ―&Βςp3|κψ ¬Λ¦Πnƒίaϋ~ –!­wˆψ¬#ΑIχrxԊ0~2‚²γ…©…ϋDœP§ΐT°νΥ`›z‘>«œAZζa{;ΐ£π:θYΘcΒ»‚Ύ\ήcήΉŸ έkχΛv ΜΛεF&δiZ.Ϋβdxƒ&π(ΛΊxνέ;‰ηh±NΐEύ)Ψ œ\?'Ϊ?Αϋm;Σ²eoυΰ}° ΝgoΨ l‹K@/ŠΛηG²>#Αg:Η4‡ga ψ¬wWΈ₯ί›Βξ0Ό?τ“―o–Ώ,G€ Εiΰ΅j1°mNλ‘–χŒŸ―Bά±k{ω\λό)Ψ^ΐ8Π㣑-άΆ§υΛ‚εpœž o€Κ‡ Pi-N€> N'&aΫΙ~· ߁mώ#θccXƒΗΑΆ±­υ¨(ϋΒ3™Pα§eΦρiοBO0ŸόΟƒuχ™Ώ@/ψrL―Fšy8·IDAT‡C'8μ«Φw0έΌϊ‚mX–ΐώ^lσ'a.4ύ΅έ|}Γ2θγq0μ7ˁεΧΛ·‘μζ>„ςλEψ Ž…μςΧ#m°ζ|!˜ϊΎ#μ“ΐϊό `&θ±²NφAηdΥ žΛ±ΐT†5q•©J=haι£J™ ˜?Έi6²ž'Χ³ˆ];OJεF\τ――ά,cnσΩ7α»Μηg,RΩ/μ5Ρ`Tt :P³Kυ}'η»Πκ*ίAηUΧ ΔrG’•ΰΐΒήUUBbсκΰ@ψ·:”5Wλ’Ψ4Χ‰˜X”XΨkbό€aQκM±.сς;0‹[ύ;zuΥl >£Ί>–;:P]ˆ›†κR±œΡθ@t :XΘ,j›†Ό”ŸΧnžŠWΥΰlε ΏdύVοόίxφc_u)΄Μ„μ‹ίΖΎ²”τΫΦ~C]]m3‘κχβϊφ¨€bΫ?ξΏΉΎt9στW ΚyoEnΫ„›-·}―ΌrŽ ύvΒ=Λ›Q5»―7ε]»’ΚΌωψ‹ωω%ΦPΤZόU‚ςΧ6ώr₯’Z’ nηΐ_{”GKpS›ςάΈ¨έ³(mό)“?‡ κNΐŸΜTeYΎ^πm9 ι—Ώ>9εΌΏ€ΫΖpAX„ύI” Π‚ΦRtΧ…6ί•puΨ4ζͺZΛu’iώl/‘<ς'gn΄,σp_|!χϊS:Υ6Λ„ύ—TΡMeθr2ωTFf%δq6ηMY…γV%\_šΣ{rΡΊΰ΄ ₯Ή!Η5ƒH ›™§kN’΄ipιŽ2\L•_ξΪ ό nXόfδδ±/”τ.ΩœpšΐZ :‚οάς‘¨UΑΝΚ–ΰŽYytΗμο¬Νǁ΄Ρΰ.Φ…q5(JyœπθζΙ.*ϋƒum€Ωžž7==a4%ή|ΗυPn²τfgΠ§l9Θ‚e ˜ΟΪvΛ¬ΌΖw4«ΓvΌΧ'λ<jC ”Υ4γyY}`EXτ;Θv°ώ‡„¬£ν’Η›‚ύ@ιΗΆ°+„4‚ιύ`Š“ύΓI'Μ/ΘzΫ―lSύ ²θ³uρ>}Φί 'Ažoϊ«oιΌΧ'ώ6¨Εaθ–„9dԎWοίμϋΉd9Φ€ΙπόΆ₯εήτ+-½Πέ lΎ:ΆŸλ{Π77rAΖm'Λi9Ϊ‚ύ*τ)Ϋε@°Όiyνι8¬•>‘„ν~ΚdΩΏMμӎΩ|~ωά Α²ο Φ/¨}φœωlΗH:Νsφ;σο ΅!ΘΆΆNϊ’ν™ώΆ{Q*ξ~Λ~0δCxζς„-›ε γ`fM{λu^Σœ?Œ%―ρ9ϊ‘Vο΅o†ηό[ΆŸΎ‡v0―φύR>Χrv: <ΪgμG^k?UyΰΈχό[Ž#Λιβœξ/yΔm―U ­e‰Xφm <7}ή°σžν6 &Αt°όŽ'Ϋ9[ΦΟgΩ‚£Ξ+ΦΕΎš=ΒΨpNτœυπ™Ζ-—Ο±ΞΩ}‡€¨Jw  ί½”‡;)φk|˜Δ‡se#>CαJ(;u.ΩYΎ€«a< ·z€Α°/\οΓωπάͺ=| Γ₯`§έΤi`Ÿ…Λα;pΠdk{Μ»XŽ›‘%Όcα °C[Η‡α"˜ω N€ηΰ-ΈΦλΈ L―iMd.ά KΓgπ τ‡!πθ₯ƒδKζγΐoΒυ`9eυ)Θxί$r<Ηπ=™ ϊ―?~λ½εNηA4#½œ χCsψ,Σx,{¨£}d$\ŸC(ΑyδDα}ϊώ„6]‹°uΦΏα+hΚώπ"ά“„_ΰhΏ±―M„ξ ^ΧΑt‚j 32‘ΒΎ£ο7ƒν¦/ @=Ά΅ε:rigG}ηhΦGΜsθ―:μK—ΐ³0lKΫεwπY+Γ{AΖm§V`[ φΛfπ?πόπ)ΨΥ>`½Ξ‡Wα ΘΦΉ$όΦskΠ·ιp= ΗA}pA› ΦΣηZΗ δΡ`ϊ^Π ¬³χή ίΓΆ œΤCώ–i84€¦P>χ&πYΦU]Žλ7Nƒl-CBx0 ς 6<φΝKΰ#}ΐώh[[Ž{α;°? „7`¨#@_‡€eψl υΨώκ20―яޠvqpΨoνKΩ:„oα9θ ^cΓc`WΠ·ΐs# Q‚εφΊΫ@ίϊƒύήzθ}KHkm"–SO]Έ{ΒD°-nσϋ(λiΥWϋΟ›`ΘΦI$ό O‚~tƒρp)<@Υ‚ϋaœ–γ|¨z£·φg5ς  %ΐρΡΆΛay‡Φπ> „kΐΉf9¨Κ°&Vθ9Υφζ2ԘJΞIUΤΑΉw·a…Α k$ΨIΤZ`‡qΣ”χΨ)Χ3‚μίΓα°8όςά­ΙΡxرԁΰΐ ϊΒƒzƒπμ‘„w„lΩY/J—ζxKv ’„<‘S›ΤΒδβ„σ,ΤŸεη„δΒ愐–ΧΝ…I’“όaIΨ|Ύ ?³‘¨}a$Τ6‚ΪΓ/Π.‡ !Θx_π?A˜D–%μ@wbθ³`yP[Α§™ΠΏ_~#Ι{•ƒώΈL¨°,S―lΫ£¨•ΰG¨o$K–5xjΈ¬»“υ€·φ+=³έε$ε„ΌιGΈθΟμͺόΝΑ>ϋ<(' Πnj?ϋ@φΩΣ3‘β_‚Η^uΨO‚Ž"πxΉ’cΫp‚γ—°Y·­’π{σ“°γΆ“ηηΒj 6†I oͺXη₯a œ Κ±dΦ“lΩΞλ‚禃yΩgOΫΟηΪOrΙϊž™œΠϋχΑvRΐM`ώΣ Τ—`fμœΚΡvΆŽ‹²λΣ:ΰ=a|,Cψ°žimGΔEΠzͺ} ¬7@(K{Βή―œc£αά›„/eώsΐ2Ϋ?: Μ{h&τΟ¦‘q½k˜€/Ερ{p<έ ƒ2Ϋ!ΤΣ΄ w l™D^βxY8ΑqtIş lΫΪ§mΫO_@hη λeΆ.%αό$±'Η±κg;φυμž Ύ86|FΆ–$αw°<Ξ Ξ[«‚2ίI°2δc ”oΒ#@ΩodB…mχuφ°1ΨVκ °ΌΞuκaύάΈω—fΜzm‘*ÚXd9αΈ(ΛΑͺμ4ΞF°"ΌA-˜69$pά&‚FΝ„0ΦΟ 5 |֍Πμ„΅!ά ή―μ΄υ3‘s μΰͺkΐ(ΘΦ$< [Γωp¨υαξL¨pg\$Θ”ι.>³ΑN„A—GΨΙΌ$?ΝΗzθη_ a ­Kψε$Cf9iu4R„<ηύζ£Ύ…3‘ΒΟMMβ}niΚkΝ#΄ΏƒΪE'h NΪ.iιν³`ϊ…p(¨`8μ;›A_h ΅ΑφVkΓ#Ό1~¬φ₯ΗAύ φΡ%ΑΆzΌΖΎ::ƒ§Χ %,ΧBYd~ΦύΥδ¦Ζ]„ΤEpΨ·ΧλUΚ’ŸΈψΣδλQ%q.HΰNxzΐsΠl£’d}m£·R &Ό81 £‘4²_9ήΤW`ގύF0‚Μ_ΏnίΐώnŸ±άγαp° ž zτο₯ΐΡ<ιπ"„ϋηΎ ϊ@‡„΄ίφ±PΞοκG°…k‡6/ε³nΟ„ώy±ϊώτ?I™ ³εμOΑ>`;υƒ?‘$Ω?Uspσσͺ‘DϊζάΖ±u ‘­‡~•€Q\κg{9ηΩ―μG§A/P­ΐ1”­uHψcέ““½“£Ϋpi(ΫψNXZΓ{ 6€™Paxdφ°!„ΈΧ]ΦΝqΆΨVϊͺ:Β+™P5~ ―W‘Ψ’‡Ξ–ΎΘΟΑΫ7HxZVάΑ09•f§lΓQžœΌ—…‹`"œ.Pʎδσ‚6"ΰ@h–:γ*Δ€IΦRDΜΣσϋƒ;ήcαp²±cϋ,―A>Η‰Τα³BΗΆΓ‘/(7+`œ‘”ΛOoIO4³ˆ;Α§εΐό-IpyέLp°7‰Ι1/κΉY·ό+šλ>Ϋ_Ϗ˺zjVΌ1ρΠφ‚λaΠΗGαHψΤg`xΠ[Ÿ‘lϋG2‘Β Ah―=Iϋ4Iχΰ3,«ύΝ{ΫΙ<ρ ΫΨvΪ†BΪw’%Κrέ .A>·.X―—ΐΊΓ (e·aΈΞΆςYΐ‘!!9Nγh{―[ΐ.π¬ Φ-—Μ·ψά°θΨ§Β"‘~.ΙΕΚϊfΛϋλ$„σ!_Hί l»ΰpA™Γ!ΦΧια_asχ_DΈ 8~‡€ρ«ΐEνmΏ΄BΉi†-wΠ’Β8 iζ3²Λiω{+ε³nφ©Mΰ#(N‘ίy΄Mlη.₯o‘ s §Λ{mi”λϋ€yŸΦ#H―³εx Ύ:¦ΫPYΦΛαsx>+`τ‡ Μγ™Pα¦edφ°- Nβ^wZ^‘γ 8%‰{Ȟ£S§ͺO0=ψ«O©s—ΤΑQ Λ}ϊοTwz[ƒƒ΄VΑN˜–‚Θ…’άΨΞρ­Cΰnp°ymxLξθΧe§ν·Γϊΰ‚oΉΥ†:w&!y9ˆγ½ΰνƒμΪ0~_α0•ΗΐΝΰsށ0Ω:@V…o‘\θ'΄τΓΊ.‘N,EψUιM“kwδθ’τ ό«ͺ]2‘ΒsΦAo”Ρ:–UzYRymMΑΊΐπ&XΖ΄¬Γ“ΰuχΓ@°--Χ‡π L‚³aΨ>Ωmh? mΊ.a½·ž¦Ϋ‹ƒ:nΙ„ ?Ξυžq‘ŸκΥξI™McΎdέm7'`ίΊšA[θ ƒ`p\Τ•φυ{β«eR B―Λ₯!$ΪχΜ£\Π,³νσμ/ΑΙ0Z@QšΒ 'ς’ q΄―ΨΧJRiϊπ42ωτC5„Ώύΰnζ`(8­OΤƒ‘ΰ½iνJΔz†ϋο#μύλΑh0OοΫ²ϋ I%ͺ+WΈ©ƒαωLθŸ—Χv‚ϊPΞω>OϋΑQ §,₯•γΩ±pHrƒΟΨJΣ.Ι-:Μ&EO‹“›Ε7`;(€‰pμΩr¬…18Žπκΰυφ§3ΰHπ™λΐm`›΄ƒΑ9\ŸŒ[O帝” ^c»™ΏΎ-γAy}Α6-€φπόΥZει€U΅Β.zοΓTp‚.Jƒ81μc!'—΄^'βunLvT›Ϊ.Λ„ _nβ`‡3Ώοΐiηp±σ8±‡εαhψφ‡Π™ ώkΑ1Mέ »€φ'°c:8μ€+ΓƒΠ'€X œŒ-οYΙ‘CFορϊ| ϊoΑžKΓHt‘Ϋ<ΧΙ"^$ύ^Π_Π²°Μ‚{@―?€Ϊΰb«ώ‚½aάoΑΘn’Š•mfž]ŠΉΚ‰ΰΠ‡O ΏBZΩτIš@7° ώ Ά§uϊ ΒDΰu (ΫΐIτS#Θs‘½ ›Ώq}Џσ #Έ(Ou-xξk°ωόΉΰύA™uΠΛ’t)'lƒq0μKζ9ξ½™>λ50ίW`Ψ—vΛ4‚Ω`z.ΩώgΒ›ΰ‚ΩN…ΰrx…f0ή€βt'νλǁγH8™―u“ŠΉΠό‚c XΎ»AŽ‚±0μŽGϋφI0lγ6`}Ώ΄ž'β˜χ/EΨϋ†S`θ£ύρWheΡ/\μxω 쇻AZ–Σώm;~­αΏ`ŸΊž}`° /CYt0? ‡mωά  <².ζ·$8Ÿ₯žœx φΫφ°½²εXΉ>I΄Ώ8¦ν›uΰu8Τu0ΎL˜ΖρO¨ ?‚ν΄.؞Γπ,θ‡°-Œ„ Ϋχdp@CpόόQσӁr|ιΓΖ)ΌΈΪ…σ«€2²wMΕ³ƒHp²IΛΙζή$ΑΙ­vϊdΓκ²ξ± 'Έ<¨Š96εœy–€ϊ™χεe'·l¦/Aΰ#Α£r ~ m”Q₯-―ΎεAImβ$Ώd+„zΩ‰₯ˆ[o'ψ₯‘1Έ'―[ΆΈ ’sύJqMΈΔΆ_.DRGΣ›§βι`ΪW'ι–ι“Ε„—ΰ\„Ά —†~`ύΚ"Η¬c­,²Ό>―4²^ΆKΆτΕMzΆΒ˜³Ο'Τ\χΫ·ΚZŸπ{ΐΎ\ζΙ~—ΉΚ©§Ά}E€o%υεζo}²ϋKQχZoΗhYdίOχηp―σa.lγtݏΉϊHΘ'}Μ΅.€Ο—9\Ž5±ΜΟ¨Φ7,Dƒ.ΖΈ7 ά―CI —Μ£»ˆ4OJŒε€ο0τω`xΑ’¨υ©”ŸNT¦œμφ©Μ c^Ձ°i¨……¬βšX9˜ίΉ,DƒάιϊQ_θξΤΛͺܐλ]jYσ© Χϋqζ‘ ίByήΕs[•WkJΈM•/e,`up`U ΉEu(h,cε9°ΧΔΚ«ΔόΜ)4?ݍyG’сθ@ur`a―‰eύΈ½:yΛˆD’сθ@%:PΣ6 ₯ύ2T%Z\¬όf/M•+³xSt ŒψΕ―Ν‹ΈΗ?·­VΔΉ˜\1ΦδvΏ0YΥUUΛ©wώBΑ?ΫDΥT*ρ£˜~xΨͺšψψ?Κ~ ΄0Šό υη•₯Θ¨Ό?Αͺ¬2Δ|Jο€ί„±ˆΛO$έ/ψ–WΧrc»2άμΟ+o/Γυ λR]Q_.㾎I‘ύςοIΈ*ž§p{WbσΙλΤ ζηwΠ ΰ9ΨYUβšX.jΚ' NR;Β„rΉT³n²OTζ„ΰ§;~‘΄Όr‚:Β§Dώ$Κ/Kκ 0ξβRω.iAJ?{Γ‹9κ/,» %ύlŽKώ–>”εϊΏo¬`ΐEΏgςΨ‹{Λ2ο΄αϊ *πΌuk+΄q9Ά'χΤ-Η}‹-~™³΄γ·¨z;–?ix¬¨‹bz p Œ»*'ΡΑπn‚±L…ί!ό› ώϋ oΓ§p?„ίIοOψ"x†‚ΏΎΖΒΗp”€ΝΈΰu˜ŸΓι ΦŸ5ΎŸ&CίΥϋ.exοσξ#ψ·κΊΎΛ~(Ωαdξύp1ΈσVΦΓw‡γ`2„½ΏŸοώόo œκQ˜ Π†Α…0v†7a2ξΔv= Ÿεΐυ§€ώ|2δΧ‚πaπ ψά'Α^œΞΞq2;ΝE·΄*Λ΅₯Ν³4Χεzν~Rrs3ŽΆΑzIΌ€ΓF\°CI•pήχ㠘φo7`ͺ%ό 7Β°CΉzΎΤp/xmzεx μ ŽIσόΜϋαZΰ;ΒQ`{ύŠhFSxύ^/Œf~yβΨ±Μ―ΒIzϊ`ΉήΛοΨ ƒr¬\ŽσxZƒϊ|–ηehͺ¨τΒ³…―γ9ό #’ΔΞIΨg …΅“ττα*"³ΐ:nƒΰ6° ΞWC-Pϊ―ΐ1©ΉΤΔwα+ ϋCΠρ¬›m’7uA•~$ηΌΎϊACPΞO{gB…υ²~–k8¬›€ΫΆ·Β`;9] ζ§'ΦA7οαGπSu2|ϊ2–e~—€ι§CΠβ&Β°Vσ΅ψ,‘ύmΛΫα=° ΥNe\«]ύ*\ΰ2δ@8!yhWŽ.LκZ87*\ψί!άjƒςJPX'ΏΝ`yxœθκΑ²ΰΐhEΙAh'έ6Ή #G'Iο΅vZuΌ” NΒ.Φ>§| ιAA4£3y}ΐ`ύΊΐ₯π8¨›ΐkπ_P_ΐX œ`λζdς,聃ΙΞΑf|.Έ )α=°,“au2ξΉvΰD·&¨λΰN¨ζg™ΝΣΕHο•“¦Dq:;ΗΙμ΄>©kΜϋ"Έτ%]ΦΔ_―—C!­Γ‰x–:al£‘=\>£¨•αb°™ΏΩJ—1œΣ“B„£Ο }±aΫΠϋΞƒ!¨ΫαuY.œδΈs’nŸ? κA. #Ρςϊ\Ϋp t†–0N΅'Lσ1ν.Pƒε΅ΟθΑ—ΰδl?±/䃲ί>Ÿ >Γ~¦¦ƒύ]ϊš­ΝHψ I\‰£ ΜVIό@Ž_>¦εx³ό§%‰zg?] zΑ‹°Τ…{αZP^³V&TΨ.W'α’“Σ™ΓΌŽNl‹οΐ±©φiϊ‹iA_cfα‘ΰΨΆ{υo SaPm%Œ€Τ‚π<΅X΅| +€>Ψφ½’υ,\oϋήφ9ε½{C°ΟμΚϊ·=l›™–ηž3ΏΫΰNPηΐU™PaΎn,lkΫΗg>ΚώφΨWν£iιΉs‹Z Ύ…P.}˜φ‹}α7Ψ:B΅SΧΔjWΏ ΈŒ9I8χ†NΩY»fB…»μM’°‡€U½{eB…οz~'μΝOΒ±§Ε9ι ίŽNkπ™–-ΘΙΰΓ$βΰ4t ΣC$utΐ„z˜ά¬£χoA;yϋ̍à Ž°δƒΜΕ°%ΈH)΄ƒ[‡-2‘Β'ΠΥ³ββγαξTΊZ¨ζηS&Αƒ°4†’tvŽ ,σΑ)Β„γ₯7ΟRΦΙsιητρDυ$-?•ή…π©ΈAΛν"lΎΐϊΘ­όj˜Δ½&­\Ουž“ΞU_/Ή0뺍ˆΫ7KRG.Ψ/ΗEΝHϋμ―AΠ?'δ`Ϋُ|ζ‰pXφΩΰ†"?α!Ž—i―@Ή‘PS`ΥL¨πέήs„Λ“KŽ‹’‡s|"뒏‰oŸ•ζβΰ’nƒOˆ;&Υ’ /»ΒΣΠ”γk0τ€ζTTz8οq$Ψ7­WZοΩ%„υuΝ$<ˆ£χί vƒqŸ’ s˜GΆ§eίzΓP·Α™Pα‹cBŠJΘΉ~Ÿ`}κyΨv‚ΘOa9·ƒαUHΛ²u}xu\• ~{|φ°Μη7 ½ —Ό.lΊ~=λ’!Δχ‡}aT[•qM¬τzΊΰ,JΊ‚Κ| –s‘ Lƒ΅αmXV‚΄2oΐΤœD”δΙΡΈrŒΚ„rΏ8Ιή Αη 'Φ0ωώJ8θ/!έΕυ‡p‚£»δ\ZŠΔτu3’‹Ό? h“Lwρ šη€Ο»ΐp7¨06šχεϋy£σΔj'1½M—νwβκθeώMΰ08œ”N› ,ϊƒ‹‡¦nX% ·ζΨ NMΣ³­ ΄kκΤ<Α|bφ― 2―{B„£ύΑ ‹ϊΉπΩ=K8xό αamxΚͺΉΙ ŽΝ 9ό.xzmΏ)I[rΑnπ#8QO„lι‹eΥΛ ϋ(λΚbόhd ‘}Ξς¬‘LŸ~œΐΣ}ΖvOχO’mΛλαp(ά —C(J>3;γιΎξ΅ΌsB„£6†Νΐ6οƒΧΩ-ΟΑΰ8ΎzCQιœΚ©²”3;Ϋ$ΘΉCΓ†k«p‚c?ψ:7θFΞ…Έ-ΈΡω‚Ÿι6ΡU\ΊηΣΟΌŽxz<[ΉYΧ8—|- έVλ¦Β{ΰσυ)[K‘ΎΟ>ιXpΣ u(LωχkQώ7ΰRϋu:ίSŠuΐΖXTδΔc§ΈMθΑρippΉ7†:`t’7άn€u`ΈΠ)'Sο9ΟZφwEi]NμΛΓo° \όβ4†“kƒ“ς]ƒ>[.ZΰΝδΔeGΒηΰ³Gƒ2lZqrΒ^ŽH.r@w‡ΎΰDP”\&'υsΩ$¬/ωIΨƒ‹άQp„όV | τMΨƒγ…p”EN¦©,“rBaa7­΄r’»Β˜λ>ϋDΆœδWΝJtβ3½¬rBn”άΤ‹£›‘±IΌ€>”\–Yτέ4œœ$8yw 'SΗI„mΦ0>I·‡ηΩ‡Γ‚b™ΪύΤώ§¦ΓO`νƒκψ¬ϋκ`}lϋΆ`oŸΔ9dκiΟ†Kac } ­ΠwLϋ €―γΨ±bzΆ–#ΑΎιΆž«γΓrά >SΉ1XκΓι`y.‡uΑΎdŸΘ•ή›τ΄²Λy$'C9#¬oΉΚ™Ύ/_:ό9λ`™ϋΒy ΧiνBΔM_°۞–A…Ή£0VΨVŽΗ’}¦νξsT{Ψε΅>ϊ¦_ΏΟκ ?@ΆN#αAΠ_εΈ·)σ ε4O½Ώ”u˜‘ιη{>—,ϋšPœ+κΐZpΨ/’*ΰ€¦.*rΡzμΌ3ΑNr=΄λΉ;<Βxφ…©pτ‚‘4˜ΐΥΰ€αΔs"j8φ4’$Βvώc`2?A(Nηs24…VΠή‚l]AΒύΠ–Λο€χ+X'ˆΊpt‡βδ=ΑηΝλδ rP~>λ\ΘΦλ$\>Οgx­zœn†w@Ββύaσ»fπ!Xώη‘,rςt²1oΐ…a)¨NVs  Ό N»Β κOXlw'Ν– Χϊoωl‹ί‘ l χΚŸγQ&€ϊλ<Φ‚Υΰ(IΦcΘ–Cΰ^PΆ―Ύ©Ν‘ά _²ό¦λŸχΊ |Ό/΄‰ύ©7θS=°ƒ¬£νψ4Ψξ`38―}¬O°\S HŸΟƒϋΑΆmΗΒ¦0|¦χ†£Αq§fΐ9`ίΚ‡~0v―Ν–Χ·ƒ³ΐηΨΗ,Χ 8^Q­Ϊ$<ΰpμOϋŽπ)ΨV»€m蜱 τ‡—a° ϊ­W-‘€eV†πΨ/νοMΰ° φιp&μ‘„‹KΏŒk>‡“` €5‚ΘΫ`νzΆ \Ωχΐqφ†Ι μηΑ#ϋβpψ <XΟΉPZ½Α…Ÿε²―μ _ΒPθQp Vξ] ·ϊχ›™3g–Άœ«S(':πΌKΒΰ„ρ8Τ…ύ 5|OΓΨ~„±΄ƒΐλ€σ(ζmΠΥH–,C7ψ†6`ηw°n O‚rΡKΗΧ"ΎL'AEΆΦ Α:š=π5¨5agp²p²Κ‰απω*ߐψφΰ=Α/‚™ΕrkŽ·Γ–π28 *ύ<š‚~Ά‚aπ4'hΟ½NFjc0Ώ;Œ ―icΐ²ϊό’t6'œ„ƒτw}°­žλμσ>€A™ΆΜλξ$Τ€@(Β>?Τ-°}cqΑLXΆ…΄œ‚§φ©ƒΐΆώlϋTZ}ˆœ—N ¬χφ1eYŸ‡FP}°œKΑG0:Α½δύzα³ΌW”ιkƒe,ϋk`²΅ ›€·y{Τ#&ϋ­G#½°N‚5 '¨-‘3ό χΓPKΐ wNβφevŸ₯φ…ea˜6 ²΅ +ΓΥ`σσ ΦΟr₯e~Ž#ΛΆ#|‚ΧΥΛe~£ΐΎ›ŽΥΖ`ގMο·„’95v'ΦŸc^+ώ™ΘVφ½σ^ϋ^¨.π%Ψnζη‚Χ,Ϋ}π;dkSΆ½υšΝ`LΛ²/Ψ_‡± ŠJ_ŽsϋA#°Ό Κό-gX½‘L{ΑΉΛrΆ„pΟ4/λθΈ μ[–ΡΎr8LΛ΅"XNϋ‘ύΪς«Ξ 7z’­%IΠŸ―σΘ―ΧίΩΰσε¨–*γšX-λX‘B/μ/}Qx;έeEœ‹Ι•λΐƒdwΤͺάlXn΅“ς»hTw9ρ_WVρŠΈipኊ,rTΡ5±κψ\E ͺSuŠ%‰,0nγIŸ€ŸbTe5‘pƒ«rcΩ’εu Š‰ε­Nεί ͺ|OcŽΡθ@t :P=XΨk’FE’сθ@t :(сΈi(Ρ’xAt :ˆD’:°¨mzS'Ώi|•«"ς[φ~;θf~Σψ1πά•‘£Θ€gedT†<ΦδZΖˆD’сͺα@~σ+₯ήόωMUΡw$lό₯?σRώj΅L¨β/WΕΟ¦L9lΙΥο–ιŽxqt :ˆTȁ2‰zV›ύ Υ’"“λfa˜€Ώvaώξ7ώφxuh+ΐtx~χMΐί€« ΰwp‘ί*ΑŸώ g¬ΊƒΏYήξ p˜Χs–ω½“JΘ#μo΄ύyXψ”χZφ†ΰζΒ²Ο~ηώa{\i5#βoΣ-£Χ>jS𞷌 Σό2ψμ}ΐŸ‘κΩ£`}ΜG/—‚–pCξΑQ}QxψΧkSRό·|–Ο–ΣΈ2­ ψ)ΧΫ<²]„ϊ GΟ‚**έs;ΟšΑ6TΓφ`}¬XˆŠD’с :°¨ύy"mΗ"Gƒ‹Έa°΄A‘ΐΕΩcWPΓΰ‚©ΑCαFψ όδΐkΆu=xnih.π{‚ΟυώF΄ηC„γ΅ΰ†ΖΫE~ p#3άΠΐώp¨|x~„?α X ‚,«η}ζt°,αω>Λς(Η~ ό €υ/Σί‚%Α?‘x!Π ‚‹Ό›70WA.Yύ9>Γ ‡ΝΐwΑA¦)Ίρΰ¨Ύ…Ι™Π?›…ο’Έ‡Saί•»Ω)€ζπ<ŒƒK ΄“ω{]Π}>N"Ύ3wN—λ)βہυyή„-`gπcϋΏΰ}θΦq,αy jUH?ΫϋŠNΈ¨+=ψ άXΦξπ|]ΐςzώθ^Μ…’υ©Όϊ-ƒe||Ύeύvt? ˆD’εq ,εΉ·*ίγ»ε¦YtA Ά I\¨Ž†aπ2ΈΘ†…•`ζί­―ΓqpαιΕΩΕW™ϊΉ^ΣΘh[x1ϊη%}m3’}‡~TBoŽναΟϊ™°Κ'°VΉ™γp)΄JΒ¦ΑwπΦKv€-`0¬/ΐ#°9¬ a“@pž+?»nMΌ ₯Λθβξ΅Φm`vγΰΒοΖΘΊΉQyV½wΓp7•ξfβp“Π:Απ5ΰ§<'Γ2p<μ ½ *:ˆD*θΐ’Ίix_\Œά(Δνΰ#9δβz&ψnάOά€ωεˆϋ.}Έ°ξ‹AΆ> α8,9ασΓΖΐΕΪE2-7uΑςm –―ψΞΩΌ\ƒπ¬' Ÿ ΑO#nƒ°x»Qz?I»ƒ£υ·Όn4vƒ‘ΰ¦αPπγ{ί}/ .Оσ~ΧFžGπoω~θK€e/J^Σ-9ιζG旅>Ž„ ΰ5»‚Οj Φ·CrΑΡΝKQι_pΞOϊ€υ\>„Nΰ†δ!¨―‚ΈΑP{AΛL(ΎD’сθΐ’η@~γ’_?qΑEΙwφ.PoΑv Φ†[2‘^\l\`\°Υιpy&Tψβ‚οΗέ.~χΒ±p¨ΗaωL¨πΕΜr€‹ϊƒΰ? \Θ‚#p.ΈΎ]@y­› 7–Ι…nΈΈϊύ*ψ\4uτΜ„ τ'o™Δδθ&"θI.ΦAψ †ƒεΉΆύ»²,ƒŽι Φ/[λ“P%GŸόιLψ#ΠG7!–ν|Pΐh°½^‚Υ@•ή’sΦ³Όο PnΒnσ7‚ή)ΫbσL(ΎD’сjθ@ΧΔjXΓ 9TAόνnάΠDE’сθ@%;°°ΧΔEυΟ•άL1»θ@t :ˆDͺΌ {WUε ͺzτ;V½bΕE’сκο@\KhΓhP ΕΣсθ@t :PcXΨkbόσDιj±’сθ@t :¨˜qΣP1βέсθ@t :¨1ΔMCiκXΡθ@t :ˆT́Έi¨˜ρξθ@t :ˆΤ⦑Ζ4u¬ht :ˆD*ζ@ά4TΜΏxwt :ˆDjŒqΣPcš:V4:ˆD’s n*ζ_Ό;:ˆD’5ƁΈi¨1M+ˆD’сŠ97 σ/ήˆD’сγ@ά4Ԙ¦ŽD’сθ@Εˆ›†ŠωοŽD’сθ@q njLSNJF’сθ@t bΔMCΕό‹wG’сθ@t Ζ87 5¦©cE£сθ@t :P1⦑bώΕ»£сθ@t :Pcˆ›†ͺΣΤ[P”ZU§8±$сθ@t :˜ΧEmΣ°&ΥλΝ[ΝJGnGWjŽχΆΓ.0·’σνF~VRž½ΙηZθXIωeg³ we'Ζxt :ˆDJν@ύϊυΛ³ž]κ”οB7&•©έΙlΣΚΜ0•We–΅ ωnœΚ»²ƒ•YΦΚ.[Μ/:ˆ,tΚΉ&VZΉλVZNΥ#£=(ζz0 n…―@-ϋΑJ0ΒdZ›ΐήΰ}ύBbr\‘γa°Œ{ h/ναBΨΪΒΝ0rΓpj*αPŸ‚ t#x†CPk=‘ ƒAdy‡zπTHLŽ 8ϊ)‰Η_ΰ&ψ *ͺPžΊdτ€Λ³ ρΞΰΉχαiͺO 4†wBbrτO5Βͺ οΆΧ4Pyp0˜—είτg0§8ω1΄€Nπ+ά?€e9 &Ý`Ÿ0ξ3o‡¨θ@t :¨κ”sWuvŽzΉ`였»HέΝ’ψ’[&aΫK“°7 g‹˜η.‡Λ@yυΰΒ6š΄ϊΉ6‚†ΠΚ'²s:°eq3Ππ*§‡%—θοc°IΧσIX―VƒμφJNΗCt :¨y„w€5‘ζώΩ!-Ή°ψŸDxœ.r—A‹ώάαψ'ΜIβq|&%qCSaƒ^γ‚—Kk‘θ† ληm³?². ερΣ‡°x‡KΒΒrXΗtω O –σX\yΪ’gΠϟ  Ο‹+«υι²ύΏο|I©‚¨θ@t :H‹K•5ΔΏί̜9³΄ετ{ ώύ[όXήwΧ—€Ÿ2Τ’¦P|ω¨NΠΌξpΡRw2~πγtΛαŸv†Απ΄™Yΰ—!Mσœrqo–E KΥv„­ή$Έ¨] .ΐΐ$π; Κ…Ξ2ψ ΓdΈ ΒFfΒnd¬‹Ÿ,4σΈšΐρP\LΏ„ OΕΙηm .π^λžKΊOτϊΒΣπ.DE’с*γ@ΧΔ*SξV…ύ₯ωXΡΝΘ»Α|Μ?fΫ7 ΕɍαEΕ]ΟE’с…εΐΒ^}΅pΞc]8±O=‘š― gƒ›ƒ΄ό4Δ ΕΥΠVƒ¨θ@t :H9=q¦NU`ό(f΄ΓΑ<%―ˆ'ωgΟŠ8“£сθ@t`:ΧΔΜ^ΨΕ”PΌx::ˆD’ ́…½&Ζ?O,°¦ŽŠD’сθ@υv nͺNϋΥ―:E‰%‰D’сθΐΏXΤ6 ­©’ŸίείU­΄u*-·2:—`ϊ;&ϋ·.ώδ0—όBίΦΉN“V‘²οKΎ–§Y1ωWδTϋ$ŠδοD’сωθΐ’Άi˜ŽWCaγωθΩ;δνΏ―P™Z…ΜΖΓάT¦―φίDh“J«h°"eχΧώ{m+Zˆ"ξχίmp#ˆD’UԁEmΣΰO όΣΙΩΆϋ³:!’\ Ÿ0ΣϊΰžKk‘θΟπ¦@ψg›Γuζ·9d‰ΑL©Ur‘ηB8Iϊϋp‘Η 9LΛJ Ρ₯ όo‡„δXγς°8ψο@ψK₯•.»Ÿ`δAϊӍτ΅ΩαβΚγ΅zγfM³•G‚ŸΞθΆόδΒ6ρŒ²NiωοXl­Σ‰„-s4Υ¬sIςYzg[˜οrΦJDόG΅‚V$ΰ?ˆD’8P»Ήΰ»Ψ+ΐE`GΈΒ‚ιBr*ΈΘn'A ΧΐEqΈόΧ •χ_ζηΑ<]¬ƒΜΛίύw…Ύ°l iΉpύΏ₯KϋŒ GΦun^n…KΑ…ρdXr©?‰ϋ€ cE€ώ Ž[uΉ―‰ηCcΠ»τ"άƒψaΰB}t‚ ­ τ7ώYΕ?ίΉΈοΗ‚Ολ ½ $™ηέpXžƒ!΄΅~ΩΞ]@ΉΑ9Ž4ˆD’ΥΐrώΌδμU»΄°Iπτ6ΠΝ@Jy„}7κbt τ;R7ζ₯\Μ\°rc’ΦωDφM'd…-λYi!Ί-/‡\rΡΛ΅P^Kzέδλ{NΜΎΰ»ϋ²Κu㬛!Ύ~*­aː–…<π™nv”47jA–uX1|M8‘ݘε§άμ< αΣ†Τ©bƒƒ³Ξκa›$Ν͎›˜ 72uB$£сθΐΒv œkb₯;,.•–aΞθgΚ67U>¨3ΰqπO έa¨u ™Χ&ΌNKΕ? >X•° bϊΣ™Ϊπ¨9p&το~―ώߧʝb^³“»­o(GΘp3οCAH¨ΰ±χχKε1‹pψσοΦ/©π5ΈΉͺ#Ό› ΎXΦ·’ΈοϊΧ„>IάƒžΪfi=Nδ—tB)Ÿd]σ1q7 _Βηΰ'#n|lΫ±`ϋEE’сθΤ€MƒοHΣ‹ω6ΔΓ’΅=ατ»φ5ˆ}GΐΓ'' Ύσχ?JR.b..”C‡rΟ]εΈ―"· εfL WAzCD΄ΜΝ~‚πFrηβέ,¨ΰnxΟrγ`(kλͺdϊ™Pα?―mΎ@φ¦'Ή€ΔC{pCσLΦ•~bq)LŸωHoNlK7m³‰FE’сšνΐ’Άi8‘ζl[‚uσέ«‹£οΌo‚ΰupΰ»IΟ© p όλ‚S·Γ-ΰ"σ!ψύσwAΩ \€Χ‚³ΐwͺζ»,τŸ}&δCΠσF$W29‰§›ι>ηwθ ΎC yp0xΪΰχ,Ϋΰ=ωΰfΘΕr7Ψ€Ορ;ΦΑO?ήτƒβ΄'·6π'ΈΙ uΉŸp_ΨΎ7ϊ₯ށ“ΐz΄ο―–w θϋΉ0Φƒ/ΰTΈn„kΑ͈ΎΥΰaΠΛ’'ΆΣŠπ-x}Z%²Ψ^zτΫZτβ.ψ ‚,‹m?f‡ΔxŒD’сjΰ@%ύ¦Un .΄ΩjAB«μΔT|%ΒΝRρt°D¨jo.μPΪ‹«Αun–τ6[nLwΞ₯Ζ$zΎN“Ι9½/«|nφκSŠŒϊrM½R\/‰D’ ԁJ^hΩΘΓ’A Δζšς? ρS•ΝsTΨMΛVΰŸ+όT*ΧΖ’δ¨θ@t :°πXΨkbέ…Wυψδ*δ@GΚΌˆςψ§ŽŸ‹8Wέ’kSΰΰ‘Μ… Ο™ζŸ*‚’>ωΰTTt :¨™ΔMCΝlχμZ/‘Š/IxQΩ4Ό™ͺWvΠοy ΙNŒρθ@t :ψǁΈiψΗ‹š Ώn¨ΙΔΊG’сθ@ ψ‘lTt :ˆD’сˆ›†-ŠD’сθ@t : qΣϋAt :ˆD’₯r :|§aΖΒώ‰I©œŒE’сθ@t`ώ;0cώ?">!:ˆD’сθ@t :ˆD’сθ@t :ˆD’сθ@t :ˆD’сθ@t :ˆD’сθ@t :ˆD’сθ@t :ˆD’сθ@t ώ}δolΆbσIENDB`‚pytorch/docs/source/_figures/to_hetero.svg000066400000000000000000002044661444262304300213300ustar00rootroot00000000000000 pytorch/docs/source/_figures/to_hetero.tex000066400000000000000000000057741444262304300213320ustar00rootroot00000000000000\documentclass{standalone} \usepackage{tikz} \definecolor{green}{RGB}{159,213,179} \definecolor{blue}{RGB}{10,153,201} \begin{document} \begin{tikzpicture} \tikzset{rect/.style={draw,rectangle,inner sep=0pt,minimum width=2cm,minimum height=0.6cm,rounded corners=2pt}} \tikzset{arrow/.style={draw,->,>=stealth}} \def\offset{1.2} \node[inner sep=0pt] at (0,0.7) {\strut\textbf{Homogeneous Model}}; \node[rect] (x) at (0,0) {\strut\texttt{x}}; \node[rect,fill=green!20!white] (conv1) at (0,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu1) at (0,-2*\offset) {\texttt{ReLU}}; \node[rect,fill=green!20!white] (conv2) at (0,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu2) at (0,-4*\offset) {\texttt{ReLU}}; \node[rect] (out) at (0,-5*\offset) {\strut\texttt{out}}; \draw[arrow] (x) -- (conv1); \draw[arrow] (conv1) -- (relu1); \draw[arrow] (relu1) -- (conv2); \draw[arrow] (conv2) -- (relu2); \draw[arrow] (relu2) -- (out); \node[inner sep=0pt] at (6,0.7) {\strut\textbf{Heterogeneous Model}}; \node[rect] (xpaper) at (3.5,-0) {\strut\texttt{x\_paper}}; \node[rect,fill=green!20!white] (conv1paper) at (3.5,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu1paper) at (3.5,-2*\offset) {\texttt{ReLU}}; \node[rect,fill=green!20!white] (conv2paper) at (3.5,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu2paper) at (3.5,-4*\offset) {\texttt{ReLU}}; \node[rect] (outpaper) at (3.5,-5*\offset) {\strut\texttt{out\_paper}}; \node[rect,fill=green!20!white] (conv1middle) at (6,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=green!20!white] (conv2middle) at (6,-3*\offset) {\texttt{SAGEConv}}; \node[rect] (xauthor) at (8.5,-0) {\strut\texttt{x\_author}}; \node[rect,fill=green!20!white] (conv1author) at (8.5,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu1author) at (8.5,-2*\offset) {\texttt{ReLU}}; \node[rect,fill=green!20!white] (conv2author) at (8.5,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu2author) at (8.5,-4*\offset) {\texttt{ReLU}}; \node[rect] (outauthor) at (8.5,-5*\offset) {\strut\texttt{out\_author}}; \draw[arrow] (xpaper) -- (conv1paper); \draw[arrow,out=270+45,in=90+45] (xpaper.south) to (conv1middle.north); \draw[arrow] (xauthor) -- (conv1author); \draw[arrow] (conv1paper) -- (relu1paper); \draw[arrow,out=270+45,in=90+45] (conv1middle.south) to (relu1author.north); \draw[arrow,out=270-60,in=0] (conv1author.south) to (relu1paper.east); \draw[arrow] (relu1paper) -- (conv2paper); \draw[arrow,out=270+45,in=90+45] (relu1paper.south) to (conv2middle.north); \draw[arrow] (relu1author) -- (conv2author); \draw[arrow] (conv2paper) -- (relu2paper); \draw[arrow,out=270+45,in=90+45] (conv2middle.south) to (relu2author.north); \draw[arrow,out=270-60,in=0] (conv2author.south) to (relu2paper.east); \draw[arrow] (relu2paper) -- (outpaper); \draw[arrow] (relu2author) -- (outauthor); \end{tikzpicture} \end{document} pytorch/docs/source/_figures/to_hetero_with_bases.svg000066400000000000000000002372201444262304300235320ustar00rootroot00000000000000 pytorch/docs/source/_figures/to_hetero_with_bases.tex000066400000000000000000000067721444262304300235410ustar00rootroot00000000000000\documentclass{standalone} \usepackage{tikz} \definecolor{green}{RGB}{159,213,179} \definecolor{blue}{RGB}{10,153,201} \begin{document} \begin{tikzpicture} \tikzset{rect/.style={draw,rectangle,inner sep=0pt,minimum width=2cm,minimum height=0.6cm,rounded corners=2pt}} \tikzset{arrow/.style={draw,->,>=stealth}} \def\offset{1.2} \node[inner sep=0pt] at (0,0.7) {\strut\textbf{Homogeneous Model}}; \node[rect] (x) at (0,0) {\strut\texttt{x}}; \node[rect,fill=green!20!white] (conv1) at (0,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu1) at (0,-2*\offset) {\texttt{ReLU}}; \node[rect,fill=green!20!white] (conv2) at (0,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=blue!20!white] (relu2) at (0,-4*\offset) {\texttt{ReLU}}; \node[rect] (out) at (0,-5*\offset) {\texttt{out}}; \draw[arrow] (x) -- (conv1); \draw[arrow] (conv1) -- (relu1); \draw[arrow] (relu1) -- (conv2); \draw[arrow] (conv2) -- (relu2); \draw[arrow] (relu2) -- (out); \node[inner sep=0pt] at (6,2*\offset+0.7) {\strut\textbf{Heterogeneous Model}}; \node[rect] (xpaper) at (3.5,2*\offset) {\strut\texttt{x\_paper}}; \node[rect] (xauthor) at (8.5,2*\offset) {\strut\texttt{x\_author}}; \node[rect] (linpaper) at (3.5,1*\offset) {\strut\texttt{Linear}}; \node[rect] (linauthor) at (8.5,1*\offset) {\strut\texttt{Linear}}; \node[rect] (x) at (6,0*\offset) {\strut\texttt{x}}; \node[rect,fill=green!20!white] (conv11) at (3.5,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=green!20!white] (conv12) at (6,-1*\offset) {\texttt{SAGEConv}}; \node[rect,fill=green!20!white] (conv13) at (8.5,-1*\offset) {\texttt{SAGEConv}}; \node[inner sep=1pt] (aggr1) at (6,-1.5*\offset) {\footnotesize$+$}; \node[rect,fill=blue!20!white] (relu1) at (6,-2*\offset) {\texttt{ReLU}}; \node[rect,fill=green!20!white] (conv21) at (3.5,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=green!20!white] (conv22) at (6,-3*\offset) {\texttt{SAGEConv}}; \node[rect,fill=green!20!white] (conv23) at (8.5,-3*\offset) {\texttt{SAGEConv}}; \node[inner sep=1pt] (aggr2) at (6,-3.5*\offset) {\footnotesize$+$}; \node[rect,fill=blue!20!white] (relu2) at (6,-4*\offset) {\texttt{ReLU}}; \node[rect] (outpaper) at (3.5,-5*\offset) {\strut\texttt{out\_paper}}; \node[rect] (outauthor) at (8.5,-5*\offset) {\strut\texttt{out\_author}}; \draw[arrow] (xpaper) -- (linpaper); \draw[arrow] (xauthor) -- (linauthor); \draw[arrow] (linpaper) -- (x); \draw[arrow] (linauthor) -- (x); \draw[arrow] (x) -- node[fill=white,inner sep=1pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 1}$} (conv11) ; \draw[arrow] (x) -- node[fill=white,inner sep=1pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 2}$} (conv12) ; \draw[arrow] (x) -- node[fill=white,inner sep=1pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 3}$} (conv13) ; \draw[arrow] (conv11) -- (aggr1); \draw[arrow] (conv12) -- (aggr1); \draw[arrow] (conv13) -- (aggr1); \draw[arrow] (aggr1) -- (relu1); \draw[arrow] (relu1) -- node[fill=white,inner sep=0pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 1}$} (conv21) ; \draw[arrow] (relu1) -- node[fill=white,inner sep=0pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 2}$} (conv22) ; \draw[arrow] (relu1) -- node[fill=white,inner sep=0pt] {\footnotesize $\mathbf{a}_{\mathcal{R}, 3}$} (conv23) ; \draw[arrow] (conv21) -- (aggr2); \draw[arrow] (conv22) -- (aggr2); \draw[arrow] (conv23) -- (aggr2); \draw[arrow] (aggr2) -- (relu2); \draw[arrow] (relu2) -- (outpaper); \draw[arrow] (relu2) -- (outauthor); \end{tikzpicture} \end{document} pytorch/docs/source/_figures/training_affinity.png000066400000000000000000003057761444262304300230370ustar00rootroot00000000000000‰PNG  IHDR@X:Ζ*B IDATx^μέxUίώρ;zo,(φˆ‚)D@₯* ˆtQ@@₯J‘ͺτ"(’ˆQQ)*Š" ŠΨyDzo!mίλ o LΨM²3σέλςώφΜ™s>Ώ³χΞ™βσω|β… € € € € €ΈH „ΔEΥd* € € € € € ` €°@@@@@@ΐu +)B@@@@@Φ € € € € €ΈN€Δu%eB € € € € € @Β@@@@@@Χ €Έ€L@@@@@@X € € € € € ΰ:Χ•” !€ € € € € €k@@@@@\'@βΊ’2!@@@@@ a € € € € € €€λ@\WR&„ € € € € € ¬@@@@@pˆλJΚ„@@@@@@€„5€ € € € € € q]I™ € € € € €€°@@@@@@ΐu +)B@@@@@Φ € € € € €ΈN€Δu%eB € € € € € @Β@@@@@@Χ €Έ€L@@@@@@X € € € € € ΰ:Χ•” !€ € € € € €k@@@@@\'@βΊ’2!@@@@@ a € € € € € €€λ@\WR&„ € € € € € ¬@@@@@pˆλJΚ„@@@@@@€„5€ € € € € € q]I™ € € € € €€8p ŒΆPSζ.Υ ]š«YΓϋ:ƒ5λ·¨C―Qͺ]υnz©c@ϋvZgMŸ€-Ώό©₯s†κΪβΕ.iψ£'/Πτ·—λΕξ-υhύj—ΤGf^δτρg¦χF@@@@Μ €Άίw¨QΫ/ΨSDΦ,*R(ΏΚ•ΎI­­£«(βΧ]€|Ώy»~ων5oT#q<ίύΈM}†NUΥ{οTί-ό§?ίR₯uš.ουΜcjΩΈVšI­qη~γ,Ÿ£{©ψε…SkžβϋΣήϊPσ―R·Ά¨nς—ΤG .ϊvγ6=Ρ}XšΊZ8mΦ~»%(Ɵ¦Σ@@@@D€e€dΟ‘ΚεξH£Ο'<|TΫψŸŽ;!†LΪ]εΚά|ΙwTsΠλϊiΫ_ϊψ­—<–τΊ°sί±ηu½~γ/:q2J₯o+©ΌΉs$yΏαƒ•UυήRι5ΗχϋΫ_jά΄…Iζ§5λ7[V­Βωv½:=+Šrόά™ € € € €xS€$uO@Ό¬πΓ„SQΡ1ιm-XςΉŠΚ§OηRXXθ%έ=PHυΖ=”%KxP )Α4lΣ_Ώώρ?½9ώ+±σ2_ς›W–π0;Ν=Υζθρ“*_·£BCC΄eΥLO͝Ι"€ € € € €€ϋ@Pc;ˆΉM\\ΌΚΧλhνb0ΊρΊβ‰wχω|Zψαj½Ώ|΅~ϋk§bbcuyΡ‚ͺQω.΅}όAεΜ‘-±ν…“§’4ού•ϊψσoυίξύ:uZŊP΅Š₯υtΛϊ‰}˜6CΖΝ=oζ›>›―7όœδ ‰3iμΕjΥΈ–žζ±σ1»Z*?άEY³†kυ’ρΚ™ΥjcΉ4󝏴ω—?tόψ)εΟ—KεΛά’-RρΛ/ν`© _oΨͺΆ=_΅ΖZ΅B) =[»[s'τΣ7—°ΖυΓ–ί4{ΑΗΪΊύoν?pX‘‘Ίε†«υdΣT‘μ­Iζ— ­ΊΥ†MΏjύ‡―λσu5ϋέOτ׎]VˆpϋM%Τ£}«Ώ„WJgh€΅Σ—©η˜©οκ« [uκΤi]wΝεjΧ¬Κήq£*Το€ΫnΊVσ_ΏπcΨRZζv”ΖŸ°&Ζξͺ,ααš0γ}™&‘‘Y­χλΦBΉrfΧδ9K΄ψ“uΪwπˆ΅–›Φ―fY’d8iY+ϋΡδ7—ΘœU³{ίA+Ψ2—«^±΄UΓ<ΙvΰγM € € € € ΰPΞnbnU§ΩσΪ±s―υe΅ω:αΥkπd-ϋτk+œί ,²g‹”yδΣζŸΠυΧ\‘yϋ)GφH«yJˆΩιΠφΩΦ—σ&X){η2‘ΚΊο~²Ύ Ώυ†kτΦ€ώΦ3ή/Ώή€qΣ*ožœjίΌžΥo‹Gjjν·?% @ώωwhήΛ RVΎ3κ<-Ψτ1CυkUА>ν¬χί]φ…Œœ₯Ȉ¬ͺSνkNΫόW+Χ|o…0³ΗφIώΨ-AjΘ[Ά«Eη!zΈvEλμŠJ\©ΒσYA9weυ7›Τ©οXλKϋϋ+•QΡΒωυο}ϊtυ+œJΛ>α•RηH}΅α'΅~΄ΆV­ύAM’…ςΛά{ξΒO•;gv}όΦ«‰_Δ§ €΅#GO¨A›~Ϊ³οJέz½Κ—ΉY»φΤ‡Ÿ}£§šΧΥ„‹Tζφ’š3ξ»”V»K @¦Ξ[¦Χ¦Ύ§&υͺhΝ·[Τς‘šΦ:ϊ֚࣡5Λ›;§φξ?d{r::FSη.³‹Wz΅Qƒ:•Η™–΅bvQ5lΣΟϊό˜GΘέRςLΠ΄ρ§ί- -˜<ΐzΜ/@@@@@€$kΐnbΒ„[τ–ωψ―–L΄~K޼̎gN²B™―υΆΎD7/`˜o-ϊLO4­£ž΅ώ<₯Δ|oν6‘Κ[ϋ[;Μ+&&V Ϊτ·B CΊ&ž“qθΘ1U¬ίYΙΫe~³ΎC―Qͺ]υnz©£ΥGγ§θηνλΙ/YAΚΉ―§ži…,ΣG=o})½kΟΥiήKαa‘šϊKΦ— ―OΎψN=L΄ϊ0}₯υ•ZςγΦίΥμ™W¬ Θ6ί±Uύ$·0Υ›Z™3Xξ+φ¬–%+֩ϐ©Ίιϊ«τήԁ‰Χ€€$ΜΧ„:‹fΌ’XCsQ·'XaΚΉ_ς§€€΅„zΧͺRV£<“8>³›Ε„^&`ΈλŽ¬`)-/σψ΄ίfνξ0/PTzΈ“υ͎›7ΗχM|Μ› €Ϊ??J•ξΉMo ΦjŸΦ΅blq½šχjΨ O%™κˆ‰o[»ŸΖΎάYUξ½3- ΄E@@@@— € °© ³~C}ΤοXΏ½nΤΏ{ΛΔ;·ι1Bίόπ³^ΦύΌCΤΝ—Τ•κwΆΎh_»xΌuMJΘΞέϋυΛo¨XαIΑdΪ'| ί₯M#΅oqf·GZY >Φ«“ζ[βκώTγΔq>r\•vQ|ΉυΩ‚1VθςƜ%?γύ$ΝΉΔMΪΠΦ_Φ♃“„#vʐZ²ιη?τxΗ—­sMV/—$%τmvn8xTή_>1 2ο™‘tΝvΦ—υ›?›‘8”‹ ζq`ζQ[ηΎœΜŽ“nν±ήΊXb·Oφ³vΠ$ί5dϊ7Α p2#©Uεnp&$Kx%„e& 0AEΒ+a½]SΌ˜–ΝjύqZΧJΒn£GλWΣ‹η|~L_±qq MRW;kŠ6 € € € € ΰ^Τ6!±Σ•yTΤ‹=ZY‡Jx•­Σ^'OΦWK'*Oηu“°saΕό‘ΦoΫ§vΊΩ9bϊ‹Ž‰±ϊzgρηV(ρTσzκΪΆ‘υgi @φξ?¬j»[»E>š7c¬Ηk™ίΔ7™Jώ>ρmΝyχ“σ…dΗΞnb}υώτ—/ΪettŒ΅S!ήo΅«Ωτ9™3TΜ9(αagLΏX2udOέ{W3C<Μ£Δzwzάκγbˆ>βγ}*]³­bγβ΅qΕT+ά9χ•°{(3¦™PνάWΒω&Ιڄι²’υιό‘Φ%i]+&δ«Ϋ²΅«©ΙCUUχώςΊύζkλeg Ρ@@@@Ό#@€Z' Ω³Eœ·ƒΓœƒ`UtmρbzcΔ³‰ JΈmΤιh•©•τq>’9δΞ[»`²ό³υzkΡJλ€oσςΧ₯ ¦Ÿ'Ί³67Α‚ Μ«Ν³#τΝχ?'ω³¦jΛΆΏRU5;$ΜN‰΄Όμ εοΊEΣF>w^Χζ퉳>Π_m” uRzΩ @RzΨ{ΛΎΤK#gͺy£κΣΉ™ΥύΕ;}?qJχ<ψ΄uvŠ9x=ωkΛ/ZAMf ζ‘l&ό:χ•€,3ΤZσ ―„δά³d.e­˜ƒξ_9Σ:ήΌΜY9εJί€ΪΥξQν*w'>r+-늢 € € € € ΰNΤυbΐ2GύΦ}­ΓΆΝΩζ „s_ _ ›?K~fEς‘Υ―]QW+”bbΞ <φMλQYζ[7—ΌZ9sD*D!ϊtΝχzgρͺKήbΖ‘πε~‡–©σ“ eλu_Γ.ΊξκΛ­³0^ »&ΜAδ—)pAέ»ξΌQχ”Ί)Mϊvs·yœΨΉ―'£d%ev”+}³jW»[E ζ³D7―ΞύΖZ;B‚-9vό€ΚΥνhΥυ›e“Ξσ2a“ €<=H&ΐIλZ1ΦoΨό«u¨½ ΰΜηΟΌL88cL/AOΣ§ŠΖ € € € €ΈW€$΅Mν „ƒΕΝΑΩ‹g ±ι>χe~Γίό¦Ώ9·Βœ§‘Ϊ+₯G`έΧ°«Μ‡ΉϊͺΤ­Χ'ι"α¬v€˜³H*?άYΕ//’%³‡hΑ’Ο5pτlλ`φswtξ;V«Φm΄P7©ςεO²`ι8j–*”½U“G<«sύΏ3tˆω²ŽϋΫΘ wγ§ΣΞ{άSΒΑΰN @΅Vvμάc…ςγΦίΟ[\τ… € € € €Ξ @½R @Μ-z ˜¨OΎψN)ΰάηH™ΊΗ 쀚χέuވŽ=‘<ΉΟž ’<IxŒ–9WδϋO¦œw}³g^±Ύφ'1&|aύΡΌV˜°~γ/Zυξ.˜7ρžSη-ΣkSίΣγ ͺ«oΧηΟεΨ λpςs»%π'υΖ͘Ώ<Ε/ΘΧ}χ“žzξΜΉΑΆČ©N³η΅cη^k§MΙk―HΒΥoψt-ϊh#w€€u­˜:ό»[7^WόΌ%c3Χ’σ`₯t0»έυE;@@@@p—Hκi'1gNΤmΩ[ζQL³Ηφ±Ύ°Nx%d}ύ5Wθ­Iύ¬s ^ίoήnΏQ£ς]Φ σJiΘ½=#”¬|g”Μ9 /³ϋγν>³v‡<φpuυλv&”0~—­ΣΑ:t}έ’ ‰DΒn³{#α~ΙΗi5iφ*{獚>κω$‚»φPζ½’w§T‰«.K|߁Γzό™Wdφ^˜0Ω"Οo§ ώ σήTCΖΝSƒ:•¬Ψ^ζ,‰½ΗhΟΎƒΦ.œ΅‹Η+_ž\ΦΫ;έΞω¦Ο1} 3Ηz„™9ψϋ₯­ΗnB-ž™Z:qHZΧJBgΞw1缜ϋzσ½6α-{½5E@@@@p―Hjk'1·™»πS ?OΕ//lύ6ΏΩ±‘πzξεΧe17Ιͺy_Yλ1YΏύ΅SŸ―Ϋ¨ˆˆ,Φ‘ή·ί\ΒjžR2bβۚύξ'Φ#ͺΜ™ qρρΦ ζKύΎ]š«mΟW­]$OR+ΙγΨ„Ac@@@@pˆλJΚ„Ό$`‚¦'»·vρΌ>¬»—¦Ξ\@@@@@‹ €°@rs~‹ΩR€P~kΧΚΉ―yο―ԐqsΥ¬αύz‘Kσ Ÿ ΓC@@@@2N€$γ¬Ή—$°lεΧΦγΉξΌε:ΝΫGYΒΓ¬~Μ‘χMŸ(sFΘΤ‘=uο]·^R\„ € € € €ΈQ€ΔUeNˆ‰‰U›gGθϋΝΫUβκΛ­σ.N:­ΟΏΪ¨=ϋ©V•²=ΰWΝ™Ι € € € € €ώ €ψ+Θυd€ΐΙSQšσξ -_΅^»φμW\\Ό)^LΥͺ ζ k(,,4FΑ-@@@@@η€8§VŒ@@@@@l €Ψ„’ € € € € €8G€Δ9΅b€ € € € € € `S€Δ&Ν@@@@@@ΐ9 Ξ©#E@@@@@› 6‘h† € € € € €Ξ qN­) € € € € €Ψ ± E3@@@@@pŽˆsjΕH@@@@@@ΐ¦ˆM(š!€ € € € € €€s@œS+FŠ € € € € €6@lBΡ @@@@@œ#@βœZ1R@@@@@°)@bŠf € € € € € ΰηԊ‘"€ € € € € €€M›P4C@@@@@η€8§VŒ@@@@@l €Ψ„’ € € € € €8G€Δ9΅b€ € € € € € `S€Δ&Ν@@@@@@ΐ9 Ξ©#E@@@@@› 6‘h† € € € € €Ξ qN­) € € € € €Ψ ± E3@@@@@pŽˆsjΕH@@@@@@ΐ¦ˆM(š!€ € € € € €€s@œS+FŠ € € € € €6@lBΡ @@@@@œ#@βœZ1R@@@@@°)@bŠf € € € € € ΰηԊ‘"€ € € € € €€M›P4C@@@@@η€8§VŒ@@@@@l €Ψ„’ € € € € €8G€Δ9΅b€ € € € € € `S€Δ&Ν@@@@@@ΐ9 Ξ©#E@@@@@› 6‘h† € € € € €Ξ qN­) € € € € €Ψ ± E3@@@@@pŽˆsjΕH@@@@@@ΐ¦ˆM(š!€ € € € € €€s@œS+FŠ € € € € €6@lBΡ @@@@@œ#@βœZ1R@@@@@°)@bκBΝώ;pΚΟΈ@@@@H?Λ dKΏΞι  ρ³8 ~r9 €ΈR |ΥϋΚςΕ 9΄_ρŠ*¦ΞcŠ»§ΖEηΎα …4O‘{U||Š»ο!ΕΤjκJ&… € €@F €d€6χ &?«Aβ' —#€ €€λΒW/UΦύp;Ε_{³ΒΆ}―,Λζ(κιAŠΏ­\Šσ Ϋφƒ"ΖφRL床+SEaΏo±‰~€ƒb«5tB@@ #@2R›{“ˆŸΥ ρΛ@@ΐu‘/<Έ2•Σ¨Cβά²NE!φκτσγRœoδθžςωβtϊΩ1g―y{œΒ6Ρ©aοH‘‘sbB € €d”HFIsŸ` ρ³" ~r9 €ΈJ tΟ9ΰIEuUρ%οLœ[Ψϊ•Š˜5\§F _ΆπŠΣ IDATηΝ9Ϋs(φΎzŠ©Ϋ*ρ½Π_7*ς΅ηuκ…7€ˆlŠό”b΄Sl•ϊgڜ:l/=‘Έ2χ)ϊΡNrd2 € €R€$šτε$?«Eβ' —#€ €€«Β6­ˆΧ_TΤΰyŠΟ_ψlςΗVEŒμ¦¨ήUΙσξυ[£±bh~6ωοoEΎάN§ŸzIq₯**|εBeYώ¦’Μ”/w>e}g‚Β~Z―¨ΎSδ‹δPGW-$&ƒ € P€r™ƒ@ό,ˆŸ€\Ž €ϋφ3EΜ¦“£IΩrž 3vώ₯ΘWžRT·ŠΏ‘ΤysŽΪQΎΌuϊιAgC“οΏTΔ΄Wές9Ε–―)ΕΗ[!Š―πŠΉΏ‘² }FQ]†¦ΨŸ«P™  € €~ €ψ ΘεŽ ρ³t ~r9 €ΈJΰRπ5Λ”υνqŠnΦ]±e«)τΏΏ1wŒBvώ©ΣOτVάέΥ-§Π]+rθ3ŠΟWΘzΔVt³nςc2 € €€‡Hz¨§@ό¬ˆŸ€\Ž €Ϋ²^“ϊιΤΰyςσ¬ΠίRδ¨ξŠκ3IρΕ―?ΞρqVΎξ#Ιη³Ž˜‡ΫX»INw¦Έ›Κ$^ρΪσ ϋu£’zMPόΥ7ΈΚΙ € €  ι‘JŸN ρ³J ~r9 €ΈJ dοΚφR«σB ld}λ5΅HΎΘμžσ‰£ 9yBΎ…φ돊Χ[§†Ύm=ΛΌ¬€eΚ@Ε•ΈUŠ9­Σώ‘BC]eΘd@@΄H EιΟ) ~VŠΔO@.G@Χ ˜$έ΄sβά"&φUHt΄’ΊΏšβ|C·oVȩ㊻γή³ΧΜ¦};ΥkΌυg!Q'1¨­βξ­­ΨJ*r`ΕΤm©Ψj ]gΘ„@@€@° ±q>ΕΖΕ۞rXh¨²„‡ΨnOCο €ψY{?Ή@\'ΆώSEΜe=Β*ξΊ[Ύιk…―˜―¨n―*ΎδΦ|³Ξ-eΙͺθG;Y;ό‹ΕΚΊπ E7ξ¨ψ«JZ;=²|4OQ]†%ržε­±Φ¨~“­kΓΏ\’,οOQTΏ©ς*ζ:G&„ € €@ ‚=ΩόK΄-“bγμΝψRΉYr ΘΆίw¨QΫ“ά(OξΊκŠ’jΦΰ~Υ­Qήή Ψκή‡žQΛGj©CΛ‡4ού•6ažΆ¬šΐ;x·+?kOβ' —#€ €€+ΒW/S–O(δπ>ΕΎB1υZ)ξΊ‰sήYΎˆl:έmΔ™?σω”εΓ7Άξ#…?,_‘βŠ~¨΅βn?σf‡Hδ˜guΊλpΕέXϊΜ5ρρŠ|΅‹”5›’L?—ς_?ΤgR € €$φδΗ­1š‘:ΥξIl|μψI?qΚΪQ‘πzu|­ώf“–ΞjύQωΊΥ²I-=έ²~b›Ν?‘ˊTΑόyΤ΄Γ@eΙ7ΗχM|Ω§_«ΧΰΙZυξ)”/ΥdΥΊϊςύ±‰»N–ψJ½‡LΡ†§(66V•ξ¬6?(sIΒλΕWgXΑΙςΉΓΣum{η ~VˆΔO@.G@@@@t 9Λ›€<χtSέuη Φ'OžΦ–mjΚά₯jΡ¨¦:=Ωΐϊσ£ΗO*!π8pθ¨|>ŸυηςεΦκEγ¬ΏαΣeqυXύjͺ\ξ•Ή½€x˜—y„V©šmΥΎy==σΔ™>ΝkΟΎCͺΦΈ»Μ.”•οJ5ωγŸτήԁ‰ΧρՏzζ…Χ¬ενU«C5sLoέ]κΖΔ6 !ΛΊΕ”7OΞt]_Αά9ˆŸΥ!ρΛ@@ΐρζ?B‚dζΏGB‚e0AbΒ0@@ 9».tΊiρξ²/4pΤl}όΦ]Q¬žω }ϋγ6θΧAwήRBY³fΡΨi ΅πΓ/rΜ_ό™–~ϊ΅Άός§΅“€y£κΨϊaEGΗ¨l Uhhh’…«ώέ[ͺiύj© ;wοΧά gw$ Ÿ½;ZΫ~Ϋa…!YΒΓ’όǐ/ήg b“eΞ ρκ‹ΔΟΚ€ψ Θε € ΰxποΏPΨ'οΕω4ςŎVΨ1ςω*~y-ϊh>˜9XωσζR­ΗžΣύ•Κ¨Ε#5•=[€~ωνoλ\>›©α•υικ κώDukχˆuήΗ©¨Σš2w™~Ψ²]Λη³ρ')Z(Ώ†Œ›kCΏ[KέvΣ΅2»B†M˜§ΛŠΤ„!]]ΎΊ/>=?ΛOβ' —#€ €€γ@_B&€ € ΰr`@6ύ­ΟΎ4‡†Ϋ+DΕrRω2Yν5NΦ*!9χΝ#’Μ£’κT»G­­£μΩ"¬··ω―ϊŸνύ«+‹R·§λ–’W[‡Ž;qR &°vZŒφžΆnϋKQΡ1ΊΌhA=Rχ>΅nR;ρ&œ˜6o™ώΪ±Λ <ΜAι=Ÿnͺ«(b΅ρ71g}Ό>{±ΌVϋQώ|ΉT½bυhίΨΊŸ—_ ~VŸΔO@.G@Η €8Ύ„L@@ΐεΑ€Δ›»γ}Ά«"σ/R IM(•χ @όδr@pΌˆγKΘ@@\.μˆΛω™^& €ψ‰Oβ' —#€ €€γ@_B&€ € ΰr—˜ι]P€ΔΟΕAβ' —#€ €€γ@_B&€ € ΰr—˜ι€€Χ I/YϊE@§€8₯RŒ@@ΐ« ^­<σfˆŸk€ΔO@.G@Η €8Ύ„L@@ΐε ./0Σ» ˆŸ‹ƒΔO@.G@Η €8Ύ„L@@ΐε ./0Σ#I―5@’^²τ‹ €N qJ₯' € €€W@ΌZyζΝ?ΧˆŸ€\Ž €Ž q| ™ € €€Λ@\^`¦wA?ˆŸ€\Ž €Ž q| ™ € €€Λ@\^`¦G’^k€$½dι@œ"@β”J1N@@― €x΅ςΜ› ~?Ή@/@βψ2@@— {uπbŽΆ]…°9”½paΫνiθ]?kOβ' —#€ €€γ@_B&€ € ΰr`@oΪ€ψ7^VHΜi{•hύœςV¨¬{Ν/ΤκΟ»T―e.˜WŸ-£ΠΠ€ΖΕΕλف“΄fύfUΊηv½6¨“¦½υ‘υOXX¨Φ-žΰίΈ:έ@ό$&ρΛ@@ΐρ Ž/!@@pΉ@Π ?n”ο՞Ά_§—•·RUΏQo,Πκυ›τη?iςˆguο]·&Y ίύΈM­» Σ‹έ[ͺΒέ·©hαό*]³κΧͺ¨ΤTΙk―pωΚqώτ@ό¬!ˆŸ€\Ž €Ž q| ™ € €€Λ@Ξ/°ΩέQ½I΅~΄ΆΦ|³Y δΡπΎν“4\ρεui‚Φ|0^ωσζα#ΗU‘~'ΠQ΅ͺάνςUγŽι€ψYG?Ή@/@βψ2@@— €œ_`σX«§{ΡͺwΗhν·›5dά\­^4NΩ³EZΗN[¨)s—&^xKΙ«΅uϋί‰;K–pύψι4—―ηOΔΟ€ψ Θε € ΰxΗ— € €Έ\€δόχ0IΗOœΤ”W{κΔΙ(UnΠEύΊ΅Pƒ:•¬Ζ§’’υρηλΥoψt}ψζ0εɝSGŽΧƒ-zλ•^mTυήRΚ›'§ΛWŽσ§Gβg @όδr@pΌˆγKΘ@@\.@’΄ΐGŸΤ} »jp―Άz ϊ=Φ›½OΦήύ‡4sLοΔƟ|ρz ˜¨΅‹Η+_ž\:tδ˜*Φο¬ΡžQ­*e]Ύjά1=?λHβ' —#€ €€γ@_B&€ € ΰr€žΏx•FO^ Οί{MY¬7Χπ‹žzn€VΎ3JŊ°ώŒΔω ?kHβ' —#€ €€γ@_B&€ € ΰr€nΪa Άlϋ+ΕͺwiΣHν[Τ#qΙg‚ΔΟB€ψ Θε € ΰxΗ— € €Έ\€δlά±KυZφѐ>νTβκΛ’T~Α’Ο΅aΣ―Z>w8ˆK> ~’ΔO@.G@Η €8Ύ„L@@ΐε g <κZ²bυψ«ΠА$•ίς˟jϊτ ½=©ΏnΏΉΐrΑη‚ΔΟ"€ψ Θε € ΰxΗ— € €Έ\€δLγββU½IΥ¨\F}»ΆH±κ5šφT₯{nΧ‹έ[€ΈΰsAβg @όδr@pΌˆγKΘ@@\.μΘΡ_·)φ§’/ήV%B―»Iyο,m«νΉΦ¬ί’½FiφΨ>ΊλŽRΌώΥΧηkΡς5ϊςύ±Z΅n£z ˜¨΅‹Η+_ž\:tδ˜*Φο¬ΡžQ­*eΣ|.Θx?Ν @όδr@pΌˆγKΘ@@\.μˆΟ'Ε›Ηζ+4$D!IŸ^eσJšyM€Δϊ€ψ Θε € ΰxΗ— € €Έ\ Ψ—σ3½L Iί<nΒΜEš2w©z=σ˜Z6•δ L\½ά@‚B€$(Κΐ @@Έ ‹Γ« ©όΎ‡ΥsΠλ:xθ¨ώΩΉG=;ΕΕE ¬A„…K‘rn ‚GςGο“β‚θίaa!rπΗς DWšŸ-qfaΙ+<ΜόέΠΏΑπχKόΜΏ7ΜΏ?‚ε•%άΟ,qπ8bΜΟσΝ`x…HΦΟ‰`‹KƐ="ά%3a€M€Δ¦Χ…Γ'blφp¦™ωΛΕϊ Ί―tYΊ5._N*·dΎΰδ•9ρρ>­Z-ύΈ)sξŸόΥͺϊTκ6σ#¬‰ΜͺˆYK?ςι·ίƒ£υ”nΈ.c1m€n2«.™y_’ΎϋΎOνΚ|@σ₯f½|Ίξš4~;›™€.Όχ‘£ρZ΄DΪ σΧDή|γE–)ϋό @j­Πϊ-e‚:§ΎΜwQiύΧπž}>-YζΣ‘£™?ο<Ή}ͺW7DE eώXœΊ1ξΖiις:•ωu(P@jπ”7wζ%ΆNν㏿γ΅μΓΕ€ύwψ>εbΕ|j0D~αΝί/ύͺΝφ?}Z²Τ―.vρ΅%|ͺ€ω%~N 5™ΏlϊΙ§•«xa:5Ών6Ÿξ―"…₯υ·BiŸZ4‹WόόœDm/΅]»C4g^¨NœΘό5Q€Θ™wδΜΑšΈΤzβΊύ/D³ή Šδͺβ>΅nǎ@֏>ώψ;D³η`«·cHΈτ–[|jΘόw(―ΜΨ²5Lο. Ž*”/―Ϊ5ψn"λG`R“Ύœ$@b³Z 6‘hvI —Δζκ‹@\]ή4OŽ$ΝdΏ€δΒ%&Ιό/-@‚γGHpΤ!˜FALΥŽ±€G‚i ΑTΐ%ؐ­Gι―¨cΆŸ^4k6έ•·Αiΰ—Šλz$ΉHIήώ·ŽŸˆ²Z΄λωͺΧ«’šχ•΅ώχ·”PDΦ,ϊοΐ©4- v€€‰Λ3 @{Οj|σ²ϋΥ¬θυ~οόώsΗ.ΥkΩG… ζΥg Ɯχ(έΈΈx=;p’Φ¬ί¬JχάΧu΄·>΄ώ ΥΊΕl―…;χjκΌeϊvγ/Ϊ{ΰ°²EfU‰«.S³†5T»κέΆϋI­αoύ«‡Ÿθ§7ΗΏ ·•L­yšήξΗmšώφrύϊΗ:|LyrηΤν7—Pϋυtλ Χ€©―ŒjLrι¦O–_ώL±ΕŠω#uyΡ‚ ΅R]~—ψ¦Gr h.Ύ„ΔΕΕ½Δ©€€€\β‡ΗC—€x¨Ψ6§JbΚCΝ@ρm­ύ`ΌςζΙyΡΕΆ{ίA˜8__oψI§’Nλͺ+‹ͺέγuU·FωΔλφξ?¬aζiν·[ͺrenVοNΝT€P>₯€˜y˜ΗoΝz­nΉαj8tT#&½m=Ϊλδ©ΣΊαΪ+Υ½}c•+}³u[ͺ΄NΌWΞΩ΄ώΓΧΥΉί8νά΅OοO9ΥKjύ›£'?5@γwΥΌ…ŸjγOΏ)[Ά+lιρT“σE–κ /€Δ=‰G`ωιΗεgΨΒJH.@š8W€ ¬‡δ ^‚Ξ!θόΔ8#@ΒJH.@šH.@šH.@βξ5Ar~}{ ˜€γ'NjΚ«=ev1TnΠEύΊ΅Hάρp**ZΎ^ύ†OΧ‡o³Ξ»8rτΈlΡ[―τj£ͺχ–Jf\,ωγŸΤ¨M+ΜθΦξ•+}‹uHςWLL¬΄ιo/―k ̟ΗzDΦs–hΦ=cγβΤΈέK Χ³š(KxΈ†O|KζZNόώχΞ$g€|ςΕwzώε7¬λΝ9&fηK“φ¬9xΆ΅ Μ«ω|¦w—~‘w§ΤυΧ\‘=ϋ©Zγξz‘KsΥ½ΏΌςδΞ‘…Φ‹―ΞPυJ₯υdΣtۍΧZγLώ²Σφ?Uƒ'ϋ©ψεE4ΈwλόUλ6κΉ—_·‚œf οΨ’ΔOJv€ψ Θε ¬@X ¬‹ €€$ΰt~f$ aM€°R IMΘ{ο€Έ»ζ Iλ{τψIέΧ°«χj«ͺίc½ΩkπdνέH3ΗτNllΒƒ&jνβρΚ—'—9¦Šυ;kτ€gT«JΩ$^,1 WωήΪebΒ…π°0έzγ5V av=˜ζ΅jνΦN‹ΉϊͺΤ­Χ'φovK˜bΪΘη¬]zΦ’YƒUβκΛ­6Ϋ~ί‘Ιo.±Βƒ£ΗO$ ζ>ζ μ4| ²Υ6αϊ食OάροS½V}TφŽ5 gkŽŽ±Ξ:1AOΒ ζΪI³>ЌωΛeΒ‘Ω#­CΦ«U,­z5ξM tμτŸ°KΕžή₯M£ΔyΆ~”Ž?©·&υΨ’ΔOJ?Ήά` !Ήkβ\v€°’ €\xM°„ όΔ8#@ΒJH.ΐΦDrΦDrw― €υL±£ IDATΏx•FO^ Οί{MY¬7Χπ‹žzn€VΎ3JŊ°ώ,΅Δe)2~ΖϋΦc·6­œ‘δ1PƒΖΜъ/Ύ³‚˜Χη,Φ΄yκϋO¦€xί„pΑœgbsυHέϋ’„ “f/Φsλ‡S­ &αevwlώωO}0σ•  ¦­ydΦ7ίoΥ7?ό¬―6lΥ_;vYΞΤW{ZŒώƘ°«%a ―Nš―χ—―ΦΧΛ&μIβ'%ˆŸ€\n €°’ €°&Ξ a=$ Ήπš !α'ΖVBrΦDrΦDrw― €υmΪa Άlϋ+Ε’› fg‚y₯€˜³.φ8l΅έΉkΏΊτ§…Σ%φ{yΡ‚Κ•3ϋΧΞέϋΥŜ­±{ΏΦ,§‘oΌ£₯+ΎWK'&ΉΖ„,³|l9bB‚E―ΡWK’ΆIΈ !\0;4LXavpœ{ΐΊΉήτeΞ09χ§ΌΉsZ΅_hHJYΏρuλ?^wάRBo Φ_jύ'ŒΡμΆΉ»Τ‰έšhϊΫΛ“œ­βο'“ΔOA?Ήά a!$ aMœ+@ΒzH.@rα5ABΒOŒ3 ¬„δ ¬‰δ ¬‰δ ξ^ gλ›π¨ͺ!}Ϊ©ΔΥ—%)ό‚%ŸkΓ¦_΅|ξpλΟS @Ξ½ψbΐ:~β”u y‘yΟ[h&πθ=dŠuΞΘ²OΏΦδΉK΄ω³ 9ϋwϋ£gkεκ V8aCev$ί%’ΠqBΈπhύj*_ζfu{q‚Ξέ‰b2wY’ &αZs ϊUWI11―›PΕό“όΥΔ ­ώf“Ύ|¬5ΆΤϊOγΨ—;λώJe»6α--Y±ξ‚αΞ₯|J @.Eνœk@όδrK€„…\€„5qλ!ΉΘ…Χ?1Ξ€°’ €°&’ €°&’ €Έ{M€œ­ο¨7X_²›Η_…†&ύϋσ–_ώTΣ§ινIύuϋΝ%€Τ’―rηΜY―υ9οΰπ„Η^­]οΓ²ΛλPu³ϋΕN ΘMλ¨g‡Gϋ3η•ΔΗΗ[s Τ‹ΔOI?Ήœ„5’ ƒ„5p1δ‚Ξόδ ¬ Φ@j © yο}wלδL}γββU½IΥ¨\F}»ΆH±θ5šφ΄'±{Λ€ «ΦmT·Η[‡†7WEΕ °sΘά…+Τ΄~5k<1±qjΤφEe Σ‹=ZY€Zψα—Φc‘-7m>ΩO>I}:7³=ω];qRοO{Yξψ/Ibξc”kŠΣΓ{ȜC€ύkξζzsή‰9‹δ•1sΤ‘U}΅j\ΛzΟ‚^»κέjή¨†ΏφJφžfΞΘz€–9ΧČΝ#‹?^g,£t΄Ξ11Ζ©υŸ€˜έ&Ο<Ρΐ {ΎψκG ?O{>a[¨ˆŸ’ ~r9k€„5ͺ;@R%ς\Ο}μΣΡ–mκΰ‘£Κ“;§υΈ©&υͺκκεw£μΩwHΓ'Ύ­uίmΡιΣΡΊξš+Τ±U}U«xφ0υ]{jΨψyΦ!δaa‘Ί§ΤMκmŒΒω•.Ό9ώ+pIΈwΛ.CΥ§σγjΦ°†ΜΩ%fžζ±U朐+ŠR“zUΤςΓs9Μ|ΖΫZg˜,ž5ΔΪΑbvΞΌ·μKλΰσc'N©`ώ<ΊΉδUjέ€ŽJίv}’Gjύ'ŒΡ„&|ΌNίnόE‘‘YΥΈnumΫ(ΙγΏ„œBc? @όδrΦk UT‰<Χ€„„Δsϋ4O˜$ΝdΏ€Δυ%Nσ @Lζϊ @ά]β`@ά­Ομ’ €€—ˆŸ² ~r9k€„5ͺHͺDžk@BB⹏}š'L’f2Χ_@βϊ§y‚ i&sύ ξ.1ˆ»λλ΄Ω€8¨b *V•CΠƒΈ8™44ΞΙ$ψ ½-H&‡EBB’‰@‡άšΔ!…Κΐa€d ΆCnEβBeΰ0 @2;nE’ θάς‚ Z *V•$ˆ‹“IC#Ι$ψ ½-H&‡EBB’‰@‡άšΔ!…Κΐa€d ΆCnEβBeΰ0 @2;nE’ θά2(x–Ÿe ρΛ-BrΦΔΉ ¬‡δ   ό\HM€$5!ο½O⽚§6cΤ„Όχ>ˆ»kNβξϊ2» €ψΉ:@όδrΦ@Š , ΦΐΕ@@@ψ‘šHjBή{ŸΔ{5OmΖ © yο}wלΔέυev ιΆ@ΦS³ΔSεΆ5Y[LžiΔΟ”ΪφD @@@l\<ېij₯ΏΰΔ @XΙ@XΙ@ά½&@ά]_fG’nk€$έh=Υ1ˆ§Κmk² Ά˜<ӈΔ3₯Ά=QΫΟ6$ρlι @(½mΫTžiHβξR€Έ»ΎΜŽ$έΦHΊΡzͺcO•ΫΦd @l1y¦ˆgJm{’   Ά?.žmHβΩ€PzΫ Ά©<ӐΔέ₯&qw}™HΊ­t£υTΗ ž*·­Ι€ΨbςL#Ο”ΪφD @@@l\<ېij₯'‘τΆ@lSy¦!ˆ»KMβξϊ2;t[ ιF멎 @Ÿ/ύxΆ Ο–> ' (§+:#qE6 €QΊ¦#Χ|œΣm" ιFλ؎χΏΝz3L1±™?…«ŠϋΤΊeœΒB3,^Ϋ+μθΧR1χΦTLύΆ ‰Άv|„υ‰’Ÿθ­Ψ»ͺ(δπ~eΤN§[<«¬σFλt»ݞ] ξ.=ˆ»λΛμ.,@βηκ ρΛ-BrΦΔΉ ¬‡δ   ό\HM€$5!ο½O⽚§6cΧ ’Β~ω^YηŽVΘ‘RΦlŠ~°…².žΣOτQ\©ŠQψΪεΚ:oŒb*ΧUΜc]Scsυϋ .―@ά]_fG’nk€$έh=Υ1ˆ§Κmk² Ά˜<ӈΔ3₯Ά=QΫΟ6$ρlι/8qΦDr/ ‰s>qLŠΜγG”­χ£Šκ5^ρWίh½εƒι b±uφΗι#€„qyoΝ€Έ»ζ ξ/³#I·5@’n΄žκ˜ΔSεΆ5Y[LžiDβ™R۞(ˆν‹g€xΆτ ”ήΆ€λΈ8…°Zq7–:sΘΉΩν±z©xœρžE‘lWʞŠzvŒ"§ PLG[ΉmC·5$q[E“Ξ‡Δέυev ιΆ@ΦS€xͺάΆ&Kb‹Ι3@^a?SΦestͺίdk>ΎΉ₯¬ _³Μz?φΎ‡¬?·mu‘φ‘ƒ;(Ύθ•ŠiΤ^‘ϋw+λ΄—[Ήžbh.ΕΖ(rhGλ,Ψ:ΝΆυ[eϊŠ’ϊM‘―`QێnjHβ¦jž?wΧ—Ω€€Ϋ I7ZOuLβ©rۚ,ˆ-&Ο4"ρL©mO”„„ΔφΗΕ³ @<[zJo[ μ}š*δπσζύδ Š-[USΚ£θŽ―Xm²₯>tχλ|Ϋ₯l9[ρΕ<ΠB U–%³Άi’ϊΌ.…‡[ύeώŠBŽρμ£°@l|ِđecΠΰt? @όδrK€„…\€„5qλ!Ή?R IMΘ{οsˆχjžΪŒƒ1ρξι©U+cή'ΙηΜΊ HfΙsίΜ ρ³ ~r9k Ekΰb   όŒHM€$5!ο½O⽚§6cΤ„Όχ>ˆ»kNβξϊ2» €ψΉ:@όδrΦk Uv€€JδΉ φONξΠσΎΦ=…5₯pΥ‹“οOοΥΰCίλ§Σ”=4‹ͺg»B/ε/«ά‘YΟ»nΪџυΑo΅€Ψƒ*Q((Φ߁ƒ!zs^¨Κόߝ͟ϧΝβU Ώ/(lΌ:―VώΒσ&aM$ aM$ qχš qw}™HΊ­t£υTΗ<ΛSεΆ5Yv€ΨbςL#Ο”ΪφD @R@ϊόFߜޣ|‘Φ? @ώ‹=‘ͺ;?ΠcΉW›ά7kOάIuέΏFε#‹jd InφoμqΥέυ‘φǝb Abϋ£œ @•Χ‘€8²lι:htεudη Ž,›νA€Ψ¦’‘ΛΨβgA @όδrK€„…\€„5qλ!ΉHκΘΤ£[Υ2χκwπ‹Ύh²>jζϋUγ UNμxΚ‘­šs|›Φ^ή(Ι͚οωTe# kΤᡨΨ֐ΗΣΐƒ΄ζςΚ©xωTχΏeΊ3’ †(Ÿ! ˜ Β쨛€8ͺ\2X avΤM@U ,H†0gΪM@2žg²ˆŸ ρΛ @X) €°0@X I= ω'Aυ<°.Υ$₯ήϊN[OΠ‚’΅ί^xβ½~δ'}T¬žJόσfbb΄ά»RC³itΑ š}l›ή8ς“V^V_9B³dΘb&ΙfGέ„ΔQεʐΑ€d³£nLΘ­·ϊΤΈaΌ2AŽη–Π{r$qΤG8Νƒ%I3ΈD ΰΘΑΓΗ΄aΣ―Ϊ³ο jUΉ[… ζΥ©¨he‹<ωΙn0$qC3μΙόΫ@‚­"™;v€d0ޝ$}―£v[Ζ[…kͺldaλfβN©ΪK4«pu•Š(¨βΟN€μŠ;©κ;?ΠΘ‚τάώ―4₯pUˆ,–aΛ‡$Γ¨s#Η”*ΓJ’aԎΉQ0 5+ŸTΥΠ 9z8Σύ|ααŠ+UAρEΜτ±dτ@2Z#! "]π,Φΐy ,Šs@XΙ@/Y|ό/½pπ)XQ5³Ÿύβγ™}_ͺPX6 Θ·uσδˆυwίcΫυό―Τ"Χ šAΎJ αηDrΦDrΦDr` @jίw\UΏι©°ΏΝόBeΟ₯¨ΞCυ ™?– Hƒgπν@2œΫ@ΐw― 1st₯2ͺrοκ7|zb²μΣ―ΥwΨ4ulύ°Ϊ·¨4“Δ@@‘Hμa $ aM€°.&@’>ΘθΓ?Z!ƌΒΥt{DΑΔ›|vςκπ[λ‘VΩCΓ/€ =τ½fύEwF;Ekeθc<@ψ™AΒHM€$5!ο½Orš€Ε‡‘|ωxΥ§ ύUPL=έA’n΄tδ @΅}Q·έx­τl­ΣΡ1*]³]bb ΖLyWŸυ£–Μδ$iHΪΌh²+ƒ„5p1v€°>’ €>™pd³ϋC ŠΥRΡ°μInΠ~ίηZyς_e 9~˜Χ‘ψΣΚšUr\«W ”Σ¦θjΌλ#λ±XmχΣynSΛ\χ›£ όœH.ΐΦDrΦDr ¬  ψ°€Ύ  7₯Gg,)U³& ξͺ eoM1Yϋνuν?^ί2Ε26GIbŠf a€°@Xi ΉxU§™φƟ΄ >ΈAΗ|1φ€2αF¨B4θΰwͺšνrUΚv™~‰>¨w-ӌΒΥU2kή$ Λ‘Γρ§uΚ›δΟΛύο=M/RMe# +{HΥωo©Κq΅Ίζ½CŸŸϊWνχ~‘Ο.XW†ηLKi/Ή-Θ%ΣΉφBΧ–φ’'FrΙt½„$ΉΐrνΗݚˆ»λΛμ.,°€L­§τΪ NͺtΟν) ΛV~­£f黏&»ͺ *g¦M†$ΣθƒφΖ<+hK“)cH¦°υM @.€lΏ!UϊούmΌ’‰ ‡gΧ­;ήV§Ό·©Cξ[5βπF=Ό)Εφ?LyB#Ξ{οά3@ΜυŸœΨ‘/«§,!‘VΫ§χ}©qQφ(, ώΘfΚΰ@2…=¨oJΤεΙ”Α€€€dΚG/ΣnJ’iτά8“€<Φρe-”Ocv:/‰‹Σέ†Ιη“ζNθ›ΙSμν @λιΥή@ΌZω Ο›„5qλ!ΉΘŐ˜:Νβ­uCβ­zΫ™-ˆ%o΅!ρV½νΜ–„„ΔΞ'Ε=m@άSKf’6€ ώ­ž8IwέqƒͺW,­αίΦ3­V–,αϊΰγ΅ϊϋ»­"5*ί•Άyk /C†GβBeΰ0 @2Ϋ·"q@‘2xˆ  Ι@2ψCθ€Ϋ€8 H€Υ[αΒ‘o,Ϊ4{ιœ<{ΛgC©β…πηίΠgΔ<τιΪ Eθ §‡Σ-'$β–a΅Λ) vΙηv €Έ]Hνvˆˆu ož‚n~QxhJ΄[g{(‘Κ‹-‘α(¦π³w¨lϋΙV"k@ΔγBž­Γ@²•Θγ±r ’X @„α5₯]CΑH»O&@o0’SλΖΘ`>$<³ bμ™4nήΎέk¦½bΞ–οOπ;C΄rωλDΩdb0fζJωιwώ>·«₯{ϋpŒέ<Ξ³xmŽ"Νν,9󛇏¦’.φΥΌ)ƒ†υΘdDΤŊΛ‡PΰΜ―bYπκΌ;0¨^…‘oYˆξ9±m—ΧKc]φξeBΉΒΚ$S’Foœ:„Ρ¬^/Γƒ‡βη„R|ά“A©βN•€&Λ€@b"‹΅εˆz*~Nƒ+€οΆ…’ˆΣ­!SΡ5O€dΘΆ°¦(©Ξλpm’Ÿ³Ψ°IŽΨ8ρs"0€ΕG=ί‡ /α =a±a³ΙΙβΗ‘@ ΟG,ςζqFA8 EdΣξήΦo”Γ ώ9+Κ’οΗ,” ϋDαώ–‚Œς*·*ήωX­AΘνδ™ϊ5o”„w~ ΕύۍhΗ0>~Π™ Y©7μ$SWWHSΈ|]ŽΩR―.ƒ–ΝΘι}•`y’Oάσ™s„"r¨€`€ZΣΎΨ΄t*–+‘Cί<·„Ϋ9rπΔ9ώΰτ’Ε ς;@ζ-ߊ»΄@ίξ­xΓυF&G˜Η6αΐaρα oٜEσ&Ήφ1ζ IDAT (, 9Fs­€ΙΔbη§NηzA;vιΘ’Am}ΘT՜ f41X»™ΑU‰>½YΤ¨ψ²|`ΞΌyΩΪΔ0VαxnΗτ”~:=ƒΕ+Ιξ…E₯rφ焧ΔΟ~ΖΔ™πυjV2dHΕπΩ* ςηΠ©ψΠW:dO±ζxΣ7 Giΐ}^Μιg³G‘&¬\ΛJ€τλ#CΡ0iΌTΛ‘ψnΤψο{F¬Zɐ}eΖ—ΐά(Μ9rεΖm#V―“I€ ν/‡Fm_Np_ZTΘν#G"ΊYγΛ7ό—l€pI €°ΓηΒχ͊ΒI# ™³τ‡ŠgΞ›°u‡pnΫ3Rύϊ,Ί΄•C© 5nŽϋͺ•€₯PZ8₯€`€I§αXσε(”(&9|³ΗO^ΐρm ^±ν›ϋ±nΫaœ?όz ¬Θ§1p$ζNψ -›Τ1χΫςύXπΝœΩ»Ύ>P ,Ι…Ϋ% ’X.6‡M%°*―Λ N%°\.d7˜J`Y—˜J`‰†…Ϋ³;ƒ @WψΊ«Γ—«hP ,Ρ€—μΔTK²‘Ν0*eEz*%ZNfœ˜J` *%Ό¦4’k( α`ΒΣηq˜τy/Ι•SΪ}θ¦.άΐ‚ξ₯V™#3vΦ*ά…m+&½­ίώΈ…>Γηρe³ή(]Τ|τοΧπΏ1 ±ΓLώ\ ‘θR·’ˆΤ#δ|ϋ€8_s)ΟHDΚΡΗ6 @2+@g€ˆ³₯<+)GGΫ€ˆ£»”g%B$³tΊ”W¬ύΆ±_CΑ5 €pηc ™°χE’Fε²Θ—7EE†ΦΩιJ₯οζψrς€ŸA’ͺΥγ½.Γω3=|Τφ5›>‰F‹ξ£1oβ^Ω²iΧqΜYφpΊ7νqz0έtB nX;ά"b‡xnΨ•ˆΥN—€!bη"ς€ξ@< Θ9t‘Hσ€ζ@€ρ€…žΑE žoςφ₯‚Υ[bρκ]Ωj{σδ†lΫ8’Α„ΉkΑνήΰ0Μ‡΅[αΖ_χ°oΓLψϋρSNœ·ή5Ζ ιΑ»ί¨Έϋ0’ί՝rηίG˜Ίπ[ΌύV%Μϋ)ί†v€8"Zž7&Ο‹yvΙN!ΟΊOΔ³βm‹·@€±e₯xv žKή‘œΘ¬ @<λΉ@Δ³βMή:€4ξ0 εJαΟœ …εC …‹’ΏNoΐΒ•;pθΔoHNΥ’j…R?€'J—(dΆηΓώΣΰλ­ΑΪ…£ωŸ½HJΑς {ρΣ™?π,6‘!hώΞ[θΧ£5J€ˆJ·œ”ˆ[†Υ.§€Ψ%ŸΫu&βv!΅Ϋ! @€Ψ½Œά~ nβ;H$Η’Ή} @€Έύ2ΕA žoςΦ€κ{}°gέ I‚ξΘ€ΣGͺλ9cρœXΫκ)[•ςŒv@<#Ξ9ρ’ 9Y1žΩ–ˆgΖ=+― €PNdV€ žυ\ βYρ&o@ΪτΉγϋαΝ2ΕX<³ΨιηkΈK€ΈK$ΕυƒˆΈϊKqv RŒŠx6O{©ΞL„©NιΨED:±Š%@€ ιΨA„ι¬GgXBΔ*ΣRT@0R£Y?(rΘd²,ύόύΠ )κk›€δZ:κ˜A ”™ B9‘Q ”™ B„=²S€Hv yή} žσμ<&B„Hv«Δ½ξq―x’7Ά+ ±}JχjIΔ½β)–7@ΔR^ΊσΊQόq κM_‚)[ΊώΣr$¦ςΤ¨·.Ύϋη0Ύύ~Žϊzbc žυ¬}&B„=²S€Hv yή} žσμ<&B„Hv«Δ½ξq―x’7Ά+@Δv­,Ά$b§€ΤW€%Bf$ @Œ¨w―„βχ`½}Α.•#"Kˆχ΄Ύ€6ϊC €Ψώ@lΙÚ!BΔΓ}.ά%’ ΡάΌ 7p.ά#B„H.Ž w!βΒΑ#ΣνRΐ.²qη1ΌY¦jU}άΫrυκΤΜ–f.Σ†ˆΛ„J†‘txD1NΚDώΰΤηCχΩ¨·/”ͺυš€LΕΝσ0΄οGΔ† #bƒHΦ„ Άθsα.\ˆζζ]€Έy€sα @r±p\Έ ™n—v {£G‡p|1Έ;ΈΆεΊyrƒ-Ν\¦ — •€ %"ιπˆbœ”’!S(ΑjΌα΅τ‹ōσP―›ν€5ΠLϋΤ @dΡΠΜμC»Ύ06n›¦yjΌ' SFΠw$J€2)©DB:v!BD:λQͺ–‘jdΔ³‹ˆxΪKuf @€Huu:Ζ. ŽΡ•F•Ύv?ίAώΰ χίΆ\Υ+•΅₯™Λ΄!β2‘’΄‘@$QŒ“4Ι HNˆL§…fϊ§0„w†±Qxΰ• Κ»‘:Ό Ϊ)λΑζ ΰw—(nόνψUθcvΥαnz- ΕνΛЎY¦x9J€ΫƒυxπP&ΊJπQŠaE·Ε“ B„ˆ'?lσˆm:yR+ žmΫ|%B„ˆmkΕ]Zq—H’9U€HNΛԞˆRw ”p'"‹y ο =ψ²W―\ά[}™ ¬Rμζo)―USa*U0θ ±θυ~˜3@<0θΩΈL„z.d§μςΌϋ@0΄κγ»ν-Ώδζ#"Ήˆn @D_†’7€ˆδCδt €8]rΙOH„Ι/SA $"¨œ4˜ )`7Ι©―7OnΘiI·'"ιπΈŒq@\&TN3TΚDφ,²Έ§ΌκοWƒUͺ`hΣ›7[ X (ΩΕ…Ÿ ΉΨͺf™K`©Ύϋ ŠΫW °P©ω1Tί―‚vΒj°!aNΣ^Š‘bTΔ΅‰ βAW˜ˆ+DΙΉ6qή0 @\a₯ g#α΄€‘\K»Hϋχβν·*Ωμu³Ζ΅lnλ €ΈB”€o#ιΗΘΩJ€¨φ¬κψv‹’θ{„±n3πmNμDΚΧΗl ς;Χ Y4Ί‘saz£zZώμ!€ΪΪaσψrYžzρΤΘ[χ› τ\ΘN Ω)δyχ €x^Μ³σ˜ Ω­χΊOΔ½βIήΨ€έdܐθήώ=Ϋgt³–@ά, "ΉCD$α%<­”ˆ„es[Σ€Έmhsν @r½|<¦# ΅ΝŽ±Y*iH„YξΌ£@<+ήδνK€Ψ™ @쐺σ ‘DΘ¬Κ‰Œ ‘|Θ¬ @蹐@²SΘσξρΌ˜gη1 @²[%ξuŸˆ{Ε“Ό±] Άke±%;€ξ@(,*@„ƒε@V !B„žΩ)@$;…<ο>Ο‹yv!B$»Uβ^χ €ΈW<ΙΫ° €΄ι=Ÿυl–MκΨ>£›΅$βfΙΪ"’πž–ˆ„ƒ#‚i΄DΡ%>% @$ΎH%` Ab&‘X@$` @$°h'ŠMSIJ»ˆ€<Ι " οfΣq³€ ΰDt£!€ΈQ0r… -&7†ˆ7—Ι₯pn܍ nΌΐ-ΈFΔ³βMήΎT€ˆΩ@ΔN©;―J„Μ H €ŒμώΖ§’«φSβMIΨβL#€8Smט‹ ±VΕ΄’ˆ˜κKsn Œ‹˜V!BDΜθόΉ €8_sšQ ±3@쐺‘°¨€”Θψ.#`ε(€eE–©BMhΜ„L&Ί)N5€ˆSεv‰Ι€!βKUT# €ˆ*Ώ$''"Ι°ˆj @D]‚NŸœˆΣ%§ %’;AΔN©;ΚΧ ίŒ©νΐY@D\7JπQŠˆ‰(ƒθS!BDτe(y€H>DN7ˆΣ%—ό„@€‘ό2Τ@ ‚ΚIƒΉ@μ ;€ξ@(€δ L€δ@.‡4%βYs<( @rΌl< yΆΙV"k@„ΟZφ@<+ήδνK€Ψ™ @쐺‘ ’ƒ "~ν/ 9HX6%BΔeHr"ΤΫ–@yρ$΄V)TΒjπΤ[—@yκΐkχSg}6 HI„zΟ(€&ƒ-P†=`ͺφΆW›λMΔucη(Λ €8JYΧ— »~sc9ά¨F}άAΑHl|".^½§ΟbΡ¬ρ[Θœ©Z=Ό5jwΠλ5€ΈeXξ‚ξtΙ%?‘δΞ‘X’ζ "ͺό’œœW Α©ΒkωD˜*ΎΥ©ƒΩ΅³• †Φ½_qυδ x­˜$ΕΓΠy XΏ(Ο‡κπ&h'¬V\’kUL£€ˆ©Ύ4η&"ΝΈˆi @Δ\ΞŸ›ˆσ5§₯‘€ δ«5»±~ΫaŒ&ή»ο–OD•ς₯°ΰ›νxτ$ &€J©†ηYAD !=| žά'b9'h푧Eš@€Έ y~2… L`~xλ–-ρZ6L±r0΄ώΘb€•?οσfu0‘EΝχ}ΏύG£a¬\šYƒ©f#Z₯υW\;υΊYόΌlp˜Η=>$ @„ά”!ͺςw ™7ϊ6½alΦΥγβm‹Γ@lQΙ³Ϊ!BΔ³Φ<ϊ7yϋRΑΘφ}?aΪ’x―A 4Wζ5ƒ?œΓψ9k0 χψ¬gk·Ÿˆ[…S4g€ˆ&½d'&B$£΄D²KU4Γ€qΘςfΚβžΩ@4sƒΝ$%@σLhΫ| SΙς―\¦ΣByϊ ”GΏKω‚!Ώϋ'ΌΎέ+ Λ΄>0„w†±QΡΦͺ˜K€ψΕ »+Θ¬―ΙΝάAEGΐΠβC Vˆ˜+RšsΙ€˜ΐbYό5,JΈŠΙ΅πqή7­σ@ς}όοΩΙΧξ/ i€Ύ₯έ})dΙυ› μά-ώ—8-κΦeΠ<ά€aβc― @μUϊ»ͺ‚ŸNB₯7JbΚΘήΠι ¨ή΄―€pβ,Z΅?Ÿ½‚ύfΊͺVν&βVαΝ ’I/Ω‰ €!"Ωε) Γ€qGβ΅`ΐ²0ΎίLή@(Ο„κμ1hΗ―δaHϊΕ½θ–ίΏΝοΡ}2l‘Rζ{ͺ]+!rlρrέϋΊ‘σ™gΎ8‘"ΙϋDΨ]AιWΫ Εν«Γΐτf5€Θ£B3«?΄g€)WoͺΪ΅‚o§³ P*%ρ†fώPj¨·,2xίΡ&¬„μι#ΎD–nδW`Š”φΘΌ β‘aΟi Yθ"Eq<εΊω•Eύ'»ρ©_ω,ΘΧ Χρcκc|ΪΒβΐYέΧ²&4‹ΨΦΎ%02_UΎ©0ΰΩ/ψ±`[Uϊ9% €8EfΡ&!"šτ4±Θ @j4λ‡ΕΣ‘AνΚΘΑη0υΛ Έpd₯Θ. ;=aυτΤΡ€xjδ­ϋM„ˆΛΖΥΑPωϊŽύal>Ϋ…-KŒ‡zη7ίψίΪn*Wϊ.ƒΑζ ΚΆ―§7 BΔ#υšόθϊOΎ°t―/?[€4τςχe 1ΠLο MΠ ˜¦l}\ΈΙvWɍ‚©RmΒ;ΑkρθWGL•κ@~λ"˜7ͺΓΠͺ—Ηζ ½UΗ €d @˜βεΜ l ³β.αdκVϊβš.! o|δχ>τ+ϓέύKΊgόŽ“γa­¦πΕ;{ΡίΏ"?†³. ΞRZœy€ˆ£;Ν*Ύ‚¦#4$‹¦z €M&|‹Υx{δ3ΕΘkΛ΄+ˆ?ζΜα΄ίrΕλ€<ς>4³€-P©cΏφΘWι:ρΘGA–N±"Ο%°r @fΔ^ΔΡΤκ_•ΤA8§Β΄Έ ˜Tς”Fvχ9k¦Ε^ΐ-}ͺyƒ"ΫC›9υ  ξύœ βήρ%ο¬+ 9ϊσyŒ˜Ί5«”C“·«cξΧ[1°χP©”Ψ{τ ξ?Šβwˆ„7¬ιVρ βVαΝ ’I/Ω‰ €q%Β½d”ί ϊž#α=ͺ#τ|š-Q\ϊ…‡&©3·€υOΫρΑΥk—ί½γ[οyτ *[L@€Έρz™^ Ω‹8hζ „nθ\0Š€υ>~ίΉεΥ³Πwΐ»£αvs0&θ? xϋBωΛ^(O‚vZ°ωωoϊ#5 ϊCψη†βΪ9¨w| ݐ90½QŠ ?ς»ΚR'―|ςΐkΑP°EΛBίe-ΛΚνΪΈ%Ι°+ˆ+‹ΖŸρ1xŽω˜Χv€PάΌυκ€Z νΔΥ`ύςΉ]¬muˆˆ­JyN; ΒK£Mˆω —uΟp¨`k‹“eΎŸΚΉOM©ψ©ΰ(’ΜγΤ„$βTΉ>§KNJDΑηΟ¦]Ηρ՚]HΥκ_qΟΗ[ƒΟϋuB·vM$βΆpfNKO‰ˆ'Gί²ο@€Έ‘Ε=Β›μ3ψ}›ˆzέlΘR’ 4Σb°eΡΠΜμC»Ύ06n›Φ&5 ή“?†©F#}‘™.λΏ7nž‚n~QxhJύ—K U^l G1…γλvΗΔΚ°i‹±qβψΐ’gwχΞζ‘DζΛψvK軃ςδ>¨φAκβ|YB,Ώ Dρη%ΐd[Έ4τϊ)V6ν~b…‘σ@ίz7­ύ“»πž3Aξ‰₯°\€d·+ˆκ+ΐzϋšSL¦M«Pς»=΄γV@–š ―ι}ahϋ17~ηΫιϋLύΩ –@ΔR^Ίσ±›\ξ±4ΪΖΔΫψ*ώ .ιbq²Μχ£)x'bŒ,ƒ š ΖΉgY‘ξzΒ2 B¨HcΈ’‚N€€δT\ΌzQΟbωmzCCP£rpΔ/ ξUηϋDΔωšK}F –#dͺP ځ³ ύžSSˆa€υxπP|Η•*ΰ£&+’vmζΛV’™ΡL™Κ`}ό 8w r6ν ΞΑζ ζ‡UžΨ ΥαMΠNY6oΤΫ—ρ/°΄γWylI›t½ €X_‚@ΔN€ @ΛΟ §>@=x2I·+ˆΫEāρŒ—zέ,°₯*ΑπN[~—w(Ί,ζ)ΏKˆ;Ζ{κ§ΠυSυ™@<2μY:MĊ<Ή K‘΄ΚοϋόWΚΐ1ηpΫΟŒžέ}ΚO’D€\ƒ*^Αψ&αNl ΉiΙKΔiR‹2Qd§I% €ΰD>9Υ N•Ϋm'#βΆ‘Ν΅c@,KGDό›Bο±iί‚3U¬ cύό‹)χMή|AЍZ’– ξ\6aήλοΩ‘2LΉjΉ^[ξ‘Μ HqqŸ8R AΫ…έ”YαŒ%°ύυŠIΠNX 6$ŒoͺΑؘsXά-}‹e{Oς]L‰½€_ }ΉΪFBeufΥqZϊqšΤ’LDDΩiR ( (Ή|γoœ»xρ/’ΐ0–Ώρ5aXO Έ-œ @„Σ“G"βΙΡ·μ; pΗ ή#;€ΝνΔUHίΓΥiχZ6Ϊ‹ΐ”ΘKΐ^;{ ˜€0e«ςesθ€!BO‚μ"ΙγK»‚²‹›#οq€Ί96¬Θζ FŜ}­Q)₯?Nn‡Ώτqh±‡ ΆFuL`± ξ2v'‹“%UώψŸtπ-ŏ‘Υύη¦T4~²3kγƒ<%ωφκcρ~δA|W ©ΣJaqΝ΅l«Υ@lUŠΪΉ›‚=GNcΒά΅Ωκsσδ†lΫΈR -ιΪJDΊ±Λ2 @ά€Lθ¦dEθ>kv•«χξύyθ» ƒ±AKσΟΉoτ*n_†vΜ20ΕΛ‰΅,%5/ @$΅$%i I†ET£€ˆ*Ώ$''’5ρΔϝ@$ΉT3Šˆ`R@.¦€`€U―/P°@ωƒ PΘ-Jθοbem.· §hΞMzΙNL„ˆ»―%iΰƒ«Λž~Ι΄)πώΌ-t‚©NSώNJλΏΓkΥT˜JU :θF,δ–?cHvA;ΐ0 @€8`aΉΩ@ά, ΈCDέl @2+@ΔΝy&w€Έw|Ι;λ @ͺ5ν‹]«§’T±‚₯7 ·Γœ%β0i]v` @ά€πœXνŒ-`ύς₯ΑŽkηΰΕΥkb9˜’eΐ―iŸΒT―9Ώ#D3΅ ­zΑψn{—]ΫBN„ˆΤHο^ ό₯QξHV΅ϊm‡ˆνZyJK žiΫύ$B„ˆνλΕZq‡(’ΉQ@0¨ύPlZ:E ȍ.Ϋ‡ˆΛ†NR†‘T8$a  @δξΊήdΝ’±0ΎέΖκ ψ3ΕΛj5”Ώμ‡βΒOЍ\Μ<n°ω‚alΣHI‚zηr0…JB7hίFυέWPάΎν„•€*m Υχ«^9ΠV V#€Xύζΰ)θζ…‡¦D"σκ”%Ty±%4<c IDATΕ~·EJ‡ /Ζ OρύπΊqΞα~Ϋ2‘n3ώ™$ƒΜ–ζnΣFj€w/|}$‚£X.$b‹3ŽˆΕv‘©€X ”΄ƒg{dιUΪβ"‹7—fΙ₯pΤΝε €Μ^Ίωόσ ―Ά./JN ’΅¨­5€PndV€ˆεœ0U¨νΐYιgf{LβHύtΝάΑίΛb›/ˆο§Έuͺ ‹|h|ahΤΖέs&΅&βAΑΆΡU @2* υX6¦55P @€ΨΎ €ɜ-΄Δφυγ¨–΄ΔQΚζl\ ΦΘ¦αc0δΕ…œ κ ΦƒςVΖΥ€8H_O–ˆ'E›|Ν¨€`ΔSeuβ΅}δ_|ς‚Νγ—-Q―yΒsθ:„L‘δk»³ώΠwς'χ ™3ϊχ{ΒT'ς¨‡Po˜ C«^06hε©©’₯ί@(-2+@„z.d₯ @lF!B$MΪbϋsΓSZ!’YΪβή«Ÿˆ{Η—Ό³;³Γ]ˆϊΔ.ΆzΗ2ΘR³ ²ηQπžόΑΆ―ΧΔTή Ε§ °Κ¬ς§=P=ŠΤ +‘ξeB‘BtΊ˜O @€ΨΎ €!BΔφ'†g΅$B„ˆg­y žoςV gψ<„7ͺΙCξΏmΉΦ.mK3—iγ‘δΘ¨ŽnεˆΎΥG%%ΐkέ,°Aψ#ς;Χ Y<ΊO'ΒT½d1O‘^?Šo"uΞvΎT–,ώ9Ό§υ…η¨·,„ο$0εͺΊLά…6”ˆΠŠΊώx@€H€xi€‘­#Ογλ’XhάοcF’°Ε™F!BΔφG„ Ά?1<«% @f€,6©άί‹V.ş—ΰ΅d¬Ε»ΊΟ¦ΐT΅ώ+χδξ@3o0τmzΓΨ¬«–yV3 @€x֚'βYρ&o=€€€κ°pε;yΙ)Z”/[cuC₯7Jd™ύσ³–lƍΏξΑ/Z6©ƒαλ ε/ώ=€(OξA2~8ε:ΧLνƒΤ™[ΐζOΣ”e!K~Φ7/”—~jΣ€.>όχ’J΅w-Έ±Έ³?tCη™ξ‰ ”ˆ'F=kŸ €d@nβ0δΩ<1&αFQλԞNΐ‡O[pmώwΡά§(ήy²w ρ―΄)ςΓ―…:H"9XΏQΕ‰ED) @€ΨΎ €ˆ»ƒˆνωκΘ–R<]}ρG¨Ώ_ SΩ*PάΊ˜%AΟWΘxΙ ΝϊΉH΄lΎΰ—·L&hζ‚,:†±’X@€qδSWzc‘^LΘ"η(`Χ ύΗ.Κ±•+ζ|žγ>Bt>e9.ίΈƒ±ƒΊ!$([ΎΏ^Έ}λg’@H€Ε)"ŸΖ }Ÿ‰hT―*Ί΄yO"ŸcΖW›ψζΞ=α.O ς{B3oR§n›Ώ―ƒβΪ9x­šΚοαQ—y ¦·š˜uU―™™6ΊA³ψŸqίΖρZ4Ϊ‹ Y5†π.06l%D¨]r  .6‡MΔ²Ό¦ ΅ 8 {RώΕΜΨ‹¨« ΓΙΤ'YkB £}eΐ[ΊX y~?ό”>¨ώh;†η«Šw} ›Ϋ)Yς+₯q.λˍv€ˆb38Tώςl_θΠη’­ƒί<έό’πΠ”hk‡΅+‘Κ‹-‘α(¦πsΨιΗΔΚ°i‹±qβηDΙβ > ψšΓξ·-?'€Ψ’©Žo#Eβυϋq˜ή¬ων+Pο\‘5Ι,ΓπΓX―9ŒΪΌrWyl+·― Σ›ΥxΒioVhΞSί^΅kίN;f T:>›ˆ•€…aΣπ1ςβ‚$"6(oe|PΝ)_½~S»Ε½Α _·.ƒζα&Θ {$‘ vAΔN©»Λ*`©ώiŽΏςÚχ±·ΓΓ'OΡ’ϋ,›5οΤKϋ c0šΠΌλ(΄ ―k†™η™Ύh#ώΈ~ί―n.­qξβMŒF4¬S…oξδ-δ ±iΏ_ ™6ϊnΓψ³ώA€\Υ•`*Τ„ιΝόΞ―C!“+ ο>¬A―υsΐ”ͺ}χa%ΔΒ{BΪ~cνχ Έρ;_κJ;d.˜²U£šΩ`¬ΩΖέωΣΥ«g@;aΨΰP{Γν’ύ €ΈdΨj4Λς¦]Ιΰmο‚8«Δδ˜σYΧώV‹χ#’kž2ψΘο ώv©›πmχπΆ&졉1$ YΔώWξO‰=_΅Q8Φ *™ά‘Ήΐ NΔΊΔ@Δƒˆυό$BΔαΏ ²˜ 2J†[δHN9ADΜLx9·ˆβΏQŠίOδ€(O„ςΜAhΗ.ηfMΏdOC3(5ΈΏCΣχξρνPœ=ν„•=}ėȍό L‘’“­ bEp NΞDΛΣe Š«gωυΜ”ΘŸ=kν²΅Tžκ‡PόΌς€0aΕ‘οψ?0e*IΒoGAΔ‘κΨRVΐ."eΗ2ΪΆcΟ|« GVB₯zω­Ž‰σց+q΅sΥ‹4ξ0 =;6EŸο[uΥH½’ΰ3Ή·ESηlγ!ˆχΘ06λCxgΎ,!κνΛ ΏuΠψΐT₯> νϋεNΗύoGˆjΟȟG‚ …‘Mo˜ͺ7δο©φo€βκ―Π~±Βό-υڐ%&xl), ς4qž@,k@{؝όoŽΘΖΔΫ؜xG Ά†2hY@Ίϋ•ΕEm4’YjxεΗΔΐZS€νYžpΫΑ‰Bmρ―!­"aΨϋ¨ rJR±.3ρ_l±žŸ@€@Ž₯<Δθ˜s¨ν•«ςΏc5xsγώΐ’„kοί*Ϊώr/pc-JΈ ”η‘©ΠΨ»&ΤD€"νσ°Ψλ 3@ΔΞN γ όߏ9 zΌ'υ‚ΎηHpŸΝΛB³hL•jΓή ^‹GΏ@ΐ˜x8bͺT‡[–y£: ­z‰/ˆH±"<‘2ςΥiΣˆΧφ―!ϋΰ“lΏ,ˆ-₯ς”§Bux3τ½F )ՙÐί:νhχί FD©MFˆ €GωΛ·αΔιK8Άuώ+―Ϊ|«·Β…#ίΌ&}Β‹dΤk3ΣG‚ί.έΒ™ Χ‘V©Πς½:Φ·TΚ΄ΓΏέ€Τ­ΓΠ†B_Ζ) €ˆ NOΔrPμ ©ŒυžμΖ’ΰ·ωUάυά”Šχ"φ£& η} X̊»„ηF-<Τ29L`Ρ6βΒ}ŠΰηΤ'hθ]/™ε¬‹ˆu₯ €ɜTKόœj ¬ ±Ώγ7έSΘ½ψe@=YΓ+ι΅%ρN₯F`XKόmˆGψ“}Xς6Zω”ΰK-φ>‰…«σΏλ¬_YΞC„HfάiΏϋγδ>h'­~ΕMξΕ¦ςΜahΗ,δŠΧWŽ9ς>4³€-P©cΏφΘWι’!’Y)–ΐRŸΨ CΓ6PοXΖ—\ΟjΘk΅P*―XςΑ§|U’Μ—ςά1¨v―„vς:°~ωψ­ψšΉƒω³kυ]‡Hβχ»=F±G=κλΚ  @’ŸΗcύφ#8{ρž>‹Γκω#QιΝ’όYyύ|³=pάQBNšΏ7oίΗξ5Σ^ύ#ζϋόΐλ?­‡\ώκ‹χF’U―/ˆΒQ«ΪΈzσ,ψfzulκΆg€8*4nΦ ‘ Ι¬ΗnχΗ†β§Bd™t©φh;6δo‚&>EψΆά!ιΝ#φ£”*ŸΣJ_₯I„Hfθtλ9A„HζμH?dΥ‹[θ•χ Lˆύ /Lϊ,Hζ1"LΙ²;C›‘Ό:ϋ’ξaTΜ―ΈS¬‡ΉιΚ„XŸψ~+άޟΑccv…6ηοG™RΠθρ,ΦΎΕςΑ‡ˆu™iˆSR0ΛIμέβ5(˜Κuahφ‘yYόσ΄3>Ο[€σΧv€p»Mn^ΰK0C­†vβκ΄—œz±xΪ"‰‘ω υζ…9 ™KεΙbžς%Ϋu½GCul;d1Q`Š–ώΓΑ` •L{n|=¬_τ½FBωΛ~¨Nμ„vόJ°iœiOp€Ψ£υue "’Ρ΅tΔ%$’HΑόΰώύές‰¨RΎΖΟYƒΓ?ύŽMKΗ‘bΉNΧ+7δο{ρΑΗπaΫw1ρσ—[b­Ϊ‰»Žγόαoψ] z#“#L ‹γ?›pΰ°ψœr†·lΞ’y™PŽœ’Ζv)`2±Ψy€Α©Σv #Xη.X4¨­€<½Ζ`#Σ@Ά*`41X»‰ΑΥλxNLhwωV ηΟϋβv€x™΅2­Ξσwqcδ“_QΡrΏΜφ6ώ{ZωΓΘόΩοή(rs#¦„ΦBŸ 7ωaŽ'>B·ϋ?ΐ[Δεr¬t^‰žΑβ <?'4ήΐ˜wNΓoν$±ΣŸŸiΤš># ό/'$a”Œˆ‰3αλΥ,’žŠŸ\ ¬!ΐg«4AsθT|θ)™CΠχkŽ7}r”άηŜ~6{iΒΚ΅¬dAοψ-ΌI£–ι£π ocώΒΣŸDΌΡ€mΕΓmŽK'_鍅…κσ}" )¨r{;ζ†ΥEχΐ²xnLΧ¦†wζͺ‡x“ΥnοΐτΠΪθXέόφ{«˜νsΪlœ•†ί3bΥzHζ ^ϊ/Α};_ —iπ δ©ΣžφqχΖm#V―“Α`? Ϊ_:ν3ρτq6-χͺμs„ML@jΆΠL_ΆXσσpό{6.…ΜΗΧμ ›š (”, Ν¬΅@J2RΗ|UηΎ0]9Λ·σ'Γᣎ?4ΧΓZ4eΠ°>k>δέ–>ΤFXΈ‡ŽΛqζWaΗΝνh;0¨Q……Œ £εVB»ϋqΟ‰m»δ’ γΫ]FΐŠ’ Ζ!³ΝθξLόžύŽΫ%»g«{¬I‹ςχΆβ‡"mPΙλεΩ7u±X›p sCκ™4jLA₯ϋΫ°»P 4πC’Iφb\`uός˜Ÿkuθλ[¦³5"— Œ&πœJ€Œn|yΧI€˜΄‚©Η°ΧvrζRj—ι–˜ΘbνFΉdΘΰ ΰ»Mδ֐©θšG:d[XS”TηuxnE?g±a“\2δ“€o‘9, ’ο9²-̟w‡EŸΖ “λΒl+UuMχ<9‚ Ε:!(Γω'S"πIԏHdJe5φ)ˆMaαΠΘJεO~„‘Ρ§1'€Ζ>ϋ §‹ΆC°ΒΫṐ>Α£',6l–!θ_Bu&ϋ—ΫΞH7`:δUλyΉ{XΏQ.χcΚ€ΘL&ΰς―Pμ¦Ι«ψ`σψσ/"姂5™ΐΎΣ֜²[— X<ΖεGΐ*U/jΡ¦ςίΟxΙΦΜJWΣδΐ?ς_‚ϋΈισy@B ”?IΩiηUzΪuϋ_`ύ·x7!5²yψ ~qA)18oeL ©αψχ6,πΗ5vμ–«W—A«ζ¬ωΛ™ςo@–’SΛηψf–όδΘ~ή ΣΤ΅/Ÿ·―@±`L£ƒM?τ\› εΠΆ0 ™ΆBMΎ­ότaώyΑ4n ¦ϋ0IδFϋ{ 1 A Έœ‚ ΖπΟ:£]‹Πι ―N•ΗΟbώŠm8΅g‰ΣEΪ}θ¦.άΐ‚ξ₯V™η;kξ?ŠΒΆ―ΏΔ1™ΤjρϊΤ}»·2χωωμe χNla‚\ώ ΝΤ­Ν½μ–ΖΕBόo˜‹‘•ΐCuiΟI%°,Η'ύ g¦e,Ž&?Δ—ρWπC‘6|‡™ΏCƒ;δœ;·γ#Ώ7̝N@·§Ηρo±^όΉιWœI‹·ŸμAKίbμ_)¬“bΞ#ŽΡαHΑΦP@†Ρ1gρȘ„­š"ژ‚Ζ{1?¨>ίΗ•ΐ²2"ώop:έz~!θ ι%°ίvŒω5G%°ϊ?ϋr/Μ ͺc6wπyλΘC˜ψZϋ–ΩGŜεαΗΪ g€ ~v {’οbEH#Ύ3/*e]m*εΜL΄ζ΅FϊOΖΑXλx­š Φh€~ΐ sεΉγPο\”…{²u$c ,Ε_@½b΄Vƒ γϋrεqTΏυΨRXTΛJ Q ¬lΧ–3Ψ[ΛR©<ω“{ΠΜθν„U` ½ό½μ=¦3τν>…©NSή5Υή΅ό9CάΩΊ‘σΰ.ĜJ`9#si)* ©άδ¬[85«”³@ΜΏŽAγγΚ‰—δΥY‚D>αws|9yš5ΕOΛνJy―ΛptoŽ½ό6IF›8Π‡+_εy_oΕχGNγμώ―ωo›Ίϊ!θ}[=FΙδ?㫇<:+6ηaΥ˜*Υ›χε7³Ε°CŒ9 €ˆ‘Ί΄η$b9>ι€Φγˆ4₯ΌΦhyp#΄ΝSύ’† 6ζobn³3ιLŽ=[E»½ΦοŠώ9fΖ^Δ5] PήΦ„ab@MPϊΰŒ6½£amQLεΗχεΞαΐΛΟΫ"0Γ·•U@¬+K„Hζμ 3@ΔΟ ©‚ž’ΒQρΡwΨΪΥ½BΜι6#φ".ι£±'τ}σΟ.ι’Ρ&ς0.ξŒόJ0`Ρ.ςώΤΗ’ΏE|ž/ϋς‹Bώ!b]M BfZξΖΚ|HξF‘^φ*@Ċ‚@μM-AϊΫ@dI ΰ †vΜ2ώŒσe4Β{T:|γΫύON„Ο¨Π[¦leΘ܁ג‘ΠŽXΝͺ)0„w±αΛ/F βœHƒIxšVt οt†Ύέ[£[»&ΘςoχaχΑ_πγNqΚ#L˜»§Ώ†qCΊ#80Φn=„έΓΎ 3ΰŸφBkβΌuπΦ¨1nHΪa†ΧΊ‡g uΣzh~Cάψλ.Ϊ‰A·Γ§έZςm\€ hyΕw<ώΉθΙΘ„‡~Π,0/Έέ('@ΔIB»Π4@,+€xZΝn Φ/ρ_vΣλωI;@\ΘΡ”‡ώό nνφʎi€\ΠEc_ΨKς›6 ’Žβj‘.|©«U 7ρ], n€φObθϋόκΞΊ€Ι¬ΐ£G2lΨ€L ¬ή½Lpπ‘ΞZn.; ™Έ~S)Ε&zΘβx3Uί―†L› }·΄’T¬ —§/Ε006J« ΐ]Š?/ΑkιHYrPΎ¬ÏΓνξ8wΊ>γΑ‡Bυύ*Θ#@;α~Νμ0Φl c‹ξPά<υκόŽ­«_@\=‚dn €LY°?œΎˆ…S’j…ζXά‘η‡ό “¬Gϋ ^9P<·Fη¦W–kαΚ8tβ7$§jQ΅B)Œ₯K2χaiπυΦ`νΒΡζŸq;WΈƒΟ½ωόΠ±e#τθs­q Ή‰†ε>@„Σž‘:΄cPΉ’‰Ξ±GDϊ±,"ρ_vs‡ j|ώ9„ˆψ9AΔϊCŸˆtˆΎ^3“?ΆΨ>uΞ6‚δΈTžΙ”A~0θΑ”}—Αi0d(ώ ν+₯’ŸW½vd‰ nQ ‹ˆM©GάPΑΘσΨt0O’žΓ?―/^$£@HZE εΗζeΰψC ' Β©MD8-ν‰ˆ=κ Χ—Œ Πλk‹ΜΩA%°ΔΟ )–ΐϊ§V=ΌaωΌ€τ’UnΕ |•πΏΌΝiυωσ3xΑθ_9Χ#ύζ‘δXϊβξκΰ+W‘Ύ& γkς@₯cΤ”Vϊcnp=Ύy*cΔ»{Ρ9Oi§•Β"B$³΄DΈΟκξ2 ™άpϊr¦Ο BͺIcΉ’‚Ξι„Δdlάy §~»†¨θΘd2 FΓΪ•Ρ³cSψερq%ml²•ˆM2ΩԈˆM29Ό‡KlΣ@€±i©€ˆψ/»iˆυ\₯ ΩbͺΧάm0΅νι!BΔΦΥβΉν€!βY럈gΕ›Ό}©€ Δ…%"\Τ €§₯=#±G=αϊ!BΔΆυD„HζL‘ βη„w€±ν™κ¨VεΛ=u  :sΘQSδhάτXŒŒΕ²ψkX”p“jαγΌof9Ξ±”‡|Ϋ Θ#S‘±w!L¨‰…†οχKκ̏»ŒΫΖxδ•©ΠΛο ΝW%GΆ9²±”v€/Ζ’w/†+ί/‘‹X‰˜βD3€!βΔ'©€H d‚(  @^$₯ 1)…BƒygRR΅ψώπiΔΔ½@³Ζ΅πFι’’8ιΘI €§.α΄΄g$ φ¨'\_ –΅4V©ύgSρΚI΄ΒΙžγ‘ΈΏ“ρͺ‘J`Y gd`ΦKƒv€XΧ‡v€Πb°ν±.&@+ΤΒ‡ΡΗͺπΖΊgδ_9KrǏπ'ϋ°(δm΄ς)F‹ώΡ'’Π`uώwρ—>-"π%Ξ:ζ)…τ ςό4F䫊ž~εŒlξ‡’©]¨xsu IDAT‹A»€₯ί»{‡„κ)“AΧρ`Kd ΐ„šNJγ!BDJ+ρΆqΌΖ4ƒ4 €όsο z ™…^šα½Ϊΐ`4‘ΛgSpϋίGΌηJ…›Ώž€Jo”¦Ή΄ŠH.…³ΠˆpZΪ3{Τ/ΛZ&·θŠαuΛβ¦!V8±s9’F¦ΐ΄ :¨ͺΚεΆw#b]+ @2gν?'hˆνΟwGΆ$b]]€<-_?€>B7Ώ²¨d7>υ+Ÿ%Ω—t£b~ŝb=Μ―LΈυ‰α·ΒρUόUHΉۚο―}ρ'Ά%ύ ΆΑ°ηgπؘ„]‘ΝωϋQ¦4zΌ ‚룡oqG¦‚yl©Œ†όΦ%§ψžε$r9΄Γζƒ)SY|[œl+‚Σ!θNΞDΛΣΥ«Ο Y:DΘ`RMΛ• € ΈχFβλΩΓP€`~Ύ“cΗα°ΦxC€ύοΩIΜ ͺ‹·5qI‘ΟOc~p}tπ-%X\νˆˆυ€Ψ“V‚υνή4•…"ς‘`cζv & ϊΑ³Α„Λν’θGDa #DP@0Rυ½>X9$jWK«›Ι‘–οΥΕΨAέψŸ½xΖ.Β•kEpΣqSN[ ΒiiΟH@μQOΈΎ@,kIDό—έR · qςμ ž“p£hΧ,δ]Γ L‹= Ίg`ΐ’©OΜ ͺΓjΛ]ο<Ω ή{Ζ«ΈΚΏκ άB·c€δdΰΫΝ D=?'€X$i γb μΔ3z;V0]Λ©°<€‚δ^Β ˜Ε(@¬‹“Β|ή:ς¦Ύ…ΦΎ%kbTΜY~¬Ν.?Ωͺ„›ψ2ώ Œ`¬Π ‡_9Μ‰ϋƒπ_Μ?;…=Ιw±"€?Ž3/ VΤώ€D—,ƒγ)Ά—Eλu eΥω0=°Άy`nΗOA₯/¦ΦΖ¦Δۈct’aΗGο§'’τΑό z’(‹FΔJNqζ£Ικ\@„α5₯]CΑHγΓ0²΄z―.nήΎΞŸMΑŠ9Ÿ£a*Ό8‡yΛ·βԞ%‘ŒV±Q(š±A$'4!β‘m˜‚ˆe‘€ˆ²[ͺdOΚ]̌½ˆΊš0œL}’%IeŒh±Ձψ" RY#ΖƜCaeώΕ$wU΄ΓσUΕ»>…ΝΙ¨deΘ―τ±a;Ύ λΣ β?'€vΘωq Π•½…XFηψΕ™Ν ΥAψ.΄)‘#‘2#φ".ι£±'τ}³υά7ϊΫDΖεͿXρ&-ς)48|ίρρwΡΰV&άΫEΑŸϊXτχ―Θ˜ξΜ‹HΦ$γ Ά”EkuΥΥ!˜XΣ<π”ΨσΈ’{Ž½a/σ$γ¬-"π_ΊΰbŸΐθD/‹FΔJNqζ£Ικ\@„α5₯]CΑΘη“—αξΓH|Ϊ΅%6οώ"’ρσξΕπR«`21θ;r>”JVΝιΚΨh%…²‘DrB“ŒDqυ,Τ[)]Ί~“³œ]qλ"T6@ωΠψΒΠ¨5Œ-Ίσ}TϋΧCuδ;‹ύSΎόπρs‚g5Λρ""ώ‹M©]Ιβmο‚8«Δδ˜σYσΪ§huδ•oδήΗ’yΔΎLI BƒR6αΫοαmMΨkΙΈ#ιoL½ˆΣ…ΪρmΉZ­"ςε/fΥuΚΓ†Μ ΔΔΚ°i‹±qβ?'€XΟO ‡œςŒΜn’άœΥΎ /ΆΉ²W’Žβj‘.<ΰψEφJq叒6ηMβvˆ|—tK‚ ύΣ£Ψϊ>Κ«³3W°ϋ@¬Hiαt[Θβψ+ؐx;B›‘Œ*ΈΟžDΉΏjχΪdKρη…όP¨­yGΨeΡ€XΙ  ‚=wμˆˆ=κYξKDxMiDΧP@0ςχ½ΗψxΨ\Δ%$ςžΟΣνZ4ΰ{μ¬U8tβV΁z5+Ί†26ZIΔF‘lhFΔ‘œΠ$€xmςΏ―>yΑζρΛ€ΘŸάƒfΞθίο SpΘ£B½a. ­z«Ο/KM΄―\¦ΨΒ₯ ώv>δ±QΠ~ώ%ίVΝ”O ο9¦iεh<ν"b9β@ı鄃uοFgΏ—―t0ρ/ΉΆ'ώ₯! QOŠχ"φ£& η}“ίαΑ½¨znΤβD‘ΆPΛδˆ4₯ Ι“½X\£žŸΕͺόQίΒnG=»€XW–ˆψΟ  Φσ“ˆtΘ¦Asφ΅`•RϊγTα΄oοW|ΈƒςUΒς¦}‰οPς,}q wυ <πζžϋγkšα8w~Δ¬Έ‹xhLD₯F嫆VΎΕΑ•Εκu₯•ώ˜\‹+ΗψnΔ^tΞSΪi₯°€XY›Ή ι£X†/§™WζKj>1&cSχΜqώμyΪN 5!οπ;G3^b—E#b%'€8κ#lŽΖ%’#ΉljLΔ&™¨‘* (ΙJŸ;wcΩϊο±dϊ·’‘ˆpα$"œ–φŒ”ω υζ…ِίBqσ<΄cΏ6OΝ•ΟςZ9)_TκWLR―ψ@ίy`ΪΟSα=­/τ|S¦P―žΞXίw’=Έt_ –ΓGDό›ξ@Έμ:‘1Ώβ©)>2%ςiNά%€ΌοSμ΅|nJE΅GΫ±!4ρ)Βί.ρς¬§_9ΜvRι«tΓ€XΔ9AΔz~‘1Uͺλq› €XY›Ή w ρˆ4&£‘w!~`XΌυh>Ι[ύ+ρ_ ψθιπ–)°,€ŠΜ—ΨeΡ€XΙ  ’ψ[šˆπa "Ό¦4’k( (1MΈσο#D<}†αΎη’v1 ƒ§/αδΩ+Έtμε7Δ]C’¬­$"\ €§₯=#ε€Θο\ƒfρHθ>SυΕ<…zύl(ώ½‰Τ9ΫΑϊΏ¬m,p^‹GA;ν[°~ωΜ¦*[ A΅c΄WΏrίŸ\±/ΛQ#"ώ‹Mw ιΖ•0Ι#S!ŽΡ‘κ£ν8Φ ΥΌ‚-&`•GΫ1:_Utχ+Ηߟw λ_ό‰ͺ^!Ψڌ―ˆu₯ €83-ǁˆυό$BΔYΏ',ΝCĊϊδRεΗZ/‹eJΑβΈ+œ― )}q4ε!??…΅!οβMuΎŠΏŠύ)χωσ?δ^ؘx_Ε_ΑΞΠΠΘζΙՐ#Xα-‰²h@¬δ1UζΉ € ΒkJ#Ί†‚¨g±θ?f!ΈΦwκUΓ²YC]C­$b£P64#bƒHNh’Β™₯:± Κƒ› cŒ<Έ04hυΎuHY°π}Yη–ίύαλύ‡ƒ_σΖkέ(.ό}Ÿ 0ΦτΜWι’±œμ@Δ±ιΔΘ2|“·½Cτ_wξ%4ωβ±!ρ/Μ ¬Υί֌6¦ ϊγΨڌ?ύͺ>"`OΨϋψ4ϊ'τχ―„^'<ͺA„Hfθtλ9q~άteo!–I{±)ζE„ˆ˜ωG$k²9TƒQY”EϋK‡&ϋpΈ`kTQρƒqΠƒ; ,Ρ£ŠW0fΦ6lί.κ0Ξk£_›”;ψώxΑ6’(‹FΔJNσQež›ˆπa "Ό¦4’k( ?g ŽrέΫΏ‡βEB1yώz νΫ, lίχ:΄l„Ύέ[A.卐‘!"œš@„Σž‘r @ψ9Y²δ`}σByι¨6-@κβζCΞeΪTxκνπ/Α”xσU3^_~ξ@ucx'Zφ΄Η —οK„HFXΏQΕ*UΒ>k”±8šό_Ζ_Α…ΪπΘΌΰ-WbsβmΎ4ΕG~oπ?Ψ2*L¨…‡Ζ$τ{φ3zϋ½‘ωͺ Ξ€ΕΫOφ ₯o1 φ―ŒΦ€I1ηω]"G Ά†‰eΡ"βΪψηۜϊŸEŸΔ…>@e§<€X—™v€ˆœ  Φσ“§ό’°2 ¬S¦²˜αen @2+pύ¦;w‹Y‚³‹ˆπ ΒkJ#Ί†‚&†£w—ζθΩ±)οyΥπOρύΪιόaθIΙ©θ>hΖμ†Ί5+Έ†26ZIΔF‘lh–@΄AXί'ύ‹xV’ΚΌP αήi5Χ3_\mΥyq—±#ιop₯LΚ«1=¨Ξ+%LN¦>ΑŒΈ‹ΈkH@¨Βύύ+ς5Ϋ₯rQ‰3ΏJÚάYb<δ^‚ι­&f'Τkf@¦MnΠ,σΟW~…ΧΖHως{3IΏ©<±Κ_Cί{4ΌŽ€vΤWόιžz±yΪ"ώ#R ΅οδ&Ο|-n„ΆyJ _τΟΠ‚ΑΖόiΟ© όωWuΟωCKΉ²VΓσUόΏBVWτΟ13φ"ιbx€Βνϊ˜P”>˜Η’βhΑΦζ"ύŸύ‚“Φi₯°€Xν@DόηλωI„ˆ˜Ÿm €XQίΒ bΖΙ™s±’6νqfZ‹ˆπa "Ό¦4’k( ©ϊ^|3wκΤ(Ο{^­i_μX9eJζ½ψ―Ψ±$6/οΚΨh%…²‘Y:™„{8œς ƒίFq₯Ά$ήΑ7/nΰη‚νP\εχΪHKaΝ‹[Xά%”y±.ρv%έΕ™ΒνωΪ«κcΡ6ς0&ΦΒ;ή…q^χ”©Ε•1)ύφξΊŠjίγψ/•„ή ‚ EAA― *ME¬‚½ΣDA° ―Š‚ŠT‘jCŠpAP¬ ΟŠ½Π«t„΄·frΙ•CŽ{Nf9 ί¬υΦσ23ϋΜόfοNΞημ= e<œYπ»D€Τέ―ΨΫά OxσΕμKΥώλzΈ;»L)6V 3^RVέ†Κ<ω ΕμΨͺδ;)½ν-Κ8λ"Ε}Ή'=£}ݟTVνSsL˜9V±Ώ|£΄^>(ԘMk•4δ.₯u}\YΗΧSΒ[£χγ—ΪΧg„χΏυrƒΏΡσ Hήχ)ψ6£@bb >›ΓYAπi ?πύ‰ωρΑη΄ͺόžEw8λwθk9Ζ>+©Κa9$LΜΘaι¦iuώn5ϋ¬—βVόlΪ5ψνHπ{xΔCHξ€D»™¬HΣ«z¨ΗνWλςVrΓ9ϋ².zϊαΞΉ3>ΎωαwέΪσI}ρA|i¬ζΌ¨σ%¨ΦŒ>ŠέώWŸΟι—ύ»ΞKΕ7γ〕“4€bc]YβЍVOW·²υsΧ^Ο–ΤdΝ έU¦ž»œΙ½}€b1qz’B“CΡωpΛu35ΎΚEξ·{Ÿ[—κγ}4§κ₯Ήίμ :œhS«¬RρGnΞσ²χ>1ΕEδ^W)£ε΅JoήΞέΟyˆΉ±­WV…₯_v“2O?ο 6' U̞έJ»kΰώ=;[IΟά§¬*5΄Ώγ½9Ύ?MɏަŒΖ-ŽΨ₯°ΌGRπl AFπΦ>>'€ΰλΎ Ρ  ‘N{ΎPš2½έχrπcJJ Εόwb€/%$LΊHέΞsΫHψ¨Ί–―Κ58dΟαF°#K`εVVΉJΪίνqνK©a}Εη ½γwώ€u™©ξrΊύΛ7 »J·2Ο]Ώ r|aMΐ€τπ’Ύύρw=σ&{Φ)κΤu°*”+­gvuŸϋ1rΒ;zύΝZςφσ…5«<Ο› φngΈg€8k7X=USͺ΄Pγ€”ƒ^pKζ^Υ_=Uσ«΅Qέ>ˆΞΩΑYŠ$)&NΟTό—―™‘N₯jλΓ½λυeΪ&_Rχ—=έ]ΫέωyqΗwš²λ7-¨ήVΏ§oΧ₯λgkfΥ‹jΟήUζέR΄HύS2ΛaAηZ˜Ϋ@Ώ'ΐ3@ζŒs/UZ‡ξbHΑUΌŠ₯uήUΙ©Γ ξ$ώφΚΪyzΑχ‰:u€λΣ†)α£θΕG¨ύž₯Q M’R4=₯εayο €ό€ SVνS’βχ˜œoτ¦–ΐ 43@Sόη—‰Ζ ΫhuΕη9…ƒΆ.Σ³•ΞΣiΕ*jΑžΥzdληšS΅NJ,gύ> Φ#₯ΑB’€5ωνΟ΅Ί‘ϋΥ―s¬»–σΰσAΟLPΕςeTͺdqύΉj½Ϊ΄8[Oτ»£Dγν4o9yΩ+/IΟΞ­›9 /iBε‹iζ—τνjΆφm-«qͺΕ•ΘέήΛgZΉΫ=¦ζŠρͺ_RΟU:Χύςζξ?τΰ–Οά΅Ϋλ$–wŠΫvέl5/^C‹φΥyΙΥΤ³μi^NΩΪ>ˆ΅(‹LC€xΞHΑ° €„ο«Η$”Φ€”ζͺwθžήzΈχ½ΆlΡΔI±ΪΊ­ΰϋ3@Βί7f€ ‘½ω{‘ jΟΈ8ύ0p”V–H κ"j·dl’ΞH¬Ρ1ωέ@B`Hή}βΐ ~‰λ­Xrν†yͺXV–?+χ…oΫτΎͺΕ—PΏr έKΪ”8F½ώϋΥφVηΝha΅Ά::>ςχ—H~«%Ηφ¬ˆΔζ-Ϋυηͺ :³ΑIΚΞΞΦ+“fιΝ9K”‘‘‘ΖgΤUο»Ϋ«Lι}H]ΨΓsΞ±wCΔ™ωqη¦EŠ‰ΥθJ¨xlό!/φsϊ6]°φ}Q£ͺΖΟέήoΛ§Z“ΉGγ+_¨+Ζ©wΩΊ§μžEΡfύl“TU}˝ξγ@J«u3u\BΩΓΊτՁ@μυ£’€ ήF3Rπv Hhš3@Βχ $$^s=«λv~βνΝGΐ{]Yς8χω–‡γ7<€ ήτΙͺš³’ȁΏ+–\ΉaNO¬€Λ7ΜmΣY–ύλ΄ΏτvΥ‹υeΪf]»qžζWm£ͺq%ΤlέΫΊ»L=έXκ$o'²’―Ψ8¨$`@Š@_qdaψ;€¬Hί₯Ž›ζ«YRu (¦‹ yύlΝά§SVOΡΌͺmT―X…ά]ξάΌH%cυtΕsToΥdέWφ4έ\ϊδάνΞYΙ1ρ^ρχίο]«;7-VRlΌUk«ςqIφ.ΜCKˆ‡Ž°]Όo8Ο9 σ`‹Άg€€^r½ώly…² >g²’*ΖS™ΨΰΏAΚ3@ΒχS–ΐ*ψΑ€„οŸΜaHhο@Π>€πτΠ>Α3@Vψ§yˆKώ½ύkΫυ³¦₯΄Τ e΅tίFuήΌXΞL°ͺ_α^·³DΦϋ·©A±Š.ˆLu—MΜί’Ώά8ͺπ'(€dffiύ¦-*_Ά΄Š'+όiεqˆ½Ϋz@6”.‘6fλ–R'λΞ2υŒ/pφš7tG™ΊξϝgI«F«§©WΩΊTmuΪψ•‹-¦η+ύοά-Χ½«VΕkθή²§iWΦ~wIŸr§kαή5n#+o|]›;D€\}e–’ι Ω‡q \›χΤo[ςχxHψ΅φΖ{ΥΉVΌ~ΝΨαwΨω>Ύbl’†Wό—κ°^oθΙ Hh,ΎOπ ό~Lβ»,ζ6ΐ πY Hhο@Π>€D€ΨX±dBε ΅/;S}Ά|’wRTœbtJbχo‰₯i›΄ Z[·+μΝΚPσυοhcζ^½_νrχAιωύ@ς›Ηφ|Hzz†>ύrΉΆνΨ­ΪΗ₯“OΘ™φρ²οΥη±—΄mΗ.%&&¨σmu{ΗK {^‡œ?b;2—+5+COVlrPγ%cT:6QŸμΫ y{VjΰΧH|yΗr=Ώσ;=_ρ\Պ/­v~§…{VλΓ£”sΜ’½ktσ¦χυDω&j–\]oξωCC·ύŸ>¨~₯ϋ‹γώ-ŸhuΖnMB›2φ¨ιΊ·5΄Β9ΉI·w…α[Š&ιrΝzU]£˜(‡μ„DeΧΐWΙi 13@ ώΓnf€„οŸΜaHhο@$'VηοV³Οz)nΪ'θ›+–ΈΧΞl’½Ωξg^}·|ͺ΅©šX%η9ΈΞηTΝΦ½£Œμ,«r‘œίωύ@ς›Ηφ|Ȏ©Ίαž!r€~ΰηͺKΞΣ}w]«ΦοWι’%tβq5τΝΏ»Ο=¬·š4¬[Ψ3;θό{·Σ½]먝sςlτΆ'»θ1v珲ύKύzt'w?g‚ΐΣΫΏΦ€]?kGVšN+VICΚ7vx~ΰgϊξίτοίhCΖΥN(λ―θ<δ£}λtΣ¦χ΅°j[ΥLΘy€Τ„]?»νΞ₯°’ @z^Ά\U'τQΜήT{77Ÿ-eΦ:Iiχ<%9kξa?Hή7œ%° ώƒΝh[ )ψ>Α3@Β‚β!θ’β·73@ ΎN0$όP@ΠήΑ ΙI c‹υͺ?Ώ―βΦG€l¨TΙκŠ%ΞσhΧg€κόδκξ5;+šœΉzšn)]G]ʜβώΫ-›ͺ\l’N-VQ£v|οΞ ΙλΉ^ήt ^RbŸ’˜€/ϊ⍛φžZ6=S O=QZ§©3ι’ ›θ—?Vk *©X’vμJΥ•·<δΞ1δž"•#bοv†>έ^ΛΡί’χ=@ ώ ηΞτΏό+•uŸ³‚‚οHψί)ΜΏp™Rπu‚ α‡3@˜Ϊ;ΙI€ αw0$zδŽψ?­XςޞUκφΧ‡z΅:9±œžέώfξYα>ΓYΚύ­Τ?4`λ2}Pύr•‰-¦Άλg«~b=V‘qΎήw ωŠƒŠ@Ύδ’λϋͺzJE½<΄Wnγ¦Ύ§‘#§¨_χNκxeΞt-ηgψKΣτή’₯š?eXˆν—€Ψ»ˆ½,ύ΄Δ ?ιΩ;– yg €ό›š3@Βχ f€0$΄w ’“ΐY²tΥoχ+φ@LE Ρ {» ΡQi‹σΌ]~V,qΠcόŸ΄#kΏ;Λγ±ςg©Nbyύ•ΉWMΧΎνώοΛK©³ΑΊ IDATλΎξϋ·κβυ³τz•ωZ 16ΆΥ|Hƒ·«Ϋ-Wθ–φηζσϋΚuΊμΖ~zaH5=ϋ΄άŸόφB=9βu}½ΰΥ"•%bοv φ²τΣβ'={Η Θίΰ!θαΗK`<Š HhΜ ί'@L1°K`…φ$:d·Η•U5ηΉΗ…υ)¬wŽσφ›€/©Ϋτ& μu³Ύτόάσpzώ―ΆέyήnjYθ‘ac΅|ρ8ΏηUΗ φnb/K?- ~³w,€ ήΖ€„φ–ΐ*ψ>€ ‘ π π}‚%°˜Ϊ;ΙI Ÿ€xϋ½H Ϊ@|ήΔg€;±—₯Ÿ–?ιΩ;@oγ )ψ»™Ύ―²K`…φf€0$΄O ’“Ο ~‚ Μρφ—‘y/f€˜3b’™βσΎ >όΫα™ΗΦΣ»{kw|¬½F}΄T1.I'&”υΡ‚χCyzήYρτ‚`ΣΉ3<=οώY<6^“«΄TΓb•Όφ|ξΙXαƒ@ ΎN HhΜ ί'ΙI€g€„―ΜaHhο@|ώyΘaˆ­$i§°%ΰ@*U(«₯Jδ^wVV–ώ\΅^ΥR**9©XξΏοά•ͺΝ[Ά³–€Ο—ΖjΞ{Ρρ!ηKώP­}»ύ―ο»™υΟVΏkZkτξ(xΓ#ι…Κη«MρšŠQπ. λ²5hλ2-Ω»N™ΚVƒb5 ό™ͺ¨φggiΨφ―τζξί΅={ΏŽ/­ήε¨yr χ…veνΧΫώOσφ¬Ά¬4^¬’―ΠDΗ'”)πqpΰ˜’χ­ΰ!θΑΧ#Σ ΰ!θαͺ—¬‰Uš«nbySŒΎ·§¦Jγ_‹Σ†ί'ρ>€@L‰‚Ο*TΥĞ}Τ}η2Σν:,ΫΔVG@l%I;…-_rΦ%wG|½ŸΟρ1Ρ|3@μέ@Έ¦΅^Ωύ“½F}΄4²rSΔG~6=0€εΦ*›¨ε)1&N·-Σι;τQυ«ς|™[—jΞ•^ρ_ͺ_J“vύ’Q;ΏΧ’jW¨VB)uΩό~ΨΏMOW€<€dV¨ͺ?ο’Τ’₯MCψ°lOŽ‰ΥQρ%#~-$βΘ8 ˆ$ΰ @ŠHΎ.ρίA φ²τΣR΄=d}·GΥwχWz°|C_Κ½΄οώR«υ³τEvͺWόΛ}pΛg:/ΉΊZΟ™ραόœ΄r’†Tl¬Ά%ŽΥ +'jD₯σuqρšξΆŒμ,±fΊž¨ΠD'$”QΛu35ΎΚEϊWRUw»*οΫ 9U/UBΜα™½€δέ‹™Rπl2$|…eΘp?Ώ~¬ €|@Βwg ν<„g€„φ  νHH­5ϊŽ»4*υkοYύ4Τ₯l}u(y|Δλ• ~RηΨœΐaΏΆξΠ’ΟΏΥ­Ο-Μyrξˆ½Ϋ €ΨΛOKΡ iχ<%9ŸΈώνη?{W«σζτνQν•oΌ\g™««§jJ•:½Xe@ΖTΎ@ώ HZ¬›© “RŸr§λΕίiΚί΄ z[ύžΎ]—Ÿ­™U/VέΔ ΖΧ²΅€ό=f€„YΜ)ψ»Y+|δ!θ<=΄w ’“Ο »@œ:ΆX―ϊσϋ*n}tΘs·έͺA»Ύ±υ'Ώ―vUh¬[J€ψJ‘ƒ€μμl}σ }ψιΧϊΰ³o΄όηnˏ+Rω φn'b/K?-E;€lΜΨ£Λ6ΜΦM₯OΦέ₯λ/5=;K·n^$)[*_δξίnΓ&V£+] β-²*c—Ίo^’KΫ¬ΔΨ8]ZΌ–Φe€κΔΔ²z΄όYn³Ώ€oW«u3u\BΩΓΊτՁk@Δ4ΒsΆ a73@ΒχUf€0$΄w €˜ήα €˜κbJˆν$`@ώX΅^~ϊ>ψμkύί·Ώ*#3S5ͺUΦyλλΌΖ§κΜΣNRbbB‘Ν±wk{Yϊi)ZdαžΥΊη―τHωFΊ¦δρΖK\‘ΎK7ΝW³€κPώLABR³«w­σΧΎ₯kK―ΞeNqw[Όw­ξά΄XI±ρZT­­ΚΗ%_Σζ€xQΪS˜Rπ}‚ αλ€ ˆι€ ˆ©N ¦„ΨN€e‡ήZ³~³βγtFύu^“Sݙǝσΰΰ#α±w—{Yϊi)δνΠ›ι•ΚΝδ<(τ³)cΪl˜­[J¬;ΛΊLΦΗϋΦ«ll1ΥMΜ™‘φ[ϊ@ή­z‰N/VI»²φλ‚΅οΈ³Lξ]γξ3²ω¦—΅Ί@oC )ψ»™Ύ―2„ ‘½@Δτ@ΔT'SBl'‹ςπΠ1ξ,³ΦUρδΓϋνθhΉ‘ˆ½;€ΨΛOKΡ Ϋ» Q³­σΤΎδρjWꄃ.­|l’’bβτΙΎ š·g₯ώwωͺ»6/VjV†ž¬Ψδ ύKΖ$¨tl’ΩϊΉ–μ]―—*7U‚βΤsΛGJP¬¦¦΄tχΏΛ'Z±[“«΄ƒ)MΧ½­‘ΞΡ%%jϊ‰6’cρ6d$΄§0€ΰϋ3@ΒΧ/@Σ;@S@L ±,H^a:Ούψβ›Ÿ΅qσV΅lz¦*W,«½ϋφ+9)±Hf€Ψ»­ˆ½,ύ΄m²πvλy^«•/P«βGkμΞ5dϋ—ϊυθNξ̏^1>Ούo+}²‹${²2\δpfwdeg»;΄ΒY*[Lν[§›6½―…UΫͺfB)· »~ΦΣΫΏ>¬Ka β­’ a73@ΒχUf€0$΄w €˜ήα €˜κbJˆν$ €<;ϊ 2Gι™ξ«ΌώβC:΅Ξq6jͺV―έ€atv—Λ*J?ˆ½» €ΨΛOKΡ iχ<%%%ϋΉ€By,€ ή†.€„φf€|Ÿ`Hψϊ€ bz‡€ bͺˆ)!Ά“@@2υχ5θ™ Ίθά3ΤτμΣτΰ“―ζΘ¬|ͺώOŒVη›.ם׷)Rχ±w;{Yϊi ρ“ž½cρ6ž‚°› αϋ*3@˜Ϊ;@Lοp@Lu1%Δv@Ίνar±Πλ&₯νOΧι-nΟη%ŸyyΊ}ς΅fŽ\€ξbοv φ²τΣβ'={Η βm< HhOaHΑχ f€„―_€ ¦w8€ ¦:€˜b;  ZάƒοΡ9κε -ύNχ<τΌΎœχr‘Ίˆ½Ϋ €ΨΛOKˆŸτμ € ˆ·ρ€ό‡έΜ ίW™Β Πή€ bz‡€ bͺˆ)!Ά“@@rFΛ;τοA]uξYυσY >Υΐ§ΗiΩά—ŠΤ=@μέNΔ^–~ZŠ>*%GΛ3@²₯l?ιz?@oγ@B{ 3@ ΎO0$|ύ@1½Γ@1Υ Δ”ΫI ιΠωQ₯T*§gv=@223us'”-½6’‘Ίˆ½Ϋ €ΨΛOKΡ ι ΞΥ΄λ:jsvšŸK²vμYΕͺθΤΔ ΦΪϋ§†ρ6Ԑ‚°› αϋ*3@˜Ϊ;@Lοp@Lu1%Δv@ή[΄Tχ |Q O=Qώλt=ωΒduΉιr%$Δλνχ>ŠΥά"ΝΟkX€ξbοv φ²τΣRTH“–ΊΉΥš½g₯ŸK²vμ΄”V:')ΕZ{HδQ¦Άξ +₯hYΪ¦ΘΆ|DρΨxMR ‹U²άς‘ΝeeIc'Δiεͺ‚°;)Yκέt‰ΚŒy8πλφςHΑχ  M€ αϋ€ ˆιύ € ˆ©N ¦„ΨNˆΣμΔσυμθΪ»oA―R<9Iχήq»βΒ"—?bο– φ²τΣ>=δ0­υ)ψ»π΄J\²&ViΊ‰εύ”aOΗ¦¦Jγ_‹Σ†ί'ρ4lݝ@L@ΔT'SBl'Δizwκ^}ρΝΟΪ°y«œ?Ι«₯TυOƒ Eρ±wW{Yϊi @BθωW*7κ>ΉλπRπv Hh€xε€ bͺ€ ¦:€˜b;  GZΐˆ½;€ΨΛOK€xA,ΕX‘=… Γ½ ž€χβ!θ₯,Ύ“ €˜~  €˜κbJˆν$ €|ώՏZφΥOΪΉ;UYYySψΑΧ©{€Ψ»ˆ½,ύ΄€ ˆ·€ HNΜ _3x:Aν€ ¦wš€ ¦:€˜b;  㦽§‘/N1ζ»|ρ8γ>Aμ°goš†Ώ4Mσ/Uκž}ͺS»–ϊt½N§œtŒ§—Ϋ—Ά_—έΨOιZ4γίΉΗ žβσ΄β)¦ΐw@ΔΫ0@1U @r¨SGΊ>m˜>@Δτ»@ΔT'SBl'€δ’kοSΪ5uΧυ—©rΕrŠ‹‹Ν3λreJΘ=θ9ΰE}υύ/κΫυ:UͺPN“ήό>^φ½ή;XU*•3žΣΣ£¦iότχT‘\iĘVώv@ς—›ν£ρ6ͺ@LΥ@ΔT'š$₯hzJKΕΈOΠ φgυꍛ§τŒ`_ΗKλg5ΚUΏέ―ΨΎτ²{°ϋΔΕkξ guέΞO‚}­ HhWιZΊΎ(Χ@Š ΎN|·1δ5;»{~ι™jΥ‘·.mήDχήqΝ?žσ/¬Qϋ»Ίϋ.ωό[$ ; €l„Ν α›–Jη$₯D˜hώvŸρVœΎύ.:ήxςτΌο!€„φ žΒ3@Bϋ€„φ –ΐbHhŸ@Π>€ HNHψϐό}ΞΒQGnΦδς›ΤΰΎ·©ξ‰΅’.Νi3iΘs―iΩά—”Ÿ{~=5F?ύΆJΣ_φœg™tμϊ˜Ξ¨_[•+”ΥΨ©s€ξ0P°6 € ‘ ΘίHJ–z7]’2c$'žώχ€ 9 °Vψ:€ HN­Ξί­fŸυRά @Lγ ¦„ΨN'` @,ωR¦ΟΣ“ή₯ͺ•ΛGUΞΞ³Iœσ›7yθAηυςkοκ•I³΅lξ¨°η;ων…3efŽ’ιο.@Ό³H€αFΠ4€ ή 3@1U @S@1Υ –ΐb ¬Π>€˜F ΫI q2~ίΐυΡouTΥJ*SΊdžYOyψ?0yxθ-y…ή=θ sšτζwfΘwοUlμ‘KΝlή²]—ήπ€†>t—Ξk|ͺ <‘3@ΆνΪQŸΚΚΞΦ‡K³ίΛϋ)5faη.—ώ‘šΣϋ(vϋ_ZσΧ„ ύΉX/οώΡ_C–ŽU₯©.}¬b^g3+[š5/FK>²tβ>›Ή―νr₯Œο£˜½©>[ςxz“–ΊΉΥš½g₯Ζ,΄πF΅ΦjV²ΪZ h…*gζΩλΣbτM΄,uΕW*7ς>ΙΊ‰ΎξDjλΊ’QŠ–₯mςՎƒ™V­•Ξ)ό²hιΩυͺ΄rU@.‚@œ }š.Q©(™²φΖΊ¦F†~LίΑU³«³Φδj-Τ x₯`^ΰo­ξΨ™₯WΖΕhΓΖ‚οΞ nuίU‰)Ρ±ΦέͺCΙυZ•Ή+πϋ`z@¦Uo©γ‹•1νzΠφleGό<€ ›²5f‚΄u[Αχ‰ckeιΦrγUlNtΘ²~ΓΤ>ϋmΝJ‹θ>±³³Φ5Z©r|rΝΤζΚ5Y3!F©©ί'rf€<­„f~έ^^ΰƒA#tmκR₯)ΣΛξξγΘΜ£/V|Lπώϊg–ΖŒ‹ε ‘w4.^ο=ϊœ:μψ8Π{ν΅ρ«J§QUΟW\ŸQxmΓλ~?ώ’­1γƒο{^Ξ'Ϊf€ΌΦ³―Ίν\κεΤί§{™ϊzΈrΓΐ?›PΆτΕ7Τ{Γ 5κ–ΐΊύ6 ΪωuΰχΫΛ QT’—ζΩ‡Š\Φf€τς²ή‰»ΔTΉ2%—gX ¦>}ΨCΜ/€άϋΘΕΔΔhψ€.ξ9η {"{‚œσΑζ‚²5kntόBιζ՜€δΥ)_Ni¦Ž‹ψJ€<3+[oΝ’>ŒιΥvΉͺ yήΖ7«·VΛrGύo[v€wΫΫώ™™Ω79Ι#hΥ[ιό2U½έX{νOΟΦσ/eG€4[’R―ώ/4δαΊ{θκ(iG΅TΓRΑΘΆνYωͺ’@ΊΧ{WΕ'G €ά3PJD€8vŸX<2ΙΚ’b#ό½€™γ‡¨R…²adέ–½]’σyιηKc5'Jf€tΎδ՚=ςΐ5­υΚξŸ"Κ4¨GVnͺ6ΕkFό ΜόœΟό…ρϊ(:Ύp$–ΐ yz@βΑ ‰6™\₯₯ ώΓnηΓΠ±β’@xHޝ–‡ G€,ο6@Χ•Ϊ53@&₯4WΝΈRTΊόνΊekŒ&NŠš ·”―€(™²΄ί0uˆ"y=₯…*ΔΛߍŽΰ¨υb4aRlΤΘυiΓ”πA½…,ΕX‘}‚‡ σ Π>ΑX,Ϊ'X+‚7DμJ’¬Θι-nΧ»Wυ”ŠQμ³?Τΐαγά‡ KLΘ=?gΦʊΥ”Χ²\ύŸ­·ίϋθ Mu4<;;Ϋύ·>]S§«š ±w»yˆ½,ύ΄€ ‘ πτΌϋΟ‰Ž knθ‘vGGΟX«4WέΔΰŸ…–š*-.jf€t­σJN@B«Ο‰Ž%°ΰΏEnzοΙCΠΓ'€ HNΡ6dbΟ>κΎs™©Ό–ν€–‘Ζ‹α¬ΘM=žΠν/Υ9κE]\λ7nQσφ½ττ#Υ²i#χόφξΫ―‹ν©ŽW6WηΫrΞ7oΣΞέ?ΑYβΛA‘W‡ί―JεΛͺl™’ˆΕ» €X ΣGS€x@ΪS˜€ 9 8ΟaHήΏKœg€0ΔΫοΩ χZΑ f€„φ @rˆΊ‡ ίv«νϊΖΫ‹οΕXLσE.kκΊϋ΅cηnνά½Η ΙY&*―Ÿοή[ !¦νOΧπ—¦iφ‚Ο”ΊwŸN«{œϊwΏ^ΗS=χ|Ϊί=H%’“άyύ Αή:$Ψ|½Ά€ ˆ·Ρ€ bͺ€δ$ΐXα«€ 9 °Vψ:Α f€„φΔτ.œν$ppΦδ>çóϾ§Hέf€Ψ»ˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫIΰŸ° GjЈ½;€ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$ €t묚žέ@W]ržœφςΓXηKc5η½X/qΎOηKώP­}»ύ―ΐ_Λτˆ)‘Γ³@oc @ΔT-@Lu@ΔT'1Ά“@€rΚ7λϊ«Zθώ.δό·—Ÿ‚zΊ—sΛΟ>ΜΙOjy€ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°Β•{½±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a;  g_ΦEχέy­»Φ‘ϊ€Ψ»σˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI @©Ϋτ&υλήI―ΌθˆΝ±wλ{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$0€4?―‘N?εΟ9ίpMKΟϋ†{w ±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a;  ‘Ό|ρΈH‰κύ{·±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a;  W^|žώuζ)žsnΩ΄‘η} Έ½»€ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°~έ;©γ•±9 φn=b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°Λnκ―;―o£K.l|Δζ €Ψ»υˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI @!\ ±Χ {Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$FK}* IDAT€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Έ μΩ›¦α/MΣΌΕK•ΊgŸκΤ₯>]―Σ)'6‘ΜΜ,Mœ1_Σή]€ ›Ά*₯ry]uΙωΊ©]+ΕΕΕΊΗ φ†b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@άzxQ_}‹ϊv½N•*”Σ€7£—}―wΖV•JεςLιΩΡohμΤΉΊηΦ«tΚΙΗκΛoΡσcήΤ}wΆΣΝν[ –Gb9Π|6€ ˆ·Α€ bͺ€ ¦:€ bͺ€˜F ΫIΡͺ΅ΥΊcršέΐM$=#S­:τΦ₯Ν›θή;9$₯ŒΜLέ¦‹»β"υΈύκάν=Ό Uk7iΖ+Λ£ ±h>›@ΔΫΰ@1U @S@1Υ @L£„ν$€hΪΜEςάkZ6χ%%$Δη&ςΠScτΣo«4ύ净”••­5λ7©Lι’*SͺDξφ§^˜¬y‹—iαταˆεΡ€X4ŸΝ βmπ €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@4τΕ)Z°δKΝ›<τ 4^~ν]½2iΆ–Ν婟8³BΎνplu }θnΔSjήw@Όgδž€xa€ ¦j€ bͺ€ ¦:€ ˆi”°=[JoR·΄j¨Y{Vδ³»‡½}ΤΕjSιh»ζΡZzF–^ž©oΎ Άοy½―όZe_μ)eGVοΌΆΙ~©­;θŠF)Z–Ά)’ΓΩא·j΄ΦEͺώίέ·?KOΘΤΚUί'’’₯>Ν–¨Τ«Ρ λnμ‘«kdθΗτmίΣ T‰KΦ5[©IΩ*¦]}okk¦ž{)K6|Ÿ¨XQκ^ο]Ÿœ³ΆV–n-?^ΕfG€|Ρ˜ΝϊA[³<^Ap»ΥK¬ 9Η_’κIΕƒ{‘ΆόΣοziLΆRS ΎOΤ©#έ°iΕ/™ψu{y‘v»—*M™^vt@Φn£bρqΎŽΣψ·?¦λ₯Wc”žψK_ΰ¬FYΊκ·ϋϋCtΘΌGŸSϋΟϋpμpU©γτZ­ •ψΛ}ρ]Ί^Sπ5ΒΉΠVηοV³Οz)nΕρ_·ρ*TΥλχυU—K»Žξ)[_OέXρqΑφ ηΟ½>ΛΠλΣΗU™_£SΛ :e^Ε­yώŽΫ4pΗΧζ? {<^Ή‰ξ«Z_qΎΜύχΣρςεοΓpϊΌ φb²³£ΰ­€/{Ψ¨©šΏx™ζOvΠ+š0Sc¦ΜΡ9α—ΐr€cΐΣc5χύ₯zξΡnj°ξAm¬ίΊ7’³w>>όμσXΝy/Ψ_\^Oͺσ%¨ΦŒθi­Wvδυτέodε¦Ί¬DMΕ(Ψ7„ΞœΏ0^EΗϋm1$|·šžJη$§ΪοάΖ³₯ιoΕιΫ(ώ—₯r£ξ@BξΌ Sͺ΄T£€Κχ‰Μ,iΜψΨ¨ήM—¨Μ˜θ57τP»££@^«\υŠUΌOμN•ΖMŒιZη]•œ²ΌΫ]WjCΤΘλ)ΝU+Ύtd}"&[ʎμύΗ_[₯ ―ΕF €άRnΌ’ζD€,λ7Lν³£@¦Tm‘ ±I‘υ‰|μ½v½4aRlΤΘυiΓ”πΡμ|\‰ύC>8BΧQ΅•bώ›ΓIqε*ηwGΪ₯ββ5wΠ³Ίnη'φ;[>ZΌ²δqzΎβΉ ψ{xξ™ύώgŒΖMίΌΔm2±guίΉΜΛ©Ύ;€|Ε†NρνχqšώFdοA‚  c‹υͺ?ΏoΤΘs·έͺA»Ύ κr#jχΡ uK©“"UΛ'Gτ:μLE%#@ޘύ‘η>½XbBξ½λ;δe­X½ASF†ηΡg&hξ’Ο5zXoΥ©]λϋΎnKδςωR$―ΔXΡQVπχaš I‡@$Ν@ςΌΡ6dr•–jX¬Rΰƒ7+K;!Ι#ιh‰Uš«nbωΐϋDjͺ4ώ΅8$€£ @&₯4WΝΈRχ‰-[c4q’WΠKϋ S‡(ΧS)xŸXΏ! “ςβ#Τ>ŠdzJΛΐΏtεD±zu ’WŸˆByΐΐ«„τϋŠ@MΊBUE€”k ˆ?νΞGϊn9’ηηU΅NT4Θ α=V‰02v/" ²~γ5oίKO?Y-›6roέή}ϋuΡ΅=ΥρΚζκ|cΫΦ€§ΗiβσύTοΔcςά±7{Yϊi @B`Hή}‚g€0$΄g8K` ~~Ω99™ύσ½r–ΐŠ¦ HΑχ g ¬hš€Ψ©·~Z‰Ά%°’mβ§wY8±’&˜>CΔ’…#+#@œ[ϊΰ“―jΙηίͺ_χŽͺXΎ¬^<[ίτ§ή7XεΚδ|+Ζ(9)QύΊwΎ΄ύΊΈSVχx΅o{α!½’A½γ•/Δή€@μeι§%@Ό @rpžΒXyΧ ‚K` Hθ@Ό½Χ r/$|ΊΞXH½ΟCۈ‡‚ί@‚οeΌΒ‘’ΐ iϋΣε<Θ|φ‚Ο”ΊwŸN«{œϊwΏ^ΗσΏ‡ΧΆΏ{J$'ιΥαχλ§ίVιͺΫΒΈσΑ›Οͺbω2ˆΕ‘€X ΣGS€x@€ ¦j€ ‘}Δy:K`™FNπېΰ36½€„&ΐ3@Βχ χ ,e*+nΟd ¬@σ₯q:#@‚ ’ φ’@μeι§%@Ό @S΅@$'–ΐ _-š$₯ˆg€˜ͺiΐΫyHΐ{k@B`Hψ>ΑXήκ {‘ΐŸ}ρΰί@μeι§%@Ό @S΅@1Υ δKSDΑo@‚ΟΨΓ+ βa όwΔ{VμINˆΟ~€ψ q˜Ώ0^}l/K?- βm €˜ͺ€ ˆ©N Hhα ?›†MπΫyHπ{xf€„ ρЁ؅ώ–β³; >@θBιMZκζVghφž•φ:š–¦₯΄9I)>Zπ~茷βτνwΠRηŒϋ_ώ•ΚΊOΚΞφ~ν™ΪΊƒh”’ei›zοΝ €˜*€ ˆ©N ’“3@ΒW ž²ΚTJίΞ3@˜ @@|Ζ €ψ @<@ςξHΑ£XR²Τ»ι•€ ˆι]€ ˆ©N €˜κ€„φf€˜F ΫIΰΰŸ=ρ € He@B»Λšz¨έΡϊ1}[=)˜]«Δ%kb•ζͺ›X>˜ψ[«©©ψΧβ΄acΑχ‰Š₯uήUΙ©ΓΏn//°ΌΫ]WjƒVeξς²{ ϋ €˜Š € ˆ©N b%l'NρΩCŸ €DPF‚°› α;,€„φ@ΣΫ@S@Δ4JΨNH }±ofύ³υΐ5­υΚξŸμ5κ£₯‘•›ͺMρšŠQπ8ςτΌoΟ Ύοy"<$ο”xK`…φ @r8ΆV–n)7^Is1½Χ@1Υ @L£„ν$€Ϊ{ρ φ²τΣRΟΛ–«κ„>ŠΩ›κ§+Η HhGbHΑχ f€„/o€ ¦7@υ+θυ”ͺ[Μ΄«ονλ7ΔhΒ€X₯¦όοŽ:u€λΣ†)α£ΩΎ―ΛF‹ŽPϋ=K•¦LΝωj@ΔTD1Ά“h@μΕ €ΨΛOKHψτ¦₯΄9I)~βυ|,AΟ;*€ΰ?Δ@ΠxHψ>ΑXΜ νΪ'@L$ b%l'$Π>€Ψ‹±—₯Ÿ–$4–ΐΚ»O°K`…φ f€0$΄O ’“3@ΒΏΏ@1ύύ€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@$'f€„―€ ¦w€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ@1U @S@1Υ @L£„ν$€Ϊ{ρ φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$Π>€Ψ‹±—₯Ÿ–ρ6‚@LΥ@ΔT'@Lu@Σ(a; φΔ^Όˆ½,ύ΄€ ˆ·€ bͺ€ ¦:€ bͺ€˜F ΫI ΄ φβ@μeι§%@Ό @S΅@1Υ @S@Δ4JΨNH }±/b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@νˆ½x{Yϊi @o#@ΔT-@Lu@ΔT'1Ά“h@μΕ €ΨΛOK€xA€ ¦j€ bͺ€ ¦:€ ˆi”°@ϋb/^Δ^–~Z@ΔΫ*—ζπ IDAT@1U @S@1Υ @L£„ν$€ψκgΜΧ€7hΓζ­:ͺj%έΡιR]Φβœά6_ρt0b/K?- βm €˜ͺ€ ˆ©N €˜κ€ ¦QΒv@ςέ¦ΌσΎn’zήΥN§Υ=^Ÿ~±\#ΖΎ₯†τΠωMNuΫ@ςο! φ²τΣ€ ήF€ ˆ©Z €˜κ€ ˆ©N b%l'$ί}ΰΒkzͺeΣFΊΏK‡ά6ξψ’Φmά’Ι/>€δ;ΩΌ@,šΟζρ6x@LΥ@ΔT'@Lu@Σ(a; ωκZ―Kox@cŸι«3œ”ΫΖ»σ?Qί!/λσΩ#U²D23@ς•.šΐό…ρϊθc‹aϊh @o@ΔT-@Lu@ΔT'1Ά“’―>°ψ“―Υ₯ίΏ΅`ΪpU­\>·―ΎUΊΦτ—¨NνZHΎ@o'½IKέάκ Νή³Ϋο5-₯•ΞIJ ψUršŸρVœΎύ.ζ°Ό–ιEϊ_ώ•ΚΊOΚΞ6νψφΤΦtE£-KΫψk™^@ΔT'@Lu@ΔT'1Ά“’―>0λ?ŸͺΟΰ—τΩ¬Uͺdρά6~ύs.ΏωA½:ό~5>½NΔmgfeλύΣ΅τ‹ˆ δ€+R•Η(fχφ@ڏ€Ρ¬γNΡ‹η7Τ{ϋΦDrX`ϋv/__νͺ«Ψ˜`?„ΞΘΜΦμιϊώ‡ΐ.%’†―=σ•{¬bφFt\;gΤ=SCΞ¬­₯iƒh>β6©ΤH-+ρq‘ž‘₯ιοfθΟ‘ΜώΧ7ώA%ηΌ²ηΜ τπΙυSϊΆ`.6‚V“bβτp₯†jZ±ZGεoΧ½iYšτF†6FΑPH,&u¨χ…ŠΟ-cω¨υ-Τc)ʌ]–[ŽΌΉr±ΕτP•Fj\RδGxΔ_Ϋ25ύLm-ψ‘ ₯₯«ϊ@Ε–ΌαU³ϋ―Wά 'JμΠΖ¬=ΑΌ@­V/‘AU©n©r•Ώ]W­ΝΠΫs³΄«ΰ‡‚ͺ¦dλ’δYJψbaώ.ΖςQίvΌSOh΅vf§[n9ςζŽ‰/­§j4Vυδ‘α?‘Yσ²΅o_„°{͚RΛ΄Χ·|i­Gήδη7wΣiΏ(]Y‘lωˆΊ‰ευ|­))>ΞrΛ‡6χύOιš5?[™™Ώ”ρN>)[η―©˜•Ώχ |‡Έ8-Ή₯‹žLύ1π—ςςη$WΥΰg*1>ΦΛξΎφωΏεϋ5wΎ―&¬|f½T5ψω%Εn(ψ»U¦‚f]Σ^£φDA”Τ¦Δ1κ}T}ΕΗΫ'œο»}ςeΊ/)ψ/Ύ9«ωιuμQμ–‚#(+εh½~i+MΩσ§΅>六˜€Ϋͺž€ΈΨ`?―ςsŽKΡ”@Lvv|₯7šωοΉ Qx©œ           Ή0·τΓΟΎΡέ}ŸΡ‚©O«j• Ή{ύίwΏθϊnC4㕁:ω„šECpA$@$@$@$@$@$@$@$@$@$@$P@ΒάΕUk7©uΗϋ5zXo5iX7w―7η|¨G†u‚^<9©(τH€H€H€H€H€H€H€H€H€H€Š\Θ?άR@ΞitŠμq}ξ^wυ΄ύϋ5φ™ΎE3pA$@$@$@$@$@$@$@$@$@$@$PT@ώαNΜ±|ςUυΌ£N?ε½ρWύϊl}¦vRQι\           ΉΓ-ϊΞϋ3e6nήͺš5RΤν–+uΡΉgΉŽΐ‘          @QJ)Jw“k!          p@θ$@$@$@$@$@$@$@$@$@$@$@E.€ΘέR.ˆH€H€H€H€H€H€H€H€H€H€H‘      Š>ϋΏτΣo«tS»V…β|9Ιΰ OŸqa{…μμlνά½GeJ•(l§Ξω”ΐώύιΚΘΜRρδb½Ν’ DsH4ߝBtn;v¦jΤΔ™ϊκ»_T©BY΅»¬™Ξ=«~!ΊNΥvΏύΉV/ΏφV¬Ω ;Z7^ΣRΗSέφΛΠ^!J`Ιηίκ©&kΓζ­j°θΦIU+—/DWΐ©ΪN€>a;ΡΒίޘ)s4~ΪSr±D=ωΰΤˆΊ'ύ²τ‰‚ΎΡχϊoΜώP£:΅k©F΅JϊxΩχμ:ιψ££οd9£Γ’}β°Δ\¨^δώGGιλεΏiμΏϋj՚zς…Ι:ζθ Π…χ…κNΪ;Ω¦WυpαΓω|κ£₯ίiθΘ)κ|γεκtUs{/BK$@QΥ·§pœ\Ο/κ?.Σ£UνcrOϊ―­;ΤΊγύφpgίδΤΒq!œ₯•φ₯νW›ϋΙ™vξ7Ί_}ϊ~•(žδ.qγΌ—ψςΫ_άόB6mUγ3κdGH8p™}{IΏώΉFoŒδΦ‚»RΥ’}/=Ρ5;»Α–—λ$Πμκj~^CυλήΙ ΔY6οŽήΓ΄δνηY&.BGHΘr£ƒΊLgύέ«n{X)•Κ+..V½ξn―η7t_ύ]Υπ΄“Τλkρε)ͺUώ{|PηI»‡/‰3ζλ‰―»}’zΥŠξ› η[yŒœΡς|β^ΧψŸQlυΊMͺQ­ςα;i^)Π^šψœoρΎ=汃–A{ύ­…ςάk‡υτh<Ί ODΧύˆ†³q–ͺXΉzƒfΏφ€Š%&䞽ŒΠ>όRέo½Rwtj §Κ9¦θ‡)θBτ2Ξ7vχ3Η >θ}βΠ§hά΄χ\Ω³wŸN9ι=?ΈGξ. Ρ%rͺ&WŸΨ»oΏξΌ˜ώο»_uμΡU΅fύfχχΗ]7\aλμ^XψcΥz]{η@]Φβlχ‚<ϊ̝Vχx΅iqva½,ΞΫG³ώσ©xόe=ΡNwωDηΛXΞκŠU΅’–9”H °$€–;₯ηωοWfΈK-šρŒς†Ώ4UGU«¬·_­Ϋ{ u§ήάΎ΅23³4oρ2­έ°YΟ¨λώaβό€§gθ²›ϊ«ρι'λ‘ϋnŠ«δ΄"IΐY-))Q/>~―ΖN™«W'ΟQλ Ξ”σ ^gΘΤ—Q½Ρέ{4kΑ§ΪŸž‘η7Κ}ĊΥΤφ¦ώzΌίΊψΒ³"yiφZvθ­ΪΗΥΠσu?θ ϋ=ώŠΫή;Ψύ5mΊœgBdffΊu‚‡Fι ΅pZ^ϋΔ–m;υΙ²οUΆLI·O˜=dαh"ŠXΏq‹.Ίφ>υνzΏΊEξ™93 Ύβ,­ι|ΰ™˜˜ΰ~ΈΉμλŸ•˜―3κΧvŸ’—}’θέSΏW΄jνF]vc?έzέ%κvΛ•ΉΝ½5w‰|ςUχߜΈηυςŠΞlp²’“‹ι’sΟ8UύžΗGOαϊΔC^ΡάEŸkδγχΊΟœs–kΎρž'τDΏΫ΅zέf΅lΪ(z.‚3±š€3λΗ™V<9IΛΎώIwυ«.9_ύΊwΜ}ηoΠǞθΞsώώp–@2}9ΟκIΨaK`ϋŽέϊξ§?τ―3OqΜy¦Ψ3/OwW)ω{ OΆ[Β ‘@&€hό…Εϋ?1ZΞΦ“^xΠ½η[ώc&Οq?τvώ{φΔ'”RΉΌξξ;άύΐΒ™–ξ|»Ί+.r‘δΐ/‘― r·ρSψhήΎ—ϋ­ ηώ:?7oΣπ—¦Ήt;³Bώ3υi·8Οqώ qžβόσDΏ;tΑΏNwί¨ώφη͚ψΎ-όέΑEΣ[άξΞκxεEΉW΄~ΣVw‹vmšκn9”<2l¬ή™χ±ϋLjσκΝΧΆέ7Άe‹"Πώ~ ^ϋDzF¦;Γπ―­ΫU¦TIνKKS{w?Μβ§h%°δσοtWŸ§υφΨΗtΒ19Ki:?σ/•³ΜζsvΧ…ηžξ~˜qο#/(m~•(ž,ηY¦O=x—vRΡ „«}‚Nš@—~֏ΏΤμ‰O*9)Ρέμ,Τ’C/χ[έΞoόόπΛ υ4R+ΧlΤ+Γzρ Ϋ"ڝςκΞ¬λ» Vο»Ϋλ¦k[ε^Ή3ChΪΜErΎX1wS,»ZDϋD£Φwκ‰~wΊοœŸOΏXΫ{Σ³ƒΊεώ›σχηΰg_ΣΩ λjοΎ4½2iΆš69MzρeΜ’Φ-–ΌBνξ Og½θ.•θόŒσ–ϋΤΌΙCU©BYχίθEνΞs=$w=ΓWΞμηFΏ‘…Σ‡»ί΄8πγ|ΈΉθγ―tέΊίʚ9c½0δ^{Φ)ξœΞtΓ»oh«ήŽΤ%5Ρ#=oτu= 8³<œυ˜C—4rΦeήΊm§ϋ ηCΝΤ½ϋ4ρωώͺžRQΏό±Fχ<τΌzwn―nύŸΥ°‡οVλ ˜ύ=wΥί™΄ξΨGujΧΜ]ΏίYšΐYsυχ•kέ?BΜτpf„­Z·IΗΥ¬¦ίW¬Υέ}ŸqϋΑ½w\sΠ όφηΪƒ–ςwv] xν›·lwΏύ_‘\i-\ςκυθH=χh7{VύάΣvΎν»fέfΥͺ‘R—ΒkZHΐωΆΛλz»PΪΎνn‹Œwκ:XΗΧͺξ>#hΥΪMξοŽz'Υ CzΈο9ޜσ‘~’fŽ7!ŠNΑω]ΰΜώθίγz]zQ“ά3s>Τv–Ώzgμΰƒή8ίμΎ‘ϋχOPω)z „λPdήλC•Ÿ{α#'Όγ~πι|醇½ώpΰŠϊyYΞοgΙεŸM8Kax| κ}‹»›σΎ1>.ηy„ΞΟΪ ΉΟ.1ψf‚±α|™κλϋκ’σΞp—ewf8‡6lu‡žμ§.mžσϋ„>QΔn<—Ca@θΎp~©άΡ{¨vνή«ϋξl§ΚΛΚYsΣyΐ”σsΰ[8]nΎBol›ϋZο-Zκ~hα|xξ€'U Εφu#’θ`ηMd§ιŒϊ'κ–φ­έ{μό4¨w‚ϋ_~ν]=;ϊ ϋwίƒΎ΅λ<{α’/utυΚ.ŒπSψψ}ε:χ›5«ΧnRχŸSκ•έ™^Ξ2WΞ·2ΊχχΓNηΨ™σ?QlLŒšΣ χƒlg ΅Ρ“gλγwFΘY η«οsg½0ξm½χϊΠάo€ώ€Žœ+πΪ'œηK-ψπKU©\Νρqΰw„ƒbIΕτΜΐΪΊ}—vμά­‘γίў½i1δž#'Θ"t₯ΞƒηDΝ^π©?;Ρ}8©σ‘Εό–iWκ½1ϊQΥ>φ(wv ³ŒΑ¬ τžΑY:λζk[»3̜υœY!BFŠκˆΉ›}β―­;\@u>τΰ§p'ΰ|*tiLη91iiιzmΔΑοZ2}ΦzwόχY!Ξ—*ύχDχƒΡΣO©­[:΄>hΆYαNζΘ=ϋΠ>α,Ήά°υΊ©]+έsΫUΉΑ8ο!―Όυ!wIζ7Η<κ~ψν|©bŒyξο&gΤΥν.Νύ†ψ‘›hαΏrg™Μ>ƒ_vg‹9Π+η91~8Λf:³JbΗΥͺk/k¦k.=_™YΩj¦³ϋ98Vψϋ@θόΉj½ξyx„JO5mšκΫΠ›³?ΤΜρCCŸ(z·œ+"° tί 8ςڌωzύν…rh~ό1Υ5fxχNη[8ί,] ¦=}ΠrF“ή\ΰ>όΈO—Ίαš–ΊϊΗ•-SŠoqϊΎ#ί€σΑδ‹γήv—½Jέ³Χύsψ€.ΚΘΘΤωWέsΘRΞχxx„,ωRΣ^zDujΧ’Oόmτ}Oš¦OΎψ^oŒδ>οησ―~t?„zvτ Ζήύ¨Άοά­vw pοχ©uΣWίύκβ™3λcΨΘ)ξ{οNx\6oU§.ύ?{W]Υ±Ewq+ξ.Εέ΅Έ'Έ Iˆ'„ΰ ά%Έ»—βPάέ­F)―}ΪϋΘΪ¦%φnζ¬υΧ_ /οΞμ;Ήw朳χΩe< Π[H…ι!5qβμeτ6…ςηBΞμ™pϊό5ΡoξΥΉ™$0Χ­OK‘Θζ5YŠš·ι!’Fά΅Ώ7*–-&ϋj5³s;yς€"_cή€–άk²AΘfΫ}»΅4{‰Ε ‘M³Ϊ’λΌϋΠIιWaΊΔδšθnη+{ šΰͺΠVC‘6M*a…iqωϊtκη%IpMŽ•Eu6O΄jRK’Ÿ WmIΕΆΝλθ”D<#žI«4λΫNr΅XΎ~7Ό‚#0}¬£ψCΜ[ΎASΛ{£z₯R"·xυΖ=,σTX:Y?{ΒŠυ»ρΰρs4­W6ζbŠΞ’™Ώ‹5ήΎύ ΜG\ΊvΛΕ†ν‡%!N^Κθ%ςΰΡ3Τ¨\Jš4T˜6ήΏΊπ5vλω8υ10ΝL©Δ3ƒθ¬‰ήxυϊ')j°K“ΟTιΓχΚκΩ~"“φθΙ ˜υtFŽl™AσBv³ΣW…i!ΐg@δζύ†ΔΦ‡xLΔ±S±ρO†θšΝϋAΟ±=«& kζ Ÿ Ω‡ζ–#…eFƒt[»΄n({ ¦…@L­‰;ŽˆΤj½šερτΩ+a”ΩX΄BΩ…L 5ΪΏD`λžcpς ωxώ=VΜπ‘€…φάΠ|cR€H¦€σβδΖόEώiM΄ΆrC΅ %Œ:Ά)[ͺΧH΄kρ­tν2ψά8pτ¬$Ή˜όfβ[ΙάΔόύŠνodQ‹Ή;ω+zόœ»t‘ήφ†"禝Ld‡γθΖp€Iς³C"‹pοαS"sA―)>WΌBζHe8U˜1±&XL7λαΎ+όœ­Q o°FuήωNRaΊ°@Κ}·£•°ΖΨ\Αd%²Θ6υsξ+ο v{‹£ό Yη|Ξ>~^˜†«fωΚ»cξς-²G Ÿ‰υσ‘#[&Σ'‘Žœ¬ξ)‰Έlέ.iš ΄*%‘έΌΘ‘X₯:ΑισWΕ9ͺ\"½#^ΎϊQΌ#Θ˜2g υi‡―4NN€°κfΪd|°ΩŽBZc&6ά{*Aφ’l°0›vAΨhΤ¬RZš±z D « nί{Œfυ«κ“Δ>΅&ϋ PσOŒ¨HbΌλq8g'ίp?{Y›€2ω9άk*ͺU,‰π aπ‰ΐš-°j¦Θ0X8‘ρ)7ΝT“Š } ΐ„DΫήξΘ“+«tf₯: ¦Ο_' ‰αύ;£OΧ–jMθγVΝβφ½GB;&•œ:ΛQ;―.]»#F₯­›ΦFϋ–ίŠœΡτkEήhN¨ ͺU,7?ώŒ–=œQ½RIY7Ÿ JT(ύ:˜˜Θϊω»5Α$₯„"]Bι²)‡φόΕkρ~a·ξ©σWAi›QΓ­ΠΉU}΅&LδΎέ0Ι€?χmšΥA₯²ψF1ψ »£QJ"•ΕΝΕkv gΗ¦"sτĐ9τρz αmΔ²M8ΆyΊJ|Sψ’51eΞjL_°+fzy=π}“6MjIrͺ0]Ξ^Ό.%τΞ”ΪμίΙ’%ΕπQxώς št.gϊAΠƒjΦ’ ςΜH’$ ξά{Œ-KƊG€wΘ\P&©zΕ’˜2ΪAyŒ™ΰ oά‚•ΫδžjΧμήocε&~sLz³RΓάύ-ΫySrΊό7ΰΛ•*‚ΐΙ‹@–ΠίA"εKiV¦ύ₯ΘK*₯<7Έ=wρ†A^Sσ.% (jσ& ¬ΛΦξΒΣη―°iα%“fΪΛΐhτjMθθfͺ©(’€*€D$υ‘Ž+λLnσPΑ'i菟Όι«Τ©S’iWGΡh₯wΤ¨Φ²Ώθoς`“+G–>υ› ‡Ÿ‹ξξήΓ'EζΞύ'Ț9=ΦFΰι‹WjM$Έ;ϋbχ~ΔM’ΏLωφ R™’˜δ‡©υΨ©K@Φΐ†"KAŸvχ±“3UΚbtHΖ½FhjψqD•a‹ύΩ¨+ΔL|2ΑΝnLΟg-ήhπŒβ{€k‘ΓZ>έ[BvMΔΔΥwΔ-L^:ϋMΝ,ι€Ζτ±ΓQβ›θhγ‰δΙ’aIΈ§ ˆΖιχ>5b˜i£f²K™₯Ην=Œι«ύΥšΘ”1=Μz8£[ΫFF¬lΒhΨΙ-Φψ¬'ˆZ1}‡bχϋ(ovρκΈι‰ΌΉ²"dΪ2Ρν_8ΕCdΞ4 ΄ΛΗ$­(ECd²’Jα؍ 2Κ/$,³€‘6±;xυν1Ž ε,€)˜ƒϊ΄Fλθ"©IφGΊ΄© ~Rš ληΑw ›΅2fH‡†u*‰Ώ%#ώ#mΔ3@…i#ΐgY¦)S$ǎύίƒ…³}k&Ι“Δ3Ε–EcE‘@‹°y‘`CΩλšI:σά‡ςχT˜6jM˜φύS£WόTδί ₯>ϋEp“Π¨Σ0΄hX.»K—ΞθI ?ΡτΦ:{)YaΧ«Mv<)ζΘΩ²d_¬«0mx_™πnΣΫ “όΛ!C­ ΣΎ§11z+Xεί> ΩύίΊ—+l-[ΊυήΈ+Ιτ}ήΏ ­ή’Eσ‹ψΗΑn>abˆIΪ» ΣC€ς&μΪ^4ΥCΌd4?ˆΉFŠV;γί¬ ΣC@8*LNΌzσ“0Ύ˜xXΉ>‘σ°pŠ»όL ­“S“ΒωΕ^CF£Yύjθή‘ΨΔψxMPδθ JšŒ‘€§Σ¬Γδ9«₯α‚R8QƒΕVsKΜ‘ oMd=0iE†ΧΜE€XΞ{ν5άJΞ z=t°φ4ςcφ™ά^<ΥC ¦ϋž­SΖyΩΙ3Ν[)’$5ωίI’&1° LžD9Lξ YΰZ»υ Θʟ'‡°8ŠΞ+xpY―έ`Ω_φξΪB~FζqTiΕώΞ!8ι¦4XhRhdL˜ΉΫ–₯U’άΔ'MifϊΉΩ’^Ν ¨ΒV3ΙΣ‚ηΥφ}=€1sΥl_ρ?cΉμ=ΘF­X¦¨0–΅ύ¨‰C’θ‡―ΦD’_ #  :ΏΑ ς‹Ώ IDATmz4wα[w9ίΠyΈrγ.XXUaZ°ΑŠ›Ω²dό€›ς›d‘’I˜ώλ΄έMY$­ ΞίmΧΫ™2~y]?;qv~_½y‘ήM ˜DԚΠΗ}T³P|UQλ"ήΠΊz™΄`g6!γΒ—‰!uώ[6ͺ.tΣπωk1‘ ?iΗߝΌλαc±tΪ(Υ₯ow/v.¬ΦDμΰjΚί:iφ*9œŽσˆo«—5š ½θ 'gVœsΡztiέ‡Ύ?'ζΆάt¦I“ dJΊ`²=y¦έEΓ•fθ”Ίαα…Ρ©ŸκΦ('―*L΅&LΖΦ ΨΑ½$r‡$/|ϊ]&|ήZ9œ’Nv٘)‹!}:)xόψΣ/νΛβ;=΅8ιFŽž!IŽ“έ•œ^lέΈXό^0f.ή€Y3‰ vΤψ»5Α.=vρΡτ’v‡Ϋ[΅Sf·±x―βγ«΅ΏύΚεŠaœ—½Ρ._Ώƒ‘ž“%ιE“Σ›ΙΪ0QΪΒΒYž ”ΐ‘ό€^m€P’%:βcNκš_†χ–]ψˆο}A’Ζƒ'0Μk*R§L@·~¨[£ό—]Lύv‚F€Zώ43žκ"¬fέF yύjπr΄’q³kΡͺν˜:7₯‹ƒt&SFI…Ύ Μ;ω΅b˜6ΫΓߟGίαc„eΘ†=zD•Φ£RΟͺŠ!dΊλƒΕ0Κ£‘ DΑΘ-δ«―ΐ=Α΅›χΕC†²‰]l½ρ£ψ‡₯ι4gϋn’P±σΐ r›€c›§«=₯ι.£‘«5‘“©¦‘ψUQK"Ξ`Χvυ–ύ1Ϊ΅ŸΘXQ¦†uμΜœΌΛ¦yΙ‘ƒ¦d‘3–ΛSgsύφΓ±IŠ2[Υ}BT˜>jM˜ώ=Œλ°£»QgLπ,t-Έ–θ Ύ ³—l¦ΨΗ’WLW„]Uβzκz1‡ΐί­‰KWo£ηΰ1Ζυpθ%[²‹(…2’›‘ώw̍H}S| ΐCkθŒX?o΄@£†&ƒΤ£CSlΪyΙ’%9$νBFϊ Π€ŒλΆΔΈiΛ%ΩΕ= ׏ ΣC€‰©[χaΥLίO˜_,€8ϋM“½%χlΒαšΠx‚soΘZβήT…ι#@¬‚ωraϋήc"…DVΠΗέϋΟ_Ύ†η˜Ω’Ψ4kTC<Ζ>–Λ3}$Τ ’"ΠΝΞW$)Ÿ§›pΪ[{ mκTβ3υσ/Ώ’IWGa“5B&Yo‡ ρR Σ]O|φSjuΛ’±†¦ξΫXΉ Cp„]Wa’Φjm/ŎfυďŽΜrz¦L‘τ0έ½rό' œΉp·ο>e ¦ƒ@l‰υΫ‘PΎ\jOa:ΛATG¨ˆŽn¦)MΕ;$ΗΟ^cΒΌΉ²‰|Mίac`ή€¦aγIšϊ•λw%Ι ]9pςB\Όr/_(τdvxͺΠjMθγ>Ζε,hpJɊƒlΦVBCnΫ]Ϋ6όΔΌ”Ώ—―έΖΊy£γrΘκZ±ŒΐηΦνϊΈƒΕ±•3} ›dž™υtFGσϊθΫ­e,L}}\!ΐϋzαΚM”*φi²š2HLl† ƒ\Κί°s·Ώek)†QRŒΣ@Χ~†α²ΛΣΒ|)˜°X¦’ΰqu'cξ:,z?zς…ςη2ϊRΦVnθΤͺϊtm‰§Ο_IB‹:ξuͺ•IJ3¦O'ο“ά9²ΐίΕE *˜˜»;ρχMτ•κ6ΐG$2™τΞ”αkΓ`؜աΏ Θ…‡ŸαΒε[Ψ§Ί΅m,Hτ:vκ"¦I…JeŠ"yςdρ7uεA€ ²ƒyO΅X΄z‡<—„yό Ι$=i!έΊ/Ύ–MκVΑ0ΫΞ12υ%ρƒY=τ +R0·¨Kp_ΰ:z†4h’ύΑ&*S4θ8ΤΰΒ‘ςέBζρό[`Ω©™¬>*•-&Ο >GXXcΎ‚¬S¦ƒ@L­‰A};ˆΒI₯²EeςOž½”Fή`Ο¨WS1NMgE¨‘κUΡ˝4±yp“9~Ζr,\΅] κξά„²% cFπΓΖsʜΥBAέ²x¬‘žζΦ=Η°yΧQ1¨S‘ԚΠΟ½Œο™°Ϋχπρ0b@L›ΏNž)ΞšrlμoΣΫMτΌ•dA|ί­ΨΏώΆ½ΗDφˆrHTP‰rΕŠύϋίW`χf«^#₯yBλξ:¦ΑqΰθYιϞ5£Ρp™Δ τ&M–Γβ{*κϊ1„ΐ‘cηΰ?qVΟςύ$Νξ^6βτθΠNvέdΊpΥ6LXƒ3}>aΕАΤΧΔ1άlάy=Ϊ71bΡ ˆΕ° ‚dΑχ™΄’‡£eή!sΖτb μ>ΔR|Tθ6Ϋ΅°pBύš0擉ω†ΞÒȝΒqd‘ *τ‰€ZϊΌ―q5+M«™]XμκΧL’§/\β…σ‘hαΌ’Ό`ΧΩe…?κŽ«qͺλΔd—ν:xRΜ)£š—άΆYχp°ι$ς7χ=̅λρΰΡSΤͺRζ³Μ‘Έ΅ΊRl ΐξ^z{|GNό€>Aςc&1™π&;5j tύCFim„l M}g?†zL‚Ÿ³΅*‚ΔΥ Šƒλά{π~ζ ;ˆΕο¨Af }ζl,ΜπςυO˜΅hƒxA„Œ²“BΪΡ@y=Jg5kPE }H’ΗΑΠΥ%ΎΪ?{ρJ4?6ΌΏyη!ϊ–8Ω’WnάC“Ί• \τ:XΊYF1ΈoXuiώɈXx­šΦ’χΗ uύ#₯Ώr―°|ύnμ=t I’$AχφΠ VΕO&:fΚb¬άΈWΨ$”XμΫέΜ`N =²‹Xσp°?{τδτξΆ=[‹—‘ ΣA Ίk‚ ’ΫχΙ:Φ―³HΏGe™ΞŒΥH¦‹€*€˜ξ½KT#§ εΝ;d£πqΞ?Αͺϋͺ{₯γ»f•‘£ΒτPkΒτοaLΟ€Ts&'\μ»…ΩsP€°?<,.K>&.R₯L’Ω δΝ‘6(R wLO}_‘±ˆΙ±¨οJ°hJΝξ=’NŸΏ*zξ~Ξ}…0{ΙFΜ]Άο~mšΦΖ`λς.Q‘Oψόoe9RΨ?ϋFyGΰθΙDκ†ρΣΟΏΰκ{(/§Hγ¨Π,š7!^QcΟ‘S° ‹φ Εςg/^ΓΒήνZΤAΏjί ―•πωΩ°Yžb”9b‚›ž1lΞZ !ώ7Ο+fx―Έvθ7ΓwΘτ`G₯x‘³…B92~6,ΔΞύ'0aζ Τ«YA$xY$#}Ηώγ’xΌAutjUΙ“%Υ j: ψG@@β¨Δ"Lb؍/ -vβψξ Κ•,‚ΰQvκ₯‹Έ'δ―Vk"!ߝ/Ϋ_IάΨ8#S†tγΡίp‘-»‚f耦΅ιΚωΞ^ΌQŒ‘#ηψ#kζ _> υ πe ¦{³ΊB?§τM)Γ‡aΣ# 96εL’vρρpΚJ–L9ΗO€“lB€Βξ_Υ‘— nν BλΒ€ΖχœP£οκ:ΐGΌ@jV.-&Ψ,ŽΠθ˜Ε1Jͺ±0–-K˜5©X©_tή/3ΉyωΪm1½}‡Α₯k·±eq0f-ή ζ·μψ^±Φo?ŒΤ)S ΘέΦ¨£σύϋύΙͺfυ«%ΌΙͺEΎCXLί»z’Ρί< ₯» ΙήΡ6βΒΟQJο {«ΆRl8˜τβϋθco€hH}0A!ΐΏusKyO¬œιc$λνDjρζνΨ° Θ(Ξίm{ΏΗΰΎνEκ†žΤy―^©‚=ΔνΤΥbJιΆκ5­šΤYDJΫψξ,μ\BΡ¦ymŒf…v}=Τ§{δΠψŒ˜Ά`Žžψ“ύ‡ nςωK?σ&΅>‘ζŒρ«/ŒΈ—€Ϊ±ΝΣ ^ΌπΓΗΟ₯α¦GΗ¦pμίE|ΔΨ\AO:J₯•-Q“ό‡~’L@ι,ϊ”/U$NΖ―.ϋhidS2―tρ‚rφˆΪ€Α‚ιƝGpxύTΕ4Žύ[―WΰY£aΗ‘¨P¦¨‘AO}†XtίΌhLΌŽQ]\! 7TDowTΗσ9tμ}ΓΠ¨N% ±ξψ·’LVΥnm– k|bTΙΞ>J΄oY‹Vοΐ˜)‹°iα1FN“:₯ŽΤίΤbcM0Aέfκ²ςpK*» ΣG€Ι¬Ε«w`ϊXG™ŒηΨΩ8yφ ΦΞ »ΊΩ™3yΞ*”.VΪzΣzU$Ή0qθ9C)·!=ΔGβ΅;θ`ν!‰sšͺ0MΨ΅Νwel˜Π‚8+Χο‘nMΡ΅­ž›’ΟΦ²\ό§IwVxΠpӜΌυί"0Ψc"~ύυ7ΡπfN1„t°>˜› ‡\’―ΎΒκ9~xωκGPmΝ¦ύhVΏ*κT/‹³n`αͺmΒ@§&,@’θ§ ΔζN‰*ε‹αa54PΌANq—Ÿ+ΦX’_. €B@@bHυ5qƒ»o'Ν^IΩX˜£G‡&ŸMPΣ ™]waiυΉxωϊGω ;ΏσηɁ«7ο‘t±‚πΡΫ`”7³RWωbrMπ KΩ“ΒωsIςβζιηdίέ¨›λKΖ«~7ώΰαC3”£!ο7uš΅ΰa„Eέ‡NaΓΌΡ8~φ2ϊ―ˆžšβμΕλX»υ ΦΜφΓ χ bšMCCU$‹Ώ{ϊ%W&%ˆk6ο‡WHlΊ›‘Δ7°aΗaΠΖ²S38Ϋw»0ω¨W£Ό‘‡ “ύCΔ(™Ο ϊBΐΙ7\ΜΟMυ‰ΝY² ‘3–‹TΣιω`ΥΉ™h6·°p) ϋήνδ³μψ}πψ™tuF5Ε&«„Οe„kΊkEϋ»gq΄h‘Ό†‰hžR}£Ρ·•pσΞCaqΙ„wκT)aΡ‘ ΜΥ0έΙ«‘ ,dϊώ._Ώƒ"r£N΅²0Γ’)eŽθ5hnΛnވρ.¨Z‘Xιlλ '»hΩ¨†4tρχU˜6\½‘,YRΈΨ s¦τ˜±Σ¬ƒλ` ΄iVGŠθτ€€ \,’π™1#Ψ΅ͺ”ΖYG›QaΧ½:53m`Τθ²~XΈ@0q‘x>dψ: ~^Φ dšT“ΰ_%ωΚΘSFAͺΘ£bΙρ­3Œξ/›u)ΟΛ3Ο"Ρe­ί~HφœJΩDλCΝ"vPΨΑU}k,#ΐCGΰδEΈχΰ)œμ»Χς”¨έz έϊύ₯.77LbΠψ–Zή”8π žƒ#ΗΟcγ‚1*αΛχ0¦ΏώKΧ.ϊ csBΕ”Œ’kLXε͝ Lj¨Π―ήF[/ψΉXΓΌρš«Z’LošYfΘ^ ZGMo―ίΎυΫ}bš­tΗL‚Γ—Κ;„G|~πž2 uψψy‘`gnϊtiDΏŸ…―νKCŒ˜‡τ 7»½‹Μ#Ζκ”[δϋD…ι#pλξC,[·[$K”₯ΈrύH]1ψ ‘ΘΕ+·ε½1o’«7(“5mΑZI’²³ΫΟΉ―μQΨΞ©– 7}„η ΘiΦ}\χ@Χ6 •ύρMΑ<˜κ,?£©ιΑcη0jX/4­WUΨ„\CΆ=[)).nvΎΘ™-“Αψ˜ΣdcχμξΥ ‘{ςΗρ3—EϊθωΛ7Βςt°DΝ*₯uŠLβ™Ycγg,Ηςυ{πΏχΎ‚½U[iΪΣΰ‘sόρM‘<PΨΩMΟ!N΅³›Ν«gϋ}"•xΠΤΟLΙ4g# χŠlΌfΔ' ώ0ΪΏQ&’j{ŸD€IEv“ϋJ§©Π;0κ3ω¨ξφ~xώβ5ΦΟΔζ]G’Ε“ΖήξN¨_«FΪθ 5 …@,   ±ͺϊΚΈC€”bGεΝ•.»ΙΖB v[ΡπΦΓΑ9²fέξv-ΎσZ²=Ϊυq‡]―ΆBAΦβΘ‰ΠΗ!KΒ% ©ys"{֌Έqϋfsι‚iσΧ 5ΙΞ¨~0Z›‰Š™Α#ώv"L’³¦"a"°vλψOX …ŽΜ« <…΅γXIFQKλΒdΧv»­ΡDϊ;“ίΉ<τO=w'τθΠ•Κi”ίδO˜W£ϊOL™³46έ²x¬Q·φΦ=ΗD§™…΄yΛ· hΚbIvΡSζάΕp3Kž\/,ρ£Β4`W&7l? σ₯°Eσκ­{ΎΓλΒΚ™Ύ"sΔbΩƒ^ŽVθdώΑδvά΄eΒ8£DŽ ύ!ΐ}Γάe›1wβHa‘‘μΰ5'Ο]Vώ7ξ8"2Y|>Н’x³oΔ€Y+±jΆŸbκdYΠ/†,ς<Ή²Œ¬ιDYΕ“έŒfΙg &λζȞQc“9ΩwCΆΜQ‘t•τΦΑΊΰσΰ—·o‘3Ϋ™ZπΉ±|έ.l_6Ξπ3zΓt±υ©=‡^ȟ'»ψš’)²hŠ δΝ‘DΤˆσXwYUΙWΝ[+ ΣƒE™$Ί¬1>GΈΏΨΈ Ω²dTΰ*€*€¨₯aς:oΕVμ;rZθεZP™Ή³—lΒ½ODŠΒΧ©―Žρ`zύφ9hέό=jύ―ή΄{VM?ξvΎ  ϋО&SbšΐΏ]άXŒςc£R‘φ}=$!uuͺ5‘―•DΆ›,~πo½IέΚ’ά€–;šgˆ6k?!³|₯ΰϊϊΝO εψνoο€ΛWσΰχ΅±rΓhΧ~hΩ¨ΊΎ@ΣΙlψχ=Β7g~Έ&sjq‡ΟD±"ω0'Τ…›ΥΝ‚ΉEϋΖFώμήe/5š©Υμμ?M$Τ}Ο•= v8r₯Š`τH€K›Z'ˆ%ξi<}ώ m-‚Γ—aξ)†’ΖB(½_˜ΠΎxε(}B_‡!Φd‚―ήό„φ}άΡ€^UI–ͺΠl¦π?_€2ΩωϋοΏΛί8™’,ŽήΌ[ £L 2FΜΉ$2R.ψΩΖίV‘  ©s#Ήy?‡y kŒ2gC<&J·& Žω|X°rθ1ενΨΝλ&Αb{“.ΓQ½RI„Œ²ƒ¦ρΝuAσcΪ4«-Ο¦…eϋ#ΝSY3gbϊΟ? · ™8}ώštδςoš~S"ΦHa¬dΡ†Ir/Α&Κε‘©~ύΦ} χž*^QŸ%¦…Šνǐ-θδ&ϋG³Ζ5qβμexŒ™%Ο‘M ΉFRImʜ5˜δ?Ψ ο|ψϋσ {„XΥ+–Tΐκ 6k²8~fηliΐ’”Q½φCP©LQiΐa§σΐξβ7D†ιρΣ— Rœ%:Ι"©X¦¨ΞΠJ\Σ‰.kŒΜυσo S«ϊrί›7¨.’¬* OP΅*mz»Ι!5*[„ P―™ZήLf°σΧ¬‡ R€H?gkιg2œ—ΙώCP»j™D‡›^'LŸ&%˜Ό΄³jk˜&7Lvsσ8ήg tξ©5‘ΧU`‘Υc–Aφ2q"ηψIvκ±Δ3‹θ°ΖΘ8η™³|©"°κBTO¦D¬–>ϋήνXj¦ h"  ΡJ}L?άΊϋH Ž£05½ζ±D†Zίάp¬˜ιm$WASΊ΄iRb`¦tηLΈω,Wͺ°tκkAR(“–7W6΅&Ωΰ!ƒΚ’U-_›vΕισWα6€'Ί·k-™₯±ƒ4x|BηaΗΎοE―—…-˜ e°»K…i"ΐDεkΎN›ν› GΆLβ α:z†ψ:DMvž½x] ZbΓ4g¬F]4ΦΨΘAθΡ‘‰αΧΨαΛD„λ` Y3{Β`χ‰b€ΛN^2RY$eρ²Iμώ_8ΕL„ rŸk%™έ›‡Ÿ;~ζ’< ˜€jΣ¬Žψώ0Xμ^°r«$‘φ=ƒ1ξύ₯(ΚΟ“-HIš“’YFΩ<šb7­χGŒοˆŽ6£€Λ—Ι-¦…ΪξA³pμΤdʘ^’Pά+pΟΐ ›ψΠχηP0oNƒ€*Nι6St4―/r»\7ΆN!Rπΰο³Ψ:kΡρ€’D’ ΣC@ΫπJφ†Ξ~ΣDξjλ’#ζ&ΉIƒt₯s rŸ( ς¨¬uΣCFXCΰΡ“Hu9‡RV—kαδΆ™Β4fάΉc¦.Ǝ}Η…V£RI،ΖΙ³—ΡΊYmρ a!„μDώ?ŸΎͺaΣ΄ΧΨ?±Ζ’'K†ΦVH—6x i²ξ<›π³cΩ89«¨P(>   j5$z4½f-ρ@³Ξ­ΝNn-ΘhΨΙ-ΦPž :_5άh²SOΣδUkBη7ό/¦GΝM;HΑ”›I²@θύΑΧa*”ώSG;ύ6‹&Ϋχ}eΣF‰§ΘȝΨ{δ4&ψέ^>o–―ۍ-λŠFΌ ΣE€IoΚ–°xΚΔDΤ ?=VΟρ“ϋBtΆυ’Γητ±#„!Ζχ›&R§LU³}ρόΕk΄κε*Y!#δοŸFΉ=ϊIbœEχ₯ΣF‘LρB-ΞvΪι•$’Ι->ην\BρΰΡ3Θ—S–Ε ηΕlρϊJΊΌK-hΔc„Ο“²% «ˆΙέρΎpε–°7ΘdGo~όχ>EΠ”EΒκάͺ>yλαc$yΉfŽΏΘd΅λνŽT©RbΡwCς“{ ²K7ΜTμ ]4ΎΏxυά‡τDή\Y2m™tϊ/œβ!ΚZ±œμ°%ažF³$#ˆΜΑΛC₯ ΪΎ―'^ΏωQήLœ7¨]QΊ΅dΉ‰B€†ύ',¦6ο>.-ŒdωΟlΔ’Ώ\Hψ2aπ BYfΖςυ»…•Έ}οχ(Q4Ώ‘ΧŒΧ΄ˆ.kŒk€k τ¨ΛZc―&·ΘΌΖ‹Woδ™‘B!ΨPΔΎΤόA:R §./κξ=ρƒΘUD5―eΗχδ9«±6"ΰ“»τq²ο†Βvχ(hM&»'ΝZ‰΅sDΚ@­ Σ½—11rΚx…DˆΙ9MΩΝΫsP€.θλ‘…Θ©uuD«&΅DƒτσN¨S­¬θΑΟ[ΎΧnέ—Κ²i^H’D±AbβώΔΧwμ>xŽ>S±zΆςεΞn‹f4Eζ;…χώΖνŸMZ½}ϋφ>e@Š―y¨λΖ μψžΚ^‚|&h]˜cfKgη†y£Ε Y‹€‰ ±lέ.˜5ͺ!Œ&?2gL―ž1s[βν[(“GO&2Θψ‘Χ»{'ΞZ‰Ω‹7Κώ"ž†ρ‘QΒ½Ζ¬'Τ¨όA:+ή& .+άΎχHŠcό;σΣΟΒš6ΖΕ ηΣcjό“PΊxAΓυΧm=—€ιρOI^¦‡ŸΛ6aζ’ xωκG9_z · ƒοs― Œ,@-Όsqa{ΠKΜ²S3,\΅”LcΑƒΝχŽχΈΉΒœκ¬XΖ¦·4>;bΚΉψOC³ϊΥD2ρΜΧ0Έo{‘=c2›’ΜνΝκbΤ°^†ίΧΌCπΏ!2"ΰoŸ\79²e6’mΥ tΊ˜FtXclcΣDωRί`F°£ΡΌ©F°vΛ~쏜,ϋ‡Q“qυΖ=ρŸβ³EKu±LΤ$ώ#ͺςSΏ¦/(cΐΞM-±ω±y%“|ΙP—•Τp₯!2 &6©χKzσηΊΊ£nλ 9ύΞ&ΆΧ„~‘ΣΜ4­~1’&)(i²λΐ IR0©A#ά•φ`Γό yΎh]|ΦέΝDΚBΙa™ώZa"ƒ&ιZ)hήΣE η©DΎ'švuD£:•ΰ8 «xΕhA‰½qΣ–!rŽΏxCΠ›J…ι#@_‡wο~;~)΅ΙMCλΆ,Μ―ΠΩ€qξβ 1Α₯ώ?eQŠΘƒΡ6†b™ώšΠfΐ€“™εHτhίΓϋw6LŒϋ ³ž.ψ¦`„ ΣΟ„ΥL>‹¦=‹_~ύuͺ•CΪ4©„1Θ†‰FίVιD-Ψ€A‰¬―Σ₯6’ΥTК&Όχd†1‘­ΙΥh3!«”ϋFzΪX˜‹6=‡J͏9‘. ‹ˆk€v΅2 d³”η€ΔžΖ 4MdΤ¨?F€,RzΠC,[–L°·jƒ"σHΗό[°eq°‘ΌΡw'/HC§U—ζ1 +ψμΨ²ϋ;ά}π5*—¦ƒςk­­άDFkΤp+|EΰŸXc”ί]»υ ϊχl}>x}pέ0gEΟIOKiΒθ5d΄4θ2GE:6κ9ΩwΆ² …@bC@@ΫWσύ[˜¬Z²f'χν`Τ}92`φ>)¬šΦ‘ŽΞM)“ύ-[K·/;Ύ53Σ¨‘vλŸ08Ψv#D¦…@l¬ ΣB@φc˜δnΫΫyre•C(u{§Ο_‡)k$©Υ§kKazPΚb@―6ςŒΰΆ……³θ»³K‹>Κ]k‹fιΣζ­Ed„Ώυqώ?Ί;/έΤΦMkγωΛΧ²Τ灀D‘όπΡۈM’?„njΨΕKŸ‡RΕ ΚίΌV47Ρ•Λ}03ζΪ y²CΏN`atυ¦}β#@o!v}*֘ΎΦΛΙsW0fΚbΜ!Io-Βη­EΨάH¬™γ'Ο&›5aδKΖΟr]ΊvGδ9Uθ vρoΨ~ΘΠ@‘ΝnIδNψ†ΞΓβ©"}£BίΠGhτ€ς. k„,γ@Χ~Βππ?‘[φiŽΡepB§/mΝ[ί(%ξΩYΨϋIρlvθ{H-¨Dqϊ‡«²>θ-3ΐeΎ;yΕ‹δ™gέΫ5–ζ+m½¬˜α#FΙΎLΏFΞl™7° lφΗcqΛ{\rηΘ ΎΈ§ ͺ ¬τ:{αΊ4εfΙ”ϊ’s'bd«³©—ŸΙ›;Ϋ'ς½ 5…@¬   ±«ϊR=!@ΪiΧ>βϋA_-^ΏωI¬ Vm“)»x?ήx2 ΩΦχ>ΑΫ·ο„β>r°…`4ρς%kΒΔ§†'4)¦TΕήΓ'eCyηώd͜^$ςH-ξο‚+7ξ‰f7*μτ'KŒMΪωϋ½Wή:\M‹Vοΐσ―`ίϋC7–6Ν ;£xα|"‡γUχ!ΔˍΏ­,fΚ€¨SkDΡΣυ΅84}w&,h~Ξ`¦xΕΌϊk#όεΰΚ’ ΅iŽΝŽpΕΣΧ:ψάlψ.‘'M°]vΦXΧώ>H–4 †χοbΘaΒ£½΅ΎN›}ΊΆ@Κ”)P£R)%a’“%’5U±{[ MFσΫκε$ Ξ½'ΟBψ,ι1Π_­Έ&T˜>j 1ƒΐυ[χEjS“ΡΤΎ•²ΊLb’YΎ}ιΈ? ₯Nr–jΣ οίΏGHψR”/ύ BFΩΕΜ`Τ·Δ;lžΛ”!½$6΅ΈqϋΪXΉ‘―ΦΩOoΏΘ*LnσωαλΤΝκWχΉ¨|9τ‡aα“ρ$_%‘‚ΈΦεoο:§Ξ]Εφe!F2Š WmGΐΔpΆοΛNΝDŠ5lξZΨ[΅5xΩqMQ)Y€_>Hυ ρŠε6—έ…υσŠ[k6ο‡[ΰL‘Z­Z‘DΌŽQ]\!¨H| © |wΓΌ¦H²*sΖ?ΜkISό _ S„Q ?gJΗzλ^¨U΅Œx‡¨Π·&­ήŽΐΙ‹ΠΉuτ³0Ην{α?q>*”ώF h*τΐ“g/αδŽv]Q²h=q½1Ζ£?ΜΥΠΗ$Υ,ώ3[χ©+Κ)RVQ &1ΫφqG’―ΎΒκΩ~R@g‚χtiΣΘ;…‰qΥΑωŸ‘OPΏH ύΔΨ–άƒά'ŠΌ »xΣ€N©Xc κnΕ`θ!΄bύn‘ΗΛ•#‹a@dΜZΌ'·Ν”ŸρύΓDψW_}ƒV#ψ"θ5FOΒnν}b–Mmψ Wn’T±‚π;[ŒŽWΝς•Ζ†Ζ>§'Δ‹Wo$ωE©^ϊB€Lγ+ΧοH‚“EΡΰπ₯X°bͺ”/—AέQ΄P^™0%7WlΨƒFu*γςυ;(_ͺˆΘλiςZϊB%ρΝΖ'tž=%Ν2ΜCTin ‹vε5؜ΕΒ9½¦XΰΠ|‡˜Η`±o 2KΈ/)]Ό`βRG3&γ«I—αθΧ£μ¬Ϊf¦1‹+–)*녍9<_|.(Η©|at΄(ΤT ¨ˆZ ˆ_μΐ ―ΗΗ‘Iήπ:άΆ3ΊΆmψI7Ε0―©Έ|ν6ΦΝύG ~-‘!πWk‚KvχΦ―Y^΄έ΅Έπ)wŽ%ažΒP‘?(wΰ»3Ψ±lœJJιοφώ«‘υgn9RΜ)Ԉ>N5ήq^φ‘Ιξ&6)‘V΅BqPΊ€vUωW'θΣμ–Œ°ΓΗΟ£ο°1’ίήΊim³b%θ[§ƒc±¬mo7tnΥΐ¨hΚApέψύχί1Β7\$"ηψ©}©>–΄ύε¬*7λΗώ]ΥOh冽R, .,’KΧξ ƒ΅†υλ,Ÿ³>Vφ"uk”GX ƒNPIάΣ`χ©R¦0A)φˆe›Ε σžƒ€>°O;ƒ'?ΜJ 'ψ9[£yƒj‰H5{έ!  Ί»₯jB zD>ώF θ"z½iΣ€‚σΐξ¨]΅Œ /zΠ 5jB~. !M‚U“ΐασακΝ»"ρ\³riU ΡΗ²ϊρ³—1ΩJ|“»žΐp―©¨V±$Βƒ†Ι,ΙLπθΦFH‘”^2ΛΧνB†―Σ‘yΓjF ::%ΡOC{7XunŽ‘6έObΜΤΕxωϊG8ΨtBGσzςsΧΡ3Ήε€Δnέ}(g•A}Ϊ«&¬DΏŠτ€*€θγ>ͺY$ ¨ΩNoaΆΡ·[KΖ>{ρFL_ΈΕ ηCΡΒyεpJ‰ΚcώŒ·HšŽJ `ΦΣEQMΘψ΅v–ͺ&“FγΊ+·I‡Vλf΅Ρ VΕΈΊϊŠψF€xό U$NXτ¦'Χ»³ωpτ “Ϊ—―~D₯²E₯ψA™’rHH“:•FΆ˜εΡxωκ šΦ«*E^IaW -}­)ΕΣΧύό’Ωμ9t v#CQ«J΄jZΣΐ’Œ‰…½―4ΣΠ[Š…3σž."β3’²dNο(ΏΨ΅Ε—\^ύnF`ΣΞ#ςaBσΫκε #e‘γΜ…kX?ο”ςsz’)Β„wΚ₯π¬ΤΠbJ£QͺyJΐPΤ―UΑπΥ\B±οΘit4«‡ά9³ˆίX™…0ή{μIT˜6dΣo’Μβ΄iR ;πρ“"}Εb—V%C¨aνŠΠό†μ{·–χ–σε4mΤθ?A€I³VbνάΓΩ‚β~bΑŠ­’—"Dk¨V±„ψq8i!Xpe!•—*¦Œ€*€˜ςέScOpP‰”d=ΦΝύ£«B‹ΞΆ^’οž;gVdHŸ–›ŒΥŸ>%Κ›°[C…Ύ ΜU³ΊbR¨\+τŒα¦tηώγ:j2Ύ)˜Τζάsθ$ΊΆmλξfŸ€>o­fΤZΡΧ:Q³Ρ/SηFΚƒZΛ§•‰²γκΰwgΕγƒμ@&>'‡D½χξφΎ…Eφˆv0Ώb+Θ*‘o„wΧΟΪQ¬1ύάΛ/™ “m¬\‘%Sψ»XΓrp€Hѝɭ]NHς»N΅²˜<{5f,\…Sݍ:Β™Θ›»wδ±θτmΫARα~‚¬1ΰΉnT(L U1΅;¦Ζ›ΰ`wΕη:(lƒ‘)C:Œρθo˜Γ–έGA3t ΅ι„―Ύ‚ΘeΝ]ΎEL-³fΐΰη«ψΟότσ/δ6QΜJ[6¬‹WoΤτώ–­ESΣ=hΈΨΙΥδx¨ηdςY[7l?ŒŒιΣ‘mσΪ(RP±Ζτ°Šψώθ6ΐeK–’§Ζ2Μ‘-ζ„ΊθaŠj_€ΐΟΏΌ…YOgQ¦ΨΎ,ΔθΌ1kρFŒ›ΆLrSΉrdAσξ#P‘τ7Β`η{eˆuεςΨ«_[T$nρVWKΔΠύέοοθ (πEΣ¬›£Π =†Z’£y}C‡.iΛԊœόe֜ϋ IDATQ"†Π€§ΞηΦ=Η°cίχ’ΣmΦ¨¦°„x0­Ϋn°PJG θj4Gj6ίΌσ@ #,†ΡHέ{ά\ιξγΖC…B@! κwŠVMj‰ΩqΤ𠝇•χbΟΚ Hž<)Fψ†K7£XαΌυ(rY£'-ΔΚ {Δ+€] …€i @V˜–΄ώ܈ŸΉ&/½{ΜKω9vωŽŸ±KΒG‘l‰&ΙlΠ©ΨԁύΠͺi­Ώϋ2MςζΚf@©Q Lz—R<Χ|Bf@Ν*₯Eκhϋήο‘#{&a|hZξτ„H•2ΉΌ3TθJΨΠ‡¬N΅rP₯Εζ]G₯ρbF°£x 1)nγ8Vώ™ŒUώ;cNvέ‹TKƒηΟ·oί‰΄ΡΞ'ΐάΗQ›nt0M5…€™A,tT-_BžZPΧ¬‡ J/(ͺ,„πsΝκWCΥ ΕεAΣ¨ œαςκWq†€*€ΔΤκB‰š>|ό σ'Ή «7ν“Ξ Ω•“4IΈ ΄M§ΨΉΗΓ‰CΏN [ΨΌH/’_LΜT˜&‡Žƒ΅γΨO:²4ΣJ&=ϋtm ^ZX8‹aa…EeσJ_‘|Ή³›ζΔΥ¨ h!Πu€Κ•,ΧΑ†Ο_ΊvϋBηΦ ΰ:Έ‡§SΖbτHay¬ί~K#w"x”Ϊυq‡]―ΆΒ8S‘P聏Ÿaɚά·ƒQr’žsiΣ€ϊ€«W“ΠΤ’τ’!.;½›7¨f0FεώΤ;$Bš/θ[§Βτ ‘yO”,Z4;ΦΨ?4AΞ™=NŸΏ†fυΠ«s3΄ολΖu«|"Ρkz³V#ώ+hzξ0kfϋ•τΰy³D‘ό˜0Δπ«d›³Κβλύ‡OΑη‰mΟΦ"}£B?hžsŽύ»HžςΠΟ½ύ·3αί9ŸέΪ5Βχ§.ΒsX/”*VP6‡xLΒΑcη„!F‰nzȘ7 JqG 6Mœ»xCΨ₯,–¨P$TT$‘ή5.έ!°λΰ ,^½Γ`Fη9v6Nž½‚΅sΐ.Ώ₯kwaςœU(]¬Θ±«i½*b\0qtγΥ¬\”1aW`Τϋe‘„ r¦‰ΐ—o’£Ν(CχgAv OίόψΦFˆΤEΠ”Ε˜·| Ϊ4«&υͺˆεͺϋDώζΫκεLsςjΤ …ΐ?"ΐwΐPΟIπp°„yγZΈvσžh63AΑ ,°σί©ρ}CίΊjχn˜ˆ”)’γ΅Τ ΣG€¬±Nζυ`ί»Ρdϊ;ΓΉ‹Χ±sy(˜ζ=ι€FζLιρσ/Ώ‚L²MΩ)^©lQΕ0α₯πβεψOœ/2«ωσδ@o‡@Όzύ(‘Ei5Κ`MXƒΘ-$Ι₯yEάΌσ>γζβΔΩΛ(ώM~ ·νŒ*ε‹›0jθ<[€N•Β}BΒηE"2"@ΨΤυίΈγ0rfΛ σ&5Q·Fyω,ŸL|²¦B_μ=| ΛΦξF€«ψ„¨HœPŽύ»“Hižώαͺœ)*•)&*7ξ<@[΄lT]š­θiΊqA ‘a‚ˆM_°NΌeΨLAωέά9²ΐίΕZΙη%Ξε”ΰg­ ώ©κ R‹Ή‰dP–„Yšζ*φςՏRΩ}θ6̍γg/£ίˆ`T.W =;4ΕΩ‹Χ±vλAιΰδ>=3$Θυ„Sb™ ;τR₯LΓvξ?!6dC1gΙ&°0ΒΓΩ=LbΆνν.LrjA©LnͺP(τ‹ ύ',;ΆϋΣγƒEςόy²ƒ’‰|―¬šεk€ΩΛη;ΉΗyΩ UύΪ­ϋ’σ]¦ψIύ"¦f¦HΌ tύC/ͺ€‘ζA&YΝ*eΠΡΪ•ΚΓd!²Ω°γ0h€Z΄PΜ_Ή ληV>!:ZBτ‹ͺV‘„˜‘kΑίV½FŠΔ=>~.έώ,–θΥo~ό3mΐ¬')˜[ωΞιh=°ΦΒΒ ]Ϊ4ω]Ν„¬R6Ω±Βζ+J1;ω†£o·–†‚*χWoάC|9U\GkBM%q"ΐ#ϊHC– ΣG€ΚτkΩ°:₯M-ϋˆœΩ3cJΐP™¦›v…WΘ‘hfg8ŸΫφωfΗ(™doΥVωUšώrP3HΔΠ,rλthYο/Ο|^π‘*U ρŒαωƒΑη@ӎ(ρM~ΜηdT$g1½F₯Rfkμ_˜ˆ‘VSO ¨HΉj‰ROY5/7'²g͈·Θ „Ζ„Σζ―ΓΔY+…΅cWλ⣑!ε ώ*”‰eΒ_SΦΡ/S²kgυ,_ΡVT&–¦»œ¨ΝO]f,Ω½}ωΪΜ]Ύύz˜Kχ%₯nΪτvΓ­»€0V½bIΓdΉf¨Ω2ΚNΜK©χOVΠ·ΥΛ"Wφ,Ψyΰ„lRhŠΜξ. …€ώΰ‘€Y7GtnΥNφέd‚‡Ώ?/EΣ6Νλ₯έ]4-€¬Ÿ#QΓ+8MλWA­*eτŽš‘B@!π dOž³‡ΦMΑΧQ4ίΩΩ=Τs2]ϋ‘UΣZιM[ΛΦ°λΥF!©3Ύ?}IΌζΘ0/_ͺˆtυ³)kϋ#3dΚ(ςίΘώθ`VWg(¨ιhPןΝ•Λ†Π­;±ψOVωš9~ψ _‘]_RC°lΊ—ώέΙ ˜Ά`ŽžψAδτ4ί…¬B@! \GΟOuσFMφτω«θfηk`§+?[ύ―Sš‘*€˜έRcM4ότσ―¨ΪΒ#μΊΒͺ󇽏Ÿ‘……³˜[ΊϋGΛΔ’r(“f­BίξfȜρλDƒ£)L”χ&bιfPfΰΧ·Ώ‘}Λo…NΞ"ο·GΠl”.^P€ZβΙ³—BE₯DΑ’)ξ²ωΰF£_Vbύ‡lΕ«7?‘}w4©WU:όT(ϊD€ΧΐRH%σλΨ© Hš4©HVπyO6™e+gϊ’XαΌv<)’'γ}’Iέ*ϊGΝJ! 0B@“Ξ<Ό>Μΐέ[I'šφνώΗΡσάΐ³ΘΛWo€‘ŠUΪώ`γŽ#α&zl΄βό₯Β(γήbΕ $Mšδ―£> Pθ²?._»ω“άŒ&ΤwΨ\Ίv[‹©Ίς³ΥΗύΦΛ,TD/wRΝCWPή¨AGΤͺR#mdnνπΛΫίΰβ? ”Σ[ζ)f§τ$£χεpθS¦…ζ-ΒNo~N…~£Pθ3 “pœ—Α“vςμΥ ¬U›-Τ¨όlur―υ4 UΡΣέTsя9‚κ•J‘jωβΨ΄λ(H)tέΫ5‚΅γΨebΙN.Υ™cΊKδρΣrΘ8vκ"Ύ)”Gτ»Y̊άr€ RΪ†fcZœ½x,€|lŽlΊ¨‘+„;ΉY ΅μΨΙ“'½χ€‰ °sy¨C΅ˆXΆΑaK !'Ξ^Ζ¨±s€˜Ι―ξνΓΎw;$O¦ !„Ήϊw…€©!pςά1Eg’³PΎ\ΰ~‘uΣΪνj#Ν6m{»‰έ‚ΙΖ]]'ΰόεβ’:U S›Άοg £ψη_~Eϊti₯QŠέ VY”ΙjVΏš$·T$>Ξ]Ό!MwΝλI£UΞl™κύ!D/sKT)_άθη‰)5c…@βE`ΧΑπ3ysgΙ]6έ°lƒ}ΪEΫΟ–ΝZΙ’&AšΤίU(bU‰MtΥw+Ύκ»oΪyοίΏ—D7¦τώΈxυV΄M,Iaζ·ΏekΝ»ίǝ{δ οMBψuJ₯Ρd¬\©Β˜θ;ΨhH4³|ρς VΟρSέά αf©1(βκύΟYΊ ΦN6<ξ=x"Zή-TsBVh€Jx·!=ΐηŠίψωΘ—';‚=?’21–γΟίότ3ʐδHSƒP(b2ΗXάΰίŸn-E*olΨIpQJ1S†t"sργOΏ`Υ,_ƒτΙ•λwΡ¦·Ά- –χEΚ”Ιaο:ALp{uj#cS_’PΔ?οίOXΪ>€Zέo{g$Η=h{k€M §Έγέοοα8λ·’Δ磧/ΠͺI-Έ퉔)’Η€Τ’e[tw9Κ&2NΏ ΛA°Ά0Γ >ν£ύ]κƒϊCΰꍻ8ύΓ5)zRΦfɚRϋνέ;pŸ`caŽ‘6υ7q5#…€Bΰ‹ˆŸ-e9ι55¬_gτξΪβ‹©~Y!T$:(©Ο(Ρ1±όί€ΞΆ^2ςeΣΌpΰ»³ζ5Y’YK§RIοvOλpHAΪ‘MSuσž.(Z8ΒΔH½iWG4ͺS ŽΊuiΡ™EES<{A₯Š°+σ8œ%0oQΌ.]Ύz׍‰;ˆίRE ¨HΑ<Άΐ΅„ο1ΞΫ~mV|šUΟ—)n?Τp €€3Ύα–k‡Οˆι:uζΌM”ήan~Τkγ£|yrhϊ˜>2Ιχb…σιvνWΪ΄κΕ²₯h”ξΜ— QΉ±€idj›š:¦ΟΗ’Υ›]=BΌ‡NΥ¦―ψ˜ »n›ή‘‹4=°M˜˜Η™]c ¦QžœΩάX“ΠHέf‡ΧΐQαϊlοe}ς2}Ě7¬’=ίK݁1zxθχκg›¨DΥjαm㆏λ{Χ8LΏ±“§ΟiΩ¬α]ˆf7ϋγe€―ιCŸ1η_€ˆση˜.π{M,§ΜΧΌΈ΅bΞH[ςΔΤσmή5@Ύ½ΫΘ£ξ[!Όϋ DΖ}¬αγηΊvέy|HΨB͎]­«§έοό;8@ΐ”:176b¦ϊΪΔ†Iž›ΰ₯‹΄Ρ…Ν[‘ρα ΅(<ΐώ»ω}ΣΡ+P»??€fύ‡νΠ·k3ΥͺRή„€ΏˆZ―7ΏW»¦?­Β¬Φ΄ν-ζΫ«υ=±fΞ_₯±Sη«q·τιž46εe…€>‘/WΡByυάΣΕ! €@R όΊŸνͺυ;ΤΫ’Lχw5M7c1‹/Ίω„($ »ͺΎρ’vο;$ί13tδψ76ΌaU»Θ3]ΪG“jθ\Ηa$@6‘„γžΏlbyβτ9Υkέ_ν›ΤΌΠά°jΪyˆ­Ϋ;͟%ξωqEmz†˜έA/<[R†χΌKΓ4N75»M9s˜]CO/δjŒιζt„€#Lٚ˜₯λm­έƒGOΪϊήΛg°Ώ3ήnα­†oΏ.Ÿ-m즜ž)‡1iΔ‡zσΥηdΆ·{L±MP›Τ!ΉξΘA!π :­ϊΫR½:yό‹Yνiήc˜¦θƒ>l%³›δζ·τٞΆk™R…ρD@7ˆSΤ’xm]6Ρ₯`ϊ˜6l7Θξς–IΊΆπjzΆ΄}l‡ΟQ|98Έ‹λyζσΚ©oΞ»J»1+‘  ‰‡ šL¦|άΤXMέΫ6¦[Όj³Œ Σ¬ώ*χ\©Τ c}ΓBζΘ”Ιηη©o½δΊΒ·η/©F³>z·~eυοήB¦œE³z―quϋ¦’d±όκΊ+.œpJ’[ΐ”Ή‰ίΌΣn-―S­‚ύπa~o˜•X+#G+KζΗνΝΝΝ*½Τ£}#׊πωKΦ)"z₯VGIξ0Έ>š;JΉsdUsΟ‘ϊβΰΧjγQSY³d֎=4iΔτqο—Ρ»‰€Ω)φFΓξκΡώuh^ϋ¨c–mP@Π,ΠI΅«Όbϋ-X±Ρξ8δ@η LšΉXαQ)}Ί΄Ίšp]jU΄I sCΓ£“ΏϊwoͺYˆ™1«<-V·70’―SπτX͝0ΠUjΟωbDˆ €Ώ0‹-MΩvS½€D‘όvWΉG½JΆΔζ†­{διl{ΩΦxλeΧΣΞΏh#αcϋκόΏ―¨—ί›ό˜?ΥΧ.Κΰ@ΰ~$@ξ'ΔΏ#Κήλ>\Χk¬_W+”7•GΓπŽ€i*vόΤ· πj'Sφfβˆ”+{V͘ΏRc§ΔΘ«KS[ξjٚ­κ7|š†χο¨ϊ5^s]ςΔιo΅hεf[’ΒKΟθ΅—~ZρɁΞ0Ώ3*7ιe“€fΫωη_ΥΉ —ΥΈΞ›6PsσΣάΔ θaΛαp €€{ ˜ΔΗώƒΗ”+GV)˜ΗίΊη]Ί|MqF₯λΆμVχ!ŠžUΤ>ΞΤύΞ”1ƒ†z·w/4’E@ί˜ώ€!a ΄jήe{2³‚f+~σ.[–ϋ‘Gάυx“πHŸ>­.^Ί¦ΪοyΫςšζsΛά lSuώH€―.pωJ‚Β’VΨέ¦œ‡{ άΉ δίMΥί,§ι‘Λ5!b‘2eΚ`ί4΄hTΥΦωΏqσΥjΩWO—,¬‰Γ?pamήρΉz ΅_-WFŸν=¨ά9²ΩRZ¦ό8Kΐl37?'ή­_IŸΤœk΄2r”+ΘαγηΚ41Œ§tι:+x’AIΐ¬Κ¬€—†x΅Σ;΅+ήυ\³ΓtΫΞ/lMo³ΚΣτ;zόŒ+‘ϊ?]ˆ#€ €€£L?Ϋ―Žžtν υ1]§ΞœΧμρ>χŒΣτ*έ²cŸύlrϊμ•*V@7n~―tiΣςΉΔQ―Ž λΙΩ@)ΠφΓ‘ΆώφΜΰ~ρόζœLϘb…ςΉ“Ž_¨ˆ¨΄tΦpΜ—Λ>Φτ©ΧΖΗξ ΫΧφ–1 ΗΜM ³ 4,Π+EΖΜ @ΰΑ\Ό|M;VΝ·^Φ‡cΕΗΫT‘\[fσ™—ŸΆφ„&A’ΫP[ͺ"thΏ~ž‰ €Ž0ο#b–Wω=­ƒGOκƍον½Sy’iη!v1fμ4{Α”ψΥΫήΏx»RyΝ_Ί^ΡKΦiΚ¨^zΎLqΗ[ΰŸ ςη­x$ ΰh³:sτΔ(Mτ²ΝJΝaϋ}ΤσTέjμ*‹_Q‹γ54xŽΆ.hߌ˜7"_:‘,™WΑ|?Χΰ4ε.™aΟ›'η½0ŽΖ%8"°ν³/΄}Χ~ϋώέz•]?+a €ΐ0;@:υ«€ΎνU«Jy{Ζ‘‘Ϊςι>{ƒ@ψ#―ŽžRό杢―Xjμ}„Ε«6kΐΘ0Ν ι―rΟ•²OΏ“ MΥ*–³ί ˆΣΖm{΅`Ί?ΘΈH€πb@Έ§ΐ7fΕεΛΊ—˜˜¨ϊm*S†τš?ΥWΗNž•§O°ύΣ¦ΟΘ°~”1Cytς³ί‹™κΗ.^k €Έ€I–¦œ|R%ŠΠ²5[δΩΆ‘:Ά¨γΡ" €QΜ4?•*VΐΖlvˆ΄μ6L§Ξœ·ΫSΗϊvu„A € € € €@ \Ύ’ °¨Κ•=«M€,]³Eqαv·~Ό­Ξ}ΗκςΥ[ϊ*Mš4I;8–βH€€ψ)b€ €@ς ˜~g,VΒυΆ™iΝJ/k¨wΫX½zΣ>jZΏ²|z΄Όk 5šyιόΏ/kŜ‘Κ“λŸ2MΣ‹Κ«œΩŸLώ€ € € €€:#ΗN«Ιϋ~ άE•_ΑŽ“ν{Υ₯_ΦΕ)WŽ¬©.&όpH€<\_Ύ8Fΐ¬ͺ0o4L“s“Π0ǜk4rΒ<­‰TΎάΩ]±ξ;ψ΅ήνδ―.­κ«[»†Ίšp]5›{©ςk/h¨w{ǘ € € €$­ΐϊ­»mSτΧ_zVoVx^¦LχΩs΅2r4½G“v*RΕΥH€€Šib €@Κ39Ϊ–ΏΪΆ|]|ί+P‡ΦŠ9£”)cz ©Έ6Ϊ―Y‘2η’Q!€ € € Z.\Ό’ΘΈ΅}Χ~»PΣ«kS+”7΅ Ÿq&‘ $ΔζR €€Σ6nΫ«ύƒ:¬§*Ώφ/ή”ΩKeϊ†ŒΨIuͺΎͺ#ΗΏQΓvΥ΅uunUΟE`˜ ‘^š¨`Ύ\N£!@@@@d ’Μΐε@Τ.0+v΅&D,³₯‹(Q‰Ϊ±ϋ€ž/S\‘ΪΠΜn£ΗΏΡς9#•1CzWΈc&E+zΙ:­˜;RΉsdKν Œ@@@@ … IaΒp@Τ(`z|:zJ·ε’§ψΪ„ˆΩŠΪΎΧhۜμνΚε]‘?υ­κ·ρΡϋ-λͺk›φϋωO’^ϋNOfy"50f@@@@&@$…MΓAR³ΐΩσ΅~Λn5kPΕ†ΡΛo’}ZKfSš4i\‘yϊλΐ‘vWˆι²`ωFO_ KW)OΞlκΩ‘±κV―š); € € € €@2 Iζ ΰς €€“ΜξτιΣiςΘ]aΫΌKέŽΧθAU»Κ+Š]ΎA~3ΥͺI ΅j\]_|uLώcgͺ{»Fς¨WΙυΌc'Ο*ήJϋθ£N@@@@$@δAr@ί ˜Xέ|‚Υ¨Φ›ͺSυ8rR“£U’H~Ϋ#δΜΉ‹ͺσ^?U|₯¬‚ό»ΉN0oQΌfΜ_©΅ΡΪ»ˆΜή‘}ΖhFP?•)Uj@@@@ξ+@δΎD<@ΰο=qF‘ ΧκςΥ{šUλwΈz„˜Fθ‘qk΅:*PΉrdu]Ζμρ œ‘OW‡ήcl/‘Š―qF Ϊ”gΫΆGΘΩsUΈ@n]MΈ'Λ€GωΉ±Ί#± @@@@ΰ/ ωΛt<@ΰο ψ›₯=ϋiNθ=ρx&™FηζΟμΩ²ΨSwφ«£ΗΟhωμΆ™ϊ­[?ͺΗ P}²}―,£Χ~SžmΪΏs € € € € πK Ό@’Mΰ»λ7Υ}@ˆLŸ²OΥφϋυL©"ŠςΦ†­{δι¬ΰ!έT­b9;ΖΨεδ8Ss' Πc™2jΜΕ:τυ)Ν›8HOfy"ΩβΰΒ € € € €)O€HΚ›F„Έ•@bb’Ά|ΊO»>JΉsdSj―*]ΊtͺίΖΗ6FŸΤΟε±`ωFΝV|μ8»KΔ<·Ν#U¦daυυlζVn‹ € € € πΗ$@x… €€8u[v۝!Q“©μΣΕ\γ»ωύjΦeˆ2dH―°@/[.kΡΚM:pψ„ϊwo‘ββ`@ € € € €Ι'@$ωμΉ2 €ΐ=ΆοΪ―φ½F«OηwΥ’QUνΪwHE δ±;BΞ]Έ¬Φ=‡+k–Μ η­LΣγˆ € € € €ΐoH€π’@H‘¦ΡyΜ ξΣQύ†MUΌ9]»<Ύ=IόT³ΛμόH‘³Η @@@@δ ’όsΐ@ξ#χΡ' [¨9‘>*˜/—}tΰ”ωŠί΄S+#Gγ‡ € € € €ΐoH€π’@H“f.Φ¬ΨΥς¨[I%‹ζΧΈi1ͺPξ λΧ!UŒŸA"€ € € €$­ €υζj €Cΰΰ‘“š³`mz^ΊxAωτh‘Η2eόgδ© € € € €N βΤ™%.@@@@@άX€ˆO>‘#€ € € € € ΰT NYβB@@@@@ΐH€Έρδ: € € € € €N βΤ™%.@@@@@άX€ˆO>‘#€ € € € € ΰT NYβB@@@@@ΐH€Έρδ: € € € € €N βΤ™%.@@@@@άX€ˆO>‘#€ € € € € ΰT NYβB@@@@@ΐH€Έρδ: € € € € €N βΤ™%.@@@@@άX€ˆO>‘#€ € € € € ΰT NYβB@@@@@ΐH€Έρδ: € € € € €Nψ/Ε|Ίa” "IENDB`‚pytorch/docs/source/_static/000077500000000000000000000000001444262304300164265ustar00rootroot00000000000000pytorch/docs/source/_static/js/000077500000000000000000000000001444262304300170425ustar00rootroot00000000000000pytorch/docs/source/_static/js/version_alert.js000066400000000000000000000013551444262304300222600ustar00rootroot00000000000000function warnOnLatestVersion() { if (!window.READTHEDOCS_DATA || window.READTHEDOCS_DATA.version !== "latest") { return; // not on ReadTheDocs and not latest. } var note = document.createElement('div'); note.setAttribute('class', 'admonition note'); note.innerHTML = "

Note

" + "

" + "This documentation is for an unreleased development version. " + "Click here to access the documentation of the current stable release." + "

"; var parent = document.querySelector('#pyg-documentation'); if (parent) parent.insertBefore(note, parent.querySelector('h1')); } document.addEventListener('DOMContentLoaded', warnOnLatestVersion); pytorch/docs/source/_templates/000077500000000000000000000000001444262304300171355ustar00rootroot00000000000000pytorch/docs/source/_templates/autosummary/000077500000000000000000000000001444262304300215235ustar00rootroot00000000000000pytorch/docs/source/_templates/autosummary/class.rst000066400000000000000000000002051444262304300233570ustar00rootroot00000000000000{{ fullname | escape | underline}} .. currentmodule:: {{ module }} .. autoclass:: {{ objname }} :show-inheritance: :members: pytorch/docs/source/_templates/autosummary/inherited_class.rst000066400000000000000000000003061444262304300254140ustar00rootroot00000000000000{{ fullname | escape | underline}} .. currentmodule:: {{ module }} .. autoclass:: {{ objname }} :show-inheritance: :members: :inherited-members: :special-members: __cat_dim__, __inc__ pytorch/docs/source/_templates/autosummary/nn.rst000066400000000000000000000006601444262304300226720ustar00rootroot00000000000000{{ fullname | escape | underline}} .. currentmodule:: {{ module }} {% if objname != "MessagePassing" %} .. autoclass:: {{ objname }} :show-inheritance: :members: :exclude-members: forward, reset_parameters, message, message_and_aggregate, edge_update, aggregate, update .. automethod:: forward .. automethod:: reset_parameters {% else %} .. autoclass:: {{ objname }} :show-inheritance: :members: {% endif %} pytorch/docs/source/_templates/autosummary/only_class.rst000066400000000000000000000001701444262304300244210ustar00rootroot00000000000000{{ fullname | escape | underline}} .. currentmodule:: {{ module }} .. autoclass:: {{ objname }} :show-inheritance: pytorch/docs/source/advanced/000077500000000000000000000000001444262304300165455ustar00rootroot00000000000000pytorch/docs/source/advanced/batching.rst000066400000000000000000000271201444262304300210600ustar00rootroot00000000000000Advanced Mini-Batching ====================== The creation of mini-batching is crucial for letting the training of a deep learning model scale to huge amounts of data. Instead of processing examples one-by-one, a mini-batch groups a set of examples into a unified representation where it can efficiently be processed in parallel. In the image or language domain, this procedure is typically achieved by rescaling or padding each example into a set to equally-sized shapes, and examples are then grouped in an additional dimension. The length of this dimension is then equal to the number of examples grouped in a mini-batch and is typically referred to as the :obj:`batch_size`. Since graphs are one of the most general data structures that can hold *any* number of nodes or edges, the two approaches described above are either not feasible or may result in a lot of unnecessary memory consumption. In :pyg:`PyG`, we opt for another approach to achieve parallelization across a number of examples. Here, adjacency matrices are stacked in a diagonal fashion (creating a giant graph that holds multiple isolated subgraphs), and node and target features are simply concatenated in the node dimension, *i.e.* .. math:: \mathbf{A} = \begin{bmatrix} \mathbf{A}_1 & & \\ & \ddots & \\ & & \mathbf{A}_n \end{bmatrix}, \qquad \mathbf{X} = \begin{bmatrix} \mathbf{X}_1 \\ \vdots \\ \mathbf{X}_n \end{bmatrix}, \qquad \mathbf{Y} = \begin{bmatrix} \mathbf{Y}_1 \\ \vdots \\ \mathbf{Y}_n \end{bmatrix}. This procedure has some crucial advantages over other batching procedures: 1. GNN operators that rely on a message passing scheme do not need to be modified since messages still cannot be exchanged between two nodes that belong to different graphs. 2. There is no computational or memory overhead. For example, this batching procedure works completely without any padding of node or edge features. Note that there is no additional memory overhead for adjacency matrices since they are saved in a sparse fashion holding only non-zero entries, *i.e.*, the edges. :pyg:`PyG` automatically takes care of batching multiple graphs into a single giant graph with the help of the :class:`torch_geometric.loader.DataLoader` class. Internally, :class:`~torch_geometric.loader.DataLoader` is just a regular :pytorch:`PyTorch` :class:`torch.utils.data.DataLoader` that overwrites its :func:`collate` functionality, *i.e.*, the definition of how a list of examples should be grouped together. Therefore, all arguments that can be passed to a :pytorch:`PyTorch` :class:`~torch.utils.data.DataLoader` can also be passed to a :pyg:`PyG` :class:`~torch_geometric.loader.DataLoader`, *e.g.*, the number of workers :obj:`num_workers`. In its most general form, the :pyg:`PyG` :class:`~torch_geometric.loader.DataLoader` will automatically increment the :obj:`edge_index` tensor by the cumulated number of nodes of all graphs that got collated before the currently processed graph, and will concatenate :obj:`edge_index` tensors (that are of shape :obj:`[2, num_edges]`) in the second dimension. The same is true for :obj:`face` tensors, *i.e.*, face indices in meshes. All other tensors will just get concatenated in the first dimension without any further increasement of their values. However, there are a few special use-cases (as outlined below) where the user actively wants to modify this behavior to its own needs. :pyg:`PyG` allows modification to the underlying batching procedure by overwriting the :func:`torch_geometric.data.Data.__inc__` and :func:`torch_geometric.data.Data.__cat_dim__` functionalities. Without any modifications, these are defined as follows in the :class:`~torch_geometric.data.Data` class: .. code-block:: python def __inc__(self, key, value, *args, **kwargs): if 'index' in key or 'face' in key: return self.num_nodes else: return 0 def __cat_dim__(self, key, value, *args, **kwargs): if 'index' in key or 'face' in key: return 1 else: return 0 We can see that :meth:`__inc__` defines the incremental count between two consecutive graph attributes, where as :meth:`__cat_dim__` defines in which dimension graph tensors of the same attribute should be concatenated together. Both functions are called for each attribute stored in the :class:`~torch_geometric.data.Data` class, and get passed their specific :obj:`key` and value :obj:`item` as arguments. In what follows, we present a few use-cases where the modification of :func:`__inc__` and :func:`__cat_dim__` might be absolutely necessary. Pairs of Graphs --------------- In case you want to store multiple graphs in a single :class:`~torch_geometric.data.Data` object, *e.g.*, for applications such as graph matching, you need to ensure correct batching behavior across all those graphs. For example, consider storing two graphs, a source graph :math:`\mathcal{G}_s` and a target graph :math:`\mathcal{G}_t` in a :class:`~torch_geometric.data.Data`, *e.g.*: .. code-block:: python from torch_geometric.data import Data class PairData(Data): pass data = PairData(x_s=x_s, edge_index_s=edge_index_s, # Source graph. x_t=x_t, edge_index_t=edge_index_t) # Target graph. In this case, :obj:`edge_index_s` should be increased by the number of nodes in the source graph :math:`\mathcal{G}_s`, *e.g.*, :obj:`x_s.size(0)`, and :obj:`edge_index_t` should be increased by the number of nodes in the target graph :math:`\mathcal{G}_t`, *e.g.*, :obj:`x_t.size(0)`: .. code-block:: python class PairData(Data): def __inc__(self, key, value, *args, **kwargs): if key == 'edge_index_s': return self.x_s.size(0) if key == 'edge_index_t': return self.x_t.size(0) return super().__inc__(key, value, *args, **kwargs) We can test our :class:`PairData` batching behavior by setting up a simple test script: .. code-block:: python from torch_geometric.loader import DataLoader x_s = torch.randn(5, 16) # 5 nodes. edge_index_s = torch.tensor([ [0, 0, 0, 0], [1, 2, 3, 4], ]) x_t = torch.randn(4, 16) # 4 nodes. edge_index_t = torch.tensor([ [0, 0, 0], [1, 2, 3], ]) data = PairData(x_s=x_s, edge_index_s=edge_index_s, x_t=x_t, edge_index_t=edge_index_t) data_list = [data, data] loader = DataLoader(data_list, batch_size=2) batch = next(iter(loader)) print(batch) >>> PairDataBatch(x_s=[10, 16], edge_index_s=[2, 8], x_t=[8, 16], edge_index_t=[2, 6]) print(batch.edge_index_s) >>> tensor([[0, 0, 0, 0, 5, 5, 5, 5], [1, 2, 3, 4, 6, 7, 8, 9]]) print(batch.edge_index_t) >>> tensor([[0, 0, 0, 4, 4, 4], [1, 2, 3, 5, 6, 7]]) Everything looks good so far! :obj:`edge_index_s` and :obj:`edge_index_t` get correctly batched together, even when using a different numbers of nodes for :math:`\mathcal{G}_s` and :math:`\mathcal{G}_t`. However, the :obj:`batch` attribute (that maps each node to its respective graph) is missing since :pyg:`PyG` fails to identify the actual graph in the :class:`PairData` object. That is where the :obj:`follow_batch` argument of the :class:`~torch_geometric.loader.DataLoader` comes into play. Here, we can specify for which attributes we want to maintain the batch information: .. code-block:: python loader = DataLoader(data_list, batch_size=2, follow_batch=['x_s', 'x_t']) batch = next(iter(loader)) print(batch) >>> PairDataBatch(x_s=[10, 16], edge_index_s=[2, 8], x_s_batch=[10], x_t=[8, 16], edge_index_t=[2, 6], x_t_batch=[8]) print(batch.x_s_batch) >>> tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1]) print(batch.x_t_batch) >>> tensor([0, 0, 0, 0, 1, 1, 1, 1]) As one can see, :obj:`follow_batch=['x_s', 'x_t']` now successfully creates assignment vectors :obj:`x_s_batch` and :obj:`x_t_batch` for the node features :obj:`x_s` and :obj:`x_t`, respectively. That information can now be used to perform reduce operations, *e.g.*, global pooling, on multiple graphs in a single :class:`Batch` object. Bipartite Graphs ---------------- The adjacency matrix of a bipartite graph defines the relationship between nodes of two different node types. In general, the number of nodes for each node type do not need to match, resulting in a non-quadratic adjacency matrix of shape :math:`\mathbf{A} \in \{ 0, 1 \}^{N \times M}` with :math:`N \neq M` potentially. In a mini-batching procedure of bipartite graphs, the source nodes of edges in :obj:`edge_index` should get increased differently than the target nodes of edges in :obj:`edge_index`. To achieve this, consider a bipartite graph between two node types with corresponding node features :obj:`x_s` and :obj:`x_t`, respectively: .. code-block:: python from torch_geometric.data import Data class BipartiteData(Data): pass data = BipartiteData(x_s=x_s, x_t=x_t, edge_index=edge_index) For a correct mini-batching procedure in bipartite graphs, we need to tell :pyg:`PyG` that it should increment source and target nodes of edges in :obj:`edge_index` independently: .. code-block:: python class BipartiteData(Data): def __inc__(self, key, value, *args, **kwargs): if key == 'edge_index': return torch.tensor([[self.x_s.size(0)], [self.x_t.size(0)]]) return super().__inc__(key, value, *args, **kwargs) Here, :obj:`edge_index[0]` (the source nodes of edges) get incremented by :obj:`x_s.size(0)` while :obj:`edge_index[1]` (the target nodes of edges) get incremented by :obj:`x_t.size(0)`. We can again test our implementation by running a simple test script: .. code-block:: python from torch_geometric.loader import DataLoader x_s = torch.randn(2, 16) # 2 nodes. x_t = torch.randn(3, 16) # 3 nodes. edge_index = torch.tensor([ [0, 0, 1, 1], [0, 1, 1, 2], ]) data = BipartiteData(x_s=x_s, x_t=x_t, edge_index=edge_index) data_list = [data, data] loader = DataLoader(data_list, batch_size=2) batch = next(iter(loader)) print(batch) >>> BipartiteDataBatch(x_s=[4, 16], x_t=[6, 16], edge_index=[2, 8]) print(batch.edge_index) >>> tensor([[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 1, 2, 3, 4, 4, 5]]) Again, this is exactly the behavior we aimed for! Batching Along New Dimensions ----------------------------- Sometimes, attributes of :obj:`data` objects should be batched by gaining a new batch dimension (as in classical mini-batching), *e.g.*, for graph-level properties or targets. Specifically, a list of attributes of shape :obj:`[num_features]` should be returned as :obj:`[num_examples, num_features]` rather than :obj:`[num_examples * num_features]`. :pyg:`PyG` achieves this by returning a concatenation dimension of :obj:`None` in :meth:`~torch_geometric.data.Data.__cat_dim__`: .. code-block:: python from torch_geometric.data import Data from torch_geometric.loader import DataLoader class MyData(Data): def __cat_dim__(self, key, value, *args, **kwargs): if key == 'foo': return None return super().__cat_dim__(key, value, *args, **kwargs) edge_index = torch.tensor([ [0, 1, 1, 2], [1, 0, 2, 1], ]) foo = torch.randn(16) data = MyData(num_nodes=3, edge_index=edge_index, foo=foo) data_list = [data, data] loader = DataLoader(data_list, batch_size=2) batch = next(iter(loader)) print(batch) >>> MyDataBatch(num_nodes=6, edge_index=[2, 8], foo=[2, 16]) As desired, :obj:`batch.foo` is now described by two dimensions: The batch dimension and the feature dimension. pytorch/docs/source/advanced/cpu_affinity.rst000066400000000000000000000316061444262304300217650ustar00rootroot00000000000000CPU Affinity for PyG Workloads ============================== The performance of :pyg:`PyG` workloads using CPU can be significantly improved by setting a proper affinity mask. Processor affinity, or core binding, is a modification of the native OS queue scheduling algorithm that enables an application to assign a specific set of cores to processes or threads launched during its execution on the CPU. In consequence, it increases the overall effective hardware utilisation by minimizing core stalls and memory bounds. It also secures CPU resources to critical processes or threads, even if the system is under heavy load. CPU affinity targets the two main performance-critical regions: * **Execution bind:** Indicates a core where process/thread will run. * **Memory bind:** Indicates a preferred memory area where memory pages will be bound (local areas in NUMA machine). The following article discusses readily available tools and environment settings that one can use to maximize the performance of Intel CPUs with :pyg:`PyG`. .. note:: Overall, CPU affinity can be a useful tool for improving the performance and predictability of certain types of applications, but one configuration does not necessarily fit all cases: it is important to carefully consider whether CPU affinity is appropriate for your use case, and to test and measure the impact of any changes you make. Using CPU affinity ------------------ Each :pyg:`PyG` workload can be parallelized using the :pytorch:`PyTorch` iterator class :class:`MultiProcessingDataLoaderIter`, which is automatically enabled in case :obj:`num_workers > 0` is passed to a :class:`torch.utils.data.DataLoader`. Under the hood, it creates :obj:`num_workers` many sub-processes that will run in parallel to the main process. Setting a CPU affinity mask for the data loading processes places :class:`~torch.utils.data.DataLoader` worker threads on specific CPU cores. In effect, it allows for more efficient data batch preparation by allocating pre-fetched batches in local memory. Every time a process or thread moves from one core to another, registers and caches need to be flushed and reloaded. This can become very costly if it happens often, and threads may also no longer be close to their data, or be able to share data in a cache. Since :pyg:`PyG` (2.3 and beyond), :class:`~torch_geometric.loader.NodeLoader` and :class:`~torch_geometric.loader.LinkLoader` classes officially support a native solution for CPU affinity using the :class:`torch_geometric.loader.AffinityMixin` context manager. CPU affinity can be enabled via the :meth:`~torch_geometric.loader.AffinityMixin.enable_cpu_affinity` method for :obj:`num_workers > 0` use-cases, and will guarantee that a separate core is assigned to each worker at initialization. A user-defined list of core IDs may be assigned using the :attr:`loader_cores` argument. Otherwise, cores will be assigned automatically, starting at core ID 0. As of now, only a single core can be assigned to a worker, hence multi-threading is disabled in workers' processes by default. The recommended number of workers to start with lies between :obj:`[2, 4]`, and the optimum may vary based on workload characteristics: .. code-block:: python loader = NeigborLoader( data, num_workers=3, ..., ) with loader.enable_cpu_affinity(loader_cores=[0, 1, 2]): for batch in loader: pass It is generally adivisable to use :obj:`filter_per_worker=True` for any multi-process CPU workloads (:obj:`True` by default). The workers then prepare each mini-batch: first by sampling the node indices using pre-defined a sampler, and secondly filtering node and edge features according to sampled nodes and edges. The filtering function selects node feature vectors from the complete input :class:`~torch_geometric.data.Data` tensor loaded into DRAM. When :attr:`filter_per_worker` is set to :attr:`True`, each worker's subprocess performs the filtering within it's CPU resource. Hence, main process resources are relieved and can be secured only for GNN computation. Binding processes to physical cores ----------------------------------- Following general performance tuning principles, it is advisable to use only physical cores for deep learning workloads. For example, while two logical threads run :obj:`GEMM` at the same time, they will be sharing the same core resources causing front end bound, such that the overhead from this front end bound is greater than the gain from running both logical threads at the same time. This is because OpenMP threads will contend for the same :obj:`GEMM` execution units, see `here `__. The binding can be done in many ways, however the most common tools are: * :obj:`numactl` (only on Linux): .. code-block:: console --physcpubind=, -C or --cpunodebind=, -N * `Intel OMP `__ :obj:`libiomp`: .. code-block:: console export KMP_AFFINITY=granularity=fine,proclist=[0-],explicit * GNU :obj:`libgomp`: .. code-block:: console export GOMP_CPU_AFFINITY="0-" Isolating the :class:`~torch.utils.data.DataLoader` process ----------------------------------------------------------- For best performance, it is required combine main process affinity using the tools listed above, with the multi-process :class:`~torch.utils.data.DataLoader` affinity settings. In each parallelized :pyg:`PyG` workload execution, the main process performs message passing updates over GNN layers, while the :class:`~torch.utils.data.DataLoader` workers sub-processes take care of fetching and pre-processing data to be passed to a GNN model. It is advisable to isolate the CPU resources made available to these two processes to achieve the best results. To do this, CPUs assigned to each affinity mask should be mutually exclusive. For example, if four :class:`~torch.utils.data.DataLoader` workers are assigned to CPUs :obj:`[0, 1, 2, 3]`, the main process should use the rest of available cores, *i.e.* by calling: .. code-block:: console numactl -C 4-(N-1) --localalloc python … where :obj:`N` is the total number of physical cores, with the last CPU having core ID :obj:`N-1`. Adding :obj:`--localalloc` improves local memory allocation and keeps the cache closer to active cores. Dual socket CPU separation ~~~~~~~~~~~~~~~~~~~~~~~~~~ With dual-socket CPUs, it might be beneficial to further isolate the processes between the sockets. This leads to decreased frequency of remote memory calls for the main process. The goal is to `utilize high-speed cache on local memory and reduces memory bound caused by migrating cached data between NUMA nodes `__. This can be achieved by using :class:`~torch.utils.data.DataLoader` affinity, and launching main process on the cores of the second socket, *i.e.* with: .. code-block:: console numactl -C M-(N-1) -m 1 python … where :obj:`M` is the :obj:`cpuid` of the first core of the second CPU socket. Adding a complementary memory-allocation flag :obj:`-m 1` prioritizes cache allocation on the same NUMA node, where the main process is running (alternatively for less strict memory allocation use :obj:`--preferred 1`). This makes the data readily available on the same socket where the computation takes place. Using this setting is very workload-specific and may require some fine-tuning, as one needs to manage a trade-off between using more OMP threads vs. limiting the number of remote memory calls. Improving memory bounds ----------------------- Following the CPU performance optimization guidelines for :pytorch:`PyTorch`, it is also advised for :pyg:`PyG` to use :obj:`jemalloc` or :obj:`TCMalloc`. These generally can reach better memory usage than the default :pytorch:`PyTorch` `memory allocator `__ :obj:`PTMalloc`. A `non-default memory allocator `__ can be specified using :obj:`LD_PRELOAD` prior to script execution. Quick start guidelines ---------------------- The general guidelines for achieving the best performance with CPU affinity can be summarized in the following steps: #. Test if your dataset benefits from using parallel data loaders. For some datasets, it might be more beneficial to use a plain serial data loader, especially when the dimensions of the input :class:`~torch_geometric.data.Data` are relatively small. #. Enable multi-process data loaders by setting :attr:`num_workers > 0`. A good estimate for :obj:`num_workers` lies in the range :obj:`[2, 4]`. However, for more complex datasets you might want to experiment with larger number of workers. Use the :meth:`~torch_geometric.loader.AffinityMixin.enable_cpu_affinity` feature to affinitize :class:`~torch.utils.data.DataLoader` cores. #. Bind execution to physical cores. Alternatively, hyperthreading can be disabled completely at a system-level. #. Separate the cores used for main process from the data loader workers' cores by using :obj:`numactl`, :obj:`KMP_AFFINITY` of the :obj:`libiomp5` library, or :obj:`GOMP_CPU_AFFINITY` of the :obj:`libgomp` library. #. Find the optimum number of OMP threads for your workload. A good starting point is :obj:`N - num_workers`. Generally, well-parallelized models will benefit from many OMP threads. However, if your model computation flow has interlaced parallel and serial regions, the performance will decrease due to resource allocation needed for spawning and maintaining threads between parallel regions. #. When using a dual-socket CPU, you might want to experiment with assigning data loading to one socket and main process to another socket with memory allocation (:obj:`numactl -m`) on the same socket where the main process is executed. This leads to best cache-allocation and often overweighs the benefit of using more OMP threads. #. An additional boost in performance can be obtained by using non-default memory allocator, such as :obj:`jemalloc` or :obj:`TCMalloc`. #. Finding an optimal setup for the CPU affinity mask is a problem of managing the proportion of CPU time spent in each iteration for loading and preparing the data vs. time spent during GNN execution. Different results may be obtained by changing model hyperparameters, such as the batch size, number of sampled neighbors, and the number of layers. As a general rule, workloads which require sampling a complex graph may benefit more from reserving some CPU resources just for the data preparation step. Example results --------------- The figure below presents the outcome of applying CPU affinity mask to :obj:`benchmark/training/training_benchmark.py`. Measurements were taken for a variable number of workers, while other hyperparameters for each benchmark were constant: :obj:`--warmup 0 --use-sparse-tensor --num-layers 3 --num-hidden-channels 128 --batch-sizes 2048`. Three different affinity configurations are presented: * **Baseline** - only :obj:`OMP_NUM_THREADS` changes: .. code-block:: console OMP_NUM_THREADS=(N-num_workers) python training_benchmark.py --num-workers … * **Aff** - data loader process on first socket, main process on first and second socket, 98-110 threads: .. code-block:: console LD_PRELOAD=(path)/libjemalloc.so (path)/libiomp5.so MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto OMP_NUM_THREADS=(N-num_workers) KMP_AFFINITY=granularity=fine,compact,1,0 KMP_BLOCKTIME=0 numactl -C --localalloc python training_benchmark.py --cpu-affinity --num-workers … * **Aff+SocketSep** - data loader process on first socket, main process on second socket, 60 threads: .. code-block:: console LD_PRELOAD=(path)/libjemalloc.so (path)/libiomp5.so MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto OMP_NUM_THREADS=(N-M) KMP_AFFINITY=granularity=fine,compact,1,0 KMP_BLOCKTIME=0 numactl -C -m 1 python training_benchmark.py --cpu-affinity --num-workers ... Training times for each model/dataset combination were obtained by taking a mean of results at a variable number of dataloader workers: :obj:`[0, 2, 4, 8, 16]` for the baseline and :obj:`[2, 4, 8, 16]` workers for each affinity configuration. Then, the affinity means were normalized with respect to the mean baseline measurement. This value is denoted on the :math:`y`-axis. The labels above each result indicate the end-to-end performance gain from using the discussed configuration. Over all model/dataset samples, the average training time is decreased by **1.53x** for plain affinity and **1.85x** for the affinity with socket separation. .. figure:: ../_figures/training_affinity.png :width: 100% Pre-production dual-socket Intel(R) Xeon(R) Platinum 8481C @ 2.0Ghz (2 x 56) cores CPU. pytorch/docs/source/advanced/graphgym.rst000066400000000000000000000310441444262304300211170ustar00rootroot00000000000000Managing Experiments with GraphGym ================================== GraphGym is a platform for **designing and evaluating Graph Neural Networks (GNNs)**, as originally proposed in the `"Design Space for Graph Neural Networks" `__ paper. We now officially support GraphGym as part of of :pyg:`PyG`. .. warning:: GraphGym API may change in the future as we are continuously working on better and deeper integration with :pyg:`PyG`. Highlights ---------- #. **Highly modularized pipeline for GNN:** - **Data:** Data loading and data splitting - **Model:** Modularized GNN implementations - **Tasks:** Node-level, edge-level and graph-level tasks - **Evaluation:** Accuracy, ROC AUC, ... #. **Reproducible experiment configuration:** - Each experiment is *fully described by a configuration file* #. **Scalable experiment management:** - Easily launch *thousands of GNN experiments in parallel* - *Auto-generate* experiment analyses and figures across random seeds and experiments #. **Flexible user customization:** - Easily *register your own modules*, such as data loaders, GNN layers, loss functions, etc Why GraphGym? ------------- **TL;DR:** GraphGym is great for GNN beginners, domain experts and GNN researchers. **Scenario 1:** You are a beginner to graph representation learning and want to understand how GNNs work: You probably have read many exciting papers on GNNs, and try to write your own GNN implementation. Even if using raw :pyg:`PyG`, you still have to code up the essential pipeline on your own. GraphGym is a perfect place for your to start learning about *standardized GNN implementation and evaluation*. .. figure:: ../_figures/graphgym_design_space.png :align: center :width: 450px **Figure 1:** Modularized GNN implementation. **Scenario 2:** You want to apply GNNs to your exciting application: You probably know that there are hundreds of possible GNN models, and selecting the best model is notoriously hard. Even worse, the `GraphGym paper `__ shows that the best GNN designs for different tasks differ drastically. GraphGym provides a *simple interface to try out thousands of GNNs in parallel* and understand the best designs for your specific task. GraphGym also recommends a "go-to" GNN design space, after investigating 10 million GNN model-task combinations. .. figure:: ../_figures/graphgym_results.png :align: center :width: 100% **Figure 2:** A guideline for desirable GNN design choices. **Scenario 3:** You are a GNN researcher, who wants to innovate new GNN models or propose new GNN tasks: Say you have proposed a new GNN layer :class:`ExampleConv`. GraphGym can help you convincingly argue that :class:`ExampleConv` is better than, *e.g.*, :class:`~torch_geometric.nn.conv.GCNConv`: When randomly sampling from 10 million possible model-task combinations, how often will :class:`ExampleConv` will outperform :class:`~torch_geometric.nn.conv.GCNConv` when everything else is fixed (including computational costs)? Moreover, GraphGym can help you easily do hyper-parameter search, and *visualize* what design choices are better. In sum, GraphGym can greatly facilitate your GNN research. .. figure:: ../_figures/graphgym_evaluation.png :align: center :width: 100% **Figure 3:** Evaluation of a given GNN design dimension, *e.g.*, :obj:`BatchNorm`. Basic Usage ----------- To use GraphGym, you need to clone :pyg:`PyG` from :github:`GitHub`, then change to the :obj:`graphgym/` directory. .. code-block:: bash git clone https://github.com/pyg-team/pytorch_geometric.git cd pytorch_geometric/graphgym #. **Run a single experiment:** Run an experiment using GraphGym via :obj:`run_single.sh`. Configurations are specified in :obj:`configs/example_node.yaml`. The default experiment is about node classification on the :class:`~torch_geometric.datasets.Planetoid` datasets (using a random 80/20 train/validation split). .. code-block:: bash bash run_single.sh # run a single experiment #. **Run a batch of experiments:** Run a batch of experiments using GraphGym via :obj:`run_batch.sh`. Configurations are specified in :obj:`configs/example_node.yaml` (controls the basic architecture) and :obj:`grids/example.txt` (controls how to do grid search). The experiment examines 96 models in the recommended GNN design space, on 2 graph classification datasets. Each experiment is repeated 3 times, and we set up that 8 jobs can be concurrently run. Depending on your infrastructure, finishing all the experiments may take a long time; you can quit the experiment via :obj:`Ctrl-C` (GraphGym will properly kill all the processes). .. code-block:: bash bash run_batch.sh # run a batch of experiments #. **Run GraphGym with CPU backend:** GraphGym supports CPU backend as well -- you only need to add the line :obj:`accelerator: cpu` to the :obj:`*.yaml` file. In-Depth Usage -------------- To use GraphGym, you need to clone :pyg:`PyG` from :github:`GitHub`, then change to the :obj:`graphgym/` directory. .. code-block:: bash git clone https://github.com/pyg-team/pytorch_geometric.git cd pytorch_geometric/graphgym #. **Run a single experiment:** A full example is specified in :obj:`run_single.sh`. #. **Specify a configuration file:** In GraphGym, an experiment is fully specified by a :obj:`*.yaml` file. Unspecified configurations in the :obj:`*.yaml` file will be populated by the default values in :meth:`torch_geometric.graphgym.set_cfg`. For example, in :obj:`configs/example_node.yaml`, there are configurations for the dataset, training procedure, model, etc. Concrete description for each configuration is described in :meth:`~torch_geometric.graphgym.set_cfg`. #. **Launch an experiment:** For example, in :obj:`run_single.sh`: .. code-block:: bash python main.py --cfg configs/example_node.yaml --repeat 3 You can specify the number of different random seeds to repeat via :obj:`--repeat`. #. **Understand the results:** Experimental results will be automatically saved in :obj:`results/${CONFIG_NAME}/`. In the example above, this amounts to :obj:`results/example_node/`. Results for different random seeds will be saved in different subdirectories, *e.g.*, :obj:`results/example_node/2`. The aggregated results over all the random seeds are *automatically* generated into :obj:`results/example/agg`, including the mean and standard deviation :obj:`_std` for each metric. Train/validation/test results are further saved into subdirectories, such as :obj:`results/example/agg/val`. Here, :obj:`stats.json` stores the results after each epoch aggregated across random seeds, and :obj:`best.json` stores the results of *the epoch with the highest validation accuracy*. #. **Run a batch of experiments:** A full example is specified in :obj:`run_batch.sh`. #. **Specify a base file:** GraphGym supports running a batch of experiments. To start, a user needs to select a base architecture via :obj:`--config`. The batch of experiments will be created by perturbing certain configurations of the base architecture. #. **(Optionally) specify a base file for computational budget:** Additionally, GraphGym allows a user to select a base architecture to *control the computational budget* for the grid search via :obj:`--config_budget`. The computational budget is currently measured by the number of trainable parameters, and the control is achieved by auto-adjusting the hidden dimensionality of the underlying GNN. If no :obj:`--config_budget` is provided, GraphGym will not control the computational budget. #. **Specify a grid file:** A grid file describes how to perturb the base file, in order to generate the batch of experiments. For example, the base file could specify an experiment of a 3-layer GCN for node classification on the :class:`~torch_geometric.datasets.Planetoid` datasets. Then, the grid file specifies how to perturb the experiment along different dimensions, such as the number of layers, the model architecture, the dataset, the level of task, etc. #. **Generate configuration files for the batch of experiments** based on the information specified above: For example, in :obj:`run_batch.sh`: .. code-block:: bash python configs_gen.py --config configs/${DIR}/${CONFIG}.yaml \ --config_budget configs/${DIR}/${CONFIG}.yaml \ --grid grids/${DIR}/${GRID}.txt \ --out_dir configs #. **Launch the batch of experiments:** For example, in :obj:`run_batch.sh`: .. code-block:: bash bash parallel.sh configs/${CONFIG}_grid_${GRID} $REPEAT $MAX_JOBS $SLEEP Each experiment will be repeated for :obj:`$REPEAT` times. We implemented a queue system to sequentially launch all the jobs, with :obj:`$MAX_JOBS` concurrent jobs running at the same time. In practice, our system works great when handling thousands of jobs. #. **Understand the results:** Experimental results will be automatically saved in directory :obj:`results/${CONFIG_NAME}_grid_${GRID_NAME}/`. In the example above, this amounts to :obj:`results/example_grid_example/`. After running each experiment, GraphGym additionally automatically averages across different models, saved in :obj:`results/example_grid_example/agg`. There, :obj:`val.csv` represents the validation accuracy for each model configuration at the *final* epoch, :obj:`val_best.csv` represents the results at the epoch with the highest average validation accuracy, and :obj:`val_best_epoch.csv` represents the results at the epoch with the highest validation accuracy averaged over different random seeds. When a test set split is provided, :obj:`test.csv` represents the test accuracy for each model configuration at the *final* epoch, :obj:`test_best.csv` represents the test set results at the epoch with the highest average validation accuracy, and :obj:`test_best_epoch.csv` represents the test set results at the epoch with the highest validation accuracy averaged over different random seeds. Customizing GraphGym -------------------- A highlight of GraphGym is that it allows you to easily register customized modules. For each project, you can have a unique GraphGym copy with different customized modules. For example, the `"Design Space for Graph Neural Networks" `__ and `"Identity-aware Graph Neural Networks" `__ papers represent two successful projects using customized GraphGym, and you may find more details about them `here `__. Eventually, every GraphGym-powered project will be unique. There are two ways for customizing GraphGym: #. Use the :obj:`graphgym/custom_graphgym` directory outside the :pyg:`PyG` package: You can register your customized modules here without touching :pyg:`PyG`. This use case will be great for your own customized project. #. Use the :obj:`torch_geometric/graphgym/contrib` directory inside the :pyg:`PyG` package: If you have come up with a nice customized module, you can directly copy your files into :obj:`torch_geometric/graphgym/contrib`, and **create a pull request** to :pyg:`PyG`. This way, your idea can ship with :pyg:`PyG` installations, and will have a much higher visibility and impact. Concretely, the supported customized modules includes - Activations: :obj:`custom_graphgym/act/` - Customized configurations: :obj:`custom_graphgym/config/` - Feature augmentations: :obj:`custom_graphgym/feature_augment/` - Feature encoders: :obj:`custom_graphgym/feature_encoder/` - GNN heads: :obj:`custom_graphgym/head/` - GNN layers: :obj:`custom_graphgym/layer/` - Data loaders: :obj:`custom_graphgym/loader/` - Loss functions: :obj:`custom_graphgym/loss/` - GNN network architectures: :obj:`custom_graphgym/network/` - Optimizers: :obj:`custom_graphgym/optimizer/` - GNN global pooling layers (for graph classification only): :obj:`custom_graphgym/pooling/` - GNN stages: :obj:`custom_graphgym/stage/` - GNN training pipelines: :obj:`custom_graphgym/train/` - Data transformations: :obj:`custom_graphgym/transform/` Within each directory, at least one example is provided that shows how to register customized modules via :meth:`torch_geometric.graphgym.register`. Note that new customized modules may result in new configurations. In these cases, new configuration fields can be registered via :obj:`custom_graphgym/config/`. pytorch/docs/source/advanced/jit.rst000066400000000000000000000137211444262304300200710ustar00rootroot00000000000000TorchScript Support =================== TorchScript is a way to create serializable and optimizable models from :pytorch:`PyTorch` code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency. If you are unfamilar with TorchScript, we recommend to read the official "`Introduction to TorchScript `_" tutorial first. Converting GNN Models --------------------- Converting your :pyg:`PyG` model to a TorchScript program is straightforward and requires only a few code changes. As always, it is best understood by an example, so let's consider the following model for now: .. code-block:: python import torch import torch.nn.functional as F from torch_geometric.nn import GCNConv class GNN(torch.nn.Module): def __init__(self, in_channels, out_channels): super().__init__() self.conv1 = GCNConv(in_channels, 64) self.conv2 = GCNConv(64, out_channels) def forward(self, x, edge_index): x = self.conv1(x, edge_index) x = F.relu(x) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) model = GNN(dataset.num_features, dataset.num_classes) For TorchScript support, we need to convert our GNN operators into "jittable" instances. This is done by calling the :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.jittable` function provided by the underlying :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` interface: .. code-block:: python def __init__(self, in_channels, out_channels): super().__init__() self.conv1 = GCNConv(in_channels, 64).jittable() self.conv2 = GCNConv(64, out_channels).jittable() This will create temporary instances of the :class:`~torch_geometric.nn.conv.GCNConv` operator that can now be passed into :func:`torch.jit.script`: .. code-block:: python model = torch.jit.script(model) Under the hood, the :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.jittable` call applies the following two modifications to the original class: 1. It parses and converts the arguments of the internal :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate` function into a :obj:`NamedTuple` which can be handled by the TorchScript compiler. 2. It replaces any :obj:`Union` arguments of the :func:`forward` function (*i.e.*, arguments that may contain different types) with :obj:`@torch.jit._overload_method` annotations. With this, we can do the following while everything remains jittable: .. code-block:: python from typing import Union, Tuple from torch import Tensor def forward(self, x: Union[Tensor, Tuple[Tensor, Tensor]], edge_index: Tensor) -> Tensor: pass conv(x, edge_index) conv((x_src, x_dst), edge_index) This technique is, *e.g.*, applied in the :class:`~torch_geometric.nn.conv.SAGEConv` class, which can operate on both single node feature matrices and tuples of node feature matrices at the same time. And that is all you need to know on how to convert your :pyg:`PyG` models to TorchScript programs. You can have a further look at our JIT examples that show-case how to obtain TorchScript programs for `node `_ and `graph classification `_ models. .. note:: TorchScript support is still experimental. If you have problems converting your model to a TorchScript program, *e.g.*, because an internal function does not support TorchScript yet, please `let us know `_. Creating Jittable GNN Operators -------------------------------- All :pyg:`PyG` :class:`~torch_geometric.nn.conv.MessagePassing` operators are tested to be convertible to a TorchScript program. However, if you want your own GNN module to be jittable, you need to account for the following two things: 1. As one would expect, your :meth:`forward` code may need to be adjusted so that it passes the TorchScript compiler requirements, *e.g.*, by adding type notations. 2. You need to tell the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` module the types that you pass to its :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate` function. This can be achieved in two different ways: 1. Declaring the type of propagation arguments in a dictionary called :obj:`propagate_type`: .. code-block:: python from typing import Optional from torch import Tensor from torch_geometric.nn import MessagePassing class MyConv(MessagePassing): propagate_type = {'x': Tensor, 'edge_weight': Optional[Tensor] } def forward(self, x: Tensor, edge_index: Tensor, edge_weight: Optional[Tensor]) -> Tensor: return self.propagate(edge_index, x=x, edge_weight=edge_weight, size=None) 2. Declaring the type of propagation arguments as a comment anywhere inside your module: .. code-block:: python from typing import Optional from torch import Tensor from torch_geometric.nn import MessagePassing class MyConv(MessagePassing): def forward(self, x: Tensor, edge_index: Tensor, edge_weight: Optional[Tensor]) -> Tensor: # propagate_type: (x: Tensor, edge_weight: Optional[Tensor]) return self.propagate(edge_index, x=x, edge_weight=edge_weight, size=None) .. warning:: Importantly, due to TorchScript limitations, one also has to pass in the :obj:`size` attribute to :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate`. In most cases, this can be simply set to :obj:`None` in which case it will be automatically inferred. pytorch/docs/source/advanced/remote.rst000066400000000000000000000302261444262304300205750ustar00rootroot00000000000000Scaling Up GNNs via Remote Backends =================================== :pyg:`PyG` (2.2 and beyond) includes numerous primitives to easily integrate with simple paradigms for scalable graph machine learning, enabling users to train GNNs on graphs far larger than the size of their machine's available memory. It does so by introducing simple, easy-to-use, and extensible abstractions of a :class:`torch_geometric.data.FeatureStore` and a :class:`torch_geometric.data.GraphStore` that plug directly into existing familiar :pyg:`PyG` interfaces. Defining a :class:`~torch_geometric.data.FeatureStore` allows users to leverage node (and soon, edge) features stored remotely, and defining a :class:`~torch_geometric.data.GraphStore` allows users to leverage graph structure information stored remotely. Together, they allow for powerful GNN scalability with low developer friction. .. warning:: The remote backend APIs discussed here may change in the future as we continuously work to improve their ease-of-use and generalizability. .. note:: Currently, the :class:`~torch_geometric.data.FeatureStore` and :class:`~torch_geometric.data.GraphStore` only support *heterogeneous graphs*, and do not support edge features. Homogeneous graph and edge feature support is coming soon. Background ---------- An instantiated Graph Neural Network consists of two types of data: - **Node and/or edge feature information:** Dense vectors corresponding to attributes of the nodes and edges in a graph - **Graph structure information:** The nodes in the graph and the edges that connect them An immediate observation of GNNs is that scaling to data larger than the available memory of a chosen accelerator requires training on sampled subgraphs (which form mini-batches), instead of the full graph at once (full-batch training). While this method adds stochasticity to the learning process, it reduces the memory requirements of the accelerator to those of the sampled subgraphs. .. figure:: ../_figures/remote_1.png :align: center :width: 100% **Figure 1:** The classical mini-batch GNN training paradigm. However, while mini-batch training reduces the memory requirements of the chosen accelerator, it is not a silver bullet for all graph learning scalability problems. In particular, since one must sample subgraphs to pass to the accelerator at each iteration of the learning process, the graph and features are traditionally required to be stored in the CPU DRAM of a user's machine. At large scale, this requirement can become quite burdensome: - Acquiring instances with enough CPU DRAM to store a graph and features is challenging - Training with data parallelism requires replicating the graph and features in each compute node - Graphs and features can easily be much larger than the memory of a single machine Scalability to very large graphs and features beyond the memory requirements of a single machine thus requires moving these data structures out-of-core and only processing sampled subgraphs on a node that performs computation. In order to achieve this goal, :pyg:`PyG` relies on two primary abstractions to store feature information and graph structure: Features are stored in a key-value :class:`~torch_geometric.data.FeatureStore`, which must support efficient random access. Graph information is stored in a :class:`~torch_geometric.data.GraphStore`, which must support efficient sampling for the samplers defined to operate on the :class:`~torch_geometric.data.GraphStore` instance. .. figure:: ../_figures/remote_2.png :align: center :width: 100% **Figure 2:** Graph data storage layout between remote storage and a training instance. In :pyg:`PyG` (2.2 and beyond), the separation of graph data into its features and structure information, the storage of this information in locations potentially remote to the actual training node, and the interactions between these components, are all completely abstracted from the end user. As long as the :class:`~torch_geometric.data.FeatureStore` and :class:`~torch_geometric.data.GraphStore` are defined appropriately (keeping in mind the aforementioned performance requirements), :pyg:`PyG` handles the rest! Feature Store ------------- A :class:`torch_geometric.data.FeatureStore` holds features for the nodes and edges of a graph. Feature storage is often the primary storage bottleneck in graph learning applications, as storing a graph's layout information (*i.e.* the :obj:`edge_index`) is relatively cheap (~32 bytes per edge). :pyg:`PyG` provides a common interface for various :class:`~torch_geometric.data.FeatureStore` implementations to interface with its core learning API. The implementation details of a :class:`~torch_geometric.data.FeatureStore` are abstracted from :pyg:`PyG` through a CRUD-like interface. In particular, implementors of the :class:`~torch_geometric.data.FeatureStore` abstraction are expected to primarily override :meth:`~torch_geometric.data.FeatureStore.put_tensor`, :meth:`~torch_geometric.data.FeatureStore.get_tensor`, and :meth:`~torch_geometric.data.FeatureStore.remove_tensor` functionalities. Doing so both enables :pyg:`PyG` to leverage the features stored in the implementation and allows a user to employ a pythonic interface to inspect and modify the :class:`~torch_geometric.data.FeatureStore` elements: .. code-block:: python feature_store = CustomFeatureStore() paper_features = ... # [num_papers, num_paper_features] author_features = ... # [num_authors, num_author_features] # Add features: feature_store['paper', 'x', None] = paper_features feature_store['author', 'x', None] = author_features # Access features: assert torch.equal(feature_store['paper', 'x'], paper_features) assert torch.equal(feature_store['paper'].x, paper_features) assert torch.equal(feature_store['author', 'x', 0:20], author_features[0:20]) Common implementations of the :class:`~torch_geometric.data.FeatureStore` abstractions are key-value stores, *e.g.*, backends such as :obj:`memcached`, :obj:`LevelDB`, :obj:`RocksDB` are all viable performant options. Graph Store and Sampler ----------------------- A :class:`torch_geometric.data.GraphStore` holds the edge indices that define relationships between nodes in a graph. The goal of the :class:`~torch_geometric.data.GraphStore` is to store graph information in a manner that allows for efficient sampling from root nodes, according to a sampling algorithm of the developer's choice. Similar to the :class:`~torch_geometric.data.FeatureStore`, :pyg:`PyG` provides a common interface for various :class:`~torch_geometric.data.GraphStore` implementations to interface with its core learning API. However, unlike the :class:`~torch_geometric.data.FeatureStore`, the :class:`~torch_geometric.data.GraphStore` does not need to provide efficient random access for all its elements; rather, it needs to define a representation that provides efficient subgraph sampling. An example usage of the interface is shown below: .. code-block:: python graph_store = CustomGraphStore() edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]]) # Put edges: graph_store['edge', 'coo'] = coo # Access edges: row, col = graph_store['edge', 'coo'] assert torch.equal(row, edge_index[0]) assert torch.equal(col, edge_index[1]) Common implementations of the :class:`~torch_geometric.data.GraphStore` are graph databases, *e.g.*, :obj:`Neo4j`, :obj:`TigerGraph`, :obj:`ArangoDB`, :obj:`KΓΉzu` are all viable performant options. We provide an example of using :pyg:`PyG` in combination with the :obj:`KΓΉzu` database `here `__. A graph sampler is tightly coupled to the given :class:`~torch_geometric.data.GraphStore`, and operates on the :class:`~torch_geometric.data.GraphStore` to produce sampled subgraphs from input nodes. Different sampling algorithms are implemented behind the :class:`torch_geometric.sampler.BaseSampler` interface. By default, :pyg:`PyG's` default in-memory sampler pulls all edge indices from the :class:`~torch_geometric.data.GraphStore` into the training node memory, converts them to compressed sparse column (CSC) format, and leverages pre-built in-memory sampling routines. However, custom sampler implementations may choose to call specialized :class:`~torch_geometric.data.GraphStore` methods by implementing the :meth:`~torch_geometric.sampler.BaseSampler.sample_from_nodes` and/or :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges` of the :class:`~torch_geometric.sampler.BaseSampler` class for efficiency reasons (*e.g.*, for performing sampling directly on the remote :class:`~torch_geometric.data.GraphStore`): .. code-block:: python # `CustomGraphSampler` knows how to sample on `CustomGraphStore`: node_sampler = CustomGraphSampler( graph_store=graph_store, num_neighbors=[10, 20], ... ) Data Loader ----------- :pyg:`PyG` does not define a domain-specific language for sampling that must be implemented by the :class:`~torch_geometric.data.GraphStore`; rather, the sampler and the :class:`~torch_geometric.data.GraphStore` are tightly coupled together through a data loader. :pyg:`PyG` provides two data loaders out-of-the-box: a :class:`torch_geometric.loader.NodeLoader` that samples subgraphs from input nodes for use in node classification tasks, and a :class:`torch_geometric.loader.LinkLoader` that samples subgraphs from either side of an edge for use in link prediction tasks. These data loaders require a :class:`~torch_geometric.data.FeatureStore`, a :class:`~torch_geometric.data.GraphStore`, and a graph sampler as input, and internally call the sampler's :meth:`~torch_geometric.sampler.BaseSampler.sample_from_nodes` or :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges` method to perform subgraph sampling: .. code-block:: python # Instead of passing PyG data objects, we now pass a tuple # of the `FeatureStore` and `GraphStore as input data: loader = NodeLoader( data=(feature_store, graph_store), node_sampler=node_sampler, batch_size=20, input_nodes='paper', ) for batch in loader: pass Putting it All Together ----------------------- At a high level, the components listed above all work together to provide support for scaling up GNNs within :pyg:`PyG`. - The **data loader** (precisely, each worker) leverages a :class:`~torch_geometric.sampler.BaseSampler` to make a sampling request to the :class:`~torch_geometric.data.GraphStore`. - Upon receipt of a response, the data loader subsequently queries the :class:`~torch_geometric.data.FeatureStore` for features associated with the nodes and edges of the sampled subgraphs. - The data loader subsequently constructs a final mini-batch from graph structure and feature information to send to the accelerator for forward/backward passes. - Repeat until convergence. All of the outlined classes speak through common interfaces, making them extensible, generalizable, and easy to integrate with the :pyg:`PyG` you use today: .. figure:: ../_figures/remote_3.png :align: center :width: 80% **Figure 3:** The common interfaces (and data flow) uniting the :class:~torch_geometric.data.`FeatureStore`, :class:`~torch_geometric.data.GraphStore`, graph sampler, and data loader. To get started with scalability, we recommend inspecting the interfaces listed above and defining your own :class:`~torch_geometric.data.FeatureStore`, :class:`~torch_geometric.data.GraphStore`, and :class:`~torch_geometric.sampler.BaseSampler` implementations behind them. Once a :class:`~torch_geometric.data.FeatureStore`, a :class:`~torch_geometric.data.GraphStore`, and a :class:`~torch_geometric.sampler.BaseSampler` are correctly implemented, simply pass them as parameters to a :class:`~torch_geometric.loader.NodeLoader` or a :class:`~torch_geometric.loader.LinkLoader`, and the rest of :pyg:`PyG` will work seamlessly and similar to any pure in-memory application. Since this feature is still undergoing heavy development, please feel free to reach out to the :pyg:`PyG` core team either on :github:`null` `GitHub `_ or :slack:`null` `Slack `_ if you have any questions, comments or concerns. pytorch/docs/source/advanced/sparse_tensor.rst000066400000000000000000000214361444262304300221740ustar00rootroot00000000000000Memory-Efficient Aggregations ============================= The :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` interface of :pyg:`PyG` relies on a gather-scatter scheme to aggregate messages from neighboring nodes. For example, consider the message passing layer .. math:: \mathbf{x}^{\prime}_i = \sum_{j \in \mathcal{N}(i)} \textrm{MLP}(\mathbf{x}_j - \mathbf{x}_i), that can be implemented as: .. code-block:: python from torch_geometric.nn import MessagePassing x = ... # Node features of shape [num_nodes, num_features] edge_index = ... # Edge indices of shape [2, num_edges] class MyConv(MessagePassing): def __init__(self): super().__init__(aggr="add") def forward(self, x, edge_index): return self.propagate(edge_index, x=x) def message(self, x_i, x_j): return MLP(x_j - x_i) Under the hood, the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` implementation produces a code that looks as follows: .. code-block:: python from torch_geometric.utils import scatter x = ... # Node features of shape [num_nodes, num_features] edge_index = ... # Edge indices of shape [2, num_edges] x_j = x[edge_index[0]] # Source node features [num_edges, num_features] x_i = x[edge_index[1]] # Target node features [num_edges, num_features] msg = MLP(x_j - x_i) # Compute message for each edge # Aggregate messages based on target node indices out = scatter(msg, edge_index[1], dim=0, dim_size=x.size(0), reduce='sum') While the gather-scatter formulation generalizes to a lot of useful GNN implementations, it has the disadvantage of explicitely materalizing :obj:`x_j` and :obj:`x_i`, resulting in a high memory footprint on large and dense graphs. Luckily, not all GNNs need to be implemented by explicitely materalizing :obj:`x_j` and/or :obj:`x_i`. In some cases, GNNs can also be implemented as a simple-sparse matrix multiplication. As a general rule of thumb, this holds true for GNNs that do not make use of the central node features :obj:`x_i` or multi-dimensional edge features when computing messages. For example, the :class:`~torch_geometric.nn.conv.GINConv` layer .. math:: \mathbf{x}^{\prime}_i = \textrm{MLP} \left( (1 + \epsilon) \cdot \mathbf{x}_i + \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j \right), is equivalent to computing .. math:: \mathbf{X}^{\prime} = \textrm{MLP} \left( (1 + \epsilon) \cdot \mathbf{X} + \mathbf{A}\mathbf{X} \right), where :math:`\mathbf{A}` denotes a sparse adjacency matrix of shape :obj:`[num_nodes, num_nodes]`. This formulation allows to leverage dedicated and fast sparse-matrix multiplication implementations. In :pyg:`null` **PyG >= 1.6.0**, we officially introduce better support for sparse-matrix multiplication GNNs, resulting in a **lower memory footprint** and a **faster execution time**. As a result, we introduce the :class:`SparseTensor` class (from the :obj:`torch_sparse` package), which implements fast forward and backward passes for sparse-matrix multiplication based on the `"Design Principles for Sparse Matrix Multiplication on the GPU" `_ paper. Using the :class:`SparseTensor` class is straightforward and similar to the way :obj:`scipy` treats sparse matrices: .. code-block:: python from torch_sparse import SparseTensor adj = SparseTensor(row=edge_index[0], col=edge_index[1], value=..., sparse_sizes=(num_nodes, num_nodes)) # value is optional and can be None # Obtain different representations (COO, CSR, CSC): row, col, value = adj.coo() rowptr, col, value = adj.csr() colptr, row, value = adj.csc() adj = adj[:100, :100] # Slicing, indexing and masking support adj = adj.set_diag() # Add diagonal entries adj_t = adj.t() # Transpose out = adj.matmul(x) # Sparse-dense matrix multiplication adj = adj.matmul(adj) # Sparse-sparse matrix multiplication # Creating SparseTensor instances: adj = SparseTensor.from_dense(mat) adj = SparseTensor.eye(100, 100) adj = SparseTensor.from_scipy(mat) Our :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` interface can handle both :obj:`torch.Tensor` and :class:`SparseTensor` as input for propagating messages. However, when holding a directed graph in :class:`SparseTensor`, you need to make sure to input the **transposed sparse matrix** to :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate`: .. code-block:: python conv = GCNConv(16, 32) out1 = conv(x, edge_index) out2 = conv(x, adj.t()) assert torch.allclose(out1, out2) conv = GINConv(nn=Sequential(Linear(16, 32), ReLU(), Linear(32, 32))) out1 = conv(x, edge_index) out2 = conv(x, adj.t()) assert torch.allclose(out1, out2) To leverage sparse-matrix multiplications, the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` interface introduces the :func:`~torch_geometric.nn.conv.message_passing.message_and_aggregate` function (which fuses the :func:`~torch_geometric.nn.conv.message_passing.message` and :func:`~torch_geometric.nn.conv.message_passing.aggregate` functions into a single computation step), which gets called whenever it is implemented and receives a :class:`SparseTensor` as input for :obj:`edge_index`. With it, the :class:`~torch_geometric.nn.conv.GINConv` layer can now be implemented as follows: .. code-block:: python import torch_sparse class GINConv(MessagePassing): def __init__(self): super().__init__(aggr="add") def forward(self, x, edge_index): out = self.propagate(edge_index, x=x) return MLP((1 + eps) x + out) def message(self, x_j): return x_j def message_and_aggregate(self, adj_t, x): return torch_sparse.matmul(adj_t, x, reduce=self.aggr) Playing around with the new :class:`SparseTensor` format is straightforward since all of our GNNs work with it out-of-the-box. To convert the :obj:`edge_index` format to the newly introduced :class:`SparseTensor` format, you can make use of the :class:`torch_geometric.transforms.ToSparseTensor` transform: .. code-block:: python import torch import torch.nn.functional as F from torch_geometric.nn import GCNConv import torch_geometric.transforms as T from torch_geometric.datasets import Planetoid dataset = Planetoid("Planetoid", name="Cora", transform=T.ToSparseTensor()) data = dataset[0] >>> Data(adj_t=[2708, 2708, nnz=10556], x=[2708, 1433], y=[2708], ...) class GNN(torch.nn.Module): def __init__(self): super().__init__() self.conv1 = GCNConv(dataset.num_features, 16, cached=True) self.conv2 = GCNConv(16, dataset.num_classes, cached=True) def forward(self, x, adj_t): x = self.conv1(x, adj_t) x = F.relu(x) x = self.conv2(x, adj_t) return F.log_softmax(x, dim=1) model = GNN() optimizer = torch.optim.Adam(model.parameters(), lr=0.01) def train(data): model.train() optimizer.zero_grad() out = model(data.x, data.adj_t) loss = F.nll_loss(out, data.y) loss.backward() optimizer.step() return float(loss) for epoch in range(1, 201): loss = train(data) All code remains the same as before, except for the :obj:`data` transform via :obj:`T.ToSparseTensor()`. As an additional advantage, :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` implementations that utilize the :class:`SparseTensor` class are deterministic on the GPU since aggregations no longer rely on atomic operations. Notably, the GNN layer execution slightly changes in case GNNs incorporate single or multi-dimensional edge information :obj:`edge_weight` or :obj:`edge_attr` into their message passing formulation, respectively. In particular, it is now expected that these attributes are directly added as values to the :class:`SparseTensor` object. Instead of calling the GNN as .. code-block:: python conv = GMMConv(16, 32, dim=3) out = conv(x, edge_index, edge_attr) we now execute our GNN operator as .. code-block:: python conv = GMMConv(16, 32, dim=3) adj = SparseTensor(row=edge_index[0], col=edge_index[1], value=edge_attr) out = conv(x, adj.t()) .. note:: Since this feature is still experimental, some operations, *e.g.*, graph pooling methods, may still require you to input the :obj:`edge_index` format. You can convert :obj:`adj_t` back to :obj:`(edge_index, edge_attr)` via: .. code-block:: python row, col, edge_attr = adj_t.t().coo() edge_index = torch.stack([row, col], dim=0) Please let us know what you think of :class:`SparseTensor`, how we can improve it, and whenever you encounter any unexpected behavior. pytorch/docs/source/cheatsheet/000077500000000000000000000000001444262304300171155ustar00rootroot00000000000000pytorch/docs/source/cheatsheet/data_cheatsheet.rst000066400000000000000000000115351444262304300227620ustar00rootroot00000000000000Dataset Cheatsheet ================== .. note:: This dataset statistics table is a **work in progress**. Please consider helping us filling its content by providing statistics for individual datasets. See `here `__ and `here `__ for examples on how to do so. Homogeneous Datasets -------------------- .. list-table:: :widths: 50 10 10 10 10 10 :header-rows: 1 * - Name - #graphs - #nodes - #edges - #features - #classes/#tasks {% for cls in torch_geometric.datasets.homo_datasets %} * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %} - {%if torch_geometric.datasets.utils.has_stats(cls) %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default=1) }}{% else %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default='') }}{% endif %} - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', default='') }} {% for child in torch_geometric.datasets.utils.get_children(cls) %} * - └─ {{ child }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', child, default=1) }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }} {% endfor %} {% endfor %} Heterogeneous Datasets ---------------------- .. list-table:: :widths: 50 30 10 10 :header-rows: 1 * - Name - #nodes/#edges - #features - #classes/#tasks {% for cls in torch_geometric.datasets.hetero_datasets %} * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %} - - - {% for child in torch_geometric.datasets.utils.get_children(cls) %} * - └─ **{{torch_geometric.datasets.utils.get_type(child)}} Type**: {{ child }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes/#edges', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }} {% endfor %} {% endfor %} Synthetic Datasets ------------------ .. list-table:: :widths: 50 10 10 10 10 10 :header-rows: 1 * - Name - #graphs - #nodes - #edges - #features - #classes/#tasks {% for cls in torch_geometric.datasets.synthetic_datasets %} * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %} - {%if torch_geometric.datasets.utils.has_stats(cls) %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default=1) }}{% else %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default='') }}{% endif %} - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', default='') }} {% for child in torch_geometric.datasets.utils.get_children(cls) %} * - └─ {{ child }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', child, default=1) }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }} - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }} {% endfor %} {% endfor %} pytorch/docs/source/cheatsheet/gnn_cheatsheet.rst000066400000000000000000000123171444262304300226320ustar00rootroot00000000000000GNN Cheatsheet ============== * :class:`~torch_sparse.SparseTensor`: If checked (βœ“), supports message passing based on :class:`torch_sparse.SparseTensor`, *e.g.*, :obj:`GCNConv(...).forward(x, adj_t)`. See `here <../advanced/sparse_tensor.html>`__ for the accompanying tutorial. * :obj:`edge_weight`: If checked (βœ“), supports message passing with one-dimensional edge weight information, *e.g.*, :obj:`GraphConv(...).forward(x, edge_index, edge_weight)`. * :obj:`edge_attr`: If checked (βœ“), supports message passing with multi-dimensional edge feature information, *e.g.*, :obj:`GINEConv(...).forward(x, edge_index, edge_attr)`. * **bipartite**: If checked (βœ“), supports message passing in bipartite graphs with potentially different feature dimensionalities for source and destination nodes, *e.g.*, :obj:`SAGEConv(in_channels=(16, 32), out_channels=64)`. * **static**: If checked (βœ“), supports message passing in static graphs, *e.g.*, :obj:`GCNConv(...).forward(x, edge_index)` with :obj:`x` having shape :obj:`[batch_size, num_nodes, in_channels]`. * **lazy**: If checked (βœ“), supports lazy initialization of message passing layers, *e.g.*, :obj:`SAGEConv(in_channels=-1, out_channels=64)`. Graph Neural Network Operators ------------------------------ .. list-table:: :widths: 40 10 10 10 10 10 10 :header-rows: 1 * - Name - :class:`~torch_sparse.SparseTensor` - :obj:`edge_weight` - :obj:`edge_attr` - bipartite - static - lazy {% for cls in torch_geometric.nn.conv.classes[1:] %} {% if not torch_geometric.nn.conv.utils.processes_heterogeneous_graphs(cls) and not torch_geometric.nn.conv.utils.processes_hypergraphs(cls) and not torch_geometric.nn.conv.utils.processes_point_clouds(cls) %} * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__) - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %} {% endif %} {% endfor %} Heterogeneous Graph Neural Network Operators -------------------------------------------- .. list-table:: :widths: 40 10 10 10 10 10 10 :header-rows: 1 * - Name - :class:`~torch_sparse.SparseTensor` - :obj:`edge_weight` - :obj:`edge_attr` - bipartite - static - lazy {% for cls in torch_geometric.nn.conv.classes[1:] %} {% if torch_geometric.nn.conv.utils.processes_heterogeneous_graphs(cls) %} * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__) - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %} {% endif %} {% endfor %} Hypergraph Neural Network Operators ----------------------------------- .. list-table:: :widths: 40 10 10 10 10 10 10 :header-rows: 1 * - Name - :class:`~torch_sparse.SparseTensor` - :obj:`edge_weight` - :obj:`edge_attr` - bipartite - static - lazy {% for cls in torch_geometric.nn.conv.classes[1:] %} {% if torch_geometric.nn.conv.utils.processes_hypergraphs(cls) %} * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__) - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %} {% endif %} {% endfor %} Point Cloud Neural Network Operators ------------------------------------ .. list-table:: :widths: 80 10 10 :header-rows: 1 * - Name - bipartite - lazy {% for cls in torch_geometric.nn.conv.classes[1:] %} {% if torch_geometric.nn.conv.utils.processes_point_clouds(cls) %} * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__) - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %} - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %} {% endif %} {% endfor %} pytorch/docs/source/conf.py000066400000000000000000000030211444262304300162730ustar00rootroot00000000000000import datetime import os.path as osp import sys import pyg_sphinx_theme import torch_geometric author = 'PyG Team' project = 'pytorch_geometric' version = torch_geometric.__version__ copyright = f'{datetime.datetime.now().year}, {author}' sys.path.append(osp.join(osp.dirname(pyg_sphinx_theme.__file__), 'extension')) extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.autosummary', 'sphinx.ext.intersphinx', 'sphinx.ext.mathjax', 'sphinx.ext.napoleon', 'sphinx.ext.viewcode', 'pyg', ] html_theme = 'pyg_sphinx_theme' html_logo = ('https://raw.githubusercontent.com/pyg-team/pyg_sphinx_theme/' 'master/pyg_sphinx_theme/static/img/pyg_logo.png') html_favicon = ('https://raw.githubusercontent.com/pyg-team/pyg_sphinx_theme/' 'master/pyg_sphinx_theme/static/img/favicon.png') html_static_path = ['_static'] templates_path = ['_templates'] add_module_names = False autodoc_member_order = 'bysource' suppress_warnings = ['autodoc.import_object'] intersphinx_mapping = { 'python': ('https://docs.python.org/', None), 'numpy': ('http://docs.scipy.org/doc/numpy', None), 'pandas': ('http://pandas.pydata.org/pandas-docs/dev', None), 'torch': ('https://pytorch.org/docs/master', None), } def setup(app): def rst_jinja_render(app, _, source): rst_context = {'torch_geometric': torch_geometric} source[0] = app.builder.templates.render_string(source[0], rst_context) app.connect('source-read', rst_jinja_render) app.add_js_file('js/version_alert.js') pytorch/docs/source/external/000077500000000000000000000000001444262304300166225ustar00rootroot00000000000000pytorch/docs/source/external/resources.rst000066400000000000000000000115651444262304300213760ustar00rootroot00000000000000External Resources ================== * Matthias Fey and Jan E. Lenssen: **Fast Graph Representation Learning with** :pyg:`null` **PyTorch Geometric** [`Paper `_, `Slides (3.3MB) `__, `Poster (2.3MB) `__, `Notebook `__] * :stanford:`Stanford CS224W: Machine Learning with Graphs`: **Graph Machine Learning lectures** [:youtube:`null` `Youtube `__] * :stanford:`Stanford University`: **A collection of graph machine learning tutorial blog posts**, fully realized with :pyg:`null` **PyG** [`Website `__] * Soumith Chintala: **Automatic Differentiation,** :pytorch:`null` **PyTorch and Graph Neural Networks** [`Talk (starting from 26:15) `__] * Stanford University: **Graph Neural Networks using** :pyg:`null` **PyTorch Geometric** [:youtube:`null` `YouTube (starting from 33:33) `__] * Antonio Longa, Gabriele Santin and Giovanni Pellegrini: :pyg:`null` **PyTorch Geometric Tutorial** [`Website `__, :github:`null` `GitHub `__] * DAIR.AI | elvis: **Introduction to GNNs with** :pyg:`null` **PyTorch Geometric** [`Website `__, :colab:`null` `Colab `__] * Nicolas Chaulet *et al.*: **PyTorch Points 3D** - A framework for running common deep learning models for point cloud analysis tasks that heavily relies on Pytorch Geometric [:github:`null` `GitHub `__, `Documentation `__] * Weihua Hu *et al.*: :ogb:`null` **Open Graph Benchmark** - A collection of large-scale benchmark datasets, data loaders, and evaluators for graph machine learning, including :pyg:`PyG` support and examples [`Website `__, :github:`null` `GitHub `__] * **DeepSNAP** - A :pytorch:`PyTorch` library that bridges between graph libraries such as NetworkX and :pyg:`PyG` [:github:`null` `GitHub `__, `Documentation `__] * **Quiver** - A distributed graph learning library for :pyg:`PyG` [:github:`null` `GitHub `__] * Benedek Rozemberczki: **PyTorch Geometric Temporal** - A temporal GNN library built upon :pyg:`PyG` [:github:`null` `GitHub `__, `Documentation `__] * Yixuan He: **PyTorch Geometric Signed Directed** - A signed and directed GNN library built upon :pyg:`PyG` [:github:`null` `GitHub `__, `Documentation `__] * Steeve Huang: **Hands-on Graph Neural Networks with** :pytorch:`null` **PyTorch &** :pyg:`null` **PyTorch Geometric** [`Tutorial `__, `Code `__] * Francesco Landolfi: :pyg:`null` **PyTorch Geometric Tutorial** [`PDF (0.4MB) `__] * Sachin Sharma: **How to Deploy (almost) any** :pyg:`null` **PyTorch Geometric Model on Nvidia's Triton Inference Server with an Application to Amazon Product Recommendation and ArangoDB** [`Blog `__] * Amitoz Azad: **torch_pdegraph** - Solving PDEs on Graphs with :pyg:`PyG` [`Devpost `__, :github:`null` `GitHub `__] * Amitoz Azad: **Primal-Dual Algorithm for Total Variation Processing on Graphs** [`Jupyter `__] * Manan Goel: **Recommending Amazon Products using Graph Neural Networks in** :pyg:`null` **PyTorch Geometric** [:wandb:`null` `W&B Report `__] * KΓΉzu: **Remote Backend for** :pyg:`null` **PyTorch Geometric** [:colab:`null` `Colab `__] pytorch/docs/source/get_started/000077500000000000000000000000001444262304300173055ustar00rootroot00000000000000pytorch/docs/source/get_started/colabs.rst000066400000000000000000000164361444262304300213140ustar00rootroot00000000000000Colab Notebooks and Video Tutorials =================================== Official Examples ----------------- We have prepared a list of :colab:`Colab` notebooks that practically introduces you to the world of **Graph Neural Networks** with :pyg:`PyG`: 1. `Introduction: Hands-on Graph Neural Networks `__ 2. `Node Classification with Graph Neural Networks `__ 3. `Graph Classification with Graph Neural Networks `__ 4. `Scaling Graph Neural Networks `__ 5. `Point Cloud Classification with Graph Neural Networks `__ 6. `Explaining GNN Model Predictions using `__ :captum:`null` `Captum `__ 7. `Customizing Aggregations within Message Passing `__ 8. `Node Classification Instrumented with `__ :wandb:`null` `Weights&Biases `__ 9. `Graph Classification Instrumented with `__ :wandb:`null` `Weights&Biases `__ 10. `Link Prediction on MovieLens `__ 11. `Link Regression on MovieLens `__ All :colab:`Colab` notebooks are released under the MIT license. Stanford CS224W Tutorials ------------------------- .. image:: https://data.pyg.org/img/cs224w_tutorials.png :align: center :width: 941px :target: https://medium.com/stanford-cs224w .. raw:: html
The :stanford:`null` `Stanford CS224W `__ course has collected a set of `graph machine learning tutorial blog posts `__, fully realized with :pyg:`PyG`. Students worked on projects spanning all kinds of tasks, model architectures and applications. All tutorials also link to a :colab:`Colab` with the code in the tutorial for you to follow along with as you read it! PyTorch Geometric Tutorial Project ---------------------------------- The :pyg:`null` `PyTorch Geometric Tutorial `__ project provides **video tutorials and** :colab:`null` **Colab notebooks** for a variety of different methods in :pyg:`PyG`: 1. Introduction [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 2. :pytorch:`PyTorch` basics [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 3. Graph Attention Networks (GATs) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 4. Spectral Graph Convolutional Layers [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 5. Aggregation Functions in GNNs [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 6. (Variational) Graph Autoencoders (GAE and VGAE) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 7. Adversarially Regularized Graph Autoencoders (ARGA and ARGVA) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 8. Graph Generation [:youtube:`null` `YouTube `__] 9. Recurrent Graph Neural Networks [:youtube:`null` `YouTube `__, :colab:`null` `Colab (Part 1) `__, :colab:`null` `Colab (Part 2) `__] 10. DeepWalk and Node2Vec [:youtube:`null` `YouTube (Theory) `__, :youtube:`null` `YouTube (Practice) `__, :colab:`null` `Colab `__] 11. Edge analysis [:youtube:`null` `YouTube `__, :colab:`null` `Colab (Link Prediction) `__, :colab:`null` `Colab (Label Prediction) `__] 12. Data handling in :pyg:`PyG` (Part 1) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 13. Data handling in :pyg:`PyG` (Part 2) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 14. MetaPath2vec [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] 15. Graph pooling (DiffPool) [:youtube:`null` `YouTube `__, :colab:`null` `Colab `__] pytorch/docs/source/get_started/introduction.rst000066400000000000000000000403731444262304300225670ustar00rootroot00000000000000Introduction by Example ======================= We shortly introduce the fundamental concepts of :pyg:`PyG` through self-contained examples. For an introduction to Graph Machine Learning, we refer the interested reader to the :stanford:`null` `Stanford CS224W: Machine Learning with Graphs `__ lectures. For an interactive introduction to :pyg:`PyG`, we recommend our carefully curated :colab:`null` `Google Colab `__ notebooks. At its core, :pyg:`PyG` provides the following main features: .. contents:: :local: Data Handling of Graphs ----------------------- A graph is used to model pairwise relations (edges) between objects (nodes). A single graph in :pyg:`PyG` is described by an instance of :class:`torch_geometric.data.Data`, which holds the following attributes by default: - :obj:`data.x`: Node feature matrix with shape :obj:`[num_nodes, num_node_features]` - :obj:`data.edge_index`: Graph connectivity in `COO format `_ with shape :obj:`[2, num_edges]` and type :obj:`torch.long` - :obj:`data.edge_attr`: Edge feature matrix with shape :obj:`[num_edges, num_edge_features]` - :obj:`data.y`: Target to train against (may have arbitrary shape), *e.g.*, node-level targets of shape :obj:`[num_nodes, *]` or graph-level targets of shape :obj:`[1, *]` - :obj:`data.pos`: Node position matrix with shape :obj:`[num_nodes, num_dimensions]` None of these attributes are required. In fact, the :class:`~torch_geometric.data.Data` object is not even restricted to these attributes. We can, *e.g.*, extend it by :obj:`data.face` to save the connectivity of triangles from a 3D mesh in a tensor with shape :obj:`[3, num_faces]` and type :obj:`torch.long`. .. Note:: :pytorch:`PyTorch` and :obj:`torchvision` define an example as a tuple of an image and a target. We omit this notation in :pyg:`PyG` to allow for various data structures in a clean and understandable way. We show a simple example of an unweighted and undirected graph with three nodes and four edges. Each node contains exactly one feature: .. code-block:: python import torch from torch_geometric.data import Data edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], dtype=torch.long) x = torch.tensor([[-1], [0], [1]], dtype=torch.float) data = Data(x=x, edge_index=edge_index) >>> Data(edge_index=[2, 4], x=[3, 1]) .. image:: ../_figures/graph.svg :align: center :width: 300px | Note that :obj:`edge_index`, *i.e.* the tensor defining the source and target nodes of all edges, is **not** a list of index tuples. If you want to write your indices this way, you should transpose and call :obj:`contiguous` on it before passing them to the data constructor: .. code-block:: python import torch from torch_geometric.data import Data edge_index = torch.tensor([[0, 1], [1, 0], [1, 2], [2, 1]], dtype=torch.long) x = torch.tensor([[-1], [0], [1]], dtype=torch.float) data = Data(x=x, edge_index=edge_index.t().contiguous()) >>> Data(edge_index=[2, 4], x=[3, 1]) Although the graph has only two edges, we need to define four index tuples to account for both directions of a edge. .. Note:: You can print out your data object anytime and receive a short information about its attributes and their shapes. Note that it is necessary that the elements in :obj:`edge_index` only hold indices in the range :obj:`{ 0, ..., num_nodes - 1}`. This is needed as we want our final data representation to be as compact as possible, *e.g.*, we want to index the source and destination node features of the first edge :obj:`(0, 1)` via :obj:`x[0]` and :obj:`x[1]`, respectively. You can always check that your final :class:`~torch_geometric.data.Data` objects fulfill these requirements by running :meth:`~torch_geometric.data.Data.validate`: .. code-block:: python data.validate(raise_on_error=True) Besides holding a number of node-level, edge-level or graph-level attributes, :class:`~torch_geometric.data.Data` provides a number of useful utility functions, *e.g.*: .. code-block:: python print(data.keys) >>> ['x', 'edge_index'] print(data['x']) >>> tensor([[-1.0], [0.0], [1.0]]) for key, item in data: print(f'{key} found in data') >>> x found in data >>> edge_index found in data 'edge_attr' in data >>> False data.num_nodes >>> 3 data.num_edges >>> 4 data.num_node_features >>> 1 data.has_isolated_nodes() >>> False data.has_self_loops() >>> False data.is_directed() >>> False # Transfer data object to GPU. device = torch.device('cuda') data = data.to(device) You can find a complete list of all methods at :class:`torch_geometric.data.Data`. Common Benchmark Datasets ------------------------- :pyg:`PyG` contains a large number of common benchmark datasets, *e.g.*, all Planetoid datasets (Cora, Citeseer, Pubmed), all graph classification datasets from `http://graphkernels.cs.tu-dortmund.de `_ and their `cleaned versions `_, the QM7 and QM9 dataset, and a handful of 3D mesh/point cloud datasets like FAUST, ModelNet10/40 and ShapeNet. Initializing a dataset is straightforward. An initialization of a dataset will automatically download its raw files and process them to the previously described :class:`~torch_geometric.data.Data` format. *E.g.*, to load the ENZYMES dataset (consisting of 600 graphs within 6 classes), type: .. code-block:: python from torch_geometric.datasets import TUDataset dataset = TUDataset(root='/tmp/ENZYMES', name='ENZYMES') >>> ENZYMES(600) len(dataset) >>> 600 dataset.num_classes >>> 6 dataset.num_node_features >>> 3 We now have access to all 600 graphs in the dataset: .. code-block:: python data = dataset[0] >>> Data(edge_index=[2, 168], x=[37, 3], y=[1]) data.is_undirected() >>> True We can see that the first graph in the dataset contains 37 nodes, each one having 3 features. There are 168/2 = 84 undirected edges and the graph is assigned to exactly one class. In addition, the data object is holding exactly one graph-level target. We can even use slices, long or bool tensors to split the dataset. *E.g.*, to create a 90/10 train/test split, type: .. code-block:: python train_dataset = dataset[:540] >>> ENZYMES(540) test_dataset = dataset[540:] >>> ENZYMES(60) If you are unsure whether the dataset is already shuffled before you split, you can randomly permutate it by running: .. code-block:: python dataset = dataset.shuffle() >>> ENZYMES(600) This is equivalent of doing: .. code-block:: python perm = torch.randperm(len(dataset)) dataset = dataset[perm] >> ENZYMES(600) Let's try another one! Let's download Cora, the standard benchmark dataset for semi-supervised graph node classification: .. code-block:: python from torch_geometric.datasets import Planetoid dataset = Planetoid(root='/tmp/Cora', name='Cora') >>> Cora() len(dataset) >>> 1 dataset.num_classes >>> 7 dataset.num_node_features >>> 1433 Here, the dataset contains only a single, undirected citation graph: .. code-block:: python data = dataset[0] >>> Data(edge_index=[2, 10556], test_mask=[2708], train_mask=[2708], val_mask=[2708], x=[2708, 1433], y=[2708]) data.is_undirected() >>> True data.train_mask.sum().item() >>> 140 data.val_mask.sum().item() >>> 500 data.test_mask.sum().item() >>> 1000 This time, the :class:`~torch_geometric.data.Data` objects holds a label for each node, and additional node-level attributes: :obj:`train_mask`, :obj:`val_mask` and :obj:`test_mask`, where - :obj:`train_mask` denotes against which nodes to train (140 nodes), - :obj:`val_mask` denotes which nodes to use for validation, *e.g.*, to perform early stopping (500 nodes), - :obj:`test_mask` denotes against which nodes to test (1000 nodes). Mini-batches ------------ Neural networks are usually trained in a batch-wise fashion. :pyg:`PyG` achieves parallelization over a mini-batch by creating sparse block diagonal adjacency matrices (defined by :obj:`edge_index`) and concatenating feature and target matrices in the node dimension. This composition allows differing number of nodes and edges over examples in one batch: .. math:: \mathbf{A} = \begin{bmatrix} \mathbf{A}_1 & & \\ & \ddots & \\ & & \mathbf{A}_n \end{bmatrix}, \qquad \mathbf{X} = \begin{bmatrix} \mathbf{X}_1 \\ \vdots \\ \mathbf{X}_n \end{bmatrix}, \qquad \mathbf{Y} = \begin{bmatrix} \mathbf{Y}_1 \\ \vdots \\ \mathbf{Y}_n \end{bmatrix} :pyg:`PyG` contains its own :class:`torch_geometric.loader.DataLoader`, which already takes care of this concatenation process. Let's learn about it in an example: .. code-block:: python from torch_geometric.datasets import TUDataset from torch_geometric.loader import DataLoader dataset = TUDataset(root='/tmp/ENZYMES', name='ENZYMES', use_node_attr=True) loader = DataLoader(dataset, batch_size=32, shuffle=True) for batch in loader: batch >>> DataBatch(batch=[1082], edge_index=[2, 4066], x=[1082, 21], y=[32]) batch.num_graphs >>> 32 :class:`torch_geometric.data.Batch` inherits from :class:`torch_geometric.data.Data` and contains an additional attribute called :obj:`batch`. :obj:`batch` is a column vector which maps each node to its respective graph in the batch: .. math:: \mathrm{batch} = {\begin{bmatrix} 0 & \cdots & 0 & 1 & \cdots & n - 2 & n -1 & \cdots & n - 1 \end{bmatrix}}^{\top} You can use it to, *e.g.*, average node features in the node dimension for each graph individually: .. code-block:: python from torch_geometric.utils import scatter from torch_geometric.datasets import TUDataset from torch_geometric.loader import DataLoader dataset = TUDataset(root='/tmp/ENZYMES', name='ENZYMES', use_node_attr=True) loader = DataLoader(dataset, batch_size=32, shuffle=True) for data in loader: data >>> DataBatch(batch=[1082], edge_index=[2, 4066], x=[1082, 21], y=[32]) data.num_graphs >>> 32 x = scatter(data.x, data.batch, dim=0, reduce='mean') x.size() >>> torch.Size([32, 21]) You can learn more about the internal batching procedure of :pyg:`PyG`, *e.g.*, how to modify its behavior, `here <../advanced/batching.html>`__. For documentation of scatter operations, we refer the interested reader to the :obj:`torch_scatter` `documentation `_. Data Transforms --------------- Transforms are a common way in :obj:`torchvision` to transform images and perform augmentation. :pyg:`PyG` comes with its own transforms, which expect a :class:`~torch_geometric.data.Data` object as input and return a new transformed :class:`~torch_geometric.data.Data` object. Transforms can be chained together using :class:`torch_geometric.transforms.Compose` and are applied before saving a processed dataset on disk (:obj:`pre_transform`) or before accessing a graph in a dataset (:obj:`transform`). Let's look at an example, where we apply transforms on the ShapeNet dataset (containing 17,000 3D shape point clouds and per point labels from 16 shape categories). .. code-block:: python from torch_geometric.datasets import ShapeNet dataset = ShapeNet(root='/tmp/ShapeNet', categories=['Airplane']) dataset[0] >>> Data(pos=[2518, 3], y=[2518]) We can convert the point cloud dataset into a graph dataset by generating nearest neighbor graphs from the point clouds via transforms: .. code-block:: python import torch_geometric.transforms as T from torch_geometric.datasets import ShapeNet dataset = ShapeNet(root='/tmp/ShapeNet', categories=['Airplane'], pre_transform=T.KNNGraph(k=6)) dataset[0] >>> Data(edge_index=[2, 15108], pos=[2518, 3], y=[2518]) .. note:: We use the :obj:`pre_transform` to convert the data before saving it to disk (leading to faster loading times). Note that the next time the dataset is initialized it will already contain graph edges, even if you do not pass any transform. If the :obj:`pre_transform` does not match with the one from the already processed dataset, you will be given a warning. In addition, we can use the :obj:`transform` argument to randomly augment a :class:`~torch_geometric.data.Data` object, *e.g.*, translating each node position by a small number: .. code-block:: python import torch_geometric.transforms as T from torch_geometric.datasets import ShapeNet dataset = ShapeNet(root='/tmp/ShapeNet', categories=['Airplane'], pre_transform=T.KNNGraph(k=6), transform=T.RandomJitter(0.01)) dataset[0] >>> Data(edge_index=[2, 15108], pos=[2518, 3], y=[2518]) You can find a complete list of all implemented transforms at :mod:`torch_geometric.transforms`. Learning Methods on Graphs -------------------------- After learning about data handling, datasets, loader and transforms in :pyg:`PyG`, it's time to implement our first graph neural network! We will use a simple GCN layer and replicate the experiments on the Cora citation dataset. For a high-level explanation on GCN, have a look at its `blog post `_. We first need to load the Cora dataset: .. code-block:: python from torch_geometric.datasets import Planetoid dataset = Planetoid(root='/tmp/Cora', name='Cora') >>> Cora() Note that we do not need to use transforms or a dataloader. Now let's implement a two-layer GCN: .. code-block:: python import torch import torch.nn.functional as F from torch_geometric.nn import GCNConv class GCN(torch.nn.Module): def __init__(self): super().__init__() self.conv1 = GCNConv(dataset.num_node_features, 16) self.conv2 = GCNConv(16, dataset.num_classes) def forward(self, data): x, edge_index = data.x, data.edge_index x = self.conv1(x, edge_index) x = F.relu(x) x = F.dropout(x, training=self.training) x = self.conv2(x, edge_index) return F.log_softmax(x, dim=1) The constructor defines two :class:`~torch_geometric.nn.conv.GCNConv` layers which get called in the forward pass of our network. Note that the non-linearity is not integrated in the :obj:`conv` calls and hence needs to be applied afterwards (something which is consistent across all operators in :pyg:`PyG`). Here, we chose to use ReLU as our intermediate non-linearity and finally output a softmax distribution over the number of classes. Let's train this model on the training nodes for 200 epochs: .. code-block:: python device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = GCN().to(device) data = dataset[0].to(device) optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4) model.train() for epoch in range(200): optimizer.zero_grad() out = model(data) loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask]) loss.backward() optimizer.step() Finally, we can evaluate our model on the test nodes: .. code-block:: python model.eval() pred = model(data).argmax(dim=1) correct = (pred[data.test_mask] == data.y[data.test_mask]).sum() acc = int(correct) / int(data.test_mask.sum()) print(f'Accuracy: {acc:.4f}') >>> Accuracy: 0.8150 This is all it takes to implement your first graph neural network. The easiest way to learn more about Graph Neural Networks is to study the examples in the :obj:`examples/` directory and to browse :mod:`torch_geometric.nn`. Happy hacking! Exercises --------- 1. What does :obj:`edge_index.t().contiguous()` do? 2. Load the :obj:`"IMDB-BINARY"` dataset from the :class:`~torch_geometric.datasets.TUDataset` benchmark suite and randomly split it into 80%/10%/10% training, validation and test graphs. 3. What does each number of the following output mean? .. code-block:: python print(batch) >>> DataBatch(batch=[1082], edge_index=[2, 4066], x=[1082, 21], y=[32]) pytorch/docs/source/index.rst000066400000000000000000000043431444262304300166450ustar00rootroot00000000000000:github_url: https://github.com/pyg-team/pytorch_geometric PyG Documentation ================= :pyg:`null` **PyG** *(PyTorch Geometric)* is a library built upon :pytorch:`null` `PyTorch `_ to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data. It consists of various methods for deep learning on graphs and other irregular structures, also known as `geometric deep learning `_, from a variety of published papers. In addition, it consists of easy-to-use mini-batch loaders for operating on many small and single giant graphs, `multi GPU-support `_, `torch.compile `_ support, `DataPipe `_ support, a large number of common benchmark datasets (based on simple interfaces to create your own), the `GraphGym `__ experiment manager, and helpful transforms, both for learning on arbitrary graphs as well as on 3D meshes or point clouds. .. slack_button:: .. toctree:: :maxdepth: 1 :caption: Install PyG install/installation .. toctree:: :maxdepth: 1 :caption: Get Started get_started/introduction get_started/colabs .. toctree:: :maxdepth: 1 :caption: Tutorials tutorial/create_gnn tutorial/create_dataset tutorial/heterogeneous tutorial/load_csv tutorial/explain tutorial/compile .. toctree:: :maxdepth: 1 :caption: Advanced Concepts advanced/batching advanced/sparse_tensor advanced/jit advanced/remote advanced/graphgym advanced/cpu_affinity .. toctree:: :maxdepth: 1 :caption: Package Reference modules/root modules/nn modules/data modules/loader modules/sampler modules/datasets modules/transforms modules/utils modules/explain modules/contrib modules/graphgym modules/profile .. toctree:: :maxdepth: 1 :caption: Cheatsheets cheatsheet/gnn_cheatsheet cheatsheet/data_cheatsheet .. toctree:: :maxdepth: 1 :caption: External Resources external/resources pytorch/docs/source/install/000077500000000000000000000000001444262304300164465ustar00rootroot00000000000000pytorch/docs/source/install/installation.rst000066400000000000000000000210341444262304300217010ustar00rootroot00000000000000Installation ============ :pyg:`PyG` is available for Python 3.7 to Python 3.11. .. note:: We do not recommend installation as a root user on your system Python. Please setup a virtual environment, *e.g.*, via `Anaconda or Miniconda `_, or create a `Docker image `_. Quick Start ----------- .. raw:: html :file: quick-start.html Installation via Anaconda ------------------------- You can now install :pyg:`PyG` via `Anaconda `_ for all major OS, :pytorch:`PyTorch` and CUDA combinations πŸ€— If you have not yet installed :pytorch:`PyTorch`, install it via :obj:`conda` as described in its `official documentation `_. Given that you have :pytorch:`PyTorch` installed (:obj:`>=1.12.0`), simply run .. code-block:: none conda install pyg -c pyg .. warning:: Conda packages are currently not available for Windows and M1/M2/M3 macs. Installation via PyPi --------------------- From :pyg:`null` **PyG 2.3** onwards, you can install and use :pyg:`PyG` **without any external library** required except for :pytorch:`PyTorch`. For this, simply run: .. code-block:: none pip install torch_geometric PyG 2.3 requires that at least PyTorch 1.12 is installed. Additional Libraries -------------------- If you want to utilize the full set of features from :pyg:`PyG`, there exists several additional libraries you may want to install: * `pyg-lib `__: Heterogeneous GNN operators and graph sampling routines * `torch-scatter `__: Accelerated and efficient sparse reductions * `torch-sparse `__: :class:`SparseTensor` support, see `here `__ * `torch-cluster `__: Graph clustering routines * `torch-spline-conv `__: :class:`~torch_geometric.nn.conv.SplineConv` support These packages come with their own CPU and GPU kernel implementations based on the :pytorch:`null` `PyTorch C++/CUDA/hip(ROCm) extension interface `_. For a basic usage of :pyg:`PyG`, these dependencies are **fully optional**. We recommend to start with a minimal installation, and install additional dependencies once you start to actually need them. Installation from Wheels ~~~~~~~~~~~~~~~~~~~~~~~~ For ease of installation of these extensions, we provide :obj:`pip` wheels for these packages for all major OS, :pytorch:`PyTorch` and CUDA combinations, see `here `__: .. warning:: Wheels are currently not available for M1/M2/M3 macs. Please install the extension packages `from source `__. #. Ensure that at least :pytorch:`PyTorch` 1.12.0 is installed: .. code-block:: none python -c "import torch; print(torch.__version__)" >>> 2.0.0 #. Find the CUDA version :pytorch:`PyTorch` was installed with: .. code-block:: none python -c "import torch; print(torch.version.cuda)" >>> 11.7 #. Install the relevant packages: .. code-block:: none pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-${TORCH}+${CUDA}.html where :obj:`${TORCH}` and :obj:`${CUDA}` should be replaced by the specific :pytorch:`PyTorch` and CUDA versions, respectively: * :pytorch:`PyTorch` 2.0: :obj:`${TORCH}=2.0.0` and :obj:`${CUDA}=cpu|cu117|cu118` * :pytorch:`PyTorch` 1.13: :obj:`${TORCH}=1.13.0` and :obj:`${CUDA}=cpu|cu116|cu117` For example, for :pytorch:`PyTorch` 2.0.* and CUDA 11.7, type: .. code-block:: none pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.0.0+cu117.html For :pytorch:`PyTorch` 1.13.* and CUDA 11.6, type: .. code-block:: none pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-1.13.0+cu116.html **Note:** Binaries of older versions are also provided for :pytorch:`PyTorch` 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0 and 1.12.1 (following the same procedure). **For older versions, you need to explicitly specify the latest supported version number** or install via :obj:`pip install --no-index` in order to prevent a manual installation from source. You can look up the latest supported version number `here `__. **ROCm:** The external `pyg-rocm-build repository `__ provides wheels and detailed instructions on how to install :pyg:`PyG` for ROCm. If you have any questions about it, please open an issue `here `__. Installation from Source ~~~~~~~~~~~~~~~~~~~~~~~~ In case a specific version is not supported by `our wheels `_, you can alternatively install them from source: #. Ensure that your CUDA is setup correctly (optional): #. Check if :pytorch:`PyTorch` is installed with CUDA support: .. code-block:: none python -c "import torch; print(torch.cuda.is_available())" >>> True #. Add CUDA to :obj:`$PATH` and :obj:`$CPATH` (note that your actual CUDA path may vary from :obj:`/usr/local/cuda`): .. code-block:: none export PATH=/usr/local/cuda/bin:$PATH echo $PATH >>> /usr/local/cuda/bin:... export CPATH=/usr/local/cuda/include:$CPATH echo $CPATH >>> /usr/local/cuda/include:... #. Add CUDA to :obj:`$LD_LIBRARY_PATH` on Linux and to :obj:`$DYLD_LIBRARY_PATH` on macOS (note that your actual CUDA path may vary from :obj:`/usr/local/cuda`): .. code-block:: none export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH echo $LD_LIBRARY_PATH >>> /usr/local/cuda/lib64:... export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH echo $DYLD_LIBRARY_PATH >>> /usr/local/cuda/lib:... #. Verify that :obj:`nvcc` is accessible from terminal: .. code-block:: none nvcc --version >>> 11.6 #. Ensure that :pytorch:`PyTorch` and system CUDA versions match: .. code-block:: none python -c "import torch; print(torch.version.cuda)" >>> 11.6 nvcc --version >>> 11.6 #. Install the relevant packages: .. code-block:: none pip install --verbose git+https://github.com/pyg-team/pyg-lib.git pip install --verbose torch_scatter pip install --verbose torch_sparse pip install --verbose torch_cluster pip install --verbose torch_spline_conv In rare cases, CUDA or Python path problems can prevent a successful installation. :obj:`pip` may even signal a successful installation, but execution simply crashes with :obj:`Segmentation fault (core dumped)`. We collected common installation errors in the `Frequently Asked Questions `__ subsection. In case the FAQ does not help you in solving your problem, please create an `issue `_. Before, please verify that your CUDA is set up correctly by following the official `installation guide `_. Frequently Asked Questions -------------------------- #. :obj:`undefined symbol: **make_function_schema**`: This issue signals (1) a **version conflict** between your installed :pytorch:`PyTorch` version and the :obj:`${TORCH}` version specified to install the extension packages, or (2) a version conflict between the installed CUDA version of :pytorch:`PyTorch` and the :obj:`${CUDA}` version specified to install the extension packages. Please verify that your :pytorch:`PyTorch` version and its CUDA version **match** with your installation command: .. code-block:: none python -c "import torch; print(torch.__version__)" python -c "import torch; print(torch.version.cuda)" nvcc --version For re-installation, ensure that you do not run into any caching issues by using the :obj:`pip --force-reinstall --no-cache-dir` flags. In addition, the :obj:`pip --verbose` option may help to track down any issues during installation. If you still do not find any success in installation, please try to install the extension packages `from source `__. pytorch/docs/source/install/quick-start.html000066400000000000000000000073041444262304300216070ustar00rootroot00000000000000
PyTorch
Your OS
Package
CUDA
Run:

  





pytorch/docs/source/modules/000077500000000000000000000000001444262304300164505ustar00rootroot00000000000000pytorch/docs/source/modules/contrib.rst000066400000000000000000000040521444262304300206430ustar00rootroot00000000000000torch_geometric.contrib
=======================

.. currentmodule:: torch_geometric.contrib

:obj:`torch_geometric.contrib` is a staging area for early stage experimental code.
Modules might be moved to the main library in the future.

.. warning::

    This module contains experimental code, which is not guaranteed to be stable.

.. contents:: Contents
    :local:

Convolutional Layers
--------------------

.. currentmodule:: torch_geometric.contrib.nn.conv

.. autosummary::
   :nosignatures:
   {% for cls in torch_geometric.contrib.nn.conv.classes %}
     {{ cls }}
   {% endfor %}

.. automodule:: torch_geometric.contrib.nn.conv
   :members:
   :undoc-members:
   :exclude-members: message, aggregate, message_and_aggregate, update, MessagePassing, training, initialize_parameters

Models
------

.. currentmodule:: torch_geometric.contrib.nn.models

.. autosummary::
   :nosignatures:
   {% for cls in torch_geometric.contrib.nn.models.classes %}
     {{ cls }}
   {% endfor %}

.. automodule:: torch_geometric.contrib.nn.models
   :members:
   :undoc-members:
   :exclude-members: message, aggregate, message_and_aggregate, update, MessagePassing, training, init_conv

Datasets
--------

.. currentmodule:: torch_geometric.contrib.datasets

.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.contrib.datasets.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.contrib.datasets
    :members:
    :exclude-members: download, process, processed_file_names, raw_file_names, num_classes, get

Transforms
----------

.. currentmodule:: torch_geometric.contrib.transforms

.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.contrib.transforms.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.contrib.transforms
    :members:


Explainer
---------

.. currentmodule:: torch_geometric.contrib.explain

.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.contrib.explain.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.contrib.explain
    :members:
pytorch/docs/source/modules/data.rst000066400000000000000000000021501444262304300201110ustar00rootroot00000000000000torch_geometric.data
====================

.. contents:: Contents
    :local:

Data Objects
------------

.. currentmodule:: torch_geometric.data

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/inherited_class.rst

   {% for name in torch_geometric.data.data_classes %}
     {{ name }}
   {% endfor %}

Remote Backend Interfaces
-------------------------

.. currentmodule:: torch_geometric.data

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.data.remote_backend_classes %}
     {{ name }}
   {% endfor %}

PyTorch Lightning Wrappers
--------------------------

.. currentmodule:: torch_geometric.data.lightning

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.data.lightning.classes %}
     {{ name }}
   {% endfor %}

Helper Functions
----------------

.. currentmodule:: torch_geometric.data

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.data.helper_functions %}
     {{ name }}
   {% endfor %}
pytorch/docs/source/modules/datasets.rst000066400000000000000000000030151444262304300210110ustar00rootroot00000000000000torch_geometric.datasets
========================

.. contents:: Contents
    :local:

Homogeneous Datasets
--------------------

.. currentmodule:: torch_geometric.datasets

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.datasets.homo_datasets %}
     {{ name }}
   {% endfor %}

Heterogeneous Datasets
----------------------

.. currentmodule:: torch_geometric.datasets

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.datasets.hetero_datasets %}
     {{ name }}
   {% endfor %}

Synthetic Datasets
------------------

.. currentmodule:: torch_geometric.datasets

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.datasets.synthetic_datasets %}
     {{ name }}
   {% endfor %}

Graph Generators
----------------

.. currentmodule:: torch_geometric.datasets.graph_generator

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.datasets.graph_generator.classes %}
     {{ name }}
   {% endfor %}

Motif Generators
----------------

.. currentmodule:: torch_geometric.datasets.motif_generator

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.datasets.motif_generator.classes %}
     {{ name }}
   {% endfor %}
pytorch/docs/source/modules/explain.rst000066400000000000000000000055021444262304300206440ustar00rootroot00000000000000torch_geometric.explain
=======================

.. currentmodule:: torch_geometric.explain

.. warning::

    This module is in active development and may not be stable.
    Access requires installing :pyg:`PyG` from master.

.. contents:: Contents
    :local:

Philoshopy
----------

This module provides a set of tools to explain the predictions of a PyG model or to explain the underlying phenomenon of a dataset (see the `"GraphFramEx: Towards Systematic Evaluation of Explainability Methods for Graph Neural Networks" `_ paper for more details).

We represent explanations using the :class:`torch_geometric.explain.Explanation` class, which is a :class:`~torch_geometric.data.Data` object containing masks for the nodes, edges, features and any attributes of the data.

The :class:`torch_geometric.explain.Explainer` class is designed to handle all explainability parameters (see the :class:`torch_geometric.explain.config.ExplainerConfig` class for more details):

- which algorithm from the :class:`torch_geometric.explain.algorithm` module to use (*e.g.*, :class:`~torch_geometric.explain.algorithm.GNNExplainer`)
- the type of explanation to compute (*e.g.*, :obj:`explanation_type="phenomenon"` or :obj:`explanation_type="model"`)
- the different type of masks for node and edges (*e.g.*, :obj:`mask="object"` or :obj:`mask="attributes"`)
- any postprocessing of the masks (*e.g.*, :obj:`threshold_type="topk"` or :obj:`threshold_type="hard"`)

This class allows the user to easily compare different explainability methods and to easily switch between different types of masks, while making sure the high-level framework stays the same.

Explainer
---------

.. autoclass:: torch_geometric.explain.Explainer
   :show-inheritance:
   :members:
   :special-members: __call__

.. autoclass:: torch_geometric.explain.config.ExplainerConfig
   :members:

.. autoclass:: torch_geometric.explain.config.ModelConfig
   :members:

.. autoclass:: torch_geometric.explain.config.ThresholdConfig
   :members:

Explanations
------------

.. autoclass:: torch_geometric.explain.Explanation
   :show-inheritance:
   :members:

.. autoclass:: torch_geometric.explain.HeteroExplanation
   :show-inheritance:
   :members:

Explainer Algorithms
--------------------

.. currentmodule:: torch_geometric.explain.algorithm

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.explain.algorithm.classes %}
     {{ name }}
   {% endfor %}

Explanation Metrics
-------------------

The quality of an explanation can be judged by a variety of different methods.
PyG supports the following metrics out-of-the-box:

.. currentmodule:: torch_geometric.explain.metric

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.explain.metric.classes %}
     {{ name }}
   {% endfor %}
pytorch/docs/source/modules/graphgym.rst000066400000000000000000000017321444262304300210230ustar00rootroot00000000000000torch_geometric.graphgym
========================

.. contents:: Contents
    :local:

Workflow and Register Modules
-----------------------------

.. currentmodule:: torch_geometric.graphgym
.. autosummary::
   :nosignatures:
   {% for cls in torch_geometric.graphgym.classes %}
     {{ cls }}
   {% endfor %}

.. automodule:: torch_geometric.graphgym
   :members:
   :exclude-members:

Model Modules
-------------

.. currentmodule:: torch_geometric.graphgym.models
.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.graphgym.models.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.graphgym.models
    :members:
    :exclude-members: forward

Utility Modules
---------------

.. currentmodule:: torch_geometric.graphgym.utils
.. autosummary::
   :nosignatures:
    {% for cls in torch_geometric.graphgym.utils.classes %}
     {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.graphgym.utils
    :members:
    :exclude-members:
pytorch/docs/source/modules/loader.rst000066400000000000000000000004051444262304300204470ustar00rootroot00000000000000torch_geometric.loader
======================

.. currentmodule:: torch_geometric.loader

.. autosummary::
   :nosignatures:
   {% for cls in torch_geometric.loader.classes %}
     {{ cls }}
   {% endfor %}

.. automodule:: torch_geometric.loader
    :members:
pytorch/docs/source/modules/nn.rst000066400000000000000000000247421444262304300176260ustar00rootroot00000000000000torch_geometric.nn
==================

.. contents:: Contents
    :local:

.. autoclass:: torch_geometric.nn.sequential.Sequential

.. currentmodule:: torch_geometric.nn.dense

   {% for name in torch_geometric.nn.dense.lin_classes %}
.. autoclass:: {{ name }}
   :members:
   {% endfor %}

Convolutional Layers
--------------------

.. currentmodule:: torch_geometric.nn.conv

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/nn.rst

   {% for name in torch_geometric.nn.conv.classes %}
     {{ name }}
   {% endfor %}

Aggregation Operators
---------------------

.. currentmodule:: torch_geometric.nn.aggr

Aggregation functions play an important role in the message passing framework and the readout functions of Graph Neural Networks.
Specifically, many works in the literature (`Hamilton et al. (2017) `__, `Xu et al. (2018) `__, `Corso et al. (2020) `__, `Li et al. (2020) `__, `Tailor et al. (2021) `__) demonstrate that the choice of aggregation functions contributes significantly to the representational power and performance of the model.
For example, **mean aggregation** captures the distribution (or proportions) of elements, **max aggregation** proves to be advantageous to identify representative elements, and **sum aggregation** enables the learning of structural graph properties (`Xu et al. (2018) `__).
Recent works also show that using **multiple aggregations** (`Corso et al. (2020) `__, `Tailor et al. (2021) `__) and **learnable aggregations** (`Li et al. (2020) `__) can potentially provide substantial improvements.
Another line of research studies optimization-based and implicitly-defined aggregations (`Bartunov et al. (2022) `__).
Furthermore, an interesting discussion concerns the trade-off between representational power (usually gained through learnable functions implemented as neural networks) and the formal property of permutation invariance (`Buterez et al. (2022) `__).

To facilitate further experimentation and unify the concepts of aggregation within GNNs across both :class:`~torch_geometric.nn.conv.MessagePassing` and global readouts, we have made the concept of :class:`~torch_geometric.nn.aggr.Aggregation` a first-class principle in :pyg:`PyG`.
As of now, :pyg:`PyG` provides support for various aggregations --- from rather simple ones (*e.g.*, :obj:`mean`, :obj:`max`, :obj:`sum`), to advanced ones (*e.g.*, :obj:`median`, :obj:`var`, :obj:`std`), learnable ones (*e.g.*, :class:`~torch_geometric.nn.aggr.SoftmaxAggregation`, :class:`~torch_geometric.nn.aggr.PowerMeanAggregation`, :class:`~torch_geometric.nn.aggr.SetTransformerAggregation`), and exotic ones (*e.g.*, :class:`~torch_geometric.nn.aggr.MLPAggregation`, :class:`~torch_geometric.nn.aggr.LSTMAggregation`, :class:`~torch_geometric.nn.aggr.SortAggregation`, :class:`~torch_geometric.nn.aggr.EquilibriumAggregation`):

.. code-block:: python

   from torch_geometric.nn import aggr

   # Simple aggregations:
   mean_aggr = aggr.MeanAggregation()
   max_aggr = aggr.MaxAggregation()

   # Advanced aggregations:
   median_aggr = aggr.MedianAggregation()

   # Learnable aggregations:
   softmax_aggr = aggr.SoftmaxAggregation(learn=True)
   powermean_aggr = aggr.PowerMeanAggregation(learn=True)

   # Exotic aggregations:
   lstm_aggr = aggr.LSTMAggregation(in_channels=..., out_channels=...)
   sort_aggr = aggr.SortAggregation(k=4)

We can then easily apply these aggregations over a batch of sets of potentially varying size.
For this, an :obj:`index` vector defines the mapping from input elements to their location in the output:

.. code-block:: python

   # Feature matrix holding 1000 elements with 64 features each:
   x = torch.randn(1000, 64)

   # Randomly assign elements to 100 sets:
   index = torch.randint(0, 100, (1000, ))

   output = mean_aggr(x, index)  #  Output shape: [100, 64]

Notably, all aggregations share the same set of forward arguments, as described in detail in the :class:`torch_geometric.nn.aggr.Aggregation` base class.

Each of the provided aggregations can be used within :class:`~torch_geometric.nn.conv.MessagePassing` as well as for hierachical/global pooling to obtain graph-level representations:

.. code-block:: python

   import torch
   from torch_geometric.nn import MessagePassing

   class MyConv(MessagePassing):
       def __init__(self, ...):
           # Use a learnable softmax neighborhood aggregation:
           super().__init__(aggr=aggr.SoftmaxAggregation(learn=True))

      def forward(self, x, edge_index):
          ....


   class MyGNN(torch.nn.Module)
       def __init__(self, ...):
           super().__init__()

           self.conv = MyConv(...)
           # Use a global sort aggregation:
           self.global_pool = aggr.SortAggregation(k=4)
           self.classifier = torch.nn.Linear(...)

        def foward(self, x, edge_index, batch):
            x = self.conv(x, edge_index).relu()
            x = self.global_pool(x, batch)
            x = self.classifier(x)
            return x

In addition, the aggregation package of :pyg:`PyG` introduces two new concepts:
First, aggregations can be **resolved from pure strings** via a lookup table, following the design principles of the `class-resolver `__ library, *e.g.*, by simply passing in :obj:`"median"` to the :class:`~torch_geometric.nn.conv.MessagePassing` module.
This will automatically resolve to the :obj:`~torch_geometric.nn.aggr.MedianAggregation` class:

.. code-block:: python

   class MyConv(MessagePassing):
       def __init__(self, ...):
           super().__init__(aggr="median")

Secondly, **multiple aggregations** can be combined and stacked via the :class:`~torch_geometric.nn.aggr.MultiAggregation` module in order to enhance the representational power of GNNs (`Corso et al. (2020) `__, `Tailor et al. (2021) `__):

.. code-block:: python

   class MyConv(MessagePassing):
       def __init__(self, ...):
           # Combines a set of aggregations and concatenates their results,
           # i.e. its output will be `[num_nodes, 3 * out_channels]` here.
           # Note that the interface also supports automatic resolution.
           super().__init__(aggr=aggr.MultiAggregation(
               ['mean', 'std', aggr.SoftmaxAggregation(learn=True)]))

Importantly, :class:`~torch_geometric.nn.aggr.MultiAggregation` provides various options to combine the outputs of its underlying aggegations (*e.g.*, using concatenation, summation, attention, ...) via its :obj:`mode` argument.
The default :obj:`mode` performs concatenation (:obj:`"cat"`).
For combining via attention, we need to additionally specify the :obj:`in_channels` :obj:`out_channels`, and :obj:`num_heads`:

.. code-block:: python

   multi_aggr = aggr.MultiAggregation(
       aggrs=['mean', 'std'],
       mode='attn',
       mode_kwargs=dict(in_channels=64, out_channels=64, num_heads=4),
   )

If aggregations are given as a list, they will be automatically resolved to a :class:`~torch_geometric.nn.aggr.MultiAggregation`, *e.g.*, :obj:`aggr=['mean', 'std', 'median']`.

Finally, we added full support for customization of aggregations into the :class:`~torch_geometric.nn.conv.SAGEConv` layer --- simply override its :obj:`aggr` argument and **utilize the power of aggregation within your GNN**.

.. note::

   You can read more about the :class:`torch_geometric.nn.aggr` package in this [blog post](https://medium.com/@pytorch_geometric/a-principled-approach-to-aggregations-983c086b10b3).

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.aggr.classes %}
     {{ name }}
   {% endfor %}

Normalization Layers
--------------------

.. currentmodule:: torch_geometric.nn.norm

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.norm.classes %}
     {{ name }}
   {% endfor %}

Pooling Layers
--------------

.. currentmodule:: torch_geometric.nn.pool

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.pool.classes %}
     {{ name }}
   {% endfor %}

Unpooling Layers
----------------

.. currentmodule:: torch_geometric.nn.unpool

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.unpool.classes %}
     {{ name }}
   {% endfor %}

Models
------

.. currentmodule:: torch_geometric.nn.models

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/nn.rst

   {% for name in torch_geometric.nn.models.classes %}
     {{ name }}
   {% endfor %}

KGE Models
----------

.. currentmodule:: torch_geometric.nn.kge

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.kge.classes %}
     {{ name }}
   {% endfor %}

Encodings
---------

.. automodule:: torch_geometric.nn.encoding
   :members:
   :undoc-members:
   :exclude-members: training

Functional
----------

.. py:currentmodule:: torch_geometric.nn.functional

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.functional.classes %}
     {{ name }}
   {% endfor %}

Dense Convolutional Layers
--------------------------

.. currentmodule:: torch_geometric.nn.dense

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.dense.conv_classes %}
     {{ name }}
   {% endfor %}

Dense Pooling Layers
--------------------

.. currentmodule:: torch_geometric.nn.dense

.. autosummary::
   :nosignatures:
   :toctree: ../generated

   {% for name in torch_geometric.nn.dense.pool_classes %}
     {{ name }}
   {% endfor %}

Model Transformations
---------------------

.. autoclass:: torch_geometric.nn.fx.Transformer
   :members:
   :undoc-members:
   :exclude-members: graph, find_by_target, find_by_name

.. autofunction:: torch_geometric.nn.to_hetero_transformer.to_hetero

.. autofunction:: torch_geometric.nn.to_hetero_with_bases_transformer.to_hetero_with_bases

DataParallel Layers
-------------------

.. automodule:: torch_geometric.nn.data_parallel
   :members:

Model Hub
---------

.. automodule:: torch_geometric.nn.model_hub
   :members:

Model Summary
-------------

.. automodule:: torch_geometric.nn.summary
   :members:
pytorch/docs/source/modules/profile.rst000066400000000000000000000004421444262304300206420ustar00rootroot00000000000000torch_geometric.profile
=======================

.. currentmodule:: torch_geometric.profile

.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.profile.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.profile
    :members:
    :undoc-members:
pytorch/docs/source/modules/root.rst000066400000000000000000000004601444262304300201650ustar00rootroot00000000000000torch_geometric
===============

.. automodule:: torch_geometric.seed
    :members:

.. automodule:: torch_geometric.home
    :members:

.. automodule:: torch_geometric.compile
    :members:

.. automodule:: torch_geometric.debug
    :members:

.. automodule:: torch_geometric.experimental
    :members:
pytorch/docs/source/modules/sampler.rst000066400000000000000000000006431444262304300206500ustar00rootroot00000000000000torch_geometric.sampler
=======================

.. currentmodule:: torch_geometric.sampler

.. autosummary::
   :nosignatures:
   {% for cls in torch_geometric.sampler.classes %}
     {{ cls }}
   {% endfor %}

.. autoclass:: torch_geometric.sampler.BaseSampler
   :members:

.. automodule:: torch_geometric.sampler
   :members:
   :exclude-members: sample_from_nodes, sample_from_edges, edge_permutation, BaseSampler
pytorch/docs/source/modules/transforms.rst000066400000000000000000000033061444262304300214020ustar00rootroot00000000000000torch_geometric.transforms
==========================

.. contents:: Contents
    :local:

Transforms are a general way to modify and customize :class:`~torch_geometric.data.Data` or :class:`~torch_geometric.data.HeteroData` objects, either by implicitly passing them as an argument to a :class:`~torch_geometric.data.Dataset`, or by applying them explicitly to individual :class:`~torch_geometric.data.Data` or :class:`~torch_geometric.data.HeteroData` objects:

.. code-block:: python

   import torch_geometric.transforms as T
   from torch_geometric.datasets import TUDataset

   transform = T.Compose([T.ToUndirected(), T.AddSelfLoops()])

   dataset = TUDataset(path, name='MUTAG', transform=transform)
   data = dataset[0]  # Implicitly transform data on every access.

   data = TUDataset(path, name='MUTAG')[0]
   data = transform(data)  # Explicitly transform data.

General Transforms
------------------

.. currentmodule:: torch_geometric.transforms

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.transforms.general_transforms %}
     {{ name }}
   {% endfor %}

Graph Transforms
----------------

.. currentmodule:: torch_geometric.transforms

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.transforms.graph_transforms %}
     {{ name }}
   {% endfor %}

Vision Transforms
-----------------

.. currentmodule:: torch_geometric.transforms

.. autosummary::
   :nosignatures:
   :toctree: ../generated
   :template: autosummary/only_class.rst

   {% for name in torch_geometric.transforms.vision_transforms %}
     {{ name }}
   {% endfor %}
pytorch/docs/source/modules/utils.rst000066400000000000000000000004041444262304300203400ustar00rootroot00000000000000torch_geometric.utils
=====================

.. currentmodule:: torch_geometric.utils

.. autosummary::
    :nosignatures:
    {% for cls in torch_geometric.utils.classes %}
      {{ cls }}
    {% endfor %}

.. automodule:: torch_geometric.utils
    :members:
pytorch/docs/source/notes/000077500000000000000000000000001444262304300161305ustar00rootroot00000000000000pytorch/docs/source/notes/batching.rst000066400000000000000000000000601444262304300204350ustar00rootroot00000000000000:orphan:

.. include:: ../advanced/batching.rst
pytorch/docs/source/notes/cheatsheet.rst000066400000000000000000000123311444262304300207770ustar00rootroot00000000000000:orphan:

GNN Cheatsheet
==============

* :class:`~torch_sparse.SparseTensor`: If checked (βœ“), supports message passing based on :class:`torch_sparse.SparseTensor`, *e.g.*, :obj:`GCNConv(...).forward(x, adj_t)`. See `here <../advanced/sparse_tensor.html>`__ for the accompanying tutorial.

* :obj:`edge_weight`: If checked (βœ“), supports message passing with one-dimensional edge weight information, *e.g.*, :obj:`GraphConv(...).forward(x, edge_index, edge_weight)`.

* :obj:`edge_attr`: If checked (βœ“), supports message passing with multi-dimensional edge feature information, *e.g.*, :obj:`GINEConv(...).forward(x, edge_index, edge_attr)`.

* **bipartite**: If checked (βœ“), supports message passing in bipartite graphs with potentially different feature dimensionalities for source and destination nodes, *e.g.*, :obj:`SAGEConv(in_channels=(16, 32), out_channels=64)`.

* **static**: If checked (βœ“), supports message passing in static graphs, *e.g.*, :obj:`GCNConv(...).forward(x, edge_index)` with :obj:`x` having shape :obj:`[batch_size, num_nodes, in_channels]`.

* **lazy**: If checked (βœ“), supports lazy initialization of message passing layers, *e.g.*, :obj:`SAGEConv(in_channels=-1, out_channels=64)`.

Graph Neural Network Operators
------------------------------

.. list-table::
    :widths: 40 10 10 10 10 10 10
    :header-rows: 1

    * - Name
      - :class:`~torch_sparse.SparseTensor`
      - :obj:`edge_weight`
      - :obj:`edge_attr`
      - bipartite
      - static
      - lazy
{% for cls in torch_geometric.nn.conv.classes[1:] %}
{% if not torch_geometric.nn.conv.utils.processes_heterogeneous_graphs(cls) and
      not torch_geometric.nn.conv.utils.processes_hypergraphs(cls) and
      not torch_geometric.nn.conv.utils.processes_point_clouds(cls) %}
    * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__)
      - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %}
{% endif %}
{% endfor %}

Heterogeneous Graph Neural Network Operators
--------------------------------------------

.. list-table::
    :widths: 40 10 10 10 10 10 10
    :header-rows: 1

    * - Name
      - :class:`~torch_sparse.SparseTensor`
      - :obj:`edge_weight`
      - :obj:`edge_attr`
      - bipartite
      - static
      - lazy
{% for cls in torch_geometric.nn.conv.classes[1:] %}
{% if torch_geometric.nn.conv.utils.processes_heterogeneous_graphs(cls) %}
    * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__)
      - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %}
{% endif %}
{% endfor %}

Hypergraph Neural Network Operators
-----------------------------------

.. list-table::
    :widths: 40 10 10 10 10 10 10
    :header-rows: 1

    * - Name
      - :class:`~torch_sparse.SparseTensor`
      - :obj:`edge_weight`
      - :obj:`edge_attr`
      - bipartite
      - static
      - lazy
{% for cls in torch_geometric.nn.conv.classes[1:] %}
{% if torch_geometric.nn.conv.utils.processes_hypergraphs(cls) %}
    * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__)
      - {% if torch_geometric.nn.conv.utils.supports_sparse_tensor(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_weights(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_edge_features(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_static_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %}
{% endif %}
{% endfor %}

Point Cloud Neural Network Operators
------------------------------------

.. list-table::
    :widths: 80 10 10
    :header-rows: 1

    * - Name
      - bipartite
      - lazy
{% for cls in torch_geometric.nn.conv.classes[1:] %}
{% if torch_geometric.nn.conv.utils.processes_point_clouds(cls) %}
    * - :class:`~torch_geometric.nn.conv.{{ cls }}` (`Paper <{{ torch_geometric.nn.conv.utils.paper_link(cls) }}>`__)
      - {% if torch_geometric.nn.conv.utils.supports_bipartite_graphs(cls) %}βœ“{% endif %}
      - {% if torch_geometric.nn.conv.utils.supports_lazy_initialization(cls) %}βœ“{% endif %}
{% endif %}
{% endfor %}
pytorch/docs/source/notes/colabs.rst000066400000000000000000000000611444262304300201220ustar00rootroot00000000000000:orphan:

.. include:: ../get_started/colabs.rst
pytorch/docs/source/notes/create_dataset.rst000066400000000000000000000000661444262304300216340ustar00rootroot00000000000000:orphan:

.. include:: ../tutorial/create_dataset.rst
pytorch/docs/source/notes/create_gnn.rst000066400000000000000000000000621444262304300207650ustar00rootroot00000000000000:orphan:

.. include:: ../tutorial/create_gnn.rst
pytorch/docs/source/notes/data_cheatsheet.rst000066400000000000000000000115471444262304300220000ustar00rootroot00000000000000:orphan:

Dataset Cheatsheet
==================

.. note::

    This dataset statistics table is a **work in progress**.
    Please consider helping us filling its content by providing statistics for individual datasets.
    See `here `__ and `here `__ for examples on how to do so.

Homogeneous Datasets
--------------------

.. list-table::
    :widths: 50 10 10 10 10 10
    :header-rows: 1

    * - Name
      - #graphs
      - #nodes
      - #edges
      - #features
      - #classes/#tasks
{% for cls in torch_geometric.datasets.homo_datasets %}
    * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %}
      - {%if torch_geometric.datasets.utils.has_stats(cls) %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default=1) }}{% else %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default='') }}{% endif %}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', default='') }}
    {% for child in torch_geometric.datasets.utils.get_children(cls) %}
    * - └─ {{ child }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', child, default=1) }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }}
    {% endfor %}
{% endfor %}

Heterogeneous Datasets
----------------------

.. list-table::
    :widths: 50 30 10 10
    :header-rows: 1

    * - Name
      - #nodes/#edges
      - #features
      - #classes/#tasks
{% for cls in torch_geometric.datasets.hetero_datasets %}
    * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %}
      -
      -
      -
    {% for child in torch_geometric.datasets.utils.get_children(cls) %}
    * - └─ **{{torch_geometric.datasets.utils.get_type(child)}} Type**: {{ child }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes/#edges', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }}
    {% endfor %}
{% endfor %}

Synthetic Datasets
------------------

.. list-table::
    :widths: 50 10 10 10 10 10
    :header-rows: 1

    * - Name
      - #graphs
      - #nodes
      - #edges
      - #features
      - #classes/#tasks
{% for cls in torch_geometric.datasets.synthetic_datasets %}
    * - :class:`~torch_geometric.datasets.{{ cls }}` {% if torch_geometric.datasets.utils.paper_link(cls) %}(`Paper <{{ torch_geometric.datasets.utils.paper_link(cls) }}>`__){% endif %}
      - {%if torch_geometric.datasets.utils.has_stats(cls) %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default=1) }}{% else %}{{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', default='') }}{% endif %}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', default='') }}
    {% for child in torch_geometric.datasets.utils.get_children(cls) %}
    * - └─ {{ child }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#graphs', child, default=1) }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#nodes', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#edges', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#features', child, default='') }}
      - {{ torch_geometric.datasets.utils.get_stat(cls, '#classes', child, default='') }}{{ torch_geometric.datasets.utils.get_stat(cls, '#tasks', child, default='') }}
    {% endfor %}
{% endfor %}
pytorch/docs/source/notes/explain.rst000066400000000000000000000000571444262304300203240ustar00rootroot00000000000000:orphan:

.. include:: ../tutorial/explain.rst
pytorch/docs/source/notes/graphgym.rst000066400000000000000000000000601444262304300204740ustar00rootroot00000000000000:orphan:

.. include:: ../advanced/graphgym.rst
pytorch/docs/source/notes/heterogeneous.rst000066400000000000000000000000651444262304300215370ustar00rootroot00000000000000:orphan:

.. include:: ../tutorial/heterogeneous.rst
pytorch/docs/source/notes/installation.rst000066400000000000000000000000631444262304300213620ustar00rootroot00000000000000:orphan:

.. include:: ../install/installation.rst
pytorch/docs/source/notes/introduction.rst000066400000000000000000000000671444262304300214060ustar00rootroot00000000000000:orphan:

.. include:: ../get_started/introduction.rst
pytorch/docs/source/notes/jit.rst000066400000000000000000000000531444262304300174460ustar00rootroot00000000000000:orphan:

.. include:: ../advanced/jit.rst
pytorch/docs/source/notes/load_csv.rst000066400000000000000000000000601444262304300204500ustar00rootroot00000000000000:orphan:

.. include:: ../tutorial/load_csv.rst
pytorch/docs/source/notes/remote.rst000066400000000000000000000000561444262304300201560ustar00rootroot00000000000000:orphan:

.. include:: ../advanced/remote.rst
pytorch/docs/source/notes/resources.rst000066400000000000000000000000611444262304300206710ustar00rootroot00000000000000:orphan:

.. include:: ../external/resources.rst
pytorch/docs/source/notes/sparse_tensor.rst000066400000000000000000000000651444262304300215520ustar00rootroot00000000000000:orphan:

.. include:: ../advanced/sparse_tensor.rst
pytorch/docs/source/tutorial/000077500000000000000000000000001444262304300166435ustar00rootroot00000000000000pytorch/docs/source/tutorial/compile.rst000066400000000000000000000142431444262304300210310ustar00rootroot00000000000000Compiled Graph Neural Networks
==============================

:meth:`torch.compile` is the latest method to speed up your :pytorch:`PyTorch` code in :obj:`torch >= 2.0.0`!
:meth:`torch.compile` makes PyTorch code run faster by JIT-compiling it into optimized kernels, all while required minimal code changes.

Under the hood, :meth:`torch.compile` captures :pytorch:`PyTorch` programs via :obj:`TorchDynamo`, canonicalizes over 2,000 :pytorch:`PyTorch` operators via :obj:`PrimTorch`, and finally generates fast code out of it across multiple accelerators and backends via the deep learning compiler :obj:`TorchInductor`.

.. note::
    See `here `__ for a general tutorial on how to leverage :meth:`torch.compile`, and `here `__ for a description of its interface.

In this tutorial, we show how to optimize your custom :pyg:`PyG` model via :meth:`torch.compile`.

Introducing :meth:`torch_geometric.compile`
-------------------------------------------

By default, :meth:`torch.compile` struggles to optimize a custom :pyg:`PyG` model since its underlying :class:`~torch_geometric.nn.conv.MessagePassing` interface is JIT-unfriendly due to its generality.
As such, in :pyg:`PyG 2.3`, we introduce :meth:`torch_geometric.compile`, a wrapper around :meth:`torch.compile` with the same signature.

:meth:`torch_geometric.compile` applies further optimizations to make :pyg:`PyG` models more compiler-friendly.
Specifically, it:

#. Temporarily disables the usage of the extension packages :obj:`torch_scatter`, :obj:`torch_sparse` and :obj:`pyg_lib` during GNN execution workflows (since these are not *yet* directly optimizable by :pytorch:`PyTorch`).
   From :pyg:`PyG 2.3` onwards, these packages are purely optional and not required anymore for running :pyg:`PyG` models (but :obj:`pyg_lib` may be required for graph sampling routines).

#. Converts all instances of :class:`~torch_geometric.nn.conv.MessagePassing` modules into their jittable instances (see :meth:`torch_geometric.nn.conv.MessagePassing.jittable`)

Without these adjustments, :meth:`torch.compile` may currently fail to correctly optimize your :pyg:`PyG` model.
We are working on fully relying on :meth:`torch.compile` for future releases.

Basic Usage
-----------

Leveraging :meth:`torch_geometric.compile` is as simple as the usage of :meth:`torch.compile`.
Once you have a :pyg:`PyG` model defined, simply wrap it with :meth:`torch_geometric.compile` to obtain its optimized version:

.. code-block:: python

    import torch_geometric
    from torch_geometric.nn import GraphSAGE

    model = GraphSAGE(in_channels, hidden_channels, num_layers, out_channels)
    model = model.to(device)

    model = torch_geometric.compile(model)

and execute it as usual:

.. code-block:: python

    from torch_geometric.datasets import Planetoid

    dataset = Planetoid(root, name="Cora")
    data = dataset[0].to(device)

    out = model(data.x, data.edge_index)

We have incorporated multiple examples in :obj:`examples/compile` that further show the practical usage of :meth:`torch_geometric.compile`:

#. `Node Classification `__ via :class:`~torch_geometric.nn.models.GCN`
#. `Graph Classification `__ via :class:`~torch_geometric.nn.models.GIN`

Note that :meth:`torch.compile(model, dynamic=True)` does sadly not yet work for :pyg:`PyG` models on :pytorch:`PyTorch 2.0`.
While static compilation via :meth:`torch.compile(model, dynamic=False)` works fine, it will re-compile the model everytime it sees an input with a different shape.
That currently does not play that nicely with the way :pyg:`PyG` performs mini-batching, and will hence lead to major slow-downs.
We are working with the :pytorch:`PyTorch` team to fix this limitation (see `this `_ :github:`GitHub` issue).
A temporary workaround is to utilize the :class:`torch_geometric.transforms.Pad` transformation to ensure that all inputs are of equal shape.

If you notice that :meth:`~torch_geometric.compile` fails for a certain :pyg:`PyG` model, do not hesitate to reach out either on :github:`null` `GitHub `_ or :slack:`null` `Slack `_.
We are very eager to improve :meth:`~torch_geometric.compile` support across the whole :pyg:`PyG` code base.

Benchmark
---------

:meth:`torch.compile` works **fantastically well** for many :pyg:`PyG` models.
**Overall, we observe runtime improvements of nearly up to 300%.**

Specifically, we benchmark :class:`~torch_geometric.nn.models.GCN`, :class:`~torch_geometric.nn.models.GraphSAGE` and :class:`~torch_geometric.nn.models.GIN` and compare runtimes obtained from traditional eager mode and :meth:`torch_geometric.compile`.
We use a synthetic graph with 10,000 nodes and 200,000 edges, and a hidden feature dimensionality of 64.
We report runtimes over 500 optimization steps:

.. list-table::
   :widths: 15 15 15 15 15 15
   :header-rows: 1

   * - Model
     - Mode
     - Forward
     - Backward
     - Total
     - Speedup
   * - :class:`~torch_geometric.nn.models.GCN`
     - Eager
     - 2.6396s
     - 2.1697s
     - 4.8093s
     -
   * - :class:`~torch_geometric.nn.models.GCN`
     - **Compiled**
     - **1.1082s**
     - **0.5896s**
     - **1.6978s**
     - **2.83x**
   * - :class:`~torch_geometric.nn.models.GraphSAGE`
     - Eager
     - 1.6023s
     - 1.6428s
     - 3.2451s
     -
   * - :class:`~torch_geometric.nn.models.GraphSAGE`
     - **Compiled**
     - **0.7033s**
     - **0.7465s**
     - **1.4498s**
     - **2.24x**
   * - :class:`~torch_geometric.nn.models.GIN`
     - Eager
     - 1.6701s
     - 1.6990s
     - 3.3690s
     -
   * - :class:`~torch_geometric.nn.models.GIN`
     - **Compiled**
     - **0.7320s**
     - **0.7407s**
     - **1.4727s**
     - **2.29x**

To reproduce these results, run

.. code-block:: console

    python test/nn/models/test_basic_gnn.py

from the root folder of your checked out :pyg:`PyG` repository from :github:`GitHub`.
pytorch/docs/source/tutorial/create_dataset.rst000066400000000000000000000206321444262304300223500ustar00rootroot00000000000000Creating Your Own Datasets
==========================

Although :pyg:`PyG` already contains a lot of useful datasets, you may wish to create your own dataset with self-recorded or non-publicly available data.

Implementing datasets by yourself is straightforward and you may want to take a look at the source code to find out how the various datasets are implemented.
However, we give a brief introduction on what is needed to setup your own dataset.

We provide two abstract classes for datasets: :class:`torch_geometric.data.Dataset` and :class:`torch_geometric.data.InMemoryDataset`.
:class:`torch_geometric.data.InMemoryDataset` inherits from :class:`torch_geometric.data.Dataset` and should be used if the whole dataset fits into CPU memory.

Following the :obj:`torchvision` convention, each dataset gets passed a root folder which indicates where the dataset should be stored.
We split up the root folder into two folders: the :obj:`raw_dir`, where the dataset gets downloaded to, and the :obj:`processed_dir`, where the processed dataset is being saved.

In addition, each dataset can be passed a :obj:`transform`, a :obj:`pre_transform` and a :obj:`pre_filter` function, which are :obj:`None` by default.
The :obj:`transform` function dynamically transforms the data object before accessing (so it is best used for data augmentation).
The :obj:`pre_transform` function applies the transformation before saving the data objects to disk (so it is best used for heavy precomputation which needs to be only done once).
The :obj:`pre_filter` function can manually filter out data objects before saving.
Use cases may involve the restriction of data objects being of a specific class.

Creating "In Memory Datasets"
-----------------------------

In order to create a :class:`torch_geometric.data.InMemoryDataset`, you need to implement four fundamental methods:

* :func:`torch_geometric.data.InMemoryDataset.raw_file_names`: A list of files in the :obj:`raw_dir` which needs to be found in order to skip the download.

* :func:`torch_geometric.data.InMemoryDataset.processed_file_names`: A list of files in the :obj:`processed_dir` which needs to be found in order to skip the processing.

* :func:`torch_geometric.data.InMemoryDataset.download`: Downloads raw data into :obj:`raw_dir`.

* :func:`torch_geometric.data.InMemoryDataset.process`: Processes raw data and saves it into the :obj:`processed_dir`.

You can find helpful methods to download and extract data in :mod:`torch_geometric.data`.

The real magic happens in the body of :meth:`~torch_geometric.data.InMemoryDataset.process`.
Here, we need to read and create a list of :class:`~torch_geometric.data.Data` objects and save it into the :obj:`processed_dir`.
Because saving a huge python list is rather slow, we collate the list into one huge :class:`~torch_geometric.data.Data` object via :meth:`torch_geometric.data.InMemoryDataset.collate` before saving .
The collated data object has concatenated all examples into one big data object and, in addition, returns a :obj:`slices` dictionary to reconstruct single examples from this object.
Finally, we need to load these two objects in the constructor into the properties :obj:`self.data` and :obj:`self.slices`.

Let's see this process in a simplified example:

.. code-block:: python

    import torch
    from torch_geometric.data import InMemoryDataset, download_url


    class MyOwnDataset(InMemoryDataset):
        def __init__(self, root, transform=None, pre_transform=None, pre_filter=None):
            super().__init__(root, transform, pre_transform, pre_filter)
            self.data, self.slices = torch.load(self.processed_paths[0])

        @property
        def raw_file_names(self):
            return ['some_file_1', 'some_file_2', ...]

        @property
        def processed_file_names(self):
            return ['data.pt']

        def download(self):
            # Download to `self.raw_dir`.
            download_url(url, self.raw_dir)
            ...

        def process(self):
            # Read data into huge `Data` list.
            data_list = [...]

            if self.pre_filter is not None:
                data_list = [data for data in data_list if self.pre_filter(data)]

            if self.pre_transform is not None:
                data_list = [self.pre_transform(data) for data in data_list]

            data, slices = self.collate(data_list)
            torch.save((data, slices), self.processed_paths[0])

Creating "Larger" Datasets
--------------------------

For creating datasets which do not fit into memory, the :class:`torch_geometric.data.Dataset` can be used, which closely follows the concepts of the :obj:`torchvision` datasets.
It expects the following methods to be implemented in addition:

* :func:`torch_geometric.data.Dataset.len`: Returns the number of examples in your dataset.

* :func:`torch_geometric.data.Dataset.get`: Implements the logic to load a single graph.

Internally, :meth:`torch_geometric.data.Dataset.__getitem__` gets data objects from :meth:`torch_geometric.data.Dataset.get` and optionally transforms them according to :obj:`transform`.

Let's see this process in a simplified example:

.. code-block:: python

    import os.path as osp

    import torch
    from torch_geometric.data import Dataset, download_url


    class MyOwnDataset(Dataset):
        def __init__(self, root, transform=None, pre_transform=None, pre_filter=None):
            super().__init__(root, transform, pre_transform, pre_filter)

        @property
        def raw_file_names(self):
            return ['some_file_1', 'some_file_2', ...]

        @property
        def processed_file_names(self):
            return ['data_1.pt', 'data_2.pt', ...]

        def download(self):
            # Download to `self.raw_dir`.
            path = download_url(url, self.raw_dir)
            ...

        def process(self):
            idx = 0
            for raw_path in self.raw_paths:
                # Read data from `raw_path`.
                data = Data(...)

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                torch.save(data, osp.join(self.processed_dir, f'data_{idx}.pt'))
                idx += 1

        def len(self):
            return len(self.processed_file_names)

        def get(self, idx):
            data = torch.load(osp.join(self.processed_dir, f'data_{idx}.pt'))
            return data

Here, each graph data object gets saved individually in :meth:`~torch_geometric.data.Dataset.process`, and is manually loaded in :meth:`~torch_geometric.data.Dataset.get`.

Frequently Asked Questions
--------------------------

#. **How can I skip the execution of** :meth:`download` **and/or** :meth:`process` **?**

    You can skip downloading and/or processing by just not overriding the :meth:`download()` and :meth:`process()` methods:

    .. code-block:: python

        class MyOwnDataset(Dataset):
            def __init__(self, transform=None, pre_transform=None):
                super().__init__(None, transform, pre_transform)

#. **Do I really need to use these dataset interfaces?**

    No! Just as in regular :pytorch:`PyTorch`, you do not have to use datasets, *e.g.*, when you want to create synthetic data on the fly without saving them explicitly to disk.
    In this case, simply pass a regular python list holding :class:`torch_geometric.data.Data` objects and pass them to :class:`torch_geometric.loader.DataLoader`:

    .. code-block:: python

        from torch_geometric.data import Data
        from torch_geometric.loader import DataLoader

        data_list = [Data(...), ..., Data(...)]
        loader = DataLoader(data_list, batch_size=32)

Exercises
---------

Consider the following :class:`~torch_geometric.data.InMemoryDataset` constructed from a list of :obj:`~torch_geometric.data.Data` objects:

.. code-block:: python

    class MyDataset(InMemoryDataset):
        def __init__(self, root, data_list, transform=None):
            self.data_list = data_list
            super().__init__(root, transform)
            self.data, self.slices = torch.load(self.processed_paths[0])

        @property
        def processed_file_names(self):
            return 'data.pt'

        def process(self):
            torch.save(self.collate(self.data_list), self.processed_paths[0])

1. What is the output of :obj:`self.processed_paths[0]`?

2. What does :meth:`~torch_geometric.data.InMemoryDataset.collate` do?
pytorch/docs/source/tutorial/create_gnn.rst000066400000000000000000000316461444262304300215140ustar00rootroot00000000000000Creating Message Passing Networks
=================================

Generalizing the convolution operator to irregular domains is typically expressed as a *neighborhood aggregation* or *message passing* scheme.
With :math:`\mathbf{x}^{(k-1)}_i \in \mathbb{R}^F` denoting node features of node :math:`i` in layer :math:`(k-1)` and :math:`\mathbf{e}_{j,i} \in \mathbb{R}^D` denoting (optional) edge features from node :math:`j` to node :math:`i`, message passing graph neural networks can be described as

.. math::
  \mathbf{x}_i^{(k)} = \gamma^{(k)} \left( \mathbf{x}_i^{(k-1)}, \bigoplus_{j \in \mathcal{N}(i)} \, \phi^{(k)}\left(\mathbf{x}_i^{(k-1)}, \mathbf{x}_j^{(k-1)},\mathbf{e}_{j,i}\right) \right),

where :math:`\bigoplus` denotes a differentiable, permutation invariant function, *e.g.*, sum, mean or max, and :math:`\gamma` and :math:`\phi` denote differentiable functions such as MLPs (Multi Layer Perceptrons).

.. contents::
    :local:

The "MessagePassing" Base Class
-------------------------------

:pyg:`PyG` provides the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` base class, which helps in creating such kinds of message passing graph neural networks by automatically taking care of message propagation.
The user only has to define the functions :math:`\phi` , *i.e.* :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.message`, and :math:`\gamma` , *i.e.* :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.update`, as well as the aggregation scheme to use, *i.e.* :obj:`aggr="add"`, :obj:`aggr="mean"` or :obj:`aggr="max"`.

This is done with the help of the following methods:

* :obj:`MessagePassing(aggr="add", flow="source_to_target", node_dim=-2)`: Defines the aggregation scheme to use (:obj:`"add"`, :obj:`"mean"` or :obj:`"max"`) and the flow direction of message passing (either :obj:`"source_to_target"` or :obj:`"target_to_source"`).
  Furthermore, the :obj:`node_dim` attribute indicates along which axis to propagate.
* :obj:`MessagePassing.propagate(edge_index, size=None, **kwargs)`:
  The initial call to start propagating messages.
  Takes in the edge indices and all additional data which is needed to construct messages and to update node embeddings.
  Note that :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate` is not limited to exchanging messages in square adjacency matrices of shape :obj:`[N, N]` only, but can also exchange messages in general sparse assignment matrices, *e.g.*, bipartite graphs, of shape :obj:`[N, M]` by passing :obj:`size=(N, M)` as an additional argument.
  If set to :obj:`None`, the assignment matrix is assumed to be a square matrix.
  For bipartite graphs with two independent sets of nodes and indices, and each set holding its own information, this split can be marked by passing the information as a tuple, *e.g.* :obj:`x=(x_N, x_M)`.
* :obj:`MessagePassing.message(...)`: Constructs messages to node :math:`i` in analogy to :math:`\phi` for each edge :math:`(j,i) \in \mathcal{E}` if :obj:`flow="source_to_target"` and :math:`(i,j) \in \mathcal{E}` if :obj:`flow="target_to_source"`.
  Can take any argument which was initially passed to :meth:`propagate`.
  In addition, tensors passed to :meth:`propagate` can be mapped to the respective nodes :math:`i` and :math:`j` by appending :obj:`_i` or :obj:`_j` to the variable name, *e.g.* :obj:`x_i` and :obj:`x_j`.
  Note that we generally refer to :math:`i` as the central nodes that aggregates information, and refer to :math:`j` as the neighboring nodes, since this is the most common notation.
* :obj:`MessagePassing.update(aggr_out, ...)`: Updates node embeddings in analogy to :math:`\gamma` for each node :math:`i \in \mathcal{V}`.
  Takes in the output of aggregation as first argument and any argument which was initially passed to :func:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate`.

Let us verify this by re-implementing two popular GNN variants, the `GCN layer from Kipf and Welling `_ and the `EdgeConv layer from Wang et al. `_.

Implementing the GCN Layer
--------------------------

The `GCN layer `_ is mathematically defined as

.. math::

    \mathbf{x}_i^{(k)} = \sum_{j \in \mathcal{N}(i) \cup \{ i \}} \frac{1}{\sqrt{\deg(i)} \cdot \sqrt{\deg(j)}} \cdot \left( \mathbf{W}^{\top} \cdot \mathbf{x}_j^{(k-1)} \right) + \mathbf{b},

where neighboring node features are first transformed by a weight matrix :math:`\mathbf{W}`, normalized by their degree, and finally summed up.
Lastly, we apply the bias vector :math:`\mathbf{b}` to the aggregated output.
This formula can be divided into the following steps:

1. Add self-loops to the adjacency matrix.
2. Linearly transform node feature matrix.
3. Compute normalization coefficients.
4. Normalize node features in :math:`\phi`.
5. Sum up neighboring node features (:obj:`"add"` aggregation).
6. Apply a final bias vector.

Steps 1-3 are typically computed before message passing takes place.
Steps 4-5 can be easily processed using the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` base class.
The full layer implementation is shown below:

.. code-block:: python

    import torch
    from torch.nn import Linear, Parameter
    from torch_geometric.nn import MessagePassing
    from torch_geometric.utils import add_self_loops, degree

    class GCNConv(MessagePassing):
        def __init__(self, in_channels, out_channels):
            super().__init__(aggr='add')  # "Add" aggregation (Step 5).
            self.lin = Linear(in_channels, out_channels, bias=False)
            self.bias = Parameter(torch.Tensor(out_channels))

            self.reset_parameters()

        def reset_parameters(self):
            self.lin.reset_parameters()
            self.bias.data.zero_()

        def forward(self, x, edge_index):
            # x has shape [N, in_channels]
            # edge_index has shape [2, E]

            # Step 1: Add self-loops to the adjacency matrix.
            edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))

            # Step 2: Linearly transform node feature matrix.
            x = self.lin(x)

            # Step 3: Compute normalization.
            row, col = edge_index
            deg = degree(col, x.size(0), dtype=x.dtype)
            deg_inv_sqrt = deg.pow(-0.5)
            deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0
            norm = deg_inv_sqrt[row] * deg_inv_sqrt[col]

            # Step 4-5: Start propagating messages.
            out = self.propagate(edge_index, x=x, norm=norm)

            # Step 6: Apply a final bias vector.
            out += self.bias

            return out

        def message(self, x_j, norm):
            # x_j has shape [E, out_channels]

            # Step 4: Normalize node features.
            return norm.view(-1, 1) * x_j

:class:`~torch_geometric.nn.conv.GCNConv` inherits from :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` with :obj:`"add"` propagation.
All the logic of the layer takes place in its :meth:`forward` method.
Here, we first add self-loops to our edge indices using the :meth:`torch_geometric.utils.add_self_loops` function (step 1), as well as linearly transform node features by calling the :class:`torch.nn.Linear` instance (step 2).

The normalization coefficients are derived by the node degrees :math:`\deg(i)` for each node :math:`i` which gets transformed to :math:`1/(\sqrt{\deg(i)} \cdot \sqrt{\deg(j)})` for each edge :math:`(j,i) \in \mathcal{E}`.
The result is saved in the tensor :obj:`norm` of shape :obj:`[num_edges, ]` (step 3).

We then call :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.propagate`, which internally calls :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.message`, :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.aggregate` and :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.update`.
We pass the node embeddings :obj:`x` and the normalization coefficients :obj:`norm` as additional arguments for message propagation.

In the :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.message` function, we need to normalize the neighboring node features :obj:`x_j` by :obj:`norm`.
Here, :obj:`x_j` denotes a *lifted* tensor, which contains the source node features of each edge, *i.e.*, the neighbors of each node.
Node features can be automatically lifted by appending :obj:`_i` or :obj:`_j` to the variable name.
In fact, any tensor can be converted this way, as long as they hold source or destination node features.

That is all that it takes to create a simple message passing layer.
You can use this layer as a building block for deep architectures.
Initializing and calling it is straightforward:

.. code-block:: python

    conv = GCNConv(16, 32)
    x = conv(x, edge_index)

Implementing the Edge Convolution
---------------------------------

The `edge convolutional layer `_ processes graphs or point clouds and is mathematically defined as

.. math::

    \mathbf{x}_i^{(k)} = \max_{j \in \mathcal{N}(i)} h_{\mathbf{\Theta}} \left( \mathbf{x}_i^{(k-1)}, \mathbf{x}_j^{(k-1)} - \mathbf{x}_i^{(k-1)} \right),

where :math:`h_{\mathbf{\Theta}}` denotes an MLP.
In analogy to the GCN layer, we can use the :class:`~torch_geometric.nn.conv.message_passing.MessagePassing` class to implement this layer, this time using the :obj:`"max"` aggregation:

.. code-block:: python

    import torch
    from torch.nn import Sequential as Seq, Linear, ReLU
    from torch_geometric.nn import MessagePassing

    class EdgeConv(MessagePassing):
        def __init__(self, in_channels, out_channels):
            super().__init__(aggr='max') #  "Max" aggregation.
            self.mlp = Seq(Linear(2 * in_channels, out_channels),
                           ReLU(),
                           Linear(out_channels, out_channels))

        def forward(self, x, edge_index):
            # x has shape [N, in_channels]
            # edge_index has shape [2, E]

            return self.propagate(edge_index, x=x)

        def message(self, x_i, x_j):
            # x_i has shape [E, in_channels]
            # x_j has shape [E, in_channels]

            tmp = torch.cat([x_i, x_j - x_i], dim=1)  # tmp has shape [E, 2 * in_channels]
            return self.mlp(tmp)

Inside the :meth:`~torch_geometric.nn.conv.message_passing.MessagePassing.message` function, we use :obj:`self.mlp` to transform both the target node features :obj:`x_i` and the relative source node features :obj:`x_j - x_i` for each edge :math:`(j,i) \in \mathcal{E}`.

The edge convolution is actually a dynamic convolution, which recomputes the graph for each layer using nearest neighbors in the feature space.
Luckily, :pyg:`PyG` comes with a GPU accelerated batch-wise k-NN graph generation method named :meth:`torch_geometric.nn.pool.knn_graph`:

.. code-block:: python

    from torch_geometric.nn import knn_graph

    class DynamicEdgeConv(EdgeConv):
        def __init__(self, in_channels, out_channels, k=6):
            super().__init__(in_channels, out_channels)
            self.k = k

        def forward(self, x, batch=None):
            edge_index = knn_graph(x, self.k, batch, loop=False, flow=self.flow)
            return super().forward(x, edge_index)

Here, :meth:`~torch_geometric.nn.pool.knn_graph` computes a nearest neighbor graph, which is further used to call the :meth:`forward` method of :class:`~torch_geometric.nn.conv.EdgeConv`.

This leaves us with a clean interface for initializing and calling this layer:

.. code-block:: python

    conv = DynamicEdgeConv(3, 128, k=6)
    x = conv(x, batch)

Exercises
---------

Imagine we are given the following :obj:`~torch_geometric.data.Data` object:

.. code-block:: python

    import torch
    from torch_geometric.data import Data

    edge_index = torch.tensor([[0, 1],
                               [1, 0],
                               [1, 2],
                               [2, 1]], dtype=torch.long)
    x = torch.tensor([[-1], [0], [1]], dtype=torch.float)

    data = Data(x=x, edge_index=edge_index.t().contiguous())

Try to answer the following questions related to :class:`~torch_geometric.nn.conv.GCNConv`:

1. What information does :obj:`row` and :obj:`col` hold?

2. What does :meth:`~torch_geometric.utils.degree` do?

3. Why do we use :obj:`degree(col, ...)` rather than :obj:`degree(row, ...)`?

4. What does :obj:`deg_inv_sqrt[col]` and :obj:`deg_inv_sqrt[row]` do?

5. What information does :obj:`x_j` hold in the :meth:`~torch_geometric.nn.conv.MessagePassing.message` function? If :obj:`self.lin` denotes the identity function, what is the exact content of :obj:`x_j`?

6. Add an :meth:`~torch_geometric.nn.conv.MessagePassing.update` function to :class:`~torch_geometric.nn.conv.GCNConv` that adds transformed central node features to the aggregated output.

Try to answer the following questions related to :class:`~torch_geometric.nn.conv.EdgeConv`:

1. What is :obj:`x_i` and :obj:`x_j - x_i`?

2. What does :obj:`torch.cat([x_i, x_j - x_i], dim=1)` do? Why :obj:`dim = 1`?
pytorch/docs/source/tutorial/explain.rst000066400000000000000000000222361444262304300210420ustar00rootroot00000000000000GNN Explainability
==================

Interpreting GNN models is crucial for many use cases.
:pyg:`PyG` (2.3 and beyond) provides the :class:`torch_geometric.explain` package for first-class GNN explainability support that currently includes

#. a flexible interface to generate a variety of explanations via the :class:`~torch_geometric.explain.Explainer` class,

#. several underlying explanation algorithms including, *e.g.*, :class:`~torch_geometric.explain.algorithm.GNNExplainer`,  :class:`~torch_geometric.explain.algorithm.PGExplainer` and :class:`~torch_geometric.explain.algorithm.CaptumExplainer`,

#. support to visualize explanations via the :class:`~torch_geometric.explain.Explanation` or the :class:`~torch_geometric.explain.HeteroExplanation` class,

#. and metrics to evaluate explanations via the :class:`~torch_geometric.explain.metric` package.

.. warning::

   The explanation APIs discussed here may change in the future as we continuously work to improve their ease-of-use and generalizability.

Explainer Interface
-------------------

The :class:`torch_geometric.explain.Explainer` class is designed to handle all explainability parameters (see the :class:`~torch_geometric.explain.config.ExplainerConfig` class for more details):

#. which algorithm from the :class:`torch_geometric.explain.algorithm` module to use (*e.g.*, :class:`~torch_geometric.explain.algorithm.GNNExplainer`)

#. the type of explanation to compute, *i.e.* :obj:`explanation_type="phenomenon"` to explain the underlying phenomenon of a dataset, and :obj:`explanation_type="model"` to explain the prediction of a GNN model (see the `"GraphFramEx: Towards Systematic Evaluation of Explainability Methods for Graph Neural Networks" `_ paper for more details).

#. the different type of masks for node and edges (*e.g.*, :obj:`mask="object"` or :obj:`mask="attributes"`)

#. any postprocessing of the masks (*e.g.*, :obj:`threshold_type="topk"` or :obj:`threshold_type="hard"`)

This class allows the user to easily compare different explainability methods and to easily switch between different types of masks, while making sure the high-level framework stays the same.
The :class:`~torch_geometric.explain.Explainer` generates an :class:`~torch_geometric.explain.Explanation` or :class:`~torch_geometric.explain.HeteroExplanation` object which contains the final information about which nodes, edges and features are crucial to explain a GNN model.

.. note::

   You can read more about the :class:`torch_geometric.explain` package in this `blog post `__.

Examples
--------

In what follows, we discuss a few use-cases with corresponding code examples.

Explaining node classification on a homogeneous graph
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Assume we have a GNN :obj:`model` that does node classification on a homogeneous graph.
We can use the :class:`torch_geometric.explain.algorithm.GNNExplainer` algorithm to generate an :class:`~torch_geometric.explain.Explanation`.
We configure the :class:`~torch_geometric.explain.Explainer` to use both a :obj:`node_mask_type` and an :obj:`edge_mask_type` such that the final :class:`~torch_geometric.explain.Explanation` object contains (1) a :obj:`node_mask` (indicating which nodes and features are crucial for prediction), and (2) an :obj:`edge_mask` (indicating which edges are crucial for prediction).

.. code-block:: python

    from torch_geometric.data import Data
    from torch_geometric.explain import Explainer, GNNExplainer

    data = Data(...)  # A homogeneous graph data object.

    explainer = Explainer(
        model=model,
        algorithm=GNNExplainer(epochs=200),
        explanation_type='model',
        node_mask_type='attributes',
        edge_mask_type='object',
        model_config=dict(
            mode='multiclass_classification',
            task_level='node',
            return_type='log_probs',  # Model returns log probabilities.
        ),
    )

    # Generate explanation for the node at index `10`:
    explanation = explainer(data.x, data.edge_index, index=10)
    print(explanation.edge_mask)
    print(explanation.node_mask)

Finally, we can visualize both feature importance and the crucial subgraph of the explanation:

.. code-block:: python

    explanation.visualize_feature_importance(top_k=10)

    explanation.visualize_graph()

To evaluate the explanation from the :class:`~torch_geometric.explain.algorithm.GNNExplainer`, we can utilize the :class:`torch_geometric.explain.metric` module.
For example, to compute the :meth:`~torch_geometric.explain.metric.unfaithfulness` of an explanation, run:

.. code-block:: python

    from torch_geometric.explain import unfaithfulness

    metric = unfaithfulness(explainer, explanation)
    print(metric)

Explaining node classification on a heterogeneous graph
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Assume we have a heterogeneous GNN :obj:`model` that does node classification on a heterogeneous graph.
We can use the :class:`IntegratedGradient` attribution method from :captum:`null` `Captum `__ via the :class:`torch_geometric.explain.algorithm.CaptumExplainer` algorithm to generate a :class:`~torch_geometric.explain.HeteroExplanation`.

.. note::
    :class:`~torch_geometric.explain.algorithm.CaptumExplainer` is a wrapper around the :captum:`null` `Captum `__ library with support for most of attribution methods to explain *any* homogeneous or heterogeneous :pyg:`PyG` model.

We configure the :class:`~torch_geometric.explain.Explainer` to use both a :obj:`node_mask_type` and an :obj:`edge_mask_type` such that the final :class:`~torch_geometric.explain.HeteroExplanation` object contains (1) a :obj:`node_mask` for *each* node type (indicating which nodes and features for each node type are crucial for prediction), and (2) an :obj:`edge_mask` for *each* edge type (indicating which edges for each edge type are crucial for prediction).

.. code-block:: python

    from torch_geometric.data import HeteroData
    from torch_geometric.explain import Explainer, CaptumExplainer

    hetero_data = HeteroData(...)  # A heterogeneous graph data object.

    explainer = Explainer(
        model,  # It is assumed that model outputs a single tensor.
        algorithm=CaptumExplainer('IntegratedGradients'),
        explanation_type='model',
        node_mask_type='attributes',
        edge_mask_type='object',
        model_config = dict(
            mode='multiclass_classification',
            task_level=task_level,
            return_type='probs',  # Model returns probabilities.
        ),
    )

    # Generate batch-wise heterogeneous explanations for
    # the nodes at index `1` and `3`:
    hetero_explanation = explainer(
        hetero_data.x_dict,
        hetero_data.edge_index_dict,
        index=torch.tensor([1, 3]),
    )
    print(hetero_explanation.edge_mask_dict)
    print(hetero_explanation.node_mask_dict)

Explaining graph regression on a homogeneous graph
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Assume we have a GNN :obj:`model` that does graph regression on a homogeneous graph.
We can use the :class:`torch_geometric.explain.algorithm.PGExplainer` algorithm to generate an :class:`~torch_geometric.explain.Explanation`.
We configure the :class:`~torch_geometric.explain.Explainer` to use an :obj:`edge_mask_type` such that the final :class:`~torch_geometric.explain.Explanation` object contains an :obj:`edge_mask` (indicating which edges are crucial for prediction).
Importantly, passing a :obj:`node_mask_type` to the :class:`~torch_geometric.explain.Explainer` will throw an error since :class:`~torch_geometric.explain.algorithm.PGExplainer` cannot explain the importance of nodes:

.. code-block:: python

    from torch_geometric.data import Data
    from torch_geometric.explain import Explainer, PGExplainer

    dataset = ...
    loader = DataLoader(dataset, batch_size=1, shuffle=True)

    explainer = Explainer(
        model=model,
        algorithm=PGExplainer(epochs=30, lr=0.003),
        explanation_type='phenomenon',
        edge_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
            return_type='raw',
        ),
        # Include only the top 10 most important edges:
        threshold_config=dict(threshold_type='topk', value=10),
    )

    # PGExplainer needs to be trained separately since it is a parametric
    # explainer i.e it uses a neural network to generate explanations:
    for epoch in range(30):
        for batch in loader:
            loss = explainer.algorithm.train(
                epoch, model, batch.x, batch.edge_index, target=batch.target)

    # Generate the explanation for a particular graph:
    explanation = explainer(dataset[0].x, dataset[0].edge_index)
    print(explanation.edge_mask)

Since this feature is still undergoing heavy development, please feel free to reach out to the :pyg:`PyG` core team either on :github:`null` `GitHub `_ or :slack:`null` `Slack `_ if you have any questions, comments or concerns.
pytorch/docs/source/tutorial/heterogeneous.rst000066400000000000000000000567311444262304300222650ustar00rootroot00000000000000Heterogeneous Graph Learning
============================

A large set of real-world datasets are stored as heterogeneous graphs, motivating the introduction of specialized functionality for them in :pyg:`PyG`.
For example, most graphs in the area of recommendation, such as social graphs, are heterogeneous, as they store information about different types of entities and their different types of relations.
This tutorial introduces how heterogeneous graphs are mapped to :pyg:`PyG` and how they can be used as input to Graph Neural Network models.

Heterogeneous graphs come with different types of information attached to nodes and edges.
Thus, a single node or edge feature tensor cannot hold all node or edge features of the whole graph, due to differences in type and dimensionality.
Instead, a set of types need to be specified for nodes and edges, respectively, each having its own data tensors.
As a consequence of the different data structure, the message passing formulation changes accordingly, allowing the computation of message and update function conditioned on node or edge type.

Example Graph
-------------

As a guiding example, we take a look at the heterogeneous `ogbn-mag `__ network from the :ogb:`null` `dataset suite `_:

.. image:: ../_figures/hg_example.svg
  :align: center
  :width: 500px

The given heterogeneous graph has 1,939,743 nodes, split between the four node types **author**, **paper**, **institution** and **field of study**.
It further has 21,111,007 edges, which also are of one of four types:

* **writes**: An author *writes* a specific paper
* **affiliated with**: An author is *affiliated with* a specific institution
* **cites**: A paper *cites* another paper
* **has topic**: A paper *has a topic* of a specific field of study

The task for this graph is to infer the venue of each paper (conference or journal) given the information stored in the graph.

Creating Heterogeneous Graphs
-----------------------------

First, we can create a data object of type :class:`torch_geometric.data.HeteroData`, for which we define node feature tensors, edge index tensors and edge feature tensors individually for each type:

.. code-block:: python

   from torch_geometric.data import HeteroData

   data = HeteroData()

   data['paper'].x = ... # [num_papers, num_features_paper]
   data['author'].x = ... # [num_authors, num_features_author]
   data['institution'].x = ... # [num_institutions, num_features_institution]
   data['field_of_study'].x = ... # [num_field, num_features_field]

   data['paper', 'cites', 'paper'].edge_index = ... # [2, num_edges_cites]
   data['author', 'writes', 'paper'].edge_index = ... # [2, num_edges_writes]
   data['author', 'affiliated_with', 'institution'].edge_index = ... # [2, num_edges_affiliated]
   data['paper', 'has_topic', 'field_of_study'].edge_index = ... # [2, num_edges_topic]

   data['paper', 'cites', 'paper'].edge_attr = ... # [num_edges_cites, num_features_cites]
   data['author', 'writes', 'paper'].edge_attr = ... # [num_edges_writes, num_features_writes]
   data['author', 'affiliated_with', 'institution'].edge_attr = ... # [num_edges_affiliated, num_features_affiliated]
   data['paper', 'has_topic', 'field_of_study'].edge_attr = ... # [num_edges_topic, num_features_topic]

Node or edge tensors will be automatically created upon first access and indexed by string keys.
Node types are identified by a single string while edge types are identified by using a triplet :obj:`(source_node_type, edge_type, destination_node_type)` of strings: the edge type identifier and the two node types between which the edge type can exist.
As such, the data object allows different feature dimensionalities for each type.

Dictionaries containing the heterogeneous information grouped by attribute names rather than by node or edge type can directly be accessed via :obj:`data.{attribute_name}_dict` and serve as input to GNN models later:

.. code-block:: python

    model = HeteroGNN(...)

    output = model(data.x_dict, data.edge_index_dict, data.edge_attr_dict)

If the dataset exists in the `list of Pytorch Geometric datasets `_, it can directly be imported and used.
In particular, it will be downloaded to :obj:`root` and processed automatically.

.. code-block:: python

    from torch_geometric.datasets import OGB_MAG

    dataset = OGB_MAG(root='./data', preprocess='metapath2vec')
    data = dataset[0]

The :obj:`data` object can be printed for verification.

.. code-block:: text

    HeteroData(
      paper={
        x=[736389, 128],
        y=[736389],
        train_mask=[736389],
        val_mask=[736389],
        test_mask=[736389]
      },
      author={ x=[1134649, 128] },
      institution={ x=[8740, 128] },
      field_of_study={ x=[59965, 128] },
      (author, affiliated_with, institution)={ edge_index=[2, 1043998] },
      (author, writes, paper)={ edge_index=[2, 7145660] },
      (paper, cites, paper)={ edge_index=[2, 5416271] },
      (paper, has_topic, field_of_study)={ edge_index=[2, 7505078] }
    )

.. note::

   The original `ogbn-mag `__ network does only provide features for "paper" nodes.
   In :class:`~torch_geometric.datasets.OGB_MAG`, we provide the option to download a processed version of it in which structural features (obtained from either :obj:`"metapath2vec"` or :obj:`"TransE"`) are added to featureless nodes, as it is commonly done in the top ranked submissions to the `OGB leaderboards `_.

Utility Functions
~~~~~~~~~~~~~~~~~

The :class:`torch_geometric.data.HeteroData` class provides a number of useful utility functions to modify and analyze the given graph.

For example, single node or edge stores can be indiviually indexed:

.. code-block:: python

    paper_node_data = data['paper']
    cites_edge_data = data['paper', 'cites', 'paper']

In case the edge type can be uniquely identified by only the pair of source and destination node types or the edge type, the following operations work as well:

.. code-block:: python

    cites_edge_data = data['paper', 'paper']
    cites_edge_data = data['cites']

We can add new node types or tensors and remove them:

.. code-block:: python

    data['paper'].year = ...    # Setting a new paper attribute
    del data['field_of_study']  # Deleting 'field_of_study' node type
    del data['has_topic']       # Deleting 'has_topic' edge type

We can access the meta-data of the :obj:`data` object, holding information of all present node and edge types:

.. code-block:: python

    node_types, edge_types = data.metadata()
    print(node_types)
    ['paper', 'author', 'institution']
    print(edge_types)
    [('paper', 'cites', 'paper'),
    ('author', 'writes', 'paper'),
    ('author', 'affiliated_with', 'institution')]

The :obj:`data` object can be transferred between devices as usual:

.. code-block:: python

    data = data.to('cuda:0')
    data = data.cpu()

We further have access to additional helper functions to analyze the given graph

.. code-block:: python

    data.has_isolated_nodes()
    data.has_self_loops()
    data.is_undirected()

and can convert it to a homogeneous "typed" graph via :meth:`~torch_geometric.data.HeteroData.to_homogeneous` which is able to maintain features in case their dimensionalities match across different types:

.. code-block:: python

    homogeneous_data = data.to_homogeneous()
    print(homogeneous_data)
    Data(x=[1879778, 128], edge_index=[2, 13605929], edge_type=[13605929])

Here, :obj:`homogeneous_data.edge_type` represents an edge-level vector that holds the edge type of each edge as an integer.

Heterogeneous Graph Transformations
-----------------------------------

Most `transformations `_ for preprocessing regular graphs work as well on the heterogeneous graph :obj:`data` object.

.. code-block:: python

    import torch_geometric.transforms as T

    data = T.ToUndirected()(data)
    data = T.AddSelfLoops()(data)
    data = T.NormalizeFeatures()(data)

Here, :meth:`~torch_geometric.transforms.ToUndirected` transforms a directed graph into (the :pyg:`PyG` representation of) an undirected graph, by adding reverse edges for all edges in the graph.
Thus, future message passing is performed in both direction of all edges.
The function may add reverse edge types to the heterogeneous graph, if necessary.

For all nodes of type :obj:`'node_type'` and all existing edge types of the form :obj:`('node_type', 'edge_type', 'node_type')`, the function :meth:`~torch_geometric.transforms.AddSelfLoops` will add self-loop edges.
As a result, each node might receive one or more (one per appropriate edge type) messages from itself during message passing.

The transform :meth:`~torch_geometric.transforms.NormalizeFeatures` works like in the homogeneous case, and normalizes all specified features (of all types) to sum up to one.

Creating Heterogeneous GNNs
---------------------------

Standard Message Passing GNNs (MP-GNNs) can not trivially be applied to heterogeneous graph data, as node and edge features from different types can not be processed by the same functions due to differences in feature type.
A natural way to circumvent this is to implement message and update functions individually for each edge type.
During runtime, the MP-GNN algorithm would need to iterate over edge type dictionaries during message computation and over node type dictionaries during node updates.

To avoid unnecessary runtime overheads and to make the creation of heterogeneous MP-GNNs as simple as possible, Pytorch Geometric provides three ways for the user to create models on heterogeneous graph data:

#. Automatically convert a homogeneous GNN model to a heterogeneous GNN model by making use of :meth:`torch_geometric.nn.to_hetero` or :meth:`torch_geometric.nn.to_hetero_with_bases`
#. Define individual functions for different types using :pyg:`PyG's` wrapper :class:`torch_geometric.nn.conv.HeteroConv` for heterogeneous convolution
#. Deploy existing (or write your own) heterogeneous GNN operators

In the following, each option is introduced in detail.

Automatically Converting GNN Models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Pytorch Geometric allows to automatically convert any :pyg:`PyG` GNN model to a model for heterogeneous input graphs, using the built in functions :meth:`torch_geometric.nn.to_hetero` or :meth:`torch_geometric.nn.to_hetero_with_bases`.
The following `example `__ shows how to apply it:

.. code-block:: python

    import torch_geometric.transforms as T
    from torch_geometric.datasets import OGB_MAG
    from torch_geometric.nn import SAGEConv, to_hetero


    dataset = OGB_MAG(root='./data', preprocess='metapath2vec', transform=T.ToUndirected())
    data = dataset[0]

    class GNN(torch.nn.Module):
        def __init__(self, hidden_channels, out_channels):
            super().__init__()
            self.conv1 = SAGEConv((-1, -1), hidden_channels)
            self.conv2 = SAGEConv((-1, -1), out_channels)

        def forward(self, x, edge_index):
            x = self.conv1(x, edge_index).relu()
            x = self.conv2(x, edge_index)
            return x


    model = GNN(hidden_channels=64, out_channels=dataset.num_classes)
    model = to_hetero(model, data.metadata(), aggr='sum')

The process takes an existing GNN model and duplicates the message functions to work on each edge type individually, as detailed in the following figure.

.. image:: ../_figures/to_hetero.svg
   :align: center
   :width: 90%

As a result, the model now expects dictionaries with node and edge types as keys as input arguments, rather than single tensors utilized in homogeneous graphs.
Note that we pass in a tuple of :obj:`in_channels` to :class:`~torch_geometric.nn.conv.SAGEConv` in order to allow for message passing in bipartite graphs.

.. _lazyinit:

.. note::
   Since the number of input features and thus the size of tensors varies between different types, :pyg:`PyG` can make use of **lazy initialization** to initialize parameters in heterogeneous GNNs (as denoted by :obj:`-1` as the :obj:`in_channels` argument).
   This allows us to avoid calculating and keeping track of all tensor sizes of the computation graph.
   Lazy initialization is supported for all existing :pyg:`PyG` operators.
   We can initialize the model's parameters by calling it once:

   .. code-block:: python

        with torch.no_grad():  # Initialize lazy modules.
            out = model(data.x_dict, data.edge_index_dict)

Both :meth:`~torch_geometric.nn.to_hetero` and :meth:`~torch_geometric.nn.to_hetero_with_bases` are very flexible with respect to the homogeneous architectures that can be automatically converted to heterogeneous ones, *e.g.*, applying skip-connections, jumping knowledge or other techniques are supported out-of-the-box.
For example, this is all it takes to implement a heterogeneous graph attention network with learnable skip-connections:

.. code-block:: python

    from torch_geometric.nn import GATConv, Linear, to_hetero

    class GAT(torch.nn.Module):
        def __init__(self, hidden_channels, out_channels):
            super().__init__()
            self.conv1 = GATConv((-1, -1), hidden_channels, add_self_loops=False)
            self.lin1 = Linear(-1, hidden_channels)
            self.conv2 = GATConv((-1, -1), out_channels, add_self_loops=False)
            self.lin2 = Linear(-1, out_channels)

        def forward(self, x, edge_index):
            x = self.conv1(x, edge_index) + self.lin1(x)
            x = x.relu()
            x = self.conv2(x, edge_index) + self.lin2(x)
            return x


    model = GAT(hidden_channels=64, out_channels=dataset.num_classes)
    model = to_hetero(model, data.metadata(), aggr='sum')

Afterwards, the created model can be trained as usual:

.. _trainfunc:

.. code-block:: python

    def train():
        model.train()
        optimizer.zero_grad()
        out = model(data.x_dict, data.edge_index_dict)
        mask = data['paper'].train_mask
        loss = F.cross_entropy(out['paper'][mask], data['paper'].y[mask])
        loss.backward()
        optimizer.step()
        return float(loss)

Using the Heterogeneous Convolution Wrapper
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The heterogeneous convolution wrapper :class:`torch_geometric.nn.conv.HeteroConv` allows to define custom heterogeneous message and update functions to build arbitrary MP-GNNs for heterogeneous graphs from scratch.
While the automatic converter :meth:`~torch_geometric.nn.to_hetero` uses the same operator for all edge types, the wrapper allows to define different operators for different edge types.
Here, :class:`~torch_geometric.nn.conv.HeteroConv` takes a dictionary of submodules as input, one for each edge type in the graph data.
The following `example `__ shows how to apply it.

.. code-block:: python

    import torch_geometric.transforms as T
    from torch_geometric.datasets import OGB_MAG
    from torch_geometric.nn import HeteroConv, GCNConv, SAGEConv, GATConv, Linear


    dataset = OGB_MAG(root='./data', preprocess='metapath2vec', transform=T.ToUndirected())
    data = dataset[0]

    class HeteroGNN(torch.nn.Module):
        def __init__(self, hidden_channels, out_channels, num_layers):
            super().__init__()

            self.convs = torch.nn.ModuleList()
            for _ in range(num_layers):
                conv = HeteroConv({
                    ('paper', 'cites', 'paper'): GCNConv(-1, hidden_channels),
                    ('author', 'writes', 'paper'): SAGEConv((-1, -1), hidden_channels),
                    ('paper', 'rev_writes', 'author'): GATConv((-1, -1), hidden_channels),
                }, aggr='sum')
                self.convs.append(conv)

            self.lin = Linear(hidden_channels, out_channels)

        def forward(self, x_dict, edge_index_dict):
            for conv in self.convs:
                x_dict = conv(x_dict, edge_index_dict)
                x_dict = {key: x.relu() for key, x in x_dict.items()}
            return self.lin(x_dict['author'])

    model = HeteroGNN(hidden_channels=64, out_channels=dataset.num_classes,
                      num_layers=2)

We can initialize the model by calling it once (see :ref:`here` for more details about lazy initialization)

.. code-block:: python

    with torch.no_grad():  # Initialize lazy modules.
         out = model(data.x_dict, data.edge_index_dict)

and run the standard training procedure as outlined :ref:`here`.

Deploy Existing Heterogeneous Operators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:pyg:`PyG` provides operators (*e.g.*, :class:`torch_geometric.nn.conv.HGTConv`), which are specifically designed for heterogeneous graphs.
These operators can be directly used to build heterogeneous GNN models as can be seen in the following `example `__:

.. code-block:: python

    import torch_geometric.transforms as T
    from torch_geometric.datasets import OGB_MAG
    from torch_geometric.nn import HGTConv, Linear


    dataset = OGB_MAG(root='./data', preprocess='metapath2vec', transform=T.ToUndirected())
    data = dataset[0]

    class HGT(torch.nn.Module):
        def __init__(self, hidden_channels, out_channels, num_heads, num_layers):
            super().__init__()

            self.lin_dict = torch.nn.ModuleDict()
            for node_type in data.node_types:
                self.lin_dict[node_type] = Linear(-1, hidden_channels)

            self.convs = torch.nn.ModuleList()
            for _ in range(num_layers):
                conv = HGTConv(hidden_channels, hidden_channels, data.metadata(),
                               num_heads, group='sum')
                self.convs.append(conv)

            self.lin = Linear(hidden_channels, out_channels)

        def forward(self, x_dict, edge_index_dict):
            for node_type, x in x_dict.items():
                x_dict[node_type] = self.lin_dict[node_type](x).relu_()

            for conv in self.convs:
                x_dict = conv(x_dict, edge_index_dict)

            return self.lin(x_dict['author'])

    model = HGT(hidden_channels=64, out_channels=dataset.num_classes,
                num_heads=2, num_layers=2)

We can initialize the model by calling it once (see :ref:`here` for more details about lazy initialization).

.. code-block:: python

    with torch.no_grad():  # Initialize lazy modules.
         out = model(data.x_dict, data.edge_index_dict)

and run the standard training procedure as outlined :ref:`here`.

Heterogeneous Graph Samplers
----------------------------

:pyg:`PyG` provides various functionalities for sampling heterogeneous graphs, *i.e.* in the standard :class:`torch_geometric.loader.NeighborLoader` class  or in dedicated heterogeneous graph samplers such as :class:`torch_geometric.loader.HGTLoader`.
This is especially useful for efficient representation learning on large heterogeneous graphs, where processing the full number of neighbors is too computationally expensive.
Heterogeneous graph support for other samplers such as :class:`torch_geometric.loader.ClusterLoader` or :class:`torch_geometric.loader.GraphSAINTLoader` will be added soon.
Overall, all heterogeneous graph loaders will produce a :class:`~torch_geometric.data.HeteroData` object as output, holding a subset of the original data, and mainly differ in the way their sampling procedures works.
As such, only minimal code changes are required to convert the training procedure from :ref:`full-batch training` to mini-batch training.

Performing neighbor sampling using :class:`~torch_geometric.loader.NeighborLoader` works as outlined in the following `example `__:

.. code-block:: python

    import torch_geometric.transforms as T
    from torch_geometric.datasets import OGB_MAG
    from torch_geometric.loader import NeighborLoader

    transform = T.ToUndirected()  # Add reverse edge types.
    data = OGB_MAG(root='./data', preprocess='metapath2vec', transform=transform)[0]

    train_loader = NeighborLoader(
        data,
        # Sample 15 neighbors for each node and each edge type for 2 iterations:
        num_neighbors=[15] * 2,
        # Use a batch size of 128 for sampling training nodes of type "paper":
        batch_size=128,
        input_nodes=('paper', data['paper'].train_mask),
    )

    batch = next(iter(train_loader))


Notably, :class:`~torch_geometric.loader.NeighborLoader` works for both homogeneous *and* heterogeneous graphs.
When operating in heterogeneous graphs, more fine-grained control over the amount of sampled neighbors of individual edge types is possible, but not necessary, *e.g.*:

.. code-block:: python

    num_neighbors = {key: [15] * 2 for key in data.edge_types}

Using the :obj:`input_nodes` argument, we further specify the type and indices of nodes from which we want to sample local neighborhoods, *i.e.* all the "paper" nodes marked as training nodes according to :obj:`data['paper'].train_mask`.

Printing :obj:`batch` then yields the following output:

.. code-block:: text

    HeteroData(
      paper={
        x=[20799, 256],
        y=[20799],
        train_mask=[20799],
        val_mask=[20799],
        test_mask=[20799],
        batch_size=128
      },
      author={ x=[4419, 128] },
      institution={ x=[302, 128] },
      field_of_study={ x=[2605, 128] },
      (author, affiliated_with, institution)={ edge_index=[2, 0] },
      (author, writes, paper)={ edge_index=[2, 5927] },
      (paper, cites, paper)={ edge_index=[2, 11829] },
      (paper, has_topic, field_of_study)={ edge_index=[2, 10573] },
      (institution, rev_affiliated_with, author)={ edge_index=[2, 829] },
      (paper, rev_writes, author)={ edge_index=[2, 5512] },
      (field_of_study, rev_has_topic, paper)={ edge_index=[2, 10499] }
    )

As such, :obj:`batch` holds a total of 28,187 nodes involved for computing the embeddings of 128 "paper" nodes.
Sampled nodes are always sorted based on the order in which they were sampled.
Thus, the first :obj:`batch['paper'].batch_size` nodes represent the set of original mini-batch nodes, making it easy to obtain the final output embeddings via slicing.

Training our heterogeneous GNN model in mini-batch mode is then similar to training it in full-batch mode, except that we now iterate over the mini-batches produced by :obj:`train_loader` and optimize model parameters based on individual mini-batches:

.. code-block:: python

    def train():
        model.train()

        total_examples = total_loss = 0
        for batch in train_loader:
            optimizer.zero_grad()
            batch = batch.to('cuda:0')
            batch_size = batch['paper'].batch_size
            out = model(batch.x_dict, batch.edge_index_dict)
            loss = F.cross_entropy(out['paper'][:batch_size],
                                   batch['paper'].y[:batch_size])
            loss.backward()
            optimizer.step()

            total_examples += batch_size
            total_loss += float(loss) * batch_size

        return total_loss / total_examples

Importantly, we only make use of the first 128 "paper" nodes during loss computation.
We do so by slicing both "paper" labels :obj:`batch['paper'].y` and "paper" output predictions :obj:`out['paper']` based on :obj:`batch['paper'].batch_size`, representing the labels and final output predictions of original mini-batch nodes, respectively.
pytorch/docs/source/tutorial/load_csv.rst000066400000000000000000000247301444262304300211750ustar00rootroot00000000000000Loading Graphs from CSV
=======================

In this example, we will show how to load a set of :obj:`*.csv` files as input and construct a **heterogeneous graph** from it, which can be used as input to a `heterogeneous graph model `__.
This tutorial is also available as an executable `example script `_ in the :obj:`examples/hetero` directory.

We are going to use the `MovieLens dataset `_ collected by the GroupLens research group.
This toy dataset describes 5-star rating and tagging activity from MovieLens.
The dataset contains approximately 100k ratings across more than 9k movies from more than 600 users.
We are going to use this dataset to generate two node types holding data for **movies** and **users**, respectively, and one edge type connecting **users and movies**, representing the relation of how a user has rated a specific movie.

First, we download the dataset to an arbitrary folder (in this case, the current directory):

.. code-block:: python

    from torch_geometric.data import download_url, extract_zip

    url = 'https://files.grouplens.org/datasets/movielens/ml-latest-small.zip'
    extract_zip(download_url(url, '.'), '.')

    movie_path = './ml-latest-small/movies.csv'
    rating_path = './ml-latest-small/ratings.csv'

Before we create the heterogeneous graph, let's take a look at the data.

.. code-block:: python

    import pandas as pd

    print(pd.read_csv(movie_path).head())
    print(pd.read_csv(rating_path).head())

.. list-table:: Head of :obj:`movies.csv`
    :widths: 5 40 60
    :header-rows: 1

    * - movieId
      - title
      - genres
    * - 1
      - Toy Story (1995)
      - Adventure|Animation|Children|Comedy|Fantasy
    * - 2
      - Jumanji (1995)
      - Adventure|Children|Fantasy
    * - 3
      - Grumpier Old Men (1995)
      - Comedy|Romance
    * - 4
      - Waiting to Exhale (1995)
      - Comedy|Drama|Romance
    * - 5
      - Father of the Bride Part II (1995)
      - Comedy

We see that the :obj:`movies.csv` file provides three columns: :obj:`movieId` assigns a unique identifier to each movie, while the :obj:`title` and :obj:`genres` columns represent title and genres of the given movie.
We can make use of those two columns to define a feature representation that can be easily interpreted by machine learning models.

.. list-table:: Head of :obj:`ratings.csv`
    :widths: 5 5 10 30
    :header-rows: 1

    * - userId
      - movieId
      - rating
      - timestamp
    * - 1
      - 1
      - 4.0
      - 964982703
    * - 1
      - 3
      - 4.0
      - 964981247
    * - 1
      - 6
      - 4.0
      - 964982224
    * - 1
      - 47
      - 5.0
      - 964983815
    * - 1
      - 50
      - 5.0
      - 964982931

The :obj:`ratings.csv` data connects users (as given by :obj:`userId`) and movies (as given by :obj:`movieId`), and defines how a given user has rated a specific movie (:obj:`rating`).
Due to simplicity, we do not make use of the additional :obj:`timestamp` information.

For representing this data in the :pyg:`PyG` data format, we first define a method :meth:`load_node_csv` that reads in a :obj:`*.csv` file and returns a node-level feature representation :obj:`x` of shape :obj:`[num_nodes, num_features]`:

.. code-block:: python

    import torch

    def load_node_csv(path, index_col, encoders=None, **kwargs):
        df = pd.read_csv(path, index_col=index_col, **kwargs)
        mapping = {index: i for i, index in enumerate(df.index.unique())}

        x = None
        if encoders is not None:
            xs = [encoder(df[col]) for col, encoder in encoders.items()]
            x = torch.cat(xs, dim=-1)

        return x, mapping

Here, :meth:`load_node_csv` reads the :obj:`*.csv` file from :obj:`path`, and creates a dictionary :obj:`mapping` that maps its index column to a consecutive value in the range :obj:`{ 0, ..., num_rows - 1 }`.
This is needed as we want our final data representation to be as compact as possible, *e.g.*, the representation of a movie in the first row should be accessible via :obj:`x[0]`.

We further utilize the concept of encoders, which define how the values of specific columns should be encoded into a numerical feature representation.
For example, we can define a sentence encoder that encodes raw column strings into low-dimensional embeddings.
For this, we make use of the excellent `sentence-transformers `_ library which provides a large number of state-of-the-art pretrained NLP embedding models:

.. code-block:: bash

    pip install sentence-transformers

.. code-block:: python

    class SequenceEncoder:
        def __init__(self, model_name='all-MiniLM-L6-v2', device=None):
            self.device = device
            self.model = SentenceTransformer(model_name, device=device)

        @torch.no_grad()
        def __call__(self, df):
            x = self.model.encode(df.values, show_progress_bar=True,
                                  convert_to_tensor=True, device=self.device)
            return x.cpu()

The :class:`SequenceEncoder` class loads a pre-trained NLP model as given by :obj:`model_name`, and uses it to encode a list of strings into a :pytorch:`PyTorch` tensor of shape :obj:`[num_strings, embedding_dim]`.
We can use this :class:`SequenceEncoder` to encode the :obj:`title` of the :obj:`movies.csv` file.

In a similar fashion, we can create another encoder that converts the genres of movies, *e.g.*, :obj:`Adventure|Children|Fantasy`, into categorical labels.
For this, we first need to find all existing genres present in the data, create a feature representation :obj:`x` of shape :obj:`[num_movies, num_genres]`, and assign a :obj:`1` to :obj:`x[i, j]` in case the genre :obj:`j` is present in movie :obj:`i`:

.. code-block:: python

    class GenresEncoder:
        def __init__(self, sep='|'):
            self.sep = sep

        def __call__(self, df):
            genres = set(g for col in df.values for g in col.split(self.sep))
            mapping = {genre: i for i, genre in enumerate(genres)}

            x = torch.zeros(len(df), len(mapping))
            for i, col in enumerate(df.values):
                for genre in col.split(self.sep):
                    x[i, mapping[genre]] = 1
            return x

With this, we can obtain our final representation of movies via:

.. code-block:: python

    movie_x, movie_mapping = load_node_csv(
        movie_path, index_col='movieId', encoders={
            'title': SequenceEncoder(),
            'genres': GenresEncoder()
        })

Similarly, we can utilize :meth:`load_node_csv` for obtaining a user mapping from :obj:`userId` to consecutive values as well.
However, there is no additional feature information for users present in this dataset.
As such, we do not define any encoders:

.. code-block:: python

    _, user_mapping = load_node_csv(rating_path, index_col='userId')

With this, we are ready to initialize our :class:`~torch_geometric.data.HeteroData` object and pass two node types into it:

.. code-block:: python

    from torch_geometric.data import HeteroData

    data = HeteroData()

    data['user'].num_nodes = len(user_mapping)  # Users do not have any features.
    data['movie'].x = movie_x

    print(data)
    HeteroData(
      user={ num_nodes=610 },
      movie={ x[9742, 404] }
    )

As users do not have any node-level information, we solely define its number of nodes.
As a result, we likely need to learn distinct user embeddings via :class:`torch.nn.Embedding` in an end-to-end fashion during training of a heterogeneous graph model.

Next, we take a look at connecting users with movies as defined by their ratings.
For this, we define a method :meth:`load_edge_csv` that returns the final :obj:`edge_index` representation of shape :obj:`[2, num_ratings]` from :obj:`ratings.csv`, as well as any additional features present in the raw :obj:`*.csv` file:

.. code-block:: python

    def load_edge_csv(path, src_index_col, src_mapping, dst_index_col, dst_mapping,
                      encoders=None, **kwargs):
        df = pd.read_csv(path, **kwargs)

        src = [src_mapping[index] for index in df[src_index_col]]
        dst = [dst_mapping[index] for index in df[dst_index_col]]
        edge_index = torch.tensor([src, dst])

        edge_attr = None
        if encoders is not None:
            edge_attrs = [encoder(df[col]) for col, encoder in encoders.items()]
            edge_attr = torch.cat(edge_attrs, dim=-1)

        return edge_index, edge_attr

Here, :obj:`src_index_col` and :obj:`dst_index_col` define the index columns of source and destination nodes, respectively.
We further make use of the node-level mappings :obj:`src_mapping` and :obj:`dst_mapping` to ensure that raw indices are mapped to the correct consecutive indices in our final representation.
For every edge defined in the file, it looks up the forward indices in :obj:`src_mapping` and :obj:`dst_mapping`, and moves the data appropriately.

Similarly to :meth:`load_node_csv`, encoders are used to return additional edge-level feature information.
For example, for loading the ratings from the :obj:`rating` column in :obj:`ratings.csv`, we can define an :class:`IdentityEncoder` that simply converts a list of floating-point values into a :pytorch:`PyTorch` tensor:

.. code-block:: python

    class IdentityEncoder:
        def __init__(self, dtype=None):
            self.dtype = dtype

        def __call__(self, df):
            return torch.from_numpy(df.values).view(-1, 1).to(self.dtype)

With this, we are ready to finalize our :class:`~torch_geometric.data.HeteroData` object:

.. code-block:: python

    edge_index, edge_label = load_edge_csv(
        rating_path,
        src_index_col='userId',
        src_mapping=user_mapping,
        dst_index_col='movieId',
        dst_mapping=movie_mapping,
        encoders={'rating': IdentityEncoder(dtype=torch.long)},
    )

    data['user', 'rates', 'movie'].edge_index = edge_index
    data['user', 'rates', 'movie'].edge_label = edge_label

    print(data)
    HeteroData(
      user={ num_nodes=610 },
      movie={ x=[9742, 404] },
      (user, rates, movie)={
        edge_index=[2, 100836],
        edge_label=[100836, 1]
      }
    )

This :class:`~torch_geometric.data.HeteroData` object is the native format of heterogeneous graphs in :pyg:`PyG` and can be used as input for `heterogeneous graph models `__.

.. note::

    Click `here `_ to see the final example script.
pytorch/examples/000077500000000000000000000000001444262304300143665ustar00rootroot00000000000000pytorch/examples/agnn.py000066400000000000000000000036451444262304300156730ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import AGNNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = torch.nn.Linear(dataset.num_features, 16)
        self.prop1 = AGNNConv(requires_grad=False)
        self.prop2 = AGNNConv(requires_grad=True)
        self.lin2 = torch.nn.Linear(16, dataset.num_classes)

    def forward(self):
        x = F.dropout(data.x, training=self.training)
        x = F.relu(self.lin1(x))
        x = self.prop1(x, data.edge_index)
        x = self.prop2(x, data.edge_index)
        x = F.dropout(x, training=self.training)
        x = self.lin2(x)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 201):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/argva_node_clustering.py000066400000000000000000000102151444262304300213030ustar00rootroot00000000000000import os.path as osp

import matplotlib.pyplot as plt
import torch
from sklearn.cluster import KMeans
from sklearn.manifold import TSNE
from sklearn.metrics.cluster import (
    completeness_score,
    homogeneity_score,
    v_measure_score,
)
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import ARGVA, GCNConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.ToDevice(device),
    T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,
                      split_labels=True, add_negative_train_samples=False),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=transform)
train_data, val_data, test_data = dataset[0]


class Encoder(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv_mu = GCNConv(hidden_channels, out_channels)
        self.conv_logstd = GCNConv(hidden_channels, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv_mu(x, edge_index), self.conv_logstd(x, edge_index)


class Discriminator(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.lin1 = Linear(in_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, hidden_channels)
        self.lin3 = Linear(hidden_channels, out_channels)

    def forward(self, x):
        x = self.lin1(x).relu()
        x = self.lin2(x).relu()
        return self.lin3(x)


encoder = Encoder(train_data.num_features, hidden_channels=32, out_channels=32)
discriminator = Discriminator(in_channels=32, hidden_channels=64,
                              out_channels=32)
model = ARGVA(encoder, discriminator).to(device)

encoder_optimizer = torch.optim.Adam(encoder.parameters(), lr=0.005)
discriminator_optimizer = torch.optim.Adam(discriminator.parameters(),
                                           lr=0.001)


def train():
    model.train()
    encoder_optimizer.zero_grad()
    z = model.encode(train_data.x, train_data.edge_index)

    # We optimize the discriminator more frequently than the encoder.
    for i in range(5):
        discriminator_optimizer.zero_grad()
        discriminator_loss = model.discriminator_loss(z)
        discriminator_loss.backward()
        discriminator_optimizer.step()

    loss = model.recon_loss(z, train_data.pos_edge_label_index)
    loss = loss + model.reg_loss(z)
    loss = loss + (1 / train_data.num_nodes) * model.kl_loss()
    loss.backward()
    encoder_optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    z = model.encode(data.x, data.edge_index)

    # Cluster embedded values using k-means.
    kmeans_input = z.cpu().numpy()
    kmeans = KMeans(n_clusters=7, random_state=0).fit(kmeans_input)
    pred = kmeans.predict(kmeans_input)

    labels = data.y.cpu().numpy()
    completeness = completeness_score(labels, pred)
    hm = homogeneity_score(labels, pred)
    nmi = v_measure_score(labels, pred)

    auc, ap = model.test(z, data.pos_edge_label_index,
                         data.neg_edge_label_index)

    return auc, ap, completeness, hm, nmi


for epoch in range(1, 151):
    loss = train()
    auc, ap, completeness, hm, nmi = test(test_data)
    print((f'Epoch: {epoch:03d}, Loss: {loss:.3f}, AUC: {auc:.3f}, '
           f'AP: {ap:.3f}, Completeness: {completeness:.3f}, '
           f'Homogeneity: {hm:.3f}, NMI: {nmi:.3f}'))


@torch.no_grad()
def plot_points(data, colors):
    model.eval()
    z = model.encode(data.x, data.edge_index)
    z = TSNE(n_components=2).fit_transform(z.cpu().numpy())
    y = data.y.cpu().numpy()

    plt.figure(figsize=(8, 8))
    for i in range(dataset.num_classes):
        plt.scatter(z[y == i, 0], z[y == i, 1], s=20, color=colors[i])
    plt.axis('off')
    plt.show()


colors = [
    '#ffc0cb', '#bada55', '#008080', '#420420', '#7fe5f0', '#065535', '#ffd700'
]
plot_points(test_data, colors)
pytorch/examples/arma.py000066400000000000000000000042351444262304300156640ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import ARMAConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()

        self.conv1 = ARMAConv(in_channels, hidden_channels, num_stacks=3,
                              num_layers=2, shared_weights=True, dropout=0.25)

        self.conv2 = ARMAConv(hidden_channels, out_channels, num_stacks=3,
                              num_layers=2, shared_weights=True, dropout=0.25,
                              act=lambda x: x)

    def forward(self, x, edge_index):
        x = F.dropout(x, training=self.training)
        x = F.relu(self.conv1(x, edge_index))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net(dataset.num_features, 16,
                  dataset.num_classes).to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()


def test():
    model.eval()
    out, accs = model(data.x, data.edge_index), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 401):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/attentive_fp.py000066400000000000000000000124371444262304300174370ustar00rootroot00000000000000import os.path as osp
from math import sqrt

import torch
import torch.nn.functional as F
from rdkit import Chem

from torch_geometric.datasets import MoleculeNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn.models import AttentiveFP


class GenFeatures:
    def __init__(self):
        self.symbols = [
            'B', 'C', 'N', 'O', 'F', 'Si', 'P', 'S', 'Cl', 'As', 'Se', 'Br',
            'Te', 'I', 'At', 'other'
        ]

        self.hybridizations = [
            Chem.rdchem.HybridizationType.SP,
            Chem.rdchem.HybridizationType.SP2,
            Chem.rdchem.HybridizationType.SP3,
            Chem.rdchem.HybridizationType.SP3D,
            Chem.rdchem.HybridizationType.SP3D2,
            'other',
        ]

        self.stereos = [
            Chem.rdchem.BondStereo.STEREONONE,
            Chem.rdchem.BondStereo.STEREOANY,
            Chem.rdchem.BondStereo.STEREOZ,
            Chem.rdchem.BondStereo.STEREOE,
        ]

    def __call__(self, data):
        # Generate AttentiveFP features according to Table 1.
        mol = Chem.MolFromSmiles(data.smiles)

        xs = []
        for atom in mol.GetAtoms():
            symbol = [0.] * len(self.symbols)
            symbol[self.symbols.index(atom.GetSymbol())] = 1.
            degree = [0.] * 6
            degree[atom.GetDegree()] = 1.
            formal_charge = atom.GetFormalCharge()
            radical_electrons = atom.GetNumRadicalElectrons()
            hybridization = [0.] * len(self.hybridizations)
            hybridization[self.hybridizations.index(
                atom.GetHybridization())] = 1.
            aromaticity = 1. if atom.GetIsAromatic() else 0.
            hydrogens = [0.] * 5
            hydrogens[atom.GetTotalNumHs()] = 1.
            chirality = 1. if atom.HasProp('_ChiralityPossible') else 0.
            chirality_type = [0.] * 2
            if atom.HasProp('_CIPCode'):
                chirality_type[['R', 'S'].index(atom.GetProp('_CIPCode'))] = 1.

            x = torch.tensor(symbol + degree + [formal_charge] +
                             [radical_electrons] + hybridization +
                             [aromaticity] + hydrogens + [chirality] +
                             chirality_type)
            xs.append(x)

        data.x = torch.stack(xs, dim=0)

        edge_indices = []
        edge_attrs = []
        for bond in mol.GetBonds():
            edge_indices += [[bond.GetBeginAtomIdx(), bond.GetEndAtomIdx()]]
            edge_indices += [[bond.GetEndAtomIdx(), bond.GetBeginAtomIdx()]]

            bond_type = bond.GetBondType()
            single = 1. if bond_type == Chem.rdchem.BondType.SINGLE else 0.
            double = 1. if bond_type == Chem.rdchem.BondType.DOUBLE else 0.
            triple = 1. if bond_type == Chem.rdchem.BondType.TRIPLE else 0.
            aromatic = 1. if bond_type == Chem.rdchem.BondType.AROMATIC else 0.
            conjugation = 1. if bond.GetIsConjugated() else 0.
            ring = 1. if bond.IsInRing() else 0.
            stereo = [0.] * 4
            stereo[self.stereos.index(bond.GetStereo())] = 1.

            edge_attr = torch.tensor(
                [single, double, triple, aromatic, conjugation, ring] + stereo)

            edge_attrs += [edge_attr, edge_attr]

        if len(edge_attrs) == 0:
            data.edge_index = torch.zeros((2, 0), dtype=torch.long)
            data.edge_attr = torch.zeros((0, 10), dtype=torch.float)
        else:
            data.edge_index = torch.tensor(edge_indices).t().contiguous()
            data.edge_attr = torch.stack(edge_attrs, dim=0)

        return data


path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'AFP_Mol')
dataset = MoleculeNet(path, name='ESOL', pre_transform=GenFeatures()).shuffle()

N = len(dataset) // 10
val_dataset = dataset[:N]
test_dataset = dataset[N:2 * N]
train_dataset = dataset[2 * N:]

train_loader = DataLoader(train_dataset, batch_size=200, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=200)
test_loader = DataLoader(test_dataset, batch_size=200)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = AttentiveFP(in_channels=39, hidden_channels=200, out_channels=1,
                    edge_dim=10, num_layers=2, num_timesteps=2,
                    dropout=0.2).to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=10**-2.5,
                             weight_decay=10**-5)


def train():
    total_loss = total_examples = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
        loss = F.mse_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs
        total_examples += data.num_graphs
    return sqrt(total_loss / total_examples)


@torch.no_grad()
def test(loader):
    mse = []
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
        mse.append(F.mse_loss(out, data.y, reduction='none').cpu())
    return float(torch.cat(mse, dim=0).mean().sqrt())


for epoch in range(1, 201):
    train_rmse = train()
    val_rmse = test(val_loader)
    test_rmse = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {train_rmse:.4f} Val: {val_rmse:.4f} '
          f'Test: {test_rmse:.4f}')
pytorch/examples/autoencoder.py000066400000000000000000000071631444262304300172570ustar00rootroot00000000000000import argparse
import os.path as osp

import torch

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GAE, VGAE, GCNConv

parser = argparse.ArgumentParser()
parser.add_argument('--variational', action='store_true')
parser.add_argument('--linear', action='store_true')
parser.add_argument('--dataset', type=str, default='Cora',
                    choices=['Cora', 'CiteSeer', 'PubMed'])
parser.add_argument('--epochs', type=int, default=400)
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.NormalizeFeatures(),
    T.ToDevice(device),
    T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,
                      split_labels=True, add_negative_train_samples=False),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, args.dataset, transform=transform)
train_data, val_data, test_data = dataset[0]


class GCNEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, 2 * out_channels)
        self.conv2 = GCNConv(2 * out_channels, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)


class VariationalGCNEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, 2 * out_channels)
        self.conv_mu = GCNConv(2 * out_channels, out_channels)
        self.conv_logstd = GCNConv(2 * out_channels, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv_mu(x, edge_index), self.conv_logstd(x, edge_index)


class LinearEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = GCNConv(in_channels, out_channels)

    def forward(self, x, edge_index):
        return self.conv(x, edge_index)


class VariationalLinearEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv_mu = GCNConv(in_channels, out_channels)
        self.conv_logstd = GCNConv(in_channels, out_channels)

    def forward(self, x, edge_index):
        return self.conv_mu(x, edge_index), self.conv_logstd(x, edge_index)


in_channels, out_channels = dataset.num_features, 16

if not args.variational and not args.linear:
    model = GAE(GCNEncoder(in_channels, out_channels))
elif not args.variational and args.linear:
    model = GAE(LinearEncoder(in_channels, out_channels))
elif args.variational and not args.linear:
    model = VGAE(VariationalGCNEncoder(in_channels, out_channels))
elif args.variational and args.linear:
    model = VGAE(VariationalLinearEncoder(in_channels, out_channels))

model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()
    optimizer.zero_grad()
    z = model.encode(train_data.x, train_data.edge_index)
    loss = model.recon_loss(z, train_data.pos_edge_label_index)
    if args.variational:
        loss = loss + (1 / train_data.num_nodes) * model.kl_loss()
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    z = model.encode(data.x, data.edge_index)
    return model.test(z, data.pos_edge_label_index, data.neg_edge_label_index)


for epoch in range(1, args.epochs + 1):
    loss = train()
    auc, ap = test(test_data)
    print(f'Epoch: {epoch:03d}, AUC: {auc:.4f}, AP: {ap:.4f}')
pytorch/examples/cluster_gcn_ppi.py000066400000000000000000000060641444262304300201260ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import f1_score

from torch_geometric.data import Batch
from torch_geometric.datasets import PPI
from torch_geometric.loader import ClusterData, ClusterLoader, DataLoader
from torch_geometric.nn import BatchNorm, SAGEConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')

train_data = Batch.from_data_list(train_dataset)
cluster_data = ClusterData(train_data, num_parts=50, recursive=False,
                           save_dir=train_dataset.processed_dir)
train_loader = ClusterLoader(cluster_data, batch_size=1, shuffle=True,
                             num_workers=12)

val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers):
        super().__init__()
        self.convs = torch.nn.ModuleList()
        self.batch_norms = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        self.batch_norms.append(BatchNorm(hidden_channels))
        for _ in range(num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
            self.batch_norms.append(BatchNorm(hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x, edge_index):
        for conv, batch_norm in zip(self.convs[:-1], self.batch_norms):
            x = conv(x, edge_index)
            x = batch_norm(x)
            x = F.relu(x)
            x = F.dropout(x, p=0.2, training=self.training)
        return self.convs[-1](x, edge_index)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(in_channels=train_dataset.num_features, hidden_channels=1024,
            out_channels=train_dataset.num_classes, num_layers=6).to(device)
loss_op = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = loss_op(model(data.x, data.edge_index), data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * data.num_nodes
    return total_loss / train_data.num_nodes


@torch.no_grad()
def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        out = model(data.x.to(device), data.edge_index.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 201):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/cluster_gcn_reddit.py000066400000000000000000000066751444262304300206210ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch.nn import ModuleList
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import ClusterData, ClusterLoader, NeighborLoader
from torch_geometric.nn import SAGEConv

dataset = Reddit('../data/Reddit')
data = dataset[0]

cluster_data = ClusterData(data, num_parts=1500, recursive=False,
                           save_dir=dataset.processed_dir)
train_loader = ClusterLoader(cluster_data, batch_size=20, shuffle=True,
                             num_workers=12)

subgraph_loader = NeighborLoader(data, num_neighbors=[-1], batch_size=1024,
                                 shuffle=False, num_workers=12)


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.convs = ModuleList(
            [SAGEConv(in_channels, 128),
             SAGEConv(128, out_channels)])

    def forward(self, x, edge_index):
        for i, conv in enumerate(self.convs):
            x = conv(x, edge_index)
            if i != len(self.convs) - 1:
                x = F.relu(x)
                x = F.dropout(x, p=0.5, training=self.training)
        return F.log_softmax(x, dim=-1)

    def inference(self, x_all):
        pbar = tqdm(total=x_all.size(0) * len(self.convs))
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch.
        for i, conv in enumerate(self.convs):
            xs = []
            for batch in subgraph_loader:
                edge_index = batch.edge_index.to(device)
                x = x_all[batch.n_id].to(device)
                x_target = x[:batch.batch_size]
                x = conv((x, x_target), edge_index)
                if i != len(self.convs) - 1:
                    x = F.relu(x)
                xs.append(x.cpu())

                pbar.update(batch.batch_size)

            x_all = torch.cat(xs, dim=0)

        pbar.close()

        return x_all


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)


def train():
    model.train()

    total_loss = total_nodes = 0
    for batch in train_loader:
        batch = batch.to(device)
        optimizer.zero_grad()
        out = model(batch.x, batch.edge_index)
        loss = F.nll_loss(out[batch.train_mask], batch.y[batch.train_mask])
        loss.backward()
        optimizer.step()

        nodes = batch.train_mask.sum().item()
        total_loss += loss.item() * nodes
        total_nodes += nodes

    return total_loss / total_nodes


@torch.no_grad()
def test():  # Inference should be performed on the full graph.
    model.eval()

    out = model.inference(data.x)
    y_pred = out.argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        correct = y_pred[mask].eq(data.y[mask]).sum().item()
        accs.append(correct / mask.sum().item())
    return accs


for epoch in range(1, 31):
    loss = train()
    if epoch % 5 == 0:
        train_acc, val_acc, test_acc = test()
        print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
              f'Val: {val_acc:.4f}, test: {test_acc:.4f}')
    else:
        print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}')
pytorch/examples/colors_topk_pool.py000066400000000000000000000073611444262304300203360ustar00rootroot00000000000000import copy
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GINConv, TopKPooling, global_add_pool
from torch_geometric.utils import scatter


class HandleNodeAttention:
    def __call__(self, data):
        data = copy.copy(data)
        data.attn = torch.softmax(data.x[:, 0], dim=0)
        data.x = data.x[:, 1:]
        return data


path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'COLORS-3')
dataset = TUDataset(path, 'COLORS-3', use_node_attr=True,
                    transform=HandleNodeAttention())

train_loader = DataLoader(dataset[:500], batch_size=60, shuffle=True)
val_loader = DataLoader(dataset[500:3000], batch_size=60)
test_loader = DataLoader(dataset[3000:], batch_size=60)


class Net(torch.nn.Module):
    def __init__(self, in_channels):
        super().__init__()

        self.conv1 = GINConv(Seq(Lin(in_channels, 64), ReLU(), Lin(64, 64)))
        self.pool1 = TopKPooling(in_channels, min_score=0.05)
        self.conv2 = GINConv(Seq(Lin(64, 64), ReLU(), Lin(64, 64)))

        self.lin = torch.nn.Linear(64, 1)

    def forward(self, data):
        x, edge_index, batch = data.x, data.edge_index, data.batch

        out = F.relu(self.conv1(x, edge_index))

        out, edge_index, _, batch, perm, score = self.pool1(
            out, edge_index, None, batch, attn=x)
        ratio = out.size(0) / x.size(0)

        out = F.relu(self.conv2(out, edge_index))
        out = global_add_pool(out, batch)
        out = self.lin(out).view(-1)

        attn_loss = F.kl_div(torch.log(score + 1e-14), data.attn[perm],
                             reduction='none')
        attn_loss = scatter(attn_loss, batch, reduce='mean')

        return out, attn_loss, ratio


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Initialize to optimal attention weights:
# model.pool1.weight.data = torch.tensor([0., 1., 0., 0.]).view(1,4).to(device)


def train(epoch):
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out, attn_loss, _ = model(data)
        loss = ((out - data.y).pow(2) + 100 * attn_loss).mean()
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()

    return total_loss / len(train_loader.dataset)


def test(loader):
    model.eval()

    corrects, total_ratio = [], 0
    for data in loader:
        data = data.to(device)
        out, _, ratio = model(data)
        pred = out.round().to(torch.long)
        corrects.append(pred.eq(data.y.to(torch.long)))
        total_ratio += ratio
    return torch.cat(corrects, dim=0), total_ratio / len(loader)


for epoch in range(1, 301):
    loss = train(epoch)
    train_correct, train_ratio = test(train_loader)
    val_correct, val_ratio = test(val_loader)
    test_correct, test_ratio = test(test_loader)

    train_acc = train_correct.sum().item() / train_correct.size(0)
    val_acc = val_correct.sum().item() / val_correct.size(0)

    test_acc1 = test_correct[:2500].sum().item() / 2500
    test_acc2 = test_correct[2500:5000].sum().item() / 2500
    test_acc3 = test_correct[5000:].sum().item() / 2500

    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.3f}, '
          f'Val: {val_acc:.3f}, Test Orig: {test_acc1:.3f}, '
          f'Test Large: {test_acc2:.3f}, Test LargeC: {test_acc3:.3f}, '
          f'Train/Val/Test Ratio='
          f'{train_ratio:.3f}/{val_ratio:.3f}/{test_ratio:.3f}')
pytorch/examples/compile/000077500000000000000000000000001444262304300160165ustar00rootroot00000000000000pytorch/examples/compile/gcn.py000066400000000000000000000034751444262304300171500ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric
import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join('data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=T.NormalizeFeatures())
data = dataset[0].to(device)


class GCN(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, 16)
        self.conv2 = GCNConv(16, out_channels)

    def forward(self, x, edge_index):
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        return x


model = GCN(dataset.num_features, dataset.num_classes).to(device)

# Compile the model into an optimized version:
model = torch_geometric.compile(model)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


for epoch in range(1, 201):
    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/compile/gin.py000066400000000000000000000052231444262304300171470ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric
import torch_geometric.transforms as T
from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, GINConv, global_add_pool

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join('data', 'TUDatasets')
transform = T.Pad(max_num_nodes=28, max_num_edges=66)
dataset = TUDataset(path, name='MUTAG', pre_transform=transform).shuffle()

train_dataset = dataset[len(dataset) // 10:]
train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True,
                          drop_last=True)

test_dataset = dataset[:len(dataset) // 10]
test_loader = DataLoader(test_dataset, batch_size=128)


class GIN(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        for _ in range(5):
            mlp = MLP([in_channels, 32, 32])
            self.convs.append(GINConv(mlp, train_eps=False))
            in_channels = 32

        self.mlp = MLP([32, 32, out_channels], norm=None, dropout=0.5)

    def forward(self, x, edge_index, batch):
        for conv in self.convs:
            x = conv(x, edge_index).relu()
        x = global_add_pool(x, batch)
        return self.mlp(x)


model = GIN(dataset.num_features, dataset.num_classes).to(device)

# Compile the model into an optimized version:
# Note that `compile(model, dynamic=True)` does not work yet in PyTorch 2.0, so
# we use `transforms.Pad` and static compilation as a current workaround.
# See: https://github.com/pytorch/pytorch/issues/94640
model = torch_geometric.compile(model)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = F.cross_entropy(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = 0
    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.edge_index, data.batch).argmax(dim=-1)
        total_correct += int((pred == data.y).sum())
    return total_correct / len(loader.dataset)


for epoch in range(1, 101):
    loss = train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/contrib/000077500000000000000000000000001444262304300160265ustar00rootroot00000000000000pytorch/examples/contrib/graphmask_explainer.py000066400000000000000000000060321444262304300224250ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.contrib.explain import GraphMaskExplainer
from torch_geometric.datasets import Planetoid
from torch_geometric.explain import Explainer
from torch_geometric.nn import GATConv, GCNConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora')
data = dataset[0].to(device)

# GCN Node Classification =====================================================


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


model = GCN().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

for epoch in range(1, 201):
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()

explainer = Explainer(
    model=model,
    algorithm=GraphMaskExplainer(2, epochs=5),
    explanation_type='model',
    node_mask_type='attributes',
    edge_mask_type='object',
    model_config=dict(
        mode='multiclass_classification',
        task_level='node',
        return_type='log_probs',
    ),
)

node_index = 10
explanation = explainer(data.x, data.edge_index, index=node_index)
print(f'Generated explanations in {explanation.available_explanations}')

# GAT Node Classification =====================================================


class GAT(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GATConv(dataset.num_features, 8, heads=8)
        self.conv2 = GATConv(64, dataset.num_classes, heads=1, concat=False)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


model = GAT().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

for epoch in range(1, 201):
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()

explainer = Explainer(
    model=model,
    algorithm=GraphMaskExplainer(2, epochs=5),
    explanation_type='model',
    node_mask_type='attributes',
    edge_mask_type='object',
    model_config=dict(
        mode='multiclass_classification',
        task_level='node',
        return_type='log_probs',
    ),
)

node_index = torch.tensor([10, 20])
explanation = explainer(data.x, data.edge_index, index=node_index)
print(f'Generated explanations in {explanation.available_explanations}')
pytorch/examples/contrib/pgm_explainer_graph_classification.py000066400000000000000000000076401444262304300254750ustar00rootroot00000000000000"""
This is an example of using the PGM explainer algorithm
on a graph classification task
"""
import os.path as osp

import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.contrib.explain import PGMExplainer
from torch_geometric.datasets import MNISTSuperpixels
from torch_geometric.explain import Explainer
from torch_geometric.loader import DataLoader
from torch_geometric.nn import (
    NNConv,
    global_mean_pool,
    graclus,
    max_pool,
    max_pool_x,
)
from torch_geometric.utils import normalized_cut

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')
transform = T.Cartesian(cat=False)
train_dataset = MNISTSuperpixels(path, True, transform=transform)
test_dataset = MNISTSuperpixels(path, False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=512, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False)
d = train_dataset


def normalized_cut_2d(edge_index, pos):
    row, col = edge_index
    edge_attr = torch.norm(pos[row] - pos[col], p=2, dim=1)
    return normalized_cut(edge_index, edge_attr, num_nodes=pos.size(0))


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        nn1 = nn.Sequential(nn.Linear(2, 25), nn.ReLU(),
                            nn.Linear(25, d.num_features * 32))
        self.conv1 = NNConv(d.num_features, 32, nn1, aggr='mean')

        nn2 = nn.Sequential(nn.Linear(2, 25), nn.ReLU(),
                            nn.Linear(25, 32 * 64))
        self.conv2 = NNConv(32, 64, nn2, aggr='mean')

        self.fc1 = torch.nn.Linear(64, 128)
        self.fc2 = torch.nn.Linear(128, d.num_classes)

    def forward(self, x, edge_index, **kwargs):
        data = kwargs.get('data')
        data = data.detach().clone()
        x = F.elu(self.conv1(x, edge_index, data.edge_attr))
        weight = normalized_cut_2d(edge_index, data.pos)
        cluster = graclus(edge_index, weight, x.size(0))
        data.edge_attr = None
        data.x = x
        data.edge_index = edge_index
        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv2(data.x, data.edge_index, data.edge_attr))
        weight = normalized_cut_2d(data.edge_index, data.pos)
        cluster = graclus(data.edge_index, weight, data.x.size(0))
        x, batch = max_pool_x(cluster, data.x, data.batch)

        x = global_mean_pool(x, batch)
        x = F.elu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        return F.log_softmax(self.fc2(x), dim=1)


def train(model, dataloader):
    model.train()

    for data in dataloader:
        data = data.to(device)
        optimizer.zero_grad()
        F.nll_loss(model(data.x, data), data.y).backward()
        optimizer.step()


if __name__ == "__main__":

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    print(f'current device: {device}')
    model = Net().to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

    for epoch in range(2):
        train(model, train_loader)

    explainer = Explainer(
        model=model, algorithm=PGMExplainer(perturb_feature_list=[0],
                                            perturbation_mode="mean"),
        explanation_type='phenomenon', node_mask_type="object",
        model_config=dict(mode="multiclass_classification", task_level="graph",
                          return_type="raw"))
    i = 0

    for explain_dataset in test_loader:
        explain_dataset.to(device)
        explanation = explainer(x=explain_dataset.x,
                                edge_index=explain_dataset.edge_index,
                                target=explain_dataset.y,
                                edge_attr=explain_dataset.edge_attr,
                                data=explain_dataset)
        for k in explanation.available_explanations:
            print(explanation[k])
        i += 1
        if i > 2:
            break
pytorch/examples/contrib/pgm_explainer_node_classification.py000066400000000000000000000045341444262304300253200ustar00rootroot00000000000000"""
This is an example of using the PGM explainer algorithm
on a node classification task
"""
import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.contrib.explain import PGMExplainer
from torch_geometric.datasets import Planetoid
from torch_geometric.explain import Explainer, ModelConfig
from torch_geometric.nn import GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
transform = T.Compose([T.GCNNorm(), T.NormalizeFeatures()])
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, 16, normalize=False)
        self.conv2 = GCNConv(16, dataset.num_classes, normalize=False)

    def forward(self, x, edge_index, edge_weight):
        x = F.relu(self.conv1(x, edge_index, edge_weight))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_weight)
        return F.log_softmax(x, dim=1)


if __name__ == "__main__":
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = Net().to(device)
    data = data.to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01,
                                 weight_decay=5e-4)
    x, edge_index, edge_weight, target = \
        data.x, data.edge_index, data.edge_weight, data.y

    model.train()
    for epoch in range(1, 500):
        optimizer.zero_grad()
        log_logits = model(x, edge_index, edge_weight)
        loss = F.nll_loss(log_logits[data.train_mask], data.y[data.train_mask])
        loss.backward()
        optimizer.step()

    model.eval()
    log_logits = model(x, edge_index, edge_weight)
    predicted_target = log_logits.argmax(dim=1)

    explainer = Explainer(
        model=model, algorithm=PGMExplainer(), node_mask_type='attributes',
        explanation_type='phenomenon',
        model_config=ModelConfig(mode='multiclass_classification',
                                 task_level='node', return_type='raw'))
    node_idx = 100
    explanation = explainer(x=data.x, edge_index=edge_index, index=node_idx,
                            target=predicted_target, edge_weight=edge_weight)
    print(f'Significance of relevant neighbors: {explanation.pgm_stats}')
pytorch/examples/contrib/rbcd_attack.py000066400000000000000000000171171444262304300206500ustar00rootroot00000000000000import copy
import os.path as osp
from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.optim import Adam

import torch_geometric.transforms as T
from torch_geometric.contrib.nn import GRBCDAttack, PRBCDAttack
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GATConv, GCNConv
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.utils import softmax

path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'Planetoid')


class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.norm = gcn_norm
        self.conv1 = GCNConv(in_channels, hidden_channels, normalize=False)
        self.conv2 = GCNConv(hidden_channels, out_channels, normalize=False)

    def reset_parameters(self):
        self.conv1.reset_parameters()
        self.conv2.reset_parameters()

    def forward(self, x, edge_index, edge_weight=None, **kwargs):
        # Normalize edge indices only once:
        if not kwargs.get('skip_norm', False):
            edge_index, edge_weight = self.norm(
                edge_index,
                edge_weight,
                num_nodes=x.size(0),
                add_self_loops=True,
            )

        x = self.conv1(x, edge_index, edge_weight).relu()
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_weight)
        return x


class WeightedGATConv(GATConv):
    """Extended GAT to allow for weighted edges."""
    def edge_update(self, alpha_j: Tensor, alpha_i: Optional[Tensor],
                    edge_attr: Optional[Tensor], index: Tensor,
                    ptr: Optional[Tensor], size_i: Optional[int]) -> Tensor:
        # Given edge-level attention coefficients for source and target nodes,
        # we simply need to sum them up to "emulate" concatenation:
        alpha = alpha_j if alpha_i is None else alpha_j + alpha_i
        alpha = F.leaky_relu(alpha, self.negative_slope)

        if edge_attr is not None:
            assert edge_attr.dim() == 1, 'Only scalar edge weights supported'
            edge_attr = edge_attr.view(-1, 1)
            # `alpha` unchanged if edge_attr == 1 and -Inf if edge_attr == 0;
            # We choose log to counteract underflow in subsequent exp/softmax
            alpha = alpha + torch.log2(edge_attr)

        alpha = softmax(alpha, index, ptr, size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return alpha


class GAT(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        # Initialize edge weights of self-loops with 1:
        self.conv1 = WeightedGATConv(in_channels, hidden_channels,
                                     fill_value=1.)
        self.conv2 = WeightedGATConv(hidden_channels, out_channels,
                                     fill_value=1.)

    def reset_parameters(self):
        self.conv1.reset_parameters()
        self.conv2.reset_parameters()

    def forward(self, x, edge_index, edge_weight=None):
        x = self.conv1(x, edge_index, edge_weight).relu()
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_weight)
        return x


def train(model, data, epochs=200, lr=0.01, weight_decay=5e-4):
    model.train()
    optimizer = Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
    for _ in range(epochs):
        optimizer.zero_grad()
        pred = model(data.x, data.edge_index, data.edge_weight)
        loss = F.cross_entropy(pred[data.train_mask], data.y[data.train_mask])
        loss.backward()
        optimizer.step()


def accuracy(pred, y, mask):
    return (pred.argmax(-1)[mask] == y[mask]).float().mean()


@torch.no_grad()
def test(model, data):
    model.eval()
    pred = model(data.x, data.edge_index, data.edge_weight)
    return float(accuracy(pred, data.y, data.test_mask))


# The metric in PRBCD is assumed to be best if lower (like a loss).
def metric(*args, **kwargs):
    return -accuracy(*args, **kwargs)


if __name__ == '__main__':
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    dataset = Planetoid(path, name='Cora', transform=T.NormalizeFeatures())
    data = dataset[0].to(device)

    gcn = GCN(dataset.num_features, 16, dataset.num_classes).to(device)
    gat = GAT(dataset.num_features, 16, dataset.num_classes).to(device)

    train(gcn, data)
    gcn.eval()

    train(gat, data)
    gat.eval()

    node_idx = 42
    local_budget = 2  # Degree of (training) node 42 is 2.

    # Perturb 5% of edges:
    global_budget = int(0.05 * data.edge_index.size(1) / 2)

    print('------------- GAT: Local Evasion -------------')
    # Note: GRBCD is faster than PRBCD for small budgets but not as consistent

    grbcd = GRBCDAttack(gat, block_size=250_000)
    # The learning rate is one of the most important parameters for PRBCD and a
    # good heuristic is to choose it s.t. the budget is exhausted within a few
    # steps. Moreover, a high learning rate mitigates the impact of the
    # relaxation gap ({0, 1} -> [0, 1]) of the edge weights. See poisoning
    #  example for a debug plot.
    prbcd = PRBCDAttack(gat, block_size=250_000, metric=metric, lr=2_000)

    clean_acc = test(gat, data)
    print(f'Clean accuracy: {clean_acc:.3f}')

    # GRBCD: Attack a single node:
    pert_edge_index, perts = grbcd.attack(
        data.x,
        data.edge_index,
        data.y,
        budget=local_budget,
        idx_attack=[node_idx],
    )

    clean_margin = -PRBCDAttack._probability_margin_loss(
        gat(data.x, data.edge_index), data.y, [node_idx])
    pert_margin = -PRBCDAttack._probability_margin_loss(
        gat(data.x, pert_edge_index), data.y, [node_idx])
    print(f'GRBCD: Confidence margin of target to best non-target dropped '
          f'from {clean_margin:.3f} to {pert_margin:.3f}')
    adv_edges = ', '.join(str((u, v)) for u, v in perts.T.tolist())
    print(f'Adv. edges: {adv_edges}')

    # PRBCD: Attack single node:
    pert_edge_index, perts = prbcd.attack(
        data.x,
        data.edge_index,
        data.y,
        budget=local_budget,
        idx_attack=[node_idx],
    )
    clean_margin = -PRBCDAttack._probability_margin_loss(
        gat(data.x, data.edge_index), data.y, [node_idx])
    pert_margin = -PRBCDAttack._probability_margin_loss(
        gat(data.x, pert_edge_index), data.y, [node_idx])
    print(f'PRBCD: Confidence margin of target to best non-target dropped '
          f'from {clean_margin:.3f} to {pert_margin:.3f}')
    adv_edges = ', '.join(str((u, v)) for u, v in perts.T.tolist())
    print(f'Adv. edges: {adv_edges}\n')

    print('------------- GCN: Global Evasion -------------')

    grbcd = GRBCDAttack(gcn, block_size=250_000)
    prbcd = PRBCDAttack(gcn, block_size=250_000, metric=metric, lr=2_000)

    clean_acc = test(gcn, data)

    # GRBCD: Attack test set:
    pert_edge_index, perts = grbcd.attack(
        data.x,
        data.edge_index,
        data.y,
        budget=global_budget,
        idx_attack=data.test_mask,
    )

    pert_data = copy.copy(data)
    pert_data.edge_index = pert_edge_index
    pert_acc = test(gcn, pert_data)
    print(f'GRBCD: Accuracy dropped from {clean_acc:.3f} to {pert_acc:.3f}')

    # PRBCD: Attack test set:
    pert_edge_index, perts = prbcd.attack(
        data.x,
        data.edge_index,
        data.y,
        budget=global_budget,
        idx_attack=data.test_mask,
    )

    pert_data = copy.copy(data)
    pert_data.edge_index = pert_edge_index
    pert_acc = test(gcn, pert_data)
    print(f'PRBCD: Accuracy dropped from {clean_acc:.3f} to {pert_acc:.3f}')
pytorch/examples/contrib/rbcd_attack_poisoning.py000066400000000000000000000112741444262304300227330ustar00rootroot00000000000000import copy
import os.path as osp
import sys
from typing import Optional, Tuple

import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F
from rbcd_attack import GCN, metric, test, train
from torch import Tensor
from torch.optim import Adam

import torch_geometric.transforms as T
from torch_geometric.contrib.nn import PRBCDAttack
from torch_geometric.datasets import Planetoid

try:
    import higher
except ImportError:
    sys.exit('Install `higher` via `pip install higher` for poisoning example')

path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'Planetoid')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# IMPORTANT: Edge weights are being ignored later and most adjacency matrix
# preprocessing should be part of the model (part of backpropagation):
dataset = Planetoid(path, name='Cora', transform=T.NormalizeFeatures())
data = dataset[0].to(device)

gcn = GCN(dataset.num_features, 16, dataset.num_classes).to(device)
train(gcn, data)

print('------------- GCN: Global Poisoning -------------')

clean_acc = test(gcn, data)
print(f'Clean accuracy: {clean_acc:.3f}')

n_epochs = 50
lr = 0.04
weight_decay = 5e-4


class PoisoningPRBCDAttack(PRBCDAttack):
    def _forward(self, x: Tensor, edge_index: Tensor, edge_weight: Tensor,
                 **kwargs) -> Tensor:
        """Forward model."""
        self.model.reset_parameters()

        with torch.enable_grad():
            ped = copy.copy(data)
            ped.x, ped.edge_index, ped.edge_weight = x, edge_index, edge_weight
            train(self.model, ped, n_epochs, lr, weight_decay)

        self.model.eval()
        return self.model(x, edge_index, edge_weight)

    def _forward_and_gradient(self, x: Tensor, labels: Tensor,
                              idx_attack: Optional[Tensor] = None,
                              **kwargs) -> Tuple[Tensor, Tensor]:
        """Forward and update edge weights."""
        self.block_edge_weight.requires_grad = True

        self.model.reset_parameters()

        self.model.train()
        opt = Adam(self.model.parameters(), lr=lr, weight_decay=weight_decay)

        with higher.innerloop_ctx(self.model, opt) as (fmodel, diffopt):
            edge_index, edge_weight = self._get_modified_adj(
                self.edge_index, self.edge_weight, self.block_edge_index,
                self.block_edge_weight)

            # Normalize only once (only relevant if model normalizes adj)
            if hasattr(fmodel, 'norm'):
                edge_index, edge_weight = fmodel.norm(
                    edge_index,
                    edge_weight,
                    num_nodes=x.size(0),
                    add_self_loops=True,
                )

            for _ in range(n_epochs):
                pred = fmodel.forward(x, edge_index, edge_weight,
                                      skip_norm=True)
                loss = F.cross_entropy(pred[data.train_mask],
                                       data.y[data.train_mask])
                diffopt.step(loss)

            pred = fmodel(x, edge_index, edge_weight)
            loss = self.loss(pred, labels, idx_attack)

            gradient = torch.autograd.grad(loss, self.block_edge_weight)[0]

        # Clip gradient for stability:
        clip_norm = 0.5
        grad_len_sq = gradient.square().sum()
        if grad_len_sq > clip_norm:
            gradient *= clip_norm / grad_len_sq.sqrt()

        self.model.eval()

        return loss, gradient


prbcd = PoisoningPRBCDAttack(gcn, block_size=250_000, metric=metric, lr=100)

# PRBCD: Attack test set:
global_budget = int(0.05 * data.edge_index.size(1) / 2)  # Perturb 5% of edges

pert_edge_index, perts = prbcd.attack(
    data.x,
    data.edge_index,
    data.y,
    budget=global_budget,
    idx_attack=data.test_mask,
)

gcn.reset_parameters()
pert_data = copy.copy(data)
pert_data.edge_index = pert_edge_index
train(gcn, pert_data)
pert_acc = test(gcn, pert_data)
# Note that the values here a bit more noisy than in the evasion case:
print(f'PRBCD: Accuracy dropped from {clean_acc:.3f} to {pert_acc:.3f}')

fig, ax1 = plt.subplots()
plt.title('Global Poisoning GCN')
color = 'tab:red'
ax1.plot(prbcd.attack_statistics['loss'], color=color, label='Loss')
ax1.tick_params(axis='y', labelcolor=color)
ax1.set_ylabel('Loss')
ax1.set_xlabel('Steps')

# It is best practice choosing the learning rate s.t. the budget is exhausted:
ax2 = ax1.twinx()
color = 'tab:blue'
ax2.plot(prbcd.attack_statistics['prob_mass_after_update'], color=color,
         linestyle='--', label='Before projection')
ax2.plot(prbcd.attack_statistics['prob_mass_after_projection'], color=color,
         label='After projection')
ax2.tick_params(axis='y', labelcolor=color)
ax2.set_ylabel('Used budget')
plt.legend()
fig.show()
pytorch/examples/cora.py000066400000000000000000000037001444262304300156640ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import SplineConv
from torch_geometric.typing import WITH_TORCH_SPLINE_CONV

if not WITH_TORCH_SPLINE_CONV:
    quit("This example requires 'torch-spline-conv'")

dataset = 'Cora'
transform = T.Compose([
    T.RandomNodeSplit(num_val=500, num_test=500),
    T.TargetIndegree(),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SplineConv(dataset.num_features, 16, dim=1, kernel_size=2)
        self.conv2 = SplineConv(16, dataset.num_classes, dim=1, kernel_size=2)

    def forward(self):
        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        x = F.dropout(x, training=self.training)
        x = F.elu(self.conv1(x, edge_index, edge_attr))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_attr)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-3)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    log_probs, accs = model(), []
    for _, mask in data('train_mask', 'test_mask'):
        pred = log_probs[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


for epoch in range(1, 201):
    train()
    train_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/correct_and_smooth.py000066400000000000000000000055701444262304300206230ustar00rootroot00000000000000import os.path as osp

import torch
from ogb.nodeproppred import Evaluator, PygNodePropPredDataset

import torch_geometric.transforms as T
from torch_geometric.nn import MLP, CorrectAndSmooth
from torch_geometric.typing import WITH_TORCH_SPARSE

if not WITH_TORCH_SPARSE:
    quit("This example requires 'torch-sparse'")

root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'OGB')
dataset = PygNodePropPredDataset('ogbn-products', root,
                                 transform=T.ToSparseTensor())
evaluator = Evaluator(name='ogbn-products')
split_idx = dataset.get_idx_split()
data = dataset[0]

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = MLP([dataset.num_features, 200, 200, dataset.num_classes], dropout=0.5,
            norm="batch_norm", act_first=True).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

x, y = data.x.to(device), data.y.to(device)
train_idx = split_idx['train'].to(device)
val_idx = split_idx['valid'].to(device)
test_idx = split_idx['test'].to(device)
x_train, y_train = x[train_idx], y[train_idx]


def train():
    model.train()
    optimizer.zero_grad()
    out = model(x_train)
    loss = criterion(out, y_train.view(-1))
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(out=None):
    model.eval()
    out = model(x) if out is None else out
    pred = out.argmax(dim=-1, keepdim=True)
    train_acc = evaluator.eval({
        'y_true': y[train_idx],
        'y_pred': pred[train_idx]
    })['acc']
    val_acc = evaluator.eval({
        'y_true': y[val_idx],
        'y_pred': pred[val_idx]
    })['acc']
    test_acc = evaluator.eval({
        'y_true': y[test_idx],
        'y_pred': pred[test_idx]
    })['acc']
    return train_acc, val_acc, test_acc, out


best_val_acc = 0
for epoch in range(1, 301):
    loss = train()
    train_acc, val_acc, test_acc, out = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        y_soft = out.softmax(dim=-1)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, Test: {test_acc:.4f}')

adj_t = data.adj_t.to(device)
deg = adj_t.sum(dim=1).to(torch.float)
deg_inv_sqrt = deg.pow_(-0.5)
deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0
DAD = deg_inv_sqrt.view(-1, 1) * adj_t * deg_inv_sqrt.view(1, -1)
DA = deg_inv_sqrt.view(-1, 1) * deg_inv_sqrt.view(-1, 1) * adj_t

post = CorrectAndSmooth(num_correction_layers=50, correction_alpha=1.0,
                        num_smoothing_layers=50, smoothing_alpha=0.8,
                        autoscale=False, scale=20.)

print('Correct and smooth...')
y_soft = post.correct(y_soft, y_train, train_idx, DAD)
y_soft = post.smooth(y_soft, y_train, train_idx, DA)
print('Done!')
train_acc, val_acc, test_acc, _ = test(y_soft)
print(f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/cpp/000077500000000000000000000000001444262304300151505ustar00rootroot00000000000000pytorch/examples/cpp/CMakeLists.txt000066400000000000000000000016661444262304300177210ustar00rootroot00000000000000cmake_minimum_required(VERSION 3.10)
project(hello-world)

# The first thing do is to tell cmake to find the TorchScatter
# and TorchSparse libraries. The package pulls in all the necessary
# torch libraries, so there is no need to add `find_package(Torch)`.
find_package(TorchScatter REQUIRED)
find_package(TorchSparse REQUIRED)

find_package(Python3 COMPONENTS Development)

add_executable(hello-world main.cpp)

# We now need to link the TorchScatter and TorchSparse libraries
# to our executable. We can do that by using the
# TorchScatter::TorchScatter and TorchSparse::TorchSparse targets,
# which also adds all the necessary torch dependencies.
target_compile_features(hello-world PUBLIC cxx_range_for)
target_link_libraries(hello-world TorchScatter::TorchScatter)
target_link_libraries(hello-world TorchSparse::TorchSparse)
target_link_libraries(hello-world ${CUDA_cusparse_LIBRARY})
set_property(TARGET hello-world PROPERTY CXX_STANDARD 14)
pytorch/examples/cpp/README.md000066400000000000000000000022751444262304300164350ustar00rootroot00000000000000# PyG in C++

This is a minimal example of getting PyG to work in C++ with CMake.

In order to successfully compile this example, make sure you have both the C++ APIs of [`TorchScatter`](https://github.com/rusty1s/pytorch_scatter#c-api) and [`TorchSparse`](https://github.com/rusty1s/pytorch_sparse/#c-api) installed.

For this, we need to add `TorchLib` to the `-DCMAKE_PREFIX_PATH` (*e.g.*, it may exists in `{CONDA}/lib/python{X.X}/site-packages/torch` if installed via `conda`).
Then, *e.g.*, to install `TorchScatter`, run:

```
git clone https://github.com/rusty1s/pytorch_scatter.git
cd pytorch_scatter
mkdir build && cd build
cmake -DWITH_CUDA=on -DCMAKE_PREFIX_PATH="..." ..
make
(sudo) make install
```

Once both dependencies are sorted, we can start the CMake fun:

1. Run `save_model.py` to create and save a PyG GNN model.
2. Create a `build` directory inside the current one.
3. From within the `build` directory, run the following commands:
   * `cmake -DCMAKE_PREFIX_PATH=";;" ..`
   * `cmake --build .`

That's it!
You should now have a `hello-world` executable in your `build` folder.
Run it via:

```
./hello-world ../model.pt
```
pytorch/examples/cpp/main.cpp000066400000000000000000000014471444262304300166060ustar00rootroot00000000000000#include 
#include 
#include 

#include 

int main(int argc, const char *argv[]) {
  if (argc != 2) {
    std::cerr << "usage: hello-world \n";
    return -1;
  }

  torch::jit::script::Module model;
  try {
    model = torch::jit::load(argv[1]);
  } catch (const c10::Error &e) {
    std::cerr << "error loading the model\n";
    return -1;
  }

  auto x = torch::randn({5, 32});
  auto edge_index = torch::tensor({
      {0, 1, 1, 2, 2, 3, 3, 4},
      {1, 0, 2, 1, 3, 2, 4, 3},
  });

  std::vector inputs;
  inputs.push_back(x);
  inputs.push_back(edge_index);

  auto out = model.forward(inputs).toTensor();
  std::cout << "output tensor shape: " << out.sizes() << std::endl;
}
pytorch/examples/cpp/save_model.py000066400000000000000000000025601444262304300176430ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import BatchNorm1d, Linear, ReLU, Sequential

from torch_geometric.nn import GINConv, global_mean_pool


class GIN(torch.nn.Module):
    def __init__(self, in_channels: int, hidden_channels: int,
                 out_channels: int, num_layers: int):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            mlp = Sequential(
                Linear(in_channels, hidden_channels),
                BatchNorm1d(hidden_channels),
                ReLU(),
                Linear(hidden_channels, hidden_channels),
            )
            self.convs.append(GINConv(mlp).jittable())
            in_channels = hidden_channels

        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        batch: Optional[Tensor] = None,
    ) -> Tensor:

        for conv in self.convs:
            x = conv(x, edge_index).relu()

        x = global_mean_pool(x, batch)

        x = self.lin1(x).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin2(x)

        return x


model = GIN(32, 64, 16, num_layers=3)
model = torch.jit.script(model)
model.save('model.pt')
pytorch/examples/datapipe.py000066400000000000000000000070141444262304300165310ustar00rootroot00000000000000# In this example, you will find data loading implementations using PyTorch
# DataPipes (https://pytorch.org/data/) across various tasks:

# (1) molecular graph data loading pipe
# (2) mesh/point cloud data loading pipe

# In particular, we make use of PyG's built-in DataPipes, e.g., for batching
# multiple PyG data objects together or for converting SMILES strings into
# molecular graph representations. We also showcase how to write your own
# DataPipe (i.e. for loading and parsing mesh data into PyG data objects).

import argparse
import os.path as osp
import time

import torch
from torchdata.datapipes.iter import FileLister, FileOpener, IterDataPipe

from torch_geometric.data import Data, download_url, extract_zip


def molecule_datapipe() -> IterDataPipe:
    # Download HIV dataset from MoleculeNet:
    url = 'https://deepchemdata.s3-us-west-1.amazonaws.com/datasets'
    root_dir = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data')
    path = download_url(f'{url}/HIV.csv', root_dir)

    datapipe = FileOpener([path])
    datapipe = datapipe.parse_csv_as_dict()
    datapipe = datapipe.parse_smiles(target_key='HIV_active')
    datapipe = datapipe.in_memory_cache()  # Cache graph instances in-memory.

    return datapipe


@torch.utils.data.functional_datapipe('read_mesh')
class MeshOpener(IterDataPipe):
    # A custom DataPipe to load and parse mesh data into PyG data objects.
    def __init__(self, dp: IterDataPipe):
        super().__init__()
        self.dp = dp

    def __iter__(self):
        import meshio

        for path in self.dp:
            category = osp.basename(path).split('_')[0]

            mesh = meshio.read(path)
            pos = torch.from_numpy(mesh.points).to(torch.float)
            face = torch.from_numpy(mesh.cells[0].data).t().contiguous()

            yield Data(pos=pos, face=face, category=category)


def mesh_datapipe() -> IterDataPipe:
    # Download ModelNet10 dataset from Princeton:
    url = 'http://vision.princeton.edu/projects/2014/3DShapeNets'
    root_dir = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data')
    path = download_url(f'{url}/ModelNet10.zip', root_dir)
    root_dir = osp.join(root_dir, 'ModelNet10')
    if not osp.exists(root_dir):
        extract_zip(path, root_dir)

    def is_train(path: str) -> bool:
        return 'train' in path

    datapipe = FileLister([root_dir], masks='*.off', recursive=True)
    datapipe = datapipe.filter(is_train)
    datapipe = datapipe.read_mesh()
    datapipe = datapipe.in_memory_cache()  # Cache graph instances in-memory.
    datapipe = datapipe.sample_points(1024)  # Use PyG transforms from here.
    datapipe = datapipe.knn_graph(k=8)

    return datapipe


DATAPIPES = {
    'molecule': molecule_datapipe,
    'mesh': mesh_datapipe,
}

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--task', default='molecule', choices=DATAPIPES.keys())

    args = parser.parse_args()

    datapipe = DATAPIPES[args.task]()

    print('Example output:')
    print(next(iter(datapipe)))

    # Shuffling + Batching support:
    datapipe = datapipe.shuffle()
    datapipe = datapipe.batch_graphs(batch_size=32)

    # The first epoch will take longer than the remaining ones...
    print('Iterating over all data...')
    t = time.perf_counter()
    for batch in datapipe:
        pass
    print(f'Done! [{time.perf_counter() - t:.2f}s]')

    print('Iterating over all data a second time...')
    t = time.perf_counter()
    for batch in datapipe:
        pass
    print(f'Done! [{time.perf_counter() - t:.2f}s]')
pytorch/examples/dgcnn_classification.py000066400000000000000000000047671444262304300211220ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import ModelNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, DynamicEdgeConv, global_max_pool

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data/ModelNet10')
pre_transform, transform = T.NormalizeScale(), T.SamplePoints(1024)
train_dataset = ModelNet(path, '10', True, transform, pre_transform)
test_dataset = ModelNet(path, '10', False, transform, pre_transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True,
                          num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False,
                         num_workers=6)


class Net(torch.nn.Module):
    def __init__(self, out_channels, k=20, aggr='max'):
        super().__init__()

        self.conv1 = DynamicEdgeConv(MLP([2 * 3, 64, 64, 64]), k, aggr)
        self.conv2 = DynamicEdgeConv(MLP([2 * 64, 128]), k, aggr)
        self.lin1 = Linear(128 + 64, 1024)

        self.mlp = MLP([1024, 512, 256, out_channels], dropout=0.5, norm=None)

    def forward(self, data):
        pos, batch = data.pos, data.batch
        x1 = self.conv1(pos, batch)
        x2 = self.conv2(x1, batch)
        out = self.lin1(torch.cat([x1, x2], dim=1))
        out = global_max_pool(out, batch)
        out = self.mlp(out)
        return F.log_softmax(out, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(train_dataset.num_classes, k=20).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()
    return total_loss / len(train_dataset)


def test(loader):
    model.eval()

    correct = 0
    for data in loader:
        data = data.to(device)
        with torch.no_grad():
            pred = model(data).max(dim=1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(loader.dataset)


for epoch in range(1, 201):
    loss = train()
    test_acc = test(test_loader)
    print(f'Epoch {epoch:03d}, Loss: {loss:.4f}, Test: {test_acc:.4f}')
    scheduler.step()
pytorch/examples/dgcnn_segmentation.py000066400000000000000000000077361444262304300206230ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torchmetrics.functional import jaccard_index

import torch_geometric.transforms as T
from torch_geometric.datasets import ShapeNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, DynamicEdgeConv
from torch_geometric.utils import scatter

category = 'Airplane'  # Pass in `None` to train on all categories.
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')
transform = T.Compose([
    T.RandomJitter(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2)
])
pre_transform = T.NormalizeScale()
train_dataset = ShapeNet(path, category, split='trainval', transform=transform,
                         pre_transform=pre_transform)
test_dataset = ShapeNet(path, category, split='test',
                        pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=10, shuffle=True,
                          num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=10, shuffle=False,
                         num_workers=6)


class Net(torch.nn.Module):
    def __init__(self, out_channels, k=30, aggr='max'):
        super().__init__()

        self.conv1 = DynamicEdgeConv(MLP([2 * 6, 64, 64]), k, aggr)
        self.conv2 = DynamicEdgeConv(MLP([2 * 64, 64, 64]), k, aggr)
        self.conv3 = DynamicEdgeConv(MLP([2 * 64, 64, 64]), k, aggr)

        self.mlp = MLP([3 * 64, 1024, 256, 128, out_channels], dropout=0.5,
                       norm=None)

    def forward(self, data):
        x, pos, batch = data.x, data.pos, data.batch
        x0 = torch.cat([x, pos], dim=-1)
        x1 = self.conv1(x0, batch)
        x2 = self.conv2(x1, batch)
        x3 = self.conv3(x2, batch)
        out = self.mlp(torch.cat([x1, x2, x3], dim=1))
        return F.log_softmax(out, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(train_dataset.num_classes, k=30).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.8)


def train():
    model.train()

    total_loss = correct_nodes = total_nodes = 0
    for i, data in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        correct_nodes += out.argmax(dim=1).eq(data.y).sum().item()
        total_nodes += data.num_nodes

        if (i + 1) % 10 == 0:
            print(f'[{i+1}/{len(train_loader)}] Loss: {total_loss / 10:.4f} '
                  f'Train Acc: {correct_nodes / total_nodes:.4f}')
            total_loss = correct_nodes = total_nodes = 0


@torch.no_grad()
def test(loader):
    model.eval()

    ious, categories = [], []
    y_map = torch.empty(loader.dataset.num_classes, device=device).long()
    for data in loader:
        data = data.to(device)
        outs = model(data)

        sizes = (data.ptr[1:] - data.ptr[:-1]).tolist()
        for out, y, category in zip(outs.split(sizes), data.y.split(sizes),
                                    data.category.tolist()):
            category = list(ShapeNet.seg_classes.keys())[category]
            part = ShapeNet.seg_classes[category]
            part = torch.tensor(part, device=device)

            y_map[part] = torch.arange(part.size(0), device=device)

            iou = jaccard_index(out[:, part].argmax(dim=-1), y_map[y],
                                num_classes=part.size(0), absent_score=1.0)
            ious.append(iou)

        categories.append(data.category)

    iou = torch.tensor(ious, device=device)
    category = torch.cat(categories, dim=0)

    mean_iou = scatter(iou, category, reduce='mean')  # Per-category IoU.
    return float(mean_iou.mean())  # Global IoU.


for epoch in range(1, 31):
    train()
    iou = test(test_loader)
    print(f'Epoch: {epoch:02d}, Test IoU: {iou:.4f}')
pytorch/examples/dir_gnn.py000066400000000000000000000053351444262304300163660ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import WikipediaNetwork
from torch_geometric.nn import DirGNNConv, GCNConv, SAGEConv

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='chameleon')
parser.add_argument('--hidden_channels', type=int, default=128)
parser.add_argument('--lr', type=float, default=0.01)
parser.add_argument('--epochs', type=int, default=1000)
parser.add_argument('--alpha', type=float, default=1)
parser.add_argument('--conv', type=str, default='gcn')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Wikipedia')
dataset = WikipediaNetwork(
    root=path,
    name=args.dataset,
    transform=T.NormalizeFeatures(),
)

data = dataset[0].to(device)
data.train_mask = data.train_mask[:, 0]
data.val_mask = data.val_mask[:, 0]
data.test_mask = data.test_mask[:, 0]

if args.conv == 'gcn':
    Conv = GCNConv
elif args.conv == 'sage':
    Conv = SAGEConv
else:
    raise NotImplementedError


class DirGNN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, alpha):
        super().__init__()
        self.conv1 = Conv(in_channels, hidden_channels)
        self.conv1 = DirGNNConv(self.conv1, alpha, root_weight=False)

        self.conv2 = Conv(hidden_channels, out_channels)
        self.conv2 = DirGNNConv(self.conv2, alpha, root_weight=False)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


model = DirGNN(
    dataset.num_features,
    args.hidden_channels,
    dataset.num_classes,
    alpha=args.alpha,
).to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = final_test_acc = 0
for epoch in range(1, args.epochs + 1):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc

    print(f'Epoch: {epoch:04d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/dna.py000066400000000000000000000062211444262304300155030ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.model_selection import StratifiedKFold

from torch_geometric.datasets import Planetoid
from torch_geometric.nn import DNAConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset)
data = dataset[0]
data.train_mask = data.val_mask = data.test_mask = None


def gen_uniform_20_20_60_split(data):
    skf = StratifiedKFold(5, shuffle=True, random_state=55)
    idx = [torch.from_numpy(i) for _, i in skf.split(data.y, data.y)]
    data.train_idx = idx[0].to(torch.long)
    data.val_idx = idx[1].to(torch.long)
    data.test_idx = torch.cat(idx[2:], dim=0).to(torch.long)
    return data


data = gen_uniform_20_20_60_split(data)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers,
                 heads=1, groups=1):
        super().__init__()
        self.hidden_channels = hidden_channels
        self.lin1 = torch.nn.Linear(in_channels, hidden_channels)
        self.convs = torch.nn.ModuleList()
        for i in range(num_layers):
            self.convs.append(
                DNAConv(hidden_channels, heads, groups, dropout=0.8,
                        cached=True))
        self.lin2 = torch.nn.Linear(hidden_channels, out_channels)

    def reset_parameters(self):
        self.lin1.reset_parameters()
        for conv in self.convs:
            conv.reset_parameters()
        self.lin2.reset_parameters()

    def forward(self, x, edge_index):
        x = F.relu(self.lin1(x))
        x = F.dropout(x, p=0.5, training=self.training)
        x_all = x.view(-1, 1, self.hidden_channels)
        for conv in self.convs:
            x = F.relu(conv(x_all, edge_index))
            x = x.view(-1, 1, self.hidden_channels)
            x_all = torch.cat([x_all, x], dim=1)
        x = x_all[:, -1]
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin2(x)
        return torch.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(in_channels=dataset.num_features, hidden_channels=128,
            out_channels=dataset.num_classes, num_layers=5, heads=8, groups=16)
model, data = model.to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=0.0005)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_idx], data.y[data.train_idx])
    loss.backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(data.x, data.edge_index), []
    for _, idx in data('train_idx', 'val_idx', 'test_idx'):
        pred = out[idx].argmax(1)
        acc = pred.eq(data.y[idx]).sum().item() / idx.numel()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 201):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/egc.py000066400000000000000000000103351444262304300155000ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F
from ogb.graphproppred import Evaluator
from ogb.graphproppred import PygGraphPropPredDataset as OGBG
from ogb.graphproppred.mol_encoder import AtomEncoder
from torch.nn import BatchNorm1d, Linear, ReLU, Sequential
from torch.optim.lr_scheduler import ReduceLROnPlateau

import torch_geometric.transforms as T
from torch_geometric.loader import DataLoader
from torch_geometric.nn import EGConv, global_mean_pool
from torch_geometric.typing import WITH_TORCH_SPARSE

if not WITH_TORCH_SPARSE:
    quit("This example requires 'torch-sparse'")

parser = argparse.ArgumentParser()
parser.add_argument('--use_multi_aggregators', action='store_true',
                    help='Switch between EGC-S and EGC-M')
args = parser.parse_args()

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'OGB')
dataset = OGBG('ogbg-molhiv', path, pre_transform=T.ToSparseTensor())
evaluator = Evaluator('ogbg-molhiv')

split_idx = dataset.get_idx_split()
train_dataset = dataset[split_idx['train']]
val_dataset = dataset[split_idx['valid']]
test_dataset = dataset[split_idx['test']]

train_loader = DataLoader(train_dataset, batch_size=32, num_workers=4,
                          shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=256)
test_loader = DataLoader(test_dataset, batch_size=256)


class Net(torch.nn.Module):
    def __init__(self, hidden_channels, num_layers, num_heads, num_bases):
        super().__init__()
        if args.use_multi_aggregators:
            aggregators = ['sum', 'mean', 'max']
        else:
            aggregators = ['symnorm']

        self.encoder = AtomEncoder(hidden_channels)

        self.convs = torch.nn.ModuleList()
        self.norms = torch.nn.ModuleList()
        for _ in range(num_layers):
            self.convs.append(
                EGConv(hidden_channels, hidden_channels, aggregators,
                       num_heads, num_bases))
            self.norms.append(BatchNorm1d(hidden_channels))

        self.mlp = Sequential(
            Linear(hidden_channels, hidden_channels // 2, bias=False),
            BatchNorm1d(hidden_channels // 2),
            ReLU(inplace=True),
            Linear(hidden_channels // 2, hidden_channels // 4, bias=False),
            BatchNorm1d(hidden_channels // 4),
            ReLU(inplace=True),
            Linear(hidden_channels // 4, 1),
        )

    def forward(self, x, adj_t, batch):
        adj_t = adj_t.set_value(None)  # EGConv works without any edge features

        x = self.encoder(x)

        for conv, norm in zip(self.convs, self.norms):
            h = conv(x, adj_t)
            h = norm(h)
            h = h.relu_()
            x = x + h

        x = global_mean_pool(x, batch)

        return self.mlp(x)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(hidden_channels=236, num_layers=4, num_heads=4,
            num_bases=4).to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=20,
                              min_lr=1e-5)


def train():
    model.train()

    total_loss = total_examples = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()

        out = model(data.x, data.adj_t, data.batch)
        loss = F.binary_cross_entropy_with_logits(out, data.y.to(torch.float))
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * data.num_graphs
        total_examples += data.num_graphs

    return total_loss / total_examples


@torch.no_grad()
def evaluate(loader):
    model.eval()

    y_pred, y_true = [], []
    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.adj_t, data.batch)
        y_pred.append(pred.cpu())
        y_true.append(data.y.cpu())

    y_true = torch.cat(y_true, dim=0)
    y_pred = torch.cat(y_pred, dim=0)
    return evaluator.eval({'y_true': y_true, 'y_pred': y_pred})['rocauc']


for epoch in range(1, 31):
    loss = train()
    val_rocauc = evaluate(val_loader)
    test_rocauc = evaluate(test_loader)
    scheduler.step(val_rocauc)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_rocauc:.4f}, '
          f'Test: {test_rocauc:.4f}')
pytorch/examples/equilibrium_median.py000066400000000000000000000021531444262304300206050ustar00rootroot00000000000000r"""
Replicates the experiment from `"Deep Graph Infomax"
`_ to try and teach
`EquilibriumAggregation` to learn to take the median of
a set of numbers

This example converges slowly to being able to predict the
median similar to what is observed in the paper.
"""

import numpy as np
import torch

from torch_geometric.nn import EquilibriumAggregation

input_size = 100
steps = 10000000
embedding_size = 10
eval_each = 1000

model = EquilibriumAggregation(1, 10, [256, 256], 1)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

norm = torch.distributions.normal.Normal(0.5, 0.4)
gamma = torch.distributions.gamma.Gamma(0.2, 0.5)
uniform = torch.distributions.uniform.Uniform(0, 1)
total_loss = 0
n_loss = 0

for i in range(1, steps + 1):
    optimizer.zero_grad()
    dist = np.random.choice([norm, gamma, uniform])
    x = dist.sample((input_size, 1))
    y = model(x)
    loss = (y - x.median()).norm(2) / input_size
    loss.backward()
    optimizer.step()
    total_loss += loss
    n_loss += 1
    if i % eval_each == 0:
        print(f"Epoch: {i}, Loss {total_loss / n_loss:.6f}")
pytorch/examples/explain/000077500000000000000000000000001444262304300160265ustar00rootroot00000000000000pytorch/examples/explain/captum_explainer.py000066400000000000000000000037301444262304300217430ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Planetoid
from torch_geometric.explain import CaptumExplainer, Explainer
from torch_geometric.nn import GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, dataset)
data = dataset[0]


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, x, edge_index):
        x = F.relu(self.conv1(x, edge_index))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = GCN().to(device)
data = data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

for _ in range(1, 201):
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()

explainer = Explainer(
    model=model,
    algorithm=CaptumExplainer('IntegratedGradients'),
    explanation_type='model',
    model_config=dict(
        mode='multiclass_classification',
        task_level='node',
        return_type='log_probs',
    ),
    node_mask_type='attributes',
    edge_mask_type='object',
    threshold_config=dict(
        threshold_type='topk',
        value=200,
    ),
)

node_index = 10
explanation = explainer(data.x, data.edge_index, index=node_index)
print(f'Generated explanations in {explanation.available_explanations}')

path = 'feature_importance.png'
explanation.visualize_feature_importance(path, top_k=10)
print(f"Feature importance plot has been saved to '{path}'")

path = 'subgraph.pdf'
explanation.visualize_graph(path)
print(f"Subgraph plot has been saved to '{path}'")
pytorch/examples/explain/captum_explainer_hetero_link.py000066400000000000000000000073211444262304300243260ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import MovieLens
from torch_geometric.explain import CaptumExplainer, Explainer
from torch_geometric.nn import SAGEConv, to_hetero

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/MovieLens')
dataset = MovieLens(path, model_name='all-MiniLM-L6-v2')
data = dataset[0].to(device)

# Add user node features for message passing:
data['user'].x = torch.eye(data['user'].num_nodes, device=device)
del data['user'].num_nodes

# Add a reverse ('movie', 'rev_rates', 'user') relation for message passing:
data = T.ToUndirected()(data)
data['user', 'movie'].edge_label = data['user',
                                        'movie'].edge_label.to(torch.float)
del data['movie', 'rev_rates', 'user'].edge_label  # Remove "reverse" label.

# Perform a link-level split into training, validation, and test edges:
data, _, _ = T.RandomLinkSplit(
    num_val=0.1,
    num_test=0.1,
    neg_sampling_ratio=0.0,
    edge_types=[('user', 'rates', 'movie')],
    rev_edge_types=[('movie', 'rev_rates', 'user')],
)(data)


class GNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_dict, edge_label_index):
        row, col = edge_label_index
        z = torch.cat([z_dict['user'][row], z_dict['movie'][col]], dim=-1)

        z = self.lin1(z).relu()
        z = self.lin2(z)
        return z.view(-1)


class Model(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.encoder = GNNEncoder(hidden_channels, hidden_channels)
        self.encoder = to_hetero(self.encoder, data.metadata(), aggr='sum')
        self.decoder = EdgeDecoder(hidden_channels)

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = self.encoder(x_dict, edge_index_dict)
        return self.decoder(z_dict, edge_label_index)


model = Model(hidden_channels=32).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(1, 10):
    model.train()
    optimizer.zero_grad()
    pred = model(
        data.x_dict,
        data.edge_index_dict,
        data['user', 'movie'].edge_label_index,
    )
    loss = F.mse_loss(pred, data['user', 'movie'].edge_label)
    loss.backward()
    optimizer.step()

explainer = Explainer(
    model=model,
    algorithm=CaptumExplainer('IntegratedGradients'),
    explanation_type='model',
    model_config=dict(
        mode='regression',
        task_level='edge',
        return_type='raw',
    ),
    node_mask_type='attributes',
    edge_mask_type='object',
    threshold_config=dict(
        threshold_type='topk',
        value=200,
    ),
)

index = torch.tensor([2, 10])  # Explain edge labels with index 2 and 10.
explanation = explainer(
    data.x_dict,
    data.edge_index_dict,
    index=index,
    edge_label_index=data['user', 'movie'].edge_label_index,
)
print(f'Generated explanations in {explanation.available_explanations}')

path = 'feature_importance.png'
explanation.visualize_feature_importance(path, top_k=10)
print(f"Feature importance plot has been saved to '{path}'")
pytorch/examples/explain/gnn_explainer.py000066400000000000000000000036041444262304300212340ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Planetoid
from torch_geometric.explain import Explainer, GNNExplainer
from torch_geometric.nn import GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, dataset)
data = dataset[0]


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, x, edge_index):
        x = F.relu(self.conv1(x, edge_index))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = GCN().to(device)
data = data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

for epoch in range(1, 201):
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()

explainer = Explainer(
    model=model,
    algorithm=GNNExplainer(epochs=200),
    explanation_type='model',
    node_mask_type='attributes',
    edge_mask_type='object',
    model_config=dict(
        mode='multiclass_classification',
        task_level='node',
        return_type='log_probs',
    ),
)
node_index = 10
explanation = explainer(data.x, data.edge_index, index=node_index)
print(f'Generated explanations in {explanation.available_explanations}')

path = 'feature_importance.png'
explanation.visualize_feature_importance(path, top_k=10)
print(f"Feature importance plot has been saved to '{path}'")

path = 'subgraph.pdf'
explanation.visualize_graph(path)
print(f"Subgraph visualization plot has been saved to '{path}'")
pytorch/examples/explain/gnn_explainer_ba_shapes.py000066400000000000000000000062461444262304300232460ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.datasets import ExplainerDataset
from torch_geometric.datasets.graph_generator import BAGraph
from torch_geometric.explain import Explainer, GNNExplainer
from torch_geometric.nn import GCN
from torch_geometric.utils import k_hop_subgraph

dataset = ExplainerDataset(
    graph_generator=BAGraph(num_nodes=300, num_edges=5),
    motif_generator='house',
    num_motifs=80,
    transform=T.Constant(),
)
data = dataset[0]

idx = torch.arange(data.num_nodes)
train_idx, test_idx = train_test_split(idx, train_size=0.8, stratify=data.y)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data = data.to(device)
model = GCN(data.num_node_features, hidden_channels=20, num_layers=3,
            out_channels=dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.005)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.cross_entropy(out[train_idx], data.y[train_idx])
    torch.nn.utils.clip_grad_norm_(model.parameters(), 2.0)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)

    train_correct = int((pred[train_idx] == data.y[train_idx]).sum())
    train_acc = train_correct / train_idx.size(0)

    test_correct = int((pred[test_idx] == data.y[test_idx]).sum())
    test_acc = test_correct / test_idx.size(0)

    return train_acc, test_acc


pbar = tqdm(range(1, 2001))
for epoch in pbar:
    loss = train()
    if epoch == 1 or epoch % 200 == 0:
        train_acc, test_acc = test()
        pbar.set_description(f'Loss: {loss:.4f}, Train: {train_acc:.4f}, '
                             f'Test: {test_acc:.4f}')
pbar.close()
model.eval()

for explanation_type in ['phenomenon', 'model']:
    explainer = Explainer(
        model=model,
        algorithm=GNNExplainer(epochs=300),
        explanation_type=explanation_type,
        node_mask_type='attributes',
        edge_mask_type='object',
        model_config=dict(
            mode='multiclass_classification',
            task_level='node',
            return_type='raw',
        ),
    )

    # Explanation ROC AUC over all test nodes:
    targets, preds = [], []
    node_indices = range(400, data.num_nodes, 5)
    for node_index in tqdm(node_indices, leave=False, desc='Train Explainer'):
        target = data.y if explanation_type == 'phenomenon' else None
        explanation = explainer(data.x, data.edge_index, index=node_index,
                                target=target)

        _, _, _, hard_edge_mask = k_hop_subgraph(node_index, num_hops=3,
                                                 edge_index=data.edge_index)

        targets.append(data.edge_mask[hard_edge_mask].cpu())
        preds.append(explanation.edge_mask[hard_edge_mask].cpu())

    auc = roc_auc_score(torch.cat(targets), torch.cat(preds))
    print(f'Mean ROC AUC (explanation type {explanation_type:10}): {auc:.4f}')
pytorch/examples/explain/gnn_explainer_link_pred.py000066400000000000000000000067131444262304300232670ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import roc_auc_score

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.explain import Explainer, GNNExplainer, ModelConfig
from torch_geometric.nn import GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.NormalizeFeatures(),
    T.ToDevice(device),
    T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True),
])
dataset = Planetoid(path, dataset, transform=transform)
train_data, val_data, test_data = dataset[0]


class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)

    def encode(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x

    def decode(self, z, edge_label_index):
        src, dst = edge_label_index
        return (z[src] * z[dst]).sum(dim=-1)

    def forward(self, x, edge_index, edge_label_index):
        z = model.encode(x, edge_index)
        return model.decode(z, edge_label_index).view(-1)


model = GCN(dataset.num_features, 128, 64).to(device)
optimizer = torch.optim.Adam(params=model.parameters(), lr=0.01)


def train():
    model.train()
    optimizer.zero_grad()

    out = model(train_data.x, train_data.edge_index,
                train_data.edge_label_index)
    loss = F.binary_cross_entropy_with_logits(out, train_data.edge_label)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    out = model(data.x, data.edge_index, data.edge_label_index).sigmoid()
    return roc_auc_score(data.edge_label.cpu().numpy(), out.cpu().numpy())


for epoch in range(1, 201):
    loss = train()
    if epoch % 20 == 0:
        val_auc = test(val_data)
        test_auc = test(test_data)
        print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_auc:.4f}, '
              f'Test: {test_auc:.4f}')

model_config = ModelConfig(
    mode='binary_classification',
    task_level='edge',
    return_type='raw',
)

# Explain model output for a single edge:
edge_label_index = val_data.edge_label_index[:, 0]

explainer = Explainer(
    model=model,
    explanation_type='model',
    algorithm=GNNExplainer(epochs=200),
    node_mask_type='attributes',
    edge_mask_type='object',
    model_config=model_config,
)
explanation = explainer(
    x=train_data.x,
    edge_index=train_data.edge_index,
    edge_label_index=edge_label_index,
)
print(f'Generated model explanations in {explanation.available_explanations}')

# Explain a selected target (phenomenon) for a single edge:
edge_label_index = val_data.edge_label_index[:, 0]
target = val_data.edge_label[0].unsqueeze(dim=0).long()

explainer = Explainer(
    model=model,
    explanation_type='phenomenon',
    algorithm=GNNExplainer(epochs=200),
    node_mask_type='attributes',
    edge_mask_type='object',
    model_config=model_config,
)
explanation = explainer(
    x=train_data.x,
    edge_index=train_data.edge_index,
    target=target,
    edge_label_index=edge_label_index,
)
available_explanations = explanation.available_explanations
print(f'Generated phenomenon explanations in {available_explanations}')
pytorch/examples/faust.py000066400000000000000000000054471444262304300160740ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import FAUST
from torch_geometric.loader import DataLoader
from torch_geometric.nn import SplineConv
from torch_geometric.typing import WITH_TORCH_SPLINE_CONV

if not WITH_TORCH_SPLINE_CONV:
    quit("This example requires 'torch-spline-conv'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'FAUST')
pre_transform = T.Compose([T.FaceToEdge(), T.Constant(value=1)])
train_dataset = FAUST(path, True, T.Cartesian(), pre_transform)
test_dataset = FAUST(path, False, T.Cartesian(), pre_transform)
train_loader = DataLoader(train_dataset, batch_size=1, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=1)
d = train_dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SplineConv(1, 32, dim=3, kernel_size=5, aggr='add')
        self.conv2 = SplineConv(32, 64, dim=3, kernel_size=5, aggr='add')
        self.conv3 = SplineConv(64, 64, dim=3, kernel_size=5, aggr='add')
        self.conv4 = SplineConv(64, 64, dim=3, kernel_size=5, aggr='add')
        self.conv5 = SplineConv(64, 64, dim=3, kernel_size=5, aggr='add')
        self.conv6 = SplineConv(64, 64, dim=3, kernel_size=5, aggr='add')
        self.lin1 = torch.nn.Linear(64, 256)
        self.lin2 = torch.nn.Linear(256, d.num_nodes)

    def forward(self, data):
        x, edge_index, pseudo = data.x, data.edge_index, data.edge_attr
        x = F.elu(self.conv1(x, edge_index, pseudo))
        x = F.elu(self.conv2(x, edge_index, pseudo))
        x = F.elu(self.conv3(x, edge_index, pseudo))
        x = F.elu(self.conv4(x, edge_index, pseudo))
        x = F.elu(self.conv5(x, edge_index, pseudo))
        x = F.elu(self.conv6(x, edge_index, pseudo))
        x = F.elu(self.lin1(x))
        x = F.dropout(x, training=self.training)
        x = self.lin2(x)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
target = torch.arange(d.num_nodes, dtype=torch.long, device=device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(epoch):
    model.train()

    if epoch == 61:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.001

    for data in train_loader:
        optimizer.zero_grad()
        F.nll_loss(model(data.to(device)), target).backward()
        optimizer.step()


def test():
    model.eval()
    correct = 0

    for data in test_loader:
        pred = model(data.to(device)).max(1)[1]
        correct += pred.eq(target).sum().item()
    return correct / (len(test_dataset) * d.num_nodes)


for epoch in range(1, 101):
    train(epoch)
    test_acc = test()
    print(f'Epoch: {epoch:03d}, Test: {test_acc:.4f}')
pytorch/examples/film.py000066400000000000000000000055041444262304300156730ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import f1_score
from torch.nn import BatchNorm1d

from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader
from torch_geometric.nn import FiLMConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers,
                 dropout=0.0):
        super().__init__()
        self.dropout = dropout

        self.convs = torch.nn.ModuleList()
        self.convs.append(FiLMConv(in_channels, hidden_channels))
        for _ in range(num_layers - 2):
            self.convs.append(FiLMConv(hidden_channels, hidden_channels))
        self.convs.append(FiLMConv(hidden_channels, out_channels, act=None))

        self.norms = torch.nn.ModuleList()
        for _ in range(num_layers - 1):
            self.norms.append(BatchNorm1d(hidden_channels))

    def forward(self, x, edge_index):
        for conv, norm in zip(self.convs[:-1], self.norms):
            x = norm(conv(x, edge_index))
            x = F.dropout(x, p=self.dropout, training=self.training)
        x = self.convs[-1](x, edge_index)
        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(in_channels=train_dataset.num_features, hidden_channels=320,
            out_channels=train_dataset.num_classes, num_layers=4,
            dropout=0.1).to(device)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = criterion(model(data.x, data.edge_index), data.y)
        total_loss += loss.item() * data.num_graphs
        loss.backward()
        optimizer.step()
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        out = model(data.x.to(device), data.edge_index.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 501):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/gat.py000066400000000000000000000053341444262304300155200ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.logging import init_wandb, log
from torch_geometric.nn import GATConv

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='Cora')
parser.add_argument('--hidden_channels', type=int, default=8)
parser.add_argument('--heads', type=int, default=8)
parser.add_argument('--lr', type=float, default=0.005)
parser.add_argument('--epochs', type=int, default=200)
parser.add_argument('--wandb', action='store_true', help='Track experiment')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
init_wandb(name=f'GAT-{args.dataset}', heads=args.heads, epochs=args.epochs,
           hidden_channels=args.hidden_channels, lr=args.lr, device=device)

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, args.dataset, transform=T.NormalizeFeatures())
data = dataset[0].to(device)


class GAT(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, heads):
        super().__init__()
        self.conv1 = GATConv(in_channels, hidden_channels, heads, dropout=0.6)
        # On the Pubmed dataset, use `heads` output heads in `conv2`.
        self.conv2 = GATConv(hidden_channels * heads, out_channels, heads=1,
                             concat=False, dropout=0.6)

    def forward(self, x, edge_index):
        x = F.dropout(x, p=0.6, training=self.training)
        x = F.elu(self.conv1(x, edge_index))
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, edge_index)
        return x


model = GAT(dataset.num_features, args.hidden_channels, dataset.num_classes,
            args.heads).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = final_test_acc = 0
for epoch in range(1, args.epochs + 1):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    log(Epoch=epoch, Loss=loss, Train=train_acc, Val=val_acc, Test=test_acc)
pytorch/examples/gcn.py000066400000000000000000000063141444262304300155130ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.logging import init_wandb, log
from torch_geometric.nn import GCNConv

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='Cora')
parser.add_argument('--hidden_channels', type=int, default=16)
parser.add_argument('--lr', type=float, default=0.01)
parser.add_argument('--epochs', type=int, default=200)
parser.add_argument('--use_gdc', action='store_true', help='Use GDC')
parser.add_argument('--wandb', action='store_true', help='Track experiment')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
init_wandb(name=f'GCN-{args.dataset}', lr=args.lr, epochs=args.epochs,
           hidden_channels=args.hidden_channels, device=device)

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, args.dataset, transform=T.NormalizeFeatures())
data = dataset[0]

if args.use_gdc:
    transform = T.GDC(
        self_loop_weight=1,
        normalization_in='sym',
        normalization_out='col',
        diffusion_kwargs=dict(method='ppr', alpha=0.05),
        sparsification_kwargs=dict(method='topk', k=128, dim=0),
        exact=True,
    )
    data = transform(data)


class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels, cached=True,
                             normalize=not args.use_gdc)
        self.conv2 = GCNConv(hidden_channels, out_channels, cached=True,
                             normalize=not args.use_gdc)

    def forward(self, x, edge_index, edge_weight=None):
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv1(x, edge_index, edge_weight).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index, edge_weight)
        return x


model = GCN(dataset.num_features, args.hidden_channels, dataset.num_classes)
model, data = model.to(device), data.to(device)
optimizer = torch.optim.Adam([
    dict(params=model.conv1.parameters(), weight_decay=5e-4),
    dict(params=model.conv2.parameters(), weight_decay=0)
], lr=args.lr)  # Only perform weight-decay on first convolution.


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index, data.edge_attr)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index, data.edge_attr).argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = final_test_acc = 0
for epoch in range(1, args.epochs + 1):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    log(Epoch=epoch, Loss=loss, Train=train_acc, Val=val_acc, Test=test_acc)
pytorch/examples/gcn2_cora.py000066400000000000000000000053731444262304300166050ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCN2Conv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
transform = T.Compose([T.NormalizeFeatures(), T.GCNNorm(), T.ToSparseTensor()])
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self, hidden_channels, num_layers, alpha, theta,
                 shared_weights=True, dropout=0.0):
        super().__init__()

        self.lins = torch.nn.ModuleList()
        self.lins.append(Linear(dataset.num_features, hidden_channels))
        self.lins.append(Linear(hidden_channels, dataset.num_classes))

        self.convs = torch.nn.ModuleList()
        for layer in range(num_layers):
            self.convs.append(
                GCN2Conv(hidden_channels, alpha, theta, layer + 1,
                         shared_weights, normalize=False))

        self.dropout = dropout

    def forward(self, x, adj_t):
        x = F.dropout(x, self.dropout, training=self.training)
        x = x_0 = self.lins[0](x).relu()

        for conv in self.convs:
            x = F.dropout(x, self.dropout, training=self.training)
            x = conv(x, x_0, adj_t)
            x = x.relu()

        x = F.dropout(x, self.dropout, training=self.training)
        x = self.lins[1](x)

        return x.log_softmax(dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(hidden_channels=64, num_layers=64, alpha=0.1, theta=0.5,
            shared_weights=True, dropout=0.6).to(device)
data = data.to(device)
optimizer = torch.optim.Adam([
    dict(params=model.convs.parameters(), weight_decay=0.01),
    dict(params=model.lins.parameters(), weight_decay=5e-4)
], lr=0.01)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.adj_t)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred, accs = model(data.x, data.adj_t).argmax(dim=-1), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 1001):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:04d}, Loss: {loss:.4f} Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {tmp_test_acc:.4f}, '
          f'Final Test: {test_acc:.4f}')
pytorch/examples/gcn2_ppi.py000066400000000000000000000062331444262304300164450ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import f1_score
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GCN2Conv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'GCN2_PPI')
pre_transform = T.Compose([T.GCNNorm(), T.ToSparseTensor()])
train_dataset = PPI(path, split='train', pre_transform=pre_transform)
val_dataset = PPI(path, split='val', pre_transform=pre_transform)
test_dataset = PPI(path, split='test', pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)


class Net(torch.nn.Module):
    def __init__(self, hidden_channels, num_layers, alpha, theta,
                 shared_weights=True, dropout=0.0):
        super().__init__()

        self.lins = torch.nn.ModuleList()
        self.lins.append(Linear(train_dataset.num_features, hidden_channels))
        self.lins.append(Linear(hidden_channels, train_dataset.num_classes))

        self.convs = torch.nn.ModuleList()
        for layer in range(num_layers):
            self.convs.append(
                GCN2Conv(hidden_channels, alpha, theta, layer + 1,
                         shared_weights, normalize=False))

        self.dropout = dropout

    def forward(self, x, adj_t):
        x = F.dropout(x, self.dropout, training=self.training)
        x = x_0 = self.lins[0](x).relu()

        for conv in self.convs:
            h = F.dropout(x, self.dropout, training=self.training)
            h = conv(h, x_0, adj_t)
            x = h + x
            x = x.relu()

        x = F.dropout(x, self.dropout, training=self.training)
        x = self.lins[1](x)

        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(hidden_channels=2048, num_layers=9, alpha=0.5, theta=1.0,
            shared_weights=False, dropout=0.2).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = torch.nn.BCEWithLogitsLoss()


def train():
    model.train()

    total_loss = total_examples = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = criterion(model(data.x, data.adj_t), data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * data.num_nodes
        total_examples += data.num_nodes
    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        out = model(data.x.to(device), data.adj_t.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 2001):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:04d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/geniepath.py000066400000000000000000000112641444262304300167100ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
from sklearn.metrics import f1_score

from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GATConv

parser = argparse.ArgumentParser()
parser.add_argument('--model', type=str, default='GeniePathLazy')
args = parser.parse_args()
assert args.model in ['GeniePath', 'GeniePathLazy']

path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')
train_loader = DataLoader(train_dataset, batch_size=1, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)

dim = 256
lstm_hidden = 256
layer_num = 4


class Breadth(torch.nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.gatconv = GATConv(in_dim, out_dim, heads=1)

    def forward(self, x, edge_index):
        x = torch.tanh(self.gatconv(x, edge_index))
        return x


class Depth(torch.nn.Module):
    def __init__(self, in_dim, hidden):
        super().__init__()
        self.lstm = torch.nn.LSTM(in_dim, hidden, 1, bias=False)

    def forward(self, x, h, c):
        x, (h, c) = self.lstm(x, (h, c))
        return x, (h, c)


class GeniePathLayer(torch.nn.Module):
    def __init__(self, in_dim):
        super().__init__()
        self.breadth_func = Breadth(in_dim, dim)
        self.depth_func = Depth(dim, lstm_hidden)

    def forward(self, x, edge_index, h, c):
        x = self.breadth_func(x, edge_index)
        x = x[None, :]
        x, (h, c) = self.depth_func(x, h, c)
        x = x[0]
        return x, (h, c)


class GeniePath(torch.nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.lin1 = torch.nn.Linear(in_dim, dim)
        self.gplayers = torch.nn.ModuleList(
            [GeniePathLayer(dim) for i in range(layer_num)])
        self.lin2 = torch.nn.Linear(dim, out_dim)

    def forward(self, x, edge_index):
        x = self.lin1(x)
        h = torch.zeros(1, x.shape[0], lstm_hidden, device=x.device)
        c = torch.zeros(1, x.shape[0], lstm_hidden, device=x.device)
        for i, l in enumerate(self.gplayers):
            x, (h, c) = self.gplayers[i](x, edge_index, h, c)
        x = self.lin2(x)
        return x


class GeniePathLazy(torch.nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.lin1 = torch.nn.Linear(in_dim, dim)
        self.breadths = torch.nn.ModuleList(
            [Breadth(dim, dim) for i in range(layer_num)])
        self.depths = torch.nn.ModuleList(
            [Depth(dim * 2, lstm_hidden) for i in range(layer_num)])
        self.lin2 = torch.nn.Linear(dim, out_dim)

    def forward(self, x, edge_index):
        x = self.lin1(x)
        h = torch.zeros(1, x.shape[0], lstm_hidden, device=x.device)
        c = torch.zeros(1, x.shape[0], lstm_hidden, device=x.device)
        h_tmps = []
        for i, l in enumerate(self.breadths):
            h_tmps.append(self.breadths[i](x, edge_index))
        x = x[None, :]
        for i, l in enumerate(self.depths):
            in_cat = torch.cat((h_tmps[i][None, :], x), -1)
            x, (h, c) = self.depths[i](in_cat, h, c)
        x = self.lin2(x[0])
        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
kwargs = {'GeniePath': GeniePath, 'GeniePathLazy': GeniePathLazy}
model = kwargs[args.model](train_dataset.num_features,
                           train_dataset.num_classes).to(device)
loss_op = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        num_graphs = data.num_graphs
        data.batch = None
        data = data.to(device)
        optimizer.zero_grad()
        loss = loss_op(model(data.x, data.edge_index), data.y)
        total_loss += loss.item() * num_graphs
        loss.backward()
        optimizer.step()
    return total_loss / len(train_loader.dataset)


def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        with torch.no_grad():
            out = model(data.x.to(device), data.edge_index.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 101):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/glnn.py000066400000000000000000000060161444262304300157010ustar00rootroot00000000000000# Implementation of:
# Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

import argparse
import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCN, MLP

parser = argparse.ArgumentParser()
parser.add_argument('--lamb', type=float, default=0.0,
                    help='Balances loss from hard labels and teacher outputs')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=T.NormalizeFeatures())
data = dataset[0].to(device)

gnn = GCN(dataset.num_node_features, hidden_channels=16,
          out_channels=dataset.num_classes, num_layers=2).to(device)
mlp = MLP([dataset.num_node_features, 64, dataset.num_classes], dropout=0.5,
          norm=None).to(device)

gnn_optimizer = torch.optim.Adam(gnn.parameters(), lr=0.01, weight_decay=5e-4)
mlp_optimizer = torch.optim.Adam(mlp.parameters(), lr=0.01, weight_decay=5e-4)


def train_teacher():
    gnn.train()
    gnn_optimizer.zero_grad()
    out = gnn(data.x, data.edge_index)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    gnn_optimizer.step()
    return float(loss)


@torch.no_grad()
def test_teacher():
    gnn.eval()
    pred = gnn(data.x, data.edge_index).argmax(dim=-1)
    accs = []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


print('Training Teacher GNN:')
for epoch in range(1, 201):
    loss = train_teacher()
    if epoch % 20 == 0:
        train_acc, val_acc, test_acc = test_teacher()
        print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
              f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')

with torch.no_grad():  # Obtain soft labels from the GNN:
    y_soft = gnn(data.x, data.edge_index).log_softmax(dim=-1)


def train_student():
    mlp.train()
    mlp_optimizer.zero_grad()
    out = mlp(data.x)
    loss1 = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss2 = F.kl_div(out.log_softmax(dim=-1), y_soft, reduction='batchmean',
                     log_target=True)
    loss = args.lamb * loss1 + (1 - args.lamb) * loss2
    loss.backward()
    mlp_optimizer.step()
    return float(loss)


@torch.no_grad()
def test_student():
    mlp.eval()
    pred = mlp(data.x).argmax(dim=-1)
    accs = []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


print('Training Student MLP:')
for epoch in range(1, 501):
    loss = train_student()
    if epoch % 20 == 0:
        train_acc, val_acc, test_acc = test_student()
        print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
              f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/graph_gps.py000066400000000000000000000117101444262304300167120ustar00rootroot00000000000000import argparse
import os.path as osp
from typing import Any, Dict, Optional

import torch
from torch.nn import (
    BatchNorm1d,
    Embedding,
    Linear,
    ModuleList,
    ReLU,
    Sequential,
)
from torch.optim.lr_scheduler import ReduceLROnPlateau

import torch_geometric.transforms as T
from torch_geometric.datasets import ZINC
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GINEConv, GPSConv, global_add_pool
from torch_geometric.nn.attention import PerformerAttention

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ZINC-PE')
transform = T.AddRandomWalkPE(walk_length=20, attr_name='pe')
train_dataset = ZINC(path, subset=True, split='train', pre_transform=transform)
val_dataset = ZINC(path, subset=True, split='val', pre_transform=transform)
test_dataset = ZINC(path, subset=True, split='test', pre_transform=transform)

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=64)
test_loader = DataLoader(test_dataset, batch_size=64)

parser = argparse.ArgumentParser()
parser.add_argument(
    '--attn_type', default='multihead',
    help="Global attention type such as 'multihead' or 'performer'.")
args = parser.parse_args()


class GPS(torch.nn.Module):
    def __init__(self, channels: int, pe_dim: int, num_layers: int,
                 attn_type: str, attn_kwargs: Dict[str, Any]):
        super().__init__()

        self.node_emb = Embedding(28, channels - pe_dim)
        self.pe_lin = Linear(20, pe_dim)
        self.pe_norm = BatchNorm1d(20)
        self.edge_emb = Embedding(4, channels)

        self.convs = ModuleList()
        for _ in range(num_layers):
            nn = Sequential(
                Linear(channels, channels),
                ReLU(),
                Linear(channels, channels),
            )
            conv = GPSConv(channels, GINEConv(nn), heads=4,
                           attn_type=attn_type, attn_kwargs=attn_kwargs)
            self.convs.append(conv)

        self.mlp = Sequential(
            Linear(channels, channels // 2),
            ReLU(),
            Linear(channels // 2, channels // 4),
            ReLU(),
            Linear(channels // 4, 1),
        )
        self.redraw_projection = RedrawProjection(
            self.convs,
            redraw_interval=1000 if attn_type == 'performer' else None)

    def forward(self, x, pe, edge_index, edge_attr, batch):
        x_pe = self.pe_norm(pe)
        x = torch.cat((self.node_emb(x.squeeze(-1)), self.pe_lin(x_pe)), 1)
        edge_attr = self.edge_emb(edge_attr)

        for conv in self.convs:
            x = conv(x, edge_index, batch, edge_attr=edge_attr)
        x = global_add_pool(x, batch)
        return self.mlp(x)


class RedrawProjection:
    def __init__(self, model: torch.nn.Module,
                 redraw_interval: Optional[int] = None):
        self.model = model
        self.redraw_interval = redraw_interval
        self.num_last_redraw = 0

    def redraw_projections(self):
        if not self.model.training or self.redraw_interval is None:
            return
        if self.num_last_redraw >= self.redraw_interval:
            fast_attentions = [
                module for module in self.model.modules()
                if isinstance(module, PerformerAttention)
            ]
            for fast_attention in fast_attentions:
                fast_attention.redraw_projection_matrix()
            self.num_last_redraw = 0
            return
        self.num_last_redraw += 1


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
attn_kwargs = {'dropout': 0.5}
model = GPS(channels=64, pe_dim=8, num_layers=10, attn_type=args.attn_type,
            attn_kwargs=attn_kwargs).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-5)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=20,
                              min_lr=0.00001)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        model.redraw_projection.redraw_projections()
        out = model(data.x, data.pe, data.edge_index, data.edge_attr,
                    data.batch)
        loss = (out.squeeze() - data.y).abs().mean()
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_error = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.pe, data.edge_index, data.edge_attr,
                    data.batch)
        total_error += (out.squeeze() - data.y).abs().sum().item()
    return total_error / len(loader.dataset)


for epoch in range(1, 101):
    loss = train()
    val_mae = test(val_loader)
    test_mae = test(test_loader)
    scheduler.step(val_mae)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_mae:.4f}, '
          f'Test: {test_mae:.4f}')
pytorch/examples/graph_sage_unsup.py000066400000000000000000000036701444262304300203000ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.linear_model import LogisticRegression

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.loader import LinkNeighborLoader
from torch_geometric.nn import GraphSAGE

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]

train_loader = LinkNeighborLoader(
    data,
    batch_size=256,
    shuffle=True,
    neg_sampling_ratio=1.0,
    num_neighbors=[10, 10],
)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data = data.to(device, 'x', 'edge_index')

model = GraphSAGE(
    data.num_node_features,
    hidden_channels=64,
    num_layers=2,
).to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0
    for batch in train_loader:
        batch = batch.to(device)
        optimizer.zero_grad()
        h = model(batch.x, batch.edge_index)
        h_src = h[batch.edge_label_index[0]]
        h_dst = h[batch.edge_label_index[1]]
        pred = (h_src * h_dst).sum(dim=-1)
        loss = F.binary_cross_entropy_with_logits(pred, batch.edge_label)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * pred.size(0)

    return total_loss / data.num_nodes


@torch.no_grad()
def test():
    model.eval()
    out = model(data.x, data.edge_index).cpu()

    clf = LogisticRegression()
    clf.fit(out[data.train_mask], data.y[data.train_mask])

    val_acc = clf.score(out[data.val_mask], data.y[data.val_mask])
    test_acc = clf.score(out[data.test_mask], data.y[data.test_mask])

    return val_acc, test_acc


for epoch in range(1, 51):
    loss = train()
    val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/graph_sage_unsup_ppi.py000066400000000000000000000060321444262304300211430ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
import tqdm
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import f1_score
from sklearn.multioutput import MultiOutputClassifier

from torch_geometric.data import Batch
from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader, LinkNeighborLoader
from torch_geometric.nn import GraphSAGE

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')

# Group all training graphs into a single graph to perform sampling:
train_data = Batch.from_data_list(train_dataset)
loader = LinkNeighborLoader(train_data, batch_size=2048, shuffle=True,
                            neg_sampling_ratio=1.0, num_neighbors=[10, 10],
                            num_workers=6, persistent_workers=True)

# Evaluation loaders (one datapoint corresponds to a graph)
train_loader = DataLoader(train_dataset, batch_size=2)
val_loader = DataLoader(val_dataset, batch_size=2)
test_loader = DataLoader(test_dataset, batch_size=2)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = GraphSAGE(
    in_channels=train_dataset.num_features,
    hidden_channels=64,
    num_layers=2,
    out_channels=64,
).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)


def train():
    model.train()

    total_loss = total_examples = 0
    for data in tqdm.tqdm(loader):
        data = data.to(device)
        optimizer.zero_grad()
        h = model(data.x, data.edge_index)

        h_src = h[data.edge_label_index[0]]
        h_dst = h[data.edge_label_index[1]]
        link_pred = (h_src * h_dst).sum(dim=-1)  # Inner product.

        loss = F.binary_cross_entropy_with_logits(link_pred, data.edge_label)
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * link_pred.numel()
        total_examples += link_pred.numel()

    return total_loss / total_examples


@torch.no_grad()
def encode(loader):
    model.eval()

    xs, ys = [], []
    for data in loader:
        data = data.to(device)
        xs.append(model(data.x, data.edge_index).cpu())
        ys.append(data.y.cpu())
    return torch.cat(xs, dim=0), torch.cat(ys, dim=0)


@torch.no_grad()
def test():
    # Train classifier on training set:
    x, y = encode(train_loader)

    clf = MultiOutputClassifier(SGDClassifier(loss='log', penalty='l2'))
    clf.fit(x, y)

    train_f1 = f1_score(y, clf.predict(x), average='micro')

    # Evaluate on validation set:
    x, y = encode(val_loader)
    val_f1 = f1_score(y, clf.predict(x), average='micro')

    # Evaluate on test set:
    x, y = encode(test_loader)
    test_f1 = f1_score(y, clf.predict(x), average='micro')

    return train_f1, val_f1, test_f1


for epoch in range(1, 6):
    loss = train()
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}')
    train_f1, val_f1, test_f1 = test()
    print(f'Train F1: {train_f1:.4f}, Val F1: {val_f1:.4f}, '
          f'Test F1: {test_f1:.4f}')
pytorch/examples/graph_saint.py000066400000000000000000000071441444262304300172450ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Flickr
from torch_geometric.loader import GraphSAINTRandomWalkSampler
from torch_geometric.nn import GraphConv
from torch_geometric.typing import WITH_TORCH_SPARSE
from torch_geometric.utils import degree

if not WITH_TORCH_SPARSE:
    quit("This example requires 'torch-sparse'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Flickr')
dataset = Flickr(path)
data = dataset[0]
row, col = data.edge_index
data.edge_weight = 1. / degree(col, data.num_nodes)[col]  # Norm by in-degree.

parser = argparse.ArgumentParser()
parser.add_argument('--use_normalization', action='store_true')
args = parser.parse_args()

loader = GraphSAINTRandomWalkSampler(data, batch_size=6000, walk_length=2,
                                     num_steps=5, sample_coverage=100,
                                     save_dir=dataset.processed_dir,
                                     num_workers=4)


class Net(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        in_channels = dataset.num_node_features
        out_channels = dataset.num_classes
        self.conv1 = GraphConv(in_channels, hidden_channels)
        self.conv2 = GraphConv(hidden_channels, hidden_channels)
        self.conv3 = GraphConv(hidden_channels, hidden_channels)
        self.lin = torch.nn.Linear(3 * hidden_channels, out_channels)

    def set_aggr(self, aggr):
        self.conv1.aggr = aggr
        self.conv2.aggr = aggr
        self.conv3.aggr = aggr

    def forward(self, x0, edge_index, edge_weight=None):
        x1 = F.relu(self.conv1(x0, edge_index, edge_weight))
        x1 = F.dropout(x1, p=0.2, training=self.training)
        x2 = F.relu(self.conv2(x1, edge_index, edge_weight))
        x2 = F.dropout(x2, p=0.2, training=self.training)
        x3 = F.relu(self.conv3(x2, edge_index, edge_weight))
        x3 = F.dropout(x3, p=0.2, training=self.training)
        x = torch.cat([x1, x2, x3], dim=-1)
        x = self.lin(x)
        return x.log_softmax(dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(hidden_channels=256).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train():
    model.train()
    model.set_aggr('add' if args.use_normalization else 'mean')

    total_loss = total_examples = 0
    for data in loader:
        data = data.to(device)
        optimizer.zero_grad()

        if args.use_normalization:
            edge_weight = data.edge_norm * data.edge_weight
            out = model(data.x, data.edge_index, edge_weight)
            loss = F.nll_loss(out, data.y, reduction='none')
            loss = (loss * data.node_norm)[data.train_mask].sum()
        else:
            out = model(data.x, data.edge_index)
            loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])

        loss.backward()
        optimizer.step()
        total_loss += loss.item() * data.num_nodes
        total_examples += data.num_nodes
    return total_loss / total_examples


@torch.no_grad()
def test():
    model.eval()
    model.set_aggr('mean')

    out = model(data.x.to(device), data.edge_index.to(device))
    pred = out.argmax(dim=-1)
    correct = pred.eq(data.y.to(device))

    accs = []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        accs.append(correct[mask].sum().item() / mask.sum().item())
    return accs


for epoch in range(1, 51):
    loss = train()
    accs = test()
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Train: {accs[0]:.4f}, '
          f'Val: {accs[1]:.4f}, Test: {accs[2]:.4f}')
pytorch/examples/graph_unet.py000066400000000000000000000036031444262304300170760ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GraphUNet
from torch_geometric.utils import dropout_edge

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        pool_ratios = [2000 / data.num_nodes, 0.5]
        self.unet = GraphUNet(dataset.num_features, 32, dataset.num_classes,
                              depth=3, pool_ratios=pool_ratios)

    def forward(self):
        edge_index, _ = dropout_edge(data.edge_index, p=0.2,
                                     force_undirected=True,
                                     training=self.training)
        x = F.dropout(data.x, p=0.92, training=self.training)

        x = self.unet(x, edge_index)
        return F.log_softmax(x, dim=1)


device = 'cuda' if torch.cuda.is_available() else 'cpu'
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=0.001)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 201):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/hetero/000077500000000000000000000000001444262304300156545ustar00rootroot00000000000000pytorch/examples/hetero/bipartite_sage.py000066400000000000000000000125121444262304300212110ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Embedding, Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import MovieLens
from torch_geometric.nn import SAGEConv
from torch_geometric.nn.conv.gcn_conv import gcn_norm

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/MovieLens')
dataset = MovieLens(path, model_name='all-MiniLM-L6-v2')
data = dataset[0]
data['user'].x = torch.arange(data['user'].num_nodes)
data['user', 'movie'].edge_label = data['user', 'movie'].edge_label.float()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data = data.to(device)

# Add a reverse ('movie', 'rev_rates', 'user') relation for message passing:
data = T.ToUndirected()(data)
del data['movie', 'rev_rates', 'user'].edge_label  # Remove "reverse" label.

# Perform a link-level split into training, validation, and test edges:
train_data, val_data, test_data = T.RandomLinkSplit(
    num_val=0.1,
    num_test=0.1,
    neg_sampling_ratio=0.0,
    edge_types=[('user', 'rates', 'movie')],
    rev_edge_types=[('movie', 'rev_rates', 'user')],
)(data)

# Generate the co-occurence matrix of movies<>movies:
metapath = [('movie', 'rev_rates', 'user'), ('user', 'rates', 'movie')]
train_data = T.AddMetaPaths(metapaths=[metapath])(train_data)

# Apply normalization to filter the metapath:
_, edge_weight = gcn_norm(
    train_data['movie', 'movie'].edge_index,
    num_nodes=train_data['movie'].num_nodes,
    add_self_loops=False,
)
edge_index = train_data['movie', 'movie'].edge_index[:, edge_weight > 0.002]

train_data['movie', 'metapath_0', 'movie'].edge_index = edge_index
val_data['movie', 'metapath_0', 'movie'].edge_index = edge_index
test_data['movie', 'metapath_0', 'movie'].edge_index = edge_index


class MovieGNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()

        self.conv1 = SAGEConv(-1, hidden_channels)
        self.conv2 = SAGEConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index).relu()
        return self.lin(x)


class UserGNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), hidden_channels)
        self.conv3 = SAGEConv((-1, -1), hidden_channels)
        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict):
        movie_x = self.conv1(
            x_dict['movie'],
            edge_index_dict[('movie', 'metapath_0', 'movie')],
        ).relu()

        user_x = self.conv2(
            (x_dict['movie'], x_dict['user']),
            edge_index_dict[('movie', 'rev_rates', 'user')],
        ).relu()

        user_x = self.conv3(
            (movie_x, user_x),
            edge_index_dict[('movie', 'rev_rates', 'user')],
        ).relu()

        return self.lin(user_x)


class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_src, z_dst, edge_label_index):
        row, col = edge_label_index
        z = torch.cat([z_src[row], z_dst[col]], dim=-1)

        z = self.lin1(z).relu()
        z = self.lin2(z)
        return z.view(-1)


class Model(torch.nn.Module):
    def __init__(self, num_users, hidden_channels, out_channels):
        super().__init__()
        self.user_emb = Embedding(num_users, hidden_channels)
        self.user_encoder = UserGNNEncoder(hidden_channels, out_channels)
        self.movie_encoder = MovieGNNEncoder(hidden_channels, out_channels)
        self.decoder = EdgeDecoder(out_channels)

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = {}
        x_dict['user'] = self.user_emb(x_dict['user'])
        z_dict['user'] = self.user_encoder(x_dict, edge_index_dict)
        z_dict['movie'] = self.movie_encoder(
            x_dict['movie'],
            edge_index_dict[('movie', 'metapath_0', 'movie')],
        )
        return self.decoder(z_dict['user'], z_dict['movie'], edge_label_index)


model = Model(data['user'].num_nodes, hidden_channels=64, out_channels=64)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0003)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(
        train_data.x_dict,
        train_data.edge_index_dict,
        train_data['user', 'movie'].edge_label_index,
    )
    loss = F.mse_loss(out, train_data['user', 'movie'].edge_label)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    out = model(
        data.x_dict,
        data.edge_index_dict,
        data['user', 'movie'].edge_label_index,
    ).clamp(min=0, max=5)
    rmse = F.mse_loss(out, data['user', 'movie'].edge_label).sqrt()
    return float(rmse)


for epoch in range(1, 701):
    loss = train()
    train_rmse = test(train_data)
    val_rmse = test(val_data)
    test_rmse = test(test_data)
    print(f'Epoch: {epoch:04d}, Loss: {loss:.4f}, Train: {train_rmse:.4f}, '
          f'Val: {val_rmse:.4f}, Test: {test_rmse:.4f}')
pytorch/examples/hetero/bipartite_sage_unsup.py000066400000000000000000000163341444262304300224510ustar00rootroot00000000000000# An implementation of unsupervised bipartite GraphSAGE using the Alibaba
# Taobao dataset.
import os.path as osp

import torch
import torch.nn.functional as F
import tqdm
from sklearn.metrics import roc_auc_score
from torch.nn import Embedding, Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import Taobao
from torch_geometric.loader import LinkNeighborLoader
from torch_geometric.nn import SAGEConv
from torch_geometric.utils.convert import to_scipy_sparse_matrix

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/Taobao')

dataset = Taobao(path)
data = dataset[0]

data['user'].x = torch.arange(0, data['user'].num_nodes)
data['item'].x = torch.arange(0, data['item'].num_nodes)

# Only consider user<>item relationships for simplicity:
del data['category']
del data['item', 'category']
del data['user', 'item'].time
del data['user', 'item'].behavior

# Add a reverse ('item', 'rev_to', 'user') relation for message passing:
data = T.ToUndirected()(data)

# Perform a link-level split into training, validation, and test edges:
print('Computing data splits...')
train_data, val_data, test_data = T.RandomLinkSplit(
    num_val=0.1,
    num_test=0.1,
    neg_sampling_ratio=1.0,
    add_negative_train_samples=False,
    edge_types=[('user', 'to', 'item')],
    rev_edge_types=[('item', 'rev_to', 'user')],
)(data)
print('Done!')

# Compute sparsified item<>item relationships through users:
print('Computing item<>item relationships...')
mat = to_scipy_sparse_matrix(data['user', 'item'].edge_index).tocsr()
mat = mat[:data['user'].num_nodes, :data['item'].num_nodes]
comat = mat.T @ mat
comat.setdiag(0)
comat = comat >= 3.
comat = comat.tocoo()
row = torch.from_numpy(comat.row).to(torch.long)
col = torch.from_numpy(comat.col).to(torch.long)
item_to_item_edge_index = torch.stack([row, col], dim=0)

# Add the generated item<>item relationships for high-order information:
train_data['item', 'item'].edge_index = item_to_item_edge_index
val_data['item', 'item'].edge_index = item_to_item_edge_index
test_data['item', 'item'].edge_index = item_to_item_edge_index
print('Done!')

train_loader = LinkNeighborLoader(
    data=train_data,
    num_neighbors=[8, 4],
    edge_label_index=('user', 'to', 'item'),
    neg_sampling='binary',
    batch_size=2048,
    shuffle=True,
    num_workers=16,
    drop_last=True,
)

val_loader = LinkNeighborLoader(
    data=val_data,
    num_neighbors=[8, 4],
    edge_label_index=(
        ('user', 'to', 'item'),
        val_data[('user', 'to', 'item')].edge_label_index,
    ),
    edge_label=val_data[('user', 'to', 'item')].edge_label,
    batch_size=2048,
    shuffle=False,
    num_workers=16,
)

test_loader = LinkNeighborLoader(
    data=test_data,
    num_neighbors=[8, 4],
    edge_label_index=(
        ('user', 'to', 'item'),
        test_data[('user', 'to', 'item')].edge_label_index,
    ),
    edge_label=test_data[('user', 'to', 'item')].edge_label,
    batch_size=2048,
    shuffle=False,
    num_workers=16,
)


class ItemGNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv(-1, hidden_channels)
        self.conv2 = SAGEConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index).relu()
        return self.lin(x)


class UserGNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), hidden_channels)
        self.conv3 = SAGEConv((-1, -1), hidden_channels)
        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict):
        item_x = self.conv1(
            x_dict['item'],
            edge_index_dict[('item', 'to', 'item')],
        ).relu()

        user_x = self.conv2(
            (x_dict['item'], x_dict['user']),
            edge_index_dict[('item', 'rev_to', 'user')],
        ).relu()

        user_x = self.conv3(
            (item_x, user_x),
            edge_index_dict[('item', 'rev_to', 'user')],
        ).relu()

        return self.lin(user_x)


class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_src, z_dst, edge_label_index):
        row, col = edge_label_index
        z = torch.cat([z_src[row], z_dst[col]], dim=-1)

        z = self.lin1(z).relu()
        z = self.lin2(z)
        return z.view(-1)


class Model(torch.nn.Module):
    def __init__(self, num_users, num_items, hidden_channels, out_channels):
        super().__init__()
        self.user_emb = Embedding(num_users, hidden_channels, device=device)
        self.item_emb = Embedding(num_items, hidden_channels, device=device)
        self.item_encoder = ItemGNNEncoder(hidden_channels, out_channels)
        self.user_encoder = UserGNNEncoder(hidden_channels, out_channels)
        self.decoder = EdgeDecoder(out_channels)

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = {}
        x_dict['user'] = self.user_emb(x_dict['user'])
        x_dict['item'] = self.item_emb(x_dict['item'])
        z_dict['item'] = self.item_encoder(
            x_dict['item'],
            edge_index_dict[('item', 'to', 'item')],
        )
        z_dict['user'] = self.user_encoder(x_dict, edge_index_dict)

        return self.decoder(z_dict['user'], z_dict['item'], edge_label_index)


model = Model(
    num_users=data['user'].num_nodes,
    num_items=data['item'].num_nodes,
    hidden_channels=64,
    out_channels=64,
).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train():
    model.train()

    total_loss = total_examples = 0
    for batch in tqdm.tqdm(train_loader):
        batch = batch.to(device)
        optimizer.zero_grad()

        pred = model(
            batch.x_dict,
            batch.edge_index_dict,
            batch['user', 'item'].edge_label_index,
        )
        loss = F.binary_cross_entropy_with_logits(
            pred, batch['user', 'item'].edge_label)

        loss.backward()
        optimizer.step()
        total_loss += float(loss)
        total_examples += pred.numel()

    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()

    preds, targets = [], []
    for batch in tqdm.tqdm(loader):
        batch = batch.to(device)

        pred = model(
            batch.x_dict,
            batch.edge_index_dict,
            batch['user', 'item'].edge_label_index,
        ).sigmoid().view(-1).cpu()
        target = batch['user', 'item'].edge_label.long().cpu()

        preds.append(pred)
        targets.append(target)

    pred = torch.cat(preds, dim=0).numpy()
    target = torch.cat(targets, dim=0).numpy()

    return roc_auc_score(target, pred)


for epoch in range(1, 21):
    loss = train()
    val_auc = test(val_loader)
    test_auc = test(test_loader)

    print(f'Epoch: {epoch:02d}, Loss: {loss:4f}, Val: {val_auc:.4f}, '
          f'Test: {test_auc:.4f}')
pytorch/examples/hetero/dmgi_unsup.py000066400000000000000000000076611444262304300204120ustar00rootroot00000000000000# An implementation of "Unsupervised Attributed Multiplex Network
# Embedding" (DMGI) for unsupervised learning on  heterogeneous graphs:
# * Paper:  (AAAI 2020)

import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.linear_model import LogisticRegression
from torch.optim import Adam

import torch_geometric.transforms as T
from torch_geometric.datasets import IMDB
from torch_geometric.nn import GCNConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/IMDB')
dataset = IMDB(path)

metapaths = [
    [('movie', 'actor'), ('actor', 'movie')],  # MAM
    [('movie', 'director'), ('director', 'movie')],  # MDM
]
data = T.AddMetaPaths(metapaths, drop_orig_edge_types=True)(dataset[0])


class DMGI(torch.nn.Module):
    def __init__(self, num_nodes, in_channels, out_channels, num_relations):
        super().__init__()
        self.convs = torch.nn.ModuleList(
            [GCNConv(in_channels, out_channels) for _ in range(num_relations)])
        self.M = torch.nn.Bilinear(out_channels, out_channels, 1)
        self.Z = torch.nn.Parameter(torch.Tensor(num_nodes, out_channels))
        self.reset_parameters()

    def reset_parameters(self):
        for conv in self.convs:
            conv.reset_parameters()
        torch.nn.init.xavier_uniform_(self.M.weight)
        self.M.bias.data.zero_()
        torch.nn.init.xavier_uniform_(self.Z)

    def forward(self, x, edge_indices):
        pos_hs, neg_hs, summaries = [], [], []
        for conv, edge_index in zip(self.convs, edge_indices):
            pos_h = F.dropout(x, p=0.5, training=self.training)
            pos_h = conv(pos_h, edge_index).relu()
            pos_hs.append(pos_h)

            neg_h = F.dropout(x, p=0.5, training=self.training)
            neg_h = neg_h[torch.randperm(neg_h.size(0), device=neg_h.device)]
            neg_h = conv(neg_h, edge_index).relu()
            neg_hs.append(neg_h)

            summaries.append(pos_h.mean(dim=0, keepdim=True))

        return pos_hs, neg_hs, summaries

    def loss(self, pos_hs, neg_hs, summaries):
        loss = 0.
        for pos_h, neg_h, s in zip(pos_hs, neg_hs, summaries):
            s = s.expand_as(pos_h)
            loss += -torch.log(self.M(pos_h, s).sigmoid() + 1e-15).mean()
            loss += -torch.log(1 - self.M(neg_h, s).sigmoid() + 1e-15).mean()

        pos_mean = torch.stack(pos_hs, dim=0).mean(dim=0)
        neg_mean = torch.stack(neg_hs, dim=0).mean(dim=0)

        pos_reg_loss = (self.Z - pos_mean).pow(2).sum()
        neg_reg_loss = (self.Z - neg_mean).pow(2).sum()
        loss += 0.001 * (pos_reg_loss - neg_reg_loss)

        return loss


model = DMGI(data['movie'].num_nodes, data['movie'].x.size(-1),
             out_channels=64, num_relations=len(data.edge_types))
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data, model = data.to(device), model.to(device)

optimizer = Adam(model.parameters(), lr=0.0005, weight_decay=0.0001)


def train():
    model.train()
    optimizer.zero_grad()
    x = data['movie'].x
    edge_indices = data.edge_index_dict.values()
    pos_hs, neg_hs, summaries = model(x, edge_indices)
    loss = model.loss(pos_hs, neg_hs, summaries)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    train_emb = model.Z[data['movie'].train_mask].cpu()
    val_emb = model.Z[data['movie'].val_mask].cpu()
    test_emb = model.Z[data['movie'].test_mask].cpu()

    train_y = data['movie'].y[data['movie'].train_mask].cpu()
    val_y = data['movie'].y[data['movie'].val_mask].cpu()
    test_y = data['movie'].y[data['movie'].test_mask].cpu()

    clf = LogisticRegression().fit(train_emb, train_y)
    return clf.score(val_emb, val_y), clf.score(test_emb, test_y)


for epoch in range(1, 1001):
    loss = train()
    if epoch % 50 == 0:
        val_acc, test_acc = test()
        print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
              f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/hetero/han_imdb.py000066400000000000000000000052551444262304300177760ustar00rootroot00000000000000import os.path as osp
from typing import Dict, List, Union

import torch
import torch.nn.functional as F
from torch import nn

import torch_geometric.transforms as T
from torch_geometric.datasets import IMDB
from torch_geometric.nn import HANConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/IMDB')
metapaths = [[('movie', 'actor'), ('actor', 'movie')],
             [('movie', 'director'), ('director', 'movie')]]
transform = T.AddMetaPaths(metapaths=metapaths, drop_orig_edge_types=True,
                           drop_unconnected_node_types=True)
dataset = IMDB(path, transform=transform)
data = dataset[0]
print(data)


class HAN(nn.Module):
    def __init__(self, in_channels: Union[int, Dict[str, int]],
                 out_channels: int, hidden_channels=128, heads=8):
        super().__init__()
        self.han_conv = HANConv(in_channels, hidden_channels, heads=heads,
                                dropout=0.6, metadata=data.metadata())
        self.lin = nn.Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict):
        out = self.han_conv(x_dict, edge_index_dict)
        out = self.lin(out['movie'])
        return out


model = HAN(in_channels=-1, out_channels=3)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data, model = data.to(device), model.to(device)

with torch.no_grad():  # Initialize lazy modules.
    out = model(data.x_dict, data.edge_index_dict)

optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=0.001)


def train() -> float:
    model.train()
    optimizer.zero_grad()
    out = model(data.x_dict, data.edge_index_dict)
    mask = data['movie'].train_mask
    loss = F.cross_entropy(out[mask], data['movie'].y[mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test() -> List[float]:
    model.eval()
    pred = model(data.x_dict, data.edge_index_dict).argmax(dim=-1)

    accs = []
    for split in ['train_mask', 'val_mask', 'test_mask']:
        mask = data['movie'][split]
        acc = (pred[mask] == data['movie'].y[mask]).sum() / mask.sum()
        accs.append(float(acc))
    return accs


best_val_acc = 0
start_patience = patience = 100
for epoch in range(1, 200):

    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')

    if best_val_acc <= val_acc:
        patience = start_patience
        best_val_acc = val_acc
    else:
        patience -= 1

    if patience <= 0:
        print('Stopping training as validation accuracy did not improve '
              f'for {start_patience} epochs')
        break
pytorch/examples/hetero/hetero_conv_dblp.py000066400000000000000000000045641444262304300215530ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import DBLP
from torch_geometric.nn import HeteroConv, Linear, SAGEConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/DBLP')
# We initialize conference node features with a single one-vector as feature:
dataset = DBLP(path, transform=T.Constant(node_types='conference'))
data = dataset[0]
print(data)


class HeteroGNN(torch.nn.Module):
    def __init__(self, metadata, hidden_channels, out_channels, num_layers):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            conv = HeteroConv({
                edge_type: SAGEConv((-1, -1), hidden_channels)
                for edge_type in metadata[1]
            })
            self.convs.append(conv)

        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict):
        for conv in self.convs:
            x_dict = conv(x_dict, edge_index_dict)
            x_dict = {key: F.leaky_relu(x) for key, x in x_dict.items()}
        return self.lin(x_dict['author'])


model = HeteroGNN(data.metadata(), hidden_channels=64, out_channels=4,
                  num_layers=2)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data, model = data.to(device), model.to(device)

with torch.no_grad():  # Initialize lazy modules.
    out = model(data.x_dict, data.edge_index_dict)

optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=0.001)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x_dict, data.edge_index_dict)
    mask = data['author'].train_mask
    loss = F.cross_entropy(out[mask], data['author'].y[mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x_dict, data.edge_index_dict).argmax(dim=-1)

    accs = []
    for split in ['train_mask', 'val_mask', 'test_mask']:
        mask = data['author'][split]
        acc = (pred[mask] == data['author'].y[mask]).sum() / mask.sum()
        accs.append(float(acc))
    return accs


for epoch in range(1, 101):
    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/hetero/hetero_link_pred.py000066400000000000000000000100421444262304300215400ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

import torch_geometric.transforms as T
from torch_geometric.datasets import MovieLens
from torch_geometric.nn import SAGEConv, to_hetero

parser = argparse.ArgumentParser()
parser.add_argument('--use_weighted_loss', action='store_true',
                    help='Whether to use weighted MSE loss.')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/MovieLens')
dataset = MovieLens(path, model_name='all-MiniLM-L6-v2')
data = dataset[0].to(device)

# Add user node features for message passing:
data['user'].x = torch.eye(data['user'].num_nodes, device=device)
del data['user'].num_nodes

# Add a reverse ('movie', 'rev_rates', 'user') relation for message passing:
data = T.ToUndirected()(data)
del data['movie', 'rev_rates', 'user'].edge_label  # Remove "reverse" label.

# Perform a link-level split into training, validation, and test edges:
train_data, val_data, test_data = T.RandomLinkSplit(
    num_val=0.1,
    num_test=0.1,
    neg_sampling_ratio=0.0,
    edge_types=[('user', 'rates', 'movie')],
    rev_edge_types=[('movie', 'rev_rates', 'user')],
)(data)

# We have an unbalanced dataset with many labels for rating 3 and 4, and very
# few for 0 and 1. Therefore we use a weighted MSE loss.
if args.use_weighted_loss:
    weight = torch.bincount(train_data['user', 'movie'].edge_label)
    weight = weight.max() / weight
else:
    weight = None


def weighted_mse_loss(pred, target, weight=None):
    weight = 1. if weight is None else weight[target].to(pred.dtype)
    return (weight * (pred - target.to(pred.dtype)).pow(2)).mean()


class GNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_dict, edge_label_index):
        row, col = edge_label_index
        z = torch.cat([z_dict['user'][row], z_dict['movie'][col]], dim=-1)

        z = self.lin1(z).relu()
        z = self.lin2(z)
        return z.view(-1)


class Model(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.encoder = GNNEncoder(hidden_channels, hidden_channels)
        self.encoder = to_hetero(self.encoder, data.metadata(), aggr='sum')
        self.decoder = EdgeDecoder(hidden_channels)

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = self.encoder(x_dict, edge_index_dict)
        return self.decoder(z_dict, edge_label_index)


model = Model(hidden_channels=32).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()
    optimizer.zero_grad()
    pred = model(train_data.x_dict, train_data.edge_index_dict,
                 train_data['user', 'movie'].edge_label_index)
    target = train_data['user', 'movie'].edge_label
    loss = weighted_mse_loss(pred, target, weight)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    pred = model(data.x_dict, data.edge_index_dict,
                 data['user', 'movie'].edge_label_index)
    pred = pred.clamp(min=0, max=5)
    target = data['user', 'movie'].edge_label.float()
    rmse = F.mse_loss(pred, target).sqrt()
    return float(rmse)


for epoch in range(1, 301):
    loss = train()
    train_rmse = test(train_data)
    val_rmse = test(val_data)
    test_rmse = test(test_data)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_rmse:.4f}, '
          f'Val: {val_rmse:.4f}, Test: {test_rmse:.4f}')
pytorch/examples/hetero/hgt_dblp.py000066400000000000000000000050101444262304300200050ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import DBLP
from torch_geometric.nn import HGTConv, Linear

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/DBLP')
# We initialize conference node features with a single one-vector as feature:
dataset = DBLP(path, transform=T.Constant(node_types='conference'))
data = dataset[0]
print(data)


class HGT(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels, num_heads, num_layers):
        super().__init__()

        self.lin_dict = torch.nn.ModuleDict()
        for node_type in data.node_types:
            self.lin_dict[node_type] = Linear(-1, hidden_channels)

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            conv = HGTConv(hidden_channels, hidden_channels, data.metadata(),
                           num_heads, group='sum')
            self.convs.append(conv)

        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict):
        x_dict = {
            node_type: self.lin_dict[node_type](x).relu_()
            for node_type, x in x_dict.items()
        }

        for conv in self.convs:
            x_dict = conv(x_dict, edge_index_dict)

        return self.lin(x_dict['author'])


model = HGT(hidden_channels=64, out_channels=4, num_heads=2, num_layers=1)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data, model = data.to(device), model.to(device)

with torch.no_grad():  # Initialize lazy modules.
    out = model(data.x_dict, data.edge_index_dict)

optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=0.001)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x_dict, data.edge_index_dict)
    mask = data['author'].train_mask
    loss = F.cross_entropy(out[mask], data['author'].y[mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x_dict, data.edge_index_dict).argmax(dim=-1)

    accs = []
    for split in ['train_mask', 'val_mask', 'test_mask']:
        mask = data['author'][split]
        acc = (pred[mask] == data['author'].y[mask]).sum() / mask.sum()
        accs.append(float(acc))
    return accs


for epoch in range(1, 101):
    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/hetero/hierarchical_sage.py000066400000000000000000000101101444262304300216340ustar00rootroot00000000000000import argparse

import torch
import torch.nn.functional as F
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.datasets import OGB_MAG
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import HeteroConv, Linear, SAGEConv
from torch_geometric.utils import trim_to_layer

parser = argparse.ArgumentParser()
parser.add_argument('--device', type=str, default='cuda')
parser.add_argument('--use-sparse-tensor', action='store_true')
args = parser.parse_args()

device = args.device if torch.cuda.is_available() else 'cpu'

transforms = [T.ToUndirected(merge=True)]
if args.use_sparse_tensor:
    transforms.append(T.ToSparseTensor())
dataset = OGB_MAG(root='../../data', preprocess='metapath2vec',
                  transform=T.Compose(transforms))
data = dataset[0].to(device, 'x', 'y')


class HierarchicalHeteroGraphSage(torch.nn.Module):
    def __init__(self, edge_types, hidden_channels, out_channels, num_layers):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            conv = HeteroConv(
                {
                    edge_type: SAGEConv((-1, -1), hidden_channels)
                    for edge_type in edge_types
                }, aggr='sum')
            self.convs.append(conv)

        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x_dict, edge_index_dict, num_sampled_edges_dict,
                num_sampled_nodes_dict):

        for i, conv in enumerate(self.convs):
            x_dict, edge_index_dict, _ = trim_to_layer(
                layer=i,
                num_sampled_nodes_per_hop=num_sampled_nodes_dict,
                num_sampled_edges_per_hop=num_sampled_edges_dict,
                x=x_dict,
                edge_index=edge_index_dict,
            )

            x_dict = conv(x_dict, edge_index_dict)
            x_dict = {key: x.relu() for key, x in x_dict.items()}

        return self.lin(x_dict['paper'])


model = HierarchicalHeteroGraphSage(
    edge_types=data.edge_types,
    hidden_channels=64,
    out_channels=dataset.num_classes,
    num_layers=2,
).to(args.device)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

kwargs = {'batch_size': 1024, 'num_workers': 0}
train_loader = NeighborLoader(
    data,
    num_neighbors=[10] * 2,
    shuffle=True,
    input_nodes=('paper', data['paper'].train_mask),
    **kwargs,
)

val_loader = NeighborLoader(
    data,
    num_neighbors=[10] * 2,
    shuffle=False,
    input_nodes=('paper', data['paper'].val_mask),
    **kwargs,
)


def train():
    model.train()

    total_examples = total_loss = 0
    for batch in tqdm(train_loader):
        batch = batch.to(device)
        optimizer.zero_grad()

        out = model(
            batch.x_dict,
            batch.adj_t_dict
            if args.use_sparse_tensor else batch.edge_index_dict,
            num_sampled_nodes_dict=batch.num_sampled_nodes_dict,
            num_sampled_edges_dict=batch.num_sampled_edges_dict,
        )

        batch_size = batch['paper'].batch_size
        loss = F.cross_entropy(out[:batch_size], batch['paper'].y[:batch_size])
        loss.backward()
        optimizer.step()

        total_examples += batch_size
        total_loss += float(loss) * batch_size

    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()

    total_examples = total_correct = 0
    for batch in tqdm(loader):
        batch = batch.to(device)
        out = model(
            batch.x_dict,
            batch.adj_t_dict
            if args.use_sparse_tensor else batch.edge_index_dict,
            num_sampled_nodes_dict=batch.num_sampled_nodes_dict,
            num_sampled_edges_dict=batch.num_sampled_edges_dict,
        )

        batch_size = batch['paper'].batch_size
        pred = out[:batch_size].argmax(dim=-1)
        total_examples += batch_size
        total_correct += int((pred == batch['paper'].y[:batch_size]).sum())

    return total_correct / total_examples


for epoch in range(1, 6):
    loss = train()
    val_acc = test(val_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_acc:.4f}')
pytorch/examples/hetero/load_csv.py000066400000000000000000000101351444262304300200200ustar00rootroot00000000000000import os.path as osp

import pandas as pd
import torch
from sentence_transformers import SentenceTransformer

from torch_geometric.data import HeteroData, download_url, extract_zip
from torch_geometric.transforms import RandomLinkSplit, ToUndirected

url = 'https://files.grouplens.org/datasets/movielens/ml-latest-small.zip'
root = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/MovieLens')
extract_zip(download_url(url, root), root)
movie_path = osp.join(root, 'ml-latest-small', 'movies.csv')
rating_path = osp.join(root, 'ml-latest-small', 'ratings.csv')


def load_node_csv(path, index_col, encoders=None, **kwargs):
    df = pd.read_csv(path, index_col=index_col, **kwargs)
    mapping = {index: i for i, index in enumerate(df.index.unique())}

    x = None
    if encoders is not None:
        xs = [encoder(df[col]) for col, encoder in encoders.items()]
        x = torch.cat(xs, dim=-1)

    return x, mapping


def load_edge_csv(path, src_index_col, src_mapping, dst_index_col, dst_mapping,
                  encoders=None, **kwargs):
    df = pd.read_csv(path, **kwargs)

    src = [src_mapping[index] for index in df[src_index_col]]
    dst = [dst_mapping[index] for index in df[dst_index_col]]
    edge_index = torch.tensor([src, dst])

    edge_attr = None
    if encoders is not None:
        edge_attrs = [encoder(df[col]) for col, encoder in encoders.items()]
        edge_attr = torch.cat(edge_attrs, dim=-1)

    return edge_index, edge_attr


class SequenceEncoder:
    # The 'SequenceEncoder' encodes raw column strings into embeddings.
    def __init__(self, model_name='all-MiniLM-L6-v2', device=None):
        self.device = device
        self.model = SentenceTransformer(model_name, device=device)

    @torch.no_grad()
    def __call__(self, df):
        x = self.model.encode(df.values, show_progress_bar=True,
                              convert_to_tensor=True, device=self.device)
        return x.cpu()


class GenresEncoder:
    # The 'GenreEncoder' splits the raw column strings by 'sep' and converts
    # individual elements to categorical labels.
    def __init__(self, sep='|'):
        self.sep = sep

    def __call__(self, df):
        genres = set(g for col in df.values for g in col.split(self.sep))
        mapping = {genre: i for i, genre in enumerate(genres)}

        x = torch.zeros(len(df), len(mapping))
        for i, col in enumerate(df.values):
            for genre in col.split(self.sep):
                x[i, mapping[genre]] = 1
        return x


class IdentityEncoder:
    # The 'IdentityEncoder' takes the raw column values and converts them to
    # PyTorch tensors.
    def __init__(self, dtype=None):
        self.dtype = dtype

    def __call__(self, df):
        return torch.from_numpy(df.values).view(-1, 1).to(self.dtype)


user_x, user_mapping = load_node_csv(rating_path, index_col='userId')

movie_x, movie_mapping = load_node_csv(
    movie_path, index_col='movieId', encoders={
        'title': SequenceEncoder(),
        'genres': GenresEncoder()
    })

edge_index, edge_label = load_edge_csv(
    rating_path,
    src_index_col='userId',
    src_mapping=user_mapping,
    dst_index_col='movieId',
    dst_mapping=movie_mapping,
    encoders={'rating': IdentityEncoder(dtype=torch.long)},
)

data = HeteroData()
data['user'].num_nodes = len(user_mapping)  # Users do not have any features.
data['movie'].x = movie_x
data['user', 'rates', 'movie'].edge_index = edge_index
data['user', 'rates', 'movie'].edge_label = edge_label
print(data)

# We can now convert `data` into an appropriate format for training a
# graph-based machine learning model:

# 1. Add a reverse ('movie', 'rev_rates', 'user') relation for message passing.
data = ToUndirected()(data)
del data['movie', 'rev_rates', 'user'].edge_label  # Remove "reverse" label.

# 2. Perform a link-level split into training, validation, and test edges.
transform = RandomLinkSplit(
    num_val=0.05,
    num_test=0.1,
    neg_sampling_ratio=0.0,
    edge_types=[('user', 'rates', 'movie')],
    rev_edge_types=[('movie', 'rev_rates', 'user')],
)
train_data, val_data, test_data = transform(data)
print(train_data)
print(val_data)
print(test_data)
pytorch/examples/hetero/metapath2vec.py000066400000000000000000000040741444262304300206160ustar00rootroot00000000000000# Reaches around 91.8% Micro-F1 after 5 epochs.

import os.path as osp

import torch

from torch_geometric.datasets import AMiner
from torch_geometric.nn import MetaPath2Vec

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/AMiner')
dataset = AMiner(path)
data = dataset[0]

metapath = [
    ('author', 'writes', 'paper'),
    ('paper', 'published_in', 'venue'),
    ('venue', 'publishes', 'paper'),
    ('paper', 'written_by', 'author'),
]

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = MetaPath2Vec(data.edge_index_dict, embedding_dim=128,
                     metapath=metapath, walk_length=50, context_size=7,
                     walks_per_node=5, num_negative_samples=5,
                     sparse=True).to(device)

loader = model.loader(batch_size=128, shuffle=True, num_workers=6)
optimizer = torch.optim.SparseAdam(list(model.parameters()), lr=0.01)


def train(epoch, log_steps=100, eval_steps=2000):
    model.train()

    total_loss = 0
    for i, (pos_rw, neg_rw) in enumerate(loader):
        optimizer.zero_grad()
        loss = model.loss(pos_rw.to(device), neg_rw.to(device))
        loss.backward()
        optimizer.step()

        total_loss += loss.item()
        if (i + 1) % log_steps == 0:
            print((f'Epoch: {epoch}, Step: {i + 1:05d}/{len(loader)}, '
                   f'Loss: {total_loss / log_steps:.4f}'))
            total_loss = 0

        if (i + 1) % eval_steps == 0:
            acc = test()
            print((f'Epoch: {epoch}, Step: {i + 1:05d}/{len(loader)}, '
                   f'Acc: {acc:.4f}'))


@torch.no_grad()
def test(train_ratio=0.1):
    model.eval()

    z = model('author', batch=data['author'].y_index.to(device))
    y = data['author'].y

    perm = torch.randperm(z.size(0))
    train_perm = perm[:int(z.size(0) * train_ratio)]
    test_perm = perm[int(z.size(0) * train_ratio):]

    return model.test(z[train_perm], y[train_perm], z[test_perm], y[test_perm],
                      max_iter=150)


for epoch in range(1, 6):
    train(epoch)
    acc = test()
    print(f'Epoch: {epoch}, Accuracy: {acc:.4f}')
pytorch/examples/hetero/to_hetero_mag.py000066400000000000000000000066001444262304300210440ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import ReLU
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.datasets import OGB_MAG
from torch_geometric.loader import HGTLoader, NeighborLoader
from torch_geometric.nn import Linear, SAGEConv, Sequential, to_hetero

parser = argparse.ArgumentParser()
parser.add_argument('--use_hgt_loader', action='store_true')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data/OGB')
transform = T.ToUndirected(merge=True)
dataset = OGB_MAG(path, preprocess='metapath2vec', transform=transform)

# Already send node features/labels to GPU for faster access during sampling:
data = dataset[0].to(device, 'x', 'y')

train_input_nodes = ('paper', data['paper'].train_mask)
val_input_nodes = ('paper', data['paper'].val_mask)
kwargs = {'batch_size': 1024, 'num_workers': 6, 'persistent_workers': True}

if not args.use_hgt_loader:
    train_loader = NeighborLoader(data, num_neighbors=[10] * 2, shuffle=True,
                                  input_nodes=train_input_nodes, **kwargs)
    val_loader = NeighborLoader(data, num_neighbors=[10] * 2,
                                input_nodes=val_input_nodes, **kwargs)
else:
    train_loader = HGTLoader(data, num_samples=[1024] * 4, shuffle=True,
                             input_nodes=train_input_nodes, **kwargs)
    val_loader = HGTLoader(data, num_samples=[1024] * 4,
                           input_nodes=val_input_nodes, **kwargs)

model = Sequential('x, edge_index', [
    (SAGEConv((-1, -1), 64), 'x, edge_index -> x'),
    ReLU(inplace=True),
    (SAGEConv((-1, -1), 64), 'x, edge_index -> x'),
    ReLU(inplace=True),
    (Linear(-1, dataset.num_classes), 'x -> x'),
])
model = to_hetero(model, data.metadata(), aggr='sum').to(device)


@torch.no_grad()
def init_params():
    # Initialize lazy parameters via forwarding a single batch to the model:
    batch = next(iter(train_loader))
    batch = batch.to(device, 'edge_index')
    model(batch.x_dict, batch.edge_index_dict)


def train():
    model.train()

    total_examples = total_loss = 0
    for batch in tqdm(train_loader):
        optimizer.zero_grad()
        batch = batch.to(device, 'edge_index')
        batch_size = batch['paper'].batch_size
        out = model(batch.x_dict, batch.edge_index_dict)['paper'][:batch_size]
        loss = F.cross_entropy(out, batch['paper'].y[:batch_size])
        loss.backward()
        optimizer.step()

        total_examples += batch_size
        total_loss += float(loss) * batch_size

    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()

    total_examples = total_correct = 0
    for batch in tqdm(loader):
        batch = batch.to(device, 'edge_index')
        batch_size = batch['paper'].batch_size
        out = model(batch.x_dict, batch.edge_index_dict)['paper'][:batch_size]
        pred = out.argmax(dim=-1)

        total_examples += batch_size
        total_correct += int((pred == batch['paper'].y[:batch_size]).sum())

    return total_correct / total_examples


init_params()  # Initialize parameters.
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(1, 21):
    loss = train()
    val_acc = test(val_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_acc:.4f}')
pytorch/examples/hierarchical_sampling.py000066400000000000000000000032561444262304300212560ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn.models.basic_gnn import GraphSAGE

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Reddit')
dataset = Reddit(path)

# Already send node features/labels to GPU for faster access during sampling:
data = dataset[0].to(device, 'x', 'y')

kwargs = {'batch_size': 1024, 'num_workers': 6, 'persistent_workers': True}
loader = NeighborLoader(data, input_nodes=data.train_mask,
                        num_neighbors=[20, 10, 5], shuffle=True, **kwargs)

model = GraphSAGE(
    dataset.num_features,
    hidden_channels=64,
    out_channels=dataset.num_classes,
    num_layers=3,
).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(trim=False):
    for batch in tqdm(loader):
        optimizer.zero_grad()
        batch = batch.to(device)

        if not trim:
            out = model(batch.x, batch.edge_index)
        else:
            out = model(
                batch.x,
                batch.edge_index,
                num_sampled_nodes_per_hop=batch.num_sampled_nodes,
                num_sampled_edges_per_hop=batch.num_sampled_edges,
            )

        out = out[:batch.batch_size]
        y = batch.y[:batch.batch_size]

        loss = F.cross_entropy(out, y)
        loss.backward()
        optimizer.step()


print('One epoch training without Hierarchical Graph Sampling:')
train(trim=False)

print('One epoch training with Hierarchical Graph Sampling:')
train(trim=True)
pytorch/examples/infomax_inductive.py000066400000000000000000000056621444262304300204640ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn as nn
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import DeepGraphInfomax, SAGEConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Reddit')
dataset = Reddit(path)
data = dataset[0].to(device, 'x', 'edge_index')

train_loader = NeighborLoader(data, num_neighbors=[10, 10, 25], batch_size=256,
                              shuffle=True, num_workers=12)
test_loader = NeighborLoader(data, num_neighbors=[10, 10, 25], batch_size=256,
                             num_workers=12)


class Encoder(nn.Module):
    def __init__(self, in_channels, hidden_channels):
        super().__init__()
        self.convs = torch.nn.ModuleList([
            SAGEConv(in_channels, hidden_channels),
            SAGEConv(hidden_channels, hidden_channels),
            SAGEConv(hidden_channels, hidden_channels)
        ])

        self.activations = torch.nn.ModuleList()
        self.activations.extend([
            nn.PReLU(hidden_channels),
            nn.PReLU(hidden_channels),
            nn.PReLU(hidden_channels)
        ])

    def forward(self, x, edge_index, batch_size):
        for conv, act in zip(self.convs, self.activations):
            x = conv(x, edge_index)
            x = act(x)
        return x[:batch_size]


def corruption(x, edge_index, batch_size):
    return x[torch.randperm(x.size(0))], edge_index, batch_size


model = DeepGraphInfomax(
    hidden_channels=512, encoder=Encoder(dataset.num_features, 512),
    summary=lambda z, *args, **kwargs: torch.sigmoid(z.mean(dim=0)),
    corruption=corruption).to(device)

model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)


def train(epoch):
    model.train()

    total_loss = total_examples = 0
    for batch in tqdm(train_loader, desc=f'Epoch {epoch:02d}'):
        optimizer.zero_grad()
        pos_z, neg_z, summary = model(batch.x, batch.edge_index,
                                      batch.batch_size)
        loss = model.loss(pos_z, neg_z, summary)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * pos_z.size(0)
        total_examples += pos_z.size(0)

    return total_loss / total_examples


@torch.no_grad()
def test():
    model.eval()

    zs = []
    for batch in tqdm(test_loader, desc='Evaluating'):
        pos_z, _, _ = model(batch.x, batch.edge_index, batch.batch_size)
        zs.append(pos_z.cpu())
    z = torch.cat(zs, dim=0)
    train_val_mask = data.train_mask | data.val_mask
    acc = model.test(z[train_val_mask], data.y[train_val_mask],
                     z[data.test_mask], data.y[data.test_mask], max_iter=10000)
    return acc


for epoch in range(1, 31):
    loss = train(epoch)
    print(f'Epoch {epoch:02d}, Loss: {loss:.4f}')

test_acc = test()
print(f'Test Accuracy: {test_acc:.4f}')
pytorch/examples/infomax_transductive.py000066400000000000000000000032321444262304300211740ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn as nn

from torch_geometric.datasets import Planetoid
from torch_geometric.nn import DeepGraphInfomax, GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset)


class Encoder(nn.Module):
    def __init__(self, in_channels, hidden_channels):
        super().__init__()
        self.conv = GCNConv(in_channels, hidden_channels, cached=True)
        self.prelu = nn.PReLU(hidden_channels)

    def forward(self, x, edge_index):
        x = self.conv(x, edge_index)
        x = self.prelu(x)
        return x


def corruption(x, edge_index):
    return x[torch.randperm(x.size(0))], edge_index


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = DeepGraphInfomax(
    hidden_channels=512, encoder=Encoder(dataset.num_features, 512),
    summary=lambda z, *args, **kwargs: torch.sigmoid(z.mean(dim=0)),
    corruption=corruption).to(device)
data = dataset[0].to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train():
    model.train()
    optimizer.zero_grad()
    pos_z, neg_z, summary = model(data.x, data.edge_index)
    loss = model.loss(pos_z, neg_z, summary)
    loss.backward()
    optimizer.step()
    return loss.item()


def test():
    model.eval()
    z, _, _ = model(data.x, data.edge_index)
    acc = model.test(z[data.train_mask], data.y[data.train_mask],
                     z[data.test_mask], data.y[data.test_mask], max_iter=150)
    return acc


for epoch in range(1, 301):
    loss = train()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')
acc = test()
print(f'Accuracy: {acc:.4f}')
pytorch/examples/jit/000077500000000000000000000000001444262304300151545ustar00rootroot00000000000000pytorch/examples/jit/film.py000066400000000000000000000057201444262304300164610ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import f1_score
from torch import Tensor
from torch.nn import BatchNorm1d

from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader
from torch_geometric.nn import FiLMConv

path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)


class FiLM(torch.nn.Module):
    def __init__(self, in_channels: int, hidden_channels: int,
                 out_channels: int, num_layers: int, dropout: float = 0.0):
        super().__init__()
        self.dropout = dropout

        self.convs = torch.nn.ModuleList()
        self.convs.append(FiLMConv(in_channels, hidden_channels).jittable())
        for _ in range(num_layers - 2):
            conv = FiLMConv(hidden_channels, hidden_channels).jittable()
            self.convs.append(conv)
        self.last_conv = FiLMConv(hidden_channels, out_channels,
                                  act=None).jittable()

        self.norms = torch.nn.ModuleList()
        for _ in range(num_layers - 1):
            self.norms.append(BatchNorm1d(hidden_channels))

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for conv, norm in zip(self.convs, self.norms):
            x = norm(conv(x, edge_index))
            x = F.dropout(x, p=self.dropout, training=self.training)
        x = self.last_conv(x, edge_index)
        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = FiLM(train_dataset.num_features, 320, train_dataset.num_classes,
             num_layers=4, dropout=0.1)
model = torch.jit.script(model).to(device)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = criterion(model(data.x, data.edge_index), data.y)
        total_loss += loss.item() * data.num_graphs
        loss.backward()
        optimizer.step()
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        out = model(data.x.to(device), data.edge_index.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 501):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/jit/gat.py000066400000000000000000000036711444262304300163100ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GATConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', '..', 'data',
                dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GATConv(dataset.num_features, 8, heads=8,
                             dropout=0.6).jittable()

        self.conv2 = GATConv(64, dataset.num_classes, heads=1, concat=True,
                             dropout=0.6).jittable()

    def forward(self, x, edge_index):
        x = F.dropout(x, p=0.6, training=self.training)
        x = F.elu(self.conv1(x, edge_index))
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), dataset[0].to(device)
model = torch.jit.script(model)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return loss.item()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(data.x, data.edge_index), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


for epoch in range(1, 201):
    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/jit/gcn.py000066400000000000000000000042761444262304300163060ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch import Tensor

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
path = osp.join(osp.dirname(osp.realpath(__file__)), 'data', 'Planetoid')
dataset = Planetoid(path, 'Cora', transform=T.NormalizeFeatures())
data = dataset[0]


class GCN(torch.nn.Module):
    def __init__(self, in_channels: int, hidden_channels: int,
                 out_channels: int):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels).jittable()
        self.conv2 = GCNConv(hidden_channels, out_channels).jittable()

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        return x


model = GCN(dataset.num_features, 16, dataset.num_classes)
model = torch.jit.script(model).to(device)
data = data.to(device)
optimizer = torch.optim.Adam([
    dict(params=model.conv1.parameters(), weight_decay=5e-4),
    dict(params=model.conv2.parameters(), weight_decay=0)
], lr=0.01)  # Only perform weight-decay on first convolution.


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = final_test_acc = 0
for epoch in range(1, 201):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/jit/gin.py000066400000000000000000000062011444262304300163020ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import BatchNorm1d as BatchNorm
from torch.nn import Linear, ReLU, Sequential

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GINConv, global_add_pool
from torch_geometric.transforms import OneHotDegree

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', '..', 'data', 'TU')
dataset = TUDataset(path, name='IMDB-BINARY', transform=OneHotDegree(135))
dataset = dataset.shuffle()
test_dataset = dataset[:len(dataset) // 10]
train_dataset = dataset[len(dataset) // 10:]
test_loader = DataLoader(test_dataset, batch_size=128)
train_loader = DataLoader(train_dataset, batch_size=128)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        self.batch_norms = torch.nn.ModuleList()

        for i in range(num_layers):
            mlp = Sequential(
                Linear(in_channels, 2 * hidden_channels),
                BatchNorm(2 * hidden_channels),
                ReLU(),
                Linear(2 * hidden_channels, hidden_channels),
            )
            conv = GINConv(mlp, train_eps=True).jittable()

            self.convs.append(conv)
            self.batch_norms.append(BatchNorm(hidden_channels))

            in_channels = hidden_channels

        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.batch_norm1 = BatchNorm(hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, batch):
        for conv, batch_norm in zip(self.convs, self.batch_norms):
            x = F.relu(batch_norm(conv(x, edge_index)))
        x = global_add_pool(x, batch)
        x = F.relu(self.batch_norm1(self.lin1(x)))
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin2(x)
        return F.log_softmax(x, dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, 64, dataset.num_classes, num_layers=3)
model = model.to(device)
model = torch.jit.script(model)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = 0.
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()
    return total_loss / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.batch)
        pred = out.max(dim=1)[1]
        total_correct += pred.eq(data.y).sum().item()
    return total_correct / len(loader.dataset)


for epoch in range(1, 101):
    loss = train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Train: {train_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/kge_fb15k_237.py000066400000000000000000000047331444262304300171000ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.optim as optim

from torch_geometric.datasets import FB15k_237
from torch_geometric.nn import ComplEx, DistMult, RotatE, TransE

model_map = {
    'transe': TransE,
    'complex': ComplEx,
    'distmult': DistMult,
    'rotate': RotatE,
}

parser = argparse.ArgumentParser()
parser.add_argument('--model', choices=model_map.keys(), type=str.lower,
                    required=True)
args = parser.parse_args()

device = 'cuda' if torch.cuda.is_available() else 'cpu'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'FB15k')

train_data = FB15k_237(path, split='train')[0].to(device)
val_data = FB15k_237(path, split='val')[0].to(device)
test_data = FB15k_237(path, split='test')[0].to(device)

model_arg_map = {'rotate': {'margin': 9.0}}
model = model_map[args.model](
    num_nodes=train_data.num_nodes,
    num_relations=train_data.num_edge_types,
    hidden_channels=50,
    **model_arg_map.get(args.model, {}),
).to(device)

loader = model.loader(
    head_index=train_data.edge_index[0],
    rel_type=train_data.edge_type,
    tail_index=train_data.edge_index[1],
    batch_size=1000,
    shuffle=True,
)

optimizer_map = {
    'transe': optim.Adam(model.parameters(), lr=0.01),
    'complex': optim.Adagrad(model.parameters(), lr=0.001, weight_decay=1e-6),
    'distmult': optim.Adam(model.parameters(), lr=0.0001, weight_decay=1e-6),
    'rotate': optim.Adam(model.parameters(), lr=1e-3),
}
optimizer = optimizer_map[args.model]


def train():
    model.train()
    total_loss = total_examples = 0
    for head_index, rel_type, tail_index in loader:
        optimizer.zero_grad()
        loss = model.loss(head_index, rel_type, tail_index)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * head_index.numel()
        total_examples += head_index.numel()
    return total_loss / total_examples


@torch.no_grad()
def test(data):
    model.eval()
    return model.test(
        head_index=data.edge_index[0],
        rel_type=data.edge_type,
        tail_index=data.edge_index[1],
        batch_size=20000,
        k=10,
    )


for epoch in range(1, 501):
    loss = train()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')
    if epoch % 25 == 0:
        rank, hits = test(val_data)
        print(f'Epoch: {epoch:03d}, Val Mean Rank: {rank:.2f}, '
              f'Val Hits@10: {hits:.4f}')

rank, hits_at_10 = test(test_data)
print(f'Test Mean Rank: {rank:.2f}, Test Hits@10: {hits_at_10:.4f}')
pytorch/examples/kuzu/000077500000000000000000000000001444262304300153645ustar00rootroot00000000000000pytorch/examples/kuzu/README.md000066400000000000000000000043311444262304300166440ustar00rootroot00000000000000# Using KΓΉzu as a Remote Backend for PyG

[KΓΉzu](https://kuzudb.com/) is an in-process property graph database management system built for query speed and scalability.
It provides an integration with PyG via the [remote backend interface](https://pytorch-geometric.readthedocs.io/en/latest/advanced/remote.html) of PyG.
The Python API of KΓΉzu outputs a [`torch_geometric.data.FeatureStore`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.data.FeatureStore.html) and a [`torch_geometric.data.GraphStore`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.data.GraphStore.html) that can be plugged directly into existing familiar PyG interfaces such as [`NeighborLoader`](https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/loader/neighbor_loader.html) and enables training GNNs directly on graphs stored in KΓΉzu.
This is particularly useful if you would like to train graphs that don't fit on your CPU's memory.

## Installation

You can install KΓΉzu as follows:

```bash
pip install kuzu
```

## Usage

The API and design documentation of KΓΉzu can be found at [https://kuzudb.com/docs/](https://kuzudb.com/docs/).

## Examples

We provide the following examples to showcase the usage of KΓΉzu remote backend within PyG:

### PubMed


  Open In Colab


The PubMed example is hosted on [Google Colab](https://colab.research.google.com/drive/12fOSqPm1HQTz_m9caRW7E_92vaeD9xq6).
In this example, we work on a small dataset for demonstrative purposes.
The [PubMed](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Planetoid.html) dataset consists of 19,717 papers as nodes and 88,648 citation relationships between them.

### `papers_100M`

This example shows how to use the remote backend feature of KΓΉzu to work with a large graph of papers and citations on a single machine.
The data used in this example is `ogbn-papers100M` from the [Open Graph Benchmark](https://ogb.stanford.edu/).
The dataset contains approximately 111 million nodes and 1.6 billion edges.
pytorch/examples/kuzu/papers_100M/000077500000000000000000000000001444262304300173535ustar00rootroot00000000000000pytorch/examples/kuzu/papers_100M/README.md000066400000000000000000000020401444262304300206260ustar00rootroot00000000000000# `papers_100M` Example

This example shows how to use the remote backend feature of [KΓΉzu](https://kuzudb.com) to work with a large graph of papers and citations on a single machine.
The data used in this example is `ogbn-papers100M` from the [Open Graph Benchmark](https://ogb.stanford.edu/).
The dataset contains approximately 100 million nodes and 1.6 billion edges.

## Prepare the data

1. Download the dataset from [`http://snap.stanford.edu/ogb/data/nodeproppred/papers100M-bin.zip`](http://snap.stanford.edu/ogb/data/nodeproppred/papers100M-bin.zip) and put the `*.zip` file into this directory.
2. Run `python prepare_data.py`.
   The script will automatically extract the data and convert it to the format that KΓΉzu can read.
   A KΓΉzu database instance is then created under `papers_100M` and the data is loaded into the it.

## Train a Model

Afterwards, run `python train.py` to train a three-layer [`GraphSAGE`](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GraphSAGE.html) model on this dataset.
pytorch/examples/kuzu/papers_100M/prepare_data.py000066400000000000000000000036741444262304300223660ustar00rootroot00000000000000from multiprocessing import cpu_count
from os import path
from zipfile import ZipFile

import kuzu
import numpy as np
from tqdm import tqdm

with ZipFile("papers100M-bin.zip", 'r') as papers100M_zip:
    print('Extracting papers100M-bin.zip...')
    papers100M_zip.extractall()

with ZipFile("papers100M-bin/raw/data.npz", 'r') as data_zip:
    print('Extracting data.npz...')
    data_zip.extractall()

with ZipFile("papers100M-bin/raw/node-label.npz", 'r') as node_label_zip:
    print('Extracting node-label.npz...')
    node_label_zip.extractall()

print("Converting edge_index to CSV...")
edge_index = np.load('edge_index.npy', mmap_mode='r')
csvfile = open('edge_index.csv', 'w')
csvfile.write('src,dst\n')
for i in tqdm(range(edge_index.shape[1])):
    csvfile.write(str(edge_index[0, i]) + ',' + str(edge_index[1, i]) + '\n')
csvfile.close()

print("Generating IDs for nodes...")
node_year = np.load('node_year.npy', mmap_mode='r')
length = node_year.shape[0]
ids = np.arange(length)
np.save('ids.npy', ids)

ids_path = path.abspath(path.join('.', 'ids.npy'))
edge_index_path = path.abspath(path.join('.', 'edge_index.csv'))
node_label_path = path.abspath(path.join('.', 'node_label.npy'))
node_feature_path = path.abspath(path.join('.', 'node_feat.npy'))
node_year_path = path.abspath(path.join('.', 'node_year.npy'))

print("Creating KΓΉzu database...")
db = kuzu.Database('papers100M')
conn = kuzu.Connection(db, num_threads=cpu_count())
print("Creating KΓΉzu tables...")
conn.execute(
    "CREATE NODE TABLE paper(id INT64, x FLOAT[128], year INT64, y FLOAT, "
    "PRIMARY KEY (id));")
conn.execute("CREATE REL TABLE cites(FROM paper TO paper, MANY_MANY);")
print("Copying nodes to KΓΉzu tables...")
conn.execute('COPY paper FROM ("%s",  "%s",  "%s", "%s") BY COLUMN;' %
             (ids_path, node_feature_path, node_year_path, node_label_path))
print("Copying edges to KΓΉzu tables...")
conn.execute('COPY cites FROM "%s";' % (edge_index_path))
print("All done!")
pytorch/examples/kuzu/papers_100M/train.py000066400000000000000000000075671444262304300210610ustar00rootroot00000000000000import multiprocessing as mp
import os.path as osp

import kuzu
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from tqdm import tqdm

from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import MLP, BatchNorm, SAGEConv

NUM_EPOCHS = 1
LOADER_BATCH_SIZE = 1024

print('Batch size:', LOADER_BATCH_SIZE)
print('Number of epochs:', NUM_EPOCHS)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)

# Load the train set:
train_path = osp.join('.', 'papers100M-bin', 'split', 'time', 'train.csv.gz')
train_df = pd.read_csv(
    osp.abspath(train_path),
    compression='gzip',
    header=None,
)
input_nodes = torch.tensor(train_df[0].values, dtype=torch.long)

########################################################################
# The below code sets up the remote backend of KΓΉzu for PyG.
# Please refer to: https://kuzudb.com/docs/client-apis/python-api/overview.html
# for how to use the Python API of KΓΉzu.
########################################################################

# The buffer pool size of KΓΉzu is set to 40GB. You can change it to a smaller
# value if you have less memory.
KUZU_BM_SIZE = 40 * 1024**3

# Create KΓΉzu database:
db = kuzu.Database(osp.abspath(osp.join('.', 'papers100M')), KUZU_BM_SIZE)

# Get remote backend for PyG:
feature_store, graph_store = db.get_torch_geometric_remote_backend(
    mp.cpu_count())

# Plug the graph store and feature store into the `NeighborLoader`.
# Note that `filter_per_worker` is set to `False`. This is because the KΓΉzu
# database is already using multi-threading to scan the features in parallel
# and the database object is not fork-safe.
loader = NeighborLoader(
    data=(feature_store, graph_store),
    num_neighbors={('paper', 'cites', 'paper'): [12, 12, 12]},
    batch_size=LOADER_BATCH_SIZE,
    input_nodes=('paper', input_nodes),
    num_workers=4,
    filter_per_worker=False,
)


class GraphSAGE(nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers,
                 dropout=0.2):
        super().__init__()

        self.convs = nn.ModuleList()
        self.norms = nn.ModuleList()

        self.convs.append(SAGEConv(in_channels, hidden_channels))
        self.bns.append(BatchNorm(hidden_channels))
        for i in range(1, num_layers):
            self.layers.append(SAGEConv(hidden_channels, hidden_channels))
            self.bns.append(BatchNorm(hidden_channels))

        self.mlp = MLP(
            in_channels=in_channels + num_layers * hidden_channels,
            hidden_channels=2 * out_channels,
            out_channels=out_channels,
            num_layers=2,
            norm='batch_norm',
            act='leaky_relu',
        )

    def forward(self, x, edge_index):
        x = F.dropout(x, p=self.dropout, training=self.training)
        xs = [x]
        for conv, norm in zip(self.convs, self.norms):
            x = conv(x, edge_index)
            x = norm(x)
            x = x.relu()
            x = F.dropout(x, p=self.dropout, training=self.training)
            xs.append(x)
        return self.mlp(torch.cat(xs, dim=-1))


model = GraphSAGE(in_channels=128, hidden_channels=1024, out_channels=172,
                  num_layers=3).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

for epoch in range(1, NUM_EPOCHS + 1):
    total_loss = total_examples = 0
    for batch in tqdm(loader):
        batch = batch.to(device)
        batch_size = batch['paper'].batch_size

        optimizer.zero_grad()
        out = model(batch.x, batch.edge_index)[:batch_size]
        y = batch.y[:batch_size].long().view(-1)
        loss = F.cross_entropy_loss(out, y)

        loss.backward()
        optimizer.step()

        total_loss += float(loss) * y.numel()
        total_examples += y.numel()

    print(f'Epoch: {epoch:02d}, Loss: {total_loss / total_examples:.4f}')
pytorch/examples/label_prop.py000066400000000000000000000016431444262304300170630ustar00rootroot00000000000000import os.path as osp

from ogb.nodeproppred import Evaluator, PygNodePropPredDataset

import torch_geometric.transforms as T
from torch_geometric.nn import LabelPropagation

root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'OGB')
dataset = PygNodePropPredDataset(
    'ogbn-arxiv', root, transform=T.Compose([
        T.ToUndirected(),
        T.ToSparseTensor(),
    ]))
split_idx = dataset.get_idx_split()
evaluator = Evaluator(name='ogbn-arxiv')
data = dataset[0]

model = LabelPropagation(num_layers=3, alpha=0.9)
out = model(data.y, data.adj_t, mask=split_idx['train'])

y_pred = out.argmax(dim=-1, keepdim=True)

val_acc = evaluator.eval({
    'y_true': data.y[split_idx['valid']],
    'y_pred': y_pred[split_idx['valid']],
})['acc']
test_acc = evaluator.eval({
    'y_true': data.y[split_idx['test']],
    'y_pred': y_pred[split_idx['test']],
})['acc']

print(f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/link_pred.py000066400000000000000000000062011444262304300167060ustar00rootroot00000000000000import os.path as osp

import torch
from sklearn.metrics import roc_auc_score

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv
from torch_geometric.utils import negative_sampling

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.NormalizeFeatures(),
    T.ToDevice(device),
    T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,
                      add_negative_train_samples=False),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=transform)
# After applying the `RandomLinkSplit` transform, the data is transformed from
# a data object to a list of tuples (train_data, val_data, test_data), with
# each element representing the corresponding split.
train_data, val_data, test_data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)

    def encode(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

    def decode(self, z, edge_label_index):
        return (z[edge_label_index[0]] * z[edge_label_index[1]]).sum(dim=-1)

    def decode_all(self, z):
        prob_adj = z @ z.t()
        return (prob_adj > 0).nonzero(as_tuple=False).t()


model = Net(dataset.num_features, 128, 64).to(device)
optimizer = torch.optim.Adam(params=model.parameters(), lr=0.01)
criterion = torch.nn.BCEWithLogitsLoss()


def train():
    model.train()
    optimizer.zero_grad()
    z = model.encode(train_data.x, train_data.edge_index)

    # We perform a new round of negative sampling for every training epoch:
    neg_edge_index = negative_sampling(
        edge_index=train_data.edge_index, num_nodes=train_data.num_nodes,
        num_neg_samples=train_data.edge_label_index.size(1), method='sparse')

    edge_label_index = torch.cat(
        [train_data.edge_label_index, neg_edge_index],
        dim=-1,
    )
    edge_label = torch.cat([
        train_data.edge_label,
        train_data.edge_label.new_zeros(neg_edge_index.size(1))
    ], dim=0)

    out = model.decode(z, edge_label_index).view(-1)
    loss = criterion(out, edge_label)
    loss.backward()
    optimizer.step()
    return loss


@torch.no_grad()
def test(data):
    model.eval()
    z = model.encode(data.x, data.edge_index)
    out = model.decode(z, data.edge_label_index).view(-1).sigmoid()
    return roc_auc_score(data.edge_label.cpu().numpy(), out.cpu().numpy())


best_val_auc = final_test_auc = 0
for epoch in range(1, 101):
    loss = train()
    val_auc = test(val_data)
    test_auc = test(test_data)
    if val_auc > best_val_auc:
        best_val_auc = val_auc
        final_test_auc = test_auc
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val: {val_auc:.4f}, '
          f'Test: {test_auc:.4f}')

print(f'Final Test: {final_test_auc:.4f}')

z = model.encode(test_data.x, test_data.edge_index)
final_edge_index = model.decode_all(z)
pytorch/examples/linkx.py000066400000000000000000000027551444262304300160760ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import LINKXDataset
from torch_geometric.nn import LINKX

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'LINKX')
dataset = LINKXDataset(path, name='Penn94')
data = dataset[0].to(device)

model = LINKX(data.num_nodes, data.num_features, hidden_channels=32,
              out_channels=dataset.num_classes, num_layers=1,
              num_edge_layers=1, num_node_layers=1, dropout=0.5).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=1e-3)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    mask = data.train_mask[:, 0]  # Use the first set of the five masks.
    loss = F.cross_entropy(out[mask], data.y[mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    accs = []
    model.eval()
    pred = model(data.x, data.edge_index).argmax(dim=-1)
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        mask = mask[:, 0]  # Use the first set of the five masks.
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


for epoch in range(1, 201):
    loss = train()
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/mem_pool.py000066400000000000000000000070511444262304300165520ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import BatchNorm1d, LeakyReLU, Linear

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import DeepGCNLayer, GATConv, MemPooling

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'TUD')
dataset = TUDataset(path, name="PROTEINS_full", use_node_attr=True)
dataset.data.x = dataset.data.x[:, :-3]  # only use non-binary features.
dataset = dataset.shuffle()

n = (len(dataset)) // 10
test_dataset = dataset[:n]
val_dataset = dataset[n:2 * n]
train_dataset = dataset[2 * n:]

test_loader = DataLoader(test_dataset, batch_size=20)
val_loader = DataLoader(val_dataset, batch_size=20)
train_loader = DataLoader(train_dataset, batch_size=20)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, dropout):
        super().__init__()
        self.dropout = dropout

        self.lin = Linear(in_channels, hidden_channels)

        self.convs = torch.nn.ModuleList()
        for i in range(2):
            conv = GATConv(hidden_channels, hidden_channels, dropout=dropout)
            norm = BatchNorm1d(hidden_channels)
            act = LeakyReLU()
            self.convs.append(
                DeepGCNLayer(conv, norm, act, block='res+', dropout=dropout))

        self.mem1 = MemPooling(hidden_channels, 80, heads=5, num_clusters=10)
        self.mem2 = MemPooling(80, out_channels, heads=5, num_clusters=1)

    def forward(self, x, edge_index, batch):
        x = self.lin(x)
        for conv in self.convs:
            x = conv(x, edge_index)

        x, S1 = self.mem1(x, batch)
        x = F.leaky_relu(x)
        x = F.dropout(x, p=self.dropout)
        x, S2 = self.mem2(x)

        return (
            F.log_softmax(x.squeeze(1), dim=-1),
            MemPooling.kl_loss(S1) + MemPooling.kl_loss(S2),
        )


device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = Net(dataset.num_features, 32, dataset.num_classes, dropout=0.1)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=4e-5)


def train():
    model.train()

    model.mem1.k.requires_grad = False
    model.mem2.k.requires_grad = False
    for data in train_loader:
        optimizer.zero_grad()
        data = data.to(device)
        out = model(data.x, data.edge_index, data.batch)[0]
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()

    kl_loss = 0.
    model.mem1.k.requires_grad = True
    model.mem2.k.requires_grad = True
    optimizer.zero_grad()
    for data in train_loader:
        data = data.to(device)
        kl_loss += model(data.x, data.edge_index, data.batch)[1]
    kl_loss /= len(train_loader.dataset)
    kl_loss.backward()
    optimizer.step()


@torch.no_grad()
def test(loader):
    model.eval()
    total_correct = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.batch)[0]
        total_correct += int((out.argmax(dim=-1) == data.y).sum())
    return total_correct / len(loader.dataset)


patience = start_patience = 250
test_acc = best_val_acc = 0.
for epoch in range(1, 2001):
    train()
    val_acc = test(val_loader)
    if epoch % 500 == 0:
        optimizer.param_groups[0]['lr'] *= 0.5
    if best_val_acc < val_acc:
        patience = start_patience
        best_val_acc = val_acc
        test_acc = test(test_loader)
    else:
        patience -= 1
    print(f'Epoch {epoch:02d}, Val: {val_acc:.3f}, Test: {test_acc:.3f}')
    if patience <= 0:
        break
pytorch/examples/mnist_graclus.py000066400000000000000000000062141444262304300176150ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import MNISTSuperpixels
from torch_geometric.loader import DataLoader
from torch_geometric.nn import (
    SplineConv,
    global_mean_pool,
    graclus,
    max_pool,
    max_pool_x,
)
from torch_geometric.typing import WITH_TORCH_CLUSTER, WITH_TORCH_SPLINE_CONV
from torch_geometric.utils import normalized_cut

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")
if not WITH_TORCH_SPLINE_CONV:
    quit("This example requires 'torch-spline-conv'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')
transform = T.Cartesian(cat=False)
train_dataset = MNISTSuperpixels(path, True, transform=transform)
test_dataset = MNISTSuperpixels(path, False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64)
d = train_dataset


def normalized_cut_2d(edge_index, pos):
    row, col = edge_index
    edge_attr = torch.norm(pos[row] - pos[col], p=2, dim=1)
    return normalized_cut(edge_index, edge_attr, num_nodes=pos.size(0))


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SplineConv(d.num_features, 32, dim=2, kernel_size=5)
        self.conv2 = SplineConv(32, 64, dim=2, kernel_size=5)
        self.fc1 = torch.nn.Linear(64, 128)
        self.fc2 = torch.nn.Linear(128, d.num_classes)

    def forward(self, data):
        data.x = F.elu(self.conv1(data.x, data.edge_index, data.edge_attr))
        weight = normalized_cut_2d(data.edge_index, data.pos)
        cluster = graclus(data.edge_index, weight, data.x.size(0))
        data.edge_attr = None
        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv2(data.x, data.edge_index, data.edge_attr))
        weight = normalized_cut_2d(data.edge_index, data.pos)
        cluster = graclus(data.edge_index, weight, data.x.size(0))
        x, batch = max_pool_x(cluster, data.x, data.batch)

        x = global_mean_pool(x, batch)
        x = F.elu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        return F.log_softmax(self.fc2(x), dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(epoch):
    model.train()

    if epoch == 16:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.001

    if epoch == 26:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.0001

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        F.nll_loss(model(data), data.y).backward()
        optimizer.step()


def test():
    model.eval()
    correct = 0

    for data in test_loader:
        data = data.to(device)
        pred = model(data).max(1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(test_dataset)


for epoch in range(1, 31):
    train(epoch)
    test_acc = test()
    print(f'Epoch: {epoch:02d}, Test: {test_acc:.4f}')
pytorch/examples/mnist_nn_conv.py000066400000000000000000000064221444262304300176160ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import MNISTSuperpixels
from torch_geometric.loader import DataLoader
from torch_geometric.nn import (
    NNConv,
    global_mean_pool,
    graclus,
    max_pool,
    max_pool_x,
)
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import normalized_cut

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')
transform = T.Cartesian(cat=False)
train_dataset = MNISTSuperpixels(path, True, transform=transform)
test_dataset = MNISTSuperpixels(path, False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
d = train_dataset


def normalized_cut_2d(edge_index, pos):
    row, col = edge_index
    edge_attr = torch.norm(pos[row] - pos[col], p=2, dim=1)
    return normalized_cut(edge_index, edge_attr, num_nodes=pos.size(0))


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        nn1 = nn.Sequential(nn.Linear(2, 25), nn.ReLU(),
                            nn.Linear(25, d.num_features * 32))
        self.conv1 = NNConv(d.num_features, 32, nn1, aggr='mean')

        nn2 = nn.Sequential(nn.Linear(2, 25), nn.ReLU(),
                            nn.Linear(25, 32 * 64))
        self.conv2 = NNConv(32, 64, nn2, aggr='mean')

        self.fc1 = torch.nn.Linear(64, 128)
        self.fc2 = torch.nn.Linear(128, d.num_classes)

    def forward(self, data):
        data.x = F.elu(self.conv1(data.x, data.edge_index, data.edge_attr))
        weight = normalized_cut_2d(data.edge_index, data.pos)
        cluster = graclus(data.edge_index, weight, data.x.size(0))
        data.edge_attr = None
        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv2(data.x, data.edge_index, data.edge_attr))
        weight = normalized_cut_2d(data.edge_index, data.pos)
        cluster = graclus(data.edge_index, weight, data.x.size(0))
        x, batch = max_pool_x(cluster, data.x, data.batch)

        x = global_mean_pool(x, batch)
        x = F.elu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        return F.log_softmax(self.fc2(x), dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(epoch):
    model.train()

    if epoch == 16:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.001

    if epoch == 26:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.0001

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        F.nll_loss(model(data), data.y).backward()
        optimizer.step()


def test():
    model.eval()
    correct = 0

    for data in test_loader:
        data = data.to(device)
        pred = model(data).max(1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(test_dataset)


for epoch in range(1, 31):
    train(epoch)
    test_acc = test()
    print(f'Epoch: {epoch:02d}, Test: {test_acc:.4f}')
pytorch/examples/mnist_voxel_grid.py000066400000000000000000000061341444262304300203200ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import MNISTSuperpixels
from torch_geometric.loader import DataLoader
from torch_geometric.nn import SplineConv, max_pool, max_pool_x, voxel_grid
from torch_geometric.typing import WITH_TORCH_SPLINE_CONV

if not WITH_TORCH_SPLINE_CONV:
    quit("This example requires 'torch-spline-conv'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')
transform = T.Cartesian(cat=False)
train_dataset = MNISTSuperpixels(path, True, transform=transform)
test_dataset = MNISTSuperpixels(path, False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64)
d = train_dataset


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SplineConv(d.num_features, 32, dim=2, kernel_size=5)
        self.conv2 = SplineConv(32, 64, dim=2, kernel_size=5)
        self.conv3 = SplineConv(64, 64, dim=2, kernel_size=5)
        self.fc1 = torch.nn.Linear(4 * 64, 128)
        self.fc2 = torch.nn.Linear(128, d.num_classes)

    def forward(self, data):
        data.x = F.elu(self.conv1(data.x, data.edge_index, data.edge_attr))
        cluster = voxel_grid(data.pos, batch=data.batch, size=5, start=0,
                             end=28)
        data.edge_attr = None
        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv2(data.x, data.edge_index, data.edge_attr))
        cluster = voxel_grid(data.pos, batch=data.batch, size=7, start=0,
                             end=28)
        data.edge_attr = None
        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv3(data.x, data.edge_index, data.edge_attr))
        cluster = voxel_grid(data.pos, batch=data.batch, size=14, start=0,
                             end=27.99)
        x, _ = max_pool_x(cluster, data.x, data.batch, size=4)

        x = x.view(-1, self.fc1.weight.size(1))
        x = F.elu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(epoch):
    model.train()

    if epoch == 6:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.001

    if epoch == 16:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.0001

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        F.nll_loss(model(data), data.y).backward()
        optimizer.step()


def test():
    model.eval()
    correct = 0

    for data in test_loader:
        data = data.to(device)
        pred = model(data).max(1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(test_dataset)


for epoch in range(1, 21):
    train(epoch)
    test_acc = test()
    print(f'Epoch: {epoch:02d}, Test: {test_acc:.4f}')
pytorch/examples/multi_gpu/000077500000000000000000000000001444262304300163735ustar00rootroot00000000000000pytorch/examples/multi_gpu/data_parallel.py000066400000000000000000000045021444262304300215330ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear, ReLU, Sequential

import torch_geometric.transforms as T
from torch_geometric.datasets import MNISTSuperpixels
from torch_geometric.loader import DataListLoader
from torch_geometric.nn import (
    DataParallel,
    NNConv,
    SplineConv,
    global_mean_pool,
)
from torch_geometric.typing import WITH_TORCH_SPLINE_CONV

path = osp.join(osp.dirname(osp.realpath(__file__)), '../../data', 'MNIST')
dataset = MNISTSuperpixels(path, transform=T.Cartesian()).shuffle()
loader = DataListLoader(dataset, batch_size=1024, shuffle=True)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        if WITH_TORCH_SPLINE_CONV:
            self.conv1 = SplineConv(dataset.num_features, 32, dim=2,
                                    kernel_size=5)
            self.conv2 = SplineConv(32, 64, dim=2, kernel_size=5)
        else:
            nn1 = Sequential(Linear(2, 25), ReLU(),
                             Linear(25, dataset.num_features * 32))
            self.conv1 = NNConv(dataset.num_features, 32, nn1, aggr='mean')

            nn2 = Sequential(Linear(2, 25), ReLU(), Linear(25, 32 * 64))
            self.conv2 = NNConv(32, 64, nn2, aggr='mean')

        self.lin1 = torch.nn.Linear(64, 128)
        self.lin2 = torch.nn.Linear(128, dataset.num_classes)

    def forward(self, data):
        print(f'Inside model - num graphs: {data.num_graphs}, '
              f'device: {data.batch.device}')

        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        x = F.elu(self.conv1(x, edge_index, edge_attr))
        x = F.elu(self.conv2(x, edge_index, edge_attr))
        x = global_mean_pool(x, data.batch)
        x = F.elu(self.lin1(x))
        return F.log_softmax(self.lin2(x), dim=1)


model = Net()
print(f"Let's use {torch.cuda.device_count()} GPUs!")
model = DataParallel(model)
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for data_list in loader:
    optimizer.zero_grad()
    output = model(data_list)
    print(f'Outside model - num graphs: {output.size(0)}')
    y = torch.cat([data.y for data in data_list]).to(output.device)
    loss = F.nll_loss(output, y)
    loss.backward()
    optimizer.step()
pytorch/examples/multi_gpu/distributed_batching.py000066400000000000000000000123111444262304300231240ustar00rootroot00000000000000import os

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn.functional as F
from ogb.graphproppred import Evaluator
from ogb.graphproppred import PygGraphPropPredDataset as Dataset
from ogb.graphproppred.mol_encoder import AtomEncoder, BondEncoder
from torch.nn import BatchNorm1d as BatchNorm
from torch.nn import Linear, ReLU, Sequential
from torch.nn.parallel import DistributedDataParallel
from torch.utils.data.distributed import DistributedSampler

import torch_geometric.transforms as T
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GINEConv, global_mean_pool
from torch_geometric.typing import WITH_TORCH_SPARSE

if not WITH_TORCH_SPARSE:
    quit("This example requires 'torch-sparse'")


class GIN(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels, num_layers=3,
                 dropout=0.5):
        super().__init__()

        self.dropout = dropout

        self.atom_encoder = AtomEncoder(hidden_channels)
        self.bond_encoder = BondEncoder(hidden_channels)

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            nn = Sequential(
                Linear(hidden_channels, 2 * hidden_channels),
                BatchNorm(2 * hidden_channels),
                ReLU(),
                Linear(2 * hidden_channels, hidden_channels),
                BatchNorm(hidden_channels),
                ReLU(),
            )
            self.convs.append(GINEConv(nn, train_eps=True))

        self.lin = Linear(hidden_channels, out_channels)

    def forward(self, x, adj_t, batch):
        x = self.atom_encoder(x)
        edge_attr = adj_t.coo()[2]
        adj_t = adj_t.set_value(self.bond_encoder(edge_attr), layout='coo')

        for conv in self.convs:
            x = conv(x, adj_t)
            x = F.dropout(x, p=self.dropout, training=self.training)

        x = global_mean_pool(x, batch)
        x = self.lin(x)
        return x


def run(rank, world_size: int, dataset_name: str, root: str):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group('nccl', rank=rank, world_size=world_size)

    dataset = Dataset(dataset_name, root,
                      pre_transform=T.ToSparseTensor(attr='edge_attr'))
    split_idx = dataset.get_idx_split()
    evaluator = Evaluator(dataset_name)

    train_dataset = dataset[split_idx['train']]
    train_sampler = DistributedSampler(train_dataset, num_replicas=world_size,
                                       rank=rank)
    train_loader = DataLoader(train_dataset, batch_size=128,
                              sampler=train_sampler)

    torch.manual_seed(12345)
    model = GIN(128, dataset.num_tasks, num_layers=3, dropout=0.5).to(rank)
    model = DistributedDataParallel(model, device_ids=[rank])
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    criterion = torch.nn.BCEWithLogitsLoss()

    if rank == 0:
        val_loader = DataLoader(dataset[split_idx['valid']], batch_size=256)
        test_loader = DataLoader(dataset[split_idx['test']], batch_size=256)

    for epoch in range(1, 51):
        model.train()

        total_loss = torch.zeros(2).to(rank)
        for data in train_loader:
            data = data.to(rank)
            optimizer.zero_grad()
            logits = model(data.x, data.adj_t, data.batch)
            loss = criterion(logits, data.y.to(torch.float))
            loss.backward()
            optimizer.step()
            total_loss[0] += float(loss) * logits.size(0)
            total_loss[1] += data.num_graphs

        dist.all_reduce(total_loss, op=dist.ReduceOp.SUM)
        loss = float(total_loss[0] / total_loss[1])

        if rank == 0:  # We evaluate on a single GPU for now.
            model.eval()

            y_pred, y_true = [], []
            for data in val_loader:
                data = data.to(rank)
                with torch.no_grad():
                    y_pred.append(model.module(data.x, data.adj_t, data.batch))
                    y_true.append(data.y)
            val_rocauc = evaluator.eval({
                'y_pred': torch.cat(y_pred, dim=0),
                'y_true': torch.cat(y_true, dim=0),
            })['rocauc']

            y_pred, y_true = [], []
            for data in test_loader:
                data = data.to(rank)
                with torch.no_grad():
                    y_pred.append(model.module(data.x, data.adj_t, data.batch))
                    y_true.append(data.y)
            test_rocauc = evaluator.eval({
                'y_pred': torch.cat(y_pred, dim=0),
                'y_true': torch.cat(y_true, dim=0),
            })['rocauc']

            print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
                  f'Val: {val_rocauc:.4f}, Test: {test_rocauc:.4f}')

        dist.barrier()

    dist.destroy_process_group()


if __name__ == '__main__':
    dataset_name = 'ogbg-molhiv'
    root = '../../data/OGB'

    # Download and process the dataset on main process.
    Dataset(dataset_name, root,
            pre_transform=T.ToSparseTensor(attr='edge_attr'))

    world_size = torch.cuda.device_count()
    print('Let\'s use', world_size, 'GPUs!')
    args = (world_size, dataset_name, root)
    mp.spawn(run, args=args, nprocs=world_size, join=True)
pytorch/examples/multi_gpu/distributed_sampling.py000066400000000000000000000114071444262304300231640ustar00rootroot00000000000000import copy
import os

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn.functional as F
from torch import Tensor
from torch.nn.parallel import DistributedDataParallel
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import SAGEConv


class SAGE(torch.nn.Module):
    def __init__(self, in_channels: int, hidden_channels: int,
                 out_channels: int, num_layers: int = 2):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        for _ in range(num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for i, conv in enumerate(self.convs):
            x = conv(x, edge_index)
            if i < len(self.convs) - 1:
                x = x.relu_()
                x = F.dropout(x, p=0.5, training=self.training)
        return x

    @torch.no_grad()
    def inference(self, x_all: Tensor, device: torch.device,
                  subgraph_loader: NeighborLoader) -> Tensor:

        pbar = tqdm(total=len(subgraph_loader) * len(self.convs))
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch:
        for i, conv in enumerate(self.convs):
            xs = []
            for batch in subgraph_loader:
                x = x_all[batch.node_id.to(x_all.device)].to(device)
                x = conv(x, batch.edge_index.to(device))
                x = x[:batch.batch_size]
                if i < len(self.convs) - 1:
                    x = x.relu_()
                xs.append(x.cpu())
                pbar.update(1)
            x_all = torch.cat(xs, dim=0)

        pbar.close()
        return x_all


def run(rank, world_size, dataset):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group('nccl', rank=rank, world_size=world_size)

    data = dataset[0]
    data = data.to(rank, 'x', 'y')  # Move to device for faster feature fetch.

    # Split training indices into `world_size` many chunks:
    train_idx = data.train_mask.nonzero(as_tuple=False).view(-1)
    train_idx = train_idx.split(train_idx.size(0) // world_size)[rank]

    kwargs = dict(batch_size=1024, num_workers=4, persistent_workers=True)
    train_loader = NeighborLoader(data, input_nodes=train_idx,
                                  num_neighbors=[25, 10], shuffle=True,
                                  drop_last=True, **kwargs)

    if rank == 0:  # Create single-hop evaluation neighbor loader:
        subgraph_loader = NeighborLoader(copy.copy(data), num_neighbors=[-1],
                                         shuffle=False, **kwargs)
        # No need to maintain these features during evaluation:
        del subgraph_loader.data.x, subgraph_loader.data.y
        # Add global node index information:
        subgraph_loader.data.node_id = torch.arange(data.num_nodes)

    torch.manual_seed(12345)
    model = SAGE(dataset.num_features, 256, dataset.num_classes).to(rank)
    model = DistributedDataParallel(model, device_ids=[rank])
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

    for epoch in range(1, 21):
        model.train()
        for batch in train_loader:
            optimizer.zero_grad()
            out = model(batch.x, batch.edge_index.to(rank))[:batch.batch_size]
            loss = F.cross_entropy(out, batch.y[:batch.batch_size])
            loss.backward()
            optimizer.step()

        dist.barrier()

        if rank == 0:
            print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}')

        if rank == 0 and epoch % 5 == 0:  # We evaluate on a single GPU for now
            model.eval()
            with torch.no_grad():
                out = model.module.inference(data.x, rank, subgraph_loader)
            res = out.argmax(dim=-1) == data.y.to(out.device)
            acc1 = int(res[data.train_mask].sum()) / int(data.train_mask.sum())
            acc2 = int(res[data.val_mask].sum()) / int(data.val_mask.sum())
            acc3 = int(res[data.test_mask].sum()) / int(data.test_mask.sum())
            print(f'Train: {acc1:.4f}, Val: {acc2:.4f}, Test: {acc3:.4f}')

        dist.barrier()

    dist.destroy_process_group()


if __name__ == '__main__':
    dataset = Reddit('../../data/Reddit')

    world_size = torch.cuda.device_count()
    print('Let\'s use', world_size, 'GPUs!')
    mp.spawn(run, args=(world_size, dataset), nprocs=world_size, join=True)
pytorch/examples/mutag_gin.py000066400000000000000000000061471444262304300167220ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.logging import init_wandb, log
from torch_geometric.nn import MLP, GINConv, global_add_pool

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='MUTAG')
parser.add_argument('--batch_size', type=int, default=128)
parser.add_argument('--hidden_channels', type=int, default=32)
parser.add_argument('--num_layers', type=int, default=5)
parser.add_argument('--lr', type=float, default=0.01)
parser.add_argument('--epochs', type=int, default=100)
parser.add_argument('--wandb', action='store_true', help='Track experiment')
args = parser.parse_args()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
init_wandb(name=f'GIN-{args.dataset}', batch_size=args.batch_size, lr=args.lr,
           epochs=args.epochs, hidden_channels=args.hidden_channels,
           num_layers=args.num_layers, device=device)

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'TU')
dataset = TUDataset(path, name=args.dataset).shuffle()

train_dataset = dataset[len(dataset) // 10:]
train_loader = DataLoader(train_dataset, args.batch_size, shuffle=True)

test_dataset = dataset[:len(dataset) // 10]
test_loader = DataLoader(test_dataset, args.batch_size)


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers):
        super().__init__()

        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            mlp = MLP([in_channels, hidden_channels, hidden_channels])
            self.convs.append(GINConv(nn=mlp, train_eps=False))
            in_channels = hidden_channels

        self.mlp = MLP([hidden_channels, hidden_channels, out_channels],
                       norm=None, dropout=0.5)

    def forward(self, x, edge_index, batch):
        for conv in self.convs:
            x = conv(x, edge_index).relu()
        x = global_add_pool(x, batch)
        return self.mlp(x)


model = Net(dataset.num_features, args.hidden_channels, dataset.num_classes,
            args.num_layers).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = F.cross_entropy(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = 0
    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.edge_index, data.batch).argmax(dim=-1)
        total_correct += int((pred == data.y).sum())
    return total_correct / len(loader.dataset)


for epoch in range(1, args.epochs + 1):
    loss = train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    log(Epoch=epoch, Loss=loss, Train=train_acc, Test=test_acc)
pytorch/examples/node2vec.py000066400000000000000000000043711444262304300164520ustar00rootroot00000000000000import os.path as osp
import sys

import matplotlib.pyplot as plt
import torch
from sklearn.manifold import TSNE

from torch_geometric.datasets import Planetoid
from torch_geometric.nn import Node2Vec


def main():
    dataset = 'Cora'
    path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
    dataset = Planetoid(path, dataset)
    data = dataset[0]

    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    model = Node2Vec(
        data.edge_index,
        embedding_dim=128,
        walk_length=20,
        context_size=10,
        walks_per_node=10,
        num_negative_samples=1,
        p=1,
        q=1,
        sparse=True,
    ).to(device)

    num_workers = 0 if sys.platform.startswith('win') else 4
    loader = model.loader(batch_size=128, shuffle=True,
                          num_workers=num_workers)
    optimizer = torch.optim.SparseAdam(list(model.parameters()), lr=0.01)

    def train():
        model.train()
        total_loss = 0
        for pos_rw, neg_rw in loader:
            optimizer.zero_grad()
            loss = model.loss(pos_rw.to(device), neg_rw.to(device))
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        return total_loss / len(loader)

    @torch.no_grad()
    def test():
        model.eval()
        z = model()
        acc = model.test(z[data.train_mask], data.y[data.train_mask],
                         z[data.test_mask], data.y[data.test_mask],
                         max_iter=150)
        return acc

    for epoch in range(1, 101):
        loss = train()
        acc = test()
        print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Acc: {acc:.4f}')

    @torch.no_grad()
    def plot_points(colors):
        model.eval()
        z = model(torch.arange(data.num_nodes, device=device))
        z = TSNE(n_components=2).fit_transform(z.cpu().numpy())
        y = data.y.cpu().numpy()

        plt.figure(figsize=(8, 8))
        for i in range(dataset.num_classes):
            plt.scatter(z[y == i, 0], z[y == i, 1], s=20, color=colors[i])
        plt.axis('off')
        plt.show()

    colors = [
        '#ffc0cb', '#bada55', '#008080', '#420420', '#7fe5f0', '#065535',
        '#ffd700'
    ]
    plot_points(colors)


if __name__ == "__main__":
    main()
pytorch/examples/ogbn_products_gat.py000066400000000000000000000130551444262304300204470ustar00rootroot00000000000000# Reaches around 0.7945 Β± 0.0059 test accuracy.

import os.path as osp

import torch
import torch.nn.functional as F
from ogb.nodeproppred import Evaluator, PygNodePropPredDataset
from torch.nn import Linear as Lin
from tqdm import tqdm

from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import GATConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'products')
dataset = PygNodePropPredDataset('ogbn-products', root)
split_idx = dataset.get_idx_split()
evaluator = Evaluator(name='ogbn-products')
data = dataset[0].to(device, 'x', 'y')

train_loader = NeighborLoader(
    data,
    input_nodes=split_idx['train'],
    num_neighbors=[10, 10, 5],
    batch_size=512,
    shuffle=True,
    num_workers=12,
    persistent_workers=True,
)
subgraph_loader = NeighborLoader(
    data,
    input_nodes=None,
    num_neighbors=[-1],
    batch_size=2048,
    num_workers=12,
    persistent_workers=True,
)


class GAT(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers,
                 heads):
        super().__init__()

        self.num_layers = num_layers

        self.convs = torch.nn.ModuleList()
        self.convs.append(GATConv(dataset.num_features, hidden_channels,
                                  heads))
        for _ in range(num_layers - 2):
            self.convs.append(
                GATConv(heads * hidden_channels, hidden_channels, heads))
        self.convs.append(
            GATConv(heads * hidden_channels, out_channels, heads,
                    concat=False))

        self.skips = torch.nn.ModuleList()
        self.skips.append(Lin(dataset.num_features, hidden_channels * heads))
        for _ in range(num_layers - 2):
            self.skips.append(
                Lin(hidden_channels * heads, hidden_channels * heads))
        self.skips.append(Lin(hidden_channels * heads, out_channels))

    def reset_parameters(self):
        for conv in self.convs:
            conv.reset_parameters()
        for skip in self.skips:
            skip.reset_parameters()

    def forward(self, x, edge_index):
        for i, (conv, skip) in enumerate(zip(self.convs, self.skips)):
            x = conv(x, edge_index) + skip(x)
            if i != self.num_layers - 1:
                x = F.elu(x)
                x = F.dropout(x, p=0.5, training=self.training)
        return x

    def inference(self, x_all):
        pbar = tqdm(total=x_all.size(0) * self.num_layers)
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch.
        for i in range(self.num_layers):
            xs = []
            for batch in subgraph_loader:
                x = x_all[batch.n_id].to(device)
                edge_index = batch.edge_index.to(device)
                x = self.convs[i](x, edge_index) + self.skips[i](x)
                x = x[:batch.batch_size]
                if i != self.num_layers - 1:
                    x = F.elu(x)
                xs.append(x.cpu())

                pbar.update(batch.batch_size)

            x_all = torch.cat(xs, dim=0)

        pbar.close()

        return x_all


model = GAT(dataset.num_features, 128, dataset.num_classes, num_layers=3,
            heads=4).to(device)


def train(epoch):
    model.train()

    pbar = tqdm(total=split_idx['train'].size(0))
    pbar.set_description(f'Epoch {epoch:02d}')

    total_loss = total_correct = 0
    for batch in train_loader:
        optimizer.zero_grad()
        out = model(batch.x, batch.edge_index.to(device))[:batch.batch_size]
        y = batch.y[:batch.batch_size].squeeze()
        loss = F.cross_entropy(out, y)
        loss.backward()
        optimizer.step()

        total_loss += float(loss)
        total_correct += int(out.argmax(dim=-1).eq(y).sum())
        pbar.update(batch.batch_size)

    pbar.close()

    loss = total_loss / len(train_loader)
    approx_acc = total_correct / split_idx['train'].size(0)

    return loss, approx_acc


@torch.no_grad()
def test():
    model.eval()

    out = model.inference(data.x)

    y_true = data.y.cpu()
    y_pred = out.argmax(dim=-1, keepdim=True)

    train_acc = evaluator.eval({
        'y_true': y_true[split_idx['train']],
        'y_pred': y_pred[split_idx['train']],
    })['acc']
    val_acc = evaluator.eval({
        'y_true': y_true[split_idx['valid']],
        'y_pred': y_pred[split_idx['valid']],
    })['acc']
    test_acc = evaluator.eval({
        'y_true': y_true[split_idx['test']],
        'y_pred': y_pred[split_idx['test']],
    })['acc']

    return train_acc, val_acc, test_acc


test_accs = []
for run in range(1, 11):
    print(f'\nRun {run:02d}:\n')

    model.reset_parameters()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

    best_val_acc = final_test_acc = 0
    for epoch in range(1, 101):
        loss, acc = train(epoch)
        print(f'Epoch {epoch:02d}, Loss: {loss:.4f}, Approx. Train: {acc:.4f}')

        if epoch > 50 and epoch % 10 == 0:
            train_acc, val_acc, test_acc = test()
            print(f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
                  f'Test: {test_acc:.4f}')

            if val_acc > best_val_acc:
                best_val_acc = val_acc
                final_test_acc = test_acc
    test_accs.append(final_test_acc)

test_acc = torch.tensor(test_accs)
print('============================')
print(f'Final Test: {test_acc.mean():.4f} Β± {test_acc.std():.4f}')
pytorch/examples/ogbn_products_sage.py000066400000000000000000000116041444262304300206110ustar00rootroot00000000000000# Reaches around 0.7870 Β± 0.0036 test accuracy.

import os.path as osp

import torch
import torch.nn.functional as F
from ogb.nodeproppred import Evaluator, PygNodePropPredDataset
from tqdm import tqdm

from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import SAGEConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'products')
dataset = PygNodePropPredDataset('ogbn-products', root)
split_idx = dataset.get_idx_split()
evaluator = Evaluator(name='ogbn-products')
data = dataset[0].to(device, 'x', 'y')

train_loader = NeighborLoader(
    data,
    input_nodes=split_idx['train'],
    num_neighbors=[15, 10, 5],
    batch_size=1024,
    shuffle=True,
    num_workers=12,
    persistent_workers=True,
)
subgraph_loader = NeighborLoader(
    data,
    input_nodes=None,
    num_neighbors=[-1],
    batch_size=4096,
    num_workers=12,
    persistent_workers=True,
)


class SAGE(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers):
        super().__init__()

        self.num_layers = num_layers

        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        for _ in range(num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def reset_parameters(self):
        for conv in self.convs:
            conv.reset_parameters()

    def forward(self, x, edge_index):
        for i, conv in enumerate(self.convs):
            x = conv(x, edge_index)
            if i != self.num_layers - 1:
                x = x.relu()
                x = F.dropout(x, p=0.5, training=self.training)
        return x

    def inference(self, x_all):
        pbar = tqdm(total=x_all.size(0) * self.num_layers)
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch.
        for i in range(self.num_layers):
            xs = []
            for batch in subgraph_loader:
                x = x_all[batch.n_id].to(device)
                edge_index = batch.edge_index.to(device)
                x = self.convs[i](x, edge_index)
                x = x[:batch.batch_size]
                if i != self.num_layers - 1:
                    x = x.relu()
                xs.append(x.cpu())

                pbar.update(batch.batch_size)

            x_all = torch.cat(xs, dim=0)

        pbar.close()

        return x_all


model = SAGE(dataset.num_features, 256, dataset.num_classes, num_layers=3)
model = model.to(device)


def train(epoch):
    model.train()

    pbar = tqdm(total=split_idx['train'].size(0))
    pbar.set_description(f'Epoch {epoch:02d}')

    total_loss = total_correct = 0
    for batch in train_loader:
        optimizer.zero_grad()
        out = model(batch.x, batch.edge_index.to(device))[:batch.batch_size]
        y = batch.y[:batch.batch_size].squeeze()
        loss = F.cross_entropy(out, y)
        loss.backward()
        optimizer.step()

        total_loss += float(loss)
        total_correct += int(out.argmax(dim=-1).eq(y).sum())
        pbar.update(batch.batch_size)

    pbar.close()

    loss = total_loss / len(train_loader)
    approx_acc = total_correct / split_idx['train'].size(0)

    return loss, approx_acc


@torch.no_grad()
def test():
    model.eval()

    out = model.inference(data.x)

    y_true = data.y.cpu()
    y_pred = out.argmax(dim=-1, keepdim=True)

    train_acc = evaluator.eval({
        'y_true': y_true[split_idx['train']],
        'y_pred': y_pred[split_idx['train']],
    })['acc']
    val_acc = evaluator.eval({
        'y_true': y_true[split_idx['valid']],
        'y_pred': y_pred[split_idx['valid']],
    })['acc']
    test_acc = evaluator.eval({
        'y_true': y_true[split_idx['test']],
        'y_pred': y_pred[split_idx['test']],
    })['acc']

    return train_acc, val_acc, test_acc


test_accs = []
for run in range(1, 11):
    print(f'\nRun {run:02d}:\n')

    model.reset_parameters()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.003)

    best_val_acc = final_test_acc = 0
    for epoch in range(1, 21):
        loss, acc = train(epoch)
        print(f'Epoch {epoch:02d}, Loss: {loss:.4f}, Approx. Train: {acc:.4f}')

        if epoch > 5:
            train_acc, val_acc, test_acc = test()
            print(f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
                  f'Test: {test_acc:.4f}')

            if val_acc > best_val_acc:
                best_val_acc = val_acc
                final_test_acc = test_acc
    test_accs.append(final_test_acc)

test_acc = torch.tensor(test_accs)
print('============================')
print(f'Final Test: {test_acc.mean():.4f} Β± {test_acc.std():.4f}')
pytorch/examples/ogbn_proteins_deepgcn.py000066400000000000000000000107461444262304300213050ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from ogb.nodeproppred import Evaluator, PygNodePropPredDataset
from torch.nn import LayerNorm, Linear, ReLU
from tqdm import tqdm

from torch_geometric.loader import RandomNodeLoader
from torch_geometric.nn import DeepGCNLayer, GENConv
from torch_geometric.utils import scatter

dataset = PygNodePropPredDataset('ogbn-proteins', root='../data')
splitted_idx = dataset.get_idx_split()
data = dataset[0]
data.node_species = None
data.y = data.y.to(torch.float)

# Initialize features of nodes by aggregating edge features.
row, col = data.edge_index
data.x = scatter(data.edge_attr, col, dim_size=data.num_nodes, reduce='sum')

# Set split indices to masks.
for split in ['train', 'valid', 'test']:
    mask = torch.zeros(data.num_nodes, dtype=torch.bool)
    mask[splitted_idx[split]] = True
    data[f'{split}_mask'] = mask

train_loader = RandomNodeLoader(data, num_parts=40, shuffle=True,
                                num_workers=5)
test_loader = RandomNodeLoader(data, num_parts=5, num_workers=5)


class DeeperGCN(torch.nn.Module):
    def __init__(self, hidden_channels, num_layers):
        super().__init__()

        self.node_encoder = Linear(data.x.size(-1), hidden_channels)
        self.edge_encoder = Linear(data.edge_attr.size(-1), hidden_channels)

        self.layers = torch.nn.ModuleList()
        for i in range(1, num_layers + 1):
            conv = GENConv(hidden_channels, hidden_channels, aggr='softmax',
                           t=1.0, learn_t=True, num_layers=2, norm='layer')
            norm = LayerNorm(hidden_channels, elementwise_affine=True)
            act = ReLU(inplace=True)

            layer = DeepGCNLayer(conv, norm, act, block='res+', dropout=0.1,
                                 ckpt_grad=i % 3)
            self.layers.append(layer)

        self.lin = Linear(hidden_channels, data.y.size(-1))

    def forward(self, x, edge_index, edge_attr):
        x = self.node_encoder(x)
        edge_attr = self.edge_encoder(edge_attr)

        x = self.layers[0].conv(x, edge_index, edge_attr)

        for layer in self.layers[1:]:
            x = layer(x, edge_index, edge_attr)

        x = self.layers[0].act(self.layers[0].norm(x))
        x = F.dropout(x, p=0.1, training=self.training)

        return self.lin(x)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = DeeperGCN(hidden_channels=64, num_layers=28).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = torch.nn.BCEWithLogitsLoss()
evaluator = Evaluator('ogbn-proteins')


def train(epoch):
    model.train()

    pbar = tqdm(total=len(train_loader))
    pbar.set_description(f'Training epoch: {epoch:04d}')

    total_loss = total_examples = 0
    for data in train_loader:
        optimizer.zero_grad()
        data = data.to(device)
        out = model(data.x, data.edge_index, data.edge_attr)
        loss = criterion(out[data.train_mask], data.y[data.train_mask])
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * int(data.train_mask.sum())
        total_examples += int(data.train_mask.sum())

        pbar.update(1)

    pbar.close()

    return total_loss / total_examples


@torch.no_grad()
def test():
    model.eval()

    y_true = {'train': [], 'valid': [], 'test': []}
    y_pred = {'train': [], 'valid': [], 'test': []}

    pbar = tqdm(total=len(test_loader))
    pbar.set_description(f'Evaluating epoch: {epoch:04d}')

    for data in test_loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.edge_attr)

        for split in y_true.keys():
            mask = data[f'{split}_mask']
            y_true[split].append(data.y[mask].cpu())
            y_pred[split].append(out[mask].cpu())

        pbar.update(1)

    pbar.close()

    train_rocauc = evaluator.eval({
        'y_true': torch.cat(y_true['train'], dim=0),
        'y_pred': torch.cat(y_pred['train'], dim=0),
    })['rocauc']

    valid_rocauc = evaluator.eval({
        'y_true': torch.cat(y_true['valid'], dim=0),
        'y_pred': torch.cat(y_pred['valid'], dim=0),
    })['rocauc']

    test_rocauc = evaluator.eval({
        'y_true': torch.cat(y_true['test'], dim=0),
        'y_pred': torch.cat(y_pred['test'], dim=0),
    })['rocauc']

    return train_rocauc, valid_rocauc, test_rocauc


for epoch in range(1, 1001):
    loss = train(epoch)
    train_rocauc, valid_rocauc, test_rocauc = test()
    print(f'Loss: {loss:.4f}, Train: {train_rocauc:.4f}, '
          f'Val: {valid_rocauc:.4f}, Test: {test_rocauc:.4f}')
pytorch/examples/pmlp.py000066400000000000000000000030511444262304300157070ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import PMLP

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=T.NormalizeFeatures())
data = dataset[0].to(device)

model = PMLP(
    in_channels=dataset.num_features,
    hidden_channels=16,
    out_channels=dataset.num_classes,
    num_layers=2,
    dropout=0.5,
    norm=False,
).to(device)

optimizer = torch.optim.Adam(model.parameters(), weight_decay=5e-4, lr=0.01)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x)  # MLP during training.
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    out = model(data.x, data.edge_index)
    pred = out.argmax(dim=-1)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
    return accs


best_val_acc = final_test_acc = 0
for epoch in range(1, 201):
    loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/pna.py000066400000000000000000000072521444262304300155240ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Embedding, Linear, ModuleList, ReLU, Sequential
from torch.optim.lr_scheduler import ReduceLROnPlateau

from torch_geometric.datasets import ZINC
from torch_geometric.loader import DataLoader
from torch_geometric.nn import BatchNorm, PNAConv, global_add_pool
from torch_geometric.utils import degree

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ZINC')
train_dataset = ZINC(path, subset=True, split='train')
val_dataset = ZINC(path, subset=True, split='val')
test_dataset = ZINC(path, subset=True, split='test')

train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=128)
test_loader = DataLoader(test_dataset, batch_size=128)

# Compute the maximum in-degree in the training data.
max_degree = -1
for data in train_dataset:
    d = degree(data.edge_index[1], num_nodes=data.num_nodes, dtype=torch.long)
    max_degree = max(max_degree, int(d.max()))

# Compute the in-degree histogram tensor
deg = torch.zeros(max_degree + 1, dtype=torch.long)
for data in train_dataset:
    d = degree(data.edge_index[1], num_nodes=data.num_nodes, dtype=torch.long)
    deg += torch.bincount(d, minlength=deg.numel())


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.node_emb = Embedding(21, 75)
        self.edge_emb = Embedding(4, 50)

        aggregators = ['mean', 'min', 'max', 'std']
        scalers = ['identity', 'amplification', 'attenuation']

        self.convs = ModuleList()
        self.batch_norms = ModuleList()
        for _ in range(4):
            conv = PNAConv(in_channels=75, out_channels=75,
                           aggregators=aggregators, scalers=scalers, deg=deg,
                           edge_dim=50, towers=5, pre_layers=1, post_layers=1,
                           divide_input=False)
            self.convs.append(conv)
            self.batch_norms.append(BatchNorm(75))

        self.mlp = Sequential(Linear(75, 50), ReLU(), Linear(50, 25), ReLU(),
                              Linear(25, 1))

    def forward(self, x, edge_index, edge_attr, batch):
        x = self.node_emb(x.squeeze())
        edge_attr = self.edge_emb(edge_attr)

        for conv, batch_norm in zip(self.convs, self.batch_norms):
            x = F.relu(batch_norm(conv(x, edge_index, edge_attr)))

        x = global_add_pool(x, batch)
        return self.mlp(x)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=20,
                              min_lr=0.00001)


def train(epoch):
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
        loss = (out.squeeze() - data.y).abs().mean()
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_error = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
        total_error += (out.squeeze() - data.y).abs().sum().item()
    return total_error / len(loader.dataset)


for epoch in range(1, 301):
    loss = train(epoch)
    val_mae = test(val_loader)
    test_mae = test(test_loader)
    scheduler.step(val_mae)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_mae:.4f}, '
          f'Test: {test_mae:.4f}')
pytorch/examples/point_transformer_classification.py000066400000000000000000000142321444262304300235700ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear as Lin

import torch_geometric.transforms as T
from torch_geometric.datasets import ModelNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import (
    MLP,
    PointTransformerConv,
    fps,
    global_mean_pool,
    knn,
    knn_graph,
)
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import scatter

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data/ModelNet10')
pre_transform, transform = T.NormalizeScale(), T.SamplePoints(1024)
train_dataset = ModelNet(path, '10', True, transform, pre_transform)
test_dataset = ModelNet(path, '10', False, transform, pre_transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)


class TransformerBlock(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.lin_in = Lin(in_channels, in_channels)
        self.lin_out = Lin(out_channels, out_channels)

        self.pos_nn = MLP([3, 64, out_channels], norm=None, plain_last=False)

        self.attn_nn = MLP([out_channels, 64, out_channels], norm=None,
                           plain_last=False)

        self.transformer = PointTransformerConv(in_channels, out_channels,
                                                pos_nn=self.pos_nn,
                                                attn_nn=self.attn_nn)

    def forward(self, x, pos, edge_index):
        x = self.lin_in(x).relu()
        x = self.transformer(x, pos, edge_index)
        x = self.lin_out(x).relu()
        return x


class TransitionDown(torch.nn.Module):
    '''
        Samples the input point cloud by a ratio percentage to reduce
        cardinality and uses an mlp to augment features dimensionnality
    '''
    def __init__(self, in_channels, out_channels, ratio=0.25, k=16):
        super().__init__()
        self.k = k
        self.ratio = ratio
        self.mlp = MLP([in_channels, out_channels], plain_last=False)

    def forward(self, x, pos, batch):
        # FPS sampling
        id_clusters = fps(pos, ratio=self.ratio, batch=batch)

        # compute for each cluster the k nearest points
        sub_batch = batch[id_clusters] if batch is not None else None

        # beware of self loop
        id_k_neighbor = knn(pos, pos[id_clusters], k=self.k, batch_x=batch,
                            batch_y=sub_batch)

        # transformation of features through a simple MLP
        x = self.mlp(x)

        # Max pool onto each cluster the features from knn in points
        x_out = scatter(x[id_k_neighbor[1]], id_k_neighbor[0], dim=0,
                        dim_size=id_clusters.size(0), reduce='max')

        # keep only the clusters and their max-pooled features
        sub_pos, out = pos[id_clusters], x_out
        return out, sub_pos, sub_batch


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, dim_model, k=16):
        super().__init__()
        self.k = k

        # dummy feature is created if there is none given
        in_channels = max(in_channels, 1)

        # first block
        self.mlp_input = MLP([in_channels, dim_model[0]], plain_last=False)

        self.transformer_input = TransformerBlock(in_channels=dim_model[0],
                                                  out_channels=dim_model[0])
        # backbone layers
        self.transformers_down = torch.nn.ModuleList()
        self.transition_down = torch.nn.ModuleList()

        for i in range(len(dim_model) - 1):
            # Add Transition Down block followed by a Transformer block
            self.transition_down.append(
                TransitionDown(in_channels=dim_model[i],
                               out_channels=dim_model[i + 1], k=self.k))

            self.transformers_down.append(
                TransformerBlock(in_channels=dim_model[i + 1],
                                 out_channels=dim_model[i + 1]))

        # class score computation
        self.mlp_output = MLP([dim_model[-1], 64, out_channels], norm=None)

    def forward(self, x, pos, batch=None):

        # add dummy features in case there is none
        if x is None:
            x = torch.ones((pos.shape[0], 1), device=pos.get_device())

        # first block
        x = self.mlp_input(x)
        edge_index = knn_graph(pos, k=self.k, batch=batch)
        x = self.transformer_input(x, pos, edge_index)

        # backbone
        for i in range(len(self.transformers_down)):
            x, pos, batch = self.transition_down[i](x, pos, batch=batch)

            edge_index = knn_graph(pos, k=self.k, batch=batch)
            x = self.transformers_down[i](x, pos, edge_index)

        # GlobalAveragePooling
        x = global_mean_pool(x, batch)

        # Class score
        out = self.mlp_output(x)

        return F.log_softmax(out, dim=-1)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.pos, data.batch)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()
    return total_loss / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    correct = 0
    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.pos, data.batch).max(dim=1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(loader.dataset)


if __name__ == '__main__':

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = Net(0, train_dataset.num_classes,
                dim_model=[32, 64, 128, 256, 512], k=16).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20,
                                                gamma=0.5)

    for epoch in range(1, 201):
        loss = train()
        test_acc = test(test_loader)
        print(f'Epoch {epoch:03d}, Loss: {loss:.4f}, Test: {test_acc:.4f}')
        scheduler.step()
pytorch/examples/point_transformer_segmentation.py000066400000000000000000000201201444262304300232630ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from point_transformer_classification import TransformerBlock, TransitionDown
from torchmetrics.functional import jaccard_index

import torch_geometric.transforms as T
from torch_geometric.datasets import ShapeNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, knn_graph, knn_interpolate
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import scatter

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

category = 'Airplane'  # Pass in `None` to train on all categories.
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')
transform = T.Compose([
    T.RandomJitter(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2),
])
pre_transform = T.NormalizeScale()
train_dataset = ShapeNet(path, category, split='trainval', transform=transform,
                         pre_transform=pre_transform)
test_dataset = ShapeNet(path, category, split='test',
                        pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=10, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=10, shuffle=False)


class TransitionUp(torch.nn.Module):
    '''
        Reduce features dimensionnality and interpolate back to higher
        resolution and cardinality
    '''
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.mlp_sub = MLP([in_channels, out_channels], plain_last=False)
        self.mlp = MLP([out_channels, out_channels], plain_last=False)

    def forward(self, x, x_sub, pos, pos_sub, batch=None, batch_sub=None):
        # transform low-res features and reduce the number of features
        x_sub = self.mlp_sub(x_sub)

        # interpolate low-res feats to high-res points
        x_interpolated = knn_interpolate(x_sub, pos_sub, pos, k=3,
                                         batch_x=batch_sub, batch_y=batch)

        x = self.mlp(x) + x_interpolated

        return x


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, dim_model, k=16):
        super().__init__()
        self.k = k

        # dummy feature is created if there is none given
        in_channels = max(in_channels, 1)

        # first block
        self.mlp_input = MLP([in_channels, dim_model[0]], plain_last=False)

        self.transformer_input = TransformerBlock(
            in_channels=dim_model[0],
            out_channels=dim_model[0],
        )

        # backbone layers
        self.transformers_up = torch.nn.ModuleList()
        self.transformers_down = torch.nn.ModuleList()
        self.transition_up = torch.nn.ModuleList()
        self.transition_down = torch.nn.ModuleList()

        for i in range(0, len(dim_model) - 1):

            # Add Transition Down block followed by a Point Transformer block
            self.transition_down.append(
                TransitionDown(in_channels=dim_model[i],
                               out_channels=dim_model[i + 1], k=self.k))

            self.transformers_down.append(
                TransformerBlock(in_channels=dim_model[i + 1],
                                 out_channels=dim_model[i + 1]))

            # Add Transition Up block followed by Point Transformer block
            self.transition_up.append(
                TransitionUp(in_channels=dim_model[i + 1],
                             out_channels=dim_model[i]))

            self.transformers_up.append(
                TransformerBlock(in_channels=dim_model[i],
                                 out_channels=dim_model[i]))

        # summit layers
        self.mlp_summit = MLP([dim_model[-1], dim_model[-1]], norm=None,
                              plain_last=False)

        self.transformer_summit = TransformerBlock(
            in_channels=dim_model[-1],
            out_channels=dim_model[-1],
        )

        # class score computation
        self.mlp_output = MLP([dim_model[0], 64, out_channels], norm=None)

    def forward(self, x, pos, batch=None):

        # add dummy features in case there is none
        if x is None:
            x = torch.ones((pos.shape[0], 1)).to(pos.get_device())

        out_x = []
        out_pos = []
        out_batch = []

        # first block
        x = self.mlp_input(x)
        edge_index = knn_graph(pos, k=self.k, batch=batch)
        x = self.transformer_input(x, pos, edge_index)

        # save outputs for skipping connections
        out_x.append(x)
        out_pos.append(pos)
        out_batch.append(batch)

        # backbone down : #reduce cardinality and augment dimensionnality
        for i in range(len(self.transformers_down)):
            x, pos, batch = self.transition_down[i](x, pos, batch=batch)
            edge_index = knn_graph(pos, k=self.k, batch=batch)
            x = self.transformers_down[i](x, pos, edge_index)

            out_x.append(x)
            out_pos.append(pos)
            out_batch.append(batch)

        # summit
        x = self.mlp_summit(x)
        edge_index = knn_graph(pos, k=self.k, batch=batch)
        x = self.transformer_summit(x, pos, edge_index)

        # backbone up : augment cardinality and reduce dimensionnality
        n = len(self.transformers_down)
        for i in range(n):
            x = self.transition_up[-i - 1](x=out_x[-i - 2], x_sub=x,
                                           pos=out_pos[-i - 2],
                                           pos_sub=out_pos[-i - 1],
                                           batch_sub=out_batch[-i - 1],
                                           batch=out_batch[-i - 2])

            edge_index = knn_graph(out_pos[-i - 2], k=self.k,
                                   batch=out_batch[-i - 2])
            x = self.transformers_up[-i - 1](x, out_pos[-i - 2], edge_index)

        # Class score
        out = self.mlp_output(x)

        return F.log_softmax(out, dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(3, train_dataset.num_classes, dim_model=[32, 64, 128, 256, 512],
            k=16).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)


def train():
    model.train()

    total_loss = correct_nodes = total_nodes = 0
    for i, data in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.pos, data.batch)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        correct_nodes += out.argmax(dim=1).eq(data.y).sum().item()
        total_nodes += data.num_nodes

        if (i + 1) % 10 == 0:
            print(f'[{i+1}/{len(train_loader)}] Loss: {total_loss / 10:.4f} '
                  f'Train Acc: {correct_nodes / total_nodes:.4f}')
            total_loss = correct_nodes = total_nodes = 0


def test(loader):
    model.eval()

    ious, categories = [], []
    y_map = torch.empty(loader.dataset.num_classes, device=device).long()
    for data in loader:
        data = data.to(device)
        outs = model(data.x, data.pos, data.batch)

        sizes = (data.ptr[1:] - data.ptr[:-1]).tolist()
        for out, y, category in zip(outs.split(sizes), data.y.split(sizes),
                                    data.category.tolist()):
            category = list(ShapeNet.seg_classes.keys())[category]
            part = ShapeNet.seg_classes[category]
            part = torch.tensor(part, device=device)

            y_map[part] = torch.arange(part.size(0), device=device)

            iou = jaccard_index(out[:, part].argmax(dim=-1), y_map[y],
                                num_classes=part.size(0), absent_score=1.0)
            ious.append(iou)

        categories.append(data.category)

    iou = torch.tensor(ious, device=device)
    category = torch.cat(categories, dim=0)

    mean_iou = scatter(iou, category, reduce='mean')  # Per-category IoU.
    return float(mean_iou.mean())  # Global IoU.


for epoch in range(1, 100):
    train()
    iou = test(test_loader)
    print(f'Epoch: {epoch:03d}, Test IoU: {iou:.4f}')
    scheduler.step()
pytorch/examples/pointnet2_classification.py000066400000000000000000000067401444262304300217440ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import ModelNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, PointNetConv, fps, global_max_pool, radius
from torch_geometric.typing import WITH_TORCH_CLUSTER

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")


class SAModule(torch.nn.Module):
    def __init__(self, ratio, r, nn):
        super().__init__()
        self.ratio = ratio
        self.r = r
        self.conv = PointNetConv(nn, add_self_loops=False)

    def forward(self, x, pos, batch):
        idx = fps(pos, batch, ratio=self.ratio)
        row, col = radius(pos, pos[idx], self.r, batch, batch[idx],
                          max_num_neighbors=64)
        edge_index = torch.stack([col, row], dim=0)
        x_dst = None if x is None else x[idx]
        x = self.conv((x, x_dst), (pos, pos[idx]), edge_index)
        pos, batch = pos[idx], batch[idx]
        return x, pos, batch


class GlobalSAModule(torch.nn.Module):
    def __init__(self, nn):
        super().__init__()
        self.nn = nn

    def forward(self, x, pos, batch):
        x = self.nn(torch.cat([x, pos], dim=1))
        x = global_max_pool(x, batch)
        pos = pos.new_zeros((x.size(0), 3))
        batch = torch.arange(x.size(0), device=batch.device)
        return x, pos, batch


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Input channels account for both `pos` and node features.
        self.sa1_module = SAModule(0.5, 0.2, MLP([3, 64, 64, 128]))
        self.sa2_module = SAModule(0.25, 0.4, MLP([128 + 3, 128, 128, 256]))
        self.sa3_module = GlobalSAModule(MLP([256 + 3, 256, 512, 1024]))

        self.mlp = MLP([1024, 512, 256, 10], dropout=0.5, norm=None)

    def forward(self, data):
        sa0_out = (data.x, data.pos, data.batch)
        sa1_out = self.sa1_module(*sa0_out)
        sa2_out = self.sa2_module(*sa1_out)
        sa3_out = self.sa3_module(*sa2_out)
        x, pos, batch = sa3_out

        return self.mlp(x).log_softmax(dim=-1)


def train(epoch):
    model.train()

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = F.nll_loss(model(data), data.y)
        loss.backward()
        optimizer.step()


def test(loader):
    model.eval()

    correct = 0
    for data in loader:
        data = data.to(device)
        with torch.no_grad():
            pred = model(data).max(1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(loader.dataset)


if __name__ == '__main__':
    path = osp.join(osp.dirname(osp.realpath(__file__)), '..',
                    'data/ModelNet10')
    pre_transform, transform = T.NormalizeScale(), T.SamplePoints(1024)
    train_dataset = ModelNet(path, '10', True, transform, pre_transform)
    test_dataset = ModelNet(path, '10', False, transform, pre_transform)
    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True,
                              num_workers=6)
    test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False,
                             num_workers=6)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = Net().to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

    for epoch in range(1, 201):
        train(epoch)
        test_acc = test(test_loader)
        print(f'Epoch: {epoch:03d}, Test: {test_acc:.4f}')
pytorch/examples/pointnet2_segmentation.py000066400000000000000000000117101444262304300214370ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from pointnet2_classification import GlobalSAModule, SAModule
from torchmetrics.functional import jaccard_index

import torch_geometric.transforms as T
from torch_geometric.datasets import ShapeNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, knn_interpolate
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import scatter

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

category = 'Airplane'  # Pass in `None` to train on all categories.
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')
transform = T.Compose([
    T.RandomJitter(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2)
])
pre_transform = T.NormalizeScale()
train_dataset = ShapeNet(path, category, split='trainval', transform=transform,
                         pre_transform=pre_transform)
test_dataset = ShapeNet(path, category, split='test',
                        pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=12, shuffle=True,
                          num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=12, shuffle=False,
                         num_workers=6)


class FPModule(torch.nn.Module):
    def __init__(self, k, nn):
        super().__init__()
        self.k = k
        self.nn = nn

    def forward(self, x, pos, batch, x_skip, pos_skip, batch_skip):
        x = knn_interpolate(x, pos, pos_skip, batch, batch_skip, k=self.k)
        if x_skip is not None:
            x = torch.cat([x, x_skip], dim=1)
        x = self.nn(x)
        return x, pos_skip, batch_skip


class Net(torch.nn.Module):
    def __init__(self, num_classes):
        super().__init__()

        # Input channels account for both `pos` and node features.
        self.sa1_module = SAModule(0.2, 0.2, MLP([3 + 3, 64, 64, 128]))
        self.sa2_module = SAModule(0.25, 0.4, MLP([128 + 3, 128, 128, 256]))
        self.sa3_module = GlobalSAModule(MLP([256 + 3, 256, 512, 1024]))

        self.fp3_module = FPModule(1, MLP([1024 + 256, 256, 256]))
        self.fp2_module = FPModule(3, MLP([256 + 128, 256, 128]))
        self.fp1_module = FPModule(3, MLP([128 + 3, 128, 128, 128]))

        self.mlp = MLP([128, 128, 128, num_classes], dropout=0.5, norm=None)

        self.lin1 = torch.nn.Linear(128, 128)
        self.lin2 = torch.nn.Linear(128, 128)
        self.lin3 = torch.nn.Linear(128, num_classes)

    def forward(self, data):
        sa0_out = (data.x, data.pos, data.batch)
        sa1_out = self.sa1_module(*sa0_out)
        sa2_out = self.sa2_module(*sa1_out)
        sa3_out = self.sa3_module(*sa2_out)

        fp3_out = self.fp3_module(*sa3_out, *sa2_out)
        fp2_out = self.fp2_module(*fp3_out, *sa1_out)
        x, _, _ = self.fp1_module(*fp2_out, *sa0_out)

        return self.mlp(x).log_softmax(dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(train_dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train():
    model.train()

    total_loss = correct_nodes = total_nodes = 0
    for i, data in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        correct_nodes += out.argmax(dim=1).eq(data.y).sum().item()
        total_nodes += data.num_nodes

        if (i + 1) % 10 == 0:
            print(f'[{i+1}/{len(train_loader)}] Loss: {total_loss / 10:.4f} '
                  f'Train Acc: {correct_nodes / total_nodes:.4f}')
            total_loss = correct_nodes = total_nodes = 0


@torch.no_grad()
def test(loader):
    model.eval()

    ious, categories = [], []
    y_map = torch.empty(loader.dataset.num_classes, device=device).long()
    for data in loader:
        data = data.to(device)
        outs = model(data)

        sizes = (data.ptr[1:] - data.ptr[:-1]).tolist()
        for out, y, category in zip(outs.split(sizes), data.y.split(sizes),
                                    data.category.tolist()):
            category = list(ShapeNet.seg_classes.keys())[category]
            part = ShapeNet.seg_classes[category]
            part = torch.tensor(part, device=device)

            y_map[part] = torch.arange(part.size(0), device=device)

            iou = jaccard_index(out[:, part].argmax(dim=-1), y_map[y],
                                num_classes=part.size(0), absent_score=1.0)
            ious.append(iou)

        categories.append(data.category)

    iou = torch.tensor(ious, device=device)
    category = torch.cat(categories, dim=0)

    mean_iou = scatter(iou, category, reduce='mean')  # Per-category IoU.
    return float(mean_iou.mean())  # Global IoU.


for epoch in range(1, 31):
    train()
    iou = test(test_loader)
    print(f'Epoch: {epoch:02d}, Test IoU: {iou:.4f}')
pytorch/examples/ppi.py000066400000000000000000000047461444262304300155430ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from sklearn.metrics import f1_score

from torch_geometric.datasets import PPI
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GATConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PPI')
train_dataset = PPI(path, split='train')
val_dataset = PPI(path, split='val')
test_dataset = PPI(path, split='test')
train_loader = DataLoader(train_dataset, batch_size=1, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GATConv(train_dataset.num_features, 256, heads=4)
        self.lin1 = torch.nn.Linear(train_dataset.num_features, 4 * 256)
        self.conv2 = GATConv(4 * 256, 256, heads=4)
        self.lin2 = torch.nn.Linear(4 * 256, 4 * 256)
        self.conv3 = GATConv(4 * 256, train_dataset.num_classes, heads=6,
                             concat=False)
        self.lin3 = torch.nn.Linear(4 * 256, train_dataset.num_classes)

    def forward(self, x, edge_index):
        x = F.elu(self.conv1(x, edge_index) + self.lin1(x))
        x = F.elu(self.conv2(x, edge_index) + self.lin2(x))
        x = self.conv3(x, edge_index) + self.lin3(x)
        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
loss_op = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = loss_op(model(data.x, data.edge_index), data.y)
        total_loss += loss.item() * data.num_graphs
        loss.backward()
        optimizer.step()
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    ys, preds = [], []
    for data in loader:
        ys.append(data.y)
        out = model(data.x.to(device), data.edge_index.to(device))
        preds.append((out > 0).float().cpu())

    y, pred = torch.cat(ys, dim=0).numpy(), torch.cat(preds, dim=0).numpy()
    return f1_score(y, pred, average='micro') if pred.sum() > 0 else 0


for epoch in range(1, 101):
    loss = train()
    val_f1 = test(val_loader)
    test_f1 = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val: {val_f1:.4f}, '
          f'Test: {test_f1:.4f}')
pytorch/examples/proteins_diff_pool.py000066400000000000000000000107101444262304300206230ustar00rootroot00000000000000import os.path as osp
from math import ceil

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DenseDataLoader
from torch_geometric.nn import DenseSAGEConv, dense_diff_pool

max_nodes = 150

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data',
                'PROTEINS_dense')
dataset = TUDataset(
    path,
    name='PROTEINS',
    transform=T.ToDense(max_nodes),
    pre_filter=lambda data: data.num_nodes <= max_nodes,
)
dataset = dataset.shuffle()
n = (len(dataset) + 9) // 10
test_dataset = dataset[:n]
val_dataset = dataset[n:2 * n]
train_dataset = dataset[2 * n:]
test_loader = DenseDataLoader(test_dataset, batch_size=20)
val_loader = DenseDataLoader(val_dataset, batch_size=20)
train_loader = DenseDataLoader(train_dataset, batch_size=20)


class GNN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels,
                 normalize=False, lin=True):
        super().__init__()

        self.conv1 = DenseSAGEConv(in_channels, hidden_channels, normalize)
        self.bn1 = torch.nn.BatchNorm1d(hidden_channels)
        self.conv2 = DenseSAGEConv(hidden_channels, hidden_channels, normalize)
        self.bn2 = torch.nn.BatchNorm1d(hidden_channels)
        self.conv3 = DenseSAGEConv(hidden_channels, out_channels, normalize)
        self.bn3 = torch.nn.BatchNorm1d(out_channels)

        if lin is True:
            self.lin = torch.nn.Linear(2 * hidden_channels + out_channels,
                                       out_channels)
        else:
            self.lin = None

    def bn(self, i, x):
        batch_size, num_nodes, num_channels = x.size()

        x = x.view(-1, num_channels)
        x = getattr(self, f'bn{i}')(x)
        x = x.view(batch_size, num_nodes, num_channels)
        return x

    def forward(self, x, adj, mask=None):
        batch_size, num_nodes, in_channels = x.size()

        x0 = x
        x1 = self.bn(1, self.conv1(x0, adj, mask).relu())
        x2 = self.bn(2, self.conv2(x1, adj, mask).relu())
        x3 = self.bn(3, self.conv3(x2, adj, mask).relu())

        x = torch.cat([x1, x2, x3], dim=-1)

        if self.lin is not None:
            x = self.lin(x).relu()

        return x


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        num_nodes = ceil(0.25 * max_nodes)
        self.gnn1_pool = GNN(dataset.num_features, 64, num_nodes)
        self.gnn1_embed = GNN(dataset.num_features, 64, 64, lin=False)

        num_nodes = ceil(0.25 * num_nodes)
        self.gnn2_pool = GNN(3 * 64, 64, num_nodes)
        self.gnn2_embed = GNN(3 * 64, 64, 64, lin=False)

        self.gnn3_embed = GNN(3 * 64, 64, 64, lin=False)

        self.lin1 = torch.nn.Linear(3 * 64, 64)
        self.lin2 = torch.nn.Linear(64, dataset.num_classes)

    def forward(self, x, adj, mask=None):
        s = self.gnn1_pool(x, adj, mask)
        x = self.gnn1_embed(x, adj, mask)

        x, adj, l1, e1 = dense_diff_pool(x, adj, s, mask)

        s = self.gnn2_pool(x, adj)
        x = self.gnn2_embed(x, adj)

        x, adj, l2, e2 = dense_diff_pool(x, adj, s)

        x = self.gnn3_embed(x, adj)

        x = x.mean(dim=1)
        x = self.lin1(x).relu()
        x = self.lin2(x)
        return F.log_softmax(x, dim=-1), l1 + l2, e1 + e2


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train(epoch):
    model.train()
    loss_all = 0

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        output, _, _ = model(data.x, data.adj, data.mask)
        loss = F.nll_loss(output, data.y.view(-1))
        loss.backward()
        loss_all += data.y.size(0) * float(loss)
        optimizer.step()
    return loss_all / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()
    correct = 0

    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.adj, data.mask)[0].max(dim=1)[1]
        correct += int(pred.eq(data.y.view(-1)).sum())
    return correct / len(loader.dataset)


best_val_acc = test_acc = 0
for epoch in range(1, 151):
    train_loss = train(epoch)
    val_acc = test(val_loader)
    if val_acc > best_val_acc:
        test_acc = test(test_loader)
        best_val_acc = val_acc
    print(f'Epoch: {epoch:03d}, Train Loss: {train_loss:.4f}, '
          f'Val Acc: {val_acc:.4f}, Test Acc: {test_acc:.4f}')
pytorch/examples/proteins_dmon_pool.py000066400000000000000000000067231444262304300206610ustar00rootroot00000000000000import os.path as osp
from math import ceil

import torch
import torch.nn.functional as F
from torch.nn import Linear

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import DenseGraphConv, DMoNPooling, GCNConv
from torch_geometric.utils import to_dense_adj, to_dense_batch

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PROTEINS')
dataset = TUDataset(path, name='PROTEINS').shuffle()
avg_num_nodes = int(dataset.data.x.size(0) / len(dataset))
n = (len(dataset) + 9) // 10
test_dataset = dataset[:n]
val_dataset = dataset[n:2 * n]
train_dataset = dataset[2 * n:]
test_loader = DataLoader(test_dataset, batch_size=20)
val_loader = DataLoader(val_dataset, batch_size=20)
train_loader = DataLoader(train_dataset, batch_size=20)


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, hidden_channels=32):
        super().__init__()

        self.conv1 = GCNConv(in_channels, hidden_channels)
        num_nodes = ceil(0.5 * avg_num_nodes)
        self.pool1 = DMoNPooling([hidden_channels, hidden_channels], num_nodes)

        self.conv2 = DenseGraphConv(hidden_channels, hidden_channels)
        num_nodes = ceil(0.5 * num_nodes)
        self.pool2 = DMoNPooling([hidden_channels, hidden_channels], num_nodes)

        self.conv3 = DenseGraphConv(hidden_channels, hidden_channels)

        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, batch):
        x = self.conv1(x, edge_index).relu()

        x, mask = to_dense_batch(x, batch)
        adj = to_dense_adj(edge_index, batch)

        _, x, adj, sp1, o1, c1 = self.pool1(x, adj, mask)

        x = self.conv2(x, adj).relu()

        _, x, adj, sp2, o2, c2 = self.pool2(x, adj)

        x = self.conv3(x, adj)

        x = x.mean(dim=1)
        x = self.lin1(x).relu()
        x = self.lin2(x)
        return F.log_softmax(x, dim=-1), sp1 + sp2 + o1 + o2 + c1 + c2


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train(train_loader):
    model.train()
    loss_all = 0

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out, tot_loss = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(out, data.y.view(-1)) + tot_loss
        loss.backward()
        loss_all += data.y.size(0) * float(loss)
        optimizer.step()
    return loss_all / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()
    correct = 0
    loss_all = 0

    for data in loader:
        data = data.to(device)
        pred, tot_loss = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(pred, data.y.view(-1)) + tot_loss
        loss_all += data.y.size(0) * float(loss)
        correct += int(pred.max(dim=1)[1].eq(data.y.view(-1)).sum())

    return loss_all / len(loader.dataset), correct / len(loader.dataset)


for epoch in range(1, 101):
    train_loss = train(train_loader)
    _, train_acc = test(train_loader)
    val_loss, val_acc = test(val_loader)
    test_loss, test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Loss: {train_loss:.3f}, '
          f'Train Acc: {train_acc:.3f}, Val Loss: {val_loss:.3f}, '
          f'Val Acc: {val_acc:.3f}, Test Loss: {test_loss:.3f}, '
          f'Test Acc: {test_acc:.3f}')
pytorch/examples/proteins_gmt.py000066400000000000000000000047741444262304300174660ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GCNConv, GraphMultisetTransformer

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PROTEINS')
dataset = TUDataset(path, name='PROTEINS').shuffle()

n = (len(dataset) + 9) // 10
train_dataset = dataset[2 * n:]
val_dataset = dataset[n:2 * n]
test_dataset = dataset[:n]

train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=128)
test_loader = DataLoader(test_dataset, batch_size=128)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = GCNConv(dataset.num_features, 32)
        self.conv2 = GCNConv(32, 32)
        self.conv3 = GCNConv(32, 32)

        self.pool = GraphMultisetTransformer(96, k=10, heads=4)

        self.lin1 = Linear(96, 16)
        self.lin2 = Linear(16, dataset.num_classes)

    def forward(self, x0, edge_index, batch):
        x1 = self.conv1(x0, edge_index).relu()
        x2 = self.conv2(x1, edge_index).relu()
        x3 = self.conv3(x2, edge_index).relu()
        x = torch.cat([x1, x2, x3], dim=-1)

        x = self.pool(x, batch)

        x = self.lin1(x).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin2(x)

        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = F.cross_entropy(out, data.y)
        loss.backward()
        total_loss += data.num_graphs * float(loss)
        optimizer.step()
    return total_loss / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.batch)
        total_correct += int((out.argmax(dim=-1) == data.y).sum())
    return total_correct / len(loader.dataset)


for epoch in range(1, 201):
    train_loss = train()
    val_acc = test(val_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {train_loss:.4f}, '
          f'Val Acc: {val_acc:.4f}, Test Acc: {test_acc:.4f}')
pytorch/examples/proteins_mincut_pool.py000066400000000000000000000074061444262304300212220ustar00rootroot00000000000000import os.path as osp
from math import ceil

import torch
import torch.nn.functional as F
from torch.nn import Linear

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import DenseGraphConv, GCNConv, dense_mincut_pool
from torch_geometric.utils import to_dense_adj, to_dense_batch

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PROTEINS')
dataset = TUDataset(path, name='PROTEINS').shuffle()
average_nodes = int(dataset.data.x.size(0) / len(dataset))
n = (len(dataset) + 9) // 10
test_dataset = dataset[:n]
val_dataset = dataset[n:2 * n]
train_dataset = dataset[2 * n:]
test_loader = DataLoader(test_dataset, batch_size=20)
val_loader = DataLoader(val_dataset, batch_size=20)
train_loader = DataLoader(train_dataset, batch_size=20)


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, hidden_channels=32):
        super().__init__()

        self.conv1 = GCNConv(in_channels, hidden_channels)
        num_nodes = ceil(0.5 * average_nodes)
        self.pool1 = Linear(hidden_channels, num_nodes)

        self.conv2 = DenseGraphConv(hidden_channels, hidden_channels)
        num_nodes = ceil(0.5 * num_nodes)
        self.pool2 = Linear(hidden_channels, num_nodes)

        self.conv3 = DenseGraphConv(hidden_channels, hidden_channels)

        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, batch):
        x = self.conv1(x, edge_index).relu()

        x, mask = to_dense_batch(x, batch)
        adj = to_dense_adj(edge_index, batch)

        s = self.pool1(x)
        x, adj, mc1, o1 = dense_mincut_pool(x, adj, s, mask)

        x = self.conv2(x, adj).relu()
        s = self.pool2(x)

        x, adj, mc2, o2 = dense_mincut_pool(x, adj, s)

        x = self.conv3(x, adj)

        x = x.mean(dim=1)
        x = self.lin1(x).relu()
        x = self.lin2(x)
        return F.log_softmax(x, dim=-1), mc1 + mc2, o1 + o2


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=5e-4, weight_decay=1e-4)


def train(epoch):
    model.train()
    loss_all = 0

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out, mc_loss, o_loss = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(out, data.y.view(-1)) + mc_loss + o_loss
        loss.backward()
        loss_all += data.y.size(0) * float(loss)
        optimizer.step()
    return loss_all / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()
    correct = 0
    loss_all = 0

    for data in loader:
        data = data.to(device)
        pred, mc_loss, o_loss = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(pred, data.y.view(-1)) + mc_loss + o_loss
        loss_all += data.y.size(0) * float(loss)
        correct += int(pred.max(dim=1)[1].eq(data.y.view(-1)).sum())

    return loss_all / len(loader.dataset), correct / len(loader.dataset)


best_val_acc = test_acc = 0
best_val_loss = float('inf')
patience = start_patience = 50
for epoch in range(1, 15000):
    train_loss = train(epoch)
    _, train_acc = test(train_loader)
    val_loss, val_acc = test(val_loader)
    if val_loss < best_val_loss:
        test_loss, test_acc = test(test_loader)
        best_val_acc = val_acc
        patience = start_patience
    else:
        patience -= 1
        if patience == 0:
            break
    print(f'Epoch: {epoch:03d}, Train Loss: {train_loss:.3f}, '
          f'Train Acc: {train_acc:.3f}, Val Loss: {val_loss:.3f}, '
          f'Val Acc: {val_acc:.3f}, Test Loss: {test_loss:.3f}, '
          f'Test Acc: {test_acc:.3f}')
pytorch/examples/proteins_topk_pool.py000066400000000000000000000057611444262304300207020ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GraphConv, TopKPooling
from torch_geometric.nn import global_max_pool as gmp
from torch_geometric.nn import global_mean_pool as gap

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'PROTEINS')
dataset = TUDataset(path, name='PROTEINS')
dataset = dataset.shuffle()
n = len(dataset) // 10
test_dataset = dataset[:n]
train_dataset = dataset[n:]
test_loader = DataLoader(test_dataset, batch_size=60)
train_loader = DataLoader(train_dataset, batch_size=60)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = GraphConv(dataset.num_features, 128)
        self.pool1 = TopKPooling(128, ratio=0.8)
        self.conv2 = GraphConv(128, 128)
        self.pool2 = TopKPooling(128, ratio=0.8)
        self.conv3 = GraphConv(128, 128)
        self.pool3 = TopKPooling(128, ratio=0.8)

        self.lin1 = torch.nn.Linear(256, 128)
        self.lin2 = torch.nn.Linear(128, 64)
        self.lin3 = torch.nn.Linear(64, dataset.num_classes)

    def forward(self, data):
        x, edge_index, batch = data.x, data.edge_index, data.batch

        x = F.relu(self.conv1(x, edge_index))
        x, edge_index, _, batch, _, _ = self.pool1(x, edge_index, None, batch)
        x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)

        x = F.relu(self.conv2(x, edge_index))
        x, edge_index, _, batch, _, _ = self.pool2(x, edge_index, None, batch)
        x2 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)

        x = F.relu(self.conv3(x, edge_index))
        x, edge_index, _, batch, _, _ = self.pool3(x, edge_index, None, batch)
        x3 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)

        x = x1 + x2 + x3

        x = F.relu(self.lin1(x))
        x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(self.lin2(x))
        x = F.log_softmax(self.lin3(x), dim=-1)

        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0005)


def train(epoch):
    model.train()

    loss_all = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, data.y)
        loss.backward()
        loss_all += data.num_graphs * loss.item()
        optimizer.step()
    return loss_all / len(train_dataset)


def test(loader):
    model.eval()

    correct = 0
    for data in loader:
        data = data.to(device)
        pred = model(data).max(dim=1)[1]
        correct += pred.eq(data.y).sum().item()
    return correct / len(loader.dataset)


for epoch in range(1, 201):
    loss = train(epoch)
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.5f}, Train Acc: {train_acc:.5f}, '
          f'Test Acc: {test_acc:.5f}')
pytorch/examples/pytorch_ignite/000077500000000000000000000000001444262304300174155ustar00rootroot00000000000000pytorch/examples/pytorch_ignite/gin.py000066400000000000000000000135211444262304300205460ustar00rootroot00000000000000import os.path as osp

import ignite
import ignite.contrib.handlers.tensorboard_logger
import ignite.contrib.handlers.tqdm_logger
import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric import seed_everything
from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GIN, MLP, global_add_pool


class Model(torch.nn.Module):
    def __init__(self, in_channels: int, out_channels: int,
                 hidden_channels: int = 64, num_layers: int = 3,
                 dropout: float = 0.5):
        super().__init__()

        self.gnn = GIN(in_channels, hidden_channels, num_layers,
                       dropout=dropout, jk='cat')

        self.classifier = MLP([hidden_channels, hidden_channels, out_channels],
                              norm="batch_norm", dropout=dropout)

    def forward(self, data):
        x = self.gnn(data.x, data.edge_index)
        x = global_add_pool(x, data.batch)
        x = self.classifier(x)
        return x


def main():
    seed_everything(42)

    root = osp.join('data', 'TUDataset')
    dataset = TUDataset(root, 'IMDB-BINARY', pre_transform=T.OneHotDegree(135))

    dataset = dataset.shuffle()
    test_dataset = dataset[:len(dataset) // 10]
    val_dataset = dataset[len(dataset) // 10:2 * len(dataset) // 10]
    train_dataset = dataset[2 * len(dataset) // 10:]

    train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True,
                              pin_memory=True)
    val_loader = DataLoader(val_dataset, batch_size=64, pin_memory=True)
    test_loader = DataLoader(test_dataset, batch_size=64, pin_memory=True)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = Model(dataset.num_node_features, dataset.num_classes).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    metrics = {'acc': ignite.metrics.Accuracy()}

    def prepare_batch_fn(batch, device, non_blocking):
        return (batch.to(device, non_blocking=non_blocking),
                batch.y.to(device, non_blocking=non_blocking))

    trainer = ignite.engine.create_supervised_trainer(
        model=model,
        optimizer=optimizer,
        loss_fn=F.cross_entropy,
        device=device,
        prepare_batch=prepare_batch_fn,
        output_transform=lambda x, y, y_pred, loss: loss.item(),
        amp_mode='amp',
    )

    # Progress bar for each epoch:
    pbar = ignite.contrib.handlers.tqdm_logger.ProgressBar()
    pbar.attach(trainer, output_transform=lambda x: {'loss': x})

    def log_metrics(evaluator, loader, tag):
        def logger(trainer):
            evaluator.run(loader)
            print(f'{tag:10} Epoch: {trainer.state.epoch:02d}, '
                  f'Acc: {evaluator.state.metrics["acc"]:.4f}')

        return logger

    train_evaluator = ignite.engine.create_supervised_evaluator(
        model=model,
        metrics=metrics,
        device=device,
        prepare_batch=prepare_batch_fn,
        output_transform=lambda x, y, y_pred: (y_pred, y),
        amp_mode='amp',
    )
    trainer.on(ignite.engine.Events.EPOCH_COMPLETED(every=1))(log_metrics(
        train_evaluator, train_loader, 'Training'))

    val_evaluator = ignite.engine.create_supervised_evaluator(
        model=model,
        metrics=metrics,
        device=device,
        prepare_batch=prepare_batch_fn,
        output_transform=lambda x, y, y_pred: (y_pred, y),
        amp_mode='amp',
    )
    trainer.on(ignite.engine.Events.EPOCH_COMPLETED(every=1))(log_metrics(
        val_evaluator, val_loader, 'Validation'))

    test_evaluator = ignite.engine.create_supervised_evaluator(
        model=model,
        metrics=metrics,
        device=device,
        prepare_batch=prepare_batch_fn,
        output_transform=lambda x, y, y_pred: (y_pred, y),
        amp_mode='amp',
    )
    trainer.on(ignite.engine.Events.EPOCH_COMPLETED(every=1))(log_metrics(
        test_evaluator, test_loader, 'Test'))

    # Save checkpoint of the model based on Accuracy on the validation set:
    checkpoint_handler = ignite.handlers.Checkpoint(
        {'model': model},
        'runs/gin',
        n_saved=2,
        score_name=list(metrics.keys())[0],
        filename_pattern='best-{global_step}-{score_name}-{score}.pt',
        global_step_transform=ignite.handlers.global_step_from_engine(trainer),
    )
    val_evaluator.add_event_handler(ignite.engine.Events.EPOCH_COMPLETED,
                                    checkpoint_handler)

    # Create a tensorboard logger to write logs:
    tb_logger = ignite.contrib.handlers.tensorboard_logger.TensorboardLogger(
        log_dir=osp.join('runs/example', 'tb_logs'))

    tb_logger.attach_output_handler(
        trainer, event_name=ignite.engine.Events.ITERATION_COMPLETED,
        tag='training', output_transform=lambda loss: {'loss_iteration': loss})
    tb_logger.attach_output_handler(
        trainer, event_name=ignite.engine.Events.EPOCH_COMPLETED,
        tag='training', output_transform=lambda loss: {'loss_epoch': loss})
    tb_logger.attach_output_handler(
        train_evaluator,
        event_name=ignite.engine.Events.EPOCH_COMPLETED,
        tag='training',
        metric_names='all',
        global_step_transform=ignite.handlers.global_step_from_engine(trainer),
    )
    tb_logger.attach_output_handler(
        val_evaluator,
        event_name=ignite.engine.Events.EPOCH_COMPLETED,
        tag='validation',
        metric_names='all',
        global_step_transform=ignite.handlers.global_step_from_engine(trainer),
    )
    tb_logger.attach_output_handler(
        test_evaluator,
        event_name=ignite.engine.Events.EPOCH_COMPLETED,
        tag='test',
        metric_names='all',
        global_step_transform=ignite.handlers.global_step_from_engine(trainer),
    )
    tb_logger.close()

    trainer.run(train_loader, max_epochs=50)


if __name__ == '__main__':
    main()
pytorch/examples/pytorch_lightning/000077500000000000000000000000001444262304300201215ustar00rootroot00000000000000pytorch/examples/pytorch_lightning/gin.py000066400000000000000000000062251444262304300212550ustar00rootroot00000000000000import os.path as osp

import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from torchmetrics import Accuracy

import torch_geometric.transforms as T
from torch_geometric.data.lightning import LightningDataset
from torch_geometric.datasets import TUDataset
from torch_geometric.nn import GIN, MLP, global_add_pool


class Model(pl.LightningModule):
    def __init__(self, in_channels: int, out_channels: int,
                 hidden_channels: int = 64, num_layers: int = 3,
                 dropout: float = 0.5):
        super().__init__()
        self.gnn = GIN(in_channels, hidden_channels, num_layers,
                       dropout=dropout, jk='cat')

        self.classifier = MLP([hidden_channels, hidden_channels, out_channels],
                              norm="batch_norm", dropout=dropout)

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(self, x, edge_index, batch):
        x = self.gnn(x, edge_index)
        x = global_add_pool(x, batch)
        x = self.classifier(x)
        return x

    def training_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index, data.batch)
        loss = F.cross_entropy(y_hat, data.y)
        self.train_acc(y_hat.softmax(dim=-1), data.y)
        self.log('train_acc', self.train_acc, prog_bar=True, on_step=False,
                 on_epoch=True)
        return loss

    def validation_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index, data.batch)
        self.val_acc(y_hat.softmax(dim=-1), data.y)
        self.log('val_acc', self.val_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def test_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index, data.batch)
        self.test_acc(y_hat.softmax(dim=-1), data.y)
        self.log('test_acc', self.test_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


if __name__ == '__main__':
    root = osp.join('data', 'TUDataset')
    dataset = TUDataset(root, 'IMDB-BINARY', pre_transform=T.OneHotDegree(135))

    dataset = dataset.shuffle()
    test_dataset = dataset[:len(dataset) // 10]
    val_dataset = dataset[len(dataset) // 10:2 * len(dataset) // 10]
    train_dataset = dataset[2 * len(dataset) // 10:]

    datamodule = LightningDataset(train_dataset, val_dataset, test_dataset,
                                  batch_size=64, num_workers=4)

    model = Model(dataset.num_node_features, dataset.num_classes)

    devices = torch.cuda.device_count()
    strategy = pl.strategies.DDPStrategy(accelerator='gpu')
    checkpoint = pl.callbacks.ModelCheckpoint(monitor='val_acc', save_top_k=1,
                                              mode='max')
    trainer = pl.Trainer(strategy=strategy, devices=devices, max_epochs=50,
                         log_every_n_steps=5, callbacks=[checkpoint])

    trainer.fit(model, datamodule)
    trainer.test(ckpt_path='best', datamodule=datamodule)
pytorch/examples/pytorch_lightning/graph_sage.py000066400000000000000000000056401444262304300226000ustar00rootroot00000000000000import os.path as osp

import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from torch.nn import BatchNorm1d
from torchmetrics import Accuracy

from torch_geometric.data.lightning import LightningNodeData
from torch_geometric.datasets import Reddit
from torch_geometric.nn import GraphSAGE


class Model(pl.LightningModule):
    def __init__(self, in_channels: int, out_channels: int,
                 hidden_channels: int = 256, num_layers: int = 2,
                 dropout: float = 0.5):
        super().__init__()
        self.gnn = GraphSAGE(in_channels, hidden_channels, num_layers,
                             out_channels, dropout=dropout,
                             norm=BatchNorm1d(hidden_channels))

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(self, x, edge_index):
        return self.gnn(x, edge_index)

    def training_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index)[:data.batch_size]
        y = data.y[:data.batch_size]
        loss = F.cross_entropy(y_hat, y)
        self.train_acc(y_hat.softmax(dim=-1), y)
        self.log('train_acc', self.train_acc, prog_bar=True, on_step=False,
                 on_epoch=True)
        return loss

    def validation_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index)[:data.batch_size]
        y = data.y[:data.batch_size]
        self.val_acc(y_hat.softmax(dim=-1), y)
        self.log('val_acc', self.val_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def test_step(self, data, batch_idx):
        y_hat = self(data.x, data.edge_index)[:data.batch_size]
        y = data.y[:data.batch_size]
        self.test_acc(y_hat.softmax(dim=-1), y)
        self.log('test_acc', self.test_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


if __name__ == '__main__':
    dataset = Reddit(osp.join('data', 'Reddit'))
    data = dataset[0]

    datamodule = LightningNodeData(
        data,
        input_train_nodes=data.train_mask,
        input_val_nodes=data.val_mask,
        input_test_nodes=data.test_mask,
        loader='neighbor',
        num_neighbors=[25, 10],
        batch_size=1024,
        num_workers=8,
    )

    model = Model(dataset.num_node_features, dataset.num_classes)

    strategy = pl.strategies.SingleDeviceStrategy('cuda:0')
    checkpoint = pl.callbacks.ModelCheckpoint(monitor='val_acc', save_top_k=1,
                                              mode='max')
    trainer = pl.Trainer(strategy=strategy, devices=1, max_epochs=20,
                         callbacks=[checkpoint])

    trainer.fit(model, datamodule)
    trainer.test(ckpt_path='best', datamodule=datamodule)
pytorch/examples/pytorch_lightning/relational_gnn.py000066400000000000000000000111231444262304300234650ustar00rootroot00000000000000import os.path as osp
from typing import Dict, List, Tuple

import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.callbacks import ModelCheckpoint
from torch import Tensor
from torchmetrics import Accuracy

import torch_geometric.transforms as T
from torch_geometric.data import Batch
from torch_geometric.data.lightning import LightningNodeData
from torch_geometric.datasets import OGB_MAG
from torch_geometric.nn import Linear, SAGEConv, to_hetero
from torch_geometric.typing import EdgeType, NodeType


class GNN(torch.nn.Module):
    def __init__(self, hidden_channels: int, out_channels: int,
                 dropout: float):
        super().__init__()
        self.dropout = dropout

        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), hidden_channels)
        self.lin = Linear(-1, out_channels)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, p=self.dropout, training=self.training)
        x = self.conv2(x, edge_index).relu()
        x = F.dropout(x, p=self.dropout, training=self.training)
        return self.lin(x)


class RelationalGNN(LightningModule):
    def __init__(
        self,
        metadata: Tuple[List[NodeType], List[EdgeType]],
        hidden_channels: int,
        out_channels: int,
        dropout: float,
    ):
        super().__init__()
        self.save_hyperparameters()

        model = GNN(hidden_channels, out_channels, dropout)
        # Convert the homogeneous GNN model to a heterogeneous variant in
        # which distinct parameters are learned for each node and edge type.
        self.model = to_hetero(model, metadata, aggr='sum')

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(
        self,
        x_dict: Dict[NodeType, Tensor],
        edge_index_dict: Dict[EdgeType, Tensor],
    ) -> Dict[NodeType, Tensor]:
        return self.model(x_dict, edge_index_dict)

    def common_step(self, batch: Batch) -> Tuple[Tensor, Tensor]:
        batch_size = batch['paper'].batch_size
        y_hat = self(batch.x_dict, batch.edge_index_dict)['paper'][:batch_size]
        y = batch['paper'].y[:batch_size]
        return y_hat, y

    def training_step(self, batch: Batch, batch_idx: int) -> Tensor:
        y_hat, y = self.common_step(batch)
        loss = F.cross_entropy(y_hat, y)
        self.train_acc(y_hat.softmax(dim=-1), y)
        self.log('train_acc', self.train_acc, prog_bar=True, on_step=False,
                 on_epoch=True)
        return loss

    def validation_step(self, batch: Batch, batch_idx: int):
        y_hat, y = self.common_step(batch)
        self.val_acc(y_hat.softmax(dim=-1), y)
        self.log('val_acc', self.val_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def test_step(self, batch: Batch, batch_idx: int):
        y_hat, y = self.common_step(batch)
        self.test_acc(y_hat.softmax(dim=-1), y)
        self.log('test_acc', self.test_acc, prog_bar=True, on_step=False,
                 on_epoch=True)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


def main():
    dataset = OGB_MAG(osp.join('data', 'OGB'), preprocess='metapath2vec',
                      transform=T.ToUndirected(merge=False))
    data = dataset[0]

    datamodule = LightningNodeData(
        data,
        input_train_nodes=('paper', data['paper'].train_mask),
        input_val_nodes=('paper', data['paper'].val_mask),
        input_test_nodes=('paper', data['paper'].test_mask),
        loader='neighbor',
        num_neighbors=[10, 10],
        batch_size=1024,
        num_workers=8,
    )

    model = RelationalGNN(data.metadata(), hidden_channels=64,
                          out_channels=349, dropout=0.0)

    with torch.no_grad():  # Run a dummy forward pass to initialize lazy model
        loader = datamodule.train_dataloader()
        batch = next(iter(loader))
        model.common_step(batch)

    strategy = pl.strategies.SingleDeviceStrategy('cuda:0')
    checkpoint = ModelCheckpoint(monitor='val_acc', save_top_k=1, mode='max')
    trainer = Trainer(strategy=strategy, devices=1, max_epochs=20,
                      log_every_n_steps=5, callbacks=[checkpoint])

    trainer.fit(model, datamodule)
    trainer.test(ckpt_path='best', datamodule=datamodule)


if __name__ == "__main__":
    main()
pytorch/examples/qm9_nn_conv.py000066400000000000000000000106611444262304300171720ustar00rootroot00000000000000import copy
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import GRU, Linear, ReLU, Sequential

import torch_geometric.transforms as T
from torch_geometric.datasets import QM9
from torch_geometric.loader import DataLoader
from torch_geometric.nn import NNConv, Set2Set
from torch_geometric.utils import remove_self_loops

target = 0
dim = 64


class MyTransform:
    def __call__(self, data):
        data = copy.copy(data)
        data.y = data.y[:, target]  # Specify target.
        return data


class Complete:
    def __call__(self, data):
        data = copy.copy(data)
        device = data.edge_index.device

        row = torch.arange(data.num_nodes, dtype=torch.long, device=device)
        col = torch.arange(data.num_nodes, dtype=torch.long, device=device)

        row = row.view(-1, 1).repeat(1, data.num_nodes).view(-1)
        col = col.repeat(data.num_nodes)
        edge_index = torch.stack([row, col], dim=0)

        edge_attr = None
        if data.edge_attr is not None:
            idx = data.edge_index[0] * data.num_nodes + data.edge_index[1]
            size = list(data.edge_attr.size())
            size[0] = data.num_nodes * data.num_nodes
            edge_attr = data.edge_attr.new_zeros(size)
            edge_attr[idx] = data.edge_attr

        edge_index, edge_attr = remove_self_loops(edge_index, edge_attr)
        data.edge_attr = edge_attr
        data.edge_index = edge_index

        return data


path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'QM9')
transform = T.Compose([MyTransform(), Complete(), T.Distance(norm=False)])
dataset = QM9(path, transform=transform).shuffle()

# Normalize targets to mean = 0 and std = 1.
mean = dataset.data.y.mean(dim=0, keepdim=True)
std = dataset.data.y.std(dim=0, keepdim=True)
dataset.data.y = (dataset.data.y - mean) / std
mean, std = mean[:, target].item(), std[:, target].item()

# Split datasets.
test_dataset = dataset[:10000]
val_dataset = dataset[10000:20000]
train_dataset = dataset[20000:]
test_loader = DataLoader(test_dataset, batch_size=128, shuffle=False)
val_loader = DataLoader(val_dataset, batch_size=128, shuffle=False)
train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin0 = torch.nn.Linear(dataset.num_features, dim)

        nn = Sequential(Linear(5, 128), ReLU(), Linear(128, dim * dim))
        self.conv = NNConv(dim, dim, nn, aggr='mean')
        self.gru = GRU(dim, dim)

        self.set2set = Set2Set(dim, processing_steps=3)
        self.lin1 = torch.nn.Linear(2 * dim, dim)
        self.lin2 = torch.nn.Linear(dim, 1)

    def forward(self, data):
        out = F.relu(self.lin0(data.x))
        h = out.unsqueeze(0)

        for i in range(3):
            m = F.relu(self.conv(out, data.edge_index, data.edge_attr))
            out, h = self.gru(m.unsqueeze(0), h)
            out = out.squeeze(0)

        out = self.set2set(out, data.batch)
        out = F.relu(self.lin1(out))
        out = self.lin2(out)
        return out.view(-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min',
                                                       factor=0.7, patience=5,
                                                       min_lr=0.00001)


def train(epoch):
    model.train()
    loss_all = 0

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = F.mse_loss(model(data), data.y)
        loss.backward()
        loss_all += loss.item() * data.num_graphs
        optimizer.step()
    return loss_all / len(train_loader.dataset)


def test(loader):
    model.eval()
    error = 0

    for data in loader:
        data = data.to(device)
        error += (model(data) * std - data.y * std).abs().sum().item()  # MAE
    return error / len(loader.dataset)


best_val_error = None
for epoch in range(1, 301):
    lr = scheduler.optimizer.param_groups[0]['lr']
    loss = train(epoch)
    val_error = test(val_loader)
    scheduler.step(val_error)

    if best_val_error is None or val_error <= best_val_error:
        test_error = test(test_loader)
        best_val_error = val_error

    print(f'Epoch: {epoch:03d}, LR: {lr:7f}, Loss: {loss:.7f}, '
          f'Val MAE: {val_error:.7f}, Test MAE: {test_error:.7f}')
pytorch/examples/qm9_pretrained_dimenet.py000066400000000000000000000031041444262304300213660ustar00rootroot00000000000000import argparse
import os.path as osp

import torch

from torch_geometric.datasets import QM9
from torch_geometric.loader import DataLoader
from torch_geometric.nn import DimeNet, DimeNetPlusPlus

parser = argparse.ArgumentParser()
parser.add_argument('--use_dimenet_plus_plus', action='store_true')
args = parser.parse_args()

Model = DimeNetPlusPlus if args.use_dimenet_plus_plus else DimeNet

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'QM9')
dataset = QM9(path)

# DimeNet uses the atomization energy for targets U0, U, H, and G, i.e.:
# 7 -> 12, 8 -> 13, 9 -> 14, 10 -> 15
idx = torch.tensor([0, 1, 2, 3, 4, 5, 6, 12, 13, 14, 15, 11])
dataset.data.y = dataset.data.y[:, idx]

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

for target in range(12):
    # Skip target \delta\epsilon, since it can be computed via
    # \epsilon_{LUMO} - \epsilon_{HOMO}:
    if target == 4:
        continue

    model, datasets = Model.from_qm9_pretrained(path, dataset, target)
    train_dataset, val_dataset, test_dataset = datasets

    model = model.to(device)
    loader = DataLoader(test_dataset, batch_size=256)

    maes = []
    for data in loader:
        data = data.to(device)
        with torch.no_grad():
            pred = model(data.z, data.pos, data.batch)
        mae = (pred.view(-1) - data.y[:, target]).abs()
        maes.append(mae)

    mae = torch.cat(maes, dim=0)

    # Report meV instead of eV:
    mae = 1000 * mae if target in [2, 3, 4, 6, 7, 8, 9, 10] else mae

    print(f'Target: {target:02d}, MAE: {mae.mean():.5f} Β± {mae.std():.5f}')
pytorch/examples/qm9_pretrained_schnet.py000066400000000000000000000023461444262304300212340ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
from tqdm import tqdm

from torch_geometric.datasets import QM9
from torch_geometric.loader import DataLoader
from torch_geometric.nn import SchNet

parser = argparse.ArgumentParser()
parser.add_argument('--cutoff', type=float, default=10.0,
                    help='Cutoff distance for interatomic interactions')
args = parser.parse_args()

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'QM9')
dataset = QM9(path)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

for target in range(12):
    model, datasets = SchNet.from_qm9_pretrained(path, dataset, target)
    train_dataset, val_dataset, test_dataset = datasets

    model = model.to(device)
    loader = DataLoader(test_dataset, batch_size=256)

    maes = []
    for data in tqdm(loader):
        data = data.to(device)
        with torch.no_grad():
            pred = model(data.z, data.pos, data.batch)
        mae = (pred.view(-1) - data.y[:, target]).abs()
        maes.append(mae)

    mae = torch.cat(maes, dim=0)

    # Report meV instead of eV.
    mae = 1000 * mae if target in [2, 3, 4, 6, 7, 8, 9, 10] else mae

    print(f'Target: {target:02d}, MAE: {mae.mean():.5f} Β± {mae.std():.5f}')
pytorch/examples/quiver/000077500000000000000000000000001444262304300157015ustar00rootroot00000000000000pytorch/examples/quiver/README.md000066400000000000000000000023771444262304300171710ustar00rootroot00000000000000# Using Quiver for PyG Examples

**[Quiver](https://github.com/quiver-team/torch-quiver)** is a **GPU-optimized distributed library** for PyG.
It can speed up graph sampling and feature aggregation through GPU when running PyG examples.

## Installation

Assuming you have installed PyTorch and PyG, you can install Quiver as follows:

```bash
pip install torch-quiver>=0.1.1
```

## Usage

The API and design documentation of Quiver can be found [here](https://github.com/quiver-team/torch-quiver).

## Examples

We provide several examples to showcase the usage of Quiver within PyG:

### Single-GPU Training

The single-GPU example leverages Quiver's ability of **(i)** GPU-based graph sampling and feature aggregation, and **(ii)** GNN data caching algorithm (which cache hot data in GPU memory) while enabling fast access to CPU data using a Quiver shared tensor implementation:

```bash
python single_gpu_quiver.py
```

### Multi-GPU Training

The multi-GPU example further leverages Quiver's ability of **(i)** distributing sampling and feature aggregation to multiple GPUs, and **(ii)** using multi-GPU memories to cache and replicate hot GNN data:

```bash
python multi_gpu_quiver.py
```

### Distributed Training

A Quiver-based distributed PyG example is coming soon.
pytorch/examples/quiver/multi_gpu_quiver.py000066400000000000000000000127561444262304300216660ustar00rootroot00000000000000# This script shows how to use Quiver in an existing PyG example:
# https://github.com/pyg-team/pytorch_geometric/blob/master/examples/multi_gpu/distributed_sampling.py
import os

import quiver
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn.functional as F
from torch.nn.parallel import DistributedDataParallel
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborSampler
from torch_geometric.nn import SAGEConv


class SAGE(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels,
                 num_layers=2):
        super().__init__()
        self.num_layers = num_layers

        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        for _ in range(self.num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x, adjs):
        for i, (edge_index, _, size) in enumerate(adjs):
            x_target = x[:size[1]]  # Target nodes are always placed first.
            x = self.convs[i]((x, x_target), edge_index)
            if i != self.num_layers - 1:
                x = F.relu(x)
                x = F.dropout(x, p=0.5, training=self.training)
        return x.log_softmax(dim=-1)

    @torch.no_grad()
    def inference(self, x_all, device, subgraph_loader):
        pbar = tqdm(total=x_all.size(0) * self.num_layers)
        pbar.set_description('Evaluating')

        for i in range(self.num_layers):
            xs = []
            for batch_size, n_id, adj in subgraph_loader:
                edge_index, _, size = adj.to(device)
                x = x_all[n_id].to(device)
                x_target = x[:size[1]]
                x = self.convs[i]((x, x_target), edge_index)
                if i != self.num_layers - 1:
                    x = F.relu(x)
                xs.append(x.cpu())

                pbar.update(batch_size)

            x_all = torch.cat(xs, dim=0)

        pbar.close()

        return x_all


def run(rank, world_size, dataset, quiver_feature, quiver_sampler):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group('nccl', rank=rank, world_size=world_size)
    torch.cuda.set_device(rank)

    data = dataset[0]
    train_idx = data.train_mask.nonzero(as_tuple=False).view(-1)
    train_idx = train_idx.split(train_idx.size(0) // world_size)[rank]

    train_loader = torch.utils.data.DataLoader(train_idx, batch_size=1024,
                                               shuffle=True, num_workers=0)

    if rank == 0:
        subgraph_loader = NeighborSampler(data.edge_index, node_idx=None,
                                          sizes=[-1], batch_size=2048,
                                          shuffle=False, num_workers=6)

    torch.manual_seed(12345)
    model = SAGE(dataset.num_features, 256, dataset.num_classes).to(rank)
    model = DistributedDataParallel(model, device_ids=[rank])
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

    y = data.y.to(rank)

    for epoch in range(1, 21):
        model.train()

        for seeds in train_loader:
            n_id, batch_size, adjs = quiver_sampler.sample(seeds)
            adjs = [adj.to(rank) for adj in adjs]

            optimizer.zero_grad()
            out = model(quiver_feature[n_id].to(rank), adjs)
            loss = F.nll_loss(out, y[n_id[:batch_size]])
            loss.backward()
            optimizer.step()

        dist.barrier()

        if rank == 0:
            print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')

        if rank == 0 and epoch % 5 == 0:  # We evaluate on a single GPU for now
            model.eval()
            with torch.no_grad():
                out = model.module.inference(quiver_feature, rank,
                                             subgraph_loader)
            res = out.argmax(dim=-1) == data.y
            acc1 = int(res[data.train_mask].sum()) / int(data.train_mask.sum())
            acc2 = int(res[data.val_mask].sum()) / int(data.val_mask.sum())
            acc3 = int(res[data.test_mask].sum()) / int(data.test_mask.sum())
            print(f'Train: {acc1:.4f}, Val: {acc2:.4f}, Test: {acc3:.4f}')

        dist.barrier()

    dist.destroy_process_group()


if __name__ == '__main__':
    dataset = Reddit('../../data/Reddit')
    data = dataset[0]
    world_size = torch.cuda.device_count()
    print('Let\'s use', world_size, 'GPUs!')

    ########################################################################
    # The below code enable Quiver for PyG.
    # Please refer to: https://torch-quiver.readthedocs.io/en/latest/api/ for
    # how to configure the CSRTopo, Sampler and Feature of Quiver.
    ########################################################################
    csr_topo = quiver.CSRTopo(data.edge_index)  # Quiver
    quiver_sampler = quiver.pyg.GraphSageSampler(csr_topo, [25, 10], 0,
                                                 mode='GPU')  # Quiver
    quiver_feature = quiver.Feature(rank=0,
                                    device_list=list(range(world_size)),
                                    device_cache_size="2G",
                                    cache_policy="device_replicate",
                                    csr_topo=csr_topo)  # Quiver
    quiver_feature.from_cpu_tensor(data.x)  # Quiver

    mp.spawn(run, args=(world_size, dataset, quiver_feature, quiver_sampler),
             nprocs=world_size, join=True)
pytorch/examples/quiver/single_gpu_quiver.py000066400000000000000000000134151444262304300220060ustar00rootroot00000000000000# This script shows how to use Quiver in an existing PyG example:
# https://github.com/pyg-team/pytorch_geometric/blob/master/examples/reddit.py
import os.path as osp

import quiver
import torch
import torch.nn.functional as F
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborSampler
from torch_geometric.nn import SAGEConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Reddit')
dataset = Reddit(path)
data = dataset[0]

train_idx = data.train_mask.nonzero(as_tuple=False).view(-1)

################################
# Step 1: Using Quiver's sampler
################################

train_loader = torch.utils.data.DataLoader(train_idx, batch_size=1024,
                                           shuffle=True,
                                           drop_last=True)  # Quiver
########################################################################
# The below code enable Quiver for PyG.
# Please refer to: https://torch-quiver.readthedocs.io/en/latest/api/ for
# how to configure the CSRTopo, Sampler and Feature of Quiver.
########################################################################
csr_topo = quiver.CSRTopo(data.edge_index)  # Quiver
quiver_sampler = quiver.pyg.GraphSageSampler(csr_topo, sizes=[25, 10],
                                             device=0, mode='GPU')  # Quiver

subgraph_loader = NeighborSampler(data.edge_index, node_idx=None, sizes=[-1],
                                  batch_size=1024, shuffle=False,
                                  num_workers=12)


class SAGE(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()

        self.num_layers = 2

        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x, adjs):
        # `train_loader` computes the k-hop neighborhood of a batch of nodes,
        # and returns, for each layer, a bipartite graph object, holding the
        # bipartite edges `edge_index`, the index `e_id` of the original edges,
        # and the size/shape `size` of the bipartite graph.
        # Target nodes are also included in the source nodes so that one can
        # easily apply skip-connections or add self-loops.
        for i, (edge_index, _, size) in enumerate(adjs):
            x_target = x[:size[1]]  # Target nodes are always placed first.
            x = self.convs[i]((x, x_target), edge_index)
            if i != self.num_layers - 1:
                x = F.relu(x)
                x = F.dropout(x, p=0.5, training=self.training)
        return x.log_softmax(dim=-1)

    def inference(self, x_all):
        pbar = tqdm(total=x_all.size(0) * self.num_layers)
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch.
        for i in range(self.num_layers):
            xs = []
            for batch_size, n_id, adj in subgraph_loader:
                edge_index, _, size = adj.to(device)
                x = x_all[n_id].to(device)
                x_target = x[:size[1]]
                x = self.convs[i]((x, x_target), edge_index)
                if i != self.num_layers - 1:
                    x = F.relu(x)
                xs.append(x.cpu())

                pbar.update(batch_size)

            x_all = torch.cat(xs, dim=0)

        pbar.close()

        return x_all


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = SAGE(dataset.num_features, 256, dataset.num_classes)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

################################
# Step 2: Using Quiver's Feature
################################
x = quiver.Feature(rank=0, device_list=[0], device_cache_size="4G",
                   cache_policy="device_replicate",
                   csr_topo=csr_topo)  # Quiver
x.from_cpu_tensor(data.x)  # Quiver

y = data.y.squeeze().to(device)


def train(epoch):
    model.train()

    pbar = tqdm(total=int(data.train_mask.sum()))
    pbar.set_description(f'Epoch {epoch:02d}')

    total_loss = total_correct = 0
    ############################################
    # Step 3: Training the PyG Model with Quiver
    ############################################
    # for batch_size, n_id, adjs in train_loader: # Original PyG Code
    for seeds in train_loader:  # Quiver
        n_id, batch_size, adjs = quiver_sampler.sample(seeds)  # Quiver
        # `adjs` holds a list of `(edge_index, e_id, size)` tuples.
        adjs = [adj.to(device) for adj in adjs]

        optimizer.zero_grad()
        out = model(x[n_id], adjs)
        loss = F.nll_loss(out, y[n_id[:batch_size]])
        loss.backward()
        optimizer.step()

        total_loss += float(loss)
        total_correct += int(out.argmax(dim=-1).eq(y[n_id[:batch_size]]).sum())
        pbar.update(batch_size)

    pbar.close()

    loss = total_loss / len(train_loader)
    approx_acc = total_correct / int(data.train_mask.sum())

    return loss, approx_acc


@torch.no_grad()
def test():
    model.eval()

    out = model.inference(x)

    y_true = y.cpu().unsqueeze(-1)
    y_pred = out.argmax(dim=-1, keepdim=True)

    results = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        results += [int(y_pred[mask].eq(y_true[mask]).sum()) / int(mask.sum())]

    return results


for epoch in range(1, 11):
    loss, acc = train(epoch)
    print(f'Epoch {epoch:02d}, Loss: {loss:.4f}, Approx. Train: {acc:.4f}')
    train_acc, val_acc, test_acc = test()
    print(f'Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/randlanet_classification.py000066400000000000000000000201171444262304300217640ustar00rootroot00000000000000"""
An adaptation of RandLA-Net to the classification task, which was not
addressed in the paper:
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Reference: https://arxiv.org/abs/1911.11236
"""
import os.path as osp

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.datasets import ModelNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP
from torch_geometric.nn.aggr import MaxAggregation
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.pool import knn_graph
from torch_geometric.nn.pool.decimation import decimation_indices
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import softmax

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

# Default activation and batch norm parameters used by RandLA-Net:
lrelu02_kwargs = {'negative_slope': 0.2}
bn099_kwargs = {'momentum': 0.01, 'eps': 1e-6}


class SharedMLP(MLP):
    """SharedMLP following RandLA-Net paper."""
    def __init__(self, *args, **kwargs):
        # BN + Act always active even at last layer.
        kwargs['plain_last'] = False
        # LeakyRelu with 0.2 slope by default.
        kwargs['act'] = kwargs.get('act', 'LeakyReLU')
        kwargs['act_kwargs'] = kwargs.get('act_kwargs', lrelu02_kwargs)
        # BatchNorm with 1 - 0.99 = 0.01 momentum
        # and 1e-6 eps by defaut (tensorflow momentum != pytorch momentum)
        kwargs['norm_kwargs'] = kwargs.get('norm_kwargs', bn099_kwargs)
        super().__init__(*args, **kwargs)


class LocalFeatureAggregation(MessagePassing):
    """Positional encoding of points in a neighborhood."""
    def __init__(self, channels):
        super().__init__(aggr='add')
        self.mlp_encoder = SharedMLP([10, channels // 2])
        self.mlp_attention = SharedMLP([channels, channels], bias=False,
                                       act=None, norm=None)
        self.mlp_post_attention = SharedMLP([channels, channels])

    def forward(self, edge_index, x, pos):
        out = self.propagate(edge_index, x=x, pos=pos)  # N, d_out
        out = self.mlp_post_attention(out)  # N, d_out
        return out

    def message(self, x_j: Tensor, pos_i: Tensor, pos_j: Tensor,
                index: Tensor) -> Tensor:
        """Local Spatial Encoding (locSE) and attentive pooling of features.

        Args:
            x_j (Tensor): neighboors features (K,d)
            pos_i (Tensor): centroid position (repeated) (K,3)
            pos_j (Tensor): neighboors positions (K,3)
            index (Tensor): index of centroid positions
                (e.g. [0,...,0,1,...,1,...,N,...,N])

        returns:
            (Tensor): locSE weighted by feature attention scores.

        """
        # Encode local neighboorhod structural information
        pos_diff = pos_j - pos_i
        distance = torch.sqrt((pos_diff * pos_diff).sum(1, keepdim=True))
        relative_infos = torch.cat([pos_i, pos_j, pos_diff, distance],
                                   dim=1)  # N * K, d
        local_spatial_encoding = self.mlp_encoder(relative_infos)  # N * K, d
        local_features = torch.cat([x_j, local_spatial_encoding],
                                   dim=1)  # N * K, 2d

        # Attention will weight the different features of x
        # along the neighborhood dimension.
        att_features = self.mlp_attention(local_features)  # N * K, d_out
        att_scores = softmax(att_features, index=index)  # N * K, d_out

        return att_scores * local_features  # N * K, d_out


class DilatedResidualBlock(torch.nn.Module):
    def __init__(
        self,
        num_neighbors,
        d_in: int,
        d_out: int,
    ):
        super().__init__()
        self.num_neighbors = num_neighbors
        self.d_in = d_in
        self.d_out = d_out

        # MLP on input
        self.mlp1 = SharedMLP([d_in, d_out // 8])
        # MLP on input, and the result is summed with the output of mlp2
        self.shortcut = SharedMLP([d_in, d_out], act=None)
        # MLP on output
        self.mlp2 = SharedMLP([d_out // 2, d_out], act=None)

        self.lfa1 = LocalFeatureAggregation(d_out // 4)
        self.lfa2 = LocalFeatureAggregation(d_out // 2)

        self.lrelu = torch.nn.LeakyReLU(**lrelu02_kwargs)

    def forward(self, x, pos, batch):
        edge_index = knn_graph(pos, self.num_neighbors, batch=batch, loop=True)

        shortcut_of_x = self.shortcut(x)  # N, d_out
        x = self.mlp1(x)  # N, d_out//8
        x = self.lfa1(edge_index, x, pos)  # N, d_out//2
        x = self.lfa2(edge_index, x, pos)  # N, d_out//2
        x = self.mlp2(x)  # N, d_out
        x = self.lrelu(x + shortcut_of_x)  # N, d_out

        return x, pos, batch


def decimate(tensors, ptr: Tensor, decimation_factor: int):
    """Decimates each element of the given tuple of tensors."""
    idx_decim, ptr_decim = decimation_indices(ptr, decimation_factor)
    tensors_decim = tuple(tensor[idx_decim] for tensor in tensors)
    return tensors_decim, ptr_decim


class Net(torch.nn.Module):
    def __init__(
        self,
        num_features,
        num_classes,
        decimation: int = 4,
        num_neighboors: int = 16,
        return_logits: bool = False,
    ):
        super().__init__()
        self.decimation = decimation
        # An option to return logits instead of log probabilities:
        self.return_logits = return_logits
        self.fc0 = Linear(in_features=num_features, out_features=8)
        # 2 DilatedResidualBlock converges better than 4 on ModelNet.
        self.block1 = DilatedResidualBlock(num_neighboors, 8, 32)
        self.block2 = DilatedResidualBlock(num_neighboors, 32, 128)
        self.mlp1 = SharedMLP([128, 128])
        self.max_agg = MaxAggregation()
        self.mlp_classif = SharedMLP([128, 32], dropout=[0.5])
        self.fc_classif = Linear(32, num_classes)

    def forward(self, x, pos, batch, ptr):
        x = x if x is not None else pos
        b1 = self.block1(self.fc0(x), pos, batch)
        b1_decimated, ptr1 = decimate(b1, ptr, self.decimation)

        b2 = self.block2(*b1_decimated)
        b2_decimated, _ = decimate(b2, ptr1, self.decimation)

        x = self.mlp1(b2_decimated[0])
        x = self.max_agg(x, b2_decimated[2])

        x = self.mlp_classif(x)
        logits = self.fc_classif(x)

        return logits if self.return_logits else logits.log_softmax(dim=-1)


def train(epoch):
    model.train()

    total_loss = 0
    for data in tqdm(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.pos, data.batch, data.ptr)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += data.num_graphs * float(loss)
    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    correct = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.pos, data.batch, data.ptr)
        correct += int((out.argmax(dim=-1) == data.y).sum())
    return correct / len(loader.dataset)


if __name__ == '__main__':
    path = osp.dirname(osp.realpath(__file__))
    path = osp.join(path, '..', 'data/ModelNet10')
    pre_transform, transform = T.NormalizeScale(), T.SamplePoints(1024)
    train_dataset = ModelNet(path, '10', True, transform, pre_transform)
    test_dataset = ModelNet(path, '10', False, transform, pre_transform)
    train_loader = DataLoader(train_dataset, 32, shuffle=True, num_workers=6)
    test_loader = DataLoader(test_dataset, 32, shuffle=False, num_workers=6)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = Net(3, train_dataset.num_classes).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20,
                                                gamma=0.5)

    for epoch in range(1, 201):
        loss = train(epoch)
        test_acc = test(test_loader)
        print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Test: {test_acc:.4f}')
        scheduler.step()
pytorch/examples/randlanet_segmentation.py000066400000000000000000000150701444262304300214700ustar00rootroot00000000000000"""
An implementation of RandLA-Net based on the paper:
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Reference: https://arxiv.org/abs/1911.11236
"""
import os.path as osp

import torch
import torch.nn.functional as F
from randlanet_classification import DilatedResidualBlock, SharedMLP, decimate
from torch.nn import Linear
from torchmetrics.functional import jaccard_index
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.datasets import ShapeNet
from torch_geometric.loader import DataLoader
from torch_geometric.nn import knn_interpolate
from torch_geometric.typing import WITH_TORCH_CLUSTER
from torch_geometric.utils import scatter

if not WITH_TORCH_CLUSTER:
    quit("This example requires 'torch-cluster'")

category = 'Airplane'  # Pass in `None` to train on all categories.
category_num_classes = 4  # 4 for Airplane - see ShapeNet for details
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')

transform = T.Compose([
    T.RandomJitter(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2),
])
pre_transform = T.NormalizeScale()

train_dataset = ShapeNet(
    path,
    category,
    split='trainval',
    transform=transform,
    pre_transform=pre_transform,
)
test_dataset = ShapeNet(
    path,
    category,
    split='test',
    pre_transform=pre_transform,
)

train_loader = DataLoader(train_dataset, 12, shuffle=True, num_workers=6)
test_loader = DataLoader(test_dataset, 12, shuffle=False, num_workers=6)


class FPModule(torch.nn.Module):
    """Upsampling with a skip connection."""
    def __init__(self, k, nn):
        super().__init__()
        self.k = k
        self.nn = nn

    def forward(self, x, pos, batch, x_skip, pos_skip, batch_skip):
        x = knn_interpolate(x, pos, pos_skip, batch, batch_skip, k=self.k)
        x = torch.cat([x, x_skip], dim=1)
        x = self.nn(x)
        return x, pos_skip, batch_skip


class Net(torch.nn.Module):
    def __init__(
        self,
        num_features: int,
        num_classes: int,
        decimation: int = 4,
        num_neighbors: int = 16,
        return_logits: bool = False,
    ):
        super().__init__()

        self.decimation = decimation
        # An option to return logits instead of log probabilities:
        self.return_logits = return_logits

        # Authors use 8, which is a bottleneck
        # for the final MLP, and also when num_classes>8
        # or num_features>8.
        d_bottleneck = max(32, num_classes, num_features)

        self.fc0 = Linear(num_features, d_bottleneck)
        self.block1 = DilatedResidualBlock(num_neighbors, d_bottleneck, 32)
        self.block2 = DilatedResidualBlock(num_neighbors, 32, 128)
        self.block3 = DilatedResidualBlock(num_neighbors, 128, 256)
        self.block4 = DilatedResidualBlock(num_neighbors, 256, 512)
        self.mlp_summit = SharedMLP([512, 512])
        self.fp4 = FPModule(1, SharedMLP([512 + 256, 256]))
        self.fp3 = FPModule(1, SharedMLP([256 + 128, 128]))
        self.fp2 = FPModule(1, SharedMLP([128 + 32, 32]))
        self.fp1 = FPModule(1, SharedMLP([32 + 32, d_bottleneck]))
        self.mlp_classif = SharedMLP([d_bottleneck, 64, 32],
                                     dropout=[0.0, 0.5])
        self.fc_classif = Linear(32, num_classes)

    def forward(self, x, pos, batch, ptr):
        x = x if x is not None else pos

        b1_out = self.block1(self.fc0(x), pos, batch)
        b1_out_decimated, ptr1 = decimate(b1_out, ptr, self.decimation)

        b2_out = self.block2(*b1_out_decimated)
        b2_out_decimated, ptr2 = decimate(b2_out, ptr1, self.decimation)

        b3_out = self.block3(*b2_out_decimated)
        b3_out_decimated, ptr3 = decimate(b3_out, ptr2, self.decimation)

        b4_out = self.block4(*b3_out_decimated)
        b4_out_decimated, _ = decimate(b4_out, ptr3, self.decimation)

        mlp_out = (
            self.mlp_summit(b4_out_decimated[0]),
            b4_out_decimated[1],
            b4_out_decimated[2],
        )

        fp4_out = self.fp4(*mlp_out, *b3_out_decimated)
        fp3_out = self.fp3(*fp4_out, *b2_out_decimated)
        fp2_out = self.fp2(*fp3_out, *b1_out_decimated)
        fp1_out = self.fp1(*fp2_out, *b1_out)

        x = self.mlp_classif(fp1_out[0])
        logits = self.fc_classif(x)

        if self.return_logits:
            return logits

        probas = logits.log_softmax(dim=-1)
        return probas


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(3, category_num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = correct_nodes = total_nodes = 0
    for i, data in tqdm(enumerate(train_loader)):
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.pos, data.batch, data.ptr)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        correct_nodes += out.argmax(dim=1).eq(data.y).sum().item()
        total_nodes += data.num_nodes

        if (i + 1) % 10 == 0:
            print(f'[{i+1}/{len(train_loader)}] Loss: {total_loss / 10:.4f} '
                  f'Train Acc: {correct_nodes / total_nodes:.4f}')
            total_loss = correct_nodes = total_nodes = 0


@torch.no_grad()
def test(loader):
    model.eval()

    ious, categories = [], []
    y_map = torch.empty(loader.dataset.num_classes, device=device).long()
    for data in loader:
        data = data.to(device)
        outs = model(data.x, data.pos, data.batch, data.ptr)

        sizes = (data.ptr[1:] - data.ptr[:-1]).tolist()
        for out, y, category in zip(outs.split(sizes), data.y.split(sizes),
                                    data.category.tolist()):
            category = list(ShapeNet.seg_classes.keys())[category]
            part = ShapeNet.seg_classes[category]
            part = torch.tensor(part, device=device)

            y_map[part] = torch.arange(part.size(0), device=device)

            iou = jaccard_index(
                out[:, part].argmax(dim=-1),
                y_map[y],
                num_classes=part.size(0),
                absent_score=1.0,
            )
            ious.append(iou)

        categories.append(data.category)

    iou = torch.tensor(ious, device=device)
    category = torch.cat(categories, dim=0)

    mean_iou = scatter(iou, category, reduce='mean')  # Per-category IoU.
    return float(mean_iou.mean())  # Global IoU.


for epoch in range(1, 31):
    train()
    iou = test(test_loader)
    print(f'Epoch: {epoch:02d}, Test IoU: {iou:.4f}')
pytorch/examples/rect.py000066400000000000000000000051431444262304300157000ustar00rootroot00000000000000import argparse
import copy
import os.path as osp

import torch
from sklearn.linear_model import LogisticRegression

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import RECT_L

# RECT focuses on the zero-shot, i.e. completely-imbalanced label setting:
# For this, we first remove "unseen" classes from the training set and train a
# RECT (or more specifically its supervised part RECT-L) model in the zero-shot
# label scenario. Lastly, we train a simple classifier to evaluate the final
# performance of the embeddings based on the original labels.

# Datasets              Citeseer             Cora          Pubmed
# Unseen Classes  [1, 2, 5]  [3, 4]  [1, 2, 3]  [3, 4, 6]  [2]
# RECT-L          66.30      68.20   74.60      71.20      75.30
# GCN             51.80      55.70   55.80      57.10      59.80
# NodeFeats       61.40      61.40   57.50      57.50      73.10

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='Cora',
                    choices=['Cora', 'CiteSeer', 'PubMed'])
parser.add_argument('--unseen-classes', type=int, nargs='*', default=[1, 2, 3])
args = parser.parse_args()

path = osp.join(osp.dirname(osp.realpath(__file__)), '../data/Planetoid')
train_mask_original = Planetoid(path, args.dataset)[0].train_mask.clone()
transform = T.Compose([
    T.NormalizeFeatures(),
    T.SVDFeatureReduction(200),
    T.GDC(),
])
dataset = Planetoid(path, args.dataset, transform=transform)
data = dataset[0]
zs_data = T.RemoveTrainingClasses(args.unseen_classes)(copy.copy(data))

model = RECT_L(200, 200, normalize=False, dropout=0.0)
zs_data.y = model.get_semantic_labels(zs_data.x, zs_data.y, zs_data.train_mask)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, zs_data = model.to(device), zs_data.to(device)

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=5e-4)

model.train()
for epoch in range(1, 201):
    optimizer.zero_grad()
    out = model(zs_data.x, zs_data.edge_index, zs_data.edge_attr)
    loss = criterion(out[zs_data.train_mask], zs_data.y)
    loss.backward()
    optimizer.step()
    print(f'Epoch {epoch:03d}, Loss {loss:.4f}')

model.eval()
with torch.no_grad():
    h = model.embed(zs_data.x, zs_data.edge_index, zs_data.edge_attr).cpu()

reg = LogisticRegression()
reg.fit(h[data.train_mask].numpy(), data.y[data.train_mask].numpy())
test_acc = reg.score(h[data.test_mask].numpy(), data.y[data.test_mask].numpy())
print(f'Test Acc: {test_acc:.4f}')
pytorch/examples/reddit.py000066400000000000000000000100151444262304300162100ustar00rootroot00000000000000import copy
import os.path as osp

import torch
import torch.nn.functional as F
from tqdm import tqdm

from torch_geometric.datasets import Reddit
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import SAGEConv

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Reddit')
dataset = Reddit(path)

# Already send node features/labels to GPU for faster access during sampling:
data = dataset[0].to(device, 'x', 'y')

kwargs = {'batch_size': 1024, 'num_workers': 6, 'persistent_workers': True}
train_loader = NeighborLoader(data, input_nodes=data.train_mask,
                              num_neighbors=[25, 10], shuffle=True, **kwargs)

subgraph_loader = NeighborLoader(copy.copy(data), input_nodes=None,
                                 num_neighbors=[-1], shuffle=False, **kwargs)

# No need to maintain these features during evaluation:
del subgraph_loader.data.x, subgraph_loader.data.y
# Add global node index information.
subgraph_loader.data.num_nodes = data.num_nodes
subgraph_loader.data.n_id = torch.arange(data.num_nodes)


class SAGE(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x, edge_index):
        for i, conv in enumerate(self.convs):
            x = conv(x, edge_index)
            if i < len(self.convs) - 1:
                x = x.relu_()
                x = F.dropout(x, p=0.5, training=self.training)
        return x

    @torch.no_grad()
    def inference(self, x_all, subgraph_loader):
        pbar = tqdm(total=len(subgraph_loader.dataset) * len(self.convs))
        pbar.set_description('Evaluating')

        # Compute representations of nodes layer by layer, using *all*
        # available edges. This leads to faster computation in contrast to
        # immediately computing the final representations of each batch:
        for i, conv in enumerate(self.convs):
            xs = []
            for batch in subgraph_loader:
                x = x_all[batch.n_id.to(x_all.device)].to(device)
                x = conv(x, batch.edge_index.to(device))
                if i < len(self.convs) - 1:
                    x = x.relu_()
                xs.append(x[:batch.batch_size].cpu())
                pbar.update(batch.batch_size)
            x_all = torch.cat(xs, dim=0)
        pbar.close()
        return x_all


model = SAGE(dataset.num_features, 256, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train(epoch):
    model.train()

    pbar = tqdm(total=int(len(train_loader.dataset)))
    pbar.set_description(f'Epoch {epoch:02d}')

    total_loss = total_correct = total_examples = 0
    for batch in train_loader:
        optimizer.zero_grad()
        y = batch.y[:batch.batch_size]
        y_hat = model(batch.x, batch.edge_index.to(device))[:batch.batch_size]
        loss = F.cross_entropy(y_hat, y)
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * batch.batch_size
        total_correct += int((y_hat.argmax(dim=-1) == y).sum())
        total_examples += batch.batch_size
        pbar.update(batch.batch_size)
    pbar.close()

    return total_loss / total_examples, total_correct / total_examples


@torch.no_grad()
def test():
    model.eval()
    y_hat = model.inference(data.x, subgraph_loader).argmax(dim=-1)
    y = data.y.to(y_hat.device)

    accs = []
    for mask in [data.train_mask, data.val_mask, data.test_mask]:
        accs.append(int((y_hat[mask] == y[mask]).sum()) / int(mask.sum()))
    return accs


for epoch in range(1, 11):
    loss, acc = train(epoch)
    print(f'Epoch {epoch:02d}, Loss: {loss:.4f}, Approx. Train: {acc:.4f}')
    train_acc, val_acc, test_acc = test()
    print(f'Epoch: {epoch:02d}, Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/renet.py000066400000000000000000000050451444262304300160610ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import GDELT, ICEWS18  # noqa
from torch_geometric.loader import DataLoader
from torch_geometric.nn.models.re_net import RENet

seq_len = 10

# Load the dataset and precompute history objects.
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ICEWS18')
train_dataset = ICEWS18(path, pre_transform=RENet.pre_transform(seq_len))
test_dataset = ICEWS18(path, split='test')

# path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'GDELT')
# train_dataset = GDELT(path, pre_transform=RENet.pre_transform(seq_len))
# test_dataset = ICEWS18(path, split='test')

# Create dataloader for training and test dataset.
train_loader = DataLoader(train_dataset, batch_size=1024, shuffle=True,
                          follow_batch=['h_sub', 'h_obj'], num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=1024, shuffle=False,
                         follow_batch=['h_sub', 'h_obj'], num_workers=6)

# Initialize model and optimizer.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = RENet(
    train_dataset.num_nodes,
    train_dataset.num_rels,
    hidden_channels=200,
    seq_len=seq_len,
    dropout=0.5,
).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001,
                             weight_decay=0.00001)


def train():
    model.train()

    # Train model via multi-class classification against the corresponding
    # object and subject entities.
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        log_prob_obj, log_prob_sub = model(data)
        loss_obj = F.nll_loss(log_prob_obj, data.obj)
        loss_sub = F.nll_loss(log_prob_sub, data.sub)
        loss = loss_obj + loss_sub
        loss.backward()
        optimizer.step()


def test(loader):
    model.eval()

    # Compute Mean Reciprocal Rank (MRR) and Hits@1/3/10.
    result = torch.tensor([0, 0, 0, 0], dtype=torch.float)
    for data in loader:
        data = data.to(device)
        with torch.no_grad():
            log_prob_obj, log_prob_sub = model(data)
        result += model.test(log_prob_obj, data.obj) * data.obj.size(0)
        result += model.test(log_prob_sub, data.sub) * data.sub.size(0)
    result = result / (2 * len(loader.dataset))
    return result.tolist()


for epoch in range(1, 21):
    train()
    mrr, hits1, hits3, hits10 = test(test_loader)
    print(f'Epoch: {epoch:02d}, MRR: {mrr:.4f}, Hits@1: {hits1:.4f}, '
          f'Hits@3: {hits3:.4f}, Hits@10: {hits10:.4f}')
pytorch/examples/rev_gnn.py000066400000000000000000000141151444262304300164000ustar00rootroot00000000000000# Peak GPU memory usage is around 1.57 G
# | RevGNN Models           | Test Acc        | Val Acc         |
# |-------------------------|-----------------|-----------------|
# | 112 layers 160 channels | 0.8307 Β± 0.0030 | 0.9290 Β± 0.0007 |
# | 7 layers 160 channels   | 0.8276 Β± 0.0027 | 0.9272 Β± 0.0006 |

import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import LayerNorm, Linear
from tqdm import tqdm

import torch_geometric.transforms as T
from torch_geometric.loader import RandomNodeLoader
from torch_geometric.nn import GroupAddRev, SAGEConv
from torch_geometric.utils import index_to_mask


class GNNBlock(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.norm = LayerNorm(in_channels, elementwise_affine=True)
        self.conv = SAGEConv(in_channels, out_channels)

    def reset_parameters(self):
        self.norm.reset_parameters()
        self.conv.reset_parameters()

    def forward(self, x, edge_index, dropout_mask=None):
        x = self.norm(x).relu()
        if self.training and dropout_mask is not None:
            x = x * dropout_mask
        return self.conv(x, edge_index)


class RevGNN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers,
                 dropout, num_groups=2):
        super().__init__()

        self.dropout = dropout

        self.lin1 = Linear(in_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)
        self.norm = LayerNorm(hidden_channels, elementwise_affine=True)

        assert hidden_channels % num_groups == 0
        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            conv = GNNBlock(
                hidden_channels // num_groups,
                hidden_channels // num_groups,
            )
            self.convs.append(GroupAddRev(conv, num_groups=num_groups))

    def reset_parameters(self):
        self.lin1.reset_parameters()
        self.lin2.reset_parameters()
        self.norm.reset_parameters()
        for conv in self.convs:
            conv.reset_parameters()

    def forward(self, x, edge_index):
        x = self.lin1(x)

        # Generate a dropout mask which will be shared across GNN blocks:
        mask = None
        if self.training and self.dropout > 0:
            mask = torch.zeros_like(x).bernoulli_(1 - self.dropout)
            mask = mask.requires_grad_(False)
            mask = mask / (1 - self.dropout)

        for conv in self.convs:
            x = conv(x, edge_index, mask)
        x = self.norm(x).relu()
        x = F.dropout(x, p=self.dropout, training=self.training)
        return self.lin2(x)


from ogb.nodeproppred import Evaluator, PygNodePropPredDataset  # noqa

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([T.ToDevice(device), T.ToSparseTensor()])
root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'products')
dataset = PygNodePropPredDataset('ogbn-products', root,
                                 transform=T.AddSelfLoops())
evaluator = Evaluator(name='ogbn-products')

data = dataset[0]
split_idx = dataset.get_idx_split()
for split in ['train', 'valid', 'test']:
    data[f'{split}_mask'] = index_to_mask(split_idx[split], data.y.shape[0])

train_loader = RandomNodeLoader(data, num_parts=10, shuffle=True,
                                num_workers=5)
# Increase the num_parts of the test loader if you cannot fit
# the full batch graph into your GPU:
test_loader = RandomNodeLoader(data, num_parts=1, num_workers=5)

model = RevGNN(
    in_channels=dataset.num_features,
    hidden_channels=160,
    out_channels=dataset.num_classes,
    num_layers=7,  # You can try 1000 layers for fun
    dropout=0.5,
    num_groups=2,
).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.003)


def train(epoch):
    model.train()

    pbar = tqdm(total=len(train_loader))
    pbar.set_description(f'Training epoch: {epoch:03d}')

    total_loss = total_examples = 0
    for data in train_loader:
        optimizer.zero_grad()

        # Memory-efficient aggregations:
        data = transform(data)
        out = model(data.x, data.adj_t)[data.train_mask]
        loss = F.cross_entropy(out, data.y[data.train_mask].view(-1))
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * int(data.train_mask.sum())
        total_examples += int(data.train_mask.sum())
        pbar.update(1)

    pbar.close()

    return total_loss / total_examples


@torch.no_grad()
def test(epoch):
    model.eval()

    y_true = {"train": [], "valid": [], "test": []}
    y_pred = {"train": [], "valid": [], "test": []}

    pbar = tqdm(total=len(test_loader))
    pbar.set_description(f'Evaluating epoch: {epoch:03d}')

    for data in test_loader:
        # Memory-efficient aggregations
        data = transform(data)
        out = model(data.x, data.adj_t).argmax(dim=-1, keepdim=True)

        for split in ['train', 'valid', 'test']:
            mask = data[f'{split}_mask']
            y_true[split].append(data.y[mask].cpu())
            y_pred[split].append(out[mask].cpu())

        pbar.update(1)

    pbar.close()

    train_acc = evaluator.eval({
        'y_true': torch.cat(y_true['train'], dim=0),
        'y_pred': torch.cat(y_pred['train'], dim=0),
    })['acc']

    valid_acc = evaluator.eval({
        'y_true': torch.cat(y_true['valid'], dim=0),
        'y_pred': torch.cat(y_pred['valid'], dim=0),
    })['acc']

    test_acc = evaluator.eval({
        'y_true': torch.cat(y_true['test'], dim=0),
        'y_pred': torch.cat(y_pred['test'], dim=0),
    })['acc']

    return train_acc, valid_acc, test_acc


best_val = 0.0
final_train = 0.0
final_test = 0.0
for epoch in range(1, 1001):
    loss = train(epoch)
    train_acc, val_acc, test_acc = test(epoch)
    if val_acc > best_val:
        best_val = val_acc
        final_train = train_acc
        final_test = test_acc
    print(f'Loss: {loss:.4f}, Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')

print(f'Final Train: {final_train:.4f}, Best Val: {best_val:.4f}, '
      f'Final Test: {final_test:.4f}')
pytorch/examples/rgat.py000066400000000000000000000035701444262304300157020ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Entities
from torch_geometric.nn import RGATConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Entities')
dataset = Entities(path, 'AIFB')
data = dataset[0]
data.x = torch.randn(data.num_nodes, 16)


class RGAT(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels,
                 num_relations):
        super().__init__()
        self.conv1 = RGATConv(in_channels, hidden_channels, num_relations)
        self.conv2 = RGATConv(hidden_channels, hidden_channels, num_relations)
        self.lin = torch.nn.Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, edge_type):
        x = self.conv1(x, edge_index, edge_type).relu()
        x = self.conv2(x, edge_index, edge_type).relu()
        x = self.lin(x)
        return F.log_softmax(x, dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data = data.to(device)
model = RGAT(16, 16, dataset.num_classes, dataset.num_relations).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0005)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index, data.edge_type)
    loss = F.nll_loss(out[data.train_idx], data.train_y)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.x, data.edge_index, data.edge_type).argmax(dim=-1)
    train_acc = float((pred[data.train_idx] == data.train_y).float().mean())
    test_acc = float((pred[data.test_idx] == data.test_y).float().mean())
    return train_acc, test_acc


for epoch in range(1, 51):
    loss = train()
    train_acc, test_acc = test()
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Train: {train_acc:.4f} '
          f'Test: {test_acc:.4f}')
pytorch/examples/rgcn.py000066400000000000000000000053701444262304300156760ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Entities
from torch_geometric.nn import FastRGCNConv, RGCNConv
from torch_geometric.utils import k_hop_subgraph

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='AIFB',
                    choices=['AIFB', 'MUTAG', 'BGS', 'AM'])
args = parser.parse_args()

# Trade memory consumption for faster computation.
if args.dataset in ['AIFB', 'MUTAG']:
    RGCNConv = FastRGCNConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Entities')
dataset = Entities(path, args.dataset)
data = dataset[0]

# BGS and AM graphs are too big to process them in a full-batch fashion.
# Since our model does only make use of a rather small receptive field, we
# filter the graph to only contain the nodes that are at most 2-hop neighbors
# away from any training/test node.
node_idx = torch.cat([data.train_idx, data.test_idx], dim=0)
node_idx, edge_index, mapping, edge_mask = k_hop_subgraph(
    node_idx, 2, data.edge_index, relabel_nodes=True)

data.num_nodes = node_idx.size(0)
data.edge_index = edge_index
data.edge_type = data.edge_type[edge_mask]
data.train_idx = mapping[:data.train_idx.size(0)]
data.test_idx = mapping[data.train_idx.size(0):]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = RGCNConv(data.num_nodes, 16, dataset.num_relations,
                              num_bases=30)
        self.conv2 = RGCNConv(16, dataset.num_classes, dataset.num_relations,
                              num_bases=30)

    def forward(self, edge_index, edge_type):
        x = F.relu(self.conv1(None, edge_index, edge_type))
        x = self.conv2(x, edge_index, edge_type)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device = torch.device('cpu') if args.dataset == 'AM' else device
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0005)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.edge_index, data.edge_type)
    loss = F.nll_loss(out[data.train_idx], data.train_y)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    pred = model(data.edge_index, data.edge_type).argmax(dim=-1)
    train_acc = float((pred[data.train_idx] == data.train_y).float().mean())
    test_acc = float((pred[data.test_idx] == data.test_y).float().mean())
    return train_acc, test_acc


for epoch in range(1, 51):
    loss = train()
    train_acc, test_acc = test()
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Train: {train_acc:.4f} '
          f'Test: {test_acc:.4f}')
pytorch/examples/rgcn_link_pred.py000066400000000000000000000141761444262304300177310ustar00rootroot00000000000000""""
Implements the link prediction task on the FB15k237 datasets according to the
`"Modeling Relational Data with Graph Convolutional Networks"
`_ paper.

Caution: This script is executed in a full-batch fashion, and therefore needs
to run on CPU (following the experimental setup in the official paper).
"""
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Parameter
from tqdm import tqdm

from torch_geometric.datasets import RelLinkPredDataset
from torch_geometric.nn import GAE, RGCNConv

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'RLPD')
dataset = RelLinkPredDataset(path, 'FB15k-237')
data = dataset[0]


class RGCNEncoder(torch.nn.Module):
    def __init__(self, num_nodes, hidden_channels, num_relations):
        super().__init__()
        self.node_emb = Parameter(torch.Tensor(num_nodes, hidden_channels))
        self.conv1 = RGCNConv(hidden_channels, hidden_channels, num_relations,
                              num_blocks=5)
        self.conv2 = RGCNConv(hidden_channels, hidden_channels, num_relations,
                              num_blocks=5)
        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.node_emb)
        self.conv1.reset_parameters()
        self.conv2.reset_parameters()

    def forward(self, edge_index, edge_type):
        x = self.node_emb
        x = self.conv1(x, edge_index, edge_type).relu_()
        x = F.dropout(x, p=0.2, training=self.training)
        x = self.conv2(x, edge_index, edge_type)
        return x


class DistMultDecoder(torch.nn.Module):
    def __init__(self, num_relations, hidden_channels):
        super().__init__()
        self.rel_emb = Parameter(torch.Tensor(num_relations, hidden_channels))
        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.rel_emb)

    def forward(self, z, edge_index, edge_type):
        z_src, z_dst = z[edge_index[0]], z[edge_index[1]]
        rel = self.rel_emb[edge_type]
        return torch.sum(z_src * rel * z_dst, dim=1)


model = GAE(
    RGCNEncoder(data.num_nodes, hidden_channels=500,
                num_relations=dataset.num_relations),
    DistMultDecoder(dataset.num_relations // 2, hidden_channels=500),
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def negative_sampling(edge_index, num_nodes):
    # Sample edges by corrupting either the subject or the object of each edge.
    mask_1 = torch.rand(edge_index.size(1)) < 0.5
    mask_2 = ~mask_1

    neg_edge_index = edge_index.clone()
    neg_edge_index[0, mask_1] = torch.randint(num_nodes, (mask_1.sum(), ))
    neg_edge_index[1, mask_2] = torch.randint(num_nodes, (mask_2.sum(), ))
    return neg_edge_index


def train():
    model.train()
    optimizer.zero_grad()

    z = model.encode(data.edge_index, data.edge_type)

    pos_out = model.decode(z, data.train_edge_index, data.train_edge_type)

    neg_edge_index = negative_sampling(data.train_edge_index, data.num_nodes)
    neg_out = model.decode(z, neg_edge_index, data.train_edge_type)

    out = torch.cat([pos_out, neg_out])
    gt = torch.cat([torch.ones_like(pos_out), torch.zeros_like(neg_out)])
    cross_entropy_loss = F.binary_cross_entropy_with_logits(out, gt)
    reg_loss = z.pow(2).mean() + model.decoder.rel_emb.pow(2).mean()
    loss = cross_entropy_loss + 1e-2 * reg_loss

    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.)
    optimizer.step()

    return float(loss)


@torch.no_grad()
def test():
    model.eval()
    z = model.encode(data.edge_index, data.edge_type)

    valid_mrr = compute_mrr(z, data.valid_edge_index, data.valid_edge_type)
    test_mrr = compute_mrr(z, data.test_edge_index, data.test_edge_type)

    return valid_mrr, test_mrr


@torch.no_grad()
def compute_rank(ranks):
    # fair ranking prediction as the average
    # of optimistic and pessimistic ranking
    true = ranks[0]
    optimistic = (ranks > true).sum() + 1
    pessimistic = (ranks >= true).sum()
    return (optimistic + pessimistic).float() * 0.5


@torch.no_grad()
def compute_mrr(z, edge_index, edge_type):
    ranks = []
    for i in tqdm(range(edge_type.numel())):
        (src, dst), rel = edge_index[:, i], edge_type[i]

        # Try all nodes as tails, but delete true triplets:
        tail_mask = torch.ones(data.num_nodes, dtype=torch.bool)
        for (heads, tails), types in [
            (data.train_edge_index, data.train_edge_type),
            (data.valid_edge_index, data.valid_edge_type),
            (data.test_edge_index, data.test_edge_type),
        ]:
            tail_mask[tails[(heads == src) & (types == rel)]] = False

        tail = torch.arange(data.num_nodes)[tail_mask]
        tail = torch.cat([torch.tensor([dst]), tail])
        head = torch.full_like(tail, fill_value=src)
        eval_edge_index = torch.stack([head, tail], dim=0)
        eval_edge_type = torch.full_like(tail, fill_value=rel)

        out = model.decode(z, eval_edge_index, eval_edge_type)
        rank = compute_rank(out)
        ranks.append(rank)

        # Try all nodes as heads, but delete true triplets:
        head_mask = torch.ones(data.num_nodes, dtype=torch.bool)
        for (heads, tails), types in [
            (data.train_edge_index, data.train_edge_type),
            (data.valid_edge_index, data.valid_edge_type),
            (data.test_edge_index, data.test_edge_type),
        ]:
            head_mask[heads[(tails == dst) & (types == rel)]] = False

        head = torch.arange(data.num_nodes)[head_mask]
        head = torch.cat([torch.tensor([src]), head])
        tail = torch.full_like(head, fill_value=dst)
        eval_edge_index = torch.stack([head, tail], dim=0)
        eval_edge_type = torch.full_like(head, fill_value=rel)

        out = model.decode(z, eval_edge_index, eval_edge_type)
        rank = compute_rank(out)
        ranks.append(rank)

    return (1. / torch.tensor(ranks, dtype=torch.float)).mean()


for epoch in range(1, 10001):
    loss = train()
    print(f'Epoch: {epoch:05d}, Loss: {loss:.4f}')
    if (epoch % 500) == 0:
        valid_mrr, test_mrr = test()
        print(f'Val MRR: {valid_mrr:.4f}, Test MRR: {test_mrr:.4f}')
pytorch/examples/seal_link_pred.py000066400000000000000000000210551444262304300177160ustar00rootroot00000000000000import math
import os.path as osp
from itertools import chain

import numpy as np
import torch
import torch.nn.functional as F
from scipy.sparse.csgraph import shortest_path
from sklearn.metrics import roc_auc_score
from torch.nn import BCEWithLogitsLoss, Conv1d, MaxPool1d, ModuleList

from torch_geometric.data import Data, InMemoryDataset
from torch_geometric.datasets import Planetoid
from torch_geometric.loader import DataLoader
from torch_geometric.nn import MLP, GCNConv, SortAggregation
from torch_geometric.transforms import RandomLinkSplit
from torch_geometric.utils import k_hop_subgraph, to_scipy_sparse_matrix


class SEALDataset(InMemoryDataset):
    def __init__(self, dataset, num_hops, split='train'):
        self.data = dataset[0]
        self.num_hops = num_hops
        super().__init__(dataset.root)
        index = ['train', 'val', 'test'].index(split)
        self.data, self.slices = torch.load(self.processed_paths[index])

    @property
    def processed_file_names(self):
        return ['SEAL_train_data.pt', 'SEAL_val_data.pt', 'SEAL_test_data.pt']

    def process(self):
        transform = RandomLinkSplit(num_val=0.05, num_test=0.1,
                                    is_undirected=True, split_labels=True)
        train_data, val_data, test_data = transform(self.data)

        self._max_z = 0

        # Collect a list of subgraphs for training, validation and testing:
        train_pos_data_list = self.extract_enclosing_subgraphs(
            train_data.edge_index, train_data.pos_edge_label_index, 1)
        train_neg_data_list = self.extract_enclosing_subgraphs(
            train_data.edge_index, train_data.neg_edge_label_index, 0)

        val_pos_data_list = self.extract_enclosing_subgraphs(
            val_data.edge_index, val_data.pos_edge_label_index, 1)
        val_neg_data_list = self.extract_enclosing_subgraphs(
            val_data.edge_index, val_data.neg_edge_label_index, 0)

        test_pos_data_list = self.extract_enclosing_subgraphs(
            test_data.edge_index, test_data.pos_edge_label_index, 1)
        test_neg_data_list = self.extract_enclosing_subgraphs(
            test_data.edge_index, test_data.neg_edge_label_index, 0)

        # Convert node labeling to one-hot features.
        for data in chain(train_pos_data_list, train_neg_data_list,
                          val_pos_data_list, val_neg_data_list,
                          test_pos_data_list, test_neg_data_list):
            # We solely learn links from structure, dropping any node features:
            data.x = F.one_hot(data.z, self._max_z + 1).to(torch.float)

        torch.save(self.collate(train_pos_data_list + train_neg_data_list),
                   self.processed_paths[0])
        torch.save(self.collate(val_pos_data_list + val_neg_data_list),
                   self.processed_paths[1])
        torch.save(self.collate(test_pos_data_list + test_neg_data_list),
                   self.processed_paths[2])

    def extract_enclosing_subgraphs(self, edge_index, edge_label_index, y):
        data_list = []
        for src, dst in edge_label_index.t().tolist():
            sub_nodes, sub_edge_index, mapping, _ = k_hop_subgraph(
                [src, dst], self.num_hops, edge_index, relabel_nodes=True)
            src, dst = mapping.tolist()

            # Remove target link from the subgraph.
            mask1 = (sub_edge_index[0] != src) | (sub_edge_index[1] != dst)
            mask2 = (sub_edge_index[0] != dst) | (sub_edge_index[1] != src)
            sub_edge_index = sub_edge_index[:, mask1 & mask2]

            # Calculate node labeling.
            z = self.drnl_node_labeling(sub_edge_index, src, dst,
                                        num_nodes=sub_nodes.size(0))

            data = Data(x=self.data.x[sub_nodes], z=z,
                        edge_index=sub_edge_index, y=y)
            data_list.append(data)

        return data_list

    def drnl_node_labeling(self, edge_index, src, dst, num_nodes=None):
        # Double-radius node labeling (DRNL).
        src, dst = (dst, src) if src > dst else (src, dst)
        adj = to_scipy_sparse_matrix(edge_index, num_nodes=num_nodes).tocsr()

        idx = list(range(src)) + list(range(src + 1, adj.shape[0]))
        adj_wo_src = adj[idx, :][:, idx]

        idx = list(range(dst)) + list(range(dst + 1, adj.shape[0]))
        adj_wo_dst = adj[idx, :][:, idx]

        dist2src = shortest_path(adj_wo_dst, directed=False, unweighted=True,
                                 indices=src)
        dist2src = np.insert(dist2src, dst, 0, axis=0)
        dist2src = torch.from_numpy(dist2src)

        dist2dst = shortest_path(adj_wo_src, directed=False, unweighted=True,
                                 indices=dst - 1)
        dist2dst = np.insert(dist2dst, src, 0, axis=0)
        dist2dst = torch.from_numpy(dist2dst)

        dist = dist2src + dist2dst
        dist_over_2, dist_mod_2 = dist // 2, dist % 2

        z = 1 + torch.min(dist2src, dist2dst)
        z += dist_over_2 * (dist_over_2 + dist_mod_2 - 1)
        z[src] = 1.
        z[dst] = 1.
        z[torch.isnan(z)] = 0.

        self._max_z = max(int(z.max()), self._max_z)

        return z.to(torch.long)


path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora')

train_dataset = SEALDataset(dataset, num_hops=2, split='train')
val_dataset = SEALDataset(dataset, num_hops=2, split='val')
test_dataset = SEALDataset(dataset, num_hops=2, split='test')

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32)
test_loader = DataLoader(test_dataset, batch_size=32)


class DGCNN(torch.nn.Module):
    def __init__(self, hidden_channels, num_layers, GNN=GCNConv, k=0.6):
        super().__init__()

        if k < 1:  # Transform percentile to number.
            num_nodes = sorted([data.num_nodes for data in train_dataset])
            k = num_nodes[int(math.ceil(k * len(num_nodes))) - 1]
            k = int(max(10, k))

        self.convs = ModuleList()
        self.convs.append(GNN(train_dataset.num_features, hidden_channels))
        for i in range(0, num_layers - 1):
            self.convs.append(GNN(hidden_channels, hidden_channels))
        self.convs.append(GNN(hidden_channels, 1))

        conv1d_channels = [16, 32]
        total_latent_dim = hidden_channels * num_layers + 1
        conv1d_kws = [total_latent_dim, 5]
        self.conv1 = Conv1d(1, conv1d_channels[0], conv1d_kws[0],
                            conv1d_kws[0])
        self.pool = SortAggregation(k)
        self.maxpool1d = MaxPool1d(2, 2)
        self.conv2 = Conv1d(conv1d_channels[0], conv1d_channels[1],
                            conv1d_kws[1], 1)
        dense_dim = int((k - 2) / 2 + 1)
        dense_dim = (dense_dim - conv1d_kws[1] + 1) * conv1d_channels[1]
        self.mlp = MLP([dense_dim, 128, 1], dropout=0.5, norm=None)

    def forward(self, x, edge_index, batch):
        xs = [x]
        for conv in self.convs:
            xs += [conv(xs[-1], edge_index).tanh()]
        x = torch.cat(xs[1:], dim=-1)

        # Global pooling.
        x = self.pool(x, batch)
        x = x.unsqueeze(1)  # [num_graphs, 1, k * hidden]
        x = self.conv1(x).relu()
        x = self.maxpool1d(x)
        x = self.conv2(x).relu()
        x = x.view(x.size(0), -1)  # [num_graphs, dense_dim]

        return self.mlp(x)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = DGCNN(hidden_channels=32, num_layers=3).to(device)
optimizer = torch.optim.Adam(params=model.parameters(), lr=0.0001)
criterion = BCEWithLogitsLoss()


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = criterion(out.view(-1), data.y.to(torch.float))
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs

    return total_loss / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    y_pred, y_true = [], []
    for data in loader:
        data = data.to(device)
        logits = model(data.x, data.edge_index, data.batch)
        y_pred.append(logits.view(-1).cpu())
        y_true.append(data.y.view(-1).cpu().to(torch.float))

    return roc_auc_score(torch.cat(y_true), torch.cat(y_pred))


best_val_auc = test_auc = 0
for epoch in range(1, 51):
    loss = train()
    val_auc = test(val_loader)
    if val_auc > best_val_auc:
        best_val_auc = val_auc
        test_auc = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_auc:.4f}, '
          f'Test: {test_auc:.4f}')
pytorch/examples/sgc.py000066400000000000000000000031171444262304300155160ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Planetoid
from torch_geometric.nn import SGConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SGConv(dataset.num_features, dataset.num_classes, K=2,
                            cached=True)

    def forward(self):
        x, edge_index = data.x, data.edge_index
        x = self.conv1(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.2, weight_decay=0.005)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 101):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/shadow.py000066400000000000000000000056361444262304300162370ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

from torch_geometric.datasets import Flickr
from torch_geometric.loader import ShaDowKHopSampler
from torch_geometric.nn import SAGEConv, global_mean_pool

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Flickr')
dataset = Flickr(path)
data = dataset[0]

kwargs = {'batch_size': 1024, 'num_workers': 6, 'persistent_workers': True}
train_loader = ShaDowKHopSampler(data, depth=2, num_neighbors=5,
                                 node_idx=data.train_mask, **kwargs)
val_loader = ShaDowKHopSampler(data, depth=2, num_neighbors=5,
                               node_idx=data.val_mask, **kwargs)
test_loader = ShaDowKHopSampler(data, depth=2, num_neighbors=5,
                                node_idx=data.test_mask, **kwargs)


class GNN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv(in_channels, hidden_channels)
        self.conv2 = SAGEConv(hidden_channels, hidden_channels)
        self.conv3 = SAGEConv(hidden_channels, hidden_channels)
        self.lin = torch.nn.Linear(2 * hidden_channels, out_channels)

    def forward(self, x, edge_index, batch, root_n_id):
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, p=0.3)
        x = self.conv2(x, edge_index).relu()
        x = F.dropout(x, p=0.3, training=self.training)
        x = self.conv3(x, edge_index).relu()
        x = F.dropout(x, p=0.3, training=self.training)

        # We merge both central node embeddings and subgraph embeddings:
        x = torch.cat([x[root_n_id], global_mean_pool(x, batch)], dim=-1)

        x = self.lin(x)
        return x


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = GNN(dataset.num_features, 256, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train():
    model.train()
    total_loss = total_examples = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch, data.root_n_id)
        loss = F.cross_entropy(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs
        total_examples += data.num_graphs
    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()
    total_correct = total_examples = 0
    for data in loader:
        data = data.to(device)
        out = model(data.x, data.edge_index, data.batch, data.root_n_id)
        total_correct += int((out.argmax(dim=-1) == data.y).sum())
        total_examples += data.num_graphs
    return total_correct / total_examples


for epoch in range(1, 51):
    loss = train()
    val_acc = test(val_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, ',
          f'Val: {val_acc:.4f} Test: {test_acc:.4f}')
pytorch/examples/sign.py000066400000000000000000000052671444262304300157120ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear
from torch.utils.data import DataLoader

import torch_geometric.transforms as T
from torch_geometric.datasets import Flickr

K = 2
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Flickr')
transform = T.Compose([T.NormalizeFeatures(), T.SIGN(K)])
dataset = Flickr(path, transform=transform)
data = dataset[0]

train_idx = data.train_mask.nonzero(as_tuple=False).view(-1)
val_idx = data.val_mask.nonzero(as_tuple=False).view(-1)
test_idx = data.test_mask.nonzero(as_tuple=False).view(-1)

train_loader = DataLoader(train_idx, batch_size=16 * 1024, shuffle=True)
val_loader = DataLoader(val_idx, batch_size=32 * 1024)
test_loader = DataLoader(test_idx, batch_size=32 * 1024)


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.lins = torch.nn.ModuleList()
        for _ in range(K + 1):
            self.lins.append(Linear(dataset.num_node_features, 1024))
        self.lin = Linear((K + 1) * 1024, dataset.num_classes)

    def forward(self, xs):
        hs = []
        for x, lin in zip(xs, self.lins):
            h = lin(x).relu()
            h = F.dropout(h, p=0.5, training=self.training)
            hs.append(h)
        h = torch.cat(hs, dim=-1)
        h = self.lin(h)
        return h.log_softmax(dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


def train():
    model.train()

    total_loss = total_examples = 0
    for idx in train_loader:
        xs = [data.x[idx].to(device)]
        xs += [data[f'x{i}'][idx].to(device) for i in range(1, K + 1)]
        y = data.y[idx].to(device)

        optimizer.zero_grad()
        out = model(xs)
        loss = F.nll_loss(out, y)
        loss.backward()
        optimizer.step()

        total_loss += float(loss) * idx.numel()
        total_examples += idx.numel()

    return total_loss / total_examples


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = total_examples = 0
    for idx in loader:
        xs = [data.x[idx].to(device)]
        xs += [data[f'x{i}'][idx].to(device) for i in range(1, K + 1)]
        y = data.y[idx].to(device)

        out = model(xs)
        total_correct += int((out.argmax(dim=-1) == y).sum())
        total_examples += idx.numel()

    return total_correct / total_examples


for epoch in range(1, 201):
    loss = train()
    train_acc = test(train_loader)
    val_acc = test(val_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/signed_gcn.py000066400000000000000000000032311444262304300170370ustar00rootroot00000000000000import os.path as osp

import torch

from torch_geometric.datasets import BitcoinOTC
from torch_geometric.nn import SignedGCN

name = 'BitcoinOTC-1'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', name)
dataset = BitcoinOTC(path, edge_window_size=1)

# Generate dataset.
pos_edge_indices, neg_edge_indices = [], []
for data in dataset:
    pos_edge_indices.append(data.edge_index[:, data.edge_attr > 0])
    neg_edge_indices.append(data.edge_index[:, data.edge_attr < 0])
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pos_edge_index = torch.cat(pos_edge_indices, dim=1).to(device)
neg_edge_index = torch.cat(neg_edge_indices, dim=1).to(device)

# Build and train model.
model = SignedGCN(64, 64, num_layers=2, lamb=5).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)

train_pos_edge_index, test_pos_edge_index = model.split_edges(pos_edge_index)
train_neg_edge_index, test_neg_edge_index = model.split_edges(neg_edge_index)
x = model.create_spectral_features(train_pos_edge_index, train_neg_edge_index)


def train():
    model.train()
    optimizer.zero_grad()
    z = model(x, train_pos_edge_index, train_neg_edge_index)
    loss = model.loss(z, train_pos_edge_index, train_neg_edge_index)
    loss.backward()
    optimizer.step()
    return loss.item()


def test():
    model.eval()
    with torch.no_grad():
        z = model(x, train_pos_edge_index, train_neg_edge_index)
    return model.test(z, test_pos_edge_index, test_neg_edge_index)


for epoch in range(101):
    loss = train()
    auc, f1 = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, AUC: {auc:.4f}, '
          f'F1: {f1:.4f}')
pytorch/examples/super_gat.py000066400000000000000000000043731444262304300167400ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import SuperGATConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = SuperGATConv(dataset.num_features, 8, heads=8,
                                  dropout=0.6, attention_type='MX',
                                  edge_sample_ratio=0.8, is_undirected=True)
        self.conv2 = SuperGATConv(8 * 8, dataset.num_classes, heads=8,
                                  concat=False, dropout=0.6,
                                  attention_type='MX', edge_sample_ratio=0.8,
                                  is_undirected=True)

    def forward(self, x, edge_index):
        x = F.dropout(x, p=0.6, training=self.training)
        x = F.elu(self.conv1(x, edge_index))
        att_loss = self.conv1.get_attention_loss()
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, data.edge_index)
        att_loss += self.conv2.get_attention_loss()
        return F.log_softmax(x, dim=-1), att_loss


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)


def train(data):
    model.train()
    optimizer.zero_grad()
    out, att_loss = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss += 4.0 * att_loss
    loss.backward()
    optimizer.step()


@torch.no_grad()
def test(data):
    model.eval()
    out, accs = model(data.x, data.edge_index)[0], []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


for epoch in range(1, 501):
    train(data)
    train_acc, val_acc, test_acc = test(data)
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/tagcn.py000066400000000000000000000033571444262304300160440ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import TAGConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = TAGConv(dataset.num_features, 16)
        self.conv2 = TAGConv(16, dataset.num_classes)

    def forward(self):
        x, edge_index = data.x, data.edge_index
        x = F.relu(self.conv1(x, edge_index))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


best_val_acc = test_acc = 0
for epoch in range(1, 201):
    train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/tensorboard_logging.py000066400000000000000000000043421444262304300207730ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from torch.utils.tensorboard import SummaryWriter

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv

dataset = 'Cora'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, 16, cached=True)
        self.conv2 = GCNConv(16, dataset.num_classes, cached=True)

    def forward(self, x, edge_index):
        x = F.relu(self.conv1(x, edge_index, None))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, None)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)


def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()
    return loss.item()


@torch.no_grad()
def test():
    model.eval()
    out, accs = model(data.x, data.edge_index), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = out[mask].argmax(1)
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


model(data.x, data.edge_index)
writer = SummaryWriter()
writer.add_graph(model, [data.x, data.edge_index])

best_val_acc = test_acc = 0
for epoch in range(1, 201):
    train_loss = train()
    train_acc, val_acc, tmp_test_acc = test()
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        test_acc = tmp_test_acc
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, '
          f'Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')

    writer.add_scalar('Loss/train', train_loss, epoch)
    writer.add_scalar('Accuracy/train', train_acc, epoch)
    writer.add_scalar('Accuracy/val', val_acc, epoch)
    writer.add_scalar('Accuracy/test', test_acc, epoch)
pytorch/examples/tgn.py000066400000000000000000000157331444262304300155410ustar00rootroot00000000000000# This code achieves a performance of around 96.60%. However, it is not
# directly comparable to the results reported by the TGN paper since a
# slightly different evaluation setup is used here.
# In particular, predictions in the same batch are made in parallel, i.e.
# predictions for interactions later in the batch have no access to any
# information whatsoever about previous interactions in the same batch.
# On the contrary, when sampling node neighborhoods for interactions later in
# the batch, the TGN paper code has access to previous interactions in the
# batch.
# While both approaches are correct, together with the authors of the paper we
# decided to present this version here as it is more realsitic and a better
# test bed for future methods.

import os.path as osp

import torch
from sklearn.metrics import average_precision_score, roc_auc_score
from torch.nn import Linear

from torch_geometric.datasets import JODIEDataset
from torch_geometric.loader import TemporalDataLoader
from torch_geometric.nn import TGNMemory, TransformerConv
from torch_geometric.nn.models.tgn import (
    IdentityMessage,
    LastAggregator,
    LastNeighborLoader,
)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'JODIE')
dataset = JODIEDataset(path, name='wikipedia')
data = dataset[0]

# For small datasets, we can put the whole dataset on GPU and thus avoid
# expensive memory transfer costs for mini-batches:
data = data.to(device)

# Ensure to only sample actual destination nodes as negatives.
min_dst_idx, max_dst_idx = int(data.dst.min()), int(data.dst.max())
train_data, val_data, test_data = data.train_val_test_split(
    val_ratio=0.15, test_ratio=0.15)

train_loader = TemporalDataLoader(train_data, batch_size=200)
val_loader = TemporalDataLoader(val_data, batch_size=200)
test_loader = TemporalDataLoader(test_data, batch_size=200)

neighbor_loader = LastNeighborLoader(data.num_nodes, size=10, device=device)


class GraphAttentionEmbedding(torch.nn.Module):
    def __init__(self, in_channels, out_channels, msg_dim, time_enc):
        super().__init__()
        self.time_enc = time_enc
        edge_dim = msg_dim + time_enc.out_channels
        self.conv = TransformerConv(in_channels, out_channels // 2, heads=2,
                                    dropout=0.1, edge_dim=edge_dim)

    def forward(self, x, last_update, edge_index, t, msg):
        rel_t = last_update[edge_index[0]] - t
        rel_t_enc = self.time_enc(rel_t.to(x.dtype))
        edge_attr = torch.cat([rel_t_enc, msg], dim=-1)
        return self.conv(x, edge_index, edge_attr)


class LinkPredictor(torch.nn.Module):
    def __init__(self, in_channels):
        super().__init__()
        self.lin_src = Linear(in_channels, in_channels)
        self.lin_dst = Linear(in_channels, in_channels)
        self.lin_final = Linear(in_channels, 1)

    def forward(self, z_src, z_dst):
        h = self.lin_src(z_src) + self.lin_dst(z_dst)
        h = h.relu()
        return self.lin_final(h)


memory_dim = time_dim = embedding_dim = 100

memory = TGNMemory(
    data.num_nodes,
    data.msg.size(-1),
    memory_dim,
    time_dim,
    message_module=IdentityMessage(data.msg.size(-1), memory_dim, time_dim),
    aggregator_module=LastAggregator(),
).to(device)

gnn = GraphAttentionEmbedding(
    in_channels=memory_dim,
    out_channels=embedding_dim,
    msg_dim=data.msg.size(-1),
    time_enc=memory.time_enc,
).to(device)

link_pred = LinkPredictor(in_channels=embedding_dim).to(device)

optimizer = torch.optim.Adam(
    set(memory.parameters()) | set(gnn.parameters())
    | set(link_pred.parameters()), lr=0.0001)
criterion = torch.nn.BCEWithLogitsLoss()

# Helper vector to map global node indices to local ones.
assoc = torch.empty(data.num_nodes, dtype=torch.long, device=device)


def train():
    memory.train()
    gnn.train()
    link_pred.train()

    memory.reset_state()  # Start with a fresh memory.
    neighbor_loader.reset_state()  # Start with an empty graph.

    total_loss = 0
    for batch in train_loader:
        batch = batch.to(device)
        optimizer.zero_grad()

        src, pos_dst, t, msg = batch.src, batch.dst, batch.t, batch.msg

        # Sample negative destination nodes.
        neg_dst = torch.randint(min_dst_idx, max_dst_idx + 1, (src.size(0), ),
                                dtype=torch.long, device=device)

        n_id = torch.cat([src, pos_dst, neg_dst]).unique()
        n_id, edge_index, e_id = neighbor_loader(n_id)
        assoc[n_id] = torch.arange(n_id.size(0), device=device)

        # Get updated memory of all nodes involved in the computation.
        z, last_update = memory(n_id)
        z = gnn(z, last_update, edge_index, data.t[e_id].to(device),
                data.msg[e_id].to(device))

        pos_out = link_pred(z[assoc[src]], z[assoc[pos_dst]])
        neg_out = link_pred(z[assoc[src]], z[assoc[neg_dst]])

        loss = criterion(pos_out, torch.ones_like(pos_out))
        loss += criterion(neg_out, torch.zeros_like(neg_out))

        # Update memory and neighbor loader with ground-truth state.
        memory.update_state(src, pos_dst, t, msg)
        neighbor_loader.insert(src, pos_dst)

        loss.backward()
        optimizer.step()
        memory.detach()
        total_loss += float(loss) * batch.num_events

    return total_loss / train_data.num_events


@torch.no_grad()
def test(loader):
    memory.eval()
    gnn.eval()
    link_pred.eval()

    torch.manual_seed(12345)  # Ensure deterministic sampling across epochs.

    aps, aucs = [], []
    for batch in loader:
        batch = batch.to(device)
        src, pos_dst, t, msg = batch.src, batch.dst, batch.t, batch.msg

        neg_dst = torch.randint(min_dst_idx, max_dst_idx + 1, (src.size(0), ),
                                dtype=torch.long, device=device)

        n_id = torch.cat([src, pos_dst, neg_dst]).unique()
        n_id, edge_index, e_id = neighbor_loader(n_id)
        assoc[n_id] = torch.arange(n_id.size(0), device=device)

        z, last_update = memory(n_id)
        z = gnn(z, last_update, edge_index, data.t[e_id].to(device),
                data.msg[e_id].to(device))

        pos_out = link_pred(z[assoc[src]], z[assoc[pos_dst]])
        neg_out = link_pred(z[assoc[src]], z[assoc[neg_dst]])

        y_pred = torch.cat([pos_out, neg_out], dim=0).sigmoid().cpu()
        y_true = torch.cat(
            [torch.ones(pos_out.size(0)),
             torch.zeros(neg_out.size(0))], dim=0)

        aps.append(average_precision_score(y_true, y_pred))
        aucs.append(roc_auc_score(y_true, y_pred))

        memory.update_state(src, pos_dst, t, msg)
        neighbor_loader.insert(src, pos_dst)

    return float(torch.tensor(aps).mean()), float(torch.tensor(aucs).mean())


for epoch in range(1, 51):
    loss = train()
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}')
    val_ap, val_auc = test(val_loader)
    test_ap, test_auc = test(test_loader)
    print(f'Val AP: {val_ap:.4f}, Val AUC: {val_auc:.4f}')
    print(f'Test AP: {test_ap:.4f}, Test AUC: {test_auc:.4f}')
pytorch/examples/triangles_sag_pool.py000066400000000000000000000076071444262304300206250ustar00rootroot00000000000000import copy
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.transforms as T
from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GCNConv, GINConv, SAGPooling, global_max_pool
from torch_geometric.utils import scatter


class HandleNodeAttention:
    def __call__(self, data):
        data = copy.copy(data)
        data.attn = torch.softmax(data.x, dim=0).flatten()
        data.x = None
        return data


transform = T.Compose([HandleNodeAttention(), T.OneHotDegree(max_degree=14)])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'TRIANGLES')
dataset = TUDataset(path, name='TRIANGLES', use_node_attr=True,
                    transform=transform)

train_loader = DataLoader(dataset[:30000], batch_size=60, shuffle=True)
val_loader = DataLoader(dataset[30000:35000], batch_size=60)
test_loader = DataLoader(dataset[35000:], batch_size=60)


class Net(torch.nn.Module):
    def __init__(self, in_channels):
        super().__init__()

        self.conv1 = GINConv(Seq(Lin(in_channels, 64), ReLU(), Lin(64, 64)))
        self.pool1 = SAGPooling(64, min_score=0.001, GNN=GCNConv)
        self.conv2 = GINConv(Seq(Lin(64, 64), ReLU(), Lin(64, 64)))
        self.pool2 = SAGPooling(64, min_score=0.001, GNN=GCNConv)
        self.conv3 = GINConv(Seq(Lin(64, 64), ReLU(), Lin(64, 64)))

        self.lin = torch.nn.Linear(64, 1)

    def forward(self, data):
        x, edge_index, batch = data.x, data.edge_index, data.batch

        x = F.relu(self.conv1(x, edge_index))
        x, edge_index, _, batch, perm, score = self.pool1(
            x, edge_index, None, batch)
        x = F.relu(self.conv2(x, edge_index))
        x, edge_index, _, batch, perm, score = self.pool2(
            x, edge_index, None, batch)
        ratio = x.size(0) / data.x.size(0)

        x = F.relu(self.conv3(x, edge_index))
        x = global_max_pool(x, batch)
        x = self.lin(x).view(-1)

        attn_loss = F.kl_div(torch.log(score + 1e-14), data.attn[perm],
                             reduction='none')
        attn_loss = scatter(attn_loss, batch, reduce='mean')

        return x, attn_loss, ratio


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train(epoch):
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out, attn_loss, _ = model(data)
        loss = ((out - data.y).pow(2) + 100 * attn_loss).mean()
        loss.backward()
        total_loss += loss.item() * data.num_graphs
        optimizer.step()

    return total_loss / len(train_loader.dataset)


def test(loader):
    model.eval()

    corrects, total_ratio = [], 0
    for data in loader:
        data = data.to(device)
        out, _, ratio = model(data)
        pred = out.round().to(torch.long)
        corrects.append(pred.eq(data.y.to(torch.long)))
        total_ratio += ratio
    return torch.cat(corrects, dim=0), total_ratio / len(loader)


for epoch in range(1, 301):
    loss = train(epoch)
    train_correct, train_ratio = test(train_loader)
    val_correct, val_ratio = test(val_loader)
    test_correct, test_ratio = test(test_loader)

    train_acc = train_correct.sum().item() / train_correct.size(0)
    val_acc = val_correct.sum().item() / val_correct.size(0)

    test_acc1 = test_correct[:5000].sum().item() / 5000
    test_acc2 = test_correct[5000:].sum().item() / 5000

    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_acc:.3f}, '
          f'Val: {val_acc:.3f}, Test Orig: {test_acc1:.3f}, '
          f'Test Large: {test_acc2:.3f}, Train/Val/Test Ratio='
          f'{train_ratio:.3f}/{val_ratio:.3f}/{test_ratio:.3f}')
pytorch/examples/unimp_arxiv.py000066400000000000000000000063231444262304300173050ustar00rootroot00000000000000import os.path as osp

import torch
import torch.nn.functional as F
from ogb.nodeproppred import PygNodePropPredDataset

import torch_geometric.transforms as T
from torch_geometric.nn import MaskLabel, TransformerConv
from torch_geometric.utils import index_to_mask

root = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'OGB')
dataset = PygNodePropPredDataset('ogbn-arxiv', root, T.ToUndirected())


class UniMP(torch.nn.Module):
    def __init__(self, in_channels, num_classes, hidden_channels, num_layers,
                 heads, dropout=0.3):
        super().__init__()

        self.label_emb = MaskLabel(num_classes, in_channels)

        self.convs = torch.nn.ModuleList()
        self.norms = torch.nn.ModuleList()
        for i in range(1, num_layers + 1):
            if i < num_layers:
                out_channels = hidden_channels // heads
                concat = True
            else:
                out_channels = num_classes
                concat = False
            conv = TransformerConv(in_channels, out_channels, heads,
                                   concat=concat, beta=True, dropout=dropout)
            self.convs.append(conv)
            in_channels = hidden_channels

            if i < num_layers:
                self.norms.append(torch.nn.LayerNorm(hidden_channels))

    def forward(self, x, y, edge_index, label_mask):
        x = self.label_emb(x, y, label_mask)
        for conv, norm in zip(self.convs, self.norms):
            x = norm(conv(x, edge_index)).relu()
        return self.convs[-1](x, edge_index)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data = dataset[0].to(device)
data.y = data.y.view(-1)
model = UniMP(dataset.num_features, dataset.num_classes, hidden_channels=64,
              num_layers=3, heads=2).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.0005)

split_idx = dataset.get_idx_split()
train_mask = index_to_mask(split_idx['train'], size=data.num_nodes)
val_mask = index_to_mask(split_idx['valid'], size=data.num_nodes)
test_mask = index_to_mask(split_idx['test'], size=data.num_nodes)


def train(label_rate=0.65):  # How many labels to use for propagation.
    model.train()

    propagation_mask = MaskLabel.ratio_mask(train_mask, ratio=label_rate)
    supervision_mask = train_mask ^ propagation_mask

    optimizer.zero_grad()
    out = model(data.x, data.y, data.edge_index, propagation_mask)
    loss = F.cross_entropy(out[supervision_mask], data.y[supervision_mask])
    loss.backward()
    optimizer.step()

    return float(loss)


@torch.no_grad()
def test():
    model.eval()

    propagation_mask = train_mask
    out = model(data.x, data.y, data.edge_index, propagation_mask)
    pred = out[val_mask].argmax(dim=-1)
    val_acc = int((pred == data.y[val_mask]).sum()) / pred.size(0)

    propagation_mask = train_mask | val_mask
    out = model(data.x, data.y, data.edge_index, propagation_mask)
    pred = out[test_mask].argmax(dim=-1)
    test_acc = int((pred == data.y[test_mask]).sum()) / pred.size(0)

    return val_acc, test_acc


for epoch in range(1, 501):
    loss = train()
    val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val: {val_acc:.4f}, '
          f'Test: {test_acc:.4f}')
pytorch/examples/upfd.py000066400000000000000000000073431444262304300157050ustar00rootroot00000000000000import argparse
import os.path as osp

import torch
import torch.nn.functional as F
from torch.nn import Linear

from torch_geometric.datasets import UPFD
from torch_geometric.loader import DataLoader
from torch_geometric.nn import GATConv, GCNConv, SAGEConv, global_max_pool
from torch_geometric.transforms import ToUndirected

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='politifact',
                    choices=['politifact', 'gossipcop'])
parser.add_argument('--feature', type=str, default='spacy',
                    choices=['profile', 'spacy', 'bert', 'content'])
parser.add_argument('--model', type=str, default='GCN',
                    choices=['GCN', 'GAT', 'SAGE'])
args = parser.parse_args()

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'UPFD')
train_dataset = UPFD(path, args.dataset, args.feature, 'train', ToUndirected())
val_dataset = UPFD(path, args.dataset, args.feature, 'val', ToUndirected())
test_dataset = UPFD(path, args.dataset, args.feature, 'test', ToUndirected())

train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=128, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=128, shuffle=False)


class Net(torch.nn.Module):
    def __init__(self, model, in_channels, hidden_channels, out_channels,
                 concat=False):
        super().__init__()
        self.concat = concat

        if model == 'GCN':
            self.conv1 = GCNConv(in_channels, hidden_channels)
        elif model == 'SAGE':
            self.conv1 = SAGEConv(in_channels, hidden_channels)
        elif model == 'GAT':
            self.conv1 = GATConv(in_channels, hidden_channels)

        if self.concat:
            self.lin0 = Linear(in_channels, hidden_channels)
            self.lin1 = Linear(2 * hidden_channels, hidden_channels)

        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, batch):
        h = self.conv1(x, edge_index).relu()
        h = global_max_pool(h, batch)

        if self.concat:
            # Get the root node (tweet) features of each graph:
            root = (batch[1:] - batch[:-1]).nonzero(as_tuple=False).view(-1)
            root = torch.cat([root.new_zeros(1), root + 1], dim=0)
            news = x[root]

            news = self.lin0(news).relu()
            h = self.lin1(torch.cat([news, h], dim=-1)).relu()

        h = self.lin2(h)
        return h.log_softmax(dim=-1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(args.model, train_dataset.num_features, 128,
            train_dataset.num_classes, concat=True).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)


def train():
    model.train()

    total_loss = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(out, data.y)
        loss.backward()
        optimizer.step()
        total_loss += float(loss) * data.num_graphs

    return total_loss / len(train_loader.dataset)


@torch.no_grad()
def test(loader):
    model.eval()

    total_correct = total_examples = 0
    for data in loader:
        data = data.to(device)
        pred = model(data.x, data.edge_index, data.batch).argmax(dim=-1)
        total_correct += int((pred == data.y).sum())
        total_examples += data.num_graphs

    return total_correct / total_examples


for epoch in range(1, 61):
    loss = train()
    train_acc = test(train_loader)
    val_acc = test(val_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Train: {train_acc:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')
pytorch/examples/wl_kernel.py000066400000000000000000000043371444262304300167310ustar00rootroot00000000000000import argparse
import os.path as osp
import warnings

import torch
from sklearn.exceptions import ConvergenceWarning
from sklearn.metrics import accuracy_score
from sklearn.svm import LinearSVC

from torch_geometric.data import Batch
from torch_geometric.datasets import TUDataset
from torch_geometric.nn import WLConv

warnings.filterwarnings('ignore', category=ConvergenceWarning)

parser = argparse.ArgumentParser()
parser.add_argument('--runs', type=int, default=10)
args = parser.parse_args()

torch.manual_seed(42)

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'TU')
dataset = TUDataset(path, name='ENZYMES')
data = Batch.from_data_list(dataset)


class WL(torch.nn.Module):
    def __init__(self, num_layers):
        super().__init__()
        self.convs = torch.nn.ModuleList([WLConv() for _ in range(num_layers)])

    def forward(self, x, edge_index, batch=None):
        hists = []
        for conv in self.convs:
            x = conv(x, edge_index)
            hists.append(conv.histogram(x, batch, norm=True))
        return hists


wl = WL(num_layers=5)
hists = wl(data.x, data.edge_index, data.batch)

test_accs = torch.empty(args.runs, dtype=torch.float)

for run in range(1, args.runs + 1):
    perm = torch.randperm(data.num_graphs)
    val_index = perm[:data.num_graphs // 10]
    test_index = perm[data.num_graphs // 10:data.num_graphs // 5]
    train_index = perm[data.num_graphs // 5:]

    best_val_acc = 0

    for hist in hists:
        train_hist, train_y = hist[train_index], data.y[train_index]
        val_hist, val_y = hist[val_index], data.y[val_index]
        test_hist, test_y = hist[test_index], data.y[test_index]

        for C in [10**3, 10**2, 10**1, 10**0, 10**-1, 10**-2, 10**-3]:
            model = LinearSVC(C=C, tol=0.01, dual=True)
            model.fit(train_hist, train_y)
            val_acc = accuracy_score(val_y, model.predict(val_hist))
            if val_acc > best_val_acc:
                best_val_acc = val_acc
                test_acc = accuracy_score(test_y, model.predict(test_hist))
                test_accs[run - 1] = test_acc

    print(f'Run: {run:02d}, Val: {best_val_acc:.4f}, Test: {test_acc:.4f}')

print(f'Final Test Performance: {test_accs.mean():.4f}Β±{test_accs.std():.4f}')
pytorch/graphgym/000077500000000000000000000000001444262304300143665ustar00rootroot00000000000000pytorch/graphgym/agg_batch.py000066400000000000000000000011431444262304300166360ustar00rootroot00000000000000import argparse

from torch_geometric.graphgym.utils.agg_runs import agg_batch


def parse_args():
    """Parses the arguments."""
    parser = argparse.ArgumentParser(
        description='Train a classification model')
    parser.add_argument('--dir', dest='dir', help='Dir for batch of results',
                        required=True, type=str)
    parser.add_argument('--metric', dest='metric',
                        help='metric to select best epoch', required=False,
                        type=str, default='auto')
    return parser.parse_args()


args = parse_args()
agg_batch(args.dir, args.metric)
pytorch/graphgym/configs/000077500000000000000000000000001444262304300160165ustar00rootroot00000000000000pytorch/graphgym/configs/example.yaml000066400000000000000000000011321444262304300203320ustar00rootroot00000000000000# The recommended basic settings for GNN
out_dir: results
dataset:
  format: PyG
  name: Cora
  task: node
  task_type: classification
  transductive: true
  split: [0.8, 0.2]
  transform: none
train:
  batch_size: 32
  eval_period: 20
  ckpt_period: 100
model:
  type: gnn
  loss_fun: cross_entropy
  edge_decoding: dot
  graph_pooling: add
gnn:
  layers_pre_mp: 1
  layers_mp: 2
  layers_post_mp: 1
  dim_inner: 256
  layer_type: generalconv
  stage_type: stack
  batchnorm: true
  act: prelu
  dropout: 0.0
  agg: add
  normalize_adj: false
optim:
  optimizer: adam
  base_lr: 0.01
  max_epoch: 400
pytorch/graphgym/configs/pyg/000077500000000000000000000000001444262304300166155ustar00rootroot00000000000000pytorch/graphgym/configs/pyg/example_graph.yaml000066400000000000000000000011631444262304300223160ustar00rootroot00000000000000out_dir: results
dataset:
  format: OGB
  name: ogbg-molhiv
  task: graph
  task_type: classification
  node_encoder: true
  node_encoder_name: Atom
  edge_encoder: true
  edge_encoder_name: Bond
train:
  batch_size: 128
  eval_period: 1
  ckpt_period: 100
  sampler: full_batch
model:
  type: gnn
  loss_fun: cross_entropy
  edge_decoding: dot
  graph_pooling: add
gnn:
  layers_pre_mp: 1
  layers_mp: 2
  layers_post_mp: 1
  dim_inner: 300
  layer_type: generalconv
  stage_type: stack
  batchnorm: true
  act: prelu
  dropout: 0.0
  agg: mean
  normalize_adj: false
optim:
  optimizer: adam
  base_lr: 0.01
  max_epoch: 100
pytorch/graphgym/configs/pyg/example_link.yaml000066400000000000000000000011651444262304300221540ustar00rootroot00000000000000out_dir: results
dataset:
  format: OGB
  name: ogbl-collab
  task: link_pred
  task_type: classification
  node_encoder: false
  node_encoder_name: Atom
  edge_encoder: false
  edge_encoder_name: Bond
train:
  batch_size: 128
  eval_period: 1
  ckpt_period: 100
  sampler: full_batch
model:
  type: gnn
  loss_fun: cross_entropy
  edge_decoding: dot
  graph_pooling: add
gnn:
  layers_pre_mp: 1
  layers_mp: 2
  layers_post_mp: 1
  dim_inner: 300
  layer_type: gcnconv
  stage_type: stack
  batchnorm: true
  act: prelu
  dropout: 0.0
  agg: mean
  normalize_adj: false
optim:
  optimizer: adam
  base_lr: 0.01
  max_epoch: 100
pytorch/graphgym/configs/pyg/example_node.yaml000066400000000000000000000011511444262304300221370ustar00rootroot00000000000000out_dir: results
dataset:
  format: PyG
  name: Cora
  task: node
  task_type: classification
  node_encoder: false
  node_encoder_name: Atom
  edge_encoder: false
  edge_encoder_name: Bond
train:
  batch_size: 128
  eval_period: 1
  ckpt_period: 100
  sampler: full_batch
model:
  type: gnn
  loss_fun: cross_entropy
  edge_decoding: dot
  graph_pooling: add
gnn:
  layers_pre_mp: 0
  layers_mp: 2
  layers_post_mp: 1
  dim_inner: 16
  layer_type: gcnconv
  stage_type: stack
  batchnorm: false
  act: prelu
  dropout: 0.1
  agg: mean
  normalize_adj: false
optim:
  optimizer: adam
  base_lr: 0.01
  max_epoch: 200
pytorch/graphgym/configs_gen.py000066400000000000000000000212121444262304300172170ustar00rootroot00000000000000import argparse
import copy
import csv
import os
import random

import numpy as np
import yaml

from torch_geometric.graphgym.utils.comp_budget import match_baseline_cfg
from torch_geometric.graphgym.utils.io import (
    makedirs_rm_exist,
    string_to_python,
)

random.seed(123)


def parse_args():
    """Parses the arguments."""
    parser = argparse.ArgumentParser()
    parser.add_argument('--config', dest='config',
                        help='the base configuration file used for edit',
                        default=None, type=str)
    parser.add_argument('--grid', dest='grid',
                        help='configuration file for grid search',
                        required=True, type=str)
    parser.add_argument('--sample_alias', dest='sample_alias',
                        help='configuration file for sample alias',
                        default=None, required=False, type=str)
    parser.add_argument('--sample_num', dest='sample_num',
                        help='Number of random samples in the space',
                        default=10, type=int)
    parser.add_argument('--out_dir', dest='out_dir',
                        help='output directory for generated config files',
                        default='configs', type=str)
    parser.add_argument(
        '--config_budget', dest='config_budget',
        help='the base configuration file used for matching computation',
        default=None, type=str)
    return parser.parse_args()


def get_fname(string):
    if string is not None:
        return string.split('/')[-1].split('.')[0]
    else:
        return 'default'


def grid2list(grid):
    list_in = [[]]
    for grid_temp in grid:
        list_out = []
        for val in grid_temp:
            for list_temp in list_in:
                list_out.append(list_temp + [val])
        list_in = list_out
    return list_in


def lists_distance(l1, l2):
    assert len(l1) == len(l2)
    dist = 0
    for i in range(len(l1)):
        if l1[i] != l2[i]:
            dist += 1
    return dist


def grid2list_sample(grid, sample=10):
    configs = []
    while len(configs) < sample:
        config = []
        for grid_temp in grid:
            config.append(random.choice(grid_temp))
        if config not in configs:
            configs.append(config)
    return configs


def load_config(fname):
    if fname is not None:
        with open(fname) as f:
            return yaml.load(f, Loader=yaml.FullLoader)
    else:
        return {}


def load_search_file(fname):
    with open(fname, 'r') as f:
        out_raw = csv.reader(f, delimiter=' ')
        outs = []
        out = []
        for row in out_raw:
            if '#' in row:
                continue
            elif len(row) > 0:
                assert len(row) == 3, \
                    'Exact 1 space between each grid argument file' \
                    'And no spaces within each argument is allowed'
                out.append(row)
            else:
                if len(out) > 0:
                    outs.append(out)
                out = []
        if len(out) > 0:
            outs.append(out)
    return outs


def load_alias_file(fname):
    with open(fname, 'r') as f:
        file = csv.reader(f, delimiter=' ')
        for line in file:
            break
    return line


def exclude_list_id(list, id):
    return [list[i] for i in range(len(list)) if i != id]


def gen_grid(args, config, config_budget={}):
    task_name = f'{get_fname(args.config)}_grid_{get_fname(args.grid)}'
    fname_start = get_fname(args.config)
    out_dir = f'{args.out_dir}/{task_name}'
    makedirs_rm_exist(out_dir)
    config['out_dir'] = os.path.join(config['out_dir'], task_name)

    outs = load_search_file(args.grid)
    for i, out in enumerate(outs):
        vars_label = [row[0].split('.') for row in out]
        vars_alias = [row[1] for row in out]
        vars_value = grid2list([string_to_python(row[2]) for row in out])
        if i == 0:
            print(f'Variable label: {vars_label}')
            print(f'Variable alias: {vars_alias}')

        for vars in vars_value:
            config_out = config.copy()
            fname_out = fname_start
            for id, var in enumerate(vars):
                if len(vars_label[id]) == 1:
                    config_out[vars_label[id][0]] = var
                elif len(vars_label[id]) == 2:
                    if vars_label[id][0] in config_out:  # if key1 exist
                        config_out[vars_label[id][0]][vars_label[id][1]] = var
                    else:
                        config_out[vars_label[id][0]] = {
                            vars_label[id][1]: var
                        }
                else:
                    raise ValueError('Only 2-level config files are supported')
                var_repr = str(var).strip("[]").strip("''")
                fname_out += f'-{vars_alias[id]}={var_repr}'
            if len(config_budget) > 0:
                config_out = match_baseline_cfg(config_out, config_budget)
            with open(f'{out_dir}/{fname_out}.yaml', 'w') as f:
                yaml.dump(config_out, f, default_flow_style=False)
        print(f'{len(vars_value)} configurations saved to: {out_dir}')


def gen_grid_sample(args, config, config_budget={}, compare_alias_list=[]):
    task_name = f'{get_fname(args.config)}_grid_{get_fname(args.grid)}'
    fname_start = get_fname(args.config)
    out_dir = f'{args.out_dir}/{task_name}'
    makedirs_rm_exist(out_dir)
    config['out_dir'] = os.path.join(config['out_dir'], task_name)
    outs = load_search_file(args.grid)

    counts = []
    for out in outs:
        vars_grid = [string_to_python(row[2]) for row in out]
        count = 1
        for var in vars_grid:
            count *= len(var)
        counts.append(count)
    counts = np.array(counts)
    print('Total size of each chunk of experiment space:', counts)
    counts = counts / np.sum(counts)
    counts = np.round(counts * args.sample_num)
    counts[0] += args.sample_num - np.sum(counts)
    print('Total sample size of each chunk of experiment space:', counts)

    for i, out in enumerate(outs):
        vars_label = [row[0].split('.') for row in out]
        vars_alias = [row[1] for row in out]
        if i == 0:
            print(f'Variable label: {vars_label}')
            print(f'Variable alias: {vars_alias}')
        vars_grid = [string_to_python(row[2]) for row in out]
        for alias in compare_alias_list:
            alias_id = vars_alias.index(alias)
            vars_grid_select = copy.deepcopy(vars_grid[alias_id])
            vars_grid[alias_id] = [vars_grid[alias_id][0]]
            vars_value = grid2list_sample(vars_grid, counts[i])

            vars_value_new = []
            for vars in vars_value:
                for grid in vars_grid_select:
                    vars[alias_id] = grid
                    vars_value_new.append(copy.deepcopy(vars))
            vars_value = vars_value_new

            vars_grid[alias_id] = vars_grid_select
            for vars in vars_value:
                config_out = config.copy()
                fname_out = fname_start + f'-sample={vars_alias[alias_id]}'
                for id, var in enumerate(vars):
                    if len(vars_label[id]) == 1:
                        config_out[vars_label[id][0]] = var
                    elif len(vars_label[id]) == 2:
                        if vars_label[id][0] in config_out:  # if key1 exist
                            config_out[vars_label[id][0]][vars_label[id]
                                                          [1]] = var
                        else:
                            config_out[vars_label[id][0]] = {
                                vars_label[id][1]: var
                            }
                    else:
                        raise ValueError(
                            'Only 2-level config files are supported')
                    var_repr = str(var).strip("[]").strip("''")
                    fname_out += f'-{vars_alias[id]}={var_repr}'
                if len(config_budget) > 0:
                    config_out = match_baseline_cfg(config_out, config_budget,
                                                    verbose=False)
                with open(f'{out_dir}/{fname_out}.yaml', "w") as f:
                    yaml.dump(config_out, f, default_flow_style=False)
            print(f'Chunk {i + 1}/{len(outs)}: '
                  f'Perturbing design dimension {alias}, '
                  f'{len(vars_value)} configurations saved to: {out_dir}')


args = parse_args()
config = load_config(args.config)
config_budget = load_config(args.config_budget)
if args.sample_alias is None:
    gen_grid(args, config, config_budget)
else:
    alias_list = load_alias_file(args.sample_alias)
    gen_grid_sample(args, config, config_budget, alias_list)
pytorch/graphgym/custom_graphgym/000077500000000000000000000000001444262304300175765ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/__init__.py000066400000000000000000000006051444262304300217100ustar00rootroot00000000000000from .act import *  # noqa
from .config import *  # noqa
from .encoder import *  # noqa
from .head import *  # noqa
from .layer import *  # noqa
from .loader import *  # noqa
from .loss import *  # noqa
from .network import *  # noqa
from .optimizer import *  # noqa
from .pooling import *  # noqa
from .stage import *  # noqa
from .train import *  # noqa
from .transform import *  # noqa
pytorch/graphgym/custom_graphgym/act/000077500000000000000000000000001444262304300203455ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/act/__init__.py000066400000000000000000000003351444262304300224570ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/act/example.py000066400000000000000000000011341444262304300223510ustar00rootroot00000000000000from functools import partial

import torch
import torch.nn as nn

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.register import register_act


class SWISH(nn.Module):
    def __init__(self, inplace=False):
        super().__init__()
        self.inplace = inplace

    def forward(self, x):
        if self.inplace:
            x.mul_(torch.sigmoid(x))
            return x
        else:
            return x * torch.sigmoid(x)


register_act('swish', partial(SWISH, inplace=cfg.mem.inplace))
register_act('lrelu_03', partial(nn.LeakyReLU, 0.3, inplace=cfg.mem.inplace))
pytorch/graphgym/custom_graphgym/config/000077500000000000000000000000001444262304300210435ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/config/__init__.py000066400000000000000000000003351444262304300231550ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/config/example.py000066400000000000000000000013041444262304300230460ustar00rootroot00000000000000from yacs.config import CfgNode as CN

from torch_geometric.graphgym.register import register_config


@register_config('example')
def set_cfg_example(cfg):
    r'''
    This function sets the default config value for customized options
    :return: customized configuration use by the experiment.
    '''

    # ----------------------------------------------------------------------- #
    # Customized options
    # ----------------------------------------------------------------------- #

    # example argument
    cfg.example_arg = 'example'

    # example argument group
    cfg.example_group = CN()

    # then argument can be specified within the group
    cfg.example_group.example_arg = 'example'
pytorch/graphgym/custom_graphgym/encoder/000077500000000000000000000000001444262304300212155ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/encoder/__init__.py000066400000000000000000000003351444262304300233270ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/encoder/example.py000066400000000000000000000027661444262304300232350ustar00rootroot00000000000000import torch
from ogb.utils.features import get_bond_feature_dims

from torch_geometric.graphgym.register import (
    register_edge_encoder,
    register_node_encoder,
)


@register_node_encoder('example')
class ExampleNodeEncoder(torch.nn.Module):
    """
        Provides an encoder for integer node features

        Parameters:
        num_classes - the number of classes for the embedding mapping to learn
    """
    def __init__(self, emb_dim, num_classes=None):
        super().__init__()

        self.encoder = torch.nn.Embedding(num_classes, emb_dim)
        torch.nn.init.xavier_uniform_(self.encoder.weight.data)

    def forward(self, batch):
        # Encode just the first dimension if more exist
        batch.x = self.encoder(batch.x[:, 0])

        return batch


@register_edge_encoder('example')
class ExampleEdgeEncoder(torch.nn.Module):
    def __init__(self, emb_dim):
        super().__init__()

        self.bond_embedding_list = torch.nn.ModuleList()
        full_bond_feature_dims = get_bond_feature_dims()

        for i, dim in enumerate(full_bond_feature_dims):
            emb = torch.nn.Embedding(dim, emb_dim)
            torch.nn.init.xavier_uniform_(emb.weight.data)
            self.bond_embedding_list.append(emb)

    def forward(self, batch):
        bond_embedding = 0
        for i in range(batch.edge_feature.shape[1]):
            bond_embedding += \
                self.bond_embedding_list[i](batch.edge_attr[:, i])

        batch.edge_attr = bond_embedding
        return batch
pytorch/graphgym/custom_graphgym/head/000077500000000000000000000000001444262304300204775ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/head/__init__.py000066400000000000000000000003351444262304300226110ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/head/example.py000066400000000000000000000013761444262304300225130ustar00rootroot00000000000000import torch.nn as nn

from torch_geometric.graphgym.register import register_head


@register_head('head')
class ExampleNodeHead(nn.Module):
    '''Head of GNN, node prediction'''
    def __init__(self, dim_in, dim_out):
        super().__init__()
        self.layer_post_mp = nn.Linear(dim_in, dim_out, bias=True)

    def _apply_index(self, batch):
        if batch.node_label_index.shape[0] == batch.node_label.shape[0]:
            return batch.x[batch.node_label_index], batch.node_label
        else:
            return batch.x[batch.node_label_index], \
                batch.node_label[batch.node_label_index]

    def forward(self, batch):
        batch = self.layer_post_mp(batch)
        pred, label = self._apply_index(batch)
        return pred, label
pytorch/graphgym/custom_graphgym/layer/000077500000000000000000000000001444262304300207125ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/layer/__init__.py000066400000000000000000000003351444262304300230240ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/layer/example.py000066400000000000000000000054301444262304300227210ustar00rootroot00000000000000import torch
import torch.nn as nn
from torch.nn import Parameter

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.register import register_layer
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import glorot, zeros

# Note: A registered GNN layer should take 'batch' as input
# and 'batch' as output


# Example 1: Directly define a GraphGym format Conv
# take 'batch' as input and 'batch' as output
@register_layer('exampleconv1')
class ExampleConv1(MessagePassing):
    r"""Example GNN layer

    """
    def __init__(self, in_channels, out_channels, bias=True, **kwargs):
        super().__init__(aggr=cfg.gnn.agg, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels

        self.weight = Parameter(torch.Tensor(in_channels, out_channels))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        glorot(self.weight)
        zeros(self.bias)

    def forward(self, batch):
        x, edge_index = batch.x, batch.edge_index
        x = torch.matmul(x, self.weight)

        batch.x = self.propagate(edge_index, x=x)

        return batch

    def message(self, x_j):
        return x_j

    def update(self, aggr_out):
        if self.bias is not None:
            aggr_out = aggr_out + self.bias
        return aggr_out


# Example 2: First define a PyG format Conv layer
# Then wrap it to become GraphGym format
class ExampleConv2Layer(MessagePassing):
    r"""Example GNN layer

    """
    def __init__(self, in_channels, out_channels, bias=True, **kwargs):
        super().__init__(aggr=cfg.gnn.agg, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels

        self.weight = Parameter(torch.Tensor(in_channels, out_channels))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        glorot(self.weight)
        zeros(self.bias)

    def forward(self, x, edge_index):
        x = torch.matmul(x, self.weight)

        return self.propagate(edge_index, x=x)

    def message(self, x_j):
        return x_j

    def update(self, aggr_out):
        if self.bias is not None:
            aggr_out = aggr_out + self.bias
        return aggr_out


@register_layer('exampleconv2')
class ExampleConv2(nn.Module):
    def __init__(self, dim_in, dim_out, bias=False, **kwargs):
        super().__init__()
        self.model = ExampleConv2Layer(dim_in, dim_out, bias=bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch
pytorch/graphgym/custom_graphgym/loader/000077500000000000000000000000001444262304300210445ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/loader/__init__.py000066400000000000000000000003351444262304300231560ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/loader/example.py000066400000000000000000000005431444262304300230530ustar00rootroot00000000000000from torch_geometric.datasets import QM7b
from torch_geometric.graphgym.register import register_loader


@register_loader('example')
def load_dataset_example(format, name, dataset_dir):
    dataset_dir = f'{dataset_dir}/{name}'
    if format == 'PyG':
        if name == 'QM7b':
            dataset_raw = QM7b(dataset_dir)
            return dataset_raw
pytorch/graphgym/custom_graphgym/loss/000077500000000000000000000000001444262304300205565ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/loss/__init__.py000066400000000000000000000003351444262304300226700ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/loss/example.py000066400000000000000000000005051444262304300225630ustar00rootroot00000000000000import torch

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.register import register_loss


@register_loss('smoothl1')
def loss_example(pred, true):
    if cfg.model.loss_fun == 'smoothl1':
        l1_loss = torch.nn.SmoothL1Loss()
        loss = l1_loss(pred, true)
        return loss, pred
pytorch/graphgym/custom_graphgym/network/000077500000000000000000000000001444262304300212675ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/network/__init__.py000066400000000000000000000003351444262304300234010ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/network/example.py000066400000000000000000000027741444262304300233060ustar00rootroot00000000000000import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.graphgym.models.head  # noqa, register module
import torch_geometric.graphgym.register as register
import torch_geometric.nn as pyg_nn
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.register import register_network


@register_network('example')
class ExampleGNN(torch.nn.Module):
    def __init__(self, dim_in, dim_out, num_layers=2, model_type='GCN'):
        super().__init__()
        conv_model = self.build_conv_model(model_type)
        self.convs = nn.ModuleList()
        self.convs.append(conv_model(dim_in, dim_in))

        for _ in range(num_layers - 1):
            self.convs.append(conv_model(dim_in, dim_in))

        GNNHead = register.head_dict[cfg.dataset.task]
        self.post_mp = GNNHead(dim_in=dim_in, dim_out=dim_out)

    def build_conv_model(self, model_type):
        if model_type == 'GCN':
            return pyg_nn.GCNConv
        elif model_type == 'GAT':
            return pyg_nn.GATConv
        elif model_type == "GraphSage":
            return pyg_nn.SAGEConv
        else:
            raise ValueError(f'Model {model_type} unavailable')

    def forward(self, batch):
        x, edge_index = batch.x, batch.edge_index

        for i in range(len(self.convs)):
            x = self.convs[i](x, edge_index)
            x = F.relu(x)
            x = F.dropout(x, p=0.1, training=self.training)

        batch.x = x
        batch = self.post_mp(batch)

        return batch
pytorch/graphgym/custom_graphgym/optimizer/000077500000000000000000000000001444262304300216205ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/optimizer/__init__.py000066400000000000000000000003351444262304300237320ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/optimizer/example.py000066400000000000000000000012511444262304300236240ustar00rootroot00000000000000from typing import Iterator

from torch.nn import Parameter
from torch.optim import Adagrad, Optimizer
from torch.optim.lr_scheduler import ReduceLROnPlateau

import torch_geometric.graphgym.register as register


@register.register_optimizer('adagrad')
def adagrad_optimizer(params: Iterator[Parameter], base_lr: float,
                      weight_decay: float) -> Adagrad:
    return Adagrad(params, lr=base_lr, weight_decay=weight_decay)


@register.register_scheduler('pleateau')
def plateau_scheduler(optimizer: Optimizer, patience: int,
                      lr_decay: float) -> ReduceLROnPlateau:
    return ReduceLROnPlateau(optimizer, patience=patience, factor=lr_decay)
pytorch/graphgym/custom_graphgym/pooling/000077500000000000000000000000001444262304300212455ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/pooling/__init__.py000066400000000000000000000003351444262304300233570ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/pooling/example.py000066400000000000000000000004631444262304300232550ustar00rootroot00000000000000from torch_geometric.graphgym.register import register_pooling
from torch_geometric.utils import scatter


@register_pooling('example')
def global_example_pool(x, batch, size=None):
    size = batch.max().item() + 1 if size is None else size
    return scatter(x, batch, dim=0, dim_size=size, reduce='sum')
pytorch/graphgym/custom_graphgym/stage/000077500000000000000000000000001444262304300207015ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/stage/__init__.py000066400000000000000000000003351444262304300230130ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/stage/example.py000066400000000000000000000016601444262304300227110ustar00rootroot00000000000000import torch.nn as nn
import torch.nn.functional as F

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.models.layer import GeneralLayer
from torch_geometric.graphgym.register import register_stage


def GNNLayer(dim_in, dim_out, has_act=True):
    return GeneralLayer(cfg.gnn.layer_type, dim_in, dim_out, has_act)


@register_stage('example')
class GNNStackStage(nn.Module):
    '''Simple Stage that stack GNN layers'''
    def __init__(self, dim_in, dim_out, num_layers):
        super().__init__()
        for i in range(num_layers):
            d_in = dim_in if i == 0 else dim_out
            layer = GNNLayer(d_in, dim_out)
            self.add_module(f'layer{i}', layer)
        self.dim_out = dim_out

    def forward(self, batch):
        for layer in self.children():
            batch = layer(batch)
        if cfg.gnn.l2norm:
            batch.x = F.normalize(batch.x, p=2, dim=-1)
        return batch
pytorch/graphgym/custom_graphgym/train/000077500000000000000000000000001444262304300207135ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/train/__init__.py000066400000000000000000000003351444262304300230250ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/custom_graphgym/train/example.py000066400000000000000000000052111444262304300227170ustar00rootroot00000000000000import logging
import time

import torch

from torch_geometric.graphgym.checkpoint import (
    clean_ckpt,
    load_ckpt,
    save_ckpt,
)
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.loss import compute_loss
from torch_geometric.graphgym.register import register_train
from torch_geometric.graphgym.utils.epoch import is_ckpt_epoch, is_eval_epoch


def train_epoch(logger, loader, model, optimizer, scheduler):
    model.train()
    time_start = time.time()
    for batch in loader:
        optimizer.zero_grad()
        batch.to(torch.device(cfg.device))
        pred, true = model(batch)
        loss, pred_score = compute_loss(pred, true)
        loss.backward()
        optimizer.step()
        logger.update_stats(true=true.detach().cpu(),
                            pred=pred_score.detach().cpu(), loss=loss.item(),
                            lr=scheduler.get_last_lr()[0],
                            time_used=time.time() - time_start,
                            params=cfg.params)
        time_start = time.time()
    scheduler.step()


def eval_epoch(logger, loader, model):
    model.eval()
    time_start = time.time()
    for batch in loader:
        batch.to(torch.device(cfg.device))
        pred, true = model(batch)
        loss, pred_score = compute_loss(pred, true)
        logger.update_stats(true=true.detach().cpu(),
                            pred=pred_score.detach().cpu(), loss=loss.item(),
                            lr=0, time_used=time.time() - time_start,
                            params=cfg.params)
        time_start = time.time()


@register_train('example')
def train_example(loggers, loaders, model, optimizer, scheduler):
    start_epoch = 0
    if cfg.train.auto_resume:
        start_epoch = load_ckpt(model, optimizer, scheduler,
                                cfg.train.epoch_resume)
    if start_epoch == cfg.optim.max_epoch:
        logging.info('Checkpoint found, Task already done')
    else:
        logging.info('Start from epoch %s', start_epoch)

    num_splits = len(loggers)
    for cur_epoch in range(start_epoch, cfg.optim.max_epoch):
        train_epoch(loggers[0], loaders[0], model, optimizer, scheduler)
        loggers[0].write_epoch(cur_epoch)
        if is_eval_epoch(cur_epoch):
            for i in range(1, num_splits):
                eval_epoch(loggers[i], loaders[i], model)
                loggers[i].write_epoch(cur_epoch)
        if is_ckpt_epoch(cur_epoch):
            save_ckpt(model, optimizer, scheduler, cur_epoch)
    for logger in loggers:
        logger.close()
    if cfg.train.ckpt_clean:
        clean_ckpt()

    logging.info('Task done, results saved in %s', cfg.run_dir)
pytorch/graphgym/custom_graphgym/transform/000077500000000000000000000000001444262304300216115ustar00rootroot00000000000000pytorch/graphgym/custom_graphgym/transform/__init__.py000066400000000000000000000003351444262304300237230ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/graphgym/grids/000077500000000000000000000000001444262304300154765ustar00rootroot00000000000000pytorch/graphgym/grids/example.txt000066400000000000000000000011431444262304300176710ustar00rootroot00000000000000# Format for each row: name in config.py; alias; range to search
# No spaces, except between these 3 fields
# Line breaks are used to union different grid search spaces
# Feel free to add '#' to add comments


# (1) dataset configurations
dataset.format format ['PyG']
dataset.name dataset ['TU_ENZYMES','TU_PROTEINS']
dataset.task task ['graph']
dataset.transductive trans [False]
# (2) The recommended GNN design space, 96 models in total
gnn.layers_pre_mp l_pre [1,2]
gnn.layers_mp l_mp [2,4,6,8]
gnn.layers_post_mp l_post [2,3]
gnn.stage_type stage ['skipsum','skipconcat']
gnn.agg agg ['add','mean','max']
pytorch/graphgym/grids/pyg/000077500000000000000000000000001444262304300162755ustar00rootroot00000000000000pytorch/graphgym/grids/pyg/example.txt000066400000000000000000000006541444262304300204760ustar00rootroot00000000000000# Format for each row: name in config.py; alias; range to search
# No spaces, except between these 3 fields
# Line breaks are used to union different grid search spaces
# Feel free to add '#' to add comments


gnn.layers_pre_mp l_pre [1,2]
gnn.layers_mp l_mp [2,4,6]
gnn.layers_post_mp l_post [1,2]
gnn.stage_type stage ['stack','skipsum','skipconcat']
gnn.dim_inner dim [64]
optim.base_lr lr [0.01]
optim.max_epoch epoch [200]
pytorch/graphgym/main.py000066400000000000000000000034041444262304300156650ustar00rootroot00000000000000import logging
import os

import custom_graphgym  # noqa, register custom modules
import torch

from torch_geometric import seed_everything
from torch_geometric.graphgym.cmd_args import parse_args
from torch_geometric.graphgym.config import (
    cfg,
    dump_cfg,
    load_cfg,
    set_out_dir,
    set_run_dir,
)
from torch_geometric.graphgym.logger import set_printing
from torch_geometric.graphgym.model_builder import create_model
from torch_geometric.graphgym.train import GraphGymDataModule, train
from torch_geometric.graphgym.utils.agg_runs import agg_runs
from torch_geometric.graphgym.utils.comp_budget import params_count
from torch_geometric.graphgym.utils.device import auto_select_device

if __name__ == '__main__':
    # Load cmd line args
    args = parse_args()
    # Load config file
    load_cfg(cfg, args)
    set_out_dir(cfg.out_dir, args.cfg_file)
    # Set Pytorch environment
    torch.set_num_threads(cfg.num_threads)
    dump_cfg(cfg)
    # Repeat for different random seeds
    for i in range(args.repeat):
        set_run_dir(cfg.out_dir)
        set_printing()
        # Set configurations for each run
        cfg.seed = cfg.seed + 1
        seed_everything(cfg.seed)
        auto_select_device()
        # Set machine learning pipeline
        datamodule = GraphGymDataModule()
        model = create_model()
        # Print model info
        logging.info(model)
        logging.info(cfg)
        cfg.params = params_count(model)
        logging.info('Num parameters: %s', cfg.params)
        train(model, datamodule, logger=True)

    # Aggregate results from different seeds
    agg_runs(cfg.out_dir, cfg.metric_best)
    # When being launched in batch mode, mark a yaml as done
    if args.mark_done:
        os.rename(args.cfg_file, f'{args.cfg_file}_done')
pytorch/graphgym/parallel.sh000066400000000000000000000006121444262304300165150ustar00rootroot00000000000000CONFIG_DIR=$1
REPEAT=$2
MAX_JOBS=${3:-2}
SLEEP=${4:-1}
MAIN=${5:-main}

(
  trap 'kill 0' SIGINT
  CUR_JOBS=0
  for CONFIG in "$CONFIG_DIR"/*.yaml; do
    if [ "$CONFIG" != "$CONFIG_DIR/*.yaml" ]; then
      ((CUR_JOBS >= MAX_JOBS)) && wait -n
      python $MAIN.py --cfg $CONFIG --repeat $REPEAT --mark_done &
      echo $CONFIG
      sleep $SLEEP
      ((++CUR_JOBS))
    fi
  done

  wait
)
pytorch/graphgym/results/000077500000000000000000000000001444262304300160675ustar00rootroot00000000000000pytorch/graphgym/results/example_node_grid_example/000077500000000000000000000000001444262304300232475ustar00rootroot00000000000000pytorch/graphgym/results/example_node_grid_example/agg/000077500000000000000000000000001444262304300240055ustar00rootroot00000000000000pytorch/graphgym/results/example_node_grid_example/agg/test.csv000066400000000000000000000054461444262304300255120ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,199,0.9023,0.0044,117128.0,0.009,0.0022,0.735,0.0043
1,2,1,skipsum,64,0.01,199,1.0758,0.0525,108936.0,0.0154,0.0096,0.6883,0.0167
1,2,1,stack,64,0.01,199,0.8435,0.0093,108936.0,0.0171,0.0064,0.7633,0.0065
1,2,2,skipconcat,64,0.01,199,1.5254,0.0357,121352.0,0.0075,0.0021,0.7013,0.0061
1,2,2,skipsum,64,0.01,199,1.4988,0.0776,113160.0,0.0147,0.001,0.6547,0.0215
1,2,2,stack,64,0.01,199,1.403,0.0358,113160.0,0.014,0.0035,0.7267,0.0063
1,4,1,skipconcat,64,0.01,199,0.848,0.0294,174728.0,0.0066,0.0013,0.761,0.012
1,4,1,skipsum,64,0.01,199,0.9693,0.042,125576.0,0.0278,0.0161,0.7087,0.0152
1,4,1,stack,64,0.01,199,0.8618,0.0592,125576.0,0.0113,0.0025,0.7767,0.0155
1,4,2,skipconcat,64,0.01,199,1.3837,0.0563,178952.0,0.0107,0.0019,0.7303,0.0201
1,4,2,skipsum,64,0.01,199,1.4925,0.0543,129800.0,0.0084,0.001,0.6687,0.0115
1,4,2,stack,64,0.01,199,1.582,0.0665,129800.0,0.0082,0.0006,0.747,0.0149
1,6,1,skipconcat,64,0.01,199,0.8481,0.0145,265096.0,0.0091,0.002,0.76,0.0033
1,6,1,skipsum,64,0.01,199,0.9459,0.0473,142216.0,0.0201,0.0007,0.7327,0.0119
1,6,1,stack,64,0.01,199,0.9047,0.0376,142216.0,0.0106,0.0008,0.7683,0.0141
1,6,2,skipconcat,64,0.01,199,1.2463,0.0601,269320.0,0.0237,0.0105,0.7543,0.0177
1,6,2,skipsum,64,0.01,199,1.4777,0.0305,146440.0,0.0189,0.0036,0.6987,0.0071
1,6,2,stack,64,0.01,199,1.7604,0.0518,146440.0,0.0166,0.0036,0.74,0.0079
2,2,1,skipconcat,64,0.01,199,0.9243,0.0162,121352.0,0.0145,0.007,0.734,0.0043
2,2,1,skipsum,64,0.01,199,1.05,0.0274,113160.0,0.0088,0.0045,0.686,0.0135
2,2,1,stack,64,0.01,199,0.8104,0.0149,113160.0,0.0045,0.0002,0.771,0.0022
2,2,2,skipconcat,64,0.01,199,1.5122,0.0801,125576.0,0.0195,0.0042,0.7137,0.0021
2,2,2,skipsum,64,0.01,199,1.6425,0.0708,117384.0,0.0111,0.0023,0.6497,0.0137
2,2,2,stack,64,0.01,199,1.4422,0.1121,117384.0,0.0141,0.0065,0.731,0.0248
2,4,1,skipconcat,64,0.01,199,0.9101,0.0135,178952.0,0.0141,0.0026,0.7437,0.009
2,4,1,skipsum,64,0.01,199,1.0268,0.0254,129800.0,0.0131,0.008,0.6983,0.0005
2,4,1,stack,64,0.01,199,0.8549,0.0273,129800.0,0.0133,0.0057,0.778,0.0088
2,4,2,skipconcat,64,0.01,199,1.4913,0.0804,183176.0,0.0119,0.0022,0.716,0.0205
2,4,2,skipsum,64,0.01,199,1.3682,0.0238,134024.0,0.0069,0.0004,0.7057,0.0098
2,4,2,stack,64,0.01,199,1.4885,0.0288,134024.0,0.0091,0.0022,0.7623,0.0031
2,6,1,skipconcat,64,0.01,199,0.8715,0.0474,269320.0,0.0118,0.0027,0.7587,0.014
2,6,1,skipsum,64,0.01,199,0.9767,0.0409,146440.0,0.0091,0.0017,0.729,0.0096
2,6,1,stack,64,0.01,199,0.9098,0.0871,146440.0,0.0138,0.005,0.7737,0.0189
2,6,2,skipconcat,64,0.01,199,1.4138,0.0312,273544.0,0.0155,0.0026,0.748,0.0036
2,6,2,skipsum,64,0.01,199,1.4558,0.1642,150664.0,0.0132,0.0012,0.6967,0.0344
2,6,2,stack,64,0.01,199,1.6375,0.1628,150664.0,0.0118,0.0026,0.75,0.0224
pytorch/graphgym/results/example_node_grid_example/agg/test_best.csv000066400000000000000000000054151444262304300265230ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,35,0.8785,0.025,117128.0,0.0093,0.0011,0.7297,0.0042
1,2,1,skipsum,64,0.01,32,1.0665,0.0663,108936.0,0.0097,0.005,0.6803,0.0208
1,2,1,stack,64,0.01,74,0.8302,0.0157,108936.0,0.0161,0.0029,0.7633,0.0068
1,2,2,skipconcat,64,0.01,70,1.5818,0.0575,121352.0,0.0073,0.0006,0.6977,0.0105
1,2,2,skipsum,64,0.01,34,1.2786,0.131,113160.0,0.0199,0.0036,0.662,0.0252
1,2,2,stack,64,0.01,48,1.4485,0.0237,113160.0,0.0218,0.0032,0.7227,0.0046
1,4,1,skipconcat,64,0.01,167,0.822,0.0288,174728.0,0.0083,0.0019,0.7663,0.0103
1,4,1,skipsum,64,0.01,194,1.0017,0.0975,125576.0,0.0156,0.006,0.7053,0.0142
1,4,1,stack,64,0.01,106,0.8089,0.0359,125576.0,0.0118,0.0019,0.77,0.0227
1,4,2,skipconcat,64,0.01,47,1.401,0.0394,178952.0,0.0114,0.0023,0.7297,0.0098
1,4,2,skipsum,64,0.01,40,1.3821,0.1176,129800.0,0.0098,0.0026,0.6733,0.009
1,4,2,stack,64,0.01,176,1.35,0.345,129800.0,0.0084,0.0016,0.7493,0.0169
1,6,1,skipconcat,64,0.01,73,0.837,0.0146,265096.0,0.0107,0.0029,0.7587,0.005
1,6,1,skipsum,64,0.01,32,0.9428,0.0471,142216.0,0.0135,0.0012,0.726,0.0108
1,6,1,stack,64,0.01,86,0.8625,0.0637,142216.0,0.0134,0.0036,0.767,0.0113
1,6,2,skipconcat,64,0.01,20,1.0512,0.2101,269320.0,0.0167,0.0025,0.7513,0.018
1,6,2,skipsum,64,0.01,35,1.3902,0.0995,146440.0,0.019,0.0042,0.6967,0.0092
1,6,2,stack,64,0.01,191,1.4292,0.4493,146440.0,0.0171,0.0004,0.7467,0.0165
2,2,1,skipconcat,64,0.01,148,0.8838,0.0424,121352.0,0.0147,0.0034,0.7403,0.0052
2,2,1,skipsum,64,0.01,27,1.0086,0.0355,113160.0,0.0088,0.0029,0.6747,0.0153
2,2,1,stack,64,0.01,137,0.7704,0.0363,113160.0,0.0064,0.001,0.7707,0.0098
2,2,2,skipconcat,64,0.01,108,1.36,0.3055,125576.0,0.0094,0.003,0.7143,0.0103
2,2,2,skipsum,64,0.01,71,1.6004,0.1345,117384.0,0.0123,0.005,0.649,0.0142
2,2,2,stack,64,0.01,80,1.4425,0.0911,117384.0,0.0149,0.0031,0.7277,0.0258
2,4,1,skipconcat,64,0.01,86,0.8901,0.0257,178952.0,0.012,0.0022,0.7443,0.0085
2,4,1,skipsum,64,0.01,173,1.0763,0.1099,129800.0,0.0114,0.0044,0.6927,0.0038
2,4,1,stack,64,0.01,59,0.8332,0.0466,129800.0,0.0222,0.0109,0.7757,0.0074
2,4,2,skipconcat,64,0.01,9,1.2834,0.2037,183176.0,0.0132,0.0029,0.7133,0.0197
2,4,2,skipsum,64,0.01,27,1.2735,0.1646,134024.0,0.0111,0.0047,0.71,0.0222
2,4,2,stack,64,0.01,105,1.2851,0.2719,134024.0,0.01,0.0027,0.7597,0.0141
2,6,1,skipconcat,64,0.01,92,0.8324,0.0181,269320.0,0.0138,0.0071,0.7637,0.0091
2,6,1,skipsum,64,0.01,37,0.9787,0.0625,146440.0,0.0082,0.0019,0.719,0.0079
2,6,1,stack,64,0.01,122,0.8407,0.0848,146440.0,0.0111,0.0049,0.77,0.0214
2,6,2,skipconcat,64,0.01,21,0.9791,0.0546,273544.0,0.0126,0.0011,0.7373,0.0012
2,6,2,skipsum,64,0.01,10,1.2423,0.3052,150664.0,0.0132,0.0028,0.7147,0.0449
2,6,2,stack,64,0.01,75,1.6255,0.1883,150664.0,0.0124,0.0024,0.7467,0.0233
pytorch/graphgym/results/example_node_grid_example/agg/test_bestepoch.csv000066400000000000000000000054331444262304300275420ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,151,0.8951,0.0066,117128.0,0.0105,0.0015,0.7373,0.0053
1,2,1,skipsum,64,0.01,179,1.0776,0.0515,108936.0,0.0075,0.0011,0.6893,0.0172
1,2,1,stack,64,0.01,145,0.8396,0.0126,108936.0,0.0167,0.0049,0.766,0.0083
1,2,2,skipconcat,64,0.01,194,1.5253,0.0358,121352.0,0.0101,0.0044,0.702,0.0057
1,2,2,skipsum,64,0.01,35,1.3806,0.0881,113160.0,0.0191,0.0039,0.6673,0.019
1,2,2,stack,64,0.01,178,1.3988,0.0353,113160.0,0.0157,0.0038,0.728,0.0059
1,4,1,skipconcat,64,0.01,101,0.8278,0.0208,174728.0,0.0114,0.0025,0.7663,0.012
1,4,1,skipsum,64,0.01,167,0.9749,0.0464,125576.0,0.0247,0.017,0.7093,0.0167
1,4,1,stack,64,0.01,173,0.8621,0.0625,125576.0,0.0134,0.0036,0.778,0.0159
1,4,2,skipconcat,64,0.01,45,1.4025,0.0445,178952.0,0.0105,0.0025,0.7313,0.011
1,4,2,skipsum,64,0.01,69,1.5035,0.0562,129800.0,0.0083,0.0019,0.6783,0.009
1,4,2,stack,64,0.01,25,1.2012,0.0141,129800.0,0.0138,0.0086,0.75,0.0107
1,6,1,skipconcat,64,0.01,171,0.8471,0.0152,265096.0,0.0116,0.0038,0.761,0.0028
1,6,1,skipsum,64,0.01,190,0.9466,0.0473,142216.0,0.0176,0.0053,0.733,0.0124
1,6,1,stack,64,0.01,166,0.8976,0.0361,142216.0,0.0155,0.004,0.771,0.0131
1,6,2,skipconcat,64,0.01,196,1.2471,0.0595,269320.0,0.0193,0.0057,0.7543,0.0186
1,6,2,skipsum,64,0.01,49,1.4946,0.0375,146440.0,0.0149,0.0061,0.7013,0.0037
1,6,2,stack,64,0.01,184,1.754,0.0538,146440.0,0.0152,0.0044,0.7407,0.0074
2,2,1,skipconcat,64,0.01,140,0.9168,0.0218,121352.0,0.013,0.0018,0.74,0.0033
2,2,1,skipsum,64,0.01,197,1.0497,0.0276,113160.0,0.0066,0.0011,0.687,0.0142
2,2,1,stack,64,0.01,123,0.7874,0.0201,113160.0,0.0064,0.0014,0.7763,0.0021
2,2,2,skipconcat,64,0.01,38,1.4176,0.1167,125576.0,0.0132,0.0051,0.7153,0.0128
2,2,2,skipsum,64,0.01,51,1.6633,0.0889,117384.0,0.0123,0.0035,0.6547,0.0172
2,2,2,stack,64,0.01,193,1.442,0.1116,117384.0,0.0208,0.0017,0.7313,0.0245
2,4,1,skipconcat,64,0.01,138,0.9106,0.0237,178952.0,0.0125,0.0029,0.745,0.0078
2,4,1,skipsum,64,0.01,166,1.0312,0.026,129800.0,0.0173,0.0134,0.6993,0.0021
2,4,1,stack,64,0.01,163,0.8533,0.0355,129800.0,0.0139,0.0012,0.7793,0.0108
2,4,2,skipconcat,64,0.01,15,0.9239,0.0392,183176.0,0.0147,0.0005,0.7183,0.0233
2,4,2,skipsum,64,0.01,16,0.8927,0.0487,134024.0,0.0097,0.0037,0.725,0.0212
2,4,2,stack,64,0.01,108,1.4865,0.0188,134024.0,0.012,0.003,0.765,0.0078
2,6,1,skipconcat,64,0.01,183,0.8717,0.0467,269320.0,0.0116,0.0013,0.7597,0.0132
2,6,1,skipsum,64,0.01,191,0.9782,0.0413,146440.0,0.0079,0.0,0.729,0.0096
2,6,1,stack,64,0.01,139,0.8858,0.1117,146440.0,0.0356,0.0401,0.7767,0.021
2,6,2,skipconcat,64,0.01,115,1.4276,0.0397,273544.0,0.0142,0.0007,0.749,0.0014
2,6,2,skipsum,64,0.01,51,1.4454,0.1903,150664.0,0.0202,0.0064,0.703,0.0372
2,6,2,stack,64,0.01,137,1.6217,0.177,150664.0,0.0101,0.0014,0.7513,0.0216
pytorch/graphgym/results/example_node_grid_example/agg/train.csv000066400000000000000000000051201444262304300256350ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,199,0.0292,0.0003,117128.0,0.017,0.0033,1.0,0.0
1,2,1,skipsum,64,0.01,199,0.0263,0.0004,108936.0,0.0135,0.0008,1.0,0.0
1,2,1,stack,64,0.01,199,0.03,0.0004,108936.0,0.0292,0.0117,1.0,0.0
1,2,2,skipconcat,64,0.01,199,0.0008,0.0001,121352.0,0.015,0.0018,1.0,0.0
1,2,2,skipsum,64,0.01,199,0.0009,0.0001,113160.0,0.0312,0.0102,1.0,0.0
1,2,2,stack,64,0.01,199,0.0009,0.0001,113160.0,0.0247,0.0035,1.0,0.0
1,4,1,skipconcat,64,0.01,199,0.0291,0.0005,174728.0,0.0194,0.0033,1.0,0.0
1,4,1,skipsum,64,0.01,199,0.0258,0.0006,125576.0,0.0354,0.0059,1.0,0.0
1,4,1,stack,64,0.01,199,0.0315,0.0012,125576.0,0.0182,0.001,1.0,0.0
1,4,2,skipconcat,64,0.01,199,0.001,0.0,178952.0,0.0237,0.0059,1.0,0.0
1,4,2,skipsum,64,0.01,199,0.0009,0.0001,129800.0,0.0233,0.008,1.0,0.0
1,4,2,stack,64,0.01,199,0.0008,0.0,129800.0,0.0197,0.0025,1.0,0.0
1,6,1,skipconcat,64,0.01,199,0.0295,0.0007,265096.0,0.0215,0.0016,1.0,0.0
1,6,1,skipsum,64,0.01,199,0.0261,0.0007,142216.0,0.037,0.0109,1.0,0.0
1,6,1,stack,64,0.01,199,0.0337,0.0012,142216.0,0.028,0.0051,1.0,0.0
1,6,2,skipconcat,64,0.01,199,0.0009,0.0001,269320.0,0.0325,0.006,1.0,0.0
1,6,2,skipsum,64,0.01,199,0.0007,0.0,146440.0,0.0321,0.0009,1.0,0.0
1,6,2,stack,64,0.01,199,0.0008,0.0,146440.0,0.0312,0.0069,1.0,0.0
2,2,1,skipconcat,64,0.01,199,0.0295,0.0005,121352.0,0.0305,0.0121,1.0,0.0
2,2,1,skipsum,64,0.01,199,0.0263,0.0002,113160.0,0.0116,0.0007,1.0,0.0
2,2,1,stack,64,0.01,199,0.0297,0.0007,113160.0,0.0113,0.0007,1.0,0.0
2,2,2,skipconcat,64,0.01,199,0.0008,0.0002,125576.0,0.0241,0.0025,1.0,0.0
2,2,2,skipsum,64,0.01,199,0.0009,0.0001,117384.0,0.0247,0.0037,1.0,0.0
2,2,2,stack,64,0.01,199,0.0014,0.0007,117384.0,0.0261,0.0035,1.0,0.0
2,4,1,skipconcat,64,0.01,199,0.03,0.0005,178952.0,0.0253,0.0036,1.0,0.0
2,4,1,skipsum,64,0.01,199,0.0262,0.0008,129800.0,0.0211,0.0063,1.0,0.0
2,4,1,stack,64,0.01,199,0.0318,0.0007,129800.0,0.0457,0.025,1.0,0.0
2,4,2,skipconcat,64,0.01,199,0.0009,0.0,183176.0,0.0253,0.0034,1.0,0.0
2,4,2,skipsum,64,0.01,199,0.0008,0.0001,134024.0,0.0158,0.0006,1.0,0.0
2,4,2,stack,64,0.01,199,0.001,0.0004,134024.0,0.0202,0.0012,1.0,0.0
2,6,1,skipconcat,64,0.01,199,0.031,0.0008,269320.0,0.0214,0.0026,1.0,0.0
2,6,1,skipsum,64,0.01,199,0.0266,0.0002,146440.0,0.0214,0.0038,1.0,0.0
2,6,1,stack,64,0.01,199,0.0336,0.0004,146440.0,0.0237,0.0027,1.0,0.0
2,6,2,skipconcat,64,0.01,199,0.0009,0.0002,273544.0,0.032,0.0064,1.0,0.0
2,6,2,skipsum,64,0.01,199,0.0008,0.0001,150664.0,0.0272,0.0054,1.0,0.0
2,6,2,stack,64,0.01,199,0.001,0.0002,150664.0,0.0228,0.0021,1.0,0.0
pytorch/graphgym/results/example_node_grid_example/agg/train_best.csv000066400000000000000000000051031444262304300266530ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,35,0.074,0.0347,117128.0,0.0175,0.0076,1.0,0.0
1,2,1,skipsum,64,0.01,32,0.2247,0.0759,108936.0,0.0159,0.0089,1.0,0.0
1,2,1,stack,64,0.01,74,0.0371,0.0069,108936.0,0.0242,0.0018,1.0,0.0
1,2,2,skipconcat,64,0.01,70,0.0007,0.0001,121352.0,0.0161,0.0043,1.0,0.0
1,2,2,skipsum,64,0.01,34,0.0013,0.001,113160.0,0.0246,0.0054,1.0,0.0
1,2,2,stack,64,0.01,48,0.0007,0.0001,113160.0,0.0277,0.0072,1.0,0.0
1,4,1,skipconcat,64,0.01,167,0.0336,0.0044,174728.0,0.0177,0.0014,1.0,0.0
1,4,1,skipsum,64,0.01,194,0.2488,0.2567,125576.0,0.0257,0.0078,1.0,0.0
1,4,1,stack,64,0.01,106,0.0787,0.0517,125576.0,0.0181,0.0016,1.0,0.0
1,4,2,skipconcat,64,0.01,47,0.0008,0.0001,178952.0,0.0258,0.01,1.0,0.0
1,4,2,skipsum,64,0.01,40,0.0007,0.0001,129800.0,0.018,0.0019,1.0,0.0
1,4,2,stack,64,0.01,176,0.0023,0.0024,129800.0,0.0188,0.0035,1.0,0.0
1,6,1,skipconcat,64,0.01,73,0.036,0.0077,265096.0,0.0203,0.0013,1.0,0.0
1,6,1,skipsum,64,0.01,32,0.0604,0.0474,142216.0,0.0263,0.0037,1.0,0.0
1,6,1,stack,64,0.01,86,0.2398,0.2512,142216.0,0.0244,0.0047,1.0,0.0
1,6,2,skipconcat,64,0.01,20,0.0013,0.0004,269320.0,0.0275,0.0045,1.0,0.0
1,6,2,skipsum,64,0.01,35,0.0007,0.0001,146440.0,0.0353,0.0088,1.0,0.0
1,6,2,stack,64,0.01,191,0.0038,0.004,146440.0,0.0317,0.0015,1.0,0.0
2,2,1,skipconcat,64,0.01,148,0.1225,0.0693,121352.0,0.0215,0.0055,1.0,0.0
2,2,1,skipsum,64,0.01,27,0.1388,0.0247,113160.0,0.0149,0.0038,1.0,0.0
2,2,1,stack,64,0.01,137,0.0502,0.0135,113160.0,0.011,0.0001,1.0,0.0
2,2,2,skipconcat,64,0.01,108,0.0011,0.0005,125576.0,0.0196,0.0037,1.0,0.0
2,2,2,skipsum,64,0.01,71,0.0007,0.0,117384.0,0.0226,0.0051,1.0,0.0
2,2,2,stack,64,0.01,80,0.0009,0.0002,117384.0,0.0261,0.0017,1.0,0.0
2,4,1,skipconcat,64,0.01,86,0.0377,0.0052,178952.0,0.0231,0.0029,1.0,0.0
2,4,1,skipsum,64,0.01,173,0.3034,0.329,129800.0,0.0193,0.0017,1.0,0.0
2,4,1,stack,64,0.01,59,0.0439,0.0144,129800.0,0.025,0.0045,1.0,0.0
2,4,2,skipconcat,64,0.01,9,0.0052,0.006,183176.0,0.0283,0.0088,1.0,0.0
2,4,2,skipsum,64,0.01,27,0.001,0.0003,134024.0,0.02,0.0021,1.0,0.0
2,4,2,stack,64,0.01,105,0.0039,0.0046,134024.0,0.0205,0.0009,1.0,0.0
2,6,1,skipconcat,64,0.01,92,0.0782,0.0568,269320.0,0.0265,0.0073,1.0,0.0
2,6,1,skipsum,64,0.01,37,0.051,0.0336,146440.0,0.0182,0.0013,1.0,0.0
2,6,1,stack,64,0.01,122,0.1761,0.1235,146440.0,0.0226,0.0015,1.0,0.0
2,6,2,skipconcat,64,0.01,21,0.0035,0.0007,273544.0,0.0273,0.001,1.0,0.0
2,6,2,skipsum,64,0.01,10,0.0034,0.0035,150664.0,0.0374,0.0011,1.0,0.0
2,6,2,stack,64,0.01,75,0.0011,0.0001,150664.0,0.0216,0.0008,1.0,0.0
pytorch/graphgym/results/example_node_grid_example/agg/train_bestepoch.csv000066400000000000000000000050401444262304300276720ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,3,0.7977,0.0686,117128.0,0.0147,0.0024,1.0,0.0
1,2,1,skipsum,64,0.01,4,0.7508,0.0695,108936.0,0.0154,0.001,1.0,0.0
1,2,1,stack,64,0.01,5,0.6964,0.0622,108936.0,0.0282,0.0053,1.0,0.0
1,2,2,skipconcat,64,0.01,5,0.0342,0.0063,121352.0,0.0152,0.0018,1.0,0.0
1,2,2,skipsum,64,0.01,4,0.0525,0.0117,113160.0,0.022,0.0081,1.0,0.0
1,2,2,stack,64,0.01,6,0.0268,0.0056,113160.0,0.0309,0.0105,1.0,0.0
1,4,1,skipconcat,64,0.01,4,0.6782,0.0586,174728.0,0.0249,0.0118,1.0,0.0
1,4,1,skipsum,64,0.01,3,0.8329,0.0849,125576.0,0.0325,0.0083,1.0,0.0
1,4,1,stack,64,0.01,5,0.7008,0.065,125576.0,0.0221,0.0057,1.0,0.0
1,4,2,skipconcat,64,0.01,4,0.0488,0.0103,178952.0,0.0232,0.0026,1.0,0.0
1,4,2,skipsum,64,0.01,4,0.0371,0.0087,129800.0,0.0198,0.0046,1.0,0.0
1,4,2,stack,64,0.01,7,0.0309,0.0056,129800.0,0.0178,0.0002,1.0,0.0
1,6,1,skipconcat,64,0.01,4,0.704,0.0503,265096.0,0.0301,0.0043,1.0,0.0
1,6,1,skipsum,64,0.01,4,0.7519,0.0592,142216.0,0.023,0.0027,1.0,0.0
1,6,1,stack,64,0.01,7,0.6592,0.0402,142216.0,0.0228,0.0004,1.0,0.0
1,6,2,skipconcat,64,0.01,4,0.0587,0.0106,269320.0,0.0337,0.0053,1.0,0.0
1,6,2,skipsum,64,0.01,5,0.0261,0.0071,146440.0,0.0363,0.0079,1.0,0.0
1,6,2,stack,64,0.01,16,0.0108,0.0026,146440.0,0.0301,0.0059,1.0,0.0
2,2,1,skipconcat,64,0.01,3,0.8443,0.0874,121352.0,0.0291,0.0095,1.0,0.0
2,2,1,skipsum,64,0.01,4,0.7817,0.0907,113160.0,0.0122,0.0013,1.0,0.0
2,2,1,stack,64,0.01,4,0.7775,0.079,113160.0,0.0159,0.0058,1.0,0.0
2,2,2,skipconcat,64,0.01,7,0.0165,0.0046,125576.0,0.0288,0.0079,1.0,0.0
2,2,2,skipsum,64,0.01,4,0.0652,0.0197,117384.0,0.0217,0.0029,1.0,0.0
2,2,2,stack,64,0.01,8,0.0229,0.0142,117384.0,0.0246,0.0026,1.0,0.0
2,4,1,skipconcat,64,0.01,6,0.6253,0.0566,178952.0,0.0219,0.0002,1.0,0.0
2,4,1,skipsum,64,0.01,3,0.8844,0.1032,129800.0,0.0199,0.0044,1.0,0.0
2,4,1,stack,64,0.01,7,0.6471,0.0474,129800.0,0.0208,0.0012,1.0,0.0
2,4,2,skipconcat,64,0.01,6,0.0309,0.002,183176.0,0.0296,0.0107,1.0,0.0
2,4,2,skipsum,64,0.01,6,0.0335,0.01,134024.0,0.0177,0.0011,1.0,0.0
2,4,2,stack,64,0.01,8,0.0305,0.0118,134024.0,0.0208,0.0047,1.0,0.0
2,6,1,skipconcat,64,0.01,6,0.6552,0.0174,269320.0,0.0246,0.0055,1.0,0.0
2,6,1,skipsum,64,0.01,4,0.7697,0.0991,146440.0,0.0184,0.0002,1.0,0.0
2,6,1,stack,64,0.01,9,0.6096,0.0445,146440.0,0.0271,0.0071,1.0,0.0
2,6,2,skipconcat,64,0.01,6,0.0401,0.003,273544.0,0.0308,0.0054,1.0,0.0
2,6,2,skipsum,64,0.01,6,0.0277,0.0045,150664.0,0.0311,0.0043,1.0,0.0
2,6,2,stack,64,0.01,16,0.0175,0.0085,150664.0,0.0252,0.0019,1.0,0.0
pytorch/graphgym/results/example_node_grid_example/agg/val.csv000066400000000000000000000054631444262304300253140ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,199,0.9696,0.0213,117128.0,0.0098,0.0018,0.7193,0.0025
1,2,1,skipsum,64,0.01,199,1.1788,0.0399,108936.0,0.0111,0.0041,0.6467,0.0118
1,2,1,stack,64,0.01,199,0.9178,0.0054,108936.0,0.0203,0.0019,0.7487,0.0093
1,2,2,skipconcat,64,0.01,199,1.5872,0.0508,121352.0,0.0104,0.0017,0.6933,0.0019
1,2,2,skipsum,64,0.01,199,1.5791,0.0927,113160.0,0.0174,0.0033,0.6407,0.0247
1,2,2,stack,64,0.01,199,1.5224,0.0502,113160.0,0.0227,0.0031,0.714,0.0043
1,4,1,skipconcat,64,0.01,199,0.9399,0.0354,174728.0,0.0088,0.0012,0.7387,0.0068
1,4,1,skipsum,64,0.01,199,1.0484,0.0444,125576.0,0.0257,0.0088,0.6847,0.0269
1,4,1,stack,64,0.01,199,0.9333,0.0415,125576.0,0.0141,0.0025,0.7533,0.0081
1,4,2,skipconcat,64,0.01,199,1.5291,0.1317,178952.0,0.0104,0.001,0.6933,0.0238
1,4,2,skipsum,64,0.01,199,1.602,0.1771,129800.0,0.0096,0.0029,0.6613,0.0313
1,4,2,stack,64,0.01,199,1.6831,0.1264,129800.0,0.0102,0.0015,0.7373,0.029
1,6,1,skipconcat,64,0.01,199,0.9324,0.0086,265096.0,0.0143,0.0031,0.732,0.0043
1,6,1,skipsum,64,0.01,199,1.0403,0.0379,142216.0,0.0161,0.0031,0.706,0.0102
1,6,1,stack,64,0.01,199,1.0158,0.057,142216.0,0.0123,0.0008,0.7353,0.0152
1,6,2,skipconcat,64,0.01,199,1.3808,0.0346,269320.0,0.0159,0.0028,0.7267,0.0109
1,6,2,skipsum,64,0.01,199,1.6229,0.0814,146440.0,0.0128,0.0024,0.6827,0.0057
1,6,2,stack,64,0.01,199,1.979,0.0522,146440.0,0.0139,0.0017,0.7087,0.0025
2,2,1,skipconcat,64,0.01,199,0.9576,0.0433,121352.0,0.0117,0.0027,0.7253,0.0068
2,2,1,skipsum,64,0.01,199,1.1686,0.0232,113160.0,0.0058,0.0008,0.6547,0.0186
2,2,1,stack,64,0.01,199,0.8924,0.0211,113160.0,0.0048,0.0002,0.748,0.0091
2,2,2,skipconcat,64,0.01,199,1.6285,0.0639,125576.0,0.0163,0.0045,0.6967,0.0131
2,2,2,skipsum,64,0.01,199,1.7467,0.0979,117384.0,0.0129,0.0021,0.64,0.0016
2,2,2,stack,64,0.01,199,1.655,0.0243,117384.0,0.0172,0.0048,0.6993,0.0123
2,4,1,skipconcat,64,0.01,199,0.9584,0.0324,178952.0,0.016,0.0047,0.7227,0.0104
2,4,1,skipsum,64,0.01,199,1.0906,0.0782,129800.0,0.0146,0.0091,0.694,0.0145
2,4,1,stack,64,0.01,199,0.9331,0.0115,129800.0,0.017,0.0026,0.752,0.0128
2,4,2,skipconcat,64,0.01,199,1.5703,0.0973,183176.0,0.0144,0.0019,0.702,0.0201
2,4,2,skipsum,64,0.01,199,1.5258,0.0604,134024.0,0.0097,0.0004,0.6933,0.0204
2,4,2,stack,64,0.01,199,1.6077,0.0358,134024.0,0.0163,0.0028,0.7467,0.0062
2,6,1,skipconcat,64,0.01,199,0.9485,0.0513,269320.0,0.0102,0.0014,0.7433,0.0195
2,6,1,skipsum,64,0.01,199,1.0677,0.0586,146440.0,0.0119,0.0053,0.692,0.017
2,6,1,stack,64,0.01,199,1.0101,0.0993,146440.0,0.0112,0.0019,0.7533,0.0251
2,6,2,skipconcat,64,0.01,199,1.6208,0.0486,273544.0,0.0132,0.0031,0.712,0.0016
2,6,2,skipsum,64,0.01,199,1.5597,0.1875,150664.0,0.0183,0.0072,0.6833,0.0374
2,6,2,stack,64,0.01,199,1.7315,0.203,150664.0,0.0119,0.0012,0.7467,0.0299
pytorch/graphgym/results/example_node_grid_example/agg/val_best.csv000066400000000000000000000054011444262304300263210ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,35,0.9324,0.0398,117128.0,0.0112,0.0012,0.734,0.0059
1,2,1,skipsum,64,0.01,32,1.0959,0.0469,108936.0,0.0089,0.0039,0.68,0.0114
1,2,1,stack,64,0.01,74,0.8924,0.0154,108936.0,0.0128,0.0004,0.76,0.0085
1,2,2,skipconcat,64,0.01,70,1.6288,0.0505,121352.0,0.0093,0.0031,0.6987,0.0081
1,2,2,skipsum,64,0.01,34,1.3387,0.181,113160.0,0.0133,0.0015,0.6693,0.0229
1,2,2,stack,64,0.01,48,1.5709,0.035,113160.0,0.0169,0.002,0.7213,0.0019
1,4,1,skipconcat,64,0.01,167,0.9054,0.0324,174728.0,0.0085,0.0017,0.746,0.0065
1,4,1,skipsum,64,0.01,194,1.0419,0.0853,125576.0,0.016,0.0035,0.7047,0.0148
1,4,1,stack,64,0.01,106,0.8555,0.008,125576.0,0.0131,0.0015,0.766,0.0086
1,4,2,skipconcat,64,0.01,47,1.5369,0.1013,178952.0,0.0118,0.0048,0.712,0.014
1,4,2,skipsum,64,0.01,40,1.4829,0.2136,129800.0,0.0114,0.0045,0.6733,0.0248
1,4,2,stack,64,0.01,176,1.43,0.3857,129800.0,0.008,0.0011,0.748,0.0237
1,6,1,skipconcat,64,0.01,73,0.914,0.0238,265096.0,0.0103,0.0036,0.7393,0.0052
1,6,1,skipsum,64,0.01,32,1.0163,0.0439,142216.0,0.0253,0.0078,0.714,0.0091
1,6,1,stack,64,0.01,86,0.9281,0.0403,142216.0,0.0194,0.0097,0.7513,0.0105
1,6,2,skipconcat,64,0.01,20,1.1469,0.2318,269320.0,0.0197,0.0061,0.738,0.0113
1,6,2,skipsum,64,0.01,35,1.5108,0.0892,146440.0,0.0215,0.0078,0.6887,0.0052
1,6,2,stack,64,0.01,191,1.5966,0.4699,146440.0,0.0212,0.007,0.7213,0.0146
2,2,1,skipconcat,64,0.01,148,0.918,0.0722,121352.0,0.0191,0.0077,0.738,0.0113
2,2,1,skipsum,64,0.01,27,1.054,0.0398,113160.0,0.0055,0.0003,0.6853,0.0116
2,2,1,stack,64,0.01,137,0.8381,0.0381,113160.0,0.0052,0.0003,0.76,0.0065
2,2,2,skipconcat,64,0.01,108,1.4578,0.2674,125576.0,0.0124,0.0051,0.704,0.0071
2,2,2,skipsum,64,0.01,71,1.6973,0.1969,117384.0,0.0146,0.003,0.66,0.0128
2,2,2,stack,64,0.01,80,1.656,0.0444,117384.0,0.0161,0.0024,0.7073,0.0188
2,4,1,skipconcat,64,0.01,86,0.9336,0.05,178952.0,0.012,0.0019,0.736,0.0059
2,4,1,skipsum,64,0.01,173,1.0928,0.1133,129800.0,0.0086,0.0019,0.706,0.0033
2,4,1,stack,64,0.01,59,0.9015,0.0248,129800.0,0.0196,0.0019,0.7627,0.0136
2,4,2,skipconcat,64,0.01,9,1.3413,0.2281,183176.0,0.0123,0.0023,0.7107,0.0141
2,4,2,skipsum,64,0.01,27,1.4201,0.145,134024.0,0.0092,0.0018,0.702,0.0201
2,4,2,stack,64,0.01,105,1.3714,0.3009,134024.0,0.0099,0.0014,0.7513,0.0025
2,6,1,skipconcat,64,0.01,92,0.8977,0.0223,269320.0,0.0124,0.0044,0.752,0.0114
2,6,1,skipsum,64,0.01,37,1.062,0.0807,146440.0,0.0112,0.0008,0.6973,0.0152
2,6,1,stack,64,0.01,122,0.9048,0.081,146440.0,0.0115,0.0033,0.7613,0.0195
2,6,2,skipconcat,64,0.01,21,1.0935,0.0689,273544.0,0.0131,0.0012,0.726,0.0086
2,6,2,skipsum,64,0.01,10,1.3273,0.3521,150664.0,0.0184,0.0078,0.702,0.044
2,6,2,stack,64,0.01,75,1.7189,0.2232,150664.0,0.0103,0.0009,0.7533,0.0303
pytorch/graphgym/results/example_node_grid_example/agg/val_bestepoch.csv000066400000000000000000000054161444262304300273460ustar00rootroot00000000000000l_pre,l_mp,l_post,stage,dim,lr,epoch,loss,loss_std,params,time_iter,time_iter_std,accuracy,accuracy_std
1,2,1,skipconcat,64,0.01,153,0.9607,0.0242,117128.0,0.0102,0.0007,0.7267,0.0025
1,2,1,skipsum,64,0.01,17,1.1199,0.0168,108936.0,0.0068,0.0008,0.674,0.0057
1,2,1,stack,64,0.01,105,0.893,0.0071,108936.0,0.0207,0.0078,0.7527,0.0094
1,2,2,skipconcat,64,0.01,63,1.6305,0.0503,121352.0,0.0093,0.003,0.6973,0.009
1,2,2,skipsum,64,0.01,43,1.5459,0.063,113160.0,0.0208,0.0077,0.6647,0.0265
1,2,2,stack,64,0.01,67,1.5803,0.0462,113160.0,0.0152,0.0029,0.718,0.0043
1,4,1,skipconcat,64,0.01,152,0.9274,0.0354,174728.0,0.0076,0.0017,0.7427,0.0081
1,4,1,skipsum,64,0.01,7,1.1327,0.0193,125576.0,0.0199,0.0023,0.6913,0.0057
1,4,1,stack,64,0.01,106,0.9245,0.0906,125576.0,0.0117,0.0016,0.756,0.0188
1,4,2,skipconcat,64,0.01,44,1.5306,0.1077,178952.0,0.0132,0.0039,0.7113,0.0139
1,4,2,skipsum,64,0.01,30,1.3797,0.1598,129800.0,0.0084,0.0016,0.664,0.0188
1,4,2,stack,64,0.01,195,1.6813,0.1259,129800.0,0.0195,0.0146,0.738,0.0287
1,6,1,skipconcat,64,0.01,165,0.9285,0.0063,265096.0,0.0109,0.0049,0.7353,0.0047
1,6,1,skipsum,64,0.01,157,1.0426,0.0404,142216.0,0.019,0.007,0.7093,0.012
1,6,1,stack,64,0.01,57,0.9045,0.0361,142216.0,0.0132,0.006,0.7433,0.012
1,6,2,skipconcat,64,0.01,33,1.2341,0.0892,269320.0,0.0137,0.0032,0.7367,0.0104
1,6,2,skipsum,64,0.01,35,1.4584,0.0534,146440.0,0.0158,0.0056,0.6867,0.0074
1,6,2,stack,64,0.01,182,1.9735,0.0553,146440.0,0.0107,0.0013,0.71,0.0028
2,2,1,skipconcat,64,0.01,26,0.8947,0.0292,121352.0,0.0157,0.002,0.734,0.0118
2,2,1,skipsum,64,0.01,32,1.0537,0.0367,113160.0,0.0053,0.0004,0.6813,0.0116
2,2,1,stack,64,0.01,72,0.829,0.0323,113160.0,0.0052,0.0004,0.7527,0.0098
2,2,2,skipconcat,64,0.01,115,1.6346,0.0839,125576.0,0.0241,0.0158,0.698,0.0091
2,2,2,skipsum,64,0.01,48,1.7637,0.1458,117384.0,0.0154,0.0065,0.6547,0.0111
2,2,2,stack,64,0.01,80,1.7137,0.0388,117384.0,0.0184,0.0049,0.702,0.0229
2,4,1,skipconcat,64,0.01,102,0.9602,0.0689,178952.0,0.0108,0.0037,0.73,0.0075
2,4,1,skipsum,64,0.01,195,1.0911,0.0779,129800.0,0.0083,0.0008,0.694,0.0145
2,4,1,stack,64,0.01,171,0.9323,0.0103,129800.0,0.0146,0.0,0.7533,0.0133
2,4,2,skipconcat,64,0.01,142,1.559,0.0978,183176.0,0.0121,0.0002,0.7033,0.0213
2,4,2,skipsum,64,0.01,179,1.5227,0.0625,134024.0,0.0214,0.0154,0.6947,0.0205
2,4,2,stack,64,0.01,158,1.5934,0.0387,134024.0,0.0159,0.0092,0.748,0.0043
2,6,1,skipconcat,64,0.01,103,0.9367,0.057,269320.0,0.0301,0.0213,0.746,0.017
2,6,1,skipsum,64,0.01,166,1.0794,0.0552,146440.0,0.0146,0.0058,0.694,0.0123
2,6,1,stack,64,0.01,122,0.9687,0.1362,146440.0,0.0084,0.0014,0.7547,0.0274
2,6,2,skipconcat,64,0.01,15,1.0278,0.0185,273544.0,0.0111,0.0009,0.7233,0.0109
2,6,2,skipsum,64,0.01,17,1.0332,0.1557,150664.0,0.0147,0.0012,0.688,0.0438
2,6,2,stack,64,0.01,64,1.7142,0.2182,150664.0,0.0171,0.006,0.75,0.03
pytorch/graphgym/run_batch.sh000077500000000000000000000020311444262304300166660ustar00rootroot00000000000000#!/usr/bin/env bash

CONFIG=${CONFIG:-example_node}
GRID=${GRID:-example}
REPEAT=${REPEAT:-3}
MAX_JOBS=${MAX_JOBS:-8}
SLEEP=${SLEEP:-1}
MAIN=${MAIN:-main}

# generate configs (after controlling computational budget)
# please remove --config_budget, if don't control computational budget
python configs_gen.py --config configs/pyg/${CONFIG}.yaml \
  --grid grids/pyg/${GRID}.txt \
  --out_dir configs
#python configs_gen.py --config configs/ChemKG/${CONFIG}.yaml --config_budget configs/ChemKG/${CONFIG}.yaml --grid grids/ChemKG/${GRID}.txt --out_dir configs
# run batch of configs
# Args: config_dir, num of repeats, max jobs running, sleep time
bash parallel.sh configs/${CONFIG}_grid_${GRID} $REPEAT $MAX_JOBS $SLEEP $MAIN
# rerun missed / stopped experiments
bash parallel.sh configs/${CONFIG}_grid_${GRID} $REPEAT $MAX_JOBS $SLEEP $MAIN
# rerun missed / stopped experiments
bash parallel.sh configs/${CONFIG}_grid_${GRID} $REPEAT $MAX_JOBS $SLEEP $MAIN

# aggregate results for the batch
python agg_batch.py --dir results/${CONFIG}_grid_${GRID}
pytorch/graphgym/run_single.sh000077500000000000000000000005531444262304300170750ustar00rootroot00000000000000#!/usr/bin/env bash

# Test for running a single experiment. --repeat means run how many different random seeds.
python main.py --cfg configs/pyg/example_node.yaml --repeat 3 # node classification
python main.py --cfg configs/pyg/example_link.yaml --repeat 3 # link prediction
python main.py --cfg configs/pyg/example_graph.yaml --repeat 3 # graph classification
pytorch/graphgym/sample/000077500000000000000000000000001444262304300156475ustar00rootroot00000000000000pytorch/graphgym/sample/dimensions.txt000066400000000000000000000000751444262304300205620ustar00rootroot00000000000000act bn drop agg l_mp l_pre l_post stage batch lr optim epoch
pytorch/graphgym/sample/dimensionsatt.txt000066400000000000000000000000051444262304300212640ustar00rootroot00000000000000l_tw
pytorch/pyproject.toml000066400000000000000000000077641444262304300155020ustar00rootroot00000000000000[build-system]
requires=["flit_core >=3.2,<4"]
build-backend="flit_core.buildapi"

[project]
name="torch_geometric"
version="2.4.0"
authors=[
    {name="Matthias Fey", email="matthias@pyg.org"},
]
description="Graph Neural Network Library for PyTorch"
readme="README.md"
requires-python=">=3.7"
keywords=[
    "deep-learning",
    "pytorch",
    "geometric-deep-learning",
    "graph-neural-networks",
    "graph-convolutional-networks",
]
classifiers=[
    "Development Status :: 5 - Production/Stable",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3 :: Only",
]
dependencies=[
    "tqdm",
    "numpy",
    "scipy",
    "jinja2",
    "requests",
    "pyparsing",
    "scikit-learn",
    "psutil>=5.8.0",
]

[project.optional-dependencies]
graphgym=[
    "yacs",
    "hydra-core",
    "protobuf<4.21",
    "pytorch-lightning",
]
modelhub=[
    "huggingface_hub"
]
benchmark=[
    "protobuf<4.21",
    "wandb",
    "pandas",
    "networkx",
    "matplotlib",
]
test=[
    "pytest",
    "pytest-cov",
    "onnx",
    "onnxruntime",
]
dev=[
    "torch_geometric[test]",
    "pre-commit",
]
full = [
    "torch_geometric[graphgym, modelhub]",
    "ase",
    "h5py",
    "numba",
    "sympy",
    "pandas",
    "captum",
    "rdflib",
    "trimesh",
    "networkx",
    "graphviz",
    "tabulate",
    "matplotlib",
    "pynndescent",
    "torchmetrics",
    "scikit-image",
    "pytorch-memlab",
    "pgmpy",
    "opt_einsum",
    "statsmodels",
    "rdkit",
]

[project.urls]
homepage="https://pyg.org"
documentation="https://pytorch-geometric.readthedocs.io"
repository="https://github.com/pyg-team/pytorch_geometric.git"
changelog="https://github.com/pyg-team/pytorch_geometric/blob/master/CHANGELOG.md"

[tool.flit.module]
name="torch_geometric"

[tool.yapf]
based_on_style = "pep8"
split_before_named_assigns = false
blank_line_before_nested_class_or_def = false

[tool.pyright]
include = ["torch_geometric/utils/*"]

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
skip = [".gitingore", "__init__.py"]

[tool.pytest.ini_options]
addopts = "--capture=no"
filterwarnings = [
    "ignore:distutils:DeprecationWarning",
    "ignore:'torch_geometric.contrib' contains experimental code:UserWarning",
    # Filter `torch` warnings:
    "ignore:The PyTorch API of nested tensors is in prototype stage:UserWarning",
    "ignore:scatter_reduce():UserWarning",
    "ignore:Sparse CSR tensor support is in beta state:UserWarning",
    "ignore:Sparse CSC tensor support is in beta state:UserWarning",
    "ignore:torch.distributed._sharded_tensor will be deprecated:DeprecationWarning",
    # Filter `captum` warnings:
    "ignore:Setting backward hooks on ReLU activations:UserWarning",
    "ignore:.*did not already require gradients, required_grads has been set automatically:UserWarning",
    # Filter `pytorch_lightning` warnings:
    "ignore:GPU available but not used:UserWarning",
]

[tool.pylint.messages_control]
disable = [
    "import-outside-toplevel",
    "missing-module-docstring",
    "missing-class-docstring",
    "missing-function-docstring",
    "empty-docstring",
    "import-error",
    "too-many-arguments",
    "arguments-differ",
    "invalid-name",
    "redefined-builtin",
]
attr-rgx = "[A-Za-z_][A-Za-z0-9_]*$"
argument-rgx = "[A-Za-z_][A-Za-z0-9_]*$"
variable-rgx = "[A-Za-z_][A-Za-z0-9_]*$"
generated-members = ["torch.*"]

[tool.coverage.run]
source = ["torch_geometric"]
omit = [
    "torch_geometric/datasets/*",
    "torch_geometric/data/extract.py",
    "torch_geometric/nn/data_parallel.py",
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "pass",
    "raise",
    "except",
    "register_parameter",
    "warn",
    "torch.cuda.is_available",
    "WITH_PT2",
]

[tool.flake8]
ignore = ["F811", "W503", "W504"]
pytorch/readthedocs.yml000066400000000000000000000002671444262304300155650ustar00rootroot00000000000000version: 2

build:
   os: ubuntu-22.04
   tools:
      python: "3.8"

python:
   install:
      - requirements: docs/requirements.txt
      - method: pip
        path: .

formats: []
pytorch/test/000077500000000000000000000000001444262304300135275ustar00rootroot00000000000000pytorch/test/conftest.py000066400000000000000000000051771444262304300157400ustar00rootroot00000000000000import functools
import os.path as osp
import shutil

import pytest

import torch_geometric.typing
from torch_geometric.data import Dataset


def load_dataset(root: str, name: str, *args, **kwargs) -> Dataset:
    r"""Returns a variety of datasets according to :obj:`name`."""
    if 'karate' in name.lower():
        from torch_geometric.datasets import KarateClub
        return KarateClub(*args, **kwargs)
    if name.lower() in ['cora', 'citeseer', 'pubmed']:
        from torch_geometric.datasets import Planetoid
        path = osp.join(root, 'Planetoid', name)
        return Planetoid(path, name, *args, **kwargs)
    if name in ['BZR', 'ENZYMES', 'IMDB-BINARY', 'MUTAG']:
        from torch_geometric.datasets import TUDataset
        path = osp.join(root, 'TUDataset')
        return TUDataset(path, name, *args, **kwargs)
    if name in ['ego-facebook', 'soc-Slashdot0811', 'wiki-vote']:
        from torch_geometric.datasets import SNAPDataset
        path = osp.join(root, 'SNAPDataset')
        return SNAPDataset(path, name, *args, **kwargs)
    if name.lower() in ['bashapes']:
        from torch_geometric.datasets import BAShapes
        return BAShapes(*args, **kwargs)
    if name in ['citationCiteseer', 'illc1850']:
        from torch_geometric.datasets import SuiteSparseMatrixCollection
        path = osp.join(root, 'SuiteSparseMatrixCollection')
        return SuiteSparseMatrixCollection(path, name=name, *args, **kwargs)
    if 'elliptic' in name.lower():
        from torch_geometric.datasets import EllipticBitcoinDataset
        path = osp.join(root, 'EllipticBitcoinDataset')
        return EllipticBitcoinDataset(path, *args, **kwargs)
    if name.lower() in ['hetero']:
        from torch_geometric.testing import FakeHeteroDataset
        return FakeHeteroDataset(*args, **kwargs)

    raise ValueError(f"Cannot load dataset with name '{name}'")


@pytest.fixture(scope='session')
def get_dataset():
    root = osp.join('/', 'tmp', 'pyg_test_datasets')
    yield functools.partial(load_dataset, root)
    if osp.exists(root):
        shutil.rmtree(root)


@pytest.fixture
def disable_extensions():
    prev_state = {
        'WITH_PYG_LIB': torch_geometric.typing.WITH_PYG_LIB,
        'WITH_SAMPLED_OP': torch_geometric.typing.WITH_SAMPLED_OP,
        'WITH_INDEX_SORT': torch_geometric.typing.WITH_INDEX_SORT,
        'WITH_TORCH_SCATTER': torch_geometric.typing.WITH_TORCH_SCATTER,
        'WITH_TORCH_SPARSE': torch_geometric.typing.WITH_TORCH_SPARSE,
    }
    for key in prev_state.keys():
        setattr(torch_geometric.typing, key, False)
    yield
    for key, value in prev_state.items():
        setattr(torch_geometric.typing, key, value)
pytorch/test/contrib/000077500000000000000000000000001444262304300151675ustar00rootroot00000000000000pytorch/test/contrib/explain/000077500000000000000000000000001444262304300166275ustar00rootroot00000000000000pytorch/test/contrib/explain/test_graphmask_explainer.py000066400000000000000000000043371444262304300242730ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.contrib.explain import GraphMaskExplainer
from torch_geometric.explain import Explainer
from torch_geometric.explain.config import (
    ModelConfig,
    ModelMode,
    ModelTaskLevel,
)
from torch_geometric.nn import GCNConv, global_add_pool


class GCN(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config
        assert model_config.mode == ModelMode.binary_classification

        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, 1)

    def forward(self, x, edge_index, batch=None, edge_label_index=None):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)

        if self.model_config.task_level == ModelTaskLevel.graph:
            x = global_add_pool(x, batch)
        elif self.model_config.task_level == ModelTaskLevel.edge:
            assert edge_label_index is not None
            x = x[edge_label_index[0]] * x[edge_label_index[1]]

        return x


x = torch.randn(8, 3)
edge_index = torch.tensor([
    [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
    [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
])
batch = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2])
edge_label_index = torch.tensor([[0, 1, 2], [3, 4, 5]])


@pytest.mark.parametrize('task_level', ['node', 'edge', 'graph'])
def test_graph_mask_explainer(task_level):
    model_config = ModelConfig(
        mode='binary_classification',
        task_level=task_level,
        return_type='raw',
    )

    model = GCN(model_config)

    explainer = Explainer(
        model=model,
        algorithm=GraphMaskExplainer(2, epochs=5, log=False),
        explanation_type='model',
        node_mask_type='attributes',
        edge_mask_type='object',
        model_config=model_config,
    )

    explanation = explainer(
        x,
        edge_index,
        batch=batch,
        edge_label_index=edge_label_index,
    )

    assert explanation.node_mask.size() == explanation.x.size()
    assert explanation.node_mask.min() >= 0
    assert explanation.node_mask.max() <= 1

    assert explanation.edge_mask.size() == (explanation.num_edges, )
    assert explanation.edge_mask.min() >= 0
    assert explanation.edge_mask.max() <= 1
pytorch/test/contrib/explain/test_pgm_explainer.py000066400000000000000000000051531444262304300230760ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.contrib.explain import PGMExplainer
from torch_geometric.explain import Explainer
from torch_geometric.explain.config import ModelConfig
from torch_geometric.nn import GCNConv, global_add_pool
from torch_geometric.testing import withPackage


class GCN(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config

        if model_config.mode.value == 'multiclass_classification':
            out_channels = 7
        else:
            out_channels = 1

        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, out_channels)

    def forward(self, x, edge_index, edge_weight=None, batch=None, **kwargs):
        x = self.conv1(x, edge_index, edge_weight).relu()
        x = self.conv2(x, edge_index, edge_weight).relu()

        if self.model_config.task_level.value == 'graph':
            x = global_add_pool(x, batch)

        if self.model_config.mode.value == 'binary_classification':
            x = x.sigmoid()
        elif self.model_config.mode.value == 'multiclass_classification':
            x = x.log_softmax(dim=-1)

        return x


x = torch.randn(8, 3)
edge_index = torch.tensor([
    [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
    [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
])
target = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2])
edge_label_index = torch.tensor([[0, 1, 2], [3, 4, 5]])


@withPackage('pgmpy')
@withPackage('pandas')
@pytest.mark.parametrize('node_idx', [2, 6])
@pytest.mark.parametrize('task_level, perturbation_mode', [
    ('node', 'randint'),
    ('graph', 'mean'),
    ('graph', 'max'),
    ('graph', 'min'),
    ('graph', 'zero'),
])
def test_pgm_explainer_classification(node_idx, task_level, perturbation_mode):
    model_config = ModelConfig(
        mode='multiclass_classification',
        task_level=task_level,
        return_type='raw',
    )

    model = GCN(model_config)
    logits = model(x, edge_index)
    target = logits.argmax(dim=1)

    explainer = Explainer(
        model=model,
        algorithm=PGMExplainer(feature_index=[0],
                               perturbation_mode=perturbation_mode),
        explanation_type='phenomenon',
        node_mask_type='object',
        model_config=model_config,
    )

    explanation = explainer(
        x=x,
        edge_index=edge_index,
        index=node_idx,
        target=target,
    )

    assert 'node_mask' in explanation
    assert 'pgm_stats' in explanation
    assert explanation.node_mask.size(0) == explanation.num_nodes
    assert explanation.node_mask.min() >= 0
    assert explanation.node_mask.max() <= 1
pytorch/test/contrib/nn/000077500000000000000000000000001444262304300156025ustar00rootroot00000000000000pytorch/test/contrib/nn/models/000077500000000000000000000000001444262304300170655ustar00rootroot00000000000000pytorch/test/contrib/nn/models/test_rbcd_attack.py000066400000000000000000000071271444262304300227460ustar00rootroot00000000000000import pytest
import torch
from torch.nn import Linear

from torch_geometric.contrib.nn import GRBCDAttack, PRBCDAttack
from torch_geometric.nn import GCNConv, global_add_pool
from torch_geometric.utils import to_undirected


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, 7)

    def forward(self, x, edge_index, edge_weight):
        x = self.conv1(x, edge_index, edge_weight).relu()
        x = self.conv2(x, edge_index, edge_weight)
        return x


class GNN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, 16)
        self.lin = Linear(16, 7)

    def forward(self, x, edge_index, edge_weight, batch=None):
        x = self.conv1(x, edge_index, edge_weight).relu()
        x = self.conv2(x, edge_index, edge_weight).relu()
        x = global_add_pool(x, batch)
        x = self.lin(x)
        return x


@pytest.mark.parametrize('model', [GCN, GNN])
@pytest.mark.parametrize('budget', [1])
@pytest.mark.parametrize('loss',
                         ['masked', 'margin', 'prob_margin', 'tanh_margin'])
@pytest.mark.parametrize('is_undirected', [False, True])
@pytest.mark.parametrize('with_early_stopping', [False, True])
def test_prbcd_attack(model, budget, loss, is_undirected, with_early_stopping):
    attack = PRBCDAttack(model(), block_size=10_000, epochs=4,
                         epochs_resampling=2, loss=loss, max_final_samples=2,
                         log=False, is_undirected=is_undirected,
                         with_early_stopping=with_early_stopping)
    assert str(attack) == 'PRBCDAttack()'

    x = torch.randn(8, 3)
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6, 7]])
    if is_undirected:
        edge_index = to_undirected(edge_index)

    if model == GNN:
        y = torch.tensor([0])
        # All nodes belong to same graph:
        kwargs = dict(batch=edge_index.new_zeros(x.size(0)))
    else:
        y = torch.tensor([0, 1, 1, 0, 1, 0, 1, 0])
        kwargs = {}

    pert_edge_index, pert = attack.attack(x, edge_index, y, budget, **kwargs)

    m = edge_index.size(1)
    if budget == 1:
        assert pert.size() in [(2, 0), (2, 1)]

        if pert.size(1):
            if is_undirected:
                possible_m = [m - 2, m + 2]
            else:
                possible_m = [m - 1, m + 1]
        else:
            possible_m = [m]
        assert pert_edge_index.size(1) in possible_m


@pytest.mark.parametrize('model', [GCN, GNN])
@pytest.mark.parametrize('budget', [1])
@pytest.mark.parametrize('is_undirected', [False, True])
def test_grbcd_attack(model, budget, is_undirected):
    attack = GRBCDAttack(model(), block_size=10_000, epochs=4, log=False,
                         is_undirected=is_undirected)
    assert str(attack) == 'GRBCDAttack()'

    x = torch.randn(8, 3)
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6, 7]])
    if is_undirected:
        edge_index = to_undirected(edge_index)

    if model == GNN:
        y = torch.tensor([0])
        # All nodes belong to same graph:
        kwargs = dict(batch=edge_index.new_zeros(x.size(0)))
    else:
        y = torch.tensor([0, 1, 1, 0, 1, 0, 1, 0])
        kwargs = {}

    pert_edge_index, pert = attack.attack(x, edge_index, y, budget, **kwargs)

    m = edge_index.size(1)
    if budget == 1:
        assert pert.size() == (2, 1)

        if is_undirected:
            possible_m = [m - 2, m + 2]
        else:
            possible_m = [m - 1, m + 1]
        assert pert_edge_index.size(1) in possible_m
pytorch/test/data/000077500000000000000000000000001444262304300144405ustar00rootroot00000000000000pytorch/test/data/lightning/000077500000000000000000000000001444262304300164235ustar00rootroot00000000000000pytorch/test/data/lightning/test_datamodule.py000066400000000000000000000417071444262304300221640ustar00rootroot00000000000000import math

import pytest
import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.lightning import (
    LightningDataset,
    LightningLinkData,
    LightningNodeData,
)
from torch_geometric.nn import global_mean_pool
from torch_geometric.sampler import BaseSampler, NeighborSampler
from torch_geometric.testing import (
    MyFeatureStore,
    MyGraphStore,
    get_random_edge_index,
    onlyCUDA,
    onlyFullTest,
    withPackage,
)

try:
    from pytorch_lightning import LightningModule
except ImportError:
    LightningModule = torch.nn.Module


class LinearGraphModule(LightningModule):
    def __init__(self, in_channels: int, hidden_channels: int,
                 out_channels: int):
        super().__init__()
        from torchmetrics import Accuracy

        self.lin1 = torch.nn.Linear(in_channels, hidden_channels)
        self.lin2 = torch.nn.Linear(hidden_channels, out_channels)

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(self, x: Tensor, batch: Data) -> Tensor:
        # Basic test to ensure that the dataset is not replicated:
        self.trainer.datamodule.train_dataset._data.x.add_(1)

        x = self.lin1(x).relu()
        x = global_mean_pool(x, batch)
        x = self.lin2(x)
        return x

    def training_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x, data.batch)
        loss = F.cross_entropy(y_hat, data.y)
        self.train_acc(y_hat.softmax(dim=-1), data.y)
        self.log('loss', loss, batch_size=data.num_graphs)
        self.log('train_acc', self.train_acc, batch_size=data.num_graphs)
        return loss

    def validation_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x, data.batch)
        self.val_acc(y_hat.softmax(dim=-1), data.y)
        self.log('val_acc', self.val_acc, batch_size=data.num_graphs)

    def test_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x, data.batch)
        self.test_acc(y_hat.softmax(dim=-1), data.y)
        self.log('test_acc', self.test_acc, batch_size=data.num_graphs)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


@onlyCUDA
@onlyFullTest
@withPackage('pytorch_lightning>=2.0.0')
@withPackage('torchmetrics>=0.11.0')
@pytest.mark.parametrize('strategy_type', [None, 'ddp'])
def test_lightning_dataset(get_dataset, strategy_type):
    import pytorch_lightning as pl

    dataset = get_dataset(name='MUTAG').shuffle()
    train_dataset = dataset[:50]
    val_dataset = dataset[50:80]
    test_dataset = dataset[80:90]
    pred_dataset = dataset[90:]

    devices = 1 if strategy_type is None else torch.cuda.device_count()
    if strategy_type == 'ddp':
        strategy = pl.strategies.DDPStrategy(accelerator='gpu')
    else:
        strategy = pl.strategies.SingleDeviceStrategy(device='cuda:0')

    model = LinearGraphModule(dataset.num_features, 64, dataset.num_classes)

    trainer = pl.Trainer(strategy=strategy, devices=devices, max_epochs=1,
                         log_every_n_steps=1)
    with pytest.warns(UserWarning, match="'shuffle=True' option is ignored"):
        datamodule = LightningDataset(train_dataset, val_dataset, test_dataset,
                                      pred_dataset, batch_size=5,
                                      num_workers=3, shuffle=True)
        assert 'shuffle' not in datamodule.kwargs
    old_x = train_dataset._data.x.clone()
    assert str(datamodule) == ('LightningDataset(train_dataset=MUTAG(50), '
                               'val_dataset=MUTAG(30), '
                               'test_dataset=MUTAG(10), '
                               'pred_dataset=MUTAG(98), batch_size=5, '
                               'num_workers=3, pin_memory=True, '
                               'persistent_workers=True)')
    trainer.fit(model, datamodule)
    trainer.test(model, datamodule)
    new_x = train_dataset._data.x
    assert torch.all(new_x > old_x)  # Ensure shared data.
    assert trainer.validate_loop._data_source.is_defined()
    assert trainer.test_loop._data_source.is_defined()

    # Test with `val_dataset=None` and `test_dataset=None`:
    if strategy_type is None:
        trainer = pl.Trainer(strategy=strategy, devices=devices, max_epochs=1,
                             log_every_n_steps=1)

        datamodule = LightningDataset(train_dataset, batch_size=5)
        assert str(datamodule) == ('LightningDataset(train_dataset=MUTAG(50), '
                                   'batch_size=5, num_workers=0, '
                                   'pin_memory=True, '
                                   'persistent_workers=False)')

        with pytest.warns(UserWarning, match="defined a `validation_step`"):
            trainer.fit(model, datamodule)

        assert not trainer.validate_loop._data_source.is_defined()
        assert not trainer.test_loop._data_source.is_defined()


class LinearNodeModule(LightningModule):
    def __init__(self, in_channels: int, out_channels: int):
        super().__init__()
        from torchmetrics import Accuracy

        self.lin = torch.nn.Linear(in_channels, out_channels)

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(self, x: Tensor) -> Tensor:
        # Basic test to ensure that the dataset is not replicated:
        self.trainer.datamodule.data.x.add_(1)

        return self.lin(x)

    def training_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x)[data.train_mask]
        y = data.y[data.train_mask]
        loss = F.cross_entropy(y_hat, y)
        self.train_acc(y_hat.softmax(dim=-1), y)
        self.log('loss', loss, batch_size=y.size(0))
        self.log('train_acc', self.train_acc, batch_size=y.size(0))
        return loss

    def validation_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x)[data.val_mask]
        y = data.y[data.val_mask]
        self.val_acc(y_hat.softmax(dim=-1), y)
        self.log('val_acc', self.val_acc, batch_size=y.size(0))

    def test_step(self, data: Data, batch_idx: int):
        y_hat = self(data.x)[data.test_mask]
        y = data.y[data.test_mask]
        self.test_acc(y_hat.softmax(dim=-1), y)
        self.log('test_acc', self.test_acc, batch_size=y.size(0))

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


@onlyCUDA
@onlyFullTest
@withPackage('pyg_lib')
@withPackage('pytorch_lightning>=2.0.0')
@withPackage('torchmetrics>=0.11.0')
@pytest.mark.parametrize('loader', ['full', 'neighbor'])
@pytest.mark.parametrize('strategy_type', [None, 'ddp'])
def test_lightning_node_data(get_dataset, strategy_type, loader):
    import pytorch_lightning as pl

    dataset = get_dataset(name='Cora')
    data = dataset[0]
    data_repr = ('Data(x=[2708, 1433], edge_index=[2, 10556], y=[2708], '
                 'train_mask=[2708], val_mask=[2708], test_mask=[2708])')

    model = LinearNodeModule(dataset.num_features, dataset.num_classes)

    if strategy_type is None or loader == 'full':
        devices = 1
    else:
        devices = torch.cuda.device_count()

    if strategy_type == 'ddp':
        strategy = pl.strategies.DDPStrategy(accelerator='gpu')
    else:
        strategy = pl.strategies.SingleDeviceStrategy(device='cuda:0')

    if loader == 'full':  # Set reasonable defaults for full-batch training:
        batch_size = 1
        num_workers = 0
    else:
        batch_size = 32
        num_workers = 3
    kwargs, kwargs_repr = {}, ''
    if loader == 'neighbor':
        kwargs['num_neighbors'] = [5]
        kwargs_repr += 'num_neighbors=[5], '

    trainer = pl.Trainer(strategy=strategy, devices=devices, max_epochs=5,
                         log_every_n_steps=1)
    datamodule = LightningNodeData(data, loader=loader, batch_size=batch_size,
                                   num_workers=num_workers, **kwargs)

    old_x = data.x.clone().cpu()
    assert str(datamodule) == (f'LightningNodeData(data={data_repr}, '
                               f'loader={loader}, batch_size={batch_size}, '
                               f'num_workers={num_workers}, {kwargs_repr}'
                               f'pin_memory={loader != "full"}, '
                               f'persistent_workers={loader != "full"})')
    trainer.fit(model, datamodule)
    trainer.test(model, datamodule)
    new_x = data.x.cpu()
    assert torch.all(new_x > old_x)  # Ensure shared data.
    assert trainer.validate_loop._data_source.is_defined()
    assert trainer.test_loop._data_source.is_defined()


class LinearHeteroNodeModule(LightningModule):
    def __init__(self, in_channels: int, out_channels: int):
        super().__init__()
        from torchmetrics import Accuracy

        self.lin = torch.nn.Linear(in_channels, out_channels)

        self.train_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.val_acc = Accuracy(task='multiclass', num_classes=out_channels)
        self.test_acc = Accuracy(task='multiclass', num_classes=out_channels)

    def forward(self, x: Tensor) -> Tensor:
        # Basic test to ensure that the dataset is not replicated:
        self.trainer.datamodule.data['paper'].x.add_(1)

        return self.lin(x)

    def training_step(self, data: HeteroData, batch_idx: int):
        y_hat = self(data['paper'].x)[data['paper'].train_mask]
        y = data['paper'].y[data['paper'].train_mask]
        loss = F.cross_entropy(y_hat, y)
        self.train_acc(y_hat.softmax(dim=-1), y)
        self.log('loss', loss, batch_size=y.size(0))
        self.log('train_acc', self.train_acc, batch_size=y.size(0))
        return loss

    def validation_step(self, data: HeteroData, batch_idx: int):
        y_hat = self(data['paper'].x)[data['paper'].val_mask]
        y = data['paper'].y[data['paper'].val_mask]
        self.val_acc(y_hat.softmax(dim=-1), y)
        self.log('val_acc', self.val_acc, batch_size=y.size(0))

    def test_step(self, data: HeteroData, batch_idx: int):
        y_hat = self(data['paper'].x)[data['paper'].test_mask]
        y = data['paper'].y[data['paper'].test_mask]
        self.test_acc(y_hat.softmax(dim=-1), y)
        self.log('test_acc', self.test_acc, batch_size=y.size(0))

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.01)


@onlyCUDA
@onlyFullTest
@withPackage('pyg_lib')
@withPackage('pytorch_lightning>=2.0.0')
@withPackage('torchmetrics>=0.11.0')
def test_lightning_hetero_node_data(get_dataset):
    import pytorch_lightning as pl

    data = get_dataset(name='hetero')[0]

    model = LinearHeteroNodeModule(data['paper'].num_features,
                                   int(data['paper'].y.max()) + 1)

    devices = torch.cuda.device_count()
    strategy = pl.strategies.DDPStrategy(accelerator='gpu')

    trainer = pl.Trainer(strategy=strategy, devices=devices, max_epochs=5,
                         log_every_n_steps=1)
    datamodule = LightningNodeData(data, loader='neighbor', num_neighbors=[5],
                                   batch_size=32, num_workers=3)
    assert isinstance(datamodule.graph_sampler, NeighborSampler)
    original_x = data['paper'].x.clone()
    trainer.fit(model, datamodule)
    trainer.test(model, datamodule)
    assert torch.all(data['paper'].x > original_x)  # Ensure shared data.
    assert trainer.validate_loop._data_source.is_defined()
    assert trainer.test_loop._data_source.is_defined()


@withPackage('pytorch_lightning')
def test_lightning_data_custom_sampler():
    class DummySampler(BaseSampler):
        def sample_from_edges(self, *args, **kwargs):
            pass

        def sample_from_nodes(self, *args, **kwargs):
            pass

    data = Data(num_nodes=2, edge_index=torch.tensor([[0, 1], [1, 0]]))

    datamodule = LightningNodeData(data, node_sampler=DummySampler(),
                                   input_train_nodes=torch.arange(2))
    assert isinstance(datamodule.graph_sampler, DummySampler)

    datamodule = LightningLinkData(
        data, link_sampler=DummySampler(),
        input_train_edges=torch.tensor([[0, 1], [0, 1]]))
    assert isinstance(datamodule.graph_sampler, DummySampler)


@onlyCUDA
@onlyFullTest
@withPackage('pyg_lib')
@withPackage('pytorch_lightning')
def test_lightning_hetero_link_data():
    torch.manual_seed(12345)

    data = HeteroData()

    data['paper'].x = torch.arange(10)
    data['author'].x = torch.arange(10)
    data['term'].x = torch.arange(10)

    data['paper', 'author'].edge_index = get_random_edge_index(10, 10, 10)
    data['author', 'paper'].edge_index = get_random_edge_index(10, 10, 10)
    data['paper', 'term'].edge_index = get_random_edge_index(10, 10, 10)
    data['author', 'term'].edge_index = get_random_edge_index(10, 10, 10)

    datamodule = LightningLinkData(
        data,
        input_train_edges=('author', 'paper'),
        input_val_edges=('paper', 'author'),
        input_test_edges=('paper', 'term'),
        input_pred_edges=('author', 'term'),
        loader='neighbor',
        num_neighbors=[5],
        batch_size=32,
        num_workers=0,
    )

    assert isinstance(datamodule.graph_sampler, NeighborSampler)
    assert isinstance(datamodule.eval_graph_sampler, NeighborSampler)

    for batch in datamodule.train_dataloader():
        assert 'edge_label_index' in batch['author', 'paper']
    for batch in datamodule.val_dataloader():
        assert 'edge_label_index' in batch['paper', 'author']
    for batch in datamodule.test_dataloader():
        assert 'edge_label_index' in batch['paper', 'term']
    for batch in datamodule.predict_dataloader():
        assert 'edge_label_index' in batch['author', 'term']

    data['author'].time = torch.arange(data['author'].num_nodes)
    data['paper'].time = torch.arange(data['paper'].num_nodes)
    data['term'].time = torch.arange(data['term'].num_nodes)

    datamodule = LightningLinkData(
        data,
        input_train_edges=('author', 'paper'),
        input_train_time=torch.arange(data['author', 'paper'].num_edges),
        loader='neighbor',
        num_neighbors=[5],
        batch_size=32,
        num_workers=0,
        time_attr='time',
    )

    for batch in datamodule.train_dataloader():
        assert 'edge_label_index' in batch['author', 'paper']
        assert 'edge_label_time' in batch['author', 'paper']


@withPackage('pyg_lib')
@withPackage('pytorch_lightning')
def test_lightning_hetero_link_data_custom_store():
    torch.manual_seed(12345)

    feature_store = MyFeatureStore()
    graph_store = MyGraphStore()

    x = torch.arange(10)
    feature_store.put_tensor(x, group_name='paper', attr_name='x', index=None)
    feature_store.put_tensor(x, group_name='author', attr_name='x', index=None)
    feature_store.put_tensor(x, group_name='term', attr_name='x', index=None)

    edge_index = get_random_edge_index(10, 10, 10)
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('paper', 'to', 'author'),
                               layout='coo', size=(10, 10))
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('author', 'to', 'paper'),
                               layout='coo', size=(10, 10))
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('paper', 'to', 'term'), layout='coo',
                               size=(10, 10))

    datamodule = LightningLinkData(
        (feature_store, graph_store),
        input_train_edges=('author', 'to', 'paper'),
        loader='neighbor',
        num_neighbors=[5],
        batch_size=32,
        num_workers=0,
    )

    batch = next(iter(datamodule.train_dataloader()))
    assert 'edge_label_index' in batch['author', 'paper']


@withPackage('pyg_lib')
@withPackage('pytorch_lightning')
def test_eval_loader_kwargs(get_dataset):
    data = get_dataset(name='Cora')[0]

    node_sampler = NeighborSampler(data, num_neighbors=[5])

    datamodule = LightningNodeData(
        data,
        node_sampler=node_sampler,
        batch_size=32,
        eval_loader_kwargs=dict(num_neighbors=[-1], batch_size=64),
    )

    assert datamodule.loader_kwargs['batch_size'] == 32
    assert datamodule.graph_sampler.num_neighbors.values == [5]
    assert datamodule.eval_loader_kwargs['batch_size'] == 64
    assert datamodule.eval_graph_sampler.num_neighbors.values == [-1]

    train_loader = datamodule.train_dataloader()
    assert math.ceil(int(data.train_mask.sum()) / 32) == len(train_loader)

    val_loader = datamodule.val_dataloader()
    assert math.ceil(int(data.val_mask.sum()) / 64) == len(val_loader)

    test_loader = datamodule.test_dataloader()
    assert math.ceil(int(data.test_mask.sum()) / 64) == len(test_loader)

    pred_loader = datamodule.predict_dataloader()
    assert math.ceil(data.num_nodes / 64) == len(pred_loader)
pytorch/test/data/test_batch.py000066400000000000000000000460431444262304300171410ustar00rootroot00000000000000import os.path as osp

import numpy as np
import pytest
import torch

import torch_geometric
from torch_geometric.data import Batch, Data, HeteroData
from torch_geometric.testing import get_random_edge_index, withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_edge_index, to_torch_sparse_tensor


def test_batch_basic():
    torch_geometric.set_debug(True)

    x = torch.tensor([1.0, 2.0, 3.0])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    data1 = Data(x=x, y=1, edge_index=edge_index, string='1', array=['1', '2'],
                 num_nodes=3)

    x = torch.tensor([1.0, 2.0])
    edge_index = torch.tensor([[0, 1], [1, 0]])
    data2 = Data(x=x, y=2, edge_index=edge_index, string='2',
                 array=['3', '4', '5'], num_nodes=2)

    x = torch.tensor([1.0, 2.0, 3.0, 4.0])
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    data3 = Data(x=x, y=3, edge_index=edge_index, string='3',
                 array=['6', '7', '8', '9'], num_nodes=4)

    batch = Batch.from_data_list([data1])
    assert str(batch) == ('DataBatch(x=[3], edge_index=[2, 4], y=[1], '
                          'string=[1], array=[1], num_nodes=3, batch=[3], '
                          'ptr=[2])')
    assert batch.num_graphs == len(batch) == 1
    assert batch.x.tolist() == [1, 2, 3]
    assert batch.y.tolist() == [1]
    assert batch.edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert batch.string == ['1']
    assert batch.array == [['1', '2']]
    assert batch.num_nodes == 3
    assert batch.batch.tolist() == [0, 0, 0]
    assert batch.ptr.tolist() == [0, 3]

    batch = Batch.from_data_list([data1, data2, data3],
                                 follow_batch=['string'])

    assert str(batch) == ('DataBatch(x=[9], edge_index=[2, 12], y=[3], '
                          'string=[3], string_batch=[3], string_ptr=[4], '
                          'array=[3], num_nodes=9, batch=[9], ptr=[4])')
    assert batch.num_graphs == len(batch) == 3
    assert batch.x.tolist() == [1, 2, 3, 1, 2, 1, 2, 3, 4]
    assert batch.y.tolist() == [1, 2, 3]
    assert batch.edge_index.tolist() == [[0, 1, 1, 2, 3, 4, 5, 6, 6, 7, 7, 8],
                                         [1, 0, 2, 1, 4, 3, 6, 5, 7, 6, 8, 7]]
    assert batch.string == ['1', '2', '3']
    assert batch.string_batch.tolist() == [0, 1, 2]
    assert batch.string_ptr.tolist() == [0, 1, 2, 3]
    assert batch.array == [['1', '2'], ['3', '4', '5'], ['6', '7', '8', '9']]
    assert batch.num_nodes == 9
    assert batch.batch.tolist() == [0, 0, 0, 1, 1, 2, 2, 2, 2]
    assert batch.ptr.tolist() == [0, 3, 5, 9]

    assert str(batch[0]) == ("Data(x=[3], edge_index=[2, 4], y=[1], "
                             "string='1', array=[2], num_nodes=3)")
    assert str(batch[1]) == ("Data(x=[2], edge_index=[2, 2], y=[1], "
                             "string='2', array=[3], num_nodes=2)")
    assert str(batch[2]) == ("Data(x=[4], edge_index=[2, 6], y=[1], "
                             "string='3', array=[4], num_nodes=4)")

    assert len(batch.index_select([1, 0])) == 2
    assert len(batch.index_select(torch.tensor([1, 0]))) == 2
    assert len(batch.index_select(torch.tensor([True, False]))) == 1
    assert len(batch.index_select(np.array([1, 0], dtype=np.int64))) == 2
    assert len(batch.index_select(np.array([True, False]))) == 1
    assert len(batch[:2]) == 2

    data_list = batch.to_data_list()
    assert len(data_list) == 3

    assert len(data_list[0]) == 6
    assert data_list[0].x.tolist() == [1, 2, 3]
    assert data_list[0].y.tolist() == [1]
    assert data_list[0].edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert data_list[0].string == '1'
    assert data_list[0].array == ['1', '2']
    assert data_list[0].num_nodes == 3

    assert len(data_list[1]) == 6
    assert data_list[1].x.tolist() == [1, 2]
    assert data_list[1].y.tolist() == [2]
    assert data_list[1].edge_index.tolist() == [[0, 1], [1, 0]]
    assert data_list[1].string == '2'
    assert data_list[1].array == ['3', '4', '5']
    assert data_list[1].num_nodes == 2

    assert len(data_list[2]) == 6
    assert data_list[2].x.tolist() == [1, 2, 3, 4]
    assert data_list[2].y.tolist() == [3]
    assert data_list[2].edge_index.tolist() == [[0, 1, 1, 2, 2, 3],
                                                [1, 0, 2, 1, 3, 2]]
    assert data_list[2].string == '3'
    assert data_list[2].array == ['6', '7', '8', '9']
    assert data_list[2].num_nodes == 4

    torch_geometric.set_debug(True)


@withPackage('torch_sparse')
def test_batch_with_sparse_tensor():
    x = SparseTensor.from_dense(torch.tensor([[1.0], [2.0], [3.0]]))
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    adj = SparseTensor.from_edge_index(edge_index)
    data1 = Data(x=x, adj=adj)

    x = SparseTensor.from_dense(torch.tensor([[1.0], [2.0]]))
    edge_index = torch.tensor([[0, 1], [1, 0]])
    adj = SparseTensor.from_edge_index(edge_index)
    data2 = Data(x=x, adj=adj)

    x = SparseTensor.from_dense(torch.tensor([[1.0], [2.0], [3.0], [4.0]]))
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    adj = SparseTensor.from_edge_index(edge_index)
    data3 = Data(x=x, adj=adj)

    batch = Batch.from_data_list([data1])
    assert str(batch) == ('DataBatch(x=[3, 1, nnz=3], adj=[3, 3, nnz=4], '
                          'batch=[3], ptr=[2])')
    assert batch.num_graphs == len(batch) == 1
    assert batch.x.to_dense().tolist() == [[1], [2], [3]]
    assert batch.adj.coo()[0].tolist() == [0, 1, 1, 2]
    assert batch.adj.coo()[1].tolist() == [1, 0, 2, 1]
    assert batch.batch.tolist() == [0, 0, 0]
    assert batch.ptr.tolist() == [0, 3]

    batch = Batch.from_data_list([data1, data2, data3])

    assert str(batch) == ('DataBatch(x=[9, 1, nnz=9], adj=[9, 9, nnz=12], '
                          'batch=[9], ptr=[4])')
    assert batch.num_graphs == len(batch) == 3
    assert batch.x.to_dense().view(-1).tolist() == [1, 2, 3, 1, 2, 1, 2, 3, 4]
    assert batch.adj.coo()[0].tolist() == [0, 1, 1, 2, 3, 4, 5, 6, 6, 7, 7, 8]
    assert batch.adj.coo()[1].tolist() == [1, 0, 2, 1, 4, 3, 6, 5, 7, 6, 8, 7]
    assert batch.batch.tolist() == [0, 0, 0, 1, 1, 2, 2, 2, 2]
    assert batch.ptr.tolist() == [0, 3, 5, 9]

    assert str(batch[0]) == ("Data(x=[3, 1, nnz=3], adj=[3, 3, nnz=4])")
    assert str(batch[1]) == ("Data(x=[2, 1, nnz=2], adj=[2, 2, nnz=2])")
    assert str(batch[2]) == ("Data(x=[4, 1, nnz=4], adj=[4, 4, nnz=6])")

    data_list = batch.to_data_list()
    assert len(data_list) == 3

    assert len(data_list[0]) == 2
    assert data_list[0].x.to_dense().tolist() == [[1], [2], [3]]
    assert data_list[0].adj.coo()[0].tolist() == [0, 1, 1, 2]
    assert data_list[0].adj.coo()[1].tolist() == [1, 0, 2, 1]

    assert len(data_list[1]) == 2
    assert data_list[1].x.to_dense().tolist() == [[1], [2]]
    assert data_list[1].adj.coo()[0].tolist() == [0, 1]
    assert data_list[1].adj.coo()[1].tolist() == [1, 0]

    assert len(data_list[2]) == 2
    assert data_list[2].x.to_dense().tolist() == [[1], [2], [3], [4]]
    assert data_list[2].adj.coo()[0].tolist() == [0, 1, 1, 2, 2, 3]
    assert data_list[2].adj.coo()[1].tolist() == [1, 0, 2, 1, 3, 2]


def test_batch_with_torch_coo_tensor():
    x = torch.tensor([[1.0], [2.0], [3.0]]).to_sparse_coo()
    data1 = Data(x=x)

    x = torch.tensor([[1.0], [2.0]]).to_sparse_coo()
    data2 = Data(x=x)

    x = torch.tensor([[1.0], [2.0], [3.0], [4.0]]).to_sparse_coo()
    data3 = Data(x=x)

    batch = Batch.from_data_list([data1])
    assert str(batch) == ('DataBatch(x=[3, 1], batch=[3], ptr=[2])')
    assert batch.num_graphs == len(batch) == 1
    assert batch.x.to_dense().tolist() == [[1], [2], [3]]
    assert batch.batch.tolist() == [0, 0, 0]
    assert batch.ptr.tolist() == [0, 3]

    batch = Batch.from_data_list([data1, data2, data3])

    assert str(batch) == ('DataBatch(x=[9, 1], batch=[9], ptr=[4])')
    assert batch.num_graphs == len(batch) == 3
    assert batch.x.to_dense().view(-1).tolist() == [1, 2, 3, 1, 2, 1, 2, 3, 4]
    assert batch.batch.tolist() == [0, 0, 0, 1, 1, 2, 2, 2, 2]
    assert batch.ptr.tolist() == [0, 3, 5, 9]

    assert str(batch[0]) == ("Data(x=[3, 1])")
    assert str(batch[1]) == ("Data(x=[2, 1])")
    assert str(batch[2]) == ("Data(x=[4, 1])")

    data_list = batch.to_data_list()
    assert len(data_list) == 3

    assert len(data_list[0]) == 1
    assert data_list[0].x.to_dense().tolist() == [[1], [2], [3]]

    assert len(data_list[1]) == 1
    assert data_list[1].x.to_dense().tolist() == [[1], [2]]

    assert len(data_list[2]) == 1
    assert data_list[2].x.to_dense().tolist() == [[1], [2], [3], [4]]


def test_batching_with_new_dimension():
    torch_geometric.set_debug(True)

    class MyData(Data):
        def __cat_dim__(self, key, value, *args, **kwargs):
            if key == 'foo':
                return None
            else:
                return super().__cat_dim__(key, value, *args, **kwargs)

    x1 = torch.tensor([1, 2, 3], dtype=torch.float)
    foo1 = torch.randn(4)
    y1 = torch.tensor(1)

    x2 = torch.tensor([1, 2], dtype=torch.float)
    foo2 = torch.randn(4)
    y2 = torch.tensor(2)

    batch = Batch.from_data_list(
        [MyData(x=x1, foo=foo1, y=y1),
         MyData(x=x2, foo=foo2, y=y2)])

    assert str(batch) == ('MyDataBatch(x=[5], y=[2], foo=[2, 4], batch=[5], '
                          'ptr=[3])')
    assert batch.num_graphs == len(batch) == 2
    assert batch.x.tolist() == [1, 2, 3, 1, 2]
    assert batch.foo.size() == (2, 4)
    assert batch.foo[0].tolist() == foo1.tolist()
    assert batch.foo[1].tolist() == foo2.tolist()
    assert batch.y.tolist() == [1, 2]
    assert batch.batch.tolist() == [0, 0, 0, 1, 1]
    assert batch.ptr.tolist() == [0, 3, 5]
    assert batch.num_graphs == 2

    data = batch[0]
    assert str(data) == ('MyData(x=[3], y=[1], foo=[4])')
    data = batch[1]
    assert str(data) == ('MyData(x=[2], y=[1], foo=[4])')

    torch_geometric.set_debug(True)


def test_pickling(tmp_path):
    data = Data(x=torch.randn(5, 16))
    batch = Batch.from_data_list([data, data, data, data])
    assert id(batch._store._parent()) == id(batch)
    assert batch.num_nodes == 20

    # filename = f'{random.randrange(sys.maxsize)}.pt'
    path = osp.join(tmp_path, 'batch.pt')
    torch.save(batch, path)
    assert id(batch._store._parent()) == id(batch)
    assert batch.num_nodes == 20

    batch = torch.load(path)
    assert id(batch._store._parent()) == id(batch)
    assert batch.num_nodes == 20

    assert batch.__class__.__name__ == 'DataBatch'
    assert batch.num_graphs == len(batch) == 4


def test_recursive_batch():
    data1 = Data(
        x={
            '1': torch.randn(10, 32),
            '2': torch.randn(20, 48)
        },
        edge_index=[
            get_random_edge_index(30, 30, 50),
            get_random_edge_index(30, 30, 70)
        ],
        num_nodes=30,
    )

    data2 = Data(
        x={
            '1': torch.randn(20, 32),
            '2': torch.randn(40, 48)
        },
        edge_index=[
            get_random_edge_index(60, 60, 80),
            get_random_edge_index(60, 60, 90)
        ],
        num_nodes=60,
    )

    batch = Batch.from_data_list([data1, data2])

    assert batch.num_graphs == len(batch) == 2
    assert batch.num_nodes == 90

    assert torch.allclose(batch.x['1'],
                          torch.cat([data1.x['1'], data2.x['1']], dim=0))
    assert torch.allclose(batch.x['2'],
                          torch.cat([data1.x['2'], data2.x['2']], dim=0))
    assert (batch.edge_index[0].tolist() == torch.cat(
        [data1.edge_index[0], data2.edge_index[0] + 30], dim=1).tolist())
    assert (batch.edge_index[1].tolist() == torch.cat(
        [data1.edge_index[1], data2.edge_index[1] + 30], dim=1).tolist())
    assert batch.batch.size() == (90, )
    assert batch.ptr.size() == (3, )

    out1 = batch[0]
    assert len(out1) == 3
    assert out1.num_nodes == 30
    assert torch.allclose(out1.x['1'], data1.x['1'])
    assert torch.allclose(out1.x['2'], data1.x['2'])
    assert out1.edge_index[0].tolist(), data1.edge_index[0].tolist()
    assert out1.edge_index[1].tolist(), data1.edge_index[1].tolist()

    out2 = batch[1]
    assert len(out2) == 3
    assert out2.num_nodes == 60
    assert torch.allclose(out2.x['1'], data2.x['1'])
    assert torch.allclose(out2.x['2'], data2.x['2'])
    assert out2.edge_index[0].tolist(), data2.edge_index[0].tolist()
    assert out2.edge_index[1].tolist(), data2.edge_index[1].tolist()


def test_batching_of_batches():
    data = Data(x=torch.randn(2, 16))
    batch = Batch.from_data_list([data, data])

    batch = Batch.from_data_list([batch, batch])
    assert batch.num_graphs == len(batch) == 2
    assert batch.x[0:2].tolist() == data.x.tolist()
    assert batch.x[2:4].tolist() == data.x.tolist()
    assert batch.x[4:6].tolist() == data.x.tolist()
    assert batch.x[6:8].tolist() == data.x.tolist()
    assert batch.batch.tolist() == [0, 0, 1, 1, 2, 2, 3, 3]


def test_hetero_batch():
    e1 = ('p', 'a')
    e2 = ('a', 'p')
    data1 = HeteroData()
    data1['p'].x = torch.randn(100, 128)
    data1['a'].x = torch.randn(200, 128)
    data1[e1].edge_index = get_random_edge_index(100, 200, 500)
    data1[e1].edge_attr = torch.randn(500, 32)
    data1[e2].edge_index = get_random_edge_index(200, 100, 400)
    data1[e2].edge_attr = torch.randn(400, 32)

    data2 = HeteroData()
    data2['p'].x = torch.randn(50, 128)
    data2['a'].x = torch.randn(100, 128)
    data2[e1].edge_index = get_random_edge_index(50, 100, 300)
    data2[e1].edge_attr = torch.randn(300, 32)
    data2[e2].edge_index = get_random_edge_index(100, 50, 200)
    data2[e2].edge_attr = torch.randn(200, 32)

    batch = Batch.from_data_list([data1, data2])

    assert batch.num_graphs == len(batch) == 2
    assert batch.num_nodes == 450

    assert torch.allclose(batch['p'].x[:100], data1['p'].x)
    assert torch.allclose(batch['a'].x[:200], data1['a'].x)
    assert torch.allclose(batch['p'].x[100:], data2['p'].x)
    assert torch.allclose(batch['a'].x[200:], data2['a'].x)
    assert (batch[e1].edge_index.tolist() == torch.cat([
        data1[e1].edge_index,
        data2[e1].edge_index + torch.tensor([[100], [200]])
    ], 1).tolist())
    assert torch.allclose(
        batch[e1].edge_attr,
        torch.cat([data1[e1].edge_attr, data2[e1].edge_attr], 0))
    assert (batch[e2].edge_index.tolist() == torch.cat([
        data1[e2].edge_index,
        data2[e2].edge_index + torch.tensor([[200], [100]])
    ], 1).tolist())
    assert torch.allclose(
        batch[e2].edge_attr,
        torch.cat([data1[e2].edge_attr, data2[e2].edge_attr], 0))
    assert batch['p'].batch.size() == (150, )
    assert batch['p'].ptr.size() == (3, )
    assert batch['a'].batch.size() == (300, )
    assert batch['a'].ptr.size() == (3, )

    out1 = batch[0]
    assert len(out1) == 3
    assert out1.num_nodes == 300
    assert torch.allclose(out1['p'].x, data1['p'].x)
    assert torch.allclose(out1['a'].x, data1['a'].x)
    assert out1[e1].edge_index.tolist() == data1[e1].edge_index.tolist()
    assert torch.allclose(out1[e1].edge_attr, data1[e1].edge_attr)
    assert out1[e2].edge_index.tolist() == data1[e2].edge_index.tolist()
    assert torch.allclose(out1[e2].edge_attr, data1[e2].edge_attr)

    out2 = batch[1]
    assert len(out2) == 3
    assert out2.num_nodes == 150
    assert torch.allclose(out2['p'].x, data2['p'].x)
    assert torch.allclose(out2['a'].x, data2['a'].x)
    assert out2[e1].edge_index.tolist() == data2[e1].edge_index.tolist()
    assert torch.allclose(out2[e1].edge_attr, data2[e1].edge_attr)
    assert out2[e2].edge_index.tolist() == data2[e2].edge_index.tolist()
    assert torch.allclose(out2[e2].edge_attr, data2[e2].edge_attr)


def test_pair_data_batching():
    class PairData(Data):
        def __inc__(self, key, value, *args, **kwargs):
            if key == 'edge_index_s':
                return self.x_s.size(0)
            if key == 'edge_index_t':
                return self.x_t.size(0)
            return super().__inc__(key, value, *args, **kwargs)

    x_s = torch.randn(5, 16)
    edge_index_s = torch.tensor([
        [0, 0, 0, 0],
        [1, 2, 3, 4],
    ])
    x_t = torch.randn(4, 16)
    edge_index_t = torch.tensor([
        [0, 0, 0],
        [1, 2, 3],
    ])

    data = PairData(x_s=x_s, edge_index_s=edge_index_s, x_t=x_t,
                    edge_index_t=edge_index_t)
    batch = Batch.from_data_list([data, data])

    assert torch.allclose(batch.x_s, torch.cat([x_s, x_s], dim=0))
    assert batch.edge_index_s.tolist() == [[0, 0, 0, 0, 5, 5, 5, 5],
                                           [1, 2, 3, 4, 6, 7, 8, 9]]

    assert torch.allclose(batch.x_t, torch.cat([x_t, x_t], dim=0))
    assert batch.edge_index_t.tolist() == [[0, 0, 0, 4, 4, 4],
                                           [1, 2, 3, 5, 6, 7]]


def test_batch_with_empty_list():
    x = torch.randn(4, 1)
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    data = Data(x=x, edge_index=edge_index, nontensor=[])

    batch = Batch.from_data_list([data, data])
    assert batch.nontensor == [[], []]
    assert batch[0].nontensor == []
    assert batch[1].nontensor == []


def test_nested_follow_batch():
    def tr(n, m):
        return torch.rand((n, m))

    d1 = Data(xs=[tr(4, 3), tr(11, 4), tr(1, 2)], a={"aa": tr(11, 3)},
              x=tr(10, 5))
    d2 = Data(xs=[tr(5, 3), tr(14, 4), tr(3, 2)], a={"aa": tr(2, 3)},
              x=tr(11, 5))
    d3 = Data(xs=[tr(6, 3), tr(15, 4), tr(2, 2)], a={"aa": tr(4, 3)},
              x=tr(9, 5))
    d4 = Data(xs=[tr(4, 3), tr(16, 4), tr(1, 2)], a={"aa": tr(8, 3)},
              x=tr(8, 5))

    data_list = [d1, d2, d3, d4]

    batch = Batch.from_data_list(data_list, follow_batch=['xs', 'a'])

    assert batch.xs[0].shape == (19, 3)
    assert batch.xs[1].shape == (56, 4)
    assert batch.xs[2].shape == (7, 2)
    assert batch.a['aa'].shape == (25, 3)

    assert len(batch.xs_batch) == 3
    assert len(batch.a_batch) == 1

    assert batch.xs_batch[0].tolist() == \
           [0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3]
    assert batch.xs_batch[1].tolist() == \
           [0] * 11 + [1] * 14 + [2] * 15 + [3] * 16
    assert batch.xs_batch[2].tolist() == \
           [0] * 1 + [1] * 3 + [2] * 2 + [3] * 1

    assert batch.a_batch['aa'].tolist() == \
           [0] * 11 + [1] * 2 + [2] * 4 + [3] * 8


@withPackage('torch>=2.0.0')
@pytest.mark.parametrize('layout', [
    torch.sparse_coo,
    torch.sparse_csr,
    torch.sparse_csc,
])
def test_torch_sparse_batch(layout):
    x_dense = torch.randn(3, 4)
    x = x_dense.to_sparse(layout=layout)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.rand(4)
    adj = to_torch_sparse_tensor(edge_index, edge_attr, layout=layout)

    data = Data(x=x, adj=adj)

    batch = Batch.from_data_list([data, data])

    assert batch.x.size() == (6, 4)
    assert batch.x.layout == layout
    assert torch.equal(batch.x.to_dense(), torch.cat([x_dense, x_dense], 0))

    assert batch.adj.size() == (6, 6)
    assert batch.adj.layout == layout
    out = to_edge_index(batch.adj.to_sparse(layout=torch.sparse_csr))
    assert torch.equal(out[0], torch.cat([edge_index, edge_index + 3], 1))
    assert torch.equal(out[1], torch.cat([edge_attr, edge_attr], 0))
pytorch/test/data/test_data.py000066400000000000000000000335421444262304300167710ustar00rootroot00000000000000import copy

import pytest
import torch
import torch.multiprocessing as mp

import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.storage import AttrType
from torch_geometric.testing import withPackage


def test_data():
    torch_geometric.set_debug(True)

    x = torch.tensor([[1, 3, 5], [2, 4, 6]], dtype=torch.float).t()
    edge_index = torch.tensor([[0, 0, 1, 1, 2], [1, 1, 0, 2, 1]])
    data = Data(x=x, edge_index=edge_index).to(torch.device('cpu'))
    data.validate(raise_on_error=True)

    N = data.num_nodes
    assert N == 3

    assert data.node_attrs() == ['x']
    assert data.edge_attrs() == ['edge_index']

    assert data.x.tolist() == x.tolist()
    assert data['x'].tolist() == x.tolist()
    assert data.get('x').tolist() == x.tolist()
    assert data.get('y', 2) == 2
    assert data.get('y', None) is None

    assert sorted(data.keys) == ['edge_index', 'x']
    assert len(data) == 2
    assert 'x' in data and 'edge_index' in data and 'pos' not in data

    D = data.to_dict()
    assert len(D) == 2
    assert 'x' in D and 'edge_index' in D

    D = data.to_namedtuple()
    assert len(D) == 2
    assert D.x is not None and D.edge_index is not None

    assert data.__cat_dim__('x', data.x) == 0
    assert data.__cat_dim__('edge_index', data.edge_index) == -1
    assert data.__inc__('x', data.x) == 0
    assert data.__inc__('edge_index', data.edge_index) == data.num_nodes

    assert not data.x.is_contiguous()
    data.contiguous()
    assert data.x.is_contiguous()

    assert not data.is_coalesced()
    data = data.coalesce()
    assert data.is_coalesced()

    clone = data.clone()
    assert clone != data
    assert len(clone) == len(data)
    assert clone.x.data_ptr() != data.x.data_ptr()
    assert clone.x.tolist() == data.x.tolist()
    assert clone.edge_index.data_ptr() != data.edge_index.data_ptr()
    assert clone.edge_index.tolist() == data.edge_index.tolist()

    # Test `data.to_heterogeneous()`:
    out = data.to_heterogeneous()
    assert torch.allclose(data.x, out['0'].x)
    assert torch.allclose(data.edge_index, out['0', '0'].edge_index)

    data.edge_type = torch.tensor([0, 0, 1, 0])
    out = data.to_heterogeneous()
    assert torch.allclose(data.x, out['0'].x)
    assert [store.num_edges for store in out.edge_stores] == [3, 1]
    data.edge_type = None

    data['x'] = x + 1
    assert data.x.tolist() == (x + 1).tolist()

    assert str(data) == 'Data(x=[3, 2], edge_index=[2, 4])'

    dictionary = {'x': data.x, 'edge_index': data.edge_index}
    data = Data.from_dict(dictionary)
    assert sorted(data.keys) == ['edge_index', 'x']

    assert not data.has_isolated_nodes()
    assert not data.has_self_loops()
    assert data.is_undirected()
    assert not data.is_directed()

    assert data.num_nodes == 3
    assert data.num_edges == 4
    with pytest.warns(UserWarning, match='deprecated'):
        assert data.num_faces is None
    assert data.num_node_features == 2
    assert data.num_features == 2

    data.edge_attr = torch.randn(data.num_edges, 2)
    assert data.num_edge_features == 2
    data.edge_attr = None

    data.x = None
    with pytest.warns(UserWarning, match='Unable to accurately infer'):
        assert data.num_nodes == 3

    data.edge_index = None
    with pytest.warns(UserWarning, match='Unable to accurately infer'):
        assert data.num_nodes is None
    assert data.num_edges == 0

    data.num_nodes = 4
    assert data.num_nodes == 4

    data = Data(x=x, attribute=x)
    assert len(data) == 2
    assert data.x.tolist() == x.tolist()
    assert data.attribute.tolist() == x.tolist()

    face = torch.tensor([[0, 1], [1, 2], [2, 3]])
    data = Data(num_nodes=4, face=face)
    with pytest.warns(UserWarning, match='deprecated'):
        assert data.num_faces == 2
    assert data.num_nodes == 4

    data = Data(title='test')
    assert str(data) == "Data(title='test')"
    assert data.num_node_features == 0
    assert data.num_edge_features == 0

    key = value = 'test_value'
    data[key] = value
    assert data[key] == value
    del data[value]
    del data[value]  # Deleting unset attributes should work as well.

    assert data.get(key) is None
    assert data.get('title') == 'test'

    torch_geometric.set_debug(False)


def test_data_attr_cache():
    x = torch.randn(3, 16)
    edge_index = torch.tensor([[0, 0, 1, 1, 2], [1, 1, 0, 2, 1]])
    edge_attr = torch.randn(5, 4)
    y = torch.tensor([0])

    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr, y=y)

    assert data.is_node_attr('x')
    assert 'x' in data._store._cached_attr[AttrType.NODE]
    assert 'x' not in data._store._cached_attr[AttrType.EDGE]
    assert 'x' not in data._store._cached_attr[AttrType.OTHER]

    assert not data.is_node_attr('edge_index')
    assert 'edge_index' not in data._store._cached_attr[AttrType.NODE]
    assert 'edge_index' in data._store._cached_attr[AttrType.EDGE]
    assert 'edge_index' not in data._store._cached_attr[AttrType.OTHER]

    assert data.is_edge_attr('edge_attr')
    assert 'edge_attr' not in data._store._cached_attr[AttrType.NODE]
    assert 'edge_attr' in data._store._cached_attr[AttrType.EDGE]
    assert 'edge_attr' not in data._store._cached_attr[AttrType.OTHER]

    assert not data.is_edge_attr('y')
    assert 'y' not in data._store._cached_attr[AttrType.NODE]
    assert 'y' not in data._store._cached_attr[AttrType.EDGE]
    assert 'y' in data._store._cached_attr[AttrType.OTHER]


def test_to_heterogeneous_empty_edge_index():
    data = Data(
        x=torch.randn(5, 10),
        edge_index=torch.empty(2, 0, dtype=torch.long),
    )
    hetero_data = data.to_heterogeneous()
    assert hetero_data.node_types == ['0']
    assert hetero_data.edge_types == []
    assert len(hetero_data) == 1
    assert torch.equal(hetero_data['0'].x, data.x)

    hetero_data = data.to_heterogeneous(
        node_type_names=['0'],
        edge_type_names=[('0', 'to', '0')],
    )
    assert hetero_data.node_types == ['0']
    assert hetero_data.edge_types == [('0', 'to', '0')]
    assert len(hetero_data) == 2
    assert torch.equal(hetero_data['0'].x, data.x)
    assert torch.equal(hetero_data['0', 'to', '0'].edge_index, data.edge_index)


def test_data_subgraph():
    x = torch.arange(5)
    y = torch.tensor([0.])
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4],
                               [1, 0, 2, 1, 3, 2, 4, 3]])
    edge_weight = torch.arange(edge_index.size(1))

    data = Data(x=x, y=y, edge_index=edge_index, edge_weight=edge_weight,
                num_nodes=5)

    out = data.subgraph(torch.tensor([1, 2, 3]))
    assert len(out) == 5
    assert torch.equal(out.x, torch.arange(1, 4))
    assert torch.equal(out.y, data.y)
    assert out.edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert torch.equal(out.edge_weight, edge_weight[torch.arange(2, 6)])
    assert out.num_nodes == 3

    # Test unordered selection:
    out = data.subgraph(torch.tensor([3, 1, 2]))
    assert len(out) == 5
    assert torch.equal(out.x, torch.tensor([3, 1, 2]))
    assert torch.equal(out.y, data.y)
    assert out.edge_index.tolist() == [[1, 2, 2, 0], [2, 1, 0, 2]]
    assert torch.equal(out.edge_weight, edge_weight[torch.arange(2, 6)])
    assert out.num_nodes == 3

    out = data.subgraph(torch.tensor([False, False, False, True, True]))
    assert len(out) == 5
    assert torch.equal(out.x, torch.arange(3, 5))
    assert torch.equal(out.y, data.y)
    assert out.edge_index.tolist() == [[0, 1], [1, 0]]
    assert torch.equal(out.edge_weight, edge_weight[torch.arange(6, 8)])
    assert out.num_nodes == 2

    out = data.edge_subgraph(torch.tensor([1, 2, 3]))
    assert len(out) == 5
    assert out.num_nodes == data.num_nodes
    assert torch.equal(out.x, data.x)
    assert torch.equal(out.y, data.y)
    assert out.edge_index.tolist() == [[1, 1, 2], [0, 2, 1]]
    assert torch.equal(out.edge_weight, edge_weight[torch.tensor([1, 2, 3])])

    out = data.edge_subgraph(
        torch.tensor([False, True, True, True, False, False, False, False]))
    assert len(out) == 5
    assert out.num_nodes == data.num_nodes
    assert torch.equal(out.x, data.x)
    assert torch.equal(out.y, data.y)
    assert out.edge_index.tolist() == [[1, 1, 2], [0, 2, 1]]
    assert torch.equal(out.edge_weight, edge_weight[torch.tensor([1, 2, 3])])


def test_data_subgraph_with_list_field():
    x = torch.arange(5)
    y = list(range(5))
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4],
                               [1, 0, 2, 1, 3, 2, 4, 3]])
    data = Data(x=x, y=y, edge_index=edge_index)

    out = data.subgraph(torch.tensor([1, 2, 3]))
    assert len(out) == 3
    assert out.x.tolist() == out.y == [1, 2, 3]

    out = data.subgraph(torch.tensor([False, True, True, True, False]))
    assert len(out) == 3
    assert out.x.tolist() == out.y == [1, 2, 3]


def test_copy_data():
    data = Data(x=torch.randn(20, 5))

    out = copy.copy(data)
    assert id(data) != id(out)
    assert id(data._store) != id(out._store)
    assert len(data.stores) == len(out.stores)
    for store1, store2 in zip(data.stores, out.stores):
        assert id(store1) != id(store2)
        assert id(data) == id(store1._parent())
        assert id(out) == id(store2._parent())
    assert data.x.data_ptr() == out.x.data_ptr()

    out = copy.deepcopy(data)
    assert id(data) != id(out)
    assert id(data._store) != id(out._store)
    assert len(data.stores) == len(out.stores)
    for store1, store2 in zip(data.stores, out.stores):
        assert id(store1) != id(store2)
        assert id(data) == id(store1._parent())
        assert id(out) == id(store2._parent())
    assert data.x.data_ptr() != out.x.data_ptr()
    assert data.x.tolist() == out.x.tolist()


def test_debug_data():
    torch_geometric.set_debug(True)

    Data()
    Data(edge_index=torch.zeros((2, 0), dtype=torch.long), num_nodes=10)
    Data(face=torch.zeros((3, 0), dtype=torch.long), num_nodes=10)
    Data(edge_index=torch.tensor([[0, 1], [1, 0]]), edge_attr=torch.randn(2))
    Data(x=torch.torch.randn(5, 3), num_nodes=5)
    Data(pos=torch.torch.randn(5, 3), num_nodes=5)
    Data(norm=torch.torch.randn(5, 3), num_nodes=5)

    torch_geometric.set_debug(False)


def run(rank, data_list):
    for data in data_list:
        assert data.x.is_shared()
        data.x.add_(1)


def test_data_share_memory():
    data_list = [Data(x=torch.zeros(8)) for _ in range(10)]

    for data in data_list:
        assert not data.x.is_shared()

    mp.spawn(run, args=(data_list, ), nprocs=4, join=True)

    for data in data_list:
        assert data.x.is_shared()
        assert torch.allclose(data.x, torch.full((8, ), 4.))


def test_data_setter_properties():
    class MyData(Data):
        def __init__(self):
            super().__init__()
            self.my_attr1 = 1
            self.my_attr2 = 2

        @property
        def my_attr1(self):
            return self._my_attr1

        @my_attr1.setter
        def my_attr1(self, value):
            self._my_attr1 = value

    data = MyData()
    assert data.my_attr2 == 2

    assert 'my_attr1' not in data._store
    assert data.my_attr1 == 1

    data.my_attr1 = 2
    assert 'my_attr1' not in data._store
    assert data.my_attr1 == 2


def test_data_update():
    data = Data(x=torch.arange(0, 5), y=torch.arange(5, 10))
    other = Data(z=torch.arange(10, 15), x=torch.arange(15, 20))
    data.update(other)

    assert len(data) == 3
    assert torch.equal(data.x, torch.arange(15, 20))
    assert torch.equal(data.y, torch.arange(5, 10))
    assert torch.equal(data.z, torch.arange(10, 15))


# Feature Store ###############################################################


def test_basic_feature_store():
    data = Data()
    x = torch.randn(20, 20)
    data.not_a_tensor_attr = 10  # don't include, not a tensor attr
    data.bad_attr = torch.randn(10, 20)  # don't include, bad cat_dim

    # Put tensor:
    assert data.put_tensor(copy.deepcopy(x), attr_name='x', index=None)
    assert torch.equal(data.x, x)

    # Put (modify) tensor slice:
    x[15:] = 0
    data.put_tensor(0, attr_name='x', index=slice(15, None, None))

    # Get tensor:
    out = data.get_tensor(attr_name='x', index=None)
    assert torch.equal(x, out)

    # Get tensor size:
    assert data.get_tensor_size(attr_name='x') == (20, 20)

    # Get tensor attrs:
    tensor_attrs = data.get_all_tensor_attrs()
    assert len(tensor_attrs) == 1
    assert tensor_attrs[0].attr_name == 'x'

    # Remove tensor:
    assert 'x' in data.__dict__['_store']
    data.remove_tensor(attr_name='x', index=None)
    assert 'x' not in data.__dict__['_store']


# Graph Store #################################################################


@withPackage('torch_sparse')
def test_basic_graph_store():
    r"""Test the core graph store API."""
    data = Data()

    def assert_equal_tensor_tuple(expected, actual):
        assert len(expected) == len(actual)
        for i in range(len(expected)):
            assert torch.equal(expected[i], actual[i])

    # We put all three tensor types: COO, CSR, and CSC, and we get them back
    # to confirm that `GraphStore` works as intended.
    coo = (torch.tensor([0, 1]), torch.tensor([1, 2]))
    csr = (torch.tensor([0, 1, 2, 2]), torch.tensor([1, 2]))
    csc = (torch.tensor([0, 1]), torch.tensor([0, 0, 1, 2]))

    # Put:
    data.put_edge_index(coo, layout='coo', size=(3, 3))
    data.put_edge_index(csr, layout='csr')
    data.put_edge_index(csc, layout='csc')

    # Get:
    assert_equal_tensor_tuple(coo, data.get_edge_index('coo'))
    assert_equal_tensor_tuple(csr, data.get_edge_index('csr'))
    assert_equal_tensor_tuple(csc, data.get_edge_index('csc'))

    # Get attrs:
    edge_attrs = data.get_all_edge_attrs()
    assert len(edge_attrs) == 3


def test_data_generate_ids():
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 0, 1, 1, 2], [1, 1, 0, 2, 1]])

    data = Data(x=x, edge_index=edge_index)
    assert len(data) == 2

    data.generate_ids()
    assert len(data) == 4
    assert data.n_id.tolist() == [0, 1, 2]
    assert data.e_id.tolist() == [0, 1, 2, 3, 4]
pytorch/test/data/test_datapipes.py000066400000000000000000000031141444262304300200220ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import DatasetAdapter
from torch_geometric.loader import DataLoader
from torch_geometric.testing import withPackage
from torch_geometric.utils import to_smiles


@pytest.fixture()
def dataset_adapter() -> DatasetAdapter:
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1], [1, 0, 2]])
    data = Data(x=x, edge_index=edge_index)
    return DatasetAdapter([data, data, data, data])


def test_dataset_adapter(dataset_adapter):
    loader = DataLoader(dataset_adapter, batch_size=2)
    batch = next(iter(loader))
    assert batch.x.shape == (6, 8)
    assert len(loader) == 2

    # Test sharding:
    dataset_adapter.apply_sharding(2, 0)
    assert len([data for data in dataset_adapter]) == 2

    assert dataset_adapter.is_shardable()


def test_datapipe_batch_graphs(dataset_adapter):
    dp = dataset_adapter.batch_graphs(batch_size=2)
    assert len(dp) == 2
    batch = next(iter(dp))
    assert batch.x.shape == (6, 8)


def test_functional_transform(dataset_adapter):
    assert next(iter(dataset_adapter)).is_directed()
    dataset_adapter = dataset_adapter.to_undirected()
    assert next(iter(dataset_adapter)).is_undirected()


@withPackage('rdkit')
def test_datapipe_parse_smiles():
    smiles = 'F/C=C/F'

    dp = DatasetAdapter([smiles])
    dp = dp.parse_smiles()
    assert to_smiles(next(iter(dp))) == smiles

    dp = DatasetAdapter([{'abc': smiles, 'cba': '1.0'}])
    dp = dp.parse_smiles(smiles_key='abc', target_key='cba')
    assert to_smiles(next(iter(dp))) == smiles
pytorch/test/data/test_dataset.py000066400000000000000000000273661444262304300175140ustar00rootroot00000000000000import copy

import pytest
import torch

from torch_geometric.data import Data, HeteroData, InMemoryDataset
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor


class MyTestDataset(InMemoryDataset):
    def __init__(self, data_list, transform=None):
        super().__init__(None, transform=transform)
        self.data, self.slices = self.collate(data_list)


class MyStoredTestDataset(InMemoryDataset):
    def __init__(self, root, data_list, transform=None):
        self.data_list = data_list
        super().__init__(root, transform=transform)
        self.load(self.processed_paths[0], data_cls=data_list[0].__class__)

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def process(self):
        self.save(self.data_list, self.processed_paths[0])


def test_in_memory_dataset():
    x1 = torch.Tensor([[1], [1], [1]])
    x2 = torch.Tensor([[2], [2], [2]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    face = torch.tensor([[0], [1], [2]])

    data1 = Data(x1, edge_index, face=face, test_int=1, test_str='1')
    data1.num_nodes = 10

    data2 = Data(x2, edge_index, face=face, test_int=2, test_str='2')
    data2.num_nodes = 5

    dataset = MyTestDataset([data1, data2])
    assert str(dataset) == 'MyTestDataset(2)'
    assert len(dataset) == 2

    assert len(dataset[0]) == 6
    assert dataset[0].num_nodes == 10
    assert dataset[0].x.tolist() == x1.tolist()
    assert dataset[0].edge_index.tolist() == edge_index.tolist()
    assert dataset[0].face.tolist() == face.tolist()
    assert dataset[0].test_int == 1
    assert dataset[0].test_str == '1'

    assert len(dataset[1]) == 6
    assert dataset[1].num_nodes == 5
    assert dataset[1].x.tolist() == x2.tolist()
    assert dataset[1].edge_index.tolist() == edge_index.tolist()
    assert dataset[1].face.tolist() == face.tolist()
    assert dataset[1].test_int == 2
    assert dataset[1].test_str == '2'

    with pytest.warns(UserWarning, match="internal storage format"):
        dataset.data

    assert torch.equal(dataset.x, torch.cat([x1, x2], dim=0))
    assert dataset.edge_index.tolist() == [
        [0, 1, 1, 2, 10, 11, 11, 12],
        [1, 0, 2, 1, 11, 10, 12, 11],
    ]
    assert torch.equal(dataset[1:].x, x2)


def test_stored_in_memory_dataset(tmp_path):
    x1 = torch.Tensor([[1], [1], [1]])
    x2 = torch.Tensor([[2], [2], [2], [2]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    data1 = Data(x1, edge_index, num_nodes=3, test_int=1, test_str='1')
    data2 = Data(x2, edge_index, num_nodes=4, test_int=2, test_str='2')

    dataset = MyStoredTestDataset(tmp_path, [data1, data2])
    assert dataset._data.num_nodes == 7
    assert dataset._data._num_nodes == [3, 4]

    assert torch.equal(dataset[0].x, x1)
    assert torch.equal(dataset[0].edge_index, edge_index)
    assert dataset[0].num_nodes == 3
    assert torch.equal(dataset[0].test_int, torch.tensor([1]))
    assert dataset[0].test_str == '1'

    assert torch.equal(dataset[1].x, x2)
    assert torch.equal(dataset[1].edge_index, edge_index)
    assert dataset[1].num_nodes == 4
    assert torch.equal(dataset[1].test_int, torch.tensor([2]))
    assert dataset[1].test_str == '2'


def test_stored_hetero_in_memory_dataset(tmp_path):
    x1 = torch.Tensor([[1], [1], [1]])
    x2 = torch.Tensor([[2], [2], [2], [2]])

    data1 = HeteroData()
    data1['paper'].x = x1
    data1['paper'].num_nodes = 3

    data2 = HeteroData()
    data2['paper'].x = x2
    data2['paper'].num_nodes = 4

    dataset = MyStoredTestDataset(tmp_path, [data1, data2])
    assert dataset._data['paper'].num_nodes == 7
    assert dataset._data['paper']._num_nodes == [3, 4]

    assert torch.equal(dataset[0]['paper'].x, x1)
    assert dataset[0]['paper'].num_nodes == 3

    assert torch.equal(dataset[1]['paper'].x, x2)
    assert dataset[1]['paper'].num_nodes == 4


def test_in_memory_num_classes():
    dataset = MyTestDataset([Data(), Data()])
    assert dataset.num_classes == 0

    dataset = MyTestDataset([Data(y=0), Data(y=1)])
    assert dataset.num_classes == 2

    dataset = MyTestDataset([Data(y=1.5), Data(y=2.5), Data(y=3.5)])
    assert dataset.num_classes == 3

    dataset = MyTestDataset([
        Data(y=torch.tensor([[0, 1, 0, 1]])),
        Data(y=torch.tensor([[1, 0, 0, 0]])),
        Data(y=torch.tensor([[0, 0, 1, 0]])),
    ])
    assert dataset.num_classes == 4

    # Test when `__getitem__` returns a tuple of data objects.
    def transform(data):
        copied_data = copy.copy(data)
        copied_data.y += 1
        return data, copied_data, 'foo'

    dataset = MyTestDataset([Data(y=0), Data(y=1)], transform=transform)
    assert dataset.num_classes == 3


def test_in_memory_dataset_copy():
    data_list = [Data(x=torch.randn(5, 16)) for _ in range(4)]
    dataset = MyTestDataset(data_list)

    copied_dataset = dataset.copy()
    assert id(copied_dataset) != id(dataset)

    assert len(copied_dataset) == len(dataset) == 4
    for copied_data, data in zip(copied_dataset, dataset):
        assert torch.equal(copied_data.x, data.x)

    copied_dataset = dataset.copy([1, 2])
    assert len(copied_dataset) == 2
    assert torch.equal(copied_dataset[0].x, data_list[1].x)
    assert torch.equal(copied_dataset[1].x, data_list[2].x)


def test_to_datapipe():
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    data = Data(x=x, edge_index=edge_index)
    dataset = MyTestDataset([data, data])

    dp = dataset.to_datapipe()

    assert isinstance(dp, torch.utils.data.IterDataPipe)
    assert len(dp) == 2

    assert torch.equal(dataset[0].x, list(dp)[0].x)
    assert torch.equal(dataset[0].edge_index, list(dp)[0].edge_index)
    assert torch.equal(dataset[1].x, list(dp)[1].x)
    assert torch.equal(dataset[1].edge_index, list(dp)[1].edge_index)


@withPackage('torch_sparse')
def test_in_memory_sparse_tensor_dataset():
    x = torch.randn(11, 16)
    adj = SparseTensor(
        row=torch.tensor([4, 1, 3, 2, 2, 3]),
        col=torch.tensor([1, 3, 2, 3, 3, 2]),
        sparse_sizes=(11, 11),
    )
    data = Data(x=x, adj=adj)

    dataset = MyTestDataset([data, data])
    assert len(dataset) == 2
    assert torch.allclose(dataset[0].x, x)
    assert dataset[0].adj.sparse_sizes() == (11, 11)
    assert torch.allclose(dataset[1].x, x)
    assert dataset[1].adj.sparse_sizes() == (11, 11)


def test_collate_with_new_dimension():
    class MyData(Data):
        def __cat_dim__(self, key, value, *args, **kwargs):
            if key == 'foo':
                return None
            else:
                return super().__cat_dim__(key, value, *args, **kwargs)

    x = torch.tensor([1, 2, 3], dtype=torch.float)
    foo = torch.randn(4)
    y = torch.tensor(1)

    data = MyData(x=x, foo=foo, y=y)

    dataset = MyTestDataset([data, data])
    assert str(dataset) == 'MyTestDataset(2)'
    assert len(dataset) == 2

    data1 = dataset[0]
    assert len(data1) == 3
    assert data1.x.tolist() == x.tolist()
    assert data1.foo.tolist() == foo.tolist()
    assert data1.y.tolist() == [1]

    data2 = dataset[0]
    assert len(data2) == 3
    assert data2.x.tolist() == x.tolist()
    assert data2.foo.tolist() == foo.tolist()
    assert data2.y.tolist() == [1]


def test_hetero_in_memory_dataset():
    data1 = HeteroData()
    data1.y = torch.randn(5)
    data1['paper'].x = torch.randn(10, 16)
    data1['paper', 'paper'].edge_index = torch.randint(0, 10, (2, 30)).long()

    data2 = HeteroData()
    data2.y = torch.randn(5)
    data2['paper'].x = torch.randn(10, 16)
    data2['paper', 'paper'].edge_index = torch.randint(0, 10, (2, 30)).long()

    dataset = MyTestDataset([data1, data2])
    assert str(dataset) == 'MyTestDataset(2)'
    assert len(dataset) == 2

    assert len(dataset[0]) == 3
    assert dataset[0].y.tolist() == data1.y.tolist()
    assert dataset[0]['paper'].x.tolist() == data1['paper'].x.tolist()
    assert (dataset[0]['paper', 'paper'].edge_index.tolist() == data1[
        'paper', 'paper'].edge_index.tolist())

    assert len(dataset[1]) == 3
    assert dataset[1].y.tolist() == data2.y.tolist()
    assert dataset[1]['paper'].x.tolist() == data2['paper'].x.tolist()
    assert (dataset[1]['paper', 'paper'].edge_index.tolist() == data2[
        'paper', 'paper'].edge_index.tolist())


def test_override_behavior():
    class DS1(InMemoryDataset):
        def __init__(self):
            self.enter_download = False
            self.enter_process = False
            super().__init__()

        def _download(self):
            self.enter_download = True

        def _process(self):
            self.enter_process = True

        def download(self):
            pass

        def process(self):
            pass

    class DS2(InMemoryDataset):
        def __init__(self):
            self.enter_download = False
            self.enter_process = False
            super().__init__()

        def _download(self):
            self.enter_download = True

        def _process(self):
            self.enter_process = True

        def process(self):
            pass

    class DS3(InMemoryDataset):
        def __init__(self):
            self.enter_download = False
            self.enter_process = False
            super().__init__()

        def _download(self):
            self.enter_download = True

        def _process(self):
            self.enter_process = True

    class DS4(DS1):
        pass

    ds = DS1()
    assert ds.enter_download
    assert ds.enter_process

    ds = DS2()
    assert not ds.enter_download
    assert ds.enter_process

    ds = DS3()
    assert not ds.enter_download
    assert not ds.enter_process

    ds = DS4()
    assert ds.enter_download
    assert ds.enter_process


def test_lists_of_tensors_in_memory_dataset():
    def tr(n, m):
        return torch.rand((n, m))

    d1 = Data(xs=[tr(4, 3), tr(11, 4), tr(1, 2)])
    d2 = Data(xs=[tr(5, 3), tr(14, 4), tr(3, 2)])
    d3 = Data(xs=[tr(6, 3), tr(15, 4), tr(2, 2)])
    d4 = Data(xs=[tr(4, 3), tr(16, 4), tr(1, 2)])

    data_list = [d1, d2, d3, d4]

    dataset = MyTestDataset(data_list)
    assert len(dataset) == 4
    assert dataset[0].xs[1].size() == (11, 4)
    assert dataset[0].xs[2].size() == (1, 2)
    assert dataset[1].xs[0].size() == (5, 3)
    assert dataset[2].xs[1].size() == (15, 4)
    assert dataset[3].xs[1].size() == (16, 4)


@withPackage('torch_sparse')
def test_lists_of_sparse_tensors():
    e1 = torch.tensor([[4, 1, 3, 2, 2, 3], [1, 3, 2, 3, 3, 2]])
    e2 = torch.tensor([[0, 1, 4, 7, 2, 9], [7, 2, 2, 1, 4, 7]])
    e3 = torch.tensor([[3, 5, 1, 2, 3, 3], [5, 0, 2, 1, 3, 7]])
    e4 = torch.tensor([[0, 1, 9, 2, 0, 3], [1, 1, 2, 1, 3, 2]])
    adj1 = SparseTensor.from_edge_index(e1, sparse_sizes=(11, 11))
    adj2 = SparseTensor.from_edge_index(e2, sparse_sizes=(22, 22))
    adj3 = SparseTensor.from_edge_index(e3, sparse_sizes=(12, 12))
    adj4 = SparseTensor.from_edge_index(e4, sparse_sizes=(15, 15))

    d1 = Data(adj_test=[adj1, adj2])
    d2 = Data(adj_test=[adj3, adj4])

    data_list = [d1, d2]
    dataset = MyTestDataset(data_list)
    assert len(dataset) == 2
    assert dataset[0].adj_test[0].sparse_sizes() == (11, 11)
    assert dataset[0].adj_test[1].sparse_sizes() == (22, 22)
    assert dataset[1].adj_test[0].sparse_sizes() == (12, 12)
    assert dataset[1].adj_test[1].sparse_sizes() == (15, 15)


def test_file_names_as_property_and_method():
    class MyTestDataset(InMemoryDataset):
        def __init__(self):
            super().__init__('/tmp/MyTestDataset')

        @property
        def raw_file_names(self):
            return ['test_file']

        def download(self):
            pass

    MyTestDataset()

    class MyTestDataset(InMemoryDataset):
        def __init__(self):
            super().__init__('/tmp/MyTestDataset')

        def raw_file_names(self):
            return ['test_file']

        def download(self):
            pass

    MyTestDataset()
pytorch/test/data/test_dataset_summary.py000066400000000000000000000064551444262304300212650ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.data.summary import Stats, Summary
from torch_geometric.datasets import FakeDataset, FakeHeteroDataset
from torch_geometric.testing import withPackage


def check_stats(stats: Stats, expected: Tensor):
    assert stats.mean == float(expected.mean())
    assert stats.std == float(expected.std())
    assert stats.min == float(expected.min())
    assert stats.quantile25 == float(expected.quantile(0.25))
    assert stats.median == float(expected.median())
    assert stats.quantile75 == float(expected.quantile(0.75))
    assert stats.max == float(expected.max())


def test_dataset_summary():
    dataset = FakeDataset(num_graphs=10)
    num_nodes = torch.Tensor([data.num_nodes for data in dataset])
    num_edges = torch.Tensor([data.num_edges for data in dataset])

    summary = dataset.get_summary()

    assert summary.name == 'FakeDataset'
    assert summary.num_graphs == 10

    check_stats(summary.num_nodes, num_nodes)
    check_stats(summary.num_edges, num_edges)


@withPackage('tabulate')
def test_dataset_summary_representation():
    dataset = FakeDataset(num_graphs=10)

    summary1 = Summary.from_dataset(dataset, per_type=False)
    summary2 = Summary.from_dataset(dataset, per_type=True)

    assert str(summary1) == str(summary2)


@withPackage('tabulate')
def test_dataset_summary_hetero():
    dataset1 = FakeHeteroDataset(num_graphs=10)
    summary1 = Summary.from_dataset(dataset1, per_type=False)

    dataset2 = [data.to_homogeneous() for data in dataset1]
    summary2 = Summary.from_dataset(dataset2)
    summary2.name = 'FakeHeteroDataset'

    assert summary1 == summary2
    assert str(summary1) == str(summary2)


@withPackage('tabulate')
def test_dataset_summary_hetero_representation_length():
    dataset = FakeHeteroDataset(num_graphs=10)
    summary = Summary.from_dataset(dataset)
    num_lines = len(str(summary).splitlines())

    stats_len = len(Stats.__dataclass_fields__)
    len_header_and_border = 5
    num_tables = 3  # general, stats per node type, stats per edge type

    assert num_lines == num_tables * (stats_len + len_header_and_border)


def test_dataset_summary_hetero_per_type_check():
    dataset = FakeHeteroDataset(num_graphs=10)
    exp_num_nodes = torch.Tensor([data.num_nodes for data in dataset])
    exp_num_edges = torch.Tensor([data.num_edges for data in dataset])

    summary = dataset.get_summary()

    assert summary.name == 'FakeHeteroDataset'
    assert summary.num_graphs == 10

    check_stats(summary.num_nodes, exp_num_nodes)
    check_stats(summary.num_edges, exp_num_edges)

    num_nodes_per_type = {}
    for node_type in dataset.node_types:
        num_nodes_per_type[node_type] = torch.Tensor(
            [data[node_type].num_nodes for data in dataset])

    assert len(summary.num_nodes_per_type) == len(dataset.node_types)
    for node_type, stats in summary.num_nodes_per_type.items():
        check_stats(stats, num_nodes_per_type[node_type])

    num_edges_per_type = {}
    for edge_type in dataset.edge_types:
        num_edges_per_type[edge_type] = torch.Tensor(
            [data[edge_type].num_edges for data in dataset])

    assert len(summary.num_edges_per_type) == len(dataset.edge_types)
    for edge_type, stats in summary.num_edges_per_type.items():
        check_stats(stats, num_edges_per_type[edge_type])
pytorch/test/data/test_feature_store.py000066400000000000000000000074471444262304300207340ustar00rootroot00000000000000from dataclasses import dataclass

import pytest
import torch

from torch_geometric.data import TensorAttr
from torch_geometric.data.feature_store import AttrView, _field_status
from torch_geometric.testing import MyFeatureStore


@dataclass
class MyTensorAttrNoGroupName(TensorAttr):
    def __init__(self, attr_name=_field_status.UNSET,
                 index=_field_status.UNSET):
        # Treat group_name as optional, and move it to the end
        super().__init__(None, attr_name, index)


class MyFeatureStoreNoGroupName(MyFeatureStore):
    def __init__(self):
        super().__init__()
        self._tensor_attr_cls = MyTensorAttrNoGroupName


def test_feature_store():
    store = MyFeatureStore()
    tensor = torch.Tensor([[0, 0, 0], [1, 1, 1], [2, 2, 2]])

    group_name = 'A'
    attr_name = 'feat'
    index = torch.tensor([0, 1, 2])
    attr = TensorAttr(group_name, attr_name, index)
    assert TensorAttr(group_name).update(attr) == attr

    # Normal API:
    store.put_tensor(tensor, attr)
    assert torch.equal(store.get_tensor(attr), tensor)
    assert torch.equal(
        store.get_tensor(group_name, attr_name, index=torch.tensor([0, 2])),
        tensor[torch.tensor([0, 2])],
    )

    assert store.update_tensor(tensor + 1, attr)
    assert torch.equal(store.get_tensor(attr), tensor + 1)

    store.remove_tensor(attr)
    with pytest.raises(KeyError):
        _ = store.get_tensor(attr)

    # Views:
    view = store.view(group_name=group_name)
    view.attr_name = attr_name
    view['index'] = index
    assert view != "not a 'AttrView' object"
    assert view == AttrView(store, TensorAttr(group_name, attr_name, index))
    assert str(view) == ("AttrView(store=MyFeatureStore(), "
                         "attr=TensorAttr(group_name='A', attr_name='feat', "
                         "index=tensor([0, 1, 2])))")

    # Indexing:
    store[group_name, attr_name, index] = tensor

    # Fully-specified forms, all of which produce a tensor output
    assert torch.equal(store[group_name, attr_name, index], tensor)
    assert torch.equal(store[group_name, attr_name, None], tensor)
    assert torch.equal(store[group_name, attr_name, :], tensor)
    assert torch.equal(store[group_name][attr_name][:], tensor)
    assert torch.equal(store[group_name].feat[:], tensor)
    assert torch.equal(store.view().A.feat[:], tensor)

    with pytest.raises(AttributeError) as exc_info:
        _ = store.view(group_name=group_name, index=None).feat.A
        print(exc_info)

    # Partially-specified forms, which produce an AttrView object
    assert store[group_name] == store.view(TensorAttr(group_name=group_name))
    assert store[group_name].feat == store.view(
        TensorAttr(group_name=group_name, attr_name=attr_name))

    # Partially-specified forms, when called, produce a Tensor output
    # from the `TensorAttr` that has been partially specified.
    store[group_name] = tensor
    assert isinstance(store[group_name], AttrView)
    assert torch.equal(store[group_name](), tensor)

    # Deletion:
    del store[group_name, attr_name, index]
    with pytest.raises(KeyError):
        _ = store[group_name, attr_name, index]
    del store[group_name]
    with pytest.raises(KeyError):
        _ = store[group_name]()


def test_feature_store_override():
    store = MyFeatureStoreNoGroupName()
    tensor = torch.Tensor([[0, 0, 0], [1, 1, 1], [2, 2, 2]])

    attr_name = 'feat'
    index = torch.tensor([0, 1, 2])

    # Only use attr_name and index, in that order:
    store[attr_name, index] = tensor

    # A few assertions to ensure group_name is not needed:
    assert isinstance(store[attr_name], AttrView)
    assert torch.equal(store[attr_name, index], tensor)
    assert torch.equal(store[attr_name][index], tensor)
    assert torch.equal(store[attr_name][:], tensor)
    assert torch.equal(store[attr_name, :], tensor)
pytorch/test/data/test_graph_store.py000066400000000000000000000073401444262304300203720ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data.graph_store import EdgeAttr, EdgeLayout
from torch_geometric.testing import MyGraphStore, get_random_edge_index
from torch_geometric.utils import (
    to_torch_coo_tensor,
    to_torch_csc_tensor,
    to_torch_csr_tensor,
)


def test_graph_store():
    graph_store = MyGraphStore()

    assert str(graph_store) == 'MyGraphStore()'

    coo = torch.tensor([0, 1]), torch.tensor([1, 2])
    csr = torch.tensor([0, 1, 2]), torch.tensor([1, 2])
    csc = torch.tensor([0, 1]), torch.tensor([0, 0, 1, 2])

    graph_store['edge_type', 'coo'] = coo
    graph_store['edge_type', 'csr'] = csr
    graph_store['edge_type', 'csc'] = csc

    assert torch.equal(graph_store['edge_type', 'coo'][0], coo[0])
    assert torch.equal(graph_store['edge_type', 'coo'][1], coo[1])
    assert torch.equal(graph_store['edge_type', 'csr'][0], csr[0])
    assert torch.equal(graph_store['edge_type', 'csr'][1], csr[1])
    assert torch.equal(graph_store['edge_type', 'csc'][0], csc[0])
    assert torch.equal(graph_store['edge_type', 'csc'][1], csc[1])

    assert len(graph_store.get_all_edge_attrs()) == 3

    del graph_store['edge_type', 'coo']
    with pytest.raises(KeyError):
        graph_store['edge_type', 'coo']

    with pytest.raises(KeyError):
        graph_store['edge_type_2', 'coo']


def test_graph_store_conversion():
    graph_store = MyGraphStore()

    edge_index = get_random_edge_index(100, 100, 300)
    adj = to_torch_coo_tensor(edge_index, size=(100, 100))
    coo = (adj.indices()[0], adj.indices()[1])
    adj = to_torch_csr_tensor(edge_index, size=(100, 100))
    csr = (adj.crow_indices(), adj.col_indices())
    adj = to_torch_csc_tensor(edge_index, size=(100, 100))
    csc = (adj.row_indices(), adj.ccol_indices())

    graph_store.put_edge_index(coo, ('v', '1', 'v'), 'coo', size=(100, 100))
    graph_store.put_edge_index(csr, ('v', '2', 'v'), 'csr', size=(100, 100))
    graph_store.put_edge_index(csc, ('v', '3', 'v'), 'csc', size=(100, 100))

    # Convert to COO:
    row_dict, col_dict, perm_dict = graph_store.coo()
    assert len(row_dict) == len(col_dict) == len(perm_dict) == 3
    for row, col, perm in zip(row_dict.values(), col_dict.values(),
                              perm_dict.values()):
        assert torch.equal(row.sort()[0], coo[0].sort()[0])
        assert torch.equal(col.sort()[0], coo[1].sort()[0])
        assert perm is None

    # Convert to CSR:
    row_dict, col_dict, perm_dict = graph_store.csr()
    assert len(row_dict) == len(col_dict) == len(perm_dict) == 3
    for row, col in zip(row_dict.values(), col_dict.values()):
        assert torch.equal(row, csr[0])
        assert torch.equal(col.sort()[0], csr[1].sort()[0])

    # Convert to CSC:
    row_dict, col_dict, perm_dict = graph_store.csc()
    assert len(row_dict) == len(col_dict) == len(perm_dict) == 3
    for row, col in zip(row_dict.values(), col_dict.values()):
        assert torch.equal(row.sort()[0], csc[0].sort()[0])
        assert torch.equal(col, csc[1])

    # Ensure that 'edge_types' parameters work as intended:
    out = graph_store.coo([('v', '1', 'v')])
    assert torch.equal(list(out[0].values())[0], coo[0])
    assert torch.equal(list(out[1].values())[0], coo[1])

    # Ensure that 'store' parameter works as intended:
    key = EdgeAttr(edge_type=('v', '1', 'v'), layout=EdgeLayout.CSR,
                   is_sorted=False, size=(100, 100))
    with pytest.raises(KeyError):
        graph_store[key]

    out = graph_store.csr([('v', '1', 'v')], store=True)
    assert torch.equal(list(out[0].values())[0], csr[0])
    assert torch.equal(list(out[1].values())[0].sort()[0], csr[1].sort()[0])

    out = graph_store[key]
    assert torch.equal(out[0], csr[0])
    assert torch.equal(out[1].sort()[0], csr[1].sort()[0])
pytorch/test/data/test_hetero_data.py000066400000000000000000000547331444262304300203440ustar00rootroot00000000000000import copy

import pytest
import torch

from torch_geometric.data import HeteroData
from torch_geometric.data.storage import EdgeStorage
from torch_geometric.testing import get_random_edge_index, withPackage

x_paper = torch.randn(10, 16)
x_author = torch.randn(5, 32)
x_conference = torch.randn(5, 8)

idx_paper = torch.randint(x_paper.size(0), (100, ), dtype=torch.long)
idx_author = torch.randint(x_author.size(0), (100, ), dtype=torch.long)
idx_conference = torch.randint(x_conference.size(0), (100, ), dtype=torch.long)

edge_index_paper_paper = torch.stack([idx_paper[:50], idx_paper[:50]], dim=0)
edge_index_paper_author = torch.stack([idx_paper[:30], idx_author[:30]], dim=0)
edge_index_author_paper = torch.stack([idx_author[:30], idx_paper[:30]], dim=0)
edge_index_paper_conference = torch.stack(
    [idx_paper[:25], idx_conference[:25]], dim=0)

edge_attr_paper_paper = torch.randn(edge_index_paper_paper.size(1), 8)
edge_attr_author_paper = torch.randn(edge_index_author_paper.size(1), 8)


def test_init_hetero_data():
    data = HeteroData()
    data['v1'].x = 1
    data['paper'].x = x_paper
    data['author'].x = x_author
    data['paper', 'paper'].edge_index = edge_index_paper_paper
    data['paper', 'author'].edge_index = edge_index_paper_author
    data['author', 'paper'].edge_index = edge_index_author_paper
    data.validate(raise_on_error=True)

    assert len(data) == 2
    assert data.node_types == ['v1', 'paper', 'author']
    assert len(data.node_stores) == 3
    assert len(data.node_items()) == 3
    assert len(data.edge_types) == 3
    assert len(data.edge_stores) == 3
    assert len(data.edge_items()) == 3

    data = HeteroData(
        v1={'x': 1},
        paper={'x': x_paper},
        author={'x': x_author},
        paper__paper={'edge_index': edge_index_paper_paper},
        paper__author={'edge_index': edge_index_paper_author},
        author__paper={'edge_index': edge_index_author_paper},
    )

    assert len(data) == 2
    assert data.node_types == ['v1', 'paper', 'author']
    assert len(data.node_stores) == 3
    assert len(data.node_items()) == 3
    assert len(data.edge_types) == 3
    assert len(data.edge_stores) == 3
    assert len(data.edge_items()) == 3

    data = HeteroData({
        'v1': {
            'x': 1
        },
        'paper': {
            'x': x_paper
        },
        'author': {
            'x': x_author
        },
        ('paper', 'paper'): {
            'edge_index': edge_index_paper_paper
        },
        ('paper', 'author'): {
            'edge_index': edge_index_paper_author
        },
        ('author', 'paper'): {
            'edge_index': edge_index_author_paper
        },
    })

    assert len(data) == 2
    assert data.node_types == ['v1', 'paper', 'author']
    assert len(data.node_stores) == 3
    assert len(data.node_items()) == 3
    assert len(data.edge_types) == 3
    assert len(data.edge_stores) == 3
    assert len(data.edge_items()) == 3


def test_hetero_data_to_from_dict():
    data = HeteroData()
    data.global_id = '1'
    data['v1'].x = torch.randn(5, 16)
    data['v2'].y = torch.randn(4, 16)
    data['v1', 'v2'].edge_index = torch.tensor([[0, 1, 2, 3], [0, 1, 2, 3]])

    out = HeteroData.from_dict(data.to_dict())
    assert out.global_id == data.global_id
    assert torch.equal(out['v1'].x, data['v1'].x)
    assert torch.equal(out['v2'].y, data['v2'].y)
    assert torch.equal(out['v1', 'v2'].edge_index, data['v1', 'v2'].edge_index)


def test_hetero_data_functions():
    data = HeteroData()
    data['paper'].x = x_paper
    data['author'].x = x_author
    data['paper', 'paper'].edge_index = edge_index_paper_paper
    data['paper', 'author'].edge_index = edge_index_paper_author
    data['author', 'paper'].edge_index = edge_index_author_paper
    data['paper', 'paper'].edge_attr = edge_attr_paper_paper
    assert len(data) == 3
    assert sorted(data.keys) == ['edge_attr', 'edge_index', 'x']
    assert 'x' in data and 'edge_index' in data and 'edge_attr' in data
    assert data.num_nodes == 15
    assert data.num_edges == 110

    assert data.node_attrs() == ['x']
    assert sorted(data.edge_attrs()) == ['edge_attr', 'edge_index']

    assert data.num_node_features == {'paper': 16, 'author': 32}
    assert data.num_edge_features == {
        ('paper', 'to', 'paper'): 8,
        ('paper', 'to', 'author'): 0,
        ('author', 'to', 'paper'): 0,
    }

    node_types, edge_types = data.metadata()
    assert node_types == ['paper', 'author']
    assert edge_types == [
        ('paper', 'to', 'paper'),
        ('paper', 'to', 'author'),
        ('author', 'to', 'paper'),
    ]

    x_dict = data.collect('x')
    assert len(x_dict) == 2
    assert x_dict['paper'].tolist() == x_paper.tolist()
    assert x_dict['author'].tolist() == x_author.tolist()
    assert x_dict == data.x_dict

    data.y = 0
    assert data['y'] == 0 and data.y == 0
    assert len(data) == 4
    assert sorted(data.keys) == ['edge_attr', 'edge_index', 'x', 'y']

    del data['paper', 'author']
    node_types, edge_types = data.metadata()
    assert node_types == ['paper', 'author']
    assert edge_types == [('paper', 'to', 'paper'), ('author', 'to', 'paper')]

    assert len(data.to_dict()) == 5
    assert len(data.to_namedtuple()) == 5
    assert data.to_namedtuple().y == 0
    assert len(data.to_namedtuple().paper) == 1


def test_hetero_data_set_value_dict():
    data = HeteroData()
    data.set_value_dict('x', {
        'paper': torch.randn(4, 16),
        'author': torch.randn(8, 32),
    })
    assert data.node_types == ['paper', 'author']
    assert data.edge_types == []
    assert data['paper'].x.size() == (4, 16)
    assert data['author'].x.size() == (8, 32)


def test_hetero_data_rename():
    data = HeteroData()
    data['paper'].x = x_paper
    data['author'].x = x_author
    data['paper', 'paper'].edge_index = edge_index_paper_paper
    data['paper', 'author'].edge_index = edge_index_paper_author
    data['author', 'paper'].edge_index = edge_index_author_paper

    data = data.rename('paper', 'article')
    assert data.node_types == ['author', 'article']
    assert data.edge_types == [
        ('article', 'to', 'article'),
        ('article', 'to', 'author'),
        ('author', 'to', 'article'),
    ]

    assert data['article'].x.tolist() == x_paper.tolist()
    edge_index = data['article', 'article'].edge_index
    assert edge_index.tolist() == edge_index_paper_paper.tolist()


def test_hetero_data_subgraph():
    data = HeteroData()
    data.num_node_types = 3
    data['paper'].x = x_paper
    data['paper'].name = 'paper'
    data['paper'].num_nodes = x_paper.size(0)
    data['author'].x = x_author
    data['author'].num_nodes = x_author.size(0)
    data['conf'].x = x_conference
    data['conf'].num_nodes = x_conference.size(0)
    data['paper', 'paper'].edge_index = edge_index_paper_paper
    data['paper', 'paper'].edge_attr = edge_attr_paper_paper
    data['paper', 'paper'].name = 'cites'
    data['author', 'paper'].edge_index = edge_index_author_paper
    data['paper', 'author'].edge_index = edge_index_paper_author
    data['paper', 'conf'].edge_index = edge_index_paper_conference

    subset = {
        'paper': torch.randperm(x_paper.size(0))[:4],
        'author': torch.randperm(x_author.size(0))[:2],
        'conf': torch.randperm(x_conference.size(0))[:2],
    }

    out = data.subgraph(subset)
    out.validate(raise_on_error=True)

    assert out.num_node_types == data.num_node_types
    assert out.node_types == ['paper', 'author', 'conf']

    for key in out.node_types:
        assert len(out[key]) == len(data[key])
        assert torch.allclose(out[key].x, data[key].x[subset[key]])
        assert out[key].num_nodes == subset[key].size(0)
        if key == 'paper':
            assert out['paper'].name == 'paper'

    # Construct correct edge index manually:
    node_mask = {}  # for each node type a mask of nodes in the subgraph
    node_map = {}  # for each node type a map from old node id to new node id
    for key in out.node_types:
        node_mask[key] = torch.zeros((data[key].num_nodes, ), dtype=torch.bool)
        node_map[key] = torch.zeros((data[key].num_nodes, ), dtype=torch.long)
        node_mask[key][subset[key]] = True
        node_map[key][subset[key]] = torch.arange(subset[key].size(0))

    edge_mask = {}  # for each edge type a mask of edges in the subgraph
    subgraph_edge_index = {
    }  # for each edge type the edge index of the subgraph
    for key in out.edge_types:
        edge_mask[key] = (node_mask[key[0]][data[key].edge_index[0]]
                          & node_mask[key[-1]][data[key].edge_index[1]])
        subgraph_edge_index[key] = data[key].edge_index[:, edge_mask[key]]
        subgraph_edge_index[key][0] = node_map[key[0]][subgraph_edge_index[key]
                                                       [0]]
        subgraph_edge_index[key][1] = node_map[key[-1]][
            subgraph_edge_index[key][1]]

    assert out.edge_types == [
        ('paper', 'to', 'paper'),
        ('author', 'to', 'paper'),
        ('paper', 'to', 'author'),
        ('paper', 'to', 'conf'),
    ]

    for key in out.edge_types:
        assert len(out[key]) == len(data[key])
        assert torch.equal(out[key].edge_index, subgraph_edge_index[key])
        if key == ('paper', 'to', 'paper'):
            assert torch.allclose(out[key].edge_attr,
                                  data[key].edge_attr[edge_mask[key]])
            assert out[key].name == 'cites'

    # Test for bool and long in `subset_dict`.
    author_mask = torch.zeros((x_author.size(0), ), dtype=torch.bool)
    author_mask[subset['author']] = True
    subset_mixed = {
        'paper': subset['paper'],
        'author': author_mask,
    }
    out = data.subgraph(subset_mixed)
    out.validate(raise_on_error=True)

    assert out.num_node_types == data.num_node_types
    assert out.node_types == ['paper', 'author', 'conf']
    assert out['paper'].num_nodes == subset['paper'].size(0)
    assert out['author'].num_nodes == subset['author'].size(0)
    assert out['conf'].num_nodes == data['conf'].num_nodes
    assert out.edge_types == [
        ('paper', 'to', 'paper'),
        ('author', 'to', 'paper'),
        ('paper', 'to', 'author'),
        ('paper', 'to', 'conf'),
    ]

    out = data.node_type_subgraph(['paper', 'author'])
    assert out.node_types == ['paper', 'author']
    assert out.edge_types == [('paper', 'to', 'paper'),
                              ('author', 'to', 'paper'),
                              ('paper', 'to', 'author')]

    out = data.edge_type_subgraph([('paper', 'author')])
    assert out.node_types == ['paper', 'author']
    assert out.edge_types == [('paper', 'to', 'author')]

    subset = {
        ('paper', 'to', 'paper'): torch.arange(4),
    }

    out = data.edge_subgraph(subset)
    assert out.node_types == data.node_types
    assert out.edge_types == data.edge_types
    assert data['paper'] == out['paper']
    assert data['author'] == out['author']
    assert data['paper', 'author'] == out['paper', 'author']
    assert data['author', 'paper'] == out['author', 'paper']

    assert out['paper', 'paper'].num_edges == 4
    assert out['paper', 'paper'].edge_index.size() == (2, 4)
    assert out['paper', 'paper'].edge_attr.size() == (4, 8)


def test_copy_hetero_data():
    data = HeteroData()
    data['paper'].x = x_paper
    data['paper', 'to', 'paper'].edge_index = edge_index_paper_paper

    out = copy.copy(data)
    assert id(data) != id(out)
    assert len(data.stores) == len(out.stores)
    for store1, store2 in zip(data.stores, out.stores):
        assert id(store1) != id(store2)
        assert id(data) == id(store1._parent())
        assert id(out) == id(store2._parent())
    assert out['paper']._key == 'paper'
    assert data['paper'].x.data_ptr() == out['paper'].x.data_ptr()
    assert out['to']._key == ('paper', 'to', 'paper')
    assert data['to'].edge_index.data_ptr() == out['to'].edge_index.data_ptr()

    out = copy.deepcopy(data)
    assert id(data) != id(out)
    assert len(data.stores) == len(out.stores)
    for store1, store2 in zip(data.stores, out.stores):
        assert id(store1) != id(store2)
    assert id(out) == id(out['paper']._parent())
    assert out['paper']._key == 'paper'
    assert data['paper'].x.data_ptr() != out['paper'].x.data_ptr()
    assert data['paper'].x.tolist() == out['paper'].x.tolist()
    assert id(out) == id(out['to']._parent())
    assert out['to']._key == ('paper', 'to', 'paper')
    assert data['to'].edge_index.data_ptr() != out['to'].edge_index.data_ptr()
    assert data['to'].edge_index.tolist() == out['to'].edge_index.tolist()


def test_to_homogeneous_and_vice_versa():
    data = HeteroData()

    data['paper'].x = torch.randn(100, 128)
    data['paper'].y = torch.randint(0, 10, (100, ))
    data['author'].x = torch.randn(200, 128)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 250)
    data['paper', 'paper'].edge_weight = torch.randn(250, )
    data['paper', 'paper'].edge_attr = torch.randn(250, 64)

    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 500)
    data['paper', 'author'].edge_weight = torch.randn(500, )
    data['paper', 'author'].edge_attr = torch.randn(500, 64)

    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)
    data['author', 'paper'].edge_weight = torch.randn(1000, )
    data['author', 'paper'].edge_attr = torch.randn(1000, 64)

    out = data.to_homogeneous()
    assert len(out) == 7
    assert out.num_nodes == 300
    assert out.num_edges == 1750
    assert out.num_node_features == 128
    assert out.num_edge_features == 64
    assert out.node_type.size() == (300, )
    assert out.node_type.min() == 0
    assert out.node_type.max() == 1
    assert out.edge_type.size() == (1750, )
    assert out.edge_type.min() == 0
    assert out.edge_type.max() == 2
    assert len(out._node_type_names) == 2
    assert len(out._edge_type_names) == 3
    assert out.y.size() == (300, )
    assert torch.allclose(out.y[:100], data['paper'].y)
    assert torch.all(out.y[100:] == -1)
    assert 'y' not in data['author']

    out = out.to_heterogeneous()
    assert len(out) == 5
    assert torch.allclose(data['paper'].x, out['paper'].x)
    assert torch.allclose(data['author'].x, out['author'].x)
    assert torch.allclose(data['paper'].y, out['paper'].y)

    edge_index1 = data['paper', 'paper'].edge_index
    edge_index2 = out['paper', 'paper'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['paper', 'paper'].edge_weight,
        out['paper', 'paper'].edge_weight,
    )
    assert torch.allclose(
        data['paper', 'paper'].edge_attr,
        out['paper', 'paper'].edge_attr,
    )

    edge_index1 = data['paper', 'author'].edge_index
    edge_index2 = out['paper', 'author'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['paper', 'author'].edge_weight,
        out['paper', 'author'].edge_weight,
    )
    assert torch.allclose(
        data['paper', 'author'].edge_attr,
        out['paper', 'author'].edge_attr,
    )

    edge_index1 = data['author', 'paper'].edge_index
    edge_index2 = out['author', 'paper'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['author', 'paper'].edge_weight,
        out['author', 'paper'].edge_weight,
    )
    assert torch.allclose(
        data['author', 'paper'].edge_attr,
        out['author', 'paper'].edge_attr,
    )

    out = data.to_homogeneous()
    node_type = out.node_type
    edge_type = out.edge_type
    del out.node_type
    del out.edge_type
    del out._edge_type_names
    del out._node_type_names
    out = out.to_heterogeneous(node_type, edge_type)
    assert len(out) == 5
    assert torch.allclose(data['paper'].x, out['0'].x)
    assert torch.allclose(data['author'].x, out['1'].x)

    edge_index1 = data['paper', 'paper'].edge_index
    edge_index2 = out['0', '0'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['paper', 'paper'].edge_weight,
        out['0', '0'].edge_weight,
    )
    assert torch.allclose(
        data['paper', 'paper'].edge_attr,
        out['0', '0'].edge_attr,
    )

    edge_index1 = data['paper', 'author'].edge_index
    edge_index2 = out['0', '1'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['paper', 'author'].edge_weight,
        out['0', '1'].edge_weight,
    )
    assert torch.allclose(
        data['paper', 'author'].edge_attr,
        out['0', '1'].edge_attr,
    )

    edge_index1 = data['author', 'paper'].edge_index
    edge_index2 = out['1', '0'].edge_index
    assert edge_index1.tolist() == edge_index2.tolist()
    assert torch.allclose(
        data['author', 'paper'].edge_weight,
        out['1', '0'].edge_weight,
    )
    assert torch.allclose(
        data['author', 'paper'].edge_attr,
        out['1', '0'].edge_attr,
    )

    data = HeteroData()

    data['paper'].num_nodes = 100
    data['author'].num_nodes = 200

    out = data.to_homogeneous(add_node_type=False)
    assert len(out) == 1
    assert out.num_nodes == 300

    out = data.to_homogeneous().to_heterogeneous()
    assert len(out) == 1
    assert out['paper'].num_nodes == 100
    assert out['author'].num_nodes == 200


def test_to_homogeneous_padding():
    data = HeteroData()
    data['paper'].x = torch.randn(100, 128)
    data['author'].x = torch.randn(50, 64)

    out = data.to_homogeneous()
    assert len(out) == 2
    assert out.node_type.size() == (150, )
    assert out.node_type[:100].abs().sum() == 0
    assert out.node_type[100:].sub(1).abs().sum() == 0
    assert out.x.size() == (150, 128)
    assert torch.equal(out.x[:100], data['paper'].x)
    assert torch.equal(out.x[100:, :64], data['author'].x)
    assert out.x[100:, 64:].abs().sum() == 0


def test_hetero_data_to_canonical():
    data = HeteroData()
    assert isinstance(data['user', 'product'], EdgeStorage)
    assert len(data.edge_types) == 1
    assert isinstance(data['user', 'to', 'product'], EdgeStorage)
    assert len(data.edge_types) == 1

    data = HeteroData()
    assert isinstance(data['user', 'buys', 'product'], EdgeStorage)
    assert isinstance(data['user', 'clicks', 'product'], EdgeStorage)
    assert len(data.edge_types) == 2

    with pytest.raises(TypeError, match="missing 1 required"):
        data['user', 'product']


def test_hetero_data_invalid_names():
    data = HeteroData()
    with pytest.warns(UserWarning, match="single underscores"):
        data['my test', 'a__b', 'my test'].edge_attr = torch.randn(10, 16)
    assert data.edge_types == [('my test', 'a__b', 'my test')]


def test_hetero_data_update():
    data = HeteroData()
    data['paper'].x = torch.arange(0, 5)
    data['paper'].y = torch.arange(5, 10)
    data['author'].x = torch.arange(10, 15)

    other = HeteroData()
    other['paper'].x = torch.arange(15, 20)
    other['author'].y = torch.arange(20, 25)
    other['paper', 'paper'].edge_index = torch.randint(5, (2, 20))

    data.update(other)
    assert len(data) == 3
    assert torch.equal(data['paper'].x, torch.arange(15, 20))
    assert torch.equal(data['paper'].y, torch.arange(5, 10))
    assert torch.equal(data['author'].x, torch.arange(10, 15))
    assert torch.equal(data['author'].y, torch.arange(20, 25))
    assert torch.equal(data['paper', 'paper'].edge_index,
                       other['paper', 'paper'].edge_index)


# Feature Store ###############################################################


def test_basic_feature_store():
    data = HeteroData()
    x = torch.randn(20, 20)

    # Put tensor:
    assert data.put_tensor(copy.deepcopy(x), group_name='paper', attr_name='x',
                           index=None)
    assert torch.equal(data['paper'].x, x)

    # Put (modify) tensor slice:
    x[15:] = 0
    data.put_tensor(0, group_name='paper', attr_name='x',
                    index=slice(15, None, None))

    # Get tensor:
    out = data.get_tensor(group_name='paper', attr_name='x', index=None)
    assert torch.equal(x, out)

    # Get tensor size:
    assert data.get_tensor_size(group_name='paper', attr_name='x') == (20, 20)

    # Get tensor attrs:
    data['paper'].num_nodes = 20  # don't include, not a tensor attr
    data['paper'].bad_attr = torch.randn(10, 20)  # don't include, bad cat_dim

    tensor_attrs = data.get_all_tensor_attrs()
    assert len(tensor_attrs) == 1
    assert tensor_attrs[0].group_name == 'paper'
    assert tensor_attrs[0].attr_name == 'x'

    # Remove tensor:
    assert 'x' in data['paper'].__dict__['_mapping']
    data.remove_tensor(group_name='paper', attr_name='x', index=None)
    assert 'x' not in data['paper'].__dict__['_mapping']


# Graph Store #################################################################


@withPackage('torch_sparse')
def test_basic_graph_store():
    data = HeteroData()

    def assert_equal_tensor_tuple(expected, actual):
        assert len(expected) == len(actual)
        for i in range(len(expected)):
            assert torch.equal(expected[i], actual[i])

    # We put all three tensor types: COO, CSR, and CSC, and we get them back
    # to confirm that `GraphStore` works as intended.
    coo = (torch.tensor([0, 1]), torch.tensor([1, 2]))
    csr = (torch.tensor([0, 1, 2, 2]), torch.tensor([1, 2]))
    csc = (torch.tensor([0, 1]), torch.tensor([0, 0, 1, 2]))

    # Put:
    data.put_edge_index(coo, layout='coo', edge_type=('a', 'to', 'b'),
                        size=(3, 3))
    data.put_edge_index(csr, layout='csr', edge_type=('a', 'to', 'c'),
                        size=(3, 3))
    data.put_edge_index(csc, layout='csc', edge_type=('b', 'to', 'c'),
                        size=(3, 3))

    # Get:
    assert_equal_tensor_tuple(
        coo, data.get_edge_index(layout='coo', edge_type=('a', 'to', 'b')))
    assert_equal_tensor_tuple(
        csr, data.get_edge_index(layout='csr', edge_type=('a', 'to', 'c')))
    assert_equal_tensor_tuple(
        csc, data.get_edge_index(layout='csc', edge_type=('b', 'to', 'c')))

    # Get attrs:
    edge_attrs = data.get_all_edge_attrs()
    assert len(edge_attrs) == 3


def test_data_generate_ids():
    data = HeteroData()

    data['paper'].x = torch.randn(100, 128)
    data['author'].x = torch.randn(200, 128)

    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 300)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 400)
    assert len(data) == 2

    data.generate_ids()
    assert len(data) == 4
    assert data['paper'].n_id.tolist() == list(range(100))
    assert data['author'].n_id.tolist() == list(range(200))
    assert data['paper', 'author'].e_id.tolist() == list(range(300))
    assert data['author', 'paper'].e_id.tolist() == list(range(400))
pytorch/test/data/test_inherit.py000066400000000000000000000026371444262304300175230ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, Dataset, InMemoryDataset


class MyData(Data):
    def __init__(self, x=None, edge_index=None, arg=None):
        super().__init__(x=x, edge_index=edge_index, arg=arg)

    def random(self):
        return torch.randn(list(self.x.size()) + list(self.arg.size()))


class MyInMemoryDataset(InMemoryDataset):
    def __init__(self):
        super().__init__('/tmp/MyInMemoryDataset')

        x = torch.randn(4, 5)
        edge_index = torch.tensor([[0, 0, 0], [1, 2, 3]])
        arg = torch.randn(4, 3)

        data_list = [MyData(x, edge_index, arg) for _ in range(10)]
        self.data, self.slices = self.collate(data_list)

    def _download(self):
        pass

    def _process(self):
        pass


class MyDataset(Dataset):
    def __init__(self):
        super().__init__('/tmp/MyDataset')

    def _download(self):
        pass

    def _process(self):
        pass

    def len(self):
        return 10

    def get(self, idx):
        x = torch.randn(4, 5)
        edge_index = torch.tensor([[0, 0, 0], [1, 2, 3]])
        arg = torch.randn(4, 3)
        return MyData(x, edge_index, arg)


def test_inherit():
    dataset = MyDataset()
    assert len(dataset) == 10
    data = dataset[0]
    assert data.random().size() == (4, 5, 4, 3)

    dataset = MyInMemoryDataset()
    assert len(dataset) == 10
    data = dataset[0]
    assert data.random().size() == (4, 5, 4, 3)
pytorch/test/data/test_remote_backend_utils.py000066400000000000000000000023351444262304300222360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import HeteroData
from torch_geometric.data.remote_backend_utils import num_nodes, size
from torch_geometric.testing import (
    MyFeatureStore,
    MyGraphStore,
    get_random_edge_index,
)


@pytest.mark.parametrize('FeatureStore', [MyFeatureStore, HeteroData])
@pytest.mark.parametrize('GraphStore', [MyGraphStore, HeteroData])
def test_num_nodes_size(FeatureStore, GraphStore):
    feature_store = FeatureStore()
    graph_store = GraphStore()

    # Infer num nodes from features:
    x = torch.arange(100)
    feature_store.put_tensor(x, group_name='x', attr_name='x', index=None)
    assert num_nodes(feature_store, graph_store, 'x') == 100

    # Infer num nodes and size from edges:
    xy = get_random_edge_index(100, 50, 20)
    graph_store.put_edge_index(xy, edge_type=('x', 'to', 'y'), layout='coo',
                               size=(100, 50))
    assert num_nodes(feature_store, graph_store, 'y') == 50
    assert size(feature_store, graph_store, ('x', 'to', 'y')) == (100, 50)

    # Throw an error if we cannot infer for an unknown node type:
    with pytest.raises(ValueError, match="Unable to accurately infer"):
        _ = num_nodes(feature_store, graph_store, 'z')
pytorch/test/data/test_storage.py000066400000000000000000000056221444262304300175220ustar00rootroot00000000000000import copy
from typing import Any

import torch

from torch_geometric.data.storage import BaseStorage


def test_base_storage():
    storage = BaseStorage()
    assert storage._mapping == {}
    storage.x = torch.zeros(1)
    storage.y = torch.ones(1)
    assert len(storage) == 2
    assert storage._mapping == {'x': torch.zeros(1), 'y': torch.ones(1)}
    assert storage.x is not None
    assert storage.y is not None

    assert torch.allclose(storage.get('x', None), storage.x)
    assert torch.allclose(storage.get('y', None), storage.y)
    assert storage.get('z', 2) == 2
    assert storage.get('z', None) is None
    assert len(list(storage.keys('x', 'y', 'z'))) == 2
    assert len(list(storage.keys('x', 'y', 'z'))) == 2
    assert len(list(storage.values('x', 'y', 'z'))) == 2
    assert len(list(storage.items('x', 'y', 'z'))) == 2

    del storage.y
    assert len(storage) == 1
    assert storage.x is not None

    storage = BaseStorage({'x': torch.zeros(1)})
    assert len(storage) == 1
    assert storage.x is not None

    storage = BaseStorage(x=torch.zeros(1))
    assert len(storage) == 1
    assert storage.x is not None

    storage = BaseStorage(x=torch.zeros(1))
    copied_storage = copy.copy(storage)
    assert storage == copied_storage
    assert id(storage) != id(copied_storage)
    assert storage.x.data_ptr() == copied_storage.x.data_ptr()
    assert int(storage.x) == 0
    assert int(copied_storage.x) == 0

    deepcopied_storage = copy.deepcopy(storage)
    assert storage == deepcopied_storage
    assert id(storage) != id(deepcopied_storage)
    assert storage.x.data_ptr() != deepcopied_storage.x.data_ptr()
    assert int(storage.x) == 0
    assert int(deepcopied_storage.x) == 0


def test_storage_tensor_methods():
    x = torch.randn(5)
    storage = BaseStorage({'x': x})

    storage = storage.clone()
    assert storage.x.data_ptr() != x.data_ptr()

    storage = storage.contiguous()
    assert storage.x.is_contiguous()

    storage = storage.to('cpu')
    assert storage.x.device == torch.device('cpu')

    storage = storage.cpu()
    assert storage.x.device == torch.device('cpu')

    if torch.cuda.is_available():
        storage = storage.pin_memory()
        assert storage.x.is_pinned()

    storage = storage.share_memory_()
    assert storage.x.is_shared

    storage = storage.detach_()
    assert not storage.x.requires_grad

    storage = storage.detach()
    assert not storage.x.requires_grad

    storage = storage.requires_grad_()
    assert storage.x.requires_grad


def test_setter_and_getter():
    class MyStorage(BaseStorage):
        @property
        def my_property(self) -> Any:
            return self._my_property

        @my_property.setter
        def my_property(self, value: Any):
            self._my_property = value

    storage = MyStorage()
    storage.my_property = 'hello'
    assert storage.my_property == 'hello'
    assert storage._my_property == storage._my_property
pytorch/test/data/test_temporal.py000066400000000000000000000110431444262304300176730ustar00rootroot00000000000000import copy

import torch

from torch_geometric.data import TemporalData


def get_temporal_data(num_events, msg_channels):
    return TemporalData(
        src=torch.arange(num_events),
        dst=torch.arange(num_events, num_events * 2),
        t=torch.arange(0, num_events * 1000, step=1000),
        msg=torch.randn(num_events, msg_channels),
        y=torch.randint(0, 2, (num_events, )),
    )


def test_temporal_data():
    data = get_temporal_data(num_events=3, msg_channels=16)
    assert str(data) == ("TemporalData(src=[3], dst=[3], t=[3], "
                         "msg=[3, 16], y=[3])")

    assert data.num_nodes == 6
    assert data.num_events == data.num_edges == len(data) == 3

    assert data.src.tolist() == [0, 1, 2]
    assert data['src'].tolist() == [0, 1, 2]

    assert data.edge_index.tolist() == [[0, 1, 2], [3, 4, 5]]
    data.edge_index = 'edge_index'
    assert data.edge_index == 'edge_index'
    del data.edge_index
    assert data.edge_index.tolist() == [[0, 1, 2], [3, 4, 5]]

    assert sorted(data.keys) == ['dst', 'msg', 'src', 't', 'y']
    assert sorted(data.to_dict().keys()) == sorted(data.keys)

    data_tuple = data.to_namedtuple()
    assert len(data_tuple) == 5
    assert data_tuple.src is not None
    assert data_tuple.dst is not None
    assert data_tuple.t is not None
    assert data_tuple.msg is not None
    assert data_tuple.y is not None

    assert data.__cat_dim__('src', data.src) == 0
    assert data.__inc__('src', data.src) == 6

    clone = data.clone()
    assert clone != data
    assert len(clone) == len(data)
    assert clone.src.data_ptr() != data.src.data_ptr()
    assert clone.src.tolist() == data.src.tolist()
    assert clone.dst.data_ptr() != data.dst.data_ptr()
    assert clone.dst.tolist() == data.dst.tolist()

    deepcopy = copy.deepcopy(data)
    assert deepcopy != data
    assert len(deepcopy) == len(data)
    assert deepcopy.src.data_ptr() != data.src.data_ptr()
    assert deepcopy.src.tolist() == data.src.tolist()
    assert deepcopy.dst.data_ptr() != data.dst.data_ptr()
    assert deepcopy.dst.tolist() == data.dst.tolist()

    key = value = 'test_value'
    data[key] = value
    assert data[key] == value
    assert data.test_value == value
    del data[key]
    del data[key]  # Deleting unset attributes should work as well.

    assert data.get(key, 10) == 10

    assert len([event for event in data]) == 3

    assert len([attr for attr in data()]) == 5

    assert data.size() == (2, 5)

    del data.src
    assert 'src' not in data


def test_train_val_test_split():
    data = get_temporal_data(num_events=100, msg_channels=16)

    train_data, val_data, test_data = data.train_val_test_split(
        val_ratio=0.2, test_ratio=0.15)

    assert len(train_data) == 65
    assert len(val_data) == 20
    assert len(test_data) == 15

    assert train_data.t.max() < val_data.t.min()
    assert val_data.t.max() < test_data.t.min()


def test_temporal_indexing():
    data = get_temporal_data(num_events=10, msg_channels=16)

    elem = data[0]
    assert isinstance(elem, TemporalData)
    assert len(elem) == 1
    assert elem.src.tolist() == data.src[0:1].tolist()
    assert elem.dst.tolist() == data.dst[0:1].tolist()
    assert elem.t.tolist() == data.t[0:1].tolist()
    assert elem.msg.tolist() == data.msg[0:1].tolist()
    assert elem.y.tolist() == data.y[0:1].tolist()

    subset = data[0:5]
    assert isinstance(subset, TemporalData)
    assert len(subset) == 5
    assert subset.src.tolist() == data.src[0:5].tolist()
    assert subset.dst.tolist() == data.dst[0:5].tolist()
    assert subset.t.tolist() == data.t[0:5].tolist()
    assert subset.msg.tolist() == data.msg[0:5].tolist()
    assert subset.y.tolist() == data.y[0:5].tolist()

    index = [0, 4, 8]
    subset = data[torch.tensor(index)]
    assert isinstance(subset, TemporalData)
    assert len(subset) == 3
    assert subset.src.tolist() == data.src[0::4].tolist()
    assert subset.dst.tolist() == data.dst[0::4].tolist()
    assert subset.t.tolist() == data.t[0::4].tolist()
    assert subset.msg.tolist() == data.msg[0::4].tolist()
    assert subset.y.tolist() == data.y[0::4].tolist()

    mask = [True, False, True, False, True, False, True, False, True, False]
    subset = data[torch.tensor(mask)]
    assert isinstance(subset, TemporalData)
    assert len(subset) == 5
    assert subset.src.tolist() == data.src[0::2].tolist()
    assert subset.dst.tolist() == data.dst[0::2].tolist()
    assert subset.t.tolist() == data.t[0::2].tolist()
    assert subset.msg.tolist() == data.msg[0::2].tolist()
    assert subset.y.tolist() == data.y[0::2].tolist()
pytorch/test/data/test_view.py000066400000000000000000000021671444262304300170310ustar00rootroot00000000000000from torch_geometric.data.storage import BaseStorage


def test_views():
    storage = BaseStorage(x=1, y=2, z=3)

    assert str(storage.keys()) == "KeysView({'x': 1, 'y': 2, 'z': 3})"
    assert len(storage.keys()) == 3
    assert list(storage.keys()) == ['x', 'y', 'z']

    assert str(storage.values()) == "ValuesView({'x': 1, 'y': 2, 'z': 3})"
    assert len(storage.values()) == 3
    assert list(storage.values()) == [1, 2, 3]

    assert str(storage.items()) == "ItemsView({'x': 1, 'y': 2, 'z': 3})"
    assert len(storage.items()) == 3
    assert list(storage.items()) == [('x', 1), ('y', 2), ('z', 3)]

    args = ['x', 'z', 'foo']

    assert str(storage.keys(*args)) == "KeysView({'x': 1, 'z': 3})"
    assert len(storage.keys(*args)) == 2
    assert list(storage.keys(*args)) == ['x', 'z']

    assert str(storage.values(*args)) == "ValuesView({'x': 1, 'z': 3})"
    assert len(storage.values(*args)) == 2
    assert list(storage.values(*args)) == [1, 3]

    assert str(storage.items(*args)) == "ItemsView({'x': 1, 'z': 3})"
    assert len(storage.items(*args)) == 2
    assert list(storage.items(*args)) == [('x', 1), ('z', 3)]
pytorch/test/datasets/000077500000000000000000000000001444262304300153375ustar00rootroot00000000000000pytorch/test/datasets/graph_generator/000077500000000000000000000000001444262304300205065ustar00rootroot00000000000000pytorch/test/datasets/graph_generator/test_ba_graph.py000066400000000000000000000005311444262304300236610ustar00rootroot00000000000000from torch_geometric.datasets.graph_generator import BAGraph


def test_ba_graph():
    graph_generator = BAGraph(num_nodes=300, num_edges=5)
    assert str(graph_generator) == 'BAGraph(num_nodes=300, num_edges=5)'

    data = graph_generator()
    assert len(data) == 2
    assert data.num_nodes == 300
    assert data.num_edges <= 2 * 300 * 5
pytorch/test/datasets/graph_generator/test_er_graph.py000066400000000000000000000006201444262304300237040ustar00rootroot00000000000000from torch_geometric.datasets.graph_generator import ERGraph


def test_er_graph():
    graph_generator = ERGraph(num_nodes=300, edge_prob=0.1)
    assert str(graph_generator) == 'ERGraph(num_nodes=300, edge_prob=0.1)'

    data = graph_generator()
    assert len(data) == 2
    assert data.num_nodes == 300
    assert data.num_edges >= 300 * 300 * 0.05
    assert data.num_edges <= 300 * 300 * 0.15
pytorch/test/datasets/graph_generator/test_grid_graph.py000066400000000000000000000005131444262304300242240ustar00rootroot00000000000000from torch_geometric.datasets.graph_generator import GridGraph


def test_grid_graph():
    graph_generator = GridGraph(height=10, width=10)
    assert str(graph_generator) == 'GridGraph(height=10, width=10)'

    data = graph_generator()
    assert len(data) == 2
    assert data.num_nodes == 100
    assert data.num_edges == 784
pytorch/test/datasets/motif_generator/000077500000000000000000000000001444262304300205235ustar00rootroot00000000000000pytorch/test/datasets/motif_generator/test_custom_motif.py000066400000000000000000000017221444262304300246460ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.datasets.motif_generator import CustomMotif
from torch_geometric.testing import withPackage


def test_custom_motif_pyg_data():
    structure = Data(
        num_nodes=3,
        edge_index=torch.tensor([[0, 1, 2, 1, 2, 0], [1, 2, 0, 0, 1, 2]]),
    )

    motif_generator = CustomMotif(structure)
    assert str(motif_generator) == 'CustomMotif()'

    assert structure == motif_generator()


@withPackage('networkx')
def test_custom_motif_networkx():
    import networkx as nx

    structure = nx.gnm_random_graph(5, 10, seed=2000)

    motif_generator = CustomMotif(structure)
    assert str(motif_generator) == 'CustomMotif()'

    out = motif_generator()
    assert len(out) == 2
    assert out.num_nodes == 5
    assert out.num_edges == 20


def test_custom_motif_unknown():
    with pytest.raises(ValueError, match="motif structure of type"):
        CustomMotif(structure='unknown')
pytorch/test/datasets/motif_generator/test_cycle_motif.py000066400000000000000000000006541444262304300244360ustar00rootroot00000000000000from torch_geometric.datasets.motif_generator import CycleMotif


def test_cycle_motif():
    motif_generator = CycleMotif(5)
    assert str(motif_generator) == 'CycleMotif(5)'

    motif = motif_generator()
    assert len(motif) == 2
    assert motif.num_nodes == 5
    assert motif.num_edges == 10
    assert motif.edge_index.tolist() == [
        [0, 0, 1, 1, 2, 2, 3, 3, 4, 4],
        [1, 4, 0, 2, 1, 3, 2, 4, 0, 3],
    ]
pytorch/test/datasets/motif_generator/test_house_motif.py000066400000000000000000000005371444262304300244620ustar00rootroot00000000000000from torch_geometric.datasets.motif_generator import HouseMotif


def test_house_motif():
    motif_generator = HouseMotif()
    assert str(motif_generator) == 'HouseMotif()'

    motif = motif_generator()
    assert len(motif) == 3
    assert motif.num_nodes == 5
    assert motif.num_edges == 12
    assert motif.y.min() == 0 and motif.y.max() == 2
pytorch/test/datasets/test_ba_shapes.py000066400000000000000000000010371444262304300206760ustar00rootroot00000000000000import pytest


def test_ba_shapes(get_dataset):
    with pytest.warns(UserWarning, match="is deprecated"):
        dataset = get_dataset(name='BAShapes')

    assert str(dataset) == 'BAShapes()'
    assert len(dataset) == 1
    assert dataset.num_features == 10
    assert dataset.num_classes == 4

    data = dataset[0]
    assert len(data) == 5
    assert data.edge_index.size(1) >= 1120
    assert data.x.size() == (700, 10)
    assert data.y.size() == (700, )
    assert data.expl_mask.sum() == 60
    assert data.edge_label.sum() == 960
pytorch/test/datasets/test_bzr.py000066400000000000000000000011651444262304300175500ustar00rootroot00000000000000from torch_geometric.testing import onlyFullTest


@onlyFullTest
def test_bzr(get_dataset):
    dataset = get_dataset(name='BZR')
    assert len(dataset) == 405
    assert dataset.num_features == 53
    assert dataset.num_node_labels == 53
    assert dataset.num_node_attributes == 3
    assert dataset.num_classes == 2
    assert str(dataset) == 'BZR(405)'
    assert len(dataset[0]) == 3


@onlyFullTest
def test_bzr_with_node_attr(get_dataset):
    dataset = get_dataset(name='BZR', use_node_attr=True)
    assert dataset.num_features == 56
    assert dataset.num_node_labels == 53
    assert dataset.num_node_attributes == 3
pytorch/test/datasets/test_elliptic.py000066400000000000000000000020331444262304300205530ustar00rootroot00000000000000from torch_geometric.testing import onlyFullTest


@onlyFullTest
def test_elliptic_bitcoin_dataset(get_dataset):
    dataset = get_dataset(name='EllipticBitcoinDataset')
    assert str(dataset) == 'EllipticBitcoinDataset()'
    assert len(dataset) == 1
    assert dataset.num_features == 165
    assert dataset.num_classes == 2

    data = dataset[0]
    assert len(data) == 5
    assert data.x.size() == (203769, 165)
    assert data.edge_index.size() == (2, 234355)
    assert data.y.size() == (203769, )

    assert data.train_mask.size() == (203769, )
    assert data.train_mask.sum() > 0
    assert data.test_mask.size() == (203769, )
    assert data.test_mask.sum() > 0
    assert data.train_mask.sum() + data.test_mask.sum() == 4545 + 42019
    assert data.y[data.train_mask].sum() == 3462
    assert data.y[data.test_mask].sum() == 1083
    assert data.y[data.train_mask].sum() + data.y[data.test_mask].sum() == 4545
    assert data.y[data.test_mask | data.train_mask].min() == 0
    assert data.y[data.test_mask | data.train_mask].max() == 1
pytorch/test/datasets/test_enzymes.py000066400000000000000000000044311444262304300204440ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.loader import DataListLoader, DataLoader, DenseDataLoader
from torch_geometric.transforms import ToDense


def test_enzymes(get_dataset):
    dataset = get_dataset(name='ENZYMES')
    assert len(dataset) == 600
    assert dataset.num_features == 3
    assert dataset.num_classes == 6
    assert str(dataset) == 'ENZYMES(600)'

    assert len(dataset[0]) == 3
    assert len(dataset.shuffle()) == 600
    assert len(dataset.shuffle(return_perm=True)) == 2
    assert len(dataset[:100]) == 100
    assert len(dataset[torch.arange(100, dtype=torch.long)]) == 100
    mask = torch.zeros(600, dtype=torch.bool)
    mask[:100] = 1
    assert len(dataset[mask]) == 100

    loader = DataLoader(dataset, batch_size=len(dataset))
    for batch in loader:
        assert batch.num_graphs == len(batch) == 600

        avg_num_nodes = batch.num_nodes / batch.num_graphs
        assert pytest.approx(avg_num_nodes, abs=1e-2) == 32.63

        avg_num_edges = batch.num_edges / (2 * batch.num_graphs)
        assert pytest.approx(avg_num_edges, abs=1e-2) == 62.14

        assert list(batch.x.size()) == [batch.num_nodes, 3]
        assert list(batch.y.size()) == [batch.num_graphs]
        assert batch.y.max() + 1 == 6
        assert list(batch.batch.size()) == [batch.num_nodes]
        assert batch.ptr.numel() == batch.num_graphs + 1

        assert batch.has_isolated_nodes()
        assert not batch.has_self_loops()
        assert batch.is_undirected()

    loader = DataListLoader(dataset, batch_size=len(dataset))
    for data_list in loader:
        assert len(data_list) == 600

    dataset.transform = ToDense(num_nodes=126)
    loader = DenseDataLoader(dataset, batch_size=len(dataset))
    for data in loader:
        assert list(data.x.size()) == [600, 126, 3]
        assert list(data.adj.size()) == [600, 126, 126]
        assert list(data.mask.size()) == [600, 126]
        assert list(data.y.size()) == [600, 1]


def test_enzymes_with_node_attr(get_dataset):
    dataset = get_dataset(name='ENZYMES', use_node_attr=True)
    assert dataset.num_node_features == 21
    assert dataset.num_features == 21
    assert dataset.num_edge_features == 0


def test_cleaned_enzymes(get_dataset):
    dataset = get_dataset(name='ENZYMES', cleaned=True)
    assert len(dataset) == 595
pytorch/test/datasets/test_explainer_dataset.py000066400000000000000000000033711444262304300224500ustar00rootroot00000000000000import pytest
import torch

from torch_geometric import seed_everything
from torch_geometric.datasets import ExplainerDataset
from torch_geometric.datasets.graph_generator import BAGraph
from torch_geometric.datasets.motif_generator import HouseMotif


@pytest.mark.parametrize('graph_generator', [
    BAGraph(num_nodes=80, num_edges=5),
])
@pytest.mark.parametrize('motif_generator', [
    HouseMotif(),
    'house',
])
def test_explainer_dataset_ba_house(graph_generator, motif_generator):
    dataset = ExplainerDataset(graph_generator, motif_generator, num_motifs=2)
    assert str(dataset) == ('ExplainerDataset(1, graph_generator='
                            'BAGraph(num_nodes=80, num_edges=5), '
                            'motif_generator=HouseMotif(), num_motifs=2)')
    assert len(dataset) == 1

    data = dataset[0]
    assert len(data) == 4
    assert data.num_nodes == 80 + (2 * 5)
    assert data.edge_index.min() == 0
    assert data.edge_index.max() == data.num_nodes - 1
    assert data.y.min() == 0 and data.y.max() == 3
    assert data.node_mask.size() == (data.num_nodes, )
    assert data.edge_mask.size() == (data.num_edges, )
    assert data.node_mask.min() == 0 and data.node_mask.max() == 1
    assert data.node_mask.sum() == 2 * 5
    assert data.edge_mask.min() == 0 and data.edge_mask.max() == 1
    assert data.edge_mask.sum() == 2 * 12


def test_explainer_dataset_reproducibility():
    seed_everything(12345)
    data1 = ExplainerDataset(BAGraph(num_nodes=80, num_edges=5), HouseMotif(),
                             num_motifs=2)[0]

    seed_everything(12345)
    data2 = ExplainerDataset(BAGraph(num_nodes=80, num_edges=5), HouseMotif(),
                             num_motifs=2)[0]

    assert torch.equal(data1.edge_index, data2.edge_index)
pytorch/test/datasets/test_fake.py000066400000000000000000000055151444262304300176640ustar00rootroot00000000000000import pytest

from torch_geometric.datasets import FakeDataset, FakeHeteroDataset


@pytest.mark.parametrize('num_graphs', [1, 10])
@pytest.mark.parametrize('edge_dim', [0, 1, 4])
@pytest.mark.parametrize('task', ['node', 'graph', 'auto'])
def test_fake_dataset(num_graphs, edge_dim, task):
    dataset = FakeDataset(num_graphs, edge_dim=edge_dim, task=task,
                          global_features=3)

    if num_graphs > 1:
        assert str(dataset) == f'FakeDataset({num_graphs})'
    else:
        assert str(dataset) == 'FakeDataset()'

    assert len(dataset) == num_graphs

    data = dataset[0]

    assert data.num_features == 64

    if edge_dim == 0:
        assert len(data) == 4
    elif edge_dim == 1:
        assert len(data) == 5
        assert data.edge_weight.size() == (data.num_edges, )
        assert data.edge_weight.min() >= 0 and data.edge_weight.max() < 1
    else:
        assert len(data) == 5
        assert data.edge_attr.size() == (data.num_edges, edge_dim)
        assert data.edge_attr.min() >= 0 and data.edge_attr.max() < 1

    assert data.y.min() >= 0 and data.y.max() < 10
    if task == 'node' or (task == 'auto' and num_graphs == 1):
        assert data.y.size() == (data.num_nodes, )
    else:
        assert data.y.size() == (1, )

    assert data.global_features.size() == (3, )


@pytest.mark.parametrize('num_graphs', [1, 10])
@pytest.mark.parametrize('edge_dim', [0, 1, 4])
@pytest.mark.parametrize('task', ['node', 'graph', 'auto'])
def test_fake_hetero_dataset(num_graphs, edge_dim, task):
    dataset = FakeHeteroDataset(num_graphs, edge_dim=edge_dim, task=task,
                                global_features=3)

    if num_graphs > 1:
        assert str(dataset) == f'FakeHeteroDataset({num_graphs})'
    else:
        assert str(dataset) == 'FakeHeteroDataset()'

    assert len(dataset) == num_graphs

    data = dataset[0]

    for store in data.node_stores:
        assert store.num_features > 0

        if task == 'node' or (task == 'auto' and num_graphs == 1):
            if store._key == 'v0':
                assert store.y.min() >= 0 and store.y.max() < 10
                assert store.y.size() == (store.num_nodes, )

    for store in data.edge_stores:
        if edge_dim == 0:
            assert len(data) == 4
        elif edge_dim == 1:
            assert len(data) == 5
            assert store.edge_weight.size() == (store.num_edges, )
            assert store.edge_weight.min() >= 0 and store.edge_weight.max() < 1
        else:
            assert len(data) == 5
            assert store.edge_attr.size() == (store.num_edges, edge_dim)
            assert store.edge_attr.min() >= 0 and store.edge_attr.max() < 1

    if task == 'graph' or (task == 'auto' and num_graphs > 1):
        assert data.y.min() >= 0 and data.y.max() < 10
        assert data.y.size() == (1, )

    assert data.global_features.size() == (3, )
pytorch/test/datasets/test_imdb_binary.py000066400000000000000000000007131444262304300212300ustar00rootroot00000000000000from torch_geometric.testing import onlyFullTest


@onlyFullTest
def test_imdb_binary(get_dataset):
    dataset = get_dataset(name='IMDB-BINARY')
    assert len(dataset) == 1000
    assert dataset.num_features == 0
    assert dataset.num_classes == 2
    assert str(dataset) == 'IMDB-BINARY(1000)'

    data = dataset[0]
    assert len(data) == 3
    assert data.edge_index.size() == (2, 146)
    assert data.y.size() == (1, )
    assert data.num_nodes == 20
pytorch/test/datasets/test_infection_dataset.py000066400000000000000000000040471444262304300224400ustar00rootroot00000000000000import torch

from torch_geometric import seed_everything
from torch_geometric.data import Data
from torch_geometric.datasets import InfectionDataset
from torch_geometric.datasets.graph_generator import ERGraph, GraphGenerator


class DummyGraph(GraphGenerator):
    def __call__(self) -> Data:
        edge_index = torch.tensor([
            [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9],
            [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6, 8, 7, 9, 8],
        ])
        return Data(num_nodes=10, edge_index=edge_index)


def test_infection_dataset():
    seed_everything(12345)
    graph_generator = DummyGraph()
    dataset = InfectionDataset(graph_generator, num_infected_nodes=2,
                               max_path_length=2)
    assert str(dataset) == ('InfectionDataset(1, '
                            'graph_generator=DummyGraph(), '
                            'num_infected_nodes=2, '
                            'max_path_length=2)')
    assert len(dataset) == 1

    data = dataset[0]
    assert len(data) == 4
    assert data.x.size() == (10, 2)
    assert data.x[:, 0].sum() == 8 and data.x[:, 1].sum() == 2
    assert torch.equal(data.edge_index, graph_generator().edge_index)
    assert data.y.size() == (10, )

    # With `seed=12345`, node 0 and node 7 will be infected:
    assert data.x[0].tolist() == [0, 1]  # First infected node.
    assert data.x[7].tolist() == [0, 1]  # Second infected node.
    assert data.y.tolist() == [0, 1, 2, 3, 3, 2, 1, 0, 1, 2]
    assert data.edge_mask.tolist() == [
        1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0
    ]


def test_infection_dataset_reproducibility():
    graph_generator = ERGraph(num_nodes=500, edge_prob=0.004)

    seed_everything(12345)
    dataset1 = InfectionDataset(graph_generator, num_infected_nodes=50,
                                max_path_length=5)

    seed_everything(12345)
    dataset2 = InfectionDataset(graph_generator, num_infected_nodes=50,
                                max_path_length=5)

    assert torch.equal(dataset1[0].edge_mask, dataset2[0].edge_mask)
pytorch/test/datasets/test_karate.py000066400000000000000000000007521444262304300202230ustar00rootroot00000000000000def test_karate(get_dataset):
    dataset = get_dataset(name='KarateClub')
    assert str(dataset) == 'KarateClub()'
    assert len(dataset) == 1
    assert dataset.num_features == 34
    assert dataset.num_classes == 4

    assert len(dataset[0]) == 4
    assert dataset[0].edge_index.size() == (2, 156)
    assert dataset[0].x.size() == (34, 34)
    assert dataset[0].y.size() == (34, )
    assert dataset[0].train_mask.size() == (34, )
    assert dataset[0].train_mask.sum().item() == 4
pytorch/test/datasets/test_mutag.py000066400000000000000000000006621444262304300200710ustar00rootroot00000000000000def test_mutag(get_dataset):
    dataset = get_dataset(name='MUTAG')
    assert len(dataset) == 188
    assert dataset.num_features == 7
    assert dataset.num_classes == 2
    assert str(dataset) == 'MUTAG(188)'

    assert len(dataset[0]) == 4
    assert dataset[0].edge_attr.size(1) == 4


def test_mutag_with_node_attr(get_dataset):
    dataset = get_dataset(name='MUTAG', use_node_attr=True)
    assert dataset.num_features == 7
pytorch/test/datasets/test_planetoid.py000066400000000000000000000033651444262304300207360ustar00rootroot00000000000000from torch_geometric.loader import DataLoader


def test_citeseer(get_dataset):
    dataset = get_dataset(name='CiteSeer')
    loader = DataLoader(dataset, batch_size=len(dataset))

    assert len(dataset) == 1
    assert str(dataset) == 'CiteSeer()'

    for batch in loader:
        assert batch.num_graphs == len(batch) == 1
        assert batch.num_nodes == 3327
        assert batch.num_edges / 2 == 4552

        assert list(batch.x.size()) == [batch.num_nodes, 3703]
        assert list(batch.y.size()) == [batch.num_nodes]
        assert batch.y.max() + 1 == 6
        assert batch.train_mask.sum() == 6 * 20
        assert batch.val_mask.sum() == 500
        assert batch.test_mask.sum() == 1000
        assert (batch.train_mask & batch.val_mask & batch.test_mask).sum() == 0
        assert list(batch.batch.size()) == [batch.num_nodes]
        assert batch.ptr.tolist() == [0, batch.num_nodes]

        assert batch.has_isolated_nodes()
        assert not batch.has_self_loops()
        assert batch.is_undirected()


def test_citeseer_with_full_split(get_dataset):
    dataset = get_dataset(name='CiteSeer', split='full')
    data = dataset[0]
    assert data.val_mask.sum() == 500
    assert data.test_mask.sum() == 1000
    assert data.train_mask.sum() == data.num_nodes - 1500
    assert (data.train_mask & data.val_mask & data.test_mask).sum() == 0


def test_citeseer_with_random_split(get_dataset):
    dataset = get_dataset(name='CiteSeer', split='random',
                          num_train_per_class=11, num_val=29, num_test=41)
    data = dataset[0]
    assert data.train_mask.sum() == dataset.num_classes * 11
    assert data.val_mask.sum() == 29
    assert data.test_mask.sum() == 41
    assert (data.train_mask & data.val_mask & data.test_mask).sum() == 0
pytorch/test/datasets/test_snap_dataset.py000066400000000000000000000011571444262304300214220ustar00rootroot00000000000000from torch_geometric.testing import onlyFullTest


@onlyFullTest
def test_ego_facebook_snap_dataset(get_dataset):
    dataset = get_dataset(name='ego-facebook')
    assert str(dataset) == 'SNAP-ego-facebook(10)'
    assert len(dataset) == 10


@onlyFullTest
def test_soc_slashdot_snap_dataset(get_dataset):
    dataset = get_dataset(name='soc-Slashdot0811')
    assert str(dataset) == 'SNAP-soc-slashdot0811(1)'
    assert len(dataset) == 1


@onlyFullTest
def test_wiki_vote_snap_dataset(get_dataset):
    dataset = get_dataset(name='wiki-vote')
    assert str(dataset) == 'SNAP-wiki-vote(1)'
    assert len(dataset) == 1
pytorch/test/datasets/test_suite_sparse.py000066400000000000000000000011341444262304300214550ustar00rootroot00000000000000from torch_geometric.testing import onlyFullTest


@onlyFullTest
def test_suite_sparse_dataset(get_dataset):
    dataset = get_dataset(group='DIMACS10', name='citationCiteseer')
    assert str(dataset) == ('SuiteSparseMatrixCollection('
                            'group=DIMACS10, name=citationCiteseer)')
    assert len(dataset) == 1


@onlyFullTest
def test_illc1850_suite_sparse_dataset(get_dataset):
    dataset = get_dataset(group='HB', name='illc1850')
    assert str(dataset) == ('SuiteSparseMatrixCollection('
                            'group=HB, name=illc1850)')
    assert len(dataset) == 1
pytorch/test/distributed/000077500000000000000000000000001444262304300160515ustar00rootroot00000000000000pytorch/test/distributed/test_local_feature_store.py000066400000000000000000000076251444262304300235150ustar00rootroot00000000000000import torch

from torch_geometric.distributed import LocalFeatureStore


def test_local_feature_store_global_id():
    store = LocalFeatureStore()

    feat = torch.Tensor([
        [0, 0, 0],
        [1, 1, 1],
        [2, 2, 2],
        [3, 3, 3],
        [4, 4, 4],
        [5, 5, 5],
        [6, 6, 6],
        [7, 7, 7],
        [8, 8, 8],
    ])

    paper_global_id = torch.tensor([1, 2, 3, 5, 8, 4])
    paper_feat = feat[paper_global_id]

    store.put_global_id(paper_global_id, group_name='paper')
    store.put_tensor(paper_feat, group_name='paper', attr_name='feat')

    out = store.get_tensor_from_global_id(group_name='paper', attr_name='feat',
                                          index=torch.tensor([3, 8, 4]))
    assert torch.equal(out, feat[torch.tensor([3, 8, 4])])


def test_local_feature_store_utils():
    store = LocalFeatureStore()

    feat = torch.Tensor([
        [0, 0, 0],
        [1, 1, 1],
        [2, 2, 2],
        [3, 3, 3],
        [4, 4, 4],
        [5, 5, 5],
        [6, 6, 6],
        [7, 7, 7],
        [8, 8, 8],
    ])

    paper_global_id = torch.tensor([1, 2, 3, 5, 8, 4])
    paper_feat = feat[paper_global_id]

    store.put_tensor(paper_feat, group_name='paper', attr_name='feat')

    assert len(store.get_all_tensor_attrs()) == 1
    attr = store.get_all_tensor_attrs()[0]
    assert attr.group_name == 'paper'
    assert attr.attr_name == 'feat'
    assert attr.index is None
    assert store.get_tensor_size(attr) == (6, 3)


def test_homogeneous_feature_store():
    node_id = torch.randperm(6)
    x = torch.randn(6, 32)
    y = torch.randint(0, 2, (6, ))
    edge_id = torch.randperm(12)
    edge_attr = torch.randn(12, 16)

    store = LocalFeatureStore.from_data(node_id, x, y, edge_id, edge_attr)

    assert len(store.get_all_tensor_attrs()) == 3
    attrs = store.get_all_tensor_attrs()

    assert attrs[0].group_name is None
    assert attrs[0].attr_name == 'x'
    assert attrs[1].group_name is None
    assert attrs[1].attr_name == 'y'
    assert attrs[2].group_name == (None, None)
    assert attrs[2].attr_name == 'edge_attr'

    assert torch.equal(store.get_global_id(group_name=None), node_id)
    assert torch.equal(store.get_tensor(group_name=None, attr_name='x'), x)
    assert torch.equal(store.get_tensor(group_name=None, attr_name='y'), y)
    assert torch.equal(store.get_global_id(group_name=(None, None)), edge_id)
    assert torch.equal(
        store.get_tensor(group_name=(None, None), attr_name='edge_attr'),
        edge_attr,
    )


def test_heterogeneous_feature_store():
    node_type = 'paper'
    edge_type = ('paper', 'to', 'paper')
    node_id_dict = {node_type: torch.randperm(6)}
    x_dict = {node_type: torch.randn(6, 32)}
    y_dict = {node_type: torch.randint(0, 2, (6, ))}
    edge_id_dict = {edge_type: torch.randperm(12)}
    edge_attr_dict = {edge_type: torch.randn(12, 16)}

    store = LocalFeatureStore.from_hetero_data(
        node_id_dict,
        x_dict,
        y_dict,
        edge_id_dict,
        edge_attr_dict,
    )

    assert len(store.get_all_tensor_attrs()) == 3
    attrs = store.get_all_tensor_attrs()

    assert attrs[0].group_name == node_type
    assert attrs[0].attr_name == 'x'
    assert attrs[1].group_name == node_type
    assert attrs[1].attr_name == 'y'
    assert attrs[2].group_name == edge_type
    assert attrs[2].attr_name == 'edge_attr'

    assert torch.equal(
        store.get_global_id(group_name=node_type),
        node_id_dict[node_type],
    )
    assert torch.equal(
        store.get_tensor(group_name=node_type, attr_name='x'),
        x_dict[node_type],
    )
    assert torch.equal(
        store.get_tensor(group_name=node_type, attr_name='y'),
        y_dict[node_type],
    )
    assert torch.equal(
        store.get_global_id(group_name=edge_type),
        edge_id_dict[edge_type],
    )
    assert torch.equal(
        store.get_tensor(group_name=edge_type, attr_name='edge_attr'),
        edge_attr_dict[edge_type],
    )
pytorch/test/distributed/test_local_graph_store.py000066400000000000000000000046441444262304300231610ustar00rootroot00000000000000import torch

from torch_geometric.distributed import LocalGraphStore
from torch_geometric.testing import get_random_edge_index


def test_local_graph_store():
    graph_store = LocalGraphStore()

    edge_index = get_random_edge_index(100, 100, 300)
    edge_id = torch.tensor([1, 2, 3, 5, 8, 4])

    graph_store.put_edge_index(edge_index, edge_type=None, layout='coo',
                               size=(100, 100))

    graph_store.put_edge_id(edge_id, edge_type=None, layout='coo',
                            size=(100, 100))

    assert len(graph_store.get_all_edge_attrs()) == 1
    edge_attr = graph_store.get_all_edge_attrs()[0]
    assert torch.equal(graph_store.get_edge_index(edge_attr), edge_index)
    assert torch.equal(graph_store.get_edge_id(edge_attr), edge_id)

    graph_store.remove_edge_index(edge_attr)
    graph_store.remove_edge_id(edge_attr)
    assert len(graph_store.get_all_edge_attrs()) == 0


def test_homogeneous_graph_store():
    edge_id = torch.randperm(300)
    edge_index = get_random_edge_index(100, 100, 300)

    graph_store = LocalGraphStore.from_data(edge_id, edge_index, num_nodes=100)

    assert len(graph_store.get_all_edge_attrs()) == 1
    edge_attr = graph_store.get_all_edge_attrs()[0]
    assert edge_attr.edge_type is None
    assert edge_attr.layout.value == 'coo'
    assert not edge_attr.is_sorted
    assert edge_attr.size == (100, 100)

    assert torch.equal(
        graph_store.get_edge_id(edge_type=None, layout='coo'),
        edge_id,
    )
    assert torch.equal(
        graph_store.get_edge_index(edge_type=None, layout='coo'),
        edge_index,
    )


def test_heterogeneous_graph_store():
    edge_type = ('paper', 'to', 'paper')
    edge_id_dict = {edge_type: torch.randperm(300)}
    edge_index_dict = {edge_type: get_random_edge_index(100, 100, 300)}

    graph_store = LocalGraphStore.from_hetero_data(
        edge_id_dict, edge_index_dict, num_nodes_dict={'paper': 100})

    assert len(graph_store.get_all_edge_attrs()) == 1
    edge_attr = graph_store.get_all_edge_attrs()[0]
    assert edge_attr.edge_type == edge_type
    assert edge_attr.layout.value == 'coo'
    assert not edge_attr.is_sorted
    assert edge_attr.size == (100, 100)

    assert torch.equal(
        graph_store.get_edge_id(edge_type, layout='coo'),
        edge_id_dict[edge_type],
    )
    assert torch.equal(
        graph_store.get_edge_index(edge_type, layout='coo'),
        edge_index_dict[edge_type],
    )
pytorch/test/explain/000077500000000000000000000000001444262304300151675ustar00rootroot00000000000000pytorch/test/explain/algorithm/000077500000000000000000000000001444262304300171555ustar00rootroot00000000000000pytorch/test/explain/algorithm/test_attention_explainer.py000066400000000000000000000040111444262304300246360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import AttentionExplainer, Explainer
from torch_geometric.explain.config import ExplanationType, MaskType
from torch_geometric.nn import GATConv, GATv2Conv, TransformerConv


class AttentionGNN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GATConv(3, 16, heads=4)
        self.conv2 = GATv2Conv(4 * 16, 16, heads=2)
        self.conv3 = TransformerConv(2 * 16, 7, heads=1)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        x = self.conv3(x, edge_index)
        return x


x = torch.randn(8, 3)
edge_index = torch.tensor([
    [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
    [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
])


@pytest.mark.parametrize('index', [None, 2, torch.arange(3)])
def test_attention_explainer(index, check_explanation):
    explainer = Explainer(
        model=AttentionGNN(),
        algorithm=AttentionExplainer(),
        explanation_type='model',
        edge_mask_type='object',
        model_config=dict(
            mode='multiclass_classification',
            task_level='node',
            return_type='raw',
        ),
    )

    explanation = explainer(x, edge_index, index=index)
    check_explanation(explanation, None, explainer.edge_mask_type)


@pytest.mark.parametrize('explanation_type', [e for e in ExplanationType])
@pytest.mark.parametrize('node_mask_type', [m for m in MaskType])
def test_attention_explainer_supports(explanation_type, node_mask_type):
    with pytest.raises(ValueError, match="not support the given explanation"):
        Explainer(
            model=AttentionGNN(),
            algorithm=AttentionExplainer(),
            explanation_type=explanation_type,
            node_mask_type=node_mask_type,
            edge_mask_type='object',
            model_config=dict(
                mode='multiclass_classification',
                task_level='node',
                return_type='raw',
            ),
        )
pytorch/test/explain/algorithm/test_captum.py000066400000000000000000000176471444262304300220760ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.explain.algorithm.captum import to_captum_input
from torch_geometric.nn import GAT, GCN, SAGEConv
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.models import to_captum_model
from torch_geometric.testing import withPackage

x = torch.randn(8, 3, requires_grad=True)
edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
                           [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6]])

GCN = GCN(3, 16, 2, 7, dropout=0.5)
GAT = GAT(3, 16, 2, 7, heads=2, concat=False)
mask_types = ['edge', 'node_and_edge', 'node']
methods = [
    'Saliency',
    'InputXGradient',
    'Deconvolution',
    'FeatureAblation',
    'ShapleyValueSampling',
    'IntegratedGradients',
    'GradientShap',
    'Occlusion',
    'GuidedBackprop',
    'KernelShap',
    'Lime',
]


@pytest.mark.parametrize('mask_type', mask_types)
@pytest.mark.parametrize('model', [GCN, GAT])
@pytest.mark.parametrize('output_idx', [None, 1])
def test_to_captum(model, mask_type, output_idx):
    captum_model = to_captum_model(model, mask_type=mask_type,
                                   output_idx=output_idx)
    pre_out = model(x, edge_index)
    if mask_type == 'node':
        mask = x * 0.0
        out = captum_model(mask.unsqueeze(0), edge_index)
    elif mask_type == 'edge':
        mask = torch.ones(edge_index.shape[1], dtype=torch.float,
                          requires_grad=True) * 0.5
        out = captum_model(mask.unsqueeze(0), x, edge_index)
    elif mask_type == 'node_and_edge':
        node_mask = x * 0.0
        edge_mask = torch.ones(edge_index.shape[1], dtype=torch.float,
                               requires_grad=True) * 0.5
        out = captum_model(node_mask.unsqueeze(0), edge_mask.unsqueeze(0),
                           edge_index)

    if output_idx is not None:
        assert out.shape == (1, 7)
        assert torch.any(out != pre_out[[output_idx]])
    else:
        assert out.shape == (8, 7)
        assert torch.any(out != pre_out)


@withPackage('captum')
@pytest.mark.parametrize('mask_type', mask_types)
@pytest.mark.parametrize('method', methods)
def test_captum_attribution_methods(mask_type, method):
    from captum import attr  # noqa

    captum_model = to_captum_model(GCN, mask_type, 0)
    explainer = getattr(attr, method)(captum_model)
    data = Data(x, edge_index)
    input, additional_forward_args = to_captum_input(data.x, data.edge_index,
                                                     mask_type)
    if mask_type == 'node':
        sliding_window_shapes = (3, 3)
    elif mask_type == 'edge':
        sliding_window_shapes = (5, )
    elif mask_type == 'node_and_edge':
        sliding_window_shapes = ((3, 3), (5, ))

    if method == 'IntegratedGradients':
        attributions, delta = explainer.attribute(
            input, target=0, internal_batch_size=1,
            additional_forward_args=additional_forward_args,
            return_convergence_delta=True)
    elif method == 'GradientShap':
        attributions, delta = explainer.attribute(
            input, target=0, return_convergence_delta=True, baselines=input,
            n_samples=1, additional_forward_args=additional_forward_args)
    elif method == 'DeepLiftShap' or method == 'DeepLift':
        attributions, delta = explainer.attribute(
            input, target=0, return_convergence_delta=True, baselines=input,
            additional_forward_args=additional_forward_args)
    elif method == 'Occlusion':
        attributions = explainer.attribute(
            input, target=0, sliding_window_shapes=sliding_window_shapes,
            additional_forward_args=additional_forward_args)
    else:
        attributions = explainer.attribute(
            input, target=0, additional_forward_args=additional_forward_args)
    if mask_type == 'node':
        assert attributions[0].shape == (1, 8, 3)
    elif mask_type == 'edge':
        assert attributions[0].shape == (1, 14)
    else:
        assert attributions[0].shape == (1, 8, 3)
        assert attributions[1].shape == (1, 14)


def test_custom_explain_message():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])

    conv = SAGEConv(8, 32)

    def explain_message(self, inputs, x_i, x_j):
        assert isinstance(self, SAGEConv)
        assert inputs.size() == (6, 8)
        assert inputs.size() == x_i.size() == x_j.size()
        assert torch.allclose(inputs, x_j)
        self.x_i = x_i
        self.x_j = x_j
        return inputs

    conv.explain_message = explain_message.__get__(conv, MessagePassing)
    conv.explain = True

    conv(x, edge_index)

    assert torch.allclose(conv.x_i, x[edge_index[1]])
    assert torch.allclose(conv.x_j, x[edge_index[0]])


@withPackage('captum')
@pytest.mark.parametrize('mask_type', ['node', 'edge', 'node_and_edge'])
def test_to_captum_input(mask_type):
    num_nodes = x.shape[0]
    num_node_feats = x.shape[1]
    num_edges = edge_index.shape[1]

    # Check for Data:
    data = Data(x, edge_index)
    args = 'test_args'
    inputs, additional_forward_args = to_captum_input(data.x, data.edge_index,
                                                      mask_type, args)
    if mask_type == 'node':
        assert len(inputs) == 1
        assert inputs[0].shape == (1, num_nodes, num_node_feats)
        assert len(additional_forward_args) == 2
        assert torch.allclose(additional_forward_args[0], edge_index)
    elif mask_type == 'edge':
        assert len(inputs) == 1
        assert inputs[0].shape == (1, num_edges)
        assert inputs[0].sum() == num_edges
        assert len(additional_forward_args) == 3
        assert torch.allclose(additional_forward_args[0], x)
        assert torch.allclose(additional_forward_args[1], edge_index)
    else:
        assert len(inputs) == 2
        assert inputs[0].shape == (1, num_nodes, num_node_feats)
        assert inputs[1].shape == (1, num_edges)
        assert inputs[1].sum() == num_edges
        assert len(additional_forward_args) == 2
        assert torch.allclose(additional_forward_args[0], edge_index)

    # Check for HeteroData:
    data = HeteroData()
    x2 = torch.rand(8, 3)
    data['paper'].x = x
    data['author'].x = x2
    data['paper', 'to', 'author'].edge_index = edge_index
    data['author', 'to', 'paper'].edge_index = edge_index.flip([0])
    inputs, additional_forward_args = to_captum_input(data.x_dict,
                                                      data.edge_index_dict,
                                                      mask_type, args)
    if mask_type == 'node':
        assert len(inputs) == 2
        assert inputs[0].shape == (1, num_nodes, num_node_feats)
        assert inputs[1].shape == (1, num_nodes, num_node_feats)
        assert len(additional_forward_args) == 2
        for key in data.edge_types:
            torch.allclose(additional_forward_args[0][key],
                           data[key].edge_index)
    elif mask_type == 'edge':
        assert len(inputs) == 2
        assert inputs[0].shape == (1, num_edges)
        assert inputs[1].shape == (1, num_edges)
        assert inputs[1].sum() == inputs[0].sum() == num_edges
        assert len(additional_forward_args) == 3
        for key in data.node_types:
            torch.allclose(additional_forward_args[0][key], data[key].x)
        for key in data.edge_types:
            torch.allclose(additional_forward_args[1][key],
                           data[key].edge_index)
    else:
        assert len(inputs) == 4
        assert inputs[0].shape == (1, num_nodes, num_node_feats)
        assert inputs[1].shape == (1, num_nodes, num_node_feats)
        assert inputs[2].shape == (1, num_edges)
        assert inputs[3].shape == (1, num_edges)
        assert inputs[3].sum() == inputs[2].sum() == num_edges
        assert len(additional_forward_args) == 2
        for key in data.edge_types:
            torch.allclose(additional_forward_args[0][key],
                           data[key].edge_index)
pytorch/test/explain/algorithm/test_captum_explainer.py000066400000000000000000000141631444262304300241330ustar00rootroot00000000000000from typing import Optional

import pytest
import torch

from torch_geometric.explain import Explainer, Explanation
from torch_geometric.explain.algorithm import CaptumExplainer
from torch_geometric.explain.config import (
    MaskType,
    ModelConfig,
    ModelMode,
    ModelReturnType,
    ModelTaskLevel,
)
from torch_geometric.nn import GCNConv, global_add_pool
from torch_geometric.testing import withPackage

methods = [
    'Saliency',
    'InputXGradient',
    'Deconvolution',
    'ShapleyValueSampling',
    'IntegratedGradients',
    'GuidedBackprop',
]

unsupported_methods = [
    'FeatureAblation',
    'Occlusion',
    'DeepLift',
    'DeepLiftShap',
    'GradientShap',
    'KernelShap',
    'Lime',
]


class GCN(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config

        if model_config.mode == ModelMode.multiclass_classification:
            out_channels = 7
        else:
            out_channels = 1

        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, out_channels)

    def forward(self, x, edge_index, batch=None, edge_label_index=None):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)

        if self.model_config.task_level == ModelTaskLevel.graph:
            x = global_add_pool(x, batch)
        elif self.model_config.task_level == ModelTaskLevel.edge:
            assert edge_label_index is not None
            x = x[edge_label_index[0]] * x[edge_label_index[1]]

        if self.model_config.mode == ModelMode.binary_classification:
            if self.model_config.return_type == ModelReturnType.probs:
                x = x.sigmoid()
        elif self.model_config.mode == ModelMode.multiclass_classification:
            if self.model_config.return_type == ModelReturnType.probs:
                x = x.softmax(dim=-1)
            elif self.model_config.return_type == ModelReturnType.log_probs:
                x = x.log_softmax(dim=-1)

        return x


node_mask_types = [
    MaskType.attributes,
    None,
]
edge_mask_types = [
    MaskType.object,
    None,
]


def check_explanation(
    explanation: Explanation,
    node_mask_type: Optional[MaskType],
    edge_mask_type: Optional[MaskType],
):
    if node_mask_type == MaskType.attributes:
        assert explanation.node_mask.size() == explanation.x.size()
    elif node_mask_type is None:
        assert 'node_mask' not in explanation

    if edge_mask_type == MaskType.object:
        assert explanation.edge_mask.size() == (explanation.num_edges, )
    elif edge_mask_type is None:
        assert 'edge_mask' not in explanation


@withPackage('captum')
@pytest.mark.parametrize('method', unsupported_methods)
def test_unsupported_methods(method):
    model_config = ModelConfig(mode='regression', task_level='node')

    with pytest.raises(ValueError, match="does not support attribution"):
        Explainer(
            GCN(model_config),
            algorithm=CaptumExplainer(method),
            explanation_type='model',
            edge_mask_type='object',
            node_mask_type='attributes',
            model_config=model_config,
        )


@withPackage('captum')
@pytest.mark.parametrize('method', methods)
@pytest.mark.parametrize('node_mask_type', node_mask_types)
@pytest.mark.parametrize('edge_mask_type', edge_mask_types)
@pytest.mark.parametrize('task_level', ['node', 'edge', 'graph'])
@pytest.mark.parametrize('index', [1, torch.arange(2)])
def test_captum_explainer_multiclass_classification(
    method,
    data,
    node_mask_type,
    edge_mask_type,
    task_level,
    index,
):
    if node_mask_type is None and edge_mask_type is None:
        return

    batch = torch.tensor([0, 0, 1, 1])
    edge_label_index = torch.tensor([[0, 1, 2], [2, 3, 1]])

    model_config = ModelConfig(
        mode='multiclass_classification',
        task_level=task_level,
        return_type='probs',
    )

    explainer = Explainer(
        GCN(model_config),
        algorithm=CaptumExplainer(method),
        explanation_type='model',
        edge_mask_type=edge_mask_type,
        node_mask_type=node_mask_type,
        model_config=model_config,
    )

    explanation = explainer(
        data.x,
        data.edge_index,
        index=index,
        batch=batch,
        edge_label_index=edge_label_index,
    )
    check_explanation(explanation, node_mask_type, edge_mask_type)


@withPackage('captum')
@pytest.mark.parametrize('method', methods)
@pytest.mark.parametrize('node_mask_type', node_mask_types)
@pytest.mark.parametrize('edge_mask_type', edge_mask_types)
@pytest.mark.parametrize('index', [1, torch.arange(2)])
def test_captum_hetero_data(method, node_mask_type, edge_mask_type, index,
                            hetero_data, hetero_model):

    if method == 'ShapleyValueSampling':
        # This currently takes too long to test and is already covered by
        # by the homogeneous graph test case.
        return

    if node_mask_type is None or edge_mask_type is None:
        return

    model_config = ModelConfig(mode='regression', task_level='node')

    explainer = Explainer(
        hetero_model(hetero_data.metadata()),
        algorithm=CaptumExplainer(method),
        edge_mask_type=edge_mask_type,
        node_mask_type=node_mask_type,
        model_config=model_config,
        explanation_type='model',
    )

    explanation = explainer(hetero_data.x_dict, hetero_data.edge_index_dict,
                            index=index)

    explanation.validate(raise_on_error=True)


@withPackage('captum')
@pytest.mark.parametrize('node_mask_type', [
    MaskType.object,
    MaskType.common_attributes,
])
def test_captum_explainer_supports(node_mask_type):
    model_config = ModelConfig(
        mode='multiclass_classification',
        task_level='node',
        return_type='probs',
    )

    with pytest.raises(ValueError, match="not support the given explanation"):
        Explainer(
            GCN(model_config),
            algorithm=CaptumExplainer('IntegratedGradients'),
            edge_mask_type=MaskType.object,
            node_mask_type=node_mask_type,
            model_config=model_config,
            explanation_type='model',
        )
pytorch/test/explain/algorithm/test_captum_hetero.py000066400000000000000000000076131444262304300234340ustar00rootroot00000000000000import pytest

from torch_geometric.explain.algorithm.captum import (
    CaptumHeteroModel,
    captum_output_to_dicts,
    to_captum_input,
)
from torch_geometric.nn import to_captum_model
from torch_geometric.testing import withPackage

mask_types = [
    'node',
    'edge',
    'node_and_edge',
]

methods = [
    'Saliency',
    'InputXGradient',
    'Deconvolution',
    'FeatureAblation',
    'ShapleyValueSampling',
    'IntegratedGradients',
    'GradientShap',
    'Occlusion',
    'GuidedBackprop',
    'KernelShap',
    'Lime',
]


@withPackage('captum')
@pytest.mark.parametrize('mask_type', mask_types)
@pytest.mark.parametrize('method', methods)
def test_captum_attribution_methods_hetero(mask_type, method, hetero_data,
                                           hetero_model):
    from captum import attr  # noqa
    data = hetero_data
    metadata = data.metadata()
    model = hetero_model(metadata)
    captum_model = to_captum_model(model, mask_type, 0, metadata)
    explainer = getattr(attr, method)(captum_model)
    assert isinstance(captum_model, CaptumHeteroModel)

    inputs, additional_forward_args = to_captum_input(
        data.x_dict,
        data.edge_index_dict,
        mask_type,
        'additional_arg',
    )

    if mask_type == 'node':
        sliding_window_shapes = ((3, 3), (3, 3))
    elif mask_type == 'edge':
        sliding_window_shapes = ((5, ), (5, ), (5, ))
    else:
        sliding_window_shapes = ((3, 3), (3, 3), (5, ), (5, ), (5, ))

    if method == 'IntegratedGradients':
        attributions, delta = explainer.attribute(
            inputs, target=0, internal_batch_size=1,
            additional_forward_args=additional_forward_args,
            return_convergence_delta=True)
    elif method == 'GradientShap':
        attributions, delta = explainer.attribute(
            inputs, target=0, return_convergence_delta=True, baselines=inputs,
            n_samples=1, additional_forward_args=additional_forward_args)
    elif method == 'DeepLiftShap' or method == 'DeepLift':
        attributions, delta = explainer.attribute(
            inputs, target=0, return_convergence_delta=True, baselines=inputs,
            additional_forward_args=additional_forward_args)
    elif method == 'Occlusion':
        attributions = explainer.attribute(
            inputs, target=0, sliding_window_shapes=sliding_window_shapes,
            additional_forward_args=additional_forward_args)
    else:
        attributions = explainer.attribute(
            inputs, target=0, additional_forward_args=additional_forward_args)

    if mask_type == 'node':
        assert len(attributions) == len(metadata[0])
        x_attr_dict, _ = captum_output_to_dicts(attributions, mask_type,
                                                metadata)
        for node_type in metadata[0]:
            num_nodes = data[node_type].num_nodes
            num_node_feats = data[node_type].x.shape[1]
            assert x_attr_dict[node_type].shape == (num_nodes, num_node_feats)
    elif mask_type == 'edge':
        assert len(attributions) == len(metadata[1])
        _, edge_attr_dict = captum_output_to_dicts(attributions, mask_type,
                                                   metadata)
        for edge_type in metadata[1]:
            num_edges = data[edge_type].edge_index.shape[1]
            assert edge_attr_dict[edge_type].shape == (num_edges, )
    else:
        assert len(attributions) == len(metadata[0]) + len(metadata[1])
        x_attr_dict, edge_attr_dict = captum_output_to_dicts(
            attributions, mask_type, metadata)
        for edge_type in metadata[1]:
            num_edges = data[edge_type].edge_index.shape[1]
            assert edge_attr_dict[edge_type].shape == (num_edges, )

        for node_type in metadata[0]:
            num_nodes = data[node_type].num_nodes
            num_node_feats = data[node_type].x.shape[1]
            assert x_attr_dict[node_type].shape == (num_nodes, num_node_feats)
pytorch/test/explain/algorithm/test_explain_algorithm_utils.py000066400000000000000000000051621444262304300255200ustar00rootroot00000000000000import torch

from torch_geometric.explain.algorithm.utils import (
    clear_masks,
    set_hetero_masks,
)
from torch_geometric.nn import GCNConv, HeteroConv, SAGEConv, to_hetero


class HeteroModel(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = HeteroConv({
            ('paper', 'to', 'paper'):
            GCNConv(-1, 32),
            ('author', 'to', 'paper'):
            SAGEConv((-1, -1), 32),
            ('paper', 'to', 'author'):
            SAGEConv((-1, -1), 32),
        })

        self.conv2 = HeteroConv({
            ('paper', 'to', 'paper'):
            GCNConv(-1, 32),
            ('author', 'to', 'paper'):
            SAGEConv((-1, -1), 32),
            ('paper', 'to', 'author'):
            SAGEConv((-1, -1), 32),
        })


class GraphSAGE(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), 32)
        self.conv2 = SAGEConv((-1, -1), 32)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)


def test_set_clear_mask(hetero_data):
    edge_mask_dict = {
        ('paper', 'to', 'paper'): torch.ones(200),
        ('author', 'to', 'paper'): torch.ones(100),
        ('paper', 'to', 'author'): torch.ones(100),
    }

    model = HeteroModel()

    set_hetero_masks(model, edge_mask_dict, hetero_data.edge_index_dict)
    for edge_type in hetero_data.edge_types:
        # Check that masks are correctly set:
        str_edge_type = '__'.join(edge_type)
        assert torch.allclose(model.conv1.convs[str_edge_type]._edge_mask,
                              edge_mask_dict[edge_type])
        assert model.conv1.convs[str_edge_type].explain

    clear_masks(model)
    for edge_type in hetero_data.edge_types:
        str_edge_type = '__'.join(edge_type)
        assert model.conv1.convs[str_edge_type]._edge_mask is None
        assert not model.conv1.convs[str_edge_type].explain

    model = to_hetero(GraphSAGE(), hetero_data.metadata(), debug=False)

    set_hetero_masks(model, edge_mask_dict, hetero_data.edge_index_dict)
    for edge_type in hetero_data.edge_types:
        # Check that masks are correctly set:
        str_edge_type = '__'.join(edge_type)
        assert torch.allclose(model.conv1[str_edge_type]._edge_mask,
                              edge_mask_dict[edge_type])
        assert model.conv1[str_edge_type].explain

    clear_masks(model)
    for edge_type in hetero_data.edge_types:
        str_edge_type = '__'.join(edge_type)
        assert model.conv1[str_edge_type]._edge_mask is None
        assert not model.conv1[str_edge_type].explain
pytorch/test/explain/algorithm/test_gnn_explainer.py000066400000000000000000000163731444262304300234310ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import Explainer, GNNExplainer
from torch_geometric.explain.config import (
    ExplanationType,
    MaskType,
    ModelConfig,
    ModelMode,
    ModelReturnType,
    ModelTaskLevel,
)
from torch_geometric.nn import ChebConv, GCNConv, global_add_pool


class GCN(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config

        if model_config.mode == ModelMode.multiclass_classification:
            out_channels = 7
        else:
            out_channels = 1

        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, out_channels)

    def forward(self, x, edge_index, batch=None, edge_label_index=None):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)

        if self.model_config.task_level == ModelTaskLevel.graph:
            x = global_add_pool(x, batch)
        elif self.model_config.task_level == ModelTaskLevel.edge:
            assert edge_label_index is not None
            x = x[edge_label_index[0]] * x[edge_label_index[1]]

        if self.model_config.mode == ModelMode.binary_classification:
            if self.model_config.return_type == ModelReturnType.probs:
                x = x.sigmoid()
        elif self.model_config.mode == ModelMode.multiclass_classification:
            if self.model_config.return_type == ModelReturnType.probs:
                x = x.softmax(dim=-1)
            elif self.model_config.return_type == ModelReturnType.log_probs:
                x = x.log_softmax(dim=-1)

        return x


node_mask_types = [
    MaskType.object,
    MaskType.common_attributes,
    MaskType.attributes,
]
edge_mask_types = [MaskType.object, None]
explanation_types = [ExplanationType.model, ExplanationType.phenomenon]
task_levels = [ModelTaskLevel.node, ModelTaskLevel.edge, ModelTaskLevel.graph]
indices = [None, 2, torch.arange(3)]

x = torch.randn(8, 3)
edge_index = torch.tensor([
    [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
    [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
])
batch = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2])
edge_label_index = torch.tensor([[0, 1, 2], [3, 4, 5]])


@pytest.mark.parametrize('edge_mask_type', edge_mask_types)
@pytest.mark.parametrize('node_mask_type', node_mask_types)
@pytest.mark.parametrize('explanation_type', explanation_types)
@pytest.mark.parametrize('task_level', task_levels)
@pytest.mark.parametrize('return_type', [
    ModelReturnType.probs,
    ModelReturnType.raw,
])
@pytest.mark.parametrize('index', indices)
def test_gnn_explainer_binary_classification(
    edge_mask_type,
    node_mask_type,
    explanation_type,
    task_level,
    return_type,
    index,
    check_explanation,
):
    model_config = ModelConfig(
        mode='binary_classification',
        task_level=task_level,
        return_type=return_type,
    )

    model = GCN(model_config)

    target = None
    if explanation_type == ExplanationType.phenomenon:
        with torch.no_grad():
            out = model(x, edge_index, batch, edge_label_index)
            if model_config.return_type == ModelReturnType.raw:
                target = (out > 0).long().view(-1)
            if model_config.return_type == ModelReturnType.probs:
                target = (out > 0.5).long().view(-1)

    explainer = Explainer(
        model=model,
        algorithm=GNNExplainer(epochs=2),
        explanation_type=explanation_type,
        node_mask_type=node_mask_type,
        edge_mask_type=edge_mask_type,
        model_config=model_config,
    )

    explanation = explainer(
        x,
        edge_index,
        target=target,
        index=index,
        batch=batch,
        edge_label_index=edge_label_index,
    )

    assert explainer.algorithm.node_mask is None
    assert explainer.algorithm.edge_mask is None

    check_explanation(explanation, node_mask_type, edge_mask_type)


@pytest.mark.parametrize('edge_mask_type', edge_mask_types)
@pytest.mark.parametrize('node_mask_type', node_mask_types)
@pytest.mark.parametrize('explanation_type', explanation_types)
@pytest.mark.parametrize('task_level', task_levels)
@pytest.mark.parametrize('return_type', [
    ModelReturnType.log_probs,
    ModelReturnType.probs,
    ModelReturnType.raw,
])
@pytest.mark.parametrize('index', indices)
def test_gnn_explainer_multiclass_classification(
    edge_mask_type,
    node_mask_type,
    explanation_type,
    task_level,
    return_type,
    index,
    check_explanation,
):
    model_config = ModelConfig(
        mode='multiclass_classification',
        task_level=task_level,
        return_type=return_type,
    )

    model = GCN(model_config)

    target = None
    if explanation_type == ExplanationType.phenomenon:
        with torch.no_grad():
            target = model(x, edge_index, batch, edge_label_index).argmax(-1)

    explainer = Explainer(
        model=model,
        algorithm=GNNExplainer(epochs=2),
        explanation_type=explanation_type,
        node_mask_type=node_mask_type,
        edge_mask_type=edge_mask_type,
        model_config=model_config,
    )

    explanation = explainer(
        x,
        edge_index,
        target=target,
        index=index,
        batch=batch,
        edge_label_index=edge_label_index,
    )

    assert explainer.algorithm.node_mask is None
    assert explainer.algorithm.edge_mask is None

    check_explanation(explanation, node_mask_type, edge_mask_type)


@pytest.mark.parametrize('edge_mask_type', edge_mask_types)
@pytest.mark.parametrize('node_mask_type', node_mask_types)
@pytest.mark.parametrize('explanation_type', explanation_types)
@pytest.mark.parametrize('task_level', task_levels)
@pytest.mark.parametrize('index', indices)
def test_gnn_explainer_regression(
    edge_mask_type,
    node_mask_type,
    explanation_type,
    task_level,
    index,
    check_explanation,
):
    model_config = ModelConfig(
        mode='regression',
        task_level=task_level,
    )

    model = GCN(model_config)

    target = None
    if explanation_type == ExplanationType.phenomenon:
        with torch.no_grad():
            target = model(x, edge_index, batch, edge_label_index)

    explainer = Explainer(
        model=model,
        algorithm=GNNExplainer(epochs=2),
        explanation_type=explanation_type,
        node_mask_type=node_mask_type,
        edge_mask_type=edge_mask_type,
        model_config=model_config,
    )

    explanation = explainer(
        x,
        edge_index,
        target=target,
        index=index,
        batch=batch,
        edge_label_index=edge_label_index,
    )

    assert explainer.algorithm.node_mask is None
    assert explainer.algorithm.edge_mask is None

    check_explanation(explanation, node_mask_type, edge_mask_type)


def test_gnn_explainer_cheb_conv(check_explanation):
    explainer = Explainer(
        model=ChebConv(3, 1, K=2),
        algorithm=GNNExplainer(epochs=2),
        explanation_type='model',
        node_mask_type='object',
        edge_mask_type='object',
        model_config=dict(
            mode='binary_classification',
            task_level='node',
            return_type='raw',
        ),
    )

    explanation = explainer(x, edge_index)

    assert explainer.algorithm.node_mask is None
    assert explainer.algorithm.edge_mask is None

    check_explanation(explanation, MaskType.object, MaskType.object)
pytorch/test/explain/algorithm/test_pg_explainer.py000066400000000000000000000130571444262304300232510ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import Explainer, PGExplainer
from torch_geometric.explain.config import (
    ModelConfig,
    ModelMode,
    ModelTaskLevel,
)
from torch_geometric.nn import GCNConv, global_add_pool
from torch_geometric.testing import withCUDA


class GCN(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config

        if model_config.mode == ModelMode.multiclass_classification:
            out_channels = 7
        else:
            out_channels = 1

        self.conv1 = GCNConv(3, 16)
        self.conv2 = GCNConv(16, out_channels)

    def forward(self, x, edge_index, batch=None, edge_label_index=None):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        if self.model_config.task_level == ModelTaskLevel.graph:
            x = global_add_pool(x, batch)
        return x


@withCUDA
@pytest.mark.parametrize('mode', [
    ModelMode.binary_classification,
    ModelMode.multiclass_classification,
    ModelMode.regression,
])
def test_pg_explainer_node(device, check_explanation, mode):
    x = torch.randn(8, 3, device=device)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
    ], device=device)

    if mode == ModelMode.binary_classification:
        target = torch.randint(2, (x.size(0), ), device=device)
    elif mode == ModelMode.multiclass_classification:
        target = torch.randint(7, (x.size(0), ), device=device)
    elif mode == ModelMode.regression:
        target = torch.randn((x.size(0), 1), device=device)

    model_config = ModelConfig(mode=mode, task_level='node', return_type='raw')

    model = GCN(model_config).to(device)

    explainer = Explainer(
        model=model,
        algorithm=PGExplainer(epochs=2).to(device),
        explanation_type='phenomenon',
        edge_mask_type='object',
        model_config=model_config,
    )

    with pytest.raises(ValueError, match="not yet fully trained"):
        explainer(x, edge_index, target=target)

    explainer.algorithm.reset_parameters()
    for epoch in range(2):
        for index in range(x.size(0)):
            loss = explainer.algorithm.train(epoch, model, x, edge_index,
                                             target=target, index=index)
            assert loss >= 0.0

    explanation = explainer(x, edge_index, target=target, index=0)

    check_explanation(explanation, None, explainer.edge_mask_type)


@withCUDA
@pytest.mark.parametrize('mode', [
    ModelMode.binary_classification,
    ModelMode.multiclass_classification,
    ModelMode.regression,
])
def test_pg_explainer_graph(device, check_explanation, mode):
    x = torch.randn(8, 3, device=device)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
    ], device=device)

    if mode == ModelMode.binary_classification:
        target = torch.randint(2, (1, ), device=device)
    elif mode == ModelMode.multiclass_classification:
        target = torch.randint(7, (1, ), device=device)
    elif mode == ModelMode.regression:
        target = torch.randn((1, 1), device=device)

    model_config = ModelConfig(mode=mode, task_level='graph',
                               return_type='raw')

    model = GCN(model_config).to(device)

    explainer = Explainer(
        model=model,
        algorithm=PGExplainer(epochs=2).to(device),
        explanation_type='phenomenon',
        edge_mask_type='object',
        model_config=model_config,
    )

    with pytest.raises(ValueError, match="not yet fully trained"):
        explainer(x, edge_index, target=target)

    explainer.algorithm.reset_parameters()
    for epoch in range(2):
        loss = explainer.algorithm.train(epoch, model, x, edge_index,
                                         target=target)
        assert loss >= 0.0

    explanation = explainer(x, edge_index, target=target)

    check_explanation(explanation, None, explainer.edge_mask_type)


def test_pg_explainer_supports():
    # Test unsupported model task level:
    with pytest.raises(ValueError, match="not support the given explanation"):
        model_config = ModelConfig(
            mode='binary_classification',
            task_level='edge',
            return_type='raw',
        )
        Explainer(
            model=GCN(model_config),
            algorithm=PGExplainer(epochs=2),
            explanation_type='phenomenon',
            edge_mask_type='object',
            model_config=model_config,
        )

    # Test unsupported explanation type:
    with pytest.raises(ValueError, match="not support the given explanation"):
        model_config = ModelConfig(
            mode='binary_classification',
            task_level='node',
            return_type='raw',
        )
        Explainer(
            model=GCN(model_config),
            algorithm=PGExplainer(epochs=2),
            explanation_type='model',
            edge_mask_type='object',
            model_config=model_config,
        )

    # Test unsupported node mask:
    with pytest.raises(ValueError, match="not support the given explanation"):
        model_config = ModelConfig(
            mode='binary_classification',
            task_level='node',
            return_type='raw',
        )
        Explainer(
            model=GCN(model_config),
            algorithm=PGExplainer(epochs=2),
            explanation_type='model',
            node_mask_type='object',
            edge_mask_type='object',
            model_config=model_config,
        )
pytorch/test/explain/conftest.py000066400000000000000000000061061444262304300173710ustar00rootroot00000000000000from typing import Optional

import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.explain import Explanation
from torch_geometric.explain.config import MaskType
from torch_geometric.nn import SAGEConv, to_hetero
from torch_geometric.testing import get_random_edge_index


@pytest.fixture()
def data():
    return Data(
        x=torch.randn(4, 3),
        edge_index=get_random_edge_index(4, 4, num_edges=6),
        edge_attr=torch.randn(6, 3),
    )


@pytest.fixture()
def hetero_data():
    data = HeteroData()
    data['paper'].x = torch.randn(8, 16)
    data['author'].x = torch.randn(10, 8)

    data['paper', 'paper'].edge_index = get_random_edge_index(8, 8, 10)
    data['paper', 'paper'].edge_attr = torch.randn(10, 16)
    data['paper', 'author'].edge_index = get_random_edge_index(8, 10, 10)
    data['paper', 'author'].edge_attr = torch.randn(10, 8)
    data['author', 'paper'].edge_index = get_random_edge_index(10, 8, 10)
    data['author', 'paper'].edge_attr = torch.randn(10, 8)

    return data


@pytest.fixture()
def hetero_model():
    return HeteroSAGE


class GraphSAGE(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), 32)
        self.conv2 = SAGEConv((-1, -1), 32)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)


class HeteroSAGE(torch.nn.Module):
    def __init__(self, metadata):
        super().__init__()
        self.graph_sage = to_hetero(GraphSAGE(), metadata, debug=False)
        self.lin = torch.nn.Linear(32, 1)

    def forward(self, x_dict, edge_index_dict,
                additonal_arg=None) -> torch.Tensor:
        return self.lin(self.graph_sage(x_dict, edge_index_dict)['paper'])


@pytest.fixture()
def check_explanation():
    def _check_explanation(
        explanation: Explanation,
        node_mask_type: Optional[MaskType],
        edge_mask_type: Optional[MaskType],
    ):
        if node_mask_type == MaskType.attributes:
            assert explanation.node_mask.size() == explanation.x.size()
            assert explanation.node_mask.min() >= 0
            assert explanation.node_mask.max() <= 1
        elif node_mask_type == MaskType.object:
            assert explanation.node_mask.size() == (explanation.num_nodes, 1)
            assert explanation.node_mask.min() >= 0
            assert explanation.node_mask.max() <= 1
        elif node_mask_type == MaskType.common_attributes:
            assert explanation.node_mask.size() == (1, explanation.x.size(-1))
            assert explanation.node_mask.min() >= 0
            assert explanation.node_mask.max() <= 1
        elif node_mask_type is None:
            assert 'node_mask' not in explanation

        if edge_mask_type == MaskType.object:
            assert explanation.edge_mask.size() == (explanation.num_edges, )
            assert explanation.edge_mask.min() >= 0
            assert explanation.edge_mask.max() <= 1
        elif edge_mask_type is None:
            assert 'edge_mask' not in explanation

    return _check_explanation
pytorch/test/explain/metric/000077500000000000000000000000001444262304300164525ustar00rootroot00000000000000pytorch/test/explain/metric/test_basic_metric.py000066400000000000000000000027341444262304300225150ustar00rootroot00000000000000import warnings

import torch

from torch_geometric.explain import groundtruth_metrics
from torch_geometric.testing import withPackage


@withPackage('torchmetrics>=0.10.0')
def test_groundtruth_metrics():
    pred_mask = torch.rand(10)
    target_mask = torch.rand(10)

    accuracy, recall, precision, f1_score, auroc = groundtruth_metrics(
        pred_mask, target_mask)

    assert accuracy >= 0.0 and accuracy <= 1.0
    assert recall >= 0.0 and recall <= 1.0
    assert precision >= 0.0 and precision <= 1.0
    assert f1_score >= 0.0 and f1_score <= 1.0
    assert auroc >= 0.0 and auroc <= 1.0


@withPackage('torchmetrics>=0.10.0')
def test_perfect_groundtruth_metrics():
    pred_mask = target_mask = torch.rand(10)

    accuracy, recall, precision, f1_score, auroc = groundtruth_metrics(
        pred_mask, target_mask)

    assert round(accuracy, 6) == 1.0
    assert round(recall, 6) == 1.0
    assert round(precision, 6) == 1.0
    assert round(f1_score, 6) == 1.0
    assert round(auroc, 6) == 1.0


@withPackage('torchmetrics>=0.10.0')
def test_groundtruth_true_negative():
    warnings.filterwarnings('ignore', '.*No positive samples in targets.*')
    pred_mask = target_mask = torch.zeros(10)

    accuracy, recall, precision, f1_score, auroc = groundtruth_metrics(
        pred_mask, target_mask)

    assert round(accuracy, 6) == 1.0
    assert round(recall, 6) == 0.0
    assert round(precision, 6) == 0.0
    assert round(f1_score, 6) == 0.0
    assert round(auroc, 6) == 0.0
pytorch/test/explain/metric/test_faithfulness.py000066400000000000000000000033761444262304300225670ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import (
    DummyExplainer,
    Explainer,
    ModelConfig,
    unfaithfulness,
)


class DummyModel(torch.nn.Module):
    def __init__(self, model_config: ModelConfig):
        super().__init__()
        self.model_config = model_config

    def forward(self, x, edge_index):
        if self.model_config.return_type.value == 'probs':
            x = x.softmax(dim=-1)
        elif self.model_config.return_type.value == 'log_probs':
            x = x.log_softmax(dim=-1)
        return x


@pytest.mark.parametrize('top_k', [None, 2])
@pytest.mark.parametrize('explanation_type', ['model', 'phenomenon'])
@pytest.mark.parametrize('node_mask_type', ['common_attributes', 'attributes'])
@pytest.mark.parametrize('return_type', ['raw', 'probs', 'log_probs'])
def test_unfaithfulness(top_k, explanation_type, node_mask_type, return_type):
    x = torch.randn(8, 4)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
    ])

    model_config = ModelConfig(
        mode='multiclass_classification',
        task_level='node',
        return_type=return_type,
    )

    explainer = Explainer(
        DummyModel(model_config),
        algorithm=DummyExplainer(),
        explanation_type=explanation_type,
        node_mask_type=node_mask_type,
        edge_mask_type='object',
        model_config=model_config,
    )

    target = None
    if explanation_type == 'phenomenon':
        target = torch.randint(0, x.size(1), (x.size(0), ))

    explanation = explainer(x, edge_index, target=target,
                            index=torch.arange(4))

    metric = unfaithfulness(explainer, explanation, top_k)
    assert metric >= 0. and metric <= 1.
pytorch/test/explain/metric/test_fidelity.py000066400000000000000000000040351444262304300216760ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import (
    DummyExplainer,
    Explainer,
    characterization_score,
    fidelity,
    fidelity_curve_auc,
)


class DummyModel(torch.nn.Module):
    def forward(self, x, edge_index):
        return x


@pytest.mark.parametrize('explanation_type', ['model', 'phenomenon'])
def test_fidelity(explanation_type):
    x = torch.randn(8, 4)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6],
    ])

    explainer = Explainer(
        DummyModel(),
        algorithm=DummyExplainer(),
        explanation_type=explanation_type,
        node_mask_type='object',
        edge_mask_type='object',
        model_config=dict(
            mode='multiclass_classification',
            return_type='raw',
            task_level='node',
        ),
    )

    target = None
    if explanation_type == 'phenomenon':
        target = torch.randint(0, x.size(1), (x.size(0), ))

    explanation = explainer(x, edge_index, target=target,
                            index=torch.arange(4))

    pos_fidelity, neg_fidelity = fidelity(explainer, explanation)
    assert pos_fidelity == 0.0 and neg_fidelity == 0.0


def test_characterization_score():
    out = characterization_score(
        pos_fidelity=torch.tensor([1.0, 0.6, 0.5, 1.0]),
        neg_fidelity=torch.tensor([0.0, 0.2, 0.5, 1.0]),
        pos_weight=0.2,
        neg_weight=0.8,
    )
    assert out.tolist() == [1.0, 0.75, 0.5, 0.0]


def test_fidelity_curve_auc():
    pos_fidelity = torch.tensor([1.0, 1.0, 0.5, 1.0])
    neg_fidelity = torch.tensor([0.0, 0.5, 0.5, 0.9])

    x = torch.tensor([0, 1, 2, 3])
    out = round(float(fidelity_curve_auc(pos_fidelity, neg_fidelity, x)), 4)
    assert out == 8.5

    x = torch.tensor([10, 11, 12, 13])
    out = round(float(fidelity_curve_auc(pos_fidelity, neg_fidelity, x)), 4)
    assert out == 8.5

    x = torch.tensor([0, 1, 2, 5])
    out = round(float(fidelity_curve_auc(pos_fidelity, neg_fidelity, x)), 4)
    assert out == 19.5
pytorch/test/explain/test_explain_config.py000066400000000000000000000026771444262304300216010ustar00rootroot00000000000000import pytest

from torch_geometric.explain.config import ExplainerConfig, ThresholdConfig


@pytest.mark.parametrize('threshold_pairs', [
    ('hard', 0.5, True),
    ('hard', 1.1, False),
    ('hard', -1, False),
    ('topk', 1, True),
    ('topk', 0, False),
    ('topk', -1, False),
    ('topk', 0.5, False),
    ('invalid', None, False),
    ('hard', None, False),
])
def test_threshold_config(threshold_pairs):
    threshold_type, threshold_value, valid = threshold_pairs
    if valid:
        threshold = ThresholdConfig(threshold_type, threshold_value)
        assert threshold.type.value == threshold_type
        assert threshold.value == threshold_value
    else:
        with pytest.raises(ValueError):
            ThresholdConfig(threshold_type, threshold_value)


@pytest.mark.parametrize('explanation_type', [
    'model',
    'phenomenon',
    'invalid',
])
@pytest.mark.parametrize('mask_type', [
    None,
    'object',
    'common_attributes',
    'attributes',
    'invalid',
])
def test_configuration_config(explanation_type, mask_type):
    if (explanation_type != 'invalid' and mask_type is not None
            and mask_type != 'invalid'):
        config = ExplainerConfig(explanation_type, mask_type, None)
        assert config.explanation_type.value == explanation_type
        assert config.node_mask_type.value == mask_type
    else:
        with pytest.raises(ValueError):
            ExplainerConfig(explanation_type, mask_type, mask_type)
pytorch/test/explain/test_explainer.py000066400000000000000000000100251444262304300205650ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import DummyExplainer, Explainer, Explanation
from torch_geometric.explain.config import ExplanationType


class DummyModel(torch.nn.Module):
    def forward(self, x, edge_index):
        return x.mean().view(-1)


def test_get_prediction(data):
    model = DummyModel()
    assert model.training

    explainer = Explainer(
        model,
        algorithm=DummyExplainer(),
        explanation_type='phenomenon',
        node_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
    )
    pred = explainer.get_prediction(data.x, data.edge_index)
    assert model.training
    assert pred.size() == (1, )


@pytest.mark.parametrize('target', [None, torch.randn(2)])
@pytest.mark.parametrize('explanation_type', [x for x in ExplanationType])
def test_forward(data, target, explanation_type):
    model = DummyModel()
    assert model.training

    explainer = Explainer(
        model,
        algorithm=DummyExplainer(),
        explanation_type=explanation_type,
        node_mask_type='attributes',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
    )

    if target is None and explanation_type == ExplanationType.phenomenon:
        with pytest.raises(ValueError):
            explainer(data.x, data.edge_index, target=target)
    else:
        explanation = explainer(
            data.x,
            data.edge_index,
            target=target
            if explanation_type == ExplanationType.phenomenon else None,
        )
        assert model.training
        assert isinstance(explanation, Explanation)
        assert 'x' in explanation
        assert 'edge_index' in explanation
        assert 'target' in explanation
        assert 'node_mask' in explanation.available_explanations
        assert explanation.node_mask.size() == data.x.size()


@pytest.mark.parametrize('threshold_value', [0.2, 0.5, 0.8])
@pytest.mark.parametrize('node_mask_type', ['object', 'attributes'])
def test_hard_threshold(data, threshold_value, node_mask_type):
    explainer = Explainer(
        DummyModel(),
        algorithm=DummyExplainer(),
        explanation_type='model',
        node_mask_type=node_mask_type,
        edge_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
        threshold_config=('hard', threshold_value),
    )
    explanation = explainer(data.x, data.edge_index)

    assert 'node_mask' in explanation.available_explanations
    assert 'edge_mask' in explanation.available_explanations

    for key in explanation.available_explanations:
        mask = explanation[key]
        assert set(mask.unique().tolist()).issubset({0, 1})


@pytest.mark.parametrize('threshold_value', [1, 5, 10])
@pytest.mark.parametrize('threshold_type', ['topk', 'topk_hard'])
@pytest.mark.parametrize('node_mask_type', ['object', 'attributes'])
def test_topk_threshold(data, threshold_value, threshold_type, node_mask_type):
    explainer = Explainer(
        DummyModel(),
        algorithm=DummyExplainer(),
        explanation_type='model',
        node_mask_type=node_mask_type,
        edge_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
        threshold_config=(threshold_type, threshold_value),
    )
    explanation = explainer(data.x, data.edge_index)

    assert 'node_mask' in explanation.available_explanations
    assert 'edge_mask' in explanation.available_explanations

    for key in explanation.available_explanations:
        mask = explanation[key]
        if threshold_type == 'topk':
            assert (mask > 0).sum() == min(mask.numel(), threshold_value)
            assert ((mask == 0).sum() == mask.numel() -
                    min(mask.numel(), threshold_value))
        else:
            assert (mask == 1).sum() == min(mask.numel(), threshold_value)
            assert ((mask == 0).sum() == mask.numel() -
                    min(mask.numel(), threshold_value))
pytorch/test/explain/test_explanation.py000066400000000000000000000115641444262304300211310ustar00rootroot00000000000000import os.path as osp
from typing import Optional, Union

import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.explain import Explanation
from torch_geometric.explain.config import MaskType
from torch_geometric.testing import withPackage


def create_random_explanation(
    data: Data,
    node_mask_type: Optional[Union[MaskType, str]] = None,
    edge_mask_type: Optional[Union[MaskType, str]] = None,
):
    if node_mask_type is not None:
        node_mask_type = MaskType(node_mask_type)
    if edge_mask_type is not None:
        edge_mask_type = MaskType(edge_mask_type)

    if node_mask_type == MaskType.object:
        node_mask = torch.rand(data.x.size(0), 1)
    elif node_mask_type == MaskType.common_attributes:
        node_mask = torch.rand(1, data.x.size(1))
    elif node_mask_type == MaskType.attributes:
        node_mask = torch.rand_like(data.x)
    else:
        node_mask = None

    if edge_mask_type == MaskType.object:
        edge_mask = torch.rand(data.edge_index.size(1))
    else:
        edge_mask = None

    return Explanation(  # Create explanation.
        node_mask=node_mask,
        edge_mask=edge_mask,
        x=data.x,
        edge_index=data.edge_index,
        edge_attr=data.edge_attr,
    )


@pytest.mark.parametrize('node_mask_type',
                         [None, 'object', 'common_attributes', 'attributes'])
@pytest.mark.parametrize('edge_mask_type', [None, 'object'])
def test_available_explanations(data, node_mask_type, edge_mask_type):
    expected = []
    if node_mask_type is not None:
        expected.append('node_mask')
    if edge_mask_type is not None:
        expected.append('edge_mask')

    explanation = create_random_explanation(
        data,
        node_mask_type=node_mask_type,
        edge_mask_type=edge_mask_type,
    )

    assert set(explanation.available_explanations) == set(expected)


def test_validate_explanation(data):
    explanation = create_random_explanation(data)
    explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 5 nodes"):
        explanation = create_random_explanation(data, node_mask_type='object')
        explanation.x = torch.randn(5, 5)
        explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 4 features"):
        explanation = create_random_explanation(data, 'attributes')
        explanation.x = torch.randn(4, 4)
        explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 7 edges"):
        explanation = create_random_explanation(data, edge_mask_type='object')
        explanation.edge_index = torch.randint(0, 4, (2, 7))
        explanation.validate(raise_on_error=True)


def test_node_mask(data):
    node_mask = torch.tensor([[1.], [0.], [1.], [0.]])

    explanation = Explanation(
        node_mask=node_mask,
        x=data.x,
        edge_index=data.edge_index,
        edge_attr=data.edge_attr,
    )
    explanation.validate(raise_on_error=True)

    out = explanation.get_explanation_subgraph()
    assert out.node_mask.size() == (2, 1)
    assert (out.node_mask > 0.0).sum() == 2
    assert out.x.size() == (2, 3)
    assert out.edge_index.size(1) <= 6
    assert out.edge_index.size(1) == out.edge_attr.size(0)

    out = explanation.get_complement_subgraph()
    assert out.node_mask.size() == (2, 1)
    assert (out.node_mask == 0.0).sum() == 2
    assert out.x.size() == (2, 3)
    assert out.edge_index.size(1) <= 6
    assert out.edge_index.size(1) == out.edge_attr.size(0)


def test_edge_mask(data):
    edge_mask = torch.tensor([1., 0., 1., 0., 0., 1.])

    explanation = Explanation(
        edge_mask=edge_mask,
        x=data.x,
        edge_index=data.edge_index,
        edge_attr=data.edge_attr,
    )
    explanation.validate(raise_on_error=True)

    out = explanation.get_explanation_subgraph()
    assert out.x.size() == (4, 3)
    assert out.edge_mask.size() == (3, )
    assert (out.edge_mask > 0.0).sum() == 3
    assert out.edge_index.size() == (2, 3)
    assert out.edge_attr.size() == (3, 3)

    out = explanation.get_complement_subgraph()
    assert out.x.size() == (4, 3)
    assert out.edge_mask.size() == (3, )
    assert (out.edge_mask == 0.0).sum() == 3
    assert out.edge_index.size() == (2, 3)
    assert out.edge_attr.size() == (3, 3)


@withPackage('matplotlib', 'pandas')
@pytest.mark.parametrize('top_k', [2, None])
@pytest.mark.parametrize('node_mask_type', [None, 'attributes'])
def test_visualize_feature_importance(tmp_path, data, top_k, node_mask_type):
    explanation = create_random_explanation(data, node_mask_type)

    path = osp.join(tmp_path, 'feature_importance.png')

    if node_mask_type is None:
        with pytest.raises(ValueError, match="node_mask' is not"):
            explanation.visualize_feature_importance(path, top_k=top_k)
    else:
        explanation.visualize_feature_importance(path, top_k=top_k)
        assert osp.exists(path)
pytorch/test/explain/test_hetero_explainer.py000066400000000000000000000105651444262304300221440ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.explain import (
    DummyExplainer,
    Explainer,
    HeteroExplanation,
)
from torch_geometric.explain.config import ExplanationType


class DummyModel(torch.nn.Module):
    def forward(self, x_dict, edge_index_dict, *args) -> torch.Tensor:
        return x_dict['paper'].mean().view(-1)


def test_get_prediction(hetero_data):
    model = DummyModel()
    assert model.training

    explainer = Explainer(
        model,
        algorithm=DummyExplainer(),
        explanation_type='phenomenon',
        node_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
    )
    pred = explainer.get_prediction(hetero_data.x_dict,
                                    hetero_data.edge_index_dict)
    assert model.training
    assert pred.size() == (1, )


@pytest.mark.parametrize('target', [None, torch.randn(2)])
@pytest.mark.parametrize('explanation_type', [x for x in ExplanationType])
def test_forward(hetero_data, target, explanation_type):
    model = DummyModel()

    explainer = Explainer(
        model,
        algorithm=DummyExplainer(),
        explanation_type=explanation_type,
        node_mask_type='attributes',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
    )

    if target is None and explanation_type == ExplanationType.phenomenon:
        with pytest.raises(ValueError):
            explainer(hetero_data.x_dict, hetero_data.edge_index_dict,
                      target=target)
    else:
        explanation = explainer(
            hetero_data.x_dict,
            hetero_data.edge_index_dict,
            target=target
            if explanation_type == ExplanationType.phenomenon else None,
        )
        assert model.training
        assert isinstance(explanation, HeteroExplanation)
        assert 'node_mask' in explanation.available_explanations
        for key in explanation.node_types:
            expected_size = hetero_data[key].x.size()
            assert explanation[key].node_mask.size() == expected_size


@pytest.mark.parametrize('threshold_value', [0.2, 0.5, 0.8])
@pytest.mark.parametrize('node_mask_type', ['object', 'attributes'])
def test_hard_threshold(hetero_data, threshold_value, node_mask_type):

    explainer = Explainer(
        DummyModel(),
        algorithm=DummyExplainer(),
        explanation_type='model',
        node_mask_type=node_mask_type,
        edge_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
        threshold_config=('hard', threshold_value),
    )
    explanation = explainer(hetero_data.x_dict, hetero_data.edge_index_dict)
    assert 'node_mask' in explanation.available_explanations
    assert 'edge_mask' in explanation.available_explanations

    for key in explanation.available_explanations:
        for mask in explanation.collect(key).values():
            assert set(mask.unique().tolist()).issubset({0, 1})


@pytest.mark.parametrize('threshold_value', [1, 5, 10])
@pytest.mark.parametrize('threshold_type', ['topk', 'topk_hard'])
@pytest.mark.parametrize('node_mask_type', ['object', 'attributes'])
def test_topk_threshold(hetero_data, threshold_value, threshold_type,
                        node_mask_type):
    explainer = Explainer(
        DummyModel(),
        algorithm=DummyExplainer(),
        explanation_type='model',
        node_mask_type=node_mask_type,
        edge_mask_type='object',
        model_config=dict(
            mode='regression',
            task_level='graph',
        ),
        threshold_config=(threshold_type, threshold_value),
    )
    explanation = explainer(hetero_data.x_dict, hetero_data.edge_index_dict)

    assert 'node_mask' in explanation.available_explanations
    assert 'edge_mask' in explanation.available_explanations

    for key in explanation.available_explanations:
        for mask in explanation.collect(key).values():
            if threshold_type == 'topk':
                assert (mask > 0).sum() == min(mask.numel(), threshold_value)
                assert ((mask == 0).sum() == mask.numel() -
                        min(mask.numel(), threshold_value))
            else:
                assert (mask == 1).sum() == min(mask.numel(), threshold_value)
                assert ((mask == 0).sum() == mask.numel() -
                        min(mask.numel(), threshold_value))
pytorch/test/explain/test_hetero_explanation.py000066400000000000000000000117501444262304300224740ustar00rootroot00000000000000import os.path as osp
from typing import Optional, Union

import pytest
import torch

from torch_geometric.data import HeteroData
from torch_geometric.explain import HeteroExplanation
from torch_geometric.explain.config import MaskType
from torch_geometric.testing import withPackage


def create_random_explanation(
    hetero_data: HeteroData,
    node_mask_type: Optional[Union[MaskType, str]] = None,
    edge_mask_type: Optional[Union[MaskType, str]] = None,
):
    if node_mask_type is not None:
        node_mask_type = MaskType(node_mask_type)
    if edge_mask_type is not None:
        edge_mask_type = MaskType(edge_mask_type)

    out = HeteroExplanation()

    for key in ['paper', 'author']:
        out[key].x = hetero_data[key].x
        if node_mask_type == MaskType.object:
            out[key].node_mask = torch.rand(hetero_data[key].num_nodes, 1)
        elif node_mask_type == MaskType.common_attributes:
            out[key].node_mask = torch.rand(1, hetero_data[key].num_features)
        elif node_mask_type == MaskType.attributes:
            out[key].node_mask = torch.rand_like(hetero_data[key].x)

    for key in [('paper', 'paper'), ('paper', 'author')]:
        out[key].edge_index = hetero_data[key].edge_index
        out[key].edge_attr = hetero_data[key].edge_attr
        if edge_mask_type == MaskType.object:
            out[key].edge_mask = torch.rand(hetero_data[key].num_edges)

    return out


@pytest.mark.parametrize('node_mask_type',
                         [None, 'object', 'common_attributes', 'attributes'])
@pytest.mark.parametrize('edge_mask_type', [None, 'object'])
def test_available_explanations(hetero_data, node_mask_type, edge_mask_type):
    expected = []
    if node_mask_type:
        expected.append('node_mask')
    if edge_mask_type:
        expected.append('edge_mask')

    explanation = create_random_explanation(
        hetero_data,
        node_mask_type=node_mask_type,
        edge_mask_type=edge_mask_type,
    )

    assert set(explanation.available_explanations) == set(expected)


def test_validate_explanation(hetero_data):
    explanation = create_random_explanation(hetero_data)
    explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 8 nodes"):
        explanation = create_random_explanation(hetero_data)
        explanation['paper'].node_mask = torch.rand(5, 5)
        explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 5 features"):
        explanation = create_random_explanation(hetero_data, 'attributes')
        explanation['paper'].x = torch.randn(8, 5)
        explanation.validate(raise_on_error=True)

    with pytest.raises(ValueError, match="with 10 edges"):
        explanation = create_random_explanation(hetero_data)
        explanation['paper', 'paper'].edge_mask = torch.randn(5)
        explanation.validate(raise_on_error=True)


def test_node_mask():
    explanation = HeteroExplanation()
    explanation['paper'].node_mask = torch.tensor([[1.], [0.], [1.], [1.]])
    explanation['author'].node_mask = torch.tensor([[1.], [0.], [1.], [1.]])
    explanation.validate(raise_on_error=True)

    out = explanation.get_explanation_subgraph()
    assert out['paper'].node_mask.size() == (3, 1)
    assert out['author'].node_mask.size() == (3, 1)

    out = explanation.get_complement_subgraph()
    assert out['paper'].node_mask.size() == (1, 1)
    assert out['author'].node_mask.size() == (1, 1)


def test_edge_mask():
    explanation = HeteroExplanation()
    explanation['paper'].num_nodes = 4
    explanation['author'].num_nodes = 4
    explanation['paper', 'author'].edge_index = torch.tensor([
        [0, 1, 2, 3],
        [0, 1, 2, 3],
    ])
    explanation['paper', 'author'].edge_mask = torch.tensor([1., 0., 1., 1.])

    out = explanation.get_explanation_subgraph()
    assert out['paper'].num_nodes == 4
    assert out['author'].num_nodes == 4
    assert out['paper', 'author'].edge_mask.size() == (3, )
    assert torch.equal(out['paper', 'author'].edge_index,
                       torch.tensor([[0, 2, 3], [0, 2, 3]]))

    out = explanation.get_complement_subgraph()
    assert out['paper'].num_nodes == 4
    assert out['author'].num_nodes == 4
    assert out['paper', 'author'].edge_mask.size() == (1, )
    assert torch.equal(out['paper', 'author'].edge_index,
                       torch.tensor([[1], [1]]))


@withPackage('matplotlib')
@pytest.mark.parametrize('top_k', [2, None])
@pytest.mark.parametrize('node_mask_type', [None, 'attributes'])
def test_visualize_feature_importance(
    top_k,
    node_mask_type,
    tmp_path,
    hetero_data,
):
    explanation = create_random_explanation(
        hetero_data,
        node_mask_type=node_mask_type,
    )

    path = osp.join(tmp_path, 'feature_importance.png')

    if node_mask_type is None:
        with pytest.raises(ValueError, match="node_mask' is not"):
            explanation.visualize_feature_importance(path, top_k=top_k)
    else:
        explanation.visualize_feature_importance(path, top_k=top_k)
        assert osp.exists(path)
pytorch/test/graphgym/000077500000000000000000000000001444262304300153455ustar00rootroot00000000000000pytorch/test/graphgym/example_node.yml000066400000000000000000000012661444262304300205350ustar00rootroot00000000000000tensorboard_each_run: false
tensorboard_agg: false
dataset:
  format: PyG
  name: Cora
  task: node
  task_type: classification
  node_encoder: false
  node_encoder_name: Atom
  edge_encoder: false
  edge_encoder_name: Bond
train:
  batch_size: 128
  eval_period: 2
  ckpt_period: 100
  enable_ckpt: false
  skip_train_eval: true
  sampler: full_batch
model:
  type: gnn
  loss_fun: cross_entropy
  edge_decoding: dot
  graph_pooling: add
gnn:
  layers_pre_mp: 2
  layers_mp: 2
  layers_post_mp: 1
  dim_inner: 16
  layer_type: gcnconv
  stage_type: stack
  batchnorm: false
  act: prelu
  dropout: 0.1
  agg: mean
  normalize_adj: false
optim:
  optimizer: adam
  base_lr: 0.01
  max_epoch: 6
pytorch/test/graphgym/test_config.py000066400000000000000000000011451444262304300202240ustar00rootroot00000000000000from dataclasses import dataclass

from torch_geometric.graphgym.config import from_config


@dataclass
class MyConfig:
    a: int
    b: int = 4


def my_func(a: int, b: int = 2) -> str:
    return f'a={a},b={b}'


def test_from_config():
    assert my_func(a=1) == 'a=1,b=2'

    assert my_func.__name__ == from_config(my_func).__name__
    assert from_config(my_func)(cfg=MyConfig(a=1)) == 'a=1,b=4'
    assert from_config(my_func)(cfg=MyConfig(a=1, b=1)) == 'a=1,b=1'
    assert from_config(my_func)(2, cfg=MyConfig(a=1, b=3)) == 'a=2,b=3'
    assert from_config(my_func)(cfg=MyConfig(a=1), b=3) == 'a=1,b=3'
pytorch/test/graphgym/test_graphgym.py000066400000000000000000000142141444262304300205760ustar00rootroot00000000000000import os.path as osp
import warnings
from collections import namedtuple

import pytest
import torch

from torch_geometric import seed_everything
from torch_geometric.graphgym import register
from torch_geometric.graphgym.checkpoint import get_ckpt_dir
from torch_geometric.graphgym.config import (
    cfg,
    dump_cfg,
    load_cfg,
    set_out_dir,
    set_run_dir,
)
from torch_geometric.graphgym.loader import create_loader
from torch_geometric.graphgym.logger import LoggerCallback, set_printing
from torch_geometric.graphgym.model_builder import create_model
from torch_geometric.graphgym.models.gnn import FeatureEncoder, GNNStackStage
from torch_geometric.graphgym.models.head import GNNNodeHead
from torch_geometric.graphgym.train import GraphGymDataModule, train
from torch_geometric.graphgym.utils import (
    agg_runs,
    auto_select_device,
    params_count,
)
from torch_geometric.testing import withPackage

num_trivial_metric_calls = 0

Args = namedtuple('Args', ['cfg_file', 'opts'])
root = osp.join(osp.dirname(osp.realpath(__file__)))
args = Args(osp.join(root, 'example_node.yml'), [])


def trivial_metric(true, pred, task_type):
    global num_trivial_metric_calls
    num_trivial_metric_calls += 1
    return 1


@withPackage('yacs')
@withPackage('pytorch_lightning')
@pytest.mark.parametrize('auto_resume', [True, False])
@pytest.mark.parametrize('skip_train_eval', [True, False])
@pytest.mark.parametrize('use_trivial_metric', [True, False])
def test_run_single_graphgym(tmp_path, capfd, auto_resume, skip_train_eval,
                             use_trivial_metric):
    warnings.filterwarnings('ignore', ".*does not have many workers.*")
    warnings.filterwarnings('ignore', ".*lower value for log_every_n_steps.*")

    load_cfg(cfg, args)
    cfg.out_dir = osp.join(tmp_path, 'out_dir')
    cfg.run_dir = osp.join(tmp_path, 'run_dir')
    cfg.dataset.dir = osp.join('/', 'tmp', 'pyg_test_datasets', 'Planetoid')
    cfg.train.auto_resume = auto_resume

    set_out_dir(cfg.out_dir, args.cfg_file)
    dump_cfg(cfg)
    set_printing()

    seed_everything(cfg.seed)
    auto_select_device()
    set_run_dir(cfg.out_dir)

    cfg.train.skip_train_eval = skip_train_eval
    cfg.train.enable_ckpt = use_trivial_metric and skip_train_eval
    if use_trivial_metric:
        if 'trivial' not in register.metric_dict:
            register.register_metric('trivial', trivial_metric)
        global num_trivial_metric_calls
        num_trivial_metric_calls = 0
        cfg.metric_best = 'trivial'
        cfg.custom_metrics = ['trivial']
    else:
        cfg.metric_best = 'auto'
        cfg.custom_metrics = []

    datamodule = GraphGymDataModule()
    assert len(datamodule.loaders) == 3

    model = create_model()
    assert isinstance(model, torch.nn.Module)
    assert isinstance(model.encoder, FeatureEncoder)
    assert isinstance(model.mp, GNNStackStage)
    assert isinstance(model.post_mp, GNNNodeHead)
    assert len(list(model.pre_mp.children())) == cfg.gnn.layers_pre_mp

    optimizer, scheduler = model.configure_optimizers()
    assert isinstance(optimizer[0], torch.optim.Adam)
    assert isinstance(scheduler[0], torch.optim.lr_scheduler.CosineAnnealingLR)

    cfg.params = params_count(model)
    assert cfg.params == 23883

    train(model, datamodule, logger=True,
          trainer_config={"enable_progress_bar": False})

    assert osp.isdir(get_ckpt_dir()) is cfg.train.enable_ckpt

    agg_runs(cfg.out_dir, cfg.metric_best)

    out, _ = capfd.readouterr()
    assert "train: {'epoch': 0," in out
    assert "val: {'epoch': 0," in out
    assert "train: {'epoch': 5," in out
    assert "val: {'epoch': 5," in out


@withPackage('yacs')
@withPackage('pytorch_lightning')
def test_graphgym_module(tmp_path):
    import pytorch_lightning as pl

    load_cfg(cfg, args)
    cfg.out_dir = osp.join(tmp_path, 'out_dir')
    cfg.run_dir = osp.join(tmp_path, 'run_dir')
    cfg.dataset.dir = osp.join('/', 'tmp', 'pyg_test_datasets', 'Planetoid')

    set_out_dir(cfg.out_dir, args.cfg_file)
    dump_cfg(cfg)
    set_printing()

    seed_everything(cfg.seed)
    auto_select_device()
    set_run_dir(cfg.out_dir)

    loaders = create_loader()
    assert len(loaders) == 3

    model = create_model()
    assert isinstance(model, pl.LightningModule)

    optimizer, scheduler = model.configure_optimizers()
    assert isinstance(optimizer[0], torch.optim.Adam)
    assert isinstance(scheduler[0], torch.optim.lr_scheduler.CosineAnnealingLR)

    cfg.params = params_count(model)
    assert cfg.params == 23883

    keys = {"loss", "true", "pred_score", "step_end_time"}
    # test training step
    batch = next(iter(loaders[0]))
    batch.to(model.device)
    outputs = model.training_step(batch)
    assert keys == set(outputs.keys())
    assert isinstance(outputs["loss"], torch.Tensor)

    # test validation step
    batch = next(iter(loaders[1]))
    batch.to(model.device)
    outputs = model.validation_step(batch)
    assert keys == set(outputs.keys())
    assert isinstance(outputs["loss"], torch.Tensor)

    # test test step
    batch = next(iter(loaders[2]))
    batch.to(model.device)
    outputs = model.test_step(batch)
    assert keys == set(outputs.keys())
    assert isinstance(outputs["loss"], torch.Tensor)


@withPackage('yacs')
@withPackage('pytorch_lightning')
def test_train(tmp_path, capfd):
    warnings.filterwarnings('ignore', ".*does not have many workers.*")

    import pytorch_lightning as pl

    load_cfg(cfg, args)
    cfg.out_dir = osp.join(tmp_path, 'out_dir')
    cfg.run_dir = osp.join(tmp_path, 'run_dir')
    cfg.dataset.dir = osp.join('/', 'tmp', 'pyg_test_datasets', 'Planetoid')

    set_out_dir(cfg.out_dir, args.cfg_file)
    dump_cfg(cfg)
    set_printing()

    seed_everything(cfg.seed)
    auto_select_device()
    set_run_dir(cfg.out_dir)

    loaders = create_loader()
    model = create_model()
    cfg.params = params_count(model)
    logger = LoggerCallback()
    trainer = pl.Trainer(max_epochs=1, max_steps=4, callbacks=logger,
                         log_every_n_steps=1)
    train_loader, val_loader = loaders[0], loaders[1]
    trainer.fit(model, train_loader, val_loader)

    out, err = capfd.readouterr()
    assert 'Sanity Checking' in out
    assert 'Epoch 0:' in out
pytorch/test/graphgym/test_logger.py000066400000000000000000000005741444262304300202430ustar00rootroot00000000000000from torch_geometric.graphgym.logger import Logger, LoggerCallback
from torch_geometric.testing import withPackage


@withPackage('yacs')
@withPackage('pytorch_lightning')
def test_logger_callback():
    logger = LoggerCallback()
    assert isinstance(logger.train_logger, Logger)
    assert isinstance(logger.val_logger, Logger)
    assert isinstance(logger.test_logger, Logger)
pytorch/test/graphgym/test_register.py000066400000000000000000000012121444262304300205760ustar00rootroot00000000000000import torch

import torch_geometric.graphgym.register as register
from torch_geometric.testing import withPackage


@register.register_act('identity')
def identity_act(x: torch.Tensor) -> torch.Tensor:
    return x


@withPackage('yacs')
def test_register():
    assert len(register.act_dict) == 8
    assert list(register.act_dict.keys()) == [
        'relu', 'selu', 'prelu', 'elu', 'lrelu_01', 'lrelu_025', 'lrelu_05',
        'identity'
    ]
    assert str(register.act_dict['relu']()) == 'ReLU()'

    register.register_act('lrelu_03', torch.nn.LeakyReLU(0.3))
    assert len(register.act_dict) == 9
    assert 'lrelu_03' in register.act_dict
pytorch/test/io/000077500000000000000000000000001444262304300141365ustar00rootroot00000000000000pytorch/test/io/example1.off000066400000000000000000000001121444262304300163400ustar00rootroot00000000000000OFF
4 2 0
0.0 0.0 0.0
0.0 1.0 0.0
1.0 0.0 0.0
1.0 1.0 0.0
3 0 1 2
3 1 2 3
pytorch/test/io/example2.off000066400000000000000000000001041444262304300163420ustar00rootroot00000000000000OFF
4 1 0
0.0 0.0 0.0
0.0 1.0 0.0
1.0 0.0 0.0
1.0 1.0 0.0
4 0 1 2 3
pytorch/test/io/test_off.py000066400000000000000000000021171444262304300163220ustar00rootroot00000000000000import os
import os.path as osp
import random
import sys

import torch

from torch_geometric.data import Data
from torch_geometric.io import read_off, write_off


def test_read_off():
    root_dir = osp.join(osp.dirname(osp.realpath(__file__)))

    data = read_off(osp.join(root_dir, 'example1.off'))
    assert len(data) == 2
    assert data.pos.tolist() == [[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 0]]
    assert data.face.tolist() == [[0, 1], [1, 2], [2, 3]]

    data = read_off(osp.join(root_dir, 'example2.off'))
    assert len(data) == 2
    assert data.pos.tolist() == [[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 0]]
    assert data.face.tolist() == [[0, 0], [1, 2], [2, 3]]


def test_write_off():
    pos = torch.tensor([[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 0]])
    face = torch.tensor([[0, 1], [1, 2], [2, 3]])

    name = str(random.randrange(sys.maxsize))
    path = osp.join('/', 'tmp', f'{name}.off')
    write_off(Data(pos=pos, face=face), path)
    data = read_off(path)
    os.unlink(path)

    assert data.pos.tolist() == pos.tolist()
    assert data.face.tolist() == face.tolist()
pytorch/test/loader/000077500000000000000000000000001444262304300147755ustar00rootroot00000000000000pytorch/test/loader/test_cluster.py000066400000000000000000000143221444262304300200710ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.loader import ClusterData, ClusterLoader
from torch_geometric.testing import onlyFullTest
from torch_geometric.utils import sort_edge_index

try:
    # TODO Using `pyg-lib` metis partitioning leads to some weird bugs in the
    # CI. As such, we require `torch-sparse` for these tests for now.
    rowptr = torch.tensor([0, 1])
    col = torch.tensor([0])
    torch.ops.torch_sparse.partition(rowptr, col, None, 1, True)
    WITH_METIS = True
except (AttributeError, RuntimeError):
    WITH_METIS = False


@pytest.mark.skipif(not WITH_METIS, reason='Not compiled with METIS support')
def test_cluster_gcn():
    adj = torch.tensor([
        [1, 1, 1, 0, 1, 0],
        [1, 1, 0, 1, 0, 1],
        [1, 0, 1, 0, 1, 0],
        [0, 1, 0, 1, 0, 1],
        [1, 0, 1, 0, 1, 0],
        [0, 1, 0, 1, 0, 1],
    ])

    x = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
    edge_index = adj.nonzero(as_tuple=False).t()
    edge_attr = torch.arange(edge_index.size(1))
    n_id = torch.arange(6)
    data = Data(x=x, n_id=n_id, edge_index=edge_index, edge_attr=edge_attr)
    data.num_nodes = 6

    cluster_data = ClusterData(data, num_parts=2, log=False)

    partition = cluster_data._partition(
        edge_index, cluster=torch.tensor([0, 1, 0, 1, 0, 1]))
    assert partition.partptr.tolist() == [0, 3, 6]
    assert partition.node_perm.tolist() == [0, 2, 4, 1, 3, 5]
    assert partition.edge_perm.tolist() == [
        0, 2, 3, 1, 8, 9, 10, 14, 15, 16, 4, 5, 6, 7, 11, 12, 13, 17, 18, 19
    ]

    assert cluster_data.partition.partptr.tolist() == [0, 3, 6]
    assert torch.equal(
        cluster_data.partition.node_perm.sort()[0],
        torch.arange(data.num_nodes),
    )
    assert torch.equal(
        cluster_data.partition.edge_perm.sort()[0],
        torch.arange(data.num_edges),
    )

    out = cluster_data[0]
    expected = data.subgraph(out.n_id)
    out.validate()
    assert out.num_nodes == 3
    assert out.n_id.size() == (3, )
    assert torch.equal(out.x, expected.x)
    tmp = sort_edge_index(expected.edge_index, expected.edge_attr)
    assert torch.equal(out.edge_index, tmp[0])
    assert torch.equal(out.edge_attr, tmp[1])

    out = cluster_data[1]
    out.validate()
    assert out.num_nodes == 3
    assert out.n_id.size() == (3, )
    expected = data.subgraph(out.n_id)
    assert torch.equal(out.x, expected.x)
    tmp = sort_edge_index(expected.edge_index, expected.edge_attr)
    assert torch.equal(out.edge_index, tmp[0])
    assert torch.equal(out.edge_attr, tmp[1])

    loader = ClusterLoader(cluster_data, batch_size=1)
    iterator = iter(loader)

    out = next(iterator)
    out.validate()
    assert out.num_nodes == 3
    assert out.n_id.size() == (3, )
    expected = data.subgraph(out.n_id)
    assert torch.equal(out.x, expected.x)
    tmp = sort_edge_index(expected.edge_index, expected.edge_attr)
    assert torch.equal(out.edge_index, tmp[0])
    assert torch.equal(out.edge_attr, tmp[1])

    out = next(iterator)
    out.validate()
    assert out.num_nodes == 3
    assert out.n_id.size() == (3, )
    expected = data.subgraph(out.n_id)
    assert torch.equal(out.x, expected.x)
    tmp = sort_edge_index(expected.edge_index, expected.edge_attr)
    assert torch.equal(out.edge_index, tmp[0])
    assert torch.equal(out.edge_attr, tmp[1])

    loader = ClusterLoader(cluster_data, batch_size=2, shuffle=False)
    out = next(iter(loader))
    out.validate()
    assert out.num_nodes == 6
    assert out.n_id.size() == (6, )
    expected = data.subgraph(out.n_id)
    assert torch.equal(out.x, expected.x)
    tmp = sort_edge_index(expected.edge_index, expected.edge_attr)
    assert torch.equal(out.edge_index, tmp[0])
    assert torch.equal(out.edge_attr, tmp[1])


@pytest.mark.skipif(not WITH_METIS, reason='Not compiled with METIS support')
def test_keep_inter_cluster_edges():
    adj = torch.tensor([
        [1, 1, 1, 0, 1, 0],
        [1, 1, 0, 1, 0, 1],
        [1, 0, 1, 0, 1, 0],
        [0, 1, 0, 1, 0, 1],
        [1, 0, 1, 0, 1, 0],
        [0, 1, 0, 1, 0, 1],
    ])

    x = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
    edge_index = adj.nonzero(as_tuple=False).t()
    edge_attr = torch.arange(edge_index.size(1))
    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr)
    data.num_nodes = 6

    cluster_data = ClusterData(data, num_parts=2, log=False,
                               keep_inter_cluster_edges=True)

    data = cluster_data[0]
    assert data.edge_index[0].min() == 0
    assert data.edge_index[0].max() == 2
    assert data.edge_index[1].min() == 0
    assert data.edge_index[1].max() > 2
    assert data.edge_index.size(1) == data.edge_attr.size(0)

    data = cluster_data[1]
    assert data.edge_index[0].min() == 0
    assert data.edge_index[0].max() == 2
    assert data.edge_index[1].min() == 0
    assert data.edge_index[1].max() > 2
    assert data.edge_index.size(1) == data.edge_attr.size(0)


@onlyFullTest
@pytest.mark.skipif(not WITH_METIS, reason='Not compiled with METIS support')
def test_cluster_gcn_correctness(get_dataset):
    dataset = get_dataset('Cora')
    data = dataset[0].clone()
    data.n_id = torch.arange(data.num_nodes)
    cluster_data = ClusterData(data, num_parts=10, log=False)
    loader = ClusterLoader(cluster_data, batch_size=3, shuffle=False)

    for batch1 in loader:
        batch1.validate()
        batch2 = data.subgraph(batch1.n_id)
        assert batch1.num_nodes == batch2.num_nodes
        assert batch1.num_edges == batch2.num_edges
        assert torch.equal(batch1.x, batch2.x)
        assert torch.equal(
            batch1.edge_index,
            sort_edge_index(batch2.edge_index),
        )


if __name__ == '__main__':
    import argparse

    from ogb.nodeproppred import PygNodePropPredDataset
    from tqdm import tqdm

    parser = argparse.ArgumentParser()
    parser.add_argument('--num_workers', type=int, default=0)
    args = parser.parse_args()

    data = PygNodePropPredDataset('ogbn-products', root='/tmp/ogb')[0]

    loader = ClusterLoader(
        ClusterData(data, num_parts=15_000, save_dir='/tmp/ogb/ogbn_products'),
        batch_size=32,
        shuffle=True,
        num_workers=args.num_workers,
    )

    for batch in tqdm(loader):
        pass
pytorch/test/loader/test_dataloader.py000066400000000000000000000145261444262304300205160ustar00rootroot00000000000000import multiprocessing
import sys
from collections import namedtuple

import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.loader import DataLoader
from torch_geometric.testing import get_random_edge_index, withCUDA

with_mp = sys.platform not in ['win32']
num_workers_list = [0, 2] if with_mp else [0]

if sys.platform == 'darwin':
    multiprocessing.set_start_method('spawn')


@withCUDA
@pytest.mark.parametrize('num_workers', num_workers_list)
def test_dataloader(num_workers, device):
    if num_workers > 0 and device != torch.device('cpu'):
        return

    x = torch.Tensor([[1], [1], [1]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    face = torch.tensor([[0], [1], [2]])
    y = 2.
    z = torch.tensor(0.)
    name = 'data'

    data = Data(x=x, edge_index=edge_index, y=y, z=z, name=name).to(device)
    assert str(data) == ("Data(x=[3, 1], edge_index=[2, 4], y=2.0, z=0.0, "
                         "name='data')")
    data.face = face

    loader = DataLoader([data, data, data, data], batch_size=2, shuffle=False,
                        num_workers=num_workers)
    assert len(loader) == 2

    for batch in loader:
        assert batch.x.device == device
        assert batch.edge_index.device == device
        assert batch.z.device == device
        assert batch.num_graphs == len(batch) == 2
        assert batch.batch.tolist() == [0, 0, 0, 1, 1, 1]
        assert batch.ptr.tolist() == [0, 3, 6]
        assert batch.x.tolist() == [[1], [1], [1], [1], [1], [1]]
        assert batch.edge_index.tolist() == [[0, 1, 1, 2, 3, 4, 4, 5],
                                             [1, 0, 2, 1, 4, 3, 5, 4]]
        assert batch.y.tolist() == [2.0, 2.0]
        assert batch.z.tolist() == [0.0, 0.0]
        assert batch.name == ['data', 'data']
        assert batch.face.tolist() == [[0, 3], [1, 4], [2, 5]]

        for store in batch.stores:
            assert id(batch) == id(store._parent())

    loader = DataLoader([data, data, data, data], batch_size=2, shuffle=False,
                        follow_batch=['edge_index'], num_workers=num_workers,
                        collate_fn=None)
    assert len(loader) == 2

    for batch in loader:
        assert batch.num_graphs == len(batch) == 2
        assert batch.edge_index_batch.tolist() == [0, 0, 0, 0, 1, 1, 1, 1]


def test_dataloader_fallbacks():
    # Test inputs of type List[torch.Tensor]:
    data_list = [torch.ones(3) for _ in range(4)]
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch, torch.ones(4, 3))

    # Test inputs of type List[float]:
    data_list = [1.0, 1.0, 1.0, 1.0]
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch, torch.ones(4))

    # Test inputs of type List[int]:
    data_list = [1, 1, 1, 1]
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch, torch.ones(4, dtype=torch.long))

    # Test inputs of type List[str]:
    data_list = ['test'] * 4
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert batch == data_list

    # Test inputs of type List[Mapping]:
    data_list = [{'x': torch.ones(3), 'y': 1}] * 4
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch['x'], torch.ones(4, 3))
    assert torch.equal(batch['y'], torch.ones(4, dtype=torch.long))

    # Test inputs of type List[Tuple]:
    DataTuple = namedtuple('DataTuple', 'x y')
    data_list = [DataTuple(0.0, 1)] * 4
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch.x, torch.zeros(4))
    assert torch.equal(batch[1], torch.ones(4, dtype=torch.long))

    # Test inputs of type List[Sequence]:
    data_list = [[0.0, 1]] * 4
    batch = next(iter(DataLoader(data_list, batch_size=4)))
    assert torch.equal(batch[0], torch.zeros(4))
    assert torch.equal(batch[1], torch.ones(4, dtype=torch.long))

    # Test that inputs of unsupported types raise an error:
    class DummyClass:
        pass

    with pytest.raises(TypeError):
        data_list = [DummyClass()] * 4
        next(iter(DataLoader(data_list, batch_size=4)))


@pytest.mark.skipif(not with_mp, reason='Multi-processing not available')
def test_multiprocessing():
    queue = torch.multiprocessing.Manager().Queue()
    data = Data(x=torch.randn(5, 16))
    data_list = [data, data, data, data]
    loader = DataLoader(data_list, batch_size=2)
    for batch in loader:
        queue.put(batch)

    batch = queue.get()
    assert batch.num_graphs == len(batch) == 2

    batch = queue.get()
    assert batch.num_graphs == len(batch) == 2


def test_pin_memory():
    x = torch.randn(3, 16)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    data = Data(x=x, edge_index=edge_index)

    loader = DataLoader([data] * 4, batch_size=2, pin_memory=True)
    for batch in loader:
        assert batch.x.is_pinned() or not torch.cuda.is_available()
        assert batch.edge_index.is_pinned() or not torch.cuda.is_available()


@pytest.mark.parametrize('num_workers', num_workers_list)
def test_heterogeneous_dataloader(num_workers):
    data = HeteroData()
    data['p'].x = torch.randn(100, 128)
    data['a'].x = torch.randn(200, 128)
    data['p', 'a'].edge_index = get_random_edge_index(100, 200, 500)
    data['p'].edge_attr = torch.randn(500, 32)
    data['a', 'p'].edge_index = get_random_edge_index(200, 100, 400)
    data['a', 'p'].edge_attr = torch.randn(400, 32)

    loader = DataLoader([data, data, data, data], batch_size=2, shuffle=False,
                        num_workers=num_workers)
    assert len(loader) == 2

    for batch in loader:
        assert batch.num_graphs == len(batch) == 2
        assert batch.num_nodes == 600

        for store in batch.stores:
            assert id(batch) == id(store._parent())


if __name__ == '__main__':
    import argparse
    import time

    from torch_geometric.datasets import QM9

    parser = argparse.ArgumentParser()
    parser.add_argument('--num_workers', type=int, default=0)
    args = parser.parse_args()

    dataset = QM9('/tmp/QM9')
    loader = DataLoader(dataset, batch_size=128, shuffle=True,
                        num_workers=args.num_workers)

    for _ in range(2):
        print(f'Start loading {len(loader)} mini-batches ... ', end='')
        t = time.perf_counter()
        for batch in loader:
            pass
        print(f'Done! [{time.perf_counter() - t:.4f}s]')
pytorch/test/loader/test_dynamic_batch_sampler.py000066400000000000000000000026701444262304300227230ustar00rootroot00000000000000from typing import List

import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.loader import DataLoader, DynamicBatchSampler


def test_dataloader_with_dynamic_batches():
    data_list: List[Data] = []
    for num_nodes in range(100, 110):
        data_list.append(Data(num_nodes=num_nodes))

    torch.manual_seed(12345)
    batch_sampler = DynamicBatchSampler(data_list, 300, shuffle=True)
    loader = DataLoader(data_list, batch_sampler=batch_sampler)

    num_nodes_total = 0
    for data in loader:
        assert data.num_nodes <= 300
        num_nodes_total += data.num_nodes
    assert num_nodes_total == 1045

    # Test skipping
    data_list = [Data(num_nodes=400)] + data_list
    batch_sampler = DynamicBatchSampler(data_list, 300, skip_too_big=True,
                                        num_steps=2)
    loader = DataLoader(data_list, batch_sampler=batch_sampler)

    num_nodes_total = 0
    for data in loader:
        num_nodes_total += data.num_nodes
    assert num_nodes_total == 404

    # Test warning
    batch_sampler = DynamicBatchSampler(data_list, 300, skip_too_big=False,
                                        num_steps=2)
    loader = DataLoader(data_list, batch_sampler=batch_sampler)

    with pytest.warns(UserWarning, match="is larger than 300 nodes"):
        num_nodes_total = 0
        for data in loader:
            num_nodes_total += data.num_nodes
        assert num_nodes_total == 601
pytorch/test/loader/test_graph_saint.py000066400000000000000000000061731444262304300207140ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.loader import (
    GraphSAINTEdgeSampler,
    GraphSAINTNodeSampler,
    GraphSAINTRandomWalkSampler,
)
from torch_geometric.testing import withPackage


@withPackage('torch_sparse')
def test_graph_saint():
    adj = torch.tensor([
        [+1, +2, +3, +0, +4, +0],
        [+5, +6, +0, +7, +0, +8],
        [+9, +0, 10, +0, 11, +0],
        [+0, 12, +0, 13, +0, 14],
        [15, +0, 16, +0, 17, +0],
        [+0, 18, +0, 19, +0, 20],
    ])

    edge_index = adj.nonzero(as_tuple=False).t()
    edge_id = adj[edge_index[0], edge_index[1]]
    x = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
    n_id = torch.arange(6)
    data = Data(edge_index=edge_index, x=x, n_id=n_id, edge_id=edge_id,
                num_nodes=6)

    loader = GraphSAINTNodeSampler(data, batch_size=3, num_steps=4,
                                   sample_coverage=10, log=False)

    assert len(loader) == 4
    for sample in loader:
        assert sample.num_nodes <= data.num_nodes
        assert sample.n_id.min() >= 0 and sample.n_id.max() < 6
        assert sample.num_nodes == sample.n_id.numel()
        assert sample.x.tolist() == x[sample.n_id].tolist()
        assert sample.edge_index.min() >= 0
        assert sample.edge_index.max() < sample.num_nodes
        assert sample.edge_id.min() >= 1 and sample.edge_id.max() <= 21
        assert sample.edge_id.numel() == sample.num_edges
        assert sample.node_norm.numel() == sample.num_nodes
        assert sample.edge_norm.numel() == sample.num_edges

    loader = GraphSAINTEdgeSampler(data, batch_size=2, num_steps=4,
                                   sample_coverage=10, log=False)

    assert len(loader) == 4
    for sample in loader:
        assert sample.num_nodes <= data.num_nodes
        assert sample.n_id.min() >= 0 and sample.n_id.max() < 6
        assert sample.num_nodes == sample.n_id.numel()
        assert sample.x.tolist() == x[sample.n_id].tolist()
        assert sample.edge_index.min() >= 0
        assert sample.edge_index.max() < sample.num_nodes
        assert sample.edge_id.min() >= 1 and sample.edge_id.max() <= 21
        assert sample.edge_id.numel() == sample.num_edges
        assert sample.node_norm.numel() == sample.num_nodes
        assert sample.edge_norm.numel() == sample.num_edges

    loader = GraphSAINTRandomWalkSampler(data, batch_size=2, walk_length=1,
                                         num_steps=4, sample_coverage=10,
                                         log=False)

    assert len(loader) == 4
    for sample in loader:
        assert sample.num_nodes <= data.num_nodes
        assert sample.n_id.min() >= 0 and sample.n_id.max() < 6
        assert sample.num_nodes == sample.n_id.numel()
        assert sample.x.tolist() == x[sample.n_id].tolist()
        assert sample.edge_index.min() >= 0
        assert sample.edge_index.max() < sample.num_nodes
        assert sample.edge_id.min() >= 1 and sample.edge_id.max() <= 21
        assert sample.edge_id.numel() == sample.num_edges
        assert sample.node_norm.numel() == sample.num_nodes
        assert sample.edge_norm.numel() == sample.num_edges
pytorch/test/loader/test_hgt_loader.py000066400000000000000000000217761444262304300205330ustar00rootroot00000000000000import numpy as np
import torch

from torch_geometric.data import HeteroData
from torch_geometric.loader import HGTLoader
from torch_geometric.nn import GraphConv, to_hetero
from torch_geometric.testing import get_random_edge_index, withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import k_hop_subgraph


def is_subset(subedge_index, edge_index, src_idx, dst_idx):
    num_nodes = int(edge_index.max()) + 1
    idx = num_nodes * edge_index[0] + edge_index[1]
    subidx = num_nodes * src_idx[subedge_index[0]] + dst_idx[subedge_index[1]]
    mask = torch.from_numpy(np.isin(subidx, idx))
    return int(mask.sum()) == mask.numel()


@withPackage('torch_sparse')
def test_hgt_loader():
    torch.manual_seed(12345)

    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['author'].x = torch.arange(100, 300)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'paper'].edge_attr = torch.arange(500)
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 1000)
    data['paper', 'author'].edge_attr = torch.arange(500, 1500)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)
    data['author', 'paper'].edge_attr = torch.arange(1500, 2500)

    r1, c1 = data['paper', 'paper'].edge_index
    r2, c2 = data['paper', 'author'].edge_index + torch.tensor([[0], [100]])
    r3, c3 = data['author', 'paper'].edge_index + torch.tensor([[100], [0]])
    full_adj = SparseTensor(
        row=torch.cat([r1, r2, r3]),
        col=torch.cat([c1, c2, c3]),
        value=torch.arange(2500),
    )

    batch_size = 20
    loader = HGTLoader(data, num_samples=[5] * 4, batch_size=batch_size,
                       input_nodes='paper')
    assert str(loader) == 'HGTLoader()'
    assert len(loader) == (100 + batch_size - 1) // batch_size

    for batch in loader:
        assert isinstance(batch, HeteroData)

        # Test node and types:
        assert set(batch.node_types) == {'paper', 'author'}
        assert set(batch.edge_types) == set(data.edge_types)

        assert len(batch['paper']) == 4
        assert batch['paper'].n_id.size() == (batch['paper'].num_nodes, )
        assert batch['paper'].x.size() == (40, )  # 20 + 4 * 5
        assert batch['paper'].input_id.numel() == batch_size
        assert batch['paper'].batch_size == batch_size
        assert batch['paper'].x.min() >= 0 and batch['paper'].x.max() < 100

        assert len(batch['author']) == 2
        assert batch['author'].n_id.size() == (batch['author'].num_nodes, )
        assert batch['author'].x.size() == (20, )  # 4 * 5
        assert batch['author'].x.min() >= 100 and batch['author'].x.max() < 300

        # Test edge type selection:
        assert set(batch.edge_types) == {('paper', 'to', 'paper'),
                                         ('paper', 'to', 'author'),
                                         ('author', 'to', 'paper')}

        assert len(batch['paper', 'paper']) == 3
        num_edges = batch['paper', 'paper'].num_edges
        assert batch['paper', 'paper'].e_id.size() == (num_edges, )
        row, col = batch['paper', 'paper'].edge_index
        value = batch['paper', 'paper'].edge_attr
        adj = full_adj[batch['paper'].x, batch['paper'].x]
        assert row.min() >= 0 and row.max() < 40
        assert col.min() >= 0 and col.max() < 40
        assert value.min() >= 0 and value.max() < 500
        assert adj.nnz() == row.size(0)
        assert torch.allclose(row.unique(), adj.storage.row().unique())
        assert torch.allclose(col.unique(), adj.storage.col().unique())
        assert torch.allclose(value.unique(), adj.storage.value().unique())

        assert is_subset(batch['paper', 'paper'].edge_index,
                         data['paper', 'paper'].edge_index, batch['paper'].x,
                         batch['paper'].x)

        assert len(batch['paper', 'author']) == 3
        num_edges = batch['paper', 'author'].num_edges
        assert batch['paper', 'author'].e_id.size() == (num_edges, )
        row, col = batch['paper', 'author'].edge_index
        value = batch['paper', 'author'].edge_attr
        adj = full_adj[batch['paper'].x, batch['author'].x]
        assert row.min() >= 0 and row.max() < 40
        assert col.min() >= 0 and col.max() < 20
        assert value.min() >= 500 and value.max() < 1500
        assert adj.nnz() == row.size(0)
        assert torch.allclose(row.unique(), adj.storage.row().unique())
        assert torch.allclose(col.unique(), adj.storage.col().unique())
        assert torch.allclose(value.unique(), adj.storage.value().unique())

        assert is_subset(batch['paper', 'author'].edge_index,
                         data['paper', 'author'].edge_index, batch['paper'].x,
                         batch['author'].x - 100)

        assert len(batch['author', 'paper']) == 3
        num_edges = batch['author', 'paper'].num_edges
        assert batch['author', 'paper'].e_id.size() == (num_edges, )
        row, col = batch['author', 'paper'].edge_index
        value = batch['author', 'paper'].edge_attr
        adj = full_adj[batch['author'].x, batch['paper'].x]
        assert row.min() >= 0 and row.max() < 20
        assert col.min() >= 0 and col.max() < 40
        assert value.min() >= 1500 and value.max() < 2500
        assert adj.nnz() == row.size(0)
        assert torch.allclose(row.unique(), adj.storage.row().unique())
        assert torch.allclose(col.unique(), adj.storage.col().unique())
        assert torch.allclose(value.unique(), adj.storage.value().unique())

        assert is_subset(batch['author', 'paper'].edge_index,
                         data['author', 'paper'].edge_index,
                         batch['author'].x - 100, batch['paper'].x)

        # Test for isolated nodes (there shouldn't exist any):
        n_id = torch.cat([batch['paper'].x, batch['author'].x])
        row, col, _ = full_adj[n_id, n_id].coo()
        assert torch.cat([row, col]).unique().numel() >= 59


@withPackage('torch_sparse')
def test_hgt_loader_on_cora(get_dataset):
    dataset = get_dataset(name='Cora')
    data = dataset[0]
    data.edge_weight = torch.rand(data.num_edges)

    hetero_data = HeteroData()
    hetero_data['paper'].x = data.x
    hetero_data['paper'].n_id = torch.arange(data.num_nodes)
    hetero_data['paper', 'paper'].edge_index = data.edge_index
    hetero_data['paper', 'paper'].edge_weight = data.edge_weight

    split_idx = torch.arange(5, 8)

    # Sample the complete two-hop neighborhood:
    loader = HGTLoader(hetero_data, num_samples=[data.num_nodes] * 2,
                       batch_size=split_idx.numel(),
                       input_nodes=('paper', split_idx))
    assert len(loader) == 1

    hetero_batch = next(iter(loader))
    batch_size = hetero_batch['paper'].batch_size

    n_id, _, _, e_mask = k_hop_subgraph(split_idx, num_hops=2,
                                        edge_index=data.edge_index,
                                        num_nodes=data.num_nodes)

    n_id = n_id.sort()[0]
    assert n_id.tolist() == hetero_batch['paper'].n_id.sort()[0].tolist()
    assert hetero_batch['paper', 'paper'].num_edges == int(e_mask.sum())

    class GNN(torch.nn.Module):
        def __init__(self, in_channels, hidden_channels, out_channels):
            super().__init__()
            self.conv1 = GraphConv(in_channels, hidden_channels)
            self.conv2 = GraphConv(hidden_channels, out_channels)

        def forward(self, x, edge_index, edge_weight):
            x = self.conv1(x, edge_index, edge_weight).relu()
            x = self.conv2(x, edge_index, edge_weight).relu()
            return x

    model = GNN(dataset.num_features, 16, dataset.num_classes)
    hetero_model = to_hetero(model, hetero_data.metadata())

    out1 = model(data.x, data.edge_index, data.edge_weight)[split_idx]
    out2 = hetero_model(hetero_batch.x_dict, hetero_batch.edge_index_dict,
                        hetero_batch.edge_weight_dict)['paper'][:batch_size]
    assert torch.allclose(out1, out2, atol=1e-6)


@withPackage('torch_sparse')
def test_hgt_loader_disconnected():
    data = HeteroData()

    data['paper'].x = torch.randn(10, 16)
    data['author'].x = torch.randn(10, 16)

    # Paper nodes are disconnected from author nodes:
    data['paper', 'paper'].edge_index = get_random_edge_index(10, 10, 15)
    data['paper', 'paper'].edge_attr = torch.randn(15, 8)
    data['author', 'author'].edge_index = get_random_edge_index(10, 10, 15)
    data['author', 'author'].edge_attr = torch.randn(15, 8)

    loader = HGTLoader(data, num_samples=[2], batch_size=2,
                       input_nodes='paper')

    for batch in loader:
        assert isinstance(batch, HeteroData)

        # Test node and edge types:
        assert set(batch.node_types) == set(data.node_types)
        assert set(batch.edge_types) == set(data.edge_types)

        assert batch['author'].num_nodes == 0
        assert batch['author'].x.size() == (0, 16)
        assert batch['author', 'author'].num_edges == 0
        assert batch['author', 'author'].edge_index.size() == (2, 0)
        assert batch['author', 'author'].edge_attr.size() == (0, 8)
pytorch/test/loader/test_imbalanced_sampler.py000066400000000000000000000057451444262304300222230ustar00rootroot00000000000000from typing import List

import torch

from torch_geometric.data import Data
from torch_geometric.datasets import FakeDataset
from torch_geometric.loader import (
    DataLoader,
    ImbalancedSampler,
    NeighborLoader,
)
from torch_geometric.testing import onlyNeighborSampler


def test_dataloader_with_imbalanced_sampler():
    data_list: List[Data] = []
    for _ in range(10):
        data_list.append(Data(num_nodes=10, y=0))
    for _ in range(90):
        data_list.append(Data(num_nodes=10, y=1))

    torch.manual_seed(12345)
    sampler = ImbalancedSampler(data_list)
    loader = DataLoader(data_list, batch_size=10, sampler=sampler)

    y = torch.cat([batch.y for batch in loader])

    histogram = y.bincount()
    prob = histogram / histogram.sum()

    assert histogram.sum() == len(data_list)
    assert prob.min() > 0.4 and prob.max() < 0.6

    # Test with label tensor as input:
    torch.manual_seed(12345)
    sampler = ImbalancedSampler(torch.tensor([data.y for data in data_list]))
    loader = DataLoader(data_list, batch_size=10, sampler=sampler)

    assert torch.allclose(y, torch.cat([batch.y for batch in loader]))

    # Test with list of data objects as input where each y is a tensor:
    torch.manual_seed(12345)
    for data in data_list:
        data.y = torch.tensor([data.y])
    sampler = ImbalancedSampler(data_list)
    loader = DataLoader(data_list, batch_size=100, sampler=sampler)

    assert torch.allclose(y, torch.cat([batch.y for batch in loader]))


def test_in_memory_dataset_imbalanced_sampler():
    torch.manual_seed(12345)
    dataset = FakeDataset(num_graphs=100, avg_num_nodes=10, avg_degree=0,
                          num_channels=0, num_classes=2)
    sampler = ImbalancedSampler(dataset)
    loader = DataLoader(dataset, batch_size=10, sampler=sampler)

    y = torch.cat([batch.y for batch in loader])
    histogram = y.bincount()
    prob = histogram / histogram.sum()

    assert histogram.sum() == len(dataset)
    assert prob.min() > 0.4 and prob.max() < 0.6


@onlyNeighborSampler
def test_neighbor_loader_with_imbalanced_sampler():
    zeros = torch.zeros(10, dtype=torch.long)
    ones = torch.ones(90, dtype=torch.long)

    y = torch.cat([zeros, ones], dim=0)
    edge_index = torch.empty((2, 0), dtype=torch.long)
    data = Data(edge_index=edge_index, y=y, num_nodes=y.size(0))

    torch.manual_seed(12345)
    sampler = ImbalancedSampler(data)
    loader = NeighborLoader(data, batch_size=10, sampler=sampler,
                            num_neighbors=[-1])

    y = torch.cat([batch.y for batch in loader])

    histogram = y.bincount()
    prob = histogram / histogram.sum()

    assert histogram.sum() == data.num_nodes
    assert prob.min() > 0.4 and prob.max() < 0.6

    # Test with label tensor as input:
    torch.manual_seed(12345)
    sampler = ImbalancedSampler(data.y)
    loader = NeighborLoader(data, batch_size=10, sampler=sampler,
                            num_neighbors=[-1])

    assert torch.allclose(y, torch.cat([batch.y for batch in loader]))
pytorch/test/loader/test_link_neighbor_loader.py000066400000000000000000000466661444262304300225700ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.loader import LinkNeighborLoader
from torch_geometric.testing import (
    MyFeatureStore,
    MyGraphStore,
    get_random_edge_index,
    onlyNeighborSampler,
    withCUDA,
    withPackage,
)


def unique_edge_pairs(edge_index):
    return set(map(tuple, edge_index.t().tolist()))


@withCUDA
@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', ['directional', 'bidirectional'])
@pytest.mark.parametrize('neg_sampling_ratio', [None, 1.0])
@pytest.mark.parametrize('filter_per_worker', [None, True, False])
def test_homo_link_neighbor_loader_basic(device, subgraph_type,
                                         neg_sampling_ratio,
                                         filter_per_worker):
    pos_edge_index = get_random_edge_index(50, 50, 500, device=device)
    neg_edge_index = get_random_edge_index(50, 50, 500, device=device)
    neg_edge_index += 50

    edge_label_index = torch.cat([pos_edge_index, neg_edge_index], dim=-1)
    edge_label = torch.cat([
        torch.ones(500, device=device),
        torch.zeros(500, device=device),
    ], dim=0)

    data = Data()

    data.edge_index = pos_edge_index
    data.x = torch.arange(100, device=device)
    data.edge_attr = torch.arange(500, device=device)

    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        batch_size=20,
        edge_label_index=edge_label_index,
        edge_label=edge_label if neg_sampling_ratio is None else None,
        subgraph_type=subgraph_type,
        neg_sampling_ratio=neg_sampling_ratio,
        shuffle=True,
        filter_per_worker=filter_per_worker,
    )

    assert str(loader) == 'LinkNeighborLoader()'
    assert len(loader) == 1000 / 20

    batch = loader([0])
    assert isinstance(batch, Data)
    assert int(edge_label_index[0, 0]) in batch.n_id.tolist()
    assert int(edge_label_index[1, 0]) in batch.n_id.tolist()

    for batch in loader:
        assert isinstance(batch, Data)

        assert batch.n_id.size() == (batch.num_nodes, )
        assert batch.e_id.size() == (batch.num_edges, )
        assert batch.x.device == device
        assert batch.x.size(0) <= 100
        assert batch.x.min() >= 0 and batch.x.max() < 100
        assert batch.input_id.numel() == 20
        assert batch.edge_index.device == device
        assert batch.edge_index.min() >= 0
        assert batch.edge_index.max() < batch.num_nodes
        assert batch.edge_attr.device == device
        assert batch.edge_attr.min() >= 0
        assert batch.edge_attr.max() < 500

        if neg_sampling_ratio is None:
            assert batch.edge_label_index.size(1) == 20

            # Assert positive samples are present in the original graph:
            edge_index = unique_edge_pairs(batch.edge_index)
            edge_label_index = batch.edge_label_index[:, batch.edge_label == 1]
            edge_label_index = unique_edge_pairs(edge_label_index)
            assert len(edge_index | edge_label_index) == len(edge_index)

            # Assert negative samples are not present in the original graph:
            edge_index = unique_edge_pairs(batch.edge_index)
            edge_label_index = batch.edge_label_index[:, batch.edge_label == 0]
            edge_label_index = unique_edge_pairs(edge_label_index)
            assert len(edge_index & edge_label_index) == 0

        else:
            assert batch.edge_label_index.size(1) == 40
            assert torch.all(batch.edge_label[:20] == 1)
            assert torch.all(batch.edge_label[20:] == 0)


@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', ['directional', 'bidirectional'])
@pytest.mark.parametrize('neg_sampling_ratio', [None, 1.0])
def test_hetero_link_neighbor_loader_basic(subgraph_type, neg_sampling_ratio):
    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['author'].x = torch.arange(100, 300)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'paper'].edge_attr = torch.arange(500)
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 1000)
    data['paper', 'author'].edge_attr = torch.arange(500, 1500)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)
    data['author', 'paper'].edge_attr = torch.arange(1500, 2500)

    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        edge_label_index=('paper', 'author'),
        batch_size=20,
        subgraph_type=subgraph_type,
        neg_sampling_ratio=neg_sampling_ratio,
        shuffle=True,
    )

    assert str(loader) == 'LinkNeighborLoader()'
    assert len(loader) == 1000 / 20

    for batch in loader:
        assert isinstance(batch, HeteroData)
        if neg_sampling_ratio is None:
            # Assert only positive samples are present in the original graph:
            edge_index = unique_edge_pairs(batch['paper', 'author'].edge_index)
            edge_label_index = batch['paper', 'author'].edge_label_index
            edge_label_index = unique_edge_pairs(edge_label_index)
            assert len(edge_index | edge_label_index) == len(edge_index)

        else:
            assert batch['paper', 'author'].edge_label_index.size(1) == 40
            assert torch.all(batch['paper', 'author'].edge_label[:20] == 1)
            assert torch.all(batch['paper', 'author'].edge_label[20:] == 0)


@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', ['directional', 'bidirectional'])
def test_hetero_link_neighbor_loader_loop(subgraph_type):
    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['author'].x = torch.arange(100, 300)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 1000)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)

    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        edge_label_index=('paper', 'paper'),
        batch_size=20,
        subgraph_type=subgraph_type,
    )

    for batch in loader:
        assert batch['paper'].x.size(0) <= 100
        assert batch['paper'].x.min() >= 0 and batch['paper'].x.max() < 100

        # Assert positive samples are present in the original graph:
        edge_index = unique_edge_pairs(batch['paper', 'paper'].edge_index)
        edge_label_index = batch['paper', 'paper'].edge_label_index
        edge_label_index = unique_edge_pairs(edge_label_index)
        assert len(edge_index | edge_label_index) == len(edge_index)


@onlyNeighborSampler
def test_link_neighbor_loader_edge_label():
    edge_index = get_random_edge_index(100, 100, 500)
    data = Data(edge_index=edge_index, x=torch.arange(100))

    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        batch_size=10,
        neg_sampling_ratio=1.0,
    )

    for batch in loader:
        assert batch.edge_label.dtype == torch.float
        assert torch.all(batch.edge_label[:10] == 1.0)
        assert torch.all(batch.edge_label[10:] == 0.0)

    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        batch_size=10,
        edge_label=torch.ones(500, dtype=torch.long),
        neg_sampling_ratio=1.0,
    )

    for batch in loader:
        assert batch.edge_label.dtype == torch.long
        assert torch.all(batch.edge_label[:10] == 1)
        assert torch.all(batch.edge_label[10:] == 0)


@withPackage('pyg_lib')
def test_temporal_hetero_link_neighbor_loader():
    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['paper'].time = torch.arange(data['paper'].num_nodes) - 200
    data['author'].x = torch.arange(100, 300)
    data['author'].time = torch.arange(data['author'].num_nodes)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 1000)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)

    with pytest.raises(ValueError, match=r"'edge_label_time' is not set"):
        loader = LinkNeighborLoader(
            data,
            num_neighbors=[-1] * 2,
            edge_label_index=('paper', 'paper'),
            batch_size=32,
            time_attr='time',
        )

    # With edge_time:
    edge_time = torch.arange(data['paper', 'paper'].edge_index.size(1))
    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        edge_label_index=('paper', 'paper'),
        edge_label_time=edge_time,
        batch_size=32,
        time_attr='time',
        neg_sampling_ratio=0.5,
        drop_last=True,
    )
    for batch in loader:
        # Check if each seed edge has a different batch:
        assert int(batch['paper'].batch.max()) + 1 == 32

        author_max = batch['author'].time.max()
        edge_max = batch['paper', 'paper'].edge_label_time.max()
        assert edge_max >= author_max
        author_min = batch['author'].time.min()
        edge_min = batch['paper', 'paper'].edge_label_time.min()
        assert edge_min >= author_min


@onlyNeighborSampler
def test_custom_hetero_link_neighbor_loader():
    data = HeteroData()
    feature_store = MyFeatureStore()
    graph_store = MyGraphStore()

    # Set up node features:
    x = torch.arange(100)
    data['paper'].x = x
    feature_store.put_tensor(x, group_name='paper', attr_name='x', index=None)

    x = torch.arange(100, 300)
    data['author'].x = x
    feature_store.put_tensor(x, group_name='author', attr_name='x', index=None)

    # Set up edge indices (GraphStore does not support `edge_attr` at the
    # moment):
    edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'to', 'paper'].edge_index = edge_index
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('paper', 'to', 'paper'),
                               layout='coo', size=(100, 100))

    edge_index = get_random_edge_index(100, 200, 1000)
    data['paper', 'to', 'author'].edge_index = edge_index
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('paper', 'to', 'author'),
                               layout='coo', size=(100, 200))

    edge_index = get_random_edge_index(200, 100, 1000)
    data['author', 'to', 'paper'].edge_index = edge_index
    graph_store.put_edge_index(edge_index=(edge_index[0], edge_index[1]),
                               edge_type=('author', 'to', 'paper'),
                               layout='coo', size=(200, 100))

    loader1 = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        edge_label_index=('paper', 'to', 'author'),
        batch_size=20,
    )

    loader2 = LinkNeighborLoader(
        (feature_store, graph_store),
        num_neighbors=[-1] * 2,
        edge_label_index=('paper', 'to', 'author'),
        batch_size=20,
    )

    assert str(loader1) == str(loader2)

    for (batch1, batch2) in zip(loader1, loader2):
        # Mapped indices of neighbors may be differently sorted:
        assert torch.allclose(batch1['paper'].x.sort()[0],
                              batch2['paper'].x.sort()[0])
        assert torch.allclose(batch1['author'].x.sort()[0],
                              batch2['author'].x.sort()[0])

        # Assert that edge indices have the same size:
        assert (batch1['paper', 'to', 'paper'].edge_index.size() == batch1[
            'paper', 'to', 'paper'].edge_index.size())
        assert (batch1['paper', 'to', 'author'].edge_index.size() == batch1[
            'paper', 'to', 'author'].edge_index.size())
        assert (batch1['author', 'to', 'paper'].edge_index.size() == batch1[
            'author', 'to', 'paper'].edge_index.size())


@onlyNeighborSampler
def test_homo_link_neighbor_loader_no_edges():
    loader = LinkNeighborLoader(
        Data(num_nodes=100),
        num_neighbors=[],
        batch_size=20,
        edge_label_index=get_random_edge_index(100, 100, 100),
    )

    for batch in loader:
        assert isinstance(batch, Data)
        assert batch.input_id.numel() == 20
        assert batch.edge_label_index.size(1) == 20
        assert batch.num_nodes == batch.edge_label_index.unique().numel()


@onlyNeighborSampler
def test_hetero_link_neighbor_loader_no_edges():
    loader = LinkNeighborLoader(
        HeteroData(paper=dict(num_nodes=100)),
        num_neighbors=[],
        edge_label_index=(
            ('paper', 'paper'),
            get_random_edge_index(100, 100, 100),
        ),
        batch_size=20,
    )

    for batch in loader:
        assert isinstance(batch, HeteroData)
        assert batch['paper', 'paper'].input_id.numel() == 20
        assert batch['paper', 'paper'].edge_label_index.size(1) == 20
        assert batch['paper'].num_nodes == batch[
            'paper', 'paper'].edge_label_index.unique().numel()


@withPackage('pyg_lib')
@pytest.mark.parametrize('disjoint', [False, True])
@pytest.mark.parametrize('temporal', [False, True])
@pytest.mark.parametrize('amount', [1, 2])
def test_homo_link_neighbor_loader_triplet(disjoint, temporal, amount):
    if not disjoint and temporal:
        return

    data = Data()
    data.x = torch.arange(100)
    data.edge_index = get_random_edge_index(100, 100, 400)
    data.edge_label_index = get_random_edge_index(100, 100, 500)
    data.edge_attr = torch.arange(data.num_edges)

    time_attr = edge_label_time = None
    if temporal:
        time_attr = 'time'
        data.time = torch.arange(data.num_nodes)

        edge_label_time = torch.max(data.time[data.edge_label_index[0]],
                                    data.time[data.edge_label_index[1]])
        edge_label_time = edge_label_time + 50

    batch_size = 20
    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        batch_size=batch_size,
        edge_label_index=data.edge_label_index,
        edge_label_time=edge_label_time,
        time_attr=time_attr,
        disjoint=disjoint,
        neg_sampling=dict(mode='triplet', amount=amount),
        shuffle=True,
    )

    assert str(loader) == 'LinkNeighborLoader()'
    assert len(loader) == 500 / batch_size

    for batch in loader:
        assert isinstance(batch, Data)

        # Check that `src_index` and `dst_pos_index` point to valid edges:
        assert torch.equal(batch.x[batch.src_index],
                           data.edge_label_index[0, batch.input_id])
        assert torch.equal(batch.x[batch.dst_pos_index],
                           data.edge_label_index[1, batch.input_id])

        # Check that `dst_neg_index` points to valid nodes in the batch:
        if amount == 1:
            assert batch.dst_neg_index.size() == (batch_size, )
        else:
            assert batch.dst_neg_index.size() == (batch_size, amount)
        assert batch.dst_neg_index.min() >= 0
        assert batch.dst_neg_index.max() < batch.num_nodes

        if disjoint:
            # In disjoint mode, seed nodes should always be placed first:
            assert batch.src_index.min() == 0
            assert batch.src_index.max() == batch_size - 1

            assert batch.dst_pos_index.min() == batch_size
            assert batch.dst_pos_index.max() == 2 * batch_size - 1

            assert batch.dst_neg_index.min() == 2 * batch_size
            max_seed_nodes = 2 * batch_size + batch_size * amount
            assert batch.dst_neg_index.max() == max_seed_nodes - 1

            assert batch.batch.min() == 0
            assert batch.batch.max() == batch_size - 1

            # Check that `batch` is always increasing:
            for i in range(0, max_seed_nodes, batch_size):
                batch_vector = batch.batch[i:i + batch_size]
                assert torch.equal(batch_vector, torch.arange(batch_size))

        if temporal:
            for i in range(batch_size):
                assert batch.time[batch.batch == i].max() <= batch.seed_time[i]


@withPackage('pyg_lib')
@pytest.mark.parametrize('disjoint', [False, True])
@pytest.mark.parametrize('temporal', [False, True])
@pytest.mark.parametrize('amount', [1, 2])
def test_hetero_link_neighbor_loader_triplet(disjoint, temporal, amount):
    if not disjoint and temporal:
        return

    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['author'].x = torch.arange(100, 300)

    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 400)
    edge_label_index = get_random_edge_index(100, 100, 500)
    data['paper', 'paper'].edge_label_index = edge_label_index
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 1000)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 1000)

    time_attr = edge_label_time = None
    if temporal:
        time_attr = 'time'
        data['paper'].time = torch.arange(data['paper'].num_nodes)
        data['author'].time = torch.arange(data['author'].num_nodes)

        edge_label_time = torch.max(
            data['paper'].time[data['paper', 'paper'].edge_label_index[0]],
            data['paper'].time[data['paper', 'paper'].edge_label_index[1]],
        )
        edge_label_time = edge_label_time + 50

    weight = torch.rand(data['paper'].num_nodes) if not temporal else None

    batch_size = 20
    index = (('paper', 'paper'), data['paper', 'paper'].edge_label_index)
    loader = LinkNeighborLoader(
        data,
        num_neighbors=[-1] * 2,
        batch_size=batch_size,
        edge_label_index=index,
        edge_label_time=edge_label_time,
        time_attr=time_attr,
        disjoint=disjoint,
        neg_sampling=dict(mode='triplet', amount=amount, weight=weight),
        shuffle=True,
    )

    assert str(loader) == 'LinkNeighborLoader()'
    assert len(loader) == 500 / batch_size

    for batch in loader:
        assert isinstance(batch, HeteroData)

        node_store = batch['paper']
        edge_store = batch['paper', 'paper']

        # Check that `src_index` and `dst_pos_index` point to valid edges:
        assert torch.equal(
            node_store.x[node_store.src_index],
            data['paper', 'paper'].edge_label_index[0, edge_store.input_id])
        assert torch.equal(
            node_store.x[node_store.dst_pos_index],
            data['paper', 'paper'].edge_label_index[1, edge_store.input_id])

        # Check that `dst_neg_index` points to valid nodes in the batch:
        if amount == 1:
            assert node_store.dst_neg_index.size() == (batch_size, )
        else:
            assert node_store.dst_neg_index.size() == (batch_size, amount)
        assert node_store.dst_neg_index.min() >= 0
        assert node_store.dst_neg_index.max() < node_store.num_nodes

        if disjoint:
            # In disjoint mode, seed nodes should always be placed first:
            assert node_store.src_index.min() == 0
            assert node_store.src_index.max() == batch_size - 1

            assert node_store.dst_pos_index.min() == batch_size
            assert node_store.dst_pos_index.max() == 2 * batch_size - 1

            assert node_store.dst_neg_index.min() == 2 * batch_size
            max_seed_nodes = 2 * batch_size + batch_size * amount
            assert node_store.dst_neg_index.max() == max_seed_nodes - 1

            assert node_store.batch.min() == 0
            assert node_store.batch.max() == batch_size - 1

            # Check that `batch` is always increasing:
            for i in range(0, max_seed_nodes, batch_size):
                batch_vector = node_store.batch[i:i + batch_size]
                assert torch.equal(batch_vector, torch.arange(batch_size))

        if temporal:
            for i in range(batch_size):
                assert (node_store.time[node_store.batch == i].max() <=
                        node_store.seed_time[i])
pytorch/test/loader/test_neighbor_loader.py000066400000000000000000000602051444262304300215340ustar00rootroot00000000000000import os.path as osp
import subprocess
from time import sleep

import numpy as np
import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import GraphConv, to_hetero
from torch_geometric.sampler.base import SubgraphType
from torch_geometric.testing import (
    MyFeatureStore,
    MyGraphStore,
    get_random_edge_index,
    onlyLinux,
    onlyNeighborSampler,
    withCUDA,
    withPackage,
)
from torch_geometric.typing import WITH_PYG_LIB, WITH_TORCH_SPARSE
from torch_geometric.utils import (
    is_undirected,
    sort_edge_index,
    to_torch_csc_tensor,
    to_torch_csr_tensor,
    to_undirected,
)


def is_subset(subedge_index, edge_index, src_idx, dst_idx):
    num_nodes = int(edge_index.max()) + 1
    idx = num_nodes * edge_index[0] + edge_index[1]
    subidx = num_nodes * src_idx[subedge_index[0]] + dst_idx[subedge_index[1]]
    mask = torch.from_numpy(np.isin(subidx.cpu().numpy(), idx.cpu().numpy()))
    return int(mask.sum()) == mask.numel()


@withCUDA
@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', list(SubgraphType))
@pytest.mark.parametrize('dtype', [torch.int64, torch.int32])
@pytest.mark.parametrize('filter_per_worker', [None, True, False])
def test_homo_neighbor_loader_basic(device, subgraph_type, dtype,
                                    filter_per_worker):
    if subgraph_type == SubgraphType.induced and not WITH_TORCH_SPARSE:
        return
    if (dtype != torch.int64
            and (not WITH_PYG_LIB or subgraph_type == SubgraphType.induced)):
        return

    torch.manual_seed(12345)

    data = Data()

    data.x = torch.arange(100, device=device)
    data.edge_index = get_random_edge_index(100, 100, 500, dtype, device)
    data.edge_attr = torch.arange(500, device=device)

    loader = NeighborLoader(
        data,
        num_neighbors=[5] * 2,
        batch_size=20,
        subgraph_type=subgraph_type,
        filter_per_worker=filter_per_worker,
    )

    assert str(loader) == 'NeighborLoader()'
    assert len(loader) == 5

    batch = loader([0])
    assert isinstance(batch, Data)
    assert batch.n_id[:1].tolist() == [0]

    for i, batch in enumerate(loader):
        assert isinstance(batch, Data)
        assert batch.x.device == device
        assert batch.x.size(0) <= 100
        assert batch.n_id.size() == (batch.num_nodes, )
        assert batch.input_id.numel() == batch.batch_size == 20
        assert batch.x.min() >= 0 and batch.x.max() < 100
        assert batch.edge_index.device == device
        assert batch.edge_index.min() >= 0
        assert batch.edge_index.max() < batch.num_nodes
        assert batch.edge_attr.device == device
        assert batch.edge_attr.size(0) == batch.edge_index.size(1)

        # Input nodes are always sampled first:
        assert torch.equal(
            batch.x[:batch.batch_size],
            torch.arange(i * batch.batch_size, (i + 1) * batch.batch_size,
                         device=device),
        )

        if subgraph_type != SubgraphType.bidirectional:
            assert batch.e_id.size() == (batch.num_edges, )
            assert batch.edge_attr.min() >= 0
            assert batch.edge_attr.max() < 500

            assert is_subset(
                batch.edge_index.to(torch.int64),
                data.edge_index.to(torch.int64),
                batch.x,
                batch.x,
            )


@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', list(SubgraphType))
@pytest.mark.parametrize('dtype', [torch.int64, torch.int32])
def test_hetero_neighbor_loader_basic(subgraph_type, dtype):
    if subgraph_type == SubgraphType.induced and not WITH_TORCH_SPARSE:
        return
    if (dtype != torch.int64
            and (not WITH_PYG_LIB or subgraph_type == SubgraphType.induced)):
        return

    torch.manual_seed(12345)

    data = HeteroData()

    data['paper'].x = torch.arange(100)
    data['author'].x = torch.arange(100, 300)

    edge_index = get_random_edge_index(100, 100, 500, dtype)
    data['paper', 'paper'].edge_index = edge_index
    data['paper', 'paper'].edge_attr = torch.arange(500)
    edge_index = get_random_edge_index(100, 200, 1000, dtype)
    data['paper', 'author'].edge_index = edge_index
    data['paper', 'author'].edge_attr = torch.arange(500, 1500)
    edge_index = get_random_edge_index(200, 100, 1000, dtype)
    data['author', 'paper'].edge_index = edge_index
    data['author', 'paper'].edge_attr = torch.arange(1500, 2500)

    r1, c1 = data['paper', 'paper'].edge_index
    r2, c2 = data['paper', 'author'].edge_index + torch.tensor([[0], [100]])
    r3, c3 = data['author', 'paper'].edge_index + torch.tensor([[100], [0]])

    batch_size = 20

    with pytest.raises(ValueError, match="hops must be the same across all"):
        loader = NeighborLoader(
            data,
            num_neighbors={
                ('paper', 'to', 'paper'): [-1],
                ('paper', 'to', 'author'): [-1, -1],
                ('author', 'to', 'paper'): [-1, -1],
            },
            input_nodes='paper',
            batch_size=batch_size,
            subgraph_type=subgraph_type,
        )
        next(iter(loader))

    loader = NeighborLoader(
        data,
        num_neighbors=[10] * 2,
        input_nodes='paper',
        batch_size=batch_size,
        subgraph_type=subgraph_type,
    )

    assert str(loader) == 'NeighborLoader()'
    assert len(loader) == (100 + batch_size - 1) // batch_size

    for batch in loader:
        assert isinstance(batch, HeteroData)

        # Test node type selection:
        assert set(batch.node_types) == {'paper', 'author'}

        assert batch['paper'].n_id.size() == (batch['paper'].num_nodes, )
        assert batch['paper'].x.size(0) <= 100
        assert batch['paper'].input_id.numel() == batch_size
        assert batch['paper'].batch_size == batch_size
        assert batch['paper'].x.min() >= 0 and batch['paper'].x.max() < 100

        assert batch['author'].n_id.size() == (batch['author'].num_nodes, )
        assert batch['author'].x.size(0) <= 200
        assert batch['author'].x.min() >= 100 and batch['author'].x.max() < 300

        # Test edge type selection:
        assert set(batch.edge_types) == {('paper', 'to', 'paper'),
                                         ('paper', 'to', 'author'),
                                         ('author', 'to', 'paper')}

        row, col = batch['paper', 'paper'].edge_index
        assert row.min() >= 0 and row.max() < batch['paper'].num_nodes
        assert col.min() >= 0 and col.max() < batch['paper'].num_nodes

        if subgraph_type != SubgraphType.bidirectional:
            assert batch['paper', 'paper'].e_id.size() == (row.numel(), )
            value = batch['paper', 'paper'].edge_attr
            assert value.min() >= 0 and value.max() < 500

            assert is_subset(
                batch['paper', 'paper'].edge_index.to(torch.int64),
                data['paper', 'paper'].edge_index.to(torch.int64),
                batch['paper'].x,
                batch['paper'].x,
            )
        elif subgraph_type != SubgraphType.directional:
            assert 'e_id' not in batch['paper', 'paper']
            assert 'edge_attr' not in batch['paper', 'paper']

            assert is_undirected(batch['paper', 'paper'].edge_index)

        row, col = batch['paper', 'author'].edge_index
        assert row.min() >= 0 and row.max() < batch['paper'].num_nodes
        assert col.min() >= 0 and col.max() < batch['author'].num_nodes

        if subgraph_type != SubgraphType.bidirectional:
            assert batch['paper', 'author'].e_id.size() == (row.numel(), )
            value = batch['paper', 'author'].edge_attr
            assert value.min() >= 500 and value.max() < 1500

            assert is_subset(
                batch['paper', 'author'].edge_index.to(torch.int64),
                data['paper', 'author'].edge_index.to(torch.int64),
                batch['paper'].x,
                batch['author'].x - 100,
            )
        elif subgraph_type != SubgraphType.directional:
            assert 'e_id' not in batch['paper', 'author']
            assert 'edge_attr' not in batch['paper', 'author']

            assert torch.equal(
                batch['paper', 'author'].edge_index,
                sort_edge_index(batch['author', 'paper'].edge_index.flip([0])),
            )

        row, col = batch['author', 'paper'].edge_index
        assert row.min() >= 0 and row.max() < batch['author'].num_nodes
        assert col.min() >= 0 and col.max() < batch['paper'].num_nodes

        if subgraph_type != SubgraphType.bidirectional:
            assert batch['author', 'paper'].e_id.size() == (row.numel(), )
            value = batch['author', 'paper'].edge_attr
            assert value.min() >= 1500 and value.max() < 2500

            assert is_subset(
                batch['author', 'paper'].edge_index.to(torch.int64),
                data['author', 'paper'].edge_index.to(torch.int64),
                batch['author'].x - 100,
                batch['paper'].x,
            )
        elif subgraph_type != SubgraphType.directional:
            assert 'e_id' not in batch['author', 'paper']
            assert 'edge_attr' not in batch['author', 'paper']

            assert torch.equal(
                batch['author', 'paper'].edge_index,
                sort_edge_index(batch['paper', 'author'].edge_index.flip([0])),
            )

        # Test for isolated nodes (there shouldn't exist any):
        assert not batch.has_isolated_nodes()


@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', list(SubgraphType))
def test_homo_neighbor_loader_on_cora(get_dataset, subgraph_type):
    if subgraph_type == SubgraphType.induced and not WITH_TORCH_SPARSE:
        return
    dataset = get_dataset(name='Cora')
    data = dataset[0]

    mask = data.edge_index[0] < data.edge_index[1]
    edge_index = data.edge_index[:, mask]
    edge_weight = torch.rand(edge_index.size(1))
    data.edge_index, data.edge_weight = to_undirected(edge_index, edge_weight)

    split_idx = torch.arange(5, 8)

    loader = NeighborLoader(
        data,
        num_neighbors=[-1, -1],
        batch_size=split_idx.numel(),
        input_nodes=split_idx,
        subgraph_type=subgraph_type,
    )
    assert len(loader) == 1

    batch = next(iter(loader))
    batch_size = batch.batch_size

    class GNN(torch.nn.Module):
        def __init__(self, in_channels, hidden_channels, out_channels):
            super().__init__()
            self.conv1 = GraphConv(in_channels, hidden_channels)
            self.conv2 = GraphConv(hidden_channels, out_channels)

        def forward(self, x, edge_index, edge_weight):
            x = self.conv1(x, edge_index, edge_weight).relu()
            x = self.conv2(x, edge_index, edge_weight)
            return x

    model = GNN(dataset.num_features, 16, dataset.num_classes)

    out1 = model(data.x, data.edge_index, data.edge_weight)[split_idx]
    out2 = model(batch.x, batch.edge_index, batch.edge_weight)[:batch_size]
    assert torch.allclose(out1, out2, atol=1e-6)


@onlyNeighborSampler
@pytest.mark.parametrize('subgraph_type', list(SubgraphType))
def test_hetero_neighbor_loader_on_cora(get_dataset, subgraph_type):
    if subgraph_type == SubgraphType.induced and not WITH_TORCH_SPARSE:
        return
    dataset = get_dataset(name='Cora')
    data = dataset[0]

    hetero_data = HeteroData()
    hetero_data['paper'].x = data.x
    hetero_data['paper', 'paper'].edge_index = data.edge_index

    split_idx = torch.arange(5, 8)

    loader = NeighborLoader(
        hetero_data,
        num_neighbors=[-1, -1],
        batch_size=split_idx.numel(),
        input_nodes=('paper', split_idx),
        subgraph_type=subgraph_type,
    )
    assert len(loader) == 1

    hetero_batch = next(iter(loader))
    batch_size = hetero_batch['paper'].batch_size

    class GNN(torch.nn.Module):
        def __init__(self, in_channels, hidden_channels, out_channels):
            super().__init__()
            self.conv1 = GraphConv(in_channels, hidden_channels)
            self.conv2 = GraphConv(hidden_channels, out_channels)

        def forward(self, x, edge_index):
            x = self.conv1(x, edge_index).relu()
            x = self.conv2(x, edge_index)
            return x

    model = GNN(dataset.num_features, 16, dataset.num_classes)
    hetero_model = to_hetero(model, hetero_data.metadata())

    out1 = model(data.x, data.edge_index)[split_idx]
    out2 = hetero_model(hetero_batch.x_dict,
                        hetero_batch.edge_index_dict)['paper'][:batch_size]
    assert torch.allclose(out1, out2, atol=1e-6)


@withPackage('pyg_lib')
def test_temporal_hetero_neighbor_loader_on_cora(get_dataset):
    dataset = get_dataset(name='Cora')
    data = dataset[0]

    hetero_data = HeteroData()
    hetero_data['paper'].x = data.x
    hetero_data['paper'].time = torch.arange(data.num_nodes, 0, -1)
    hetero_data['paper', 'paper'].edge_index = data.edge_index

    loader = NeighborLoader(hetero_data, num_neighbors=[-1, -1],
                            input_nodes='paper', time_attr='time',
                            batch_size=1)

    for batch in loader:
        mask = batch['paper'].time[0] >= batch['paper'].time[1:]
        assert torch.all(mask)


@onlyNeighborSampler
def test_custom_neighbor_loader():
    # Initialize feature store, graph store, and reference:
    feature_store = MyFeatureStore()
    graph_store = MyGraphStore()
    data = HeteroData()

    # Set up node features:
    x = torch.arange(100)
    data['paper'].x = x
    feature_store.put_tensor(x, group_name='paper', attr_name='x', index=None)

    x = torch.arange(100, 300)
    data['author'].x = x
    feature_store.put_tensor(x, group_name='author', attr_name='x', index=None)

    # COO:
    edge_index = get_random_edge_index(100, 100, 500)
    data['paper', 'to', 'paper'].edge_index = edge_index
    coo = (edge_index[0], edge_index[1])
    graph_store.put_edge_index(edge_index=coo,
                               edge_type=('paper', 'to', 'paper'),
                               layout='coo', size=(100, 100))

    # CSR:
    edge_index = get_random_edge_index(100, 200, 1000)
    data['paper', 'to', 'author'].edge_index = edge_index
    adj = to_torch_csr_tensor(edge_index, size=(100, 200))
    csr = (adj.crow_indices(), adj.col_indices())
    graph_store.put_edge_index(edge_index=csr,
                               edge_type=('paper', 'to', 'author'),
                               layout='csr', size=(100, 200))

    # CSC:
    edge_index = get_random_edge_index(200, 100, 1000)
    data['author', 'to', 'paper'].edge_index = edge_index
    adj = to_torch_csc_tensor(edge_index, size=(200, 100))
    csc = (adj.row_indices(), adj.ccol_indices())
    graph_store.put_edge_index(edge_index=csc,
                               edge_type=('author', 'to', 'paper'),
                               layout='csc', size=(200, 100))

    # COO (sorted):
    edge_index = get_random_edge_index(200, 200, 100)
    edge_index = edge_index[:, edge_index[1].argsort()]
    data['author', 'to', 'author'].edge_index = edge_index
    coo = (edge_index[0], edge_index[1])
    graph_store.put_edge_index(edge_index=coo,
                               edge_type=('author', 'to', 'author'),
                               layout='coo', size=(200, 200), is_sorted=True)

    # Construct neighbor loaders:
    loader1 = NeighborLoader(data, batch_size=20,
                             input_nodes=('paper', range(100)),
                             num_neighbors=[-1] * 2)

    loader2 = NeighborLoader((feature_store, graph_store), batch_size=20,
                             input_nodes=('paper', range(100)),
                             num_neighbors=[-1] * 2)

    assert str(loader1) == str(loader2)
    assert len(loader1) == len(loader2)

    for batch1, batch2 in zip(loader1, loader2):
        # loader2 explicitly adds `num_nodes` to the batch
        assert len(batch1) + 1 == len(batch2)
        assert batch1['paper'].batch_size == batch2['paper'].batch_size

        # Mapped indices of neighbors may be differently sorted:
        assert torch.allclose(batch1['paper'].x.sort()[0],
                              batch2['paper'].x.sort()[0])
        assert torch.allclose(batch1['author'].x.sort()[0],
                              batch2['author'].x.sort()[0])

        assert (batch1['paper', 'to', 'paper'].edge_index.size() == batch1[
            'paper', 'to', 'paper'].edge_index.size())
        assert (batch1['paper', 'to', 'author'].edge_index.size() == batch1[
            'paper', 'to', 'author'].edge_index.size())
        assert (batch1['author', 'to', 'paper'].edge_index.size() == batch1[
            'author', 'to', 'paper'].edge_index.size())


@withPackage('pyg_lib')
def test_temporal_custom_neighbor_loader_on_cora(get_dataset):
    # Initialize dataset (once):
    dataset = get_dataset(name='Cora')
    data = dataset[0]
    data.time = torch.arange(data.num_nodes, 0, -1)

    # Initialize feature store, graph store, and reference:
    feature_store = MyFeatureStore()
    graph_store = MyGraphStore()
    hetero_data = HeteroData()

    feature_store.put_tensor(
        data.x,
        group_name='paper',
        attr_name='x',
        index=None,
    )
    hetero_data['paper'].x = data.x

    feature_store.put_tensor(
        data.time,
        group_name='paper',
        attr_name='time',
        index=None,
    )
    hetero_data['paper'].time = data.time

    # Sort according to time in local neighborhoods:
    row, col = data.edge_index
    perm = ((col * (data.num_nodes + 1)) + data.time[row]).argsort()
    edge_index = data.edge_index[:, perm]

    graph_store.put_edge_index(
        edge_index,
        edge_type=('paper', 'to', 'paper'),
        layout='coo',
        is_sorted=True,
        size=(data.num_nodes, data.num_nodes),
    )
    hetero_data['paper', 'to', 'paper'].edge_index = data.edge_index

    loader1 = NeighborLoader(
        hetero_data,
        num_neighbors=[-1, -1],
        input_nodes='paper',
        time_attr='time',
        batch_size=128,
    )

    loader2 = NeighborLoader(
        (feature_store, graph_store),
        num_neighbors=[-1, -1],
        input_nodes='paper',
        time_attr='time',
        batch_size=128,
    )

    for batch1, batch2 in zip(loader1, loader2):
        assert torch.equal(batch1['paper'].time, batch2['paper'].time)


@withPackage('pyg_lib')
@withPackage('torch_sparse')
def test_pyg_lib_and_torch_sparse_homo_equality():
    edge_index = get_random_edge_index(20, 20, 100)
    adj = to_torch_csc_tensor(edge_index, size=(20, 20))
    colptr, row = adj.ccol_indices(), adj.row_indices()

    seed = torch.arange(10)

    sample = torch.ops.pyg.neighbor_sample
    out1 = sample(colptr, row, seed, [-1, -1], None, None, True)
    sample = torch.ops.torch_sparse.neighbor_sample
    out2 = sample(colptr, row, seed, [-1, -1], False, True)

    row1, col1, node_id1, edge_id1 = out1[:4]
    node_id2, row2, col2, edge_id2 = out2
    assert torch.equal(node_id1, node_id2)
    assert torch.equal(row1, row2)
    assert torch.equal(col1, col2)
    assert torch.equal(edge_id1, edge_id2)


@withPackage('pyg_lib')
@withPackage('torch_sparse')
def test_pyg_lib_and_torch_sparse_hetero_equality():
    edge_index = get_random_edge_index(20, 10, 50)
    adj = to_torch_csc_tensor(edge_index, size=(20, 10))
    colptr1, row1 = adj.ccol_indices(), adj.row_indices()

    edge_index = get_random_edge_index(10, 20, 50)
    adj = to_torch_csc_tensor(edge_index, size=(10, 20))
    colptr2, row2 = adj.ccol_indices(), adj.row_indices()

    node_types = ['paper', 'author']
    edge_types = [('paper', 'to', 'author'), ('author', 'to', 'paper')]
    colptr_dict = {
        'paper__to__author': colptr1,
        'author__to__paper': colptr2,
    }
    row_dict = {
        'paper__to__author': row1,
        'author__to__paper': row2,
    }
    seed_dict = {'paper': torch.arange(1)}
    num_neighbors_dict = {
        'paper__to__author': [-1, -1],
        'author__to__paper': [-1, -1],
    }

    sample = torch.ops.pyg.hetero_neighbor_sample
    out1 = sample(node_types, edge_types, colptr_dict, row_dict, seed_dict,
                  num_neighbors_dict, None, None, True, False, True, False,
                  "uniform", True)
    sample = torch.ops.torch_sparse.hetero_neighbor_sample
    out2 = sample(node_types, edge_types, colptr_dict, row_dict, seed_dict,
                  num_neighbors_dict, 2, False, True)

    row1_dict, col1_dict, node_id1_dict, edge_id1_dict = out1[:4]
    node_id2_dict, row2_dict, col2_dict, edge_id2_dict = out2
    assert len(node_id1_dict) == len(node_id2_dict)
    for key in node_id1_dict.keys():
        assert torch.equal(node_id1_dict[key], node_id2_dict[key])
    assert len(row1_dict) == len(row2_dict)
    for key in row1_dict.keys():
        assert torch.equal(row1_dict[key], row2_dict[key])
    assert len(col1_dict) == len(col2_dict)
    for key in col1_dict.keys():
        assert torch.equal(col1_dict[key], col2_dict[key])
    assert len(edge_id1_dict) == len(edge_id2_dict)
    for key in edge_id1_dict.keys():
        assert torch.equal(edge_id1_dict[key], edge_id2_dict[key])


@onlyLinux
@onlyNeighborSampler
def test_memmap_neighbor_loader(tmp_path):
    path = osp.join(tmp_path, 'x.npy')
    x = np.memmap(path, dtype=np.float32, mode='w+', shape=(100, 32))
    x[:] = np.random.randn(100, 32)

    data = Data()
    data.x = np.memmap(path, dtype=np.float32, mode='r', shape=(100, 32))
    data.edge_index = get_random_edge_index(100, 100, 500)

    assert str(data) == 'Data(x=[100, 32], edge_index=[2, 500])'
    assert data.num_nodes == 100

    loader = NeighborLoader(data, num_neighbors=[5] * 2, batch_size=20,
                            num_workers=2)
    batch = next(iter(loader))
    assert batch.num_nodes <= 100
    assert isinstance(batch.x, torch.Tensor)
    assert batch.x.size() == (batch.num_nodes, 32)


@onlyLinux
@onlyNeighborSampler
@pytest.mark.parametrize('loader_cores', [None, [1]])
def test_cpu_affinity_neighbor_loader(loader_cores):
    data = Data(x=torch.randn(1, 1))
    loader = NeighborLoader(data, num_neighbors=[-1], batch_size=1,
                            num_workers=1)

    out = []
    with loader.enable_cpu_affinity(loader_cores):
        iterator = loader._get_iterator()
        workers = iterator._workers
        for worker in workers:
            sleep(1)  # Gives time for worker to initialize.
            process = subprocess.Popen(
                ['taskset', '-c', '-p', f'{worker.pid}'],
                stdout=subprocess.PIPE)
            stdout = process.communicate()[0].decode('utf-8')
            out.append(int(stdout.split(':')[1].strip()))
        if not loader_cores:
            assert out == [0]
        else:
            assert out == loader_cores


@withPackage('pyg_lib')
def test_homo_neighbor_loader_sampled_info():
    edge_index = torch.tensor([
        [2, 3, 4, 5, 7, 7, 10, 11, 12, 13],
        [0, 1, 2, 3, 2, 3, 7, 7, 7, 7],
    ])

    data = Data(edge_index=edge_index, num_nodes=14)

    loader = NeighborLoader(
        data,
        num_neighbors=[1, 2, 4],
        batch_size=2,
        shuffle=False,
    )
    batch = next(iter(loader))

    assert batch.num_sampled_nodes == [2, 2, 3, 4]
    assert batch.num_sampled_edges == [2, 4, 4]


@withPackage('pyg_lib')
def test_hetero_neighbor_loader_sampled_info():
    edge_index = torch.tensor([
        [2, 3, 4, 5, 7, 7, 10, 11, 12, 13],
        [0, 1, 2, 3, 2, 3, 7, 7, 7, 7],
    ])

    data = HeteroData()
    data['paper'].num_nodes = data['author'].num_nodes = 14
    data['paper', 'paper'].edge_index = edge_index
    data['paper', 'author'].edge_index = edge_index
    data['author', 'paper'].edge_index = edge_index

    loader = NeighborLoader(
        data,
        num_neighbors=[1, 2, 4],
        batch_size=2,
        input_nodes='paper',
        shuffle=False,
    )
    batch = next(iter(loader))

    expected_num_sampled_nodes = {
        'paper': [2, 2, 3, 4],
        'author': [0, 2, 3, 4],
    }
    expected_num_sampled_edges = {
        ('paper', 'to', 'paper'): [2, 4, 4],
        ('paper', 'to', 'author'): [0, 4, 4],
        ('author', 'to', 'paper'): [2, 4, 4],
    }

    for node_type in batch.node_types:
        assert (batch[node_type].num_sampled_nodes ==
                expected_num_sampled_nodes[node_type])
    for edge_type in batch.edge_types:
        assert (batch[edge_type].num_sampled_edges ==
                expected_num_sampled_edges[edge_type])
pytorch/test/loader/test_neighbor_sampler.py000066400000000000000000000075601444262304300217360ustar00rootroot00000000000000import numpy as np
import torch

from torch_geometric.loader import NeighborSampler
from torch_geometric.nn.conv import GATConv, SAGEConv
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import erdos_renyi_graph


@withPackage('torch_sparse')
def test_neighbor_sampler_basic():
    edge_index = erdos_renyi_graph(num_nodes=10, edge_prob=0.5)
    adj_t = SparseTensor.from_edge_index(edge_index, sparse_sizes=(10, 10)).t()
    E = edge_index.size(1)

    loader = NeighborSampler(edge_index, sizes=[2, 4], batch_size=2)
    assert str(loader) == 'NeighborSampler(sizes=[2, 4])'
    assert len(loader) == 5

    for batch_size, n_id, adjs in loader:
        assert batch_size == 2
        assert all(np.isin(n_id, torch.arange(10)).tolist())
        assert n_id.unique().size(0) == n_id.size(0)
        for (edge_index, e_id, size) in adjs:
            assert int(edge_index[0].max() + 1) <= size[0]
            assert int(edge_index[1].max() + 1) <= size[1]
            assert all(np.isin(e_id, torch.arange(E)).tolist())
            assert e_id.unique().size(0) == e_id.size(0)
            assert size[0] >= size[1]

    out = loader.sample([1, 2])
    assert len(out) == 3

    loader = NeighborSampler(adj_t, sizes=[2, 4], batch_size=2)

    for batch_size, n_id, adjs in loader:
        for (adj_t, e_id, size) in adjs:
            assert adj_t.size(0) == size[1]
            assert adj_t.size(1) == size[0]


@withPackage('torch_sparse')
def test_neighbor_sampler_invalid_kwargs():
    # Ignore `collate_fn` and `dataset` arguments:
    edge_index = torch.tensor([[0, 1], [1, 0]])
    NeighborSampler(edge_index, sizes=[-1], collate_fn=None, dataset=None)


@withPackage('torch_sparse')
def test_neighbor_sampler_on_cora(get_dataset):
    dataset = get_dataset(name='Cora')
    data = dataset[0]

    batch = torch.arange(10)
    loader = NeighborSampler(data.edge_index, sizes=[-1, -1, -1],
                             node_idx=batch, batch_size=10)

    class SAGE(torch.nn.Module):
        def __init__(self, in_channels, out_channels):
            super().__init__()

            self.convs = torch.nn.ModuleList()
            self.convs.append(SAGEConv(in_channels, 16))
            self.convs.append(SAGEConv(16, 16))
            self.convs.append(SAGEConv(16, out_channels))

        def batch(self, x, adjs):
            for i, (edge_index, _, size) in enumerate(adjs):
                x_target = x[:size[1]]  # Target nodes are always placed first.
                x = self.convs[i]((x, x_target), edge_index)
            return x

        def full(self, x, edge_index):
            for conv in self.convs:
                x = conv(x, edge_index)
            return x

    model = SAGE(dataset.num_features, dataset.num_classes)

    _, n_id, adjs = next(iter(loader))
    out1 = model.batch(data.x[n_id], adjs)
    out2 = model.full(data.x, data.edge_index)[batch]
    assert torch.allclose(out1, out2, atol=1e-7)

    class GAT(torch.nn.Module):
        def __init__(self, in_channels, out_channels):
            super().__init__()

            self.convs = torch.nn.ModuleList()
            self.convs.append(GATConv(in_channels, 16, heads=2))
            self.convs.append(GATConv(32, 16, heads=2))
            self.convs.append(GATConv(32, out_channels, heads=2, concat=False))

        def batch(self, x, adjs):
            for i, (edge_index, _, size) in enumerate(adjs):
                x_target = x[:size[1]]  # Target nodes are always placed first.
                x = self.convs[i]((x, x_target), edge_index)
            return x

        def full(self, x, edge_index):
            for conv in self.convs:
                x = conv(x, edge_index)
            return x

    _, n_id, adjs = next(iter(loader))
    out1 = model.batch(data.x[n_id], adjs)
    out2 = model.full(data.x, data.edge_index)[batch]
    assert torch.allclose(out1, out2, atol=1e-7)
pytorch/test/loader/test_prefetch.py000066400000000000000000000030531444262304300202070ustar00rootroot00000000000000import torch

from torch_geometric.loader import NeighborLoader, PrefetchLoader
from torch_geometric.nn import GraphSAGE
from torch_geometric.testing import withCUDA


@withCUDA
def test_prefetch_loader(device):
    data = [torch.randn(5, 5) for _ in range(10)]

    loader = PrefetchLoader(data, device=device)
    assert str(loader).startswith('PrefetchLoader')
    assert len(loader) == 10

    for i, batch in enumerate(loader):
        assert batch.device == device
        assert torch.equal(batch.cpu(), data[i])


if __name__ == '__main__':
    import argparse

    from ogb.nodeproppred import PygNodePropPredDataset
    from tqdm import tqdm

    parser = argparse.ArgumentParser()
    parser.add_argument('--num_workers', type=int, default=0)
    args = parser.parse_args()

    data = PygNodePropPredDataset('ogbn-products', root='/tmp/ogb')[0]

    model = GraphSAGE(
        in_channels=data.x.size(-1),
        hidden_channels=64,
        num_layers=2,
    ).cuda()

    loader = NeighborLoader(
        data,
        input_nodes=torch.arange(1024 * 200),
        batch_size=1024,
        num_neighbors=[10, 10],
        num_workers=args.num_workers,
        persistent_workers=args.num_workers > 0,
    )

    print('Forward pass without prefetching...')
    for batch in tqdm(loader):
        with torch.no_grad():
            batch = batch.cuda()
            model(batch.x, batch.edge_index)

    print('Forward pass with prefetching...')
    for batch in tqdm(PrefetchLoader(loader)):
        with torch.no_grad():
            model(batch.x, batch.edge_index)
pytorch/test/loader/test_random_node_loader.py000066400000000000000000000032761444262304300222310ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.loader import RandomNodeLoader
from torch_geometric.testing import get_random_edge_index


def test_random_node_loader():
    data = Data()
    data.x = torch.randn(100, 128)
    data.node_id = torch.arange(100)
    data.edge_index = get_random_edge_index(100, 100, 500)
    data.edge_attr = torch.randn(500, 32)

    loader = RandomNodeLoader(data, num_parts=4, shuffle=True)
    assert len(loader) == 4

    for batch in loader:
        assert len(batch) == 4
        assert batch.node_id.min() >= 0
        assert batch.node_id.max() < 100
        assert batch.edge_index.size(1) == batch.edge_attr.size(0)
        assert torch.allclose(batch.x, data.x[batch.node_id])
        batch.validate()


def test_heterogeneous_random_node_loader():
    data = HeteroData()
    data['paper'].x = torch.randn(100, 128)
    data['paper'].node_id = torch.arange(100)
    data['author'].x = torch.randn(200, 128)
    data['author'].node_id = torch.arange(200)
    data['paper', 'author'].edge_index = get_random_edge_index(100, 200, 500)
    data['paper', 'author'].edge_attr = torch.randn(500, 32)
    data['author', 'paper'].edge_index = get_random_edge_index(200, 100, 400)
    data['author', 'paper'].edge_attr = torch.randn(400, 32)
    data['paper', 'paper'].edge_index = get_random_edge_index(100, 100, 600)
    data['paper', 'paper'].edge_attr = torch.randn(600, 32)

    loader = RandomNodeLoader(data, num_parts=4, shuffle=True)
    assert len(loader) == 4

    for batch in loader:
        assert len(batch) == 4
        assert batch.node_types == data.node_types
        assert batch.edge_types == data.edge_types
        batch.validate()
pytorch/test/loader/test_shadow.py000066400000000000000000000045771444262304300177100ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.loader import ShaDowKHopSampler
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor


@withPackage('torch_sparse')
def test_shadow_k_hop_sampler():
    row = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 4, 4, 5, 5])
    col = torch.tensor([1, 2, 3, 0, 2, 0, 1, 4, 5, 0, 2, 5, 2, 4])
    edge_index = torch.stack([row, col], dim=0)
    edge_weight = torch.arange(row.size(0))
    x = torch.randn(6, 16)
    y = torch.randint(3, (6, ), dtype=torch.long)
    data = Data(edge_index=edge_index, edge_weight=edge_weight, x=x, y=y)

    train_mask = torch.tensor([1, 1, 0, 0, 0, 0], dtype=torch.bool)
    loader = ShaDowKHopSampler(data, depth=1, num_neighbors=3,
                               node_idx=train_mask, batch_size=2)
    assert len(loader) == 1

    batch1 = next(iter(loader))
    assert batch1.num_graphs == len(batch1) == 2

    assert batch1.batch.tolist() == [0, 0, 0, 0, 1, 1, 1]
    assert batch1.ptr.tolist() == [0, 4, 7]
    assert batch1.root_n_id.tolist() == [0, 5]
    assert batch1.x.tolist() == x[torch.tensor([0, 1, 2, 3, 0, 1, 2])].tolist()
    assert batch1.y.tolist() == y[train_mask].tolist()
    row, col = batch1.edge_index
    assert row.tolist() == [0, 0, 0, 1, 1, 2, 2, 3, 4, 4, 5, 5, 6, 6]
    assert col.tolist() == [1, 2, 3, 0, 2, 0, 1, 0, 5, 6, 4, 6, 4, 5]
    e_id = torch.tensor([0, 1, 2, 3, 4, 5, 6, 9, 0, 1, 3, 4, 5, 6])
    assert batch1.edge_weight.tolist() == edge_weight[e_id].tolist()

    adj_t = SparseTensor(row=edge_index[0], col=edge_index[1],
                         value=edge_weight).t()
    data = Data(adj_t=adj_t, x=x, y=y)

    loader = ShaDowKHopSampler(data, depth=1, num_neighbors=3,
                               node_idx=train_mask, batch_size=2)
    assert len(loader) == 1

    batch2 = next(iter(loader))
    assert batch2.num_graphs == len(batch2) == 2

    assert batch1.batch.tolist() == batch2.batch.tolist()
    assert batch1.ptr.tolist() == batch2.ptr.tolist()
    assert batch1.root_n_id.tolist() == batch2.root_n_id.tolist()
    assert batch1.x.tolist() == batch2.x.tolist()
    assert batch1.y.tolist() == batch2.y.tolist()
    row, col, value = batch2.adj_t.t().coo()
    assert batch1.edge_index[0].tolist() == row.tolist()
    assert batch1.edge_index[1].tolist() == col.tolist()
    assert batch1.edge_weight.tolist() == value.tolist()
pytorch/test/loader/test_temporal_dataloader.py000066400000000000000000000013441444262304300224130ustar00rootroot00000000000000import torch

from torch_geometric.data import TemporalData
from torch_geometric.loader import TemporalDataLoader


def test_temporal_dataloader():
    src = dst = t = torch.arange(10)
    msg = torch.randn(10, 16)

    data = TemporalData(src=src, dst=dst, t=t, msg=msg)

    loader = TemporalDataLoader(data, batch_size=2)
    assert len(loader) == 5

    for i, batch in enumerate(loader):
        assert len(batch) == 2
        arange = range(len(batch) * i, len(batch) * i + len(batch))
        assert batch.src.tolist() == data.src[arange].tolist()
        assert batch.dst.tolist() == data.dst[arange].tolist()
        assert batch.t.tolist() == data.t[arange].tolist()
        assert batch.msg.tolist() == data.msg[arange].tolist()
pytorch/test/loader/test_zip_loader.py000066400000000000000000000030051444262304300205340ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.loader import NeighborLoader, ZipLoader
from torch_geometric.testing import onlyNeighborSampler


@onlyNeighborSampler
@pytest.mark.parametrize('filter_per_worker', [True, False])
def test_zip_loader(filter_per_worker):
    x = torch.arange(100)
    edge_index = torch.randint(0, 100, (2, 1000))
    data = Data(x=x, edge_index=edge_index)

    loaders = [
        NeighborLoader(data, [5], input_nodes=torch.arange(0, 50)),
        NeighborLoader(data, [5], input_nodes=torch.arange(50, 95)),
    ]

    loader = ZipLoader(loaders, batch_size=10,
                       filter_per_worker=filter_per_worker)

    assert str(loader) == ('ZipLoader(loaders=[NeighborLoader(), '
                           'NeighborLoader()])')
    assert len(loader) == 5
    assert loader.dataset == range(0, 45)

    for i, (batch1, batch2) in enumerate(loader):
        n_id1 = batch1.n_id[:batch1.batch_size]
        n_id2 = batch2.n_id[:batch2.batch_size]

        if i < 4:
            assert batch1.batch_size == 10
            assert batch2.batch_size == 10
            assert torch.equal(n_id1, torch.arange(0 + i * 10, 10 + i * 10))
            assert torch.equal(n_id2, torch.arange(50 + i * 10, 60 + i * 10))
        else:
            assert batch1.batch_size == 5
            assert batch2.batch_size == 5
            assert torch.equal(n_id1, torch.arange(0 + i * 10, 5 + i * 10))
            assert torch.equal(n_id2, torch.arange(50 + i * 10, 55 + i * 10))
pytorch/test/my_config.yaml000066400000000000000000000004261444262304300163670ustar00rootroot00000000000000defaults:
  - dataset: KarateClub
  - transform@dataset.transform:
      - NormalizeFeatures
      - AddSelfLoops
  - model: GCN
  - optimizer: Adam
  - lr_scheduler: ReduceLROnPlateau
  - _self_

model:
  in_channels: 34
  out_channels: 4
  hidden_channels: 16
  num_layers: 2
pytorch/test/nn/000077500000000000000000000000001444262304300141425ustar00rootroot00000000000000pytorch/test/nn/aggr/000077500000000000000000000000001444262304300150625ustar00rootroot00000000000000pytorch/test/nn/aggr/test_aggr_utils.py000066400000000000000000000046151444262304300206410ustar00rootroot00000000000000import torch

from torch_geometric.nn.aggr.utils import (
    InducedSetAttentionBlock,
    MultiheadAttentionBlock,
    PoolingByMultiheadAttention,
    SetAttentionBlock,
)


def test_multihead_attention_block():
    x = torch.randn(2, 4, 8)
    y = torch.randn(2, 3, 8)
    x_mask = torch.tensor([[1, 1, 1, 1], [1, 1, 0, 0]], dtype=torch.bool)
    y_mask = torch.tensor([[1, 1, 0], [1, 1, 1]], dtype=torch.bool)

    block = MultiheadAttentionBlock(8, heads=2)
    block.reset_parameters()
    assert str(block) == ('MultiheadAttentionBlock(8, heads=2, '
                          'layer_norm=True, dropout=0.0)')

    out = block(x, y, x_mask, y_mask)
    assert out.size() == (2, 4, 8)

    jit = torch.jit.script(block)
    assert torch.allclose(jit(x, y, x_mask, y_mask), out)


def test_multihead_attention_block_dropout():
    x = torch.randn(2, 4, 8)

    block = MultiheadAttentionBlock(8, dropout=0.5)
    assert not torch.allclose(block(x, x), block(x, x))


def test_set_attention_block():
    x = torch.randn(2, 4, 8)
    mask = torch.tensor([[1, 1, 1, 1], [1, 1, 0, 0]], dtype=torch.bool)

    block = SetAttentionBlock(8, heads=2)
    block.reset_parameters()
    assert str(block) == ('SetAttentionBlock(8, heads=2, layer_norm=True, '
                          'dropout=0.0)')

    out = block(x, mask)
    assert out.size() == (2, 4, 8)

    jit = torch.jit.script(block)
    assert torch.allclose(jit(x, mask), out)


def test_induced_set_attention_block():
    x = torch.randn(2, 4, 8)
    mask = torch.tensor([[1, 1, 1, 1], [1, 1, 0, 0]], dtype=torch.bool)

    block = InducedSetAttentionBlock(8, num_induced_points=2, heads=2)
    assert str(block) == ('InducedSetAttentionBlock(8, num_induced_points=2, '
                          'heads=2, layer_norm=True, dropout=0.0)')

    out = block(x, mask)
    assert out.size() == (2, 4, 8)

    jit = torch.jit.script(block)
    assert torch.allclose(jit(x, mask), out)


def test_pooling_by_multihead_attention():
    x = torch.randn(2, 4, 8)
    mask = torch.tensor([[1, 1, 1, 1], [1, 1, 0, 0]], dtype=torch.bool)

    block = PoolingByMultiheadAttention(8, num_seed_points=2, heads=2)
    assert str(block) == ('PoolingByMultiheadAttention(8, num_seed_points=2, '
                          'heads=2, layer_norm=True, dropout=0.0)')

    out = block(x, mask)
    assert out.size() == (2, 2, 8)

    jit = torch.jit.script(block)
    assert torch.allclose(jit(x, mask), out)
pytorch/test/nn/aggr/test_attention.py000066400000000000000000000015651444262304300205070ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import MLP
from torch_geometric.nn.aggr import AttentionalAggregation


def test_attentional_aggregation():
    channels = 16
    x = torch.randn(6, channels)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])

    gate_nn = MLP([channels, 1], act='relu')
    nn = MLP([channels, channels], act='relu')
    aggr = AttentionalAggregation(gate_nn, nn)
    assert str(aggr) == (f'AttentionalAggregation(gate_nn=MLP({channels}, 1), '
                         f'nn=MLP({channels}, {channels}))')

    out = aggr(x, index)
    assert out.size() == (3, channels)

    if not torch_geometric.typing.WITH_TORCH_SCATTER:
        with pytest.raises(ImportError, match="'segment' requires"):
            aggr(x, ptr=ptr)
    else:
        assert torch.allclose(out, aggr(x, ptr=ptr))
pytorch/test/nn/aggr/test_basic.py000066400000000000000000000072671444262304300175700ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import (
    MaxAggregation,
    MeanAggregation,
    MinAggregation,
    MulAggregation,
    PowerMeanAggregation,
    SoftmaxAggregation,
    StdAggregation,
    SumAggregation,
    VarAggregation,
)


def test_validate():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])

    aggr = MeanAggregation()

    with pytest.raises(ValueError, match="invalid dimension"):
        aggr(x, index, dim=-3)

    with pytest.raises(ValueError, match="invalid 'dim_size'"):
        aggr(x, ptr=ptr, dim_size=2)

    with pytest.raises(ValueError, match="invalid 'dim_size'"):
        aggr(x, index, dim_size=2)


@pytest.mark.parametrize('Aggregation', [
    MeanAggregation,
    SumAggregation,
    MaxAggregation,
    MinAggregation,
    MulAggregation,
    VarAggregation,
    StdAggregation,
])
def test_basic_aggregation(Aggregation):
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])

    aggr = Aggregation()
    assert str(aggr) == f'{Aggregation.__name__}()'

    out = aggr(x, index)
    assert out.size() == (3, x.size(1))

    if isinstance(aggr, MulAggregation):
        with pytest.raises(NotImplementedError, match="requires 'index'"):
            aggr(x, ptr=ptr)
    elif not torch_geometric.typing.WITH_TORCH_SCATTER:
        with pytest.raises(ImportError, match="'segment' requires"):
            aggr(x, ptr=ptr)
    else:
        assert torch.allclose(out, aggr(x, ptr=ptr))


def test_var_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    var_aggr = VarAggregation()
    out = var_aggr(x, index)

    mean_aggr = MeanAggregation()
    expected = mean_aggr((x - mean_aggr(x, index)[index]).pow(2), index)
    assert torch.allclose(out, expected, atol=1e-6)


def test_empty_std_aggregation():
    aggr = StdAggregation()

    x = torch.empty(0, 6).reshape(0, 6)
    index = torch.empty(0, dtype=torch.long)

    out = aggr(x, index, dim_size=5)
    assert out.size() == (5, 6)
    assert float(out.abs().sum()) == 0.0


@pytest.mark.parametrize('Aggregation', [
    SoftmaxAggregation,
    PowerMeanAggregation,
])
@pytest.mark.parametrize('learn', [True, False])
def test_learnable_aggregation(Aggregation, learn):
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])

    aggr = Aggregation(learn=learn)
    assert str(aggr) == f'{Aggregation.__name__}(learn={learn})'

    out = aggr(x, index)
    assert out.size() == (3, x.size(1))

    if not torch_geometric.typing.WITH_TORCH_SCATTER:
        with pytest.raises(ImportError, match="'segment' requires"):
            aggr(x, ptr=ptr)
    else:
        assert torch.allclose(out, aggr(x, ptr=ptr))

    if learn:
        out.mean().backward()
        for param in aggr.parameters():
            assert not torch.isnan(param.grad).any()


@pytest.mark.parametrize('Aggregation', [
    SoftmaxAggregation,
    PowerMeanAggregation,
])
def test_learnable_channels_aggregation(Aggregation):
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])

    aggr = Aggregation(learn=True, channels=16)
    assert str(aggr) == f'{Aggregation.__name__}(learn=True)'

    out = aggr(x, index)
    assert out.size() == (3, x.size(1))

    if not torch_geometric.typing.WITH_TORCH_SCATTER:
        with pytest.raises(ImportError, match="'segment' requires"):
            aggr(x, ptr=ptr)
    else:
        assert torch.allclose(out, aggr(x, ptr=ptr))

    out.mean().backward()
    for param in aggr.parameters():
        assert not torch.isnan(param.grad).any()
pytorch/test/nn/aggr/test_deep_sets.py000066400000000000000000000010471444262304300204500ustar00rootroot00000000000000import torch

from torch_geometric.nn import DeepSetsAggregation, Linear


def test_deep_sets_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = DeepSetsAggregation(
        local_nn=Linear(16, 32),
        global_nn=Linear(32, 64),
    )
    aggr.reset_parameters()
    assert str(aggr) == ('DeepSetsAggregation('
                         'local_nn=Linear(16, 32, bias=True), '
                         'global_nn=Linear(32, 64, bias=True))')

    out = aggr(x, index)
    assert out.size() == (3, 64)
pytorch/test/nn/aggr/test_equilibrium.py000066400000000000000000000026051444262304300210250ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn.aggr import EquilibriumAggregation


@pytest.mark.parametrize('iter', [0, 1, 5])
@pytest.mark.parametrize('alpha', [0, .1, 5])
def test_equilibrium(iter, alpha):
    batch_size = 10
    feature_channels = 3
    output_channels = 2
    x = torch.randn(batch_size, feature_channels)
    model = EquilibriumAggregation(feature_channels, output_channels,
                                   num_layers=[10, 10], grad_iter=iter)

    assert str(model) == 'EquilibriumAggregation()'
    out = model(x)
    assert out.size() == (1, 2)

    out = model(x, dim_size=3)
    assert out.size() == (3, 2)
    assert torch.all(out[1:, :] == 0)


@pytest.mark.parametrize('iter', [0, 1, 5])
@pytest.mark.parametrize('alpha', [0, .1, 5])
def test_equilibrium_batch(iter, alpha):
    batch_1, batch_2 = 4, 6
    feature_channels = 3
    output_channels = 2
    x = torch.randn(batch_1 + batch_2, feature_channels)
    batch = torch.tensor([0 for _ in range(batch_1)] +
                         [1 for _ in range(batch_2)])

    model = EquilibriumAggregation(feature_channels, output_channels,
                                   num_layers=[10, 10], grad_iter=iter)

    assert str(model) == 'EquilibriumAggregation()'
    out = model(x, batch)
    assert out.size() == (2, 2)

    out = model(x, dim_size=3)
    assert out.size() == (3, 2)
    assert torch.all(out[1:, :] == 0)
pytorch/test/nn/aggr/test_fused.py000066400000000000000000000054601444262304300176060ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn.aggr.fused import FusedAggregation
from torch_geometric.nn.resolver import aggregation_resolver
from torch_geometric.profile import benchmark


@pytest.mark.parametrize('aggrs', [
    ['sum', 'mean', 'min', 'max', 'mul', 'var', 'std'],
    ['sum', 'min', 'max', 'mul', 'var', 'std'],
    ['min', 'max', 'mul', 'var', 'std'],
    ['mean', 'min', 'max', 'mul', 'var', 'std'],
    ['sum', 'min', 'max', 'mul', 'std'],
    ['mean', 'min', 'max', 'mul', 'std'],
    ['min', 'max', 'mul', 'std'],
])
def test_fused_aggregation(aggrs):
    aggrs = [aggregation_resolver(aggr) for aggr in aggrs]

    x = torch.randn(6, 1)
    y = x.clone()
    index = torch.tensor([0, 0, 1, 1, 1, 3])

    x.requires_grad_(True)
    y.requires_grad_(True)

    aggr = FusedAggregation(aggrs)
    assert str(aggr) == 'FusedAggregation()'
    out = torch.cat(aggr(x, index), dim=-1)

    expected = torch.cat([aggr(y, index) for aggr in aggrs], dim=-1)
    assert torch.allclose(out, expected, atol=1e-5)

    jit = torch.jit.script(aggr)
    assert torch.allclose(torch.cat(jit(x, index), dim=-1), out, atol=1e-5)

    out.mean().backward()
    assert x.grad is not None
    expected.mean().backward()
    assert y.grad is not None
    assert torch.allclose(x.grad, y.grad, atol=1e-5)


def test_empty_fused_std_aggregation():
    aggrs = [aggregation_resolver(aggr) for aggr in ['mean', 'var', 'std']]
    aggr = FusedAggregation(aggrs)

    x = torch.empty(0, 6).reshape(0, 6)
    index = torch.empty(0, dtype=torch.long)

    out = torch.cat(aggr(x, index, dim_size=5), dim=-1)
    assert out.size() == (5, 18)
    assert float(out.abs().sum()) == 0.0


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 1_000, 50_000
    x = torch.randn(num_edges, 64, device=args.device)
    index = torch.randint(num_nodes, (num_edges, ), device=args.device)

    aggrs = ['sum', 'mean', 'max', 'std']
    print(f'Aggregators: {", ".join(aggrs)}')

    aggrs = [aggregation_resolver(aggr) for aggr in aggrs]
    fused_aggregation = FusedAggregation(aggrs)

    def naive_aggr(x, index, dim_size):
        outs = [aggr(x, index, dim_size=dim_size) for aggr in aggrs]
        return torch.cat(outs, dim=-1)

    def fused_aggr(x, index, dim_size):
        outs = fused_aggregation(x, index, dim_size=dim_size)
        return torch.cat(outs, dim=-1)

    benchmark(
        funcs=[naive_aggr, fused_aggr],
        func_names=['Naive', 'Fused'],
        args=(x, index, num_nodes),
        num_steps=100 if args.device == 'cpu' else 1000,
        num_warmups=50 if args.device == 'cpu' else 500,
        backward=args.backward,
    )
pytorch/test/nn/aggr/test_gmt.py000066400000000000000000000011541444262304300172630ustar00rootroot00000000000000import torch

from torch_geometric.nn.aggr import GraphMultisetTransformer
from torch_geometric.testing import is_full_test


def test_graph_multiset_transformer():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = GraphMultisetTransformer(16, k=2, heads=2)
    aggr.reset_parameters()
    assert str(aggr) == ('GraphMultisetTransformer(16, k=2, heads=2, '
                         'layer_norm=False, dropout=0.0)')

    out = aggr(x, index)
    assert out.size() == (3, 16)

    if is_full_test():
        jit = torch.jit.script(aggr)
        assert torch.allclose(jit(x, index), out)
pytorch/test/nn/aggr/test_gru.py000066400000000000000000000004611444262304300172710ustar00rootroot00000000000000import torch

from torch_geometric.nn import GRUAggregation


def test_gru_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = GRUAggregation(16, 32)
    assert str(aggr) == 'GRUAggregation(16, 32)'

    out = aggr(x, index)
    assert out.size() == (3, 32)
pytorch/test/nn/aggr/test_lstm.py000066400000000000000000000006611444262304300174550ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import LSTMAggregation


def test_lstm_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = LSTMAggregation(16, 32)
    assert str(aggr) == 'LSTMAggregation(16, 32)'

    with pytest.raises(ValueError, match="is not sorted"):
        aggr(x, torch.tensor([0, 1, 0, 1, 2, 1]))

    out = aggr(x, index)
    assert out.size() == (3, 32)
pytorch/test/nn/aggr/test_mlp_aggr.py000066400000000000000000000006471444262304300202720ustar00rootroot00000000000000import torch

from torch_geometric.nn import MLPAggregation


def test_mlp_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = MLPAggregation(
        in_channels=16,
        out_channels=32,
        max_num_elements=3,
        num_layers=1,
    )
    assert str(aggr) == 'MLPAggregation(16, 32, max_num_elements=3)'

    out = aggr(x, index)
    assert out.size() == (3, 32)
pytorch/test/nn/aggr/test_multi.py000066400000000000000000000032531444262304300176300ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import MultiAggregation


@pytest.mark.parametrize('multi_aggr_tuple', [
    (dict(mode='cat'), 3),
    (dict(mode='proj', mode_kwargs=dict(in_channels=16, out_channels=16)), 1),
    (dict(mode='attn', mode_kwargs=dict(in_channels=16, out_channels=16,
                                        num_heads=4)), 1),
    (dict(mode='sum'), 1),
    (dict(mode='mean'), 1),
    (dict(mode='max'), 1),
    (dict(mode='min'), 1),
    (dict(mode='logsumexp'), 1),
    (dict(mode='std'), 1),
    (dict(mode='var'), 1),
])
def test_multi_aggr(multi_aggr_tuple):
    # The 'cat' combine mode will expand the output dimensions by
    # the number of aggregators which is 3 here, while the other
    # modes keep output dimensions unchanged.
    aggr_kwargs, expand = multi_aggr_tuple
    x = torch.randn(7, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2, 3])
    ptr = torch.tensor([0, 2, 5, 6, 7])

    aggrs = ['mean', 'sum', 'max']
    aggr = MultiAggregation(aggrs, **aggr_kwargs)
    aggr.reset_parameters()
    assert str(aggr) == ('MultiAggregation([\n'
                         '  MeanAggregation(),\n'
                         '  SumAggregation(),\n'
                         '  MaxAggregation(),\n'
                         f"], mode={aggr_kwargs['mode']})")

    out = aggr(x, index)
    assert out.size() == (4, expand * x.size(1))

    if not torch_geometric.typing.WITH_TORCH_SCATTER:
        with pytest.raises(ImportError, match="'segment' requires"):
            aggr(x, ptr=ptr)
    else:
        assert torch.allclose(out, aggr(x, ptr=ptr))

    jit = torch.jit.script(aggr)
    assert torch.allclose(out, jit(x, index))
pytorch/test/nn/aggr/test_quantile.py000066400000000000000000000060651444262304300203240ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import MedianAggregation, QuantileAggregation


@pytest.mark.parametrize('q', [0., .1, .2, .3, .4, .5, .6, .7, .8, .9, 1.])
@pytest.mark.parametrize('interpolation', QuantileAggregation.interpolations)
@pytest.mark.parametrize('dim', [0, 1])
@pytest.mark.parametrize('dim_size', [None, 15])
@pytest.mark.parametrize('fill_value', [0.0, 10.0])
def test_quantile_aggregation(q, interpolation, dim, dim_size, fill_value):
    x = torch.tensor([
        [0.0, 1.0, 2.0],
        [3.0, 4.0, 5.0],
        [6.0, 7.0, 8.0],
        [9.0, 0.0, 1.0],
        [2.0, 3.0, 4.0],
        [5.0, 6.0, 7.0],
        [8.0, 9.0, 0.0],
        [1.0, 2.0, 3.0],
        [4.0, 5.0, 6.0],
        [7.0, 8.0, 9.0],
    ])
    index = torch.zeros(x.size(dim), dtype=torch.long)

    aggr = QuantileAggregation(q=q, interpolation=interpolation,
                               fill_value=fill_value)
    assert str(aggr) == f"QuantileAggregation(q={q})"

    out = aggr(x, index, dim=dim, dim_size=dim_size)
    expected = x.quantile(q, dim, interpolation=interpolation, keepdim=True)

    assert torch.allclose(out.narrow(dim, 0, 1), expected)

    if out.size(0) > index.max() + 1:
        padding = out.narrow(dim, 1, out.size(dim) - 1)
        assert torch.allclose(padding, torch.tensor(fill_value))


def test_median_aggregation():
    x = torch.tensor([
        [0.0, 1.0, 2.0],
        [3.0, 4.0, 5.0],
        [6.0, 7.0, 8.0],
        [9.0, 0.0, 1.0],
        [2.0, 3.0, 4.0],
        [5.0, 6.0, 7.0],
        [8.0, 9.0, 0.0],
        [1.0, 2.0, 3.0],
        [4.0, 5.0, 6.0],
        [7.0, 8.0, 9.0],
    ])

    aggr = MedianAggregation()
    assert str(aggr) == "MedianAggregation()"

    index = torch.tensor([0, 0, 0, 0, 1, 1, 1, 2, 2, 2])
    assert aggr(x, index).tolist() == [
        [3.0, 1.0, 2.0],
        [5.0, 6.0, 4.0],
        [4.0, 5.0, 6.0],
    ]

    index = torch.tensor([0, 1, 0])
    assert aggr(x, index, dim=1).tolist() == [
        [0.0, 1.0],
        [3.0, 4.0],
        [6.0, 7.0],
        [1.0, 0.0],
        [2.0, 3.0],
        [5.0, 6.0],
        [0.0, 9.0],
        [1.0, 2.0],
        [4.0, 5.0],
        [7.0, 8.0],
    ]


def test_quantile_aggregation_multi():
    x = torch.tensor([
        [0.0, 1.0, 2.0],
        [3.0, 4.0, 5.0],
        [6.0, 7.0, 8.0],
        [9.0, 0.0, 1.0],
        [2.0, 3.0, 4.0],
        [5.0, 6.0, 7.0],
        [8.0, 9.0, 0.0],
        [1.0, 2.0, 3.0],
        [4.0, 5.0, 6.0],
        [7.0, 8.0, 9.0],
    ])
    index = torch.tensor([0, 0, 0, 0, 1, 1, 1, 2, 2, 2])

    qs = [0.25, 0.5, 0.75]

    assert torch.allclose(
        QuantileAggregation(qs)(x, index),
        torch.cat([QuantileAggregation(q)(x, index) for q in qs], dim=-1),
    )


def test_quantile_aggregation_validate():
    with pytest.raises(ValueError, match="at least one quantile"):
        QuantileAggregation(q=[])

    with pytest.raises(ValueError, match="must be in the range"):
        QuantileAggregation(q=-1)

    with pytest.raises(ValueError, match="Invalid interpolation method"):
        QuantileAggregation(q=0.5, interpolation=None)
pytorch/test/nn/aggr/test_scaler.py000066400000000000000000000014371444262304300177510ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import DegreeScalerAggregation


@pytest.mark.parametrize('train_norm', [True, False])
def test_degree_scaler_aggregation(train_norm):
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])
    ptr = torch.tensor([0, 2, 5, 6])
    deg = torch.tensor([0, 3, 0, 1, 1, 0])

    aggr = ['mean', 'sum', 'max']
    scaler = [
        'identity', 'amplification', 'attenuation', 'linear', 'inverse_linear'
    ]
    aggr = DegreeScalerAggregation(aggr, scaler, deg, train_norm=train_norm)
    assert str(aggr) == 'DegreeScalerAggregation()'

    out = aggr(x, index)
    assert out.size() == (3, 240)
    assert torch.allclose(torch.jit.script(aggr)(x, index), out)

    with pytest.raises(NotImplementedError):
        aggr(x, ptr=ptr)
pytorch/test/nn/aggr/test_set2set.py000066400000000000000000000015341444262304300200670ustar00rootroot00000000000000import torch

from torch_geometric.nn.aggr import Set2Set


def test_set2set():
    set2set = Set2Set(in_channels=2, processing_steps=1)
    assert str(set2set) == 'Set2Set(2, 4)'

    N = 4
    x_1, batch_1 = torch.randn(N, 2), torch.zeros(N, dtype=torch.long)
    out_1 = set2set(x_1, batch_1).view(-1)

    N = 6
    x_2, batch_2 = torch.randn(N, 2), torch.zeros(N, dtype=torch.long)
    out_2 = set2set(x_2, batch_2).view(-1)

    x, batch = torch.cat([x_1, x_2]), torch.cat([batch_1, batch_2 + 1])
    out = set2set(x, batch)
    assert out.size() == (2, 4)
    assert torch.allclose(out_1, out[0])
    assert torch.allclose(out_2, out[1])

    x, batch = torch.cat([x_2, x_1]), torch.cat([batch_2, batch_1 + 1])
    out = set2set(x, batch)
    assert out.size() == (2, 4)
    assert torch.allclose(out_1, out[1])
    assert torch.allclose(out_2, out[0])
pytorch/test/nn/aggr/test_set_transformer.py000066400000000000000000000012201444262304300217030ustar00rootroot00000000000000import torch

from torch_geometric.nn.aggr import SetTransformerAggregation
from torch_geometric.testing import is_full_test


def test_set_transformer_aggregation():
    x = torch.randn(6, 16)
    index = torch.tensor([0, 0, 1, 1, 1, 2])

    aggr = SetTransformerAggregation(16, num_seed_points=2, heads=2)
    aggr.reset_parameters()
    assert str(aggr) == ('SetTransformerAggregation(16, num_seed_points=2, '
                         'heads=2, layer_norm=False, dropout=0.0)')

    out = aggr(x, index)
    assert out.size() == (3, 2 * 16)

    if is_full_test():
        jit = torch.jit.script(aggr)
        assert torch.allclose(jit(x, index), out)
pytorch/test/nn/aggr/test_sort.py000066400000000000000000000040721444262304300174650ustar00rootroot00000000000000import torch

from torch_geometric.nn.aggr import SortAggregation


def test_sort_aggregation():
    N_1, N_2 = 4, 6
    x = torch.randn(N_1 + N_2, 4)
    index = torch.tensor([0 for _ in range(N_1)] + [1 for _ in range(N_2)])

    aggr = SortAggregation(k=5)
    assert str(aggr) == 'SortAggregation(k=5)'

    out = aggr(x, index)
    assert out.size() == (2, 5 * 4)

    out_dim = out = aggr(x, index, dim=0)
    assert torch.allclose(out_dim, out)

    out = out.view(2, 5, 4)

    # First graph output has been filled up with zeros.
    assert out[0, -1].tolist() == [0, 0, 0, 0]

    # Nodes are sorted.
    assert torch.equal(out[0, :4, -1].argsort(), 3 - torch.arange(4))
    assert torch.equal(out[1, :, -1].argsort(), 4 - torch.arange(5))


def test_sort_aggregation_smaller_than_k():
    N_1, N_2 = 4, 6
    x = torch.randn(N_1 + N_2, 4)
    index = torch.tensor([0 for _ in range(N_1)] + [1 for _ in range(N_2)])

    # Set k which is bigger than both N_1=4 and N_2=6.
    aggr = SortAggregation(k=10)
    assert str(aggr) == 'SortAggregation(k=10)'

    out = aggr(x, index)
    assert out.size() == (2, 10 * 4)

    out_dim = out = aggr(x, index, dim=0)
    assert torch.allclose(out_dim, out)

    out = out.view(2, 10, 4)

    # Both graph outputs have been filled up with zeros.
    assert out[0, -1].tolist() == [0, 0, 0, 0]
    assert out[1, -1].tolist() == [0, 0, 0, 0]

    # Nodes are sorted.
    assert torch.equal(out[0, :4, -1].argsort(), 3 - torch.arange(4))
    assert torch.equal(out[1, :6, -1].argsort(), 5 - torch.arange(6))


def test_sort_aggregation_dim_size():
    N_1, N_2 = 4, 6
    x = torch.randn(N_1 + N_2, 4)
    index = torch.tensor([0 for _ in range(N_1)] + [1 for _ in range(N_2)])

    aggr = SortAggregation(k=5)
    assert str(aggr) == 'SortAggregation(k=5)'

    # expand batch output by 1
    out = aggr(x, index, dim_size=3)
    assert out.size() == (3, 5 * 4)

    out = out.view(3, 5, 4)

    # Both first and last graph outputs have been filled up with zeros.
    assert out[0, -1].tolist() == [0, 0, 0, 0]
    assert out[2, -1].tolist() == [0, 0, 0, 0]
pytorch/test/nn/attention/000077500000000000000000000000001444262304300161475ustar00rootroot00000000000000pytorch/test/nn/attention/test_performer.py000066400000000000000000000006351444262304300215650ustar00rootroot00000000000000import torch

from torch_geometric.nn.attention import PerformerAttention


def test_performer_attention():
    x = torch.randn(1, 4, 16)
    mask = torch.ones([1, 4], dtype=torch.bool)
    attn = PerformerAttention(channels=16, heads=4)
    out = attn(x, mask)
    assert out.shape == (1, 4, 16)
    assert str(attn) == ('PerformerAttention(heads=4, '
                         'head_channels=64 kernel=ReLU())')
pytorch/test/nn/conv/000077500000000000000000000000001444262304300151075ustar00rootroot00000000000000pytorch/test/nn/conv/cugraph/000077500000000000000000000000001444262304300165405ustar00rootroot00000000000000pytorch/test/nn/conv/cugraph/test_cugraph_gat_conv.py000066400000000000000000000041561444262304300234700ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import CuGraphGATConv, GATConv
from torch_geometric.testing import onlyCUDA, withPackage


@onlyCUDA
@withPackage('pylibcugraphops>=23.02')
@pytest.mark.parametrize('bias', [True, False])
@pytest.mark.parametrize('bipartite', [True, False])
@pytest.mark.parametrize('concat', [True, False])
@pytest.mark.parametrize('heads', [1, 2, 3])
@pytest.mark.parametrize('max_num_neighbors', [8, None])
def test_gat_conv_equality(bias, bipartite, concat, heads, max_num_neighbors):
    in_channels, out_channels = (5, 2)
    kwargs = dict(bias=bias, concat=concat)

    size = (10, 8) if bipartite else (10, 10)
    x = torch.rand(size[0], in_channels, device='cuda')
    edge_index = torch.tensor([
        [7, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 8, 9],
        [0, 1, 2, 3, 4, 5, 6, 7, 7, 7, 7, 7, 7, 7, 7],
    ], device='cuda')

    conv1 = GATConv(in_channels, out_channels, heads, add_self_loops=False,
                    **kwargs).cuda()
    conv2 = CuGraphGATConv(in_channels, out_channels, heads, **kwargs).cuda()

    with torch.no_grad():
        conv2.lin.weight.data[:, :] = conv1.lin_src.weight.data
        conv2.att.data[:heads * out_channels] = conv1.att_src.data.flatten()
        conv2.att.data[heads * out_channels:] = conv1.att_dst.data.flatten()

    if bipartite:
        out1 = conv1((x, x[:size[1]]), edge_index)
    else:
        out1 = conv1(x, edge_index)

    csc = CuGraphGATConv.to_csc(edge_index, size)
    out2 = conv2(x, csc, max_num_neighbors=max_num_neighbors)
    assert torch.allclose(out1, out2, atol=1e-3)

    grad_output = torch.rand_like(out1)
    out1.backward(grad_output)
    out2.backward(grad_output)

    assert torch.allclose(conv1.lin_src.weight.grad, conv2.lin.weight.grad,
                          atol=1e-3)
    assert torch.allclose(conv1.att_src.grad.flatten(),
                          conv2.att.grad[:heads * out_channels], atol=1e-3)
    assert torch.allclose(conv1.att_dst.grad.flatten(),
                          conv2.att.grad[heads * out_channels:], atol=1e-3)
    if bias:
        assert torch.allclose(conv1.bias.grad, conv2.bias.grad, atol=1e-3)
pytorch/test/nn/conv/cugraph/test_cugraph_rgcn_conv.py000066400000000000000000000043651444262304300236500ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import CuGraphRGCNConv
from torch_geometric.nn import FastRGCNConv as RGCNConv
from torch_geometric.testing import onlyCUDA, withPackage


@onlyCUDA
@withPackage('pylibcugraphops>=23.02')
@pytest.mark.parametrize('aggr', ['add', 'sum', 'mean'])
@pytest.mark.parametrize('bias', [True, False])
@pytest.mark.parametrize('bipartite', [True, False])
@pytest.mark.parametrize('max_num_neighbors', [8, None])
@pytest.mark.parametrize('num_bases', [1, 2, None])
@pytest.mark.parametrize('root_weight', [True, False])
def test_rgcn_conv_equality(aggr, bias, bipartite, max_num_neighbors,
                            num_bases, root_weight):

    in_channels, out_channels, num_relations = (4, 2, 3)
    kwargs = dict(aggr=aggr, bias=bias, num_bases=num_bases,
                  root_weight=root_weight)

    size = (10, 8) if bipartite else (10, 10)
    x = torch.rand(size[0], in_channels, device='cuda')
    edge_index = torch.tensor([
        [7, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 8, 9],
        [0, 1, 2, 3, 4, 5, 6, 7, 7, 7, 7, 7, 7, 7, 7],
    ], device='cuda')
    edge_type = torch.tensor([1, 2, 1, 0, 2, 1, 2, 0, 2, 2, 1, 1, 1, 2,
                              2]).cuda()

    torch.manual_seed(12345)
    conv1 = RGCNConv(in_channels, out_channels, num_relations, **kwargs).cuda()
    torch.manual_seed(12345)
    conv2 = CuGraphRGCNConv(in_channels, out_channels, num_relations,
                            **kwargs).cuda()

    if bipartite:
        out1 = conv1((x, x[:size[1]]), edge_index, edge_type)
    else:
        out1 = conv1(x, edge_index, edge_type)

    csc, edge_type = CuGraphRGCNConv.to_csc(edge_index, size, edge_type)
    out2 = conv2(x, csc, edge_type, max_num_neighbors=max_num_neighbors)
    assert torch.allclose(out1, out2, atol=1e-3)

    grad_out = torch.rand_like(out1)
    out1.backward(grad_out)
    out2.backward(grad_out)

    end = -1 if root_weight else None
    assert torch.allclose(conv1.weight.grad, conv2.weight.grad[:end],
                          atol=1e-3)

    if root_weight:
        assert torch.allclose(conv1.root.grad, conv2.weight.grad[-1],
                              atol=1e-3)

    if num_bases is not None:
        assert torch.allclose(conv1.comp.grad, conv2.comp.grad, atol=1e-3)
pytorch/test/nn/conv/cugraph/test_cugraph_sage_conv.py000066400000000000000000000043751444262304300236370ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import CuGraphSAGEConv, SAGEConv
from torch_geometric.testing import onlyCUDA, withPackage


@onlyCUDA
@withPackage('pylibcugraphops>=23.02')
@pytest.mark.parametrize('aggr', ['sum', 'mean', 'min', 'max'])
@pytest.mark.parametrize('bias', [True, False])
@pytest.mark.parametrize('bipartite', [True, False])
@pytest.mark.parametrize('max_num_neighbors', [8, None])
@pytest.mark.parametrize('normalize', [True, False])
@pytest.mark.parametrize('root_weight', [True, False])
def test_sage_conv_equality(aggr, bias, bipartite, max_num_neighbors,
                            normalize, root_weight):

    in_channels, out_channels = (8, 16)
    kwargs = dict(aggr=aggr, bias=bias, normalize=normalize,
                  root_weight=root_weight)

    size = (10, 8) if bipartite else (10, 10)
    x = torch.rand(size[0], in_channels, device='cuda')
    edge_index = torch.tensor([
        [7, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 8, 9],
        [0, 1, 2, 3, 4, 5, 6, 7, 7, 7, 7, 7, 7, 7, 7],
    ], device='cuda')

    conv1 = SAGEConv(in_channels, out_channels, **kwargs).cuda()
    conv2 = CuGraphSAGEConv(in_channels, out_channels, **kwargs).cuda()

    with torch.no_grad():
        conv2.lin.weight.data[:, :in_channels] = conv1.lin_l.weight.data
        if root_weight:
            conv2.lin.weight.data[:, in_channels:] = conv1.lin_r.weight.data
        if bias:
            conv2.lin.bias.data[:] = conv1.lin_l.bias.data

    if bipartite:
        out1 = conv1((x, x[:size[1]]), edge_index)
    else:
        out1 = conv1(x, edge_index)

    csc = CuGraphSAGEConv.to_csc(edge_index, size)
    out2 = conv2(x, csc, max_num_neighbors=max_num_neighbors)
    assert torch.allclose(out1, out2, atol=1e-6)

    grad_out = torch.rand_like(out1)
    out1.backward(grad_out)
    out2.backward(grad_out)

    assert torch.allclose(
        conv1.lin_l.weight.grad,
        conv2.lin.weight.grad[:, :in_channels],
        atol=1e-6,
    )

    if root_weight:
        assert torch.allclose(
            conv1.lin_r.weight.grad,
            conv2.lin.weight.grad[:, in_channels:],
            atol=1e-6,
        )

    if bias:
        assert torch.allclose(
            conv1.lin_l.bias.grad,
            conv2.lin.bias.grad,
            atol=1e-6,
        )
pytorch/test/nn/conv/test_agnn_conv.py000066400000000000000000000024361444262304300204750ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import AGNNConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


@pytest.mark.parametrize('requires_grad', [True, False])
def test_agnn_conv(requires_grad):
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = AGNNConv(requires_grad=requires_grad)
    assert str(conv) == 'AGNNConv()'
    out = conv(x, edge_index)
    assert out.size() == (4, 16)
    assert torch.allclose(conv(x, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_antisymmetric_conv.py000066400000000000000000000023351444262304300224400ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import AntiSymmetricConv
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_antisymmetric_conv():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = AntiSymmetricConv(8)
    assert str(conv) == ('AntiSymmetricConv(8, phi=GCNConv(8, 8), '
                         'num_iters=1, epsilon=0.1, gamma=0.1)')

    out1 = conv(x, edge_index)
    assert out1.size() == (4, 8)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 8)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)
pytorch/test/nn/conv/test_appnp.py000066400000000000000000000043051444262304300176400ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import APPNP
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_appnp():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = APPNP(K=3, alpha=0.1, cached=True)
    assert str(conv) == 'APPNP(K=3, alpha=0.1)'
    out = conv(x, edge_index)
    assert out.size() == (4, 16)
    assert torch.allclose(conv(x, adj1.t()), out)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out)

    # Run again to test the cached functionality:
    assert conv._cached_edge_index is not None
    assert torch.allclose(conv(x, edge_index), conv(x, adj1.t()))
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, edge_index), conv(x, adj2.t()))

    conv.reset_parameters()
    assert conv._cached_edge_index is None
    assert conv._cached_adj_t is None

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out)


def test_appnp_dropout():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    # With dropout probability of 1.0, the final output equals to alpha * x:
    conv = APPNP(K=2, alpha=0.1, dropout=1.0)
    assert torch.allclose(0.1 * x, conv(x, edge_index))
    assert torch.allclose(0.1 * x, conv(x, adj1.t()))

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(0.1 * x, conv(x, adj2.t()))
pytorch/test/nn/conv/test_arma_conv.py000066400000000000000000000032001444262304300204600ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import ARMAConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_arma_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = ARMAConv(16, 32, num_stacks=8, num_layers=4)
    assert str(conv) == 'ARMAConv(16, 32, num_stacks=8, num_layers=4)'
    out = conv(x, edge_index)
    assert out.size() == (4, 32)
    with pytest.raises(RuntimeError):  # No 3D feature tensor support.
        assert torch.allclose(conv(x, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out, atol=1e-6)


def test_lazy_arma_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = ARMAConv(-1, 32, num_stacks=8, num_layers=4)
    assert str(conv) == 'ARMAConv(-1, 32, num_stacks=8, num_layers=4)'
    out = conv(x, edge_index)
    assert out.size() == (4, 32)
pytorch/test/nn/conv/test_cg_conv.py000066400000000000000000000077011444262304300201430ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import CGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csr_tensor


@pytest.mark.parametrize('batch_norm', [False, True])
def test_cg_conv(batch_norm):
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csr_tensor(edge_index, size=(4, 4))

    conv = CGConv(8, batch_norm=batch_norm)
    assert str(conv) == 'CGConv(8, dim=0)'
    out = conv(x1, edge_index)
    assert out.size() == (4, 8)
    assert torch.allclose(conv(x1, adj1.t()), out)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csr_tensor(edge_index, size=(4, 2))

    conv = CGConv((8, 16))
    assert str(conv) == 'CGConv((8, 16), dim=0)'
    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 16)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)


def test_cg_conv_with_edge_features():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1), 3)

    conv = CGConv(8, dim=3)
    assert str(conv) == 'CGConv(8, dim=3)'
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 8)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, value), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out)

    # Test bipartite message passing:
    conv = CGConv((8, 16), dim=3)
    assert str(conv) == 'CGConv((8, 16), dim=3)'
    out = conv((x1, x2), edge_index, value)
    assert out.size() == (2, 16)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out)

    if is_full_test():
        t = '(PairTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out)
pytorch/test/nn/conv/test_cheb_conv.py000066400000000000000000000052041444262304300204470ustar00rootroot00000000000000import torch

from torch_geometric.data import Batch, Data
from torch_geometric.nn import ChebConv
from torch_geometric.testing import is_full_test


def test_cheb_conv():
    in_channels, out_channels = (16, 32)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    num_nodes = edge_index.max().item() + 1
    edge_weight = torch.rand(edge_index.size(1))
    x = torch.randn((num_nodes, in_channels))

    conv = ChebConv(in_channels, out_channels, K=3)
    assert str(conv) == 'ChebConv(16, 32, K=3, normalization=sym)'
    out1 = conv(x, edge_index)
    assert out1.size() == (num_nodes, out_channels)
    out2 = conv(x, edge_index, edge_weight)
    assert out2.size() == (num_nodes, out_channels)
    out3 = conv(x, edge_index, edge_weight, lambda_max=3.0)
    assert out3.size() == (num_nodes, out_channels)

    if is_full_test():
        jit = torch.jit.script(conv.jittable())
        assert torch.allclose(jit(x, edge_index), out1)
        assert torch.allclose(jit(x, edge_index, edge_weight), out2)
        assert torch.allclose(
            jit(x, edge_index, edge_weight, lambda_max=torch.tensor(3.0)),
            out3)

    batch = torch.tensor([0, 0, 1, 1])
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 3, 2]])
    num_nodes = edge_index.max().item() + 1
    edge_weight = torch.rand(edge_index.size(1))
    x = torch.randn((num_nodes, in_channels))
    lambda_max = torch.tensor([2.0, 3.0])

    out4 = conv(x, edge_index, edge_weight, batch)
    assert out4.size() == (num_nodes, out_channels)
    out5 = conv(x, edge_index, edge_weight, batch, lambda_max)
    assert out5.size() == (num_nodes, out_channels)

    if is_full_test():
        assert torch.allclose(jit(x, edge_index, edge_weight, batch), out4)
        assert torch.allclose(
            jit(x, edge_index, edge_weight, batch, lambda_max), out5)


def test_cheb_conv_batch():
    x1 = torch.randn(4, 8)
    edge_index1 = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    edge_weight1 = torch.rand(edge_index1.size(1))
    data1 = Data(x=x1, edge_index=edge_index1, edge_weight=edge_weight1)

    x2 = torch.randn(3, 8)
    edge_index2 = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight2 = torch.rand(edge_index2.size(1))
    data2 = Data(x=x2, edge_index=edge_index2, edge_weight=edge_weight2)

    conv = ChebConv(8, 16, K=2)

    out1 = conv(x1, edge_index1, edge_weight1)
    out2 = conv(x2, edge_index2, edge_weight2)

    batch = Batch.from_data_list([data1, data2])
    out = conv(batch.x, batch.edge_index, batch.edge_weight, batch.batch)

    assert out.size() == (7, 16)
    assert torch.allclose(out1, out[:4])
    assert torch.allclose(out2, out[4:])
pytorch/test/nn/conv/test_cluster_gcn_conv.py000066400000000000000000000023361444262304300220610ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import ClusterGCNConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_cluster_gcn_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = ClusterGCNConv(16, 32, diag_lambda=1.)
    assert str(conv) == 'ClusterGCNConv(16, 32, diag_lambda=1.0)'
    out = conv(x, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out, atol=1e-5)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-5)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out, atol=1e-5)
pytorch/test/nn/conv/test_create_gnn.py000066400000000000000000000020431444262304300206240ustar00rootroot00000000000000import torch

from torch_geometric.nn import MessagePassing
from torch_geometric.utils import add_self_loops, degree


class GCNConv(MessagePassing):
    def __init__(self, in_channels, out_channels):
        super().__init__(aggr='add')
        self.lin = torch.nn.Linear(in_channels, out_channels)

    def forward(self, x, edge_index):
        edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))

        row, col = edge_index
        deg = degree(row, x.size(0), dtype=x.dtype)
        deg_inv_sqrt = deg.pow(-0.5)
        norm = deg_inv_sqrt[row] * deg_inv_sqrt[col]

        x = self.lin(x)
        return self.propagate(edge_index, size=(x.size(0), x.size(0)), x=x,
                              norm=norm)

    def message(self, x_j, norm):
        return norm.view(-1, 1) * x_j

    def update(self, aggr_out):
        return aggr_out


def test_create_gnn():
    conv = GCNConv(16, 32)
    x = torch.randn(5, 16)
    edge_index = torch.randint(5, (2, 64), dtype=torch.long)
    out = conv(x, edge_index)
    assert out.size() == (5, 32)
pytorch/test/nn/conv/test_dir_gnn_conv.py000066400000000000000000000011061444262304300211630ustar00rootroot00000000000000import torch

from torch_geometric.nn import DirGNNConv, SAGEConv


def test_dir_gnn_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2], [1, 2, 3]])

    conv = DirGNNConv(SAGEConv(16, 32))
    assert str(conv) == 'DirGNNConv(SAGEConv(16, 32, aggr=mean), alpha=0.5)'

    out = conv(x, edge_index)
    assert out.size() == (4, 32)


def test_static_dir_gnn_conv():
    x = torch.randn(3, 4, 16)
    edge_index = torch.tensor([[0, 1, 2], [1, 2, 3]])

    conv = DirGNNConv(SAGEConv(16, 32))

    out = conv(x, edge_index)
    assert out.size() == (3, 4, 32)
pytorch/test/nn/conv/test_dna_conv.py000066400000000000000000000073421444262304300203150ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import DNAConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


@pytest.mark.parametrize('channels', [32])
@pytest.mark.parametrize('num_layers', [3])
def test_dna_conv(channels, num_layers):
    x = torch.randn((4, num_layers, channels))
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = DNAConv(channels, heads=4, groups=8, dropout=0.0)
    assert str(conv) == 'DNAConv(32, heads=4, groups=8)'
    out = conv(x, edge_index)
    assert out.size() == (4, channels)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)

    conv = DNAConv(channels, heads=1, groups=1, dropout=0.0)
    assert str(conv) == 'DNAConv(32, heads=1, groups=1)'
    out = conv(x, edge_index)
    assert out.size() == (4, channels)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)

    conv = DNAConv(channels, heads=1, groups=1, dropout=0.0, cached=True)
    out = conv(x, edge_index)
    assert conv._cached_edge_index is not None
    out = conv(x, edge_index)
    assert out.size() == (4, channels)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)


@pytest.mark.parametrize('channels', [32])
@pytest.mark.parametrize('num_layers', [3])
def test_dna_conv_sparse_tensor(channels, num_layers):
    x = torch.randn((4, num_layers, channels))
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = DNAConv(32, heads=4, groups=8, dropout=0.0)
    assert str(conv) == 'DNAConv(32, heads=4, groups=8)'
    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)
    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)

    conv = DNAConv(channels, heads=1, groups=1, dropout=0.0, cached=True)

    out1 = conv(x, adj1.t())
    assert conv._cached_edge_index is not None
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
pytorch/test/nn/conv/test_edge_conv.py000066400000000000000000000104161444262304300204530ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.typing
from torch_geometric.nn import DynamicEdgeConv, EdgeConv
from torch_geometric.testing import is_full_test, withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_edge_conv_conv():
    x1 = torch.randn(4, 16)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    nn = Seq(Lin(32, 16), ReLU(), Lin(16, 32))
    conv = EdgeConv(nn)
    assert str(conv) == (
        'EdgeConv(nn=Sequential(\n'
        '  (0): Linear(in_features=32, out_features=16, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=16, out_features=32, bias=True)\n'
        '))')
    out = conv(x1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv((x1, x1), edge_index), out, atol=1e-6)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)
    assert torch.allclose(conv((x1, x1), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)
        assert torch.allclose(conv((x1, x1), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)

        t = '(PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x1), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

        t = '(PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x1), adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)


@withPackage('torch_cluster')
def test_dynamic_edge_conv_conv():
    x1 = torch.randn(8, 16)
    x2 = torch.randn(4, 16)
    batch1 = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])
    batch2 = torch.tensor([0, 0, 1, 1])

    nn = Seq(Lin(32, 16), ReLU(), Lin(16, 32))
    conv = DynamicEdgeConv(nn, k=2)
    assert str(conv) == (
        'DynamicEdgeConv(nn=Sequential(\n'
        '  (0): Linear(in_features=32, out_features=16, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=16, out_features=32, bias=True)\n'
        '), k=2)')
    out11 = conv(x1)
    assert out11.size() == (8, 32)

    out12 = conv(x1, batch1)
    assert out12.size() == (8, 32)

    out21 = conv((x1, x2))
    assert out21.size() == (4, 32)

    out22 = conv((x1, x2), (batch1, batch2))
    assert out22.size() == (4, 32)

    if is_full_test():
        t = '(Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1), out11)
        assert torch.allclose(jit(x1, batch1), out12)

        t = '(PairTensor, Optional[PairTensor]) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2)), out21)
        assert torch.allclose(jit((x1, x2), (batch1, batch2)), out22)

        torch.jit.script(conv.jittable())  # Test without explicit typing.
pytorch/test/nn/conv/test_eg_conv.py000066400000000000000000000046461444262304300201520ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import EGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_eg_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = EGConv(16, 32)
    assert str(conv) == "EGConv(16, 32, aggregators=['symnorm'])"
    out = conv(x, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-6)

    conv.cached = True
    assert torch.allclose(conv(x, edge_index), out, atol=1e-6)
    assert conv._cached_edge_index is not None
    assert torch.allclose(conv(x, edge_index), out, atol=1e-6)
    assert torch.allclose(conv(x, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-6)
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out, atol=1e-6)


def test_eg_conv_multiple_aggregators():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = EGConv(16, 32, aggregators=["max", "min"])
    assert str(conv) == "EGConv(16, 32, aggregators=['max', 'min'])"
    out = conv(x, edge_index)
    assert out.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj.t()), out, atol=1e-6)


def test_eg_conv_with_sparse_input_feature():
    x = torch.randn(4, 16).to_sparse_coo()
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = EGConv(16, 32)
    assert conv(x, edge_index).size() == (4, 32)
pytorch/test/nn/conv/test_fa_conv.py000066400000000000000000000066111444262304300201370ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import FAConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_fa_conv():
    x = torch.randn(4, 16)
    x_0 = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    # adj1 = SparseTensor(row=row, col=col, sparse_sizes=(4, 4))

    conv = FAConv(16, eps=1.0, cached=True)
    assert str(conv) == 'FAConv(16, eps=1.0)'
    out = conv(x, x_0, edge_index)
    assert conv._cached_edge_index is not None
    assert out.size() == (4, 16)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, x_0, adj2.t()), out)
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, x_0, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, x_0, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, Tensor, SparseTensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, x_0, adj2.t()), out)

    conv.reset_parameters()
    assert conv._cached_edge_index is None
    assert conv._cached_adj_t is None

    # Test without caching:
    conv.cached = False
    out = conv(x, x_0, edge_index)
    assert torch.allclose(conv(x, x_0, adj1.t()), out)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, x_0, adj2.t()), out)

    # Test `return_attention_weights`:
    result = conv(x, x_0, edge_index, return_attention_weights=True)
    assert torch.allclose(result[0], out)
    assert result[1][0].size() == (2, 10)
    assert result[1][1].size() == (10, )
    assert conv._alpha is None

    result = conv(x, x_0, adj1.t(), return_attention_weights=True)
    assert torch.allclose(result[0], out)
    assert result[1][0].size() == torch.Size([4, 4])
    assert result[1][0]._nnz() == 10
    assert conv._alpha is None

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        result = conv(x, x_0, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1].sizes() == [4, 4] and result[1].nnz() == 10
        assert conv._alpha is None

    if is_full_test():
        t = ('(Tensor, Tensor, Tensor, OptTensor, bool) '
             '-> Tuple[Tensor, Tuple[Tensor, Tensor]]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x, x_0, edge_index, return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1][0].size() == (2, 10)
        assert result[1][1].size() == (10, )
        assert conv._alpha is None

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = ('(Tensor, Tensor, SparseTensor, OptTensor, bool) '
             '-> Tuple[Tensor, SparseTensor]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x, x_0, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1].sizes() == [4, 4] and result[1].nnz() == 10
        assert conv._alpha is None
pytorch/test/nn/conv/test_feast_conv.py000066400000000000000000000041101444262304300206430ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import FeaStConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_feast_conv():
    x1 = torch.randn(4, 16)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = FeaStConv(16, 32, heads=2)
    assert str(conv) == 'FeaStConv(16, 32, heads=2)'

    out = conv(x1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_film_conv.py000066400000000000000000000071731444262304300205040ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import FiLMConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


def test_film_conv():
    x1 = torch.randn(4, 4)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [0, 0, 1, 0, 1, 1]])
    edge_type = torch.tensor([0, 1, 1, 0, 0, 1])

    conv = FiLMConv(4, 32)
    assert str(conv) == 'FiLMConv(4, 32, num_relations=1)'
    out = conv(x1, edge_index)
    assert out.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out, atol=1e-6)

    conv = FiLMConv(4, 32, num_relations=2)
    assert str(conv) == 'FiLMConv(4, 32, num_relations=2)'
    out = conv(x1, edge_index, edge_type)
    assert out.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_type, (4, 4))
        assert torch.allclose(conv(x1, adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, edge_type), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out, atol=1e-6)

    # Test bipartite message passing:
    conv = FiLMConv((4, 16), 32)
    assert str(conv) == 'FiLMConv((4, 16), 32, num_relations=1)'
    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out, atol=1e-6)

    conv = FiLMConv((4, 16), 32, num_relations=2)
    assert str(conv) == 'FiLMConv((4, 16), 32, num_relations=2)'
    out = conv((x1, x2), edge_index, edge_type)
    assert out.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_type, (4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, edge_type), out,
                              atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_fused_gat_conv.py000066400000000000000000000011301444262304300215010ustar00rootroot00000000000000import torch

from torch_geometric.nn import FusedGATConv
from torch_geometric.testing import onlyCUDA, withPackage


@onlyCUDA
@withPackage('dgNN')
def test_fused_gat_conv():
    device = torch.device('cuda')

    x = torch.randn(4, 8, device=device)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]], device=device)

    csr, csc, perm = FusedGATConv.to_graph_format(edge_index, size=(4, 4))

    conv = FusedGATConv(8, 32, heads=2, add_self_loops=False).to(device)
    assert str(conv) == 'FusedGATConv(8, 32, heads=2)'

    out = conv(x, csr, csc, perm)
    assert out.size() == (4, 64)
pytorch/test/nn/conv/test_gat_conv.py000066400000000000000000000145151444262304300203260ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GATConv
from torch_geometric.testing import is_full_test, withCUDA
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_gat_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = GATConv(8, 32, heads=2)
    assert str(conv) == 'GATConv(8, 32, heads=2)'
    out = conv(x1, edge_index)
    assert out.size() == (4, 64)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test `return_attention_weights`.
    result = conv(x1, edge_index, return_attention_weights=True)
    assert torch.allclose(result[0], out)
    assert result[1][0].size() == (2, 7)
    assert result[1][1].size() == (7, 2)
    assert result[1][1].min() >= 0 and result[1][1].max() <= 1

    result = conv(x1, adj1.t(), return_attention_weights=True)
    assert torch.allclose(result[0], out, atol=1e-6)
    assert result[1][0].size() == torch.Size([4, 4, 2])
    assert result[1][0]._nnz() == 7

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        result = conv(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 7

    if is_full_test():
        t = ('(Tensor, Tensor, OptTensor, Size, bool) -> '
             'Tuple[Tensor, Tuple[Tensor, Tensor]]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, edge_index, return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1][0].size() == (2, 7)
        assert result[1][1].size() == (7, 2)
        assert result[1][1].min() >= 0 and result[1][1].max() <= 1

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = ('(Tensor, SparseTensor, OptTensor, Size, bool) -> '
             'Tuple[Tensor, SparseTensor]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 7

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    conv = GATConv((8, 16), 32, heads=2)
    assert str(conv) == 'GATConv((8, 16), 32, heads=2)'

    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 64)
    assert torch.allclose(conv((x1, x2), edge_index, size=(4, 2)), out1)
    assert torch.allclose(conv((x1, x2), adj1.t()), out1, atol=1e-6)

    out2 = conv((x1, None), edge_index, size=(4, 2))
    assert out2.size() == (2, 64)
    assert torch.allclose(conv((x1, None), adj1.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj2.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1)
        assert torch.allclose(jit((x1, None), edge_index, size=(4, 2)), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = ('(OptPairTensor, SparseTensor, OptTensor, Size, NoneType) -> '
             'Tensor')
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj2.t()), out2, atol=1e-6)


def test_gat_conv_with_edge_attr():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 1, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 4)

    conv = GATConv(8, 32, heads=2, edge_dim=1, fill_value=0.5)
    out = conv(x, edge_index, edge_weight)
    assert out.size() == (4, 64)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj1 = SparseTensor.from_edge_index(edge_index, edge_weight, (4, 4))
        with pytest.raises(NotImplementedError):
            assert torch.allclose(conv(x, adj1.t()), out)

    conv = GATConv(8, 32, heads=2, edge_dim=1, fill_value='mean')
    out = conv(x, edge_index, edge_weight)
    assert out.size() == (4, 64)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        with pytest.raises(NotImplementedError):
            assert torch.allclose(conv(x, adj1.t()), out)

    conv = GATConv(8, 32, heads=2, edge_dim=4, fill_value=0.5)
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 64)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, edge_attr, (4, 4))
        with pytest.raises(NotImplementedError):
            assert torch.allclose(conv(x, adj2.t()), out)

    conv = GATConv(8, 32, heads=2, edge_dim=4, fill_value='mean')
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 64)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        with pytest.raises(NotImplementedError):
            assert torch.allclose(conv(x, adj2.t()), out)


@withCUDA
def test_gat_conv_empty_edge_index(device):
    x = torch.randn(0, 8, device=device)
    edge_index = torch.empty(2, 0, dtype=torch.long, device=device)

    conv = GATConv(8, 32, heads=2).to(device)
    out = conv(x, edge_index)
    assert out.size() == (0, 64)
pytorch/test/nn/conv/test_gated_graph_conv.py000066400000000000000000000034101444262304300220100ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import GatedGraphConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_gated_graph_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = GatedGraphConv(32, num_layers=3)
    assert str(conv) == 'GatedGraphConv(32, num_layers=3)'
    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)
    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)
pytorch/test/nn/conv/test_gatv2_conv.py000066400000000000000000000113471444262304300205760ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import GATv2Conv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_gatv2_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = GATv2Conv(8, 32, heads=2)
    assert str(conv) == 'GATv2Conv(8, 32, heads=2)'
    out = conv(x1, edge_index)
    assert out.size() == (4, 64)
    assert torch.allclose(conv(x1, edge_index), out)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test `return_attention_weights`.
    result = conv(x1, edge_index, return_attention_weights=True)
    assert torch.allclose(result[0], out)
    assert result[1][0].size() == (2, 7)
    assert result[1][1].size() == (7, 2)
    assert result[1][1].min() >= 0 and result[1][1].max() <= 1
    assert conv._alpha is None

    result = conv(x1, adj1.t(), return_attention_weights=True)
    assert torch.allclose(result[0], out, atol=1e-6)
    assert result[1][0].size() == torch.Size([4, 4, 2])
    assert result[1][0]._nnz() == 7

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        result = conv(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 7
        assert conv._alpha is None

    if is_full_test():
        t = ('(Tensor, Tensor, OptTensor, bool) -> '
             'Tuple[Tensor, Tuple[Tensor, Tensor]]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, edge_index, return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1][0].size() == (2, 7)
        assert result[1][1].size() == (7, 2)
        assert result[1][1].min() >= 0 and result[1][1].max() <= 1
        assert conv._alpha is None

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = ('(Tensor, SparseTensor, OptTensor, bool) -> '
             'Tuple[Tensor, SparseTensor]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 7
        assert conv._alpha is None

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 64)
    assert torch.allclose(conv((x1, x2), edge_index), out)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)


def test_gatv2_conv_with_edge_attr():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 1, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 4)

    conv = GATv2Conv(8, 32, heads=2, edge_dim=1, fill_value=0.5)
    out = conv(x, edge_index, edge_weight)
    assert out.size() == (4, 64)

    conv = GATv2Conv(8, 32, heads=2, edge_dim=1, fill_value='mean')
    out = conv(x, edge_index, edge_weight)
    assert out.size() == (4, 64)

    conv = GATv2Conv(8, 32, heads=2, edge_dim=4, fill_value=0.5)
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 64)

    conv = GATv2Conv(8, 32, heads=2, edge_dim=4, fill_value='mean')
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 64)
pytorch/test/nn/conv/test_gcn2_conv.py000066400000000000000000000044041444262304300204000ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import GCN2Conv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_gcn2_conv():
    x = torch.randn(4, 16)
    x_0 = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = GCN2Conv(16, alpha=0.2)
    assert str(conv) == 'GCN2Conv(16, alpha=0.2, beta=1.0)'
    out1 = conv(x, x_0, edge_index)
    assert out1.size() == (4, 16)
    assert torch.allclose(conv(x, x_0, adj1.t()), out1, atol=1e-6)
    out2 = conv(x, x_0, edge_index, value)
    assert out2.size() == (4, 16)
    assert torch.allclose(conv(x, x_0, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, x_0, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, x_0, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, x_0, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, x_0, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, x_0, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, x_0, adj4.t()), out2, atol=1e-6)

    conv.cached = True
    conv(x, x_0, edge_index)
    assert conv._cached_edge_index is not None
    assert torch.allclose(conv(x, x_0, edge_index), out1, atol=1e-6)
    assert torch.allclose(conv(x, x_0, adj1.t()), out1, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        conv(x, x_0, adj3.t())
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, x_0, adj3.t()), out1, atol=1e-6)
pytorch/test/nn/conv/test_gcn_conv.py000066400000000000000000000105441444262304300203200ustar00rootroot00000000000000import copy

import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GCNConv
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_coo_tensor, to_torch_csc_tensor


def test_gcn_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = GCNConv(16, 32)
    assert str(conv) == 'GCNConv(16, 32)'

    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)

    conv.cached = True
    conv(x, edge_index)
    assert conv._cached_edge_index is not None
    assert torch.allclose(conv(x, edge_index), out1, atol=1e-6)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        conv(x, adj3.t())
        assert conv._cached_adj_t is not None
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)


def test_gcn_conv_with_decomposed_layers():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = GCNConv(16, 32)

    decomposed_conv = copy.deepcopy(conv)
    decomposed_conv.decomposed_layers = 2

    out1 = conv(x, edge_index)
    out2 = decomposed_conv(x, edge_index)
    assert torch.allclose(out1, out2)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(decomposed_conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1)


def test_gcn_conv_with_sparse_input_feature():
    x = torch.sparse_coo_tensor(
        indices=torch.tensor([[0, 0], [0, 1]]),
        values=torch.tensor([1., 1.]),
        size=torch.Size([4, 16]),
    )
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = GCNConv(16, 32)
    assert conv(x, edge_index).size() == (4, 32)


def test_static_gcn_conv():
    x = torch.randn(3, 4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = GCNConv(16, 32)
    out = conv(x, edge_index)
    assert out.size() == (3, 4, 32)


def test_gcn_conv_norm():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0], [1, 2, 3]])
    row, col = edge_index

    conv = GCNConv(16, 32, flow="source_to_target")
    out1 = conv(x, edge_index)
    conv.flow = "target_to_source"
    out2 = conv(x, edge_index.flip(0))
    assert torch.allclose(out1, out2, atol=1e-6)


@pytest.mark.parametrize('requires_grad', [False, True])
@pytest.mark.parametrize('layout', [torch.sparse_coo, torch.sparse_csr])
def test_gcn_norm_gradient(requires_grad, layout):
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_weight = torch.ones(edge_index.size(1), requires_grad=requires_grad)
    adj = to_torch_coo_tensor(edge_index, edge_weight)
    if layout == torch.sparse_csr:
        adj = adj.to_sparse_csr()

    # TODO Sparse CSR tensor does not yet inherit `requires_grad` from `value`.
    if layout == torch.sparse_csr:
        assert not gcn_norm(adj)[0].requires_grad
    else:
        assert adj.requires_grad == gcn_norm(adj)[0].requires_grad
pytorch/test/nn/conv/test_gen_conv.py000066400000000000000000000150271444262304300203230ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GENConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_coo_tensor


@pytest.mark.parametrize('aggr', [
    'softmax',
    'powermean',
    ['softmax', 'powermean'],
])
def test_gen_conv(aggr):
    x1 = torch.randn(4, 16)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1), 16)
    adj1 = to_torch_coo_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_coo_tensor(edge_index, value, size=(4, 4))

    conv = GENConv(16, 32, aggr, edge_dim=16, msg_norm=True)
    assert str(conv) == f'GENConv(16, 32, aggr={aggr})'
    out1 = conv(x1, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out1)
    assert torch.allclose(conv(x1, adj1.t().coalesce()), out1)

    out2 = conv(x1, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, (4, 4)), out2)
    # t() expects a tensor with <= 2 sparse and 0 dense dimensions
    assert torch.allclose(conv(x1, adj2.transpose(1, 0).coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj3.t()), out1)
        assert torch.allclose(conv(x1, adj4.t()), out2)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out1,
                              atol=1e-6)
        assert torch.allclose(jit(x1, edge_index, value), out2, atol=1e-6)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out2,
                              atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj3.t()), out1)
        assert torch.allclose(jit(x1, adj4.t()), out2)

    # Test bipartite message passing:
    adj1 = to_torch_coo_tensor(edge_index, size=(4, 2))
    adj2 = to_torch_coo_tensor(edge_index, value, size=(4, 2))

    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, size=(4, 2)), out1)
    assert torch.allclose(conv((x1, x2), adj1.t().coalesce()), out1)

    out2 = conv((x1, x2), edge_index, value)
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out2)
    assert torch.allclose(conv((x1, x2),
                               adj2.transpose(1, 0).coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj3.t()), out1)
        assert torch.allclose(conv((x1, x2), adj4.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1, atol=1e-6)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1,
                              atol=1e-6)
        assert torch.allclose(jit((x1, x2), edge_index, value), out2,
                              atol=1e-6)
        assert torch.allclose(jit((x1, x2), edge_index, value, (4, 2)), out2,
                              atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj3.t()), out1)
        assert torch.allclose(jit((x1, x2), adj4.t()), out2)

    # Test bipartite message passing with unequal feature dimensions:
    conv.reset_parameters()
    assert float(conv.msg_norm.scale) == 1

    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)

    conv = GENConv((8, 16), 32, aggr)
    assert str(conv) == f'GENConv((8, 16), 32, aggr={aggr})'

    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, size=(4, 2)), out1)
    assert torch.allclose(conv((x1, x2), adj1.t().coalesce()), out1)

    out2 = conv((x1, None), edge_index, size=(4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None), adj1.t().coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv((x1, x2), adj3.t()), out1)
        assert torch.allclose(conv((x1, None), adj3.t()), out2)

    # Test lazy initialization:
    conv = GENConv((-1, -1), 32, aggr, edge_dim=-1)
    assert str(conv) == f'GENConv((-1, -1), 32, aggr={aggr})'
    out1 = conv((x1, x2), edge_index, value)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, size=(4, 2)), out1)
    assert torch.allclose(conv((x1, x2),
                               adj2.transpose(1, 0).coalesce()), out1)

    out2 = conv((x1, None), edge_index, value, size=(4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None),
                               adj2.transpose(1, 0).coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv((x1, x2), adj4.t()), out1)
        assert torch.allclose(conv((x1, None), adj4.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out1,
                              atol=1e-6)
        assert torch.allclose(jit((x1, x2), edge_index, value, size=(4, 2)),
                              out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), edge_index, value, size=(4, 2)),
                              out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj4.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj4.t()), out2, atol=1e-6)
pytorch/test/nn/conv/test_general_conv.py000066400000000000000000000043401444262304300211630ustar00rootroot00000000000000import torch

from torch_geometric.nn import GeneralConv


def test_general_conv():
    x1 = torch.randn(4, 8)
    e1 = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = GeneralConv(8, 32, 16)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, skip_linear=True)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, directed_msg=False)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, heads=3)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, attention=True)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, heads=3, attention=True)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, heads=3, attention=True,
                       attention_type='dot_product')
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)

    conv = GeneralConv(8, 32, 16, l2_normalize=True)
    assert str(conv) == 'GeneralConv(8, 32)'
    out = conv(x1, edge_index, edge_attr=e1)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, edge_attr=e1), out, atol=1e-7)
pytorch/test/nn/conv/test_gin_conv.py000066400000000000000000000153641444262304300203330ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.typing
from torch_geometric.nn import GINConv, GINEConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_gin_conv():
    x1 = torch.randn(4, 16)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    nn = Seq(Lin(16, 32), ReLU(), Lin(32, 32))
    conv = GINConv(nn, train_eps=True)
    assert str(conv) == (
        'GINConv(nn=Sequential(\n'
        '  (0): Linear(in_features=16, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=32, bias=True)\n'
        '))')
    out = conv(x1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out, atol=1e-6)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, (4, 2)), out1, atol=1e-6)
    assert torch.allclose(conv((x1, x2), adj1.t()), out1, atol=1e-6)

    out2 = conv((x1, None), edge_index, (4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None), adj1.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj2.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(OptPairTensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1)
        assert torch.allclose(jit((x1, None), edge_index, size=(4, 2)), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out1)
        assert torch.allclose(jit((x1, None), adj2.t()), out2)


def test_gine_conv():
    x1 = torch.randn(4, 16)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1), 16)

    nn = Seq(Lin(16, 32), ReLU(), Lin(32, 32))
    conv = GINEConv(nn, train_eps=True)
    assert str(conv) == (
        'GINEConv(nn=Sequential(\n'
        '  (0): Linear(in_features=16, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=32, bias=True)\n'
        '))')
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, size=(4, 4)), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, value), out)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out)

    # Test bipartite message passing:
    out1 = conv((x1, x2), edge_index, value)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)

    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out2.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out1)
        assert torch.allclose(conv((x1, None), adj.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out1)
        assert torch.allclose(jit((x1, x2), edge_index, value, size=(4, 2)),
                              out1)
        assert torch.allclose(jit((x1, None), edge_index, value, size=(4, 2)),
                              out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1)
        assert torch.allclose(jit((x1, None), adj.t()), out2)


def test_gine_conv_edge_dim():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_attr = torch.randn(edge_index.size(1), 8)

    nn = Seq(Lin(16, 32), ReLU(), Lin(32, 32))
    conv = GINEConv(nn, train_eps=True, edge_dim=8)
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 32)

    nn = Lin(16, 32)
    conv = GINEConv(nn, train_eps=True, edge_dim=8)
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 32)


def test_static_gin_conv():
    x = torch.randn(3, 4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    nn = Seq(Lin(16, 32), ReLU(), Lin(32, 32))
    conv = GINConv(nn, train_eps=True)
    out = conv(x, edge_index)
    assert out.size() == (3, 4, 32)


def test_static_gine_conv():
    x = torch.randn(3, 4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.randn(edge_index.size(1), 16)

    nn = Seq(Lin(16, 32), ReLU(), Lin(32, 32))
    conv = GINEConv(nn, train_eps=True)
    out = conv(x, edge_index, edge_attr)
    assert out.size() == (3, 4, 32)
pytorch/test/nn/conv/test_gmm_conv.py000066400000000000000000000101231444262304300203220ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GMMConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_coo_tensor


@pytest.mark.parametrize('separate_gaussians', [True, False])
def test_gmm_conv(separate_gaussians):
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1), 3)
    adj1 = to_torch_coo_tensor(edge_index, value, size=(4, 4))

    conv = GMMConv(8, 32, dim=3, kernel_size=25,
                   separate_gaussians=separate_gaussians)
    assert str(conv) == 'GMMConv(8, 32, dim=3)'
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, size=(4, 4)), out)
    # t() expects a tensor with <= 2 sparse and 0 dense dimensions
    assert torch.allclose(conv(x1, adj1.transpose(0, 1).coalesce()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, value), out)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out)

    # Test bipartite message passing:
    adj1 = to_torch_coo_tensor(edge_index, value, size=(4, 2))

    conv = GMMConv((8, 16), 32, dim=3, kernel_size=5,
                   separate_gaussians=separate_gaussians)
    assert str(conv) == 'GMMConv((8, 16), 32, dim=3)'

    out1 = conv((x1, x2), edge_index, value)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)
    assert torch.allclose(conv((x1, x2),
                               adj1.transpose(0, 1).coalesce()), out1)

    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None),
                               adj1.transpose(0, 1).coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out1)
        assert torch.allclose(conv((x1, None), adj2.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out1)
        assert torch.allclose(jit((x1, x2), edge_index, value, size=(4, 2)),
                              out1)
        assert torch.allclose(jit((x1, None), edge_index, value, size=(4, 2)),
                              out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out1)
        assert torch.allclose(jit((x1, None), adj2.t()), out2)


@pytest.mark.parametrize('separate_gaussians', [True, False])
def test_lazy_gmm_conv(separate_gaussians):
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1), 3)

    conv = GMMConv(-1, 32, dim=3, kernel_size=25,
                   separate_gaussians=separate_gaussians)
    assert str(conv) == 'GMMConv(-1, 32, dim=3)'
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)

    conv = GMMConv((-1, -1), 32, dim=3, kernel_size=25,
                   separate_gaussians=separate_gaussians)
    assert str(conv) == 'GMMConv((-1, -1), 32, dim=3)'
    out = conv((x1, x2), edge_index, value)
    assert out.size() == (2, 32)
pytorch/test/nn/conv/test_gps_conv.py000066400000000000000000000025651444262304300203460ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GPSConv, SAGEConv
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor

x = torch.randn(4, 16)
edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 3, 2]])
batch = torch.tensor([0, 0, 1, 1])
adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))


@pytest.mark.parametrize('attn_type', ['multihead', 'performer'])
@pytest.mark.parametrize('norm', [None, 'batch_norm', 'layer_norm'])
def test_gps_conv(norm, attn_type):
    conv = GPSConv(16, conv=SAGEConv(16, 16), heads=4, norm=norm,
                   attn_type=attn_type)
    conv.reset_parameters()
    assert str(conv) == (f'GPSConv(16, conv=SAGEConv(16, 16, aggr=mean), '
                         f'heads=4, attn_type={attn_type})')

    out = conv(x, edge_index)
    assert out.size() == (4, 16)
    assert torch.allclose(conv(x, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out, atol=1e-6)

    out = conv(x, edge_index, batch)
    assert out.size() == (4, 16)
    assert torch.allclose(conv(x, adj1.t(), batch), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj2.t(), batch), out, atol=1e-6)
pytorch/test/nn/conv/test_graph_conv.py000066400000000000000000000111641444262304300206510ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import GraphConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_graph_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = GraphConv(8, 32)
    assert str(conv) == 'GraphConv(8, 32)'
    out1 = conv(x1, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out1, atol=1e-6)
    assert torch.allclose(conv(x1, adj1.t()), out1, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj3.t()), out1, atol=1e-6)

    out2 = conv(x1, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, size=(4, 4)), out2,
                          atol=1e-6)
    assert torch.allclose(conv(x1, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out1)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out1)
        assert torch.allclose(jit(x1, edge_index, value), out2)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x1, adj4.t()), out2, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 2))

    conv = GraphConv((8, 16), 32)
    assert str(conv) == 'GraphConv((8, 16), 32)'
    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, size=(4, 2)), out1)
    assert torch.allclose(conv((x1, x2), adj1.t()), out1, atol=1e-6)

    out2 = conv((x1, None), edge_index, size=(4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None), adj1.t()), out2, atol=1e-6)

    out3 = conv((x1, x2), edge_index, value)
    assert out3.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out3)
    assert torch.allclose(conv((x1, x2), adj2.t()), out3, atol=1e-6)

    out4 = conv((x1, None), edge_index, value, size=(4, 2))
    assert out4.size() == (2, 32)
    assert torch.allclose(conv((x1, None), adj2.t()), out4, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj3.t()), out2, atol=1e-6)
        assert torch.allclose(conv((x1, x2), adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj4.t()), out4, atol=1e-6)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1)
        assert torch.allclose(jit((x1, None), edge_index, size=(4, 2)), out2)
        assert torch.allclose(jit((x1, x2), edge_index, value), out3)
        assert torch.allclose(jit((x1, x2), edge_index, value, (4, 2)), out3)
        assert torch.allclose(jit((x1, None), edge_index, value, (4, 2)), out4)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj3.t()), out2, atol=1e-6)
        assert torch.allclose(jit((x1, x2), adj4.t()), out3, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj4.t()), out4, atol=1e-6)
pytorch/test/nn/conv/test_gravnet_conv.py000066400000000000000000000023361444262304300212170ustar00rootroot00000000000000import torch

from torch_geometric.nn import GravNetConv
from torch_geometric.testing import is_full_test, withPackage


@withPackage('torch_cluster')
def test_gravnet_conv():
    x1 = torch.randn(8, 16)
    x2 = torch.randn(4, 16)
    batch1 = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])
    batch2 = torch.tensor([0, 0, 1, 1])

    conv = GravNetConv(16, 32, space_dimensions=4, propagate_dimensions=8, k=2)
    assert str(conv) == 'GravNetConv(16, 32, k=2)'

    out11 = conv(x1)
    assert out11.size() == (8, 32)

    out12 = conv(x1, batch1)
    assert out12.size() == (8, 32)

    out21 = conv((x1, x2))
    assert out21.size() == (4, 32)

    out22 = conv((x1, x2), (batch1, batch2))
    assert out22.size() == (4, 32)

    if is_full_test():
        t = '(Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1), out11)
        assert torch.allclose(jit(x1, batch1), out12)

        t = '(PairTensor, Optional[PairTensor]) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2)), out21)
        assert torch.allclose(jit((x1, x2), (batch1, batch2)), out22)

        torch.jit.script(conv.jittable())  # Test without explicit typing.
pytorch/test/nn/conv/test_han_conv.py000066400000000000000000000120031444262304300203070ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import HANConv
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import coalesce, to_torch_csc_tensor


def test_han_conv():
    x_dict = {
        'author': torch.randn(6, 16),
        'paper': torch.randn(5, 12),
        'term': torch.randn(4, 3)
    }
    edge_index1 = coalesce(torch.randint(0, 6, (2, 7)))
    edge_index2 = coalesce(torch.randint(0, 5, (2, 4)))
    edge_index3 = coalesce(torch.randint(0, 3, (2, 5)))
    edge_index_dict = {
        ('author', 'metapath0', 'author'): edge_index1,
        ('paper', 'metapath1', 'paper'): edge_index2,
        ('paper', 'metapath2', 'paper'): edge_index3,
    }

    adj_t_dict1 = {}
    for edge_type, edge_index in edge_index_dict.items():
        src_type, _, dst_type = edge_type
        adj_t_dict1[edge_type] = to_torch_csc_tensor(
            edge_index,
            size=(x_dict[src_type].size(0), x_dict[dst_type].size(0)),
        ).t()

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))
    in_channels = {'author': 16, 'paper': 12, 'term': 3}

    conv = HANConv(in_channels, 16, metadata, heads=2)
    assert str(conv) == 'HANConv(16, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 3
    assert out_dict1['author'].size() == (6, 16)
    assert out_dict1['paper'].size() == (5, 16)
    assert out_dict1['term'] is None
    del out_dict1['term']
    del x_dict['term']

    out_dict2 = conv(x_dict, adj_t_dict1)
    assert len(out_dict1) == len(out_dict2)
    for key in out_dict1.keys():
        assert torch.allclose(out_dict1[key], out_dict2[key], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict2 = {}
        for edge_type, edge_index in edge_index_dict.items():
            adj_t_dict2[edge_type] = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=adj_t_dict1[edge_type].size()[::-1],
            ).t()
        out_dict3 = conv(x_dict, adj_t_dict2)
        assert len(out_dict1) == len(out_dict3)
        for key in out_dict3.keys():
            assert torch.allclose(out_dict1[key], out_dict3[key], atol=1e-6)

    # Test non-zero dropout:
    conv = HANConv(in_channels, 16, metadata, heads=2, dropout=0.1)
    assert str(conv) == 'HANConv(16, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 2
    assert out_dict1['author'].size() == (6, 16)
    assert out_dict1['paper'].size() == (5, 16)


def test_han_conv_lazy():
    x_dict = {
        'author': torch.randn(6, 16),
        'paper': torch.randn(5, 12),
    }
    edge_index1 = coalesce(torch.randint(0, 6, (2, 8)))
    edge_index2 = coalesce(torch.randint(0, 5, (2, 6)))
    edge_index_dict = {
        ('author', 'to', 'author'): edge_index1,
        ('paper', 'to', 'paper'): edge_index2,
    }

    adj_t_dict1 = {}
    for edge_type, edge_index in edge_index_dict.items():
        src_type, _, dst_type = edge_type
        adj_t_dict1[edge_type] = to_torch_csc_tensor(
            edge_index,
            size=(x_dict[src_type].size(0), x_dict[dst_type].size(0)),
        ).t()

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))
    conv = HANConv(-1, 16, metadata, heads=2)
    assert str(conv) == 'HANConv(16, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 2
    assert out_dict1['author'].size() == (6, 16)
    assert out_dict1['paper'].size() == (5, 16)

    out_dict2 = conv(x_dict, adj_t_dict1)
    assert len(out_dict1) == len(out_dict2)
    for key in out_dict1.keys():
        assert torch.allclose(out_dict1[key], out_dict2[key], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict2 = {}
        for edge_type, edge_index in edge_index_dict.items():
            adj_t_dict2[edge_type] = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=adj_t_dict1[edge_type].size()[::-1],
            ).t()
        out_dict3 = conv(x_dict, adj_t_dict2)
        assert len(out_dict1) == len(out_dict3)
        for key in out_dict1.keys():
            assert torch.allclose(out_dict1[key], out_dict3[key], atol=1e-6)


def test_han_conv_empty_tensor():
    x_dict = {
        'author': torch.randn(6, 16),
        'paper': torch.empty(0, 12),
    }
    edge_index_dict = {
        ('paper', 'to', 'author'): torch.empty((2, 0), dtype=torch.long),
        ('author', 'to', 'paper'): torch.empty((2, 0), dtype=torch.long),
        ('paper', 'to', 'paper'): torch.empty((2, 0), dtype=torch.long),
    }

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))
    in_channels = {'author': 16, 'paper': 12}
    conv = HANConv(in_channels, 16, metadata, heads=2)

    out_dict = conv(x_dict, edge_index_dict)
    assert len(out_dict) == 2
    assert out_dict['author'].size() == (6, 16)
    assert torch.all(out_dict['author'] == 0)
    assert out_dict['paper'].size() == (0, 16)
pytorch/test/nn/conv/test_heat_conv.py000066400000000000000000000030331444262304300204650ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import HEATConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


@pytest.mark.parametrize('concat', [True, False])
def test_heat_conv(concat):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_attr = torch.randn((4, 2))
    node_type = torch.tensor([0, 0, 1, 2])
    edge_type = torch.tensor([0, 2, 1, 2])

    conv = HEATConv(in_channels=8, out_channels=16, num_node_types=3,
                    num_edge_types=3, edge_type_emb_dim=5, edge_dim=2,
                    edge_attr_emb_dim=6, heads=2, concat=concat)
    assert str(conv) == 'HEATConv(8, 16, heads=2)'

    out = conv(x, edge_index, node_type, edge_type, edge_attr)
    assert out.size() == (4, 32 if concat else 16)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_attr, (4, 4))
        assert torch.allclose(conv(x, adj.t(), node_type, edge_type), out)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(
            jit(x, edge_index, node_type, edge_type, edge_attr), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t(), node_type, edge_type), out)
pytorch/test/nn/conv/test_hetero_conv.py000066400000000000000000000107141444262304300210360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import HeteroData
from torch_geometric.nn import (
    GATConv,
    GCNConv,
    HeteroConv,
    Linear,
    MessagePassing,
    SAGEConv,
)
from torch_geometric.testing import get_random_edge_index


@pytest.mark.parametrize('aggr', ['sum', 'mean', 'min', 'max', 'cat', None])
def test_hetero_conv(aggr):
    data = HeteroData()
    data['paper'].x = torch.randn(50, 32)
    data['author'].x = torch.randn(30, 64)
    data['paper', 'paper'].edge_index = get_random_edge_index(50, 50, 200)
    data['paper', 'author'].edge_index = get_random_edge_index(50, 30, 100)
    data['paper', 'author'].edge_attr = torch.randn(100, 3)
    data['author', 'paper'].edge_index = get_random_edge_index(30, 50, 100)
    data['paper', 'paper'].edge_weight = torch.rand(200)

    # Unspecified edge types should be ignored:
    data['author', 'author'].edge_index = get_random_edge_index(30, 30, 100)

    conv = HeteroConv(
        {
            ('paper', 'to', 'paper'):
            GCNConv(-1, 64),
            ('author', 'to', 'paper'):
            SAGEConv((-1, -1), 64),
            ('paper', 'to', 'author'):
            GATConv((-1, -1), 64, edge_dim=3, add_self_loops=False),
        }, aggr=aggr)

    assert len(list(conv.parameters())) > 0
    assert str(conv) == 'HeteroConv(num_relations=3)'

    out = conv(data.x_dict, data.edge_index_dict, data.edge_attr_dict,
               edge_weight_dict=data.edge_weight_dict)

    assert len(out) == 2
    if aggr == 'cat':
        assert out['paper'].size() == (50, 128)
        assert out['author'].size() == (30, 64)
    elif aggr is not None:
        assert out['paper'].size() == (50, 64)
        assert out['author'].size() == (30, 64)
    else:
        assert out['paper'].size() == (50, 2, 64)
        assert out['author'].size() == (30, 1, 64)


class CustomConv(MessagePassing):
    def __init__(self, out_channels):
        super().__init__(aggr='add')
        self.lin = Linear(-1, out_channels)

    def forward(self, x, edge_index, y, z):
        return self.propagate(edge_index, x=x, y=y, z=z)

    def message(self, x_j, y_j, z_j):
        return self.lin(torch.cat([x_j, y_j, z_j], dim=-1))


def test_hetero_conv_with_custom_conv():
    data = HeteroData()
    data['paper'].x = torch.randn(50, 32)
    data['paper'].y = torch.randn(50, 3)
    data['paper'].z = torch.randn(50, 3)
    data['author'].x = torch.randn(30, 64)
    data['author'].y = torch.randn(30, 3)
    data['author'].z = torch.randn(30, 3)
    data['paper', 'paper'].edge_index = get_random_edge_index(50, 50, 200)
    data['paper', 'author'].edge_index = get_random_edge_index(50, 30, 100)
    data['author', 'paper'].edge_index = get_random_edge_index(30, 50, 100)

    conv = HeteroConv({key: CustomConv(64) for key in data.edge_types})
    # Test node `args_dict` and `kwargs_dict` with `y_dict` and `z_dict`:
    out = conv(data.x_dict, data.edge_index_dict, data.y_dict,
               z_dict=data.z_dict)
    assert len(out) == 2
    assert out['paper'].size() == (50, 64)
    assert out['author'].size() == (30, 64)


class MessagePassingLoops(MessagePassing):
    def __init__(self):
        super().__init__()
        self.add_self_loops = True


def test_hetero_conv_self_loop_error():
    HeteroConv({('a', 'to', 'a'): MessagePassingLoops()})
    with pytest.raises(ValueError, match="incorrect message passing"):
        HeteroConv({('a', 'to', 'b'): MessagePassingLoops()})


def test_hetero_conv_with_dot_syntax_node_types():
    data = HeteroData()
    data['src.paper'].x = torch.randn(50, 32)
    data['author'].x = torch.randn(30, 64)
    edge_index = get_random_edge_index(50, 50, 200)
    data['src.paper', 'src.paper'].edge_index = edge_index
    data['src.paper', 'author'].edge_index = get_random_edge_index(50, 30, 100)
    data['author', 'src.paper'].edge_index = get_random_edge_index(30, 50, 100)
    data['src.paper', 'src.paper'].edge_weight = torch.rand(200)

    conv = HeteroConv({
        ('src.paper', 'to', 'src.paper'):
        GCNConv(-1, 64),
        ('author', 'to', 'src.paper'):
        SAGEConv((-1, -1), 64),
        ('src.paper', 'to', 'author'):
        GATConv((-1, -1), 64, add_self_loops=False),
    })

    assert len(list(conv.parameters())) > 0
    assert str(conv) == 'HeteroConv(num_relations=3)'

    out = conv(data.x_dict, data.edge_index_dict,
               edge_weight_dict=data.edge_weight_dict)

    assert len(out) == 2
    assert out['src.paper'].size() == (50, 64)
    assert out['author'].size() == (30, 64)
pytorch/test/nn/conv/test_hgt_conv.py000066400000000000000000000223241444262304300203320ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.data import HeteroData
from torch_geometric.nn import HGTConv
from torch_geometric.profile import benchmark
from torch_geometric.testing import get_random_edge_index
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import coalesce, to_torch_csc_tensor


def test_hgt_conv_same_dimensions():
    x_dict = {
        'author': torch.randn(4, 16),
        'paper': torch.randn(6, 16),
    }
    edge_index = coalesce(get_random_edge_index(4, 6, num_edges=20))

    edge_index_dict = {
        ('author', 'writes', 'paper'): edge_index,
        ('paper', 'written_by', 'author'): edge_index.flip([0]),
    }

    adj_t_dict1 = {}
    for edge_type, edge_index in edge_index_dict.items():
        src_type, _, dst_type = edge_type
        adj_t_dict1[edge_type] = to_torch_csc_tensor(
            edge_index,
            size=(x_dict[src_type].size(0), x_dict[dst_type].size(0)),
        ).t()

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))

    conv = HGTConv(16, 16, metadata, heads=2)
    assert str(conv) == 'HGTConv(-1, 16, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 2
    assert out_dict1['author'].size() == (4, 16)
    assert out_dict1['paper'].size() == (6, 16)

    out_dict2 = conv(x_dict, adj_t_dict1)
    assert len(out_dict1) == len(out_dict2)
    for key in out_dict1.keys():
        assert torch.allclose(out_dict1[key], out_dict2[key], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict2 = {}
        for edge_type, edge_index in edge_index_dict.items():
            adj_t_dict2[edge_type] = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=adj_t_dict1[edge_type].size()[::-1],
            ).t()
        out_dict3 = conv(x_dict, adj_t_dict2)
        assert len(out_dict1) == len(out_dict3)
        for key in out_dict1.keys():
            assert torch.allclose(out_dict1[key], out_dict3[key], atol=1e-6)

    # TODO: Test JIT functionality. We need to wait on this one until PyTorch
    # allows indexing `ParameterDict` mappings :(


def test_hgt_conv_different_dimensions():
    x_dict = {
        'author': torch.randn(4, 16),
        'paper': torch.randn(6, 32),
    }
    edge_index = coalesce(get_random_edge_index(4, 6, num_edges=20))

    edge_index_dict = {
        ('author', 'writes', 'paper'): edge_index,
        ('paper', 'written_by', 'author'): edge_index.flip([0]),
    }

    adj_t_dict1 = {}
    for edge_type, edge_index in edge_index_dict.items():
        src_type, _, dst_type = edge_type
        adj_t_dict1[edge_type] = to_torch_csc_tensor(
            edge_index,
            size=(x_dict[src_type].size(0), x_dict[dst_type].size(0)),
        ).t()

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))

    conv = HGTConv(in_channels={
        'author': 16,
        'paper': 32
    }, out_channels=32, metadata=metadata, heads=2)
    assert str(conv) == 'HGTConv(-1, 32, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 2
    assert out_dict1['author'].size() == (4, 32)
    assert out_dict1['paper'].size() == (6, 32)

    out_dict2 = conv(x_dict, adj_t_dict1)
    assert len(out_dict1) == len(out_dict2)
    for key in out_dict1.keys():
        assert torch.allclose(out_dict1[key], out_dict2[key], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict2 = {}
        for edge_type, edge_index in edge_index_dict.items():
            adj_t_dict2[edge_type] = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=adj_t_dict1[edge_type].size()[::-1],
            ).t()
        out_dict3 = conv(x_dict, adj_t_dict2)
        assert len(out_dict1) == len(out_dict3)
        for node_type in out_dict1.keys():
            assert torch.allclose(out_dict1[key], out_dict3[key], atol=1e-6)


def test_hgt_conv_lazy():
    x_dict = {
        'author': torch.randn(4, 16),
        'paper': torch.randn(6, 32),
    }
    edge_index = coalesce(get_random_edge_index(4, 6, num_edges=20))

    edge_index_dict = {
        ('author', 'writes', 'paper'): edge_index,
        ('paper', 'written_by', 'author'): edge_index.flip([0]),
    }

    adj_t_dict1 = {}
    for edge_type, edge_index in edge_index_dict.items():
        src_type, _, dst_type = edge_type
        adj_t_dict1[edge_type] = to_torch_csc_tensor(
            edge_index,
            size=(x_dict[src_type].size(0), x_dict[dst_type].size(0)),
        ).t()

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))

    conv = HGTConv(-1, 32, metadata, heads=2)
    assert str(conv) == 'HGTConv(-1, 32, heads=2)'
    out_dict1 = conv(x_dict, edge_index_dict)
    assert len(out_dict1) == 2
    assert out_dict1['author'].size() == (4, 32)
    assert out_dict1['paper'].size() == (6, 32)

    out_dict2 = conv(x_dict, adj_t_dict1)
    assert len(out_dict1) == len(out_dict2)
    for key in out_dict1.keys():
        assert torch.allclose(out_dict1[key], out_dict2[key], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict2 = {}
        for edge_type, edge_index in edge_index_dict.items():
            adj_t_dict2[edge_type] = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=adj_t_dict1[edge_type].size()[::-1],
            ).t()
        out_dict3 = conv(x_dict, adj_t_dict2)
        assert len(out_dict1) == len(out_dict3)
        for key in out_dict1.keys():
            assert torch.allclose(out_dict1[key], out_dict3[key], atol=1e-6)


def test_hgt_conv_out_of_place():
    data = HeteroData()
    data['author'].x = torch.randn(4, 16)
    data['paper'].x = torch.randn(6, 32)

    edge_index = coalesce(get_random_edge_index(4, 6, num_edges=20))

    data['author', 'paper'].edge_index = edge_index
    data['paper', 'author'].edge_index = edge_index.flip([0])

    conv = HGTConv(-1, 64, data.metadata(), heads=1)

    x_dict, edge_index_dict = data.x_dict, data.edge_index_dict
    assert x_dict['author'].size() == (4, 16)
    assert x_dict['paper'].size() == (6, 32)

    _ = conv(x_dict, edge_index_dict)

    assert x_dict['author'].size() == (4, 16)
    assert x_dict['paper'].size() == (6, 32)


def test_hgt_conv_missing_dst_node_type():
    data = HeteroData()
    data['author'].x = torch.randn(4, 16)
    data['paper'].x = torch.randn(6, 32)
    data['university'].x = torch.randn(10, 32)

    data['author', 'paper'].edge_index = get_random_edge_index(4, 6, 20)
    data['paper', 'author'].edge_index = get_random_edge_index(6, 4, 20)
    data['university', 'author'].edge_index = get_random_edge_index(10, 4, 10)

    conv = HGTConv(-1, 64, data.metadata(), heads=1)

    out_dict = conv(data.x_dict, data.edge_index_dict)
    assert out_dict['author'].size() == (4, 64)
    assert out_dict['paper'].size() == (6, 64)
    assert 'university' not in out_dict


def test_hgt_conv_missing_input_node_type():
    data = HeteroData()
    data['author'].x = torch.randn(4, 16)
    data['paper'].x = torch.randn(6, 32)
    data['author', 'writes',
         'paper'].edge_index = get_random_edge_index(4, 6, 20)

    # Some nodes from metadata are missing in data.
    # This might happen while using NeighborLoader.
    metadata = (['author', 'paper',
                 'university'], [('author', 'writes', 'paper')])
    conv = HGTConv(-1, 64, metadata, heads=1)

    out_dict = conv(data.x_dict, data.edge_index_dict)
    assert out_dict['paper'].size() == (6, 64)
    assert 'university' not in out_dict


def test_hgt_conv_missing_edge_type():
    data = HeteroData()
    data['author'].x = torch.randn(4, 16)
    data['paper'].x = torch.randn(6, 32)
    data['university'].x = torch.randn(10, 32)

    data['author', 'writes',
         'paper'].edge_index = get_random_edge_index(4, 6, 20)

    metadata = (['author', 'paper',
                 'university'], [('author', 'writes', 'paper'),
                                 ('university', 'employs', 'author')])
    conv = HGTConv(-1, 64, metadata, heads=1)

    out_dict = conv(data.x_dict, data.edge_index_dict)
    assert out_dict['author'].size() == (4, 64)
    assert out_dict['paper'].size() == (6, 64)
    assert 'university' not in out_dict


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    args = parser.parse_args()

    num_nodes, num_edges = 30_000, 300_000
    x_dict = {
        'paper': torch.randn(num_nodes, 64, device=args.device),
        'author': torch.randn(num_nodes, 64, device=args.device),
    }
    edge_index_dict = {
        ('paper', 'to', 'paper'):
        torch.randint(num_nodes, (2, num_edges), device=args.device),
        ('author', 'to', 'paper'):
        torch.randint(num_nodes, (2, num_edges), device=args.device),
        ('paper', 'to', 'author'):
        torch.randint(num_nodes, (2, num_edges), device=args.device),
    }

    conv = HGTConv(
        in_channels=64,
        out_channels=64,
        metadata=(list(x_dict.keys()), list(edge_index_dict.keys())),
        heads=4,
    ).to(args.device)

    benchmark(
        funcs=[conv],
        args=(x_dict, edge_index_dict),
        num_steps=10 if args.device == 'cpu' else 100,
        num_warmups=5 if args.device == 'cpu' else 50,
        backward=False,
    )
pytorch/test/nn/conv/test_hypergraph_conv.py000066400000000000000000000040521444262304300217170ustar00rootroot00000000000000import torch

from torch_geometric.nn import HypergraphConv


def test_hypergraph_conv_with_more_nodes_than_edges():
    in_channels, out_channels = (16, 32)
    hyperedge_index = torch.tensor([[0, 0, 1, 1, 2, 3], [0, 1, 0, 1, 0, 1]])
    hyperedge_weight = torch.tensor([1.0, 0.5])
    num_nodes = hyperedge_index[0].max().item() + 1
    num_edges = hyperedge_index[1].max().item() + 1
    x = torch.randn((num_nodes, in_channels))
    hyperedge_attr = torch.randn((num_edges, in_channels))

    conv = HypergraphConv(in_channels, out_channels)
    assert str(conv) == 'HypergraphConv(16, 32)'
    out = conv(x, hyperedge_index)
    assert out.size() == (num_nodes, out_channels)
    out = conv(x, hyperedge_index, hyperedge_weight)
    assert out.size() == (num_nodes, out_channels)

    conv = HypergraphConv(in_channels, out_channels, use_attention=True,
                          heads=2)
    out = conv(x, hyperedge_index, hyperedge_attr=hyperedge_attr)
    assert out.size() == (num_nodes, 2 * out_channels)
    out = conv(x, hyperedge_index, hyperedge_weight, hyperedge_attr)
    assert out.size() == (num_nodes, 2 * out_channels)

    conv = HypergraphConv(in_channels, out_channels, use_attention=True,
                          heads=2, concat=False, dropout=0.5)
    out = conv(x, hyperedge_index, hyperedge_weight, hyperedge_attr)
    assert out.size() == (num_nodes, out_channels)


def test_hypergraph_conv_with_more_edges_than_nodes():
    in_channels, out_channels = (16, 32)
    hyperedge_index = torch.tensor([[0, 0, 1, 1, 2, 3, 3, 3, 2, 1, 2],
                                    [0, 1, 2, 1, 2, 1, 0, 3, 3, 4, 4]])
    hyperedge_weight = torch.tensor([1.0, 0.5, 0.8, 0.2, 0.7])
    num_nodes = hyperedge_index[0].max().item() + 1
    x = torch.randn((num_nodes, in_channels))

    conv = HypergraphConv(in_channels, out_channels)
    assert str(conv) == 'HypergraphConv(16, 32)'
    out = conv(x, hyperedge_index)
    assert out.size() == (num_nodes, out_channels)
    out = conv(x, hyperedge_index, hyperedge_weight)
    assert out.size() == (num_nodes, out_channels)
pytorch/test/nn/conv/test_le_conv.py000066400000000000000000000022261444262304300201470ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import LEConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_le_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = LEConv(16, 32)
    assert str(conv) == 'LEConv(16, 32)'
    out = conv(x, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        torch.allclose(jit(x, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj2.t()), out)
pytorch/test/nn/conv/test_lg_conv.py000066400000000000000000000031411444262304300201460ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import LGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_lg_conv():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = LGConv()
    assert str(conv) == 'LGConv()'
    out1 = conv(x, edge_index)
    assert out1.size() == (4, 8)
    assert torch.allclose(conv(x, adj1.t()), out1)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 8)
    assert torch.allclose(conv(x, adj2.t()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1)
        assert torch.allclose(conv(x, adj4.t()), out2)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1)
        assert torch.allclose(jit(x, edge_index, value), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1)
        assert torch.allclose(jit(x, adj4.t()), out2)
pytorch/test/nn/conv/test_message_passing.py000066400000000000000000000534701444262304300217010ustar00rootroot00000000000000import copy
from typing import Tuple, Union

import pytest
import torch
from torch import Tensor
from torch.nn import Linear

import torch_geometric.typing
from torch_geometric.nn import MessagePassing, aggr
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
)
from torch_geometric.utils import (
    add_self_loops,
    scatter,
    spmm,
    to_torch_csc_tensor,
)


class MyConv(MessagePassing):
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, aggr: str = 'add'):
        super().__init__(aggr=aggr)

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lin_l = Linear(in_channels[0], out_channels)
        self.lin_r = Linear(in_channels[1], out_channels)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_weight: OptTensor = None, size: Size = None) -> Tensor:
        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=size)
        out = self.lin_l(out)

        x_r = x[1]
        if x_r is not None:
            out += self.lin_r(x_r)

        return out

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        return spmm(adj_t, x[0], reduce=self.aggr)


class MyConvWithSelfLoops(MessagePassing):
    def __init__(self, aggr: str = 'add'):
        super().__init__(aggr=aggr)

    def forward(self, x: Tensor, edge_index: torch.Tensor) -> Tensor:
        edge_index, _ = add_self_loops(edge_index)

        # propagate_type: (x: Tensor)
        return self.propagate(edge_index, x=x, size=None)


def test_my_conv_basic():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, value, size=(4, 4))
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 4))

    conv = MyConv(8, 32)
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, (4, 4)), out, atol=1e-6)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)
    conv.fuse = False
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)
    conv.fuse = True

    # Bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, value, size=(4, 2))
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 2))

    conv = MyConv((8, 16), 32)
    out1 = conv((x1, x2), edge_index, value)
    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out1.size() == (2, 32)
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)
    assert torch.allclose(conv((x1, x2), adj1.t()), out1, atol=1e-6)
    assert torch.allclose(conv((x1, None), adj1.t()), out2, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv((x1, x2), adj2.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj2.t()), out2, atol=1e-6)
    conv.fuse = False
    assert torch.allclose(conv((x1, x2), adj1.t()), out1, atol=1e-6)
    assert torch.allclose(conv((x1, None), adj1.t()), out2, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv((x1, x2), adj2.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj2.t()), out2, atol=1e-6)

    # Test gradient computation for `torch.sparse` tensors:
    conv.fuse = True
    torch_adj_t = adj1.t().requires_grad_()
    out = conv((x1, x2), torch_adj_t)
    out.sum().backward()
    assert torch_adj_t.grad is not None


def test_my_conv_out_of_bounds():
    x = torch.randn(3, 8)
    value = torch.randn(4)

    conv = MyConv(8, 32)

    with pytest.raises(IndexError, match="valid indices"):
        edge_index = torch.tensor([[-1, 1, 2, 2], [0, 0, 1, 1]])
        conv(x, edge_index, value)

    with pytest.raises(IndexError, match="valid indices"):
        edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
        conv(x, edge_index, value)


def test_my_conv_jittable():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1))

    conv = MyConv(8, 32)
    out = conv(x1, edge_index, value)

    conv.jittable()  # Should succeed.
    torch.jit.script(conv.jittable())  # Should succeed.

    t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
    jit = torch.jit.script(conv.jittable(t))
    assert torch.allclose(jit(x1, edge_index, value), out, atol=1e-6)
    assert torch.allclose(jit(x1, edge_index, value, (4, 4)), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))

        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out, atol=1e-6)
        jit.fuse = False
        assert torch.allclose(jit(x1, adj.t()), out, atol=1e-6)
        jit.fuse = True

    conv = MyConv((8, 16), 32)
    out1 = conv((x1, x2), edge_index, value)
    out2 = conv((x1, None), edge_index, value, (4, 2))

    t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
    jit = torch.jit.script(conv.jittable(t))
    assert torch.allclose(jit((x1, x2), edge_index, value), out1)
    assert torch.allclose(jit((x1, x2), edge_index, value, (4, 2)), out1)
    assert torch.allclose(jit((x1, None), edge_index, value, (4, 2)), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 2))

        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj.t()), out2, atol=1e-6)
        jit.fuse = False
        assert torch.allclose(jit((x1, x2), adj.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj.t()), out2, atol=1e-6)
        jit.fuse = True


@pytest.mark.parametrize('aggr', ['add', 'sum', 'mean', 'min', 'max', 'mul'])
def test_my_conv_aggr(aggr):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_weight = torch.randn(edge_index.size(1))

    conv = MyConv(8, 32, aggr=aggr)
    out = conv(x, edge_index, edge_weight)
    assert out.size() == (4, 32)


def test_my_static_graph_conv():
    x1 = torch.randn(3, 4, 8)
    x2 = torch.randn(3, 2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1))
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))

    conv = MyConv(8, 32)
    out = conv(x1, edge_index, value)
    assert out.size() == (3, 4, 32)
    assert torch.allclose(conv(x1, edge_index, value, (4, 4)), out)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x1, adj.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 2))

    conv = MyConv((8, 16), 32)
    out1 = conv((x1, x2), edge_index, value)
    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out1.size() == (3, 2, 32)
    assert out2.size() == (3, 2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv((x1, x2), adj.t()), out1)
        assert torch.allclose(conv((x1, None), adj.t()), out2)


class MyMultipleAggrConv(MessagePassing):
    def __init__(self, **kwargs):
        super().__init__(aggr=['add', 'mean', 'max'], **kwargs)

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        # propagate_type: (x: Tensor)
        return self.propagate(edge_index, x=x, size=None)


@pytest.mark.parametrize('multi_aggr_tuple', [
    (dict(mode='cat'), 3),
    (dict(mode='proj', mode_kwargs=dict(in_channels=16, out_channels=16)), 1)
])
def test_my_multiple_aggr_conv(multi_aggr_tuple):
    # The 'cat' combine mode will expand the output dimensions by
    # the number of aggregators which is 3 here, while the 'proj'
    # mode keeps output dimensions unchanged.
    aggr_kwargs, expand = multi_aggr_tuple
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))

    conv = MyMultipleAggrConv(aggr_kwargs=aggr_kwargs)
    out = conv(x, edge_index)
    assert out.size() == (4, 16 * expand)
    assert torch.allclose(conv(x, adj1.t()), out)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj2.t()), out)


def test_my_multiple_aggr_conv_jittable():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = MyMultipleAggrConv()
    out = conv(x, edge_index)

    t = '(Tensor, Tensor) -> Tensor'
    jit = torch.jit.script(conv.jittable(t))
    assert torch.allclose(jit(x, edge_index), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out)


def test_copy():
    conv = MyConv(8, 32)
    conv2 = copy.copy(conv)

    assert conv != conv2
    assert torch.equal(conv.lin_l.weight, conv2.lin_l.weight)
    assert torch.equal(conv.lin_r.weight, conv2.lin_r.weight)
    assert conv.lin_l.weight.data_ptr == conv2.lin_l.weight.data_ptr
    assert conv.lin_r.weight.data_ptr == conv2.lin_r.weight.data_ptr

    conv = copy.deepcopy(conv)
    assert conv != conv2
    assert torch.equal(conv.lin_l.weight, conv2.lin_l.weight)
    assert torch.equal(conv.lin_r.weight, conv2.lin_r.weight)
    assert conv.lin_l.weight.data_ptr != conv2.lin_l.weight.data_ptr
    assert conv.lin_r.weight.data_ptr != conv2.lin_r.weight.data_ptr


class MyEdgeConv(MessagePassing):
    def __init__(self):
        super().__init__(aggr='add')

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        # edge_updater_type: (x: Tensor)
        edge_attr = self.edge_updater(edge_index, x=x)

        # propagate_type: (edge_attr: Tensor)
        return self.propagate(edge_index, edge_attr=edge_attr,
                              size=(x.size(0), x.size(0)))

    def edge_update(self, x_j: Tensor, x_i: Tensor) -> Tensor:
        return x_j - x_i

    def message(self, edge_attr: Tensor) -> Tensor:
        return edge_attr


def test_my_edge_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    row, col = edge_index
    expected = scatter(x[row] - x[col], col, dim=0, dim_size=4, reduce='sum')

    conv = MyEdgeConv()
    out = conv(x, edge_index)
    assert out.size() == (4, 16)
    assert torch.allclose(out, expected)
    assert torch.allclose(conv(x, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out)


def test_my_edge_conv_jittable():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = MyEdgeConv()
    out = conv(x, edge_index)

    t = '(Tensor, Tensor) -> Tensor'
    jit = torch.jit.script(conv.jittable(t))
    assert torch.allclose(jit(x, edge_index), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out)


num_pre_hook_calls = 0
num_hook_calls = 0


def test_message_passing_hooks():
    conv = MyConv(8, 32)

    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.randn(edge_index.size(1))
    adj = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    def pre_hook(module, inputs):
        assert module == conv
        global num_pre_hook_calls
        num_pre_hook_calls += 1
        return inputs

    def hook(module, inputs, output):
        assert module == conv
        global num_hook_calls
        num_hook_calls += 1
        return output

    handle1 = conv.register_propagate_forward_pre_hook(pre_hook)
    assert len(conv._propagate_forward_pre_hooks) == 1
    handle2 = conv.register_propagate_forward_hook(hook)
    assert len(conv._propagate_forward_hooks) == 1

    handle3 = conv.register_message_forward_pre_hook(pre_hook)
    assert len(conv._message_forward_pre_hooks) == 1
    handle4 = conv.register_message_forward_hook(hook)
    assert len(conv._message_forward_hooks) == 1

    handle5 = conv.register_aggregate_forward_pre_hook(pre_hook)
    assert len(conv._aggregate_forward_pre_hooks) == 1
    handle6 = conv.register_aggregate_forward_hook(hook)
    assert len(conv._aggregate_forward_hooks) == 1

    handle7 = conv.register_message_and_aggregate_forward_pre_hook(pre_hook)
    assert len(conv._message_and_aggregate_forward_pre_hooks) == 1
    handle8 = conv.register_message_and_aggregate_forward_hook(hook)
    assert len(conv._message_and_aggregate_forward_hooks) == 1

    out1 = conv(x, edge_index, value)
    assert num_pre_hook_calls == 3
    assert num_hook_calls == 3
    out2 = conv(x, adj.t())
    assert num_pre_hook_calls == 5
    assert num_hook_calls == 5
    assert torch.allclose(out1, out2)

    handle1.remove()
    assert len(conv._propagate_forward_pre_hooks) == 0
    handle2.remove()
    assert len(conv._propagate_forward_hooks) == 0

    handle3.remove()
    assert len(conv._message_forward_pre_hooks) == 0
    handle4.remove()
    assert len(conv._message_forward_hooks) == 0

    handle5.remove()
    assert len(conv._aggregate_forward_pre_hooks) == 0
    handle6.remove()
    assert len(conv._aggregate_forward_hooks) == 0

    handle7.remove()
    assert len(conv._message_and_aggregate_forward_pre_hooks) == 0
    handle8.remove()
    assert len(conv._message_and_aggregate_forward_hooks) == 0

    conv = MyEdgeConv()

    handle1 = conv.register_edge_update_forward_pre_hook(pre_hook)
    assert len(conv._edge_update_forward_pre_hooks) == 1
    handle2 = conv.register_edge_update_forward_hook(hook)
    assert len(conv._edge_update_forward_hooks) == 1

    out1 = conv(x, edge_index)
    assert num_pre_hook_calls == 6
    assert num_hook_calls == 6
    out2 = conv(x, adj.t())
    assert num_pre_hook_calls == 7
    assert num_hook_calls == 7
    assert torch.allclose(out1, out2)

    handle1.remove()
    assert len(conv._propagate_forward_pre_hooks) == 0
    handle2.remove()
    assert len(conv._propagate_forward_hooks) == 0


def test_modified_message_passing_hook():
    conv = MyConv(8, 32)

    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_weight = torch.randn(edge_index.size(1))

    out1 = conv(x, edge_index, edge_weight)

    def hook(module, inputs, output):
        assert len(inputs) == 1
        assert len(inputs[-1]) == 2
        assert 'x_j' in inputs[-1]
        assert 'edge_weight' in inputs[-1]
        return output + 1.

    conv.register_message_forward_hook(hook)

    out2 = conv(x, edge_index, edge_weight)
    assert not torch.allclose(out1, out2)


class MyDefaultArgConv(MessagePassing):
    def __init__(self):
        super().__init__(aggr='mean')

    # propagate_type: (x: Tensor)
    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        return self.propagate(edge_index, x=x)

    def message(self, x_j, zeros: bool = True):
        return x_j * 0 if zeros else x_j


def test_my_default_arg_conv():
    x = torch.randn(4, 1)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = MyDefaultArgConv()
    assert conv(x, edge_index).view(-1).tolist() == [0, 0, 0, 0]
    assert conv(x, adj1.t()).view(-1).tolist() == [0, 0, 0, 0]

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert conv(x, adj2.t()).view(-1).tolist() == [0, 0, 0, 0]


def test_my_default_arg_conv_jittable():
    conv = MyDefaultArgConv()

    # This should not succeed in JIT mode.
    with pytest.raises((RuntimeError, AttributeError)):
        torch.jit.script(conv.jittable())


class MyMultipleOutputConv(MessagePassing):
    def __init__(self):
        super().__init__()

    def forward(self, x: Tensor, edge_index: Tensor) -> Tuple[Tensor, Tensor]:
        # propagate_type: (x: Tensor)
        return self.propagate(edge_index, x=x, size=None)

    def message(self, x_j: Tensor) -> Tuple[Tensor, Tensor]:
        return x_j, x_j

    def aggregate(self, inputs: Tuple[Tensor, Tensor],
                  index: Tensor) -> Tuple[Tensor, Tensor]:
        return (scatter(inputs[0], index, dim=0, reduce='sum'),
                scatter(inputs[0], index, dim=0, reduce='mean'))

    def update(self, inputs: Tuple[Tensor, Tensor]) -> Tuple[Tensor, Tensor]:
        return inputs


def test_tuple_output():
    conv = MyMultipleOutputConv()

    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    out1 = conv(x, edge_index)
    assert isinstance(out1, tuple) and len(out1) == 2


def test_tuple_output_jittable():
    conv = MyMultipleOutputConv()

    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    out1 = conv(x, edge_index)
    assert isinstance(out1, tuple) and len(out1) == 2

    jit = torch.jit.script(conv.jittable())
    out2 = jit(x, edge_index)
    assert isinstance(out2, tuple) and len(out2) == 2
    assert torch.allclose(out1[0], out2[0])
    assert torch.allclose(out1[1], out2[1])


class MyExplainConv(MessagePassing):
    def __init__(self):
        super().__init__(aggr='add')

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        return self.propagate(edge_index, x=x)


def test_explain_message():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = MyExplainConv()
    assert conv(x, edge_index).abs().sum() != 0.

    conv.explain = True

    with pytest.raises(ValueError, match="pre-defined 'edge_mask'"):
        conv(x, edge_index)

    conv._edge_mask = torch.tensor([0, 0, 0, 0], dtype=torch.float)
    conv._apply_sigmoid = False
    assert conv(x, edge_index).abs().sum() == 0.


class MyAggregatorConv(MessagePassing):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        # propagate_type: (x: TEnsor)
        return self.propagate(edge_index, x=x, size=None)


@pytest.mark.parametrize('aggr_module', [
    aggr.MeanAggregation(),
    aggr.SumAggregation(),
    aggr.MaxAggregation(),
    aggr.SoftmaxAggregation(),
    aggr.PowerMeanAggregation(),
    aggr.MultiAggregation(['mean', 'max'])
])
def test_message_passing_with_aggr_module(aggr_module):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    row, col = edge_index
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = MyAggregatorConv(aggr=aggr_module)
    assert isinstance(conv.aggr_module, aggr.Aggregation)
    out = conv(x, edge_index)
    assert out.size(0) == 4 and out.size(1) in {8, 16}
    assert torch.allclose(conv(x, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj2.t()), out)


def test_message_passing_int32_edge_index():
    # Check that we can dispatch an int32 edge_index up to aggregation
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]], dtype=torch.int32)
    edge_weight = torch.randn(edge_index.shape[1])

    # Use a hook to promote the edge_index to long to workaround PyTorch CPU
    # backend restriction to int64 for the index.
    def cast_index_hook(module, inputs):
        input_dict = inputs[-1]
        input_dict['index'] = input_dict['index'].long()
        return (input_dict, )

    conv = MyConv(8, 32)
    conv.register_aggregate_forward_pre_hook(cast_index_hook)

    assert conv(x, edge_index, edge_weight).size() == (4, 32)


@pytest.mark.parametrize('num_nodes', [4, 8, 2, 0])
def test_traceable_my_conv_with_self_loops(num_nodes):
    # `torch.jit.trace` a `MessagePassing` layer that adds self loops and test
    # it across different input sizes.
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])

    conv = MyConvWithSelfLoops()
    traced_conv = torch.jit.trace(conv, ((x, edge_index)))
    scripted_conv = torch.jit.script(conv.jittable())

    x = torch.randn(num_nodes, 16)
    if num_nodes > 0:
        edge_index = torch.stack([
            torch.arange(0, num_nodes - 1),
            torch.arange(1, num_nodes),
        ], dim=0)
    else:
        edge_index = torch.empty((2, 0), dtype=torch.long)

    out = conv(x, edge_index)
    traced_out = traced_conv(x, edge_index)
    scripted_out = scripted_conv(x, edge_index)

    assert torch.allclose(out, traced_out)
    assert torch.allclose(out, scripted_out)
pytorch/test/nn/conv/test_mf_conv.py000066400000000000000000000045331444262304300201540ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import MFConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


def test_mf_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = MFConv(8, 32)
    assert str(conv) == 'MFConv(8, 32)'
    out = conv(x1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out)

    # Test bipartite message passing:
    conv = MFConv((8, 16), 32)
    assert str(conv) == 'MFConv((8, 16), 32)'

    out1 = conv((x1, x2), edge_index)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, (4, 2)), out1)

    out2 = conv((x1, None), edge_index, (4, 2))
    assert out2.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out1)
        assert torch.allclose(conv((x1, None), adj.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1)
        assert torch.allclose(jit((x1, None), edge_index, size=(4, 2)), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1)
        assert torch.allclose(jit((x1, None), adj.t()), out2)
pytorch/test/nn/conv/test_nn_conv.py000066400000000000000000000073271444262304300201710ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.typing
from torch_geometric.nn import NNConv
from torch_geometric.testing import is_full_test, withCUDA
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_coo_tensor


@withCUDA
def test_nn_conv(device):
    x1 = torch.randn(4, 8, device=device)
    x2 = torch.randn(2, 16, device=device)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]], device=device)
    value = torch.rand(edge_index.size(1), 3, device=device)
    adj1 = to_torch_coo_tensor(edge_index, value, size=(4, 4))

    nn = Seq(Lin(3, 32), ReLU(), Lin(32, 8 * 32))
    conv = NNConv(8, 32, nn=nn).to(device)
    assert str(conv) == (
        'NNConv(8, 32, aggr=add, nn=Sequential(\n'
        '  (0): Linear(in_features=3, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=256, bias=True)\n'
        '))')

    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, size=(4, 4)), out)
    assert torch.allclose(conv(x1, adj1.transpose(0, 1).coalesce()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, value), out)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out)

    # Test bipartite message passing:
    adj1 = to_torch_coo_tensor(edge_index, value, size=(4, 2))

    conv = NNConv((8, 16), 32, nn=nn).to(device)
    assert str(conv) == (
        'NNConv((8, 16), 32, aggr=add, nn=Sequential(\n'
        '  (0): Linear(in_features=3, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=256, bias=True)\n'
        '))')

    out1 = conv((x1, x2), edge_index, value)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)
    assert torch.allclose(conv((x1, x2),
                               adj1.transpose(0, 1).coalesce()), out1)

    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out2.size() == (2, 32)
    assert torch.allclose(conv((x1, None),
                               adj1.transpose(0, 1).coalesce()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out1)
        assert torch.allclose(conv((x1, None), adj2.t()), out2)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out1)
        assert torch.allclose(jit((x1, x2), edge_index, value, size=(4, 2)),
                              out1)
        assert torch.allclose(jit((x1, None), edge_index, value, size=(4, 2)),
                              out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out1)
        assert torch.allclose(jit((x1, None), adj2.t()), out2)
pytorch/test/nn/conv/test_pan_conv.py000066400000000000000000000020521444262304300203220ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import PANConv
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


@withPackage('torch_sparse')  # TODO `PANConv` returns a `SparseTensor`.
def test_pan_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))

    conv = PANConv(16, 32, filter_size=2)
    assert str(conv) == 'PANConv(16, 32, filter_size=2)'
    out1, M1 = conv(x, edge_index)
    assert out1.size() == (4, 32)

    out2, M2 = conv(x, adj1.t())
    assert torch.allclose(out1, out2, atol=1e-6)
    assert torch.allclose(M1.to_dense(), M2.to_dense())

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3, M3 = conv(x, adj2.t())
        assert torch.allclose(out1, out3, atol=1e-6)
        assert torch.allclose(M1.to_dense(), M3.to_dense())
pytorch/test/nn/conv/test_pdn_conv.py000066400000000000000000000031301444262304300203230ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import PDNConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


def test_pdn_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.randn(edge_index.size(1), 8)

    conv = PDNConv(16, 32, edge_dim=8, hidden_channels=128)
    assert str(conv) == "PDNConv(16, 32)"

    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_attr, (4, 4))
        assert torch.allclose(conv(x, adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index, edge_attr), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out, atol=1e-6)


def test_pdn_conv_with_sparse_node_input_feature():
    x = torch.sparse_coo_tensor(
        indices=torch.tensor([[0, 0], [0, 1]]),
        values=torch.tensor([1.0, 1.0]),
        size=torch.Size([4, 16]),
    )
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.randn(edge_index.size(1), 8)

    conv = PDNConv(16, 32, edge_dim=8, hidden_channels=128)

    out = conv(x, edge_index, edge_attr)
    assert out.size() == (4, 32)
pytorch/test/nn/conv/test_pna_conv.py000066400000000000000000000057331444262304300203330ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.data import Data
from torch_geometric.loader import DataLoader, NeighborLoader
from torch_geometric.nn import PNAConv
from torch_geometric.testing import is_full_test, onlyNeighborSampler
from torch_geometric.typing import SparseTensor

aggregators = ['sum', 'mean', 'min', 'max', 'var', 'std']
scalers = [
    'identity', 'amplification', 'attenuation', 'linear', 'inverse_linear'
]


@pytest.mark.parametrize('divide_input', [True, False])
def test_pna_conv(divide_input):
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    deg = torch.tensor([0, 3, 0, 1])
    value = torch.rand(edge_index.size(1), 3)

    conv = PNAConv(16, 32, aggregators, scalers, deg=deg, edge_dim=3, towers=4,
                   pre_layers=2, post_layers=2, divide_input=divide_input)
    assert str(conv) == 'PNAConv(16, 32, towers=4, edge_dim=3)'

    out = conv(x, edge_index, value)
    assert out.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index, value), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out, atol=1e-6)


@onlyNeighborSampler
def test_pna_conv_get_degree_histogram_neighbor_loader():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 2, 3], [1, 2, 3, 2, 0, 0, 0]])
    data = Data(num_nodes=5, edge_index=edge_index)
    loader = NeighborLoader(
        data,
        num_neighbors=[-1],
        input_nodes=None,
        batch_size=5,
        shuffle=False,
    )
    deg_hist = PNAConv.get_degree_histogram(loader)
    assert torch.equal(deg_hist, torch.tensor([1, 2, 1, 1]))


def test_pna_conv_get_degree_histogram_dataloader():
    edge_index_1 = torch.tensor([[0, 0, 0, 1, 1, 2, 3], [1, 2, 3, 2, 0, 0, 0]])
    edge_index_2 = torch.tensor([[1, 1, 2, 2, 0, 3, 3], [2, 3, 3, 1, 1, 0, 2]])
    edge_index_3 = torch.tensor([[1, 3, 2, 0, 0, 4, 2], [2, 0, 4, 1, 1, 0, 3]])
    edge_index_4 = torch.tensor([[0, 1, 2, 4, 0, 1, 3], [2, 3, 3, 1, 1, 0, 2]])

    data_1 = Data(num_nodes=5, edge_index=edge_index_1)  # hist = [1, 2 ,1 ,1]
    data_2 = Data(num_nodes=5, edge_index=edge_index_2)  # hist = [1, 1, 3]
    data_3 = Data(num_nodes=5, edge_index=edge_index_3)  # hist = [0, 3, 2]
    data_4 = Data(num_nodes=5, edge_index=edge_index_4)  # hist = [1, 1, 3]

    loader = DataLoader(
        [data_1, data_2, data_3, data_4],
        batch_size=1,
        shuffle=False,
    )
    deg_hist = PNAConv.get_degree_histogram(loader)
    assert torch.equal(deg_hist, torch.tensor([3, 7, 9, 1]))
pytorch/test/nn/conv/test_point_conv.py000066400000000000000000000056701444262304300207060ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.typing
from torch_geometric.nn import PointNetConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_point_net_conv():
    x1 = torch.randn(4, 16)
    pos1 = torch.randn(4, 3)
    pos2 = torch.randn(2, 3)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    local_nn = Seq(Lin(16 + 3, 32), ReLU(), Lin(32, 32))
    global_nn = Seq(Lin(32, 32))
    conv = PointNetConv(local_nn, global_nn)
    assert str(conv) == (
        'PointNetConv(local_nn=Sequential(\n'
        '  (0): Linear(in_features=19, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=32, bias=True)\n'
        '), global_nn=Sequential(\n'
        '  (0): Linear(in_features=32, out_features=32, bias=True)\n'
        '))')
    out = conv(x1, pos1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, pos1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, pos1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(OptTensor, Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptTensor, Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out = conv(x1, (pos1, pos2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, None), (pos1, pos2), edge_index), out)
    assert torch.allclose(conv(x1, (pos1, pos2), adj1.t()), out, atol=1e-6)
    assert torch.allclose(conv((x1, None), (pos1, pos2), adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv(x1, (pos1, pos2), adj2.t()), out, atol=1e-6)
        assert torch.allclose(conv((x1, None), (pos1, pos2), adj2.t()), out)

    if is_full_test():
        t = '(PairOptTensor, PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, None), (pos1, pos2), edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairOptTensor, PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, None), (pos1, pos2), adj2.t()), out)
pytorch/test/nn/conv/test_point_gnn_conv.py000066400000000000000000000030541444262304300215420ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import MLP, PointGNNConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_point_gnn_conv():
    x = torch.randn(6, 8)
    pos = torch.randn(6, 3)
    edge_index = torch.tensor([[0, 1, 1, 1, 2, 5], [1, 2, 3, 4, 3, 4]])
    adj1 = to_torch_csc_tensor(edge_index, size=(6, 6))

    conv = PointGNNConv(
        mlp_h=MLP([8, 16, 3]),
        mlp_f=MLP([3 + 8, 16, 8]),
        mlp_g=MLP([8, 16, 8]),
    )
    assert str(conv) == ('PointGNNConv(\n'
                         '  mlp_h=MLP(8, 16, 3),\n'
                         '  mlp_f=MLP(11, 16, 8),\n'
                         '  mlp_g=MLP(8, 16, 8),\n'
                         ')')

    out = conv(x, pos, edge_index)
    assert out.size() == (6, 8)
    assert torch.allclose(conv(x, pos, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(6, 6))
        assert torch.allclose(conv(x, pos, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, pos, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, pos, adj2.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_point_transformer_conv.py000066400000000000000000000055521444262304300233270ustar00rootroot00000000000000import torch
from torch.nn import Linear, ReLU, Sequential

import torch_geometric.typing
from torch_geometric.nn import PointTransformerConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_point_transformer_conv():
    x1 = torch.rand(4, 16)
    x2 = torch.randn(2, 8)
    pos1 = torch.rand(4, 3)
    pos2 = torch.randn(2, 3)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = PointTransformerConv(in_channels=16, out_channels=32)
    assert str(conv) == 'PointTransformerConv(16, 32)'

    out = conv(x1, pos1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, pos1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, pos1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, adj2.t()), out, atol=1e-6)

    pos_nn = Sequential(Linear(3, 16), ReLU(), Linear(16, 32))
    attn_nn = Sequential(Linear(32, 32), ReLU(), Linear(32, 32))
    conv = PointTransformerConv(16, 32, pos_nn, attn_nn)

    out = conv(x1, pos1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, pos1, adj1.t()), out, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x1, pos1, adj2.t()), out, atol=1e-6)

    # Test biparitite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    conv = PointTransformerConv((16, 8), 32)
    assert str(conv) == 'PointTransformerConv((16, 8), 32)'

    out = conv((x1, x2), (pos1, pos2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), (pos1, pos2), adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), (pos1, pos2), adj2.t()), out)

    if is_full_test():
        t = '(PairTensor, PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), (pos1, pos2), edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), (pos1, pos2), adj2.t()), out)
pytorch/test/nn/conv/test_ppf_conv.py000066400000000000000000000065041444262304300203370ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

import torch_geometric.typing
from torch_geometric.nn import PPFConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_ppf_conv():
    x1 = torch.randn(4, 16)
    pos1 = torch.randn(4, 3)
    pos2 = torch.randn(2, 3)
    n1 = F.normalize(torch.rand(4, 3), dim=-1)
    n2 = F.normalize(torch.rand(2, 3), dim=-1)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    local_nn = Seq(Lin(16 + 4, 32), ReLU(), Lin(32, 32))
    global_nn = Seq(Lin(32, 32))
    conv = PPFConv(local_nn, global_nn)
    assert str(conv) == (
        'PPFConv(local_nn=Sequential(\n'
        '  (0): Linear(in_features=20, out_features=32, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=32, out_features=32, bias=True)\n'
        '), global_nn=Sequential(\n'
        '  (0): Linear(in_features=32, out_features=32, bias=True)\n'
        '))')

    out = conv(x1, pos1, n1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, pos1, n1, adj1.t()), out, atol=1e-3)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, pos1, n1, adj2.t()), out, atol=1e-3)

    if is_full_test():
        t = '(OptTensor, Tensor, Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, n1, edge_index), out, atol=1e-3)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptTensor, Tensor, Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, pos1, n1, adj2.t()), out, atol=1e-3)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    out = conv(x1, (pos1, pos2), (n1, n2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, None), (pos1, pos2), (n1, n2), edge_index),
                          out, atol=1e-3)
    assert torch.allclose(conv(x1, (pos1, pos2), (n1, n2), adj1.t()), out,
                          atol=1e-3)
    assert torch.allclose(conv((x1, None), (pos1, pos2), (n1, n2), adj1.t()),
                          out, atol=1e-3)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv(x1, (pos1, pos2), (n1, n2), adj2.t()), out)
        assert torch.allclose(
            conv((x1, None), (pos1, pos2), (n1, n2), adj2.t()), out, atol=1e-3)

    if is_full_test():
        t = '(PairOptTensor, PairTensor, PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(
            jit((x1, None), (pos1, pos2), (n1, n2), edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairOptTensor, PairTensor, PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(
            jit((x1, None), (pos1, pos2), (n1, n2), adj2.t()), out, atol=1e-3)
pytorch/test/nn/conv/test_res_gated_graph_conv.py000066400000000000000000000042711444262304300226670ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import ResGatedGraphConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_res_gated_graph_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = ResGatedGraphConv(8, 32)
    assert str(conv) == 'ResGatedGraphConv(8, 32)'

    out = conv(x1, edge_index)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    conv = ResGatedGraphConv((8, 32), 32)
    assert str(conv) == 'ResGatedGraphConv((8, 32), 32)'

    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_rgat_conv.py000066400000000000000000000045631444262304300205120ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import RGATConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_coo_tensor


@pytest.mark.parametrize('mod', [
    'additive',
    'scaled',
    'f-additive',
    'f-scaled',
])
@pytest.mark.parametrize('attention_mechanism', [
    'within-relation',
    'across-relation',
])
@pytest.mark.parametrize('attention_mode', [
    'additive-self-attention',
    'multiplicative-self-attention',
])
def test_rgat_conv(mod, attention_mechanism, attention_mode):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_type = torch.tensor([0, 2, 1, 2])
    edge_attr = torch.randn((4, 8))

    conv = RGATConv(8, 20, num_relations=4, num_bases=4, mod=mod,
                    attention_mechanism=attention_mechanism,
                    attention_mode=attention_mode, heads=2, dim=1, edge_dim=8)
    assert str(conv) == 'RGATConv(8, 20, heads=2)'

    out = conv(x, edge_index, edge_type, edge_attr)
    assert out.size() == (4, 40)


def test_rgat_conv_jittable():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_attr = torch.randn((edge_index.size(1), 8))
    edge_type = torch.tensor([0, 2, 1, 2])
    adj1 = to_torch_coo_tensor(edge_index, edge_attr, size=(4, 4))

    conv = RGATConv(8, 20, num_relations=4, num_bases=4, mod='additive',
                    attention_mechanism='across-relation',
                    attention_mode='additive-self-attention', heads=2, dim=1,
                    edge_dim=8, bias=False)

    out = conv(x, edge_index, edge_type, edge_attr)
    assert out.size() == (4, 40)
    # t() expects a tensor with <= 2 sparse and 0 dense dimensions
    adj1_t = adj1.transpose(0, 1).coalesce()
    assert torch.allclose(conv(x, adj1_t, edge_type), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, edge_attr, (4, 4))
        assert torch.allclose(conv(x, adj2.t(), edge_type), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, OptTensor, Size, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index, edge_type),
                              conv(x, edge_index, edge_type))
pytorch/test/nn/conv/test_rgcn_conv.py000066400000000000000000000127041444262304300205020ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import FastRGCNConv, RGCNConv
from torch_geometric.testing import is_full_test, withCUDA
from torch_geometric.typing import SparseTensor

classes = [RGCNConv, FastRGCNConv]
confs = [(None, None), (2, None), (None, 2)]


@withCUDA
@pytest.mark.parametrize('conf', confs)
def test_rgcn_conv_equality(conf, device):
    num_bases, num_blocks = conf

    x1 = torch.randn(4, 4, device=device)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [0, 0, 1, 0, 1, 1],
    ], device=device)
    edge_type = torch.tensor([0, 1, 1, 0, 0, 1], device=device)

    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 0, 1, 1, 2, 2, 3],
        [0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1],
    ], device=device)
    edge_type = torch.tensor([0, 1, 1, 0, 0, 1, 2, 3, 3, 2, 2, 3],
                             device=device)

    torch.manual_seed(12345)
    conv1 = RGCNConv(4, 32, 4, num_bases, num_blocks, aggr='sum').to(device)

    torch.manual_seed(12345)
    conv2 = FastRGCNConv(4, 32, 4, num_bases, num_blocks,
                         aggr='sum').to(device)

    out1 = conv1(x1, edge_index, edge_type)
    out2 = conv2(x1, edge_index, edge_type)
    assert torch.allclose(out1, out2, atol=1e-2)

    if num_blocks is None:
        out1 = conv1(None, edge_index, edge_type)
        out2 = conv2(None, edge_index, edge_type)
        assert torch.allclose(out1, out2, atol=1e-2)


@withCUDA
@pytest.mark.parametrize('cls', classes)
@pytest.mark.parametrize('conf', confs)
def test_rgcn_conv(cls, conf, device):
    num_bases, num_blocks = conf

    x1 = torch.randn(4, 4, device=device)
    x2 = torch.randn(2, 16, device=device)
    idx1 = torch.arange(4, device=device)
    idx2 = torch.arange(2, device=device)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [0, 0, 1, 0, 1, 1],
    ], device=device)
    edge_type = torch.tensor([0, 1, 1, 0, 0, 1], device=device)

    conv = cls(4, 32, 2, num_bases, num_blocks, aggr='sum').to(device)
    assert str(conv) == f'{cls.__name__}(4, 32, num_relations=2)'

    out1 = conv(x1, edge_index, edge_type)
    assert out1.size() == (4, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_type, (4, 4))
        assert torch.allclose(conv(x1, adj.t()), out1, atol=1e-3)

    if num_blocks is None:
        out2 = conv(None, edge_index, edge_type)
        assert torch.allclose(conv(idx1, edge_index, edge_type), out2, 1e-3)
        assert out2.size() == (4, 32)
        if torch_geometric.typing.WITH_TORCH_SPARSE:
            assert torch.allclose(conv(None, adj.t()), out2, atol=1e-3)
            assert torch.allclose(conv(idx1, adj.t()), out2, atol=1e-3)

    if is_full_test():
        t = '(OptTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, edge_type), out1, atol=1e-3)
        if num_blocks is None:
            assert torch.allclose(jit(idx1, edge_index, edge_type), out2,
                                  atol=1e-3)
            assert torch.allclose(jit(None, edge_index, edge_type), out2,
                                  atol=1e-3)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out1)
        if num_blocks is None:
            assert torch.allclose(jit(idx1, adj.t()), out2, atol=1e-3)
            assert torch.allclose(jit(None, adj.t()), out2, atol=1e-3)

    # Test bipartite message passing:
    conv = cls((4, 16), 32, 2, num_bases, num_blocks, aggr='sum').to(device)
    assert str(conv) == f'{cls.__name__}((4, 16), 32, num_relations=2)'

    out1 = conv((x1, x2), edge_index, edge_type)
    assert out1.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_type, (4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out1, atol=1e-3)

    if num_blocks is None:
        out2 = conv((None, idx2), edge_index, edge_type)
        assert out2.size() == (2, 32)
        assert torch.allclose(conv((idx1, idx2), edge_index, edge_type), out2,
                              atol=1e-3)
        if torch_geometric.typing.WITH_TORCH_SPARSE:
            assert torch.allclose(conv((None, idx2), adj.t()), out2, atol=1e-3)
            assert torch.allclose(conv((idx1, idx2), adj.t()), out2, atol=1e-3)

    if is_full_test():
        t = '(Tuple[OptTensor, Tensor], Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, edge_type), out1,
                              atol=1e-3)
        if num_blocks is None:
            assert torch.allclose(jit((None, idx2), edge_index, edge_type),
                                  out2, atol=1e-3)
            assert torch.allclose(jit((idx1, idx2), edge_index, edge_type),
                                  out2, atol=1e-3)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tuple[OptTensor, Tensor], SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1, atol=1e-3)
        if num_blocks is None:
            assert torch.allclose(jit((None, idx2), adj.t()), out2, atol=1e-3)
            assert torch.allclose(jit((idx1, idx2), adj.t()), out2, atol=1e-3)
pytorch/test/nn/conv/test_sage_conv.py000066400000000000000000000076421444262304300204750ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import SAGEConv
from torch_geometric.testing import assert_module, is_full_test
from torch_geometric.typing import SparseTensor


@pytest.mark.parametrize('project', [False, True])
@pytest.mark.parametrize('aggr', ['mean', 'sum'])
def test_sage_conv(project, aggr):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = SAGEConv(8, 32, project=project, aggr=aggr)
    assert str(conv) == f'SAGEConv(8, 32, aggr={aggr})'

    out = assert_module(conv, x, edge_index, expected_size=(4, 32))

    if is_full_test():
        t = '(Tensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, size=(4, 4)), out, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        t = '(Tensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out, atol=1e-6)

    # Test bipartite message passing:
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)

    conv = SAGEConv((8, 16), 32, project=project, aggr=aggr)
    assert str(conv) == f'SAGEConv((8, 16), 32, aggr={aggr})'

    out1 = assert_module(conv, (x1, x2), edge_index, expected_size=(2, 32))
    out2 = assert_module(conv, (x1, None), edge_index, size=(4, 2),
                         expected_size=(2, 32))

    if is_full_test():
        t = '(OptPairTensor, Tensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out1, atol=1e-6)
        assert torch.allclose(jit((x1, x2), edge_index, size=(4, 2)), out1)
        assert torch.allclose(jit((x1, None), edge_index, size=(4, 2)), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        t = '(OptPairTensor, SparseTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj.t()), out2, atol=1e-6)


@pytest.mark.parametrize('project', [False, True])
def test_lazy_sage_conv(project):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    if project:
        with pytest.raises(ValueError, match="does not support lazy"):
            SAGEConv(-1, 32, project=project)
    else:
        conv = SAGEConv(-1, 32, project=project)
        assert str(conv) == 'SAGEConv(-1, 32, aggr=mean)'

        out = conv(x, edge_index)
        assert out.size() == (4, 32)


def test_lstm_aggr_sage_conv():
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = SAGEConv(8, 32, aggr='lstm')
    assert str(conv) == 'SAGEConv(8, 32, aggr=lstm)'

    assert_module(conv, x, edge_index, expected_size=(4, 32),
                  test_edge_permutation=False)

    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 1, 0]])
    with pytest.raises(ValueError, match="'index' tensor is not sorted"):
        conv(x, edge_index)


@pytest.mark.parametrize('aggr_kwargs', [
    dict(mode='cat'),
    dict(mode='proj', mode_kwargs=dict(in_channels=8, out_channels=16)),
    dict(mode='attn', mode_kwargs=dict(in_channels=8, out_channels=16,
                                       num_heads=4)),
    dict(mode='sum'),
])
def test_multi_aggr_sage_conv(aggr_kwargs):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    aggr_kwargs['aggrs_kwargs'] = [{}, {}, {}, dict(learn=True, t=1)]
    conv = SAGEConv(8, 32, aggr=['mean', 'max', 'sum', 'softmax'],
                    aggr_kwargs=aggr_kwargs)

    assert_module(conv, x, edge_index, expected_size=(4, 32))
pytorch/test/nn/conv/test_sg_conv.py000066400000000000000000000040501444262304300201550ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import SGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_sg_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = SGConv(16, 32, K=10)
    assert str(conv) == 'SGConv(16, 32, K=10)'

    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)

    conv.cached = True
    conv(x, edge_index)
    assert conv._cached_x is not None
    assert torch.allclose(conv(x, edge_index), out1, atol=1e-6)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
pytorch/test/nn/conv/test_signed_conv.py000066400000000000000000000072611444262304300210240ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import SignedConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_signed_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv1 = SignedConv(16, 32, first_aggr=True)
    assert str(conv1) == 'SignedConv(16, 32, first_aggr=True)'

    conv2 = SignedConv(32, 48, first_aggr=False)
    assert str(conv2) == 'SignedConv(32, 48, first_aggr=False)'

    out1 = conv1(x, edge_index, edge_index)
    assert out1.size() == (4, 64)
    assert torch.allclose(conv1(x, adj1.t(), adj1.t()), out1)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv1(x, adj2.t(), adj2.t()), out1)

    out2 = conv2(out1, edge_index, edge_index)
    assert out2.size() == (4, 96)
    assert torch.allclose(conv2(out1, adj1.t(), adj1.t()), out2)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv2(out1, adj2.t(), adj2.t()), out2)

    if is_full_test():
        t = '(Tensor, Tensor, Tensor) -> Tensor'
        jit1 = torch.jit.script(conv1.jittable(t))
        jit2 = torch.jit.script(conv2.jittable(t))
        assert torch.allclose(jit1(x, edge_index, edge_index), out1)
        assert torch.allclose(jit2(out1, edge_index, edge_index), out2)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, SparseTensor) -> Tensor'
        jit1 = torch.jit.script(conv1.jittable(t))
        jit2 = torch.jit.script(conv2.jittable(t))
        assert torch.allclose(jit1(x, adj2.t(), adj2.t()), out1)
        assert torch.allclose(jit2(out1, adj2.t(), adj2.t()), out2)

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    assert torch.allclose(conv1((x, x[:2]), edge_index, edge_index), out1[:2],
                          atol=1e-6)
    assert torch.allclose(conv1((x, x[:2]), adj1.t(), adj1.t()), out1[:2],
                          atol=1e-6)
    assert torch.allclose(conv2((out1, out1[:2]), edge_index, edge_index),
                          out2[:2], atol=1e-6)
    assert torch.allclose(conv2((out1, out1[:2]), adj1.t(), adj1.t()),
                          out2[:2], atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv1((x, x[:2]), adj2.t(), adj2.t()), out1[:2],
                              atol=1e-6)
        assert torch.allclose(conv2((out1, out1[:2]), adj2.t(), adj2.t()),
                              out2[:2], atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor, Tensor) -> Tensor'
        jit1 = torch.jit.script(conv1.jittable(t))
        jit2 = torch.jit.script(conv2.jittable(t))
        assert torch.allclose(jit1((x, x[:2]), edge_index, edge_index),
                              out1[:2], atol=1e-6)
        assert torch.allclose(jit2((out1, out1[:2]), edge_index, edge_index),
                              out2[:2], atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, SparseTensor) -> Tensor'
        jit1 = torch.jit.script(conv1.jittable(t))
        jit2 = torch.jit.script(conv2.jittable(t))
        assert torch.allclose(jit1((x, x[:2]), adj2.t(), adj2.t()), out1[:2],
                              atol=1e-6)
        assert torch.allclose(jit2((out1, out1[:2]), adj2.t(), adj2.t()),
                              out2[:2], atol=1e-6)
pytorch/test/nn/conv/test_simple_conv.py000066400000000000000000000044011444262304300210350ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import SimpleConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


@pytest.mark.parametrize('aggr, combine_root', [
    ('mean', None),
    ('sum', 'sum'),
    (['mean', 'max'], 'cat'),
    ('mean', 'self_loop'),
])
def test_simple_conv(aggr, combine_root):
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = SimpleConv(aggr, combine_root)
    assert str(conv) == 'SimpleConv()'

    num_aggrs = 1 if isinstance(aggr, str) else len(aggr)
    output_size = sum([8] * num_aggrs) + (8 if combine_root == 'cat' else 0)

    out = conv(x1, edge_index)
    assert out.size() == (4, output_size)
    assert torch.allclose(conv(x1, edge_index, size=(4, 4)), out)
    assert torch.allclose(conv(x1, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out)
        assert torch.allclose(jit(x1, edge_index, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out)

    # Test bipartite message passing:
    if combine_root != 'self_loop':
        adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

        out = conv((x1, x2), edge_index)
        assert out.size() == (2, output_size)
        assert torch.allclose(conv((x1, x2), edge_index, size=(4, 2)), out)
        assert torch.allclose(conv((x1, x2), adj1.t()), out)

        if torch_geometric.typing.WITH_TORCH_SPARSE:
            adj2 = SparseTensor.from_edge_index(edge_index,
                                                sparse_sizes=(4, 2))
            assert torch.allclose(conv((x1, x2), adj2.t()), out)
pytorch/test/nn/conv/test_spline_conv.py000066400000000000000000000070041444262304300210400ustar00rootroot00000000000000import warnings

import torch

import torch_geometric.typing
from torch_geometric.nn import SplineConv
from torch_geometric.testing import is_full_test, withPackage
from torch_geometric.typing import SparseTensor


@withPackage('torch_spline_conv')
def test_spline_conv():
    warnings.filterwarnings('ignore', '.*non-optimized CPU version.*')

    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1), 3)

    conv = SplineConv(8, 32, dim=3, kernel_size=5)
    assert str(conv) == 'SplineConv(8, 32, dim=3)'
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)
    assert torch.allclose(conv(x1, edge_index, value, size=(4, 4)), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x1, adj.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index, value), out, atol=1e-6)
        assert torch.allclose(jit(x1, edge_index, value, size=(4, 4)), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj.t()), out, atol=1e-6)

    # Test bipartite message passing:
    conv = SplineConv((8, 16), 32, dim=3, kernel_size=5)
    assert str(conv) == 'SplineConv((8, 16), 32, dim=3)'

    out1 = conv((x1, x2), edge_index, value)
    assert out1.size() == (2, 32)
    assert torch.allclose(conv((x1, x2), edge_index, value, (4, 2)), out1)

    out2 = conv((x1, None), edge_index, value, (4, 2))
    assert out2.size() == (2, 32)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, value, (4, 2))
        assert torch.allclose(conv((x1, x2), adj.t()), out1, atol=1e-6)
        assert torch.allclose(conv((x1, None), adj.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(OptPairTensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index, value), out1)
        assert torch.allclose(jit((x1, x2), edge_index, value, size=(4, 2)),
                              out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), edge_index, value, size=(4, 2)),
                              out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptPairTensor, SparseTensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj.t()), out1, atol=1e-6)
        assert torch.allclose(jit((x1, None), adj.t()), out2, atol=1e-6)


@withPackage('torch_spline_conv')
def test_lazy_spline_conv():
    warnings.filterwarnings('ignore', '.*non-optimized CPU version.*')

    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    value = torch.rand(edge_index.size(1), 3)

    conv = SplineConv(-1, 32, dim=3, kernel_size=5)
    assert str(conv) == 'SplineConv(-1, 32, dim=3)'
    out = conv(x1, edge_index, value)
    assert out.size() == (4, 32)

    conv = SplineConv((-1, -1), 32, dim=3, kernel_size=5)
    assert str(conv) == 'SplineConv((-1, -1), 32, dim=3)'
    out = conv((x1, x2), edge_index, value)
    assert out.size() == (2, 32)
pytorch/test/nn/conv/test_ssg_conv.py000066400000000000000000000041021444262304300203360ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import SSGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_ssg_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = SSGConv(16, 32, alpha=0.1, K=10)
    assert str(conv) == 'SSGConv(16, 32, K=10, alpha=0.1)'

    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)

    conv.cached = True
    conv(x, edge_index)
    assert conv._cached_h is not None
    assert torch.allclose(conv(x, edge_index), out1, atol=1e-6)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
pytorch/test/nn/conv/test_static_graph.py000066400000000000000000000015501444262304300211710ustar00rootroot00000000000000import torch

from torch_geometric.data import Batch, Data
from torch_geometric.nn import ChebConv, GCNConv, MessagePassing


class MyConv(MessagePassing):
    def forward(self, x, edge_index):
        return self.propagate(edge_index, x=x)


def test_static_graph():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    x1, x2 = torch.randn(3, 8), torch.randn(3, 8)

    data1 = Data(edge_index=edge_index, x=x1)
    data2 = Data(edge_index=edge_index, x=x2)
    batch = Batch.from_data_list([data1, data2])

    x = torch.stack([x1, x2], dim=0)
    for conv in [MyConv(), GCNConv(8, 16), ChebConv(8, 16, K=2)]:
        out1 = conv(batch.x, batch.edge_index)
        assert out1.size(0) == 6
        conv.node_dim = 1
        out2 = conv(x, edge_index)
        assert out2.size()[:2] == (2, 3)
        assert torch.allclose(out1, out2.view(-1, out2.size(-1)))
pytorch/test/nn/conv/test_supergat_conv.py000066400000000000000000000031641444262304300214030ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import SuperGATConv
from torch_geometric.typing import SparseTensor


@pytest.mark.parametrize('att_type', ['MX', 'SD'])
def test_supergat_conv(att_type):
    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    conv = SuperGATConv(8, 32, heads=2, attention_type=att_type,
                        neg_sample_ratio=1.0, edge_sample_ratio=1.0)
    assert str(conv) == f'SuperGATConv(8, 32, heads=2, type={att_type})'

    out = conv(x, edge_index)
    assert out.size() == (4, 64)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x, adj.t()), out, atol=1e-6)

    # Negative samples are given:
    neg_edge_index = conv.negative_sampling(edge_index, x.size(0))
    assert torch.allclose(conv(x, edge_index, neg_edge_index), out)
    att_loss = conv.get_attention_loss()
    assert isinstance(att_loss, torch.Tensor) and att_loss > 0

    # Batch of graphs:
    x = torch.randn(8, 8)
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 6, 7],
                               [0, 0, 1, 1, 4, 4, 5, 5]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])
    out = conv(x, edge_index, batch=batch)
    assert out.size() == (8, 64)

    # Batch of graphs and negative samples are given:
    neg_edge_index = conv.negative_sampling(edge_index, x.size(0), batch)
    assert torch.allclose(conv(x, edge_index, neg_edge_index), out)
    att_loss = conv.get_attention_loss()
    assert isinstance(att_loss, torch.Tensor) and att_loss > 0
pytorch/test/nn/conv/test_tag_conv.py000066400000000000000000000037001444262304300203200ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import TAGConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_tag_conv():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    value = torch.rand(edge_index.size(1))
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))
    adj2 = to_torch_csc_tensor(edge_index, value, size=(4, 4))

    conv = TAGConv(16, 32)
    assert str(conv) == 'TAGConv(16, 32, K=3)'

    out1 = conv(x, edge_index)
    assert out1.size() == (4, 32)
    assert torch.allclose(conv(x, adj1.t()), out1, atol=1e-6)

    out2 = conv(x, edge_index, value)
    assert out2.size() == (4, 32)
    assert torch.allclose(conv(x, adj2.t()), out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj3 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        adj4 = SparseTensor.from_edge_index(edge_index, value, (4, 4))
        assert torch.allclose(conv(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(conv(x, adj4.t()), out2, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out1, atol=1e-6)
        assert torch.allclose(jit(x, edge_index, value), out2, atol=1e-6)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, adj3.t()), out1, atol=1e-6)
        assert torch.allclose(jit(x, adj4.t()), out2, atol=1e-6)


def test_static_tag_conv():
    x = torch.randn(3, 4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    conv = TAGConv(16, 32)
    out = conv(x, edge_index)
    assert out.size() == (3, 4, 32)
pytorch/test/nn/conv/test_transformer_conv.py000066400000000000000000000074371444262304300221220ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import TransformerConv
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import to_torch_csc_tensor


def test_transformer_conv():
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 16)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = TransformerConv(8, 32, heads=2, beta=True)
    assert str(conv) == 'TransformerConv(8, 32, heads=2)'

    out = conv(x1, edge_index)
    assert out.size() == (4, 64)
    assert torch.allclose(conv(x1, adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(conv(x1, adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(Tensor, Tensor, NoneType, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, NoneType, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x1, adj2.t()), out, atol=1e-6)

    # Test `return_attention_weights`.
    result = conv(x1, edge_index, return_attention_weights=True)
    assert torch.allclose(result[0], out)
    assert result[1][0].size() == (2, 4)
    assert result[1][1].size() == (4, 2)
    assert result[1][1].min() >= 0 and result[1][1].max() <= 1
    assert conv._alpha is None

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        result = conv(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 4
        assert conv._alpha is None

    if is_full_test():
        t = ('(Tensor, Tensor, NoneType, bool) -> '
             'Tuple[Tensor, Tuple[Tensor, Tensor]]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, edge_index, return_attention_weights=True)
        assert torch.allclose(result[0], out)
        assert result[1][0].size() == (2, 4)
        assert result[1][1].size() == (4, 2)
        assert result[1][1].min() >= 0 and result[1][1].max() <= 1
        assert conv._alpha is None

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = ('(Tensor, SparseTensor, NoneType, bool) -> '
             'Tuple[Tensor, SparseTensor]')
        jit = torch.jit.script(conv.jittable(t))
        result = jit(x1, adj2.t(), return_attention_weights=True)
        assert torch.allclose(result[0], out, atol=1e-6)
        assert result[1].sizes() == [4, 4, 2] and result[1].nnz() == 4
        assert conv._alpha is None

    # Test bipartite message passing:
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 2))

    conv = TransformerConv((8, 16), 32, heads=2, beta=True)
    assert str(conv) == 'TransformerConv((8, 16), 32, heads=2)'

    out = conv((x1, x2), edge_index)
    assert out.size() == (2, 64)
    assert torch.allclose(conv((x1, x2), adj1.t()), out, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 2))
        assert torch.allclose(conv((x1, x2), adj2.t()), out, atol=1e-6)

    if is_full_test():
        t = '(PairTensor, Tensor, NoneType, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(PairTensor, SparseTensor, NoneType, NoneType) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit((x1, x2), adj2.t()), out, atol=1e-6)
pytorch/test/nn/conv/test_wl_conv.py000066400000000000000000000020641444262304300201710ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import WLConv
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import one_hot, to_torch_csc_tensor


def test_wl_conv():
    x1 = torch.tensor([1, 0, 0, 1])
    x2 = one_hot(x1)
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    adj1 = to_torch_csc_tensor(edge_index, size=(4, 4))

    conv = WLConv()
    assert str(conv) == 'WLConv()'

    out = conv(x1, edge_index)
    assert out.tolist() == [0, 1, 1, 0]
    assert torch.equal(conv(x2, edge_index), out)
    assert torch.equal(conv(x1, adj1.t()), out)
    assert torch.equal(conv(x2, adj1.t()), out)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.equal(conv(x1, adj2.t()), out)
        assert torch.equal(conv(x2, adj2.t()), out)

    assert conv.histogram(out).tolist() == [[2, 2]]
    assert torch.allclose(conv.histogram(out, norm=True),
                          torch.tensor([[0.7071, 0.7071]]))
pytorch/test/nn/conv/test_wl_conv_continuous.py000066400000000000000000000017441444262304300224630ustar00rootroot00000000000000import torch

from torch_geometric.nn import WLConvContinuous
from torch_geometric.testing import is_full_test


def test_wl_conv():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], dtype=torch.long)
    x = torch.tensor([[-1], [0], [1]], dtype=torch.float)

    conv = WLConvContinuous()
    assert str(conv) == 'WLConvContinuous()'

    out = conv(x, edge_index)
    assert out.tolist() == [[-0.5], [0.0], [0.5]]

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor, Size) -> Tensor'
        jit = torch.jit.script(conv.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)

    # Test bipartite message passing:
    x1 = torch.randn(4, 8)
    x2 = torch.randn(2, 8)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_weight = torch.randn(edge_index.size(1))

    out = conv((x1, None), edge_index, edge_weight, size=(4, 2))
    assert out.size() == (2, 8)

    out = conv((x1, x2), edge_index, edge_weight)
    assert out.size() == (2, 8)
pytorch/test/nn/conv/test_x_conv.py000066400000000000000000000014501444262304300200140ustar00rootroot00000000000000import torch

from torch_geometric.nn import XConv
from torch_geometric.testing import is_full_test, withPackage


@withPackage('torch_cluster')
def test_x_conv():
    x = torch.randn(8, 16)
    pos = torch.rand(8, 3)
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])

    conv = XConv(16, 32, dim=3, kernel_size=2, dilation=2)
    assert str(conv) == 'XConv(16, 32)'

    torch.manual_seed(12345)
    out1 = conv(x, pos)
    assert out1.size() == (8, 32)

    torch.manual_seed(12345)
    out2 = conv(x, pos, batch)
    assert out2.size() == (8, 32)

    if is_full_test():
        jit = torch.jit.script(conv)

        torch.manual_seed(12345)
        assert torch.allclose(jit(x, pos), out1, atol=1e-6)

        torch.manual_seed(12345)
        assert torch.allclose(jit(x, pos, batch), out2, atol=1e-6)
pytorch/test/nn/conv/utils/000077500000000000000000000000001444262304300162475ustar00rootroot00000000000000pytorch/test/nn/conv/utils/test_gnn_cheatsheet.py000066400000000000000000000027331444262304300226440ustar00rootroot00000000000000from torch_geometric.nn.conv import utils


def test_gnn_cheatsheet():
    assert utils.paper_title('GCNConv') == ('Semi-supervised Classification '
                                            'with Graph Convolutional '
                                            'Networks')
    assert utils.paper_link('GCNConv') == 'https://arxiv.org/abs/1609.02907'

    assert utils.supports_sparse_tensor('GCNConv')
    assert not utils.supports_sparse_tensor('ChebConv')

    assert utils.supports_edge_weights('GraphConv')
    assert not utils.supports_edge_weights('SAGEConv')

    assert utils.supports_edge_features('GATConv')
    assert not utils.supports_edge_features('SimpleConv')

    assert utils.supports_bipartite_graphs('SAGEConv')
    assert not utils.supports_bipartite_graphs('GCNConv')

    assert utils.supports_static_graphs('GCNConv')
    assert not utils.supports_static_graphs('GATConv')

    assert utils.supports_lazy_initialization('SAGEConv')
    assert not utils.supports_lazy_initialization('GatedGraphConv')

    assert utils.processes_heterogeneous_graphs('RGCNConv')
    assert utils.processes_heterogeneous_graphs('HeteroConv')
    assert not utils.processes_heterogeneous_graphs('GCNConv')

    assert utils.processes_hypergraphs('HypergraphConv')
    assert not utils.processes_hypergraphs('SAGEConv')

    assert utils.processes_point_clouds('DynamicEdgeConv')
    assert utils.processes_point_clouds('XConv')
    assert not utils.processes_point_clouds('CuGraphSAGEConv')
pytorch/test/nn/dense/000077500000000000000000000000001444262304300152405ustar00rootroot00000000000000pytorch/test/nn/dense/test_dense_gat_conv.py000066400000000000000000000037521444262304300216360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import DenseGATConv, GATConv
from torch_geometric.testing import is_full_test


@pytest.mark.parametrize('heads', [1, 4])
@pytest.mark.parametrize('concat', [True, False])
def test_dense_gat_conv(heads, concat):
    channels = 16
    sparse_conv = GATConv(channels, channels, heads=heads, concat=concat)
    dense_conv = DenseGATConv(channels, channels, heads=heads, concat=concat)
    assert str(dense_conv) == f'DenseGATConv(16, 16, heads={heads})'

    # Ensure same weights and bias:
    dense_conv.lin = sparse_conv.lin_src
    dense_conv.att_src = sparse_conv.att_src
    dense_conv.att_dst = sparse_conv.att_dst
    dense_conv.bias = sparse_conv.bias

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 1, 1, 2, 3, 4], [1, 0, 2, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)

    x = torch.cat([x, x.new_zeros(1, channels)], dim=0).view(2, 3, channels)
    adj = torch.Tensor([
        [
            [0, 1, 0],
            [1, 0, 1],
            [0, 1, 0],
        ],
        [
            [0, 1, 0],
            [1, 0, 0],
            [0, 0, 0],
        ],
    ])
    mask = torch.tensor([[1, 1, 1], [1, 1, 0]], dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)

    if is_full_test():
        jit = torch.jit.script(dense_conv)
        assert torch.allclose(jit(x, adj, mask), dense_out)

    assert dense_out[1, 2].abs().sum() == 0
    dense_out = dense_out.view(6, dense_out.size(-1))[:-1]
    assert torch.allclose(sparse_out, dense_out, atol=1e-4)


def test_dense_gat_conv_with_broadcasting():
    batch_size, num_nodes, channels = 8, 3, 16
    conv = DenseGATConv(channels, channels, heads=4)

    x = torch.randn(batch_size, num_nodes, channels)
    adj = torch.Tensor([
        [0, 1, 1],
        [1, 0, 1],
        [1, 1, 0],
    ])

    assert conv(x, adj).size() == (batch_size, num_nodes, 64)
    mask = torch.tensor([1, 1, 1], dtype=torch.bool)
    assert conv(x, adj, mask).size() == (batch_size, num_nodes, 64)
pytorch/test/nn/dense/test_dense_gcn_conv.py000066400000000000000000000035351444262304300216310ustar00rootroot00000000000000import torch

from torch_geometric.nn import DenseGCNConv, GCNConv
from torch_geometric.testing import is_full_test


def test_dense_gcn_conv():
    channels = 16
    sparse_conv = GCNConv(channels, channels)
    dense_conv = DenseGCNConv(channels, channels)
    assert str(dense_conv) == 'DenseGCNConv(16, 16)'

    # Ensure same weights and bias:
    dense_conv.lin.weight = sparse_conv.lin.weight
    dense_conv.bias = sparse_conv.bias

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 4],
                               [1, 2, 0, 2, 0, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)
    assert sparse_out.size() == (5, channels)

    x = torch.cat([x, x.new_zeros(1, channels)], dim=0).view(2, 3, channels)
    adj = torch.Tensor([
        [
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 0],
        ],
        [
            [0, 1, 0],
            [1, 0, 0],
            [0, 0, 0],
        ],
    ])
    mask = torch.tensor([[1, 1, 1], [1, 1, 0]], dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)
    assert dense_out.size() == (2, 3, channels)

    if is_full_test():
        jit = torch.jit.script(dense_conv)
        assert torch.allclose(jit(x, adj, mask), dense_out)

    assert dense_out[1, 2].abs().sum() == 0
    dense_out = dense_out.view(6, channels)[:-1]
    assert torch.allclose(sparse_out, dense_out, atol=1e-4)


def test_dense_gcn_conv_with_broadcasting():
    batch_size, num_nodes, channels = 8, 3, 16
    conv = DenseGCNConv(channels, channels)

    x = torch.randn(batch_size, num_nodes, channels)
    adj = torch.Tensor([
        [0, 1, 1],
        [1, 0, 1],
        [1, 1, 0],
    ])

    assert conv(x, adj).size() == (batch_size, num_nodes, channels)
    mask = torch.tensor([1, 1, 1], dtype=torch.bool)
    assert conv(x, adj, mask).size() == (batch_size, num_nodes, channels)
pytorch/test/nn/dense/test_dense_gin_conv.py000066400000000000000000000042411444262304300216320ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

from torch_geometric.nn import DenseGINConv, GINConv
from torch_geometric.testing import is_full_test


def test_dense_gin_conv():
    channels = 16
    nn = Seq(Lin(channels, channels), ReLU(), Lin(channels, channels))
    sparse_conv = GINConv(nn)
    dense_conv = DenseGINConv(nn)
    dense_conv = DenseGINConv(nn, train_eps=True)
    assert str(dense_conv) == (
        'DenseGINConv(nn=Sequential(\n'
        '  (0): Linear(in_features=16, out_features=16, bias=True)\n'
        '  (1): ReLU()\n'
        '  (2): Linear(in_features=16, out_features=16, bias=True)\n'
        '))')

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 4],
                               [1, 2, 0, 2, 0, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)
    assert sparse_out.size() == (5, channels)

    x = torch.cat([x, x.new_zeros(1, channels)], dim=0).view(2, 3, channels)
    adj = torch.Tensor([
        [
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 0],
        ],
        [
            [0, 1, 0],
            [1, 0, 0],
            [0, 0, 0],
        ],
    ])
    mask = torch.tensor([[1, 1, 1], [1, 1, 0]], dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)
    assert dense_out.size() == (2, 3, channels)

    if is_full_test():
        jit = torch.jit.script(dense_conv)
        assert torch.allclose(jit(x, adj, mask), dense_out)

    assert dense_out[1, 2].abs().sum().item() == 0
    dense_out = dense_out.view(6, channels)[:-1]
    assert torch.allclose(sparse_out, dense_out, atol=1e-04)


def test_dense_gin_conv_with_broadcasting():
    batch_size, num_nodes, channels = 8, 3, 16
    nn = Seq(Lin(channels, channels), ReLU(), Lin(channels, channels))
    conv = DenseGINConv(nn)

    x = torch.randn(batch_size, num_nodes, channels)
    adj = torch.Tensor([
        [0, 1, 1],
        [1, 0, 1],
        [1, 1, 0],
    ])

    assert conv(x, adj).size() == (batch_size, num_nodes, channels)
    mask = torch.tensor([1, 1, 1], dtype=torch.bool)
    assert conv(x, adj, mask).size() == (batch_size, num_nodes, channels)
pytorch/test/nn/dense/test_dense_graph_conv.py000066400000000000000000000056161444262304300221650ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import DenseGraphConv, GraphConv
from torch_geometric.testing import is_full_test
from torch_geometric.utils import to_dense_adj


@pytest.mark.parametrize('aggr', ['add', 'mean', 'max'])
def test_dense_graph_conv(aggr):
    channels = 16
    sparse_conv = GraphConv(channels, channels, aggr=aggr)
    dense_conv = DenseGraphConv(channels, channels, aggr=aggr)
    assert str(dense_conv) == 'DenseGraphConv(16, 16)'

    # Ensure same weights and bias.
    dense_conv.lin_rel = sparse_conv.lin_rel
    dense_conv.lin_root = sparse_conv.lin_root

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 4],
                               [1, 2, 0, 2, 0, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)
    assert sparse_out.size() == (5, channels)

    adj = to_dense_adj(edge_index)
    mask = torch.ones(5, dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)[0]

    assert dense_out.size() == (5, channels)
    assert torch.allclose(sparse_out, dense_out, atol=1e-04)

    if is_full_test():
        jit = torch.jit.script(dense_conv)
        assert torch.allclose(jit(x, adj, mask), dense_out)


@pytest.mark.parametrize('aggr', ['add', 'mean', 'max'])
def test_dense_graph_conv_batch(aggr):
    channels = 16
    sparse_conv = GraphConv(channels, channels, aggr=aggr)
    dense_conv = DenseGraphConv(channels, channels, aggr=aggr)

    # Ensure same weights and bias.
    dense_conv.lin_rel = sparse_conv.lin_rel
    dense_conv.lin_root = sparse_conv.lin_root

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 4],
                               [1, 2, 0, 2, 0, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)
    assert sparse_out.size() == (5, channels)

    x = torch.cat([x, x.new_zeros(1, channels)], dim=0).view(2, 3, channels)
    adj = torch.Tensor([
        [
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 0],
        ],
        [
            [0, 1, 0],
            [1, 0, 0],
            [0, 0, 0],
        ],
    ])
    mask = torch.tensor([[1, 1, 1], [1, 1, 0]], dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)
    assert dense_out.size() == (2, 3, channels)
    dense_out = dense_out.view(-1, channels)

    assert torch.allclose(sparse_out, dense_out[:5], atol=1e-04)
    assert dense_out[-1].abs().sum() == 0


@pytest.mark.parametrize('aggr', ['add', 'mean', 'max'])
def test_dense_graph_conv_with_broadcasting(aggr):
    batch_size, num_nodes, channels = 8, 3, 16
    conv = DenseGraphConv(channels, channels, aggr=aggr)

    x = torch.randn(batch_size, num_nodes, channels)
    adj = torch.Tensor([
        [0, 1, 1],
        [1, 0, 1],
        [1, 1, 0],
    ])

    assert conv(x, adj).size() == (batch_size, num_nodes, channels)
    mask = torch.tensor([1, 1, 1], dtype=torch.bool)
    assert conv(x, adj, mask).size() == (batch_size, num_nodes, channels)
pytorch/test/nn/dense/test_dense_sage_conv.py000066400000000000000000000036121444262304300217750ustar00rootroot00000000000000import torch

from torch_geometric.nn import DenseSAGEConv, SAGEConv
from torch_geometric.testing import is_full_test


def test_dense_sage_conv():
    channels = 16
    sparse_conv = SAGEConv(channels, channels, normalize=True)
    dense_conv = DenseSAGEConv(channels, channels, normalize=True)
    assert str(dense_conv) == 'DenseSAGEConv(16, 16)'

    # Ensure same weights and bias.
    dense_conv.lin_rel = sparse_conv.lin_l
    dense_conv.lin_root = sparse_conv.lin_r

    x = torch.randn((5, channels))
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 4],
                               [1, 2, 0, 2, 0, 1, 4, 3]])

    sparse_out = sparse_conv(x, edge_index)
    assert sparse_out.size() == (5, channels)

    x = torch.cat([x, x.new_zeros(1, channels)], dim=0).view(2, 3, channels)
    adj = torch.Tensor([
        [
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 0],
        ],
        [
            [0, 1, 0],
            [1, 0, 0],
            [0, 0, 0],
        ],
    ])
    mask = torch.tensor([[1, 1, 1], [1, 1, 0]], dtype=torch.bool)

    dense_out = dense_conv(x, adj, mask)
    assert dense_out.size() == (2, 3, channels)

    if is_full_test():
        jit = torch.jit.script(dense_conv)
        assert torch.allclose(jit(x, adj, mask), dense_out)

    assert dense_out[1, 2].abs().sum().item() == 0
    dense_out = dense_out.view(6, channels)[:-1]
    assert torch.allclose(sparse_out, dense_out, atol=1e-04)


def test_dense_sage_conv_with_broadcasting():
    batch_size, num_nodes, channels = 8, 3, 16
    conv = DenseSAGEConv(channels, channels)

    x = torch.randn(batch_size, num_nodes, channels)
    adj = torch.Tensor([
        [0, 1, 1],
        [1, 0, 1],
        [1, 1, 0],
    ])

    assert conv(x, adj).size() == (batch_size, num_nodes, channels)
    mask = torch.tensor([1, 1, 1], dtype=torch.bool)
    assert conv(x, adj, mask).size() == (batch_size, num_nodes, channels)
pytorch/test/nn/dense/test_diff_pool.py000066400000000000000000000041301444262304300206100ustar00rootroot00000000000000from itertools import product

import torch

from torch_geometric.nn import dense_diff_pool
from torch_geometric.profile import benchmark
from torch_geometric.testing import is_full_test


def test_dense_diff_pool():
    batch_size, num_nodes, channels, num_clusters = (2, 20, 16, 10)
    x = torch.randn((batch_size, num_nodes, channels))
    adj = torch.rand((batch_size, num_nodes, num_nodes))
    s = torch.randn((batch_size, num_nodes, num_clusters))
    mask = torch.randint(0, 2, (batch_size, num_nodes), dtype=torch.bool)

    x_out, adj_out, link_loss, ent_loss = dense_diff_pool(x, adj, s, mask)
    assert x_out.size() == (2, 10, 16)
    assert adj_out.size() == (2, 10, 10)
    assert link_loss.item() >= 0
    assert ent_loss.item() >= 0

    if is_full_test():
        jit = torch.jit.script(dense_diff_pool)
        x_jit, adj_jit, link_loss, ent_loss = jit(x, adj, s, mask)
        assert torch.allclose(x_jit, x_out)
        assert torch.allclose(adj_jit, adj_out)
        assert link_loss.item() >= 0
        assert ent_loss.item() >= 0


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    BS = [2**i for i in range(4, 8)]
    NS = [2**i for i in range(4, 8)]
    FS = [2**i for i in range(5, 9)]
    CS = [2**i for i in range(5, 9)]

    funcs = []
    func_names = []
    args_list = []
    for B, N, F, C in product(BS, NS, FS, CS):
        x = torch.randn(B, N, F, device=args.device)
        adj = torch.randint(0, 2, (B, N, N), dtype=x.dtype, device=args.device)
        s = torch.randn(B, N, C, device=args.device)

        funcs.append(dense_diff_pool)
        func_names.append(f'B={B}, N={N}, F={F}, C={C}')
        args_list.append((x, adj, s))

    benchmark(
        funcs=funcs,
        func_names=func_names,
        args=args_list,
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
        backward=args.backward,
        progress_bar=True,
    )
pytorch/test/nn/dense/test_dmon_pool.py000066400000000000000000000013501444262304300206360ustar00rootroot00000000000000import torch

from torch_geometric.nn import DMoNPooling


def test_dmon_pooling():
    batch_size, num_nodes, channels, num_clusters = (2, 20, 16, 10)
    x = torch.randn((batch_size, num_nodes, channels))
    adj = torch.ones((batch_size, num_nodes, num_nodes))
    mask = torch.randint(0, 2, (batch_size, num_nodes), dtype=torch.bool)

    pool = DMoNPooling([channels, channels], num_clusters)
    assert str(pool) == 'DMoNPooling(16, num_clusters=10)'

    s, x, adj, spectral_loss, ortho_loss, cluster_loss = pool(x, adj, mask)
    assert s.size() == (2, 20, 10)
    assert x.size() == (2, 10, 16)
    assert adj.size() == (2, 10, 10)
    assert -1 <= spectral_loss <= 0
    assert 0 <= ortho_loss <= 2
    assert -1 <= cluster_loss <= 0
pytorch/test/nn/dense/test_linear.py000066400000000000000000000236341444262304300201330ustar00rootroot00000000000000import copy
import warnings
from typing import List

import pytest
import torch
from torch import Tensor
from torch.nn import Linear as PTLinear
from torch.nn.parameter import UninitializedParameter

from torch_geometric.nn import HeteroDictLinear, HeteroLinear, Linear
from torch_geometric.profile import benchmark
from torch_geometric.testing import withCUDA, withPackage
from torch_geometric.typing import pyg_lib

weight_inits = ['glorot', 'kaiming_uniform', None]
bias_inits = ['zeros', None]


@withCUDA
@pytest.mark.parametrize('weight', weight_inits)
@pytest.mark.parametrize('bias', bias_inits)
def test_linear(weight, bias, device):
    x = torch.randn(3, 4, 16, device=device)
    lin = Linear(16, 32, weight_initializer=weight, bias_initializer=bias)
    lin = lin.to(device)
    assert str(lin) == 'Linear(16, 32, bias=True)'
    assert lin(x).size() == (3, 4, 32)


@withCUDA
@pytest.mark.parametrize('weight', weight_inits)
@pytest.mark.parametrize('bias', bias_inits)
def test_lazy_linear(weight, bias, device):
    x = torch.randn(3, 4, 16, device=device)
    lin = Linear(-1, 32, weight_initializer=weight, bias_initializer=bias)
    lin = lin.to(device)
    assert str(lin) == 'Linear(-1, 32, bias=True)'
    assert lin(x).size() == (3, 4, 32)
    assert str(lin) == 'Linear(16, 32, bias=True)'


@withCUDA
@pytest.mark.parametrize('dim1', [-1, 16])
@pytest.mark.parametrize('dim2', [-1, 16])
def test_load_lazy_linear(dim1, dim2, device):
    lin1 = Linear(dim1, 32).to(device)
    lin2 = Linear(dim1, 32).to(device)
    lin2.load_state_dict(lin1.state_dict())

    if dim1 != -1:
        assert torch.allclose(lin1.weight, lin2.weight)
        assert torch.allclose(lin1.bias, lin2.bias)
        assert not hasattr(lin1, '_hook')
        assert not hasattr(lin2, '_hook')
    else:
        assert isinstance(lin1.weight, UninitializedParameter)
        assert isinstance(lin2.weight, UninitializedParameter)
        assert hasattr(lin1, '_hook')
        assert hasattr(lin2, '_hook')

    with pytest.raises(RuntimeError, match="in state_dict"):
        lin1.load_state_dict({}, strict=True)
    lin1.load_state_dict({}, strict=False)


@pytest.mark.parametrize('lazy', [True, False])
def test_identical_linear_default_initialization(lazy):
    x = torch.randn(3, 4, 16)

    torch.manual_seed(12345)
    lin1 = Linear(-1 if lazy else 16, 32)
    lin1(x)

    torch.manual_seed(12345)
    lin2 = PTLinear(16, 32)

    assert torch.equal(lin1.weight, lin2.weight)
    assert torch.equal(lin1.bias, lin2.bias)
    assert torch.allclose(lin1(x), lin2(x))


@withPackage('torch<=1.12')
def test_copy_unintialized_parameter():
    weight = UninitializedParameter()
    with pytest.raises(Exception):
        copy.deepcopy(weight)


@withCUDA
@pytest.mark.parametrize('lazy', [True, False])
def test_copy_linear(lazy, device):
    lin = Linear(-1 if lazy else 16, 32).to(device)

    copied_lin = copy.copy(lin).to(device)
    assert id(copied_lin) != id(lin)
    assert id(copied_lin.weight) == id(lin.weight)
    if not isinstance(copied_lin.weight, UninitializedParameter):
        assert copied_lin.weight.data_ptr() == lin.weight.data_ptr()
    assert id(copied_lin.bias) == id(lin.bias)
    assert copied_lin.bias.data_ptr() == lin.bias.data_ptr()

    copied_lin = copy.deepcopy(lin).to(device)
    assert id(copied_lin) != id(lin)
    assert id(copied_lin.weight) != id(lin.weight)
    if not isinstance(copied_lin.weight, UninitializedParameter):
        assert copied_lin.weight.data_ptr() != lin.weight.data_ptr()
        assert torch.allclose(copied_lin.weight, lin.weight)
    assert id(copied_lin.bias) != id(lin.bias)
    assert copied_lin.bias.data_ptr() != lin.bias.data_ptr()
    if int(torch.isnan(lin.bias).sum()) == 0:
        assert torch.allclose(copied_lin.bias, lin.bias)


@withCUDA
def test_hetero_linear(device):
    x = torch.randn(3, 16, device=device)
    type_vec = torch.tensor([0, 1, 2], device=device)

    lin = HeteroLinear(16, 32, num_types=3).to(device)
    assert str(lin) == 'HeteroLinear(16, 32, num_types=3, bias=True)'

    out = lin(x, type_vec)
    assert out.size() == (3, 32)

    jit = torch.jit.script(lin)
    assert torch.allclose(jit(x, type_vec), out, atol=1e-3)


@withCUDA
@pytest.mark.parametrize('use_segmm', [True, False])
def test_hetero_linear_amp(device, use_segmm):
    warnings.filterwarnings('ignore', '.*but CUDA is not available.*')

    x = torch.randn(3, 16, device=device)
    type_vec = torch.tensor([0, 1, 2], device=device)

    lin = HeteroLinear(16, 32, num_types=3).to(device)
    lin.use_segmm = use_segmm

    with torch.cuda.amp.autocast():
        assert lin(x, type_vec).size() == (3, 32)


@withCUDA
def test_lazy_hetero_linear(device):
    x = torch.randn(3, 16, device=device)
    type_vec = torch.tensor([0, 1, 2], device=device)

    lin = HeteroLinear(-1, 32, num_types=3).to(device)
    assert str(lin) == 'HeteroLinear(-1, 32, num_types=3, bias=True)'

    out = lin(x, type_vec)
    assert out.size() == (3, 32)


@withCUDA
@pytest.mark.parametrize('bias', [True, False])
def test_hetero_dict_linear(bias, device):
    x_dict = {
        'v': torch.randn(3, 16, device=device),
        'w': torch.randn(2, 8, device=device),
    }

    lin = HeteroDictLinear({'v': 16, 'w': 8}, 32, bias=bias).to(device)
    assert str(lin) == (f"HeteroDictLinear({{'v': 16, 'w': 8}}, 32, "
                        f"bias={bias})")

    out_dict = lin(x_dict)
    assert len(out_dict) == 2
    assert out_dict['v'].size() == (3, 32)
    assert out_dict['w'].size() == (2, 32)

    x_dict = {
        'v': torch.randn(3, 16, device=device),
        'w': torch.randn(2, 16, device=device),
    }

    lin = HeteroDictLinear(16, 32, types=['v', 'w'], bias=bias).to(device)
    assert str(lin) == (f"HeteroDictLinear({{'v': 16, 'w': 16}}, 32, "
                        f"bias={bias})")

    out_dict = lin(x_dict)
    assert len(out_dict) == 2
    assert out_dict['v'].size() == (3, 32)
    assert out_dict['w'].size() == (2, 32)


def test_hetero_dict_linear_jit():
    x_dict = {
        'v': torch.randn(3, 16),
        'w': torch.randn(2, 8),
    }

    lin = HeteroDictLinear({'v': 16, 'w': 8}, 32)

    jit = torch.jit.script(lin)
    assert len(jit(x_dict)) == 2


@withCUDA
def test_lazy_hetero_dict_linear(device):
    x_dict = {
        'v': torch.randn(3, 16, device=device),
        'w': torch.randn(2, 8, device=device),
    }

    lin = HeteroDictLinear(-1, 32, types=['v', 'w']).to(device)
    assert str(lin) == "HeteroDictLinear({'v': -1, 'w': -1}, 32, bias=True)"

    out_dict = lin(x_dict)
    assert len(out_dict) == 2
    assert out_dict['v'].size() == (3, 32)
    assert out_dict['w'].size() == (2, 32)


@withCUDA
@withPackage('pyg_lib')
@pytest.mark.parametrize('type_vec', [
    torch.tensor([0, 0, 1, 1, 2, 2]),
    torch.tensor([0, 1, 2, 0, 1, 2]),
])
def test_hetero_linear_sort(type_vec, device):
    x = torch.randn(type_vec.numel(), 16, device=device)

    lin = HeteroLinear(16, 32, num_types=3).to(device)
    out = lin(x, type_vec)

    for i in range(type_vec.numel()):
        node_type = int(type_vec[i])
        expected = x[i] @ lin.weight[node_type] + lin.bias[node_type]
        assert torch.allclose(out[i], expected, atol=1e-3)


if __name__ == '__main__':
    import argparse
    try:
        import dgl
        WITH_DLG = True
    except:  # noqa
        WITH_DGL = False

    warnings.filterwarnings('ignore', '.*API of nested tensors.*')
    warnings.filterwarnings('ignore', '.*TypedStorage is deprecated.*')

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    torch.manual_seed(12345)

    def get_xs(mean: float, std: float, num_types: int,
               channels: int) -> List[Tensor]:
        num_nodes_list = torch.normal(
            mean=torch.tensor([mean] * num_types, dtype=torch.float),
            std=torch.tensor([std] * num_types, dtype=torch.float),
        ).round().to(torch.long).tolist()

        return [
            torch.randn(num_nodes, channels, device=args.device)
            for num_nodes in num_nodes_list
        ]

    def sequential(xs: List[Tensor], weights: List[Tensor]) -> List[Tensor]:
        return [x @ weight for x, weight in zip(xs, weights)]

    def nested(xs: List[Tensor], weights: List[Tensor]) -> List[Tensor]:
        x = torch.nested.nested_tensor(xs)
        weight = torch.nested.nested_tensor(weights)
        return list(torch.matmul(x, weight).unbind(0))

    def grouped(x: Tensor, ptr: Tensor, weight: Tensor) -> Tensor:
        return pyg_lib.ops.segment_matmul(x, ptr, weight)

    def padded(x: Tensor, weight: Tensor) -> Tensor:
        return torch.matmul(x, weight)

    def dgl_mm(x: Tensor, count: Tensor, weight: Tensor) -> Tensor:
        return dgl.ops.segment_mm(x, weight, count)

    num_nodes, channels = 1_000_000, 64

    for num_types in [3, 5, 10, 50, 100, 200, 500, 1000]:
        print(f'Number of types: {num_types}')
        mean = num_nodes // num_types
        std = mean // 4

        xs = get_xs(mean, std, num_types, channels)
        count = torch.tensor([x.size(0) for x in xs])
        ptr = torch.tensor([0] + [x.size(0) for x in xs]).cumsum(0)
        x = torch.cat(xs, dim=0)
        padded_x = torch.nested.nested_tensor(xs).to_padded_tensor(padding=0.0)
        weight = torch.randn(num_types, channels, channels, device=args.device)
        weights = list(weight.unbind(0))

        funcs = [sequential, grouped, padded]
        func_names = ['Sequential', 'Grouped', 'Padded']
        args_list = [(xs, weights), (x, ptr, weight), (padded_x, weight)]

        if WITH_DGL:
            funcs.append(dgl_mm)
            func_names.append('DGL')
            args_list.append((x, count, weight))

        benchmark(
            funcs=funcs,
            func_names=func_names,
            args=args_list,
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )
pytorch/test/nn/dense/test_mincut_pool.py000066400000000000000000000017521444262304300212060ustar00rootroot00000000000000import torch

from torch_geometric.nn import dense_mincut_pool
from torch_geometric.testing import is_full_test


def test_dense_mincut_pool():
    batch_size, num_nodes, channels, num_clusters = (2, 20, 16, 10)
    x = torch.randn((batch_size, num_nodes, channels))
    adj = torch.ones((batch_size, num_nodes, num_nodes))
    s = torch.randn((batch_size, num_nodes, num_clusters))
    mask = torch.randint(0, 2, (batch_size, num_nodes), dtype=torch.bool)

    x_out, adj_out, mincut_loss, ortho_loss = dense_mincut_pool(
        x, adj, s, mask)
    assert x_out.size() == (2, 10, 16)
    assert adj_out.size() == (2, 10, 10)
    assert -1 <= mincut_loss <= 0
    assert 0 <= ortho_loss <= 2

    if is_full_test():
        jit = torch.jit.script(dense_mincut_pool)

        x_jit, adj_jit, mincut_loss, ortho_loss = jit(x, adj, s, mask)
        assert x_jit.size() == (2, 10, 16)
        assert adj_jit.size() == (2, 10, 10)
        assert -1 <= mincut_loss <= 0
        assert 0 <= ortho_loss <= 2
pytorch/test/nn/functional/000077500000000000000000000000001444262304300163045ustar00rootroot00000000000000pytorch/test/nn/functional/test_bro.py000066400000000000000000000012471444262304300205030ustar00rootroot00000000000000import torch

from torch_geometric.nn.functional import bro


def test_bro():
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 2, 2])

    g1 = torch.tensor([
        [0.2, 0.2, 0.2, 0.2],
        [0.0, 0.2, 0.2, 0.2],
        [0.2, 0.0, 0.2, 0.2],
        [0.2, 0.2, 0.0, 0.2],
    ])

    g2 = torch.tensor([
        [0.2, 0.2, 0.2, 0.2],
        [0.0, 0.2, 0.2, 0.2],
        [0.2, 0.0, 0.2, 0.2],
    ])

    g3 = torch.tensor([
        [0.2, 0.2, 0.2, 0.2],
        [0.2, 0.0, 0.2, 0.2],
    ])

    s = 0.
    for g in [g1, g2, g3]:
        s += torch.norm(g @ g.t() - torch.eye(g.shape[0]), p=2)

    assert torch.isclose(s / 3., bro(torch.cat([g1, g2, g3], dim=0), batch))
pytorch/test/nn/functional/test_gini.py000066400000000000000000000003041444262304300206400ustar00rootroot00000000000000import torch

from torch_geometric.nn.functional import gini


def test_gini():
    w = torch.tensor([[0., 0., 0., 0.], [0., 0., 0., 1000.0]])
    assert torch.isclose(gini(w), torch.tensor(0.5))
pytorch/test/nn/kge/000077500000000000000000000000001444262304300147105ustar00rootroot00000000000000pytorch/test/nn/kge/test_complex.py000066400000000000000000000026361444262304300177770ustar00rootroot00000000000000import torch

from torch_geometric.nn import ComplEx


def test_complex_scoring():
    model = ComplEx(num_nodes=5, num_relations=2, hidden_channels=1)

    model.node_emb.weight.data = torch.tensor([
        [2.],
        [3.],
        [5.],
        [1.],
        [2.],
    ])
    model.node_emb_im.weight.data = torch.tensor([
        [4.],
        [1.],
        [3.],
        [1.],
        [2.],
    ])
    model.rel_emb.weight.data = torch.tensor([
        [2.],
        [3.],
    ])
    model.rel_emb_im.weight.data = torch.tensor([
        [3.],
        [1.],
    ])

    score = model(
        head_index=torch.tensor([1, 3]),
        rel_type=torch.tensor([1, 0]),
        tail_index=torch.tensor([2, 4]),
    )
    assert score.tolist() == [58., 8.]


def test_complex():
    model = ComplEx(num_nodes=10, num_relations=5, hidden_channels=32)
    assert str(model) == 'ComplEx(10, num_relations=5, hidden_channels=32)'

    head_index = torch.tensor([0, 2, 4, 6, 8])
    rel_type = torch.tensor([0, 1, 2, 3, 4])
    tail_index = torch.tensor([1, 3, 5, 7, 9])

    loader = model.loader(head_index, rel_type, tail_index, batch_size=5)
    for h, r, t in loader:
        out = model(h, r, t)
        assert out.size() == (5, )

        loss = model.loss(h, r, t)
        assert loss >= 0.

        mean_rank, hits_at_10 = model.test(h, r, t, batch_size=5, log=False)
        assert mean_rank <= 10
        assert hits_at_10 == 1.0
pytorch/test/nn/kge/test_distmult.py000066400000000000000000000013401444262304300201640ustar00rootroot00000000000000import torch

from torch_geometric.nn import DistMult


def test_distmult():
    model = DistMult(num_nodes=10, num_relations=5, hidden_channels=32)
    assert str(model) == 'DistMult(10, num_relations=5, hidden_channels=32)'

    head_index = torch.tensor([0, 2, 4, 6, 8])
    rel_type = torch.tensor([0, 1, 2, 3, 4])
    tail_index = torch.tensor([1, 3, 5, 7, 9])

    loader = model.loader(head_index, rel_type, tail_index, batch_size=5)
    for h, r, t in loader:
        out = model(h, r, t)
        assert out.size() == (5, )

        loss = model.loss(h, r, t)
        assert loss >= 0.

        mean_rank, hits_at_10 = model.test(h, r, t, batch_size=5, log=False)
        assert mean_rank <= 10
        assert hits_at_10 == 1.0
pytorch/test/nn/kge/test_rotate.py000066400000000000000000000013301444262304300176140ustar00rootroot00000000000000import torch

from torch_geometric.nn import RotatE


def test_rotate():
    model = RotatE(num_nodes=10, num_relations=5, hidden_channels=32)
    assert str(model) == 'RotatE(10, num_relations=5, hidden_channels=32)'

    head_index = torch.tensor([0, 2, 4, 6, 8])
    rel_type = torch.tensor([0, 1, 2, 3, 4])
    tail_index = torch.tensor([1, 3, 5, 7, 9])

    loader = model.loader(head_index, rel_type, tail_index, batch_size=5)
    for h, r, t in loader:
        out = model(h, r, t)
        assert out.size() == (5, )

        loss = model.loss(h, r, t)
        assert loss >= 0.

        mean_rank, hits_at_10 = model.test(h, r, t, batch_size=5, log=False)
        assert mean_rank <= 10
        assert hits_at_10 == 1.0
pytorch/test/nn/kge/test_transe.py000066400000000000000000000013301444262304300176120ustar00rootroot00000000000000import torch

from torch_geometric.nn import TransE


def test_transe():
    model = TransE(num_nodes=10, num_relations=5, hidden_channels=32)
    assert str(model) == 'TransE(10, num_relations=5, hidden_channels=32)'

    head_index = torch.tensor([0, 2, 4, 6, 8])
    rel_type = torch.tensor([0, 1, 2, 3, 4])
    tail_index = torch.tensor([1, 3, 5, 7, 9])

    loader = model.loader(head_index, rel_type, tail_index, batch_size=5)
    for h, r, t in loader:
        out = model(h, r, t)
        assert out.size() == (5, )

        loss = model.loss(h, r, t)
        assert loss >= 0.

        mean_rank, hits_at_10 = model.test(h, r, t, batch_size=5, log=False)
        assert mean_rank <= 10
        assert hits_at_10 == 1.0
pytorch/test/nn/models/000077500000000000000000000000001444262304300154255ustar00rootroot00000000000000pytorch/test/nn/models/test_attentive_fp.py000066400000000000000000000014671444262304300215360ustar00rootroot00000000000000import torch

from torch_geometric.nn import AttentiveFP
from torch_geometric.testing import is_full_test


def test_attentive_fp():
    model = AttentiveFP(8, 16, 32, edge_dim=3, num_layers=2, num_timesteps=2)
    assert str(model) == ('AttentiveFP(in_channels=8, hidden_channels=16, '
                          'out_channels=32, edge_dim=3, num_layers=2, '
                          'num_timesteps=2)')

    x = torch.randn(4, 8)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.randn(edge_index.size(1), 3)
    batch = torch.tensor([0, 0, 0, 0])

    out = model(x, edge_index, edge_attr, batch)
    assert out.size() == (1, 32)

    if is_full_test():
        jit = torch.jit.script(model.jittable())
        assert torch.allclose(jit(x, edge_index, edge_attr, batch), out)
pytorch/test/nn/models/test_autoencoder.py000066400000000000000000000060341444262304300213510ustar00rootroot00000000000000import torch
from torch import Tensor as T

from torch_geometric.data import Data
from torch_geometric.nn import ARGA, ARGVA, GAE, VGAE
from torch_geometric.testing import is_full_test
from torch_geometric.transforms import RandomLinkSplit


def test_gae():
    model = GAE(encoder=lambda x: x)
    model.reset_parameters()

    x = torch.Tensor([[1, -1], [1, 2], [2, 1]])
    z = model.encode(x)
    assert torch.allclose(z, x)

    adj = model.decoder.forward_all(z)
    assert torch.allclose(
        adj,
        torch.Tensor([[+2, -1, +1], [-1, +5, +4], [+1, +4, +5]]).sigmoid())

    edge_index = torch.tensor([[0, 1], [1, 2]])
    value = model.decode(z, edge_index)
    assert torch.allclose(value, torch.sigmoid(torch.Tensor([-1, 4])))

    if is_full_test():
        jit = torch.jit.export(model)
        assert torch.allclose(jit.encode(x), z)
        assert torch.allclose(jit.decode(z, edge_index), value)

    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
                               [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]])
    data = Data(edge_index=edge_index, num_nodes=11)
    transform = RandomLinkSplit(split_labels=True,
                                add_negative_train_samples=False)
    train_data, val_data, test_data = transform(data)

    z = torch.randn(11, 16)
    loss = model.recon_loss(z, train_data.pos_edge_label_index)
    assert float(loss) > 0

    auc, ap = model.test(z, val_data.pos_edge_label_index,
                         val_data.neg_edge_label_index)
    assert auc >= 0 and auc <= 1 and ap >= 0 and ap <= 1


def test_vgae():
    model = VGAE(encoder=lambda x: (x, x))

    x = torch.Tensor([[1, -1], [1, 2], [2, 1]])
    model.encode(x)
    assert float(model.kl_loss()) > 0

    model.eval()
    model.encode(x)

    if is_full_test():
        jit = torch.jit.export(model)
        jit.encode(x)
        assert float(jit.kl_loss()) > 0


def test_arga():
    model = ARGA(encoder=lambda x: x, discriminator=lambda x: T([0.5]))
    model.reset_parameters()

    x = torch.Tensor([[1, -1], [1, 2], [2, 1]])
    z = model.encode(x)

    assert float(model.reg_loss(z)) > 0
    assert float(model.discriminator_loss(z)) > 0

    if is_full_test():
        jit = torch.jit.export(model)
        assert torch.allclose(jit.encode(x), z)
        assert float(jit.reg_loss(z)) > 0
        assert float(jit.discriminator_loss(z)) > 0


def test_argva():
    model = ARGVA(encoder=lambda x: (x, x), discriminator=lambda x: T([0.5]))

    x = torch.Tensor([[1, -1], [1, 2], [2, 1]])
    model.encode(x)
    model.reparametrize(model.__mu__, model.__logstd__)
    assert float(model.kl_loss()) > 0

    if is_full_test():
        jit = torch.jit.export(model)
        jit.encode(x)
        jit.reparametrize(jit.__mu__, jit.__logstd__)
        assert float(jit.kl_loss()) > 0


def test_init():
    encoder = torch.nn.Linear(16, 32)
    decoder = torch.nn.Linear(32, 16)
    discriminator = torch.nn.Linear(32, 1)

    GAE(encoder, decoder)
    VGAE(encoder, decoder)
    ARGA(encoder, discriminator, decoder)
    ARGVA(encoder, discriminator, decoder)
pytorch/test/nn/models/test_basic_gnn.py000066400000000000000000000275721444262304300207760ustar00rootroot00000000000000import os
import os.path as osp
import warnings

import pytest
import torch
import torch.nn.functional as F

import torch_geometric.typing
from torch_geometric.data import Data
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import SAGEConv
from torch_geometric.nn.models import GAT, GCN, GIN, PNA, EdgeCNN, GraphSAGE
from torch_geometric.profile import benchmark
from torch_geometric.testing import (
    disableExtensions,
    onlyFullTest,
    onlyLinux,
    onlyNeighborSampler,
    withCUDA,
    withPackage,
)

out_dims = [None, 8]
dropouts = [0.0, 0.5]
acts = [None, 'leaky_relu', torch.relu_, F.elu, torch.nn.ReLU()]
norms = [None, 'batch_norm', 'layer_norm']
jks = [None, 'last', 'cat', 'max', 'lstm']


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_gcn(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    model = GCN(8, 16, num_layers=3, out_channels=out_dim, dropout=dropout,
                act=act, norm=norm, jk=jk)
    assert str(model) == f'GCN(8, {out_channels}, num_layers=3)'
    assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_graph_sage(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    model = GraphSAGE(8, 16, num_layers=3, out_channels=out_dim,
                      dropout=dropout, act=act, norm=norm, jk=jk)
    assert str(model) == f'GraphSAGE(8, {out_channels}, num_layers=3)'
    assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_gin(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    model = GIN(8, 16, num_layers=3, out_channels=out_dim, dropout=dropout,
                act=act, norm=norm, jk=jk)
    assert str(model) == f'GIN(8, {out_channels}, num_layers=3)'
    assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_gat(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    for v2 in [False, True]:
        model = GAT(8, 16, num_layers=3, out_channels=out_dim, v2=v2,
                    dropout=dropout, act=act, norm=norm, jk=jk)
        assert str(model) == f'GAT(8, {out_channels}, num_layers=3)'
        assert model(x, edge_index).size() == (3, out_channels)

        model = GAT(8, 16, num_layers=3, out_channels=out_dim, v2=v2,
                    dropout=dropout, act=act, norm=norm, jk=jk, heads=4)
        assert str(model) == f'GAT(8, {out_channels}, num_layers=3)'
        assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_pna(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    deg = torch.tensor([0, 2, 1])
    out_channels = 16 if out_dim is None else out_dim
    aggregators = ['mean', 'min', 'max', 'std', 'var', 'sum']
    scalers = [
        'identity', 'amplification', 'attenuation', 'linear', 'inverse_linear'
    ]

    model = PNA(8, 16, num_layers=3, out_channels=out_dim, dropout=dropout,
                act=act, norm=norm, jk=jk, aggregators=aggregators,
                scalers=scalers, deg=deg)
    assert str(model) == f'PNA(8, {out_channels}, num_layers=3)'
    assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('dropout', dropouts)
@pytest.mark.parametrize('act', acts)
@pytest.mark.parametrize('norm', norms)
@pytest.mark.parametrize('jk', jks)
def test_edge_cnn(out_dim, dropout, act, norm, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    model = EdgeCNN(8, 16, num_layers=3, out_channels=out_dim, dropout=dropout,
                    act=act, norm=norm, jk=jk)
    assert str(model) == f'EdgeCNN(8, {out_channels}, num_layers=3)'
    assert model(x, edge_index).size() == (3, out_channels)


@pytest.mark.parametrize('out_dim', out_dims)
@pytest.mark.parametrize('jk', jks)
def test_one_layer_gnn(out_dim, jk):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    out_channels = 16 if out_dim is None else out_dim

    model = GraphSAGE(8, 16, num_layers=1, out_channels=out_dim, jk=jk)
    assert model(x, edge_index).size() == (3, out_channels)


@onlyNeighborSampler
@pytest.mark.parametrize('jk', [None, 'last'])
def test_basic_gnn_inference(get_dataset, jk):
    dataset = get_dataset(name='Cora')
    data = dataset[0]

    model = GraphSAGE(dataset.num_features, hidden_channels=16, num_layers=2,
                      out_channels=dataset.num_classes, jk=jk)
    model.eval()

    out1 = model(data.x, data.edge_index)
    assert out1.size() == (data.num_nodes, dataset.num_classes)

    loader = NeighborLoader(data, num_neighbors=[-1], batch_size=128)
    out2 = model.inference(loader)
    assert out1.size() == out2.size()
    assert torch.allclose(out1, out2, atol=1e-4)

    assert 'n_id' not in data


@withCUDA
@onlyLinux
@onlyFullTest
@disableExtensions
@withPackage('torch>=2.0.0')
def test_compile(device):
    x = torch.randn(3, 8, device=device)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], device=device)

    model = GCN(8, 16, num_layers=3).to(device)
    compiled_model = torch_geometric.compile(model)

    expected = model(x, edge_index)
    out = compiled_model(x, edge_index)
    assert torch.allclose(out, expected, atol=1e-6)


def test_packaging():
    warnings.filterwarnings('ignore', '.*TypedStorage is deprecated.*')

    os.makedirs(torch.hub._get_torch_home(), exist_ok=True)

    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    model = GraphSAGE(8, 16, num_layers=3)
    path = osp.join(torch.hub._get_torch_home(), 'pyg_test_model.pt')
    torch.save(model, path)

    model = torch.load(path)
    with torch.no_grad():
        assert model(x, edge_index).size() == (3, 16)

    model = GraphSAGE(8, 16, num_layers=3)
    path = osp.join(torch.hub._get_torch_home(), 'pyg_test_package.pt')
    with torch.package.PackageExporter(path) as pe:
        pe.extern('torch_geometric.nn.**')
        pe.extern('torch_geometric.utils.trim_to_layer')
        pe.extern('_operator')
        pe.save_pickle('models', 'model.pkl', model)

    pi = torch.package.PackageImporter(path)
    model = pi.load_pickle('models', 'model.pkl')
    with torch.no_grad():
        assert model(x, edge_index).size() == (3, 16)


@withPackage('onnx', 'onnxruntime')
def test_onnx(tmp_path, capfd):
    import onnx
    import onnxruntime as ort

    warnings.filterwarnings('ignore', '.*tensor to a Python boolean.*')
    warnings.filterwarnings('ignore', '.*shape inference of prim::Constant.*')

    class MyModel(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = SAGEConv(8, 16)
            self.conv2 = SAGEConv(16, 16)

        def forward(self, x, edge_index):
            x = self.conv1(x, edge_index).relu()
            x = self.conv2(x, edge_index)
            return x

    model = MyModel()
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 2], [1, 0, 2]])
    expected = model(x, edge_index)
    assert expected.size() == (3, 16)

    path = osp.join(tmp_path, 'model.onnx')
    torch.onnx.export(model, (x, edge_index), path,
                      input_names=('x', 'edge_index'), opset_version=16)
    if torch_geometric.typing.WITH_PT2:
        out, _ = capfd.readouterr()
        assert '0 NONE 0 NOTE 0 WARNING 0 ERROR' in out

    model = onnx.load(path)
    onnx.checker.check_model(model)

    providers = ['CPUExecutionProvider']
    ort_session = ort.InferenceSession(path, providers=providers)

    out = ort_session.run(None, {
        'x': x.numpy(),
        'edge_index': edge_index.numpy()
    })[0]
    out = torch.from_numpy(out)
    assert torch.allclose(out, expected, atol=1e-6)


@withPackage('pyg_lib')
def test_trim_to_layer():
    x = torch.randn(14, 16)
    edge_index = torch.tensor([
        [2, 3, 4, 5, 7, 7, 10, 11, 12, 13],
        [0, 1, 2, 3, 2, 3, 7, 7, 7, 7],
    ])
    data = Data(x=x, edge_index=edge_index)

    loader = NeighborLoader(
        data,
        num_neighbors=[1, 2, 4],
        batch_size=2,
        shuffle=False,
    )
    batch = next(iter(loader))

    model = GraphSAGE(in_channels=16, hidden_channels=16, num_layers=3)
    out1 = model(batch.x, batch.edge_index)[:2]
    assert out1.size() == (2, 16)

    out2 = model(
        batch.x,
        batch.edge_index,
        num_sampled_nodes_per_hop=batch.num_sampled_nodes,
        num_sampled_edges_per_hop=batch.num_sampled_edges,
    )[:2]
    assert out2.size() == (2, 16)

    assert torch.allclose(out1, out2)


num_compile_calls = 0


@onlyLinux
@disableExtensions
@withPackage('torch>=2.0.0')
@pytest.mark.parametrize('Model', [GCN, GraphSAGE, GIN, GAT, EdgeCNN, PNA])
@pytest.mark.skip(reason="Does not work yet in the full test suite")
def test_compile_graph_breaks(Model):
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    kwargs = {}
    if Model in {GCN, GAT}:
        # Adding self-loops inside the model leads to graph breaks :(
        kwargs['add_self_loops'] = False

    if Model in {PNA}:  # `PNA` requires additional arguments:
        kwargs['aggregators'] = ['sum', 'mean', 'min', 'max', 'var', 'std']
        kwargs['scalers'] = ['identity', 'amplification', 'attenuation']
        kwargs['deg'] = torch.tensor([1, 2, 1])

    model = Model(in_channels=8, hidden_channels=16, num_layers=2, **kwargs)

    def my_custom_backend(gm, *args):
        global num_compile_calls
        num_compile_calls += 1
        return gm.forward

    model = torch_geometric.compile(model, backend=my_custom_backend)

    num_previous_compile_calls = num_compile_calls
    model(x, edge_index)
    assert num_compile_calls - num_previous_compile_calls == 1


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    x = torch.randn(num_nodes, 64, device=args.device)
    edge_index = torch.randint(num_nodes, (2, num_edges), device=args.device)

    for Model in [GCN, GraphSAGE, GIN, EdgeCNN]:
        print(f'Model: {Model.__name__}')

        model = Model(64, 64, num_layers=3).to(args.device)
        compiled_model = torch_geometric.compile(model)

        benchmark(
            funcs=[model, compiled_model],
            func_names=['Vanilla', 'Compiled'],
            args=(x, edge_index),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )
pytorch/test/nn/models/test_correct_and_smooth.py000066400000000000000000000035111444262304300227120ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn.models import CorrectAndSmooth
from torch_geometric.typing import SparseTensor


def test_correct_and_smooth():
    y_soft = torch.tensor([0.1, 0.5, 0.4]).repeat(6, 1)
    y_true = torch.tensor([1, 0, 0, 2, 1, 1])
    edge_index = torch.tensor([[0, 1, 1, 2, 4, 5], [1, 0, 2, 1, 5, 4]])
    mask = torch.randint(0, 2, (6, ), dtype=torch.bool)

    model = CorrectAndSmooth(
        num_correction_layers=2,
        correction_alpha=0.5,
        num_smoothing_layers=2,
        smoothing_alpha=0.5,
    )
    assert str(model) == ('CorrectAndSmooth(\n'
                          '  correct: num_layers=2, alpha=0.5\n'
                          '  smooth:  num_layers=2, alpha=0.5\n'
                          '  autoscale=True, scale=1.0\n'
                          ')')

    out = model.correct(y_soft, y_true[mask], mask, edge_index)
    assert out.size() == (6, 3)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(6, 6))
        assert torch.allclose(
            out, model.correct(y_soft, y_true[mask], mask, adj.t()))

    out = model.smooth(y_soft, y_true[mask], mask, edge_index)
    assert out.size() == (6, 3)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(
            out, model.smooth(y_soft, y_true[mask], mask, adj.t()))

    # Test without autoscale:
    model = CorrectAndSmooth(
        num_correction_layers=2,
        correction_alpha=0.5,
        num_smoothing_layers=2,
        smoothing_alpha=0.5,
        autoscale=False,
    )
    out = model.correct(y_soft, y_true[mask], mask, edge_index)
    assert out.size() == (6, 3)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(
            out, model.correct(y_soft, y_true[mask], mask, adj.t()))
pytorch/test/nn/models/test_deep_graph_infomax.py000066400000000000000000000020371444262304300226570ustar00rootroot00000000000000import torch

from torch_geometric.nn import DeepGraphInfomax
from torch_geometric.testing import is_full_test


def test_deep_graph_infomax():
    def corruption(z):
        return z + 1

    model = DeepGraphInfomax(hidden_channels=16, encoder=lambda x: x,
                             summary=lambda z, *args: z.mean(dim=0),
                             corruption=lambda x: x + 1)

    assert str(model) == 'DeepGraphInfomax(16)'

    x = torch.ones(20, 16)

    pos_z, neg_z, summary = model(x)
    assert pos_z.size() == (20, 16) and neg_z.size() == (20, 16)
    assert summary.size() == (16, )

    if is_full_test():
        jit = torch.jit.export(model)
        pos_z, neg_z, summary = jit(x)
        assert pos_z.size() == (20, 16) and neg_z.size() == (20, 16)
        assert summary.size() == (16, )

    loss = model.loss(pos_z, neg_z, summary)
    assert 0 <= loss.item()

    acc = model.test(torch.ones(20, 16), torch.randint(10, (20, )),
                     torch.ones(20, 16), torch.randint(10, (20, )))
    assert 0 <= acc and acc <= 1
pytorch/test/nn/models/test_deepgcn.py000066400000000000000000000013141444262304300204420ustar00rootroot00000000000000import pytest
import torch
from torch.nn import ReLU

from torch_geometric.nn import DeepGCNLayer, GENConv, LayerNorm


@pytest.mark.parametrize(
    'block_tuple',
    [('res+', 1), ('res', 1), ('dense', 2), ('plain', 1)],
)
@pytest.mark.parametrize('ckpt_grad', [True, False])
def test_deepgcn(block_tuple, ckpt_grad):
    block, expansion = block_tuple
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    conv = GENConv(8, 8)
    norm = LayerNorm(8)
    act = ReLU()
    layer = DeepGCNLayer(conv, norm, act, block=block, ckpt_grad=ckpt_grad)
    assert str(layer) == f'DeepGCNLayer(block={block})'

    out = layer(x, edge_index)
    assert out.size() == (3, 8 * expansion)
pytorch/test/nn/models/test_dimenet.py000066400000000000000000000034721444262304300204710ustar00rootroot00000000000000import pytest
import torch
import torch.nn.functional as F

from torch_geometric.nn import DimeNet, DimeNetPlusPlus
from torch_geometric.nn.models.dimenet import (
    BesselBasisLayer,
    Envelope,
    ResidualLayer,
)
from torch_geometric.testing import is_full_test, withPackage


def test_dimenet_modules():
    env = Envelope(exponent=5)
    x = torch.randn(10, 3)
    assert env(x).size() == (10, 3)  # Isotonic layer.

    bbl = BesselBasisLayer(5)
    x = torch.randn(10, 3)
    assert bbl(x).size() == (10, 3, 5)  # Non-isotonic layer.

    rl = ResidualLayer(128, torch.nn.functional.relu)
    x = torch.randn(128, 128)
    assert rl(x).size() == (128, 128)  # Isotonic layer.


@withPackage('torch_sparse')  # TODO `triplet` requires `SparseTensor` for now.
@pytest.mark.parametrize('Model', [DimeNet, DimeNetPlusPlus])
def test_dimenet(Model):
    z = torch.randint(1, 10, (20, ))
    pos = torch.randn(20, 3)

    if Model == DimeNet:
        kwargs = dict(num_bilinear=3)
    else:
        kwargs = dict(out_emb_channels=3, int_emb_size=5, basis_emb_size=5)

    model = Model(
        hidden_channels=5,
        out_channels=1,
        num_blocks=5,
        num_spherical=5,
        num_radial=5,
        **kwargs,
    )
    model.reset_parameters()

    with torch.no_grad():
        out = model(z, pos)
        assert out.size() == (1, )

        jit = torch.jit.export(model)
        assert torch.allclose(jit(z, pos), out)

    if is_full_test():
        optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

        min_loss = float('inf')
        for i in range(100):
            optimizer.zero_grad()
            out = model(z, pos)
            loss = F.l1_loss(out, torch.tensor([1.0]))
            loss.backward()
            optimizer.step()
            min_loss = min(float(loss), min_loss)
        assert min_loss < 2
pytorch/test/nn/models/test_gnnff.py000066400000000000000000000011141444262304300201310ustar00rootroot00000000000000import torch

from torch_geometric.nn import GNNFF
from torch_geometric.testing import is_full_test, withPackage


@withPackage('torch_sparse')  # TODO `triplet` requires `SparseTensor` for now.
def test_gnnff():
    z = torch.randint(1, 10, (20, ))
    pos = torch.randn(20, 3)

    model = GNNFF(
        hidden_node_channels=5,
        hidden_edge_channels=5,
        num_layers=5,
    )
    model.reset_parameters()

    out = model(z, pos)
    assert out.size() == (20, 3)

    if is_full_test():
        jit = torch.jit.export(model)
        assert torch.allclose(jit(z, pos), out)
pytorch/test/nn/models/test_graph_mixer.py000066400000000000000000000015411444262304300213440ustar00rootroot00000000000000import torch

from torch_geometric.nn.models.graph_mixer import NodeEncoder


def test_node_encoder():
    x = torch.arange(4, dtype=torch.float).view(-1, 1)
    edge_index = torch.tensor([[1, 2, 0, 0, 1, 3], [0, 0, 1, 2, 2, 2]])
    edge_time = torch.tensor([0, 1, 1, 1, 2, 3])
    seed_time = torch.tensor([2, 2, 2, 2])

    encoder = NodeEncoder(time_window=2)
    assert str(encoder) == 'NodeEncoder(time_window=2)'

    out = encoder(x, edge_index, edge_time, seed_time)
    # Node 0 aggregates information from node 2 (excluding node 1).
    # Node 1 aggregates information from node 0.
    # Node 2 aggregates information from node 0 and node 1 (exluding node 3).
    # Node 3 aggregates no information.
    expected = torch.tensor([
        [0 + 2],
        [1 + 0],
        [2 + 0.5 * (0 + 1)],
        [3],
    ])
    assert torch.allclose(out, expected)
pytorch/test/nn/models/test_graph_unet.py000066400000000000000000000011761444262304300211770ustar00rootroot00000000000000import torch

from torch_geometric.nn import GraphUNet
from torch_geometric.testing import is_full_test, onlyLinux


@onlyLinux  # TODO  (matthias) Investigate CSR @ CSR support on Windows.
def test_graph_unet():
    model = GraphUNet(16, 32, 8, depth=3)
    out = 'GraphUNet(16, 32, 8, depth=3, pool_ratios=[0.5, 0.5, 0.5])'
    assert str(model) == out

    x = torch.randn(3, 16)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    out = model(x, edge_index)
    assert out.size() == (3, 8)

    if is_full_test():
        jit = torch.jit.export(model)
        out = jit(x, edge_index)
        assert out.size() == (3, 8)
pytorch/test/nn/models/test_jumping_knowledge.py000066400000000000000000000022111444262304300225420ustar00rootroot00000000000000import torch

from torch_geometric.nn import JumpingKnowledge
from torch_geometric.testing import is_full_test


def test_jumping_knowledge():
    num_nodes, channels, num_layers = 100, 17, 5
    xs = list([torch.randn(num_nodes, channels) for _ in range(num_layers)])

    model = JumpingKnowledge('cat')
    assert str(model) == 'JumpingKnowledge(cat)'

    out = model(xs)
    assert out.size() == (num_nodes, channels * num_layers)

    if is_full_test():
        jit = torch.jit.script(model)
        assert torch.allclose(jit(xs), out)

    model = JumpingKnowledge('max')
    assert str(model) == 'JumpingKnowledge(max)'

    out = model(xs)
    assert out.size() == (num_nodes, channels)

    if is_full_test():
        jit = torch.jit.script(model)
        assert torch.allclose(jit(xs), out)

    model = JumpingKnowledge('lstm', channels, num_layers)
    assert str(model) == (f'JumpingKnowledge(lstm, channels='
                          f'{channels}, layers={num_layers})')

    out = model(xs)
    assert out.size() == (num_nodes, channels)

    if is_full_test():
        jit = torch.jit.script(model)
        assert torch.allclose(jit(xs), out)
pytorch/test/nn/models/test_label_prop.py000066400000000000000000000020531444262304300211550ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn.models import LabelPropagation
from torch_geometric.typing import SparseTensor


def test_label_prop():
    y = torch.tensor([1, 0, 0, 2, 1, 1])
    edge_index = torch.tensor([[0, 1, 1, 2, 4, 5], [1, 0, 2, 1, 5, 4]])
    mask = torch.randint(0, 2, (6, ), dtype=torch.bool)

    model = LabelPropagation(num_layers=2, alpha=0.5)
    assert str(model) == 'LabelPropagation(num_layers=2, alpha=0.5)'

    # Test without mask:
    out = model(y, edge_index)
    assert out.size() == (6, 3)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(6, 6))
        assert torch.allclose(model(y, adj.t()), out)

    # Test with mask:
    out = model(y, edge_index, mask)
    assert out.size() == (6, 3)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(model(y, adj.t(), mask), out)

    # Test post step:
    out = model(y, edge_index, mask, post_step=lambda y: torch.zeros_like(y))
    assert torch.sum(out) == 0.
pytorch/test/nn/models/test_lightgcn.py000066400000000000000000000053031444262304300206360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn.models import LightGCN


@pytest.mark.parametrize('embedding_dim', [32, 64])
@pytest.mark.parametrize('with_edge_weight', [False, True])
@pytest.mark.parametrize('lambda_reg', [0, 1e-4])
@pytest.mark.parametrize('alpha', [0, .25, torch.tensor([0.4, 0.3, 0.2])])
def test_lightgcn_ranking(embedding_dim, with_edge_weight, lambda_reg, alpha):
    num_nodes = 500
    num_edges = 400
    edge_index = torch.randint(0, num_nodes, (2, num_edges))
    edge_weight = torch.rand(num_edges) if with_edge_weight else None
    edge_label_index = torch.randint(0, num_nodes, (2, 100))

    model = LightGCN(num_nodes, embedding_dim, num_layers=2, alpha=alpha)
    assert str(model) == f'LightGCN(500, {embedding_dim}, num_layers=2)'

    pred = model(edge_index, edge_label_index, edge_weight)
    assert pred.size() == (100, )

    loss = model.recommendation_loss(
        pos_edge_rank=pred[:50],
        neg_edge_rank=pred[50:],
        node_id=edge_index.unique(),
        lambda_reg=lambda_reg,
    )
    assert loss.dim() == 0 and loss > 0

    out = model.recommend(edge_index, edge_weight, k=2)
    assert out.size() == (500, 2)
    assert out.min() >= 0 and out.max() < 500

    src_index = torch.arange(0, 250)
    dst_index = torch.arange(250, 500)

    out = model.recommend(edge_index, edge_weight, src_index, dst_index, k=2)
    assert out.size() == (250, 2)
    assert out.min() >= 250 and out.max() < 500


@pytest.mark.parametrize('embedding_dim', [32, 64])
@pytest.mark.parametrize('with_edge_weight', [False, True])
@pytest.mark.parametrize('alpha', [0, .25, torch.tensor([0.4, 0.3, 0.2])])
def test_lightgcn_link_prediction(embedding_dim, with_edge_weight, alpha):
    num_nodes = 500
    num_edges = 400
    edge_index = torch.randint(0, num_nodes, (2, num_edges))
    edge_weight = torch.rand(num_edges) if with_edge_weight else None
    edge_label_index = torch.randint(0, num_nodes, (2, 100))
    edge_label = torch.randint(0, 2, (edge_label_index.size(1), ))

    model = LightGCN(num_nodes, embedding_dim, num_layers=2, alpha=alpha)
    assert str(model) == f'LightGCN(500, {embedding_dim}, num_layers=2)'

    pred = model(edge_index, edge_label_index, edge_weight)
    assert pred.size() == (100, )

    loss = model.link_pred_loss(pred, edge_label)
    assert loss.dim() == 0 and loss > 0

    prob = model.predict_link(edge_index, edge_label_index, edge_weight,
                              prob=True)
    assert prob.size() == (100, )
    assert prob.min() > 0 and prob.max() < 1

    prob = model.predict_link(edge_index, edge_label_index, edge_weight,
                              prob=False)
    assert prob.size() == (100, )
    assert ((prob == 0) | (prob == 1)).sum() == 100
pytorch/test/nn/models/test_linkx.py000066400000000000000000000037671444262304300202000ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import LINKX
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


@pytest.mark.parametrize('num_edge_layers', [1, 2])
def test_linkx(num_edge_layers):
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 2], [1, 2, 3]])
    edge_weight = torch.rand(edge_index.size(1))

    model = LINKX(num_nodes=4, in_channels=16, hidden_channels=32,
                  out_channels=8, num_layers=2,
                  num_edge_layers=num_edge_layers)
    assert str(model) == 'LINKX(num_nodes=4, in_channels=16, out_channels=8)'

    out = model(x, edge_index)
    assert out.size() == (4, 8)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(4, 4))
        assert torch.allclose(out, model(x, adj.t()), atol=1e-6)

    if is_full_test():
        t = '(OptTensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(model.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(OptTensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(model.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out, atol=1e-6)

    out = model(None, edge_index)
    assert out.size() == (4, 8)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(out, model(None, adj.t()), atol=1e-6)

    out = model(x, edge_index, edge_weight)
    assert out.size() == (4, 8)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, edge_weight,
                                           sparse_sizes=(4, 4))
        assert torch.allclose(model(x, adj.t()), out, atol=1e-6)

    out = model(None, edge_index, edge_weight)
    assert out.size() == (4, 8)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(model(None, adj.t()), out, atol=1e-6)
pytorch/test/nn/models/test_mask_label.py000066400000000000000000000013201444262304300211240ustar00rootroot00000000000000import torch

from torch_geometric.nn import MaskLabel


def test_mask_label():
    model = MaskLabel(2, 10)
    assert str(model) == 'MaskLabel()'

    x = torch.rand(4, 10)
    y = torch.tensor([1, 0, 1, 0])
    mask = torch.tensor([False, False, True, True])

    out = model(x, y, mask)
    assert out.size() == (4, 10)
    assert torch.allclose(out[~mask], x[~mask])

    model = MaskLabel(2, 10, method='concat')
    out = model(x, y, mask)
    assert out.size() == (4, 20)
    assert torch.allclose(out[:, :10], x)


def test_ratio_mask():
    mask = torch.tensor([True, True, True, True, False, False, False, False])
    out = MaskLabel.ratio_mask(mask, 0.5)
    assert out[:4].sum() <= 4 and out[4:].sum() == 0
pytorch/test/nn/models/test_meta.py000066400000000000000000000100161444262304300177620ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

from torch_geometric.nn import MetaLayer
from torch_geometric.testing import is_full_test
from torch_geometric.utils import scatter

count = 0


def test_meta_layer():
    assert str(MetaLayer()) == ('MetaLayer(\n'
                                '  edge_model=None,\n'
                                '  node_model=None,\n'
                                '  global_model=None\n'
                                ')')

    def dummy_model(*args):
        global count
        count += 1
        return None

    x = torch.randn(20, 10)
    edge_index = torch.randint(0, high=10, size=(2, 20), dtype=torch.long)

    for edge_model in (dummy_model, None):
        for node_model in (dummy_model, None):
            for global_model in (dummy_model, None):
                model = MetaLayer(edge_model, node_model, global_model)
                out = model(x, edge_index)
                assert isinstance(out, tuple) and len(out) == 3

    assert count == 12


def test_meta_layer_example():
    class EdgeModel(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.edge_mlp = Seq(Lin(2 * 10 + 5 + 20, 5), ReLU(), Lin(5, 5))

        def forward(
            self,
            src: Tensor,
            dest: Tensor,
            edge_attr: Optional[Tensor],
            u: Optional[Tensor],
            batch: Optional[Tensor],
        ) -> Tensor:
            assert edge_attr is not None
            assert u is not None
            assert batch is not None
            out = torch.cat([src, dest, edge_attr, u[batch]], 1)
            return self.edge_mlp(out)

    class NodeModel(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.node_mlp_1 = Seq(Lin(15, 10), ReLU(), Lin(10, 10))
            self.node_mlp_2 = Seq(Lin(2 * 10 + 20, 10), ReLU(), Lin(10, 10))

        def forward(
            self,
            x: Tensor,
            edge_index: Tensor,
            edge_attr: Optional[Tensor],
            u: Optional[Tensor],
            batch: Optional[Tensor],
        ) -> Tensor:
            assert edge_attr is not None
            assert u is not None
            assert batch is not None
            row = edge_index[0]
            col = edge_index[1]
            out = torch.cat([x[row], edge_attr], dim=1)
            out = self.node_mlp_1(out)
            out = scatter(out, col, dim=0, dim_size=x.size(0), reduce='mean')
            out = torch.cat([x, out, u[batch]], dim=1)
            return self.node_mlp_2(out)

    class GlobalModel(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.global_mlp = Seq(Lin(20 + 10, 20), ReLU(), Lin(20, 20))

        def forward(
            self,
            x: Tensor,
            edge_index: Tensor,
            edge_attr: Optional[Tensor],
            u: Optional[Tensor],
            batch: Optional[Tensor],
        ) -> Tensor:
            assert u is not None
            assert batch is not None
            out = torch.cat([
                u,
                scatter(x, batch, dim=0, reduce='mean'),
            ], dim=1)
            return self.global_mlp(out)

    op = MetaLayer(EdgeModel(), NodeModel(), GlobalModel())

    x = torch.randn(20, 10)
    edge_attr = torch.randn(40, 5)
    u = torch.randn(2, 20)
    batch = torch.tensor([0] * 10 + [1] * 10)
    edge_index = torch.randint(0, high=10, size=(2, 20), dtype=torch.long)
    edge_index = torch.cat([edge_index, 10 + edge_index], dim=1)

    x_out, edge_attr_out, u_out = op(x, edge_index, edge_attr, u, batch)
    assert x_out.size() == (20, 10)
    assert edge_attr_out.size() == (40, 5)
    assert u_out.size() == (2, 20)

    if is_full_test():
        jit = torch.jit.script(op)

        x_out, edge_attr_out, u_out = jit(x, edge_index, edge_attr, u, batch)
        assert x_out.size() == (20, 10)
        assert edge_attr_out.size() == (40, 5)
        assert u_out.size() == (2, 20)
pytorch/test/nn/models/test_metapath2vec.py000066400000000000000000000022621444262304300214230ustar00rootroot00000000000000import torch

from torch_geometric.nn import MetaPath2Vec
from torch_geometric.testing import withCUDA


@withCUDA
def test_metapath2vec(device):
    edge_index_dict = {
        ('author', 'writes', 'paper'):
        torch.tensor([[0, 1, 1, 2], [0, 0, 1, 1]], device=device),
        ('paper', 'written_by', 'author'):
        torch.tensor([[0, 0, 1, 1], [0, 1, 1, 2]], device=device)
    }

    metapath = [
        ('author', 'writes', 'paper'),
        ('paper', 'written_by', 'author'),
    ]

    model = MetaPath2Vec(edge_index_dict, embedding_dim=16, metapath=metapath,
                         walk_length=2, context_size=2).to(device)
    assert str(model) == 'MetaPath2Vec(5, 16)'

    z = model('author')
    assert z.size() == (3, 16)

    z = model('paper')
    assert z.size() == (2, 16)

    z = model('author', torch.arange(2, device=device))
    assert z.size() == (2, 16)

    pos_rw, neg_rw = model._sample(torch.arange(3))

    loss = model.loss(pos_rw.to(device), neg_rw.to(device))
    assert 0 <= loss.item()

    acc = model.test(torch.ones(20, 16), torch.randint(10, (20, )),
                     torch.ones(20, 16), torch.randint(10, (20, )))
    assert 0 <= acc and acc <= 1
pytorch/test/nn/models/test_mlp.py000066400000000000000000000026201444262304300176260ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import MLP


@pytest.mark.parametrize('norm', ['batch_norm', None])
@pytest.mark.parametrize('act_first', [False, True])
@pytest.mark.parametrize('plain_last', [False, True])
def test_mlp(norm, act_first, plain_last):
    x = torch.randn(4, 16)

    torch.manual_seed(12345)
    mlp = MLP(
        [16, 32, 32, 64],
        norm=norm,
        act_first=act_first,
        plain_last=plain_last,
    )
    assert str(mlp) == 'MLP(16, 32, 32, 64)'
    out = mlp(x)
    assert out.size() == (4, 64)

    jit = torch.jit.script(mlp)
    assert torch.allclose(jit(x), out)

    torch.manual_seed(12345)
    mlp = MLP(
        16,
        hidden_channels=32,
        out_channels=64,
        num_layers=3,
        norm=norm,
        act_first=act_first,
        plain_last=plain_last,
    )
    assert torch.allclose(mlp(x), out)


def test_mlp_return_emb():
    x = torch.randn(4, 16)

    mlp = MLP([16, 32, 1])

    out, emb = mlp(x, return_emb=True)
    assert out.size() == (4, 1)
    assert emb.size() == (4, 32)

    out, emb = mlp(x, return_emb=False)
    assert out.size() == (4, 1)
    assert emb is None


@pytest.mark.parametrize('plain_last', [False, True])
def test_fine_grained_mlp(plain_last):
    mlp = MLP(
        [16, 32, 32, 64],
        dropout=[0.1, 0.2, 0.3],
        bias=[False, True, False],
    )
    assert mlp(torch.randn(4, 16)).size() == (4, 64)
pytorch/test/nn/models/test_node2vec.py000066400000000000000000000023011444262304300205370ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import Node2Vec
from torch_geometric.testing import is_full_test, withCUDA, withPackage


@withCUDA
@withPackage('pyg_lib')
@withPackage('torch_cluster')
@pytest.mark.parametrize('p', [1.0])
@pytest.mark.parametrize('q', [1.0, 0.5])
def test_node2vec(device, p, q):
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], device=device)

    model = Node2Vec(
        edge_index,
        embedding_dim=16,
        walk_length=2,
        context_size=2,
        p=p,
        q=q,
    ).to(device)
    assert str(model) == 'Node2Vec(3, 16)'

    assert model(torch.arange(3, device=device)).size() == (3, 16)

    pos_rw, neg_rw = model.sample(torch.arange(3))
    assert float(model.loss(pos_rw.to(device), neg_rw.to(device))) >= 0

    acc = model.test(torch.ones(20, 16), torch.randint(10, (20, )),
                     torch.ones(20, 16), torch.randint(10, (20, )))
    assert 0 <= acc and acc <= 1

    if is_full_test():
        jit = torch.jit.script(model)

        assert jit(torch.arange(3, device=device)).size() == (3, 16)

        pos_rw, neg_rw = jit.sample(torch.arange(3))
        assert float(jit.loss(pos_rw.to(device), neg_rw.to(device))) >= 0
pytorch/test/nn/models/test_pmlp.py000066400000000000000000000011161444262304300200050ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn.models import PMLP


def test_pmlp():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    pmlp = PMLP(in_channels=16, hidden_channels=32, out_channels=2,
                num_layers=4)
    assert str(pmlp) == 'PMLP(16, 2, num_layers=4)'

    pmlp.training = True
    assert pmlp(x).size() == (4, 2)

    pmlp.training = False
    assert pmlp(x, edge_index).size() == (4, 2)

    with pytest.raises(ValueError, match="'edge_index' needs to be present"):
        pmlp.training = False
        pmlp(x)
pytorch/test/nn/models/test_re_net.py000066400000000000000000000040331444262304300203120ustar00rootroot00000000000000import torch

from torch_geometric.datasets.icews import EventDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import RENet
from torch_geometric.testing import is_full_test


class MyTestEventDataset(EventDataset):
    def __init__(self, root, seq_len):
        super().__init__(root, pre_transform=RENet.pre_transform(seq_len))
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def num_nodes(self):
        return 16

    @property
    def num_rels(self):
        return 8

    @property
    def processed_file_names(self):
        return 'data.pt'

    def _download(self):
        pass

    def process_events(self):
        sub = torch.randint(self.num_nodes, (64, ), dtype=torch.long)
        rel = torch.randint(self.num_rels, (64, ), dtype=torch.long)
        obj = torch.randint(self.num_nodes, (64, ), dtype=torch.long)
        t = torch.arange(8, dtype=torch.long).view(-1, 1).repeat(1, 8).view(-1)
        return torch.stack([sub, rel, obj, t], dim=1)

    def process(self):
        data_list = super().process()
        torch.save(self.collate(data_list), self.processed_paths[0])


def test_re_net(tmp_path):
    dataset = MyTestEventDataset(tmp_path, seq_len=4)
    loader = DataLoader(dataset, 2, follow_batch=['h_sub', 'h_obj'])

    model = RENet(dataset.num_nodes, dataset.num_rels, hidden_channels=16,
                  seq_len=4)

    if is_full_test():
        jit = torch.jit.export(model)

    logits = torch.randn(6, 6)
    y = torch.tensor([0, 1, 2, 3, 4, 5])

    mrr, hits1, hits3, hits10 = model.test(logits, y)
    assert 0.15 < mrr <= 1
    assert hits1 <= hits3 and hits3 <= hits10 and hits10 == 1

    for data in loader:
        log_prob_obj, log_prob_sub = model(data)
        if is_full_test():
            log_prob_obj_jit, log_prob_sub_jit = jit(data)
            assert torch.allclose(log_prob_obj_jit, log_prob_obj)
            assert torch.allclose(log_prob_sub_jit, log_prob_sub)
        model.test(log_prob_obj, data.obj)
        model.test(log_prob_sub, data.sub)
pytorch/test/nn/models/test_rect.py000066400000000000000000000033721444262304300200000ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.nn import RECT_L
from torch_geometric.testing import is_full_test
from torch_geometric.typing import SparseTensor


def test_rect():
    x = torch.randn(6, 8)
    y = torch.tensor([1, 0, 0, 2, 1, 1])
    edge_index = torch.tensor([[0, 1, 1, 2, 4, 5], [1, 0, 2, 1, 5, 4]])
    mask = torch.randint(0, 2, (6, ), dtype=torch.bool)

    model = RECT_L(8, 16)
    assert str(model) == 'RECT_L(8, 16)'

    out = model(x, edge_index)
    assert out.size() == (6, 8)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=(6, 6))
        assert torch.allclose(out, model(x, adj.t()), atol=1e-6)

    # Test `embed`:
    embed_out = model.embed(x, edge_index)
    assert embed_out.size() == (6, 16)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert torch.allclose(embed_out, model.embed(x, adj.t()), atol=1e-6)

    # Test `get_semantic_labels`:
    labeds_out = model.get_semantic_labels(x, y, mask)
    assert labeds_out.size() == (int(mask.sum()), 8)

    if is_full_test():
        t = '(Tensor, Tensor, OptTensor) -> Tensor'
        jit = torch.jit.script(model.jittable(t))
        assert torch.allclose(jit(x, edge_index), out)
        assert torch.allclose(embed_out, jit.embed(x, edge_index))
        assert torch.allclose(labeds_out, jit.get_semantic_labels(x, y, mask))

    if is_full_test() and torch_geometric.typing.WITH_TORCH_SPARSE:
        t = '(Tensor, SparseTensor, OptTensor) -> Tensor'
        jit = torch.jit.script(model.jittable(t))
        assert torch.allclose(jit(x, adj.t()), out)
        assert torch.allclose(embed_out, jit.embed(x, adj.t()))
        assert torch.allclose(labeds_out, jit.get_semantic_labels(x, y, mask))
pytorch/test/nn/models/test_rev_gnn.py000066400000000000000000000061111444262304300204730ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.nn import GraphConv, GroupAddRev, SAGEConv
from torch_geometric.nn.dense.linear import Linear


@pytest.mark.parametrize('num_groups', [2, 4, 8, 16])
def test_revgnn_forward_inverse(num_groups):
    x = torch.randn(4, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    lin = Linear(32, 32)
    conv = SAGEConv(32 // num_groups, 32 // num_groups)
    conv = GroupAddRev(conv, num_groups=num_groups)
    assert str(conv) == (f'GroupAddRev(SAGEConv({32 // num_groups}, '
                         f'{32 // num_groups}, aggr=mean), '
                         f'num_groups={num_groups})')

    h = lin(x)
    h_o = h.clone().detach()

    out = conv(h, edge_index)
    if torch_geometric.typing.WITH_PT2:
        assert h.untyped_storage().size() == 0
    else:
        assert h.storage().size() == 0

    h_rev = conv.inverse(out, edge_index)
    assert torch.allclose(h_o, h_rev, atol=0.001)


@pytest.mark.parametrize('num_groups', [2, 4, 8, 16])
def test_revgnn_backward(num_groups):
    x = torch.randn(4, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    lin = Linear(32, 32)
    conv = SAGEConv(32 // num_groups, 32 // num_groups)
    conv = GroupAddRev(conv, num_groups=num_groups)

    h = lin(x)
    out = conv(h, edge_index)
    target = out.mean()
    target.backward()


@pytest.mark.parametrize('num_groups', [2, 4, 8, 16])
def test_revgnn_multi_backward(num_groups):
    x = torch.randn(4, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    lin = Linear(32, 32)
    conv = SAGEConv(32 // num_groups, 32 // num_groups)
    conv = GroupAddRev(conv, num_groups=num_groups, num_bwd_passes=4)

    h = lin(x)
    out = conv(h, edge_index)
    target = out.mean()
    target.backward(retain_graph=True)
    target.backward(retain_graph=True)
    torch.autograd.grad(outputs=target, inputs=[h] + list(conv.parameters()),
                        retain_graph=True)
    torch.autograd.grad(outputs=target, inputs=[h] + list(conv.parameters()))


@pytest.mark.parametrize('num_groups', [2, 4, 8, 16])
def test_revgnn_diable(num_groups):
    x = torch.randn(4, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])

    lin = Linear(32, 32)
    conv = SAGEConv(32 // num_groups, 32 // num_groups)
    conv = GroupAddRev(conv, num_groups=num_groups, disable=True)

    h = lin(x)
    out = conv(h, edge_index)
    target = out.mean()
    target.backward()

    # Memory will not be freed if disable:
    if torch_geometric.typing.WITH_PT2:
        assert h.untyped_storage().size() == 4 * 4 * 32
    else:
        assert h.storage().size() == 4 * 32


@pytest.mark.parametrize('num_groups', [2, 4, 8, 16])
def test_revgnn_with_args(num_groups):
    x = torch.randn(4, 32)
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    edge_weight = torch.rand(4)

    lin = Linear(32, 32)
    conv = GraphConv(32 // num_groups, 32 // num_groups)
    conv = GroupAddRev(conv, num_groups=num_groups)

    h = lin(x)
    out = conv(h, edge_index, edge_weight)
    target = out.mean()
    target.backward()
pytorch/test/nn/models/test_schnet.py000066400000000000000000000035151444262304300203260ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Batch, Data
from torch_geometric.nn import SchNet
from torch_geometric.nn.models.schnet import RadiusInteractionGraph
from torch_geometric.testing import is_full_test, withPackage


def generate_data():
    return Data(
        z=torch.randint(1, 10, (20, )),
        pos=torch.randn(20, 3),
    )


@withPackage('torch_cluster')
@withPackage('ase')
@pytest.mark.parametrize('use_interaction_graph', [False, True])
@pytest.mark.parametrize('use_atomref', [False, True])
def test_schnet(use_interaction_graph, use_atomref):
    data = generate_data()

    interaction_graph = None
    if use_interaction_graph:
        interaction_graph = RadiusInteractionGraph(cutoff=6.0)

    model = SchNet(
        hidden_channels=16,
        num_filters=16,
        num_interactions=2,
        interaction_graph=interaction_graph,
        num_gaussians=10,
        cutoff=6.0,
        dipole=True,
        atomref=torch.randn(100, 1) if use_atomref else None,
    )

    assert str(model) == ('SchNet(hidden_channels=16, num_filters=16, '
                          'num_interactions=2, num_gaussians=10, cutoff=6.0)')

    with torch.no_grad():
        out = model(data.z, data.pos)
        assert out.size() == (1, 1)

        if is_full_test():
            jit = torch.jit.export(model)
            out = jit(data.z, data.pos)
            assert out.size() == (1, 1)


@withPackage('torch_cluster')
def test_schnet_batch():
    num_graphs = 3
    batch = [generate_data() for _ in range(num_graphs)]
    batch = Batch.from_data_list(batch)

    model = SchNet(
        hidden_channels=16,
        num_filters=16,
        num_interactions=2,
        num_gaussians=10,
        cutoff=6.0,
    )

    with torch.no_grad():
        out = model(batch.z, batch.pos, batch.batch)
        assert out.size() == (num_graphs, 1)
pytorch/test/nn/models/test_signed_gcn.py000066400000000000000000000022731444262304300211420ustar00rootroot00000000000000import torch

from torch_geometric.nn import SignedGCN
from torch_geometric.testing import is_full_test


def test_signed_gcn():
    model = SignedGCN(8, 16, num_layers=2, lamb=5)
    assert str(model) == 'SignedGCN(8, 16, num_layers=2)'

    pos_index = torch.randint(high=10, size=(2, 40), dtype=torch.long)
    neg_index = torch.randint(high=10, size=(2, 40), dtype=torch.long)

    train_pos_index, test_pos_index = model.split_edges(pos_index)
    train_neg_index, test_neg_index = model.split_edges(neg_index)

    assert train_pos_index.size() == (2, 32)
    assert test_pos_index.size() == (2, 8)
    assert train_neg_index.size() == (2, 32)
    assert test_neg_index.size() == (2, 8)

    x = model.create_spectral_features(train_pos_index, train_neg_index, 10)
    assert x.size() == (10, 8)

    z = model(x, train_pos_index, train_neg_index)
    assert z.size() == (10, 16)

    loss = model.loss(z, train_pos_index, train_neg_index)
    assert loss.item() >= 0

    auc, f1 = model.test(z, test_pos_index, test_neg_index)
    assert auc >= 0
    assert f1 >= 0

    if is_full_test():
        jit = torch.jit.export(model)
        assert torch.allclose(jit(x, train_pos_index, train_neg_index), z)
pytorch/test/nn/models/test_tgn.py000066400000000000000000000052721444262304300176340ustar00rootroot00000000000000import torch

from torch_geometric.data import TemporalData
from torch_geometric.loader import TemporalDataLoader
from torch_geometric.nn import TGNMemory
from torch_geometric.nn.models.tgn import (
    IdentityMessage,
    LastAggregator,
    LastNeighborLoader,
)


def test_tgn():
    memory_dim = 16
    time_dim = 16

    src = torch.tensor([0, 1, 0, 2, 0, 3, 1, 4, 2, 3])
    dst = torch.tensor([1, 2, 1, 1, 3, 2, 4, 3, 3, 4])
    t = torch.arange(10)
    msg = torch.randn(10, 16)
    data = TemporalData(src=src, dst=dst, t=t, msg=msg)

    loader = TemporalDataLoader(data, batch_size=5)

    neighbor_loader = LastNeighborLoader(data.num_nodes, size=3)
    assert neighbor_loader.cur_e_id == 0
    assert neighbor_loader.e_id.size() == (data.num_nodes, 3)

    memory = TGNMemory(
        num_nodes=data.num_nodes,
        raw_msg_dim=msg.size(-1),
        memory_dim=memory_dim,
        time_dim=time_dim,
        message_module=IdentityMessage(msg.size(-1), memory_dim, time_dim),
        aggregator_module=LastAggregator(),
    )
    assert memory.memory.size() == (data.num_nodes, memory_dim)
    assert memory.last_update.size() == (data.num_nodes, )

    # Test TGNMemory training:
    for i, batch in enumerate(loader):
        n_id = torch.cat([batch.src, batch.dst]).unique()
        n_id, edge_index, e_id = neighbor_loader(n_id)
        z, last_update = memory(n_id)
        memory.update_state(batch.src, batch.dst, batch.t, batch.msg)
        neighbor_loader.insert(batch.src, batch.dst)
        if i == 0:
            assert n_id.size(0) == 4
            assert edge_index.numel() == 0
            assert e_id.numel() == 0
            assert z.size() == (n_id.size(0), memory_dim)
            assert torch.sum(last_update) == 0
        else:
            assert n_id.size(0) == 5
            assert edge_index.numel() == 12
            assert e_id.numel() == 6
            assert z.size() == (n_id.size(0), memory_dim)
            assert torch.equal(last_update, torch.tensor([4, 3, 3, 4, 0]))

    # Test TGNMemory inference:
    memory.eval()
    all_n_id = torch.arange(data.num_nodes)
    z, last_update = memory(all_n_id)
    assert z.size() == (data.num_nodes, memory_dim)
    assert torch.equal(last_update, torch.tensor([4, 6, 8, 9, 9]))

    post_src = torch.tensor([3, 4])
    post_dst = torch.tensor([4, 3])
    post_t = torch.tensor([10, 10])
    post_msg = torch.randn(2, 16)
    memory.update_state(post_src, post_dst, post_t, post_msg)
    post_z, post_last_update = memory(all_n_id)
    assert torch.allclose(z[0:3], post_z[0:3])
    assert torch.equal(post_last_update, torch.tensor([4, 6, 8, 10, 10]))

    memory.reset_state()
    assert memory.memory.sum() == 0
    assert memory.last_update.sum() == 0
pytorch/test/nn/norm/000077500000000000000000000000001444262304300151155ustar00rootroot00000000000000pytorch/test/nn/norm/test_batch_norm.py000066400000000000000000000044331444262304300206460ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import BatchNorm, HeteroBatchNorm
from torch_geometric.testing import is_full_test, withCUDA


@withCUDA
@pytest.mark.parametrize('conf', [True, False])
def test_batch_norm(device, conf):
    x = torch.randn(100, 16, device=device)

    norm = BatchNorm(16, affine=conf, track_running_stats=conf).to(device)
    norm.reset_running_stats()
    norm.reset_parameters()
    assert str(norm) == 'BatchNorm(16)'

    if is_full_test():
        torch.jit.script(norm)

    out = norm(x)
    assert out.size() == (100, 16)


def test_batch_norm_single_element():
    x = torch.randn(1, 16)

    norm = BatchNorm(16)
    with pytest.raises(ValueError, match="Expected more than 1 value"):
        norm(x)

    with pytest.raises(ValueError, match="requires 'track_running_stats'"):
        norm = BatchNorm(16, track_running_stats=False,
                         allow_single_element=True)

    norm = BatchNorm(16, track_running_stats=True, allow_single_element=True)
    out = norm(x)
    assert torch.allclose(out, x)


@withCUDA
@pytest.mark.parametrize('conf', [True, False])
def test_hetero_batch_norm(device, conf):
    x = torch.randn((100, 16), device=device)

    # Test single type:
    norm = BatchNorm(16, affine=conf, track_running_stats=conf).to(device)
    expected = norm(x)

    type_vec = torch.zeros(100, dtype=torch.long, device=device)
    norm = HeteroBatchNorm(16, num_types=1, affine=conf,
                           track_running_stats=conf).to(device)
    norm.reset_running_stats()
    norm.reset_parameters()
    assert str(norm) == 'HeteroBatchNorm(16, num_types=1)'

    out = norm(x, type_vec)
    assert out.size() == (100, 16)
    assert torch.allclose(out, expected, atol=1e-3)

    # Test multiple types:
    type_vec = torch.randint(5, (100, ), device=device)
    norm = HeteroBatchNorm(16, num_types=5, affine=conf,
                           track_running_stats=conf).to(device)
    out = norm(x, type_vec)
    assert out.size() == (100, 16)

    for i in range(5):  # Check that mean=0 and std=1 across all types:
        mean = out[type_vec == i].mean()
        std = out[type_vec == i].std(unbiased=False)
        assert torch.allclose(mean, torch.zeros_like(mean), atol=1e-7)
        assert torch.allclose(std, torch.ones_like(std), atol=1e-7)
pytorch/test/nn/norm/test_diff_group_norm.py000066400000000000000000000016701444262304300217110ustar00rootroot00000000000000import torch

from torch_geometric.nn import DiffGroupNorm
from torch_geometric.testing import is_full_test


def test_diff_group_norm():
    x = torch.randn(6, 16)

    norm = DiffGroupNorm(16, groups=4, lamda=0)
    assert str(norm) == 'DiffGroupNorm(16, groups=4)'

    assert torch.allclose(norm(x), x)

    if is_full_test():
        jit = torch.jit.script(norm)
        assert torch.allclose(jit(x), x)

    norm = DiffGroupNorm(16, groups=4, lamda=0.01)
    assert str(norm) == 'DiffGroupNorm(16, groups=4)'

    out = norm(x)
    assert out.size() == x.size()

    if is_full_test():
        jit = torch.jit.script(norm)
        assert torch.allclose(jit(x), out)


def test_group_distance_ratio():
    x = torch.randn(6, 16)
    y = torch.tensor([0, 1, 0, 1, 1, 1])

    assert DiffGroupNorm.group_distance_ratio(x, y) > 0

    if is_full_test():
        jit = torch.jit.script(DiffGroupNorm.group_distance_ratio)
        assert jit(x, y) > 0
pytorch/test/nn/norm/test_graph_norm.py000066400000000000000000000015311444262304300206620ustar00rootroot00000000000000import torch

from torch_geometric.nn import GraphNorm
from torch_geometric.testing import is_full_test


def test_graph_norm():
    torch.manual_seed(42)
    x = torch.randn(200, 16)
    batch = torch.arange(4).view(-1, 1).repeat(1, 50).view(-1)

    norm = GraphNorm(16)
    assert str(norm) == 'GraphNorm(16)'

    if is_full_test():
        torch.jit.script(norm)

    out = norm(x)
    assert out.size() == (200, 16)
    assert torch.allclose(out.mean(dim=0), torch.zeros(16), atol=1e-6)
    assert torch.allclose(out.std(dim=0, unbiased=False), torch.ones(16),
                          atol=1e-6)

    out = norm(x, batch)
    assert out.size() == (200, 16)
    assert torch.allclose(out[:50].mean(dim=0), torch.zeros(16), atol=1e-6)
    assert torch.allclose(out[:50].std(dim=0, unbiased=False), torch.ones(16),
                          atol=1e-6)
pytorch/test/nn/norm/test_graph_size_norm.py000066400000000000000000000007471444262304300217240ustar00rootroot00000000000000import torch

from torch_geometric.nn import GraphSizeNorm
from torch_geometric.testing import is_full_test


def test_graph_size_norm():
    x = torch.randn(100, 16)
    batch = torch.repeat_interleave(torch.full((10, ), 10, dtype=torch.long))

    norm = GraphSizeNorm()
    assert str(norm) == 'GraphSizeNorm()'

    out = norm(x, batch)
    assert out.size() == (100, 16)

    if is_full_test():
        jit = torch.jit.script(norm)
        assert torch.allclose(jit(x, batch), out)
pytorch/test/nn/norm/test_instance_norm.py000066400000000000000000000030101444262304300213570ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import InstanceNorm
from torch_geometric.testing import is_full_test


@pytest.mark.parametrize('conf', [True, False])
def test_instance_norm(conf):
    batch = torch.zeros(100, dtype=torch.long)

    x1 = torch.randn(100, 16)
    x2 = torch.randn(100, 16)

    norm1 = InstanceNorm(16, affine=conf, track_running_stats=conf)
    norm2 = InstanceNorm(16, affine=conf, track_running_stats=conf)
    assert str(norm1) == 'InstanceNorm(16)'

    if is_full_test():
        torch.jit.script(norm1)

    out1 = norm1(x1)
    out2 = norm2(x1, batch)
    assert out1.size() == (100, 16)
    assert torch.allclose(out1, out2, atol=1e-7)
    if conf:
        assert torch.allclose(norm1.running_mean, norm2.running_mean)
        assert torch.allclose(norm1.running_var, norm2.running_var)

    out1 = norm1(x2)
    out2 = norm2(x2, batch)
    assert torch.allclose(out1, out2, atol=1e-7)
    if conf:
        assert torch.allclose(norm1.running_mean, norm2.running_mean)
        assert torch.allclose(norm1.running_var, norm2.running_var)

    norm1.eval()
    norm2.eval()

    out1 = norm1(x1)
    out2 = norm2(x1, batch)
    assert torch.allclose(out1, out2, atol=1e-7)

    out1 = norm1(x2)
    out2 = norm2(x2, batch)
    assert torch.allclose(out1, out2, atol=1e-7)

    out1 = norm2(x1)
    out2 = norm2(x2)
    out3 = norm2(torch.cat([x1, x2], dim=0), torch.cat([batch, batch + 1]))
    assert torch.allclose(out1, out3[:100], atol=1e-7)
    assert torch.allclose(out2, out3[100:], atol=1e-7)
pytorch/test/nn/norm/test_layer_norm.py000066400000000000000000000046371444262304300207070ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import HeteroLayerNorm, LayerNorm
from torch_geometric.testing import is_full_test, withCUDA


@withCUDA
@pytest.mark.parametrize('affine', [True, False])
@pytest.mark.parametrize('mode', ['graph', 'node'])
def test_layer_norm(device, affine, mode):
    x = torch.randn(100, 16, device=device)
    batch = torch.zeros(100, dtype=torch.long, device=device)

    norm = LayerNorm(16, affine=affine, mode=mode).to(device)
    assert str(norm) == f'LayerNorm(16, affine={affine}, mode={mode})'

    if is_full_test():
        torch.jit.script(norm)

    out1 = norm(x)
    assert out1.size() == (100, 16)
    assert torch.allclose(norm(x, batch), out1, atol=1e-6)

    out2 = norm(torch.cat([x, x], dim=0), torch.cat([batch, batch + 1], dim=0))
    assert torch.allclose(out1, out2[:100], atol=1e-6)
    assert torch.allclose(out1, out2[100:], atol=1e-6)


@withCUDA
@pytest.mark.parametrize('affine', [False, True])
def test_hetero_layer_norm(device, affine):
    x = torch.randn((100, 16), device=device)
    expected = LayerNorm(16, affine=affine, mode='node').to(device)(x)

    # Test single type:
    type_vec = torch.zeros(100, dtype=torch.long, device=device)
    type_ptr = [0, 100]

    norm = HeteroLayerNorm(16, num_types=1, affine=affine).to(device)
    assert str(norm) == 'HeteroLayerNorm(16, num_types=1)'

    out = norm(x, type_vec)
    assert out.size() == (100, 16)
    assert torch.allclose(out, expected, atol=1e-3)
    assert torch.allclose(norm(out, type_ptr=type_ptr), expected, atol=1e-3)

    mean = out.mean(dim=-1)
    std = out.std(unbiased=False, dim=-1)
    assert torch.allclose(mean, torch.zeros_like(mean), atol=1e-2)
    assert torch.allclose(std, torch.ones_like(std), atol=1e-2)

    # Test multiple types:
    type_vec = torch.arange(5, device=device)
    type_vec = type_vec.view(-1, 1).repeat(1, 20).view(-1)
    type_ptr = [0, 20, 40, 60, 80, 100]

    norm = HeteroLayerNorm(16, num_types=5, affine=affine).to(device)
    assert str(norm) == 'HeteroLayerNorm(16, num_types=5)'

    out = norm(x, type_vec)
    assert out.size() == (100, 16)
    assert torch.allclose(out, expected, atol=1e-3)
    assert torch.allclose(norm(out, type_ptr=type_ptr), expected, atol=1e-3)

    mean = out.mean(dim=-1)
    std = out.std(unbiased=False, dim=-1)
    assert torch.allclose(mean, torch.zeros_like(mean), atol=1e-2)
    assert torch.allclose(std, torch.ones_like(std), atol=1e-2)
pytorch/test/nn/norm/test_mean_subtraction_norm.py000066400000000000000000000012511444262304300231150ustar00rootroot00000000000000import torch

from torch_geometric.nn import MeanSubtractionNorm
from torch_geometric.testing import is_full_test


def test_mean_subtraction_norm():
    x = torch.randn(6, 16)
    batch = torch.tensor([0, 0, 1, 1, 1, 2])

    norm = MeanSubtractionNorm()
    assert str(norm) == 'MeanSubtractionNorm()'

    if is_full_test():
        torch.jit.script(norm)

    out = norm(x)
    assert out.size() == (6, 16)
    assert torch.allclose(out.mean(), torch.tensor(0.), atol=1e-6)

    out = norm(x, batch)
    assert out.size() == (6, 16)
    assert torch.allclose(out[0:2].mean(), torch.tensor(0.), atol=1e-6)
    assert torch.allclose(out[0:2].mean(), torch.tensor(0.), atol=1e-6)
pytorch/test/nn/norm/test_msg_norm.py000066400000000000000000000013241444262304300203470ustar00rootroot00000000000000import torch

from torch_geometric.nn import MessageNorm
from torch_geometric.testing import is_full_test


def test_message_norm():
    norm = MessageNorm(learn_scale=True)
    assert str(norm) == 'MessageNorm(learn_scale=True)'
    x = torch.randn(100, 16)
    msg = torch.randn(100, 16)
    out = norm(x, msg)
    assert out.size() == (100, 16)

    if is_full_test():
        jit = torch.jit.script(norm)
        assert torch.allclose(jit(x, msg), out)

    norm = MessageNorm(learn_scale=False)
    assert str(norm) == 'MessageNorm(learn_scale=False)'
    out = norm(x, msg)
    assert out.size() == (100, 16)

    if is_full_test():
        jit = torch.jit.script(norm)
        assert torch.allclose(jit(x, msg), out)
pytorch/test/nn/norm/test_pair_norm.py000066400000000000000000000012701444262304300205140ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import PairNorm
from torch_geometric.testing import is_full_test


@pytest.mark.parametrize('scale_individually', [False, True])
def test_pair_norm(scale_individually):
    x = torch.randn(100, 16)
    batch = torch.zeros(100, dtype=torch.long)

    norm = PairNorm(scale_individually=scale_individually)
    assert str(norm) == 'PairNorm()'

    if is_full_test():
        torch.jit.script(norm)

    out1 = norm(x)
    assert out1.size() == (100, 16)

    out2 = norm(torch.cat([x, x], dim=0), torch.cat([batch, batch + 1], dim=0))
    assert torch.allclose(out1, out2[:100], atol=1e-6)
    assert torch.allclose(out1, out2[100:], atol=1e-6)
pytorch/test/nn/pool/000077500000000000000000000000001444262304300151135ustar00rootroot00000000000000pytorch/test/nn/pool/select/000077500000000000000000000000001444262304300163725ustar00rootroot00000000000000pytorch/test/nn/pool/select/test_select_topk.py000066400000000000000000000051371444262304300223250ustar00rootroot00000000000000from itertools import product

import pytest
import torch

from torch_geometric.nn.pool.select import SelectOutput, SelectTopK
from torch_geometric.nn.pool.select.topk import topk
from torch_geometric.profile import benchmark
from torch_geometric.testing import is_full_test


def test_topk_ratio():
    x = torch.Tensor([2, 4, 5, 6, 2, 9])
    batch = torch.tensor([0, 0, 1, 1, 1, 1])

    perm1 = topk(x, 0.5, batch)
    assert perm1.tolist() == [1, 5, 3]
    assert x[perm1].tolist() == [4, 9, 6]
    assert batch[perm1].tolist() == [0, 1, 1]

    perm2 = topk(x, 2, batch)
    assert perm2.tolist() == [1, 0, 5, 3]
    assert x[perm2].tolist() == [4, 2, 9, 6]
    assert batch[perm2].tolist() == [0, 0, 1, 1]

    perm3 = topk(x, 3, batch)
    assert perm3.tolist() == [1, 0, 5, 3, 2]
    assert x[perm3].tolist() == [4, 2, 9, 6, 5]
    assert batch[perm3].tolist() == [0, 0, 1, 1, 1]

    if is_full_test():
        jit = torch.jit.script(topk)
        assert torch.equal(jit(x, 0.5, batch), perm1)
        assert torch.equal(jit(x, 2, batch), perm2)
        assert torch.equal(jit(x, 3, batch), perm3)


@pytest.mark.parametrize('min_score', [None, 2.0])
def test_select_topk(min_score):
    if min_score is not None:
        return
    x = torch.randn(6, 16)
    batch = torch.tensor([0, 0, 1, 1, 1, 1])

    pool = SelectTopK(16, min_score=min_score)

    if min_score is None:
        assert str(pool) == 'SelectTopK(16, ratio=0.5)'
    else:
        assert str(pool) == 'SelectTopK(16, min_score=2.0)'

    out = pool(x, batch)
    assert isinstance(out, SelectOutput)

    assert out.num_nodes == 6
    assert out.num_clusters <= out.num_nodes
    assert out.node_index.min() >= 0
    assert out.node_index.max() < out.num_nodes
    assert out.cluster_index.min() == 0
    assert out.cluster_index.max() == out.num_clusters - 1


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    args = parser.parse_args()

    BS = [2**i for i in range(6, 8)]
    NS = [2**i for i in range(8, 16)]

    funcs = []
    func_names = []
    args_list = []
    for B, N in product(BS, NS):
        x = torch.randn(N, device=args.device)
        batch = torch.randint(0, B, (N, ), device=args.device).sort()[0]

        funcs.append(topk)
        func_names.append(f'B={B}, N={N}')
        args_list.append((x, 0.5, batch))

    benchmark(
        funcs=funcs,
        func_names=func_names,
        args=args_list,
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
        progress_bar=True,
    )
pytorch/test/nn/pool/test_approx_knn.py000066400000000000000000000033031444262304300207020ustar00rootroot00000000000000import warnings

import torch

from torch_geometric.nn import approx_knn, approx_knn_graph
from torch_geometric.testing import onlyFullTest, withPackage


def to_set(edge_index):
    return set([(i, j) for i, j in edge_index.t().tolist()])


@onlyFullTest  # JIT compile makes this test too slow :(
@withPackage('pynndescent')
def test_approx_knn():
    warnings.filterwarnings('ignore', '.*find n_neighbors.*')

    x = torch.tensor([
        [-1.0, -1.0],
        [-1.0, +1.0],
        [+1.0, +1.0],
        [+1.0, -1.0],
        [-1.0, -1.0],
        [-1.0, +1.0],
        [+1.0, +1.0],
        [+1.0, -1.0],
    ])
    y = torch.tensor([
        [+1.0, 0.0],
        [-1.0, 0.0],
    ])

    batch_x = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])
    batch_y = torch.tensor([0, 1])

    edge_index = approx_knn(x, y, 2)
    assert to_set(edge_index) == set([(0, 2), (0, 3), (1, 0), (1, 1)])

    edge_index = approx_knn(x, y, 2, batch_x, batch_y)
    assert to_set(edge_index) == set([(0, 2), (0, 3), (1, 4), (1, 5)])


@onlyFullTest  # JIT compile makes this test too slow :(
@withPackage('pynndescent')
def test_approx_knn_graph():
    warnings.filterwarnings('ignore', '.*find n_neighbors.*')

    x = torch.tensor([
        [-1.0, -1.0],
        [-1.0, +1.0],
        [+1.0, +1.0],
        [+1.0, -1.0],
    ])

    edge_index = approx_knn_graph(x, k=2, flow='target_to_source')
    assert to_set(edge_index) == set([(0, 1), (0, 3), (1, 0), (1, 2), (2, 1),
                                      (2, 3), (3, 0), (3, 2)])

    edge_index = approx_knn_graph(x, k=2, flow='source_to_target')
    assert to_set(edge_index) == set([(1, 0), (3, 0), (0, 1), (2, 1), (1, 2),
                                      (3, 2), (0, 3), (2, 3)])
pytorch/test/nn/pool/test_asap.py000066400000000000000000000033211444262304300174470ustar00rootroot00000000000000import io

import pytest
import torch

from torch_geometric.nn import ASAPooling, GCNConv, GraphConv
from torch_geometric.testing import is_full_test, onlyFullTest, onlyLinux


@onlyLinux  # TODO  (matthias) Investigate CSR @ CSR support on Windows.
def test_asap():
    in_channels = 16
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]])
    num_nodes = edge_index.max().item() + 1
    x = torch.randn((num_nodes, in_channels))

    for GNN in [GraphConv, GCNConv]:
        pool = ASAPooling(in_channels, ratio=0.5, GNN=GNN,
                          add_self_loops=False)
        assert str(pool) == ('ASAPooling(16, ratio=0.5)')
        out = pool(x, edge_index)
        assert out[0].size() == (num_nodes // 2, in_channels)
        assert out[1].size() == (2, 2)

        if is_full_test():
            torch.jit.script(pool.jittable())

        pool = ASAPooling(in_channels, ratio=0.5, GNN=GNN, add_self_loops=True)
        assert str(pool) == ('ASAPooling(16, ratio=0.5)')
        out = pool(x, edge_index)
        assert out[0].size() == (num_nodes // 2, in_channels)
        assert out[1].size() == (2, 4)

        pool = ASAPooling(in_channels, ratio=2, GNN=GNN, add_self_loops=False)
        assert str(pool) == ('ASAPooling(16, ratio=2)')
        out = pool(x, edge_index)
        assert out[0].size() == (2, in_channels)
        assert out[1].size() == (2, 2)


@onlyFullTest
def test_asap_jit_save():
    pool = ASAPooling(in_channels=16)
    pool_jit = pool.jittable()
    model = torch.jit.script(pool_jit)
    try:
        torch.jit.save(model, io.BytesIO())
    except RuntimeError:
        pytest.fail('ASAP model serialization failed.')
pytorch/test/nn/pool/test_avg_pool.py000066400000000000000000000053341444262304300203370ustar00rootroot00000000000000import torch

from torch_geometric.data import Batch
from torch_geometric.nn import avg_pool, avg_pool_neighbor_x, avg_pool_x
from torch_geometric.testing import is_full_test


def test_avg_pool_x():
    cluster = torch.tensor([0, 1, 0, 1, 2, 2])
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    out = avg_pool_x(cluster, x, batch)
    assert out[0].tolist() == [[3, 4], [5, 6], [10, 11]]
    assert out[1].tolist() == [0, 0, 1]

    if is_full_test():
        jit = torch.jit.script(avg_pool_x)
        out = jit(cluster, x, batch)
        assert out[0].tolist() == [[3, 4], [5, 6], [10, 11]]
        assert out[1].tolist() == [0, 0, 1]

    out, _ = avg_pool_x(cluster, x, batch, size=2)
    assert out.tolist() == [[3, 4], [5, 6], [10, 11], [0, 0]]

    batch_size = int(batch.max().item()) + 1
    out2, _ = avg_pool_x(cluster, x, batch, batch_size=batch_size, size=2)
    assert torch.equal(out, out2)

    if is_full_test():
        jit = torch.jit.script(avg_pool_x)
        out, _ = jit(cluster, x, batch, size=2)
        assert out.tolist() == [[3, 4], [5, 6], [10, 11], [0, 0]]

        out2, _ = jit(cluster, x, batch, batch_size=batch_size, size=2)
        assert torch.equal(out, out2)


def test_avg_pool():
    cluster = torch.tensor([0, 1, 0, 1, 2, 2])
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    pos = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    edge_attr = torch.Tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    data = Batch(x=x, pos=pos, edge_index=edge_index, edge_attr=edge_attr,
                 batch=batch)

    data = avg_pool(cluster, data, transform=lambda x: x)

    assert data.x.tolist() == [[3, 4], [5, 6], [10, 11]]
    assert data.pos.tolist() == [[1, 1], [2, 2], [4.5, 4.5]]
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]
    assert data.edge_attr.tolist() == [4, 4]
    assert data.batch.tolist() == [0, 0, 1]


def test_avg_pool_neighbor_x():
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    data = Batch(x=x, edge_index=edge_index, batch=batch)
    data = avg_pool_neighbor_x(data)

    assert data.x.tolist() == [
        [4, 5],
        [4, 5],
        [4, 5],
        [4, 5],
        [10, 11],
        [10, 11],
    ]
    assert torch.equal(data.edge_index, edge_index)
pytorch/test/nn/pool/test_consecutive.py000066400000000000000000000004621444262304300210550ustar00rootroot00000000000000import torch

from torch_geometric.nn.pool.consecutive import consecutive_cluster


def test_consecutive_cluster():
    src = torch.tensor([8, 2, 10, 15, 100, 1, 100])

    out, perm = consecutive_cluster(src)
    assert out.tolist() == [2, 1, 3, 4, 5, 0, 5]
    assert perm.tolist() == [5, 1, 0, 2, 3, 6]
pytorch/test/nn/pool/test_decimation.py000066400000000000000000000021731444262304300206430ustar00rootroot00000000000000import torch

from torch_geometric.nn.pool.decimation import decimation_indices


def test_decimation_basic():
    N_1, N_2 = 4, 6
    decimation_factor = 2
    ptr = torch.tensor([0, N_1, N_1 + N_2])

    idx_decim, ptr_decim = decimation_indices(ptr, decimation_factor)

    expected_size = (N_1 // decimation_factor) + (N_2 // decimation_factor)
    assert idx_decim.size(0) == expected_size

    expected = torch.tensor([0, N_1 // decimation_factor, expected_size])
    assert torch.equal(ptr_decim, expected)


def test_decimation_single_cloud():
    N_1 = 4
    decimation_factor = 2
    ptr = torch.tensor([0, N_1])

    idx_decim, ptr_decim = decimation_indices(ptr, decimation_factor)

    expected_size = N_1 // decimation_factor
    assert idx_decim.size(0) == expected_size
    assert torch.equal(ptr_decim, torch.tensor([0, expected_size]))


def test_decimation_almost_empty():
    N_1 = 4
    decimation_factor = 666  # greater than N_1
    ptr = torch.tensor([0, N_1])

    idx_decim, ptr_decim = decimation_indices(ptr, decimation_factor)

    assert idx_decim.size(0) == 1
    assert torch.equal(ptr_decim, torch.tensor([0, 1]))
pytorch/test/nn/pool/test_edge_pool.py000066400000000000000000000077111444262304300204670ustar00rootroot00000000000000import torch

from torch_geometric.nn import EdgePooling
from torch_geometric.testing import is_full_test
from torch_geometric.utils import scatter


def test_compute_edge_score_softmax():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    raw = torch.randn(edge_index.size(1))
    e = EdgePooling.compute_edge_score_softmax(raw, edge_index, 6)
    assert torch.all(e >= 0) and torch.all(e <= 1)

    # Test whether all incoming edge scores sum up to one.
    assert torch.allclose(scatter(e, edge_index[1], reduce='sum'),
                          torch.Tensor([1, 1, 1, 1, 1, 1]))

    if is_full_test():
        jit = torch.jit.script(EdgePooling.compute_edge_score_softmax)
        assert torch.allclose(jit(raw, edge_index, 6), e)


def test_compute_edge_score_tanh():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    raw = torch.randn(edge_index.size(1))
    e = EdgePooling.compute_edge_score_tanh(raw, edge_index, 6)
    assert torch.all(e >= -1) and torch.all(e <= 1)
    assert torch.all(torch.argsort(raw) == torch.argsort(e))

    if is_full_test():
        jit = torch.jit.script(EdgePooling.compute_edge_score_tanh)
        assert torch.allclose(jit(raw, edge_index, 6), e)


def test_compute_edge_score_sigmoid():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    raw = torch.randn(edge_index.size(1))
    e = EdgePooling.compute_edge_score_sigmoid(raw, edge_index, 6)
    assert torch.all(e >= 0) and torch.all(e <= 1)
    assert torch.all(torch.argsort(raw) == torch.argsort(e))

    if is_full_test():
        jit = torch.jit.script(EdgePooling.compute_edge_score_sigmoid)
        assert torch.allclose(jit(raw, edge_index, 6), e)


def test_edge_pooling():
    x = torch.Tensor([[0], [1], [2], [3], [4], [5], [-1]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5, 6],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4, 0]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 0])

    op = EdgePooling(in_channels=1)
    assert str(op) == 'EdgePooling(1)'

    # Setting parameters fixed so we can test the expected outcome:
    op.lin.weight.data.fill_(1.)
    op.lin.bias.data.fill_(0.)

    # Test pooling:
    new_x, new_edge_index, new_batch, unpool_info = op(x, edge_index, batch)

    assert new_x.size(0) == new_batch.size(0) == 4
    assert new_edge_index.tolist() == [[0, 1, 1, 2, 2, 3], [0, 1, 2, 1, 2, 2]]
    assert new_batch.tolist() == [1, 0, 0, 0]

    if is_full_test():
        jit = torch.jit.script(op)
        out = jit(x, edge_index, batch)
        assert torch.allclose(new_x, out[0])
        assert torch.equal(new_edge_index, out[1])
        assert torch.equal(new_batch, out[2])

    # Test unpooling:
    out = op.unpool(new_x, unpool_info)
    assert out[0].size() == x.size()
    assert out[0].tolist() == [[1], [1], [5], [5], [9], [9], [-1]]
    assert torch.equal(out[1], edge_index)
    assert torch.equal(out[2], batch)

    if is_full_test():
        jit = torch.jit.export(op)
        out = jit.unpool(new_x, unpool_info)
        assert out[0].size() == x.size()
        assert out[0].tolist() == [[1], [1], [5], [5], [9], [9], [-1]]
        assert torch.equal(out[1], edge_index)
        assert torch.equal(out[2], batch)

    # Test edge cases.
    x = torch.Tensor([[0], [1], [2], [3], [4], [5]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])
    new_x, new_edge_index, new_batch, _ = op(x, edge_index, batch)

    assert new_x.size(0) == new_batch.size(0) == 3
    assert new_batch.tolist() == [1, 0, 0]
    assert new_edge_index.tolist() == [[0, 1, 1, 2, 2], [0, 1, 2, 1, 2]]
pytorch/test/nn/pool/test_glob.py000066400000000000000000000041211444262304300174450ustar00rootroot00000000000000import torch

from torch_geometric.nn import (
    global_add_pool,
    global_max_pool,
    global_mean_pool,
)


def test_global_pool():
    N_1, N_2 = 4, 6
    x = torch.randn(N_1 + N_2, 4)
    batch = torch.tensor([0 for _ in range(N_1)] + [1 for _ in range(N_2)])

    out = global_add_pool(x, batch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], x[:4].sum(dim=0))
    assert torch.allclose(out[1], x[4:].sum(dim=0))

    out = global_add_pool(x, None)
    assert out.size() == (1, 4)
    assert torch.allclose(out, x.sum(dim=0, keepdim=True))

    out = global_mean_pool(x, batch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], x[:4].mean(dim=0))
    assert torch.allclose(out[1], x[4:].mean(dim=0))

    out = global_mean_pool(x, None)
    assert out.size() == (1, 4)
    assert torch.allclose(out, x.mean(dim=0, keepdim=True))

    out = global_max_pool(x, batch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], x[:4].max(dim=0)[0])
    assert torch.allclose(out[1], x[4:].max(dim=0)[0])

    out = global_max_pool(x, None)
    assert out.size() == (1, 4)
    assert torch.allclose(out, x.max(dim=0, keepdim=True)[0])


def test_permuted_global_pool():
    N_1, N_2 = 4, 6
    x = torch.randn(N_1 + N_2, 4)
    batch = torch.cat([torch.zeros(N_1), torch.ones(N_2)]).to(torch.long)
    perm = torch.randperm(N_1 + N_2)

    px = x[perm]
    pbatch = batch[perm]
    px1 = px[pbatch == 0]
    px2 = px[pbatch == 1]

    out = global_add_pool(px, pbatch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], px1.sum(dim=0))
    assert torch.allclose(out[1], px2.sum(dim=0))

    out = global_mean_pool(px, pbatch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], px1.mean(dim=0))
    assert torch.allclose(out[1], px2.mean(dim=0))

    out = global_max_pool(px, pbatch)
    assert out.size() == (2, 4)
    assert torch.allclose(out[0], px1.max(dim=0)[0])
    assert torch.allclose(out[1], px2.max(dim=0)[0])


def test_dense_global_pool():
    x = torch.randn(3, 16, 32)
    assert torch.allclose(global_add_pool(x, None), x.sum(dim=1))
pytorch/test/nn/pool/test_graclus.py000066400000000000000000000003731444262304300201670ustar00rootroot00000000000000import torch

from torch_geometric.nn import graclus
from torch_geometric.testing import withPackage


@withPackage('torch_cluster')
def test_graclus():
    edge_index = torch.tensor([[0, 1], [1, 0]])
    assert graclus(edge_index).tolist() == [0, 0]
pytorch/test/nn/pool/test_max_pool.py000066400000000000000000000053341444262304300203470ustar00rootroot00000000000000import torch

from torch_geometric.data import Batch
from torch_geometric.nn import max_pool, max_pool_neighbor_x, max_pool_x
from torch_geometric.testing import is_full_test


def test_max_pool_x():
    cluster = torch.tensor([0, 1, 0, 1, 2, 2])
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    out = max_pool_x(cluster, x, batch)
    assert out[0].tolist() == [[5, 6], [7, 8], [11, 12]]
    assert out[1].tolist() == [0, 0, 1]

    if is_full_test():
        jit = torch.jit.script(max_pool_x)
        out = jit(cluster, x, batch)
        assert out[0].tolist() == [[5, 6], [7, 8], [11, 12]]
        assert out[1].tolist() == [0, 0, 1]

    out, _ = max_pool_x(cluster, x, batch, size=2)
    assert out.tolist() == [[5, 6], [7, 8], [11, 12], [0, 0]]

    batch_size = int(batch.max().item()) + 1
    out2, _ = max_pool_x(cluster, x, batch, batch_size=batch_size, size=2)
    assert torch.equal(out, out2)

    if is_full_test():
        jit = torch.jit.script(max_pool_x)
        out, _ = jit(cluster, x, batch, size=2)
        assert out.tolist() == [[5, 6], [7, 8], [11, 12], [0, 0]]

        out2, _ = jit(cluster, x, batch, batch_size=batch_size, size=2)
        assert torch.equal(out, out2)


def test_max_pool():
    cluster = torch.tensor([0, 1, 0, 1, 2, 2])
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    pos = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    edge_attr = torch.Tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    data = Batch(x=x, pos=pos, edge_index=edge_index, edge_attr=edge_attr,
                 batch=batch)

    data = max_pool(cluster, data, transform=lambda x: x)

    assert data.x.tolist() == [[5, 6], [7, 8], [11, 12]]
    assert data.pos.tolist() == [[1, 1], [2, 2], [4.5, 4.5]]
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]
    assert data.edge_attr.tolist() == [4, 4]
    assert data.batch.tolist() == [0, 0, 1]


def test_max_pool_neighbor_x():
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    batch = torch.tensor([0, 0, 0, 0, 1, 1])

    data = Batch(x=x, edge_index=edge_index, batch=batch)
    data = max_pool_neighbor_x(data)

    assert data.x.tolist() == [
        [7, 8],
        [7, 8],
        [7, 8],
        [7, 8],
        [11, 12],
        [11, 12],
    ]
    assert torch.equal(data.edge_index, edge_index)
pytorch/test/nn/pool/test_mem_pool.py000066400000000000000000000014421444262304300203340ustar00rootroot00000000000000import torch

from torch_geometric.nn import MemPooling
from torch_geometric.utils import to_dense_batch


def test_mem_pool():
    mpool1 = MemPooling(4, 8, heads=3, num_clusters=2)
    assert str(mpool1) == 'MemPooling(4, 8, heads=3, num_clusters=2)'
    mpool2 = MemPooling(8, 4, heads=2, num_clusters=1)

    x = torch.randn(17, 4)
    batch = torch.tensor([0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 4, 4, 4])
    _, mask = to_dense_batch(x, batch)

    out1, S = mpool1(x, batch)
    loss = MemPooling.kl_loss(S)
    with torch.autograd.set_detect_anomaly(True):
        loss.backward()
    out2, _ = mpool2(out1)

    assert out1.size() == (5, 2, 8)
    assert out2.size() == (5, 1, 4)
    assert S[~mask].sum() == 0
    assert round(S[mask].sum().item()) == x.size(0)
    assert float(loss) > 0
pytorch/test/nn/pool/test_pan_pool.py000066400000000000000000000022251444262304300203340ustar00rootroot00000000000000import torch

from torch_geometric.nn import PANConv, PANPooling
from torch_geometric.testing import is_full_test, withPackage


@withPackage('torch_sparse')
def test_pan_pooling():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]])
    num_nodes = edge_index.max().item() + 1
    x = torch.randn((num_nodes, 16))

    conv = PANConv(16, 32, filter_size=2)
    pool = PANPooling(32, ratio=0.5)
    assert str(pool) == 'PANPooling(32, ratio=0.5, multiplier=1.0)'

    x, M = conv(x, edge_index)
    h, edge_index, edge_weight, batch, perm, score = pool(x, M)

    assert h.size() == (2, 32)
    assert edge_index.size() == (2, 4)
    assert edge_weight.size() == (4, )
    assert perm.size() == (2, )
    assert score.size() == (2, )

    if is_full_test():
        jit = torch.jit.script(pool)
        out = jit(x, M)
        assert torch.allclose(h, out[0])
        assert torch.equal(edge_index, out[1])
        assert torch.allclose(edge_weight, out[2])
        assert torch.equal(batch, out[3])
        assert torch.equal(perm, out[4])
        assert torch.allclose(score, out[5])
pytorch/test/nn/pool/test_pool.py000066400000000000000000000012211444262304300174710ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.nn import radius_graph
from torch_geometric.testing import onlyFullTest, withPackage


@onlyFullTest
@withPackage('torch_cluster')
def test_radius_graph_jit():
    class Net(torch.nn.Module):
        def forward(self, x: Tensor, batch: Optional[Tensor] = None) -> Tensor:
            return radius_graph(x, r=2.5, batch=batch, loop=False)

    x = torch.tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]], dtype=torch.float)
    batch = torch.tensor([0, 0, 0, 0])

    model = Net()
    jit = torch.jit.script(model)
    assert model(x, batch).size() == jit(x, batch).size()
pytorch/test/nn/pool/test_sag_pool.py000066400000000000000000000037501444262304300203340ustar00rootroot00000000000000import torch

from torch_geometric.nn import (
    GATConv,
    GCNConv,
    GraphConv,
    SAGEConv,
    SAGPooling,
)
from torch_geometric.testing import is_full_test


def test_sag_pooling():
    in_channels = 16
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]])
    num_nodes = edge_index.max().item() + 1
    x = torch.randn((num_nodes, in_channels))

    for GNN in [GraphConv, GCNConv, GATConv, SAGEConv]:
        pool1 = SAGPooling(in_channels, ratio=0.5, GNN=GNN)
        assert str(pool1) == (f'SAGPooling({GNN.__name__}, 16, '
                              f'ratio=0.5, multiplier=1.0)')
        out1 = pool1(x, edge_index)
        assert out1[0].size() == (num_nodes // 2, in_channels)
        assert out1[1].size() == (2, 2)

        pool2 = SAGPooling(in_channels, ratio=None, GNN=GNN, min_score=0.1)
        assert str(pool2) == (f'SAGPooling({GNN.__name__}, 16, '
                              f'min_score=0.1, multiplier=1.0)')
        out2 = pool2(x, edge_index)
        assert out2[0].size(0) <= x.size(0) and out2[0].size(1) == (16)
        assert out2[1].size(0) == 2 and out2[1].size(1) <= edge_index.size(1)

        pool3 = SAGPooling(in_channels, ratio=2, GNN=GNN)
        assert str(pool3) == (f'SAGPooling({GNN.__name__}, 16, '
                              f'ratio=2, multiplier=1.0)')
        out3 = pool3(x, edge_index)
        assert out3[0].size() == (2, in_channels)
        assert out3[1].size() == (2, 2)

        if is_full_test():
            pool1.gnn = pool1.gnn.jittable()
            jit1 = torch.jit.script(pool1)
            assert torch.allclose(jit1(x, edge_index)[0], out1[0])

            pool2.gnn = pool2.gnn.jittable()
            jit2 = torch.jit.script(pool2)
            assert torch.allclose(jit2(x, edge_index)[0], out2[0])

            pool3.gnn = pool3.gnn.jittable()
            jit3 = torch.jit.script(pool3)
            assert torch.allclose(jit3(x, edge_index)[0], out3[0])
pytorch/test/nn/pool/test_topk_pool.py000066400000000000000000000041261444262304300205350ustar00rootroot00000000000000import torch

from torch_geometric.nn.pool import TopKPooling
from torch_geometric.nn.pool.topk_pool import filter_adj
from torch_geometric.testing import is_full_test


def test_filter_adj():
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 3],
                               [1, 3, 0, 2, 1, 3, 0, 2]])
    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6, 7, 8])
    perm = torch.tensor([2, 3])

    out = filter_adj(edge_index, edge_attr, perm)
    assert out[0].tolist() == [[0, 1], [1, 0]]
    assert out[1].tolist() == [6, 8]

    if is_full_test():
        jit = torch.jit.script(filter_adj)

        out = jit(edge_index, edge_attr, perm)
        assert out[0].tolist() == [[0, 1], [1, 0]]
        assert out[1].tolist() == [6, 8]


def test_topk_pooling():
    in_channels = 16
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]])
    num_nodes = edge_index.max().item() + 1
    x = torch.randn((num_nodes, in_channels))

    pool1 = TopKPooling(in_channels, ratio=0.5)
    assert str(pool1) == 'TopKPooling(16, ratio=0.5, multiplier=1.0)'
    out1 = pool1(x, edge_index)
    assert out1[0].size() == (num_nodes // 2, in_channels)
    assert out1[1].size() == (2, 2)

    pool2 = TopKPooling(in_channels, ratio=None, min_score=0.1)
    assert str(pool2) == 'TopKPooling(16, min_score=0.1, multiplier=1.0)'
    out2 = pool2(x, edge_index)
    assert out2[0].size(0) <= x.size(0) and out2[0].size(1) == (16)
    assert out2[1].size(0) == 2 and out2[1].size(1) <= edge_index.size(1)

    pool3 = TopKPooling(in_channels, ratio=2)
    assert str(pool3) == 'TopKPooling(16, ratio=2, multiplier=1.0)'
    out3 = pool3(x, edge_index)
    assert out3[0].size() == (2, in_channels)
    assert out3[1].size() == (2, 2)

    if is_full_test():
        jit1 = torch.jit.script(pool1)
        assert torch.allclose(jit1(x, edge_index)[0], out1[0])

        jit2 = torch.jit.script(pool2)
        assert torch.allclose(jit2(x, edge_index)[0], out2[0])

        jit3 = torch.jit.script(pool3)
        assert torch.allclose(jit3(x, edge_index)[0], out3[0])
pytorch/test/nn/pool/test_voxel_grid.py000066400000000000000000000025611444262304300206720ustar00rootroot00000000000000import torch

from torch_geometric.data import Batch
from torch_geometric.nn import avg_pool, voxel_grid
from torch_geometric.testing import withPackage


@withPackage('torch_cluster')
def test_voxel_grid():
    pos = torch.Tensor([[0, 0], [11, 9], [2, 8], [2, 2], [8, 3]])
    batch = torch.tensor([0, 0, 0, 1, 1])

    assert voxel_grid(pos, size=5, batch=batch).tolist() == [0, 5, 3, 6, 7]
    assert voxel_grid(pos, size=5).tolist() == [0, 5, 3, 0, 1]

    cluster = voxel_grid(pos, size=5, batch=batch, start=-1, end=[18, 14])
    assert cluster.tolist() == [0, 10, 4, 16, 17]

    cluster_no_batch = voxel_grid(pos, size=5, start=-1, end=[18, 14])
    assert cluster_no_batch.tolist() == [0, 10, 4, 0, 1]


@withPackage('torch_cluster')
def test_single_voxel_grid():
    pos = torch.Tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]])
    edge_index = torch.tensor([[0, 0, 3], [1, 2, 4]])
    batch = torch.tensor([0, 0, 0, 1, 1])
    x = torch.randn(5, 16)

    cluster = voxel_grid(pos, size=5, batch=batch)
    assert cluster.tolist() == [0, 0, 0, 1, 1]

    data = Batch(x=x, edge_index=edge_index, pos=pos, batch=batch)
    data = avg_pool(cluster, data)

    cluster_no_batch = voxel_grid(pos, size=5)
    assert cluster_no_batch.tolist() == [0, 0, 0, 0, 0]

    data_no_batch = Batch(x=x, edge_index=edge_index, pos=pos)
    data_no_batch = avg_pool(cluster_no_batch, data_no_batch)
pytorch/test/nn/test_compile_basic.py000066400000000000000000000124171444262304300203510ustar00rootroot00000000000000import torch

import torch_geometric
from torch_geometric.profile import benchmark
from torch_geometric.testing import (
    disableExtensions,
    onlyFullTest,
    onlyLinux,
    withCUDA,
    withPackage,
)
from torch_geometric.utils import scatter


# Basic "Gather-Apply-Scatter" patterns commonly used in PyG:
def gather_scatter(x, edge_index, reduce='sum'):
    row, col = edge_index
    x_j = x[row]
    return scatter(x_j, col, dim_size=x.size(0), reduce=reduce)


def gather_cat_scatter(x, edge_index, reduce='sum'):
    row, col = edge_index
    x_ij = torch.cat([x[col], x[row]], dim=-1)
    return scatter(x_ij, col, dim_size=x.size(0), reduce=reduce)


def gather_weight_scatter(x, edge_index, edge_weight, reduce='sum'):
    row, col = edge_index
    x_j = x[row] * edge_weight.view(-1, 1)
    return scatter(x_j, col, dim_size=x.size(0), reduce=reduce)


def gather_transform_scatter(x, edge_index, matrix, reduce='sum'):
    row, col = edge_index
    x_j = x[row] @ matrix
    return scatter(x_j, col, dim_size=x.size(0), reduce=reduce)


def fused_gather_scatter(x, edge_index, reduce=['sum', 'mean', 'max']):
    row, col = edge_index
    x_j = x[row]
    outs = [scatter(x_j, col, dim_size=x.size(0), reduce=r) for r in reduce]
    return torch.cat(outs, dim=-1)


@withCUDA
@onlyLinux
@onlyFullTest
@disableExtensions
@withPackage('torch>=2.0.0')
def test_torch_compile(device):
    x = torch.randn(10, 16, device=device)
    edge_index = torch.randint(0, x.size(0), (2, 40), device=device)
    edge_weight = torch.rand(edge_index.size(1), device=device)
    matrix = torch.randn(x.size(-1), x.size(-1), device=device)

    expected = gather_scatter(x, edge_index)
    compiled_op = torch_geometric.compile(gather_scatter)
    out = compiled_op(x, edge_index)
    assert torch.allclose(out, expected, atol=1e-6)

    expected = gather_cat_scatter(x, edge_index)
    compiled_op = torch_geometric.compile(gather_cat_scatter)
    out = compiled_op(x, edge_index)
    assert torch.allclose(out, expected, atol=1e-6)

    expected = gather_weight_scatter(x, edge_index, edge_weight)
    compiled_op = torch_geometric.compile(gather_weight_scatter)
    out = compiled_op(x, edge_index, edge_weight)
    assert torch.allclose(out, expected, atol=1e-6)

    expected = gather_transform_scatter(x, edge_index, matrix)
    compiled_op = torch_geometric.compile(gather_transform_scatter)
    out = compiled_op(x, edge_index, matrix)
    assert torch.allclose(out, expected, atol=1e-6)

    expected = fused_gather_scatter(x, edge_index)
    compiled_op = torch_geometric.compile(fused_gather_scatter)
    out = compiled_op(x, edge_index)
    assert torch.allclose(out, expected, atol=1e-6)


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    x = torch.randn(num_nodes, 64, device=args.device)
    edge_index = torch.randint(num_nodes, (2, num_edges), device=args.device)
    edge_weight = torch.rand(num_edges, device=args.device)
    matrix = torch.randn(64, 64, device=args.device)

    for reduce in ['sum', 'mean', 'max']:
        print(f'Aggregator: {reduce}')

        benchmark(
            funcs=[
                gather_scatter,
                torch_geometric.compile(gather_scatter),
            ],
            func_names=['Vanilla', 'Compiled'],
            args=(x, edge_index, reduce),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )

        benchmark(
            funcs=[
                gather_cat_scatter,
                torch_geometric.compile(gather_cat_scatter),
            ],
            func_names=['Vanilla Cat', 'Compiled Cat'],
            args=(x, edge_index, reduce),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )

        benchmark(
            funcs=[
                gather_weight_scatter,
                torch_geometric.compile(gather_weight_scatter),
            ],
            func_names=['Vanilla Weight', 'Compiled Weight'],
            args=(x, edge_index, edge_weight, reduce),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )

        benchmark(
            funcs=[
                gather_transform_scatter,
                torch_geometric.compile(gather_transform_scatter),
            ],
            func_names=['Vanilla Transform', 'Compiled Transform'],
            args=(x, edge_index, matrix, reduce),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )

    benchmark(
        funcs=[
            fused_gather_scatter,
            torch_geometric.compile(fused_gather_scatter),
        ],
        func_names=['Vanilla Fused', 'Compiled Fused'],
        args=(x, edge_index),
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
        backward=args.backward,
    )
pytorch/test/nn/test_compile_conv.py000066400000000000000000000047121444262304300202340ustar00rootroot00000000000000import pytest
import torch
from torch import Tensor

import torch_geometric
from torch_geometric.nn import GCNConv, SAGEConv
from torch_geometric.profile import benchmark
from torch_geometric.testing import (
    disableExtensions,
    onlyFullTest,
    onlyLinux,
    withCUDA,
    withPackage,
)
from torch_geometric.utils import scatter


class MySAGEConv(torch.nn.Module):
    def __init__(self, in_channels: int, out_channels: int):
        super().__init__()
        self.lin_src = torch.nn.Linear(in_channels, out_channels)
        self.lin_dst = torch.nn.Linear(in_channels, out_channels)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x_j = x[edge_index[0]]
        out = scatter(x_j, edge_index[1], dim_size=x.size(0), reduce='mean')
        return self.lin_src(out) + self.lin_dst(x)


@withCUDA
@onlyLinux
@onlyFullTest
@disableExtensions
@withPackage('torch>=2.0.0')
@pytest.mark.parametrize('Conv', [GCNConv, SAGEConv])
def test_compile_conv(device, Conv):
    x = torch.randn(10, 16, device=device)
    edge_index = torch.randint(0, x.size(0), (2, 40), device=device)

    conv = Conv(16, 32).to(device)
    out = torch_geometric.compile(conv)(x, edge_index)
    assert torch.allclose(conv(x, edge_index), out, atol=1e-6)


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    x = torch.randn(num_nodes, 64, device=args.device)
    edge_index = torch.randint(num_nodes, (2, num_edges), device=args.device)

    conv = MySAGEConv(64, 64).to(args.device)
    benchmark(
        funcs=[conv, torch_geometric.compile(conv)],
        func_names=['Vanilla', 'Compiled'],
        args=(x, edge_index),
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
        backward=args.backward,
    )

    for Conv in [GCNConv, SAGEConv]:
        print(f'Conv: {Conv.__name__}')

        conv = Conv(64, 64).to(args.device)
        compiled_conv = torch_geometric.compile(conv)

        benchmark(
            funcs=[conv, compiled_conv],
            func_names=['Vanilla', 'Compiled'],
            args=(x, edge_index),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )
pytorch/test/nn/test_compile_dynamic.py000066400000000000000000000025501444262304300207110ustar00rootroot00000000000000import random

import torch
from torch import Tensor

import torch_geometric
from torch_geometric.testing import (
    disableExtensions,
    get_random_edge_index,
    onlyFullTest,
    onlyLinux,
    withCUDA,
    withPackage,
)
from torch_geometric.utils import scatter


class MySAGEConv(torch.nn.Module):
    def __init__(self, in_channels: int, out_channels: int):
        super().__init__()
        self.lin_src = torch.nn.Linear(in_channels, out_channels)
        self.lin_dst = torch.nn.Linear(in_channels, out_channels)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x_j = x[edge_index[0]]
        out = scatter(x_j, edge_index[1], dim_size=x.size(0), reduce='mean')
        return self.lin_src(out) + self.lin_dst(x)


@withCUDA
@onlyLinux
@onlyFullTest
@disableExtensions
@withPackage('torch>2.0.0')
def test_dynamic_torch_compile(device):
    conv = MySAGEConv(64, 64).to(device)
    conv = torch_geometric.compile(conv, dynamic=True)

    optimizer = torch.optim.Adam(conv.parameters(), lr=0.01)

    for _ in range(10):
        N = random.randrange(100, 500)
        E = random.randrange(200, 1000)

        x = torch.randn(N, 64, device=device)
        edge_index = get_random_edge_index(N, N, E, device=device)

        optimizer.zero_grad()
        expected = conv(x, edge_index)
        expected.mean().backward()
        optimizer.step()
pytorch/test/nn/test_data_parallel.py000066400000000000000000000007201444262304300203370ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.nn import DataParallel
from torch_geometric.testing import onlyCUDA


@onlyCUDA
@pytest.mark.skipif(torch.cuda.device_count() < 2, reason='No multiple GPUs')
def test_data_parallel():
    module = DataParallel(None)
    data_list = [Data(x=torch.randn(x, 1)) for x in [2, 3, 10, 4]]
    batches = module.scatter(data_list, device_ids=[0, 1, 0, 1])
    assert len(batches) == 3
pytorch/test/nn/test_encoding.py000066400000000000000000000011411444262304300173360ustar00rootroot00000000000000import torch

from torch_geometric.nn import PositionalEncoding, TemporalEncoding
from torch_geometric.testing import withCUDA


@withCUDA
def test_positional_encoding(device):
    encoder = PositionalEncoding(64).to(device)
    assert str(encoder) == 'PositionalEncoding(64)'

    x = torch.tensor([1.0, 2.0, 3.0], device=device)
    assert encoder(x).size() == (3, 64)


@withCUDA
def test_temporal_encoding(device):
    encoder = TemporalEncoding(64).to(device)
    assert str(encoder) == 'TemporalEncoding(64)'

    x = torch.tensor([1.0, 2.0, 3.0], device=device)
    assert encoder(x).size() == (3, 64)
pytorch/test/nn/test_inits.py000066400000000000000000000024561444262304300167100ustar00rootroot00000000000000import torch
from torch.nn import Linear as Lin
from torch.nn import ReLU
from torch.nn import Sequential as Seq

from torch_geometric.nn.inits import (
    glorot,
    glorot_orthogonal,
    ones,
    reset,
    uniform,
    zeros,
)


def test_inits():
    x = torch.empty(1, 4)

    uniform(size=4, value=x)
    assert x.min() >= -0.5
    assert x.max() <= 0.5

    glorot(x)
    assert x.min() >= -1.1
    assert x.max() <= 1.1

    glorot_orthogonal(x, scale=1.0)
    assert x.min() >= -2.5
    assert x.max() <= 2.5

    zeros(x)
    assert x.tolist() == [[0, 0, 0, 0]]

    ones(x)
    assert x.tolist() == [[1, 1, 1, 1]]

    nn = Lin(16, 16)
    uniform(size=4, value=nn.weight)
    assert nn.weight[0].min() >= -0.5
    assert nn.weight[0].max() <= 0.5

    glorot(nn.weight)
    assert nn.weight[0].min() >= -0.45
    assert nn.weight[0].max() <= 0.45

    glorot_orthogonal(nn.weight, scale=1.0)
    assert nn.weight[0].min() >= -2.5
    assert nn.weight[0].max() <= 2.5


def test_reset():
    nn = Lin(16, 16)
    w = nn.weight.clone()
    reset(nn)
    assert not torch.allclose(nn.weight, w)

    nn = Seq(Lin(16, 16), ReLU(), Lin(16, 16))
    w_1, w_2 = nn[0].weight.clone(), nn[2].weight.clone()
    reset(nn)
    assert not torch.allclose(nn[0].weight, w_1)
    assert not torch.allclose(nn[2].weight, w_2)
pytorch/test/nn/test_model_hub.py000066400000000000000000000067061444262304300175220ustar00rootroot00000000000000import os
from pathlib import Path
from unittest.mock import Mock

import pytest
import torch

from torch_geometric.nn import GCN
from torch_geometric.nn.model_hub import PyGModelHubMixin
from torch_geometric.testing import withPackage

REPO_NAME = 'pyg_hugging_test'
MODEL_NAME = 'pyg_test_model'
DATASET_NAME = 'pyg_dataset'
CONFIG = {'hello': 'world'}


class DummyModel(GCN, PyGModelHubMixin):
    def __init__(self, model_name, dataset_name, model_kwargs):
        GCN.__init__(self, in_channels=3, hidden_channels=5, num_layers=2)
        PyGModelHubMixin.__init__(self, model_name, dataset_name, model_kwargs)


@pytest.fixture
def model():
    return DummyModel(MODEL_NAME, DATASET_NAME, CONFIG)


@withPackage('huggingface_hub')
def test_model_init():
    model = DummyModel(
        MODEL_NAME, DATASET_NAME, model_kwargs={
            **CONFIG, 'tensor': torch.Tensor([1, 2, 3])
        })
    assert model.model_config == CONFIG


@withPackage('huggingface_hub')
def test_save_pretrained(model, tmp_path):
    save_directory = f'{str(tmp_path / REPO_NAME)}'
    model.save_pretrained(save_directory)
    files = os.listdir(save_directory)
    assert 'model.pth' in files
    assert len(files) >= 1


@withPackage('huggingface_hub')
def test_save_pretrained_internal(model, tmp_path):
    save_directory = f'{str(tmp_path / REPO_NAME)}'
    model._save_pretrained = Mock()
    model.save_pretrained(save_directory)
    model._save_pretrained.assert_called_with(Path(save_directory))


@withPackage('huggingface_hub')
def test_save_pretrained_with_push_to_hub(model, tmp_path):
    save_directory = f'{str(tmp_path / REPO_NAME)}'

    model.push_to_hub = Mock()
    model.construct_model_card = Mock()
    model._save_pretrained = Mock()  # disable _save_pretrained to speed-up

    # Not pushed to hub
    model.save_pretrained(save_directory)
    model.push_to_hub.assert_not_called()
    model.construct_model_card.assert_called_with(MODEL_NAME, DATASET_NAME)

    # Push to hub with repo_id
    model.save_pretrained(save_directory, push_to_hub=True, repo_id='CustomID',
                          config=CONFIG)
    model.push_to_hub.assert_called_with(repo_id='CustomID', config=CONFIG)

    # Push to hub with default repo_id (based on dir name)
    model.save_pretrained(save_directory, push_to_hub=True, config=CONFIG)
    model.push_to_hub.assert_called_with(repo_id=REPO_NAME, config=CONFIG)


@withPackage('huggingface_hub')
def test_from_pretrained(model, tmp_path):
    save_directory = f'{str(tmp_path / REPO_NAME)}'
    model.save_pretrained(save_directory)

    model = model.from_pretrained(save_directory)
    assert model.model_config == CONFIG


@withPackage('huggingface_hub')
def test_from_pretrained_internal(model, monkeypatch):
    hf_hub_download = Mock(side_effect='model')
    monkeypatch.setattr('torch_geometric.nn.model_hub.hf_hub_download',
                        hf_hub_download)
    monkeypatch.setattr('torch_geometric.nn.model_hub.torch.load',
                        lambda x, **kwargs: {'state_dict': 1})

    model = model._from_pretrained(
        model_id=MODEL_NAME,
        revision=None,
        cache_dir=None,
        force_download=False,
        proxies=None,
        resume_download=True,
        local_files_only=False,
        token=False,
        dataset_name=DATASET_NAME,
        model_name=MODEL_NAME,
        map_location='cpu',
        strict=False,
        **CONFIG,
    )

    assert hf_hub_download.call_count == 1
    assert model.model_config == CONFIG
pytorch/test/nn/test_model_summary.py000066400000000000000000000221731444262304300204350ustar00rootroot00000000000000from typing import Optional

import pytest
import torch
from torch import Tensor, nn

import torch_geometric.typing
from torch_geometric.nn import Linear, SAGEConv, summary, to_hetero
from torch_geometric.nn.models import GCN
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor


class GraphSAGE(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(16, 16)
        self.conv1 = SAGEConv(16, 32)
        self.lin2 = Linear(32, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.lin1(x).relu()
        x = self.conv1(x, edge_index).relu()
        x = self.lin2(x)
        return x


class ModuleDictModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.acts = nn.ModuleDict({
            "lrelu": nn.LeakyReLU(),
            "prelu": nn.PReLU()
        })

    def forward(self, x: torch.Tensor, act_type: str) -> torch.Tensor:
        return self.acts[act_type](x)


@pytest.fixture
def gcn():
    torch.manual_seed(1)
    model = GCN(32, 16, num_layers=2, out_channels=32)
    x = torch.randn(100, 32)
    edge_index = torch.randint(100, size=(2, 20))
    adj_t: Optional[SparseTensor] = None
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t = SparseTensor.from_edge_index(
            edge_index,
            sparse_sizes=(100, 100),
        ).t()
    return dict(model=model, x=x, edge_index=edge_index, adj_t=adj_t)


@withPackage('tabulate')
def test_summary_basic(gcn):
    expected = """
+---------------------+--------------------+----------------+----------+
| Layer               | Input Shape        | Output Shape   | #Param   |
|---------------------+--------------------+----------------+----------|
| GCN                 | [100, 32], [2, 20] | [100, 32]      | 1,072    |
| β”œβ”€(act)ReLU         | [100, 16]          | [100, 16]      | --       |
| β”œβ”€(convs)ModuleList | --                 | --             | 1,072    |
| β”‚    └─(0)GCNConv   | [100, 32], [2, 20] | [100, 16]      | 528      |
| β”‚    └─(1)GCNConv   | [100, 16], [2, 20] | [100, 32]      | 544      |
+---------------------+--------------------+----------------+----------+
"""
    assert summary(gcn['model'], gcn['x'], gcn['edge_index']) == expected[1:-1]


@withPackage('tabulate')
@withPackage('torch_sparse')
def test_summary_with_sparse_tensor(gcn):
    expected = """
+---------------------+-----------------------+----------------+----------+
| Layer               | Input Shape           | Output Shape   | #Param   |
|---------------------+-----------------------+----------------+----------|
| GCN                 | [100, 32], [100, 100] | [100, 32]      | 1,072    |
| β”œβ”€(act)ReLU         | [100, 16]             | [100, 16]      | --       |
| β”œβ”€(convs)ModuleList | --                    | --             | 1,072    |
| β”‚    └─(0)GCNConv   | [100, 32], [100, 100] | [100, 16]      | 528      |
| β”‚    └─(1)GCNConv   | [100, 16], [100, 100] | [100, 32]      | 544      |
+---------------------+-----------------------+----------------+----------+
"""
    assert summary(gcn['model'], gcn['x'], gcn['adj_t']) == expected[1:-1]


@withPackage('tabulate')
def test_summary_with_max_depth(gcn):
    expected = """
+---------------------+--------------------+----------------+----------+
| Layer               | Input Shape        | Output Shape   | #Param   |
|---------------------+--------------------+----------------+----------|
| GCN                 | [100, 32], [2, 20] | [100, 32]      | 1,072    |
| β”œβ”€(act)ReLU         | [100, 16]          | [100, 16]      | --       |
| β”œβ”€(convs)ModuleList | --                 | --             | 1,072    |
+---------------------+--------------------+----------------+----------+
"""
    assert summary(gcn['model'], gcn['x'], gcn['edge_index'],
                   max_depth=1) == expected[1:-1]


@withPackage('tabulate')
def test_summary_with_leaf_module(gcn):
    expected = """# noqa: E501
+-----------------------------------------+--------------------+----------------+----------+
| Layer                                   | Input Shape        | Output Shape   | #Param   |
|-----------------------------------------+--------------------+----------------+----------|
| GCN                                     | [100, 32], [2, 20] | [100, 32]      | 1,072    |
| β”œβ”€(act)ReLU                             | [100, 16]          | [100, 16]      | --       |
| β”œβ”€(convs)ModuleList                     | --                 | --             | 1,072    |
| β”‚    └─(0)GCNConv                       | [100, 32], [2, 20] | [100, 16]      | 528      |
| β”‚    β”‚    └─(aggr_module)SumAggregation | [120, 16], [120]   | [100, 16]      | --       |
| β”‚    β”‚    └─(lin)Linear                 | [100, 32]          | [100, 16]      | 512      |
| β”‚    └─(1)GCNConv                       | [100, 16], [2, 20] | [100, 32]      | 544      |
| β”‚    β”‚    └─(aggr_module)SumAggregation | [120, 32], [120]   | [100, 32]      | --       |
| β”‚    β”‚    └─(lin)Linear                 | [100, 16]          | [100, 32]      | 512      |
+-----------------------------------------+--------------------+----------------+----------+
"""
    assert summary(gcn['model'], gcn['x'], gcn['edge_index'],
                   leaf_module=None) == expected[13:-1]


@withPackage('tabulate')
def test_summary_with_reusing_layers():
    act = nn.ReLU(inplace=True)
    model1 = nn.Sequential(act, nn.Identity(), act, nn.Identity(), act)
    model2 = nn.Sequential(
        nn.ReLU(inplace=True),
        nn.Identity(),
        nn.ReLU(inplace=True),
        nn.Identity(),
        nn.ReLU(inplace=True),
    )
    x = torch.randn(10)

    assert summary(model1, x) == summary(model2, x)


@withPackage('tabulate')
def test_summary_with_to_hetero_model():
    x_dict = {
        'p': torch.randn(100, 16),
        'a': torch.randn(100, 16),
    }
    edge_index_dict = {
        ('p', 'to', 'p'): torch.randint(100, (2, 200)),
        ('p', 'to', 'a'): torch.randint(100, (2, 200)),
        ('a', 'to', 'p'): torch.randint(100, (2, 200)),
    }
    metadata = list(x_dict.keys()), list(edge_index_dict.keys())
    model = to_hetero(GraphSAGE(), metadata)

    expected = """
+---------------------------+---------------------+----------------+----------+
| Layer                     | Input Shape         | Output Shape   | #Param   |
|---------------------------+---------------------+----------------+----------|
| GraphModule               |                     |                | 5,824    |
| β”œβ”€(lin1)ModuleDict        | --                  | --             | 544      |
| β”‚    └─(p)Linear          | [100, 16]           | [100, 16]      | 272      |
| β”‚    └─(a)Linear          | [100, 16]           | [100, 16]      | 272      |
| β”œβ”€(conv1)ModuleDict       | --                  | --             | 3,168    |
| β”‚    └─(p__to__p)SAGEConv | [100, 16], [2, 200] | [100, 32]      | 1,056    |
| β”‚    └─(p__to__a)SAGEConv | [2, 200]            | [100, 32]      | 1,056    |
| β”‚    └─(a__to__p)SAGEConv | [2, 200]            | [100, 32]      | 1,056    |
| β”œβ”€(lin2)ModuleDict        | --                  | --             | 2,112    |
| β”‚    └─(p)Linear          | [100, 32]           | [100, 32]      | 1,056    |
| β”‚    └─(a)Linear          | [100, 32]           | [100, 32]      | 1,056    |
+---------------------------+---------------------+----------------+----------+
"""
    assert summary(model, x_dict, edge_index_dict) == expected[1:-1]


@withPackage('tabulate')
def test_summary_with_module_dict_model():
    model = ModuleDictModel()
    x = torch.randn(100, 32)

    expected = """
+-------------------------+---------------+----------------+----------+
| Layer                   | Input Shape   | Output Shape   | #Param   |
|-------------------------+---------------+----------------+----------|
| ModuleDictModel         | [100, 32]     | [100, 32]      | 1        |
| β”œβ”€(acts)ModuleDict      | --            | --             | 1        |
| β”‚    └─(lrelu)LeakyReLU | --            | --             | --       |
| β”‚    └─(prelu)PReLU     | [100, 32]     | [100, 32]      | 1        |
+-------------------------+---------------+----------------+----------+
"""
    assert summary(model, x, 'prelu') == expected[1:-1]


@withPackage('tabulate')
def test_summary_with_jit_model():
    model = nn.Sequential(nn.Linear(32, 16), nn.ReLU(), nn.Linear(16, 8))
    model = torch.jit.script(model)
    x = torch.randn(100, 32)

    expected = """
+----------------------------+---------------+----------------+----------+
| Layer                      | Input Shape   | Output Shape   | #Param   |
|----------------------------+---------------+----------------+----------|
| RecursiveScriptModule      | --            | --             | 664      |
| β”œβ”€(0)RecursiveScriptModule | --            | --             | 528      |
| β”œβ”€(1)RecursiveScriptModule | --            | --             | --       |
| β”œβ”€(2)RecursiveScriptModule | --            | --             | 136      |
+----------------------------+---------------+----------------+----------+
"""
    assert summary(model, x) == expected[1:-1]
pytorch/test/nn/test_module_dict.py000066400000000000000000000033101444262304300200400ustar00rootroot00000000000000import torch

from torch_geometric.nn.module_dict import ModuleDict


def test_internal_external_key_conversion():
    assert ModuleDict.to_internal_key('a.b') == 'a#b'
    assert ModuleDict.to_internal_key('ab') == 'ab'
    assert ModuleDict.to_internal_key('a.b.c') == 'a#b#c'
    assert ModuleDict.to_internal_key(('a', 'b')) == ''
    assert ModuleDict.to_internal_key(('a.b', 'c')) == ''

    assert ModuleDict.to_external_key('a#b') == 'a.b'
    assert ModuleDict.to_external_key('a#b#c') == 'a.b.c'
    assert ModuleDict.to_external_key('') == ('a', 'b')
    assert ModuleDict.to_external_key('') == ('a.b', 'c')


def test_dot_syntax_keys():
    module_dict = ModuleDict({
        'lin1': torch.nn.Linear(16, 16),
        'model.lin2': torch.nn.Linear(8, 8),
        'model.sub_model.lin3': torch.nn.Linear(4, 4),
    })

    expected_keys = {'lin1', 'model.lin2', 'model.sub_model.lin3'}
    assert set(module_dict.keys()) == expected_keys
    assert set([key for key, _ in module_dict.items()]) == expected_keys

    for key in expected_keys:
        assert key in module_dict

    assert 'model.lin2' in module_dict
    del module_dict['model.lin2']
    assert 'model.lin2' not in module_dict


def test_tuple_keys():
    module_dict = ModuleDict({
        ('a', 'b'): torch.nn.Linear(16, 16),
        ('a.b', 'c'): torch.nn.Linear(8, 8),
    })

    expected_keys = {('a', 'b'), ('a.b', 'c')}
    assert set(module_dict.keys()) == expected_keys
    assert set([key for key, _ in module_dict.items()]) == expected_keys

    for key in expected_keys:
        assert key in module_dict

    assert ('a', 'b') in module_dict
    del module_dict['a', 'b']
    assert ('a', 'b') not in module_dict
pytorch/test/nn/test_parameter_dict.py000066400000000000000000000037101444262304300205370ustar00rootroot00000000000000import torch

from torch_geometric.nn.parameter_dict import ParameterDict


def test_internal_external_key_conversion():
    assert ParameterDict.to_internal_key('a.b') == 'a#b'
    assert ParameterDict.to_internal_key('ab') == 'ab'
    assert ParameterDict.to_internal_key('a.b.c') == 'a#b#c'
    assert ParameterDict.to_internal_key(('a', 'b')) == ''
    assert ParameterDict.to_internal_key(('a.b', 'c')) == ''

    assert ParameterDict.to_external_key('a#b') == 'a.b'
    assert ParameterDict.to_external_key('a#b#c') == 'a.b.c'
    assert ParameterDict.to_external_key('') == ('a', 'b')
    assert ParameterDict.to_external_key('') == ('a.b', 'c')


def test_dot_syntax_keys():
    parameter_dict = {
        'param1': torch.nn.Parameter(torch.Tensor(16, 16)),
        'model.param2': torch.nn.Parameter(torch.Tensor(8, 8)),
        'model.sub_model.param3': torch.nn.Parameter(torch.Tensor(4, 4)),
    }
    parameter_dict = ParameterDict(parameter_dict)

    expected_keys = {'param1', 'model.param2', 'model.sub_model.param3'}
    assert set(parameter_dict.keys()) == expected_keys
    assert set([key for key, _ in parameter_dict.items()]) == expected_keys

    for key in expected_keys:
        assert key in parameter_dict

    assert 'model.param2' in parameter_dict
    del parameter_dict['model.param2']
    assert 'model.param2' not in parameter_dict


def test_tuple_keys():
    parameter_dict = {
        ('a', 'b'): torch.nn.Parameter(torch.Tensor(16, 16)),
        ('a.b', 'c'): torch.nn.Parameter(torch.Tensor(8, 8)),
    }
    parameter_dict = ParameterDict(parameter_dict)

    expected_keys = {('a', 'b'), ('a.b', 'c')}
    assert set(parameter_dict.keys()) == expected_keys
    assert set([key for key, _ in parameter_dict.items()]) == expected_keys

    for key in expected_keys:
        assert key in parameter_dict

    assert ('a', 'b') in parameter_dict
    del parameter_dict['a', 'b']
    assert ('a', 'b') not in parameter_dict
pytorch/test/nn/test_reshape.py000066400000000000000000000004051444262304300172010ustar00rootroot00000000000000import torch

from torch_geometric.nn.reshape import Reshape


def test_reshape():
    x = torch.randn(10, 4)

    op = Reshape(5, 2, 4)
    assert str(op) == 'Reshape(5, 2, 4)'

    assert op(x).size() == (5, 2, 4)
    assert torch.equal(op(x).view(10, 4), x)
pytorch/test/nn/test_resolver.py000066400000000000000000000102041444262304300174110ustar00rootroot00000000000000import pytest
import torch
from torch.optim.lr_scheduler import ConstantLR, LambdaLR, ReduceLROnPlateau

import torch_geometric
from torch_geometric.nn.resolver import (
    activation_resolver,
    aggregation_resolver,
    lr_scheduler_resolver,
    normalization_resolver,
    optimizer_resolver,
)


def test_activation_resolver():
    assert isinstance(activation_resolver(torch.nn.ELU()), torch.nn.ELU)
    assert isinstance(activation_resolver(torch.nn.ReLU()), torch.nn.ReLU)
    assert isinstance(activation_resolver(torch.nn.PReLU()), torch.nn.PReLU)

    assert isinstance(activation_resolver('elu'), torch.nn.ELU)
    assert isinstance(activation_resolver('relu'), torch.nn.ReLU)
    assert isinstance(activation_resolver('prelu'), torch.nn.PReLU)


@pytest.mark.parametrize('aggr_tuple', [
    (torch_geometric.nn.MeanAggregation, 'mean'),
    (torch_geometric.nn.SumAggregation, 'sum'),
    (torch_geometric.nn.SumAggregation, 'add'),
    (torch_geometric.nn.MaxAggregation, 'max'),
    (torch_geometric.nn.MinAggregation, 'min'),
    (torch_geometric.nn.MulAggregation, 'mul'),
    (torch_geometric.nn.VarAggregation, 'var'),
    (torch_geometric.nn.StdAggregation, 'std'),
    (torch_geometric.nn.SoftmaxAggregation, 'softmax'),
    (torch_geometric.nn.PowerMeanAggregation, 'powermean'),
])
def test_aggregation_resolver(aggr_tuple):
    aggr_module, aggr_repr = aggr_tuple
    assert isinstance(aggregation_resolver(aggr_module()), aggr_module)
    assert isinstance(aggregation_resolver(aggr_repr), aggr_module)


def test_multi_aggregation_resolver():
    aggr = aggregation_resolver(None)
    assert aggr is None

    aggr = aggregation_resolver(['sum', 'mean', None])
    assert isinstance(aggr, torch_geometric.nn.MultiAggregation)
    assert len(aggr.aggrs) == 3
    assert isinstance(aggr.aggrs[0], torch_geometric.nn.SumAggregation)
    assert isinstance(aggr.aggrs[1], torch_geometric.nn.MeanAggregation)
    assert aggr.aggrs[2] is None


@pytest.mark.parametrize('norm_tuple', [
    (torch_geometric.nn.BatchNorm, 'batch', (16, )),
    (torch_geometric.nn.BatchNorm, 'batch_norm', (16, )),
    (torch_geometric.nn.InstanceNorm, 'instance_norm', (16, )),
    (torch_geometric.nn.LayerNorm, 'layer_norm', (16, )),
    (torch_geometric.nn.GraphNorm, 'graph_norm', (16, )),
    (torch_geometric.nn.GraphSizeNorm, 'graphsize_norm', ()),
    (torch_geometric.nn.PairNorm, 'pair_norm', ()),
    (torch_geometric.nn.MessageNorm, 'message_norm', ()),
    (torch_geometric.nn.DiffGroupNorm, 'diffgroup_norm', (16, 4)),
])
def test_normalization_resolver(norm_tuple):
    norm_module, norm_repr, norm_args = norm_tuple
    assert isinstance(normalization_resolver(norm_module(*norm_args)),
                      norm_module)
    assert isinstance(normalization_resolver(norm_repr, *norm_args),
                      norm_module)


def test_optimizer_resolver():
    params = [torch.nn.Parameter(torch.Tensor(1))]

    assert isinstance(optimizer_resolver(torch.optim.SGD(params, lr=0.01)),
                      torch.optim.SGD)
    assert isinstance(optimizer_resolver(torch.optim.Adam(params)),
                      torch.optim.Adam)
    assert isinstance(optimizer_resolver(torch.optim.Rprop(params)),
                      torch.optim.Rprop)

    assert isinstance(optimizer_resolver('sgd', params, lr=0.01),
                      torch.optim.SGD)
    assert isinstance(optimizer_resolver('adam', params), torch.optim.Adam)
    assert isinstance(optimizer_resolver('rprop', params), torch.optim.Rprop)


@pytest.mark.parametrize('scheduler_args', [
    ('constant_with_warmup', LambdaLR),
    ('linear_with_warmup', LambdaLR),
    ('cosine_with_warmup', LambdaLR),
    ('cosine_with_warmup_restarts', LambdaLR),
    ('polynomial_with_warmup', LambdaLR),
    ('constant', ConstantLR),
    ('ReduceLROnPlateau', ReduceLROnPlateau),
])
def test_lr_scheduler_resolver(scheduler_args):
    scheduler_name, scheduler_cls = scheduler_args

    model = torch.nn.Linear(10, 5)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

    lr_scheduler = lr_scheduler_resolver(
        scheduler_name,
        optimizer,
        num_training_steps=100,
    )
    assert isinstance(lr_scheduler, scheduler_cls)
pytorch/test/nn/test_sequential.py000066400000000000000000000104141444262304300177250ustar00rootroot00000000000000from collections import OrderedDict

import torch
import torch.fx
from torch.nn import Dropout, Linear, ReLU

import torch_geometric.typing
from torch_geometric.nn import (
    GCNConv,
    JumpingKnowledge,
    MessagePassing,
    Sequential,
    global_mean_pool,
)
from torch_geometric.typing import SparseTensor


def test_sequential():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    batch = torch.zeros(4, dtype=torch.long)

    model = Sequential('x, edge_index', [
        (GCNConv(16, 64), 'x, edge_index -> x'),
        ReLU(inplace=True),
        (GCNConv(64, 64), 'x, edge_index -> x'),
        ReLU(inplace=True),
        Linear(64, 7),
    ])
    model.reset_parameters()

    assert len(model) == 5
    assert str(model) == (
        'Sequential(\n'
        '  (0): GCNConv(16, 64)\n'
        '  (1): ReLU(inplace=True)\n'
        '  (2): GCNConv(64, 64)\n'
        '  (3): ReLU(inplace=True)\n'
        '  (4): Linear(in_features=64, out_features=7, bias=True)\n'
        ')')

    assert isinstance(model[0], GCNConv)
    assert isinstance(model[1], ReLU)
    assert isinstance(model[2], GCNConv)
    assert isinstance(model[3], ReLU)
    assert isinstance(model[4], Linear)

    out = model(x, edge_index)
    assert out.size() == (4, 7)

    model = Sequential('x, edge_index, batch', [
        (Dropout(p=0.5), 'x -> x'),
        (GCNConv(16, 64), 'x, edge_index -> x1'),
        ReLU(inplace=True),
        (GCNConv(64, 64), 'x1, edge_index -> x2'),
        ReLU(inplace=True),
        (lambda x1, x2: [x1, x2], 'x1, x2 -> xs'),
        (JumpingKnowledge('cat', 64, num_layers=2), 'xs -> x'),
        (global_mean_pool, 'x, batch -> x'),
        Linear(2 * 64, 7),
    ])
    model.reset_parameters()

    out = model(x, edge_index, batch)
    assert out.size() == (1, 7)


def test_sequential_jittable():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    model = Sequential('x: Tensor, edge_index: Tensor', [
        (GCNConv(16, 64).jittable(), 'x, edge_index -> x'),
        ReLU(inplace=True),
        (GCNConv(64, 64).jittable(), 'x, edge_index -> x'),
        ReLU(inplace=True),
        Linear(64, 7),
    ])
    torch.jit.script(model)(x, edge_index)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t = SparseTensor.from_edge_index(edge_index).t()

        model = Sequential('x: Tensor, edge_index: SparseTensor', [
            (GCNConv(16, 64).jittable(), 'x, edge_index -> x'),
            ReLU(inplace=True),
            (GCNConv(64, 64).jittable(), 'x, edge_index -> x'),
            ReLU(inplace=True),
            Linear(64, 7),
        ])
        torch.jit.script(model)(x, adj_t)


def symbolic_trace(module):
    class Tracer(torch.fx.Tracer):
        def is_leaf_module(self, module, *args, **kwargs) -> bool:
            return (isinstance(module, MessagePassing)
                    or super().is_leaf_module(module, *args, **kwargs))

    return torch.fx.GraphModule(module, Tracer().trace(module))


def test_sequential_tracable():
    model = Sequential('x, edge_index', [
        (GCNConv(16, 64), 'x, edge_index -> x1'),
        ReLU(inplace=True),
        (GCNConv(64, 64), 'x1, edge_index -> x2'),
        ReLU(inplace=True),
        (lambda x1, x2: x1 + x2, 'x1, x2 -> x'),
        Linear(64, 7),
    ])
    symbolic_trace(model)


def test_sequential_with_multiple_return_values():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    model = Sequential('x, edge_index', [
        (GCNConv(16, 32), 'x, edge_index -> x1'),
        (GCNConv(32, 64), 'x1, edge_index -> x2'),
        (lambda x1, x2: (x1, x2), 'x1, x2 -> x1, x2'),
    ])

    x1, x2 = model(x, edge_index)
    assert x1.size() == (4, 32)
    assert x2.size() == (4, 64)


def test_sequential_with_ordered_dict():
    x = torch.randn(4, 16)
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])

    model = Sequential(
        'x, edge_index', modules=OrderedDict([
            ('conv1', (GCNConv(16, 32), 'x, edge_index -> x')),
            ('conv2', (GCNConv(32, 64), 'x, edge_index -> x')),
        ]))

    assert isinstance(model.conv1, GCNConv)
    assert isinstance(model.conv2, GCNConv)

    x = model(x, edge_index)
    assert x.size() == (4, 64)
pytorch/test/nn/test_to_fixed_size_transformer.py000066400000000000000000000011161444262304300230270ustar00rootroot00000000000000import torch

from torch_geometric.nn import SumAggregation
from torch_geometric.nn.to_fixed_size_transformer import to_fixed_size


class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.aggr = SumAggregation()

    def forward(self, x, batch):
        return self.aggr(x, batch, dim=0)


def test_to_fixed_size():
    x = torch.randn(10, 16)
    batch = torch.zeros(10, dtype=torch.long)

    model = Model()
    assert model(x, batch).size() == (1, 16)

    model = to_fixed_size(model, batch_size=10)
    assert model(x, batch).size() == (10, 16)
pytorch/test/nn/test_to_hetero_module.py000066400000000000000000000036521444262304300211160ustar00rootroot00000000000000import pytest  # noqa
import torch

from torch_geometric.nn.conv import SAGEConv
from torch_geometric.nn.dense import Linear
from torch_geometric.nn.to_hetero_module import (
    ToHeteroLinear,
    ToHeteroMessagePassing,
)


@pytest.mark.parametrize('LinearCls', [torch.nn.Linear, Linear])
def test_to_hetero_linear(LinearCls):
    x_dict = {'1': torch.randn(5, 16), '2': torch.randn(4, 16)}
    x = torch.cat([x_dict['1'], x_dict['2']], dim=0)
    type_vec = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1])

    module = ToHeteroLinear(LinearCls(16, 32), list(x_dict.keys()))

    out_dict = module(x_dict)
    assert len(out_dict) == 2
    assert out_dict['1'].size() == (5, 32)
    assert out_dict['2'].size() == (4, 32)

    out = module(x, type_vec)
    assert out.size() == (9, 32)

    assert torch.allclose(out_dict['1'], out[0:5])
    assert torch.allclose(out_dict['2'], out[5:9])


def test_to_hetero_message_passing():
    x_dict = {'1': torch.randn(5, 16), '2': torch.randn(4, 16)}
    x = torch.cat([x_dict['1'], x_dict['2']], dim=0)
    node_type = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1])

    edge_index_dict = {
        ('1', 'to', '2'): torch.tensor([[0, 1, 2, 3, 4], [0, 0, 1, 2, 3]]),
        ('2', 'to', '1'): torch.tensor([[0, 0, 1, 2, 3], [0, 1, 2, 3, 4]]),
    }
    edge_index = torch.tensor([
        [0, 1, 2, 3, 4, 5, 5, 6, 7, 8],
        [5, 5, 6, 7, 8, 0, 1, 2, 3, 4],
    ])
    edge_type = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

    module = ToHeteroMessagePassing(SAGEConv(16, 32), list(x_dict.keys()),
                                    list(edge_index_dict.keys()))

    out_dict = module(x_dict, edge_index_dict)
    assert len(out_dict) == 2
    assert out_dict['1'].size() == (5, 32)
    assert out_dict['2'].size() == (4, 32)

    out = module(x, edge_index, node_type, edge_type)
    assert out.size() == (9, 32)

    assert torch.allclose(out_dict['1'], out[0:5])
    assert torch.allclose(out_dict['2'], out[5:9])
pytorch/test/nn/test_to_hetero_transformer.py000066400000000000000000000401741444262304300221730ustar00rootroot00000000000000from typing import Tuple

import pytest
import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear, ReLU, Sequential

import torch_geometric.typing
from torch_geometric.nn import GAT, BatchNorm, GCNConv, GINEConv, GraphSAGE
from torch_geometric.nn import Linear as LazyLinear
from torch_geometric.nn import (
    MeanAggregation,
    MessagePassing,
    RGCNConv,
    SAGEConv,
    to_hetero,
)
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import dropout_edge

torch.fx.wrap('dropout_edge')


class Net1(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(16, 32)
        self.lin2 = Linear(8, 16)

    def forward(self, x: Tensor, edge_attr: Tensor) -> Tuple[Tensor, Tensor]:
        x = self.lin1(x)
        edge_attr = self.lin2(edge_attr)
        return x, edge_attr


class Net2(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(16, 16)
        self.conv1 = SAGEConv(16, 32)
        self.lin2 = Linear(32, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.lin1(x).relu_()
        x = self.conv1(x, edge_index).relu_()
        x = self.lin2(x)
        return x


class Net3(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(8, 16)
        self.conv1 = GINEConv(nn=Linear(16, 32))

    def forward(self, x: Tensor, edge_index: Tensor,
                edge_attr: Tensor) -> Tensor:
        x = self.conv1(x, edge_index, self.lin1(edge_attr))
        return x


class Net4(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SAGEConv(16, 16)
        self.conv2 = SAGEConv(16, 16)
        self.lin1 = Linear(3 * 16, 32)

    def forward(self, x0: Tensor, edge_index: Tensor) -> Tensor:
        x1 = self.conv1(x0, edge_index).relu_()
        x2 = self.conv2(x1, edge_index).relu_()
        return self.lin1(torch.cat([x0, x1, x2], dim=-1))


class Net5(torch.nn.Module):
    def __init__(self, num_layers):
        super().__init__()
        self.lins = torch.nn.ModuleList()
        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            self.lins.append(Linear(16, 16))
            self.convs.append(SAGEConv(16, 16))

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for lin, conv in zip(self.lins, self.convs):
            x = (conv(x, edge_index) + lin(x))
        return x


class Net6(torch.nn.Module):
    def __init__(self, num_layers):
        super().__init__()
        self.lins = torch.nn.ModuleDict()
        self.convs = torch.nn.ModuleDict()
        for i in range(num_layers):
            self.lins[str(i)] = Linear(16, 16)
            self.convs[str(i)] = SAGEConv(16, 16)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for i in range(len(self.lins)):
            x = (self.convs[str(i)](x, edge_index) + self.lins[str(i)](x))
        return x


class Net7(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.mlp1 = Sequential(Linear(16, 16), ReLU(), Linear(16, 16))
        self.conv1 = SAGEConv(16, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.mlp1(x)
        x = self.conv1(x, edge_index)
        return x


class Net8(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = LazyLinear(-1, 32)

    def forward(self, x: Tensor) -> Tensor:
        x = self.lin1(x)
        return x


class Net9(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.batch_norm = BatchNorm(16)

    def forward(self, x: Tensor) -> Tensor:
        return self.batch_norm(x)


class Net10(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = SAGEConv(16, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = F.dropout(x, p=0.5, training=self.training)
        edge_index, _ = dropout_edge(edge_index, p=0.5, training=self.training)
        return self.conv(x, edge_index)


class Net11(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = SAGEConv(16, 16)
        self.num_layers = 3

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        xs = [x]
        for _ in range(self.num_layers):
            xs.append(self.conv(xs[-1], edge_index))
        return torch.cat(xs, dim=-1)


class Net12(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = Net8()

    def forward(self, x: Tensor) -> Tensor:
        return self.conv(x)


def test_to_hetero_basic():
    x_dict = {
        'paper': torch.randn(100, 16),
        'author': torch.randn(100, 16),
    }
    edge_index_dict = {
        ('paper', 'cites', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('paper', 'written_by', 'author'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('author', 'writes', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
    }
    edge_attr_dict = {
        ('paper', 'cites', 'paper'): torch.randn(200, 8),
        ('paper', 'written_by', 'author'): torch.randn(200, 8),
        ('author', 'writes', 'paper'): torch.randn(200, 8),
    }

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict = {}
        for edge_type, (row, col) in edge_index_dict.items():
            adj_t_dict[edge_type] = SparseTensor(
                row=col,
                col=row,
                sparse_sizes=(100, 100),
            )

    metadata = list(x_dict.keys()), list(edge_index_dict.keys())

    model = Net1()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_attr_dict)
    assert isinstance(out, tuple) and len(out) == 2
    assert isinstance(out[0], dict) and len(out[0]) == 2
    assert out[0]['paper'].size() == (100, 32)
    assert out[0]['author'].size() == (100, 32)
    assert isinstance(out[1], dict) and len(out[1]) == 3
    assert out[1][('paper', 'cites', 'paper')].size() == (200, 16)
    assert out[1][('paper', 'written_by', 'author')].size() == (200, 16)
    assert out[1][('author', 'writes', 'paper')].size() == (200, 16)
    assert sum(p.numel() for p in model.parameters()) == 1520

    for aggr in ['sum', 'mean', 'min', 'max', 'mul']:
        model = Net2()
        model = to_hetero(model, metadata, aggr='mean', debug=False)
        assert sum(p.numel() for p in model.parameters()) == 5824

        out1 = model(x_dict, edge_index_dict)
        assert isinstance(out1, dict) and len(out1) == 2
        assert out1['paper'].size() == (100, 32)
        assert out1['author'].size() == (100, 32)

        if torch_geometric.typing.WITH_TORCH_SPARSE:
            out2 = model(x_dict, adj_t_dict)
            assert isinstance(out2, dict) and len(out2) == 2
            for key in x_dict.keys():
                assert torch.allclose(out1[key], out2[key], atol=1e-6)

    model = Net3()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict, edge_attr_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net4()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net5(num_layers=2)
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 16)
    assert out['author'].size() == (100, 16)

    model = Net6(num_layers=2)
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 16)
    assert out['author'].size() == (100, 16)

    model = Net7()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net8()
    model = to_hetero(model, metadata, debug=False)
    out = model({'paper': torch.randn(4, 8), 'author': torch.randn(8, 16)})
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (4, 32)
    assert out['author'].size() == (8, 32)

    model = Net9()
    model = to_hetero(model, metadata, debug=False)
    out = model({'paper': torch.randn(4, 16), 'author': torch.randn(8, 16)})
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (4, 16)
    assert out['author'].size() == (8, 16)

    model = Net10()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net11()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 64)
    assert out['author'].size() == (100, 64)

    model = Net12()
    with pytest.warns(UserWarning, match="parameters cannot be reset"):
        model = to_hetero(model, metadata, debug=False)
    out = model({'paper': torch.randn(4, 8), 'author': torch.randn(8, 16)})
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (4, 32)
    assert out['author'].size() == (8, 32)


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(16, 32)
        self.conv2 = GCNConv(32, 64)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index).relu()
        return x


def test_to_hetero_with_gcn():
    x_dict = {
        'paper': torch.randn(100, 16),
    }
    edge_index_dict = {
        ('paper', 'cites', 'paper'): torch.randint(100, (2, 200)),
        ('paper', 'rev_cites', 'paper'): torch.randint(100, (2, 200)),
    }
    metadata = list(x_dict.keys()), list(edge_index_dict.keys())

    model = GCN()
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 1
    assert out['paper'].size() == (100, 64)


def test_to_hetero_with_basic_model():
    x_dict = {
        'paper': torch.randn(100, 16),
        'author': torch.randn(100, 16),
    }
    edge_index_dict = {
        ('paper', 'cites', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('paper', 'written_by', 'author'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('author', 'writes', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
    }

    metadata = list(x_dict.keys()), list(edge_index_dict.keys())

    model = GraphSAGE((-1, -1), 32, num_layers=3)
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2

    model = GAT((-1, -1), 32, num_layers=3, add_self_loops=False)
    model = to_hetero(model, metadata, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2


class GraphConv(MessagePassing):
    def __init__(self, in_channels, out_channels):
        super().__init__(aggr='sum')
        self.lin = Linear(in_channels, out_channels, bias=False)

    def reset_parameters(self):
        self.lin.reset_parameters()

    def forward(self, x, edge_index):
        if isinstance(x, Tensor):
            x = (x, x)
        return self.propagate(edge_index, x=(self.lin(x[0]), x[1]))


class RGCN(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = GraphConv(in_channels, out_channels)
        self.lin = Linear(in_channels, out_channels, bias=True)

    def forward(self, x, edge_index):
        return self.lin(x) + self.conv(x, edge_index)


def test_to_hetero_and_rgcn_equal_output():
    torch.manual_seed(1234)

    # Run `RGCN`:
    x = torch.randn(10, 16)  # 6 paper nodes, 4 author nodes
    adj = (torch.rand(10, 10) > 0.5)
    adj[6:, 6:] = False
    edge_index = adj.nonzero(as_tuple=False).t().contiguous()
    row, col = edge_index

    # # 0 = paper<->paper, 1 = paper->author, 2 = author->paper
    edge_type = torch.full((edge_index.size(1), ), -1, dtype=torch.long)
    edge_type[(row < 6) & (col < 6)] = 0
    edge_type[(row < 6) & (col >= 6)] = 1
    edge_type[(row >= 6) & (col < 6)] = 2
    assert edge_type.min() == 0

    conv = RGCNConv(16, 32, num_relations=3, aggr='sum')
    out1 = conv(x, edge_index, edge_type)

    # Run `to_hetero`:
    x_dict = {
        'paper': x[:6],
        'author': x[6:],
    }
    edge_index_dict = {
        ('paper', '_', 'paper'):
        edge_index[:, edge_type == 0],
        ('paper', '_', 'author'):
        edge_index[:, edge_type == 1] - torch.tensor([[0], [6]]),
        ('author', '_', 'paper'):
        edge_index[:, edge_type == 2] - torch.tensor([[6], [0]]),
    }

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict = {
            key: SparseTensor.from_edge_index(edge_index).t()
            for key, edge_index in edge_index_dict.items()
        }

    node_types, edge_types = list(x_dict.keys()), list(edge_index_dict.keys())

    model = to_hetero(RGCN(16, 32), (node_types, edge_types))

    # Set model weights:
    for i, edge_type in enumerate(edge_types):
        weight = model.conv['__'.join(edge_type)].lin.weight
        weight.data = conv.weight[i].data.t()
    for i, node_type in enumerate(node_types):
        model.lin[node_type].weight.data = conv.root.data.t()
        model.lin[node_type].bias.data = conv.bias.data

    out2 = model(x_dict, edge_index_dict)
    out2 = torch.cat([out2['paper'], out2['author']], dim=0)
    assert torch.allclose(out1, out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3 = model(x_dict, adj_t_dict)
        out3 = torch.cat([out3['paper'], out3['author']], dim=0)
        assert torch.allclose(out1, out3, atol=1e-6)


class GraphLevelGNN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = SAGEConv(16, 32)
        self.pool = MeanAggregation()
        self.lin = Linear(32, 64)

    def forward(self, x: Tensor, edge_index: Tensor, batch: Tensor) -> Tensor:
        x = self.conv(x, edge_index)
        x = self.pool(x, batch)
        x = self.lin(x)
        return x


def test_graph_level_to_hetero():
    x_dict = {
        'paper': torch.randn(100, 16),
        'author': torch.randn(100, 16),
    }
    edge_index_dict = {
        ('paper', 'written_by', 'author'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('author', 'writes', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
    }
    batch_dict = {
        'paper': torch.zeros(100, dtype=torch.long),
        'author': torch.zeros(100, dtype=torch.long),
    }

    metadata = list(x_dict.keys()), list(edge_index_dict.keys())

    model = GraphLevelGNN()
    model = to_hetero(model, metadata, aggr='mean', debug=False)
    out = model(x_dict, edge_index_dict, batch_dict)
    assert out.size() == (1, 64)


class MessagePassingLoops(MessagePassing):
    def __init__(self):
        super().__init__()
        self.add_self_loops = True

    def forward(self, x):
        return x


class ModelLoops(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = MessagePassingLoops()

    def forward(self, x):
        return self.conv(x)


def test_hetero_transformer_self_loop_error():
    to_hetero(ModelLoops(), metadata=(['a'], [('a', 'to', 'a')]))
    with pytest.raises(ValueError, match="incorrect message passing"):
        to_hetero(ModelLoops(), metadata=(['a', 'b'], [('a', 'to', 'b'),
                                                       ('b', 'to', 'a')]))


def test_to_hetero_validate():
    model = Net1()
    metadata = (['my test'], [('my test', 'rel', 'my test')])

    with pytest.warns(UserWarning, match="letters, numbers and underscores"):
        model = to_hetero(model, metadata, debug=False)
pytorch/test/nn/test_to_hetero_with_bases_transformer.py000066400000000000000000000240701444262304300244000ustar00rootroot00000000000000from typing import Tuple

import pytest
import torch
from torch import Tensor
from torch.nn import Linear, ReLU, Sequential

import torch_geometric.typing
from torch_geometric.nn import (
    GINEConv,
    MessagePassing,
    RGCNConv,
    SAGEConv,
    to_hetero_with_bases,
)
from torch_geometric.typing import SparseTensor


class Net1(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(16, 32)
        self.lin2 = Linear(8, 16)

    def forward(self, x: Tensor, edge_attr: Tensor) -> Tuple[Tensor, Tensor]:
        x = self.lin1(x)
        edge_attr = self.lin2(edge_attr)
        return x, edge_attr


class Net2(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(16, 16)
        self.conv1 = SAGEConv(16, 32)
        self.lin2 = Linear(32, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.lin1(x).relu_()
        x = self.conv1(x, edge_index).relu_()
        x = self.lin2(x)
        return x


class Net3(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = Linear(8, 16)
        self.conv1 = GINEConv(nn=Linear(16, 32))

    def forward(self, x: Tensor, edge_index: Tensor,
                edge_attr: Tensor) -> Tensor:
        x = self.conv1(x, edge_index, self.lin1(edge_attr))
        return x


class Net4(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = SAGEConv(16, 16)
        self.conv2 = SAGEConv(16, 16)
        self.lin1 = Linear(3 * 16, 32)

    def forward(self, x0: Tensor, edge_index: Tensor) -> Tensor:
        x1 = self.conv1(x0, edge_index).relu_()
        x2 = self.conv2(x1, edge_index).relu_()
        return self.lin1(torch.cat([x0, x1, x2], dim=-1))


class Net5(torch.nn.Module):
    def __init__(self, num_layers):
        super().__init__()
        self.lins = torch.nn.ModuleList()
        self.convs = torch.nn.ModuleList()
        for _ in range(num_layers):
            self.lins.append(Linear(16, 16))
            self.convs.append(SAGEConv(16, 16))

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for lin, conv in zip(self.lins, self.convs):
            x = (conv(x, edge_index) + lin(x))
        return x


class Net6(torch.nn.Module):
    def __init__(self, num_layers):
        super().__init__()
        self.lins = torch.nn.ModuleDict()
        self.convs = torch.nn.ModuleDict()
        for i in range(num_layers):
            self.lins[str(i)] = Linear(16, 16)
            self.convs[str(i)] = SAGEConv(16, 16)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        for i in range(len(self.lins)):
            x = (self.convs[str(i)](x, edge_index) + self.lins[str(i)](x))
        return x


class Net7(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.mlp1 = Sequential(Linear(16, 16), ReLU(), Linear(16, 16))
        self.conv1 = SAGEConv(16, 32)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        x = self.mlp1(x)
        x = self.conv1(x, edge_index)
        return x


def test_to_hetero_with_bases():
    metadata = (['paper', 'author'], [('paper', 'cites', 'paper'),
                                      ('paper', 'written_by', 'author'),
                                      ('author', 'writes', 'paper')])

    x_dict = {'paper': torch.randn(100, 16), 'author': torch.randn(100, 8)}
    edge_index_dict = {
        ('paper', 'cites', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('paper', 'written_by', 'author'):
        torch.randint(100, (2, 200), dtype=torch.long),
        ('author', 'writes', 'paper'):
        torch.randint(100, (2, 200), dtype=torch.long),
    }
    edge_attr_dict = {
        ('paper', 'cites', 'paper'): torch.randn(200, 8),
        ('paper', 'written_by', 'author'): torch.randn(200, 8),
        ('author', 'writes', 'paper'): torch.randn(200, 8),
    }

    model = Net1()
    in_channels = {'x': 16, 'edge_attr': 8}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_attr_dict)
    assert isinstance(out, tuple) and len(out) == 2
    assert isinstance(out[0], dict) and len(out[0]) == 2
    assert out[0]['paper'].size() == (100, 32)
    assert out[0]['author'].size() == (100, 32)
    assert isinstance(out[1], dict) and len(out[1]) == 3
    assert out[1][('paper', 'cites', 'paper')].size() == (200, 16)
    assert out[1][('paper', 'written_by', 'author')].size() == (200, 16)
    assert out[1][('author', 'writes', 'paper')].size() == (200, 16)
    assert sum(p.numel() for p in model.parameters()) == 1264

    model = Net2()
    in_channels = {'x': 16}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)
    assert sum(p.numel() for p in model.parameters()) == 5948

    model = Net3()
    in_channels = {'x': 16, 'edge_attr': 8}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict, edge_attr_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net4()
    in_channels = {'x0': 16}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)

    model = Net5(num_layers=2)
    in_channels = {'x': 16}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 16)
    assert out['author'].size() == (100, 16)

    model = Net6(num_layers=2)
    in_channels = {'x': 16}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 16)
    assert out['author'].size() == (100, 16)

    model = Net7()
    in_channels = {'x': 16}
    model = to_hetero_with_bases(model, metadata, num_bases=4,
                                 in_channels=in_channels, debug=False)
    out = model(x_dict, edge_index_dict)
    assert isinstance(out, dict) and len(out) == 2
    assert out['paper'].size() == (100, 32)
    assert out['author'].size() == (100, 32)


class GraphConv(MessagePassing):
    def __init__(self, in_channels, out_channels):
        super().__init__(aggr='add')
        self.lin = Linear(in_channels, out_channels, bias=False)

    def reset_parameters(self):
        self.lin.reset_parameters()

    def forward(self, x, edge_index):
        if isinstance(x, Tensor):
            x = (x, x)
        return self.propagate(edge_index, x=(self.lin(x[0]), x[1]))


class RGCN(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = GraphConv(in_channels, out_channels)
        self.lin = Linear(in_channels, out_channels, bias=True)

    def forward(self, x, edge_index):
        return self.lin(x) + self.conv(x, edge_index)


def test_to_hetero_with_bases_and_rgcn_equal_output():
    torch.manual_seed(1234)

    # Run `RGCN` with basis decomposition:
    x = torch.randn(10, 16)  # 6 paper nodes, 4 author nodes
    adj = (torch.rand(10, 10) > 0.5)
    adj[6:, 6:] = False
    edge_index = adj.nonzero(as_tuple=False).t().contiguous()
    row, col = edge_index

    # # 0 = paper<->paper, 1 = author->paper, 2 = paper->author
    edge_type = torch.full((edge_index.size(1), ), -1, dtype=torch.long)
    edge_type[(row < 6) & (col < 6)] = 0
    edge_type[(row < 6) & (col >= 6)] = 1
    edge_type[(row >= 6) & (col < 6)] = 2
    assert edge_type.min() == 0

    num_bases = 4
    conv = RGCNConv(16, 32, num_relations=3, num_bases=num_bases, aggr='add')
    out1 = conv(x, edge_index, edge_type)

    # Run `to_hetero_with_bases`:
    x_dict = {
        'paper': x[:6],
        'author': x[6:],
    }
    edge_index_dict = {
        ('paper', '_', 'paper'):
        edge_index[:, edge_type == 0],
        ('paper', '_', 'author'):
        edge_index[:, edge_type == 1] - torch.tensor([[0], [6]]),
        ('author', '_', 'paper'):
        edge_index[:, edge_type == 2] - torch.tensor([[6], [0]]),
    }

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj_t_dict = {
            key: SparseTensor.from_edge_index(edge_index).t()
            for key, edge_index in edge_index_dict.items()
        }

    metadata = (list(x_dict.keys()), list(edge_index_dict.keys()))
    model = to_hetero_with_bases(RGCN(16, 32), metadata, num_bases=num_bases,
                                 debug=False)

    # Set model weights:
    for i in range(num_bases):
        model.conv.convs[i].lin.weight.data = conv.weight[i].data.t()
        model.conv.convs[i].edge_type_weight.data = conv.comp[:, i].data.t()

    model.lin.weight.data = conv.root.data.t()
    model.lin.bias.data = conv.bias.data

    out2 = model(x_dict, edge_index_dict)
    out2 = torch.cat([out2['paper'], out2['author']], dim=0)
    assert torch.allclose(out1, out2, atol=1e-6)

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3 = model(x_dict, adj_t_dict)
        out3 = torch.cat([out3['paper'], out3['author']], dim=0)
        assert torch.allclose(out1, out3, atol=1e-6)


def test_to_hetero_with_bases_validate():
    model = Net1()
    metadata = (['my test'], [('my test', 'rel', 'my test')])

    with pytest.warns(UserWarning, match="letters, numbers and underscores"):
        model = to_hetero_with_bases(model, metadata, num_bases=4, debug=False)
pytorch/test/nn/unpool/000077500000000000000000000000001444262304300154565ustar00rootroot00000000000000pytorch/test/nn/unpool/test_knn_interpolate.py000066400000000000000000000010761444262304300222670ustar00rootroot00000000000000import torch

from torch_geometric.nn import knn_interpolate
from torch_geometric.testing import withPackage


@withPackage('torch_cluster')
def test_knn_interpolate():
    x = torch.Tensor([[1], [10], [100], [-1], [-10], [-100]])
    pos_x = torch.Tensor([[-1, 0], [0, 0], [1, 0], [-2, 0], [0, 0], [2, 0]])
    pos_y = torch.Tensor([[-1, -1], [1, 1], [-2, -2], [2, 2]])
    batch_x = torch.tensor([0, 0, 0, 1, 1, 1])
    batch_y = torch.tensor([0, 0, 1, 1])

    y = knn_interpolate(x, pos_x, pos_y, batch_x, batch_y, k=2)
    assert y.tolist() == [[4], [70], [-4], [-70]]
pytorch/test/profile/000077500000000000000000000000001444262304300151675ustar00rootroot00000000000000pytorch/test/profile/test_benchmark.py000066400000000000000000000007531444262304300205370ustar00rootroot00000000000000import torch

from torch_geometric.profile import benchmark
from torch_geometric.testing import withPackage


@withPackage('tabulate')
def test_benchmark(capfd):
    def add(x, y):
        return x + y

    benchmark(
        funcs=[add],
        args=(torch.randn(10), torch.randn(10)),
        num_steps=1,
        num_warmups=1,
        backward=True,
    )

    out, _ = capfd.readouterr()
    assert '| Name   | Forward   | Backward   | Total   |' in out
    assert '| add    |' in out
pytorch/test/profile/test_profile.py000066400000000000000000000055661444262304300202540ustar00rootroot00000000000000import os.path
import warnings

import torch
import torch.nn.functional as F

from torch_geometric.nn import GraphSAGE
from torch_geometric.profile import (
    get_stats_summary,
    profileit,
    rename_profile_file,
    timeit,
)
from torch_geometric.profile.profile import torch_profile
from torch_geometric.testing import onlyCUDA, onlyLinux, withCUDA, withPackage


@withCUDA
@onlyLinux
def test_timeit(device):
    x = torch.randn(100, 16, device=device)
    lin = torch.nn.Linear(16, 32).to(device)

    with timeit(log=False) as t:
        assert not hasattr(t, 'duration')

        with torch.no_grad():
            lin(x)
        t.reset()
        assert t.duration > 0

        del t.duration
        assert not hasattr(t, 'duration')
    assert t.duration > 0


@onlyCUDA
@withPackage('pytorch_memlab')
def test_profileit(get_dataset):
    warnings.filterwarnings('ignore', '.*arguments of DataFrame.drop.*')

    dataset = get_dataset(name='Cora')
    data = dataset[0].cuda()
    model = GraphSAGE(dataset.num_features, hidden_channels=64, num_layers=3,
                      out_channels=dataset.num_classes).cuda()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

    @profileit()
    def train(model, x, edge_index, y):
        model.train()
        optimizer.zero_grad()
        out = model(x, edge_index)
        loss = F.cross_entropy(out, y)
        loss.backward()
        return float(loss)

    stats_list = []
    for epoch in range(5):
        _, stats = train(model, data.x, data.edge_index, data.y)
        assert len(stats) == 6
        assert stats.time > 0
        assert stats.max_allocated_cuda > 0
        assert stats.max_reserved_cuda > 0
        assert stats.max_active_cuda > 0
        assert stats.nvidia_smi_free_cuda > 0
        assert stats.nvidia_smi_used_cuda > 0

        if epoch >= 2:  # Warm-up
            stats_list.append(stats)

    stats_summary = get_stats_summary(stats_list)
    assert len(stats_summary) == 7
    assert stats_summary.time_mean > 0
    assert stats_summary.time_std > 0
    assert stats_summary.max_allocated_cuda > 0
    assert stats_summary.max_reserved_cuda > 0
    assert stats_summary.max_active_cuda > 0
    assert stats_summary.min_nvidia_smi_free_cuda > 0
    assert stats_summary.max_nvidia_smi_used_cuda > 0


@withCUDA
def test_torch_profile(capfd, get_dataset, device):
    dataset = get_dataset(name='Cora')
    data = dataset[0].to(device)
    model = GraphSAGE(dataset.num_features, hidden_channels=64, num_layers=3,
                      out_channels=dataset.num_classes).to(device)

    with torch_profile():
        model(data.x, data.edge_index)

    out, _ = capfd.readouterr()
    assert 'Self CPU time total' in out
    if data.x.is_cuda:
        assert 'Self CUDA time total' in out

    rename_profile_file('test_profile')
    assert os.path.exists('profile-test_profile.json')
    os.remove('profile-test_profile.json')
pytorch/test/profile/test_profile_utils.py000066400000000000000000000037001444262304300214600ustar00rootroot00000000000000import torch
from torch.nn import Linear

from torch_geometric.data import Data
from torch_geometric.profile import (
    count_parameters,
    get_cpu_memory_from_gc,
    get_data_size,
    get_gpu_memory_from_gc,
    get_gpu_memory_from_nvidia_smi,
    get_model_size,
)
from torch_geometric.profile.utils import (
    byte_to_megabyte,
    medibyte_to_megabyte,
)
from torch_geometric.testing import onlyCUDA, withPackage
from torch_geometric.typing import SparseTensor


def test_count_parameters():
    assert count_parameters(Linear(32, 128)) == 32 * 128 + 128


def test_get_model_size():
    model_size = get_model_size(Linear(32, 128, bias=False))
    assert model_size >= 32 * 128 * 4 and model_size < 32 * 128 * 4 + 2000


def test_get_data_size():
    x = torch.randn(10, 128)
    data = Data(x=x, y=x)

    data_size = get_data_size(data)
    assert data_size == 10 * 128 * 4


@withPackage('torch_sparse')
def test_get_data_size_with_sparse_tensor():
    x = torch.randn(10, 128)
    row, col = torch.randint(0, 10, (2, 100), dtype=torch.long)
    adj_t = SparseTensor(row=row, col=col, value=None, sparse_sizes=(10, 10))
    data = Data(x=x, y=x, adj_t=adj_t)

    data_size = get_data_size(data)
    assert data_size == 10 * 128 * 4 + 11 * 8 + 100 * 8


def test_get_cpu_memory_from_gc():
    old_mem = get_cpu_memory_from_gc()
    _ = torch.randn(10, 128)
    new_mem = get_cpu_memory_from_gc()
    assert new_mem - old_mem == 10 * 128 * 4


@onlyCUDA
def test_get_gpu_memory_from_gc():
    old_mem = get_gpu_memory_from_gc()
    _ = torch.randn(10, 128, device='cuda')
    new_mem = get_gpu_memory_from_gc()
    assert new_mem - old_mem == 10 * 128 * 4


@onlyCUDA
def test_get_gpu_memory_from_nvidia_smi():
    free_mem, used_mem = get_gpu_memory_from_nvidia_smi(device=0, digits=2)
    assert free_mem >= 0
    assert used_mem >= 0


def test_bytes_function():
    assert byte_to_megabyte((1024 * 1024)) == 1.00
    assert medibyte_to_megabyte(1 / 1.0485) == 1.00
pytorch/test/profile/test_profiler.py000066400000000000000000000015731444262304300204300ustar00rootroot00000000000000import torch

from torch_geometric.nn import GraphSAGE
from torch_geometric.profile.profiler import Profiler
from torch_geometric.testing import withCUDA


@withCUDA
def test_profiler(capfd, get_dataset, device):
    x = torch.randn(10, 16, device=device)
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6, 8, 7, 9, 8],
    ], device=device)

    model = GraphSAGE(16, hidden_channels=32, num_layers=2).to(device)

    with Profiler(model, profile_memory=True, use_cuda=x.is_cuda) as prof:
        model(x, edge_index)

    _, err = capfd.readouterr()
    assert 'Completed Stage' in err

    _, heading_list, raw_results, layer_names, layer_stats = prof.get_trace()
    assert 'Self CPU total' in heading_list
    assert 'aten::relu' in raw_results
    assert '-act--aten::relu' in layer_names
pytorch/test/sampler/000077500000000000000000000000001444262304300151725ustar00rootroot00000000000000pytorch/test/sampler/test_sampler_base.py000066400000000000000000000076531444262304300212530ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.sampler.base import (
    HeteroSamplerOutput,
    NumNeighbors,
    SamplerOutput,
)
from torch_geometric.testing import get_random_edge_index
from torch_geometric.utils import is_undirected


def test_homogeneous_num_neighbors():
    with pytest.raises(ValueError, match="'default' must be set to 'None'"):
        num_neighbors = NumNeighbors([25, 10], default=[-1, -1])

    num_neighbors = NumNeighbors([25, 10])
    assert str(num_neighbors) == 'NumNeighbors(values=[25, 10], default=None)'

    assert num_neighbors.get_values() == [25, 10]
    assert num_neighbors.__dict__['_values'] == [25, 10]
    assert num_neighbors.get_values() == [25, 10]  # Test caching.

    assert num_neighbors.get_mapped_values() == [25, 10]
    assert num_neighbors.__dict__['_mapped_values'] == [25, 10]
    assert num_neighbors.get_mapped_values() == [25, 10]  # Test caching.

    assert num_neighbors.num_hops == 2
    assert num_neighbors.__dict__['_num_hops'] == 2
    assert num_neighbors.num_hops == 2  # Test caching.


def test_heterogeneous_num_neighbors_list():
    num_neighbors = NumNeighbors([25, 10])

    values = num_neighbors.get_values([('A', 'B'), ('B', 'A')])
    assert values == {('A', 'B'): [25, 10], ('B', 'A'): [25, 10]}

    values = num_neighbors.get_mapped_values([('A', 'B'), ('B', 'A')])
    assert values == {'A__to__B': [25, 10], 'B__to__A': [25, 10]}

    assert num_neighbors.num_hops == 2


def test_heterogeneous_num_neighbors_dict_and_default():
    num_neighbors = NumNeighbors({('A', 'B'): [25, 10]}, default=[-1])
    with pytest.raises(ValueError, match="hops must be the same across all"):
        values = num_neighbors.get_values([('A', 'B'), ('B', 'A')])

    num_neighbors = NumNeighbors({('A', 'B'): [25, 10]}, default=[-1, -1])

    values = num_neighbors.get_values([('A', 'B'), ('B', 'A')])
    assert values == {('A', 'B'): [25, 10], ('B', 'A'): [-1, -1]}

    values = num_neighbors.get_mapped_values([('A', 'B'), ('B', 'A')])
    assert values == {'A__to__B': [25, 10], 'B__to__A': [-1, -1]}

    assert num_neighbors.num_hops == 2


def test_heterogeneous_num_neighbors_empty_dict():
    num_neighbors = NumNeighbors({}, default=[25, 10])

    values = num_neighbors.get_values([('A', 'B'), ('B', 'A')])
    assert values == {('A', 'B'): [25, 10], ('B', 'A'): [25, 10]}

    values = num_neighbors.get_mapped_values([('A', 'B'), ('B', 'A')])
    assert values == {'A__to__B': [25, 10], 'B__to__A': [25, 10]}

    assert num_neighbors.num_hops == 2


def test_homogeneous_to_bidirectional():
    edge_index = get_random_edge_index(10, 10, num_edges=20)

    obj = SamplerOutput(
        node=torch.arange(10),
        row=edge_index[0],
        col=edge_index[0],
        edge=torch.arange(edge_index.size(1)),
    ).to_bidirectional()

    assert is_undirected(torch.stack([obj.row, obj.col], dim=0))


def test_heterogeneous_to_bidirectional():
    edge_index1 = get_random_edge_index(10, 5, num_edges=20)
    edge_index2 = get_random_edge_index(5, 10, num_edges=20)
    edge_index3 = get_random_edge_index(10, 10, num_edges=20)

    obj = HeteroSamplerOutput(
        node={
            'v1': torch.arange(10),
            'v2': torch.arange(5)
        },
        row={
            ('v1', 'to', 'v2'): edge_index1[0],
            ('v2', 'rev_to', 'v1'): edge_index2[0],
            ('v1', 'to', 'v1'): edge_index3[0],
        },
        col={
            ('v1', 'to', 'v2'): edge_index1[1],
            ('v2', 'rev_to', 'v1'): edge_index2[1],
            ('v1', 'to', 'v1'): edge_index3[1],
        },
        edge={},
    ).to_bidirectional()

    assert torch.equal(
        obj.row['v1', 'to', 'v2'].sort().values,
        obj.col['v2', 'rev_to', 'v1'].sort().values,
    )
    assert torch.equal(
        obj.col['v1', 'to', 'v2'].sort().values,
        obj.row['v2', 'rev_to', 'v1'].sort().values,
    )
    assert is_undirected(
        torch.stack([obj.row['v1', 'to', 'v1'], obj.col['v1', 'to', 'v1']], 0))
pytorch/test/test_config_store.py000066400000000000000000000107031444262304300176220ustar00rootroot00000000000000from typing import Any, Optional

from torch_geometric.config_store import (
    class_from_dataclass,
    dataclass_from_class,
    fill_config_store,
    get_config_store,
    register,
    to_dataclass,
)
from torch_geometric.testing import withPackage


def test_to_dataclass():
    from torch_geometric.transforms import AddSelfLoops

    AddSelfLoopsConfig = to_dataclass(AddSelfLoops, with_target=True)
    assert AddSelfLoopsConfig.__name__ == 'AddSelfLoops'

    fields = AddSelfLoopsConfig.__dataclass_fields__

    assert fields['attr'].name == 'attr'
    assert fields['attr'].type == Optional[str]
    assert fields['attr'].default == 'edge_weight'

    assert fields['fill_value'].name == 'fill_value'
    assert fields['fill_value'].type == Any
    assert fields['fill_value'].default == 1.0

    assert fields['_target_'].name == '_target_'
    assert fields['_target_'].type == str
    assert fields['_target_'].default == (
        'torch_geometric.transforms.add_self_loops.AddSelfLoops')

    cfg = AddSelfLoopsConfig()
    assert str(cfg) == ("AddSelfLoops(attr='edge_weight', fill_value=1.0, "
                        "_target_='torch_geometric.transforms.add_self_loops."
                        "AddSelfLoops')")


def test_register():
    from torch_geometric.transforms import AddSelfLoops

    register(AddSelfLoops, group='transforms')
    assert 'transforms' in get_config_store().repo

    AddSelfLoopsConfig = dataclass_from_class('AddSelfLoops')

    Cls = class_from_dataclass('AddSelfLoops')
    assert Cls == AddSelfLoops
    Cls = class_from_dataclass(AddSelfLoopsConfig)
    assert Cls == AddSelfLoops

    ConfigCls = dataclass_from_class('AddSelfLoops')
    assert ConfigCls == AddSelfLoopsConfig
    ConfigCls = dataclass_from_class(ConfigCls)
    assert ConfigCls == AddSelfLoopsConfig


@withPackage('hydra')
def test_hydra_config_store():
    import hydra
    from omegaconf import DictConfig

    fill_config_store()

    with hydra.initialize(config_path='.', version_base='1.1'):
        cfg = hydra.compose(config_name='my_config')

    assert len(cfg) == 4
    assert 'dataset' in cfg
    assert 'model' in cfg
    assert 'optimizer' in cfg
    assert 'lr_scheduler' in cfg

    # Check `cfg.dataset`:
    assert len(cfg.dataset) == 2
    assert cfg.dataset._target_.split('.')[-1] == 'KarateClub'

    # Check `cfg.dataset.transform`:
    assert isinstance(cfg.dataset.transform, DictConfig)
    assert len(cfg.dataset.transform) == 2
    assert 'NormalizeFeatures' in cfg.dataset.transform
    assert 'AddSelfLoops' in cfg.dataset.transform

    assert isinstance(cfg.dataset.transform.NormalizeFeatures, DictConfig)
    assert (cfg.dataset.transform.NormalizeFeatures._target_.split('.')[-1] ==
            'NormalizeFeatures')
    assert cfg.dataset.transform.NormalizeFeatures.attrs == ['x']

    assert isinstance(cfg.dataset.transform.AddSelfLoops, DictConfig)
    assert (cfg.dataset.transform.AddSelfLoops._target_.split('.')[-1] ==
            'AddSelfLoops')
    assert cfg.dataset.transform.AddSelfLoops.attr == 'edge_weight'
    assert cfg.dataset.transform.AddSelfLoops.fill_value == 1.0

    # Check `cfg.model`:
    assert len(cfg.model) == 12
    assert cfg.model._target_.split('.')[-1] == 'GCN'
    assert cfg.model.in_channels == 34
    assert cfg.model.out_channels == 4
    assert cfg.model.hidden_channels == 16
    assert cfg.model.num_layers == 2
    assert cfg.model.dropout == 0.0
    assert cfg.model.act == 'relu'
    assert cfg.model.norm is None
    assert cfg.model.norm_kwargs is None
    assert cfg.model.jk is None
    assert not cfg.model.act_first
    assert cfg.model.act_kwargs is None

    # Check `cfg.optimizer`:
    assert cfg.optimizer._target_.split('.')[-1] == 'Adam'
    assert cfg.optimizer.lr == 0.001
    assert cfg.optimizer.betas == [0.9, 0.999]
    assert cfg.optimizer.eps == 1e-08
    assert cfg.optimizer.weight_decay == 0
    assert not cfg.optimizer.amsgrad
    if hasattr(cfg.optimizer, 'maximize'):
        assert not cfg.optimizer.maximize

    # Check `cfg.lr_scheduler`:
    assert cfg.lr_scheduler._target_.split('.')[-1] == 'ReduceLROnPlateau'
    assert cfg.lr_scheduler.mode == 'min'
    assert cfg.lr_scheduler.factor == 0.1
    assert cfg.lr_scheduler.patience == 10
    assert cfg.lr_scheduler.threshold == 0.0001
    assert cfg.lr_scheduler.threshold_mode == 'rel'
    assert cfg.lr_scheduler.cooldown == 0
    assert cfg.lr_scheduler.min_lr == 0
    assert cfg.lr_scheduler.eps == 1e-08
    assert not cfg.lr_scheduler.verbose
pytorch/test/test_debug.py000066400000000000000000000014301444262304300162240ustar00rootroot00000000000000from torch_geometric import debug, is_debug_enabled, set_debug


def test_debug():
    assert is_debug_enabled() is False
    set_debug(True)
    assert is_debug_enabled() is True
    set_debug(False)
    assert is_debug_enabled() is False

    assert is_debug_enabled() is False
    with set_debug(True):
        assert is_debug_enabled() is True
    assert is_debug_enabled() is False

    assert is_debug_enabled() is False
    set_debug(True)
    assert is_debug_enabled() is True
    with set_debug(False):
        assert is_debug_enabled() is False
    assert is_debug_enabled() is True
    set_debug(False)
    assert is_debug_enabled() is False

    assert is_debug_enabled() is False
    with debug():
        assert is_debug_enabled() is True
    assert is_debug_enabled() is False
pytorch/test/test_experimental.py000066400000000000000000000017271444262304300176440ustar00rootroot00000000000000import pytest

from torch_geometric import (
    experimental_mode,
    is_experimental_mode_enabled,
    set_experimental_mode,
)


@pytest.mark.parametrize('options', ['disable_dynamic_shapes'])
def test_experimental_mode(options):
    assert is_experimental_mode_enabled(options) is False
    with experimental_mode(options):
        assert is_experimental_mode_enabled(options) is True
    assert is_experimental_mode_enabled(options) is False

    with set_experimental_mode(True, options):
        assert is_experimental_mode_enabled(options) is True
    assert is_experimental_mode_enabled(options) is False

    with set_experimental_mode(False, options):
        assert is_experimental_mode_enabled(options) is False
    assert is_experimental_mode_enabled(options) is False

    set_experimental_mode(True, options)
    assert is_experimental_mode_enabled(options) is True
    set_experimental_mode(False, options)
    assert is_experimental_mode_enabled(options) is False
pytorch/test/test_home.py000066400000000000000000000007511444262304300160730ustar00rootroot00000000000000import os
import os.path as osp

from torch_geometric import get_home_dir, set_home_dir
from torch_geometric.home import DEFAULT_CACHE_DIR


def test_home():
    os.environ.pop('PYG_HOME', None)
    home_dir = osp.expanduser(DEFAULT_CACHE_DIR)
    assert get_home_dir() == home_dir

    home_dir = '/tmp/test_pyg1'
    os.environ['PYG_HOME'] = home_dir
    assert get_home_dir() == home_dir

    home_dir = '/tmp/test_pyg2'
    set_home_dir(home_dir)
    assert get_home_dir() == home_dir
pytorch/test/test_seed.py000066400000000000000000000006361444262304300160650ustar00rootroot00000000000000import random

import numpy as np
import torch

from torch_geometric import seed_everything


def test_seed_everything():
    seed_everything(0)

    assert random.randint(0, 100) == 49
    assert random.randint(0, 100) == 97
    assert np.random.randint(0, 100) == 44
    assert np.random.randint(0, 100) == 47
    assert int(torch.randint(0, 100, (1, ))) == 44
    assert int(torch.randint(0, 100, (1, ))) == 39
pytorch/test/test_typing.py000066400000000000000000000023521444262304300164540ustar00rootroot00000000000000import pytest

from torch_geometric.typing import EdgeTypeStr


def test_edge_type_str():
    edge_type_str = EdgeTypeStr('a__links__b')
    assert isinstance(edge_type_str, str)
    assert edge_type_str == 'a__links__b'
    assert edge_type_str.to_tuple() == ('a', 'links', 'b')

    edge_type_str = EdgeTypeStr('a', 'b')
    assert isinstance(edge_type_str, str)
    assert edge_type_str == 'a__to__b'
    assert edge_type_str.to_tuple() == ('a', 'to', 'b')

    edge_type_str = EdgeTypeStr(('a', 'b'))
    assert isinstance(edge_type_str, str)
    assert edge_type_str == 'a__to__b'
    assert edge_type_str.to_tuple() == ('a', 'to', 'b')

    edge_type_str = EdgeTypeStr('a', 'links', 'b')
    assert isinstance(edge_type_str, str)
    assert edge_type_str == 'a__links__b'
    assert edge_type_str.to_tuple() == ('a', 'links', 'b')

    edge_type_str = EdgeTypeStr(('a', 'links', 'b'))
    assert isinstance(edge_type_str, str)
    assert edge_type_str == 'a__links__b'
    assert edge_type_str.to_tuple() == ('a', 'links', 'b')

    with pytest.raises(ValueError, match="invalid edge type"):
        EdgeTypeStr('a', 'b', 'c', 'd')

    with pytest.raises(ValueError, match="Cannot convert the edge type"):
        EdgeTypeStr('a__b__c__d').to_tuple()
pytorch/test/testing/000077500000000000000000000000001444262304300152045ustar00rootroot00000000000000pytorch/test/testing/test_decorators.py000066400000000000000000000016161444262304300207660ustar00rootroot00000000000000import torch_geometric.typing
from torch_geometric.testing import disableExtensions


def test_enable_extensions():
    try:
        import pyg_lib  # noqa
        assert torch_geometric.typing.WITH_PYG_LIB
    except (ImportError, OSError):
        assert not torch_geometric.typing.WITH_PYG_LIB

    try:
        import torch_scatter  # noqa
        assert torch_geometric.typing.WITH_TORCH_SCATTER
    except (ImportError, OSError):
        assert not torch_geometric.typing.WITH_TORCH_SCATTER

    try:
        import torch_sparse  # noqa
        assert torch_geometric.typing.WITH_TORCH_SPARSE
    except (ImportError, OSError):
        assert not torch_geometric.typing.WITH_TORCH_SPARSE


@disableExtensions
def test_disable_extensions():
    assert not torch_geometric.typing.WITH_PYG_LIB
    assert not torch_geometric.typing.WITH_TORCH_SCATTER
    assert not torch_geometric.typing.WITH_TORCH_SPARSE
pytorch/test/transforms/000077500000000000000000000000001444262304300157255ustar00rootroot00000000000000pytorch/test/transforms/test_add_metapaths.py000066400000000000000000000222731444262304300221420ustar00rootroot00000000000000import copy

import torch
from torch import tensor

from torch_geometric.data import HeteroData
from torch_geometric.testing import withPackage
from torch_geometric.transforms import AddMetaPaths, AddRandomMetaPaths
from torch_geometric.utils import coalesce


def generate_data() -> HeteroData:
    data = HeteroData()
    data['p'].x = torch.ones(5)
    data['a'].x = torch.ones(6)
    data['c'].x = torch.ones(3)
    data['p', 'p'].edge_index = tensor([[0, 1, 2, 3], [1, 2, 4, 2]])
    data['p', 'a'].edge_index = tensor([[0, 1, 2, 3, 4], [2, 2, 5, 2, 5]])
    data['a', 'p'].edge_index = data['p', 'a'].edge_index.flip([0])
    data['c', 'p'].edge_index = tensor([[0, 0, 1, 2, 2], [0, 1, 2, 3, 4]])
    data['p', 'c'].edge_index = data['c', 'p'].edge_index.flip([0])
    return data


@withPackage('torch_sparse')
def test_add_metapaths():
    data = generate_data()
    # Test transform options:
    metapaths = [[('p', 'c'), ('c', 'p')]]

    transform = AddMetaPaths(metapaths)
    assert str(transform) == 'AddMetaPaths()'
    meta1 = transform(copy.copy(data))

    transform = AddMetaPaths(metapaths, drop_orig_edge_types=True)
    assert str(transform) == 'AddMetaPaths()'
    meta2 = transform(copy.copy(data))

    transform = AddMetaPaths(metapaths, drop_orig_edge_types=True,
                             keep_same_node_type=True)
    assert str(transform) == 'AddMetaPaths()'
    meta3 = transform(copy.copy(data))

    transform = AddMetaPaths(metapaths, drop_orig_edge_types=True,
                             keep_same_node_type=True,
                             drop_unconnected_node_types=True)
    assert str(transform) == 'AddMetaPaths()'
    meta4 = transform(copy.copy(data))

    assert meta1['metapath_0'].edge_index.size() == (2, 9)
    assert meta2['metapath_0'].edge_index.size() == (2, 9)
    assert meta3['metapath_0'].edge_index.size() == (2, 9)
    assert meta4['metapath_0'].edge_index.size() == (2, 9)

    assert all([i in meta1.edge_types for i in data.edge_types])
    assert meta2.edge_types == [('p', 'metapath_0', 'p')]
    assert meta3.edge_types == [('p', 'to', 'p'), ('p', 'metapath_0', 'p')]
    assert meta4.edge_types == [('p', 'to', 'p'), ('p', 'metapath_0', 'p')]

    assert meta3.node_types == ['p', 'a', 'c']
    assert meta4.node_types == ['p']

    # Test 4-hop metapath:
    metapaths = [
        [('a', 'p'), ('p', 'c')],
        [('a', 'p'), ('p', 'c'), ('c', 'p'), ('p', 'a')],
    ]
    transform = AddMetaPaths(metapaths)
    meta = transform(copy.copy(data))
    new_edge_types = [('a', 'metapath_0', 'c'), ('a', 'metapath_1', 'a')]
    assert meta['metapath_0'].edge_index.size() == (2, 4)
    assert meta['metapath_1'].edge_index.size() == (2, 4)

    # Test `metapath_dict` information:
    assert list(meta.metapath_dict.values()) == metapaths
    assert list(meta.metapath_dict.keys()) == new_edge_types


@withPackage('torch_sparse')
def test_add_metapaths_max_sample():
    torch.manual_seed(12345)

    data = generate_data()

    metapaths = [[('p', 'c'), ('c', 'p')]]
    transform = AddMetaPaths(metapaths, max_sample=1)

    meta = transform(data)
    assert meta['metapath_0'].edge_index.size(1) < 9


@withPackage('torch_sparse')
def test_add_weighted_metapaths():
    torch.manual_seed(12345)

    data = HeteroData()
    data['a'].num_nodes = 2
    data['b'].num_nodes = 3
    data['c'].num_nodes = 2
    data['d'].num_nodes = 2
    data['a', 'b'].edge_index = tensor([[0, 1, 1], [0, 1, 2]])
    data['b', 'a'].edge_index = data['a', 'b'].edge_index.flip([0])
    data['b', 'c'].edge_index = tensor([[0, 1, 2], [0, 1, 1]])
    data['c', 'b'].edge_index = data['b', 'c'].edge_index.flip([0])
    data['c', 'd'].edge_index = tensor([[0, 1], [0, 0]])
    data['d', 'c'].edge_index = data['c', 'd'].edge_index.flip([0])

    metapaths = [
        [('a', 'b'), ('b', 'c')],
        [('a', 'b'), ('b', 'c'), ('c', 'd')],
        [('a', 'b'), ('b', 'c'), ('c', 'd'), ('d', 'c'), ('c', 'b'),
         ('b', 'a')],
    ]
    transform = AddMetaPaths(metapaths, weighted=True)
    out = transform(copy.copy(data))

    # Make sure manually added metapaths compute the correct number of edges:
    edge_index = out['a', 'a'].edge_index
    edge_weight = out['a', 'a'].edge_weight
    edge_index, edge_weight = coalesce(edge_index, edge_weight)
    assert edge_index.tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert edge_weight.tolist() == [1, 2, 2, 4]

    edge_index = out['a', 'c'].edge_index
    edge_weight = out['a', 'c'].edge_weight
    edge_index, edge_weight = coalesce(edge_index, edge_weight)
    assert edge_index.tolist() == [[0, 1], [0, 1]]
    assert edge_weight.tolist() == [1, 2]

    edge_index = out['a', 'd'].edge_index
    edge_weight = out['a', 'd'].edge_weight
    edge_index, edge_weight = coalesce(edge_index, edge_weight)
    assert edge_index.tolist() == [[0, 1], [0, 0]]
    assert edge_weight.tolist() == [1, 2]

    # Compute intra-table metapaths efficiently:
    metapaths = [[('a', 'b'), ('b', 'c'), ('c', 'd')]]
    out = AddMetaPaths(metapaths, weighted=True)(copy.copy(data))
    out['d', 'a'].edge_index = out['a', 'd'].edge_index.flip([0])
    out['d', 'a'].edge_weight = out['a', 'd'].edge_weight
    metapaths = [[('a', 'd'), ('d', 'a')]]
    out = AddMetaPaths(metapaths, weighted=True)(out)

    edge_index = out['a', 'a'].edge_index
    edge_weight = out['a', 'a'].edge_weight
    edge_index, edge_weight = coalesce(edge_index, edge_weight)
    assert edge_index.tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert edge_weight.tolist() == [1, 2, 2, 4]


@withPackage('torch_sparse')
def test_add_random_metapaths():
    data = generate_data()

    # Test transform options:
    metapaths = [[('p', 'c'), ('c', 'p')]]
    torch.manual_seed(12345)

    transform = AddRandomMetaPaths(metapaths)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[1])'
    meta1 = transform(copy.copy(data))

    transform = AddRandomMetaPaths(metapaths, drop_orig_edge_types=True)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[1])'
    meta2 = transform(copy.copy(data))

    transform = AddRandomMetaPaths(metapaths, drop_orig_edge_types=True,
                                   keep_same_node_type=True)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[1])'
    meta3 = transform(copy.copy(data))

    transform = AddRandomMetaPaths(metapaths, drop_orig_edge_types=True,
                                   keep_same_node_type=True,
                                   drop_unconnected_node_types=True)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[1])'
    meta4 = transform(copy.copy(data))

    transform = AddRandomMetaPaths(metapaths, sample_ratio=0.8,
                                   drop_orig_edge_types=True,
                                   keep_same_node_type=True,
                                   drop_unconnected_node_types=True)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=0.8, walks_per_node=[1])'
    meta5 = transform(copy.copy(data))

    transform = AddRandomMetaPaths(metapaths, walks_per_node=5,
                                   drop_orig_edge_types=True,
                                   keep_same_node_type=True,
                                   drop_unconnected_node_types=True)
    assert str(transform
               ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[5])'
    meta6 = transform(copy.copy(data))

    assert meta1['metapath_0'].edge_index.size() == (2, 5)
    assert meta2['metapath_0'].edge_index.size() == (2, 5)
    assert meta3['metapath_0'].edge_index.size() == (2, 5)
    assert meta4['metapath_0'].edge_index.size() == (2, 5)
    assert meta5['metapath_0'].edge_index.size() == (2, 4)
    assert meta6['metapath_0'].edge_index.size() == (2, 7)

    assert all([i in meta1.edge_types for i in data.edge_types])
    assert meta2.edge_types == [('p', 'metapath_0', 'p')]
    assert meta3.edge_types == [('p', 'to', 'p'), ('p', 'metapath_0', 'p')]
    assert meta4.edge_types == [('p', 'to', 'p'), ('p', 'metapath_0', 'p')]

    assert meta3.node_types == ['p', 'a', 'c']
    assert meta4.node_types == ['p']

    # Test 4-hop metapath:
    metapaths = [
        [('a', 'p'), ('p', 'c')],
        [('a', 'p'), ('p', 'c'), ('c', 'p'), ('p', 'a')],
    ]
    transform = AddRandomMetaPaths(metapaths)
    assert str(
        transform
    ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[1, 1])'

    meta1 = transform(copy.copy(data))
    new_edge_types = [('a', 'metapath_0', 'c'), ('a', 'metapath_1', 'a')]
    assert meta1['metapath_0'].edge_index.size() == (2, 2)
    assert meta1['metapath_1'].edge_index.size() == (2, 2)

    # Test `metapath_dict` information:
    assert list(meta1.metapath_dict.values()) == metapaths
    assert list(meta1.metapath_dict.keys()) == new_edge_types

    transform = AddRandomMetaPaths(metapaths, walks_per_node=[2, 5])
    assert str(
        transform
    ) == 'AddRandomMetaPaths(sample_ratio=1.0, walks_per_node=[2, 5])'

    meta2 = transform(copy.copy(data))
    new_edge_types = [('a', 'metapath_0', 'c'), ('a', 'metapath_1', 'a')]
    assert meta2['metapath_0'].edge_index.size() == (2, 2)
    assert meta2['metapath_1'].edge_index.size() == (2, 3)
pytorch/test/transforms/test_add_positional_encoding.py000066400000000000000000000062071444262304300242020ustar00rootroot00000000000000import copy

import torch

from torch_geometric.data import Data
from torch_geometric.testing import onlyLinux
from torch_geometric.transforms import (
    AddLaplacianEigenvectorPE,
    AddRandomWalkPE,
)


def test_add_laplacian_eigenvector_pe():
    x = torch.randn(6, 4)
    edge_index = torch.tensor([[0, 1, 0, 4, 1, 4, 2, 3, 3, 5],
                               [1, 0, 4, 0, 4, 1, 3, 2, 5, 3]])
    data = Data(x=x, edge_index=edge_index)

    transform = AddLaplacianEigenvectorPE(k=3)
    assert str(transform) == 'AddLaplacianEigenvectorPE()'
    out = transform(copy.copy(data))
    assert out.laplacian_eigenvector_pe.size() == (6, 3)

    transform = AddLaplacianEigenvectorPE(k=3, attr_name=None)
    out = transform(copy.copy(data))
    assert out.x.size() == (6, 4 + 3)

    transform = AddLaplacianEigenvectorPE(k=3, attr_name='x')
    out = transform(copy.copy(data))
    assert out.x.size() == (6, 3)

    # Output tests:
    edge_index = torch.tensor([[0, 1, 0, 4, 1, 4, 2, 3, 3, 5, 2, 5],
                               [1, 0, 4, 0, 4, 1, 3, 2, 5, 3, 5, 2]])
    data = Data(x=x, edge_index=edge_index)

    transform1 = AddLaplacianEigenvectorPE(k=1, is_undirected=True)
    transform2 = AddLaplacianEigenvectorPE(k=1, is_undirected=False)

    # Clustering test with first non-trivial eigenvector (Fiedler vector)
    pe = transform1(copy.copy(data)).laplacian_eigenvector_pe
    pe_cluster_1 = pe[[0, 1, 4]]
    pe_cluster_2 = pe[[2, 3, 5]]
    assert not torch.allclose(pe_cluster_1, pe_cluster_2)
    assert torch.allclose(pe_cluster_1, pe_cluster_1.mean())
    assert torch.allclose(pe_cluster_2, pe_cluster_2.mean())

    pe = transform2(copy.copy(data)).laplacian_eigenvector_pe
    pe_cluster_1 = pe[[0, 1, 4]]
    pe_cluster_2 = pe[[2, 3, 5]]
    assert not torch.allclose(pe_cluster_1, pe_cluster_2)
    assert torch.allclose(pe_cluster_1, pe_cluster_1.mean())
    assert torch.allclose(pe_cluster_2, pe_cluster_2.mean())


@onlyLinux  # TODO  (matthias) Investigate CSR @ CSR support on Windows.
def test_add_random_walk_pe():
    x = torch.randn(6, 4)
    edge_index = torch.tensor([[0, 1, 0, 4, 1, 4, 2, 3, 3, 5],
                               [1, 0, 4, 0, 4, 1, 3, 2, 5, 3]])
    data = Data(x=x, edge_index=edge_index)

    transform = AddRandomWalkPE(walk_length=3)
    assert str(transform) == 'AddRandomWalkPE()'
    out = transform(copy.copy(data))
    assert out.random_walk_pe.size() == (6, 3)

    transform = AddRandomWalkPE(walk_length=3, attr_name=None)
    out = transform(copy.copy(data))
    assert out.x.size() == (6, 4 + 3)

    transform = AddRandomWalkPE(walk_length=3, attr_name='x')
    out = transform(copy.copy(data))
    assert out.x.size() == (6, 3)

    # Output tests:
    assert out.x.tolist() == [
        [0.0, 0.5, 0.25],
        [0.0, 0.5, 0.25],
        [0.0, 0.5, 0.00],
        [0.0, 1.0, 0.00],
        [0.0, 0.5, 0.25],
        [0.0, 0.5, 0.00],
    ]

    edge_index = torch.tensor([[0, 1, 2], [0, 1, 2]])
    data = Data(edge_index=edge_index, num_nodes=4)
    out = transform(copy.copy(data))

    assert out.x.tolist() == [
        [1.0, 1.0, 1.0],
        [1.0, 1.0, 1.0],
        [1.0, 1.0, 1.0],
        [0.0, 0.0, 0.0],
    ]
pytorch/test/transforms/test_add_remaining_self_loops.py000066400000000000000000000056271444262304300243560ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import AddRemainingSelfLoops


def test_add_remaining_self_loops():
    assert str(AddRemainingSelfLoops()) == 'AddRemainingSelfLoops()'

    assert len(AddRemainingSelfLoops()(Data())) == 0

    # No self-loops in `edge_index`.
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.tensor([1, 2, 3, 4])
    edge_attr = torch.tensor([[1, 2], [3, 4], [5, 6], [7, 8]])

    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddRemainingSelfLoops()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                        [1, 0, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3

    # Single self-loop in `edge_index`.
    edge_index = torch.tensor([[0, 0, 1, 2], [1, 0, 2, 1]])
    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddRemainingSelfLoops()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 1, 2, 0, 1, 2], [1, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3

    data = Data(edge_index=edge_index, edge_weight=edge_weight, num_nodes=3)
    data = AddRemainingSelfLoops(attr='edge_weight', fill_value=5)(data)
    assert data.edge_index.tolist() == [[0, 1, 2, 0, 1, 2], [1, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3
    assert data.edge_weight.tolist() == [1, 3, 4, 2, 5, 5]

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    data = AddRemainingSelfLoops(attr='edge_attr', fill_value='add')(data)
    assert data.edge_index.tolist() == [[0, 1, 2, 0, 1, 2], [1, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3
    assert data.edge_attr.tolist() == [[1, 2], [5, 6], [7, 8], [3, 4], [8, 10],
                                       [5, 6]]


def test_add_remaining_self_loops_all_loops_exist():
    # All self-loops already exist in the data object.
    edge_index = torch.tensor([[0, 1, 2], [0, 1, 2]])
    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddRemainingSelfLoops()(data)
    assert data.edge_index.tolist() == edge_index.tolist()

    # All self-loops already exist in the data object, some of them appear
    # multiple times.
    edge_index = torch.tensor([[0, 0, 1, 1, 2], [0, 0, 1, 1, 2]])
    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddRemainingSelfLoops()(data)
    assert data.edge_index.tolist() == [[0, 1, 2], [0, 1, 2]]


def test_hetero_add_remaining_self_loops():
    edge_index = torch.tensor([[0, 0, 1, 2], [1, 0, 2, 1]])

    data = HeteroData()
    data['v'].num_nodes = 3
    data['w'].num_nodes = 3
    data['v', 'v'].edge_index = edge_index
    data['v', 'w'].edge_index = edge_index
    data = AddRemainingSelfLoops()(data)
    assert data['v', 'v'].edge_index.tolist() == [[0, 1, 2, 0, 1, 2],
                                                  [1, 2, 1, 0, 1, 2]]
    assert data['v', 'w'].edge_index.tolist() == edge_index.tolist()
pytorch/test/transforms/test_add_self_loops.py000066400000000000000000000043541444262304300223210ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import AddSelfLoops


def test_add_self_loops():
    assert str(AddSelfLoops()) == 'AddSelfLoops()'

    assert len(AddSelfLoops()(Data())) == 0

    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.tensor([1, 2, 3, 4])
    edge_attr = torch.tensor([[1, 2], [3, 4], [5, 6], [7, 8]])

    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddSelfLoops()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                        [1, 0, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3

    data = Data(edge_index=edge_index, edge_weight=edge_weight, num_nodes=3)
    data = AddSelfLoops(attr='edge_weight', fill_value=5)(data)
    assert data.edge_index.tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                        [1, 0, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3
    assert data.edge_weight.tolist() == [1, 2, 3, 4, 5, 5, 5]

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    data = AddSelfLoops(attr='edge_attr', fill_value='add')(data)
    assert data.edge_index.tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                        [1, 0, 2, 1, 0, 1, 2]]
    assert data.num_nodes == 3
    assert data.edge_attr.tolist() == [[1, 2], [3, 4], [5, 6], [7, 8], [3, 4],
                                       [8, 10], [5, 6]]


def test_add_self_loops_with_existing_self_loops():
    edge_index = torch.tensor([[0, 1, 2], [0, 1, 2]])
    data = Data(edge_index=edge_index, num_nodes=3)
    data = AddSelfLoops()(data)
    assert data.edge_index.tolist() == [[0, 1, 2, 0, 1, 2], [0, 1, 2, 0, 1, 2]]
    assert data.num_nodes == 3


def test_hetero_add_self_loops():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    data = HeteroData()
    data['v'].num_nodes = 3
    data['w'].num_nodes = 3
    data['v', 'v'].edge_index = edge_index
    data['v', 'w'].edge_index = edge_index
    data = AddSelfLoops()(data)
    assert data['v', 'v'].edge_index.tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                                  [1, 0, 2, 1, 0, 1, 2]]
    assert data['v', 'w'].edge_index.tolist() == edge_index.tolist()
pytorch/test/transforms/test_cartesian.py000066400000000000000000000017661444262304300213210ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Cartesian


def test_cartesian():
    assert str(Cartesian()) == 'Cartesian(norm=True, max_value=None)'

    pos = torch.Tensor([[-1, 0], [0, 0], [2, 0]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.Tensor([1, 2, 3, 4])

    data = Data(edge_index=edge_index, pos=pos)
    data = Cartesian(norm=False)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[-1, 0], [1, 0], [-2, 0], [2, 0]]

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = Cartesian(norm=True)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[1, 0.25, 0.5], [2, 0.75, 0.5],
                                       [3, 0, 0.5], [4, 1, 0.5]]
pytorch/test/transforms/test_center.py000066400000000000000000000005631444262304300206220ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Center


def test_center():
    transform = Center()
    assert str(transform) == 'Center()'

    pos = torch.Tensor([[0, 0], [2, 0], [4, 0]])
    data = Data(pos=pos)

    data = transform(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[-2, 0], [0, 0], [2, 0]]
pytorch/test/transforms/test_compose.py000066400000000000000000000024141444262304300210040ustar00rootroot00000000000000import torch

import torch_geometric.transforms as T
from torch_geometric.data import Data


def test_compose():
    transform = T.Compose([T.Center(), T.AddSelfLoops()])
    assert str(transform) == ('Compose([\n'
                              '  Center(),\n'
                              '  AddSelfLoops()\n'
                              '])')

    pos = torch.Tensor([[0, 0], [2, 0], [4, 0]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    data = Data(edge_index=edge_index, pos=pos)
    data = transform(data)
    assert len(data) == 2
    assert data.pos.tolist() == [[-2, 0], [0, 0], [2, 0]]
    assert data.edge_index.size() == (2, 7)


def test_compose_filters():
    filter_fn = T.ComposeFilters([
        lambda d: d.num_nodes > 2,
        lambda d: d.num_edges > 2,
    ])
    assert str(filter_fn)[:16] == 'ComposeFilters(['

    data1 = Data(x=torch.arange(3))
    assert not filter_fn(data1)

    data2 = Data(x=torch.arange(2), edge_index=torch.tensor([
        [0, 0, 1],
        [0, 1, 1],
    ]))
    assert not filter_fn(data2)

    data3 = Data(x=torch.arange(3), edge_index=torch.tensor([
        [0, 0, 1],
        [0, 1, 1],
    ]))
    assert filter_fn(data3)

    # Test tuple of data objects:
    assert filter_fn((data1, data2, data3)) is False
pytorch/test/transforms/test_constant.py000066400000000000000000000022431444262304300211700ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import Constant


def test_constant():
    assert str(Constant()) == 'Constant(value=1.0)'

    x = torch.tensor([[-1, 0], [0, 0], [2, 0]], dtype=torch.float)
    edge_index = torch.tensor([[0, 1], [1, 2]])

    data = Data(edge_index=edge_index, num_nodes=3)
    data = Constant()(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.x.tolist() == [[1], [1], [1]]
    assert data.num_nodes == 3

    data = Data(edge_index=edge_index, x=x)
    data = Constant()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.x.tolist() == [[-1, 0, 1], [0, 0, 1], [2, 0, 1]]

    data = HeteroData()
    data['v'].x = x
    data = Constant()(data)
    assert len(data) == 1
    assert data['v'].x.tolist() == [[-1, 0, 1], [0, 0, 1], [2, 0, 1]]

    data = HeteroData()
    data['v'].x = x
    data['w'].x = x
    data = Constant(node_types='w')(data)
    assert len(data) == 1
    assert data['v'].x.tolist() == x.tolist()
    assert data['w'].x.tolist() == [[-1, 0, 1], [0, 0, 1], [2, 0, 1]]
pytorch/test/transforms/test_delaunay.py000066400000000000000000000017041444262304300211420ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Delaunay


def test_delaunay():
    assert str(Delaunay()) == 'Delaunay()'

    pos = torch.tensor([[-1, -1], [-1, 1], [1, 1], [1, -1]], dtype=torch.float)
    data = Data(pos=pos)
    data = Delaunay()(data)
    assert len(data) == 2
    assert data.face.tolist() == [[3, 1], [1, 3], [0, 2]]

    pos = torch.tensor([[-1, -1], [-1, 1], [1, 1]], dtype=torch.float)
    data = Data(pos=pos)
    data = Delaunay()(data)
    assert len(data) == 2
    assert data.face.tolist() == [[0], [1], [2]]

    pos = torch.tensor([[-1, -1], [1, 1]], dtype=torch.float)
    data = Data(pos=pos)
    data = Delaunay()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]

    pos = torch.tensor([[-1, -1]], dtype=torch.float)
    data = Data(pos=pos)
    data = Delaunay()(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[], []]
pytorch/test/transforms/test_distance.py000066400000000000000000000016451444262304300211360ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Distance


def test_distance():
    assert str(Distance()) == 'Distance(norm=True, max_value=None)'

    pos = torch.Tensor([[-1, 0], [0, 0], [2, 0]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.Tensor([1, 1, 1, 1])

    data = Data(edge_index=edge_index, pos=pos)
    data = Distance(norm=False)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[1], [1], [2], [2]]

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = Distance(norm=True)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[1, 0.5], [1, 0.5], [1, 1], [1, 1]]
pytorch/test/transforms/test_face_to_edge.py000066400000000000000000000010121444262304300217140ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import FaceToEdge


def test_face_to_edge():
    transform = FaceToEdge()
    assert str(transform) == 'FaceToEdge()'

    face = torch.tensor([[0, 0], [1, 1], [2, 3]])
    data = Data(face=face, num_nodes=4)

    data = transform(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 0, 0, 1, 1, 1, 2, 2, 3, 3],
                                        [1, 2, 3, 0, 2, 3, 0, 1, 0, 1]]
    assert data.num_nodes == 4
pytorch/test/transforms/test_feature_propagation.py000066400000000000000000000020211444262304300233670ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import FeaturePropagation, ToSparseTensor


def test_feature_propagation():
    x = torch.randn(6, 4)
    x[0, 1] = float('nan')
    x[2, 3] = float('nan')
    missing_mask = torch.isnan(x)
    edge_index = torch.tensor([[0, 1, 0, 4, 1, 4, 2, 3, 3, 5],
                               [1, 0, 4, 0, 4, 1, 3, 2, 5, 3]])

    transform = FeaturePropagation(missing_mask)
    assert str(transform) == ('FeaturePropagation(missing_features=8.3%, '
                              'num_iterations=40)')

    data1 = Data(x=x, edge_index=edge_index)
    assert torch.isnan(data1.x).sum() == 2
    data1 = FeaturePropagation(missing_mask)(data1)
    assert torch.isnan(data1.x).sum() == 0
    assert data1.x.size() == x.size()

    data2 = Data(x=x, edge_index=edge_index)
    assert torch.isnan(data2.x).sum() == 2
    data2 = ToSparseTensor()(data2)
    data2 = transform(data2)
    assert torch.isnan(data2.x).sum() == 0
    assert torch.allclose(data1.x, data2.x)
pytorch/test/transforms/test_fixed_points.py000066400000000000000000000040211444262304300220260ustar00rootroot00000000000000from copy import copy

import torch

from torch_geometric.data import Data
from torch_geometric.transforms import FixedPoints


def test_fixed_points():
    assert str(FixedPoints(1024)) == 'FixedPoints(1024, replace=True)'

    data = Data(
        pos=torch.randn(100, 3),
        x=torch.randn(100, 16),
        y=torch.randn(1),
        edge_attr=torch.randn(100, 3),
        num_nodes=100,
    )

    out = FixedPoints(50, replace=True)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (50, 3)
    assert out.x.size() == (50, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 50

    out = FixedPoints(200, replace=True)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (200, 3)
    assert out.x.size() == (200, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 200

    out = FixedPoints(50, replace=False, allow_duplicates=False)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (50, 3)
    assert out.x.size() == (50, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 50

    out = FixedPoints(200, replace=False, allow_duplicates=False)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (100, 3)
    assert out.x.size() == (100, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 100

    out = FixedPoints(50, replace=False, allow_duplicates=True)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (50, 3)
    assert out.x.size() == (50, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 50

    out = FixedPoints(200, replace=False, allow_duplicates=True)(copy(data))
    assert len(out) == 5
    assert out.pos.size() == (200, 3)
    assert out.x.size() == (200, 16)
    assert out.y.size() == (1, )
    assert out.edge_attr.size() == (100, 3)
    assert out.num_nodes == 200
pytorch/test/transforms/test_gcn_norm.py000066400000000000000000000034561444262304300211500ustar00rootroot00000000000000import torch

import torch_geometric.typing
from torch_geometric.data import Data
from torch_geometric.transforms import GCNNorm
from torch_geometric.typing import SparseTensor


def test_gcn_norm():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.ones(edge_index.size(1))

    transform = GCNNorm()
    assert str(transform) == 'GCNNorm(add_self_loops=True)'

    expected_edge_index = [[0, 1, 1, 2, 0, 1, 2], [1, 0, 2, 1, 0, 1, 2]]
    expected_edge_weight = torch.tensor(
        [0.4082, 0.4082, 0.4082, 0.4082, 0.5000, 0.3333, 0.5000])

    data = Data(edge_index=edge_index, edge_weight=edge_weight, num_nodes=3)
    data = transform(data)
    assert len(data) == 3
    assert data.num_nodes == 3
    assert data.edge_index.tolist() == expected_edge_index
    assert torch.allclose(data.edge_weight, expected_edge_weight, atol=1e-4)

    data = Data(edge_index=edge_index, num_nodes=3)
    data = transform(data)
    assert len(data) == 3
    assert data.num_nodes == 3
    assert data.edge_index.tolist() == expected_edge_index
    assert torch.allclose(data.edge_weight, expected_edge_weight, atol=1e-4)

    # For `SparseTensor`, expected outputs will be sorted:
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        expected_edge_index = [[0, 0, 1, 1, 1, 2, 2], [0, 1, 0, 1, 2, 1, 2]]
        expected_edge_weight = torch.tensor(
            [0.500, 0.4082, 0.4082, 0.3333, 0.4082, 0.4082, 0.5000])

        adj_t = SparseTensor.from_edge_index(edge_index, edge_weight).t()
        data = Data(adj_t=adj_t)
        data = transform(data)
        assert len(data) == 1
        row, col, value = data.adj_t.coo()
        assert row.tolist() == expected_edge_index[0]
        assert col.tolist() == expected_edge_index[1]
        assert torch.allclose(value, expected_edge_weight, atol=1e-4)
pytorch/test/transforms/test_gdc.py000066400000000000000000000077131444262304300201030ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import withPackage
from torch_geometric.transforms import GDC
from torch_geometric.utils import to_dense_adj


@withPackage('numba')
def test_gdc():
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 2, 3, 3, 4],
                               [1, 2, 0, 2, 0, 1, 3, 2, 4, 3]])

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='sym',
              normalization_out='sym',
              diffusion_kwargs=dict(method='ppr', alpha=0.15),
              sparsification_kwargs=dict(method='threshold',
                                         avg_degree=2), exact=True)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    assert torch.all(mat >= -1e-8)
    assert torch.allclose(mat, mat.t(), atol=1e-4)

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='sym',
              normalization_out='sym',
              diffusion_kwargs=dict(method='heat', t=10),
              sparsification_kwargs=dict(method='threshold',
                                         avg_degree=2), exact=True)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    assert torch.all(mat >= -1e-8)
    assert torch.allclose(mat, mat.t(), atol=1e-4)

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='col',
              normalization_out='col',
              diffusion_kwargs=dict(method='heat', t=10),
              sparsification_kwargs=dict(method='topk', k=2,
                                         dim=0), exact=True)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    col_sum = mat.sum(0)
    assert torch.all(mat >= -1e-8)
    assert torch.all(
        torch.isclose(col_sum, torch.tensor(1.0))
        | torch.isclose(col_sum, torch.tensor(0.0)))
    assert torch.all((~torch.isclose(mat, torch.tensor(0.0))).sum(0) == 2)

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='row',
              normalization_out='row',
              diffusion_kwargs=dict(method='heat', t=5),
              sparsification_kwargs=dict(method='topk', k=2,
                                         dim=1), exact=True)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    row_sum = mat.sum(1)
    assert torch.all(mat >= -1e-8)
    assert torch.all(
        torch.isclose(row_sum, torch.tensor(1.0))
        | torch.isclose(row_sum, torch.tensor(0.0)))
    assert torch.all((~torch.isclose(mat, torch.tensor(0.0))).sum(1) == 2)

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='row',
              normalization_out='row',
              diffusion_kwargs=dict(method='coeff', coeffs=[0.8, 0.3, 0.1]),
              sparsification_kwargs=dict(method='threshold',
                                         eps=0.1), exact=True)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    row_sum = mat.sum(1)
    assert torch.all(mat >= -1e-8)
    assert torch.all(
        torch.isclose(row_sum, torch.tensor(1.0))
        | torch.isclose(row_sum, torch.tensor(0.0)))

    data = Data(edge_index=edge_index, num_nodes=5)
    gdc = GDC(self_loop_weight=1, normalization_in='sym',
              normalization_out='col',
              diffusion_kwargs=dict(method='ppr', alpha=0.15, eps=1e-4),
              sparsification_kwargs=dict(method='threshold',
                                         avg_degree=2), exact=False)
    data = gdc(data)
    mat = to_dense_adj(data.edge_index, edge_attr=data.edge_attr).squeeze()
    col_sum = mat.sum(0)
    assert torch.all(mat >= -1e-8)
    assert torch.all(
        torch.isclose(col_sum, torch.tensor(1.0))
        | torch.isclose(col_sum, torch.tensor(0.0)))
pytorch/test/transforms/test_generate_normals.py000066400000000000000000000012711444262304300226640ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import GenerateMeshNormals


def test_generate_normals():
    transform = GenerateMeshNormals()
    assert str(transform) == 'GenerateMeshNormals()'

    pos = torch.Tensor([
        [0, 0, 0],
        [-2, 1, 0],
        [-1, 1, 0],
        [0, 1, 0],
        [1, 1, 0],
        [2, 1, 0],
    ])
    face = torch.tensor([
        [0, 0, 0, 0],
        [1, 2, 3, 4],
        [2, 3, 4, 5],
    ])

    data = transform(Data(pos=pos, face=face))
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.face.tolist() == face.tolist()
    assert data.norm.tolist() == [[0, 0, -1]] * 6
pytorch/test/transforms/test_grid_sampling.py000066400000000000000000000012221444262304300221520ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import withPackage
from torch_geometric.transforms import GridSampling


@withPackage('torch_cluster')
def test_grid_sampling():
    assert str(GridSampling(5)) == 'GridSampling(size=5)'

    pos = torch.Tensor([[0, 2], [3, 2], [3, 2], [2, 8], [2, 6]])
    y = torch.tensor([0, 1, 1, 2, 2])
    batch = torch.tensor([0, 0, 0, 0, 0])

    data = Data(pos=pos, y=y, batch=batch)
    data = GridSampling(size=5, start=0)(data)
    assert len(data) == 3
    assert data.pos.tolist() == [[2, 2], [2, 7]]
    assert data.y.tolist() == [1, 2]
    assert data.batch.tolist() == [0, 0]
pytorch/test/transforms/test_knn_graph.py000066400000000000000000000013361444262304300213100ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import withPackage
from torch_geometric.transforms import KNNGraph


@withPackage('torch_cluster')
def test_knn_graph():
    assert str(KNNGraph()) == 'KNNGraph(k=6)'

    pos = torch.Tensor([[0, 0], [1, 0], [2, 0], [0, 1], [-2, 0], [0, -2]])

    expected_row = [0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 3, 4, 4, 5, 5]
    expected_col = [1, 2, 3, 4, 5, 0, 2, 3, 5, 0, 1, 0, 1, 4, 0, 3, 0, 1]

    data = Data(pos=pos)
    data = KNNGraph(k=2, force_undirected=True)(data)
    assert len(data) == 2
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index[0].tolist() == expected_row
    assert data.edge_index[1].tolist() == expected_col
pytorch/test/transforms/test_laplacian_lambda_max.py000066400000000000000000000025531444262304300234340ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LaplacianLambdaMax


def test_laplacian_lambda_max():
    out = str(LaplacianLambdaMax())
    assert out == 'LaplacianLambdaMax(normalization=None)'

    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], dtype=torch.long)
    edge_attr = torch.tensor([1, 1, 2, 2], dtype=torch.float)

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    out = LaplacianLambdaMax(normalization=None, is_undirected=True)(data)
    assert len(out) == 4
    assert torch.allclose(torch.tensor(out.lambda_max), torch.tensor(4.732049))

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    out = LaplacianLambdaMax(normalization='sym', is_undirected=True)(data)
    assert len(out) == 4
    assert torch.allclose(torch.tensor(out.lambda_max), torch.tensor(2.0))

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    out = LaplacianLambdaMax(normalization='rw', is_undirected=True)(data)
    assert len(out) == 4
    assert torch.allclose(torch.tensor(out.lambda_max), torch.tensor(2.0))

    data = Data(edge_index=edge_index, edge_attr=torch.randn(4, 2),
                num_nodes=3)
    out = LaplacianLambdaMax(normalization=None)(data)
    assert len(out) == 4
    assert torch.allclose(torch.tensor(out.lambda_max), torch.tensor(3.0))
pytorch/test/transforms/test_largest_connected_components.py000066400000000000000000000032351444262304300252710ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LargestConnectedComponents


def test_largest_connected_components():
    assert str(LargestConnectedComponents()) == 'LargestConnectedComponents(1)'

    edge_index = torch.tensor([
        [0, 0, 1, 1, 2, 2, 2, 3, 3, 4, 5, 6, 8, 9],
        [1, 2, 0, 2, 0, 1, 3, 2, 4, 3, 6, 7, 9, 8],
    ])
    data = Data(edge_index=edge_index, num_nodes=10)

    # Testing without `connection` specified:
    transform = LargestConnectedComponents(num_components=2)
    out = transform(data)
    assert out.num_nodes == 8
    assert out.edge_index.tolist() == data.edge_index[:, :12].tolist()

    # Testing with `connection = strong`:
    transform = LargestConnectedComponents(num_components=2,
                                           connection='strong')
    out = transform(data)
    assert out.num_nodes == 7
    assert out.edge_index.tolist() == [[0, 0, 1, 1, 2, 2, 2, 3, 3, 4, 5, 6],
                                       [1, 2, 0, 2, 0, 1, 3, 2, 4, 3, 6, 5]]

    edge_index = torch.tensor([
        [0, 1, 2, 3, 3, 4],
        [1, 0, 3, 2, 4, 3],
    ])
    data = Data(edge_index=edge_index, num_nodes=5)

    # Testing without `num_components` and `connection` specified:
    transform = LargestConnectedComponents()
    out = transform(data)
    assert out.num_nodes == 3
    assert out.edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]

    # Testing with larger `num_components` than actual number of components:
    transform = LargestConnectedComponents(num_components=3)
    out = transform(data)
    assert out.num_nodes == 5
    assert out.edge_index.tolist() == data.edge_index.tolist()
pytorch/test/transforms/test_line_graph.py000066400000000000000000000022311444262304300214440ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LineGraph


def test_line_graph():
    transform = LineGraph()
    assert str(transform) == 'LineGraph()'

    # Directed.
    edge_index = torch.tensor([
        [0, 1, 2, 2, 3],
        [1, 2, 0, 3, 0],
    ])
    data = Data(edge_index=edge_index, num_nodes=4)
    data = transform(data)
    assert data.edge_index.tolist() == [[0, 1, 1, 2, 3, 4], [1, 2, 3, 0, 4, 0]]
    assert data.num_nodes == data.edge_index.max().item() + 1

    # Undirected.
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 2, 2, 3, 3, 3, 4, 4],
                               [1, 2, 3, 0, 4, 0, 3, 0, 2, 4, 1, 3]])
    edge_attr = torch.ones(edge_index.size(1))
    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=5)
    data = transform(data)
    assert data.edge_index.max().item() + 1 == data.x.size(0)
    assert data.edge_index.tolist() == [
        [0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 4, 4, 4, 5, 5, 5],
        [1, 2, 3, 0, 2, 4, 0, 1, 4, 5, 0, 5, 1, 2, 5, 2, 3, 4],
    ]
    assert data.x.tolist() == [2, 2, 2, 2, 2, 2]
    assert data.num_nodes == data.edge_index.max().item() + 1
pytorch/test/transforms/test_linear_transformation.py000066400000000000000000000013241444262304300237360ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LinearTransformation


@pytest.mark.parametrize('matrix', [
    [[2., 0.], [0., 2.]],
    torch.tensor([[2., 0.], [0., 2.]]),
])
def test_linear_transformation(matrix):
    pos = torch.Tensor([[-1, 1], [-3, 0], [2, -1]])

    transform = LinearTransformation(matrix)
    assert str(transform) == ('LinearTransformation(\n'
                              '[[2. 0.]\n'
                              ' [0. 2.]]\n'
                              ')')

    out = transform(Data(pos=pos))
    assert len(out) == 1
    assert out.pos.tolist() == [[-2, 2], [-6, 0], [4, -2]]

    out = transform(Data())
    assert len(out) == 0
pytorch/test/transforms/test_local_cartesian.py000066400000000000000000000020561444262304300224640ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LocalCartesian


def test_local_cartesian():
    transform = LocalCartesian()
    assert str(transform) == 'LocalCartesian()'

    pos = torch.Tensor([[-1, 0], [0, 0], [2, 0]])
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.Tensor([1, 2, 3, 4])
    data = Data(edge_index=edge_index, pos=pos)

    data = transform(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[0.25, 0.5], [1.0, 0.5], [0.0, 0.5],
                                       [1.0, 0.5]]

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = transform(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[1, 0.25, 0.5], [2, 1.0, 0.5],
                                       [3, 0.0, 0.5], [4, 1.0, 0.5]]
pytorch/test/transforms/test_local_degree_profile.py000066400000000000000000000014751444262304300234720ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import LocalDegreeProfile


def test_target_indegree():
    assert str(LocalDegreeProfile()) == 'LocalDegreeProfile()'

    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    x = torch.Tensor([[1], [1], [1], [1]])  # One isolated node.

    expected = torch.tensor([
        [1, 2, 2, 2, 0],
        [2, 1, 1, 1, 0],
        [1, 2, 2, 2, 0],
        [0, 0, 0, 0, 0],
    ], dtype=torch.float)

    data = Data(edge_index=edge_index, num_nodes=x.size(0))
    data = LocalDegreeProfile()(data)
    assert torch.allclose(data.x, expected, atol=1e-2)

    data = Data(edge_index=edge_index, x=x)
    data = LocalDegreeProfile()(data)
    assert torch.allclose(data.x[:, :1], x)
    assert torch.allclose(data.x[:, 1:], expected, atol=1e-2)
pytorch/test/transforms/test_mask_transform.py000066400000000000000000000062721444262304300223730ustar00rootroot00000000000000import copy

import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import IndexToMask, MaskToIndex


def test_index_to_mask():
    assert str(IndexToMask()) == ('IndexToMask(attrs=None, sizes=None, '
                                  'replace=False)')

    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4],
                               [1, 0, 2, 1, 3, 2, 4, 3]])
    train_index = torch.arange(0, 3)
    test_index = torch.arange(3, 5)
    data = Data(edge_index=edge_index, train_index=train_index,
                test_index=test_index, num_nodes=5)

    out = IndexToMask(replace=True)(copy.copy(data))
    assert len(out) == len(data)
    assert out.train_mask.tolist() == [True, True, True, False, False]
    assert out.test_mask.tolist() == [False, False, False, True, True]

    out = IndexToMask(replace=False)(copy.copy(data))
    assert len(out) == len(data) + 2

    out = IndexToMask(sizes=6, replace=True)(copy.copy(data))
    assert out.train_mask.tolist() == [True, True, True, False, False, False]
    assert out.test_mask.tolist() == [False, False, False, True, True, False]

    out = IndexToMask(attrs='train_index')(copy.copy(data))
    assert len(out) == len(data) + 1
    assert 'train_index' in out
    assert 'train_mask' in out
    assert 'test_index' in out
    assert 'test_mask' not in out


def test_mask_to_index():
    assert str(MaskToIndex()) == 'MaskToIndex(attrs=None, replace=False)'

    train_mask = torch.tensor([True, True, True, False, False])
    test_mask = torch.tensor([False, False, False, True, True])
    data = Data(train_mask=train_mask, test_mask=test_mask)

    out = MaskToIndex(replace=True)(copy.copy(data))
    assert len(out) == len(data)
    assert out.train_index.tolist() == [0, 1, 2]
    assert out.test_index.tolist() == [3, 4]

    out = MaskToIndex(replace=False)(copy.copy(data))
    assert len(out) == len(data) + 2

    out = MaskToIndex(attrs='train_mask')(copy.copy(data))
    assert len(out) == len(data) + 1
    assert 'train_mask' in out
    assert 'train_index' in out
    assert 'test_mask' in out
    assert 'test_index' not in out


def test_hetero_index_to_mask():
    data = HeteroData()
    data['u'].train_index = torch.arange(0, 3)
    data['u'].test_index = torch.arange(3, 5)
    data['u'].num_nodes = 5

    data['v'].train_index = torch.arange(0, 3)
    data['v'].test_index = torch.arange(3, 5)
    data['v'].num_nodes = 5

    out = IndexToMask()(copy.copy(data))
    assert len(out) == len(data) + 2
    assert 'train_mask' in out['u']
    assert 'test_mask' in out['u']
    assert 'train_mask' in out['v']
    assert 'test_mask' in out['v']


def test_hetero_mask_to_index():
    data = HeteroData()
    data['u'].train_mask = torch.tensor([True, True, True, False, False])
    data['u'].test_mask = torch.tensor([False, False, False, True, True])

    data['v'].train_mask = torch.tensor([True, True, True, False, False])
    data['v'].test_mask = torch.tensor([False, False, False, True, True])

    out = MaskToIndex()(copy.copy(data))
    assert len(out) == len(data) + 2
    assert 'train_index' in out['u']
    assert 'test_index' in out['u']
    assert 'train_index' in out['v']
    assert 'test_index' in out['v']
pytorch/test/transforms/test_node_property_split.py000066400000000000000000000022551444262304300234460ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.datasets import graph_generator
from torch_geometric.testing import withPackage
from torch_geometric.transforms import NodePropertySplit


@withPackage('networkx')
@pytest.mark.parametrize('property_name', [
    'popularity',
    'locality',
    'density',
])
def test_node_property_split(property_name):
    ratios = [0.3, 0.1, 0.1, 0.2, 0.3]

    transform = NodePropertySplit(property_name, ratios)
    assert str(transform) == f'NodePropertySplit({property_name})'

    data = graph_generator.ERGraph(num_nodes=100, edge_prob=0.4)()
    data = transform(data)

    node_ids = []
    for name, ratio in zip([
            'id_train_mask',
            'id_val_mask',
            'id_test_mask',
            'ood_val_mask',
            'ood_test_mask',
    ], ratios):
        assert data[name].dtype == torch.bool
        assert data[name].size() == (100, )
        assert int(data[name].sum()) == 100 * ratio
        node_ids.extend(data[name].nonzero().view(-1).tolist())

    # Check that masks are non-intersecting and cover all nodes:
    node_ids = torch.tensor(node_ids)
    assert node_ids.numel() == torch.unique(node_ids).numel() == 100
pytorch/test/transforms/test_normalize_features.py000066400000000000000000000014611444262304300232360ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import NormalizeFeatures


def test_normalize_scale():
    transform = NormalizeFeatures()
    assert str(transform) == 'NormalizeFeatures()'

    x = torch.tensor([[1, 0, 1], [0, 1, 0], [0, 0, 0]], dtype=torch.float)
    data = Data(x=x)

    data = transform(data)
    assert len(data) == 1
    assert data.x.tolist() == [[0.5, 0, 0.5], [0, 1, 0], [0, 0, 0]]


def test_hetero_normalize_scale():
    x = torch.tensor([[1, 0, 1], [0, 1, 0], [0, 0, 0]], dtype=torch.float)

    data = HeteroData()
    data['v'].x = x
    data['w'].x = x
    data = NormalizeFeatures()(data)
    assert data['v'].x.tolist() == [[0.5, 0, 0.5], [0, 1, 0], [0, 0, 0]]
    assert data['w'].x.tolist() == [[0.5, 0, 0.5], [0, 1, 0], [0, 0, 0]]
pytorch/test/transforms/test_normalize_rotation.py000066400000000000000000000024141444262304300232560ustar00rootroot00000000000000from math import sqrt

import torch

from torch_geometric.data import Data
from torch_geometric.transforms import NormalizeRotation


def test_normalize_rotation():
    assert str(NormalizeRotation()) == 'NormalizeRotation()'

    pos = torch.Tensor([[-2, -2], [-1, -1], [0, 0], [1, 1], [2, 2]])
    normal = torch.Tensor([[-1, 1], [-1, 1], [-1, 1], [-1, 1], [-1, 1]])
    data = Data(pos=pos)
    data.normal = normal
    data = NormalizeRotation()(data)
    assert len(data) == 2

    expected_pos = torch.Tensor([
        [-2 * sqrt(2), 0],
        [-sqrt(2), 0],
        [0, 0],
        [sqrt(2), 0],
        [2 * sqrt(2), 0],
    ])
    expected_normal = [[0, 1], [0, 1], [0, 1], [0, 1], [0, 1]]

    assert torch.allclose(data.pos, expected_pos, atol=1e-04)
    assert data.normal.tolist() == expected_normal

    data = Data(pos=pos)
    data.normal = normal
    data = NormalizeRotation(max_points=3)(data)
    assert len(data) == 2

    assert torch.allclose(data.pos, expected_pos, atol=1e-04)
    assert data.normal.tolist() == expected_normal

    data = Data(pos=pos)
    data.normal = normal
    data = NormalizeRotation(sort=True)(data)
    assert len(data) == 2

    assert torch.allclose(data.pos, expected_pos, atol=1e-04)
    assert data.normal.tolist() == expected_normal
pytorch/test/transforms/test_normalize_scale.py000066400000000000000000000006231444262304300225060ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import NormalizeScale


def test_normalize_scale():
    transform = NormalizeScale()
    assert str(transform) == 'NormalizeScale()'

    pos = torch.randn((10, 3))
    data = Data(pos=pos)

    data = transform(data)
    assert len(data) == 1
    assert data.pos.min().item() > -1
    assert data.pos.max().item() < 1
pytorch/test/transforms/test_one_hot_degree.py000066400000000000000000000016541444262304300223120ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import OneHotDegree


def test_one_hot_degree():
    assert str(OneHotDegree(max_degree=3)) == 'OneHotDegree(3)'

    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    x = torch.Tensor([1, 1, 1, 1])

    data = Data(edge_index=edge_index, num_nodes=4)
    data = OneHotDegree(max_degree=3)(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.x.tolist() == [[0, 0, 0, 1], [0, 1, 0, 0], [0, 1, 0, 0],
                               [0, 1, 0, 0]]
    assert data.num_nodes == 4

    data = Data(edge_index=edge_index, x=x)
    data = OneHotDegree(max_degree=3)(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.x.tolist() == [[1, 0, 0, 0, 1], [1, 0, 1, 0, 0],
                               [1, 0, 1, 0, 0], [1, 0, 1, 0, 0]]
pytorch/test/transforms/test_pad.py000066400000000000000000000511221444262304300201030ustar00rootroot00000000000000import numbers
from copy import deepcopy
from typing import Dict, Generator, List, Optional, Tuple, Union

import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.datasets import FakeDataset, FakeHeteroDataset
from torch_geometric.transforms import Pad
from torch_geometric.transforms.pad import (
    AttrNamePadding,
    EdgeTypePadding,
    NodeTypePadding,
    Padding,
    UniformPadding,
)
from torch_geometric.typing import EdgeType, NodeType


def fake_data() -> Data:
    return FakeDataset(avg_num_nodes=10, avg_degree=5, edge_dim=2)[0]


def fake_hetero_data(node_types=2, edge_types=5) -> HeteroData:
    return FakeHeteroDataset(num_node_types=node_types,
                             num_edge_types=edge_types, avg_num_nodes=10,
                             edge_dim=2)[0]


def _generate_homodata_node_attrs(data: Data) -> Generator[str, None, None]:
    for attr in data.keys:
        if data.is_node_attr(attr):
            yield attr


def _generate_homodata_edge_attrs(data: Data) -> Generator[str, None, None]:
    for attr in data.keys:
        if data.is_edge_attr(attr):
            yield attr


def _generate_heterodata_nodes(
    data: HeteroData
) -> Generator[Tuple[NodeType, str, torch.Tensor], None, None]:
    for node_type, store in data.node_items():
        for attr in store.keys():
            yield node_type, attr


def _generate_heterodata_edges(
    data: HeteroData
) -> Generator[Tuple[EdgeType, str, torch.Tensor], None, None]:
    for edge_type, store in data.edge_items():
        for attr in store.keys():
            yield edge_type, attr


def _check_homo_data_nodes(
    original: Data,
    padded: Data,
    max_num_nodes: Union[int, Dict[NodeType, int]],
    node_pad_value: Optional[Padding] = None,
    is_mask_available: bool = False,
    exclude_keys: Optional[List[str]] = None,
):
    assert padded.num_nodes == max_num_nodes

    compare_pad_start_idx = original.num_nodes

    if is_mask_available:
        assert padded.pad_node_mask.numel() == padded.num_nodes
        assert torch.all(padded.pad_node_mask[:compare_pad_start_idx])
        assert not torch.any(padded.pad_node_mask[compare_pad_start_idx:])

    for attr in _generate_homodata_node_attrs(original):
        if attr in exclude_keys:
            assert attr not in padded.keys
            continue

        assert attr in padded.keys

        if not isinstance(padded[attr], torch.Tensor):
            continue

        assert padded[attr].shape[0] == max_num_nodes

        # Check values in padded area.
        pad_value = node_pad_value.get_value(
            None, attr) if node_pad_value is not None else 0.0
        assert all(
            i == pad_value
            for i in torch.flatten(padded[attr][compare_pad_start_idx:]))

        # Check values in non-padded area.
        assert torch.equal(original[attr],
                           padded[attr][:compare_pad_start_idx])


def _check_homo_data_edges(
    original: Data,
    padded: Data,
    max_num_edges: Optional[int] = None,
    edge_pad_value: Optional[Padding] = None,
    is_mask_available: bool = False,
    exclude_keys: Optional[List[str]] = None,
):
    # Check edge index attribute.
    if max_num_edges is None:
        max_num_edges = padded.num_nodes**2
    assert padded.num_edges == max_num_edges
    assert padded.edge_index.shape[1] == max_num_edges
    assert padded.edge_index.shape[1] == max_num_edges

    compare_pad_start_idx = original.num_edges
    expected_node = original.num_nodes

    # Check values in padded area.
    assert all(
        padded.edge_index[0, i] == padded.edge_index[1, i] == expected_node
        for i in range(compare_pad_start_idx, max_num_edges))
    # Check values in non-padded area.
    assert torch.equal(original.edge_index,
                       padded.edge_index[:, :compare_pad_start_idx])

    if is_mask_available:
        assert padded.pad_edge_mask.numel() == padded.num_edges
        assert torch.all(padded.pad_edge_mask[:compare_pad_start_idx])
        assert not torch.any(padded.pad_edge_mask[compare_pad_start_idx:])

    # Check other attributes.
    for attr in _generate_homodata_edge_attrs(original):
        if attr == 'edge_index':
            continue
        if attr in exclude_keys:
            assert attr not in padded.keys
            continue

        assert attr in padded.keys

        if not isinstance(padded[attr], torch.Tensor):
            continue

        assert padded[attr].shape[0] == max_num_edges

        # Check values in padded area.
        pad_value = edge_pad_value.get_value(
            None, attr) if edge_pad_value is not None else 0.0
        assert all(
            i == pad_value
            for i in torch.flatten(padded[attr][compare_pad_start_idx:, :]))

        # Check values in non-padded area.
        assert torch.equal(original[attr],
                           padded[attr][:compare_pad_start_idx, :])


def _check_hetero_data_nodes(
    original: HeteroData,
    padded: HeteroData,
    max_num_nodes: Union[int, Dict[NodeType, int]],
    node_pad_value: Optional[Padding] = None,
    is_mask_available: bool = False,
    exclude_keys: Optional[List[str]] = None,
):
    if is_mask_available:
        for store in padded.node_stores:
            assert 'pad_node_mask' in store

    expected_nodes = max_num_nodes

    for node_type, attr in _generate_heterodata_nodes(original):
        if attr in exclude_keys:
            assert attr not in padded[node_type].keys()
            continue

        assert attr in padded[node_type].keys()

        if not isinstance(padded[node_type][attr], torch.Tensor):
            continue

        compare_pad_start_idx = original[node_type].num_nodes
        padded_tensor = padded[node_type][attr]

        if attr == 'pad_node_mask':
            assert padded_tensor.numel() == padded[node_type].num_nodes
            assert torch.all(padded_tensor[:compare_pad_start_idx])
            assert not torch.any(padded_tensor[compare_pad_start_idx:])
            continue

        original_tensor = original[node_type][attr]

        # Check the number of nodes.
        if isinstance(max_num_nodes, dict):
            expected_nodes = max_num_nodes[node_type]

        assert padded_tensor.shape[0] == expected_nodes

        compare_pad_start_idx = original_tensor.shape[0]
        pad_value = node_pad_value.get_value(
            node_type, attr) if node_pad_value is not None else 0.0
        assert all(
            i == pad_value
            for i in torch.flatten(padded_tensor[compare_pad_start_idx:]))
        # Compare non-padded area with the original.
        assert torch.equal(original_tensor,
                           padded_tensor[:compare_pad_start_idx])


def _check_hetero_data_edges(
    original: HeteroData,
    padded: HeteroData,
    max_num_edges: Optional[Union[int, Dict[EdgeType, int]]] = None,
    edge_pad_value: Optional[Padding] = None,
    is_mask_available: bool = False,
    exclude_keys: Optional[List[str]] = None,
):
    if is_mask_available:
        for store in padded.edge_stores:
            assert 'pad_edge_mask' in store

    for edge_type, attr in _generate_heterodata_edges(padded):
        if attr in exclude_keys:
            assert attr not in padded[edge_type].keys()
            continue

        assert attr in padded[edge_type].keys()

        if not isinstance(padded[edge_type][attr], torch.Tensor):
            continue

        compare_pad_start_idx = original[edge_type].num_edges
        padded_tensor = padded[edge_type][attr]

        if attr == 'pad_edge_mask':
            assert padded_tensor.numel() == padded[edge_type].num_edges
            assert torch.all(padded_tensor[:compare_pad_start_idx])
            assert not torch.any(padded_tensor[compare_pad_start_idx:])
            continue

        original_tensor = original[edge_type][attr]

        if isinstance(max_num_edges, numbers.Number):
            expected_num_edges = max_num_edges
        elif max_num_edges is None or edge_type not in max_num_edges.keys():
            v1, _, v2 = edge_type
            expected_num_edges = padded[v1].num_nodes * padded[v2].num_nodes
        else:
            expected_num_edges = max_num_edges[edge_type]

        if attr == 'edge_index':
            # Check the number of edges.
            assert padded_tensor.shape[1] == expected_num_edges

            # Check padded area values.
            src_nodes = original[edge_type[0]].num_nodes
            assert all(
                i == src_nodes
                for i in torch.flatten(padded_tensor[0,
                                                     compare_pad_start_idx:]))
            dst_nodes = original[edge_type[2]].num_nodes
            assert all(
                i == dst_nodes
                for i in torch.flatten(padded_tensor[1,
                                                     compare_pad_start_idx:]))

            # Compare non-padded area with the original.
            assert torch.equal(original_tensor,
                               padded_tensor[:, :compare_pad_start_idx])
        else:
            # Check padded area size.
            assert padded_tensor.shape[0] == expected_num_edges

            # Check padded area values.
            pad_value = edge_pad_value.get_value(
                edge_type, attr) if edge_pad_value is not None else 0.0
            assert all(i == pad_value for i in torch.flatten(padded_tensor[
                compare_pad_start_idx:, :]))

            # Compare non-padded area with the original.
            assert torch.equal(original_tensor,
                               padded_tensor[:compare_pad_start_idx, :])


def _check_data(
    original: Union[Data, HeteroData],
    padded: Union[Data, HeteroData],
    max_num_nodes: Union[int, Dict[NodeType, int]],
    max_num_edges: Optional[Union[int, Dict[EdgeType, int]]] = None,
    node_pad_value: Optional[Union[Padding, int, float]] = None,
    edge_pad_value: Optional[Union[Padding, int, float]] = None,
    is_mask_available: bool = False,
    exclude_keys: Optional[List[str]] = None,
):

    if not isinstance(node_pad_value, Padding) and node_pad_value is not None:
        node_pad_value = UniformPadding(node_pad_value)
    if not isinstance(edge_pad_value, Padding) and edge_pad_value is not None:
        edge_pad_value = UniformPadding(edge_pad_value)

    if is_mask_available is None:
        is_mask_available = False

    if exclude_keys is None:
        exclude_keys = []

    if isinstance(original, Data):
        _check_homo_data_nodes(original, padded, max_num_nodes, node_pad_value,
                               is_mask_available, exclude_keys)
        _check_homo_data_edges(original, padded, max_num_edges, edge_pad_value,
                               is_mask_available, exclude_keys)
    else:
        _check_hetero_data_nodes(original, padded, max_num_nodes,
                                 node_pad_value, is_mask_available,
                                 exclude_keys)
        _check_hetero_data_edges(original, padded, max_num_edges,
                                 edge_pad_value, is_mask_available,
                                 exclude_keys)


def test_pad_repr():
    pad_str = 'Pad(max_num_nodes=10, max_num_edges=15, ' \
        'node_pad_value=UniformPadding(value=3.0), ' \
        'edge_pad_value=UniformPadding(value=1.5))'
    assert str(eval(pad_str)) == pad_str


@pytest.mark.parametrize('data', [fake_data(), fake_hetero_data()])
@pytest.mark.parametrize('num_nodes', [32, 64])
@pytest.mark.parametrize('add_pad_mask', [True, False])
def test_pad_auto_edges(data, num_nodes, add_pad_mask):
    original = data
    data = deepcopy(data)
    transform = Pad(max_num_nodes=num_nodes, add_pad_mask=add_pad_mask)

    padded = transform(data)
    _check_data(original, padded, num_nodes, is_mask_available=add_pad_mask)


@pytest.mark.parametrize('num_nodes', [32, 64])
@pytest.mark.parametrize('num_edges', [300, 411])
@pytest.mark.parametrize('add_pad_mask', [True, False])
def test_pad_data_explicit_edges(num_nodes, num_edges, add_pad_mask):
    data = fake_data()
    original = deepcopy(data)
    transform = Pad(max_num_nodes=num_nodes, max_num_edges=num_edges,
                    add_pad_mask=add_pad_mask)

    padded = transform(data)
    _check_data(original, padded, num_nodes, num_edges,
                is_mask_available=add_pad_mask)


@pytest.mark.parametrize('num_nodes', [32, {'v0': 64, 'v1': 36}])
@pytest.mark.parametrize('num_edges', [300, {('v0', 'e0', 'v1'): 397}])
@pytest.mark.parametrize('add_pad_mask', [True, False])
def test_pad_heterodata_explicit_edges(num_nodes, num_edges, add_pad_mask):
    data = fake_hetero_data()
    original = deepcopy(data)
    transform = Pad(max_num_nodes=num_nodes, max_num_edges=num_edges,
                    add_pad_mask=add_pad_mask)

    padded = transform(data)
    _check_data(original, padded, num_nodes, num_edges,
                is_mask_available=add_pad_mask)


@pytest.mark.parametrize('node_pad_value', [10, AttrNamePadding({'x': 3.0})])
@pytest.mark.parametrize('edge_pad_value',
                         [11, AttrNamePadding({'edge_attr': 2.0})])
def test_pad_data_pad_values(node_pad_value, edge_pad_value):
    data = fake_data()
    original = deepcopy(data)
    num_nodes = 32
    transform = Pad(max_num_nodes=num_nodes, node_pad_value=node_pad_value,
                    edge_pad_value=edge_pad_value)
    padded = transform(data)
    _check_data(original, padded, num_nodes, node_pad_value=node_pad_value,
                edge_pad_value=edge_pad_value)


@pytest.mark.parametrize('node_pad_value', [
    UniformPadding(12),
    AttrNamePadding({'x': 0}),
    NodeTypePadding({
        'v0': UniformPadding(12),
        'v1': AttrNamePadding({'x': 7})
    })
])
@pytest.mark.parametrize('edge_pad_value', [
    UniformPadding(13),
    EdgeTypePadding({
        ('v0', 'e0', 'v1'):
        UniformPadding(13),
        ('v1', 'e0', 'v0'):
        AttrNamePadding({'edge_attr': UniformPadding(-1.0)})
    })
])
def test_pad_heterodata_pad_values(node_pad_value, edge_pad_value):
    data = fake_hetero_data()
    original = deepcopy(data)
    num_nodes = 32
    transform = Pad(max_num_nodes=num_nodes, node_pad_value=node_pad_value,
                    edge_pad_value=edge_pad_value)

    padded = transform(data)
    _check_data(original, padded, num_nodes, node_pad_value=node_pad_value,
                edge_pad_value=edge_pad_value)


@pytest.mark.parametrize('data', [fake_data(), fake_hetero_data()])
@pytest.mark.parametrize('add_pad_mask', [True, False])
@pytest.mark.parametrize('exclude_keys', [
    ['y'],
    ['edge_attr'],
    ['y', 'edge_attr'],
])
def test_pad_data_exclude_keys(data, add_pad_mask, exclude_keys):
    original = data
    data = deepcopy(data)
    num_nodes = 32
    transform = Pad(max_num_nodes=num_nodes, add_pad_mask=add_pad_mask,
                    exclude_keys=exclude_keys)

    padded = transform(data)
    _check_data(original, padded, num_nodes, is_mask_available=add_pad_mask,
                exclude_keys=exclude_keys)


@pytest.mark.parametrize('data', [fake_data(), fake_hetero_data(node_types=1)])
def test_pad_invalid_max_num_nodes(data):
    transform = Pad(max_num_nodes=data.num_nodes - 1)

    with pytest.raises(AssertionError,
                       match='The number of nodes after padding'):
        transform(data)


@pytest.mark.parametrize(
    'data',
    [fake_data(), fake_hetero_data(node_types=1, edge_types=1)])
def test_pad_invalid_max_num_edges(data):
    transform = Pad(max_num_nodes=data.num_nodes + 10,
                    max_num_edges=data.num_edges - 1)

    with pytest.raises(AssertionError,
                       match='The number of edges after padding'):
        transform(data)


def test_pad_num_nodes_not_complete():
    data = fake_hetero_data(node_types=2, edge_types=1)
    transform = Pad(max_num_nodes={'v0': 100})

    with pytest.raises(AssertionError, match='The number of v1 nodes'):
        transform(data)


def test_pad_invalid_padding_type():
    with pytest.raises(ValueError, match="to be an integer or float"):
        Pad(max_num_nodes=100, node_pad_value='somestring')
    with pytest.raises(ValueError, match="to be an integer or float"):
        Pad(max_num_nodes=100, edge_pad_value='somestring')


def test_pad_data_non_tensor_attr():
    data = deepcopy(fake_data())
    batch_size = 13
    data.batch_size = batch_size

    transform = Pad(max_num_nodes=100)
    padded = transform(data)
    assert padded.batch_size == batch_size

    exclude_transform = Pad(max_num_nodes=101, exclude_keys=('batch_size', ))
    padded = exclude_transform(data)
    assert 'batch_size' not in padded.keys


@pytest.mark.parametrize('mask_pad_value', [True, False])
def test_pad_node_additional_attr_mask(mask_pad_value):
    data = fake_data()
    mask = torch.randn(data.num_nodes) > 0
    mask_names = ['train_mask', 'test_mask', 'val_mask']
    for mask_name in mask_names:
        setattr(data, mask_name, mask)
    padding_num = 20

    max_num_nodes = int(data.num_nodes) + padding_num
    max_num_edges = data.num_edges + padding_num

    transform = Pad(max_num_nodes, max_num_edges, node_pad_value=0.1,
                    mask_pad_value=mask_pad_value)
    padded = transform(data)
    padded_masks = [getattr(padded, mask_name) for mask_name in mask_names]

    for padded_mask in padded_masks:
        assert padded_mask.ndim == 1
        assert padded_mask.size()[0] == max_num_nodes
        assert torch.all(padded_mask[-padding_num:] == mask_pad_value)


def test_uniform_padding():
    pad_val = 10.0
    p = UniformPadding(pad_val)
    assert p.get_value() == pad_val
    assert p.get_value("v1", "x") == pad_val

    p = UniformPadding()
    assert p.get_value() == 0.0

    with pytest.raises(ValueError, match="to be an integer or float"):
        UniformPadding('')


def test_attr_name_padding():
    x_val = 10.0
    y_val = 15.0
    default = 3.0
    padding_dict = {'x': x_val, 'y': UniformPadding(y_val)}
    padding = AttrNamePadding(padding_dict, default=default)

    assert padding.get_value(attr_name='x') == x_val
    assert padding.get_value('v1', 'x') == x_val
    assert padding.get_value(attr_name='y') == y_val
    assert padding.get_value('v1', 'y') == y_val
    assert padding.get_value(attr_name='x2') == default

    padding = AttrNamePadding({})
    assert padding.get_value(attr_name='x') == 0.0


def test_attr_name_padding_invalid():
    with pytest.raises(ValueError, match="to be a dictionary"):
        AttrNamePadding(10.0)

    with pytest.raises(ValueError, match="to be a string"):
        AttrNamePadding({10: 10.0})

    with pytest.raises(ValueError, match="to be of type"):
        AttrNamePadding({"x": {}})

    with pytest.raises(ValueError, match="to be of type"):
        AttrNamePadding({"x": {}})

    node_type_padding = NodeTypePadding({"x": 10.0})
    with pytest.raises(ValueError, match="to be of type"):
        AttrNamePadding({'x': node_type_padding})


@pytest.mark.parametrize('store_type', ['node', 'edge'])
def test_node_edge_type_padding(store_type):
    if store_type == "node":
        stores = ['v1', 'v2', 'v3', 'v4']
        padding_cls = NodeTypePadding
    else:
        stores = [('v1', 'e1', 'v1'), ('v1', 'e2', 'v1'), ('v1', 'e1', 'v2'),
                  ('v2', 'e1', 'v1')]
        padding_cls = EdgeTypePadding

    s0_default = 3.0
    s0_padding_dict = {'x': 10.0, 'y': -12.0}
    s0_padding = AttrNamePadding(s0_padding_dict, s0_default)
    s1_default = 0.1
    s1_padding_dict = {'y': 0.0, 'p': 13.0}
    s1_padding = AttrNamePadding(s1_padding_dict, s1_default)

    s2_default = 7.5
    store_default = -11.0
    padding_dict = {
        stores[0]: s0_padding,
        stores[1]: s1_padding,
        stores[2]: s2_default
    }
    padding = padding_cls(padding_dict, store_default)

    assert padding.get_value(stores[0], 'x') == s0_padding_dict['x']
    assert padding.get_value(stores[0], 'y') == s0_padding_dict['y']
    assert padding.get_value(stores[0], 'p') == s0_default
    assert padding.get_value(stores[0], 'z') == s0_default

    assert padding.get_value(stores[1], 'x') == s1_default
    assert padding.get_value(stores[1], 'y') == s1_padding_dict['y']
    assert padding.get_value(stores[1], 'p') == s1_padding_dict['p']
    assert padding.get_value(stores[1], 'z') == s1_default

    assert padding.get_value(stores[2], 'x') == s2_default
    assert padding.get_value(stores[2], 'z') == s2_default

    assert padding.get_value(stores[3], 'x') == store_default


def test_edge_padding_invalid():
    with pytest.raises(ValueError, match="to be a tuple"):
        EdgeTypePadding({'v1': 10.0})

    with pytest.raises(ValueError, match="got 1"):
        EdgeTypePadding({('v1', ): 10.0})

    with pytest.raises(ValueError, match="got 2"):
        EdgeTypePadding({('v1', 'v2'): 10.0})

    with pytest.raises(ValueError, match="got 4"):
        EdgeTypePadding({('v1', 'e2', 'v1', 'v2'): 10.0})
pytorch/test/transforms/test_point_pair_features.py000066400000000000000000000024211444262304300233770ustar00rootroot00000000000000from math import pi as PI

import torch

from torch_geometric.data import Data
from torch_geometric.transforms import PointPairFeatures


def test_point_pair_features():
    transform = PointPairFeatures()
    assert str(transform) == 'PointPairFeatures()'

    pos = torch.Tensor([[0, 0, 0], [1, 0, 0]])
    edge_index = torch.tensor([[0, 1], [1, 0]])
    norm = torch.Tensor([[1, 0, 0], [1, 0, 0]])
    edge_attr = torch.Tensor([1, 1])
    data = Data(edge_index=edge_index, pos=pos, norm=norm)

    data = transform(data)
    assert len(data) == 4
    assert data.pos.tolist() == pos.tolist()
    assert data.norm.tolist() == norm.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 0, 0, 0], [1, PI, PI, 0]]),
                          atol=1e-04)

    data = Data(edge_index=edge_index, pos=pos, norm=norm, edge_attr=edge_attr)
    data = transform(data)
    assert len(data) == 4
    assert data.pos.tolist() == pos.tolist()
    assert data.norm.tolist() == norm.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 1, 0, 0, 0], [1, 1, PI, PI, 0]]),
                          atol=1e-04)
pytorch/test/transforms/test_polar.py000066400000000000000000000017661444262304300204650ustar00rootroot00000000000000from math import pi as PI

import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Polar


def test_polar():
    assert str(Polar()) == 'Polar(norm=True, max_value=None)'

    pos = torch.Tensor([[0, 0], [1, 0]])
    edge_index = torch.tensor([[0, 1], [1, 0]])
    edge_attr = torch.Tensor([1, 1])

    data = Data(edge_index=edge_index, pos=pos)
    data = Polar(norm=False)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr, torch.Tensor([[1, 0], [1, PI]]),
                          atol=1e-04)

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = Polar(norm=True)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 1, 0], [1, 1, 0.5]]), atol=1e-04)
pytorch/test/transforms/test_radius_graph.py000066400000000000000000000012331444262304300220050ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import withPackage
from torch_geometric.transforms import RadiusGraph
from torch_geometric.utils import coalesce


@withPackage('torch_cluster')
def test_radius_graph():
    assert str(RadiusGraph(r=1)) == 'RadiusGraph(r=1)'

    pos = torch.Tensor([[0, 0], [1, 0], [2, 0], [0, 1], [-2, 0], [0, -2]])

    data = Data(pos=pos)
    data = RadiusGraph(r=1.5)(data)
    assert len(data) == 2
    assert data.pos.tolist() == pos.tolist()
    assert coalesce(data.edge_index).tolist() == [[0, 0, 1, 1, 1, 2, 3, 3],
                                                  [1, 3, 0, 2, 3, 1, 0, 1]]
pytorch/test/transforms/test_random_flip.py000066400000000000000000000010501444262304300216240ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RandomFlip


def test_random_flip():
    assert str(RandomFlip(axis=0)) == 'RandomFlip(axis=0, p=0.5)'

    pos = torch.Tensor([[-1, 1], [-3, 0], [2, -1]])

    data = Data(pos=pos)
    data = RandomFlip(axis=0, p=1)(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[1, 1], [3, 0], [-2, -1]]

    data = Data(pos=pos)
    data = RandomFlip(axis=1, p=1)(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[-1, -1], [-3, 0], [2, 1]]
pytorch/test/transforms/test_random_jitter.py000066400000000000000000000014631444262304300222030ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RandomJitter


def test_random_jitter():
    assert str(RandomJitter(0.1)) == 'RandomJitter(0.1)'

    pos = torch.Tensor([[0, 0], [0, 0], [0, 0], [0, 0]])

    data = Data(pos=pos)
    data = RandomJitter(0)(data)
    assert len(data) == 1
    assert data.pos.tolist() == pos.tolist()

    data = Data(pos=pos)
    data = RandomJitter(0.1)(data)
    assert len(data) == 1
    assert data.pos.min().item() >= -0.1
    assert data.pos.max().item() <= 0.1

    data = Data(pos=pos)
    data = RandomJitter([0.1, 1])(data)
    assert len(data) == 1
    assert data.pos[:, 0].min().item() >= -0.1
    assert data.pos[:, 0].max().item() <= 0.1
    assert data.pos[:, 1].min().item() >= -1
    assert data.pos[:, 1].max().item() <= 1
pytorch/test/transforms/test_random_link_split.py000066400000000000000000000317341444262304300230560ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.testing import get_random_edge_index, onlyFullTest
from torch_geometric.transforms import RandomLinkSplit
from torch_geometric.utils import is_undirected, to_undirected


def test_random_link_split():
    assert str(RandomLinkSplit()) == ('RandomLinkSplit('
                                      'num_val=0.1, num_test=0.2)')

    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5],
                               [1, 0, 2, 1, 3, 2, 4, 3, 5, 4]])
    edge_attr = torch.randn(edge_index.size(1), 3)

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=100)

    # No test split:
    transform = RandomLinkSplit(num_val=2, num_test=0, is_undirected=True)
    train_data, val_data, test_data = transform(data)

    assert len(train_data) == 5
    assert train_data.num_nodes == 100
    assert train_data.edge_index.size() == (2, 6)
    assert train_data.edge_attr.size() == (6, 3)
    assert train_data.edge_label_index.size(1) == 6
    assert train_data.edge_label.size(0) == 6

    assert len(val_data) == 5
    assert val_data.num_nodes == 100
    assert val_data.edge_index.size() == (2, 6)
    assert val_data.edge_attr.size() == (6, 3)
    assert val_data.edge_label_index.size(1) == 4
    assert val_data.edge_label.size(0) == 4

    assert len(test_data) == 5
    assert test_data.num_nodes == 100
    assert test_data.edge_index.size() == (2, 10)
    assert test_data.edge_attr.size() == (10, 3)
    assert test_data.edge_label_index.size() == (2, 0)
    assert test_data.edge_label.size() == (0, )

    # Percentage split:
    transform = RandomLinkSplit(num_val=0.2, num_test=0.2,
                                neg_sampling_ratio=2.0, is_undirected=False)
    train_data, val_data, test_data = transform(data)

    assert len(train_data) == 5
    assert train_data.num_nodes == 100
    assert train_data.edge_index.size() == (2, 6)
    assert train_data.edge_attr.size() == (6, 3)
    assert train_data.edge_label_index.size(1) == 18
    assert train_data.edge_label.size(0) == 18

    assert len(val_data) == 5
    assert val_data.num_nodes == 100
    assert val_data.edge_index.size() == (2, 6)
    assert val_data.edge_attr.size() == (6, 3)
    assert val_data.edge_label_index.size(1) == 6
    assert val_data.edge_label.size(0) == 6

    assert len(test_data) == 5
    assert test_data.num_nodes == 100
    assert test_data.edge_index.size() == (2, 8)
    assert test_data.edge_attr.size() == (8, 3)
    assert test_data.edge_label_index.size(1) == 6
    assert test_data.edge_label.size(0) == 6

    # Disjoint training split:
    transform = RandomLinkSplit(num_val=0.2, num_test=0.2, is_undirected=False,
                                disjoint_train_ratio=0.5)
    train_data, val_data, test_data = transform(data)

    assert len(train_data) == 5
    assert train_data.num_nodes == 100
    assert train_data.edge_index.size() == (2, 3)
    assert train_data.edge_attr.size() == (3, 3)
    assert train_data.edge_label_index.size(1) == 6
    assert train_data.edge_label.size(0) == 6


def test_random_link_split_with_label():
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5],
                               [1, 0, 2, 1, 3, 2, 4, 3, 5, 4]])
    edge_label = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

    data = Data(edge_index=edge_index, edge_label=edge_label, num_nodes=6)

    transform = RandomLinkSplit(num_val=0.2, num_test=0.2,
                                neg_sampling_ratio=0.0)
    train_data, _, _ = transform(data)
    assert len(train_data) == 4
    assert train_data.num_nodes == 6
    assert train_data.edge_index.size() == (2, 6)
    assert train_data.edge_label_index.size() == (2, 6)
    assert train_data.edge_label.size() == (6, )
    assert train_data.edge_label.min() == 0
    assert train_data.edge_label.max() == 1

    transform = RandomLinkSplit(num_val=0.2, num_test=0.2,
                                neg_sampling_ratio=1.0)
    train_data, _, _ = transform(data)
    assert len(train_data) == 4
    assert train_data.num_nodes == 6
    assert train_data.edge_index.size() == (2, 6)
    assert train_data.edge_label_index.size() == (2, 12)
    assert train_data.edge_label.size() == (12, )
    assert train_data.edge_label.min() == 0
    assert train_data.edge_label.max() == 2
    assert train_data.edge_label[6:].sum() == 0


def test_random_link_split_increment_label():
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5],
                               [1, 0, 2, 1, 3, 2, 4, 3, 5, 4]])
    edge_label = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

    data = Data(edge_index=edge_index, edge_label=edge_label, num_nodes=6)

    transform = RandomLinkSplit(num_val=0, num_test=0, neg_sampling_ratio=0.0)
    train_data, _, _ = transform(data)
    assert train_data.edge_label.numel() == edge_index.size(1)
    assert train_data.edge_label.min() == 0
    assert train_data.edge_label.max() == 1

    transform = RandomLinkSplit(num_val=0, num_test=0, neg_sampling_ratio=1.0)
    train_data, _, _ = transform(data)
    assert train_data.edge_label.numel() == 2 * edge_index.size(1)
    assert train_data.edge_label.min() == 0
    assert train_data.edge_label.max() == 2
    assert train_data.edge_label[edge_index.size(1):].sum() == 0


def test_random_link_split_on_hetero_data():
    data = HeteroData()

    data['p'].x = torch.arange(100)
    data['a'].x = torch.arange(100, 300)

    data['p', 'p'].edge_index = get_random_edge_index(100, 100, 500)
    data['p', 'p'].edge_index = to_undirected(data['p', 'p'].edge_index)
    data['p', 'p'].edge_attr = torch.arange(data['p', 'p'].num_edges)
    data['p', 'a'].edge_index = get_random_edge_index(100, 200, 1000)
    data['p', 'a'].edge_attr = torch.arange(500, 1500)
    data['a', 'p'].edge_index = data['p', 'a'].edge_index.flip([0])
    data['a', 'p'].edge_attr = torch.arange(1500, 2500)

    transform = RandomLinkSplit(num_val=0.2, num_test=0.2, is_undirected=True,
                                edge_types=('p', 'p'))
    train_data, val_data, test_data = transform(data)

    assert len(train_data['p']) == 1
    assert len(train_data['a']) == 1
    assert len(train_data['p', 'p']) == 4
    assert len(train_data['p', 'a']) == 2
    assert len(train_data['a', 'p']) == 2

    assert is_undirected(train_data['p', 'p'].edge_index,
                         train_data['p', 'p'].edge_attr)
    assert is_undirected(val_data['p', 'p'].edge_index,
                         val_data['p', 'p'].edge_attr)
    assert is_undirected(test_data['p', 'p'].edge_index,
                         test_data['p', 'p'].edge_attr)

    transform = RandomLinkSplit(num_val=0.2, num_test=0.2,
                                edge_types=('p', 'a'),
                                rev_edge_types=('a', 'p'))
    train_data, val_data, test_data = transform(data)

    assert len(train_data['p']) == 1
    assert len(train_data['a']) == 1
    assert len(train_data['p', 'p']) == 2
    assert len(train_data['p', 'a']) == 4
    assert len(train_data['a', 'p']) == 2

    assert train_data['p', 'a'].edge_index.size() == (2, 600)
    assert train_data['p', 'a'].edge_attr.size() == (600, )
    assert train_data['p', 'a'].edge_attr.min() >= 500
    assert train_data['p', 'a'].edge_attr.max() <= 1500
    assert train_data['a', 'p'].edge_index.size() == (2, 600)
    assert train_data['a', 'p'].edge_attr.size() == (600, )
    assert train_data['a', 'p'].edge_attr.min() >= 500
    assert train_data['a', 'p'].edge_attr.max() <= 1500
    assert train_data['p', 'a'].edge_label_index.size() == (2, 1200)
    assert train_data['p', 'a'].edge_label.size() == (1200, )

    assert val_data['p', 'a'].edge_index.size() == (2, 600)
    assert val_data['p', 'a'].edge_attr.size() == (600, )
    assert val_data['p', 'a'].edge_attr.min() >= 500
    assert val_data['p', 'a'].edge_attr.max() <= 1500
    assert val_data['a', 'p'].edge_index.size() == (2, 600)
    assert val_data['a', 'p'].edge_attr.size() == (600, )
    assert val_data['a', 'p'].edge_attr.min() >= 500
    assert val_data['a', 'p'].edge_attr.max() <= 1500
    assert val_data['p', 'a'].edge_label_index.size() == (2, 400)
    assert val_data['p', 'a'].edge_label.size() == (400, )

    assert test_data['p', 'a'].edge_index.size() == (2, 800)
    assert test_data['p', 'a'].edge_attr.size() == (800, )
    assert test_data['p', 'a'].edge_attr.min() >= 500
    assert test_data['p', 'a'].edge_attr.max() <= 1500
    assert test_data['a', 'p'].edge_index.size() == (2, 800)
    assert test_data['a', 'p'].edge_attr.size() == (800, )
    assert test_data['a', 'p'].edge_attr.min() >= 500
    assert test_data['a', 'p'].edge_attr.max() <= 1500
    assert test_data['p', 'a'].edge_label_index.size() == (2, 400)
    assert test_data['p', 'a'].edge_label.size() == (400, )

    transform = RandomLinkSplit(num_val=0.2, num_test=0.2, is_undirected=True,
                                edge_types=[('p', 'p'), ('p', 'a')],
                                rev_edge_types=[None, ('a', 'p')])
    train_data, val_data, test_data = transform(data)

    assert len(train_data['p']) == 1
    assert len(train_data['a']) == 1
    assert len(train_data['p', 'p']) == 4
    assert len(train_data['p', 'a']) == 4
    assert len(train_data['a', 'p']) == 2

    assert is_undirected(train_data['p', 'p'].edge_index,
                         train_data['p', 'p'].edge_attr)
    assert train_data['p', 'a'].edge_index.size() == (2, 600)
    assert train_data['a', 'p'].edge_index.size() == (2, 600)

    # No reverse edge types specified:
    transform = RandomLinkSplit(edge_types=[('p', 'p'), ('p', 'a')])
    train_data, val_data, test_data = transform(data)
    assert train_data['p', 'p'].num_edges < data['p', 'p'].num_edges
    assert train_data['p', 'a'].num_edges < data['p', 'a'].num_edges
    assert train_data['a', 'p'].num_edges == data['a', 'p'].num_edges


def test_random_link_split_on_undirected_hetero_data():
    data = HeteroData()
    data['p'].x = torch.arange(100)
    data['p', 'p'].edge_index = get_random_edge_index(100, 100, 500)
    data['p', 'p'].edge_index = to_undirected(data['p', 'p'].edge_index)

    transform = RandomLinkSplit(is_undirected=True, edge_types=('p', 'p'))
    train_data, val_data, test_data = transform(data)
    assert train_data['p', 'p'].is_undirected()

    transform = RandomLinkSplit(is_undirected=True, edge_types=('p', 'p'),
                                rev_edge_types=('p', 'p'))
    train_data, val_data, test_data = transform(data)
    assert train_data['p', 'p'].is_undirected()

    transform = RandomLinkSplit(is_undirected=True, edge_types=('p', 'p'),
                                rev_edge_types=('p', 'p'))
    train_data, val_data, test_data = transform(data)
    assert train_data['p', 'p'].is_undirected()


def test_random_link_split_insufficient_negative_edges():
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2], [1, 3, 0, 2, 0, 1]])
    data = Data(edge_index=edge_index, num_nodes=4)

    transform = RandomLinkSplit(num_val=0.34, num_test=0.34,
                                is_undirected=False, neg_sampling_ratio=2,
                                split_labels=True)

    with pytest.warns(UserWarning, match="not enough negative edges"):
        train_data, val_data, test_data = transform(data)

    assert train_data.neg_edge_label_index.size() == (2, 2)
    assert val_data.neg_edge_label_index.size() == (2, 2)
    assert test_data.neg_edge_label_index.size() == (2, 2)


def test_random_link_split_non_contiguous():
    edge_index = get_random_edge_index(40, 40, num_edges=150)
    edge_index = edge_index[:, :100]
    assert not edge_index.is_contiguous()

    data = Data(edge_index=edge_index, num_nodes=40)
    transform = RandomLinkSplit(num_val=0.2, num_test=0.2)
    train_data, val_data, test_data = transform(data)
    assert train_data.num_edges == 60
    assert train_data.edge_index.is_contiguous()

    data = HeteroData()
    data['p'].num_nodes = 40
    data['p', 'p'].edge_index = edge_index
    transform = RandomLinkSplit(num_val=0.2, num_test=0.2,
                                edge_types=('p', 'p'))
    train_data, val_data, test_data = transform(data)
    assert train_data['p', 'p'].num_edges == 60
    assert train_data['p', 'p'].edge_index.is_contiguous()


@onlyFullTest
def test_random_link_split_on_dataset(get_dataset):
    dataset = get_dataset(name='MUTAG')

    dataset.transform = RandomLinkSplit(
        num_val=0.1,
        num_test=0.1,
        disjoint_train_ratio=0.3,
        add_negative_train_samples=False,
    )

    train_dataset, val_dataset, test_dataset = zip(*dataset)
    assert len(train_dataset) == len(dataset)
    assert len(val_dataset) == len(dataset)
    assert len(test_dataset) == len(dataset)

    assert isinstance(train_dataset[0], Data)
    assert train_dataset[0].edge_label.min() == 1.0
    assert train_dataset[0].edge_label.max() == 1.0

    assert isinstance(val_dataset[0], Data)
    assert val_dataset[0].edge_label.min() == 0.0
    assert val_dataset[0].edge_label.max() == 1.0

    assert isinstance(test_dataset[0], Data)
    assert test_dataset[0].edge_label.min() == 0.0
    assert test_dataset[0].edge_label.max() == 1.0
pytorch/test/transforms/test_random_node_split.py000066400000000000000000000152501444262304300230410ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import RandomNodeSplit


@pytest.mark.parametrize('num_splits', [1, 2])
def test_random_node_split(num_splits):
    num_nodes, num_classes = 1000, 4
    x = torch.randn(num_nodes, 16)
    y = torch.randint(num_classes, (num_nodes, ), dtype=torch.long)
    data = Data(x=x, y=y)

    transform = RandomNodeSplit(split='train_rest', num_splits=num_splits,
                                num_val=100, num_test=200)
    assert str(transform) == 'RandomNodeSplit(split=train_rest)'
    data = transform(data)
    assert len(data) == 5

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    assert train_mask.size() == (num_nodes, num_splits)
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    assert val_mask.size() == (num_nodes, num_splits)
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask
    assert test_mask.size() == (num_nodes, num_splits)

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == num_nodes - 100 - 200
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == 200
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == num_nodes)

    transform = RandomNodeSplit(split='train_rest', num_splits=num_splits,
                                num_val=0.1, num_test=0.2)
    data = transform(data)

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == num_nodes - 100 - 200
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == 200
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == num_nodes)

    transform = RandomNodeSplit(split='test_rest', num_splits=num_splits,
                                num_train_per_class=10, num_val=100)
    assert str(transform) == 'RandomNodeSplit(split=test_rest)'
    data = transform(data)
    assert len(data) == 5

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == 10 * num_classes
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == num_nodes - 10 * num_classes - 100
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == num_nodes)

    transform = RandomNodeSplit(split='test_rest', num_splits=num_splits,
                                num_train_per_class=10, num_val=0.1)
    data = transform(data)

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == 10 * num_classes
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == num_nodes - 10 * num_classes - 100
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == num_nodes)

    transform = RandomNodeSplit(split='random', num_splits=num_splits,
                                num_train_per_class=10, num_val=100,
                                num_test=200)
    assert str(transform) == 'RandomNodeSplit(split=random)'
    data = transform(data)
    assert len(data) == 5

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == 10 * num_classes
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == 200
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == 10 * num_classes + 100 + 200)

    transform = RandomNodeSplit(split='random', num_splits=num_splits,
                                num_train_per_class=10, num_val=0.1,
                                num_test=0.2)
    assert str(transform) == 'RandomNodeSplit(split=random)'
    data = transform(data)

    train_mask = data.train_mask
    train_mask = train_mask.unsqueeze(-1) if num_splits == 1 else train_mask
    val_mask = data.val_mask
    val_mask = val_mask.unsqueeze(-1) if num_splits == 1 else val_mask
    test_mask = data.test_mask
    test_mask = test_mask.unsqueeze(-1) if num_splits == 1 else test_mask

    for i in range(train_mask.size(-1)):
        assert train_mask[:, i].sum() == 10 * num_classes
        assert val_mask[:, i].sum() == 100
        assert test_mask[:, i].sum() == 200
        assert (train_mask[:, i] & val_mask[:, i] & test_mask[:, i]).sum() == 0
        assert ((train_mask[:, i] | val_mask[:, i]
                 | test_mask[:, i]).sum() == 10 * num_classes + 100 + 200)


def test_random_node_split_on_hetero_data():
    data = HeteroData()

    data['paper'].x = torch.randn(2000, 16)
    data['paper'].y = torch.randint(4, (2000, ), dtype=torch.long)
    data['author'].x = torch.randn(300, 16)

    transform = RandomNodeSplit()
    assert str(transform) == 'RandomNodeSplit(split=train_rest)'
    data = transform(data)
    assert len(data) == 5

    assert len(data['author']) == 1
    assert len(data['paper']) == 5

    assert data['paper'].train_mask.sum() == 500
    assert data['paper'].val_mask.sum() == 500
    assert data['paper'].test_mask.sum() == 1000
pytorch/test/transforms/test_random_rotate.py000066400000000000000000000025051444262304300221760ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RandomRotate


def test_random_rotate():
    assert str(RandomRotate([-180, 180])) == ('RandomRotate('
                                              '[-180, 180], axis=0)')

    pos = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])

    data = Data(pos=pos)
    data = RandomRotate(0)(data)
    assert len(data) == 1
    assert data.pos.tolist() == pos.tolist()

    data = Data(pos=pos)
    data = RandomRotate([180, 180])(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[1, 1], [1, -1], [-1, 1], [-1, -1]]

    pos = torch.Tensor([[-1, -1, 1], [-1, 1, 1], [1, -1, -1], [1, 1, -1]])

    data = Data(pos=pos)
    data = RandomRotate([180, 180], axis=0)(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[-1, 1, -1], [-1, -1, -1], [1, 1, 1],
                                 [1, -1, 1]]

    data = Data(pos=pos)
    data = RandomRotate([180, 180], axis=1)(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[1, -1, -1], [1, 1, -1], [-1, -1, 1],
                                 [-1, 1, 1]]

    data = Data(pos=pos)
    data = RandomRotate([180, 180], axis=2)(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[1, 1, 1], [1, -1, 1], [-1, 1, -1],
                                 [-1, -1, -1]]
pytorch/test/transforms/test_random_scale.py000066400000000000000000000010411444262304300217610ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RandomScale


def test_random_scale():
    assert str(RandomScale([1, 2])) == 'RandomScale([1, 2])'

    pos = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])

    data = Data(pos=pos)
    data = RandomScale([1, 1])(data)
    assert len(data) == 1
    assert data.pos.tolist() == pos.tolist()

    data = Data(pos=pos)
    data = RandomScale([2, 2])(data)
    assert len(data) == 1
    assert data.pos.tolist() == [[-2, -2], [-2, 2], [2, -2], [2, 2]]
pytorch/test/transforms/test_random_shear.py000066400000000000000000000007731444262304300220070ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RandomShear


def test_random_shear():
    assert str(RandomShear(0.1)) == 'RandomShear(0.1)'

    pos = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])

    data = Data(pos=pos)
    data = RandomShear(0)(data)
    assert len(data) == 1
    assert data.pos.tolist() == pos.tolist()

    data = Data(pos=pos)
    data = RandomShear(0.1)(data)
    assert len(data) == 1
    assert data.pos.tolist() != pos.tolist()
pytorch/test/transforms/test_remove_duplicated_edges.py000066400000000000000000000012731444262304300242030ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RemoveDuplicatedEdges


def test_remove_duplicated_edges():
    edge_index = torch.tensor([[0, 0, 0, 0, 1, 1, 1, 1],
                               [0, 0, 1, 1, 0, 0, 1, 1]])
    edge_weight = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8])
    data = Data(edge_index=edge_index, edge_weight=edge_weight, num_nodes=2)

    transform = RemoveDuplicatedEdges()
    assert str(transform) == 'RemoveDuplicatedEdges()'

    out = transform(data)
    assert len(out) == 3
    assert out.num_nodes == 2
    assert out.edge_index.tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert out.edge_weight.tolist() == [3, 7, 11, 15]
pytorch/test/transforms/test_remove_isolated_nodes.py000066400000000000000000000031431444262304300237100ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import RemoveIsolatedNodes


def test_remove_isolated_nodes():
    assert str(RemoveIsolatedNodes()) == 'RemoveIsolatedNodes()'

    data = Data()
    data.x = torch.arange(3)
    data.edge_index = torch.tensor([[0, 2], [2, 0]])
    data.edge_attr = torch.arange(2)

    data = RemoveIsolatedNodes()(data)

    assert len(data) == 3
    assert data.x.tolist() == [0, 2]
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]
    assert data.edge_attr.tolist() == [0, 1]


def test_remove_isolated_nodes_in_hetero_data():
    data = HeteroData()

    data['p'].x = torch.arange(6)
    data['a'].x = torch.arange(6)
    data['i'].num_nodes = 4

    # isolated paper nodes: {4}
    # isolated author nodes: {3, 4, 5}
    # isolated institution nodes: {0, 1, 2, 3}
    data['p', '1', 'p'].edge_index = torch.tensor([[0, 1, 2], [0, 1, 3]])
    data['p', '2', 'a'].edge_index = torch.tensor([[1, 3, 5], [0, 1, 2]])
    data['p', '2', 'a'].edge_attr = torch.arange(3)
    data['p', '3', 'a'].edge_index = torch.tensor([[5], [2]])

    data = RemoveIsolatedNodes()(data)

    assert len(data) == 4
    assert data['p'].num_nodes == 5
    assert data['a'].num_nodes == 3
    assert data['i'].num_nodes == 0

    assert data['p'].x.tolist() == [0, 1, 2, 3, 5]
    assert data['a'].x.tolist() == [0, 1, 2]

    assert data['1'].edge_index.tolist() == [[0, 1, 2], [0, 1, 3]]
    assert data['2'].edge_index.tolist() == [[1, 3, 4], [0, 1, 2]]
    assert data['2'].edge_attr.tolist() == [0, 1, 2]
    assert data['3'].edge_index.tolist() == [[4], [2]]
pytorch/test/transforms/test_remove_training_classes.py000066400000000000000000000011171444262304300242430ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import RemoveTrainingClasses


def test_remove_training_classes():
    y = torch.tensor([1, 0, 0, 2, 1, 3])
    train_mask = torch.tensor([False, False, True, True, True, True])

    data = Data(y=y, train_mask=train_mask)

    transform = RemoveTrainingClasses(classes=[0, 1])
    assert str(transform) == 'RemoveTrainingClasses([0, 1])'

    data = transform(data)
    assert len(data) == 2
    assert torch.equal(data.y, y)
    assert data.train_mask.tolist() == [False, False, False, True, False, True]
pytorch/test/transforms/test_rooted_subgraph.py000066400000000000000000000056501444262304300225330ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.loader import DataLoader
from torch_geometric.testing import withPackage
from torch_geometric.transforms import RootedEgoNets, RootedRWSubgraph


def test_rooted_ego_nets():
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.randn(4, 8)
    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr)

    transform = RootedEgoNets(num_hops=1)
    assert str(transform) == 'RootedEgoNets(num_hops=1)'

    out = transform(data)
    assert len(out) == 8

    assert torch.equal(out.x, data.x)
    assert torch.equal(out.edge_index, data.edge_index)
    assert torch.equal(out.edge_attr, data.edge_attr)

    assert out.sub_edge_index.tolist() == [[0, 1, 2, 3, 3, 4, 5, 6],
                                           [1, 0, 3, 2, 4, 3, 6, 5]]
    assert out.n_id.tolist() == [0, 1, 0, 1, 2, 1, 2]
    assert out.n_sub_batch.tolist() == [0, 0, 1, 1, 1, 2, 2]
    assert out.e_id.tolist() == [0, 1, 0, 1, 2, 3, 2, 3]
    assert out.e_sub_batch.tolist() == [0, 0, 1, 1, 1, 1, 2, 2]

    out = out.map_data()
    assert len(out) == 4

    assert torch.allclose(out.x, x[[0, 1, 0, 1, 2, 1, 2]])
    assert out.edge_index.tolist() == [[0, 1, 2, 3, 3, 4, 5, 6],
                                       [1, 0, 3, 2, 4, 3, 6, 5]]
    assert torch.allclose(out.edge_attr, edge_attr[[0, 1, 0, 1, 2, 3, 2, 3]])
    assert out.n_sub_batch.tolist() == [0, 0, 1, 1, 1, 2, 2]


@withPackage('torch_cluster')
def test_rooted_rw_subgraph():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    data = Data(edge_index=edge_index, num_nodes=3)

    transform = RootedRWSubgraph(walk_length=1)
    assert str(transform) == 'RootedRWSubgraph(walk_length=1)'

    out = transform(data)
    assert len(out) == 7

    assert out.n_sub_batch.tolist() == [0, 0, 1, 1, 2, 2]
    assert out.sub_edge_index.size() == (2, 6)

    out = out.map_data()
    assert len(out) == 3

    assert out.edge_index.size() == (2, 6)
    assert out.num_nodes == 6
    assert out.n_sub_batch.tolist() == [0, 0, 1, 1, 2, 2]


def test_rooted_subgraph_minibatch():
    x = torch.randn(3, 8)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.randn(4, 8)
    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr)

    transform = RootedEgoNets(num_hops=1)
    data = transform(data)

    loader = DataLoader([data, data], batch_size=2)
    batch = next(iter(loader))
    batch = batch.map_data()
    assert batch.num_graphs == len(batch) == 2

    assert batch.x.size() == (14, 8)
    assert batch.edge_index.size() == (2, 16)
    assert batch.edge_attr.size() == (16, 8)
    assert batch.n_sub_batch.size() == (14, )
    assert batch.batch.size() == (14, )
    assert batch.ptr.size() == (3, )

    assert batch.edge_index.min() == 0
    assert batch.edge_index.max() == 13

    assert batch.n_sub_batch.min() == 0
    assert batch.n_sub_batch.max() == 5
pytorch/test/transforms/test_sample_points.py000066400000000000000000000015071444262304300222160ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import SamplePoints


def test_sample_points():
    assert str(SamplePoints(1024)) == 'SamplePoints(1024)'

    pos = torch.Tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 0]])
    face = torch.tensor([[0, 1], [1, 2], [2, 3]])

    data = Data(pos=pos)
    data.face = face
    data = SamplePoints(8)(data)
    assert len(data) == 1
    assert pos[:, 0].min().item() >= 0 and pos[:, 0].max().item() <= 1
    assert pos[:, 1].min().item() >= 0 and pos[:, 1].max().item() <= 1
    assert pos[:, 2].abs().sum().item() == 0

    data = Data(pos=pos)
    data.face = face
    data = SamplePoints(8, include_normals=True)(data)
    assert len(data) == 2
    assert data.normal[:, :2].abs().sum().item() == 0
    assert data.normal[:, 2].abs().sum().item() == 8
pytorch/test/transforms/test_sign.py000066400000000000000000000014471444262304300203040ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import SIGN


def test_sign():
    x = torch.ones(5, 3)
    edge_index = torch.tensor([
        [0, 1, 2, 3, 3, 4],
        [1, 0, 3, 2, 4, 3],
    ])
    data = Data(x=x, edge_index=edge_index)

    transform = SIGN(K=2)
    assert str(transform) == 'SIGN(K=2)'

    expected_x1 = torch.tensor([
        [1, 1, 1],
        [1, 1, 1],
        [0.7071, 0.7071, 0.7071],
        [1.4142, 1.4142, 1.4142],
        [0.7071, 0.7071, 0.7071],
    ])
    expected_x2 = torch.ones(5, 3)

    out = transform(data)
    assert len(out) == 4
    assert torch.equal(out.edge_index, edge_index)
    assert torch.allclose(out.x, x)
    assert torch.allclose(out.x1, expected_x1, atol=1e-4)
    assert torch.allclose(out.x2, expected_x2)
pytorch/test/transforms/test_spherical.py000066400000000000000000000036271444262304300213200ustar00rootroot00000000000000from math import pi as PI

import torch

from torch_geometric.data import Data
from torch_geometric.transforms import Spherical


def test_spherical():
    assert str(Spherical()) == 'Spherical(norm=True, max_value=None)'

    pos = torch.Tensor([[0, 0, 0], [1, 0, 0]])
    edge_index = torch.tensor([[0, 1], [1, 0]])
    edge_attr = torch.Tensor([1, 1])

    data = Data(edge_index=edge_index, pos=pos)
    data = Spherical(norm=False)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 0, PI / 2], [1, PI, PI / 2]]),
                          atol=1e-04)

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = Spherical(norm=True)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 1, 0, 0.5], [1, 1, 0.5, 0.5]]),
                          atol=1e-04)

    pos = torch.Tensor([[0, 0, 0], [0, 0, 1]])
    edge_index = torch.tensor([[0, 1], [1, 0]])

    data = Data(edge_index=edge_index, pos=pos)
    data = Spherical(norm=False)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 0, 0], [1, 0, PI]]), atol=1e-04)

    data = Data(edge_index=edge_index, pos=pos, edge_attr=edge_attr)
    data = Spherical(norm=True)(data)
    assert len(data) == 3
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index.tolist()
    assert torch.allclose(data.edge_attr,
                          torch.Tensor([[1, 1, 0, 0], [1, 1, 0, 1]]),
                          atol=1e-04)
pytorch/test/transforms/test_svd_feature_reduction.py000066400000000000000000000010341444262304300237170ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import SVDFeatureReduction


def test_svd_feature_reduction():
    assert str(SVDFeatureReduction(10)) == 'SVDFeatureReduction(10)'

    x = torch.randn(4, 16)
    U, S, _ = torch.linalg.svd(x)
    data = Data(x=x)
    data = SVDFeatureReduction(10)(data)
    assert torch.allclose(data.x, torch.mm(U[:, :10], torch.diag(S[:10])))

    x = torch.randn(4, 8)
    data.x = x
    data = SVDFeatureReduction(10)(Data(x=x))
    assert torch.allclose(data.x, x)
pytorch/test/transforms/test_target_indegree.py000066400000000000000000000016041444262304300224670ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import TargetIndegree


def test_target_indegree():
    assert str(TargetIndegree()) == 'TargetIndegree(norm=True, max_value=None)'

    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_attr = torch.Tensor([1, 1, 1, 1])

    data = Data(edge_index=edge_index, num_nodes=3)
    data = TargetIndegree(norm=False)(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[2], [1], [1], [2]]
    assert data.num_nodes == 3

    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=3)
    data = TargetIndegree(norm=True)(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == edge_index.tolist()
    assert data.edge_attr.tolist() == [[1, 1], [1, 0.5], [1, 0.5], [1, 1]]
    assert data.num_nodes == 3
pytorch/test/transforms/test_to_dense.py000066400000000000000000000034341444262304300211420ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import ToDense


def test_to_dense():
    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6])
    num_nodes = edge_index.max().item() + 1
    x = torch.randn((num_nodes, 4))
    pos = torch.randn((num_nodes, 3))
    y = torch.randint(0, 4, (num_nodes, ), dtype=torch.long)

    transform = ToDense()
    assert str(transform) == 'ToDense()'
    data = Data(x=x, pos=pos, edge_index=edge_index, edge_attr=edge_attr, y=y)
    data = transform(data)
    assert len(data) == 5
    assert data.x.tolist() == x.tolist()
    assert data.pos.tolist() == pos.tolist()
    assert data.y.tolist() == y.tolist()
    assert data.adj.size() == (num_nodes, num_nodes)
    assert data.adj.tolist() == [
        [0, 1, 2, 3],
        [4, 0, 0, 0],
        [5, 0, 0, 0],
        [6, 0, 0, 0],
    ]
    assert data.mask.tolist() == [1, 1, 1, 1]

    transform = ToDense(num_nodes=5)
    assert str(transform) == 'ToDense(num_nodes=5)'
    data = Data(x=x, pos=pos, edge_index=edge_index, edge_attr=edge_attr, y=y)
    data = transform(data)
    assert len(data) == 5
    assert data.x.size() == (5, 4)
    assert data.x[:4].tolist() == x.tolist()
    assert data.x[4].tolist() == [0, 0, 0, 0]
    assert data.pos.size() == (5, 3)
    assert data.pos[:4].tolist() == pos.tolist()
    assert data.pos[4].tolist() == [0, 0, 0]
    assert data.y.size() == (5, )
    assert data.y[:4].tolist() == y.tolist()
    assert data.y[4].tolist() == 0
    assert data.adj.size() == (5, 5)
    assert data.adj.tolist() == [
        [0, 1, 2, 3, 0],
        [4, 0, 0, 0, 0],
        [5, 0, 0, 0, 0],
        [6, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
    ]
    assert data.mask.tolist() == [1, 1, 1, 1, 0]
pytorch/test/transforms/test_to_device.py000066400000000000000000000010721444262304300212770ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import withCUDA
from torch_geometric.transforms import ToDevice


@withCUDA
def test_to_device(device):
    x = torch.randn(3, 4)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.randn(edge_index.size(1))

    data = Data(x=x, edge_index=edge_index, edge_weight=edge_weight)

    transform = ToDevice(device)
    assert str(transform) == f'ToDevice({device})'

    data = transform(data)
    for key, value in data:
        assert value.device == device
pytorch/test/transforms/test_to_sparse_tensor.py000066400000000000000000000100761444262304300227330ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import ToSparseTensor


@pytest.mark.parametrize('layout', [None, torch.sparse_coo, torch.sparse_csr])
def test_to_sparse_tensor_basic(layout):
    transform = ToSparseTensor(layout=layout)
    assert str(transform) == (f'ToSparseTensor(attr=edge_weight, '
                              f'layout={layout})')

    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    perm = torch.tensor([1, 0, 3, 2])

    data = Data(edge_index=edge_index, edge_weight=edge_weight,
                edge_attr=edge_attr, num_nodes=3)
    data = transform(data)

    assert len(data) == 3
    assert data.num_nodes == 3
    assert torch.equal(data.edge_attr, edge_attr[perm])
    assert 'adj_t' in data

    if layout is None and torch_geometric.typing.WITH_TORCH_SPARSE:
        row, col, value = data.adj_t.coo()
        assert row.tolist() == [0, 1, 1, 2]
        assert col.tolist() == [1, 0, 2, 1]
        assert torch.equal(value, edge_weight[perm])
    else:
        adj_t = data.adj_t
        assert adj_t.layout == layout or torch.sparse_csr
        if layout != torch.sparse_coo:
            adj_t = adj_t.to_sparse_coo()
        assert adj_t.indices().tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
        assert torch.equal(adj_t.values(), edge_weight[perm])


def test_to_sparse_tensor_and_keep_edge_index():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    perm = torch.tensor([1, 0, 3, 2])

    data = Data(edge_index=edge_index, edge_weight=edge_weight,
                edge_attr=edge_attr, num_nodes=3)
    data = ToSparseTensor(remove_edge_index=False)(data)

    assert len(data) == 5
    assert torch.equal(data.edge_index, edge_index[:, perm])
    assert torch.equal(data.edge_weight, edge_weight[perm])
    assert torch.equal(data.edge_attr, edge_attr[perm])


@pytest.mark.parametrize('layout', [None, torch.sparse_coo, torch.sparse_csr])
def test_hetero_to_sparse_tensor(layout):
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])

    data = HeteroData()
    data['v'].num_nodes = 3
    data['w'].num_nodes = 3
    data['v', 'v'].edge_index = edge_index
    data['v', 'w'].edge_index = edge_index

    data = ToSparseTensor(layout=layout)(data)

    if layout is None and torch_geometric.typing.WITH_TORCH_SPARSE:
        row, col, value = data['v', 'v'].adj_t.coo()
        assert row.tolist() == [0, 1, 1, 2]
        assert col.tolist() == [1, 0, 2, 1]
        assert value is None

        row, col, value = data['v', 'w'].adj_t.coo()
        assert row.tolist() == [0, 1, 1, 2]
        assert col.tolist() == [1, 0, 2, 1]
        assert value is None
    else:
        adj_t = data['v', 'v'].adj_t
        assert adj_t.layout == layout or torch.sparse_csr
        if layout != torch.sparse_coo:
            adj_t = adj_t.to_sparse_coo()
        assert adj_t.indices().tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
        assert adj_t.values().tolist() == [1., 1., 1., 1.]

        adj_t = data['v', 'w'].adj_t
        assert adj_t.layout == layout or torch.sparse_csr
        if layout != torch.sparse_coo:
            adj_t = adj_t.to_sparse_coo()
        assert adj_t.indices().tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
        assert adj_t.values().tolist() == [1., 1., 1., 1.]


def test_to_sparse_tensor_num_nodes_equals_num_edges():
    x = torch.arange(4)
    y = torch.arange(4)
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    perm = torch.tensor([1, 0, 3, 2])

    data = Data(x=x, edge_index=edge_index, edge_weight=edge_weight,
                edge_attr=edge_attr, y=y)
    data = ToSparseTensor()(data)

    assert len(data) == 4
    assert torch.equal(data.x, x)
    assert torch.equal(data.y, y)
    assert torch.equal(data.edge_attr, edge_attr[perm])
pytorch/test/transforms/test_to_superpixels.py000066400000000000000000000060701444262304300224260ustar00rootroot00000000000000import os.path as osp

import torch

from torch_geometric.data import download_url, extract_gz
from torch_geometric.data.makedirs import makedirs
from torch_geometric.loader import DataLoader
from torch_geometric.testing import withPackage
from torch_geometric.transforms import ToSLIC

resources = [
    'https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz',
    'https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz',
]


@withPackage('torchvision', 'skimage')
def test_to_superpixels(tmp_path):
    import torchvision.transforms as T
    from torchvision.datasets.mnist import (
        MNIST,
        read_image_file,
        read_label_file,
    )

    raw_folder = osp.join(tmp_path, 'MNIST', 'raw')
    processed_folder = osp.join(tmp_path, 'MNIST', 'processed')

    makedirs(raw_folder)
    makedirs(processed_folder)
    for resource in resources:
        path = download_url(resource, raw_folder)
        extract_gz(path, osp.join(tmp_path, raw_folder))

    test_set = (
        read_image_file(osp.join(raw_folder, 't10k-images-idx3-ubyte')),
        read_label_file(osp.join(raw_folder, 't10k-labels-idx1-ubyte')),
    )

    torch.save(test_set, osp.join(processed_folder, 'training.pt'))
    torch.save(test_set, osp.join(processed_folder, 'test.pt'))

    dataset = MNIST(tmp_path, download=False)

    dataset.transform = T.Compose([T.ToTensor(), ToSLIC()])

    data, y = dataset[0]
    assert len(data) == 2
    assert data.pos.dim() == 2 and data.pos.size(1) == 2
    assert data.x.dim() == 2 and data.x.size(1) == 1
    assert data.pos.size(0) == data.x.size(0)
    assert y == 7

    loader = DataLoader(dataset, batch_size=2, shuffle=False)
    for batch, y in loader:
        assert batch.num_graphs == len(batch) == 2
        assert batch.pos.dim() == 2 and batch.pos.size(1) == 2
        assert batch.x.dim() == 2 and batch.x.size(1) == 1
        assert batch.batch.dim() == 1
        assert batch.ptr.dim() == 1
        assert batch.pos.size(0) == batch.x.size(0) == batch.batch.size(0)
        assert y.tolist() == [7, 2]
        break

    dataset.transform = T.Compose(
        [T.ToTensor(), ToSLIC(add_seg=True, add_img=True)])

    data, y = dataset[0]
    assert len(data) == 4
    assert data.pos.dim() == 2 and data.pos.size(1) == 2
    assert data.x.dim() == 2 and data.x.size(1) == 1
    assert data.pos.size(0) == data.x.size(0)
    assert data.seg.size() == (1, 28, 28)
    assert data.img.size() == (1, 1, 28, 28)
    assert data.seg.max().item() + 1 == data.x.size(0)
    assert y == 7

    loader = DataLoader(dataset, batch_size=2, shuffle=False)
    for batch, y in loader:
        assert batch.num_graphs == len(batch) == 2
        assert batch.pos.dim() == 2 and batch.pos.size(1) == 2
        assert batch.x.dim() == 2 and batch.x.size(1) == 1
        assert batch.batch.dim() == 1
        assert batch.ptr.dim() == 1
        assert batch.pos.size(0) == batch.x.size(0) == batch.batch.size(0)
        assert batch.seg.size() == (2, 28, 28)
        assert batch.img.size() == (2, 1, 28, 28)
        assert y.tolist() == [7, 2]
        break
pytorch/test/transforms/test_to_undirected.py000066400000000000000000000050531444262304300221710ustar00rootroot00000000000000import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import ToUndirected


def test_to_undirected():
    assert str(ToUndirected()) == 'ToUndirected()'

    edge_index = torch.tensor([[2, 0, 2], [3, 1, 0]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    perm = torch.tensor([1, 2, 1, 2, 0, 0])

    data = Data(edge_index=edge_index, edge_weight=edge_weight,
                edge_attr=edge_attr, num_nodes=4)
    data = ToUndirected()(data)
    assert len(data) == 4
    assert data.edge_index.tolist() == [[0, 0, 1, 2, 2, 3], [1, 2, 0, 0, 3, 2]]
    assert data.edge_weight.tolist() == edge_weight[perm].tolist()
    assert data.edge_attr.tolist() == edge_attr[perm].tolist()
    assert data.num_nodes == 4


def test_to_undirected_with_duplicates():
    edge_index = torch.tensor([[0, 0, 1, 1], [0, 1, 0, 2]])
    edge_weight = torch.ones(4)

    data = Data(edge_index=edge_index, edge_weight=edge_weight, num_nodes=3)
    data = ToUndirected()(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == [[0, 0, 1, 1, 2], [0, 1, 0, 2, 1]]
    assert data.edge_weight.tolist() == [2, 2, 2, 1, 1]
    assert data.num_nodes == 3


def test_hetero_to_undirected():
    edge_index = torch.tensor([[2, 0], [3, 1]])
    edge_weight = torch.randn(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    perm = torch.tensor([1, 1, 0, 0])

    data = HeteroData()
    data['v'].num_nodes = 4
    data['w'].num_nodes = 4
    data['v', 'v'].edge_index = edge_index
    data['v', 'v'].edge_weight = edge_weight
    data['v', 'v'].edge_attr = edge_attr
    data['v', 'w'].edge_index = edge_index
    data['v', 'w'].edge_weight = edge_weight
    data['v', 'w'].edge_attr = edge_attr

    from torch_geometric.transforms import ToUndirected

    assert not data.is_undirected()
    data = ToUndirected()(data)
    assert data.is_undirected()

    assert data['v', 'v'].edge_index.tolist() == [[0, 1, 2, 3], [1, 0, 3, 2]]
    assert data['v', 'v'].edge_weight.tolist() == edge_weight[perm].tolist()
    assert data['v', 'v'].edge_attr.tolist() == edge_attr[perm].tolist()
    assert data['v', 'w'].edge_index.tolist() == edge_index.tolist()
    assert data['v', 'w'].edge_weight.tolist() == edge_weight.tolist()
    assert data['v', 'w'].edge_attr.tolist() == edge_attr.tolist()
    assert data['w', 'v'].edge_index.tolist() == [[3, 1], [2, 0]]
    assert data['w', 'v'].edge_weight.tolist() == edge_weight.tolist()
    assert data['w', 'v'].edge_attr.tolist() == edge_attr.tolist()
pytorch/test/transforms/test_two_hop.py000066400000000000000000000021471444262304300210210ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.testing import onlyLinux
from torch_geometric.transforms import TwoHop


@onlyLinux  # TODO  (matthias) Investigate CSR @ CSR support on Windows.
def test_two_hop():
    transform = TwoHop()
    assert str(transform) == 'TwoHop()'

    edge_index = torch.tensor([[0, 0, 0, 1, 2, 3], [1, 2, 3, 0, 0, 0]])
    edge_attr = torch.tensor([1, 2, 3, 1, 2, 3], dtype=torch.float)
    data = Data(edge_index=edge_index, edge_attr=edge_attr, num_nodes=4)

    data = transform(data)
    assert len(data) == 3
    assert data.edge_index.tolist() == [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                                        [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]]
    assert data.edge_attr.tolist() == [1, 2, 3, 1, 0, 0, 2, 0, 0, 3, 0, 0]
    assert data.num_nodes == 4

    data = Data(edge_index=edge_index, num_nodes=4)
    data = transform(data)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
                                        [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]]
    assert data.num_nodes == 4
pytorch/test/transforms/test_virtual_node.py000066400000000000000000000023111444262304300220260ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.transforms import VirtualNode


def test_virtual_node():
    assert str(VirtualNode()) == 'VirtualNode()'

    x = torch.randn(4, 16)
    edge_index = torch.tensor([[2, 0, 2], [3, 1, 0]])
    edge_weight = torch.rand(edge_index.size(1))
    edge_attr = torch.randn(edge_index.size(1), 8)

    data = Data(x=x, edge_index=edge_index, edge_weight=edge_weight,
                edge_attr=edge_attr, num_nodes=x.size(0))

    data = VirtualNode()(data)
    assert len(data) == 6

    assert data.x.size() == (5, 16)
    assert torch.allclose(data.x[:4], x)
    assert data.x[4:].abs().sum() == 0

    assert data.edge_index.tolist() == [[2, 0, 2, 0, 1, 2, 3, 4, 4, 4, 4],
                                        [3, 1, 0, 4, 4, 4, 4, 0, 1, 2, 3]]

    assert data.edge_weight.size() == (11, )
    assert torch.allclose(data.edge_weight[:3], edge_weight)
    assert data.edge_weight[3:].abs().sum() == 8

    assert data.edge_attr.size() == (11, 8)
    assert torch.allclose(data.edge_attr[:3], edge_attr)
    assert data.edge_attr[3:].abs().sum() == 0

    assert data.num_nodes == 5

    assert data.edge_type.tolist() == [0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]
pytorch/test/utils/000077500000000000000000000000001444262304300146675ustar00rootroot00000000000000pytorch/test/utils/test_assortativity.py000066400000000000000000000021441444262304300212260ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import assortativity


def test_assortativity():
    # Completely assortative graph:
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5],
                               [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2, 5, 4]])
    out = assortativity(edge_index)
    assert pytest.approx(out, abs=1e-5) == 1.0

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=[6, 6])
        out = assortativity(adj)
        assert pytest.approx(out, abs=1e-5) == 1.0

    # Completely disassortative graph:
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 5, 5, 5, 5],
                               [5, 5, 5, 5, 5, 0, 1, 2, 3, 4]])
    out = assortativity(edge_index)
    assert pytest.approx(out, abs=1e-5) == -1.0

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=[6, 6])
        out = assortativity(adj)
        assert pytest.approx(out, abs=1e-5) == -1.0
pytorch/test/utils/test_augmentation.py000066400000000000000000000073741444262304300210060ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.utils import (
    add_random_edge,
    is_undirected,
    mask_feature,
    shuffle_node,
)


def test_shuffle_node():
    x = torch.tensor([[0, 1, 2], [3, 4, 5]], dtype=torch.float)

    out = shuffle_node(x, training=False)
    assert out[0].tolist() == x.tolist()
    assert out[1].tolist() == list(range(len(x)))

    torch.manual_seed(5)
    out = shuffle_node(x)
    assert out[0].tolist() == [[3.0, 4.0, 5.0], [0.0, 1.0, 2.0]]
    assert out[1].tolist() == [1, 0]

    torch.manual_seed(10)
    x = torch.arange(21).view(7, 3).to(torch.float)
    batch = torch.tensor([0, 0, 1, 1, 2, 2, 2])
    out = shuffle_node(x, batch)
    assert out[0].tolist() == [[3.0, 4.0, 5.0], [0.0, 1.0, 2.0],
                               [9.0, 10.0, 11.0], [6.0, 7.0, 8.0],
                               [12.0, 13.0, 14.0], [18.0, 19.0, 20.0],
                               [15.0, 16.0, 17.0]]
    assert out[1].tolist() == [1, 0, 3, 2, 4, 6, 5]


def test_mask_feature():
    x = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]],
                     dtype=torch.float)

    out = mask_feature(x, training=False)
    assert out[0].tolist() == x.tolist()
    assert torch.all(out[1])

    torch.manual_seed(4)
    out = mask_feature(x)
    assert out[0].tolist() == [[1.0, 2.0, 0.0, 0.0], [5.0, 6.0, 0.0, 0.0],
                               [9.0, 10.0, 0.0, 0.0]]
    assert out[1].tolist() == [[True, True, False, False]]

    torch.manual_seed(5)
    out = mask_feature(x, mode='row')
    assert out[0].tolist() == [[1.0, 2.0, 3.0, 4.0], [0.0, 0.0, 0.0, 0.0],
                               [9.0, 10.0, 11.0, 12.0]]
    assert out[1].tolist() == [[True], [False], [True]]

    torch.manual_seed(7)
    out = mask_feature(x, mode='all')
    assert out[0].tolist() == [[1.0, 0.0, 3.0, 4.0], [0.0, 0.0, 0.0, 8.0],
                               [0.0, 10.0, 11.0, 12.0]]

    assert out[1].tolist() == [[True, False, True, True],
                               [False, False, False, True],
                               [False, True, True, True]]

    torch.manual_seed(7)
    out = mask_feature(x, mode='all', fill_value=-1)
    assert out[0].tolist() == [[1.0, -1.0, 3.0, 4.0], [-1.0, -1.0, -1.0, 8.0],
                               [-1.0, 10.0, 11.0, 12.0]]

    assert out[1].tolist() == [[True, False, True, True],
                               [False, False, False, True],
                               [False, True, True, True]]


def test_add_random_edge():
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])
    out = add_random_edge(edge_index, p=0.5, training=False)
    assert out[0].tolist() == edge_index.tolist()
    assert out[1].tolist() == [[], []]

    torch.manual_seed(5)
    out = add_random_edge(edge_index, p=0.5)
    assert out[0].tolist() == [[0, 1, 1, 2, 2, 3, 3, 2, 3],
                               [1, 0, 2, 1, 3, 2, 1, 2, 2]]

    assert out[1].tolist() == [[3, 2, 3], [1, 2, 2]]

    torch.manual_seed(6)
    out = add_random_edge(edge_index, p=0.5, force_undirected=True)
    assert out[0].tolist() == [[0, 1, 1, 2, 2, 3, 1, 2],
                               [1, 0, 2, 1, 3, 2, 2, 1]]
    assert out[1].tolist() == [[1, 2], [2, 1]]
    assert is_undirected(out[0])
    assert is_undirected(out[1])

    # test with bipartite graph
    torch.manual_seed(7)
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5], [2, 3, 1, 4, 2, 1]])
    with pytest.raises(RuntimeError,
                       match="not supported for heterogeneous graphs"):
        out = add_random_edge(edge_index, p=0.5, force_undirected=True,
                              num_nodes=(6, 5))
    out = add_random_edge(edge_index, p=0.5, num_nodes=(6, 5))
    out[0].tolist() == [[0, 1, 2, 3, 4, 5, 3, 4, 1],
                        [2, 3, 1, 4, 2, 1, 1, 3, 2]]
pytorch/test/utils/test_coalesce.py000066400000000000000000000053441444262304300200640ustar00rootroot00000000000000from typing import List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.utils import coalesce


def test_coalesce():
    edge_index = torch.tensor([[2, 1, 1, 0, 2], [1, 2, 0, 1, 1]])
    edge_attr = torch.tensor([[1], [2], [3], [4], [5]])

    out = coalesce(edge_index)
    assert out.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]

    out = coalesce(edge_index, None)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1] is None

    out = coalesce(edge_index, edge_attr)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1].tolist() == [[4], [3], [2], [6]]

    out = coalesce(edge_index, [edge_attr, edge_attr.view(-1)])
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1][0].tolist() == [[4], [3], [2], [6]]
    assert out[1][1].tolist() == [4, 3, 2, 6]


def test_coalesce_without_duplicates():
    edge_index = torch.tensor([[2, 1, 1, 0], [1, 2, 0, 1]])
    edge_attr = torch.tensor([[1], [2], [3], [4]])

    out = coalesce(edge_index)
    assert out.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]

    out = coalesce(edge_index, None)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1] is None

    out = coalesce(edge_index, edge_attr)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1].tolist() == [[4], [3], [2], [1]]

    out = coalesce(edge_index, [edge_attr, edge_attr.view(-1)])
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1][0].tolist() == [[4], [3], [2], [1]]
    assert out[1][1].tolist() == [4, 3, 2, 1]


def test_coalesce_jit():
    @torch.jit.script
    def wrapper1(edge_index: Tensor) -> Tensor:
        return coalesce(edge_index)

    @torch.jit.script
    def wrapper2(
        edge_index: Tensor,
        edge_attr: Optional[Tensor],
    ) -> Tuple[Tensor, Optional[Tensor]]:
        return coalesce(edge_index, edge_attr)

    @torch.jit.script
    def wrapper3(
        edge_index: Tensor,
        edge_attr: List[Tensor],
    ) -> Tuple[Tensor, List[Tensor]]:
        return coalesce(edge_index, edge_attr)

    edge_index = torch.tensor([[2, 1, 1, 0], [1, 2, 0, 1]])
    edge_attr = torch.tensor([[1], [2], [3], [4]])

    out = wrapper1(edge_index)
    assert out.size() == edge_index.size()

    out = wrapper2(edge_index, None)
    assert out[0].size() == edge_index.size()
    assert out[1] is None

    out = wrapper2(edge_index, edge_attr)
    assert out[0].size() == edge_index.size()
    assert out[1].size() == edge_attr.size()

    out = wrapper3(edge_index, [edge_attr, edge_attr.view(-1)])
    assert out[0].size() == edge_index.size()
    assert len(out[1]) == 2
    assert out[1][0].size() == edge_attr.size()
    assert out[1][1].size() == edge_attr.view(-1).size()
pytorch/test/utils/test_convert.py000066400000000000000000000432771444262304300177750ustar00rootroot00000000000000import pytest
import scipy.sparse
import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.testing import withPackage
from torch_geometric.utils import (
    from_cugraph,
    from_dgl,
    from_networkit,
    from_networkx,
    from_scipy_sparse_matrix,
    from_trimesh,
    sort_edge_index,
    subgraph,
    to_cugraph,
    to_dgl,
    to_networkit,
    to_networkx,
    to_scipy_sparse_matrix,
    to_trimesh,
)


def test_to_scipy_sparse_matrix():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])

    adj = to_scipy_sparse_matrix(edge_index)
    assert isinstance(adj, scipy.sparse.coo_matrix) is True
    assert adj.shape == (2, 2)
    assert adj.row.tolist() == edge_index[0].tolist()
    assert adj.col.tolist() == edge_index[1].tolist()
    assert adj.data.tolist() == [1, 1, 1]

    edge_attr = torch.Tensor([1, 2, 3])
    adj = to_scipy_sparse_matrix(edge_index, edge_attr)
    assert isinstance(adj, scipy.sparse.coo_matrix) is True
    assert adj.shape == (2, 2)
    assert adj.row.tolist() == edge_index[0].tolist()
    assert adj.col.tolist() == edge_index[1].tolist()
    assert adj.data.tolist() == edge_attr.tolist()


def test_from_scipy_sparse_matrix():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    adj = to_scipy_sparse_matrix(edge_index)

    out = from_scipy_sparse_matrix(adj)
    assert out[0].tolist() == edge_index.tolist()
    assert out[1].tolist() == [1, 1, 1]


@withPackage('networkx')
def test_to_networkx():
    import networkx as nx

    x = torch.Tensor([[1, 2], [3, 4]])
    pos = torch.Tensor([[0, 0], [1, 1]])
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_attr = torch.Tensor([1, 2, 3])
    data = Data(x=x, pos=pos, edge_index=edge_index, weight=edge_attr)

    for remove_self_loops in [True, False]:
        G = to_networkx(data, node_attrs=['x', 'pos'], edge_attrs=['weight'],
                        remove_self_loops=remove_self_loops)

        assert G.nodes[0]['x'] == [1, 2]
        assert G.nodes[1]['x'] == [3, 4]
        assert G.nodes[0]['pos'] == [0, 0]
        assert G.nodes[1]['pos'] == [1, 1]

        if remove_self_loops:
            assert nx.to_numpy_array(G).tolist() == [[0, 1], [2, 0]]
        else:
            assert nx.to_numpy_array(G).tolist() == [[3, 1], [2, 0]]


@withPackage('networkx')
def test_from_networkx_set_node_attributes():
    import networkx as nx

    G = nx.path_graph(3)
    attrs = {
        0: {
            'x': torch.tensor([1, 0, 0])
        },
        1: {
            'x': torch.tensor([0, 1, 0])
        },
        2: {
            'x': torch.tensor([0, 0, 1])
        },
    }
    nx.set_node_attributes(G, attrs)

    assert from_networkx(G).x.tolist() == [[1, 0, 0], [0, 1, 0], [0, 0, 1]]


@withPackage('networkx')
def test_to_networkx_undirected():
    import networkx as nx

    x = torch.Tensor([[1, 2], [3, 4]])
    pos = torch.Tensor([[0, 0], [1, 1]])
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_attr = torch.Tensor([1, 2, 3])
    data = Data(x=x, pos=pos, edge_index=edge_index, weight=edge_attr)

    for remove_self_loops in [True, False]:
        G = to_networkx(data, node_attrs=['x', 'pos'], edge_attrs=['weight'],
                        remove_self_loops=remove_self_loops,
                        to_undirected=True)

        assert G.nodes[0]['x'] == [1, 2]
        assert G.nodes[1]['x'] == [3, 4]
        assert G.nodes[0]['pos'] == [0, 0]
        assert G.nodes[1]['pos'] == [1, 1]

        if remove_self_loops:
            assert nx.to_numpy_array(G).tolist() == [[0, 1], [1, 0]]
        else:
            assert nx.to_numpy_array(G).tolist() == [[3, 1], [1, 0]]

    G = to_networkx(data, edge_attrs=['weight'], to_undirected=False)
    assert nx.to_numpy_array(G).tolist() == [[3, 1], [2, 0]]

    G = to_networkx(data, edge_attrs=['weight'], to_undirected="upper")
    assert nx.to_numpy_array(G).tolist() == [[3, 1], [1, 0]]

    G = to_networkx(data, edge_attrs=['weight'], to_undirected="lower")
    assert nx.to_numpy_array(G).tolist() == [[3, 2], [2, 0]]


@withPackage('networkx')
def test_from_networkx():
    x = torch.randn(2, 8)
    pos = torch.randn(2, 3)
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_attr = torch.randn(edge_index.size(1))
    perm = torch.tensor([0, 2, 1])
    data = Data(x=x, pos=pos, edge_index=edge_index, edge_attr=edge_attr)
    G = to_networkx(data, node_attrs=['x', 'pos'], edge_attrs=['edge_attr'])
    data = from_networkx(G)
    assert len(data) == 4
    assert data.x.tolist() == x.tolist()
    assert data.pos.tolist() == pos.tolist()
    assert data.edge_index.tolist() == edge_index[:, perm].tolist()
    assert data.edge_attr.tolist() == edge_attr[perm].tolist()


@withPackage('networkx')
def test_from_networkx_group_attrs():
    x = torch.randn(2, 2)
    x1 = torch.randn(2, 4)
    x2 = torch.randn(2, 8)
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_attr1 = torch.randn(edge_index.size(1))
    edge_attr2 = torch.randn(edge_index.size(1))
    perm = torch.tensor([0, 2, 1])
    data = Data(x=x, x1=x1, x2=x2, edge_index=edge_index,
                edge_attr1=edge_attr1, edge_attr2=edge_attr2)
    G = to_networkx(data, node_attrs=['x', 'x1', 'x2'],
                    edge_attrs=['edge_attr1', 'edge_attr2'])
    data = from_networkx(G, group_node_attrs=['x', 'x2'], group_edge_attrs=all)
    assert len(data) == 4
    assert data.x.tolist() == torch.cat([x, x2], dim=-1).tolist()
    assert data.x1.tolist() == x1.tolist()
    assert data.edge_index.tolist() == edge_index[:, perm].tolist()
    assert data.edge_attr.tolist() == torch.stack([edge_attr1, edge_attr2],
                                                  dim=-1)[perm].tolist()


@withPackage('networkx')
def test_networkx_vice_versa_convert():
    import networkx as nx

    G = nx.complete_graph(5)
    assert G.is_directed() is False
    data = from_networkx(G)
    assert data.is_directed() is False
    G = to_networkx(data)
    assert G.is_directed() is True
    G = nx.to_undirected(G)
    assert G.is_directed() is False


@withPackage('networkx')
def test_from_networkx_non_consecutive():
    import networkx as nx

    graph = nx.Graph()
    graph.add_node(4)
    graph.add_node(2)
    graph.add_edge(4, 2)
    for node in graph.nodes():
        graph.nodes[node]['x'] = node

    data = from_networkx(graph)
    assert len(data) == 2
    assert data.x.tolist() == [4, 2]
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]


@withPackage('networkx')
def test_from_networkx_inverse():
    import networkx as nx

    graph = nx.Graph()
    graph.add_node(3)
    graph.add_node(2)
    graph.add_node(1)
    graph.add_node(0)
    graph.add_edge(3, 1)
    graph.add_edge(2, 1)
    graph.add_edge(1, 0)

    data = from_networkx(graph)
    assert len(data) == 2
    assert data.edge_index.tolist() == [[0, 1, 2, 2, 2, 3], [2, 2, 0, 1, 3, 2]]
    assert data.num_nodes == 4


@withPackage('networkx')
def test_from_networkx_non_numeric_labels():
    import networkx as nx

    graph = nx.Graph()
    graph.add_node('4')
    graph.add_node('2')
    graph.add_edge('4', '2')
    for node in graph.nodes():
        graph.nodes[node]['x'] = node
    data = from_networkx(graph)
    assert len(data) == 2
    assert data.x == ['4', '2']
    assert data.edge_index.tolist() == [[0, 1], [1, 0]]


@withPackage('networkx')
def test_from_networkx_without_edges():
    import networkx as nx

    graph = nx.Graph()
    graph.add_node(1)
    graph.add_node(2)
    data = from_networkx(graph)
    assert len(data) == 2
    assert data.edge_index.size() == (2, 0)
    assert data.num_nodes == 2


@withPackage('networkx')
def test_from_networkx_with_same_node_and_edge_attributes():
    import networkx as nx

    G = nx.Graph()
    G.add_nodes_from([(0, {'age': 1}), (1, {'age': 6}), (2, {'age': 5})])
    G.add_edges_from([(0, 1, {'age': 2}), (1, 2, {'age': 7})])

    data = from_networkx(G)
    assert len(data) == 4
    assert data.age.tolist() == [1, 6, 5]
    assert data.num_nodes == 3
    assert data.edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert data.edge_age.tolist() == [2, 2, 7, 7]

    data = from_networkx(G, group_node_attrs=all, group_edge_attrs=all)
    assert len(data) == 3
    assert data.x.tolist() == [[1], [6], [5]]
    assert data.edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert data.edge_attr.tolist() == [[2], [2], [7], [7]]


@withPackage('networkx')
def test_from_networkx_subgraph_convert():
    import networkx as nx

    G = nx.complete_graph(5)

    edge_index = from_networkx(G).edge_index
    sub_edge_index_1, _ = subgraph([0, 1, 3, 4], edge_index,
                                   relabel_nodes=True)

    sub_edge_index_2 = from_networkx(G.subgraph([0, 1, 3, 4])).edge_index

    assert sub_edge_index_1.tolist() == sub_edge_index_2.tolist()


@withPackage('networkit')
def test_to_networkit_vice_versa():
    edge_index = torch.tensor([[0, 1], [1, 0]])

    g = to_networkit(edge_index, directed=False)
    assert not g.isDirected()
    assert not g.isWeighted()

    edge_index, edge_weight = from_networkit(g)
    assert edge_index.tolist() == [[0, 1], [1, 0]]
    assert edge_weight is None


@withPackage('networkit')
@pytest.mark.parametrize('directed', [True, False])
@pytest.mark.parametrize('num_nodes', [None, 3])
@pytest.mark.parametrize('edge_weight', [None, torch.rand(3)])
def test_to_networkit(directed, edge_weight, num_nodes):
    import networkit

    edge_index = torch.tensor([[0, 1, 1], [1, 0, 2]], dtype=torch.long)
    g = to_networkit(edge_index, edge_weight, num_nodes, directed)

    assert isinstance(g, networkit.Graph)
    assert g.isDirected() == directed
    assert g.numberOfNodes() == 3

    if edge_weight is None:
        edge_weight = torch.tensor([1., 1., 1.])

    assert g.weight(0, 1) == float(edge_weight[0])
    assert g.weight(1, 2) == float(edge_weight[2])

    if directed:
        assert g.numberOfEdges() == 3
        assert g.weight(1, 0) == float(edge_weight[1])
    else:
        assert g.numberOfEdges() == 2


@pytest.mark.parametrize('directed', [True, False])
@pytest.mark.parametrize('weighted', [True, False])
@withPackage('networkit')
def test_from_networkit(directed, weighted):
    import networkit

    g = networkit.Graph(3, weighted=weighted, directed=directed)
    g.addEdge(0, 1)
    g.addEdge(1, 2)
    if directed:
        g.addEdge(1, 0)

    if weighted:
        for i, (u, v) in enumerate(g.iterEdges()):
            g.setWeight(u, v, i + 1)

    edge_index, edge_weight = from_networkit(g)

    if directed:
        assert edge_index.tolist() == [[0, 1, 1], [1, 2, 0]]
        if weighted:
            assert edge_weight.tolist() == [1, 2, 3]
        else:
            assert edge_weight is None
    else:
        assert edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
        if weighted:
            assert edge_weight.tolist() == [1, 1, 2, 2]
        else:
            assert edge_weight is None


@withPackage('trimesh')
def test_trimesh_vice_versa():
    pos = torch.tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 0]],
                       dtype=torch.float)
    face = torch.tensor([[0, 1, 2], [1, 2, 3]]).t()

    data = Data(pos=pos, face=face)
    mesh = to_trimesh(data)
    data = from_trimesh(mesh)

    assert pos.tolist() == data.pos.tolist()
    assert face.tolist() == data.face.tolist()


@withPackage('trimesh')
def test_to_trimesh():
    import trimesh

    pos = torch.tensor([[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 0]])
    face = torch.tensor([[0, 1, 2], [2, 1, 3]]).t()
    data = Data(pos=pos, face=face)

    obj = to_trimesh(data)

    assert isinstance(obj, trimesh.Trimesh)
    assert obj.vertices.shape == (4, 3)
    assert obj.faces.shape == (2, 3)
    assert obj.vertices.tolist() == data.pos.tolist()
    assert obj.faces.tolist() == data.face.t().contiguous().tolist()


@withPackage('trimesh')
def test_from_trimesh():
    import trimesh

    vertices = [[0, 0, 0], [1, 0, 0], [0, 1, 0]]
    faces = [[0, 1, 2]]
    mesh = trimesh.Trimesh(vertices=vertices, faces=faces, process=False)

    data = from_trimesh(mesh)

    assert data.pos.tolist() == vertices
    assert data.face.t().contiguous().tolist() == faces


@withPackage('cudf')
@withPackage('cugraph')
@pytest.mark.parametrize('edge_weight', [None, torch.rand(4)])
@pytest.mark.parametrize('relabel_nodes', [True, False])
@pytest.mark.parametrize('directed', [True, False])
def test_to_cugraph(edge_weight, directed, relabel_nodes):
    import cugraph

    if directed:
        edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    else:
        edge_index = torch.tensor([[0, 1], [1, 2]])

    if edge_weight is not None:
        edge_weight = edge_weight[:edge_index.size(1)]

    graph = to_cugraph(edge_index, edge_weight, relabel_nodes, directed)
    assert isinstance(graph, cugraph.Graph)
    assert graph.number_of_nodes() == 3

    edge_list = graph.view_edge_list()
    assert edge_list is not None

    edge_list = edge_list.sort_values(by=['src', 'dst'])

    cu_edge_index = edge_list[['src', 'dst']].to_pandas().values
    cu_edge_index = torch.from_numpy(cu_edge_index).t()
    cu_edge_weight = None
    if edge_weight is not None:
        cu_edge_weight = edge_list['weights'].to_pandas().values
        cu_edge_weight = torch.from_numpy(cu_edge_weight)

    cu_edge_index, cu_edge_weight = sort_edge_index(cu_edge_index,
                                                    cu_edge_weight)

    assert torch.equal(edge_index, cu_edge_index.cpu())
    if edge_weight is not None:
        assert torch.allclose(edge_weight, cu_edge_weight.cpu())


@withPackage('cudf')
@withPackage('cugraph')
@pytest.mark.parametrize('edge_weight', [None, torch.randn(4)])
@pytest.mark.parametrize('directed', [True, False])
@pytest.mark.parametrize('relabel_nodes', [True, False])
def test_from_cugraph(edge_weight, directed, relabel_nodes):
    import cudf
    import cugraph
    from torch.utils.dlpack import to_dlpack

    if directed:
        edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]])
    else:
        edge_index = torch.tensor([[0, 1], [1, 2]])

    if edge_weight is not None:
        edge_weight = edge_weight[:edge_index.size(1)]

    G = cugraph.Graph(directed=directed)
    df = cudf.from_dlpack(to_dlpack(edge_index.t()))
    if edge_weight is not None:
        df['2'] = cudf.from_dlpack(to_dlpack(edge_weight))

    G.from_cudf_edgelist(
        df,
        source=0,
        destination=1,
        edge_attr='2' if edge_weight is not None else None,
        renumber=relabel_nodes,
    )

    cu_edge_index, cu_edge_weight = from_cugraph(G)
    cu_edge_index, cu_edge_weight = sort_edge_index(cu_edge_index,
                                                    cu_edge_weight)

    assert torch.equal(edge_index, cu_edge_index.cpu())
    if edge_weight is not None:
        assert torch.allclose(edge_weight, cu_edge_weight.cpu())
    else:
        assert cu_edge_weight is None


@withPackage('dgl')
def test_to_dgl_graph():
    x = torch.randn(5, 3)
    edge_index = torch.tensor([[0, 1, 1, 2, 3, 0], [1, 0, 2, 1, 4, 4]])
    edge_attr = torch.randn(edge_index.size(1), 2)
    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr)

    g = to_dgl(data)

    assert torch.equal(data.x, g.ndata['x'])
    row, col = g.edges()
    assert torch.equal(row, edge_index[0])
    assert torch.equal(col, edge_index[1])
    assert torch.equal(data.edge_attr, g.edata['edge_attr'])


@withPackage('dgl')
def test_to_dgl_hetero_graph():
    data = HeteroData()
    data['v1'].x = torch.randn(4, 3)
    data['v2'].x = torch.randn(4, 3)
    data['v1', 'v2'].edge_index = torch.tensor([[0, 1, 2, 3], [0, 1, 2, 3]])
    data['v1', 'v2'].edge_attr = torch.randn(4, 2)

    g = to_dgl(data)

    assert data['v1', 'v2'].num_edges == g.num_edges(('v1', 'to', 'v2'))
    assert data['v1'].num_nodes == g.num_nodes('v1')
    assert data['v2'].num_nodes == g.num_nodes('v2')
    assert torch.equal(data['v1'].x, g.nodes['v1'].data['x'])
    assert torch.equal(data['v2'].x, g.nodes['v2'].data['x'])
    row, col = g.edges()
    assert torch.equal(row, data['v1', 'v2'].edge_index[0])
    assert torch.equal(col, data['v1', 'v2'].edge_index[1])
    assert torch.equal(g.edata['edge_attr'], data['v1', 'v2'].edge_attr)


@withPackage('dgl')
@withPackage('torch_sparse')
def test_to_dgl_sparse():
    from torch_geometric.transforms import ToSparseTensor
    x = torch.randn(5, 3)
    edge_index = torch.tensor([[0, 1, 1, 2, 3, 0], [1, 0, 2, 1, 4, 4]])
    edge_attr = torch.randn(edge_index.size(1), 2)
    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr)
    data = ToSparseTensor()(data)

    g = to_dgl(data)

    assert torch.equal(data.x, g.ndata["x"])
    pyg_row, pyg_col, _ = data.adj_t.t().coo()
    dgl_row, dgl_col = g.edges()
    assert torch.equal(pyg_row, dgl_row)
    assert torch.equal(pyg_col, dgl_col)
    assert torch.equal(data.edge_attr, g.edata['edge_attr'])


@withPackage('dgl')
def test_from_dgl_graph():
    import dgl
    g = dgl.graph(([0, 0, 1, 5], [1, 2, 2, 0]))
    g.ndata['x'] = torch.randn(g.num_nodes(), 3)
    g.edata['edge_attr'] = torch.randn(g.num_edges())

    data = from_dgl(g)

    assert torch.equal(data.x, g.ndata['x'])
    row, col = g.edges()
    assert torch.equal(data.edge_index[0], row)
    assert torch.equal(data.edge_index[1], col)
    assert torch.equal(data.edge_attr, g.edata['edge_attr'])


@withPackage('dgl')
def test_from_dgl_hetero_graph():
    import dgl
    g = dgl.heterograph({
        ('v1', 'to', 'v2'): (
            [0, 1, 1, 2, 3, 3, 4],
            [0, 0, 1, 1, 1, 2, 2],
        )
    })
    g.nodes['v1'].data['x'] = torch.randn(5, 3)
    g.nodes['v2'].data['x'] = torch.randn(3, 3)

    data = from_dgl(g)

    assert data['v1', 'v2'].num_edges == g.num_edges(('v1', 'to', 'v2'))
    assert data['v1'].num_nodes == g.num_nodes('v1')
    assert data['v2'].num_nodes == g.num_nodes('v2')
    assert torch.equal(data['v1'].x, g.nodes['v1'].data['x'])
    assert torch.equal(data['v2'].x, g.nodes['v2'].data['x'])
pytorch/test/utils/test_cross_entropy.py000066400000000000000000000017161444262304300212160ustar00rootroot00000000000000import torch
import torch.nn.functional as F

from torch_geometric.utils.cross_entropy import sparse_cross_entropy


def test_sparse_cross_entropy_multiclass():
    x = torch.randn(5, 5, requires_grad=True)
    y = torch.eye(5)
    edge_label_index = y.nonzero().t()

    expected = F.cross_entropy(x, y)
    expected.backward()
    expected_grad = x.grad

    x.grad = None
    out = sparse_cross_entropy(x, edge_label_index)
    out.backward()

    assert torch.allclose(expected, out)
    assert torch.allclose(expected_grad, x.grad)


def test_sparse_cross_entropy_multilabel():
    x = torch.randn(4, 4, requires_grad=True)
    y = torch.randint_like(x, 0, 2)
    edge_label_index = y.nonzero().t()

    expected = F.cross_entropy(x, y)
    expected.backward()
    expected_grad = x.grad

    x.grad = None
    out = sparse_cross_entropy(x, edge_label_index)
    out.backward()

    assert torch.allclose(expected, out)
    assert torch.allclose(expected_grad, x.grad)
pytorch/test/utils/test_degree.py000066400000000000000000000003441444262304300175340ustar00rootroot00000000000000import torch

from torch_geometric.utils import degree


def test_degree():
    row = torch.tensor([0, 1, 0, 2, 0])
    deg = degree(row, dtype=torch.long)
    assert deg.dtype == torch.long
    assert deg.tolist() == [3, 1, 1]
pytorch/test/utils/test_dropout.py000066400000000000000000000067611444262304300200060ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.testing import withPackage
from torch_geometric.utils import (
    dropout_adj,
    dropout_edge,
    dropout_node,
    dropout_path,
)


def test_dropout_adj():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])
    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6])

    with pytest.warns(UserWarning, match="'dropout_adj' is deprecated"):
        out = dropout_adj(edge_index, edge_attr, training=False)
    assert edge_index.tolist() == out[0].tolist()
    assert edge_attr.tolist() == out[1].tolist()

    torch.manual_seed(5)
    with pytest.warns(UserWarning, match="'dropout_adj' is deprecated"):
        out = dropout_adj(edge_index, edge_attr)
    assert out[0].tolist() == [[0, 1, 2, 2], [1, 2, 1, 3]]
    assert out[1].tolist() == [1, 3, 4, 5]

    torch.manual_seed(6)
    with pytest.warns(UserWarning, match="'dropout_adj' is deprecated"):
        out = dropout_adj(edge_index, edge_attr, force_undirected=True)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 2, 0, 1]]
    assert out[1].tolist() == [1, 3, 1, 3]


def test_dropout_node():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])

    out = dropout_node(edge_index, training=False)
    assert edge_index.tolist() == out[0].tolist()
    assert out[1].tolist() == [True, True, True, True, True, True]
    assert out[2].tolist() == [True, True, True, True]

    torch.manual_seed(5)
    out = dropout_node(edge_index)
    assert out[0].tolist() == [[2, 3], [3, 2]]
    assert out[1].tolist() == [False, False, False, False, True, True]
    assert out[2].tolist() == [True, False, True, True]


def test_dropout_edge():
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])

    out = dropout_edge(edge_index, training=False)
    assert edge_index.tolist() == out[0].tolist()
    assert out[1].tolist() == [True, True, True, True, True, True]

    torch.manual_seed(5)
    out = dropout_edge(edge_index)
    assert out[0].tolist() == [[0, 1, 2, 2], [1, 2, 1, 3]]
    assert out[1].tolist() == [True, False, True, True, True, False]

    torch.manual_seed(6)
    out = dropout_edge(edge_index, force_undirected=True)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 2, 0, 1]]
    assert out[1].tolist() == [0, 2, 0, 2]


@withPackage('torch_cluster')
def test_dropout_path():
    edge_index = torch.tensor([[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]])

    out = dropout_path(edge_index, training=False)
    assert edge_index.tolist() == out[0].tolist()
    assert out[1].tolist() == [True, True, True, True, True, True]

    torch.manual_seed(4)
    out = dropout_path(edge_index, p=0.2)
    assert out[0].tolist() == [[0, 1], [1, 0]]
    assert out[1].tolist() == [True, True, False, False, False, False]
    assert edge_index[:, out[1]].tolist() == out[0].tolist()

    # test with unsorted edges
    torch.manual_seed(5)
    edge_index = torch.tensor([[3, 5, 2, 2, 2, 1], [1, 0, 0, 1, 3, 2]])
    out = dropout_path(edge_index, p=0.2)
    assert out[0].tolist() == [[3, 2, 2, 1], [1, 1, 3, 2]]
    assert out[1].tolist() == [True, False, False, True, True, True]
    assert edge_index[:, out[1]].tolist() == out[0].tolist()

    # test with isolated nodes
    torch.manual_seed(7)
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 0, 2, 4]])
    out = dropout_path(edge_index, p=0.2)
    assert out[0].tolist() == [[2, 3], [2, 4]]
    assert out[1].tolist() == [False, False, True, True]
    assert edge_index[:, out[1]].tolist() == out[0].tolist()
pytorch/test/utils/test_embedding.py000066400000000000000000000017161444262304300202230ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.nn import GCNConv, Linear
from torch_geometric.utils import get_embeddings


class GNN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(5, 6)
        self.conv2 = GCNConv(6, 7)

    def forward(self, x0, edge_index):
        x1 = self.conv1(x0, edge_index)
        x2 = self.conv2(x1, edge_index)
        return [x1, x2]


def test_get_embeddings():
    x = torch.randn(6, 5)
    edge_index = torch.tensor([[0, 1, 2, 3, 4], [1, 2, 3, 4, 5]])

    with pytest.warns(UserWarning, match="any 'MessagePassing' layers"):
        intermediate_outs = get_embeddings(Linear(5, 5), x)
    assert len(intermediate_outs) == 0

    model = GNN()
    expected_embeddings = model(x, edge_index)

    embeddings = get_embeddings(model, x, edge_index)
    assert len(embeddings) == 2
    for expected, out in zip(expected_embeddings, embeddings):
        assert torch.allclose(expected, out)
pytorch/test/utils/test_geodesic.py000066400000000000000000000024361444262304300200670ustar00rootroot00000000000000from math import sqrt

import pytest
import torch

from torch_geometric.testing import withPackage
from torch_geometric.utils import geodesic_distance


@withPackage('gdist')
@pytest.mark.skip(reason="No way of currently testing this")
def test_geodesic_distance():
    pos = torch.Tensor([[0, 0, 0], [2, 0, 0], [0, 2, 0], [2, 2, 0]])
    face = torch.tensor([[0, 1, 3], [0, 2, 3]]).t()

    out = geodesic_distance(pos, face)
    expected = [
        [0, 1, 1, sqrt(2)],
        [1, 0, sqrt(2), 1],
        [1, sqrt(2), 0, 1],
        [sqrt(2), 1, 1, 0],
    ]
    assert torch.allclose(out, torch.tensor(expected))
    assert torch.allclose(out, geodesic_distance(pos, face, num_workers=-1))

    out = geodesic_distance(pos, face, norm=False)
    expected = [
        [0, 2, 2, 2 * sqrt(2)],
        [2, 0, 2 * sqrt(2), 2],
        [2, 2 * sqrt(2), 0, 2],
        [2 * sqrt(2), 2, 2, 0],
    ]
    assert torch.allclose(out, torch.tensor(expected))

    src = torch.tensor([0, 0, 0, 0])
    dest = torch.tensor([0, 1, 2, 3])
    out = geodesic_distance(pos, face, src=src, dest=dest)
    expected = [0, 1, 1, sqrt(2)]
    assert torch.allclose(out, torch.tensor(expected))

    out = geodesic_distance(pos, face, dest=dest)
    expected = [0, 0, 0, 0]
    assert torch.allclose(out, torch.Tensor(expected))
pytorch/test/utils/test_get_laplacian.py000066400000000000000000000021471444262304300210670ustar00rootroot00000000000000import torch

from torch_geometric.testing import is_full_test
from torch_geometric.utils import get_laplacian


def test_get_laplacian():
    edge_index = torch.tensor([[0, 1, 1, 2], [1, 0, 2, 1]], dtype=torch.long)
    edge_weight = torch.tensor([1, 2, 2, 4], dtype=torch.float)

    lap = get_laplacian(edge_index, edge_weight)
    assert lap[0].tolist() == [[0, 1, 1, 2, 0, 1, 2], [1, 0, 2, 1, 0, 1, 2]]
    assert lap[1].tolist() == [-1, -2, -2, -4, 1, 4, 4]

    if is_full_test():
        jit = torch.jit.script(get_laplacian)
        lap = jit(edge_index, edge_weight)
        assert lap[0].tolist() == [[0, 1, 1, 2, 0, 1, 2],
                                   [1, 0, 2, 1, 0, 1, 2]]
        assert lap[1].tolist() == [-1, -2, -2, -4, 1, 4, 4]

    lap_sym = get_laplacian(edge_index, edge_weight, normalization='sym')
    assert lap_sym[0].tolist() == lap[0].tolist()
    assert lap_sym[1].tolist() == [-0.5, -1, -0.5, -1, 1, 1, 1]

    lap_rw = get_laplacian(edge_index, edge_weight, normalization='rw')
    assert lap_rw[0].tolist() == lap[0].tolist()
    assert lap_rw[1].tolist() == [-1, -0.5, -0.5, -1, 1, 1, 1]
pytorch/test/utils/test_get_mesh_laplacian.py000066400000000000000000000057341444262304300221100ustar00rootroot00000000000000import torch

from torch_geometric.utils import get_mesh_laplacian


def test_get_mesh_laplacian_of_cube():
    pos = torch.tensor([
        [1.0, 1.0, 1.0],
        [1.0, -1.0, 1.0],
        [-1.0, -1.0, 1.0],
        [-1.0, 1.0, 1.0],
        [1.0, 1.0, -1.0],
        [1.0, -1.0, -1.0],
        [-1.0, -1.0, -1.0],
        [-1.0, 1.0, -1.0],
    ])

    face = torch.tensor([
        [0, 1, 2],
        [0, 3, 2],
        [4, 5, 1],
        [4, 0, 1],
        [7, 6, 5],
        [7, 4, 5],
        [3, 2, 6],
        [3, 7, 6],
        [4, 0, 3],
        [4, 7, 3],
        [1, 5, 6],
        [1, 2, 6],
    ])

    edge_index, edge_weight = get_mesh_laplacian(pos, face.t(),
                                                 normalization='rw')

    assert edge_index.tolist() == [
        [
            0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4,
            4, 5, 5, 5, 5, 6, 6, 6, 6, 6, 7, 7, 7, 7, 0, 1, 2, 3, 4, 5, 6, 7
        ],
        [
            1, 2, 3, 4, 0, 2, 4, 5, 6, 0, 1, 3, 6, 0, 2, 4, 6, 7, 0, 1, 3, 5,
            7, 1, 4, 6, 7, 1, 2, 3, 5, 7, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7
        ],
    ]

    assert torch.allclose(
        edge_weight,
        torch.tensor([
            0.375, 0.0, 0.375, 0.375, 0.3, 0.3, 0.0, 0.3, 0.0, 0.0, 0.375,
            0.375, 0.375, 0.3, 0.3, 0.0, 0.0, 0.3, 0.3, 0.0, 0.0, 0.3, 0.3,
            0.375, 0.375, 0.375, 0.0, 0.0, 0.3, 0.0, 0.3, 0.3, 0.375, 0.375,
            0.0, 0.375, -1.125, -0.9, -1.125, -0.9, -0.9, -1.125, -0.9, -1.125
        ]))


def test_get_mesh_laplacian_of_irregular_triangular_prism():
    pos = torch.tensor([
        [0.0, 0.0, 0.0],
        [4.0, 0.0, 0.0],
        [0.0, 0.0, -3.0],
        [1.0, 5.0, -1.0],
        [3.0, 5.0, -1.0],
        [2.0, 5.0, -2.0],
    ])

    face = torch.tensor([
        [0, 1, 2],
        [3, 4, 5],
        [0, 1, 4],
        [0, 3, 4],
        [1, 2, 5],
        [1, 4, 5],
        [2, 0, 3],
        [2, 5, 3],
    ])

    edge_index, edge_weight = get_mesh_laplacian(pos, face.t(),
                                                 normalization='rw')

    assert edge_index.tolist() == [
        [
            0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5,
            5, 5, 0, 1, 2, 3, 4, 5
        ],
        [
            1, 2, 3, 4, 0, 2, 4, 5, 0, 1, 3, 5, 0, 2, 4, 5, 0, 1, 3, 5, 1, 2,
            3, 4, 0, 1, 2, 3, 4, 5
        ],
    ]

    assert torch.allclose(
        edge_weight,
        torch.tensor([
            0.09730332, 0.15039921, 0.05081503, 0.00000000, 0.08726977,
            0.03521059, 0.05363689, 0.00723919, 0.14497279, 0.03784235,
            0.01629947, 0.03438699, 0.08362866, 0.02782887, 0.24252312,
            0.40727590, 0.00000000, 0.08728313, 0.21507657, 0.38582093,
            0.01117009, 0.04936920, 0.34247482, 0.36583540, -0.29851755,
            -0.18335645, -0.23350160, -0.76125660, -0.68818060, -0.76884955
        ]))
pytorch/test/utils/test_grid.py000066400000000000000000000015221444262304300172250ustar00rootroot00000000000000import torch

from torch_geometric.testing import is_full_test
from torch_geometric.utils import grid


def test_grid():
    (row, col), pos = grid(height=3, width=2)

    expected_row = [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2]
    expected_col = [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5]
    expected_row += [3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5]
    expected_col += [0, 1, 2, 3, 4, 5, 2, 3, 4, 5, 2, 3, 4, 5]

    expected_pos = [[0, 2], [1, 2], [0, 1], [1, 1], [0, 0], [1, 0]]

    assert row.tolist() == expected_row
    assert col.tolist() == expected_col
    assert pos.tolist() == expected_pos

    if is_full_test():
        jit = torch.jit.script(grid)
        (row, col), pos = jit(height=3, width=2)
        assert row.tolist() == expected_row
        assert col.tolist() == expected_col
        assert pos.tolist() == expected_pos
pytorch/test/utils/test_hetero.py000066400000000000000000000022761444262304300175750ustar00rootroot00000000000000import torch

from torch_geometric.testing import get_random_edge_index
from torch_geometric.utils.hetero import construct_bipartite_edge_index


def test_construct_bipartite_edge_index():
    edge_index = get_random_edge_index(4, 6, num_edges=20)

    edge_index_dict = {
        ('author', 'paper'): edge_index,
        ('paper', 'author'): edge_index.flip([0]),
    }
    edge_attr_dict = {
        ('author', 'paper'): torch.randn(edge_index.size(1), 16),
        ('paper', 'author'): torch.randn(edge_index.size(1), 16)
    }

    edge_index, edge_attr = construct_bipartite_edge_index(
        edge_index_dict,
        src_offset_dict={
            ('author', 'paper'): 0,
            ('paper', 'author'): 4
        },
        dst_offset_dict={
            'author': 0,
            'paper': 4
        },
        edge_attr_dict=edge_attr_dict,
    )

    assert edge_index.size() == (2, 40)
    assert edge_index.min() >= 0
    assert edge_index[0].max() > 4 and edge_index[1].max() > 6
    assert edge_index.max() <= 10
    assert edge_attr.size() == (40, 16)
    assert torch.equal(edge_attr[:20], edge_attr_dict['author', 'paper'])
    assert torch.equal(edge_attr[20:], edge_attr_dict['paper', 'author'])
pytorch/test/utils/test_homophily.py000066400000000000000000000024701444262304300203130ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import homophily


def test_homophily():
    edge_index = torch.tensor([[0, 1, 2, 3], [1, 2, 0, 4]])
    y = torch.tensor([0, 0, 0, 0, 1])
    batch = torch.tensor([0, 0, 0, 1, 1])
    row, col = edge_index
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj = SparseTensor(row=row, col=col, sparse_sizes=(5, 5))

    method = 'edge'
    assert pytest.approx(homophily(edge_index, y, method=method)) == 0.75
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert pytest.approx(homophily(adj, y, method=method)) == 0.75
    assert homophily(edge_index, y, batch, method).tolist() == [1., 0.]

    method = 'node'
    assert pytest.approx(homophily(edge_index, y, method=method)) == 0.6
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert pytest.approx(homophily(adj, y, method=method)) == 0.6
    assert homophily(edge_index, y, batch, method).tolist() == [1., 0.]

    method = 'edge_insensitive'
    assert pytest.approx(homophily(edge_index, y, method=method)) == 0.1999999
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert pytest.approx(homophily(adj, y, method=method)) == 0.1999999
    assert homophily(edge_index, y, batch, method).tolist() == [0., 0.]
pytorch/test/utils/test_isolated.py000066400000000000000000000030021444262304300200770ustar00rootroot00000000000000import torch

from torch_geometric.testing import is_full_test
from torch_geometric.utils import (
    contains_isolated_nodes,
    remove_isolated_nodes,
)


def test_contains_isolated_nodes():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    assert not contains_isolated_nodes(edge_index)
    assert contains_isolated_nodes(edge_index, num_nodes=3)

    if is_full_test():
        jit = torch.jit.script(contains_isolated_nodes)
        assert not jit(edge_index)
        assert jit(edge_index, num_nodes=3)

    edge_index = torch.tensor([[0, 1, 2, 0], [1, 0, 2, 0]])
    assert contains_isolated_nodes(edge_index)


def test_remove_isolated_nodes():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])

    out, _, mask = remove_isolated_nodes(edge_index)
    assert out.tolist() == [[0, 1, 0], [1, 0, 0]]
    assert mask.tolist() == [1, 1]

    if is_full_test():
        jit = torch.jit.script(remove_isolated_nodes)
        out, _, mask = jit(edge_index)
        assert out.tolist() == [[0, 1, 0], [1, 0, 0]]
        assert mask.tolist() == [1, 1]

    out, _, mask = remove_isolated_nodes(edge_index, num_nodes=3)
    assert out.tolist() == [[0, 1, 0], [1, 0, 0]]
    assert mask.tolist() == [1, 1, 0]

    edge_index = torch.tensor([[0, 2, 1, 0, 2], [2, 0, 1, 0, 2]])
    edge_attr = torch.tensor([1, 2, 3, 4, 5])
    out1, out2, mask = remove_isolated_nodes(edge_index, edge_attr)
    assert out1.tolist() == [[0, 1, 0, 1], [1, 0, 0, 1]]
    assert out2.tolist() == [1, 2, 4, 5]
    assert mask.tolist() == [1, 0, 1]
pytorch/test/utils/test_loop.py000066400000000000000000000162411444262304300172550ustar00rootroot00000000000000import torch

from torch_geometric.utils import (
    add_remaining_self_loops,
    add_self_loops,
    contains_self_loops,
    get_self_loop_attr,
    remove_self_loops,
    segregate_self_loops,
    to_torch_coo_tensor,
)


def test_contains_self_loops():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    assert contains_self_loops(edge_index)

    edge_index = torch.tensor([[0, 1, 1], [1, 0, 2]])
    assert not contains_self_loops(edge_index)


def test_remove_self_loops():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_attr = torch.tensor([[1, 2], [3, 4], [5, 6]])

    expected = [[0, 1], [1, 0]]

    out = remove_self_loops(edge_index)
    assert out[0].tolist() == expected
    assert out[1] is None

    out = remove_self_loops(edge_index, edge_attr)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [[1, 2], [3, 4]]

    adj = to_torch_coo_tensor(edge_index)
    adj, _ = remove_self_loops(adj)
    assert torch.diag(adj.to_dense()).tolist() == [0, 0]


def test_segregate_self_loops():
    edge_index = torch.tensor([[0, 0, 1], [0, 1, 0]])

    out = segregate_self_loops(edge_index)
    assert out[0].tolist() == [[0, 1], [1, 0]]
    assert out[1] is None
    assert out[2].tolist() == [[0], [0]]
    assert out[3] is None

    edge_attr = torch.tensor([1, 2, 3])
    out = segregate_self_loops(edge_index, edge_attr)
    assert out[0].tolist() == [[0, 1], [1, 0]]
    assert out[1].tolist() == [2, 3]
    assert out[2].tolist() == [[0], [0]]
    assert out[3].tolist() == [1]


def test_add_self_loops():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_weight = torch.tensor([0.5, 0.5, 0.5])
    edge_attr = torch.eye(3)
    adj = to_torch_coo_tensor(edge_index, edge_weight)

    expected = [[0, 1, 0, 0, 1], [1, 0, 0, 0, 1]]
    assert add_self_loops(edge_index)[0].tolist() == expected

    out = add_self_loops(edge_index, edge_weight)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 1., 1.]

    out = add_self_loops(adj)[0]
    assert out._indices().tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert out._values().tolist() == [1.5, 0.5, 0.5, 1.0]

    out = add_self_loops(edge_index, edge_weight, fill_value=5)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 5.0, 5.0]

    out = add_self_loops(adj, fill_value=5)[0]
    assert out._indices().tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert out._values().tolist() == [5.5, 0.5, 0.5, 5.0]

    out = add_self_loops(edge_index, edge_weight, fill_value=torch.tensor(2.))
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 2., 2.]

    out = add_self_loops(adj, fill_value=torch.tensor(2.))[0]
    assert out._indices().tolist() == [[0, 0, 1, 1], [0, 1, 0, 1]]
    assert out._values().tolist() == [2.5, 0.5, 0.5, 2.0]

    out = add_self_loops(edge_index, edge_weight, fill_value='add')
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 1, 0.5]

    # Tests with `edge_attr`:
    out = add_self_loops(edge_index, edge_attr)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [[1., 0., 0.], [0., 1., 0.], [0., 0., 1.],
                               [1., 1., 1.], [1., 1., 1.]]

    out = add_self_loops(edge_index, edge_attr,
                         fill_value=torch.tensor([0., 1., 0.]))
    assert out[0].tolist() == expected
    assert out[1].tolist() == [[1., 0., 0.], [0., 1., 0.], [0., 0., 1.],
                               [0., 1., 0.], [0., 1., 0.]]

    out = add_self_loops(edge_index, edge_attr, fill_value='add')
    assert out[0].tolist() == expected
    assert out[1].tolist() == [[1., 0., 0.], [0., 1., 0.], [0., 0., 1.],
                               [0., 1., 1.], [1., 0., 0.]]

    # Test empty `edge_index` and `edge_weight`:
    edge_index = torch.empty(2, 0, dtype=torch.long)
    edge_weight = torch.empty(0)
    out = add_self_loops(edge_index, edge_weight, num_nodes=1)
    assert out[0].tolist() == [[0], [0]]
    assert out[1].tolist() == [1.]


def test_add_self_loops_bipartite():
    edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
    adj = to_torch_coo_tensor(edge_index, size=(4, 2))

    edge_index, _ = add_self_loops(edge_index, num_nodes=(4, 2))
    assert edge_index.tolist() == [[0, 1, 2, 3, 0, 1], [0, 0, 1, 1, 0, 1]]

    adj, _ = add_self_loops(adj)
    assert adj._indices().tolist() == [[0, 1, 1, 2, 3], [0, 0, 1, 1, 1]]


def test_add_remaining_self_loops():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_weight = torch.tensor([0.5, 0.5, 0.5])
    edge_attr = torch.eye(3)

    expected = [[0, 1, 0, 1], [1, 0, 0, 1]]

    out = add_remaining_self_loops(edge_index, edge_weight)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 1]

    out = add_remaining_self_loops(edge_index, edge_weight, fill_value=5)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 5.0]

    out = add_remaining_self_loops(edge_index, edge_weight,
                                   fill_value=torch.tensor(2.))
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 2.0]

    out = add_remaining_self_loops(edge_index, edge_weight, fill_value='add')
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 0.5]

    # Test with `edge_attr`:
    out = add_remaining_self_loops(edge_index, edge_attr,
                                   fill_value=torch.tensor([0., 1., 0.]))
    assert out[0].tolist() == expected
    assert out[1].tolist() == [[1., 0., 0.], [0., 1., 0.], [0., 0., 1.],
                               [0., 1., 0.]]


def test_add_remaining_self_loops_without_initial_loops():
    edge_index = torch.tensor([[0, 1], [1, 0]])
    edge_weight = torch.tensor([0.5, 0.5])

    expected = [[0, 1, 0, 1], [1, 0, 0, 1]]

    out = add_remaining_self_loops(edge_index, edge_weight)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 1, 1]

    out = add_remaining_self_loops(edge_index, edge_weight, fill_value=5)
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 5.0, 5.0]

    out = add_remaining_self_loops(edge_index, edge_weight,
                                   fill_value=torch.tensor(2.0))
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 2.0, 2.0]

    # Test string `fill_value`:
    out = add_remaining_self_loops(edge_index, edge_weight, fill_value='add')
    assert out[0].tolist() == expected
    assert out[1].tolist() == [0.5, 0.5, 0.5, 0.5]


def test_get_self_loop_attr():
    edge_index = torch.tensor([[0, 1, 0], [1, 0, 0]])
    edge_weight = torch.tensor([0.2, 0.3, 0.5])

    full_loop_weight = get_self_loop_attr(edge_index, edge_weight)
    assert full_loop_weight.tolist() == [0.5, 0.0]

    full_loop_weight = get_self_loop_attr(edge_index, edge_weight, num_nodes=4)
    assert full_loop_weight.tolist() == [0.5, 0.0, 0.0, 0.0]

    full_loop_weight = get_self_loop_attr(edge_index)
    assert full_loop_weight.tolist() == [1.0, 0.0]

    edge_attr = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.5, 1.0]])
    full_loop_attr = get_self_loop_attr(edge_index, edge_attr)
    assert full_loop_attr.tolist() == [[0.5, 1.0], [0.0, 0.0]]
pytorch/test/utils/test_map.py000066400000000000000000000043711444262304300170620ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.profile import benchmark
from torch_geometric.testing import withCUDA, withPackage
from torch_geometric.utils.map import map_index


@withCUDA
@withPackage('pandas')
@pytest.mark.parametrize('max_index', [3, 100_000_000])
def test_map_index(device, max_index):
    src = torch.tensor([2, 0, 1, 0, max_index], device=device)
    index = torch.tensor([max_index, 2, 0, 1], device=device)

    out, mask = map_index(src, index, inclusive=True)
    assert out.device == device
    assert mask is None
    assert out.tolist() == [1, 2, 3, 2, 0]


@withCUDA
@withPackage('pandas')
@pytest.mark.parametrize('max_index', [3, 100_000_000])
def test_map_index_na(device, max_index):
    src = torch.tensor([2, 0, 1, 0, max_index], device=device)
    index = torch.tensor([max_index, 2, 0], device=device)

    out, mask = map_index(src, index, inclusive=False)
    assert out.device == device
    assert mask.device == device
    assert out.tolist() == [1, 2, 2, 0]
    assert mask.tolist() == [True, True, False, True, True]


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    args = parser.parse_args()

    src = torch.randint(0, 100_000_000, (100_000, ), device=args.device)
    index = src.unique()

    def trivial_map(src, index, max_index, inclusive):
        if max_index is None:
            max_index = max(src.max(), index.max())

        if inclusive:
            assoc = src.new_empty(max_index + 1)
        else:
            assoc = src.new_full((max_index + 1, ), -1)
        assoc[index] = torch.arange(index.numel(), device=index.device)
        out = assoc[src]

        if inclusive:
            return out, None
        else:
            mask = out != -1
            return out[mask], mask

    print('Inclusive:')
    benchmark(
        funcs=[trivial_map, map_index],
        func_names=['trivial', 'map_index'],
        args=(src, index, None, True),
        num_steps=100,
        num_warmups=50,
    )

    print('Exclusive:')
    benchmark(
        funcs=[trivial_map, map_index],
        func_names=['trivial', 'map_index'],
        args=(src, index[:50_000], None, False),
        num_steps=100,
        num_warmups=50,
    )
pytorch/test/utils/test_mask.py000066400000000000000000000015461444262304300172410ustar00rootroot00000000000000import torch

from torch_geometric.utils import index_to_mask, mask_select, mask_to_index


def test_mask_select():
    src = torch.randn(6, 8)
    mask = torch.tensor([False, True, False, True, False, True])

    out = mask_select(src, 0, mask)
    assert out.size() == (3, 8)
    assert torch.equal(src[torch.tensor([1, 3, 5])], out)

    jit = torch.jit.script(mask_select)
    assert torch.equal(jit(src, 0, mask), out)


def test_index_to_mask():
    index = torch.tensor([1, 3, 5])

    mask = index_to_mask(index)
    assert mask.tolist() == [False, True, False, True, False, True]

    mask = index_to_mask(index, size=7)
    assert mask.tolist() == [False, True, False, True, False, True, False]


def test_mask_to_index():
    mask = torch.tensor([False, True, False, True, False, True])

    index = mask_to_index(mask)
    assert index.tolist() == [1, 3, 5]
pytorch/test/utils/test_negative_sampling.py000066400000000000000000000150231444262304300217750ustar00rootroot00000000000000import torch

from torch_geometric.utils import (
    batched_negative_sampling,
    contains_self_loops,
    is_undirected,
    negative_sampling,
    structured_negative_sampling,
    structured_negative_sampling_feasible,
    to_undirected,
)
from torch_geometric.utils.negative_sampling import (
    edge_index_to_vector,
    vector_to_edge_index,
)


def is_negative(edge_index, neg_edge_index, size, bipartite):
    adj = torch.zeros(size, dtype=torch.bool)
    neg_adj = torch.zeros(size, dtype=torch.bool)

    adj[edge_index[0], edge_index[1]] = True
    neg_adj[neg_edge_index[0], neg_edge_index[1]] = True

    if not bipartite:
        arange = torch.arange(size[0])
        assert neg_adj[arange, arange].sum() == 0

    return (adj & neg_adj).sum() == 0


def test_edge_index_to_vector_and_vice_versa():
    # Create a fully-connected graph:
    N = 10
    row = torch.arange(N).view(-1, 1).repeat(1, N).view(-1)
    col = torch.arange(N).view(1, -1).repeat(N, 1).view(-1)
    edge_index = torch.stack([row, col], dim=0)

    idx, population = edge_index_to_vector(edge_index, (N, N), bipartite=True)
    assert population == N * N
    assert idx.tolist() == list(range(population))
    edge_index2 = vector_to_edge_index(idx, (N, N), bipartite=True)
    assert is_undirected(edge_index2)
    assert edge_index.tolist() == edge_index2.tolist()

    idx, population = edge_index_to_vector(edge_index, (N, N), bipartite=False)
    assert population == N * N - N
    assert idx.tolist() == list(range(population))
    mask = edge_index[0] != edge_index[1]  # Remove self-loops.
    edge_index2 = vector_to_edge_index(idx, (N, N), bipartite=False)
    assert is_undirected(edge_index2)
    assert edge_index[:, mask].tolist() == edge_index2.tolist()

    idx, population = edge_index_to_vector(edge_index, (N, N), bipartite=False,
                                           force_undirected=True)
    assert population == (N * (N + 1)) / 2 - N
    assert idx.tolist() == list(range(population))
    mask = edge_index[0] != edge_index[1]  # Remove self-loops.
    edge_index2 = vector_to_edge_index(idx, (N, N), bipartite=False,
                                       force_undirected=True)
    assert is_undirected(edge_index2)
    assert edge_index[:, mask].tolist() == to_undirected(edge_index2).tolist()


def test_negative_sampling():
    edge_index = torch.as_tensor([[0, 0, 1, 2], [0, 1, 2, 3]])

    neg_edge_index = negative_sampling(edge_index)
    assert neg_edge_index.size(1) == edge_index.size(1)
    assert is_negative(edge_index, neg_edge_index, (4, 4), bipartite=False)

    neg_edge_index = negative_sampling(edge_index, method='dense')
    assert neg_edge_index.size(1) == edge_index.size(1)
    assert is_negative(edge_index, neg_edge_index, (4, 4), bipartite=False)

    neg_edge_index = negative_sampling(edge_index, num_neg_samples=2)
    assert neg_edge_index.size(1) == 2
    assert is_negative(edge_index, neg_edge_index, (4, 4), bipartite=False)

    edge_index = to_undirected(edge_index)
    neg_edge_index = negative_sampling(edge_index, force_undirected=True)
    assert neg_edge_index.size(1) == edge_index.size(1) - 1
    assert is_undirected(neg_edge_index)
    assert is_negative(edge_index, neg_edge_index, (4, 4), bipartite=False)


def test_bipartite_negative_sampling():
    edge_index = torch.as_tensor([[0, 0, 1, 2], [0, 1, 2, 3]])

    neg_edge_index = negative_sampling(edge_index, num_nodes=(3, 4))
    assert neg_edge_index.size(1) == edge_index.size(1)
    assert is_negative(edge_index, neg_edge_index, (3, 4), bipartite=True)

    neg_edge_index = negative_sampling(edge_index, num_nodes=(3, 4),
                                       num_neg_samples=2)
    assert neg_edge_index.size(1) == 2
    assert is_negative(edge_index, neg_edge_index, (3, 4), bipartite=True)


def test_batched_negative_sampling():
    edge_index = torch.as_tensor([[0, 0, 1, 2], [0, 1, 2, 3]])
    edge_index = torch.cat([edge_index, edge_index + 4], dim=1)
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])

    neg_edge_index = batched_negative_sampling(edge_index, batch)
    assert neg_edge_index.size(1) <= edge_index.size(1)

    adj = torch.zeros(8, 8, dtype=torch.bool)
    adj[edge_index[0], edge_index[1]] = True
    neg_adj = torch.zeros(8, 8, dtype=torch.bool)
    neg_adj[neg_edge_index[0], neg_edge_index[1]] = True

    assert (adj & neg_adj).sum() == 0
    assert (adj | neg_adj).sum() == edge_index.size(1) + neg_edge_index.size(1)
    assert neg_adj[:4, 4:].sum() == 0
    assert neg_adj[4:, :4].sum() == 0


def test_bipartite_batched_negative_sampling():
    edge_index1 = torch.as_tensor([[0, 0, 1, 1], [0, 1, 2, 3]])
    edge_index2 = edge_index1 + torch.tensor([[2], [4]])
    edge_index3 = edge_index2 + torch.tensor([[2], [4]])
    edge_index = torch.cat([edge_index1, edge_index2, edge_index3], dim=1)
    src_batch = torch.tensor([0, 0, 1, 1, 2, 2])
    dst_batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2])

    neg_edge_index = batched_negative_sampling(edge_index,
                                               (src_batch, dst_batch))
    assert neg_edge_index.size(1) <= edge_index.size(1)

    adj = torch.zeros(6, 12, dtype=torch.bool)
    adj[edge_index[0], edge_index[1]] = True
    neg_adj = torch.zeros(6, 12, dtype=torch.bool)
    neg_adj[neg_edge_index[0], neg_edge_index[1]] = True

    assert (adj & neg_adj).sum() == 0
    assert (adj | neg_adj).sum() == edge_index.size(1) + neg_edge_index.size(1)


def test_structured_negative_sampling():
    edge_index = torch.as_tensor([[0, 0, 1, 2], [0, 1, 2, 3]])

    i, j, k = structured_negative_sampling(edge_index)
    assert i.size(0) == edge_index.size(1)
    assert j.size(0) == edge_index.size(1)
    assert k.size(0) == edge_index.size(1)

    adj = torch.zeros(4, 4, dtype=torch.bool)
    adj[i, j] = 1

    neg_adj = torch.zeros(4, 4, dtype=torch.bool)
    neg_adj[i, k] = 1
    assert (adj & neg_adj).sum() == 0

    # Test with no self-loops:
    edge_index = torch.LongTensor([[0, 0, 1, 1, 2], [1, 2, 0, 2, 1]])
    i, j, k = structured_negative_sampling(edge_index, num_nodes=4,
                                           contains_neg_self_loops=False)
    neg_edge_index = torch.vstack([i, k])
    assert not contains_self_loops(neg_edge_index)


def test_structured_negative_sampling_feasible():
    edge_index = torch.LongTensor([[0, 0, 1, 1, 2, 2, 2],
                                   [1, 2, 0, 2, 0, 1, 1]])
    assert not structured_negative_sampling_feasible(edge_index, 3, False)
    assert structured_negative_sampling_feasible(edge_index, 3, True)
    assert structured_negative_sampling_feasible(edge_index, 4, False)
pytorch/test/utils/test_nested.py000066400000000000000000000035521444262304300175670ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.utils import from_nested_tensor, to_nested_tensor


def test_to_nested_tensor():
    x = torch.randn(5, 4, 3)

    out = to_nested_tensor(x, batch=torch.tensor([0, 0, 1, 1, 1]))
    out = out.to_padded_tensor(padding=0)
    assert out.size() == (2, 3, 4, 3)
    assert torch.allclose(out[0, :2], x[0:2])
    assert torch.allclose(out[1, :3], x[2:5])

    out = to_nested_tensor(x, ptr=torch.tensor([0, 2, 5]))
    out = out.to_padded_tensor(padding=0)
    assert out.size() == (2, 3, 4, 3)
    assert torch.allclose(out[0, :2], x[0:2])
    assert torch.allclose(out[1, :3], x[2:5])

    out = to_nested_tensor(x)
    out = out.to_padded_tensor(padding=0)
    assert out.size() == (1, 5, 4, 3)
    assert torch.allclose(out[0], x)


def test_from_nested_tensor():
    x = torch.randn(5, 4, 3)

    nested = to_nested_tensor(x, batch=torch.tensor([0, 0, 1, 1, 1]))
    out, batch = from_nested_tensor(nested, return_batch=True)

    assert torch.equal(x, out)
    assert batch.tolist() == [0, 0, 1, 1, 1]

    nested = torch.nested.nested_tensor([torch.randn(4, 3), torch.randn(5, 4)])
    with pytest.raises(ValueError, match="have the same size in dimension 1"):
        from_nested_tensor(nested)

    # Test zero-copy:
    nested = to_nested_tensor(x, batch=torch.tensor([0, 0, 1, 1, 1]))
    out = from_nested_tensor(nested)
    out += 1  # Increment in-place (which should increment `nested` as well).
    assert torch.equal(nested.to_padded_tensor(padding=0)[0, :2], out[0:2])
    assert torch.equal(nested.to_padded_tensor(padding=0)[1, :3], out[2:5])


def test_to_and_from_nested_tensor_autograd():
    x = torch.randn(5, 4, 3, requires_grad=True)
    grad = torch.randn_like(x)

    out = to_nested_tensor(x, batch=torch.tensor([0, 0, 1, 1, 1]))
    out = from_nested_tensor(out)
    out.backward(grad)
    assert torch.equal(x.grad, grad)
pytorch/test/utils/test_normalized_cut.py000066400000000000000000000012401444262304300213140ustar00rootroot00000000000000import torch

from torch_geometric.testing import is_full_test
from torch_geometric.utils import normalized_cut


def test_normalized_cut():
    row = torch.tensor([0, 1, 1, 1, 2, 2, 3, 3, 4, 4])
    col = torch.tensor([1, 0, 2, 3, 1, 4, 1, 4, 2, 3])
    edge_attr = torch.Tensor([3, 3, 6, 3, 6, 1, 3, 2, 1, 2])
    expected_output = [4, 4, 5, 2.5, 5, 1, 2.5, 2, 1, 2]

    output = normalized_cut(torch.stack([row, col], dim=0), edge_attr)
    assert output.tolist() == expected_output

    if is_full_test():
        jit = torch.jit.script(normalized_cut)
        output = jit(torch.stack([row, col], dim=0), edge_attr)
        assert output.tolist() == expected_output
pytorch/test/utils/test_num_nodes.py000066400000000000000000000015671444262304300203000ustar00rootroot00000000000000import torch

from torch_geometric.utils import to_torch_coo_tensor
from torch_geometric.utils.num_nodes import (
    maybe_num_nodes,
    maybe_num_nodes_dict,
)


def test_maybe_num_nodes():
    edge_index = torch.tensor([[0, 0, 1, 1, 2, 2, 2], [1, 2, 0, 2, 0, 1, 1]])

    assert maybe_num_nodes(edge_index, 4) == 4
    assert maybe_num_nodes(edge_index) == 3

    adj = to_torch_coo_tensor(edge_index)
    assert maybe_num_nodes(adj, 4) == 4
    assert maybe_num_nodes(adj) == 3


def test_maybe_num_nodes_dict():
    edge_index_dict = {
        '1': torch.tensor([[0, 0, 1, 1, 2, 2, 2], [1, 2, 0, 2, 0, 1, 1]]),
        '2': torch.tensor([[0, 0, 1, 3], [1, 2, 0, 4]])
    }
    num_nodes_dict = {'2': 6}

    assert maybe_num_nodes_dict(edge_index_dict) == {'1': 3, '2': 5}
    assert maybe_num_nodes_dict(edge_index_dict, num_nodes_dict) == {
        '1': 3,
        '2': 6,
    }
pytorch/test/utils/test_one_hot.py000066400000000000000000000007211444262304300177330ustar00rootroot00000000000000import torch

from torch_geometric.utils import one_hot


def test_one_hot():
    index = torch.tensor([0, 1, 2])

    out = one_hot(index)
    assert out.size() == (3, 3)
    assert out.dtype == torch.float
    assert out.tolist() == [[1, 0, 0], [0, 1, 0], [0, 0, 1]]

    out = one_hot(index, num_classes=4, dtype=torch.long)
    assert out.size() == (3, 4)
    assert out.dtype == torch.long
    assert out.tolist() == [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]]
pytorch/test/utils/test_random.py000066400000000000000000000026411444262304300175630ustar00rootroot00000000000000import numpy as np
import torch

from torch_geometric.utils import (
    barabasi_albert_graph,
    erdos_renyi_graph,
    stochastic_blockmodel_graph,
)


def test_erdos_renyi_graph():
    torch.manual_seed(1234)
    edge_index = erdos_renyi_graph(5, 0.2, directed=False)
    assert edge_index.tolist() == [
        [0, 1, 1, 1, 2, 4],
        [1, 0, 2, 4, 1, 1],
    ]

    edge_index = erdos_renyi_graph(5, 0.5, directed=True)
    assert edge_index.tolist() == [
        [1, 1, 2, 2, 3, 4, 4, 4],
        [0, 3, 0, 4, 0, 0, 1, 3],
    ]


def test_stochastic_blockmodel_graph():
    torch.manual_seed(12345)

    block_sizes = [2, 2, 4]
    edge_probs = [
        [0.25, 0.05, 0.02],
        [0.05, 0.35, 0.07],
        [0.02, 0.07, 0.40],
    ]

    edge_index = stochastic_blockmodel_graph(block_sizes, edge_probs,
                                             directed=False)
    assert edge_index.tolist() == [
        [2, 3, 4, 4, 5, 5, 6, 7, 7, 7],
        [3, 2, 5, 7, 4, 7, 7, 4, 5, 6],
    ]

    edge_index = stochastic_blockmodel_graph(block_sizes, edge_probs,
                                             directed=True)
    assert edge_index.tolist() == [
        [0, 1, 3, 5, 6, 6, 7, 7],
        [3, 3, 2, 4, 4, 7, 5, 6],
    ]


def test_barabasi_albert_graph():
    torch.manual_seed(12345)
    np.random.seed(12345)

    edge_index = barabasi_albert_graph(num_nodes=8, num_edges=3)
    assert edge_index.size() == (2, 26)
pytorch/test/utils/test_repeat.py000066400000000000000000000005141444262304300175600ustar00rootroot00000000000000from torch_geometric.utils.repeat import repeat


def test_repeat():
    assert repeat(None, length=4) is None
    assert repeat(4, length=4) == [4, 4, 4, 4]
    assert repeat([2, 3, 4], length=4) == [2, 3, 4, 4]
    assert repeat([1, 2, 3, 4], length=4) == [1, 2, 3, 4]
    assert repeat([1, 2, 3, 4, 5], length=4) == [1, 2, 3, 4]
pytorch/test/utils/test_scatter.py000066400000000000000000000116511444262304300177510ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.profile import benchmark
from torch_geometric.testing import disableExtensions, withCUDA, withPackage
from torch_geometric.utils import scatter
from torch_geometric.utils.scatter import scatter_argmax


def test_scatter_validate():
    src = torch.randn(100, 32)
    index = torch.randint(0, 10, (100, ), dtype=torch.long)

    with pytest.raises(ValueError, match="must be one-dimensional"):
        scatter(src, index.view(-1, 1))

    with pytest.raises(ValueError, match="must lay between 0 and 1"):
        scatter(src, index, dim=2)

    with pytest.raises(ValueError, match="invalid `reduce` argument 'std'"):
        scatter(src, index, reduce='std')


@withCUDA
@withPackage('torch_scatter')
@pytest.mark.parametrize('reduce', ['sum', 'add', 'mean', 'min', 'max'])
def test_scatter(reduce, device):
    import torch_scatter

    src = torch.randn(100, 16, device=device)
    index = torch.randint(0, 8, (100, ), device=device)

    out1 = scatter(src, index, dim=0, reduce=reduce)
    out2 = torch_scatter.scatter(src, index, dim=0, reduce=reduce)
    assert out1.device == device
    assert torch.allclose(out1, out2, atol=1e-6)

    jit = torch.jit.script(scatter)
    out3 = jit(src, index, dim=0, reduce=reduce)
    assert torch.allclose(out1, out3, atol=1e-6)

    src = torch.randn(8, 100, 16, device=device)
    out1 = scatter(src, index, dim=1, reduce=reduce)
    out2 = torch_scatter.scatter(src, index, dim=1, reduce=reduce)
    assert out1.device == device
    assert torch.allclose(out1, out2, atol=1e-6)


@withCUDA
@pytest.mark.parametrize('reduce', ['sum', 'add', 'mean', 'min', 'max'])
def test_scatter_backward(reduce, device):
    src = torch.randn(8, 100, 16, device=device, requires_grad=True)
    index = torch.randint(0, 8, (100, ), device=device)

    out = scatter(src, index, dim=1, reduce=reduce)

    assert src.grad is None
    out.mean().backward()
    assert src.grad is not None


@withCUDA
def test_scatter_any(device):
    src = torch.randn(6, 4, device=device)
    index = torch.tensor([0, 0, 1, 1, 2, 2], device=device)

    out = scatter(src, index, dim=0, reduce='any')

    for i in range(3):
        for j in range(4):
            assert float(out[i, j]) in src[2 * i:2 * i + 2, j].tolist()


@withCUDA
@disableExtensions
def test_scatter_argmax(device):
    src = torch.arange(5, device=device)
    index = torch.tensor([2, 2, 0, 0, 3], device=device)

    argmax = scatter_argmax(src, index, dim_size=6)
    assert argmax.tolist() == [3, 5, 1, 4, 5, 5]


if __name__ == '__main__':
    # Insights on GPU:
    # ================
    # * "sum": Prefer `scatter_add_` implementation
    # * "mean": Prefer manual implementation via `scatter_add_` + `count`
    # * "min"/"max":
    #   * Prefer `scatter_reduce_` implementation without gradients
    #   * Prefer `torch_sparse` implementation with gradients
    # * "mul": Prefer `torch_sparse` implementation
    #
    # Insights on CPU:
    # ================
    # * "sum": Prefer `scatter_add_` implementation
    # * "mean": Prefer manual implementation via `scatter_add_` + `count`
    # * "min"/"max": Prefer `scatter_reduce_` implementation
    # * "mul" (probably not worth branching for this):
    #   * Prefer `scatter_reduce_` implementation without gradients
    #   * Prefer `torch_sparse` implementation with gradients
    import argparse

    import torch_scatter

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 1_000, 50_000
    x = torch.randn(num_edges, 64, device=args.device)
    index = torch.randint(num_nodes, (num_edges, ), device=args.device)

    def pytorch_scatter(x, index, dim_size, reduce):
        if reduce == 'min' or reduce == 'max':
            reduce = f'a{aggr}'  # `amin` or `amax`
        elif reduce == 'mul':
            reduce = 'prod'
        out = x.new_zeros((dim_size, x.size(-1)))
        include_self = reduce in ['sum', 'mean']
        index = index.view(-1, 1).expand(-1, x.size(-1))
        out.scatter_reduce_(0, index, x, reduce, include_self=include_self)
        return out

    def own_scatter(x, index, dim_size, reduce):
        return torch_scatter.scatter(x, index, dim=0, dim_size=num_nodes,
                                     reduce=reduce)

    def optimized_scatter(x, index, dim_size, reduce):
        return scatter(x, index, dim=0, dim_size=dim_size, reduce=reduce)

    aggrs = ['sum', 'mean', 'min', 'max', 'mul']
    for aggr in aggrs:
        print(f'Aggregator: {aggr}')
        benchmark(
            funcs=[pytorch_scatter, own_scatter, optimized_scatter],
            func_names=['PyTorch', 'torch_scatter', 'Optimized'],
            args=(x, index, num_nodes, aggr),
            num_steps=100 if args.device == 'cpu' else 1000,
            num_warmups=50 if args.device == 'cpu' else 500,
            backward=args.backward,
        )
pytorch/test/utils/test_segment.py000066400000000000000000000011251444262304300177410ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.testing import withCUDA, withPackage
from torch_geometric.utils import segment


@withCUDA
@withPackage('torch_scatter')
@pytest.mark.parametrize('reduce', ['sum', 'mean', 'min', 'max'])
def test_segment(device, reduce):
    src = torch.randn(20, 16, device=device)
    ptr = torch.tensor([0, 5, 10, 15, 20], device=device)

    out = segment(src, ptr, reduce=reduce)

    expected = getattr(torch, reduce)(src.view(4, 5, -1), dim=1)
    expected = expected[0] if isinstance(expected, tuple) else expected

    assert torch.allclose(out, expected)
pytorch/test/utils/test_select.py000066400000000000000000000012751444262304300175640ustar00rootroot00000000000000import torch

from torch_geometric.utils import narrow, select


def test_select():
    src = torch.randn(5, 3)
    index = torch.tensor([0, 2, 4])
    mask = torch.tensor([True, False, True, False, True])

    out = select(src, index, dim=0)
    assert torch.equal(out, src[index])
    assert torch.equal(out, select(src, mask, dim=0))
    assert torch.equal(out, torch.tensor(select(src.tolist(), index, dim=0)))
    assert torch.equal(out, torch.tensor(select(src.tolist(), mask, dim=0)))


def test_narrow():
    src = torch.randn(5, 3)

    out = narrow(src, dim=0, start=2, length=2)
    assert torch.equal(out, src[2:4])
    assert torch.equal(out, torch.tensor(narrow(src.tolist(), 0, 2, 2)))
pytorch/test/utils/test_smiles.py000066400000000000000000000012631444262304300175760ustar00rootroot00000000000000import pytest

from torch_geometric.testing import withPackage
from torch_geometric.utils import from_smiles, to_smiles


@withPackage('rdkit')
@pytest.mark.parametrize('smiles', [
    r'F/C=C/F',
    r'F/C=C\F',
    r'F/C=C\F',
    (r'COc1cccc([C@@H]2Oc3ccc(OC)cc3/C(=N/OC[C@@H](C)[C@H](OCc3ccccc3)'
     r'C(C)C)[C@@H]2O)c1'),
    r'C/C(=C\C(=O)c1ccc(C)o1)Nc1ccc2c(c1)OCO2',
    r'F[B-](F)(F)c1cnc2ccccc2c1',
    r'COC(=O)[C@@]1(Cc2ccccc2)[C@H]2C(=O)N(C)C(=O)[C@H]2[C@H]2CN=C(SC)N21',
    (r'O=C(O)c1ccc(NS(=O)(=O)c2ccc3c(c2)C(=O)c2cc(S(=O)(=O)Nc4ccc(C(=O)O)'
     r'cc4)ccc2-3)cc1'),
])
def test_from_to_smiles(smiles):
    data = from_smiles(smiles)
    assert to_smiles(data) == smiles
pytorch/test/utils/test_softmax.py000066400000000000000000000064061444262304300177670ustar00rootroot00000000000000import pytest
import torch

import torch_geometric.typing
from torch_geometric.profile import benchmark
from torch_geometric.utils import softmax


def test_softmax():
    src = torch.tensor([1., 1., 1., 1.])
    index = torch.tensor([0, 0, 1, 2])
    ptr = torch.tensor([0, 2, 3, 4])

    out = softmax(src, index)
    assert out.tolist() == [0.5, 0.5, 1, 1]
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert softmax(src, None, ptr).tolist() == out.tolist()
    else:
        with pytest.raises(ImportError):
            softmax(src, None, ptr)

    src = src.view(-1, 1)
    out = softmax(src, index)
    assert out.tolist() == [[0.5], [0.5], [1], [1]]
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert softmax(src, None, ptr).tolist() == out.tolist()

    jit = torch.jit.script(softmax)
    assert torch.allclose(jit(src, index), out)


def test_softmax_backward():
    src_sparse = torch.rand(4, 8)
    index = torch.tensor([0, 0, 1, 1])
    src_dense = src_sparse.clone().view(2, 2, src_sparse.size(-1))

    src_sparse.requires_grad_(True)
    src_dense.requires_grad_(True)

    out_sparse = softmax(src_sparse, index)
    out_sparse.mean().backward()
    out_dense = src_dense.softmax(dim=1)
    out_dense.mean().backward()

    assert torch.allclose(out_sparse, out_dense.view_as(out_sparse))
    assert torch.allclose(src_sparse.grad, src_dense.grad.view_as(src_sparse))


def test_softmax_dim():
    index = torch.tensor([0, 0, 0, 0])
    ptr = torch.tensor([0, 4])

    src = torch.randn(4)
    assert torch.allclose(softmax(src, index, dim=0), src.softmax(dim=0))
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert torch.allclose(softmax(src, ptr=ptr, dim=0), src.softmax(dim=0))

    src = torch.randn(4, 16)
    assert torch.allclose(softmax(src, index, dim=0), src.softmax(dim=0))
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert torch.allclose(softmax(src, ptr=ptr, dim=0), src.softmax(dim=0))

    src = torch.randn(4, 4)
    assert torch.allclose(softmax(src, index, dim=-1), src.softmax(dim=-1))
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert torch.allclose(softmax(src, ptr=ptr, dim=-1), src.softmax(-1))

    src = torch.randn(4, 4, 16)
    assert torch.allclose(softmax(src, index, dim=1), src.softmax(dim=1))
    if torch_geometric.typing.WITH_TORCH_SCATTER:
        assert torch.allclose(softmax(src, ptr=ptr, dim=1), src.softmax(dim=1))


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    x = torch.randn(num_edges, 64, device=args.device)
    index = torch.randint(num_nodes, (num_edges, ), device=args.device)

    compiled_softmax = torch_geometric.compile(softmax)

    def dense_softmax(x, index):
        x = x.view(num_nodes, -1, x.size(-1))
        return x.softmax(dim=-1)

    benchmark(
        funcs=[dense_softmax, softmax, compiled_softmax],
        func_names=['Dense Softmax', 'Vanilla', 'Compiled'],
        args=(x, index),
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
        backward=args.backward,
    )
pytorch/test/utils/test_sort_edge_index.py000066400000000000000000000040721444262304300214450ustar00rootroot00000000000000from typing import List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.utils import sort_edge_index


def test_sort_edge_index():
    edge_index = torch.tensor([[2, 1, 1, 0], [1, 2, 0, 1]])
    edge_attr = torch.tensor([[1], [2], [3], [4]])

    out = sort_edge_index(edge_index)
    assert out.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]

    out = sort_edge_index(edge_index, None)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1] is None

    out = sort_edge_index(edge_index, edge_attr)
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1].tolist() == [[4], [3], [2], [1]]

    out = sort_edge_index(edge_index, [edge_attr, edge_attr.view(-1)])
    assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
    assert out[1][0].tolist() == [[4], [3], [2], [1]]
    assert out[1][1].tolist() == [4, 3, 2, 1]


def test_sort_edge_index_jit():
    @torch.jit.script
    def wrapper1(edge_index: Tensor) -> Tensor:
        return sort_edge_index(edge_index)

    @torch.jit.script
    def wrapper2(
        edge_index: Tensor,
        edge_attr: Optional[Tensor],
    ) -> Tuple[Tensor, Optional[Tensor]]:
        return sort_edge_index(edge_index, edge_attr)

    @torch.jit.script
    def wrapper3(
        edge_index: Tensor,
        edge_attr: List[Tensor],
    ) -> Tuple[Tensor, List[Tensor]]:
        return sort_edge_index(edge_index, edge_attr)

    edge_index = torch.tensor([[2, 1, 1, 0], [1, 2, 0, 1]])
    edge_attr = torch.tensor([[1], [2], [3], [4]])

    out = wrapper1(edge_index)
    assert out.size() == edge_index.size()

    out = wrapper2(edge_index, None)
    assert out[0].size() == edge_index.size()
    assert out[1] is None

    out = wrapper2(edge_index, edge_attr)
    assert out[0].size() == edge_index.size()
    assert out[1].size() == edge_attr.size()

    out = wrapper3(edge_index, [edge_attr, edge_attr.view(-1)])
    assert out[0].size() == edge_index.size()
    assert len(out[1]) == 2
    assert out[1][0].size() == edge_attr.size()
    assert out[1][1].size() == edge_attr.view(-1).size()
pytorch/test/utils/test_sparse.py000066400000000000000000000167431444262304300176100ustar00rootroot00000000000000import os.path as osp

import torch

import torch_geometric.typing
from torch_geometric.profile import benchmark
from torch_geometric.testing import is_full_test, withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import (
    dense_to_sparse,
    is_sparse,
    is_torch_sparse_tensor,
    to_edge_index,
    to_torch_coo_tensor,
    to_torch_csc_tensor,
    to_torch_csr_tensor,
)


def test_dense_to_sparse():
    adj = torch.Tensor([
        [3, 1],
        [2, 0],
    ])
    edge_index, edge_attr = dense_to_sparse(adj)
    assert edge_index.tolist() == [[0, 0, 1], [0, 1, 0]]
    assert edge_attr.tolist() == [3, 1, 2]

    if is_full_test():
        jit = torch.jit.script(dense_to_sparse)
        edge_index, edge_attr = jit(adj)
        assert edge_index.tolist() == [[0, 0, 1], [0, 1, 0]]
        assert edge_attr.tolist() == [3, 1, 2]

    adj = torch.Tensor([[
        [3, 1],
        [2, 0],
    ], [
        [0, 1],
        [0, 2],
    ]])
    edge_index, edge_attr = dense_to_sparse(adj)
    assert edge_index.tolist() == [[0, 0, 1, 2, 3], [0, 1, 0, 3, 3]]
    assert edge_attr.tolist() == [3, 1, 2, 1, 2]

    if is_full_test():
        jit = torch.jit.script(dense_to_sparse)
        edge_index, edge_attr = jit(adj)
        assert edge_index.tolist() == [[0, 0, 1, 2, 3], [0, 1, 0, 3, 3]]
        assert edge_attr.tolist() == [3, 1, 2, 1, 2]


def test_dense_to_sparse_bipartite():
    edge_index, edge_attr = dense_to_sparse(torch.rand(2, 10, 5))
    assert edge_index[0].max() == 19
    assert edge_index[1].max() == 9


def test_is_torch_sparse_tensor():
    x = torch.randn(5, 5)

    assert not is_torch_sparse_tensor(x)
    assert is_torch_sparse_tensor(x.to_sparse())

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert not is_torch_sparse_tensor(SparseTensor.from_dense(x))


def test_is_sparse():
    x = torch.randn(5, 5)

    assert not is_sparse(x)
    assert is_sparse(x.to_sparse())

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        assert is_sparse(SparseTensor.from_dense(x))


def test_to_torch_coo_tensor():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])
    edge_attr = torch.randn(edge_index.size(1), 8)

    adj = to_torch_coo_tensor(edge_index, is_coalesced=False)
    assert adj.is_coalesced()
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_coo
    assert torch.allclose(adj.indices(), edge_index)

    adj = to_torch_coo_tensor(edge_index, is_coalesced=True)
    assert adj.is_coalesced()
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_coo
    assert torch.allclose(adj.indices(), edge_index)

    adj = to_torch_coo_tensor(edge_index, size=6)
    assert adj.size() == (6, 6)
    assert adj.layout == torch.sparse_coo
    assert torch.allclose(adj.indices(), edge_index)

    adj = to_torch_coo_tensor(edge_index, edge_attr)
    assert adj.size() == (4, 4, 8)
    assert adj.layout == torch.sparse_coo
    assert torch.allclose(adj.indices(), edge_index)
    assert torch.allclose(adj.values(), edge_attr)

    if is_full_test():
        jit = torch.jit.script(to_torch_coo_tensor)
        adj = jit(edge_index, edge_attr)
        assert adj.size() == (4, 4, 8)
        assert adj.layout == torch.sparse_coo
        assert torch.allclose(adj.indices(), edge_index)
        assert torch.allclose(adj.values(), edge_attr)


def test_to_torch_csr_tensor():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])

    adj = to_torch_csr_tensor(edge_index)
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_csr
    assert torch.allclose(adj.to_sparse_coo().indices(), edge_index)

    edge_weight = torch.randn(edge_index.size(1))
    adj = to_torch_csr_tensor(edge_index, edge_weight)
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_csr
    assert torch.allclose(adj.to_sparse_coo().indices(), edge_index)
    assert torch.allclose(adj.to_sparse_coo().values(), edge_weight)

    if torch_geometric.typing.WITH_PT2:
        edge_attr = torch.randn(edge_index.size(1), 8)
        adj = to_torch_csr_tensor(edge_index, edge_attr)
        assert adj.size() == (4, 4, 8)
        assert adj.layout == torch.sparse_csr
        assert torch.allclose(adj.to_sparse_coo().indices(), edge_index)
        assert torch.allclose(adj.to_sparse_coo().values(), edge_attr)


def test_to_torch_csc_tensor():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])

    adj = to_torch_csc_tensor(edge_index)
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_csc
    adj_coo = adj.to_sparse_coo().coalesce()
    if torch_geometric.typing.WITH_PT2:
        assert torch.allclose(adj_coo.indices(), edge_index)
    else:
        assert torch.allclose(adj_coo.indices().flip([0]), edge_index)

    edge_weight = torch.randn(edge_index.size(1))
    adj = to_torch_csc_tensor(edge_index, edge_weight)
    assert adj.size() == (4, 4)
    assert adj.layout == torch.sparse_csc
    adj_coo = adj.to_sparse_coo().coalesce()
    if torch_geometric.typing.WITH_PT2:
        assert torch.allclose(adj_coo.indices(), edge_index)
        assert torch.allclose(adj_coo.values(), edge_weight)
    else:
        perm = adj_coo.indices()[0].argsort()
        assert torch.allclose(adj_coo.indices()[:, perm], edge_index)
        assert torch.allclose(adj_coo.values()[perm], edge_weight)

    if torch_geometric.typing.WITH_PT2:
        edge_attr = torch.randn(edge_index.size(1), 8)
        adj = to_torch_csc_tensor(edge_index, edge_attr)
        assert adj.size() == (4, 4, 8)
        assert adj.layout == torch.sparse_csc
        assert torch.allclose(adj.to_sparse_coo().coalesce().indices(),
                              edge_index)
        assert torch.allclose(adj.to_sparse_coo().coalesce().values(),
                              edge_attr)


@withPackage('torch>=2.1.0')
def test_to_torch_coo_tensor_save_load(tmp_path):
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3],
        [1, 0, 2, 1, 3, 2],
    ])
    adj = to_torch_coo_tensor(edge_index, is_coalesced=False)
    assert adj.is_coalesced()

    path = osp.join(tmp_path, 'adj.t')
    torch.save(adj, path)
    adj = torch.load(path)
    assert adj.is_coalesced()


def test_to_edge_index():
    adj = torch.tensor([
        [0., 1., 0., 0.],
        [1., 0., 1., 0.],
        [0., 1., 0., 1.],
        [0., 0., 1., 0.],
    ]).to_sparse()

    edge_index, edge_attr = to_edge_index(adj)
    assert edge_index.tolist() == [[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]]
    assert edge_attr.tolist() == [1., 1., 1., 1., 1., 1.]

    if is_full_test():
        jit = torch.jit.script(to_edge_index)
        edge_index, edge_attr = jit(adj)
        assert edge_index.tolist() == [[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]]
        assert edge_attr.tolist() == [1., 1., 1., 1., 1., 1.]


if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    edge_index = torch.randint(num_nodes, (2, num_edges), device=args.device)

    benchmark(
        funcs=[
            SparseTensor.from_edge_index, to_torch_coo_tensor,
            to_torch_csr_tensor, to_torch_csc_tensor
        ],
        func_names=['SparseTensor', 'To COO', 'To CSR', 'To CSC'],
        args=(edge_index, None, (num_nodes, num_nodes)),
        num_steps=50 if args.device == 'cpu' else 500,
        num_warmups=10 if args.device == 'cpu' else 100,
    )
pytorch/test/utils/test_spmm.py000066400000000000000000000116061444262304300172600ustar00rootroot00000000000000import itertools
import warnings

import pytest
import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.profile import benchmark
from torch_geometric.testing import withCUDA, withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import spmm, to_torch_coo_tensor


@withCUDA
@pytest.mark.parametrize('reduce', ['sum', 'mean'])
def test_spmm_basic(device, reduce):
    src = torch.randn(5, 4, device=device)
    other = torch.randn(4, 8, device=device)

    out1 = (src @ other) / (src.size(1) if reduce == 'mean' else 1)
    out2 = spmm(src.to_sparse_csr(), other, reduce=reduce)
    assert out1.size() == (5, 8)
    assert torch.allclose(out1, out2, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3 = spmm(SparseTensor.from_dense(src), other, reduce=reduce)
        assert torch.allclose(out2, out3, atol=1e-6)

    # Test `mean` reduction with isolated nodes:
    src[0] = 0.
    out1 = (src @ other) / (4. if reduce == 'mean' else 1.)
    out2 = spmm(src.to_sparse_csr(), other, reduce=reduce)
    assert out1.size() == (5, 8)
    assert torch.allclose(out1, out2, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3 = spmm(SparseTensor.from_dense(src), other, reduce=reduce)
        assert torch.allclose(out2, out3, atol=1e-6)


@withCUDA
@withPackage('torch>=2.0.0')
@pytest.mark.parametrize('reduce', ['min', 'max'])
def test_spmm_reduce(device, reduce):
    src = torch.randn(5, 4, device=device)
    other = torch.randn(4, 8, device=device)

    if src.is_cuda:
        with pytest.raises(NotImplementedError, match="not yet supported"):
            spmm(src.to_sparse_csr(), other, reduce)
    else:
        out1 = spmm(src.to_sparse_csr(), other, reduce)
        assert out1.size() == (5, 8)
        if torch_geometric.typing.WITH_TORCH_SPARSE:
            out2 = spmm(SparseTensor.from_dense(src), other, reduce=reduce)
            assert torch.allclose(out1, out2)


@withCUDA
@withPackage('torch>=2.0.0')
@pytest.mark.parametrize(
    'layout', [torch.sparse_coo, torch.sparse_csr, torch.sparse_csc])
@pytest.mark.parametrize('reduce', ['sum', 'mean', 'min', 'max'])
def test_spmm_layout(device, layout, reduce):
    src = torch.randn(5, 4, device=device)
    if layout == torch.sparse_coo:
        src = src.to_sparse_coo()
    elif layout == torch.sparse_csr:
        src = src.to_sparse_csr()
    else:
        assert layout == torch.sparse_csc
        src = src.to_sparse_csc()
    other = torch.randn(4, 8, device=device)

    if src.is_cuda and reduce in {'min', 'max'}:
        with pytest.raises(NotImplementedError, match="not yet supported"):
            spmm(src, other, reduce=reduce)
    elif layout != torch.sparse_csr:
        with pytest.warns(UserWarning, match="Converting sparse tensor"):
            spmm(src, other, reduce=reduce)
    else:
        spmm(src, other, reduce=reduce)


@pytest.mark.parametrize('reduce', ['sum', 'mean'])
def test_spmm_jit(reduce):
    @torch.jit.script
    def jit_torch_sparse(src: SparseTensor, other: Tensor,
                         reduce: str) -> Tensor:
        return spmm(src, other, reduce=reduce)

    @torch.jit.script
    def jit_torch(src: Tensor, other: Tensor, reduce: str) -> Tensor:
        return spmm(src, other, reduce=reduce)

    src = torch.randn(5, 4)
    other = torch.randn(4, 8)

    out1 = src @ other
    out2 = jit_torch(src.to_sparse_csr(), other, reduce)
    assert out1.size() == (5, 8)
    if reduce == 'sum':
        assert torch.allclose(out1, out2, atol=1e-6)
    if torch_geometric.typing.WITH_TORCH_SPARSE:
        out3 = jit_torch_sparse(SparseTensor.from_dense(src), other, reduce)
        assert torch.allclose(out2, out3, atol=1e-6)


if __name__ == '__main__':
    import argparse

    warnings.filterwarnings('ignore', ".*Sparse CSR tensor support.*")
    warnings.filterwarnings('ignore', ".*Converting sparse tensor to CSR.*")

    parser = argparse.ArgumentParser()
    parser.add_argument('--device', type=str, default='cuda')
    parser.add_argument('--backward', action='store_true')
    args = parser.parse_args()

    num_nodes, num_edges = 10_000, 200_000
    x = torch.randn(num_nodes, 64, device=args.device)
    edge_index = torch.randint(num_nodes, (2, num_edges), device=args.device)

    reductions = ['sum', 'mean']
    if not x.is_cuda:
        reductions.extend(['min', 'max'])
    layouts = [torch.sparse_coo, torch.sparse_csr, torch.sparse_csc]

    for reduce, layout in itertools.product(reductions, layouts):
        print(f'Aggregator: {reduce}, Layout: {layout}')

        adj = to_torch_coo_tensor(edge_index, size=num_nodes)
        adj = adj.to_sparse(layout=layout)

        benchmark(
            funcs=[spmm],
            func_names=['spmm'],
            args=(adj, x, reduce),
            num_steps=50 if args.device == 'cpu' else 500,
            num_warmups=10 if args.device == 'cpu' else 100,
            backward=args.backward,
        )
pytorch/test/utils/test_subgraph.py000066400000000000000000000066751444262304300201310ustar00rootroot00000000000000import torch

from torch_geometric.nn import GCNConv, Linear
from torch_geometric.utils import (
    bipartite_subgraph,
    get_num_hops,
    index_to_mask,
    k_hop_subgraph,
    subgraph,
)


def test_get_num_hops():
    class GNN(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = GCNConv(3, 16, normalize=False)
            self.conv2 = GCNConv(16, 16, normalize=False)
            self.lin = Linear(16, 2)

        def forward(self, x, edge_index):
            x = torch.F.relu(self.conv1(x, edge_index))
            x = self.conv2(x, edge_index)
            return self.lin(x)

    assert get_num_hops(GNN()) == 2


def test_subgraph():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6],
        [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5],
    ])
    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])

    idx = torch.tensor([3, 4, 5], dtype=torch.long)
    mask = index_to_mask(idx, 7)
    indices = idx.tolist()

    for subset in [idx, mask, indices]:
        out = subgraph(subset, edge_index, edge_attr, return_edge_mask=True)
        assert out[0].tolist() == [[3, 4, 4, 5], [4, 3, 5, 4]]
        assert out[1].tolist() == [7, 8, 9, 10]
        assert out[2].tolist() == [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0]

        out = subgraph(subset, edge_index, edge_attr, relabel_nodes=True)
        assert out[0].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
        assert out[1].tolist() == [7, 8, 9, 10]


def test_bipartite_subgraph():
    edge_index = torch.tensor([[0, 5, 2, 3, 3, 4, 4, 3, 5, 5, 6],
                               [0, 0, 3, 2, 0, 0, 2, 1, 2, 3, 1]])
    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
    idx = (torch.tensor([2, 3, 5], dtype=torch.long),
           torch.tensor([2, 3], dtype=torch.long))
    mask = (index_to_mask(idx[0], 7), index_to_mask(idx[1], 4))
    indices = (idx[0].tolist(), idx[1].tolist())
    mixed = (mask[0], idx[1])

    for subset in [idx, mask, indices, mixed]:
        out = bipartite_subgraph(subset, edge_index, edge_attr,
                                 return_edge_mask=True)
        assert out[0].tolist() == [[2, 3, 5, 5], [3, 2, 2, 3]]
        assert out[1].tolist() == [3, 4, 9, 10]
        assert out[2].tolist() == [0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0]

        out = bipartite_subgraph(subset, edge_index, edge_attr,
                                 relabel_nodes=True)
        assert out[0].tolist() == [[0, 1, 2, 2], [1, 0, 0, 1]]
        assert out[1].tolist() == [3, 4, 9, 10]


def test_k_hop_subgraph():
    edge_index = torch.tensor([
        [0, 1, 2, 3, 4, 5],
        [2, 2, 4, 4, 6, 6],
    ])

    subset, edge_index, mapping, edge_mask = k_hop_subgraph(
        6, 2, edge_index, relabel_nodes=True)
    assert subset.tolist() == [2, 3, 4, 5, 6]
    assert edge_index.tolist() == [[0, 1, 2, 3], [2, 2, 4, 4]]
    assert mapping.tolist() == [4]
    assert edge_mask.tolist() == [False, False, True, True, True, True]

    edge_index = torch.tensor([
        [1, 2, 4, 5],
        [0, 1, 5, 6],
    ])

    subset, edge_index, mapping, edge_mask = k_hop_subgraph([0, 6], 2,
                                                            edge_index,
                                                            relabel_nodes=True)

    assert subset.tolist() == [0, 1, 2, 4, 5, 6]
    assert edge_index.tolist() == [[1, 2, 3, 4], [0, 1, 4, 5]]
    assert mapping.tolist() == [0, 5]
    assert edge_mask.tolist() == [True, True, True, True]
pytorch/test/utils/test_to_dense_adj.py000066400000000000000000000070361444262304300207240ustar00rootroot00000000000000import torch

from torch_geometric.testing import is_full_test
from torch_geometric.utils import to_dense_adj


def test_to_dense_adj():
    edge_index = torch.tensor([
        [0, 0, 1, 2, 3, 4],
        [0, 1, 0, 3, 4, 2],
    ])
    batch = torch.tensor([0, 0, 1, 1, 1])

    adj = to_dense_adj(edge_index, batch)
    assert adj.size() == (2, 3, 3)
    assert adj[0].tolist() == [[1, 1, 0], [1, 0, 0], [0, 0, 0]]
    assert adj[1].tolist() == [[0, 1, 0], [0, 0, 1], [1, 0, 0]]

    if is_full_test():
        jit = torch.jit.script(to_dense_adj)
        adj = jit(edge_index, batch)
        assert adj.size() == (2, 3, 3)
        assert adj[0].tolist() == [[1, 1, 0], [1, 0, 0], [0, 0, 0]]
        assert adj[1].tolist() == [[0, 1, 0], [0, 0, 1], [1, 0, 0]]

    adj = to_dense_adj(edge_index, batch, max_num_nodes=2)
    assert adj.size() == (2, 2, 2)
    assert adj[0].tolist() == [[1, 1], [1, 0]]
    assert adj[1].tolist() == [[0, 1], [0, 0]]

    adj = to_dense_adj(edge_index, batch, max_num_nodes=5)
    assert adj.size() == (2, 5, 5)
    assert adj[0][:3, :3].tolist() == [[1, 1, 0], [1, 0, 0], [0, 0, 0]]
    assert adj[1][:3, :3].tolist() == [[0, 1, 0], [0, 0, 1], [1, 0, 0]]

    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6])
    adj = to_dense_adj(edge_index, batch, edge_attr)
    assert adj.size() == (2, 3, 3)
    assert adj[0].tolist() == [[1, 2, 0], [3, 0, 0], [0, 0, 0]]
    assert adj[1].tolist() == [[0, 4, 0], [0, 0, 5], [6, 0, 0]]

    adj = to_dense_adj(edge_index, batch, edge_attr, max_num_nodes=5)
    assert adj.size() == (2, 5, 5)
    assert adj[0][:3, :3].tolist() == [[1, 2, 0], [3, 0, 0], [0, 0, 0]]
    assert adj[1][:3, :3].tolist() == [[0, 4, 0], [0, 0, 5], [6, 0, 0]]

    edge_attr = edge_attr.view(-1, 1)
    adj = to_dense_adj(edge_index, batch, edge_attr)
    assert adj.size() == (2, 3, 3, 1)

    edge_attr = edge_attr.view(-1, 1)
    adj = to_dense_adj(edge_index, batch, edge_attr, max_num_nodes=5)
    assert adj.size() == (2, 5, 5, 1)

    adj = to_dense_adj(edge_index)
    assert adj.size() == (1, 5, 5)
    assert adj[0].nonzero(as_tuple=False).t().tolist() == edge_index.tolist()

    adj = to_dense_adj(edge_index, max_num_nodes=10)
    assert adj.size() == (1, 10, 10)
    assert adj[0].nonzero(as_tuple=False).t().tolist() == edge_index.tolist()

    adj = to_dense_adj(edge_index, batch, batch_size=4)
    assert adj.size() == (4, 3, 3)


def test_to_dense_adj_with_empty_edge_index():
    edge_index = torch.tensor([[], []], dtype=torch.long)
    batch = torch.tensor([0, 0, 1, 1, 1])

    adj = to_dense_adj(edge_index)
    assert adj.size() == (1, 0, 0)

    adj = to_dense_adj(edge_index, max_num_nodes=10)
    assert adj.size() == (1, 10, 10) and adj.sum() == 0

    adj = to_dense_adj(edge_index, batch)
    assert adj.size() == (2, 3, 3) and adj.sum() == 0

    adj = to_dense_adj(edge_index, batch, max_num_nodes=10)
    assert adj.size() == (2, 10, 10) and adj.sum() == 0


def test_to_dense_adj_with_duplicate_entries():
    edge_index = torch.tensor([
        [0, 0, 0, 1, 2, 3, 3, 4],
        [0, 0, 1, 0, 3, 4, 4, 2],
    ])
    batch = torch.tensor([0, 0, 1, 1, 1])

    adj = to_dense_adj(edge_index, batch)
    assert adj.size() == (2, 3, 3)
    assert adj[0].tolist() == [[2, 1, 0], [1, 0, 0], [0, 0, 0]]
    assert adj[1].tolist() == [[0, 1, 0], [0, 0, 2], [1, 0, 0]]

    edge_attr = torch.Tensor([1, 2, 3, 4, 5, 6, 7, 8])
    adj = to_dense_adj(edge_index, batch, edge_attr)
    assert adj.size() == (2, 3, 3)
    assert adj[0].tolist() == [[3, 3, 0], [4, 0, 0], [0, 0, 0]]
    assert adj[1].tolist() == [[0, 5, 0], [0, 0, 13], [8, 0, 0]]
pytorch/test/utils/test_to_dense_batch.py000066400000000000000000000065041444262304300212460ustar00rootroot00000000000000from typing import Tuple

import pytest
import torch
from torch import Tensor

from torch_geometric.experimental import set_experimental_mode
from torch_geometric.testing import onlyFullTest
from torch_geometric.utils import to_dense_batch


@pytest.mark.parametrize('fill', [70.0, torch.tensor(49.0)])
def test_to_dense_batch(fill):
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    batch = torch.tensor([0, 0, 1, 2, 2, 2])

    item = fill.item() if isinstance(fill, Tensor) else fill
    expected = torch.Tensor([
        [[1, 2], [3, 4], [item, item]],
        [[5, 6], [item, item], [item, item]],
        [[7, 8], [9, 10], [11, 12]],
    ])

    out, mask = to_dense_batch(x, batch, fill_value=fill)
    assert out.size() == (3, 3, 2)
    assert torch.equal(out, expected)
    assert mask.tolist() == [[1, 1, 0], [1, 0, 0], [1, 1, 1]]

    out, mask = to_dense_batch(x, batch, max_num_nodes=2, fill_value=fill)
    assert out.size() == (3, 2, 2)
    assert torch.equal(out, expected[:, :2])
    assert mask.tolist() == [[1, 1], [1, 0], [1, 1]]

    out, mask = to_dense_batch(x, batch, max_num_nodes=5, fill_value=fill)
    assert out.size() == (3, 5, 2)
    assert torch.equal(out[:, :3], expected)
    assert mask.tolist() == [[1, 1, 0, 0, 0], [1, 0, 0, 0, 0], [1, 1, 1, 0, 0]]

    out, mask = to_dense_batch(x, fill_value=fill)
    assert out.size() == (1, 6, 2)
    assert torch.equal(out[0], x)
    assert mask.tolist() == [[1, 1, 1, 1, 1, 1]]

    out, mask = to_dense_batch(x, max_num_nodes=2, fill_value=fill)
    assert out.size() == (1, 2, 2)
    assert torch.equal(out[0], x[:2])
    assert mask.tolist() == [[1, 1]]

    out, mask = to_dense_batch(x, max_num_nodes=10, fill_value=fill)
    assert out.size() == (1, 10, 2)
    assert torch.equal(out[0, :6], x)
    assert mask.tolist() == [[1, 1, 1, 1, 1, 1, 0, 0, 0, 0]]

    out, mask = to_dense_batch(x, batch, batch_size=4, fill_value=fill)
    assert out.size() == (4, 3, 2)


def test_to_dense_batch_disable_dynamic_shapes():
    x = torch.Tensor([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])
    batch = torch.tensor([0, 0, 1, 2, 2, 2])

    with set_experimental_mode(True, 'disable_dynamic_shapes'):
        with pytest.raises(ValueError, match="'batch_size' needs to be set"):
            out, mask = to_dense_batch(x, batch, max_num_nodes=6)
        with pytest.raises(ValueError, match="'max_num_nodes' needs to be"):
            out, mask = to_dense_batch(x, batch, batch_size=4)
        with pytest.raises(ValueError, match="'batch_size' needs to be set"):
            out, mask = to_dense_batch(x)

        out, mask = to_dense_batch(x, batch_size=1, max_num_nodes=6)
        assert out.size() == (1, 6, 2)
        assert mask.size() == (1, 6)

        out, mask = to_dense_batch(x, batch, batch_size=3, max_num_nodes=10)
        assert out.size() == (3, 10, 2)
        assert mask.size() == (3, 10)


@onlyFullTest
def test_to_dense_batch_jit():
    @torch.jit.script
    def to_dense_batch_jit(
        x: Tensor,
        batch: Tensor,
        fill_value: Tensor,
    ) -> Tuple[Tensor, Tensor]:
        return to_dense_batch(x, batch, fill_value=fill_value)

    x = torch.randn(6, 2)
    batch = torch.tensor([0, 0, 1, 2, 2, 2])

    out, mask = to_dense_batch_jit(x, batch, fill_value=torch.tensor(0.0))
    assert out.size() == (3, 3, 2)
    assert mask.size() == (3, 3)
pytorch/test/utils/test_train_test_split_edges.py000066400000000000000000000022071444262304300230370ustar00rootroot00000000000000import pytest
import torch

from torch_geometric.data import Data
from torch_geometric.utils import train_test_split_edges


def test_train_test_split_edges():
    edge_index = torch.tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
                               [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]])
    edge_attr = torch.arange(edge_index.size(1))
    data = Data(edge_index=edge_index, edge_attr=edge_attr)
    data.num_nodes = edge_index.max().item() + 1

    with pytest.warns(UserWarning, match='deprecated'):
        data = train_test_split_edges(data, val_ratio=0.2, test_ratio=0.3)

    assert len(data) == 10
    assert data.val_pos_edge_index.size() == (2, 2)
    assert data.val_neg_edge_index.size() == (2, 2)
    assert data.test_pos_edge_index.size() == (2, 3)
    assert data.test_neg_edge_index.size() == (2, 3)
    assert data.train_pos_edge_index.size() == (2, 10)
    assert data.train_neg_adj_mask.size() == (11, 11)
    assert data.train_neg_adj_mask.sum().item() == (11**2 - 11) / 2 - 4 - 6 - 5
    assert data.train_pos_edge_attr.size() == (10, )
    assert data.val_pos_edge_attr.size() == (2, )
    assert data.test_pos_edge_attr.size() == (3, )
pytorch/test/utils/test_trim_to_layer.py000066400000000000000000000150311444262304300211510ustar00rootroot00000000000000from typing import List, Optional

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data import Data
from torch_geometric.loader import NeighborLoader
from torch_geometric.nn import GraphConv
from torch_geometric.testing import withPackage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import trim_to_layer
from torch_geometric.utils.trim_to_layer import trim_sparse_tensor


@withPackage('torch_sparse')
def test_trim_sparse_tensor():
    edge_index = torch.tensor([[0, 0, 1, 2], [1, 2, 3, 4]])
    adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=[5, 5])

    adj = trim_sparse_tensor(adj, size=(3, 3), num_seed_nodes=1)

    row, col, _ = adj.coo()
    assert row.tolist() == [0, 0]
    assert col.tolist() == [1, 2]


def test_trim_to_layer_basic():
    x0 = torch.arange(4)
    edge_index0 = torch.tensor([[1, 2, 3], [0, 1, 2]])
    edge_weight0 = torch.arange(3)

    num_sampled_nodes_per_hop = [1, 1, 1]
    num_sampled_edges_per_hop = [1, 1, 1]

    x1, edge_index1, edge_weight1 = trim_to_layer(
        layer=0,
        num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
        num_sampled_edges_per_hop=num_sampled_edges_per_hop,
        x=x0,
        edge_index=edge_index0,
        edge_attr=edge_weight0,
    )
    assert torch.equal(x1, torch.arange(4))
    assert edge_index1.tolist() == [[1, 2, 3], [0, 1, 2]]
    assert torch.equal(edge_weight1, torch.arange(3))

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj0 = SparseTensor.from_edge_index(edge_index0, edge_weight0, (4, 4))
        x1, adj_t1, _ = trim_to_layer(
            layer=0,
            num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
            num_sampled_edges_per_hop=num_sampled_edges_per_hop,
            x=x0,
            edge_index=adj0.t(),
            edge_attr=edge_weight0,
        )
        adj1 = adj_t1.t()
        assert adj1.sizes() == [4, 4]

        row, col, value = adj1.coo()
        assert torch.equal(x1, torch.arange(4))
        assert row.tolist() == [1, 2, 3]
        assert col.tolist() == [0, 1, 2]
        assert torch.equal(value, torch.arange(3))

    x2, edge_index2, edge_weight2 = trim_to_layer(
        layer=1,
        num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
        num_sampled_edges_per_hop=num_sampled_edges_per_hop,
        x=x1,
        edge_index=edge_index1,
        edge_attr=edge_weight1,
    )
    assert torch.equal(x2, torch.arange(3))
    assert edge_index2.tolist() == [[1, 2], [0, 1]]
    assert torch.equal(edge_weight2, torch.arange(2))

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj1 = SparseTensor.from_edge_index(edge_index1, edge_weight1, (4, 4))
        x2, adj_t2, _ = trim_to_layer(
            layer=1,
            num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
            num_sampled_edges_per_hop=num_sampled_edges_per_hop,
            x=x1,
            edge_index=adj1.t(),
        )
        adj2 = adj_t2.t()
        assert adj2.sizes() == [3, 3]

        row, col, value = adj2.coo()
        assert torch.equal(x2, torch.arange(3))
        assert row.tolist() == [1, 2]
        assert col.tolist() == [0, 1]
        assert torch.equal(value, torch.arange(2))

    x3, edge_index3, edge_weight3 = trim_to_layer(
        layer=2,
        num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
        num_sampled_edges_per_hop=num_sampled_edges_per_hop,
        x=x2,
        edge_index=edge_index2,
        edge_attr=edge_weight2,
    )
    assert torch.equal(x3, torch.arange(2))
    assert edge_index3.tolist() == [[1], [0]]
    assert torch.equal(edge_weight3, torch.arange(1))

    if torch_geometric.typing.WITH_TORCH_SPARSE:
        adj2 = SparseTensor.from_edge_index(edge_index2, edge_weight2, (3, 3))
        x3, adj_t3, _ = trim_to_layer(
            layer=2,
            num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
            num_sampled_edges_per_hop=num_sampled_edges_per_hop,
            x=x2,
            edge_index=adj2.t(),
        )
        adj3 = adj_t3.t()
        assert adj3.sizes() == [2, 2]

        row, col, value = adj3.coo()
        assert torch.equal(x3, torch.arange(2))
        assert row.tolist() == [1]
        assert col.tolist() == [0]
        assert torch.equal(value, torch.arange(1))


def test_trim_to_layer_hetero():
    x = {'v': torch.arange(4)}
    edge_index = {('v', 'to', 'v'): torch.tensor([[1, 2, 3], [0, 1, 2]])}
    edge_weight = {('v', 'to', 'v'): torch.arange(3)}

    num_sampled_nodes_per_hop = {'v': [1, 1, 1, 1]}
    num_sampled_edges_per_hop = {('v', 'to', 'v'): [1, 1, 1]}

    x, edge_index, edge_weight = trim_to_layer(
        layer=1,
        num_sampled_nodes_per_hop=num_sampled_nodes_per_hop,
        num_sampled_edges_per_hop=num_sampled_edges_per_hop,
        x=x,
        edge_index=edge_index,
        edge_attr=edge_weight,
    )
    assert torch.equal(x['v'], torch.arange(3))
    assert edge_index['v', 'to', 'v'].tolist() == [[1, 2], [0, 1]]
    assert torch.equal(edge_weight['v', 'to', 'v'], torch.arange(2))


class GNN(torch.nn.Module):
    def __init__(self, num_layers: int):
        super().__init__()

        self.convs = torch.nn.ModuleList(
            GraphConv(16, 16) for _ in range(num_layers))

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_weight: Tensor,
        num_sampled_nodes: Optional[List[int]] = None,
        num_sampled_edges: Optional[List[int]] = None,
    ) -> Tensor:
        for i, conv in enumerate(self.convs):
            if num_sampled_nodes is not None:
                x, edge_index, edge_weight = trim_to_layer(
                    i, num_sampled_nodes, num_sampled_edges, x, edge_index,
                    edge_weight)
            x = conv(x, edge_index, edge_weight)
        return x


@withPackage('pyg_lib')
def test_trim_to_layer_with_neighbor_loader():
    x = torch.randn(14, 16)
    edge_index = torch.tensor([
        [2, 3, 4, 5, 7, 7, 10, 11, 12, 13],
        [0, 1, 2, 3, 2, 3, 7, 7, 7, 7],
    ])
    edge_weight = torch.rand(edge_index.size(1))
    data = Data(x=x, edge_index=edge_index, edge_weight=edge_weight)

    loader = NeighborLoader(
        data,
        num_neighbors=[1, 2, 4],
        batch_size=2,
        shuffle=False,
    )
    batch = next(iter(loader))

    model = GNN(num_layers=3)
    out1 = model(batch.x, batch.edge_index, batch.edge_weight)[:2]
    assert out1.size() == (2, 16)

    out2 = model(batch.x, batch.edge_index, batch.edge_weight,
                 batch.num_sampled_nodes, batch.num_sampled_edges)[:2]
    assert out2.size() == (2, 16)

    assert torch.allclose(out1, out2)
pytorch/test/utils/test_unbatch.py000066400000000000000000000013261444262304300177260ustar00rootroot00000000000000import torch

from torch_geometric.utils import unbatch, unbatch_edge_index


def test_unbatch():
    src = torch.arange(10)
    batch = torch.tensor([0, 0, 0, 1, 1, 2, 2, 3, 4, 4])

    out = unbatch(src, batch)
    assert len(out) == 5
    for i in range(len(out)):
        assert torch.equal(out[i], src[batch == i])


def test_unbatch_edge_index():
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 4, 5, 5, 6],
        [1, 0, 2, 1, 3, 2, 5, 4, 6, 5],
    ])
    batch = torch.tensor([0, 0, 0, 0, 1, 1, 1])

    edge_indices = unbatch_edge_index(edge_index, batch)
    assert edge_indices[0].tolist() == [[0, 1, 1, 2, 2, 3], [1, 0, 2, 1, 3, 2]]
    assert edge_indices[1].tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]
pytorch/test/utils/test_undirected.py000066400000000000000000000024261444262304300204320ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.testing import is_full_test
from torch_geometric.utils import is_undirected, to_undirected


def test_is_undirected():
    row = torch.tensor([0, 1, 0])
    col = torch.tensor([1, 0, 0])
    sym_weight = torch.tensor([0, 0, 1])
    asym_weight = torch.tensor([0, 1, 1])

    assert is_undirected(torch.stack([row, col], dim=0))
    assert is_undirected(torch.stack([row, col], dim=0), sym_weight)
    assert not is_undirected(torch.stack([row, col], dim=0), asym_weight)

    row = torch.tensor([0, 1, 1])
    col = torch.tensor([1, 0, 2])

    assert not is_undirected(torch.stack([row, col], dim=0))

    if is_full_test():

        @torch.jit.script
        def jit(edge_index: Tensor) -> bool:
            return is_undirected(edge_index)

        assert not jit(torch.stack([row, col], dim=0))


def test_to_undirected():
    row = torch.tensor([0, 1, 1])
    col = torch.tensor([1, 0, 2])

    edge_index = to_undirected(torch.stack([row, col], dim=0))
    assert edge_index.tolist() == [[0, 1, 1, 2], [1, 0, 2, 1]]

    if is_full_test():

        @torch.jit.script
        def jit(edge_index: Tensor) -> Tensor:
            return to_undirected(edge_index)

        assert torch.equal(jit(torch.stack([row, col], dim=0)), edge_index)
pytorch/test/visualization/000077500000000000000000000000001444262304300164305ustar00rootroot00000000000000pytorch/test/visualization/test_graph_visualization.py000066400000000000000000000020501444262304300241200ustar00rootroot00000000000000import os.path as osp

import pytest
import torch

from torch_geometric.testing import onlyGraphviz, withPackage
from torch_geometric.visualization import visualize_graph


@onlyGraphviz
@pytest.mark.parametrize('backend', [None, 'graphviz'])
def test_visualize_graph_via_graphviz(tmp_path, backend):
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4],
        [1, 0, 2, 1, 3, 2, 4, 3],
    ])
    edge_weight = (torch.rand(edge_index.size(1)) > 0.5).float()

    path = osp.join(tmp_path, 'graph.pdf')
    visualize_graph(edge_index, edge_weight, path, backend)
    assert osp.exists(path)


@withPackage('networkx', 'matplotlib')
@pytest.mark.parametrize('backend', [None, 'networkx'])
def test_visualize_graph_via_networkx(tmp_path, backend):
    edge_index = torch.tensor([
        [0, 1, 1, 2, 2, 3, 3, 4],
        [1, 0, 2, 1, 3, 2, 4, 3],
    ])
    edge_weight = (torch.rand(edge_index.size(1)) > 0.5).float()

    path = osp.join(tmp_path, 'graph.pdf')
    visualize_graph(edge_index, edge_weight, path, backend)
    assert osp.exists(path)
pytorch/test/visualization/test_influence.py000066400000000000000000000015221444262304300220110ustar00rootroot00000000000000import torch

from torch_geometric.datasets import KarateClub
from torch_geometric.nn import GCNConv
from torch_geometric.visualization import influence


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, out_channels)
        self.conv2 = GCNConv(out_channels, out_channels)

    def forward(self, x, edge_index):
        x = torch.nn.functional.relu(self.conv1(x, edge_index))
        x = self.conv2(x, edge_index)
        return x


def test_influence():
    data = KarateClub()[0]
    x = torch.randn(data.num_nodes, 8)

    out = influence(Net(x.size(1), 16), x, data.edge_index)
    assert out.size() == (data.num_nodes, data.num_nodes)
    assert torch.allclose(out.sum(dim=-1), torch.ones(data.num_nodes),
                          atol=1e-04)
pytorch/torch_geometric/000077500000000000000000000000001444262304300157255ustar00rootroot00000000000000pytorch/torch_geometric/__init__.py000066400000000000000000000020231444262304300200330ustar00rootroot00000000000000import torch_geometric.utils
import torch_geometric.data
import torch_geometric.sampler
import torch_geometric.loader
import torch_geometric.transforms
import torch_geometric.datasets
import torch_geometric.nn
import torch_geometric.explain
import torch_geometric.profile

from .seed import seed_everything
from .home import get_home_dir, set_home_dir
from .compile import compile
from .debug import is_debug_enabled, debug, set_debug
from .experimental import (is_experimental_mode_enabled, experimental_mode,
                           set_experimental_mode)
from .lazy_loader import LazyLoader

contrib = LazyLoader('contrib', globals(), 'torch_geometric.contrib')
graphgym = LazyLoader('graphgym', globals(), 'torch_geometric.graphgym')

__version__ = '2.4.0'

__all__ = [
    'seed_everything',
    'get_home_dir',
    'set_home_dir',
    'compile',
    'is_debug_enabled',
    'debug',
    'set_debug',
    'is_experimental_mode_enabled',
    'experimental_mode',
    'set_experimental_mode',
    'torch_geometric',
    '__version__',
]
pytorch/torch_geometric/compile.py000066400000000000000000000065001444262304300177300ustar00rootroot00000000000000import logging
import warnings
from typing import Callable, Optional

import torch

import torch_geometric.typing

JIT_WARNING = ("Could not convert the 'model' into a jittable version. "
               "As such, 'torch.compile' may currently fail to correctly "
               "optimize your model. 'MessagePassing.jittable()' reported "
               "the following error: {error}")


def to_jittable(model: torch.nn.Module) -> torch.nn.Module:
    if isinstance(model, torch_geometric.nn.MessagePassing):
        try:
            model = model.jittable()
        except Exception as e:
            warnings.warn(JIT_WARNING.format(error=e))

    elif isinstance(model, torch.nn.Module):
        for name, child in model.named_children():
            if isinstance(child, torch_geometric.nn.MessagePassing):
                try:
                    setattr(model, name, child.jittable())
                except Exception as e:
                    warnings.warn(JIT_WARNING.format(error=e))
            else:
                to_jittable(child)

    return model


def compile(model: Optional[Callable] = None, *args, **kwargs) -> Callable:
    r"""Optimizes the given :pyg:`PyG` model/function via
    :meth:`torch.compile`.

    This function has the same signature as :meth:`torch.compile` (see
    `here `__),
    but it applies further optimization to make :pyg:`PyG` models/functions
    more compiler-friendly.

    Specifically, it

    1. temporarily disables the usage of the extension packages
       :obj:`torch_scatter`, :obj:`torch_sparse` and :obj:`pyg_lib`

    2. converts all instances of
       :class:`~torch_geometric.nn.conv.MessagePassing` modules into their
       jittable instances
       (see :meth:`torch_geometric.nn.conv.MessagePassing.jittable`)

    .. note::
        Without these adjustments, :meth:`torch.compile` may currently fail to
        correctly optimize your :pyg:`PyG` model.
        We are working on fully relying on :meth:`torch.compile` for future
        releases.
    """
    if model is None:

        def fn(model: Callable) -> Callable:
            if model is None:
                raise RuntimeError("'model' cannot be 'None'")
            return compile(model, *args, **kwargs)

        return fn

    # Disable the usage of external extension packages:
    # TODO (matthias) Disable only temporarily
    prev_state = {
        'WITH_INDEX_SORT': torch_geometric.typing.WITH_INDEX_SORT,
        'WITH_TORCH_SCATTER': torch_geometric.typing.WITH_TORCH_SCATTER,
    }
    warnings.filterwarnings('ignore', ".*the 'torch-scatter' package.*")
    for key in prev_state.keys():
        setattr(torch_geometric.typing, key, False)

    # Adjust the logging level of `torch.compile`:
    # TODO (matthias) Disable only temporarily
    prev_log_level = {
        'torch._dynamo': logging.getLogger('torch._dynamo').level,
        'torch._inductor': logging.getLogger('torch._inductor').level,
    }
    log_level = kwargs.pop('log_level', logging.WARNING)
    for key in prev_log_level.keys():
        logging.getLogger(key).setLevel(log_level)

    # Replace instances of `MessagePassing` by their jittable version:
    model = to_jittable(model)

    # Finally, run `torch.compile` to create an optimized version:
    out = torch.compile(model, *args, **kwargs)

    return out
pytorch/torch_geometric/config_store.py000066400000000000000000000343571444262304300207740ustar00rootroot00000000000000import copy
import inspect
from collections import defaultdict
from dataclasses import dataclass, field, make_dataclass
from typing import Any, Callable, Dict, List, Optional, Tuple, Union

import torch

EXCLUDE = {'self', 'args', 'kwargs'}

MAPPING = {
    torch.nn.Module: Any,
    torch.Tensor: Any,
}

try:
    from omegaconf import MISSING
except ImportError:
    MISSING: Any = '???'

try:
    from hydra.core.config_store import ConfigStore

    def get_node(cls: Union[str, Any]) -> Optional[Any]:
        if (not isinstance(cls, str)
                and cls.__module__ in {'builtins', 'typing'}):
            return None

        def _recursive_get_node(repo: Dict[str, Any]) -> Optional[Any]:
            for key, value in repo.items():
                if isinstance(value, dict):
                    out = _recursive_get_node(value)
                    if out is not None:
                        return out
                elif isinstance(cls, str) and key == f'{cls}.yaml':
                    return value.node
                elif getattr(value.node._metadata, 'object_type', None) == cls:
                    return value.node
                elif getattr(value.node._metadata, 'orig_type', None) == cls:
                    return value.node

        return _recursive_get_node(get_config_store().repo)

    def dataclass_from_class(cls: Union[str, Any]) -> Optional[Any]:
        r"""Returns the :obj:`dataclass` of a class registered in the global
        configuration store."""
        node = get_node(cls)
        return node._metadata.object_type if node is not None else None

    def class_from_dataclass(cls: Union[str, Any]) -> Optional[Any]:
        r"""Returns the original class of a :obj:`dataclass` registered in the
        global configuration store."""
        node = get_node(cls)
        return node._metadata.orig_type if node is not None else None

except ImportError:

    class Singleton(type):
        _instances: Dict[type, Any] = {}

        def __call__(cls, *args, **kwargs) -> Any:
            if cls not in cls._instances:
                instance = super(Singleton, cls).__call__(*args, **kwargs)
                cls._instances[cls] = instance
                return instance
            return cls._instances[cls]

    @dataclass
    class Metadata:
        orig_type: Optional[Any] = None

    @dataclass
    class ConfigNode:
        name: str
        node: Any
        group: Optional[str] = None
        _metadata: Metadata = field(default_factory=Metadata)

    class ConfigStore(metaclass=Singleton):
        def __init__(self):
            self.repo: Dict[str, Any] = defaultdict(dict)

        @classmethod
        def instance(cls, *args, **kwargs) -> 'ConfigStore':
            return cls(*args, **kwargs)

        def store(self, name: str, node: Any, group: Optional[str] = None):
            cur = self.repo
            if group is not None:
                cur = cur[group]
            cur[name] = ConfigNode(name, node, group)

    def get_node(cls: Union[str, Any]) -> Optional[ConfigNode]:
        if (not isinstance(cls, str)
                and cls.__module__ in {'builtins', 'typing'}):
            return None

        def _recursive_get_node(repo: Dict[str, Any]) -> Optional[ConfigNode]:
            for key, value in repo.items():
                if isinstance(value, dict):
                    out = _recursive_get_node(value)
                    if out is not None:
                        return out
                elif isinstance(cls, str) and key == cls:
                    return value
                elif value.node == cls:
                    return value
                elif value._metadata.orig_type == cls:
                    return value

        return _recursive_get_node(get_config_store().repo)

    def dataclass_from_class(cls: Union[str, Any]) -> Optional[Any]:
        r"""Returns the :obj:`dataclass` of a class registered in the global
        configuration store."""
        node = get_node(cls)
        return node.node if node is not None else None

    def class_from_dataclass(cls: Union[str, Any]) -> Optional[Any]:
        r"""Returns the original class of a :obj:`dataclass` registered in the
        global configuration store."""
        node = get_node(cls)
        return node._metadata.orig_type if node is not None else None


def map_annotation(
    annotation: Any,
    mapping: Optional[Dict[Any, Any]] = None,
) -> Any:

    origin = getattr(annotation, '__origin__', None)
    args = getattr(annotation, '__args__', [])
    if origin == Union or origin == list or origin == dict:
        annotation = copy.copy(annotation)
        annotation.__args__ = tuple(map_annotation(a, mapping) for a in args)
        return annotation

    if annotation in mapping or {}:
        return mapping[annotation]

    out = dataclass_from_class(annotation)
    if out is not None:
        return out

    return annotation


def to_dataclass(
    cls: Any,
    base_cls: Optional[Any] = None,
    with_target: Optional[bool] = None,
    map_args: Optional[Dict[str, Tuple]] = None,
    exclude_args: Optional[List[str]] = None,
    strict: bool = False,
) -> Any:
    r"""Converts the input arguments of a given class :obj:`cls` to a
    :obj:`dataclass` schema, *e.g.*,

    .. code-block:: python

        from torch_geometric.transforms import NormalizeFeatures

        dataclass = to_dataclass(NormalizeFeatures)

    will generate

    .. code-block:: python

        @dataclass
        class NormalizeFeatures:
            _target_: str = "torch_geometric.transforms.NormalizeFeatures"
            attrs: List[str] = field(default_factory = lambda: ["x"])

    Args:
        cls (Any): The class to generate a schema for.
        base_cls (Any, optional): The base class of the schema.
            (default: :obj:`None`)
        with_target (bool, optional): If set to :obj:`False`, will not add the
            :obj:`_target_` attribute to the schema. If set to :obj:`None`,
            will only add the :obj:`_target_` in case :obj:`base_cls` is given.
            (default: :obj:`None`)
        map_args (Dict[str, Tuple], optional): Arguments for which annotation
            and default values should be overridden. (default: :obj:`None`)
        exclude_args (List[str or int], optional): Arguments to exclude.
            (default: :obj:`None`)
        strict (bool, optional): If set to :obj:`True`, ensures that all
            arguments in both :obj:`map_args` and :obj:`exclude_args` are
            present in the input parameters. (default: :obj:`False`)
    """
    fields = []

    params = inspect.signature(cls.__init__).parameters

    if strict:  # Check that keys in map_args or exclude_args are present.
        args = set() if map_args is None else set(map_args.keys())
        if exclude_args is not None:
            args |= set([arg for arg in exclude_args if isinstance(arg, str)])
        diff = args - set(params.keys())
        if len(diff) > 0:
            raise ValueError(f"Expected input argument(s) {diff} in "
                             f"'{cls.__name__}'")

    for i, (name, arg) in enumerate(params.items()):
        if name in EXCLUDE:
            continue
        if exclude_args is not None:
            if name in exclude_args or i in exclude_args:
                continue
        if base_cls is not None:
            if name in base_cls.__dataclass_fields__:
                continue

        if map_args is not None and name in map_args:
            fields.append((name, ) + map_args[name])
            continue

        annotation, default = arg.annotation, arg.default
        annotation = map_annotation(annotation, mapping=MAPPING)

        if annotation != inspect.Parameter.empty:
            # `Union` types are not supported (except for `Optional`).
            # As such, we replace them with either `Any` or `Optional[Any]`.
            origin = getattr(annotation, '__origin__', None)
            args = getattr(annotation, '__args__', [])
            if origin == Union and type(None) in args and len(args) > 2:
                annotation = Optional[Any]
            elif origin == Union and type(None) not in args:
                annotation = Any
            elif origin == list:
                if getattr(args[0], '__origin__', None) == Union:
                    annotation = List[Any]
            elif origin == dict:
                if getattr(args[1], '__origin__', None) == Union:
                    annotation = Dict[args[0], Any]
        else:
            annotation = Any

        if str(default) == "":
            # Fix `torch.optim.SGD.lr = _RequiredParameter()`:
            # https://github.com/pytorch/hydra-torch/blob/main/
            # hydra-configs-torch/hydra_configs/torch/optim/sgd.py
            default = field(default=MISSING)
        elif default != inspect.Parameter.empty:
            if isinstance(default, (list, dict)):
                # Avoid late binding of default values inside a loop:
                # https://stackoverflow.com/questions/3431676/
                # creating-functions-in-a-loop
                def wrapper(default):
                    return lambda: default

                default = field(default_factory=wrapper(default))
        else:
            default = field(default=MISSING)

        fields.append((name, annotation, default))

    with_target = base_cls is not None if with_target is None else with_target
    if with_target:
        full_cls_name = f'{cls.__module__}.{cls.__qualname__}'
        fields.append(('_target_', str, field(default=full_cls_name)))

    return make_dataclass(cls.__qualname__, fields=fields,
                          bases=() if base_cls is None else (base_cls, ))


def get_config_store() -> ConfigStore:
    r"""Returns the global configuration store."""
    return ConfigStore.instance()


def register(
    cls: Optional[Any] = None,
    data_cls: Optional[Any] = None,
    group: Optional[str] = None,
    **kwargs,
) -> Union[Any, Callable]:
    r"""Registers a class in the global configuration store.

    Args:
        cls (Any, optional): The class to register. If set to :obj:`None`, will
            return a decorator. (default: :obj:`None`)
        data_cls (Any, optional): The data class to register. If set to
            :obj:`None`, will dynamically create the data class according to
            :class:`~torch_geometric.graphgym.config_store.to_dataclass`.
            (default: :obj:`None`)
        group (str, optional): The group in the global configuration store.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`~torch_geometric.graphgym.config_store.to_dataclass`.
    """
    if cls is not None:
        name = cls.__name__

        if data_cls is None:
            data_cls = to_dataclass(cls, **kwargs)

        get_config_store().store(name, data_cls, group)
        get_node(name)._metadata.orig_type = cls

        return data_cls

    def bounded_register(cls: Any) -> Any:  # Other-wise, return a decorator:
        register(cls=cls, data_cls=data_cls, group=group, **kwargs)
        return cls

    return bounded_register


###############################################################################


@dataclass
class Transform:
    pass


@dataclass
class Dataset:
    pass


@dataclass
class Model:
    pass


@dataclass
class Optimizer:
    pass


@dataclass
class LRScheduler:
    pass


@dataclass
class Config:
    dataset: Dataset = MISSING
    model: Model = MISSING
    optim: Optimizer = MISSING
    lr_scheduler: Optional[LRScheduler] = None


def fill_config_store():
    import torch_geometric

    config_store = get_config_store()

    # Register `torch_geometric.transforms` ###################################
    transforms = torch_geometric.transforms
    for cls_name in set(transforms.__all__) - set([
            'BaseTransform',
            'Compose',
            'ComposeFilters',
            'LinearTransformation',
            'AddMetaPaths',  # TODO
    ]):
        cls = to_dataclass(getattr(transforms, cls_name), base_cls=Transform)
        # We use an explicit additional nesting level inside each config to
        # allow for applying multiple transformations.
        # See: hydra.cc/docs/patterns/select_multiple_configs_from_config_group
        config_store.store(cls_name, group='transform', node={cls_name: cls})

    # Register `torch_geometric.datasets` #####################################
    datasets = torch_geometric.datasets
    map_dataset_args = {
        'transform': (Dict[str, Transform], field(default_factory=dict)),
        'pre_transform': (Dict[str, Transform], field(default_factory=dict)),
    }

    for cls_name in set(datasets.__all__) - set([]):
        cls = to_dataclass(getattr(datasets, cls_name), base_cls=Dataset,
                           map_args=map_dataset_args,
                           exclude_args=['pre_filter'])
        config_store.store(cls_name, group='dataset', node=cls)

    # Register `torch_geometric.models` #######################################
    models = torch_geometric.nn.models.basic_gnn
    for cls_name in set(models.__all__) - set([]):
        cls = to_dataclass(getattr(models, cls_name), base_cls=Model)
        config_store.store(cls_name, group='model', node=cls)

    # Register `torch.optim.Optimizer` ########################################
    for cls_name in set([
            key for key, cls in torch.optim.__dict__.items()
            if inspect.isclass(cls) and issubclass(cls, torch.optim.Optimizer)
    ]) - set([
            'Optimizer',
    ]):
        cls = to_dataclass(getattr(torch.optim, cls_name), base_cls=Optimizer,
                           exclude_args=['params'])
        config_store.store(cls_name, group='optimizer', node=cls)

    # Register `torch.optim.lr_scheduler` #####################################
    for cls_name in set([
            key for key, cls in torch.optim.lr_scheduler.__dict__.items()
            if inspect.isclass(cls)
    ]) - set([
            'Optimizer',
            '_LRScheduler',
            'Counter',
            'SequentialLR',
            'ChainedScheduler',
    ]):
        cls = to_dataclass(getattr(torch.optim.lr_scheduler, cls_name),
                           base_cls=LRScheduler, exclude_args=['optimizer'])
        config_store.store(cls_name, group='lr_scheduler', node=cls)

    # Register global schema ##################################################
    config_store.store('config', node=Config)
pytorch/torch_geometric/contrib/000077500000000000000000000000001444262304300173655ustar00rootroot00000000000000pytorch/torch_geometric/contrib/__init__.py000066400000000000000000000005401444262304300214750ustar00rootroot00000000000000import warnings

import torch_geometric.contrib.transforms  # noqa
import torch_geometric.contrib.datasets  # noqa
import torch_geometric.contrib.nn  # noqa
import torch_geometric.contrib.explain  # noqa

warnings.warn(
    "'torch_geometric.contrib' contains experimental code and is subject to "
    "change. Please use with caution.")

__all__ = []
pytorch/torch_geometric/contrib/datasets/000077500000000000000000000000001444262304300211755ustar00rootroot00000000000000pytorch/torch_geometric/contrib/datasets/__init__.py000066400000000000000000000000271444262304300233050ustar00rootroot00000000000000__all__ = classes = []
pytorch/torch_geometric/contrib/explain/000077500000000000000000000000001444262304300210255ustar00rootroot00000000000000pytorch/torch_geometric/contrib/explain/__init__.py000066400000000000000000000002431444262304300231350ustar00rootroot00000000000000from .graphmask_explainer import GraphMaskExplainer
from .pgm_explainer import PGMExplainer

__all__ = classes = [
    'GraphMaskExplainer',
    'PGMExplainer',
]
pytorch/torch_geometric/contrib/explain/graphmask_explainer.py000066400000000000000000000543031444262304300254300ustar00rootroot00000000000000import math
from typing import List, Optional, Tuple, Union

import numpy as np
import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import LayerNorm, Linear, Parameter, ReLU, Sequential
from tqdm import tqdm

from torch_geometric.explain import Explanation
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.config import (
    MaskType,
    ModelMode,
    ModelReturnType,
    ModelTaskLevel,
)
from torch_geometric.nn import MessagePassing


def explain_message(self, out: Tensor, x_i: Tensor, x_j: Tensor) -> Tensor:
    norm = Sequential(LayerNorm(out.size(-1)).to(out.device), ReLU())
    basis_messages = norm(out)

    if getattr(self, 'message_scale', None) is not None:
        basis_messages = basis_messages * self.message_scale.unsqueeze(-1)

        if self.message_replacement is not None:
            if basis_messages.shape == self.message_replacement.shape:
                basis_messages = (basis_messages +
                                  (1 - self.message_scale).unsqueeze(-1) *
                                  self.message_replacement)
            else:
                basis_messages = (basis_messages +
                                  ((1 - self.message_scale).unsqueeze(-1) *
                                   self.message_replacement.unsqueeze(0)))

    self.latest_messages = basis_messages
    self.latest_source_embeddings = x_j
    self.latest_target_embeddings = x_i

    return basis_messages


class GraphMaskExplainer(ExplainerAlgorithm):
    r"""The GraphMask-Explainer model from the `"Interpreting Graph Neural
    Networks for NLP With Differentiable Edge Masking"
    `_ paper for identifying layer-wise
    compact subgraph structures and node features that play a crucial role in
    the predictions made by a GNN.

    .. note::
        For an example of using :class:`GraphMaskExplainer`,
        see `examples/contrib/graphmask_explainer.py
        `_.

    Args:
        num_layers (int): The number of layers to use.
        epochs (int, optional): The number of epochs to train.
            (default: :obj:`100`)
        lr (float, optional): The learning rate to apply.
            (default: :obj:`0.01`)
        penalty_scaling (int, optional): Scaling value of penalty term. Value
            must lie between 0 and 10. (default: :obj:`5`)
        lambda_optimizer_lr (float, optional): The learning rate to optimize
            the Lagrange multiplier. (default: :obj:`1e-2`)
        init_lambda (float, optional): The Lagrange multiplier. Value must lie
            between :obj:`0` and `1`. (default: :obj:`0.55`)
        allowance (float, optional): A float value between :obj:`0` and
            :obj:`1` denotes tolerance level. (default: :obj:`0.03`)
        log (bool, optional): If set to :obj:`False`, will not log any
            learning progress. (default: :obj:`True`)
        **kwargs (optional): Additional hyper-parameters to override default
            settings in
            :attr:`~torch_geometric.nn.models.GraphMaskExplainer.coeffs`.
    """
    coeffs = {
        'node_feat_size': 1.0,
        'node_feat_reduction': 'mean',
        'node_feat_ent': 0.1,
        'EPS': 1e-15,
    }

    def __init__(
        self,
        num_layers: int,
        epochs: int = 100,
        lr: float = 0.01,
        penalty_scaling: int = 5,
        lambda_optimizer_lr: int = 1e-2,
        init_lambda: int = 0.55,
        allowance: int = 0.03,
        allow_multiple_explanations: bool = False,
        log: bool = True,
        **kwargs,
    ):
        super().__init__()
        assert 0 <= penalty_scaling <= 10
        assert 0 <= init_lambda <= 1
        assert 0 <= allowance <= 1

        self.num_layers = num_layers
        self.init_lambda = init_lambda
        self.lambda_optimizer_lr = lambda_optimizer_lr
        self.penalty_scaling = penalty_scaling
        self.allowance = allowance
        self.allow_multiple_explanations = allow_multiple_explanations
        self.epochs = epochs
        self.lr = lr
        self.log = log
        self.coeffs.update(kwargs)

    def forward(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Explanation:

        hard_node_mask = None

        if self.model_config.task_level == ModelTaskLevel.node:
            hard_node_mask, hard_edge_mask = self._get_hard_masks(
                model, index, edge_index, num_nodes=x.size(0))
        self._train_explainer(model, x, edge_index, target=target, index=index,
                              **kwargs)
        node_mask = self._post_process_mask(self.node_feat_mask,
                                            hard_node_mask, apply_sigmoid=True)
        edge_mask = self._explain(model, index=index)
        edge_mask = edge_mask[:edge_index.size(1)]

        return Explanation(node_mask=node_mask, edge_mask=edge_mask)

    def supports(self) -> bool:
        return True

    def _hard_concrete(
        self,
        input_element: Tensor,
        summarize_penalty: bool = True,
        beta: float = 1 / 3,
        gamma: float = -0.2,
        zeta: float = 1.2,
        loc_bias: int = 2,
        min_val: int = 0,
        max_val: int = 1,
        training: bool = True,
    ) -> Tuple[Tensor, Tensor]:
        r"""Helps to set the edge mask while sampling its values from the
        hard-concrete distribution."""
        input_element = input_element + loc_bias

        if training:
            u = torch.empty_like(input_element).uniform_(1e-6, 1.0 - 1e-6)

            s = torch.sigmoid(
                (torch.log(u) - torch.log(1 - u) + input_element) / beta)

            penalty = torch.sigmoid(input_element -
                                    beta * np.math.log(-gamma / zeta))
        else:
            s = torch.sigmoid(input_element)
            penalty = torch.zeros_like(input_element)

        if summarize_penalty:
            penalty = penalty.mean()

        s = s * (zeta - gamma) + gamma

        clipped_s = s.clamp(min_val, max_val)

        clip_value = (torch.min(clipped_s) + torch.max(clipped_s)) / 2
        hard_concrete = (clipped_s > clip_value).float()
        clipped_s = clipped_s + (hard_concrete - clipped_s).detach()

        return clipped_s, penalty

    def _set_masks(
        self,
        i_dim: List[int],
        j_dim: List[int],
        h_dim: List[int],
        x: Tensor,
    ):
        r"""Sets the node masks and edge masks."""
        (num_nodes, num_feat), std, device = x.size(), 0.1, x.device
        self.feat_mask_type = self.explainer_config.node_mask_type

        if self.feat_mask_type == MaskType.attributes:
            self.node_feat_mask = torch.nn.Parameter(
                torch.randn(num_nodes, num_feat, device=device) * std)
        elif self.feat_mask_type == MaskType.object:
            self.node_feat_mask = torch.nn.Parameter(
                torch.randn(num_nodes, 1, device=device) * std)
        else:
            self.node_feat_mask = torch.nn.Parameter(
                torch.randn(1, num_feat, device=device) * std)

        baselines, self.gates, full_biases = [], torch.nn.ModuleList(), []

        for v_dim, m_dim, h_dim in zip(i_dim, j_dim, h_dim):
            self.transform, self.layer_norm = [], []
            input_dims = [v_dim, m_dim, v_dim]
            for _, input_dim in enumerate(input_dims):
                self.transform.append(
                    Linear(input_dim, h_dim, bias=False).to(device))
                self.layer_norm.append(LayerNorm(h_dim).to(device))

            self.transforms = torch.nn.ModuleList(self.transform)
            self.layer_norms = torch.nn.ModuleList(self.layer_norm)

            self.full_bias = Parameter(
                torch.tensor(h_dim, dtype=torch.float, device=device))
            full_biases.append(self.full_bias)

            self.reset_parameters(input_dims, h_dim)

            self.non_linear = ReLU()
            self.output_layer = Linear(h_dim, 1).to(device)

            gate = [
                self.transforms, self.layer_norms, self.non_linear,
                self.output_layer
            ]
            self.gates.extend(gate)

            baseline = torch.tensor(m_dim, dtype=torch.float, device=device)
            stdv = 1. / math.sqrt(m_dim)
            baseline.uniform_(-stdv, stdv)
            baseline = torch.nn.Parameter(baseline)
            baselines.append(baseline)

        full_biases = torch.nn.ParameterList(full_biases)
        self.full_biases = full_biases

        baselines = torch.nn.ParameterList(baselines)
        self.baselines = baselines

        for parameter in self.parameters():
            parameter.requires_grad = False

    def _enable_layer(self, layer: int):
        r"""Enables the input layer's edge mask."""
        for d in range(layer * 4, (layer * 4) + 4):
            for parameter in self.gates[d].parameters():
                parameter.requires_grad = True
        self.full_biases[layer].requires_grad = True
        self.baselines[layer].requires_grad = True

    def reset_parameters(self, input_dims: List[int], h_dim: List[int]):
        r"""Resets all learnable parameters of the module."""
        fan_in = sum(input_dims)

        std = math.sqrt(2.0 / float(fan_in + h_dim))
        a = math.sqrt(3.0) * std

        for transform in self.transforms:
            torch.nn.init._no_grad_uniform_(transform.weight, -a, a)

        torch.nn.init.zeros_(self.full_bias)

        for layer_norm in self.layer_norms:
            layer_norm.reset_parameters()

    def _loss_regression(self, y_hat: Tensor, y: Tensor) -> Tensor:
        assert self.model_config.return_type == ModelReturnType.raw
        return F.mse_loss(y_hat, y)

    def _loss_binary_classification(self, y_hat: Tensor, y: Tensor) -> Tensor:
        if self.model_config.return_type == ModelReturnType.raw:
            loss_fn = F.binary_cross_entropy_with_logits
        elif self.model_config.return_type == ModelReturnType.probs:
            loss_fn = F.binary_cross_entropy
        else:
            assert False

        return loss_fn(y_hat.view_as(y), y.float())

    def _loss_multiclass_classification(
        self,
        y_hat: Tensor,
        y: Tensor,
    ) -> Tensor:
        if self.model_config.return_type == ModelReturnType.raw:
            loss_fn = F.cross_entropy
        elif self.model_config.return_type == ModelReturnType.probs:
            loss_fn = F.nll_loss
            y_hat = y_hat.log()
        elif self.model_config.return_type == ModelReturnType.log_probs:
            loss_fn = F.nll_loss
        else:
            assert False

        return loss_fn(y_hat, y)

    def _loss(self, y_hat: Tensor, y: Tensor, penalty: float) -> Tensor:
        if self.model_config.mode == ModelMode.binary_classification:
            loss = self._loss_binary_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.multiclass_classification:
            loss = self._loss_multiclass_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.regression:
            loss = self._loss_regression(y_hat, y)
        else:
            assert False

        g = torch.relu(loss - self.allowance).mean()
        f = penalty * self.penalty_scaling

        loss = f + F.softplus(self.lambda_op) * g

        m = self.node_feat_mask.sigmoid()
        node_feat_reduce = getattr(torch, self.coeffs['node_feat_reduction'])
        loss = loss + self.coeffs['node_feat_size'] * node_feat_reduce(m)
        ent = -m * torch.log(m + self.coeffs['EPS']) - (
            1 - m) * torch.log(1 - m + self.coeffs['EPS'])
        loss = loss + self.coeffs['node_feat_ent'] * ent.mean()

        return loss

    def _freeze_model(self, module: torch.nn.Module):
        r"""Freezes the parameters of the original GNN model by disabling
        their gradients."""
        for param in module.parameters():
            param.requires_grad = False

    def _set_flags(self, model: torch.nn.Module):
        r"""Initializes the underlying explainer model's parameters for each
        layer of the original GNN model."""
        for module in model.modules():
            if isinstance(module, MessagePassing):
                module.explain_message = explain_message.__get__(
                    module, MessagePassing)
                module.explain = True

    def _inject_messages(
        self,
        model: torch.nn.Module,
        message_scale: List[Tensor],
        message_replacement: torch.nn.ParameterList,
        set: bool = False,
    ):
        r"""Injects the computed messages into each layer of the original GNN
        model."""
        i = 0
        for module in model.modules():
            if isinstance(module, MessagePassing):
                if not set:
                    module.message_scale = message_scale[i]
                    module.message_replacement = message_replacement[i]
                    i = i + 1
                else:
                    module.message_scale = None
                    module.message_replacement = None

    def _train_explainer(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ):
        r"""Trains the underlying explainer model.

        Args:
            model (torch.nn.Module): The model to explain.
            x (torch.Tensor): The input node features.
            edge_index (torch.Tensor): The input edge indices.
            target (torch.Tensor): The target of the model.
            index (int or torch.Tensor, optional): The index of the model
                output to explain. Needs to be a single index.
                (default: :obj:`None`)
            **kwargs (optional): Additional keyword arguments passed to
                :obj:`model`.
        """
        if (not isinstance(index, Tensor) and not isinstance(index, int)
                and index is not None):
            raise ValueError("'index' parameter can only be a 'Tensor', "
                             "'integer' or set to 'None' instead.")

        self._freeze_model(model)
        self._set_flags(model)

        input_dims, output_dims = [], []
        for module in model.modules():
            if isinstance(module, MessagePassing):
                input_dims.append(module.in_channels)
                output_dims.append(module.out_channels)

        self._set_masks(input_dims, output_dims, output_dims, x)

        optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)

        for layer in reversed(list(range(self.num_layers))):
            if self.log:
                pbar = tqdm(total=self.epochs)
                if self.model_config.task_level == ModelTaskLevel.node:
                    pbar.set_description(
                        f'Train explainer for node(s) {index} with layer '
                        f'{layer}')
                elif self.model_config.task_level == ModelTaskLevel.edge:
                    pbar.set_description(
                        f"Train explainer for edge-level task with layer "
                        f"{layer}")
                else:
                    pbar.set_description(
                        f'Train explainer for graph {index} with layer '
                        f'{layer}')
            self._enable_layer(layer)
            for epoch in range(self.epochs):
                with torch.no_grad():
                    model(x, edge_index, **kwargs)
                gates, total_penalty = [], 0
                latest_source_embeddings, latest_messages = [], []
                latest_target_embeddings = []
                for module in model.modules():
                    if isinstance(module, MessagePassing):
                        latest_source_embeddings.append(
                            module.latest_source_embeddings)
                        latest_messages.append(module.latest_messages)
                        latest_target_embeddings.append(
                            module.latest_target_embeddings)
                gate_input = [
                    latest_source_embeddings, latest_messages,
                    latest_target_embeddings
                ]
                for i in range(self.num_layers):
                    output = self.full_biases[i]
                    for j in range(len(gate_input)):
                        try:
                            partial = self.gates[i * 4][j](gate_input[j][i])
                        except Exception:
                            try:
                                self._set_masks(output_dims, output_dims,
                                                output_dims, x)
                                partial = self.gates[i * 4][j](
                                    gate_input[j][i])
                            except Exception:
                                self._set_masks(input_dims, input_dims,
                                                output_dims, x)
                                partial = self.gates[i * 4][j](
                                    gate_input[j][i])
                        result = self.gates[(i * 4) + 1][j](partial)
                        output = output + result
                    relu_output = self.gates[(i * 4) + 2](output /
                                                          len(gate_input))
                    sampling_weights = self.gates[(i * 4) +
                                                  3](relu_output).squeeze(
                                                      dim=-1)
                    sampling_weights, penalty = self._hard_concrete(
                        sampling_weights)
                    gates.append(sampling_weights)
                    total_penalty += penalty

                self._inject_messages(model, gates, self.baselines)

                self.lambda_op = torch.tensor(self.init_lambda,
                                              requires_grad=True)
                optimizer_lambda = torch.optim.RMSprop(
                    [self.lambda_op], lr=self.lambda_optimizer_lr,
                    centered=True)

                optimizer.zero_grad()
                optimizer_lambda.zero_grad()

                h = x * self.node_feat_mask.sigmoid()
                y_hat, y = model(x=h, edge_index=edge_index, **kwargs), target

                if (self.model_config.task_level == ModelTaskLevel.node or
                        self.model_config.task_level == ModelTaskLevel.edge):
                    if index is not None:
                        y_hat, y = y_hat[index], y[index]

                self._inject_messages(model, gates, self.baselines, True)

                loss = self._loss(y_hat, y, total_penalty)

                loss.backward()
                optimizer.step()
                self.lambda_op.grad *= -1
                optimizer_lambda.step()

                if self.lambda_op.item() < -2:
                    self.lambda_op.data = torch.full_like(
                        self.lambda_op.data, -2)
                elif self.lambda_op.item() > 30:
                    self.lambda_op.data = torch.full_like(
                        self.lambda_op.data, 30)

                if self.log:
                    pbar.update(1)

            if self.log:
                pbar.close()

    def _explain(
        self,
        model: torch.nn.Module,
        *,
        index: Optional[Union[int, Tensor]] = None,
    ) -> Tensor:
        r"""Generates explanations for the original GNN model.

        Args:
            model (torch.nn.Module): The model to explain.
            index (int or torch.Tensor, optional): The index of the model
                output to explain. Needs to be a single index.
                (default: :obj:`None`).
        """
        if (not isinstance(index, Tensor) and not isinstance(index, int)
                and index is not None):
            raise ValueError("'index' parameter can only be a 'Tensor', "
                             "'integer' or set to 'None' instead.")

        self._freeze_model(model)
        self._set_flags(model)

        with torch.no_grad():
            latest_source_embeddings, latest_messages = [], []
            latest_target_embeddings = []
            for module in model.modules():
                if isinstance(module, MessagePassing):
                    latest_source_embeddings.append(
                        module.latest_source_embeddings)
                    latest_messages.append(module.latest_messages)
                    latest_target_embeddings.append(
                        module.latest_target_embeddings)
            gate_input = [
                latest_source_embeddings, latest_messages,
                latest_target_embeddings
            ]
            if self.log:
                pbar = tqdm(total=self.num_layers)
            for i in range(self.num_layers):
                if self.log:
                    pbar.set_description("Explain")
                output = self.full_biases[i]
                for j in range(len(gate_input)):
                    partial = self.gates[i * 4][j](gate_input[j][i])
                    result = self.gates[(i * 4) + 1][j](partial)
                    output = output + result
                relu_output = self.gates[(i * 4) + 2](output / len(gate_input))
                sampling_weights = self.gates[(i * 4) +
                                              3](relu_output).squeeze(dim=-1)
                sampling_weights, _ = self._hard_concrete(
                    sampling_weights, training=False)
                if i == 0:
                    edge_weight = sampling_weights
                else:
                    if edge_weight.size(-1) != sampling_weights.size(-1):
                        sampling_weights = F.pad(
                            input=sampling_weights,
                            pad=(0, edge_weight.size(-1) -
                                 sampling_weights.size(-1), 0, 0),
                            mode='constant', value=0)
                    edge_weight = torch.cat((edge_weight, sampling_weights), 0)
                if self.log:
                    pbar.update(1)
        if self.log:
            pbar.close()

        edge_mask = edge_weight.view(-1,
                                     edge_weight.size(0) // self.num_layers)
        edge_mask = torch.mean(edge_mask, 0)

        return edge_mask
pytorch/torch_geometric/contrib/explain/pgm_explainer.py000066400000000000000000000403631444262304300242370ustar00rootroot00000000000000import logging
from typing import List, Optional, Tuple, Union

import numpy as np
import torch
from torch import Tensor

from torch_geometric.explain import ExplainerAlgorithm
from torch_geometric.explain.config import ModelMode, ModelTaskLevel
from torch_geometric.explain.explanation import Explanation
from torch_geometric.utils import k_hop_subgraph
from torch_geometric.utils.subgraph import get_num_hops


class PGMExplainer(ExplainerAlgorithm):
    r"""The PGMExplainer model from the `"PGMExplainer: Probabilistic
    Graphical Model Explanations  for Graph Neural Networks"
    `_ paper.

    The generated :class:`~torch_geometric.explain.Explanation` provides a
    :obj:`node_mask` and a :obj:`pgm_stats` tensor, which stores the
    :math:`p`-values of each node as calculated by the Chi-squared test.

    Args:
        feature_index (List): The indices of the perturbed features. If set
            to :obj:`None`, all features are perturbed. (default: :obj:`None`)
        perturb_mode (str, optional): The method to generate the variations in
            features. One of :obj:`"randint"`, :obj:`"mean"`, :obj:`"zero"`,
            :obj:`"max"` or :obj:`"uniform"`. (default: :obj:`"randint"`)
        perturbations_is_positive_only (bool, optional): If set to :obj:`True`,
            restrict perturbed values to be positive. (default: :obj:`False`)
        is_perturbation_scaled (bool, optional): If set to :obj:`True`, will
            normalize the range of the perturbed features.
            (default: :obj:`False`)
        num_samples (int, optional): The number of samples of perturbations
            used to test the significance of nodes to the prediction.
            (default: :obj:`100`)
        max_subgraph_size (int, optional): The maximum number of neighbors to
            consider for the explanation. (default: :obj:`None`)
        significance_threshold (float, optional): The statistical threshold
            (:math:`p`-value) for which a node is considered to have an effect
            on the prediction. (default: :obj:`0.05`)
        pred_threshold (float, optional): The buffer value (in range
            :obj:`[0, 1]`) to consider the output from a perturbed data to be
            different from the original. (default: :obj:`0.1`)
    """
    def __init__(
        self,
        feature_index: Optional[List] = None,
        perturbation_mode: str = "randint",
        perturbations_is_positive_only: bool = False,
        is_perturbation_scaled: bool = False,
        num_samples: int = 100,
        max_subgraph_size: Optional[int] = None,
        significance_threshold: float = 0.05,
        pred_threshold: float = 0.1,
    ):
        super().__init__()
        self.feature_index = feature_index
        self.perturbation_mode = perturbation_mode
        self.perturbations_is_positive_only = perturbations_is_positive_only
        self.is_perturbation_scaled = is_perturbation_scaled
        self.num_samples = num_samples
        self.max_subgraph_size = max_subgraph_size
        self.significance_threshold = significance_threshold
        self.pred_threshold = pred_threshold

    def _perturb_features_on_nodes(
        self,
        x: Tensor,
        index: Tensor,
    ) -> Tensor:
        r"""Perturbs feature matrix :obj:`x`.

        Args:
            x (torch.Tensor): The feature matrix.
            index (torch.Tensor): The indices of nodes to perturb.
        """
        x_perturb = x.detach().clone()
        perturb_array = x_perturb[index]
        epsilon = 0.05 * torch.max(x, dim=0).values

        if self.perturbation_mode == "randint":
            perturb_array = torch.randint(high=2, size=perturb_array.size(),
                                          device=x.device)
        elif self.perturbation_mode == "mean":
            perturb_array[:, self.feature_index] = torch.mean(
                x[:, self.feature_index])
        elif self.perturbation_mode == "zero":
            perturb_array[:, self.feature_index] = 0
        elif self.perturbation_mode == "max":
            perturb_array[:, self.feature_index] = torch.max(
                x[:, self.feature_index])
        elif self.perturbation_mode == "uniform":
            random_perturbations = torch.rand(
                perturb_array.shape) * 2 * epsilon - epsilon
            perturb_array[:, self.feature_index] = perturb_array[
                self.feature_index] + random_perturbations
            perturb_array.clamp(min=0, max=torch.max(x, dim=0))

        if self.is_perturbation_scaled:
            perturb_array = torch.multiply(
                perturb_array, torch.rand(size=perturb_array.size())) * 2

        x_perturb[index] = perturb_array.type(x_perturb.dtype)
        return x_perturb

    def _batch_perturb_features_on_node(
            self,
            model: torch.nn.Module,
            x: Tensor,
            edge_index: Tensor,
            indices_to_perturb: np.array,
            percentage: float = 50.,  # % time node gets perturbed
            **kwargs) -> Tensor:
        r"""Perturbs the node features of a batch of graphs for graph
        classification tasks.

        Args:
            model (torch.nn.Module): The GNN model.
            x (torch.Tensor): The node feature matrix
            edge_index (torch.Tensor): The edge indices.
            indices_to_perturb (np.array): The indices of nodes to perturb.
            percentage (float, optional): The percentage of times a node gets
                perturbed. (default: :obj:`50.`)
        """
        pred_torch = model(x, edge_index, **kwargs)
        soft_pred = torch.softmax(pred_torch, dim=1)
        pred_label = torch.argmax(soft_pred, dim=1)
        num_nodes = x.shape[0]

        samples = []
        for _ in range(self.num_samples):
            x_perturb = x.detach().clone()

            seeds = np.random.randint(0, 100, size=len(indices_to_perturb))
            perturbed_node_indexes = indices_to_perturb[(seeds < percentage)]
            x_perturb = self._perturb_features_on_nodes(
                x=x_perturb,
                index=perturbed_node_indexes,
            )
            sample = np.zeros(num_nodes + 1)
            sample[perturbed_node_indexes] = 1

            pred_perturb_torch = model(x_perturb, edge_index, **kwargs)
            soft_pred_perturb = torch.softmax(pred_perturb_torch,
                                              dim=1).squeeze()

            pred_change = torch.max(soft_pred) - soft_pred_perturb[pred_label]

            sample[num_nodes] = pred_change
            samples.append(sample)

        samples = torch.tensor(np.array(samples))
        if self.perturbations_is_positive_only:
            samples = torch.abs(samples)

        top = int(self.num_samples / 8)
        top_idx = torch.argsort(samples[:, num_nodes])[-top:]
        for i in range(self.num_samples):
            if i in top_idx:
                samples[i, num_nodes] = 1
            else:
                samples[i, num_nodes] = 0

        return samples

    def _explain_graph(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        target=None,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        r"""Generates explanations for graph classification tasks.

        Args:
            model (torch.nn.Module): The model to explain.
            x (torch.Tensor): The node features.
            edge_index (torch.Tensor): The edge indices of the input graph.
            target (torch.Tensor): The predicted label from the model.

        Returns:
            pgm_nodes (List): The neighbor nodes that are significant in the
                selected node's prediction.
            pgm_stats (torch.Tensor): The :math:`p`-values of all the nodes in
                the graph, ordered by node index.
        """
        import pandas as pd
        from pgmpy.estimators.CITests import chi_square

        num_nodes = x.shape[0]
        if not self.max_subgraph_size:
            self.max_subgraph_size = int(num_nodes / 20)

        samples = self._batch_perturb_features_on_node(
            indices_to_perturb=np.array(range(num_nodes)),
            x=x,
            model=model,
            edge_index=edge_index,
        )

        # note: the PC estimator is in the original code, ie. est= PC(data)
        # but as it does nothing it is not included here
        data = pd.DataFrame(np.array(samples.detach().cpu()))

        p_values = []
        for node in range(num_nodes):
            chi2, p, _ = chi_square(
                node, int(target.detach().cpu()), [], data, boolean=False,
                significance_level=self.significance_threshold)
            p_values.append(p)

        # the original code uses number_candidates_nodes = int(top_nodes * 4)
        # if we consider 'top nodes' to equate to max number of nodes
        # it seems more correct to limit number_candidates_nodes to this
        candidate_nodes = np.argpartition(
            p_values, self.max_subgraph_size)[0:self.max_subgraph_size]

        # Round 2
        samples = self._batch_perturb_features_on_node(
            indices_to_perturb=candidate_nodes, x=x, edge_index=edge_index,
            model=model, **kwargs)

        # note: the PC estimator is in the original code, ie. est= PC(data)
        # but as it does nothing it is not included here
        data = pd.DataFrame(np.array(samples.detach().cpu()))

        p_values = []
        dependent_nodes = []

        target = num_nodes
        for node in range(num_nodes):
            _, p, _ = chi_square(
                node, target, [], data, boolean=False,
                significance_level=self.significance_threshold)
            p_values.append(p)
            if p < self.significance_threshold:
                dependent_nodes.append(node)

        top_p = np.min((self.max_subgraph_size, num_nodes - 1))
        ind_top_p = np.argpartition(p_values, top_p)[0:top_p]
        pgm_nodes = list(ind_top_p)

        node_mask = torch.zeros(x.size(), dtype=torch.int)
        node_mask[pgm_nodes] = 1
        pgm_stats = torch.tensor(p_values)

        return node_mask, pgm_stats

    def _explain_node(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        target: Tensor,
        index: int,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        r"""Generates explanations for node classification tasks.

        Args:
            model (torch.nn.Module): The model to explain.
            x (torch.Tensor): The node features.
            edge_index (torch.Tensor): The edge indices of the input graph.
            target (torch.Tensor): The predicted label from the model.
            index (torch.Tensor): The index of the node that the
                explanations are generated for.

        Returns:
            node_mask (torch.Tensor): A hard node mask corresponding to whether
                a node is significant in the selected node's prediction.
            pgm_stats (torch.Tensor): The :math:`p`-values of all the nodes in
                the graph, ordered by node index.
        """

        import pandas as pd
        from pgmpy.estimators.CITests import chi_square

        neighbors, _, _, _ = k_hop_subgraph(
            node_idx=index,
            num_hops=get_num_hops(model),
            edge_index=edge_index,
            relabel_nodes=False,
            num_nodes=x.size(0),
        )

        if index not in neighbors:
            neighbors = torch.cat([neighbors, index], dim=1)

        pred_model = model(x, edge_index, **kwargs)

        softmax_pred = torch.softmax(pred_model, dim=1)

        samples = []
        pred_samples = []

        for _ in range(self.num_samples):
            # A subset of neighbors are selected randomly for perturbing:
            seeds = np.random.choice([1, 0], size=(len(neighbors), ))
            x_perturb = self._perturb_features_on_nodes(
                x=x,
                index=neighbors[seeds == 1],
            )

            # prediction after perturbation
            pred_perturb = model(x_perturb, edge_index, **kwargs)
            softmax_pred_perturb = torch.softmax(pred_perturb, dim=1)
            sample_bool = np.ones(shape=(len(neighbors), ))
            sample_bool[(
                (softmax_pred_perturb[neighbors, target] + self.pred_threshold)
                >= softmax_pred[neighbors, target]).cpu()] = 0

            samples.append(seeds)
            pred_samples.append(sample_bool)

        samples = np.asarray(samples)
        pred_samples = np.asarray(pred_samples)
        combine_samples = (samples * 10 + pred_samples) + 1

        neighbors = np.array(neighbors.detach().cpu())
        data_pgm = pd.DataFrame(combine_samples)
        data_pgm = data_pgm.rename(columns={
            0: "A",
            1: "B"
        })  # Trick to use chi_square test on first two data columns
        index_original_to_subgraph = dict(
            zip(neighbors, list(data_pgm.columns)))
        index_subgraph_to_original = dict(
            zip(list(data_pgm.columns), neighbors))
        p_values = []

        dependent_neighbors = []
        dependent_neighbors_p_values = []
        for node in neighbors:
            if node == index:
                # null hypothesis is perturbing a particular
                # node has no effect on result
                p = 0
            else:
                _, p, _ = chi_square(
                    index_original_to_subgraph[node],
                    index_original_to_subgraph[index], [], data_pgm,
                    boolean=False,
                    significance_level=self.significance_threshold)
            p_values.append(p)
            if p < self.significance_threshold:
                dependent_neighbors.append(node)
                dependent_neighbors_p_values.append(p)

        pgm_stats = torch.ones(x.size(0), dtype=torch.float)
        node_mask = torch.zeros(x.size(), dtype=torch.int)

        pgm_stats[neighbors] = torch.tensor(p_values, dtype=torch.float)

        if self.max_subgraph_size is None:
            pgm_nodes = dependent_neighbors
        else:
            top_p = np.min((self.max_subgraph_size, len(neighbors) - 1))
            ind_top_p = np.argpartition(p_values, top_p)[0:top_p]
            pgm_nodes = [
                index_subgraph_to_original[node] for node in ind_top_p
            ]
        node_mask[pgm_nodes] = 1
        return node_mask, pgm_stats

    def forward(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,  # node index
        **kwargs,
    ) -> Explanation:

        if self.feature_index is None:
            self.feature_index = list(range(x.shape[-1]))

        if isinstance(index, Tensor):
            if index.numel() > 1:
                raise NotImplementedError(
                    f"'{self.__class__.__name}' only supports a single "
                    f"`index` for now")
            index = index.item()

        if self.model_config.task_level == ModelTaskLevel.node:
            node_mask, pgm_stats = self._explain_node(
                model=model,
                x=x,
                edge_index=edge_index,
                target=target[index],
                index=index,
                **kwargs,
            )
            return Explanation(
                x=x,
                edge_index=edge_index,
                node_mask=node_mask,
                pgm_stats=pgm_stats,
            )

        elif self.model_config.task_level == ModelTaskLevel.graph:
            node_mask, pgm_stats = self._explain_graph(
                model=model,
                x=x,
                target=target,
                edge_index=edge_index,
                **kwargs,
            )
            return Explanation(
                node_mask=node_mask,
                pgm_stats=pgm_stats,
            )

    def supports(self) -> bool:
        task_level = self.model_config.task_level
        if task_level not in [ModelTaskLevel.node, ModelTaskLevel.graph]:
            logging.error(f"Task level '{task_level.value}' not supported")
            return False
        if self.explainer_config.edge_mask_type is not None:
            logging.error("Generation of edge masks is not supported")
            return False
        if self.model_config.mode == ModelMode.regression:
            logging.error("'PGMExplainer' only supports classification tasks")
            return False
        return True
pytorch/torch_geometric/contrib/nn/000077500000000000000000000000001444262304300200005ustar00rootroot00000000000000pytorch/torch_geometric/contrib/nn/__init__.py000066400000000000000000000001101444262304300221010ustar00rootroot00000000000000from .conv import *  # noqa
from .models import *  # noqa

__all__ = []
pytorch/torch_geometric/contrib/nn/conv/000077500000000000000000000000001444262304300207455ustar00rootroot00000000000000pytorch/torch_geometric/contrib/nn/conv/__init__.py000066400000000000000000000000271444262304300230550ustar00rootroot00000000000000__all__ = classes = []
pytorch/torch_geometric/contrib/nn/models/000077500000000000000000000000001444262304300212635ustar00rootroot00000000000000pytorch/torch_geometric/contrib/nn/models/__init__.py000066400000000000000000000001611444262304300233720ustar00rootroot00000000000000from .rbcd_attack import PRBCDAttack, GRBCDAttack

__all__ = classes = [
    'PRBCDAttack',
    'GRBCDAttack',
]
pytorch/torch_geometric/contrib/nn/models/rbcd_attack.py000066400000000000000000001007551444262304300241060ustar00rootroot00000000000000from collections import defaultdict
from functools import partial
from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union

import numpy as np
import torch
import torch.nn.functional as F
from torch import Tensor
from tqdm import tqdm

from torch_geometric.utils import coalesce, to_undirected

# (predictions, labels, ids/mask) -> Tensor with one element
LOSS_TYPE = Callable[[Tensor, Tensor, Optional[Tensor]], Tensor]


class PRBCDAttack(torch.nn.Module):
    r"""The Projected Randomized Block Coordinate Descent (PRBCD) adversarial
    attack from the `Robustness of Graph Neural Networks at Scale
    `_ paper.

    This attack uses an efficient gradient based approach that (during the
    attack) relaxes the discrete entries in the adjacency matrix
    :math:`\{0, 1\}` to :math:`[0, 1]` and solely perturbs the adjacency matrix
    (no feature perturbations). Thus, this attack supports all models that can
    handle weighted graphs that are differentiable w.r.t. these edge weights,
    *e.g.*, :class:`~torch_geometric.nn.conv.GCNConv` or
    :class:`~torch_geometric.nn.conv.GraphConv`. For non-differentiable models
    you might need modifications, e.g., see example for
    :class:`~torch_geometric.nn.conv.GATConv`.

    The memory overhead is driven by the additional edges (at most
    :attr:`block_size`). For scalability reasons, the block is drawn with
    replacement and then the index is made unique. Thus, the actual block size
    is typically slightly smaller than specified.

    This attack can be used for both global and local attacks as well as
    test-time attacks (evasion) and training-time attacks (poisoning). Please
    see the provided examples.

    This attack is designed with a focus on node- or graph-classification,
    however, to adapt to other tasks you most likely only need to provide an
    appropriate loss and model. However, we currently do not support batching
    out of the box (sampling needs to be adapted).

    .. note::
        For examples of using the PRBCD Attack, see
        `examples/contrib/rbcd_attack.py
        `_
        for a test time attack (evasion) or
        `examples/contrib/rbcd_attack_poisoning.py
        `_
        for a training time (poisoning) attack.

    Args:
        model (torch.nn.Module): The GNN module to assess.
        block_size (int): Number of randomly selected elements in the
            adjacency matrix to consider.
        epochs (int, optional): Number of epochs (aborts early if
            :obj:`mode='greedy'` and budget is satisfied) (default: :obj:`125`)
        epochs_resampling (int, optional): Number of epochs to resample the
            random block. (default: obj:`100`)
        loss (str or callable, optional): A loss to quantify the "strength" of
            an attack. Note that this function must match the output format of
            :attr:`model`. By default, it is assumed that the task is
            classification and that the model returns raw predictions (*i.e.*,
            no output activation) or uses :obj:`logsoftmax`. Moreover, and the
            number of predictions should match the number of labels passed to
            :attr:`attack`. Either pass a callable or one of: :obj:`'masked'`,
            :obj:`'margin'`, :obj:`'prob_margin'`, :obj:`'tanh_margin'`.
            (default: :obj:`'prob_margin'`)
        metric (callable, optional): Second (potentially
            non-differentiable) loss for monitoring or early stopping (if
            :obj:`mode='greedy'`). (default: same as :attr:`loss`)
        lr (float, optional): Learning rate for updating edge weights.
            Additionally, it is heuristically corrected for :attr:`block_size`,
            budget (see :attr:`attack`) and graph size. (default: :obj:`1_000`)
        is_undirected (bool, optional): If :obj:`True` the graph is
            assumed to be undirected. (default: :obj:`True`)
        log (bool, optional): If set to :obj:`False`, will not log any learning
            progress. (default: :obj:`True`)
    """
    coeffs = {
        'max_final_samples': 20,
        'max_trials_sampling': 20,
        'with_early_stopping': True,
        'eps': 1e-7
    }

    def __init__(
        self,
        model: torch.nn.Module,
        block_size: int,
        epochs: int = 125,
        epochs_resampling: int = 100,
        loss: Optional[Union[str, LOSS_TYPE]] = 'prob_margin',
        metric: Optional[Union[str, LOSS_TYPE]] = None,
        lr: float = 1_000,
        is_undirected: bool = True,
        log: bool = True,
        **kwargs,
    ):
        super().__init__()

        self.model = model
        self.block_size = block_size
        self.epochs = epochs

        if isinstance(loss, str):
            if loss == 'masked':
                self.loss = self._masked_cross_entropy
            elif loss == 'margin':
                self.loss = partial(self._margin_loss, reduce='mean')
            elif loss == 'prob_margin':
                self.loss = self._probability_margin_loss
            elif loss == 'tanh_margin':
                self.loss = self._tanh_margin_loss
            else:
                raise ValueError(f'Unknown loss `{loss}`')
        else:
            self.loss = loss

        self.is_undirected = is_undirected
        self.log = log
        self.metric = metric or self.loss

        self.epochs_resampling = epochs_resampling
        self.lr = lr

        self.coeffs.update(kwargs)

    def attack(
        self,
        x: Tensor,
        edge_index: Tensor,
        labels: Tensor,
        budget: int,
        idx_attack: Optional[Tensor] = None,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        """Attack the predictions for the provided model and graph.

        A subset of predictions may be specified with :attr:`idx_attack`. The
        attack is allowed to flip (i.e. add or delete) :attr:`budget` edges and
        will return the strongest perturbation it can find. It returns both the
        resulting perturbed :attr:`edge_index` as well as the perturbations.

        Args:
            x (torch.Tensor): The node feature matrix.
            edge_index (torch.Tensor): The edge indices.
            labels (torch.Tensor): The labels.
            budget (int): The number of allowed perturbations (i.e.
                number of edges that are flipped at most).
            idx_attack (torch.Tensor, optional): Filter for predictions/labels.
                Shape and type must match that it can index :attr:`labels`
                and the model's predictions.
            **kwargs (optional): Additional arguments passed to the GNN module.

        :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`)
        """
        self.model.eval()

        self.device = x.device
        assert kwargs.get('edge_weight') is None
        edge_weight = torch.ones(edge_index.size(1), device=self.device)
        self.edge_index = edge_index.cpu().clone()
        self.edge_weight = edge_weight.cpu().clone()
        self.num_nodes = x.size(0)

        # For collecting attack statistics
        self.attack_statistics = defaultdict(list)

        # Prepare attack and define `self.iterable` to iterate over
        step_sequence = self._prepare(budget)

        # Loop over the epochs (Algorithm 1, line 5)
        for step in tqdm(step_sequence, disable=not self.log, desc='Attack'):
            loss, gradient = self._forward_and_gradient(
                x, labels, idx_attack, **kwargs)

            scalars = self._update(step, gradient, x, labels, budget,
                                   idx_attack, **kwargs)

            scalars['loss'] = loss.item()
            self._append_statistics(scalars)

        perturbed_edge_index, flipped_edges = self._close(
            x, labels, budget, idx_attack, **kwargs)

        assert flipped_edges.size(1) <= budget, (
            f'# perturbed edges {flipped_edges.size(1)} '
            f'exceeds budget {budget}')

        return perturbed_edge_index, flipped_edges

    def _prepare(self, budget: int) -> Iterable[int]:
        """Prepare attack."""
        if self.block_size <= budget:
            raise ValueError(
                f'The search space size ({self.block_size}) must be '
                f'greater than the number of permutations ({budget})')

        # For early stopping (not explicitly covered by pseudo code)
        self.best_metric = float('-Inf')

        # Sample initial search space (Algorithm 1, line 3-4)
        self._sample_random_block(budget)

        steps = range(self.epochs)
        return steps

    @torch.no_grad()
    def _update(self, epoch: int, gradient: Tensor, x: Tensor, labels: Tensor,
                budget: int, idx_attack: Optional[Tensor] = None,
                **kwargs) -> Dict[str, float]:
        """Update edge weights given gradient."""
        # Gradient update step (Algorithm 1, line 7)
        self.block_edge_weight = self._update_edge_weights(
            budget, self.block_edge_weight, epoch, gradient)

        # For monitoring
        pmass_update = torch.clamp(self.block_edge_weight, 0, 1)
        # Projection to stay within relaxed `L_0` budget
        # (Algorithm 1, line 8)
        self.block_edge_weight = self._project(budget, self.block_edge_weight,
                                               self.coeffs['eps'])

        # For monitoring
        scalars = dict(
            prob_mass_after_update=pmass_update.sum().item(),
            prob_mass_after_update_max=pmass_update.max().item(),
            prob_mass_after_projection=self.block_edge_weight.sum().item(),
            prob_mass_after_projection_nonzero_weights=(
                self.block_edge_weight > self.coeffs['eps']).sum().item(),
            prob_mass_after_projection_max=self.block_edge_weight.max().item())
        if not self.coeffs['with_early_stopping']:
            return scalars

        # Calculate metric after the current epoch (overhead
        # for monitoring and early stopping)
        topk_block_edge_weight = torch.zeros_like(self.block_edge_weight)
        topk_block_edge_weight[torch.topk(self.block_edge_weight,
                                          budget).indices] = 1
        edge_index, edge_weight = self._get_modified_adj(
            self.edge_index, self.edge_weight, self.block_edge_index,
            topk_block_edge_weight)
        prediction = self._forward(x, edge_index, edge_weight, **kwargs)
        metric = self.metric(prediction, labels, idx_attack)

        # Save best epoch for early stopping
        # (not explicitly covered by pseudo code)
        if metric > self.best_metric:
            self.best_metric = metric
            self.best_block = self.current_block.cpu().clone()
            self.best_edge_index = self.block_edge_index.cpu().clone()
            self.best_pert_edge_weight = self.block_edge_weight.cpu().clone()

        # Resampling of search space (Algorithm 1, line 9-14)
        if epoch < self.epochs_resampling - 1:
            self._resample_random_block(budget)
        elif epoch == self.epochs_resampling - 1:
            # Retrieve best epoch if early stopping is active
            # (not explicitly covered by pseudo code)
            self.current_block = self.best_block.to(self.device)
            self.block_edge_index = self.best_edge_index.to(self.device)
            block_edge_weight = self.best_pert_edge_weight.clone()
            self.block_edge_weight = block_edge_weight.to(self.device)

        scalars['metric'] = metric.item()
        return scalars

    @torch.no_grad()
    def _close(self, x: Tensor, labels: Tensor, budget: int,
               idx_attack: Optional[Tensor] = None,
               **kwargs) -> Tuple[Tensor, Tensor]:
        """Clean up and prepare return argument."""
        # Retrieve best epoch if early stopping is active
        # (not explicitly covered by pseudo code)
        if self.coeffs['with_early_stopping']:
            self.current_block = self.best_block.to(self.device)
            self.block_edge_index = self.best_edge_index.to(self.device)
            self.block_edge_weight = self.best_pert_edge_weight.to(self.device)

        # Sample final discrete graph (Algorithm 1, line 16)
        edge_index, flipped_edges = self._sample_final_edges(
            x, labels, budget, idx_attack=idx_attack, **kwargs)

        return edge_index, flipped_edges

    def _forward(self, x: Tensor, edge_index: Tensor, edge_weight: Tensor,
                 **kwargs) -> Tensor:
        """Forward model."""
        return self.model(x, edge_index, edge_weight, **kwargs)

    def _forward_and_gradient(self, x: Tensor, labels: Tensor,
                              idx_attack: Optional[Tensor] = None,
                              **kwargs) -> Tuple[Tensor, Tensor]:
        """Forward and update edge weights."""
        self.block_edge_weight.requires_grad = True

        # Retrieve sparse perturbed adjacency matrix `A \oplus p_{t-1}`
        # (Algorithm 1, line 6 / Algorithm 2, line 7)
        edge_index, edge_weight = self._get_modified_adj(
            self.edge_index, self.edge_weight, self.block_edge_index,
            self.block_edge_weight)

        # Get prediction (Algorithm 1, line 6 / Algorithm 2, line 7)
        prediction = self._forward(x, edge_index, edge_weight, **kwargs)
        # Calculate loss combining all each node
        # (Algorithm 1, line 7 / Algorithm 2, line 8)
        loss = self.loss(prediction, labels, idx_attack)
        # Retrieve gradient towards the current block
        # (Algorithm 1, line 7 / Algorithm 2, line 8)
        gradient = torch.autograd.grad(loss, self.block_edge_weight)[0]

        return loss, gradient

    def _get_modified_adj(self, edge_index: Tensor, edge_weight: Tensor,
                          block_edge_index: Tensor,
                          block_edge_weight: Tensor) -> Tuple[Tensor, Tensor]:
        """Merges adjacency matrix with current block (incl. weights)"""
        if self.is_undirected:
            block_edge_index, block_edge_weight = to_undirected(
                block_edge_index, block_edge_weight, num_nodes=self.num_nodes,
                reduce='mean')

        modified_edge_index = torch.cat(
            (edge_index.to(self.device), block_edge_index), dim=-1)
        modified_edge_weight = torch.cat(
            (edge_weight.to(self.device), block_edge_weight))

        modified_edge_index, modified_edge_weight = coalesce(
            modified_edge_index, modified_edge_weight,
            num_nodes=self.num_nodes, reduce='sum')

        # Allow (soft) removal of edges
        is_edge_in_clean_adj = modified_edge_weight > 1
        modified_edge_weight[is_edge_in_clean_adj] = (
            2 - modified_edge_weight[is_edge_in_clean_adj])

        return modified_edge_index, modified_edge_weight

    def _filter_self_loops_in_block(self, with_weight: bool):
        is_not_sl = self.block_edge_index[0] != self.block_edge_index[1]
        self.current_block = self.current_block[is_not_sl]
        self.block_edge_index = self.block_edge_index[:, is_not_sl]
        if with_weight:
            self.block_edge_weight = self.block_edge_weight[is_not_sl]

    def _sample_random_block(self, budget: int = 0):
        for _ in range(self.coeffs['max_trials_sampling']):
            num_possible_edges = self._num_possible_edges(
                self.num_nodes, self.is_undirected)
            self.current_block = torch.randint(num_possible_edges,
                                               (self.block_size, ),
                                               device=self.device)
            self.current_block = torch.unique(self.current_block, sorted=True)
            if self.is_undirected:
                self.block_edge_index = self._linear_to_triu_idx(
                    self.num_nodes, self.current_block)
            else:
                self.block_edge_index = self._linear_to_full_idx(
                    self.num_nodes, self.current_block)
                self._filter_self_loops_in_block(with_weight=False)

            self.block_edge_weight = torch.full(self.current_block.shape,
                                                self.coeffs['eps'],
                                                device=self.device)
            if self.current_block.size(0) >= budget:
                return
        raise RuntimeError('Sampling random block was not successful. '
                           'Please decrease `budget`.')

    def _resample_random_block(self, budget: int):
        # Keep at most half of the block (i.e. resample low weights)
        sorted_idx = torch.argsort(self.block_edge_weight)
        keep_above = (self.block_edge_weight <=
                      self.coeffs['eps']).sum().long()
        if keep_above < sorted_idx.size(0) // 2:
            keep_above = sorted_idx.size(0) // 2
        sorted_idx = sorted_idx[keep_above:]

        self.current_block = self.current_block[sorted_idx]

        # Sample until enough edges were drawn
        for _ in range(self.coeffs['max_trials_sampling']):
            n_edges_resample = self.block_size - self.current_block.size(0)
            num_possible_edges = self._num_possible_edges(
                self.num_nodes, self.is_undirected)
            lin_index = torch.randint(num_possible_edges, (n_edges_resample, ),
                                      device=self.device)

            current_block = torch.cat((self.current_block, lin_index))
            self.current_block, unique_idx = torch.unique(
                current_block, sorted=True, return_inverse=True)

            if self.is_undirected:
                self.block_edge_index = self._linear_to_triu_idx(
                    self.num_nodes, self.current_block)
            else:
                self.block_edge_index = self._linear_to_full_idx(
                    self.num_nodes, self.current_block)

            # Merge existing weights with new edge weights
            block_edge_weight_prev = self.block_edge_weight[sorted_idx]
            self.block_edge_weight = torch.full(self.current_block.shape,
                                                self.coeffs['eps'],
                                                device=self.device)
            self.block_edge_weight[
                unique_idx[:sorted_idx.size(0)]] = block_edge_weight_prev

            if not self.is_undirected:
                self._filter_self_loops_in_block(with_weight=True)

            if self.current_block.size(0) > budget:
                return
        raise RuntimeError('Sampling random block was not successful.'
                           'Please decrease `budget`.')

    def _sample_final_edges(self, x: Tensor, labels: Tensor, budget: int,
                            idx_attack: Optional[Tensor] = None,
                            **kwargs) -> Tuple[Tensor, Tensor]:
        best_metric = float('-Inf')
        block_edge_weight = self.block_edge_weight
        block_edge_weight[block_edge_weight <= self.coeffs['eps']] = 0

        for i in range(self.coeffs['max_final_samples']):
            if i == 0:
                # In first iteration employ top k heuristic instead of sampling
                sampled_edges = torch.zeros_like(block_edge_weight)
                sampled_edges[torch.topk(block_edge_weight,
                                         budget).indices] = 1
            else:
                sampled_edges = torch.bernoulli(block_edge_weight).float()

            if sampled_edges.sum() > budget:
                # Allowed budget is exceeded
                continue

            edge_index, edge_weight = self._get_modified_adj(
                self.edge_index, self.edge_weight, self.block_edge_index,
                sampled_edges)
            prediction = self._forward(x, edge_index, edge_weight, **kwargs)
            metric = self.metric(prediction, labels, idx_attack)

            # Save best sample
            if metric > best_metric:
                best_metric = metric
                self.block_edge_weight = sampled_edges.clone().cpu()

        # Recover best sample
        self.block_edge_weight = self.block_edge_weight.to(self.device)
        flipped_edges = self.block_edge_index[:, self.block_edge_weight > 0]

        edge_index, edge_weight = self._get_modified_adj(
            self.edge_index, self.edge_weight, self.block_edge_index,
            self.block_edge_weight)
        edge_mask = edge_weight == 1
        edge_index = edge_index[:, edge_mask]

        return edge_index, flipped_edges

    def _update_edge_weights(self, budget: int, block_edge_weight: Tensor,
                             epoch: int, gradient: Tensor) -> Tensor:
        # The learning rate is refined heuristically, s.t. (1) it is
        # independent of the number of perturbations (assuming an undirected
        # adjacency matrix) and (2) to decay learning rate during fine-tuning
        # (i.e. fixed search space).
        lr = (budget / self.num_nodes * self.lr /
              np.sqrt(max(0, epoch - self.epochs_resampling) + 1))
        return block_edge_weight + lr * gradient

    @staticmethod
    def _project(budget: int, values: Tensor, eps: float = 1e-7) -> Tensor:
        r"""Project :obj:`values`:
        :math:`budget \ge \sum \Pi_{[0, 1]}(\text{values})`."""
        if torch.clamp(values, 0, 1).sum() > budget:
            left = (values - 1).min()
            right = values.max()
            miu = PRBCDAttack._bisection(values, left, right, budget)
            values = values - miu
        return torch.clamp(values, min=eps, max=1 - eps)

    @staticmethod
    def _bisection(edge_weights: Tensor, a: float, b: float, n_pert: int,
                   eps=1e-5, max_iter=1e3) -> Tensor:
        """Bisection search for projection."""
        def shift(offset: float):
            return (torch.clamp(edge_weights - offset, 0, 1).sum() - n_pert)

        miu = a
        for _ in range(int(max_iter)):
            miu = (a + b) / 2
            # Check if middle point is root
            if (shift(miu) == 0.0):
                break
            # Decide the side to repeat the steps
            if (shift(miu) * shift(a) < 0):
                b = miu
            else:
                a = miu
            if ((b - a) <= eps):
                break
        return miu

    @staticmethod
    def _num_possible_edges(n: int, is_undirected: bool) -> int:
        """Determine number of possible edges for graph."""
        if is_undirected:
            return n * (n - 1) // 2
        else:
            return int(n**2)  # We filter self-loops later

    @staticmethod
    def _linear_to_triu_idx(n: int, lin_idx: Tensor) -> Tensor:
        """Linear index to upper triangular matrix without diagonal. This is
        similar to
        https://stackoverflow.com/questions/242711/algorithm-for-index-numbers-of-triangular-matrix-coefficients/28116498#28116498
        with number nodes decremented and col index incremented by one."""
        nn = n * (n - 1)
        row_idx = n - 2 - torch.floor(
            torch.sqrt(-8 * lin_idx.double() + 4 * nn - 7) / 2.0 - 0.5).long()
        col_idx = 1 + lin_idx + row_idx - nn // 2 + torch.div(
            (n - row_idx) * (n - row_idx - 1), 2, rounding_mode='floor')
        return torch.stack((row_idx, col_idx))

    @staticmethod
    def _linear_to_full_idx(n: int, lin_idx: Tensor) -> Tensor:
        """Linear index to dense matrix including diagonal."""
        row_idx = torch.div(lin_idx, n, rounding_mode='floor')
        col_idx = lin_idx % n
        return torch.stack((row_idx, col_idx))

    @staticmethod
    def _margin_loss(score: Tensor, labels: Tensor,
                     idx_mask: Optional[Tensor] = None,
                     reduce: Optional[str] = None) -> Tensor:
        r"""Margin loss between true score and highest non-target score:

        .. math::
            m = - s_{y} + max_{y' \ne y} s_{y'}

        where :math:`m` is the margin :math:`s` the score and :math:`y` the
        labels.

        Args:
            score (Tensor): Some score (*e.g.*, logits) of shape
                :obj:`[n_elem, dim]`.
            labels (LongTensor): The labels of shape :obj:`[n_elem]`.
            idx_mask (Tensor, optional): To select subset of `score` and
                `labels` of shape :obj:`[n_select]`. Defaults to None.
            reduce (str, optional): if :obj:`mean` the result is aggregated.
                Otherwise, return element wise margin.

        :rtype: (Tensor)
        """
        if idx_mask is not None:
            score = score[idx_mask]
            labels = labels[idx_mask]

        linear_idx = torch.arange(score.size(0), device=score.device)
        true_score = score[linear_idx, labels]

        score = score.clone()
        score[linear_idx, labels] = float('-Inf')
        best_non_target_score = score.amax(dim=-1)

        margin_ = best_non_target_score - true_score
        if reduce is None:
            return margin_
        return margin_.mean()

    @staticmethod
    def _tanh_margin_loss(prediction: Tensor, labels: Tensor,
                          idx_mask: Optional[Tensor] = None) -> Tensor:
        """Calculate tanh margin loss, a node-classification loss that focuses
        on nodes next to decision boundary.

        Args:
            prediction (Tensor): Prediction of shape :obj:`[n_elem, dim]`.
            labels (LongTensor): The labels of shape :obj:`[n_elem]`.
            idx_mask (Tensor, optional): To select subset of `score` and
                `labels` of shape :obj:`[n_select]`. Defaults to None.

        :rtype: (Tensor)
        """
        log_prob = F.log_softmax(prediction, dim=-1)
        margin_ = GRBCDAttack._margin_loss(log_prob, labels, idx_mask)
        loss = torch.tanh(margin_).mean()
        return loss

    @staticmethod
    def _probability_margin_loss(prediction: Tensor, labels: Tensor,
                                 idx_mask: Optional[Tensor] = None) -> Tensor:
        """Calculate probability margin loss, a node-classification loss that
        focuses  on nodes next to decision boundary. See `Are Defenses for
        Graph Neural Networks Robust?
        `_ for details.

        Args:
            prediction (Tensor): Prediction of shape :obj:`[n_elem, dim]`.
            labels (LongTensor): The labels of shape :obj:`[n_elem]`.
            idx_mask (Tensor, optional): To select subset of `score` and
                `labels` of shape :obj:`[n_select]`. Defaults to None.

        :rtype: (Tensor)
        """
        prob = F.softmax(prediction, dim=-1)
        margin_ = GRBCDAttack._margin_loss(prob, labels, idx_mask)
        return margin_.mean()

    @staticmethod
    def _masked_cross_entropy(log_prob: Tensor, labels: Tensor,
                              idx_mask: Optional[Tensor] = None) -> Tensor:
        """Calculate masked cross entropy loss, a node-classification loss that
        focuses on nodes next to decision boundary.

        Args:
            log_prob (Tensor): Log probabilities of shape :obj:`[n_elem, dim]`.
            labels (LongTensor): The labels of shape :obj:`[n_elem]`.
            idx_mask (Tensor, optional): To select subset of `score` and
                `labels` of shape :obj:`[n_select]`. Defaults to None.

        :rtype: (Tensor)
        """
        if idx_mask is not None:
            log_prob = log_prob[idx_mask]
            labels = labels[idx_mask]

        is_correct = log_prob.argmax(-1) == labels
        if is_correct.any():
            log_prob = log_prob[is_correct]
            labels = labels[is_correct]

        return F.nll_loss(log_prob, labels)

    def _append_statistics(self, mapping: Dict[str, Any]):
        for key, value in mapping.items():
            self.attack_statistics[key].append(value)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'


class GRBCDAttack(PRBCDAttack):
    r"""The Greedy Randomized Block Coordinate Descent (GRBCD) adversarial
    attack from the `Robustness of Graph Neural Networks at Scale
    `_ paper.

    GRBCD shares most of the properties and requirements with
    :class:`PRBCDAttack`. It also uses an efficient gradient based approach.
    However, it greedily flips edges based on the gradient towards the
    adjacency matrix.

    .. note::
        For examples of using the GRBCD Attack, see
        `examples/contrib/rbcd_attack.py
        `_
        for a test time attack (evasion).

    Args:
        model (torch.nn.Module): The GNN module to assess.
        block_size (int): Number of randomly selected elements in the
            adjacency matrix to consider.
        epochs (int, optional): Number of epochs (aborts early if
            :obj:`mode='greedy'` and budget is satisfied) (default: :obj:`125`)
        loss (str or callable, optional): A loss to quantify the "strength" of
            an attack. Note that this function must match the output format of
            :attr:`model`. By default, it is assumed that the task is
            classification and that the model returns raw predictions (*i.e.*,
            no output activation) or uses :obj:`logsoftmax`. Moreover, and the
            number of predictions should match the number of labels passed to
            :attr:`attack`. Either pass Callable or one of: :obj:`'masked'`,
            :obj:`'margin'`, :obj:`'prob_margin'`, :obj:`'tanh_margin'`.
            (default: :obj:`'masked'`)
        is_undirected (bool, optional): If :obj:`True` the graph is
            assumed to be undirected. (default: :obj:`True`)
        log (bool, optional): If set to :obj:`False`, will not log any learning
            progress. (default: :obj:`True`)
    """
    coeffs = {'max_trials_sampling': 20, 'eps': 1e-7}

    def __init__(
        self,
        model: torch.nn.Module,
        block_size: int,
        epochs: int = 125,
        loss: Optional[Union[str, LOSS_TYPE]] = 'masked',
        is_undirected: bool = True,
        log: bool = True,
        **kwargs,
    ):
        super().__init__(model, block_size, epochs, loss=loss,
                         is_undirected=is_undirected, log=log, **kwargs)

    @torch.no_grad()
    def _prepare(self, budget: int) -> List[int]:
        """Prepare attack."""
        self.flipped_edges = self.edge_index.new_empty(2, 0).to(self.device)

        # Determine the number of edges to be flipped in each attach step/epoch
        step_size = budget // self.epochs
        if step_size > 0:
            steps = self.epochs * [step_size]
            for i in range(budget % self.epochs):
                steps[i] += 1
        else:
            steps = [1] * budget

        # Sample initial search space (Algorithm 2, line 3-4)
        self._sample_random_block(step_size)

        return steps

    @torch.no_grad()
    def _update(self, step_size: int, gradient: Tensor, *args,
                **kwargs) -> Dict[str, Any]:
        """Update edge weights given gradient."""
        _, topk_edge_index = torch.topk(gradient, step_size)

        flip_edge_index = self.block_edge_index[:, topk_edge_index]
        flip_edge_weight = torch.ones_like(flip_edge_index[0],
                                           dtype=torch.float32)

        self.flipped_edges = torch.cat((self.flipped_edges, flip_edge_index),
                                       axis=-1)

        if self.is_undirected:
            flip_edge_index, flip_edge_weight = to_undirected(
                flip_edge_index, flip_edge_weight, num_nodes=self.num_nodes,
                reduce='mean')
        edge_index = torch.cat(
            (self.edge_index.to(self.device), flip_edge_index.to(self.device)),
            dim=-1)
        edge_weight = torch.cat((self.edge_weight.to(self.device),
                                 flip_edge_weight.to(self.device)))
        edge_index, edge_weight = coalesce(edge_index, edge_weight,
                                           num_nodes=self.num_nodes,
                                           reduce='sum')

        is_one_mask = torch.isclose(edge_weight, torch.tensor(1.))
        self.edge_index = edge_index[:, is_one_mask]
        self.edge_weight = edge_weight[is_one_mask]
        # self.edge_weight = torch.ones_like(self.edge_weight)
        assert self.edge_index.size(1) == self.edge_weight.size(0)

        # Sample initial search space (Algorithm 2, line 3-4)
        self._sample_random_block(step_size)

        # Return debug information
        scalars = {
            'number_positive_entries_in_gradient': (gradient > 0).sum().item()
        }
        return scalars

    def _close(self, *args, **kwargs) -> Tuple[Tensor, Tensor]:
        """Clean up and prepare return argument."""
        return self.edge_index, self.flipped_edges
pytorch/torch_geometric/contrib/transforms/000077500000000000000000000000001444262304300215635ustar00rootroot00000000000000pytorch/torch_geometric/contrib/transforms/__init__.py000066400000000000000000000000271444262304300236730ustar00rootroot00000000000000__all__ = classes = []
pytorch/torch_geometric/data/000077500000000000000000000000001444262304300166365ustar00rootroot00000000000000pytorch/torch_geometric/data/__init__.py000066400000000000000000000064671444262304300207640ustar00rootroot00000000000000# flake8: noqa

from .feature_store import FeatureStore, TensorAttr
from .graph_store import GraphStore, EdgeAttr
from .data import Data
from .hetero_data import HeteroData
from .batch import Batch
from .temporal import TemporalData
from .dataset import Dataset
from .in_memory_dataset import InMemoryDataset
from .makedirs import makedirs
from .download import download_url
from .extract import extract_tar, extract_zip, extract_bz2, extract_gz

from torch_geometric.lazy_loader import LazyLoader

data_classes = [
    'Data',
    'HeteroData',
    'Batch',
    'TemporalData',
    'Dataset',
    'InMemoryDataset',
]

remote_backend_classes = [
    'FeatureStore',
    'GraphStore',
    'TensorAttr',
    'EdgeAttr',
]

helper_functions = [
    'makedirs',
    'download_url',
    'extract_tar',
    'extract_zip',
    'extract_bz2',
    'extract_gz',
]

__all__ = data_classes + remote_backend_classes + helper_functions

lightning = LazyLoader('lightning', globals(),
                       'torch_geometric.data.lightning')

from torch_geometric.deprecation import deprecated
from torch_geometric.loader import NeighborSampler
from torch_geometric.loader import ClusterData
from torch_geometric.loader import ClusterLoader
from torch_geometric.loader import GraphSAINTSampler
from torch_geometric.loader import GraphSAINTNodeSampler
from torch_geometric.loader import GraphSAINTEdgeSampler
from torch_geometric.loader import GraphSAINTRandomWalkSampler
from torch_geometric.loader import ShaDowKHopSampler
from torch_geometric.loader import RandomNodeLoader
from torch_geometric.loader import DataLoader
from torch_geometric.loader import DataListLoader
from torch_geometric.loader import DenseDataLoader

NeighborSampler = deprecated("use 'loader.NeighborSampler' instead",
                             'data.NeighborSampler')(NeighborSampler)
ClusterData = deprecated("use 'loader.ClusterData' instead",
                         'data.ClusterData')(ClusterData)
ClusterLoader = deprecated("use 'loader.ClusterLoader' instead",
                           'data.ClusterLoader')(ClusterLoader)
GraphSAINTSampler = deprecated("use 'loader.GraphSAINTSampler' instead",
                               'data.GraphSAINTSampler')(GraphSAINTSampler)
GraphSAINTNodeSampler = deprecated(
    "use 'loader.GraphSAINTNodeSampler' instead",
    'data.GraphSAINTNodeSampler')(GraphSAINTNodeSampler)
GraphSAINTEdgeSampler = deprecated(
    "use 'loader.GraphSAINTEdgeSampler' instead",
    'data.GraphSAINTEdgeSampler')(GraphSAINTEdgeSampler)
GraphSAINTRandomWalkSampler = deprecated(
    "use 'loader.GraphSAINTRandomWalkSampler' instead",
    'data.GraphSAINTRandomWalkSampler')(GraphSAINTRandomWalkSampler)
ShaDowKHopSampler = deprecated("use 'loader.ShaDowKHopSampler' instead",
                               'data.ShaDowKHopSampler')(ShaDowKHopSampler)
RandomNodeSampler = deprecated("use 'loader.RandomNodeLoader' instead",
                               'data.RandomNodeSampler')(RandomNodeLoader)
DataLoader = deprecated("use 'loader.DataLoader' instead",
                        'data.DataLoader')(DataLoader)
DataListLoader = deprecated("use 'loader.DataListLoader' instead",
                            'data.DataListLoader')(DataListLoader)
DenseDataLoader = deprecated("use 'loader.DenseDataLoader' instead",
                             'data.DenseDataLoader')(DenseDataLoader)
pytorch/torch_geometric/data/batch.py000066400000000000000000000164041444262304300202760ustar00rootroot00000000000000import inspect
from collections.abc import Sequence
from typing import Any, List, Optional, Union

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data.collate import collate
from torch_geometric.data.data import BaseData, Data
from torch_geometric.data.dataset import IndexType
from torch_geometric.data.separate import separate


class DynamicInheritance(type):
    # A meta class that sets the base class of a `Batch` object, e.g.:
    # * `Batch(Data)` in case `Data` objects are batched together
    # * `Batch(HeteroData)` in case `HeteroData` objects are batched together
    def __call__(cls, *args, **kwargs):
        base_cls = kwargs.pop('_base_cls', Data)

        if issubclass(base_cls, Batch):
            new_cls = base_cls
        else:
            name = f'{base_cls.__name__}{cls.__name__}'

            # NOTE `MetaResolver` is necessary to resolve metaclass conflict
            # problems between `DynamicInheritance` and the metaclass of
            # `base_cls`. In particular, it creates a new common metaclass
            # from the defined metaclasses.
            class MetaResolver(type(cls), type(base_cls)):
                pass

            if name not in globals():
                globals()[name] = MetaResolver(name, (cls, base_cls), {})
            new_cls = globals()[name]

        params = list(inspect.signature(base_cls.__init__).parameters.items())
        for i, (k, v) in enumerate(params[1:]):
            if k == 'args' or k == 'kwargs':
                continue
            if i < len(args) or k in kwargs:
                continue
            if v.default is not inspect.Parameter.empty:
                continue
            kwargs[k] = None

        return super(DynamicInheritance, new_cls).__call__(*args, **kwargs)


class DynamicInheritanceGetter:
    def __call__(self, cls, base_cls):
        return cls(_base_cls=base_cls)


class Batch(metaclass=DynamicInheritance):
    r"""A data object describing a batch of graphs as one big (disconnected)
    graph.
    Inherits from :class:`torch_geometric.data.Data` or
    :class:`torch_geometric.data.HeteroData`.
    In addition, single graphs can be identified via the assignment vector
    :obj:`batch`, which maps each node to its respective graph identifier.
    """
    @classmethod
    def from_data_list(cls, data_list: List[BaseData],
                       follow_batch: Optional[List[str]] = None,
                       exclude_keys: Optional[List[str]] = None):
        r"""Constructs a :class:`~torch_geometric.data.Batch` object from a
        Python list of :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` objects.
        The assignment vector :obj:`batch` is created on the fly.
        In addition, creates assignment vectors for each key in
        :obj:`follow_batch`.
        Will exclude any keys given in :obj:`exclude_keys`."""

        batch, slice_dict, inc_dict = collate(
            cls,
            data_list=data_list,
            increment=True,
            add_batch=not isinstance(data_list[0], Batch),
            follow_batch=follow_batch,
            exclude_keys=exclude_keys,
        )

        batch._num_graphs = len(data_list)
        batch._slice_dict = slice_dict
        batch._inc_dict = inc_dict

        return batch

    def get_example(self, idx: int) -> BaseData:
        r"""Gets the :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` object at index :obj:`idx`.
        The :class:`~torch_geometric.data.Batch` object must have been created
        via :meth:`from_data_list` in order to be able to reconstruct the
        initial object."""

        if not hasattr(self, '_slice_dict'):
            raise RuntimeError(
                ("Cannot reconstruct 'Data' object from 'Batch' because "
                 "'Batch' was not created via 'Batch.from_data_list()'"))

        data = separate(
            cls=self.__class__.__bases__[-1],
            batch=self,
            idx=idx,
            slice_dict=self._slice_dict,
            inc_dict=self._inc_dict,
            decrement=True,
        )

        return data

    def index_select(self, idx: IndexType) -> List[BaseData]:
        r"""Creates a subset of :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` objects from specified
        indices :obj:`idx`.
        Indices :obj:`idx` can be a slicing object, *e.g.*, :obj:`[2:5]`, a
        list, a tuple, or a :obj:`torch.Tensor` or :obj:`np.ndarray` of type
        long or bool.
        The :class:`~torch_geometric.data.Batch` object must have been created
        via :meth:`from_data_list` in order to be able to reconstruct the
        initial objects."""
        if isinstance(idx, slice):
            idx = list(range(self.num_graphs)[idx])

        elif isinstance(idx, Tensor) and idx.dtype == torch.long:
            idx = idx.flatten().tolist()

        elif isinstance(idx, Tensor) and idx.dtype == torch.bool:
            idx = idx.flatten().nonzero(as_tuple=False).flatten().tolist()

        elif isinstance(idx, np.ndarray) and idx.dtype == np.int64:
            idx = idx.flatten().tolist()

        elif isinstance(idx, np.ndarray) and idx.dtype == bool:
            idx = idx.flatten().nonzero()[0].flatten().tolist()

        elif isinstance(idx, Sequence) and not isinstance(idx, str):
            pass

        else:
            raise IndexError(
                f"Only slices (':'), list, tuples, torch.tensor and "
                f"np.ndarray of dtype long or bool are valid indices (got "
                f"'{type(idx).__name__}')")

        return [self.get_example(i) for i in idx]

    def __getitem__(self, idx: Union[int, np.integer, str, IndexType]) -> Any:
        if (isinstance(idx, (int, np.integer))
                or (isinstance(idx, Tensor) and idx.dim() == 0)
                or (isinstance(idx, np.ndarray) and np.isscalar(idx))):
            return self.get_example(idx)
        elif isinstance(idx, str) or (isinstance(idx, tuple)
                                      and isinstance(idx[0], str)):
            # Accessing attributes or node/edge types:
            return super().__getitem__(idx)
        else:
            return self.index_select(idx)

    def to_data_list(self) -> List[BaseData]:
        r"""Reconstructs the list of :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` objects from the
        :class:`~torch_geometric.data.Batch` object.
        The :class:`~torch_geometric.data.Batch` object must have been created
        via :meth:`from_data_list` in order to be able to reconstruct the
        initial objects."""
        return [self.get_example(i) for i in range(self.num_graphs)]

    @property
    def num_graphs(self) -> int:
        """Returns the number of graphs in the batch."""
        if hasattr(self, '_num_graphs'):
            return self._num_graphs
        elif hasattr(self, 'ptr'):
            return self.ptr.numel() - 1
        elif hasattr(self, 'batch'):
            return int(self.batch.max()) + 1
        else:
            raise ValueError("Can not infer the number of graphs")

    def __len__(self) -> int:
        return self.num_graphs

    def __reduce__(self):
        state = self.__dict__.copy()
        return DynamicInheritanceGetter(), self.__class__.__bases__, state
pytorch/torch_geometric/data/collate.py000066400000000000000000000246421444262304300206430ustar00rootroot00000000000000from collections import defaultdict
from collections.abc import Mapping, Sequence
from typing import Any, List, Optional, Tuple, Union

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data.data import BaseData
from torch_geometric.data.storage import BaseStorage, NodeStorage
from torch_geometric.typing import SparseTensor, torch_sparse
from torch_geometric.utils import is_sparse, is_torch_sparse_tensor
from torch_geometric.utils.sparse import cat


def collate(
    cls,
    data_list: List[BaseData],
    increment: bool = True,
    add_batch: bool = True,
    follow_batch: Optional[List[str]] = None,
    exclude_keys: Optional[List[str]] = None,
) -> Tuple[BaseData, Mapping, Mapping]:
    # Collates a list of `data` objects into a single object of type `cls`.
    # `collate` can handle both homogeneous and heterogeneous data objects by
    # individually collating all their stores.
    # In addition, `collate` can handle nested data structures such as
    # dictionaries and lists.

    if not isinstance(data_list, (list, tuple)):
        # Materialize `data_list` to keep the `_parent` weakref alive.
        data_list = list(data_list)

    if cls != data_list[0].__class__:
        out = cls(_base_cls=data_list[0].__class__)  # Dynamic inheritance.
    else:
        out = cls()

    # Create empty stores:
    out.stores_as(data_list[0])

    follow_batch = set(follow_batch or [])
    exclude_keys = set(exclude_keys or [])

    # Group all storage objects of every data object in the `data_list` by key,
    # i.e. `key_to_store_list = { key: [store_1, store_2, ...], ... }`:
    key_to_stores = defaultdict(list)
    for data in data_list:
        for store in data.stores:
            key_to_stores[store._key].append(store)

    # With this, we iterate over each list of storage objects and recursively
    # collate all its attributes into a unified representation:

    # We maintain two additional dictionaries:
    # * `slice_dict` stores a compressed index representation of each attribute
    #    and is needed to re-construct individual elements from mini-batches.
    # * `inc_dict` stores how individual elements need to be incremented, e.g.,
    #   `edge_index` is incremented by the cumulated sum of previous elements.
    #   We also need to make use of `inc_dict` when re-constructuing individual
    #   elements as attributes that got incremented need to be decremented
    #   while separating to obtain original values.
    device = None
    slice_dict, inc_dict = defaultdict(dict), defaultdict(dict)
    for out_store in out.stores:
        key = out_store._key
        stores = key_to_stores[key]
        for attr in stores[0].keys():

            if attr in exclude_keys:  # Do not include top-level attribute.
                continue

            values = [store[attr] for store in stores]

            # The `num_nodes` attribute needs special treatment, as we need to
            # sum their values up instead of merging them to a list:
            if attr == 'num_nodes':
                out_store._num_nodes = values
                out_store.num_nodes = sum(values)
                continue

            # Skip batching of `ptr` vectors for now:
            if attr == 'ptr':
                continue

            # Collate attributes into a unified representation:
            value, slices, incs = _collate(attr, values, data_list, stores,
                                           increment)

            if isinstance(value, Tensor) and value.is_cuda:
                device = value.device

            out_store[attr] = value
            if key is not None:
                slice_dict[key][attr] = slices
                inc_dict[key][attr] = incs
            else:
                slice_dict[attr] = slices
                inc_dict[attr] = incs

            # Add an additional batch vector for the given attribute:
            if attr in follow_batch:
                batch, ptr = _batch_and_ptr(slices, device)
                out_store[f'{attr}_batch'] = batch
                out_store[f'{attr}_ptr'] = ptr

        # In case the storage holds node, we add a top-level batch vector it:
        if (add_batch and isinstance(stores[0], NodeStorage)
                and stores[0].can_infer_num_nodes):
            repeats = [store.num_nodes for store in stores]
            out_store.batch = repeat_interleave(repeats, device=device)
            out_store.ptr = cumsum(torch.tensor(repeats, device=device))

    return out, slice_dict, inc_dict


def _collate(
    key: str,
    values: List[Any],
    data_list: List[BaseData],
    stores: List[BaseStorage],
    increment: bool,
) -> Tuple[Any, Any, Any]:

    elem = values[0]

    if isinstance(elem, Tensor) and not is_sparse(elem):
        # Concatenate a list of `torch.Tensor` along the `cat_dim`.
        # NOTE: We need to take care of incrementing elements appropriately.
        key = str(key)
        cat_dim = data_list[0].__cat_dim__(key, elem, stores[0])
        if cat_dim is None or elem.dim() == 0:
            values = [value.unsqueeze(0) for value in values]
        slices = cumsum([value.size(cat_dim or 0) for value in values])
        if increment:
            incs = get_incs(key, values, data_list, stores)
            if incs.dim() > 1 or int(incs[-1]) != 0:
                values = [
                    value + inc.to(value.device)
                    for value, inc in zip(values, incs)
                ]
        else:
            incs = None

        if torch.utils.data.get_worker_info() is not None:
            # Write directly into shared memory to avoid an extra copy:
            numel = sum(value.numel() for value in values)
            if torch_geometric.typing.WITH_PT2:
                storage = elem.untyped_storage()._new_shared(
                    numel * elem.element_size(), device=elem.device)
            else:
                storage = elem.storage()._new_shared(numel, device=elem.device)
            shape = list(elem.size())
            if cat_dim is None or elem.dim() == 0:
                shape = [len(values)] + shape
            else:
                shape[cat_dim] = int(slices[-1])
            out = elem.new(storage).resize_(*shape)
        else:
            out = None

        value = torch.cat(values, dim=cat_dim or 0, out=out)
        return value, slices, incs

    elif is_sparse(elem) and increment:
        # Concatenate a list of `SparseTensor` along the `cat_dim`.
        # NOTE: `cat_dim` may return a tuple to allow for diagonal stacking.
        key = str(key)
        cat_dim = data_list[0].__cat_dim__(key, elem, stores[0])
        cat_dims = (cat_dim, ) if isinstance(cat_dim, int) else cat_dim
        repeats = [[value.size(dim) for dim in cat_dims] for value in values]
        slices = cumsum(repeats)
        if is_torch_sparse_tensor(elem):
            value = cat(values, dim=cat_dim)
        else:
            value = torch_sparse.cat(values, dim=cat_dim)
        return value, slices, None

    elif isinstance(elem, (int, float)):
        # Convert a list of numerical values to a `torch.Tensor`.
        value = torch.tensor(values)
        if increment:
            incs = get_incs(key, values, data_list, stores)
            if int(incs[-1]) != 0:
                value.add_(incs)
        else:
            incs = None
        slices = torch.arange(len(values) + 1)
        return value, slices, incs

    elif isinstance(elem, Mapping):
        # Recursively collate elements of dictionaries.
        value_dict, slice_dict, inc_dict = {}, {}, {}
        for key in elem.keys():
            value_dict[key], slice_dict[key], inc_dict[key] = _collate(
                key, [v[key] for v in values], data_list, stores, increment)
        return value_dict, slice_dict, inc_dict

    elif (isinstance(elem, Sequence) and not isinstance(elem, str)
          and len(elem) > 0 and isinstance(elem[0], (Tensor, SparseTensor))):
        # Recursively collate elements of lists.
        value_list, slice_list, inc_list = [], [], []
        for i in range(len(elem)):
            value, slices, incs = _collate(key, [v[i] for v in values],
                                           data_list, stores, increment)
            value_list.append(value)
            slice_list.append(slices)
            inc_list.append(incs)
        return value_list, slice_list, inc_list

    else:
        # Other-wise, just return the list of values as it is.
        slices = torch.arange(len(values) + 1)
        return values, slices, None


def _batch_and_ptr(
    slices: Any,
    device: Optional[torch.device] = None,
) -> Tuple[Any, Any]:
    if (isinstance(slices, Tensor) and slices.dim() == 1):
        # Default case, turn slices tensor into batch.
        repeats = slices[1:] - slices[:-1]
        batch = repeat_interleave(repeats.tolist(), device=device)
        ptr = cumsum(repeats.to(device))
        return batch, ptr

    elif isinstance(slices, Mapping):
        # Recursively batch elements of dictionaries.
        batch, ptr = {}, {}
        for k, v in slices.items():
            batch[k], ptr[k] = _batch_and_ptr(v, device)
        return batch, ptr

    elif (isinstance(slices, Sequence) and not isinstance(slices, str)
          and isinstance(slices[0], Tensor)):
        # Recursively batch elements of lists.
        batch, ptr = [], []
        for s in slices:
            sub_batch, sub_ptr = _batch_and_ptr(s, device)
            batch.append(sub_batch)
            ptr.append(sub_ptr)
        return batch, ptr

    else:
        # Failure of batching, usually due to slices.dim() != 1
        return None, None


###############################################################################


def repeat_interleave(
    repeats: List[int],
    device: Optional[torch.device] = None,
) -> Tensor:
    outs = [torch.full((n, ), i, device=device) for i, n in enumerate(repeats)]
    return torch.cat(outs, dim=0)


def cumsum(value: Union[Tensor, List[int]]) -> Tensor:
    if not isinstance(value, Tensor):
        value = torch.tensor(value)
    out = value.new_empty((value.size(0) + 1, ) + value.size()[1:])
    out[0] = 0
    torch.cumsum(value, 0, out=out[1:])
    return out


def get_incs(key, values: List[Any], data_list: List[BaseData],
             stores: List[BaseStorage]) -> Tensor:
    repeats = [
        data.__inc__(key, value, store)
        for value, data, store in zip(values, data_list, stores)
    ]
    if isinstance(repeats[0], Tensor):
        repeats = torch.stack(repeats, dim=0)
    else:
        repeats = torch.tensor(repeats)
    return cumsum(repeats[:-1])
pytorch/torch_geometric/data/data.py000066400000000000000000001111711444262304300201230ustar00rootroot00000000000000import copy
import warnings
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from itertools import chain
from typing import (
    Any,
    Callable,
    Dict,
    Iterable,
    List,
    NamedTuple,
    Optional,
    Tuple,
    Union,
)

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data import EdgeAttr, FeatureStore, GraphStore, TensorAttr
from torch_geometric.data.feature_store import _field_status
from torch_geometric.data.graph_store import EdgeLayout
from torch_geometric.data.storage import (
    BaseStorage,
    EdgeStorage,
    GlobalStorage,
    NodeStorage,
)
from torch_geometric.deprecation import deprecated
from torch_geometric.typing import (
    EdgeTensorType,
    EdgeType,
    FeatureTensorType,
    NodeType,
    OptTensor,
    SparseTensor,
)
from torch_geometric.utils import is_sparse, select, subgraph


class BaseData:
    def __getattr__(self, key: str) -> Any:
        raise NotImplementedError

    def __setattr__(self, key: str, value: Any):
        raise NotImplementedError

    def __delattr__(self, key: str):
        raise NotImplementedError

    def __getitem__(self, key: str) -> Any:
        raise NotImplementedError

    def __setitem__(self, key: str, value: Any):
        raise NotImplementedError

    def __delitem__(self, key: str):
        raise NotImplementedError

    def __copy__(self):
        raise NotImplementedError

    def __deepcopy__(self, memo):
        raise NotImplementedError

    def __repr__(self) -> str:
        raise NotImplementedError

    def stores_as(self, data: 'BaseData'):
        raise NotImplementedError

    @property
    def stores(self) -> List[BaseStorage]:
        raise NotImplementedError

    @property
    def node_stores(self) -> List[NodeStorage]:
        raise NotImplementedError

    @property
    def edge_stores(self) -> List[EdgeStorage]:
        raise NotImplementedError

    def to_dict(self) -> Dict[str, Any]:
        r"""Returns a dictionary of stored key/value pairs."""
        raise NotImplementedError

    def to_namedtuple(self) -> NamedTuple:
        r"""Returns a :obj:`NamedTuple` of stored key/value pairs."""
        raise NotImplementedError

    def update(self, data: 'BaseData') -> 'BaseData':
        r"""Updates the data object with the elements from another data object.
        """
        raise NotImplementedError

    def __cat_dim__(self, key: str, value: Any, *args, **kwargs) -> Any:
        r"""Returns the dimension for which the value :obj:`value` of the
        attribute :obj:`key` will get concatenated when creating mini-batches
        using :class:`torch_geometric.loader.DataLoader`.

        .. note::

            This method is for internal use only, and should only be overridden
            in case the mini-batch creation process is corrupted for a specific
            attribute.
        """
        raise NotImplementedError

    def __inc__(self, key: str, value: Any, *args, **kwargs) -> Any:
        r"""Returns the incremental count to cumulatively increase the value
        :obj:`value` of the attribute :obj:`key` when creating mini-batches
        using :class:`torch_geometric.loader.DataLoader`.

        .. note::

            This method is for internal use only, and should only be overridden
            in case the mini-batch creation process is corrupted for a specific
            attribute.
        """
        raise NotImplementedError

    def debug(self):
        raise NotImplementedError

    ###########################################################################

    @property
    def keys(self) -> List[str]:
        r"""Returns a list of all graph attribute names."""
        out = []
        for store in self.stores:
            out += list(store.keys())
        return list(set(out))

    def __len__(self) -> int:
        r"""Returns the number of graph attributes."""
        return len(self.keys)

    def __contains__(self, key: str) -> bool:
        r"""Returns :obj:`True` if the attribute :obj:`key` is present in the
        data."""
        return key in self.keys

    def __getstate__(self) -> Dict[str, Any]:
        return self.__dict__

    def __setstate__(self, mapping: Dict[str, Any]):
        for key, value in mapping.items():
            self.__dict__[key] = value

    @property
    def num_nodes(self) -> Optional[int]:
        r"""Returns the number of nodes in the graph.

        .. note::
            The number of nodes in the data object is automatically inferred
            in case node-level attributes are present, *e.g.*, :obj:`data.x`.
            In some cases, however, a graph may only be given without any
            node-level attributes.
            :pyg:`PyG` then *guesses* the number of nodes according to
            :obj:`edge_index.max().item() + 1`.
            However, in case there exists isolated nodes, this number does not
            have to be correct which can result in unexpected behavior.
            Thus, we recommend to set the number of nodes in your data object
            explicitly via :obj:`data.num_nodes = ...`.
            You will be given a warning that requests you to do so.
        """
        try:
            return sum([v.num_nodes for v in self.node_stores])
        except TypeError:
            return None

    def size(
        self, dim: Optional[int] = None
    ) -> Union[Tuple[Optional[int], Optional[int]], Optional[int]]:
        r"""Returns the size of the adjacency matrix induced by the graph."""
        size = (self.num_nodes, self.num_nodes)
        return size if dim is None else size[dim]

    @property
    def num_edges(self) -> int:
        r"""Returns the number of edges in the graph.
        For undirected graphs, this will return the number of bi-directional
        edges, which is double the amount of unique edges."""
        return sum([v.num_edges for v in self.edge_stores])

    def node_attrs(self) -> List[str]:
        r"""Returns all node-level tensor attribute names."""
        return list(set(chain(*[s.node_attrs() for s in self.node_stores])))

    def edge_attrs(self) -> List[str]:
        r"""Returns all edge-level tensor attribute names."""
        return list(set(chain(*[s.edge_attrs() for s in self.edge_stores])))

    def is_coalesced(self) -> bool:
        r"""Returns :obj:`True` if edge indices :obj:`edge_index` are sorted
        and do not contain duplicate entries."""
        return all([store.is_coalesced() for store in self.edge_stores])

    def generate_ids(self):
        r"""Generates and sets :obj:`n_id` and :obj:`e_id` attributes to assign
        each node and edge to a continuously ascending and unique ID."""
        for store in self.node_stores:
            store.n_id = torch.arange(store.num_nodes)
        for store in self.edge_stores:
            store.e_id = torch.arange(store.num_edges)

    def coalesce(self):
        r"""Sorts and removes duplicated entries from edge indices
        :obj:`edge_index`."""
        for store in self.edge_stores:
            store.coalesce()
        return self

    def has_isolated_nodes(self) -> bool:
        r"""Returns :obj:`True` if the graph contains isolated nodes."""
        return any([store.has_isolated_nodes() for store in self.edge_stores])

    def has_self_loops(self) -> bool:
        """Returns :obj:`True` if the graph contains self-loops."""
        return any([store.has_self_loops() for store in self.edge_stores])

    def is_undirected(self) -> bool:
        r"""Returns :obj:`True` if graph edges are undirected."""
        return all([store.is_undirected() for store in self.edge_stores])

    def is_directed(self) -> bool:
        r"""Returns :obj:`True` if graph edges are directed."""
        return not self.is_undirected()

    def apply_(self, func: Callable, *args: List[str]):
        r"""Applies the in-place function :obj:`func`, either to all attributes
        or only the ones given in :obj:`*args`."""
        for store in self.stores:
            store.apply_(func, *args)
        return self

    def apply(self, func: Callable, *args: List[str]):
        r"""Applies the function :obj:`func`, either to all attributes or only
        the ones given in :obj:`*args`."""
        for store in self.stores:
            store.apply(func, *args)
        return self

    def clone(self, *args: List[str]):
        r"""Performs cloning of tensors, either for all attributes or only the
        ones given in :obj:`*args`."""
        return copy.copy(self).apply(lambda x: x.clone(), *args)

    def contiguous(self, *args: List[str]):
        r"""Ensures a contiguous memory layout, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.contiguous(), *args)

    def to(self, device: Union[int, str], *args: List[str],
           non_blocking: bool = False):
        r"""Performs tensor device conversion, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(
            lambda x: x.to(device=device, non_blocking=non_blocking), *args)

    def cpu(self, *args: List[str]):
        r"""Copies attributes to CPU memory, either for all attributes or only
        the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.cpu(), *args)

    def cuda(self, device: Optional[Union[int, str]] = None, *args: List[str],
             non_blocking: bool = False):
        r"""Copies attributes to CUDA memory, either for all attributes or only
        the ones given in :obj:`*args`."""
        # Some PyTorch tensor like objects require a default value for `cuda`:
        device = 'cuda' if device is None else device
        return self.apply(lambda x: x.cuda(device, non_blocking=non_blocking),
                          *args)

    def pin_memory(self, *args: List[str]):
        r"""Copies attributes to pinned memory, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.pin_memory(), *args)

    def share_memory_(self, *args: List[str]):
        r"""Moves attributes to shared memory, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply_(lambda x: x.share_memory_(), *args)

    def detach_(self, *args: List[str]):
        r"""Detaches attributes from the computation graph, either for all
        attributes or only the ones given in :obj:`*args`."""
        return self.apply_(lambda x: x.detach_(), *args)

    def detach(self, *args: List[str]):
        r"""Detaches attributes from the computation graph by creating a new
        tensor, either for all attributes or only the ones given in
        :obj:`*args`."""
        return self.apply(lambda x: x.detach(), *args)

    def requires_grad_(self, *args: List[str], requires_grad: bool = True):
        r"""Tracks gradient computation, either for all attributes or only the
        ones given in :obj:`*args`."""
        return self.apply_(
            lambda x: x.requires_grad_(requires_grad=requires_grad), *args)

    def record_stream(self, stream: torch.cuda.Stream, *args: List[str]):
        r"""Ensures that the tensor memory is not reused for another tensor
        until all current work queued on :obj:`stream` has been completed,
        either for all attributes or only the ones given in :obj:`*args`."""
        return self.apply_(lambda x: x.record_stream(stream), *args)

    @property
    def is_cuda(self) -> bool:
        r"""Returns :obj:`True` if any :class:`torch.Tensor` attribute is
        stored on the GPU, :obj:`False` otherwise."""
        for store in self.stores:
            for value in store.values():
                if isinstance(value, Tensor) and value.is_cuda:
                    return True
        return False

    # Deprecated functions ####################################################

    @deprecated(details="use 'has_isolated_nodes' instead")
    def contains_isolated_nodes(self) -> bool:
        return self.has_isolated_nodes()

    @deprecated(details="use 'has_self_loops' instead")
    def contains_self_loops(self) -> bool:
        return self.has_self_loops()


###############################################################################


@dataclass
class DataTensorAttr(TensorAttr):
    r"""Tensor attribute for `Data` without group name."""
    def __init__(
        self,
        attr_name=_field_status.UNSET,
        index=None,
    ):
        super().__init__(None, attr_name, index)


@dataclass
class DataEdgeAttr(EdgeAttr):
    r"""Edge attribute class for `Data` without edge type."""
    def __init__(
        self,
        layout: Optional[EdgeLayout] = None,
        is_sorted: bool = False,
        size: Optional[Tuple[int, int]] = None,
    ):
        super().__init__(None, layout, is_sorted, size)


###############################################################################


class Data(BaseData, FeatureStore, GraphStore):
    r"""A data object describing a homogeneous graph.
    The data object can hold node-level, link-level and graph-level attributes.
    In general, :class:`~torch_geometric.data.Data` tries to mimic the
    behavior of a regular Python dictionary.
    In addition, it provides useful functionality for analyzing graph
    structures, and provides basic PyTorch tensor functionalities.
    See `here `__ for the accompanying
    tutorial.

    .. code-block:: python

        from torch_geometric.data import Data

        data = Data(x=x, edge_index=edge_index, ...)

        # Add additional arguments to `data`:
        data.train_idx = torch.tensor([...], dtype=torch.long)
        data.test_mask = torch.tensor([...], dtype=torch.bool)

        # Analyzing the graph structure:
        data.num_nodes
        >>> 23

        data.is_directed()
        >>> False

        # PyTorch tensor functionality:
        data = data.pin_memory()
        data = data.to('cuda:0', non_blocking=True)

    Args:
        x (torch.Tensor, optional): Node feature matrix with shape
            :obj:`[num_nodes, num_node_features]`. (default: :obj:`None`)
        edge_index (LongTensor, optional): Graph connectivity in COO format
            with shape :obj:`[2, num_edges]`. (default: :obj:`None`)
        edge_attr (torch.Tensor, optional): Edge feature matrix with shape
            :obj:`[num_edges, num_edge_features]`. (default: :obj:`None`)
        y (torch.Tensor, optional): Graph-level or node-level ground-truth
            labels with arbitrary shape. (default: :obj:`None`)
        pos (torch.Tensor, optional): Node position matrix with shape
            :obj:`[num_nodes, num_dimensions]`. (default: :obj:`None`)
        **kwargs (optional): Additional attributes.
    """
    def __init__(self, x: OptTensor = None, edge_index: OptTensor = None,
                 edge_attr: OptTensor = None, y: OptTensor = None,
                 pos: OptTensor = None, **kwargs):
        # `Data` doesn't support group_name, so we need to adjust `TensorAttr`
        # accordingly here to avoid requiring `group_name` to be set:
        super().__init__(tensor_attr_cls=DataTensorAttr)

        # `Data` doesn't support edge_type, so we need to adjust `EdgeAttr`
        # accordingly here to avoid requiring `edge_type` to be set:
        GraphStore.__init__(self, edge_attr_cls=DataEdgeAttr)

        self.__dict__['_store'] = GlobalStorage(_parent=self)

        if x is not None:
            self.x = x
        if edge_index is not None:
            self.edge_index = edge_index
        if edge_attr is not None:
            self.edge_attr = edge_attr
        if y is not None:
            self.y = y
        if pos is not None:
            self.pos = pos

        for key, value in kwargs.items():
            setattr(self, key, value)

    def __getattr__(self, key: str) -> Any:
        if '_store' not in self.__dict__:
            raise RuntimeError(
                "The 'data' object was created by an older version of PyG. "
                "If this error occurred while loading an already existing "
                "dataset, remove the 'processed/' directory in the dataset's "
                "root folder and try again.")
        return getattr(self._store, key)

    def __setattr__(self, key: str, value: Any):
        propobj = getattr(self.__class__, key, None)
        if propobj is not None and getattr(propobj, 'fset', None) is not None:
            propobj.fset(self, value)
        else:
            setattr(self._store, key, value)

    def __delattr__(self, key: str):
        delattr(self._store, key)

    # TODO consider supporting the feature store interface for
    # __getitem__, __setitem__, and __delitem__ so, for example, we
    # can accept key: Union[str, TensorAttr] in __getitem__.
    def __getitem__(self, key: str) -> Any:
        return self._store[key]

    def __setitem__(self, key: str, value: Any):
        self._store[key] = value

    def __delitem__(self, key: str):
        if key in self._store:
            del self._store[key]

    def __copy__(self):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out.__dict__['_store'] = copy.copy(self._store)
        out._store._parent = out
        return out

    def __deepcopy__(self, memo):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = copy.deepcopy(value, memo)
        out._store._parent = out
        return out

    def __repr__(self) -> str:
        cls = self.__class__.__name__
        has_dict = any([isinstance(v, Mapping) for v in self._store.values()])

        if not has_dict:
            info = [size_repr(k, v) for k, v in self._store.items()]
            info = ', '.join(info)
            return f'{cls}({info})'
        else:
            info = [size_repr(k, v, indent=2) for k, v in self._store.items()]
            info = ',\n'.join(info)
            return f'{cls}(\n{info}\n)'

    def stores_as(self, data: 'Data'):
        return self

    @property
    def stores(self) -> List[BaseStorage]:
        return [self._store]

    @property
    def node_stores(self) -> List[NodeStorage]:
        return [self._store]

    @property
    def edge_stores(self) -> List[EdgeStorage]:
        return [self._store]

    def to_dict(self) -> Dict[str, Any]:
        return self._store.to_dict()

    def to_namedtuple(self) -> NamedTuple:
        return self._store.to_namedtuple()

    def update(self, data: Union['Data', Dict[str, Any]]) -> 'Data':
        for key, value in data.items():
            self[key] = value
        return self

    def __cat_dim__(self, key: str, value: Any, *args, **kwargs) -> Any:
        if is_sparse(value) and 'adj' in key:
            return (0, 1)
        elif 'index' in key or key == 'face':
            return -1
        else:
            return 0

    def __inc__(self, key: str, value: Any, *args, **kwargs) -> Any:
        if 'batch' in key:
            return int(value.max()) + 1
        elif 'index' in key or key == 'face':
            return self.num_nodes
        else:
            return 0

    def validate(self, raise_on_error: bool = True) -> bool:
        r"""Validates the correctness of the data."""
        cls_name = self.__class__.__name__
        status = True

        num_nodes = self.num_nodes
        if num_nodes is None:
            status = False
            warn_or_raise(f"'num_nodes' is undefined in '{cls_name}'",
                          raise_on_error)

        if 'edge_index' in self:
            if self.edge_index.dim() != 2 or self.edge_index.size(0) != 2:
                status = False
                warn_or_raise(
                    f"'edge_index' needs to be of shape [2, num_edges] in "
                    f"'{cls_name}' (found {self.edge_index.size()})",
                    raise_on_error)

        if 'edge_index' in self and self.edge_index.numel() > 0:
            if self.edge_index.min() < 0:
                status = False
                warn_or_raise(
                    f"'edge_index' contains negative indices in "
                    f"'{cls_name}' (found {int(self.edge_index.min())})",
                    raise_on_error)

            if num_nodes is not None and self.edge_index.max() >= num_nodes:
                status = False
                warn_or_raise(
                    f"'edge_index' contains larger indices than the number "
                    f"of nodes ({num_nodes}) in '{cls_name}' "
                    f"(found {int(self.edge_index.max())})", raise_on_error)

        return status

    def debug(self):
        pass  # TODO

    def is_node_attr(self, key: str) -> bool:
        r"""Returns :obj:`True` if the object at key :obj:`key` denotes a
        node-level tensor attribute."""
        return self._store.is_node_attr(key)

    def is_edge_attr(self, key: str) -> bool:
        r"""Returns :obj:`True` if the object at key :obj:`key` denotes an
        edge-level tensor attribute."""
        return self._store.is_edge_attr(key)

    def subgraph(self, subset: Tensor) -> 'Data':
        r"""Returns the induced subgraph given by the node indices
        :obj:`subset`.

        Args:
            subset (LongTensor or BoolTensor): The nodes to keep.
        """
        out = subgraph(subset, self.edge_index, relabel_nodes=True,
                       num_nodes=self.num_nodes, return_edge_mask=True)
        edge_index, _, edge_mask = out

        data = copy.copy(self)

        for key, value in self:
            if key == 'edge_index':
                data.edge_index = edge_index
            elif key == 'num_nodes':
                if subset.dtype == torch.bool:
                    data.num_nodes = int(subset.sum())
                else:
                    data.num_nodes = subset.size(0)
            elif self.is_node_attr(key):
                cat_dim = self.__cat_dim__(key, value)
                data[key] = select(value, subset, dim=cat_dim)
            elif self.is_edge_attr(key):
                cat_dim = self.__cat_dim__(key, value)
                data[key] = select(value, edge_mask, dim=cat_dim)

        return data

    def edge_subgraph(self, subset: Tensor) -> 'Data':
        r"""Returns the induced subgraph given by the edge indices
        :obj:`subset`.
        Will currently preserve all the nodes in the graph, even if they are
        isolated after subgraph computation.

        Args:
            subset (LongTensor or BoolTensor): The edges to keep.
        """
        data = copy.copy(self)

        for key, value in self:
            if self.is_edge_attr(key):
                cat_dim = self.__cat_dim__(key, value)
                data[key] = select(value, subset, dim=cat_dim)

        return data

    def to_heterogeneous(
        self,
        node_type: Optional[Tensor] = None,
        edge_type: Optional[Tensor] = None,
        node_type_names: Optional[List[NodeType]] = None,
        edge_type_names: Optional[List[EdgeType]] = None,
    ):
        r"""Converts a :class:`~torch_geometric.data.Data` object to a
        heterogeneous :class:`~torch_geometric.data.HeteroData` object.
        For this, node and edge attributes are splitted according to the
        node-level and edge-level vectors :obj:`node_type` and
        :obj:`edge_type`, respectively.
        :obj:`node_type_names` and :obj:`edge_type_names` can be used to give
        meaningful node and edge type names, respectively.
        That is, the node_type :obj:`0` is given by :obj:`node_type_names[0]`.
        If the :class:`~torch_geometric.data.Data` object was constructed via
        :meth:`~torch_geometric.data.HeteroData.to_homogeneous`, the object can
        be reconstructed without any need to pass in additional arguments.

        Args:
            node_type (torch.Tensor, optional): A node-level vector denoting
                the type of each node. (default: :obj:`None`)
            edge_type (torch.Tensor, optional): An edge-level vector denoting
                the type of each edge. (default: :obj:`None`)
            node_type_names (List[str], optional): The names of node types.
                (default: :obj:`None`)
            edge_type_names (List[Tuple[str, str, str]], optional): The names
                of edge types. (default: :obj:`None`)
        """
        from torch_geometric.data import HeteroData

        if node_type is None:
            node_type = self._store.get('node_type', None)
        if node_type is None:
            node_type = torch.zeros(self.num_nodes, dtype=torch.long)

        if node_type_names is None:
            store = self._store
            node_type_names = store.__dict__.get('_node_type_names', None)
        if node_type_names is None:
            node_type_names = [str(i) for i in node_type.unique().tolist()]

        if edge_type is None:
            edge_type = self._store.get('edge_type', None)
        if edge_type is None:
            edge_type = torch.zeros(self.num_edges, dtype=torch.long)

        if edge_type_names is None:
            store = self._store
            edge_type_names = store.__dict__.get('_edge_type_names', None)
        if edge_type_names is None:
            edge_type_names = []
            edge_index = self.edge_index
            for i in edge_type.unique().tolist():
                src, dst = edge_index[:, edge_type == i]
                src_types = node_type[src].unique().tolist()
                dst_types = node_type[dst].unique().tolist()
                if len(src_types) != 1 and len(dst_types) != 1:
                    raise ValueError(
                        "Could not construct a 'HeteroData' object from the "
                        "'Data' object because single edge types span over "
                        "multiple node types")
                edge_type_names.append((node_type_names[src_types[0]], str(i),
                                        node_type_names[dst_types[0]]))

        # We iterate over node types to find the local node indices belonging
        # to each node type. Furthermore, we create a global `index_map` vector
        # that maps global node indices to local ones in the final
        # heterogeneous graph:
        node_ids, index_map = {}, torch.empty_like(node_type)
        for i, key in enumerate(node_type_names):
            node_ids[i] = (node_type == i).nonzero(as_tuple=False).view(-1)
            index_map[node_ids[i]] = torch.arange(len(node_ids[i]),
                                                  device=index_map.device)

        # We iterate over edge types to find the local edge indices:
        edge_ids = {}
        for i, key in enumerate(edge_type_names):
            edge_ids[i] = (edge_type == i).nonzero(as_tuple=False).view(-1)

        data = HeteroData()

        for i, key in enumerate(node_type_names):
            for attr, value in self.items():
                if attr in {'node_type', 'edge_type', 'ptr'}:
                    continue
                elif isinstance(value, Tensor) and self.is_node_attr(attr):
                    cat_dim = self.__cat_dim__(attr, value)
                    data[key][attr] = value.index_select(cat_dim, node_ids[i])

            if len(data[key]) == 0:
                data[key].num_nodes = node_ids[i].size(0)

        for i, key in enumerate(edge_type_names):
            src, _, dst = key
            for attr, value in self.items():
                if attr in {'node_type', 'edge_type', 'ptr'}:
                    continue
                elif attr == 'edge_index':
                    edge_index = value[:, edge_ids[i]]
                    edge_index[0] = index_map[edge_index[0]]
                    edge_index[1] = index_map[edge_index[1]]
                    data[key].edge_index = edge_index
                elif isinstance(value, Tensor) and self.is_edge_attr(attr):
                    cat_dim = self.__cat_dim__(attr, value)
                    data[key][attr] = value.index_select(cat_dim, edge_ids[i])

        # Add global attributes.
        exclude_keys = set(data.keys) | {
            'node_type', 'edge_type', 'edge_index', 'num_nodes', 'ptr'
        }
        for attr, value in self.items():
            if attr in exclude_keys:
                continue
            data[attr] = value

        return data

    ###########################################################################

    @classmethod
    def from_dict(cls, mapping: Dict[str, Any]) -> 'Data':
        r"""Creates a :class:`~torch_geometric.data.Data` object from a Python
        dictionary."""
        return cls(**mapping)

    @property
    def num_node_features(self) -> int:
        r"""Returns the number of features per node in the graph."""
        return self._store.num_node_features

    @property
    def num_features(self) -> int:
        r"""Returns the number of features per node in the graph.
        Alias for :py:attr:`~num_node_features`."""
        return self.num_node_features

    @property
    def num_edge_features(self) -> int:
        r"""Returns the number of features per edge in the graph."""
        return self._store.num_edge_features

    @property
    def num_node_types(self) -> int:
        r"""Returns the number of node types in the graph."""
        return int(self.node_type.max()) + 1 if 'node_type' in self else 1

    @property
    def num_edge_types(self) -> int:
        r"""Returns the number of edge types in the graph."""
        return int(self.edge_type.max()) + 1 if 'edge_type' in self else 1

    def __iter__(self) -> Iterable:
        r"""Iterates over all attributes in the data, yielding their attribute
        names and values."""
        for key, value in self._store.items():
            yield key, value

    def __call__(self, *args: List[str]) -> Iterable:
        r"""Iterates over all attributes :obj:`*args` in the data, yielding
        their attribute names and values.
        If :obj:`*args` is not given, will iterate over all attributes."""
        for key, value in self._store.items(*args):
            yield key, value

    @property
    def x(self) -> Any:
        return self['x'] if 'x' in self._store else None

    @property
    def edge_index(self) -> Any:
        return self['edge_index'] if 'edge_index' in self._store else None

    @property
    def edge_weight(self) -> Any:
        return self['edge_weight'] if 'edge_weight' in self._store else None

    @property
    def edge_attr(self) -> Any:
        return self['edge_attr'] if 'edge_attr' in self._store else None

    @property
    def y(self) -> Any:
        return self['y'] if 'y' in self._store else None

    @property
    def pos(self) -> Any:
        return self['pos'] if 'pos' in self._store else None

    @property
    def batch(self) -> Any:
        return self['batch'] if 'batch' in self._store else None

    # Deprecated functions ####################################################

    @property
    @deprecated(details="use 'data.face.size(-1)' instead")
    def num_faces(self) -> Optional[int]:
        r"""Returns the number of faces in the mesh."""
        if 'face' in self._store and isinstance(self.face, Tensor):
            return self.face.size(self.__cat_dim__('face', self.face))
        return None

    # FeatureStore interface ##################################################

    def _put_tensor(self, tensor: FeatureTensorType, attr: TensorAttr) -> bool:
        out = self.get(attr.attr_name)
        if out is not None and attr.index is not None:
            out[attr.index] = tensor
        else:
            assert attr.index is None
            setattr(self, attr.attr_name, tensor)
        return True

    def _get_tensor(self, attr: TensorAttr) -> Optional[FeatureTensorType]:
        tensor = getattr(self, attr.attr_name, None)
        if tensor is not None:
            # TODO this behavior is a bit odd, since TensorAttr requires that
            # we set `index`. So, we assume here that indexing by `None` is
            # equivalent to not indexing at all, which is not in line with
            # Python semantics.
            return tensor[attr.index] if attr.index is not None else tensor
        return None

    def _remove_tensor(self, attr: TensorAttr) -> bool:
        if hasattr(self, attr.attr_name):
            delattr(self, attr.attr_name)
            return True
        return False

    def _get_tensor_size(self, attr: TensorAttr) -> Tuple:
        return self._get_tensor(attr).size()

    def get_all_tensor_attrs(self) -> List[TensorAttr]:
        r"""Obtains all feature attributes stored in `Data`."""
        return [
            TensorAttr(attr_name=name) for name in self._store.keys()
            if self._store.is_node_attr(name)
        ]

    # GraphStore interface ####################################################

    def _put_edge_index(self, edge_index: EdgeTensorType,
                        edge_attr: EdgeAttr) -> bool:
        if not hasattr(self, '_edge_attrs'):
            self._edge_attrs = {}
        self._edge_attrs[edge_attr.layout] = edge_attr

        row, col = edge_index

        if edge_attr.layout == EdgeLayout.COO:
            self.edge_index = torch.stack([row, col], dim=0)
        elif edge_attr.layout == EdgeLayout.CSR:
            self.adj = SparseTensor(
                rowptr=row,
                col=col,
                sparse_sizes=edge_attr.size,
                is_sorted=True,
                trust_data=True,
            )
        else:  # edge_attr.layout == EdgeLayout.CSC:
            size = edge_attr.size[::-1] if edge_attr.size is not None else None
            self.adj_t = SparseTensor(
                rowptr=col,
                col=row,
                sparse_sizes=size,
                is_sorted=True,
                trust_data=True,
            )
        return True

    def _get_edge_index(self, edge_attr: EdgeAttr) -> Optional[EdgeTensorType]:
        if edge_attr.size is None:
            edge_attr.size = self.size()  # Modify in-place.

        if edge_attr.layout == EdgeLayout.COO and 'edge_index' in self:
            row, col = self.edge_index
            return row, col
        elif edge_attr.layout == EdgeLayout.CSR and 'adj' in self:
            rowptr, col, _ = self.adj.csr()
            return rowptr, col
        elif edge_attr.layout == EdgeLayout.CSC and 'adj_t' in self:
            colptr, row, _ = self.adj_t.csr()
            return row, colptr
        return None

    def _remove_edge_index(self, edge_attr: EdgeAttr) -> bool:
        if edge_attr.layout == EdgeLayout.COO and 'edge_index' in self:
            del self.edge_index
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop(EdgeLayout.COO, None)
            return True
        elif edge_attr.layout == EdgeLayout.CSR and 'adj' in self:
            del self.adj
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop(EdgeLayout.CSR, None)
            return True
        elif edge_attr.layout == EdgeLayout.CSC and 'adj_t' in self:
            del self.adj_t
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop(EdgeLayout.CSC, None)
            return True
        return False

    def get_all_edge_attrs(self) -> List[EdgeAttr]:
        edge_attrs = getattr(self, '_edge_attrs', {})

        if 'edge_index' in self and EdgeLayout.COO not in edge_attrs:
            edge_attrs[EdgeLayout.COO] = DataEdgeAttr('coo', is_sorted=False)
        if 'adj' in self and EdgeLayout.CSR not in edge_attrs:
            size = self.adj.sparse_sizes()
            edge_attrs[EdgeLayout.CSR] = DataEdgeAttr('csr', size=size)
        if 'adj_t' in self and EdgeLayout.CSC not in edge_attrs:
            size = self.adj_t.sparse_sizes()[::-1]
            edge_attrs[EdgeLayout.CSC] = DataEdgeAttr('csc', size=size)

        return list(edge_attrs.values())


###############################################################################


def size_repr(key: Any, value: Any, indent: int = 0) -> str:
    pad = ' ' * indent
    if isinstance(value, Tensor) and value.dim() == 0:
        out = value.item()
    elif isinstance(value, Tensor):
        out = str(list(value.size()))
    elif isinstance(value, np.ndarray):
        out = str(list(value.shape))
    elif isinstance(value, SparseTensor):
        out = str(value.sizes())[:-1] + f', nnz={value.nnz()}]'
    elif isinstance(value, str):
        out = f"'{value}'"
    elif isinstance(value, Sequence):
        out = str([len(value)])
    elif isinstance(value, Mapping) and len(value) == 0:
        out = '{}'
    elif (isinstance(value, Mapping) and len(value) == 1
          and not isinstance(list(value.values())[0], Mapping)):
        lines = [size_repr(k, v, 0) for k, v in value.items()]
        out = '{ ' + ', '.join(lines) + ' }'
    elif isinstance(value, Mapping):
        lines = [size_repr(k, v, indent + 2) for k, v in value.items()]
        out = '{\n' + ',\n'.join(lines) + ',\n' + pad + '}'
    else:
        out = str(value)

    key = str(key).replace("'", '')
    return f'{pad}{key}={out}'


def warn_or_raise(msg: str, raise_on_error: bool = True):
    if raise_on_error:
        raise ValueError(msg)
    else:
        warnings.warn(msg)
pytorch/torch_geometric/data/datapipes.py000066400000000000000000000055521444262304300211710ustar00rootroot00000000000000import copy
from typing import Any, Callable, Iterator, Optional, Sequence

import torch

from torch_geometric.data import Batch
from torch_geometric.utils import from_smiles

try:
    from torch.utils.data import IterDataPipe, functional_datapipe
    from torch.utils.data.datapipes.iter import Batcher as IterBatcher
except ImportError:
    IterDataPipe = IterBatcher = object

    def functional_datapipe(name: str) -> Callable:
        return lambda cls: cls


@functional_datapipe('batch_graphs')
class Batcher(IterBatcher):
    def __init__(
        self,
        dp: IterDataPipe,
        batch_size: int,
        drop_last: bool = False,
    ):
        super().__init__(
            dp,
            batch_size=batch_size,
            drop_last=drop_last,
            wrapper_class=Batch.from_data_list,
        )


@functional_datapipe('parse_smiles')
class SMILESParser(IterDataPipe):
    def __init__(
        self,
        dp: IterDataPipe,
        smiles_key: str = 'smiles',
        target_key: Optional[str] = None,
    ):
        super().__init__()
        self.dp = dp
        self.smiles_key = smiles_key
        self.target_key = target_key

    def __iter__(self) -> Iterator:
        for d in self.dp:
            if isinstance(d, str):
                data = from_smiles(d)
            elif isinstance(d, dict):
                data = from_smiles(d[self.smiles_key])
                if self.target_key is not None:
                    y = d.get(self.target_key, None)
                    if y is not None:
                        y = float(y) if len(y) > 0 else float('NaN')
                        data.y = torch.tensor([y], dtype=torch.float)
            else:
                raise ValueError(
                    f"'{self.__class__.__name__}' expected either a string or "
                    f"a dict as input (got '{type(d)}')")

            yield data


class DatasetAdapter(IterDataPipe):
    def __init__(self, dataset: Sequence[Any]):
        super().__init__()
        self.dataset = dataset
        self.range = range(len(self))

    def is_shardable(self) -> bool:
        return True

    def apply_sharding(self, num_shards: int, shard_idx: int):
        self.range = range(shard_idx, len(self), num_shards)

    def __iter__(self) -> Iterator:
        for i in self.range:
            yield self.dataset[i]

    def __len__(self) -> int:
        return len(self.dataset)


def functional_transform(name: str) -> Callable:
    def wrapper(cls: Any) -> Any:
        @functional_datapipe(name)
        class DynamicMapper(IterDataPipe):
            def __init__(self, dp: IterDataPipe, *args, **kwargs):
                super().__init__()
                self.dp = dp
                self.fn = cls(*args, **kwargs)

            def __iter__(self) -> Iterator:
                for data in self.dp:
                    yield self.fn(copy.copy(data))

        return cls

    return wrapper
pytorch/torch_geometric/data/dataset.py000066400000000000000000000353131444262304300206420ustar00rootroot00000000000000import copy
import os.path as osp
import re
import sys
import warnings
from abc import ABC, abstractmethod
from collections.abc import Sequence
from typing import Any, Callable, List, Optional, Tuple, Union

import numpy as np
import torch.utils.data
from torch import Tensor

from torch_geometric.data.data import BaseData
from torch_geometric.data.makedirs import makedirs

IndexType = Union[slice, Tensor, np.ndarray, Sequence]


class Dataset(torch.utils.data.Dataset, ABC):
    r"""Dataset base class for creating graph datasets.
    See `here `__ for the accompanying tutorial.

    Args:
        root (str, optional): Root directory where the dataset should be saved.
            (optional: :obj:`None`)
        transform (callable, optional): A function/transform that takes in a
            :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            transformed version.
            The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            a :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            transformed version.
            The data object will be transformed before being saved to disk.
            (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in a
            :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            boolean value, indicating whether the data object should be
            included in the final dataset. (default: :obj:`None`)
        log (bool, optional): Whether to print any console output while
            downloading and processing the dataset. (default: :obj:`True`)
    """
    @property
    def raw_file_names(self) -> Union[str, List[str], Tuple]:
        r"""The name of the files in the :obj:`self.raw_dir` folder that must
        be present in order to skip downloading."""
        raise NotImplementedError

    @property
    def processed_file_names(self) -> Union[str, List[str], Tuple]:
        r"""The name of the files in the :obj:`self.processed_dir` folder that
        must be present in order to skip processing."""
        raise NotImplementedError

    def download(self):
        r"""Downloads the dataset to the :obj:`self.raw_dir` folder."""
        raise NotImplementedError

    def process(self):
        r"""Processes the dataset to the :obj:`self.processed_dir` folder."""
        raise NotImplementedError

    @abstractmethod
    def len(self) -> int:
        r"""Returns the number of graphs stored in the dataset."""
        raise NotImplementedError

    @abstractmethod
    def get(self, idx: int) -> BaseData:
        r"""Gets the data object at index :obj:`idx`."""
        raise NotImplementedError

    def __init__(
        self,
        root: Optional[str] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
        log: bool = True,
    ):
        super().__init__()

        if isinstance(root, str):
            root = osp.expanduser(osp.normpath(root))

        self.root = root
        self.transform = transform
        self.pre_transform = pre_transform
        self.pre_filter = pre_filter
        self.log = log
        self._indices: Optional[Sequence] = None

        if self.has_download:
            self._download()

        if self.has_process:
            self._process()

    def indices(self) -> Sequence:
        return range(self.len()) if self._indices is None else self._indices

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, 'processed')

    @property
    def num_node_features(self) -> int:
        r"""Returns the number of features per node in the dataset."""
        data = self[0]
        # Do not fill cache for `InMemoryDataset`:
        if hasattr(self, '_data_list') and self._data_list is not None:
            self._data_list[0] = None
        data = data[0] if isinstance(data, tuple) else data
        if hasattr(data, 'num_node_features'):
            return data.num_node_features
        raise AttributeError(f"'{data.__class__.__name__}' object has no "
                             f"attribute 'num_node_features'")

    @property
    def num_features(self) -> int:
        r"""Returns the number of features per node in the dataset.
        Alias for :py:attr:`~num_node_features`."""
        return self.num_node_features

    @property
    def num_edge_features(self) -> int:
        r"""Returns the number of features per edge in the dataset."""
        data = self[0]
        # Do not fill cache for `InMemoryDataset`:
        if hasattr(self, '_data_list') and self._data_list is not None:
            self._data_list[0] = None
        data = data[0] if isinstance(data, tuple) else data
        if hasattr(data, 'num_edge_features'):
            return data.num_edge_features
        raise AttributeError(f"'{data.__class__.__name__}' object has no "
                             f"attribute 'num_edge_features'")

    def _infer_num_classes(self, y: Optional[Tensor]) -> int:
        if y is None:
            return 0
        elif y.numel() == y.size(0) and not torch.is_floating_point(y):
            return int(y.max()) + 1
        elif y.numel() == y.size(0) and torch.is_floating_point(y):
            return torch.unique(y).numel()
        else:
            return y.size(-1)

    @property
    def num_classes(self) -> int:
        r"""Returns the number of classes in the dataset."""
        # We iterate over the dataset and collect all labels to determine the
        # maximum number of classes. Importantly, in rare cases, `__getitem__`
        # may produce a tuple of data objects (e.g., when used in combination
        # with `RandomLinkSplit`, so we take care of this case here as well:
        data_list = _get_flattened_data_list([data for data in self])
        y = torch.cat([data.y for data in data_list if 'y' in data], dim=0)

        # Do not fill cache for `InMemoryDataset`:
        if hasattr(self, '_data_list') and self._data_list is not None:
            self._data_list = self.len() * [None]
        return self._infer_num_classes(y)

    @property
    def raw_paths(self) -> List[str]:
        r"""The absolute filepaths that must be present in order to skip
        downloading."""
        files = self.raw_file_names
        # Prevent a common source of error in which `file_names` are not
        # defined as a property.
        if isinstance(files, Callable):
            files = files()
        return [osp.join(self.raw_dir, f) for f in to_list(files)]

    @property
    def processed_paths(self) -> List[str]:
        r"""The absolute filepaths that must be present in order to skip
        processing."""
        files = self.processed_file_names
        # Prevent a common source of error in which `file_names` are not
        # defined as a property.
        if isinstance(files, Callable):
            files = files()
        return [osp.join(self.processed_dir, f) for f in to_list(files)]

    @property
    def has_download(self) -> bool:
        r"""Checks whether the dataset defines a :meth:`download` method."""
        return overrides_method(self.__class__, 'download')

    def _download(self):
        if files_exist(self.raw_paths):  # pragma: no cover
            return

        makedirs(self.raw_dir)
        self.download()

    @property
    def has_process(self) -> bool:
        r"""Checks whether the dataset defines a :meth:`process` method."""
        return overrides_method(self.__class__, 'process')

    def _process(self):
        f = osp.join(self.processed_dir, 'pre_transform.pt')
        if osp.exists(f) and torch.load(f) != _repr(self.pre_transform):
            warnings.warn(
                f"The `pre_transform` argument differs from the one used in "
                f"the pre-processed version of this dataset. If you want to "
                f"make use of another pre-processing technique, make sure to "
                f"delete '{self.processed_dir}' first")

        f = osp.join(self.processed_dir, 'pre_filter.pt')
        if osp.exists(f) and torch.load(f) != _repr(self.pre_filter):
            warnings.warn(
                "The `pre_filter` argument differs from the one used in "
                "the pre-processed version of this dataset. If you want to "
                "make use of another pre-fitering technique, make sure to "
                "delete '{self.processed_dir}' first")

        if files_exist(self.processed_paths):  # pragma: no cover
            return

        if self.log and 'pytest' not in sys.modules:
            print('Processing...', file=sys.stderr)

        makedirs(self.processed_dir)
        self.process()

        path = osp.join(self.processed_dir, 'pre_transform.pt')
        torch.save(_repr(self.pre_transform), path)
        path = osp.join(self.processed_dir, 'pre_filter.pt')
        torch.save(_repr(self.pre_filter), path)

        if self.log and 'pytest' not in sys.modules:
            print('Done!', file=sys.stderr)

    def __len__(self) -> int:
        r"""The number of examples in the dataset."""
        return len(self.indices())

    def __getitem__(
        self,
        idx: Union[int, np.integer, IndexType],
    ) -> Union['Dataset', BaseData]:
        r"""In case :obj:`idx` is of type integer, will return the data object
        at index :obj:`idx` (and transforms it in case :obj:`transform` is
        present).
        In case :obj:`idx` is a slicing object, *e.g.*, :obj:`[2:5]`, a list, a
        tuple, or a :obj:`torch.Tensor` or :obj:`np.ndarray` of type long or
        bool, will return a subset of the dataset at the specified indices."""
        if (isinstance(idx, (int, np.integer))
                or (isinstance(idx, Tensor) and idx.dim() == 0)
                or (isinstance(idx, np.ndarray) and np.isscalar(idx))):

            data = self.get(self.indices()[idx])
            data = data if self.transform is None else self.transform(data)
            return data

        else:
            return self.index_select(idx)

    def index_select(self, idx: IndexType) -> 'Dataset':
        r"""Creates a subset of the dataset from specified indices :obj:`idx`.
        Indices :obj:`idx` can be a slicing object, *e.g.*, :obj:`[2:5]`, a
        list, a tuple, or a :obj:`torch.Tensor` or :obj:`np.ndarray` of type
        long or bool."""
        indices = self.indices()

        if isinstance(idx, slice):
            indices = indices[idx]

        elif isinstance(idx, Tensor) and idx.dtype == torch.long:
            return self.index_select(idx.flatten().tolist())

        elif isinstance(idx, Tensor) and idx.dtype == torch.bool:
            idx = idx.flatten().nonzero(as_tuple=False)
            return self.index_select(idx.flatten().tolist())

        elif isinstance(idx, np.ndarray) and idx.dtype == np.int64:
            return self.index_select(idx.flatten().tolist())

        elif isinstance(idx, np.ndarray) and idx.dtype == bool:
            idx = idx.flatten().nonzero()[0]
            return self.index_select(idx.flatten().tolist())

        elif isinstance(idx, Sequence) and not isinstance(idx, str):
            indices = [indices[i] for i in idx]

        else:
            raise IndexError(
                f"Only slices (':'), list, tuples, torch.tensor and "
                f"np.ndarray of dtype long or bool are valid indices (got "
                f"'{type(idx).__name__}')")

        dataset = copy.copy(self)
        dataset._indices = indices
        return dataset

    def shuffle(
        self,
        return_perm: bool = False,
    ) -> Union['Dataset', Tuple['Dataset', Tensor]]:
        r"""Randomly shuffles the examples in the dataset.

        Args:
            return_perm (bool, optional): If set to :obj:`True`, will also
                return the random permutation used to shuffle the dataset.
                (default: :obj:`False`)
        """
        perm = torch.randperm(len(self))
        dataset = self.index_select(perm)
        return (dataset, perm) if return_perm is True else dataset

    def __repr__(self) -> str:
        arg_repr = str(len(self)) if len(self) > 1 else ''
        return f'{self.__class__.__name__}({arg_repr})'

    def get_summary(self):
        r"""Collects summary statistics for the dataset."""
        from torch_geometric.data.summary import Summary
        return Summary.from_dataset(self)

    def print_summary(self):  # pragma: no cover
        r"""Prints summary statistics of the dataset to the console."""
        print(str(self.get_summary()))

    def to_datapipe(self):
        r"""Converts the dataset into a :class:`torch.utils.data.DataPipe`.

        The returned instance can then be used with :pyg:`PyG's` built-in
        :class:`DataPipes` for baching graphs as follows:

        .. code-block:: python

            from torch_geometric.datasets import QM9

            dp = QM9(root='./data/QM9/').to_datapipe()
            dp = dp.batch_graphs(batch_size=2, drop_last=True)

            for batch in dp:
                pass

        See the `PyTorch tutorial
        `_ for further background
        on DataPipes.
        """
        from torch_geometric.data.datapipes import DatasetAdapter

        return DatasetAdapter(self)


def overrides_method(cls, method_name: str):
    from torch_geometric.data import InMemoryDataset

    if method_name in cls.__dict__:
        return True

    out = False
    for base in cls.__bases__:
        if base != Dataset and base != InMemoryDataset:
            out |= overrides_method(base, method_name)
    return out


def to_list(value: Any) -> Sequence:
    if isinstance(value, Sequence) and not isinstance(value, str):
        return value
    else:
        return [value]


def files_exist(files: List[str]) -> bool:
    # NOTE: We return `False` in case `files` is empty, leading to a
    # re-processing of files on every instantiation.
    return len(files) != 0 and all([osp.exists(f) for f in files])


def _repr(obj: Any) -> str:
    if obj is None:
        return 'None'
    return re.sub('(<.*?)\\s.*(>)', r'\1\2', str(obj))


def _get_flattened_data_list(data_list: List[Any]) -> List[BaseData]:
    outs: List[BaseData] = []
    for data in data_list:
        if isinstance(data, BaseData):
            outs.append(data)
        elif isinstance(data, (tuple, list)):
            outs.extend(_get_flattened_data_list(data))
        elif isinstance(data, dict):
            outs.extend(_get_flattened_data_list(data.values()))
    return outs
pytorch/torch_geometric/data/download.py000066400000000000000000000025171444262304300210240ustar00rootroot00000000000000import os.path as osp
import ssl
import sys
import urllib
from typing import Optional

from torch_geometric.data.makedirs import makedirs


def download_url(url: str, folder: str, log: bool = True,
                 filename: Optional[str] = None):
    r"""Downloads the content of an URL to a specific folder.

    Args:
        url (str): The URL.
        folder (str): The folder.
        log (bool, optional): If :obj:`False`, will not print anything to the
            console. (default: :obj:`True`)
    """

    if filename is None:
        filename = url.rpartition('/')[2]
        filename = filename if filename[0] == '?' else filename.split('?')[0]

    path = osp.join(folder, filename)

    if osp.exists(path):  # pragma: no cover
        if log and 'pytest' not in sys.modules:
            print(f'Using existing file {filename}', file=sys.stderr)
        return path

    if log and 'pytest' not in sys.modules:
        print(f'Downloading {url}', file=sys.stderr)

    makedirs(folder)

    context = ssl._create_unverified_context()
    data = urllib.request.urlopen(url, context=context)

    with open(path, 'wb') as f:
        # workaround for https://bugs.python.org/issue42853
        while True:
            chunk = data.read(10 * 1024 * 1024)
            if not chunk:
                break
            f.write(chunk)

    return path
pytorch/torch_geometric/data/extract.py000066400000000000000000000043141444262304300206640ustar00rootroot00000000000000import bz2
import gzip
import os.path as osp
import sys
import tarfile
import zipfile


def maybe_log(path, log=True):
    if log and 'pytest' not in sys.modules:
        print(f'Extracting {path}', file=sys.stderr)


def extract_tar(path: str, folder: str, mode: str = 'r:gz', log: bool = True):
    r"""Extracts a tar archive to a specific folder.

    Args:
        path (str): The path to the tar archive.
        folder (str): The folder.
        mode (str, optional): The compression mode. (default: :obj:`"r:gz"`)
        log (bool, optional): If :obj:`False`, will not print anything to the
            console. (default: :obj:`True`)
    """
    maybe_log(path, log)
    with tarfile.open(path, mode) as f:
        f.extractall(folder)


def extract_zip(path: str, folder: str, log: bool = True):
    r"""Extracts a zip archive to a specific folder.

    Args:
        path (str): The path to the tar archive.
        folder (str): The folder.
        log (bool, optional): If :obj:`False`, will not print anything to the
            console. (default: :obj:`True`)
    """
    maybe_log(path, log)
    with zipfile.ZipFile(path, 'r') as f:
        f.extractall(folder)


def extract_bz2(path: str, folder: str, log: bool = True):
    r"""Extracts a bz2 archive to a specific folder.

    Args:
        path (str): The path to the tar archive.
        folder (str): The folder.
        log (bool, optional): If :obj:`False`, will not print anything to the
            console. (default: :obj:`True`)
    """
    maybe_log(path, log)
    path = osp.abspath(path)
    with bz2.open(path, 'r') as r:
        with open(osp.join(folder, '.'.join(path.split('.')[:-1])), 'wb') as w:
            w.write(r.read())


def extract_gz(path: str, folder: str, log: bool = True):
    r"""Extracts a gz archive to a specific folder.

    Args:
        path (str): The path to the tar archive.
        folder (str): The folder.
        log (bool, optional): If :obj:`False`, will not print anything to the
            console. (default: :obj:`True`)
    """
    maybe_log(path, log)
    path = osp.abspath(path)
    with gzip.open(path, 'r') as r:
        with open(osp.join(folder, '.'.join(path.split('.')[:-1])), 'wb') as w:
            w.write(r.read())
pytorch/torch_geometric/data/feature_store.py000066400000000000000000000511231444262304300220610ustar00rootroot00000000000000r"""
This class defines the abstraction for a backend-agnostic feature store. The
goal of the feature store is to abstract away all node and edge feature memory
management so that varying implementations can allow for independent scale-out.

This particular feature store abstraction makes a few key assumptions:
* The features we care about storing are node and edge features of a graph.
  To this end, the attributes that the feature store supports include a
  `group_name` (e.g. a heterogeneous node name or a heterogeneous edge type),
  an `attr_name` (e.g. `x` or `edge_attr`), and an index.
* A feature can be uniquely identified from any associated attributes specified
  in `TensorAttr`.

It is the job of a feature store implementor class to handle these assumptions
properly. For example, a simple in-memory feature store implementation may
concatenate all metadata values with a feature index and use this as a unique
index in a KV store. More complicated implementations may choose to partition
features in interesting manners based on the provided metadata.

Major TODOs for future implementation:
* Async `put` and `get` functionality
"""
import copy
from abc import abstractmethod
from dataclasses import dataclass
from enum import Enum
from typing import Any, List, Optional, Tuple, Union

import numpy as np
import torch

from torch_geometric.typing import FeatureTensorType, NodeType
from torch_geometric.utils.mixin import CastMixin

_field_status = Enum("FieldStatus", "UNSET")

# We allow indexing with a tensor, numpy array, Python slicing, or a single
# integer index.
IndexType = Union[torch.Tensor, np.ndarray, slice, int]


@dataclass
class TensorAttr(CastMixin):
    r"""Defines the attributes of a :class:`FeatureStore` tensor.
    It holds all the parameters necessary to uniquely identify a tensor from
    the :class:`FeatureStore`.

    Note that the order of the attributes is important; this is the order in
    which attributes must be provided for indexing calls. :class:`FeatureStore`
    implementations can define a different ordering by overriding
    :meth:`TensorAttr.__init__`.
    """

    # The group name that the tensor corresponds to. Defaults to UNSET.
    group_name: Optional[NodeType] = _field_status.UNSET

    # The name of the tensor within its group. Defaults to UNSET.
    attr_name: Optional[str] = _field_status.UNSET

    # The node indices the rows of the tensor correspond to. Defaults to UNSET.
    index: Optional[IndexType] = _field_status.UNSET

    # Convenience methods #####################################################

    def is_set(self, key: str) -> bool:
        r"""Whether an attribute is set in :obj:`TensorAttr`."""
        assert key in self.__dataclass_fields__
        return getattr(self, key) != _field_status.UNSET

    def is_fully_specified(self) -> bool:
        r"""Whether the :obj:`TensorAttr` has no unset fields."""
        return all([self.is_set(key) for key in self.__dataclass_fields__])

    def fully_specify(self) -> 'TensorAttr':
        r"""Sets all :obj:`UNSET` fields to :obj:`None`."""
        for key in self.__dataclass_fields__:
            if not self.is_set(key):
                setattr(self, key, None)
        return self

    def update(self, attr: 'TensorAttr') -> 'TensorAttr':
        r"""Updates an :class:`TensorAttr` with set attributes from another
        :class:`TensorAttr`."""
        for key in self.__dataclass_fields__:
            if attr.is_set(key):
                setattr(self, key, getattr(attr, key))
        return self


class AttrView(CastMixin):
    r"""Defines a view of a :class:`FeatureStore` that is obtained from a
    specification of attributes on the feature store. The view stores a
    reference to the backing feature store as well as a :class:`TensorAttr`
    object that represents the view's state.

    Users can create views either using the :class:`AttrView` constructor,
    :meth:`FeatureStore.view`, or by incompletely indexing a feature store.
    For example, the following calls all create views:

    .. code-block:: python

        store[group_name]
        store[group_name].feat
        store[group_name, feat]

    While the following calls all materialize those views and produce tensors
    by either calling the view or fully-specifying the view:

    .. code-block:: python

        store[group_name]()
        store[group_name].feat[index]
        store[group_name, feat][index]
    """
    def __init__(self, store: 'FeatureStore', attr: TensorAttr):
        self.__dict__['_store'] = store
        self.__dict__['_attr'] = attr

    # Advanced indexing #######################################################

    def __getattr__(self, key: Any) -> Union['AttrView', FeatureTensorType]:
        r"""Sets the first unset field of the backing :class:`TensorAttr`
        object to the attribute.
        This allows for :class:`AttrView` to be indexed by different values of
        attributes, in order.
        In particular, for a feature store that we want to index by
        :obj:`group_name` and :obj:`attr_name`, the following code will do so:

        .. code-block:: python

            store[group, attr]
            store[group].attr
            store.group.attr
        """
        out = copy.copy(self)

        # Find the first attribute name that is UNSET:
        attr_name: Optional[str] = None
        for field in out._attr.__dataclass_fields__:
            if getattr(out._attr, field) == _field_status.UNSET:
                attr_name = field
                break

        if attr_name is None:
            raise AttributeError(f"Cannot access attribute '{key}' on view "
                                 f"'{out}' as all attributes have already "
                                 f"been set in this view")

        setattr(out._attr, attr_name, key)

        if out._attr.is_fully_specified():
            return out._store.get_tensor(out._attr)

        return out

    def __getitem__(self, key: Any) -> Union['AttrView', FeatureTensorType]:
        r"""Sets the first unset field of the backing :class:`TensorAttr`
        object to the attribute via indexing.
        This allows for :class:`AttrView` to be indexed by different values of
        attributes, in order.
        In particular, for a feature store that we want to index by
        :obj:`group_name` and :obj:`attr_name`, the following code will do so:

        .. code-block:: python

            store[group, attr]
            store[group][attr]

        """
        return self.__getattr__(key)

    # Setting attributes ######################################################

    def __setattr__(self, key: str, value: Any):
        r"""Supports attribute assignment to the backing :class:`TensorAttr` of
        an :class:`AttrView`. This allows for :class:`AttrView` objects to set
        their backing attribute values. In particular, the following operation
        sets the :obj:`index` of an :class:`AttrView`:

        .. code-block:: python

            view = store.view(group_name)
            view.index = torch.tensor([1, 2, 3])
        """
        if key not in self._attr.__dataclass_fields__:
            raise ValueError(f"Attempted to set nonexistent attribute '{key}' "
                             f"(acceptable attributes are "
                             f"{self._attr.__dataclass_fields__})")

        setattr(self._attr, key, value)

    def __setitem__(self, key: str, value: Any):
        r"""Supports attribute assignment to the backing :class:`TensorAttr` of
        an :class:`AttrView` via indexing. This allows for :class:`AttrView`
        objects to set their backing attribute values. In particular, the
        following operation sets the `index` of an :class:`AttrView`:

        .. code-block:: python

            view = store.view(TensorAttr(group_name))
            view['index'] = torch.Tensor([1, 2, 3])
        """
        self.__setattr__(key, value)

    # Miscellaneous built-ins #################################################

    def __call__(self) -> FeatureTensorType:
        r"""Supports :class:`AttrView` as a callable to force retrieval from
        the currently specified attributes. In particular, this passes the
        current :class:`TensorAttr` object to a GET call, regardless of whether
        all attributes have been specified. It returns the result of this call.
        In particular, the following operation returns a tensor by performing a
        GET operation on the backing feature store:

        .. code-block:: python

            store[group_name, attr_name]()
        """
        # Set all UNSET values to None:
        out = copy.copy(self)
        out._attr.fully_specify()
        return out._store.get_tensor(out._attr)

    def __copy__(self) -> 'AttrView':
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out.__dict__['_attr'] = copy.copy(out.__dict__['_attr'])
        return out

    def __eq__(self, obj: Any) -> bool:
        r"""Compares two :class:`AttrView` objects by checking equality of
        their :class:`FeatureStore` references and :class:`TensorAttr`
        attributes."""
        if not isinstance(obj, AttrView):
            return False
        return self._store == obj._store and self._attr == obj._attr

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(store={self._store}, '
                f'attr={self._attr})')


# TODO (manan, matthias) Ideally, we want to let `FeatureStore` inherit from
# `MutableMapping` to clearly indicate its behavior and usage to the user.
# However, having `MutableMapping` as a base class leads to strange behavior
# in combination with PyTorch and PyTorch Lightning, in particular since these
# libraries use customized logic during mini-batch for `Mapping` base classes.


class FeatureStore:
    r"""An abstract base class to access features from a remote feature store.

    Args:
        tensor_attr_cls (TensorAttr, optional): A user-defined
            :class:`TensorAttr` class to customize the required attributes and
            their ordering to unique identify tensor values.
            (default: :obj:`None`)
    """
    def __init__(self, tensor_attr_cls: Optional[Any] = None):
        super().__init__()
        self.__dict__['_tensor_attr_cls'] = tensor_attr_cls or TensorAttr

    # Core (CRUD) #############################################################

    @abstractmethod
    def _put_tensor(self, tensor: FeatureTensorType, attr: TensorAttr) -> bool:
        r"""To be implemented by :class:`FeatureStore` subclasses."""
        pass

    def put_tensor(self, tensor: FeatureTensorType, *args, **kwargs) -> bool:
        r"""Synchronously adds a :obj:`tensor` to the :class:`FeatureStore`.
        Returns whether insertion was successful.

        Args:
            tensor (torch.Tensor or np.ndarray): The feature tensor to be
                added.
            **kwargs (TensorAttr): Any relevant tensor attributes that
                correspond to the feature tensor. See the :class:`TensorAttr`
                documentation for required and optional attributes.

        Raises:
            ValueError: If the input :class:`TensorAttr` is not fully
                specified.
        """
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        if not attr.is_fully_specified():
            raise ValueError(f"The input TensorAttr '{attr}' is not fully "
                             f"specified. Please fully-specify the input by "
                             f"specifying all 'UNSET' fields")
        return self._put_tensor(tensor, attr)

    @abstractmethod
    def _get_tensor(self, attr: TensorAttr) -> Optional[FeatureTensorType]:
        r"""To be implemented by :class:`FeatureStore` subclasses."""
        pass

    def get_tensor(
        self,
        *args,
        convert_type: bool = False,
        **kwargs,
    ) -> FeatureTensorType:
        r"""Synchronously obtains a :class:`tensor` from the
        :class:`FeatureStore`.

        Args:
            **kwargs (TensorAttr): Any relevant tensor attributes that
                correspond to the feature tensor. See the :class:`TensorAttr`
                documentation for required and optional attributes.
            convert_type (bool, optional): Whether to convert the type of the
                output tensor to the type of the attribute index.
                (default: :obj:`False`)

        Raises:
            ValueError: If the input :class:`TensorAttr` is not fully
                specified.
            KeyError: If the tensor corresponding to the input
                :class:`TensorAttr` was not found.
        """
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        if not attr.is_fully_specified():
            raise ValueError(f"The input TensorAttr '{attr}' is not fully "
                             f"specified. Please fully-specify the input by "
                             f"specifying all 'UNSET' fields.")

        tensor = self._get_tensor(attr)
        if tensor is None:
            raise KeyError(f"A tensor corresponding to '{attr}' was not found")
        return self._to_type(attr, tensor) if convert_type else tensor

    def _multi_get_tensor(
        self,
        attrs: List[TensorAttr],
    ) -> List[Optional[FeatureTensorType]]:
        r"""To be implemented by :class:`FeatureStore` subclasses."""
        return [self._get_tensor(attr) for attr in attrs]

    def multi_get_tensor(
        self,
        attrs: List[TensorAttr],
        convert_type: bool = False,
    ) -> List[FeatureTensorType]:
        r"""Synchronously obtains a list of tensors from the
        :class:`FeatureStore` for each tensor associated with the attributes in
        :obj:`attrs`.

        .. note::
            The default implementation simply iterates over all calls to
            :meth:`get_tensor`. Implementor classes that can provide
            additional, more performant functionality are recommended to
            to override this method.

        Args:
            attrs (List[TensorAttr]): A list of input :class:`TensorAttr`
                objects that identify the tensors to obtain.
            convert_type (bool, optional): Whether to convert the type of the
                output tensor to the type of the attribute index.
                (default: :obj:`False`)

        Raises:
            ValueError: If any input :class:`TensorAttr` is not fully
                specified.
            KeyError: If any of the tensors corresponding to the input
                :class:`TensorAttr` was not found.
        """
        attrs = [self._tensor_attr_cls.cast(attr) for attr in attrs]
        bad_attrs = [attr for attr in attrs if not attr.is_fully_specified()]
        if len(bad_attrs) > 0:
            raise ValueError(
                f"The input TensorAttr(s) '{bad_attrs}' are not fully "
                f"specified. Please fully-specify them by specifying all "
                f"'UNSET' fields")

        tensors = self._multi_get_tensor(attrs)
        if any(v is None for v in tensors):
            bad_attrs = [attrs[i] for i, v in enumerate(tensors) if v is None]
            raise KeyError(f"Tensors corresponding to attributes "
                           f"'{bad_attrs}' were not found")

        return [
            self._to_type(attr, tensor) if convert_type else tensor
            for attr, tensor in zip(attrs, tensors)
        ]

    @abstractmethod
    def _remove_tensor(self, attr: TensorAttr) -> bool:
        r"""To be implemented by :obj:`FeatureStore` subclasses."""
        pass

    def remove_tensor(self, *args, **kwargs) -> bool:
        r"""Removes a tensor from the :class:`FeatureStore`.
        Returns whether deletion was successful.

        Args:
            **kwargs (TensorAttr): Any relevant tensor attributes that
                correspond to the feature tensor. See the :class:`TensorAttr`
                documentation for required and optional attributes.

        Raises:
            ValueError: If the input :class:`TensorAttr` is not fully
                specified.
        """
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        if not attr.is_fully_specified():
            raise ValueError(f"The input TensorAttr '{attr}' is not fully "
                             f"specified. Please fully-specify the input by "
                             f"specifying all 'UNSET' fields.")
        return self._remove_tensor(attr)

    def update_tensor(self, tensor: FeatureTensorType, *args,
                      **kwargs) -> bool:
        r"""Updates a :obj:`tensor` in the :class:`FeatureStore` with a new
        value. Returns whether the update was succesful.

        .. note::
            Implementor classes can choose to define more efficient update
            methods; the default performs a removal and insertion.

        Args:
            tensor (torch.Tensor or np.ndarray): The feature tensor to be
                updated.
            **kwargs (TensorAttr): Any relevant tensor attributes that
                correspond to the feature tensor. See the :class:`TensorAttr`
                documentation for required and optional attributes.
        """
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        self.remove_tensor(attr)
        return self.put_tensor(tensor, attr)

    # Additional methods ######################################################

    @abstractmethod
    def _get_tensor_size(self, attr: TensorAttr) -> Optional[Tuple[int, ...]]:
        pass

    def get_tensor_size(self, *args, **kwargs) -> Optional[Tuple[int, ...]]:
        r"""Obtains the size of a tensor given its :class:`TensorAttr`, or
        :obj:`None` if the tensor does not exist."""
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        if not attr.is_set('index'):
            attr.index = None
        return self._get_tensor_size(attr)

    @abstractmethod
    def get_all_tensor_attrs(self) -> List[TensorAttr]:
        r"""Obtains all tensor attributes stored in this :class:`FeatureStore`.
        """
        pass

    # `AttrView` methods ######################################################

    def view(self, *args, **kwargs) -> AttrView:
        r"""Returns a view of the :class:`FeatureStore` given a not yet
        fully-specified :class:`TensorAttr`."""
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        return AttrView(self, attr)

    # Helper functions ########################################################

    @staticmethod
    def _to_type(
        attr: TensorAttr,
        tensor: FeatureTensorType,
    ) -> FeatureTensorType:
        if (isinstance(attr.index, torch.Tensor)
                and isinstance(tensor, np.ndarray)):
            return torch.from_numpy(tensor)
        if (isinstance(attr.index, np.ndarray)
                and isinstance(tensor, torch.Tensor)):
            return tensor.detach().cpu().numpy()
        return tensor

    # Python built-ins ########################################################

    def __setitem__(self, key: TensorAttr, value: FeatureTensorType):
        r"""Supports :obj:`store[tensor_attr] = tensor`."""
        # CastMixin will handle the case of key being a tuple or TensorAttr
        # object:
        key = self._tensor_attr_cls.cast(key)
        # We need to fully-specify the key for __setitem__ as it does not make
        # sense to work with a view here:
        key.fully_specify()
        self.put_tensor(value, key)

    def __getitem__(self, key: TensorAttr) -> Any:
        r"""Supports pythonic indexing into the :class:`FeatureStore`.
        In particular, the following rules are followed for indexing:

        * A fully-specified :obj:`key` will produce a tensor output.

        * A partially-specified :obj:`key` will produce an :class:`AttrView`
          output, which is a view on the :class:`FeatureStore`. If a view is
          called, it will produce a tensor output from the corresponding
          (partially specified) attributes.
        """
        # CastMixin will handle the case of key being a tuple or TensorAttr:
        attr = self._tensor_attr_cls.cast(key)
        if attr.is_fully_specified():
            return self.get_tensor(attr)
        # If the view is not fully-specified, return a :class:`AttrView`:
        return self.view(attr)

    def __delitem__(self, key: TensorAttr):
        r"""Supports :obj:`del store[tensor_attr]`."""
        # CastMixin will handle the case of key being a tuple or TensorAttr
        # object:
        key = self._tensor_attr_cls.cast(key)
        key.fully_specify()
        self.remove_tensor(key)

    def __iter__(self):
        raise NotImplementedError

    def __eq__(self, obj: object) -> bool:
        return id(self) == id(obj)

    @abstractmethod
    def __len__(self):
        pass

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/data/graph_store.py000066400000000000000000000322671444262304300215370ustar00rootroot00000000000000r"""
This class defines the abstraction for a backend-agnostic graph store. The
goal of the graph store is to abstract away all graph edge index memory
management so that varying implementations can allow for independent scale-out.

This particular graph store abstraction makes a few key assumptions:
* The edge indices we care about storing are represented either in COO, CSC,
  or CSR format. They can be uniquely identified by an edge type (in PyG,
  this is a tuple of the source node, relation type, and destination node).
* Edge indices are static once they are stored in tthe grah. That is, we do not
  support dynamic modification of edge indices once they have been inserted
  into the graph store.

It is the job of a graph store implementor class to handle these assumptions
properly. For example, a simple in-memory graph store implementation may
concatenate all metadata values with an edge index and use this as a unique
index in a KV store. More complicated implementations may choose to partition
the graph in interesting manners based on the provided metadata.
"""
import copy
from abc import abstractmethod
from collections import defaultdict
from dataclasses import dataclass
from enum import Enum
from typing import Any, Dict, List, Optional, Tuple

from torch import Tensor

from torch_geometric.typing import EdgeTensorType, EdgeType, OptTensor
from torch_geometric.utils import index_sort
from torch_geometric.utils.mixin import CastMixin
from torch_geometric.utils.sparse import index2ptr, ptr2index

# The output of converting between two types in the GraphStore is a Tuple of
# dictionaries: row, col, and perm. The dictionaries are keyed by the edge
# type of the input edge attribute.
#   * The row dictionary contains the row tensor for COO, the row pointer for
#     CSR, or the row tensor for CSC
#   * The col dictionary contains the col tensor for COO, the col tensor for
#     CSR, or the col pointer for CSC
#   * The perm dictionary contains the permutation of edges that was applied
#     in converting between formats, if applicable.
ConversionOutputType = Tuple[Dict[EdgeType, Tensor], Dict[EdgeType, Tensor],
                             Dict[EdgeType, OptTensor]]


class EdgeLayout(Enum):
    COO = 'coo'
    CSC = 'csc'
    CSR = 'csr'


@dataclass
class EdgeAttr(CastMixin):
    r"""Defines the attributes of a :obj:`GraphStore` edge.
    It holds all the parameters necessary to uniquely identify an edge from
    the :class:`GraphStore`.

    Note that the order of the attributes is important; this is the order in
    which attributes must be provided for indexing calls. :class:`GraphStore`
    implementations can define a different ordering by overriding
    :meth:`EdgeAttr.__init__`.
    """

    # The type of the edge:
    edge_type: EdgeType

    # The layout of the edge representation:
    layout: EdgeLayout

    # Whether the edge index is sorted by destination node. Useful for
    # avoiding sorting costs when performing neighbor sampling, and only
    # meaningful for COO (CSC is sorted and CSR is not sorted by definition):
    is_sorted: bool = False

    # The number of source and destination nodes in this edge type:
    size: Optional[Tuple[int, int]] = None

    # NOTE we define __init__ to force-cast layout
    def __init__(
        self,
        edge_type: EdgeType,
        layout: EdgeLayout,
        is_sorted: bool = False,
        size: Optional[Tuple[int, int]] = None,
    ):
        layout = EdgeLayout(layout)

        if layout == EdgeLayout.CSR and is_sorted:
            raise ValueError("Cannot create a 'CSR' edge attribute with "
                             "option 'is_sorted=True'")

        if layout == EdgeLayout.CSC:
            is_sorted = True

        self.edge_type = edge_type
        self.layout = layout
        self.is_sorted = is_sorted
        self.size = size


class GraphStore:
    r"""An abstract base class to access edges from a remote graph store.

    Args:
        edge_attr_cls (EdgeAttr, optional): A user-defined
            :class:`EdgeAttr` class to customize the required attributes and
            their ordering to uniquely identify edges. (default: :obj:`None`)
    """
    def __init__(self, edge_attr_cls: Optional[Any] = None):
        super().__init__()
        self.__dict__['_edge_attr_cls'] = edge_attr_cls or EdgeAttr

    # Core (CRUD) #############################################################

    @abstractmethod
    def _put_edge_index(self, edge_index: EdgeTensorType,
                        edge_attr: EdgeAttr) -> bool:
        r"""To be implemented by :class:`GraphStore` subclasses."""
        pass

    def put_edge_index(self, edge_index: EdgeTensorType, *args,
                       **kwargs) -> bool:
        r"""Synchronously adds an :obj:`edge_index` tuple to the
        :class:`GraphStore`.
        Returns whether insertion was successful.

        Args:
            edge_index (Tuple[torch.Tensor, torch.Tensor]): The
                :obj:`edge_index` tuple in a format specified in
                :class:`EdgeAttr`.
            **kwargs (EdgeAttr): Any relevant edge attributes that
                correspond to the :obj:`edge_index` tuple. See the
                :class:`EdgeAttr` documentation for required and optional
                attributes.
        """
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        return self._put_edge_index(edge_index, edge_attr)

    @abstractmethod
    def _get_edge_index(self, edge_attr: EdgeAttr) -> Optional[EdgeTensorType]:
        r"""To be implemented by :class:`GraphStore` subclasses."""
        pass

    def get_edge_index(self, *args, **kwargs) -> EdgeTensorType:
        r"""Synchronously obtains an :obj:`edge_index` tuple from the
        :class:`GraphStore`.

        Args:
            **kwargs (EdgeAttr): Any relevant edge attributes that
                correspond to the :obj:`edge_index` tuple. See the
                :class:`EdgeAttr` documentation for required and optional
                attributes.

        Raises:
            KeyError: If the :obj:`edge_index` corresponding to the input
                :class:`EdgeAttr` was not found.
        """
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        edge_index = self._get_edge_index(edge_attr)
        if edge_index is None:
            raise KeyError(f"'edge_index' for '{edge_attr}' not found")
        return edge_index

    @abstractmethod
    def _remove_edge_index(self, edge_attr: EdgeAttr) -> bool:
        r"""To be implemented by :class:`GraphStore` subclasses."""
        pass

    def remove_edge_index(self, *args, **kwargs) -> bool:
        r"""Synchronously deletes an :obj:`edge_index` tuple from the
        :class:`GraphStore`.
        Returns whether deletion was successful.

        Args:
            **kwargs (EdgeAttr): Any relevant edge attributes that
                correspond to the :obj:`edge_index` tuple. See the
                :class:`EdgeAttr` documentation for required and optional
                attributes.
        """
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        return self._remove_edge_index(edge_attr)

    @abstractmethod
    def get_all_edge_attrs(self) -> List[EdgeAttr]:
        r"""Obtains all edge attributes stored in the :class:`GraphStore`."""
        pass

    # Layout Conversion #######################################################

    def coo(
        self,
        edge_types: Optional[List[Any]] = None,
        store: bool = False,
    ) -> ConversionOutputType:
        r"""Obtains the edge indices in the :class:`GraphStore` in COO
        format.

        Args:
            edge_types (List[Any], optional): The edge types of edge indices
                to obtain. If set to :obj:`None`, will return the edge indices
                of all existing edge types. (default: :obj:`None`)
            store (bool, optional): Whether to store converted edge indices in
                the :class:`GraphStore`. (default: :obj:`False`)
        """
        return self._edges_to_layout(EdgeLayout.COO, edge_types, store)

    def csr(
        self,
        edge_types: Optional[List[Any]] = None,
        store: bool = False,
    ) -> ConversionOutputType:
        r"""Obtains the edge indices in the :class:`GraphStore` in CSR
        format.

        Args:
            edge_types (List[Any], optional): The edge types of edge indices
                to obtain. If set to :obj:`None`, will return the edge indices
                of all existing edge types. (default: :obj:`None`)
            store (bool, optional): Whether to store converted edge indices in
                the :class:`GraphStore`. (default: :obj:`False`)
        """
        return self._edges_to_layout(EdgeLayout.CSR, edge_types, store)

    def csc(
        self,
        edge_types: Optional[List[Any]] = None,
        store: bool = False,
    ) -> ConversionOutputType:
        r"""Obtains the edge indices in the :class:`GraphStore` in CSC
        format.

        Args:
            edge_types (List[Any], optional): The edge types of edge indices
                to obtain. If set to :obj:`None`, will return the edge indices
                of all existing edge types. (default: :obj:`None`)
            store (bool, optional): Whether to store converted edge indices in
                the :class:`GraphStore`. (default: :obj:`False`)
        """
        return self._edges_to_layout(EdgeLayout.CSC, edge_types, store)

    # Python built-ins ########################################################

    def __setitem__(self, key: EdgeAttr, value: EdgeTensorType):
        self.put_edge_index(value, key)

    def __getitem__(self, key: EdgeAttr) -> Optional[EdgeTensorType]:
        return self.get_edge_index(key)

    def __delitem__(self, key: EdgeAttr):
        return self.remove_edge_index(key)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'

    # Helper methods ##########################################################

    def _edge_to_layout(
        self,
        attr: EdgeAttr,
        layout: EdgeLayout,
        store: bool = False,
    ) -> Tuple[Tensor, Tensor, OptTensor]:

        (row, col), perm = self.get_edge_index(attr), None

        if layout == EdgeLayout.COO:  # COO output requested:
            if attr.layout == EdgeLayout.CSR:  # CSR->COO
                row = ptr2index(row)
            elif attr.layout == EdgeLayout.CSC:  # CSC->COO
                col = ptr2index(col)

        elif layout == EdgeLayout.CSR:  # CSR output requested:
            if attr.layout == EdgeLayout.CSC:  # CSC->COO
                col = ptr2index(col)

            if attr.layout != EdgeLayout.CSR:  # COO->CSR
                num_rows = attr.size[0] if attr.size else int(row.max()) + 1
                row, perm = index_sort(row, max_value=num_rows)
                col = col[perm]
                row = index2ptr(row, num_rows)

        else:  # CSC output requested:
            if attr.layout == EdgeLayout.CSR:  # CSR->COO
                row = ptr2index(row)

            if attr.layout != EdgeLayout.CSC:  # COO->CSC
                num_cols = attr.size[1] if attr.size else int(col.max()) + 1
                if not attr.is_sorted:  # Not sorted by destination.
                    col, perm = index_sort(col, max_value=num_cols)
                    row = row[perm]
                col = index2ptr(col, num_cols)

        if attr.layout != layout and store:
            attr = copy.copy(attr)
            attr.layout = layout
            if perm is not None:
                attr.is_sorted = False
            self.put_edge_index((row, col), attr)

        return row, col, perm

    def _edges_to_layout(
        self,
        layout: EdgeLayout,
        edge_types: Optional[List[Any]] = None,
        store: bool = False,
    ) -> ConversionOutputType:

        # Obtain all edge attributes, grouped by type:
        edge_type_attrs: Dict[EdgeType, List[EdgeAttr]] = defaultdict(list)
        for attr in self.get_all_edge_attrs():
            edge_type_attrs[attr.edge_type].append(attr)

        # Check that requested edge types exist and filter:
        if edge_types is not None:
            for edge_type in edge_types:
                if edge_type not in edge_type_attrs:
                    raise ValueError(f"The 'edge_index' of type '{edge_type}' "
                                     f"was not found in the graph store.")

            edge_type_attrs = {
                key: attr
                for key, attr in edge_type_attrs.items() if key in edge_types
            }

        # Convert layout from its most favorable original layout:
        row_dict, col_dict, perm_dict = {}, {}, {}
        for edge_type, attrs in edge_type_attrs.items():
            layouts = [attr.layout for attr in attrs]

            if layout in layouts:  # No conversion needed.
                attr = attrs[layouts.index(layout)]
            elif EdgeLayout.COO in layouts:  # Prefer COO for conversion.
                attr = attrs[layouts.index(EdgeLayout.COO)]
            elif EdgeLayout.CSC in layouts:
                attr = attrs[layouts.index(EdgeLayout.CSC)]
            elif EdgeLayout.CSR in layouts:
                attr = attrs[layouts.index(EdgeLayout.CSR)]

            row_dict[edge_type], col_dict[edge_type], perm_dict[edge_type] = (
                self._edge_to_layout(attr, layout, store))

        return row_dict, col_dict, perm_dict
pytorch/torch_geometric/data/hetero_data.py000066400000000000000000001264511444262304300215000ustar00rootroot00000000000000import copy
import re
import warnings
from collections import defaultdict, namedtuple
from collections.abc import Mapping
from itertools import chain
from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import EdgeAttr, FeatureStore, GraphStore, TensorAttr
from torch_geometric.data.data import BaseData, Data, size_repr, warn_or_raise
from torch_geometric.data.graph_store import EdgeLayout
from torch_geometric.data.storage import BaseStorage, EdgeStorage, NodeStorage
from torch_geometric.typing import (
    DEFAULT_REL,
    EdgeTensorType,
    EdgeType,
    FeatureTensorType,
    NodeType,
    QueryType,
    SparseTensor,
)
from torch_geometric.utils import (
    bipartite_subgraph,
    contains_isolated_nodes,
    is_undirected,
    mask_select,
)

NodeOrEdgeType = Union[NodeType, EdgeType]
NodeOrEdgeStorage = Union[NodeStorage, EdgeStorage]


class HeteroData(BaseData, FeatureStore, GraphStore):
    r"""A data object describing a heterogeneous graph, holding multiple node
    and/or edge types in disjunct storage objects.
    Storage objects can hold either node-level, link-level or graph-level
    attributes.
    In general, :class:`~torch_geometric.data.HeteroData` tries to mimic the
    behavior of a regular **nested** Python dictionary.
    In addition, it provides useful functionality for analyzing graph
    structures, and provides basic PyTorch tensor functionalities.

    .. code-block::

        from torch_geometric.data import HeteroData

        data = HeteroData()

        # Create two node types "paper" and "author" holding a feature matrix:
        data['paper'].x = torch.randn(num_papers, num_paper_features)
        data['author'].x = torch.randn(num_authors, num_authors_features)

        # Create an edge type "(author, writes, paper)" and building the
        # graph connectivity:
        data['author', 'writes', 'paper'].edge_index = ...  # [2, num_edges]

        data['paper'].num_nodes
        >>> 23

        data['author', 'writes', 'paper'].num_edges
        >>> 52

        # PyTorch tensor functionality:
        data = data.pin_memory()
        data = data.to('cuda:0', non_blocking=True)

    Note that there exists multiple ways to create a heterogeneous graph data,
    *e.g.*:

    * To initialize a node of type :obj:`"paper"` holding a node feature
      matrix :obj:`x_paper` named :obj:`x`:

      .. code-block:: python

        from torch_geometric.data import HeteroData

        # (1) Assign attributes after initialization,
        data = HeteroData()
        data['paper'].x = x_paper

        # or (2) pass them as keyword arguments during initialization,
        data = HeteroData(paper={ 'x': x_paper })

        # or (3) pass them as dictionaries during initialization,
        data = HeteroData({'paper': { 'x': x_paper }})

    * To initialize an edge from source node type :obj:`"author"` to
      destination node type :obj:`"paper"` with relation type :obj:`"writes"`
      holding a graph connectivity matrix :obj:`edge_index_author_paper` named
      :obj:`edge_index`:

      .. code-block:: python

        # (1) Assign attributes after initialization,
        data = HeteroData()
        data['author', 'writes', 'paper'].edge_index = edge_index_author_paper

        # or (2) pass them as keyword arguments during initialization,
        data = HeteroData(author__writes__paper={
            'edge_index': edge_index_author_paper
        })

        # or (3) pass them as dictionaries during initialization,
        data = HeteroData({
            ('author', 'writes', 'paper'):
            { 'edge_index': edge_index_author_paper }
        })
    """
    def __init__(self, _mapping: Optional[Dict[str, Any]] = None, **kwargs):
        super().__init__()

        self.__dict__['_global_store'] = BaseStorage(_parent=self)
        self.__dict__['_node_store_dict'] = {}
        self.__dict__['_edge_store_dict'] = {}

        for key, value in chain((_mapping or {}).items(), kwargs.items()):
            if '__' in key and isinstance(value, Mapping):
                key = tuple(key.split('__'))

            if isinstance(value, Mapping):
                self[key].update(value)
            else:
                setattr(self, key, value)

    @classmethod
    def from_dict(cls, mapping: Dict[str, Any]) -> 'HeteroData':
        r"""Creates a :class:`~torch_geometric.data.HeteroData` object from a
        Python dictionary."""
        out = cls()
        for key, value in mapping.items():
            if key == '_global_store':
                out.__dict__['_global_store'] = BaseStorage(
                    _parent=out, **value)
            elif isinstance(key, str):
                out._node_store_dict[key] = NodeStorage(
                    _parent=out, _key=key, **value)
            else:
                out._edge_store_dict[key] = EdgeStorage(
                    _parent=out, _key=key, **value)
        return out

    def __getattr__(self, key: str) -> Any:
        # `data.*_dict` => Link to node and edge stores.
        # `data.*` => Link to the `_global_store`.
        # Using `data.*_dict` is the same as using `collect()` for collecting
        # nodes and edges features.
        if hasattr(self._global_store, key):
            return getattr(self._global_store, key)
        elif bool(re.search('_dict$', key)):
            return self.collect(key[:-5])
        raise AttributeError(f"'{self.__class__.__name__}' has no "
                             f"attribute '{key}'")

    def __setattr__(self, key: str, value: Any):
        # NOTE: We aim to prevent duplicates in node or edge types.
        if key in self.node_types:
            raise AttributeError(f"'{key}' is already present as a node type")
        elif key in self.edge_types:
            raise AttributeError(f"'{key}' is already present as an edge type")
        setattr(self._global_store, key, value)

    def __delattr__(self, key: str):
        delattr(self._global_store, key)

    def __getitem__(self, *args: QueryType) -> Any:
        # `data[*]` => Link to either `_global_store`, _node_store_dict` or
        # `_edge_store_dict`.
        # If neither is present, we create a new `Storage` object for the given
        # node/edge-type.
        key = self._to_canonical(*args)

        out = self._global_store.get(key, None)
        if out is not None:
            return out

        if isinstance(key, tuple):
            return self.get_edge_store(*key)
        else:
            return self.get_node_store(key)

    def __setitem__(self, key: str, value: Any):
        if key in self.node_types:
            raise AttributeError(f"'{key}' is already present as a node type")
        elif key in self.edge_types:
            raise AttributeError(f"'{key}' is already present as an edge type")
        self._global_store[key] = value

    def __delitem__(self, *args: QueryType):
        # `del data[*]` => Link to `_node_store_dict` or `_edge_store_dict`.
        key = self._to_canonical(*args)
        if key in self.edge_types:
            del self._edge_store_dict[key]
        elif key in self.node_types:
            del self._node_store_dict[key]
        else:
            del self._global_store[key]

    def __copy__(self):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out.__dict__['_global_store'] = copy.copy(self._global_store)
        out._global_store._parent = out
        out.__dict__['_node_store_dict'] = {}
        for key, store in self._node_store_dict.items():
            out._node_store_dict[key] = copy.copy(store)
            out._node_store_dict[key]._parent = out
        out.__dict__['_edge_store_dict'] = {}
        for key, store in self._edge_store_dict.items():
            out._edge_store_dict[key] = copy.copy(store)
            out._edge_store_dict[key]._parent = out
        return out

    def __deepcopy__(self, memo):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = copy.deepcopy(value, memo)
        out._global_store._parent = out
        for key in self._node_store_dict.keys():
            out._node_store_dict[key]._parent = out
        for key in out._edge_store_dict.keys():
            out._edge_store_dict[key]._parent = out
        return out

    def __repr__(self) -> str:
        info1 = [size_repr(k, v, 2) for k, v in self._global_store.items()]
        info2 = [size_repr(k, v, 2) for k, v in self._node_store_dict.items()]
        info3 = [size_repr(k, v, 2) for k, v in self._edge_store_dict.items()]
        info = ',\n'.join(info1 + info2 + info3)
        info = f'\n{info}\n' if len(info) > 0 else info
        return f'{self.__class__.__name__}({info})'

    def stores_as(self, data: 'HeteroData'):
        for node_type in data.node_types:
            self.get_node_store(node_type)
        for edge_type in data.edge_types:
            self.get_edge_store(*edge_type)
        return self

    @property
    def stores(self) -> List[BaseStorage]:
        r"""Returns a list of all storages of the graph."""
        return ([self._global_store] + list(self.node_stores) +
                list(self.edge_stores))

    @property
    def node_types(self) -> List[NodeType]:
        r"""Returns a list of all node types of the graph."""
        return list(self._node_store_dict.keys())

    @property
    def node_stores(self) -> List[NodeStorage]:
        r"""Returns a list of all node storages of the graph."""
        return list(self._node_store_dict.values())

    @property
    def edge_types(self) -> List[EdgeType]:
        r"""Returns a list of all edge types of the graph."""
        return list(self._edge_store_dict.keys())

    @property
    def edge_stores(self) -> List[EdgeStorage]:
        r"""Returns a list of all edge storages of the graph."""
        return list(self._edge_store_dict.values())

    def node_items(self) -> List[Tuple[NodeType, NodeStorage]]:
        r"""Returns a list of node type and node storage pairs."""
        return list(self._node_store_dict.items())

    def edge_items(self) -> List[Tuple[EdgeType, EdgeStorage]]:
        r"""Returns a list of edge type and edge storage pairs."""
        return list(self._edge_store_dict.items())

    def to_dict(self) -> Dict[str, Any]:
        out_dict: Dict[str, Any] = {}
        out_dict['_global_store'] = self._global_store.to_dict()
        for key, store in chain(self._node_store_dict.items(),
                                self._edge_store_dict.items()):
            out_dict[key] = store.to_dict()
        return out_dict

    def to_namedtuple(self) -> NamedTuple:
        field_names = list(self._global_store.keys())
        field_values = list(self._global_store.values())
        field_names += [
            '__'.join(key) if isinstance(key, tuple) else key
            for key in self.node_types + self.edge_types
        ]
        field_values += [
            store.to_namedtuple()
            for store in self.node_stores + self.edge_stores
        ]
        DataTuple = namedtuple('DataTuple', field_names)
        return DataTuple(*field_values)

    def set_value_dict(
        self,
        key: str,
        value_dict: Dict[str, Any],
    ) -> 'HeteroData':
        r"""Sets the values in the dictionary :obj:`value_dict` to the
        attribute with name :obj:`key` to all node/edge types present in the
        dictionary.

        .. code-block:: python

           data = HeteroData()

           data.set_value_dict('x', {
               'paper': torch.randn(4, 16),
               'author': torch.randn(8, 32),
           })

           print(data['paper'].x)
        """
        for k, v in (value_dict or {}).items():
            self[k][key] = v
        return self

    def update(self, data: 'HeteroData') -> 'HeteroData':
        for store in data.stores:
            for key, value in store.items():
                self[store._key][key] = value
        return self

    def __cat_dim__(self, key: str, value: Any,
                    store: Optional[NodeOrEdgeStorage] = None, *args,
                    **kwargs) -> Any:
        if isinstance(value, SparseTensor) and 'adj' in key:
            return (0, 1)
        elif 'index' in key or 'face' in key:
            return -1
        return 0

    def __inc__(self, key: str, value: Any,
                store: Optional[NodeOrEdgeStorage] = None, *args,
                **kwargs) -> Any:
        if 'batch' in key:
            return int(value.max()) + 1
        elif isinstance(store, EdgeStorage) and 'index' in key:
            return torch.tensor(store.size()).view(2, 1)
        else:
            return 0

    @property
    def num_nodes(self) -> Optional[int]:
        r"""Returns the number of nodes in the graph."""
        return super().num_nodes

    @property
    def num_node_features(self) -> Dict[NodeType, int]:
        r"""Returns the number of features per node type in the graph."""
        return {
            key: store.num_node_features
            for key, store in self._node_store_dict.items()
        }

    @property
    def num_features(self) -> Dict[NodeType, int]:
        r"""Returns the number of features per node type in the graph.
        Alias for :py:attr:`~num_node_features`."""
        return self.num_node_features

    @property
    def num_edge_features(self) -> Dict[EdgeType, int]:
        r"""Returns the number of features per edge type in the graph."""
        return {
            key: store.num_edge_features
            for key, store in self._edge_store_dict.items()
        }

    def has_isolated_nodes(self) -> bool:
        r"""Returns :obj:`True` if the graph contains isolated nodes."""
        edge_index, _, _ = to_homogeneous_edge_index(self)
        return contains_isolated_nodes(edge_index, num_nodes=self.num_nodes)

    def is_undirected(self) -> bool:
        r"""Returns :obj:`True` if graph edges are undirected."""
        edge_index, _, _ = to_homogeneous_edge_index(self)
        return is_undirected(edge_index, num_nodes=self.num_nodes)

    def validate(self, raise_on_error: bool = True) -> bool:
        r"""Validates the correctness of the data."""
        cls_name = self.__class__.__name__
        status = True

        for edge_type, store in self._edge_store_dict.items():
            src, _, dst = edge_type

            num_src_nodes = self[src].num_nodes
            num_dst_nodes = self[dst].num_nodes
            if num_src_nodes is None:
                status = False
                warn_or_raise(
                    f"'num_nodes' is undefined in node type '{src}' of "
                    f"'{cls_name}'", raise_on_error)

            if num_dst_nodes is None:
                status = False
                warn_or_raise(
                    f"'num_nodes' is undefined in node type '{dst}' of "
                    f"'{cls_name}'", raise_on_error)

            if 'edge_index' in store:
                if (store.edge_index.dim() != 2
                        or store.edge_index.size(0) != 2):
                    status = False
                    warn_or_raise(
                        f"'edge_index' of edge type {edge_type} needs to be "
                        f"of shape [2, num_edges] in '{cls_name}' (found "
                        f"{store.edge_index.size()})", raise_on_error)

            if 'edge_index' in store and store.edge_index.numel() > 0:
                if store.edge_index.min() < 0:
                    status = False
                    warn_or_raise(
                        f"'edge_index' of edge type {edge_type} contains "
                        f"negative indices in '{cls_name}' "
                        f"(found {int(store.edge_index.min())})",
                        raise_on_error)

                if (num_src_nodes is not None
                        and store.edge_index[0].max() >= num_src_nodes):
                    status = False
                    warn_or_raise(
                        f"'edge_index' of edge type {edge_type} contains "
                        f"larger source indices than the number of nodes "
                        f"({num_src_nodes}) of this node type in '{cls_name}' "
                        f"(found {int(store.edge_index[0].max())})",
                        raise_on_error)

                if (num_dst_nodes is not None
                        and store.edge_index[1].max() >= num_dst_nodes):
                    status = False
                    warn_or_raise(
                        f"'edge_index' of edge type {edge_type} contains "
                        f"larger destination indices than the number of nodes "
                        f"({num_dst_nodes}) of this node type in '{cls_name}' "
                        f"(found {int(store.edge_index[1].max())})",
                        raise_on_error)

        return status

    def debug(self):
        pass  # TODO

    ###########################################################################

    def _to_canonical(self, *args: QueryType) -> NodeOrEdgeType:
        # Converts a given `QueryType` to its "canonical type":
        # 1. `relation_type` will get mapped to the unique
        #    `(src_node_type, relation_type, dst_node_type)` tuple.
        # 2. `(src_node_type, dst_node_type)` will get mapped to the unique
        #    `(src_node_type, *, dst_node_type)` tuple, and
        #    `(src_node_type, 'to', dst_node_type)` otherwise.
        if len(args) == 1:
            args = args[0]

        if isinstance(args, str):
            node_types = [key for key in self.node_types if key == args]
            if len(node_types) == 1:
                args = node_types[0]
                return args

            # Try to map to edge type based on unique relation type:
            edge_types = [key for key in self.edge_types if key[1] == args]
            if len(edge_types) == 1:
                args = edge_types[0]
                return args

        elif len(args) == 2:
            # Try to find the unique source/destination node tuple:
            edge_types = [
                key for key in self.edge_types
                if key[0] == args[0] and key[-1] == args[-1]
            ]
            if len(edge_types) == 1:
                args = edge_types[0]
                return args
            elif len(edge_types) == 0:
                args = (args[0], DEFAULT_REL, args[1])
                return args

        return args

    def metadata(self) -> Tuple[List[NodeType], List[EdgeType]]:
        r"""Returns the heterogeneous meta-data, *i.e.* its node and edge
        types.

        .. code-block:: python

            data = HeteroData()
            data['paper'].x = ...
            data['author'].x = ...
            data['author', 'writes', 'paper'].edge_index = ...

            print(data.metadata())
            >>> (['paper', 'author'], [('author', 'writes', 'paper')])
        """
        return self.node_types, self.edge_types

    def collect(self, key: str) -> Dict[NodeOrEdgeType, Any]:
        r"""Collects the attribute :attr:`key` from all node and edge types.

        .. code-block:: python

            data = HeteroData()
            data['paper'].x = ...
            data['author'].x = ...

            print(data.collect('x'))
            >>> { 'paper': ..., 'author': ...}

        .. note::

            This is equivalent to writing :obj:`data.x_dict`.
        """
        mapping = {}
        for subtype, store in chain(self._node_store_dict.items(),
                                    self._edge_store_dict.items()):
            if hasattr(store, key):
                mapping[subtype] = getattr(store, key)
        return mapping

    def _check_type_name(self, name: str):
        if '__' in name:
            warnings.warn(f"The type '{name}' contains double underscores "
                          f"('__') which may lead to unexpected behavior. "
                          f"To avoid any issues, ensure that your type names "
                          f"only contain single underscores.")

    def get_node_store(self, key: NodeType) -> NodeStorage:
        r"""Gets the :class:`~torch_geometric.data.storage.NodeStorage` object
        of a particular node type :attr:`key`.
        If the storage is not present yet, will create a new
        :class:`torch_geometric.data.storage.NodeStorage` object for the given
        node type.

        .. code-block:: python

            data = HeteroData()
            node_storage = data.get_node_store('paper')
        """
        out = self._node_store_dict.get(key, None)
        if out is None:
            self._check_type_name(key)
            out = NodeStorage(_parent=self, _key=key)
            self._node_store_dict[key] = out
        return out

    def get_edge_store(self, src: str, rel: str, dst: str) -> EdgeStorage:
        r"""Gets the :class:`~torch_geometric.data.storage.EdgeStorage` object
        of a particular edge type given by the tuple :obj:`(src, rel, dst)`.
        If the storage is not present yet, will create a new
        :class:`torch_geometric.data.storage.EdgeStorage` object for the given
        edge type.

        .. code-block:: python

            data = HeteroData()
            edge_storage = data.get_edge_store('author', 'writes', 'paper')
        """
        key = (src, rel, dst)
        out = self._edge_store_dict.get(key, None)
        if out is None:
            self._check_type_name(rel)
            out = EdgeStorage(_parent=self, _key=key)
            self._edge_store_dict[key] = out
        return out

    def rename(self, name: NodeType, new_name: NodeType) -> 'HeteroData':
        r"""Renames the node type :obj:`name` to :obj:`new_name` in-place."""
        node_store = self._node_store_dict.pop(name)
        node_store._key = new_name
        self._node_store_dict[new_name] = node_store

        for edge_type in self.edge_types:
            src, rel, dst = edge_type
            if src == name or dst == name:
                edge_store = self._edge_store_dict.pop(edge_type)
                src = new_name if src == name else src
                dst = new_name if dst == name else dst
                edge_type = (src, rel, dst)
                edge_store._key = edge_type
                self._edge_store_dict[edge_type] = edge_store

        return self

    def subgraph(self, subset_dict: Dict[NodeType, Tensor]) -> 'HeteroData':
        r"""Returns the induced subgraph containing the node types and
        corresponding nodes in :obj:`subset_dict`.

        If a node type is not a key in :obj:`subset_dict` then all nodes of
        that type remain in the graph.

        .. code-block:: python

            data = HeteroData()
            data['paper'].x = ...
            data['author'].x = ...
            data['conference'].x = ...
            data['paper', 'cites', 'paper'].edge_index = ...
            data['author', 'paper'].edge_index = ...
            data['paper', 'conference'].edge_index = ...
            print(data)
            >>> HeteroData(
                paper={ x=[10, 16] },
                author={ x=[5, 32] },
                conference={ x=[5, 8] },
                (paper, cites, paper)={ edge_index=[2, 50] },
                (author, to, paper)={ edge_index=[2, 30] },
                (paper, to, conference)={ edge_index=[2, 25] }
            )

            subset_dict = {
                'paper': torch.tensor([3, 4, 5, 6]),
                'author': torch.tensor([0, 2]),
            }

            print(data.subgraph(subset_dict))
            >>> HeteroData(
                paper={ x=[4, 16] },
                author={ x=[2, 32] },
                conference={ x=[5, 8] },
                (paper, cites, paper)={ edge_index=[2, 24] },
                (author, to, paper)={ edge_index=[2, 5] },
                (paper, to, conference)={ edge_index=[2, 10] }
            )

        Args:
            subset_dict (Dict[str, LongTensor or BoolTensor]): A dictionary
                holding the nodes to keep for each node type.
        """
        data = copy.copy(self)
        subset_dict = copy.copy(subset_dict)

        for node_type, subset in subset_dict.items():
            for key, value in self[node_type].items():
                if key == 'num_nodes':
                    if subset.dtype == torch.bool:
                        data[node_type].num_nodes = int(subset.sum())
                    else:
                        data[node_type].num_nodes = subset.size(0)
                elif self[node_type].is_node_attr(key):
                    data[node_type][key] = value[subset]
                else:
                    data[node_type][key] = value

        for edge_type in self.edge_types:
            src, _, dst = edge_type

            src_subset = subset_dict.get(src)
            if src_subset is None:
                src_subset = torch.arange(data[src].num_nodes)
            dst_subset = subset_dict.get(dst)
            if dst_subset is None:
                dst_subset = torch.arange(data[dst].num_nodes)

            edge_index, _, edge_mask = bipartite_subgraph(
                (src_subset, dst_subset),
                self[edge_type].edge_index,
                relabel_nodes=True,
                size=(self[src].num_nodes, self[dst].num_nodes),
                return_edge_mask=True,
            )

            for key, value in self[edge_type].items():
                if key == 'edge_index':
                    data[edge_type].edge_index = edge_index
                elif self[edge_type].is_edge_attr(key):
                    data[edge_type][key] = value[edge_mask]
                else:
                    data[edge_type][key] = value

        return data

    def edge_subgraph(
        self,
        subset_dict: Dict[EdgeType, Tensor],
    ) -> 'HeteroData':
        r"""Returns the induced subgraph given by the edge indices in
        :obj:`subset_dict` for certain edge types.
        Will currently preserve all the nodes in the graph, even if they are
        isolated after subgraph computation.

        Args:
            subset_dict (Dict[Tuple[str, str, str], LongTensor or BoolTensor]):
                A dictionary holding the edges to keep for each edge type.
        """
        data = copy.copy(self)

        for edge_type, subset in subset_dict.items():
            edge_store, new_edge_store = self[edge_type], data[edge_type]
            for key, value in edge_store.items():
                if edge_store.is_edge_attr(key):
                    dim = self.__cat_dim__(key, value, edge_store)
                    if subset.dtype == torch.bool:
                        new_edge_store[key] = mask_select(value, dim, subset)
                    else:
                        new_edge_store[key] = value.index_select(dim, subset)

        return data

    def node_type_subgraph(self, node_types: List[NodeType]) -> 'HeteroData':
        r"""Returns the subgraph induced by the given :obj:`node_types`, *i.e.*
        the returned :class:`HeteroData` object only contains the node types
        which are included in :obj:`node_types`, and only contains the edge
        types where both end points are included in :obj:`node_types`."""
        data = copy.copy(self)
        for edge_type in self.edge_types:
            src, _, dst = edge_type
            if src not in node_types or dst not in node_types:
                del data[edge_type]
        for node_type in self.node_types:
            if node_type not in node_types:
                del data[node_type]
        return data

    def edge_type_subgraph(self, edge_types: List[EdgeType]) -> 'HeteroData':
        r"""Returns the subgraph induced by the given :obj:`edge_types`, *i.e.*
        the returned :class:`HeteroData` object only contains the edge types
        which are included in :obj:`edge_types`, and only contains the node
        types of the end points which are included in :obj:`node_types`."""
        edge_types = [self._to_canonical(e) for e in edge_types]

        data = copy.copy(self)
        for edge_type in self.edge_types:
            if edge_type not in edge_types:
                del data[edge_type]
        node_types = set(e[0] for e in edge_types)
        node_types |= set(e[-1] for e in edge_types)
        for node_type in self.node_types:
            if node_type not in node_types:
                del data[node_type]
        return data

    def to_homogeneous(
        self,
        node_attrs: Optional[List[str]] = None,
        edge_attrs: Optional[List[str]] = None,
        add_node_type: bool = True,
        add_edge_type: bool = True,
        dummy_values: bool = True,
    ) -> Data:
        """Converts a :class:`~torch_geometric.data.HeteroData` object to a
        homogeneous :class:`~torch_geometric.data.Data` object.
        By default, all features with same feature dimensionality across
        different types will be merged into a single representation, unless
        otherwise specified via the :obj:`node_attrs` and :obj:`edge_attrs`
        arguments.
        Furthermore, attributes named :obj:`node_type` and :obj:`edge_type`
        will be added to the returned :class:`~torch_geometric.data.Data`
        object, denoting node-level and edge-level vectors holding the
        node and edge type as integers, respectively.

        Args:
            node_attrs (List[str], optional): The node features to combine
                across all node types. These node features need to be of the
                same feature dimensionality. If set to :obj:`None`, will
                automatically determine which node features to combine.
                (default: :obj:`None`)
            edge_attrs (List[str], optional): The edge features to combine
                across all edge types. These edge features need to be of the
                same feature dimensionality. If set to :obj:`None`, will
                automatically determine which edge features to combine.
                (default: :obj:`None`)
            add_node_type (bool, optional): If set to :obj:`False`, will not
                add the node-level vector :obj:`node_type` to the returned
                :class:`~torch_geometric.data.Data` object.
                (default: :obj:`True`)
            add_edge_type (bool, optional): If set to :obj:`False`, will not
                add the edge-level vector :obj:`edge_type` to the returned
                :class:`~torch_geometric.data.Data` object.
                (default: :obj:`True`)
            dummy_values (bool, optional): If set to :obj:`True`, will fill
                attributes of remaining types with dummy values.
                Dummy values are :obj:`NaN` for floating point attributes,
                and :obj:`-1` for integers. (default: :obj:`True`)
        """
        def get_sizes(stores: List[BaseStorage]) -> Dict[str, List[Tuple]]:
            sizes_dict = defaultdict(list)
            for store in stores:
                for key, value in store.items():
                    if key in ['edge_index', 'adj', 'adj_t']:
                        continue
                    if isinstance(value, Tensor):
                        dim = self.__cat_dim__(key, value, store)
                        size = value.size()[:dim] + value.size()[dim + 1:]
                        sizes_dict[key].append(tuple(size))
            return sizes_dict

        def fill_dummy_(stores: List[BaseStorage],
                        keys: Optional[List[str]] = None):
            sizes_dict = get_sizes(stores)

            if keys is not None:
                sizes_dict = {
                    key: sizes
                    for key, sizes in sizes_dict.items() if key in keys
                }

            sizes_dict = {
                key: sizes
                for key, sizes in sizes_dict.items() if len(set(sizes)) == 1
            }

            for store in stores:  # Fill stores with dummy features:
                for key, sizes in sizes_dict.items():
                    if key not in store:
                        ref = list(self.collect(key).values())[0]
                        dim = self.__cat_dim__(key, ref, store)
                        dummy = float('NaN') if ref.is_floating_point() else -1
                        if isinstance(store, NodeStorage):
                            dim_size = store.num_nodes
                        else:
                            dim_size = store.num_edges
                        shape = sizes[0][:dim] + (dim_size, ) + sizes[0][dim:]
                        store[key] = torch.full(shape, dummy, dtype=ref.dtype,
                                                device=ref.device)

        def _consistent_size(stores: List[BaseStorage]) -> List[str]:
            sizes_dict = get_sizes(stores)
            keys = []
            for key, sizes in sizes_dict.items():
                # The attribute needs to exist in all types:
                if len(sizes) != len(stores):
                    continue
                # The attributes needs to have the same number of dimensions:
                lengths = set([len(size) for size in sizes])
                if len(lengths) != 1:
                    continue
                # The attributes needs to have the same size in all dimensions:
                if len(sizes[0]) != 1 and len(set(sizes)) != 1:
                    continue
                keys.append(key)
            return keys

        if dummy_values:
            self = copy.copy(self)
            fill_dummy_(self.node_stores, node_attrs)
            fill_dummy_(self.edge_stores, edge_attrs)

        edge_index, node_slices, edge_slices = to_homogeneous_edge_index(self)
        device = edge_index.device if edge_index is not None else None

        data = Data(**self._global_store.to_dict())
        if edge_index is not None:
            data.edge_index = edge_index
        data._node_type_names = list(node_slices.keys())
        data._edge_type_names = list(edge_slices.keys())

        # Combine node attributes into a single tensor:
        if node_attrs is None:
            node_attrs = _consistent_size(self.node_stores)
        for key in node_attrs:
            if key in {'ptr'}:
                continue
            values = [store[key] for store in self.node_stores]
            dim = self.__cat_dim__(key, values[0], self.node_stores[0])
            dim = values[0].dim() + dim if dim < 0 else dim
            # For two-dimensional features, we allow arbitrary shapes and pad
            # them with zeros if necessary in case their size doesn't match:
            if values[0].dim() == 2 and dim == 0:
                _max = max([value.size(-1) for value in values])
                for i, v in enumerate(values):
                    if v.size(-1) < _max:
                        values[i] = torch.cat(
                            [v, v.new_zeros(v.size(0), _max - v.size(-1))],
                            dim=-1,
                        )
            value = torch.cat(values, dim) if len(values) > 1 else values[0]
            data[key] = value

        if not data.can_infer_num_nodes:
            data.num_nodes = list(node_slices.values())[-1][1]

        # Combine edge attributes into a single tensor:
        if edge_attrs is None:
            edge_attrs = _consistent_size(self.edge_stores)
        for key in edge_attrs:
            values = [store[key] for store in self.edge_stores]
            dim = self.__cat_dim__(key, values[0], self.edge_stores[0])
            value = torch.cat(values, dim) if len(values) > 1 else values[0]
            data[key] = value

        if add_node_type:
            sizes = [offset[1] - offset[0] for offset in node_slices.values()]
            sizes = torch.tensor(sizes, dtype=torch.long, device=device)
            node_type = torch.arange(len(sizes), device=device)
            data.node_type = node_type.repeat_interleave(sizes)

        if add_edge_type and edge_index is not None:
            sizes = [offset[1] - offset[0] for offset in edge_slices.values()]
            sizes = torch.tensor(sizes, dtype=torch.long, device=device)
            edge_type = torch.arange(len(sizes), device=device)
            data.edge_type = edge_type.repeat_interleave(sizes)

        return data

    # FeatureStore interface ##################################################

    def _put_tensor(self, tensor: FeatureTensorType, attr: TensorAttr) -> bool:
        if not attr.is_set('index'):
            attr.index = None

        out = self._node_store_dict.get(attr.group_name, None)
        if out:
            # Group name exists, handle index or create new attribute name:
            val = getattr(out, attr.attr_name, None)
            if val is not None:
                val[attr.index] = tensor
            else:
                assert attr.index is None
                setattr(self[attr.group_name], attr.attr_name, tensor)
        else:
            # No node storage found, just store tensor in new one:
            setattr(self[attr.group_name], attr.attr_name, tensor)
        return True

    def _get_tensor(self, attr: TensorAttr) -> Optional[FeatureTensorType]:
        # Retrieve tensor and index accordingly:
        tensor = getattr(self[attr.group_name], attr.attr_name, None)
        if tensor is not None:
            # TODO this behavior is a bit odd, since TensorAttr requires that
            # we set `index`. So, we assume here that indexing by `None` is
            # equivalent to not indexing at all, which is not in line with
            # Python semantics.
            return tensor[attr.index] if attr.index is not None else tensor
        return None

    def _remove_tensor(self, attr: TensorAttr) -> bool:
        # Remove tensor entirely:
        if hasattr(self[attr.group_name], attr.attr_name):
            delattr(self[attr.group_name], attr.attr_name)
            return True
        return False

    def _get_tensor_size(self, attr: TensorAttr) -> Tuple:
        return self._get_tensor(attr).size()

    def get_all_tensor_attrs(self) -> List[TensorAttr]:
        out = []
        for group_name, group in self.node_items():
            for attr_name in group:
                if group.is_node_attr(attr_name):
                    out.append(TensorAttr(group_name, attr_name))
        return out

    # GraphStore interface ####################################################

    def _put_edge_index(self, edge_index: EdgeTensorType,
                        edge_attr: EdgeAttr) -> bool:
        if not hasattr(self, '_edge_attrs'):
            self._edge_attrs = {}
        self._edge_attrs[(edge_attr.edge_type, edge_attr.layout)] = edge_attr

        row, col = edge_index
        store = self[edge_attr.edge_type]

        if edge_attr.layout == EdgeLayout.COO:
            store.edge_index = torch.stack([row, col], dim=0)
        elif edge_attr.layout == EdgeLayout.CSR:
            store.adj = SparseTensor(
                rowptr=row,
                col=col,
                sparse_sizes=edge_attr.size,
                is_sorted=True,
                trust_data=True,
            )
        else:  # edge_attr.layout == EdgeLayout.CSC:
            size = edge_attr.size[::-1] if edge_attr.size is not None else None
            store.adj_t = SparseTensor(
                rowptr=col,
                col=row,
                sparse_sizes=size,
                is_sorted=True,
                trust_data=True,
            )
        return True

    def _get_edge_index(self, edge_attr: EdgeAttr) -> Optional[EdgeTensorType]:
        r"""Gets an edge index from edge storage, in the specified layout."""
        store = self[edge_attr.edge_type]

        edge_attrs = getattr(self, '_edge_attrs', {})
        if (edge_attr.edge_type, edge_attr.layout) in edge_attrs:
            edge_attr = edge_attrs[(edge_attr.edge_type, edge_attr.layout)]
        if edge_attr.size is None:
            edge_attr.size = store.size()  # Modify in-place.

        if edge_attr.layout == EdgeLayout.COO and 'edge_index' in store:
            row, col = store.edge_index
            return row, col
        elif edge_attr.layout == EdgeLayout.CSR and 'adj' in store:
            rowptr, col, _ = store.adj.csr()
            return rowptr, col
        elif edge_attr.layout == EdgeLayout.CSC and 'adj_t' in store:
            colptr, row, _ = store.adj_t.csr()
            return row, colptr
        return None

    def _remove_edge_index(self, edge_attr: EdgeAttr) -> bool:
        edge_type = edge_attr.edge_type
        store = self[edge_type]
        if edge_attr.layout == EdgeLayout.COO and 'edge_index' in store:
            del store.edge_index
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop((edge_type, EdgeLayout.COO), None)
            return True
        elif edge_attr.layout == EdgeLayout.CSR and 'adj' in store:
            del store.adj
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop((edge_type, EdgeLayout.CSR), None)
            return True
        elif edge_attr.layout == EdgeLayout.CSC and 'adj_t' in store:
            del store.adj_t
            if hasattr(self, '_edge_attrs'):
                self._edges_to_layout.pop((edge_type, EdgeLayout.CSC), None)
            return True
        return False

    def get_all_edge_attrs(self) -> List[EdgeAttr]:
        edge_attrs = getattr(self, '_edge_attrs', {})

        for store in self.edge_stores:
            if ('edge_index' in store
                    and (store._key, EdgeLayout.COO) not in edge_attrs):
                edge_attrs[(store._key, EdgeLayout.COO)] = EdgeAttr(
                    store._key, 'coo', is_sorted=False)
            if ('adj' in store
                    and (store._key, EdgeLayout.CSR) not in edge_attrs):
                size = store.adj.sparse_sizes()
                edge_attrs[(store._key, EdgeLayout.CSR)] = EdgeAttr(
                    store._key, 'csr', size=size)
            if ('adj_t' in store
                    and (store._key, EdgeLayout.CSC) not in edge_attrs):
                size = store.adj_t.sparse_sizes()[::-1]
                edge_attrs[(store._key, EdgeLayout.CSC)] = EdgeAttr(
                    store._key, 'csc', size=size)

        return list(edge_attrs.values())


# Helper functions ############################################################


def get_node_slices(num_nodes: Dict[str, int]) -> Dict[str, Tuple[int, int]]:
    r"""Returns the boundaries of each node type in a graph."""
    node_slices: Dict[NodeType, Tuple[int, int]] = {}
    cumsum = 0
    for node_type, N in num_nodes.items():
        node_slices[node_type] = (cumsum, cumsum + N)
        cumsum += N
    return node_slices


def offset_edge_index(
    node_slices: Dict[NodeType, Tuple[int, int]],
    edge_type: EdgeType,
    edge_index: Tensor,
) -> Tensor:
    r"""Increases the edge indices by the offsets of source and destination
    node types."""
    src, _, dst = edge_type
    offset = [[node_slices[src][0]], [node_slices[dst][0]]]
    offset = torch.tensor(offset, device=edge_index.device)
    return edge_index + offset


def to_homogeneous_edge_index(
    data: HeteroData,
) -> Tuple[Optional[Tensor], Dict[NodeType, Any], Dict[EdgeType, Any]]:
    r"""Converts a heterogeneous graph into a homogeneous typed graph."""
    # Record slice information per node type:
    node_slices = get_node_slices(data.num_nodes_dict)

    # Record edge indices and slice information per edge type:
    cumsum = 0
    edge_indices: List[Tensor] = []
    edge_slices: Dict[EdgeType, Tuple[int, int]] = {}
    for edge_type, edge_index in data.edge_index_dict.items():
        edge_index = offset_edge_index(node_slices, edge_type, edge_index)
        edge_indices.append(edge_index)
        edge_slices[edge_type] = (cumsum, cumsum + edge_index.size(1))
        cumsum += edge_index.size(1)

    edge_index: Optional[Tensor] = None
    if len(edge_indices) == 1:  # Memory-efficient `torch.cat`:
        edge_index = edge_indices[0]
    elif len(edge_indices) > 1:
        edge_index = torch.cat(edge_indices, dim=-1)

    return edge_index, node_slices, edge_slices
pytorch/torch_geometric/data/in_memory_dataset.py000066400000000000000000000176101444262304300227200ustar00rootroot00000000000000import copy
import warnings
from abc import ABC
from collections.abc import Mapping, Sequence
from typing import (
    Any,
    Callable,
    Dict,
    Iterable,
    List,
    Optional,
    Tuple,
    Type,
    Union,
)

import torch
from torch import Tensor

from torch_geometric.data import Batch, Data
from torch_geometric.data.collate import collate
from torch_geometric.data.data import BaseData
from torch_geometric.data.dataset import Dataset, IndexType
from torch_geometric.data.separate import separate


class InMemoryDataset(Dataset, ABC):
    r"""Dataset base class for creating graph datasets which easily fit
    into CPU memory.
    Inherits from :class:`torch_geometric.data.Dataset`.
    See `here `__ for the accompanying
    tutorial.

    Args:
        root (str, optional): Root directory where the dataset should be saved.
            (optional: :obj:`None`)
        transform (callable, optional): A function/transform that takes in a
            :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            transformed version.
            The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            a :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            transformed version.
            The data object will be transformed before being saved to disk.
            (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in a
            :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` object and returns a
            boolean value, indicating whether the data object should be
            included in the final dataset. (default: :obj:`None`)
        log (bool, optional): Whether to print any console output while
            downloading and processing the dataset. (default: :obj:`True`)
    """
    @property
    def raw_file_names(self) -> Union[str, List[str], Tuple]:
        raise NotImplementedError

    @property
    def processed_file_names(self) -> Union[str, List[str], Tuple]:
        raise NotImplementedError

    def __init__(
        self,
        root: Optional[str] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
        log: bool = True,
    ):
        super().__init__(root, transform, pre_transform, pre_filter, log)
        self._data = None
        self.slices = None
        self._data_list: Optional[List[BaseData]] = None

    @property
    def num_classes(self) -> int:
        if self.transform is None:
            return self._infer_num_classes(self._data.y)
        return super().num_classes

    def len(self) -> int:
        if self.slices is None:
            return 1
        for _, value in nested_iter(self.slices):
            return len(value) - 1
        return 0

    def get(self, idx: int) -> BaseData:
        # TODO (matthias) Avoid unnecessary copy here.
        if self.len() == 1:
            return copy.copy(self._data)

        if not hasattr(self, '_data_list') or self._data_list is None:
            self._data_list = self.len() * [None]
        elif self._data_list[idx] is not None:
            return copy.copy(self._data_list[idx])

        data = separate(
            cls=self._data.__class__,
            batch=self._data,
            idx=idx,
            slice_dict=self.slices,
            decrement=False,
        )

        self._data_list[idx] = copy.copy(data)

        return data

    @classmethod
    def save(cls, data_list: List[BaseData], path: str):
        r"""Saves a list of data objects to the file path :obj:`path`."""
        data, slices = cls.collate(data_list)
        torch.save((data.to_dict(), slices), path)

    def load(self, path: str, data_cls: Type[BaseData] = Data):
        r"""Loads the dataset from the file path :obj:`path`."""
        data, self.slices = torch.load(path)
        if isinstance(data, dict):  # Backward compatibility.
            data = data_cls.from_dict(data)
        self.data = data

    @staticmethod
    def collate(
        data_list: List[BaseData],
    ) -> Tuple[BaseData, Optional[Dict[str, Tensor]]]:
        r"""Collates a Python list of :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` objects to the internal
        storage format of :class:`~torch_geometric.data.InMemoryDataset`."""
        if len(data_list) == 1:
            return data_list[0], None

        data, slices, _ = collate(
            data_list[0].__class__,
            data_list=data_list,
            increment=False,
            add_batch=False,
        )

        return data, slices

    def copy(self, idx: Optional[IndexType] = None) -> 'InMemoryDataset':
        r"""Performs a deep-copy of the dataset. If :obj:`idx` is not given,
        will clone the full dataset. Otherwise, will only clone a subset of the
        dataset from indices :obj:`idx`.
        Indices can be slices, lists, tuples, and a :obj:`torch.Tensor` or
        :obj:`np.ndarray` of type long or bool.
        """
        if idx is None:
            data_list = [self.get(i) for i in self.indices()]
        else:
            data_list = [self.get(i) for i in self.index_select(idx).indices()]

        dataset = copy.copy(self)
        dataset._indices = None
        dataset._data_list = None
        dataset.data, dataset.slices = self.collate(data_list)
        return dataset

    @property
    def data(self) -> Any:
        msg1 = ("It is not recommended to directly access the internal "
                "storage format `data` of an 'InMemoryDataset'.")
        msg2 = ("The given 'InMemoryDataset' only references a subset of "
                "examples of the full dataset, but 'data' will contain "
                "information of the full dataset.")
        msg3 = ("The data of the dataset is already cached, so any "
                "modifications to `data` will not be reflected when accessing "
                "its elements. Clearing the cache now by removing all "
                "elements in `dataset._data_list`.")
        msg4 = ("If you are absolutely certain what you are doing, access the "
                "internal storage via `InMemoryDataset._data` instead to "
                "suppress this warning. Alternatively, you can access stacked "
                "individual attributes of every graph via "
                "`dataset.{attr_name}`.")

        msg = msg1
        if self._indices is not None:
            msg += f' {msg2}'
        if self._data_list is not None:
            msg += f' {msg3}'
            self._data_list = None
        msg += f' {msg4}'

        warnings.warn(msg)

        return self._data

    @data.setter
    def data(self, value: Any):
        self._data = value
        self._data_list = None

    def __getattr__(self, key: str) -> Any:
        data = self.__dict__.get('_data')
        if isinstance(data, Data) and key in data:
            if self._indices is None and data.__inc__(key, data[key]) == 0:
                return data[key]
            else:
                data_list = [self.get(i) for i in self.indices()]
                return Batch.from_data_list(data_list)[key]

        raise AttributeError(f"'{self.__class__.__name__}' object has no "
                             f"attribute '{key}'")


def nested_iter(node: Union[Mapping, Sequence]) -> Iterable:
    if isinstance(node, Mapping):
        for key, value in node.items():
            for inner_key, inner_value in nested_iter(value):
                yield inner_key, inner_value
    elif isinstance(node, Sequence):
        for i, inner_value in enumerate(node):
            yield i, inner_value
    else:
        yield None, node
pytorch/torch_geometric/data/lightning/000077500000000000000000000000001444262304300206215ustar00rootroot00000000000000pytorch/torch_geometric/data/lightning/__init__.py000066400000000000000000000002621444262304300227320ustar00rootroot00000000000000from .datamodule import LightningDataset, LightningNodeData, LightningLinkData

__all__ = classes = [
    'LightningDataset',
    'LightningNodeData',
    'LightningLinkData',
]
pytorch/torch_geometric/data/lightning/datamodule.py000066400000000000000000000675121444262304300233250ustar00rootroot00000000000000import copy
import inspect
import warnings
from typing import Any, Dict, Optional, Tuple, Type, Union

import torch

from torch_geometric.data import (
    Data,
    Dataset,
    FeatureStore,
    GraphStore,
    HeteroData,
)
from torch_geometric.loader import DataLoader, LinkLoader, NodeLoader
from torch_geometric.sampler import BaseSampler, NeighborSampler
from torch_geometric.typing import InputEdges, InputNodes, OptTensor

try:
    from pytorch_lightning import LightningDataModule as PLLightningDataModule
    no_pytorch_lightning = False
except (ImportError, ModuleNotFoundError):
    PLLightningDataModule = object
    no_pytorch_lightning = True


class LightningDataModule(PLLightningDataModule):
    def __init__(self, has_val: bool, has_test: bool, **kwargs):
        super().__init__()

        if no_pytorch_lightning:
            raise ModuleNotFoundError(
                "No module named 'pytorch_lightning' found on this machine. "
                "Run 'pip install pytorch_lightning' to install the library.")

        if not has_val:
            self.val_dataloader = None

        if not has_test:
            self.test_dataloader = None

        kwargs.setdefault('batch_size', 1)
        kwargs.setdefault('num_workers', 0)
        kwargs.setdefault('pin_memory', True)
        kwargs.setdefault('persistent_workers',
                          kwargs.get('num_workers', 0) > 0)

        if 'shuffle' in kwargs:
            warnings.warn(f"The 'shuffle={kwargs['shuffle']}' option is "
                          f"ignored in '{self.__class__.__name__}'. Remove it "
                          f"from the argument list to disable this warning")
            del kwargs['shuffle']

        self.kwargs = kwargs

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({kwargs_repr(**self.kwargs)})'


class LightningData(LightningDataModule):
    def __init__(
        self,
        data: Union[Data, HeteroData],
        has_val: bool,
        has_test: bool,
        loader: str = 'neighbor',
        graph_sampler: Optional[BaseSampler] = None,
        eval_loader_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ):
        kwargs.setdefault('batch_size', 1)
        kwargs.setdefault('num_workers', 0)

        if graph_sampler is not None:
            loader = 'custom'

        # For full-batch training, we use reasonable defaults for a lot of
        # data-loading options:
        if loader not in ['full', 'neighbor', 'link_neighbor', 'custom']:
            raise ValueError(f"Undefined 'loader' option (got '{loader}')")

        if loader == 'full' and kwargs['batch_size'] != 1:
            warnings.warn(f"Re-setting 'batch_size' to 1 in "
                          f"'{self.__class__.__name__}' for loader='full' "
                          f"(got '{kwargs['batch_size']}')")
            kwargs['batch_size'] = 1

        if loader == 'full' and kwargs['num_workers'] != 0:
            warnings.warn(f"Re-setting 'num_workers' to 0 in "
                          f"'{self.__class__.__name__}' for loader='full' "
                          f"(got '{kwargs['num_workers']}')")
            kwargs['num_workers'] = 0

        if loader == 'full' and kwargs.get('sampler') is not None:
            warnings.warn("'sampler' option is not supported for "
                          "loader='full'")
            kwargs.pop('sampler', None)

        if loader == 'full' and kwargs.get('batch_sampler') is not None:
            warnings.warn("'batch_sampler' option is not supported for "
                          "loader='full'")
            kwargs.pop('batch_sampler', None)

        super().__init__(has_val, has_test, **kwargs)

        if loader == 'full':
            if kwargs.get('pin_memory', False):
                warnings.warn(f"Re-setting 'pin_memory' to 'False' in "
                              f"'{self.__class__.__name__}' for loader='full' "
                              f"(got 'True')")
            self.kwargs['pin_memory'] = False

        self.data = data
        self.loader = loader

        # Determine sampler and loader arguments ##############################

        if loader in ['neighbor', 'link_neighbor']:

            # Define a new `NeighborSampler` to be re-used across data loaders:
            sampler_kwargs, self.loader_kwargs = split_kwargs(
                self.kwargs,
                NeighborSampler,
            )
            sampler_kwargs.setdefault('share_memory',
                                      self.kwargs['num_workers'] > 0)
            self.graph_sampler = NeighborSampler(data, **sampler_kwargs)

        elif graph_sampler is not None:
            sampler_kwargs, self.loader_kwargs = split_kwargs(
                self.kwargs,
                graph_sampler.__class__,
            )
            if len(sampler_kwargs) > 0:
                warnings.warn(f"Ignoring the arguments "
                              f"{list(sampler_kwargs.keys())} in "
                              f"'{self.__class__.__name__}' since a custom "
                              f"'graph_sampler' was passed")
            self.graph_sampler = graph_sampler

        else:
            assert loader == 'full'
            self.loader_kwargs = self.kwargs

        # Determine validation sampler and loader arguments ###################

        self.eval_loader_kwargs = copy.copy(self.loader_kwargs)
        if eval_loader_kwargs is not None:
            # If the user wants to override certain values during evaluation,
            # we shallow-copy the graph sampler and update its attributes.
            if hasattr(self, 'graph_sampler'):
                self.eval_graph_sampler = copy.copy(self.graph_sampler)

                eval_sampler_kwargs, eval_loader_kwargs = split_kwargs(
                    eval_loader_kwargs,
                    self.graph_sampler.__class__,
                )
                for key, value in eval_sampler_kwargs.items():
                    setattr(self.eval_graph_sampler, key, value)

            self.eval_loader_kwargs.update(eval_loader_kwargs)

        elif hasattr(self, 'graph_sampler'):
            self.eval_graph_sampler = self.graph_sampler

        self.eval_loader_kwargs.pop('sampler', None)
        self.eval_loader_kwargs.pop('batch_sampler', None)

    @property
    def train_shuffle(self) -> bool:
        shuffle = self.loader_kwargs.get('sampler', None) is None
        shuffle &= self.loader_kwargs.get('batch_sampler', None) is None
        return shuffle

    def prepare_data(self):
        if self.loader == 'full':
            try:
                num_devices = self.trainer.num_devices
            except AttributeError:
                # PyTorch Lightning < 1.6 backward compatibility:
                num_devices = self.trainer.num_processes
                num_devices = max(num_devices, self.trainer.num_gpus)

            if num_devices > 1:
                raise ValueError(
                    f"'{self.__class__.__name__}' with loader='full' requires "
                    f"training on a single device")
        super().prepare_data()

    def full_dataloader(self, **kwargs) -> DataLoader:
        warnings.filterwarnings('ignore', '.*does not have many workers.*')
        warnings.filterwarnings('ignore', '.*data loading bottlenecks.*')

        return torch.utils.data.DataLoader(
            [self.data],
            collate_fn=lambda xs: xs[0],
            **kwargs,
        )

    def __repr__(self) -> str:
        kwargs = kwargs_repr(data=self.data, loader=self.loader, **self.kwargs)
        return f'{self.__class__.__name__}({kwargs})'


class LightningDataset(LightningDataModule):
    r"""Converts a set of :class:`~torch_geometric.data.Dataset` objects into a
    :class:`pytorch_lightning.LightningDataModule` variant. It can then be
    automatically used as a :obj:`datamodule` for multi-GPU graph-level
    training via :lightning:`null`
    `PyTorch Lightning `__.
    :class:`LightningDataset` will take care of providing mini-batches via
    :class:`~torch_geometric.loader.DataLoader`.

    .. note::

        Currently only the
        :class:`pytorch_lightning.strategies.SingleDeviceStrategy` and
        :class:`pytorch_lightning.strategies.DDPStrategy` training
        strategies of :lightning:`null` `PyTorch Lightning
        `__ are supported in order to correctly share data across
        all devices/processes:

        .. code-block::

            import pytorch_lightning as pl
            trainer = pl.Trainer(strategy="ddp_spawn", accelerator="gpu",
                                 devices=4)
            trainer.fit(model, datamodule)

    Args:
        train_dataset (Dataset): The training dataset.
        val_dataset (Dataset, optional): The validation dataset.
            (default: :obj:`None`)
        test_dataset (Dataset, optional): The test dataset.
            (default: :obj:`None`)
        pred_dataset (Dataset, optional): The prediction dataset.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.loader.DataLoader`.
    """
    def __init__(
        self,
        train_dataset: Dataset,
        val_dataset: Optional[Dataset] = None,
        test_dataset: Optional[Dataset] = None,
        pred_dataset: Optional[Dataset] = None,
        **kwargs,
    ):
        super().__init__(
            has_val=val_dataset is not None,
            has_test=test_dataset is not None,
            **kwargs,
        )

        self.train_dataset = train_dataset
        self.val_dataset = val_dataset
        self.test_dataset = test_dataset
        self.pred_dataset = pred_dataset

    def dataloader(self, dataset: Dataset, **kwargs) -> DataLoader:
        return DataLoader(dataset, **kwargs)

    def train_dataloader(self) -> DataLoader:
        from torch.utils.data import IterableDataset

        shuffle = not isinstance(self.train_dataset, IterableDataset)
        shuffle &= self.kwargs.get('sampler', None) is None
        shuffle &= self.kwargs.get('batch_sampler', None) is None

        return self.dataloader(self.train_dataset, shuffle=shuffle,
                               **self.kwargs)

    def val_dataloader(self) -> DataLoader:
        kwargs = copy.copy(self.kwargs)
        kwargs.pop('sampler', None)
        kwargs.pop('batch_sampler', None)

        return self.dataloader(self.val_dataset, shuffle=False, **kwargs)

    def test_dataloader(self) -> DataLoader:
        kwargs = copy.copy(self.kwargs)
        kwargs.pop('sampler', None)
        kwargs.pop('batch_sampler', None)

        return self.dataloader(self.test_dataset, shuffle=False, **kwargs)

    def predict_dataloader(self) -> DataLoader:
        kwargs = copy.copy(self.kwargs)
        kwargs.pop('sampler', None)
        kwargs.pop('batch_sampler', None)

        return self.dataloader(self.pred_dataset, shuffle=False, **kwargs)

    def __repr__(self) -> str:
        kwargs = kwargs_repr(train_dataset=self.train_dataset,
                             val_dataset=self.val_dataset,
                             test_dataset=self.test_dataset,
                             pred_dataset=self.pred_dataset, **self.kwargs)
        return f'{self.__class__.__name__}({kwargs})'


class LightningNodeData(LightningData):
    r"""Converts a :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData` object into a
    :class:`pytorch_lightning.LightningDataModule` variant. It can then be
    automatically used as a :obj:`datamodule` for multi-GPU node-level
    training via :lightning:`null`
    `PyTorch Lightning `__.
    :class:`LightningDataset` will take care of providing mini-batches via
    :class:`~torch_geometric.loader.NeighborLoader`.

    .. note::

        Currently only the
        :class:`pytorch_lightning.strategies.SingleDeviceStrategy` and
        :class:`pytorch_lightning.strategies.DDPStrategy` training
        strategies of :lightning:`null` `PyTorch Lightning
        `__ are supported in order to correctly share data across
        all devices/processes:

        .. code-block::

            import pytorch_lightning as pl
            trainer = pl.Trainer(strategy="ddp_spawn", accelerator="gpu",
                                 devices=4)
            trainer.fit(model, datamodule)

    Args:
        data (Data or HeteroData): The :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` graph object.
        input_train_nodes (torch.Tensor or str or (str, torch.Tensor)): The
            indices of training nodes.
            If not given, will try to automatically infer them from the
            :obj:`data` object by searching for :obj:`train_mask`,
            :obj:`train_idx`, or :obj:`train_index` attributes.
            (default: :obj:`None`)
        input_train_time (torch.Tensor, optional): The timestamp
            of training nodes. (default: :obj:`None`)
        input_val_nodes (torch.Tensor or str or (str, torch.Tensor)): The
            indices of validation nodes.
            If not given, will try to automatically infer them from the
            :obj:`data` object by searching for :obj:`val_mask`,
            :obj:`valid_mask`, :obj:`val_idx`, :obj:`valid_idx`,
            :obj:`val_index`, or :obj:`valid_index` attributes.
            (default: :obj:`None`)
        input_val_time (torch.Tensor, optional): The timestamp
            of validation edges. (default: :obj:`None`)
        input_test_nodes (torch.Tensor or str or (str, torch.Tensor)): The
            indices of test nodes.
            If not given, will try to automatically infer them from the
            :obj:`data` object by searching for :obj:`test_mask`,
            :obj:`test_idx`, or :obj:`test_index` attributes.
            (default: :obj:`None`)
        input_test_time (torch.Tensor, optional): The timestamp
            of test nodes. (default: :obj:`None`)
        input_pred_nodes (torch.Tensor or str or (str, torch.Tensor)): The
            indices of prediction nodes.
            If not given, will try to automatically infer them from the
            :obj:`data` object by searching for :obj:`pred_mask`,
            :obj:`pred_idx`, or :obj:`pred_index` attributes.
            (default: :obj:`None`)
        input_pred_time (torch.Tensor, optional): The timestamp
            of prediction nodes. (default: :obj:`None`)
        loader (str): The scalability technique to use (:obj:`"full"`,
            :obj:`"neighbor"`). (default: :obj:`"neighbor"`)
        node_sampler (BaseSampler, optional): A custom sampler object to
            generate mini-batches. If set, will ignore the :obj:`loader`
            option. (default: :obj:`None`)
        eval_loader_kwargs (Dict[str, Any], optional): Custom keyword arguments
            that override the :class:`torch_geometric.loader.NeighborLoader`
            configuration during evaluation. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.loader.NeighborLoader`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData],
        input_train_nodes: InputNodes = None,
        input_train_time: OptTensor = None,
        input_val_nodes: InputNodes = None,
        input_val_time: OptTensor = None,
        input_test_nodes: InputNodes = None,
        input_test_time: OptTensor = None,
        input_pred_nodes: InputNodes = None,
        input_pred_time: OptTensor = None,
        loader: str = 'neighbor',
        node_sampler: Optional[BaseSampler] = None,
        eval_loader_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ):
        if input_train_nodes is None:
            input_train_nodes = infer_input_nodes(data, split='train')

        if input_val_nodes is None:
            input_val_nodes = infer_input_nodes(data, split='val')
            if input_val_nodes is None:
                input_val_nodes = infer_input_nodes(data, split='valid')

        if input_test_nodes is None:
            input_test_nodes = infer_input_nodes(data, split='test')

        if input_pred_nodes is None:
            input_pred_nodes = infer_input_nodes(data, split='pred')

        super().__init__(
            data=data,
            has_val=input_val_nodes is not None,
            has_test=input_test_nodes is not None,
            loader=loader,
            graph_sampler=node_sampler,
            eval_loader_kwargs=eval_loader_kwargs,
            **kwargs,
        )

        self.input_train_nodes = input_train_nodes
        self.input_train_time = input_train_time
        self.input_train_id: OptTensor = None

        self.input_val_nodes = input_val_nodes
        self.input_val_time = input_val_time
        self.input_val_id: OptTensor = None

        self.input_test_nodes = input_test_nodes
        self.input_test_time = input_test_time
        self.input_test_id: OptTensor = None

        self.input_pred_nodes = input_pred_nodes
        self.input_pred_time = input_pred_time
        self.input_pred_id: OptTensor = None

    def dataloader(
        self,
        input_nodes: InputNodes,
        input_time: OptTensor = None,
        input_id: OptTensor = None,
        node_sampler: Optional[BaseSampler] = None,
        **kwargs,
    ) -> DataLoader:
        if self.loader == 'full':
            return self.full_dataloader(**kwargs)

        assert node_sampler is not None

        return NodeLoader(
            self.data,
            node_sampler=node_sampler,
            input_nodes=input_nodes,
            input_time=input_time,
            input_id=input_id,
            **kwargs,
        )

    def train_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_train_nodes,
            self.input_train_time,
            self.input_train_id,
            node_sampler=getattr(self, 'graph_sampler', None),
            shuffle=self.train_shuffle,
            **self.loader_kwargs,
        )

    def val_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_val_nodes,
            self.input_val_time,
            self.input_val_id,
            node_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )

    def test_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_test_nodes,
            self.input_test_time,
            self.input_test_id,
            node_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )

    def predict_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_pred_nodes,
            self.input_pred_time,
            self.input_pred_id,
            node_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )


class LightningLinkData(LightningData):
    r"""Converts a :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData` object into a
    :class:`pytorch_lightning.LightningDataModule` variant. It can then be
    automatically used as a :obj:`datamodule` for multi-GPU link-level
    training via :lightning:`null`
    `PyTorch Lightning `__.
    :class:`LightningDataset` will take care of providing mini-batches via
    :class:`~torch_geometric.loader.LinkNeighborLoader`.

    .. note::

        Currently only the
        :class:`pytorch_lightning.strategies.SingleDeviceStrategy` and
        :class:`pytorch_lightning.strategies.DDPStrategy` training
        strategies of :lightning:`null` `PyTorch Lightning
        `__ are supported in order to correctly share data across
        all devices/processes:

        .. code-block::

            import pytorch_lightning as pl
            trainer = pl.Trainer(strategy="ddp_spawn", accelerator="gpu",
                                 devices=4)
            trainer.fit(model, datamodule)

    Args:
        data (Data or HeteroData or Tuple[FeatureStore, GraphStore]): The
            :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` graph object, or a
            tuple of a :class:`~torch_geometric.data.FeatureStore` and
            :class:`~torch_geometric.data.GraphStore` objects.
        input_train_edges (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The training edges. (default: :obj:`None`)
        input_train_labels (torch.Tensor, optional):
            The labels of training edges. (default: :obj:`None`)
        input_train_time (torch.Tensor, optional): The timestamp
            of training edges. (default: :obj:`None`)
        input_val_edges (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The validation edges. (default: :obj:`None`)
        input_val_labels (torch.Tensor, optional):
            The labels of validation edges. (default: :obj:`None`)
        input_val_time (torch.Tensor, optional): The timestamp
            of validation edges. (default: :obj:`None`)
        input_test_edges (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The test edges. (default: :obj:`None`)
        input_test_labels (torch.Tensor, optional):
            The labels of test edges. (default: :obj:`None`)
        input_test_time (torch.Tensor, optional): The timestamp
            of test edges. (default: :obj:`None`)
        input_pred_edges (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The prediction edges. (default: :obj:`None`)
        input_pred_labels (torch.Tensor, optional):
            The labels of prediction edges. (default: :obj:`None`)
        input_pred_time (torch.Tensor, optional): The timestamp
            of prediction edges. (default: :obj:`None`)
        loader (str): The scalability technique to use (:obj:`"full"`,
            :obj:`"neighbor"`). (default: :obj:`"neighbor"`)
        link_sampler (BaseSampler, optional): A custom sampler object to
            generate mini-batches. If set, will ignore the :obj:`loader`
            option. (default: :obj:`None`)
        eval_loader_kwargs (Dict[str, Any], optional): Custom keyword arguments
            that override the
            :class:`torch_geometric.loader.LinkNeighborLoader` configuration
            during evaluation. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.loader.LinkNeighborLoader`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        input_train_edges: InputEdges = None,
        input_train_labels: OptTensor = None,
        input_train_time: OptTensor = None,
        input_val_edges: InputEdges = None,
        input_val_labels: OptTensor = None,
        input_val_time: OptTensor = None,
        input_test_edges: InputEdges = None,
        input_test_labels: OptTensor = None,
        input_test_time: OptTensor = None,
        input_pred_edges: InputEdges = None,
        input_pred_labels: OptTensor = None,
        input_pred_time: OptTensor = None,
        loader: str = 'neighbor',
        link_sampler: Optional[BaseSampler] = None,
        eval_loader_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ):
        super().__init__(
            data=data,
            has_val=input_val_edges is not None,
            has_test=input_test_edges is not None,
            loader=loader,
            graph_sampler=link_sampler,
            eval_loader_kwargs=eval_loader_kwargs,
            **kwargs,
        )

        self.input_train_edges = input_train_edges
        self.input_train_labels = input_train_labels
        self.input_train_time = input_train_time
        self.input_train_id: OptTensor = None

        self.input_val_edges = input_val_edges
        self.input_val_labels = input_val_labels
        self.input_val_time = input_val_time
        self.input_val_id: OptTensor = None

        self.input_test_edges = input_test_edges
        self.input_test_labels = input_test_labels
        self.input_test_time = input_test_time
        self.input_test_id: OptTensor = None

        self.input_pred_edges = input_pred_edges
        self.input_pred_labels = input_pred_labels
        self.input_pred_time = input_pred_time
        self.input_pred_id: OptTensor = None

    def dataloader(
        self,
        input_edges: InputEdges,
        input_labels: OptTensor = None,
        input_time: OptTensor = None,
        input_id: OptTensor = None,
        link_sampler: Optional[BaseSampler] = None,
        **kwargs,
    ) -> DataLoader:
        if self.loader == 'full':
            return self.full_dataloader(**kwargs)

        assert link_sampler is not None

        return LinkLoader(
            self.data,
            link_sampler=link_sampler,
            edge_label_index=input_edges,
            edge_label=input_labels,
            edge_label_time=input_time,
            input_id=input_id,
            **kwargs,
        )

    def train_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_train_edges,
            self.input_train_labels,
            self.input_train_time,
            self.input_train_id,
            link_sampler=getattr(self, 'graph_sampler', None),
            shuffle=self.train_shuffle,
            **self.loader_kwargs,
        )

    def val_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_val_edges,
            self.input_val_labels,
            self.input_val_time,
            self.input_val_id,
            link_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )

    def test_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_test_edges,
            self.input_test_labels,
            self.input_test_time,
            self.input_test_id,
            link_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )

    def predict_dataloader(self) -> DataLoader:
        return self.dataloader(
            self.input_pred_edges,
            self.input_pred_labels,
            self.input_pred_time,
            self.input_pred_id,
            link_sampler=getattr(self, 'eval_graph_sampler', None),
            shuffle=False,
            **self.eval_loader_kwargs,
        )


###############################################################################


# TODO support Tuple[FeatureStore, GraphStore]
def infer_input_nodes(data: Union[Data, HeteroData], split: str) -> InputNodes:
    attr_name: Optional[str] = None
    if f'{split}_mask' in data:
        attr_name = f'{split}_mask'
    elif f'{split}_idx' in data:
        attr_name = f'{split}_idx'
    elif f'{split}_index' in data:
        attr_name = f'{split}_index'

    if attr_name is None:
        return None

    if isinstance(data, Data):
        return data[attr_name]
    if isinstance(data, HeteroData):
        input_nodes_dict = data.collect(attr_name)
        if len(input_nodes_dict) != 1:
            raise ValueError(f"Could not automatically determine the input "
                             f"nodes of {data} since there exists multiple "
                             f"types with attribute '{attr_name}'")
        return list(input_nodes_dict.items())[0]
    return None


def kwargs_repr(**kwargs) -> str:
    return ', '.join([f'{k}={v}' for k, v in kwargs.items() if v is not None])


def split_kwargs(
    kwargs: Dict[str, Any],
    sampler_cls: Type,
) -> Tuple[Dict[str, Any], Dict[str, Any]]:
    r"""Splits keyword arguments into sampler and loader arguments."""
    sampler_args = inspect.signature(sampler_cls).parameters

    sampler_kwargs: Dict[str, Any] = {}
    loader_kwargs: Dict[str, Any] = {}

    for key, value in kwargs.items():
        if key in sampler_args:
            sampler_kwargs[key] = value
        else:
            loader_kwargs[key] = value

    return sampler_kwargs, loader_kwargs
pytorch/torch_geometric/data/makedirs.py000066400000000000000000000005221444262304300210060ustar00rootroot00000000000000import errno
import os
import os.path as osp


def makedirs(path: str):
    r"""Recursively creates a directory.

    Args:
        path (str): The path to create.
    """
    try:
        os.makedirs(osp.expanduser(osp.normpath(path)))
    except OSError as e:
        if e.errno != errno.EEXIST and osp.isdir(path):
            raise e
pytorch/torch_geometric/data/remote_backend_utils.py000066400000000000000000000075721444262304300234050ustar00rootroot00000000000000# This file defines a set of utilities for remote backends (backends that are
# characterize as Tuple[FeatureStore, GraphStore]). TODO support for
# non-heterogeneous graphs (feature stores with a group_name=None).
from typing import Tuple, Union

from torch_geometric.data import FeatureStore, GraphStore
from torch_geometric.typing import EdgeType, NodeType


# NOTE PyG also supports querying by a relation type `rel` in an edge type
# (src, rel, dst). It may be worth supporting this in remote backends as well.
def _internal_num_nodes(
    feature_store: FeatureStore,
    graph_store: GraphStore,
    query: Union[NodeType, EdgeType],
) -> Union[int, Tuple[int, int]]:
    r"""Returns the number of nodes in the node type or the number of source
    and destination nodes in an edge type by sequentially accessing attributes
    in the feature and graph stores that reveal this number."""
    def _matches_node_type(query: Union[NodeType, EdgeType],
                           node_type: NodeType) -> bool:
        if isinstance(query, (list, tuple)):  # EdgeType
            return query[0] == node_type or query[-1] == node_type
        else:
            return query == node_type

    node_query = isinstance(query, NodeType)

    # TODO: In general, a feature store and graph store should be able to
    # expose methods that allow for easy access to individual attributes,
    # instead of requiring iteration to identify a particular attribute.
    # Implementing this should reduce the iteration below.

    # 1. Check the edges in the GraphStore, for each node type in each edge:
    edge_attrs = graph_store.get_all_edge_attrs()
    _num_nodes = [None] if node_query else [None, None]
    for edge_attr in edge_attrs:
        if edge_attr.size is None:
            continue
        if _matches_node_type(query, edge_attr.edge_type[0]):
            _num_nodes[0] = _num_nodes[0] or edge_attr.size[0]
        if _matches_node_type(query, edge_attr.edge_type[-1]):
            _num_nodes[-1] = _num_nodes[-1] or edge_attr.size[-1]
        if None not in _num_nodes:
            # We have filled out all the nodes:
            return _num_nodes[0] if node_query else tuple(_num_nodes)

    # 2. Check the node types stored in the FeatureStore:
    tensor_attrs = feature_store.get_all_tensor_attrs()
    matching_attrs = [
        attr for attr in tensor_attrs
        if _matches_node_type(query, attr.group_name)
    ]
    if node_query:
        if len(matching_attrs) > 0:
            return feature_store.get_tensor_size(matching_attrs[0])[0]
    else:
        matching_src_attrs = [
            attr for attr in matching_attrs if attr.group_name == query[0]
        ]
        matching_dst_attrs = [
            attr for attr in matching_attrs if attr.group_name == query[-1]
        ]
        if len(matching_src_attrs) > 0 and len(matching_dst_attrs) > 0:
            return (feature_store.get_tensor_size(matching_src_attrs[0])[0],
                    feature_store.get_tensor_size(matching_dst_attrs[0])[0])

    raise ValueError(
        f"Unable to accurately infer the number of nodes corresponding to "
        f"query {query} from feature store {feature_store} and graph store "
        f"{graph_store}. Please consider either adding an edge containing "
        f"the nodes in this query or feature tensors for the nodes in this "
        f"query.")


def num_nodes(
    feature_store: FeatureStore,
    graph_store: GraphStore,
    query: NodeType,
) -> int:
    r"""Returns the number of nodes in a given node type stored in a remote
    backend."""
    return _internal_num_nodes(feature_store, graph_store, query)


def size(
    feature_store: FeatureStore,
    graph_store: GraphStore,
    query: EdgeType,
) -> Tuple[int, int]:
    r"""Returns the size of an edge (number of source nodes, number of
    destination nodes) in an edge stored in a remote backend."""
    return _internal_num_nodes(feature_store, graph_store, query)
pytorch/torch_geometric/data/separate.py000066400000000000000000000104101444262304300210100ustar00rootroot00000000000000from collections.abc import Mapping, Sequence
from typing import Any

from torch import Tensor

from torch_geometric.data.data import BaseData
from torch_geometric.data.storage import BaseStorage
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import narrow


def separate(cls, batch: BaseData, idx: int, slice_dict: Any,
             inc_dict: Any = None, decrement: bool = True) -> BaseData:
    # Separates the individual element from a `batch` at index `idx`.
    # `separate` can handle both homogeneous and heterogeneous data objects by
    # individually separating all their stores.
    # In addition, `separate` can handle nested data structures such as
    # dictionaries and lists.

    data = cls().stores_as(batch)

    # We iterate over each storage object and recursively separate all its
    # attributes:
    for batch_store, data_store in zip(batch.stores, data.stores):
        key = batch_store._key
        if key is not None:
            attrs = slice_dict[key].keys()
        else:
            attrs = set(batch_store.keys())
            attrs = [attr for attr in slice_dict.keys() if attr in attrs]
        for attr in attrs:
            if key is not None:
                slices = slice_dict[key][attr]
                incs = inc_dict[key][attr] if decrement else None
            else:
                slices = slice_dict[attr]
                incs = inc_dict[attr] if decrement else None
            data_store[attr] = _separate(attr, batch_store[attr], idx, slices,
                                         incs, batch, batch_store, decrement)

        # The `num_nodes` attribute needs special treatment, as we cannot infer
        # the real number of nodes from the total number of nodes alone:
        if hasattr(batch_store, '_num_nodes'):
            data_store.num_nodes = batch_store._num_nodes[idx]

    return data


def _separate(
    key: str,
    value: Any,
    idx: int,
    slices: Any,
    incs: Any,
    batch: BaseData,
    store: BaseStorage,
    decrement: bool,
) -> Any:

    if isinstance(value, Tensor):
        # Narrow a `torch.Tensor` based on `slices`.
        # NOTE: We need to take care of decrementing elements appropriately.
        key = str(key)
        cat_dim = batch.__cat_dim__(key, value, store)
        start, end = int(slices[idx]), int(slices[idx + 1])
        value = narrow(value, cat_dim or 0, start, end - start)
        value = value.squeeze(0) if cat_dim is None else value
        if (decrement and incs is not None
                and (incs.dim() > 1 or int(incs[idx]) != 0)):
            value = value - incs[idx].to(value.device)
        return value

    elif isinstance(value, SparseTensor) and decrement:
        # Narrow a `SparseTensor` based on `slices`.
        # NOTE: `cat_dim` may return a tuple to allow for diagonal stacking.
        key = str(key)
        cat_dim = batch.__cat_dim__(key, value, store)
        cat_dims = (cat_dim, ) if isinstance(cat_dim, int) else cat_dim
        for i, dim in enumerate(cat_dims):
            start, end = int(slices[idx][i]), int(slices[idx + 1][i])
            value = value.narrow(dim, start, end - start)
        return value

    elif isinstance(value, Mapping):
        # Recursively separate elements of dictionaries.
        return {
            key: _separate(key, elem, idx, slices[key],
                           incs[key] if decrement else None, batch, store,
                           decrement)
            for key, elem in value.items()
        }

    elif (isinstance(value, Sequence) and isinstance(value[0], Sequence)
          and not isinstance(value[0], str) and len(value[0]) > 0
          and isinstance(value[0][0], (Tensor, SparseTensor))
          and isinstance(slices, Sequence)):
        # Recursively separate elements of lists of lists.
        return [elem[idx] for elem in value]

    elif (isinstance(value, Sequence) and not isinstance(value, str)
          and isinstance(value[0], (Tensor, SparseTensor))
          and isinstance(slices, Sequence)):
        # Recursively separate elements of lists of Tensors/SparseTensors.
        return [
            _separate(key, elem, idx, slices[i],
                      incs[i] if decrement else None, batch, store, decrement)
            for i, elem in enumerate(value)
        ]

    else:
        return value[idx]
pytorch/torch_geometric/data/storage.py000066400000000000000000000612351444262304300206630ustar00rootroot00000000000000import copy
import warnings
import weakref
from collections import defaultdict, namedtuple
from collections.abc import Mapping, MutableMapping, Sequence
from enum import Enum
from typing import (
    Any,
    Callable,
    Dict,
    Iterable,
    List,
    NamedTuple,
    Optional,
    Set,
    Tuple,
    Union,
)

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data.view import ItemsView, KeysView, ValuesView
from torch_geometric.typing import EdgeType, NodeType, SparseTensor
from torch_geometric.utils import (
    coalesce,
    contains_isolated_nodes,
    is_torch_sparse_tensor,
    is_undirected,
)

N_KEYS = {'x', 'feat', 'pos', 'batch', 'node_type', 'n_id'}
E_KEYS = {'edge_index', 'edge_weight', 'edge_attr', 'edge_type', 'e_id'}


class AttrType(Enum):
    NODE = 'NODE'
    EDGE = 'EDGE'
    OTHER = 'OTHER'


class BaseStorage(MutableMapping):
    # This class wraps a Python dictionary and extends it as follows:
    # 1. It allows attribute assignments, e.g.:
    #    `storage.x = ...` in addition to `storage['x'] = ...`
    # 2. It allows private attributes that are not exposed to the user, e.g.:
    #    `storage._{key} = ...` and accessible via `storage._{key}`
    # 3. It holds an (optional) weak reference to its parent object, e.g.:
    #    `storage._parent = weakref.ref(parent)`
    # 4. It allows iterating over only a subset of keys, e.g.:
    #    `storage.values('x', 'y')` or `storage.items('x', 'y')
    # 5. It adds additional PyTorch Tensor functionality, e.g.:
    #    `storage.cpu()`, `storage.cuda()` or `storage.share_memory_()`.
    def __init__(self, _mapping: Optional[Dict[str, Any]] = None, **kwargs):
        super().__init__()
        self._mapping = {}
        for key, value in (_mapping or {}).items():
            setattr(self, key, value)
        for key, value in kwargs.items():
            setattr(self, key, value)

    @property
    def _key(self) -> Any:
        return None

    def _pop_cache(self, key: str):
        for cache in getattr(self, '_cached_attr', {}).values():
            cache.discard(key)

    def __len__(self) -> int:
        return len(self._mapping)

    def __getattr__(self, key: str) -> Any:
        if key == '_mapping':
            self._mapping = {}
            return self._mapping
        try:
            return self[key]
        except KeyError:
            raise AttributeError(
                f"'{self.__class__.__name__}' object has no attribute '{key}'")

    def __setattr__(self, key: str, value: Any):
        propobj = getattr(self.__class__, key, None)
        if propobj is not None and getattr(propobj, 'fset', None) is not None:
            propobj.fset(self, value)
        elif key == '_parent':
            self.__dict__[key] = weakref.ref(value)
        elif key[:1] == '_':
            self.__dict__[key] = value
        else:
            self._pop_cache(key)
            self[key] = value

    def __delattr__(self, key: str):
        if key[:1] == '_':
            del self.__dict__[key]
        else:
            self._pop_cache(key)
            del self[key]

    def __getitem__(self, key: str) -> Any:
        return self._mapping[key]

    def __setitem__(self, key: str, value: Any):
        self._pop_cache(key)
        if value is None and key in self._mapping:
            del self._mapping[key]
        elif value is not None:
            self._mapping[key] = value

    def __delitem__(self, key: str):
        if key in self._mapping:
            self._pop_cache(key)
            del self._mapping[key]

    def __iter__(self) -> Iterable:
        return iter(self._mapping)

    def __copy__(self):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out._mapping = copy.copy(out._mapping)
        return out

    def __deepcopy__(self, memo):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out._mapping = copy.deepcopy(out._mapping, memo)
        return out

    def __getstate__(self) -> Dict[str, Any]:
        out = self.__dict__.copy()

        _parent = out.get('_parent', None)
        if _parent is not None:
            out['_parent'] = _parent()

        return out

    def __setstate__(self, mapping: Dict[str, Any]):
        for key, value in mapping.items():
            self.__dict__[key] = value

        _parent = self.__dict__.get('_parent', None)
        if _parent is not None:
            self.__dict__['_parent'] = weakref.ref(_parent)

    def __repr__(self) -> str:
        return repr(self._mapping)

    # Allow iterating over subsets ############################################

    # In contrast to standard `keys()`, `values()` and `items()` functions of
    # Python dictionaries, we allow to only iterate over a subset of items
    # denoted by a list of keys `args`.
    # This is especially useful for adding PyTorch Tensor functionality to the
    # storage object, e.g., in case we only want to transfer a subset of keys
    # to the GPU (i.e. the ones that are relevant to the deep learning model).

    def keys(self, *args: List[str]) -> KeysView:
        return KeysView(self._mapping, *args)

    def values(self, *args: List[str]) -> ValuesView:
        return ValuesView(self._mapping, *args)

    def items(self, *args: List[str]) -> ItemsView:
        return ItemsView(self._mapping, *args)

    def apply_(self, func: Callable, *args: List[str]):
        r"""Applies the in-place function :obj:`func`, either to all attributes
        or only the ones given in :obj:`*args`."""
        for value in self.values(*args):
            recursive_apply_(value, func)
        return self

    def apply(self, func: Callable, *args: List[str]):
        r"""Applies the function :obj:`func`, either to all attributes or only
        the ones given in :obj:`*args`."""
        for key, value in self.items(*args):
            self[key] = recursive_apply(value, func)
        return self

    # Additional functionality ################################################

    def get(self, key: str, value: Optional[Any] = None) -> Any:
        return self._mapping.get(key, value)

    def to_dict(self) -> Dict[str, Any]:
        r"""Returns a dictionary of stored key/value pairs."""
        out_dict = copy.copy(self._mapping)
        # Needed to preserve individual `num_nodes` attributes when calling
        # `BaseData.collate`.
        # TODO (matthias) Try to make this more generic.
        if '_num_nodes' in self.__dict__:
            out_dict['_num_nodes'] = self.__dict__['_num_nodes']
        return out_dict

    def to_namedtuple(self) -> NamedTuple:
        r"""Returns a :obj:`NamedTuple` of stored key/value pairs."""
        field_names = list(self.keys())
        typename = f'{self.__class__.__name__}Tuple'
        StorageTuple = namedtuple(typename, field_names)
        return StorageTuple(*[self[key] for key in field_names])

    def clone(self, *args: List[str]):
        r"""Performs a deep-copy of the object."""
        return copy.deepcopy(self)

    def contiguous(self, *args: List[str]):
        r"""Ensures a contiguous memory layout, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.contiguous(), *args)

    def to(self, device: Union[int, str], *args: List[str],
           non_blocking: bool = False):
        r"""Performs tensor dtype and/or device conversion, either for all
        attributes or only the ones given in :obj:`*args`."""
        return self.apply(
            lambda x: x.to(device=device, non_blocking=non_blocking), *args)

    def cpu(self, *args: List[str]):
        r"""Copies attributes to CPU memory, either for all attributes or only
        the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.cpu(), *args)

    def cuda(self, device: Optional[Union[int, str]] = None, *args: List[str],
             non_blocking: bool = False):  # pragma: no cover
        r"""Copies attributes to CUDA memory, either for all attributes or only
        the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.cuda(device, non_blocking=non_blocking),
                          *args)

    def pin_memory(self, *args: List[str]):  # pragma: no cover
        r"""Copies attributes to pinned memory, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.pin_memory(), *args)

    def share_memory_(self, *args: List[str]):
        r"""Moves attributes to shared memory, either for all attributes or
        only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.share_memory_(), *args)

    def detach_(self, *args: List[str]):
        r"""Detaches attributes from the computation graph, either for all
        attributes or only the ones given in :obj:`*args`."""
        return self.apply(lambda x: x.detach_(), *args)

    def detach(self, *args: List[str]):
        r"""Detaches attributes from the computation graph by creating a new
        tensor, either for all attributes or only the ones given in
        :obj:`*args`."""
        return self.apply(lambda x: x.detach(), *args)

    def requires_grad_(self, *args: List[str], requires_grad: bool = True):
        r"""Tracks gradient computation, either for all attributes or only the
        ones given in :obj:`*args`."""
        return self.apply(
            lambda x: x.requires_grad_(requires_grad=requires_grad), *args)

    def record_stream(self, stream: torch.cuda.Stream, *args: List[str]):
        r"""Ensures that the tensor memory is not reused for another tensor
        until all current work queued on :obj:`stream` has been completed,
        either for all attributes or only the ones given in :obj:`*args`."""
        return self.apply_(lambda x: x.record_stream(stream), *args)


class NodeStorage(BaseStorage):
    @property
    def _key(self) -> NodeType:
        key = self.__dict__.get('_key', None)
        if key is None or not isinstance(key, str):
            raise ValueError("'_key' does not denote a valid node type")
        return key

    @property
    def can_infer_num_nodes(self):
        keys = set(self.keys())
        num_node_keys = {
            'num_nodes', 'x', 'pos', 'batch', 'adj', 'adj_t', 'edge_index',
            'face'
        }
        if len(keys & num_node_keys) > 0:
            return True
        elif len([key for key in keys if 'node' in key]) > 0:
            return True
        else:
            return False

    @property
    def num_nodes(self) -> Optional[int]:
        # We sequentially access attributes that reveal the number of nodes.
        if 'num_nodes' in self:
            return self['num_nodes']
        for key, value in self.items():
            if isinstance(value, (Tensor, np.ndarray)) and key in N_KEYS:
                cat_dim = self._parent().__cat_dim__(key, value, self)
                return value.shape[cat_dim]
        for key, value in self.items():
            if isinstance(value, (Tensor, np.ndarray)) and 'node' in key:
                cat_dim = self._parent().__cat_dim__(key, value, self)
                return value.shape[cat_dim]
        if 'adj' in self and isinstance(self.adj, SparseTensor):
            return self.adj.size(0)
        if 'adj_t' in self and isinstance(self.adj_t, SparseTensor):
            return self.adj_t.size(1)
        warnings.warn(
            f"Unable to accurately infer 'num_nodes' from the attribute set "
            f"'{set(self.keys())}'. Please explicitly set 'num_nodes' as an "
            f"attribute of " +
            ("'data'" if self._key is None else f"'data[{self._key}]'") +
            " to suppress this warning")
        if 'edge_index' in self and isinstance(self.edge_index, Tensor):
            if self.edge_index.numel() > 0:
                return int(self.edge_index.max()) + 1
            else:
                return 0
        if 'face' in self and isinstance(self.face, Tensor):
            if self.face.numel() > 0:
                return int(self.face.max()) + 1
            else:
                return 0
        return None

    @property
    def num_node_features(self) -> int:
        if 'x' in self and isinstance(self.x, (Tensor, np.ndarray)):
            return 1 if self.x.ndim == 1 else self.x.shape[-1]
        if 'x' in self and isinstance(self.x, SparseTensor):
            return 1 if self.x.dim() == 1 else self.x.size(-1)
        return 0

    @property
    def num_features(self) -> int:
        return self.num_node_features

    def is_node_attr(self, key: str) -> bool:
        if '_cached_attr' not in self.__dict__:
            self._cached_attr: Dict[AttrType, Set[str]] = defaultdict(set)

        if key in self._cached_attr[AttrType.NODE]:
            return True
        if key in self._cached_attr[AttrType.OTHER]:
            return False

        value = self[key]

        if isinstance(value, (list, tuple)) and len(value) == self.num_nodes:
            self._cached_attr[AttrType.NODE].add(key)
            return True

        if not isinstance(value, (Tensor, np.ndarray)):
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        if value.ndim == 0:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        cat_dim = self._parent().__cat_dim__(key, value, self)
        if value.shape[cat_dim] != self.num_nodes:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        self._cached_attr[AttrType.NODE].add(key)
        return True

    def is_edge_attr(self, key: str) -> bool:
        return False

    def node_attrs(self) -> List[str]:
        return [key for key in self.keys() if self.is_node_attr(key)]


class EdgeStorage(BaseStorage):
    r"""We support multiple ways to store edge connectivity in a
    :class:`EdgeStorage` object:

    * :obj:`edge_index`: A :class:`torch.LongTensor` holding edge indices in
      COO format with shape :obj:`[2, num_edges]` (the default format)

    * :obj:`adj`: A :class:`torch_sparse.SparseTensor` holding edge indices in
      a sparse format, supporting both COO and CSR format.

    * :obj:`adj_t`: A **transposed** :class:`torch_sparse.SparseTensor` holding
      edge indices in a sparse format, supporting both COO and CSR format.
      This is the most efficient one for graph-based deep learning models as
      indices are sorted based on target nodes.
    """
    @property
    def _key(self) -> EdgeType:
        key = self.__dict__.get('_key', None)
        if key is None or not isinstance(key, tuple) or not len(key) == 3:
            raise ValueError("'_key' does not denote a valid edge type")
        return key

    @property
    def edge_index(self) -> Tensor:
        if 'edge_index' in self:
            return self['edge_index']
        if 'adj' in self and isinstance(self.adj, SparseTensor):
            return torch.stack(self.adj.coo()[:2], dim=0)
        if 'adj_t' in self and isinstance(self.adj_t, SparseTensor):
            return torch.stack(self.adj_t.coo()[:2][::-1], dim=0)
        raise AttributeError(
            f"'{self.__class__.__name__}' object has no attribute "
            f"'edge_index', 'adj' or 'adj_t'")

    @property
    def num_edges(self) -> int:
        # We sequentially access attributes that reveal the number of edges.
        if 'num_edges' in self:
            return self['num_edges']
        for key, value in self.items():
            if isinstance(value, (Tensor, np.ndarray)) and key in E_KEYS:
                cat_dim = self._parent().__cat_dim__(key, value, self)
                return value.shape[cat_dim]
        for key, value in self.items():
            if isinstance(value, (Tensor, np.ndarray)) and 'edge' in key:
                cat_dim = self._parent().__cat_dim__(key, value, self)
                return value.shape[cat_dim]
        for value in self.values('adj', 'adj_t'):
            if isinstance(value, SparseTensor):
                return value.nnz()
            elif is_torch_sparse_tensor(value):
                return value._nnz()
        return 0

    @property
    def num_edge_features(self) -> int:
        if ('edge_attr' in self and isinstance(self.edge_attr,
                                               (Tensor, np.ndarray))):
            return 1 if self.edge_attr.ndim == 1 else self.edge_attr.shape[-1]
        return 0

    @property
    def num_features(self) -> int:
        return self.num_edge_features

    def size(
        self, dim: Optional[int] = None
    ) -> Union[Tuple[Optional[int], Optional[int]], Optional[int]]:

        if self._key is None:
            raise NameError("Unable to infer 'size' without explicit "
                            "'_key' assignment")

        size = (self._parent()[self._key[0]].num_nodes,
                self._parent()[self._key[-1]].num_nodes)

        return size if dim is None else size[dim]

    def is_node_attr(self, key: str) -> bool:
        return False

    def is_edge_attr(self, key: str) -> bool:
        if '_cached_attr' not in self.__dict__:
            self._cached_attr: Dict[AttrType, Set[str]] = defaultdict(set)

        if key in self._cached_attr[AttrType.EDGE]:
            return True
        if key in self._cached_attr[AttrType.OTHER]:
            return False

        value = self[key]

        if isinstance(value, (list, tuple)) and len(value) == self.num_edges:
            self._cached_attr[AttrType.EDGE].add(key)
            return True

        if not isinstance(value, (Tensor, np.ndarray)):
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        if value.ndim == 0:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        cat_dim = self._parent().__cat_dim__(key, value, self)
        if value.shape[cat_dim] != self.num_edges:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        self._cached_attr[AttrType.EDGE].add(key)
        return True

    def edge_attrs(self) -> List[str]:
        return [key for key in self.keys() if self.is_edge_attr(key)]

    def is_coalesced(self) -> bool:
        for value in self.values('adj', 'adj_t'):
            return value.is_coalesced()

        if 'edge_index' in self:
            new_edge_index = coalesce(
                self.edge_index,
                num_nodes=max(self.size(0), self.size(1)),
            )
            return (self.edge_index.numel() == new_edge_index.numel()
                    and torch.equal(self.edge_index, new_edge_index))

        return True

    def coalesce(self, reduce: str = 'sum'):
        for key, value in self.items('adj', 'adj_t'):
            self[key] = value.coalesce(reduce)

        if 'edge_index' in self:
            if 'edge_attr' in self:
                self.edge_index, self.edge_attr = coalesce(
                    self.edge_index,
                    self.edge_attr,
                    num_nodes=max(self.size(0), self.size(1)),
                )
            else:
                self.edge_index = coalesce(
                    self.edge_index,
                    num_nodes=max(self.size(0), self.size(1)),
                )

        return self

    def has_isolated_nodes(self) -> bool:
        edge_index, num_nodes = self.edge_index, self.size(1)
        if num_nodes is None:
            raise NameError("Unable to infer 'num_nodes'")
        if self.is_bipartite():
            return torch.unique(edge_index[1]).numel() < num_nodes
        else:
            return contains_isolated_nodes(edge_index, num_nodes)

    def has_self_loops(self) -> bool:
        if self.is_bipartite():
            return False
        edge_index = self.edge_index
        return int((edge_index[0] == edge_index[1]).sum()) > 0

    def is_undirected(self) -> bool:
        if self.is_bipartite():
            return False

        for value in self.values('adj', 'adj_t'):
            return value.is_symmetric()

        edge_index = self.edge_index
        edge_attr = self.edge_attr if 'edge_attr' in self else None
        return is_undirected(edge_index, edge_attr, num_nodes=self.size(0))

    def is_directed(self) -> bool:
        return not self.is_undirected()

    def is_bipartite(self) -> bool:
        return self._key is not None and self._key[0] != self._key[-1]


class GlobalStorage(NodeStorage, EdgeStorage):
    @property
    def _key(self) -> Any:
        return None

    @property
    def num_features(self) -> int:
        return self.num_node_features

    def size(
        self, dim: Optional[int] = None
    ) -> Union[Tuple[Optional[int], Optional[int]], Optional[int]]:
        size = (self.num_nodes, self.num_nodes)
        return size if dim is None else size[dim]

    def is_node_attr(self, key: str) -> bool:
        if '_cached_attr' not in self.__dict__:
            self._cached_attr: Dict[AttrType, Set[str]] = defaultdict(set)

        if key in self._cached_attr[AttrType.NODE]:
            return True
        if key in self._cached_attr[AttrType.EDGE]:
            return False
        if key in self._cached_attr[AttrType.OTHER]:
            return False

        value = self[key]

        if isinstance(value, (list, tuple)) and len(value) == self.num_nodes:
            self._cached_attr[AttrType.NODE].add(key)
            return True

        if not isinstance(value, (Tensor, np.ndarray)):
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        if value.ndim == 0:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        cat_dim = self._parent().__cat_dim__(key, value, self)
        num_nodes, num_edges = self.num_nodes, self.num_edges

        if value.shape[cat_dim] != num_nodes:
            if value.shape[cat_dim] == num_edges:
                self._cached_attr[AttrType.EDGE].add(key)
            else:
                self._cached_attr[AttrType.OTHER].add(key)
            return False

        if num_nodes != num_edges:
            self._cached_attr[AttrType.NODE].add(key)
            return True

        if 'edge' not in key:
            self._cached_attr[AttrType.NODE].add(key)
            return True
        else:
            self._cached_attr[AttrType.EDGE].add(key)
            return False

    def is_edge_attr(self, key: str) -> bool:
        if '_cached_attr' not in self.__dict__:
            self._cached_attr: Dict[AttrType, Set[str]] = defaultdict(set)

        if key in self._cached_attr[AttrType.EDGE]:
            return True
        if key in self._cached_attr[AttrType.NODE]:
            return False
        if key in self._cached_attr[AttrType.OTHER]:
            return False

        value = self[key]

        if isinstance(value, (list, tuple)) and len(value) == self.num_edges:
            self._cached_attr[AttrType.EDGE].add(key)
            return True

        if not isinstance(value, (Tensor, np.ndarray)):
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        if value.ndim == 0:
            self._cached_attr[AttrType.OTHER].add(key)
            return False

        cat_dim = self._parent().__cat_dim__(key, value, self)
        num_nodes, num_edges = self.num_nodes, self.num_edges

        if value.shape[cat_dim] != num_edges:
            if value.shape[cat_dim] == num_nodes:
                self._cached_attr[AttrType.NODE].add(key)
            else:
                self._cached_attr[AttrType.OTHER].add(key)
            return False

        if num_edges != num_nodes:
            self._cached_attr[AttrType.EDGE].add(key)
            return True

        if 'edge' in key:
            self._cached_attr[AttrType.EDGE].add(key)
            return True
        else:
            self._cached_attr[AttrType.NODE].add(key)
            return False


def recursive_apply_(data: Any, func: Callable):
    if isinstance(data, Tensor):
        func(data)
    elif isinstance(data, tuple) and hasattr(data, '_fields'):  # namedtuple
        for value in data:
            recursive_apply_(value, func)
    elif isinstance(data, Sequence) and not isinstance(data, str):
        for value in data:
            recursive_apply_(value, func)
    elif isinstance(data, Mapping):
        for value in data.values():
            recursive_apply_(value, func)
    else:
        try:
            func(data)
        except:  # noqa
            pass


def recursive_apply(data: Any, func: Callable) -> Any:
    if isinstance(data, Tensor):
        return func(data)
    elif isinstance(data, torch.nn.utils.rnn.PackedSequence):
        return func(data)
    elif isinstance(data, tuple) and hasattr(data, '_fields'):  # namedtuple
        return type(data)(*(recursive_apply(d, func) for d in data))
    elif isinstance(data, Sequence) and not isinstance(data, str):
        return [recursive_apply(d, func) for d in data]
    elif isinstance(data, Mapping):
        return {key: recursive_apply(data[key], func) for key in data}
    else:
        try:
            return func(data)
        except:  # noqa
            return data
pytorch/torch_geometric/data/summary.py000066400000000000000000000121431444262304300207060ustar00rootroot00000000000000from collections import defaultdict
from dataclasses import dataclass
from typing import Dict, List, Optional, Union

import torch
from tqdm import tqdm

from torch_geometric.data import Dataset, HeteroData


@dataclass
class Stats:
    mean: float
    std: float
    min: float
    quantile25: float
    median: float
    quantile75: float
    max: float

    @classmethod
    def from_data(cls, data: Union[List[int], List[float], torch.Tensor]):
        if not isinstance(data, torch.Tensor):
            data = torch.tensor(data)
        data = data.to(torch.float)

        return cls(
            mean=data.mean().item(),
            std=data.std().item(),
            min=data.min().item(),
            quantile25=data.quantile(0.25).item(),
            median=data.median().item(),
            quantile75=data.quantile(0.75).item(),
            max=data.max().item(),
        )


@dataclass(repr=False)
class Summary:
    name: str
    num_graphs: int
    num_nodes: Stats
    num_edges: Stats
    num_nodes_per_type: Optional[Dict[str, Stats]] = None
    num_edges_per_type: Optional[Dict[str, Stats]] = None

    @classmethod
    def from_dataset(
        cls,
        dataset: Dataset,
        progress_bar: Optional[bool] = None,
        per_type: bool = True,
    ):
        r"""Creates a summary of a :class:`~torch_geometric.data.Dataset`
        object.

        Args:
            dataset (Dataset): The dataset.
            progress_bar (bool, optional). If set to :obj:`True`, will show a
                progress bar during stats computation. If set to :obj:`None`,
                will automatically decide whether to show a progress bar based
                on dataset size. (default: :obj:`None`)
            per_type (bool, optional). If set to :obj:`True`, will separate
                statistics per node and edge type (only applicable in
                heterogeneous graph datasets). (default: :obj:`True`)
        """
        name = dataset.__class__.__name__

        if progress_bar is None:
            progress_bar = len(dataset) >= 10000

        if progress_bar:
            dataset = tqdm(dataset)

        num_nodes, num_edges = [], []
        num_nodes_per_type = defaultdict(list)
        num_edges_per_type = defaultdict(list)

        for data in dataset:
            num_nodes.append(data.num_nodes)
            num_edges.append(data.num_edges)

            if per_type and isinstance(data, HeteroData):
                for node_type in data.node_types:
                    num_nodes_per_type[node_type].append(
                        data[node_type].num_nodes)
                for edge_type in data.edge_types:
                    num_edges_per_type[edge_type].append(
                        data[edge_type].num_edges)

        if len(num_nodes_per_type) > 0:
            num_nodes_per_type = {
                node_type: Stats.from_data(num_nodes_list)
                for node_type, num_nodes_list in num_nodes_per_type.items()
            }
        else:
            num_nodes_per_type = None

        if len(num_edges_per_type) > 0:
            num_edges_per_type = {
                edge_type: Stats.from_data(num_edges_list)
                for edge_type, num_edges_list in num_edges_per_type.items()
            }
        else:
            num_edges_per_type = None

        return cls(
            name=name,
            num_graphs=len(dataset),
            num_nodes=Stats.from_data(num_nodes),
            num_edges=Stats.from_data(num_edges),
            num_nodes_per_type=num_nodes_per_type,
            num_edges_per_type=num_edges_per_type,
        )

    def __repr__(self) -> str:
        from tabulate import tabulate

        body = f'{self.name} (#graphs={self.num_graphs}):\n'

        content = [['', '#nodes', '#edges']]
        stats = [self.num_nodes, self.num_edges]
        for field in Stats.__dataclass_fields__:
            row = [field] + [f'{getattr(s, field):.1f}' for s in stats]
            content.append(row)
        body += tabulate(content, headers='firstrow', tablefmt='psql')

        if self.num_nodes_per_type is not None:
            content = [['']]
            content[0] += list(self.num_nodes_per_type.keys())

            for field in Stats.__dataclass_fields__:
                row = [field] + [
                    f'{getattr(s, field):.1f}'
                    for s in self.num_nodes_per_type.values()
                ]
                content.append(row)
            body += "\nNumber of nodes per node type:\n"
            body += tabulate(content, headers='firstrow', tablefmt='psql')

        if self.num_edges_per_type is not None:
            content = [['']]
            content[0] += [
                f"({', '.join(edge_type)})"
                for edge_type in self.num_edges_per_type.keys()
            ]

            for field in Stats.__dataclass_fields__:
                row = [field] + [
                    f'{getattr(s, field):.1f}'
                    for s in self.num_edges_per_type.values()
                ]
                content.append(row)
            body += "\nNumber of edges per edge type:\n"
            body += tabulate(content, headers='firstrow', tablefmt='psql')

        return body
pytorch/torch_geometric/data/temporal.py000066400000000000000000000230651444262304300210410ustar00rootroot00000000000000import copy
from typing import (
    Any,
    Dict,
    Iterable,
    List,
    NamedTuple,
    Optional,
    Tuple,
    Union,
)

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data.data import BaseData, size_repr
from torch_geometric.data.storage import (
    BaseStorage,
    EdgeStorage,
    GlobalStorage,
    NodeStorage,
)


class TemporalData(BaseData):
    r"""A data object composed by a stream of events describing a temporal
    graph.
    The :class:`~torch_geometric.data.TemporalData` object can hold a list of
    events (that can be understood as temporal edges in a graph) with
    structured messages.
    An event is composed by a source node, a destination node, a timestamp
    and a message. Any *Continuous-Time Dynamic Graph* (CTDG) can be
    represented with these four values.

    In general, :class:`~torch_geometric.data.TemporalData` tries to mimic
    the behavior of a regular Python dictionary.
    In addition, it provides useful functionality for analyzing graph
    structures, and provides basic PyTorch tensor functionalities.

    .. code-block:: python

        from torch import Tensor
        from torch_geometric.data import TemporalData

        events = TemporalData(
            src=Tensor([1,2,3,4]),
            dst=Tensor([2,3,4,5]),
            t=Tensor([1000,1010,1100,2000]),
            msg=Tensor([1,1,0,0])
        )

        # Add additional arguments to `events`:
        events.y = Tensor([1,1,0,0])

        # It is also possible to set additional arguments in the constructor
        events = TemporalData(
            ...,
            y=Tensor([1,1,0,0])
        )

        # Get the number of events:
        events.num_events
        >>> 4

        # Analyzing the graph structure:
        events.num_nodes
        >>> 5

        # PyTorch tensor functionality:
        events = events.pin_memory()
        events = events.to('cuda:0', non_blocking=True)

    Args:
        src (torch.Tensor, optional): A list of source nodes for the events
            with shape :obj:`[num_events]`. (default: :obj:`None`)
        dst (torch.Tensor, optional): A list of destination nodes for the
            events with shape :obj:`[num_events]`. (default: :obj:`None`)
        t (torch.Tensor, optional): The timestamps for each event with shape
            :obj:`[num_events]`. (default: :obj:`None`)
        msg (torch.Tensor, optional): Messages feature matrix with shape
            :obj:`[num_events, num_msg_features]`. (default: :obj:`None`)
        **kwargs (optional): Additional attributes.

    .. note::
        The shape of :obj:`src`, :obj:`dst`, :obj:`t` and the first dimension
        of :obj`msg` should be the same (:obj:`num_events`).
    """
    def __init__(
        self,
        src: Optional[Tensor] = None,
        dst: Optional[Tensor] = None,
        t: Optional[Tensor] = None,
        msg: Optional[Tensor] = None,
        **kwargs,
    ):
        super().__init__()
        self.__dict__['_store'] = GlobalStorage(_parent=self)

        self.src = src
        self.dst = dst
        self.t = t
        self.msg = msg

        for key, value in kwargs.items():
            setattr(self, key, value)

    def index_select(self, idx: Any) -> 'TemporalData':
        idx = prepare_idx(idx)
        data = copy.copy(self)
        for key, value in data._store.items():
            if value.size(0) == self.num_events:
                data[key] = value[idx]
        return data

    def __getitem__(self, idx: Any) -> Any:
        if isinstance(idx, str):
            return self._store[idx]
        return self.index_select(idx)

    def __setitem__(self, key: str, value: Any):
        """Sets the attribute :obj:`key` to :obj:`value`."""
        self._store[key] = value

    def __delitem__(self, key: str):
        if key in self._store:
            del self._store[key]

    def __getattr__(self, key: str) -> Any:
        if '_store' not in self.__dict__:
            raise RuntimeError(
                "The 'data' object was created by an older version of PyG. "
                "If this error occurred while loading an already existing "
                "dataset, remove the 'processed/' directory in the dataset's "
                "root folder and try again.")
        return getattr(self._store, key)

    def __setattr__(self, key: str, value: Any):
        setattr(self._store, key, value)

    def __delattr__(self, key: str):
        delattr(self._store, key)

    def __iter__(self) -> Iterable:
        for i in range(self.num_events):
            yield self[i]

    def __len__(self) -> int:
        return self.num_events

    def __call__(self, *args: List[str]) -> Iterable:
        for key, value in self._store.items(*args):
            yield key, value

    def __copy__(self):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = value
        out.__dict__['_store'] = copy.copy(self._store)
        out._store._parent = out
        return out

    def __deepcopy__(self, memo):
        out = self.__class__.__new__(self.__class__)
        for key, value in self.__dict__.items():
            out.__dict__[key] = copy.deepcopy(value, memo)
        out._store._parent = out
        return out

    def stores_as(self, data: 'TemporalData'):
        return self

    @property
    def stores(self) -> List[BaseStorage]:
        return [self._store]

    @property
    def node_stores(self) -> List[NodeStorage]:
        return [self._store]

    @property
    def edge_stores(self) -> List[EdgeStorage]:
        return [self._store]

    def to_dict(self) -> Dict[str, Any]:
        return self._store.to_dict()

    def to_namedtuple(self) -> NamedTuple:
        return self._store.to_namedtuple()

    def debug(self):
        pass  # TODO

    @property
    def num_nodes(self) -> int:
        r"""Returns the number of nodes in the graph."""
        return max(int(self.src.max()), int(self.dst.max())) + 1

    @property
    def num_events(self) -> int:
        r"""Returns the number of events loaded.

        .. note::
            In a :class:`~torch_geometric.data.TemporalData`, each row denotes
            an event.
            Thus, they can be also understood as edges.
        """
        return self.src.size(0)

    @property
    def num_edges(self) -> int:
        r"""Alias for :meth:`~torch_geometric.data.TemporalData.num_events`."""
        return self.num_events

    @property
    def edge_index(self) -> Tensor:
        r"""Returns the edge indices of the graph."""
        if 'edge_index' in self:
            return self._store['edge_index']
        if self.src is not None and self.dst is not None:
            return torch.stack([self.src, self.dst], dim=0)
        raise ValueError(f"{self.__class__.__name__} does not contain "
                         f"'edge_index' information")

    def size(
        self, dim: Optional[int] = None
    ) -> Union[Tuple[Optional[int], Optional[int]], Optional[int]]:
        r"""Returns the size of the adjacency matrix induced by the graph."""
        size = (int(self.src.max()), int(self.dst.max()))
        return size if dim is None else size[dim]

    def __cat_dim__(self, key: str, value: Any, *args, **kwargs) -> Any:
        return 0

    def __inc__(self, key: str, value: Any, *args, **kwargs) -> Any:
        if 'batch' in key:
            return int(value.max()) + 1
        elif key in ['src', 'dst']:
            return self.num_nodes
        else:
            return 0

    def __repr__(self) -> str:
        cls = self.__class__.__name__
        info = ', '.join([size_repr(k, v) for k, v in self._store.items()])
        return f'{cls}({info})'

    ###########################################################################

    def train_val_test_split(self, val_ratio: float = 0.15,
                             test_ratio: float = 0.15):
        r"""Splits the data in training, validation and test sets according to
        time.

        Args:
            val_ratio (float, optional): The proportion (in percents) of the
                dataset to include in the validation split.
                (default: :obj:`0.15`)
            test_ratio (float, optional): The proportion (in percents) of the
                dataset to include in the test split. (default: :obj:`0.15`)
        """
        val_time, test_time = np.quantile(
            self.t.cpu().numpy(),
            [1. - val_ratio - test_ratio, 1. - test_ratio])

        val_idx = int((self.t <= val_time).sum())
        test_idx = int((self.t <= test_time).sum())

        return self[:val_idx], self[val_idx:test_idx], self[test_idx:]

    ###########################################################################

    def coalesce(self):
        raise NotImplementedError

    def has_isolated_nodes(self) -> bool:
        raise NotImplementedError

    def has_self_loops(self) -> bool:
        raise NotImplementedError

    def is_undirected(self) -> bool:
        raise NotImplementedError

    def is_directed(self) -> bool:
        raise NotImplementedError


###############################################################################


def prepare_idx(idx):
    if isinstance(idx, int):
        return slice(idx, idx + 1)
    if isinstance(idx, (list, tuple)):
        return torch.tensor(idx)
    elif isinstance(idx, slice):
        return idx
    elif isinstance(idx, torch.Tensor) and idx.dtype == torch.long:
        return idx
    elif isinstance(idx, torch.Tensor) and idx.dtype == torch.bool:
        return idx

    raise IndexError(
        f"Only strings, integers, slices (`:`), list, tuples, and long or "
        f"bool tensors are valid indices (got '{type(idx).__name__}')")
pytorch/torch_geometric/data/view.py000066400000000000000000000020161444262304300201610ustar00rootroot00000000000000from collections.abc import Mapping
from typing import Iterable, List


class MappingView:
    def __init__(self, mapping: Mapping, *args: List[str]):
        self._mapping = mapping
        self._args = args

    def _keys(self) -> Iterable:
        if len(self._args) == 0:
            return self._mapping.keys()
        else:
            return [arg for arg in self._args if arg in self._mapping]

    def __len__(self) -> int:
        return len(self._keys())

    def __repr__(self) -> str:
        mapping = {key: self._mapping[key] for key in self._keys()}
        return f'{self.__class__.__name__}({mapping})'

    __class_getitem__ = classmethod(type([]))


class KeysView(MappingView):
    def __iter__(self) -> Iterable:
        yield from self._keys()


class ValuesView(MappingView):
    def __iter__(self) -> Iterable:
        for key in self._keys():
            yield self._mapping[key]


class ItemsView(MappingView):
    def __iter__(self):
        for key in self._keys():
            yield (key, self._mapping[key])
pytorch/torch_geometric/datasets/000077500000000000000000000000001444262304300175355ustar00rootroot00000000000000pytorch/torch_geometric/datasets/__init__.py000066400000000000000000000122641444262304300216530ustar00rootroot00000000000000# flake8: noqa

from .karate import KarateClub
from .tu_dataset import TUDataset
from .gnn_benchmark_dataset import GNNBenchmarkDataset
from .planetoid import Planetoid
from .nell import NELL
from .citation_full import CitationFull, CoraFull
from .coauthor import Coauthor
from .amazon import Amazon
from .ppi import PPI
from .reddit import Reddit
from .reddit2 import Reddit2
from .flickr import Flickr
from .yelp import Yelp
from .amazon_products import AmazonProducts
from .qm7 import QM7b
from .qm9 import QM9
from .md17 import MD17
from .zinc import ZINC
from .aqsol import AQSOL
from .molecule_net import MoleculeNet
from .entities import Entities
from .rel_link_pred_dataset import RelLinkPredDataset
from .ged_dataset import GEDDataset
from .attributed_graph_dataset import AttributedGraphDataset
from .mnist_superpixels import MNISTSuperpixels
from .faust import FAUST
from .dynamic_faust import DynamicFAUST
from .shapenet import ShapeNet
from .modelnet import ModelNet
from .coma import CoMA
from .shrec2016 import SHREC2016
from .tosca import TOSCA
from .pcpnet_dataset import PCPNetDataset
from .s3dis import S3DIS
from .geometry import GeometricShapes
from .bitcoin_otc import BitcoinOTC
from .gdelt_lite import GDELTLite
from .icews import ICEWS18
from .gdelt import GDELT
from .willow_object_class import WILLOWObjectClass
from .pascal import PascalVOCKeypoints
from .pascal_pf import PascalPF
from .snap_dataset import SNAPDataset
from .suite_sparse import SuiteSparseMatrixCollection
from .word_net import WordNet18, WordNet18RR
from .freebase import FB15k_237
from .wikics import WikiCS
from .webkb import WebKB
from .wikipedia_network import WikipediaNetwork
from .heterophilous_graph_dataset import HeterophilousGraphDataset
from .actor import Actor
from .upfd import UPFD
from .github import GitHub
from .facebook import FacebookPagePage
from .lastfm_asia import LastFMAsia
from .deezer_europe import DeezerEurope
from .gemsec import GemsecDeezer
from .twitch import Twitch
from .airports import Airports
from .lrgb import LRGBDataset
from .malnet_tiny import MalNetTiny
from .omdb import OMDB
from .polblogs import PolBlogs
from .email_eu_core import EmailEUCore
from .linkx_dataset import LINKXDataset
from .elliptic import EllipticBitcoinDataset
from .elliptic_temporal import EllipticBitcoinTemporalDataset
from .dgraph import DGraphFin
from .hydro_net import HydroNet
from .airfrans import AirfRANS
from .jodie import JODIEDataset

from .dbp15k import DBP15K
from .aminer import AMiner
from .ogb_mag import OGB_MAG
from .dblp import DBLP
from .movie_lens import MovieLens
from .movie_lens_100k import MovieLens100K
from .movie_lens_1m import MovieLens1M
from .imdb import IMDB
from .last_fm import LastFM
from .hgb_dataset import HGBDataset
from .taobao import Taobao
from .igmc_dataset import IGMCDataset
from .amazon_book import AmazonBook
from .hm import HM

from .fake import FakeDataset, FakeHeteroDataset
from .sbm_dataset import StochasticBlockModelDataset
from .sbm_dataset import RandomPartitionGraphDataset
from .mixhop_synthetic_dataset import MixHopSyntheticDataset
from .explainer_dataset import ExplainerDataset
from .infection_dataset import InfectionDataset
from .ba2motif_dataset import BA2MotifDataset
from .ba_multi_shapes import BAMultiShapesDataset
from .ba_shapes import BAShapes

import torch_geometric.datasets.utils  # noqa

homo_datasets = [
    'KarateClub',
    'TUDataset',
    'GNNBenchmarkDataset',
    'Planetoid',
    'NELL',
    'CitationFull',
    'CoraFull',
    'Coauthor',
    'Amazon',
    'PPI',
    'Reddit',
    'Reddit2',
    'Flickr',
    'Yelp',
    'AmazonProducts',
    'QM7b',
    'QM9',
    'MD17',
    'ZINC',
    'AQSOL',
    'MoleculeNet',
    'Entities',
    'RelLinkPredDataset',
    'GEDDataset',
    'AttributedGraphDataset',
    'MNISTSuperpixels',
    'FAUST',
    'DynamicFAUST',
    'ShapeNet',
    'ModelNet',
    'CoMA',
    'SHREC2016',
    'TOSCA',
    'PCPNetDataset',
    'S3DIS',
    'GeometricShapes',
    'BitcoinOTC',
    'GDELTLite',
    'ICEWS18',
    'GDELT',
    'WILLOWObjectClass',
    'PascalVOCKeypoints',
    'PascalPF',
    'SNAPDataset',
    'SuiteSparseMatrixCollection',
    'WordNet18',
    'WordNet18RR',
    'FB15k_237',
    'WikiCS',
    'WebKB',
    'WikipediaNetwork',
    'HeterophilousGraphDataset',
    'Actor',
    'UPFD',
    'GitHub',
    'FacebookPagePage',
    'LastFMAsia',
    'DeezerEurope',
    'GemsecDeezer',
    'Twitch',
    'Airports',
    'LRGBDataset',
    'MalNetTiny',
    'OMDB',
    'PolBlogs',
    'EmailEUCore',
    'LINKXDataset',
    'EllipticBitcoinDataset',
    'EllipticBitcoinTemporalDataset',
    'DGraphFin',
    'HydroNet',
    'AirfRANS',
    'JODIEDataset',
]

hetero_datasets = [
    'DBP15K',
    'AMiner',
    'OGB_MAG',
    'DBLP',
    'MovieLens',
    'MovieLens100K',
    'MovieLens1M',
    'IMDB',
    'LastFM',
    'HGBDataset',
    'Taobao',
    'IGMCDataset',
    'AmazonBook',
    'HM',
]
synthetic_datasets = [
    'FakeDataset',
    'FakeHeteroDataset',
    'StochasticBlockModelDataset',
    'RandomPartitionGraphDataset',
    'MixHopSyntheticDataset',
    'ExplainerDataset',
    'InfectionDataset',
    'BA2MotifDataset',
    'BAMultiShapesDataset',
    'BAShapes',
]

__all__ = homo_datasets + hetero_datasets + synthetic_datasets
pytorch/torch_geometric/datasets/actor.py000066400000000000000000000101741444262304300212220ustar00rootroot00000000000000from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import coalesce


class Actor(InMemoryDataset):
    r"""The actor-only induced subgraph of the film-director-actor-writer
    network used in the
    `"Geom-GCN: Geometric Graph Convolutional Networks"
    `_ paper.
    Each node corresponds to an actor, and the edge between two nodes denotes
    co-occurrence on the same Wikipedia page.
    Node features correspond to some keywords in the Wikipedia pages.
    The task is to classify the nodes into five categories in term of words of
    actor's Wikipedia.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 7,600
          - 30,019
          - 932
          - 5
    """

    url = 'https://raw.githubusercontent.com/graphdml-uiuc-jlu/geom-gcn/master'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['out1_node_feature_label.txt', 'out1_graph_edges.txt'
                ] + [f'film_split_0.6_0.2_{i}.npz' for i in range(10)]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for f in self.raw_file_names[:2]:
            download_url(f'{self.url}/new_data/film/{f}', self.raw_dir)
        for f in self.raw_file_names[2:]:
            download_url(f'{self.url}/splits/{f}', self.raw_dir)

    def process(self):
        with open(self.raw_paths[0], 'r') as f:
            data = [x.split('\t') for x in f.read().split('\n')[1:-1]]

            rows, cols = [], []
            for n_id, col, _ in data:
                col = [int(x) for x in col.split(',')]
                rows += [int(n_id)] * len(col)
                cols += col
            row, col = torch.tensor(rows), torch.tensor(cols)

            x = torch.zeros(int(row.max()) + 1, int(col.max()) + 1)
            x[row, col] = 1.

            y = torch.empty(len(data), dtype=torch.long)
            for n_id, _, label in data:
                y[int(n_id)] = int(label)

        with open(self.raw_paths[1], 'r') as f:
            data = f.read().split('\n')[1:-1]
            data = [[int(v) for v in r.split('\t')] for r in data]
            edge_index = torch.tensor(data, dtype=torch.long).t().contiguous()
            edge_index = coalesce(edge_index, num_nodes=x.size(0))

        train_masks, val_masks, test_masks = [], [], []
        for f in self.raw_paths[2:]:
            tmp = np.load(f)
            train_masks += [torch.from_numpy(tmp['train_mask']).to(torch.bool)]
            val_masks += [torch.from_numpy(tmp['val_mask']).to(torch.bool)]
            test_masks += [torch.from_numpy(tmp['test_mask']).to(torch.bool)]
        train_mask = torch.stack(train_masks, dim=1)
        val_mask = torch.stack(val_masks, dim=1)
        test_mask = torch.stack(test_masks, dim=1)

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/airfrans.py000066400000000000000000000127201444262304300217160ustar00rootroot00000000000000import json
import os
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class AirfRANS(InMemoryDataset):
    r"""The AirfRANS dataset from the `"AirfRANS: High Fidelity Computational
    Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier-Stokes
    Solutions" `_ paper, consisting of 1,000
    simulations of steady-state aerodynamics over 2D airfoils in a subsonic
    flight regime.
    The different tasks (:obj:`"full"`, :obj:`"scarce"`, :obj:`"reynolds"`,
    :obj:`"aoa"`) define the utilized training and test splits.

    Each simulation is given as a point cloud defined as the nodes of the
    simulation mesh. Each point of a point cloud is described via 5
    features: the inlet velocity (two components in meter per second), the
    distance to the airfoil (one component in meter), and the normals (two
    components in meter, set to :obj:`0` if the point is not on the airfoil).
    Each point is given a target of 4 components for the underyling regression
    task: the velocity (two components in meter per second), the pressure
    divided by the specific mass (one component in meter squared per second
    squared), the turbulent kinematic viscosity (one component in meter squared
    per second).
    Finaly, a boolean is attached to each point to inform if this point lies on
    the airfoil or not.

    A library for manipulating simulations of the dataset is available `here
    `_.

    The dataset is released under the `ODbL v1.0 License
    `_.

    .. note::

        Data objects contain no edge indices to be agnostic to the simulation
        mesh. You are free to build a graph via the
        :obj:`torch_geometric.transforms.RadiusGraph` transform.

    Args:
        root (str): Root directory where the dataset should be saved.
        task (str): The task to study (:obj:`"full"`, :obj:`"scarce"`,
            :obj:`"reynolds"`, :obj:`"aoa"`) that defines the utilized training
            and test splits.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #tasks
        * - 1,000
          - ~180,000
          - 0
          - 5
          - 4
    """
    url = 'https://data.isir.upmc.fr/extrality/pytorch_geometric/AirfRANS.zip'
    tasks = ['full', 'scarce', 'reynolds', 'aoa']

    def __init__(
        self,
        root: str,
        task: str,
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        if task not in self.tasks:
            raise ValueError(f"Expected 'task' to be in {self.tasks} "
                             f"got '{task}'")

        self.task = 'full' if task == 'scarce' and not train else task
        self.split = 'train' if train else 'test'

        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['AirfRANS.pt', 'manifest.json']

    @property
    def process(self) -> List[str]:
        return ['AirfRANS.pt', 'manifest.json']

    @property
    def processed_file_names(self) -> str:
        return f'{self.task}_{self.split}.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        with open(self.raw_paths[1], 'r') as f:
            manifest = json.load(f)
        total = manifest['full_train'] + manifest['full_test']
        partial = set(manifest[f'{self.task}_{self.split}'])

        data_list = []
        raw_data = torch.load(self.raw_paths[0])
        for k, s in enumerate(total):
            if s in partial:
                data = Data(**raw_data[k])

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue
                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'task={self.task}, split={self.split})')
pytorch/torch_geometric/datasets/airports.py000066400000000000000000000071771444262304300217660ustar00rootroot00000000000000import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import coalesce


class Airports(InMemoryDataset):
    r"""The Airports dataset from the `"struc2vec: Learning Node
    Representations from Structural Identity"
    `_ paper, where nodes denote airports
    and labels correspond to activity levels.
    Features are given by one-hot encoded node identifiers, as described in the
    `"GraLSP: Graph Neural Networks with Local Structural Patterns"
    ` `_ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"USA"`, :obj:`"Brazil"`,
            :obj:`"Europe"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    edge_url = ('https://github.com/leoribeiro/struc2vec/'
                'raw/master/graph/{}-airports.edgelist')
    label_url = ('https://github.com/leoribeiro/struc2vec/'
                 'raw/master/graph/labels-{}-airports.txt')

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        assert self.name in ['usa', 'brazil', 'europe']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        return [
            f'{self.name}-airports.edgelist',
            f'labels-{self.name}-airports.txt',
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.edge_url.format(self.name), self.raw_dir)
        download_url(self.label_url.format(self.name), self.raw_dir)

    def process(self):
        index_map, ys = {}, []
        with open(self.raw_paths[1], 'r') as f:
            data = f.read().split('\n')[1:-1]
            for i, row in enumerate(data):
                idx, y = row.split()
                index_map[int(idx)] = i
                ys.append(int(y))
        y = torch.tensor(ys, dtype=torch.long)
        x = torch.eye(y.size(0))

        edge_indices = []
        with open(self.raw_paths[0], 'r') as f:
            data = f.read().split('\n')[:-1]
            for row in data:
                src, dst = row.split()
                edge_indices.append([index_map[int(src)], index_map[int(dst)]])
        edge_index = torch.tensor(edge_indices).t().contiguous()
        edge_index = coalesce(edge_index, num_nodes=y.size(0))

        data = Data(x=x, edge_index=edge_index, y=y)
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.capitalize()}Airports()'
pytorch/torch_geometric/datasets/amazon.py000066400000000000000000000057521444262304300214050ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url
from torch_geometric.io import read_npz


class Amazon(InMemoryDataset):
    r"""The Amazon Computers and Amazon Photo networks from the
    `"Pitfalls of Graph Neural Network Evaluation"
    `_ paper.
    Nodes represent goods and edges represent that two goods are frequently
    bought together.
    Given product reviews as bag-of-words node features, the task is to
    map goods to their respective product category.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Computers"`,
            :obj:`"Photo"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Computers
          - 13,752
          - 491,722
          - 767
          - 10
        * - Photo
          - 7,650
          - 238,162
          - 745
          - 8
    """

    url = 'https://github.com/shchur/gnn-benchmark/raw/master/data/npz/'

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in ['computers', 'photo']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name.capitalize(), 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name.capitalize(), 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'amazon_electronics_{self.name.lower()}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url + self.raw_file_names, self.raw_dir)

    def process(self):
        data = read_npz(self.raw_paths[0], to_undirected=True)
        data = data if self.pre_transform is None else self.pre_transform(data)
        data, slices = self.collate([data])
        torch.save((data, slices), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}{self.name.capitalize()}()'
pytorch/torch_geometric/datasets/amazon_book.py000066400000000000000000000060451444262304300224130ustar00rootroot00000000000000from typing import Callable, List, Optional

import torch

from torch_geometric.data import HeteroData, InMemoryDataset, download_url


class AmazonBook(InMemoryDataset):
    r"""A subset of the AmazonBook rating dataset from the
    `"LightGCN: Simplifying and Powering Graph Convolution Network for
    Recommendation" `_ paper.
    This is a heterogeneous dataset consisting of 52,643 users and 91,599 books
    with approximately 2.9 million ratings between them.
    No labels or features are provided.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    url = ('https://raw.githubusercontent.com/gusye1234/LightGCN-PyTorch/'
           'master/data/amazon-book')

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0], data_cls=HeteroData)

    @property
    def raw_file_names(self) -> List[str]:
        return ['user_list.txt', 'item_list.txt', 'train.txt', 'test.txt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for name in self.raw_file_names:
            download_url(f'{self.url}/{name}', self.raw_dir)

    def process(self):
        import pandas as pd

        data = HeteroData()

        # Process number of nodes for each node type:
        node_types = ['user', 'book']
        for path, node_type in zip(self.raw_paths, node_types):
            df = pd.read_csv(path, sep=' ', header=0)
            data[node_type].num_nodes = len(df)

        # Process edge information for training and testing:
        attr_names = ['edge_index', 'edge_label_index']
        for path, attr_name in zip(self.raw_paths[2:], attr_names):
            rows, cols = [], []
            with open(path, 'r') as f:
                lines = f.readlines()
            for line in lines:
                line = line.strip().split(' ')
                for dst in line[1:]:
                    rows.append(int(line[0]))
                    cols.append(int(dst))
            index = torch.tensor([rows, cols])

            data['user', 'rates', 'book'][attr_name] = index
            if attr_name == 'edge_index':
                data['book', 'rated_by', 'user'][attr_name] = index.flip([0])

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self.save([data], self.processed_paths[0])
pytorch/torch_geometric/datasets/amazon_products.py000066400000000000000000000100031444262304300233110ustar00rootroot00000000000000import json
import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class AmazonProducts(InMemoryDataset):
    r"""The Amazon dataset from the `"GraphSAINT: Graph Sampling Based
    Inductive Learning Method" `_ paper,
    containing products and its categories.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 1,569,960
          - 264,339,468
          - 200
          - 107
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    adj_full_id = '17qhNA8H1IpbkkR-T2BmPQm8QNW5do-aa'
    feats_id = '10SW8lCvAj-kb6ckkfTOC5y0l8XXdtMxj'
    class_map_id = '1LIl4kimLfftj4-7NmValuWyCQE8AaE7P'
    role_id = '1npK9xlmbnjNkV80hK2Q68wTEVOFjnt4K'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['adj_full.npz', 'feats.npy', 'class_map.json', 'role.json']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url.format(self.adj_full_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'adj_full.npz'))

        path = download_url(self.url.format(self.feats_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'feats.npy'))

        path = download_url(self.url.format(self.class_map_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'class_map.json'))

        path = download_url(self.url.format(self.role_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'role.json'))

    def process(self):
        f = np.load(osp.join(self.raw_dir, 'adj_full.npz'))
        adj = sp.csr_matrix((f['data'], f['indices'], f['indptr']), f['shape'])
        adj = adj.tocoo()
        row = torch.from_numpy(adj.row).to(torch.long)
        col = torch.from_numpy(adj.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        x = np.load(osp.join(self.raw_dir, 'feats.npy'))
        x = torch.from_numpy(x).to(torch.float)

        ys = [-1] * x.size(0)
        with open(osp.join(self.raw_dir, 'class_map.json')) as f:
            class_map = json.load(f)
            for key, item in class_map.items():
                ys[int(key)] = item
        y = torch.tensor(ys)

        with open(osp.join(self.raw_dir, 'role.json')) as f:
            role = json.load(f)

        train_mask = torch.zeros(x.size(0), dtype=torch.bool)
        train_mask[torch.tensor(role['tr'])] = True

        val_mask = torch.zeros(x.size(0), dtype=torch.bool)
        val_mask[torch.tensor(role['va'])] = True

        test_mask = torch.zeros(x.size(0), dtype=torch.bool)
        test_mask[torch.tensor(role['te'])] = True

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/aminer.py000066400000000000000000000115161444262304300213660ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.utils import coalesce


class AMiner(InMemoryDataset):
    r"""The heterogeneous AMiner dataset from the `"metapath2vec: Scalable
    Representation Learning for Heterogeneous Networks"
    `_ paper, consisting of nodes from
    type :obj:`"paper"`, :obj:`"author"` and :obj:`"venue"`.
    Venue categories and author research interests are available as ground
    truth labels for a subset of nodes.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://www.dropbox.com/s/1bnz8r7mofx0osf/net_aminer.zip?dl=1'
    y_url = 'https://www.dropbox.com/s/nkocx16rpl4ydde/label.zip?dl=1'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'id_author.txt', 'id_conf.txt', 'paper.txt', 'paper_author.txt',
            'paper_conf.txt', 'label'
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        shutil.rmtree(self.raw_dir)
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.rename(osp.join(self.root, 'net_aminer'), self.raw_dir)
        os.unlink(path)
        path = download_url(self.y_url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        import pandas as pd

        data = HeteroData()

        # Get author labels.
        path = osp.join(self.raw_dir, 'id_author.txt')
        author = pd.read_csv(path, sep='\t', names=['idx', 'name'],
                             index_col=1)

        path = osp.join(self.raw_dir, 'label',
                        'googlescholar.8area.author.label.txt')
        df = pd.read_csv(path, sep=' ', names=['name', 'y'])
        df = df.join(author, on='name')

        data['author'].y = torch.from_numpy(df['y'].values) - 1
        data['author'].y_index = torch.from_numpy(df['idx'].values)

        # Get venue labels.
        path = osp.join(self.raw_dir, 'id_conf.txt')
        venue = pd.read_csv(path, sep='\t', names=['idx', 'name'], index_col=1)

        path = osp.join(self.raw_dir, 'label',
                        'googlescholar.8area.venue.label.txt')
        df = pd.read_csv(path, sep=' ', names=['name', 'y'])
        df = df.join(venue, on='name')

        data['venue'].y = torch.from_numpy(df['y'].values) - 1
        data['venue'].y_index = torch.from_numpy(df['idx'].values)

        # Get paper<->author connectivity.
        path = osp.join(self.raw_dir, 'paper_author.txt')
        paper_author = pd.read_csv(path, sep='\t', header=None)
        paper_author = torch.from_numpy(paper_author.values)
        paper_author = paper_author.t().contiguous()
        M, N = int(paper_author[0].max() + 1), int(paper_author[1].max() + 1)
        paper_author = coalesce(paper_author, num_nodes=max(M, N))
        data['paper'].num_nodes = M
        data['author'].num_nodes = N
        data['paper', 'written_by', 'author'].edge_index = paper_author
        data['author', 'writes', 'paper'].edge_index = paper_author.flip([0])

        # Get paper<->venue connectivity.
        path = osp.join(self.raw_dir, 'paper_conf.txt')
        paper_venue = pd.read_csv(path, sep='\t', header=None)
        paper_venue = torch.from_numpy(paper_venue.values)
        paper_venue = paper_venue.t().contiguous()
        M, N = int(paper_venue[0].max() + 1), int(paper_venue[1].max() + 1)
        paper_venue = coalesce(paper_venue, num_nodes=max(M, N))
        data['venue'].num_nodes = N
        data['paper', 'published_in', 'venue'].edge_index = paper_venue
        data['venue', 'publishes', 'paper'].edge_index = paper_venue.flip([0])

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/aqsol.py000066400000000000000000000122221444262304300212250ustar00rootroot00000000000000import os
import os.path as osp
import pickle
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class AQSOL(InMemoryDataset):
    r"""The AQSOL dataset from the `Benchmarking Graph Neural Networks
    `_ paper based on
    `AqSolDB `_, a
    standardized database of 9,982 molecular graphs with their aqueous
    solubility values, collected from 9 different data sources.

    The aqueous solubility targets are collected from experimental measurements
    and standardized to LogS units in AqSolDB. These final values denote the
    property to regress in the :class:`AQSOL` dataset. After filtering out few
    graphs with no bonds/edges, the total number of molecular graphs is 9,833.
    For each molecular graph, the node features are the types of heavy atoms
    and the edge features are the types of bonds between them, similar as in
    the :class:`~torch_geometric.datasets.ZINC` dataset.

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in
            the final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 9,833
          - ~17.6
          - ~35.8
          - 1
          - 1
    """
    url = 'https://www.dropbox.com/s/lzu9lmukwov12kt/aqsol_graph_raw.zip?dl=1'

    def __init__(self, root: str, split: str = 'train',
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        assert split in ['train', 'val', 'test']
        super().__init__(root, transform, pre_transform, pre_filter)
        path = osp.join(self.processed_dir, f'{split}.pt')
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'train.pickle', 'val.pickle', 'test.pickle', 'atom_dict.pickle',
            'bond_dict.pickle'
        ]

    @property
    def processed_file_names(self) -> List[str]:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        shutil.rmtree(self.raw_dir)
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.rename(osp.join(self.root, 'asqol_graph_raw'), self.raw_dir)
        os.unlink(path)

    def process(self):
        for raw_path, path in zip(self.raw_paths, self.processed_paths):
            with open(raw_path, 'rb') as f:
                graphs = pickle.load(f)

            data_list: List[Data] = []
            for graph in graphs:
                x, edge_attr, edge_index, y = graph

                x = torch.from_numpy(x)
                edge_attr = torch.from_numpy(edge_attr)
                edge_index = torch.from_numpy(edge_index)
                y = torch.tensor([y]).float()

                if edge_index.numel() == 0:
                    continue  # Skipping for graphs with no bonds/edges.

                data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr,
                            y=y)

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)

            torch.save(self.collate(data_list), path)

    def atoms(self) -> List[str]:
        return [
            'Br', 'C', 'N', 'O', 'Cl', 'Zn', 'F', 'P', 'S', 'Na', 'Al', 'Si',
            'Mo', 'Ca', 'W', 'Pb', 'B', 'V', 'Co', 'Mg', 'Bi', 'Fe', 'Ba', 'K',
            'Ti', 'Sn', 'Cd', 'I', 'Re', 'Sr', 'H', 'Cu', 'Ni', 'Lu', 'Pr',
            'Te', 'Ce', 'Nd', 'Gd', 'Zr', 'Mn', 'As', 'Hg', 'Sb', 'Cr', 'Se',
            'La', 'Dy', 'Y', 'Pd', 'Ag', 'In', 'Li', 'Rh', 'Nb', 'Hf', 'Cs',
            'Ru', 'Au', 'Sm', 'Ta', 'Pt', 'Ir', 'Be', 'Ge'
        ]

    def bonds(self) -> List[str]:
        return ['NONE', 'SINGLE', 'DOUBLE', 'AROMATIC', 'TRIPLE']
pytorch/torch_geometric/datasets/attributed_graph_dataset.py000066400000000000000000000131471444262304300251520ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import scipy.sparse as sp
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.typing import SparseTensor


class AttributedGraphDataset(InMemoryDataset):
    r"""A variety of attributed graph datasets from the
    `"Scaling Attributed Network Embedding to Massive Graphs"
    `_ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Wiki"`, :obj:`"Cora"`
            :obj:`"CiteSeer"`, :obj:`"PubMed"`, :obj:`"BlogCatalog"`,
            :obj:`"PPI"`, :obj:`"Flickr"`, :obj:`"Facebook"`, :obj:`"Twitter"`,
            :obj:`"TWeibo"`, :obj:`"MAG"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Wiki
          - 2,405
          - 17,981
          - 4,973
          - 17
        * - Cora
          - 2,708
          - 5,429
          - 1,433
          - 7
        * - CiteSeer
          - 3,312
          - 4,715
          - 3,703
          - 6
        * - PubMed
          - 19,717
          - 44,338
          - 500
          - 3
        * - BlogCatalog
          - 5,196
          - 343,486
          - 8,189
          - 6
        * - PPI
          - 56,944
          - 1,612,348
          - 50
          - 121
        * - Flickr
          - 7,575
          - 479,476
          - 12,047
          - 9
        * - Facebook
          - 4,039
          - 88,234
          - 1,283
          - 193
        * - TWeibo
          - 2,320,895
          - 9,840,066
          - 1,657
          - 8
        * - MAG
          - 59,249,719
          - 978,147,253
          - 2,000
          - 100
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    datasets = {
        'wiki': '1EPhlbziZTQv19OsTrKrAJwsElbVPEbiV',
        'cora': '1FyVnpdsTT-lhkVPotUW8OVeuCi1vi3Ey',
        'citeseer': '1d3uQIpHiemWJPgLgTafi70RFYye7hoCp',
        'pubmed': '1DOK3FfslyJoGXUSCSrK5lzdyLfIwOz6k',
        'blogcatalog': '178PqGqh67RUYMMP6-SoRHDoIBh8ku5FS',
        'ppi': '1dvwRpPT4gGtOcNP_Q-G1TKl9NezYhtez',
        'flickr': '1tZp3EB20fAC27SYWwa-x66_8uGsuU62X',
        'facebook': '12aJWAGCM4IvdGI2fiydDNyWzViEOLZH8',
        'twitter': '1fUYggzZlDrt9JsLsSdRUHiEzQRW1kSA4',
        'tweibo': '1-2xHDPFCsuBuFdQN_7GLleWa8R_t50qU',
        'mag': '1ggraUMrQgdUyA3DjSRzzqMv0jFkU65V5',
    }

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        assert self.name in self.datasets.keys()
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        return ['attrs.npz', 'edgelist.txt', 'labels.txt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        url = self.url.format(self.datasets[self.name])
        path = download_url(url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)
        path = osp.join(self.raw_dir, f'{self.name}.attr')
        if self.name == 'mag':
            path = osp.join(self.raw_dir, self.name)
        for name in self.raw_file_names:
            os.rename(osp.join(path, name), osp.join(self.raw_dir, name))
        shutil.rmtree(path)

    def process(self):
        import pandas as pd

        x = sp.load_npz(self.raw_paths[0])
        if x.shape[-1] > 10000 or self.name == 'mag':
            x = SparseTensor.from_scipy(x).to(torch.float)
        else:
            x = torch.from_numpy(x.todense()).to(torch.float)

        df = pd.read_csv(self.raw_paths[1], header=None, sep=None,
                         engine='python')
        edge_index = torch.from_numpy(df.values).t().contiguous()

        with open(self.raw_paths[2], 'r') as f:
            ys = f.read().split('\n')[:-1]
            ys = [[int(y) - 1 for y in row.split()[1:]] for row in ys]
            multilabel = max([len(y) for y in ys]) > 1

        if not multilabel:
            y = torch.tensor(ys).view(-1)
        else:
            num_classes = max([y for row in ys for y in row]) + 1
            y = torch.zeros((len(ys), num_classes), dtype=torch.float)
            for i, row in enumerate(ys):
                for j in row:
                    y[i, j] = 1.

        data = Data(x=x, edge_index=edge_index, y=y)
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.capitalize()}()'
pytorch/torch_geometric/datasets/ba2motif_dataset.py000066400000000000000000000077371444262304300233350ustar00rootroot00000000000000import pickle
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class BA2MotifDataset(InMemoryDataset):
    r"""The synthetic BA-2motifs graph classification dataset for evaluating
    explainabilty algorithms, as described in the `"Parameterized Explainer
    for Graph Neural Network" `_ paper.
    :class:`~torch_geometric.datasets.BA2MotifDataset` contains 1000 random
    Barabasi-Albert (BA) graphs.
    Half of the graphs are attached with a
    :class:`~torch_geometric.datasets.motif_generator.HouseMotif`, and the rest
    are attached with a five-node
    :class:`~torch_geometric.datasets.motif_generator.CycleMotif`.
    The graphs are assigned to one of the two classes according to the type of
    attached motifs.

    This dataset is pre-computed from the official implementation. If you want
    to create own variations of it, you can make use of the
    :class:`~torch_geometric.datasets.ExplainerDataset`:

    .. code-block:: python

        import torch
        from torch_geometric.datasets import ExplainerDataset
        from torch_geometric.datasets.graph_generator import BAGraph
        from torch_geometric.datasets.motif_generator import HouseMotif
        from torch_geometric.datasets.motif_generator import CycleMotif

        dataset1 = ExplainerDataset(
            graph_generator=BAGraph(num_nodes=25, num_edges=1),
            motif_generator=HouseMotif(),
            num_motifs=1,
            num_graphs=500,
        )

        dataset2 = ExplainerDataset(
            graph_generator=BAGraph(num_nodes=25, num_edges=1),
            motif_generator=CycleMotif(5),
            num_motifs=1,
            num_graphs=500,
        )

        dataset = torch.utils.data.ConcatDataset([dataset1, dataset2])

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 1000
          - 25
          - ~51.0
          - 10
          - 2
    """
    url = 'https://github.com/flyingdoog/PGExplainer/raw/master/dataset'
    filename = 'BA-2motif.pkl'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    def raw_file_names(self) -> str:
        return self.filename

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(f'{self.url}/{self.filename}', self.raw_dir)

    def process(self):
        with open(self.raw_paths[0], 'rb') as f:
            adj, x, y = pickle.load(f)

        adjs = torch.from_numpy(adj)
        xs = torch.from_numpy(x).to(torch.float)
        ys = torch.from_numpy(y)

        data_list: List[Data] = []
        for i in range(xs.size(0)):
            edge_index = adjs[i].nonzero().t()
            x = xs[i]
            y = int(ys[i].nonzero())

            data = Data(x=x, edge_index=edge_index, y=y)

            if self.pre_transform is not None:
                data = self.pre_transform(data)

            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])
pytorch/torch_geometric/datasets/ba_multi_shapes.py000066400000000000000000000066761444262304300232650ustar00rootroot00000000000000import pickle
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class BAMultiShapesDataset(InMemoryDataset):
    r"""The synthetic BA-Multi-Shapes graph classification dataset for
    evaluating explainabilty algorithms, as described in the
    `"Global Explainability of GNNs via Logic Combination of Learned Concepts"
    `_ paper.
    Given three atomic motifs, namely House (H), Wheel (W), and Grid (G),
    :class:`~torch_geometric.datasets.BAMultiShapesDataset` contains 1,000
    graphs where each graph is obtained by attaching the motifs to a random
    Barabasi-Albert (BA) as follows:

    * class 0: :math:`\emptyset \lor H \lor W \lor G \lor \{ H, W, G \}`

    * class 1: :math:`(H \land W) \lor (H \land G) \lor (W \land G)`

    This dataset is pre-computed from the official implementation.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 1000
          - 40
          - ~87.0
          - 10
          - 2
    """
    url = ('https://github.com/steveazzolin/gnn_logic_global_expl/raw/master/'
           'datasets/BAMultiShapes/BAMultiShapes.pkl')

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'BAMultiShapes.pkl'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        with open(self.raw_paths[0], 'rb') as f:
            adjs, xs, ys = pickle.load(f)

        data_list: List[Data] = []
        for adj, x, y in zip(adjs, xs, ys):
            edge_index = torch.from_numpy(adj).nonzero().t()
            x = torch.from_numpy(np.array(x)).to(torch.float)

            data = Data(x=x, edge_index=edge_index, y=y)

            if self.pre_filter is not None and not self.pre_filter(data):
                continue

            if self.pre_transform is not None:
                data = self.pre_transform(data)

            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])
pytorch/torch_geometric/datasets/ba_shapes.py000066400000000000000000000073601444262304300220420ustar00rootroot00000000000000from typing import Callable, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset
from torch_geometric.deprecation import deprecated
from torch_geometric.utils import barabasi_albert_graph


def house():
    edge_index = torch.tensor([[0, 0, 0, 1, 1, 1, 2, 2, 3, 3, 4, 4],
                               [1, 3, 4, 4, 2, 0, 1, 3, 2, 0, 0, 1]])
    label = torch.tensor([1, 1, 2, 2, 3])
    return edge_index, label


@deprecated("use 'datasets.ExplainerDataset' in combination with "
            "'datasets.graph_generator.BAGraph' instead")
class BAShapes(InMemoryDataset):
    r"""The BA-Shapes dataset from the `"GNNExplainer: Generating Explanations
    for Graph Neural Networks" `__ paper,
    containing a Barabasi-Albert (BA) graph with 300 nodes and a set of 80
    "house"-structured graphs connected to it.

    .. warning::

        :class:`BAShapes` is deprecated and will be removed in a future
        release. Use :class:`ExplainerDataset` in combination with
        :class:`torch_geometric.datasets.graph_generator.BAGraph` instead.

    Args:
        connection_distribution (str, optional): Specifies how the houses
            and the BA graph get connected. Valid inputs are :obj:`"random"`
            (random BA graph nodes are selected for connection to the houses),
            and :obj:`"uniform"` (uniformly distributed BA graph nodes are
            selected for connection to the houses). (default: :obj:`"random"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
    """
    def __init__(self, connection_distribution: str = "random",
                 transform: Optional[Callable] = None):
        super().__init__('.', transform)
        assert connection_distribution in ['random', 'uniform']

        # Build the Barabasi-Albert graph:
        num_nodes = 300
        edge_index = barabasi_albert_graph(num_nodes, num_edges=5)
        edge_label = torch.zeros(edge_index.size(1), dtype=torch.int64)
        node_label = torch.zeros(num_nodes, dtype=torch.int64)

        # Select nodes to connect shapes:
        num_houses = 80
        if connection_distribution == 'random':
            connecting_nodes = torch.randperm(num_nodes)[:num_houses]
        else:
            step = num_nodes // num_houses
            connecting_nodes = torch.arange(0, num_nodes, step)

        # Connect houses to Barabasi-Albert graph:
        edge_indices = [edge_index]
        edge_labels = [edge_label]
        node_labels = [node_label]
        for i in range(num_houses):
            house_edge_index, house_label = house()

            edge_indices.append(house_edge_index + num_nodes)
            edge_indices.append(
                torch.tensor([[int(connecting_nodes[i]), num_nodes],
                              [num_nodes, int(connecting_nodes[i])]]))

            edge_labels.append(
                torch.ones(house_edge_index.size(1), dtype=torch.long))
            edge_labels.append(torch.zeros(2, dtype=torch.long))

            node_labels.append(house_label)

            num_nodes += 5

        edge_index = torch.cat(edge_indices, dim=1)
        edge_label = torch.cat(edge_labels, dim=0)
        node_label = torch.cat(node_labels, dim=0)

        x = torch.ones((num_nodes, 10), dtype=torch.float)
        expl_mask = torch.zeros(num_nodes, dtype=torch.bool)
        expl_mask[torch.arange(400, num_nodes, 5)] = True

        data = Data(x=x, edge_index=edge_index, y=node_label,
                    expl_mask=expl_mask, edge_label=edge_label)

        self.data, self.slices = self.collate([data])
pytorch/torch_geometric/datasets/bitcoin_otc.py000066400000000000000000000100511444262304300224000ustar00rootroot00000000000000import datetime
import os
from typing import Callable, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_gz,
)


class BitcoinOTC(InMemoryDataset):
    r"""The Bitcoin-OTC dataset from the `"EvolveGCN: Evolving Graph
    Convolutional Networks for Dynamic Graphs"
    `_ paper, consisting of 138
    who-trusts-whom networks of sequential time steps.

    Args:
        root (str): Root directory where the dataset should be saved.
        edge_window_size (int, optional): The window size for the existence of
            an edge in the graph sequence since its initial creation.
            (default: :obj:`10`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 138
          - 6,005
          - ~2,573.2
          - 0
          - 0
    """

    url = 'https://snap.stanford.edu/data/soc-sign-bitcoinotc.csv.gz'

    def __init__(self, root: str, edge_window_size: int = 10,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.edge_window_size = edge_window_size
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'soc-sign-bitcoinotc.csv'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    @property
    def num_nodes(self) -> int:
        return self._data.edge_index.max().item() + 1

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_gz(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        with open(self.raw_paths[0], 'r') as f:
            data = f.read().split('\n')[:-1]
            data = [[x for x in line.split(',')] for line in data]

            edge_index = [[int(line[0]), int(line[1])] for line in data]
            edge_index = torch.tensor(edge_index, dtype=torch.long)
            edge_index = edge_index - edge_index.min()
            edge_index = edge_index.t().contiguous()
            num_nodes = edge_index.max().item() + 1

            edge_attr = [int(line[2]) for line in data]
            edge_attr = torch.tensor(edge_attr, dtype=torch.long)

            stamps = [int(float(line[3])) for line in data]
            stamps = [datetime.datetime.fromtimestamp(x) for x in stamps]

        offset = datetime.timedelta(days=13.8)  # Results in 138 time steps.
        graph_idx, factor = [], 1
        for t in stamps:
            factor = factor if t < stamps[0] + factor * offset else factor + 1
            graph_idx.append(factor - 1)
        graph_idx = torch.tensor(graph_idx, dtype=torch.long)

        data_list = []
        for i in range(graph_idx.max().item() + 1):
            mask = (graph_idx > (i - self.edge_window_size)) & (graph_idx <= i)
            data = Data()
            data.edge_index = edge_index[:, mask]
            data.edge_attr = edge_attr[mask]
            data.num_nodes = num_nodes
            data_list.append(data)

        if self.pre_filter is not None:
            data_list = [d for d in data_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(d) for d in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])
pytorch/torch_geometric/datasets/citation_full.py000066400000000000000000000101551444262304300227450ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url
from torch_geometric.io import read_npz


class CitationFull(InMemoryDataset):
    r"""The full citation network datasets from the
    `"Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via
    Ranking" `_ paper.
    Nodes represent documents and edges represent citation links.
    Datasets include :obj:`"Cora"`, :obj:`"Cora_ML"`, :obj:`"CiteSeer"`,
    :obj:`"DBLP"`, :obj:`"PubMed"`.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Cora"`, :obj:`"Cora_ML"`
            :obj:`"CiteSeer"`, :obj:`"DBLP"`, :obj:`"PubMed"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        to_undirected (bool, optional): Whether the original graph is
            converted to an undirected one. (default: :obj:`True`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Cora
          - 19,793
          - 126,842
          - 8,710
          - 70
        * - Cora_ML
          - 2,995
          - 16,316
          - 2,879
          - 7
        * - CiteSeer
          - 4,230
          - 10,674
          - 602
          - 6
        * - DBLP
          - 17,716
          - 105,734
          - 1,639
          - 4
        * - PubMed
          - 19,717
          - 88,648
          - 500
          - 3
    """

    url = 'https://github.com/abojchevski/graph2gauss/raw/master/data/{}.npz'

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        to_undirected: bool = True,
    ):
        self.name = name.lower()
        self.to_undirected = to_undirected
        assert self.name in ['cora', 'cora_ml', 'citeseer', 'dblp', 'pubmed']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url.format(self.name), self.raw_dir)

    def process(self):
        data = read_npz(self.raw_paths[0], to_undirected=self.to_undirected)
        data = data if self.pre_transform is None else self.pre_transform(data)
        data, slices = self.collate([data])
        torch.save((data, slices), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.capitalize()}Full()'


class CoraFull(CitationFull):
    r"""Alias for :class:`~torch_geometric.datasets.CitationFull` with
    :obj:`name="Cora"`.

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 19,793
          - 126,842
          - 8,710
          - 70
    """
    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, 'cora', transform, pre_transform)

    def download(self):
        super().download()

    def process(self):
        super().process()
pytorch/torch_geometric/datasets/coauthor.py000066400000000000000000000057011444262304300217360ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url
from torch_geometric.io import read_npz


class Coauthor(InMemoryDataset):
    r"""The Coauthor CS and Coauthor Physics networks from the
    `"Pitfalls of Graph Neural Network Evaluation"
    `_ paper.
    Nodes represent authors that are connected by an edge if they co-authored a
    paper.
    Given paper keywords for each author's papers, the task is to map authors
    to their respective field of study.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"CS"`, :obj:`"Physics"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - CS
          - 18,333
          - 163,788
          - 6,805
          - 15
        * - Physics
          - 34,493
          - 495,924
          - 8,415
          - 5
    """

    url = 'https://github.com/shchur/gnn-benchmark/raw/master/data/npz/'

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        assert name.lower() in ['cs', 'physics']
        self.name = 'CS' if name.lower() == 'cs' else 'Physics'
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'ms_academic_{self.name[:3].lower()}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url + self.raw_file_names, self.raw_dir)

    def process(self):
        data = read_npz(self.raw_paths[0], to_undirected=True)
        data = data if self.pre_transform is None else self.pre_transform(data)
        data, slices = self.collate([data])
        torch.save((data, slices), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}{self.name}()'
pytorch/torch_geometric/datasets/coma.py000066400000000000000000000107241444262304300210320ustar00rootroot00000000000000import os.path as osp
from glob import glob
from typing import Callable, List, Optional

import torch

from torch_geometric.data import InMemoryDataset, extract_zip
from torch_geometric.io import read_ply


class CoMA(InMemoryDataset):
    r"""The CoMA 3D faces dataset from the `"Generating 3D faces using
    Convolutional Mesh Autoencoders" `_
    paper, containing 20,466 meshes of extreme expressions captured over 12
    different subjects.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 20,465
          - 5,023
          - 29,990
          - 3
          - 12
    """

    url = 'https://coma.is.tue.mpg.de/'

    categories = [
        'bareteeth',
        'cheeks_in',
        'eyebrow',
        'high_smile',
        'lips_back',
        'lips_up',
        'mouth_down',
        'mouth_extreme',
        'mouth_middle',
        'mouth_open',
        'mouth_side',
        'mouth_up',
    ]

    def __init__(self, root: str, train: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> str:
        return 'COMA_data.zip'

    @property
    def processed_file_names(self) -> List[str]:
        return ['training.pt', 'test.pt']

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download 'COMA_data.zip' from "
            f"'{self.url}' and move it to '{self.raw_dir}'")

    def process(self):
        folders = sorted(glob(osp.join(self.raw_dir, 'FaceTalk_*')))
        if len(folders) == 0:
            extract_zip(self.raw_paths[0], self.raw_dir, log=False)
            folders = sorted(glob(osp.join(self.raw_dir, 'FaceTalk_*')))

        train_data_list, test_data_list = [], []
        for folder in folders:
            for i, category in enumerate(self.categories):
                files = sorted(glob(osp.join(folder, category, '*.ply')))
                for j, f in enumerate(files):
                    data = read_ply(f)
                    data.y = torch.tensor([i], dtype=torch.long)
                    if self.pre_filter is not None and\
                       not self.pre_filter(data):
                        continue
                    if self.pre_transform is not None:
                        data = self.pre_transform(data)

                    if (j % 100) < 90:
                        train_data_list.append(data)
                    else:
                        test_data_list.append(data)

        torch.save(self.collate(train_data_list), self.processed_paths[0])
        torch.save(self.collate(test_data_list), self.processed_paths[1])
pytorch/torch_geometric/datasets/dblp.py000066400000000000000000000121101444262304300210230ustar00rootroot00000000000000import os
import os.path as osp
from itertools import product
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class DBLP(InMemoryDataset):
    r"""A subset of the DBLP computer science bibliography website, as
    collected in the `"MAGNN: Metapath Aggregated Graph Neural Network for
    Heterogeneous Graph Embedding" `_ paper.
    DBLP is a heterogeneous graph containing four types of entities - authors
    (4,057 nodes), papers (14,328 nodes), terms (7,723 nodes), and conferences
    (20 nodes).
    The authors are divided into four research areas (database, data mining,
    artificial intelligence, information retrieval).
    Each author is described by a bag-of-words representation of their paper
    keywords.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10
        :header-rows: 1

        * - Node/Edge Type
          - #nodes/#edges
          - #features
          - #classes
        * - Author
          - 4,057
          - 334
          - 4
        * - Paper
          - 14,328
          - 4,231
          -
        * - Term
          - 7,723
          - 50
          -
        * - Conference
          - 20
          - 0
          -
        * - Author-Paper
          - 196,425
          -
          -
        * - Paper-Term
          - 85,810
          -
          -
        * - Conference-Paper
          - 14,328
          -
          -
    """

    url = 'https://www.dropbox.com/s/yh4grpeks87ugr2/DBLP_processed.zip?dl=1'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'adjM.npz', 'features_0.npz', 'features_1.npz', 'features_2.npy',
            'labels.npy', 'node_types.npy', 'train_val_test_idx.npz'
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        data = HeteroData()

        node_types = ['author', 'paper', 'term', 'conference']
        for i, node_type in enumerate(node_types[:2]):
            x = sp.load_npz(osp.join(self.raw_dir, f'features_{i}.npz'))
            data[node_type].x = torch.from_numpy(x.todense()).to(torch.float)

        x = np.load(osp.join(self.raw_dir, 'features_2.npy'))
        data['term'].x = torch.from_numpy(x).to(torch.float)

        node_type_idx = np.load(osp.join(self.raw_dir, 'node_types.npy'))
        node_type_idx = torch.from_numpy(node_type_idx).to(torch.long)
        data['conference'].num_nodes = int((node_type_idx == 3).sum())

        y = np.load(osp.join(self.raw_dir, 'labels.npy'))
        data['author'].y = torch.from_numpy(y).to(torch.long)

        split = np.load(osp.join(self.raw_dir, 'train_val_test_idx.npz'))
        for name in ['train', 'val', 'test']:
            idx = split[f'{name}_idx']
            idx = torch.from_numpy(idx).to(torch.long)
            mask = torch.zeros(data['author'].num_nodes, dtype=torch.bool)
            mask[idx] = True
            data['author'][f'{name}_mask'] = mask

        s = {}
        N_a = data['author'].num_nodes
        N_p = data['paper'].num_nodes
        N_t = data['term'].num_nodes
        N_c = data['conference'].num_nodes
        s['author'] = (0, N_a)
        s['paper'] = (N_a, N_a + N_p)
        s['term'] = (N_a + N_p, N_a + N_p + N_t)
        s['conference'] = (N_a + N_p + N_t, N_a + N_p + N_t + N_c)

        A = sp.load_npz(osp.join(self.raw_dir, 'adjM.npz'))
        for src, dst in product(node_types, node_types):
            A_sub = A[s[src][0]:s[src][1], s[dst][0]:s[dst][1]].tocoo()
            if A_sub.nnz > 0:
                row = torch.from_numpy(A_sub.row).to(torch.long)
                col = torch.from_numpy(A_sub.col).to(torch.long)
                data[src, dst].edge_index = torch.stack([row, col], dim=0)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/datasets/dbp15k.py000066400000000000000000000127501444262304300212020ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, Dict, List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_txt_array
from torch_geometric.utils import sort_edge_index


class DBP15K(InMemoryDataset):
    r"""The DBP15K dataset from the
    `"Cross-lingual Entity Alignment via Joint Attribute-Preserving Embedding"
    `_ paper, where Chinese, Japanese and
    French versions of DBpedia were linked to its English version.
    Node features are given by pre-trained and aligned monolingual word
    embeddings from the `"Cross-lingual Knowledge Graph Alignment via Graph
    Matching Neural Network" `_ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        pair (str): The pair of languages (:obj:`"en_zh"`, :obj:`"en_fr"`,
            :obj:`"en_ja"`, :obj:`"zh_en"`, :obj:`"fr_en"`, :obj:`"ja_en"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'
    file_id = '1ggYlYf2_kTyi7oF9g07oTNn3VDhjl7so'

    def __init__(self, root: str, pair: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        assert pair in ['en_zh', 'en_fr', 'en_ja', 'zh_en', 'fr_en', 'ja_en']
        self.pair = pair
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['en_zh', 'en_fr', 'en_ja', 'zh_en', 'fr_en', 'ja_en']

    @property
    def processed_file_names(self) -> str:
        return f'{self.pair}.pt'

    def download(self):
        path = download_url(self.url.format(self.file_id), self.root)
        extract_zip(path, self.root)
        os.unlink(path)
        shutil.rmtree(self.raw_dir)
        os.rename(osp.join(self.root, 'DBP15K'), self.raw_dir)

    def process(self):
        embs = {}
        with open(osp.join(self.raw_dir, 'sub.glove.300d'), 'r') as f:
            for i, line in enumerate(f):
                info = line.strip().split(' ')
                if len(info) > 300:
                    embs[info[0]] = torch.tensor([float(x) for x in info[1:]])
                else:
                    embs['**UNK**'] = torch.tensor([float(x) for x in info])

        g1_path = osp.join(self.raw_dir, self.pair, 'triples_1')
        x1_path = osp.join(self.raw_dir, self.pair, 'id_features_1')
        g2_path = osp.join(self.raw_dir, self.pair, 'triples_2')
        x2_path = osp.join(self.raw_dir, self.pair, 'id_features_2')

        x1, edge_index1, rel1, assoc1 = self.process_graph(
            g1_path, x1_path, embs)
        x2, edge_index2, rel2, assoc2 = self.process_graph(
            g2_path, x2_path, embs)

        train_path = osp.join(self.raw_dir, self.pair, 'train.examples.20')
        train_y = self.process_y(train_path, assoc1, assoc2)

        test_path = osp.join(self.raw_dir, self.pair, 'test.examples.1000')
        test_y = self.process_y(test_path, assoc1, assoc2)

        data = Data(x1=x1, edge_index1=edge_index1, rel1=rel1, x2=x2,
                    edge_index2=edge_index2, rel2=rel2, train_y=train_y,
                    test_y=test_y)
        torch.save(self.collate([data]), self.processed_paths[0])

    def process_graph(
        self,
        triple_path: str,
        feature_path: str,
        embeddings: Dict[str, Tensor],
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor]:

        g1 = read_txt_array(triple_path, sep='\t', dtype=torch.long)
        subj, rel, obj = g1.t()

        x_dict = {}
        with open(feature_path, 'r') as f:
            for line in f:
                info = line.strip().split('\t')
                info = info if len(info) == 2 else info + ['**UNK**']
                seq = info[1].lower().split()
                hs = [embeddings.get(w, embeddings['**UNK**']) for w in seq]
                x_dict[int(info[0])] = torch.stack(hs, dim=0)

        idx = torch.tensor(list(x_dict.keys()))
        assoc = torch.full((idx.max().item() + 1, ), -1, dtype=torch.long)
        assoc[idx] = torch.arange(idx.size(0))

        subj, obj = assoc[subj], assoc[obj]
        edge_index = torch.stack([subj, obj], dim=0)
        edge_index, rel = sort_edge_index(edge_index, rel)

        xs = [None for _ in range(idx.size(0))]
        for i in x_dict.keys():
            xs[assoc[i]] = x_dict[i]
        x = torch.nn.utils.rnn.pad_sequence(xs, batch_first=True)

        return x, edge_index, rel, assoc

    def process_y(self, path: str, assoc1: Tensor, assoc2: Tensor) -> Tensor:
        row, col, mask = read_txt_array(path, sep='\t', dtype=torch.long).t()
        mask = mask.to(torch.bool)
        return torch.stack([assoc1[row[mask]], assoc2[col[mask]]], dim=0)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.pair})'
pytorch/torch_geometric/datasets/deezer_europe.py000066400000000000000000000044041444262304300227460ustar00rootroot00000000000000from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class DeezerEurope(InMemoryDataset):
    r"""The Deezer Europe dataset introduced in the `"Characteristic Functions
    on Graphs: Birds of a Feather, from Statistical Descriptors to Parametric
    Models" `_ paper.
    Nodes represent European users of Deezer and edges are mutual follower
    relationships.
    It contains 28,281 nodes, 185,504 edges, 128 node features and 2 classes.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://graphmining.ai/datasets/ptg/deezer_europe.npz'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'deezer_europe.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        x = torch.from_numpy(data['features']).to(torch.float)
        y = torch.from_numpy(data['target']).to(torch.long)
        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/dgraph.py000066400000000000000000000074261444262304300213650ustar00rootroot00000000000000import os
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, extract_zip
from torch_geometric.utils import index_to_mask


class DGraphFin(InMemoryDataset):
    r"""The DGraphFin networks from the
    `"DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection"
    `_ paper.
    It is a directed, unweighted dynamic graph consisting of millions of
    nodes and edges, representing a realistic user-to-user social network
    in financial industry.
    Node represents a Finvolution user, and an edge from one
    user to another means that the user regards the other user
    as the emergency contact person. Each edge is associated with a
    timestamp ranging from 1 to 821 and a type of emergency contact
    ranging from 0 to 11.


    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 3,700,550
          - 4,300,999
          - 17
          - 2
    """

    url = "https://dgraph.xinye.com"

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download '{self.raw_file_names}' from "
            f"'{self.url}' and move it to '{self.raw_dir}'")

    @property
    def raw_file_names(self) -> str:
        return 'DGraphFin.zip'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    @property
    def num_classes(self) -> int:
        return 2

    def process(self):
        extract_zip(self.raw_paths[0], self.raw_dir, log=False)
        path = os.path.join(self.raw_dir, "dgraphfin.npz")

        with np.load(path) as loader:
            x = torch.from_numpy(loader['x']).to(torch.float)
            y = torch.from_numpy(loader['y']).to(torch.long)
            edge_index = torch.from_numpy(loader['edge_index']).to(torch.long)
            edge_type = torch.from_numpy(loader['edge_type']).to(torch.long)
            edge_time = torch.from_numpy(loader['edge_timestamp']).to(
                torch.long)
            train_nodes = torch.from_numpy(loader['train_mask']).to(torch.long)
            val_nodes = torch.from_numpy(loader['valid_mask']).to(torch.long)
            test_nodes = torch.from_numpy(loader['test_mask']).to(torch.long)

            train_mask = index_to_mask(train_nodes, size=x.size(0))
            val_mask = index_to_mask(val_nodes, size=x.size(0))
            test_mask = index_to_mask(test_nodes, size=x.size(0))
            data = Data(x=x, edge_index=edge_index.t(), edge_type=edge_type,
                        edge_time=edge_time, y=y, train_mask=train_mask,
                        val_mask=val_mask, test_mask=test_mask)

        data = data if self.pre_transform is None else self.pre_transform(data)
        data, slices = self.collate([data])
        torch.save((data, slices), self.processed_paths[0])
pytorch/torch_geometric/datasets/dynamic_faust.py000066400000000000000000000133511444262304300227400ustar00rootroot00000000000000from itertools import product
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset


class DynamicFAUST(InMemoryDataset):
    r"""The dynamic FAUST humans dataset from the `"Dynamic FAUST: Registering
    Human Bodies in Motion"
    `_ paper.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        subjects (list, optional): List of subjects to include in the
            dataset. Can include the subjects :obj:`"50002"`, :obj:`"50004"`,
            :obj:`"50007"`, :obj:`"50009"`, :obj:`"50020"`, :obj:`"50021"`,
            :obj:`"50022"`, :obj:`"50025"`, :obj:`"50026"`, :obj:`"50027"`.
            If set to :obj:`None`, the dataset will contain all subjects.
            (default: :obj:`None`)
        categories (list, optional): List of categories to include in the
            dataset. Can include the categories :obj:`"chicken_wings"`,
            :obj:`"hips"`, :obj:`"jiggle_on_toes"`, :obj:`"jumping_jacks"`,
            :obj:`"knees"`, :obj:`"light_hopping_loose"`,
            :obj:`"light_hopping_stiff"`, :obj:`"one_leg_jump"`,
            :obj:`"one_leg_loose"`, :obj:`"personal_move"`, :obj:`"punching"`,
            :obj:`"running_on_spot"`, :obj:`"running_on_spot_bugfix"`,
            :obj:`"shake_arms"`, :obj:`"shake_hips"`, :obj:`"shoulders"`.
            If set to :obj:`None`, the dataset will contain all categories.
            (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'http://dfaust.is.tue.mpg.de/'

    subjects = [
        '50002', '50004', '50007', '50009', '50020', '50021', '50022', '50025',
        '50026', '50027'
    ]
    categories = [
        'chicken_wings', 'hips', 'jiggle_on_toes', 'jumping_jacks', 'knees',
        'light_hopping_loose', 'light_hopping_stiff', 'one_leg_jump',
        'one_leg_loose', 'personal_move', 'punching', 'running_on_spot',
        'running_on_spot_bugfix', 'shake_arms', 'shake_hips', 'shake_shoulders'
    ]

    def __init__(self, root: str, subjects: Optional[List[str]] = None,
                 categories: Optional[List[str]] = None,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):

        subjects = self.subjects if subjects is None else subjects
        subjects = [sid.lower() for sid in subjects]
        for sid in subjects:
            assert sid in self.subjects
        self.subjects = subjects

        categories = self.categories if categories is None else categories
        categories = [cat.lower() for cat in categories]
        for cat in categories:
            assert cat in self.categories
        self.categories = categories

        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['registrations_m.hdf5', 'registrations_f.hdf5']

    @property
    def processed_file_names(self) -> str:
        sids = '_'.join([sid[-2:] for sid in self.subjects])
        cats = '_'.join([
            ''.join([w[0] for w in cat.split('_')]) for cat in self.categories
        ])
        return f'{sids}_{cats}.pt'

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download male registrations "
            f"'registrations_m.hdf5' and female registrations "
            f"'registrations_f.hdf5' from '{self.url}' and move it to "
            f"'{self.raw_dir}'")

    def process(self):
        import h5py

        fm = h5py.File(self.raw_paths[0], 'r')
        ff = h5py.File(self.raw_paths[1], 'r')

        face = torch.from_numpy(fm['faces'][()]).to(torch.long)
        face = face.t().contiguous()

        data_list = []
        for (sid, cat) in product(self.subjects, self.categories):
            idx = f'{sid}_{cat}'
            if idx in fm:
                pos = torch.from_numpy(fm[idx][()])
            elif idx in ff:
                pos = torch.from_numpy(ff[idx][()])
            else:
                continue
            pos = pos.permute(2, 0, 1).contiguous()
            data_list.append(Data(pos=pos, face=face, num_nodes=pos.size(1)))

        if self.pre_filter is not None:
            data_list = [d for d in data_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(d) for d in data_list]

        torch.save(self.collate(data_list), self.processed_paths[0])
pytorch/torch_geometric/datasets/elliptic.py000066400000000000000000000106221444262304300217150ustar00rootroot00000000000000import os
from typing import Any, Callable, List, Optional, Tuple

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class EllipticBitcoinDataset(InMemoryDataset):
    r"""The Elliptic Bitcoin dataset of Bitcoin transactions from the
    `"Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional
    Networks for Financial Forensics" `_
    paper.

    :class:`EllipticBitcoinDataset` maps Bitcoin transactions to real entities
    belonging to licit categories (exchanges, wallet providers, miners,
    licit services, etc.) versus illicit ones (scams, malware, terrorist
    organizations, ransomware, Ponzi schemes, etc.)

    There exists 203,769 node transactions and 234,355 directed edge payments
    flows, with two percent of nodes (4,545) labelled as illicit, and
    twenty-one percent of nodes (42,019) labelled as licit.
    The remaining transactions are unknown.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 203,769
          - 234,355
          - 165
          - 2
    """
    url = 'https://data.pyg.org/datasets/elliptic'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'elliptic_txs_features.csv',
            'elliptic_txs_edgelist.csv',
            'elliptic_txs_classes.csv',
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for file_name in self.raw_file_names:
            path = download_url(f'{self.url}/{file_name}.zip', self.raw_dir)
            extract_zip(path, self.raw_dir)
            os.remove(path)

    def _process_df(self, feat_df: Any, edge_df: Any,
                    class_df: Any) -> Tuple[Any, Any, Any]:
        return feat_df, edge_df, class_df

    def process(self):
        import pandas as pd

        feat_df = pd.read_csv(self.raw_paths[0], header=None)
        edge_df = pd.read_csv(self.raw_paths[1])
        class_df = pd.read_csv(self.raw_paths[2])

        columns = {0: 'txId', 1: 'time_step'}
        feat_df = feat_df.rename(columns=columns)

        feat_df, edge_df, class_df = self._process_df(
            feat_df,
            edge_df,
            class_df,
        )

        x = torch.from_numpy(feat_df.loc[:, 2:].values).to(torch.float)

        # There exists 3 different classes in the dataset:
        # 0=licit,  1=illicit, 2=unknown
        mapping = {'unknown': 2, '1': 1, '2': 0}
        class_df['class'] = class_df['class'].map(mapping)
        y = torch.from_numpy(class_df['class'].values)

        mapping = {idx: i for i, idx in enumerate(feat_df['txId'].values)}
        edge_df['txId1'] = edge_df['txId1'].map(mapping)
        edge_df['txId2'] = edge_df['txId2'].map(mapping)
        edge_index = torch.from_numpy(edge_df.values).t().contiguous()

        # Timestamp based split:
        # train_mask: 1 - 34 time_step, test_mask: 35-49 time_step
        time_step = torch.from_numpy(feat_df['time_step'].values)
        train_mask = (time_step < 35) & (y != 2)
        test_mask = (time_step >= 35) & (y != 2)

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    test_mask=test_mask)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    @property
    def num_classes(self) -> int:
        return 2
pytorch/torch_geometric/datasets/elliptic_temporal.py000066400000000000000000000056561444262304300236330ustar00rootroot00000000000000from typing import Any, Callable, Optional, Tuple

from torch_geometric.datasets import EllipticBitcoinDataset


class EllipticBitcoinTemporalDataset(EllipticBitcoinDataset):
    r"""The time-step aware Elliptic Bitcoin dataset of Bitcoin transactions
    from the `"Anti-Money Laundering in Bitcoin: Experimenting with Graph
    Convolutional Networks for Financial Forensics"
    `_ paper.

    :class:`EllipticBitcoinTemporalDataset` maps Bitcoin transactions to real
    entities belonging to licit categories (exchanges, wallet providers,
    miners, licit services, etc.) versus illicit ones (scams, malware,
    terrorist organizations, ransomware, Ponzi schemes, etc.)

    There exists 203,769 node transactions and 234,355 directed edge payments
    flows, with two percent of nodes (4,545) labelled as illicit, and
    twenty-one percent of nodes (42,019) labelled as licit.
    The remaining transactions are unknown.

    .. note::

        In contrast to :class:`EllipticBitcoinDataset`, this dataset returns
        Bitcoin transactions only for a given timestamp :obj:`t`.

    Args:
        root (str): Root directory where the dataset should be saved.
        t (int): The Timestep for which nodes should be selected (from :obj:`1`
            to :obj:`49`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 203,769
          - 234,355
          - 165
          - 2
    """
    def __init__(
        self,
        root: str,
        t: int,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        if t < 1 or t > 49:
            raise ValueError("'t' needs to be between 1 and 49")

        self.t = t
        super().__init__(root, transform, pre_transform)

    @property
    def processed_file_names(self) -> str:
        return f'data_t_{self.t}.pt'

    def _process_df(self, feat_df: Any, edge_df: Any,
                    class_df: Any) -> Tuple[Any, Any, Any]:

        feat_df = feat_df[feat_df['time_step'] == self.t]

        mask = edge_df['txId1'].isin(feat_df['txId'].values)
        edge_df = edge_df[mask]

        class_df = class_df.merge(feat_df[['txId']], how='right',
                                  left_on='txId', right_on='txId')

        return feat_df, edge_df, class_df
pytorch/torch_geometric/datasets/email_eu_core.py000066400000000000000000000051071444262304300227020ustar00rootroot00000000000000import os
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_gz,
)


class EmailEUCore(InMemoryDataset):
    r"""An e-mail communication network of a large European research
    institution, taken from the `"Local Higher-order Graph Clustering"
    `_ paper.
    Nodes indicate members of the institution.
    An edge between a pair of members indicates that they exchanged at least
    one email.
    Node labels indicate membership to one of the 42 departments.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    urls = [
        'https://snap.stanford.edu/data/email-Eu-core.txt.gz',
        'https://snap.stanford.edu/data/email-Eu-core-department-labels.txt.gz'
    ]

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['email-Eu-core.txt', 'email-Eu-core-department-labels.txt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for url in self.urls:
            path = download_url(url, self.raw_dir)
            extract_gz(path, self.raw_dir)
            os.unlink(path)

    def process(self):
        import pandas as pd

        edge_index = pd.read_csv(self.raw_paths[0], sep=' ', header=None)
        edge_index = torch.from_numpy(edge_index.values).t().contiguous()

        y = pd.read_csv(self.raw_paths[1], sep=' ', header=None, usecols=[1])
        y = torch.from_numpy(y.values).view(-1)

        data = Data(edge_index=edge_index, y=y, num_nodes=y.size(0))

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/entities.py000066400000000000000000000155311444262304300217400ustar00rootroot00000000000000import logging
import os
import os.path as osp
from collections import Counter
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
)
from torch_geometric.utils import index_sort


class Entities(InMemoryDataset):
    r"""The relational entities networks :obj:`"AIFB"`, :obj:`"MUTAG"`,
    :obj:`"BGS"` and :obj:`"AM"` from the `"Modeling Relational Data with Graph
    Convolutional Networks" `_ paper.
    Training and test splits are given by node indices.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"AIFB"`, :obj:`"MUTAG"`,
            :obj:`"BGS"`, :obj:`"AM"`).
        hetero (bool, optional): If set to :obj:`True`, will save the dataset
            as a :class:`~torch_geometric.data.HeteroData` object.
            (default: :obj:`False`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - AIFB
          - 8,285
          - 58,086
          - 0
          - 4
        * - AM
          - 1,666,764
          - 11,976,642
          - 0
          - 11
        * - MUTAG
          - 23,644
          - 148,454
          - 0
          - 2
        * - BGS
          - 333,845
          - 1,832,398
          - 0
          - 2
    """

    url = 'https://data.dgl.ai/dataset/{}.tgz'

    def __init__(self, root: str, name: str, hetero: bool = False,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        self.hetero = hetero
        assert self.name in ['aifb', 'am', 'mutag', 'bgs']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def num_relations(self) -> int:
        return self._data.edge_type.max().item() + 1

    @property
    def num_classes(self) -> int:
        return self._data.train_y.max().item() + 1

    @property
    def raw_file_names(self) -> List[str]:
        return [
            f'{self.name}_stripped.nt.gz',
            'completeDataset.tsv',
            'trainingSet.tsv',
            'testSet.tsv',
        ]

    @property
    def processed_file_names(self) -> str:
        return 'hetero_data.pt' if self.hetero else 'data.pt'

    def download(self):
        path = download_url(self.url.format(self.name), self.root)
        extract_tar(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        import gzip

        import pandas as pd
        import rdflib as rdf

        graph_file, task_file, train_file, test_file = self.raw_paths

        with hide_stdout():
            g = rdf.Graph()
            with gzip.open(graph_file, 'rb') as f:
                g.parse(file=f, format='nt')

        freq = Counter(g.predicates())

        relations = sorted(set(g.predicates()), key=lambda p: -freq.get(p, 0))
        subjects = set(g.subjects())
        objects = set(g.objects())
        nodes = list(subjects.union(objects))

        N = len(nodes)
        R = 2 * len(relations)

        relations_dict = {rel: i for i, rel in enumerate(relations)}
        nodes_dict = {node: i for i, node in enumerate(nodes)}

        edges = []
        for s, p, o in g.triples((None, None, None)):
            src, dst, rel = nodes_dict[s], nodes_dict[o], relations_dict[p]
            edges.append([src, dst, 2 * rel])
            edges.append([dst, src, 2 * rel + 1])

        edges = torch.tensor(edges, dtype=torch.long).t().contiguous()
        _, perm = index_sort(N * R * edges[0] + R * edges[1] + edges[2])
        edges = edges[:, perm]

        edge_index, edge_type = edges[:2], edges[2]

        if self.name == 'am':
            label_header = 'label_cateogory'
            nodes_header = 'proxy'
        elif self.name == 'aifb':
            label_header = 'label_affiliation'
            nodes_header = 'person'
        elif self.name == 'mutag':
            label_header = 'label_mutagenic'
            nodes_header = 'bond'
        elif self.name == 'bgs':
            label_header = 'label_lithogenesis'
            nodes_header = 'rock'

        labels_df = pd.read_csv(task_file, sep='\t')
        labels_set = set(labels_df[label_header].values.tolist())
        labels_dict = {lab: i for i, lab in enumerate(list(labels_set))}
        nodes_dict = {str(key): val for key, val in nodes_dict.items()}

        train_labels_df = pd.read_csv(train_file, sep='\t')
        train_indices, train_labels = [], []
        for nod, lab in zip(train_labels_df[nodes_header].values,
                            train_labels_df[label_header].values):
            train_indices.append(nodes_dict[nod])
            train_labels.append(labels_dict[lab])

        train_idx = torch.tensor(train_indices, dtype=torch.long)
        train_y = torch.tensor(train_labels, dtype=torch.long)

        test_labels_df = pd.read_csv(test_file, sep='\t')
        test_indices, test_labels = [], []
        for nod, lab in zip(test_labels_df[nodes_header].values,
                            test_labels_df[label_header].values):
            test_indices.append(nodes_dict[nod])
            test_labels.append(labels_dict[lab])

        test_idx = torch.tensor(test_indices, dtype=torch.long)
        test_y = torch.tensor(test_labels, dtype=torch.long)

        data = Data(edge_index=edge_index, edge_type=edge_type,
                    train_idx=train_idx, train_y=train_y, test_idx=test_idx,
                    test_y=test_y, num_nodes=N)

        if self.hetero:
            data = data.to_heterogeneous(node_type_names=['v'])

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.upper()}{self.__class__.__name__}()'


class hide_stdout:
    def __enter__(self):
        self.level = logging.getLogger().level
        logging.getLogger().setLevel(logging.ERROR)

    def __exit__(self, *args):
        logging.getLogger().setLevel(self.level)
pytorch/torch_geometric/datasets/explainer_dataset.py000066400000000000000000000132711444262304300236070ustar00rootroot00000000000000from typing import Any, Callable, Dict, Optional, Union

import torch

from torch_geometric.data import InMemoryDataset
from torch_geometric.datasets.graph_generator import GraphGenerator
from torch_geometric.datasets.motif_generator import MotifGenerator
from torch_geometric.explain import Explanation


class ExplainerDataset(InMemoryDataset):
    r"""Generates a synthetic dataset for evaluating explainabilty algorithms,
    as described in the `"GNNExplainer: Generating Explanations for Graph
    Neural Networks" `__ paper.
    The :class:`~torch_geometric.datasets.ExplainerDataset` creates synthetic
    graphs coming from a
    :class:`~torch_geometric.datasets.graph_generator.GraphGenerator`, and
    randomly attaches :obj:`num_motifs` many motifs to it coming from a
    :class:`~torch_geometric.datasets.graph_generator.MotifGenerator`.
    Ground-truth node-level and edge-level explainabilty masks are given based
    on whether nodes and edges are part of a certain motif or not.

    For example, to generate a random Barabasi-Albert (BA) graph with 300
    nodes, in which we want to randomly attach 80 :obj:`"house"` motifs, write:

    .. code-block:: python

        from torch_geometric.datasets import ExplainerDataset
        from torch_geometric.datasets.graph_generator import BAGraph

        dataset = ExplainerDataset(
            graph_generator=BAGraph(num_nodes=300, num_edges=5),
            motif_generator='house',
            num_motifs=80,
        )

    .. note::

        For an example of using :class:`ExplainerDataset`, see
        `examples/explain/gnn_explainer_ba_shapes.py
        `_.

    Args:
        graph_generator (GraphGenerator or str): The graph generator to be
            used, *e.g.*,
            :class:`torch.geometric.datasets.graph_generator.BAGraph`
            (or any string that automatically resolves to it).
        motif_generator (MotifGenerator): The motif generator to be used,
            *e.g.*,
            :class:`torch_geometric.datasets.motif_generator.HouseMotif`
            (or any string that automatically resolves to it).
        num_motifs (int): The number of motifs to attach to the graph.
        num_graphs (int, optional): The number of graphs to generate.
            (default: :obj:`1`)
        graph_generator_kwargs (Dict[str, Any], optional): Arguments passed to
            the respective graph generator module in case it gets automatically
            resolved. (default: :obj:`None`)
        motif_generator_kwargs (Dict[str, Any], optional): Arguments passed to
            the respective motif generator module in case it gets automatically
            resolved. (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        graph_generator: Union[GraphGenerator, str],
        motif_generator: Union[MotifGenerator, str],
        num_motifs: int,
        num_graphs: int = 1,
        graph_generator_kwargs: Optional[Dict[str, Any]] = None,
        motif_generator_kwargs: Optional[Dict[str, Any]] = None,
        transform: Optional[Callable] = None,
    ):
        super().__init__(root=None, transform=transform)

        if num_motifs <= 0:
            raise ValueError(f"At least one motif needs to be attached to the "
                             f"graph (got {num_motifs})")

        self.graph_generator = GraphGenerator.resolve(
            graph_generator,
            **(graph_generator_kwargs or {}),
        )
        self.motif_generator = MotifGenerator.resolve(
            motif_generator,
            **(motif_generator_kwargs or {}),
        )
        self.num_motifs = num_motifs

        # TODO (matthias) support on-the-fly graph generation.
        data_list = [self.get_graph() for _ in range(num_graphs)]
        self.data, self.slices = self.collate(data_list)

    def get_graph(self) -> Explanation:
        data = self.graph_generator()

        edge_indices = [data.edge_index]
        num_nodes = data.num_nodes
        node_masks = [torch.zeros(data.num_nodes)]
        edge_masks = [torch.zeros(data.num_edges)]
        ys = [torch.zeros(num_nodes, dtype=torch.long)]

        connecting_nodes = torch.randperm(num_nodes)[:self.num_motifs]
        for i in connecting_nodes.tolist():
            motif = self.motif_generator()

            # Add motif to the graph.
            edge_indices.append(motif.edge_index + num_nodes)
            node_masks.append(torch.ones(motif.num_nodes))
            edge_masks.append(torch.ones(motif.num_edges))

            # Add random motif connection to the graph.
            j = int(torch.randint(0, motif.num_nodes, (1, ))) + num_nodes
            edge_indices.append(torch.tensor([[i, j], [j, i]]))
            edge_masks.append(torch.zeros(2))

            if 'y' in motif:
                ys.append(motif.y + 1 if motif.y.min() == 0 else motif.y)

            num_nodes += motif.num_nodes

        return Explanation(
            edge_index=torch.cat(edge_indices, dim=1),
            y=torch.cat(ys, dim=0) if len(ys) > 1 else None,
            edge_mask=torch.cat(edge_masks, dim=0),
            node_mask=torch.cat(node_masks, dim=0),
        )

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'graph_generator={self.graph_generator}, '
                f'motif_generator={self.motif_generator}, '
                f'num_motifs={self.num_motifs})')
pytorch/torch_geometric/datasets/facebook.py000066400000000000000000000042641444262304300216660ustar00rootroot00000000000000from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class FacebookPagePage(InMemoryDataset):
    r"""The Facebook Page-Page network dataset introduced in the
    `"Multi-scale Attributed Node Embedding"
    `_ paper.
    Nodes represent verified pages on Facebook and edges are mutual likes.
    It contains 22,470 nodes, 342,004 edges, 128 node features and 4 classes.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://graphmining.ai/datasets/ptg/facebook.npz'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'facebook.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        x = torch.from_numpy(data['features']).to(torch.float)
        y = torch.from_numpy(data['target']).to(torch.long)
        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/fake.py000066400000000000000000000237471444262304300210320ustar00rootroot00000000000000import random
from collections import defaultdict
from itertools import product
from typing import Callable, Optional

import torch

from torch_geometric.data import Data, HeteroData, InMemoryDataset
from torch_geometric.utils import coalesce, remove_self_loops, to_undirected


class FakeDataset(InMemoryDataset):
    r"""A fake dataset that returns randomly generated
    :class:`~torch_geometric.data.Data` objects.

    Args:
        num_graphs (int, optional): The number of graphs. (default: :obj:`1`)
        avg_num_nodes (int, optional): The average number of nodes in a graph.
            (default: :obj:`1000`)
        avg_degree (int, optional): The average degree per node.
            (default: :obj:`10`)
        num_channels (int, optional): The number of node features.
            (default: :obj:`64`)
        edge_dim (int, optional): The number of edge features.
            (default: :obj:`0`)
        num_classes (int, optional): The number of classes in the dataset.
            (default: :obj:`10`)
        task (str, optional): Whether to return node-level or graph-level
            labels (:obj:`"node"`, :obj:`"graph"`, :obj:`"auto"`).
            If set to :obj:`"auto"`, will return graph-level labels if
            :obj:`num_graphs > 1`, and node-level labels other-wise.
            (default: :obj:`"auto"`)
        is_undirected (bool, optional): Whether the graphs to generate are
            undirected. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        **kwargs (optional): Additional attributes and their shapes
            *e.g.* :obj:`global_features=5`.
    """
    def __init__(
        self,
        num_graphs: int = 1,
        avg_num_nodes: int = 1000,
        avg_degree: int = 10,
        num_channels: int = 64,
        edge_dim: int = 0,
        num_classes: int = 10,
        task: str = "auto",
        is_undirected: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        **kwargs,
    ):
        super().__init__('.', transform)

        if task == 'auto':
            task = 'graph' if num_graphs > 1 else 'node'
        assert task in ['node', 'graph']

        self.avg_num_nodes = max(avg_num_nodes, avg_degree)
        self.avg_degree = max(avg_degree, 1)
        self.num_channels = num_channels
        self.edge_dim = edge_dim
        self._num_classes = num_classes
        self.task = task
        self.is_undirected = is_undirected
        self.kwargs = kwargs

        data_list = [self.generate_data() for _ in range(max(num_graphs, 1))]
        self.data, self.slices = self.collate(data_list)

    def generate_data(self) -> Data:
        num_nodes = get_num_nodes(self.avg_num_nodes, self.avg_degree)

        data = Data()

        if self._num_classes > 0 and self.task == 'node':
            data.y = torch.randint(self._num_classes, (num_nodes, ))
        elif self._num_classes > 0 and self.task == 'graph':
            data.y = torch.tensor([random.randint(0, self._num_classes - 1)])

        data.edge_index = get_edge_index(num_nodes, num_nodes, self.avg_degree,
                                         self.is_undirected, remove_loops=True)

        if self.num_channels > 0 and self.task == 'graph':
            data.x = torch.randn(num_nodes, self.num_channels) + data.y
        elif self.num_channels > 0 and self.task == 'node':
            data.x = torch.randn(num_nodes,
                                 self.num_channels) + data.y.unsqueeze(1)
        else:
            data.num_nodes = num_nodes

        if self.edge_dim > 1:
            data.edge_attr = torch.rand(data.num_edges, self.edge_dim)
        elif self.edge_dim == 1:
            data.edge_weight = torch.rand(data.num_edges)

        for feature_name, feature_shape in self.kwargs.items():
            setattr(data, feature_name, torch.randn(feature_shape))

        return data


class FakeHeteroDataset(InMemoryDataset):
    r"""A fake dataset that returns randomly generated
    :class:`~torch_geometric.data.HeteroData` objects.

    Args:
        num_graphs (int, optional): The number of graphs. (default: :obj:`1`)
        num_node_types (int, optional): The number of node types.
            (default: :obj:`3`)
        num_edge_types (int, optional): The number of edge types.
            (default: :obj:`6`)
        avg_num_nodes (int, optional): The average number of nodes in a graph.
            (default: :obj:`1000`)
        avg_degree (int, optional): The average degree per node.
            (default: :obj:`10`)
        avg_num_channels (int, optional): The average number of node features.
            (default: :obj:`64`)
        edge_dim (int, optional): The number of edge features.
            (default: :obj:`0`)
        num_classes (int, optional): The number of classes in the dataset.
            (default: :obj:`10`)
        task (str, optional): Whether to return node-level or graph-level
            labels (:obj:`"node"`, :obj:`"graph"`, :obj:`"auto"`).
            If set to :obj:`"auto"`, will return graph-level labels if
            :obj:`num_graphs > 1`, and node-level labels other-wise.
            (default: :obj:`"auto"`)
        transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        **kwargs (optional): Additional attributes and their shapes
            *e.g.* :obj:`global_features=5`.
    """
    def __init__(
        self,
        num_graphs: int = 1,
        num_node_types: int = 3,
        num_edge_types: int = 6,
        avg_num_nodes: int = 1000,
        avg_degree: int = 10,
        avg_num_channels: int = 64,
        edge_dim: int = 0,
        num_classes: int = 10,
        task: str = "auto",
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        **kwargs,
    ):
        super().__init__('.', transform)

        if task == 'auto':
            task = 'graph' if num_graphs > 1 else 'node'
        assert task in ['node', 'graph']

        self.node_types = [f'v{i}' for i in range(max(num_node_types, 1))]

        edge_types = []
        edge_type_product = list(product(self.node_types, self.node_types))
        while len(edge_types) < max(num_edge_types, 1):
            edge_types.extend(edge_type_product)
        random.shuffle(edge_types)

        self.edge_types = []
        count = defaultdict(lambda: 0)
        for edge_type in edge_types[:max(num_edge_types, 1)]:
            rel = f'e{count[edge_type]}'
            count[edge_type] += 1
            self.edge_types.append((edge_type[0], rel, edge_type[1]))

        self.avg_num_nodes = max(avg_num_nodes, avg_degree)
        self.avg_degree = max(avg_degree, 1)
        self.num_channels = [
            get_num_channels(avg_num_channels) for _ in self.node_types
        ]
        self.edge_dim = edge_dim
        self._num_classes = num_classes
        self.task = task
        self.kwargs = kwargs

        data_list = [self.generate_data() for _ in range(max(num_graphs, 1))]
        self.data, self.slices = self.collate(data_list)

    def generate_data(self) -> HeteroData:
        data = HeteroData()

        iterator = zip(self.node_types, self.num_channels)
        for i, (node_type, num_channels) in enumerate(iterator):
            num_nodes = get_num_nodes(self.avg_num_nodes, self.avg_degree)

            store = data[node_type]

            if num_channels > 0:
                store.x = torch.randn(num_nodes, num_channels)
            else:
                store.num_nodes = num_nodes

            if self._num_classes > 0 and self.task == 'node' and i == 0:
                store.y = torch.randint(self._num_classes, (num_nodes, ))

        for (src, rel, dst) in self.edge_types:
            store = data[(src, rel, dst)]

            store.edge_index = get_edge_index(
                data[src].num_nodes,
                data[dst].num_nodes,
                self.avg_degree,
                is_undirected=False,
                remove_loops=False,
            )

            if self.edge_dim > 1:
                store.edge_attr = torch.rand(store.num_edges, self.edge_dim)
            elif self.edge_dim == 1:
                store.edge_weight = torch.rand(store.num_edges)

            pass

        if self._num_classes > 0 and self.task == 'graph':
            data.y = torch.tensor([random.randint(0, self._num_classes - 1)])

        for feature_name, feature_shape in self.kwargs.items():
            setattr(data, feature_name, torch.randn(feature_shape))

        return data


###############################################################################


def get_num_nodes(avg_num_nodes: int, avg_degree: int) -> int:
    min_num_nodes = max(3 * avg_num_nodes // 4, avg_degree)
    max_num_nodes = 5 * avg_num_nodes // 4
    return random.randint(min_num_nodes, max_num_nodes)


def get_num_channels(num_channels) -> int:
    min_num_channels = 3 * num_channels // 4
    max_num_channels = 5 * num_channels // 4
    return random.randint(min_num_channels, max_num_channels)


def get_edge_index(num_src_nodes: int, num_dst_nodes: int, avg_degree: int,
                   is_undirected: bool = False,
                   remove_loops: bool = False) -> torch.Tensor:
    num_edges = num_src_nodes * avg_degree
    row = torch.randint(num_src_nodes, (num_edges, ), dtype=torch.int64)
    col = torch.randint(num_dst_nodes, (num_edges, ), dtype=torch.int64)
    edge_index = torch.stack([row, col], dim=0)

    if remove_loops:
        edge_index, _ = remove_self_loops(edge_index)

    num_nodes = max(num_src_nodes, num_dst_nodes)
    if is_undirected:
        edge_index = to_undirected(edge_index, num_nodes=num_nodes)
    else:
        edge_index = coalesce(edge_index, num_nodes=num_nodes)

    return edge_index
pytorch/torch_geometric/datasets/faust.py000066400000000000000000000075241444262304300212410ustar00rootroot00000000000000import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import InMemoryDataset, extract_zip
from torch_geometric.io import read_ply


class FAUST(InMemoryDataset):
    r"""The FAUST humans dataset from the `"FAUST: Dataset and Evaluation for
    3D Mesh Registration"
    `_ paper,
    containing 100 watertight meshes representing 10 different poses for 10
    different subjects.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 100
          - 6,890
          - 41,328
          - 3
          - 10
    """

    url = 'http://faust.is.tue.mpg.de/'

    def __init__(self, root: str, train: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> str:
        return 'MPI-FAUST.zip'

    @property
    def processed_file_names(self) -> List[str]:
        return ['training.pt', 'test.pt']

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download '{self.raw_file_names}' from "
            f"'{self.url}' and move it to '{self.raw_dir}'")

    def process(self):
        extract_zip(self.raw_paths[0], self.raw_dir, log=False)

        path = osp.join(self.raw_dir, 'MPI-FAUST', 'training', 'registrations')
        path = osp.join(path, 'tr_reg_{0:03d}.ply')
        data_list = []
        for i in range(100):
            data = read_ply(path.format(i))
            data.y = torch.tensor([i % 10], dtype=torch.long)
            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)
            data_list.append(data)

        torch.save(self.collate(data_list[:80]), self.processed_paths[0])
        torch.save(self.collate(data_list[80:]), self.processed_paths[1])

        shutil.rmtree(osp.join(self.raw_dir, 'MPI-FAUST'))
pytorch/torch_geometric/datasets/flickr.py000066400000000000000000000100031444262304300213530ustar00rootroot00000000000000import json
import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class Flickr(InMemoryDataset):
    r"""The Flickr dataset from the `"GraphSAINT: Graph Sampling Based
    Inductive Learning Method" `_ paper,
    containing descriptions and common properties of images.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 89,250
          - 899,756
          - 500
          - 7
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    adj_full_id = '1crmsTbd1-2sEXsGwa2IKnIB7Zd3TmUsy'
    feats_id = '1join-XdvX3anJU_MLVtick7MgeAQiWIZ'
    class_map_id = '1uxIkbtg5drHTsKt-PAsZZ4_yJmgFmle9'
    role_id = '1htXCtuktuCW8TR8KiKfrFDAxUgekQoV7'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['adj_full.npz', 'feats.npy', 'class_map.json', 'role.json']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url.format(self.adj_full_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'adj_full.npz'))

        path = download_url(self.url.format(self.feats_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'feats.npy'))

        path = download_url(self.url.format(self.class_map_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'class_map.json'))

        path = download_url(self.url.format(self.role_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'role.json'))

    def process(self):
        f = np.load(osp.join(self.raw_dir, 'adj_full.npz'))
        adj = sp.csr_matrix((f['data'], f['indices'], f['indptr']), f['shape'])
        adj = adj.tocoo()
        row = torch.from_numpy(adj.row).to(torch.long)
        col = torch.from_numpy(adj.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        x = np.load(osp.join(self.raw_dir, 'feats.npy'))
        x = torch.from_numpy(x).to(torch.float)

        ys = [-1] * x.size(0)
        with open(osp.join(self.raw_dir, 'class_map.json')) as f:
            class_map = json.load(f)
            for key, item in class_map.items():
                ys[int(key)] = item
        y = torch.tensor(ys)

        with open(osp.join(self.raw_dir, 'role.json')) as f:
            role = json.load(f)

        train_mask = torch.zeros(x.size(0), dtype=torch.bool)
        train_mask[torch.tensor(role['tr'])] = True

        val_mask = torch.zeros(x.size(0), dtype=torch.bool)
        val_mask[torch.tensor(role['va'])] = True

        test_mask = torch.zeros(x.size(0), dtype=torch.bool)
        test_mask[torch.tensor(role['te'])] = True

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/freebase.py000066400000000000000000000073471444262304300216760ustar00rootroot00000000000000from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class FB15k_237(InMemoryDataset):
    r"""The FB15K237 dataset from the `"Translating Embeddings for Modeling
    Multi-Relational Data"
    `_ paper,
    containing 14,541 entities, 237 relations and 310,116 fact triples.

    .. note::

        The original :class:`FB15k` dataset suffers from major test leakage
        through inverse relations, where a large number of test triples could
        be obtained by inverting triples in the training set.
        In order to create a dataset without this characteristic, the
        :class:`~torch_geometric.datasets.FB15k_237` describes a subset of
        :class:`FB15k` where inverse relations are removed.

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset. (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    url = ('https://raw.githubusercontent.com/villmow/'
           'datasets_knowledge_embedding/master/FB15k-237')

    def __init__(self, root: str, split: str = "train",
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)

        if split not in {'train', 'val', 'test'}:
            raise ValueError(f"Invalid 'split' argument (got {split})")

        path = self.processed_paths[['train', 'val', 'test'].index(split)]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        return ['train.txt', 'valid.txt', 'test.txt']

    @property
    def processed_file_names(self) -> List[str]:
        return ['train_data.pt', 'val_data.pt', 'test_data.pt']

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process(self):
        data_list, node_dict, rel_dict = [], {}, {}
        for path in self.raw_paths:
            with open(path, 'r') as f:
                data = [x.split('\t') for x in f.read().split('\n')[:-1]]

            edge_index = torch.empty((2, len(data)), dtype=torch.long)
            edge_type = torch.empty(len(data), dtype=torch.long)
            for i, (src, rel, dst) in enumerate(data):
                if src not in node_dict:
                    node_dict[src] = len(node_dict)
                if dst not in node_dict:
                    node_dict[dst] = len(node_dict)
                if rel not in rel_dict:
                    rel_dict[rel] = len(rel_dict)

                edge_index[0, i] = node_dict[src]
                edge_index[1, i] = node_dict[dst]
                edge_type[i] = rel_dict[rel]

            data = Data(edge_index=edge_index, edge_type=edge_type)
            data_list.append(data)

        for data, path in zip(data_list, self.processed_paths):
            data.num_nodes = len(node_dict)
            torch.save(self.collate([data]), path)
pytorch/torch_geometric/datasets/gdelt.py000066400000000000000000000065301444262304300212120ustar00rootroot00000000000000from typing import Callable, List, Optional

import torch

from torch_geometric.data import download_url
from torch_geometric.datasets.icews import EventDataset
from torch_geometric.io import read_txt_array


class GDELT(EventDataset):
    r"""The Global Database of Events, Language, and Tone (GDELT) dataset used
    in the, *e.g.*, `"Recurrent Event Network for Reasoning over Temporal
    Knowledge Graphs" `_ paper, consisting of
    events collected from 1/1/2018 to 1/31/2018 (15 minutes time granularity).

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset. (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'https://github.com/INK-USC/RE-Net/raw/master/data/GDELT'
    splits = [0, 1734399, 1973164, 2278405]  # Train/Val/Test splits.

    def __init__(self, root: str, split: str = "train",
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        assert split in ['train', 'val', 'test']
        super().__init__(root, transform, pre_transform, pre_filter)
        idx = self.processed_file_names.index(f'{split}.pt')
        self.data, self.slices = torch.load(self.processed_paths[idx])

    @property
    def num_nodes(self) -> int:
        return 7691

    @property
    def num_rels(self) -> int:
        return 240

    @property
    def raw_file_names(self) -> List[str]:
        return [f'{name}.txt' for name in ['train', 'valid', 'test']]

    @property
    def processed_file_names(self) -> List[str]:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process_events(self) -> torch.Tensor:
        events = []
        for path in self.raw_paths:
            data = read_txt_array(path, sep='\t', end=4, dtype=torch.long)
            data[:, 3] = data[:, 3] // 15
            events += [data]
        return torch.cat(events, dim=0)

    def process(self):
        s = self.splits
        data_list = super().process()
        torch.save(self.collate(data_list[s[0]:s[1]]), self.processed_paths[0])
        torch.save(self.collate(data_list[s[1]:s[2]]), self.processed_paths[1])
        torch.save(self.collate(data_list[s[2]:s[3]]), self.processed_paths[2])
pytorch/torch_geometric/datasets/gdelt_lite.py000066400000000000000000000056271444262304300222350ustar00rootroot00000000000000import os
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class GDELTLite(InMemoryDataset):
    r"""The (reduced) version of the Global Database of Events, Language, and
    Tone (GDELT) dataset used in the `"Do We Really Need Complicated Model
    Architectures for Temporal Networks?" `_
    paper, consisting of events collected from 2016 to 2020.

    Each node (actor) holds a 413-dimensional multi-hot feature vector that
    represents CAMEO codes attached to the corresponding actor to server.

    Each edge (event) holds a timestamp and a 186-dimensional multi-hot vector
    representing CAMEO codes attached to the corresponding event to server.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 8,831
          - 1,912,909
          - 413
          -
    """
    url = 'https://data.pyg.org/datasets/gdelt_lite.zip'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['node_features.pt', 'edges.csv', 'edge_features.pt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        import pandas as pd

        x = torch.load(self.raw_paths[0])
        df = pd.read_csv(self.raw_paths[1])
        edge_attr = torch.load(self.raw_paths[2])

        row = torch.from_numpy(df['src'].values)
        col = torch.from_numpy(df['dst'].values)
        edge_index = torch.stack([row, col], dim=0)
        time = torch.from_numpy(df['time'].values).to(torch.long)

        data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr, time=time)
        data = data if self.pre_transform is None else self.pre_transform(data)

        self.save([data], self.processed_paths[0])
pytorch/torch_geometric/datasets/ged_dataset.py000066400000000000000000000220551444262304300223570ustar00rootroot00000000000000import glob
import os
import os.path as osp
import pickle
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
    extract_zip,
)
from torch_geometric.utils import one_hot, to_undirected


class GEDDataset(InMemoryDataset):
    r"""The GED datasets from the `"Graph Edit Distance Computation via Graph
    Neural Networks" `_ paper.
    GEDs can be accessed via the global attributes :obj:`ged` and
    :obj:`norm_ged` for all train/train graph pairs and all train/test graph
    pairs:

    .. code-block:: python

        dataset = GEDDataset(root, name="LINUX")
        data1, data2 = dataset[0], dataset[1]
        ged = dataset.ged[data1.i, data2.i]  # GED between `data1` and `data2`.

    Note that GEDs are not available if both graphs are from the test set.
    For evaluation, it is recommended to pair up each graph from the test set
    with each graph in the training set.

    .. note::

        :obj:`ALKANE` is missing GEDs for train/test graph pairs since they are
        not provided in the `official datasets
        `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (one of :obj:`"AIDS700nef"`,
            :obj:`"LINUX"`, :obj:`"ALKANE"`, :obj:`"IMDBMulti"`).
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - AIDS700nef
          - 700
          - ~8.9
          - ~17.6
          - 29
          - 0
        * - LINUX
          - 1,000
          - ~7.6
          - ~13.9
          - 0
          - 0
        * - ALKANE
          - 150
          - ~8.9
          - ~15.8
          - 0
          - 0
        * - IMDBMulti
          - 1,500
          - ~13.0
          - ~131.9
          - 0
          - 0
    """

    url = 'https://drive.google.com/uc?export=download&id={}'

    datasets = {
        'AIDS700nef': {
            'id': '10czBPJDEzEDI2tq7Z7mkBjLhj55F-a2z',
            'extract': extract_zip,
            'pickle': '1OpV4bCHjBkdpqI6H5Mg0-BqlA2ee2eBW',
        },
        'LINUX': {
            'id': '1nw0RRVgyLpit4V4XFQyDy0pI6wUEXSOI',
            'extract': extract_tar,
            'pickle': '14FDm3NSnrBvB7eNpLeGy5Bz6FjuCSF5v',
        },
        'ALKANE': {
            'id': '1-LmxaWW3KulLh00YqscVEflbqr0g4cXt',
            'extract': extract_tar,
            'pickle': '15BpvMuHx77-yUGYgM27_sQett02HQNYu',
        },
        'IMDBMulti': {
            'id': '12QxZ7EhYA7pJiF4cO-HuE8szhSOWcfST',
            'extract': extract_zip,
            'pickle': '1wy9VbZvZodkixxVIOuRllC-Lp-0zdoYZ',
        },
    }

    # List of atoms contained in the AIDS700nef dataset:
    types = [
        'O', 'S', 'C', 'N', 'Cl', 'Br', 'B', 'Si', 'Hg', 'I', 'Bi', 'P', 'F',
        'Cu', 'Ho', 'Pd', 'Ru', 'Pt', 'Sn', 'Li', 'Ga', 'Tb', 'As', 'Co', 'Pb',
        'Sb', 'Se', 'Ni', 'Te'
    ]

    def __init__(self, root: str, name: str, train: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        self.name = name
        assert self.name in self.datasets.keys()
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)
        path = osp.join(self.processed_dir, f'{self.name}_ged.pt')
        self.ged = torch.load(path)
        path = osp.join(self.processed_dir, f'{self.name}_norm_ged.pt')
        self.norm_ged = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        # Returns, e.g., ['LINUX/train', 'LINUX/test']
        return [osp.join(self.name, s) for s in ['train', 'test']]

    @property
    def processed_file_names(self) -> List[str]:
        # Returns, e.g., ['LINUX_training.pt', 'LINUX_test.pt']
        return [f'{self.name}_{s}.pt' for s in ['training', 'test']]

    def download(self):
        # Downloads the .tar/.zip file of the graphs and extracts them:
        name = self.datasets[self.name]['id']
        path = download_url(self.url.format(name), self.raw_dir)
        self.datasets[self.name]['extract'](path, self.raw_dir)
        os.unlink(path)

        # Downloads the pickle file containing pre-computed GEDs:
        name = self.datasets[self.name]['pickle']
        path = download_url(self.url.format(name), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, self.name, 'ged.pickle'))

    def process(self):
        import networkx as nx

        ids, Ns = [], []
        # Iterating over paths for raw and processed data (train + test):
        for r_path, p_path in zip(self.raw_paths, self.processed_paths):
            # Find the paths of all raw graphs:
            names = glob.glob(osp.join(r_path, '*.gexf'))
            # Get sorted graph IDs given filename: 123.gexf -> 123
            ids.append(sorted([int(i.split(os.sep)[-1][:-5]) for i in names]))

            data_list = []
            # Convert graphs in .gexf format to a NetworkX Graph:
            for i, idx in enumerate(ids[-1]):
                i = i if len(ids) == 1 else i + len(ids[0])
                # Reading the raw `*.gexf` graph:
                G = nx.read_gexf(osp.join(r_path, f'{idx}.gexf'))
                # Mapping of nodes in `G` to a contiguous number:
                mapping = {name: j for j, name in enumerate(G.nodes())}
                G = nx.relabel_nodes(G, mapping)
                Ns.append(G.number_of_nodes())

                edge_index = torch.tensor(list(G.edges)).t().contiguous()
                if edge_index.numel() == 0:
                    edge_index = torch.empty((2, 0), dtype=torch.long)
                edge_index = to_undirected(edge_index, num_nodes=Ns[-1])

                data = Data(edge_index=edge_index, i=i)
                data.num_nodes = Ns[-1]

                # Create a one-hot encoded feature matrix denoting the atom
                # type (for the `AIDS700nef` dataset):
                if self.name == 'AIDS700nef':
                    x = torch.zeros(data.num_nodes, dtype=torch.long)
                    for node, info in G.nodes(data=True):
                        x[int(node)] = self.types.index(info['type'])
                    data.x = one_hot(x, num_classes=len(self.types))

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)

            torch.save(self.collate(data_list), p_path)

        assoc = {idx: i for i, idx in enumerate(ids[0])}
        assoc.update({idx: i + len(ids[0]) for i, idx in enumerate(ids[1])})

        # Extracting ground-truth GEDs from the GED pickle file
        path = osp.join(self.raw_dir, self.name, 'ged.pickle')
        # Initialize GEDs as float('inf'):
        mat = torch.full((len(assoc), len(assoc)), float('inf'))
        with open(path, 'rb') as f:
            obj = pickle.load(f)
            xs, ys, gs = [], [], []
            for (x, y), g in obj.items():
                xs += [assoc[x]]
                ys += [assoc[y]]
                gs += [g]
            # The pickle file does not contain GEDs for test graph pairs, i.e.
            # GEDs for (test_graph, test_graph) pairs are still float('inf'):
            x, y = torch.tensor(xs), torch.tensor(ys)
            ged = torch.tensor(gs, dtype=torch.float)
            mat[x, y], mat[y, x] = ged, ged

        path = osp.join(self.processed_dir, f'{self.name}_ged.pt')
        torch.save(mat, path)

        # Calculate the normalized GEDs:
        N = torch.tensor(Ns, dtype=torch.float)
        norm_mat = mat / (0.5 * (N.view(-1, 1) + N.view(1, -1)))

        path = osp.join(self.processed_dir, f'{self.name}_norm_ged.pt')
        torch.save(norm_mat, path)

    def __repr__(self) -> str:
        return f'{self.name}({len(self)})'
pytorch/torch_geometric/datasets/gemsec.py000066400000000000000000000051021444262304300213500ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class GemsecDeezer(InMemoryDataset):
    r"""The Deezer User Network datasets introduced in the
    `"GEMSEC: Graph Embedding with Self Clustering"
    `_ paper.
    Nodes represent Deezer user and edges are mutual friendships.
    The task is multi-label multi-class node classification about
    the genres liked by the users.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"HU"`, :obj:`"HR"`,
            :obj:`"RO"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://graphmining.ai/datasets/ptg/gemsec'

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name
        assert self.name in ['HU', 'HR', 'RO']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(osp.join(self.url, self.name + '.npz'), self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        y = torch.from_numpy(data['target']).to(torch.long)
        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/geometry.py000066400000000000000000000075621444262304300217540ustar00rootroot00000000000000import glob
import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url, extract_zip
from torch_geometric.io import read_off


class GeometricShapes(InMemoryDataset):
    r"""Synthetic dataset of various geometric shapes like cubes, spheres or
    pyramids.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 80
          - ~148.8
          - ~859.5
          - 3
          - 40
    """

    url = 'https://github.com/Yannick-S/geometric_shapes/raw/master/raw.zip'

    def __init__(self, root: str, train: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> str:
        return '2d_circle'

    @property
    def processed_file_names(self) -> List[str]:
        return ['training.pt', 'test.pt']

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.unlink(path)

    def process(self):
        torch.save(self.process_set('train'), self.processed_paths[0])
        torch.save(self.process_set('test'), self.processed_paths[1])

    def process_set(self, dataset: str):
        categories = glob.glob(osp.join(self.raw_dir, '*', ''))
        categories = sorted([x.split(os.sep)[-2] for x in categories])

        data_list = []
        for target, category in enumerate(categories):
            folder = osp.join(self.raw_dir, category, dataset)
            paths = glob.glob(f'{folder}/*.off')
            for path in paths:
                data = read_off(path)
                data.pos = data.pos - data.pos.mean(dim=0, keepdim=True)
                data.y = torch.tensor([target])
                data_list.append(data)

        if self.pre_filter is not None:
            data_list = [d for d in data_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(d) for d in data_list]

        return self.collate(data_list)
pytorch/torch_geometric/datasets/github.py000066400000000000000000000046641444262304300214030ustar00rootroot00000000000000from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class GitHub(InMemoryDataset):
    r"""The GitHub Web and ML Developers dataset introduced in the
    `"Multi-scale Attributed Node Embedding"
    `_ paper.
    Nodes represent developers on :obj:`github:`GitHub` and edges are mutual
    follower relationships.
    It contains 37,300 nodes, 578,006 edges, 128 node features and 2 classes.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 37,700
          - 578,006
          - 0
          - 2
    """
    url = 'https://graphmining.ai/datasets/ptg/github.npz'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'github.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        x = torch.from_numpy(data['features']).to(torch.float)
        y = torch.from_numpy(data['target']).to(torch.long)
        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/gnn_benchmark_dataset.py000066400000000000000000000151521444262304300244140ustar00rootroot00000000000000import logging
import os
import os.path as osp
import pickle
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.utils import remove_self_loops


class GNNBenchmarkDataset(InMemoryDataset):
    r"""A variety of artificially and semi-artificially generated graph
    datasets from the `"Benchmarking Graph Neural Networks"
    `_ paper.

    .. note::
        The ZINC dataset is provided via
        :class:`torch_geometric.datasets.ZINC`.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (one of :obj:`"PATTERN"`,
            :obj:`"CLUSTER"`, :obj:`"MNIST"`, :obj:`"CIFAR10"`,
            :obj:`"TSP"`, :obj:`"CSL"`)
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - PATTERN
          - 10,000
          - ~118.9
          - ~6,098.9
          - 3
          - 2
        * - CLUSTER
          - 10,000
          - ~117.2
          - ~4,303.9
          - 7
          - 6
        * - MNIST
          - 55,000
          - ~70.6
          - ~564.5
          - 3
          - 10
        * - CIFAR10
          - 45,000
          - ~117.6
          - ~941.2
          - 5
          - 10
        * - TSP
          - 10,000
          - ~275.4
          - ~6,885.0
          - 2
          - 2
        * - CSL
          - 150
          - ~41.0
          - ~164.0
          - 0
          - 10
    """

    names = ['PATTERN', 'CLUSTER', 'MNIST', 'CIFAR10', 'TSP', 'CSL']

    root_url = 'https://data.pyg.org/datasets/benchmarking-gnns'
    urls = {
        'PATTERN': f'{root_url}/PATTERN_v2.zip',
        'CLUSTER': f'{root_url}/CLUSTER_v2.zip',
        'MNIST': f'{root_url}/MNIST_v2.zip',
        'CIFAR10': f'{root_url}/CIFAR10_v2.zip',
        'TSP': f'{root_url}/TSP_v2.zip',
        'CSL': 'https://www.dropbox.com/s/rnbkp5ubgk82ocu/CSL.zip?dl=1',
    }

    def __init__(self, root: str, name: str, split: str = "train",
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        self.name = name
        assert self.name in self.names

        if self.name == 'CSL' and split != 'train':
            split = 'train'
            logging.warning(
                ("Dataset 'CSL' does not provide a standardized splitting. "
                 "Instead, it is recommended to perform 5-fold cross "
                 "validation with stratifed sampling"))

        super().__init__(root, transform, pre_transform, pre_filter)

        if split == 'train':
            path = self.processed_paths[0]
        elif split == 'val':
            path = self.processed_paths[1]
        elif split == 'test':
            path = self.processed_paths[2]
        else:
            raise ValueError(f"Split '{split}' found, but expected either "
                             f"'train', 'val', or 'test'")

        self.data, self.slices = torch.load(path)

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        if self.name == 'CSL':
            return [
                'graphs_Kary_Deterministic_Graphs.pkl',
                'y_Kary_Deterministic_Graphs.pt'
            ]
        else:
            name = self.urls[self.name].split('/')[-1][:-4]
            return [f'{name}.pt']

    @property
    def processed_file_names(self) -> List[str]:
        if self.name == 'CSL':
            return ['data.pt']
        else:
            return ['train_data.pt', 'val_data.pt', 'test_data.pt']

    def download(self):
        path = download_url(self.urls[self.name], self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        if self.name == 'CSL':
            data_list = self.process_CSL()
            torch.save(self.collate(data_list), self.processed_paths[0])
        else:
            inputs = torch.load(self.raw_paths[0])
            for i in range(len(inputs)):
                data_list = [Data(**data_dict) for data_dict in inputs[i]]

                if self.pre_filter is not None:
                    data_list = [d for d in data_list if self.pre_filter(d)]

                if self.pre_transform is not None:
                    data_list = [self.pre_transform(d) for d in data_list]

                torch.save(self.collate(data_list), self.processed_paths[i])

    def process_CSL(self) -> List[Data]:
        with open(self.raw_paths[0], 'rb') as f:
            adjs = pickle.load(f)

        ys = torch.load(self.raw_paths[1]).tolist()

        data_list = []
        for adj, y in zip(adjs, ys):
            row, col = torch.from_numpy(adj.row), torch.from_numpy(adj.col)
            edge_index = torch.stack([row, col], dim=0).to(torch.long)
            edge_index, _ = remove_self_loops(edge_index)
            data = Data(edge_index=edge_index, y=y, num_nodes=adj.shape[0])
            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)
            data_list.append(data)
        return data_list

    def __repr__(self) -> str:
        return f'{self.name}({len(self)})'
pytorch/torch_geometric/datasets/graph_generator/000077500000000000000000000000001444262304300227045ustar00rootroot00000000000000pytorch/torch_geometric/datasets/graph_generator/__init__.py000066400000000000000000000003351444262304300250160ustar00rootroot00000000000000from .base import GraphGenerator
from .ba_graph import BAGraph
from .er_graph import ERGraph
from .grid_graph import GridGraph

__all__ = classes = [
    'GraphGenerator',
    'BAGraph',
    'ERGraph',
    'GridGraph',
]
pytorch/torch_geometric/datasets/graph_generator/ba_graph.py000066400000000000000000000017051444262304300250240ustar00rootroot00000000000000from torch_geometric.data import Data
from torch_geometric.datasets.graph_generator import GraphGenerator
from torch_geometric.utils import barabasi_albert_graph


class BAGraph(GraphGenerator):
    r"""Generates random Barabasi-Albert (BA) graphs.
    See :meth:`~torch_geometric.utils.barabasi_albert_graph` for more
    information.

    Args:
        num_nodes (int): The number of nodes.
        num_edges (int): The number of edges from a new node to existing nodes.
    """
    def __init__(self, num_nodes: int, num_edges: int):
        super().__init__()
        self.num_nodes = num_nodes
        self.num_edges = num_edges

    def __call__(self) -> Data:
        edge_index = barabasi_albert_graph(self.num_nodes, self.num_edges)
        return Data(num_nodes=self.num_nodes, edge_index=edge_index)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_nodes={self.num_nodes}, '
                f'num_edges={self.num_edges})')
pytorch/torch_geometric/datasets/graph_generator/base.py000066400000000000000000000016651444262304300242000ustar00rootroot00000000000000from abc import ABC, abstractmethod
from typing import Any, Union

from torch_geometric.data import Data
from torch_geometric.resolver import resolver


class GraphGenerator(ABC):
    r"""An abstract base class for generating synthetic graphs."""
    @abstractmethod
    def __call__(self) -> Data:
        r"""To be implemented by :class:`GraphGenerator` subclasses."""
        raise NotImplementedError

    @staticmethod
    def resolve(query: Union[Any, str], *args, **kwargs):
        import torch_geometric.datasets.graph_generator as graph_generators
        graph_generators = [
            gen for gen in vars(graph_generators).values()
            if isinstance(gen, type) and issubclass(gen, GraphGenerator)
        ]
        return resolver(graph_generators, {}, query, base_cls=GraphGenerator,
                        base_cls_repr='Graph', *args, **kwargs)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/datasets/graph_generator/er_graph.py000066400000000000000000000016261444262304300250520ustar00rootroot00000000000000from torch_geometric.data import Data
from torch_geometric.datasets.graph_generator import GraphGenerator
from torch_geometric.utils import erdos_renyi_graph


class ERGraph(GraphGenerator):
    r"""Generates random Erdos-Renyi (ER) graphs.
    See :meth:`~torch_geometric.utils.erdos_renyi_graph` for more information.

    Args:
        num_nodes (int): The number of nodes.
        edge_prob (float): Probability of an edge.
    """
    def __init__(self, num_nodes: int, edge_prob: float):
        super().__init__()
        self.num_nodes = num_nodes
        self.edge_prob = edge_prob

    def __call__(self) -> Data:
        edge_index = erdos_renyi_graph(self.num_nodes, self.edge_prob)
        return Data(num_nodes=self.num_nodes, edge_index=edge_index)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_nodes={self.num_nodes}, '
                f'edge_prob={self.edge_prob})')
pytorch/torch_geometric/datasets/graph_generator/grid_graph.py000066400000000000000000000022071444262304300253650ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.datasets.graph_generator import GraphGenerator
from torch_geometric.utils import grid


class GridGraph(GraphGenerator):
    r"""Generates two-dimensional grid graphs.
    See :meth:`~torch_geometric.utils.grid` for more information.

    Args:
        height (int): The height of the grid.
        width (int): The width of the grid.
        dtype (:obj:`torch.dtype`, optional): The desired data type of the
            returned position tensor. (default: :obj:`None`)
    """
    def __init__(
        self,
        height: int,
        width: int,
        dtype: Optional[torch.dtype] = None,
    ):
        super().__init__()
        self.height = height
        self.width = width
        self.dtype = dtype

    def __call__(self) -> Data:
        edge_index, pos = grid(height=self.height, width=self.width,
                               dtype=self.dtype)
        return Data(edge_index=edge_index, pos=pos)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(height={self.height}, '
                f'width={self.width})')
pytorch/torch_geometric/datasets/heterophilous_graph_dataset.py000066400000000000000000000077131444262304300256770ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import to_undirected


class HeterophilousGraphDataset(InMemoryDataset):
    r"""The heterophilous graphs :obj:`"Roman-empire"`,
    :obj:`"Amazon-ratings"`, :obj:`"Minesweeper"`, :obj:`"Tolokers"` and
    :obj:`"Questions"` from the `"A Critical Look at the Evaluation of GNNs
    under Heterophily: Are We Really Making Progress?"
    `_ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Roman-empire"`,
            :obj:`"Amazon-ratings"`, :obj:`"Minesweeper"`, :obj:`"Tolokers"`,
            :obj:`"Questions"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Roman-empire
          - 22,662
          - 32,927
          - 300
          - 18
        * - Amazon-ratings
          - 24,492
          - 93,050
          - 300
          - 5
        * - Minesweeper
          - 10,000
          - 39,402
          - 7
          - 2
        * - Tolokers
          - 11,758
          - 519,000
          - 10
          - 2
        * - Questions
          - 48,921
          - 153,540
          - 301
          - 2
    """
    url = ('https://github.com/yandex-research/heterophilous-graphs/raw/'
           'main/data')

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name.lower().replace('-', '_')
        assert self.name in [
            'roman_empire',
            'amazon_ratings',
            'minesweeper',
            'tolokers',
            'questions',
        ]

        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(f'{self.url}/{self.name}.npz', self.raw_dir)

    def process(self):
        raw = np.load(self.raw_paths[0], 'r')
        x = torch.from_numpy(raw['node_features'])
        y = torch.from_numpy(raw['node_labels'])
        edge_index = torch.from_numpy(raw['edges']).t().contiguous()
        edge_index = to_undirected(edge_index, num_nodes=x.size(0))
        train_mask = torch.from_numpy(raw['train_masks']).t().contiguous()
        val_mask = torch.from_numpy(raw['val_masks']).t().contiguous()
        test_mask = torch.from_numpy(raw['test_masks']).t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(name={self.name})'
pytorch/torch_geometric/datasets/hgb_dataset.py000066400000000000000000000204231444262304300223550ustar00rootroot00000000000000import json
import os
import os.path as osp
from collections import defaultdict
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class HGBDataset(InMemoryDataset):
    r"""A variety of heterogeneous graph benchmark datasets from the
    `"Are We Really Making Much Progress? Revisiting, Benchmarking, and
    Refining Heterogeneous Graph Neural Networks"
    `_ paper.

    .. note::
        Test labels are randomly given to prevent data leakage issues.
        If you want to obtain final test performance, you will need to submit
        your model predictions to the
        `HGB leaderboard `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (one of :obj:`"ACM"`,
            :obj:`"DBLP"`, :obj:`"Freebase"`, :obj:`"IMDB"`)
        transform (callable, optional): A function/transform that takes in an
            :class:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :class:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = ('https://cloud.tsinghua.edu.cn/d/2d965d2fc2ee41d09def/files/'
           '?p=%2F{}.zip&dl=1')

    names = {
        'acm': 'ACM',
        'dblp': 'DBLP',
        'freebase': 'Freebase',
        'imdb': 'IMDB',
    }

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        assert self.name in set(self.names.keys())
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        x = ['info.dat', 'node.dat', 'link.dat', 'label.dat', 'label.dat.test']
        return [osp.join(self.names[self.name], f) for f in x]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        url = self.url.format(self.names[self.name])
        path = download_url(url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        data = HeteroData()

        # node_types = {0: 'paper', 1, 'author', ...}
        # edge_types = {0: ('paper', 'cite', 'paper'), ...}
        if self.name in ['acm', 'dblp', 'imdb']:
            with open(self.raw_paths[0], 'r') as f:  # `info.dat`
                info = json.load(f)
            n_types = info['node.dat']['node type']
            n_types = {int(k): v for k, v in n_types.items()}
            e_types = info['link.dat']['link type']
            e_types = {int(k): tuple(v.values()) for k, v in e_types.items()}
            for key, (src, dst, rel) in e_types.items():
                src, dst = n_types[int(src)], n_types[int(dst)]
                rel = rel.split('-')[1]
                rel = rel if rel != dst and rel[1:] != dst else 'to'
                e_types[key] = (src, rel, dst)
            num_classes = len(info['label.dat']['node type']['0'])
        elif self.name in ['freebase']:
            with open(self.raw_paths[0], 'r') as f:  # `info.dat`
                info = f.read().split('\n')
            start = info.index('TYPE\tMEANING') + 1
            end = info[start:].index('')
            n_types = [v.split('\t\t') for v in info[start:start + end]]
            n_types = {int(k): v.lower() for k, v in n_types}

            e_types = {}
            start = info.index('LINK\tSTART\tEND\tMEANING') + 1
            end = info[start:].index('')
            for key, row in enumerate(info[start:start + end]):
                row = row.split('\t')[1:]
                src, dst, rel = [v for v in row if v != '']
                src, dst = n_types[int(src)], n_types[int(dst)]
                rel = rel.split('-')[1]
                e_types[key] = (src, rel, dst)
        else:  # Link prediction:
            raise NotImplementedError

        # Extract node information:
        mapping_dict = {}  # Maps global node indices to local ones.
        x_dict = defaultdict(list)
        num_nodes_dict = defaultdict(lambda: 0)
        with open(self.raw_paths[1], 'r') as f:  # `node.dat`
            xs = [v.split('\t') for v in f.read().split('\n')[:-1]]
        for x in xs:
            n_id, n_type = int(x[0]), n_types[int(x[2])]
            mapping_dict[n_id] = num_nodes_dict[n_type]
            num_nodes_dict[n_type] += 1
            if len(x) >= 4:  # Extract features (in case they are given).
                x_dict[n_type].append([float(v) for v in x[3].split(',')])
        for n_type in n_types.values():
            if len(x_dict[n_type]) == 0:
                data[n_type].num_nodes = num_nodes_dict[n_type]
            else:
                data[n_type].x = torch.tensor(x_dict[n_type])

        edge_index_dict = defaultdict(list)
        edge_weight_dict = defaultdict(list)
        with open(self.raw_paths[2], 'r') as f:  # `link.dat`
            edges = [v.split('\t') for v in f.read().split('\n')[:-1]]
        for src, dst, rel, weight in edges:
            e_type = e_types[int(rel)]
            src, dst = mapping_dict[int(src)], mapping_dict[int(dst)]
            edge_index_dict[e_type].append([src, dst])
            edge_weight_dict[e_type].append(float(weight))
        for e_type in e_types.values():
            edge_index = torch.tensor(edge_index_dict[e_type])
            edge_weight = torch.tensor(edge_weight_dict[e_type])
            data[e_type].edge_index = edge_index.t().contiguous()
            # Only add "weighted" edgel to the graph:
            if not torch.allclose(edge_weight, torch.ones_like(edge_weight)):
                data[e_type].edge_weight = edge_weight

        # Node classification:
        if self.name in ['acm', 'dblp', 'freebase', 'imdb']:
            with open(self.raw_paths[3], 'r') as f:  # `label.dat`
                train_ys = [v.split('\t') for v in f.read().split('\n')[:-1]]
            with open(self.raw_paths[4], 'r') as f:  # `label.dat.test`
                test_ys = [v.split('\t') for v in f.read().split('\n')[:-1]]
            for y in train_ys:
                n_id, n_type = mapping_dict[int(y[0])], n_types[int(y[2])]

                if not hasattr(data[n_type], 'y'):
                    num_nodes = data[n_type].num_nodes
                    if self.name in ['imdb']:  # multi-label
                        data[n_type].y = torch.zeros((num_nodes, num_classes))
                    else:
                        data[n_type].y = torch.full((num_nodes, ), -1).long()
                    data[n_type].train_mask = torch.zeros(num_nodes).bool()
                    data[n_type].test_mask = torch.zeros(num_nodes).bool()

                if data[n_type].y.dim() > 1:  # multi-label
                    for v in y[3].split(','):
                        data[n_type].y[n_id, int(v)] = 1
                else:
                    data[n_type].y[n_id] = int(y[3])
                data[n_type].train_mask[n_id] = True
            for y in test_ys:
                n_id, n_type = mapping_dict[int(y[0])], n_types[int(y[2])]
                if data[n_type].y.dim() > 1:  # multi-label
                    for v in y[3].split(','):
                        data[n_type].y[n_id, int(v)] = 1
                else:
                    data[n_type].y[n_id] = int(y[3])
                data[n_type].test_mask[n_id] = True

        else:  # Link prediction:
            raise NotImplementedError

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.names[self.name]}()'
pytorch/torch_geometric/datasets/hm.py000066400000000000000000000146151444262304300205220ustar00rootroot00000000000000from typing import Callable, List, Optional

import torch

from torch_geometric.data import HeteroData, InMemoryDataset


class HM(InMemoryDataset):
    r"""The heterogeneous H&M dataset from the `Kaggle H&M Personalized Fashion
    Recommendations
    `_
    challenge.
    The task is to develop product recommendations based on data from previous
    transactions, as well as from customer and product meta data.

    Args:
        root (str): Root directory where the dataset should be saved.
        use_all_tables_as_node_types (bool, optional): If set to :obj:`True`,
            will use the transaction table as a distinct node type.
            (default: :obj:`False`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    url = ('https://www.kaggle.com/competitions/'
           'h-and-m-personalized-fashion-recommendations/data')

    def __init__(
        self,
        root: str,
        use_all_tables_as_node_types: bool = False,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.use_all_tables_as_node_types = use_all_tables_as_node_types
        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0], data_cls=HeteroData)

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'customers.csv.zip', 'articles.csv.zip',
            'transactions_train.csv.zip'
        ]

    @property
    def processed_file_names(self) -> str:
        if self.use_all_tables_as_node_types:
            return 'data.pt'
        else:
            return 'data_merged.pt'

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download {self.raw_file_names} from "
            f"'{self.url}' and move it to '{self.raw_dir}'")

    def process(self):
        import pandas as pd

        data = HeteroData()

        # Process customer data ###############################################
        df = pd.read_csv(self.raw_paths[0], index_col='customer_id')
        customer_map = {idx: i for i, idx in enumerate(df.index)}

        xs = []
        for name in [
                'Active', 'FN', 'club_member_status', 'fashion_news_frequency'
        ]:
            x = pd.get_dummies(df[name]).values
            xs.append(torch.from_numpy(x).to(torch.float))

        x = torch.from_numpy(df['age'].values).to(torch.float).view(-1, 1)
        x = x.nan_to_num(nan=x.nanmean())
        xs.append(x / x.max())

        data['customer'].x = torch.cat(xs, dim=-1)

        # Process article data ################################################
        df = pd.read_csv(self.raw_paths[1], index_col='article_id')
        article_map = {idx: i for i, idx in enumerate(df.index)}

        xs = []
        for name in [  # We drop a few columns here that are high cardinality.
                # 'product_code',  # Drop.
                # 'prod_name',  # Drop.
                'product_type_no',
                'product_type_name',
                'product_group_name',
                'graphical_appearance_no',
                'graphical_appearance_name',
                'colour_group_code',
                'colour_group_name',
                'perceived_colour_value_id',
                'perceived_colour_value_name',
                'perceived_colour_master_id',
                'perceived_colour_master_name',
                # 'department_no',  # Drop.
                # 'department_name',  # Drop.
                'index_code',
                'index_name',
                'index_group_no',
                'index_group_name',
                'section_no',
                'section_name',
                'garment_group_no',
                'garment_group_name',
                # 'detail_desc',  # Drop.
        ]:
            x = pd.get_dummies(df[name]).values
            xs.append(torch.from_numpy(x).to(torch.float))

        data['article'].x = torch.cat(xs, dim=-1)

        # Process transaction data ############################################
        df = pd.read_csv(self.raw_paths[2], parse_dates=['t_dat'])

        x1 = pd.get_dummies(df['sales_channel_id']).values
        x1 = torch.from_numpy(x1).to(torch.float)
        x2 = torch.from_numpy(df['price'].values).to(torch.float).view(-1, 1)
        x = torch.cat([x1, x2], dim=-1)

        time = torch.from_numpy(df['t_dat'].values.astype(int))
        time = time // (60 * 60 * 24 * 10**9)  # Convert nanoseconds to days.

        src = torch.tensor([customer_map[idx] for idx in df['customer_id']])
        dst = torch.tensor([article_map[idx] for idx in df['article_id']])

        if self.use_all_tables_as_node_types:
            data['transaction'].x = x
            data['transaction'].time = time

            edge_index = torch.stack([src, torch.arange(len(df))], dim=0)
            data['customer', 'to', 'transaction'].edge_index = edge_index
            edge_index = edge_index.flip([0])
            data['transaction', 'rev_to', 'customer'].edge_index = edge_index

            edge_index = torch.stack([dst, torch.arange(len(df))], dim=0)
            data['article', 'to', 'transaction'].edge_index = edge_index
            edge_index = edge_index.flip([0])
            data['transaction', 'rev_to', 'article'].edge_index = edge_index
        else:
            edge_index = torch.stack([src, dst], dim=0)
            data['customer', 'to', 'article'].edge_index = edge_index
            data['customer', 'to', 'article'].time = time
            data['customer', 'to', 'article'].edge_attr = x

            edge_index = edge_index.flip([0])
            data['article', 'rev_to', 'customer'].edge_index = edge_index
            data['article', 'rev_to', 'customer'].time = time
            data['article', 'rev_to', 'customer'].edge_attr = x

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self.save([data], self.processed_paths[0])
pytorch/torch_geometric/datasets/hydro_net.py000066400000000000000000000254551444262304300221150ustar00rootroot00000000000000import copy
import os
import os.path as osp
from dataclasses import dataclass
from functools import lru_cache
from glob import glob
from pathlib import Path
from typing import Callable, List, Optional, Tuple, Union

import numpy as np
import torch
from torch.utils.data import ConcatDataset, Subset

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)

try:
    from functools import cached_property
except ImportError:  # Python 3.7 support.

    def cached_property(func):
        return property(fget=lru_cache(maxsize=1)(func))


class HydroNet(InMemoryDataset):
    r"""The HydroNet dataest from the
    `"HydroNet: Benchmark Tasks for Preserving Intermolecular Interactions and
    Structural Motifs in Predictive and Generative Models for Molecular Data"
    `_ paper, consisting of 5 million water
    clusters held together by hydrogen bonding networks.  This dataset
    provides atomic coordinates and total energy in kcal/mol for the cluster.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str, optional): Name of the subset of the full dataset to use:
            :obj:`"small"` uses 500k graphs sampled from the :obj:`"medium"`
            dataset, :obj:`"medium"` uses 2.7m graphs with maximum size of 75
            nodes.
            Mutually exclusive option with the clusters argument.
            (default :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        num_workers (int): Number of multiprocessing workers to use for
            pre-processing the dataset. (default :obj:`8`)
        clusters (int or List[int], optional): Select a subset of clusters
            from the full dataset. If set to :obj:`None`, will select all.
            (default :obj:`None`)
        use_processed (bool): Option to use a pre-processed version of the
            original :obj:`xyz` dataset. (default: :obj:`True`)
    """
    def __init__(
        self,
        root: str,
        name: Optional[str] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        num_workers: int = 8,
        clusters: Optional[Union[int, List[int]]] = None,
        use_processed: bool = True,
    ):
        self.name = name
        self.num_workers = num_workers
        self.use_processed = use_processed

        super().__init__(root, transform, pre_transform)
        self.select_clusters(clusters)

    @property
    def raw_file_names(self) -> List[str]:
        return [f'W{c}_geoms_all.zip' for c in range(3, 31)]

    @property
    def processed_file_names(self) -> List[str]:
        return [f'W{c}_geoms_all.npz' for c in range(3, 31)]

    def download(self):
        token_file = Path(osp.join(self.raw_dir, 'use_processed'))
        if self.use_processed and token_file.exists():
            return

        file = RemoteFile.hydronet_splits()
        file.unpack_to(self.raw_dir)

        if self.use_processed:
            file = RemoteFile.processed_dataset()
            file.unpack_to(self.raw_dir)
            token_file.touch()
            return

        file = RemoteFile.raw_dataset()
        file.unpack_to(self.raw_dir)
        folder_name, _ = osp.splitext(file.name)
        files = glob(osp.join(self.raw_dir, folder_name, '*.zip'))

        for f in files:
            dst = osp.join(self.raw_dir, osp.basename(f))
            os.rename(f, dst)

        os.rmdir(osp.join(self.raw_dir, folder_name))

    def process(self):
        if self.use_processed:
            return self._unpack_processed()

        from tqdm.contrib.concurrent import process_map

        self._partitions = process_map(
            self._create_partitions,
            self.raw_paths,
            max_workers=self.num_workers,
            position=0,
            leave=True,
        )

    def _unpack_processed(self):
        files = glob(osp.join(self.raw_dir, '*.npz'))
        for f in files:
            dst = osp.join(self.processed_dir, osp.basename(f))
            os.rename(f, dst)

    def _create_partitions(self, file) -> 'Partition':
        name = osp.basename(file)
        name, _ = osp.splitext(name)
        return Partition(self.root, name, self.transform, self.pre_transform)

    def select_clusters(
        self,
        clusters: Optional[Union[int, List[int]]],
    ) -> 'Optional[List[Partition]]':
        if self.name is not None:
            clusters = self._validate_name(clusters)

        self._partitions = [self._create_partitions(f) for f in self.raw_paths]

        if clusters is None:
            return

        clusters = [clusters] if isinstance(clusters, int) else clusters

        def is_valid_cluster(x):
            return isinstance(x, int) and x >= 3 and x <= 30

        if not all([is_valid_cluster(x) for x in clusters]):
            raise ValueError(
                "Selected clusters must be an integer in the range [3, 30]")

        self._partitions = [self._partitions[c - 3] for c in clusters]

    def _validate_name(self, clusters) -> List[int]:
        if clusters is not None:
            raise ValueError("'name' and 'clusters' are mutually exclusive")

        if self.name not in ['small', 'medium']:
            raise ValueError(f"Invalid subset name '{self.name}'. "
                             f"Must be either 'small' or 'medium'")

        return range(3, 26)

    @cached_property
    def _dataset(self) -> ConcatDataset:
        dataset = ConcatDataset(self._partitions)

        if self.name == "small":
            dataset = self._load_small_split(dataset)

        return dataset

    def _load_small_split(self, dataset: ConcatDataset) -> Subset:
        split_file = osp.join(self.processed_dir, 'split_00_small.npz')
        with np.load(split_file) as split:
            train_idx = split['train_idx']
            val_idx = split['val_idx']
            all_idx = np.concatenate([train_idx, val_idx])
            dataset = Subset(dataset, all_idx)
        return dataset

    def len(self) -> int:
        return len(self._dataset)

    def get(self, idx: int) -> Data:
        return self._dataset[idx]


def get_num_clusters(filepath) -> int:
    name = osp.basename(filepath)
    return int(name[1:name.find('_')])


def read_energy(file: str, chunk_size: int) -> np.ndarray:
    import pandas as pd

    def skipatoms(i: int):
        return (i - 1) % chunk_size != 0

    if chunk_size - 2 == 11 * 3:
        # Manually handle bad lines in W11
        df = pd.read_table(file, header=None, dtype="string",
                           skiprows=skipatoms)
        df = df[0].str.split().str[-1].astype(np.float32)
    else:
        df = pd.read_table(file, sep=r'\s+', names=["label", "energy"],
                           dtype=np.float32, skiprows=skipatoms,
                           usecols=['energy'], memory_map=True)

    return df.to_numpy().squeeze()


def read_atoms(file: str, chunk_size: int) -> Tuple[np.ndarray, np.ndarray]:
    import pandas as pd

    def skipheaders(i: int):
        return i % chunk_size == 0 or (i - 1) % chunk_size == 0

    dtypes = {
        'atom': 'string',
        'x': np.float16,
        'y': np.float16,
        'z': np.float16
    }
    df = pd.read_table(file, sep=r'\s+', names=list(dtypes.keys()),
                       dtype=dtypes, skiprows=skipheaders, memory_map=True)

    z = np.ones(len(df), dtype=np.int8)
    z[(df.atom == 'O').to_numpy(dtype=np.bool_)] = 8
    pos = df.iloc[:, 1:4].to_numpy()
    num_nodes = (chunk_size - 2)
    num_graphs = z.shape[0] // num_nodes
    z.shape = (num_graphs, num_nodes)
    pos.shape = (num_graphs, num_nodes, 3)
    return (z, pos)


@dataclass
class RemoteFile:
    url: str
    name: str

    def unpack_to(self, dest_folder: str):
        file = download_url(self.url, dest_folder, filename=self.name)
        extract_zip(file, dest_folder)
        os.unlink(file)

    @staticmethod
    def raw_dataset() -> 'RemoteFile':
        return RemoteFile(
            url='https://figshare.com/ndownloader/files/38063847',
            name='W3-W30_all_geoms_TTM2.1-F.zip')

    @staticmethod
    def processed_dataset() -> 'RemoteFile':
        return RemoteFile(
            url='https://figshare.com/ndownloader/files/38075781',
            name='W3-W30_pyg_processed.zip')

    @staticmethod
    def hydronet_splits() -> 'RemoteFile':
        return RemoteFile(
            url="https://figshare.com/ndownloader/files/38075904",
            name="hydronet_splits.zip")


class Partition(InMemoryDataset):
    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name
        self.num_clusters = get_num_clusters(name)
        super().__init__(root, transform, pre_transform, pre_filter=None,
                         log=False)
        self.is_loaded = False

    @property
    def raw_file_names(self) -> List[str]:
        return [self.name + ".zip"]

    @property
    def processed_file_names(self) -> List[str]:
        return [self.name + '.npz']

    def process(self):
        num_nodes = self.num_clusters * 3
        chunk_size = num_nodes + 2
        z, pos = read_atoms(self.raw_paths[0], chunk_size)
        y = read_energy(self.raw_paths[0], chunk_size)
        np.savez(self.processed_paths[0], z=z, pos=pos, y=y,
                 num_graphs=z.shape[0])

    def load(self):
        if self.is_loaded:
            return

        with np.load(self.processed_paths[0]) as npzfile:
            self.z = npzfile['z']
            self.pos = npzfile['pos']
            self.y = npzfile['y']
            num_graphs = int(npzfile['num_graphs'])

        self._data_list = num_graphs * [None]
        self.is_loaded = True

    @cached_property
    def num_graphs(self) -> int:
        with np.load(self.processed_paths[0]) as npzfile:
            return int(npzfile['num_graphs'])

    def len(self) -> int:
        return self.num_graphs

    def get(self, idx: int) -> Data:
        self.load()
        if self._data_list[idx] is not None:
            return copy.copy(self._data_list[idx])

        data = Data(
            z=torch.from_numpy(self.z[idx, :]),
            pos=torch.from_numpy(self.pos[idx, :, :]),
            y=torch.tensor(self.y[idx]),
        )

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self._data_list[idx] = copy.copy(data)
        return data
pytorch/torch_geometric/datasets/icews.py000066400000000000000000000105371444262304300212270ustar00rootroot00000000000000from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.io import read_txt_array


class EventDataset(InMemoryDataset):
    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform, pre_filter)

    @property
    def num_nodes(self) -> int:
        raise NotImplementedError

    @property
    def num_rels(self) -> int:
        raise NotImplementedError

    def process_events(self) -> int:
        raise NotImplementedError

    def process(self) -> List[Data]:
        events = self.process_events()
        events = events - events.min(dim=0, keepdim=True)[0]

        data_list = []
        for (sub, rel, obj, t) in events.tolist():
            data = Data(sub=sub, rel=rel, obj=obj, t=t)
            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)
            data_list.append(data)

        return data_list


class ICEWS18(EventDataset):
    r"""The Integrated Crisis Early Warning System (ICEWS) dataset used in
    the, *e.g.*, `"Recurrent Event Network for Reasoning over Temporal
    Knowledge Graphs" `_ paper, consisting of
    events collected from 1/1/2018 to 10/31/2018 (24 hours time granularity).

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset. (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'https://github.com/INK-USC/RE-Net/raw/master/data/ICEWS18'
    splits = [0, 373018, 419013, 468558]  # Train/Val/Test splits.

    def __init__(
        self,
        root: str,
        split: str = 'train',
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        assert split in ['train', 'val', 'test']
        super().__init__(root, transform, pre_transform, pre_filter)
        idx = self.processed_file_names.index(f'{split}.pt')
        self.data, self.slices = torch.load(self.processed_paths[idx])

    @property
    def num_nodes(self) -> int:
        return 23033

    @property
    def num_rels(self) -> int:
        return 256

    @property
    def raw_file_names(self) -> List[str]:
        return [f'{name}.txt' for name in ['train', 'valid', 'test']]

    @property
    def processed_file_names(self) -> List[str]:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process_events(self) -> torch.Tensor:
        events = []
        for path in self.raw_paths:
            data = read_txt_array(path, sep='\t', end=4, dtype=torch.long)
            data[:, 3] = data[:, 3] // 24
            events += [data]
        return torch.cat(events, dim=0)

    def process(self):
        s = self.splits
        data_list = super().process()
        torch.save(self.collate(data_list[s[0]:s[1]]), self.processed_paths[0])
        torch.save(self.collate(data_list[s[1]:s[2]]), self.processed_paths[1])
        torch.save(self.collate(data_list[s[2]:s[3]]), self.processed_paths[2])
pytorch/torch_geometric/datasets/igmc_dataset.py000066400000000000000000000104451444262304300225370ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch
from torch import Tensor

from torch_geometric.data import HeteroData, InMemoryDataset, download_url


class IGMCDataset(InMemoryDataset):
    r"""The user-item heterogeneous rating datasets :obj:`"Douban"`,
    :obj:`"Flixster"` and :obj:`"Yahoo-Music"` from the `"Inductive Matrix
    Completion Based on Graph Neural Networks"
    `_ paper.

    Nodes represent users and items.
    Edges and features between users and items represent a (training) rating of
    the item given by the user.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Douban"`,
            :obj:`"Flixster"`, :obj:`"Yahoo-Music"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """
    url = 'https://github.com/muhanzhang/IGMC/raw/master/raw_data'

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name.lower().replace('-', '_')
        assert self.name in ['flixster', 'douban', 'yahoo_music']

        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0], data_cls=HeteroData)

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return 'training_test_dataset.mat'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = f'{self.url}/{self.name}/training_test_dataset.mat'
        download_url(path, self.raw_dir)

    @staticmethod
    def load_matlab_file(path_file: str, name: str) -> Tensor:
        import h5py
        import numpy as np

        db = h5py.File(path_file, 'r')
        out = torch.from_numpy(np.asarray(db[name])).to(torch.float).t()
        db.close()

        return out

    def process(self):
        data = HeteroData()

        M = self.load_matlab_file(self.raw_paths[0], 'M')

        if self.name == 'flixster':
            user_x = self.load_matlab_file(self.raw_paths[0], 'W_users')
            item_x = self.load_matlab_file(self.raw_paths[0], 'W_movies')
        elif self.name == 'douban':
            user_x = self.load_matlab_file(self.raw_paths[0], 'W_users')
            item_x = torch.eye(M.size(1))
        elif self.name == 'yahoo_music':
            user_x = torch.eye(M.size(0))
            item_x = self.load_matlab_file(self.raw_paths[0], 'W_tracks')

        data['user'].x = user_x
        data['item'].x = item_x

        train_mask = self.load_matlab_file(self.raw_paths[0], 'Otraining')
        train_mask = train_mask.to(torch.bool)

        edge_index = train_mask.nonzero().t()
        rating = M[edge_index[0], edge_index[1]]

        data['user', 'rates', 'item'].edge_index = edge_index
        data['user', 'rates', 'item'].rating = rating

        data['item', 'rated_by', 'user'].edge_index = edge_index.flip([0])
        data['item', 'rated_by', 'user'].rating = rating

        test_mask = self.load_matlab_file(self.raw_paths[0], 'Otest')
        test_mask = test_mask.to(torch.bool)

        edge_label_index = test_mask.nonzero().t()
        edge_label = M[edge_label_index[0], edge_label_index[1]]

        data['user', 'rates', 'item'].edge_label_index = edge_label_index
        data['user', 'rates', 'item'].edge_label = edge_label

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self.save([data], self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(name={self.name})'
pytorch/torch_geometric/datasets/imdb.py000066400000000000000000000076401444262304300210310ustar00rootroot00000000000000import os
import os.path as osp
from itertools import product
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class IMDB(InMemoryDataset):
    r"""A subset of the Internet Movie Database (IMDB), as collected in the
    `"MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph
    Embedding" `_ paper.
    IMDB is a heterogeneous graph containing three types of entities - movies
    (4,278 nodes), actors (5,257 nodes), and directors (2,081 nodes).
    The movies are divided into three classes (action, comedy, drama) according
    to their genre.
    Movie features correspond to elements of a bag-of-words representation of
    its plot keywords.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://www.dropbox.com/s/g0btk9ctr1es39x/IMDB_processed.zip?dl=1'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'adjM.npz', 'features_0.npz', 'features_1.npz', 'features_2.npz',
            'labels.npy', 'train_val_test_idx.npz'
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        data = HeteroData()

        node_types = ['movie', 'director', 'actor']
        for i, node_type in enumerate(node_types):
            x = sp.load_npz(osp.join(self.raw_dir, f'features_{i}.npz'))
            data[node_type].x = torch.from_numpy(x.todense()).to(torch.float)

        y = np.load(osp.join(self.raw_dir, 'labels.npy'))
        data['movie'].y = torch.from_numpy(y).to(torch.long)

        split = np.load(osp.join(self.raw_dir, 'train_val_test_idx.npz'))
        for name in ['train', 'val', 'test']:
            idx = split[f'{name}_idx']
            idx = torch.from_numpy(idx).to(torch.long)
            mask = torch.zeros(data['movie'].num_nodes, dtype=torch.bool)
            mask[idx] = True
            data['movie'][f'{name}_mask'] = mask

        s = {}
        N_m = data['movie'].num_nodes
        N_d = data['director'].num_nodes
        N_a = data['actor'].num_nodes
        s['movie'] = (0, N_m)
        s['director'] = (N_m, N_m + N_d)
        s['actor'] = (N_m + N_d, N_m + N_d + N_a)

        A = sp.load_npz(osp.join(self.raw_dir, 'adjM.npz'))
        for src, dst in product(node_types, node_types):
            A_sub = A[s[src][0]:s[src][1], s[dst][0]:s[dst][1]].tocoo()
            if A_sub.nnz > 0:
                row = torch.from_numpy(A_sub.row).to(torch.long)
                col = torch.from_numpy(A_sub.col).to(torch.long)
                data[src, dst].edge_index = torch.stack([row, col], dim=0)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/datasets/infection_dataset.py000066400000000000000000000160011444262304300235700ustar00rootroot00000000000000from typing import Any, Callable, Dict, List, Optional, Union

import torch

from torch_geometric.data import InMemoryDataset
from torch_geometric.datasets.graph_generator import GraphGenerator
from torch_geometric.explain import Explanation
from torch_geometric.utils import k_hop_subgraph


class InfectionDataset(InMemoryDataset):
    r"""Generates a synthetic infection dataset for evaluating explainabilty
    algorithms, as described in the `"Explainability Techniques for Graph
    Convolutional Networks" `__ paper.
    The :class:`~torch_geometric.datasets.InfectionDataset` creates synthetic
    graphs coming from a
    :class:`~torch_geometric.datasets.graph_generator.GraphGenerator` with
    :obj:`num_infected` randomly assigned infected nodes.
    The dataset describes a node classification task of predicting the length
    of the shortest path to infected nodes, with corresponding ground-truth
    edge-level masks.

    For example, to generate a random Erdos-Renyi (ER) infection graph
    with :obj:`500` nodes and :obj:`0.004` edge probability, write:

    .. code-block:: python

        from torch_geometric.datasets import InfectionDataset
        from torch_geometric.datasets.graph_generator import ERGraph

        dataset = InfectionDataset(
            graph_generator=ERGraph(num_nodes=500, edge_prob=0.004),
            num_infected_nodes=50,
            max_path_length=3,
        )

    Args:
        graph_generator (GraphGenerator or str): The graph generator to be
            used, *e.g.*,
            :class:`torch.geometric.datasets.graph_generator.BAGraph`
            (or any string that automatically resolves to it).
        num_infected_nodes (int or List[int]): The number of randomly
            selected infected nodes in the graph.
            If given as a list, will select a different number of infected
            nodes for different graphs.
        max_path_length (int, List[int]): The maximum shortest path length to
            determine whether a node will be infected.
            If given as a list, will apply different shortest path lengths for
            different graphs. (default: :obj:`5`)
        num_graphs (int, optional): The number of graphs to generate.
            The number of graphs will be automatically determined by
            :obj:`len(num_infected_nodes)` or :obj:`len(max_path_length)` in
            case either of them is given as a list, and should only be set in
            case one wants to create multiple graphs while
            :obj:`num_infected_nodes` and :obj:`max_path_length` are given as
            an integer. (default: :obj:`None`)
        graph_generator_kwargs (Dict[str, Any], optional): Arguments passed to
            the respective graph generator module in case it gets automatically
            resolved. (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        graph_generator: Union[GraphGenerator, str],
        num_infected_nodes: Union[int, List[int]],
        max_path_length: Union[int, List[int]],
        num_graphs: Optional[int] = None,
        graph_generator_kwargs: Optional[Dict[str, Any]] = None,
        transform: Optional[Callable] = None,
    ):
        super().__init__(root=None, transform=transform)

        assert isinstance(num_infected_nodes, (int, list))
        assert isinstance(max_path_length, (int, list))

        if (num_graphs is None and isinstance(num_infected_nodes, int)
                and isinstance(max_path_length, int)):
            num_graphs = 1

        if num_graphs is None and isinstance(num_infected_nodes, list):
            num_graphs = len(num_infected_nodes)

        if num_graphs is None and isinstance(max_path_length, list):
            num_graphs = len(max_path_length)

        self.graph_generator = GraphGenerator.resolve(
            graph_generator,
            **(graph_generator_kwargs or {}),
        )
        self.num_infected_nodes = num_infected_nodes
        self.max_path_length = max_path_length
        self.num_graphs = num_graphs

        if isinstance(num_infected_nodes, int):
            num_infected_nodes = [num_infected_nodes] * num_graphs

        if isinstance(max_path_length, int):
            max_path_length = [max_path_length] * num_graphs

        if len(num_infected_nodes) != num_graphs:
            raise ValueError(f"The length of 'num_infected_nodes' "
                             f"(got {len(num_infected_nodes)} does not match "
                             f"with the number of graphs (got {num_graphs})")

        if len(max_path_length) != num_graphs:
            raise ValueError(f"The length of 'max_path_length' "
                             f"(got {len(max_path_length)} does not match "
                             f"with the number of graphs (got {num_graphs})")

        if any(num_infected_nodes) <= 0:
            raise ValueError(f"'num_infected_nodes' needs to be positive "
                             f"(got {min(num_infected_nodes)})")

        if any(max_path_length) <= 0:
            raise ValueError(f"'max_path_length' needs to be positive "
                             f"(got {min(max_path_length)})")

        data_list: List[Explanation] = []
        for N, L in zip(num_infected_nodes, max_path_length):
            data_list.append(self.get_graph(N, L))

        self.data, self.slices = self.collate(data_list)

    def get_graph(self, num_infected_nodes: int,
                  max_path_length: int) -> Explanation:
        data = self.graph_generator()

        perm = torch.randperm(data.num_nodes)
        x = torch.zeros((data.num_nodes, 2))
        x[perm[:num_infected_nodes], 1] = 1  # Infected
        x[perm[num_infected_nodes:], 0] = 1  # Healthy

        y = torch.empty(data.num_nodes, dtype=torch.long)
        y.fill_(max_path_length + 1)
        y[perm[:num_infected_nodes]] = 0  # Infected nodes have label `0`.

        edge_mask = torch.zeros(data.num_edges, dtype=torch.bool)
        for num_hops in range(1, max_path_length + 1):
            sub_node_index, _, _, sub_edge_mask = k_hop_subgraph(
                perm[:num_infected_nodes], num_hops, data.edge_index,
                num_nodes=data.num_nodes, flow='target_to_source',
                directed=True)

            value = torch.full_like(sub_node_index, fill_value=num_hops)
            y[sub_node_index] = torch.min(y[sub_node_index], value)
            edge_mask |= sub_edge_mask

        return Explanation(
            x=x,
            edge_index=data.edge_index,
            y=y,
            edge_mask=edge_mask.to(torch.float),
        )

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'graph_generator={self.graph_generator}, '
                f'num_infected_nodes={self.num_infected_nodes}, '
                f'max_path_length={self.max_path_length})')
pytorch/torch_geometric/datasets/jodie.py000066400000000000000000000065571444262304300212160ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import InMemoryDataset, TemporalData, download_url


class JODIEDataset(InMemoryDataset):
    r"""The temporal graph datasets
    from the `"JODIE: Predicting Dynamic Embedding
    Trajectory in Temporal Interaction Networks"
    `_ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Reddit"`,
            :obj:`"Wikipedia"`, :obj:`"MOOC"`, and :obj:`"LastFM"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Reddit
          - 6,509
          - 25,470
          - 172
          - 1
        * - Wikipedia
          - 9,227
          - 157,474
          - 172
          - 2
        * - MOOC
          - 7,144
          - 411,749
          - 4
          - 2
        * - LastFM
          - 1,980
          - 1,293,103
          - 2
          - 1
    """
    url = 'http://snap.stanford.edu/jodie/{}.csv'
    names = ['reddit', 'wikipedia', 'mooc', 'lastfm']

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in self.names

        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.csv'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url.format(self.name), self.raw_dir)

    def process(self):
        import pandas as pd

        df = pd.read_csv(self.raw_paths[0], skiprows=1, header=None)

        src = torch.from_numpy(df.iloc[:, 0].values).to(torch.long)
        dst = torch.from_numpy(df.iloc[:, 1].values).to(torch.long)
        dst += int(src.max()) + 1
        t = torch.from_numpy(df.iloc[:, 2].values).to(torch.long)
        y = torch.from_numpy(df.iloc[:, 3].values).to(torch.long)
        msg = torch.from_numpy(df.iloc[:, 4:].values).to(torch.float)

        data = TemporalData(src=src, dst=dst, t=t, msg=msg, y=y)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.capitalize()}()'
pytorch/torch_geometric/datasets/karate.py000066400000000000000000000065641444262304300213710ustar00rootroot00000000000000from typing import Callable, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset


class KarateClub(InMemoryDataset):
    r"""Zachary's karate club network from the `"An Information Flow Model for
    Conflict and Fission in Small Groups"
    `_ paper, containing
    34 nodes, connected by 156 (undirected and unweighted) edges.
    Every node is labeled by one of four classes obtained via modularity-based
    clustering, following the `"Semi-supervised Classification with Graph
    Convolutional Networks" `_ paper.
    Training is based on a single labeled example per class, *i.e.* a total
    number of 4 labeled nodes.

    Args:
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 34
          - 156
          - 34
          - 4
    """
    def __init__(self, transform: Optional[Callable] = None):
        super().__init__('.', transform)

        row = [
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1,
            1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4,
            5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 8, 9, 9, 10, 10,
            10, 11, 12, 12, 13, 13, 13, 13, 13, 14, 14, 15, 15, 16, 16, 17, 17,
            18, 18, 19, 19, 19, 20, 20, 21, 21, 22, 22, 23, 23, 23, 23, 23, 24,
            24, 24, 25, 25, 25, 26, 26, 27, 27, 27, 27, 28, 28, 28, 29, 29, 29,
            29, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32,
            32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33,
            33, 33, 33, 33, 33, 33
        ]
        col = [
            1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 17, 19, 21, 31, 0, 2, 3, 7,
            13, 17, 19, 21, 30, 0, 1, 3, 7, 8, 9, 13, 27, 28, 32, 0, 1, 2, 7,
            12, 13, 0, 6, 10, 0, 6, 10, 16, 0, 4, 5, 16, 0, 1, 2, 3, 0, 2, 30,
            32, 33, 2, 33, 0, 4, 5, 0, 0, 3, 0, 1, 2, 3, 33, 32, 33, 32, 33, 5,
            6, 0, 1, 32, 33, 0, 1, 33, 32, 33, 0, 1, 32, 33, 25, 27, 29, 32,
            33, 25, 27, 31, 23, 24, 31, 29, 33, 2, 23, 24, 33, 2, 31, 33, 23,
            26, 32, 33, 1, 8, 32, 33, 0, 24, 25, 28, 32, 33, 2, 8, 14, 15, 18,
            20, 22, 23, 29, 30, 31, 33, 8, 9, 13, 14, 15, 18, 19, 20, 22, 23,
            26, 27, 28, 29, 30, 31, 32
        ]
        edge_index = torch.tensor([row, col])

        y = torch.tensor([  # Create communities.
            1, 1, 1, 1, 3, 3, 3, 1, 0, 1, 3, 1, 1, 1, 0, 0, 3, 1, 0, 1, 0, 1,
            0, 0, 2, 2, 0, 0, 2, 0, 0, 2, 0, 0
        ])

        x = torch.eye(y.size(0), dtype=torch.float)

        # Select a single training node for each community
        # (we just use the first one).
        train_mask = torch.zeros(y.size(0), dtype=torch.bool)
        for i in range(int(y.max()) + 1):
            train_mask[(y == i).nonzero(as_tuple=False)[0]] = True

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask)

        self.data, self.slices = self.collate([data])
pytorch/torch_geometric/datasets/last_fm.py000066400000000000000000000103751444262304300215420ustar00rootroot00000000000000import os
import os.path as osp
from itertools import product
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class LastFM(InMemoryDataset):
    r"""A subset of the last.fm music website keeping track of users' listining
    information from various sources, as collected in the
    `"MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph
    Embedding" `_ paper.
    last.fm is a heterogeneous graph containing three types of entities - users
    (1,892 nodes), artists (17,632 nodes), and artist tags (1,088 nodes).
    This dataset can be used for link prediction, and no labels or features are
    provided.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://www.dropbox.com/s/jvlbs09pz6zwcka/LastFM_processed.zip?dl=1'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'adjM.npz', 'node_types.npy', 'train_val_test_neg_user_artist.npz',
            'train_val_test_pos_user_artist.npz'
        ]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        data = HeteroData()

        node_type_idx = np.load(osp.join(self.raw_dir, 'node_types.npy'))
        node_type_idx = torch.from_numpy(node_type_idx).to(torch.long)

        node_types = ['user', 'artist', 'tag']
        for i, node_type in enumerate(node_types):
            data[node_type].num_nodes = int((node_type_idx == i).sum())

        pos_split = np.load(
            osp.join(self.raw_dir, 'train_val_test_pos_user_artist.npz'))
        neg_split = np.load(
            osp.join(self.raw_dir, 'train_val_test_neg_user_artist.npz'))

        for name in ['train', 'val', 'test']:
            if name != 'train':
                edge_index = pos_split[f'{name}_pos_user_artist']
                edge_index = torch.from_numpy(edge_index)
                edge_index = edge_index.t().to(torch.long).contiguous()
                data['user', 'artist'][f'{name}_pos_edge_index'] = edge_index

            edge_index = neg_split[f'{name}_neg_user_artist']
            edge_index = torch.from_numpy(edge_index)
            edge_index = edge_index.t().to(torch.long).contiguous()
            data['user', 'artist'][f'{name}_neg_edge_index'] = edge_index

        s = {}
        N_u = data['user'].num_nodes
        N_a = data['artist'].num_nodes
        N_t = data['tag'].num_nodes
        s['user'] = (0, N_u)
        s['artist'] = (N_u, N_u + N_a)
        s['tag'] = (N_u + N_a, N_u + N_a + N_t)

        A = sp.load_npz(osp.join(self.raw_dir, 'adjM.npz'))
        for src, dst in product(node_types, node_types):
            A_sub = A[s[src][0]:s[src][1], s[dst][0]:s[dst][1]].tocoo()
            if A_sub.nnz > 0:
                row = torch.from_numpy(A_sub.row).to(torch.long)
                col = torch.from_numpy(A_sub.col).to(torch.long)
                data[src, dst].edge_index = torch.stack([row, col], dim=0)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/datasets/lastfm_asia.py000066400000000000000000000043531444262304300223770ustar00rootroot00000000000000from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class LastFMAsia(InMemoryDataset):
    r"""The LastFM Asia Network dataset introduced in the `"Characteristic
    Functions on Graphs: Birds of a Feather, from Statistical Descriptors to
    Parametric Models" `_ paper.
    Nodes represent LastFM users from Asia and edges are friendships.
    It contains 7,624 nodes, 55,612 edges, 128 node features and 18 classes.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://graphmining.ai/datasets/ptg/lastfm_asia.npz'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'lastfm_asia.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        x = torch.from_numpy(data['features']).to(torch.float)
        y = torch.from_numpy(data['target']).to(torch.long)
        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/linkx_dataset.py000066400000000000000000000146661444262304300227560ustar00rootroot00000000000000import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import one_hot


class LINKXDataset(InMemoryDataset):
    r"""A variety of non-homophilous graph datasets from the `"Large Scale
    Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple
    Methods" `_ paper.

    .. note::
        Some of the datasets provided in :class:`LINKXDataset` are from other
        sources, but have been updated with new features and/or labels.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"penn94"`, :obj:`"reed98"`,
            :obj:`"amherst41"`, :obj:`"cornell5"`, :obj:`"johnshopkins55"`,
            :obj:`"genius"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    github_url = ('https://github.com/CUAI/Non-Homophily-Large-Scale/'
                  'raw/master/data')
    gdrive_url = 'https://drive.google.com/uc?confirm=t&'

    facebook_datasets = [
        'penn94', 'reed98', 'amherst41', 'cornell5', 'johnshopkins55'
    ]

    datasets = {
        'penn94': {
            'data.mat': f'{github_url}/facebook100/Penn94.mat'
        },
        'reed98': {
            'data.mat': f'{github_url}/facebook100/Reed98.mat'
        },
        'amherst41': {
            'data.mat': f'{github_url}/facebook100/Amherst41.mat',
        },
        'cornell5': {
            'data.mat': f'{github_url}/facebook100/Cornell5.mat'
        },
        'johnshopkins55': {
            'data.mat': f'{github_url}/facebook100/Johns%20Hopkins55.mat'
        },
        'genius': {
            'data.mat': f'{github_url}/genius.mat'
        },
        'wiki': {
            'wiki_views2M.pt':
            f'{gdrive_url}id=1p5DlVHrnFgYm3VsNIzahSsvCD424AyvP',
            'wiki_edges2M.pt':
            f'{gdrive_url}id=14X7FlkjrlUgmnsYtPwdh-gGuFla4yb5u',
            'wiki_features2M.pt':
            f'{gdrive_url}id=1ySNspxbK-snNoAZM7oxiWGvOnTRdSyEK'
        }
    }

    splits = {
        'penn94': f'{github_url}/splits/fb100-Penn94-splits.npy',
    }

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        assert self.name in self.datasets.keys()
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        names = list(self.datasets[self.name].keys())
        if self.name in self.splits:
            names += [self.splits[self.name].split('/')[-1]]
        return names

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for filename, path in self.datasets[self.name].items():
            download_url(path, self.raw_dir, filename=filename)
        if self.name in self.splits:
            download_url(self.splits[self.name], self.raw_dir)

    def _process_wiki(self):

        paths = {x.split('/')[-1]: x for x in self.raw_paths}
        x = torch.load(paths['wiki_features2M.pt'])
        edge_index = torch.load(paths['wiki_edges2M.pt']).t().contiguous()
        y = torch.load(paths['wiki_views2M.pt'])

        return Data(x=x, edge_index=edge_index, y=y)

    def _process_facebook(self):
        from scipy.io import loadmat

        mat = loadmat(self.raw_paths[0])

        A = mat['A'].tocsr().tocoo()
        row = torch.from_numpy(A.row).to(torch.long)
        col = torch.from_numpy(A.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        metadata = torch.from_numpy(mat['local_info'].astype('int64'))

        xs = []
        y = metadata[:, 1] - 1  # gender label, -1 means unlabeled
        x = torch.cat([metadata[:, :1], metadata[:, 2:]], dim=-1)
        for i in range(x.size(1)):
            _, out = x[:, i].unique(return_inverse=True)
            xs.append(one_hot(out))
        x = torch.cat(xs, dim=-1)

        data = Data(x=x, edge_index=edge_index, y=y)

        if self.name in self.splits:
            splits = np.load(self.raw_paths[1], allow_pickle=True)
            sizes = (data.num_nodes, len(splits))
            data.train_mask = torch.zeros(sizes, dtype=torch.bool)
            data.val_mask = torch.zeros(sizes, dtype=torch.bool)
            data.test_mask = torch.zeros(sizes, dtype=torch.bool)

            for i, split in enumerate(splits):
                data.train_mask[:, i][torch.tensor(split['train'])] = True
                data.val_mask[:, i][torch.tensor(split['valid'])] = True
                data.test_mask[:, i][torch.tensor(split['test'])] = True

        return data

    def _process_genius(self):
        from scipy.io import loadmat

        mat = loadmat(self.raw_paths[0])
        edge_index = torch.from_numpy(mat['edge_index']).to(torch.long)
        x = torch.from_numpy(mat['node_feat']).to(torch.float)
        y = torch.from_numpy(mat['label']).squeeze().to(torch.long)

        return Data(x=x, edge_index=edge_index, y=y)

    def process(self):
        if self.name in self.facebook_datasets:
            data = self._process_facebook()
        elif self.name == 'genius':
            data = self._process_genius()
        elif self.name == 'wiki':
            data = self._process_wiki()
        else:
            raise NotImplementedError(
                f"chosen dataset '{self.name}' is not implemented")

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name.capitalize()}({len(self)})'
pytorch/torch_geometric/datasets/lrgb.py000066400000000000000000000264471444262304300210520ustar00rootroot00000000000000import os
import os.path as osp
import pickle
import shutil
from typing import Callable, List, Optional

import torch
from tqdm import tqdm

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class LRGBDataset(InMemoryDataset):
    r"""The `"Long Range Graph Benchmark (LRGB)"
    `_
    datasets which is a collection of 5 graph learning datasets with tasks
    that are based on long-range dependencies in graphs. See the original
    `source code `_ for more details
    on the individual datasets.

    +------------------------+-------------------+----------------------+
    | Dataset                | Domain            | Task                 |
    +========================+===================+======================+
    | :obj:`PascalVOC-SP`    | Computer Vision   | Node Classification  |
    +------------------------+-------------------+----------------------+
    | :obj:`COCO-SP`         | Computer Vision   | Node Classification  |
    +------------------------+-------------------+----------------------+
    | :obj:`PCQM-Contact`    | Quantum Chemistry | Link Prediction      |
    +------------------------+-------------------+----------------------+
    | :obj:`Peptides-func`   | Chemistry         | Graph Classification |
    +------------------------+-------------------+----------------------+
    | :obj:`Peptides-struct` | Chemistry         | Graph Regression     |
    +------------------------+-------------------+----------------------+

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (one of :obj:`"PascalVOC-SP"`,
            :obj:`"COCO-SP"`, :obj:`"PCQM-Contact"`, :obj:`"Peptides-func"`,
            :obj:`"Peptides-struct"`)
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 15 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #classes
        * - PascalVOC-SP
          - 11,355
          - ~479.40
          - ~2,710.48
          - 21
        * - COCO-SP
          - 123,286
          - ~476.88
          - ~2,693.67
          - 81
        * - PCQM-Contact
          - 529,434
          - ~30.14
          - ~61.09
          - 1
        * - Peptides-func
          - 15,535
          - ~150.94
          - ~307.30
          - 10
        * - Peptides-struct
          - 15,535
          - ~150.94
          - ~307.30
          - 11
    """

    names = [
        'pascalvoc-sp', 'coco-sp', 'pcqm-contact', 'peptides-func',
        'peptides-struct'
    ]

    urls = {
        'pascalvoc-sp':
        'https://www.dropbox.com/s/8x722ai272wqwl4/pascalvocsp.zip?dl=1',
        'coco-sp':
        'https://www.dropbox.com/s/r6ihg1f4pmyjjy0/cocosp.zip?dl=1',
        'pcqm-contact':
        'https://www.dropbox.com/s/qdag867u6h6i60y/pcqmcontact.zip?dl=1',
        'peptides-func':
        'https://www.dropbox.com/s/ycsq37q8sxs1ou8/peptidesfunc.zip?dl=1',
        'peptides-struct':
        'https://www.dropbox.com/s/zgv4z8fcpmknhs8/peptidesstruct.zip?dl=1'
    }

    dwnld_file_name = {
        'pascalvoc-sp': 'voc_superpixels_edge_wt_region_boundary',
        'coco-sp': 'coco_superpixels_edge_wt_region_boundary',
        'pcqm-contact': 'pcqmcontact',
        'peptides-func': 'peptidesfunc',
        'peptides-struct': 'peptidesstruct'
    }

    def __init__(
        self,
        root: str,
        name: str,
        split: str = "train",
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in self.names
        assert split in ['train', 'val', 'test']

        super().__init__(root, transform, pre_transform, pre_filter)
        path = osp.join(self.processed_dir, f'{split}.pt')
        self.data, self.slices = torch.load(path)

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        if self.name.split('-')[1] == 'sp':
            return ['train.pickle', 'val.pickle', 'test.pickle']
        else:
            return ['train.pt', 'val.pt', 'test.pt']

    @property
    def processed_file_names(self) -> List[str]:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        shutil.rmtree(self.raw_dir)
        path = download_url(self.urls[self.name], self.root)
        extract_zip(path, self.root)
        os.rename(osp.join(self.root, self.dwnld_file_name[self.name]),
                  self.raw_dir)
        os.unlink(path)

    def process(self):
        if self.name == 'pcqm-contact':
            # PCQM-Contact
            self.process_pcqm_contact()
        else:
            if self.name == 'coco-sp':
                # Label remapping for coco-sp.
                # See self.label_remap_coco() func
                label_map = self.label_remap_coco()

            for split in ['train', 'val', 'test']:
                if self.name.split('-')[1] == 'sp':
                    # PascalVOC-SP and COCO-SP
                    with open(osp.join(self.raw_dir, f'{split}.pickle'),
                              'rb') as f:
                        graphs = pickle.load(f)
                elif self.name.split('-')[0] == 'peptides':
                    # Peptides-func and Peptides-struct
                    with open(osp.join(self.raw_dir, f'{split}.pt'),
                              'rb') as f:
                        graphs = torch.load(f)

                data_list = []
                for graph in tqdm(graphs, desc=f'Processing {split} dataset'):
                    if self.name.split('-')[1] == 'sp':
                        """
                        PascalVOC-SP and COCO-SP
                        Each `graph` is a tuple (x, edge_attr, edge_index, y)
                            Shape of x : [num_nodes, 14]
                            Shape of edge_attr : [num_edges, 2]
                            Shape of edge_index : [2, num_edges]
                            Shape of y : [num_nodes]
                        """
                        x = graph[0].to(torch.float)
                        edge_attr = graph[1].to(torch.float)
                        edge_index = graph[2]
                        y = torch.LongTensor(graph[3])
                    elif self.name.split('-')[0] == 'peptides':
                        """
                        Peptides-func and Peptides-struct
                        Each `graph` is a tuple (x, edge_attr, edge_index, y)
                            Shape of x : [num_nodes, 9]
                            Shape of edge_attr : [num_edges, 3]
                            Shape of edge_index : [2, num_edges]
                            Shape of y : [1, 10] for Peptides-func,  or
                                         [1, 11] for Peptides-struct
                        """
                        x = graph[0]
                        edge_attr = graph[1]
                        edge_index = graph[2]
                        y = graph[3]

                    if self.name == 'coco-sp':
                        for i, label in enumerate(y):
                            y[i] = label_map[label.item()]

                    data = Data(x=x, edge_index=edge_index,
                                edge_attr=edge_attr, y=y)

                    if self.pre_filter is not None and not self.pre_filter(
                            data):
                        continue

                    if self.pre_transform is not None:
                        data = self.pre_transform(data)

                    data_list.append(data)

                torch.save(self.collate(data_list),
                           osp.join(self.processed_dir, f'{split}.pt'))

    def label_remap_coco(self):
        # Util function for name 'COCO-SP'
        # to remap the labels as the original label idxs are not contiguous
        original_label_idx = [
            0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19,
            20, 21, 22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39,
            40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57,
            58, 59, 60, 61, 62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78,
            79, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90
        ]

        label_map = {}
        for i, key in enumerate(original_label_idx):
            label_map[key] = i

        return label_map

    def process_pcqm_contact(self):
        for split in ['train', 'val', 'test']:
            with open(osp.join(self.raw_dir, f'{split}.pt'), 'rb') as f:
                graphs = torch.load(f)

            data_list = []
            for graph in tqdm(graphs, desc=f'Processing {split} dataset'):
                """
                PCQM-Contact
                Each `graph` is a tuple (x, edge_attr, edge_index,
                                        edge_label_index, edge_label)
                    Shape of x : [num_nodes, 9]
                    Shape of edge_attr : [num_edges, 3]
                    Shape of edge_index : [2, num_edges]
                    Shape of edge_label_index: [2, num_labeled_edges]
                    Shape of edge_label : [num_labeled_edges]

                    where,
                    num_labeled_edges are negative edges and link pred labels,
                    https://github.com/vijaydwivedi75/lrgb/blob/main/graphgps/loader/dataset/pcqm4mv2_contact.py#L192
                """
                x = graph[0]
                edge_attr = graph[1]
                edge_index = graph[2]
                edge_label_index = graph[3]
                edge_label = graph[4]

                data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr,
                            edge_label_index=edge_label_index,
                            edge_label=edge_label)

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)

            torch.save(self.collate(data_list),
                       osp.join(self.processed_dir, f'{split}.pt'))
pytorch/torch_geometric/datasets/malnet_tiny.py000066400000000000000000000116641444262304300224420ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
    extract_zip,
)


class MalNetTiny(InMemoryDataset):
    r"""The MalNet Tiny dataset from the
    `"A Large-Scale Database for Graph Representation Learning"
    `_ paper.
    :class:`MalNetTiny` contains 5,000 malicious and benign software function
    call graphs across 5 different types. Each graph contains at most 5k nodes.

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"trainval"`, loads the training and validation dataset.
            If :obj:`"test"`, loads the test dataset.
            If :obj:`None`, loads the entire dataset.
            (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    data_url = ('http://malnet.cc.gatech.edu/'
                'graph-data/malnet-graphs-tiny.tar.gz')
    split_url = 'http://malnet.cc.gatech.edu/split-info/split_info_tiny.zip'
    splits = ['train', 'val', 'test']

    def __init__(
        self,
        root: str,
        split: Optional[str] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        if split not in {'train', 'val', 'trainval', 'test', None}:
            raise ValueError(f'Split "{split}" found, but expected either '
                             f'"train", "val", "trainval", "test" or None')
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

        if split is not None:
            split_slices = torch.load(self.processed_paths[1])
            if split == 'train':
                self._indices = range(split_slices[0], split_slices[1])
            elif split == 'val':
                self._indices = range(split_slices[1], split_slices[2])
            elif split == 'trainval':
                self._indices = range(split_slices[0], split_slices[2])
            elif split == 'test':
                self._indices = range(split_slices[2], split_slices[3])

    @property
    def raw_file_names(self) -> List[str]:
        return ['malnet-graphs-tiny', osp.join('split_info_tiny', 'type')]

    @property
    def processed_file_names(self) -> List[str]:
        return ['data.pt', 'split_slices.pt']

    def download(self):
        path = download_url(self.data_url, self.raw_dir)
        extract_tar(path, self.raw_dir)
        os.unlink(path)

        path = download_url(self.split_url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        y_map = {}
        data_list = []
        split_slices = [0]

        for split in ['train', 'val', 'test']:
            with open(osp.join(self.raw_paths[1], f'{split}.txt'), 'r') as f:
                filenames = f.read().split('\n')[:-1]
                split_slices.append(split_slices[-1] + len(filenames))

            for filename in filenames:
                path = osp.join(self.raw_paths[0], f'{filename}.edgelist')
                malware_type = filename.split('/')[0]
                y = y_map.setdefault(malware_type, len(y_map))

                with open(path, 'r') as f:
                    edges = f.read().split('\n')[5:-1]

                edge_index = [[int(s) for s in edge.split()] for edge in edges]
                edge_index = torch.tensor(edge_index).t().contiguous()
                num_nodes = int(edge_index.max()) + 1
                data = Data(edge_index=edge_index, y=y, num_nodes=num_nodes)
                data_list.append(data)

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        torch.save(self.collate(data_list), self.processed_paths[0])
        torch.save(split_slices, self.processed_paths[1])
pytorch/torch_geometric/datasets/md17.py000066400000000000000000000401421444262304300206600ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
    extract_zip,
)


class MD17(InMemoryDataset):
    r"""A variety of ab-initio molecular dynamics trajectories from the authors
    of `sGDML `_.
    This class provides access to the original MD17 datasets, their revised
    versions, and the CCSD(T) trajectories.

    For every trajectory, the dataset contains the Cartesian positions of atoms
    (in Angstrom), their atomic numbers, as well as the total energy
    (in kcal/mol) and forces (kcal/mol/Angstrom) on each atom.
    The latter two are the regression targets for this collection.

    .. note::

        Data objects contain no edge indices as these are most commonly
        constructed via the :obj:`torch_geometric.transforms.RadiusGraph`
        transform, with its cut-off being a hyperparameter.

    The `original MD17 dataset `_ contains
    ten molecule trajectories.
    This version of the dataset was found to suffer from high numerical noise.
    The `revised MD17 dataset `_ contains the
    same molecules, but the energies and forces were recalculated at the
    PBE/def2-SVP level of theory using very tight SCF convergence and very
    dense DFT integration grid.
    The third version of the dataset contains fewer molecules, computed at the
    CCSD(T) level of theory.
    The benzene molecule at the DFT FHI-aims level of theory was
    `released separately `_.

    Check the table below for detailed information on the molecule, level of
    theory and number of data points contained in each dataset.
    Which trajectory is loaded is determined by the :attr:`name` argument.
    For the coupled cluster trajectories, the dataset comes with pre-defined
    training and testing splits which are loaded separately via the
    :attr:`train` argument.

    +--------------------+--------------------+-------------------------------+-----------+
    | Molecule           | Level of Theory    | Name                          | #Examples |
    +====================+====================+===============================+===========+
    | Benzene            | DFT                | :obj:`benzene`                | 627,983   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Uracil             | DFT                | :obj:`uracil`                 | 133,770   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Naphthalene        | DFT                | :obj:`napthalene`             | 326,250   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Aspirin            | DFT                | :obj:`aspirin`                | 211,762   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Salicylic acid     | DFT                | :obj:`salicylic acid`         | 320,231   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Malonaldehyde      | DFT                | :obj:`malonaldehyde`          | 993,237   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Ethanol            | DFT                | :obj:`ethanol`                | 555,092   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Toluene            | DFT                | :obj:`toluene`                | 442,790   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Paracetamol        | DFT                | :obj:`paracetamol`            | 106,490   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Azobenzene         | DFT                | :obj:`azobenzene`             | 99,999    |
    +--------------------+--------------------+-------------------------------+-----------+
    | Benzene (R)        | DFT (PBE/def2-SVP) | :obj:`revised benzene`        | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Uracil (R)         | DFT (PBE/def2-SVP) | :obj:`revised uracil`         | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Naphthalene (R)    | DFT (PBE/def2-SVP) | :obj:`revised napthalene`     | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Aspirin (R)        | DFT (PBE/def2-SVP) | :obj:`revised aspirin`        | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Salicylic acid (R) | DFT (PBE/def2-SVP) | :obj:`revised salicylic acid` | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Malonaldehyde (R)  | DFT (PBE/def2-SVP) | :obj:`revised malonaldehyde`  | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Ethanol (R)        | DFT (PBE/def2-SVP) | :obj:`revised ethanol`        | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Toluene (R)        | DFT (PBE/def2-SVP) | :obj:`revised toluene`        | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Paracetamol (R)    | DFT (PBE/def2-SVP) | :obj:`revised paracetamol`    | 100,000   |
    +--------------------+--------------------+-------------------------------+-----------+
    | Azobenzene (R)     | DFT (PBE/def2-SVP) | :obj:`revised azobenzene`     | 99,988    |
    +--------------------+--------------------+-------------------------------+-----------+
    | Benzene            | CCSD(T)            | :obj:`benzene CCSD(T)`        | 1,500     |
    +--------------------+--------------------+-------------------------------+-----------+
    | Aspirin            | CCSD               | :obj:`aspirin CCSD`           | 1,500     |
    +--------------------+--------------------+-------------------------------+-----------+
    | Malonaldehyde      | CCSD(T)            | :obj:`malonaldehyde CCSD(T)`  | 1,500     |
    +--------------------+--------------------+-------------------------------+-----------+
    | Ethanol            | CCSD(T)            | :obj:`ethanol CCSD(T)`        | 2,000     |
    +--------------------+--------------------+-------------------------------+-----------+
    | Toluene            | CCSD(T)            | :obj:`toluene CCSD(T)`        | 1,501     |
    +--------------------+--------------------+-------------------------------+-----------+
    | Benzene            | DFT FHI-aims       | :obj:`benzene FHI-aims`       | 49,863    |
    +--------------------+--------------------+-------------------------------+-----------+

    .. warning::

        It is advised to not train a model on more than 1,000 samples from the
        original or revised MD17 dataset.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): Keyword of the trajectory that should be loaded.
        train (bool, optional): Determines whether the train or test split
            gets loaded for the coupled cluster trajectories.
            (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #tasks
        * - Benzene
          - 627,983
          - 12
          - 0
          - 1
          - 2
        * - Uracil
          - 133,770
          - 12
          - 0
          - 1
          - 2
        * - Naphthalene
          - 326,250
          - 10
          - 0
          - 1
          - 2
        * - Aspirin
          - 211,762
          - 21
          - 0
          - 1
          - 2
        * - Salicylic acid
          - 320,231
          - 16
          - 0
          - 1
          - 2
        * - Malonaldehyde
          - 993,237
          - 9
          - 0
          - 1
          - 2
        * - Ethanol
          - 555,092
          - 9
          - 0
          - 1
          - 2
        * - Toluene
          - 442,790
          - 15
          - 0
          - 1
          - 2
        * - Paracetamol
          - 106,490
          - 20
          - 0
          - 1
          - 2
        * - Azobenzene
          - 99,999
          - 24
          - 0
          - 1
          - 2
        * - Benzene (R)
          - 100,000
          - 12
          - 0
          - 1
          - 2
        * - Uracil (R)
          - 100,000
          - 12
          - 0
          - 1
          - 2
        * - Naphthalene (R)
          - 100,000
          - 10
          - 0
          - 1
          - 2
        * - Aspirin (R)
          - 100,000
          - 21
          - 0
          - 1
          - 2
        * - Salicylic acid (R)
          - 100,000
          - 16
          - 0
          - 1
          - 2
        * - Malonaldehyde (R)
          - 100,000
          - 9
          - 0
          - 1
          - 2
        * - Ethanol (R)
          - 100,000
          - 9
          - 0
          - 1
          - 2
        * - Toluene (R)
          - 100,000
          - 15
          - 0
          - 1
          - 2
        * - Paracetamol (R)
          - 100,000
          - 20
          - 0
          - 1
          - 2
        * - Azobenzene (R)
          - 99,988
          - 24
          - 0
          - 1
          - 2
        * - Benzene CCSD-T
          - 1,500
          - 12
          - 0
          - 1
          - 2
        * - Aspirin CCSD-T
          - 1,500
          - 21
          - 0
          - 1
          - 2
        * - Malonaldehyde CCSD-T
          - 1,500
          - 9
          - 0
          - 1
          - 2
        * - Ethanol CCSD-T
          - 2000
          - 9
          - 0
          - 1
          - 2
        * - Toluene CCSD-T
          - 1,501
          - 15
          - 0
          - 1
          - 2
        * - Benzene FHI-aims
          - 49,863
          - 12
          - 0
          - 1
          - 2
    """  # noqa: E501

    gdml_url = 'http://quantum-machine.org/gdml/data/npz'
    revised_url = ('https://archive.materialscloud.org/record/'
                   'file?filename=rmd17.tar.bz2&record_id=466')

    file_names = {
        'benzene': 'md17_benzene2017.npz',
        'uracil': 'md17_uracil.npz',
        'naphtalene': 'md17_naphthalene.npz',
        'aspirin': 'md17_aspirin.npz',
        'salicylic acid': 'md17_salicylic.npz',
        'malonaldehyde': 'md17_malonaldehyde.npz',
        'ethanol': 'md17_ethanol.npz',
        'toluene': 'md17_toluene.npz',
        'paracetamol': 'paracetamol_dft.npz',
        'azobenzene': 'azobenzene_dft.npz',
        'revised benzene': 'rmd17_benzene.npz',
        'revised uracil': 'rmd17_uracil.npz',
        'revised naphthalene': 'rmd17_naphthalene.npz',
        'revised aspirin': 'rmd17_aspirin.npz',
        'revised salicylic acid': 'rmd17_salicylic.npz',
        'revised malonaldehyde': 'rmd17_malonaldehyde.npz',
        'revised ethanol': 'rmd17_ethanol.npz',
        'revised toluene': 'rmd17_toluene.npz',
        'revised paracetamol': 'rmd17_paracetamol.npz',
        'revised azobenzene': 'rmd17_azobenzene.npz',
        'benzene CCSD(T)': 'benzene_ccsd_t.zip',
        'aspirin CCSD': 'aspirin_ccsd.zip',
        'malonaldehyde CCSD(T)': 'malonaldehyde_ccsd_t.zip',
        'ethanol CCSD(T)': 'ethanol_ccsd_t.zip',
        'toluene CCSD(T)': 'toluene_ccsd_t.zip',
        'benzene FHI-aims': 'benzene2018_dft.npz',
    }

    def __init__(
        self,
        root: str,
        name: str,
        train: Optional[bool] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        if name not in self.file_names:
            raise ValueError(f"Unknown dataset name '{name}'")

        self.name = name
        self.revised = 'revised' in name
        self.ccsd = 'CCSD' in self.name

        super().__init__(root, transform, pre_transform, pre_filter)

        if len(self.processed_file_names) == 1 and train is not None:
            raise ValueError(
                f"'{self.name}' dataset does not provide pre-defined splits "
                f"but the 'train' argument is set to '{train}'")
        elif len(self.processed_file_names) == 2 and train is None:
            raise ValueError(
                f"'{self.name}' dataset does provide pre-defined splits but "
                f"the 'train' argument was not specified")

        idx = 0 if train is None or train else 1
        self.data, self.slices = torch.load(self.processed_paths[idx])

    def mean(self) -> float:
        return float(self._data.energy.mean())

    @property
    def raw_dir(self) -> str:
        if self.revised:
            return osp.join(self.root, 'raw')
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        name = self.file_names[self.name]
        if self.revised:
            return osp.join('rmd17', 'npz_data', name)
        elif self.ccsd:
            return name[:-4] + '-train.npz', name[:-4] + '-test.npz'
        return name

    @property
    def processed_file_names(self) -> List[str]:
        if self.ccsd:
            return ['train.pt', 'test.pt']
        else:
            return ['data.pt']

    def download(self):
        if self.revised:
            path = download_url(self.revised_url, self.raw_dir)
            extract_tar(path, self.raw_dir, mode='r:bz2')
            os.unlink(path)
        else:
            url = f'{self.gdml_url}/{self.file_names[self.name]}'
            path = download_url(url, self.raw_dir)
            if self.ccsd:
                extract_zip(path, self.raw_dir)
                os.unlink(path)

    def process(self):
        it = zip(self.raw_paths, self.processed_paths)
        for raw_path, processed_path in it:
            raw_data = np.load(raw_path)

            if self.revised:
                z = torch.from_numpy(raw_data['nuclear_charges']).long()
                pos = torch.from_numpy(raw_data['coords']).float()
                energy = torch.from_numpy(raw_data['energies']).float()
                force = torch.from_numpy(raw_data['forces']).float()
            else:
                z = torch.from_numpy(raw_data['z']).long()
                pos = torch.from_numpy(raw_data['R']).float()
                energy = torch.from_numpy(raw_data['E']).float()
                force = torch.from_numpy(raw_data['F']).float()

            data_list = []
            for i in range(pos.size(0)):
                data = Data(z=z, pos=pos[i], energy=energy[i], force=force[i])
                if self.pre_filter is not None and not self.pre_filter(data):
                    continue
                if self.pre_transform is not None:
                    data = self.pre_transform(data)
                data_list.append(data)

            torch.save(self.collate(data_list), processed_path)

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}({len(self)}, name='{self.name}')"
pytorch/torch_geometric/datasets/mixhop_synthetic_dataset.py000066400000000000000000000072721444262304300252220ustar00rootroot00000000000000import os.path as osp
import pickle
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class MixHopSyntheticDataset(InMemoryDataset):
    r"""The MixHop synthetic dataset from the `"MixHop: Higher-Order
    Graph Convolutional Architectures via Sparsified Neighborhood Mixing"
    `_ paper, containing 10
    graphs, each with varying degree of homophily (ranging from 0.0 to 0.9).
    All graphs have 5,000 nodes, where each node corresponds to 1 out of 10
    classes.
    The feature values of the nodes are sampled from a 2D Gaussian
    distribution, which are distinct for each class.

    Args:
        root (str): Root directory where the dataset should be saved.
        homophily (float): The degree of homophily (one of :obj:`0.0`,
            :obj:`0.1`, ..., :obj:`0.9`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = ('https://raw.githubusercontent.com/samihaija/mixhop/master/data'
           '/synthetic')

    def __init__(
        self,
        root: str,
        homophily: float,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.homophily = homophily
        assert homophily in [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
        super().__init__(root, transform, pre_transform)

        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, f'{self.homophily:0.1f}'[::2], 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, f'{self.homophily:0.1f}'[::2], 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        name = f'ind.n5000-h{self.homophily:0.1f}-c10'
        return [f'{name}.allx', f'{name}.ally', f'{name}.graph']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process(self):
        x = torch.from_numpy(np.load(self.raw_paths[0]))
        y = torch.from_numpy(np.load(self.raw_paths[1])).argmax(dim=-1)

        edges = pickle.load(open(self.raw_paths[2], 'rb'), encoding='latin1')
        row, col = [], []
        for k, v in edges.items():
            row += [k] * len(v)
            col += v

        edge_index = torch.tensor([row, col], dtype=torch.long)

        N_s = x.size(0) // 3
        train_mask = torch.zeros(x.size(0), dtype=torch.bool)
        train_mask[:N_s] = True
        val_mask = torch.zeros(x.size(0), dtype=torch.bool)
        val_mask[N_s:2 * N_s] = True
        test_mask = torch.zeros(x.size(0), dtype=torch.bool)
        test_mask[2 * N_s:] = True

        data = Data(x=x, y=y, edge_index=edge_index, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(homophily={self.homophily:.1f})'
pytorch/torch_geometric/datasets/mnist_superpixels.py000066400000000000000000000061001444262304300237010ustar00rootroot00000000000000import os
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class MNISTSuperpixels(InMemoryDataset):
    r"""MNIST superpixels dataset from the `"Geometric Deep Learning on
    Graphs and Manifolds Using Mixture Model CNNs"
    `_ paper, containing 70,000 graphs with
    75 nodes each.
    Every graph is labeled by one of 10 classes.

    Args:
        root (str): Root directory where the dataset should be saved.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 70,000
          - 75
          - ~1,393.0
          - 1
          - 10
    """

    url = 'https://data.pyg.org/datasets/MNISTSuperpixels.zip'

    def __init__(
        self,
        root: str,
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> str:
        return 'MNISTSuperpixels.pt'

    @property
    def processed_file_names(self) -> List[str]:
        return ['train_data.pt', 'test_data.pt']

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        inputs = torch.load(self.raw_paths[0])
        for i in range(len(inputs)):
            data_list = [Data(**data_dict) for data_dict in inputs[i]]

            if self.pre_filter is not None:
                data_list = [d for d in data_list if self.pre_filter(d)]

            if self.pre_transform is not None:
                data_list = [self.pre_transform(d) for d in data_list]

            torch.save(self.collate(data_list), self.processed_paths[i])
pytorch/torch_geometric/datasets/modelnet.py000066400000000000000000000122031444262304300217140ustar00rootroot00000000000000import glob
import os
import os.path as osp
import shutil
from typing import Callable, Dict, List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_off


class ModelNet(InMemoryDataset):
    r"""The ModelNet10/40 datasets from the `"3D ShapeNets: A Deep
    Representation for Volumetric Shapes"
    `_ paper,
    containing CAD models of 10 and 40 categories, respectively.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str, optional): The name of the dataset (:obj:`"10"` for
            ModelNet10, :obj:`"40"` for ModelNet40). (default: :obj:`"10"`)
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - ModelNet10
          - 4,899
          - ~9,508.2
          - ~37,450.5
          - 3
          - 10
        * - ModelNet40
          - 12,311
          - ~17,744.4
          - ~66,060.9
          - 3
          - 40
    """

    urls = {
        '10':
        'http://vision.princeton.edu/projects/2014/3DShapeNets/ModelNet10.zip',
        '40': 'http://modelnet.cs.princeton.edu/ModelNet40.zip'
    }

    def __init__(
        self,
        root: str,
        name: str = '10',
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        assert name in ['10', '40']
        self.name = name
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'bathtub', 'bed', 'chair', 'desk', 'dresser', 'monitor',
            'night_stand', 'sofa', 'table', 'toilet'
        ]

    @property
    def processed_file_names(self) -> List[str]:
        return ['training.pt', 'test.pt']

    def download(self):
        path = download_url(self.urls[self.name], self.root)
        extract_zip(path, self.root)
        os.unlink(path)
        folder = osp.join(self.root, f'ModelNet{self.name}')
        shutil.rmtree(self.raw_dir)
        os.rename(folder, self.raw_dir)

        # Delete osx metadata generated during compression of ModelNet10
        metadata_folder = osp.join(self.root, '__MACOSX')
        if osp.exists(metadata_folder):
            shutil.rmtree(metadata_folder)

    def process(self):
        torch.save(self.process_set('train'), self.processed_paths[0])
        torch.save(self.process_set('test'), self.processed_paths[1])

    def process_set(self, dataset: str) -> Tuple[Data, Dict[str, Tensor]]:
        categories = glob.glob(osp.join(self.raw_dir, '*', ''))
        categories = sorted([x.split(os.sep)[-2] for x in categories])

        data_list = []
        for target, category in enumerate(categories):
            folder = osp.join(self.raw_dir, category, dataset)
            paths = glob.glob(f'{folder}/{category}_*.off')
            for path in paths:
                data = read_off(path)
                data.y = torch.tensor([target])
                data_list.append(data)

        if self.pre_filter is not None:
            data_list = [d for d in data_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(d) for d in data_list]

        return self.collate(data_list)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}{self.name}({len(self)})'
pytorch/torch_geometric/datasets/molecule_net.py000066400000000000000000000147011444262304300225650ustar00rootroot00000000000000import os
import os.path as osp
import re
from typing import Callable, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url, extract_gz
from torch_geometric.utils import from_smiles


class MoleculeNet(InMemoryDataset):
    r"""The `MoleculeNet `_ benchmark
    collection  from the `"MoleculeNet: A Benchmark for Molecular Machine
    Learning" `_ paper, containing datasets
    from physical chemistry, biophysics and physiology.
    All datasets come with the additional node and edge features introduced by
    the :ogb:`null`
    `Open Graph Benchmark `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"ESOL"`, :obj:`"FreeSolv"`,
            :obj:`"Lipo"`, :obj:`"PCBA"`, :obj:`"MUV"`, :obj:`"HIV"`,
            :obj:`"BACE"`, :obj:`"BBBP"`, :obj:`"Tox21"`, :obj:`"ToxCast"`,
            :obj:`"SIDER"`, :obj:`"ClinTox"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - ESOL
          - 1,128
          - ~13.3
          - ~27.4
          - 9
          - 1
        * - FreeSolv
          - 642
          - ~8.7
          - ~16.8
          - 9
          - 1
        * - Lipophilicity
          - 4,200
          - ~27.0
          - ~59.0
          - 9
          - 1
        * - PCBA
          - 437,929
          - ~26.0
          - ~56.2
          - 9
          - 128
        * - MUV
          - 93,087
          - ~24.2
          - ~52.6
          - 9
          - 17
        * - HIV
          - 41,127
          - ~25.5
          - ~54.9
          - 9
          - 1
        * - BACE
          - 1513
          - ~34.1
          - ~73.7
          - 9
          - 1
        * - BBBP
          - 2,050
          - ~23.9
          - ~51.6
          - 9
          - 1
        * - Tox21
          - 7,831
          - ~18.6
          - ~38.6
          - 9
          - 12
        * - ToxCast
          - 8,597
          - ~18.7
          - ~38.4
          - 9
          - 617
        * - SIDER
          - 1,427
          - ~33.6
          - ~70.7
          - 9
          - 27
        * - ClinTox
          - 1,484
          - ~26.1
          - ~55.5
          - 9
          - 2
    """

    url = 'https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/{}'

    # Format: name: [display_name, url_name, csv_name, smiles_idx, y_idx]
    names = {
        'esol': ['ESOL', 'delaney-processed.csv', 'delaney-processed', -1, -2],
        'freesolv': ['FreeSolv', 'SAMPL.csv', 'SAMPL', 1, 2],
        'lipo': ['Lipophilicity', 'Lipophilicity.csv', 'Lipophilicity', 2, 1],
        'pcba': ['PCBA', 'pcba.csv.gz', 'pcba', -1,
                 slice(0, 128)],
        'muv': ['MUV', 'muv.csv.gz', 'muv', -1,
                slice(0, 17)],
        'hiv': ['HIV', 'HIV.csv', 'HIV', 0, -1],
        'bace': ['BACE', 'bace.csv', 'bace', 0, 2],
        'bbbp': ['BBBP', 'BBBP.csv', 'BBBP', -1, -2],
        'tox21': ['Tox21', 'tox21.csv.gz', 'tox21', -1,
                  slice(0, 12)],
        'toxcast':
        ['ToxCast', 'toxcast_data.csv.gz', 'toxcast_data', 0,
         slice(1, 618)],
        'sider': ['SIDER', 'sider.csv.gz', 'sider', 0,
                  slice(1, 28)],
        'clintox': ['ClinTox', 'clintox.csv.gz', 'clintox', 0,
                    slice(1, 3)],
    }

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in self.names.keys()
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.names[self.name][2]}.csv'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        url = self.url.format(self.names[self.name][1])
        path = download_url(url, self.raw_dir)
        if self.names[self.name][1][-2:] == 'gz':
            extract_gz(path, self.raw_dir)
            os.unlink(path)

    def process(self):
        with open(self.raw_paths[0], 'r') as f:
            dataset = f.read().split('\n')[1:-1]
            dataset = [x for x in dataset if len(x) > 0]  # Filter empty lines.

        data_list = []
        for line in dataset:
            line = re.sub(r'\".*\"', '', line)  # Replace ".*" strings.
            line = line.split(',')

            smiles = line[self.names[self.name][3]]
            ys = line[self.names[self.name][4]]
            ys = ys if isinstance(ys, list) else [ys]

            ys = [float(y) if len(y) > 0 else float('NaN') for y in ys]
            y = torch.tensor(ys, dtype=torch.float).view(1, -1)

            data = from_smiles(smiles)
            data.y = y

            if self.pre_filter is not None and not self.pre_filter(data):
                continue

            if self.pre_transform is not None:
                data = self.pre_transform(data)

            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.names[self.name][0]}({len(self)})'
pytorch/torch_geometric/datasets/motif_generator/000077500000000000000000000000001444262304300227215ustar00rootroot00000000000000pytorch/torch_geometric/datasets/motif_generator/__init__.py000066400000000000000000000003431444262304300250320ustar00rootroot00000000000000from .base import MotifGenerator
from .custom import CustomMotif
from .house import HouseMotif
from .cycle import CycleMotif

__all__ = classes = [
    'MotifGenerator',
    'CustomMotif',
    'HouseMotif',
    'CycleMotif',
]
pytorch/torch_geometric/datasets/motif_generator/base.py000066400000000000000000000016161444262304300242110ustar00rootroot00000000000000from abc import ABC, abstractmethod
from typing import Any, Union

from torch_geometric.data import Data
from torch_geometric.resolver import resolver


class MotifGenerator(ABC):
    r"""An abstract base class for generating a motif."""
    @abstractmethod
    def __call__(self) -> Data:
        r"""To be implemented by :class:`Motif` subclasses."""
        pass

    @staticmethod
    def resolve(query: Union[Any, str], *args, **kwargs):
        import torch_geometric.datasets.motif_generator as motif_generators
        motif_generators = [
            gen for gen in vars(motif_generators).values()
            if isinstance(gen, type) and issubclass(gen, MotifGenerator)
        ]
        return resolver(motif_generators, {}, query, base_cls=MotifGenerator,
                        base_cls_repr='Motif', *args, **kwargs)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/datasets/motif_generator/custom.py000066400000000000000000000022631444262304300246100ustar00rootroot00000000000000from typing import Any

from torch_geometric.data import Data
from torch_geometric.datasets.motif_generator import MotifGenerator
from torch_geometric.utils import from_networkx


class CustomMotif(MotifGenerator):
    r"""Generates a motif based on a custom structure coming from a
    :class:`torch_geometric.data.Data` or :class:`networkx.Graph` object.

    Args:
        structure (torch_geometric.data.Data or networkx.Graph): The structure
            to use as a motif.
    """
    def __init__(self, structure: Any):
        super().__init__()

        self.structure = None
        if isinstance(structure, Data):
            self.structure = structure
        else:
            try:
                import networkx as nx
                if isinstance(structure, nx.Graph):
                    self.structure = from_networkx(structure)
            except ImportError:
                pass

        if self.structure is None:
            raise ValueError(f"Expected a motif structure of type "
                             f"'torch_geometric.data.Data' or 'networkx.Graph'"
                             f"(got {type(structure)})")

    def __call__(self) -> Data:
        return self.structure
pytorch/torch_geometric/datasets/motif_generator/cycle.py000066400000000000000000000017301444262304300243730ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.datasets.motif_generator import CustomMotif


class CycleMotif(CustomMotif):
    r"""Generates the cycle motif from the `"GNNExplainer:
    Generating Explanations for Graph Neural Networks"
    `__ paper.

    Args:
        num_nodes (int): The number of nodes in the cycle.
    """
    def __init__(self, num_nodes: int):
        self.num_nodes = num_nodes

        row = torch.arange(num_nodes).view(-1, 1).repeat(1, 2).view(-1)
        col1 = torch.arange(-1, num_nodes - 1) % num_nodes
        col2 = torch.arange(1, num_nodes + 1) % num_nodes
        col = torch.stack([col1, col2], dim=1).sort(dim=-1)[0].view(-1)

        structure = Data(
            num_nodes=num_nodes,
            edge_index=torch.stack([row, col], dim=0),
        )

        super().__init__(structure)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num_nodes})'
pytorch/torch_geometric/datasets/motif_generator/house.py000066400000000000000000000014411444262304300244160ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.datasets.motif_generator import CustomMotif


class HouseMotif(CustomMotif):
    r"""Generates the house-structured motif from the `"GNNExplainer:
    Generating Explanations for Graph Neural Networks"
    `__ paper, containing 5 nodes and 6
    undirected edges. Nodes are labeled according to their structural role:
    the top, middle and bottom of the house."""
    def __init__(self):
        structure = Data(
            num_nodes=5,
            edge_index=torch.tensor([
                [0, 0, 0, 1, 1, 1, 2, 2, 3, 3, 4, 4],
                [1, 3, 4, 4, 2, 0, 1, 3, 2, 0, 0, 1],
            ]),
            y=torch.tensor([0, 0, 1, 1, 2]),
        )
        super().__init__(structure)
pytorch/torch_geometric/datasets/movie_lens.py000066400000000000000000000073371444262304300222610ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class MovieLens(InMemoryDataset):
    r"""A heterogeneous rating dataset, assembled by GroupLens Research from
    the `MovieLens web site `_, consisting of nodes of
    type :obj:`"movie"` and :obj:`"user"`.
    User ratings for movies are available as ground truth labels for the edges
    between the users and the movies :obj:`("user", "rates", "movie")`.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        model_name (str): Name of model used to transform movie titles to node
            features. The model comes from the`Huggingface SentenceTransformer
            `_.
    """
    url = 'https://files.grouplens.org/datasets/movielens/ml-latest-small.zip'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        model_name: Optional[str] = 'all-MiniLM-L6-v2',
    ):
        self.model_name = model_name
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return [
            osp.join('ml-latest-small', 'movies.csv'),
            osp.join('ml-latest-small', 'ratings.csv'),
        ]

    @property
    def processed_file_names(self) -> str:
        return f'data_{self.model_name}.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        import pandas as pd
        from sentence_transformers import SentenceTransformer

        data = HeteroData()

        df = pd.read_csv(self.raw_paths[0], index_col='movieId')
        movie_mapping = {idx: i for i, idx in enumerate(df.index)}

        genres = df['genres'].str.get_dummies('|').values
        genres = torch.from_numpy(genres).to(torch.float)

        model = SentenceTransformer(self.model_name)
        with torch.no_grad():
            emb = model.encode(df['title'].values, show_progress_bar=True,
                               convert_to_tensor=True).cpu()

        data['movie'].x = torch.cat([emb, genres], dim=-1)

        df = pd.read_csv(self.raw_paths[1])
        user_mapping = {idx: i for i, idx in enumerate(df['userId'].unique())}
        data['user'].num_nodes = len(user_mapping)

        src = [user_mapping[idx] for idx in df['userId']]
        dst = [movie_mapping[idx] for idx in df['movieId']]
        edge_index = torch.tensor([src, dst])

        rating = torch.from_numpy(df['rating'].values).to(torch.long)
        data['user', 'rates', 'movie'].edge_index = edge_index
        data['user', 'rates', 'movie'].edge_label = rating

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/movie_lens_100k.py000066400000000000000000000135621444262304300230110ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)

MOVIE_HEADERS = [
    "movieId", "title", "releaseDate", "videoReleaseDate", "IMDb URL",
    "unknown", "Action", "Adventure", "Animation", "Children's", "Comedy",
    "Crime", "Documentary", "Drama", "Fantasy", "Film-Noir", "Horror",
    "Musical", "Mystery", "Romance", "Sci-Fi", "Thriller", "War", "Western"
]
USER_HEADERS = ["userId", "age", "gender", "occupation", "zipCode"]
RATING_HEADERS = ["userId", "movieId", "rating", "timestamp"]


class MovieLens100K(InMemoryDataset):
    r"""The MovieLens 100K heterogeneous rating dataset, assembled by GroupLens
    Research from the `MovieLens web site `__,
    consisting of movies (1,682 nodes) and users (943 nodes) with 100K
    ratings between them.
    User ratings for movies are available as ground truth labels.
    Features of users and movies are encoded according to the `"Inductive
    Matrix Completion Based on Graph Neural Networks"
    `__ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10
        :header-rows: 1

        * - Node/Edge Type
          - #nodes/#edges
          - #features
          - #tasks
        * - Movie
          - 1,682
          - 18
          -
        * - User
          - 943
          - 24
          -
        * - User-Movie
          - 80,000
          - 1
          - 1
    """
    url = 'https://files.grouplens.org/datasets/movielens/ml-100k.zip'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0], data_cls=HeteroData)

    @property
    def raw_file_names(self) -> List[str]:
        return ['u.item', 'u.user', 'u1.base', 'u1.test']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.remove(path)
        folder = osp.join(self.root, 'ml-100k')
        shutil.rmtree(self.raw_dir)
        os.rename(folder, self.raw_dir)

    def process(self):
        import pandas as pd

        data = HeteroData()

        # Process movie data:
        df = pd.read_csv(
            self.raw_paths[0],
            sep='|',
            header=None,
            names=MOVIE_HEADERS,
            index_col='movieId',
            encoding='ISO-8859-1',
        )
        movie_mapping = {idx: i for i, idx in enumerate(df.index)}

        x = df[MOVIE_HEADERS[6:]].values
        data['movie'].x = torch.from_numpy(x).to(torch.float)

        # Process user data:
        df = pd.read_csv(
            self.raw_paths[1],
            sep='|',
            header=None,
            names=USER_HEADERS,
            index_col='userId',
            encoding='ISO-8859-1',
        )
        user_mapping = {idx: i for i, idx in enumerate(df.index)}

        age = df['age'].values / df['age'].values.max()
        age = torch.from_numpy(age).to(torch.float).view(-1, 1)

        gender = df['gender'].str.get_dummies().values
        gender = torch.from_numpy(gender).to(torch.float)

        occupation = df['occupation'].str.get_dummies().values
        occupation = torch.from_numpy(occupation).to(torch.float)

        data['user'].x = torch.cat([age, gender, occupation], dim=-1)

        # Process rating data for training:
        df = pd.read_csv(
            self.raw_paths[2],
            sep='\t',
            header=None,
            names=RATING_HEADERS,
        )

        src = [user_mapping[idx] for idx in df['userId']]
        dst = [movie_mapping[idx] for idx in df['movieId']]
        edge_index = torch.tensor([src, dst])
        data['user', 'rates', 'movie'].edge_index = edge_index

        rating = torch.from_numpy(df['rating'].values).to(torch.long)
        data['user', 'rates', 'movie'].rating = rating

        time = torch.from_numpy(df['timestamp'].values)
        data['user', 'rates', 'movie'].time = time

        data['movie', 'rated_by', 'user'].edge_index = edge_index.flip([0])
        data['movie', 'rated_by', 'user'].rating = rating
        data['movie', 'rated_by', 'user'].time = time

        # Process rating data for testing:
        df = pd.read_csv(
            self.raw_paths[3],
            sep='\t',
            header=None,
            names=RATING_HEADERS,
        )

        src = [user_mapping[idx] for idx in df['userId']]
        dst = [movie_mapping[idx] for idx in df['movieId']]
        edge_label_index = torch.tensor([src, dst])
        data['user', 'rates', 'movie'].edge_label_index = edge_label_index

        edge_label = torch.from_numpy(df['rating'].values).to(torch.float)
        data['user', 'rates', 'movie'].edge_label = edge_label

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self.save([data], self.processed_paths[0])
pytorch/torch_geometric/datasets/movie_lens_1m.py000066400000000000000000000120521444262304300226440ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)

MOVIE_HEADERS = ["movieId", "title", "genres"]
USER_HEADERS = ["userId", "gender", "age", "occupation", "zipCode"]
RATING_HEADERS = ['userId', 'movieId', 'rating', 'timestamp']


class MovieLens1M(InMemoryDataset):
    r"""The MovieLens 1M heterogeneous rating dataset, assembled by GroupLens
    Research from the `MovieLens web site `__,
    consisting of movies (3,883 nodes) and users (6,040 nodes) with
    approximately 1 million ratings between them.
    User ratings for movies are available as ground truth labels.
    Features of users and movies are encoded according to the `"Inductive
    Matrix Completion Based on Graph Neural Networks"
    `__ paper.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10
        :header-rows: 1

        * - Node/Edge Type
          - #nodes/#edges
          - #features
          - #tasks
        * - Movie
          - 3,883
          - 18
          -
        * - User
          - 6,040
          - 30
          -
        * - User-Movie
          - 1,000,209
          - 1
          - 1
    """
    url = 'https://files.grouplens.org/datasets/movielens/ml-1m.zip'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0], data_cls=HeteroData)

    @property
    def raw_file_names(self) -> List[str]:
        return ['movies.dat', 'users.dat', 'ratings.dat']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.remove(path)
        folder = osp.join(self.root, 'ml-1m')
        shutil.rmtree(self.raw_dir)
        os.rename(folder, self.raw_dir)

    def process(self):
        import pandas as pd

        data = HeteroData()

        # Process movie data:
        df = pd.read_csv(
            self.raw_paths[0],
            sep='::',
            header=None,
            index_col='movieId',
            names=MOVIE_HEADERS,
            encoding='ISO-8859-1',
            engine='python',
        )
        movie_mapping = {idx: i for i, idx in enumerate(df.index)}

        genres = df['genres'].str.get_dummies('|').values
        genres = torch.from_numpy(genres).to(torch.float)

        data['movie'].x = genres

        # Process user data:
        df = pd.read_csv(
            self.raw_paths[1],
            sep='::',
            header=None,
            index_col='userId',
            names=USER_HEADERS,
            dtype='str',
            encoding='ISO-8859-1',
            engine='python',
        )
        user_mapping = {idx: i for i, idx in enumerate(df.index)}

        age = df['age'].str.get_dummies().values
        age = torch.from_numpy(age).to(torch.float)

        gender = df['gender'].str.get_dummies().values
        gender = torch.from_numpy(gender).to(torch.float)

        occupation = df['occupation'].str.get_dummies().values
        occupation = torch.from_numpy(occupation).to(torch.float)

        data['user'].x = torch.cat([age, gender, occupation], dim=-1)

        # Process rating data:
        df = pd.read_csv(
            self.raw_paths[2],
            sep='::',
            header=None,
            names=RATING_HEADERS,
            encoding='ISO-8859-1',
            engine='python',
        )

        src = [user_mapping[idx] for idx in df['userId']]
        dst = [movie_mapping[idx] for idx in df['movieId']]
        edge_index = torch.tensor([src, dst])
        data['user', 'rates', 'movie'].edge_index = edge_index

        rating = torch.from_numpy(df['rating'].values).to(torch.long)
        data['user', 'rates', 'movie'].rating = rating

        time = torch.from_numpy(df['timestamp'].values)
        data['user', 'rates', 'movie'].time = time

        data['movie', 'rated_by', 'user'].edge_index = edge_index.flip([0])
        data['movie', 'rated_by', 'user'].rating = rating
        data['movie', 'rated_by', 'user'].time = time

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        self.save([data], self.processed_paths[0])
pytorch/torch_geometric/datasets/nell.py000066400000000000000000000055111444262304300210430ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url, extract_tar
from torch_geometric.io import read_planetoid_data


class NELL(InMemoryDataset):
    r"""The NELL dataset, a knowledge graph from the
    `"Toward an Architecture for Never-Ending Language Learning"
    `_ paper.
    The dataset is processed as in the
    `"Revisiting Semi-Supervised Learning with Graph Embeddings"
    `_ paper.

    .. note::

        Entity nodes are described by sparse feature vectors of type
        :class:`torch_sparse.SparseTensor`, which can be either used directly,
        or can be converted via :obj:`data.x.to_dense()`,
        :obj:`data.x.to_scipy()` or :obj:`data.x.to_torch_sparse_csr_tensor()`.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 65,755
          - 251,550
          - 61,278
          - 186
    """

    url = 'http://www.cs.cmu.edu/~zhiliny/data/nell_data.tar.gz'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        names = ['x', 'tx', 'allx', 'y', 'ty', 'ally', 'graph', 'test.index']
        return [f'ind.nell.0.001.{name}' for name in names]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.root)
        extract_tar(path, self.root)
        os.unlink(path)
        shutil.rmtree(self.raw_dir)
        os.rename(osp.join(self.root, 'nell_data'), self.raw_dir)

    def process(self):
        data = read_planetoid_data(self.raw_dir, 'nell.0.001')
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/ogb_mag.py000066400000000000000000000160261444262304300215070ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class OGB_MAG(InMemoryDataset):
    r"""The :obj:`ogbn-mag` dataset from the `"Open Graph Benchmark: Datasets
    for Machine Learning on Graphs" `_ paper.
    :obj:`ogbn-mag` is a heterogeneous graph composed of a subset of the
    Microsoft Academic Graph (MAG).
    It contains four types of entities β€” papers (736,389 nodes), authors
    (1,134,649 nodes), institutions (8,740 nodes), and fields of study
    (59,965 nodes) β€” as well as four types of directed relations connecting two
    types of entities.
    Each paper is associated with a 128-dimensional :obj:`word2vec` feature
    vector, while all other node types are not associated with any input
    features.
    The task is to predict the venue (conference or journal) of each paper.
    In total, there are 349 different venues.

    Args:
        root (str): Root directory where the dataset should be saved.
        preprocess (str, optional): Pre-processes the original dataset by
            adding structural features (:obj:`"metapath2vec"`, :obj:`"TransE"`)
            to featureless nodes. (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'http://snap.stanford.edu/ogb/data/nodeproppred/mag.zip'
    urls = {
        'metapath2vec': ('https://data.pyg.org/datasets/'
                         'mag_metapath2vec_emb.zip'),
        'transe': ('https://data.pyg.org/datasets/'
                   'mag_transe_emb.zip'),
    }

    def __init__(self, root: str, preprocess: Optional[str] = None,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        preprocess = None if preprocess is None else preprocess.lower()
        self.preprocess = preprocess
        assert self.preprocess in [None, 'metapath2vec', 'transe']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def num_classes(self) -> int:
        return int(self._data['paper'].y.max()) + 1

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, 'mag', 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, 'mag', 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        file_names = [
            'node-feat', 'node-label', 'relations', 'split',
            'num-node-dict.csv.gz'
        ]

        if self.preprocess is not None:
            file_names += [f'mag_{self.preprocess}_emb.pt']

        return file_names

    @property
    def processed_file_names(self) -> str:
        if self.preprocess is not None:
            return f'data_{self.preprocess}.pt'
        else:
            return 'data.pt'

    def download(self):
        if not all([osp.exists(f) for f in self.raw_paths[:5]]):
            path = download_url(self.url, self.raw_dir)
            extract_zip(path, self.raw_dir)
            for file_name in ['node-feat', 'node-label', 'relations']:
                path = osp.join(self.raw_dir, 'mag', 'raw', file_name)
                shutil.move(path, self.raw_dir)
            path = osp.join(self.raw_dir, 'mag', 'split')
            shutil.move(path, self.raw_dir)
            path = osp.join(self.raw_dir, 'mag', 'raw', 'num-node-dict.csv.gz')
            shutil.move(path, self.raw_dir)
            shutil.rmtree(osp.join(self.raw_dir, 'mag'))
            os.remove(osp.join(self.raw_dir, 'mag.zip'))
        if self.preprocess is not None:
            path = download_url(self.urls[self.preprocess], self.raw_dir)
            extract_zip(path, self.raw_dir)
            os.remove(path)

    def process(self):
        import pandas as pd

        data = HeteroData()

        path = osp.join(self.raw_dir, 'node-feat', 'paper', 'node-feat.csv.gz')
        x_paper = pd.read_csv(path, compression='gzip', header=None,
                              dtype=np.float32).values
        data['paper'].x = torch.from_numpy(x_paper)

        path = osp.join(self.raw_dir, 'node-feat', 'paper', 'node_year.csv.gz')
        year_paper = pd.read_csv(path, compression='gzip', header=None,
                                 dtype=np.int64).values
        data['paper'].year = torch.from_numpy(year_paper).view(-1)

        path = osp.join(self.raw_dir, 'node-label', 'paper',
                        'node-label.csv.gz')
        y_paper = pd.read_csv(path, compression='gzip', header=None,
                              dtype=np.int64).values.flatten()
        data['paper'].y = torch.from_numpy(y_paper)

        if self.preprocess is None:
            path = osp.join(self.raw_dir, 'num-node-dict.csv.gz')
            num_nodes_df = pd.read_csv(path, compression='gzip')
            for node_type in ['author', 'institution', 'field_of_study']:
                data[node_type].num_nodes = num_nodes_df[node_type].tolist()[0]
        else:
            emb_dict = torch.load(self.raw_paths[-1])
            for key, value in emb_dict.items():
                if key != 'paper':
                    data[key].x = value

        for edge_type in [('author', 'affiliated_with', 'institution'),
                          ('author', 'writes', 'paper'),
                          ('paper', 'cites', 'paper'),
                          ('paper', 'has_topic', 'field_of_study')]:

            f = '___'.join(edge_type)
            path = osp.join(self.raw_dir, 'relations', f, 'edge.csv.gz')
            edge_index = pd.read_csv(path, compression='gzip', header=None,
                                     dtype=np.int64).values
            edge_index = torch.from_numpy(edge_index).t().contiguous()
            data[edge_type].edge_index = edge_index

        for f, v in [('train', 'train'), ('valid', 'val'), ('test', 'test')]:
            path = osp.join(self.raw_dir, 'split', 'time', 'paper',
                            f'{f}.csv.gz')
            idx = pd.read_csv(path, compression='gzip', header=None,
                              dtype=np.int64).values.flatten()
            idx = torch.from_numpy(idx)
            mask = torch.zeros(data['paper'].num_nodes, dtype=torch.bool)
            mask[idx] = True
            data['paper'][f'{v}_mask'] = mask

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return 'ogbn-mag()'
pytorch/torch_geometric/datasets/omdb.py000066400000000000000000000065271444262304300210420ustar00rootroot00000000000000import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, extract_tar


class OMDB(InMemoryDataset):
    r"""The `Organic Materials Database (OMDB)
    `__ of bulk organic crystals.

    Args:
        root (str): Root directory where the dataset should be saved.
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'https://omdb.mathub.io/dataset'

    def __init__(self, root: str, train: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> str:
        return 'OMDB-GAP1_v1.1.tar.gz'

    @property
    def processed_file_names(self) -> List[str]:
        return ['train_data.pt', 'test_data.pt']

    def download(self):
        raise RuntimeError(
            f"Dataset not found. Please download '{self.raw_file_names}' from "
            f"'{self.url}' and move it to '{self.raw_dir}'")

    def process(self):
        from ase.io import read

        extract_tar(self.raw_paths[0], self.raw_dir, log=False)
        materials = read(osp.join(self.raw_dir, 'structures.xyz'), index=':')
        bandgaps = np.loadtxt(osp.join(self.raw_dir, 'bandgaps.csv'))

        data_list = []
        for material, bandgap in zip(materials, bandgaps):
            pos = torch.from_numpy(material.get_positions()).to(torch.float)
            z = torch.from_numpy(material.get_atomic_numbers()).to(torch.int64)
            y = torch.tensor([float(bandgap)])
            data_list.append(Data(z=z, pos=pos, y=y))

        train_data = data_list[:10000]
        test_data = data_list[10000:]

        if self.pre_filter is not None:
            train_data = [d for d in train_data if self.pre_filter(d)]
            test_data = [d for d in test_data if self.pre_filter(d)]

        if self.pre_transform is not None:
            train_data = [self.pre_transform(d) for d in train_data]
            test_data = [self.pre_transform(d) for d in test_data]

        torch.save(self.collate(train_data), self.processed_paths[0])
        torch.save(self.collate(test_data), self.processed_paths[1])
pytorch/torch_geometric/datasets/particle.py000066400000000000000000000075741444262304300217270ustar00rootroot00000000000000import glob
import os.path as osp

import numpy as np
import torch

from torch_geometric.data import Data, Dataset
from torch_geometric.utils import index_sort, scatter


class TrackingData(Data):
    def __inc__(self, key, value, *args, **kwargs):
        if key == 'y_index':
            return torch.tensor([value[0].max().item() + 1, self.num_nodes])
        else:
            return super().__inc__(key, value, *args, **kwargs)


class TrackMLParticleTrackingDataset(Dataset):
    r"""The `TrackML Particle Tracking Challenge
    `_ dataset to
    reconstruct particle tracks from 3D points left in the silicon detectors.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
    """

    url = 'https://www.kaggle.com/c/trackml-particle-identification'

    def __init__(self, root, transform=None):
        super().__init__(root, transform)
        events = glob.glob(osp.join(self.raw_dir, 'event*-hits.csv'))
        events = [e.split(osp.sep)[-1].split('-')[0][5:] for e in events]
        self.events = sorted(events)

    @property
    def raw_file_names(self):
        event_indices = ['000001000']
        file_names = []
        file_names += [f'event{idx}-cells.csv' for idx in event_indices]
        file_names += [f'event{idx}-hits.csv' for idx in event_indices]
        file_names += [f'event{idx}-particles.csv' for idx in event_indices]
        file_names += [f'event{idx}-truth.csv' for idx in event_indices]
        return file_names

    def download(self):
        raise RuntimeError(
            f'Dataset not found. Please download it from {self.url} and move '
            f'all *.csv files to {self.raw_dir}')

    def len(self):
        return len(glob.glob(osp.join(self.raw_dir, 'event*-hits.csv')))

    def get(self, idx):
        import pandas as pd

        idx = self.events[idx]

        # Get hit positions.
        hits_path = osp.join(self.raw_dir, f'event{idx}-hits.csv')
        pos = pd.read_csv(hits_path, usecols=['x', 'y', 'z'], dtype=np.float32)
        pos = torch.from_numpy(pos.values).div_(1000.)

        # Get hit features.
        cells_path = osp.join(self.raw_dir, f'event{idx}-cells.csv')
        cell = pd.read_csv(cells_path, usecols=['hit_id', 'value'])
        hit_id = torch.from_numpy(cell['hit_id'].values).to(torch.long).sub_(1)
        value = torch.from_numpy(cell['value'].values).to(torch.float)
        ones = torch.ones(hit_id.size(0))
        num_cells = scatter(ones, hit_id, 0, pos.size(0), 'sum').div_(10.)
        value = scatter(value, hit_id, 0, pos.size(0), 'sum')
        x = torch.stack([num_cells, value], dim=-1)

        # Get ground-truth hit assignments.
        truth_path = osp.join(self.raw_dir, f'event{idx}-truth.csv')
        y = pd.read_csv(truth_path,
                        usecols=['hit_id', 'particle_id', 'weight'])
        hit_id = torch.from_numpy(y['hit_id'].values).to(torch.long).sub_(1)
        particle_id = torch.from_numpy(y['particle_id'].values).to(torch.long)
        particle_id = particle_id.unique(return_inverse=True)[1].sub_(1)
        weight = torch.from_numpy(y['weight'].values).to(torch.float)

        # Sort.
        _, perm = index_sort(particle_id * hit_id.size(0) + hit_id)
        hit_id = hit_id[perm]
        particle_id = particle_id[perm]
        weight = weight[perm]

        # Remove invalid particle ids.
        mask = particle_id >= 0
        hit_id = hit_id[mask]
        particle_id = particle_id[mask]
        weight = weight[mask]

        y_index = torch.stack([particle_id, hit_id], dim=0)

        return TrackingData(x=x, pos=pos, y_index=y_index, y_weight=weight)
pytorch/torch_geometric/datasets/pascal.py000066400000000000000000000254011444262304300213540ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from itertools import chain
from typing import Callable, List, Optional
from xml.dom import minidom

import numpy as np
import torch
import torch.nn.functional as F
from torch.utils.data import DataLoader

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
)


class PascalVOCKeypoints(InMemoryDataset):
    r"""The Pascal VOC 2011 dataset with Berkely annotations of keypoints from
    the `"Poselets: Body Part Detectors Trained Using 3D Human Pose
    Annotations" `_ paper, containing 0 to 23 keypoints per
    example over 20 categories.
    The dataset is pre-filtered to exclude difficult, occluded and truncated
    objects.
    The keypoints contain interpolated features from a pre-trained VGG16 model
    on ImageNet (:obj:`relu4_2` and :obj:`relu5_1`).

    Args:
        root (str): Root directory where the dataset should be saved.
        category (str): The category of the images (one of
            :obj:`"Aeroplane"`, :obj:`"Bicycle"`, :obj:`"Bird"`,
            :obj:`"Boat"`, :obj:`"Bottle"`, :obj:`"Bus"`, :obj:`"Car"`,
            :obj:`"Cat"`, :obj:`"Chair"`, :obj:`"Diningtable"`, :obj:`"Dog"`,
            :obj:`"Horse"`, :obj:`"Motorbike"`, :obj:`"Person"`,
            :obj:`"Pottedplant"`, :obj:`"Sheep"`, :obj:`"Sofa"`,
            :obj:`"Train"`, :obj:`"TVMonitor"`)
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
        device (str or torch.device, optional): The device to use for
            processing the raw data. If set to :obj:`None`, will utilize
            GPU-processing if available. (default: :obj:`None`)
    """
    image_url = ('http://host.robots.ox.ac.uk/pascal/VOC/voc2011/'
                 'VOCtrainval_25-May-2011.tar')
    annotation_url = ('https://www2.eecs.berkeley.edu/Research/Projects/CS/'
                      'vision/shape/poselets/voc2011_keypoints_Feb2012.tgz')
    # annotation_url = 'http://www.roemisch-drei.de/pascal_annotations.tar'
    # split_url = 'http://cvgl.stanford.edu/projects/ucn/voc2011_pairs.npz'
    split_url = ('https://github.com/Thinklab-SJTU/PCA-GM/raw/master/data/'
                 'PascalVOC/voc2011_pairs.npz')

    categories = [
        'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat',
        'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person',
        'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'
    ]

    batch_size = 32

    def __init__(
        self,
        root: str,
        category: str,
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
        device: Optional[str] = None,
    ):
        if device is None:
            device = 'cuda' if torch.cuda.is_available() else 'cpu'

        self.category = category.lower()
        assert self.category in self.categories
        self.device = device
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.category.capitalize(), 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        return ['images', 'annotations', 'splits.npz']

    @property
    def processed_file_names(self) -> List[str]:
        return ['training.pt', 'test.pt']

    def download(self):
        path = download_url(self.image_url, self.raw_dir)
        extract_tar(path, self.raw_dir, mode='r')
        os.unlink(path)
        image_path = osp.join(self.raw_dir, 'TrainVal', 'VOCdevkit', 'VOC2011')
        os.rename(image_path, osp.join(self.raw_dir, 'images'))
        shutil.rmtree(osp.join(self.raw_dir, 'TrainVal'))

        path = download_url(self.annotation_url, self.raw_dir)
        extract_tar(path, self.raw_dir, mode='r')
        os.unlink(path)

        path = download_url(self.split_url, self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'splits.npz'))

    def process(self):
        import torchvision.models as models
        import torchvision.transforms as T
        from PIL import Image

        splits = np.load(osp.join(self.raw_dir, 'splits.npz'),
                         allow_pickle=True)
        category_idx = self.categories.index(self.category)
        train_split = list(splits['train'])[category_idx]
        test_split = list(splits['test'])[category_idx]

        image_path = osp.join(self.raw_dir, 'images', 'JPEGImages')
        info_path = osp.join(self.raw_dir, 'images', 'Annotations')
        annotation_path = osp.join(self.raw_dir, 'annotations')

        labels = {}

        vgg16_outputs = []

        def hook(module, x, y):
            vgg16_outputs.append(y)

        vgg16 = models.vgg16(pretrained=True).to(self.device)
        vgg16.eval()
        vgg16.features[20].register_forward_hook(hook)  # relu4_2
        vgg16.features[25].register_forward_hook(hook)  # relu5_1

        transform = T.Compose([
            T.ToTensor(),
            T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        ])

        train_set, test_set = [], []
        for i, name in enumerate(chain(train_split, test_split)):
            filename = '_'.join(name.split('/')[1].split('_')[:-1])
            idx = int(name.split('_')[-1].split('.')[0]) - 1

            path = osp.join(info_path, f'{filename}.xml')
            obj = minidom.parse(path).getElementsByTagName('object')[idx]

            trunc = obj.getElementsByTagName('truncated')[0].firstChild.data
            occ = obj.getElementsByTagName('occluded')
            occ = '0' if len(occ) == 0 else occ[0].firstChild.data
            diff = obj.getElementsByTagName('difficult')[0].firstChild.data

            if bool(int(trunc)) or bool(int(occ)) or bool(int(diff)):
                continue

            if self.category == 'person' and int(filename[:4]) > 2008:
                continue

            xmin = float(obj.getElementsByTagName('xmin')[0].firstChild.data)
            xmax = float(obj.getElementsByTagName('xmax')[0].firstChild.data)
            ymin = float(obj.getElementsByTagName('ymin')[0].firstChild.data)
            ymax = float(obj.getElementsByTagName('ymax')[0].firstChild.data)
            box = (xmin, ymin, xmax, ymax)

            dom = minidom.parse(osp.join(annotation_path, name))
            keypoints = dom.getElementsByTagName('keypoint')
            poss, ys = [], []
            for keypoint in keypoints:
                label = keypoint.attributes['name'].value
                if label not in labels:
                    labels[label] = len(labels)
                ys.append(labels[label])
                x = float(keypoint.attributes['x'].value)
                y = float(keypoint.attributes['y'].value)
                poss += [x, y]
            y = torch.tensor(ys, dtype=torch.long)
            pos = torch.tensor(poss, dtype=torch.float).view(-1, 2)

            if pos.numel() == 0:
                continue  # These examples do not make any sense anyway...

            # Add a small offset to the bounding because some keypoints lay
            # outside the bounding box intervals.
            box = (min(pos[:, 0].min().floor().item(), box[0]) - 16,
                   min(pos[:, 1].min().floor().item(), box[1]) - 16,
                   max(pos[:, 0].max().ceil().item(), box[2]) + 16,
                   max(pos[:, 1].max().ceil().item(), box[3]) + 16)

            # Rescale keypoints.
            pos[:, 0] = (pos[:, 0] - box[0]) * 256.0 / (box[2] - box[0])
            pos[:, 1] = (pos[:, 1] - box[1]) * 256.0 / (box[3] - box[1])

            path = osp.join(image_path, f'{filename}.jpg')
            with open(path, 'rb') as f:
                img = Image.open(f).convert('RGB').crop(box)
                img = img.resize((256, 256), resample=Image.BICUBIC)

            img = transform(img)

            data = Data(img=img, pos=pos, y=y, name=filename)

            if i < len(train_split):
                train_set.append(data)
            else:
                test_set.append(data)

        data_list = list(chain(train_set, test_set))
        imgs = [data.img for data in data_list]
        loader = DataLoader(imgs, self.batch_size, shuffle=False)
        for i, batch_img in enumerate(loader):
            vgg16_outputs.clear()

            with torch.no_grad():
                vgg16(batch_img.to(self.device))

            out1 = F.interpolate(vgg16_outputs[0], (256, 256), mode='bilinear',
                                 align_corners=False)
            out2 = F.interpolate(vgg16_outputs[1], (256, 256), mode='bilinear',
                                 align_corners=False)

            for j in range(out1.size(0)):
                data = data_list[i * self.batch_size + j]
                idx = data.pos.round().long().clamp(0, 255)
                x_1 = out1[j, :, idx[:, 1], idx[:, 0]].to('cpu')
                x_2 = out2[j, :, idx[:, 1], idx[:, 0]].to('cpu')
                data.img = None
                data.x = torch.cat([x_1.t(), x_2.t()], dim=-1)
            del out1
            del out2

        if self.pre_filter is not None:
            train_set = [data for data in train_set if self.pre_filter(data)]
            test_set = [data for data in test_set if self.pre_filter(data)]

        if self.pre_transform is not None:
            train_set = [self.pre_transform(data) for data in train_set]
            test_set = [self.pre_transform(data) for data in test_set]

        torch.save(self.collate(train_set), self.processed_paths[0])
        torch.save(self.collate(test_set), self.processed_paths[1])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'category={self.category})')
pytorch/torch_geometric/datasets/pascal_pf.py000066400000000000000000000107721444262304300220460ustar00rootroot00000000000000import glob
import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class PascalPF(InMemoryDataset):
    r"""The Pascal-PF dataset from the `"Proposal Flow"
    `_ paper, containing 4 to 16 keypoints
    per example over 20 categories.

    Args:
        root (str): Root directory where the dataset should be saved.
        category (str): The category of the images (one of
            :obj:`"Aeroplane"`, :obj:`"Bicycle"`, :obj:`"Bird"`,
            :obj:`"Boat"`, :obj:`"Bottle"`, :obj:`"Bus"`, :obj:`"Car"`,
            :obj:`"Cat"`, :obj:`"Chair"`, :obj:`"Diningtable"`, :obj:`"Dog"`,
            :obj:`"Horse"`, :obj:`"Motorbike"`, :obj:`"Person"`,
            :obj:`"Pottedplant"`, :obj:`"Sheep"`, :obj:`"Sofa"`,
            :obj:`"Train"`, :obj:`"TVMonitor"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """
    url = ('https://www.di.ens.fr/willow/research/proposalflow/dataset/'
           'PF-dataset-PASCAL.zip')

    categories = [
        'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat',
        'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person',
        'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'
    ]

    def __init__(
        self,
        root: str,
        category: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.category = category.lower()
        assert self.category in self.categories
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])
        self.pairs = torch.load(self.processed_paths[1])

    @property
    def raw_file_names(self) -> List[str]:
        return ['Annotations', 'parsePascalVOC.mat']

    @property
    def processed_file_names(self) -> List[str]:
        return [f'{self.category}.pt', f'{self.category}_pairs.pt']

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        shutil.rmtree(self.raw_dir)
        os.rename(osp.join(self.root, 'PF-dataset-PASCAL'), self.raw_dir)

    def process(self):
        from scipy.io import loadmat

        path = osp.join(self.raw_dir, 'Annotations', self.category, '*.mat')
        filenames = glob.glob(path)

        names = []
        data_list = []
        for filename in filenames:
            name = filename.split(os.sep)[-1].split('.')[0]

            pos = torch.from_numpy(loadmat(filename)['kps']).to(torch.float)
            mask = ~torch.isnan(pos[:, 0])
            pos = pos[mask]

            # Normalize points to unit sphere.
            pos = pos - pos.mean(dim=0, keepdim=True)
            pos = pos / pos.norm(dim=1).max()

            y = mask.nonzero(as_tuple=False).flatten()

            data = Data(pos=pos, y=y, name=name)

            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)

            names.append(name)
            data_list.append(data)

        pairs = loadmat(osp.join(self.raw_dir, 'parsePascalVOC.mat'))
        pairs = pairs['PascalVOC']['pair'][0, 0][
            0, self.categories.index(self.category)]

        pairs = [(names.index(x[0][0]), names.index(x[1][0])) for x in pairs]

        torch.save(self.collate(data_list), self.processed_paths[0])
        torch.save(pairs, self.processed_paths[1])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'category={self.category})')
pytorch/torch_geometric/datasets/pcpnet_dataset.py000066400000000000000000000131241444262304300231060ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_txt_array


class PCPNetDataset(InMemoryDataset):
    r"""The PCPNet dataset from the `"PCPNet: Learning Local Shape Properties
    from Raw Point Clouds" `_ paper,
    consisting of 30 shapes, each given as a point cloud, densely sampled with
    100k points.
    For each shape, surface normals and local curvatures are given as node
    features.

    Args:
        root (str): Root directory where the dataset should be saved.
        category (str): The training set category (one of :obj:`"NoNoise"`,
            :obj:`"Noisy"`, :obj:`"VarDensity"`, :obj:`"NoisyAndVarDensity"`
            for :obj:`split="train"` or :obj:`split="val"`,
            or one of :obj:`"All"`, :obj:`"LowNoise"`, :obj:`"MedNoise"`,
            :obj:`"HighNoise", :obj:`"VarDensityStriped",
            :obj:`"VarDensityGradient"` for :obj:`split="test"`).
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset. (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'http://geometry.cs.ucl.ac.uk/projects/2018/pcpnet/pclouds.zip'

    category_files_train = {
        'NoNoise': 'trainingset_no_noise.txt',
        'Noisy': 'trainingset_whitenoise.txt',
        'VarDensity': 'trainingset_vardensity.txt',
        'NoisyAndVarDensity': 'trainingset_vardensity_whitenoise.txt'
    }

    category_files_val = {
        'NoNoise': 'validationset_no_noise.txt',
        'Noisy': 'validationset_whitenoise.txt',
        'VarDensity': 'validationset_vardensity.txt',
        'NoisyAndVarDensity': 'validationset_vardensity_whitenoise.txt'
    }

    category_files_test = {
        'All': 'testset_all.txt',
        'NoNoise': 'testset_no_noise.txt',
        'LowNoise': 'testset_low_noise.txt',
        'MedNoise': 'testset_med_noise.txt',
        'HighNoise': 'testset_high_noise.txt',
        'VarDensityStriped': 'testset_vardensity_striped.txt',
        'VarDensityGradient': 'testset_vardensity_gradient.txt'
    }

    def __init__(
        self,
        root: str,
        category: str,
        split: str = 'train',
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):

        assert split in ['train', 'val', 'test']

        if split == 'train':
            assert category in self.category_files_train.keys()
        elif split == 'val':
            assert category in self.category_files_val.keys()
        else:
            assert category in self.category_files_test.keys()

        self.category = category
        self.split = split

        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        if self.split == 'train':
            return self.category_files_train[self.category]
        elif self.split == 'val':
            return self.category_files_val[self.category]
        else:
            return self.category_files_test[self.category]

    @property
    def processed_file_names(self) -> str:
        return self.split + '_' + self.category + '.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        path_file = self.raw_paths
        with open(path_file[0], "r") as f:
            filenames = f.read().split('\n')[:-1]
        data_list = []
        for filename in filenames:
            pos_path = osp.join(self.raw_dir, filename + '.xyz')
            normal_path = osp.join(self.raw_dir, filename + '.normals')
            curv_path = osp.join(self.raw_dir, filename + '.curv')
            idx_path = osp.join(self.raw_dir, filename + '.pidx')
            pos = read_txt_array(pos_path)
            normals = read_txt_array(normal_path)
            curv = read_txt_array(curv_path)
            normals_and_curv = torch.cat([normals, curv], dim=1)
            test_idx = read_txt_array(idx_path, dtype=torch.long)
            data = Data(pos=pos, x=normals_and_curv)
            data.test_idx = test_idx
            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)
            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'category={self.category})')
pytorch/torch_geometric/datasets/planetoid.py000066400000000000000000000154771444262304300221040ustar00rootroot00000000000000import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import InMemoryDataset, download_url
from torch_geometric.io import read_planetoid_data


class Planetoid(InMemoryDataset):
    r"""The citation network datasets :obj:`"Cora"`, :obj:`"CiteSeer"` and
    :obj:`"PubMed"` from the `"Revisiting Semi-Supervised Learning with Graph
    Embeddings" `_ paper.
    Nodes represent documents and edges represent citation links.
    Training, validation and test splits are given by binary masks.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Cora"`, :obj:`"CiteSeer"`,
            :obj:`"PubMed"`).
        split (str, optional): The type of dataset split (:obj:`"public"`,
            :obj:`"full"`, :obj:`"geom-gcn"`, :obj:`"random"`).
            If set to :obj:`"public"`, the split will be the public fixed split
            from the `"Revisiting Semi-Supervised Learning with Graph
            Embeddings" `_ paper.
            If set to :obj:`"full"`, all nodes except those in the validation
            and test sets will be used for training (as in the
            `"FastGCN: Fast Learning with Graph Convolutional Networks via
            Importance Sampling" `_ paper).
            If set to :obj:`"geom-gcn"`, the 10 public fixed splits from the
            `"Geom-GCN: Geometric Graph Convolutional Networks"
            `_ paper are given.
            If set to :obj:`"random"`, train, validation, and test sets will be
            randomly generated, according to :obj:`num_train_per_class`,
            :obj:`num_val` and :obj:`num_test`. (default: :obj:`"public"`)
        num_train_per_class (int, optional): The number of training samples
            per class in case of :obj:`"random"` split. (default: :obj:`20`)
        num_val (int, optional): The number of validation samples in case of
            :obj:`"random"` split. (default: :obj:`500`)
        num_test (int, optional): The number of test samples in case of
            :obj:`"random"` split. (default: :obj:`1000`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Cora
          - 2,708
          - 10,556
          - 1,433
          - 7
        * - CiteSeer
          - 3,327
          - 9,104
          - 3,703
          - 6
        * - PubMed
          - 19,717
          - 88,648
          - 500
          - 3
    """
    url = 'https://github.com/kimiyoung/planetoid/raw/master/data'
    geom_gcn_url = ('https://raw.githubusercontent.com/graphdml-uiuc-jlu/'
                    'geom-gcn/master')

    def __init__(self, root: str, name: str, split: str = "public",
                 num_train_per_class: int = 20, num_val: int = 500,
                 num_test: int = 1000, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name

        self.split = split.lower()
        assert self.split in ['public', 'full', 'geom-gcn', 'random']

        super().__init__(root, transform, pre_transform)
        self.load(self.processed_paths[0])

        if split == 'full':
            data = self.get(0)
            data.train_mask.fill_(True)
            data.train_mask[data.val_mask | data.test_mask] = False
            self.data, self.slices = self.collate([data])

        elif split == 'random':
            data = self.get(0)
            data.train_mask.fill_(False)
            for c in range(self.num_classes):
                idx = (data.y == c).nonzero(as_tuple=False).view(-1)
                idx = idx[torch.randperm(idx.size(0))[:num_train_per_class]]
                data.train_mask[idx] = True

            remaining = (~data.train_mask).nonzero(as_tuple=False).view(-1)
            remaining = remaining[torch.randperm(remaining.size(0))]

            data.val_mask.fill_(False)
            data.val_mask[remaining[:num_val]] = True

            data.test_mask.fill_(False)
            data.test_mask[remaining[num_val:num_val + num_test]] = True

            self.data, self.slices = self.collate([data])

    @property
    def raw_dir(self) -> str:
        if self.split == 'geom-gcn':
            return osp.join(self.root, self.name, 'geom-gcn', 'raw')
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        if self.split == 'geom-gcn':
            return osp.join(self.root, self.name, 'geom-gcn', 'processed')
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        names = ['x', 'tx', 'allx', 'y', 'ty', 'ally', 'graph', 'test.index']
        return [f'ind.{self.name.lower()}.{name}' for name in names]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for name in self.raw_file_names:
            download_url(f'{self.url}/{name}', self.raw_dir)
        if self.split == 'geom-gcn':
            for i in range(10):
                url = f'{self.geom_gcn_url}/splits/{self.name.lower()}'
                download_url(f'{url}_split_0.6_0.2_{i}.npz', self.raw_dir)

    def process(self):
        data = read_planetoid_data(self.raw_dir, self.name)

        if self.split == 'geom-gcn':
            train_masks, val_masks, test_masks = [], [], []
            for i in range(10):
                name = f'{self.name.lower()}_split_0.6_0.2_{i}.npz'
                splits = np.load(osp.join(self.raw_dir, name))
                train_masks.append(torch.from_numpy(splits['train_mask']))
                val_masks.append(torch.from_numpy(splits['val_mask']))
                test_masks.append(torch.from_numpy(splits['test_mask']))
            data.train_mask = torch.stack(train_masks, dim=1)
            data.val_mask = torch.stack(val_masks, dim=1)
            data.test_mask = torch.stack(test_masks, dim=1)

        data = data if self.pre_transform is None else self.pre_transform(data)
        self.save([data], self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name}()'
pytorch/torch_geometric/datasets/polblogs.py000066400000000000000000000054321444262304300217340ustar00rootroot00000000000000import os
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_tar,
)


class PolBlogs(InMemoryDataset):
    r"""The Political Blogs dataset from the `"The Political Blogosphere and
    the 2004 US Election: Divided they Blog"
    `_ paper.

    :class:`Polblogs` is a graph with 1,490 vertices (representing political
    blogs) and 19,025 edges (links between blogs).
    The links are automatically extracted from a crawl of the front page of the
    blog.
    Each vertex receives a label indicating the political leaning of the blog:
    liberal or conservative.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 1,490
          - 19,025
          - 0
          - 2
    """

    url = 'https://netset.telecom-paris.fr/datasets/polblogs.tar.gz'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['adjacency.tsv', 'labels.tsv']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_tar(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        import pandas as pd

        edge_index = pd.read_csv(self.raw_paths[0], header=None, sep='\t',
                                 usecols=[0, 1])
        edge_index = torch.from_numpy(edge_index.values).t().contiguous()

        y = pd.read_csv(self.raw_paths[1], header=None, sep='\t')
        y = torch.from_numpy(y.values).view(-1)

        data = Data(edge_index=edge_index, y=y, num_nodes=y.size(0))

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/ppi.py000066400000000000000000000114021444262304300206750ustar00rootroot00000000000000import json
import os
import os.path as osp
from itertools import product
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.utils import remove_self_loops


class PPI(InMemoryDataset):
    r"""The protein-protein interaction networks from the `"Predicting
    Multicellular Function through Multi-layer Tissue Networks"
    `_ paper, containing positional gene
    sets, motif gene sets and immunological signatures as features (50 in
    total) and gene ontology sets as labels (121 in total).

    Args:
        root (str): Root directory where the dataset should be saved.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset. (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #tasks
        * - 20
          - ~2,245.3
          - ~61,318.4
          - 50
          - 121
    """

    url = 'https://data.dgl.ai/dataset/ppi.zip'

    def __init__(
        self,
        root: str,
        split: str = 'train',
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):

        assert split in ['train', 'val', 'test']

        super().__init__(root, transform, pre_transform, pre_filter)

        if split == 'train':
            self.data, self.slices = torch.load(self.processed_paths[0])
        elif split == 'val':
            self.data, self.slices = torch.load(self.processed_paths[1])
        elif split == 'test':
            self.data, self.slices = torch.load(self.processed_paths[2])

    @property
    def raw_file_names(self) -> List[str]:
        splits = ['train', 'valid', 'test']
        files = ['feats.npy', 'graph_id.npy', 'graph.json', 'labels.npy']
        return [f'{split}_{name}' for split, name in product(splits, files)]

    @property
    def processed_file_names(self) -> str:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        import networkx as nx
        from networkx.readwrite import json_graph

        for s, split in enumerate(['train', 'valid', 'test']):
            path = osp.join(self.raw_dir, f'{split}_graph.json')
            with open(path, 'r') as f:
                G = nx.DiGraph(json_graph.node_link_graph(json.load(f)))

            x = np.load(osp.join(self.raw_dir, f'{split}_feats.npy'))
            x = torch.from_numpy(x).to(torch.float)

            y = np.load(osp.join(self.raw_dir, f'{split}_labels.npy'))
            y = torch.from_numpy(y).to(torch.float)

            data_list = []
            path = osp.join(self.raw_dir, f'{split}_graph_id.npy')
            idx = torch.from_numpy(np.load(path)).to(torch.long)
            idx = idx - idx.min()

            for i in range(idx.max().item() + 1):
                mask = idx == i

                G_s = G.subgraph(
                    mask.nonzero(as_tuple=False).view(-1).tolist())
                edge_index = torch.tensor(list(G_s.edges)).t().contiguous()
                edge_index = edge_index - edge_index.min()
                edge_index, _ = remove_self_loops(edge_index)

                data = Data(edge_index=edge_index, x=x[mask], y=y[mask])

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)
            torch.save(self.collate(data_list), self.processed_paths[s])
pytorch/torch_geometric/datasets/qm7.py000066400000000000000000000061561444262304300206230ustar00rootroot00000000000000from typing import Callable, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class QM7b(InMemoryDataset):
    r"""The QM7b dataset from the `"MoleculeNet: A Benchmark for Molecular
    Machine Learning" `_ paper, consisting of
    7,211 molecules with 14 regression targets.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #tasks
        * - 7,211
          - ~15.4
          - ~245.0
          - 0
          - 14
    """

    url = 'https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/qm7b.mat'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'qm7b.mat'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(self.url, self.raw_dir)

    def process(self):
        from scipy.io import loadmat

        data = loadmat(self.raw_paths[0])
        coulomb_matrix = torch.from_numpy(data['X'])
        target = torch.from_numpy(data['T']).to(torch.float)

        data_list = []
        for i in range(target.shape[0]):
            edge_index = coulomb_matrix[i].nonzero(
                as_tuple=False).t().contiguous()
            edge_attr = coulomb_matrix[i, edge_index[0], edge_index[1]]
            y = target[i].view(1, -1)
            data = Data(edge_index=edge_index, edge_attr=edge_attr, y=y)
            data.num_nodes = edge_index.max().item() + 1
            data_list.append(data)

        if self.pre_filter is not None:
            data_list = [d for d in data_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(d) for d in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])
pytorch/torch_geometric/datasets/qm9.py000066400000000000000000000406551444262304300206270ustar00rootroot00000000000000import os
import os.path as osp
import sys
from typing import Callable, List, Optional

import torch
from tqdm import tqdm

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.utils import one_hot, scatter

HAR2EV = 27.211386246
KCALMOL2EV = 0.04336414

conversion = torch.tensor([
    1., 1., HAR2EV, HAR2EV, HAR2EV, 1., HAR2EV, HAR2EV, HAR2EV, HAR2EV, HAR2EV,
    1., KCALMOL2EV, KCALMOL2EV, KCALMOL2EV, KCALMOL2EV, 1., 1., 1.
])

atomrefs = {
    6: [0., 0., 0., 0., 0.],
    7: [
        -13.61312172, -1029.86312267, -1485.30251237, -2042.61123593,
        -2713.48485589
    ],
    8: [
        -13.5745904, -1029.82456413, -1485.26398105, -2042.5727046,
        -2713.44632457
    ],
    9: [
        -13.54887564, -1029.79887659, -1485.2382935, -2042.54701705,
        -2713.42063702
    ],
    10: [
        -13.90303183, -1030.25891228, -1485.71166277, -2043.01812778,
        -2713.88796536
    ],
    11: [0., 0., 0., 0., 0.],
}


class QM9(InMemoryDataset):
    r"""The QM9 dataset from the `"MoleculeNet: A Benchmark for Molecular
    Machine Learning" `_ paper, consisting of
    about 130,000 molecules with 19 regression targets.
    Each molecule includes complete spatial information for the single low
    energy conformation of the atoms in the molecule.
    In addition, we provide the atom features from the `"Neural Message
    Passing for Quantum Chemistry" `_ paper.

    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | Target | Property                         | Description                                                                       | Unit                                        |
    +========+==================================+===================================================================================+=============================================+
    | 0      | :math:`\mu`                      | Dipole moment                                                                     | :math:`\textrm{D}`                          |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 1      | :math:`\alpha`                   | Isotropic polarizability                                                          | :math:`{a_0}^3`                             |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 2      | :math:`\epsilon_{\textrm{HOMO}}` | Highest occupied molecular orbital energy                                         | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 3      | :math:`\epsilon_{\textrm{LUMO}}` | Lowest unoccupied molecular orbital energy                                        | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 4      | :math:`\Delta \epsilon`          | Gap between :math:`\epsilon_{\textrm{HOMO}}` and :math:`\epsilon_{\textrm{LUMO}}` | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 5      | :math:`\langle R^2 \rangle`      | Electronic spatial extent                                                         | :math:`{a_0}^2`                             |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 6      | :math:`\textrm{ZPVE}`            | Zero point vibrational energy                                                     | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 7      | :math:`U_0`                      | Internal energy at 0K                                                             | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 8      | :math:`U`                        | Internal energy at 298.15K                                                        | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 9      | :math:`H`                        | Enthalpy at 298.15K                                                               | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 10     | :math:`G`                        | Free energy at 298.15K                                                            | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 11     | :math:`c_{\textrm{v}}`           | Heat capavity at 298.15K                                                          | :math:`\frac{\textrm{cal}}{\textrm{mol K}}` |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 12     | :math:`U_0^{\textrm{ATOM}}`      | Atomization energy at 0K                                                          | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 13     | :math:`U^{\textrm{ATOM}}`        | Atomization energy at 298.15K                                                     | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 14     | :math:`H^{\textrm{ATOM}}`        | Atomization enthalpy at 298.15K                                                   | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 15     | :math:`G^{\textrm{ATOM}}`        | Atomization free energy at 298.15K                                                | :math:`\textrm{eV}`                         |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 16     | :math:`A`                        | Rotational constant                                                               | :math:`\textrm{GHz}`                        |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 17     | :math:`B`                        | Rotational constant                                                               | :math:`\textrm{GHz}`                        |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+
    | 18     | :math:`C`                        | Rotational constant                                                               | :math:`\textrm{GHz}`                        |
    +--------+----------------------------------+-----------------------------------------------------------------------------------+---------------------------------------------+

    .. note::

        We also provide a pre-processed version of the dataset in case
        :class:`rdkit` is not installed. The pre-processed version matches with
        the manually processed version as outlined in :meth:`process`.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #tasks
        * - 130,831
          - ~18.0
          - ~37.3
          - 11
          - 19
    """  # noqa: E501

    raw_url = ('https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/'
               'molnet_publish/qm9.zip')
    raw_url2 = 'https://ndownloader.figshare.com/files/3195404'
    processed_url = 'https://data.pyg.org/datasets/qm9_v3.zip'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None):
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    def mean(self, target: int) -> float:
        y = torch.cat([self.get(i).y for i in range(len(self))], dim=0)
        return float(y[:, target].mean())

    def std(self, target: int) -> float:
        y = torch.cat([self.get(i).y for i in range(len(self))], dim=0)
        return float(y[:, target].std())

    def atomref(self, target) -> Optional[torch.Tensor]:
        if target in atomrefs:
            out = torch.zeros(100)
            out[torch.tensor([1, 6, 7, 8, 9])] = torch.tensor(atomrefs[target])
            return out.view(-1, 1)
        return None

    @property
    def raw_file_names(self) -> List[str]:
        try:
            import rdkit  # noqa
            return ['gdb9.sdf', 'gdb9.sdf.csv', 'uncharacterized.txt']
        except ImportError:
            return ['qm9_v3.pt']

    @property
    def processed_file_names(self) -> str:
        return 'data_v3.pt'

    def download(self):
        try:
            import rdkit  # noqa
            file_path = download_url(self.raw_url, self.raw_dir)
            extract_zip(file_path, self.raw_dir)
            os.unlink(file_path)

            file_path = download_url(self.raw_url2, self.raw_dir)
            os.rename(osp.join(self.raw_dir, '3195404'),
                      osp.join(self.raw_dir, 'uncharacterized.txt'))
        except ImportError:
            path = download_url(self.processed_url, self.raw_dir)
            extract_zip(path, self.raw_dir)
            os.unlink(path)

    def process(self):
        try:
            import rdkit
            from rdkit import Chem, RDLogger
            from rdkit.Chem.rdchem import BondType as BT
            from rdkit.Chem.rdchem import HybridizationType
            RDLogger.DisableLog('rdApp.*')

        except ImportError:
            rdkit = None

        if rdkit is None:
            print(("Using a pre-processed version of the dataset. Please "
                   "install 'rdkit' to alternatively process the raw data."),
                  file=sys.stderr)

            data_list = torch.load(self.raw_paths[0])
            data_list = [Data(**data_dict) for data_dict in data_list]

            if self.pre_filter is not None:
                data_list = [d for d in data_list if self.pre_filter(d)]

            if self.pre_transform is not None:
                data_list = [self.pre_transform(d) for d in data_list]

            torch.save(self.collate(data_list), self.processed_paths[0])
            return

        types = {'H': 0, 'C': 1, 'N': 2, 'O': 3, 'F': 4}
        bonds = {BT.SINGLE: 0, BT.DOUBLE: 1, BT.TRIPLE: 2, BT.AROMATIC: 3}

        with open(self.raw_paths[1], 'r') as f:
            target = f.read().split('\n')[1:-1]
            target = [[float(x) for x in line.split(',')[1:20]]
                      for line in target]
            target = torch.tensor(target, dtype=torch.float)
            target = torch.cat([target[:, 3:], target[:, :3]], dim=-1)
            target = target * conversion.view(1, -1)

        with open(self.raw_paths[2], 'r') as f:
            skip = [int(x.split()[0]) - 1 for x in f.read().split('\n')[9:-2]]

        suppl = Chem.SDMolSupplier(self.raw_paths[0], removeHs=False,
                                   sanitize=False)

        data_list = []
        for i, mol in enumerate(tqdm(suppl)):
            if i in skip:
                continue

            N = mol.GetNumAtoms()

            conf = mol.GetConformer()
            pos = conf.GetPositions()
            pos = torch.tensor(pos, dtype=torch.float)

            type_idx = []
            atomic_number = []
            aromatic = []
            sp = []
            sp2 = []
            sp3 = []
            num_hs = []
            for atom in mol.GetAtoms():
                type_idx.append(types[atom.GetSymbol()])
                atomic_number.append(atom.GetAtomicNum())
                aromatic.append(1 if atom.GetIsAromatic() else 0)
                hybridization = atom.GetHybridization()
                sp.append(1 if hybridization == HybridizationType.SP else 0)
                sp2.append(1 if hybridization == HybridizationType.SP2 else 0)
                sp3.append(1 if hybridization == HybridizationType.SP3 else 0)

            z = torch.tensor(atomic_number, dtype=torch.long)

            row, col, edge_type = [], [], []
            for bond in mol.GetBonds():
                start, end = bond.GetBeginAtomIdx(), bond.GetEndAtomIdx()
                row += [start, end]
                col += [end, start]
                edge_type += 2 * [bonds[bond.GetBondType()]]

            edge_index = torch.tensor([row, col], dtype=torch.long)
            edge_type = torch.tensor(edge_type, dtype=torch.long)
            edge_attr = one_hot(edge_type, num_classes=len(bonds))

            perm = (edge_index[0] * N + edge_index[1]).argsort()
            edge_index = edge_index[:, perm]
            edge_type = edge_type[perm]
            edge_attr = edge_attr[perm]

            row, col = edge_index
            hs = (z == 1).to(torch.float)
            num_hs = scatter(hs[row], col, dim_size=N, reduce='sum').tolist()

            x1 = one_hot(torch.tensor(type_idx), num_classes=len(types))
            x2 = torch.tensor([atomic_number, aromatic, sp, sp2, sp3, num_hs],
                              dtype=torch.float).t().contiguous()
            x = torch.cat([x1, x2], dim=-1)

            y = target[i].unsqueeze(0)
            name = mol.GetProp('_Name')

            data = Data(x=x, z=z, pos=pos, edge_index=edge_index,
                        edge_attr=edge_attr, y=y, name=name, idx=i)

            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)

            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])
pytorch/torch_geometric/datasets/reddit.py000066400000000000000000000055751444262304300213760ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.utils import coalesce


class Reddit(InMemoryDataset):
    r"""The Reddit dataset from the `"Inductive Representation Learning on
    Large Graphs" `_ paper, containing
    Reddit posts belonging to different communities.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 232,965
          - 114,615,892
          - 602
          - 41
    """

    url = 'https://data.dgl.ai/dataset/reddit.zip'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['reddit_data.npz', 'reddit_graph.npz']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        data = np.load(osp.join(self.raw_dir, 'reddit_data.npz'))
        x = torch.from_numpy(data['feature']).to(torch.float)
        y = torch.from_numpy(data['label']).to(torch.long)
        split = torch.from_numpy(data['node_types'])

        adj = sp.load_npz(osp.join(self.raw_dir, 'reddit_graph.npz'))
        row = torch.from_numpy(adj.row).to(torch.long)
        col = torch.from_numpy(adj.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)
        edge_index = coalesce(edge_index, num_nodes=x.size(0))

        data = Data(x=x, edge_index=edge_index, y=y)
        data.train_mask = split == 1
        data.val_mask = split == 2
        data.test_mask = split == 3

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/reddit2.py000066400000000000000000000105271444262304300214510ustar00rootroot00000000000000import json
import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class Reddit2(InMemoryDataset):
    r"""The Reddit dataset from the `"GraphSAINT: Graph Sampling Based
    Inductive Learning Method" `_ paper,
    containing Reddit posts belonging to different communities.

    .. note::

        This is a sparser version of the original
        :obj:`~torch_geometric.datasets.Reddit` dataset (~23M edges instead of
        ~114M edges), and is used in papers such as
        `SGC `_ and
        `GraphSAINT `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 232,965
          - 23,213,838
          - 602
          - 41
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    adj_full_id = '1sncK996BM5lpuDf75lDFqCiDZyErc1c2'
    feats_id = '1ZsHaJ0ussP1W722krmEIp_8pwKAoi5b3'
    class_map_id = '1JF3Pjv9OboMNYs2aXRQGbJbc4t_nDd5u'
    role_id = '1nJIKd77lcAGU4j-kVNx_AIGEkveIKz3A'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['adj_full.npz', 'feats.npy', 'class_map.json', 'role.json']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url.format(self.adj_full_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'adj_full.npz'))

        path = download_url(self.url.format(self.feats_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'feats.npy'))

        path = download_url(self.url.format(self.class_map_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'class_map.json'))

        path = download_url(self.url.format(self.role_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'role.json'))

    def process(self):
        f = np.load(osp.join(self.raw_dir, 'adj_full.npz'))
        adj = sp.csr_matrix((f['data'], f['indices'], f['indptr']), f['shape'])
        adj = adj.tocoo()
        row = torch.from_numpy(adj.row).to(torch.long)
        col = torch.from_numpy(adj.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        x = np.load(osp.join(self.raw_dir, 'feats.npy'))
        x = torch.from_numpy(x).to(torch.float)

        ys = [-1] * x.size(0)
        with open(osp.join(self.raw_dir, 'class_map.json')) as f:
            class_map = json.load(f)
            for key, item in class_map.items():
                ys[int(key)] = item
        y = torch.tensor(ys)

        with open(osp.join(self.raw_dir, 'role.json')) as f:
            role = json.load(f)

        train_mask = torch.zeros(x.size(0), dtype=torch.bool)
        train_mask[torch.tensor(role['tr'])] = True

        val_mask = torch.zeros(x.size(0), dtype=torch.bool)
        val_mask[torch.tensor(role['va'])] = True

        test_mask = torch.zeros(x.size(0), dtype=torch.bool)
        test_mask[torch.tensor(role['te'])] = True

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/rel_link_pred_dataset.py000066400000000000000000000103711444262304300244270ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class RelLinkPredDataset(InMemoryDataset):
    r"""The relational link prediction datasets from the
    `"Modeling Relational Data with Graph Convolutional Networks"
    `_ paper.
    Training and test splits are given by sets of triplets.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"FB15k-237"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #classes
        * - 14,541
          - 544,230
          - 0
          - 0
    """

    urls = {
        'FB15k-237': ('https://raw.githubusercontent.com/MichSchli/'
                      'RelationPrediction/master/data/FB-Toutanova')
    }

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name
        assert name in ['FB15k-237']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def num_relations(self) -> int:
        return int(self._data.edge_type.max()) + 1

    @property
    def raw_dir(self) -> str:
        return os.path.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return os.path.join(self.root, self.name, 'processed')

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'entities.dict', 'relations.dict', 'test.txt', 'train.txt',
            'valid.txt'
        ]

    def download(self):
        for file_name in self.raw_file_names:
            download_url(f'{self.urls[self.name]}/{file_name}', self.raw_dir)

    def process(self):
        with open(osp.join(self.raw_dir, 'entities.dict'), 'r') as f:
            lines = [row.split('\t') for row in f.read().split('\n')[:-1]]
            entities_dict = {key: int(value) for value, key in lines}

        with open(osp.join(self.raw_dir, 'relations.dict'), 'r') as f:
            lines = [row.split('\t') for row in f.read().split('\n')[:-1]]
            relations_dict = {key: int(value) for value, key in lines}

        kwargs = {}
        for split in ['train', 'valid', 'test']:
            with open(osp.join(self.raw_dir, f'{split}.txt'), 'r') as f:
                lines = [row.split('\t') for row in f.read().split('\n')[:-1]]
                src = [entities_dict[row[0]] for row in lines]
                rel = [relations_dict[row[1]] for row in lines]
                dst = [entities_dict[row[2]] for row in lines]
                kwargs[f'{split}_edge_index'] = torch.tensor([src, dst])
                kwargs[f'{split}_edge_type'] = torch.tensor(rel)

        # For message passing, we add reverse edges and types to the graph:
        row, col = kwargs['train_edge_index']
        edge_type = kwargs['train_edge_type']
        row, col = torch.cat([row, col], dim=0), torch.cat([col, row], dim=0)
        edge_index = torch.stack([row, col], dim=0)
        edge_type = torch.cat([edge_type, edge_type + len(relations_dict)])

        data = Data(num_nodes=len(entities_dict), edge_index=edge_index,
                    edge_type=edge_type, **kwargs)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save((self.collate([data])), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name}()'
pytorch/torch_geometric/datasets/s3dis.py000066400000000000000000000101151444262304300211320ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class S3DIS(InMemoryDataset):
    r"""The (pre-processed) Stanford Large-Scale 3D Indoor Spaces dataset from
    the `"3D Semantic Parsing of Large-Scale Indoor Spaces"
    `_
    paper, containing point clouds of six large-scale indoor parts in three
    buildings with 12 semantic elements (and one clutter class).

    Args:
        root (str): Root directory where the dataset should be saved.
        test_area (int, optional): Which area to use for testing (1-6).
            (default: :obj:`6`)
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = ('https://shapenet.cs.stanford.edu/media/'
           'indoor3d_sem_seg_hdf5_data.zip')

    def __init__(
        self,
        root: str,
        test_area: int = 6,
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        assert test_area >= 1 and test_area <= 6
        self.test_area = test_area
        super().__init__(root, transform, pre_transform, pre_filter)
        path = self.processed_paths[0] if train else self.processed_paths[1]
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        return ['all_files.txt', 'room_filelist.txt']

    @property
    def processed_file_names(self) -> List[str]:
        return [f'{split}_{self.test_area}.pt' for split in ['train', 'test']]

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.unlink(path)
        shutil.rmtree(self.raw_dir)
        name = self.url.split('/')[-1].split('.')[0]
        os.rename(osp.join(self.root, name), self.raw_dir)

    def process(self):
        import h5py

        with open(self.raw_paths[0], 'r') as f:
            filenames = [x.split('/')[-1] for x in f.read().split('\n')[:-1]]

        with open(self.raw_paths[1], 'r') as f:
            rooms = f.read().split('\n')[:-1]

        xs, ys = [], []
        for filename in filenames:
            f = h5py.File(osp.join(self.raw_dir, filename))
            xs += torch.from_numpy(f['data'][:]).unbind(0)
            ys += torch.from_numpy(f['label'][:]).to(torch.long).unbind(0)

        test_area = f'Area_{self.test_area}'
        train_data_list, test_data_list = [], []
        for i, (x, y) in enumerate(zip(xs, ys)):
            data = Data(pos=x[:, :3], x=x[:, 3:], y=y)

            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)

            if test_area not in rooms[i]:
                train_data_list.append(data)
            else:
                test_data_list.append(data)

        torch.save(self.collate(train_data_list), self.processed_paths[0])
        torch.save(self.collate(test_data_list), self.processed_paths[1])
pytorch/torch_geometric/datasets/sbm_dataset.py000066400000000000000000000177141444262304300224070ustar00rootroot00000000000000import os
from typing import Callable, List, Optional, Union

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data import Data, InMemoryDataset
from torch_geometric.utils import stochastic_blockmodel_graph


class StochasticBlockModelDataset(InMemoryDataset):
    r"""A synthetic graph dataset generated by the stochastic block model.
    The node features of each block are sampled from normal distributions where
    the centers of clusters are vertices of a hypercube, as computed by the
    :meth:`sklearn.datasets.make_classification` method.

    Args:
        root (str): Root directory where the dataset should be saved.
        block_sizes ([int] or LongTensor): The sizes of blocks.
        edge_probs ([[float]] or FloatTensor): The density of edges going from
            each block to each other block. Must be symmetric if the graph is
            undirected.
        num_channels (int, optional): The number of node features. If given
            as :obj:`None`, node features are not generated.
            (default: :obj:`None`)
        is_undirected (bool, optional): Whether the graph to generate is
            undirected. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes
            in an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed
            before being saved to disk. (default: :obj:`None`)
        **kwargs (optional): The keyword arguments that are passed down to the
            :meth:`sklearn.datasets.make_classification` method for drawing
            node features.
    """
    def __init__(
        self,
        root: str,
        block_sizes: Union[List[int], Tensor],
        edge_probs: Union[List[List[float]], Tensor],
        num_channels: Optional[int] = None,
        is_undirected: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        **kwargs,
    ):
        if not isinstance(block_sizes, torch.Tensor):
            block_sizes = torch.tensor(block_sizes, dtype=torch.long)
        if not isinstance(edge_probs, torch.Tensor):
            edge_probs = torch.tensor(edge_probs, dtype=torch.float)

        self.block_sizes = block_sizes
        self.edge_probs = edge_probs
        self.num_channels = num_channels
        self.is_undirected = is_undirected

        self.kwargs = {
            'n_informative': num_channels,
            'n_redundant': 0,
            'flip_y': 0.0,
            'shuffle': False,
        }
        self.kwargs.update(kwargs)

        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def processed_dir(self) -> str:
        return os.path.join(self.root, self.__class__.__name__, 'processed')

    @property
    def processed_file_names(self) -> str:
        block_sizes = self.block_sizes.view(-1).tolist()
        hash1 = '-'.join([f'{x:.1f}' for x in block_sizes])

        edge_probs = self.edge_probs.view(-1).tolist()
        hash2 = '-'.join([f'{x:.1f}' for x in edge_probs])

        return f'data_{self.num_channels}_{hash1}_{hash2}.pt'

    def process(self):
        from sklearn.datasets import make_classification

        edge_index = stochastic_blockmodel_graph(
            self.block_sizes, self.edge_probs, directed=not self.is_undirected)

        num_samples = int(self.block_sizes.sum())
        num_classes = self.block_sizes.size(0)

        x = None
        if self.num_channels is not None:
            x, y_not_sorted = make_classification(
                n_samples=num_samples,
                n_features=self.num_channels,
                n_classes=num_classes,
                weights=self.block_sizes / num_samples,
                **self.kwargs,
            )
            x = x[np.argsort(y_not_sorted)]
            x = torch.from_numpy(x).to(torch.float)

        y = torch.arange(num_classes).repeat_interleave(self.block_sizes)

        data = Data(x=x, edge_index=edge_index, y=y)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])


class RandomPartitionGraphDataset(StochasticBlockModelDataset):
    r"""The random partition graph dataset from the `"How to Find Your
    Friendly Neighborhood: Graph Attention Design with Self-Supervision"
    `_ paper.
    This is a synthetic graph of communities controlled by the node homophily
    and the average degree, and each community is considered as a class.
    The node features are sampled from normal distributions where the centers
    of clusters are vertices of a hypercube, as computed by the
    :meth:`sklearn.datasets.make_classification` method.

    Args:
        root (str): Root directory where the dataset should be saved.
        num_classes (int): The number of classes.
        num_nodes_per_class (int): The number of nodes per class.
        node_homophily_ratio (float): The degree of node homophily.
        average_degree (float): The average degree of the graph.
        num_channels (int, optional): The number of node features. If given
            as :obj:`None`, node features are not generated.
            (default: :obj:`None`)
        is_undirected (bool, optional): Whether the graph to generate is
            undirected. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes
            in an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        **kwargs (optional): The keyword arguments that are passed down
            to :meth:`sklearn.datasets.make_classification` method in
            drawing node features.
    """
    def __init__(self, root, num_classes: int, num_nodes_per_class: int,
                 node_homophily_ratio: float, average_degree: float,
                 num_channels: Optional[int] = None,
                 is_undirected: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None, **kwargs):

        self._num_classes = num_classes
        self.num_nodes_per_class = num_nodes_per_class
        self.node_homophily_ratio = node_homophily_ratio
        self.average_degree = average_degree

        # (p_in + (C - 1) * p_out) / C = |E|/|V|^2
        # i.e., p_in + (C - 1) * p_out = average_degree / num_nodes_per_class
        ec_over_v2 = average_degree / num_nodes_per_class
        p_in = node_homophily_ratio * ec_over_v2
        p_out = (ec_over_v2 - p_in) / (num_classes - 1)

        block_sizes = [num_nodes_per_class for _ in range(num_classes)]
        edge_probs = [[p_out for _ in range(num_classes)]
                      for _ in range(num_classes)]
        for r in range(num_classes):
            edge_probs[r][r] = p_in

        super().__init__(root, block_sizes, edge_probs, num_channels,
                         is_undirected, transform, pre_transform, **kwargs)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def processed_file_names(self) -> str:
        return (f'data_{self.num_channels}_{self._num_classes}_'
                f'{self.num_nodes_per_class}_{self.node_homophily_ratio:.1f}_'
                f'{self.average_degree:.1f}.pt')

    def process(self):
        return super().process()
pytorch/torch_geometric/datasets/shapenet.py000066400000000000000000000176301444262304300217250ustar00rootroot00000000000000import json
import os
import os.path as osp
import shutil
from typing import Callable, List, Optional, Union

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_txt_array


class ShapeNet(InMemoryDataset):
    r"""The ShapeNet part level segmentation dataset from the `"A Scalable
    Active Framework for Region Annotation in 3D Shape Collections"
    `_
    paper, containing about 17,000 3D shape point clouds from 16 shape
    categories.
    Each category is annotated with 2 to 6 parts.

    Args:
        root (str): Root directory where the dataset should be saved.
        categories (str or [str], optional): The category of the CAD models
            (one or a combination of :obj:`"Airplane"`, :obj:`"Bag"`,
            :obj:`"Cap"`, :obj:`"Car"`, :obj:`"Chair"`, :obj:`"Earphone"`,
            :obj:`"Guitar"`, :obj:`"Knife"`, :obj:`"Lamp"`, :obj:`"Laptop"`,
            :obj:`"Motorbike"`, :obj:`"Mug"`, :obj:`"Pistol"`, :obj:`"Rocket"`,
            :obj:`"Skateboard"`, :obj:`"Table"`).
            Can be explicitly set to :obj:`None` to load all categories.
            (default: :obj:`None`)
        include_normals (bool, optional): If set to :obj:`False`, will not
            include normal vectors as input features to :obj:`data.x`.
            As a result, :obj:`data.x` will be :obj:`None`.
            (default: :obj:`True`)
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"trainval"`, loads the training and validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"trainval"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - 16,881
          - ~2,616.2
          - 0
          - 3
          - 50
    """

    url = ('https://shapenet.cs.stanford.edu/media/'
           'shapenetcore_partanno_segmentation_benchmark_v0_normal.zip')

    category_ids = {
        'Airplane': '02691156',
        'Bag': '02773838',
        'Cap': '02954340',
        'Car': '02958343',
        'Chair': '03001627',
        'Earphone': '03261776',
        'Guitar': '03467517',
        'Knife': '03624134',
        'Lamp': '03636649',
        'Laptop': '03642806',
        'Motorbike': '03790512',
        'Mug': '03797390',
        'Pistol': '03948459',
        'Rocket': '04099429',
        'Skateboard': '04225987',
        'Table': '04379243',
    }

    seg_classes = {
        'Airplane': [0, 1, 2, 3],
        'Bag': [4, 5],
        'Cap': [6, 7],
        'Car': [8, 9, 10, 11],
        'Chair': [12, 13, 14, 15],
        'Earphone': [16, 17, 18],
        'Guitar': [19, 20, 21],
        'Knife': [22, 23],
        'Lamp': [24, 25, 26, 27],
        'Laptop': [28, 29],
        'Motorbike': [30, 31, 32, 33, 34, 35],
        'Mug': [36, 37],
        'Pistol': [38, 39, 40],
        'Rocket': [41, 42, 43],
        'Skateboard': [44, 45, 46],
        'Table': [47, 48, 49],
    }

    def __init__(
        self,
        root: str,
        categories: Optional[Union[str, List[str]]] = None,
        include_normals: bool = True,
        split: str = 'trainval',
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        if categories is None:
            categories = list(self.category_ids.keys())
        if isinstance(categories, str):
            categories = [categories]
        assert all(category in self.category_ids for category in categories)
        self.categories = categories
        super().__init__(root, transform, pre_transform, pre_filter)

        if split == 'train':
            path = self.processed_paths[0]
        elif split == 'val':
            path = self.processed_paths[1]
        elif split == 'test':
            path = self.processed_paths[2]
        elif split == 'trainval':
            path = self.processed_paths[3]
        else:
            raise ValueError((f'Split {split} found, but expected either '
                              'train, val, trainval or test'))

        self.data, self.slices = torch.load(path)
        self._data.x = self._data.x if include_normals else None

        self.y_mask = torch.zeros((len(self.seg_classes.keys()), 50),
                                  dtype=torch.bool)
        for i, labels in enumerate(self.seg_classes.values()):
            self.y_mask[i, labels] = 1

    @property
    def num_classes(self) -> int:
        return self.y_mask.size(-1)

    @property
    def raw_file_names(self) -> List[str]:
        return list(self.category_ids.values()) + ['train_test_split']

    @property
    def processed_file_names(self) -> str:
        cats = '_'.join([cat[:3].lower() for cat in self.categories])
        return [
            osp.join(f'{cats}_{split}.pt')
            for split in ['train', 'val', 'test', 'trainval']
        ]

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.unlink(path)
        shutil.rmtree(self.raw_dir)
        name = self.url.split('/')[-1].split('.')[0]
        os.rename(osp.join(self.root, name), self.raw_dir)

    def process_filenames(self, filenames: List[str]) -> List[Data]:
        data_list = []
        categories_ids = [self.category_ids[cat] for cat in self.categories]
        cat_idx = {categories_ids[i]: i for i in range(len(categories_ids))}

        for name in filenames:
            cat = name.split(osp.sep)[0]
            if cat not in categories_ids:
                continue

            data = read_txt_array(osp.join(self.raw_dir, name))
            pos = data[:, :3]
            x = data[:, 3:6]
            y = data[:, -1].type(torch.long)
            data = Data(pos=pos, x=x, y=y, category=cat_idx[cat])
            if self.pre_filter is not None and not self.pre_filter(data):
                continue
            if self.pre_transform is not None:
                data = self.pre_transform(data)
            data_list.append(data)

        return data_list

    def process(self):
        trainval = []
        for i, split in enumerate(['train', 'val', 'test']):
            path = osp.join(self.raw_dir, 'train_test_split',
                            f'shuffled_{split}_file_list.json')
            with open(path, 'r') as f:
                filenames = [
                    osp.sep.join(name.split('/')[1:]) + '.txt'
                    for name in json.load(f)
                ]  # Removing first directory.
            data_list = self.process_filenames(filenames)
            if split == 'train' or split == 'val':
                trainval += data_list
            torch.save(self.collate(data_list), self.processed_paths[i])
        torch.save(self.collate(trainval), self.processed_paths[3])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'categories={self.categories})')
pytorch/torch_geometric/datasets/shrec2016.py000066400000000000000000000140061444262304300215250ustar00rootroot00000000000000import glob
import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import InMemoryDataset, download_url, extract_zip
from torch_geometric.io import read_off, read_txt_array


class SHREC2016(InMemoryDataset):
    r"""The SHREC 2016 partial matching dataset from the `"SHREC'16: Partial
    Matching of Deformable Shapes"
    `_ paper.
    The reference shape can be referenced via :obj:`dataset.ref`.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        partiality (str): The partiality of the dataset (one of :obj:`"Holes"`,
            :obj:`"Cuts"`).
        category (str): The category of the dataset (one of
            :obj:`"Cat"`, :obj:`"Centaur"`, :obj:`"David"`, :obj:`"Dog"`,
            :obj:`"Horse"`, :obj:`"Michael"`, :obj:`"Victoria"`,
            :obj:`"Wolf"`).
        train (bool, optional): If :obj:`True`, loads the training dataset,
            otherwise the test dataset. (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    train_url = ('http://www.dais.unive.it/~shrec2016/data/'
                 'shrec2016_PartialDeformableShapes.zip')
    test_url = ('http://www.dais.unive.it/~shrec2016/data/'
                'shrec2016_PartialDeformableShapes_TestSet.zip')

    categories = [
        'cat', 'centaur', 'david', 'dog', 'horse', 'michael', 'victoria',
        'wolf'
    ]
    partialities = ['holes', 'cuts']

    def __init__(
        self,
        root: str,
        partiality: str,
        category: str,
        train: bool = True,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        assert partiality.lower() in self.partialities
        self.part = partiality.lower()
        assert category.lower() in self.categories
        self.cat = category.lower()
        super().__init__(root, transform, pre_transform, pre_filter)
        self.__ref__ = torch.load(self.processed_paths[0])
        path = self.processed_paths[1] if train else self.processed_paths[2]
        self.data, self.slices = torch.load(path)

    @property
    def ref(self) -> str:
        ref = self.__ref__
        if self.transform is not None:
            ref = self.transform(ref)
        return ref

    @property
    def raw_file_names(self) -> List[str]:
        return ['training', 'test']

    @property
    def processed_file_names(self) -> List[str]:
        name = f'{self.part}_{self.cat}.pt'
        return [f'{i}_{name}' for i in ['ref', 'training', 'test']]

    def download(self):
        path = download_url(self.train_url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)
        path = osp.join(self.raw_dir, 'shrec2016_PartialDeformableShapes')
        os.rename(path, osp.join(self.raw_dir, 'training'))

        path = download_url(self.test_url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)
        path = osp.join(self.raw_dir,
                        'shrec2016_PartialDeformableShapes_TestSet')
        os.rename(path, osp.join(self.raw_dir, 'test'))

    def process(self):
        ref_data = read_off(
            osp.join(self.raw_paths[0], 'null', f'{self.cat}.off'))

        train_list = []
        name = f'{self.part}_{self.cat}_*.off'
        paths = glob.glob(osp.join(self.raw_paths[0], self.part, name))
        paths = [path[:-4] for path in paths]
        paths = sorted(paths, key=lambda e: (len(e), e))

        for path in paths:
            data = read_off(f'{path}.off')
            y = read_txt_array(f'{path}.baryc_gt')
            data.y = y[:, 0].to(torch.long) - 1
            data.y_baryc = y[:, 1:]
            train_list.append(data)

        test_list = []
        name = f'{self.part}_{self.cat}_*.off'
        paths = glob.glob(osp.join(self.raw_paths[1], self.part, name))
        paths = [path[:-4] for path in paths]
        paths = sorted(paths, key=lambda e: (len(e), e))

        for path in paths:
            test_list.append(read_off(f'{path}.off'))

        if self.pre_filter is not None:
            train_list = [d for d in train_list if self.pre_filter(d)]
            test_list = [d for d in test_list if self.pre_filter(d)]

        if self.pre_transform is not None:
            ref_data = self.pre_transform(ref_data)
            train_list = [self.pre_transform(d) for d in train_list]
            test_list = [self.pre_transform(d) for d in test_list]

        torch.save(ref_data, self.processed_paths[0])
        torch.save(self.collate(train_list), self.processed_paths[1])
        torch.save(self.collate(test_list), self.processed_paths[2])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'partiality={self.part}, category={self.cat})')
pytorch/torch_geometric/datasets/snap_dataset.py000066400000000000000000000221031444262304300225530ustar00rootroot00000000000000import os
import os.path as osp
from typing import Any, Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_gz,
    extract_tar,
)
from torch_geometric.data.makedirs import makedirs
from torch_geometric.utils import coalesce


class EgoData(Data):
    def __inc__(self, key: str, value: Any, *args, **kwargs):
        if key == 'circle':
            return self.num_nodes
        elif key == 'circle_batch':
            return int(value.max()) + 1 if value.numel() > 0 else 0
        return super().__inc__(key, value, *args, **kwargs)


def read_ego(files: List[str], name: str) -> List[EgoData]:
    import pandas as pd

    all_featnames = []
    files = [
        x for x in files if x.split('.')[-1] in
        ['circles', 'edges', 'egofeat', 'feat', 'featnames']
    ]
    for i in range(4, len(files), 5):
        featnames_file = files[i]
        with open(featnames_file, 'r') as f:
            featnames = f.read().split('\n')[:-1]
            featnames = [' '.join(x.split(' ')[1:]) for x in featnames]
            all_featnames += featnames
    all_featnames = sorted(list(set(all_featnames)))
    all_featnames = {key: i for i, key in enumerate(all_featnames)}

    data_list = []
    for i in range(0, len(files), 5):
        circles_file = files[i]
        edges_file = files[i + 1]
        egofeat_file = files[i + 2]
        feat_file = files[i + 3]
        featnames_file = files[i + 4]

        x = None
        if name != 'gplus':  # Don't read node features on g-plus:
            x_ego = pd.read_csv(egofeat_file, sep=' ', header=None,
                                dtype=np.float32)
            x_ego = torch.from_numpy(x_ego.values)

            x = pd.read_csv(feat_file, sep=' ', header=None, dtype=np.float32)
            x = torch.from_numpy(x.values)[:, 1:]

            x_all = torch.cat([x, x_ego], dim=0)

            # Reorder `x` according to `featnames` ordering.
            x_all = torch.zeros(x.size(0), len(all_featnames))
            with open(featnames_file, 'r') as f:
                featnames = f.read().split('\n')[:-1]
                featnames = [' '.join(x.split(' ')[1:]) for x in featnames]
            indices = [all_featnames[featname] for featname in featnames]
            x_all[:, torch.tensor(indices)] = x
            x = x_all

        idx = pd.read_csv(feat_file, sep=' ', header=None, dtype=str,
                          usecols=[0]).squeeze()

        idx_assoc = {}
        for i, j in enumerate(idx):
            idx_assoc[j] = i

        circles = []
        circles_batch = []
        with open(circles_file, 'r') as f:
            for i, circle in enumerate(f.read().split('\n')[:-1]):
                circle = [idx_assoc[c] for c in circle.split()[1:]]
                circles += circle
                circles_batch += [i] * len(circle)
        circle = torch.tensor(circles)
        circle_batch = torch.tensor(circles_batch)

        try:
            row = pd.read_csv(edges_file, sep=' ', header=None, dtype=str,
                              usecols=[0]).squeeze()
            col = pd.read_csv(edges_file, sep=' ', header=None, dtype=str,
                              usecols=[1]).squeeze()
        except:  # noqa
            continue

        row = torch.tensor([idx_assoc[i] for i in row])
        col = torch.tensor([idx_assoc[i] for i in col])

        N = max(int(row.max()), int(col.max())) + 2
        N = x.size(0) if x is not None else N

        row_ego = torch.full((N - 1, ), N - 1, dtype=torch.long)
        col_ego = torch.arange(N - 1)

        # Ego node should be connected to every other node.
        row = torch.cat([row, row_ego, col_ego], dim=0)
        col = torch.cat([col, col_ego, row_ego], dim=0)
        edge_index = torch.stack([row, col], dim=0)
        edge_index = coalesce(edge_index, num_nodes=N)

        data = EgoData(x=x, edge_index=edge_index, circle=circle,
                       circle_batch=circle_batch)

        data_list.append(data)

    return data_list


def read_soc(files: List[str], name: str) -> List[Data]:
    import pandas as pd

    skiprows = 4
    if name == 'pokec':
        skiprows = 0

    edge_index = pd.read_csv(files[0], sep='\t', header=None,
                             skiprows=skiprows, dtype=np.int64)
    edge_index = torch.from_numpy(edge_index.values).t()
    num_nodes = edge_index.max().item() + 1
    edge_index = coalesce(edge_index, num_nodes=num_nodes)

    return [Data(edge_index=edge_index, num_nodes=num_nodes)]


def read_wiki(files: List[str], name: str) -> List[Data]:
    import pandas as pd

    edge_index = pd.read_csv(files[0], sep='\t', header=None, skiprows=4,
                             dtype=np.int64)
    edge_index = torch.from_numpy(edge_index.values).t()

    idx = torch.unique(edge_index.flatten())
    idx_assoc = torch.full((edge_index.max() + 1, ), -1, dtype=torch.long)
    idx_assoc[idx] = torch.arange(idx.size(0))

    edge_index = idx_assoc[edge_index]
    num_nodes = edge_index.max().item() + 1
    edge_index = coalesce(edge_index, num_nodes=num_nodes)

    return [Data(edge_index=edge_index, num_nodes=num_nodes)]


class SNAPDataset(InMemoryDataset):
    r"""A variety of graph datasets collected from `SNAP at Stanford University
    `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'https://snap.stanford.edu/data'

    available_datasets = {
        'ego-facebook': ['facebook.tar.gz'],
        'ego-gplus': ['gplus.tar.gz'],
        'ego-twitter': ['twitter.tar.gz'],
        'soc-ca-astroph': ['ca-AstroPh.txt.gz'],
        'soc-ca-grqc': ['ca-GrQc.txt.gz'],
        'soc-epinions1': ['soc-Epinions1.txt.gz'],
        'soc-livejournal1': ['soc-LiveJournal1.txt.gz'],
        'soc-pokec': ['soc-pokec-relationships.txt.gz'],
        'soc-slashdot0811': ['soc-Slashdot0811.txt.gz'],
        'soc-slashdot0922': ['soc-Slashdot0902.txt.gz'],
        'wiki-vote': ['wiki-Vote.txt.gz'],
    }

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in self.available_datasets.keys()
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def _download(self):
        if osp.isdir(self.raw_dir) and len(os.listdir(self.raw_dir)) > 0:
            return

        makedirs(self.raw_dir)
        self.download()

    def download(self):
        for name in self.available_datasets[self.name]:
            path = download_url(f'{self.url}/{name}', self.raw_dir)
            if name.endswith('.tar.gz'):
                extract_tar(path, self.raw_dir)
            elif name.endswith('.gz'):
                extract_gz(path, self.raw_dir)
            os.unlink(path)

    def process(self):
        raw_dir = self.raw_dir
        filenames = os.listdir(self.raw_dir)
        if len(filenames) == 1 and osp.isdir(osp.join(raw_dir, filenames[0])):
            raw_dir = osp.join(raw_dir, filenames[0])

        raw_files = sorted([osp.join(raw_dir, f) for f in os.listdir(raw_dir)])

        if self.name[:4] == 'ego-':
            data_list = read_ego(raw_files, self.name[4:])
        elif self.name[:4] == 'soc-':
            data_list = read_soc(raw_files, self.name[:4])
        elif self.name[:5] == 'wiki-':
            data_list = read_wiki(raw_files, self.name[5:])
        else:
            raise NotImplementedError

        if len(data_list) > 1 and self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        torch.save(self.collate(data_list), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'SNAP-{self.name}({len(self)})'
pytorch/torch_geometric/datasets/suite_sparse.py000066400000000000000000000057271444262304300226300ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class SuiteSparseMatrixCollection(InMemoryDataset):
    r"""A suite of sparse matrix benchmarks known as the `Suite Sparse Matrix
    Collection `_ collected from a wide range of
    applications.

    Args:
        root (str): Root directory where the dataset should be saved.
        group (str): The group of the sparse matrix.
        name (str): The name of the sparse matrix.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = 'https://sparse.tamu.edu/mat/{}/{}.mat'

    def __init__(
        self,
        root: str,
        group: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.group = group
        self.name = name
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.group, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.group, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.mat'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        url = self.url.format(self.group, self.name)
        download_url(url, self.raw_dir)

    def process(self):
        from scipy.io import loadmat

        mat = loadmat(self.raw_paths[0])['Problem'][0][0][2].tocsr().tocoo()

        row = torch.from_numpy(mat.row).to(torch.long)
        col = torch.from_numpy(mat.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        edge_attr = torch.from_numpy(mat.data).to(torch.float)
        if torch.all(edge_attr == 1.):
            edge_attr = None

        size = torch.Size(mat.shape)
        if mat.shape[0] == mat.shape[1]:
            size = None

        num_nodes = mat.shape[0]

        data = Data(edge_index=edge_index, edge_attr=edge_attr, size=size,
                    num_nodes=num_nodes)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(group={self.group}, '
                f'name={self.name})')
pytorch/torch_geometric/datasets/taobao.py000066400000000000000000000075671444262304300213730ustar00rootroot00000000000000import os
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class Taobao(InMemoryDataset):
    r"""Taobao is a dataset of user behaviors from Taobao offered by Alibaba,
    provided by the `Tianchi Alicloud platform
    `_.

    Taobao is a heterogeneous graph for recommendation.
    Nodes represent users with user IDs, items with item IDs, and categories
    with category ID.
    Edges between users and items represent different types of user behaviors
    towards items (alongside with timestamps).
    Edges between items and categories assign each item to its set of
    categories.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            every access. (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.HeteroData` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    """
    url = ('https://alicloud-dev.oss-cn-hangzhou.aliyuncs.com/'
           'UserBehavior.csv.zip')

    def __init__(
        self,
        root,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> str:
        return 'UserBehavior.csv'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        import pandas as pd

        cols = ['userId', 'itemId', 'categoryId', 'behaviorType', 'timestamp']
        df = pd.read_csv(self.raw_paths[0], names=cols)

        # Time representation (YYYY.MM.DD-HH:MM:SS -> Integer)
        # start: 1511539200 = 2017.11.25-00:00:00
        # end:   1512316799 = 2017.12.03-23:59:59
        start = 1511539200
        end = 1512316799
        df = df[(df["timestamp"] >= start) & (df["timestamp"] <= end)]

        df = df.drop_duplicates()

        behavior_dict = {'pv': 0, 'cart': 1, 'buy': 2, 'fav': 3}
        df['behaviorType'] = df['behaviorType'].map(behavior_dict)

        num_entries = {}
        for col in ['userId', 'itemId', 'categoryId']:
            # Map IDs to consecutive integers:
            value, df[col] = np.unique(df[[col]].values, return_inverse=True)
            num_entries[col] = value.shape[0]

        data = HeteroData()

        data['user'].num_nodes = num_entries['userId']
        data['item'].num_nodes = num_entries['itemId']
        data['category'].num_nodes = num_entries['categoryId']

        row = torch.from_numpy(df['userId'].values)
        col = torch.from_numpy(df['itemId'].values)
        data['user', 'item'].edge_index = torch.stack([row, col], dim=0)
        data['user', 'item'].time = torch.from_numpy(df['timestamp'].values)
        behavior = torch.from_numpy(df['behaviorType'].values)
        data['user', 'item'].behavior = behavior

        df = df[['itemId', 'categoryId']].drop_duplicates()
        row = torch.from_numpy(df['itemId'].values)
        col = torch.from_numpy(df['categoryId'].values)
        data['item', 'category'].edge_index = torch.stack([row, col], dim=0)

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/tosca.py000066400000000000000000000105431444262304300212230ustar00rootroot00000000000000import glob
import os
import os.path as osp
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_txt_array


class TOSCA(InMemoryDataset):
    r"""The TOSCA dataset from the `"Numerical Geometry of Non-Ridig Shapes"
    `_ book, containing 80 meshes.
    Meshes within the same category have the same triangulation and an equal
    number of vertices numbered in a compatible way.

    .. note::

        Data objects hold mesh faces instead of edge indices.
        To convert the mesh to a graph, use the
        :obj:`torch_geometric.transforms.FaceToEdge` as :obj:`pre_transform`.
        To convert the mesh to a point cloud, use the
        :obj:`torch_geometric.transforms.SamplePoints` as :obj:`transform` to
        sample a fixed number of points on the mesh faces according to their
        face area.

    Args:
        root (str): Root directory where the dataset should be saved.
        categories (list, optional): List of categories to include in the
            dataset. Can include the categories :obj:`"Cat"`, :obj:`"Centaur"`,
            :obj:`"David"`, :obj:`"Dog"`, :obj:`"Gorilla"`, :obj:`"Horse"`,
            :obj:`"Michael"`, :obj:`"Victoria"`, :obj:`"Wolf"`. If set to
            :obj:`None`, the dataset will contain all categories. (default:
            :obj:`None`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """

    url = 'http://tosca.cs.technion.ac.il/data/toscahires-asci.zip'

    categories = [
        'cat', 'centaur', 'david', 'dog', 'gorilla', 'horse', 'michael',
        'victoria', 'wolf'
    ]

    def __init__(
        self,
        root: str,
        categories: Optional[List[str]] = None,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        categories = self.categories if categories is None else categories
        categories = [cat.lower() for cat in categories]
        for cat in categories:
            assert cat in self.categories
        self.categories = categories
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['cat0.vert', 'cat0.tri']

    @property
    def processed_file_names(self) -> str:
        name = '_'.join([cat[:2] for cat in self.categories])
        return f'{name}.pt'

    def download(self):
        path = download_url(self.url, self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.unlink(path)

    def process(self):
        data_list = []
        for cat in self.categories:
            paths = glob.glob(osp.join(self.raw_dir, f'{cat}*.tri'))
            paths = [path[:-4] for path in paths]
            paths = sorted(paths, key=lambda e: (len(e), e))

            for path in paths:
                pos = read_txt_array(f'{path}.vert')
                face = read_txt_array(f'{path}.tri', dtype=torch.long)
                face = face - face.min()  # Ensure zero-based index.
                data = Data(pos=pos, face=face.t().contiguous())
                if self.pre_filter is not None and not self.pre_filter(data):
                    continue
                if self.pre_transform is not None:
                    data = self.pre_transform(data)
                data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])
pytorch/torch_geometric/datasets/tu_dataset.py000066400000000000000000000162061444262304300222510ustar00rootroot00000000000000import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_tu_data


class TUDataset(InMemoryDataset):
    r"""A variety of graph kernel benchmark datasets, *.e.g.*,
    :obj:`"IMDB-BINARY"`, :obj:`"REDDIT-BINARY"` or :obj:`"PROTEINS"`,
    collected from the `TU Dortmund University
    `_.
    In addition, this dataset wrapper provides `cleaned dataset versions
    `_ as motivated by the
    `"Understanding Isomorphism Bias in Graph Data Sets"
    `_ paper, containing only non-isomorphic
    graphs.

    .. note::
        Some datasets may not come with any node labels.
        You can then either make use of the argument :obj:`use_node_attr`
        to load additional continuous node attributes (if present) or provide
        synthetic node features using transforms such as
        :class:`torch_geometric.transforms.Constant` or
        :class:`torch_geometric.transforms.OneHotDegree`.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The `name
            `_ of the
            dataset.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
        use_node_attr (bool, optional): If :obj:`True`, the dataset will
            contain additional continuous node attributes (if present).
            (default: :obj:`False`)
        use_edge_attr (bool, optional): If :obj:`True`, the dataset will
            contain additional continuous edge attributes (if present).
            (default: :obj:`False`)
        cleaned (bool, optional): If :obj:`True`, the dataset will
            contain only non-isomorphic graphs. (default: :obj:`False`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - MUTAG
          - 188
          - ~17.9
          - ~39.6
          - 7
          - 2
        * - ENZYMES
          - 600
          - ~32.6
          - ~124.3
          - 3
          - 6
        * - PROTEINS
          - 1,113
          - ~39.1
          - ~145.6
          - 3
          - 2
        * - COLLAB
          - 5,000
          - ~74.5
          - ~4914.4
          - 0
          - 3
        * - IMDB-BINARY
          - 1,000
          - ~19.8
          - ~193.1
          - 0
          - 2
        * - REDDIT-BINARY
          - 2,000
          - ~429.6
          - ~995.5
          - 0
          - 2
        * - ...
          -
          -
          -
          -
          -
    """

    url = 'https://www.chrsmrrs.com/graphkerneldatasets'
    cleaned_url = ('https://raw.githubusercontent.com/nd7141/'
                   'graph_datasets/master/datasets')

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 pre_filter: Optional[Callable] = None,
                 use_node_attr: bool = False, use_edge_attr: bool = False,
                 cleaned: bool = False):
        self.name = name
        self.cleaned = cleaned
        super().__init__(root, transform, pre_transform, pre_filter)

        out = torch.load(self.processed_paths[0])
        if not isinstance(out, tuple) or len(out) != 3:
            raise RuntimeError(
                "The 'data' object was created by an older version of PyG. "
                "If this error occurred while loading an already existing "
                "dataset, remove the 'processed/' directory in the dataset's "
                "root folder and try again.")
        data, self.slices, self.sizes = out
        self.data = Data.from_dict(data) if isinstance(data, dict) else data

        if self._data.x is not None and not use_node_attr:
            num_node_attributes = self.num_node_attributes
            self._data.x = self._data.x[:, num_node_attributes:]
        if self._data.edge_attr is not None and not use_edge_attr:
            num_edge_attrs = self.num_edge_attributes
            self._data.edge_attr = self._data.edge_attr[:, num_edge_attrs:]

    @property
    def raw_dir(self) -> str:
        name = f'raw{"_cleaned" if self.cleaned else ""}'
        return osp.join(self.root, self.name, name)

    @property
    def processed_dir(self) -> str:
        name = f'processed{"_cleaned" if self.cleaned else ""}'
        return osp.join(self.root, self.name, name)

    @property
    def num_node_labels(self) -> int:
        return self.sizes['num_node_labels']

    @property
    def num_node_attributes(self) -> int:
        return self.sizes['num_node_attributes']

    @property
    def num_edge_labels(self) -> int:
        return self.sizes['num_edge_labels']

    @property
    def num_edge_attributes(self) -> int:
        return self.sizes['num_edge_attributes']

    @property
    def raw_file_names(self) -> List[str]:
        names = ['A', 'graph_indicator']
        return [f'{self.name}_{name}.txt' for name in names]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        url = self.cleaned_url if self.cleaned else self.url
        folder = osp.join(self.root, self.name)
        path = download_url(f'{url}/{self.name}.zip', folder)
        extract_zip(path, folder)
        os.unlink(path)
        shutil.rmtree(self.raw_dir)
        os.rename(osp.join(folder, self.name), self.raw_dir)

    def process(self):
        self.data, self.slices, sizes = read_tu_data(self.raw_dir, self.name)

        if self.pre_filter is not None or self.pre_transform is not None:
            data_list = [self.get(idx) for idx in range(len(self))]

            if self.pre_filter is not None:
                data_list = [d for d in data_list if self.pre_filter(d)]

            if self.pre_transform is not None:
                data_list = [self.pre_transform(d) for d in data_list]

            self.data, self.slices = self.collate(data_list)
            self._data_list = None  # Reset cache.

        torch.save((self._data.to_dict(), self.slices, sizes),
                   self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name}({len(self)})'
pytorch/torch_geometric/datasets/twitch.py000066400000000000000000000066101444262304300214140ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class Twitch(InMemoryDataset):
    r"""The Twitch Gamer networks introduced in the
    `"Multi-scale Attributed Node Embedding"
    `_ paper.
    Nodes represent gamers on Twitch and edges are followerships between them.
    Node features represent embeddings of games played by the Twitch users.
    The task is to predict whether a user streams mature content.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"DE"`, :obj:`"EN"`,
            :obj:`"ES"`, :obj:`"FR"`, :obj:`"PT"`, :obj:`"RU"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - DE
          - 9,498
          - 315,774
          - 128
          - 2
        * - EN
          - 7,126
          - 77,774
          - 128
          - 2
        * - ES
          - 4,648
          - 123,412
          - 128
          - 2
        * - FR
          - 6,551
          - 231,883
          - 128
          - 2
        * - PT
          - 1,912
          - 64,510
          - 128
          - 2
        * - RU
          - 4,385
          - 78,993
          - 128
          - 2
    """

    url = 'https://graphmining.ai/datasets/ptg/twitch'

    def __init__(self, root: str, name: str,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name
        assert self.name in ['DE', 'EN', 'ES', 'FR', 'PT', 'RU']
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        return f'{self.name}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        download_url(f'{self.url}/{self.name}.npz', self.raw_dir)

    def process(self):
        data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
        x = torch.from_numpy(data['features']).to(torch.float)
        y = torch.from_numpy(data['target']).to(torch.long)

        edge_index = torch.from_numpy(data['edges']).to(torch.long)
        edge_index = edge_index.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/upfd.py000066400000000000000000000154211444262304300210500ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)
from torch_geometric.io import read_txt_array
from torch_geometric.utils import coalesce


class UPFD(InMemoryDataset):
    r"""The tree-structured fake news propagation graph classification dataset
    from the `"User Preference-aware Fake News Detection"
    `_ paper.
    It includes two sets of tree-structured fake & real news propagation graphs
    extracted from Twitter.
    For a single graph, the root node represents the source news, and leaf
    nodes represent Twitter users who retweeted the same root news.
    A user node has an edge to the news node if and only if the user retweeted
    the root news directly.
    Two user nodes have an edge if and only if one user retweeted the root news
    from the other user.
    Four different node features are encoded using different encoders.
    Please refer to `GNN-FakeNews
    `_ repo for more details.

    .. note::

        For an example of using UPFD, see `examples/upfd.py
        `_.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the graph set (:obj:`"politifact"`,
            :obj:`"gossipcop"`).
        feature (str): The node feature type (:obj:`"profile"`, :obj:`"spacy"`,
            :obj:`"bert"`, :obj:`"content"`).
            If set to :obj:`"profile"`, the 10-dimensional node feature
            is composed of ten Twitter user profile attributes.
            If set to :obj:`"spacy"`, the 300-dimensional node feature is
            composed of Twitter user historical tweets encoded by
            the `spaCy word2vec encoder
            `_.
            If set to :obj:`"bert"`, the 768-dimensional node feature is
            composed of Twitter user historical tweets encoded by the
            `bert-as-service `_.
            If set to :obj:`"content"`, the 310-dimensional node feature is
            composed of a 300-dimensional "spacy" vector plus a
            10-dimensional "profile" vector.
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    ids = {
        'politifact': '1KOmSrlGcC50PjkvRVbyb_WoWHVql06J-',
        'gossipcop': '1VskhAQ92PrT4sWEKQ2v2-AJhEcpp4A81',
    }

    def __init__(
        self,
        root: str,
        name: str,
        feature: str,
        split: str = "train",
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.root = root
        self.name = name
        self.feature = feature
        super().__init__(root, transform, pre_transform, pre_filter)

        assert split in ['train', 'val', 'test']
        path = self.processed_paths[['train', 'val', 'test'].index(split)]
        self.data, self.slices = torch.load(path)

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed', self.feature)

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'node_graph_id.npy', 'graph_labels.npy', 'A.txt', 'train_idx.npy',
            'val_idx.npy', 'test_idx.npy', f'new_{self.feature}_feature.npz'
        ]

    @property
    def processed_file_names(self):
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        path = download_url(self.url.format(self.ids[self.name]), self.raw_dir)
        extract_zip(path, self.raw_dir)
        os.remove(path)

    def process(self):
        x = sp.load_npz(
            osp.join(self.raw_dir, f'new_{self.feature}_feature.npz'))
        x = torch.from_numpy(x.todense()).to(torch.float)

        edge_index = read_txt_array(osp.join(self.raw_dir, 'A.txt'), sep=',',
                                    dtype=torch.long).t()
        edge_index = coalesce(edge_index, num_nodes=x.size(0))

        y = np.load(osp.join(self.raw_dir, 'graph_labels.npy'))
        y = torch.from_numpy(y).to(torch.long)
        _, y = y.unique(sorted=True, return_inverse=True)

        batch = np.load(osp.join(self.raw_dir, 'node_graph_id.npy'))
        batch = torch.from_numpy(batch).to(torch.long)

        node_slice = torch.cumsum(batch.bincount(), 0)
        node_slice = torch.cat([torch.tensor([0]), node_slice])
        edge_slice = torch.cumsum(batch[edge_index[0]].bincount(), 0)
        edge_slice = torch.cat([torch.tensor([0]), edge_slice])
        graph_slice = torch.arange(y.size(0) + 1)
        self.slices = {
            'x': node_slice,
            'edge_index': edge_slice,
            'y': graph_slice
        }

        edge_index -= node_slice[batch[edge_index[0]]].view(1, -1)
        self.data = Data(x=x, edge_index=edge_index, y=y)

        for path, split in zip(self.processed_paths, ['train', 'val', 'test']):
            idx = np.load(osp.join(self.raw_dir, f'{split}_idx.npy')).tolist()
            data_list = [self.get(i) for i in idx]
            if self.pre_filter is not None:
                data_list = [d for d in data_list if self.pre_filter(d)]
            if self.pre_transform is not None:
                data_list = [self.pre_transform(d) for d in data_list]
            torch.save(self.collate(data_list), path)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, name={self.name}, '
                f'feature={self.feature})')
pytorch/torch_geometric/datasets/utils/000077500000000000000000000000001444262304300206755ustar00rootroot00000000000000pytorch/torch_geometric/datasets/utils/__init__.py000066400000000000000000000002661444262304300230120ustar00rootroot00000000000000from .cheatsheet import paper_link, has_stats, get_stat, get_children, get_type

__all__ = [
    'paper_link',
    'has_stats',
    'get_stat',
    'get_children',
    'get_type',
]
pytorch/torch_geometric/datasets/utils/cheatsheet.py000066400000000000000000000033041444262304300233640ustar00rootroot00000000000000import importlib
import inspect
import re
from typing import Any, List, Optional


def paper_link(cls: str) -> str:
    cls = importlib.import_module('torch_geometric.datasets').__dict__[cls]
    match = re.search('<.+?>', inspect.getdoc(cls), flags=re.DOTALL)
    return None if match is None else match.group().replace('\n', ' ')[1:-1]


def get_stats_table(cls: str) -> str:
    cls = importlib.import_module('torch_geometric.datasets').__dict__[cls]
    match = re.search(r'\*\*STATS:\*\*\n.*$', inspect.getdoc(cls),
                      flags=re.DOTALL)
    return '' if match is None else match.group()


def has_stats(cls: str) -> bool:
    return len(get_stats_table(cls)) > 0


def get_type(cls: str) -> str:
    return 'Edge' if '-' in cls else 'Node'


def get_stat(cls: str, name: str, child: Optional[str] = None,
             default: Any = None) -> str:
    if child is None and len(get_children(cls)) > 0:
        return ''

    stats_table = get_stats_table(cls)

    if len(stats_table) > 0:
        stats_table = '\n'.join(stats_table.split('\n')[2:])

    match = re.search(f'^.*- {name}', stats_table, flags=re.DOTALL)
    if match is None:
        return default

    column = match.group().count(' -')

    if child is not None:
        child = child.replace('(', r'\(').replace(')', r'\)')
        match = re.search(f'[*] - {child}\n.*$', stats_table, flags=re.DOTALL)
        stats_row = match.group()
    else:
        stats_row = '*' + stats_table.split('*')[2]

    return stats_row.split(' -')[column].split('\n')[0].strip()


def get_children(cls: str) -> List[str]:
    matches = re.findall('[*] -.*', get_stats_table(cls))
    return [match[4:] for match in matches[1:]] if len(matches) > 2 else []
pytorch/torch_geometric/datasets/webkb.py000066400000000000000000000110071444262304300212000ustar00rootroot00000000000000import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import coalesce


class WebKB(InMemoryDataset):
    r"""The WebKB datasets used in the
    `"Geom-GCN: Geometric Graph Convolutional Networks"
    `_ paper.
    Nodes represent web pages and edges represent hyperlinks between them.
    Node features are the bag-of-words representation of web pages.
    The task is to classify the nodes into one of the five categories, student,
    project, course, staff, and faculty.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"Cornell"`, :obj:`"Texas"`,
            :obj:`"Wisconsin"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #nodes
          - #edges
          - #features
          - #classes
        * - Cornell
          - 183
          - 298
          - 1,703
          - 5
        * - Texas
          - 183
          - 325
          - 1,703
          - 5
        * - Wisconsin
          - 251
          - 515
          - 1,703
          - 5
    """

    url = 'https://raw.githubusercontent.com/graphdml-uiuc-jlu/geom-gcn/master'

    def __init__(
        self,
        root: str,
        name: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        self.name = name.lower()
        assert self.name in ['cornell', 'texas', 'wisconsin']

        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        out = ['out1_node_feature_label.txt', 'out1_graph_edges.txt']
        out += [f'{self.name}_split_0.6_0.2_{i}.npz' for i in range(10)]
        return out

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for f in self.raw_file_names[:2]:
            download_url(f'{self.url}/new_data/{self.name}/{f}', self.raw_dir)
        for f in self.raw_file_names[2:]:
            download_url(f'{self.url}/splits/{f}', self.raw_dir)

    def process(self):
        with open(self.raw_paths[0], 'r') as f:
            data = f.read().split('\n')[1:-1]
            x = [[float(v) for v in r.split('\t')[1].split(',')] for r in data]
            x = torch.tensor(x, dtype=torch.float)

            y = [int(r.split('\t')[2]) for r in data]
            y = torch.tensor(y, dtype=torch.long)

        with open(self.raw_paths[1], 'r') as f:
            data = f.read().split('\n')[1:-1]
            data = [[int(v) for v in r.split('\t')] for r in data]
            edge_index = torch.tensor(data, dtype=torch.long).t().contiguous()
            edge_index = coalesce(edge_index, num_nodes=x.size(0))

        train_masks, val_masks, test_masks = [], [], []
        for f in self.raw_paths[2:]:
            tmp = np.load(f)
            train_masks += [torch.from_numpy(tmp['train_mask']).to(torch.bool)]
            val_masks += [torch.from_numpy(tmp['val_mask']).to(torch.bool)]
            test_masks += [torch.from_numpy(tmp['test_mask']).to(torch.bool)]
        train_mask = torch.stack(train_masks, dim=1)
        val_mask = torch.stack(val_masks, dim=1)
        test_mask = torch.stack(test_masks, dim=1)

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(self.collate([data]), self.processed_paths[0])

    def __repr__(self) -> str:
        return f'{self.name}()'
pytorch/torch_geometric/datasets/wikics.py000066400000000000000000000071321444262304300214030ustar00rootroot00000000000000import json
import warnings
from itertools import chain
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import to_undirected


class WikiCS(InMemoryDataset):
    r"""The semi-supervised Wikipedia-based dataset from the
    `"Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks"
    `_ paper, containing 11,701 nodes,
    216,123 edges, 10 classes and 20 different training splits.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        is_undirected (bool, optional): Whether the graph is undirected.
            (default: :obj:`True`)
    """

    url = 'https://github.com/pmernyei/wiki-cs-dataset/raw/master/dataset'

    def __init__(self, root: str, transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None,
                 is_undirected: Optional[bool] = None):
        if is_undirected is None:
            warnings.warn(
                f"The {self.__class__.__name__} dataset now returns an "
                f"undirected graph by default. Please explicitly specify "
                f"'is_undirected=False' to restore the old behavior.")
            is_undirected = True
        self.is_undirected = is_undirected
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['data.json']

    @property
    def processed_file_names(self) -> str:
        return 'data_undirected.pt' if self.is_undirected else 'data.pt'

    def download(self):
        for name in self.raw_file_names:
            download_url(f'{self.url}/{name}', self.raw_dir)

    def process(self):
        with open(self.raw_paths[0], 'r') as f:
            data = json.load(f)

        x = torch.tensor(data['features'], dtype=torch.float)
        y = torch.tensor(data['labels'], dtype=torch.long)

        edges = [[(i, j) for j in js] for i, js in enumerate(data['links'])]
        edges = list(chain(*edges))
        edge_index = torch.tensor(edges, dtype=torch.long).t().contiguous()
        if self.is_undirected:
            edge_index = to_undirected(edge_index, num_nodes=x.size(0))

        train_mask = torch.tensor(data['train_masks'], dtype=torch.bool)
        train_mask = train_mask.t().contiguous()

        val_mask = torch.tensor(data['val_masks'], dtype=torch.bool)
        val_mask = val_mask.t().contiguous()

        test_mask = torch.tensor(data['test_mask'], dtype=torch.bool)

        stopping_mask = torch.tensor(data['stopping_masks'], dtype=torch.bool)
        stopping_mask = stopping_mask.t().contiguous()

        data = Data(x=x, y=y, edge_index=edge_index, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask,
                    stopping_mask=stopping_mask)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/wikipedia_network.py000066400000000000000000000137031444262304300236320ustar00rootroot00000000000000import os.path as osp
from typing import Callable, Optional

import numpy as np
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import coalesce


class WikipediaNetwork(InMemoryDataset):
    r"""The Wikipedia networks introduced in the
    `"Multi-scale Attributed Node Embedding"
    `_ paper.
    Nodes represent web pages and edges represent hyperlinks between them.
    Node features represent several informative nouns in the Wikipedia pages.
    The task is to predict the average daily traffic of the web page.

    Args:
        root (str): Root directory where the dataset should be saved.
        name (str): The name of the dataset (:obj:`"chameleon"`,
            :obj:`"crocodile"`, :obj:`"squirrel"`).
        geom_gcn_preprocess (bool): If set to :obj:`True`, will load the
            pre-processed data as introduced in the `"Geom-GCN: Geometric
            Graph Convolutional Networks" _`,
            in which the average monthly traffic of the web page is converted
            into five categories to predict.
            If set to :obj:`True`, the dataset :obj:`"crocodile"` is not
            available.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    """

    raw_url = 'https://graphmining.ai/datasets/ptg/wiki'
    processed_url = ('https://raw.githubusercontent.com/graphdml-uiuc-jlu/'
                     'geom-gcn/f1fc0d14b3b019c562737240d06ec83b07d16a8f')

    def __init__(self, root: str, name: str, geom_gcn_preprocess: bool = True,
                 transform: Optional[Callable] = None,
                 pre_transform: Optional[Callable] = None):
        self.name = name.lower()
        self.geom_gcn_preprocess = geom_gcn_preprocess
        assert self.name in ['chameleon', 'crocodile', 'squirrel']
        if geom_gcn_preprocess and self.name == 'crocodile':
            raise AttributeError("The dataset 'crocodile' is not available in "
                                 "case 'geom_gcn_preprocess=True'")
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        if self.geom_gcn_preprocess:
            return osp.join(self.root, self.name, 'geom_gcn', 'raw')
        else:
            return osp.join(self.root, self.name, 'raw')

    @property
    def processed_dir(self) -> str:
        if self.geom_gcn_preprocess:
            return osp.join(self.root, self.name, 'geom_gcn', 'processed')
        else:
            return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> str:
        if self.geom_gcn_preprocess:
            return (['out1_node_feature_label.txt', 'out1_graph_edges.txt'] +
                    [f'{self.name}_split_0.6_0.2_{i}.npz' for i in range(10)])
        else:
            return f'{self.name}.npz'

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        if self.geom_gcn_preprocess:
            for filename in self.raw_file_names[:2]:
                url = f'{self.processed_url}/new_data/{self.name}/{filename}'
                download_url(url, self.raw_dir)
            for filename in self.raw_file_names[2:]:
                url = f'{self.processed_url}/splits/{filename}'
                download_url(url, self.raw_dir)
        else:
            download_url(f'{self.raw_url}/{self.name}.npz', self.raw_dir)

    def process(self):
        if self.geom_gcn_preprocess:
            with open(self.raw_paths[0], 'r') as f:
                data = f.read().split('\n')[1:-1]
            x = [[float(v) for v in r.split('\t')[1].split(',')] for r in data]
            x = torch.tensor(x, dtype=torch.float)
            y = [int(r.split('\t')[2]) for r in data]
            y = torch.tensor(y, dtype=torch.long)

            with open(self.raw_paths[1], 'r') as f:
                data = f.read().split('\n')[1:-1]
                data = [[int(v) for v in r.split('\t')] for r in data]
            edge_index = torch.tensor(data, dtype=torch.long).t().contiguous()
            edge_index = coalesce(edge_index, num_nodes=x.size(0))

            train_masks, val_masks, test_masks = [], [], []
            for filepath in self.raw_paths[2:]:
                f = np.load(filepath)
                train_masks += [torch.from_numpy(f['train_mask'])]
                val_masks += [torch.from_numpy(f['val_mask'])]
                test_masks += [torch.from_numpy(f['test_mask'])]
            train_mask = torch.stack(train_masks, dim=1).to(torch.bool)
            val_mask = torch.stack(val_masks, dim=1).to(torch.bool)
            test_mask = torch.stack(test_masks, dim=1).to(torch.bool)

            data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                        val_mask=val_mask, test_mask=test_mask)

        else:
            data = np.load(self.raw_paths[0], 'r', allow_pickle=True)
            x = torch.from_numpy(data['features']).to(torch.float)
            edge_index = torch.from_numpy(data['edges']).to(torch.long)
            edge_index = edge_index.t().contiguous()
            edge_index = coalesce(edge_index, num_nodes=x.size(0))
            y = torch.from_numpy(data['target']).to(torch.float)

            data = Data(x=x, edge_index=edge_index, y=y)

        if self.pre_transform is not None:
            data = self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/willow_object_class.py000066400000000000000000000147501444262304300241460ustar00rootroot00000000000000import glob
import os
import os.path as osp
import shutil
from typing import Callable, List, Optional

import torch
import torch.nn.functional as F
from torch.utils.data import DataLoader

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class WILLOWObjectClass(InMemoryDataset):
    r"""The WILLOW-ObjectClass dataset from the `"Learning Graphs to Match"
    `_ paper,
    containing 10 equal keypoints of at least 40 images in each category.
    The keypoints contain interpolated features from a pre-trained VGG16 model
    on ImageNet (:obj:`relu4_2` and :obj:`relu5_1`).

    Args:
        root (str): Root directory where the dataset should be saved.
        category (str): The category of the images (one of :obj:`"Car"`,
            :obj:`"Duck"`, :obj:`"Face"`, :obj:`"Motorbike"`,
            :obj:`"Winebottle"`).
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)
        device (str or torch.device, optional): The device to use for
            processing the raw data. If set to :obj:`None`, will utilize
            GPU-processing if available. (default: :obj:`None`)
    """
    url = ('http://www.di.ens.fr/willow/research/graphlearning/'
           'WILLOW-ObjectClass_dataset.zip')

    categories = ['face', 'motorbike', 'car', 'duck', 'winebottle']

    batch_size = 32

    def __init__(
        self,
        root: str,
        category: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
        device: Optional[str] = None,
    ):
        if device is None:
            device = 'cuda' if torch.cuda.is_available() else 'cpu'

        assert category.lower() in self.categories
        self.category = category
        self.device = device
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_dir(self) -> str:
        return osp.join(self.root, 'raw')

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.category.capitalize(), 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        return [category.capitalize() for category in self.categories]

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.unlink(path)
        os.unlink(osp.join(self.root, 'README'))
        os.unlink(osp.join(self.root, 'demo_showAnno.m'))
        shutil.rmtree(self.raw_dir)
        os.rename(osp.join(self.root, 'WILLOW-ObjectClass'), self.raw_dir)

    def process(self):
        import torchvision.models as models
        import torchvision.transforms as T
        from PIL import Image
        from scipy.io import loadmat

        category = self.category.capitalize()
        names = glob.glob(osp.join(self.raw_dir, category, '*.png'))
        names = sorted([name[:-4] for name in names])

        vgg16_outputs = []

        def hook(module, x, y):
            vgg16_outputs.append(y.to('cpu'))

        vgg16 = models.vgg16(pretrained=True).to(self.device)
        vgg16.eval()
        vgg16.features[20].register_forward_hook(hook)  # relu4_2
        vgg16.features[25].register_forward_hook(hook)  # relu5_1

        transform = T.Compose([
            T.ToTensor(),
            T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        ])

        data_list = []
        for name in names:
            pos = loadmat(f'{name}.mat')['pts_coord']
            x, y = torch.from_numpy(pos).to(torch.float)
            pos = torch.stack([x, y], dim=1)

            # The "face" category contains a single image with less than 10
            # keypoints, so we need to skip it.
            if pos.size(0) != 10:
                continue

            with open(f'{name}.png', 'rb') as f:
                img = Image.open(f).convert('RGB')

            # Rescale keypoints.
            pos[:, 0] = pos[:, 0] * 256.0 / (img.size[0])
            pos[:, 1] = pos[:, 1] * 256.0 / (img.size[1])

            img = img.resize((256, 256), resample=Image.BICUBIC)
            img = transform(img)

            data = Data(img=img, pos=pos, name=name)
            data_list.append(data)

        imgs = [data.img for data in data_list]
        loader = DataLoader(imgs, self.batch_size, shuffle=False)
        for i, batch_img in enumerate(loader):
            vgg16_outputs.clear()

            with torch.no_grad():
                vgg16(batch_img.to(self.device))

            out1 = F.interpolate(vgg16_outputs[0], (256, 256), mode='bilinear',
                                 align_corners=False)
            out2 = F.interpolate(vgg16_outputs[1], (256, 256), mode='bilinear',
                                 align_corners=False)

            for j in range(out1.size(0)):
                data = data_list[i * self.batch_size + j]
                idx = data.pos.round().long().clamp(0, 255)
                x_1 = out1[j, :, idx[:, 1], idx[:, 0]].to('cpu')
                x_2 = out2[j, :, idx[:, 1], idx[:, 0]].to('cpu')
                data.img = None
                data.x = torch.cat([x_1.t(), x_2.t()], dim=-1)
            del out1
            del out2

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        torch.save(self.collate(data_list), self.processed_paths[0])

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({len(self)}, '
                f'category={self.category})')
pytorch/torch_geometric/datasets/word_net.py000066400000000000000000000172611444262304300217370ustar00rootroot00000000000000from itertools import chain
from typing import Callable, List, Optional

import torch

from torch_geometric.data import Data, InMemoryDataset, download_url
from torch_geometric.utils import index_sort


class WordNet18(InMemoryDataset):
    r"""The WordNet18 dataset from the `"Translating Embeddings for Modeling
    Multi-Relational Data"
    `_ paper,
    containing 40,943 entities, 18 relations and 151,442 fact triplets,
    *e.g.*, furniture includes bed.

    .. note::

        The original :obj:`WordNet18` dataset suffers from test leakage, *i.e.*
        more than 80% of test triplets can be found in the training set with
        another relation type.
        Therefore, it should not be used for research evaluation anymore.
        We recommend to use its cleaned version
        :class:`~torch_geometric.datasets.WordNet18RR` instead.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = ('https://raw.githubusercontent.com/villmow/'
           'datasets_knowledge_embedding/master/WN18/original')

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['train.txt', 'valid.txt', 'test.txt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process(self):
        srcs, dsts, edge_types = [], [], []
        for path in self.raw_paths:
            with open(path, 'r') as f:
                data = [int(x) for x in f.read().split()[1:]]
                data = torch.tensor(data, dtype=torch.long)
                srcs.append(data[::3])
                dsts.append(data[1::3])
                edge_types.append(data[2::3])

        src = torch.cat(srcs, dim=0)
        dst = torch.cat(dsts, dim=0)
        edge_type = torch.cat(edge_types, dim=0)

        train_mask = torch.zeros(src.size(0), dtype=torch.bool)
        train_mask[:srcs[0].size(0)] = True
        val_mask = torch.zeros(src.size(0), dtype=torch.bool)
        val_mask[srcs[0].size(0):srcs[0].size(0) + srcs[1].size(0)] = True
        test_mask = torch.zeros(src.size(0), dtype=torch.bool)
        test_mask[srcs[0].size(0) + srcs[1].size(0):] = True

        num_nodes = max(int(src.max()), int(dst.max())) + 1
        _, perm = index_sort(num_nodes * src + dst)

        edge_index = torch.stack([src[perm], dst[perm]], dim=0)
        edge_type = edge_type[perm]
        train_mask = train_mask[perm]
        val_mask = val_mask[perm]
        test_mask = test_mask[perm]

        data = Data(edge_index=edge_index, edge_type=edge_type,
                    train_mask=train_mask, val_mask=val_mask,
                    test_mask=test_mask, num_nodes=num_nodes)

        if self.pre_transform is not None:
            data = self.pre_filter(data)

        torch.save(self.collate([data]), self.processed_paths[0])


class WordNet18RR(InMemoryDataset):
    r"""The WordNet18RR dataset from the `"Convolutional 2D Knowledge Graph
    Embeddings" `_ paper, containing 40,943
    entities, 11 relations and 93,003 fact triplets.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
    """

    url = ('https://raw.githubusercontent.com/villmow/'
           'datasets_knowledge_embedding/master/WN18RR/original')

    edge2id = {
        '_also_see': 0,
        '_derivationally_related_form': 1,
        '_has_part': 2,
        '_hypernym': 3,
        '_instance_hypernym': 4,
        '_member_meronym': 5,
        '_member_of_domain_region': 6,
        '_member_of_domain_usage': 7,
        '_similar_to': 8,
        '_synset_domain_topic_of': 9,
        '_verb_group': 10,
    }

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['train.txt', 'valid.txt', 'test.txt']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        for filename in self.raw_file_names:
            download_url(f'{self.url}/{filename}', self.raw_dir)

    def process(self):
        node2id, idx = {}, 0

        srcs, dsts, edge_types = [], [], []
        for path in self.raw_paths:
            with open(path, 'r') as f:
                data = f.read().split()

                src = data[::3]
                dst = data[2::3]
                edge_type = data[1::3]

                for i in chain(src, dst):
                    if i not in node2id:
                        node2id[i] = idx
                        idx += 1

                src = [node2id[i] for i in src]
                dst = [node2id[i] for i in dst]
                edge_type = [self.edge2id[i] for i in edge_type]

                srcs.append(torch.tensor(src, dtype=torch.long))
                dsts.append(torch.tensor(dst, dtype=torch.long))
                edge_types.append(torch.tensor(edge_type, dtype=torch.long))

        src = torch.cat(srcs, dim=0)
        dst = torch.cat(dsts, dim=0)
        edge_type = torch.cat(edge_types, dim=0)

        train_mask = torch.zeros(src.size(0), dtype=torch.bool)
        train_mask[:srcs[0].size(0)] = True
        val_mask = torch.zeros(src.size(0), dtype=torch.bool)
        val_mask[srcs[0].size(0):srcs[0].size(0) + srcs[1].size(0)] = True
        test_mask = torch.zeros(src.size(0), dtype=torch.bool)
        test_mask[srcs[0].size(0) + srcs[1].size(0):] = True

        num_nodes = max(int(src.max()), int(dst.max())) + 1
        _, perm = index_sort(num_nodes * src + dst)

        edge_index = torch.stack([src[perm], dst[perm]], dim=0)
        edge_type = edge_type[perm]
        train_mask = train_mask[perm]
        val_mask = val_mask[perm]
        test_mask = test_mask[perm]

        data = Data(edge_index=edge_index, edge_type=edge_type,
                    train_mask=train_mask, val_mask=val_mask,
                    test_mask=test_mask, num_nodes=num_nodes)

        if self.pre_transform is not None:
            data = self.pre_filter(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/yelp.py000066400000000000000000000100241444262304300210550ustar00rootroot00000000000000import json
import os
import os.path as osp
from typing import Callable, List, Optional

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import Data, InMemoryDataset, download_url


class Yelp(InMemoryDataset):
    r"""The Yelp dataset from the `"GraphSAINT: Graph Sampling Based
    Inductive Learning Method" `_ paper,
    containing customer reviewers and their friendship.

    Args:
        root (str): Root directory where the dataset should be saved.
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 10 10 10 10
        :header-rows: 1

        * - #nodes
          - #edges
          - #features
          - #tasks
        * - 716,847
          - 13,954,819
          - 300
          - 100
    """
    url = 'https://docs.google.com/uc?export=download&id={}&confirm=t'

    adj_full_id = '1Juwx8HtDwSzmVIJ31ooVa1WljI4U5JnA'
    feats_id = '1Zy6BZH_zLEjKlEFSduKE5tV9qqA_8VtM'
    class_map_id = '1VUcBGr0T0-klqerjAjxRmAqFuld_SMWU'
    role_id = '1NI5pa5Chpd-52eSmLW60OnB3WS5ikxq_'

    def __init__(
        self,
        root: str,
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
    ):
        super().__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self) -> List[str]:
        return ['adj_full.npz', 'feats.npy', 'class_map.json', 'role.json']

    @property
    def processed_file_names(self) -> str:
        return 'data.pt'

    def download(self):
        path = download_url(self.url.format(self.adj_full_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'adj_full.npz'))

        path = download_url(self.url.format(self.feats_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'feats.npy'))

        path = download_url(self.url.format(self.class_map_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'class_map.json'))

        path = download_url(self.url.format(self.role_id), self.raw_dir)
        os.rename(path, osp.join(self.raw_dir, 'role.json'))

    def process(self):
        f = np.load(osp.join(self.raw_dir, 'adj_full.npz'))
        adj = sp.csr_matrix((f['data'], f['indices'], f['indptr']), f['shape'])
        adj = adj.tocoo()
        row = torch.from_numpy(adj.row).to(torch.long)
        col = torch.from_numpy(adj.col).to(torch.long)
        edge_index = torch.stack([row, col], dim=0)

        x = np.load(osp.join(self.raw_dir, 'feats.npy'))
        x = torch.from_numpy(x).to(torch.float)

        ys = [-1] * x.size(0)
        with open(osp.join(self.raw_dir, 'class_map.json')) as f:
            class_map = json.load(f)
            for key, item in class_map.items():
                ys[int(key)] = item
        y = torch.tensor(ys)

        with open(osp.join(self.raw_dir, 'role.json')) as f:
            role = json.load(f)

        train_mask = torch.zeros(x.size(0), dtype=torch.bool)
        train_mask[torch.tensor(role['tr'])] = True

        val_mask = torch.zeros(x.size(0), dtype=torch.bool)
        val_mask[torch.tensor(role['va'])] = True

        test_mask = torch.zeros(x.size(0), dtype=torch.bool)
        test_mask[torch.tensor(role['te'])] = True

        data = Data(x=x, edge_index=edge_index, y=y, train_mask=train_mask,
                    val_mask=val_mask, test_mask=test_mask)

        data = data if self.pre_transform is None else self.pre_transform(data)

        torch.save(self.collate([data]), self.processed_paths[0])
pytorch/torch_geometric/datasets/zinc.py000066400000000000000000000140331444262304300210530ustar00rootroot00000000000000import os
import os.path as osp
import pickle
import shutil
from typing import Callable, List, Optional

import torch
from tqdm import tqdm

from torch_geometric.data import (
    Data,
    InMemoryDataset,
    download_url,
    extract_zip,
)


class ZINC(InMemoryDataset):
    r"""The ZINC dataset from the `ZINC database
    `_ and the
    `"Automatic Chemical Design Using a Data-Driven Continuous Representation
    of Molecules" `_ paper, containing about
    250,000 molecular graphs with up to 38 heavy atoms.
    The task is to regress the penalized :obj:`logP` (also called constrained
    solubility in some works), given by :obj:`y = logP - SAS - cycles`, where
    :obj:`logP` is the water-octanol partition coefficient, :obj:`SAS` is the
    synthetic accessibility score, and :obj:`cycles` denotes the number of
    cycles with more than six atoms.
    Penalized :obj:`logP` is a score commonly used for training molecular
    generation models, see, *e.g.*, the
    `"Junction Tree Variational Autoencoder for Molecular Graph Generation"
    `_ and
    `"Grammar Variational Autoencoder"
    `_ papers.

    Args:
        root (str): Root directory where the dataset should be saved.
        subset (bool, optional): If set to :obj:`True`, will only load a
            subset of the dataset (12,000 molecular graphs), following the
            `"Benchmarking Graph Neural Networks"
            `_ paper. (default: :obj:`False`)
        split (str, optional): If :obj:`"train"`, loads the training dataset.
            If :obj:`"val"`, loads the validation dataset.
            If :obj:`"test"`, loads the test dataset.
            (default: :obj:`"train"`)
        transform (callable, optional): A function/transform that takes in an
            :obj:`torch_geometric.data.Data` object and returns a transformed
            version. The data object will be transformed before every access.
            (default: :obj:`None`)
        pre_transform (callable, optional): A function/transform that takes in
            an :obj:`torch_geometric.data.Data` object and returns a
            transformed version. The data object will be transformed before
            being saved to disk. (default: :obj:`None`)
        pre_filter (callable, optional): A function that takes in an
            :obj:`torch_geometric.data.Data` object and returns a boolean
            value, indicating whether the data object should be included in the
            final dataset. (default: :obj:`None`)

    **STATS:**

    .. list-table::
        :widths: 20 10 10 10 10 10
        :header-rows: 1

        * - Name
          - #graphs
          - #nodes
          - #edges
          - #features
          - #classes
        * - ZINC Full
          - 249,456
          - ~23.2
          - ~49.8
          - 1
          - 1
        * - ZINC Subset
          - 12,000
          - ~23.2
          - ~49.8
          - 1
          - 1
    """

    url = 'https://www.dropbox.com/s/feo9qle74kg48gy/molecules.zip?dl=1'
    split_url = ('https://raw.githubusercontent.com/graphdeeplearning/'
                 'benchmarking-gnns/master/data/molecules/{}.index')

    def __init__(
        self,
        root: str,
        subset: bool = False,
        split: str = 'train',
        transform: Optional[Callable] = None,
        pre_transform: Optional[Callable] = None,
        pre_filter: Optional[Callable] = None,
    ):
        self.subset = subset
        assert split in ['train', 'val', 'test']
        super().__init__(root, transform, pre_transform, pre_filter)
        path = osp.join(self.processed_dir, f'{split}.pt')
        self.data, self.slices = torch.load(path)

    @property
    def raw_file_names(self) -> List[str]:
        return [
            'train.pickle', 'val.pickle', 'test.pickle', 'train.index',
            'val.index', 'test.index'
        ]

    @property
    def processed_dir(self) -> str:
        name = 'subset' if self.subset else 'full'
        return osp.join(self.root, name, 'processed')

    @property
    def processed_file_names(self) -> List[str]:
        return ['train.pt', 'val.pt', 'test.pt']

    def download(self):
        shutil.rmtree(self.raw_dir)
        path = download_url(self.url, self.root)
        extract_zip(path, self.root)
        os.rename(osp.join(self.root, 'molecules'), self.raw_dir)
        os.unlink(path)

        for split in ['train', 'val', 'test']:
            download_url(self.split_url.format(split), self.raw_dir)

    def process(self):
        for split in ['train', 'val', 'test']:
            with open(osp.join(self.raw_dir, f'{split}.pickle'), 'rb') as f:
                mols = pickle.load(f)

            indices = range(len(mols))

            if self.subset:
                with open(osp.join(self.raw_dir, f'{split}.index'), 'r') as f:
                    indices = [int(x) for x in f.read()[:-1].split(',')]

            pbar = tqdm(total=len(indices))
            pbar.set_description(f'Processing {split} dataset')

            data_list = []
            for idx in indices:
                mol = mols[idx]

                x = mol['atom_type'].to(torch.long).view(-1, 1)
                y = mol['logP_SA_cycle_normalized'].to(torch.float)

                adj = mol['bond_type']
                edge_index = adj.nonzero(as_tuple=False).t().contiguous()
                edge_attr = adj[edge_index[0], edge_index[1]].to(torch.long)

                data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr,
                            y=y)

                if self.pre_filter is not None and not self.pre_filter(data):
                    continue

                if self.pre_transform is not None:
                    data = self.pre_transform(data)

                data_list.append(data)
                pbar.update(1)

            pbar.close()

            torch.save(self.collate(data_list),
                       osp.join(self.processed_dir, f'{split}.pt'))
pytorch/torch_geometric/debug.py000066400000000000000000000022551444262304300173710ustar00rootroot00000000000000__debug_flag__ = {'enabled': False}


def is_debug_enabled():
    r"""Returns :obj:`True` if the debug mode is enabled."""
    return __debug_flag__['enabled']


def set_debug_enabled(mode: bool):
    __debug_flag__['enabled'] = mode


class debug:
    r"""Context-manager that enables the debug mode to help track down errors
    and separate usage errors from real bugs.

    .. code-block:: python

        with torch_geometric.debug():
            out = model(data.x, data.edge_index)
    """
    def __init__(self):
        self.prev = is_debug_enabled()

    def __enter__(self):
        set_debug_enabled(True)

    def __exit__(self, *args):
        set_debug_enabled(self.prev)


class set_debug:
    r"""Context-manager that sets the debug mode on or off.

    :class:`set_debug` will enable or disable the debug mode based on its
    argument :attr:`mode`.
    It can be used as a context-manager or as a function.

    See :class:`debug` above for more details.
    """
    def __init__(self, mode: bool):
        self.prev = is_debug_enabled()
        set_debug_enabled(mode)

    def __enter__(self):
        pass

    def __exit__(self, *args):
        set_debug_enabled(self.prev)
pytorch/torch_geometric/deprecation.py000066400000000000000000000013541444262304300205770ustar00rootroot00000000000000import functools
import inspect
import warnings
from typing import Optional


def deprecated(details: Optional[str] = None, func_name: Optional[str] = None):
    def decorator(func):
        name = func_name or func.__name__

        if inspect.isclass(func):
            cls = type(func.__name__, (func, ), {})
            cls.__init__ = deprecated(details, name)(func.__init__)
            cls.__doc__ = func.__doc__
            return cls

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            out = f"'{name}' is deprecated"
            if details is not None:
                out += f", {details}"
            warnings.warn(out)
            return func(*args, **kwargs)

        return wrapper

    return decorator
pytorch/torch_geometric/distributed/000077500000000000000000000000001444262304300202475ustar00rootroot00000000000000pytorch/torch_geometric/distributed/__init__.py000066400000000000000000000002531444262304300223600ustar00rootroot00000000000000from .local_feature_store import LocalFeatureStore
from .local_graph_store import LocalGraphStore

__all__ = classes = [
    'LocalFeatureStore',
    'LocalGraphStore',
]
pytorch/torch_geometric/distributed/local_feature_store.py000066400000000000000000000170071444262304300246470ustar00rootroot00000000000000import copy
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import FeatureStore, TensorAttr
from torch_geometric.data.feature_store import _field_status
from torch_geometric.typing import EdgeType, NodeType


@dataclass
class LocalTensorAttr(TensorAttr):
    r"""Tensor attribute for storing features without :obj:`index`."""
    def __init__(
        self,
        group_name: Optional[Union[NodeType, EdgeType]] = _field_status.UNSET,
        attr_name: Optional[str] = _field_status.UNSET,
        index=None,
    ):
        super().__init__(group_name, attr_name, index)


class LocalFeatureStore(FeatureStore):
    r"""This class implements the :class:`torch_geometric.data.FeatureStore`
    interface to act as a local feature store for distributed training."""
    def __init__(self):
        super().__init__(tensor_attr_cls=LocalTensorAttr)

        self._feat: Dict[Tuple[Union[NodeType, EdgeType], str], Tensor] = {}

        # Save the global node/edge IDs:
        self._global_id: Dict[Union[NodeType, EdgeType], Tensor] = {}

        # Save the mapping from global node/edge IDs to indices in `_feat`:
        self._global_id_to_index: Dict[Union[NodeType, EdgeType], Tensor] = {}

    @staticmethod
    def key(attr: TensorAttr) -> Tuple[str, str]:
        return (attr.group_name, attr.attr_name)

    def put_global_id(
        self,
        global_id: Tensor,
        group_name: Union[NodeType, EdgeType],
    ) -> bool:
        self._global_id[group_name] = global_id
        self._set_global_id_to_index(group_name)
        return True

    def get_global_id(
        self,
        group_name: Union[NodeType, EdgeType],
    ) -> Optional[Tensor]:
        return self._global_id.get(group_name)

    def remove_global_id(self, group_name: Union[NodeType, EdgeType]) -> bool:
        return self._global_id.pop(group_name) is not None

    def _set_global_id_to_index(self, group_name: Union[NodeType, EdgeType]):
        global_id = self.get_global_id(group_name)

        if global_id is None:
            return

        # TODO Compute this mapping without materializing a full-sized tensor:
        global_id_to_index = global_id.new_full((int(global_id.max()) + 1, ),
                                                fill_value=-1)
        global_id_to_index[global_id] = torch.arange(global_id.numel())
        self._global_id_to_index[group_name] = global_id_to_index

    def _put_tensor(self, tensor: Tensor, attr: TensorAttr) -> bool:
        assert attr.index is None
        self._feat[self.key(attr)] = tensor
        return True

    def _get_tensor(self, attr: TensorAttr) -> Optional[Tensor]:
        tensor = self._feat.get(self.key(attr))

        if tensor is None:
            return None

        if attr.index is None:  # Empty indices return the full tensor:
            return tensor

        return tensor[attr.index]

    def _remove_tensor(self, attr: TensorAttr) -> bool:
        assert attr.index is None
        return self._feat.pop(self.key(attr), None) is not None

    def get_tensor_from_global_id(self, *args, **kwargs) -> Optional[Tensor]:
        attr = self._tensor_attr_cls.cast(*args, **kwargs)
        assert attr.index is not None

        attr = copy.copy(attr)
        attr.index = self._global_id_to_index[attr.group_name][attr.index]

        return self.get_tensor(attr)

    def _get_tensor_size(self, attr: TensorAttr) -> Tuple[int, ...]:
        return self._get_tensor(attr).size()

    def get_all_tensor_attrs(self) -> List[LocalTensorAttr]:
        return [self._tensor_attr_cls.cast(*key) for key in self._feat.keys()]

    # Initialization ##########################################################

    @classmethod
    def from_data(
        cls,
        node_id: Tensor,
        x: Optional[Tensor] = None,
        y: Optional[Tensor] = None,
        edge_id: Optional[Tensor] = None,
        edge_attr: Optional[Tensor] = None,
    ) -> 'LocalFeatureStore':
        r"""Creates a local feature store from homogeneous :pyg:`PyG` tensors.

        Args:
            node_id (torch.Tensor): The global identifier for every local node.
            x (torch.Tensor, optional): The node features.
                (default: :obj:`None`)
            y (torch.Tensor, optional): The node labels. (default: :obj:`None`)
            edge_id (torch.Tensor, optional): The global identifier for every
                local edge. (default: :obj:`None`)
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
        """
        feat_store = cls()
        feat_store.put_global_id(node_id, group_name=None)
        if x is not None:
            feat_store.put_tensor(x, group_name=None, attr_name='x')
        if y is not None:
            feat_store.put_tensor(y, group_name=None, attr_name='y')
        if edge_id is not None:
            feat_store.put_global_id(edge_id, group_name=(None, None))
        if edge_attr is not None:
            if edge_id is None:
                raise ValueError("'edge_id' needs to be present in case "
                                 "'edge_attr' is passed")
            feat_store.put_tensor(edge_attr, group_name=(None, None),
                                  attr_name='edge_attr')
        return feat_store

    @classmethod
    def from_hetero_data(
        cls,
        node_id_dict: Dict[NodeType, Tensor],
        x_dict: Optional[Dict[NodeType, Tensor]] = None,
        y_dict: Optional[Dict[NodeType, Tensor]] = None,
        edge_id_dict: Optional[Dict[EdgeType, Tensor]] = None,
        edge_attr_dict: Optional[Dict[EdgeType, Tensor]] = None,
    ) -> 'LocalFeatureStore':
        r"""Creates a local graph store from heterogeneous :pyg:`PyG` tensors.

        Args:
            node_id_dict (Dict[NodeType, torch.Tensor]): The global identifier
                for every local node of every node type.
            x_dict (Dict[NodeType, torch.Tensor], optional): The node features
                of node types. (default: :obj:`None`)
            y_dict (Dict[NodeType, torch.Tensor], optional): The node labels of
                node types. (default: :obj:`None`)
            edge_id_dict (Dict[EdgeType, torch.Tensor], optional): The global
                identifier for every local edge of edge types.
                (default: :obj:`None`)
            edge_attr_dict (Dict[EdgeType, torch.Tensor], optional): The edge
                features of edge types. (default: :obj:`None`)
        """
        feat_store = cls()

        for node_type, node_id in node_id_dict.items():
            feat_store.put_global_id(node_id, group_name=node_type)
        if x_dict is not None:
            for node_type, x in x_dict.items():
                feat_store.put_tensor(x, group_name=node_type, attr_name='x')
        if y_dict is not None:
            for node_type, y in y_dict.items():
                feat_store.put_tensor(y, group_name=node_type, attr_name='y')
        if edge_id_dict is not None:
            for edge_type, edge_id in edge_id_dict.items():
                feat_store.put_global_id(edge_id, group_name=edge_type)
        if edge_attr_dict is not None:
            for edge_type, edge_attr in edge_attr_dict.items():
                if edge_id_dict is None or edge_type not in edge_id_dict:
                    raise ValueError("'edge_id' needs to be present in case "
                                     "'edge_attr' is passed")
                feat_store.put_tensor(edge_attr, group_name=edge_type,
                                      attr_name='edge_attr')

        return feat_store
pytorch/torch_geometric/distributed/local_graph_store.py000066400000000000000000000100761444262304300243140ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple

from torch import Tensor

from torch_geometric.data import EdgeAttr, GraphStore
from torch_geometric.typing import EdgeTensorType, EdgeType, NodeType


class LocalGraphStore(GraphStore):
    r"""This class implements the :class:`torch_geometric.data.GraphStore`
    interface to act as a local graph store for distributed training."""
    def __init__(self):
        super().__init__()
        self._edge_index: Dict[Tuple, EdgeTensorType] = {}
        self._edge_attr: Dict[Tuple, EdgeAttr] = {}
        self._edge_id: Dict[Tuple, Tensor] = {}

    @staticmethod
    def key(attr: EdgeAttr) -> Tuple:
        return (attr.edge_type, attr.layout.value)

    def put_edge_id(self, edge_id: Tensor, *args, **kwargs) -> bool:
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        self._edge_id[self.key(edge_attr)] = edge_id
        return True

    def get_edge_id(self, *args, **kwargs) -> Optional[EdgeTensorType]:
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        return self._edge_id.get(self.key(edge_attr))

    def remove_edge_id(self, *args, **kwargs) -> bool:
        edge_attr = self._edge_attr_cls.cast(*args, **kwargs)
        return self._edge_id.pop(self.key(edge_attr), None) is not None

    def _put_edge_index(self, edge_index: EdgeTensorType,
                        edge_attr: EdgeAttr) -> bool:
        self._edge_index[self.key(edge_attr)] = edge_index
        self._edge_attr[self.key(edge_attr)] = edge_attr
        return True

    def _get_edge_index(self, edge_attr: EdgeAttr) -> Optional[EdgeTensorType]:
        return self._edge_index.get(self.key(edge_attr), None)

    def _remove_edge_index(self, edge_attr: EdgeAttr) -> bool:
        self._edge_attr.pop(self.key(edge_attr), None)
        return self._edge_index.pop(self.key(edge_attr), None) is not None

    def get_all_edge_attrs(self) -> List[EdgeAttr]:
        return [self._edge_attr[key] for key in self._edge_index.keys()]

    # Initialization ##########################################################

    @classmethod
    def from_data(
        cls,
        edge_id: Tensor,
        edge_index: Tensor,
        num_nodes: int,
    ) -> 'LocalGraphStore':
        r"""Creates a local graph store from a homogeneous :pyg:`PyG` graph.

        Args:
            edge_id (torch.Tensor): The global identifier for every local edge.
            edge_index (torch.Tensor): The local edge indices.
            num_nodes (int): The number of nodes in the local graph.
        """
        attr = dict(
            edge_type=None,
            layout='coo',
            size=(num_nodes, num_nodes),
        )

        graph_store = cls()
        graph_store.put_edge_index(edge_index, **attr)
        graph_store.put_edge_id(edge_id, **attr)
        return graph_store

    @classmethod
    def from_hetero_data(
        cls,
        edge_id_dict: Dict[EdgeType, Tensor],
        edge_index_dict: Dict[EdgeType, Tensor],
        num_nodes_dict: Dict[NodeType, int],
    ) -> 'LocalGraphStore':
        r"""Creates a local graph store from a heterogeneous :pyg:`PyG` graph.

        Args:
            edge_id_dict (Dict[EdgeType, torch.Tensor]): The global identifier
                for every local edge of every edge type.
            edge_index_dict (Dict[EdgeType, torch.Tensor]): The local edge
                indices of every edge type.
            num_nodes_dict (Dict[NodeType, int]): The number of nodes in the
                local graph of every node type.
        """
        attr_dict = {}
        for edge_type in edge_index_dict.keys():
            src, _, dst = edge_type
            attr_dict[edge_type] = dict(
                edge_type=edge_type,
                layout='coo',
                size=(num_nodes_dict[src], num_nodes_dict[dst]),
            )

        graph_store = cls()
        for edge_type, edge_index in edge_index_dict.items():
            attr = attr_dict[edge_type]
            edge_id = edge_id_dict[edge_type]
            graph_store.put_edge_index(edge_index, **attr)
            graph_store.put_edge_id(edge_id, **attr)
        return graph_store
pytorch/torch_geometric/experimental.py000066400000000000000000000110121444262304300207670ustar00rootroot00000000000000import functools
import inspect
from typing import Any, Callable, Dict, List, Optional, Union

import torch

# TODO (matthias) This file currently requires manual imports to let
# TorchScript work on decorated functions. Not totally sure why :(
from torch_geometric.utils import *  # noqa

__experimental_flag__: Dict[str, bool] = {
    'disable_dynamic_shapes': False,
}

Options = Optional[Union[str, List[str]]]


def get_options(options: Options) -> List[str]:
    if options is None:
        options = list(__experimental_flag__.keys())
    if isinstance(options, str):
        options = [options]
    return options


def is_experimental_mode_enabled(options: Options = None) -> bool:
    r"""Returns :obj:`True` if the experimental mode is enabled. See
    :class:`torch_geometric.experimental_mode` for a list of (optional)
    options."""
    if torch.jit.is_scripting() or torch.jit.is_tracing():
        return False
    options = get_options(options)
    return all([__experimental_flag__[option] for option in options])


def set_experimental_mode_enabled(mode: bool, options: Options = None):
    for option in get_options(options):
        __experimental_flag__[option] = mode


class experimental_mode:
    r"""Context-manager that enables the experimental mode to test new but
    potentially unstable features.

    .. code-block:: python

        with torch_geometric.experimental_mode():
            out = model(data.x, data.edge_index)

    Args:
        options (str or list, optional): Currently there are no experimental
            features.
    """
    def __init__(self, options: Options = None):
        self.options = get_options(options)
        self.previous_state = {
            option: __experimental_flag__[option]
            for option in self.options
        }

    def __enter__(self):
        set_experimental_mode_enabled(True, self.options)

    def __exit__(self, *args) -> bool:
        for option, value in self.previous_state.items():
            __experimental_flag__[option] = value


class set_experimental_mode:
    r"""Context-manager that sets the experimental mode on or off.

    :class:`set_experimental_mode` will enable or disable the experimental mode
    based on its argument :attr:`mode`.
    It can be used as a context-manager or as a function.

    See :class:`experimental_mode` above for more details.
    """
    def __init__(self, mode: bool, options: Options = None):
        self.options = get_options(options)
        self.previous_state = {
            option: __experimental_flag__[option]
            for option in self.options
        }
        set_experimental_mode_enabled(mode, self.options)

    def __enter__(self):
        pass

    def __exit__(self, *args):
        for option, value in self.previous_state.items():
            __experimental_flag__[option] = value


def disable_dynamic_shapes(required_args: List[str]) -> Callable:
    r"""A decorator that disables the usage of dynamic shapes for the given
    arguments, i.e., it will raise an error in case :obj:`required_args` are
    not passed and needs to be automatically inferred."""
    def decorator(func: Callable) -> Callable:
        spec = inspect.getfullargspec(func)

        required_args_pos: Dict[str, int] = {}
        for arg_name in required_args:
            if arg_name not in spec.args:
                raise ValueError(f"The function '{func}' does not have a "
                                 f"'{arg_name}' argument")
            required_args_pos[arg_name] = spec.args.index(arg_name)

        num_args = len(spec.args)
        num_default_args = 0 if spec.defaults is None else len(spec.defaults)
        num_positional_args = num_args - num_default_args

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            if not is_experimental_mode_enabled('disable_dynamic_shapes'):
                return func(*args, **kwargs)

            for required_arg in required_args:
                index = required_args_pos[required_arg]

                value: Optional[Any] = None
                if index < len(args):
                    value = args[index]
                elif required_arg in kwargs:
                    value = kwargs[required_arg]
                elif num_default_args > 0:
                    value = spec.defaults[index - num_positional_args]

                if value is None:
                    raise ValueError(f"Dynamic shapes disabled. Argument "
                                     f"'{required_arg}' needs to be set")

            return func(*args, **kwargs)

        return wrapper

    return decorator
pytorch/torch_geometric/explain/000077500000000000000000000000001444262304300173655ustar00rootroot00000000000000pytorch/torch_geometric/explain/__init__.py000066400000000000000000000005471444262304300215040ustar00rootroot00000000000000from .config import ExplainerConfig, ModelConfig, ThresholdConfig
from .explanation import Explanation, HeteroExplanation
from .algorithm import *  # noqa
from .explainer import Explainer
from .metric import *  # noqa

__all__ = [
    'ExplainerConfig',
    'ModelConfig',
    'ThresholdConfig',
    'Explanation',
    'HeteroExplanation',
    'Explainer',
]
pytorch/torch_geometric/explain/algorithm/000077500000000000000000000000001444262304300213535ustar00rootroot00000000000000pytorch/torch_geometric/explain/algorithm/__init__.py000066400000000000000000000006421444262304300234660ustar00rootroot00000000000000from .base import ExplainerAlgorithm
from .dummy_explainer import DummyExplainer
from .gnn_explainer import GNNExplainer
from .captum_explainer import CaptumExplainer
from .pg_explainer import PGExplainer
from .attention_explainer import AttentionExplainer

__all__ = classes = [
    'ExplainerAlgorithm',
    'DummyExplainer',
    'GNNExplainer',
    'CaptumExplainer',
    'PGExplainer',
    'AttentionExplainer',
]
pytorch/torch_geometric/explain/algorithm/attention_explainer.py000066400000000000000000000106131444262304300260020ustar00rootroot00000000000000import logging
from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.explain import Explanation
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.config import ExplanationType, ModelTaskLevel
from torch_geometric.nn.conv.message_passing import MessagePassing


class AttentionExplainer(ExplainerAlgorithm):
    r"""An explainer that uses the attention coefficients produced by an
    attention-based GNN (*e.g.*,
    :class:`~torch_geometric.nn.conv.GATConv`,
    :class:`~torch_geometric.nn.conv.GATv2Conv`, or
    :class:`~torch_geometric.nn.conv.TransformerConv`) as edge explanation.
    Attention scores across layers and heads will be aggregated according to
    the :obj:`reduce` argument.

    Args:
        reduce (str, optional): The method to reduce the attention scores
            across layers and heads. (default: :obj:`"max"`)
    """
    def __init__(self, reduce: str = 'max'):
        super().__init__()
        self.reduce = reduce

    def forward(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Explanation:
        if isinstance(x, dict) or isinstance(edge_index, dict):
            raise ValueError(f"Heterogeneous graphs not yet supported in "
                             f"'{self.__class__.__name__}'")

        hard_edge_mask = None
        if self.model_config.task_level == ModelTaskLevel.node:
            # We need to compute the hard edge mask to properly clean up edge
            # attributions not involved during message passing:
            _, hard_edge_mask = self._get_hard_masks(model, index, edge_index,
                                                     num_nodes=x.size(0))

        alphas: List[Tensor] = []

        def hook(module, msg_kwargs, out):
            if 'alpha' in msg_kwargs[0]:
                alphas.append(msg_kwargs[0]['alpha'].detach())
            elif getattr(module, '_alpha', None) is not None:
                alphas.append(module._alpha.detach())

        hook_handles = []
        for module in model.modules():  # Register message forward hooks:
            if isinstance(module, MessagePassing):
                hook_handles.append(module.register_message_forward_hook(hook))

        model(x, edge_index, **kwargs)

        for handle in hook_handles:  # Remove hooks:
            handle.remove()

        if len(alphas) == 0:
            raise ValueError("Could not collect any attention coefficients. "
                             "Please ensure that your model is using "
                             "attention-based GNN layers.")

        for i, alpha in enumerate(alphas):
            alpha = alpha[:edge_index.size(1)]  # Respect potential self-loops.
            if alpha.dim() == 2:
                alpha = getattr(torch, self.reduce)(alpha, dim=-1)
                if isinstance(alpha, tuple):  # Respect `torch.max`:
                    alpha = alpha[0]
            elif alpha.dim() > 2:
                raise ValueError(f"Can not reduce attention coefficients of "
                                 f"shape {list(alpha.size())}")
            alphas[i] = alpha

        if len(alphas) > 1:
            alpha = torch.stack(alphas, dim=-1)
            alpha = getattr(torch, self.reduce)(alpha, dim=-1)
            if isinstance(alpha, tuple):  # Respect `torch.max`:
                alpha = alpha[0]
        else:
            alpha = alphas[0]

        alpha = self._post_process_mask(alpha, hard_edge_mask,
                                        apply_sigmoid=False)

        return Explanation(edge_mask=alpha)

    def supports(self) -> bool:
        explanation_type = self.explainer_config.explanation_type
        if explanation_type != ExplanationType.model:
            logging.error(f"'{self.__class__.__name__}' only supports "
                          f"model explanations "
                          f"got (`explanation_type={explanation_type.value}`)")
            return False

        node_mask_type = self.explainer_config.node_mask_type
        if node_mask_type is not None:
            logging.error(f"'{self.__class__.__name__}' does not support "
                          f"explaining input node features "
                          f"got (`node_mask_type={node_mask_type.value}`)")
            return False

        return True
pytorch/torch_geometric/explain/algorithm/base.py000066400000000000000000000153631444262304300226470ustar00rootroot00000000000000from abc import abstractmethod
from typing import Dict, Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.explain import Explanation, HeteroExplanation
from torch_geometric.explain.config import (
    ExplainerConfig,
    ModelConfig,
    ModelReturnType,
)
from torch_geometric.nn import MessagePassing
from torch_geometric.typing import EdgeType, NodeType
from torch_geometric.utils import k_hop_subgraph


class ExplainerAlgorithm(torch.nn.Module):
    r"""An abstract base class for implementing explainer algorithms."""
    @abstractmethod
    def forward(
        self,
        model: torch.nn.Module,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Union[Explanation, HeteroExplanation]:
        r"""Computes the explanation.

        Args:
            model (torch.nn.Module): The model to explain.
            x (Union[torch.Tensor, Dict[NodeType, torch.Tensor]]): The input
                node features of a homogeneous or heterogeneous graph.
            edge_index (Union[torch.Tensor, Dict[NodeType, torch.Tensor]]): The
                input edge indices of a homogeneous or heterogeneous graph.
            target (torch.Tensor): The target of the model.
            index (Union[int, Tensor], optional): The index of the model
                output to explain. Can be a single index or a tensor of
                indices. (default: :obj:`None`)
            **kwargs (optional): Additional keyword arguments passed to
                :obj:`model`.
        """

    @abstractmethod
    def supports(self) -> bool:
        r"""Checks if the explainer supports the user-defined settings provided
        in :obj:`self.explainer_config`, :obj:`self.model_config`."""
        pass

    ###########################################################################

    @property
    def explainer_config(self) -> ExplainerConfig:
        r"""Returns the connected explainer configuration."""
        if not hasattr(self, '_explainer_config'):
            raise ValueError(
                f"The explanation algorithm '{self.__class__.__name__}' is "
                f"not yet connected to any explainer configuration. Please "
                f"call `{self.__class__.__name__}.connect(...)` before "
                f"proceeding.")
        return self._explainer_config

    @property
    def model_config(self) -> ModelConfig:
        r"""Returns the connected model configuration."""
        if not hasattr(self, '_model_config'):
            raise ValueError(
                f"The explanation algorithm '{self.__class__.__name__}' is "
                f"not yet connected to any model configuration. Please call "
                f"`{self.__class__.__name__}.connect(...)` before "
                f"proceeding.")
        return self._model_config

    def connect(
        self,
        explainer_config: ExplainerConfig,
        model_config: ModelConfig,
    ):
        r"""Connects an explainer and model configuration to the explainer
        algorithm."""
        self._explainer_config = ExplainerConfig.cast(explainer_config)
        self._model_config = ModelConfig.cast(model_config)

        if not self.supports():
            raise ValueError(
                f"The explanation algorithm '{self.__class__.__name__}' does "
                f"not support the given explanation settings.")

    # Helper functions ########################################################

    @staticmethod
    def _post_process_mask(
        mask: Optional[Tensor],
        hard_mask: Optional[Tensor] = None,
        apply_sigmoid: bool = True,
    ) -> Optional[Tensor]:
        r""""Post processes any mask to not include any attributions of
        elements not involved during message passing."""
        if mask is None:
            return mask

        mask = mask.detach()

        if apply_sigmoid:
            mask = mask.sigmoid()

        if hard_mask is not None and mask.size(0) == hard_mask.size(0):
            mask[~hard_mask] = 0.

        return mask

    @staticmethod
    def _get_hard_masks(
        model: torch.nn.Module,
        node_index: Optional[Union[int, Tensor]],
        edge_index: Tensor,
        num_nodes: int,
    ) -> Tuple[Optional[Tensor], Optional[Tensor]]:
        r"""Returns hard node and edge masks that only include the nodes and
        edges visited during message passing."""
        if node_index is None:
            return None, None  # Consider all nodes and edges.

        index, _, _, edge_mask = k_hop_subgraph(
            node_index,
            num_hops=ExplainerAlgorithm._num_hops(model),
            edge_index=edge_index,
            num_nodes=num_nodes,
            flow=ExplainerAlgorithm._flow(model),
        )

        node_mask = edge_index.new_zeros(num_nodes, dtype=torch.bool)
        node_mask[index] = True

        return node_mask, edge_mask

    @staticmethod
    def _num_hops(model: torch.nn.Module) -> int:
        r"""Returns the number of hops the :obj:`model` is aggregating
        information from.
        """
        num_hops = 0
        for module in model.modules():
            if isinstance(module, MessagePassing):
                num_hops += 1
        return num_hops

    @staticmethod
    def _flow(model: torch.nn.Module) -> str:
        r"""Determines the message passing flow of the :obj:`model`."""
        for module in model.modules():
            if isinstance(module, MessagePassing):
                return module.flow
        return 'source_to_target'

    def _loss_binary_classification(self, y_hat: Tensor, y: Tensor) -> Tensor:
        if self.model_config.return_type == ModelReturnType.raw:
            loss_fn = F.binary_cross_entropy_with_logits
        elif self.model_config.return_type == ModelReturnType.probs:
            loss_fn = F.binary_cross_entropy
        else:
            assert False

        return loss_fn(y_hat.view_as(y), y.float())

    def _loss_multiclass_classification(
        self,
        y_hat: Tensor,
        y: Tensor,
    ) -> Tensor:
        if self.model_config.return_type == ModelReturnType.raw:
            loss_fn = F.cross_entropy
        elif self.model_config.return_type == ModelReturnType.probs:
            loss_fn = F.nll_loss
            y_hat = y_hat.log()
        elif self.model_config.return_type == ModelReturnType.log_probs:
            loss_fn = F.nll_loss
        else:
            assert False

        return loss_fn(y_hat, y)

    def _loss_regression(self, y_hat: Tensor, y: Tensor) -> Tensor:
        assert self.model_config.return_type == ModelReturnType.raw
        return F.mse_loss(y_hat, y)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/explain/algorithm/captum.py000066400000000000000000000303111444262304300232140ustar00rootroot00000000000000from enum import Enum
from typing import Dict, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.explain.algorithm.utils import (
    clear_masks,
    set_hetero_masks,
    set_masks,
)
from torch_geometric.typing import EdgeType, Metadata, NodeType


class MaskLevelType(Enum):
    """Enum class for the mask level type."""
    node = 'node'
    edge = 'edge'
    node_and_edge = 'node_and_edge'

    @property
    def with_edge(self) -> bool:
        return self in [MaskLevelType.edge, MaskLevelType.node_and_edge]


class CaptumModel(torch.nn.Module):
    def __init__(
        self,
        model: torch.nn.Module,
        mask_type: Union[str, MaskLevelType],
        output_idx: Optional[Union[int, Tensor]] = None,
    ):
        super().__init__()

        self.mask_type = MaskLevelType(mask_type)
        self.model = model
        self.output_idx = output_idx

    def forward(self, mask, *args):
        """"""
        # The mask tensor, which comes from Captum's attribution methods,
        # contains the number of samples in dimension 0. Since we are
        # working with only one sample, we squeeze the tensors below.
        assert mask.shape[0] == 1, "Dimension 0 of input should be 1"
        if self.mask_type == MaskLevelType.edge:
            assert len(args) >= 2, "Expects at least x and edge_index as args."
        if self.mask_type == MaskLevelType.node:
            assert len(args) >= 1, "Expects at least edge_index as args."
        if self.mask_type == MaskLevelType.node_and_edge:
            assert args[0].shape[0] == 1, "Dimension 0 of input should be 1"
            assert len(args[1:]) >= 1, "Expects at least edge_index as args."

        # Set edge mask:
        if self.mask_type == MaskLevelType.edge:
            set_masks(self.model, mask.squeeze(0), args[1],
                      apply_sigmoid=False)
        elif self.mask_type == MaskLevelType.node_and_edge:
            set_masks(self.model, args[0].squeeze(0), args[1],
                      apply_sigmoid=False)
            args = args[1:]

        if self.mask_type == MaskLevelType.edge:
            x = self.model(*args)

        else:
            x = self.model(mask.squeeze(0), *args)

        # Clear mask:
        if self.mask_type != MaskLevelType.node:
            clear_masks(self.model)

        if self.output_idx is not None:
            x = x[self.output_idx]
            if isinstance(self.output_idx, int) or (self.output_idx.numel()
                                                    == 1):
                x = x.unsqueeze(0)

        return x


# TODO(jinu) Is there any point of inheriting from `CaptumModel`
class CaptumHeteroModel(CaptumModel):
    def __init__(
        self,
        model: torch.nn.Module,
        mask_type: Union[str, MaskLevelType],
        output_idx: Optional[Union[int, Tensor]],
        metadata: Metadata,
    ):
        super().__init__(model, mask_type, output_idx)
        self.node_types = metadata[0]
        self.edge_types = metadata[1]
        self.num_node_types = len(self.node_types)
        self.num_edge_types = len(self.edge_types)

    def _captum_data_to_hetero_data(
        self, *args
    ) -> Tuple[Dict[NodeType, Tensor], Dict[EdgeType, Tensor], Optional[Dict[
            EdgeType, Tensor]]]:
        """Converts tuple of tensors to `x_dict`, `edge_index_dict` and
        `edge_mask_dict`."""

        if self.mask_type == MaskLevelType.node:
            node_tensors = args[:self.num_node_types]
            node_tensors = [mask.squeeze(0) for mask in node_tensors]
            x_dict = dict(zip(self.node_types, node_tensors))
            edge_index_dict = args[self.num_node_types]
        elif self.mask_type == MaskLevelType.edge:
            edge_mask_tensors = args[:self.num_edge_types]
            x_dict = args[self.num_edge_types]
            edge_index_dict = args[self.num_edge_types + 1]
        else:
            node_tensors = args[:self.num_node_types]
            node_tensors = [mask.squeeze(0) for mask in node_tensors]
            x_dict = dict(zip(self.node_types, node_tensors))
            edge_mask_tensors = args[self.num_node_types:self.num_node_types +
                                     self.num_edge_types]
            edge_index_dict = args[self.num_node_types + self.num_edge_types]

        if self.mask_type.with_edge:
            edge_mask_tensors = [mask.squeeze(0) for mask in edge_mask_tensors]
            edge_mask_dict = dict(zip(self.edge_types, edge_mask_tensors))
        else:
            edge_mask_dict = None
        return x_dict, edge_index_dict, edge_mask_dict

    def forward(self, *args):
        # Validate args:
        if self.mask_type == MaskLevelType.node:
            assert len(args) >= self.num_node_types + 1
            len_remaining_args = len(args) - (self.num_node_types + 1)
        elif self.mask_type == MaskLevelType.edge:
            assert len(args) >= self.num_edge_types + 2
            len_remaining_args = len(args) - (self.num_edge_types + 2)
        else:
            assert len(args) >= self.num_node_types + self.num_edge_types + 1
            len_remaining_args = len(args) - (self.num_node_types +
                                              self.num_edge_types + 1)

        # Get main args:
        (x_dict, edge_index_dict,
         edge_mask_dict) = self._captum_data_to_hetero_data(*args)

        if self.mask_type.with_edge:
            set_hetero_masks(self.model, edge_mask_dict, edge_index_dict)

        if len_remaining_args > 0:
            # If there are args other than `x_dict` and `edge_index_dict`
            x = self.model(x_dict, edge_index_dict,
                           *args[-len_remaining_args:])
        else:
            x = self.model(x_dict, edge_index_dict)

        if self.mask_type.with_edge:
            clear_masks(self.model)

        if self.output_idx is not None:
            x = x[self.output_idx]
            if isinstance(self.output_idx, int) or (self.output_idx.numel()
                                                    == 1):
                x = x.unsqueeze(0)
        return x


def _to_edge_mask(edge_index: Tensor) -> Tensor:
    num_edges = edge_index.shape[1]
    return torch.ones(num_edges, requires_grad=True, device=edge_index.device)


def to_captum_input(
    x: Union[Tensor, Dict[NodeType, Tensor]],
    edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
    mask_type: Union[str, MaskLevelType],
    *args,
) -> Tuple[Tuple[Tensor, ...], Tuple[Tensor, ...]]:
    r"""Given :obj:`x`, :obj:`edge_index` and :obj:`mask_type`, converts it
    to a format to use in `Captum `_ attribution
    methods. Returns :obj:`inputs` and :obj:`additional_forward_args`
    required for :captum:`Captum's` :obj:`attribute` functions.
    See :meth:`~torch_geometric.nn.models.to_captum_model` for example usage.

    Args:

        x (torch.Tensor or Dict[NodeType, torch.Tensor]): The node features.
            For heterogeneous graphs this is a dictionary holding node featues
            for each node type.
        edge_index(torch.Tensor or Dict[EdgeType, torch.Tensor]): The edge
            indices. For heterogeneous graphs this is a dictionary holding the
            :obj:`edge index` for each edge type.
        mask_type (str): Denotes the type of mask to be created with
            a Captum explainer. Valid inputs are :obj:`"edge"`, :obj:`"node"`,
            and :obj:`"node_and_edge"`.
        *args: Additional forward arguments of the model being explained
            which will be added to :obj:`additional_forward_args`.
    """
    mask_type = MaskLevelType(mask_type)

    additional_forward_args = []
    if isinstance(x, Tensor) and isinstance(edge_index, Tensor):
        if mask_type == MaskLevelType.node:
            inputs = [x.unsqueeze(0)]
        elif mask_type == MaskLevelType.edge:
            inputs = [_to_edge_mask(edge_index).unsqueeze(0)]
            additional_forward_args.append(x)
        else:
            inputs = [x.unsqueeze(0), _to_edge_mask(edge_index).unsqueeze(0)]
        additional_forward_args.append(edge_index)

    elif isinstance(x, Dict) and isinstance(edge_index, Dict):
        node_types = x.keys()
        edge_types = edge_index.keys()
        inputs = []
        if mask_type == MaskLevelType.node:
            for key in node_types:
                inputs.append(x[key].unsqueeze(0))
        elif mask_type == MaskLevelType.edge:
            for key in edge_types:
                inputs.append(_to_edge_mask(edge_index[key]).unsqueeze(0))
            additional_forward_args.append(x)
        else:
            for key in node_types:
                inputs.append(x[key].unsqueeze(0))
            for key in edge_types:
                inputs.append(_to_edge_mask(edge_index[key]).unsqueeze(0))
        additional_forward_args.append(edge_index)

    else:
        raise ValueError(
            "'x' and 'edge_index' need to be either"
            f"'Dict' or 'Tensor' got({type(x)}, {type(edge_index)})")
    additional_forward_args.extend(args)
    return tuple(inputs), tuple(additional_forward_args)


def captum_output_to_dicts(
    captum_attrs: Tuple[Tensor, ...],
    mask_type: Union[str, MaskLevelType],
    metadata: Metadata,
) -> Tuple[Optional[Dict[NodeType, Tensor]], Optional[Dict[EdgeType, Tensor]]]:
    r"""Convert the output of `Captum `_ attribution
    methods which is a tuple of attributions to two dictionaries with node and
    edge attribution tensors. This function is used while explaining
    :class:`~torch_geometric.data.HeteroData` objects.
    See :meth:`~torch_geometric.nn.models.to_captum_model` for example usage.

    Args:
        captum_attrs (tuple[torch.Tensor]): The output of attribution methods.
        mask_type (str): Denotes the type of mask to be created with
            a Captum explainer. Valid inputs are :obj:`"edge"`, :obj:`"node"`,
            and :obj:`"node_and_edge"`:

            1. :obj:`"edge"`: :obj:`captum_attrs` contains only edge
               attributions. The returned tuple has no node attributions, and
               an edge attribution dictionary edge types as keys and edge mask
               tensors of shape :obj:`[num_edges]` as values.

            2. :obj:`"node"`: :obj:`captum_attrs` contains only node
               attributions. The returned tuple has a node attribution
               dictionary with node types as keys and node mask tensors of
               shape :obj:`[num_nodes, num_features]` as values, and no edge
               attributions.

            3. :obj:`"node_and_edge"`: :obj:`captum_attrs` contains node and
                edge attributions.

        metadata (Metadata): The metadata of the heterogeneous graph.
    """
    mask_type = MaskLevelType(mask_type)
    node_types = metadata[0]
    edge_types = metadata[1]
    x_attr_dict, edge_attr_dict = None, None
    captum_attrs = [captum_attr.squeeze(0) for captum_attr in captum_attrs]
    if mask_type == MaskLevelType.node:
        assert len(node_types) == len(captum_attrs)
        x_attr_dict = dict(zip(node_types, captum_attrs))
    elif mask_type == MaskLevelType.edge:
        assert len(edge_types) == len(captum_attrs)
        edge_attr_dict = dict(zip(edge_types, captum_attrs))
    elif mask_type == MaskLevelType.node_and_edge:
        assert len(edge_types) + len(node_types) == len(captum_attrs)
        x_attr_dict = dict(zip(node_types, captum_attrs[:len(node_types)]))
        edge_attr_dict = dict(zip(edge_types, captum_attrs[len(node_types):]))
    return x_attr_dict, edge_attr_dict


def convert_captum_output(
    captum_attrs: Tuple[Tensor, ...],
    mask_type: Union[str, MaskLevelType],
    metadata: Optional[Metadata] = None,
):
    r"""Convert the output of `Captum.ai `_ attribution
    methods which is a tuple of attributions to either
    :obj:`(node_mask, edge_mask)` or :obj:`(node_mask_dict, edge_mask_dict)`.
    """
    mask_type = MaskLevelType(mask_type)
    if metadata is not None:
        return captum_output_to_dicts(captum_attrs, mask_type, metadata)

    node_mask = edge_mask = None
    if mask_type == MaskLevelType.edge:
        edge_mask = captum_attrs[0].squeeze(0)
    elif mask_type == MaskLevelType.node:
        node_mask = captum_attrs[0].squeeze(0)
    else:
        node_mask = captum_attrs[0].squeeze(0)
        edge_mask = captum_attrs[1].squeeze(0)

    return node_mask, edge_mask
pytorch/torch_geometric/explain/algorithm/captum_explainer.py000066400000000000000000000146021444262304300252700ustar00rootroot00000000000000import inspect
import logging
import warnings
from typing import Any, Dict, Optional, Union

import torch
from torch import Tensor

from torch_geometric.explain import Explanation, HeteroExplanation
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.algorithm.captum import (
    CaptumHeteroModel,
    CaptumModel,
    MaskLevelType,
    convert_captum_output,
    to_captum_input,
)
from torch_geometric.explain.config import MaskType, ModelMode
from torch_geometric.typing import EdgeType, NodeType


class CaptumExplainer(ExplainerAlgorithm):
    """A `Captum `__-based explainer for identifying compact
    subgraph structures and node features that play a crucial role in the
    predictions made by a GNN.

    This explainer algorithm uses :captum:`null` `Captum `_
    to compute attributions.

    Currently, the following attribution methods are supported:

    * :class:`captum.attr.IntegratedGradients`
    * :class:`captum.attr.Saliency`
    * :class:`captum.attr.InputXGradient`
    * :class:`captum.attr.Deconvolution`
    * :class:`captum.attr.ShapleyValueSampling`
    * :class:`captum.attr.GuidedBackprop`

    Args:
        attribution_method (Attribution or str): The Captum attribution method
            to use. Can be a string or a :class:`captum.attr` method.
        **kwargs: Additional arguments for the Captum attribution method.
    """
    SUPPORTED_METHODS = [  # TODO: Add support for more methods.
        'IntegratedGradients',
        'Saliency',
        'InputXGradient',
        'Deconvolution',
        'ShapleyValueSampling',
        'GuidedBackprop',
    ]

    def __init__(
        self,
        attribution_method: Union[str, Any],
        **kwargs,
    ):
        super().__init__()

        import captum.attr  # noqa

        if isinstance(attribution_method, str):
            self.attribution_method = getattr(
                captum.attr,
                attribution_method,
            )
        else:
            self.attribution_method = attribution_method

        if not self._is_supported_attribution_method():
            raise ValueError(f"{self.__class__.__name__} does not support "
                             f"attribution method "
                             f"{self.attribution_method.__name__}")

        if kwargs.get('internal_batch_size', 1) != 1:
            warnings.warn("Overriding 'internal_batch_size' to 1")

        if 'internal_batch_size' in self._get_attribute_parameters():
            kwargs['internal_batch_size'] = 1

        self.kwargs = kwargs

    def _get_mask_type(self) -> MaskLevelType:
        r"""Based on the explainer config, return the mask type."""
        node_mask_type = self.explainer_config.node_mask_type
        edge_mask_type = self.explainer_config.edge_mask_type
        if node_mask_type is not None and edge_mask_type is not None:
            mask_type = MaskLevelType.node_and_edge
        elif node_mask_type is not None:
            mask_type = MaskLevelType.node
        elif edge_mask_type is not None:
            mask_type = MaskLevelType.edge
        else:
            raise ValueError("Neither node mask type nor "
                             "edge mask type is specified.")
        return mask_type

    def _get_attribute_parameters(self) -> Dict[str, Any]:
        r"""Returns the attribute arguments."""
        signature = inspect.signature(self.attribution_method.attribute)
        return signature.parameters

    def _needs_baseline(self) -> bool:
        r"""Checks if the method needs a baseline."""
        parameters = self._get_attribute_parameters()
        if 'baselines' in parameters:
            param = parameters['baselines']
            if param.default is inspect.Parameter.empty:
                return True
        return False

    def _is_supported_attribution_method(self) -> bool:
        r"""Returns :obj:`True` if `self.attribution_method` is supported."""
        # This is redundant for now since all supported methods need a baseline
        if self._needs_baseline():
            return False
        elif self.attribution_method.__name__ in self.SUPPORTED_METHODS:
            return True
        return False

    def forward(
        self,
        model: torch.nn.Module,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Union[Explanation, HeteroExplanation]:

        mask_type = self._get_mask_type()

        inputs, add_forward_args = to_captum_input(
            x,
            edge_index,
            mask_type,
            *kwargs.values(),
        )

        if isinstance(x, dict):
            metadata = (list(x.keys()), list(edge_index.keys()))
            captum_model = CaptumHeteroModel(
                model,
                mask_type,
                index,
                metadata,
            )
        else:
            metadata = None
            captum_model = CaptumModel(model, mask_type, index)

        attribution_method = self.attribution_method(captum_model)

        # In captum, the target is the index for which
        # the attribution is computed.
        if self.model_config.mode == ModelMode.regression:
            target = None
        else:
            target = target[index]

        attributions = attribution_method.attribute(
            inputs=inputs,
            target=target,
            additional_forward_args=add_forward_args,
            **self.kwargs,
        )

        node_mask, edge_mask = convert_captum_output(
            attributions,
            mask_type,
            metadata,
        )

        if not isinstance(x, dict):
            return Explanation(node_mask=node_mask, edge_mask=edge_mask)

        explanation = HeteroExplanation()
        explanation.set_value_dict('node_mask', node_mask)
        explanation.set_value_dict('edge_mask', edge_mask)
        return explanation

    def supports(self) -> bool:
        node_mask_type = self.explainer_config.node_mask_type
        if node_mask_type not in [None, MaskType.attributes]:
            logging.error(f"'{self.__class__.__name__}' only supports "
                          f"'node_mask_type' None or 'attributes' "
                          f"(got '{node_mask_type.value}')")
            return False

        # TODO (ramona): Confirm that output type is valid.
        return True
pytorch/torch_geometric/explain/algorithm/dummy_explainer.py000066400000000000000000000054631444262304300251370ustar00rootroot00000000000000from collections import defaultdict
from typing import Dict, Optional, Union

import torch
from torch import Tensor

from torch_geometric.explain import Explanation, HeteroExplanation
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.config import MaskType
from torch_geometric.typing import EdgeType, NodeType


class DummyExplainer(ExplainerAlgorithm):
    r"""A dummy explainer that returns random explanations (useful for testing
    purposes)."""
    def forward(
        self,
        model: torch.nn.Module,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        edge_attr: Optional[Union[Tensor, Dict[EdgeType, Tensor]]] = None,
        **kwargs,
    ) -> Union[Explanation, HeteroExplanation]:
        assert isinstance(x, (Tensor, dict))

        node_mask_type = self.explainer_config.node_mask_type
        edge_mask_type = self.explainer_config.edge_mask_type

        if isinstance(x, Tensor):  # Homogeneous graph.
            assert isinstance(edge_index, Tensor)

            node_mask = None
            if node_mask_type == MaskType.object:
                node_mask = torch.rand(x.size(0), 1, device=x.device)
            elif node_mask_type == MaskType.common_attributes:
                node_mask = torch.rand(1, x.size(1), device=x.device)
            elif node_mask_type == MaskType.attributes:
                node_mask = torch.rand_like(x)

            edge_mask = None
            if edge_mask_type == MaskType.object:
                edge_mask = torch.rand(edge_index.size(1), device=x.device)

            return Explanation(node_mask=node_mask, edge_mask=edge_mask)
        else:  # isinstance(x, dict):  # Heterogeneous graph.
            assert isinstance(edge_index, dict)

            node_dict = defaultdict(dict)
            for k, v in x.items():
                node_mask = None
                if node_mask_type == MaskType.object:
                    node_mask = torch.rand(v.size(0), 1, device=v.device)
                elif node_mask_type == MaskType.common_attributes:
                    node_mask = torch.rand(1, v.size(1), device=v.device)
                elif node_mask_type == MaskType.attributes:
                    node_mask = torch.rand_like(v)
                if node_mask is not None:
                    node_dict[k]['node_mask'] = node_mask

            edge_dict = defaultdict(dict)
            for k, v in edge_index.items():
                edge_mask = None
                if edge_mask_type == MaskType.object:
                    edge_mask = torch.rand(v.size(1), device=v.device)
                if edge_mask is not None:
                    edge_dict[k]['edge_mask'] = edge_mask

            return HeteroExplanation({**node_dict, **edge_dict})

    def supports(self) -> bool:
        return True
pytorch/torch_geometric/explain/algorithm/gnn_explainer.py000066400000000000000000000271341444262304300245650ustar00rootroot00000000000000from math import sqrt
from typing import Optional, Tuple, Union

import torch
from torch import Tensor
from torch.nn.parameter import Parameter

from torch_geometric.explain import ExplainerConfig, Explanation, ModelConfig
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.algorithm.utils import clear_masks, set_masks
from torch_geometric.explain.config import MaskType, ModelMode, ModelTaskLevel


class GNNExplainer(ExplainerAlgorithm):
    r"""The GNN-Explainer model from the `"GNNExplainer: Generating
    Explanations for Graph Neural Networks"
    `_ paper for identifying compact subgraph
    structures and node features that play a crucial role in the predictions
    made by a GNN.

    .. note::

        For an example of using :class:`GNNExplainer`, see
        `examples/explain/gnn_explainer.py `_,
        `examples/explain/gnn_explainer_ba_shapes.py `_, and `examples/explain/
        gnn_explainer_link_pred.py `_.

    Args:
        epochs (int, optional): The number of epochs to train.
            (default: :obj:`100`)
        lr (float, optional): The learning rate to apply.
            (default: :obj:`0.01`)
        **kwargs (optional): Additional hyper-parameters to override default
            settings in
            :attr:`~torch_geometric.explain.algorithm.GNNExplainer.coeffs`.
    """

    coeffs = {
        'edge_size': 0.005,
        'edge_reduction': 'sum',
        'node_feat_size': 1.0,
        'node_feat_reduction': 'mean',
        'edge_ent': 1.0,
        'node_feat_ent': 0.1,
        'EPS': 1e-15,
    }

    def __init__(self, epochs: int = 100, lr: float = 0.01, **kwargs):
        super().__init__()
        self.epochs = epochs
        self.lr = lr
        self.coeffs.update(kwargs)

        self.node_mask = self.hard_node_mask = None
        self.edge_mask = self.hard_edge_mask = None

    def forward(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Explanation:
        if isinstance(x, dict) or isinstance(edge_index, dict):
            raise ValueError(f"Heterogeneous graphs not yet supported in "
                             f"'{self.__class__.__name__}'")

        self._train(model, x, edge_index, target=target, index=index, **kwargs)

        node_mask = self._post_process_mask(
            self.node_mask,
            self.hard_node_mask,
            apply_sigmoid=True,
        )
        edge_mask = self._post_process_mask(
            self.edge_mask,
            self.hard_edge_mask,
            apply_sigmoid=True,
        )

        self._clean_model(model)

        return Explanation(node_mask=node_mask, edge_mask=edge_mask)

    def supports(self) -> bool:
        return True

    def _train(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ):
        self._initialize_masks(x, edge_index)

        parameters = []
        if self.node_mask is not None:
            parameters.append(self.node_mask)
        if self.edge_mask is not None:
            set_masks(model, self.edge_mask, edge_index, apply_sigmoid=True)
            parameters.append(self.edge_mask)

        optimizer = torch.optim.Adam(parameters, lr=self.lr)

        for i in range(self.epochs):
            optimizer.zero_grad()

            h = x if self.node_mask is None else x * self.node_mask.sigmoid()
            y_hat, y = model(h, edge_index, **kwargs), target

            if index is not None:
                y_hat, y = y_hat[index], y[index]

            loss = self._loss(y_hat, y)

            loss.backward()
            optimizer.step()

            # In the first iteration, we collect the nodes and edges that are
            # involved into making the prediction. These are all the nodes and
            # edges with gradient != 0 (without regularization applied).
            if i == 0 and self.node_mask is not None:
                if self.node_mask.grad is None:
                    raise ValueError("Could not compute gradients for node "
                                     "features. Please make sure that node "
                                     "features are used inside the model or "
                                     "disable it via `node_mask_type=None`.")
                self.hard_node_mask = self.node_mask.grad != 0.0
            if i == 0 and self.edge_mask is not None:
                if self.edge_mask.grad is None:
                    raise ValueError("Could not compute gradients for edges. "
                                     "Please make sure that edges are used "
                                     "via message passing inside the model or "
                                     "disable it via `edge_mask_type=None`.")
                self.hard_edge_mask = self.edge_mask.grad != 0.0

    def _initialize_masks(self, x: Tensor, edge_index: Tensor):
        node_mask_type = self.explainer_config.node_mask_type
        edge_mask_type = self.explainer_config.edge_mask_type

        device = x.device
        (N, F), E = x.size(), edge_index.size(1)

        std = 0.1
        if node_mask_type is None:
            self.node_mask = None
        elif node_mask_type == MaskType.object:
            self.node_mask = Parameter(torch.randn(N, 1, device=device) * std)
        elif node_mask_type == MaskType.attributes:
            self.node_mask = Parameter(torch.randn(N, F, device=device) * std)
        elif node_mask_type == MaskType.common_attributes:
            self.node_mask = Parameter(torch.randn(1, F, device=device) * std)
        else:
            assert False

        if edge_mask_type is None:
            self.edge_mask = None
        elif edge_mask_type == MaskType.object:
            std = torch.nn.init.calculate_gain('relu') * sqrt(2.0 / (2 * N))
            self.edge_mask = Parameter(torch.randn(E, device=device) * std)
        else:
            assert False

    def _loss(self, y_hat: Tensor, y: Tensor) -> Tensor:
        if self.model_config.mode == ModelMode.binary_classification:
            loss = self._loss_binary_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.multiclass_classification:
            loss = self._loss_multiclass_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.regression:
            loss = self._loss_regression(y_hat, y)
        else:
            assert False

        if self.hard_edge_mask is not None:
            assert self.edge_mask is not None
            m = self.edge_mask[self.hard_edge_mask].sigmoid()
            edge_reduce = getattr(torch, self.coeffs['edge_reduction'])
            loss = loss + self.coeffs['edge_size'] * edge_reduce(m)
            ent = -m * torch.log(m + self.coeffs['EPS']) - (
                1 - m) * torch.log(1 - m + self.coeffs['EPS'])
            loss = loss + self.coeffs['edge_ent'] * ent.mean()

        if self.hard_node_mask is not None:
            assert self.node_mask is not None
            m = self.node_mask[self.hard_node_mask].sigmoid()
            node_reduce = getattr(torch, self.coeffs['node_feat_reduction'])
            loss = loss + self.coeffs['node_feat_size'] * node_reduce(m)
            ent = -m * torch.log(m + self.coeffs['EPS']) - (
                1 - m) * torch.log(1 - m + self.coeffs['EPS'])
            loss = loss + self.coeffs['node_feat_ent'] * ent.mean()

        return loss

    def _clean_model(self, model):
        clear_masks(model)
        self.node_mask = self.hard_node_mask = None
        self.edge_mask = self.hard_edge_mask = None


class GNNExplainer_:
    r"""Deprecated version for :class:`GNNExplainer`."""

    coeffs = GNNExplainer.coeffs

    conversion_node_mask_type = {
        'feature': 'common_attributes',
        'individual_feature': 'attributes',
        'scalar': 'object',
    }

    conversion_return_type = {
        'log_prob': 'log_probs',
        'prob': 'probs',
        'raw': 'raw',
        'regression': 'raw',
    }

    def __init__(
        self,
        model: torch.nn.Module,
        epochs: int = 100,
        lr: float = 0.01,
        return_type: str = 'log_prob',
        feat_mask_type: str = 'feature',
        allow_edge_mask: bool = True,
        **kwargs,
    ):
        assert feat_mask_type in ['feature', 'individual_feature', 'scalar']

        explainer_config = ExplainerConfig(
            explanation_type='model',
            node_mask_type=self.conversion_node_mask_type[feat_mask_type],
            edge_mask_type=MaskType.object if allow_edge_mask else None,
        )
        model_config = ModelConfig(
            mode='regression'
            if return_type == 'regression' else 'multiclass_classification',
            task_level=ModelTaskLevel.node,
            return_type=self.conversion_return_type[return_type],
        )

        self.model = model
        self._explainer = GNNExplainer(epochs=epochs, lr=lr, **kwargs)
        self._explainer.connect(explainer_config, model_config)

    @torch.no_grad()
    def get_initial_prediction(self, *args, **kwargs) -> Tensor:

        training = self.model.training
        self.model.eval()

        out = self.model(*args, **kwargs)
        if (self._explainer.model_config.mode ==
                ModelMode.multiclass_classification):
            out = out.argmax(dim=-1)

        self.model.train(training)

        return out

    def explain_graph(
        self,
        x: Tensor,
        edge_index: Tensor,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        self._explainer.model_config.task_level = ModelTaskLevel.graph

        explanation = self._explainer(
            self.model,
            x,
            edge_index,
            target=self.get_initial_prediction(x, edge_index, **kwargs),
            **kwargs,
        )
        return self._convert_output(explanation, edge_index)

    def explain_node(
        self,
        node_idx: int,
        x: Tensor,
        edge_index: Tensor,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        self._explainer.model_config.task_level = ModelTaskLevel.node
        explanation = self._explainer(
            self.model,
            x,
            edge_index,
            target=self.get_initial_prediction(x, edge_index, **kwargs),
            index=node_idx,
            **kwargs,
        )
        return self._convert_output(explanation, edge_index, index=node_idx,
                                    x=x)

    def _convert_output(self, explanation, edge_index, index=None, x=None):
        node_mask = explanation.get('node_mask')
        edge_mask = explanation.get('edge_mask')

        if node_mask is not None:
            node_mask_type = self._explainer.explainer_config.node_mask_type
            if node_mask_type in {MaskType.object, MaskType.common_attributes}:
                node_mask = node_mask.view(-1)

        if edge_mask is None:
            if index is not None:
                _, edge_mask = self._explainer._get_hard_masks(
                    self.model, index, edge_index, num_nodes=x.size(0))
                edge_mask = edge_mask.to(x.dtype)
            else:
                edge_mask = torch.ones(edge_index.size(1),
                                       device=edge_index.device)

        return node_mask, edge_mask
pytorch/torch_geometric/explain/algorithm/pg_explainer.py000066400000000000000000000242021444262304300244020ustar00rootroot00000000000000import logging
from typing import Optional, Union

import torch
from torch import Tensor
from torch.nn import ReLU, Sequential

from torch_geometric.explain import Explanation
from torch_geometric.explain.algorithm import ExplainerAlgorithm
from torch_geometric.explain.algorithm.utils import clear_masks, set_masks
from torch_geometric.explain.config import (
    ExplanationType,
    ModelMode,
    ModelTaskLevel,
)
from torch_geometric.nn import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.utils import get_embeddings


class PGExplainer(ExplainerAlgorithm):
    r"""The PGExplainer model from the `"Parameterized Explainer for Graph
    Neural Network" `_ paper.
    Internally, it utilizes a neural network to identify subgraph structures
    that play a crucial role in the predictions made by a GNN.
    Importantly, the :class:`PGExplainer` needs to be trained via
    :meth:`~PGExplainer.train` before being able to generate explanations:

    .. code-block:: python

        explainer = Explainer(
            model=model,
            algorithm=PGExplainer(epochs=30, lr=0.003),
            explanation_type='phenomenon',
            edge_mask_type='object',
            model_config=ModelConfig(...),
        )

        # Train against a variety of node-level or graph-level predictions:
        for epoch in range(30):
            for index in [...]:  # Indices to train against.
                loss = explainer.algorithm.train(epoch, model, x, edge_index,
                                                 target=target, index=index)

        # Get the final explanations:
        explanation = explainer(x, edge_index, target=target, index=0)

    Args:
        epochs (int): The number of epochs to train.
        lr (float, optional): The learning rate to apply.
            (default: :obj:`0.003`).
        **kwargs (optional): Additional hyper-parameters to override default
            settings in
            :attr:`~torch_geometric.explain.algorithm.PGExplainer.coeffs`.
    """

    coeffs = {
        'edge_size': 0.05,
        'edge_ent': 1.0,
        'temp': [5.0, 2.0],
        'bias': 0.0,
    }

    def __init__(self, epochs: int, lr: float = 0.003, **kwargs):
        super().__init__()
        self.epochs = epochs
        self.lr = lr
        self.coeffs.update(kwargs)

        self.mlp = Sequential(
            Linear(-1, 64),
            ReLU(),
            Linear(64, 1),
        )
        self.optimizer = torch.optim.Adam(self.mlp.parameters(), lr=lr)
        self._curr_epoch = -1

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset(self.mlp)

    def train(
        self,
        epoch: int,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ):
        r"""Trains the underlying explainer model.
        Needs to be called before being able to make predictions.

        Args:
            epoch (int): The current epoch of the training phase.
            model (torch.nn.Module): The model to explain.
            x (torch.Tensor): The input node features of a
                homogeneous graph.
            edge_index (torch.Tensor): The input edge indices of a homogeneous
                graph.
            target (torch.Tensor): The target of the model.
            index (int or torch.Tensor, optional): The index of the model
                output to explain. Needs to be a single index.
                (default: :obj:`None`)
            **kwargs (optional): Additional keyword arguments passed to
                :obj:`model`.
        """
        if isinstance(x, dict) or isinstance(edge_index, dict):
            raise ValueError(f"Heterogeneous graphs not yet supported in "
                             f"'{self.__class__.__name__}'")

        if self.model_config.task_level == ModelTaskLevel.node:
            if index is None:
                raise ValueError(f"The 'index' argument needs to be provided "
                                 f"in '{self.__class__.__name__}' for "
                                 f"node-level explanations")
            if isinstance(index, Tensor) and index.numel() > 1:
                raise ValueError(f"Only scalars are supported for the 'index' "
                                 f"argument in '{self.__class__.__name__}'")

        z = get_embeddings(model, x, edge_index, **kwargs)[-1]

        self.optimizer.zero_grad()
        temperature = self._get_temperature(epoch)

        inputs = self._get_inputs(z, edge_index, index)
        logits = self.mlp(inputs).view(-1)
        edge_mask = self._concrete_sample(logits, temperature)
        set_masks(model, edge_mask, edge_index, apply_sigmoid=True)

        if self.model_config.task_level == ModelTaskLevel.node:
            _, hard_edge_mask = self._get_hard_masks(model, index, edge_index,
                                                     num_nodes=x.size(0))
            edge_mask = edge_mask[hard_edge_mask]

        y_hat, y = model(x, edge_index, **kwargs), target

        if index is not None:
            y_hat, y = y_hat[index], y[index]

        loss = self._loss(y_hat, y, edge_mask)
        loss.backward()
        self.optimizer.step()

        clear_masks(model)
        self._curr_epoch = epoch

        return float(loss)

    def forward(
        self,
        model: torch.nn.Module,
        x: Tensor,
        edge_index: Tensor,
        *,
        target: Tensor,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Explanation:
        if isinstance(x, dict) or isinstance(edge_index, dict):
            raise ValueError(f"Heterogeneous graphs not yet supported in "
                             f"'{self.__class__.__name__}'")

        if self._curr_epoch < self.epochs - 1:  # Safety check:
            raise ValueError(f"'{self.__class__.__name__}' is not yet fully "
                             f"trained (got {self._curr_epoch + 1} epochs "
                             f"from {self.epochs} epochs). Please first train "
                             f"the underlying explainer model by running "
                             f"`explainer.algorithm.train(...)`.")

        hard_edge_mask = None
        if self.model_config.task_level == ModelTaskLevel.node:
            if index is None:
                raise ValueError(f"The 'index' argument needs to be provided "
                                 f"in '{self.__class__.__name__}' for "
                                 f"node-level explanations")
            if isinstance(index, Tensor) and index.numel() > 1:
                raise ValueError(f"Only scalars are supported for the 'index' "
                                 f"argument in '{self.__class__.__name__}'")

            # We need to compute hard masks to properly clean up edges and
            # nodes attributions not involved during message passing:
            _, hard_edge_mask = self._get_hard_masks(model, index, edge_index,
                                                     num_nodes=x.size(0))

        z = get_embeddings(model, x, edge_index, **kwargs)[-1]

        inputs = self._get_inputs(z, edge_index, index)
        logits = self.mlp(inputs).view(-1)

        edge_mask = self._post_process_mask(logits, hard_edge_mask,
                                            apply_sigmoid=True)

        return Explanation(edge_mask=edge_mask)

    def supports(self) -> bool:
        explanation_type = self.explainer_config.explanation_type
        if explanation_type != ExplanationType.phenomenon:
            logging.error(f"'{self.__class__.__name__}' only supports "
                          f"phenomenon explanations "
                          f"got (`explanation_type={explanation_type.value}`)")
            return False

        task_level = self.model_config.task_level
        if task_level not in {ModelTaskLevel.node, ModelTaskLevel.graph}:
            logging.error(f"'{self.__class__.__name__}' only supports "
                          f"node-level or graph-level explanations "
                          f"got (`task_level={task_level.value}`)")
            return False

        node_mask_type = self.explainer_config.node_mask_type
        if node_mask_type is not None:
            logging.error(f"'{self.__class__.__name__}' does not support "
                          f"explaining input node features "
                          f"got (`node_mask_type={node_mask_type.value}`)")
            return False

        return True

    ###########################################################################

    def _get_inputs(self, embedding: Tensor, edge_index: Tensor,
                    index: Optional[int] = None) -> Tensor:
        zs = [embedding[edge_index[0]], embedding[edge_index[1]]]
        if self.model_config.task_level == ModelTaskLevel.node:
            assert index is not None
            zs.append(embedding[index].view(1, -1).repeat(zs[0].size(0), 1))
        return torch.cat(zs, dim=-1)

    def _get_temperature(self, epoch: int) -> float:
        temp = self.coeffs['temp']
        return temp[0] * pow(temp[1] / temp[0], epoch / self.epochs)

    def _concrete_sample(self, logits: Tensor,
                         temperature: float = 1.0) -> Tensor:
        bias = self.coeffs['bias']
        eps = (1 - 2 * bias) * torch.rand_like(logits) + bias
        return (eps.log() - (1 - eps).log() + logits) / temperature

    def _loss(self, y_hat: Tensor, y: Tensor, edge_mask: Tensor) -> Tensor:
        if self.model_config.mode == ModelMode.binary_classification:
            loss = self._loss_binary_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.multiclass_classification:
            loss = self._loss_multiclass_classification(y_hat, y)
        elif self.model_config.mode == ModelMode.regression:
            loss = self._loss_regression(y_hat, y)

        # Regularization loss:
        mask = edge_mask.sigmoid()
        size_loss = mask.sum() * self.coeffs['edge_size']
        mask = 0.99 * mask + 0.005
        mask_ent = -mask * mask.log() - (1 - mask) * (1 - mask).log()
        mask_ent_loss = mask_ent.mean() * self.coeffs['edge_ent']

        return loss + size_loss + mask_ent_loss
pytorch/torch_geometric/explain/algorithm/utils.py000066400000000000000000000044041444262304300230670ustar00rootroot00000000000000from typing import Dict, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn import MessagePassing
from torch_geometric.typing import EdgeType


def set_masks(
    model: torch.nn.Module,
    mask: Union[Tensor, Parameter],
    edge_index: Tensor,
    apply_sigmoid: bool = True,
):
    r"""Apply mask to every graph layer in the :obj:`model`."""
    loop_mask = edge_index[0] != edge_index[1]

    # Loop over layers and set masks on MessagePassing layers:
    for module in model.modules():
        if isinstance(module, MessagePassing):

            # Convert mask to a param if it was previously registered as one.
            # This is a workaround for the fact that PyTorch does not allow
            # assignments of pure tensors to parameter attributes:
            if (not isinstance(mask, Parameter)
                    and '_edge_mask' in module._parameters):
                mask = Parameter(mask)

            module.explain = True
            module._edge_mask = mask
            module._loop_mask = loop_mask
            module._apply_sigmoid = apply_sigmoid


def set_hetero_masks(
    model: torch.nn.Module,
    mask_dict: Dict[EdgeType, Union[Tensor, Parameter]],
    edge_index_dict: Dict[EdgeType, Tensor],
    apply_sigmoid: bool = True,
):
    r"""Apply masks to every heterogeneous graph layer in the :obj:`model`
    according to edge types."""
    for module in model.modules():
        if isinstance(module, torch.nn.ModuleDict):
            for edge_type in mask_dict.keys():
                # TODO (jinu) Use common function get `str_edge_type`.
                str_edge_type = '__'.join(edge_type)
                if str_edge_type in module:
                    set_masks(
                        module[str_edge_type],
                        mask_dict[edge_type],
                        edge_index_dict[edge_type],
                        apply_sigmoid=apply_sigmoid,
                    )


def clear_masks(model: torch.nn.Module):
    r"""Clear all masks from the model."""
    for module in model.modules():
        if isinstance(module, MessagePassing):
            module.explain = False
            module._edge_mask = None
            module._loop_mask = None
            module._apply_sigmoid = True
    return module
pytorch/torch_geometric/explain/config.py000066400000000000000000000172321444262304300212110ustar00rootroot00000000000000from dataclasses import dataclass
from enum import Enum
from typing import Optional, Union

from torch_geometric.utils.mixin import CastMixin


class ExplanationType(Enum):
    """Enum class for the explanation type."""
    model = 'model'
    phenomenon = 'phenomenon'


class MaskType(Enum):
    """Enum class for the mask type."""
    object = 'object'
    common_attributes = 'common_attributes'
    attributes = 'attributes'


class ModelMode(Enum):
    """Enum class for the model return type."""
    binary_classification = 'binary_classification'
    multiclass_classification = 'multiclass_classification'
    regression = 'regression'


class ModelTaskLevel(Enum):
    """Enum class for the model task level."""
    node = 'node'
    edge = 'edge'
    graph = 'graph'


class ModelReturnType(Enum):
    """Enum class for the model return type."""
    raw = 'raw'
    probs = 'probs'
    log_probs = 'log_probs'


class ThresholdType(Enum):
    """Enum class for the threshold type."""
    hard = 'hard'
    topk = 'topk'
    topk_hard = 'topk_hard'
    # connected = 'connected'  # TODO


@dataclass
class ExplainerConfig(CastMixin):
    r"""Configuration class to store and validate high level explanation
    parameters.

    Args:
        explanation_type (ExplanationType or str): The type of explanation to
            compute. The possible values are:

                - :obj:`"model"`: Explains the model prediction.

                - :obj:`"phenomenon"`: Explains the phenomenon that the model
                  is trying to predict.

            In practice, this means that the explanation algorithm will either
            compute their losses with respect to the model output
            (:obj:`"model"`) or the target output (:obj:`"phenomenon"`).

        node_mask_type (MaskType or str, optional): The type of mask to apply
            on nodes. The possible values are (default: :obj:`None`):

                - :obj:`None`: Will not apply any mask on nodes.

                - :obj:`"object"`: Will mask each node.

                - :obj:`"common_attributes"`: Will mask each feature.

                - :obj:`"attributes"`: Will mask each feature across all nodes.

        edge_mask_type (MaskType or str, optional): The type of mask to apply
            on edges. Has the sample possible values as :obj:`node_mask_type`.
            (default: :obj:`None`)
    """
    explanation_type: ExplanationType
    node_mask_type: Optional[MaskType]
    edge_mask_type: Optional[MaskType]

    def __init__(
        self,
        explanation_type: Union[ExplanationType, str],
        node_mask_type: Optional[Union[MaskType, str]] = None,
        edge_mask_type: Optional[Union[MaskType, str]] = None,
    ):
        if node_mask_type is not None:
            node_mask_type = MaskType(node_mask_type)
        if edge_mask_type is not None:
            edge_mask_type = MaskType(edge_mask_type)

        if edge_mask_type is not None and edge_mask_type != MaskType.object:
            raise ValueError(f"'edge_mask_type' needs be None or of type "
                             f"'object' (got '{edge_mask_type.value}')")

        if node_mask_type is None and edge_mask_type is None:
            raise ValueError("Either 'node_mask_type' or 'edge_mask_type' "
                             "must be provided")

        self.explanation_type = ExplanationType(explanation_type)
        self.node_mask_type = node_mask_type
        self.edge_mask_type = edge_mask_type


@dataclass
class ModelConfig(CastMixin):
    r"""Configuration class to store model parameters.

    Args:
        mode (ModelMode or str): The mode of the model. The possible values
            are:

                - :obj:`"binary_classification"`: A binary classification
                  model.

                - :obj:`"multiclass_classification"`: A multiclass
                  classification model.

                - :obj:`"regression"`: A regression model.

        task_level (ModelTaskLevel or str): The task-level of the model.
            The possible values are:

                - :obj:`"node"`: A node-level prediction model.

                - :obj:`"edge"`: An edge-level prediction model.

                - :obj:`"graph"`: A graph-level prediction model.

        return_type (ModelReturnType or str, optional): The return type of the
            model. The possible values are (default: :obj:`None`):

                - :obj:`"raw"`: The model returns raw values.

                - :obj:`"probs"`: The model returns probabilities.

                - :obj:`"log_probs"`: The model returns log-probabilities.
    """
    mode: ModelMode
    task_level: ModelTaskLevel
    return_type: ModelReturnType

    def __init__(
        self,
        mode: Union[ModelMode, str],
        task_level: Union[ModelTaskLevel, str],
        return_type: Optional[Union[ModelReturnType, str]] = None,
    ):
        self.mode = ModelMode(mode)
        self.task_level = ModelTaskLevel(task_level)

        if return_type is None and self.mode == ModelMode.regression:
            return_type = ModelReturnType.raw

        self.return_type = ModelReturnType(return_type)

        if (self.mode == ModelMode.regression
                and self.return_type != ModelReturnType.raw):
            raise ValueError(f"A model for regression needs to return raw "
                             f"outputs (got {self.return_type.value})")

        if (self.mode == ModelMode.binary_classification and self.return_type
                not in [ModelReturnType.raw, ModelReturnType.probs]):
            raise ValueError(
                f"A model for binary classification needs to return raw "
                f"outputs or probabilities (got {self.return_type.value})")


@dataclass
class ThresholdConfig(CastMixin):
    r"""Configuration class to store and validate threshold parameters.

    Args:
        threshold_type (ThresholdType or str): The type of threshold to apply.
            The possible values are:

                - :obj:`None`: No threshold is applied.

                - :obj:`"hard"`: A hard threshold is applied to each mask.
                  The elements of the mask with a value below the :obj:`value`
                  are set to :obj:`0`, the others are set to :obj:`1`.

                - :obj:`"topk"`: A soft threshold is applied to each mask.
                  The top obj:`value` elements of each mask are kept, the
                  others are set to :obj:`0`.

                - :obj:`"topk_hard"`: Same as :obj:`"topk"` but values are set
                  to :obj:`1` for all elements which are kept.

        value (int or float, optional): The value to use when thresholding.
            (default: :obj:`None`)
    """
    type: ThresholdType
    value: Union[float, int]

    def __init__(
        self,
        threshold_type: Union[ThresholdType, str],
        value: Union[float, int],
    ):
        self.type = ThresholdType(threshold_type)
        self.value = value

        if not isinstance(self.value, (int, float)):
            raise ValueError(f"Threshold value must be a float or int "
                             f"(got {type(self.value)}).")

        if (self.type == ThresholdType.hard
                and (self.value < 0 or self.value > 1)):
            raise ValueError(f"Threshold value must be between 0 and 1 "
                             f"(got {self.value})")

        if self.type in [ThresholdType.topk, ThresholdType.topk_hard]:
            if not isinstance(self.value, int):
                raise ValueError(f"Threshold value needs to be an integer "
                                 f"(got {type(self.value)}).")
            if self.value <= 0:
                raise ValueError(f"Threshold value needs to be positive "
                                 f"(got {self.value}).")
pytorch/torch_geometric/explain/explainer.py000066400000000000000000000242071444262304300217330ustar00rootroot00000000000000import warnings
from typing import Any, Dict, Optional, Union

import torch
from torch import Tensor

from torch_geometric.explain import (
    ExplainerAlgorithm,
    Explanation,
    HeteroExplanation,
)
from torch_geometric.explain.algorithm.utils import (
    clear_masks,
    set_hetero_masks,
    set_masks,
)
from torch_geometric.explain.config import (
    ExplainerConfig,
    ExplanationType,
    MaskType,
    ModelConfig,
    ModelMode,
    ModelReturnType,
    ThresholdConfig,
)
from torch_geometric.typing import EdgeType, NodeType


class Explainer:
    r"""An explainer class for instance-level explanations of Graph Neural
    Networks.

    Args:
        model (torch.nn.Module): The model to explain.
        algorithm (ExplainerAlgorithm): The explanation algorithm.
        explanation_type (ExplanationType or str): The type of explanation to
            compute. The possible values are:

                - :obj:`"model"`: Explains the model prediction.

                - :obj:`"phenomenon"`: Explains the phenomenon that the model
                  is trying to predict.

            In practice, this means that the explanation algorithm will either
            compute their losses with respect to the model output
            (:obj:`"model"`) or the target output (:obj:`"phenomenon"`).
        model_config (ModelConfig): The model configuration.
            See :class:`~torch_geometric.explain.config.ModelConfig` for
            available options. (default: :obj:`None`)
        node_mask_type (MaskType or str, optional): The type of mask to apply
            on nodes. The possible values are (default: :obj:`None`):

                - :obj:`None`: Will not apply any mask on nodes.

                - :obj:`"object"`: Will mask each node.

                - :obj:`"common_attributes"`: Will mask each feature.

                - :obj:`"attributes"`: Will mask each feature across all nodes.

        edge_mask_type (MaskType or str, optional): The type of mask to apply
            on edges. Has the sample possible values as :obj:`node_mask_type`.
            (default: :obj:`None`)
        threshold_config (ThresholdConfig, optional): The threshold
            configuration.
            See :class:`~torch_geometric.explain.config.ThresholdConfig` for
            available options. (default: :obj:`None`)
    """
    def __init__(
        self,
        model: torch.nn.Module,
        algorithm: ExplainerAlgorithm,
        explanation_type: Union[ExplanationType, str],
        model_config: Union[ModelConfig, Dict[str, Any]],
        node_mask_type: Optional[Union[MaskType, str]] = None,
        edge_mask_type: Optional[Union[MaskType, str]] = None,
        threshold_config: Optional[ThresholdConfig] = None,
    ):
        explainer_config = ExplainerConfig(
            explanation_type=explanation_type,
            node_mask_type=node_mask_type,
            edge_mask_type=edge_mask_type,
        )

        self.model = model
        self.algorithm = algorithm

        self.explanation_type = explainer_config.explanation_type
        self.model_config = ModelConfig.cast(model_config)
        self.node_mask_type = explainer_config.node_mask_type
        self.edge_mask_type = explainer_config.edge_mask_type
        self.threshold_config = ThresholdConfig.cast(threshold_config)

        self.algorithm.connect(explainer_config, self.model_config)

    @torch.no_grad()
    def get_prediction(self, *args, **kwargs) -> Tensor:
        r"""Returns the prediction of the model on the input graph.

        If the model mode is :obj:`"regression"`, the prediction is returned as
        a scalar value.
        If the model mode is :obj:`"multiclass_classification"` or
        :obj:`"binary_classification"`, the prediction is returned as the
        predicted class label.

        Args:
            *args: Arguments passed to the model.
            **kwargs (optional): Additional keyword arguments passed to the
                model.
        """
        training = self.model.training
        self.model.eval()

        with torch.no_grad():
            out = self.model(*args, **kwargs)

        self.model.train(training)

        return out

    def get_masked_prediction(
        self,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        node_mask: Optional[Union[Tensor, Dict[NodeType, Tensor]]] = None,
        edge_mask: Optional[Union[Tensor, Dict[EdgeType, Tensor]]] = None,
        **kwargs,
    ) -> Tensor:
        r"""Returns the prediction of the model on the input graph with node
        and edge masks applied."""
        if isinstance(x, Tensor) and node_mask is not None:
            x = node_mask * x
        elif isinstance(x, dict) and node_mask is not None:
            x = {key: value * node_mask[key] for key, value in x.items()}

        if isinstance(edge_mask, Tensor):
            set_masks(self.model, edge_mask, edge_index, apply_sigmoid=False)
        elif isinstance(edge_mask, dict):
            set_hetero_masks(self.model, edge_mask, edge_index,
                             apply_sigmoid=False)

        out = self.get_prediction(x, edge_index, **kwargs)
        clear_masks(self.model)
        return out

    def __call__(
        self,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        *,
        target: Optional[Tensor] = None,
        index: Optional[Union[int, Tensor]] = None,
        **kwargs,
    ) -> Union[Explanation, HeteroExplanation]:
        r"""Computes the explanation of the GNN for the given inputs and
        target.

        .. note::

            If you get an error message like "Trying to backward through the
            graph a second time", make sure that the target you provided
            was computed with :meth:`torch.no_grad`.

        Args:
            x (Union[torch.Tensor, Dict[NodeType, torch.Tensor]]): The input
                node features of a homogeneous or heterogeneous graph.
            edge_index (Union[torch.Tensor, Dict[NodeType, torch.Tensor]]): The
                input edge indices of a homogeneous or heterogeneous graph.
            target (torch.Tensor): The target of the model.
                If the explanation type is :obj:`"phenomenon"`, the target has
                to be provided.
                If the explanation type is :obj:`"model"`, the target should be
                set to :obj:`None` and will get automatically inferred.
                (default: :obj:`None`)
            index (Union[int, Tensor], optional): The index of the model
                output to explain. Can be a single index or a tensor of
                indices. (default: :obj:`None`)
            **kwargs: additional arguments to pass to the GNN.
        """
        # Choose the `target` depending on the explanation type:
        prediction: Optional[Tensor] = None
        if self.explanation_type == ExplanationType.phenomenon:
            if target is None:
                raise ValueError(
                    f"The 'target' has to be provided for the explanation "
                    f"type '{self.explanation_type.value}'")
        elif self.explanation_type == ExplanationType.model:
            if target is not None:
                warnings.warn(
                    f"The 'target' should not be provided for the explanation "
                    f"type '{self.explanation_type.value}'")
            prediction = self.get_prediction(x, edge_index, **kwargs)
            target = self.get_target(prediction)

        training = self.model.training
        self.model.eval()

        explanation = self.algorithm(
            self.model,
            x,
            edge_index,
            target=target,
            index=index,
            **kwargs,
        )

        self.model.train(training)

        # Add explainer objectives to the `Explanation` object:
        explanation._model_config = self.model_config
        explanation.prediction = prediction
        explanation.target = target
        explanation.index = index

        # Add model inputs to the `Explanation` object:
        if isinstance(explanation, Explanation):
            explanation._model_args = list(kwargs.keys())
            explanation.x = x
            explanation.edge_index = edge_index

            for key, arg in kwargs.items():  # Add remaining `kwargs`:
                explanation[key] = arg

        elif isinstance(explanation, HeteroExplanation):
            # TODO Add `explanation._model_args`

            assert isinstance(x, dict)
            explanation.set_value_dict('x', x)

            assert isinstance(edge_index, dict)
            explanation.set_value_dict('edge_index', edge_index)

            for key, arg in kwargs.items():  # Add remaining `kwargs`:
                if isinstance(arg, dict):
                    # Keyword arguments are likely named `{attr_name}_dict`
                    # while we only want to assign the `{attr_name}` to the
                    # `HeteroExplanation` object:
                    key = key[:-5] if key.endswith('_dict') else key
                    explanation.set_value_dict(key, arg)
                else:
                    explanation[key] = arg

        explanation.validate_masks()
        return explanation.threshold(self.threshold_config)

    def get_target(self, prediction: Tensor) -> Tensor:
        r"""Returns the target of the model from a given prediction.

        If the model mode is of type :obj:`"regression"`, the prediction is
        returned as it is.
        If the model mode is of type :obj:`"multiclass_classification"` or
        :obj:`"binary_classification"`, the prediction is returned as the
        predicted class label.
        """
        if self.model_config.mode == ModelMode.binary_classification:
            # TODO: Allow customization of the thresholds used below.
            if self.model_config.return_type == ModelReturnType.raw:
                return (prediction > 0).long().view(-1)
            if self.model_config.return_type == ModelReturnType.probs:
                return (prediction > 0.5).long().view(-1)
            assert False

        if self.model_config.mode == ModelMode.multiclass_classification:
            return prediction.argmax(dim=-1)

        return prediction
pytorch/torch_geometric/explain/explanation.py000066400000000000000000000347721444262304300222760ustar00rootroot00000000000000import copy
from typing import Dict, List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.data.data import Data, warn_or_raise
from torch_geometric.data.hetero_data import HeteroData
from torch_geometric.explain.config import ThresholdConfig, ThresholdType
from torch_geometric.typing import EdgeType, NodeType
from torch_geometric.visualization import visualize_graph


class ExplanationMixin:
    @property
    def available_explanations(self) -> List[str]:
        """Returns the available explanation masks."""
        return [key for key in self.keys if key.endswith('_mask')]

    def validate_masks(self, raise_on_error: bool = True) -> bool:
        r"""Validates the correctness of the :class:`Explanation` masks."""
        status = True

        for store in self.node_stores:
            if 'node_mask' not in store:
                continue

            if store.node_mask.dim() != 2:
                status = False
                warn_or_raise(
                    f"Expected a 'node_mask' with two dimensions (got "
                    f"{store.node_mask.dim()} dimensions)", raise_on_error)

            if store.node_mask.size(0) not in {1, store.num_nodes}:
                status = False
                warn_or_raise(
                    f"Expected a 'node_mask' with {store.num_nodes} nodes "
                    f"(got {store.node_mask.size(0)} nodes)", raise_on_error)

            if 'x' in store:
                num_features = store.x.size(-1)
            else:
                num_features = store.node_mask.size(-1)

            if store.node_mask.size(1) not in {1, num_features}:
                status = False
                warn_or_raise(
                    f"Expected a 'node_mask' with {num_features} features ("
                    f"got {store.node_mask.size(1)} features)", raise_on_error)

        for store in self.edge_stores:
            if 'edge_mask' not in store:
                continue

            if store.edge_mask.dim() != 1:
                status = False
                warn_or_raise(
                    f"Expected an 'edge_mask' with one dimension (got "
                    f"{store.edge_mask.dim()} dimensions)", raise_on_error)

            if store.edge_mask.size(0) != store.num_edges:
                status = False
                warn_or_raise(
                    f"Expected an 'edge_mask' with {store.num_edges} edges "
                    f"(got {store.edge_mask.size(0)} edges)", raise_on_error)

        return status

    def _threshold_mask(
        self,
        mask: Optional[Tensor],
        threshold_config: ThresholdConfig,
    ) -> Optional[Tensor]:

        if mask is None:
            return None

        if threshold_config.type == ThresholdType.hard:
            return (mask > threshold_config.value).float()

        if threshold_config.type in [
                ThresholdType.topk,
                ThresholdType.topk_hard,
        ]:
            if threshold_config.value >= mask.numel():
                if threshold_config.type == ThresholdType.topk:
                    return mask
                else:
                    return torch.ones_like(mask)

            value, index = torch.topk(
                mask.flatten(),
                k=threshold_config.value,
            )

            out = torch.zeros_like(mask.flatten())
            if threshold_config.type == ThresholdType.topk:
                out[index] = value
            else:
                out[index] = 1.0
            return out.view(mask.size())

        assert False

    def threshold(
        self,
        *args,
        **kwargs,
    ) -> Union['Explanation', 'HeteroExplanation']:
        """Thresholds the explanation masks according to the thresholding
        method.

        Args:
            threshold_config (ThresholdConfig): The threshold configuration.
        """
        threshold_config = ThresholdConfig.cast(*args, **kwargs)

        if threshold_config is None:
            return self

        # Avoid modification of the original explanation:
        out = copy.copy(self)

        for store in out.node_stores:
            store.node_mask = self._threshold_mask(store.get('node_mask'),
                                                   threshold_config)

        for store in out.edge_stores:
            store.edge_mask = self._threshold_mask(store.get('edge_mask'),
                                                   threshold_config)

        return out


class Explanation(Data, ExplanationMixin):
    r"""Holds all the obtained explanations of a homogeneous graph.

    The explanation object is a :obj:`~torch_geometric.data.Data` object and
    can hold node attributions and edge attributions.
    It can also hold the original graph if needed.

    Args:
        node_mask (Tensor, optional): Node-level mask with shape
            :obj:`[num_nodes, 1]`, :obj:`[1, num_features]` or
            :obj:`[num_nodes, num_features]`. (default: :obj:`None`)
        edge_mask (Tensor, optional): Edge-level mask with shape
            :obj:`[num_edges]`. (default: :obj:`None`)
        **kwargs (optional): Additional attributes.
    """
    def validate(self, raise_on_error: bool = True) -> bool:
        r"""Validates the correctness of the :class:`Explanation` object."""
        status = super().validate(raise_on_error)
        status &= self.validate_masks(raise_on_error)
        return status

    def get_explanation_subgraph(self) -> 'Explanation':
        r"""Returns the induced subgraph, in which all nodes and edges with
        zero attribution are masked out."""
        node_mask = self.get('node_mask')
        if node_mask is not None:
            node_mask = node_mask.sum(dim=-1) > 0
        edge_mask = self.get('edge_mask')
        if edge_mask is not None:
            edge_mask = edge_mask > 0
        return self._apply_masks(node_mask, edge_mask)

    def get_complement_subgraph(self) -> 'Explanation':
        r"""Returns the induced subgraph, in which all nodes and edges with any
        attribution are masked out."""
        node_mask = self.get('node_mask')
        if node_mask is not None:
            node_mask = node_mask.sum(dim=-1) == 0
        edge_mask = self.get('edge_mask')
        if edge_mask is not None:
            edge_mask = edge_mask == 0
        return self._apply_masks(node_mask, edge_mask)

    def _apply_masks(
        self,
        node_mask: Optional[Tensor] = None,
        edge_mask: Optional[Tensor] = None,
    ) -> 'Explanation':
        out = copy.copy(self)

        if edge_mask is not None:
            for key, value in self.items():
                if key == 'edge_index':
                    out.edge_index = value[:, edge_mask]
                elif self.is_edge_attr(key):
                    out[key] = value[edge_mask]

        if node_mask is not None:
            out = out.subgraph(node_mask)

        return out

    def visualize_feature_importance(
        self,
        path: Optional[str] = None,
        feat_labels: Optional[List[str]] = None,
        top_k: Optional[int] = None,
    ):
        r"""Creates a bar plot of the node feature importances by summing up
        the node mask across all nodes.

        Args:
            path (str, optional): The path to where the plot is saved.
                If set to :obj:`None`, will visualize the plot on-the-fly.
                (default: :obj:`None`)
            feat_labels (List[str], optional): The labels of features.
                (default :obj:`None`)
            top_k (int, optional): Top k features to plot. If :obj:`None`
                plots all features. (default: :obj:`None`)
        """
        node_mask = self.get('node_mask')
        if node_mask is None:
            raise ValueError(f"The attribute 'node_mask' is not available "
                             f"in '{self.__class__.__name__}' "
                             f"(got {self.available_explanations})")
        if node_mask.dim() != 2 or node_mask.size(1) <= 1:
            raise ValueError(f"Cannot compute feature importance for "
                             f"object-level 'node_mask' "
                             f"(got shape {node_mask.size()})")

        if feat_labels is None:
            feat_labels = range(node_mask.size(1))

        score = node_mask.sum(dim=0)

        return _visualize_score(score, feat_labels, path, top_k)

    def visualize_graph(self, path: Optional[str] = None,
                        backend: Optional[str] = None):
        r"""Visualizes the explanation graph with edge opacity corresponding to
        edge importance.

        Args:
            path (str, optional): The path to where the plot is saved.
                If set to :obj:`None`, will visualize the plot on-the-fly.
                (default: :obj:`None`)
            backend (str, optional): The graph drawing backend to use for
                visualization (:obj:`"graphviz"`, :obj:`"networkx"`).
                If set to :obj:`None`, will use the most appropriate
                visualization backend based on available system packages.
                (default: :obj:`None`)
        """
        edge_mask = self.get('edge_mask')
        if edge_mask is None:
            raise ValueError(f"The attribute 'edge_mask' is not available "
                             f"in '{self.__class__.__name__}' "
                             f"(got {self.available_explanations})")
        visualize_graph(self.edge_index, edge_mask, path, backend)


class HeteroExplanation(HeteroData, ExplanationMixin):
    r"""Holds all the obtained explanations of a heterogeneous graph.

    The explanation object is a :obj:`~torch_geometric.data.HeteroData` object
    and can hold node attributions and edge attributions.
    It can also hold the original graph if needed.
    """
    def validate(self, raise_on_error: bool = True) -> bool:
        r"""Validates the correctness of the :class:`Explanation` object."""
        status = super().validate(raise_on_error)
        status &= self.validate_masks(raise_on_error)
        return status

    def get_explanation_subgraph(self) -> 'HeteroExplanation':
        r"""Returns the induced subgraph, in which all nodes and edges with
        zero attribution are masked out."""
        return self._apply_masks(
            node_mask_dict={
                key: mask.sum(dim=-1) > 0
                for key, mask in self.node_mask_dict.items()
            },
            edge_mask_dict={
                key: mask > 0
                for key, mask in self.edge_mask_dict.items()
            },
        )

    def get_complement_subgraph(self) -> 'HeteroExplanation':
        r"""Returns the induced subgraph, in which all nodes and edges with any
        attribution are masked out."""
        return self._apply_masks(
            node_mask_dict={
                key: mask.sum(dim=-1) == 0
                for key, mask in self.node_mask_dict.items()
            },
            edge_mask_dict={
                key: mask == 0
                for key, mask in self.edge_mask_dict.items()
            },
        )

    def _apply_masks(
        self,
        node_mask_dict: Dict[NodeType, Tensor],
        edge_mask_dict: Dict[EdgeType, Tensor],
    ) -> 'HeteroExplanation':
        out = copy.copy(self)

        for edge_type, edge_mask in edge_mask_dict.items():
            for key, value in self[edge_type].items():
                if key == 'edge_index':
                    out[edge_type].edge_index = value[:, edge_mask]
                elif self[edge_type].is_edge_attr(key):
                    out[edge_type][key] = value[edge_mask]

        return out.subgraph(node_mask_dict)

    def visualize_feature_importance(
        self,
        path: Optional[str] = None,
        feat_labels: Optional[Dict[NodeType, List[str]]] = None,
        top_k: Optional[int] = None,
    ):
        r"""Creates a bar plot of the node feature importances by summing up
        node masks across all nodes for each node type.

        Args:
            path (str, optional): The path to where the plot is saved.
                If set to :obj:`None`, will visualize the plot on-the-fly.
                (default: :obj:`None`)
            feat_labels (Dict[NodeType, List[str]], optional): The labels of
                features for each node type. (default :obj:`None`)
            top_k (int, optional): Top k features to plot. If :obj:`None`
                plots all features. (default: :obj:`None`)
        """
        node_mask_dict = self.node_mask_dict
        if len(node_mask_dict) == 0:
            raise ValueError(f"The attribute 'node_mask' is not available "
                             f"in '{self.__class__.__name__}' "
                             f"(got {self.available_explanations})")
        for node_mask in node_mask_dict.values():
            if node_mask.dim() != 2 or node_mask.size(1) <= 1:
                raise ValueError(f"Cannot compute feature importance for "
                                 f"object-level 'node_mask' "
                                 f"(got shape {node_mask_dict.size()})")

        if feat_labels is None:
            feat_labels = {}
            for node_type, node_mask in node_mask_dict.items():
                feat_labels[node_type] = range(node_mask.size(1))

        score = torch.cat(
            [node_mask.sum(dim=0) for node_mask in node_mask_dict.values()],
            dim=0)

        all_feat_labels = []
        for node_type in node_mask_dict.keys():
            all_feat_labels += [
                f'{node_type}#{label}' for label in feat_labels[node_type]
            ]

        return _visualize_score(score, all_feat_labels, path, top_k)


def _visualize_score(
    score: torch.Tensor,
    labels: List[str],
    path: Optional[str] = None,
    top_k: Optional[int] = None,
):
    import matplotlib.pyplot as plt
    import pandas as pd

    if len(labels) != score.numel():
        raise ValueError(f"The number of labels (got {len(labels)}) must "
                         f"match the number of scores (got {score.numel()})")

    score = score.cpu().numpy()

    df = pd.DataFrame({'score': score}, index=labels)
    df = df.sort_values('score', ascending=False)
    df = df.round(decimals=3)

    if top_k is not None:
        df = df.head(top_k)
        title = f"Feature importance for top {len(df)} features"
    else:
        title = f"Feature importance for {len(df)} features"

    ax = df.plot(
        kind='barh',
        figsize=(10, 7),
        title=title,
        ylabel='Feature label',
        xlim=[0, float(df['score'].max()) + 0.3],
        legend=False,
    )
    plt.gca().invert_yaxis()
    ax.bar_label(container=ax.containers[0], label_type='edge')

    if path is not None:
        plt.savefig(path)
    else:
        plt.show()

    plt.close()
pytorch/torch_geometric/explain/metric/000077500000000000000000000000001444262304300206505ustar00rootroot00000000000000pytorch/torch_geometric/explain/metric/__init__.py000066400000000000000000000004551444262304300227650ustar00rootroot00000000000000from .basic import groundtruth_metrics
from .fidelity import fidelity, characterization_score, fidelity_curve_auc
from .faithfulness import unfaithfulness

__all__ = classes = [
    'groundtruth_metrics',
    'fidelity',
    'characterization_score',
    'fidelity_curve_auc',
    'unfaithfulness',
]
pytorch/torch_geometric/explain/metric/basic.py000066400000000000000000000035401444262304300223050ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

from torch import Tensor

METRICS = ['accuracy', 'recall', 'precision', 'f1_score', 'auroc']


def groundtruth_metrics(
    pred_mask: Tensor,
    target_mask: Tensor,
    metrics: Optional[Union[str, List[str]]] = None,
    threshold: float = 0.5,
) -> Union[float, Tuple[float, ...]]:
    r"""Compares and evaluates an explanation mask with the ground-truth
    explanation mask.

    Args:
        pred_mask (torch.Tensor): The prediction mask to evaluate.
        target_mask (torch.Tensor): The ground-truth target mask.
        metrics (str or List[str], optional). The metrics to return
            (:obj:`"accuracy"`, :obj:`"recall"`, :obj:`"precision"`,
            :obj:`"f1_score"`, :obj:`"auroc"`). (default: :obj:`["accuracy",
            "recall", "precision", "f1_score", "auroc"]`)
        threshold (float, optional): The threshold value to perform hard
            thresholding of :obj:`mask` and :obj:`groundtruth`.
            (default: :obj:`0.5`)
    """
    import torchmetrics

    if metrics is None:
        metrics = METRICS

    if isinstance(metrics, str):
        metrics = [metrics]

    if not isinstance(metrics, (tuple, list)):
        raise ValueError(f"Expected metrics to be a string or a list of "
                         f"strings (got {type(metrics)})")

    pred_mask = pred_mask.view(-1)
    target_mask = (target_mask >= threshold).view(-1)

    outs = []
    for metric in metrics:
        if metric not in METRICS:
            raise ValueError(f"Encountered invalid metric {metric}")

        fn = getattr(torchmetrics.functional, metric)
        if metric in {'auroc'}:
            out = fn(pred_mask, target_mask, 'binary')
        else:
            out = fn(pred_mask, target_mask, 'binary', threshold)

        outs.append(float(out))

    return tuple(outs) if len(outs) > 1 else outs[0]
pytorch/torch_geometric/explain/metric/faithfulness.py000066400000000000000000000057671444262304300237340ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F

from torch_geometric.explain import Explainer, Explanation
from torch_geometric.explain.config import MaskType, ModelMode, ModelReturnType


def unfaithfulness(
    explainer: Explainer,
    explanation: Explanation,
    top_k: Optional[int] = None,
) -> float:
    r"""Evaluates how faithful an :class:`~torch_geometric.explain.Explanation`
    is to an underyling GNN predictor, as described in the
    `"Evaluating Explainability for Graph Neural Networks"
    `_ paper.

    In particular, the graph explanation unfaithfulness metric is defined as

    .. math::
        \textrm{GEF}(y, \hat{y}) = 1 - \exp(- \textrm{KL}(y || \hat{y}))

    where :math:`y` refers to the prediction probability vector obtained from
    the original graph, and :math:`\hat{y}` refers to the prediction
    probability vector obtained from the masked subgraph.
    Finally, the Kullback-Leibler (KL) divergence score quantifies the distance
    between the two probability distributions.

    Args:
        explainer (Explainer): The explainer to evaluate.
        explanation (Explanation): The explanation to evaluate.
        top_k (int, optional): If set, will only keep the original values of
            the top-:math:`k` node features identified by an explanation.
            If set to :obj:`None`, will use :obj:`explanation.node_mask` as it
            is for masking node features. (default: :obj:`None`)
    """
    if explainer.model_config.mode == ModelMode.regression:
        raise ValueError("Fidelity not defined for 'regression' models")

    if top_k is not None and explainer.node_mask_type == MaskType.object:
        raise ValueError("Cannot apply top-k feature selection based on a "
                         "node mask of type 'object'")

    node_mask = explanation.get('node_mask')
    edge_mask = explanation.get('edge_mask')
    x, edge_index = explanation.x, explanation.edge_index
    kwargs = {key: explanation[key] for key in explanation._model_args}

    y = explanation.get('prediction')
    if y is None:  # == ExplanationType.phenomenon
        y = explainer.get_prediction(x, edge_index, **kwargs)

    if node_mask is not None and top_k is not None:
        feat_importance = node_mask.sum(dim=0)
        _, top_k_index = feat_importance.topk(top_k)
        node_mask = torch.zeros_like(node_mask)
        node_mask[:, top_k_index] = 1.0

    y_hat = explainer.get_masked_prediction(x, edge_index, node_mask,
                                            edge_mask, **kwargs)

    if explanation.get('index') is not None:
        y, y_hat = y[explanation.index], y_hat[explanation.index]

    if explainer.model_config.return_type == ModelReturnType.raw:
        y, y_hat = y.softmax(dim=-1), y_hat.softmax(dim=-1)
    elif explainer.model_config.return_type == ModelReturnType.log_probs:
        y, y_hat = y.exp(), y_hat.exp()

    kl_div = F.kl_div(y.log(), y_hat, reduction='batchmean')
    return 1 - float(torch.exp(-kl_div))
pytorch/torch_geometric/explain/metric/fidelity.py000066400000000000000000000140151444262304300230340ustar00rootroot00000000000000from typing import Tuple

import torch
from torch import Tensor

from torch_geometric.explain import Explainer, Explanation
from torch_geometric.explain.config import ExplanationType, ModelMode


def fidelity(
    explainer: Explainer,
    explanation: Explanation,
) -> Tuple[float, float]:
    r"""Evaluates the fidelity of an
    :class:`~torch_geometric.explain.Explainer` given an
    :class:`~torch_geometric.explain.Explanation`, as described in the
    `"GraphFramEx: Towards Systematic Evaluation of Explainability Methods for
    Graph Neural Networks" `_ paper.

    Fidelity evaluates the contribution of the produced explanatory subgraph
    to the initial prediction, either by giving only the subgraph to the model
    (fidelity-) or by removing it from the entire graph (fidelity+).
    The fidelity scores capture how good an explainable model reproduces the
    natural phenomenon or the GNN model logic.

    For **phenomenon** explanations, the fidelity scores are given by:

    .. math::
        \textrm{fid}_{+} &= \frac{1}{N} \sum_{i = 1}^N
        \| \mathbb{1}(\hat{y}_i = y_i) -
        \mathbb{1}( \hat{y}_i^{G_{C \setminus S}} = y_i) \|

        \textrm{fid}_{-} &= \frac{1}{N} \sum_{i = 1}^N
        \| \mathbb{1}(\hat{y}_i = y_i) -
        \mathbb{1}( \hat{y}_i^{G_S} = y_i) \|

    For **model** explanations, the fidelity scores are given by:

    .. math::
        \textrm{fid}_{+} &= 1 - \frac{1}{N} \sum_{i = 1}^N
        \mathbb{1}( \hat{y}_i^{G_{C \setminus S}} = \hat{y}_i)

        \textrm{fid}_{-} &= 1 - \frac{1}{N} \sum_{i = 1}^N
        \mathbb{1}( \hat{y}_i^{G_S} = \hat{y}_i)

    Args:
        explainer (Explainer): The explainer to evaluate.
        explanation (Explanation): The explanation to evaluate.
    """
    if explainer.model_config.mode == ModelMode.regression:
        raise ValueError("Fidelity not defined for 'regression' models")

    node_mask = explanation.get('node_mask')
    edge_mask = explanation.get('edge_mask')
    kwargs = {key: explanation[key] for key in explanation._model_args}

    y = explanation.target
    if explainer.explanation_type == ExplanationType.phenomenon:
        y_hat = explainer.get_prediction(
            explanation.x,
            explanation.edge_index,
            **kwargs,
        )
        y_hat = explainer.get_target(y_hat)

    explain_y_hat = explainer.get_masked_prediction(
        explanation.x,
        explanation.edge_index,
        node_mask,
        edge_mask,
        **kwargs,
    )
    explain_y_hat = explainer.get_target(explain_y_hat)

    complement_y_hat = explainer.get_masked_prediction(
        explanation.x,
        explanation.edge_index,
        1. - node_mask if node_mask is not None else None,
        1. - edge_mask if edge_mask is not None else None,
        **kwargs,
    )
    complement_y_hat = explainer.get_target(complement_y_hat)

    if explanation.get('index') is not None:
        y = y[explanation.index]
        if explainer.explanation_type == ExplanationType.phenomenon:
            y_hat = y_hat[explanation.index]
        explain_y_hat = explain_y_hat[explanation.index]
        complement_y_hat = complement_y_hat[explanation.index]

    if explainer.explanation_type == ExplanationType.model:
        pos_fidelity = 1. - (complement_y_hat == y).float().mean()
        neg_fidelity = 1. - (explain_y_hat == y).float().mean()
    else:
        pos_fidelity = ((y_hat == y).float() -
                        (complement_y_hat == y).float()).abs().mean()
        neg_fidelity = ((y_hat == y).float() -
                        (explain_y_hat == y).float()).abs().mean()

    return float(pos_fidelity), float(neg_fidelity)


def characterization_score(
    pos_fidelity: Tensor,
    neg_fidelity: Tensor,
    pos_weight: float = 0.5,
    neg_weight: float = 0.5,
) -> Tensor:
    r"""Returns the componentwise characterization score as described in the
    `"GraphFramEx: Towards Systematic Evaluation of Explainability Methods for
    Graph Neural Networks" `_ paper:

    ..  math::
       \textrm{charact} = \frac{w_{+} + w_{-}}{\frac{w_{+}}{\textrm{fid}_{+}} +
        \frac{w_{-}}{1 - \textrm{fid}_{-}}}

    Args:
        pos_fidelity (torch.Tensor): The positive fidelity
            :math:`\textrm{fid}_{+}`.
        neg_fidelity (torch.Tensor): The negative fidelity
            :math:`\textrm{fid}_{-}`.
        pos_weight (float, optional): The weight :math:`w_{+}` for
            :math:`\textrm{fid}_{+}`. (default: :obj:`0.5`)
        neg_weight (float, optional): The weight :math:`w_{-}` for
            :math:`\textrm{fid}_{-}`. (default: :obj:`0.5`)
    """
    if (pos_weight + neg_weight) != 1.0:
        raise ValueError(f"The weights need to sum up to 1 "
                         f"(got {pos_weight} and {neg_weight})")

    denom = (pos_weight / pos_fidelity) + (neg_weight / (1. - neg_fidelity))
    return 1. / denom


def fidelity_curve_auc(
    pos_fidelity: Tensor,
    neg_fidelity: Tensor,
    x: Tensor,
) -> Tensor:
    r"""Returns the AUC for the fidelity curve as described in the
    `"GraphFramEx: Towards Systematic Evaluation of Explainability Methods for
    Graph Neural Networks" `_ paper.

    More precisely, returns the AUC of

    .. math::
        f(x) = \frac{\textrm{fid}_{+}}{1 - \textrm{fid}_{-}}

    Args:
        pos_fidelity (torch.Tensor): The positive fidelity
            :math:`\textrm{fid}_{+}`.
        neg_fidelity (torch.Tensor): The negative fidelity
            :math:`\textrm{fid}_{-}`.
        x (torch.Tensor): Tensor containing the points on the :math:`x`-axis.
            Needs to be sorted in ascending order.
    """
    if torch.any(neg_fidelity == 1):
        raise ValueError("There exists negative fidelity values containing 1, "
                         "leading to a division by zero")

    y = pos_fidelity / (1. - neg_fidelity)
    return auc(x, y)


def auc(x: Tensor, y: Tensor) -> Tensor:
    if torch.any(x.diff() < 0):
        raise ValueError("'x' must be given in ascending order")
    return torch.trapezoid(y, x)
pytorch/torch_geometric/graphgym/000077500000000000000000000000001444262304300175435ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/__init__.py000066400000000000000000000034271444262304300216620ustar00rootroot00000000000000from .contrib import *  # noqa
from .models import *  # noqa
from .utils import *  # noqa
from .checkpoint import load_ckpt, save_ckpt, remove_ckpt, clean_ckpt
from .cmd_args import parse_args
from .config import (cfg, set_cfg, load_cfg, dump_cfg, set_run_dir,
                     set_out_dir, get_fname)
from .init import init_weights
from .loader import create_loader
from .logger import set_printing, create_logger
from .loss import compute_loss
from .model_builder import create_model
from .optim import create_optimizer, create_scheduler
from .train import train
from .register import (register_base, register_act, register_node_encoder,
                       register_edge_encoder, register_stage, register_head,
                       register_layer, register_pooling, register_network,
                       register_config, register_dataset, register_loader,
                       register_optimizer, register_scheduler, register_loss,
                       register_train, register_metric)

__all__ = classes = [
    'load_ckpt',
    'save_ckpt',
    'remove_ckpt',
    'clean_ckpt',
    'parse_args',
    'cfg',
    'set_cfg',
    'load_cfg',
    'dump_cfg',
    'set_run_dir',
    'set_out_dir',
    'get_fname',
    'init_weights',
    'create_loader',
    'set_printing',
    'create_logger',
    'compute_loss',
    'create_model',
    'create_optimizer',
    'create_scheduler',
    'train',
    'register_base',
    'register_act',
    'register_node_encoder',
    'register_edge_encoder',
    'register_stage',
    'register_head',
    'register_layer',
    'register_pooling',
    'register_network',
    'register_config',
    'register_dataset',
    'register_loader',
    'register_optimizer',
    'register_scheduler',
    'register_loss',
    'register_train',
    'register_metric',
]
pytorch/torch_geometric/graphgym/benchmark.py000066400000000000000000000007761444262304300220610ustar00rootroot00000000000000# Do not change; required for benchmarking

import torch_geometric_benchmark.torchprof_local as torchprof  # noqa
from pytorch_memlab import LineProfiler  # noqa
from torch_geometric_benchmark.utils import count_parameters  # noqa
from torch_geometric_benchmark.utils import get_gpu_memory_nvdia  # noqa
from torch_geometric_benchmark.utils import get_memory_status  # noqa
from torch_geometric_benchmark.utils import get_model_size  # noqa

global_line_profiler = LineProfiler()
global_line_profiler.enable()
pytorch/torch_geometric/graphgym/checkpoint.py000066400000000000000000000045031444262304300222460ustar00rootroot00000000000000import glob
import os
import os.path as osp
from typing import Any, Dict, List, Optional, Union

import torch

from torch_geometric.graphgym.config import cfg

MODEL_STATE = 'model_state'
OPTIMIZER_STATE = 'optimizer_state'
SCHEDULER_STATE = 'scheduler_state'


def load_ckpt(
    model: torch.nn.Module,
    optimizer: Optional[torch.optim.Optimizer] = None,
    scheduler: Optional[Any] = None,
    epoch: int = -1,
) -> int:
    r"""Loads the model checkpoint at a given epoch."""
    epoch = get_ckpt_epoch(epoch)
    path = get_ckpt_path(epoch)

    if not osp.exists(path):
        return 0

    ckpt = torch.load(path)
    model.load_state_dict(ckpt[MODEL_STATE])
    if optimizer is not None and OPTIMIZER_STATE in ckpt:
        optimizer.load_state_dict(ckpt[OPTIMIZER_STATE])
    if scheduler is not None and SCHEDULER_STATE in ckpt:
        scheduler.load_state_dict(ckpt[SCHEDULER_STATE])

    return epoch + 1


def save_ckpt(
    model: torch.nn.Module,
    optimizer: Optional[torch.optim.Optimizer] = None,
    scheduler: Optional[Any] = None,
    epoch: int = 0,
):
    r"""Saves the model checkpoint at a given epoch."""
    ckpt: Dict[str, Any] = {}
    ckpt[MODEL_STATE] = model.state_dict()
    if optimizer is not None:
        ckpt[OPTIMIZER_STATE] = optimizer.state_dict()
    if scheduler is not None:
        ckpt[SCHEDULER_STATE] = scheduler.state_dict()

    os.makedirs(get_ckpt_dir(), exist_ok=True)
    torch.save(ckpt, get_ckpt_path(get_ckpt_epoch(epoch)))


def remove_ckpt(epoch: int = -1):
    r"""Removes the model checkpoint at a given epoch."""
    os.remove(get_ckpt_path(get_ckpt_epoch(epoch)))


def clean_ckpt():
    r"""Removes all but the last model checkpoint."""
    for epoch in get_ckpt_epochs()[:-1]:
        os.remove(get_ckpt_path(epoch))


###############################################################################


def get_ckpt_dir() -> str:
    return osp.join(cfg.run_dir, 'ckpt')


def get_ckpt_path(epoch: Union[int, str]) -> str:
    return osp.join(get_ckpt_dir(), f'{epoch}.ckpt')


def get_ckpt_epochs() -> List[int]:
    paths = glob.glob(get_ckpt_path('*'))
    return sorted([int(osp.basename(path).split('.')[0]) for path in paths])


def get_ckpt_epoch(epoch: int) -> int:
    if epoch < 0:
        epochs = get_ckpt_epochs()
        epoch = epochs[epoch] if len(epochs) > 0 else 0
    return epoch
pytorch/torch_geometric/graphgym/cmd_args.py000066400000000000000000000013421444262304300216740ustar00rootroot00000000000000import argparse


def parse_args() -> argparse.Namespace:
    r"""Parses the command line arguments."""
    parser = argparse.ArgumentParser(description='GraphGym')

    parser.add_argument('--cfg', dest='cfg_file', type=str, required=True,
                        help='The configuration file path.')
    parser.add_argument('--repeat', type=int, default=1,
                        help='The number of repeated jobs.')
    parser.add_argument('--mark_done', action='store_true',
                        help='Mark yaml as done after a job has finished.')
    parser.add_argument('opts', default=None, nargs=argparse.REMAINDER,
                        help='See graphgym/config.py for remaining options.')

    return parser.parse_args()
pytorch/torch_geometric/graphgym/config.py000066400000000000000000000415051444262304300213670ustar00rootroot00000000000000import functools
import inspect
import logging
import os
import shutil
import warnings
from collections.abc import Iterable
from dataclasses import asdict
from typing import Any

import torch_geometric.graphgym.register as register
from torch_geometric.data.makedirs import makedirs

try:  # Define global config object
    from yacs.config import CfgNode as CN
    cfg = CN()
except ImportError:
    cfg = None
    warnings.warn("Could not define global config object. Please install "
                  "'yacs' via 'pip install yacs' in order to use GraphGym")


def set_cfg(cfg):
    r'''
    This function sets the default config value.
    1) Note that for an experiment, only part of the arguments will be used
    The remaining unused arguments won't affect anything.
    So feel free to register any argument in graphgym.contrib.config
    2) We support *at most* two levels of configs, e.g., cfg.dataset.name

    :return: configuration use by the experiment.
    '''
    if cfg is None:
        return cfg

    # ----------------------------------------------------------------------- #
    # Basic options
    # ----------------------------------------------------------------------- #

    # Set print destination: stdout / file / both
    cfg.print = 'both'

    # Select device: 'cpu', 'cuda', 'auto'
    cfg.accelerator = 'auto'

    # number of devices: eg. for 2 GPU set cfg.devices=2
    cfg.devices = 1

    # Output directory
    cfg.out_dir = 'results'

    # Config name (in out_dir)
    cfg.cfg_dest = 'config.yaml'

    # Names of registered custom metric funcs to be used (use defaults if none)
    cfg.custom_metrics = []

    # Random seed
    cfg.seed = 0

    # Print rounding
    cfg.round = 4

    # Tensorboard support for each run
    cfg.tensorboard_each_run = False

    # Tensorboard support for aggregated results
    cfg.tensorboard_agg = True

    # Additional num of worker for data loading
    cfg.num_workers = 0

    # Max threads used by PyTorch
    cfg.num_threads = 6

    # The metric for selecting the best epoch for each run
    cfg.metric_best = 'auto'

    # argmax or argmin in aggregating results
    cfg.metric_agg = 'argmax'

    # If visualize embedding.
    cfg.view_emb = False

    # If get GPU usage
    cfg.gpu_mem = False

    # If do benchmark analysis
    cfg.benchmark = False

    # ----------------------------------------------------------------------- #
    # Globally shared variables:
    # These variables will be set dynamically based on the input dataset
    # Do not directly set them here or in .yaml files
    # ----------------------------------------------------------------------- #

    cfg.share = CN()

    # Size of input dimension
    cfg.share.dim_in = 1

    # Size of out dimension, i.e., number of labels to be predicted
    cfg.share.dim_out = 1

    # Number of dataset splits: train/val/test
    cfg.share.num_splits = 1

    # ----------------------------------------------------------------------- #
    # Dataset options
    # ----------------------------------------------------------------------- #
    cfg.dataset = CN()

    # Name of the dataset
    cfg.dataset.name = 'Cora'

    # if PyG: look for it in Pytorch Geometric dataset
    # if NetworkX/nx: load data in NetworkX format
    cfg.dataset.format = 'PyG'

    # Dir to load the dataset. If the dataset is downloaded, this is the
    # cache dir
    cfg.dataset.dir = './datasets'

    # Task: node, edge, graph, link_pred
    cfg.dataset.task = 'node'

    # Type of task: classification, regression, classification_binary
    # classification_multi
    cfg.dataset.task_type = 'classification'

    # Transductive / Inductive
    # Graph classification is always inductive
    cfg.dataset.transductive = True

    # Split ratio of dataset. Len=2: Train, Val. Len=3: Train, Val, Test
    cfg.dataset.split = [0.8, 0.1, 0.1]

    # Whether to shuffle the graphs for splitting
    cfg.dataset.shuffle_split = True

    # Whether random split or use custom split: random / custom
    cfg.dataset.split_mode = 'random'

    # Whether to use an encoder for general attribute features
    cfg.dataset.encoder = True

    # Name of general encoder
    cfg.dataset.encoder_name = 'db'

    # If add batchnorm after general encoder
    cfg.dataset.encoder_bn = True

    # Whether to use an encoder for the node features
    cfg.dataset.node_encoder = False

    # Name of node encoder
    cfg.dataset.node_encoder_name = 'Atom'

    # If add batchnorm after node encoder
    cfg.dataset.node_encoder_bn = True

    # Whether to use an encoder for the edge features
    cfg.dataset.edge_encoder = False

    # Name of edge encoder
    cfg.dataset.edge_encoder_name = 'Bond'

    # If add batchnorm after edge encoder
    cfg.dataset.edge_encoder_bn = True

    # Dimension of the encoded features.
    # For now the node and edge encoding dimensions
    # are the same.
    cfg.dataset.encoder_dim = 128

    # Dimension for edge feature. Updated by the real dim of the dataset
    cfg.dataset.edge_dim = 128

    # ============== Link/edge tasks only

    # all or disjoint
    cfg.dataset.edge_train_mode = 'all'

    # Used in disjoint edge_train_mode. The proportion of edges used for
    # message-passing
    cfg.dataset.edge_message_ratio = 0.8

    # The ratio of negative samples to positive samples
    cfg.dataset.edge_negative_sampling_ratio = 1.0

    # Whether resample disjoint when dataset.edge_train_mode is 'disjoint'
    cfg.dataset.resample_disjoint = False

    # Whether resample negative edges at training time (link prediction only)
    cfg.dataset.resample_negative = False

    # What transformation function is applied to the dataset
    cfg.dataset.transform = 'none'

    # Whether cache the splitted dataset
    # NOTE: it should be cautiouslly used, as cached dataset may not have
    # exactly the same setting as the config file
    cfg.dataset.cache_save = False
    cfg.dataset.cache_load = False

    # Whether remove the original node features in the dataset
    cfg.dataset.remove_feature = False

    # Simplify TU dataset for synthetic tasks
    cfg.dataset.tu_simple = True

    # Convert to undirected graph (save 2*E edges)
    cfg.dataset.to_undirected = False

    # dataset location: local, snowflake
    cfg.dataset.location = 'local'

    # Define label: Table name
    cfg.dataset.label_table = 'none'

    # Define label: Column name
    cfg.dataset.label_column = 'none'

    # ----------------------------------------------------------------------- #
    # Training options
    # ----------------------------------------------------------------------- #
    cfg.train = CN()

    # Total graph mini-batch size
    cfg.train.batch_size = 16

    # Sampling strategy for a train loader
    cfg.train.sampler = 'full_batch'

    # Minibatch node
    cfg.train.sample_node = False

    # Num of sampled node per graph
    cfg.train.node_per_graph = 32

    # Radius: same, extend. same: same as cfg.gnn.layers_mp, extend: layers+1
    cfg.train.radius = 'extend'

    # Evaluate model on test data every eval period epochs
    cfg.train.eval_period = 10

    # Option to skip training epoch evaluation
    cfg.train.skip_train_eval = False

    # Save model checkpoint every checkpoint period epochs
    cfg.train.ckpt_period = 100

    # Enabling checkpoint, set False to disable and save I/O
    cfg.train.enable_ckpt = True

    # Resume training from the latest checkpoint in the output directory
    cfg.train.auto_resume = False

    # The epoch to resume. -1 means resume the latest epoch.
    cfg.train.epoch_resume = -1

    # Clean checkpoint: only keep the last ckpt
    cfg.train.ckpt_clean = True

    # Number of iterations per epoch (for sampling based loaders only)
    cfg.train.iter_per_epoch = 32

    # GraphSAINTRandomWalkSampler: random walk length
    cfg.train.walk_length = 4

    # NeighborSampler: number of sampled nodes per layer
    cfg.train.neighbor_sizes = [20, 15, 10, 5]

    # ----------------------------------------------------------------------- #
    # Validation options
    # ----------------------------------------------------------------------- #
    cfg.val = CN()

    # Minibatch node
    cfg.val.sample_node = False

    # Sampling strategy for a val/test loader
    cfg.val.sampler = 'full_batch'

    # Num of sampled node per graph
    cfg.val.node_per_graph = 32

    # Radius: same, extend. same: same as cfg.gnn.layers_mp, extend: layers+1
    cfg.val.radius = 'extend'

    # ----------------------------------------------------------------------- #
    # Model options
    # ----------------------------------------------------------------------- #
    cfg.model = CN()

    # Model type to use
    cfg.model.type = 'gnn'

    # Auto match computational budget, match upper bound / lower bound
    cfg.model.match_upper = True

    # Loss function: cross_entropy, mse
    cfg.model.loss_fun = 'cross_entropy'

    # size average for loss function. 'mean' or 'sum'
    cfg.model.size_average = 'mean'

    # Threshold for binary classification
    cfg.model.thresh = 0.5

    # ============== Link/edge tasks only
    # Edge decoding methods.
    #   - dot: compute dot(u, v) to predict link (binary)
    #   - cosine_similarity: use cosine similarity (u, v) to predict link (
    #   binary)
    #   - concat: use u||v followed by an nn.Linear to obtain edge embedding
    #   (multi-class)
    cfg.model.edge_decoding = 'dot'
    # ===================================

    # ================== Graph tasks only
    # Pooling methods.
    #   - add: global add pool
    #   - mean: global mean pool
    #   - max: global max pool
    cfg.model.graph_pooling = 'add'
    # ===================================

    # ----------------------------------------------------------------------- #
    # GNN options
    # ----------------------------------------------------------------------- #
    cfg.gnn = CN()

    # Prediction head. Use cfg.dataset.task by default
    cfg.gnn.head = 'default'

    # Number of layers before message passing
    cfg.gnn.layers_pre_mp = 0

    # Number of layers for message passing
    cfg.gnn.layers_mp = 2

    # Number of layers after message passing
    cfg.gnn.layers_post_mp = 0

    # Hidden layer dim. Automatically set if train.auto_match = True
    cfg.gnn.dim_inner = 16

    # Type of graph conv: generalconv, gcnconv, sageconv, gatconv, ...
    cfg.gnn.layer_type = 'generalconv'

    # Stage type: 'stack', 'skipsum', 'skipconcat'
    cfg.gnn.stage_type = 'stack'

    # How many layers to skip each time
    cfg.gnn.skip_every = 1

    # Whether use batch norm
    cfg.gnn.batchnorm = True

    # Activation
    cfg.gnn.act = 'relu'

    # Dropout
    cfg.gnn.dropout = 0.0

    # Aggregation type: add, mean, max
    # Note: only for certain layers that explicitly set aggregation type
    # e.g., when cfg.gnn.layer_type = 'generalconv'
    cfg.gnn.agg = 'add'

    # Normalize adj
    cfg.gnn.normalize_adj = False

    # Message direction: single, both
    cfg.gnn.msg_direction = 'single'

    # Whether add message from node itself: none, add, cat
    cfg.gnn.self_msg = 'concat'

    # Number of attention heads
    cfg.gnn.att_heads = 1

    # After concat attention heads, add a linear layer
    cfg.gnn.att_final_linear = False

    # After concat attention heads, add a linear layer
    cfg.gnn.att_final_linear_bn = False

    # Normalize after message passing
    cfg.gnn.l2norm = True

    # randomly use fewer edges for message passing
    cfg.gnn.keep_edge = 0.5

    # clear cached feature_new
    cfg.gnn.clear_feature = True

    # ----------------------------------------------------------------------- #
    # Optimizer options
    # ----------------------------------------------------------------------- #
    cfg.optim = CN()

    # optimizer: sgd, adam
    cfg.optim.optimizer = 'adam'

    # Base learning rate
    cfg.optim.base_lr = 0.01

    # L2 regularization
    cfg.optim.weight_decay = 5e-4

    # SGD momentum
    cfg.optim.momentum = 0.9

    # scheduler: none, steps, cos
    cfg.optim.scheduler = 'cos'

    # Steps for 'steps' policy (in epochs)
    cfg.optim.steps = [30, 60, 90]

    # Learning rate multiplier for 'steps' policy
    cfg.optim.lr_decay = 0.1

    # Maximal number of epochs
    cfg.optim.max_epoch = 200

    # ----------------------------------------------------------------------- #
    # Batch norm options
    # ----------------------------------------------------------------------- #
    cfg.bn = CN()

    # BN epsilon
    cfg.bn.eps = 1e-5

    # BN momentum (BN momentum in PyTorch = 1 - BN momentum in Caffe2)
    cfg.bn.mom = 0.1

    # ----------------------------------------------------------------------- #
    # Memory options
    # ----------------------------------------------------------------------- #
    cfg.mem = CN()

    # Perform ReLU inplace
    cfg.mem.inplace = False

    # Set user customized cfgs
    for func in register.config_dict.values():
        func(cfg)


def assert_cfg(cfg):
    r"""Checks config values, do necessary post processing to the configs"""
    if cfg.dataset.task not in ['node', 'edge', 'graph', 'link_pred']:
        raise ValueError('Task {} not supported, must be one of node, '
                         'edge, graph, link_pred'.format(cfg.dataset.task))
    if 'classification' in cfg.dataset.task_type and cfg.model.loss_fun == \
            'mse':
        cfg.model.loss_fun = 'cross_entropy'
        logging.warning(
            'model.loss_fun changed to cross_entropy for classification.')
    if cfg.dataset.task_type == 'regression' and cfg.model.loss_fun == \
            'cross_entropy':
        cfg.model.loss_fun = 'mse'
        logging.warning('model.loss_fun changed to mse for regression.')
    if cfg.dataset.task == 'graph' and cfg.dataset.transductive:
        cfg.dataset.transductive = False
        logging.warning('dataset.transductive changed '
                        'to False for graph task.')
    if cfg.gnn.layers_post_mp < 1:
        cfg.gnn.layers_post_mp = 1
        logging.warning('Layers after message passing should be >=1')
    if cfg.gnn.head == 'default':
        cfg.gnn.head = cfg.dataset.task
    cfg.run_dir = cfg.out_dir


def dump_cfg(cfg):
    r"""
    Dumps the config to the output directory specified in
    :obj:`cfg.out_dir`

    Args:
        cfg (CfgNode): Configuration node
    """
    makedirs(cfg.out_dir)
    cfg_file = os.path.join(cfg.out_dir, cfg.cfg_dest)
    with open(cfg_file, 'w') as f:
        cfg.dump(stream=f)


def load_cfg(cfg, args):
    r"""
    Load configurations from file system and command line

    Args:
        cfg (CfgNode): Configuration node
        args (ArgumentParser): Command argument parser
    """
    cfg.merge_from_file(args.cfg_file)
    cfg.merge_from_list(args.opts)
    assert_cfg(cfg)


def makedirs_rm_exist(dir):
    if os.path.isdir(dir):
        shutil.rmtree(dir)
    os.makedirs(dir, exist_ok=True)


def get_fname(fname):
    r"""
    Extract filename from file name path

    Args:
        fname (str): Filename for the yaml format configuration file
    """
    fname = os.path.basename(fname)
    if fname.endswith('.yaml'):
        fname = fname[:-5]
    elif fname.endswith('.yml'):
        fname = fname[:-4]
    return fname


def set_out_dir(out_dir, fname):
    r"""
    Create the directory for full experiment run

    Args:
        out_dir (str): Directory for output, specified in :obj:`cfg.out_dir`
        fname (str): Filename for the yaml format configuration file
    """
    fname = get_fname(fname)
    cfg.out_dir = os.path.join(out_dir, fname)
    # Make output directory
    if cfg.train.auto_resume:
        os.makedirs(cfg.out_dir, exist_ok=True)
    else:
        makedirs_rm_exist(cfg.out_dir)


def set_run_dir(out_dir):
    r"""
    Create the directory for each random seed experiment run

    Args:
        out_dir (str): Directory for output, specified in :obj:`cfg.out_dir`
    """
    cfg.run_dir = os.path.join(out_dir, str(cfg.seed))
    # Make output directory
    if cfg.train.auto_resume:
        os.makedirs(cfg.run_dir, exist_ok=True)
    else:
        makedirs_rm_exist(cfg.run_dir)


set_cfg(cfg)


def from_config(func):
    if inspect.isclass(func):
        params = list(inspect.signature(func.__init__).parameters.values())[1:]
    else:
        params = list(inspect.signature(func).parameters.values())

    arg_names = [p.name for p in params]
    has_defaults = [p.default != inspect.Parameter.empty for p in params]

    @functools.wraps(func)
    def wrapper(*args, cfg: Any = None, **kwargs):
        if cfg is not None:
            cfg = dict(cfg) if isinstance(cfg, Iterable) else asdict(cfg)

            iterator = zip(arg_names[len(args):], has_defaults[len(args):])
            for arg_name, has_default in iterator:
                if arg_name in kwargs:
                    continue
                elif arg_name in cfg:
                    kwargs[arg_name] = cfg[arg_name]
                elif not has_default:
                    raise ValueError(f"'cfg.{arg_name}' undefined")
        return func(*args, **kwargs)

    return wrapper
pytorch/torch_geometric/graphgym/contrib/000077500000000000000000000000001444262304300212035ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/__init__.py000066400000000000000000000006051444262304300233150ustar00rootroot00000000000000from .act import *  # noqa
from .config import *  # noqa
from .encoder import *  # noqa
from .head import *  # noqa
from .layer import *  # noqa
from .loader import *  # noqa
from .loss import *  # noqa
from .network import *  # noqa
from .optimizer import *  # noqa
from .pooling import *  # noqa
from .stage import *  # noqa
from .train import *  # noqa
from .transform import *  # noqa
pytorch/torch_geometric/graphgym/contrib/act/000077500000000000000000000000001444262304300217525ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/act/__init__.py000066400000000000000000000003351444262304300240640ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/config/000077500000000000000000000000001444262304300224505ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/config/__init__.py000066400000000000000000000003351444262304300245620ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/encoder/000077500000000000000000000000001444262304300226225ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/encoder/__init__.py000066400000000000000000000003351444262304300247340ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/head/000077500000000000000000000000001444262304300221045ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/head/__init__.py000066400000000000000000000003351444262304300242160ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/layer/000077500000000000000000000000001444262304300223175ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/layer/__init__.py000066400000000000000000000003351444262304300244310ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/layer/generalconv.py000066400000000000000000000201601444262304300251730ustar00rootroot00000000000000import torch
import torch.nn as nn
from torch.nn import Parameter

from torch_geometric.graphgym.config import cfg
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.utils import add_remaining_self_loops, scatter


class GeneralConvLayer(MessagePassing):
    r"""General GNN layer
    """
    def __init__(self, in_channels, out_channels, improved=False, cached=False,
                 bias=True, **kwargs):
        super().__init__(aggr=cfg.gnn.agg, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.improved = improved
        self.cached = cached
        self.normalize = cfg.gnn.normalize_adj

        self.weight = Parameter(torch.Tensor(in_channels, out_channels))
        if cfg.gnn.self_msg == 'concat':
            self.weight_self = Parameter(
                torch.Tensor(in_channels, out_channels))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        glorot(self.weight)
        if cfg.gnn.self_msg == 'concat':
            glorot(self.weight_self)
        zeros(self.bias)
        self.cached_result = None
        self.cached_num_edges = None

    @staticmethod
    def norm(edge_index, num_nodes, edge_weight=None, improved=False,
             dtype=None):
        if edge_weight is None:
            edge_weight = torch.ones((edge_index.size(1), ), dtype=dtype,
                                     device=edge_index.device)

        fill_value = 1 if not improved else 2
        edge_index, edge_weight = add_remaining_self_loops(
            edge_index, edge_weight, fill_value, num_nodes)

        row, col = edge_index
        deg = scatter(edge_weight, row, 0, num_nodes, reduce='sum')
        deg_inv_sqrt = deg.pow(-0.5)
        deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0

        return edge_index, deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]

    def forward(self, x, edge_index, edge_weight=None, edge_feature=None):
        if cfg.gnn.self_msg == 'concat':
            x_self = torch.matmul(x, self.weight_self)
        x = torch.matmul(x, self.weight)

        if self.cached and self.cached_result is not None:
            if edge_index.size(1) != self.cached_num_edges:
                raise RuntimeError(
                    'Cached {} number of edges, but found {}. Please '
                    'disable the caching behavior of this layer by removing '
                    'the `cached=True` argument in its constructor.'.format(
                        self.cached_num_edges, edge_index.size(1)))

        if not self.cached or self.cached_result is None:
            self.cached_num_edges = edge_index.size(1)
            if self.normalize:
                edge_index, norm = self.norm(edge_index, x.size(self.node_dim),
                                             edge_weight, self.improved,
                                             x.dtype)
            else:
                norm = edge_weight
            self.cached_result = edge_index, norm

        edge_index, norm = self.cached_result
        x_msg = self.propagate(edge_index, x=x, norm=norm,
                               edge_feature=edge_feature)
        if cfg.gnn.self_msg == 'none':
            return x_msg
        elif cfg.gnn.self_msg == 'add':
            return x_msg + x
        elif cfg.gnn.self_msg == 'concat':
            return x_msg + x_self
        else:
            raise ValueError('self_msg {} not defined'.format(
                cfg.gnn.self_msg))

    def message(self, x_j, norm, edge_feature):
        if edge_feature is None:
            return norm.view(-1, 1) * x_j if norm is not None else x_j
        else:
            return norm.view(-1, 1) * (
                x_j + edge_feature) if norm is not None else (x_j +
                                                              edge_feature)

    def update(self, aggr_out):
        if self.bias is not None:
            aggr_out = aggr_out + self.bias
        return aggr_out

    def __repr__(self):
        return '{}({}, {})'.format(self.__class__.__name__, self.in_channels,
                                   self.out_channels)


class GeneralEdgeConvLayer(MessagePassing):
    r"""General GNN layer, with edge features
    """
    def __init__(self, in_channels, out_channels, edge_dim, improved=False,
                 cached=False, bias=True, **kwargs):
        super().__init__(aggr=cfg.gnn.agg, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.improved = improved
        self.cached = cached
        self.normalize = cfg.gnn.normalize_adj
        self.msg_direction = cfg.gnn.msg_direction

        if self.msg_direction == 'single':
            self.linear_msg = nn.Linear(in_channels + edge_dim, out_channels,
                                        bias=False)
        else:
            self.linear_msg = nn.Linear(in_channels * 2 + edge_dim,
                                        out_channels, bias=False)
        if cfg.gnn.self_msg == 'concat':
            self.linear_self = nn.Linear(in_channels, out_channels, bias=False)

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        zeros(self.bias)
        self.cached_result = None
        self.cached_num_edges = None

    @staticmethod
    def norm(edge_index, num_nodes, edge_weight=None, improved=False,
             dtype=None):
        if edge_weight is None:
            edge_weight = torch.ones((edge_index.size(1), ), dtype=dtype,
                                     device=edge_index.device)

        fill_value = 1 if not improved else 2
        edge_index, edge_weight = add_remaining_self_loops(
            edge_index, edge_weight, fill_value, num_nodes)

        row, col = edge_index
        deg = scatter(edge_weight, row, 0, num_nodes, reduce='sum')
        deg_inv_sqrt = deg.pow(-0.5)
        deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0

        return edge_index, deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]

    def forward(self, x, edge_index, edge_weight=None, edge_feature=None):
        if self.cached and self.cached_result is not None:
            if edge_index.size(1) != self.cached_num_edges:
                raise RuntimeError(
                    'Cached {} number of edges, but found {}. Please '
                    'disable the caching behavior of this layer by removing '
                    'the `cached=True` argument in its constructor.'.format(
                        self.cached_num_edges, edge_index.size(1)))

        if not self.cached or self.cached_result is None:
            self.cached_num_edges = edge_index.size(1)
            if self.normalize:
                edge_index, norm = self.norm(edge_index, x.size(self.node_dim),
                                             edge_weight, self.improved,
                                             x.dtype)
            else:
                norm = edge_weight
            self.cached_result = edge_index, norm

        edge_index, norm = self.cached_result

        x_msg = self.propagate(edge_index, x=x, norm=norm,
                               edge_feature=edge_feature)

        if cfg.gnn.self_msg == 'concat':
            x_self = self.linear_self(x)
            return x_self + x_msg
        elif cfg.gnn.self_msg == 'add':
            return x + x_msg
        else:
            return x_msg

    def message(self, x_i, x_j, norm, edge_feature):
        if self.msg_direction == 'both':
            x_j = torch.cat((x_i, x_j, edge_feature), dim=-1)
        else:
            x_j = torch.cat((x_j, edge_feature), dim=-1)
        x_j = self.linear_msg(x_j)
        return norm.view(-1, 1) * x_j if norm is not None else x_j

    def update(self, aggr_out):
        if self.bias is not None:
            aggr_out = aggr_out + self.bias
        return aggr_out

    def __repr__(self):
        return '{}({}, {})'.format(self.__class__.__name__, self.in_channels,
                                   self.out_channels)
pytorch/torch_geometric/graphgym/contrib/loader/000077500000000000000000000000001444262304300224515ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/loader/__init__.py000066400000000000000000000003351444262304300245630ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/loss/000077500000000000000000000000001444262304300221635ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/loss/__init__.py000066400000000000000000000003351444262304300242750ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/network/000077500000000000000000000000001444262304300226745ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/network/__init__.py000066400000000000000000000003351444262304300250060ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/optimizer/000077500000000000000000000000001444262304300232255ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/optimizer/__init__.py000066400000000000000000000003351444262304300253370ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/pooling/000077500000000000000000000000001444262304300226525ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/pooling/__init__.py000066400000000000000000000003351444262304300247640ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/stage/000077500000000000000000000000001444262304300223065ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/stage/__init__.py000066400000000000000000000003351444262304300244200ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/train/000077500000000000000000000000001444262304300223205ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/train/__init__.py000066400000000000000000000003351444262304300244320ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/contrib/transform/000077500000000000000000000000001444262304300232165ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/contrib/transform/__init__.py000066400000000000000000000003351444262304300253300ustar00rootroot00000000000000from os.path import dirname, basename, isfile, join
import glob

modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [
    basename(f)[:-3] for f in modules
    if isfile(f) and not f.endswith('__init__.py')
]
pytorch/torch_geometric/graphgym/imports.py000066400000000000000000000005671444262304300216220ustar00rootroot00000000000000import warnings

import torch

try:
    import pytorch_lightning as pl
    LightningModule = pl.LightningModule
    Callback = pl.Callback
except ImportError:
    pl = object
    LightningModule = torch.nn.Module
    Callback = object

    warnings.warn("Please install 'pytorch_lightning' via  "
                  "'pip install pytorch_lightning' in order to use GraphGym")
pytorch/torch_geometric/graphgym/init.py000066400000000000000000000007521444262304300210640ustar00rootroot00000000000000import torch.nn as nn


def init_weights(m):
    r"""
    Performs weight initialization

    Args:
        m (nn.Module): PyTorch module

    """
    if isinstance(m, nn.BatchNorm2d) or isinstance(m, nn.BatchNorm1d):
        m.weight.data.fill_(1.0)
        m.bias.data.zero_()
    elif isinstance(m, nn.Linear):
        m.weight.data = nn.init.xavier_uniform_(
            m.weight.data, gain=nn.init.calculate_gain('relu'))
        if m.bias is not None:
            m.bias.data.zero_()
pytorch/torch_geometric/graphgym/loader.py000066400000000000000000000267631444262304300214010ustar00rootroot00000000000000from typing import Callable

import torch

import torch_geometric.graphgym.register as register
import torch_geometric.transforms as T
from torch_geometric.datasets import (
    PPI,
    Amazon,
    Coauthor,
    KarateClub,
    MNISTSuperpixels,
    Planetoid,
    QM7b,
    TUDataset,
)
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.models.transform import (
    create_link_label,
    neg_sampling_transform,
)
from torch_geometric.loader import (
    ClusterLoader,
    DataLoader,
    GraphSAINTEdgeSampler,
    GraphSAINTNodeSampler,
    GraphSAINTRandomWalkSampler,
    NeighborSampler,
    RandomNodeLoader,
)
from torch_geometric.utils import (
    index_to_mask,
    negative_sampling,
    to_undirected,
)

index2mask = index_to_mask  # TODO Backward compatibility


def planetoid_dataset(name: str) -> Callable:
    return lambda root: Planetoid(root, name)


register.register_dataset('Cora', planetoid_dataset('Cora'))
register.register_dataset('CiteSeer', planetoid_dataset('CiteSeer'))
register.register_dataset('PubMed', planetoid_dataset('PubMed'))
register.register_dataset('PPI', PPI)


def load_pyg(name, dataset_dir):
    """
    Load PyG dataset objects. (More PyG datasets will be supported)

    Args:
        name (str): dataset name
        dataset_dir (str): data directory

    Returns: PyG dataset object

    """
    dataset_dir = '{}/{}'.format(dataset_dir, name)
    if name in ['Cora', 'CiteSeer', 'PubMed']:
        dataset = Planetoid(dataset_dir, name)
    elif name[:3] == 'TU_':
        # TU_IMDB doesn't have node features
        if name[3:] == 'IMDB':
            name = 'IMDB-MULTI'
            dataset = TUDataset(dataset_dir, name, transform=T.Constant())
        else:
            dataset = TUDataset(dataset_dir, name[3:])
    elif name == 'Karate':
        dataset = KarateClub()
    elif 'Coauthor' in name:
        if 'CS' in name:
            dataset = Coauthor(dataset_dir, name='CS')
        else:
            dataset = Coauthor(dataset_dir, name='Physics')
    elif 'Amazon' in name:
        if 'Computers' in name:
            dataset = Amazon(dataset_dir, name='Computers')
        else:
            dataset = Amazon(dataset_dir, name='Photo')
    elif name == 'MNIST':
        dataset = MNISTSuperpixels(dataset_dir)
    elif name == 'PPI':
        dataset = PPI(dataset_dir)
    elif name == 'QM7b':
        dataset = QM7b(dataset_dir)
    else:
        raise ValueError('{} not support'.format(name))

    return dataset


def set_dataset_attr(dataset, name, value, size):
    dataset._data_list = None
    dataset.data[name] = value
    if dataset.slices is not None:
        dataset.slices[name] = torch.tensor([0, size], dtype=torch.long)


def load_ogb(name, dataset_dir):
    r"""

    Load OGB dataset objects.


    Args:
        name (str): dataset name
        dataset_dir (str): data directory

    Returns: PyG dataset object

    """
    from ogb.graphproppred import PygGraphPropPredDataset
    from ogb.linkproppred import PygLinkPropPredDataset
    from ogb.nodeproppred import PygNodePropPredDataset

    if name[:4] == 'ogbn':
        dataset = PygNodePropPredDataset(name=name, root=dataset_dir)
        splits = dataset.get_idx_split()
        split_names = ['train_mask', 'val_mask', 'test_mask']
        for i, key in enumerate(splits.keys()):
            mask = index_to_mask(splits[key], size=dataset._data.y.shape[0])
            set_dataset_attr(dataset, split_names[i], mask, len(mask))
        edge_index = to_undirected(dataset._data.edge_index)
        set_dataset_attr(dataset, 'edge_index', edge_index,
                         edge_index.shape[1])

    elif name[:4] == 'ogbg':
        dataset = PygGraphPropPredDataset(name=name, root=dataset_dir)
        splits = dataset.get_idx_split()
        split_names = [
            'train_graph_index', 'val_graph_index', 'test_graph_index'
        ]
        for i, key in enumerate(splits.keys()):
            id = splits[key]
            set_dataset_attr(dataset, split_names[i], id, len(id))

    elif name[:4] == "ogbl":
        dataset = PygLinkPropPredDataset(name=name, root=dataset_dir)
        splits = dataset.get_edge_split()
        id = splits['train']['edge'].T
        if cfg.dataset.resample_negative:
            set_dataset_attr(dataset, 'train_pos_edge_index', id, id.shape[1])
            dataset.transform = neg_sampling_transform
        else:
            id_neg = negative_sampling(edge_index=id,
                                       num_nodes=dataset._data.num_nodes,
                                       num_neg_samples=id.shape[1])
            id_all = torch.cat([id, id_neg], dim=-1)
            label = create_link_label(id, id_neg)
            set_dataset_attr(dataset, 'train_edge_index', id_all,
                             id_all.shape[1])
            set_dataset_attr(dataset, 'train_edge_label', label, len(label))

        id, id_neg = splits['valid']['edge'].T, splits['valid']['edge_neg'].T
        id_all = torch.cat([id, id_neg], dim=-1)
        label = create_link_label(id, id_neg)
        set_dataset_attr(dataset, 'val_edge_index', id_all, id_all.shape[1])
        set_dataset_attr(dataset, 'val_edge_label', label, len(label))

        id, id_neg = splits['test']['edge'].T, splits['test']['edge_neg'].T
        id_all = torch.cat([id, id_neg], dim=-1)
        label = create_link_label(id, id_neg)
        set_dataset_attr(dataset, 'test_edge_index', id_all, id_all.shape[1])
        set_dataset_attr(dataset, 'test_edge_label', label, len(label))

    else:
        raise ValueError('OGB dataset: {} non-exist')
    return dataset


def load_dataset():
    r"""

    Load dataset objects.

    Returns: PyG dataset object

    """
    format = cfg.dataset.format
    name = cfg.dataset.name
    dataset_dir = cfg.dataset.dir
    # Try to load customized data format
    for func in register.loader_dict.values():
        dataset = func(format, name, dataset_dir)
        if dataset is not None:
            return dataset
    # Load from Pytorch Geometric dataset
    if format == 'PyG':
        dataset = load_pyg(name, dataset_dir)
    # Load from OGB formatted data
    elif format == 'OGB':
        dataset = load_ogb(name.replace('_', '-'), dataset_dir)
    else:
        raise ValueError('Unknown data format: {}'.format(format))
    return dataset


def set_dataset_info(dataset):
    r"""
    Set global dataset information

    Args:
        dataset: PyG dataset object

    """

    # get dim_in and dim_out
    try:
        cfg.share.dim_in = dataset._data.x.shape[1]
    except Exception:
        cfg.share.dim_in = 1
    try:
        if cfg.dataset.task_type == 'classification':
            cfg.share.dim_out = torch.unique(dataset._data.y).shape[0]
        else:
            cfg.share.dim_out = dataset._data.y.shape[1]
    except Exception:
        cfg.share.dim_out = 1

    # count number of dataset splits
    cfg.share.num_splits = 1
    for key in dataset._data.keys:
        if 'val' in key:
            cfg.share.num_splits += 1
            break
    for key in dataset._data.keys:
        if 'test' in key:
            cfg.share.num_splits += 1
            break


def create_dataset():
    r"""
    Create dataset object

    Returns: PyG dataset object

    """
    dataset = load_dataset()
    set_dataset_info(dataset)

    return dataset


def get_loader(dataset, sampler, batch_size, shuffle=True):
    pw = cfg.num_workers > 0
    if sampler == "full_batch" or len(dataset) > 1:
        loader_train = DataLoader(dataset, batch_size=batch_size,
                                  shuffle=shuffle, num_workers=cfg.num_workers,
                                  pin_memory=True, persistent_workers=pw)
    elif sampler == "neighbor":
        loader_train = NeighborSampler(
            dataset[0], sizes=cfg.train.neighbor_sizes[:cfg.gnn.layers_mp],
            batch_size=batch_size, shuffle=shuffle,
            num_workers=cfg.num_workers, pin_memory=True)
    elif sampler == "random_node":
        loader_train = RandomNodeLoader(dataset[0],
                                        num_parts=cfg.train.train_parts,
                                        shuffle=shuffle,
                                        num_workers=cfg.num_workers,
                                        pin_memory=True, persistent_workers=pw)
    elif sampler == "saint_rw":
        loader_train = \
            GraphSAINTRandomWalkSampler(dataset[0],
                                        batch_size=batch_size,
                                        walk_length=cfg.train.walk_length,
                                        num_steps=cfg.train.iter_per_epoch,
                                        sample_coverage=0,
                                        shuffle=shuffle,
                                        num_workers=cfg.num_workers,
                                        pin_memory=True,
                                        persistent_workers=pw)
    elif sampler == "saint_node":
        loader_train = \
            GraphSAINTNodeSampler(dataset[0], batch_size=batch_size,
                                  num_steps=cfg.train.iter_per_epoch,
                                  sample_coverage=0, shuffle=shuffle,
                                  num_workers=cfg.num_workers,
                                  pin_memory=True,
                                  persistent_workers=pw)
    elif sampler == "saint_edge":
        loader_train = \
            GraphSAINTEdgeSampler(dataset[0], batch_size=batch_size,
                                  num_steps=cfg.train.iter_per_epoch,
                                  sample_coverage=0, shuffle=shuffle,
                                  num_workers=cfg.num_workers,
                                  pin_memory=True,
                                  persistent_workers=pw)
    elif sampler == "cluster":
        loader_train = \
            ClusterLoader(dataset[0],
                          num_parts=cfg.train.train_parts,
                          save_dir="{}/{}".format(cfg.dataset.dir,
                                                  cfg.dataset.name.replace(
                                                      "-", "_")),
                          batch_size=batch_size, shuffle=shuffle,
                          num_workers=cfg.num_workers,
                          pin_memory=True,
                          persistent_workers=pw)

    else:
        raise NotImplementedError("%s sampler is not implemented!" % sampler)
    return loader_train


def create_loader():
    """
    Create data loader object

    Returns: List of PyTorch data loaders

    """
    dataset = create_dataset()
    # train loader
    if cfg.dataset.task == 'graph':
        id = dataset.data['train_graph_index']
        loaders = [
            get_loader(dataset[id], cfg.train.sampler, cfg.train.batch_size,
                       shuffle=True)
        ]
        delattr(dataset.data, 'train_graph_index')
    else:
        loaders = [
            get_loader(dataset, cfg.train.sampler, cfg.train.batch_size,
                       shuffle=True)
        ]

    # val and test loaders
    for i in range(cfg.share.num_splits - 1):
        if cfg.dataset.task == 'graph':
            split_names = ['val_graph_index', 'test_graph_index']
            id = dataset.data[split_names[i]]
            loaders.append(
                get_loader(dataset[id], cfg.val.sampler, cfg.train.batch_size,
                           shuffle=False))
            delattr(dataset.data, split_names[i])
        else:
            loaders.append(
                get_loader(dataset, cfg.val.sampler, cfg.train.batch_size,
                           shuffle=False))

    return loaders
pytorch/torch_geometric/graphgym/logger.py000066400000000000000000000261011444262304300213740ustar00rootroot00000000000000import logging
import math
import sys
import time
from typing import Any, Dict, Optional

import torch

from torch_geometric.data.makedirs import makedirs
from torch_geometric.graphgym import register
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.imports import Callback, pl
from torch_geometric.graphgym.utils.device import get_current_gpu_usage
from torch_geometric.graphgym.utils.io import dict_to_json, dict_to_tb


def set_printing():
    """
    Set up printing options

    """
    logging.root.handlers = []
    logging_cfg = {'level': logging.INFO, 'format': '%(message)s'}
    makedirs(cfg.run_dir)
    h_file = logging.FileHandler('{}/logging.log'.format(cfg.run_dir))
    h_stdout = logging.StreamHandler(sys.stdout)
    if cfg.print == 'file':
        logging_cfg['handlers'] = [h_file]
    elif cfg.print == 'stdout':
        logging_cfg['handlers'] = [h_stdout]
    elif cfg.print == 'both':
        logging_cfg['handlers'] = [h_file, h_stdout]
    else:
        raise ValueError('Print option not supported')
    logging.basicConfig(**logging_cfg)


class Logger:
    def __init__(self, name='train', task_type=None):
        self.name = name
        self.task_type = task_type

        self._epoch_total = cfg.optim.max_epoch
        self._time_total = 0  # won't be reset

        self.out_dir = '{}/{}'.format(cfg.run_dir, name)
        makedirs(self.out_dir)
        if cfg.tensorboard_each_run:
            from tensorboardX import SummaryWriter
            self.tb_writer = SummaryWriter(self.out_dir)

        self.reset()

    def __getitem__(self, key):
        return getattr(self, key, None)

    def __setitem__(self, key, value):
        setattr(self, key, value)

    def reset(self):
        self._iter = 0
        self._size_current = 0
        self._loss = 0
        self._lr = 0
        self._params = 0
        self._time_used = 0
        self._true = []
        self._pred = []
        self._custom_stats = {}

    # basic properties
    def basic(self):
        stats = {
            'loss': round(self._loss / self._size_current, cfg.round),
            'lr': round(self._lr, cfg.round),
            'params': self._params,
            'time_iter': round(self.time_iter(), cfg.round),
        }
        gpu_memory = get_current_gpu_usage()
        if gpu_memory > 0:
            stats['gpu_memory'] = gpu_memory
        return stats

    # customized input properties
    def custom(self):
        if len(self._custom_stats) == 0:
            return {}
        out = {}
        for key, val in self._custom_stats.items():
            out[key] = val / self._size_current
        return out

    def _get_pred_int(self, pred_score):
        if len(pred_score.shape) == 1 or pred_score.shape[1] == 1:
            return (pred_score > cfg.model.thresh).long()
        else:
            return pred_score.max(dim=1)[1]

    # task properties
    def classification_binary(self):
        from sklearn.metrics import (
            accuracy_score,
            f1_score,
            precision_score,
            recall_score,
            roc_auc_score,
        )

        true, pred_score = torch.cat(self._true), torch.cat(self._pred)
        pred_int = self._get_pred_int(pred_score)
        try:
            r_a_score = roc_auc_score(true, pred_score)
        except ValueError:
            r_a_score = 0.0
        return {
            'accuracy': round(accuracy_score(true, pred_int), cfg.round),
            'precision': round(precision_score(true, pred_int), cfg.round),
            'recall': round(recall_score(true, pred_int), cfg.round),
            'f1': round(f1_score(true, pred_int), cfg.round),
            'auc': round(r_a_score, cfg.round),
        }

    def classification_multi(self):
        from sklearn.metrics import accuracy_score

        true, pred_score = torch.cat(self._true), torch.cat(self._pred)
        pred_int = self._get_pred_int(pred_score)
        return {'accuracy': round(accuracy_score(true, pred_int), cfg.round)}

    def regression(self):
        from sklearn.metrics import mean_absolute_error, mean_squared_error

        true, pred = torch.cat(self._true), torch.cat(self._pred)
        return {
            'mae':
            float(round(mean_absolute_error(true, pred), cfg.round)),
            'mse':
            float(round(mean_squared_error(true, pred), cfg.round)),
            'rmse':
            float(round(math.sqrt(mean_squared_error(true, pred)), cfg.round))
        }

    def time_iter(self):
        return self._time_used / self._iter

    def eta(self, epoch_current):
        epoch_current += 1  # since counter starts from 0
        time_per_epoch = self._time_total / epoch_current
        return time_per_epoch * (self._epoch_total - epoch_current)

    def update_stats(self, true, pred, loss, lr, time_used, params, **kwargs):
        assert true.shape[0] == pred.shape[0]
        self._iter += 1
        self._true.append(true)
        self._pred.append(pred)
        batch_size = true.shape[0]
        self._size_current += batch_size
        self._loss += loss * batch_size
        self._lr = lr
        self._params = params
        self._time_used += time_used
        self._time_total += time_used
        for key, val in kwargs.items():
            if key not in self._custom_stats:
                self._custom_stats[key] = val * batch_size
            else:
                self._custom_stats[key] += val * batch_size

    def write_iter(self):
        raise NotImplementedError

    def write_epoch(self, cur_epoch):
        basic_stats = self.basic()

        # Try to load customized metrics
        task_stats = {}
        for custom_metric in cfg.custom_metrics:
            func = register.metric_dict.get(custom_metric)
            if not func:
                raise ValueError(
                    f'Unknown custom metric function name: {custom_metric}')
            custom_metric_score = func(self._true, self._pred, self.task_type)
            task_stats[custom_metric] = custom_metric_score

        if not task_stats:  # use default metrics if no matching custom metric
            if self.task_type == 'regression':
                task_stats = self.regression()
            elif self.task_type == 'classification_binary':
                task_stats = self.classification_binary()
            elif self.task_type == 'classification_multi':
                task_stats = self.classification_multi()
            else:
                raise ValueError('Task has to be regression or classification')

        epoch_stats = {'epoch': cur_epoch}
        eta_stats = {'eta': round(self.eta(cur_epoch), cfg.round)}
        custom_stats = self.custom()

        if self.name == 'train':
            stats = {
                **epoch_stats,
                **eta_stats,
                **basic_stats,
                **task_stats,
                **custom_stats
            }
        else:
            stats = {
                **epoch_stats,
                **basic_stats,
                **task_stats,
                **custom_stats
            }

        # print
        logging.info('{}: {}'.format(self.name, stats))
        # json
        dict_to_json(stats, '{}/stats.json'.format(self.out_dir))
        # tensorboard
        if cfg.tensorboard_each_run:
            dict_to_tb(stats, self.tb_writer, cur_epoch)
        self.reset()

    def close(self):
        if cfg.tensorboard_each_run:
            self.tb_writer.close()


def infer_task():
    num_label = cfg.share.dim_out
    if cfg.dataset.task_type == 'classification':
        if num_label <= 2:
            task_type = 'classification_binary'
        else:
            task_type = 'classification_multi'
    else:
        task_type = cfg.dataset.task_type
    return task_type


def create_logger():
    r"""Create logger for the experiment."""
    loggers = []
    names = ['train', 'val', 'test']
    for i, dataset in enumerate(range(cfg.share.num_splits)):
        loggers.append(Logger(name=names[i], task_type=infer_task()))
    return loggers


class LoggerCallback(Callback):
    def __init__(self):
        self._logger = create_logger()
        self._train_epoch_start_time = None
        self._val_epoch_start_time = None
        self._test_epoch_start_time = None

    @property
    def train_logger(self) -> Any:
        return self._logger[0]

    @property
    def val_logger(self) -> Any:
        return self._logger[1]

    @property
    def test_logger(self) -> Any:
        return self._logger[2]

    def close(self):
        for logger in self._logger:
            logger.close()

    def _get_stats(
        self,
        epoch_start_time: int,
        outputs: Dict[str, Any],
        trainer: 'pl.Trainer',
    ) -> Dict:
        return dict(
            true=outputs['true'].detach().cpu(),
            pred=outputs['pred_score'].detach().cpu(),
            loss=float(outputs['loss']),
            lr=trainer.lr_scheduler_configs[0].scheduler.get_last_lr()[0],
            time_used=time.time() - epoch_start_time,
            params=cfg.params,
        )

    def on_train_epoch_start(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self._train_epoch_start_time = time.time()

    def on_validation_epoch_start(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self._val_epoch_start_time = time.time()

    def on_test_epoch_start(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self._test_epoch_start_time = time.time()

    def on_train_batch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
        outputs: Dict[str, Any],
        batch: Any,
        batch_idx: int,
        unused: int = 0,
    ):
        stats = self._get_stats(self._train_epoch_start_time, outputs, trainer)
        self.train_logger.update_stats(**stats)

    def on_validation_batch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
        outputs: Optional[Dict[str, Any]],
        batch: Any,
        batch_idx: int,
        dataloader_idx: int = 0,
    ):
        stats = self._get_stats(self._val_epoch_start_time, outputs, trainer)
        self.val_logger.update_stats(**stats)

    def on_test_batch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
        outputs: Optional[Dict[str, Any]],
        batch: Any,
        batch_idx: int,
        dataloader_idx: int = 0,
    ):
        stats = self._get_stats(self._test_epoch_start_time, outputs, trainer)
        self.test_logger.update_stats(**stats)

    def on_train_epoch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self.train_logger.write_epoch(trainer.current_epoch)

    def on_validation_epoch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self.val_logger.write_epoch(trainer.current_epoch)

    def on_test_epoch_end(
        self,
        trainer: 'pl.Trainer',
        pl_module: 'pl.LightningModule',
    ):
        self.test_logger.write_epoch(trainer.current_epoch)

    def on_fit_end(self, trainer, pl_module):
        self.close()
pytorch/torch_geometric/graphgym/loss.py000066400000000000000000000027021444262304300210760ustar00rootroot00000000000000import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.graphgym.register as register
from torch_geometric.graphgym.config import cfg


def compute_loss(pred, true):
    """
    Compute loss and prediction score

    Args:
        pred (torch.tensor): Unnormalized prediction
        true (torch.tensor): Grou

    Returns: Loss, normalized prediction score

    """
    bce_loss = nn.BCEWithLogitsLoss(reduction=cfg.model.size_average)
    mse_loss = nn.MSELoss(reduction=cfg.model.size_average)

    # default manipulation for pred and true
    # can be skipped if special loss computation is needed
    pred = pred.squeeze(-1) if pred.ndim > 1 else pred
    true = true.squeeze(-1) if true.ndim > 1 else true

    # Try to load customized loss
    for func in register.loss_dict.values():
        value = func(pred, true)
        if value is not None:
            return value

    if cfg.model.loss_fun == 'cross_entropy':
        # multiclass
        if pred.ndim > 1 and true.ndim == 1:
            pred = F.log_softmax(pred, dim=-1)
            return F.nll_loss(pred, true), pred
        # binary or multilabel
        else:
            true = true.float()
            return bce_loss(pred, true), torch.sigmoid(pred)
    elif cfg.model.loss_fun == 'mse':
        true = true.float()
        return mse_loss(pred, true), pred
    else:
        raise ValueError('Loss func {} not supported'.format(
            cfg.model.loss_fun))
pytorch/torch_geometric/graphgym/model_builder.py000066400000000000000000000060461444262304300227310ustar00rootroot00000000000000import time
from typing import Any, Dict, Tuple

import torch

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.imports import LightningModule
from torch_geometric.graphgym.loss import compute_loss
from torch_geometric.graphgym.models.gnn import GNN
from torch_geometric.graphgym.optim import create_optimizer, create_scheduler
from torch_geometric.graphgym.register import network_dict, register_network

register_network('gnn', GNN)


class GraphGymModule(LightningModule):
    def __init__(self, dim_in, dim_out, cfg):
        super().__init__()
        self.cfg = cfg
        self.model = network_dict[cfg.model.type](dim_in=dim_in,
                                                  dim_out=dim_out)

    def forward(self, *args, **kwargs):
        return self.model(*args, **kwargs)

    def configure_optimizers(self) -> Tuple[Any, Any]:
        optimizer = create_optimizer(self.model.parameters(), self.cfg.optim)
        scheduler = create_scheduler(optimizer, self.cfg.optim)
        return [optimizer], [scheduler]

    def _shared_step(self, batch, split: str) -> Dict:
        batch.split = split
        pred, true = self(batch)
        loss, pred_score = compute_loss(pred, true)
        step_end_time = time.time()
        return dict(loss=loss, true=true, pred_score=pred_score.detach(),
                    step_end_time=step_end_time)

    def training_step(self, batch, *args, **kwargs):
        return self._shared_step(batch, split="train")

    def validation_step(self, batch, *args, **kwargs):
        return self._shared_step(batch, split="val")

    def test_step(self, batch, *args, **kwargs):
        return self._shared_step(batch, split="test")

    @property
    def encoder(self) -> torch.nn.Module:
        return self.model.encoder

    @property
    def mp(self) -> torch.nn.Module:
        return self.model.mp

    @property
    def post_mp(self) -> torch.nn.Module:
        return self.model.post_mp

    @property
    def pre_mp(self) -> torch.nn.Module:
        return self.model.pre_mp

    def lr_scheduler_step(self, *args, **kwargs):
        # Needed for PyTorch 2.0 since the base class of LR schedulers changed.
        # TODO Remove once we only want to support PyTorch Lightning >= 2.0.
        return super().lr_scheduler_step(*args, **kwargs)


def create_model(to_device=True, dim_in=None, dim_out=None) -> GraphGymModule:
    r"""Create model for graph machine learning.

    Args:
        to_device (bool, optional): Whether to transfer the model to the
            specified device. (default: :obj:`True`)
        dim_in (int, optional): Input dimension to the model
        dim_out (int, optional): Output dimension to the model
    """
    dim_in = cfg.share.dim_in if dim_in is None else dim_in
    dim_out = cfg.share.dim_out if dim_out is None else dim_out
    # binary classification, output dim = 1
    if 'classification' == cfg.dataset.task_type and dim_out == 2:
        dim_out = 1

    model = GraphGymModule(dim_in, dim_out, cfg)
    if to_device:
        model.to(torch.device(cfg.accelerator))
    return model
pytorch/torch_geometric/graphgym/models/000077500000000000000000000000001444262304300210265ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/models/__init__.py000066400000000000000000000021411444262304300231350ustar00rootroot00000000000000from .encoder import (IntegerFeatureEncoder, AtomEncoder, BondEncoder)
from .gnn import (GNNLayer, GNNPreMP, GNNStackStage, FeatureEncoder, GNN)
from .head import (GNNNodeHead, GNNEdgeHead, GNNGraphHead)
from .layer import (GeneralLayer, GeneralMultiLayer, Linear, BatchNorm1dNode,
                    BatchNorm1dEdge, MLP, GCNConv, SAGEConv, GATConv, GINConv,
                    SplineConv, GeneralConv, GeneralEdgeConv,
                    GeneralSampleEdgeConv)
from .pooling import (global_add_pool, global_mean_pool, global_max_pool)

__all__ = [
    'IntegerFeatureEncoder',
    'AtomEncoder',
    'BondEncoder',
    'GNNLayer',
    'GNNPreMP',
    'GNNStackStage',
    'FeatureEncoder',
    'GNN',
    'GNNNodeHead',
    'GNNEdgeHead',
    'GNNGraphHead',
    'GeneralLayer',
    'GeneralMultiLayer',
    'Linear',
    'BatchNorm1dNode',
    'BatchNorm1dEdge',
    'MLP',
    'GCNConv',
    'SAGEConv',
    'GATConv',
    'GINConv',
    'SplineConv',
    'GeneralConv',
    'GeneralEdgeConv',
    'GeneralSampleEdgeConv',
    'global_add_pool',
    'global_mean_pool',
    'global_max_pool',
]

classes = __all__
pytorch/torch_geometric/graphgym/models/act.py000066400000000000000000000014661444262304300221560ustar00rootroot00000000000000import torch.nn as nn

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.register import register_act


def relu():
    return nn.ReLU(inplace=cfg.mem.inplace)


def selu():
    return nn.SELU(inplace=cfg.mem.inplace)


def prelu():
    return nn.PReLU()


def elu():
    return nn.ELU(inplace=cfg.mem.inplace)


def lrelu_01():
    return nn.LeakyReLU(0.1, inplace=cfg.mem.inplace)


def lrelu_025():
    return nn.LeakyReLU(0.25, inplace=cfg.mem.inplace)


def lrelu_05():
    return nn.LeakyReLU(0.5, inplace=cfg.mem.inplace)


if cfg is not None:
    register_act('relu', relu)
    register_act('selu', selu)
    register_act('prelu', prelu)
    register_act('elu', elu)
    register_act('lrelu_01', lrelu_01)
    register_act('lrelu_025', lrelu_025)
    register_act('lrelu_05', lrelu_05)
pytorch/torch_geometric/graphgym/models/encoder.py000066400000000000000000000047621444262304300230300ustar00rootroot00000000000000import torch

from torch_geometric.graphgym.register import (
    register_edge_encoder,
    register_node_encoder,
)


@register_node_encoder('Integer')
class IntegerFeatureEncoder(torch.nn.Module):
    """
    Provides an encoder for integer node features.

    Args:
        emb_dim (int): Output embedding dimension
        num_classes (int): the number of classes for the
        embedding mapping to learn from
    """
    def __init__(self, emb_dim, num_classes=None):
        super().__init__()

        self.encoder = torch.nn.Embedding(num_classes, emb_dim)
        torch.nn.init.xavier_uniform_(self.encoder.weight.data)

    def forward(self, batch):
        # Encode just the first dimension if more exist
        batch.x = self.encoder(batch.x[:, 0])

        return batch


@register_node_encoder('Atom')
class AtomEncoder(torch.nn.Module):
    """
    The atom Encoder used in OGB molecule dataset.

    Args:
        emb_dim (int): Output embedding dimension
        num_classes: None
    """
    def __init__(self, emb_dim, num_classes=None):
        super().__init__()

        from ogb.utils.features import get_atom_feature_dims

        self.atom_embedding_list = torch.nn.ModuleList()

        for i, dim in enumerate(get_atom_feature_dims()):
            emb = torch.nn.Embedding(dim, emb_dim)
            torch.nn.init.xavier_uniform_(emb.weight.data)
            self.atom_embedding_list.append(emb)

    def forward(self, batch):
        encoded_features = 0
        for i in range(batch.x.shape[1]):
            encoded_features += self.atom_embedding_list[i](batch.x[:, i])

        batch.x = encoded_features
        return batch


@register_edge_encoder('Bond')
class BondEncoder(torch.nn.Module):
    """
    The bond Encoder used in OGB molecule dataset.

    Args:
        emb_dim (int): Output edge embedding dimension
    """
    def __init__(self, emb_dim):
        super().__init__()

        from ogb.utils.features import get_bond_feature_dims

        self.bond_embedding_list = torch.nn.ModuleList()

        for i, dim in enumerate(get_bond_feature_dims()):
            emb = torch.nn.Embedding(dim, emb_dim)
            torch.nn.init.xavier_uniform_(emb.weight.data)
            self.bond_embedding_list.append(emb)

    def forward(self, batch):
        bond_embedding = 0
        for i in range(batch.edge_attr.shape[1]):
            edge_attr = batch.edge_attr
            bond_embedding += self.bond_embedding_list[i](edge_attr[:, i])

        batch.edge_attr = bond_embedding
        return batch
pytorch/torch_geometric/graphgym/models/gnn.py000066400000000000000000000120151444262304300221610ustar00rootroot00000000000000import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.graphgym.register as register
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.init import init_weights
from torch_geometric.graphgym.models.layer import (
    BatchNorm1dNode,
    GeneralLayer,
    GeneralMultiLayer,
    new_layer_config,
)
from torch_geometric.graphgym.register import register_stage


def GNNLayer(dim_in, dim_out, has_act=True):
    """
    Wrapper for a GNN layer

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        has_act (bool): Whether has activation function after the layer

    """
    return GeneralLayer(
        cfg.gnn.layer_type,
        layer_config=new_layer_config(dim_in, dim_out, 1, has_act=has_act,
                                      has_bias=False, cfg=cfg))


def GNNPreMP(dim_in, dim_out, num_layers):
    """
    Wrapper for NN layer before GNN message passing

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        num_layers (int): Number of layers

    """
    return GeneralMultiLayer(
        'linear',
        layer_config=new_layer_config(dim_in, dim_out, num_layers,
                                      has_act=False, has_bias=False, cfg=cfg))


@register_stage('stack')
@register_stage('skipsum')
@register_stage('skipconcat')
class GNNStackStage(nn.Module):
    """
    Simple Stage that stack GNN layers

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        num_layers (int): Number of GNN layers
    """
    def __init__(self, dim_in, dim_out, num_layers):
        super().__init__()
        self.num_layers = num_layers
        for i in range(num_layers):
            if cfg.gnn.stage_type == 'skipconcat':
                d_in = dim_in if i == 0 else dim_in + i * dim_out
            else:
                d_in = dim_in if i == 0 else dim_out
            layer = GNNLayer(d_in, dim_out)
            self.add_module('layer{}'.format(i), layer)

    def forward(self, batch):
        for i, layer in enumerate(self.children()):
            x = batch.x
            batch = layer(batch)
            if cfg.gnn.stage_type == 'skipsum':
                batch.x = x + batch.x
            elif cfg.gnn.stage_type == 'skipconcat' and \
                    i < self.num_layers - 1:
                batch.x = torch.cat([x, batch.x], dim=1)
        if cfg.gnn.l2norm:
            batch.x = F.normalize(batch.x, p=2, dim=-1)
        return batch


class FeatureEncoder(nn.Module):
    """
    Encoding node and edge features

    Args:
        dim_in (int): Input feature dimension
    """
    def __init__(self, dim_in):
        super().__init__()
        self.dim_in = dim_in
        if cfg.dataset.node_encoder:
            # Encode integer node features via nn.Embeddings
            NodeEncoder = register.node_encoder_dict[
                cfg.dataset.node_encoder_name]
            self.node_encoder = NodeEncoder(cfg.gnn.dim_inner)
            if cfg.dataset.node_encoder_bn:
                self.node_encoder_bn = BatchNorm1dNode(
                    new_layer_config(cfg.gnn.dim_inner, -1, -1, has_act=False,
                                     has_bias=False, cfg=cfg))
            # Update dim_in to reflect the new dimension fo the node features
            self.dim_in = cfg.gnn.dim_inner
        if cfg.dataset.edge_encoder:
            # Encode integer edge features via nn.Embeddings
            EdgeEncoder = register.edge_encoder_dict[
                cfg.dataset.edge_encoder_name]
            self.edge_encoder = EdgeEncoder(cfg.gnn.dim_inner)
            if cfg.dataset.edge_encoder_bn:
                self.edge_encoder_bn = BatchNorm1dNode(
                    new_layer_config(cfg.gnn.dim_inner, -1, -1, has_act=False,
                                     has_bias=False, cfg=cfg))

    def forward(self, batch):
        for module in self.children():
            batch = module(batch)
        return batch


class GNN(nn.Module):
    """
    General GNN model: encoder + stage + head

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        **kwargs (optional): Optional additional args
    """
    def __init__(self, dim_in, dim_out, **kwargs):
        super().__init__()
        GNNStage = register.stage_dict[cfg.gnn.stage_type]
        GNNHead = register.head_dict[cfg.gnn.head]

        self.encoder = FeatureEncoder(dim_in)
        dim_in = self.encoder.dim_in

        if cfg.gnn.layers_pre_mp > 0:
            self.pre_mp = GNNPreMP(dim_in, cfg.gnn.dim_inner,
                                   cfg.gnn.layers_pre_mp)
            dim_in = cfg.gnn.dim_inner
        if cfg.gnn.layers_mp > 0:
            self.mp = GNNStage(dim_in=dim_in, dim_out=cfg.gnn.dim_inner,
                               num_layers=cfg.gnn.layers_mp)
        self.post_mp = GNNHead(dim_in=cfg.gnn.dim_inner, dim_out=dim_out)

        self.apply(init_weights)

    def forward(self, batch):
        for module in self.children():
            batch = module(batch)
        return batch
pytorch/torch_geometric/graphgym/models/head.py000066400000000000000000000105221444262304300223010ustar00rootroot00000000000000""" GNN heads are the last layer of a GNN right before loss computation.
They are constructed in the init function of the gnn.GNN.
"""

import torch
import torch.nn as nn

import torch_geometric.graphgym.register as register
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.models.layer import MLP, new_layer_config
from torch_geometric.graphgym.register import register_head


@register_head('node')
class GNNNodeHead(nn.Module):
    """
    GNN prediction head for node prediction tasks.

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension. For binary prediction, dim_out=1.
    """
    def __init__(self, dim_in, dim_out):
        super().__init__()
        self.layer_post_mp = MLP(
            new_layer_config(dim_in, dim_out, cfg.gnn.layers_post_mp,
                             has_act=False, has_bias=True, cfg=cfg))

    def _apply_index(self, batch):
        x = batch.x
        y = batch.y if 'y' in batch else None

        if 'split' not in batch:
            return x, y

        mask = batch[f'{batch.split}_mask']
        return x[mask], y[mask] if y is not None else None

    def forward(self, batch):
        batch = self.layer_post_mp(batch)
        pred, label = self._apply_index(batch)
        return pred, label


@register_head('edge')
@register_head('link_pred')
class GNNEdgeHead(nn.Module):
    """
    GNN prediction head for edge/link prediction tasks.

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension. For binary prediction, dim_out=1.
    """
    def __init__(self, dim_in, dim_out):
        super().__init__()
        # module to decode edges from node embeddings
        if cfg.model.edge_decoding == 'concat':
            self.layer_post_mp = MLP(
                new_layer_config(dim_in * 2, dim_out, cfg.gnn.layers_post_mp,
                                 has_act=False, has_bias=True, cfg=cfg))
            # requires parameter
            self.decode_module = lambda v1, v2: \
                self.layer_post_mp(torch.cat((v1, v2), dim=-1))
        else:
            if dim_out > 1:
                raise ValueError(
                    'Binary edge decoding ({})is used for multi-class '
                    'edge/link prediction.'.format(cfg.model.edge_decoding))
            self.layer_post_mp = MLP(
                new_layer_config(dim_in, dim_in, cfg.gnn.layers_post_mp,
                                 has_act=False, has_bias=True, cfg=cfg))
            if cfg.model.edge_decoding == 'dot':
                self.decode_module = lambda v1, v2: torch.sum(v1 * v2, dim=-1)
            elif cfg.model.edge_decoding == 'cosine_similarity':
                self.decode_module = nn.CosineSimilarity(dim=-1)
            else:
                raise ValueError('Unknown edge decoding {}.'.format(
                    cfg.model.edge_decoding))

    def _apply_index(self, batch):
        index = '{}_edge_index'.format(batch.split)
        label = '{}_edge_label'.format(batch.split)
        return batch.x[batch[index]], \
            batch[label]

    def forward(self, batch):
        if cfg.model.edge_decoding != 'concat':
            batch = self.layer_post_mp(batch)
        pred, label = self._apply_index(batch)
        nodes_first = pred[0]
        nodes_second = pred[1]
        pred = self.decode_module(nodes_first, nodes_second)
        return pred, label


@register_head('graph')
class GNNGraphHead(nn.Module):
    """
    GNN prediction head for graph prediction tasks.
    The optional post_mp layer (specified by cfg.gnn.post_mp) is used
    to transform the pooled embedding using an MLP.

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension. For binary prediction, dim_out=1.
    """
    def __init__(self, dim_in, dim_out):
        super().__init__()
        self.layer_post_mp = MLP(
            new_layer_config(dim_in, dim_out, cfg.gnn.layers_post_mp,
                             has_act=False, has_bias=True, cfg=cfg))
        self.pooling_fun = register.pooling_dict[cfg.model.graph_pooling]

    def _apply_index(self, batch):
        return batch.graph_feature, batch.y

    def forward(self, batch):
        graph_emb = self.pooling_fun(batch.x, batch.batch)
        graph_emb = self.layer_post_mp(graph_emb)
        batch.graph_feature = graph_emb
        pred, label = self._apply_index(batch)
        return pred, label
pytorch/torch_geometric/graphgym/models/layer.py000066400000000000000000000303061444262304300225160ustar00rootroot00000000000000import copy
from dataclasses import dataclass, replace

import torch
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric as pyg
import torch_geometric.graphgym.models.act
import torch_geometric.graphgym.register as register
from torch_geometric.graphgym.contrib.layer.generalconv import (
    GeneralConvLayer,
    GeneralEdgeConvLayer,
)
from torch_geometric.graphgym.register import register_layer
from torch_geometric.nn import Linear as Linear_pyg


@dataclass
class LayerConfig:
    # batchnorm parameters.
    has_batchnorm: bool = False
    bn_eps: float = 1e-5
    bn_mom: float = 0.1

    # mem parameters.
    mem_inplace: bool = False

    # gnn parameters.
    dim_in: int = -1
    dim_out: int = -1
    edge_dim: int = -1
    dim_inner: int = None
    num_layers: int = 2
    has_bias: bool = True
    # regularizer parameters.
    has_l2norm: bool = True
    dropout: float = 0.0
    # activation parameters.
    has_act: bool = True
    final_act: bool = True
    act: str = 'relu'

    # other parameters.
    keep_edge: float = 0.5


def new_layer_config(dim_in, dim_out, num_layers, has_act, has_bias, cfg):
    return LayerConfig(
        has_batchnorm=cfg.gnn.batchnorm,
        bn_eps=cfg.bn.eps,
        bn_mom=cfg.bn.mom,
        mem_inplace=cfg.mem.inplace,
        dim_in=dim_in,
        dim_out=dim_out,
        edge_dim=cfg.dataset.edge_dim,
        has_l2norm=cfg.gnn.l2norm,
        dropout=cfg.gnn.dropout,
        has_act=has_act,
        final_act=True,
        act=cfg.gnn.act,
        has_bias=has_bias,
        keep_edge=cfg.gnn.keep_edge,
        dim_inner=cfg.gnn.dim_inner,
        num_layers=num_layers,
    )


# General classes
class GeneralLayer(nn.Module):
    """
    General wrapper for layers

    Args:
        name (str): Name of the layer in registered :obj:`layer_dict`
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        has_act (bool): Whether has activation after the layer
        has_bn (bool):  Whether has BatchNorm in the layer
        has_l2norm (bool): Wheter has L2 normalization after the layer
        **kwargs (optional): Additional args
    """
    def __init__(self, name, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.has_l2norm = layer_config.has_l2norm
        has_bn = layer_config.has_batchnorm
        layer_config.has_bias = not has_bn
        self.layer = register.layer_dict[name](layer_config, **kwargs)
        layer_wrapper = []
        if has_bn:
            layer_wrapper.append(
                nn.BatchNorm1d(layer_config.dim_out, eps=layer_config.bn_eps,
                               momentum=layer_config.bn_mom))
        if layer_config.dropout > 0:
            layer_wrapper.append(
                nn.Dropout(p=layer_config.dropout,
                           inplace=layer_config.mem_inplace))
        if layer_config.has_act:
            layer_wrapper.append(register.act_dict[layer_config.act]())
        self.post_layer = nn.Sequential(*layer_wrapper)

    def forward(self, batch):
        batch = self.layer(batch)
        if isinstance(batch, torch.Tensor):
            batch = self.post_layer(batch)
            if self.has_l2norm:
                batch = F.normalize(batch, p=2, dim=1)
        else:
            batch.x = self.post_layer(batch.x)
            if self.has_l2norm:
                batch.x = F.normalize(batch.x, p=2, dim=1)
        return batch


class GeneralMultiLayer(nn.Module):
    """
    General wrapper for a stack of multiple layers

    Args:
        name (str): Name of the layer in registered :obj:`layer_dict`
        num_layers (int): Number of layers in the stack
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        dim_inner (int): The dimension for the inner layers
        final_act (bool): Whether has activation after the layer stack
        **kwargs (optional): Additional args
    """
    def __init__(self, name, layer_config: LayerConfig, **kwargs):
        super().__init__()
        dim_inner = layer_config.dim_out \
            if layer_config.dim_inner is None \
            else layer_config.dim_inner
        for i in range(layer_config.num_layers):
            d_in = layer_config.dim_in if i == 0 else dim_inner
            d_out = layer_config.dim_out \
                if i == layer_config.num_layers - 1 else dim_inner
            has_act = layer_config.final_act \
                if i == layer_config.num_layers - 1 else True
            inter_layer_config = copy.deepcopy(layer_config)
            inter_layer_config.dim_in = d_in
            inter_layer_config.dim_out = d_out
            inter_layer_config.has_act = has_act
            layer = GeneralLayer(name, inter_layer_config, **kwargs)
            self.add_module('Layer_{}'.format(i), layer)

    def forward(self, batch):
        for layer in self.children():
            batch = layer(batch)
        return batch


# ---------- Core basic layers. Input: batch; Output: batch ----------------- #


@register_layer('linear')
class Linear(nn.Module):
    """
    Basic Linear layer.

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        bias (bool): Whether has bias term
        **kwargs (optional): Additional args
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = Linear_pyg(layer_config.dim_in, layer_config.dim_out,
                                bias=layer_config.has_bias)

    def forward(self, batch):
        if isinstance(batch, torch.Tensor):
            batch = self.model(batch)
        else:
            batch.x = self.model(batch.x)
        return batch


class BatchNorm1dNode(nn.Module):
    """
    BatchNorm for node feature.

    Args:
        dim_in (int): Input dimension
    """
    def __init__(self, layer_config: LayerConfig):
        super().__init__()
        self.bn = nn.BatchNorm1d(layer_config.dim_in, eps=layer_config.bn_eps,
                                 momentum=layer_config.bn_mom)

    def forward(self, batch):
        batch.x = self.bn(batch.x)
        return batch


class BatchNorm1dEdge(nn.Module):
    """
    BatchNorm for edge feature.

    Args:
        dim_in (int): Input dimension
    """
    def __init__(self, layer_config: LayerConfig):
        super().__init__()
        self.bn = nn.BatchNorm1d(layer_config.dim_in, eps=layer_config.bn_eps,
                                 momentum=layer_config.bn_mom)

    def forward(self, batch):
        batch.edge_attr = self.bn(batch.edge_attr)
        return batch


@register_layer('mlp')
class MLP(nn.Module):
    """
    Basic MLP model.
    Here 1-layer MLP is equivalent to a Liner layer.

    Args:
        dim_in (int): Input dimension
        dim_out (int): Output dimension
        bias (bool): Whether has bias term
        dim_inner (int): The dimension for the inner layers
        num_layers (int): Number of layers in the stack
        **kwargs (optional): Additional args
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        dim_inner = layer_config.dim_in \
            if layer_config.dim_inner is None \
            else layer_config.dim_inner
        layer_config.has_bias = True
        layers = []
        if layer_config.num_layers > 1:
            sub_layer_config = LayerConfig(
                num_layers=layer_config.num_layers - 1,
                dim_in=layer_config.dim_in, dim_out=dim_inner,
                dim_inner=dim_inner, final_act=True)
            layers.append(GeneralMultiLayer('linear', sub_layer_config))
            layer_config = replace(layer_config, dim_in=dim_inner)
            layers.append(Linear(layer_config))
        else:
            layers.append(Linear(layer_config))
        self.model = nn.Sequential(*layers)

    def forward(self, batch):
        if isinstance(batch, torch.Tensor):
            batch = self.model(batch)
        else:
            batch.x = self.model(batch.x)
        return batch


@register_layer('gcnconv')
class GCNConv(nn.Module):
    """
    Graph Convolutional Network (GCN) layer
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = pyg.nn.GCNConv(layer_config.dim_in, layer_config.dim_out,
                                    bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch


@register_layer('sageconv')
class SAGEConv(nn.Module):
    """
    GraphSAGE Conv layer
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = pyg.nn.SAGEConv(layer_config.dim_in, layer_config.dim_out,
                                     bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch


@register_layer('gatconv')
class GATConv(nn.Module):
    """
    Graph Attention Network (GAT) layer
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = pyg.nn.GATConv(layer_config.dim_in, layer_config.dim_out,
                                    bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch


@register_layer('ginconv')
class GINConv(nn.Module):
    """
    Graph Isomorphism Network (GIN) layer
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        gin_nn = nn.Sequential(
            Linear_pyg(layer_config.dim_in, layer_config.dim_out), nn.ReLU(),
            Linear_pyg(layer_config.dim_out, layer_config.dim_out))
        self.model = pyg.nn.GINConv(gin_nn)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch


@register_layer('splineconv')
class SplineConv(nn.Module):
    """
    SplineCNN layer
    """
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = pyg.nn.SplineConv(layer_config.dim_in,
                                       layer_config.dim_out, dim=1,
                                       kernel_size=2,
                                       bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index, batch.edge_attr)
        return batch


@register_layer('generalconv')
class GeneralConv(nn.Module):
    """A general GNN layer"""
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = GeneralConvLayer(layer_config.dim_in,
                                      layer_config.dim_out,
                                      bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index)
        return batch


@register_layer('generaledgeconv')
class GeneralEdgeConv(nn.Module):
    """A general GNN layer that supports edge features as well"""
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = GeneralEdgeConvLayer(layer_config.dim_in,
                                          layer_config.dim_out,
                                          layer_config.edge_dim,
                                          bias=layer_config.has_bias)

    def forward(self, batch):
        batch.x = self.model(batch.x, batch.edge_index,
                             edge_feature=batch.edge_attr)
        return batch


@register_layer('generalsampleedgeconv')
class GeneralSampleEdgeConv(nn.Module):
    """A general GNN layer that supports edge features and edge sampling"""
    def __init__(self, layer_config: LayerConfig, **kwargs):
        super().__init__()
        self.model = GeneralEdgeConvLayer(layer_config.dim_in,
                                          layer_config.dim_out,
                                          layer_config.edge_dim,
                                          bias=layer_config.has_bias)
        self.keep_edge = layer_config.keep_edge

    def forward(self, batch):
        edge_mask = torch.rand(batch.edge_index.shape[1]) < self.keep_edge
        edge_index = batch.edge_index[:, edge_mask]
        edge_feature = batch.edge_attr[edge_mask, :]
        batch.x = self.model(batch.x, edge_index, edge_feature=edge_feature)
        return batch
pytorch/torch_geometric/graphgym/models/pooling.py000066400000000000000000000004401444262304300230450ustar00rootroot00000000000000from torch_geometric.graphgym.register import register_pooling
from torch_geometric.nn import (
    global_add_pool,
    global_max_pool,
    global_mean_pool,
)

register_pooling('add', global_add_pool)
register_pooling('mean', global_mean_pool)
register_pooling('max', global_max_pool)
pytorch/torch_geometric/graphgym/models/transform.py000066400000000000000000000025571444262304300234240ustar00rootroot00000000000000import torch

from torch_geometric.utils import negative_sampling


def create_link_label(pos_edge_index, neg_edge_index):
    """
    Create labels for link prediction, based on positive and negative edges

    Args:
        pos_edge_index (torch.tensor): Positive edge index [2, num_edges]
        neg_edge_index (torch.tensor): Negative edge index [2, num_edges]

    Returns: Link label tensor, [num_positive_edges + num_negative_edges]

    """
    num_links = pos_edge_index.size(1) + neg_edge_index.size(1)
    link_labels = torch.zeros(num_links, dtype=torch.float,
                              device=pos_edge_index.device)
    link_labels[:pos_edge_index.size(1)] = 1.
    return link_labels


def neg_sampling_transform(data):
    """
    Do negative sampling for link prediction tasks

    Args:
        data (torch_geometric.data): Input data object

    Returns: Transformed data object with negative edges + link pred labels

    """
    train_neg_edge_index = negative_sampling(
        edge_index=data.train_pos_edge_index, num_nodes=data.num_nodes,
        num_neg_samples=data.train_pos_edge_index.size(1))
    data.train_edge_index = torch.cat(
        [data.train_pos_edge_index, train_neg_edge_index], dim=-1)
    data.train_edge_label = create_link_label(data.train_pos_edge_index,
                                              train_neg_edge_index)

    return data
pytorch/torch_geometric/graphgym/optim.py000066400000000000000000000047601444262304300212540ustar00rootroot00000000000000from dataclasses import dataclass, field
from typing import Any, Iterator, List, Optional

from torch.nn import Parameter
from torch.optim import SGD, Adam, Optimizer
from torch.optim.lr_scheduler import CosineAnnealingLR, MultiStepLR, StepLR

import torch_geometric.graphgym.register as register
from torch_geometric.graphgym.config import from_config


@dataclass
class OptimizerConfig:
    optimizer: str = 'adam'  # ['sgd', 'adam']
    base_lr: float = 0.01
    weight_decay: float = 5e-4
    momentum: float = 0.9  # 'sgd' policy


@register.register_optimizer('adam')
def adam_optimizer(params: Iterator[Parameter], base_lr: float,
                   weight_decay: float) -> Adam:
    return Adam(params, lr=base_lr, weight_decay=weight_decay)


@register.register_optimizer('sgd')
def sgd_optimizer(params: Iterator[Parameter], base_lr: float, momentum: float,
                  weight_decay: float) -> SGD:
    return SGD(params, lr=base_lr, momentum=momentum,
               weight_decay=weight_decay)


def create_optimizer(params: Iterator[Parameter], cfg: Any) -> Any:
    r"""Creates a config-driven optimizer."""
    params = filter(lambda p: p.requires_grad, params)
    func = register.optimizer_dict.get(cfg.optimizer, None)
    if func is not None:
        return from_config(func)(params, cfg=cfg)
    raise ValueError(f"Optimizer '{cfg.optimizer}' not supported")


@dataclass
class SchedulerConfig:
    scheduler: Optional[str] = 'cos'  # [None, 'steps', 'cos']
    steps: List[int] = field(default_factory=[30, 60, 90])  # 'steps' policy
    lr_decay: float = 0.1  # 'steps' policy
    max_epoch: int = 200


@register.register_scheduler(None)
@register.register_scheduler('none')
def none_scheduler(optimizer: Optimizer, max_epoch: int) -> StepLR:
    return StepLR(optimizer, step_size=max_epoch + 1)


@register.register_scheduler('step')
def step_scheduler(optimizer: Optimizer, steps: List[int],
                   lr_decay: float) -> MultiStepLR:
    return MultiStepLR(optimizer, milestones=steps, gamma=lr_decay)


@register.register_scheduler('cos')
def cos_scheduler(optimizer: Optimizer, max_epoch: int) -> CosineAnnealingLR:
    return CosineAnnealingLR(optimizer, T_max=max_epoch)


def create_scheduler(optimizer: Optimizer, cfg: Any) -> Any:
    r"""Creates a config-driven learning rate scheduler."""
    func = register.scheduler_dict.get(cfg.scheduler, None)
    if func is not None:
        return from_config(func)(optimizer, cfg=cfg)
    raise ValueError(f"Scheduler '{cfg.scheduler}' not supported")
pytorch/torch_geometric/graphgym/register.py000066400000000000000000000075501444262304300217500ustar00rootroot00000000000000from typing import Any, Callable, Dict, Union

act_dict: Dict[str, Any] = {}
node_encoder_dict: Dict[str, Any] = {}
edge_encoder_dict: Dict[str, Any] = {}
stage_dict: Dict[str, Any] = {}
head_dict: Dict[str, Any] = {}
layer_dict: Dict[str, Any] = {}
pooling_dict: Dict[str, Any] = {}
network_dict: Dict[str, Any] = {}
config_dict: Dict[str, Any] = {}
dataset_dict: Dict[str, Any] = {}
loader_dict: Dict[str, Any] = {}
optimizer_dict: Dict[str, Any] = {}
scheduler_dict: Dict[str, Any] = {}
loss_dict: Dict[str, Any] = {}
train_dict: Dict[str, Any] = {}
metric_dict: Dict[str, Any] = {}


def register_base(mapping: Dict[str, Any], key: str,
                  module: Any = None) -> Union[None, Callable]:
    r"""Base function for registering a module in GraphGym.

    Args:
        mapping (dict): Python dictionary to register the module.
            hosting all the registered modules
        key (str): The name of the module.
        module (any, optional): The module. If set to :obj:`None`, will return
            a decorator to register a module.
    """
    if module is not None:
        if key in mapping:
            raise KeyError(f"Module with '{key}' already defined")
        mapping[key] = module
        return

    # Other-wise, use it as a decorator:
    def bounded_register(module):
        register_base(mapping, key, module)
        return module

    return bounded_register


def register_act(key: str, module: Any = None):
    r"""Registers an activation function in GraphGym."""
    return register_base(act_dict, key, module)


def register_node_encoder(key: str, module: Any = None):
    r"""Registers a node feature encoder in GraphGym."""
    return register_base(node_encoder_dict, key, module)


def register_edge_encoder(key: str, module: Any = None):
    r"""Registers an edge feature encoder in GraphGym."""
    return register_base(edge_encoder_dict, key, module)


def register_stage(key: str, module: Any = None):
    r"""Registers a customized GNN stage in GraphGym."""
    return register_base(stage_dict, key, module)


def register_head(key: str, module: Any = None):
    r"""Registers a GNN prediction head in GraphGym."""
    return register_base(head_dict, key, module)


def register_layer(key: str, module: Any = None):
    r"""Registers a GNN layer in GraphGym."""
    return register_base(layer_dict, key, module)


def register_pooling(key: str, module: Any = None):
    r"""Registers a GNN global pooling/readout layer in GraphGym."""
    return register_base(pooling_dict, key, module)


def register_network(key: str, module: Any = None):
    r"""Registers a GNN model in GraphGym."""
    return register_base(network_dict, key, module)


def register_config(key: str, module: Any = None):
    r"""Registers a configuration group in GraphGym."""
    return register_base(config_dict, key, module)


def register_dataset(key: str, module: Any = None):
    r"""Registers a dataset in GraphGym."""
    return register_base(dataset_dict, key, module)


def register_loader(key: str, module: Any = None):
    r"""Registers a data loader in GraphGym."""
    return register_base(loader_dict, key, module)


def register_optimizer(key: str, module: Any = None):
    r"""Registers an optimizer in GraphGym."""
    return register_base(optimizer_dict, key, module)


def register_scheduler(key: str, module: Any = None):
    r"""Registers a learning rate scheduler in GraphGym."""
    return register_base(scheduler_dict, key, module)


def register_loss(key: str, module: Any = None):
    r"""Registers a loss function in GraphGym."""
    return register_base(loss_dict, key, module)


def register_train(key: str, module: Any = None):
    r"""Registers a training function in GraphGym."""
    return register_base(train_dict, key, module)


def register_metric(key: str, module: Any = None):
    r"""Register a metric function in GraphGym."""
    return register_base(metric_dict, key, module)
pytorch/torch_geometric/graphgym/train.py000066400000000000000000000035371444262304300212420ustar00rootroot00000000000000import warnings
from typing import Optional

import torch
from torch.utils.data import DataLoader

from torch_geometric.data.lightning.datamodule import LightningDataModule
from torch_geometric.graphgym import create_loader
from torch_geometric.graphgym.checkpoint import get_ckpt_dir
from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.imports import pl
from torch_geometric.graphgym.logger import LoggerCallback
from torch_geometric.graphgym.model_builder import GraphGymModule


class GraphGymDataModule(LightningDataModule):
    def __init__(self):
        self.loaders = create_loader()
        super().__init__(has_val=True, has_test=True)

    def train_dataloader(self) -> DataLoader:
        return self.loaders[0]

    def val_dataloader(self) -> DataLoader:
        # better way would be to test after fit.
        # First call trainer.fit(...) then trainer.test(...)
        return self.loaders[1]

    def test_dataloader(self) -> DataLoader:
        return self.loaders[2]


def train(model: GraphGymModule, datamodule, logger: bool = True,
          trainer_config: Optional[dict] = None):
    warnings.filterwarnings('ignore', '.*use `CSVLogger` as the default.*')

    callbacks = []
    if logger:
        callbacks.append(LoggerCallback())
    if cfg.train.enable_ckpt:
        ckpt_cbk = pl.callbacks.ModelCheckpoint(dirpath=get_ckpt_dir())
        callbacks.append(ckpt_cbk)

    trainer_config = trainer_config or {}
    trainer = pl.Trainer(
        **trainer_config,
        enable_checkpointing=cfg.train.enable_ckpt,
        callbacks=callbacks,
        default_root_dir=cfg.out_dir,
        max_epochs=cfg.optim.max_epoch,
        accelerator=cfg.accelerator,
        devices='auto' if not torch.cuda.is_available() else cfg.devices,
    )

    trainer.fit(model, datamodule=datamodule)
    trainer.test(model, datamodule=datamodule)
pytorch/torch_geometric/graphgym/utils/000077500000000000000000000000001444262304300207035ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/utils/LICENSE000066400000000000000000000000001444262304300216760ustar00rootroot00000000000000pytorch/torch_geometric/graphgym/utils/__init__.py000066400000000000000000000012011444262304300230060ustar00rootroot00000000000000from .agg_runs import agg_runs, agg_batch
from .comp_budget import params_count, match_baseline_cfg
from .device import get_current_gpu_usage, auto_select_device
from .epoch import is_eval_epoch, is_ckpt_epoch
from .io import dict_to_json, dict_list_to_json, dict_to_tb, makedirs_rm_exist
from .tools import dummy_context

__all__ = [
    'agg_runs',
    'agg_batch',
    'params_count',
    'match_baseline_cfg',
    'get_current_gpu_usage',
    'auto_select_device',
    'is_eval_epoch',
    'is_ckpt_epoch',
    'dict_to_json',
    'dict_list_to_json',
    'dict_to_tb',
    'makedirs_rm_exist',
    'dummy_context',
]

classes = __all__
pytorch/torch_geometric/graphgym/utils/agg_runs.py000066400000000000000000000224511444262304300230660ustar00rootroot00000000000000import logging
import os

import numpy as np

from torch_geometric.graphgym.config import cfg
from torch_geometric.graphgym.utils.io import (
    dict_list_to_json,
    dict_list_to_tb,
    dict_to_json,
    json_to_dict_list,
    makedirs_rm_exist,
    string_to_python,
)

try:
    from tensorboardX import SummaryWriter
except ImportError:
    SummaryWriter = None


def is_seed(s):
    try:
        int(s)
        return True
    except Exception:
        return False


def is_split(s):
    if s in ['train', 'val']:
        return True
    else:
        return False


def join_list(l1, l2):
    assert len(l1) == len(l2), \
        'Results with different seeds must have the save format'
    for i in range(len(l1)):
        l1[i] += l2[i]
    return l1


def agg_dict_list(dict_list):
    """
    Aggregate a list of dictionaries: mean + std
    Args:
        dict_list: list of dictionaries

    """
    dict_agg = {'epoch': dict_list[0]['epoch']}
    for key in dict_list[0]:
        if key != 'epoch':
            value = np.array([dict[key] for dict in dict_list])
            dict_agg[key] = np.mean(value).round(cfg.round)
            dict_agg['{}_std'.format(key)] = np.std(value).round(cfg.round)
    return dict_agg


def name_to_dict(run):
    run = run.split('-', 1)[-1]
    cols = run.split('=')
    keys, vals = [], []
    keys.append(cols[0])
    for col in cols[1:-1]:
        try:
            val, key = col.rsplit('-', 1)
        except Exception:
            print(col)
        keys.append(key)
        vals.append(string_to_python(val))
    vals.append(cols[-1])
    return dict(zip(keys, vals))


def rm_keys(dict, keys):
    for key in keys:
        dict.pop(key, None)


def agg_runs(dir, metric_best='auto'):
    r'''
    Aggregate over different random seeds of a single experiment

    Args:
        dir (str): Directory of the results, containing 1 experiment
        metric_best (str, optional): The metric for selecting the best
        validation performance. Options: auto, accuracy, auc.

    '''
    results = {'train': None, 'val': None}
    results_best = {'train': None, 'val': None}
    for seed in os.listdir(dir):
        if is_seed(seed):
            dir_seed = os.path.join(dir, seed)

            split = 'val'
            if split in os.listdir(dir_seed):
                dir_split = os.path.join(dir_seed, split)
                fname_stats = os.path.join(dir_split, 'stats.json')
                stats_list = json_to_dict_list(fname_stats)
                if metric_best == 'auto':
                    metric = 'auc' if 'auc' in stats_list[0] else 'accuracy'
                else:
                    metric = metric_best
                performance_np = np.array(  # noqa
                    [stats[metric] for stats in stats_list])
                best_epoch = \
                    stats_list[
                        eval("performance_np.{}()".format(cfg.metric_agg))][
                        'epoch']
                print(best_epoch)

            for split in os.listdir(dir_seed):
                if is_split(split):
                    dir_split = os.path.join(dir_seed, split)
                    fname_stats = os.path.join(dir_split, 'stats.json')
                    stats_list = json_to_dict_list(fname_stats)
                    stats_best = [
                        stats for stats in stats_list
                        if stats['epoch'] == best_epoch
                    ][0]
                    print(stats_best)
                    stats_list = [[stats] for stats in stats_list]
                    if results[split] is None:
                        results[split] = stats_list
                    else:
                        results[split] = join_list(results[split], stats_list)
                    if results_best[split] is None:
                        results_best[split] = [stats_best]
                    else:
                        results_best[split] += [stats_best]
    results = {k: v for k, v in results.items() if v is not None}  # rm None
    results_best = {k: v
                    for k, v in results_best.items()
                    if v is not None}  # rm None
    for key in results:
        for i in range(len(results[key])):
            results[key][i] = agg_dict_list(results[key][i])
    for key in results_best:
        results_best[key] = agg_dict_list(results_best[key])
    # save aggregated results
    for key, value in results.items():
        dir_out = os.path.join(dir, 'agg', key)
        makedirs_rm_exist(dir_out)
        fname = os.path.join(dir_out, 'stats.json')
        dict_list_to_json(value, fname)

        if cfg.tensorboard_agg:
            if SummaryWriter is None:
                raise ImportError(
                    'Tensorboard support requires `tensorboardX`.')
            writer = SummaryWriter(dir_out)
            dict_list_to_tb(value, writer)
            writer.close()
    for key, value in results_best.items():
        dir_out = os.path.join(dir, 'agg', key)
        fname = os.path.join(dir_out, 'best.json')
        dict_to_json(value, fname)
    logging.info('Results aggregated across runs saved in {}'.format(
        os.path.join(dir, 'agg')))


def agg_batch(dir, metric_best='auto'):
    r'''
    Aggregate across results from multiple experiments via grid search

    Args:
        dir (str): Directory of the results, containing multiple experiments
        metric_best (str, optional): The metric for selecting the best
        validation performance. Options: auto, accuracy, auc.

    '''
    import pandas as pd
    results = {'train': [], 'val': [], 'test': []}
    for run in os.listdir(dir):
        if run != 'agg':
            dict_name = name_to_dict(run)
            dir_run = os.path.join(dir, run, 'agg')
            if os.path.isdir(dir_run):
                for split in os.listdir(dir_run):
                    dir_split = os.path.join(dir_run, split)
                    fname_stats = os.path.join(dir_split, 'best.json')
                    dict_stats = json_to_dict_list(fname_stats)[
                        -1]  # get best val epoch
                    rm_keys(dict_stats,
                            ['lr', 'lr_std', 'eta', 'eta_std', 'params_std'])
                    results[split].append({**dict_name, **dict_stats})
    dir_out = os.path.join(dir, 'agg')
    makedirs_rm_exist(dir_out)
    for key in results:
        if len(results[key]) > 0:
            results[key] = pd.DataFrame(results[key])
            results[key] = results[key].sort_values(
                list(dict_name.keys()), ascending=[True] * len(dict_name))
            fname = os.path.join(dir_out, '{}_best.csv'.format(key))
            results[key].to_csv(fname, index=False)

    results = {'train': [], 'val': [], 'test': []}
    for run in os.listdir(dir):
        if run != 'agg':
            dict_name = name_to_dict(run)
            dir_run = os.path.join(dir, run, 'agg')
            if os.path.isdir(dir_run):
                for split in os.listdir(dir_run):
                    dir_split = os.path.join(dir_run, split)
                    fname_stats = os.path.join(dir_split, 'stats.json')
                    dict_stats = json_to_dict_list(fname_stats)[
                        -1]  # get last epoch
                    rm_keys(dict_stats,
                            ['lr', 'lr_std', 'eta', 'eta_std', 'params_std'])
                    results[split].append({**dict_name, **dict_stats})
    dir_out = os.path.join(dir, 'agg')
    for key in results:
        if len(results[key]) > 0:
            results[key] = pd.DataFrame(results[key])
            results[key] = results[key].sort_values(
                list(dict_name.keys()), ascending=[True] * len(dict_name))
            fname = os.path.join(dir_out, '{}.csv'.format(key))
            results[key].to_csv(fname, index=False)

    results = {'train': [], 'val': [], 'test': []}
    for run in os.listdir(dir):
        if run != 'agg':
            dict_name = name_to_dict(run)
            dir_run = os.path.join(dir, run, 'agg')
            if os.path.isdir(dir_run):
                for split in os.listdir(dir_run):
                    dir_split = os.path.join(dir_run, split)
                    fname_stats = os.path.join(dir_split, 'stats.json')
                    dict_stats = json_to_dict_list(
                        fname_stats)  # get best epoch
                    if metric_best == 'auto':
                        metric = 'auc' if 'auc' in dict_stats[0] \
                            else 'accuracy'
                    else:
                        metric = metric_best
                    performance_np = np.array(  # noqa
                        [stats[metric] for stats in dict_stats])
                    dict_stats = dict_stats[eval("performance_np.{}()".format(
                        cfg.metric_agg))]
                    rm_keys(dict_stats,
                            ['lr', 'lr_std', 'eta', 'eta_std', 'params_std'])
                    results[split].append({**dict_name, **dict_stats})
    dir_out = os.path.join(dir, 'agg')
    for key in results:
        if len(results[key]) > 0:
            results[key] = pd.DataFrame(results[key])
            results[key] = results[key].sort_values(
                list(dict_name.keys()), ascending=[True] * len(dict_name))
            fname = os.path.join(dir_out, '{}_bestepoch.csv'.format(key))
            results[key].to_csv(fname, index=False)

    print('Results aggregated across models saved in {}'.format(dir_out))
pytorch/torch_geometric/graphgym/utils/comp_budget.py000066400000000000000000000057601444262304300235550ustar00rootroot00000000000000import math

from torch_geometric.graphgym.config import cfg, set_cfg
from torch_geometric.graphgym.model_builder import create_model


def params_count(model):
    '''
    Computes the number of parameters.

    Args:
        model (nn.Module): PyTorch model

    '''
    return sum([p.numel() for p in model.parameters()])


def get_stats():
    model = create_model(to_device=False, dim_in=1, dim_out=1)
    return params_count(model)


def match_computation(stats_baseline, key=['gnn', 'dim_inner'], mode='sqrt'):
    '''Match computation budget by modifying cfg.gnn.dim_inner'''
    stats = get_stats()
    if stats != stats_baseline:
        # Phase 1: fast approximation
        while True:
            if mode == 'sqrt':
                scale = math.sqrt(stats_baseline / stats)
            elif mode == 'linear':
                scale = stats_baseline / stats
            step = int(round(cfg[key[0]][key[1]] * scale)) \
                - cfg[key[0]][key[1]]
            cfg[key[0]][key[1]] += step
            stats = get_stats()
            if abs(step) <= 1:
                break
        # Phase 2: fine tune
        flag_init = 1 if stats < stats_baseline else -1
        step = 1
        while True:
            cfg[key[0]][key[1]] += flag_init * step
            stats = get_stats()
            flag = 1 if stats < stats_baseline else -1
            if stats == stats_baseline:
                return stats
            if flag != flag_init:
                if not cfg.model.match_upper:  # stats is SMALLER
                    if flag < 0:
                        cfg[key[0]][key[1]] -= flag_init * step
                    return get_stats()
                else:
                    if flag > 0:
                        cfg[key[0]][key[1]] -= flag_init * step
                    return get_stats()
    return stats


def dict_to_stats(cfg_dict):
    from yacs.config import CfgNode as CN
    set_cfg(cfg)
    cfg_new = CN(cfg_dict)
    cfg.merge_from_other_cfg(cfg_new)
    stats = get_stats()
    set_cfg(cfg)
    return stats


def match_baseline_cfg(cfg_dict, cfg_dict_baseline, verbose=True):
    '''
    Match the computational budget of a given baseline model. THe current
    configuration dictionary will be modifed and returned.

    Args:
        cfg_dict (dict): Current experiment's configuration
        cfg_dict_baseline (dict): Baseline configuration
        verbose (str, optional): If printing matched paramter conunts


    '''
    from yacs.config import CfgNode as CN
    stats_baseline = dict_to_stats(cfg_dict_baseline)
    set_cfg(cfg)
    cfg_new = CN(cfg_dict)
    cfg.merge_from_other_cfg(cfg_new)
    stats = match_computation(stats_baseline, key=['gnn', 'dim_inner'])
    if 'gnn' in cfg_dict:
        cfg_dict['gnn']['dim_inner'] = cfg.gnn.dim_inner
    else:
        cfg_dict['gnn'] = {'dim_inner', cfg.gnn.dim_inner}
    set_cfg(cfg)
    if verbose:
        print('Computational budget has matched: Baseline params {}, '
              'Current params {}'.format(stats_baseline, stats))
    return cfg_dict
pytorch/torch_geometric/graphgym/utils/device.py000066400000000000000000000025101444262304300225120ustar00rootroot00000000000000import os
import subprocess

import numpy as np
import torch

from torch_geometric.graphgym.config import cfg


def get_gpu_memory_map():
    '''Get the current gpu usage.'''
    result = subprocess.check_output([
        'nvidia-smi', '--query-gpu=memory.used',
        '--format=csv,nounits,noheader'
    ], encoding='utf-8')
    gpu_memory = np.array([int(x) for x in result.strip().split('\n')])
    return gpu_memory


def get_current_gpu_usage():
    '''
    Get the current GPU memory usage.
    '''
    if cfg.gpu_mem and cfg.device != 'cpu' and torch.cuda.is_available():
        result = subprocess.check_output([
            'nvidia-smi', '--query-compute-apps=pid,used_memory',
            '--format=csv,nounits,noheader'
        ], encoding='utf-8')
        current_pid = os.getpid()
        used_memory = 0
        for line in result.strip().split('\n'):
            line = line.split(', ')
            if current_pid == int(line[0]):
                used_memory += int(line[1])
        return used_memory
    else:
        return -1


def auto_select_device():
    r"""Auto select device for the current experiment."""
    if cfg.accelerator == 'auto':
        if torch.cuda.is_available():
            cfg.accelerator = 'cuda'
            cfg.devices = 1
        else:
            cfg.accelerator = 'cpu'
            cfg.devices = None
pytorch/torch_geometric/graphgym/utils/epoch.py000066400000000000000000000012621444262304300223540ustar00rootroot00000000000000from torch_geometric.graphgym.config import cfg


def is_train_eval_epoch(cur_epoch):
    """Determines if the model should be evaluated at the training epoch."""
    return is_eval_epoch(cur_epoch) or not cfg.train.skip_train_eval


def is_eval_epoch(cur_epoch):
    """Determines if the model should be evaluated at the current epoch."""
    return ((cur_epoch + 1) % cfg.train.eval_period == 0 or cur_epoch == 0
            or (cur_epoch + 1) == cfg.optim.max_epoch)


def is_ckpt_epoch(cur_epoch):
    """Determines if the model should be evaluated at the current epoch."""
    return ((cur_epoch + 1) % cfg.train.ckpt_period == 0
            or (cur_epoch + 1) == cfg.optim.max_epoch)
pytorch/torch_geometric/graphgym/utils/io.py000066400000000000000000000040021444262304300216600ustar00rootroot00000000000000import ast
import json
import os
import shutil


def string_to_python(string):
    try:
        return ast.literal_eval(string)
    except Exception:
        return string


def dict_to_json(dict, fname):
    '''
    Dump a Python dictionary to JSON file

    Args:
        dict (dict): Python dictionary
        fname (str): Output file name

    '''
    with open(fname, 'a') as f:
        json.dump(dict, f)
        f.write('\n')


def dict_list_to_json(dict_list, fname):
    '''
    Dump a list of Python dictionaries to JSON file

    Args:
        dict_list (list of dict): List of Python dictionaries
        fname (str): Output file name

    '''
    with open(fname, 'a') as f:
        for dict in dict_list:
            json.dump(dict, f)
            f.write('\n')


def json_to_dict_list(fname):
    dict_list = []
    epoch_set = set()
    with open(fname) as f:
        lines = f.readlines()
        for line in lines:
            line = line.rstrip()
            dict = json.loads(line)
            if dict['epoch'] not in epoch_set:
                dict_list.append(dict)
            epoch_set.add(dict['epoch'])
    return dict_list


def dict_to_tb(dict, writer, epoch):
    '''
    Add a dictionary of statistics to a Tensorboard writer

    Args:
        dict (dict): Statistics of experiments, the keys are attribute names,
        the values are the attribute values
        writer: Tensorboard writer object
        epoch (int): The current epoch

    '''
    for key in dict:
        writer.add_scalar(key, dict[key], epoch)


def dict_list_to_tb(dict_list, writer):
    for dict in dict_list:
        assert 'epoch' in dict, 'Key epoch must exist in stats dict'
        dict_to_tb(dict, writer, dict['epoch'])


def makedirs(dir):
    os.makedirs(dir, exist_ok=True)


def makedirs_rm_exist(dir):
    '''
    Make a directory, remove any existing data.

    Args:
        dir (str): The directory to be created.

    Returns:

    '''
    if os.path.isdir(dir):
        shutil.rmtree(dir)
    os.makedirs(dir, exist_ok=True)
pytorch/torch_geometric/graphgym/utils/plot.py000066400000000000000000000011361444262304300222340ustar00rootroot00000000000000def view_emb(emb, dir):
    '''
    Visualize a embedding matrix.

    Args:
        emb (torch.tensor): Embedding matrix with shape (N, D). D is the
        feature dimension.
        dir (str): Output directory for the embedding figure.

    '''
    import matplotlib.pyplot as plt
    import seaborn as sns
    from sklearn.decomposition import PCA

    sns.set_context('poster')

    if emb.shape[1] > 2:
        pca = PCA(n_components=2)
        emb = pca.fit_transform(emb)
    plt.figure(figsize=(10, 10))
    plt.scatter(emb[:, 0], emb[:, 1])
    plt.savefig('{}/emb_pca.png'.format(dir), dpi=100)
pytorch/torch_geometric/graphgym/utils/tools.py000066400000000000000000000003061444262304300224140ustar00rootroot00000000000000class dummy_context():
    '''Default context manager that does nothing'''
    def __enter__(self):
        return None

    def __exit__(self, exc_type, exc_value, traceback):
        return False
pytorch/torch_geometric/home.py000066400000000000000000000014761444262304300172370ustar00rootroot00000000000000import os
import os.path as osp
from typing import Optional

ENV_PYG_HOME = 'PYG_HOME'
DEFAULT_CACHE_DIR = osp.join('~', '.cache', 'pyg')

_home_dir: Optional[str] = None


def get_home_dir() -> str:
    r"""Get the cache directory used for storing all :pyg:`PyG`-related data.

    If :meth:`set_home_dir` is not called, the path is given by the environment
    variable :obj:`$PYG_HOME` which defaults to :obj:`"~/.cache/pyg"`.
    """
    if _home_dir is not None:
        return _home_dir

    home_dir = os.getenv(ENV_PYG_HOME, DEFAULT_CACHE_DIR)
    home_dir = osp.expanduser(home_dir)
    return home_dir


def set_home_dir(path: str):
    r"""Set the cache directory used for storing all :pyg:`PyG`-related data.

    Args:
        path (str): The path to a local folder.
    """
    global _home_dir
    _home_dir = path
pytorch/torch_geometric/io/000077500000000000000000000000001444262304300163345ustar00rootroot00000000000000pytorch/torch_geometric/io/__init__.py000066400000000000000000000010201444262304300204360ustar00rootroot00000000000000from .txt_array import parse_txt_array, read_txt_array
from .tu import read_tu_data
from .planetoid import read_planetoid_data
from .ply import read_ply
from .obj import read_obj
from .sdf import read_sdf, parse_sdf
from .off import read_off, write_off
from .npz import read_npz, parse_npz

__all__ = [
    'read_off',
    'write_off',
    'parse_txt_array',
    'read_txt_array',
    'read_tu_data',
    'read_planetoid_data',
    'read_ply',
    'read_obj',
    'read_sdf',
    'parse_sdf',
    'read_npz',
    'parse_npz',
]
pytorch/torch_geometric/io/npz.py000066400000000000000000000021741444262304300175210ustar00rootroot00000000000000from typing import Any, Dict

import numpy as np
import scipy.sparse as sp
import torch

from torch_geometric.data import Data
from torch_geometric.utils import remove_self_loops
from torch_geometric.utils import to_undirected as to_undirected_fn


def read_npz(path: str, to_undirected: bool = True) -> Data:
    with np.load(path) as f:
        return parse_npz(f)


def parse_npz(f: Dict[str, Any], to_undirected: bool = True) -> Data:
    x = sp.csr_matrix((f['attr_data'], f['attr_indices'], f['attr_indptr']),
                      f['attr_shape']).todense()
    x = torch.from_numpy(x).to(torch.float)
    x[x > 0] = 1

    adj = sp.csr_matrix((f['adj_data'], f['adj_indices'], f['adj_indptr']),
                        f['adj_shape']).tocoo()
    row = torch.from_numpy(adj.row).to(torch.long)
    col = torch.from_numpy(adj.col).to(torch.long)
    edge_index = torch.stack([row, col], dim=0)
    edge_index, _ = remove_self_loops(edge_index)
    if to_undirected:
        edge_index = to_undirected_fn(edge_index, num_nodes=x.size(0))

    y = torch.from_numpy(f['labels']).to(torch.long)

    return Data(x=x, edge_index=edge_index, y=y)
pytorch/torch_geometric/io/obj.py000066400000000000000000000016751444262304300174710ustar00rootroot00000000000000import torch

from torch_geometric.data import Data


def yield_file(in_file):
    f = open(in_file)
    buf = f.read()
    f.close()
    for b in buf.split('\n'):
        if b.startswith('v '):
            yield ['v', [float(x) for x in b.split(" ")[1:]]]
        elif b.startswith('f '):
            triangles = b.split(' ')[1:]
            # -1 as .obj is base 1 but the Data class expects base 0 indices
            yield ['f', [int(t.split("/")[0]) - 1 for t in triangles]]
        else:
            yield ['', ""]


def read_obj(in_file):
    vertices = []
    faces = []

    for k, v in yield_file(in_file):
        if k == 'v':
            vertices.append(v)
        elif k == 'f':
            faces.append(v)

    if not len(faces) or not len(vertices):
        return None

    pos = torch.tensor(vertices, dtype=torch.float)
    face = torch.tensor(faces, dtype=torch.long).t().contiguous()

    data = Data(pos=pos, face=face)

    return data
pytorch/torch_geometric/io/off.py000066400000000000000000000050321444262304300174600ustar00rootroot00000000000000import re

import torch
from torch._tensor_str import PRINT_OPTS, _tensor_str

from torch_geometric.data import Data
from torch_geometric.io import parse_txt_array


def parse_off(src):
    # Some files may contain a bug and do not have a carriage return after OFF.
    if src[0] == 'OFF':
        src = src[1:]
    else:
        src[0] = src[0][3:]

    num_nodes, num_faces = [int(item) for item in src[0].split()[:2]]

    pos = parse_txt_array(src[1:1 + num_nodes])

    face = src[1 + num_nodes:1 + num_nodes + num_faces]
    face = face_to_tri(face)

    data = Data(pos=pos)
    data.face = face

    return data


def face_to_tri(face):
    face = [[int(x) for x in line.strip().split()] for line in face]

    triangle = torch.tensor([line[1:] for line in face if line[0] == 3])
    triangle = triangle.to(torch.int64)

    rect = torch.tensor([line[1:] for line in face if line[0] == 4])
    rect = rect.to(torch.int64)

    if rect.numel() > 0:
        first, second = rect[:, [0, 1, 2]], rect[:, [0, 2, 3]]
        return torch.cat([triangle, first, second], dim=0).t().contiguous()
    else:
        return triangle.t().contiguous()


def read_off(path):
    r"""Reads an OFF (Object File Format) file, returning both the position of
    nodes and their connectivity in a :class:`torch_geometric.data.Data`
    object.

    Args:
        path (str): The path to the file.
    """
    with open(path, 'r') as f:
        src = f.read().split('\n')[:-1]
    return parse_off(src)


def write_off(data, path):
    r"""Writes a :class:`torch_geometric.data.Data` object to an OFF (Object
    File Format) file.

    Args:
        data (:class:`torch_geometric.data.Data`): The data object.
        path (str): The path to the file.
    """
    num_nodes, num_faces = data.pos.size(0), data.face.size(1)

    pos = data.pos.to(torch.float)
    face = data.face.t()
    num_vertices = torch.full((num_faces, 1), face.size(1), dtype=torch.long)
    face = torch.cat([num_vertices, face], dim=-1)

    threshold = PRINT_OPTS.threshold
    torch.set_printoptions(threshold=float('inf'))

    pos_repr = re.sub(',', '', _tensor_str(pos, indent=0))
    pos_repr = '\n'.join([x[2:-1] for x in pos_repr.split('\n')])[:-1]

    face_repr = re.sub(',', '', _tensor_str(face, indent=0))
    face_repr = '\n'.join([x[2:-1] for x in face_repr.split('\n')])[:-1]

    with open(path, 'w') as f:
        f.write(f'OFF\n{num_nodes} {num_faces} 0\n')
        f.write(pos_repr)
        f.write('\n')
        f.write(face_repr)
        f.write('\n')
    torch.set_printoptions(threshold=threshold)
pytorch/torch_geometric/io/planetoid.py000066400000000000000000000102231444262304300206630ustar00rootroot00000000000000import os.path as osp
import sys
import warnings
from itertools import repeat

import torch

from torch_geometric.data import Data
from torch_geometric.io import read_txt_array
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import coalesce, index_to_mask, remove_self_loops

try:
    import cPickle as pickle
except ImportError:
    import pickle


def read_planetoid_data(folder, prefix):
    names = ['x', 'tx', 'allx', 'y', 'ty', 'ally', 'graph', 'test.index']
    items = [read_file(folder, prefix, name) for name in names]
    x, tx, allx, y, ty, ally, graph, test_index = items
    train_index = torch.arange(y.size(0), dtype=torch.long)
    val_index = torch.arange(y.size(0), y.size(0) + 500, dtype=torch.long)
    sorted_test_index = test_index.sort()[0]

    if prefix.lower() == 'citeseer':
        # There are some isolated nodes in the Citeseer graph, resulting in
        # none consecutive test indices. We need to identify them and add them
        # as zero vectors to `tx` and `ty`.
        len_test_indices = (test_index.max() - test_index.min()).item() + 1

        tx_ext = torch.zeros(len_test_indices, tx.size(1), dtype=tx.dtype)
        tx_ext[sorted_test_index - test_index.min(), :] = tx
        ty_ext = torch.zeros(len_test_indices, ty.size(1), dtype=ty.dtype)
        ty_ext[sorted_test_index - test_index.min(), :] = ty

        tx, ty = tx_ext, ty_ext

    if prefix.lower() == 'nell.0.001':
        tx_ext = torch.zeros(len(graph) - allx.size(0), x.size(1))
        tx_ext[sorted_test_index - allx.size(0)] = tx

        ty_ext = torch.zeros(len(graph) - ally.size(0), y.size(1))
        ty_ext[sorted_test_index - ally.size(0)] = ty

        tx, ty = tx_ext, ty_ext

        x = torch.cat([allx, tx], dim=0)
        x[test_index] = x[sorted_test_index]

        # Creating feature vectors for relations.
        row, col, value = SparseTensor.from_dense(x).coo()
        rows, cols, values = [row], [col], [value]

        mask1 = index_to_mask(test_index, size=len(graph))
        mask2 = index_to_mask(torch.arange(allx.size(0), len(graph)),
                              size=len(graph))
        mask = ~mask1 | ~mask2
        isolated_index = mask.nonzero(as_tuple=False).view(-1)[allx.size(0):]

        rows += [isolated_index]
        cols += [torch.arange(isolated_index.size(0)) + x.size(1)]
        values += [torch.ones(isolated_index.size(0))]

        x = SparseTensor(row=torch.cat(rows), col=torch.cat(cols),
                         value=torch.cat(values))
    else:
        x = torch.cat([allx, tx], dim=0)
        x[test_index] = x[sorted_test_index]

    y = torch.cat([ally, ty], dim=0).max(dim=1)[1]
    y[test_index] = y[sorted_test_index]

    train_mask = index_to_mask(train_index, size=y.size(0))
    val_mask = index_to_mask(val_index, size=y.size(0))
    test_mask = index_to_mask(test_index, size=y.size(0))

    edge_index = edge_index_from_dict(graph, num_nodes=y.size(0))

    data = Data(x=x, edge_index=edge_index, y=y)
    data.train_mask = train_mask
    data.val_mask = val_mask
    data.test_mask = test_mask

    return data


def read_file(folder, prefix, name):
    path = osp.join(folder, f'ind.{prefix.lower()}.{name}')

    if name == 'test.index':
        return read_txt_array(path, dtype=torch.long)

    with open(path, 'rb') as f:
        if sys.version_info > (3, 0):
            warnings.filterwarnings('ignore', '.*`scipy.sparse.csr` name.*')
            out = pickle.load(f, encoding='latin1')
        else:
            out = pickle.load(f)

    if name == 'graph':
        return out

    out = out.todense() if hasattr(out, 'todense') else out
    out = torch.from_numpy(out).to(torch.float)
    return out


def edge_index_from_dict(graph_dict, num_nodes=None):
    row, col = [], []
    for key, value in graph_dict.items():
        row += repeat(key, len(value))
        col += value
    edge_index = torch.stack([torch.tensor(row), torch.tensor(col)], dim=0)

    # NOTE: There are some duplicated edges and self loops in the datasets.
    #       Other implementations do not remove them!
    edge_index, _ = remove_self_loops(edge_index)
    edge_index = coalesce(edge_index, num_nodes=num_nodes)

    return edge_index
pytorch/torch_geometric/io/ply.py000066400000000000000000000007341444262304300175160ustar00rootroot00000000000000import torch

from torch_geometric.data import Data

try:
    import openmesh
except ImportError:
    openmesh = None


def read_ply(path):
    if openmesh is None:
        raise ImportError('`read_ply` requires the `openmesh` package.')

    mesh = openmesh.read_trimesh(path)
    pos = torch.from_numpy(mesh.points()).to(torch.float)
    face = torch.from_numpy(mesh.face_vertex_indices())
    face = face.t().to(torch.long).contiguous()
    return Data(pos=pos, face=face)
pytorch/torch_geometric/io/sdf.py000066400000000000000000000021601444262304300174610ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.io import parse_txt_array
from torch_geometric.utils import coalesce, one_hot

elems = {'H': 0, 'C': 1, 'N': 2, 'O': 3, 'F': 4}


def parse_sdf(src):
    src = src.split('\n')[3:]
    num_atoms, num_bonds = [int(item) for item in src[0].split()[:2]]

    atom_block = src[1:num_atoms + 1]
    pos = parse_txt_array(atom_block, end=3)
    x = torch.tensor([elems[item.split()[3]] for item in atom_block])
    x = one_hot(x, num_classes=len(elems))

    bond_block = src[1 + num_atoms:1 + num_atoms + num_bonds]
    row, col = parse_txt_array(bond_block, end=2, dtype=torch.long).t() - 1
    row, col = torch.cat([row, col], dim=0), torch.cat([col, row], dim=0)
    edge_index = torch.stack([row, col], dim=0)
    edge_attr = parse_txt_array(bond_block, start=2, end=3) - 1
    edge_attr = torch.cat([edge_attr, edge_attr], dim=0)
    edge_index, edge_attr = coalesce(edge_index, edge_attr, num_atoms)

    return Data(x=x, edge_index=edge_index, edge_attr=edge_attr, pos=pos)


def read_sdf(path):
    with open(path, 'r') as f:
        return parse_sdf(f.read())
pytorch/torch_geometric/io/tu.py000066400000000000000000000111751444262304300173430ustar00rootroot00000000000000import glob
import os
import os.path as osp

import numpy as np
import torch

from torch_geometric.data import Data
from torch_geometric.io import read_txt_array
from torch_geometric.utils import coalesce, one_hot, remove_self_loops

names = [
    'A', 'graph_indicator', 'node_labels', 'node_attributes'
    'edge_labels', 'edge_attributes', 'graph_labels', 'graph_attributes'
]


def read_tu_data(folder, prefix):
    files = glob.glob(osp.join(folder, f'{prefix}_*.txt'))
    names = [f.split(os.sep)[-1][len(prefix) + 1:-4] for f in files]

    edge_index = read_file(folder, prefix, 'A', torch.long).t() - 1
    batch = read_file(folder, prefix, 'graph_indicator', torch.long) - 1

    node_attributes = torch.empty((batch.size(0), 0))
    if 'node_attributes' in names:
        node_attributes = read_file(folder, prefix, 'node_attributes')
        if node_attributes.dim() == 1:
            node_attributes = node_attributes.unsqueeze(-1)

    node_labels = torch.empty((batch.size(0), 0))
    if 'node_labels' in names:
        node_labels = read_file(folder, prefix, 'node_labels', torch.long)
        if node_labels.dim() == 1:
            node_labels = node_labels.unsqueeze(-1)
        node_labels = node_labels - node_labels.min(dim=0)[0]
        node_labels = node_labels.unbind(dim=-1)
        node_labels = [one_hot(x) for x in node_labels]
        if len(node_labels) == 1:
            node_labels = node_labels[0]
        else:
            node_labels = torch.cat(node_labels, dim=-1)

    edge_attributes = torch.empty((edge_index.size(1), 0))
    if 'edge_attributes' in names:
        edge_attributes = read_file(folder, prefix, 'edge_attributes')
        if edge_attributes.dim() == 1:
            edge_attributes = edge_attributes.unsqueeze(-1)

    edge_labels = torch.empty((edge_index.size(1), 0))
    if 'edge_labels' in names:
        edge_labels = read_file(folder, prefix, 'edge_labels', torch.long)
        if edge_labels.dim() == 1:
            edge_labels = edge_labels.unsqueeze(-1)
        edge_labels = edge_labels - edge_labels.min(dim=0)[0]
        edge_labels = edge_labels.unbind(dim=-1)
        edge_labels = [one_hot(e) for e in edge_labels]
        if len(edge_labels) == 1:
            edge_labels = edge_labels[0]
        else:
            edge_labels = torch.cat(edge_labels, dim=-1)

    x = cat([node_attributes, node_labels])
    edge_attr = cat([edge_attributes, edge_labels])

    y = None
    if 'graph_attributes' in names:  # Regression problem.
        y = read_file(folder, prefix, 'graph_attributes')
    elif 'graph_labels' in names:  # Classification problem.
        y = read_file(folder, prefix, 'graph_labels', torch.long)
        _, y = y.unique(sorted=True, return_inverse=True)

    num_nodes = edge_index.max().item() + 1 if x is None else x.size(0)
    edge_index, edge_attr = remove_self_loops(edge_index, edge_attr)
    edge_index, edge_attr = coalesce(edge_index, edge_attr, num_nodes)

    data = Data(x=x, edge_index=edge_index, edge_attr=edge_attr, y=y)
    data, slices = split(data, batch)

    sizes = {
        'num_node_attributes': node_attributes.size(-1),
        'num_node_labels': node_labels.size(-1),
        'num_edge_attributes': edge_attributes.size(-1),
        'num_edge_labels': edge_labels.size(-1),
    }

    return data, slices, sizes


def read_file(folder, prefix, name, dtype=None):
    path = osp.join(folder, f'{prefix}_{name}.txt')
    return read_txt_array(path, sep=',', dtype=dtype)


def cat(seq):
    seq = [item for item in seq if item is not None]
    seq = [item for item in seq if item.numel() > 0]
    seq = [item.unsqueeze(-1) if item.dim() == 1 else item for item in seq]
    return torch.cat(seq, dim=-1) if len(seq) > 0 else None


def split(data, batch):
    node_slice = torch.cumsum(torch.from_numpy(np.bincount(batch)), 0)
    node_slice = torch.cat([torch.tensor([0]), node_slice])

    row, _ = data.edge_index
    edge_slice = torch.cumsum(torch.from_numpy(np.bincount(batch[row])), 0)
    edge_slice = torch.cat([torch.tensor([0]), edge_slice])

    # Edge indices should start at zero for every graph.
    data.edge_index -= node_slice[batch[row]].unsqueeze(0)

    slices = {'edge_index': edge_slice}
    if data.x is not None:
        slices['x'] = node_slice
    else:
        # Imitate `collate` functionality:
        data._num_nodes = torch.bincount(batch).tolist()
        data.num_nodes = batch.numel()
    if data.edge_attr is not None:
        slices['edge_attr'] = edge_slice
    if data.y is not None:
        if data.y.size(0) == batch.size(0):
            slices['y'] = node_slice
        else:
            slices['y'] = torch.arange(0, batch[-1] + 2, dtype=torch.long)

    return data, slices
pytorch/torch_geometric/io/txt_array.py000066400000000000000000000010621444262304300207220ustar00rootroot00000000000000import torch


def parse_txt_array(src, sep=None, start=0, end=None, dtype=None, device=None):
    to_number = int
    if torch.is_floating_point(torch.empty(0, dtype=dtype)):
        to_number = float

    src = [[to_number(x) for x in line.split(sep)[start:end]] for line in src]
    src = torch.tensor(src).to(dtype).squeeze()
    return src


def read_txt_array(path, sep=None, start=0, end=None, dtype=None, device=None):
    with open(path, 'r') as f:
        src = f.read().split('\n')[:-1]
    return parse_txt_array(src, sep, start, end, dtype, device)
pytorch/torch_geometric/lazy_loader.py000066400000000000000000000014001444262304300205770ustar00rootroot00000000000000from importlib import import_module
from types import ModuleType


# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/
# python/util/lazy_loader.py
class LazyLoader(ModuleType):
    def __init__(self, local_name, parent_module_globals, name):
        self._local_name = local_name
        self._parent_module_globals = parent_module_globals
        super().__init__(name)

    def _load(self):
        module = import_module(self.__name__)
        self._parent_module_globals[self._local_name] = module
        self.__dict__.update(module.__dict__)
        return module

    def __getattr__(self, item):
        module = self._load()
        return getattr(module, item)

    def __dir__(self):
        module = self._load()
        return dir(module)
pytorch/torch_geometric/loader/000077500000000000000000000000001444262304300171735ustar00rootroot00000000000000pytorch/torch_geometric/loader/__init__.py000066400000000000000000000032131444262304300213030ustar00rootroot00000000000000from torch_geometric.deprecation import deprecated

from .dataloader import DataLoader
from .node_loader import NodeLoader
from .link_loader import LinkLoader
from .neighbor_loader import NeighborLoader
from .link_neighbor_loader import LinkNeighborLoader
from .hgt_loader import HGTLoader
from .cluster import ClusterData, ClusterLoader
from .graph_saint import (GraphSAINTSampler, GraphSAINTNodeSampler,
                          GraphSAINTEdgeSampler, GraphSAINTRandomWalkSampler)
from .shadow import ShaDowKHopSampler
from .random_node_loader import RandomNodeLoader
from .zip_loader import ZipLoader
from .data_list_loader import DataListLoader
from .dense_data_loader import DenseDataLoader
from .temporal_dataloader import TemporalDataLoader
from .neighbor_sampler import NeighborSampler
from .imbalanced_sampler import ImbalancedSampler
from .dynamic_batch_sampler import DynamicBatchSampler
from .prefetch import PrefetchLoader
from .mixin import AffinityMixin

__all__ = classes = [
    'DataLoader',
    'NodeLoader',
    'LinkLoader',
    'NeighborLoader',
    'LinkNeighborLoader',
    'HGTLoader',
    'ClusterData',
    'ClusterLoader',
    'GraphSAINTSampler',
    'GraphSAINTNodeSampler',
    'GraphSAINTEdgeSampler',
    'GraphSAINTRandomWalkSampler',
    'ShaDowKHopSampler',
    'RandomNodeLoader',
    'ZipLoader',
    'DataListLoader',
    'DenseDataLoader',
    'TemporalDataLoader',
    'NeighborSampler',
    'ImbalancedSampler',
    'DynamicBatchSampler',
    'PrefetchLoader',
    'AffinityMixin',
]

RandomNodeSampler = deprecated(
    details="use 'loader.RandomNodeLoader' instead",
    func_name='loader.RandomNodeSampler',
)(RandomNodeLoader)
pytorch/torch_geometric/loader/base.py000066400000000000000000000026311444262304300204610ustar00rootroot00000000000000from typing import Any, Callable

from torch.utils.data.dataloader import _BaseDataLoaderIter


class DataLoaderIterator:
    r"""A data loader iterator extended by a simple post transformation
    function :meth:`transform_fn`. While the iterator may request items from
    different sub-processes, :meth:`transform_fn` will always be executed in
    the main process.

    This iterator is used in PyG's sampler classes, and is responsible for
    feature fetching and filtering data objects after sampling has taken place
    in a sub-process. This has the following advantages:

    * We do not need to share feature matrices across processes which may
      prevent any errors due to too many open file handles.
    * We can execute any expensive post-processing commands on the main thread
      with full parallelization power (which usually executes faster).
    * It lets us naturally support data already being present on the GPU.
    """
    def __init__(self, iterator: _BaseDataLoaderIter, transform_fn: Callable):
        self.iterator = iterator
        self.transform_fn = transform_fn

    def __iter__(self) -> 'DataLoaderIterator':
        return self

    def _reset(self, loader: Any, first_iter: bool = False):
        self.iterator._reset(loader, first_iter)

    def __len__(self) -> int:
        return len(self.iterator)

    def __next__(self) -> Any:
        return self.transform_fn(next(self.iterator))
pytorch/torch_geometric/loader/cluster.py000066400000000000000000000265121444262304300212340ustar00rootroot00000000000000import copy
import os.path as osp
import sys
from dataclasses import dataclass
from typing import List, Optional

import torch
import torch.utils.data
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data import Data
from torch_geometric.typing import pyg_lib
from torch_geometric.utils import index_sort, narrow, select, sort_edge_index
from torch_geometric.utils.map import map_index
from torch_geometric.utils.sparse import index2ptr, ptr2index


@dataclass
class Partition:
    rowptr: Tensor
    col: Tensor
    partptr: Tensor
    node_perm: Tensor
    edge_perm: Tensor


class ClusterData(torch.utils.data.Dataset):
    r"""Clusters/partitions a graph data object into multiple subgraphs, as
    motivated by the `"Cluster-GCN: An Efficient Algorithm for Training Deep
    and Large Graph Convolutional Networks"
    `_ paper.

    .. note::
        The underlying METIS algorithm requires undirected graphs as input.

    Args:
        data (torch_geometric.data.Data): The graph data object.
        num_parts (int): The number of partitions.
        recursive (bool, optional): If set to :obj:`True`, will use multilevel
            recursive bisection instead of multilevel k-way partitioning.
            (default: :obj:`False`)
        save_dir (str, optional): If set, will save the partitioned data to the
            :obj:`save_dir` directory for faster re-use. (default: :obj:`None`)
        log (bool, optional): If set to :obj:`False`, will not log any
            progress. (default: :obj:`True`)
        keep_inter_cluster_edges (bool, optional): If set to :obj:`True`,
            will keep inter-cluster edge connections. (default: :obj:`False`)
    """
    def __init__(
        self,
        data,
        num_parts: int,
        recursive: bool = False,
        save_dir: Optional[str] = None,
        log: bool = True,
        keep_inter_cluster_edges: bool = False,
    ):
        assert data.edge_index is not None

        self.num_parts = num_parts
        self.recursive = recursive
        self.keep_inter_cluster_edges = keep_inter_cluster_edges

        recursive_str = '_recursive' if recursive else ''
        filename = f'metis_{num_parts}{recursive_str}.pt'
        path = osp.join(save_dir or '', filename)
        if save_dir is not None and osp.exists(path):
            self.partition = torch.load(path)
        else:
            if log:  # pragma: no cover
                print('Computing METIS partitioning...', file=sys.stderr)

            cluster = self._metis(data.edge_index, data.num_nodes)
            self.partition = self._partition(data.edge_index, cluster)

            if save_dir is not None:
                torch.save(self.partition, path)

            if log:  # pragma: no cover
                print('Done!', file=sys.stderr)

        self.data = self._permute_data(data, self.partition)

    def _metis(self, edge_index: Tensor, num_nodes: int) -> Tensor:
        # Computes a node-level partition assignment vector via METIS.

        # Calculate CSR representation:
        row, col = sort_edge_index(edge_index, num_nodes=num_nodes)
        rowptr = index2ptr(row, size=num_nodes)

        # Compute METIS partitioning:
        cluster: Optional[Tensor] = None

        if torch_geometric.typing.WITH_TORCH_SPARSE:
            try:
                cluster = torch.ops.torch_sparse.partition(
                    rowptr.cpu(),
                    col.cpu(),
                    None,
                    self.num_parts,
                    self.recursive,
                ).to(edge_index.device)
            except (AttributeError, RuntimeError):
                pass

        if cluster is None and torch_geometric.typing.WITH_METIS:
            cluster = pyg_lib.partition.metis(
                rowptr.cpu(),
                col.cpu(),
                self.num_parts,
                recursive=self.recursive,
            ).to(edge_index.device)

        if cluster is None:
            raise ImportError(f"'{self.__class__.__name__}' requires either "
                              f"'pyg-lib' or 'torch-sparse'")

        return cluster

    def _partition(self, edge_index: Tensor, cluster: Tensor) -> Partition:
        # Computes node-level and edge-level permutations and permutes the edge
        # connectivity accordingly:

        # Sort `cluster` and compute boundaries `partptr`:
        cluster, node_perm = index_sort(cluster, max_value=self.num_parts)
        partptr = index2ptr(cluster, size=self.num_parts)

        # Permute `edge_index` based on node permutation:
        edge_perm = torch.arange(edge_index.size(1), device=edge_index.device)
        arange = torch.empty_like(node_perm)
        arange[node_perm] = torch.arange(cluster.numel(),
                                         device=cluster.device)
        edge_index = arange[edge_index]

        # Compute final CSR representation:
        (row, col), edge_perm = sort_edge_index(
            edge_index,
            edge_attr=edge_perm,
            num_nodes=cluster.numel(),
        )
        rowptr = index2ptr(row, size=cluster.numel())

        return Partition(rowptr, col, partptr, node_perm, edge_perm)

    def _permute_data(self, data: Data, partition: Partition) -> Data:
        # Permute node-level and edge-level attributes according to the
        # calculated permutations in `Partition`:
        out = copy.copy(data)
        for key, value in data.items():
            if key == 'edge_index':
                continue
            elif data.is_node_attr(key):
                cat_dim = data.__cat_dim__(key, value)
                out[key] = select(value, partition.node_perm, dim=cat_dim)
            elif data.is_edge_attr(key):
                cat_dim = data.__cat_dim__(key, value)
                out[key] = select(value, partition.edge_perm, dim=cat_dim)
        out.edge_index = None

        return out

    def __len__(self) -> int:
        return self.partition.partptr.numel() - 1

    def __getitem__(self, idx: int) -> Data:
        node_start = int(self.partition.partptr[idx])
        node_end = int(self.partition.partptr[idx + 1])
        node_length = node_end - node_start

        rowptr = self.partition.rowptr[node_start:node_end + 1]
        edge_start = int(rowptr[0])
        edge_end = int(rowptr[-1])
        edge_length = edge_end - edge_start
        rowptr = rowptr - edge_start
        row = ptr2index(rowptr)
        col = self.partition.col[edge_start:edge_end]
        if not self.keep_inter_cluster_edges:
            edge_mask = (col >= node_start) & (col < node_end)
            row = row[edge_mask]
            col = col[edge_mask] - node_start

        out = copy.copy(self.data)

        for key, value in self.data.items():
            if key == 'num_nodes':
                out.num_nodes = node_length
            elif self.data.is_node_attr(key):
                cat_dim = self.data.__cat_dim__(key, value)
                out[key] = narrow(value, cat_dim, node_start, node_length)
            elif self.data.is_edge_attr(key):
                cat_dim = self.data.__cat_dim__(key, value)
                out[key] = narrow(value, cat_dim, edge_start, edge_length)
                if not self.keep_inter_cluster_edges:
                    out[key] = out[key][edge_mask]

        out.edge_index = torch.stack([row, col], dim=0)

        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num_parts})'


class ClusterLoader(torch.utils.data.DataLoader):
    r"""The data loader scheme from the `"Cluster-GCN: An Efficient Algorithm
    for Training Deep and Large Graph Convolutional Networks"
    `_ paper which merges partioned subgraphs
    and their between-cluster links from a large-scale graph data object to
    form a mini-batch.

    .. note::

        Use :class:`~torch_geometric.loader.ClusterData` and
        :class:`~torch_geometric.loader.ClusterLoader` in conjunction to
        form mini-batches of clusters.
        For an example of using Cluster-GCN, see
        `examples/cluster_gcn_reddit.py `_ or
        `examples/cluster_gcn_ppi.py `_.

    Args:
        cluster_data (torch_geometric.loader.ClusterData): The already
            partioned data object.
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(self, cluster_data, **kwargs):
        self.cluster_data = cluster_data
        iterator = range(len(cluster_data))
        super().__init__(iterator, collate_fn=self._collate, **kwargs)

    def _collate(self, batch: List[int]) -> Data:
        if not isinstance(batch, torch.Tensor):
            batch = torch.tensor(batch)

        global_rowptr = self.cluster_data.partition.rowptr
        global_col = self.cluster_data.partition.col

        # Get all node-level and edge-level start and end indices for the
        # current mini-batch:
        node_start = self.cluster_data.partition.partptr[batch]
        node_end = self.cluster_data.partition.partptr[batch + 1]
        edge_start = global_rowptr[node_start]
        edge_end = global_rowptr[node_end]

        # Iterate over each partition in the batch and calculate new edge
        # connectivity. This is done by slicing the corresponding source and
        # destination indices for each partition and adjusting their indices to
        # start from zero:
        rows, cols, nodes, cumsum = [], [], [], 0
        for i in range(batch.numel()):
            nodes.append(torch.arange(node_start[i], node_end[i]))
            rowptr = global_rowptr[node_start[i]:node_end[i] + 1]
            rowptr = rowptr - edge_start[i]
            row = ptr2index(rowptr) + cumsum
            col = global_col[edge_start[i]:edge_end[i]]
            rows.append(row)
            cols.append(col)
            cumsum += rowptr.numel() - 1

        node = torch.cat(nodes, dim=0)
        row = torch.cat(rows, dim=0)
        col = torch.cat(cols, dim=0)

        # Map `col` vector to valid entries and remove any entries that do not
        # connect two nodes within the same mini-batch:
        col, edge_mask = map_index(col, node)
        row = row[edge_mask]

        out = copy.copy(self.cluster_data.data)

        # Slice node-level and edge-level attributes according to its offsets:
        for key, value in self.cluster_data.data.items():
            if key == 'num_nodes':
                out.num_nodes = cumsum
            elif self.cluster_data.data.is_node_attr(key):
                cat_dim = self.cluster_data.data.__cat_dim__(key, value)
                out[key] = torch.cat([
                    narrow(out[key], cat_dim, s, e - s)
                    for s, e in zip(node_start, node_end)
                ], dim=cat_dim)
            elif self.cluster_data.data.is_edge_attr(key):
                cat_dim = self.cluster_data.data.__cat_dim__(key, value)
                value = torch.cat([
                    narrow(out[key], cat_dim, s, e - s)
                    for s, e in zip(edge_start, edge_end)
                ], dim=cat_dim)
                out[key] = select(value, edge_mask, dim=cat_dim)

        out.edge_index = torch.stack([row, col], dim=0)

        return out
pytorch/torch_geometric/loader/data_list_loader.py000066400000000000000000000026511444262304300230430ustar00rootroot00000000000000from typing import List, Union

import torch

from torch_geometric.data import Dataset
from torch_geometric.data.data import BaseData


def collate_fn(data_list):
    return data_list


class DataListLoader(torch.utils.data.DataLoader):
    r"""A data loader which batches data objects from a
    :class:`torch_geometric.data.dataset` to a Python list.
    Data objects can be either of type :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData`.

    .. note::

        This data loader should be used for multi-GPU support via
        :class:`torch_geometric.nn.DataParallel`.

    Args:
        dataset (Dataset): The dataset from which to load the data.
        batch_size (int, optional): How many samples per batch to load.
            (default: :obj:`1`)
        shuffle (bool, optional): If set to :obj:`True`, the data will be
            reshuffled at every epoch. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`drop_last` or
            :obj:`num_workers`.
    """
    def __init__(self, dataset: Union[Dataset, List[BaseData]],
                 batch_size: int = 1, shuffle: bool = False, **kwargs):
        # Remove for PyTorch Lightning:
        kwargs.pop('collate_fn', None)

        super().__init__(dataset, batch_size=batch_size, shuffle=shuffle,
                         collate_fn=collate_fn, **kwargs)
pytorch/torch_geometric/loader/dataloader.py000066400000000000000000000062261444262304300216530ustar00rootroot00000000000000from collections.abc import Mapping
from typing import List, Optional, Sequence, Union

import torch.utils.data
from torch.utils.data.dataloader import default_collate

from torch_geometric.data import Batch, Dataset
from torch_geometric.data.data import BaseData
from torch_geometric.data.datapipes import DatasetAdapter


class Collater:
    def __init__(self, follow_batch, exclude_keys):
        self.follow_batch = follow_batch
        self.exclude_keys = exclude_keys

    def __call__(self, batch):
        elem = batch[0]
        if isinstance(elem, BaseData):
            return Batch.from_data_list(batch, self.follow_batch,
                                        self.exclude_keys)
        elif isinstance(elem, torch.Tensor):
            return default_collate(batch)
        elif isinstance(elem, float):
            return torch.tensor(batch, dtype=torch.float)
        elif isinstance(elem, int):
            return torch.tensor(batch)
        elif isinstance(elem, str):
            return batch
        elif isinstance(elem, Mapping):
            return {key: self([data[key] for data in batch]) for key in elem}
        elif isinstance(elem, tuple) and hasattr(elem, '_fields'):
            return type(elem)(*(self(s) for s in zip(*batch)))
        elif isinstance(elem, Sequence) and not isinstance(elem, str):
            return [self(s) for s in zip(*batch)]

        raise TypeError(f'DataLoader found invalid type: {type(elem)}')

    def collate(self, batch):  # pragma: no cover
        # TODO Deprecated, remove soon.
        return self(batch)


class DataLoader(torch.utils.data.DataLoader):
    r"""A data loader which merges data objects from a
    :class:`torch_geometric.data.Dataset` to a mini-batch.
    Data objects can be either of type :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData`.

    Args:
        dataset (Dataset): The dataset from which to load the data.
        batch_size (int, optional): How many samples per batch to load.
            (default: :obj:`1`)
        shuffle (bool, optional): If set to :obj:`True`, the data will be
            reshuffled at every epoch. (default: :obj:`False`)
        follow_batch (List[str], optional): Creates assignment batch
            vectors for each key in the list. (default: :obj:`None`)
        exclude_keys (List[str], optional): Will exclude each key in the
            list. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`.
    """
    def __init__(
        self,
        dataset: Union[Dataset, Sequence[BaseData], DatasetAdapter],
        batch_size: int = 1,
        shuffle: bool = False,
        follow_batch: Optional[List[str]] = None,
        exclude_keys: Optional[List[str]] = None,
        **kwargs,
    ):
        # Remove for PyTorch Lightning:
        kwargs.pop('collate_fn', None)

        # Save for PyTorch Lightning < 1.6:
        self.follow_batch = follow_batch
        self.exclude_keys = exclude_keys

        super().__init__(
            dataset,
            batch_size,
            shuffle,
            collate_fn=Collater(follow_batch, exclude_keys),
            **kwargs,
        )
pytorch/torch_geometric/loader/dense_data_loader.py000066400000000000000000000032231444262304300231620ustar00rootroot00000000000000from typing import List, Union

import torch
from torch.utils.data.dataloader import default_collate

from torch_geometric.data import Batch, Data, Dataset


def collate_fn(data_list: List[Data]) -> Batch:
    batch = Batch()
    for key in data_list[0].keys:
        batch[key] = default_collate([data[key] for data in data_list])
    return batch


class DenseDataLoader(torch.utils.data.DataLoader):
    r"""A data loader which batches data objects from a
    :class:`torch_geometric.data.dataset` to a
    :class:`torch_geometric.data.Batch` object by stacking all attributes in a
    new dimension.

    .. note::

        To make use of this data loader, all graph attributes in the dataset
        need to have the same shape.
        In particular, this data loader should only be used when working with
        *dense* adjacency matrices.

    Args:
        dataset (Dataset): The dataset from which to load the data.
        batch_size (int, optional): How many samples per batch to load.
            (default: :obj:`1`)
        shuffle (bool, optional): If set to :obj:`True`, the data will be
            reshuffled at every epoch. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`drop_last` or
            :obj:`num_workers`.
    """
    def __init__(self, dataset: Union[Dataset, List[Data]],
                 batch_size: int = 1, shuffle: bool = False, **kwargs):
        # Remove for PyTorch Lightning:
        kwargs.pop('collate_fn', None)

        super().__init__(dataset, batch_size=batch_size, shuffle=shuffle,
                         collate_fn=collate_fn, **kwargs)
pytorch/torch_geometric/loader/dynamic_batch_sampler.py000066400000000000000000000102751444262304300240620ustar00rootroot00000000000000import warnings
from typing import Iterator, List, Optional

import torch

from torch_geometric.data import Dataset


class DynamicBatchSampler(torch.utils.data.sampler.Sampler):
    r"""Dynamically adds samples to a mini-batch up to a maximum size (either
    based on number of nodes or number of edges). When data samples have a
    wide range in sizes, specifying a mini-batch size in terms of number of
    samples is not ideal and can cause CUDA OOM errors.

    Within the :class:`DynamicBatchSampler`, the number of steps per epoch is
    ambiguous, depending on the order of the samples. By default the
    :meth:`__len__` will be undefined. This is fine for most cases but
    progress bars will be infinite. Alternatively, :obj:`num_steps` can be
    supplied to cap the number of mini-batches produced by the sampler.

    .. code-block:: python

        from torch_geometric.loader import DataLoader, DynamicBatchSampler

        sampler = DynamicBatchSampler(dataset, max_num=10000, mode="node")
        loader = DataLoader(dataset, batch_sampler=sampler, ...)

    Args:
        dataset (Dataset): Dataset to sample from.
        max_num (int): Size of mini-batch to aim for in number of nodes or
            edges.
        mode (str, optional): :obj:`"node"` or :obj:`"edge"` to measure
            batch size. (default: :obj:`"node"`)
        shuffle (bool, optional): If set to :obj:`True`, will have the data
            reshuffled at every epoch. (default: :obj:`False`)
        skip_too_big (bool, optional): If set to :obj:`True`, skip samples
            which cannot fit in a batch by itself. (default: :obj:`False`)
        num_steps (int, optional): The number of mini-batches to draw for a
            single epoch. If set to :obj:`None`, will iterate through all the
            underlying examples, but :meth:`__len__` will be :obj:`None` since
            it is be ambiguous. (default: :obj:`None`)
    """
    def __init__(self, dataset: Dataset, max_num: int, mode: str = 'node',
                 shuffle: bool = False, skip_too_big: bool = False,
                 num_steps: Optional[int] = None):
        if not isinstance(max_num, int) or max_num <= 0:
            raise ValueError("`max_num` should be a positive integer value "
                             "(got {max_num}).")
        if mode not in ['node', 'edge']:
            raise ValueError("`mode` choice should be either "
                             f"'node' or 'edge' (got '{mode}').")

        if num_steps is None:
            num_steps = len(dataset)

        self.dataset = dataset
        self.max_num = max_num
        self.mode = mode
        self.shuffle = shuffle
        self.skip_too_big = skip_too_big
        self.num_steps = num_steps

    def __iter__(self) -> Iterator[List[int]]:
        batch = []
        batch_n = 0
        num_steps = 0
        num_processed = 0

        if self.shuffle:
            indices = torch.randperm(len(self.dataset), dtype=torch.long)
        else:
            indices = torch.arange(len(self.dataset), dtype=torch.long)

        while (num_processed < len(self.dataset)
               and num_steps < self.num_steps):
            # Fill batch
            for idx in indices[num_processed:]:
                # Size of sample
                data = self.dataset[idx]
                n = data.num_nodes if self.mode == 'node' else data.num_edges

                if batch_n + n > self.max_num:
                    if batch_n == 0:
                        if self.skip_too_big:
                            continue
                        else:
                            warnings.warn("Size of data sample at index "
                                          f"{idx} is larger than "
                                          f"{self.max_num} {self.mode}s "
                                          f"(got {n} {self.mode}s.")
                    else:
                        # Mini-batch filled
                        break

                # Add sample to current batch
                batch.append(idx.item())
                num_processed += 1
                batch_n += n

            yield batch
            batch = []
            batch_n = 0
            num_steps += 1

    def __len__(self) -> int:
        return self.num_steps
pytorch/torch_geometric/loader/graph_saint.py000066400000000000000000000204001444262304300220400ustar00rootroot00000000000000import os.path as osp
from typing import Optional

import torch
from tqdm import tqdm

from torch_geometric.typing import SparseTensor


class GraphSAINTSampler(torch.utils.data.DataLoader):
    r"""The GraphSAINT sampler base class from the `"GraphSAINT: Graph
    Sampling Based Inductive Learning Method"
    `_ paper.
    Given a graph in a :obj:`data` object, this class samples nodes and
    constructs subgraphs that can be processed in a mini-batch fashion.
    Normalization coefficients for each mini-batch are given via
    :obj:`node_norm` and :obj:`edge_norm` data attributes.

    .. note::

        See :class:`~torch_geometric.loader.GraphSAINTNodeSampler`,
        :class:`~torch_geometric.loader.GraphSAINTEdgeSampler` and
        :class:`~torch_geometric.loader.GraphSAINTRandomWalkSampler` for
        currently supported samplers.
        For an example of using GraphSAINT sampling, see
        `examples/graph_saint.py `_.

    Args:
        data (torch_geometric.data.Data): The graph data object.
        batch_size (int): The approximate number of samples per batch.
        num_steps (int, optional): The number of iterations per epoch.
            (default: :obj:`1`)
        sample_coverage (int): How many samples per node should be used to
            compute normalization statistics. (default: :obj:`0`)
        save_dir (str, optional): If set, will save normalization statistics to
            the :obj:`save_dir` directory for faster re-use.
            (default: :obj:`None`)
        log (bool, optional): If set to :obj:`False`, will not log any
            pre-processing progress. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size` or
            :obj:`num_workers`.
    """
    def __init__(self, data, batch_size: int, num_steps: int = 1,
                 sample_coverage: int = 0, save_dir: Optional[str] = None,
                 log: bool = True, **kwargs):

        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        assert data.edge_index is not None
        assert 'node_norm' not in data
        assert 'edge_norm' not in data
        assert not data.edge_index.is_cuda

        self.num_steps = num_steps
        self._batch_size = batch_size
        self.sample_coverage = sample_coverage
        self.save_dir = save_dir
        self.log = log

        self.N = N = data.num_nodes
        self.E = data.num_edges

        self.adj = SparseTensor(
            row=data.edge_index[0], col=data.edge_index[1],
            value=torch.arange(self.E, device=data.edge_index.device),
            sparse_sizes=(N, N))

        self.data = data

        super().__init__(self, batch_size=1, collate_fn=self._collate,
                         **kwargs)

        if self.sample_coverage > 0:
            path = osp.join(save_dir or '', self._filename)
            if save_dir is not None and osp.exists(path):  # pragma: no cover
                self.node_norm, self.edge_norm = torch.load(path)
            else:
                self.node_norm, self.edge_norm = self._compute_norm()
                if save_dir is not None:  # pragma: no cover
                    torch.save((self.node_norm, self.edge_norm), path)

    @property
    def _filename(self):
        return f'{self.__class__.__name__.lower()}_{self.sample_coverage}.pt'

    def __len__(self):
        return self.num_steps

    def _sample_nodes(self, batch_size):
        raise NotImplementedError

    def __getitem__(self, idx):
        node_idx = self._sample_nodes(self._batch_size).unique()
        adj, _ = self.adj.saint_subgraph(node_idx)
        return node_idx, adj

    def _collate(self, data_list):
        assert len(data_list) == 1
        node_idx, adj = data_list[0]

        data = self.data.__class__()
        data.num_nodes = node_idx.size(0)
        row, col, edge_idx = adj.coo()
        data.edge_index = torch.stack([row, col], dim=0)

        for key, item in self.data:
            if key in ['edge_index', 'num_nodes']:
                continue
            if isinstance(item, torch.Tensor) and item.size(0) == self.N:
                data[key] = item[node_idx]
            elif isinstance(item, torch.Tensor) and item.size(0) == self.E:
                data[key] = item[edge_idx]
            else:
                data[key] = item

        if self.sample_coverage > 0:
            data.node_norm = self.node_norm[node_idx]
            data.edge_norm = self.edge_norm[edge_idx]

        return data

    def _compute_norm(self):
        node_count = torch.zeros(self.N, dtype=torch.float)
        edge_count = torch.zeros(self.E, dtype=torch.float)

        loader = torch.utils.data.DataLoader(self, batch_size=200,
                                             collate_fn=lambda x: x,
                                             num_workers=self.num_workers)

        if self.log:  # pragma: no cover
            pbar = tqdm(total=self.N * self.sample_coverage)
            pbar.set_description('Compute GraphSAINT normalization')

        num_samples = total_sampled_nodes = 0
        while total_sampled_nodes < self.N * self.sample_coverage:
            for data in loader:
                for node_idx, adj in data:
                    edge_idx = adj.storage.value()
                    node_count[node_idx] += 1
                    edge_count[edge_idx] += 1
                    total_sampled_nodes += node_idx.size(0)

                    if self.log:  # pragma: no cover
                        pbar.update(node_idx.size(0))
            num_samples += self.num_steps

        if self.log:  # pragma: no cover
            pbar.close()

        row, _, edge_idx = self.adj.coo()
        t = torch.empty_like(edge_count).scatter_(0, edge_idx, node_count[row])
        edge_norm = (t / edge_count).clamp_(0, 1e4)
        edge_norm[torch.isnan(edge_norm)] = 0.1

        node_count[node_count == 0] = 0.1
        node_norm = num_samples / node_count / self.N

        return node_norm, edge_norm


class GraphSAINTNodeSampler(GraphSAINTSampler):
    r"""The GraphSAINT node sampler class (see
    :class:`~torch_geometric.loader.GraphSAINTSampler`).
    """
    def _sample_nodes(self, batch_size):
        edge_sample = torch.randint(0, self.E, (batch_size, self.batch_size),
                                    dtype=torch.long)

        return self.adj.storage.row()[edge_sample]


class GraphSAINTEdgeSampler(GraphSAINTSampler):
    r"""The GraphSAINT edge sampler class (see
    :class:`~torch_geometric.loader.GraphSAINTSampler`).
    """
    def _sample_nodes(self, batch_size):
        row, col, _ = self.adj.coo()

        deg_in = 1. / self.adj.storage.colcount()
        deg_out = 1. / self.adj.storage.rowcount()
        prob = (1. / deg_in[row]) + (1. / deg_out[col])

        # Parallel multinomial sampling (without replacement)
        # https://github.com/pytorch/pytorch/issues/11931#issuecomment-625882503
        rand = torch.rand(batch_size, self.E).log() / (prob + 1e-10)
        edge_sample = rand.topk(self.batch_size, dim=-1).indices

        source_node_sample = col[edge_sample]
        target_node_sample = row[edge_sample]

        return torch.cat([source_node_sample, target_node_sample], -1)


class GraphSAINTRandomWalkSampler(GraphSAINTSampler):
    r"""The GraphSAINT random walk sampler class (see
    :class:`~torch_geometric.loader.GraphSAINTSampler`).

    Args:
        walk_length (int): The length of each random walk.
    """
    def __init__(self, data, batch_size: int, walk_length: int,
                 num_steps: int = 1, sample_coverage: int = 0,
                 save_dir: Optional[str] = None, log: bool = True, **kwargs):
        self.walk_length = walk_length
        super().__init__(data, batch_size, num_steps, sample_coverage,
                         save_dir, log, **kwargs)

    @property
    def _filename(self):
        return (f'{self.__class__.__name__.lower()}_{self.walk_length}_'
                f'{self.sample_coverage}.pt')

    def _sample_nodes(self, batch_size):
        start = torch.randint(0, self.N, (batch_size, ), dtype=torch.long)
        node_idx = self.adj.random_walk(start.flatten(), self.walk_length)
        return node_idx.view(-1)
pytorch/torch_geometric/loader/hgt_loader.py000066400000000000000000000135741444262304300216670ustar00rootroot00000000000000from typing import Callable, Dict, List, Optional, Tuple, Union

from torch import Tensor

from torch_geometric.data import FeatureStore, GraphStore, HeteroData
from torch_geometric.loader import NodeLoader
from torch_geometric.sampler import HGTSampler
from torch_geometric.typing import NodeType


class HGTLoader(NodeLoader):
    r"""The Heterogeneous Graph Sampler from the `"Heterogeneous Graph
    Transformer" `_ paper.
    This loader allows for mini-batch training of GNNs on large-scale graphs
    where full-batch training is not feasible.

    :class:`~torch_geometric.data.HGTLoader` tries to (1) keep a similar
    number of nodes and edges for each type and (2) keep the sampled sub-graph
    dense to minimize the information loss and reduce the sample variance.

    Methodically, :class:`~torch_geometric.data.HGTLoader` keeps track of a
    node budget for each node type, which is then used to determine the
    sampling probability of a node.
    In particular, the probability of sampling a node is determined by the
    number of connections to already sampled nodes and their node degrees.
    With this, :class:`~torch_geometric.data.HGTLoader` will sample a fixed
    amount of neighbors for each node type in each iteration, as given by the
    :obj:`num_samples` argument.

    Sampled nodes are sorted based on the order in which they were sampled.
    In particular, the first :obj:`batch_size` nodes represent the set of
    original mini-batch nodes.

    .. note::

        For an example of using :class:`~torch_geometric.data.HGTLoader`, see
        `examples/hetero/to_hetero_mag.py
        `_.

    .. code-block:: python

        from torch_geometric.loader import HGTLoader
        from torch_geometric.datasets import OGB_MAG

        hetero_data = OGB_MAG(path)[0]

        loader = HGTLoader(
            hetero_data,
            # Sample 512 nodes per type and per iteration for 4 iterations
            num_samples={key: [512] * 4 for key in hetero_data.node_types},
            # Use a batch size of 128 for sampling training nodes of type paper
            batch_size=128,
            input_nodes=('paper', hetero_data['paper'].train_mask),
        )

        sampled_hetero_data = next(iter(loader))
        print(sampled_data.batch_size)
        >>> 128

    Args:
        data (Any): A :class:`~torch_geometric.data.Data`,
            :class:`~torch_geometric.data.HeteroData`, or
            (:class:`~torch_geometric.data.FeatureStore`,
            :class:`~torch_geometric.data.GraphStore`) data object.
        num_samples (List[int] or Dict[str, List[int]]): The number of nodes to
            sample in each iteration and for each node type.
            If given as a list, will sample the same amount of nodes for each
            node type.
        input_nodes (str or Tuple[str, torch.Tensor]): The indices of nodes for
            which neighbors are sampled to create mini-batches.
            Needs to be passed as a tuple that holds the node type and
            corresponding node indices.
            Node indices need to be either given as a :obj:`torch.LongTensor`
            or :obj:`torch.BoolTensor`.
            If node indices are set to :obj:`None`, all nodes of this specific
            type will be considered.
        transform (callable, optional): A function/transform that takes in
            an a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        transform_sampler_output (callable, optional): A function/transform
            that takes in a :class:`torch_geometric.sampler.SamplerOutput` and
            returns a transformed version. (default: :obj:`None`)
        is_sorted (bool, optional): If set to :obj:`True`, assumes that
            :obj:`edge_index` is sorted by column. This avoids internal
            re-sorting of the data and can improve runtime and memory
            efficiency. (default: :obj:`False`)
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[HeteroData, Tuple[FeatureStore, GraphStore]],
        num_samples: Union[List[int], Dict[NodeType, List[int]]],
        input_nodes: Union[NodeType, Tuple[NodeType, Optional[Tensor]]],
        is_sorted: bool = False,
        transform: Optional[Callable] = None,
        transform_sampler_output: Optional[Callable] = None,
        filter_per_worker: Optional[bool] = None,
        **kwargs,
    ):
        hgt_sampler = HGTSampler(
            data,
            num_samples=num_samples,
            is_sorted=is_sorted,
            share_memory=kwargs.get('num_workers', 0) > 0,
        )

        super().__init__(
            data=data,
            node_sampler=hgt_sampler,
            input_nodes=input_nodes,
            transform=transform,
            transform_sampler_output=transform_sampler_output,
            filter_per_worker=filter_per_worker,
            **kwargs,
        )
pytorch/torch_geometric/loader/imbalanced_sampler.py000066400000000000000000000072521444262304300233550ustar00rootroot00000000000000from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, Dataset, InMemoryDataset


class ImbalancedSampler(torch.utils.data.WeightedRandomSampler):
    r"""A weighted random sampler that randomly samples elements according to
    class distribution.
    As such, it will either remove samples from the majority class
    (under-sampling) or add more examples from the minority class
    (over-sampling).

    **Graph-level sampling:**

    .. code-block:: python

        from torch_geometric.loader import DataLoader, ImbalancedSampler

        sampler = ImbalancedSampler(dataset)
        loader = DataLoader(dataset, batch_size=64, sampler=sampler, ...)

    **Node-level sampling:**

    .. code-block:: python

        from torch_geometric.loader import NeighborLoader, ImbalancedSampler

        sampler = ImbalancedSampler(data, input_nodes=data.train_mask)
        loader = NeighborLoader(data, input_nodes=data.train_mask,
                                batch_size=64, num_neighbors=[-1, -1],
                                sampler=sampler, ...)

    You can also pass in the class labels directly as a :class:`torch.Tensor`:

    .. code-block:: python

        from torch_geometric.loader import NeighborLoader, ImbalancedSampler

        sampler = ImbalancedSampler(data.y)
        loader = NeighborLoader(data, input_nodes=data.train_mask,
                                batch_size=64, num_neighbors=[-1, -1],
                                sampler=sampler, ...)

    Args:
        dataset (Dataset or Data or Tensor): The dataset or class distribution
            from which to sample the data, given either as a
            :class:`~torch_geometric.data.Dataset`,
            :class:`~torch_geometric.data.Data`, or :class:`torch.Tensor`
            object.
        input_nodes (Tensor, optional): The indices of nodes that are used by
            the corresponding loader, *e.g.*, by
            :class:`~torch_geometric.loader.NeighborLoader`.
            If set to :obj:`None`, all nodes will be considered.
            This argument should only be set for node-level loaders and does
            not have any effect when operating on a set of graphs as given by
            :class:`~torch_geometric.data.Dataset`. (default: :obj:`None`)
        num_samples (int, optional): The number of samples to draw for a single
            epoch. If set to :obj:`None`, will sample as much elements as there
            exists in the underlying data. (default: :obj:`None`)
    """
    def __init__(
        self,
        dataset: Union[Dataset, Data, List[Data], Tensor],
        input_nodes: Optional[Tensor] = None,
        num_samples: Optional[int] = None,
    ):

        if isinstance(dataset, Data):
            y = dataset.y.view(-1)
            assert dataset.num_nodes == y.numel()
            y = y[input_nodes] if input_nodes is not None else y

        elif isinstance(dataset, Tensor):
            y = dataset.view(-1)
            y = y[input_nodes] if input_nodes is not None else y

        elif isinstance(dataset, InMemoryDataset):
            y = dataset.y.view(-1)
            assert len(dataset) == y.numel()

        else:
            ys = [data.y for data in dataset]
            if isinstance(ys[0], Tensor):
                y = torch.cat(ys, dim=0).view(-1)
            else:
                y = torch.tensor(ys).view(-1)
            assert len(dataset) == y.numel()

        assert y.dtype == torch.long  # Require classification.

        num_samples = y.numel() if num_samples is None else num_samples

        class_weight = 1. / y.bincount()
        weight = class_weight[y]

        return super().__init__(weight, num_samples, replacement=True)
pytorch/torch_geometric/loader/link_loader.py000066400000000000000000000335731444262304300220430ustar00rootroot00000000000000from typing import Any, Callable, Iterator, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, FeatureStore, GraphStore, HeteroData
from torch_geometric.loader.base import DataLoaderIterator
from torch_geometric.loader.mixin import AffinityMixin
from torch_geometric.loader.utils import (
    filter_custom_store,
    filter_data,
    filter_hetero_data,
    get_edge_label_index,
    infer_filter_per_worker,
)
from torch_geometric.sampler import (
    BaseSampler,
    EdgeSamplerInput,
    HeteroSamplerOutput,
    NegativeSampling,
    SamplerOutput,
)
from torch_geometric.typing import InputEdges, OptTensor


class LinkLoader(torch.utils.data.DataLoader, AffinityMixin):
    r"""A data loader that performs mini-batch sampling from link information,
    using a generic :class:`~torch_geometric.sampler.BaseSampler`
    implementation that defines a
    :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges` function and
    is supported on the provided input :obj:`data` object.

    .. note::
        Negative sampling is currently implemented in an approximate
        way, *i.e.* negative edges may contain false negatives.

    Args:
        data (Any): A :class:`~torch_geometric.data.Data`,
            :class:`~torch_geometric.data.HeteroData`, or
            (:class:`~torch_geometric.data.FeatureStore`,
            :class:`~torch_geometric.data.GraphStore`) data object.
        link_sampler (torch_geometric.sampler.BaseSampler): The sampler
            implementation to be used with this loader.
            Needs to implement
            :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges`.
            The sampler implementation must be compatible with the input
            :obj:`data` object.
        edge_label_index (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The edge indices, holding source and destination nodes to start
            sampling from.
            If set to :obj:`None`, all edges will be considered.
            In heterogeneous graphs, needs to be passed as a tuple that holds
            the edge type and corresponding edge indices.
            (default: :obj:`None`)
        edge_label (Tensor, optional): The labels of edge indices from which to
            start sampling from. Must be the same length as
            the :obj:`edge_label_index`. (default: :obj:`None`)
        edge_label_time (Tensor, optional): The timestamps of edge indices from
            which to start sampling from. Must be the same length as
            :obj:`edge_label_index`. If set, temporal sampling will be
            used such that neighbors are guaranteed to fulfill temporal
            constraints, *i.e.*, neighbors have an earlier timestamp than
            the ouput edge. The :obj:`time_attr` needs to be set for this
            to work. (default: :obj:`None`)
        neg_sampling (NegativeSampling, optional): The negative sampling
            configuration.
            For negative sampling mode :obj:`"binary"`, samples can be accessed
            via the attributes :obj:`edge_label_index` and :obj:`edge_label` in
            the respective edge type of the returned mini-batch.
            In case :obj:`edge_label` does not exist, it will be automatically
            created and represents a binary classification task (:obj:`0` =
            negative edge, :obj:`1` = positive edge).
            In case :obj:`edge_label` does exist, it has to be a categorical
            label from :obj:`0` to :obj:`num_classes - 1`.
            After negative sampling, label :obj:`0` represents negative edges,
            and labels :obj:`1` to :obj:`num_classes` represent the labels of
            positive edges.
            Note that returned labels are of type :obj:`torch.float` for binary
            classification (to facilitate the ease-of-use of
            :meth:`F.binary_cross_entropy`) and of type
            :obj:`torch.long` for multi-class classification (to facilitate the
            ease-of-use of :meth:`F.cross_entropy`).
            For negative sampling mode :obj:`"triplet"`, samples can be
            accessed via the attributes :obj:`src_index`, :obj:`dst_pos_index`
            and :obj:`dst_neg_index` in the respective node types of the
            returned mini-batch.
            :obj:`edge_label` needs to be :obj:`None` for :obj:`"triplet"`
            negative sampling mode.
            If set to :obj:`None`, no negative sampling strategy is applied.
            (default: :obj:`None`)
        neg_sampling_ratio (int or float, optional): The ratio of sampled
            negative edges to the number of positive edges.
            Deprecated in favor of the :obj:`neg_sampling` argument.
            (default: :obj:`None`).
        transform (callable, optional): A function/transform that takes in
            a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        transform_sampler_output (callable, optional): A function/transform
            that takes in a :class:`torch_geometric.sampler.SamplerOutput` and
            returns a transformed version. (default: :obj:`None`)
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        custom_cls (HeteroData, optional): A custom
            :class:`~torch_geometric.data.HeteroData` class to return for
            mini-batches in case of remote backends. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        link_sampler: BaseSampler,
        edge_label_index: InputEdges = None,
        edge_label: OptTensor = None,
        edge_label_time: OptTensor = None,
        neg_sampling: Optional[NegativeSampling] = None,
        neg_sampling_ratio: Optional[Union[int, float]] = None,
        transform: Optional[Callable] = None,
        transform_sampler_output: Optional[Callable] = None,
        filter_per_worker: Optional[bool] = None,
        custom_cls: Optional[HeteroData] = None,
        input_id: OptTensor = None,
        **kwargs,
    ):
        if filter_per_worker is None:
            filter_per_worker = infer_filter_per_worker(data)

        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        if neg_sampling_ratio is not None and neg_sampling_ratio != 0.0:
            # TODO: Deprecation warning.
            neg_sampling = NegativeSampling("binary", neg_sampling_ratio)

        # Get edge type (or `None` for homogeneous graphs):
        input_type, edge_label_index = get_edge_label_index(
            data, edge_label_index)

        self.data = data
        self.link_sampler = link_sampler
        self.neg_sampling = NegativeSampling.cast(neg_sampling)
        self.transform = transform
        self.transform_sampler_output = transform_sampler_output
        self.filter_per_worker = filter_per_worker
        self.custom_cls = custom_cls

        if (self.neg_sampling is not None and self.neg_sampling.is_binary()
                and edge_label is not None and edge_label.min() == 0):
            # Increment labels such that `zero` now denotes "negative".
            edge_label = edge_label + 1

        if (self.neg_sampling is not None and self.neg_sampling.is_triplet()
                and edge_label is not None):
            raise ValueError("'edge_label' needs to be undefined for "
                             "'triplet'-based negative sampling. Please use "
                             "`src_index`, `dst_pos_index` and "
                             "`neg_pos_index` of the returned mini-batch "
                             "instead to differentiate between positive and "
                             "negative samples.")

        self.input_data = EdgeSamplerInput(
            input_id=input_id,
            row=edge_label_index[0],
            col=edge_label_index[1],
            label=edge_label,
            time=edge_label_time,
            input_type=input_type,
        )

        iterator = range(edge_label_index.size(1))
        super().__init__(iterator, collate_fn=self.collate_fn, **kwargs)

    def __call__(
        self,
        index: Union[Tensor, List[int]],
    ) -> Union[Data, HeteroData]:
        r"""Samples a subgraph from a batch of input edges."""
        out = self.collate_fn(index)
        if not self.filter_per_worker:
            out = self.filter_fn(out)
        return out

    def collate_fn(self, index: Union[Tensor, List[int]]) -> Any:
        r"""Samples a subgraph from a batch of input edges."""
        input_data: EdgeSamplerInput = self.input_data[index]

        out = self.link_sampler.sample_from_edges(
            input_data, neg_sampling=self.neg_sampling)

        if self.filter_per_worker:  # Execute `filter_fn` in the worker process
            out = self.filter_fn(out)

        return out

    def filter_fn(
        self,
        out: Union[SamplerOutput, HeteroSamplerOutput],
    ) -> Union[Data, HeteroData]:
        r"""Joins the sampled nodes with their corresponding features,
        returning the resulting :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` object to be used downstream.
        """
        if self.transform_sampler_output:
            out = self.transform_sampler_output(out)

        if isinstance(out, SamplerOutput):
            data = filter_data(self.data, out.node, out.row, out.col, out.edge,
                               self.link_sampler.edge_permutation)

            if 'n_id' not in data:
                data.n_id = out.node
            if out.edge is not None and 'e_id' not in data:
                data.e_id = out.edge

            data.batch = out.batch
            data.num_sampled_nodes = out.num_sampled_nodes
            data.num_sampled_edges = out.num_sampled_edges

            data.input_id = out.metadata[0]

            if self.neg_sampling is None or self.neg_sampling.is_binary():
                data.edge_label_index = out.metadata[1]
                data.edge_label = out.metadata[2]
                data.edge_label_time = out.metadata[3]
            elif self.neg_sampling.is_triplet():
                data.src_index = out.metadata[1]
                data.dst_pos_index = out.metadata[2]
                data.dst_neg_index = out.metadata[3]
                data.seed_time = out.metadata[4]
                # Sanity removals in case `edge_label_index` and
                # `edge_label_time` are attributes of the base `data` object:
                del data.edge_label_index  # Sanity removals.
                del data.edge_label_time

        elif isinstance(out, HeteroSamplerOutput):
            if isinstance(self.data, HeteroData):
                data = filter_hetero_data(self.data, out.node, out.row,
                                          out.col, out.edge,
                                          self.link_sampler.edge_permutation)
            else:  # Tuple[FeatureStore, GraphStore]
                data = filter_custom_store(*self.data, out.node, out.row,
                                           out.col, out.edge, self.custom_cls)

            for key, node in out.node.items():
                if 'n_id' not in data[key]:
                    data[key].n_id = node

            for key, edge in (out.edge or {}).items():
                if 'e_id' not in data[key]:
                    data[key].e_id = edge

            data.set_value_dict('batch', out.batch)
            data.set_value_dict('num_sampled_nodes', out.num_sampled_nodes)
            data.set_value_dict('num_sampled_edges', out.num_sampled_edges)

            input_type = self.input_data.input_type
            data[input_type].input_id = out.metadata[0]

            if self.neg_sampling is None or self.neg_sampling.is_binary():
                data[input_type].edge_label_index = out.metadata[1]
                data[input_type].edge_label = out.metadata[2]
                data[input_type].edge_label_time = out.metadata[3]
            elif self.neg_sampling.is_triplet():
                data[input_type[0]].src_index = out.metadata[1]
                data[input_type[-1]].dst_pos_index = out.metadata[2]
                data[input_type[-1]].dst_neg_index = out.metadata[3]
                data[input_type[0]].seed_time = out.metadata[4]
                data[input_type[-1]].seed_time = out.metadata[4]
                # Sanity removals in case `edge_label_index` and
                # `edge_label_time` are attributes of the base `data` object:
                if input_type in data.edge_types:
                    del data[input_type].edge_label_index
                    del data[input_type].edge_label_time

        else:
            raise TypeError(f"'{self.__class__.__name__}'' found invalid "
                            f"type: '{type(out)}'")

        return data if self.transform is None else self.transform(data)

    def _get_iterator(self) -> Iterator:
        if self.filter_per_worker:
            return super()._get_iterator()

        # Execute `filter_fn` in the main process:
        return DataLoaderIterator(super()._get_iterator(), self.filter_fn)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/loader/link_neighbor_loader.py000066400000000000000000000304531444262304300237120ustar00rootroot00000000000000from typing import Callable, Dict, List, Optional, Tuple, Union

from torch_geometric.data import Data, FeatureStore, GraphStore, HeteroData
from torch_geometric.loader.link_loader import LinkLoader
from torch_geometric.sampler import NegativeSampling, NeighborSampler
from torch_geometric.sampler.base import SubgraphType
from torch_geometric.typing import EdgeType, InputEdges, OptTensor


class LinkNeighborLoader(LinkLoader):
    r"""A link-based data loader derived as an extension of the node-based
    :class:`torch_geometric.loader.NeighborLoader`.
    This loader allows for mini-batch training of GNNs on large-scale graphs
    where full-batch training is not feasible.

    More specifically, this loader first selects a sample of edges from the
    set of input edges :obj:`edge_label_index` (which may or not be edges in
    the original graph) and then constructs a subgraph from all the nodes
    present in this list by sampling :obj:`num_neighbors` neighbors in each
    iteration.

    .. code-block:: python

        from torch_geometric.datasets import Planetoid
        from torch_geometric.loader import LinkNeighborLoader

        data = Planetoid(path, name='Cora')[0]

        loader = LinkNeighborLoader(
            data,
            # Sample 30 neighbors for each node for 2 iterations
            num_neighbors=[30] * 2,
            # Use a batch size of 128 for sampling training nodes
            batch_size=128,
            edge_label_index=data.edge_index,
        )

        sampled_data = next(iter(loader))
        print(sampled_data)
        >>> Data(x=[1368, 1433], edge_index=[2, 3103], y=[1368],
                 train_mask=[1368], val_mask=[1368], test_mask=[1368],
                 edge_label_index=[2, 128])

    It is additionally possible to provide edge labels for sampled edges, which
    are then added to the batch:

    .. code-block:: python

        loader = LinkNeighborLoader(
            data,
            num_neighbors=[30] * 2,
            batch_size=128,
            edge_label_index=data.edge_index,
            edge_label=torch.ones(data.edge_index.size(1))
        )

        sampled_data = next(iter(loader))
        print(sampled_data)
        >>> Data(x=[1368, 1433], edge_index=[2, 3103], y=[1368],
                 train_mask=[1368], val_mask=[1368], test_mask=[1368],
                 edge_label_index=[2, 128], edge_label=[128])

    The rest of the functionality mirrors that of
    :class:`~torch_geometric.loader.NeighborLoader`, including support for
    heterogeneous graphs.

    .. note::
        Negative sampling is currently implemented in an approximate
        way, *i.e.* negative edges may contain false negatives.

    Args:
        data (Any): A :class:`~torch_geometric.data.Data`,
            :class:`~torch_geometric.data.HeteroData`, or
            (:class:`~torch_geometric.data.FeatureStore`,
            :class:`~torch_geometric.data.GraphStore`) data object.
        num_neighbors (List[int] or Dict[Tuple[str, str, str], List[int]]): The
            number of neighbors to sample for each node in each iteration.
            If an entry is set to :obj:`-1`, all neighbors will be included.
            In heterogeneous graphs, may also take in a dictionary denoting
            the amount of neighbors to sample for each individual edge type.
        edge_label_index (Tensor or EdgeType or Tuple[EdgeType, Tensor]):
            The edge indices for which neighbors are sampled to create
            mini-batches.
            If set to :obj:`None`, all edges will be considered.
            In heterogeneous graphs, needs to be passed as a tuple that holds
            the edge type and corresponding edge indices.
            (default: :obj:`None`)
        edge_label (Tensor, optional): The labels of edge indices for
            which neighbors are sampled. Must be the same length as
            the :obj:`edge_label_index`. If set to :obj:`None` its set to
            `torch.zeros(...)` internally. (default: :obj:`None`)
        edge_label_time (Tensor, optional): The timestamps for edge indices
            for which neighbors are sampled. Must be the same length as
            :obj:`edge_label_index`. If set, temporal sampling will be
            used such that neighbors are guaranteed to fulfill temporal
            constraints, *i.e.*, neighbors have an earlier timestamp than
            the ouput edge. The :obj:`time_attr` needs to be set for this
            to work. (default: :obj:`None`)
        replace (bool, optional): If set to :obj:`True`, will sample with
            replacement. (default: :obj:`False`)
        subgraph_type (SubgraphType or str, optional): The type of the returned
            subgraph.
            If set to :obj:`"directional"`, the returned subgraph only holds
            the sampled (directed) edges which are necessary to compute
            representations for the sampled seed nodes.
            If set to :obj:`"bidirectional"`, sampled edges are converted to
            bidirectional edges.
            If set to :obj:`"induced"`, the returned subgraph contains the
            induced subgraph of all sampled nodes.
            (default: :obj:`"directional"`)
        disjoint (bool, optional): If set to :obj: `True`, each seed node will
            create its own disjoint subgraph.
            If set to :obj:`True`, mini-batch outputs will have a :obj:`batch`
            vector holding the mapping of nodes to their respective subgraph.
            Will get automatically set to :obj:`True` in case of temporal
            sampling. (default: :obj:`False`)
        temporal_strategy (str, optional): The sampling strategy when using
            temporal sampling (:obj:`"uniform"`, :obj:`"last"`).
            If set to :obj:`"uniform"`, will sample uniformly across neighbors
            that fulfill temporal constraints.
            If set to :obj:`"last"`, will sample the last `num_neighbors` that
            fulfill temporal constraints.
            (default: :obj:`"uniform"`)
        neg_sampling (NegativeSampling, optional): The negative sampling
            configuration.
            For negative sampling mode :obj:`"binary"`, samples can be accessed
            via the attributes :obj:`edge_label_index` and :obj:`edge_label` in
            the respective edge type of the returned mini-batch.
            In case :obj:`edge_label` does not exist, it will be automatically
            created and represents a binary classification task (:obj:`0` =
            negative edge, :obj:`1` = positive edge).
            In case :obj:`edge_label` does exist, it has to be a categorical
            label from :obj:`0` to :obj:`num_classes - 1`.
            After negative sampling, label :obj:`0` represents negative edges,
            and labels :obj:`1` to :obj:`num_classes` represent the labels of
            positive edges.
            Note that returned labels are of type :obj:`torch.float` for binary
            classification (to facilitate the ease-of-use of
            :meth:`F.binary_cross_entropy`) and of type
            :obj:`torch.long` for multi-class classification (to facilitate the
            ease-of-use of :meth:`F.cross_entropy`).
            For negative sampling mode :obj:`"triplet"`, samples can be
            accessed via the attributes :obj:`src_index`, :obj:`dst_pos_index`
            and :obj:`dst_neg_index` in the respective node types of the
            returned mini-batch.
            :obj:`edge_label` needs to be :obj:`None` for :obj:`"triplet"`
            negative sampling mode.
            If set to :obj:`None`, no negative sampling strategy is applied.
            (default: :obj:`None`)
        neg_sampling_ratio (int or float, optional): The ratio of sampled
            negative edges to the number of positive edges.
            Deprecated in favor of the :obj:`neg_sampling` argument.
            (default: :obj:`None`)
        time_attr (str, optional): The name of the attribute that denotes
            timestamps for the nodes in the graph.
            If set, temporal sampling will be used such that neighbors are
            guaranteed to fulfill temporal constraints, *i.e.* neighbors have
            an earlier or equal timestamp than the center node.
            Only used if :obj:`edge_label_time` is set. (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in
            a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        transform_sampler_output (callable, optional): A function/transform
            that takes in a :class:`torch_geometric.sampler.SamplerOutput` and
            returns a transformed version. (default: :obj:`None`)
        is_sorted (bool, optional): If set to :obj:`True`, assumes that
            :obj:`edge_index` is sorted by column.
            If :obj:`time_attr` is set, additionally requires that rows are
            sorted according to time within individual neighborhoods.
            This avoids internal re-sorting of the data and can improve
            runtime and memory efficiency. (default: :obj:`False`)
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        num_neighbors: Union[List[int], Dict[EdgeType, List[int]]],
        edge_label_index: InputEdges = None,
        edge_label: OptTensor = None,
        edge_label_time: OptTensor = None,
        replace: bool = False,
        subgraph_type: Union[SubgraphType, str] = 'directional',
        disjoint: bool = False,
        temporal_strategy: str = 'uniform',
        neg_sampling: Optional[NegativeSampling] = None,
        neg_sampling_ratio: Optional[Union[int, float]] = None,
        time_attr: Optional[str] = None,
        transform: Optional[Callable] = None,
        transform_sampler_output: Optional[Callable] = None,
        is_sorted: bool = False,
        filter_per_worker: Optional[bool] = None,
        neighbor_sampler: Optional[NeighborSampler] = None,
        directed: bool = True,  # Deprecated.
        **kwargs,
    ):
        if (edge_label_time is not None) != (time_attr is not None):
            raise ValueError(
                f"Received conflicting 'edge_label_time' and 'time_attr' "
                f"arguments: 'edge_label_time' is "
                f"{'set' if edge_label_time is not None else 'not set'} "
                f"while 'time_attr' is "
                f"{'set' if time_attr is not None else 'not set'}. "
                f"Both arguments must be provided for temporal sampling.")

        if neighbor_sampler is None:
            neighbor_sampler = NeighborSampler(
                data,
                num_neighbors=num_neighbors,
                replace=replace,
                subgraph_type=subgraph_type,
                disjoint=disjoint,
                temporal_strategy=temporal_strategy,
                time_attr=time_attr,
                is_sorted=is_sorted,
                share_memory=kwargs.get('num_workers', 0) > 0,
                directed=directed,
            )

        super().__init__(
            data=data,
            link_sampler=neighbor_sampler,
            edge_label_index=edge_label_index,
            edge_label=edge_label,
            edge_label_time=edge_label_time,
            neg_sampling=neg_sampling,
            neg_sampling_ratio=neg_sampling_ratio,
            transform=transform,
            transform_sampler_output=transform_sampler_output,
            filter_per_worker=filter_per_worker,
            **kwargs,
        )
pytorch/torch_geometric/loader/mixin.py000066400000000000000000000142551444262304300207000ustar00rootroot00000000000000import glob
import logging
import os
from contextlib import contextmanager
from typing import Any, Dict, List, Optional

import psutil
import torch


def get_numa_nodes_cores() -> Dict[str, Any]:
    """ Returns numa nodes info in format:

    ..code-block::

        {: [(, [, 
        ...]), ...], ...}

        # For example:
        {0: [(0, [0, 4]), (1, [1, 5])], 1: [(2, [2, 6]), (3, [3, 7])]}

    If not available, returns an empty dictionary.
    """
    numa_node_paths = glob.glob('/sys/devices/system/node/node[0-9]*')

    if not numa_node_paths:
        return {}

    nodes = {}
    try:
        for node_path in numa_node_paths:
            numa_node_id = int(os.path.basename(node_path)[4:])

            thread_siblings = {}
            for cpu_dir in glob.glob(os.path.join(node_path, 'cpu[0-9]*')):
                cpu_id = int(os.path.basename(cpu_dir)[3:])
                if cpu_id > 0:
                    with open(os.path.join(cpu_dir,
                                           'online')) as core_online_file:
                        core_online = int(
                            core_online_file.read().splitlines()[0])
                else:
                    core_online = 1  # cpu0 is always online (special case)
                if core_online == 1:
                    with open(os.path.join(cpu_dir, 'topology',
                                           'core_id')) as core_id_file:
                        core_id = int(core_id_file.read().strip())
                        if core_id in thread_siblings:
                            thread_siblings[core_id].append(cpu_id)
                        else:
                            thread_siblings[core_id] = [cpu_id]

            nodes[numa_node_id] = sorted([(k, sorted(v))
                                          for k, v in thread_siblings.items()])

    except (OSError, ValueError, IndexError, IOError):
        Warning('Failed to read NUMA info')
        return {}

    return nodes


class WorkerInitWrapper:
    r"""Wraps the :attr:`worker_init_fn` argument for
    :class:`torch.utils.data.DataLoader` workers."""
    def __init__(self, func):
        self.func = func

    def __call__(self, worker_id):
        if self.func is not None:
            self.func(worker_id)


class AffinityMixin:
    r"""A context manager to enable CPU affinity for data loader workers
    (only used when running on CPU devices).

    Affinitization places data loader workers threads on specific CPU cores.
    In effect, it allows for more efficient local memory allocation and reduces
    remote memory calls.
    Every time a process or thread moves from one core to another, registers
    and caches need to be flushed and reloaded.
    This can become very costly if it happens often, and our threads may also
    no longer be close to their data, or be able to share data in a cache.

    See `here `__ for the accompanying tutorial.

    .. warning::

        To correctly affinitize compute threads (*i.e.* with OMP), please make
        sure that you exclude :obj:`loader_cores` from the list of cores
        available for the main process.
        This will cause core oversubsription and exacerbate performance.

    .. code-block:: python

        loader = NeigborLoader(data, num_workers=3)
        with loader.enable_cpu_affinity(loader_cores=[0, 1, 2]):
            for batch in loader:
                pass
    """
    @contextmanager
    def enable_cpu_affinity(self, loader_cores: Optional[List[int]] = None):
        r"""Enables CPU affinity.

        Args:
            loader_cores ([int], optional): List of CPU cores to which data
                loader workers should affinitize to.
                By default, will affinitize to cores starting at :obj:`cpu0`.
                (default: :obj:`node0_cores[:num_workers]`)
        """
        if not self.num_workers > 0:
            raise ValueError(
                f"'enable_cpu_affinity' should be used with at least one "
                f"worker (got {self.num_workers})")

        if loader_cores and len(loader_cores) != self.num_workers:
            raise ValueError(
                f"The number of loader cores (got {len(loader_cores)}) "
                f"in 'enable_cpu_affinity' should match with the number "
                f"of workers (got {self.num_workers})")
        worker_init_fn_old = WorkerInitWrapper(self.worker_init_fn)
        affinity_old = psutil.Process().cpu_affinity()
        nthreads_old = torch.get_num_threads()
        loader_cores = loader_cores[:] if loader_cores else None

        def init_fn(worker_id):
            try:
                psutil.Process().cpu_affinity([loader_cores[worker_id]])
            except IndexError:
                raise ValueError(f"Cannot use CPU affinity for worker ID "
                                 f"{worker_id} on CPU {loader_cores}")

            worker_init_fn_old(worker_id)

        if loader_cores is None:

            numa_info = get_numa_nodes_cores()

            if numa_info and len(numa_info[0]) > self.num_workers:
                # Take one thread per each node 0 core:
                node0_cores = [cpus[0] for core_id, cpus in numa_info[0]]
                node0_cores.sort()
            else:
                node0_cores = list(range(psutil.cpu_count(logical=False)))

            if len(node0_cores) < self.num_workers:
                raise ValueError(
                    f"More workers (got {self.num_workers}) than available "
                    f"cores (got {len(node0_cores)})")

            # Set default loader core IDs:
            loader_cores = node0_cores[:self.num_workers]

        try:
            # Set CPU affinity for dataloader:
            self.worker_init_fn = init_fn
            logging.debug(f"{self.num_workers} data loader workers are "
                          f"assigned to CPUs {loader_cores}")
            yield
        finally:
            # Restore omp_num_threads and cpu affinity:
            psutil.Process().cpu_affinity(affinity_old)
            torch.set_num_threads(nthreads_old)
            self.worker_init_fn = worker_init_fn_old
            self.cpu_affinity_enabled = False
pytorch/torch_geometric/loader/neighbor_loader.py000066400000000000000000000261661444262304300227030ustar00rootroot00000000000000from typing import Callable, Dict, List, Optional, Tuple, Union

from torch_geometric.data import Data, FeatureStore, GraphStore, HeteroData
from torch_geometric.loader.node_loader import NodeLoader
from torch_geometric.sampler import NeighborSampler
from torch_geometric.sampler.base import SubgraphType
from torch_geometric.typing import EdgeType, InputNodes, OptTensor


class NeighborLoader(NodeLoader):
    r"""A data loader that performs neighbor sampling as introduced in the
    `"Inductive Representation Learning on Large Graphs"
    `_ paper.
    This loader allows for mini-batch training of GNNs on large-scale graphs
    where full-batch training is not feasible.

    More specifically, :obj:`num_neighbors` denotes how much neighbors are
    sampled for each node in each iteration.
    :class:`~torch_geometric.loader.NeighborLoader` takes in this list of
    :obj:`num_neighbors` and iteratively samples :obj:`num_neighbors[i]` for
    each node involved in iteration :obj:`i - 1`.

    Sampled nodes are sorted based on the order in which they were sampled.
    In particular, the first :obj:`batch_size` nodes represent the set of
    original mini-batch nodes.

    .. code-block:: python

        from torch_geometric.datasets import Planetoid
        from torch_geometric.loader import NeighborLoader

        data = Planetoid(path, name='Cora')[0]

        loader = NeighborLoader(
            data,
            # Sample 30 neighbors for each node for 2 iterations
            num_neighbors=[30] * 2,
            # Use a batch size of 128 for sampling training nodes
            batch_size=128,
            input_nodes=data.train_mask,
        )

        sampled_data = next(iter(loader))
        print(sampled_data.batch_size)
        >>> 128

    By default, the data loader will only include the edges that were
    originally sampled (:obj:`directed = True`).
    This option should only be used in case the number of hops is equivalent to
    the number of GNN layers.
    In case the number of GNN layers is greater than the number of hops,
    consider setting :obj:`directed = False`, which will include all edges
    between all sampled nodes (but is slightly slower as a result).

    Furthermore, :class:`~torch_geometric.loader.NeighborLoader` works for both
    **homogeneous** graphs stored via :class:`~torch_geometric.data.Data` as
    well as **heterogeneous** graphs stored via
    :class:`~torch_geometric.data.HeteroData`.
    When operating in heterogeneous graphs, up to :obj:`num_neighbors`
    neighbors will be sampled for each :obj:`edge_type`.
    However, more fine-grained control over
    the amount of sampled neighbors of individual edge types is possible:

    .. code-block:: python

        from torch_geometric.datasets import OGB_MAG
        from torch_geometric.loader import NeighborLoader

        hetero_data = OGB_MAG(path)[0]

        loader = NeighborLoader(
            hetero_data,
            # Sample 30 neighbors for each node and edge type for 2 iterations
            num_neighbors={key: [30] * 2 for key in hetero_data.edge_types},
            # Use a batch size of 128 for sampling training nodes of type paper
            batch_size=128,
            input_nodes=('paper', hetero_data['paper'].train_mask),
        )

        sampled_hetero_data = next(iter(loader))
        print(sampled_hetero_data['paper'].batch_size)
        >>> 128

    .. note::

        For an example of using
        :class:`~torch_geometric.loader.NeighborLoader`, see
        `examples/hetero/to_hetero_mag.py `_.

    The :class:`~torch_geometric.loader.NeighborLoader` will return subgraphs
    where global node indices are mapped to local indices corresponding to this
    specific subgraph. However, often times it is desired to map the nodes of
    the current subgraph back to the global node indices. The
    :class:`~torch_geometric.loader.NeighborLoader` will include this mapping
    as part of the :obj:`data` object:

    .. code-block:: python

        loader = NeighborLoader(data, ...)
        sampled_data = next(iter(loader))
        print(sampled_data.n_id)  # Global node index of each node in batch.

    Args:
        data (Any): A :class:`~torch_geometric.data.Data`,
            :class:`~torch_geometric.data.HeteroData`, or
            (:class:`~torch_geometric.data.FeatureStore`,
            :class:`~torch_geometric.data.GraphStore`) data object.
        num_neighbors (List[int] or Dict[Tuple[str, str, str], List[int]]): The
            number of neighbors to sample for each node in each iteration.
            If an entry is set to :obj:`-1`, all neighbors will be included.
            In heterogeneous graphs, may also take in a dictionary denoting
            the amount of neighbors to sample for each individual edge type.
        input_nodes (torch.Tensor or str or Tuple[str, torch.Tensor]): The
            indices of nodes for which neighbors are sampled to create
            mini-batches.
            Needs to be either given as a :obj:`torch.LongTensor` or
            :obj:`torch.BoolTensor`.
            If set to :obj:`None`, all nodes will be considered.
            In heterogeneous graphs, needs to be passed as a tuple that holds
            the node type and node indices. (default: :obj:`None`)
        input_time (torch.Tensor, optional): Optional values to override the
            timestamp for the input nodes given in :obj:`input_nodes`. If not
            set, will use the timestamps in :obj:`time_attr` as default (if
            present). The :obj:`time_attr` needs to be set for this to work.
            (default: :obj:`None`)
        replace (bool, optional): If set to :obj:`True`, will sample with
            replacement. (default: :obj:`False`)
        subgraph_type (SubgraphType or str, optional): The type of the returned
            subgraph.
            If set to :obj:`"directional"`, the returned subgraph only holds
            the sampled (directed) edges which are necessary to compute
            representations for the sampled seed nodes.
            If set to :obj:`"bidirectional"`, sampled edges are converted to
            bidirectional edges.
            If set to :obj:`"induced"`, the returned subgraph contains the
            induced subgraph of all sampled nodes.
            (default: :obj:`"directional"`)
        disjoint (bool, optional): If set to :obj: `True`, each seed node will
            create its own disjoint subgraph.
            If set to :obj:`True`, mini-batch outputs will have a :obj:`batch`
            vector holding the mapping of nodes to their respective subgraph.
            Will get automatically set to :obj:`True` in case of temporal
            sampling. (default: :obj:`False`)
        temporal_strategy (str, optional): The sampling strategy when using
            temporal sampling (:obj:`"uniform"`, :obj:`"last"`).
            If set to :obj:`"uniform"`, will sample uniformly across neighbors
            that fulfill temporal constraints.
            If set to :obj:`"last"`, will sample the last `num_neighbors` that
            fulfill temporal constraints.
            (default: :obj:`"uniform"`)
        time_attr (str, optional): The name of the attribute that denotes
            timestamps for the nodes in the graph.
            If set, temporal sampling will be used such that neighbors are
            guaranteed to fulfill temporal constraints, *i.e.* neighbors have
            an earlier or equal timestamp than the center node.
            (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in
            a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        transform_sampler_output (callable, optional): A function/transform
            that takes in a :class:`torch_geometric.sampler.SamplerOutput` and
            returns a transformed version. (default: :obj:`None`)
        is_sorted (bool, optional): If set to :obj:`True`, assumes that
            :obj:`edge_index` is sorted by column.
            If :obj:`time_attr` is set, additionally requires that rows are
            sorted according to time within individual neighborhoods.
            This avoids internal re-sorting of the data and can improve
            runtime and memory efficiency. (default: :obj:`False`)
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        num_neighbors: Union[List[int], Dict[EdgeType, List[int]]],
        input_nodes: InputNodes = None,
        input_time: OptTensor = None,
        replace: bool = False,
        subgraph_type: Union[SubgraphType, str] = 'directional',
        disjoint: bool = False,
        temporal_strategy: str = 'uniform',
        time_attr: Optional[str] = None,
        transform: Optional[Callable] = None,
        transform_sampler_output: Optional[Callable] = None,
        is_sorted: bool = False,
        filter_per_worker: Optional[bool] = None,
        neighbor_sampler: Optional[NeighborSampler] = None,
        directed: bool = True,  # Deprecated.
        **kwargs,
    ):
        if input_time is not None and time_attr is None:
            raise ValueError("Received conflicting 'input_time' and "
                             "'time_attr' arguments: 'input_time' is set "
                             "while 'time_attr' is not set.")

        if neighbor_sampler is None:
            neighbor_sampler = NeighborSampler(
                data,
                num_neighbors=num_neighbors,
                replace=replace,
                subgraph_type=subgraph_type,
                disjoint=disjoint,
                temporal_strategy=temporal_strategy,
                time_attr=time_attr,
                is_sorted=is_sorted,
                share_memory=kwargs.get('num_workers', 0) > 0,
                directed=directed,
            )

        super().__init__(
            data=data,
            node_sampler=neighbor_sampler,
            input_nodes=input_nodes,
            input_time=input_time,
            transform=transform,
            transform_sampler_output=transform_sampler_output,
            filter_per_worker=filter_per_worker,
            **kwargs,
        )
pytorch/torch_geometric/loader/neighbor_sampler.py000066400000000000000000000205011444262304300230630ustar00rootroot00000000000000from typing import Callable, List, NamedTuple, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import SparseTensor


class EdgeIndex(NamedTuple):
    edge_index: Tensor
    e_id: Optional[Tensor]
    size: Tuple[int, int]

    def to(self, *args, **kwargs):
        edge_index = self.edge_index.to(*args, **kwargs)
        e_id = self.e_id.to(*args, **kwargs) if self.e_id is not None else None
        return EdgeIndex(edge_index, e_id, self.size)


class Adj(NamedTuple):
    adj_t: SparseTensor
    e_id: Optional[Tensor]
    size: Tuple[int, int]

    def to(self, *args, **kwargs):
        adj_t = self.adj_t.to(*args, **kwargs)
        e_id = self.e_id.to(*args, **kwargs) if self.e_id is not None else None
        return Adj(adj_t, e_id, self.size)


class NeighborSampler(torch.utils.data.DataLoader):
    r"""The neighbor sampler from the `"Inductive Representation Learning on
    Large Graphs" `_ paper, which allows
    for mini-batch training of GNNs on large-scale graphs where full-batch
    training is not feasible.

    Given a GNN with :math:`L` layers and a specific mini-batch of nodes
    :obj:`node_idx` for which we want to compute embeddings, this module
    iteratively samples neighbors and constructs bipartite graphs that simulate
    the actual computation flow of GNNs.

    More specifically, :obj:`sizes` denotes how much neighbors we want to
    sample for each node in each layer.
    This module then takes in these :obj:`sizes` and iteratively samples
    :obj:`sizes[l]` for each node involved in layer :obj:`l`.
    In the next layer, sampling is repeated for the union of nodes that were
    already encountered.
    The actual computation graphs are then returned in reverse-mode, meaning
    that we pass messages from a larger set of nodes to a smaller one, until we
    reach the nodes for which we originally wanted to compute embeddings.

    Hence, an item returned by :class:`NeighborSampler` holds the current
    :obj:`batch_size`, the IDs :obj:`n_id` of all nodes involved in the
    computation, and a list of bipartite graph objects via the tuple
    :obj:`(edge_index, e_id, size)`, where :obj:`edge_index` represents the
    bipartite edges between source and target nodes, :obj:`e_id` denotes the
    IDs of original edges in the full graph, and :obj:`size` holds the shape
    of the bipartite graph.
    For each bipartite graph, target nodes are also included at the beginning
    of the list of source nodes so that one can easily apply skip-connections
    or add self-loops.

    .. warning::

        :class:`~torch_geometric.loader.NeighborSampler` is deprecated and will
        be removed in a future release.
        Use :class:`torch_geometric.loader.NeighborLoader` instead.

    .. note::

        For an example of using :obj:`NeighborSampler`, see
        `examples/reddit.py
        `_ or
        `examples/ogbn_products_sage.py
        `_.

    Args:
        edge_index (Tensor or SparseTensor): A :obj:`torch.LongTensor` or a
            :class:`torch_sparse.SparseTensor` that defines the underlying
            graph connectivity/message passing flow.
            :obj:`edge_index` holds the indices of a (sparse) symmetric
            adjacency matrix.
            If :obj:`edge_index` is of type :obj:`torch.LongTensor`, its shape
            must be defined as :obj:`[2, num_edges]`, where messages from nodes
            :obj:`edge_index[0]` are sent to nodes in :obj:`edge_index[1]`
            (in case :obj:`flow="source_to_target"`).
            If :obj:`edge_index` is of type :class:`torch_sparse.SparseTensor`,
            its sparse indices :obj:`(row, col)` should relate to
            :obj:`row = edge_index[1]` and :obj:`col = edge_index[0]`.
            The major difference between both formats is that we need to input
            the *transposed* sparse adjacency matrix.
        sizes ([int]): The number of neighbors to sample for each node in each
            layer. If set to :obj:`sizes[l] = -1`, all neighbors are included
            in layer :obj:`l`.
        node_idx (LongTensor, optional): The nodes that should be considered
            for creating mini-batches. If set to :obj:`None`, all nodes will be
            considered.
        num_nodes (int, optional): The number of nodes in the graph.
            (default: :obj:`None`)
        return_e_id (bool, optional): If set to :obj:`False`, will not return
            original edge indices of sampled edges. This is only useful in case
            when operating on graphs without edge features to save memory.
            (default: :obj:`True`)
        transform (callable, optional): A function/transform that takes in
            a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(self, edge_index: Union[Tensor, SparseTensor],
                 sizes: List[int], node_idx: Optional[Tensor] = None,
                 num_nodes: Optional[int] = None, return_e_id: bool = True,
                 transform: Callable = None, **kwargs):

        edge_index = edge_index.to('cpu')

        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        # Save for Pytorch Lightning < 1.6:
        self.edge_index = edge_index
        self.node_idx = node_idx
        self.num_nodes = num_nodes

        self.sizes = sizes
        self.return_e_id = return_e_id
        self.transform = transform
        self.is_sparse_tensor = isinstance(edge_index, SparseTensor)
        self.__val__ = None

        # Obtain a *transposed* `SparseTensor` instance.
        if not self.is_sparse_tensor:
            if (num_nodes is None and node_idx is not None
                    and node_idx.dtype == torch.bool):
                num_nodes = node_idx.size(0)
            if (num_nodes is None and node_idx is not None
                    and node_idx.dtype == torch.long):
                num_nodes = max(int(edge_index.max()), int(node_idx.max())) + 1
            if num_nodes is None:
                num_nodes = int(edge_index.max()) + 1

            value = torch.arange(edge_index.size(1)) if return_e_id else None
            self.adj_t = SparseTensor(row=edge_index[0], col=edge_index[1],
                                      value=value,
                                      sparse_sizes=(num_nodes, num_nodes)).t()
        else:
            adj_t = edge_index
            if return_e_id:
                self.__val__ = adj_t.storage.value()
                value = torch.arange(adj_t.nnz())
                adj_t = adj_t.set_value(value, layout='coo')
            self.adj_t = adj_t

        self.adj_t.storage.rowptr()

        if node_idx is None:
            node_idx = torch.arange(self.adj_t.sparse_size(0))
        elif node_idx.dtype == torch.bool:
            node_idx = node_idx.nonzero(as_tuple=False).view(-1)

        super().__init__(
            node_idx.view(-1).tolist(), collate_fn=self.sample, **kwargs)

    def sample(self, batch):
        if not isinstance(batch, Tensor):
            batch = torch.tensor(batch)

        batch_size: int = len(batch)

        adjs = []
        n_id = batch
        for size in self.sizes:
            adj_t, n_id = self.adj_t.sample_adj(n_id, size, replace=False)
            e_id = adj_t.storage.value()
            size = adj_t.sparse_sizes()[::-1]
            if self.__val__ is not None:
                adj_t.set_value_(self.__val__[e_id], layout='coo')

            if self.is_sparse_tensor:
                adjs.append(Adj(adj_t, e_id, size))
            else:
                row, col, _ = adj_t.coo()
                edge_index = torch.stack([col, row], dim=0)
                adjs.append(EdgeIndex(edge_index, e_id, size))

        adjs = adjs[0] if len(adjs) == 1 else adjs[::-1]
        out = (batch_size, n_id, adjs)
        out = self.transform(*out) if self.transform is not None else out
        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(sizes={self.sizes})'
pytorch/torch_geometric/loader/node_loader.py000066400000000000000000000222251444262304300220230ustar00rootroot00000000000000from typing import Any, Callable, Iterator, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, FeatureStore, GraphStore, HeteroData
from torch_geometric.loader.base import DataLoaderIterator
from torch_geometric.loader.mixin import AffinityMixin
from torch_geometric.loader.utils import (
    filter_custom_store,
    filter_data,
    filter_hetero_data,
    get_input_nodes,
    infer_filter_per_worker,
)
from torch_geometric.sampler import (
    BaseSampler,
    HeteroSamplerOutput,
    NodeSamplerInput,
    SamplerOutput,
)
from torch_geometric.typing import InputNodes, OptTensor


class NodeLoader(torch.utils.data.DataLoader, AffinityMixin):
    r"""A data loader that performs mini-batch sampling from node information,
    using a generic :class:`~torch_geometric.sampler.BaseSampler`
    implementation that defines a
    :meth:`~torch_geometric.sampler.BaseSampler.sample_from_nodes` function and
    is supported on the provided input :obj:`data` object.

    Args:
        data (Any): A :class:`~torch_geometric.data.Data`,
            :class:`~torch_geometric.data.HeteroData`, or
            (:class:`~torch_geometric.data.FeatureStore`,
            :class:`~torch_geometric.data.GraphStore`) data object.
        node_sampler (torch_geometric.sampler.BaseSampler): The sampler
            implementation to be used with this loader.
            Needs to implement
            :meth:`~torch_geometric.sampler.BaseSampler.sample_from_nodes`.
            The sampler implementation must be compatible with the input
            :obj:`data` object.
        input_nodes (torch.Tensor or str or Tuple[str, torch.Tensor]): The
            indices of seed nodes to start sampling from.
            Needs to be either given as a :obj:`torch.LongTensor` or
            :obj:`torch.BoolTensor`.
            If set to :obj:`None`, all nodes will be considered.
            In heterogeneous graphs, needs to be passed as a tuple that holds
            the node type and node indices. (default: :obj:`None`)
        input_time (torch.Tensor, optional): Optional values to override the
            timestamp for the input nodes given in :obj:`input_nodes`. If not
            set, will use the timestamps in :obj:`time_attr` as default (if
            present). The :obj:`time_attr` needs to be set for this to work.
            (default: :obj:`None`)
        transform (callable, optional): A function/transform that takes in
            a sampled mini-batch and returns a transformed version.
            (default: :obj:`None`)
        transform_sampler_output (callable, optional): A function/transform
            that takes in a :class:`torch_geometric.sampler.SamplerOutput` and
            returns a transformed version. (default: :obj:`None`)
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        custom_cls (HeteroData, optional): A custom
            :class:`~torch_geometric.data.HeteroData` class to return for
            mini-batches in case of remote backends. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        node_sampler: BaseSampler,
        input_nodes: InputNodes = None,
        input_time: OptTensor = None,
        transform: Optional[Callable] = None,
        transform_sampler_output: Optional[Callable] = None,
        filter_per_worker: Optional[bool] = None,
        custom_cls: Optional[HeteroData] = None,
        input_id: OptTensor = None,
        **kwargs,
    ):
        if filter_per_worker is None:
            filter_per_worker = infer_filter_per_worker(data)

        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        # Get node type (or `None` for homogeneous graphs):
        input_type, input_nodes = get_input_nodes(data, input_nodes)

        self.data = data
        self.node_sampler = node_sampler
        self.transform = transform
        self.transform_sampler_output = transform_sampler_output
        self.filter_per_worker = filter_per_worker
        self.custom_cls = custom_cls

        self.input_data = NodeSamplerInput(
            input_id=input_id,
            node=input_nodes,
            time=input_time,
            input_type=input_type,
        )

        iterator = range(input_nodes.size(0))
        super().__init__(iterator, collate_fn=self.collate_fn, **kwargs)

    def __call__(
        self,
        index: Union[Tensor, List[int]],
    ) -> Union[Data, HeteroData]:
        r"""Samples a subgraph from a batch of input nodes."""
        out = self.collate_fn(index)
        if not self.filter_per_worker:
            out = self.filter_fn(out)
        return out

    def collate_fn(self, index: Union[Tensor, List[int]]) -> Any:
        r"""Samples a subgraph from a batch of input nodes."""
        input_data: NodeSamplerInput = self.input_data[index]

        out = self.node_sampler.sample_from_nodes(input_data)

        if self.filter_per_worker:  # Execute `filter_fn` in the worker process
            out = self.filter_fn(out)

        return out

    def filter_fn(
        self,
        out: Union[SamplerOutput, HeteroSamplerOutput],
    ) -> Union[Data, HeteroData]:
        r"""Joins the sampled nodes with their corresponding features,
        returning the resulting :class:`~torch_geometric.data.Data` or
        :class:`~torch_geometric.data.HeteroData` object to be used downstream.
        """
        if self.transform_sampler_output:
            out = self.transform_sampler_output(out)

        if isinstance(out, SamplerOutput):
            data = filter_data(self.data, out.node, out.row, out.col, out.edge,
                               self.node_sampler.edge_permutation)

            if 'n_id' not in data:
                data.n_id = out.node
            if out.edge is not None and 'e_id' not in data:
                data.e_id = out.edge

            data.batch = out.batch
            data.num_sampled_nodes = out.num_sampled_nodes
            data.num_sampled_edges = out.num_sampled_edges

            data.input_id = out.metadata[0]
            data.seed_time = out.metadata[1]
            data.batch_size = out.metadata[0].size(0)

        elif isinstance(out, HeteroSamplerOutput):
            if isinstance(self.data, HeteroData):
                data = filter_hetero_data(self.data, out.node, out.row,
                                          out.col, out.edge,
                                          self.node_sampler.edge_permutation)
            else:  # Tuple[FeatureStore, GraphStore]
                data = filter_custom_store(*self.data, out.node, out.row,
                                           out.col, out.edge, self.custom_cls)

            for key, node in out.node.items():
                if 'n_id' not in data[key]:
                    data[key].n_id = node

            for key, edge in (out.edge or {}).items():
                if 'e_id' not in data[key]:
                    data[key].e_id = edge

            data.set_value_dict('batch', out.batch)
            data.set_value_dict('num_sampled_nodes', out.num_sampled_nodes)
            data.set_value_dict('num_sampled_edges', out.num_sampled_edges)

            input_type = self.input_data.input_type
            data[input_type].input_id = out.metadata[0]
            data[input_type].seed_time = out.metadata[1]
            data[input_type].batch_size = out.metadata[0].size(0)

        else:
            raise TypeError(f"'{self.__class__.__name__}'' found invalid "
                            f"type: '{type(out)}'")

        return data if self.transform is None else self.transform(data)

    def _get_iterator(self) -> Iterator:
        if self.filter_per_worker:
            return super()._get_iterator()

        # if not self.is_cuda_available and not self.cpu_affinity_enabled:
        # TODO: Add manual page for best CPU practices
        # link = ...
        # Warning('Dataloader CPU affinity opt is not enabled, consider '
        #          'switching it on with enable_cpu_affinity() or see CPU '
        #          f'best practices for PyG [{link}])')

        # Execute `filter_fn` in the main process:
        return DataLoaderIterator(super()._get_iterator(), self.filter_fn)

    def __enter__(self):
        return self

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/loader/prefetch.py000066400000000000000000000041351444262304300213500ustar00rootroot00000000000000from contextlib import nullcontext
from functools import partial
from typing import Any, Optional

import torch
from torch.utils.data import DataLoader


class PrefetchLoader:
    r"""A GPU prefetcher class for asynchronously transferring data of a
    :class:`torch.utils.data.DataLoader` from host memory to device memory.

    Args:
        loader (torch.utils.data.DataLoader): The data loader.
        device (torch.device, optional): The device to load the data to.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        loader: DataLoader,
        device: Optional[torch.device] = None,
    ):
        if device is None:
            device = 'cuda' if torch.cuda.is_available() else 'cpu'

        self.loader = loader
        self.device = torch.device(device)

        self.is_cuda = torch.cuda.is_available() and self.device.type == 'cuda'

    def non_blocking_transfer(self, batch: Any) -> Any:
        if not self.is_cuda:
            return batch
        if isinstance(batch, (list, tuple)):
            return [self.non_blocking_transfer(v) for v in batch]
        if isinstance(batch, dict):
            return {k: self.non_blocking_transfer(v) for k, v in batch.items()}

        batch = batch.pin_memory()
        return batch.to(self.device, non_blocking=True)

    def __iter__(self) -> Any:
        first = True
        if self.is_cuda:
            stream = torch.cuda.Stream()
            stream_context = partial(torch.cuda.stream, stream=stream)
        else:
            stream = None
            stream_context = nullcontext

        for next_batch in self.loader:

            with stream_context():
                next_batch = self.non_blocking_transfer(next_batch)

            if not first:
                yield batch  # noqa
            else:
                first = False

            if stream is not None:
                torch.cuda.current_stream().wait_stream(stream)

            batch = next_batch

        yield batch

    def __len__(self) -> int:
        return len(self.loader)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.loader})'
pytorch/torch_geometric/loader/random_node_loader.py000066400000000000000000000042241444262304300233620ustar00rootroot00000000000000import math
from typing import Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.hetero_data import to_homogeneous_edge_index


class RandomNodeLoader(torch.utils.data.DataLoader):
    r"""A data loader that randomly samples nodes within a graph and returns
    their induced subgraph.

    .. note::

        For an example of using
        :class:`~torch_geometric.loader.RandomNodeLoader`, see
        `examples/ogbn_proteins_deepgcn.py
        `_.

    Args:
        data (torch_geometric.data.Data or torch_geometric.data.HeteroData):
            The :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` graph object.
        num_parts (int): The number of partitions.
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`num_workers`.
    """
    def __init__(
        self,
        data: Union[Data, HeteroData],
        num_parts: int,
        **kwargs,
    ):
        self.data = data
        self.num_parts = num_parts

        if isinstance(data, HeteroData):
            edge_index, node_dict, edge_dict = to_homogeneous_edge_index(data)
            self.node_dict, self.edge_dict = node_dict, edge_dict
        else:
            edge_index = data.edge_index

        self.edge_index = edge_index
        self.num_nodes = data.num_nodes

        super().__init__(
            range(self.num_nodes),
            batch_size=math.ceil(self.num_nodes / num_parts),
            collate_fn=self.collate_fn,
            **kwargs,
        )

    def collate_fn(self, index):
        if not isinstance(index, Tensor):
            index = torch.tensor(index)

        if isinstance(self.data, Data):
            return self.data.subgraph(index)

        elif isinstance(self.data, HeteroData):
            node_dict = {
                key: index[(index >= start) & (index < end)] - start
                for key, (start, end) in self.node_dict.items()
            }
            return self.data.subgraph(node_dict)
pytorch/torch_geometric/loader/shadow.py000066400000000000000000000101151444262304300210300ustar00rootroot00000000000000import copy
from typing import Optional

import torch
from torch import Tensor

from torch_geometric.data import Batch, Data
from torch_geometric.typing import WITH_TORCH_SPARSE, SparseTensor


class ShaDowKHopSampler(torch.utils.data.DataLoader):
    r"""The ShaDow :math:`k`-hop sampler from the `"Decoupling the Depth and
    Scope of Graph Neural Networks" `_ paper.
    Given a graph in a :obj:`data` object, the sampler will create shallow,
    localized subgraphs.
    A deep GNN on this local graph then smooths the informative local signals.

    .. note::

        For an example of using :class:`ShaDowKHopSampler`, see
        `examples/shadow.py `_.

    Args:
        data (torch_geometric.data.Data): The graph data object.
        depth (int): The depth/number of hops of the localized subgraph.
        num_neighbors (int): The number of neighbors to sample for each node in
            each hop.
        node_idx (LongTensor or BoolTensor, optional): The nodes that should be
            considered for creating mini-batches.
            If set to :obj:`None`, all nodes will be
            considered.
        replace (bool, optional): If set to :obj:`True`, will sample neighbors
            with replacement. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size` or
            :obj:`num_workers`.
    """
    def __init__(self, data: Data, depth: int, num_neighbors: int,
                 node_idx: Optional[Tensor] = None, replace: bool = False,
                 **kwargs):

        if not WITH_TORCH_SPARSE:
            raise ImportError(
                f"'{self.__class__.__name__}' requires 'torch-sparse'")

        self.data = copy.copy(data)
        self.depth = depth
        self.num_neighbors = num_neighbors
        self.replace = replace

        if data.edge_index is not None:
            self.is_sparse_tensor = False
            row, col = data.edge_index.cpu()
            self.adj_t = SparseTensor(
                row=row, col=col, value=torch.arange(col.size(0)),
                sparse_sizes=(data.num_nodes, data.num_nodes)).t()
        else:
            self.is_sparse_tensor = True
            self.adj_t = data.adj_t.cpu()

        if node_idx is None:
            node_idx = torch.arange(self.adj_t.sparse_size(0))
        elif node_idx.dtype == torch.bool:
            node_idx = node_idx.nonzero(as_tuple=False).view(-1)
        self.node_idx = node_idx

        super().__init__(node_idx.tolist(), collate_fn=self.__collate__,
                         **kwargs)

    def __collate__(self, n_id):
        n_id = torch.tensor(n_id)

        rowptr, col, value = self.adj_t.csr()
        out = torch.ops.torch_sparse.ego_k_hop_sample_adj(
            rowptr, col, n_id, self.depth, self.num_neighbors, self.replace)
        rowptr, col, n_id, e_id, ptr, root_n_id = out

        adj_t = SparseTensor(rowptr=rowptr, col=col,
                             value=value[e_id] if value is not None else None,
                             sparse_sizes=(n_id.numel(), n_id.numel()),
                             is_sorted=True, trust_data=True)

        batch = Batch(batch=torch.ops.torch_sparse.ptr2ind(ptr, n_id.numel()),
                      ptr=ptr)
        batch.root_n_id = root_n_id

        if self.is_sparse_tensor:
            batch.adj_t = adj_t
        else:
            row, col, e_id = adj_t.t().coo()
            batch.edge_index = torch.stack([row, col], dim=0)

        for k, v in self.data:
            if k in ['edge_index', 'adj_t', 'num_nodes', 'batch', 'ptr']:
                continue
            if k == 'y' and v.size(0) == self.data.num_nodes:
                batch[k] = v[n_id][root_n_id]
            elif isinstance(v, Tensor) and v.size(0) == self.data.num_nodes:
                batch[k] = v[n_id]
            elif isinstance(v, Tensor) and v.size(0) == self.data.num_edges:
                batch[k] = v[e_id]
            else:
                batch[k] = v

        return batch
pytorch/torch_geometric/loader/temporal_dataloader.py000066400000000000000000000024471444262304300235570ustar00rootroot00000000000000from typing import List

import torch

from torch_geometric.data import TemporalData


class TemporalDataLoader(torch.utils.data.DataLoader):
    r"""A data loader which merges succesive events of a
    :class:`torch_geometric.data.TemporalData` to a mini-batch.

    Args:
        data (TemporalData): The :obj:`~torch_geometric.data.TemporalData`
            from which to load the data.
        batch_size (int, optional): How many samples per batch to load.
            (default: :obj:`1`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`.
    """
    def __init__(self, data: TemporalData, batch_size: int = 1, **kwargs):
        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        kwargs.pop('shuffle', None)

        self.data = data
        self.events_per_batch = batch_size

        if kwargs.get('drop_last', False) and len(data) % batch_size != 0:
            arange = range(0, len(data) - batch_size, batch_size)
        else:
            arange = range(0, len(data), batch_size)

        super().__init__(arange, 1, shuffle=False, collate_fn=self, **kwargs)

    def __call__(self, arange: List[int]) -> TemporalData:
        return self.data[arange[0]:arange[0] + self.events_per_batch]
pytorch/torch_geometric/loader/utils.py000066400000000000000000000277611444262304300207220ustar00rootroot00000000000000import copy
import logging
import math
from collections.abc import Sequence
from typing import Any, Dict, Optional, Tuple, Union

import numpy as np
import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data import (
    Data,
    FeatureStore,
    GraphStore,
    HeteroData,
    TensorAttr,
    remote_backend_utils,
)
from torch_geometric.data.storage import EdgeStorage, NodeStorage
from torch_geometric.typing import (
    EdgeType,
    FeatureTensorType,
    InputEdges,
    InputNodes,
    NodeType,
    OptTensor,
    SparseTensor,
)


def index_select(value: FeatureTensorType, index: Tensor,
                 dim: int = 0) -> Tensor:

    # PyTorch currently only supports indexing via `torch.int64` :(
    index = index.to(torch.int64)

    if isinstance(value, Tensor):
        out: Optional[Tensor] = None
        if torch.utils.data.get_worker_info() is not None:
            # If we are in a background process, we write directly into a
            # shared memory tensor to avoid an extra copy:
            size = list(value.shape)
            size[dim] = index.numel()
            numel = math.prod(size)
            if torch_geometric.typing.WITH_PT2:
                storage = value.untyped_storage()._new_shared(
                    numel * value.element_size())
            else:
                storage = value.storage()._new_shared(numel)
            out = value.new(storage).view(size)

        return torch.index_select(value, dim, index, out=out)

    elif isinstance(value, np.ndarray):
        return torch.from_numpy(np.take(value, index, axis=dim))

    raise ValueError(f"Encountered invalid feature tensor type "
                     f"(got '{type(value)}')")


def filter_node_store_(store: NodeStorage, out_store: NodeStorage,
                       index: Tensor):
    # Filters a node storage object to only hold the nodes in `index`:
    for key, value in store.items():
        if key == 'num_nodes':
            out_store.num_nodes = index.numel()

        elif store.is_node_attr(key):
            if isinstance(value, Tensor):
                index = index.to(value.device)
            elif isinstance(value, np.ndarray):
                index = index.cpu()
            dim = store._parent().__cat_dim__(key, value, store)
            out_store[key] = index_select(value, index, dim=dim)


def filter_edge_store_(store: EdgeStorage, out_store: EdgeStorage, row: Tensor,
                       col: Tensor, index: OptTensor, perm: OptTensor = None):
    # Filters a edge storage object to only hold the edges in `index`,
    # which represents the new graph as denoted by `(row, col)`:
    for key, value in store.items():
        if key == 'edge_index':
            edge_index = torch.stack([row, col], dim=0)
            out_store.edge_index = edge_index.to(value.device)

        elif key == 'adj_t':
            # NOTE: We expect `(row, col)` to be sorted by `col` (CSC layout).
            row = row.to(value.device())
            col = col.to(value.device())
            edge_attr = value.storage.value()
            if edge_attr is not None:
                if index is not None:
                    index = index.to(edge_attr.device)
                    edge_attr = index_select(edge_attr, index, dim=0)
                else:
                    edge_attr = None
            sparse_sizes = out_store.size()[::-1]
            # TODO Currently, we set `is_sorted=False`, see:
            # https://github.com/pyg-team/pytorch_geometric/issues/4346
            out_store.adj_t = SparseTensor(row=col, col=row, value=edge_attr,
                                           sparse_sizes=sparse_sizes,
                                           is_sorted=False, trust_data=True)

        elif store.is_edge_attr(key):
            if index is None:
                out_store[key] = None
                continue

            dim = store._parent().__cat_dim__(key, value, store)
            if isinstance(value, Tensor):
                index = index.to(value.device)
            elif isinstance(value, np.ndarray):
                index = index.cpu()
            if perm is None:
                out_store[key] = index_select(value, index, dim=dim)
            else:
                if isinstance(value, Tensor):
                    perm = perm.to(value.device)
                elif isinstance(value, np.ndarray):
                    perm = perm.cpu()
                out_store[key] = index_select(
                    value,
                    perm[index.to(torch.int64)],
                    dim=dim,
                )


def filter_data(data: Data, node: Tensor, row: Tensor, col: Tensor,
                edge: OptTensor, perm: OptTensor = None) -> Data:
    # Filters a data object to only hold nodes in `node` and edges in `edge`:
    out = copy.copy(data)
    filter_node_store_(data._store, out._store, node)
    filter_edge_store_(data._store, out._store, row, col, edge, perm)
    return out


def filter_hetero_data(
    data: HeteroData,
    node_dict: Dict[NodeType, Tensor],
    row_dict: Dict[EdgeType, Tensor],
    col_dict: Dict[EdgeType, Tensor],
    edge_dict: Dict[EdgeType, OptTensor],
    perm_dict: Optional[Dict[EdgeType, OptTensor]] = None,
) -> HeteroData:
    # Filters a heterogeneous data object to only hold nodes in `node` and
    # edges in `edge` for each node and edge type, respectively:
    out = copy.copy(data)

    for node_type in out.node_types:
        # Handle the case of disconneted graph sampling:
        if node_type not in node_dict:
            node_dict[node_type] = torch.empty(0, dtype=torch.long)

        filter_node_store_(data[node_type], out[node_type],
                           node_dict[node_type])

    for edge_type in out.edge_types:
        # Handle the case of disconneted graph sampling:
        if edge_type not in row_dict:
            row_dict[edge_type] = torch.empty(0, dtype=torch.long)
        if edge_type not in col_dict:
            col_dict[edge_type] = torch.empty(0, dtype=torch.long)
        if edge_type not in edge_dict:
            edge_dict[edge_type] = torch.empty(0, dtype=torch.long)

        filter_edge_store_(
            data[edge_type],
            out[edge_type],
            row_dict[edge_type],
            col_dict[edge_type],
            edge_dict[edge_type],
            perm_dict.get(edge_type, None) if perm_dict else None,
        )

    return out


def filter_custom_store(
    feature_store: FeatureStore,
    graph_store: GraphStore,
    node_dict: Dict[str, Tensor],
    row_dict: Dict[str, Tensor],
    col_dict: Dict[str, Tensor],
    edge_dict: Dict[str, OptTensor],
    custom_cls: Optional[HeteroData] = None,
) -> HeteroData:
    r"""Constructs a `HeteroData` object from a feature store that only holds
    nodes in `node` end edges in `edge` for each node and edge type,
    respectively."""

    # Construct a new `HeteroData` object:
    data = custom_cls() if custom_cls is not None else HeteroData()

    # Filter edge storage:
    # TODO support edge attributes
    for attr in graph_store.get_all_edge_attrs():
        key = attr.edge_type
        if key in row_dict and key in col_dict:
            edge_index = torch.stack([row_dict[key], col_dict[key]], dim=0)
            data[attr.edge_type].edge_index = edge_index

    # Filter node storage:
    required_attrs = []
    for attr in feature_store.get_all_tensor_attrs():
        if attr.group_name in node_dict:
            attr.index = node_dict[attr.group_name]
            required_attrs.append(attr)
            data[attr.group_name].num_nodes = attr.index.size(0)

    # NOTE Here, we utilize `feature_store.multi_get` to give the feature store
    # full control over optimizing how it returns features (since the call is
    # synchronous, this amounts to giving the feature store control over all
    # iteration).
    tensors = feature_store.multi_get_tensor(required_attrs)
    for i, attr in enumerate(required_attrs):
        data[attr.group_name][attr.attr_name] = tensors[i]

    return data


# Input Utilities #############################################################


def get_input_nodes(
    data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
    input_nodes: Union[InputNodes, TensorAttr],
) -> Tuple[Optional[str], Sequence]:
    def to_index(tensor):
        if isinstance(tensor, Tensor) and tensor.dtype == torch.bool:
            return tensor.nonzero(as_tuple=False).view(-1)
        if not isinstance(tensor, Tensor):
            return torch.tensor(tensor, dtype=torch.long)
        return tensor

    if isinstance(data, Data):
        if input_nodes is None:
            return None, torch.arange(data.num_nodes)
        return None, to_index(input_nodes)

    elif isinstance(data, HeteroData):
        assert input_nodes is not None

        if isinstance(input_nodes, str):
            return input_nodes, torch.arange(data[input_nodes].num_nodes)

        assert isinstance(input_nodes, (list, tuple))
        assert len(input_nodes) == 2
        assert isinstance(input_nodes[0], str)

        node_type, input_nodes = input_nodes
        if input_nodes is None:
            return node_type, torch.arange(data[node_type].num_nodes)
        return node_type, to_index(input_nodes)

    else:  # Tuple[FeatureStore, GraphStore]
        feature_store, graph_store = data
        assert input_nodes is not None

        if isinstance(input_nodes, Tensor):
            return None, to_index(input_nodes)

        if isinstance(input_nodes, str):
            return input_nodes, torch.arange(
                remote_backend_utils.num_nodes(feature_store, graph_store,
                                               input_nodes))

        if isinstance(input_nodes, (list, tuple)):
            assert len(input_nodes) == 2
            assert isinstance(input_nodes[0], str)

            node_type, input_nodes = input_nodes
            if input_nodes is None:
                return node_type, torch.arange(
                    remote_backend_utils.num_nodes(feature_store, graph_store,
                                                   input_nodes))
            return node_type, to_index(input_nodes)


def get_edge_label_index(
    data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
    edge_label_index: InputEdges,
) -> Tuple[Optional[str], Tensor]:
    edge_type = None
    if isinstance(data, Data):
        if edge_label_index is None:
            return None, data.edge_index
        return None, edge_label_index

    assert edge_label_index is not None
    assert isinstance(edge_label_index, (list, tuple))

    if isinstance(data, HeteroData):
        if isinstance(edge_label_index[0], str):
            edge_type = edge_label_index
            edge_type = data._to_canonical(*edge_type)
            assert edge_type in data.edge_types
            return edge_type, data[edge_type].edge_index

        assert len(edge_label_index) == 2

        edge_type, edge_label_index = edge_label_index
        edge_type = data._to_canonical(*edge_type)

        if edge_label_index is None:
            return edge_type, data[edge_type].edge_index

        return edge_type, edge_label_index

    else:  # Tuple[FeatureStore, GraphStore]
        _, graph_store = data

        # Need the edge index in COO for LinkNeighborLoader:
        def _get_edge_index(edge_type):
            row_dict, col_dict, _ = graph_store.coo([edge_type])
            row = list(row_dict.values())[0]
            col = list(col_dict.values())[0]
            return torch.stack((row, col), dim=0)

        if isinstance(edge_label_index[0], str):
            edge_type = edge_label_index
            return edge_type, _get_edge_index(edge_type)

        assert len(edge_label_index) == 2
        edge_type, edge_label_index = edge_label_index

        if edge_label_index is None:
            return edge_type, _get_edge_index(edge_type)

        return edge_type, edge_label_index


def infer_filter_per_worker(data: Any) -> bool:
    out = True
    if isinstance(data, (Data, HeteroData)) and data.is_cuda:
        out = False
    logging.debug(f"Inferred 'filter_per_worker={out}' option for feature "
                  f"fetching routines of the data loader")
    return out
pytorch/torch_geometric/loader/zip_loader.py000066400000000000000000000066761444262304300217140ustar00rootroot00000000000000from typing import Any, Iterator, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.loader import LinkLoader, NodeLoader
from torch_geometric.loader.base import DataLoaderIterator
from torch_geometric.loader.utils import infer_filter_per_worker


class ZipLoader(torch.utils.data.DataLoader):
    r"""A loader that returns a tuple of data objects by sampling from multiple
    :class:`NodeLoader` or :class:`LinkLoader` instances.

    Args:
        loaders (List[NodeLoader] or List[LinkLoader]): The loader instances.
        filter_per_worker (bool, optional): If set to :obj:`True`, will filter
            the returned data in each worker's subprocess.
            If set to :obj:`False`, will filter the returned data in the main
            process.
            If set to :obj:`None`, will automatically infer the decision based
            on whether data partially lives on the GPU
            (:obj:`filter_per_worker=True`) or entirely on the CPU
            (:obj:`filter_per_worker=False`).
            There exists different trade-offs for setting this option.
            Specifically, setting this option to :obj:`True` for in-memory
            datasets will move all features to shared memory, which may result
            in too many open file handles. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch.utils.data.DataLoader`, such as :obj:`batch_size`,
            :obj:`shuffle`, :obj:`drop_last` or :obj:`num_workers`.
    """
    def __init__(
        self,
        loaders: Union[List[NodeLoader], List[LinkLoader]],
        filter_per_worker: Optional[bool] = None,
        **kwargs,
    ):
        if filter_per_worker is None:
            filter_per_worker = infer_filter_per_worker(loaders[0].data)

        # Remove for PyTorch Lightning:
        kwargs.pop('dataset', None)
        kwargs.pop('collate_fn', None)

        for loader in loaders:
            if not callable(getattr(loader, 'collate_fn', None)):
                raise ValueError("'{loader.__class__.__name__}' does not have "
                                 "a 'collate_fn' method")
            if not callable(getattr(loader, 'filter_fn', None)):
                raise ValueError("'{loader.__class__.__name__}' does not have "
                                 "a 'filter_fn' method")
            loader.filter_per_worker = filter_per_worker

        iterator = range(min([len(loader.dataset) for loader in loaders]))
        super().__init__(iterator, collate_fn=self.collate_fn, **kwargs)

        self.loaders = loaders
        self.filter_per_worker = filter_per_worker

    def collate_fn(self, index: List[int]) -> Tuple[Any, ...]:
        if not isinstance(index, Tensor):
            index = torch.tensor(index, dtype=torch.long)

        return tuple(loader.collate_fn(index) for loader in self.loaders)

    def filter_fn(
        self,
        outs: Tuple[Any, ...],
    ) -> Tuple[Union[Data, HeteroData], ...]:
        loaders = self.loaders
        return tuple(loader.filter_fn(v) for loader, v in zip(loaders, outs))

    def _get_iterator(self) -> Iterator:
        if self.filter_per_worker:
            return super()._get_iterator()

        # Execute `filter_fn` in the main process:
        return DataLoaderIterator(super()._get_iterator(), self.filter_fn)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(loaders={self.loaders})'
pytorch/torch_geometric/logging.py000066400000000000000000000015001444262304300177210ustar00rootroot00000000000000import sys
from typing import Any

_wandb_initialized: bool = False


def init_wandb(name: str, **kwargs):
    if '--wandb' not in sys.argv:
        return

    from datetime import datetime

    import wandb

    wandb.init(
        project=name,
        entity='pytorch-geometric',
        name=datetime.now().strftime('%Y-%m-%d_%H:%M'),
        config=kwargs,
    )

    global _wandb_initialized
    _wandb_initialized = True


def log(**kwargs):
    def _map(value: Any) -> str:
        if isinstance(value, int) and not isinstance(value, bool):
            return f'{value:03d}'
        if isinstance(value, float):
            return f'{value:.4f}'
        return value

    print(', '.join(f'{key}: {_map(value)}' for key, value in kwargs.items()))

    if _wandb_initialized:
        import wandb
        wandb.log(kwargs)
pytorch/torch_geometric/nn/000077500000000000000000000000001444262304300163405ustar00rootroot00000000000000pytorch/torch_geometric/nn/__init__.py000066400000000000000000000015171444262304300204550ustar00rootroot00000000000000from .reshape import Reshape
from .sequential import Sequential
from .data_parallel import DataParallel
from .to_hetero_transformer import to_hetero
from .to_hetero_with_bases_transformer import to_hetero_with_bases
from .to_fixed_size_transformer import to_fixed_size
from .encoding import PositionalEncoding, TemporalEncoding
from .summary import summary

from .aggr import *  # noqa
from .conv import *  # noqa
from .pool import *  # noqa
from .glob import *  # noqa
from .norm import *  # noqa
from .unpool import *  # noqa
from .dense import *  # noqa
from .kge import *  # noqa
from .models import *  # noqa
from .functional import *  # noqa

__all__ = [
    'Reshape',
    'Sequential',
    'DataParallel',
    'to_hetero',
    'to_hetero_with_bases',
    'to_fixed_size',
    'PositionalEncoding',
    'TemporalEncoding',
    'summary',
]
pytorch/torch_geometric/nn/aggr/000077500000000000000000000000001444262304300172605ustar00rootroot00000000000000pytorch/torch_geometric/nn/aggr/__init__.py000066400000000000000000000025651444262304300214010ustar00rootroot00000000000000from .base import Aggregation
from .multi import MultiAggregation
from .basic import (
    MeanAggregation,
    SumAggregation,
    MaxAggregation,
    MinAggregation,
    MulAggregation,
    VarAggregation,
    StdAggregation,
    SoftmaxAggregation,
    PowerMeanAggregation,
)
from .quantile import MedianAggregation, QuantileAggregation
from .lstm import LSTMAggregation
from .gru import GRUAggregation
from .set2set import Set2Set
from .scaler import DegreeScalerAggregation
from .equilibrium import EquilibriumAggregation
from .sort import SortAggregation
from .gmt import GraphMultisetTransformer
from .attention import AttentionalAggregation
from .mlp import MLPAggregation
from .deep_sets import DeepSetsAggregation
from .set_transformer import SetTransformerAggregation

__all__ = classes = [
    'Aggregation',
    'MultiAggregation',
    'SumAggregation',
    'MeanAggregation',
    'MaxAggregation',
    'MinAggregation',
    'MulAggregation',
    'VarAggregation',
    'StdAggregation',
    'SoftmaxAggregation',
    'PowerMeanAggregation',
    'MedianAggregation',
    'QuantileAggregation',
    'LSTMAggregation',
    'GRUAggregation',
    'Set2Set',
    'DegreeScalerAggregation',
    'SortAggregation',
    'GraphMultisetTransformer',
    'AttentionalAggregation',
    'EquilibriumAggregation',
    'MLPAggregation',
    'DeepSetsAggregation',
    'SetTransformerAggregation',
]
pytorch/torch_geometric/nn/aggr/attention.py000066400000000000000000000045331444262304300216440ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.inits import reset
from torch_geometric.utils import softmax


class AttentionalAggregation(Aggregation):
    r"""The soft attention aggregation layer from the `"Graph Matching Networks
    for Learning the Similarity of Graph Structured Objects"
    `_ paper

    .. math::
        \mathbf{r}_i = \sum_{n=1}^{N_i} \mathrm{softmax} \left(
        h_{\mathrm{gate}} ( \mathbf{x}_n ) \right) \cdot
        h_{\mathbf{\Theta}} ( \mathbf{x}_n ),

    where :math:`h_{\mathrm{gate}} \colon \mathbb{R}^F \to
    \mathbb{R}` and :math:`h_{\mathbf{\Theta}}` denote neural networks, *i.e.*
    MLPs.

    Args:
        gate_nn (torch.nn.Module): A neural network :math:`h_{\mathrm{gate}}`
            that computes attention scores by mapping node features :obj:`x` of
            shape :obj:`[-1, in_channels]` to shape :obj:`[-1, 1]` (for
            node-level gating) or :obj:`[1, out_channels]` (for feature-level
            gating), *e.g.*, defined by :class:`torch.nn.Sequential`.
        nn (torch.nn.Module, optional): A neural network
            :math:`h_{\mathbf{\Theta}}` that maps node features :obj:`x` of
            shape :obj:`[-1, in_channels]` to shape :obj:`[-1, out_channels]`
            before combining them with the attention scores, *e.g.*, defined by
            :class:`torch.nn.Sequential`. (default: :obj:`None`)
    """
    def __init__(self, gate_nn: torch.nn.Module,
                 nn: Optional[torch.nn.Module] = None):
        super().__init__()
        self.gate_nn = gate_nn
        self.nn = nn
        self.reset_parameters()

    def reset_parameters(self):
        reset(self.gate_nn)
        reset(self.nn)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        self.assert_two_dimensional_input(x, dim)
        gate = self.gate_nn(x)
        x = self.nn(x) if self.nn is not None else x
        gate = softmax(gate, index, ptr, dim_size, dim)
        return self.reduce(gate * x, index, ptr, dim_size, dim)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(gate_nn={self.gate_nn}, '
                f'nn={self.nn})')
pytorch/torch_geometric/nn/aggr/base.py000066400000000000000000000164311444262304300205510ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.utils import scatter, segment, to_dense_batch


class Aggregation(torch.nn.Module):
    r"""An abstract base class for implementing custom aggregations.

    Aggregation can be either performed via an :obj:`index` vector, which
    defines the mapping from input elements to their location in the output:

    |

    .. image:: https://raw.githubusercontent.com/rusty1s/pytorch_scatter/
            master/docs/source/_figures/add.svg?sanitize=true
        :align: center
        :width: 400px

    |

    Notably, :obj:`index` does not have to be sorted (for most aggregation
    operators):

    .. code-block::

       # Feature matrix holding 10 elements with 64 features each:
       x = torch.randn(10, 64)

       # Assign each element to one of three sets:
       index = torch.tensor([0, 0, 1, 0, 2, 0, 2, 1, 0, 2])

       output = aggr(x, index)  #  Output shape: [3, 64]

    Alternatively, aggregation can be achieved via a "compressed" index vector
    called :obj:`ptr`. Here, elements within the same set need to be grouped
    together in the input, and :obj:`ptr` defines their boundaries:

    .. code-block::

       # Feature matrix holding 10 elements with 64 features each:
       x = torch.randn(10, 64)

       # Define the boundary indices for three sets:
       ptr = torch.tensor([0, 4, 7, 10])

       output = aggr(x, ptr=ptr)  #  Output shape: [4, 64]

    Note that at least one of :obj:`index` or :obj:`ptr` must be defined.

    Shapes:
        - **input:**
          node features :math:`(*, |\mathcal{V}|, F_{in})` or edge features
          :math:`(*, |\mathcal{E}|, F_{in})`,
          index vector :math:`(|\mathcal{V}|)` or :math:`(|\mathcal{E}|)`,
        - **output:** graph features :math:`(*, |\mathcal{G}|, F_{out})` or
          node features :math:`(*, |\mathcal{V}|, F_{out})`
    """
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            index (torch.Tensor, optional): The indices of elements for
                applying the aggregation.
                One of :obj:`index` or :obj:`ptr` must be defined.
                (default: :obj:`None`)
            ptr (torch.Tensor, optional): If given, computes the aggregation
                based on sorted inputs in CSR representation.
                One of :obj:`index` or :obj:`ptr` must be defined.
                (default: :obj:`None`)
            dim_size (int, optional): The size of the output tensor at
                dimension :obj:`dim` after aggregation. (default: :obj:`None`)
            dim (int, optional): The dimension in which to aggregate.
                (default: :obj:`-2`)
            max_num_elements: (int, optional): The maximum number of elements
                within a single aggregation group. (default: :obj:`None`)
        """
        pass

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        pass

    @disable_dynamic_shapes(required_args=['dim_size'])
    def __call__(self, x: Tensor, index: Optional[Tensor] = None,
                 ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                 dim: int = -2, **kwargs) -> Tensor:

        if dim >= x.dim() or dim < -x.dim():
            raise ValueError(f"Encountered invalid dimension '{dim}' of "
                             f"source tensor with {x.dim()} dimensions")

        if index is None and ptr is None:
            index = x.new_zeros(x.size(dim), dtype=torch.long)

        if ptr is not None:
            if dim_size is None:
                dim_size = ptr.numel() - 1
            elif dim_size != ptr.numel() - 1:
                raise ValueError(f"Encountered invalid 'dim_size' (got "
                                 f"'{dim_size}' but expected "
                                 f"'{ptr.numel() - 1}')")

        if index is not None and dim_size is None:
            dim_size = int(index.max()) + 1 if index.numel() > 0 else 0

        try:
            return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
        except (IndexError, RuntimeError) as e:
            if index is not None:
                if index.numel() > 0 and dim_size <= int(index.max()):
                    raise ValueError(f"Encountered invalid 'dim_size' (got "
                                     f"'{dim_size}' but expected "
                                     f">= '{int(index.max()) + 1}')")
            raise e

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'

    # Assertions ##############################################################

    def assert_index_present(self, index: Optional[Tensor]):
        # TODO Currently, not all aggregators support `ptr`. This assert helps
        # to ensure that we require `index` to be passed to the computation:
        if index is None:
            raise NotImplementedError(
                "Aggregation requires 'index' to be specified")

    def assert_sorted_index(self, index: Optional[Tensor]):
        if index is not None and not torch.all(index[:-1] <= index[1:]):
            raise ValueError("Can not perform aggregation since the 'index' "
                             "tensor is not sorted")

    def assert_two_dimensional_input(self, x: Tensor, dim: int):
        if x.dim() != 2:
            raise ValueError(f"Aggregation requires two-dimensional inputs "
                             f"(got '{x.dim()}')")

        if dim not in [-2, 0]:
            raise ValueError(f"Aggregation needs to perform aggregation in "
                             f"first dimension (got '{dim}')")

    # Helper methods ##########################################################

    def reduce(self, x: Tensor, index: Optional[Tensor] = None,
               ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
               dim: int = -2, reduce: str = 'sum') -> Tensor:

        if ptr is not None:
            ptr = expand_left(ptr, dim, dims=x.dim())
            return segment(x, ptr, reduce=reduce)

        assert index is not None
        return scatter(x, index, dim, dim_size, reduce)

    def to_dense_batch(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        fill_value: float = 0.0,
        max_num_elements: Optional[int] = None,
    ) -> Tuple[Tensor, Tensor]:

        # TODO Currently, `to_dense_batch` can only operate on `index`:
        self.assert_index_present(index)
        self.assert_sorted_index(index)
        self.assert_two_dimensional_input(x, dim)

        return to_dense_batch(
            x,
            index,
            batch_size=dim_size,
            fill_value=fill_value,
            max_num_nodes=max_num_elements,
        )


###############################################################################


def expand_left(ptr: Tensor, dim: int, dims: int) -> Tensor:
    for _ in range(dims + dim if dim < 0 else dim):
        ptr = ptr.unsqueeze(0)
    return ptr
pytorch/torch_geometric/nn/aggr/basic.py000066400000000000000000000253701444262304300207220ustar00rootroot00000000000000import math
from typing import Optional

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.utils import softmax


class SumAggregation(Aggregation):
    r"""An aggregation operator that sums up features across a set of elements

    .. math::
        \mathrm{sum}(\mathcal{X}) = \sum_{\mathbf{x}_i \in \mathcal{X}}
        \mathbf{x}_i.
    """
    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')


class MeanAggregation(Aggregation):
    r"""An aggregation operator that averages features across a set of elements

    .. math::
        \mathrm{mean}(\mathcal{X}) = \frac{1}{|\mathcal{X}|}
        \sum_{\mathbf{x}_i \in \mathcal{X}} \mathbf{x}_i.
    """
    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        return self.reduce(x, index, ptr, dim_size, dim, reduce='mean')


class MaxAggregation(Aggregation):
    r"""An aggregation operator that takes the feature-wise maximum across a
    set of elements

    .. math::
        \mathrm{max}(\mathcal{X}) = \max_{\mathbf{x}_i \in \mathcal{X}}
        \mathbf{x}_i.
    """
    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        return self.reduce(x, index, ptr, dim_size, dim, reduce='max')


class MinAggregation(Aggregation):
    r"""An aggregation operator that takes the feature-wise minimum across a
    set of elements

    .. math::
        \mathrm{min}(\mathcal{X}) = \min_{\mathbf{x}_i \in \mathcal{X}}
        \mathbf{x}_i.
    """
    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        return self.reduce(x, index, ptr, dim_size, dim, reduce='min')


class MulAggregation(Aggregation):
    r"""An aggregation operator that multiples features across a set of
    elements

    .. math::
        \mathrm{mul}(\mathcal{X}) = \prod_{\mathbf{x}_i \in \mathcal{X}}
        \mathbf{x}_i.
    """
    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        # TODO Currently, `mul` reduction can only operate on `index`:
        self.assert_index_present(index)
        return self.reduce(x, index, None, dim_size, dim, reduce='mul')


class VarAggregation(Aggregation):
    r"""An aggregation operator that takes the feature-wise variance across a
    set of elements

    .. math::
        \mathrm{var}(\mathcal{X}) = \mathrm{mean}(\{ \mathbf{x}_i^2 : x \in
        \mathcal{X} \}) - \mathrm{mean}(\mathcal{X})^2.

    Args:
        semi_grad (bool, optional): If set to :obj:`True`, will turn off
            gradient calculation during :math:`E[X^2]` computation. Therefore,
            only semi-gradients are used during backpropagation. Useful for
            saving memory and accelerating backward computation.
            (default: :obj:`False`)
    """
    def __init__(self, semi_grad: bool = False):
        super().__init__()
        self.semi_grad = semi_grad

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        mean = self.reduce(x, index, ptr, dim_size, dim, reduce='mean')
        if self.semi_grad:
            with torch.no_grad():
                mean2 = self.reduce(x * x, index, ptr, dim_size, dim, 'mean')
        else:
            mean2 = self.reduce(x * x, index, ptr, dim_size, dim, 'mean')
        return mean2 - mean * mean


class StdAggregation(Aggregation):
    r"""An aggregation operator that takes the feature-wise standard deviation
    across a set of elements

    .. math::
        \mathrm{std}(\mathcal{X}) = \sqrt{\mathrm{var}(\mathcal{X})}.

    Args:
        semi_grad (bool, optional): If set to :obj:`True`, will turn off
            gradient calculation during :math:`E[X^2]` computation. Therefore,
            only semi-gradients are used during backpropagation. Useful for
            saving memory and accelerating backward computation.
            (default: :obj:`False`)
    """
    def __init__(self, semi_grad: bool = False):
        super().__init__()
        self.var_aggr = VarAggregation(semi_grad)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        var = self.var_aggr(x, index, ptr, dim_size, dim)
        # Allow "undefined" gradient at `sqrt(0.0)`:
        out = var.clamp(min=1e-5).sqrt()
        out = out.masked_fill(out <= math.sqrt(1e-5), 0.0)
        return out


class SoftmaxAggregation(Aggregation):
    r"""The softmax aggregation operator based on a temperature term, as
    described in the `"DeeperGCN: All You Need to Train Deeper GCNs"
    `_ paper

    .. math::
        \mathrm{softmax}(\mathcal{X}|t) = \sum_{\mathbf{x}_i\in\mathcal{X}}
        \frac{\exp(t\cdot\mathbf{x}_i)}{\sum_{\mathbf{x}_j\in\mathcal{X}}
        \exp(t\cdot\mathbf{x}_j)}\cdot\mathbf{x}_{i},

    where :math:`t` controls the softness of the softmax when aggregating over
    a set of features :math:`\mathcal{X}`.

    Args:
        t (float, optional): Initial inverse temperature for softmax
            aggregation. (default: :obj:`1.0`)
        learn (bool, optional): If set to :obj:`True`, will learn the value
            :obj:`t` for softmax aggregation dynamically.
            (default: :obj:`False`)
        semi_grad (bool, optional): If set to :obj:`True`, will turn off
            gradient calculation during softmax computation. Therefore, only
            semi-gradients are used during backpropagation. Useful for saving
            memory and accelerating backward computation when :obj:`t` is not
            learnable. (default: :obj:`False`)
        channels (int, optional): Number of channels to learn from :math:`t`.
            If set to a value greater than :obj:`1`, :math:`t` will be learned
            per input feature channel. This requires compatible shapes for the
            input to the forward calculation. (default: :obj:`1`)
    """
    def __init__(self, t: float = 1.0, learn: bool = False,
                 semi_grad: bool = False, channels: int = 1):
        super().__init__()

        if learn and semi_grad:
            raise ValueError(
                f"Cannot enable 'semi_grad' in '{self.__class__.__name__}' in "
                f"case the temperature term 't' is learnable")

        if not learn and channels != 1:
            raise ValueError(f"Cannot set 'channels' greater than '1' in case "
                             f"'{self.__class__.__name__}' is not trainable")

        self._init_t = t
        self.learn = learn
        self.semi_grad = semi_grad
        self.channels = channels

        self.t = Parameter(torch.Tensor(channels)) if learn else t
        self.reset_parameters()

    def reset_parameters(self):
        if isinstance(self.t, Tensor):
            self.t.data.fill_(self._init_t)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        t = self.t
        if self.channels != 1:
            self.assert_two_dimensional_input(x, dim)
            assert isinstance(t, Tensor)
            t = t.view(-1, self.channels)

        alpha = x
        if not isinstance(t, (int, float)) or t != 1:
            alpha = x * t

        if not self.learn and self.semi_grad:
            with torch.no_grad():
                alpha = softmax(alpha, index, ptr, dim_size, dim)
        else:
            alpha = softmax(alpha, index, ptr, dim_size, dim)
        return self.reduce(x * alpha, index, ptr, dim_size, dim, reduce='sum')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(learn={self.learn})')


class PowerMeanAggregation(Aggregation):
    r"""The powermean aggregation operator based on a power term, as
    described in the `"DeeperGCN: All You Need to Train Deeper GCNs"
    `_ paper

    .. math::
        \mathrm{powermean}(\mathcal{X}|p) = \left(\frac{1}{|\mathcal{X}|}
        \sum_{\mathbf{x}_i\in\mathcal{X}}\mathbf{x}_i^{p}\right)^{1/p},

    where :math:`p` controls the power of the powermean when aggregating over
    a set of features :math:`\mathcal{X}`.

    Args:
        p (float, optional): Initial power for powermean aggregation.
            (default: :obj:`1.0`)
        learn (bool, optional): If set to :obj:`True`, will learn the value
            :obj:`p` for powermean aggregation dynamically.
            (default: :obj:`False`)
        channels (int, optional): Number of channels to learn from :math:`p`.
            If set to a value greater than :obj:`1`, :math:`p` will be learned
            per input feature channel. This requires compatible shapes for the
            input to the forward calculation. (default: :obj:`1`)
    """
    def __init__(self, p: float = 1.0, learn: bool = False, channels: int = 1):
        super().__init__()

        if not learn and channels != 1:
            raise ValueError(f"Cannot set 'channels' greater than '1' in case "
                             f"'{self.__class__.__name__}' is not trainable")

        self._init_p = p
        self.learn = learn
        self.channels = channels

        self.p = Parameter(torch.Tensor(channels)) if learn else p
        self.reset_parameters()

    def reset_parameters(self):
        if isinstance(self.p, Tensor):
            self.p.data.fill_(self._init_p)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        p = self.p
        if self.channels != 1:
            assert isinstance(p, Tensor)
            self.assert_two_dimensional_input(x, dim)
            p = p.view(-1, self.channels)

        if not isinstance(p, (int, float)) or p != 1:
            x = x.clamp(min=0, max=100).pow(p)

        out = self.reduce(x, index, ptr, dim_size, dim, reduce='mean')

        if not isinstance(p, (int, float)) or p != 1:
            out = out.clamp(min=0, max=100).pow(1. / p)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(learn={self.learn})')
pytorch/torch_geometric/nn/aggr/deep_sets.py000066400000000000000000000040201444262304300216010ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.inits import reset


class DeepSetsAggregation(Aggregation):
    r"""Performs Deep Sets aggregation in which the elements to aggregate are
    first transformed by a Multi-Layer Perceptron (MLP)
    :math:`\phi_{\mathbf{\Theta}}`, summed, and then transformed by another MLP
    :math:`\rho_{\mathbf{\Theta}}`, as suggested in the `"Graph Neural Networks
    with Adaptive Readouts" `_ paper.

    Args:
        local_nn (torch.nn.Module, optional): The neural network
            :math:`\phi_{\mathbf{\Theta}}`, *e.g.*, defined by
            :class:`torch.nn.Sequential` or
            :class:`torch_geometric.nn.models.MLP`. (default: :obj:`None`)
        global_nn (torch.nn.Module, optional): The neural network
            :math:`\rho_{\mathbf{\Theta}}`, *e.g.*, defined by
            :class:`torch.nn.Sequential` or
            :class:`torch_geometric.nn.models.MLP`. (default: :obj:`None`)
    """
    def __init__(
        self,
        local_nn: Optional[torch.nn.Module] = None,
        global_nn: Optional[torch.nn.Module] = None,
    ):
        super().__init__()
        self.local_nn = local_nn
        self.global_nn = global_nn

    def reset_parameters(self):
        if self.local_nn is not None:
            reset(self.local_nn)
        if self.global_nn is not None:
            reset(self.global_nn)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        if self.local_nn is not None:
            x = self.local_nn(x)
        x = self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
        if self.global_nn is not None:
            x = self.global_nn(x)
        return x

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(local_nn={self.local_nn}, '
                f'global_nn={self.global_nn})')
pytorch/torch_geometric/nn/aggr/equilibrium.py000066400000000000000000000147631444262304300221740ustar00rootroot00000000000000from typing import Callable, List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.inits import reset
from torch_geometric.utils import scatter


class ResNetPotential(torch.nn.Module):
    def __init__(self, in_channels: int, out_channels: int,
                 num_layers: List[int]):

        super().__init__()
        sizes = [in_channels] + num_layers + [out_channels]
        self.layers = torch.nn.ModuleList([
            torch.nn.Sequential(torch.nn.Linear(in_size, out_size),
                                torch.nn.LayerNorm(out_size), torch.nn.Tanh())
            for in_size, out_size in zip(sizes[:-2], sizes[1:-1])
        ])
        self.layers.append(torch.nn.Linear(sizes[-2], sizes[-1]))

        self.res_trans = torch.nn.ModuleList([
            torch.nn.Linear(in_channels, layer_size)
            for layer_size in num_layers + [out_channels]
        ])

    def forward(self, x: Tensor, y: Tensor, index: Optional[Tensor],
                dim_size: Optional[int] = None) -> Tensor:
        if index is None:
            inp = torch.cat([x, y.expand(x.size(0), -1)], dim=1)
        else:
            inp = torch.cat([x, y[index]], dim=1)

        h = inp
        for layer, res in zip(self.layers, self.res_trans):
            h = layer(h)
            h = res(inp) + h

        if index is None:
            return h.mean()

        if dim_size is None:
            dim_size = int(index.max().item() + 1)

        return scatter(h, index, 0, dim_size, reduce='mean').sum()


class MomentumOptimizer(torch.nn.Module):
    r"""
    Provides an inner loop optimizer for the implicitly defined output
    layer. It is based on an unrolled Nesterov momentum algorithm.

    Args:
        learning_rate (flaot): learning rate for optimizer.
        momentum (float): momentum for optimizer.
        learnable (bool): If :obj:`True` then the :obj:`learning_rate` and
            :obj:`momentum` will be learnable parameters. If False they
            are fixed. (default: :obj:`True`)
    """
    def __init__(self, learning_rate: float = 0.1, momentum: float = 0.9,
                 learnable: bool = True):
        super().__init__()

        self._initial_lr = learning_rate
        self._initial_mom = momentum
        self._lr = torch.nn.Parameter(Tensor([learning_rate]),
                                      requires_grad=learnable)
        self._mom = torch.nn.Parameter(Tensor([momentum]),
                                       requires_grad=learnable)
        self.softplus = torch.nn.Softplus()
        self.sigmoid = torch.nn.Sigmoid()

    def reset_parameters(self):
        self._lr.data.fill_(self._initial_lr)
        self._mom.data.fill_(self._initial_mom)

    @property
    def learning_rate(self):
        return self.softplus(self._lr)

    @property
    def momentum(self):
        return self.sigmoid(self._mom)

    def forward(
        self,
        x: Tensor,
        y: Tensor,
        index: Optional[Tensor],
        dim_size: Optional[int],
        func: Callable[[Tensor, Tensor, Optional[Tensor]], Tensor],
        iterations: int = 5,
    ) -> Tuple[Tensor, float]:

        momentum_buffer = torch.zeros_like(y)
        for _ in range(iterations):
            val = func(x, y, index, dim_size)
            grad = torch.autograd.grad(val, y, create_graph=True,
                                       retain_graph=True)[0]
            delta = self.learning_rate * grad
            momentum_buffer = self.momentum * momentum_buffer - delta
            y = y + momentum_buffer
        return y


class EquilibriumAggregation(Aggregation):
    r"""The equilibrium aggregation layer from the `"Equilibrium Aggregation:
    Encoding Sets via Optimization" `_ paper.
    The output of this layer :math:`\mathbf{y}` is defined implicitly via a
    potential function :math:`F(\mathbf{x}, \mathbf{y})`, a regularization term
    :math:`R(\mathbf{y})`, and the condition

    .. math::
        \mathbf{y} = \min_\mathbf{y} R(\mathbf{y}) + \sum_{i}
        F(\mathbf{x}_i, \mathbf{y}).

    The given implementation uses a ResNet-like model for the potential
    function and a simple :math:`L_2` norm :math:`R(\mathbf{y}) =
    \textrm{softplus}(\lambda) \cdot {\| \mathbf{y} \|}^2_2` for the
    regularizer with learnable weight :math:`\lambda`.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        num_layers (List[int): List of hidden channels in the potential
            function.
        grad_iter (int): The number of steps to take in the internal gradient
            descent. (default: :obj:`5`)
        lamb (float): The initial regularization constant.
            (default: :obj:`0.1`)
    """
    def __init__(self, in_channels: int, out_channels: int,
                 num_layers: List[int], grad_iter: int = 5, lamb: float = 0.1):
        super().__init__()

        self.potential = ResNetPotential(in_channels + out_channels, 1,
                                         num_layers)
        self.optimizer = MomentumOptimizer()
        self.initial_lamb = lamb
        self.lamb = torch.nn.Parameter(Tensor(1), requires_grad=True)
        self.softplus = torch.nn.Softplus()
        self.grad_iter = grad_iter
        self.output_dim = out_channels
        self.reset_parameters()

    def reset_parameters(self):
        self.lamb.data.fill_(self.initial_lamb)
        reset(self.optimizer)
        reset(self.potential)

    def init_output(self, dim_size: int) -> Tensor:
        return torch.zeros(dim_size, self.output_dim, requires_grad=True,
                           device=self.lamb.device).float()

    def reg(self, y: Tensor) -> Tensor:
        return self.softplus(self.lamb) * y.square().sum(dim=-1).mean()

    def energy(self, x: Tensor, y: Tensor, index: Optional[Tensor],
               dim_size: Optional[int] = None):
        return self.potential(x, y, index, dim_size) + self.reg(y)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        self.assert_index_present(index)

        dim_size = int(index.max()) + 1 if dim_size is None else dim_size

        with torch.enable_grad():
            y = self.optimizer(x, self.init_output(dim_size), index, dim_size,
                               self.energy, iterations=self.grad_iter)

        return y

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}()')
pytorch/torch_geometric/nn/aggr/fused.py000066400000000000000000000277051444262304300207530ustar00rootroot00000000000000import math
from typing import Dict, List, Optional, Tuple, Union

from torch import Tensor

from torch_geometric.nn.aggr.base import Aggregation
from torch_geometric.nn.aggr.basic import (
    MaxAggregation,
    MeanAggregation,
    MinAggregation,
    MulAggregation,
    StdAggregation,
    SumAggregation,
    VarAggregation,
)
from torch_geometric.nn.resolver import aggregation_resolver
from torch_geometric.utils import scatter


class FusedAggregation(Aggregation):
    r"""Helper class to fuse computation of multiple aggregations together.
    Used internally in :class:`~torch_geometric.nn.aggr.MultiAggregation` to
    speed-up computation.
    Currently, the following optimizations are performed:

    * :class:`MeanAggregation` will share the output with
      :class:`SumAggregation` in case it is present as well.

    * :class:`VarAggregation` will share the output with either
      :class:`MeanAggregation` or :class:`SumAggregation` in case one of them
      is present as well.

    * :class:`StdAggregation` will share the output with either
      :class:`VarAggregation`, :class:`MeanAggregation` or
      :class:`SumAggregation` in case one of them is present as well.

    In addition, temporary values such as the count per group index are shared
    as well.

    Benchmarking results on PyTorch 1.12 (summed over 1000 runs):

    +------------------------------+---------+---------+
    | Aggregators                  | Vanilla | Fusion  |
    +==============================+=========+=========+
    | :obj:`[sum, mean]`           | 0.3325s | 0.1996s |
    +------------------------------+---------+---------+
    | :obj:`[sum, mean, min, max]` | 0.7139s | 0.5037s |
    +------------------------------+---------+---------+
    | :obj:`[sum, mean, var]`      | 0.6849s | 0.3871s |
    +------------------------------+---------+---------+
    | :obj:`[sum, mean, var, std]` | 1.0955s | 0.3973s |
    +------------------------------+---------+---------+

    Args:
        aggrs (list): The list of aggregation schemes to use.
    """
    # We can fuse all aggregations together that rely on `scatter` directives.
    FUSABLE_AGGRS = {
        SumAggregation,
        MeanAggregation,
        MinAggregation,
        MaxAggregation,
        MulAggregation,
        VarAggregation,
        StdAggregation,
    }

    # All aggregations that rely on computing the degree of indices.
    DEGREE_BASED_AGGRS = {
        MeanAggregation,
        VarAggregation,
        StdAggregation,
    }

    # Map aggregations to `reduce` options in `scatter` directives.
    REDUCE = {
        'SumAggregation': 'sum',
        'MeanAggregation': 'sum',
        'MinAggregation': 'min',
        'MaxAggregation': 'max',
        'MulAggregation': 'mul',
        'VarAggregation': 'pow_sum',
        'StdAggregation': 'pow_sum',
    }

    def __init__(self, aggrs: List[Union[Aggregation, str]]):
        super().__init__()

        if not isinstance(aggrs, (list, tuple)):
            raise ValueError(f"'aggrs' of '{self.__class__.__name__}' should "
                             f"be a list or tuple (got '{type(aggrs)}').")

        if len(aggrs) == 0:
            raise ValueError(f"'aggrs' of '{self.__class__.__name__}' should "
                             f"not be empty.")

        aggrs = [aggregation_resolver(aggr) for aggr in aggrs]
        aggr_classes = [aggr.__class__ for aggr in aggrs]
        self.aggr_names = [cls.__name__ for cls in aggr_classes]
        self.aggr_index: Dict[str, int] = {
            name: i
            for i, name in enumerate(self.aggr_names)
        }

        for cls in aggr_classes:
            if cls not in self.FUSABLE_AGGRS:
                raise ValueError(f"Received aggregation '{cls.__name__}' in "
                                 f"'{self.__class__.__name__}' which is not "
                                 f"fusable")

        self.semi_grad = False
        for aggr in aggrs:
            if hasattr(aggr, 'semi_grad'):
                self.semi_grad = self.semi_grad or aggr.semi_grad

        # Check whether we need to compute degree information:
        self.need_degree = False
        for cls in aggr_classes:
            if cls in self.DEGREE_BASED_AGGRS:
                self.need_degree = True

        # Determine which reduction to use for each aggregator:
        # An entry of `None` means that this operator re-uses intermediate
        # outputs from other aggregators.
        reduce_ops: List[Optional[str]] = []
        # Determine which `(Aggregator, index)` to use as intermediate output:
        lookup_ops: List[Optional[Tuple[str, int]]] = []

        for name in self.aggr_names:
            if name == 'MeanAggregation':
                # Directly use output of `SumAggregation`:
                if 'SumAggregation' in self.aggr_index:
                    reduce_ops.append(None)
                    lookup_ops.append((
                        'SumAggregation',
                        self.aggr_index['SumAggregation'],
                    ))
                else:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append(None)

            elif name == 'VarAggregation':
                if 'MeanAggregation' in self.aggr_index:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append((
                        'MeanAggregation',
                        self.aggr_index['MeanAggregation'],
                    ))
                elif 'SumAggregation' in self.aggr_index:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append((
                        'SumAggregation',
                        self.aggr_index['SumAggregation'],
                    ))
                else:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append(None)

            elif name == 'StdAggregation':
                # Directly use output of `VarAggregation`:
                if 'VarAggregation' in self.aggr_index:
                    reduce_ops.append(None)
                    lookup_ops.append((
                        'VarAggregation',
                        self.aggr_index['VarAggregation'],
                    ))
                elif 'MeanAggregation' in self.aggr_index:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append((
                        'MeanAggregation',
                        self.aggr_index['MeanAggregation'],
                    ))
                elif 'SumAggregation' in self.aggr_index:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append((
                        'SumAggregation',
                        self.aggr_index['SumAggregation'],
                    ))
                else:
                    reduce_ops.append(self.REDUCE[name])
                    lookup_ops.append(None)

            else:
                reduce_ops.append(self.REDUCE[name])
                lookup_ops.append(None)

        self.reduce_ops: List[Optional[str]] = reduce_ops
        self.lookup_ops: List[Optional[Tuple[str, int]]] = lookup_ops

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> List[Tensor]:

        # Assert two-dimensional input for now to simplify computation:
        # TODO refactor this to support any dimension.
        self.assert_index_present(index)
        self.assert_two_dimensional_input(x, dim)

        assert index is not None

        if dim_size is None:
            if ptr is not None:
                dim_size = ptr.numel() - 1
            else:
                dim_size = int(index.max()) + 1 if index.numel() > 0 else 0

        count: Optional[Tensor] = None
        if self.need_degree:
            count = x.new_zeros(dim_size)
            count.scatter_add_(0, index, x.new_ones(x.size(0)))
            count = count.clamp_(min=1).view(-1, 1)

        #######################################################################

        outs: List[Optional[Tensor]] = []

        # Iterate over all reduction ops to compute first results:
        for i, reduce in enumerate(self.reduce_ops):
            if reduce is None:
                outs.append(None)
                continue
            assert isinstance(reduce, str)

            if reduce == 'pow_sum':
                if self.semi_grad:
                    out = scatter(x.detach() * x.detach(), index, 0, dim_size,
                                  reduce='sum')
                else:
                    out = scatter(x * x, index, 0, dim_size, reduce='sum')
            else:
                out = scatter(x, index, 0, dim_size, reduce=reduce)

            outs.append(out)

        #######################################################################

        # Compute `MeanAggregation` first to be able to re-use it:
        i = self.aggr_index.get('MeanAggregation')
        if i is not None:
            assert count is not None

            if self.lookup_ops[i] is None:
                sum_ = outs[i]
            else:
                lookup_op = self.lookup_ops[i]
                assert lookup_op is not None
                tmp_aggr, j = lookup_op
                assert tmp_aggr == 'SumAggregation'

                sum_ = outs[j]

            assert sum_ is not None
            outs[i] = sum_ / count

        # Compute `VarAggregation` second to be able to re-use it:
        i = self.aggr_index.get('VarAggregation')
        if i is not None:
            assert count is not None

            if self.lookup_ops[i] is None:
                sum_ = scatter(x, index, 0, dim_size, reduce='sum')
                mean = sum_ / count
            else:
                lookup_op = self.lookup_ops[i]
                assert lookup_op is not None
                tmp_aggr, j = lookup_op

                if tmp_aggr == 'SumAggregation':
                    sum_ = outs[j]
                    assert sum_ is not None
                    mean = sum_ / count
                elif tmp_aggr == 'MeanAggregation':
                    mean = outs[j]
                else:
                    raise NotImplementedError

            pow_sum = outs[i]

            assert pow_sum is not None
            assert mean is not None
            outs[i] = (pow_sum / count) - (mean * mean)

        # Compute `StdAggregation` last:
        i = self.aggr_index.get('StdAggregation')
        if i is not None:
            var: Optional[Tensor] = None
            pow_sum: Optional[Tensor] = None
            mean: Optional[Tensor] = None

            if self.lookup_ops[i] is None:
                pow_sum = outs[i]
                sum_ = scatter(x, index, 0, dim_size, reduce='sum')
                assert count is not None
                mean = sum_ / count
            else:
                lookup_op = self.lookup_ops[i]
                assert lookup_op is not None
                tmp_aggr, j = lookup_op

                if tmp_aggr == 'VarAggregation':
                    var = outs[j]
                elif tmp_aggr == 'SumAggregation':
                    pow_sum = outs[i]
                    sum_ = outs[j]
                    assert sum_ is not None
                    assert count is not None
                    mean = sum_ / count
                elif tmp_aggr == 'MeanAggregation':
                    pow_sum = outs[i]
                    mean = outs[j]
                else:
                    raise NotImplementedError

            if var is None:
                assert pow_sum is not None
                assert count is not None
                assert mean is not None
                var = (pow_sum / count) - (mean * mean)

            # Allow "undefined" gradient at `sqrt(0.0)`:
            out = var.clamp(min=1e-5).sqrt()
            out = out.masked_fill(out <= math.sqrt(1e-5), 0.0)

            outs[i] = out

        #######################################################################

        vals: List[Tensor] = []
        for out in outs:
            assert out is not None
            vals.append(out)

        return vals
pytorch/torch_geometric/nn/aggr/gmt.py000066400000000000000000000064221444262304300204250ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.aggr.utils import (
    PoolingByMultiheadAttention,
    SetAttentionBlock,
)


class GraphMultisetTransformer(Aggregation):
    r"""The Graph Multiset Transformer pooling operator from the
    `"Accurate Learning of Graph Representations
    with Graph Multiset Pooling" `_ paper.

    The :class:`GraphMultisetTransformer` aggregates elements into
    :math:`k` representative elements via attention-based pooling, computes the
    interaction among them via :obj:`num_encoder_blocks` self-attention blocks,
    and finally pools the representative elements via attention-based pooling
    into a single cluster.

    Args:
        channels (int): Size of each input sample.
        k (int): Number of :math:`k` representative nodes after pooling.
        num_encoder_blocks (int, optional): Number of Set Attention Blocks
            (SABs) between the two pooling blocks. (default: :obj:`1`)
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        norm (str, optional): If set to :obj:`True`, will apply layer
            normalization. (default: :obj:`False`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(
        self,
        channels: int,
        k: int,
        num_encoder_blocks: int = 1,
        heads: int = 1,
        layer_norm: bool = False,
        dropout: float = 0.0,
    ):
        super().__init__()

        self.channels = channels
        self.k = k
        self.heads = heads
        self.layer_norm = layer_norm
        self.dropout = dropout

        self.pma1 = PoolingByMultiheadAttention(channels, k, heads, layer_norm,
                                                dropout)
        self.encoders = torch.nn.ModuleList([
            SetAttentionBlock(channels, heads, layer_norm, dropout)
            for _ in range(num_encoder_blocks)
        ])
        self.pma2 = PoolingByMultiheadAttention(channels, 1, heads, layer_norm,
                                                dropout)

    def reset_parameters(self):
        self.pma1.reset_parameters()
        for encoder in self.encoders:
            encoder.reset_parameters()
        self.pma2.reset_parameters()

    @disable_dynamic_shapes(required_args=['dim_size', 'max_num_elements'])
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:

        x, mask = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                      max_num_elements=max_num_elements)

        x = self.pma1(x, mask)

        for encoder in self.encoders:
            x = encoder(x)

        x = self.pma2(x)

        return x.squeeze(1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.channels}, '
                f'k={self.k}, heads={self.heads}, '
                f'layer_norm={self.layer_norm}, '
                f'dropout={self.dropout})')
pytorch/torch_geometric/nn/aggr/gru.py000066400000000000000000000033231444262304300204300ustar00rootroot00000000000000from typing import Optional

from torch import Tensor
from torch.nn import GRU

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.aggr import Aggregation


class GRUAggregation(Aggregation):
    r"""Performs GRU aggregation in which the elements to aggregate are
    interpreted as a sequence, as described in the `"Graph Neural Networks
    with Adaptive Readouts" `_ paper.

    .. warning::
        :class:`GRUAggregation` is not a permutation-invariant operator.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        **kwargs (optional): Additional arguments of :class:`torch.nn.GRU`.
    """
    def __init__(self, in_channels: int, out_channels: int, **kwargs):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.gru = GRU(in_channels, out_channels, batch_first=True, **kwargs)
        self.reset_parameters()

    def reset_parameters(self):
        self.gru.reset_parameters()

    @disable_dynamic_shapes(required_args=['dim_size', 'max_num_elements'])
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:

        x, _ = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                   max_num_elements=max_num_elements)

        return self.gru(x)[0][:, -1]

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/aggr/lstm.py000066400000000000000000000033471444262304300206200ustar00rootroot00000000000000from typing import Optional

from torch import Tensor
from torch.nn import LSTM

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.aggr import Aggregation


class LSTMAggregation(Aggregation):
    r"""Performs LSTM-style aggregation in which the elements to aggregate are
    interpreted as a sequence, as described in the `"Inductive Representation
    Learning on Large Graphs" `_ paper.

    .. warning::
        :class:`LSTMAggregation` is not a permutation-invariant operator.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        **kwargs (optional): Additional arguments of :class:`torch.nn.LSTM`.
    """
    def __init__(self, in_channels: int, out_channels: int, **kwargs):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.lstm = LSTM(in_channels, out_channels, batch_first=True, **kwargs)
        self.reset_parameters()

    def reset_parameters(self):
        self.lstm.reset_parameters()

    @disable_dynamic_shapes(required_args=['dim_size', 'max_num_elements'])
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:

        x, _ = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                   max_num_elements=max_num_elements)

        return self.lstm(x)[0][:, -1]

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/aggr/mlp.py000066400000000000000000000037131444262304300204260ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.nn.aggr import Aggregation


class MLPAggregation(Aggregation):
    r"""Performs MLP aggregation in which the elements to aggregate are
    flattened into a single vectorial representation, and are then processed by
    a Multi-Layer Perceptron (MLP), as described in the `"Graph Neural Networks
    with Adaptive Readouts" `_ paper.

    .. warning::
        :class:`MLPAggregation` is not a permutation-invariant operator.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        max_num_elements (int): The maximum number of elements to aggregate per
            group.
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.models.MLP`.
    """
    def __init__(self, in_channels: int, out_channels: int,
                 max_num_elements: int, **kwargs):
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.max_num_elements = max_num_elements

        from torch_geometric.nn import MLP
        self.mlp = MLP(in_channels=in_channels * max_num_elements,
                       out_channels=out_channels, **kwargs)

        self.reset_parameters()

    def reset_parameters(self):
        self.mlp.reset_parameters()

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:
        x, _ = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                   max_num_elements=self.max_num_elements)
        return self.mlp(x.view(-1, x.size(1) * x.size(2)))

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, '
                f'max_num_elements={self.max_num_elements})')
pytorch/torch_geometric/nn/aggr/multi.py000066400000000000000000000177521444262304300210000ustar00rootroot00000000000000import copy
from typing import Any, Dict, List, Optional, Union

import torch
from torch import Tensor
from torch.nn import Linear, MultiheadAttention

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.aggr.fused import FusedAggregation
from torch_geometric.nn.dense import HeteroDictLinear
from torch_geometric.nn.resolver import aggregation_resolver


class MultiAggregation(Aggregation):
    r"""Performs aggregations with one or more aggregators and combines
    aggregated results, as described in the `"Principal Neighbourhood
    Aggregation for Graph Nets" `_ and
    `"Adaptive Filters and Aggregator Fusion for Efficient Graph Convolutions"
    `_ papers.

    Args:
        aggrs (list): The list of aggregation schemes to use.
        aggrs_kwargs (dict, optional): Arguments passed to the
            respective aggregation function in case it gets automatically
            resolved. (default: :obj:`None`)
        mode (str, optional): The combine mode to use for combining
            aggregated results from multiple aggregations (:obj:`"cat"`,
            :obj:`"proj"`, :obj:`"sum"`, :obj:`"mean"`, :obj:`"max"`,
            :obj:`"min"`, :obj:`"logsumexp"`, :obj:`"std"`, :obj:`"var"`,
            :obj:`"attn"`). (default: :obj:`"cat"`)
        mode_kwargs (dict, optional): Arguments passed for the combine
            :obj:`mode`. When :obj:`"proj"` or :obj:`"attn"` is used as the
            combine :obj:`mode`, :obj:`in_channels` (int or tuple) and
            :obj:`out_channels` (int) are needed to be specified respectively
            for the size of each input sample to combine from the respective
            aggregation outputs and the size of each output sample after
            combination. When :obj:`"attn"` mode is used, :obj:`num_heads`
            (int) is needed to be specified for the number of parallel
            attention heads. (default: :obj:`None`)
    """
    fused_out_index: List[int]
    is_fused_aggr: List[bool]

    def __init__(
        self,
        aggrs: List[Union[Aggregation, str]],
        aggrs_kwargs: Optional[List[Dict[str, Any]]] = None,
        mode: Optional[str] = 'cat',
        mode_kwargs: Optional[Dict[str, Any]] = None,
    ):

        super().__init__()

        if not isinstance(aggrs, (list, tuple)):
            raise ValueError(f"'aggrs' of '{self.__class__.__name__}' should "
                             f"be a list or tuple (got '{type(aggrs)}').")

        if len(aggrs) == 0:
            raise ValueError(f"'aggrs' of '{self.__class__.__name__}' should "
                             f"not be empty.")

        if aggrs_kwargs is None:
            aggrs_kwargs = [{}] * len(aggrs)
        elif len(aggrs) != len(aggrs_kwargs):
            raise ValueError(f"'aggrs_kwargs' with invalid length passed to "
                             f"'{self.__class__.__name__}' "
                             f"(got '{len(aggrs_kwargs)}', "
                             f"expected '{len(aggrs)}'). Ensure that both "
                             f"'aggrs' and 'aggrs_kwargs' are consistent.")

        self.aggrs = torch.nn.ModuleList([
            aggregation_resolver(aggr, **aggr_kwargs)
            for aggr, aggr_kwargs in zip(aggrs, aggrs_kwargs)
        ])

        # Divide the set into fusable and non-fusable aggregations:
        fused_aggrs: List[Aggregation] = []
        self.fused_out_index: List[int] = []
        self.is_fused_aggr: List[bool] = []
        for i, aggr in enumerate(self.aggrs):
            if aggr.__class__ in FusedAggregation.FUSABLE_AGGRS:
                fused_aggrs.append(aggr)
                self.fused_out_index.append(i)
                self.is_fused_aggr.append(True)
            else:
                self.is_fused_aggr.append(False)

        if len(fused_aggrs) > 0:
            self.fused_aggr = FusedAggregation(fused_aggrs)
        else:
            self.fused_aggr = None

        self.mode = mode
        mode_kwargs = copy.copy(mode_kwargs) or {}

        self.in_channels = mode_kwargs.pop('in_channels', None)
        self.out_channels = mode_kwargs.pop('out_channels', None)

        if mode == 'proj' or mode == 'attn':
            if len(aggrs) == 1:
                raise ValueError("Multiple aggregations are required for "
                                 "'proj' or 'attn' combine mode.")

            if (self.in_channels and self.out_channels) is None:
                raise ValueError(
                    f"Combine mode '{mode}' must have `in_channels` "
                    f"and `out_channels` specified.")

            if isinstance(self.in_channels, int):
                self.in_channels = [self.in_channels] * len(aggrs)

            if mode == 'proj':
                self.lin = Linear(
                    sum(self.in_channels),
                    self.out_channels,
                    **mode_kwargs,
                )

            elif mode == 'attn':
                channels = {str(k): v for k, v, in enumerate(self.in_channels)}
                self.lin_heads = HeteroDictLinear(channels, self.out_channels)
                num_heads = mode_kwargs.pop('num_heads', 1)
                self.multihead_attn = MultiheadAttention(
                    self.out_channels,
                    num_heads,
                    **mode_kwargs,
                )

        dense_combine_modes = [
            'sum', 'mean', 'max', 'min', 'logsumexp', 'std', 'var'
        ]
        if mode in dense_combine_modes:
            self.dense_combine = getattr(torch, mode)

    def reset_parameters(self):
        for aggr in self.aggrs:
            aggr.reset_parameters()
        if self.mode == 'proj':
            self.lin.reset_parameters()
        if self.mode == 'attn':
            self.lin_heads.reset_parameters()
            self.multihead_attn._reset_parameters()

    def get_out_channels(self, in_channels: int) -> int:
        if self.out_channels is not None:
            return self.out_channels
        # TODO Support having customized `out_channels` in each aggregation.
        if self.mode == 'cat':
            return in_channels * len(self.aggrs)
        return in_channels

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        # `FusedAggregation` is currently limited to two-dimensional inputs:
        if index is None or x.dim() != 2 or self.fused_aggr is None:
            outs = [aggr(x, index, ptr, dim_size, dim) for aggr in self.aggrs]
            return self.combine(outs)

        outs: List[Tensor] = [x] * len(self.aggrs)  # Fill with dummy tensors.

        fused_outs = self.fused_aggr(x, index, ptr, dim_size, dim)
        for i, out in zip(self.fused_out_index, fused_outs):
            outs[i] = out

        for i, aggr in enumerate(self.aggrs):
            if not self.is_fused_aggr[i]:
                outs[i] = aggr(x, index, ptr, dim_size, dim)

        return self.combine(outs)

    def combine(self, inputs: List[Tensor]) -> Tensor:
        if len(inputs) == 1:
            return inputs[0]

        if self.mode == 'cat':
            return torch.cat(inputs, dim=-1)

        if hasattr(self, 'lin'):
            return self.lin(torch.cat(inputs, dim=-1))

        if hasattr(self, 'multihead_attn'):
            x_dict = {str(k): v for k, v, in enumerate(inputs)}
            x_dict = self.lin_heads(x_dict)
            xs = [x_dict[str(key)] for key in range(len(inputs))]
            x = torch.stack(xs, dim=0)
            attn_out, _ = self.multihead_attn(x, x, x)
            return torch.mean(attn_out, dim=0)

        if hasattr(self, 'dense_combine'):
            out = self.dense_combine(torch.stack(inputs, dim=0), dim=0)
            return out if isinstance(out, Tensor) else out[0]

        raise ValueError(f"Combine mode '{self.mode}' is not supported.")

    def __repr__(self) -> str:
        aggrs = ',\n'.join([f'  {aggr}' for aggr in self.aggrs]) + ',\n'
        return f'{self.__class__.__name__}([\n{aggrs}], mode={self.mode})'
pytorch/torch_geometric/nn/aggr/quantile.py000066400000000000000000000140361444262304300214600ustar00rootroot00000000000000from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation


class QuantileAggregation(Aggregation):
    r"""An aggregation operator that returns the feature-wise :math:`q`-th
    quantile of a set :math:`\mathcal{X}`. That is, for every feature
    :math:`d`, it computes

    .. math::
        {\mathrm{Q}_q(\mathcal{X})}_d = \begin{cases}
            x_{\pi_i,d} & i = q \cdot n, \\
            f(x_{\pi_i,d}, x_{\pi_{i+1},d}) & i < q \cdot n < i + 1,\\
        \end{cases}

    where :math:`x_{\pi_1,d} \le \dots \le x_{\pi_i,d} \le \dots \le
    x_{\pi_n,d}` and :math:`f(a, b)` is an interpolation
    function defined by :obj:`interpolation`.

    Args:
        q (float or list): The quantile value(s) :math:`q`. Can be a scalar or
            a list of scalars in the range :math:`[0, 1]`. If more than a
            quantile is passed, the results are concatenated.
        interpolation (str): Interpolation method applied if the quantile point
            :math:`q\cdot n` lies between two values
            :math:`a \le b`. Can be one of the following:

            * :obj:`"lower"`: Returns the one with lowest value.

            * :obj:`"higher"`: Returns the one with highest value.

            * :obj:`"midpoint"`: Returns the average of the two values.

            * :obj:`"nearest"`: Returns the one whose index is nearest to the
              quantile point.

            * :obj:`"linear"`: Returns a linear combination of the two
              elements, defined as
              :math:`f(a, b) = a + (b - a)\cdot(q\cdot n - i)`.

            (default: :obj:`"linear"`)
        fill_value (float, optional): The default value in the case no entry is
            found for a given index (default: :obj:`0.0`).
    """
    interpolations = {'linear', 'lower', 'higher', 'nearest', 'midpoint'}

    def __init__(self, q: Union[float, List[float]],
                 interpolation: str = 'linear', fill_value: float = 0.0):
        super().__init__()

        qs = [q] if not isinstance(q, (list, tuple)) else q
        if len(qs) == 0:
            raise ValueError("Provide at least one quantile value for `q`.")
        if not all(0. <= quantile <= 1. for quantile in qs):
            raise ValueError("`q` must be in the range [0, 1].")
        if interpolation not in self.interpolations:
            raise ValueError(f"Invalid interpolation method "
                             f"got ('{interpolation}')")

        self._q = q
        self.register_buffer('q', torch.Tensor(qs).view(-1, 1))
        self.interpolation = interpolation
        self.fill_value = fill_value

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        dim = x.dim() + dim if dim < 0 else dim

        self.assert_index_present(index)
        assert index is not None  # Required for TorchScript.

        count = torch.bincount(index, minlength=dim_size or 0)
        cumsum = torch.cumsum(count, dim=0) - count

        # In case there exists dangling indices (`dim_size > index.max()`), we
        # need to clamp them to prevent out-of-bound issues:
        if dim_size is not None:
            cumsum = cumsum.clamp(max=x.size(dim) - 1)

        q_point = self.q * (count - 1) + cumsum
        q_point = q_point.t().reshape(-1)

        shape = [1] * x.dim()
        shape[dim] = -1
        index = index.view(shape).expand_as(x)

        # Two sorts: the first one on the value,
        # the second (stable) on the indices:
        x, x_perm = torch.sort(x, dim=dim)
        index = index.take_along_dim(x_perm, dim=dim)
        index, index_perm = torch.sort(index, dim=dim, stable=True)
        x = x.take_along_dim(index_perm, dim=dim)

        # Compute the quantile interpolations:
        if self.interpolation == 'lower':
            quantile = x.index_select(dim, q_point.floor().long())
        elif self.interpolation == 'higher':
            quantile = x.index_select(dim, q_point.ceil().long())
        elif self.interpolation == 'nearest':
            quantile = x.index_select(dim, q_point.round().long())
        else:
            l_quant = x.index_select(dim, q_point.floor().long())
            r_quant = x.index_select(dim, q_point.ceil().long())

            if self.interpolation == 'linear':
                q_frac = q_point.frac().view(shape)
                quantile = l_quant + (r_quant - l_quant) * q_frac
            else:  # 'midpoint'
                quantile = 0.5 * l_quant + 0.5 * r_quant

        # If the number of elements is zero, fill with pre-defined value:
        repeats = self.q.numel()
        mask = (count == 0).repeat_interleave(
            repeats, output_size=repeats * count.numel()).view(shape)
        out = quantile.masked_fill(mask, self.fill_value)

        if self.q.numel() > 1:
            shape = list(out.shape)
            shape = (shape[:dim] + [shape[dim] // self.q.numel(), -1] +
                     shape[dim + 2:])
            out = out.view(shape)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(q={self._q})')


class MedianAggregation(QuantileAggregation):
    r"""An aggregation operator that returns the feature-wise median of a set.
    That is, for every feature :math:`d`, it computes

    .. math::
        {\mathrm{median}(\mathcal{X})}_d = x_{\pi_i,d}

    where :math:`x_{\pi_1,d} \le x_{\pi_2,d} \le \dots \le
    x_{\pi_n,d}` and :math:`i = \lfloor \frac{n}{2} \rfloor`.

    .. note::
        If the median lies between two values, the lowest one is returned.
        To compute the midpoint (or other kind of interpolation) of the two
        values, use :class:`QuantileAggregation` instead.

    Args:
        fill_value (float, optional): The default value in the case no entry is
            found for a given index (default: :obj:`0.0`).
    """
    def __init__(self, fill_value: float = 0.0):
        super().__init__(0.5, 'lower', fill_value)

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}()"
pytorch/torch_geometric/nn/aggr/scaler.py000066400000000000000000000110421444262304300211010ustar00rootroot00000000000000from typing import Any, Dict, List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation, MultiAggregation
from torch_geometric.nn.resolver import aggregation_resolver as aggr_resolver
from torch_geometric.utils import degree


class DegreeScalerAggregation(Aggregation):
    r"""Combines one or more aggregators and transforms its output with one or
    more scalers as introduced in the `"Principal Neighbourhood Aggregation for
    Graph Nets" `_ paper.
    The scalers are normalised by the in-degree of the training set and so must
    be provided at time of construction.
    See :class:`torch_geometric.nn.conv.PNAConv` for more information.

    Args:
        aggr (str or [str] or Aggregation): The aggregation scheme to use.
            See :class:`~torch_geometric.nn.conv.MessagePassing` for more
            information.
        scaler (str or list): Set of scaling function identifiers, namely one
            or more of :obj:`"identity"`, :obj:`"amplification"`,
            :obj:`"attenuation"`, :obj:`"linear"` and :obj:`"inverse_linear"`.
        deg (Tensor): Histogram of in-degrees of nodes in the training set,
            used by scalers to normalize.
        train_norm (bool, optional): Whether normalization parameters
            are trainable. (default: :obj:`False`)
        aggr_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective aggregation function in case it gets automatically
            resolved. (default: :obj:`None`)
    """
    def __init__(
        self,
        aggr: Union[str, List[str], Aggregation],
        scaler: Union[str, List[str]],
        deg: Tensor,
        train_norm: bool = False,
        aggr_kwargs: Optional[List[Dict[str, Any]]] = None,
    ):
        super().__init__()

        if isinstance(aggr, (str, Aggregation)):
            self.aggr = aggr_resolver(aggr, **(aggr_kwargs or {}))
        elif isinstance(aggr, (tuple, list)):
            self.aggr = MultiAggregation(aggr, aggr_kwargs)
        else:
            raise ValueError(f"Only strings, list, tuples and instances of"
                             f"`torch_geometric.nn.aggr.Aggregation` are "
                             f"valid aggregation schemes (got '{type(aggr)}')")

        self.scaler = [scaler] if isinstance(aggr, str) else scaler

        deg = deg.to(torch.float)
        N = int(deg.sum())
        bin_degree = torch.arange(deg.numel(), device=deg.device)

        self.init_avg_deg_lin = float((bin_degree * deg).sum()) / N
        self.init_avg_deg_log = float(((bin_degree + 1).log() * deg).sum()) / N

        if train_norm:
            self.avg_deg_lin = torch.nn.Parameter(torch.Tensor(1))
            self.avg_deg_log = torch.nn.Parameter(torch.Tensor(1))
        else:
            self.register_buffer('avg_deg_lin', torch.Tensor(1))
            self.register_buffer('avg_deg_log', torch.Tensor(1))

        self.reset_parameters()

    def reset_parameters(self):
        self.avg_deg_lin.data.fill_(self.init_avg_deg_lin)
        self.avg_deg_log.data.fill_(self.init_avg_deg_log)

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        # TODO Currently, `degree` can only operate on `index`:
        self.assert_index_present(index)

        out = self.aggr(x, index, ptr, dim_size, dim)

        assert index is not None
        deg = degree(index, num_nodes=dim_size, dtype=out.dtype)
        size = [1] * len(out.size())
        size[dim] = -1
        deg = deg.view(size)

        outs = []
        for scaler in self.scaler:
            if scaler == 'identity':
                out_scaler = out
            elif scaler == 'amplification':
                out_scaler = out * (torch.log(deg + 1) / self.avg_deg_log)
            elif scaler == 'attenuation':
                # Clamp minimum degree to one to avoid dividing by zero:
                out_scaler = out * (self.avg_deg_log /
                                    torch.log(deg.clamp(min=1) + 1))
            elif scaler == 'linear':
                out_scaler = out * (deg / self.avg_deg_lin)
            elif scaler == 'inverse_linear':
                # Clamp minimum degree to one to avoid dividing by zero:
                out_scaler = out * (self.avg_deg_lin / deg.clamp(min=1))
            else:
                raise ValueError(f"Unknown scaler '{scaler}'")
            outs.append(out_scaler)

        return torch.cat(outs, dim=-1) if len(outs) > 1 else outs[0]
pytorch/torch_geometric/nn/aggr/set2set.py000066400000000000000000000047251444262304300212330ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.utils import softmax


class Set2Set(Aggregation):
    r"""The Set2Set aggregation operator based on iterative content-based
    attention, as described in the `"Order Matters: Sequence to sequence for
    Sets" `_ paper

    .. math::
        \mathbf{q}_t &= \mathrm{LSTM}(\mathbf{q}^{*}_{t-1})

        \alpha_{i,t} &= \mathrm{softmax}(\mathbf{x}_i \cdot \mathbf{q}_t)

        \mathbf{r}_t &= \sum_{i=1}^N \alpha_{i,t} \mathbf{x}_i

        \mathbf{q}^{*}_t &= \mathbf{q}_t \, \Vert \, \mathbf{r}_t,

    where :math:`\mathbf{q}^{*}_T` defines the output of the layer with twice
    the dimensionality as the input.

    Args:
        in_channels (int): Size of each input sample.
        processing_steps (int): Number of iterations :math:`T`.
        **kwargs (optional): Additional arguments of :class:`torch.nn.LSTM`.
    """
    def __init__(self, in_channels: int, processing_steps: int, **kwargs):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = 2 * in_channels
        self.processing_steps = processing_steps
        self.lstm = torch.nn.LSTM(self.out_channels, in_channels, **kwargs)
        self.reset_parameters()

    def reset_parameters(self):
        self.lstm.reset_parameters()

    def forward(self, x: Tensor, index: Optional[Tensor] = None,
                ptr: Optional[Tensor] = None, dim_size: Optional[int] = None,
                dim: int = -2) -> Tensor:

        # TODO Currently, `to_dense_batch` can only operate on `index`:
        self.assert_index_present(index)
        self.assert_two_dimensional_input(x, dim)

        h = (x.new_zeros((self.lstm.num_layers, dim_size, x.size(-1))),
             x.new_zeros((self.lstm.num_layers, dim_size, x.size(-1))))
        q_star = x.new_zeros(dim_size, self.out_channels)

        for _ in range(self.processing_steps):
            q, h = self.lstm(q_star.unsqueeze(0), h)
            q = q.view(dim_size, self.in_channels)
            e = (x * q[index]).sum(dim=-1, keepdim=True)
            a = softmax(e, index, ptr, dim_size, dim)
            r = self.reduce(a * x, index, ptr, dim_size, dim, reduce='sum')
            q_star = torch.cat([q, r], dim=-1)

        return q_star

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/aggr/set_transformer.py000066400000000000000000000072131444262304300230520ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.aggr.utils import (
    PoolingByMultiheadAttention,
    SetAttentionBlock,
)


class SetTransformerAggregation(Aggregation):
    r"""Performs "Set Transformer" aggregation in which the elements to
    aggregate are processed by multi-head attention blocks, as described in
    the `"Graph Neural Networks with Adaptive Readouts"
    `_ paper.

    Args:
        channels (int): Size of each input sample.
        num_seed_points (int, optional): Number of seed points.
            (default: :obj:`1`)
        num_encoder_blocks (int, optional): Number of Set Attention Blocks
            (SABs) in the encoder. (default: :obj:`1`).
        num_decoder_blocks (int, optional): Number of Set Attention Blocks
            (SABs) in the decoder. (default: :obj:`1`).
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the seed embeddings
            are averaged instead of concatenated. (default: :obj:`True`)
        norm (str, optional): If set to :obj:`True`, will apply layer
            normalization. (default: :obj:`False`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(
        self,
        channels: int,
        num_seed_points: int = 1,
        num_encoder_blocks: int = 1,
        num_decoder_blocks: int = 1,
        heads: int = 1,
        concat: bool = True,
        layer_norm: bool = False,
        dropout: float = 0.0,
    ):
        super().__init__()

        self.channels = channels
        self.num_seed_points = num_seed_points
        self.heads = heads
        self.concat = concat
        self.layer_norm = layer_norm
        self.dropout = dropout

        self.encoders = torch.nn.ModuleList([
            SetAttentionBlock(channels, heads, layer_norm, dropout)
            for _ in range(num_encoder_blocks)
        ])

        self.pma = PoolingByMultiheadAttention(channels, num_seed_points,
                                               heads, layer_norm, dropout)

        self.decoders = torch.nn.ModuleList([
            SetAttentionBlock(channels, heads, layer_norm, dropout)
            for _ in range(num_decoder_blocks)
        ])

    def reset_parameters(self):
        for encoder in self.encoders:
            encoder.reset_parameters()
        self.pma.reset_parameters()
        for decoder in self.decoders:
            decoder.reset_parameters()

    @disable_dynamic_shapes(required_args=['dim_size', 'max_num_elements'])
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:

        x, mask = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                      max_num_elements=max_num_elements)

        for encoder in self.encoders:
            x = encoder(x, mask)

        x = self.pma(x, mask)

        for decoder in self.decoders:
            x = decoder(x)

        return x.flatten(1, 2) if self.concat else x.mean(dim=1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.channels}, '
                f'num_seed_points={self.num_seed_points}, '
                f'heads={self.heads}, '
                f'layer_norm={self.layer_norm}, '
                f'dropout={self.dropout})')
pytorch/torch_geometric/nn/aggr/sort.py000066400000000000000000000040151444262304300206210ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.aggr import Aggregation


class SortAggregation(Aggregation):
    r"""The pooling operator from the `"An End-to-End Deep Learning
    Architecture for Graph Classification"
    `_ paper,
    where node features are sorted in descending order based on their last
    feature channel. The first :math:`k` nodes form the output of the layer.

    Args:
        k (int): The number of nodes to hold for each graph.
    """
    def __init__(self, k: int):
        super().__init__()
        self.k = k

    @disable_dynamic_shapes(required_args=['dim_size', 'max_num_elements'])
    def forward(
        self,
        x: Tensor,
        index: Optional[Tensor] = None,
        ptr: Optional[Tensor] = None,
        dim_size: Optional[int] = None,
        dim: int = -2,
        max_num_elements: Optional[int] = None,
    ) -> Tensor:

        fill_value = x.detach().min() - 1
        batch_x, _ = self.to_dense_batch(x, index, ptr, dim_size, dim,
                                         fill_value=fill_value,
                                         max_num_elements=max_num_elements)
        B, N, D = batch_x.size()

        _, perm = batch_x[:, :, -1].sort(dim=-1, descending=True)
        arange = torch.arange(B, dtype=torch.long, device=perm.device) * N
        perm = perm + arange.view(-1, 1)

        batch_x = batch_x.view(B * N, D)
        batch_x = batch_x[perm]
        batch_x = batch_x.view(B, N, D)

        if N >= self.k:
            batch_x = batch_x[:, :self.k].contiguous()
        else:
            expand_batch_x = batch_x.new_full((B, self.k - N, D), fill_value)
            batch_x = torch.cat([batch_x, expand_batch_x], dim=1)

        batch_x[batch_x == fill_value] = 0
        x = batch_x.view(B, self.k * D)

        return x

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(k={self.k})')
pytorch/torch_geometric/nn/aggr/utils.py000066400000000000000000000202021444262304300207660ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import LayerNorm, Linear, MultiheadAttention, Parameter


class MultiheadAttentionBlock(torch.nn.Module):
    r"""The Multihead Attention Block (MAB) from the `"Set Transformer: A
    Framework for Attention-based Permutation-Invariant Neural Networks"
    `_ paper

    .. math::

        \mathrm{MAB}(\mathbf{x}, \mathbf{y}) &= \mathrm{LayerNorm}(\mathbf{h} +
        \mathbf{W} \mathbf{h})

        \mathbf{h} &= \mathrm{LayerNorm}(\mathbf{x} +
        \mathrm{Multihead}(\mathbf{x}, \mathbf{y}, \mathbf{y}))

    Args:
        channels (int): Size of each input sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        norm (str, optional): If set to :obj:`False`, will not apply layer
            normalization. (default: :obj:`True`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(self, channels: int, heads: int = 1, layer_norm: bool = True,
                 dropout: float = 0.0):
        super().__init__()

        self.channels = channels
        self.heads = heads
        self.dropout = dropout

        self.attn = MultiheadAttention(
            channels,
            heads,
            batch_first=True,
            dropout=dropout,
        )
        self.lin = Linear(channels, channels)
        self.layer_norm1 = LayerNorm(channels) if layer_norm else None
        self.layer_norm2 = LayerNorm(channels) if layer_norm else None

    def reset_parameters(self):
        self.attn._reset_parameters()
        self.lin.reset_parameters()
        if self.layer_norm1 is not None:
            self.layer_norm1.reset_parameters()
        if self.layer_norm2 is not None:
            self.layer_norm2.reset_parameters()

    def forward(self, x: Tensor, y: Tensor, x_mask: Optional[Tensor] = None,
                y_mask: Optional[Tensor] = None) -> Tensor:
        """"""
        if y_mask is not None:
            y_mask = ~y_mask

        out, _ = self.attn(x, y, y, y_mask, need_weights=False)

        if x_mask is not None:
            out[~x_mask] = 0.

        out = out + x

        if self.layer_norm1 is not None:
            out = self.layer_norm1(out)

        out = out + self.lin(out).relu()

        if self.layer_norm2 is not None:
            out = self.layer_norm2(out)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.channels}, '
                f'heads={self.heads}, '
                f'layer_norm={self.layer_norm1 is not None}, '
                f'dropout={self.dropout})')


class SetAttentionBlock(torch.nn.Module):
    r"""The Set Attention Block (SAB) from the `"Set Transformer: A
    Framework for Attention-based Permutation-Invariant Neural Networks"
    `_ paper

    .. math::

        \mathrm{SAB}(\mathbf{X}) = \mathrm{MAB}(\mathbf{x}, \mathbf{y})

    Args:
        channels (int): Size of each input sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        norm (str, optional): If set to :obj:`False`, will not apply layer
            normalization. (default: :obj:`True`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(self, channels: int, heads: int = 1, layer_norm: bool = True,
                 dropout: float = 0.0):
        super().__init__()
        self.mab = MultiheadAttentionBlock(channels, heads, layer_norm,
                                           dropout)

    def reset_parameters(self):
        self.mab.reset_parameters()

    def forward(self, x: Tensor, mask: Optional[Tensor] = None) -> Tensor:
        return self.mab(x, x, mask, mask)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.mab.channels}, '
                f'heads={self.mab.heads}, '
                f'layer_norm={self.mab.layer_norm1 is not None}, '
                f'dropout={self.mab.dropout})')


class InducedSetAttentionBlock(torch.nn.Module):
    r"""The Induced Set Attention Block (SAB) from the `"Set Transformer: A
    Framework for Attention-based Permutation-Invariant Neural Networks"
    `_ paper

    .. math::

        \mathrm{ISAB}(\mathbf{X}) &= \mathrm{MAB}(\mathbf{x}, \mathbf{h})

        \mathbf{h} &= \mathrm{MAB}(\mathbf{I}, \mathbf{x})

    where :math:`\mathbf{I}` denotes :obj:`num_induced_points` learnable
    vectors.

    Args:
        channels (int): Size of each input sample.
        num_induced_points (int): Number of induced points.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        norm (str, optional): If set to :obj:`False`, will not apply layer
            normalization. (default: :obj:`True`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(self, channels: int, num_induced_points: int, heads: int = 1,
                 layer_norm: bool = True, dropout: float = 0.0):
        super().__init__()
        self.ind = Parameter(torch.Tensor(1, num_induced_points, channels))
        self.mab1 = MultiheadAttentionBlock(channels, heads, layer_norm,
                                            dropout)
        self.mab2 = MultiheadAttentionBlock(channels, heads, layer_norm,
                                            dropout)
        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.ind)
        self.mab1.reset_parameters()
        self.mab2.reset_parameters()

    def forward(self, x: Tensor, mask: Optional[Tensor] = None) -> Tensor:
        h = self.mab1(self.ind.expand(x.size(0), -1, -1), x, y_mask=mask)
        return self.mab2(x, h, x_mask=mask)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.ind.size(2)}, '
                f'num_induced_points={self.ind.size(1)}, '
                f'heads={self.mab1.heads}, '
                f'layer_norm={self.mab1.layer_norm1 is not None}, '
                f'dropout={self.mab1.dropout})')


class PoolingByMultiheadAttention(torch.nn.Module):
    r"""The Pooling by Multihead Attention (PMA) layer from the `"Set
    Transformer: A Framework for Attention-based Permutation-Invariant Neural
    Networks" `_ paper

    .. math::

        \mathrm{PMA}(\mathbf{X}) = \mathrm{MAB}(\mathbf{S}, \mathbf{x})

    where :math:`\mathbf{S}` denotes :obj:`num_seed_points` learnable vectors.

    Args:
        channels (int): Size of each input sample.
        num_seed_points (int, optional): Number of seed points.
            (default: :obj:`1`)
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        norm (str, optional): If set to :obj:`False`, will not apply layer
            normalization. (default: :obj:`True`)
        dropout (float, optional): Dropout probability of attention weights.
            (default: :obj:`0`)
    """
    def __init__(self, channels: int, num_seed_points: int = 1, heads: int = 1,
                 layer_norm: bool = True, dropout: float = 0.0):
        super().__init__()
        self.lin = Linear(channels, channels)
        self.seed = Parameter(torch.Tensor(1, num_seed_points, channels))
        self.mab = MultiheadAttentionBlock(channels, heads, layer_norm,
                                           dropout)
        self.reset_parameters()

    def reset_parameters(self):
        self.lin.reset_parameters()
        torch.nn.init.xavier_uniform_(self.seed)
        self.mab.reset_parameters()

    def forward(self, x: Tensor, mask: Optional[Tensor] = None) -> Tensor:
        x = self.lin(x).relu()
        return self.mab(self.seed.expand(x.size(0), -1, -1), x, y_mask=mask)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.seed.size(2)}, '
                f'num_seed_points={self.seed.size(1)}, '
                f'heads={self.mab.heads}, '
                f'layer_norm={self.mab.layer_norm1 is not None}, '
                f'dropout={self.mab.dropout})')
pytorch/torch_geometric/nn/attention/000077500000000000000000000000001444262304300203455ustar00rootroot00000000000000pytorch/torch_geometric/nn/attention/__init__.py000066400000000000000000000001141444262304300224520ustar00rootroot00000000000000from .performer import PerformerAttention

__all__ = ['PerformerAttention']
pytorch/torch_geometric/nn/attention/performer.py000066400000000000000000000170071444262304300227250ustar00rootroot00000000000000import math
from typing import Callable, Optional

import torch
from torch import Tensor


def _orthogonal_matrix(dim: int) -> Tensor:
    r"""Get an orthogonal matrix by applying QR decomposition."""
    # Random matrix from normal distribution
    mat = torch.randn((dim, dim))
    # QR decomposition to two orthogonal matrices
    q, _ = torch.linalg.qr(mat.cpu(), mode='reduced')
    return q.t()


def orthogonal_matrix(num_rows: int, num_cols: int) -> Tensor:
    r"""Generate an orthogonal matrix with `num_rows` rows
    and `num_cols` columns.
    """
    num_full_blocks = int(num_rows / num_cols)
    blocks = []
    for _ in range(num_full_blocks):
        q = _orthogonal_matrix(num_cols)
        blocks.append(q)
    remain_rows = num_rows - num_full_blocks * num_cols
    if remain_rows > 0:
        q = _orthogonal_matrix(num_cols)
        blocks.append(q[:remain_rows])
    mat = torch.cat(blocks)
    # multiplier = torch.randn((num_rows, num_cols)).norm(dim=1)
    # scaler = torch.diag(multiplier)
    # mat = scaler @ mat
    return mat


def linear_attention(q: Tensor, k: Tensor, v: Tensor) -> Tensor:
    r"""Efficient attention mechanism from the
    `"Rethinking Attention with Performers"
    `_ paper.

    .. math::
        \mathbf{\hat{D}}^{-1}(\mathbf{Q}'((\mathbf{K}')^{\top} \mathbf{V}))

    """
    k_contract = k.sum(dim=-2)
    D_inv = 1.0 / torch.einsum('...Lr,...r->...L', q, k_contract)
    kv = torch.einsum('...Lr,...Ld->...rd', k, v)
    qkv = torch.einsum('...Lr,...rd->...Ld', q, kv)
    out = torch.einsum('...L,...Ld->...Ld', D_inv, qkv)
    return out


def generalized_kernel(x: Tensor, mat: Tensor,
                       kernel: Callable = torch.nn.ReLU(),
                       epsilon: float = 0.001) -> Tensor:
    r"""Apply generalized kernelizable attention with
    kernel functions such as the ReLU.
    """
    num_batches, num_heads, *_ = x.shape
    # Expand projection matrix to number of batches and number of heads
    projection = mat.expand(num_batches, num_heads, *mat.shape)
    # "Inner" product x with projection matrix
    x = torch.einsum('...id,...jd->...ij', x, projection)
    out = kernel(x) + epsilon
    return out


class PerformerProjection(torch.nn.Module):
    r"""The fast attention that uses a projection matrix
    from the `"Rethinking Attention with Performers"
    `_ paper. This class
    projects :math:`\mathbf{Q}` and :math:`\mathbf{K}` matrices
    with specified kernel.

    Args:
        num_cols (int): Projection matrix number of columns.
        kernel (Callable, optional): Kernels for generalized attention.
            If not specified, `ReLU` kernel will be used.
            (default: :obj:`torch.nn.ReLU()`)
    """
    def __init__(self, num_cols: int, kernel: Callable = torch.nn.ReLU()):
        super().__init__()
        num_rows = int(num_cols * math.log(num_cols))
        self.num_rows = num_rows
        self.num_cols = num_cols
        # Generate an orthogonal projection matrix
        # with the shape (num_rows, num_cols)
        projection_matrix = orthogonal_matrix(self.num_rows, self.num_cols)
        self.register_buffer('projection_matrix', projection_matrix)
        assert kernel is not None
        self.kernel = kernel

    def forward(self, q: Tensor, k: Tensor, v: Tensor) -> Tensor:
        q = generalized_kernel(q, self.projection_matrix, self.kernel)
        k = generalized_kernel(k, self.projection_matrix, self.kernel)
        out = linear_attention(q, k, v)
        return out


class PerformerAttention(torch.nn.Module):
    r"""The linear scaled attention mechanism from the
    `"Rethinking Attention with Performers"
    `_ paper.

    Args:
        channels (int): Size of each input sample.
        heads (int, optional): Number of parallel attention heads.
        head_channels (int, optional): Size of each attention head.
            (default: :obj:`64.`)
        kernel (Callable, optional): Kernels for generalized attention.
            If not specified, `ReLU` kernel will be used.
            (default: :obj:`torch.nn.ReLU()`)
        qkv_bias (bool, optional): If specified, add bias to query, key
            and value in the self attention. (default: :obj:`False`)
        attn_out_bias (bool, optional): If specified, add bias to the
            attention output. (default: :obj:`True`)
        dropout (float, optional): Dropout probability of the final
            attention output. (default: :obj:`0.0`)

    """
    def __init__(
        self,
        channels: int,
        heads: int,
        head_channels: int = 64,
        kernel: Callable = torch.nn.ReLU(),
        qkv_bias: bool = False,
        attn_out_bias: bool = True,
        dropout: float = 0.0,
    ):
        super().__init__()
        assert channels % heads == 0
        if head_channels is None:
            head_channels = channels // heads

        self.heads = heads
        self.head_channels = head_channels
        self.kernel = kernel
        self.fast_attn = PerformerProjection(head_channels, kernel)

        inner_channels = head_channels * heads
        self.q = torch.nn.Linear(channels, inner_channels, bias=qkv_bias)
        self.k = torch.nn.Linear(channels, inner_channels, bias=qkv_bias)
        self.v = torch.nn.Linear(channels, inner_channels, bias=qkv_bias)
        self.attn_out = torch.nn.Linear(inner_channels, channels,
                                        bias=attn_out_bias)
        self.dropout = torch.nn.Dropout(dropout)

    def forward(self, x: Tensor, mask: Optional[Tensor] = None) -> Tensor:
        r"""

        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
        """
        B, N, *_ = x.shape
        q, k, v = self.q(x), self.k(x), self.v(x)
        # Reshape and permute q, k and v to proper shape
        # (B, N, num_heads * head_channels) to (b, num_heads, n, head_channels)
        q, k, v = map(
            lambda t: t.reshape(B, N, self.heads, self.head_channels).permute(
                0, 2, 1, 3), (q, k, v))
        if mask is not None:
            mask = mask[:, None, :, None]
            v.masked_fill_(~mask, 0.)
        out = self.fast_attn(q, k, v)
        out = out.permute(0, 2, 1, 3).reshape(B, N, -1)
        out = self.attn_out(out)
        out = self.dropout(out)
        return out

    @torch.no_grad()
    def redraw_projection_matrix(self):
        r"""As described in the paper, periodically redraw
        examples to improve overall approximation of attention."""
        num_rows = self.fast_attn.num_rows
        num_cols = self.fast_attn.num_cols
        projection_matrix = orthogonal_matrix(num_rows, num_cols)
        self.fast_attn.projection_matrix.copy_(projection_matrix)
        del projection_matrix

    def _reset_parameters(self):
        self.q.reset_parameters()
        self.k.reset_parameters()
        self.v.reset_parameters()
        self.attn_out.reset_parameters()
        self.redraw_projection_matrix()

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'heads={self.heads}, '
                f'head_channels={self.head_channels} '
                f'kernel={self.kernel})')
pytorch/torch_geometric/nn/conv/000077500000000000000000000000001444262304300173055ustar00rootroot00000000000000pytorch/torch_geometric/nn/conv/__init__.py000066400000000000000000000065271444262304300214300ustar00rootroot00000000000000from .message_passing import MessagePassing
from .simple_conv import SimpleConv
from .gcn_conv import GCNConv
from .cheb_conv import ChebConv
from .sage_conv import SAGEConv
from .cugraph.sage_conv import CuGraphSAGEConv
from .graph_conv import GraphConv
from .gravnet_conv import GravNetConv
from .gated_graph_conv import GatedGraphConv
from .res_gated_graph_conv import ResGatedGraphConv
from .gat_conv import GATConv
from .cugraph.gat_conv import CuGraphGATConv
from .fused_gat_conv import FusedGATConv
from .gatv2_conv import GATv2Conv
from .transformer_conv import TransformerConv
from .agnn_conv import AGNNConv
from .tag_conv import TAGConv
from .gin_conv import GINConv, GINEConv
from .arma_conv import ARMAConv
from .sg_conv import SGConv
from .appnp import APPNP
from .mf_conv import MFConv
from .rgcn_conv import RGCNConv, FastRGCNConv
from .cugraph.rgcn_conv import CuGraphRGCNConv
from .rgat_conv import RGATConv
from .signed_conv import SignedConv
from .dna_conv import DNAConv
from .point_conv import PointNetConv
from .gmm_conv import GMMConv
from .spline_conv import SplineConv
from .nn_conv import NNConv
from .cg_conv import CGConv
from .edge_conv import EdgeConv, DynamicEdgeConv
from .x_conv import XConv
from .ppf_conv import PPFConv
from .feast_conv import FeaStConv
from .point_transformer_conv import PointTransformerConv
from .hypergraph_conv import HypergraphConv
from .le_conv import LEConv
from .pna_conv import PNAConv
from .cluster_gcn_conv import ClusterGCNConv
from .gen_conv import GENConv
from .gcn2_conv import GCN2Conv
from .pan_conv import PANConv
from .wl_conv import WLConv
from .wl_conv_continuous import WLConvContinuous
from .film_conv import FiLMConv
from .supergat_conv import SuperGATConv
from .fa_conv import FAConv
from .eg_conv import EGConv
from .pdn_conv import PDNConv
from .general_conv import GeneralConv
from .hgt_conv import HGTConv
from .heat_conv import HEATConv
from .hetero_conv import HeteroConv
from .han_conv import HANConv
from .lg_conv import LGConv
from .ssg_conv import SSGConv
from .point_gnn_conv import PointGNNConv
from .gps_conv import GPSConv
from .antisymmetric_conv import AntiSymmetricConv
from .dir_gnn_conv import DirGNNConv

__all__ = [
    'MessagePassing',
    'SimpleConv',
    'GCNConv',
    'ChebConv',
    'SAGEConv',
    'CuGraphSAGEConv',
    'GraphConv',
    'GravNetConv',
    'GatedGraphConv',
    'ResGatedGraphConv',
    'GATConv',
    'CuGraphGATConv',
    'FusedGATConv',
    'GATv2Conv',
    'TransformerConv',
    'AGNNConv',
    'TAGConv',
    'GINConv',
    'GINEConv',
    'ARMAConv',
    'SGConv',
    'SSGConv',
    'APPNP',
    'MFConv',
    'RGCNConv',
    'FastRGCNConv',
    'CuGraphRGCNConv',
    'RGATConv',
    'SignedConv',
    'DNAConv',
    'PointNetConv',
    'GMMConv',
    'SplineConv',
    'NNConv',
    'CGConv',
    'EdgeConv',
    'DynamicEdgeConv',
    'XConv',
    'PPFConv',
    'FeaStConv',
    'PointTransformerConv',
    'HypergraphConv',
    'LEConv',
    'PNAConv',
    'ClusterGCNConv',
    'GENConv',
    'GCN2Conv',
    'PANConv',
    'WLConv',
    'WLConvContinuous',
    'FiLMConv',
    'SuperGATConv',
    'FAConv',
    'EGConv',
    'PDNConv',
    'GeneralConv',
    'HGTConv',
    'HEATConv',
    'HeteroConv',
    'HANConv',
    'LGConv',
    'PointGNNConv',
    'GPSConv',
    'AntiSymmetricConv',
    'DirGNNConv',
]

classes = __all__

ECConv = NNConv
PointConv = PointNetConv
pytorch/torch_geometric/nn/conv/agnn_conv.py000066400000000000000000000060201444262304300216250ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.typing import Adj, OptTensor, SparseTensor, torch_sparse
from torch_geometric.utils import add_self_loops, remove_self_loops, softmax


class AGNNConv(MessagePassing):
    r"""The graph attentional propagation layer from the
    `"Attention-based Graph Neural Network for Semi-Supervised Learning"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} = \mathbf{P} \mathbf{X},

    where the propagation matrix :math:`\mathbf{P}` is computed as

    .. math::
        P_{i,j} = \frac{\exp( \beta \cdot \cos(\mathbf{x}_i, \mathbf{x}_j))}
        {\sum_{k \in \mathcal{N}(i)\cup \{ i \}} \exp( \beta \cdot
        \cos(\mathbf{x}_i, \mathbf{x}_k))}

    with trainable parameter :math:`\beta`.

    Args:
        requires_grad (bool, optional): If set to :obj:`False`, :math:`\beta`
            will not be trainable. (default: :obj:`True`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)`,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F)`
    """
    def __init__(self, requires_grad: bool = True, add_self_loops: bool = True,
                 **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.requires_grad = requires_grad
        self.add_self_loops = add_self_loops

        if requires_grad:
            self.beta = Parameter(torch.Tensor(1))
        else:
            self.register_buffer('beta', torch.ones(1))

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if self.requires_grad:
            self.beta.data.fill_(1)

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(edge_index,
                                               num_nodes=x.size(self.node_dim))
            elif isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.set_diag(edge_index)

        x_norm = F.normalize(x, p=2., dim=-1)

        # propagate_type: (x: Tensor, x_norm: Tensor)
        return self.propagate(edge_index, x=x, x_norm=x_norm, size=None)

    def message(self, x_j: Tensor, x_norm_i: Tensor, x_norm_j: Tensor,
                index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:
        alpha = self.beta * (x_norm_i * x_norm_j).sum(dim=-1)
        alpha = softmax(alpha, index, ptr, size_i)
        return x_j * alpha.view(-1, 1)
pytorch/torch_geometric/nn/conv/antisymmetric_conv.py000066400000000000000000000104441444262304300235770ustar00rootroot00000000000000import math
from typing import Any, Callable, Dict, Optional, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import GCNConv, MessagePassing
from torch_geometric.nn.inits import zeros
from torch_geometric.nn.resolver import activation_resolver
from torch_geometric.typing import Adj


class AntiSymmetricConv(torch.nn.Module):
    r"""The anti-symmetric graph convolutional operator from the
    `"Anti-Symmetric DGN: a stable architecture for Deep Graph Networks"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{x}_i + \epsilon \cdot \sigma \left(
            (\mathbf{W}-\mathbf{W}^T-\gamma \mathbf{I}) \mathbf{x}_i +
            \Phi(\mathbf{X}, \mathcal{N}_i) + \mathbf{b}\right),

    where :math:`\Phi(\mathbf{X}, \mathcal{N}_i)` denotes a
    :class:`~torch.nn.conv.MessagePassing` layer.

    Args:
        in_channels (int): Size of each input sample.
        phi (MessagePassing, optional): The message passing module
            :math:`\Phi`. If set to :obj:`None`, will use a
            :class:`~torch_geometric.nn.conv.GCNConv` layer as default.
            (default: :obj:`None`)
        num_iters (int, optional): The number of times the anti-symmetric deep
            graph network operator is called. (default: :obj:`1`)
        epsilon (float, optional): The discretization step size
            :math:`\epsilon`. (default: :obj:`0.1`)
        gamma (float, optional): The strength of the diffusion :math:`\gamma`.
            It regulates the stability of the method. (default: :obj:`0.1`)
        act (str, optional): The non-linear activation function :math:`\sigma`,
            *e.g.*, :obj:`"tanh"` or :obj:`"relu"`. (default: :class:`"tanh"`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{in})`
    """
    def __init__(
        self,
        in_channels: int,
        phi: Optional[MessagePassing] = None,
        num_iters: int = 1,
        epsilon: float = 0.1,
        gamma: float = 0.1,
        act: Union[str, Callable, None] = 'tanh',
        act_kwargs: Optional[Dict[str, Any]] = None,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.num_iters = num_iters
        self.gamma = gamma
        self.epsilon = epsilon
        self.act = activation_resolver(act, **(act_kwargs or {}))

        if phi is None:
            phi = GCNConv(in_channels, in_channels, bias=False)

        self.W = Parameter(torch.Tensor(in_channels, in_channels))
        self.register_buffer('eye', torch.eye(in_channels))
        self.phi = phi

        if bias:
            self.bias = Parameter(torch.Tensor(in_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        torch.nn.init.kaiming_uniform_(self.W, a=math.sqrt(5))
        self.phi.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj, *args, **kwargs) -> Tensor:
        r"""Runs the forward pass of the module."""
        antisymmetric_W = self.W - self.W.t() - self.gamma * self.eye

        for _ in range(self.num_iters):
            h = self.phi(x, edge_index, *args, **kwargs)
            h = x @ antisymmetric_W.t() + h

            if self.bias is not None:
                h += self.bias

            if self.act is not None:
                h = self.act(h)

            x = x + self.epsilon * h

        return x

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'{self.in_channels}, '
                f'phi={self.phi}, '
                f'num_iters={self.num_iters}, '
                f'epsilon={self.epsilon}, '
                f'gamma={self.gamma})')
pytorch/torch_geometric/nn/conv/appnp.py000066400000000000000000000135721444262304300210050ustar00rootroot00000000000000from typing import Optional

import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, SparseTensor
from torch_geometric.utils import is_torch_sparse_tensor, spmm, to_edge_index
from torch_geometric.utils.sparse import set_sparse_value


class APPNP(MessagePassing):
    r"""The approximate personalized propagation of neural predictions layer
    from the `"Predict then Propagate: Graph Neural Networks meet Personalized
    PageRank" `_ paper

    .. math::
        \mathbf{X}^{(0)} &= \mathbf{X}

        \mathbf{X}^{(k)} &= (1 - \alpha) \mathbf{\hat{D}}^{-1/2}
        \mathbf{\hat{A}} \mathbf{\hat{D}}^{-1/2} \mathbf{X}^{(k-1)} + \alpha
        \mathbf{X}^{(0)}

        \mathbf{X}^{\prime} &= \mathbf{X}^{(K)},

    where :math:`\mathbf{\hat{A}} = \mathbf{A} + \mathbf{I}` denotes the
    adjacency matrix with inserted self-loops and
    :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij}` its diagonal degree matrix.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.

    Args:
        K (int): Number of iterations :math:`K`.
        alpha (float): Teleport probability :math:`\alpha`.
        dropout (float, optional): Dropout probability of edges during
            training. (default: :obj:`0`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
            \mathbf{\hat{D}}^{-1/2}` on first execution, and will use the
            cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        normalize (bool, optional): Whether to add self-loops and apply
            symmetric normalization. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)`
    """
    _cached_edge_index: Optional[OptPairTensor]
    _cached_adj_t: Optional[SparseTensor]

    def __init__(self, K: int, alpha: float, dropout: float = 0.,
                 cached: bool = False, add_self_loops: bool = True,
                 normalize: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)
        self.K = K
        self.alpha = alpha
        self.dropout = dropout
        self.cached = cached
        self.add_self_loops = add_self_loops
        self.normalize = normalize

        self._cached_edge_index = None
        self._cached_adj_t = None

    def reset_parameters(self):
        super().reset_parameters()
        self._cached_edge_index = None
        self._cached_adj_t = None

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if self.normalize:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, edge_weight = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, edge_weight)
                else:
                    edge_index, edge_weight = cache[0], cache[1]

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache

        h = x
        for k in range(self.K):
            if self.dropout > 0 and self.training:
                if isinstance(edge_index, Tensor):
                    if is_torch_sparse_tensor(edge_index):
                        _, edge_weight = to_edge_index(edge_index)
                        edge_weight = F.dropout(edge_weight, p=self.dropout)
                        edge_index = set_sparse_value(edge_index, edge_weight)
                    else:
                        assert edge_weight is not None
                        edge_weight = F.dropout(edge_weight, p=self.dropout)
                else:
                    value = edge_index.storage.value()
                    assert value is not None
                    value = F.dropout(value, p=self.dropout)
                    edge_index = edge_index.set_value(value, layout='coo')

            # propagate_type: (x: Tensor, edge_weight: OptTensor)
            x = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                               size=None)
            x = x * (1 - self.alpha)
            x = x + self.alpha * h

        return x

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(K={self.K}, alpha={self.alpha})'
pytorch/torch_geometric/nn/conv/arma_conv.py000066400000000000000000000147551444262304300216400ustar00rootroot00000000000000from typing import Callable, Optional

import torch
import torch.nn.functional as F
from torch import Tensor, nn
from torch.nn import Parameter, ReLU

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class ARMAConv(MessagePassing):
    r"""The ARMA graph convolutional operator from the `"Graph Neural Networks
    with Convolutional ARMA Filters" `_ paper

    .. math::
        \mathbf{X}^{\prime} = \frac{1}{K} \sum_{k=1}^K \mathbf{X}_k^{(T)},

    with :math:`\mathbf{X}_k^{(T)}` being recursively defined by

    .. math::
        \mathbf{X}_k^{(t+1)} = \sigma \left( \mathbf{\hat{L}}
        \mathbf{X}_k^{(t)} \mathbf{W} + \mathbf{X}^{(0)} \mathbf{V} \right),

    where :math:`\mathbf{\hat{L}} = \mathbf{I} - \mathbf{L} = \mathbf{D}^{-1/2}
    \mathbf{A} \mathbf{D}^{-1/2}` denotes the
    modified Laplacian :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1/2}
    \mathbf{A} \mathbf{D}^{-1/2}`.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample
            :math:`\mathbf{x}^{(t+1)}`.
        num_stacks (int, optional): Number of parallel stacks :math:`K`.
            (default: :obj:`1`).
        num_layers (int, optional): Number of layers :math:`T`.
            (default: :obj:`1`)
        act (callable, optional): Activation function :math:`\sigma`.
            (default: :meth:`torch.nn.ReLU()`)
        shared_weights (int, optional): If set to :obj:`True` the layers in
            each stack will share the same parameters. (default: :obj:`False`)
        dropout (float, optional): Dropout probability of the skip connection.
            (default: :obj:`0.`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int,
                 num_stacks: int = 1, num_layers: int = 1,
                 shared_weights: bool = False,
                 act: Optional[Callable] = ReLU(), dropout: float = 0.,
                 bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_stacks = num_stacks
        self.num_layers = num_layers
        self.act = act
        self.shared_weights = shared_weights
        self.dropout = dropout

        K, T, F_in, F_out = num_stacks, num_layers, in_channels, out_channels
        T = 1 if self.shared_weights else T

        self.weight = Parameter(torch.Tensor(max(1, T - 1), K, F_out, F_out))
        if in_channels > 0:
            self.init_weight = Parameter(torch.Tensor(K, F_in, F_out))
            self.root_weight = Parameter(torch.Tensor(T, K, F_in, F_out))
        else:
            self.init_weight = torch.nn.parameter.UninitializedParameter()
            self.root_weight = torch.nn.parameter.UninitializedParameter()
            self._hook = self.register_forward_pre_hook(
                self.initialize_parameters)

        if bias:
            self.bias = Parameter(torch.Tensor(T, K, 1, F_out))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        glorot(self.weight)
        if not isinstance(self.init_weight, torch.nn.UninitializedParameter):
            glorot(self.init_weight)
            glorot(self.root_weight)
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if isinstance(edge_index, Tensor):
            edge_index, edge_weight = gcn_norm(  # yapf: disable
                edge_index, edge_weight, x.size(self.node_dim),
                add_self_loops=False, flow=self.flow, dtype=x.dtype)

        elif isinstance(edge_index, SparseTensor):
            edge_index = gcn_norm(  # yapf: disable
                edge_index, edge_weight, x.size(self.node_dim),
                add_self_loops=False, flow=self.flow, dtype=x.dtype)

        x = x.unsqueeze(-3)
        out = x
        for t in range(self.num_layers):
            if t == 0:
                out = out @ self.init_weight
            else:
                out = out @ self.weight[0 if self.shared_weights else t - 1]

            # propagate_type: (x: Tensor, edge_weight: OptTensor)
            out = self.propagate(edge_index, x=out, edge_weight=edge_weight,
                                 size=None)

            root = F.dropout(x, p=self.dropout, training=self.training)
            root = root @ self.root_weight[0 if self.shared_weights else t]
            out = out + root

            if self.bias is not None:
                out = out + self.bias[0 if self.shared_weights else t]

            if self.act is not None:
                out = self.act(out)

        return out.mean(dim=-3)

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        if isinstance(self.init_weight, nn.parameter.UninitializedParameter):
            F_in, F_out = input[0].size(-1), self.out_channels
            T, K = self.weight.size(0) + 1, self.weight.size(1)
            self.init_weight.materialize((K, F_in, F_out))
            self.root_weight.materialize((T, K, F_in, F_out))
            glorot(self.init_weight)
            glorot(self.root_weight)

        module._hook.remove()
        delattr(module, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_stacks={self.num_stacks}, '
                f'num_layers={self.num_layers})')
pytorch/torch_geometric/nn/conv/cg_conv.py000066400000000000000000000077041444262304300213050ustar00rootroot00000000000000from typing import Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import BatchNorm1d, Linear

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.typing import Adj, OptTensor, PairTensor


class CGConv(MessagePassing):
    r"""The crystal graph convolutional operator from the
    `"Crystal Graph Convolutional Neural Networks for an
    Accurate and Interpretable Prediction of Material Properties"
    `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{x}_i + \sum_{j \in \mathcal{N}(i)}
        \sigma \left( \mathbf{z}_{i,j} \mathbf{W}_f + \mathbf{b}_f \right)
        \odot g \left( \mathbf{z}_{i,j} \mathbf{W}_s + \mathbf{b}_s  \right)

    where :math:`\mathbf{z}_{i,j} = [ \mathbf{x}_i, \mathbf{x}_j,
    \mathbf{e}_{i,j} ]` denotes the concatenation of central node features,
    neighboring node features and edge features.
    In addition, :math:`\sigma` and :math:`g` denote the sigmoid and softplus
    functions, respectively.

    Args:
        channels (int or tuple): Size of each input sample. A tuple
            corresponds to the sizes of source and target dimensionalities.
        dim (int, optional): Edge feature dimensionality. (default: :obj:`0`)
        aggr (str, optional): The aggregation operator to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"add"`)
        batch_norm (bool, optional): If set to :obj:`True`, will make use of
            batch normalization. (default: :obj:`False`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)` or
          :math:`(|\mathcal{V_t}|, F_{t})` if bipartite
    """
    def __init__(self, channels: Union[int, Tuple[int, int]], dim: int = 0,
                 aggr: str = 'add', batch_norm: bool = False,
                 bias: bool = True, **kwargs):
        super().__init__(aggr=aggr, **kwargs)
        self.channels = channels
        self.dim = dim
        self.batch_norm = batch_norm

        if isinstance(channels, int):
            channels = (channels, channels)

        self.lin_f = Linear(sum(channels) + dim, channels[1], bias=bias)
        self.lin_s = Linear(sum(channels) + dim, channels[1], bias=bias)
        if batch_norm:
            self.bn = BatchNorm1d(channels[1])
        else:
            self.bn = None

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_f.reset_parameters()
        self.lin_s.reset_parameters()
        if self.bn is not None:
            self.bn.reset_parameters()

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj,
                edge_attr: OptTensor = None) -> Tensor:

        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        # propagate_type: (x: PairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=None)
        out = out if self.bn is None else self.bn(out)
        out = out + x[1]
        return out

    def message(self, x_i, x_j, edge_attr: OptTensor) -> Tensor:
        if edge_attr is None:
            z = torch.cat([x_i, x_j], dim=-1)
        else:
            z = torch.cat([x_i, x_j, edge_attr], dim=-1)
        return self.lin_f(z).sigmoid() * F.softplus(self.lin_s(z))

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.channels}, dim={self.dim})'
pytorch/torch_geometric/nn/conv/cheb_conv.py000066400000000000000000000144541444262304300216150ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import OptTensor
from torch_geometric.utils import get_laplacian


class ChebConv(MessagePassing):
    r"""The chebyshev spectral graph convolutional operator from the
    `"Convolutional Neural Networks on Graphs with Fast Localized Spectral
    Filtering" `_ paper

    .. math::
        \mathbf{X}^{\prime} = \sum_{k=1}^{K} \mathbf{Z}^{(k)} \cdot
        \mathbf{\Theta}^{(k)}

    where :math:`\mathbf{Z}^{(k)}` is computed recursively by

    .. math::
        \mathbf{Z}^{(1)} &= \mathbf{X}

        \mathbf{Z}^{(2)} &= \mathbf{\hat{L}} \cdot \mathbf{X}

        \mathbf{Z}^{(k)} &= 2 \cdot \mathbf{\hat{L}} \cdot
        \mathbf{Z}^{(k-1)} - \mathbf{Z}^{(k-2)}

    and :math:`\mathbf{\hat{L}}` denotes the scaled and normalized Laplacian
    :math:`\frac{2\mathbf{L}}{\lambda_{\max}} - \mathbf{I}`.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        K (int): Chebyshev filter size :math:`K`.
        normalization (str, optional): The normalization scheme for the graph
            Laplacian (default: :obj:`"sym"`):

            1. :obj:`None`: No normalization
            :math:`\mathbf{L} = \mathbf{D} - \mathbf{A}`

            2. :obj:`"sym"`: Symmetric normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1/2} \mathbf{A}
            \mathbf{D}^{-1/2}`

            3. :obj:`"rw"`: Random-walk normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1} \mathbf{A}`

            :obj:`\lambda_max` should be a :class:`torch.Tensor` of size
            :obj:`[num_graphs]` in a mini-batch scenario and a
            scalar/zero-dimensional tensor when operating on single graphs.
            You can pre-compute :obj:`lambda_max` via the
            :class:`torch_geometric.transforms.LaplacianLambdaMax` transform.
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*,
          batch vector :math:`(|\mathcal{V}|)` *(optional)*,
          maximum :obj:`lambda` value :math:`(|\mathcal{G}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        K: int,
        normalization: Optional[str] = 'sym',
        bias: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        assert K > 0
        assert normalization in [None, 'sym', 'rw'], 'Invalid normalization'

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.normalization = normalization
        self.lins = torch.nn.ModuleList([
            Linear(in_channels, out_channels, bias=False,
                   weight_initializer='glorot') for _ in range(K)
        ])

        if bias:
            self.bias = Parameter(Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        for lin in self.lins:
            lin.reset_parameters()
        zeros(self.bias)

    def __norm__(
        self,
        edge_index: Tensor,
        num_nodes: Optional[int],
        edge_weight: OptTensor,
        normalization: Optional[str],
        lambda_max: OptTensor = None,
        dtype: Optional[int] = None,
        batch: OptTensor = None,
    ):
        edge_index, edge_weight = get_laplacian(edge_index, edge_weight,
                                                normalization, dtype,
                                                num_nodes)
        assert edge_weight is not None

        if lambda_max is None:
            lambda_max = 2.0 * edge_weight.max()
        elif not isinstance(lambda_max, Tensor):
            lambda_max = torch.tensor(lambda_max, dtype=dtype,
                                      device=edge_index.device)
        assert lambda_max is not None

        if batch is not None and lambda_max.numel() > 1:
            lambda_max = lambda_max[batch[edge_index[0]]]

        edge_weight = (2.0 * edge_weight) / lambda_max
        edge_weight.masked_fill_(edge_weight == float('inf'), 0)

        loop_mask = edge_index[0] == edge_index[1]
        edge_weight[loop_mask] -= 1

        return edge_index, edge_weight

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_weight: OptTensor = None,
        batch: OptTensor = None,
        lambda_max: OptTensor = None,
    ) -> Tensor:

        edge_index, norm = self.__norm__(
            edge_index,
            x.size(self.node_dim),
            edge_weight,
            self.normalization,
            lambda_max,
            dtype=x.dtype,
            batch=batch,
        )

        Tx_0 = x
        Tx_1 = x  # Dummy.
        out = self.lins[0](Tx_0)

        # propagate_type: (x: Tensor, norm: Tensor)
        if len(self.lins) > 1:
            Tx_1 = self.propagate(edge_index, x=x, norm=norm, size=None)
            out = out + self.lins[1](Tx_1)

        for lin in self.lins[2:]:
            Tx_2 = self.propagate(edge_index, x=Tx_1, norm=norm, size=None)
            Tx_2 = 2. * Tx_2 - Tx_0
            out = out + lin.forward(Tx_2)
            Tx_0, Tx_1 = Tx_1, Tx_2

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, norm: Tensor) -> Tensor:
        return norm.view(-1, 1) * x_j

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, K={len(self.lins)}, '
                f'normalization={self.normalization})')
pytorch/torch_geometric/nn/conv/cluster_gcn_conv.py000066400000000000000000000122671444262304300232240ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptTensor, SparseTensor, torch_sparse
from torch_geometric.utils import (
    add_self_loops,
    degree,
    is_torch_sparse_tensor,
    remove_self_loops,
    spmm,
    to_edge_index,
)
from torch_geometric.utils.sparse import set_sparse_value


class ClusterGCNConv(MessagePassing):
    r"""The ClusterGCN graph convolutional operator from the
    `"Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph
    Convolutional Networks" `_ paper

    .. math::
        \mathbf{X}^{\prime} = \left( \mathbf{\hat{A}} + \lambda \cdot
        \textrm{diag}(\mathbf{\hat{A}}) \right) \mathbf{X} \mathbf{W}_1 +
        \mathbf{X} \mathbf{W}_2

    where :math:`\mathbf{\hat{A}} = {(\mathbf{D} + \mathbf{I})}^{-1}(\mathbf{A}
    + \mathbf{I})`.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        diag_lambda (float, optional): Diagonal enhancement value
            :math:`\lambda`. (default: :obj:`0.`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int,
                 diag_lambda: float = 0., add_self_loops: bool = True,
                 bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.diag_lambda = diag_lambda
        self.add_self_loops = add_self_loops

        self.lin_out = Linear(in_channels, out_channels, bias=bias,
                              weight_initializer='glorot')
        self.lin_root = Linear(in_channels, out_channels, bias=False,
                               weight_initializer='glorot')

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_out.reset_parameters()
        self.lin_root.reset_parameters()

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        num_nodes = x.size(self.node_dim)
        edge_weight: OptTensor = None

        if isinstance(edge_index, SparseTensor):
            assert edge_index.size(0) == edge_index.size(1)

            if self.add_self_loops:
                edge_index = torch_sparse.set_diag(edge_index)

            col, row, _ = edge_index.coo()  # Transposed.
            deg_inv = 1. / torch_sparse.sum(edge_index, dim=1).clamp_(1.)

            edge_weight = deg_inv[col]
            edge_weight[row == col] += self.diag_lambda * deg_inv
            edge_index = edge_index.set_value(edge_weight, layout='coo')

        elif is_torch_sparse_tensor(edge_index):
            assert edge_index.size(0) == edge_index.size(1)

            if edge_index.layout == torch.sparse_csc:
                raise NotImplementedError("Sparse CSC matrices are not yet "
                                          "supported in 'gcn_norm'")

            if self.add_self_loops:
                edge_index, _ = add_self_loops(edge_index, num_nodes=num_nodes)

            col_and_row, value = to_edge_index(edge_index)
            col, row = col_and_row[0], col_and_row[1]
            deg_inv = 1. / degree(col, num_nodes=edge_index.size(0)).clamp_(1.)

            edge_weight = deg_inv[col]
            edge_weight[row == col] += self.diag_lambda * deg_inv

            edge_index = set_sparse_value(edge_index, edge_weight)

        else:
            if self.add_self_loops:
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(edge_index, num_nodes=num_nodes)

            row, col = edge_index[0], edge_index[1]
            deg_inv = 1. / degree(col, num_nodes=num_nodes).clamp_(1.)

            edge_weight = deg_inv[col]
            edge_weight[row == col] += self.diag_lambda * deg_inv

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=None)
        out = self.lin_out(out) + self.lin_root(x)

        return out

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, diag_lambda={self.diag_lambda})')
pytorch/torch_geometric/nn/conv/cugraph/000077500000000000000000000000001444262304300207365ustar00rootroot00000000000000pytorch/torch_geometric/nn/conv/cugraph/__init__.py000066400000000000000000000003731444262304300230520ustar00rootroot00000000000000from .base import CuGraphModule
from .sage_conv import CuGraphSAGEConv
from .gat_conv import CuGraphGATConv
from .rgcn_conv import CuGraphRGCNConv

__all__ = [
    'CuGraphModule',
    'CuGraphSAGEConv',
    'CuGraphGATConv',
    'CuGraphRGCNConv',
]
pytorch/torch_geometric/nn/conv/cugraph/base.py000066400000000000000000000200031444262304300222150ustar00rootroot00000000000000import warnings
from typing import Any, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.utils import index_sort
from torch_geometric.utils.sparse import index2ptr

try:  # pragma: no cover
    LEGACY_MODE = False
    from pylibcugraphops.pytorch import (
        SampledCSC,
        SampledHeteroCSC,
        StaticCSC,
        StaticHeteroCSC,
    )
    HAS_PYLIBCUGRAPHOPS = True
except ImportError:
    HAS_PYLIBCUGRAPHOPS = False
    try:  # pragma: no cover
        from pylibcugraphops import (
            make_fg_csr,
            make_fg_csr_hg,
            make_mfg_csr,
            make_mfg_csr_hg,
        )
        LEGACY_MODE = True
    except ImportError:
        pass


class CuGraphModule(torch.nn.Module):  # pragma: no cover
    r"""An abstract base class for implementing cugraph message passing layers.
    """
    def __init__(self):
        super().__init__()

        if not HAS_PYLIBCUGRAPHOPS and not LEGACY_MODE:
            raise ModuleNotFoundError(f"'{self.__class__.__name__}' requires "
                                      f"'pylibcugraphops>=23.02'")

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        pass

    @staticmethod
    def to_csc(
        edge_index: Tensor,
        size: Optional[Tuple[int, int]] = None,
        edge_attr: Optional[Tensor] = None,
    ) -> Union[Tuple[Tensor, Tensor, int], Tuple[Tuple[Tensor, Tensor, int],
                                                 Tensor]]:
        r"""Returns a CSC representation of an :obj:`edge_index` tensor to be
        used as input to a :class:`CuGraphModule`.

        Args:
            edge_index (torch.Tensor): The edge indices.
            size ((int, int), optional). The shape of :obj:`edge_index` in each
                dimension. (default: :obj:`None`)
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
        """
        if size is None:
            warnings.warn(f"Inferring the graph size from 'edge_index' causes "
                          f"a decline in performance and does not work for "
                          f"bipartite graphs. To suppress this warning, pass "
                          f"the 'size' explicitly in '{__name__}.to_csc()'.")
            num_src_nodes = num_dst_nodes = int(edge_index.max()) + 1
        else:
            num_src_nodes, num_dst_nodes = size

        row, col = edge_index
        col, perm = index_sort(col, max_value=num_dst_nodes)
        row = row[perm]

        colptr = index2ptr(col, num_dst_nodes)

        if edge_attr is not None:
            return (row, colptr, num_src_nodes), edge_attr[perm]

        return row, colptr, num_src_nodes

    def get_cugraph(
        self,
        csc: Tuple[Tensor, Tensor, int],
        max_num_neighbors: Optional[int] = None,
    ) -> Any:
        r"""Constructs a :obj:`cugraph` graph object from CSC representation.
        Supports both bipartite and non-bipartite graphs.

        Args:
            csc ((torch.Tensor, torch.Tensor, int)): A tuple containing the CSC
                representation of a graph, given as a tuple of
                :obj:`(row, colptr, num_src_nodes)`. Use the
                :meth:`CuGraphModule.to_csc` method to convert an
                :obj:`edge_index` representation to the desired format.
            max_num_neighbors (int, optional): The maximum number of neighbors
                of a target node. It is only effective when operating in a
                bipartite graph. When not given, will be computed on-the-fly,
                leading to slightly worse performance. (default: :obj:`None`)
        """
        row, colptr, num_src_nodes = csc

        if not row.is_cuda:
            raise RuntimeError(f"'{self.__class__.__name__}' requires GPU-"
                               f"based processing (got CPU tensor)")

        if num_src_nodes != colptr.numel() - 1:  # Bipartite graph:
            if max_num_neighbors is None:
                max_num_neighbors = int((colptr[1:] - colptr[:-1]).max())

            if LEGACY_MODE:
                dst_nodes = torch.arange(colptr.numel() - 1, device=row.device)
                return make_mfg_csr(dst_nodes, colptr, row, max_num_neighbors,
                                    num_src_nodes)

            return SampledCSC(colptr, row, max_num_neighbors, num_src_nodes)

        if LEGACY_MODE:
            return make_fg_csr(colptr, row)

        return StaticCSC(colptr, row)

    def get_typed_cugraph(
        self,
        csc: Tuple[Tensor, Tensor, int],
        edge_type: Tensor,
        num_edge_types: Optional[int] = None,
        max_num_neighbors: Optional[int] = None,
    ) -> Any:
        r"""Constructs a typed :obj:`cugraph` graph object from a CSC
        representation where each edge corresponds to a given edge type.
        Supports both bipartite and non-bipartite graphs.

        Args:
            csc ((torch.Tensor, torch.Tensor, int)): A tuple containing the CSC
                representation of a graph, given as a tuple of
                :obj:`(row, colptr, num_src_nodes)`. Use the
                :meth:`CuGraphModule.to_csc` method to convert an
                :obj:`edge_index` representation to the desired format.
            edge_type (torch.Tensor): The edge type.
            num_edge_types (int, optional): The maximum number of edge types.
                When not given, will be computed on-the-fly, leading to
                slightly worse performance. (default: :obj:`None`)
            max_num_neighbors (int, optional): The maximum number of neighbors
                of a target node. It is only effective when operating in a
                bipartite graph. When not given, will be computed on-the-fly,
                leading to slightly worse performance. (default: :obj:`None`)
        """
        if num_edge_types is None:
            num_edge_types = int(edge_type.max()) + 1

        row, colptr, num_src_nodes = csc
        edge_type = edge_type.int()

        if num_src_nodes != colptr.numel() - 1:  # Bipartite graph:
            if max_num_neighbors is None:
                max_num_neighbors = int((colptr[1:] - colptr[:-1]).max())

            if LEGACY_MODE:
                dst_nodes = torch.arange(colptr.numel() - 1, device=row.device)
                return make_mfg_csr_hg(dst_nodes, colptr, row,
                                       max_num_neighbors, num_src_nodes,
                                       n_node_types=0,
                                       n_edge_types=num_edge_types,
                                       out_node_types=None, in_node_types=None,
                                       edge_types=edge_type)

            return SampledHeteroCSC(colptr, row, edge_type, max_num_neighbors,
                                    num_src_nodes, num_edge_types)

        if LEGACY_MODE:
            return make_fg_csr_hg(colptr, row, n_node_types=0,
                                  n_edge_types=num_edge_types, node_types=None,
                                  edge_types=edge_type)

        return StaticHeteroCSC(colptr, row, edge_type, num_edge_types)

    def forward(
        self,
        x: Tensor,
        csc: Tuple[Tensor, Tensor, int],
        max_num_neighbors: Optional[int] = None,
    ) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor): The node features.
            csc ((torch.Tensor, torch.Tensor, int)): A tuple containing the CSC
                representation of a graph, given as a tuple of
                :obj:`(row, colptr, num_src_nodes)`. Use the
                :meth:`CuGraphModule.to_csc` method to convert an
                :obj:`edge_index` representation to the desired format.
            max_num_neighbors (int, optional): The maximum number of neighbors
                of a target node. It is only effective when operating in a
                bipartite graph. When not given, the value will be computed
                on-the-fly, leading to slightly worse performance.
                (default: :obj:`None`)
        """
        raise NotImplementedError
pytorch/torch_geometric/nn/conv/cugraph/gat_conv.py000066400000000000000000000054411444262304300231140ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Linear, Parameter

from torch_geometric.nn.conv.cugraph import CuGraphModule
from torch_geometric.nn.conv.cugraph.base import LEGACY_MODE
from torch_geometric.nn.inits import zeros

try:
    if LEGACY_MODE:
        from pylibcugraphops.torch.autograd import mha_gat_n2n as GATConvAgg
    else:
        from pylibcugraphops.pytorch.operators import mha_gat_n2n as GATConvAgg
except ImportError:
    pass


class CuGraphGATConv(CuGraphModule):  # pragma: no cover
    r"""The graph attentional operator from the `"Graph Attention Networks"
    `_ paper.

    :class:`CuGraphGATConv` is an optimized version of
    :class:`~torch_geometric.nn.conv.GATConv` based on the :obj:`cugraph-ops`
    package that fuses message passing computation for accelerated execution
    and lower memory footprint.
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        heads: int = 1,
        concat: bool = True,
        negative_slope: float = 0.2,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope

        self.lin = Linear(in_channels, heads * out_channels, bias=False)
        self.att = Parameter(torch.Tensor(2 * heads * out_channels))

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        self.lin.reset_parameters()
        gain = torch.nn.init.calculate_gain('relu')
        torch.nn.init.xavier_normal_(
            self.att.view(2, self.heads, self.out_channels), gain=gain)
        zeros(self.bias)

    def forward(
        self,
        x: Tensor,
        csc: Tuple[Tensor, Tensor, int],
        max_num_neighbors: Optional[int] = None,
    ) -> Tensor:
        graph = self.get_cugraph(csc, max_num_neighbors)

        x = self.lin(x)

        if LEGACY_MODE:
            out = GATConvAgg(x, self.att, graph, self.heads, 'LeakyReLU',
                             self.negative_slope, False, self.concat)
        else:
            out = GATConvAgg(x, self.att, graph, self.heads, 'LeakyReLU',
                             self.negative_slope, self.concat)

        if self.bias is not None:
            out = out + self.bias

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/cugraph/rgcn_conv.py000066400000000000000000000101721444262304300232670ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv.cugraph import CuGraphModule
from torch_geometric.nn.conv.cugraph.base import LEGACY_MODE
from torch_geometric.nn.inits import glorot, zeros

try:
    if LEGACY_MODE:
        from pylibcugraphops.torch.autograd import \
            agg_hg_basis_n2n_post as RGCNConvAgg
    else:
        from pylibcugraphops.pytorch.operators import \
            agg_hg_basis_n2n_post as RGCNConvAgg
except ImportError:
    pass


class CuGraphRGCNConv(CuGraphModule):  # pragma: no cover
    r"""The relational graph convolutional operator from the `"Modeling
    Relational Data with Graph Convolutional Networks"
    `_ paper.

    :class:`CuGraphRGCNConv` is an optimized version of
    :class:`~torch_geometric.nn.conv.RGCNConv` based on the :obj:`cugraph-ops`
    package that fuses message passing computation for accelerated execution
    and lower memory footprint.
    """
    def __init__(self, in_channels: int, out_channels: int, num_relations: int,
                 num_bases: Optional[int] = None, aggr: str = 'mean',
                 root_weight: bool = True, bias: bool = True):
        super().__init__()

        if aggr not in ['sum', 'add', 'mean']:
            raise ValueError(f"Aggregation function must be either 'mean' "
                             f"or 'sum' (got '{aggr}')")

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_relations = num_relations
        self.num_bases = num_bases
        self.aggr = aggr
        self.root_weight = root_weight

        dim_root_weight = 1 if root_weight else 0

        if num_bases is not None:
            self.weight = Parameter(
                torch.Tensor(num_bases + dim_root_weight, in_channels,
                             out_channels))
            self.comp = Parameter(torch.Tensor(num_relations, num_bases))
        else:
            self.weight = Parameter(
                torch.Tensor(num_relations + dim_root_weight, in_channels,
                             out_channels))
            self.register_parameter('comp', None)

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        end = -1 if self.root_weight else None
        glorot(self.weight[:end])
        glorot(self.comp)
        if self.root_weight:
            glorot(self.weight[-1])
        zeros(self.bias)

    def forward(
        self,
        x: Tensor,
        csc: Tuple[Tensor, Tensor, int],
        edge_type: Tensor,
        max_num_neighbors: Optional[int] = None,
    ) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor): The node features.
            csc ((torch.Tensor, torch.Tensor)): A tuple containing the CSC
                representation of a graph, given as a tuple of
                :obj:`(row, colptr)`. Use the :meth:`to_csc` method to convert
                an :obj:`edge_index` representation to the desired format.
            edge_type (torch.Tensor): The edge type.
            max_num_neighbors (int, optional): The maximum number of neighbors
                of a target node. It is only effective when operating in a
                bipartite graph.. When not given, the value will be computed
                on-the-fly, leading to slightly worse performance.
                (default: :obj:`None`)
        """
        graph = self.get_typed_cugraph(csc, edge_type, self.num_relations,
                                       max_num_neighbors)

        out = RGCNConvAgg(x, self.comp, graph, concat_own=self.root_weight,
                          norm_by_out_degree=bool(self.aggr == 'mean'))

        out = out @ self.weight.view(-1, self.out_channels)

        if self.bias is not None:
            out = out + self.bias

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_relations={self.num_relations})')
pytorch/torch_geometric/nn/conv/cugraph/sage_conv.py000066400000000000000000000053621444262304300232620ustar00rootroot00000000000000from typing import Optional, Tuple

import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear

from torch_geometric.nn.conv.cugraph import CuGraphModule
from torch_geometric.nn.conv.cugraph.base import LEGACY_MODE

try:
    if LEGACY_MODE:
        from pylibcugraphops.torch.autograd import \
            agg_concat_n2n as SAGEConvAgg
    else:
        from pylibcugraphops.pytorch.operators import \
            agg_concat_n2n as SAGEConvAgg
except ImportError:
    pass


class CuGraphSAGEConv(CuGraphModule):  # pragma: no cover
    r"""The GraphSAGE operator from the `"Inductive Representation Learning on
    Large Graphs" `_ paper.

    :class:`CuGraphSAGEConv` is an optimized version of
    :class:`~torch_geometric.nn.conv.SAGEConv` based on the :obj:`cugraph-ops`
    package that fuses message passing computation for accelerated execution
    and lower memory footprint.
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        aggr: str = 'mean',
        normalize: bool = False,
        root_weight: bool = True,
        project: bool = False,
        bias: bool = True,
    ):
        super().__init__()

        if aggr not in ['mean', 'sum', 'min', 'max']:
            raise ValueError(f"Aggregation function must be either 'mean', "
                             f"'sum', 'min' or 'max' (got '{aggr}')")

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.aggr = aggr
        self.normalize = normalize
        self.root_weight = root_weight
        self.project = project

        if self.project:
            self.pre_lin = Linear(in_channels, in_channels, bias=True)

        if self.root_weight:
            self.lin = Linear(2 * in_channels, out_channels, bias=bias)
        else:
            self.lin = Linear(in_channels, out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        if self.project:
            self.pre_lin.reset_parameters()
        self.lin.reset_parameters()

    def forward(
        self,
        x: Tensor,
        csc: Tuple[Tensor, Tensor, int],
        max_num_neighbors: Optional[int] = None,
    ) -> Tensor:
        graph = self.get_cugraph(csc, max_num_neighbors)

        if self.project:
            x = self.pre_lin(x).relu()

        out = SAGEConvAgg(x, graph, self.aggr)

        if self.root_weight:
            out = self.lin(out)
        else:
            out = self.lin(out[:, :self.in_channels])

        if self.normalize:
            out = F.normalize(out, p=2., dim=-1)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, aggr={self.aggr})')
pytorch/torch_geometric/nn/conv/dir_gnn_conv.py000066400000000000000000000045731444262304300223350ustar00rootroot00000000000000import copy

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing


class DirGNNConv(torch.nn.Module):
    r"""A generic wrapper for computing graph convolution on directed
    graphs as described in the `"Edge Directionality Improves Learning on
    Heterophilic Graphs" `_ paper.
    :class:`DirGNNConv` will pass messages both from source nodes to target
    nodes and from target nodes to source nodes.

    Args:
        conv (MessagePassing): The underlying
            :class:`~torch_geometric.nn.conv.MessagePassing` layer to use.
        alpha (float, optional): The alpha coefficient used to weight the
            aggregations of in- and out-edges as part of a convex combination.
            (default: :obj:`0.5`)
        root_weight (bool, optional): If set to :obj:`True`, the layer will add
            transformed root node features to the output.
            (default: :obj:`True`)
    """
    def __init__(
        self,
        conv: MessagePassing,
        alpha: float = 0.5,
        root_weight: bool = True,
    ):
        super().__init__()

        self.alpha = alpha
        self.root_weight = root_weight

        self.conv_in = copy.deepcopy(conv)
        self.conv_out = copy.deepcopy(conv)

        if hasattr(conv, 'add_self_loops'):
            self.conv_in.add_self_loops = False
            self.conv_out.add_self_loops = False
        if hasattr(conv, 'root_weight'):
            self.conv_in.root_weight = False
            self.conv_out.root_weight = False

        if root_weight:
            self.lin = torch.nn.Linear(conv.in_channels, conv.out_channels)
        else:
            self.lin = None

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.conv_in.reset_parameters()
        self.conv_out.reset_parameters()
        if self.lin is not None:
            self.lin.reset_parameters()

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        r""""""
        x_in = self.conv_in(x, edge_index)
        x_out = self.conv_out(x, edge_index.flip([0]))

        out = self.alpha * x_out + (1 - self.alpha) * x_in

        if self.root_weight:
            out = out + self.lin(x)

        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.conv_in}, alpha={self.alpha})'
pytorch/torch_geometric/nn/conv/dna_conv.py000066400000000000000000000275111444262304300214540ustar00rootroot00000000000000import math
from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.inits import kaiming_uniform, uniform
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, SparseTensor


class Linear(torch.nn.Module):
    def __init__(self, in_channels, out_channels, groups=1, bias=True):
        super().__init__()
        assert in_channels % groups == 0 and out_channels % groups == 0

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.groups = groups

        self.weight = Parameter(
            torch.Tensor(groups, in_channels // groups,
                         out_channels // groups))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        kaiming_uniform(self.weight, fan=self.weight.size(1), a=math.sqrt(5))
        uniform(self.weight.size(1), self.bias)

    def forward(self, src):
        # Input: [*, in_channels]
        # Output: [*, out_channels]

        if self.groups > 1:
            size = src.size()[:-1]
            src = src.view(-1, self.groups, self.in_channels // self.groups)
            src = src.transpose(0, 1).contiguous()
            out = torch.matmul(src, self.weight)
            out = out.transpose(1, 0).contiguous()
            out = out.view(size + (self.out_channels, ))
        else:
            out = torch.matmul(src, self.weight.squeeze(0))

        if self.bias is not None:
            out = out + self.bias

        return out

    def __repr__(self) -> str:  # pragma: no cover
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, groups={self.groups})')


def restricted_softmax(src, dim: int = -1, margin: float = 0.):
    src_max = torch.clamp(src.max(dim=dim, keepdim=True)[0], min=0.)
    out = (src - src_max).exp()
    out = out / (out.sum(dim=dim, keepdim=True) + (margin - src_max).exp())
    return out


class Attention(torch.nn.Module):
    def __init__(self, dropout=0):
        super().__init__()
        self.dropout = dropout

    def forward(self, query, key, value):
        return self.compute_attention(query, key, value)

    def compute_attention(self, query, key, value):
        # query: [*, query_entries, dim_k]
        # key: [*, key_entries, dim_k]
        # value: [*, key_entries, dim_v]
        # Output: [*, query_entries, dim_v]

        assert query.dim() == key.dim() == value.dim() >= 2
        assert query.size(-1) == key.size(-1)
        assert key.size(-2) == value.size(-2)

        # Score: [*, query_entries, key_entries]
        score = torch.matmul(query, key.transpose(-2, -1))
        score = score / math.sqrt(key.size(-1))
        score = restricted_softmax(score, dim=-1)
        score = F.dropout(score, p=self.dropout, training=self.training)

        return torch.matmul(score, value)

    def __repr__(self) -> str:  # pragma: no cover
        return f'{self.__class__.__name__}(dropout={self.dropout})'


class MultiHead(Attention):
    def __init__(self, in_channels, out_channels, heads=1, groups=1, dropout=0,
                 bias=True):
        super().__init__(dropout)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.groups = groups
        self.bias = bias

        assert in_channels % heads == 0 and out_channels % heads == 0
        assert in_channels % groups == 0 and out_channels % groups == 0
        assert max(groups, self.heads) % min(groups, self.heads) == 0

        self.lin_q = Linear(in_channels, out_channels, groups, bias)
        self.lin_k = Linear(in_channels, out_channels, groups, bias)
        self.lin_v = Linear(in_channels, out_channels, groups, bias)

        self.reset_parameters()

    def reset_parameters(self):
        self.lin_q.reset_parameters()
        self.lin_k.reset_parameters()
        self.lin_v.reset_parameters()

    def forward(self, query, key, value):
        # query: [*, query_entries, in_channels]
        # key: [*, key_entries, in_channels]
        # value: [*, key_entries, in_channels]
        # Output: [*, query_entries, out_channels]

        assert query.dim() == key.dim() == value.dim() >= 2
        assert query.size(-1) == key.size(-1) == value.size(-1)
        assert key.size(-2) == value.size(-2)

        query = self.lin_q(query)
        key = self.lin_k(key)
        value = self.lin_v(value)

        # query: [*, heads, query_entries, out_channels // heads]
        # key: [*, heads, key_entries, out_channels // heads]
        # value: [*, heads, key_entries, out_channels // heads]
        size = query.size()[:-2]
        out_channels_per_head = self.out_channels // self.heads

        query_size = size + (query.size(-2), self.heads, out_channels_per_head)
        query = query.view(query_size).transpose(-2, -3)

        key_size = size + (key.size(-2), self.heads, out_channels_per_head)
        key = key.view(key_size).transpose(-2, -3)

        value_size = size + (value.size(-2), self.heads, out_channels_per_head)
        value = value.view(value_size).transpose(-2, -3)

        # Output: [*, heads, query_entries, out_channels // heads]
        out = self.compute_attention(query, key, value)
        # Output: [*, query_entries, heads, out_channels // heads]
        out = out.transpose(-3, -2).contiguous()
        # Output: [*, query_entries, out_channels]
        out = out.view(size + (query.size(-2), self.out_channels))

        return out

    def __repr__(self) -> str:  # pragma: no cover
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads}, '
                f'groups={self.groups}, dropout={self.droput}, '
                f'bias={self.bias})')


class DNAConv(MessagePassing):
    r"""The dynamic neighborhood aggregation operator from the `"Just Jump:
    Towards Dynamic Neighborhood Aggregation in Graph Neural Networks"
    `_ paper

    .. math::
        \mathbf{x}_v^{(t)} = h_{\mathbf{\Theta}}^{(t)} \left( \mathbf{x}_{v
        \leftarrow v}^{(t)}, \left\{ \mathbf{x}_{v \leftarrow w}^{(t)} : w \in
        \mathcal{N}(v) \right\} \right)

    based on (multi-head) dot-product attention

    .. math::
        \mathbf{x}_{v \leftarrow w}^{(t)} = \textrm{Attention} \left(
        \mathbf{x}^{(t-1)}_v \, \mathbf{\Theta}_Q^{(t)}, [\mathbf{x}_w^{(1)},
        \ldots, \mathbf{x}_w^{(t-1)}] \, \mathbf{\Theta}_K^{(t)}, \,
        [\mathbf{x}_w^{(1)}, \ldots, \mathbf{x}_w^{(t-1)}] \,
        \mathbf{\Theta}_V^{(t)} \right)

    with :math:`\mathbf{\Theta}_Q^{(t)}, \mathbf{\Theta}_K^{(t)},
    \mathbf{\Theta}_V^{(t)}` denoting (grouped) projection matrices for query,
    key and value information, respectively.
    :math:`h^{(t)}_{\mathbf{\Theta}}` is implemented as a non-trainable
    version of :class:`torch_geometric.nn.conv.GCNConv`.

    .. note::
        In contrast to other layers, this operator expects node features as
        shape :obj:`[num_nodes, num_layers, channels]`.

    Args:
        channels (int): Size of each input/output sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        groups (int, optional): Number of groups to use for all linear
            projections. (default: :obj:`1`)
        dropout (float, optional): Dropout probability of attention
            coefficients. (default: :obj:`0.`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
            \mathbf{\hat{D}}^{-1/2}` on first execution, and will use the
            cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        normalize (bool, optional): Whether to add self-loops and apply
            symmetric normalization. (default: :obj:`True`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, L, F)` where :math:`L` is the
          number of layers,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F)`
    """

    _cached_edge_index: Optional[OptPairTensor]
    _cached_adj_t: Optional[SparseTensor]

    def __init__(self, channels: int, heads: int = 1, groups: int = 1,
                 dropout: float = 0., cached: bool = False,
                 normalize: bool = True, add_self_loops: bool = True,
                 bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.bias = bias
        self.cached = cached
        self.normalize = normalize
        self.add_self_loops = add_self_loops

        self._cached_edge_index = None
        self._cached_adj_t = None

        self.multi_head = MultiHead(channels, channels, heads, groups, dropout,
                                    bias)
        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.multi_head.reset_parameters()
        self._cached_edge_index = None
        self._cached_adj_t = None

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor): The input node features of shape
                :obj:`[num_nodes, num_layers, channels]`.
        """
        if x.dim() != 3:
            raise ValueError('Feature shape must be [num_nodes, num_layers, '
                             'channels].')

        if self.normalize:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, edge_weight = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, edge_weight)
                else:
                    edge_index, edge_weight = cache[0], cache[1]

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        return self.propagate(edge_index, x=x, edge_weight=edge_weight,
                              size=None)

    def message(self, x_i: Tensor, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        x_i = x_i[:, -1:]  # [num_edges, 1, channels]
        out = self.multi_head(x_i, x_j, x_j)  # [num_edges, 1, channels]
        return edge_weight.view(-1, 1) * out.squeeze(1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.multi_head.in_channels}, '
                f'heads={self.multi_head.heads}, '
                f'groups={self.multi_head.groups})')
pytorch/torch_geometric/nn/conv/edge_conv.py000066400000000000000000000126561444262304300216220ustar00rootroot00000000000000from typing import Callable, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.typing import Adj, OptTensor, PairOptTensor, PairTensor

try:
    from torch_cluster import knn
except ImportError:
    knn = None


class EdgeConv(MessagePassing):
    r"""The edge convolutional operator from the `"Dynamic Graph CNN for
    Learning on Point Clouds" `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \sum_{j \in \mathcal{N}(i)}
        h_{\mathbf{\Theta}}(\mathbf{x}_i \, \Vert \,
        \mathbf{x}_j - \mathbf{x}_i),

    where :math:`h_{\mathbf{\Theta}}` denotes a neural network, *.i.e.* a MLP.

    Args:
        nn (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}` that
            maps pair-wise concatenated node features :obj:`x` of shape
            :obj:`[-1, 2 * in_channels]` to shape :obj:`[-1, out_channels]`,
            *e.g.*, defined by :class:`torch.nn.Sequential`.
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"max"`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V}|, F_{in}), (|\mathcal{V}|, F_{in}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, nn: Callable, aggr: str = 'max', **kwargs):
        super().__init__(aggr=aggr, **kwargs)
        self.nn = nn
        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.nn)

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj) -> Tensor:
        if isinstance(x, Tensor):
            x: PairTensor = (x, x)
        # propagate_type: (x: PairTensor)
        return self.propagate(edge_index, x=x, size=None)

    def message(self, x_i: Tensor, x_j: Tensor) -> Tensor:
        return self.nn(torch.cat([x_i, x_j - x_i], dim=-1))

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(nn={self.nn})'


class DynamicEdgeConv(MessagePassing):
    r"""The dynamic edge convolutional operator from the `"Dynamic Graph CNN
    for Learning on Point Clouds" `_ paper
    (see :class:`torch_geometric.nn.conv.EdgeConv`), where the graph is
    dynamically constructed using nearest neighbors in the feature space.

    Args:
        nn (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}` that
            maps pair-wise concatenated node features :obj:`x` of shape
            `:obj:`[-1, 2 * in_channels]` to shape :obj:`[-1, out_channels]`,
            *e.g.* defined by :class:`torch.nn.Sequential`.
        k (int): Number of nearest neighbors.
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"max"`)
        num_workers (int): Number of workers to use for k-NN computation.
            Has no effect in case :obj:`batch` is not :obj:`None`, or the input
            lies on the GPU. (default: :obj:`1`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V}|, F_{in}), (|\mathcal{V}|, F_{in}))`
          if bipartite,
          batch vector :math:`(|\mathcal{V}|)` or
          :math:`((|\mathcal{V}|), (|\mathcal{V}|))`
          if bipartite *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, nn: Callable, k: int, aggr: str = 'max',
                 num_workers: int = 1, **kwargs):
        super().__init__(aggr=aggr, flow='source_to_target', **kwargs)

        if knn is None:
            raise ImportError('`DynamicEdgeConv` requires `torch-cluster`.')

        self.nn = nn
        self.k = k
        self.num_workers = num_workers
        self.reset_parameters()

    def reset_parameters(self):
        reset(self.nn)

    def forward(
            self, x: Union[Tensor, PairTensor],
            batch: Union[OptTensor, Optional[PairTensor]] = None) -> Tensor:
        # type: (Tensor, OptTensor) -> Tensor  # noqa
        # type: (PairTensor, Optional[PairTensor]) -> Tensor  # noqa

        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        if x[0].dim() != 2:
            raise ValueError("Static graphs not supported in DynamicEdgeConv")

        b: PairOptTensor = (None, None)
        if isinstance(batch, Tensor):
            b = (batch, batch)
        elif isinstance(batch, tuple):
            assert batch is not None
            b = (batch[0], batch[1])

        edge_index = knn(x[0], x[1], self.k, b[0], b[1]).flip([0])

        # propagate_type: (x: PairTensor)
        return self.propagate(edge_index, x=x, size=None)

    def message(self, x_i: Tensor, x_j: Tensor) -> Tensor:
        return self.nn(torch.cat([x_i, x_j - x_i], dim=-1))

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(nn={self.nn}, k={self.k})'
pytorch/torch_geometric/nn/conv/eg_conv.py000066400000000000000000000243621444262304300213060ustar00rootroot00000000000000from typing import List, Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import Adj, OptTensor, SparseTensor, torch_sparse
from torch_geometric.utils import add_remaining_self_loops, scatter, spmm


class EGConv(MessagePassing):
    r"""The Efficient Graph Convolution from the `"Adaptive Filters and
    Aggregator Fusion for Efficient Graph Convolutions"
    `_ paper.

    Its node-wise formulation is given by:

    .. math::
        \mathbf{x}_i^{\prime} = {\LARGE ||}_{h=1}^H \sum_{\oplus \in
        \mathcal{A}} \sum_{b = 1}^B w_{i, h, \oplus, b} \;
        \underset{j \in \mathcal{N}(i) \cup \{i\}}{\bigoplus}
        \mathbf{W}_b \mathbf{x}_{j}

    with :math:`\mathbf{W}_b` denoting a basis weight,
    :math:`\oplus` denoting an aggregator, and :math:`w` denoting per-vertex
    weighting coefficients across different heads, bases and aggregators.

    EGC retains :math:`\mathcal{O}(|\mathcal{V}|)` memory usage, making it a
    sensible alternative to :class:`~torch_geometric.nn.conv.GCNConv`,
    :class:`~torch_geometric.nn.conv.SAGEConv` or
    :class:`~torch_geometric.nn.conv.GINConv`.

    .. note::
        For an example of using :obj:`EGConv`, see `examples/egc.py
        `_.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        aggregators (List[str], optional): Aggregators to be used.
            Supported aggregators are :obj:`"sum"`, :obj:`"mean"`,
            :obj:`"symnorm"`, :obj:`"max"`, :obj:`"min"`, :obj:`"std"`,
            :obj:`"var"`.
            Multiple aggregators can be used to improve the performance.
            (default: :obj:`["symnorm"]`)
        num_heads (int, optional): Number of heads :math:`H` to use. Must have
            :obj:`out_channels % num_heads == 0`. It is recommended to set
            :obj:`num_heads >= num_bases`. (default: :obj:`8`)
        num_bases (int, optional): Number of basis weights :math:`B` to use.
            (default: :obj:`4`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of the edge index with added self loops on first
            execution, along with caching the calculation of the symmetric
            normalized edge weights if the :obj:`"symnorm"` aggregator is
            being used. This parameter should only be set to :obj:`True` in
            transductive learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """

    _cached_edge_index: Optional[Tuple[Tensor, OptTensor]]
    _cached_adj_t: Optional[SparseTensor]

    def __init__(self, in_channels: int, out_channels: int,
                 aggregators: List[str] = ["symnorm"], num_heads: int = 8,
                 num_bases: int = 4, cached: bool = False,
                 add_self_loops: bool = True, bias: bool = True, **kwargs):
        super().__init__(node_dim=0, **kwargs)

        if out_channels % num_heads != 0:
            raise ValueError(
                'out_channels must be divisible by the number of heads')

        for a in aggregators:
            if a not in ['sum', 'mean', 'symnorm', 'min', 'max', 'var', 'std']:
                raise ValueError(f"Unsupported aggregator: '{a}'")

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_heads = num_heads
        self.num_bases = num_bases
        self.cached = cached
        self.add_self_loops = add_self_loops
        self.aggregators = aggregators

        self.bases_lin = Linear(in_channels,
                                (out_channels // num_heads) * num_bases,
                                bias=False, weight_initializer='glorot')
        self.comb_lin = Linear(in_channels,
                               num_heads * num_bases * len(aggregators))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.bases_lin.reset_parameters()
        self.comb_lin.reset_parameters()
        zeros(self.bias)
        self._cached_adj_t = None
        self._cached_edge_index = None

    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        symnorm_weight: OptTensor = None
        if "symnorm" in self.aggregators:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, symnorm_weight = gcn_norm(  # yapf: disable
                        edge_index, None, num_nodes=x.size(self.node_dim),
                        improved=False, add_self_loops=self.add_self_loops,
                        flow=self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, symnorm_weight)
                else:
                    edge_index, symnorm_weight = cache

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, None, num_nodes=x.size(self.node_dim),
                        improved=False, add_self_loops=self.add_self_loops,
                        flow=self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache

        elif self.add_self_loops:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if self.cached and cache is not None:
                    edge_index = cache[0]
                else:
                    edge_index, _ = add_remaining_self_loops(edge_index)
                    if self.cached:
                        self._cached_edge_index = (edge_index, None)

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if self.cached and cache is not None:
                    edge_index = cache
                else:
                    edge_index = torch_sparse.fill_diag(edge_index, 1.0)
                    if self.cached:
                        self._cached_adj_t = edge_index

        # [num_nodes, (out_channels // num_heads) * num_bases]
        bases = self.bases_lin(x)
        # [num_nodes, num_heads * num_bases * num_aggrs]
        weightings = self.comb_lin(x)

        # [num_nodes, num_aggregators, (out_channels // num_heads) * num_bases]
        # propagate_type: (x: Tensor, symnorm_weight: OptTensor)
        aggregated = self.propagate(edge_index, x=bases,
                                    symnorm_weight=symnorm_weight, size=None)

        weightings = weightings.view(-1, self.num_heads,
                                     self.num_bases * len(self.aggregators))
        aggregated = aggregated.view(
            -1,
            len(self.aggregators) * self.num_bases,
            self.out_channels // self.num_heads,
        )

        # [num_nodes, num_heads, out_channels // num_heads]
        out = torch.matmul(weightings, aggregated)
        out = out.view(-1, self.out_channels)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor) -> Tensor:
        return x_j

    def aggregate(self, inputs: Tensor, index: Tensor,
                  dim_size: Optional[int] = None,
                  symnorm_weight: OptTensor = None) -> Tensor:

        outs = []
        for aggr in self.aggregators:
            if aggr == 'symnorm':
                assert symnorm_weight is not None
                out = scatter(inputs * symnorm_weight.view(-1, 1), index, 0,
                              dim_size, reduce='sum')
            elif aggr == 'var' or aggr == 'std':
                mean = scatter(inputs, index, 0, dim_size, reduce='mean')
                mean_squares = scatter(inputs * inputs, index, 0, dim_size,
                                       reduce='mean')
                out = mean_squares - mean * mean
                if aggr == 'std':
                    out = out.clamp(min=1e-5).sqrt()
            else:
                out = scatter(inputs, index, 0, dim_size, reduce=aggr)

            outs.append(out)

        return torch.stack(outs, dim=1) if len(outs) > 1 else outs[0]

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        adj_t_2 = adj_t
        if len(self.aggregators) > 1 and 'symnorm' in self.aggregators:
            adj_t_2 = adj_t.set_value(None)

        outs = []
        for aggr in self.aggregators:
            if aggr == 'symnorm':
                out = spmm(adj_t, x, reduce='sum')
            elif aggr in ['var', 'std']:
                mean = spmm(adj_t_2, x, reduce='mean')
                mean_sq = spmm(adj_t_2, x * x, reduce='mean')
                out = mean_sq - mean * mean
                if aggr == 'std':
                    out = torch.sqrt(out.relu_() + 1e-5)
            else:
                out = spmm(adj_t_2, x, reduce=aggr)

            outs.append(out)

        return torch.stack(outs, dim=1) if len(outs) > 1 else outs[0]

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, aggregators={self.aggregators})')
pytorch/torch_geometric/nn/conv/fa_conv.py000066400000000000000000000173671444262304300213100ustar00rootroot00000000000000from typing import Optional

import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, SparseTensor
from torch_geometric.utils import is_torch_sparse_tensor
from torch_geometric.utils.sparse import set_sparse_value


class FAConv(MessagePassing):
    r"""The Frequency Adaptive Graph Convolution operator from the
    `"Beyond Low-Frequency Information in Graph Convolutional Networks"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i= \epsilon \cdot \mathbf{x}^{(0)}_i +
        \sum_{j \in \mathcal{N}(i)} \frac{\alpha_{i,j}}{\sqrt{d_i d_j}}
        \mathbf{x}_{j}

    where :math:`\mathbf{x}^{(0)}_i` and :math:`d_i` denote the initial feature
    representation and node degree of node :math:`i`, respectively.
    The attention coefficients :math:`\alpha_{i,j}` are computed as

    .. math::
        \mathbf{\alpha}_{i,j} = \textrm{tanh}(\mathbf{a}^{\top}[\mathbf{x}_i,
        \mathbf{x}_j])

    based on the trainable parameter vector :math:`\mathbf{a}`.

    Args:
        channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        eps (float, optional): :math:`\epsilon`-value. (default: :obj:`0.1`)
        dropout (float, optional): Dropout probability of the normalized
            coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`).
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\sqrt{d_i d_j}` on first execution, and
            will use the cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        normalize (bool, optional): Whether to add self-loops (if
            :obj:`add_self_loops` is :obj:`True`) and compute
            symmetric normalization coefficients on the fly.
            If set to :obj:`False`, :obj:`edge_weight` needs to be provided in
            the layer's :meth:`forward` method. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)`,
          initial node features :math:`(|\mathcal{V}|, F)`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)` or
          :math:`((|\mathcal{V}|, F), ((2, |\mathcal{E}|),
          (|\mathcal{E}|)))` if :obj:`return_attention_weights=True`
    """
    _cached_edge_index: Optional[OptPairTensor]
    _cached_adj_t: Optional[SparseTensor]
    _alpha: OptTensor

    def __init__(self, channels: int, eps: float = 0.1, dropout: float = 0.0,
                 cached: bool = False, add_self_loops: bool = True,
                 normalize: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.channels = channels
        self.eps = eps
        self.dropout = dropout
        self.cached = cached
        self.add_self_loops = add_self_loops
        self.normalize = normalize

        self._cached_edge_index = None
        self._cached_adj_t = None
        self._alpha = None

        self.att_l = Linear(channels, 1, bias=False)
        self.att_r = Linear(channels, 1, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.att_l.reset_parameters()
        self.att_r.reset_parameters()
        self._cached_edge_index = None
        self._cached_adj_t = None

    def forward(self, x: Tensor, x_0: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None, return_attention_weights=None):
        # type: (Tensor, Tensor, Tensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Tensor, Tensor, SparseTensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Tensor, Tensor, Tensor, OptTensor, bool) -> Tuple[Tensor, Tuple[Tensor, Tensor]]  # noqa
        # type: (Tensor, Tensor, SparseTensor, OptTensor, bool) -> Tuple[Tensor, SparseTensor]  # noqa
        r"""Runs the forward pass of the module.

        Args:
            return_attention_weights (bool, optional): If set to :obj:`True`,
                will additionally return the tuple
                :obj:`(edge_index, attention_weights)`, holding the computed
                attention weights for each edge. (default: :obj:`None`)
        """
        if self.normalize:
            if isinstance(edge_index, Tensor):
                assert edge_weight is None
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, edge_weight = gcn_norm(  # yapf: disable
                        edge_index, None, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, edge_weight)
                else:
                    edge_index, edge_weight = cache[0], cache[1]

            elif isinstance(edge_index, SparseTensor):
                assert not edge_index.has_value()
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, None, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache
        else:
            if isinstance(edge_index,
                          Tensor) and not is_torch_sparse_tensor(edge_index):
                assert edge_weight is not None
            elif isinstance(edge_index, SparseTensor):
                assert edge_index.has_value()

        alpha_l = self.att_l(x)
        alpha_r = self.att_r(x)

        # propagate_type: (x: Tensor, alpha: PairTensor, edge_weight: OptTensor)  # noqa
        out = self.propagate(edge_index, x=x, alpha=(alpha_l, alpha_r),
                             edge_weight=edge_weight, size=None)

        alpha = self._alpha
        self._alpha = None

        if self.eps != 0.0:
            out = out + self.eps * x_0

        if isinstance(return_attention_weights, bool):
            assert alpha is not None
            if isinstance(edge_index, Tensor):
                if is_torch_sparse_tensor(edge_index):
                    # TODO TorchScript requires to return a tuple
                    adj = set_sparse_value(edge_index, alpha)
                    return out, (adj, alpha)
                else:
                    return out, (edge_index, alpha)
            elif isinstance(edge_index, SparseTensor):
                return out, edge_index.set_value(alpha, layout='coo')
        else:
            return out

    def message(self, x_j: Tensor, alpha_j: Tensor, alpha_i: Tensor,
                edge_weight: OptTensor) -> Tensor:
        assert edge_weight is not None
        alpha = (alpha_j + alpha_i).tanh().squeeze(-1)
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return x_j * (alpha * edge_weight).view(-1, 1)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.channels}, eps={self.eps})'
pytorch/torch_geometric/nn/conv/feast_conv.py000066400000000000000000000105451444262304300220130ustar00rootroot00000000000000from typing import Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import normal
from torch_geometric.typing import Adj, PairTensor, SparseTensor, torch_sparse
from torch_geometric.utils import add_self_loops, remove_self_loops


class FeaStConv(MessagePassing):
    r"""The (translation-invariant) feature-steered convolutional operator from
    the `"FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{1}{|\mathcal{N}(i)|}
        \sum_{j \in \mathcal{N}(i)} \sum_{h=1}^H
        q_h(\mathbf{x}_i, \mathbf{x}_j) \mathbf{W}_h \mathbf{x}_j

    with :math:`q_h(\mathbf{x}_i, \mathbf{x}_j) = \mathrm{softmax}_j
    (\mathbf{u}_h^{\top} (\mathbf{x}_j - \mathbf{x}_i) + c_h)`, where :math:`H`
    denotes the number of attention heads, and :math:`\mathbf{W}_h`,
    :math:`\mathbf{u}_h` and :math:`c_h` are trainable parameters.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        heads (int, optional): Number of attention heads :math:`H`.
            (default: :obj:`1`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{in}), (|\mathcal{V_t}|, F_{in}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: int, out_channels: int, heads: int = 1,
                 add_self_loops: bool = True, bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'mean')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.add_self_loops = add_self_loops

        self.lin = Linear(in_channels, heads * out_channels, bias=False,
                          weight_initializer='uniform')
        self.u = Linear(in_channels, heads, bias=False,
                        weight_initializer='uniform')
        self.c = Parameter(torch.Tensor(heads))

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        self.u.reset_parameters()
        normal(self.c, mean=0, std=0.1)
        normal(self.bias, mean=0, std=0.1)

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj) -> Tensor:
        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(edge_index,
                                               num_nodes=x[1].size(0))
            elif isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.set_diag(edge_index)

        # propagate_type: (x: PairTensor)
        out = self.propagate(edge_index, x=x, size=None)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_i: Tensor, x_j: Tensor) -> Tensor:
        q = self.u(x_j - x_i) + self.c  # Translation invariance.
        q = F.softmax(q, dim=1)
        x_j = self.lin(x_j).view(x_j.size(0), self.heads, -1)
        return (x_j * q.view(-1, self.heads, 1)).sum(dim=1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/film_conv.py000066400000000000000000000142741444262304300216430ustar00rootroot00000000000000import copy
from typing import Callable, Optional, Tuple, Union

from torch import Tensor
from torch.nn import ModuleList, ReLU

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.typing import (
    Adj,
    OptTensor,
    PairTensor,
    SparseTensor,
    torch_sparse,
)


class FiLMConv(MessagePassing):
    r"""The FiLM graph convolutional operator from the
    `"GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \sum_{r \in \mathcal{R}}
        \sum_{j \in \mathcal{N}(i)} \sigma \left(
        \boldsymbol{\gamma}_{r,i} \odot \mathbf{W}_r \mathbf{x}_j +
        \boldsymbol{\beta}_{r,i} \right)

    where :math:`\boldsymbol{\beta}_{r,i}, \boldsymbol{\gamma}_{r,i} =
    g(\mathbf{x}_i)` with :math:`g` being a single linear layer by default.
    Self-loops are automatically added to the input graph and represented as
    its own relation type.

    .. note::

        For an example of using FiLM, see `examples/gcn.py
        `_.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        num_relations (int, optional): Number of relations. (default: :obj:`1`)
        nn (torch.nn.Module, optional): The neural network :math:`g` that
            maps node features :obj:`x_i` of shape
            :obj:`[-1, in_channels]` to shape :obj:`[-1, 2 * out_channels]`.
            If set to :obj:`None`, :math:`g` will be implemented as a single
            linear layer. (default: :obj:`None`)
        act (callable, optional): Activation function :math:`\sigma`.
            (default: :meth:`torch.nn.ReLU()`)
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"mean"`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge types :math:`(|\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(
            self,
            in_channels: Union[int, Tuple[int, int]],
            out_channels: int,
            num_relations: int = 1,
            nn: Optional[Callable] = None,
            act: Optional[Callable] = ReLU(),
            aggr: str = 'mean',
            **kwargs,
    ):
        super().__init__(aggr=aggr, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_relations = max(num_relations, 1)
        self.act = act

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lins = ModuleList()
        self.films = ModuleList()
        for _ in range(num_relations):
            self.lins.append(Linear(in_channels[0], out_channels, bias=False))
            if nn is None:
                film = Linear(in_channels[1], 2 * out_channels)
            else:
                film = copy.deepcopy(nn)
            self.films.append(film)

        self.lin_skip = Linear(in_channels[1], self.out_channels, bias=False)
        if nn is None:
            self.film_skip = Linear(in_channels[1], 2 * self.out_channels,
                                    bias=False)
        else:
            self.film_skip = copy.deepcopy(nn)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        for lin, film in zip(self.lins, self.films):
            lin.reset_parameters()
            reset(film)
        self.lin_skip.reset_parameters()
        reset(self.film_skip)

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj,
                edge_type: OptTensor = None) -> Tensor:

        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        beta, gamma = self.film_skip(x[1]).split(self.out_channels, dim=-1)
        out = gamma * self.lin_skip(x[1]) + beta
        if self.act is not None:
            out = self.act(out)

        # propagate_type: (x: Tensor, beta: Tensor, gamma: Tensor)
        if self.num_relations <= 1:
            beta, gamma = self.films[0](x[1]).split(self.out_channels, dim=-1)
            out = out + self.propagate(edge_index, x=self.lins[0](x[0]),
                                       beta=beta, gamma=gamma, size=None)
        else:
            for i, (lin, film) in enumerate(zip(self.lins, self.films)):
                beta, gamma = film(x[1]).split(self.out_channels, dim=-1)
                if isinstance(edge_index, SparseTensor):
                    edge_type = edge_index.storage.value()
                    assert edge_type is not None
                    mask = edge_type == i
                    adj_t = torch_sparse.masked_select_nnz(
                        edge_index, mask, layout='coo')
                    out = out + self.propagate(adj_t, x=lin(x[0]), beta=beta,
                                               gamma=gamma, size=None)
                else:
                    assert edge_type is not None
                    mask = edge_type == i
                    out = out + self.propagate(edge_index[:, mask], x=lin(
                        x[0]), beta=beta, gamma=gamma, size=None)

        return out

    def message(self, x_j: Tensor, beta_i: Tensor, gamma_i: Tensor) -> Tensor:
        out = gamma_i * x_j + beta_i
        if self.act is not None:
            out = self.act(out)
        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_relations={self.num_relations})')
pytorch/torch_geometric/nn/conv/fused_gat_conv.py000066400000000000000000000102221444262304300226420ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.nn.conv import GATConv
from torch_geometric.typing import SparseTensor


class FusedGATConv(GATConv):  # pragma: no cover
    r"""The fused graph attention operator from the
    `"Understanding GNN Computational Graph: A Coordinated Computation, IO, and
    Memory Perspective"
    `_ paper.

    :class:`FusedGATConv` is an optimized version of
    :class:`~torch_geometric.nn.conv.GATConv` based on the :obj:`dgNN` package
    that fuses message passing computation for accelerated execution and lower
    memory footprint.

    .. note::

        This implementation is based on the :obj:`dgNN` package.
        See `here `__ for instructions on how
        to install.
    """
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        if self.add_self_loops:
            raise ValueError(f"'{self.__class__.__name__}' does not support "
                             f"adding self-loops. Please add them manually "
                             f"in a pre-processing step and set "
                             f"`add_self_loops=False`.")

        if self.edge_dim is not None:
            raise ValueError(f"'{self.__class__.__name__}' does not support "
                             f"edge features. Set `edge_dim=None` in order "
                             f"to proceed.")

        from dgNN.operators import GATConvFuse
        self.op = GATConvFuse

    @staticmethod
    def to_graph_format(
        edge_index: Tensor,
        size: Optional[Tuple[int, int]] = None,
    ) -> Tuple[Tuple[Tensor, Tensor], Tuple[Tensor, Tensor], Tensor]:
        r"""Converts an :obj:`edge_index` representation of a graph to the
        desired input format of :class:`FusedGATConv`.

        Args:
            edge_index (torch.Tensor): The edge indices.
            size ((int, int), optional). The shape of :obj:`edge_index` in each
                dimension. (default: :obj:`None`)
        """
        value = torch.arange(edge_index.size(1), dtype=torch.int,
                             device=edge_index.device)

        adj = SparseTensor.from_edge_index(edge_index, sparse_sizes=size)
        adj.set_value_(value, layout='csr')

        rowptr, col, _ = adj.csr()
        colptr, row, perm = adj.csc()

        return (rowptr.int(), col.int()), (row.int(), colptr.int()), perm

    def forward(self, x: Tensor, csr: Tuple[Tensor, Tensor],
                csc: Tuple[Tensor, Tensor], perm: Tensor) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor): The node features.
            csr ((torch.Tensor, torch.Tensor)): A tuple containing the CSR
                representation of a graph, given as a tuple of
                :obj:`(rowptr, col)`.
            csc ((torch.Tensor, torch.Tensor)): A tuple containing the CSC
                representation of a graph, given as a tuple of
                :obj:`(row, colptr)`.
            perm (torch.Tensor): Permutation tensor to map the CSR
                representation to the CSC representation.

        .. note::

            Use the
            :meth:`~torch_geometric.nn.conv.FusedGATConv.to_graph_format`
            method to obtain the :obj:`(csr, csc, perm)` graph format from an
            existing :obj:`edge_index` representation.
        """
        H, C = self.heads, self.out_channels

        assert x.dim() == 2, "Static graphs not supported in 'GATConv'"
        x = self.lin_src(x).view(-1, H, C)

        alpha_src = (x * self.att_src).sum(dim=-1)
        alpha_dst = (x * self.att_dst).sum(dim=-1)

        dropout = self.dropout if self.training else 0.0

        (rowptr, col), (row, colptr) = csr, csc
        out = self.op(alpha_dst, alpha_src, rowptr, col, colptr, row, perm,
                      self.negative_slope, x, dropout)

        if self.concat:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.bias is not None:
            out += self.bias

        return out
pytorch/torch_geometric/nn/conv/gat_conv.py000066400000000000000000000324521444262304300214650ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import NoneType  # noqa
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import (
    add_self_loops,
    is_torch_sparse_tensor,
    remove_self_loops,
    softmax,
)
from torch_geometric.utils.sparse import set_sparse_value


class GATConv(MessagePassing):
    r"""The graph attentional operator from the `"Graph Attention Networks"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \alpha_{i,i}\mathbf{\Theta}\mathbf{x}_{i} +
        \sum_{j \in \mathcal{N}(i)} \alpha_{i,j}\mathbf{\Theta}\mathbf{x}_{j},

    where the attention coefficients :math:`\alpha_{i,j}` are computed as

    .. math::
        \alpha_{i,j} =
        \frac{
        \exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{\top}
        [\mathbf{\Theta}\mathbf{x}_i \, \Vert \, \mathbf{\Theta}\mathbf{x}_j]
        \right)\right)}
        {\sum_{k \in \mathcal{N}(i) \cup \{ i \}}
        \exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{\top}
        [\mathbf{\Theta}\mathbf{x}_i \, \Vert \, \mathbf{\Theta}\mathbf{x}_k]
        \right)\right)}.

    If the graph has multi-dimensional edge features :math:`\mathbf{e}_{i,j}`,
    the attention coefficients :math:`\alpha_{i,j}` are computed as

    .. math::
        \alpha_{i,j} =
        \frac{
        \exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{\top}
        [\mathbf{\Theta}\mathbf{x}_i \, \Vert \, \mathbf{\Theta}\mathbf{x}_j
        \, \Vert \, \mathbf{\Theta}_{e} \mathbf{e}_{i,j}]\right)\right)}
        {\sum_{k \in \mathcal{N}(i) \cup \{ i \}}
        \exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{\top}
        [\mathbf{\Theta}\mathbf{x}_i \, \Vert \, \mathbf{\Theta}\mathbf{x}_k
        \, \Vert \, \mathbf{\Theta}_{e} \mathbf{e}_{i,k}]\right)\right)}.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        edge_dim (int, optional): Edge feature dimensionality (in case
            there are any). (default: :obj:`None`)
        fill_value (float or torch.Tensor or str, optional): The way to
            generate edge features of self-loops (in case
            :obj:`edge_dim != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`"mean"`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, H * F_{out})` or
          :math:`((|\mathcal{V}_t|, H * F_{out})` if bipartite.
          If :obj:`return_attention_weights=True`, then
          :math:`((|\mathcal{V}|, H * F_{out}),
          ((2, |\mathcal{E}|), (|\mathcal{E}|, H)))`
          or :math:`((|\mathcal{V_t}|, H * F_{out}), ((2, |\mathcal{E}|),
          (|\mathcal{E}|, H)))` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        heads: int = 1,
        concat: bool = True,
        negative_slope: float = 0.2,
        dropout: float = 0.0,
        add_self_loops: bool = True,
        edge_dim: Optional[int] = None,
        fill_value: Union[float, Tensor, str] = 'mean',
        bias: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.add_self_loops = add_self_loops
        self.edge_dim = edge_dim
        self.fill_value = fill_value

        # In case we are operating in bipartite graphs, we apply separate
        # transformations 'lin_src' and 'lin_dst' to source and target nodes:
        if isinstance(in_channels, int):
            self.lin_src = Linear(in_channels, heads * out_channels,
                                  bias=False, weight_initializer='glorot')
            self.lin_dst = self.lin_src
        else:
            self.lin_src = Linear(in_channels[0], heads * out_channels, False,
                                  weight_initializer='glorot')
            self.lin_dst = Linear(in_channels[1], heads * out_channels, False,
                                  weight_initializer='glorot')

        # The learnable parameters to compute attention coefficients:
        self.att_src = Parameter(torch.Tensor(1, heads, out_channels))
        self.att_dst = Parameter(torch.Tensor(1, heads, out_channels))

        if edge_dim is not None:
            self.lin_edge = Linear(edge_dim, heads * out_channels, bias=False,
                                   weight_initializer='glorot')
            self.att_edge = Parameter(torch.Tensor(1, heads, out_channels))
        else:
            self.lin_edge = None
            self.register_parameter('att_edge', None)

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_src.reset_parameters()
        self.lin_dst.reset_parameters()
        if self.lin_edge is not None:
            self.lin_edge.reset_parameters()
        glorot(self.att_src)
        glorot(self.att_dst)
        glorot(self.att_edge)
        zeros(self.bias)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None,
                return_attention_weights=None):
        # type: (Union[Tensor, OptPairTensor], Tensor, OptTensor, Size, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, OptPairTensor], SparseTensor, OptTensor, Size, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, OptPairTensor], Tensor, OptTensor, Size, bool) -> Tuple[Tensor, Tuple[Tensor, Tensor]]  # noqa
        # type: (Union[Tensor, OptPairTensor], SparseTensor, OptTensor, Size, bool) -> Tuple[Tensor, SparseTensor]  # noqa
        r"""Runs the forward pass of the module.

        Args:
            return_attention_weights (bool, optional): If set to :obj:`True`,
                will additionally return the tuple
                :obj:`(edge_index, attention_weights)`, holding the computed
                attention weights for each edge. (default: :obj:`None`)
        """
        # NOTE: attention weights will be returned whenever
        # `return_attention_weights` is set to a value, regardless of its
        # actual value (might be `True` or `False`). This is a current somewhat
        # hacky workaround to allow for TorchScript support via the
        # `torch.jit._overload` decorator, as we can only change the output
        # arguments conditioned on type (`None` or `bool`), not based on its
        # actual value.

        H, C = self.heads, self.out_channels

        # We first transform the input node features. If a tuple is passed, we
        # transform source and target node features via separate weights:
        if isinstance(x, Tensor):
            assert x.dim() == 2, "Static graphs not supported in 'GATConv'"
            x_src = x_dst = self.lin_src(x).view(-1, H, C)
        else:  # Tuple of source and target node features:
            x_src, x_dst = x
            assert x_src.dim() == 2, "Static graphs not supported in 'GATConv'"
            x_src = self.lin_src(x_src).view(-1, H, C)
            if x_dst is not None:
                x_dst = self.lin_dst(x_dst).view(-1, H, C)

        x = (x_src, x_dst)

        # Next, we compute node-level attention coefficients, both for source
        # and target nodes (if present):
        alpha_src = (x_src * self.att_src).sum(dim=-1)
        alpha_dst = None if x_dst is None else (x_dst * self.att_dst).sum(-1)
        alpha = (alpha_src, alpha_dst)

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                # We only want to add self-loops for nodes that appear both as
                # source and target nodes:
                num_nodes = x_src.size(0)
                if x_dst is not None:
                    num_nodes = min(num_nodes, x_dst.size(0))
                num_nodes = min(size) if size is not None else num_nodes
                edge_index, edge_attr = remove_self_loops(
                    edge_index, edge_attr)
                edge_index, edge_attr = add_self_loops(
                    edge_index, edge_attr, fill_value=self.fill_value,
                    num_nodes=num_nodes)
            elif isinstance(edge_index, SparseTensor):
                if self.edge_dim is None:
                    edge_index = torch_sparse.set_diag(edge_index)
                else:
                    raise NotImplementedError(
                        "The usage of 'edge_attr' and 'add_self_loops' "
                        "simultaneously is currently not yet supported for "
                        "'edge_index' in a 'SparseTensor' form")

        # edge_updater_type: (alpha: OptPairTensor, edge_attr: OptTensor)
        alpha = self.edge_updater(edge_index, alpha=alpha, edge_attr=edge_attr)

        # propagate_type: (x: OptPairTensor, alpha: Tensor)
        out = self.propagate(edge_index, x=x, alpha=alpha, size=size)

        if self.concat:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.bias is not None:
            out = out + self.bias

        if isinstance(return_attention_weights, bool):
            if isinstance(edge_index, Tensor):
                if is_torch_sparse_tensor(edge_index):
                    # TODO TorchScript requires to return a tuple
                    adj = set_sparse_value(edge_index, alpha)
                    return out, (adj, alpha)
                else:
                    return out, (edge_index, alpha)
            elif isinstance(edge_index, SparseTensor):
                return out, edge_index.set_value(alpha, layout='coo')
        else:
            return out

    def edge_update(self, alpha_j: Tensor, alpha_i: OptTensor,
                    edge_attr: OptTensor, index: Tensor, ptr: OptTensor,
                    size_i: Optional[int]) -> Tensor:
        # Given edge-level attention coefficients for source and target nodes,
        # we simply need to sum them up to "emulate" concatenation:
        alpha = alpha_j if alpha_i is None else alpha_j + alpha_i
        if index.numel() == 0:
            return alpha
        if edge_attr is not None and self.lin_edge is not None:
            if edge_attr.dim() == 1:
                edge_attr = edge_attr.view(-1, 1)
            edge_attr = self.lin_edge(edge_attr)
            edge_attr = edge_attr.view(-1, self.heads, self.out_channels)
            alpha_edge = (edge_attr * self.att_edge).sum(dim=-1)
            alpha = alpha + alpha_edge

        alpha = F.leaky_relu(alpha, self.negative_slope)
        alpha = softmax(alpha, index, ptr, size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return alpha

    def message(self, x_j: Tensor, alpha: Tensor) -> Tensor:
        return alpha.unsqueeze(-1) * x_j

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/gated_graph_conv.py000066400000000000000000000067761444262304300231710ustar00rootroot00000000000000import torch
from torch import Tensor
from torch.nn import Parameter as Param

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import uniform
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class GatedGraphConv(MessagePassing):
    r"""The gated graph convolution operator from the `"Gated Graph Sequence
    Neural Networks" `_ paper

    .. math::
        \mathbf{h}_i^{(0)} &= \mathbf{x}_i \, \Vert \, \mathbf{0}

        \mathbf{m}_i^{(l+1)} &= \sum_{j \in \mathcal{N}(i)} e_{j,i} \cdot
        \mathbf{\Theta} \cdot \mathbf{h}_j^{(l)}

        \mathbf{h}_i^{(l+1)} &= \textrm{GRU} (\mathbf{m}_i^{(l+1)},
        \mathbf{h}_i^{(l)})

    up to representation :math:`\mathbf{h}_i^{(L)}`.
    The number of input channels of :math:`\mathbf{x}_i` needs to be less or
    equal than :obj:`out_channels`.
    :math:`e_{j,i}` denotes the edge weight from source node :obj:`j` to target
    node :obj:`i` (default: :obj:`1`)

    Args:
        out_channels (int): Size of each output sample.
        num_layers (int): The sequence length :math:`L`.
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"add"`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`

    """
    def __init__(self, out_channels: int, num_layers: int, aggr: str = 'add',
                 bias: bool = True, **kwargs):
        super().__init__(aggr=aggr, **kwargs)

        self.out_channels = out_channels
        self.num_layers = num_layers

        self.weight = Param(Tensor(num_layers, out_channels, out_channels))
        self.rnn = torch.nn.GRUCell(out_channels, out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        uniform(self.out_channels, self.weight)
        self.rnn.reset_parameters()

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if x.size(-1) > self.out_channels:
            raise ValueError('The number of input channels is not allowed to '
                             'be larger than the number of output channels')

        if x.size(-1) < self.out_channels:
            zero = x.new_zeros(x.size(0), self.out_channels - x.size(-1))
            x = torch.cat([x, zero], dim=1)

        for i in range(self.num_layers):
            m = torch.matmul(x, self.weight[i])
            # propagate_type: (x: Tensor, edge_weight: OptTensor)
            m = self.propagate(edge_index, x=m, edge_weight=edge_weight,
                               size=None)
            x = self.rnn(m, x)

        return x

    def message(self, x_j: Tensor, edge_weight: OptTensor):
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.out_channels}, '
                f'num_layers={self.num_layers})')
pytorch/torch_geometric/nn/conv/gatv2_conv.py000066400000000000000000000302071444262304300217310ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import (
    Adj,
    OptTensor,
    PairTensor,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import (
    add_self_loops,
    is_torch_sparse_tensor,
    remove_self_loops,
    softmax,
)
from torch_geometric.utils.sparse import set_sparse_value


class GATv2Conv(MessagePassing):
    r"""The GATv2 operator from the `"How Attentive are Graph Attention
    Networks?" `_ paper, which fixes the
    static attention problem of the standard
    :class:`~torch_geometric.conv.GATConv` layer.
    Since the linear layers in the standard GAT are applied right after each
    other, the ranking of attended nodes is unconditioned on the query node.
    In contrast, in :class:`GATv2`, every node can attend to any other node.

    .. math::
        \mathbf{x}^{\prime}_i = \alpha_{i,i}\mathbf{\Theta}\mathbf{x}_{i} +
        \sum_{j \in \mathcal{N}(i)} \alpha_{i,j}\mathbf{\Theta}\mathbf{x}_{j},

    where the attention coefficients :math:`\alpha_{i,j}` are computed as

    .. math::
        \alpha_{i,j} =
        \frac{
        \exp\left(\mathbf{a}^{\top}\mathrm{LeakyReLU}\left(\mathbf{\Theta}
        [\mathbf{x}_i \, \Vert \, \mathbf{x}_j]
        \right)\right)}
        {\sum_{k \in \mathcal{N}(i) \cup \{ i \}}
        \exp\left(\mathbf{a}^{\top}\mathrm{LeakyReLU}\left(\mathbf{\Theta}
        [\mathbf{x}_i \, \Vert \, \mathbf{x}_k]
        \right)\right)}.

    If the graph has multi-dimensional edge features :math:`\mathbf{e}_{i,j}`,
    the attention coefficients :math:`\alpha_{i,j}` are computed as

    .. math::
        \alpha_{i,j} =
        \frac{
        \exp\left(\mathbf{a}^{\top}\mathrm{LeakyReLU}\left(\mathbf{\Theta}
        [\mathbf{x}_i \, \Vert \, \mathbf{x}_j \, \Vert \, \mathbf{e}_{i,j}]
        \right)\right)}
        {\sum_{k \in \mathcal{N}(i) \cup \{ i \}}
        \exp\left(\mathbf{a}^{\top}\mathrm{LeakyReLU}\left(\mathbf{\Theta}
        [\mathbf{x}_i \, \Vert \, \mathbf{x}_k \, \Vert \, \mathbf{e}_{i,k}]
        \right)\right)}.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        edge_dim (int, optional): Edge feature dimensionality (in case
            there are any). (default: :obj:`None`)
        fill_value (float or torch.Tensor or str, optional): The way to
            generate edge features of self-loops
            (in case :obj:`edge_dim != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`"mean"`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        share_weights (bool, optional): If set to :obj:`True`, the same matrix
            will be applied to the source and the target node of every edge.
            (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, H * F_{out})` or
          :math:`((|\mathcal{V}_t|, H * F_{out})` if bipartite.
          If :obj:`return_attention_weights=True`, then
          :math:`((|\mathcal{V}|, H * F_{out}),
          ((2, |\mathcal{E}|), (|\mathcal{E}|, H)))`
          or :math:`((|\mathcal{V_t}|, H * F_{out}), ((2, |\mathcal{E}|),
          (|\mathcal{E}|, H)))` if bipartite
    """
    _alpha: OptTensor

    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        heads: int = 1,
        concat: bool = True,
        negative_slope: float = 0.2,
        dropout: float = 0.0,
        add_self_loops: bool = True,
        edge_dim: Optional[int] = None,
        fill_value: Union[float, Tensor, str] = 'mean',
        bias: bool = True,
        share_weights: bool = False,
        **kwargs,
    ):
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.add_self_loops = add_self_loops
        self.edge_dim = edge_dim
        self.fill_value = fill_value
        self.share_weights = share_weights

        if isinstance(in_channels, int):
            self.lin_l = Linear(in_channels, heads * out_channels, bias=bias,
                                weight_initializer='glorot')
            if share_weights:
                self.lin_r = self.lin_l
            else:
                self.lin_r = Linear(in_channels, heads * out_channels,
                                    bias=bias, weight_initializer='glorot')
        else:
            self.lin_l = Linear(in_channels[0], heads * out_channels,
                                bias=bias, weight_initializer='glorot')
            if share_weights:
                self.lin_r = self.lin_l
            else:
                self.lin_r = Linear(in_channels[1], heads * out_channels,
                                    bias=bias, weight_initializer='glorot')

        self.att = Parameter(torch.Tensor(1, heads, out_channels))

        if edge_dim is not None:
            self.lin_edge = Linear(edge_dim, heads * out_channels, bias=False,
                                   weight_initializer='glorot')
        else:
            self.lin_edge = None

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self._alpha = None

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_l.reset_parameters()
        self.lin_r.reset_parameters()
        if self.lin_edge is not None:
            self.lin_edge.reset_parameters()
        glorot(self.att)
        zeros(self.bias)

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj,
                edge_attr: OptTensor = None,
                return_attention_weights: bool = None):
        # type: (Union[Tensor, PairTensor], Tensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, PairTensor], Tensor, OptTensor, bool) -> Tuple[Tensor, Tuple[Tensor, Tensor]]  # noqa
        # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, bool) -> Tuple[Tensor, SparseTensor]  # noqa
        r"""Runs the forward pass of the module.

        Args:
            return_attention_weights (bool, optional): If set to :obj:`True`,
                will additionally return the tuple
                :obj:`(edge_index, attention_weights)`, holding the computed
                attention weights for each edge. (default: :obj:`None`)
        """
        H, C = self.heads, self.out_channels

        x_l: OptTensor = None
        x_r: OptTensor = None
        if isinstance(x, Tensor):
            assert x.dim() == 2
            x_l = self.lin_l(x).view(-1, H, C)
            if self.share_weights:
                x_r = x_l
            else:
                x_r = self.lin_r(x).view(-1, H, C)
        else:
            x_l, x_r = x[0], x[1]
            assert x[0].dim() == 2
            x_l = self.lin_l(x_l).view(-1, H, C)
            if x_r is not None:
                x_r = self.lin_r(x_r).view(-1, H, C)

        assert x_l is not None
        assert x_r is not None

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                num_nodes = x_l.size(0)
                if x_r is not None:
                    num_nodes = min(num_nodes, x_r.size(0))
                edge_index, edge_attr = remove_self_loops(
                    edge_index, edge_attr)
                edge_index, edge_attr = add_self_loops(
                    edge_index, edge_attr, fill_value=self.fill_value,
                    num_nodes=num_nodes)
            elif isinstance(edge_index, SparseTensor):
                if self.edge_dim is None:
                    edge_index = torch_sparse.set_diag(edge_index)
                else:
                    raise NotImplementedError(
                        "The usage of 'edge_attr' and 'add_self_loops' "
                        "simultaneously is currently not yet supported for "
                        "'edge_index' in a 'SparseTensor' form")

        # propagate_type: (x: PairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=(x_l, x_r), edge_attr=edge_attr,
                             size=None)

        alpha = self._alpha
        assert alpha is not None
        self._alpha = None

        if self.concat:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.bias is not None:
            out = out + self.bias

        if isinstance(return_attention_weights, bool):
            if isinstance(edge_index, Tensor):
                if is_torch_sparse_tensor(edge_index):
                    # TODO TorchScript requires to return a tuple
                    adj = set_sparse_value(edge_index, alpha)
                    return out, (adj, alpha)
                else:
                    return out, (edge_index, alpha)
            elif isinstance(edge_index, SparseTensor):
                return out, edge_index.set_value(alpha, layout='coo')
        else:
            return out

    def message(self, x_j: Tensor, x_i: Tensor, edge_attr: OptTensor,
                index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:
        x = x_i + x_j

        if edge_attr is not None:
            if edge_attr.dim() == 1:
                edge_attr = edge_attr.view(-1, 1)
            assert self.lin_edge is not None
            edge_attr = self.lin_edge(edge_attr)
            edge_attr = edge_attr.view(-1, self.heads, self.out_channels)
            x = x + edge_attr

        x = F.leaky_relu(x, self.negative_slope)
        alpha = (x * self.att).sum(dim=-1)
        alpha = softmax(alpha, index, ptr, size_i)
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return x_j * alpha.unsqueeze(-1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/gcn2_conv.py000066400000000000000000000155561444262304300215510ustar00rootroot00000000000000from math import log
from typing import Optional

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.inits import glorot
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class GCN2Conv(MessagePassing):
    r"""The graph convolutional operator with initial residual connections and
    identity mapping (GCNII) from the `"Simple and Deep Graph Convolutional
    Networks" `_ paper

    .. math::
        \mathbf{X}^{\prime} = \left( (1 - \alpha) \mathbf{\hat{P}}\mathbf{X} +
        \alpha \mathbf{X^{(0)}}\right) \left( (1 - \beta) \mathbf{I} + \beta
        \mathbf{\Theta} \right)

    with :math:`\mathbf{\hat{P}} = \mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
    \mathbf{\hat{D}}^{-1/2}`, where
    :math:`\mathbf{\hat{A}} = \mathbf{A} + \mathbf{I}` denotes the adjacency
    matrix with inserted self-loops and
    :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij}` its diagonal degree matrix,
    and :math:`\mathbf{X}^{(0)}` being the initial feature representation.
    Here, :math:`\alpha` models the strength of the initial residual
    connection, while :math:`\beta` models the strength of the identity
    mapping.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.

    Args:
        channels (int): Size of each input and output sample.
        alpha (float): The strength of the initial residual connection
            :math:`\alpha`.
        theta (float, optional): The hyperparameter :math:`\theta` to compute
            the strength of the identity mapping
            :math:`\beta = \log \left( \frac{\theta}{\ell} + 1 \right)`.
            (default: :obj:`None`)
        layer (int, optional): The layer :math:`\ell` in which this module is
            executed. (default: :obj:`None`)
        shared_weights (bool, optional): If set to :obj:`False`, will use
            different weight matrices for the smoothed representation and the
            initial residual ("GCNII*"). (default: :obj:`True`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
            \mathbf{\hat{D}}^{-1/2}` on first execution, and will use the
            cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        normalize (bool, optional): Whether to add self-loops and apply
            symmetric normalization. (default: :obj:`True`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)`,
          initial node features :math:`(|\mathcal{V}|, F)`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)`
    """
    _cached_edge_index: Optional[OptPairTensor]
    _cached_adj_t: Optional[SparseTensor]

    def __init__(self, channels: int, alpha: float, theta: float = None,
                 layer: int = None, shared_weights: bool = True,
                 cached: bool = False, add_self_loops: bool = True,
                 normalize: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.channels = channels
        self.alpha = alpha
        self.beta = 1.
        if theta is not None or layer is not None:
            assert theta is not None and layer is not None
            self.beta = log(theta / layer + 1)
        self.cached = cached
        self.normalize = normalize
        self.add_self_loops = add_self_loops

        self._cached_edge_index = None
        self._cached_adj_t = None

        self.weight1 = Parameter(torch.Tensor(channels, channels))

        if shared_weights:
            self.register_parameter('weight2', None)
        else:
            self.weight2 = Parameter(torch.Tensor(channels, channels))

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        glorot(self.weight1)
        glorot(self.weight2)
        self._cached_edge_index = None
        self._cached_adj_t = None

    def forward(self, x: Tensor, x_0: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if self.normalize:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, edge_weight = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, edge_weight)
                else:
                    edge_index, edge_weight = cache[0], cache[1]

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim), False,
                        self.add_self_loops, self.flow, dtype=x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        x = self.propagate(edge_index, x=x, edge_weight=edge_weight, size=None)

        x.mul_(1 - self.alpha)
        x_0 = self.alpha * x_0[:x.size(0)]

        if self.weight2 is None:
            out = x.add_(x_0)
            out = torch.addmm(out, out, self.weight1, beta=1. - self.beta,
                              alpha=self.beta)
        else:
            out = torch.addmm(x, x, self.weight1, beta=1. - self.beta,
                              alpha=self.beta)
            out = out + torch.addmm(x_0, x_0, self.weight2,
                                    beta=1. - self.beta, alpha=self.beta)

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.channels}, '
                f'alpha={self.alpha}, beta={self.beta})')
pytorch/torch_geometric/nn/conv/gcn_conv.py000066400000000000000000000221651444262304300214610ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import add_remaining_self_loops
from torch_geometric.utils import add_self_loops as add_self_loops_fn
from torch_geometric.utils import (
    is_torch_sparse_tensor,
    scatter,
    spmm,
    to_edge_index,
)
from torch_geometric.utils.num_nodes import maybe_num_nodes
from torch_geometric.utils.sparse import set_sparse_value


@torch.jit._overload
def gcn_norm(edge_index, edge_weight, num_nodes, improved, add_self_loops,
             flow, dtype):
    # type: (Tensor, OptTensor, Optional[int], bool, bool, str, Optional[int]) -> OptPairTensor  # noqa
    pass


@torch.jit._overload
def gcn_norm(edge_index, edge_weight, num_nodes, improved, add_self_loops,
             flow, dtype):
    # type: (SparseTensor, OptTensor, Optional[int], bool, bool, str, Optional[int]) -> SparseTensor  # noqa
    pass


def gcn_norm(edge_index, edge_weight=None, num_nodes=None, improved=False,
             add_self_loops=True, flow="source_to_target", dtype=None):

    fill_value = 2. if improved else 1.

    if isinstance(edge_index, SparseTensor):
        assert edge_index.size(0) == edge_index.size(1)

        adj_t = edge_index

        if not adj_t.has_value():
            adj_t = adj_t.fill_value(1., dtype=dtype)
        if add_self_loops:
            adj_t = torch_sparse.fill_diag(adj_t, fill_value)

        deg = torch_sparse.sum(adj_t, dim=1)
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0.)
        adj_t = torch_sparse.mul(adj_t, deg_inv_sqrt.view(-1, 1))
        adj_t = torch_sparse.mul(adj_t, deg_inv_sqrt.view(1, -1))

        return adj_t

    if is_torch_sparse_tensor(edge_index):
        assert edge_index.size(0) == edge_index.size(1)

        if edge_index.layout == torch.sparse_csc:
            raise NotImplementedError("Sparse CSC matrices are not yet "
                                      "supported in 'gcn_norm'")

        adj_t = edge_index
        if add_self_loops:
            adj_t, _ = add_self_loops_fn(adj_t, None, fill_value, num_nodes)

        edge_index, value = to_edge_index(adj_t)
        col, row = edge_index[0], edge_index[1]

        deg = scatter(value, col, 0, dim_size=num_nodes, reduce='sum')
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
        value = deg_inv_sqrt[row] * value * deg_inv_sqrt[col]

        return set_sparse_value(adj_t, value), None

    assert flow in ['source_to_target', 'target_to_source']
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    if add_self_loops:
        edge_index, edge_weight = add_remaining_self_loops(
            edge_index, edge_weight, fill_value, num_nodes)

    if edge_weight is None:
        edge_weight = torch.ones((edge_index.size(1), ), dtype=dtype,
                                 device=edge_index.device)

    row, col = edge_index[0], edge_index[1]
    idx = col if flow == 'source_to_target' else row
    deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum')
    deg_inv_sqrt = deg.pow_(-0.5)
    deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
    edge_weight = deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]

    return edge_index, edge_weight


class GCNConv(MessagePassing):
    r"""The graph convolutional operator from the `"Semi-supervised
    Classification with Graph Convolutional Networks"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} = \mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
        \mathbf{\hat{D}}^{-1/2} \mathbf{X} \mathbf{\Theta},

    where :math:`\mathbf{\hat{A}} = \mathbf{A} + \mathbf{I}` denotes the
    adjacency matrix with inserted self-loops and
    :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij}` its diagonal degree matrix.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.

    Its node-wise formulation is given by:

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{\Theta}^{\top} \sum_{j \in
        \mathcal{N}(i) \cup \{ i \}} \frac{e_{j,i}}{\sqrt{\hat{d}_j
        \hat{d}_i}} \mathbf{x}_j

    with :math:`\hat{d}_i = 1 + \sum_{j \in \mathcal{N}(i)} e_{j,i}`, where
    :math:`e_{j,i}` denotes the edge weight from source node :obj:`j` to target
    node :obj:`i` (default: :obj:`1.0`)

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        improved (bool, optional): If set to :obj:`True`, the layer computes
            :math:`\mathbf{\hat{A}}` as :math:`\mathbf{A} + 2\mathbf{I}`.
            (default: :obj:`False`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
            \mathbf{\hat{D}}^{-1/2}` on first execution, and will use the
            cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        normalize (bool, optional): Whether to add self-loops and compute
            symmetric normalization coefficients on the fly.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """

    _cached_edge_index: Optional[OptPairTensor]
    _cached_adj_t: Optional[SparseTensor]

    def __init__(self, in_channels: int, out_channels: int,
                 improved: bool = False, cached: bool = False,
                 add_self_loops: bool = True, normalize: bool = True,
                 bias: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.improved = improved
        self.cached = cached
        self.add_self_loops = add_self_loops
        self.normalize = normalize

        self._cached_edge_index = None
        self._cached_adj_t = None

        self.lin = Linear(in_channels, out_channels, bias=False,
                          weight_initializer='glorot')

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        zeros(self.bias)
        self._cached_edge_index = None
        self._cached_adj_t = None

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if self.normalize:
            if isinstance(edge_index, Tensor):
                cache = self._cached_edge_index
                if cache is None:
                    edge_index, edge_weight = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim),
                        self.improved, self.add_self_loops, self.flow, x.dtype)
                    if self.cached:
                        self._cached_edge_index = (edge_index, edge_weight)
                else:
                    edge_index, edge_weight = cache[0], cache[1]

            elif isinstance(edge_index, SparseTensor):
                cache = self._cached_adj_t
                if cache is None:
                    edge_index = gcn_norm(  # yapf: disable
                        edge_index, edge_weight, x.size(self.node_dim),
                        self.improved, self.add_self_loops, self.flow, x.dtype)
                    if self.cached:
                        self._cached_adj_t = edge_index
                else:
                    edge_index = cache

        x = self.lin(x)

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=None)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)
pytorch/torch_geometric/nn/conv/gen_conv.py000066400000000000000000000234101444262304300214550ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

from torch import Tensor
from torch.nn import (
    BatchNorm1d,
    Dropout,
    InstanceNorm1d,
    LayerNorm,
    ReLU,
    Sequential,
)

from torch_geometric.nn.aggr import Aggregation, MultiAggregation
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.nn.norm import MessageNorm
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
)
from torch_geometric.utils import is_torch_sparse_tensor, to_edge_index


class MLP(Sequential):
    def __init__(self, channels: List[int], norm: Optional[str] = None,
                 bias: bool = True, dropout: float = 0.):
        m = []
        for i in range(1, len(channels)):
            m.append(Linear(channels[i - 1], channels[i], bias=bias))

            if i < len(channels) - 1:
                if norm and norm == 'batch':
                    m.append(BatchNorm1d(channels[i], affine=True))
                elif norm and norm == 'layer':
                    m.append(LayerNorm(channels[i], elementwise_affine=True))
                elif norm and norm == 'instance':
                    m.append(InstanceNorm1d(channels[i], affine=False))
                elif norm:
                    raise NotImplementedError(
                        f'Normalization layer "{norm}" not supported.')
                m.append(ReLU())
                m.append(Dropout(dropout))

        super().__init__(*m)


class GENConv(MessagePassing):
    r"""The GENeralized Graph Convolution (GENConv) from the `"DeeperGCN: All
    You Need to Train Deeper GCNs" `_ paper.
    Supports SoftMax & PowerMean aggregation. The message construction is:

    .. math::
        \mathbf{x}_i^{\prime} = \mathrm{MLP} \left( \mathbf{x}_i +
        \mathrm{AGG} \left( \left\{
        \mathrm{ReLU} \left( \mathbf{x}_j + \mathbf{e_{ji}} \right) +\epsilon
        : j \in \mathcal{N}(i) \right\} \right)
        \right)

    .. note::

        For an example of using :obj:`GENConv`, see
        `examples/ogbn_proteins_deepgcn.py
        `_.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        aggr (str or Aggregation, optional): The aggregation scheme to use.
            Any aggregation of :obj:`torch_geometric.nn.aggr` can be used,
            (:obj:`"softmax"`, :obj:`"powermean"`, :obj:`"add"`, :obj:`"mean"`,
            :obj:`max`). (default: :obj:`"softmax"`)
        t (float, optional): Initial inverse temperature for softmax
            aggregation. (default: :obj:`1.0`)
        learn_t (bool, optional): If set to :obj:`True`, will learn the value
            :obj:`t` for softmax aggregation dynamically.
            (default: :obj:`False`)
        p (float, optional): Initial power for power mean aggregation.
            (default: :obj:`1.0`)
        learn_p (bool, optional): If set to :obj:`True`, will learn the value
            :obj:`p` for power mean aggregation dynamically.
            (default: :obj:`False`)
        msg_norm (bool, optional): If set to :obj:`True`, will use message
            normalization. (default: :obj:`False`)
        learn_msg_scale (bool, optional): If set to :obj:`True`, will learn the
            scaling factor of message normalization. (default: :obj:`False`)
        norm (str, optional): Norm layer of MLP layers (:obj:`"batch"`,
            :obj:`"layer"`, :obj:`"instance"`) (default: :obj:`batch`)
        num_layers (int, optional): The number of MLP layers.
            (default: :obj:`2`)
        expansion (int, optional): The expansion factor of hidden channels in
            MLP layers. (default: :obj:`2`)
        eps (float, optional): The epsilon value of the message construction
            function. (default: :obj:`1e-7`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        edge_dim (int, optional): Edge feature dimensionality. If set to
            :obj:`None`, Edge feature dimensionality is expected to match
            the `out_channels`. Other-wise, edge features are linearly
            transformed to match `out_channels` of node feature dimensionality.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.GenMessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge attributes :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        aggr: Optional[Union[str, List[str], Aggregation]] = 'softmax',
        t: float = 1.0,
        learn_t: bool = False,
        p: float = 1.0,
        learn_p: bool = False,
        msg_norm: bool = False,
        learn_msg_scale: bool = False,
        norm: str = 'batch',
        num_layers: int = 2,
        expansion: int = 2,
        eps: float = 1e-7,
        bias: bool = False,
        edge_dim: Optional[int] = None,
        **kwargs,
    ):

        # Backward compatibility:
        semi_grad = True if aggr == 'softmax_sg' else False
        aggr = 'softmax' if aggr == 'softmax_sg' else aggr
        aggr = 'powermean' if aggr == 'power' else aggr

        # Override args of aggregator if `aggr_kwargs` is specified
        if 'aggr_kwargs' not in kwargs:
            if aggr == 'softmax':
                kwargs['aggr_kwargs'] = dict(t=t, learn=learn_t,
                                             semi_grad=semi_grad)
            elif aggr == 'powermean':
                kwargs['aggr_kwargs'] = dict(p=p, learn=learn_p)

        super().__init__(aggr=aggr, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.eps = eps

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        if in_channels[0] != out_channels:
            self.lin_src = Linear(in_channels[0], out_channels, bias=bias)

        if edge_dim is not None and edge_dim != out_channels:
            self.lin_edge = Linear(edge_dim, out_channels, bias=bias)

        if isinstance(self.aggr_module, MultiAggregation):
            aggr_out_channels = self.aggr_module.get_out_channels(out_channels)
        else:
            aggr_out_channels = out_channels

        if aggr_out_channels != out_channels:
            self.lin_aggr_out = Linear(aggr_out_channels, out_channels,
                                       bias=bias)

        if in_channels[1] != out_channels:
            self.lin_dst = Linear(in_channels[1], out_channels, bias=bias)

        channels = [out_channels]
        for i in range(num_layers - 1):
            channels.append(out_channels * expansion)
        channels.append(out_channels)
        self.mlp = MLP(channels, norm=norm, bias=bias)

        if msg_norm:
            self.msg_norm = MessageNorm(learn_msg_scale)

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.mlp)
        if hasattr(self, 'msg_norm'):
            self.msg_norm.reset_parameters()
        if hasattr(self, 'lin_src'):
            self.lin_src.reset_parameters()
        if hasattr(self, 'lin_edge'):
            self.lin_edge.reset_parameters()
        if hasattr(self, 'lin_aggr_out'):
            self.lin_aggr_out.reset_parameters()
        if hasattr(self, 'lin_dst'):
            self.lin_dst.reset_parameters()

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        if hasattr(self, 'lin_src'):
            x = (self.lin_src(x[0]), x[1])

        if isinstance(edge_index, SparseTensor):
            edge_attr = edge_index.storage.value()
        elif is_torch_sparse_tensor(edge_index):
            _, value = to_edge_index(edge_index)
            if value.dim() > 1 or not value.all():
                edge_attr = value

        if edge_attr is not None and hasattr(self, 'lin_edge'):
            edge_attr = self.lin_edge(edge_attr)

        # Node and edge feature dimensionalites need to match.
        if edge_attr is not None:
            assert x[0].size(-1) == edge_attr.size(-1)

        # propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)

        if hasattr(self, 'lin_aggr_out'):
            out = self.lin_aggr_out(out)

        if hasattr(self, 'msg_norm'):
            h = x[1] if x[1] is not None else x[0]
            assert h is not None
            out = self.msg_norm(h, out)

        x_dst = x[1]
        if x_dst is not None:
            if hasattr(self, 'lin_dst'):
                x_dst = self.lin_dst(x_dst)
            out = out + x_dst

        return self.mlp(out)

    def message(self, x_j: Tensor, edge_attr: OptTensor) -> Tensor:
        msg = x_j if edge_attr is None else x_j + edge_attr
        return msg.relu() + self.eps

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, aggr={self.aggr})')
pytorch/torch_geometric/nn/conv/general_conv.py000066400000000000000000000165601444262304300223310ustar00rootroot00000000000000from typing import Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot
from torch_geometric.typing import (
    Adj,
    Optional,
    OptPairTensor,
    OptTensor,
    Size,
)
from torch_geometric.utils import softmax


class GeneralConv(MessagePassing):
    r"""A general GNN layer adapted from the `"Design Space for Graph Neural
    Networks" `_ paper.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        in_edge_channels (int, optional): Size of each input edge.
            (default: :obj:`None`)
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"mean"`)
        skip_linear (bool, optional): Whether apply linear function in skip
            connection. (default: :obj:`False`)
        directed_msg (bool, optional): If message passing is directed;
            otherwise, message passing is bi-directed. (default: :obj:`True`)
        heads (int, optional): Number of message passing ensembles.
            If :obj:`heads > 1`, the GNN layer will output an ensemble of
            multiple messages.
            If attention is used (:obj:`attention=True`), this corresponds to
            multi-head attention. (default: :obj:`1`)
        attention (bool, optional): Whether to add attention to message
            computation. (default: :obj:`False`)
        attention_type (str, optional): Type of attention: :obj:`"additive"`,
            :obj:`"dot_product"`. (default: :obj:`"additive"`)
        l2_normalize (bool, optional): If set to :obj:`True`, output features
            will be :math:`\ell_2`-normalized, *i.e.*,
            :math:`\frac{\mathbf{x}^{\prime}_i}
            {\| \mathbf{x}^{\prime}_i \|_2}`.
            (default: :obj:`False`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge attributes :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: Optional[int],
        in_edge_channels: int = None,
        aggr: str = "add",
        skip_linear: str = False,
        directed_msg: bool = True,
        heads: int = 1,
        attention: bool = False,
        attention_type: str = "additive",
        l2_normalize: bool = False,
        bias: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', aggr)
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.in_edge_channels = in_edge_channels
        self.aggr = aggr
        self.skip_linear = skip_linear
        self.directed_msg = directed_msg
        self.heads = heads
        self.attention = attention
        self.attention_type = attention_type
        self.normalize_l2 = l2_normalize

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        if self.directed_msg:
            self.lin_msg = Linear(in_channels[0], out_channels * self.heads,
                                  bias=bias)
        else:
            self.lin_msg = Linear(in_channels[0], out_channels * self.heads,
                                  bias=bias)
            self.lin_msg_i = Linear(in_channels[0], out_channels * self.heads,
                                    bias=bias)

        if self.skip_linear or self.in_channels != self.out_channels:
            self.lin_self = Linear(in_channels[1], out_channels, bias=bias)
        else:
            self.lin_self = torch.nn.Identity()

        if self.in_edge_channels is not None:
            self.lin_edge = Linear(in_edge_channels, out_channels * self.heads,
                                   bias=bias)

        # TODO: A general torch_geometric.nn.AttentionLayer
        if self.attention:
            if self.attention_type == 'additive':
                self.att_msg = Parameter(
                    torch.Tensor(1, self.heads, self.out_channels))
            elif self.attention_type == 'dot_product':
                scaler = torch.tensor(out_channels, dtype=torch.float).sqrt()
                self.register_buffer('scaler', scaler)
            else:
                raise ValueError(
                    f"Attention type '{self.attention_type}' not supported")

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_msg.reset_parameters()
        if hasattr(self.lin_self, 'reset_parameters'):
            self.lin_self.reset_parameters()
        if self.in_edge_channels is not None:
            self.lin_edge.reset_parameters()
        if self.attention and self.attention_type == 'additive':
            glorot(self.att_msg)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: Tensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)
        x_self = x[1]
        # propagate_type: (x: OptPairTensor)
        out = self.propagate(edge_index, x=x, size=size, edge_attr=edge_attr)
        out = out.mean(dim=1)  # todo: other approach to aggregate heads
        out = out + self.lin_self(x_self)
        if self.normalize_l2:
            out = F.normalize(out, p=2, dim=-1)
        return out

    def message_basic(self, x_i: Tensor, x_j: Tensor, edge_attr: OptTensor):
        if self.directed_msg:
            x_j = self.lin_msg(x_j)
        else:
            x_j = self.lin_msg(x_j) + self.lin_msg_i(x_i)
        if edge_attr is not None:
            x_j = x_j + self.lin_edge(edge_attr)
        return x_j

    def message(self, x_i: Tensor, x_j: Tensor, edge_index_i: Tensor,
                size_i: Tensor, edge_attr: Tensor) -> Tensor:
        x_j_out = self.message_basic(x_i, x_j, edge_attr)
        x_j_out = x_j_out.view(-1, self.heads, self.out_channels)
        if self.attention:
            if self.attention_type == 'dot_product':
                x_i_out = self.message_basic(x_j, x_i, edge_attr)
                x_i_out = x_i_out.view(-1, self.heads, self.out_channels)
                alpha = (x_i_out * x_j_out).sum(dim=-1) / self.scaler
            else:
                alpha = (x_j_out * self.att_msg).sum(dim=-1)
            alpha = F.leaky_relu(alpha, negative_slope=0.2)
            alpha = softmax(alpha, edge_index_i, num_nodes=size_i)
            alpha = alpha.view(-1, self.heads, 1)
            return x_j_out * alpha
        else:
            return x_j_out
pytorch/torch_geometric/nn/conv/gin_conv.py000066400000000000000000000164241444262304300214700ustar00rootroot00000000000000from typing import Callable, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
)
from torch_geometric.utils import spmm


class GINConv(MessagePassing):
    r"""The graph isomorphism operator from the `"How Powerful are
    Graph Neural Networks?" `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = h_{\mathbf{\Theta}} \left( (1 + \epsilon) \cdot
        \mathbf{x}_i + \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j \right)

    or

    .. math::
        \mathbf{X}^{\prime} = h_{\mathbf{\Theta}} \left( \left( \mathbf{A} +
        (1 + \epsilon) \cdot \mathbf{I} \right) \cdot \mathbf{X} \right),

    here :math:`h_{\mathbf{\Theta}}` denotes a neural network, *.i.e.* an MLP.

    Args:
        nn (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}` that
            maps node features :obj:`x` of shape :obj:`[-1, in_channels]` to
            shape :obj:`[-1, out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`.
        eps (float, optional): (Initial) :math:`\epsilon`-value.
            (default: :obj:`0.`)
        train_eps (bool, optional): If set to :obj:`True`, :math:`\epsilon`
            will be a trainable parameter. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, nn: Callable, eps: float = 0., train_eps: bool = False,
                 **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)
        self.nn = nn
        self.initial_eps = eps
        if train_eps:
            self.eps = torch.nn.Parameter(torch.Tensor([eps]))
        else:
            self.register_buffer('eps', torch.Tensor([eps]))
        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.nn)
        self.eps.data.fill_(self.initial_eps)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor)
        out = self.propagate(edge_index, x=x, size=size)

        x_r = x[1]
        if x_r is not None:
            out = out + (1 + self.eps) * x_r

        return self.nn(out)

    def message(self, x_j: Tensor) -> Tensor:
        return x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        if isinstance(adj_t, SparseTensor):
            adj_t = adj_t.set_value(None, layout=None)
        return spmm(adj_t, x[0], reduce=self.aggr)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(nn={self.nn})'


class GINEConv(MessagePassing):
    r"""The modified :class:`GINConv` operator from the `"Strategies for
    Pre-training Graph Neural Networks" `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = h_{\mathbf{\Theta}} \left( (1 + \epsilon) \cdot
        \mathbf{x}_i + \sum_{j \in \mathcal{N}(i)} \mathrm{ReLU}
        ( \mathbf{x}_j + \mathbf{e}_{j,i} ) \right)

    that is able to incorporate edge features :math:`\mathbf{e}_{j,i}` into
    the aggregation procedure.

    Args:
        nn (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}` that
            maps node features :obj:`x` of shape :obj:`[-1, in_channels]` to
            shape :obj:`[-1, out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`.
        eps (float, optional): (Initial) :math:`\epsilon`-value.
            (default: :obj:`0.`)
        train_eps (bool, optional): If set to :obj:`True`, :math:`\epsilon`
            will be a trainable parameter. (default: :obj:`False`)
        edge_dim (int, optional): Edge feature dimensionality. If set to
            :obj:`None`, node and edge feature dimensionality is expected to
            match. Other-wise, edge features are linearly transformed to match
            node feature dimensionality. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, nn: torch.nn.Module, eps: float = 0.,
                 train_eps: bool = False, edge_dim: Optional[int] = None,
                 **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)
        self.nn = nn
        self.initial_eps = eps
        if train_eps:
            self.eps = torch.nn.Parameter(torch.Tensor([eps]))
        else:
            self.register_buffer('eps', torch.Tensor([eps]))
        if edge_dim is not None:
            if isinstance(self.nn, torch.nn.Sequential):
                nn = self.nn[0]
            if hasattr(nn, 'in_features'):
                in_channels = nn.in_features
            elif hasattr(nn, 'in_channels'):
                in_channels = nn.in_channels
            else:
                raise ValueError("Could not infer input channels from `nn`.")
            self.lin = Linear(edge_dim, in_channels)

        else:
            self.lin = None
        self.reset_parameters()

    def reset_parameters(self):
        reset(self.nn)
        self.eps.data.fill_(self.initial_eps)
        if self.lin is not None:
            self.lin.reset_parameters()

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)

        x_r = x[1]
        if x_r is not None:
            out = out + (1 + self.eps) * x_r

        return self.nn(out)

    def message(self, x_j: Tensor, edge_attr: Tensor) -> Tensor:
        if self.lin is None and x_j.size(-1) != edge_attr.size(-1):
            raise ValueError("Node and edge feature dimensionalities do not "
                             "match. Consider setting the 'edge_dim' "
                             "attribute of 'GINEConv'")

        if self.lin is not None:
            edge_attr = self.lin(edge_attr)

        return (x_j + edge_attr).relu()

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(nn={self.nn})'
pytorch/torch_geometric/nn/conv/gmm_conv.py000066400000000000000000000202001444262304300214560ustar00rootroot00000000000000from typing import Tuple, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, Size


class GMMConv(MessagePassing):
    r"""The gaussian mixture model convolutional operator from the `"Geometric
    Deep Learning on Graphs and Manifolds using Mixture Model CNNs"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{1}{|\mathcal{N}(i)|}
        \sum_{j \in \mathcal{N}(i)} \frac{1}{K} \sum_{k=1}^K
        \mathbf{w}_k(\mathbf{e}_{i,j}) \odot \mathbf{\Theta}_k \mathbf{x}_j,

    where

    .. math::
        \mathbf{w}_k(\mathbf{e}) = \exp \left( -\frac{1}{2} {\left(
        \mathbf{e} - \mathbf{\mu}_k \right)}^{\top} \Sigma_k^{-1}
        \left( \mathbf{e} - \mathbf{\mu}_k \right) \right)

    denotes a weighting function based on trainable mean vector
    :math:`\mathbf{\mu}_k` and diagonal covariance matrix
    :math:`\mathbf{\Sigma}_k`.

    .. note::

        The edge attribute :math:`\mathbf{e}_{ij}` is usually given by
        :math:`\mathbf{e}_{ij} = \mathbf{p}_j - \mathbf{p}_i`, where
        :math:`\mathbf{p}_i` denotes the position of node :math:`i` (see
        :class:`torch_geometric.transform.Cartesian`).

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        dim (int): Pseudo-coordinate dimensionality.
        kernel_size (int): Number of kernels :math:`K`.
        separate_gaussians (bool, optional): If set to :obj:`True`, will
            learn separate GMMs for every pair of input and output channel,
            inspired by traditional CNNs. (default: :obj:`False`)
        aggr (str, optional): The aggregation operator to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"mean"`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, dim: int, kernel_size: int,
                 separate_gaussians: bool = False, aggr: str = 'mean',
                 root_weight: bool = True, bias: bool = True, **kwargs):
        super().__init__(aggr=aggr, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.dim = dim
        self.kernel_size = kernel_size
        self.separate_gaussians = separate_gaussians
        self.root_weight = root_weight

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)
        self.rel_in_channels = in_channels[0]

        if in_channels[0] > 0:
            self.g = Parameter(
                Tensor(in_channels[0], out_channels * kernel_size))

            if not self.separate_gaussians:
                self.mu = Parameter(Tensor(kernel_size, dim))
                self.sigma = Parameter(Tensor(kernel_size, dim))
            if self.separate_gaussians:
                self.mu = Parameter(
                    Tensor(in_channels[0], out_channels, kernel_size, dim))
                self.sigma = Parameter(
                    Tensor(in_channels[0], out_channels, kernel_size, dim))
        else:
            self.g = torch.nn.parameter.UninitializedParameter()
            self.mu = torch.nn.parameter.UninitializedParameter()
            self.sigma = torch.nn.parameter.UninitializedParameter()
            self._hook = self.register_forward_pre_hook(
                self.initialize_parameters)

        if root_weight:
            self.root = Linear(in_channels[1], out_channels, bias=False,
                               weight_initializer='glorot')

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if not isinstance(self.g, torch.nn.UninitializedParameter):
            glorot(self.g)
            glorot(self.mu)
            glorot(self.sigma)
        if self.root_weight:
            self.root.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None):

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
        if not self.separate_gaussians:
            out: OptPairTensor = (torch.matmul(x[0], self.g), x[1])
            out = self.propagate(edge_index, x=out, edge_attr=edge_attr,
                                 size=size)
        else:
            out = self.propagate(edge_index, x=x, edge_attr=edge_attr,
                                 size=size)

        x_r = x[1]
        if x_r is not None and self.root is not None:
            out = out + self.root(x_r)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_attr: Tensor):
        EPS = 1e-15
        F, M = self.rel_in_channels, self.out_channels
        (E, D), K = edge_attr.size(), self.kernel_size

        if not self.separate_gaussians:
            gaussian = -0.5 * (edge_attr.view(E, 1, D) -
                               self.mu.view(1, K, D)).pow(2)
            gaussian = gaussian / (EPS + self.sigma.view(1, K, D).pow(2))
            gaussian = torch.exp(gaussian.sum(dim=-1))  # [E, K]

            return (x_j.view(E, K, M) * gaussian.view(E, K, 1)).sum(dim=-2)

        else:
            gaussian = -0.5 * (edge_attr.view(E, 1, 1, 1, D) -
                               self.mu.view(1, F, M, K, D)).pow(2)
            gaussian = gaussian / (EPS + self.sigma.view(1, F, M, K, D).pow(2))
            gaussian = torch.exp(gaussian.sum(dim=-1))  # [E, F, M, K]

            gaussian = gaussian * self.g.view(1, F, M, K)
            gaussian = gaussian.sum(dim=-1)  # [E, F, M]

            return (x_j.view(E, F, 1) * gaussian).sum(dim=-2)  # [E, M]

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        if isinstance(self.g, torch.nn.parameter.UninitializedParameter):
            x = input[0][0] if isinstance(input, tuple) else input[0]
            in_channels = x.size(-1)
            out_channels, kernel_size = self.out_channels, self.kernel_size
            self.g.materialize((in_channels, out_channels * kernel_size))
            if not self.separate_gaussians:
                self.mu.materialize((kernel_size, self.dim))
                self.sigma.materialize((kernel_size, self.dim))
            else:
                self.mu.materialize(
                    (in_channels, out_channels, kernel_size, self.dim))
                self.sigma.materialize(
                    (in_channels, out_channels, kernel_size, self.dim))
            glorot(self.g)
            glorot(self.mu)
            glorot(self.sigma)

        module._hook.remove()
        delattr(module, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, dim={self.dim})')
pytorch/torch_geometric/nn/conv/gps_conv.py000066400000000000000000000150201444262304300214730ustar00rootroot00000000000000import inspect
from typing import Any, Dict, Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Dropout, Linear, Sequential

from torch_geometric.nn.attention import PerformerAttention
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.nn.resolver import (
    activation_resolver,
    normalization_resolver,
)
from torch_geometric.typing import Adj
from torch_geometric.utils import to_dense_batch


class GPSConv(torch.nn.Module):
    r"""The general, powerful, scalable (GPS) graph transformer layer from the
    `"Recipe for a General, Powerful, Scalable Graph Transformer"
    `_ paper.

    The GPS layer is based on a 3-part recipe:

    1. Inclusion of positional (PE) and structural encodings (SE) to the input
       features (done in a pre-processing step via
       :class:`torch_geometric.transforms`).
    2. A local message passing layer (MPNN) that operates on the input graph.
    3. A global attention layer that operates on the entire graph.

    .. note::

        For an example of using :class:`GPSConv`, see
        `examples/graph_gps.py
        `_.

    Args:
        channels (int): Size of each input sample.
        conv (MessagePassing, optional): The local message passing layer.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        dropout (float, optional): Dropout probability of intermediate
            embeddings. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`"batch_norm"`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        attn_type (str): Global attention type, :obj:`multihead` or
            :obj:`performer`. (default: :obj:`multihead`)
        attn_kwargs (Dict[str, Any], optional): Arguments passed to the
            attention layer. (default: :obj:`None`)
    """
    def __init__(
        self,
        channels: int,
        conv: Optional[MessagePassing],
        heads: int = 1,
        dropout: float = 0.0,
        act: str = 'relu',
        act_kwargs: Optional[Dict[str, Any]] = None,
        norm: Optional[str] = 'batch_norm',
        norm_kwargs: Optional[Dict[str, Any]] = None,
        attn_type: str = 'multihead',
        attn_kwargs: Optional[Dict[str, Any]] = None,
    ):
        super().__init__()

        self.channels = channels
        self.conv = conv
        self.heads = heads
        self.dropout = dropout
        self.attn_type = attn_type

        attn_kwargs = attn_kwargs or {}
        if attn_type == 'multihead':
            self.attn = torch.nn.MultiheadAttention(
                channels,
                heads,
                batch_first=True,
                **attn_kwargs,
            )
        elif attn_type == 'performer':
            self.attn = PerformerAttention(
                channels=channels,
                heads=heads,
                **attn_kwargs,
            )
        else:
            # TODO: Support BigBird
            raise ValueError(f'{attn_type} is not supported')

        self.mlp = Sequential(
            Linear(channels, channels * 2),
            activation_resolver(act, **(act_kwargs or {})),
            Dropout(dropout),
            Linear(channels * 2, channels),
            Dropout(dropout),
        )

        norm_kwargs = norm_kwargs or {}
        self.norm1 = normalization_resolver(norm, channels, **norm_kwargs)
        self.norm2 = normalization_resolver(norm, channels, **norm_kwargs)
        self.norm3 = normalization_resolver(norm, channels, **norm_kwargs)

        self.norm_with_batch = False
        if self.norm1 is not None:
            signature = inspect.signature(self.norm1.forward)
            self.norm_with_batch = 'batch' in signature.parameters

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        if self.conv is not None:
            self.conv.reset_parameters()
        self.attn._reset_parameters()
        reset(self.mlp)
        if self.norm1 is not None:
            self.norm1.reset_parameters()
        if self.norm2 is not None:
            self.norm2.reset_parameters()
        if self.norm3 is not None:
            self.norm3.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Adj,
        batch: Optional[torch.Tensor] = None,
        **kwargs,
    ) -> Tensor:
        r"""Runs the forward pass of the module."""
        hs = []
        if self.conv is not None:  # Local MPNN.
            h = self.conv(x, edge_index, **kwargs)
            h = F.dropout(h, p=self.dropout, training=self.training)
            h = h + x
            if self.norm1 is not None:
                if self.norm_with_batch:
                    h = self.norm1(h, batch=batch)
                else:
                    h = self.norm1(h)
            hs.append(h)

        # Global attention transformer-style model.
        h, mask = to_dense_batch(x, batch)

        if isinstance(self.attn, torch.nn.MultiheadAttention):
            h, _ = self.attn(h, h, h, key_padding_mask=~mask,
                             need_weights=False)
        elif isinstance(self.attn, PerformerAttention):
            h = self.attn(h, mask=mask)

        h = h[mask]
        h = F.dropout(h, p=self.dropout, training=self.training)
        h = h + x  # Residual connection.
        if self.norm2 is not None:
            if self.norm_with_batch:
                h = self.norm2(h, batch=batch)
            else:
                h = self.norm2(h)
        hs.append(h)

        out = sum(hs)  # Combine local and global outputs.

        out = out + self.mlp(out)
        if self.norm3 is not None:
            if self.norm_with_batch:
                out = self.norm3(out, batch=batch)
            else:
                out = self.norm3(out)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.channels}, '
                f'conv={self.conv}, heads={self.heads}, '
                f'attn_type={self.attn_type})')
pytorch/torch_geometric/nn/conv/graph_conv.py000066400000000000000000000067331444262304300220160ustar00rootroot00000000000000from typing import Tuple, Union

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
)
from torch_geometric.utils import spmm


class GraphConv(MessagePassing):
    r"""The graph neural network operator from the `"Weisfeiler and Leman Go
    Neural: Higher-order Graph Neural Networks"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i + \mathbf{W}_2
        \sum_{j \in \mathcal{N}(i)} e_{j,i} \cdot \mathbf{x}_j

    where :math:`e_{j,i}` denotes the edge weight from source node :obj:`j` to
    target node :obj:`i` (default: :obj:`1`)

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"add"`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        aggr: str = 'add',
        bias: bool = True,
        **kwargs,
    ):
        super().__init__(aggr=aggr, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lin_rel = Linear(in_channels[0], out_channels, bias=bias)
        self.lin_root = Linear(in_channels[1], out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_rel.reset_parameters()
        self.lin_root.reset_parameters()

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_weight: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=size)
        out = self.lin_rel(out)

        x_r = x[1]
        if x_r is not None:
            out = out + self.lin_root(x_r)

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        return spmm(adj_t, x[0], reduce=self.aggr)
pytorch/torch_geometric/nn/conv/gravnet_conv.py000066400000000000000000000116371444262304300223620ustar00rootroot00000000000000import warnings
from typing import Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import OptTensor, PairOptTensor, PairTensor

try:
    from torch_cluster import knn
except ImportError:
    knn = None


class GravNetConv(MessagePassing):
    r"""The GravNet operator from the `"Learning Representations of Irregular
    Particle-detector Geometry with Distance-weighted Graph
    Networks" `_ paper, where the graph is
    dynamically constructed using nearest neighbors.
    The neighbors are constructed in a learnable low-dimensional projection of
    the feature space.
    A second projection of the input feature space is then propagated from the
    neighbors to each vertex using distance weights that are derived by
    applying a Gaussian function to the distances.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): The number of output channels.
        space_dimensions (int): The dimensionality of the space used to
           construct the neighbors; referred to as :math:`S` in the paper.
        propagate_dimensions (int): The number of features to be propagated
           between the vertices; referred to as :math:`F_{\textrm{LR}}` in the
           paper.
        k (int): The number of nearest neighbors.
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{in}), (|\mathcal{V_t}|, F_{in}))`
          if bipartite,
          batch vector :math:`(|\mathcal{V}|)` or
          :math:`((|\mathcal{V}_s|), (|\mathcal{V}_t|))` if bipartite
          *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: int, out_channels: int,
                 space_dimensions: int, propagate_dimensions: int, k: int,
                 num_workers: Optional[int] = None, **kwargs):
        super().__init__(aggr=['mean', 'max'], flow='source_to_target',
                         **kwargs)

        if knn is None:
            raise ImportError('`GravNetConv` requires `torch-cluster`.')

        if num_workers is not None:
            warnings.warn(
                "'num_workers' attribute in '{self.__class__.__name__}' is "
                "deprecated and will be removed in a future release")

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.k = k

        self.lin_s = Linear(in_channels, space_dimensions)
        self.lin_h = Linear(in_channels, propagate_dimensions)

        self.lin_out1 = Linear(in_channels, out_channels, bias=False)
        self.lin_out2 = Linear(2 * propagate_dimensions, out_channels)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_s.reset_parameters()
        self.lin_h.reset_parameters()
        self.lin_out1.reset_parameters()
        self.lin_out2.reset_parameters()

    def forward(
            self, x: Union[Tensor, PairTensor],
            batch: Union[OptTensor, Optional[PairTensor]] = None) -> Tensor:
        # type: (Tensor, OptTensor) -> Tensor  # noqa
        # type: (PairTensor, Optional[PairTensor]) -> Tensor  # noqa

        is_bipartite: bool = True
        if isinstance(x, Tensor):
            x: PairTensor = (x, x)
            is_bipartite = False

        if x[0].dim() != 2:
            raise ValueError("Static graphs not supported in 'GravNetConv'")

        b: PairOptTensor = (None, None)
        if isinstance(batch, Tensor):
            b = (batch, batch)
        elif isinstance(batch, tuple):
            assert batch is not None
            b = (batch[0], batch[1])

        h_l: Tensor = self.lin_h(x[0])

        s_l: Tensor = self.lin_s(x[0])
        s_r: Tensor = self.lin_s(x[1]) if is_bipartite else s_l

        edge_index = knn(s_l, s_r, self.k, b[0], b[1]).flip([0])

        edge_weight = (s_l[edge_index[0]] - s_r[edge_index[1]]).pow(2).sum(-1)
        edge_weight = torch.exp(-10. * edge_weight)  # 10 gives a better spread

        # propagate_type: (x: OptPairTensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=(h_l, None),
                             edge_weight=edge_weight,
                             size=(s_l.size(0), s_r.size(0)))

        return self.lin_out1(x[1]) + self.lin_out2(out)

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return x_j * edge_weight.unsqueeze(1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, k={self.k})')
pytorch/torch_geometric/nn/conv/han_conv.py000066400000000000000000000162721444262304300214620ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor, nn

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense import Linear
from torch_geometric.nn.inits import glorot, reset
from torch_geometric.typing import Adj, EdgeType, Metadata, NodeType, OptTensor
from torch_geometric.utils import softmax


def group(
    xs: List[Tensor],
    q: nn.Parameter,
    k_lin: nn.Module,
) -> Tuple[OptTensor, OptTensor]:

    if len(xs) == 0:
        return None, None
    else:
        num_edge_types = len(xs)
        out = torch.stack(xs)
        if out.numel() == 0:
            return out.view(0, out.size(-1)), None
        attn_score = (q * torch.tanh(k_lin(out)).mean(1)).sum(-1)
        attn = F.softmax(attn_score, dim=0)
        out = torch.sum(attn.view(num_edge_types, 1, -1) * out, dim=0)
        return out, attn


class HANConv(MessagePassing):
    r"""
    The Heterogenous Graph Attention Operator from the
    `"Heterogenous Graph Attention Network"
    `_ paper.

    .. note::

        For an example of using HANConv, see `examples/hetero/han_imdb.py
        `_.

    Args:
        in_channels (int or Dict[str, int]): Size of each input sample of every
            node type, or :obj:`-1` to derive the size from the first input(s)
            to the forward method.
        out_channels (int): Size of each output sample.
        metadata (Tuple[List[str], List[Tuple[str, str, str]]]): The metadata
            of the heterogeneous graph, *i.e.* its node and edge types given
            by a list of strings and a list of string triplets, respectively.
            See :meth:`torch_geometric.data.HeteroData.metadata` for more
            information.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """
    def __init__(
        self,
        in_channels: Union[int, Dict[str, int]],
        out_channels: int,
        metadata: Metadata,
        heads: int = 1,
        negative_slope=0.2,
        dropout: float = 0.0,
        **kwargs,
    ):
        super().__init__(aggr='add', node_dim=0, **kwargs)

        if not isinstance(in_channels, dict):
            in_channels = {node_type: in_channels for node_type in metadata[0]}

        self.heads = heads
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.negative_slope = negative_slope
        self.metadata = metadata
        self.dropout = dropout
        self.k_lin = nn.Linear(out_channels, out_channels)
        self.q = nn.Parameter(torch.Tensor(1, out_channels))

        self.proj = nn.ModuleDict()
        for node_type, in_channels in self.in_channels.items():
            self.proj[node_type] = Linear(in_channels, out_channels)

        self.lin_src = nn.ParameterDict()
        self.lin_dst = nn.ParameterDict()
        dim = out_channels // heads
        for edge_type in metadata[1]:
            edge_type = '__'.join(edge_type)
            self.lin_src[edge_type] = nn.Parameter(torch.Tensor(1, heads, dim))
            self.lin_dst[edge_type] = nn.Parameter(torch.Tensor(1, heads, dim))

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.proj)
        glorot(self.lin_src)
        glorot(self.lin_dst)
        self.k_lin.reset_parameters()
        glorot(self.q)

    def forward(
        self,
        x_dict: Dict[NodeType, Tensor],
        edge_index_dict: Dict[EdgeType, Adj],
        return_semantic_attention_weights: bool = False,
    ) -> Union[Dict[NodeType, OptTensor], Tuple[Dict[NodeType, OptTensor],
                                                Dict[NodeType, OptTensor]]]:
        r"""Runs the forward pass of the module.

        Args:
            x_dict (Dict[str, torch.Tensor]): A dictionary holding node feature
                information for each individual node type.
            edge_index_dict (Dict[Tuple[str, str, str], torch.Tensor]): A
                dictionary holding graph connectivity information for each
                individual edge type, either as a :class:`torch.Tensor` of
                shape :obj:`[2, num_edges]` or a
                :class:`torch_sparse.SparseTensor`.
            return_semantic_attention_weights (bool, optional): If set to
                :obj:`True`, will additionally return the semantic-level
                attention weights for each destination node type.
                (default: :obj:`False`)
        """
        H, D = self.heads, self.out_channels // self.heads
        x_node_dict, out_dict = {}, {}

        # Iterate over node types:
        for node_type, x in x_dict.items():
            x_node_dict[node_type] = self.proj[node_type](x).view(-1, H, D)
            out_dict[node_type] = []

        # Iterate over edge types:
        for edge_type, edge_index in edge_index_dict.items():
            src_type, _, dst_type = edge_type
            edge_type = '__'.join(edge_type)
            lin_src = self.lin_src[edge_type]
            lin_dst = self.lin_dst[edge_type]
            x_src = x_node_dict[src_type]
            x_dst = x_node_dict[dst_type]
            alpha_src = (x_src * lin_src).sum(dim=-1)
            alpha_dst = (x_dst * lin_dst).sum(dim=-1)
            # propagate_type: (x_dst: PairTensor, alpha: PairTensor)
            out = self.propagate(edge_index, x=(x_src, x_dst),
                                 alpha=(alpha_src, alpha_dst), size=None)

            out = F.relu(out)
            out_dict[dst_type].append(out)

        # iterate over node types:
        semantic_attn_dict = {}
        for node_type, outs in out_dict.items():
            out, attn = group(outs, self.q, self.k_lin)
            out_dict[node_type] = out
            semantic_attn_dict[node_type] = attn

        if return_semantic_attention_weights:
            return out_dict, semantic_attn_dict

        return out_dict

    def message(self, x_j: Tensor, alpha_i: Tensor, alpha_j: Tensor,
                index: Tensor, ptr: Optional[Tensor],
                size_i: Optional[int]) -> Tensor:

        alpha = alpha_j + alpha_i
        alpha = F.leaky_relu(alpha, self.negative_slope)
        alpha = softmax(alpha, index, ptr, size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        out = x_j * alpha.view(-1, self.heads, 1)
        return out.view(-1, self.out_channels)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.out_channels}, '
                f'heads={self.heads})')
pytorch/torch_geometric/nn/conv/heat_conv.py000066400000000000000000000136571444262304300216410ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Embedding

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import HeteroLinear, Linear
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils import softmax


class HEATConv(MessagePassing):
    r"""The heterogeneous edge-enhanced graph attentional operator from the
    `"Heterogeneous Edge-Enhanced Graph Attention Network For Multi-Agent
    Trajectory Prediction" `_ paper, which
    enhances :class:`~torch_geometric.nn.conv.GATConv` by:

    1. type-specific transformations of nodes of different types
    2. edge type and edge feature incorporation, in which edges are assumed to
       have different types but contain the same kind of attributes

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        num_node_types (int): The number of node types.
        num_edge_types (int): The number of edge types.
        edge_type_emb_dim (int): The embedding size of edge types.
        edge_dim (int): Edge feature dimensionality.
        edge_attr_emb_dim (int): The embedding size of edge features.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          node types :math:`(|\mathcal{V}|)`,
          edge types :math:`(|\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int,
                 num_node_types: int, num_edge_types: int,
                 edge_type_emb_dim: int, edge_dim: int, edge_attr_emb_dim: int,
                 heads: int = 1, concat: bool = True,
                 negative_slope: float = 0.2, dropout: float = 0.0,
                 root_weight: bool = True, bias: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.root_weight = root_weight

        self.hetero_lin = HeteroLinear(in_channels, out_channels,
                                       num_node_types, bias=bias)

        self.edge_type_emb = Embedding(num_edge_types, edge_type_emb_dim)
        self.edge_attr_emb = Linear(edge_dim, edge_attr_emb_dim, bias=False)

        self.att = Linear(
            2 * out_channels + edge_type_emb_dim + edge_attr_emb_dim,
            self.heads, bias=False)

        self.lin = Linear(out_channels + edge_attr_emb_dim, out_channels,
                          bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.hetero_lin.reset_parameters()
        self.edge_type_emb.reset_parameters()
        self.edge_attr_emb.reset_parameters()
        self.att.reset_parameters()
        self.lin.reset_parameters()

    def forward(self, x: Tensor, edge_index: Adj, node_type: Tensor,
                edge_type: Tensor, edge_attr: OptTensor = None) -> Tensor:

        x = self.hetero_lin(x, node_type)

        edge_type_emb = F.leaky_relu(self.edge_type_emb(edge_type),
                                     self.negative_slope)

        # propagate_type: (x: Tensor, edge_type_emb: Tensor, edge_attr: OptTensor)  # noqa
        out = self.propagate(edge_index, x=x, edge_type_emb=edge_type_emb,
                             edge_attr=edge_attr, size=None)

        if self.concat:
            if self.root_weight:
                out = out + x.view(-1, 1, self.out_channels)
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)
            if self.root_weight:
                out = out + x

        return out

    def message(self, x_i: Tensor, x_j: Tensor, edge_type_emb: Tensor,
                edge_attr: Tensor, index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:

        edge_attr = F.leaky_relu(self.edge_attr_emb(edge_attr),
                                 self.negative_slope)

        alpha = torch.cat([x_i, x_j, edge_type_emb, edge_attr], dim=-1)
        alpha = F.leaky_relu(self.att(alpha), self.negative_slope)
        alpha = softmax(alpha, index, ptr, size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)

        out = self.lin(torch.cat([x_j, edge_attr], dim=-1)).unsqueeze(-2)
        return out * alpha.unsqueeze(-1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/hetero_conv.py000066400000000000000000000145061444262304300222000ustar00rootroot00000000000000import warnings
from collections import defaultdict
from typing import Dict, List, Optional

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.module_dict import ModuleDict
from torch_geometric.typing import Adj, EdgeType, NodeType
from torch_geometric.utils.hetero import check_add_self_loops


def group(xs: List[Tensor], aggr: Optional[str]) -> Optional[Tensor]:
    if len(xs) == 0:
        return None
    elif aggr is None:
        return torch.stack(xs, dim=1)
    elif len(xs) == 1:
        return xs[0]
    elif aggr == "cat":
        return torch.cat(xs, dim=-1)
    else:
        out = torch.stack(xs, dim=0)
        out = getattr(torch, aggr)(out, dim=0)
        out = out[0] if isinstance(out, tuple) else out
        return out


class HeteroConv(torch.nn.Module):
    r"""A generic wrapper for computing graph convolution on heterogeneous
    graphs.
    This layer will pass messages from source nodes to target nodes based on
    the bipartite GNN layer given for a specific edge type.
    If multiple relations point to the same destination, their results will be
    aggregated according to :attr:`aggr`.
    In comparison to :meth:`torch_geometric.nn.to_hetero`, this layer is
    especially useful if you want to apply different message passing modules
    for different edge types.

    .. code-block:: python

        hetero_conv = HeteroConv({
            ('paper', 'cites', 'paper'): GCNConv(-1, 64),
            ('author', 'writes', 'paper'): SAGEConv((-1, -1), 64),
            ('paper', 'written_by', 'author'): GATConv((-1, -1), 64),
        }, aggr='sum')

        out_dict = hetero_conv(x_dict, edge_index_dict)

        print(list(out_dict.keys()))
        >>> ['paper', 'author']

    Args:
        convs (Dict[Tuple[str, str, str], MessagePassing]): A dictionary
            holding a bipartite
            :class:`~torch_geometric.nn.conv.MessagePassing` layer for each
            individual edge type.
        aggr (str, optional): The aggregation scheme to use for grouping node
            embeddings generated by different relations
            (:obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"cat"`, :obj:`None`). (default: :obj:`"sum"`)
    """
    def __init__(
        self,
        convs: Dict[EdgeType, MessagePassing],
        aggr: Optional[str] = "sum",
    ):
        super().__init__()

        for edge_type, module in convs.items():
            check_add_self_loops(module, [edge_type])

        src_node_types = set([key[0] for key in convs.keys()])
        dst_node_types = set([key[-1] for key in convs.keys()])
        if len(src_node_types - dst_node_types) > 0:
            warnings.warn(
                f"There exist node types ({src_node_types - dst_node_types}) "
                f"whose representations do not get updated during message "
                f"passing as they do not occur as destination type in any "
                f"edge type. This may lead to unexpected behavior.")

        self.convs = ModuleDict({'__'.join(k): v for k, v in convs.items()})
        self.aggr = aggr

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for conv in self.convs.values():
            conv.reset_parameters()

    def forward(
        self,
        x_dict: Dict[NodeType, Tensor],
        edge_index_dict: Dict[EdgeType, Adj],
        *args_dict,
        **kwargs_dict,
    ) -> Dict[NodeType, Tensor]:
        r"""Runs the forward pass of the module.

        Args:
            x_dict (Dict[str, torch.Tensor]): A dictionary holding node feature
                information for each individual node type.
            edge_index_dict (Dict[Tuple[str, str, str], torch.Tensor]): A
                dictionary holding graph connectivity information for each
                individual edge type, either as a :class:`torch.Tensor` of
                shape :obj:`[2, num_edges]` or a
                :class:`torch_sparse.SparseTensor`.
            *args_dict (optional): Additional forward arguments of invididual
                :class:`torch_geometric.nn.conv.MessagePassing` layers.
            **kwargs_dict (optional): Additional forward arguments of
                individual :class:`torch_geometric.nn.conv.MessagePassing`
                layers.
                For example, if a specific GNN layer at edge type
                :obj:`edge_type` expects edge attributes :obj:`edge_attr` as a
                forward argument, then you can pass them to
                :meth:`~torch_geometric.nn.conv.HeteroConv.forward` via
                :obj:`edge_attr_dict = { edge_type: edge_attr }`.
        """
        out_dict = defaultdict(list)
        for edge_type, edge_index in edge_index_dict.items():
            src, rel, dst = edge_type

            str_edge_type = '__'.join(edge_type)
            if str_edge_type not in self.convs:
                continue

            args = []
            for value_dict in args_dict:
                if edge_type in value_dict:
                    args.append(value_dict[edge_type])
                elif src == dst and src in value_dict:
                    args.append(value_dict[src])
                elif src in value_dict or dst in value_dict:
                    args.append(
                        (value_dict.get(src, None), value_dict.get(dst, None)))

            kwargs = {}
            for arg, value_dict in kwargs_dict.items():
                arg = arg[:-5]  # `{*}_dict`
                if edge_type in value_dict:
                    kwargs[arg] = value_dict[edge_type]
                elif src == dst and src in value_dict:
                    kwargs[arg] = value_dict[src]
                elif src in value_dict or dst in value_dict:
                    kwargs[arg] = (value_dict.get(src, None),
                                   value_dict.get(dst, None))

            conv = self.convs[str_edge_type]

            if src == dst:
                out = conv(x_dict[src], edge_index, *args, **kwargs)
            else:
                out = conv((x_dict[src], x_dict[dst]), edge_index, *args,
                           **kwargs)

            out_dict[dst].append(out)

        for key, value in out_dict.items():
            out_dict[key] = group(value, self.aggr)

        return out_dict

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(num_relations={len(self.convs)})'
pytorch/torch_geometric/nn/conv/hgt_conv.py000066400000000000000000000221021444262304300214630ustar00rootroot00000000000000import math
from typing import Dict, List, Optional, Tuple, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense import HeteroDictLinear, HeteroLinear
from torch_geometric.nn.inits import ones
from torch_geometric.nn.parameter_dict import ParameterDict
from torch_geometric.typing import Adj, EdgeType, Metadata, NodeType
from torch_geometric.utils import softmax
from torch_geometric.utils.hetero import construct_bipartite_edge_index


class HGTConv(MessagePassing):
    r"""The Heterogeneous Graph Transformer (HGT) operator from the
    `"Heterogeneous Graph Transformer" `_
    paper.

    .. note::

        For an example of using HGT, see `examples/hetero/hgt_dblp.py
        `_.

    Args:
        in_channels (int or Dict[str, int]): Size of each input sample of every
            node type, or :obj:`-1` to derive the size from the first input(s)
            to the forward method.
        out_channels (int): Size of each output sample.
        metadata (Tuple[List[str], List[Tuple[str, str, str]]]): The metadata
            of the heterogeneous graph, *i.e.* its node and edge types given
            by a list of strings and a list of string triplets, respectively.
            See :meth:`torch_geometric.data.HeteroData.metadata` for more
            information.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        group (str, optional): The aggregation scheme to use for grouping node
            embeddings generated by different relations
            (:obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`).
            (default: :obj:`"sum"`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """
    def __init__(
        self,
        in_channels: Union[int, Dict[str, int]],
        out_channels: int,
        metadata: Metadata,
        heads: int = 1,
        **kwargs,
    ):
        super().__init__(aggr='add', node_dim=0, **kwargs)

        if out_channels % heads != 0:
            raise ValueError(f"'out_channels' (got {out_channels}) must be "
                             f"divisible by the number of heads (got {heads})")

        if not isinstance(in_channels, dict):
            in_channels = {node_type: in_channels for node_type in metadata[0]}

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.node_types = metadata[0]
        self.edge_types = metadata[1]
        self.edge_types_map = {
            edge_type: i
            for i, edge_type in enumerate(metadata[1])
        }

        self.dst_node_types = set([key[-1] for key in self.edge_types])

        self.kqv_lin = HeteroDictLinear(self.in_channels,
                                        self.out_channels * 3)

        self.out_lin = HeteroDictLinear(self.out_channels, self.out_channels,
                                        types=self.node_types)

        dim = out_channels // heads
        num_types = heads * len(self.edge_types)

        self.k_rel = HeteroLinear(dim, dim, num_types, bias=False,
                                  is_sorted=True)
        self.v_rel = HeteroLinear(dim, dim, num_types, bias=False,
                                  is_sorted=True)

        self.skip = ParameterDict({
            node_type: Parameter(torch.Tensor(1))
            for node_type in self.node_types
        })

        self.p_rel = ParameterDict()
        for edge_type in self.edge_types:
            edge_type = '__'.join(edge_type)
            self.p_rel[edge_type] = Parameter(torch.Tensor(1, heads))

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.kqv_lin.reset_parameters()
        self.out_lin.reset_parameters()
        self.k_rel.reset_parameters()
        self.v_rel.reset_parameters()
        ones(self.skip)
        ones(self.p_rel)

    def _cat(self, x_dict: Dict[str, Tensor]) -> Tuple[Tensor, Dict[str, int]]:
        """Concatenates a dictionary of features."""
        cumsum = 0
        outs: List[Tensor] = []
        offset: Dict[str, int] = {}
        for key, x in x_dict.items():
            outs.append(x)
            offset[key] = cumsum
            cumsum += x.size(0)
        return torch.cat(outs, dim=0), offset

    def _construct_src_node_feat(
        self, k_dict: Dict[str, Tensor], v_dict: Dict[str, Tensor],
        edge_index_dict: Dict[EdgeType, Adj]
    ) -> Tuple[Tensor, Tensor, Dict[EdgeType, int]]:
        """Constructs the source node representations."""
        cumsum = 0
        num_edge_types = len(self.edge_types)
        H, D = self.heads, self.out_channels // self.heads

        # Flatten into a single tensor with shape [num_edge_types * heads, D]:
        ks: List[Tensor] = []
        vs: List[Tensor] = []
        type_list: List[Tensor] = []
        offset: Dict[EdgeType] = {}
        for edge_type in edge_index_dict.keys():
            src = edge_type[0]
            N = k_dict[src].size(0)
            offset[edge_type] = cumsum
            cumsum += N

            # construct type_vec for curr edge_type with shape [H, D]
            edge_type_offset = self.edge_types_map[edge_type]
            type_vec = torch.arange(H, dtype=torch.long).view(-1, 1).repeat(
                1, N) * num_edge_types + edge_type_offset

            type_list.append(type_vec)
            ks.append(k_dict[src])
            vs.append(v_dict[src])

        ks = torch.cat(ks, dim=0).transpose(0, 1).reshape(-1, D)
        vs = torch.cat(vs, dim=0).transpose(0, 1).reshape(-1, D)
        type_vec = torch.cat(type_list, dim=1).flatten()

        k = self.k_rel(ks, type_vec).view(H, -1, D).transpose(0, 1)
        v = self.v_rel(vs, type_vec).view(H, -1, D).transpose(0, 1)

        return k, v, offset

    def forward(
        self,
        x_dict: Dict[NodeType, Tensor],
        edge_index_dict: Dict[EdgeType, Adj]  # Support both.
    ) -> Dict[NodeType, Optional[Tensor]]:
        r"""Runs the forward pass of the module.

        Args:
            x_dict (Dict[str, torch.Tensor]): A dictionary holding input node
                features  for each individual node type.
            edge_index_dict (Dict[Tuple[str, str, str], torch.Tensor]): A
                dictionary holding graph connectivity information for each
                individual edge type, either as a :class:`torch.Tensor` of
                shape :obj:`[2, num_edges]` or a
                :class:`torch_sparse.SparseTensor`.

        :rtype: :obj:`Dict[str, Optional[torch.Tensor]]` - The output node
            embeddings for each node type.
            In case a node type does not receive any message, its output will
            be set to :obj:`None`.
        """
        F = self.out_channels
        H = self.heads
        D = F // H

        k_dict, q_dict, v_dict, out_dict = {}, {}, {}, {}

        # Compute K, Q, V over node types:
        kqv_dict = self.kqv_lin(x_dict)
        for key, val in kqv_dict.items():
            k, q, v = torch.tensor_split(val, 3, dim=1)
            k_dict[key] = k.view(-1, H, D)
            q_dict[key] = q.view(-1, H, D)
            v_dict[key] = v.view(-1, H, D)

        q, dst_offset = self._cat(q_dict)
        k, v, src_offset = self._construct_src_node_feat(
            k_dict, v_dict, edge_index_dict)

        edge_index, edge_attr = construct_bipartite_edge_index(
            edge_index_dict, src_offset, dst_offset, edge_attr_dict=self.p_rel)

        out = self.propagate(edge_index, k=k, q=q, v=v, edge_attr=edge_attr,
                             size=None)

        # Reconstruct output node embeddings dict:
        for node_type, start_offset in dst_offset.items():
            end_offset = start_offset + q_dict[node_type].size(0)
            if node_type in self.dst_node_types:
                out_dict[node_type] = out[start_offset:end_offset]

        # Transform output node embeddings:
        a_dict = self.out_lin({
            k: torch.nn.functional.gelu(v) if v is not None else v
            for k, v in out_dict.items()
        })

        # Iterate over node types:
        for node_type, out in out_dict.items():
            out = a_dict[node_type]

            if out.size(-1) == x_dict[node_type].size(-1):
                alpha = self.skip[node_type].sigmoid()
                out = alpha * out + (1 - alpha) * x_dict[node_type]
            out_dict[node_type] = out

        return out_dict

    def message(self, k_j: Tensor, q_i: Tensor, v_j: Tensor, edge_attr: Tensor,
                index: Tensor, ptr: Optional[Tensor],
                size_i: Optional[int]) -> Tensor:
        alpha = (q_i * k_j).sum(dim=-1) * edge_attr
        alpha = alpha / math.sqrt(q_i.size(-1))
        alpha = softmax(alpha, index, ptr, size_i)
        out = v_j * alpha.view(-1, self.heads, 1)
        return out.view(-1, self.out_channels)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(-1, {self.out_channels}, '
                f'heads={self.heads})')
pytorch/torch_geometric/nn/conv/hypergraph_conv.py000066400000000000000000000173471444262304300230710ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.experimental import disable_dynamic_shapes
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.utils import scatter, softmax


class HypergraphConv(MessagePassing):
    r"""The hypergraph convolutional operator from the `"Hypergraph Convolution
    and Hypergraph Attention" `_ paper

    .. math::
        \mathbf{X}^{\prime} = \mathbf{D}^{-1} \mathbf{H} \mathbf{W}
        \mathbf{B}^{-1} \mathbf{H}^{\top} \mathbf{X} \mathbf{\Theta}

    where :math:`\mathbf{H} \in {\{ 0, 1 \}}^{N \times M}` is the incidence
    matrix, :math:`\mathbf{W} \in \mathbb{R}^M` is the diagonal hyperedge
    weight matrix, and
    :math:`\mathbf{D}` and :math:`\mathbf{B}` are the corresponding degree
    matrices.

    For example, in the hypergraph scenario
    :math:`\mathcal{G} = (\mathcal{V}, \mathcal{E})` with
    :math:`\mathcal{V} = \{ 0, 1, 2, 3 \}` and
    :math:`\mathcal{E} = \{ \{ 0, 1, 2 \}, \{ 1, 2, 3 \} \}`, the
    :obj:`hyperedge_index` is represented as:

    .. code-block:: python

        hyperedge_index = torch.tensor([
            [0, 1, 2, 1, 2, 3],
            [0, 0, 0, 1, 1, 1],
        ])

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        use_attention (bool, optional): If set to :obj:`True`, attention
            will be added to this layer. (default: :obj:`False`)
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          hyperedge indices :math:`(|\mathcal{V}|, |\mathcal{E}|)`,
          hyperedge weights :math:`(|\mathcal{E}|)` *(optional)*
          hyperedge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels, out_channels, use_attention=False, heads=1,
                 concat=True, negative_slope=0.2, dropout=0, bias=True,
                 **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(flow='source_to_target', node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.use_attention = use_attention

        if self.use_attention:
            self.heads = heads
            self.concat = concat
            self.negative_slope = negative_slope
            self.dropout = dropout
            self.lin = Linear(in_channels, heads * out_channels, bias=False,
                              weight_initializer='glorot')
            self.att = Parameter(torch.Tensor(1, heads, 2 * out_channels))
        else:
            self.heads = 1
            self.concat = True
            self.lin = Linear(in_channels, out_channels, bias=False,
                              weight_initializer='glorot')

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        if self.use_attention:
            glorot(self.att)
        zeros(self.bias)

    @disable_dynamic_shapes(required_args=['num_edges'])
    def forward(self, x: Tensor, hyperedge_index: Tensor,
                hyperedge_weight: Optional[Tensor] = None,
                hyperedge_attr: Optional[Tensor] = None,
                num_edges: Optional[int] = None) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor): Node feature matrix
                :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
            hyperedge_index (torch.Tensor): The hyperedge indices, *i.e.*
                the sparse incidence matrix
                :math:`\mathbf{H} \in {\{ 0, 1 \}}^{N \times M}` mapping from
                nodes to edges.
            hyperedge_weight (torch.Tensor, optional): Hyperedge weights
                :math:`\mathbf{W} \in \mathbb{R}^M`. (default: :obj:`None`)
            hyperedge_attr (torch.Tensor, optional): Hyperedge feature matrix
                in :math:`\mathbb{R}^{M \times F}`.
                These features only need to get passed in case
                :obj:`use_attention=True`. (default: :obj:`None`)
            num_edges (int, optional) : The number of edges :math:`M`.
                (default: :obj:`None`)
        """
        num_nodes = x.size(0)

        if num_edges is None:
            num_edges = 0
            if hyperedge_index.numel() > 0:
                num_edges = int(hyperedge_index[1].max()) + 1

        if hyperedge_weight is None:
            hyperedge_weight = x.new_ones(num_edges)

        x = self.lin(x)

        alpha = None
        if self.use_attention:
            assert hyperedge_attr is not None
            x = x.view(-1, self.heads, self.out_channels)
            hyperedge_attr = self.lin(hyperedge_attr)
            hyperedge_attr = hyperedge_attr.view(-1, self.heads,
                                                 self.out_channels)
            x_i = x[hyperedge_index[0]]
            x_j = hyperedge_attr[hyperedge_index[1]]
            alpha = (torch.cat([x_i, x_j], dim=-1) * self.att).sum(dim=-1)
            alpha = F.leaky_relu(alpha, self.negative_slope)
            alpha = softmax(alpha, hyperedge_index[0], num_nodes=x.size(0))
            alpha = F.dropout(alpha, p=self.dropout, training=self.training)

        D = scatter(hyperedge_weight[hyperedge_index[1]], hyperedge_index[0],
                    dim=0, dim_size=num_nodes, reduce='sum')
        D = 1.0 / D
        D[D == float("inf")] = 0

        B = scatter(x.new_ones(hyperedge_index.size(1)), hyperedge_index[1],
                    dim=0, dim_size=num_edges, reduce='sum')
        B = 1.0 / B
        B[B == float("inf")] = 0

        out = self.propagate(hyperedge_index, x=x, norm=B, alpha=alpha,
                             size=(num_nodes, num_edges))
        out = self.propagate(hyperedge_index.flip([0]), x=out, norm=D,
                             alpha=alpha, size=(num_edges, num_nodes))

        if self.concat is True:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, norm_i: Tensor, alpha: Tensor) -> Tensor:
        H, F = self.heads, self.out_channels

        out = norm_i.view(-1, 1, 1) * x_j.view(-1, H, F)

        if alpha is not None:
            out = alpha.view(-1, self.heads, 1) * out

        return out
pytorch/torch_geometric/nn/conv/le_conv.py000066400000000000000000000067031444262304300213120ustar00rootroot00000000000000from typing import Tuple, Union

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptTensor, PairTensor


class LEConv(MessagePassing):
    r"""The local extremum graph neural network operator from the
    `"ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph
    Representations" `_ paper, which finds
    the importance of nodes with respect to their neighbors using the
    difference operator:

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{x}_i \cdot \mathbf{\Theta}_1 +
        \sum_{j \in \mathcal{N}(i)} e_{j,i} \cdot
        (\mathbf{\Theta}_2 \mathbf{x}_i - \mathbf{\Theta}_3 \mathbf{x}_j)

    where :math:`e_{j,i}` denotes the edge weight from source node :obj:`j` to
    target node :obj:`i` (default: :obj:`1`)

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        bias (bool, optional): If set to :obj:`False`, the layer will
            not learn an additive bias. (default: :obj:`True`).
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lin1 = Linear(in_channels[0], out_channels, bias=bias)
        self.lin2 = Linear(in_channels[1], out_channels, bias=False)
        self.lin3 = Linear(in_channels[1], out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin1.reset_parameters()
        self.lin2.reset_parameters()
        self.lin3.reset_parameters()

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if isinstance(x, Tensor):
            x = (x, x)

        a = self.lin1(x[0])
        b = self.lin2(x[1])

        # propagate_type: (a: Tensor, b: Tensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, a=a, b=b, edge_weight=edge_weight,
                             size=None)

        return out + self.lin3(x[1])

    def message(self, a_j: Tensor, b_i: Tensor,
                edge_weight: OptTensor) -> Tensor:
        out = a_j - b_i
        return out if edge_weight is None else out * edge_weight.view(-1, 1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/conv/lg_conv.py000066400000000000000000000045621444262304300213150ustar00rootroot00000000000000from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class LGConv(MessagePassing):
    r"""The Light Graph Convolution (LGC) operator from the `"LightGCN:
    Simplifying and Powering Graph Convolution Network for Recommendation"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \sum_{j \in \mathcal{N}(i)}
        \frac{e_{j,i}}{\sqrt{\deg(i)\deg(j)}} \mathbf{x}_j

    Args:
        normalize (bool, optional): If set to :obj:`False`, output features
            will not be normalized via symmetric normalization.
            (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)`
    """
    def __init__(self, normalize: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)
        self.normalize = normalize

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if self.normalize and isinstance(edge_index, Tensor):
            out = gcn_norm(edge_index, edge_weight, x.size(self.node_dim),
                           add_self_loops=False, flow=self.flow, dtype=x.dtype)
            edge_index, edge_weight = out
        elif self.normalize and isinstance(edge_index, SparseTensor):
            edge_index = gcn_norm(edge_index, None, x.size(self.node_dim),
                                  add_self_loops=False, flow=self.flow,
                                  dtype=x.dtype)

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        return self.propagate(edge_index, x=x, edge_weight=edge_weight,
                              size=None)

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)
pytorch/torch_geometric/nn/conv/message_passing.jinja000066400000000000000000000264041444262304300235000ustar00rootroot00000000000000import typing
from typing import *
from torch_geometric.typing import *

import torch
from torch import Tensor
from torch_geometric.nn.conv.message_passing import *
from {{module}} import *


class Propagate_{{uid}}(NamedTuple):
{%- for k, v in prop_types.items() %}
    {{k}}: {{v}}
{%- endfor %}

{% if edge_updater_types|length > 0 %}
class EdgeUpdater_{{uid}}(NamedTuple):
{%- for k, v in edge_updater_types.items() %}
    {{k}}: {{v}}
{%- endfor %}
{% endif %}

class Collect_{{uid}}(NamedTuple):
{%- for k, v in collect_types.items() %}
    {{k}}: {{v}}
{%- endfor %}

{% if edge_updater_types|length > 0 %}
class EdgeUpdater_Collect_{{uid}}(NamedTuple):
{%- for k, v in edge_collect_types.items() %}
    {{k}}: {{v}}
{%- endfor %}
{% endif %}

class {{cls_name}}({{parent_cls_name}}):

    @torch.jit._overload_method
    def _check_input(self, edge_index, size):
        # type: (Tensor, Size) -> List[Optional[int]]
        pass

    @torch.jit._overload_method
    def _check_input(self, edge_index, size):
        # type: (SparseTensor, Size) -> List[Optional[int]]
        pass

{{check_input}}

    @torch.jit._overload_method
    def _lift(self, src, edge_index, dim):
        # type: (Tensor, Tensor, int) -> Tensor
        pass

    @torch.jit._overload_method
    def _lift(self, src, edge_index, dim):
        # type: (Tensor, SparseTensor, int) -> Tensor
        pass

    def _lift(self, src, edge_index, dim):
        if isinstance(edge_index, Tensor):
            index = edge_index[dim]
            return src.index_select(self.node_dim, index)

        elif isinstance(edge_index, SparseTensor):
            row, col, _ = edge_index.coo()
            if dim == 0:
                return src.index_select(self.node_dim, col)
            elif dim == 1:
                return src.index_select(self.node_dim, row)

        raise ValueError(
            ('`MessagePassing.propagate` only supports integer tensors of '
             'shape `[2, num_messages]` or `torch_sparse.SparseTensor` for '
             'argument `edge_index`.'))

    @torch.jit._overload_method
    def _collect(self, edge_def, size, kwargs):
        # type: (Tensor, List[Optional[int]], Propagate_{{uid}}) -> Collect_{{uid}}
        pass

    @torch.jit._overload_method
    def _collect(self, edge_def, size, kwargs):
        # type: (SparseTensor, List[Optional[int]], Propagate_{{uid}}) -> Collect_{{uid}}
        pass

    def _collect(self, edge_def, size, kwargs):
        init = torch.tensor(0.)
        i, j = (1, 0) if self.flow == 'source_to_target' else (0, 1)
{% for arg in user_args %}
{%- if arg[-2:] not in ['_i', '_j'] %}
        {{arg}} = kwargs.{{arg}}
{%- else %}
        {{arg}}: {{collect_types[arg]}} = {% if collect_types[arg][:8] == 'Optional' %}None{% else %}init{% endif %}
        data = kwargs.{{arg[:-2]}}
        if isinstance(data, (tuple, list)):
            assert len(data) == 2
{%- if arg[-2:] == '_j' %}
            tmp = data[1]
            if isinstance(tmp, Tensor):
                self._set_size(size, 1, tmp)
            {{arg}} = data[0]
{%- else %}
            tmp = data[0]
            if isinstance(tmp, Tensor):
                self._set_size(size, 0, tmp)
            {{arg}} = data[1]
{%- endif %}
        else:
            {{arg}} = data
        if isinstance({{arg}}, Tensor):
            self._set_size(size, {% if arg[-2:] == '_j'%}0{% else %}1{% endif %}, {{arg}})
            {{arg}} = self._lift({{arg}}, edge_def, {% if arg[-2:] == "_j" %}j{% else %}i{% endif %})
{%- endif %}
{%- endfor %}

        edge_index: Optional[Tensor] = None
        adj_t: Optional[SparseTensor] = None
        edge_index_i: torch.Tensor = init
        edge_index_j: torch.Tensor = init
        ptr: Optional[Tensor] = None
        if isinstance(edge_def, Tensor):
            edge_index = edge_def
            edge_index_i = edge_def[i]
            edge_index_j = edge_def[j]
        elif isinstance(edge_def, SparseTensor):
            adj_t = edge_def
            edge_index_i, edge_index_j, value = edge_def.coo()
            ptr, _, _ = edge_def.csr()

            {% if 'edge_weight' in collect_types.keys() %}
            if edge_weight is None:
                edge_weight = value
            {% endif %}
            {% if 'edge_attr' in collect_types.keys() %}
            if edge_attr is None:
                edge_attr = value
            {% endif %}
            {% if 'edge_type' in collect_types.keys() %}
            if edge_type is None:
                edge_type = value
            {% endif %}

        {% if collect_types.get('edge_weight', 'Optional')[:8] != 'Optional' %}assert edge_weight is not None{% endif %}
        {% if collect_types.get('edge_attr', 'Optional')[:8] != 'Optional' %}assert edge_attr is not None{% endif %}
        {% if collect_types.get('edge_type', 'Optional')[:8] != 'Optional' %}assert edge_type is not None{% endif %}

        index = edge_index_i
        size_i = size[1] if size[1] is not None else size[0]
        size_j = size[0] if size[0] is not None else size[1]
        dim_size = size_i

        return Collect_{{uid}}({% for k in collect_types.keys() %}{{k}}={{k}}{{ ", " if not loop.last }}{% endfor %})

{% if edge_updater_types|length > 0 %}
    @torch.jit._overload_method
    def _collect_edge(self, edge_def, size, kwargs):
        # type: (Tensor, List[Optional[int]], EdgeUpdater_{{uid}}) -> EdgeUpdater_Collect_{{uid}}
        pass

    @torch.jit._overload_method
    def _collect_edge(self, edge_def, size, kwargs):
        # type: (SparseTensor, List[Optional[int]], EdgeUpdater_{{uid}}) -> EdgeUpdater_Collect_{{uid}}
        pass

    def _collect_edge(self, edge_def, size, kwargs):
        init = torch.tensor(0.)
        i, j = (1, 0) if self.flow == 'source_to_target' else (0, 1)
{% for arg in edge_user_args %}
{%- if arg[-2:] not in ['_i', '_j'] %}
        {{arg}} = kwargs.{{arg}}
{%- else %}
        {{arg}}: {{edge_collect_types[arg]}} = {% if edge_collect_types[arg][:8] == 'Optional' %}None{% else %}init{% endif %}
        data = kwargs.{{arg[:-2]}}
        if isinstance(data, (tuple, list)):
            assert len(data) == 2
{%- if arg[-2:] == '_j' %}
            tmp = data[1]
            if isinstance(tmp, Tensor):
                self._set_size(size, 1, tmp)
            {{arg}} = data[0]
{%- else %}
            tmp = data[0]
            if isinstance(tmp, Tensor):
                self._set_size(size, 0, tmp)
            {{arg}} = data[1]
{%- endif %}
        else:
            {{arg}} = data
        if isinstance({{arg}}, Tensor):
            self._set_size(size, {% if arg[-2:] == '_j'%}0{% else %}1{% endif %}, {{arg}})
            {{arg}} = self._lift({{arg}}, edge_def, {% if arg[-2:] == "_j" %}j{% else %}i{% endif %})
{%- endif %}
{%- endfor %}

        edge_index: Optional[Tensor] = None
        adj_t: Optional[SparseTensor] = None
        edge_index_i: torch.Tensor = init
        edge_index_j: torch.Tensor = init
        ptr: Optional[Tensor] = None
        if isinstance(edge_def, Tensor):
            edge_index = edge_def
            edge_index_i = edge_def[i]
            edge_index_j = edge_def[j]
        elif isinstance(edge_def, SparseTensor):
            adj_t = edge_def
            edge_index_i, edge_index_j, value = edge_def.coo()
            ptr, _, _ = edge_def.csr()

            {% if 'edge_weight' in edge_collect_types.keys() %}
            if edge_weight is None:
                edge_weight = value
            {% endif %}
            {% if 'edge_attr' in edge_collect_types.keys() %}
            if edge_attr is None:
                edge_attr = value
            {% endif %}
            {% if 'edge_type' in edge_collect_types.keys() %}
            if edge_type is None:
                edge_type = value
            {% endif %}

        {% if edge_collect_types.get('edge_weight', 'Optional')[:8] != 'Optional' %}assert edge_weight is not None{% endif %}
        {% if edge_collect_types.get('edge_attr', 'Optional')[:8] != 'Optional' %}assert edge_attr is not None{% endif %}
        {% if edge_collect_types.get('edge_type', 'Optional')[:8] != 'Optional' %}assert edge_type is not None{% endif %}

        index = edge_index_i
        size_i = size[1] if size[1] is not None else size[0]
        size_j = size[0] if size[0] is not None else size[1]
        dim_size = size_i

        return EdgeUpdater_Collect_{{uid}}({% for k in edge_collect_types.keys() %}{{k}}={{k}}{{ ", " if not loop.last }}{% endfor %})


{% endif %}

    @torch.jit._overload_method
    def propagate(self, edge_index, {{ prop_types.keys()|join(', ') }}, size=None):
        # type: (Tensor, {{ prop_types.values()|join(', ') }}, Size) -> {{ prop_return_type }}
        pass

    @torch.jit._overload_method
    def propagate(self, edge_index, {{ prop_types.keys()|join(', ') }}, size=None):
        # type: (SparseTensor, {{ prop_types.values()|join(', ') }}, Size) -> {{ prop_return_type }}
        pass

    def propagate(self, edge_index, {{ prop_types.keys()|join(', ') }}, size=None):
        the_size = self._check_input(edge_index, size)
        in_kwargs = Propagate_{{uid}}({% for k in prop_types.keys() %}{{k}}={{k}}{{ ", " if not loop.last }}{% endfor %})

        {% if fuse %}
        if isinstance(edge_index, SparseTensor):
            out = self.message_and_aggregate(edge_index{% for k in msg_and_aggr_args %}, {{k}}=in_kwargs.{{k}}{% endfor %})
            return self.update(out{% for k in update_args %}, {{k}}=in_kwargs.{{k}}{% endfor %})
        {% endif %}

        kwargs = self._collect(edge_index, the_size, in_kwargs)
        out = self.message({% for k in msg_args %}{{k}}=kwargs.{{k}}{{ ", " if not loop.last }}{% endfor %})
        out = self.aggregate(out{% for k in aggr_args %}, {{k}}=kwargs.{{k}}{% endfor %})
        return self.update(out{% for k in update_args %}, {{k}}=kwargs.{{k}}{% endfor %})

{% if edge_updater_types|length > 0 %}
    @torch.jit._overload_method
    def edge_updater(self, edge_index{% for k in edge_updater_types.keys() %}, {{k}} {% endfor %}):
        # type: (Tensor{% for v in edge_updater_types.values() %}, {{v}} {% endfor %}) -> {{ edge_updater_return_type }}
        pass

    @torch.jit._overload_method
    def edge_updater(self, edge_index{% for k in edge_updater_types.keys() %}, {{k}} {% endfor %}):
        # type: (SparseTensor{% for v in edge_updater_types.values() %}, {{v}} {% endfor %}) -> {{ edge_updater_return_type }}
        pass

    def edge_updater(self, edge_index{% for k in edge_updater_types.keys() %}, {{k}} {% endfor %}):
        the_size = self._check_input(edge_index, size=None)
        in_kwargs = EdgeUpdater_{{uid}}({% for k in edge_updater_types.keys() %}{{k}}={{k}}{{ ", " if not loop.last }}{% endfor %})
        kwargs = self._collect_edge(edge_index, the_size, in_kwargs)
        return self.edge_update({% for k in edge_update_args %}{{k}}=kwargs.{{k}}{{ ", " if not loop.last }}{% endfor %})
{% else %}
    def edge_updater(self):
        pass
{% endif %}

    @property
    def explain(self) -> bool:
        return self._explain

    @explain.setter
    def explain(self, explain: bool):
        raise ValueError("Explainability of message passing modules "
                         "is only supported on the Python module")

{%- for (arg_types, return_type_repr) in forward_types %}

    @torch.jit._overload_method
    {{forward_header}}
        # type: ({{arg_types|join(', ')}}) -> {{return_type_repr}}
        pass
{%- endfor %}

    {{forward_header}}
{{forward_body}}
pytorch/torch_geometric/nn/conv/message_passing.py000066400000000000000000001156631444262304300230430ustar00rootroot00000000000000import inspect
import os
import os.path as osp
import random
import re
from collections import OrderedDict
from inspect import Parameter
from itertools import chain
from typing import (
    Any,
    Callable,
    Dict,
    List,
    Optional,
    Set,
    Union,
    get_type_hints,
)

import torch
from torch import Tensor
from torch.utils.hooks import RemovableHandle

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.conv.utils.inspector import (
    Inspector,
    func_body_repr,
    func_header_repr,
)
from torch_geometric.nn.conv.utils.jit import class_from_module_repr
from torch_geometric.nn.conv.utils.typing import (
    parse_types,
    resolve_types,
    sanitize,
    split_types_repr,
)
from torch_geometric.nn.resolver import aggregation_resolver as aggr_resolver
from torch_geometric.typing import Adj, Size, SparseTensor
from torch_geometric.utils import (
    is_sparse,
    is_torch_sparse_tensor,
    to_edge_index,
)
from torch_geometric.utils.sparse import ptr2index

FUSE_AGGRS = {'add', 'sum', 'mean', 'min', 'max'}


def ptr2ind(ptr: Tensor) -> Tensor:
    ind = torch.arange(ptr.numel() - 1, device=ptr.device)
    return ind.repeat_interleave(ptr[1:] - ptr[:-1])


class MessagePassing(torch.nn.Module):
    r"""Base class for creating message passing layers of the form

    .. math::
        \mathbf{x}_i^{\prime} = \gamma_{\mathbf{\Theta}} \left( \mathbf{x}_i,
        \bigoplus_{j \in \mathcal{N}(i)} \, \phi_{\mathbf{\Theta}}
        \left(\mathbf{x}_i, \mathbf{x}_j,\mathbf{e}_{j,i}\right) \right),

    where :math:`\bigoplus` denotes a differentiable, permutation invariant
    function, *e.g.*, sum, mean, min, max or mul, and
    :math:`\gamma_{\mathbf{\Theta}}` and :math:`\phi_{\mathbf{\Theta}}` denote
    differentiable functions such as MLPs.
    See `here `__ for the accompanying tutorial.

    Args:
        aggr (str or [str] or Aggregation, optional): The aggregation scheme
            to use, *e.g.*, :obj:`"add"`, :obj:`"sum"` :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"` or :obj:`"mul"`.
            In addition, can be any
            :class:`~torch_geometric.nn.aggr.Aggregation` module (or any string
            that automatically resolves to it).
            If given as a list, will make use of multiple aggregations in which
            different outputs will get concatenated in the last dimension.
            If set to :obj:`None`, the :class:`MessagePassing` instantiation is
            expected to implement its own aggregation logic via
            :meth:`aggregate`. (default: :obj:`"add"`)
        aggr_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective aggregation function in case it gets automatically
            resolved. (default: :obj:`None`)
        flow (str, optional): The flow direction of message passing
            (:obj:`"source_to_target"` or :obj:`"target_to_source"`).
            (default: :obj:`"source_to_target"`)
        node_dim (int, optional): The axis along which to propagate.
            (default: :obj:`-2`)
        decomposed_layers (int, optional): The number of feature decomposition
            layers, as introduced in the `"Optimizing Memory Efficiency of
            Graph Neural Networks on Edge Computing Platforms"
            `_ paper.
            Feature decomposition reduces the peak memory usage by slicing
            the feature dimensions into separated feature decomposition layers
            during GNN aggregation.
            This method can accelerate GNN execution on CPU-based platforms
            (*e.g.*, 2-3x speedup on the
            :class:`~torch_geometric.datasets.Reddit` dataset) for common GNN
            models such as :class:`~torch_geometric.nn.models.GCN`,
            :class:`~torch_geometric.nn.models.GraphSAGE`,
            :class:`~torch_geometric.nn.models.GIN`, etc.
            However, this method is not applicable to all GNN operators
            available, in particular for operators in which message computation
            can not easily be decomposed, *e.g.* in attention-based GNNs.
            The selection of the optimal value of :obj:`decomposed_layers`
            depends both on the specific graph dataset and available hardware
            resources.
            A value of :obj:`2` is suitable in most cases.
            Although the peak memory usage is directly associated with the
            granularity of feature decomposition, the same is not necessarily
            true for execution speedups. (default: :obj:`1`)
    """

    special_args: Set[str] = {
        'edge_index', 'adj_t', 'edge_index_i', 'edge_index_j', 'size',
        'size_i', 'size_j', 'ptr', 'index', 'dim_size'
    }

    def __init__(
        self,
        aggr: Optional[Union[str, List[str], Aggregation]] = "add",
        *,
        aggr_kwargs: Optional[Dict[str, Any]] = None,
        flow: str = "source_to_target",
        node_dim: int = -2,
        decomposed_layers: int = 1,
        **kwargs,
    ):
        super().__init__()

        if aggr is None:
            self.aggr = None
        elif isinstance(aggr, (str, Aggregation)):
            self.aggr = str(aggr)
        elif isinstance(aggr, (tuple, list)):
            self.aggr = [str(x) for x in aggr]

        self.aggr_module = aggr_resolver(aggr, **(aggr_kwargs or {}))

        self.flow = flow

        if flow not in ['source_to_target', 'target_to_source']:
            raise ValueError(f"Expected 'flow' to be either 'source_to_target'"
                             f" or 'target_to_source' (got '{flow}')")

        self.node_dim = node_dim
        self.decomposed_layers = decomposed_layers

        self.inspector = Inspector(self)
        self.inspector.inspect(self.message)
        self.inspector.inspect(self.aggregate, pop_first=True)
        self.inspector.params['aggregate'].pop('aggr', None)
        self.inspector.inspect(self.message_and_aggregate, pop_first=True)
        self.inspector.inspect(self.update, pop_first=True)
        self.inspector.inspect(self.edge_update)

        self._user_args = self.inspector.keys(
            ['message', 'aggregate', 'update']).difference(self.special_args)
        self._fused_user_args = self.inspector.keys(
            ['message_and_aggregate', 'update']).difference(self.special_args)
        self._edge_user_args = self.inspector.keys(['edge_update']).difference(
            self.special_args)

        # Support for "fused" message passing.
        self.fuse = self.inspector.implements('message_and_aggregate')
        if self.aggr is not None:
            self.fuse &= isinstance(self.aggr, str) and self.aggr in FUSE_AGGRS

        # Support for explainability.
        self._explain = False
        self._edge_mask = None
        self._loop_mask = None
        self._apply_sigmoid = True

        # Hooks:
        self._propagate_forward_pre_hooks = OrderedDict()
        self._propagate_forward_hooks = OrderedDict()
        self._message_forward_pre_hooks = OrderedDict()
        self._message_forward_hooks = OrderedDict()
        self._aggregate_forward_pre_hooks = OrderedDict()
        self._aggregate_forward_hooks = OrderedDict()
        self._message_and_aggregate_forward_pre_hooks = OrderedDict()
        self._message_and_aggregate_forward_hooks = OrderedDict()
        self._edge_update_forward_pre_hooks = OrderedDict()
        self._edge_update_forward_hooks = OrderedDict()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        if self.aggr_module is not None:
            self.aggr_module.reset_parameters()

    def forward(self, *args, **kwargs) -> Any:
        r"""Runs the forward pass of the module."""
        pass

    def _check_input(self, edge_index, size):
        the_size: List[Optional[int]] = [None, None]

        if is_sparse(edge_index):
            if self.flow == 'target_to_source':
                raise ValueError(
                    ('Flow direction "target_to_source" is invalid for '
                     'message propagation via `torch_sparse.SparseTensor` '
                     'or `torch.sparse.Tensor`. If you really want to make '
                     'use of a reverse message passing flow, pass in the '
                     'transposed sparse tensor to the message passing module, '
                     'e.g., `adj_t.t()`.'))
            the_size[0] = edge_index.size(1)
            the_size[1] = edge_index.size(0)
            return the_size
        elif isinstance(edge_index, Tensor):
            int_dtypes = (torch.uint8, torch.int8, torch.int32, torch.int64)

            if edge_index.dtype not in int_dtypes:
                raise ValueError(f"Expected 'edge_index' to be of integer "
                                 f"type (got '{edge_index.dtype}')")
            if edge_index.dim() != 2:
                raise ValueError(f"Expected 'edge_index' to be two-dimensional"
                                 f" (got {edge_index.dim()} dimensions)")
            if edge_index.size(0) != 2:
                raise ValueError(f"Expected 'edge_index' to have size '2' in "
                                 f"the first dimension (got "
                                 f"'{edge_index.size(0)}')")
            if size is not None:
                the_size[0] = size[0]
                the_size[1] = size[1]
            return the_size

        raise ValueError(
            ('`MessagePassing.propagate` only supports integer tensors of '
             'shape `[2, num_messages]`, `torch_sparse.SparseTensor` or '
             '`torch.sparse.Tensor` for argument `edge_index`.'))

    def _set_size(self, size: List[Optional[int]], dim: int, src: Tensor):
        the_size = size[dim]
        if the_size is None:
            size[dim] = src.size(self.node_dim)
        elif the_size != src.size(self.node_dim):
            raise ValueError(
                (f'Encountered tensor with size {src.size(self.node_dim)} in '
                 f'dimension {self.node_dim}, but expected size {the_size}.'))

    def _lift(self, src, edge_index, dim):
        if is_torch_sparse_tensor(edge_index):
            assert dim == 0 or dim == 1
            if edge_index.layout == torch.sparse_coo:
                index = edge_index._indices()[1 - dim]
            elif edge_index.layout == torch.sparse_csr:
                if dim == 0:
                    index = edge_index.col_indices()
                else:
                    index = ptr2index(edge_index.crow_indices())
            elif edge_index.layout == torch.sparse_csc:
                if dim == 0:
                    index = ptr2index(edge_index.ccol_indices())
                else:
                    index = edge_index.row_indices()
            else:
                raise ValueError(f"Unsupported sparse tensor layout "
                                 f"(got '{edge_index.layout}')")
            return src.index_select(self.node_dim, index)

        elif isinstance(edge_index, Tensor):
            try:
                index = edge_index[dim]
                return src.index_select(self.node_dim, index)
            except (IndexError, RuntimeError) as e:
                if index.min() < 0 or index.max() >= src.size(self.node_dim):
                    raise IndexError(
                        f"Encountered an index error. Please ensure that all "
                        f"indices in 'edge_index' point to valid indices in "
                        f"the interval [0, {src.size(self.node_dim) - 1}] "
                        f"(got interval "
                        f"[{int(index.min())}, {int(index.max())}])")
                else:
                    raise e

                if index.numel() > 0 and index.min() < 0:
                    raise ValueError(
                        f"Found negative indices in 'edge_index' (got "
                        f"{index.min().item()}). Please ensure that all "
                        f"indices in 'edge_index' point to valid indices "
                        f"in the interval [0, {src.size(self.node_dim)}) in "
                        f"your node feature matrix and try again.")

                if (index.numel() > 0
                        and index.max() >= src.size(self.node_dim)):
                    raise ValueError(
                        f"Found indices in 'edge_index' that are larger "
                        f"than {src.size(self.node_dim) - 1} (got "
                        f"{index.max().item()}). Please ensure that all "
                        f"indices in 'edge_index' point to valid indices "
                        f"in the interval [0, {src.size(self.node_dim)}) in "
                        f"your node feature matrix and try again.")

                raise e

        elif isinstance(edge_index, SparseTensor):
            row, col, _ = edge_index.coo()
            if dim == 0:
                return src.index_select(self.node_dim, col)
            elif dim == 1:
                return src.index_select(self.node_dim, row)

        raise ValueError(
            ('`MessagePassing.propagate` only supports integer tensors of '
             'shape `[2, num_messages]`, `torch_sparse.SparseTensor` '
             'or `torch.sparse.Tensor` for argument `edge_index`.'))

    def _collect(self, args, edge_index, size, kwargs):
        i, j = (1, 0) if self.flow == 'source_to_target' else (0, 1)

        out = {}
        for arg in args:
            if arg[-2:] not in ['_i', '_j']:
                out[arg] = kwargs.get(arg, Parameter.empty)
            else:
                dim = j if arg[-2:] == '_j' else i
                data = kwargs.get(arg[:-2], Parameter.empty)

                if isinstance(data, (tuple, list)):
                    assert len(data) == 2
                    if isinstance(data[1 - dim], Tensor):
                        self._set_size(size, 1 - dim, data[1 - dim])
                    data = data[dim]

                if isinstance(data, Tensor):
                    self._set_size(size, dim, data)
                    data = self._lift(data, edge_index, dim)

                out[arg] = data

        if is_torch_sparse_tensor(edge_index):
            indices, values = to_edge_index(edge_index)
            out['adj_t'] = edge_index
            out['edge_index'] = None
            out['edge_index_i'] = indices[0]
            out['edge_index_j'] = indices[1]
            out['ptr'] = None  # TODO Get `rowptr` from CSR representation.
            if out.get('edge_weight', None) is None:
                out['edge_weight'] = values
            if out.get('edge_attr', None) is None:
                out['edge_attr'] = None if values.dim() == 1 else values
            if out.get('edge_type', None) is None:
                out['edge_type'] = values

        elif isinstance(edge_index, Tensor):
            out['adj_t'] = None
            out['edge_index'] = edge_index
            out['edge_index_i'] = edge_index[i]
            out['edge_index_j'] = edge_index[j]
            out['ptr'] = None

        elif isinstance(edge_index, SparseTensor):
            row, col, value = edge_index.coo()
            rowptr, _, _ = edge_index.csr()

            out['adj_t'] = edge_index
            out['edge_index'] = None
            out['edge_index_i'] = row
            out['edge_index_j'] = col
            out['ptr'] = rowptr
            if out.get('edge_weight', None) is None:
                out['edge_weight'] = value
            if out.get('edge_attr', None) is None:
                out['edge_attr'] = value
            if out.get('edge_type', None) is None:
                out['edge_type'] = value

        out['index'] = out['edge_index_i']
        out['size'] = size
        out['size_i'] = size[i] if size[i] is not None else size[j]
        out['size_j'] = size[j] if size[j] is not None else size[i]
        out['dim_size'] = out['size_i']

        return out

    def propagate(self, edge_index: Adj, size: Size = None, **kwargs):
        r"""The initial call to start propagating messages.

        Args:
            edge_index (torch.Tensor or SparseTensor): A :class:`torch.Tensor`,
                a :class:`torch_sparse.SparseTensor` or a
                :class:`torch.sparse.Tensor` that defines the underlying
                graph connectivity/message passing flow.
                :obj:`edge_index` holds the indices of a general (sparse)
                assignment matrix of shape :obj:`[N, M]`.
                If :obj:`edge_index` is a :obj:`torch.Tensor`, its :obj:`dtype`
                should be :obj:`torch.long` and its shape needs to be defined
                as :obj:`[2, num_messages]` where messages from nodes in
                :obj:`edge_index[0]` are sent to nodes in :obj:`edge_index[1]`
                (in case :obj:`flow="source_to_target"`).
                If :obj:`edge_index` is a :class:`torch_sparse.SparseTensor` or
                a :class:`torch.sparse.Tensor`, its sparse indices
                :obj:`(row, col)` should relate to :obj:`row = edge_index[1]`
                and :obj:`col = edge_index[0]`.
                The major difference between both formats is that we need to
                input the *transposed* sparse adjacency matrix into
                :meth:`propagate`.
            size ((int, int), optional): The size :obj:`(N, M)` of the
                assignment matrix in case :obj:`edge_index` is a
                :class:`torch.Tensor`.
                If set to :obj:`None`, the size will be automatically inferred
                and assumed to be quadratic.
                This argument is ignored in case :obj:`edge_index` is a
                :class:`torch_sparse.SparseTensor` or
                a :class:`torch.sparse.Tensor`. (default: :obj:`None`)
            **kwargs: Any additional data which is needed to construct and
                aggregate messages, and to update node embeddings.
        """
        decomposed_layers = 1 if self.explain else self.decomposed_layers

        for hook in self._propagate_forward_pre_hooks.values():
            res = hook(self, (edge_index, size, kwargs))
            if res is not None:
                edge_index, size, kwargs = res

        size = self._check_input(edge_index, size)

        # Run "fused" message and aggregation (if applicable).
        if is_sparse(edge_index) and self.fuse and not self.explain:
            coll_dict = self._collect(self._fused_user_args, edge_index, size,
                                      kwargs)

            msg_aggr_kwargs = self.inspector.distribute(
                'message_and_aggregate', coll_dict)
            for hook in self._message_and_aggregate_forward_pre_hooks.values():
                res = hook(self, (edge_index, msg_aggr_kwargs))
                if res is not None:
                    edge_index, msg_aggr_kwargs = res
            out = self.message_and_aggregate(edge_index, **msg_aggr_kwargs)
            for hook in self._message_and_aggregate_forward_hooks.values():
                res = hook(self, (edge_index, msg_aggr_kwargs), out)
                if res is not None:
                    out = res

            update_kwargs = self.inspector.distribute('update', coll_dict)
            out = self.update(out, **update_kwargs)

        else:  # Otherwise, run both functions in separation.
            if decomposed_layers > 1:
                user_args = self._user_args
                decomp_args = {a[:-2] for a in user_args if a[-2:] == '_j'}
                decomp_kwargs = {
                    a: kwargs[a].chunk(decomposed_layers, -1)
                    for a in decomp_args
                }
                decomp_out = []

            for i in range(decomposed_layers):
                if decomposed_layers > 1:
                    for arg in decomp_args:
                        kwargs[arg] = decomp_kwargs[arg][i]

                coll_dict = self._collect(self._user_args, edge_index, size,
                                          kwargs)

                msg_kwargs = self.inspector.distribute('message', coll_dict)
                for hook in self._message_forward_pre_hooks.values():
                    res = hook(self, (msg_kwargs, ))
                    if res is not None:
                        msg_kwargs = res[0] if isinstance(res, tuple) else res
                out = self.message(**msg_kwargs)
                for hook in self._message_forward_hooks.values():
                    res = hook(self, (msg_kwargs, ), out)
                    if res is not None:
                        out = res

                if self.explain:
                    explain_msg_kwargs = self.inspector.distribute(
                        'explain_message', coll_dict)
                    out = self.explain_message(out, **explain_msg_kwargs)

                aggr_kwargs = self.inspector.distribute('aggregate', coll_dict)
                for hook in self._aggregate_forward_pre_hooks.values():
                    res = hook(self, (aggr_kwargs, ))
                    if res is not None:
                        aggr_kwargs = res[0] if isinstance(res, tuple) else res

                out = self.aggregate(out, **aggr_kwargs)

                for hook in self._aggregate_forward_hooks.values():
                    res = hook(self, (aggr_kwargs, ), out)
                    if res is not None:
                        out = res

                update_kwargs = self.inspector.distribute('update', coll_dict)
                out = self.update(out, **update_kwargs)

                if decomposed_layers > 1:
                    decomp_out.append(out)

            if decomposed_layers > 1:
                out = torch.cat(decomp_out, dim=-1)

        for hook in self._propagate_forward_hooks.values():
            res = hook(self, (edge_index, size, kwargs), out)
            if res is not None:
                out = res

        return out

    def edge_updater(self, edge_index: Adj, **kwargs):
        r"""The initial call to compute or update features for each edge in the
        graph.

        Args:
            edge_index (torch.Tensor or SparseTensor): A :obj:`torch.Tensor`, a
                :class:`torch_sparse.SparseTensor` or a
                :class:`torch.sparse.Tensor` that defines the underlying graph
                connectivity/message passing flow.
                See :meth:`propagate` for more information.
            **kwargs: Any additional data which is needed to compute or update
                features for each edge in the graph.
        """
        for hook in self._edge_update_forward_pre_hooks.values():
            res = hook(self, (edge_index, kwargs))
            if res is not None:
                edge_index, kwargs = res

        size = self._check_input(edge_index, size=None)

        coll_dict = self._collect(self._edge_user_args, edge_index, size,
                                  kwargs)

        edge_kwargs = self.inspector.distribute('edge_update', coll_dict)
        out = self.edge_update(**edge_kwargs)

        for hook in self._edge_update_forward_hooks.values():
            res = hook(self, (edge_index, kwargs), out)
            if res is not None:
                out = res

        return out

    def message(self, x_j: Tensor) -> Tensor:
        r"""Constructs messages from node :math:`j` to node :math:`i`
        in analogy to :math:`\phi_{\mathbf{\Theta}}` for each edge in
        :obj:`edge_index`.
        This function can take any argument as input which was initially
        passed to :meth:`propagate`.
        Furthermore, tensors passed to :meth:`propagate` can be mapped to the
        respective nodes :math:`i` and :math:`j` by appending :obj:`_i` or
        :obj:`_j` to the variable name, *.e.g.* :obj:`x_i` and :obj:`x_j`.
        """
        return x_j

    @property
    def explain(self) -> bool:
        return self._explain

    @explain.setter
    def explain(self, explain: bool):
        if explain:
            methods = ['message', 'explain_message', 'aggregate', 'update']
        else:
            methods = ['message', 'aggregate', 'update']

        self._explain = explain
        self.inspector.inspect(self.explain_message, pop_first=True)
        self._user_args = self.inspector.keys(methods).difference(
            self.special_args)

    def explain_message(self, inputs: Tensor, size_i: int) -> Tensor:
        # NOTE Replace this method in custom explainers per message-passing
        # layer to customize how messages shall be explained, e.g., via:
        # conv.explain_message = explain_message.__get__(conv, MessagePassing)
        # see stackoverflow.com: 394770/override-a-method-at-instance-level

        edge_mask = self._edge_mask

        if edge_mask is None:
            raise ValueError(f"Could not find a pre-defined 'edge_mask' as "
                             f"part of {self.__class__.__name__}.")

        if self._apply_sigmoid:
            edge_mask = edge_mask.sigmoid()

        # Some ops add self-loops to `edge_index`. We need to do the same for
        # `edge_mask` (but do not train these entries).
        if inputs.size(self.node_dim) != edge_mask.size(0):
            edge_mask = edge_mask[self._loop_mask]
            loop = edge_mask.new_ones(size_i)
            edge_mask = torch.cat([edge_mask, loop], dim=0)
        assert inputs.size(self.node_dim) == edge_mask.size(0)

        size = [1] * inputs.dim()
        size[self.node_dim] = -1
        return inputs * edge_mask.view(size)

    def aggregate(self, inputs: Tensor, index: Tensor,
                  ptr: Optional[Tensor] = None,
                  dim_size: Optional[int] = None) -> Tensor:
        r"""Aggregates messages from neighbors as
        :math:`\bigoplus_{j \in \mathcal{N}(i)}`.

        Takes in the output of message computation as first argument and any
        argument which was initially passed to :meth:`propagate`.

        By default, this function will delegate its call to the underlying
        :class:`~torch_geometric.nn.aggr.Aggregation` module to reduce messages
        as specified in :meth:`__init__` by the :obj:`aggr` argument.
        """
        return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
                                dim=self.node_dim)

    def message_and_aggregate(
        self,
        adj_t: Union[SparseTensor, Tensor],
    ) -> Tensor:
        r"""Fuses computations of :func:`message` and :func:`aggregate` into a
        single function.
        If applicable, this saves both time and memory since messages do not
        explicitly need to be materialized.
        This function will only gets called in case it is implemented and
        propagation takes place based on a :obj:`torch_sparse.SparseTensor`
        or a :obj:`torch.sparse.Tensor`.
        """
        raise NotImplementedError

    def update(self, inputs: Tensor) -> Tensor:
        r"""Updates node embeddings in analogy to
        :math:`\gamma_{\mathbf{\Theta}}` for each node
        :math:`i \in \mathcal{V}`.
        Takes in the output of aggregation as first argument and any argument
        which was initially passed to :meth:`propagate`.
        """
        return inputs

    def edge_update(self) -> Tensor:
        r"""Computes or updates features for each edge in the graph.
        This function can take any argument as input which was initially passed
        to :meth:`edge_updater`.
        Furthermore, tensors passed to :meth:`edge_updater` can be mapped to
        the respective nodes :math:`i` and :math:`j` by appending :obj:`_i` or
        :obj:`_j` to the variable name, *.e.g.* :obj:`x_i` and :obj:`x_j`.
        """
        raise NotImplementedError

    def register_propagate_forward_pre_hook(self,
                                            hook: Callable) -> RemovableHandle:
        r"""Registers a forward pre-hook on the module.
        The hook will be called every time before :meth:`propagate` is invoked.
        It should have the following signature:

        .. code-block:: python

            hook(module, inputs) -> None or modified input

        The hook can modify the input.
        Input keyword arguments are passed to the hook as a dictionary in
        :obj:`inputs[-1]`.

        Returns a :class:`torch.utils.hooks.RemovableHandle` that can be used
        to remove the added hook by calling :obj:`handle.remove()`.
        """
        handle = RemovableHandle(self._propagate_forward_pre_hooks)
        self._propagate_forward_pre_hooks[handle.id] = hook
        return handle

    def register_propagate_forward_hook(self,
                                        hook: Callable) -> RemovableHandle:
        r"""Registers a forward hook on the module.
        The hook will be called every time after :meth:`propagate` has computed
        an output.
        It should have the following signature:

        .. code-block:: python

            hook(module, inputs, output) -> None or modified output

        The hook can modify the output.
        Input keyword arguments are passed to the hook as a dictionary in
        :obj:`inputs[-1]`.

        Returns a :class:`torch.utils.hooks.RemovableHandle` that can be used
        to remove the added hook by calling :obj:`handle.remove()`.
        """
        handle = RemovableHandle(self._propagate_forward_hooks)
        self._propagate_forward_hooks[handle.id] = hook
        return handle

    def register_message_forward_pre_hook(self,
                                          hook: Callable) -> RemovableHandle:
        r"""Registers a forward pre-hook on the module.
        The hook will be called every time before :meth:`message` is invoked.
        See :meth:`register_propagate_forward_pre_hook` for more information.
        """
        handle = RemovableHandle(self._message_forward_pre_hooks)
        self._message_forward_pre_hooks[handle.id] = hook
        return handle

    def register_message_forward_hook(self, hook: Callable) -> RemovableHandle:
        r"""Registers a forward hook on the module.
        The hook will be called every time after :meth:`message` has computed
        an output.
        See :meth:`register_propagate_forward_hook` for more information.
        """
        handle = RemovableHandle(self._message_forward_hooks)
        self._message_forward_hooks[handle.id] = hook
        return handle

    def register_aggregate_forward_pre_hook(self,
                                            hook: Callable) -> RemovableHandle:
        r"""Registers a forward pre-hook on the module.
        The hook will be called every time before :meth:`aggregate` is invoked.
        See :meth:`register_propagate_forward_pre_hook` for more information.
        """
        handle = RemovableHandle(self._aggregate_forward_pre_hooks)
        self._aggregate_forward_pre_hooks[handle.id] = hook
        return handle

    def register_aggregate_forward_hook(self,
                                        hook: Callable) -> RemovableHandle:
        r"""Registers a forward hook on the module.
        The hook will be called every time after :meth:`aggregate` has computed
        an output.
        See :meth:`register_propagate_forward_hook` for more information.
        """
        handle = RemovableHandle(self._aggregate_forward_hooks)
        self._aggregate_forward_hooks[handle.id] = hook
        return handle

    def register_message_and_aggregate_forward_pre_hook(
            self, hook: Callable) -> RemovableHandle:
        r"""Registers a forward pre-hook on the module.
        The hook will be called every time before :meth:`message_and_aggregate`
        is invoked.
        See :meth:`register_propagate_forward_pre_hook` for more information.
        """
        handle = RemovableHandle(self._message_and_aggregate_forward_pre_hooks)
        self._message_and_aggregate_forward_pre_hooks[handle.id] = hook
        return handle

    def register_message_and_aggregate_forward_hook(
            self, hook: Callable) -> RemovableHandle:
        r"""Registers a forward hook on the module.
        The hook will be called every time after :meth:`message_and_aggregate`
        has computed an output.
        See :meth:`register_propagate_forward_hook` for more information.
        """
        handle = RemovableHandle(self._message_and_aggregate_forward_hooks)
        self._message_and_aggregate_forward_hooks[handle.id] = hook
        return handle

    def register_edge_update_forward_pre_hook(
            self, hook: Callable) -> RemovableHandle:
        r"""Registers a forward pre-hook on the module.
        The hook will be called every time before :meth:`edge_update` is
        invoked. See :meth:`register_propagate_forward_pre_hook` for more
        information.
        """
        handle = RemovableHandle(self._edge_update_forward_pre_hooks)
        self._edge_update_forward_pre_hooks[handle.id] = hook
        return handle

    def register_edge_update_forward_hook(self,
                                          hook: Callable) -> RemovableHandle:
        r"""Registers a forward hook on the module.
        The hook will be called every time after :meth:`edge_update` has
        computed an output.
        See :meth:`register_propagate_forward_hook` for more information.
        """
        handle = RemovableHandle(self._edge_update_forward_hooks)
        self._edge_update_forward_hooks[handle.id] = hook
        return handle

    @torch.jit.unused
    def jittable(self, typing: Optional[str] = None) -> 'MessagePassing':
        r"""Analyzes the :class:`MessagePassing` instance and produces a new
        jittable module.

        Args:
            typing (str, optional): If given, will generate a concrete instance
                with :meth:`forward` types based on :obj:`typing`, *e.g.*,
                :obj:`"(Tensor, Optional[Tensor]) -> Tensor"`.
        """
        try:
            from jinja2 import Template
        except ImportError:
            raise ModuleNotFoundError(
                "No module named 'jinja2' found on this machine. "
                "Run 'pip install jinja2' to install the library.")

        source = inspect.getsource(self.__class__)

        # Find and parse `propagate()` types to format `{arg1: type1, ...}`.
        if hasattr(self, 'propagate_type'):
            prop_types = {
                k: sanitize(str(v))
                for k, v in self.propagate_type.items()
            }
        else:
            match = re.search(r'#\s*propagate_type:\s*\((.*)\)', source)
            if match is None:
                raise TypeError(
                    'TorchScript support requires the definition of the types '
                    'passed to `propagate()`. Please specify them via\n\n'
                    'propagate_type = {"arg1": type1, "arg2": type2, ... }\n\n'
                    'or via\n\n'
                    '# propagate_type: (arg1: type1, arg2: type2, ...)\n\n'
                    'inside the `MessagePassing` module.')
            prop_types = split_types_repr(match.group(1))
            prop_types = dict([re.split(r'\s*:\s*', t) for t in prop_types])

        # Find and parse `edge_updater` types to format `{arg1: type1, ...}`.
        if 'edge_update' in self.__class__.__dict__.keys():
            if hasattr(self, 'edge_updater_type'):
                edge_updater_types = {
                    k: sanitize(str(v))
                    for k, v in self.edge_updater.items()
                }
            else:
                match = re.search(r'#\s*edge_updater_type:\s*\((.*)\)', source)
                if match is None:
                    raise TypeError(
                        'TorchScript support requires the definition of the '
                        'types passed to `edge_updater()`. Please specify '
                        'them via\n\n edge_updater_type = {"arg1": type1, '
                        '"arg2": type2, ... }\n\n or via\n\n'
                        '# edge_updater_type: (arg1: type1, arg2: type2, ...)'
                        '\n\ninside the `MessagePassing` module.')
                edge_updater_types = split_types_repr(match.group(1))
                edge_updater_types = dict(
                    [re.split(r'\s*:\s*', t) for t in edge_updater_types])
        else:
            edge_updater_types = {}

        type_hints = get_type_hints(self.__class__.update)
        prop_return_type = type_hints.get('return', 'Tensor')
        if str(prop_return_type)[:6] == ' str:
        if hasattr(self, 'in_channels') and hasattr(self, 'out_channels'):
            return (f'{self.__class__.__name__}({self.in_channels}, '
                    f'{self.out_channels})')
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/conv/mf_conv.py000066400000000000000000000103411444262304300213050ustar00rootroot00000000000000from typing import Tuple, Union

import torch
from torch import Tensor
from torch.nn import ModuleList

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptPairTensor, Size, SparseTensor
from torch_geometric.utils import degree, spmm


class MFConv(MessagePassing):
    r"""The graph neural network operator from the
    `"Convolutional Networks on Graphs for Learning Molecular Fingerprints"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{W}^{(\deg(i))}_1 \mathbf{x}_i +
        \mathbf{W}^{(\deg(i))}_2 \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j

    which trains a distinct weight matrix for each possible vertex degree.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        max_degree (int, optional): The maximum node degree to consider when
            updating weights (default: :obj:`10`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **inputs:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **outputs:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, max_degree: int = 10, bias=True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.max_degree = max_degree

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lins_l = ModuleList([
            Linear(in_channels[0], out_channels, bias=bias)
            for _ in range(max_degree + 1)
        ])

        self.lins_r = ModuleList([
            Linear(in_channels[1], out_channels, bias=False)
            for _ in range(max_degree + 1)
        ])

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        for lin in self.lins_l:
            lin.reset_parameters()
        for lin in self.lins_r:
            lin.reset_parameters()

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)
        x_r = x[1]

        deg = x[0]  # Dummy.
        if isinstance(edge_index, SparseTensor):
            deg = edge_index.storage.rowcount()
        elif isinstance(edge_index, Tensor):
            i = 1 if self.flow == 'source_to_target' else 0
            N = x[0].size(self.node_dim)
            N = size[1] if size is not None else N
            N = x_r.size(self.node_dim) if x_r is not None else N
            deg = degree(edge_index[i], N, dtype=torch.long)
        deg.clamp_(max=self.max_degree)

        # propagate_type: (x: OptPairTensor)
        h = self.propagate(edge_index, x=x, size=size)

        out = h.new_empty(list(h.size())[:-1] + [self.out_channels])
        for i, (lin_l, lin_r) in enumerate(zip(self.lins_l, self.lins_r)):
            idx = (deg == i).nonzero().view(-1)
            r = lin_l(h.index_select(self.node_dim, idx))

            if x_r is not None:
                r = r + lin_r(x_r.index_select(self.node_dim, idx))

            out.index_copy_(self.node_dim, idx, r)

        return out

    def message(self, x_j: Tensor) -> Tensor:
        return x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        adj_t = adj_t.set_value(None, layout=None)
        return spmm(adj_t, x[0], reduce=self.aggr)
pytorch/torch_geometric/nn/conv/nn_conv.py000066400000000000000000000112071444262304300213200ustar00rootroot00000000000000from typing import Callable, Tuple, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset, zeros
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, Size


class NNConv(MessagePassing):
    r"""The continuous kernel-based convolutional operator from the
    `"Neural Message Passing for Quantum Chemistry"
    `_ paper.
    This convolution is also known as the edge-conditioned convolution from the
    `"Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on
    Graphs" `_ paper (see
    :class:`torch_geometric.nn.conv.ECConv` for an alias):

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{\Theta} \mathbf{x}_i +
        \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j \cdot
        h_{\mathbf{\Theta}}(\mathbf{e}_{i,j}),

    where :math:`h_{\mathbf{\Theta}}` denotes a neural network, *.i.e.*
    a MLP.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        nn (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}` that
            maps edge features :obj:`edge_attr` of shape :obj:`[-1,
            num_edge_features]` to shape
            :obj:`[-1, in_channels * out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`.
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"add"`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add the transformed root node features to the output.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, nn: Callable, aggr: str = 'add',
                 root_weight: bool = True, bias: bool = True, **kwargs):
        super().__init__(aggr=aggr, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.nn = nn
        self.root_weight = root_weight

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.in_channels_l = in_channels[0]

        if root_weight:
            self.lin = Linear(in_channels[1], out_channels, bias=False,
                              weight_initializer='uniform')

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.nn)
        if self.root_weight:
            self.lin.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)

        x_r = x[1]
        if x_r is not None and self.root_weight:
            out = out + self.lin(x_r)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_attr: Tensor) -> Tensor:
        weight = self.nn(edge_attr)
        weight = weight.view(-1, self.in_channels_l, self.out_channels)
        return torch.matmul(x_j.unsqueeze(1), weight).squeeze(1)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, aggr={self.aggr}, nn={self.nn})')
pytorch/torch_geometric/nn/conv/pan_conv.py000066400000000000000000000115001444262304300214570ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, SparseTensor
from torch_geometric.utils import is_torch_sparse_tensor, spmm


class PANConv(MessagePassing):
    r"""The path integral based convolutional operator from the
    `"Path Integral Based Convolution and Pooling for Graph Neural Networks"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} = \mathbf{M} \mathbf{X} \mathbf{W}

    where :math:`\mathbf{M}` denotes the normalized and learned maximal entropy
    transition (MET) matrix that includes neighbors up to :obj:`filter_size`
    hops:

    .. math::

        \mathbf{M} = \mathbf{Z}^{-1/2} \sum_{n=0}^L e^{-\frac{E(n)}{T}}
        \mathbf{A}^n \mathbf{Z}^{-1/2}

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        filter_size (int): The filter size :math:`L`.
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int, filter_size: int,
                 **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.filter_size = filter_size

        self.lin = Linear(in_channels, out_channels)
        self.weight = Parameter(torch.Tensor(filter_size + 1))

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        self.weight.data.fill_(0.5)

    def forward(
        self,
        x: Tensor,
        edge_index: Adj,
    ) -> Tuple[Tensor, SparseTensor]:

        adj_t: Optional[SparseTensor] = None
        if isinstance(edge_index, Tensor):
            if is_torch_sparse_tensor(edge_index):
                # TODO Handle PyTorch sparse tensor directly.
                if edge_index.layout == torch.sparse_coo:
                    adj_t = SparseTensor.from_torch_sparse_coo_tensor(
                        edge_index)
                elif edge_index.layout == torch.sparse_csr:
                    adj_t = SparseTensor.from_torch_sparse_csr_tensor(
                        edge_index)
                else:
                    raise ValueError(f"Unexpected sparse tensor layout "
                                     f"(got '{edge_index.layout}')")
            else:
                adj_t = SparseTensor(row=edge_index[1], col=edge_index[0],
                                     sparse_sizes=(x.size(0), x.size(0)))

        elif isinstance(edge_index, SparseTensor):
            adj_t = edge_index.set_value(None)

        adj_t = self.panentropy(adj_t, dtype=x.dtype)

        deg = adj_t.storage.rowcount().to(x.dtype)
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0.
        M = deg_inv_sqrt.view(1, -1) * adj_t * deg_inv_sqrt.view(-1, 1)

        out = self.propagate(M, x=x, edge_weight=None, size=None)
        out = self.lin(out)
        return out, M

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def panentropy(self, adj_t: SparseTensor,
                   dtype: Optional[int] = None) -> SparseTensor:

        if not adj_t.has_value():
            adj_t = adj_t.fill_value(1.0)

        tmp = SparseTensor.eye(adj_t.size(0), adj_t.size(1), has_value=True,
                               dtype=dtype, device=adj_t.device())
        tmp = tmp.mul_nnz(self.weight[0], layout='coo')

        outs = [tmp]
        for i in range(1, self.filter_size + 1):
            tmp = tmp @ adj_t
            tmp = tmp.mul_nnz(self.weight[i], layout='coo')
            outs += [tmp]

        row = torch.cat([out.storage.row() for out in outs], dim=0)
        col = torch.cat([out.storage.col() for out in outs], dim=0)
        value = torch.cat([out.storage.value() for out in outs], dim=0)

        out = SparseTensor(row=row, col=col, value=value,
                           sparse_sizes=adj_t.sparse_sizes()).coalesce()

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, filter_size={self.filter_size})')
pytorch/torch_geometric/nn/conv/pdn_conv.py000066400000000000000000000114641444262304300214730ustar00rootroot00000000000000import torch
from torch import Tensor
from torch.nn import Linear, Parameter, ReLU, Sequential, Sigmoid

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class PDNConv(MessagePassing):
    r"""The pathfinder discovery network convolutional operator from the
    `"Pathfinder Discovery Networks for Neural Message Passing"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \sum_{j \in \mathcal{N}(i) \cup
        \{i\}}f_{\Theta}(\textbf{e}_{(j,i)}) \cdot f_{\Omega}(\mathbf{x}_{j})

    where :math:`z_{i,j}` denotes the edge feature vector from source node
    :math:`j` to target node :math:`i`, and :math:`\mathbf{x}_{j}` denotes the
    node feature vector of node :math:`j`.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        edge_dim (int): Edge feature dimensionality.
        hidden_channels (int): Hidden edge feature dimensionality.
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        normalize (bool, optional): Whether to add self-loops and compute
            symmetric normalization coefficients on the fly.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int, edge_dim: int,
                 hidden_channels: int, add_self_loops: bool = True,
                 normalize: bool = True, bias: bool = True, **kwargs):

        kwargs.setdefault("aggr", "add")
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.edge_dim = edge_dim
        self.hidden_channels = hidden_channels
        self.add_self_loops = add_self_loops
        self.normalize = normalize

        self.lin = Linear(in_channels, out_channels, bias=False)

        self.mlp = Sequential(
            Linear(edge_dim, hidden_channels),
            ReLU(inplace=True),
            Linear(hidden_channels, 1),
            Sigmoid(),
        )

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter("bias", None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        glorot(self.lin.weight)
        glorot(self.mlp[0].weight)
        glorot(self.mlp[2].weight)
        zeros(self.mlp[0].bias)
        zeros(self.mlp[2].bias)
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj,
                edge_attr: OptTensor = None) -> Tensor:

        if isinstance(edge_index, SparseTensor):
            edge_attr = edge_index.storage.value()

        if edge_attr is not None:
            edge_attr = self.mlp(edge_attr).squeeze(-1)

        if isinstance(edge_index, SparseTensor):
            edge_index = edge_index.set_value(edge_attr, layout='coo')

        if self.normalize:
            if isinstance(edge_index, Tensor):
                edge_index, edge_attr = gcn_norm(edge_index, edge_attr,
                                                 x.size(self.node_dim), False,
                                                 self.add_self_loops,
                                                 self.flow, x.dtype)
            elif isinstance(edge_index, SparseTensor):
                edge_index = gcn_norm(edge_index, None, x.size(self.node_dim),
                                      False, self.add_self_loops, self.flow,
                                      x.dtype)

        x = self.lin(x)

        # propagate_type: (x: Tensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_attr, size=None)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self):
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/conv/pna_conv.py000066400000000000000000000200621444262304300214620ustar00rootroot00000000000000from typing import Any, Callable, Dict, List, Optional, Union

import torch
from torch import Tensor
from torch.nn import ModuleList, Sequential
from torch.utils.data import DataLoader

from torch_geometric.nn.aggr import DegreeScalerAggregation
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.nn.resolver import activation_resolver
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils import degree


class PNAConv(MessagePassing):
    r"""The Principal Neighbourhood Aggregation graph convolution operator
    from the `"Principal Neighbourhood Aggregation for Graph Nets"
    `_ paper

    .. math::
        \mathbf{x}_i^{\prime} = \gamma_{\mathbf{\Theta}} \left(
        \mathbf{x}_i, \underset{j \in \mathcal{N}(i)}{\bigoplus}
        h_{\mathbf{\Theta}} \left( \mathbf{x}_i, \mathbf{x}_j \right)
        \right)

    with

    .. math::
        \bigoplus = \underbrace{\begin{bmatrix}
            1 \\
            S(\mathbf{D}, \alpha=1) \\
            S(\mathbf{D}, \alpha=-1)
        \end{bmatrix} }_{\text{scalers}}
        \otimes \underbrace{\begin{bmatrix}
            \mu \\
            \sigma \\
            \max \\
            \min
        \end{bmatrix}}_{\text{aggregators}},

    where :math:`\gamma_{\mathbf{\Theta}}` and :math:`h_{\mathbf{\Theta}}`
    denote MLPs.

    .. note::

        For an example of using :obj:`PNAConv`, see `examples/pna.py
        `_.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        aggregators (List[str]): Set of aggregation function identifiers,
            namely :obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"var"` and :obj:`"std"`.
        scalers (List[str]): Set of scaling function identifiers, namely
            :obj:`"identity"`, :obj:`"amplification"`,
            :obj:`"attenuation"`, :obj:`"linear"` and
            :obj:`"inverse_linear"`.
        deg (torch.Tensor): Histogram of in-degrees of nodes in the training
            set, used by scalers to normalize.
        edge_dim (int, optional): Edge feature dimensionality (in case
            there are any). (default :obj:`None`)
        towers (int, optional): Number of towers (default: :obj:`1`).
        pre_layers (int, optional): Number of transformation layers before
            aggregation (default: :obj:`1`).
        post_layers (int, optional): Number of transformation layers after
            aggregation (default: :obj:`1`).
        divide_input (bool, optional): Whether the input features should
            be split between towers or not (default: :obj:`False`).
        act (str or callable, optional): Pre- and post-layer activation
            function to use. (default: :obj:`"relu"`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        train_norm (bool, optional): Whether normalization parameters
            are trainable. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge features :math:`(|\mathcal{E}|, D)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        aggregators: List[str],
        scalers: List[str],
        deg: Tensor,
        edge_dim: Optional[int] = None,
        towers: int = 1,
        pre_layers: int = 1,
        post_layers: int = 1,
        divide_input: bool = False,
        act: Union[str, Callable, None] = "relu",
        act_kwargs: Optional[Dict[str, Any]] = None,
        train_norm: bool = False,
        **kwargs,
    ):

        aggr = DegreeScalerAggregation(aggregators, scalers, deg, train_norm)
        super().__init__(aggr=aggr, node_dim=0, **kwargs)

        if divide_input:
            assert in_channels % towers == 0
        assert out_channels % towers == 0

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.edge_dim = edge_dim
        self.towers = towers
        self.divide_input = divide_input

        self.F_in = in_channels // towers if divide_input else in_channels
        self.F_out = self.out_channels // towers

        if self.edge_dim is not None:
            self.edge_encoder = Linear(edge_dim, self.F_in)

        self.pre_nns = ModuleList()
        self.post_nns = ModuleList()
        for _ in range(towers):
            modules = [Linear((3 if edge_dim else 2) * self.F_in, self.F_in)]
            for _ in range(pre_layers - 1):
                modules += [activation_resolver(act, **(act_kwargs or {}))]
                modules += [Linear(self.F_in, self.F_in)]
            self.pre_nns.append(Sequential(*modules))

            in_channels = (len(aggregators) * len(scalers) + 1) * self.F_in
            modules = [Linear(in_channels, self.F_out)]
            for _ in range(post_layers - 1):
                modules += [activation_resolver(act, **(act_kwargs or {}))]
                modules += [Linear(self.F_out, self.F_out)]
            self.post_nns.append(Sequential(*modules))

        self.lin = Linear(out_channels, out_channels)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if self.edge_dim is not None:
            self.edge_encoder.reset_parameters()
        for nn in self.pre_nns:
            reset(nn)
        for nn in self.post_nns:
            reset(nn)
        self.lin.reset_parameters()

    def forward(self, x: Tensor, edge_index: Adj,
                edge_attr: OptTensor = None) -> Tensor:

        if self.divide_input:
            x = x.view(-1, self.towers, self.F_in)
        else:
            x = x.view(-1, 1, self.F_in).repeat(1, self.towers, 1)

        # propagate_type: (x: Tensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=None)

        out = torch.cat([x, out], dim=-1)
        outs = [nn(out[:, i]) for i, nn in enumerate(self.post_nns)]
        out = torch.cat(outs, dim=1)

        return self.lin(out)

    def message(self, x_i: Tensor, x_j: Tensor,
                edge_attr: OptTensor) -> Tensor:

        h: Tensor = x_i  # Dummy.
        if edge_attr is not None:
            edge_attr = self.edge_encoder(edge_attr)
            edge_attr = edge_attr.view(-1, 1, self.F_in)
            edge_attr = edge_attr.repeat(1, self.towers, 1)
            h = torch.cat([x_i, x_j, edge_attr], dim=-1)
        else:
            h = torch.cat([x_i, x_j], dim=-1)

        hs = [nn(h[:, i]) for i, nn in enumerate(self.pre_nns)]
        return torch.stack(hs, dim=1)

    def __repr__(self):
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, towers={self.towers}, '
                f'edge_dim={self.edge_dim})')

    @staticmethod
    def get_degree_histogram(loader: DataLoader) -> Tensor:
        r"""Returns the degree histogram to be used as input for the :obj:`deg`
        argument in :class:`PNAConv`."""
        deg_histogram = torch.zeros(1, dtype=torch.long)
        for data in loader:
            d = degree(data.edge_index[1], num_nodes=data.num_nodes,
                       dtype=torch.long)
            d_bincount = torch.bincount(d, minlength=deg_histogram.numel())
            if d_bincount.size(0) > deg_histogram.size(0):
                d_bincount[:deg_histogram.size(0)] += deg_histogram
                deg_histogram = d_bincount
            else:
                assert d_bincount.size(0) == deg_histogram.size(0)
                deg_histogram += d_bincount

        return deg_histogram
pytorch/torch_geometric/nn/conv/point_conv.py000066400000000000000000000106521444262304300220410ustar00rootroot00000000000000from typing import Callable, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.typing import (
    Adj,
    OptTensor,
    PairOptTensor,
    PairTensor,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import add_self_loops, remove_self_loops


class PointNetConv(MessagePassing):
    r"""The PointNet set layer from the `"PointNet: Deep Learning on Point Sets
    for 3D Classification and Segmentation"
    `_ and `"PointNet++: Deep Hierarchical
    Feature Learning on Point Sets in a Metric Space"
    `_ papers

    .. math::
        \mathbf{x}^{\prime}_i = \gamma_{\mathbf{\Theta}} \left( \max_{j \in
        \mathcal{N}(i) \cup \{ i \}} h_{\mathbf{\Theta}} ( \mathbf{x}_j,
        \mathbf{p}_j - \mathbf{p}_i) \right),

    where :math:`\gamma_{\mathbf{\Theta}}` and :math:`h_{\mathbf{\Theta}}`
    denote neural networks, *i.e.* MLPs, and
    :math:`\mathbf{P} \in \mathbb{R}^{N \times D}` defines the position of
    each point.

    Args:
        local_nn (torch.nn.Module, optional): A neural network
            :math:`h_{\mathbf{\Theta}}` that maps node features :obj:`x` and
            relative spatial coordinates :obj:`pos_j - pos_i` of shape
            :obj:`[-1, in_channels + num_dimensions]` to shape
            :obj:`[-1, out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`. (default: :obj:`None`)
        global_nn (torch.nn.Module, optional): A neural network
            :math:`\gamma_{\mathbf{\Theta}}` that maps aggregated node features
            of shape :obj:`[-1, out_channels]` to shape :obj:`[-1,
            final_out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`. (default: :obj:`None`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          positions :math:`(|\mathcal{V}|, 3)` or
          :math:`((|\mathcal{V_s}|, 3), (|\mathcal{V_t}|, 3))` if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, local_nn: Optional[Callable] = None,
                 global_nn: Optional[Callable] = None,
                 add_self_loops: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'max')
        super().__init__(**kwargs)

        self.local_nn = local_nn
        self.global_nn = global_nn
        self.add_self_loops = add_self_loops

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.local_nn)
        reset(self.global_nn)

    def forward(self, x: Union[OptTensor, PairOptTensor],
                pos: Union[Tensor, PairTensor], edge_index: Adj) -> Tensor:

        if not isinstance(x, tuple):
            x: PairOptTensor = (x, None)

        if isinstance(pos, Tensor):
            pos: PairTensor = (pos, pos)

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(
                    edge_index, num_nodes=min(pos[0].size(0), pos[1].size(0)))
            elif isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.set_diag(edge_index)

        # propagate_type: (x: PairOptTensor, pos: PairTensor)
        out = self.propagate(edge_index, x=x, pos=pos, size=None)

        if self.global_nn is not None:
            out = self.global_nn(out)

        return out

    def message(self, x_j: Optional[Tensor], pos_i: Tensor,
                pos_j: Tensor) -> Tensor:
        msg = pos_j - pos_i
        if x_j is not None:
            msg = torch.cat([x_j, msg], dim=1)
        if self.local_nn is not None:
            msg = self.local_nn(msg)
        return msg

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(local_nn={self.local_nn}, '
                f'global_nn={self.global_nn})')
pytorch/torch_geometric/nn/conv/point_gnn_conv.py000066400000000000000000000063301444262304300227010ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.typing import Adj


class PointGNNConv(MessagePassing):
    r"""The PointGNN operator from the `"Point-GNN: Graph Neural Network for
    3D Object Detection in a Point Cloud" `_
    paper

    .. math::

        \Delta \textrm{pos}_i &= h_{\mathbf{\Theta}}(\mathbf{x}_i)

        \mathbf{e}_{j,i} &= f_{\mathbf{\Theta}}(\textrm{pos}_j -
        \textrm{pos}_i + \Delta \textrm{pos}_i, \mathbf{x}_j)

        \mathbf{x}^{\prime}_i &= g_{\mathbf{\Theta}}(\max_{j \in
        \mathcal{N}(i)} \mathbf{e}_{j,i}) + \mathbf{x}_i

    The relative position is used in the message passing step to introduce
    global translation invariance.
    To also counter shifts in the local neighborhood of the center node, the
    authors propose to utilize an alignment offset.
    The graph should be statically constructed using radius-based cutoff.

    Args:
        mlp_h (torch.nn.Module): A neural network :math:`h_{\mathbf{\Theta}}`
            that maps node features of size :math:`F_{in}` to three-dimensional
            coordination offsets :math:`\Delta \textrm{pos}_i`.
        mlp_f (torch.nn.Module): A neural network :math:`f_{\mathbf{\Theta}}`
            that computes :math:`\mathbf{e}_{j,i}` from the features of
            neighbors of size :math:`F_{in}` and the three-dimensional vector
            :math:`\textrm{pos_j} - \textrm{pos_i} + \Delta \textrm{pos}_i`.
        mlp_g (torch.nn.Module): A neural network :math:`g_{\mathbf{\Theta}}`
            that maps the aggregated edge features back to :math:`F_{in}`.
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          positions :math:`(|\mathcal{V}|, 3)`,
          edge indices :math:`(2, |\mathcal{E}|)`,
        - **output:** node features :math:`(|\mathcal{V}|, F_{in})`
    """
    def __init__(
        self,
        mlp_h: torch.nn.Module,
        mlp_f: torch.nn.Module,
        mlp_g: torch.nn.Module,
        **kwargs,
    ):
        kwargs.setdefault('aggr', 'max')
        super().__init__(**kwargs)

        self.mlp_h = mlp_h
        self.mlp_f = mlp_f
        self.mlp_g = mlp_g

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.mlp_h)
        reset(self.mlp_f)
        reset(self.mlp_g)

    def forward(self, x: Tensor, pos: Tensor, edge_index: Adj) -> Tensor:
        # propagate_type: (x: Tensor, pos: Tensor)
        out = self.propagate(edge_index, x=x, pos=pos, size=None)
        out = self.mlp_g(out)
        return x + out

    def message(self, pos_j: Tensor, pos_i: Tensor, x_i: Tensor,
                x_j: Tensor) -> Tensor:
        delta = self.mlp_h(x_i)
        e = torch.cat([pos_j - pos_i + delta, x_j], dim=-1)
        return self.mlp_f(e)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(\n'
                f'  mlp_h={self.mlp_h},\n'
                f'  mlp_f={self.mlp_f},\n'
                f'  mlp_g={self.mlp_g},\n'
                f')')
pytorch/torch_geometric/nn/conv/point_transformer_conv.py000066400000000000000000000134011444262304300244560ustar00rootroot00000000000000from typing import Callable, Optional, Tuple, Union

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import reset
from torch_geometric.typing import (
    Adj,
    OptTensor,
    PairTensor,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import add_self_loops, remove_self_loops, softmax


class PointTransformerConv(MessagePassing):
    r"""The Point Transformer layer from the `"Point Transformer"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i =  \sum_{j \in
        \mathcal{N}(i) \cup \{ i \}} \alpha_{i,j} \left(\mathbf{W}_3
        \mathbf{x}_j + \delta_{ij} \right),

    where the attention coefficients :math:`\alpha_{i,j}` and
    positional embedding :math:`\delta_{ij}` are computed as

    .. math::
        \alpha_{i,j}= \textrm{softmax} \left( \gamma_\mathbf{\Theta}
        (\mathbf{W}_1 \mathbf{x}_i - \mathbf{W}_2 \mathbf{x}_j +
        \delta_{i,j}) \right)

    and

    .. math::
        \delta_{i,j}= h_{\mathbf{\Theta}}(\mathbf{p}_i - \mathbf{p}_j),

    with :math:`\gamma_\mathbf{\Theta}` and :math:`h_\mathbf{\Theta}`
    denoting neural networks, *i.e.* MLPs, and
    :math:`\mathbf{P} \in \mathbb{R}^{N \times D}` defines the position of
    each point.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        pos_nn (torch.nn.Module, optional): A neural network
            :math:`h_\mathbf{\Theta}` which maps relative spatial coordinates
            :obj:`pos_j - pos_i` of shape :obj:`[-1, 3]` to shape
            :obj:`[-1, out_channels]`.
            Will default to a :class:`torch.nn.Linear` transformation if not
            further specified. (default: :obj:`None`)
        attn_nn (torch.nn.Module, optional): A neural network
            :math:`\gamma_\mathbf{\Theta}` which maps transformed
            node features of shape :obj:`[-1, out_channels]`
            to shape :obj:`[-1, out_channels]`. (default: :obj:`None`)
        add_self_loops (bool, optional) : If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          positions :math:`(|\mathcal{V}|, 3)` or
          :math:`((|\mathcal{V_s}|, 3), (|\mathcal{V_t}|, 3))` if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: Union[int, Tuple[int, int]],
                 out_channels: int, pos_nn: Optional[Callable] = None,
                 attn_nn: Optional[Callable] = None,
                 add_self_loops: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.add_self_loops = add_self_loops

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.pos_nn = pos_nn
        if self.pos_nn is None:
            self.pos_nn = Linear(3, out_channels)

        self.attn_nn = attn_nn
        self.lin = Linear(in_channels[0], out_channels, bias=False)
        self.lin_src = Linear(in_channels[0], out_channels, bias=False)
        self.lin_dst = Linear(in_channels[1], out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.pos_nn)
        if self.attn_nn is not None:
            reset(self.attn_nn)
        self.lin.reset_parameters()
        self.lin_src.reset_parameters()
        self.lin_dst.reset_parameters()

    def forward(self, x: Union[Tensor, PairTensor],
                pos: Union[Tensor, PairTensor], edge_index: Adj) -> Tensor:

        if isinstance(x, Tensor):
            alpha = (self.lin_src(x), self.lin_dst(x))
            x: PairTensor = (self.lin(x), x)
        else:
            alpha = (self.lin_src(x[0]), self.lin_dst(x[1]))
            x = (self.lin(x[0]), x[1])

        if isinstance(pos, Tensor):
            pos: PairTensor = (pos, pos)

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(
                    edge_index, num_nodes=min(pos[0].size(0), pos[1].size(0)))
            elif isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.set_diag(edge_index)

        # propagate_type: (x: PairTensor, pos: PairTensor, alpha: PairTensor)
        out = self.propagate(edge_index, x=x, pos=pos, alpha=alpha, size=None)
        return out

    def message(self, x_j: Tensor, pos_i: Tensor, pos_j: Tensor,
                alpha_i: Tensor, alpha_j: Tensor, index: Tensor,
                ptr: OptTensor, size_i: Optional[int]) -> Tensor:

        delta = self.pos_nn(pos_i - pos_j)
        alpha = alpha_i - alpha_j + delta
        if self.attn_nn is not None:
            alpha = self.attn_nn(alpha)
        alpha = softmax(alpha, index, ptr, size_i)
        return alpha * (x_j + delta)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/conv/ppf_conv.py000066400000000000000000000124561444262304300215010ustar00rootroot00000000000000from typing import Callable, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.typing import (
    Adj,
    OptTensor,
    PairOptTensor,
    PairTensor,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import add_self_loops, remove_self_loops


def get_angle(v1: Tensor, v2: Tensor) -> Tensor:
    return torch.atan2(
        torch.cross(v1, v2, dim=1).norm(p=2, dim=1), (v1 * v2).sum(dim=1))


def point_pair_features(pos_i: Tensor, pos_j: Tensor, normal_i: Tensor,
                        normal_j: Tensor) -> Tensor:
    pseudo = pos_j - pos_i
    return torch.stack([
        pseudo.norm(p=2, dim=1),
        get_angle(normal_i, pseudo),
        get_angle(normal_j, pseudo),
        get_angle(normal_i, normal_j)
    ], dim=1)


class PPFConv(MessagePassing):
    r"""The PPFNet operator from the `"PPFNet: Global Context Aware Local
    Features for Robust 3D Point Matching" `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = \gamma_{\mathbf{\Theta}} \left( \max_{j \in
        \mathcal{N}(i) \cup \{ i \}} h_{\mathbf{\Theta}} ( \mathbf{x}_j, \|
        \mathbf{d_{j,i}} \|, \angle(\mathbf{n}_i, \mathbf{d_{j,i}}),
        \angle(\mathbf{n}_j, \mathbf{d_{j,i}}), \angle(\mathbf{n}_i,
        \mathbf{n}_j) \right)

    where :math:`\gamma_{\mathbf{\Theta}}` and :math:`h_{\mathbf{\Theta}}`
    denote neural networks, *.i.e.* MLPs, which takes in node features and
    :class:`torch_geometric.transforms.PointPairFeatures`.

    Args:
        local_nn (torch.nn.Module, optional): A neural network
            :math:`h_{\mathbf{\Theta}}` that maps node features :obj:`x` and
            relative spatial coordinates :obj:`pos_j - pos_i` of shape
            :obj:`[-1, in_channels + num_dimensions]` to shape
            :obj:`[-1, out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`. (default: :obj:`None`)
        global_nn (torch.nn.Module, optional): A neural network
            :math:`\gamma_{\mathbf{\Theta}}` that maps aggregated node features
            of shape :obj:`[-1, out_channels]` to shape :obj:`[-1,
            final_out_channels]`, *e.g.*, defined by
            :class:`torch.nn.Sequential`. (default: :obj:`None`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          positions :math:`(|\mathcal{V}|, 3)` or
          :math:`((|\mathcal{V_s}|, 3), (|\mathcal{V_t}|, 3))` if bipartite,
          point normals :math:`(|\mathcal{V}, 3)` or
          :math:`((|\mathcal{V_s}|, 3), (|\mathcal{V_t}|, 3))` if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V}_t|, F_{out})` if bipartite

    """
    def __init__(self, local_nn: Optional[Callable] = None,
                 global_nn: Optional[Callable] = None,
                 add_self_loops: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'max')
        super().__init__(**kwargs)

        self.local_nn = local_nn
        self.global_nn = global_nn
        self.add_self_loops = add_self_loops

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.local_nn)
        reset(self.global_nn)

    def forward(
        self,
        x: Union[OptTensor, PairOptTensor],
        pos: Union[Tensor, PairTensor],
        normal: Union[Tensor, PairTensor],
        edge_index: Adj,
    ) -> Tensor:

        if not isinstance(x, tuple):
            x: PairOptTensor = (x, None)

        if isinstance(pos, Tensor):
            pos: PairTensor = (pos, pos)

        if isinstance(normal, Tensor):
            normal: PairTensor = (normal, normal)

        if self.add_self_loops:
            if isinstance(edge_index, Tensor):
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(edge_index,
                                               num_nodes=pos[1].size(0))
            elif isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.set_diag(edge_index)

        # propagate_type: (x: PairOptTensor, pos: PairTensor, normal: PairTensor)  # noqa
        out = self.propagate(edge_index, x=x, pos=pos, normal=normal,
                             size=None)

        if self.global_nn is not None:
            out = self.global_nn(out)

        return out

    def message(self, x_j: OptTensor, pos_i: Tensor, pos_j: Tensor,
                normal_i: Tensor, normal_j: Tensor) -> Tensor:
        msg = point_pair_features(pos_i, pos_j, normal_i, normal_j)
        if x_j is not None:
            msg = torch.cat([x_j, msg], dim=1)
        if self.local_nn is not None:
            msg = self.local_nn(msg)
        return msg

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(local_nn={self.local_nn}, '
                f'global_nn={self.global_nn})')
pytorch/torch_geometric/nn/conv/res_gated_graph_conv.py000066400000000000000000000101241444262304300240200ustar00rootroot00000000000000from typing import Callable, Optional, Tuple, Union

from torch import Tensor
from torch.nn import Parameter, Sigmoid

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import Adj, PairTensor


class ResGatedGraphConv(MessagePassing):
    r"""The residual gated graph convolutional operator from the
    `"Residual Gated Graph ConvNets" `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i +
        \sum_{j \in \mathcal{N}(i)} \eta_{i,j} \odot \mathbf{W}_2 \mathbf{x}_j

    where the gate :math:`\eta_{i,j}` is defined as

    .. math::
        \eta_{i,j} = \sigma(\mathbf{W}_3 \mathbf{x}_i + \mathbf{W}_4
        \mathbf{x}_j)

    with :math:`\sigma` denoting the sigmoid function.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        act (callable, optional): Gating function :math:`\sigma`.
            (default: :meth:`torch.nn.Sigmoid()`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **inputs:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **outputs:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        act: Optional[Callable] = Sigmoid(),
        root_weight: bool = True,
        bias: bool = True,
        **kwargs,
    ):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.act = act
        self.root_weight = root_weight

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lin_key = Linear(in_channels[1], out_channels)
        self.lin_query = Linear(in_channels[0], out_channels)
        self.lin_value = Linear(in_channels[0], out_channels)

        if root_weight:
            self.lin_skip = Linear(in_channels[1], out_channels, bias=False)
        else:
            self.register_parameter('lin_skip', None)

        if bias:
            self.bias = Parameter(Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_key.reset_parameters()
        self.lin_query.reset_parameters()
        self.lin_value.reset_parameters()
        if self.lin_skip is not None:
            self.lin_skip.reset_parameters()
        if self.bias is not None:
            zeros(self.bias)

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj) -> Tensor:
        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        k = self.lin_key(x[1])
        q = self.lin_query(x[0])
        v = self.lin_value(x[0])

        # propagate_type: (k: Tensor, q: Tensor, v: Tensor)
        out = self.propagate(edge_index, k=k, q=q, v=v, size=None)

        if self.root_weight:
            out = out + self.lin_skip(x[1])

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, k_i: Tensor, q_j: Tensor, v_j: Tensor) -> Tensor:
        return self.act(k_i + q_j) * v_j
pytorch/torch_geometric/nn/conv/rgat_conv.py000066400000000000000000000544011444262304300216450ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter, ReLU

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, ones, zeros
from torch_geometric.typing import Adj, OptTensor, Size, SparseTensor
from torch_geometric.utils import is_torch_sparse_tensor, scatter, softmax
from torch_geometric.utils.sparse import set_sparse_value


class RGATConv(MessagePassing):
    r"""The relational graph attentional operator from the `"Relational Graph
    Attention Networks" `_ paper.
    Here, attention logits :math:`\mathbf{a}^{(r)}_{i,j}` are computed for each
    relation type :math:`r` with the help of both query and key kernels, *i.e.*

    .. math::
        \mathbf{q}^{(r)}_i = \mathbf{W}_1^{(r)}\mathbf{x}_{i} \cdot
        \mathbf{Q}^{(r)}
        \quad \textrm{and} \quad
        \mathbf{k}^{(r)}_i = \mathbf{W}_1^{(r)}\mathbf{x}_{i} \cdot
        \mathbf{K}^{(r)}.

    Two schemes have been proposed to compute attention logits
    :math:`\mathbf{a}^{(r)}_{i,j}` for each relation type :math:`r`:

    **Additive attention**

    .. math::
        \mathbf{a}^{(r)}_{i,j} = \mathrm{LeakyReLU}(\mathbf{q}^{(r)}_i +
        \mathbf{k}^{(r)}_j)

    or **multiplicative attention**

    .. math::
        \mathbf{a}^{(r)}_{i,j} = \mathbf{q}^{(r)}_i \cdot \mathbf{k}^{(r)}_j.

    If the graph has multi-dimensional edge features
    :math:`\mathbf{e}^{(r)}_{i,j}`, the attention logits
    :math:`\mathbf{a}^{(r)}_{i,j}` for each relation type :math:`r` are
    computed as

    .. math::
        \mathbf{a}^{(r)}_{i,j} = \mathrm{LeakyReLU}(\mathbf{q}^{(r)}_i +
        \mathbf{k}^{(r)}_j + \mathbf{W}_2^{(r)}\mathbf{e}^{(r)}_{i,j})

    or

    .. math::
        \mathbf{a}^{(r)}_{i,j} = \mathbf{q}^{(r)}_i \cdot \mathbf{k}^{(r)}_j
        \cdot \mathbf{W}_2^{(r)} \mathbf{e}^{(r)}_{i,j},

    respectively.
    The attention coefficients :math:`\alpha^{(r)}_{i,j}` for each relation
    type :math:`r` are then obtained via two different attention mechanisms:
    The **within-relation** attention mechanism

    .. math::
        \alpha^{(r)}_{i,j} =
        \frac{\exp(\mathbf{a}^{(r)}_{i,j})}
        {\sum_{k \in \mathcal{N}_r(i)} \exp(\mathbf{a}^{(r)}_{i,k})}

    or the **across-relation** attention mechanism

    .. math::
        \alpha^{(r)}_{i,j} =
        \frac{\exp(\mathbf{a}^{(r)}_{i,j})}
        {\sum_{r^{\prime} \in \mathcal{R}}
        \sum_{k \in \mathcal{N}_{r^{\prime}}(i)}
        \exp(\mathbf{a}^{(r^{\prime})}_{i,k})}

    where :math:`\mathcal{R}` denotes the set of relations, *i.e.* edge types.
    Edge type needs to be a one-dimensional :obj:`torch.long` tensor which
    stores a relation identifier :math:`\in \{ 0, \ldots, |\mathcal{R}| - 1\}`
    for each edge.

    To enhance the discriminative power of attention-based GNNs, this layer
    further implements four different cardinality preservation options as
    proposed in the `"Improving Attention Mechanism in Graph Neural Networks
    via Cardinality Preservation" `_ paper:

    .. math::
        \text{additive:}~~~\mathbf{x}^{{\prime}(r)}_i &=
        \sum_{j \in \mathcal{N}_r(i)}
        \alpha^{(r)}_{i,j} \mathbf{x}^{(r)}_j + \mathcal{W} \odot
        \sum_{j \in \mathcal{N}_r(i)} \mathbf{x}^{(r)}_j

        \text{scaled:}~~~\mathbf{x}^{{\prime}(r)}_i &=
        \psi(|\mathcal{N}_r(i)|) \odot
        \sum_{j \in \mathcal{N}_r(i)} \alpha^{(r)}_{i,j} \mathbf{x}^{(r)}_j

        \text{f-additive:}~~~\mathbf{x}^{{\prime}(r)}_i &=
        \sum_{j \in \mathcal{N}_r(i)}
        (\alpha^{(r)}_{i,j} + 1) \cdot \mathbf{x}^{(r)}_j

        \text{f-scaled:}~~~\mathbf{x}^{{\prime}(r)}_i &=
        |\mathcal{N}_r(i)| \odot \sum_{j \in \mathcal{N}_r(i)}
        \alpha^{(r)}_{i,j} \mathbf{x}^{(r)}_j

    * If :obj:`attention_mode="additive-self-attention"` and
      :obj:`concat=True`, the layer outputs :obj:`heads * out_channels`
      features for each node.

    * If :obj:`attention_mode="multiplicative-self-attention"` and
      :obj:`concat=True`, the layer outputs :obj:`heads * dim * out_channels`
      features for each node.

    * If :obj:`attention_mode="additive-self-attention"` and
      :obj:`concat=False`, the layer outputs :obj:`out_channels` features for
      each node.

    * If :obj:`attention_mode="multiplicative-self-attention"` and
      :obj:`concat=False`, the layer outputs :obj:`dim * out_channels` features
      for each node.

    Please make sure to set the :obj:`in_channels` argument of the next
    layer accordingly if more than one instance of this layer is used.

    .. note::

        For an example of using :class:`RGATConv`, see
        `examples/rgat.py `_.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        num_relations (int): Number of relations.
        num_bases (int, optional): If set, this layer will use the
            basis-decomposition regularization scheme where :obj:`num_bases`
            denotes the number of bases to use. (default: :obj:`None`)
        num_blocks (int, optional): If set, this layer will use the
            block-diagonal-decomposition regularization scheme where
            :obj:`num_blocks` denotes the number of blocks to use.
            (default: :obj:`None`)
        mod (str, optional): The cardinality preservation option to use.
            (:obj:`"additive"`, :obj:`"scaled"`, :obj:`"f-additive"`,
            :obj:`"f-scaled"`, :obj:`None`). (default: :obj:`None`)
        attention_mechanism (str, optional): The attention mechanism to use
            (:obj:`"within-relation"`, :obj:`"across-relation"`).
            (default: :obj:`"across-relation"`)
        attention_mode (str, optional): The mode to calculate attention logits.
            (:obj:`"additive-self-attention"`,
            :obj:`"multiplicative-self-attention"`).
            (default: :obj:`"additive-self-attention"`)
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        dim (int): Number of dimensions for query and key kernels.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        edge_dim (int, optional): Edge feature dimensionality (in case there
            are any). (default: :obj:`None`)
        bias (bool, optional): If set to :obj:`False`, the layer will not
            learn an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """

    _alpha: OptTensor

    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        num_relations: int,
        num_bases: Optional[int] = None,
        num_blocks: Optional[int] = None,
        mod: Optional[str] = None,
        attention_mechanism: str = "across-relation",
        attention_mode: str = "additive-self-attention",
        heads: int = 1,
        dim: int = 1,
        concat: bool = True,
        negative_slope: float = 0.2,
        dropout: float = 0.0,
        edge_dim: Optional[int] = None,
        bias: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.heads = heads
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.mod = mod
        self.activation = ReLU()
        self.concat = concat
        self.attention_mode = attention_mode
        self.attention_mechanism = attention_mechanism
        self.dim = dim
        self.edge_dim = edge_dim

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_relations = num_relations
        self.num_bases = num_bases
        self.num_blocks = num_blocks

        mod_types = ['additive', 'scaled', 'f-additive', 'f-scaled']

        if (self.attention_mechanism != "within-relation"
                and self.attention_mechanism != "across-relation"):
            raise ValueError('attention mechanism must either be '
                             '"within-relation" or "across-relation"')

        if (self.attention_mode != "additive-self-attention"
                and self.attention_mode != "multiplicative-self-attention"):
            raise ValueError('attention mode must either be '
                             '"additive-self-attention" or '
                             '"multiplicative-self-attention"')

        if self.attention_mode == "additive-self-attention" and self.dim > 1:
            raise ValueError('"additive-self-attention" mode cannot be '
                             'applied when value of d is greater than 1. '
                             'Use "multiplicative-self-attention" instead.')

        if self.dropout > 0.0 and self.mod in mod_types:
            raise ValueError('mod must be None with dropout value greater '
                             'than 0 in order to sample attention '
                             'coefficients stochastically')

        if num_bases is not None and num_blocks is not None:
            raise ValueError('Can not apply both basis-decomposition and '
                             'block-diagonal-decomposition at the same time.')

        # The learnable parameters to compute both attention logits and
        # attention coefficients:
        self.q = Parameter(
            torch.Tensor(self.heads * self.out_channels,
                         self.heads * self.dim))
        self.k = Parameter(
            torch.Tensor(self.heads * self.out_channels,
                         self.heads * self.dim))

        if bias and concat:
            self.bias = Parameter(
                torch.Tensor(self.heads * self.dim * self.out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(self.dim * self.out_channels))
        else:
            self.register_parameter('bias', None)

        if edge_dim is not None:
            self.lin_edge = Linear(self.edge_dim,
                                   self.heads * self.out_channels, bias=False,
                                   weight_initializer='glorot')
            self.e = Parameter(
                torch.Tensor(self.heads * self.out_channels,
                             self.heads * self.dim))
        else:
            self.lin_edge = None
            self.register_parameter('e', None)

        if num_bases is not None:
            self.att = Parameter(
                torch.Tensor(self.num_relations, self.num_bases))
            self.basis = Parameter(
                torch.Tensor(self.num_bases, self.in_channels,
                             self.heads * self.out_channels))
        elif num_blocks is not None:
            assert (
                self.in_channels % self.num_blocks == 0
                and (self.heads * self.out_channels) % self.num_blocks == 0), (
                    "both 'in_channels' and 'heads * out_channels' must be "
                    "multiple of 'num_blocks' used")
            self.weight = Parameter(
                torch.Tensor(self.num_relations, self.num_blocks,
                             self.in_channels // self.num_blocks,
                             (self.heads * self.out_channels) //
                             self.num_blocks))
        else:
            self.weight = Parameter(
                torch.Tensor(self.num_relations, self.in_channels,
                             self.heads * self.out_channels))

        self.w = Parameter(torch.ones(self.out_channels))
        self.l1 = Parameter(torch.Tensor(1, self.out_channels))
        self.b1 = Parameter(torch.Tensor(1, self.out_channels))
        self.l2 = Parameter(torch.Tensor(self.out_channels, self.out_channels))
        self.b2 = Parameter(torch.Tensor(1, self.out_channels))

        self._alpha = None

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if self.num_bases is not None:
            glorot(self.basis)
            glorot(self.att)
        else:
            glorot(self.weight)
        glorot(self.q)
        glorot(self.k)
        zeros(self.bias)
        ones(self.l1)
        zeros(self.b1)
        torch.full(self.l2.size(), 1 / self.out_channels)
        zeros(self.b2)
        if self.lin_edge is not None:
            glorot(self.lin_edge)
            glorot(self.e)

    def forward(self, x: Tensor, edge_index: Adj, edge_type: OptTensor = None,
                edge_attr: OptTensor = None, size: Size = None,
                return_attention_weights=None):
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor or tuple, optional): The input node features.
                Can be either a :obj:`[num_nodes, in_channels]` node feature
                matrix, or an optional one-dimensional node index tensor (in
                which case input features are treated as trainable node
                embeddings).
            edge_index (torch.Tensor or SparseTensor): The edge indices.
            edge_type (torch.Tensor, optional): The one-dimensional relation
                type/index for each edge in :obj:`edge_index`.
                Should be only :obj:`None` in case :obj:`edge_index` is of type
                :class:`torch_sparse.SparseTensor` or
                :class:`torch.sparse.Tensor`. (default: :obj:`None`)
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
            return_attention_weights (bool, optional): If set to :obj:`True`,
                will additionally return the tuple
                :obj:`(edge_index, attention_weights)`, holding the computed
                attention weights for each edge. (default: :obj:`None`)
        """
        # propagate_type: (x: Tensor, edge_type: OptTensor, edge_attr: OptTensor)  # noqa
        out = self.propagate(edge_index=edge_index, edge_type=edge_type, x=x,
                             size=size, edge_attr=edge_attr)

        alpha = self._alpha
        assert alpha is not None
        self._alpha = None

        if isinstance(return_attention_weights, bool):
            if isinstance(edge_index, Tensor):
                if is_torch_sparse_tensor(edge_index):
                    # TODO TorchScript requires to return a tuple
                    adj = set_sparse_value(edge_index, alpha)
                    return out, (adj, alpha)
                else:
                    return out, (edge_index, alpha)
            elif isinstance(edge_index, SparseTensor):
                return out, edge_index.set_value(alpha, layout='coo')
        else:
            return out

    def message(self, x_i: Tensor, x_j: Tensor, edge_type: Tensor,
                edge_attr: OptTensor, index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:

        if self.num_bases is not None:  # Basis-decomposition =================
            w = torch.matmul(self.att, self.basis.view(self.num_bases, -1))
            w = w.view(self.num_relations, self.in_channels,
                       self.heads * self.out_channels)
        if self.num_blocks is not None:  # Block-diagonal-decomposition =======
            if (x_i.dtype == torch.long and x_j.dtype == torch.long
                    and self.num_blocks is not None):
                raise ValueError('Block-diagonal decomposition not supported '
                                 'for non-continuous input features.')
            w = self.weight
            x_i = x_i.view(-1, 1, w.size(1), w.size(2))
            x_j = x_j.view(-1, 1, w.size(1), w.size(2))
            w = torch.index_select(w, 0, edge_type)
            outi = torch.einsum('abcd,acde->ace', x_i, w)
            outi = outi.contiguous().view(-1, self.heads * self.out_channels)
            outj = torch.einsum('abcd,acde->ace', x_j, w)
            outj = outj.contiguous().view(-1, self.heads * self.out_channels)
        else:  # No regularization/Basis-decomposition ========================
            if self.num_bases is None:
                w = self.weight
            w = torch.index_select(w, 0, edge_type)
            outi = torch.bmm(x_i.unsqueeze(1), w).squeeze(-2)
            outj = torch.bmm(x_j.unsqueeze(1), w).squeeze(-2)

        qi = torch.matmul(outi, self.q)
        kj = torch.matmul(outj, self.k)

        alpha_edge, alpha = 0, torch.tensor([0])
        if edge_attr is not None:
            if edge_attr.dim() == 1:
                edge_attr = edge_attr.view(-1, 1)
            assert self.lin_edge is not None, (
                "Please set 'edge_dim = edge_attr.size(-1)' while calling the "
                "RGATConv layer")
            edge_attributes = self.lin_edge(edge_attr).view(
                -1, self.heads * self.out_channels)
            if edge_attributes.size(0) != edge_attr.size(0):
                edge_attributes = torch.index_select(edge_attributes, 0,
                                                     edge_type)
            alpha_edge = torch.matmul(edge_attributes, self.e)

        if self.attention_mode == "additive-self-attention":
            if edge_attr is not None:
                alpha = torch.add(qi, kj) + alpha_edge
            else:
                alpha = torch.add(qi, kj)
            alpha = F.leaky_relu(alpha, self.negative_slope)
        elif self.attention_mode == "multiplicative-self-attention":
            if edge_attr is not None:
                alpha = (qi * kj) * alpha_edge
            else:
                alpha = qi * kj

        if self.attention_mechanism == "within-relation":
            across_out = torch.zeros_like(alpha)
            for r in range(self.num_relations):
                mask = edge_type == r
                across_out[mask] = softmax(alpha[mask], index[mask])
            alpha = across_out
        elif self.attention_mechanism == "across-relation":
            alpha = softmax(alpha, index, ptr, size_i)

        self._alpha = alpha

        if self.mod == "additive":
            if self.attention_mode == "additive-self-attention":
                ones = torch.ones_like(alpha)
                h = (outj.view(-1, self.heads, self.out_channels) *
                     ones.view(-1, self.heads, 1))
                h = torch.mul(self.w, h)

                return (outj.view(-1, self.heads, self.out_channels) *
                        alpha.view(-1, self.heads, 1) + h)
            elif self.attention_mode == "multiplicative-self-attention":
                ones = torch.ones_like(alpha)
                h = (outj.view(-1, self.heads, 1, self.out_channels) *
                     ones.view(-1, self.heads, self.dim, 1))
                h = torch.mul(self.w, h)

                return (outj.view(-1, self.heads, 1, self.out_channels) *
                        alpha.view(-1, self.heads, self.dim, 1) + h)

        elif self.mod == "scaled":
            if self.attention_mode == "additive-self-attention":
                ones = alpha.new_ones(index.size())
                degree = scatter(ones, index, dim_size=size_i,
                                 reduce='sum')[index].unsqueeze(-1)
                degree = torch.matmul(degree, self.l1) + self.b1
                degree = self.activation(degree)
                degree = torch.matmul(degree, self.l2) + self.b2

                return torch.mul(
                    outj.view(-1, self.heads, self.out_channels) *
                    alpha.view(-1, self.heads, 1),
                    degree.view(-1, 1, self.out_channels))
            elif self.attention_mode == "multiplicative-self-attention":
                ones = alpha.new_ones(index.size())
                degree = scatter(ones, index, dim_size=size_i,
                                 reduce='sum')[index].unsqueeze(-1)
                degree = torch.matmul(degree, self.l1) + self.b1
                degree = self.activation(degree)
                degree = torch.matmul(degree, self.l2) + self.b2

                return torch.mul(
                    outj.view(-1, self.heads, 1, self.out_channels) *
                    alpha.view(-1, self.heads, self.dim, 1),
                    degree.view(-1, 1, 1, self.out_channels))

        elif self.mod == "f-additive":
            alpha = torch.where(alpha > 0, alpha + 1, alpha)

        elif self.mod == "f-scaled":
            ones = alpha.new_ones(index.size())
            degree = scatter(ones, index, dim_size=size_i,
                             reduce='sum')[index].unsqueeze(-1)
            alpha = alpha * degree

        elif self.training and self.dropout > 0:
            alpha = F.dropout(alpha, p=self.dropout, training=True)

        else:
            alpha = alpha  # original

        if self.attention_mode == "additive-self-attention":
            return alpha.view(-1, self.heads, 1) * outj.view(
                -1, self.heads, self.out_channels)
        else:
            return (alpha.view(-1, self.heads, self.dim, 1) *
                    outj.view(-1, self.heads, 1, self.out_channels))

    def update(self, aggr_out: Tensor) -> Tensor:
        if self.attention_mode == "additive-self-attention":
            if self.concat is True:
                aggr_out = aggr_out.view(-1, self.heads * self.out_channels)
            else:
                aggr_out = aggr_out.mean(dim=1)

            if self.bias is not None:
                aggr_out = aggr_out + self.bias

            return aggr_out
        else:
            if self.concat is True:
                aggr_out = aggr_out.view(
                    -1, self.heads * self.dim * self.out_channels)
            else:
                aggr_out = aggr_out.mean(dim=1)
                aggr_out = aggr_out.view(-1, self.dim * self.out_channels)

            if self.bias is not None:
                aggr_out = aggr_out + self.bias

            return aggr_out

    def __repr__(self) -> str:
        return '{}({}, {}, heads={})'.format(self.__class__.__name__,
                                             self.in_channels,
                                             self.out_channels, self.heads)
pytorch/torch_geometric/nn/conv/rgcn_conv.py000066400000000000000000000352011444262304300216360ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
from torch import Tensor
from torch.nn import Parameter
from torch.nn import Parameter as Param

import torch_geometric.typing
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import (
    Adj,
    OptTensor,
    SparseTensor,
    pyg_lib,
    torch_sparse,
)
from torch_geometric.utils import index_sort, one_hot, scatter, spmm
from torch_geometric.utils.hetero import segmatmul_heuristic
from torch_geometric.utils.sparse import index2ptr


@torch.jit._overload
def masked_edge_index(edge_index, edge_mask):
    # type: (Tensor, Tensor) -> Tensor
    pass


@torch.jit._overload
def masked_edge_index(edge_index, edge_mask):
    # type: (SparseTensor, Tensor) -> SparseTensor
    pass


def masked_edge_index(edge_index, edge_mask):
    if isinstance(edge_index, Tensor):
        return edge_index[:, edge_mask]
    return torch_sparse.masked_select_nnz(edge_index, edge_mask, layout='coo')


class RGCNConv(MessagePassing):
    r"""The relational graph convolutional operator from the `"Modeling
    Relational Data with Graph Convolutional Networks"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{\Theta}_{\textrm{root}} \cdot
        \mathbf{x}_i + \sum_{r \in \mathcal{R}} \sum_{j \in \mathcal{N}_r(i)}
        \frac{1}{|\mathcal{N}_r(i)|} \mathbf{\Theta}_r \cdot \mathbf{x}_j,

    where :math:`\mathcal{R}` denotes the set of relations, *i.e.* edge types.
    Edge type needs to be a one-dimensional :obj:`torch.long` tensor which
    stores a relation identifier
    :math:`\in \{ 0, \ldots, |\mathcal{R}| - 1\}` for each edge.

    .. note::
        This implementation is as memory-efficient as possible by iterating
        over each individual relation type.
        Therefore, it may result in low GPU utilization in case the graph has a
        large number of relations.
        As an alternative approach, :class:`FastRGCNConv` does not iterate over
        each individual type, but may consume a large amount of memory to
        compensate.
        We advise to check out both implementations to see which one fits your
        needs.

    .. note::
        :class:`RGCNConv` can use `dynamic shapes
        `_, which means that the shape of the interim
        tensors can be determined at runtime.
        If your device doesn't support dynamic shapes, use
        :class:`FastRGCNConv` instead.

    Args:
        in_channels (int or tuple): Size of each input sample. A tuple
            corresponds to the sizes of source and target dimensionalities.
            In case no input features are given, this argument should
            correspond to the number of nodes in your graph.
        out_channels (int): Size of each output sample.
        num_relations (int): Number of relations.
        num_bases (int, optional): If set, this layer will use the
            basis-decomposition regularization scheme where :obj:`num_bases`
            denotes the number of bases to use. (default: :obj:`None`)
        num_blocks (int, optional): If set, this layer will use the
            block-diagonal-decomposition regularization scheme where
            :obj:`num_blocks` denotes the number of blocks to use.
            (default: :obj:`None`)
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"mean"`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        is_sorted (bool, optional): If set to :obj:`True`, assumes that
            :obj:`edge_index` is sorted by :obj:`edge_type`. This avoids
            internal re-sorting of the data and can improve runtime and memory
            efficiency. (default: :obj:`False`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        num_relations: int,
        num_bases: Optional[int] = None,
        num_blocks: Optional[int] = None,
        aggr: str = 'mean',
        root_weight: bool = True,
        is_sorted: bool = False,
        bias: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', aggr)
        super().__init__(node_dim=0, **kwargs)

        if num_bases is not None and num_blocks is not None:
            raise ValueError('Can not apply both basis-decomposition and '
                             'block-diagonal-decomposition at the same time.')

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_relations = num_relations
        self.num_bases = num_bases
        self.num_blocks = num_blocks
        self.is_sorted = is_sorted
        self.use_segmm: int = -1
        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)
        self.in_channels_l = in_channels[0]

        if num_bases is not None:
            self.weight = Parameter(
                torch.Tensor(num_bases, in_channels[0], out_channels))
            self.comp = Parameter(torch.Tensor(num_relations, num_bases))

        elif num_blocks is not None:
            assert (in_channels[0] % num_blocks == 0
                    and out_channels % num_blocks == 0)
            self.weight = Parameter(
                torch.Tensor(num_relations, num_blocks,
                             in_channels[0] // num_blocks,
                             out_channels // num_blocks))
            self.register_parameter('comp', None)

        else:
            self.weight = Parameter(
                torch.Tensor(num_relations, in_channels[0], out_channels))
            self.register_parameter('comp', None)

        if root_weight:
            self.root = Param(torch.Tensor(in_channels[1], out_channels))
        else:
            self.register_parameter('root', None)

        if bias:
            self.bias = Param(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        glorot(self.weight)
        glorot(self.comp)
        glorot(self.root)
        zeros(self.bias)

    def forward(self, x: Union[OptTensor, Tuple[OptTensor, Tensor]],
                edge_index: Adj, edge_type: OptTensor = None):
        r"""Runs the forward pass of the module.

        Args:
            x (torch.Tensor or tuple, optional): The input node features.
                Can be either a :obj:`[num_nodes, in_channels]` node feature
                matrix, or an optional one-dimensional node index tensor (in
                which case input features are treated as trainable node
                embeddings).
                Furthermore, :obj:`x` can be of type :obj:`tuple` denoting
                source and destination node features.
            edge_index (torch.Tensor or SparseTensor): The edge indices.
            edge_type (torch.Tensor, optional): The one-dimensional relation
                type/index for each edge in :obj:`edge_index`.
                Should be only :obj:`None` in case :obj:`edge_index` is of type
                :class:`torch_sparse.SparseTensor`. (default: :obj:`None`)
        """
        # Convert input features to a pair of node features or node indices.
        x_l: OptTensor = None
        if isinstance(x, tuple):
            x_l = x[0]
        else:
            x_l = x
        if x_l is None:
            x_l = torch.arange(self.in_channels_l, device=self.weight.device)

        x_r: Tensor = x_l
        if isinstance(x, tuple):
            x_r = x[1]

        size = (x_l.size(0), x_r.size(0))
        if isinstance(edge_index, SparseTensor):
            edge_type = edge_index.storage.value()
        assert edge_type is not None

        # propagate_type: (x: Tensor, edge_type_ptr: OptTensor)
        out = torch.zeros(x_r.size(0), self.out_channels, device=x_r.device)

        weight = self.weight
        if self.num_bases is not None:  # Basis-decomposition =================
            weight = (self.comp @ weight.view(self.num_bases, -1)).view(
                self.num_relations, self.in_channels_l, self.out_channels)

        if self.num_blocks is not None:  # Block-diagonal-decomposition =====

            if not torch.is_floating_point(
                    x_r) and self.num_blocks is not None:
                raise ValueError('Block-diagonal decomposition not supported '
                                 'for non-continuous input features.')

            for i in range(self.num_relations):
                tmp = masked_edge_index(edge_index, edge_type == i)
                h = self.propagate(tmp, x=x_l, edge_type_ptr=None, size=size)
                h = h.view(-1, weight.size(1), weight.size(2))
                h = torch.einsum('abc,bcd->abd', h, weight[i])
                out = out + h.contiguous().view(-1, self.out_channels)

        else:  # No regularization/Basis-decomposition ========================
            if (torch_geometric.typing.WITH_PYG_LIB and self.num_bases is None
                    and x_l.is_floating_point() and isinstance(
                        edge_index, Tensor)) and (self.use_segmm == -1
                                                  or bool(self.use_segmm)):
                if not self.is_sorted:
                    if (edge_type[1:] < edge_type[:-1]).any():
                        edge_type, perm = index_sort(
                            edge_type, max_value=self.num_relations)
                        edge_index = edge_index[:, perm]
                edge_type_ptr = index2ptr(edge_type, self.num_relations)
                if self.use_segmm == -1:
                    self.use_segmm = segmatmul_heuristic(
                        x_l, edge_type_ptr, self.weight)
                out = self.propagate(edge_index, x=x_l,
                                     edge_type_ptr=edge_type_ptr, size=size)
            else:
                for i in range(self.num_relations):
                    tmp = masked_edge_index(edge_index, edge_type == i)

                    if not torch.is_floating_point(x_r):
                        out = out + self.propagate(
                            tmp,
                            x=weight[i, x_l],
                            edge_type_ptr=None,
                            size=size,
                        )
                    else:
                        h = self.propagate(tmp, x=x_l, edge_type_ptr=None,
                                           size=size)
                        out = out + (h @ weight[i])

        root = self.root
        if root is not None:
            if not torch.is_floating_point(x_r):
                out = out + root[x_r]
            else:
                out = out + x_r @ root

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_type_ptr: OptTensor) -> Tensor:
        if torch_geometric.typing.WITH_PYG_LIB and edge_type_ptr is not None:
            # TODO Re-weight according to edge type degree for `aggr=mean`.
            return pyg_lib.ops.segment_matmul(x_j, edge_type_ptr, self.weight)

        return x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        adj_t = adj_t.set_value(None)
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_relations={self.num_relations})')


class FastRGCNConv(RGCNConv):
    r"""See :class:`RGCNConv`."""
    def forward(self, x: Union[OptTensor, Tuple[OptTensor, Tensor]],
                edge_index: Adj, edge_type: OptTensor = None):

        self.fuse = False
        assert self.aggr in ['add', 'sum', 'mean']

        # Convert input features to a pair of node features or node indices.
        x_l: OptTensor = None
        if isinstance(x, tuple):
            x_l = x[0]
        else:
            x_l = x
        if x_l is None:
            x_l = torch.arange(self.in_channels_l, device=self.weight.device)

        x_r: Tensor = x_l
        if isinstance(x, tuple):
            x_r = x[1]

        size = (x_l.size(0), x_r.size(0))

        # propagate_type: (x: Tensor, edge_type: OptTensor)
        out = self.propagate(edge_index, x=x_l, edge_type=edge_type, size=size)

        root = self.root
        if root is not None:
            if not torch.is_floating_point(x_r):
                out = out + root[x_r]
            else:
                out = out + x_r @ root

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_type: Tensor,
                edge_index_j: Tensor) -> Tensor:
        weight = self.weight
        if self.num_bases is not None:  # Basis-decomposition =================
            weight = (self.comp @ weight.view(self.num_bases, -1)).view(
                self.num_relations, self.in_channels_l, self.out_channels)

        if self.num_blocks is not None:  # Block-diagonal-decomposition =======
            if not torch.is_floating_point(x_j):
                raise ValueError('Block-diagonal decomposition not supported '
                                 'for non-continuous input features.')

            weight = weight[edge_type].view(-1, weight.size(2), weight.size(3))
            x_j = x_j.view(-1, 1, weight.size(1))
            return torch.bmm(x_j, weight).view(-1, self.out_channels)

        else:  # No regularization/Basis-decomposition ========================
            if not torch.is_floating_point(x_j):
                weight_index = edge_type * weight.size(1) + edge_index_j
                return weight.view(-1, self.out_channels)[weight_index]

            return torch.bmm(x_j.unsqueeze(-2), weight[edge_type]).squeeze(-2)

    def aggregate(self, inputs: Tensor, edge_type: Tensor, index: Tensor,
                  dim_size: Optional[int] = None) -> Tensor:

        # Compute normalization in separation for each `edge_type`.
        if self.aggr == 'mean':
            norm = one_hot(edge_type, self.num_relations, dtype=inputs.dtype)
            norm = scatter(norm, index, dim=0, dim_size=dim_size)[index]
            norm = torch.gather(norm, 1, edge_type.view(-1, 1))
            norm = 1. / norm.clamp_(1.)
            inputs = norm * inputs

        return scatter(inputs, index, dim=self.node_dim, dim_size=dim_size)
pytorch/torch_geometric/nn/conv/sage_conv.py000066400000000000000000000133221444262304300216240ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation, MultiAggregation
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptPairTensor, Size, SparseTensor
from torch_geometric.utils import spmm


class SAGEConv(MessagePassing):
    r"""The GraphSAGE operator from the `"Inductive Representation Learning on
    Large Graphs" `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i + \mathbf{W}_2 \cdot
        \mathrm{mean}_{j \in \mathcal{N(i)}} \mathbf{x}_j

    If :obj:`project = True`, then :math:`\mathbf{x}_j` will first get
    projected via

    .. math::
        \mathbf{x}_j \leftarrow \sigma ( \mathbf{W}_3 \mathbf{x}_j +
        \mathbf{b})

    as described in Eq. (3) of the paper.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        aggr (str or Aggregation, optional): The aggregation scheme to use.
            Any aggregation of :obj:`torch_geometric.nn.aggr` can be used,
            *e.g.*, :obj:`"mean"`, :obj:`"max"`, or :obj:`"lstm"`.
            (default: :obj:`"mean"`)
        normalize (bool, optional): If set to :obj:`True`, output features
            will be :math:`\ell_2`-normalized, *i.e.*,
            :math:`\frac{\mathbf{x}^{\prime}_i}
            {\| \mathbf{x}^{\prime}_i \|_2}`.
            (default: :obj:`False`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        project (bool, optional): If set to :obj:`True`, the layer will apply a
            linear transformation followed by an activation function before
            aggregation (as described in Eq. (3) of the paper).
            (default: :obj:`False`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **inputs:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{s}), (|\mathcal{V_t}|, F_{t}))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **outputs:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        aggr: Optional[Union[str, List[str], Aggregation]] = "mean",
        normalize: bool = False,
        root_weight: bool = True,
        project: bool = False,
        bias: bool = True,
        **kwargs,
    ):
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.normalize = normalize
        self.root_weight = root_weight
        self.project = project

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        if aggr == 'lstm':
            kwargs.setdefault('aggr_kwargs', {})
            kwargs['aggr_kwargs'].setdefault('in_channels', in_channels[0])
            kwargs['aggr_kwargs'].setdefault('out_channels', in_channels[0])

        super().__init__(aggr, **kwargs)

        if self.project:
            if in_channels[0] <= 0:
                raise ValueError(f"'{self.__class__.__name__}' does not "
                                 f"support lazy initialization with "
                                 f"`project=True`")
            self.lin = Linear(in_channels[0], in_channels[0], bias=True)

        if isinstance(self.aggr_module, MultiAggregation):
            aggr_out_channels = self.aggr_module.get_out_channels(
                in_channels[0])
        else:
            aggr_out_channels = in_channels[0]

        self.lin_l = Linear(aggr_out_channels, out_channels, bias=bias)
        if self.root_weight:
            self.lin_r = Linear(in_channels[1], out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if self.project:
            self.lin.reset_parameters()
        self.lin_l.reset_parameters()
        if self.root_weight:
            self.lin_r.reset_parameters()

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        if self.project and hasattr(self, 'lin'):
            x = (self.lin(x[0]).relu(), x[1])

        # propagate_type: (x: OptPairTensor)
        out = self.propagate(edge_index, x=x, size=size)
        out = self.lin_l(out)

        x_r = x[1]
        if self.root_weight and x_r is not None:
            out = out + self.lin_r(x_r)

        if self.normalize:
            out = F.normalize(out, p=2., dim=-1)

        return out

    def message(self, x_j: Tensor) -> Tensor:
        return x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        if isinstance(adj_t, SparseTensor):
            adj_t = adj_t.set_value(None, layout=None)
        return spmm(adj_t, x[0], reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, aggr={self.aggr})')
pytorch/torch_geometric/nn/conv/sg_conv.py000066400000000000000000000107651444262304300213260ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class SGConv(MessagePassing):
    r"""The simple graph convolutional operator from the `"Simplifying Graph
    Convolutional Networks" `_ paper

    .. math::
        \mathbf{X}^{\prime} = {\left(\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
        \mathbf{\hat{D}}^{-1/2} \right)}^K \mathbf{X} \mathbf{\Theta},

    where :math:`\mathbf{\hat{A}} = \mathbf{A} + \mathbf{I}` denotes the
    adjacency matrix with inserted self-loops and
    :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij}` its diagonal degree matrix.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        K (int, optional): Number of hops :math:`K`. (default: :obj:`1`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`{\left(\mathbf{\hat{D}}^{-1/2}
            \mathbf{\hat{A}} \mathbf{\hat{D}}^{-1/2} \right)}^K \mathbf{X}` on
            first execution, and will use the cached version for further
            executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:**
          node features :math:`(|\mathcal{V}|, F_{out})`
    """

    _cached_x: Optional[Tensor]

    def __init__(self, in_channels: int, out_channels: int, K: int = 1,
                 cached: bool = False, add_self_loops: bool = True,
                 bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.K = K
        self.cached = cached
        self.add_self_loops = add_self_loops

        self._cached_x = None

        self.lin = Linear(in_channels, out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        self._cached_x = None

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        cache = self._cached_x
        if cache is None:
            if isinstance(edge_index, Tensor):
                edge_index, edge_weight = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim), False,
                    self.add_self_loops, self.flow, dtype=x.dtype)
            elif isinstance(edge_index, SparseTensor):
                edge_index = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim), False,
                    self.add_self_loops, self.flow, dtype=x.dtype)

            for k in range(self.K):
                # propagate_type: (x: Tensor, edge_weight: OptTensor)
                x = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                                   size=None)
                if self.cached:
                    self._cached_x = x
        else:
            x = cache.detach()

        return self.lin(x)

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, K={self.K})')
pytorch/torch_geometric/nn/conv/signed_conv.py000066400000000000000000000142131444262304300221560ustar00rootroot00000000000000from typing import Union

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, PairTensor, SparseTensor
from torch_geometric.utils import spmm


class SignedConv(MessagePassing):
    r"""The signed graph convolutional operator from the `"Signed Graph
    Convolutional Network" `_ paper

    .. math::
        \mathbf{x}_v^{(\textrm{pos})} &= \mathbf{\Theta}^{(\textrm{pos})}
        \left[ \frac{1}{|\mathcal{N}^{+}(v)|} \sum_{w \in \mathcal{N}^{+}(v)}
        \mathbf{x}_w , \mathbf{x}_v \right]

        \mathbf{x}_v^{(\textrm{neg})} &= \mathbf{\Theta}^{(\textrm{neg})}
        \left[ \frac{1}{|\mathcal{N}^{-}(v)|} \sum_{w \in \mathcal{N}^{-}(v)}
        \mathbf{x}_w , \mathbf{x}_v \right]

    if :obj:`first_aggr` is set to :obj:`True`, and

    .. math::
        \mathbf{x}_v^{(\textrm{pos})} &= \mathbf{\Theta}^{(\textrm{pos})}
        \left[ \frac{1}{|\mathcal{N}^{+}(v)|} \sum_{w \in \mathcal{N}^{+}(v)}
        \mathbf{x}_w^{(\textrm{pos})}, \frac{1}{|\mathcal{N}^{-}(v)|}
        \sum_{w \in \mathcal{N}^{-}(v)} \mathbf{x}_w^{(\textrm{neg})},
        \mathbf{x}_v^{(\textrm{pos})} \right]

        \mathbf{x}_v^{(\textrm{neg})} &= \mathbf{\Theta}^{(\textrm{pos})}
        \left[ \frac{1}{|\mathcal{N}^{+}(v)|} \sum_{w \in \mathcal{N}^{+}(v)}
        \mathbf{x}_w^{(\textrm{neg})}, \frac{1}{|\mathcal{N}^{-}(v)|}
        \sum_{w \in \mathcal{N}^{-}(v)} \mathbf{x}_w^{(\textrm{pos})},
        \mathbf{x}_v^{(\textrm{neg})} \right]

    otherwise.
    In case :obj:`first_aggr` is :obj:`False`, the layer expects :obj:`x` to be
    a tensor where :obj:`x[:, :in_channels]` denotes the positive node features
    :math:`\mathbf{X}^{(\textrm{pos})}` and :obj:`x[:, in_channels:]` denotes
    the negative node features :math:`\mathbf{X}^{(\textrm{neg})}`.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        first_aggr (bool): Denotes which aggregation formula to use.
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})` or
          :math:`((|\mathcal{V_s}|, F_{in}), (|\mathcal{V_t}|, F_{in}))`
          if bipartite,
          positive edge indices :math:`(2, |\mathcal{E}^{(+)}|)`,
          negative edge indices :math:`(2, |\mathcal{E}^{(-)}|)`
        - **outputs:** node features :math:`(|\mathcal{V}|, F_{out})` or
          :math:`(|\mathcal{V_t}|, F_{out})` if bipartite
    """
    def __init__(self, in_channels: int, out_channels: int, first_aggr: bool,
                 bias: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'mean')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.first_aggr = first_aggr

        if first_aggr:
            self.lin_pos_l = Linear(in_channels, out_channels, False)
            self.lin_pos_r = Linear(in_channels, out_channels, bias)
            self.lin_neg_l = Linear(in_channels, out_channels, False)
            self.lin_neg_r = Linear(in_channels, out_channels, bias)
        else:
            self.lin_pos_l = Linear(2 * in_channels, out_channels, False)
            self.lin_pos_r = Linear(in_channels, out_channels, bias)
            self.lin_neg_l = Linear(2 * in_channels, out_channels, False)
            self.lin_neg_r = Linear(in_channels, out_channels, bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_pos_l.reset_parameters()
        self.lin_pos_r.reset_parameters()
        self.lin_neg_l.reset_parameters()
        self.lin_neg_r.reset_parameters()

    def forward(self, x: Union[Tensor, PairTensor], pos_edge_index: Adj,
                neg_edge_index: Adj):

        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        # propagate_type: (x: PairTensor)
        if self.first_aggr:

            out_pos = self.propagate(pos_edge_index, x=x, size=None)
            out_pos = self.lin_pos_l(out_pos)
            out_pos = out_pos + self.lin_pos_r(x[1])

            out_neg = self.propagate(neg_edge_index, x=x, size=None)
            out_neg = self.lin_neg_l(out_neg)
            out_neg = out_neg + self.lin_neg_r(x[1])

            return torch.cat([out_pos, out_neg], dim=-1)

        else:
            F_in = self.in_channels

            out_pos1 = self.propagate(pos_edge_index, size=None,
                                      x=(x[0][..., :F_in], x[1][..., :F_in]))
            out_pos2 = self.propagate(neg_edge_index, size=None,
                                      x=(x[0][..., F_in:], x[1][..., F_in:]))
            out_pos = torch.cat([out_pos1, out_pos2], dim=-1)
            out_pos = self.lin_pos_l(out_pos)
            out_pos = out_pos + self.lin_pos_r(x[1][..., :F_in])

            out_neg1 = self.propagate(pos_edge_index, size=None,
                                      x=(x[0][..., F_in:], x[1][..., F_in:]))
            out_neg2 = self.propagate(neg_edge_index, size=None,
                                      x=(x[0][..., :F_in], x[1][..., :F_in]))
            out_neg = torch.cat([out_neg1, out_neg2], dim=-1)
            out_neg = self.lin_neg_l(out_neg)
            out_neg = out_neg + self.lin_neg_r(x[1][..., F_in:])

            return torch.cat([out_pos, out_neg], dim=-1)

    def message(self, x_j: Tensor) -> Tensor:
        return x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: PairTensor) -> Tensor:
        if isinstance(adj_t, SparseTensor):
            adj_t = adj_t.set_value(None, layout=None)
        return spmm(adj_t, x[0], reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, first_aggr={self.first_aggr})')
pytorch/torch_geometric/nn/conv/simple_conv.py000066400000000000000000000074731444262304300222100ustar00rootroot00000000000000from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.aggr import Aggregation
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.typing import (
    Adj,
    OptPairTensor,
    OptTensor,
    Size,
    SparseTensor,
    torch_sparse,
)
from torch_geometric.utils import add_self_loops, spmm


class SimpleConv(MessagePassing):
    r"""A simple message passing operator that performs (non-trainable)
    propagation

    .. math::
        \mathbf{x}^{\prime}_i = \bigoplus_{j \in \mathcal{N(i)}} e_{ji} \cdot
        \mathbf{x}_j

    where :math:`\bigoplus` defines a custom aggregation scheme.

    Args:
        aggr (str or [str] or Aggregation, optional): The aggregation scheme
            to use, *e.g.*, :obj:`"add"`, :obj:`"sum"` :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"` or :obj:`"mul"`.
            In addition, can be any
            :class:`~torch_geometric.nn.aggr.Aggregation` module (or any string
            that automatically resolves to it). (default: :obj:`"sum"`)
        combine_root (str, optional): Specifies whether or how to combine the
            central node representation (one of :obj:`"sum"`, :obj:`"cat"`,
            :obj:`"self_loop"`, :obj:`None`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **inputs:**
          node features :math:`(|\mathcal{V}|, F)` or
          :math:`((|\mathcal{V_s}|, F), (|\mathcal{V_t}|, *))`
          if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **outputs:** node features :math:`(|\mathcal{V}|, F)` or
          :math:`(|\mathcal{V_t}|, F)` if bipartite
    """
    def __init__(
        self,
        aggr: Optional[Union[str, List[str], Aggregation]] = "sum",
        combine_root: Optional[str] = None,
        **kwargs,
    ):
        if combine_root not in ['sum', 'cat', 'self_loop', None]:
            raise ValueError(f"Received invalid value for 'combine_root' "
                             f"(got '{combine_root}')")

        super().__init__(aggr, **kwargs)
        self.combine_root = combine_root

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_weight: OptTensor = None, size: Size = None) -> Tensor:

        if self.combine_root is not None:
            if self.combine_root == 'self_loop':
                if not isinstance(x, Tensor) or (size is not None
                                                 and size[0] != size[1]):
                    raise ValueError("Cannot use `combine_root='self_loop'` "
                                     "for bipartite message passing")
                if isinstance(edge_index, Tensor):
                    edge_index, edge_weight = add_self_loops(
                        edge_index, edge_weight, num_nodes=x.size(0))
                elif isinstance(edge_index, SparseTensor):
                    edge_index = torch_sparse.set_diag(edge_index)

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        # propagate_type: (x: OptPairTensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=size)

        x_dst = x[1]
        if x_dst is not None and self.combine_root is not None:
            if self.combine_root == 'sum':
                out = out + x_dst
            elif self.combine_root == 'cat':
                out = torch.cat([x_dst, out], dim=-1)

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              x: OptPairTensor) -> Tensor:
        return spmm(adj_t, x[0], reduce=self.aggr)
pytorch/torch_geometric/nn/conv/spline_conv.py000066400000000000000000000142721444262304300222040ustar00rootroot00000000000000import warnings
from typing import List, Tuple, Union

import torch
from torch import Tensor, nn
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import uniform, zeros
from torch_geometric.typing import Adj, OptPairTensor, OptTensor, Size
from torch_geometric.utils.repeat import repeat

try:
    from torch_spline_conv import spline_basis, spline_weighting
except (ImportError, OSError):  # Fail gracefully on GLIBC errors
    spline_basis = None
    spline_weighting = None


class SplineConv(MessagePassing):
    r"""The spline-based convolutional operator from the `"SplineCNN: Fast
    Geometric Deep Learning with Continuous B-Spline Kernels"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{1}{|\mathcal{N}(i)|} \sum_{j \in
        \mathcal{N}(i)} \mathbf{x}_j \cdot
        h_{\mathbf{\Theta}}(\mathbf{e}_{i,j}),

    where :math:`h_{\mathbf{\Theta}}` denotes a kernel function defined
    over the weighted B-Spline tensor product basis.

    .. note::

        Pseudo-coordinates must lay in the fixed interval :math:`[0, 1]` for
        this method to work as intended.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        dim (int): Pseudo-coordinate dimensionality.
        kernel_size (int or [int]): Size of the convolving kernel.
        is_open_spline (bool or [bool], optional): If set to :obj:`False`, the
            operator will use a closed B-spline basis in this dimension.
            (default :obj:`True`)
        degree (int, optional): B-spline basis degrees. (default: :obj:`1`)
        aggr (str, optional): The aggregation scheme to use
            (:obj:`"add"`, :obj:`"mean"`, :obj:`"max"`).
            (default: :obj:`"mean"`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add transformed root node features to the output.
            (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """
    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        dim: int,
        kernel_size: Union[int, List[int]],
        is_open_spline: bool = True,
        degree: int = 1,
        aggr: str = 'mean',
        root_weight: bool = True,
        bias: bool = True,
        **kwargs,
    ):
        super().__init__(aggr=aggr, **kwargs)

        if spline_basis is None:
            raise ImportError("'SplineConv' requires 'torch-spline-conv'")

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.dim = dim
        self.degree = degree
        self.root_weight = root_weight

        kernel_size = torch.tensor(repeat(kernel_size, dim), dtype=torch.long)
        self.register_buffer('kernel_size', kernel_size)

        is_open_spline = repeat(is_open_spline, dim)
        is_open_spline = torch.tensor(is_open_spline, dtype=torch.uint8)
        self.register_buffer('is_open_spline', is_open_spline)

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.K = kernel_size.prod().item()

        if in_channels[0] > 0:
            self.weight = Parameter(
                torch.Tensor(self.K, in_channels[0], out_channels))
        else:
            self.weight = torch.nn.parameter.UninitializedParameter()
            self._hook = self.register_forward_pre_hook(
                self.initialize_parameters)

        if root_weight:
            self.lin = Linear(in_channels[1], out_channels, bias=False,
                              weight_initializer='uniform')

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        if not isinstance(self.weight, nn.UninitializedParameter):
            size = self.weight.size(0) * self.weight.size(1)
            uniform(size, self.weight)
        if self.root_weight:
            self.lin.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        if not x[0].is_cuda:
            warnings.warn(
                'We do not recommend using the non-optimized CPU version of '
                '`SplineConv`. If possible, please move your data to GPU.')

        # propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)

        x_r = x[1]
        if x_r is not None and self.root_weight:
            out = out + self.lin(x_r)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_attr: Tensor) -> Tensor:
        data = spline_basis(edge_attr, self.kernel_size, self.is_open_spline,
                            self.degree)
        return spline_weighting(x_j, self.weight, *data)

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        if isinstance(self.weight, torch.nn.parameter.UninitializedParameter):
            x = input[0][0] if isinstance(input, tuple) else input[0]
            in_channels = x.size(-1)
            self.weight.materialize((self.K, in_channels, self.out_channels))
            size = self.weight.size(0) * self.weight.size(1)
            uniform(size, self.weight)
        module._hook.remove()
        delattr(module, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, dim={self.dim})')
pytorch/torch_geometric/nn/conv/ssg_conv.py000066400000000000000000000121011444262304300214730ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class SSGConv(MessagePassing):
    r"""The simple spectral graph convolutional operator from the
    `"Simple Spectral Graph Convolution"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} = \frac{1}{K} \sum_{k=1}^K\left((1-\alpha)
        {\left(\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
        \mathbf{\hat{D}}^{-1/2} \right)}^k
        \mathbf{X}+\alpha \mathbf{X}\right) \mathbf{\Theta},

    where :math:`\mathbf{\hat{A}} = \mathbf{A} + \mathbf{I}` denotes the
    adjacency matrix with inserted self-loops and
    :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij}` its diagonal degree matrix.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.
    :class:`~torch_geometric.nn.conv.SSGConv` is an improved operator of
    :class:`~torch_geometric.nn.conv.SGConv` by introducing the :obj:`alpha`
    parameter to address the oversmoothing issue.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        alpha (float): Teleport probability :math:`\alpha \in [0, 1]`.
        K (int, optional): Number of hops :math:`K`. (default: :obj:`1`)
        cached (bool, optional): If set to :obj:`True`, the layer will cache
            the computation of :math:`\frac{1}{K} \sum_{k=1}^K\left((1-\alpha)
            {\left(\mathbf{\hat{D}}^{-1/2} \mathbf{\hat{A}}
            \mathbf{\hat{D}}^{-1/2} \right)}^k \mathbf{X}+
            \alpha \mathbf{X}\right)` on first execution, and will use the
            cached version for further executions.
            This parameter should only be set to :obj:`True` in transductive
            learning scenarios. (default: :obj:`False`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:**
          node features :math:`(|\mathcal{V}|, F_{out})`
    """

    _cached_h: Optional[Tensor]

    def __init__(self, in_channels: int, out_channels: int, alpha: float,
                 K: int = 1, cached: bool = False, add_self_loops: bool = True,
                 bias: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.alpha = alpha
        self.K = K
        self.cached = cached
        self.add_self_loops = add_self_loops

        self._cached_h = None

        self.lin = Linear(in_channels, out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        self._cached_h = None

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        cache = self._cached_h
        if cache is None:
            if isinstance(edge_index, Tensor):
                edge_index, edge_weight = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim), False,
                    self.add_self_loops, self.flow, dtype=x.dtype)
            elif isinstance(edge_index, SparseTensor):
                edge_index = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim), False,
                    self.add_self_loops, self.flow, dtype=x.dtype)

            h = x * self.alpha
            for k in range(self.K):
                # propagate_type: (x: Tensor, edge_weight: OptTensor)
                x = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                                   size=None)
                h = h + (1 - self.alpha) / self.K * x
            if self.cached:
                self._cached_h = h
        else:
            h = cache.detach()

        return self.lin(h)

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, K={self.K}, alpha={self.alpha})')
pytorch/torch_geometric/nn/conv/supergat_conv.py000066400000000000000000000273141444262304300225450ustar00rootroot00000000000000import math
from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import Adj, OptTensor, SparseTensor, torch_sparse
from torch_geometric.utils import (
    add_self_loops,
    batched_negative_sampling,
    dropout_edge,
    is_undirected,
    negative_sampling,
    remove_self_loops,
    softmax,
    to_undirected,
)


class SuperGATConv(MessagePassing):
    r"""The self-supervised graph attentional operator from the `"How to Find
    Your Friendly Neighborhood: Graph Attention Design with Self-Supervision"
    `_ paper

    .. math::

        \mathbf{x}^{\prime}_i = \alpha_{i,i}\mathbf{\Theta}\mathbf{x}_{i} +
        \sum_{j \in \mathcal{N}(i)} \alpha_{i,j}\mathbf{\Theta}\mathbf{x}_{j},

    where the two types of attention :math:`\alpha_{i,j}^{\mathrm{MX\ or\ SD}}`
    are computed as:

    .. math::

        \alpha_{i,j}^{\mathrm{MX\ or\ SD}} &=
        \frac{
        \exp\left(\mathrm{LeakyReLU}\left(
            e_{i,j}^{\mathrm{MX\ or\ SD}}
        \right)\right)}
        {\sum_{k \in \mathcal{N}(i) \cup \{ i \}}
        \exp\left(\mathrm{LeakyReLU}\left(
            e_{i,k}^{\mathrm{MX\ or\ SD}}
        \right)\right)}

        e_{i,j}^{\mathrm{MX}} &= \mathbf{a}^{\top}
            [\mathbf{\Theta}\mathbf{x}_i \, \Vert \,
             \mathbf{\Theta}\mathbf{x}_j]
            \cdot \sigma \left(
                \left( \mathbf{\Theta}\mathbf{x}_i \right)^{\top}
                \mathbf{\Theta}\mathbf{x}_j
            \right)

        e_{i,j}^{\mathrm{SD}} &= \frac{
            \left( \mathbf{\Theta}\mathbf{x}_i \right)^{\top}
            \mathbf{\Theta}\mathbf{x}_j
        }{ \sqrt{d} }

    The self-supervised task is a link prediction using the attention values
    as input to predict the likelihood :math:`\phi_{i,j}^{\mathrm{MX\ or\ SD}}`
    that an edge exists between nodes:

    .. math::

        \phi_{i,j}^{\mathrm{MX}} &= \sigma \left(
            \left( \mathbf{\Theta}\mathbf{x}_i \right)^{\top}
            \mathbf{\Theta}\mathbf{x}_j
        \right)

        \phi_{i,j}^{\mathrm{SD}} &= \sigma \left(
            \frac{
                \left( \mathbf{\Theta}\mathbf{x}_i \right)^{\top}
                \mathbf{\Theta}\mathbf{x}_j
            }{ \sqrt{d} }
        \right)

    .. note::

        For an example of using SuperGAT, see `examples/super_gat.py
        `_.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        add_self_loops (bool, optional): If set to :obj:`False`, will not add
            self-loops to the input graph. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        attention_type (str, optional): Type of attention to use
            (:obj:`'MX'`, :obj:`'SD'`). (default: :obj:`'MX'`)
        neg_sample_ratio (float, optional): The ratio of the number of sampled
            negative edges to the number of positive edges.
            (default: :obj:`0.5`)
        edge_sample_ratio (float, optional): The ratio of samples to use for
            training among the number of training edges. (default: :obj:`1.0`)
        is_undirected (bool, optional): Whether the input graph is undirected.
            If not given, will be automatically computed with the input graph
            when negative sampling is performed. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          edge indices :math:`(2, |\mathcal{E}|)`,
          negative edge indices :math:`(2, |\mathcal{E}^{(-)}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, H * F_{out})`
    """
    att_x: OptTensor
    att_y: OptTensor

    def __init__(self, in_channels: int, out_channels: int, heads: int = 1,
                 concat: bool = True, negative_slope: float = 0.2,
                 dropout: float = 0.0, add_self_loops: bool = True,
                 bias: bool = True, attention_type: str = 'MX',
                 neg_sample_ratio: float = 0.5, edge_sample_ratio: float = 1.0,
                 is_undirected: bool = False, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.add_self_loops = add_self_loops
        self.attention_type = attention_type
        self.neg_sample_ratio = neg_sample_ratio
        self.edge_sample_ratio = edge_sample_ratio
        self.is_undirected = is_undirected

        assert attention_type in ['MX', 'SD']
        assert 0.0 < neg_sample_ratio and 0.0 < edge_sample_ratio <= 1.0

        self.lin = Linear(in_channels, heads * out_channels, bias=False,
                          weight_initializer='glorot')

        if self.attention_type == 'MX':
            self.att_l = Parameter(torch.Tensor(1, heads, out_channels))
            self.att_r = Parameter(torch.Tensor(1, heads, out_channels))
        else:  # self.attention_type == 'SD'
            self.register_parameter('att_l', None)
            self.register_parameter('att_r', None)

        self.att_x = self.att_y = None  # x/y for self-supervision

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin.reset_parameters()
        glorot(self.att_l)
        glorot(self.att_r)
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj,
                neg_edge_index: OptTensor = None,
                batch: OptTensor = None) -> Tensor:
        r"""Runs the forward pass of the module.

        Args:
            neg_edge_index (torch.Tensor, optional): The negative edges to
                train against. If not given, uses negative sampling to
                calculate negative edges. (default: :obj:`None`)
        """
        N, H, C = x.size(0), self.heads, self.out_channels

        if self.add_self_loops:
            if isinstance(edge_index, SparseTensor):
                edge_index = torch_sparse.fill_diag(edge_index, 1.)
            else:
                edge_index, _ = remove_self_loops(edge_index)
                edge_index, _ = add_self_loops(edge_index, num_nodes=N)

        x = self.lin(x).view(-1, H, C)

        # propagate_type: (x: Tensor)
        out = self.propagate(edge_index, x=x, size=None)

        if self.training:
            if isinstance(edge_index, SparseTensor):
                col, row, _ = edge_index.coo()
                edge_index = torch.stack([row, col], dim=0)
            pos_edge_index = self.positive_sampling(edge_index)

            pos_att = self.get_attention(
                edge_index_i=pos_edge_index[1],
                x_i=x[pos_edge_index[1]],
                x_j=x[pos_edge_index[0]],
                num_nodes=x.size(0),
                return_logits=True,
            )

            if neg_edge_index is None:
                neg_edge_index = self.negative_sampling(edge_index, N, batch)

            neg_att = self.get_attention(
                edge_index_i=neg_edge_index[1],
                x_i=x[neg_edge_index[1]],
                x_j=x[neg_edge_index[0]],
                num_nodes=x.size(0),
                return_logits=True,
            )

            self.att_x = torch.cat([pos_att, neg_att], dim=0)
            self.att_y = self.att_x.new_zeros(self.att_x.size(0))
            self.att_y[:pos_edge_index.size(1)] = 1.

        if self.concat is True:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, edge_index_i: Tensor, x_i: Tensor, x_j: Tensor,
                size_i: Optional[int]) -> Tensor:
        alpha = self.get_attention(edge_index_i, x_i, x_j, num_nodes=size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return x_j * alpha.view(-1, self.heads, 1)

    def negative_sampling(self, edge_index: Tensor, num_nodes: int,
                          batch: OptTensor = None) -> Tensor:

        num_neg_samples = int(self.neg_sample_ratio * self.edge_sample_ratio *
                              edge_index.size(1))

        if not self.is_undirected and not is_undirected(
                edge_index, num_nodes=num_nodes):
            edge_index = to_undirected(edge_index, num_nodes=num_nodes)

        if batch is None:
            neg_edge_index = negative_sampling(edge_index, num_nodes,
                                               num_neg_samples=num_neg_samples)
        else:
            neg_edge_index = batched_negative_sampling(
                edge_index, batch, num_neg_samples=num_neg_samples)

        return neg_edge_index

    def positive_sampling(self, edge_index: Tensor) -> Tensor:
        pos_edge_index, _ = dropout_edge(edge_index,
                                         p=1. - self.edge_sample_ratio,
                                         training=self.training)
        return pos_edge_index

    def get_attention(self, edge_index_i: Tensor, x_i: Tensor, x_j: Tensor,
                      num_nodes: Optional[int],
                      return_logits: bool = False) -> Tensor:

        if self.attention_type == 'MX':
            logits = (x_i * x_j).sum(dim=-1)
            if return_logits:
                return logits

            alpha = (x_j * self.att_l).sum(-1) + (x_i * self.att_r).sum(-1)
            alpha = alpha * logits.sigmoid()

        else:  # self.attention_type == 'SD'
            alpha = (x_i * x_j).sum(dim=-1) / math.sqrt(self.out_channels)
            if return_logits:
                return alpha

        alpha = F.leaky_relu(alpha, self.negative_slope)
        alpha = softmax(alpha, edge_index_i, num_nodes=num_nodes)
        return alpha

    def get_attention_loss(self) -> Tensor:
        r"""Computes the self-supervised graph attention loss."""
        if not self.training:
            return torch.tensor([0], device=self.lin.weight.device)

        return F.binary_cross_entropy_with_logits(
            self.att_x.mean(dim=-1),
            self.att_y,
        )

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads}, '
                f'type={self.attention_type})')
pytorch/torch_geometric/nn/conv/tag_conv.py000066400000000000000000000101671444262304300214640ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class TAGConv(MessagePassing):
    r"""The topology adaptive graph convolutional networks operator from the
    `"Topology Adaptive Graph Convolutional Networks"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} = \sum_{k=0}^K \left( \mathbf{D}^{-1/2} \mathbf{A}
        \mathbf{D}^{-1/2} \right)^k \mathbf{X} \mathbf{W}_{k},

    where :math:`\mathbf{A}` denotes the adjacency matrix and
    :math:`D_{ii} = \sum_{j=0} A_{ij}` its diagonal degree matrix.
    The adjacency matrix can include other values than :obj:`1` representing
    edge weights via the optional :obj:`edge_weight` tensor.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        out_channels (int): Size of each output sample.
        K (int, optional): Number of hops :math:`K`. (default: :obj:`3`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        normalize (bool, optional): Whether to apply symmetric normalization.
            (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node_features :math:`(|\mathcal{V}|, F_{in})`,
          edge_index :math:`(2, |\mathcal{E}|)`,
          edge_weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int, K: int = 3,
                 bias: bool = True, normalize: bool = True, **kwargs):
        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.K = K
        self.normalize = normalize

        self.lins = torch.nn.ModuleList([
            Linear(in_channels, out_channels, bias=False) for _ in range(K + 1)
        ])

        if bias:
            self.bias = torch.nn.Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        for lin in self.lins:
            lin.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:

        if self.normalize:
            if isinstance(edge_index, Tensor):
                edge_index, edge_weight = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim),
                    improved=False, add_self_loops=False, flow=self.flow,
                    dtype=x.dtype)

            elif isinstance(edge_index, SparseTensor):
                edge_index = gcn_norm(  # yapf: disable
                    edge_index, edge_weight, x.size(self.node_dim),
                    add_self_loops=False, flow=self.flow, dtype=x.dtype)

        out = self.lins[0](x)
        for lin in self.lins[1:]:
            # propagate_type: (x: Tensor, edge_weight: OptTensor)
            x = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                               size=None)
            out = out + lin.forward(x)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, K={self.K})')
pytorch/torch_geometric/nn/conv/transformer_conv.py000066400000000000000000000223211444262304300232460ustar00rootroot00000000000000import math
from typing import Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense.linear import Linear
from torch_geometric.typing import Adj, OptTensor, PairTensor, SparseTensor
from torch_geometric.utils import softmax


class TransformerConv(MessagePassing):
    r"""The graph transformer operator from the `"Masked Label Prediction:
    Unified Message Passing Model for Semi-Supervised Classification"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i +
        \sum_{j \in \mathcal{N}(i)} \alpha_{i,j} \mathbf{W}_2 \mathbf{x}_{j},

    where the attention coefficients :math:`\alpha_{i,j}` are computed via
    multi-head dot product attention:

    .. math::
        \alpha_{i,j} = \textrm{softmax} \left(
        \frac{(\mathbf{W}_3\mathbf{x}_i)^{\top} (\mathbf{W}_4\mathbf{x}_j)}
        {\sqrt{d}} \right)

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        out_channels (int): Size of each output sample.
        heads (int, optional): Number of multi-head-attentions.
            (default: :obj:`1`)
        concat (bool, optional): If set to :obj:`False`, the multi-head
            attentions are averaged instead of concatenated.
            (default: :obj:`True`)
        beta (bool, optional): If set, will combine aggregation and
            skip information via

            .. math::
                \mathbf{x}^{\prime}_i = \beta_i \mathbf{W}_1 \mathbf{x}_i +
                (1 - \beta_i) \underbrace{\left(\sum_{j \in \mathcal{N}(i)}
                \alpha_{i,j} \mathbf{W}_2 \vec{x}_j \right)}_{=\mathbf{m}_i}

            with :math:`\beta_i = \textrm{sigmoid}(\mathbf{w}_5^{\top}
            [ \mathbf{W}_1 \mathbf{x}_i, \mathbf{m}_i, \mathbf{W}_1
            \mathbf{x}_i - \mathbf{m}_i ])` (default: :obj:`False`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        edge_dim (int, optional): Edge feature dimensionality (in case
            there are any). Edge features are added to the keys after
            linear transformation, that is, prior to computing the
            attention dot product. They are also added to final values
            after the same linear transformation. The model is:

            .. math::
                \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i +
                \sum_{j \in \mathcal{N}(i)} \alpha_{i,j} \left(
                \mathbf{W}_2 \mathbf{x}_{j} + \mathbf{W}_6 \mathbf{e}_{ij}
                \right),

            where the attention coefficients :math:`\alpha_{i,j}` are now
            computed via:

            .. math::
                \alpha_{i,j} = \textrm{softmax} \left(
                \frac{(\mathbf{W}_3\mathbf{x}_i)^{\top}
                (\mathbf{W}_4\mathbf{x}_j + \mathbf{W}_6 \mathbf{e}_{ij})}
                {\sqrt{d}} \right)

            (default :obj:`None`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        root_weight (bool, optional): If set to :obj:`False`, the layer will
            not add the transformed root node features to the output and the
            option  :attr:`beta` is set to :obj:`False`. (default: :obj:`True`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.
    """
    _alpha: OptTensor

    def __init__(
        self,
        in_channels: Union[int, Tuple[int, int]],
        out_channels: int,
        heads: int = 1,
        concat: bool = True,
        beta: bool = False,
        dropout: float = 0.,
        edge_dim: Optional[int] = None,
        bias: bool = True,
        root_weight: bool = True,
        **kwargs,
    ):
        kwargs.setdefault('aggr', 'add')
        super().__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.beta = beta and root_weight
        self.root_weight = root_weight
        self.concat = concat
        self.dropout = dropout
        self.edge_dim = edge_dim
        self._alpha = None

        if isinstance(in_channels, int):
            in_channels = (in_channels, in_channels)

        self.lin_key = Linear(in_channels[0], heads * out_channels)
        self.lin_query = Linear(in_channels[1], heads * out_channels)
        self.lin_value = Linear(in_channels[0], heads * out_channels)
        if edge_dim is not None:
            self.lin_edge = Linear(edge_dim, heads * out_channels, bias=False)
        else:
            self.lin_edge = self.register_parameter('lin_edge', None)

        if concat:
            self.lin_skip = Linear(in_channels[1], heads * out_channels,
                                   bias=bias)
            if self.beta:
                self.lin_beta = Linear(3 * heads * out_channels, 1, bias=False)
            else:
                self.lin_beta = self.register_parameter('lin_beta', None)
        else:
            self.lin_skip = Linear(in_channels[1], out_channels, bias=bias)
            if self.beta:
                self.lin_beta = Linear(3 * out_channels, 1, bias=False)
            else:
                self.lin_beta = self.register_parameter('lin_beta', None)

        self.reset_parameters()

    def reset_parameters(self):
        super().reset_parameters()
        self.lin_key.reset_parameters()
        self.lin_query.reset_parameters()
        self.lin_value.reset_parameters()
        if self.edge_dim:
            self.lin_edge.reset_parameters()
        self.lin_skip.reset_parameters()
        if self.beta:
            self.lin_beta.reset_parameters()

    def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj,
                edge_attr: OptTensor = None, return_attention_weights=None):
        # type: (Union[Tensor, PairTensor], Tensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, NoneType) -> Tensor  # noqa
        # type: (Union[Tensor, PairTensor], Tensor, OptTensor, bool) -> Tuple[Tensor, Tuple[Tensor, Tensor]]  # noqa
        # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, bool) -> Tuple[Tensor, SparseTensor]  # noqa
        r"""Runs the forward pass of the module.

        Args:
            return_attention_weights (bool, optional): If set to :obj:`True`,
                will additionally return the tuple
                :obj:`(edge_index, attention_weights)`, holding the computed
                attention weights for each edge. (default: :obj:`None`)
        """

        H, C = self.heads, self.out_channels

        if isinstance(x, Tensor):
            x: PairTensor = (x, x)

        query = self.lin_query(x[1]).view(-1, H, C)
        key = self.lin_key(x[0]).view(-1, H, C)
        value = self.lin_value(x[0]).view(-1, H, C)

        # propagate_type: (query: Tensor, key:Tensor, value: Tensor, edge_attr: OptTensor) # noqa
        out = self.propagate(edge_index, query=query, key=key, value=value,
                             edge_attr=edge_attr, size=None)

        alpha = self._alpha
        self._alpha = None

        if self.concat:
            out = out.view(-1, self.heads * self.out_channels)
        else:
            out = out.mean(dim=1)

        if self.root_weight:
            x_r = self.lin_skip(x[1])
            if self.lin_beta is not None:
                beta = self.lin_beta(torch.cat([out, x_r, out - x_r], dim=-1))
                beta = beta.sigmoid()
                out = beta * x_r + (1 - beta) * out
            else:
                out = out + x_r

        if isinstance(return_attention_weights, bool):
            assert alpha is not None
            if isinstance(edge_index, Tensor):
                return out, (edge_index, alpha)
            elif isinstance(edge_index, SparseTensor):
                return out, edge_index.set_value(alpha, layout='coo')
        else:
            return out

    def message(self, query_i: Tensor, key_j: Tensor, value_j: Tensor,
                edge_attr: OptTensor, index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:

        if self.lin_edge is not None:
            assert edge_attr is not None
            edge_attr = self.lin_edge(edge_attr).view(-1, self.heads,
                                                      self.out_channels)
            key_j = key_j + edge_attr

        alpha = (query_i * key_j).sum(dim=-1) / math.sqrt(self.out_channels)
        alpha = softmax(alpha, index, ptr, size_i)
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)

        out = value_j
        if edge_attr is not None:
            out = out + edge_attr

        out = out * alpha.view(-1, self.heads, 1)
        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/conv/utils/000077500000000000000000000000001444262304300204455ustar00rootroot00000000000000pytorch/torch_geometric/nn/conv/utils/__init__.py000066400000000000000000000014671444262304300225660ustar00rootroot00000000000000from .cheatsheet import paper_title, paper_link
from .cheatsheet import supports_sparse_tensor
from .cheatsheet import supports_edge_weights
from .cheatsheet import supports_edge_features
from .cheatsheet import supports_bipartite_graphs
from .cheatsheet import supports_static_graphs
from .cheatsheet import supports_lazy_initialization
from .cheatsheet import processes_heterogeneous_graphs
from .cheatsheet import processes_hypergraphs
from .cheatsheet import processes_point_clouds

__all__ = [
    'paper_title',
    'paper_link',
    'supports_sparse_tensor',
    'supports_edge_weights',
    'supports_edge_features',
    'supports_bipartite_graphs',
    'supports_static_graphs',
    'supports_lazy_initialization',
    'processes_heterogeneous_graphs',
    'processes_hypergraphs',
    'processes_point_clouds',
]
pytorch/torch_geometric/nn/conv/utils/cheatsheet.py000066400000000000000000000052041444262304300231350ustar00rootroot00000000000000import importlib
import inspect
import re


def paper_title(cls: str) -> str:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    match = re.search('`\".+?\"', inspect.getdoc(cls), flags=re.DOTALL)
    return None if match is None else match.group().replace('\n', ' ')[2:-1]


def paper_link(cls: str) -> str:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    match = re.search('<.+?>', inspect.getdoc(cls), flags=re.DOTALL)
    return None if match is None else match.group().replace('\n', ' ')[1:-1]


def supports_sparse_tensor(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'SparseTensor' in str(signature)


def supports_edge_weights(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'edge_weight' in str(signature)


def supports_edge_features(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'edge_attr' in str(signature)


def supports_bipartite_graphs(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'Union[torch.Tensor, Tuple[torch.Tensor' in str(signature)


def supports_static_graphs(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    return 'node_dim=' not in inspect.getsource(cls.__init__)


def supports_lazy_initialization(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    doc = re.sub(' +', ' ', inspect.getdoc(cls).replace('\n', ' '))
    match = re.search('or :obj:`-1` to derive the size from the first', doc)
    return match is not None


def processes_heterogeneous_graphs(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'edge_index_dict' in str(signature) or 'edge_type' in str(signature)


def processes_hypergraphs(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return 'hyperedge_index' in str(signature)


def processes_point_clouds(cls: str) -> bool:
    cls = importlib.import_module('torch_geometric.nn.conv').__dict__[cls]
    signature = inspect.signature(cls.forward)
    return (('edge_index' not in str(signature)
             and 'csc' not in str(signature)) or 'pos' in str(signature))
pytorch/torch_geometric/nn/conv/utils/helpers.py000066400000000000000000000002721444262304300224620ustar00rootroot00000000000000import torch


def expand_left(src: torch.Tensor, dim: int, dims: int) -> torch.Tensor:
    for _ in range(dims + dim if dim < 0 else dim):
        src = src.unsqueeze(0)
    return src
pytorch/torch_geometric/nn/conv/utils/inspector.py000066400000000000000000000062731444262304300230350ustar00rootroot00000000000000import inspect
import re
from collections import OrderedDict
from typing import Any, Callable, Dict, List, Optional, Set

from torch_geometric.nn.conv.utils.typing import parse_types


class Inspector:
    def __init__(self, base_class: Any):
        self.base_class: Any = base_class
        self.params: Dict[str, Dict[str, Any]] = {}

    def inspect(self, func: Callable,
                pop_first: bool = False) -> Dict[str, Any]:
        params = inspect.signature(func).parameters
        params = OrderedDict(params)
        if pop_first:
            params.popitem(last=False)
        self.params[func.__name__] = params

    def keys(self, func_names: Optional[List[str]] = None) -> Set[str]:
        keys = []
        for func in func_names or list(self.params.keys()):
            keys += self.params[func].keys()
        return set(keys)

    def _implements(self, cls, func_name: str) -> bool:
        if cls.__name__ == 'MessagePassing':
            return False
        if func_name in cls.__dict__.keys():
            return True
        return any(self._implements(c, func_name) for c in cls.__bases__)

    def implements(self, func_name: str) -> bool:
        return self._implements(self.base_class.__class__, func_name)

    def types(self, func_names: Optional[List[str]] = None) -> Dict[str, str]:
        out: Dict[str, str] = {}
        for func_name in func_names or list(self.params.keys()):
            func = getattr(self.base_class, func_name)
            arg_types = parse_types(func)[0][0]
            for key in self.params[func_name].keys():
                if key in out and out[key] != arg_types[key]:
                    raise ValueError(
                        (f'Found inconsistent types for argument {key}. '
                         f'Expected type {out[key]} but found type '
                         f'{arg_types[key]}.'))
                out[key] = arg_types[key]
        return out

    def distribute(self, func_name, kwargs: Dict[str, Any]):
        out = {}
        for key, param in self.params[func_name].items():
            data = kwargs.get(key, inspect.Parameter.empty)
            if data is inspect.Parameter.empty:
                if param.default is inspect.Parameter.empty:
                    raise TypeError(f'Required parameter {key} is empty.')
                data = param.default
            out[key] = data
        return out


def func_header_repr(func: Callable, keep_annotation: bool = True) -> str:
    source = inspect.getsource(func)
    signature = inspect.signature(func)

    if keep_annotation:
        return ''.join(re.split(r'(\).*?:.*?\n)', source,
                                maxsplit=1)[:2]).strip()

    params_repr = ['self']
    for param in signature.parameters.values():
        params_repr.append(param.name)
        if param.default is not inspect.Parameter.empty:
            params_repr[-1] += f'={param.default}'

    return f'def {func.__name__}({", ".join(params_repr)}):'


def func_body_repr(func: Callable, keep_annotation: bool = True) -> str:
    source = inspect.getsource(func)
    body_repr = re.split(r'\).*?:.*?\n', source, maxsplit=1)[1]
    if not keep_annotation:
        body_repr = re.sub(r'\s*# type:.*\n', '', body_repr)
    return body_repr
pytorch/torch_geometric/nn/conv/utils/jit.py000066400000000000000000000012471444262304300216110ustar00rootroot00000000000000import os.path as osp
import sys
from getpass import getuser
from importlib.util import module_from_spec, spec_from_file_location
from tempfile import NamedTemporaryFile as TempFile
from tempfile import gettempdir

from torch_geometric.data.makedirs import makedirs


def class_from_module_repr(cls_name, module_repr):
    path = osp.join(gettempdir(), f'{getuser()}_pyg')
    makedirs(path)
    with TempFile(mode='w+', suffix='.py', delete=False, dir=path) as f:
        f.write(module_repr)
    spec = spec_from_file_location(cls_name, f.name)
    mod = module_from_spec(spec)
    sys.modules[cls_name] = mod
    spec.loader.exec_module(mod)
    return getattr(mod, cls_name)
pytorch/torch_geometric/nn/conv/utils/typing.py000066400000000000000000000074231444262304300223370ustar00rootroot00000000000000import inspect
import re
from collections import OrderedDict
from itertools import product
from typing import Callable, Dict, List, Tuple

import pyparsing as pp


def split_types_repr(types_repr: str) -> List[str]:
    out = []
    i = depth = 0
    for j, char in enumerate(types_repr):
        if char == '[':
            depth += 1
        elif char == ']':
            depth -= 1
        elif char == ',' and depth == 0:
            out.append(types_repr[i:j].strip())
            i = j + 1
    out.append(types_repr[i:].strip())
    return out


def sanitize(type_repr: str):
    type_repr = re.sub(r'', r'\1', type_repr)
    type_repr = type_repr.replace('typing.', '')
    type_repr = type_repr.replace('torch_sparse.tensor.', '')
    type_repr = type_repr.replace('Adj', 'Union[Tensor, SparseTensor]')
    type_repr = type_repr.replace('torch_geometric.SparseTensor',
                                  'SparseTensor')

    # Replace `Union[..., NoneType]` by `Optional[...]`.
    sexp = pp.nestedExpr(opener='[', closer=']')
    tree = sexp.parseString(f'[{type_repr.replace(",", " ")}]').asList()[0]

    def union_to_optional_(tree):
        for i in range(len(tree)):
            e, n = tree[i], tree[i + 1] if i + 1 < len(tree) else []
            if e == 'Union' and n[-1] == 'NoneType':
                tree[i] = 'Optional'
                tree[i + 1] = tree[i + 1][:-1]
            elif e == 'Union' and 'NoneType' in n:
                idx = n.index('NoneType')
                n[idx] = [n[idx - 1]]
                n[idx - 1] = 'Optional'
            elif isinstance(e, list):
                tree[i] = union_to_optional_(e)
        return tree

    tree = union_to_optional_(tree)
    type_repr = re.sub(r'\'|\"', '', str(tree)[1:-1]).replace(', [', '[')

    return type_repr


def param_type_repr(param) -> str:
    if param.annotation is inspect.Parameter.empty:
        return 'torch.Tensor'
    return sanitize(re.split(r':|='.strip(), str(param))[1])


def return_type_repr(signature) -> str:
    return_type = signature.return_annotation
    if return_type is inspect.Parameter.empty:
        return 'torch.Tensor'
    elif str(return_type)[:6] != ' List[Tuple[Dict[str, str], str]]:
    source = inspect.getsource(func)
    signature = inspect.signature(func)

    # Parse `# type: (...) -> ...` annotation. Note that it is allowed to pass
    # multiple `# type:` annotations in `forward()`.
    iterator = re.finditer(r'#\s*type:\s*\((.*)\)\s*->\s*(.*)\s*\n', source)
    matches = list(iterator)

    if len(matches) > 0:
        out = []
        args = list(signature.parameters.keys())
        for match in matches:
            arg_types_repr, return_type = match.groups()
            arg_types = split_types_repr(arg_types_repr)
            arg_types = OrderedDict((k, v) for k, v in zip(args, arg_types))
            return_type = return_type.split('#')[0].strip()
            out.append((arg_types, return_type))
        return out

    # Alternatively, parse annotations using the inspected signature.
    else:
        ps = signature.parameters
        arg_types = OrderedDict((k, param_type_repr(v)) for k, v in ps.items())
        return [(arg_types, return_type_repr(signature))]


def resolve_types(arg_types: Dict[str, str],
                  return_type_repr: str) -> List[Tuple[List[str], str]]:
    out = []
    for type_repr in arg_types.values():
        if type_repr[:5] == 'Union':
            out.append(split_types_repr(type_repr[6:-1]))
        else:
            out.append([type_repr])
    return [(x, return_type_repr) for x in product(*out)]
pytorch/torch_geometric/nn/conv/wl_conv.py000066400000000000000000000060371444262304300213340ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.typing import Adj
from torch_geometric.utils import (
    degree,
    is_sparse,
    scatter,
    sort_edge_index,
    to_edge_index,
)


class WLConv(torch.nn.Module):
    r"""The Weisfeiler Lehman operator from the `"A Reduction of a Graph to a
    Canonical Form and an Algebra Arising During this Reduction"
    `_ paper, which
    iteratively refines node colorings:

    .. math::
        \mathbf{x}^{\prime}_i = \textrm{hash} \left( \mathbf{x}_i, \{
        \mathbf{x}_j \colon j \in \mathcal{N}(i) \} \right)

    Shapes:
        - **input:**
          node coloring :math:`(|\mathcal{V}|, F_{in})` *(one-hot encodings)*
          or :math:`(|\mathcal{V}|)` *(integer-based)*,
          edge indices :math:`(2, |\mathcal{E}|)`
        - **output:** node coloring :math:`(|\mathcal{V}|)` *(integer-based)*
    """
    def __init__(self):
        super().__init__()
        self.hashmap = {}

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.hashmap = {}

    @torch.no_grad()
    def forward(self, x: Tensor, edge_index: Adj) -> Tensor:
        r"""Runs the forward pass of the module."""
        if x.dim() > 1:
            assert (x.sum(dim=-1) == 1).sum() == x.size(0)
            x = x.argmax(dim=-1)  # one-hot -> integer.
        assert x.dtype == torch.long

        if is_sparse(edge_index):
            col_and_row, _ = to_edge_index(edge_index)
            col = col_and_row[0]
            row = col_and_row[1]
        else:
            edge_index = sort_edge_index(edge_index, num_nodes=x.size(0),
                                         sort_by_row=False)
            row, col = edge_index[0], edge_index[1]

        # `col` is sorted, so we can use it to `split` neighbors to groups:
        deg = degree(col, x.size(0), dtype=torch.long).tolist()

        out = []
        for node, neighbors in zip(x.tolist(), x[row].split(deg)):
            idx = hash(tuple([node] + neighbors.sort()[0].tolist()))
            if idx not in self.hashmap:
                self.hashmap[idx] = len(self.hashmap)
            out.append(self.hashmap[idx])

        return torch.tensor(out, device=x.device)

    def histogram(self, x: Tensor, batch: Optional[Tensor] = None,
                  norm: bool = False) -> Tensor:
        r"""Given a node coloring :obj:`x`, computes the color histograms of
        the respective graphs (separated by :obj:`batch`)."""

        if batch is None:
            batch = torch.zeros(x.size(0), dtype=torch.long, device=x.device)

        num_colors = len(self.hashmap)
        batch_size = int(batch.max()) + 1

        index = batch * num_colors + x
        out = scatter(torch.ones_like(index), index, dim=0,
                      dim_size=num_colors * batch_size, reduce='sum')
        out = out.view(batch_size, num_colors)

        if norm:
            out = out.to(torch.float)
            out /= out.norm(dim=-1, keepdim=True)

        return out
pytorch/torch_geometric/nn/conv/wl_conv_continuous.py000066400000000000000000000044551444262304300236240ustar00rootroot00000000000000from typing import Union

from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.typing import OptPairTensor, OptTensor, Size
from torch_geometric.utils import scatter


class WLConvContinuous(MessagePassing):
    r"""The Weisfeiler Lehman operator from the `"Wasserstein
    Weisfeiler-Lehman Graph Kernels" `_
    paper. Refinement is done though a degree-scaled mean aggregation and
    works on nodes with continuous attributes:

    .. math::
        \mathbf{x}^{\prime}_i = \frac{1}{2}\big(\mathbf{x}_i +
        \frac{1}{\textrm{deg}(i)}
        \sum_{j \in \mathcal{N}(i)} e_{j,i} \cdot \mathbf{x}_j \big)

    where :math:`e_{j,i}` denotes the edge weight from source node :obj:`j` to
    target node :obj:`i` (default: :obj:`1`)

    Args:
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.MessagePassing`.

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F)` or
          :math:`((|\mathcal{V_s}|, F), (|\mathcal{V_t}|, F))` if bipartite,
          edge indices :math:`(2, |\mathcal{E}|)`,
          edge weights :math:`(|\mathcal{E}|)` *(optional)*
        - **output:** node features :math:`(|\mathcal{V}|, F)` or
          :math:`(|\mathcal{V}_t|, F)` if bipartite
    """
    def __init__(self, **kwargs):
        super().__init__(aggr='add', **kwargs)

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Tensor,
                edge_weight: OptTensor = None, size: Size = None) -> Tensor:

        if isinstance(x, Tensor):
            x: OptPairTensor = (x, x)

        if edge_weight is None:
            edge_weight = x[0].new_ones(edge_index.size(1))

        # propagate_type: (x: OptPairTensor, edge_weight: Tensor)
        out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
                             size=size)

        deg = scatter(edge_weight, edge_index[1], 0, out.size(0), reduce='sum')
        deg_inv = 1. / deg
        deg_inv.masked_fill_(deg_inv == float('inf'), 0)
        out = deg_inv.view(-1, 1) * out

        x_dst = x[1]
        if x_dst is not None:
            out = 0.5 * (x_dst + out)

        return out

    def message(self, x_j: Tensor, edge_weight: Tensor) -> Tensor:
        return edge_weight.view(-1, 1) * x_j
pytorch/torch_geometric/nn/conv/x_conv.py000066400000000000000000000135031444262304300211550ustar00rootroot00000000000000from math import ceil
from typing import Optional

import torch
from torch import Tensor
from torch.nn import ELU
from torch.nn import BatchNorm1d as BN
from torch.nn import Conv1d
from torch.nn import Linear as L
from torch.nn import Sequential as S

from torch_geometric.nn import Reshape
from torch_geometric.nn.inits import reset

try:
    from torch_cluster import knn_graph
except ImportError:
    knn_graph = None


class XConv(torch.nn.Module):
    r"""The convolutional operator on :math:`\mathcal{X}`-transformed points
    from the `"PointCNN: Convolution On X-Transformed Points"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \mathrm{Conv}\left(\mathbf{K},
        \gamma_{\mathbf{\Theta}}(\mathbf{P}_i - \mathbf{p}_i) \times
        \left( h_\mathbf{\Theta}(\mathbf{P}_i - \mathbf{p}_i) \, \Vert \,
        \mathbf{x}_i \right) \right),

    where :math:`\mathbf{K}` and :math:`\mathbf{P}_i` denote the trainable
    filter and neighboring point positions of :math:`\mathbf{x}_i`,
    respectively.
    :math:`\gamma_{\mathbf{\Theta}}` and :math:`h_{\mathbf{\Theta}}` describe
    neural networks, *i.e.* MLPs, where :math:`h_{\mathbf{\Theta}}`
    individually lifts each point into a higher-dimensional space, and
    :math:`\gamma_{\mathbf{\Theta}}` computes the :math:`\mathcal{X}`-
    transformation matrix based on *all* points in a neighborhood.

    Args:
        in_channels (int): Size of each input sample.
        out_channels (int): Size of each output sample.
        dim (int): Point cloud dimensionality.
        kernel_size (int): Size of the convolving kernel, *i.e.* number of
            neighbors including self-loops.
        hidden_channels (int, optional): Output size of
            :math:`h_{\mathbf{\Theta}}`, *i.e.* dimensionality of lifted
            points. If set to :obj:`None`, will be automatically set to
            :obj:`in_channels / 4`. (default: :obj:`None`)
        dilation (int, optional): The factor by which the neighborhood is
            extended, from which :obj:`kernel_size` neighbors are then
            uniformly sampled. Can be interpreted as the dilation rate of
            classical convolutional operators. (default: :obj:`1`)
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        num_workers (int): Number of workers to use for k-NN computation.
            Has no effect in case :obj:`batch` is not :obj:`None`, or the input
            lies on the GPU. (default: :obj:`1`)

    Shapes:
        - **input:**
          node features :math:`(|\mathcal{V}|, F_{in})`,
          positions :math:`(|\mathcal{V}|, D)`,
          batch vector :math:`(|\mathcal{V}|)` *(optional)*
        - **output:**
          node features :math:`(|\mathcal{V}|, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int, dim: int,
                 kernel_size: int, hidden_channels: Optional[int] = None,
                 dilation: int = 1, bias: bool = True, num_workers: int = 1):
        super().__init__()

        if knn_graph is None:
            raise ImportError('`XConv` requires `torch-cluster`.')

        self.in_channels = in_channels
        if hidden_channels is None:
            hidden_channels = in_channels // 4
        assert hidden_channels > 0
        self.hidden_channels = hidden_channels
        self.out_channels = out_channels
        self.dim = dim
        self.kernel_size = kernel_size
        self.dilation = dilation
        self.num_workers = num_workers

        C_in, C_delta, C_out = in_channels, hidden_channels, out_channels
        D, K = dim, kernel_size

        self.mlp1 = S(
            L(dim, C_delta),
            ELU(),
            BN(C_delta),
            L(C_delta, C_delta),
            ELU(),
            BN(C_delta),
            Reshape(-1, K, C_delta),
        )

        self.mlp2 = S(
            L(D * K, K**2),
            ELU(),
            BN(K**2),
            Reshape(-1, K, K),
            Conv1d(K, K**2, K, groups=K),
            ELU(),
            BN(K**2),
            Reshape(-1, K, K),
            Conv1d(K, K**2, K, groups=K),
            BN(K**2),
            Reshape(-1, K, K),
        )

        C_in = C_in + C_delta
        depth_multiplier = int(ceil(C_out / C_in))
        self.conv = S(
            Conv1d(C_in, C_in * depth_multiplier, K, groups=C_in),
            Reshape(-1, C_in * depth_multiplier),
            L(C_in * depth_multiplier, C_out, bias=bias),
        )

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset(self.mlp1)
        reset(self.mlp2)
        reset(self.conv)

    def forward(self, x: Tensor, pos: Tensor, batch: Optional[Tensor] = None):
        r"""Runs the forward pass of the module."""
        pos = pos.unsqueeze(-1) if pos.dim() == 1 else pos
        (N, D), K = pos.size(), self.kernel_size

        edge_index = knn_graph(pos, K * self.dilation, batch, loop=True,
                               flow='target_to_source',
                               num_workers=self.num_workers)

        if self.dilation > 1:
            edge_index = edge_index[:, ::self.dilation]

        row, col = edge_index[0], edge_index[1]

        pos = pos[col] - pos[row]

        x_star = self.mlp1(pos)
        if x is not None:
            x = x.unsqueeze(-1) if x.dim() == 1 else x
            x = x[col].view(N, K, self.in_channels)
            x_star = torch.cat([x_star, x], dim=-1)
        x_star = x_star.transpose(1, 2).contiguous()

        transform_matrix = self.mlp2(pos.view(N, K * D))

        x_transformed = torch.matmul(x_star, transform_matrix)

        out = self.conv(x_transformed)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/data_parallel.py000066400000000000000000000075701444262304300215100ustar00rootroot00000000000000import logging
from itertools import chain

import torch

from torch_geometric.data import Batch


class DataParallel(torch.nn.DataParallel):
    r"""Implements data parallelism at the module level.

    This container parallelizes the application of the given :attr:`module` by
    splitting a list of :class:`torch_geometric.data.Data` objects and copying
    them as :class:`torch_geometric.data.Batch` objects to each device.
    In the forward pass, the module is replicated on each device, and each
    replica handles a portion of the input.
    During the backwards pass, gradients from each replica are summed into the
    original module.

    The batch size should be larger than the number of GPUs used.

    The parallelized :attr:`module` must have its parameters and buffers on
    :obj:`device_ids[0]`.

    .. note::

        You need to use the :class:`torch_geometric.loader.DataListLoader` for
        this module.

    Args:
        module (Module): Module to be parallelized.
        device_ids (list of int or torch.device): CUDA devices.
            (default: all devices)
        output_device (int or torch.device): Device location of output.
            (default: :obj:`device_ids[0]`)
        follow_batch (list or tuple, optional): Creates assignment batch
            vectors for each key in the list. (default: :obj:`None`)
        exclude_keys (list or tuple, optional): Will exclude each key in the
            list. (default: :obj:`None`)
    """
    def __init__(self, module, device_ids=None, output_device=None,
                 follow_batch=None, exclude_keys=None):
        super().__init__(module, device_ids, output_device)
        self.src_device = torch.device(f'cuda:{self.device_ids[0]}')
        self.follow_batch = follow_batch or []
        self.exclude_keys = exclude_keys or []

    def forward(self, data_list):
        """"""
        if len(data_list) == 0:
            logging.warning('DataParallel received an empty data list, which '
                            'may result in unexpected behavior.')
            return None

        if not self.device_ids or len(self.device_ids) == 1:  # Fallback
            data = Batch.from_data_list(
                data_list, follow_batch=self.follow_batch,
                exclude_keys=self.exclude_keys).to(self.src_device)
            return self.module(data)

        for t in chain(self.module.parameters(), self.module.buffers()):
            if t.device != self.src_device:
                raise RuntimeError(
                    f"Module must have its parameters and buffers on device "
                    f"'{self.src_device}' but found one of them on device "
                    f"'{t.device}'")

        inputs = self.scatter(data_list, self.device_ids)
        replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
        outputs = self.parallel_apply(replicas, inputs, None)
        return self.gather(outputs, self.output_device)

    def scatter(self, data_list, device_ids):
        num_devices = min(len(device_ids), len(data_list))

        count = torch.tensor([data.num_nodes for data in data_list])
        cumsum = count.cumsum(0)
        cumsum = torch.cat([cumsum.new_zeros(1), cumsum], dim=0)
        device_id = num_devices * cumsum.to(torch.float) / cumsum[-1].item()
        device_id = (device_id[:-1] + device_id[1:]) / 2.0
        device_id = device_id.to(torch.long)  # round.
        split = device_id.bincount().cumsum(0)
        split = torch.cat([split.new_zeros(1), split], dim=0)
        split = torch.unique(split, sorted=True)
        split = split.tolist()

        return [
            Batch.from_data_list(data_list[split[i]:split[i + 1]],
                                 follow_batch=self.follow_batch,
                                 exclude_keys=self.exclude_keys).to(
                                     torch.device(f'cuda:{device_ids[i]}'))
            for i in range(len(split) - 1)
        ]
pytorch/torch_geometric/nn/dense/000077500000000000000000000000001444262304300174365ustar00rootroot00000000000000pytorch/torch_geometric/nn/dense/__init__.py000066400000000000000000000013101444262304300215420ustar00rootroot00000000000000from .linear import Linear, HeteroLinear, HeteroDictLinear
from .dense_gat_conv import DenseGATConv
from .dense_sage_conv import DenseSAGEConv
from .dense_gcn_conv import DenseGCNConv
from .dense_graph_conv import DenseGraphConv
from .dense_gin_conv import DenseGINConv
from .diff_pool import dense_diff_pool
from .mincut_pool import dense_mincut_pool
from .dmon_pool import DMoNPooling

__all__ = [
    'Linear',
    'HeteroLinear',
    'HeteroDictLinear',
    'DenseGCNConv',
    'DenseGINConv',
    'DenseGraphConv',
    'DenseSAGEConv',
    'DenseGATConv',
    'dense_diff_pool',
    'dense_mincut_pool',
    'DMoNPooling',
]

lin_classes = __all__[:3]
conv_classes = __all__[3:8]
pool_classes = __all__[8:]
pytorch/torch_geometric/nn/dense/dense_gat_conv.py000066400000000000000000000102041444262304300227630ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import glorot, zeros


class DenseGATConv(torch.nn.Module):
    r"""See :class:`torch_geometric.nn.conv.GATConv`."""
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        heads: int = 1,
        concat: bool = True,
        negative_slope: float = 0.2,
        dropout: float = 0.0,
        bias: bool = True,
    ):
        # TODO Add support for edge features.
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.concat = concat
        self.negative_slope = negative_slope
        self.dropout = dropout

        self.lin = Linear(in_channels, heads * out_channels, bias=False,
                          weight_initializer='glorot')

        # The learnable parameters to compute attention coefficients:
        self.att_src = Parameter(torch.Tensor(1, 1, heads, out_channels))
        self.att_dst = Parameter(torch.Tensor(1, 1, heads, out_channels))

        if bias and concat:
            self.bias = Parameter(torch.Tensor(heads * out_channels))
        elif bias and not concat:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        self.lin.reset_parameters()
        glorot(self.att_src)
        glorot(self.att_dst)
        zeros(self.bias)

    def forward(self, x: Tensor, adj: Tensor, mask: Optional[Tensor] = None,
                add_loop: bool = True):
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
                The adjacency tensor is broadcastable in the batch dimension,
                resulting in a shared adjacency matrix for the complete batch.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
            add_loop (bool, optional): If set to :obj:`False`, the layer will
                not automatically add self-loops to the adjacency matrices.
                (default: :obj:`True`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x  # [B, N, F]
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj  # [B, N, N]

        H, C = self.heads, self.out_channels
        B, N, _ = x.size()

        if add_loop:
            adj = adj.clone()
            idx = torch.arange(N, dtype=torch.long, device=adj.device)
            adj[:, idx, idx] = 1.0

        x = self.lin(x).view(B, N, H, C)  # [B, N, H, C]

        alpha_src = torch.sum(x * self.att_src, dim=-1)  # [B, N, H]
        alpha_dst = torch.sum(x * self.att_dst, dim=-1)  # [B, N, H]

        alpha = alpha_src.unsqueeze(1) + alpha_dst.unsqueeze(2)  # [B, N, N, H]

        # Weighted and masked softmax:
        alpha = F.leaky_relu(alpha, self.negative_slope)
        alpha = alpha.masked_fill(adj.unsqueeze(-1) == 0, float('-inf'))
        alpha = alpha.softmax(dim=2)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)

        out = torch.matmul(alpha.movedim(3, 1), x.movedim(2, 1))
        out = out.movedim(1, 2)  # [B,N,H,C]

        if self.concat:
            out = out.reshape(B, N, H * C)
        else:
            out = out.mean(dim=2)

        if self.bias is not None:
            out = out + self.bias

        if mask is not None:
            out = out * mask.view(-1, N, 1).to(x.dtype)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads})')
pytorch/torch_geometric/nn/dense/dense_gcn_conv.py000066400000000000000000000056551444262304300227750ustar00rootroot00000000000000import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.inits import zeros
from torch_geometric.typing import OptTensor


class DenseGCNConv(torch.nn.Module):
    r"""See :class:`torch_geometric.nn.conv.GCNConv`."""
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        improved: bool = False,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.improved = improved

        self.lin = Linear(in_channels, out_channels, bias=False,
                          weight_initializer='glorot')

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin.reset_parameters()
        zeros(self.bias)

    def forward(self, x: Tensor, adj: Tensor, mask: OptTensor = None,
                add_loop: bool = True) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
                The adjacency tensor is broadcastable in the batch dimension,
                resulting in a shared adjacency matrix for the complete batch.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
            add_loop (bool, optional): If set to :obj:`False`, the layer will
                not automatically add self-loops to the adjacency matrices.
                (default: :obj:`True`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
        B, N, _ = adj.size()

        if add_loop:
            adj = adj.clone()
            idx = torch.arange(N, dtype=torch.long, device=adj.device)
            adj[:, idx, idx] = 1 if not self.improved else 2

        out = self.lin(x)
        deg_inv_sqrt = adj.sum(dim=-1).clamp(min=1).pow(-0.5)

        adj = deg_inv_sqrt.unsqueeze(-1) * adj * deg_inv_sqrt.unsqueeze(-2)
        out = torch.matmul(adj, out)

        if self.bias is not None:
            out = out + self.bias

        if mask is not None:
            out = out * mask.view(B, N, 1).to(x.dtype)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/dense/dense_gin_conv.py000066400000000000000000000044651444262304300230010ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Module

from torch_geometric.nn.inits import reset


class DenseGINConv(torch.nn.Module):
    r"""See :class:`torch_geometric.nn.conv.GINConv`."""
    def __init__(
        self,
        nn: Module,
        eps: float = 0.0,
        train_eps: bool = False,
    ):
        super().__init__()

        self.nn = nn
        self.initial_eps = eps
        if train_eps:
            self.eps = torch.nn.Parameter(torch.Tensor([eps]))
        else:
            self.register_buffer('eps', torch.Tensor([eps]))
        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset(self.nn)
        self.eps.data.fill_(self.initial_eps)

    def forward(self, x: Tensor, adj: Tensor, mask: Optional[Tensor] = None,
                add_loop: bool = True) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
                The adjacency tensor is broadcastable in the batch dimension,
                resulting in a shared adjacency matrix for the complete batch.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
            add_loop (bool, optional): If set to :obj:`False`, the layer will
                not automatically add self-loops to the adjacency matrices.
                (default: :obj:`True`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
        B, N, _ = adj.size()

        out = torch.matmul(adj, x)
        if add_loop:
            out = (1 + self.eps) * x + out

        out = self.nn(out)

        if mask is not None:
            out = out * mask.view(B, N, 1).to(x.dtype)

        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(nn={self.nn})'
pytorch/torch_geometric/nn/dense/dense_graph_conv.py000066400000000000000000000052611444262304300233200ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Linear


class DenseGraphConv(torch.nn.Module):
    r"""See :class:`torch_geometric.nn.conv.GraphConv`."""
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        aggr: str = 'add',
        bias: bool = True,
    ):
        assert aggr in ['add', 'mean', 'max']
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.aggr = aggr

        self.lin_rel = Linear(in_channels, out_channels, bias=bias)
        self.lin_root = Linear(in_channels, out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin_rel.reset_parameters()
        self.lin_root.reset_parameters()

    def forward(self, x: Tensor, adj: Tensor,
                mask: Optional[Tensor] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
                The adjacency tensor is broadcastable in the batch dimension,
                resulting in a shared adjacency matrix for the complete batch.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
        B, N, C = x.size()

        if self.aggr == 'add':
            out = torch.matmul(adj, x)
        elif self.aggr == 'mean':
            out = torch.matmul(adj, x)
            out = out / adj.sum(dim=-1, keepdim=True).clamp_(min=1)
        elif self.aggr == 'max':
            out = x.unsqueeze(-2).repeat(1, 1, N, 1)
            adj = adj.unsqueeze(-1).expand(B, N, N, C)
            out[adj == 0] = float('-inf')
            out = out.max(dim=-3)[0]
            out[out == float('-inf')] = 0.
        else:
            raise NotImplementedError

        out = self.lin_rel(out)
        out = out + self.lin_root(x)

        if mask is not None:
            out = out * mask.view(-1, N, 1).to(x.dtype)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/dense/dense_sage_conv.py000066400000000000000000000051421444262304300231340ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear

from torch_geometric.typing import OptTensor


class DenseSAGEConv(torch.nn.Module):
    r"""See :class:`torch_geometric.nn.conv.SAGEConv`.

    .. note::

        :class:`~torch_geometric.nn.dense.DenseSAGEConv` expects to work on
        binary adjacency matrices.
        If you want to make use of weighted dense adjacency matrices, please
        use :class:`torch_geometric.nn.dense.DenseGraphConv` instead.
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        normalize: bool = False,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.normalize = normalize

        self.lin_rel = Linear(in_channels, out_channels, bias=False)
        self.lin_root = Linear(in_channels, out_channels, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin_rel.reset_parameters()
        self.lin_root.reset_parameters()

    def forward(self, x: Tensor, adj: Tensor,
                mask: OptTensor = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
                The adjacency tensor is broadcastable in the batch dimension,
                resulting in a shared adjacency matrix for the complete batch.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
        B, N, _ = adj.size()

        out = torch.matmul(adj, x)
        out = out / adj.sum(dim=-1, keepdim=True).clamp(min=1)
        out = self.lin_rel(out) + self.lin_root(x)

        if self.normalize:
            out = F.normalize(out, p=2.0, dim=-1)

        if mask is not None:
            out = out * mask.view(B, N, 1).to(x.dtype)

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels})')
pytorch/torch_geometric/nn/dense/diff_pool.py000066400000000000000000000057521444262304300217620ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor


def dense_diff_pool(
    x: Tensor,
    adj: Tensor,
    s: Tensor,
    mask: Optional[Tensor] = None,
    normalize: bool = True,
) -> Tuple[Tensor, Tensor, Tensor, Tensor]:
    r"""The differentiable pooling operator from the `"Hierarchical Graph
    Representation Learning with Differentiable Pooling"
    `_ paper

    .. math::
        \mathbf{X}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{X}

        \mathbf{A}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{A} \cdot \mathrm{softmax}(\mathbf{S})

    based on dense learned assignments :math:`\mathbf{S} \in \mathbb{R}^{B
    \times N \times C}`.
    Returns the pooled node feature matrix, the coarsened adjacency matrix and
    two auxiliary objectives: (1) The link prediction loss

    .. math::
        \mathcal{L}_{LP} = {\| \mathbf{A} -
        \mathrm{softmax}(\mathbf{S}) {\mathrm{softmax}(\mathbf{S})}^{\top}
        \|}_F,

    and (2) the entropy regularization

    .. math::
        \mathcal{L}_E = \frac{1}{N} \sum_{n=1}^N H(\mathbf{S}_n).

    Args:
        x (torch.Tensor): Node feature tensor
            :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
            batch-size :math:`B`, (maximum) number of nodes :math:`N` for
            each graph, and feature dimension :math:`F`.
        adj (torch.Tensor): Adjacency tensor
            :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
        s (torch.Tensor): Assignment tensor
            :math:`\mathbf{S} \in \mathbb{R}^{B \times N \times C}`
            with number of clusters :math:`C`.
            The softmax does not have to be applied before-hand, since it is
            executed within this method.
        mask (torch.Tensor, optional): Mask matrix
            :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
            the valid nodes for each graph. (default: :obj:`None`)
        normalize (bool, optional): If set to :obj:`False`, the link
            prediction loss is not divided by :obj:`adj.numel()`.
            (default: :obj:`True`)

    :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`,
        :class:`torch.Tensor`, :class:`torch.Tensor`)
    """
    x = x.unsqueeze(0) if x.dim() == 2 else x
    adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
    s = s.unsqueeze(0) if s.dim() == 2 else s

    batch_size, num_nodes, _ = x.size()

    s = torch.softmax(s, dim=-1)

    if mask is not None:
        mask = mask.view(batch_size, num_nodes, 1).to(x.dtype)
        x, s = x * mask, s * mask

    out = torch.matmul(s.transpose(1, 2), x)
    out_adj = torch.matmul(torch.matmul(s.transpose(1, 2), adj), s)

    link_loss = adj - torch.matmul(s, s.transpose(1, 2))
    link_loss = torch.norm(link_loss, p=2)
    if normalize is True:
        link_loss = link_loss / adj.numel()

    ent_loss = (-s * torch.log(s + 1e-15)).sum(dim=-1).mean()

    return out, out_adj, link_loss, ent_loss
pytorch/torch_geometric/nn/dense/dmon_pool.py000066400000000000000000000130471444262304300220030ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.dense.mincut_pool import _rank3_trace

EPS = 1e-15


class DMoNPooling(torch.nn.Module):
    r"""The spectral modularity pooling operator from the `"Graph Clustering
    with Graph Neural Networks" `_ paper

    .. math::
        \mathbf{X}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{X}

        \mathbf{A}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{A} \cdot \mathrm{softmax}(\mathbf{S})

    based on dense learned assignments :math:`\mathbf{S} \in \mathbb{R}^{B
    \times N \times C}`.
    Returns the learned cluster assignment matrix, the pooled node feature
    matrix, the coarsened symmetrically normalized adjacency matrix, and three
    auxiliary objectives: (1) The spectral loss

    .. math::
        \mathcal{L}_s = - \frac{1}{2m}
        \cdot{\mathrm{Tr}(\mathbf{S}^{\top} \mathbf{B} \mathbf{S})}

    where :math:`\mathbf{B}` is the modularity matrix, (2) the orthogonality
    loss

    .. math::
        \mathcal{L}_o = {\left\| \frac{\mathbf{S}^{\top} \mathbf{S}}
        {{\|\mathbf{S}^{\top} \mathbf{S}\|}_F} -\frac{\mathbf{I}_C}{\sqrt{C}}
        \right\|}_F

    where :math:`C` is the number of clusters, and (3) the cluster loss

    .. math::
        \mathcal{L}_c = \frac{\sqrt{C}}{n}
        {\left\|\sum_i\mathbf{C_i}^{\top}\right\|}_F - 1.

    .. note::

        For an example of using :class:`DMoNPooling`, see
        `examples/proteins_dmon_pool.py
        `_.

    Args:
        channels (int or List[int]): Size of each input sample. If given as a
            list, will construct an MLP based on the given feature sizes.
        k (int): The number of clusters.
        dropout (float, optional): Dropout probability. (default: :obj:`0.0`)
    """
    def __init__(self, channels: Union[int, List[int]], k: int,
                 dropout: float = 0.0):
        super().__init__()

        if isinstance(channels, int):
            channels = [channels]

        from torch_geometric.nn.models.mlp import MLP
        self.mlp = MLP(channels + [k], act='selu', norm=None)

        self.dropout = dropout

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.mlp.reset_parameters()

    def forward(
        self,
        x: Tensor,
        adj: Tensor,
        mask: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): Node feature tensor
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
                batch-size :math:`B`, (maximum) number of nodes :math:`N` for
                each graph, and feature dimension :math:`F`.
                Note that the cluster assignment matrix
                :math:`\mathbf{S} \in \mathbb{R}^{B \times N \times C}` is
                being created within this method.
            adj (torch.Tensor): Adjacency tensor
                :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
            mask (torch.Tensor, optional): Mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
                the valid nodes for each graph. (default: :obj:`None`)

        :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`,
            :class:`torch.Tensor`, :class:`torch.Tensor`,
            :class:`torch.Tensor`, :class:`torch.Tensor`)
        """
        x = x.unsqueeze(0) if x.dim() == 2 else x
        adj = adj.unsqueeze(0) if adj.dim() == 2 else adj

        s = self.mlp(x)
        s = F.dropout(s, self.dropout, training=self.training)
        s = torch.softmax(s, dim=-1)

        (batch_size, num_nodes, _), k = x.size(), s.size(-1)

        if mask is not None:
            mask = mask.view(batch_size, num_nodes, 1).to(x.dtype)
            x, s = x * mask, s * mask

        out = F.selu(torch.matmul(s.transpose(1, 2), x))
        out_adj = torch.matmul(torch.matmul(s.transpose(1, 2), adj), s)

        # Spectral loss:
        degrees = torch.einsum('ijk->ik', adj).transpose(0, 1)
        m = torch.einsum('ij->', degrees)

        ca = torch.matmul(s.transpose(1, 2), degrees)
        cb = torch.matmul(degrees.transpose(0, 1), s)

        normalizer = torch.matmul(ca, cb) / 2 / m
        decompose = out_adj - normalizer
        spectral_loss = -_rank3_trace(decompose) / 2 / m
        spectral_loss = torch.mean(spectral_loss)

        # Orthogonality regularization:
        ss = torch.matmul(s.transpose(1, 2), s)
        i_s = torch.eye(k).type_as(ss)
        ortho_loss = torch.norm(
            ss / torch.norm(ss, dim=(-1, -2), keepdim=True) -
            i_s / torch.norm(i_s), dim=(-1, -2))
        ortho_loss = torch.mean(ortho_loss)

        # Cluster loss:
        cluster_loss = torch.norm(torch.einsum(
            'ijk->ij', ss)) / adj.size(1) * torch.norm(i_s) - 1

        # Fix and normalize coarsened adjacency matrix:
        ind = torch.arange(k, device=out_adj.device)
        out_adj[:, ind, ind] = 0
        d = torch.einsum('ijk->ij', out_adj)
        d = torch.sqrt(d)[:, None] + EPS
        out_adj = (out_adj / d) / d.transpose(1, 2)

        return s, out, out_adj, spectral_loss, ortho_loss, cluster_loss

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.mlp.in_channels}, '
                f'num_clusters={self.mlp.out_channels})')
pytorch/torch_geometric/nn/dense/linear.py000066400000000000000000000413651444262304300212730ustar00rootroot00000000000000import copy
import math
from typing import Any, Dict, Optional, Union

import torch
import torch.nn.functional as F
from torch import Tensor, nn
from torch.nn.parameter import Parameter

import torch_geometric.typing
from torch_geometric.nn import inits
from torch_geometric.typing import pyg_lib
from torch_geometric.utils import index_sort
from torch_geometric.utils.hetero import segmatmul_heuristic
from torch_geometric.utils.sparse import index2ptr


def is_uninitialized_parameter(x: Any) -> bool:
    if not hasattr(nn.parameter, 'UninitializedParameter'):
        return False
    return isinstance(x, nn.parameter.UninitializedParameter)


def reset_weight_(weight: Tensor, in_channels: int,
                  initializer: Optional[str] = None) -> Tensor:
    if in_channels <= 0:
        pass
    elif initializer == 'glorot':
        inits.glorot(weight)
    elif initializer == 'uniform':
        bound = 1.0 / math.sqrt(in_channels)
        torch.nn.init.uniform_(weight.data, -bound, bound)
    elif initializer == 'kaiming_uniform':
        inits.kaiming_uniform(weight, fan=in_channels, a=math.sqrt(5))
    elif initializer is None:
        inits.kaiming_uniform(weight, fan=in_channels, a=math.sqrt(5))
    else:
        raise RuntimeError(f"Weight initializer '{initializer}' not supported")

    return weight


def reset_bias_(bias: Optional[Tensor], in_channels: int,
                initializer: Optional[str] = None) -> Optional[Tensor]:
    if bias is None or in_channels <= 0:
        pass
    elif initializer == 'zeros':
        inits.zeros(bias)
    elif initializer is None:
        inits.uniform(in_channels, bias)
    else:
        raise RuntimeError(f"Bias initializer '{initializer}' not supported")

    return bias


class Linear(torch.nn.Module):
    r"""Applies a linear tranformation to the incoming data

    .. math::
        \mathbf{x}^{\prime} = \mathbf{x} \mathbf{W}^{\top} + \mathbf{b}

    similar to :class:`torch.nn.Linear`.
    It supports lazy initialization and customizable weight and bias
    initialization.

    Args:
        in_channels (int): Size of each input sample. Will be initialized
            lazily in case it is given as :obj:`-1`.
        out_channels (int): Size of each output sample.
        bias (bool, optional): If set to :obj:`False`, the layer will not learn
            an additive bias. (default: :obj:`True`)
        weight_initializer (str, optional): The initializer for the weight
            matrix (:obj:`"glorot"`, :obj:`"uniform"`, :obj:`"kaiming_uniform"`
            or :obj:`None`).
            If set to :obj:`None`, will match default weight initialization of
            :class:`torch.nn.Linear`. (default: :obj:`None`)
        bias_initializer (str, optional): The initializer for the bias vector
            (:obj:`"zeros"` or :obj:`None`).
            If set to :obj:`None`, will match default bias initialization of
            :class:`torch.nn.Linear`. (default: :obj:`None`)

    Shapes:
        - **input:** features :math:`(*, F_{in})`
        - **output:** features :math:`(*, F_{out})`
    """
    def __init__(self, in_channels: int, out_channels: int, bias: bool = True,
                 weight_initializer: Optional[str] = None,
                 bias_initializer: Optional[str] = None):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.weight_initializer = weight_initializer
        self.bias_initializer = bias_initializer

        if in_channels > 0:
            self.weight = Parameter(torch.Tensor(out_channels, in_channels))
        else:
            self.weight = nn.parameter.UninitializedParameter()
            self._hook = self.register_forward_pre_hook(
                self.initialize_parameters)

        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self._load_hook = self._register_load_state_dict_pre_hook(
            self._lazy_load_hook)

        self.reset_parameters()

    def __deepcopy__(self, memo):
        out = Linear(self.in_channels, self.out_channels, self.bias
                     is not None, self.weight_initializer,
                     self.bias_initializer)
        if self.in_channels > 0:
            out.weight = copy.deepcopy(self.weight, memo)
        if self.bias is not None:
            out.bias = copy.deepcopy(self.bias, memo)
        return out

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset_weight_(self.weight, self.in_channels, self.weight_initializer)
        reset_bias_(self.bias, self.in_channels, self.bias_initializer)

    def forward(self, x: Tensor) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The input features.
        """
        return F.linear(x, self.weight, self.bias)

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        if is_uninitialized_parameter(self.weight):
            self.in_channels = input[0].size(-1)
            self.weight.materialize((self.out_channels, self.in_channels))
            self.reset_parameters()
        self._hook.remove()
        delattr(self, '_hook')

    def _save_to_state_dict(self, destination, prefix, keep_vars):
        if (is_uninitialized_parameter(self.weight)
                or torch.onnx.is_in_onnx_export()):
            destination[prefix + 'weight'] = self.weight
        else:
            destination[prefix + 'weight'] = self.weight.detach()
        if self.bias is not None:
            if torch.onnx.is_in_onnx_export():
                destination[prefix + 'bias'] = self.bias
            else:
                destination[prefix + 'bias'] = self.bias.detach()

    def _lazy_load_hook(self, state_dict, prefix, local_metadata, strict,
                        missing_keys, unexpected_keys, error_msgs):

        weight = state_dict.get(prefix + 'weight', None)

        if weight is not None and is_uninitialized_parameter(weight):
            self.in_channels = -1
            self.weight = nn.parameter.UninitializedParameter()
            if not hasattr(self, '_hook'):
                self._hook = self.register_forward_pre_hook(
                    self.initialize_parameters)

        elif weight is not None and is_uninitialized_parameter(self.weight):
            self.in_channels = weight.size(-1)
            self.weight.materialize((self.out_channels, self.in_channels))
            if hasattr(self, '_hook'):
                self._hook.remove()
                delattr(self, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, bias={self.bias is not None})')


class HeteroLinear(torch.nn.Module):
    r"""Applies separate linear tranformations to the incoming data according
    to types

    .. math::
        \mathbf{x}^{\prime}_{\kappa} = \mathbf{x}_{\kappa}
        \mathbf{W}^{\top}_{\kappa} + \mathbf{b}_{\kappa}

    for type :math:`\kappa`.
    It supports lazy initialization and customizable weight and bias
    initialization.

    Args:
        in_channels (int): Size of each input sample. Will be initialized
            lazily in case it is given as :obj:`-1`.
        out_channels (int): Size of each output sample.
        num_types (int): The number of types.
        is_sorted (bool, optional): If set to :obj:`True`, assumes that
            :obj:`type_vec` is sorted. This avoids internal re-sorting of the
            data and can improve runtime and memory efficiency.
            (default: :obj:`False`)
        use_segmm (bool, optional): If set to :obj:`True` and :obj:`pyg-lib` is
            installed, this module will use the fused :obj:`segment_matmul`
            kernel to parallelize the linear transformation across types. If
            set to :obj:`False`, :obj:`segment_matmul` will not be used. If
            left as :obj:`None` and :obj:`pyg-lib` is installed, the module
            will determine heuristically whether to use :obj:`segment_matmul`.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.Linear`.

    Shapes:
        - **input:**
          features :math:`(*, F_{in})`,
          type vector :math:`(*)`
        - **output:** features :math:`(*, F_{out})`
    """
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        num_types: int,
        is_sorted: bool = False,
        use_segmm: Optional[bool] = None,
        **kwargs,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_types = num_types
        self.is_sorted = is_sorted
        self.use_segmm: int = -1 if use_segmm is None else int(use_segmm)
        self.kwargs = kwargs

        if self.in_channels == -1:
            self.weight = nn.parameter.UninitializedParameter()
            self._hook = self.register_forward_pre_hook(
                self.initialize_parameters)
        else:
            self.weight = torch.nn.Parameter(
                torch.Tensor(num_types, in_channels, out_channels))
        if kwargs.get('bias', True):
            self.bias = Parameter(torch.Tensor(num_types, out_channels))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset_weight_(self.weight, self.in_channels,
                      self.kwargs.get('weight_initializer', None))
        reset_weight_(self.bias, self.in_channels,
                      self.kwargs.get('bias_initializer', None))

    def forward(self, x: Tensor, type_vec: Tensor) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The input features.
            type_vec (torch.Tensor): A vector that maps each entry to a type.
        """
        if (torch_geometric.typing.WITH_PYG_LIB
                and (self.use_segmm == -1 or bool(self.use_segmm))):
            assert self.weight is not None

            perm: Optional[Tensor] = None
            if not self.is_sorted:
                if (type_vec[1:] < type_vec[:-1]).any():
                    type_vec, perm = index_sort(type_vec, self.num_types)
                    x = x[perm]

            type_vec_ptr = index2ptr(type_vec, self.num_types)
            if self.use_segmm == -1:
                self.use_segmm = segmatmul_heuristic(x, type_vec_ptr,
                                                     self.weight)
            out = pyg_lib.ops.segment_matmul(x, type_vec_ptr, self.weight)
            if self.bias is not None:
                out += self.bias[type_vec]

            if perm is not None:  # Restore original order (if necessary).
                out_unsorted = torch.empty_like(out)
                out_unsorted[perm] = out
                out = out_unsorted
        else:
            out = x.new_empty(x.size(0), self.out_channels)
            for i in range(self.num_types):
                mask = type_vec == i
                if mask.numel() == 0:
                    continue
                subset_out = F.linear(x[mask], self.weight[i].T)
                # The data type may have changed with mixed precision:
                out[mask] = subset_out.to(out.dtype)

            if self.bias is not None:
                out += self.bias[type_vec]
        return out

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        if is_uninitialized_parameter(self.weight):
            self.in_channels = input[0].size(-1)
            self.weight.materialize(
                (self.num_types, self.in_channels, self.out_channels))
            self.reset_parameters()
        self._hook.remove()
        delattr(self, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_types={self.num_types}, '
                f'bias={self.kwargs.get("bias", True)})')


class HeteroDictLinear(torch.nn.Module):
    r"""Applies separate linear tranformations to the incoming data dictionary

    .. math::
        \mathbf{x}^{\prime}_{\kappa} = \mathbf{x}_{\kappa}
        \mathbf{W}^{\top}_{\kappa} + \mathbf{b}_{\kappa}

    for key :math:`\kappa`.
    It supports lazy initialization and customizable weight and bias
    initialization.

    Args:
        in_channels (int or Dict[Any, int]): Size of each input sample. If
            passed an integer, :obj:`types` will be a mandatory argument.
            initialized lazily in case it is given as :obj:`-1`.
        out_channels (int): Size of each output sample.
        types (List[Any], optional): The keys of the input dictionary.
            (default: :obj:`None`)
        use_segmm (bool, optional): If set to :obj:`True` and :obj:`pyg-lib` is
            installed, this module will use the fused :obj:`segment_matmul`
            kernel to parallelize the linear transformation across types. If
            set to :obj:`False`, :obj:`segment_matmul` will not be used. If
            left as :obj:`None` and :obj:`pyg-lib` is installed, the module
            will determine heuristically whether to use :obj:`segment_matmul`.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.Linear`.
    """
    def __init__(
        self,
        in_channels: Union[int, Dict[Any, int]],
        out_channels: int,
        types: Optional[Any] = None,
        use_segmm: Optional[bool] = None,
        **kwargs,
    ):
        super().__init__()

        if isinstance(in_channels, dict):
            self.types = list(in_channels.keys())

            if any([i == -1 for i in in_channels.values()]):
                self._hook = self.register_forward_pre_hook(
                    self.initialize_parameters)

            if types is not None and set(self.types) != set(types):
                raise ValueError("The provided 'types' do not match with the "
                                 "keys in the 'in_channels' dictionary")

        else:
            if types is None:
                raise ValueError("Please provide a list of 'types' if passing "
                                 "'in_channels' as an integer")

            if in_channels == -1:
                self._hook = self.register_forward_pre_hook(
                    self.initialize_parameters)

            self.types = types
            in_channels = {node_type: in_channels for node_type in types}

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.use_segmm = use_segmm
        self.kwargs = kwargs

        self.lins = torch.nn.ModuleDict({
            key: Linear(channels, self.out_channels, **kwargs)
            for key, channels in self.in_channels.items()
        })

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for lin in self.lins.values():
            lin.reset_parameters()

    def forward(
        self,
        x_dict: Dict[str, Tensor],
    ) -> Dict[str, Tensor]:
        r"""
        Args:
            x_dict (Dict[Any, torch.Tensor]): A dictionary holding input
                features for each individual type.
        """
        out_dict = {}

        # Only apply fused kernel for more than 10 types, otherwise default
        # back to sequential computation (which is faster for these cases).
        if self.use_segmm is None:
            use_segmm = len(x_dict) >= 10
        else:
            use_segmm = self.use_segmm

        if (torch_geometric.typing.WITH_GMM and not torch.jit.is_scripting()
                and use_segmm):
            xs, weights, biases = [], [], []
            for key, lin in self.lins.items():
                if key in x_dict:
                    xs.append(x_dict[key])
                    weights.append(lin.weight.t())
                    biases.append(lin.bias)
            biases = None if biases[0] is None else biases
            outs = pyg_lib.ops.grouped_matmul(xs, weights, biases)
            for key, out in zip(x_dict.keys(), outs):
                if key in x_dict:
                    out_dict[key] = out
        else:
            for key, lin in self.lins.items():
                if key in x_dict:
                    out_dict[key] = lin(x_dict[key])

        return out_dict

    @torch.no_grad()
    def initialize_parameters(self, module, input):
        for key, x in input[0].items():
            lin = self.lins[key]
            if is_uninitialized_parameter(lin.weight):
                self.lins[key].initialize_parameters(None, x)
        self.reset_parameters()
        self._hook.remove()
        self.in_channels = {key: x.size(-1) for key, x in input[0].items()}
        delattr(self, '_hook')

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, bias={self.kwargs.get("bias", True)})')
pytorch/torch_geometric/nn/dense/mincut_pool.py000066400000000000000000000100171444262304300223370ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor


def dense_mincut_pool(
    x: Tensor,
    adj: Tensor,
    s: Tensor,
    mask: Optional[Tensor] = None,
    temp: float = 1.0,
) -> Tuple[Tensor, Tensor, Tensor, Tensor]:
    r"""The MinCut pooling operator from the `"Spectral Clustering in Graph
    Neural Networks for Graph Pooling" `_
    paper

    .. math::
        \mathbf{X}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{X}

        \mathbf{A}^{\prime} &= {\mathrm{softmax}(\mathbf{S})}^{\top} \cdot
        \mathbf{A} \cdot \mathrm{softmax}(\mathbf{S})

    based on dense learned assignments :math:`\mathbf{S} \in \mathbb{R}^{B
    \times N \times C}`.
    Returns the pooled node feature matrix, the coarsened and symmetrically
    normalized adjacency matrix and two auxiliary objectives: (1) The MinCut
    loss

    .. math::
        \mathcal{L}_c = - \frac{\mathrm{Tr}(\mathbf{S}^{\top} \mathbf{A}
        \mathbf{S})} {\mathrm{Tr}(\mathbf{S}^{\top} \mathbf{D}
        \mathbf{S})}

    where :math:`\mathbf{D}` is the degree matrix, and (2) the orthogonality
    loss

    .. math::
        \mathcal{L}_o = {\left\| \frac{\mathbf{S}^{\top} \mathbf{S}}
        {{\|\mathbf{S}^{\top} \mathbf{S}\|}_F} -\frac{\mathbf{I}_C}{\sqrt{C}}
        \right\|}_F.

    Args:
        x (torch.Tensor): Node feature tensor
            :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`, with
            batch-size :math:`B`, (maximum) number of nodes :math:`N` for
            each graph, and feature dimension :math:`F`.
        adj (torch.Tensor): Adjacency tensor
            :math:`\mathbf{A} \in \mathbb{R}^{B \times N \times N}`.
        s (torch.Tensor): Assignment tensor
            :math:`\mathbf{S} \in \mathbb{R}^{B \times N \times C}`
            with number of clusters :math:`C`.
            The softmax does not have to be applied before-hand, since it is
            executed within this method.
        mask (torch.Tensor, optional): Mask matrix
            :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}` indicating
            the valid nodes for each graph. (default: :obj:`None`)
        temp (float, optional): Temperature parameter for softmax function.
            (default: :obj:`1.0`)

    :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`,
        :class:`torch.Tensor`, :class:`torch.Tensor`)
    """

    x = x.unsqueeze(0) if x.dim() == 2 else x
    adj = adj.unsqueeze(0) if adj.dim() == 2 else adj
    s = s.unsqueeze(0) if s.dim() == 2 else s

    (batch_size, num_nodes, _), k = x.size(), s.size(-1)

    s = torch.softmax(s / temp if temp != 1.0 else s, dim=-1)

    if mask is not None:
        mask = mask.view(batch_size, num_nodes, 1).to(x.dtype)
        x, s = x * mask, s * mask

    out = torch.matmul(s.transpose(1, 2), x)
    out_adj = torch.matmul(torch.matmul(s.transpose(1, 2), adj), s)

    # MinCut regularization.
    mincut_num = _rank3_trace(out_adj)
    d_flat = torch.einsum('ijk->ij', adj)
    d = _rank3_diag(d_flat)
    mincut_den = _rank3_trace(
        torch.matmul(torch.matmul(s.transpose(1, 2), d), s))
    mincut_loss = -(mincut_num / mincut_den)
    mincut_loss = torch.mean(mincut_loss)

    # Orthogonality regularization.
    ss = torch.matmul(s.transpose(1, 2), s)
    i_s = torch.eye(k).type_as(ss)
    ortho_loss = torch.norm(
        ss / torch.norm(ss, dim=(-1, -2), keepdim=True) -
        i_s / torch.norm(i_s), dim=(-1, -2))
    ortho_loss = torch.mean(ortho_loss)

    EPS = 1e-15

    # Fix and normalize coarsened adjacency matrix.
    ind = torch.arange(k, device=out_adj.device)
    out_adj[:, ind, ind] = 0
    d = torch.einsum('ijk->ij', out_adj)
    d = torch.sqrt(d)[:, None] + EPS
    out_adj = (out_adj / d) / d.transpose(1, 2)

    return out, out_adj, mincut_loss, ortho_loss


def _rank3_trace(x: Tensor) -> Tensor:
    return torch.einsum('ijj->i', x)


def _rank3_diag(x: Tensor) -> Tensor:
    eye = torch.eye(x.size(1)).type_as(x)
    out = eye * x.unsqueeze(2).expand(x.size(0), x.size(1), x.size(1))

    return out
pytorch/torch_geometric/nn/encoding.py000066400000000000000000000060651444262304300205070ustar00rootroot00000000000000import math

import torch
from torch import Tensor


class PositionalEncoding(torch.nn.Module):
    r"""The positional encoding scheme from the `"Attention Is All You Need"
    `_ paper

    .. math::

        PE(x)_{2 \cdot i} &= \sin(x / 10000^{2 \cdot i / d})

        PE(x)_{2 \cdot i + 1} &= \cos(x / 10000^{2 \cdot i / d})

    where :math:`x` is the position and :math:`i` is the dimension.

    Args:
        out_channels (int): Size :math:`d` of each output sample.
        base_freq (float, optional): The base frequency of sinusoidal
            functions. (default: :obj:`1e-4`)
        granularity (float, optional): The granularity of the positions. If
            set to smaller value, the encoder will capture more fine-grained
            changes in positions. (default: :obj:`1.0`)
    """
    def __init__(
        self,
        out_channels: int,
        base_freq: float = 1e-4,
        granularity: float = 1.0,
    ):
        super().__init__()

        if out_channels % 2 != 0:
            raise ValueError(f"Cannot use sinusoidal positional encoding with "
                             f"odd 'out_channels' (got {out_channels}).")

        self.out_channels = out_channels
        self.base_freq = base_freq
        self.granularity = granularity

        frequency = torch.logspace(0, 1, out_channels // 2, base_freq)
        self.register_buffer('frequency', frequency)

        self.reset_parameters()

    def reset_parameters(self):
        pass

    def forward(self, x: Tensor) -> Tensor:
        """"""
        x = x / self.granularity if self.granularity != 1.0 else x
        out = x.view(-1, 1) * self.frequency.view(1, -1)
        return torch.cat([torch.sin(out), torch.cos(out)], dim=-1)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.out_channels})'


class TemporalEncoding(torch.nn.Module):
    r"""The time-encoding function from the `"Do We Really Need Complicated
    Model Architectures for Temporal Networks?"
    `_ paper.
    :class:`TemporalEncoding` first maps each entry to a vector with
    monotonically exponentially decreasing values, and then uses the cosine
    function to project all values to range :math:`[-1, 1]`

    .. math::
        y_{i} = \cos \left(x \cdot \sqrt{d}^{-(i - 1)/\sqrt{d}} \right)

    where :math:`d` defines the output feature dimension, and
    :math:`1 \leq i \leq d`.

    Args:
        out_channels (int): Size :math:`d` of each output sample.
    """
    def __init__(self, out_channels: int):
        super().__init__()
        self.out_channels = out_channels

        sqrt = math.sqrt(out_channels)
        weight = 1.0 / sqrt**torch.linspace(0, sqrt, out_channels).view(1, -1)
        self.register_buffer('weight', weight)

        self.reset_parameters()

    def reset_parameters(self):
        pass

    def forward(self, x: Tensor) -> Tensor:
        """"""
        return torch.cos(x.view(-1, 1) @ self.weight)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.out_channels})'
pytorch/torch_geometric/nn/functional/000077500000000000000000000000001444262304300205025ustar00rootroot00000000000000pytorch/torch_geometric/nn/functional/__init__.py000066400000000000000000000001341444262304300226110ustar00rootroot00000000000000from .bro import bro
from .gini import gini

__all__ = classes = [
    'bro',
    'gini',
]
pytorch/torch_geometric/nn/functional/bro.py000066400000000000000000000030151444262304300216350ustar00rootroot00000000000000from typing import Union

import torch


def bro(
    x: torch.Tensor,
    batch: torch.Tensor,
    p: Union[int, str] = 2,
) -> torch.Tensor:
    r"""The Batch Representation Orthogonality penalty from the `"Improving
    Molecular Graph Neural Network Explainability with Orthonormalization
    and Induced Sparsity" `_ paper.

    Computes a regularization for each graph representation in a mini-batch
    according to

    .. math::
        \mathcal{L}_{\textrm{BRO}}^\mathrm{graph} =
          || \mathbf{HH}^T - \mathbf{I}||_p

    and returns an average over all graphs in the batch.

    Args:
        x (torch.Tensor): The node feature matrix.
        batch (torch.Tensor): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each node to a specific example.
        p (int or str, optional): The norm order to use. (default: :obj:`2`)
    """
    _, counts = torch.unique(batch, return_counts=True)
    diags = torch.stack([
        torch.diag(x) for x in torch.nn.utils.rnn.pad_sequence(
            sequences=torch.ones_like(batch).split_with_sizes(counts.tolist()),
            padding_value=0.,
            batch_first=True,
        )
    ])
    x = x.split_with_sizes(split_sizes=counts.tolist())
    x = torch.nn.utils.rnn.pad_sequence(
        sequences=x,
        padding_value=0.,
        batch_first=True,
    )
    return torch.sum(torch.norm(x @ x.transpose(1, 2) - diags, p=p,
                                dim=(1, 2))) / counts.shape[0]
pytorch/torch_geometric/nn/functional/gini.py000066400000000000000000000015371444262304300220100ustar00rootroot00000000000000import torch


def gini(w: torch.Tensor) -> torch.Tensor:
    r"""The Gini coefficient from the `"Improving Molecular Graph Neural
    Network Explainability with Orthonormalization and Induced Sparsity"
    `_ paper.

    Computes a regularization penalty :math:`\in [0, 1]` for each row of a
    matrix according to

    .. math::
        \mathcal{L}_\textrm{Gini}^i = \sum_j^n \sum_{j'}^n \frac{|w_{ij}
         - w_{ij'}|}{2 (n^2 - n)\bar{w_i}}

    and returns an average over all rows.

    Args:
        w (torch.Tensor): A two-dimensional tensor.
    """
    s = 0
    for row in w:
        t = row.repeat(row.size(0), 1)
        u = (t - t.T).abs().sum() / (2 * (row.size(-1)**2 - row.size(-1)) *
                                     row.abs().mean() + torch.finfo().eps)
        s += u
    s /= w.shape[0]
    return s
pytorch/torch_geometric/nn/fx.py000066400000000000000000000362771444262304300173460ustar00rootroot00000000000000import copy
from typing import Any, Dict, Optional

import torch
from torch.nn import Module, ModuleDict, ModuleList, Sequential

try:
    from torch.fx import Graph, GraphModule, Node
except (ImportError, ModuleNotFoundError, AttributeError):
    GraphModule, Graph, Node = 'GraphModule', 'Graph', 'Node'


class Transformer:
    r"""A :class:`Transformer` executes an FX graph node-by-node, applies
    transformations to each node, and produces a new :class:`torch.nn.Module`.
    It exposes a :func:`transform` method that returns the transformed
    :class:`~torch.nn.Module`.
    :class:`Transformer` works entirely symbolically.

    Methods in the :class:`Transformer` class can be overriden to customize the
    behavior of transformation.

    .. code-block:: none

        transform()
            +-- Iterate over each node in the graph
                +-- placeholder()
                +-- get_attr()
                +-- call_function()
                +-- call_method()
                +-- call_module()
                +-- call_message_passing_module()
                +-- call_global_pooling_module()
                +-- output()
            +-- Erase unused nodes in the graph
            +-- Iterate over each children module
                +-- init_submodule()

    In contrast to the :class:`torch.fx.Transformer` class, the
    :class:`Transformer` exposes additional functionality:

    #. It subdivides :func:`call_module` into nodes that call a regular
       :class:`torch.nn.Module` (:func:`call_module`), a
       :class:`MessagePassing` module (:func:`call_message_passing_module`),
       or a :class:`GlobalPooling` module (:func:`call_global_pooling_module`).

    #. It allows to customize or initialize new children modules via
       :func:`init_submodule`

    #. It allows to infer whether a node returns node-level or edge-level
       information via :meth:`is_edge_level`.

    Args:
        module (torch.nn.Module): The module to be transformed.
        input_map (Dict[str, str], optional): A dictionary holding information
            about the type of input arguments of :obj:`module.forward`.
            For example, in case :obj:`arg` is a node-level argument, then
            :obj:`input_map['arg'] = 'node'`, and
            :obj:`input_map['arg'] = 'edge'` otherwise.
            In case :obj:`input_map` is not further specified, will try to
            automatically determine the correct type of input arguments.
            (default: :obj:`None`)
        debug (bool, optional): If set to :obj:`True`, will perform
            transformation in debug mode. (default: :obj:`False`)
    """
    def __init__(
        self,
        module: Module,
        input_map: Optional[Dict[str, str]] = None,
        debug: bool = False,
    ):
        self.module = module
        self.gm = symbolic_trace(module)
        self.input_map = input_map
        self.debug = debug

    # Methods to override #####################################################

    def placeholder(self, node: Node, target: Any, name: str):
        pass

    def get_attr(self, node: Node, target: Any, name: str):
        pass

    def call_message_passing_module(self, node: Node, target: Any, name: str):
        pass

    def call_global_pooling_module(self, node: Node, target: Any, name: str):
        pass

    def call_module(self, node: Node, target: Any, name: str):
        pass

    def call_method(self, node: Node, target: Any, name: str):
        pass

    def call_function(self, node: Node, target: Any, name: str):
        pass

    def output(self, node: Node, target: Any, name: str):
        pass

    def init_submodule(self, module: Module, target: str) -> Module:
        return module

    # Internal functionality ##################################################

    @property
    def graph(self) -> Graph:
        return self.gm.graph

    def transform(self) -> GraphModule:
        r"""Transforms :obj:`self.module` and returns a transformed
        :class:`torch.fx.GraphModule`."""

        if self.debug:
            self.graph.print_tabular()
            print()
            code = self.graph.python_code('self')
            print(code.src if hasattr(code, 'src') else code)

        # We create a private dictionary `self._state` which holds information
        # about whether a node returns node-level or edge-level information:
        # `self._state[node.name] in { 'node', 'edge' }`
        self._state = copy.copy(self.input_map or {})

        # We iterate over each node and determine its output level
        # (node-level, edge-level) by filling `self._state`:
        for node in list(self.graph.nodes):
            if node.op == 'placeholder':
                if node.name not in self._state:
                    if 'edge' in node.name or 'adj' in node.name:
                        self._state[node.name] = 'edge'
                    else:
                        self._state[node.name] = 'node'
            elif is_message_passing_op(self.module, node.op, node.target):
                self._state[node.name] = 'node'
            elif is_global_pooling_op(self.module, node.op, node.target):
                self._state[node.name] = 'graph'
            elif node.op in ['call_module', 'call_method', 'call_function']:
                if self.has_edge_level_arg(node):
                    self._state[node.name] = 'edge'
                elif self.has_node_level_arg(node):
                    self._state[node.name] = 'node'
                else:
                    self._state[node.name] = 'graph'

        # We iterate over each node and may transform it:
        for node in list(self.graph.nodes):
            # Call the corresponding `Transformer` method for each `node.op`,
            # e.g.: `call_module(...)`, `call_function(...)`, ...
            op = node.op
            if is_message_passing_op(self.module, op, node.target):
                op = 'call_message_passing_module'
            elif is_global_pooling_op(self.module, op, node.target):
                op = 'call_global_pooling_module'
            getattr(self, op)(node, node.target, node.name)

        # Remove all unused nodes in the computation graph, i.e., all nodes
        # which have been replaced by node type-wise or edge type-wise variants
        # but which are still present in the computation graph.
        # We do this by iterating over the computation graph in reversed order,
        # and try to remove every node. This does only succeed in case there
        # are no users of that node left in the computation graph.
        for node in reversed(list(self.graph.nodes)):
            try:
                if node.op not in ['placeholder', 'output']:
                    self.graph.erase_node(node)
            except RuntimeError:
                pass

        for target, submodule in dict(self.module._modules).items():
            self.gm._modules[target] = self._init_submodule(submodule, target)

        del self._state

        if self.debug:
            self.gm.graph.print_tabular()
            print()
            code = self.graph.python_code('self')
            print(code.src if hasattr(code, 'src') else code)

        self.gm.graph.lint()
        self.gm.recompile()

        return self.gm

    def _init_submodule(self, module: Module, target: str) -> Module:
        if isinstance(module, ModuleList) or isinstance(module, Sequential):
            return ModuleList([
                self._init_submodule(submodule, f'{target}.{i}')
                for i, submodule in enumerate(module)
            ])
        elif isinstance(module, ModuleDict):
            return ModuleDict({
                key: self._init_submodule(submodule, f'{target}.{key}')
                for key, submodule in module.items()
            })
        else:
            return self.init_submodule(module, target)

    def _is_level(self, node: Node, name: str) -> bool:
        return self._state[node.name] == name

    def _has_level_arg(self, node: Node, name: str) -> bool:
        def _recurse(value: Any) -> bool:
            if isinstance(value, Node):
                return getattr(self, f'is_{name}_level')(value)
            elif isinstance(value, dict):
                return any([_recurse(v) for v in value.values()])
            elif isinstance(value, (list, tuple)):
                return any([_recurse(v) for v in value])
            else:
                return False

        return (any([_recurse(value) for value in node.args])
                or any([_recurse(value) for value in node.kwargs.values()]))

    def is_node_level(self, node: Node) -> bool:
        return self._is_level(node, name='node')

    def is_edge_level(self, node: Node) -> bool:
        return self._is_level(node, name='edge')

    def is_graph_level(self, node: Node) -> bool:
        return self._is_level(node, name='graph')

    def has_node_level_arg(self, node: Node) -> bool:
        return self._has_level_arg(node, name='node')

    def has_edge_level_arg(self, node: Node) -> bool:
        return self._has_level_arg(node, name='edge')

    def has_graph_level_arg(self, node: Node) -> bool:
        return self._has_level_arg(node, name='graph')

    def find_by_name(self, name: str) -> Optional[Node]:
        for node in self.graph.nodes:
            if node.name == name:
                return node
        return None

    def find_by_target(self, target: Any) -> Optional[Node]:
        for node in self.graph.nodes:
            if node.target == target:
                return node
        return None

    def replace_all_uses_with(self, to_replace: Node, replace_with: Node):
        def maybe_replace_node(n: Node) -> Node:
            return replace_with if n == to_replace else n

        node = replace_with.next
        while node.op != 'root':
            node.args = torch.fx.map_arg(node.args, maybe_replace_node)
            node.kwargs = torch.fx.map_arg(node.kwargs, maybe_replace_node)
            node = node.next


def symbolic_trace(
        module: Module,
        concrete_args: Optional[Dict[str, Any]] = None) -> GraphModule:

    # This is to support compatibility with pytorch version 1.9 and lower
    try:
        import torch.fx._symbolic_trace as st
    except (ImportError, ModuleNotFoundError):
        import torch.fx.symbolic_trace as st

    from torch_geometric.nn import Aggregation

    class Tracer(torch.fx.Tracer):
        def is_leaf_module(self, module: Module, *args, **kwargs) -> bool:
            # TODO We currently only trace top-level modules.
            return not isinstance(module, torch.nn.Sequential)

        # Note: This is a hack around the fact that `Aggregaton.__call__`
        # is not patched by the base implementation of `trace`.
        # see https://github.com/pyg-team/pytorch_geometric/pull/5021 for
        # details on the rationale
        # TODO: Revisit https://github.com/pyg-team/pytorch_geometric/pull/5021
        @st.compatibility(is_backward_compatible=True)
        def trace(self, root: st.Union[torch.nn.Module, st.Callable[..., Any]],
                  concrete_args: Optional[Dict[str, Any]] = None) -> Graph:

            if isinstance(root, torch.nn.Module):
                self.root = root
                fn = type(root).forward
                self.submodule_paths = {
                    mod: name
                    for name, mod in root.named_modules()
                }
            else:
                self.root = torch.nn.Module()
                fn = root

            tracer_cls: Optional[st.Type['Tracer']] = getattr(
                self, '__class__', None)
            self.graph = Graph(tracer_cls=tracer_cls)

            self.tensor_attrs: Dict[st.Union[torch.Tensor, st.ScriptObject],
                                    str] = {}

            def collect_tensor_attrs(m: torch.nn.Module,
                                     prefix_atoms: st.List[str]):
                for k, v in m.__dict__.items():
                    if isinstance(v, (torch.Tensor, st.ScriptObject)):
                        self.tensor_attrs[v] = '.'.join(prefix_atoms + [k])
                for k, v in m.named_children():
                    collect_tensor_attrs(v, prefix_atoms + [k])

            collect_tensor_attrs(self.root, [])

            assert isinstance(fn, st.FunctionType)

            fn_globals = fn.__globals__  # run before it gets patched
            fn, args = self.create_args_for_root(
                fn, isinstance(root, torch.nn.Module), concrete_args)

            parameter_proxy_cache: Dict[str, st.Proxy] = {
            }  # Reduce number of get_attr calls

            @st.functools.wraps(st._orig_module_getattr)
            def module_getattr_wrapper(mod, attr):
                attr_val = st._orig_module_getattr(mod, attr)
                # Support for PyTorch > 1.12, see:
                # https://github.com/pytorch/pytorch/pull/84011
                if hasattr(self, 'getattr'):
                    return self.getattr(attr, attr_val, parameter_proxy_cache)
                return self._module_getattr(attr, attr_val,
                                            parameter_proxy_cache)

            @st.functools.wraps(st._orig_module_call)
            def module_call_wrapper(mod, *args, **kwargs):
                def forward(*args, **kwargs):
                    return st._orig_module_call(mod, *args, **kwargs)

                st._autowrap_check(
                    patcher,
                    getattr(getattr(mod, "forward", mod), "__globals__", {}),
                    self._autowrap_function_ids)
                return self.call_module(mod, forward, args, kwargs)

            with st._Patcher() as patcher:
                # allow duplicate patches to support the case of nested calls
                patcher.patch_method(torch.nn.Module, "__getattr__",
                                     module_getattr_wrapper, deduplicate=False)
                patcher.patch_method(torch.nn.Module, "__call__",
                                     module_call_wrapper, deduplicate=False)
                patcher.patch_method(Aggregation, "__call__",
                                     module_call_wrapper, deduplicate=False)
                st._patch_wrapped_functions(patcher)
                st._autowrap_check(patcher, fn_globals,
                                   self._autowrap_function_ids)
                for module in self._autowrap_search:
                    st._autowrap_check(patcher, module.__dict__,
                                       self._autowrap_function_ids)
                self.create_node(
                    'output', 'output', (self.create_arg(fn(*args)), ), {},
                    type_expr=fn.__annotations__.get('return', None))

            self.submodule_paths = None

            return self.graph

    return GraphModule(module, Tracer().trace(module, concrete_args))


def get_submodule(module: Module, target: str) -> Module:
    out = module
    for attr in target.split('.'):
        out = getattr(out, attr)
    return out


def is_message_passing_op(module: Module, op: str, target: str) -> bool:
    from torch_geometric.nn import MessagePassing
    if op == 'call_module':
        return isinstance(get_submodule(module, target), MessagePassing)
    return False


def is_global_pooling_op(module: Module, op: str, target: str) -> bool:
    from torch_geometric.nn import Aggregation
    if op == 'call_module':
        return isinstance(get_submodule(module, target), Aggregation)
    return False
pytorch/torch_geometric/nn/glob.py000066400000000000000000000021001444262304300176260ustar00rootroot00000000000000from torch_geometric.deprecation import deprecated
from torch_geometric.nn import (
    global_add_pool,
    global_max_pool,
    global_mean_pool,
)
from torch_geometric.nn.aggr import AttentionalAggregation, SortAggregation


@deprecated(
    details="use 'nn.aggr.AttentionalAggregation' instead",
    func_name='nn.glob.GlobalAttention',
)
class GlobalAttention(AttentionalAggregation):
    def __call__(self, x, batch=None, size=None):
        return super().__call__(x, batch, dim_size=size)


@deprecated(
    details="use 'nn.aggr.SortAggr' instead",
    func_name='nn.glob.global_sort_pool',
)
def global_sort_pool(x, index, k):
    module = SortAggregation(k=k)
    return module(x, index=index)


deprecated(
    details="use 'nn.pool.global_add_pool' instead",
    func_name='nn.glob.global_add_pool',
)(global_add_pool)

deprecated(
    details="use 'nn.pool.global_max_pool' instead",
    func_name='nn.glob.global_max_pool',
)(global_max_pool)

deprecated(
    details="use 'nn.pool.global_mean_pool' instead",
    func_name='nn.glob.global_mean_pool',
)(global_mean_pool)
pytorch/torch_geometric/nn/inits.py000066400000000000000000000046311444262304300200440ustar00rootroot00000000000000import math
from typing import Any

import torch
from torch import Tensor


def uniform(size: int, value: Any):
    if isinstance(value, Tensor):
        bound = 1.0 / math.sqrt(size)
        value.data.uniform_(-bound, bound)
    else:
        for v in value.parameters() if hasattr(value, 'parameters') else []:
            uniform(size, v)
        for v in value.buffers() if hasattr(value, 'buffers') else []:
            uniform(size, v)


def kaiming_uniform(value: Any, fan: int, a: float):
    if isinstance(value, Tensor):
        bound = math.sqrt(6 / ((1 + a**2) * fan))
        value.data.uniform_(-bound, bound)
    else:
        for v in value.parameters() if hasattr(value, 'parameters') else []:
            kaiming_uniform(v, fan, a)
        for v in value.buffers() if hasattr(value, 'buffers') else []:
            kaiming_uniform(v, fan, a)


def glorot(value: Any):
    if isinstance(value, Tensor):
        stdv = math.sqrt(6.0 / (value.size(-2) + value.size(-1)))
        value.data.uniform_(-stdv, stdv)
    else:
        for v in value.parameters() if hasattr(value, 'parameters') else []:
            glorot(v)
        for v in value.buffers() if hasattr(value, 'buffers') else []:
            glorot(v)


def glorot_orthogonal(tensor, scale):
    if tensor is not None:
        torch.nn.init.orthogonal_(tensor.data)
        scale /= ((tensor.size(-2) + tensor.size(-1)) * tensor.var())
        tensor.data *= scale.sqrt()


def constant(value: Any, fill_value: float):
    if isinstance(value, Tensor):
        value.data.fill_(fill_value)
    else:
        for v in value.parameters() if hasattr(value, 'parameters') else []:
            constant(v, fill_value)
        for v in value.buffers() if hasattr(value, 'buffers') else []:
            constant(v, fill_value)


def zeros(value: Any):
    constant(value, 0.)


def ones(tensor: Any):
    constant(tensor, 1.)


def normal(value: Any, mean: float, std: float):
    if isinstance(value, Tensor):
        value.data.normal_(mean, std)
    else:
        for v in value.parameters() if hasattr(value, 'parameters') else []:
            normal(v, mean, std)
        for v in value.buffers() if hasattr(value, 'buffers') else []:
            normal(v, mean, std)


def reset(value: Any):
    if hasattr(value, 'reset_parameters'):
        value.reset_parameters()
    else:
        for child in value.children() if hasattr(value, 'children') else []:
            reset(child)
pytorch/torch_geometric/nn/kge/000077500000000000000000000000001444262304300171065ustar00rootroot00000000000000pytorch/torch_geometric/nn/kge/__init__.py000066400000000000000000000003611444262304300212170ustar00rootroot00000000000000from .base import KGEModel
from .transe import TransE
from .complex import ComplEx
from .distmult import DistMult
from .rotate import RotatE

__all__ = classes = [
    'KGEModel',
    'TransE',
    'ComplEx',
    'DistMult',
    'RotatE',
]
pytorch/torch_geometric/nn/kge/base.py000066400000000000000000000131531444262304300203750ustar00rootroot00000000000000from typing import Tuple

import torch
from torch import Tensor
from torch.nn import Embedding
from tqdm import tqdm

from torch_geometric.nn.kge.loader import KGTripletLoader


class KGEModel(torch.nn.Module):
    r"""An abstract base class for implementing custom KGE models.

    Args:
        num_nodes (int): The number of nodes/entities in the graph.
        num_relations (int): The number of relations in the graph.
        hidden_channels (int): The hidden embedding size.
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to the
            embedding matrices will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_relations: int,
        hidden_channels: int,
        sparse: bool = False,
    ):
        super().__init__()

        self.num_nodes = num_nodes
        self.num_relations = num_relations
        self.hidden_channels = hidden_channels

        self.node_emb = Embedding(num_nodes, hidden_channels, sparse=sparse)
        self.rel_emb = Embedding(num_relations, hidden_channels, sparse=sparse)

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.node_emb.reset_parameters()
        self.rel_emb.reset_parameters()

    def forward(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:
        r"""Returns the score for the given triplet.

        Args:
            head_index (torch.Tensor): The head indices.
            rel_type (torch.Tensor): The relation type.
            tail_index (torch.Tensor): The tail indices.
        """
        raise NotImplementedError

    def loss(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:
        r"""Returns the loss value for the given triplet.

        Args:
            head_index (torch.Tensor): The head indices.
            rel_type (torch.Tensor): The relation type.
            tail_index (torch.Tensor): The tail indices.
        """
        raise NotImplementedError

    def loader(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
        **kwargs,
    ) -> Tensor:
        r"""Returns a mini-batch loader that samples a subset of triplets.

        Args:
            head_index (torch.Tensor): The head indices.
            rel_type (torch.Tensor): The relation type.
            tail_index (torch.Tensor): The tail indices.
            **kwargs (optional): Additional arguments of
                :class:`torch.utils.data.DataLoader`, such as
                :obj:`batch_size`, :obj:`shuffle`, :obj:`drop_last`
                or :obj:`num_workers`.
            """
        return KGTripletLoader(head_index, rel_type, tail_index, **kwargs)

    @torch.no_grad()
    def test(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
        batch_size: int,
        k: int = 10,
        log: bool = True,
    ) -> Tuple[float, float]:
        r"""Evaluates the model quality by computing Mean Rank and
        Hits @ :math:`k` across all possible tail entities.

        Args:
            head_index (torch.Tensor): The head indices.
            rel_type (torch.Tensor): The relation type.
            tail_index (torch.Tensor): The tail indices.
            batch_size (int): The batch size to use for evaluating.
            k (int, optional): The :math:`k` in Hits @ :math:`k`.
                (default: :obj:`10`)
            log (bool, optional): If set to :obj:`False`, will not print a
                progress bar to the console. (default: :obj:`True`)
        """
        arange = range(head_index.numel())
        arange = tqdm(arange) if log else arange

        mean_ranks, hits_at_k = [], []
        for i in arange:
            h, r, t = head_index[i], rel_type[i], tail_index[i]

            scores = []
            tail_indices = torch.arange(self.num_nodes, device=t.device)
            for ts in tail_indices.split(batch_size):
                scores.append(self(h.expand_as(ts), r.expand_as(ts), ts))
            rank = int((torch.cat(scores).argsort(
                descending=True) == t).nonzero().view(-1))
            mean_ranks.append(rank)
            hits_at_k.append(rank < k)

        mean_rank = float(torch.tensor(mean_ranks, dtype=torch.float).mean())
        hits_at_k = int(torch.tensor(hits_at_k).sum()) / len(hits_at_k)

        return mean_rank, hits_at_k

    @torch.no_grad()
    def random_sample(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tuple[Tensor, Tensor, Tensor]:
        r"""Randomly samples negative triplets by either replacing the head or
        the tail (but not both).

        Args:
            head_index (torch.Tensor): The head indices.
            rel_type (torch.Tensor): The relation type.
            tail_index (torch.Tensor): The tail indices.
        """
        # Random sample either `head_index` or `tail_index` (but not both):
        num_negatives = head_index.numel() // 2
        rnd_index = torch.randint(self.num_nodes, head_index.size(),
                                  device=head_index.device)

        head_index = head_index.clone()
        head_index[:num_negatives] = rnd_index[:num_negatives]
        tail_index = tail_index.clone()
        tail_index[num_negatives:] = rnd_index[num_negatives:]

        return head_index, rel_type, tail_index

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.num_nodes}, '
                f'num_relations={self.num_relations}, '
                f'hidden_channels={self.hidden_channels})')
pytorch/torch_geometric/nn/kge/complex.py000066400000000000000000000062421444262304300211330ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Embedding

from torch_geometric.nn.kge import KGEModel


class ComplEx(KGEModel):
    r"""The ComplEx model from the `"Complex Embeddings for Simple Link
    Prediction" `_ paper.

    :class:`ComplEx` models relations as complex-valued bilinear mappings
    between head and tail entities using the Hermetian dot product.
    The entities and relations are embedded in different dimensional spaces,
    resulting in the scoring function:

    .. math::
        d(h, r, t) = Re(< \mathbf{e}_h,  \mathbf{e}_r, \mathbf{e}_t>)

    .. note::

        For an example of using the :class:`ComplEx` model, see
        `examples/kge_fb15k_237.py
        `_.

    Args:
        num_nodes (int): The number of nodes/entities in the graph.
        num_relations (int): The number of relations in the graph.
        hidden_channels (int): The hidden embedding size.
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to
            the embedding matrices will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_relations: int,
        hidden_channels: int,
        sparse: bool = False,
    ):
        super().__init__(num_nodes, num_relations, hidden_channels, sparse)

        self.node_emb_im = Embedding(num_nodes, hidden_channels, sparse=sparse)
        self.rel_emb_im = Embedding(num_relations, hidden_channels,
                                    sparse=sparse)

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.node_emb.weight)
        torch.nn.init.xavier_uniform_(self.node_emb_im.weight)
        torch.nn.init.xavier_uniform_(self.rel_emb.weight)
        torch.nn.init.xavier_uniform_(self.rel_emb_im.weight)

    def forward(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        head_re = self.node_emb(head_index)
        head_im = self.node_emb_im(head_index)
        rel_re = self.rel_emb(rel_type)
        rel_im = self.rel_emb_im(rel_type)
        tail_re = self.node_emb(tail_index)
        tail_im = self.node_emb_im(tail_index)

        return (triple_dot(head_re, rel_re, tail_re) +
                triple_dot(head_im, rel_re, tail_im) +
                triple_dot(head_re, rel_im, tail_im) -
                triple_dot(head_im, rel_im, tail_re))

    def loss(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        pos_score = self(head_index, rel_type, tail_index)
        neg_score = self(*self.random_sample(head_index, rel_type, tail_index))
        scores = torch.cat([pos_score, neg_score], dim=0)

        pos_target = torch.ones_like(pos_score)
        neg_target = torch.zeros_like(neg_score)
        target = torch.cat([pos_target, neg_target], dim=0)

        return F.binary_cross_entropy_with_logits(scores, target)


def triple_dot(x: Tensor, y: Tensor, z: Tensor) -> Tensor:
    return (x * y * z).sum(dim=-1)
pytorch/torch_geometric/nn/kge/distmult.py000066400000000000000000000046361444262304300213360ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.kge import KGEModel


class DistMult(KGEModel):
    r"""The DistMult model from the `"Embedding Entities and Relations for
    Learning and Inference in Knowledge Bases"
    `_ paper.

    :class:`DistMult` models relations as diagonal matrices, which simplifies
    the bi-linear interaction between the head and tail entities to the score
    function:

    .. math::
        d(h, r, t) = < \mathbf{e}_h,  \mathbf{e}_r, \mathbf{e}_t >

    .. note::

        For an example of using the :class:`DistMult` model, see
        `examples/kge_fb15k_237.py
        `_.

    Args:
        num_nodes (int): The number of nodes/entities in the graph.
        num_relations (int): The number of relations in the graph.
        hidden_channels (int): The hidden embedding size.
        margin (float, optional): The margin of the ranking loss.
            (default: :obj:`1.0`)
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to
            the embedding matrices will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_relations: int,
        hidden_channels: int,
        margin: float = 1.0,
        sparse: bool = False,
    ):
        super().__init__(num_nodes, num_relations, hidden_channels, sparse)

        self.margin = margin

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.node_emb.weight)
        torch.nn.init.xavier_uniform_(self.rel_emb.weight)

    def forward(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        head = self.node_emb(head_index)
        rel = self.rel_emb(rel_type)
        tail = self.node_emb(tail_index)

        return (head * rel * tail).sum(dim=-1)

    def loss(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        pos_score = self(head_index, rel_type, tail_index)
        neg_score = self(*self.random_sample(head_index, rel_type, tail_index))

        return F.margin_ranking_loss(
            pos_score,
            neg_score,
            target=torch.ones_like(pos_score),
            margin=self.margin,
        )
pytorch/torch_geometric/nn/kge/loader.py000066400000000000000000000014031444262304300207240ustar00rootroot00000000000000from typing import List, Tuple

import torch
from torch import Tensor


class KGTripletLoader(torch.utils.data.DataLoader):
    def __init__(self, head_index: Tensor, rel_type: Tensor,
                 tail_index: Tensor, **kwargs):
        self.head_index = head_index
        self.rel_type = rel_type
        self.tail_index = tail_index

        super().__init__(range(head_index.numel()), collate_fn=self.sample,
                         **kwargs)

    def sample(self, index: List[int]) -> Tuple[Tensor, Tensor, Tensor]:
        index = torch.tensor(index, device=self.head_index.device)

        head_index = self.head_index[index]
        rel_type = self.rel_type[index]
        tail_index = self.tail_index[index]

        return head_index, rel_type, tail_index
pytorch/torch_geometric/nn/kge/rotate.py000066400000000000000000000062101444262304300207550ustar00rootroot00000000000000import math

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Embedding

from torch_geometric.nn.kge import KGEModel


class RotatE(KGEModel):
    r"""The RotatE model from the `"RotatE: Knowledge Graph Embedding by
    Relational Rotation in Complex Space" `_ paper.

    :class:`RotatE` models relations as a rotation in complex space
    from head to tail such that

    .. math::
        \mathbf{e}_t = \mathbf{e}_h \circ \mathbf{e}_r,

    resulting in the scoring function

    .. math::
        d(h, r, t) = - {\| \mathbf{e}_h \circ \mathbf{e}_r - \mathbf{e}_t \|}_p

    .. note::

        For an example of using the :class:`RotatE` model, see
        `examples/kge_fb15k_237.py
        `_.

    Args:
        num_nodes (int): The number of nodes/entities in the graph.
        num_relations (int): The number of relations in the graph.
        hidden_channels (int): The hidden embedding size.
        margin (float, optional): The margin of the ranking loss.
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to
            the embedding matrices will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_relations: int,
        hidden_channels: int,
        margin: float = 1.0,
        sparse: bool = False,
    ):
        super().__init__(num_nodes, num_relations, hidden_channels, sparse)

        self.margin = margin
        self.node_emb_im = Embedding(num_nodes, hidden_channels, sparse=sparse)

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.node_emb.weight)
        torch.nn.init.xavier_uniform_(self.node_emb_im.weight)
        torch.nn.init.uniform_(self.rel_emb.weight, 0, 2 * math.pi)

    def forward(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        head_re = self.node_emb(head_index)
        head_im = self.node_emb_im(head_index)
        tail_re = self.node_emb(tail_index)
        tail_im = self.node_emb_im(tail_index)

        rel_theta = self.rel_emb(rel_type)
        rel_re, rel_im = torch.cos(rel_theta), torch.sin(rel_theta)

        re_score = (rel_re * head_re - rel_im * head_im) - tail_re
        im_score = (rel_re * head_im + rel_im * head_re) - tail_im
        complex_score = torch.stack([re_score, im_score], dim=2)
        score = torch.linalg.vector_norm(complex_score, dim=(1, 2))

        return self.margin - score

    def loss(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        pos_score = self(head_index, rel_type, tail_index)
        neg_score = self(*self.random_sample(head_index, rel_type, tail_index))
        scores = torch.cat([pos_score, neg_score], dim=0)

        pos_target = torch.ones_like(pos_score)
        neg_target = torch.zeros_like(neg_score)
        target = torch.cat([pos_target, neg_target], dim=0)

        return F.binary_cross_entropy_with_logits(scores, target)
pytorch/torch_geometric/nn/kge/transe.py000066400000000000000000000060201444262304300207520ustar00rootroot00000000000000import math

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn.kge import KGEModel


class TransE(KGEModel):
    r"""The TransE model from the `"Translating Embeddings for Modeling
    Multi-Relational Data" `_ paper.

    :class:`TransE` models relations as a translation from head to tail
    entities such that

    .. math::
        \mathbf{e}_h + \mathbf{e}_r \approx \mathbf{e}_t,

    resulting in the scoring function:

    .. math::
        d(h, r, t) = - {\| \mathbf{e}_h + \mathbf{e}_r - \mathbf{e}_t \|}_p

    .. note::

        For an example of using the :class:`TransE` model, see
        `examples/kge_fb15k_237.py
        `_.

    Args:
        num_nodes (int): The number of nodes/entities in the graph.
        num_relations (int): The number of relations in the graph.
        hidden_channels (int): The hidden embedding size.
        margin (int, optional): The margin of the ranking loss.
            (default: :obj:`1.0`)
        p_norm (int, optional): The order embedding and distance normalization.
            (default: :obj:`1.0`)
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to the
            embedding matrices will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_relations: int,
        hidden_channels: int,
        margin: float = 1.0,
        p_norm: float = 1.0,
        sparse: bool = False,
    ):
        super().__init__(num_nodes, num_relations, hidden_channels, sparse)

        self.p_norm = p_norm
        self.margin = margin

        self.reset_parameters()

    def reset_parameters(self):
        bound = 6. / math.sqrt(self.hidden_channels)
        torch.nn.init.uniform_(self.node_emb.weight, -bound, bound)
        torch.nn.init.uniform_(self.rel_emb.weight, -bound, bound)
        F.normalize(self.rel_emb.weight.data, p=self.p_norm, dim=-1,
                    out=self.rel_emb.weight.data)

    def forward(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        head = self.node_emb(head_index)
        rel = self.rel_emb(rel_type)
        tail = self.node_emb(tail_index)

        head = F.normalize(head, p=self.p_norm, dim=-1)
        tail = F.normalize(tail, p=self.p_norm, dim=-1)

        # Calculate *negative* TransE norm:
        return -((head + rel) - tail).norm(p=self.p_norm, dim=-1)

    def loss(
        self,
        head_index: Tensor,
        rel_type: Tensor,
        tail_index: Tensor,
    ) -> Tensor:

        pos_score = self(head_index, rel_type, tail_index)
        neg_score = self(*self.random_sample(head_index, rel_type, tail_index))

        return F.margin_ranking_loss(
            pos_score,
            neg_score,
            target=torch.ones_like(pos_score),
            margin=self.margin,
        )
pytorch/torch_geometric/nn/lr_scheduler.py000066400000000000000000000213511444262304300213670ustar00rootroot00000000000000# See HuggingFace `transformers/optimization.py`.
import functools
import math

from torch.optim import Optimizer
from torch.optim.lr_scheduler import LambdaLR


class ConstantWithWarmupLR(LambdaLR):
    r"""Creates a LR scheduler with a constant learning rate preceded by a
    warmup period during which the learning rate increases linearly between
    :obj:`0` and the initial LR set in the optimizer.

    Args:
        optimizer (Optimizer): The optimizer to be scheduled.
        num_warmup_steps (int): The number of steps for the warmup phase.
        last_epoch (int, optional): The index of the last epoch when resuming
            training. (default: :obj:`-1`)
    """
    def __init__(
        self,
        optimizer: Optimizer,
        num_warmup_steps: int,
        last_epoch: int = -1,
    ):
        lr_lambda = functools.partial(
            self._lr_lambda,
            num_warmup_steps=num_warmup_steps,
        )
        super().__init__(optimizer, lr_lambda, last_epoch)

    @staticmethod
    def _lr_lambda(
        current_step: int,
        num_warmup_steps: int,
    ) -> float:
        if current_step < num_warmup_steps:
            return float(current_step) / float(max(1.0, num_warmup_steps))
        return 1.0


class LinearWithWarmupLR(LambdaLR):
    r"""Creates a LR scheduler with a learning rate that decreases linearly
    from the initial LR set in the optimizer to :obj:`0`, after a warmup period
    during which it increases linearly from :obj:`0` to the initial LR set in
    the optimizer.

    Args:
        optimizer (Optimizer): The optimizer to be scheduled.
        num_warmup_steps (int): The number of steps for the warmup phase.
        num_training_steps (int): The total number of training steps.
        last_epoch (int, optional): The index of the last epoch when resuming
            training. (default: :obj:`-1`)
    """
    def __init__(
        self,
        optimizer: Optimizer,
        num_warmup_steps: int,
        num_training_steps: int,
        last_epoch: int = -1,
    ):
        lr_lambda = functools.partial(
            self._lr_lambda,
            num_warmup_steps=num_warmup_steps,
            num_training_steps=num_training_steps,
        )
        super().__init__(optimizer, lr_lambda, last_epoch)

    @staticmethod
    def _lr_lambda(
        current_step: int,
        num_warmup_steps: int,
        num_training_steps: int,
    ) -> float:
        if current_step < num_warmup_steps:
            return float(current_step) / float(max(1, num_warmup_steps))
        return max(
            0.0,
            float(num_training_steps - current_step) /
            float(max(1, num_training_steps - num_warmup_steps)),
        )


class CosineWithWarmupLR(LambdaLR):
    r"""Creates a LR scheduler with a learning rate that decreases following
    the values of the cosine function between the initial LR set in the
    optimizer to :obj:`0`, after a warmup period during which it increases
    linearly between :obj:`0` and the initial LR set in the optimizer.

    Args:
        optimizer (Optimizer): The optimizer to be scheduled.
        num_warmup_steps (int): The number of steps for the warmup phase.
        num_training_steps (int): The total number of training steps.
        num_cycles (float, optional): The number of waves in the cosine
            schedule (the default decreases LR from the max value to :obj:`0`
            following a half-cosine). (default: :obj:`0.5`)
        last_epoch (int, optional): The index of the last epoch when resuming
            training. (default: :obj:`-1`)
    """
    def __init__(
        self,
        optimizer: Optimizer,
        num_warmup_steps: int,
        num_training_steps: int,
        num_cycles: float = 0.5,
        last_epoch: int = -1,
    ):
        lr_lambda = functools.partial(
            self._lr_lambda,
            num_warmup_steps=num_warmup_steps,
            num_training_steps=num_training_steps,
            num_cycles=num_cycles,
        )
        super().__init__(optimizer, lr_lambda, last_epoch)

    @staticmethod
    def _lr_lambda(
        current_step: int,
        num_warmup_steps: int,
        num_training_steps: int,
        num_cycles: float,
    ):
        if current_step < num_warmup_steps:
            return float(current_step) / float(max(1, num_warmup_steps))
        progress = float(current_step - num_warmup_steps) / float(
            max(1, num_training_steps - num_warmup_steps))
        return max(
            0.0,
            0.5 *
            (1.0 + math.cos(math.pi * float(num_cycles) * 2.0 * progress)),
        )


class CosineWithWarmupRestartsLR(LambdaLR):
    r"""Creates a LR scheduler with a learning rate that decreases following
    the values of the cosine function between the initial LR set in the
    optimizer to :obj:`0`, with several hard restarts, after a warmup period
    during which it increases linearly between :obj:`0` and the initial LR set
    in the optimizer.

    Args:
        optimizer (Optimizer): The optimizer to be scheduled.
        num_warmup_steps (int): The number of steps for the warmup phase.
        num_training_steps (int): The total number of training steps.
        num_cycles (int, optional): The number of hard restarts to use.
            (default: :obj:`3`)
        last_epoch (int, optional): The index of the last epoch when resuming
            training. (default: :obj:`-1`)
    """
    def __init__(
        self,
        optimizer: Optimizer,
        num_warmup_steps: int,
        num_training_steps: int,
        num_cycles: int = 3,
        last_epoch: int = -1,
    ):
        lr_lambda = functools.partial(
            self._lr_lambda,
            num_warmup_steps=num_warmup_steps,
            num_training_steps=num_training_steps,
            num_cycles=num_cycles,
        )
        super().__init__(optimizer, lr_lambda, last_epoch)

    @staticmethod
    def _lr_lambda(
        current_step: int,
        num_warmup_steps: int,
        num_training_steps: int,
        num_cycles: int,
    ) -> float:
        if current_step < num_warmup_steps:
            return float(current_step) / float(max(1, num_warmup_steps))
        progress = float(current_step - num_warmup_steps) / float(
            max(1, num_training_steps - num_warmup_steps))
        if progress >= 1.0:
            return 0.0
        return max(
            0.0,
            0.5 * (1.0 + math.cos(math.pi *
                                  ((float(num_cycles) * progress) % 1.0))),
        )


class PolynomialWithWarmupLR(LambdaLR):
    r"""Creates a LR scheduler with a learning rate that decreases as a
    polynomial decay from the initial LR set in the optimizer to end LR defined
    by `lr_end`, after a warmup period during which it increases linearly from
    :obj:`0` to the initial LR set in the optimizer.

    Args:
        optimizer (Optimizer): The optimizer to be scheduled.
        num_warmup_steps (int): The number of steps for the warmup phase.
        num_training_steps (int): The total number of training steps.
        lr_end (float, optional): The end learning rate. (default: :obj:`1e-7`)
        power (float, optional): The power factor of the polynomial decay.
            (default: :obj:`1.0`)
        last_epoch (int, optional): The index of the last epoch when resuming
            training. (default: :obj:`-1`)
    """
    def __init__(
        self,
        optimizer: Optimizer,
        num_warmup_steps: int,
        num_training_steps: int,
        lr_end: float = 1e-7,
        power: float = 1.0,
        last_epoch: int = -1,
    ):
        lr_init = optimizer.defaults["lr"]
        if not (lr_init > lr_end):
            raise ValueError(f"`lr_end` ({lr_end}) must be smaller than the "
                             f"initial lr ({lr_init})")

        lr_lambda = functools.partial(
            self._lr_lambda,
            num_warmup_steps=num_warmup_steps,
            num_training_steps=num_training_steps,
            lr_init=lr_init,
            lr_end=lr_end,
            power=power,
        )
        super().__init__(optimizer, lr_lambda, last_epoch)

    @staticmethod
    def _lr_lambda(
        current_step: int,
        num_warmup_steps: int,
        num_training_steps: int,
        lr_init: float,
        lr_end: float,
        power: float,
    ) -> float:
        if current_step < num_warmup_steps:
            return float(current_step) / float(max(1, num_warmup_steps))
        elif current_step > num_training_steps:
            return lr_end / lr_init  # As `LambdaLR` multiplies by `lr_init`.
        else:
            lr_range = lr_init - lr_end
            decay_steps = num_training_steps - num_warmup_steps
            pct_remaining = 1 - (current_step - num_warmup_steps) / decay_steps
            decay = lr_range * pct_remaining**power + lr_end
            return decay / lr_init  # As `LambdaLR` multiplies by `lr_init`.
pytorch/torch_geometric/nn/model_hub.py000066400000000000000000000224501444262304300206530ustar00rootroot00000000000000import os
from pathlib import Path
from typing import Any, Dict, Optional, Union

import torch

try:
    from huggingface_hub import ModelHubMixin, hf_hub_download
except ImportError:
    ModelHubMixin = object
    hf_hub_download = None

CONFIG_NAME = 'config.json'
MODEL_HUB_ORGANIZATION = "pytorch_geometric"
MODEL_WEIGHTS_NAME = 'model.pth'
TAGS = ['graph-machine-learning']


class PyGModelHubMixin(ModelHubMixin):
    r"""A mixin for saving and loading models to the
    `Huggingface Model Hub `_.

    .. code-block:: python

       from torch_geometric.datasets import Planetoid
       from torch_geometric.nn import Node2Vec
       from torch_geometric.nn.model_hub import PyGModelHubMixin

       # Define your class with the mixin:
       class N2V(Node2Vec, PyGModelHubMixin):
           def __init__(self,model_name, dataset_name, model_kwargs):
               Node2Vec.__init__(self,**model_kwargs)
               PyGModelHubMixin.__init__(self, model_name,
                   dataset_name, model_kwargs)

       # Instantiate your model:
       n2v = N2V(model_name='node2vec',
           dataset_name='Cora', model_kwargs=dict(
           edge_index=data.edge_index, embedding_dim=128,
           walk_length=20, context_size=10, walks_per_node=10,
           num_negative_samples=1, p=1, q=1, sparse=True))

       # Train the model:
       ...

       # Push to the HuggingFace hub:
       repo_id = ...  # your repo id
       n2v.save_pretrained(
           local_file_path,
           push_to_hub=True,
           repo_id=repo_id,
        )

       # Load the model for inference:
       # The required arguments are the repo id/local folder, and any model
       # initialisation arguments that are not native python types (e.g
       # Node2Vec requires the edge_index argument which is not stored in the
       # model hub).
       model = N2V.from_pretrained(
           repo_id,
           model_name='node2vec',
           dataset_name='Cora',
           edge_index=data.edge_index,
       )

    Args:
        model_name (str): Name of the model.
        dataset_name (str): Name of the dataset the model was trained against.
        model_kwargs (Dict[str, Any]): The arguments to initialise the model.
    """
    def __init__(self, model_name: str, dataset_name: str, model_kwargs: Dict):
        ModelHubMixin.__init__(self)

        # Huggingface Hub API only accepts saving the config as a dict.
        # If the model is instantiated with non-native python types
        # such as torch Tensors (node2vec being an example), we have to remove
        # these as they are not json serialisable
        self.model_config = {
            k: v
            for k, v in model_kwargs.items() if type(v) in [str, int, float]
        }
        self.model_name = model_name
        self.dataset_name = dataset_name

    def construct_model_card(self, model_name: str, dataset_name: str) -> Any:
        from huggingface_hub import ModelCard, ModelCardData
        card_data = ModelCardData(
            language='en',
            license='mit',
            library_name=MODEL_HUB_ORGANIZATION,
            tags=TAGS,
            datasets=dataset_name,
            model_name=model_name,
        )
        card = ModelCard.from_template(card_data)
        return card

    def _save_pretrained(self, save_directory: Union[Path, str]):
        path = os.path.join(save_directory, MODEL_WEIGHTS_NAME)
        model_to_save = self.module if hasattr(self, 'module') else self
        torch.save(model_to_save.state_dict(), path)

    def save_pretrained(self, save_directory: Union[str, Path],
                        push_to_hub: bool = False,
                        repo_id: Optional[str] = None, **kwargs):
        r"""Save a trained model to a local directory or to the HuggingFace
        model hub.

        Args:
            save_directory (str): The directory where weights are saved.
            push_to_hub (bool, optional): If :obj:`True`, push the model to the
                HuggingFace model hub. (default: :obj:`False`)
            repo_id (str, optional): The repository name in the hub.
                If not provided will default to the name of
                :obj:`save_directory` in your namespace. (default: :obj:`None`)
            **kwargs: Additional keyword arguments passed to
                :meth:`huggingface_hub.ModelHubMixin.save_pretrained`.
        """
        config = self.model_config
        # due to way huggingface hub handles the loading/saving of models,
        # the model config can end up in one of the items in the kwargs
        # this has to be removed to prevent a duplication of arguments to
        # ModelHubMixin.save_pretrained
        kwargs.pop('config', None)

        super().save_pretrained(
            save_directory=save_directory,
            config=config,
            push_to_hub=push_to_hub,
            repo_id=repo_id,
            **kwargs,
        )
        model_card = self.construct_model_card(self.model_name,
                                               self.dataset_name)
        if push_to_hub:
            model_card.push_to_hub(repo_id)

    @classmethod
    def _from_pretrained(
        cls,
        model_id,
        revision,
        cache_dir,
        force_download,
        proxies,
        resume_download,
        local_files_only,
        token,
        dataset_name='',
        model_name='',
        map_location='cpu',
        strict=False,
        **model_kwargs,
    ):
        map_location = torch.device(map_location)

        if os.path.isdir(model_id):
            model_file = os.path.join(model_id, MODEL_WEIGHTS_NAME)
        else:
            model_file = hf_hub_download(
                repo_id=model_id,
                filename=MODEL_WEIGHTS_NAME,
                revision=revision,
                cache_dir=cache_dir,
                force_download=force_download,
                proxies=proxies,
                resume_download=resume_download,
                token=token,
                local_files_only=local_files_only,
            )

        config = model_kwargs.pop('config', None)
        if config is not None:
            model_kwargs = {**model_kwargs, **config}

        model = cls(dataset_name, model_name, model_kwargs)

        state_dict = torch.load(model_file, map_location=map_location)
        model.load_state_dict(state_dict, strict=strict)
        model.eval()

        return model

    @classmethod
    def from_pretrained(
        cls,
        pretrained_model_name_or_path: str,
        force_download: bool = False,
        resume_download: bool = False,
        proxies: Optional[Dict] = None,
        token: Optional[Union[str, bool]] = None,
        cache_dir: Optional[str] = None,
        local_files_only: bool = False,
        **model_kwargs,
    ) -> Any:
        r"""Downloads and instantiates a model from the HuggingFace hub.

        Args:
            pretrained_model_name_or_path (str): Can be either:

                - The :obj:`model_id` of a pretrained model hosted inside the
                  HuggingFace hub.

                - You can add a :obj:`revision` by appending :obj:`@` at the
                  end of :obj:`model_id` to load a specific model version.

                - A path to a directory containing the saved model weights.

                - :obj:`None` if you are both providing the configuration
                  :obj:`config` and state dictionary :obj:`state_dict`.

            force_download (bool, optional): Whether to force the
                (re-)download of the model weights and configuration files,
                overriding the cached versions if they exist.
                (default: :obj:`False`)
            resume_download (bool, optional): Whether to delete incompletely
                received files. Will attempt to resume the download if such a
                file exists. (default: :obj:`False`)
            proxies (Dict[str, str], optional): A dictionary of proxy servers
                to use by protocol or endpoint, *e.g.*,
                :obj:`{'http': 'foo.bar:3128', 'http://host': 'foo.bar:4012'}`.
                The proxies are used on each request. (default: :obj:`None`)
            token (str or bool, optional): The token to use as HTTP bearer
                authorization for remote files. If set to :obj:`True`, will use
                the token generated when running :obj:`transformers-cli login`
                (stored in :obj:`~/.huggingface`). It is **required** if you
                want to use a private model. (default: :obj:`None`)
            cache_dir (str, optional): The path to a directory in which a
                downloaded model configuration should be cached if the
                standard cache should not be used. (default: :obj:`None`)
            local_files_only (bool, optional): Whether to only look at local
                files, *i.e.* do not try to download the model.
                (default: :obj:`False`)
            **model_kwargs: Additional keyword arguments passed to the
                model during initialization.
        """
        return super().from_pretrained(
            pretrained_model_name_or_path,
            force_download=force_download,
            resume_download=resume_download,
            proxies=proxies,
            use_auth_token=token,
            cache_dir=cache_dir,
            local_files_only=local_files_only,
            **model_kwargs,
        )
pytorch/torch_geometric/nn/models/000077500000000000000000000000001444262304300176235ustar00rootroot00000000000000pytorch/torch_geometric/nn/models/__init__.py000066400000000000000000000031571444262304300217420ustar00rootroot00000000000000from .mlp import MLP
from .basic_gnn import GCN, GraphSAGE, GIN, GAT, PNA, EdgeCNN
from .jumping_knowledge import JumpingKnowledge
from .meta import MetaLayer
from .node2vec import Node2Vec
from .deep_graph_infomax import DeepGraphInfomax
from .autoencoder import InnerProductDecoder, GAE, VGAE, ARGA, ARGVA
from .signed_gcn import SignedGCN
from .re_net import RENet
from .graph_unet import GraphUNet
from .schnet import SchNet
from .dimenet import DimeNet, DimeNetPlusPlus
from .captum import to_captum_model, to_captum_input, captum_output_to_dicts
from .metapath2vec import MetaPath2Vec
from .deepgcn import DeepGCNLayer
from .tgn import TGNMemory
from .label_prop import LabelPropagation
from .correct_and_smooth import CorrectAndSmooth
from .attentive_fp import AttentiveFP
from .rect import RECT_L
from .linkx import LINKX
from .lightgcn import LightGCN
from .mask_label import MaskLabel
from .rev_gnn import GroupAddRev
from .gnnff import GNNFF
from .pmlp import PMLP

__all__ = classes = [
    'MLP',
    'GCN',
    'GraphSAGE',
    'GIN',
    'GAT',
    'PNA',
    'EdgeCNN',
    'JumpingKnowledge',
    'MetaLayer',
    'Node2Vec',
    'DeepGraphInfomax',
    'InnerProductDecoder',
    'GAE',
    'VGAE',
    'ARGA',
    'ARGVA',
    'SignedGCN',
    'RENet',
    'GraphUNet',
    'SchNet',
    'DimeNet',
    'DimeNetPlusPlus',
    'to_captum_model',
    'to_captum_input',
    'captum_output_to_dicts',
    'MetaPath2Vec',
    'DeepGCNLayer',
    'TGNMemory',
    'LabelPropagation',
    'CorrectAndSmooth',
    'AttentiveFP',
    'RECT_L',
    'LINKX',
    'LightGCN',
    'MaskLabel',
    'GroupAddRev',
    'GNNFF',
    'PMLP',
]
pytorch/torch_geometric/nn/models/attentive_fp.py000066400000000000000000000146771444262304300227040ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import GRUCell, Linear, Parameter

from torch_geometric.nn import GATConv, MessagePassing, global_add_pool
from torch_geometric.nn.inits import glorot, zeros
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils import softmax


class GATEConv(MessagePassing):
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        edge_dim: int,
        dropout: float = 0.0,
    ):
        super().__init__(aggr='add', node_dim=0)

        self.dropout = dropout

        self.att_l = Parameter(torch.Tensor(1, out_channels))
        self.att_r = Parameter(torch.Tensor(1, in_channels))

        self.lin1 = Linear(in_channels + edge_dim, out_channels, False)
        self.lin2 = Linear(out_channels, out_channels, False)

        self.bias = Parameter(torch.Tensor(out_channels))

        self.reset_parameters()

    def reset_parameters(self):
        glorot(self.att_l)
        glorot(self.att_r)
        glorot(self.lin1.weight)
        glorot(self.lin2.weight)
        zeros(self.bias)

    def forward(self, x: Tensor, edge_index: Adj, edge_attr: Tensor) -> Tensor:
        # propagate_type: (x: Tensor, edge_attr: Tensor)
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=None)
        out = out + self.bias
        return out

    def message(self, x_j: Tensor, x_i: Tensor, edge_attr: Tensor,
                index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:

        x_j = F.leaky_relu_(self.lin1(torch.cat([x_j, edge_attr], dim=-1)))
        alpha_j = (x_j * self.att_l).sum(dim=-1)
        alpha_i = (x_i * self.att_r).sum(dim=-1)
        alpha = alpha_j + alpha_i
        alpha = F.leaky_relu_(alpha)
        alpha = softmax(alpha, index, ptr, size_i)
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return self.lin2(x_j) * alpha.unsqueeze(-1)


class AttentiveFP(torch.nn.Module):
    r"""The Attentive FP model for molecular representation learning from the
    `"Pushing the Boundaries of Molecular Representation for Drug Discovery
    with the Graph Attention Mechanism"
    `_ paper, based on
    graph attention mechanisms.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Hidden node feature dimensionality.
        out_channels (int): Size of each output sample.
        edge_dim (int): Edge feature dimensionality.
        num_layers (int): Number of GNN layers.
        num_timesteps (int): Number of iterative refinement steps for global
            readout.
        dropout (float, optional): Dropout probability. (default: :obj:`0.0`)

    """
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        out_channels: int,
        edge_dim: int,
        num_layers: int,
        num_timesteps: int,
        dropout: float = 0.0,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.out_channels = out_channels
        self.edge_dim = edge_dim
        self.num_layers = num_layers
        self.num_timesteps = num_timesteps
        self.dropout = dropout

        self.lin1 = Linear(in_channels, hidden_channels)

        self.gate_conv = GATEConv(hidden_channels, hidden_channels, edge_dim,
                                  dropout)
        self.gru = GRUCell(hidden_channels, hidden_channels)

        self.atom_convs = torch.nn.ModuleList()
        self.atom_grus = torch.nn.ModuleList()
        for _ in range(num_layers - 1):
            conv = GATConv(hidden_channels, hidden_channels, dropout=dropout,
                           add_self_loops=False, negative_slope=0.01)
            self.atom_convs.append(conv)
            self.atom_grus.append(GRUCell(hidden_channels, hidden_channels))

        self.mol_conv = GATConv(hidden_channels, hidden_channels,
                                dropout=dropout, add_self_loops=False,
                                negative_slope=0.01)
        self.mol_gru = GRUCell(hidden_channels, hidden_channels)

        self.lin2 = Linear(hidden_channels, out_channels)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin1.reset_parameters()
        self.gate_conv.reset_parameters()
        self.gru.reset_parameters()
        for conv, gru in zip(self.atom_convs, self.atom_grus):
            conv.reset_parameters()
            gru.reset_parameters()
        self.mol_conv.reset_parameters()
        self.mol_gru.reset_parameters()
        self.lin2.reset_parameters()

    def forward(self, x: Tensor, edge_index: Tensor, edge_attr: Tensor,
                batch: Tensor) -> Tensor:
        """"""
        # Atom Embedding:
        x = F.leaky_relu_(self.lin1(x))

        h = F.elu_(self.gate_conv(x, edge_index, edge_attr))
        h = F.dropout(h, p=self.dropout, training=self.training)
        x = self.gru(h, x).relu_()

        for conv, gru in zip(self.atom_convs, self.atom_grus):
            h = F.elu_(conv(x, edge_index))
            h = F.dropout(h, p=self.dropout, training=self.training)
            x = gru(h, x).relu_()

        # Molecule Embedding:
        row = torch.arange(batch.size(0), device=batch.device)
        edge_index = torch.stack([row, batch], dim=0)

        out = global_add_pool(x, batch).relu_()
        for t in range(self.num_timesteps):
            h = F.elu_(self.mol_conv((x, out), edge_index))
            h = F.dropout(h, p=self.dropout, training=self.training)
            out = self.mol_gru(h, out).relu_()

        # Predictor:
        out = F.dropout(out, p=self.dropout, training=self.training)
        return self.lin2(out)

    def jittable(self) -> 'AttentiveFP':
        self.gate_conv = self.gate_conv.jittable()
        self.atom_convs = torch.nn.ModuleList(
            [conv.jittable() for conv in self.atom_convs])
        self.mol_conv = self.mol_conv.jittable()
        return self

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'in_channels={self.in_channels}, '
                f'hidden_channels={self.hidden_channels}, '
                f'out_channels={self.out_channels}, '
                f'edge_dim={self.edge_dim}, '
                f'num_layers={self.num_layers}, '
                f'num_timesteps={self.num_timesteps}'
                f')')
pytorch/torch_geometric/nn/models/autoencoder.py000066400000000000000000000246401444262304300225130ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Module

from torch_geometric.nn.inits import reset
from torch_geometric.utils import negative_sampling

EPS = 1e-15
MAX_LOGSTD = 10


class InnerProductDecoder(torch.nn.Module):
    r"""The inner product decoder from the `"Variational Graph Auto-Encoders"
    `_ paper

    .. math::
        \sigma(\mathbf{Z}\mathbf{Z}^{\top})

    where :math:`\mathbf{Z} \in \mathbb{R}^{N \times d}` denotes the latent
    space produced by the encoder."""
    def forward(self, z: Tensor, edge_index: Tensor,
                sigmoid: bool = True) -> Tensor:
        r"""Decodes the latent variables :obj:`z` into edge probabilities for
        the given node-pairs :obj:`edge_index`.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
            sigmoid (bool, optional): If set to :obj:`False`, does not apply
                the logistic sigmoid function to the output.
                (default: :obj:`True`)
        """
        value = (z[edge_index[0]] * z[edge_index[1]]).sum(dim=1)
        return torch.sigmoid(value) if sigmoid else value

    def forward_all(self, z: Tensor, sigmoid: bool = True) -> Tensor:
        r"""Decodes the latent variables :obj:`z` into a probabilistic dense
        adjacency matrix.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
            sigmoid (bool, optional): If set to :obj:`False`, does not apply
                the logistic sigmoid function to the output.
                (default: :obj:`True`)
        """
        adj = torch.matmul(z, z.t())
        return torch.sigmoid(adj) if sigmoid else adj


class GAE(torch.nn.Module):
    r"""The Graph Auto-Encoder model from the
    `"Variational Graph Auto-Encoders" `_
    paper based on user-defined encoder and decoder models.

    Args:
        encoder (torch.nn.Module): The encoder module.
        decoder (torch.nn.Module, optional): The decoder module. If set to
            :obj:`None`, will default to the
            :class:`torch_geometric.nn.models.InnerProductDecoder`.
            (default: :obj:`None`)
    """
    def __init__(self, encoder: Module, decoder: Optional[Module] = None):
        super().__init__()
        self.encoder = encoder
        self.decoder = InnerProductDecoder() if decoder is None else decoder
        GAE.reset_parameters(self)

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset(self.encoder)
        reset(self.decoder)

    def forward(self, *args, **kwargs) -> Tensor:  # pragma: no cover
        r"""Alias for :meth:`encode`."""
        return self.encoder(*args, **kwargs)

    def encode(self, *args, **kwargs) -> Tensor:
        r"""Runs the encoder and computes node-wise latent variables."""
        return self.encoder(*args, **kwargs)

    def decode(self, *args, **kwargs) -> Tensor:
        r"""Runs the decoder and computes edge probabilities."""
        return self.decoder(*args, **kwargs)

    def recon_loss(self, z: Tensor, pos_edge_index: Tensor,
                   neg_edge_index: Optional[Tensor] = None) -> Tensor:
        r"""Given latent variables :obj:`z`, computes the binary cross
        entropy loss for positive edges :obj:`pos_edge_index` and negative
        sampled edges.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
            pos_edge_index (torch.Tensor): The positive edges to train against.
            neg_edge_index (torch.Tensor, optional): The negative edges to
                train against. If not given, uses negative sampling to
                calculate negative edges. (default: :obj:`None`)
        """
        pos_loss = -torch.log(
            self.decoder(z, pos_edge_index, sigmoid=True) + EPS).mean()

        if neg_edge_index is None:
            neg_edge_index = negative_sampling(pos_edge_index, z.size(0))
        neg_loss = -torch.log(1 -
                              self.decoder(z, neg_edge_index, sigmoid=True) +
                              EPS).mean()

        return pos_loss + neg_loss

    def test(self, z: Tensor, pos_edge_index: Tensor,
             neg_edge_index: Tensor) -> Tuple[Tensor, Tensor]:
        r"""Given latent variables :obj:`z`, positive edges
        :obj:`pos_edge_index` and negative edges :obj:`neg_edge_index`,
        computes area under the ROC curve (AUC) and average precision (AP)
        scores.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
            pos_edge_index (torch.Tensor): The positive edges to evaluate
                against.
            neg_edge_index (torch.Tensor): The negative edges to evaluate
                against.
        """
        from sklearn.metrics import average_precision_score, roc_auc_score

        pos_y = z.new_ones(pos_edge_index.size(1))
        neg_y = z.new_zeros(neg_edge_index.size(1))
        y = torch.cat([pos_y, neg_y], dim=0)

        pos_pred = self.decoder(z, pos_edge_index, sigmoid=True)
        neg_pred = self.decoder(z, neg_edge_index, sigmoid=True)
        pred = torch.cat([pos_pred, neg_pred], dim=0)

        y, pred = y.detach().cpu().numpy(), pred.detach().cpu().numpy()

        return roc_auc_score(y, pred), average_precision_score(y, pred)


class VGAE(GAE):
    r"""The Variational Graph Auto-Encoder model from the
    `"Variational Graph Auto-Encoders" `_
    paper.

    Args:
        encoder (torch.nn.Module): The encoder module to compute :math:`\mu`
            and :math:`\log\sigma^2`.
        decoder (torch.nn.Module, optional): The decoder module. If set to
            :obj:`None`, will default to the
            :class:`torch_geometric.nn.models.InnerProductDecoder`.
            (default: :obj:`None`)
    """
    def __init__(self, encoder: Module, decoder: Optional[Module] = None):
        super().__init__(encoder, decoder)

    def reparametrize(self, mu: Tensor, logstd: Tensor) -> Tensor:
        if self.training:
            return mu + torch.randn_like(logstd) * torch.exp(logstd)
        else:
            return mu

    def encode(self, *args, **kwargs) -> Tensor:
        """"""
        self.__mu__, self.__logstd__ = self.encoder(*args, **kwargs)
        self.__logstd__ = self.__logstd__.clamp(max=MAX_LOGSTD)
        z = self.reparametrize(self.__mu__, self.__logstd__)
        return z

    def kl_loss(self, mu: Optional[Tensor] = None,
                logstd: Optional[Tensor] = None) -> Tensor:
        r"""Computes the KL loss, either for the passed arguments :obj:`mu`
        and :obj:`logstd`, or based on latent variables from last encoding.

        Args:
            mu (torch.Tensor, optional): The latent space for :math:`\mu`. If
                set to :obj:`None`, uses the last computation of :math:`\mu`.
                (default: :obj:`None`)
            logstd (torch.Tensor, optional): The latent space for
                :math:`\log\sigma`.  If set to :obj:`None`, uses the last
                computation of :math:`\log\sigma^2`. (default: :obj:`None`)
        """
        mu = self.__mu__ if mu is None else mu
        logstd = self.__logstd__ if logstd is None else logstd.clamp(
            max=MAX_LOGSTD)
        return -0.5 * torch.mean(
            torch.sum(1 + 2 * logstd - mu**2 - logstd.exp()**2, dim=1))


class ARGA(GAE):
    r"""The Adversarially Regularized Graph Auto-Encoder model from the
    `"Adversarially Regularized Graph Autoencoder for Graph Embedding"
    `_ paper.

    Args:
        encoder (torch.nn.Module): The encoder module.
        discriminator (torch.nn.Module): The discriminator module.
        decoder (torch.nn.Module, optional): The decoder module. If set to
            :obj:`None`, will default to the
            :class:`torch_geometric.nn.models.InnerProductDecoder`.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        encoder: Module,
        discriminator: Module,
        decoder: Optional[Module] = None,
    ):
        super().__init__(encoder, decoder)
        self.discriminator = discriminator
        reset(self.discriminator)

    def reset_parameters(self):
        super().reset_parameters()
        reset(self.discriminator)

    def reg_loss(self, z: Tensor) -> Tensor:
        r"""Computes the regularization loss of the encoder.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
        """
        real = torch.sigmoid(self.discriminator(z))
        real_loss = -torch.log(real + EPS).mean()
        return real_loss

    def discriminator_loss(self, z: Tensor) -> Tensor:
        r"""Computes the loss of the discriminator.

        Args:
            z (torch.Tensor): The latent space :math:`\mathbf{Z}`.
        """
        real = torch.sigmoid(self.discriminator(torch.randn_like(z)))
        fake = torch.sigmoid(self.discriminator(z.detach()))
        real_loss = -torch.log(real + EPS).mean()
        fake_loss = -torch.log(1 - fake + EPS).mean()
        return real_loss + fake_loss


class ARGVA(ARGA):
    r"""The Adversarially Regularized Variational Graph Auto-Encoder model from
    the `"Adversarially Regularized Graph Autoencoder for Graph Embedding"
    `_ paper.

    Args:
        encoder (torch.nn.Module): The encoder module to compute :math:`\mu`
            and :math:`\log\sigma^2`.
        discriminator (torch.nn.Module): The discriminator module.
        decoder (torch.nn.Module, optional): The decoder module. If set to
            :obj:`None`, will default to the
            :class:`torch_geometric.nn.models.InnerProductDecoder`.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        encoder: Module,
        discriminator: Module,
        decoder: Optional[Module] = None,
    ):
        super().__init__(encoder, discriminator, decoder)
        self.VGAE = VGAE(encoder, decoder)

    @property
    def __mu__(self) -> Tensor:
        return self.VGAE.__mu__

    @property
    def __logstd__(self) -> Tensor:
        return self.VGAE.__logstd__

    def reparametrize(self, mu: Tensor, logstd: Tensor) -> Tensor:
        return self.VGAE.reparametrize(mu, logstd)

    def encode(self, *args, **kwargs) -> Tensor:
        """"""
        return self.VGAE.encode(*args, **kwargs)

    def kl_loss(
        self,
        mu: Optional[Tensor] = None,
        logstd: Optional[Tensor] = None,
    ) -> Tensor:
        return self.VGAE.kl_loss(mu, logstd)
pytorch/torch_geometric/nn/models/basic_gnn.py000066400000000000000000000677201444262304300221340ustar00rootroot00000000000000import copy
from typing import Any, Callable, Dict, List, Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear, ModuleList
from tqdm import tqdm

from torch_geometric.loader import NeighborLoader
from torch_geometric.nn.conv import (
    EdgeConv,
    GATConv,
    GATv2Conv,
    GCNConv,
    GINConv,
    MessagePassing,
    PNAConv,
    SAGEConv,
)
from torch_geometric.nn.models import MLP
from torch_geometric.nn.models.jumping_knowledge import JumpingKnowledge
from torch_geometric.nn.resolver import (
    activation_resolver,
    normalization_resolver,
)
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils.trim_to_layer import TrimToLayer


class BasicGNN(torch.nn.Module):
    r"""An abstract class for implementing basic GNN models.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of the underlying
            :class:`torch_geometric.nn.conv.MessagePassing` layers.
    """
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        num_layers: int,
        out_channels: Optional[int] = None,
        dropout: float = 0.0,
        act: Union[str, Callable, None] = "relu",
        act_first: bool = False,
        act_kwargs: Optional[Dict[str, Any]] = None,
        norm: Union[str, Callable, None] = None,
        norm_kwargs: Optional[Dict[str, Any]] = None,
        jk: Optional[str] = None,
        **kwargs,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.num_layers = num_layers

        self.dropout = dropout
        self.act = activation_resolver(act, **(act_kwargs or {}))
        self.jk_mode = jk
        self.act_first = act_first
        self.norm = norm if isinstance(norm, str) else None
        self.norm_kwargs = norm_kwargs

        if out_channels is not None:
            self.out_channels = out_channels
        else:
            self.out_channels = hidden_channels

        self.convs = ModuleList()
        if num_layers > 1:
            self.convs.append(
                self.init_conv(in_channels, hidden_channels, **kwargs))
            if isinstance(in_channels, (tuple, list)):
                in_channels = (hidden_channels, hidden_channels)
            else:
                in_channels = hidden_channels
        for _ in range(num_layers - 2):
            self.convs.append(
                self.init_conv(in_channels, hidden_channels, **kwargs))
            if isinstance(in_channels, (tuple, list)):
                in_channels = (hidden_channels, hidden_channels)
            else:
                in_channels = hidden_channels
        if out_channels is not None and jk is None:
            self._is_conv_to_out = True
            self.convs.append(
                self.init_conv(in_channels, out_channels, **kwargs))
        else:
            self.convs.append(
                self.init_conv(in_channels, hidden_channels, **kwargs))

        self.norms = None
        if norm is not None:
            norm_layer = normalization_resolver(
                norm,
                hidden_channels,
                **(norm_kwargs or {}),
            )
            self.norms = ModuleList()
            for _ in range(num_layers - 1):
                self.norms.append(copy.deepcopy(norm_layer))
            if jk is not None:
                self.norms.append(copy.deepcopy(norm_layer))

        if jk is not None and jk != 'last':
            self.jk = JumpingKnowledge(jk, hidden_channels, num_layers)

        if jk is not None:
            if jk == 'cat':
                in_channels = num_layers * hidden_channels
            else:
                in_channels = hidden_channels
            self.lin = Linear(in_channels, self.out_channels)

        # We define `trim_to_layer` functionality as a module such that we can
        # still use `to_hetero` on-top.
        self._trim = TrimToLayer()

    def init_conv(self, in_channels: Union[int, Tuple[int, int]],
                  out_channels: int, **kwargs) -> MessagePassing:
        raise NotImplementedError

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for conv in self.convs:
            conv.reset_parameters()
        for norm in self.norms or []:
            norm.reset_parameters()
        if hasattr(self, 'jk'):
            self.jk.reset_parameters()
        if hasattr(self, 'lin'):
            self.lin.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Adj,
        *,
        edge_weight: OptTensor = None,
        edge_attr: OptTensor = None,
        num_sampled_nodes_per_hop: Optional[List[int]] = None,
        num_sampled_edges_per_hop: Optional[List[int]] = None,
    ) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The input node features.
            edge_index (torch.Tensor): The edge indices.
            edge_weight (torch.Tensor, optional): The edge weights (if
                supported by the underlying GNN layer). (default: :obj:`None`)
            edge_attr (torch.Tensor, optional): The edge features (if supported
                by the underlying GNN layer). (default: :obj:`None`)
            num_sampled_nodes_per_hop (List[int], optional): The number of
                sampled nodes per hop.
                Useful in :class:~torch_geometric.loader.NeighborLoader`
                scenarios to only operate on minimal-sized representations.
                (default: :obj:`None`)
            num_sampled_edges_per_hop (List[int], optional): The number of
                sampled edges per hop.
                Useful in :class:~torch_geometric.loader.NeighborLoader`
                scenarios to only operate on minimal-sized representations.
                (default: :obj:`None`)
        """
        if (num_sampled_nodes_per_hop is not None
                and isinstance(edge_weight, Tensor)
                and isinstance(edge_attr, Tensor)):
            raise NotImplementedError("'trim_to_layer' functionality does not "
                                      "yet support trimming of both "
                                      "'edge_weight' and 'edge_attr'")

        xs: List[Tensor] = []
        for i in range(self.num_layers):
            if num_sampled_nodes_per_hop is not None:
                x, edge_index, value = self._trim(
                    i,
                    num_sampled_nodes_per_hop,
                    num_sampled_edges_per_hop,
                    x,
                    edge_index,
                    edge_weight if edge_weight is not None else edge_attr,
                )
                if edge_weight is not None:
                    edge_weight = value
                else:
                    edge_attr = value

            # Tracing the module is not allowed with *args and **kwargs :(
            # As such, we rely on a static solution to pass optional edge
            # weights and edge attributes to the module.
            if self.supports_edge_weight and self.supports_edge_attr:
                x = self.convs[i](x, edge_index, edge_weight=edge_weight,
                                  edge_attr=edge_attr)
            elif self.supports_edge_weight:
                x = self.convs[i](x, edge_index, edge_weight=edge_weight)
            elif self.supports_edge_attr:
                x = self.convs[i](x, edge_index, edge_attr=edge_attr)
            else:
                x = self.convs[i](x, edge_index)
            if i == self.num_layers - 1 and self.jk_mode is None:
                break
            if self.act is not None and self.act_first:
                x = self.act(x)
            if self.norms is not None:
                x = self.norms[i](x)
            if self.act is not None and not self.act_first:
                x = self.act(x)
            x = F.dropout(x, p=self.dropout, training=self.training)
            if hasattr(self, 'jk'):
                xs.append(x)

        x = self.jk(xs) if hasattr(self, 'jk') else x
        x = self.lin(x) if hasattr(self, 'lin') else x
        return x

    @torch.no_grad()
    def inference(
        self,
        loader: NeighborLoader,
        device: Optional[Union[str, torch.device]] = None,
        embedding_device: Union[str, torch.device] = 'cpu',
        progress_bar: bool = False,
    ) -> Tensor:
        r"""Performs layer-wise inference on large-graphs using a
        :class:`~torch_geometric.loader.NeighborLoader`, where
        :class:`~torch_geometric.loader.NeighborLoader` should sample the
        full neighborhood for only one layer.
        This is an efficient way to compute the output embeddings for all
        nodes in the graph.
        Only applicable in case :obj:`jk=None` or `jk='last'`.

        Args:
            loader (torch_geometric.loader.NeighborLoader): A neighbor loader
                object that generates full 1-hop subgraphs, *i.e.*,
                :obj:`loader.num_neighbors = [-1]`.
            device (torch.device, optional): The device to run the GNN on.
                (default: :obj:`None`)
            embedding_device (torch.device, optional): The device to store
                intermediate embeddings on. If intermediate embeddings fit on
                GPU, this option helps to avoid unnecessary device transfers.
                (default: :obj:`"cpu"`)
            progress_bar (bool, optional): If set to :obj:`True`, will print a
                progress bar during computation. (default: :obj:`False`)
        """
        assert self.jk_mode is None or self.jk_mode == 'last'
        assert isinstance(loader, NeighborLoader)
        assert len(loader.dataset) == loader.data.num_nodes
        assert len(loader.node_sampler.num_neighbors) == 1
        assert not self.training
        # assert not loader.shuffle  # TODO (matthias) does not work :(
        if progress_bar:
            pbar = tqdm(total=len(self.convs) * len(loader))
            pbar.set_description('Inference')

        x_all = loader.data.x.to(embedding_device)
        loader.data.n_id = torch.arange(x_all.size(0))

        for i in range(self.num_layers):
            xs: List[Tensor] = []
            for batch in loader:
                x = x_all[batch.n_id].to(device)
                if hasattr(batch, 'adj_t'):
                    edge_index = batch.adj_t.to(device)
                else:
                    edge_index = batch.edge_index.to(device)
                x = self.convs[i](x, edge_index)[:batch.batch_size]
                if i == self.num_layers - 1 and self.jk_mode is None:
                    xs.append(x.to(embedding_device))
                    if progress_bar:
                        pbar.update(1)
                    continue
                if self.act is not None and self.act_first:
                    x = self.act(x)
                if self.norms is not None:
                    x = self.norms[i](x)
                if self.act is not None and not self.act_first:
                    x = self.act(x)
                if i == self.num_layers - 1 and hasattr(self, 'lin'):
                    x = self.lin(x)
                xs.append(x.to(embedding_device))
                if progress_bar:
                    pbar.update(1)
            x_all = torch.cat(xs, dim=0)
        if progress_bar:
            pbar.close()
        del loader.data.n_id

        return x_all

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_layers={self.num_layers})')


class GCN(BasicGNN):
    r"""The Graph Neural Network from the `"Semi-supervised
    Classification with Graph Convolutional Networks"
    `_ paper, using the
    :class:`~torch_geometric.nn.conv.GCNConv` operator for message passing.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.GCNConv`.
    """
    supports_edge_weight = True
    supports_edge_attr = False

    def init_conv(self, in_channels: int, out_channels: int,
                  **kwargs) -> MessagePassing:
        return GCNConv(in_channels, out_channels, **kwargs)


class GraphSAGE(BasicGNN):
    r"""The Graph Neural Network from the `"Inductive Representation Learning
    on Large Graphs" `_ paper, using the
    :class:`~torch_geometric.nn.SAGEConv` operator for message passing.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.SAGEConv`.
    """
    supports_edge_weight = False
    supports_edge_attr = False

    def init_conv(self, in_channels: Union[int, Tuple[int, int]],
                  out_channels: int, **kwargs) -> MessagePassing:
        return SAGEConv(in_channels, out_channels, **kwargs)


class GIN(BasicGNN):
    r"""The Graph Neural Network from the `"How Powerful are Graph Neural
    Networks?" `_ paper, using the
    :class:`~torch_geometric.nn.GINConv` operator for message passing.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.GINConv`.
    """
    supports_edge_weight = False
    supports_edge_attr = False

    def init_conv(self, in_channels: int, out_channels: int,
                  **kwargs) -> MessagePassing:
        mlp = MLP(
            [in_channels, out_channels, out_channels],
            act=self.act,
            act_first=self.act_first,
            norm=self.norm,
            norm_kwargs=self.norm_kwargs,
        )
        return GINConv(mlp, **kwargs)


class GAT(BasicGNN):
    r"""The Graph Neural Network from `"Graph Attention Networks"
    `_ or `"How Attentive are Graph Attention
    Networks?" `_ papers, using the
    :class:`~torch_geometric.nn.GATConv` or
    :class:`~torch_geometric.nn.GATv2Conv` operator for message passing,
    respectively.

    Args:
        in_channels (int or tuple): Size of each input sample, or :obj:`-1` to
            derive the size from the first input(s) to the forward method.
            A tuple corresponds to the sizes of source and target
            dimensionalities.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        v2 (bool, optional): If set to :obj:`True`, will make use of
            :class:`~torch_geometric.nn.conv.GATv2Conv` rather than
            :class:`~torch_geometric.nn.conv.GATConv`. (default: :obj:`False`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.GATConv` or
            :class:`torch_geometric.nn.conv.GATv2Conv`.
    """
    supports_edge_weight = False
    supports_edge_attr = True

    def init_conv(self, in_channels: Union[int, Tuple[int, int]],
                  out_channels: int, **kwargs) -> MessagePassing:

        v2 = kwargs.pop('v2', False)
        heads = kwargs.pop('heads', 1)
        concat = kwargs.pop('concat', True)

        # Do not use concatenation in case the layer `GATConv` layer maps to
        # the desired output channels (out_channels != None and jk != None):
        if getattr(self, '_is_conv_to_out', False):
            concat = False

        if concat and out_channels % heads != 0:
            raise ValueError(f"Ensure that the number of output channels of "
                             f"'GATConv' (got '{out_channels}') is divisible "
                             f"by the number of heads (got '{heads}')")

        if concat:
            out_channels = out_channels // heads

        Conv = GATConv if not v2 else GATv2Conv
        return Conv(in_channels, out_channels, heads=heads, concat=concat,
                    dropout=self.dropout, **kwargs)


class PNA(BasicGNN):
    r"""The Graph Neural Network from the `"Principal Neighbourhood Aggregation
    for Graph Nets" `_ paper, using the
    :class:`~torch_geometric.nn.conv.PNAConv` operator for message passing.

    Args:
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.PNAConv`.
    """
    supports_edge_weight = False
    supports_edge_attr = True

    def init_conv(self, in_channels: int, out_channels: int,
                  **kwargs) -> MessagePassing:
        return PNAConv(in_channels, out_channels, **kwargs)


class EdgeCNN(BasicGNN):
    r"""The Graph Neural Network from the `"Dynamic Graph CNN for Learning on
    Point Clouds" `_ paper, using the
    :class:`~torch_geometric.nn.conv.EdgeConv` operator for message passing.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of message passing layers.
        out_channels (int, optional): If not set to :obj:`None`, will apply a
            final linear transformation to convert hidden node embeddings to
            output size :obj:`out_channels`. (default: :obj:`None`)
        dropout (float, optional): Dropout probability. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`None`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        jk (str, optional): The Jumping Knowledge mode. If specified, the model
            will additionally apply a final linear transformation to transform
            node embeddings to the expected output feature dimensionality.
            (:obj:`None`, :obj:`"last"`, :obj:`"cat"`, :obj:`"max"`,
            :obj:`"lstm"`). (default: :obj:`None`)
        **kwargs (optional): Additional arguments of
            :class:`torch_geometric.nn.conv.EdgeConv`.
    """
    supports_edge_weight = False
    supports_edge_attr = False

    def init_conv(self, in_channels: int, out_channels: int,
                  **kwargs) -> MessagePassing:
        mlp = MLP(
            [2 * in_channels, out_channels, out_channels],
            act=self.act,
            act_first=self.act_first,
            norm=self.norm,
            norm_kwargs=self.norm_kwargs,
        )
        return EdgeConv(mlp, **kwargs)


__all__ = ['GCN', 'GraphSAGE', 'GIN', 'GAT', 'PNA', 'EdgeCNN']
pytorch/torch_geometric/nn/models/captum.py000066400000000000000000000100521444262304300214640ustar00rootroot00000000000000from typing import Optional, Union

import torch

from torch_geometric.explain.algorithm.captum import \
    captum_output_to_dicts  # noqa
from torch_geometric.explain.algorithm.captum import to_captum_input  # noqa
from torch_geometric.explain.algorithm.captum import (
    CaptumHeteroModel,
    CaptumModel,
    MaskLevelType,
)
from torch_geometric.typing import Metadata


def to_captum_model(
    model: torch.nn.Module,
    mask_type: Union[str, MaskLevelType] = MaskLevelType.edge,
    output_idx: Optional[int] = None,
    metadata: Optional[Metadata] = None,
) -> Union[CaptumModel, CaptumHeteroModel]:
    r"""Converts a model to a model that can be used for
    `Captum `_ attribution methods.

    Sample code for homogeneous graphs:

    .. code-block:: python

        from captum.attr import IntegratedGradients

        from torch_geometric.data import Data
        from torch_geometric.nn import GCN
        from torch_geometric.nn import to_captum_model, to_captum_input

        data = Data(x=(...), edge_index(...))
        model = GCN(...)
        ...  # Train the model.

        # Explain predictions for node `10`:
        mask_type="edge"
        output_idx = 10
        captum_model = to_captum_model(model, mask_type, output_idx)
        inputs, additional_forward_args = to_captum_input(data.x,
                                            data.edge_index,mask_type)

        ig = IntegratedGradients(captum_model)
        ig_attr = ig.attribute(inputs = inputs,
                               target=int(y[output_idx]),
                               additional_forward_args=additional_forward_args,
                               internal_batch_size=1)


    Sample code for heterogeneous graphs:

    .. code-block:: python

        from captum.attr import IntegratedGradients

        from torch_geometric.data import HeteroData
        from torch_geometric.nn import HeteroConv
        from torch_geometric.nn import (captum_output_to_dicts,
                                        to_captum_model, to_captum_input)

        data = HeteroData(...)
        model = HeteroConv(...)
        ...  # Train the model.

        # Explain predictions for node `10`:
        mask_type="edge"
        metadata = data.metadata
        output_idx = 10
        captum_model = to_captum_model(model, mask_type, output_idx, metadata)
        inputs, additional_forward_args = to_captum_input(data.x_dict,
                                            data.edge_index_dict, mask_type)

        ig = IntegratedGradients(captum_model)
        ig_attr = ig.attribute(inputs=inputs,
                               target=int(y[output_idx]),
                               additional_forward_args=additional_forward_args,
                               internal_batch_size=1)
        edge_attr_dict = captum_output_to_dicts(ig_attr, mask_type, metadata)


    .. note::
        For an example of using a :captum:`Captum` attribution method within
        :pyg:`PyG`, see `examples/explain/captum_explainer.py
        `_.

    Args:
        model (torch.nn.Module): The model to be explained.
        mask_type (str, optional): Denotes the type of mask to be created with
            a :captum:`Captum` explainer. Valid inputs are :obj:`"edge"`,
            :obj:`"node"`, and :obj:`"node_and_edge"`. (default: :obj:`"edge"`)
        output_idx (int, optional): Index of the output element (node or link
            index) to be explained. With :obj:`output_idx` set, the forward
            function will return the output of the model for the element at
            the index specified. (default: :obj:`None`)
        metadata (Metadata, optional): The metadata of the heterogeneous graph.
            Only required if explaning a
            :class:`~torch_geometric.data.HeteroData` object.
            (default: :obj:`None`)
    """
    if metadata is None:
        return CaptumModel(model, mask_type, output_idx)
    else:
        return CaptumHeteroModel(model, mask_type, output_idx, metadata)
pytorch/torch_geometric/nn/models/correct_and_smooth.py000066400000000000000000000152171444262304300240570ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.nn.models import LabelPropagation
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils import one_hot


class CorrectAndSmooth(torch.nn.Module):
    r"""The correct and smooth (C&S) post-processing model from the
    `"Combining Label Propagation And Simple Models Out-performs Graph Neural
    Networks"
    `_ paper, where soft predictions
    :math:`\mathbf{Z}` (obtained from a simple base predictor) are
    first corrected based on ground-truth training
    label information :math:`\mathbf{Y}` and residual propagation

    .. math::
        \mathbf{e}^{(0)}_i &= \begin{cases}
            \mathbf{y}_i - \mathbf{z}_i, & \text{if }i
            \text{ is training node,}\\
            \mathbf{0}, & \text{else}
        \end{cases}

    .. math::
        \mathbf{E}^{(\ell)} &= \alpha_1 \mathbf{D}^{-1/2}\mathbf{A}
        \mathbf{D}^{-1/2} \mathbf{E}^{(\ell - 1)} +
        (1 - \alpha_1) \mathbf{E}^{(\ell - 1)}

        \mathbf{\hat{Z}} &= \mathbf{Z} + \gamma \cdot \mathbf{E}^{(L_1)},

    where :math:`\gamma` denotes the scaling factor (either fixed or
    automatically determined), and then smoothed over the graph via label
    propagation

    .. math::
        \mathbf{\hat{z}}^{(0)}_i &= \begin{cases}
            \mathbf{y}_i, & \text{if }i\text{ is training node,}\\
            \mathbf{\hat{z}}_i, & \text{else}
        \end{cases}

    .. math::
        \mathbf{\hat{Z}}^{(\ell)} = \alpha_2 \mathbf{D}^{-1/2}\mathbf{A}
        \mathbf{D}^{-1/2} \mathbf{\hat{Z}}^{(\ell - 1)} +
        (1 - \alpha_2) \mathbf{\hat{Z}}^{(\ell - 1)}

    to obtain the final prediction :math:`\mathbf{\hat{Z}}^{(L_2)}`.

    .. note::

        For an example of using the C&S model, see
        `examples/correct_and_smooth.py
        `_.

    Args:
        num_correction_layers (int): The number of propagations :math:`L_1`.
        correction_alpha (float): The :math:`\alpha_1` coefficient.
        num_smoothing_layers (int): The number of propagations :math:`L_2`.
        smoothing_alpha (float): The :math:`\alpha_2` coefficient.
        autoscale (bool, optional): If set to :obj:`True`, will automatically
            determine the scaling factor :math:`\gamma`. (default: :obj:`True`)
        scale (float, optional): The scaling factor :math:`\gamma`, in case
            :obj:`autoscale = False`. (default: :obj:`1.0`)
    """
    def __init__(self, num_correction_layers: int, correction_alpha: float,
                 num_smoothing_layers: int, smoothing_alpha: float,
                 autoscale: bool = True, scale: float = 1.0):
        super().__init__()
        self.autoscale = autoscale
        self.scale = scale

        self.prop1 = LabelPropagation(num_correction_layers, correction_alpha)
        self.prop2 = LabelPropagation(num_smoothing_layers, smoothing_alpha)

    def forward(self, y_soft: Tensor, *args) -> Tensor:  # pragma: no cover
        r"""Applies both :meth:`correct` and :meth:`smooth`."""
        y_soft = self.correct(y_soft, *args)
        return self.smooth(y_soft, *args)

    def correct(self, y_soft: Tensor, y_true: Tensor, mask: Tensor,
                edge_index: Adj, edge_weight: OptTensor = None) -> Tensor:
        r"""
        Args:
            y_soft (torch.Tensor): The soft predictions :math:`\mathbf{Z}`
                obtained from a simple base predictor.
            y_true (torch.Tensor): The ground-truth label information
                :math:`\mathbf{Y}` of training nodes.
            mask (torch.Tensor): A mask or index tensor denoting which nodes
                were used for training.
            edge_index (torch.Tensor or SparseTensor): The edge connectivity.
            edge_weight (torch.Tensor, optional): The edge weights.
                (default: :obj:`None`)
        """

        numel = int(mask.sum()) if mask.dtype == torch.bool else mask.size(0)
        assert y_true.size(0) == numel

        if y_true.dtype == torch.long and y_true.size(0) == y_true.numel():
            y_true = one_hot(y_true.view(-1), num_classes=y_soft.size(-1),
                             dtype=y_soft.dtype)

        error = torch.zeros_like(y_soft)
        error[mask] = y_true - y_soft[mask]

        if self.autoscale:
            smoothed_error = self.prop1(error, edge_index,
                                        edge_weight=edge_weight,
                                        post_step=lambda x: x.clamp_(-1., 1.))

            sigma = error[mask].abs().sum() / numel
            scale = sigma / smoothed_error.abs().sum(dim=1, keepdim=True)
            scale[scale.isinf() | (scale > 1000)] = 1.0
            return y_soft + scale * smoothed_error
        else:

            def fix_input(x):
                x[mask] = error[mask]
                return x

            smoothed_error = self.prop1(error, edge_index,
                                        edge_weight=edge_weight,
                                        post_step=fix_input)
            return y_soft + self.scale * smoothed_error

    def smooth(self, y_soft: Tensor, y_true: Tensor, mask: Tensor,
               edge_index: Adj, edge_weight: OptTensor = None) -> Tensor:
        r"""
        Args:
            y_soft (torch.Tensor): The corrected predictions :math:`\mathbf{Z}`
                obtained from :meth:`correct`.
            y_true (torch.Tensor): The ground-truth label information
                :math:`\mathbf{Y}` of training nodes.
            mask (torch.Tensor): A mask or index tensor denoting which nodes
                were used for training.
            edge_index (torch.Tensor or SparseTensor): The edge connectivity.
            edge_weight (torch.Tensor, optional): The edge weights.
                (default: :obj:`None`)
        """
        numel = int(mask.sum()) if mask.dtype == torch.bool else mask.size(0)
        assert y_true.size(0) == numel

        if y_true.dtype == torch.long and y_true.size(0) == y_true.numel():
            y_true = one_hot(y_true.view(-1), num_classes=y_soft.size(-1),
                             dtype=y_soft.dtype)

        y_soft = y_soft.clone()
        y_soft[mask] = y_true

        return self.prop2(y_soft, edge_index, edge_weight=edge_weight)

    def __repr__(self):
        L1, alpha1 = self.prop1.num_layers, self.prop1.alpha
        L2, alpha2 = self.prop2.num_layers, self.prop2.alpha
        return (f'{self.__class__.__name__}(\n'
                f'  correct: num_layers={L1}, alpha={alpha1}\n'
                f'  smooth:  num_layers={L2}, alpha={alpha2}\n'
                f'  autoscale={self.autoscale}, scale={self.scale}\n'
                ')')
pytorch/torch_geometric/nn/models/deep_graph_infomax.py000066400000000000000000000076041444262304300240230ustar00rootroot00000000000000from typing import Callable, Tuple

import torch
from torch import Tensor
from torch.nn import Module, Parameter

from torch_geometric.nn.inits import reset, uniform

EPS = 1e-15


class DeepGraphInfomax(torch.nn.Module):
    r"""The Deep Graph Infomax model from the
    `"Deep Graph Infomax" `_
    paper based on user-defined encoder and summary model :math:`\mathcal{E}`
    and :math:`\mathcal{R}` respectively, and a corruption function
    :math:`\mathcal{C}`.

    Args:
        hidden_channels (int): The latent space dimensionality.
        encoder (torch.nn.Module): The encoder module :math:`\mathcal{E}`.
        summary (callable): The readout function :math:`\mathcal{R}`.
        corruption (callable): The corruption function :math:`\mathcal{C}`.
    """
    def __init__(
        self,
        hidden_channels: int,
        encoder: Module,
        summary: Callable,
        corruption: Callable,
    ):
        super().__init__()
        self.hidden_channels = hidden_channels
        self.encoder = encoder
        self.summary = summary
        self.corruption = corruption

        self.weight = Parameter(torch.Tensor(hidden_channels, hidden_channels))

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        reset(self.encoder)
        reset(self.summary)
        uniform(self.hidden_channels, self.weight)

    def forward(self, *args, **kwargs) -> Tuple[Tensor, Tensor, Tensor]:
        """Returns the latent space for the input arguments, their
        corruptions and their summary representation."""
        pos_z = self.encoder(*args, **kwargs)
        cor = self.corruption(*args, **kwargs)
        cor = cor if isinstance(cor, tuple) else (cor, )
        neg_z = self.encoder(*cor)
        summary = self.summary(pos_z, *args, **kwargs)
        return pos_z, neg_z, summary

    def discriminate(self, z: Tensor, summary: Tensor,
                     sigmoid: bool = True) -> Tensor:
        r"""Given the patch-summary pair :obj:`z` and :obj:`summary`, computes
        the probability scores assigned to this patch-summary pair.

        Args:
            z (torch.Tensor): The latent space.
            summary (torch.Tensor): The summary vector.
            sigmoid (bool, optional): If set to :obj:`False`, does not apply
                the logistic sigmoid function to the output.
                (default: :obj:`True`)
        """
        summary = summary.t() if summary.dim() > 1 else summary
        value = torch.matmul(z, torch.matmul(self.weight, summary))
        return torch.sigmoid(value) if sigmoid else value

    def loss(self, pos_z: Tensor, neg_z: Tensor, summary: Tensor) -> Tensor:
        r"""Computes the mutual information maximization objective."""
        pos_loss = -torch.log(
            self.discriminate(pos_z, summary, sigmoid=True) + EPS).mean()
        neg_loss = -torch.log(1 -
                              self.discriminate(neg_z, summary, sigmoid=True) +
                              EPS).mean()

        return pos_loss + neg_loss

    def test(
        self,
        train_z: Tensor,
        train_y: Tensor,
        test_z: Tensor,
        test_y: Tensor,
        solver: str = 'lbfgs',
        multi_class: str = 'auto',
        *args,
        **kwargs,
    ) -> float:
        r"""Evaluates latent space quality via a logistic regression downstream
        task."""
        from sklearn.linear_model import LogisticRegression

        clf = LogisticRegression(solver=solver, multi_class=multi_class, *args,
                                 **kwargs).fit(train_z.detach().cpu().numpy(),
                                               train_y.detach().cpu().numpy())
        return clf.score(test_z.detach().cpu().numpy(),
                         test_y.detach().cpu().numpy())

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.hidden_channels})'
pytorch/torch_geometric/nn/models/deepgcn.py000066400000000000000000000101771444262304300216100ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Module
from torch.utils.checkpoint import checkpoint


class DeepGCNLayer(torch.nn.Module):
    r"""The skip connection operations from the
    `"DeepGCNs: Can GCNs Go as Deep as CNNs?"
    `_ and `"All You Need to Train Deeper
    GCNs" `_ papers.
    The implemented skip connections includes the pre-activation residual
    connection (:obj:`"res+"`), the residual connection (:obj:`"res"`),
    the dense connection (:obj:`"dense"`) and no connections (:obj:`"plain"`).

    * **Res+** (:obj:`"res+"`):

    .. math::
        \text{Normalization}\to\text{Activation}\to\text{Dropout}\to
        \text{GraphConv}\to\text{Res}

    * **Res** (:obj:`"res"`) / **Dense** (:obj:`"dense"`) / **Plain**
      (:obj:`"plain"`):

    .. math::
        \text{GraphConv}\to\text{Normalization}\to\text{Activation}\to
        \text{Res/Dense/Plain}\to\text{Dropout}

    .. note::

        For an example of using :obj:`GENConv`, see
        `examples/ogbn_proteins_deepgcn.py
        `_.

    Args:
        conv (torch.nn.Module, optional): the GCN operator.
            (default: :obj:`None`)
        norm (torch.nn.Module): the normalization layer. (default: :obj:`None`)
        act (torch.nn.Module): the activation layer. (default: :obj:`None`)
        block (str, optional): The skip connection operation to use
            (:obj:`"res+"`, :obj:`"res"`, :obj:`"dense"` or :obj:`"plain"`).
            (default: :obj:`"res+"`)
        dropout (float, optional): Whether to apply or dropout.
            (default: :obj:`0.`)
        ckpt_grad (bool, optional): If set to :obj:`True`, will checkpoint this
            part of the model. Checkpointing works by trading compute for
            memory, since intermediate activations do not need to be kept in
            memory. Set this to :obj:`True` in case you encounter out-of-memory
            errors while going deep. (default: :obj:`False`)
    """
    def __init__(
        self,
        conv: Optional[Module] = None,
        norm: Optional[Module] = None,
        act: Optional[Module] = None,
        block: str = 'res+',
        dropout: float = 0.,
        ckpt_grad: bool = False,
    ):
        super().__init__()

        self.conv = conv
        self.norm = norm
        self.act = act
        self.block = block.lower()
        assert self.block in ['res+', 'res', 'dense', 'plain']
        self.dropout = dropout
        self.ckpt_grad = ckpt_grad

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.conv.reset_parameters()
        self.norm.reset_parameters()

    def forward(self, *args, **kwargs) -> Tensor:
        """"""
        args = list(args)
        x = args.pop(0)

        if self.block == 'res+':
            h = x
            if self.norm is not None:
                h = self.norm(h)
            if self.act is not None:
                h = self.act(h)
            h = F.dropout(h, p=self.dropout, training=self.training)
            if self.conv is not None and self.ckpt_grad and h.requires_grad:
                h = checkpoint(self.conv, h, *args, **kwargs)
            else:
                h = self.conv(h, *args, **kwargs)

            return x + h

        else:
            if self.conv is not None and self.ckpt_grad and x.requires_grad:
                h = checkpoint(self.conv, x, *args, **kwargs)
            else:
                h = self.conv(x, *args, **kwargs)
            if self.norm is not None:
                h = self.norm(h)
            if self.act is not None:
                h = self.act(h)

            if self.block == 'res':
                h = x + h
            elif self.block == 'dense':
                h = torch.cat([x, h], dim=-1)
            elif self.block == 'plain':
                pass

            return F.dropout(h, p=self.dropout, training=self.training)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(block={self.block})'
pytorch/torch_geometric/nn/models/dimenet.py000066400000000000000000001032001444262304300216160ustar00rootroot00000000000000import os
import os.path as osp
from math import pi as PI
from math import sqrt
from typing import Callable, Dict, Optional, Tuple, Union

import numpy as np
import torch
from torch import Tensor
from torch.nn import Embedding, Linear

from torch_geometric.data import Dataset, download_url
from torch_geometric.data.makedirs import makedirs
from torch_geometric.nn import radius_graph
from torch_geometric.nn.inits import glorot_orthogonal
from torch_geometric.nn.resolver import activation_resolver
from torch_geometric.typing import OptTensor, SparseTensor
from torch_geometric.utils import scatter

qm9_target_dict: Dict[int, str] = {
    0: 'mu',
    1: 'alpha',
    2: 'homo',
    3: 'lumo',
    5: 'r2',
    6: 'zpve',
    7: 'U0',
    8: 'U',
    9: 'H',
    10: 'G',
    11: 'Cv',
}


class Envelope(torch.nn.Module):
    def __init__(self, exponent: int):
        super().__init__()
        self.p = exponent + 1
        self.a = -(self.p + 1) * (self.p + 2) / 2
        self.b = self.p * (self.p + 2)
        self.c = -self.p * (self.p + 1) / 2

    def forward(self, x: Tensor) -> Tensor:
        p, a, b, c = self.p, self.a, self.b, self.c
        x_pow_p0 = x.pow(p - 1)
        x_pow_p1 = x_pow_p0 * x
        x_pow_p2 = x_pow_p1 * x
        return (1.0 / x + a * x_pow_p0 + b * x_pow_p1 +
                c * x_pow_p2) * (x < 1.0).to(x.dtype)


class BesselBasisLayer(torch.nn.Module):
    def __init__(self, num_radial: int, cutoff: float = 5.0,
                 envelope_exponent: int = 5):
        super().__init__()
        self.cutoff = cutoff
        self.envelope = Envelope(envelope_exponent)

        self.freq = torch.nn.Parameter(torch.Tensor(num_radial))

        self.reset_parameters()

    def reset_parameters(self):
        with torch.no_grad():
            torch.arange(1, self.freq.numel() + 1, out=self.freq).mul_(PI)
        self.freq.requires_grad_()

    def forward(self, dist: Tensor) -> Tensor:
        dist = dist.unsqueeze(-1) / self.cutoff
        return self.envelope(dist) * (self.freq * dist).sin()


class SphericalBasisLayer(torch.nn.Module):
    def __init__(
        self,
        num_spherical: int,
        num_radial: int,
        cutoff: float = 5.0,
        envelope_exponent: int = 5,
    ):
        super().__init__()
        import sympy as sym

        from torch_geometric.nn.models.dimenet_utils import (
            bessel_basis,
            real_sph_harm,
        )

        assert num_radial <= 64
        self.num_spherical = num_spherical
        self.num_radial = num_radial
        self.cutoff = cutoff
        self.envelope = Envelope(envelope_exponent)

        bessel_forms = bessel_basis(num_spherical, num_radial)
        sph_harm_forms = real_sph_harm(num_spherical)
        self.sph_funcs = []
        self.bessel_funcs = []

        x, theta = sym.symbols('x theta')
        modules = {'sin': torch.sin, 'cos': torch.cos}
        for i in range(num_spherical):
            if i == 0:
                sph1 = sym.lambdify([theta], sph_harm_forms[i][0], modules)(0)
                self.sph_funcs.append(lambda x: torch.zeros_like(x) + sph1)
            else:
                sph = sym.lambdify([theta], sph_harm_forms[i][0], modules)
                self.sph_funcs.append(sph)
            for j in range(num_radial):
                bessel = sym.lambdify([x], bessel_forms[i][j], modules)
                self.bessel_funcs.append(bessel)

    def forward(self, dist: Tensor, angle: Tensor, idx_kj: Tensor) -> Tensor:
        dist = dist / self.cutoff
        rbf = torch.stack([f(dist) for f in self.bessel_funcs], dim=1)
        rbf = self.envelope(dist).unsqueeze(-1) * rbf

        cbf = torch.stack([f(angle) for f in self.sph_funcs], dim=1)

        n, k = self.num_spherical, self.num_radial
        out = (rbf[idx_kj].view(-1, n, k) * cbf.view(-1, n, 1)).view(-1, n * k)
        return out


class EmbeddingBlock(torch.nn.Module):
    def __init__(self, num_radial: int, hidden_channels: int, act: Callable):
        super().__init__()
        self.act = act

        self.emb = Embedding(95, hidden_channels)
        self.lin_rbf = Linear(num_radial, hidden_channels)
        self.lin = Linear(3 * hidden_channels, hidden_channels)

        self.reset_parameters()

    def reset_parameters(self):
        self.emb.weight.data.uniform_(-sqrt(3), sqrt(3))
        self.lin_rbf.reset_parameters()
        self.lin.reset_parameters()

    def forward(self, x: Tensor, rbf: Tensor, i: Tensor, j: Tensor) -> Tensor:
        x = self.emb(x)
        rbf = self.act(self.lin_rbf(rbf))
        return self.act(self.lin(torch.cat([x[i], x[j], rbf], dim=-1)))


class ResidualLayer(torch.nn.Module):
    def __init__(self, hidden_channels: int, act: Callable):
        super().__init__()
        self.act = act
        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, hidden_channels)

        self.reset_parameters()

    def reset_parameters(self):
        glorot_orthogonal(self.lin1.weight, scale=2.0)
        self.lin1.bias.data.fill_(0)
        glorot_orthogonal(self.lin2.weight, scale=2.0)
        self.lin2.bias.data.fill_(0)

    def forward(self, x: Tensor) -> Tensor:
        return x + self.act(self.lin2(self.act(self.lin1(x))))


class InteractionBlock(torch.nn.Module):
    def __init__(
        self,
        hidden_channels: int,
        num_bilinear: int,
        num_spherical: int,
        num_radial: int,
        num_before_skip: int,
        num_after_skip: int,
        act: Callable,
    ):
        super().__init__()
        self.act = act

        self.lin_rbf = Linear(num_radial, hidden_channels, bias=False)
        self.lin_sbf = Linear(num_spherical * num_radial, num_bilinear,
                              bias=False)

        # Dense transformations of input messages.
        self.lin_kj = Linear(hidden_channels, hidden_channels)
        self.lin_ji = Linear(hidden_channels, hidden_channels)

        self.W = torch.nn.Parameter(
            torch.Tensor(hidden_channels, num_bilinear, hidden_channels))

        self.layers_before_skip = torch.nn.ModuleList([
            ResidualLayer(hidden_channels, act) for _ in range(num_before_skip)
        ])
        self.lin = Linear(hidden_channels, hidden_channels)
        self.layers_after_skip = torch.nn.ModuleList([
            ResidualLayer(hidden_channels, act) for _ in range(num_after_skip)
        ])

        self.reset_parameters()

    def reset_parameters(self):
        glorot_orthogonal(self.lin_rbf.weight, scale=2.0)
        glorot_orthogonal(self.lin_sbf.weight, scale=2.0)
        glorot_orthogonal(self.lin_kj.weight, scale=2.0)
        self.lin_kj.bias.data.fill_(0)
        glorot_orthogonal(self.lin_ji.weight, scale=2.0)
        self.lin_ji.bias.data.fill_(0)
        self.W.data.normal_(mean=0, std=2 / self.W.size(0))
        for res_layer in self.layers_before_skip:
            res_layer.reset_parameters()
        glorot_orthogonal(self.lin.weight, scale=2.0)
        self.lin.bias.data.fill_(0)
        for res_layer in self.layers_after_skip:
            res_layer.reset_parameters()

    def forward(self, x: Tensor, rbf: Tensor, sbf: Tensor, idx_kj: Tensor,
                idx_ji: Tensor) -> Tensor:
        rbf = self.lin_rbf(rbf)
        sbf = self.lin_sbf(sbf)

        x_ji = self.act(self.lin_ji(x))
        x_kj = self.act(self.lin_kj(x))
        x_kj = x_kj * rbf
        x_kj = torch.einsum('wj,wl,ijl->wi', sbf, x_kj[idx_kj], self.W)
        x_kj = scatter(x_kj, idx_ji, dim=0, dim_size=x.size(0), reduce='sum')

        h = x_ji + x_kj
        for layer in self.layers_before_skip:
            h = layer(h)
        h = self.act(self.lin(h)) + x
        for layer in self.layers_after_skip:
            h = layer(h)

        return h


class InteractionPPBlock(torch.nn.Module):
    def __init__(
        self,
        hidden_channels: int,
        int_emb_size: int,
        basis_emb_size: int,
        num_spherical: int,
        num_radial: int,
        num_before_skip: int,
        num_after_skip: int,
        act: Callable,
    ):
        super().__init__()
        self.act = act

        # Transformation of Bessel and spherical basis representations:
        self.lin_rbf1 = Linear(num_radial, basis_emb_size, bias=False)
        self.lin_rbf2 = Linear(basis_emb_size, hidden_channels, bias=False)

        self.lin_sbf1 = Linear(num_spherical * num_radial, basis_emb_size,
                               bias=False)
        self.lin_sbf2 = Linear(basis_emb_size, int_emb_size, bias=False)

        # Hidden transformation of input message:
        self.lin_kj = Linear(hidden_channels, hidden_channels)
        self.lin_ji = Linear(hidden_channels, hidden_channels)

        # Embedding projections for interaction triplets:
        self.lin_down = Linear(hidden_channels, int_emb_size, bias=False)
        self.lin_up = Linear(int_emb_size, hidden_channels, bias=False)

        # Residual layers before and after skip connection:
        self.layers_before_skip = torch.nn.ModuleList([
            ResidualLayer(hidden_channels, act) for _ in range(num_before_skip)
        ])
        self.lin = Linear(hidden_channels, hidden_channels)
        self.layers_after_skip = torch.nn.ModuleList([
            ResidualLayer(hidden_channels, act) for _ in range(num_before_skip)
        ])

        self.reset_parameters()

    def reset_parameters(self):
        glorot_orthogonal(self.lin_rbf1.weight, scale=2.0)
        glorot_orthogonal(self.lin_rbf2.weight, scale=2.0)
        glorot_orthogonal(self.lin_sbf1.weight, scale=2.0)
        glorot_orthogonal(self.lin_sbf2.weight, scale=2.0)

        glorot_orthogonal(self.lin_kj.weight, scale=2.0)
        self.lin_kj.bias.data.fill_(0)
        glorot_orthogonal(self.lin_ji.weight, scale=2.0)
        self.lin_ji.bias.data.fill_(0)

        glorot_orthogonal(self.lin_down.weight, scale=2.0)
        glorot_orthogonal(self.lin_up.weight, scale=2.0)

        for res_layer in self.layers_before_skip:
            res_layer.reset_parameters()
        glorot_orthogonal(self.lin.weight, scale=2.0)
        self.lin.bias.data.fill_(0)
        for res_layer in self.layers_after_skip:
            res_layer.reset_parameters()

    def forward(self, x: Tensor, rbf: Tensor, sbf: Tensor, idx_kj: Tensor,
                idx_ji: Tensor) -> Tensor:
        # Initial transformation:
        x_ji = self.act(self.lin_ji(x))
        x_kj = self.act(self.lin_kj(x))

        # Transformation via Bessel basis:
        rbf = self.lin_rbf1(rbf)
        rbf = self.lin_rbf2(rbf)
        x_kj = x_kj * rbf

        # Down project embedding and generating triple-interactions:
        x_kj = self.act(self.lin_down(x_kj))

        # Transform via 2D spherical basis:
        sbf = self.lin_sbf1(sbf)
        sbf = self.lin_sbf2(sbf)
        x_kj = x_kj[idx_kj] * sbf

        # Aggregate interactions and up-project embeddings:
        x_kj = scatter(x_kj, idx_ji, dim=0, dim_size=x.size(0), reduce='sum')
        x_kj = self.act(self.lin_up(x_kj))

        h = x_ji + x_kj
        for layer in self.layers_before_skip:
            h = layer(h)
        h = self.act(self.lin(h)) + x
        for layer in self.layers_after_skip:
            h = layer(h)

        return h


class OutputBlock(torch.nn.Module):
    def __init__(
        self,
        num_radial: int,
        hidden_channels: int,
        out_channels: int,
        num_layers: int,
        act: Callable,
    ):
        super().__init__()
        self.act = act

        self.lin_rbf = Linear(num_radial, hidden_channels, bias=False)
        self.lins = torch.nn.ModuleList()
        for _ in range(num_layers):
            self.lins.append(Linear(hidden_channels, hidden_channels))
        self.lin = Linear(hidden_channels, out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        glorot_orthogonal(self.lin_rbf.weight, scale=2.0)
        for lin in self.lins:
            glorot_orthogonal(lin.weight, scale=2.0)
            lin.bias.data.fill_(0)
        self.lin.weight.data.fill_(0)

    def forward(self, x: Tensor, rbf: Tensor, i: Tensor,
                num_nodes: Optional[int] = None) -> Tensor:
        x = self.lin_rbf(rbf) * x
        x = scatter(x, i, dim=0, dim_size=num_nodes, reduce='sum')
        for lin in self.lins:
            x = self.act(lin(x))
        return self.lin(x)


class OutputPPBlock(torch.nn.Module):
    def __init__(
        self,
        num_radial: int,
        hidden_channels: int,
        out_emb_channels: int,
        out_channels: int,
        num_layers: int,
        act: Callable,
    ):
        super().__init__()
        self.act = act

        self.lin_rbf = Linear(num_radial, hidden_channels, bias=False)

        # The up-projection layer:
        self.lin_up = Linear(hidden_channels, out_emb_channels, bias=False)
        self.lins = torch.nn.ModuleList()
        for _ in range(num_layers):
            self.lins.append(Linear(out_emb_channels, out_emb_channels))
        self.lin = Linear(out_emb_channels, out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        glorot_orthogonal(self.lin_rbf.weight, scale=2.0)
        glorot_orthogonal(self.lin_up.weight, scale=2.0)
        for lin in self.lins:
            glorot_orthogonal(lin.weight, scale=2.0)
            lin.bias.data.fill_(0)
        self.lin.weight.data.fill_(0)

    def forward(self, x: Tensor, rbf: Tensor, i: Tensor,
                num_nodes: Optional[int] = None) -> Tensor:
        x = self.lin_rbf(rbf) * x
        x = scatter(x, i, dim=0, dim_size=num_nodes, reduce='sum')
        x = self.lin_up(x)
        for lin in self.lins:
            x = self.act(lin(x))
        return self.lin(x)


def triplets(
    edge_index: Tensor,
    num_nodes: int,
) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]:
    row, col = edge_index  # j->i

    value = torch.arange(row.size(0), device=row.device)
    adj_t = SparseTensor(row=col, col=row, value=value,
                         sparse_sizes=(num_nodes, num_nodes))
    adj_t_row = adj_t[row]
    num_triplets = adj_t_row.set_value(None).sum(dim=1).to(torch.long)

    # Node indices (k->j->i) for triplets.
    idx_i = col.repeat_interleave(num_triplets)
    idx_j = row.repeat_interleave(num_triplets)
    idx_k = adj_t_row.storage.col()
    mask = idx_i != idx_k  # Remove i == k triplets.
    idx_i, idx_j, idx_k = idx_i[mask], idx_j[mask], idx_k[mask]

    # Edge indices (k-j, j->i) for triplets.
    idx_kj = adj_t_row.storage.value()[mask]
    idx_ji = adj_t_row.storage.row()[mask]

    return col, row, idx_i, idx_j, idx_k, idx_kj, idx_ji


class DimeNet(torch.nn.Module):
    r"""The directional message passing neural network (DimeNet) from the
    `"Directional Message Passing for Molecular Graphs"
    `_ paper.
    DimeNet transforms messages based on the angle between them in a
    rotation-equivariant fashion.

    .. note::

        For an example of using a pretrained DimeNet variant, see
        `examples/qm9_pretrained_dimenet.py
        `_.

    Args:
        hidden_channels (int): Hidden embedding size.
        out_channels (int): Size of each output sample.
        num_blocks (int): Number of building blocks.
        num_bilinear (int): Size of the bilinear layer tensor.
        num_spherical (int): Number of spherical harmonics.
        num_radial (int): Number of radial basis functions.
        cutoff (float, optional): Cutoff distance for interatomic
            interactions. (default: :obj:`5.0`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            collect for each node within the :attr:`cutoff` distance.
            (default: :obj:`32`)
        envelope_exponent (int, optional): Shape of the smooth cutoff.
            (default: :obj:`5`)
        num_before_skip (int, optional): Number of residual layers in the
            interaction blocks before the skip connection. (default: :obj:`1`)
        num_after_skip (int, optional): Number of residual layers in the
            interaction blocks after the skip connection. (default: :obj:`2`)
        num_output_layers (int, optional): Number of linear layers for the
            output blocks. (default: :obj:`3`)
        act (str or Callable, optional): The activation function.
            (default: :obj:`"swish"`)
    """

    url = ('https://github.com/klicperajo/dimenet/raw/master/pretrained/'
           'dimenet')

    def __init__(
        self,
        hidden_channels: int,
        out_channels: int,
        num_blocks: int,
        num_bilinear: int,
        num_spherical: int,
        num_radial: int,
        cutoff: float = 5.0,
        max_num_neighbors: int = 32,
        envelope_exponent: int = 5,
        num_before_skip: int = 1,
        num_after_skip: int = 2,
        num_output_layers: int = 3,
        act: Union[str, Callable] = 'swish',
    ):
        super().__init__()

        if num_spherical < 2:
            raise ValueError("'num_spherical' should be greater than 1")

        act = activation_resolver(act)

        self.cutoff = cutoff
        self.max_num_neighbors = max_num_neighbors
        self.num_blocks = num_blocks

        self.rbf = BesselBasisLayer(num_radial, cutoff, envelope_exponent)
        self.sbf = SphericalBasisLayer(num_spherical, num_radial, cutoff,
                                       envelope_exponent)

        self.emb = EmbeddingBlock(num_radial, hidden_channels, act)

        self.output_blocks = torch.nn.ModuleList([
            OutputBlock(num_radial, hidden_channels, out_channels,
                        num_output_layers, act) for _ in range(num_blocks + 1)
        ])

        self.interaction_blocks = torch.nn.ModuleList([
            InteractionBlock(
                hidden_channels,
                num_bilinear,
                num_spherical,
                num_radial,
                num_before_skip,
                num_after_skip,
                act,
            ) for _ in range(num_blocks)
        ])

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.rbf.reset_parameters()
        self.emb.reset_parameters()
        for out in self.output_blocks:
            out.reset_parameters()
        for interaction in self.interaction_blocks:
            interaction.reset_parameters()

    @classmethod
    def from_qm9_pretrained(
        cls,
        root: str,
        dataset: Dataset,
        target: int,
    ) -> Tuple['DimeNet', Dataset, Dataset, Dataset]:  # pragma: no cover
        r"""Returns a pre-trained :class:`DimeNet` model on the
        :class:`~torch_geometric.datasets.QM9` dataset, trained on the
        specified target :obj:`target`."""
        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
        import tensorflow as tf

        assert target >= 0 and target <= 12 and not target == 4

        root = osp.expanduser(osp.normpath(root))
        path = osp.join(root, 'pretrained_dimenet', qm9_target_dict[target])

        makedirs(path)
        url = f'{cls.url}/{qm9_target_dict[target]}'

        if not osp.exists(osp.join(path, 'checkpoint')):
            download_url(f'{url}/checkpoint', path)
            download_url(f'{url}/ckpt.data-00000-of-00002', path)
            download_url(f'{url}/ckpt.data-00001-of-00002', path)
            download_url(f'{url}/ckpt.index', path)

        path = osp.join(path, 'ckpt')
        reader = tf.train.load_checkpoint(path)

        model = cls(
            hidden_channels=128,
            out_channels=1,
            num_blocks=6,
            num_bilinear=8,
            num_spherical=7,
            num_radial=6,
            cutoff=5.0,
            envelope_exponent=5,
            num_before_skip=1,
            num_after_skip=2,
            num_output_layers=3,
        )

        def copy_(src, name, transpose=False):
            init = reader.get_tensor(f'{name}/.ATTRIBUTES/VARIABLE_VALUE')
            init = torch.from_numpy(init)
            if name[-6:] == 'kernel':
                init = init.t()
            src.data.copy_(init)

        copy_(model.rbf.freq, 'rbf_layer/frequencies')
        copy_(model.emb.emb.weight, 'emb_block/embeddings')
        copy_(model.emb.lin_rbf.weight, 'emb_block/dense_rbf/kernel')
        copy_(model.emb.lin_rbf.bias, 'emb_block/dense_rbf/bias')
        copy_(model.emb.lin.weight, 'emb_block/dense/kernel')
        copy_(model.emb.lin.bias, 'emb_block/dense/bias')

        for i, block in enumerate(model.output_blocks):
            copy_(block.lin_rbf.weight, f'output_blocks/{i}/dense_rbf/kernel')
            for j, lin in enumerate(block.lins):
                copy_(lin.weight, f'output_blocks/{i}/dense_layers/{j}/kernel')
                copy_(lin.bias, f'output_blocks/{i}/dense_layers/{j}/bias')
            copy_(block.lin.weight, f'output_blocks/{i}/dense_final/kernel')

        for i, block in enumerate(model.interaction_blocks):
            copy_(block.lin_rbf.weight, f'int_blocks/{i}/dense_rbf/kernel')
            copy_(block.lin_sbf.weight, f'int_blocks/{i}/dense_sbf/kernel')
            copy_(block.lin_kj.weight, f'int_blocks/{i}/dense_kj/kernel')
            copy_(block.lin_kj.bias, f'int_blocks/{i}/dense_kj/bias')
            copy_(block.lin_ji.weight, f'int_blocks/{i}/dense_ji/kernel')
            copy_(block.lin_ji.bias, f'int_blocks/{i}/dense_ji/bias')
            copy_(block.W, f'int_blocks/{i}/bilinear')
            for j, layer in enumerate(block.layers_before_skip):
                copy_(layer.lin1.weight,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_1/kernel')
                copy_(layer.lin1.bias,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_1/bias')
                copy_(layer.lin2.weight,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_2/kernel')
                copy_(layer.lin2.bias,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_2/bias')
            copy_(block.lin.weight, f'int_blocks/{i}/final_before_skip/kernel')
            copy_(block.lin.bias, f'int_blocks/{i}/final_before_skip/bias')
            for j, layer in enumerate(block.layers_after_skip):
                copy_(layer.lin1.weight,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_1/kernel')
                copy_(layer.lin1.bias,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_1/bias')
                copy_(layer.lin2.weight,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_2/kernel')
                copy_(layer.lin2.bias,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_2/bias')

        # Use the same random seed as the official DimeNet` implementation.
        random_state = np.random.RandomState(seed=42)
        perm = torch.from_numpy(random_state.permutation(np.arange(130831)))
        perm = perm.long()
        train_idx = perm[:110000]
        val_idx = perm[110000:120000]
        test_idx = perm[120000:]

        return model, (dataset[train_idx], dataset[val_idx], dataset[test_idx])

    def forward(
        self,
        z: Tensor,
        pos: Tensor,
        batch: OptTensor = None,
    ) -> Tensor:
        r"""
        Args:
            z (torch.Tensor): Atomic number of each atom with shape
                :obj:`[num_atoms]`.
            pos (torch.Tensor): Coordinates of each atom with shape
                :obj:`[num_atoms, 3]`.
            batch (torch.Tensor, optional): Batch indices assigning each atom
                to a separate molecule with shape :obj:`[num_atoms]`.
                (default: :obj:`None`)
        """
        edge_index = radius_graph(pos, r=self.cutoff, batch=batch,
                                  max_num_neighbors=self.max_num_neighbors)

        i, j, idx_i, idx_j, idx_k, idx_kj, idx_ji = triplets(
            edge_index, num_nodes=z.size(0))

        # Calculate distances.
        dist = (pos[i] - pos[j]).pow(2).sum(dim=-1).sqrt()

        # Calculate angles.
        pos_i = pos[idx_i]
        pos_ji, pos_ki = pos[idx_j] - pos_i, pos[idx_k] - pos_i
        a = (pos_ji * pos_ki).sum(dim=-1)
        b = torch.cross(pos_ji, pos_ki).norm(dim=-1)
        angle = torch.atan2(b, a)

        rbf = self.rbf(dist)
        sbf = self.sbf(dist, angle, idx_kj)

        # Embedding block.
        x = self.emb(z, rbf, i, j)
        P = self.output_blocks[0](x, rbf, i, num_nodes=pos.size(0))

        # Interaction blocks.
        for interaction_block, output_block in zip(self.interaction_blocks,
                                                   self.output_blocks[1:]):
            x = interaction_block(x, rbf, sbf, idx_kj, idx_ji)
            P = P + output_block(x, rbf, i, num_nodes=pos.size(0))

        if batch is None:
            return P.sum(dim=0)
        else:
            return scatter(P, batch, dim=0, reduce='sum')


class DimeNetPlusPlus(DimeNet):
    r"""The DimeNet++ from the `"Fast and Uncertainty-Aware
    Directional Message Passing for Non-Equilibrium Molecules"
    `_ paper.

    :class:`DimeNetPlusPlus` is an upgrade to the :class:`DimeNet` model with
    8x faster and 10% more accurate than :class:`DimeNet`.

    Args:
        hidden_channels (int): Hidden embedding size.
        out_channels (int): Size of each output sample.
        num_blocks (int): Number of building blocks.
        int_emb_size (int): Size of embedding in the interaction block.
        basis_emb_size (int): Size of basis embedding in the interaction block.
        out_emb_channels (int): Size of embedding in the output block.
        num_spherical (int): Number of spherical harmonics.
        num_radial (int): Number of radial basis functions.
        cutoff: (float, optional): Cutoff distance for interatomic
            interactions. (default: :obj:`5.0`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            collect for each node within the :attr:`cutoff` distance.
            (default: :obj:`32`)
        envelope_exponent (int, optional): Shape of the smooth cutoff.
            (default: :obj:`5`)
        num_before_skip: (int, optional): Number of residual layers in the
            interaction blocks before the skip connection. (default: :obj:`1`)
        num_after_skip: (int, optional): Number of residual layers in the
            interaction blocks after the skip connection. (default: :obj:`2`)
        num_output_layers: (int, optional): Number of linear layers for the
            output blocks. (default: :obj:`3`)
        act: (str or Callable, optional): The activation funtion.
            (default: :obj:`"swish"`)
    """

    url = ('https://raw.githubusercontent.com/gasteigerjo/dimenet/'
           'master/pretrained/dimenet_pp')

    def __init__(
        self,
        hidden_channels: int,
        out_channels: int,
        num_blocks: int,
        int_emb_size: int,
        basis_emb_size: int,
        out_emb_channels: int,
        num_spherical: int,
        num_radial: int,
        cutoff: float = 5.0,
        max_num_neighbors: int = 32,
        envelope_exponent: int = 5,
        num_before_skip: int = 1,
        num_after_skip: int = 2,
        num_output_layers: int = 3,
        act: Union[str, Callable] = 'swish',
    ):
        act = activation_resolver(act)

        super().__init__(
            hidden_channels=hidden_channels,
            out_channels=out_channels,
            num_blocks=num_blocks,
            num_bilinear=1,
            num_spherical=num_spherical,
            num_radial=num_radial,
            cutoff=cutoff,
            max_num_neighbors=max_num_neighbors,
            envelope_exponent=envelope_exponent,
            num_before_skip=num_before_skip,
            num_after_skip=num_after_skip,
            num_output_layers=num_output_layers,
            act=act,
        )

        # We are re-using the RBF, SBF and embedding layers of `DimeNet` and
        # redefine output_block and interaction_block in DimeNet++.
        # Hence, it is to be noted that in the above initalization, the
        # variable `num_bilinear` does not have any purpose as it is used
        # solely in the `OutputBlock` of DimeNet:
        self.output_blocks = torch.nn.ModuleList([
            OutputPPBlock(
                num_radial,
                hidden_channels,
                out_emb_channels,
                out_channels,
                num_output_layers,
                act,
            ) for _ in range(num_blocks + 1)
        ])

        self.interaction_blocks = torch.nn.ModuleList([
            InteractionPPBlock(
                hidden_channels,
                int_emb_size,
                basis_emb_size,
                num_spherical,
                num_radial,
                num_before_skip,
                num_after_skip,
                act,
            ) for _ in range(num_blocks)
        ])

        self.reset_parameters()

    @classmethod
    def from_qm9_pretrained(
        cls,
        root: str,
        dataset: Dataset,
        target: int,
    ) -> Tuple['DimeNetPlusPlus', Dataset, Dataset,
               Dataset]:  # pragma: no cover
        r"""Returns a pre-trained :class:`DimeNetPlusPlus` model on the
        :class:`~torch_geometric.datasets.QM9` dataset, trained on the
        specified target :obj:`target`."""
        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
        import tensorflow as tf

        assert target >= 0 and target <= 12 and not target == 4

        root = osp.expanduser(osp.normpath(root))
        path = osp.join(root, 'pretrained_dimenet_pp', qm9_target_dict[target])

        makedirs(path)
        url = f'{cls.url}/{qm9_target_dict[target]}'

        if not osp.exists(osp.join(path, 'checkpoint')):
            download_url(f'{url}/checkpoint', path)
            download_url(f'{url}/ckpt.data-00000-of-00002', path)
            download_url(f'{url}/ckpt.data-00001-of-00002', path)
            download_url(f'{url}/ckpt.index', path)

        path = osp.join(path, 'ckpt')
        reader = tf.train.load_checkpoint(path)

        # Configuration from DimeNet++:
        # https://github.com/gasteigerjo/dimenet/blob/master/config_pp.yaml
        model = cls(
            hidden_channels=128,
            out_channels=1,
            num_blocks=4,
            int_emb_size=64,
            basis_emb_size=8,
            out_emb_channels=256,
            num_spherical=7,
            num_radial=6,
            cutoff=5.0,
            max_num_neighbors=32,
            envelope_exponent=5,
            num_before_skip=1,
            num_after_skip=2,
            num_output_layers=3,
        )

        def copy_(src, name, transpose=False):
            init = reader.get_tensor(f'{name}/.ATTRIBUTES/VARIABLE_VALUE')
            init = torch.from_numpy(init)
            if name[-6:] == 'kernel':
                init = init.t()
            src.data.copy_(init)

        copy_(model.rbf.freq, 'rbf_layer/frequencies')
        copy_(model.emb.emb.weight, 'emb_block/embeddings')
        copy_(model.emb.lin_rbf.weight, 'emb_block/dense_rbf/kernel')
        copy_(model.emb.lin_rbf.bias, 'emb_block/dense_rbf/bias')
        copy_(model.emb.lin.weight, 'emb_block/dense/kernel')
        copy_(model.emb.lin.bias, 'emb_block/dense/bias')

        for i, block in enumerate(model.output_blocks):
            copy_(block.lin_rbf.weight, f'output_blocks/{i}/dense_rbf/kernel')
            copy_(block.lin_up.weight,
                  f'output_blocks/{i}/up_projection/kernel')
            for j, lin in enumerate(block.lins):
                copy_(lin.weight, f'output_blocks/{i}/dense_layers/{j}/kernel')
                copy_(lin.bias, f'output_blocks/{i}/dense_layers/{j}/bias')
            copy_(block.lin.weight, f'output_blocks/{i}/dense_final/kernel')

        for i, block in enumerate(model.interaction_blocks):
            copy_(block.lin_rbf1.weight, f'int_blocks/{i}/dense_rbf1/kernel')
            copy_(block.lin_rbf2.weight, f'int_blocks/{i}/dense_rbf2/kernel')
            copy_(block.lin_sbf1.weight, f'int_blocks/{i}/dense_sbf1/kernel')
            copy_(block.lin_sbf2.weight, f'int_blocks/{i}/dense_sbf2/kernel')

            copy_(block.lin_ji.weight, f'int_blocks/{i}/dense_ji/kernel')
            copy_(block.lin_ji.bias, f'int_blocks/{i}/dense_ji/bias')
            copy_(block.lin_kj.weight, f'int_blocks/{i}/dense_kj/kernel')
            copy_(block.lin_kj.bias, f'int_blocks/{i}/dense_kj/bias')

            copy_(block.lin_down.weight,
                  f'int_blocks/{i}/down_projection/kernel')
            copy_(block.lin_up.weight, f'int_blocks/{i}/up_projection/kernel')

            for j, layer in enumerate(block.layers_before_skip):
                copy_(layer.lin1.weight,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_1/kernel')
                copy_(layer.lin1.bias,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_1/bias')
                copy_(layer.lin2.weight,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_2/kernel')
                copy_(layer.lin2.bias,
                      f'int_blocks/{i}/layers_before_skip/{j}/dense_2/bias')

            copy_(block.lin.weight, f'int_blocks/{i}/final_before_skip/kernel')
            copy_(block.lin.bias, f'int_blocks/{i}/final_before_skip/bias')

            for j, layer in enumerate(block.layers_after_skip):
                copy_(layer.lin1.weight,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_1/kernel')
                copy_(layer.lin1.bias,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_1/bias')
                copy_(layer.lin2.weight,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_2/kernel')
                copy_(layer.lin2.bias,
                      f'int_blocks/{i}/layers_after_skip/{j}/dense_2/bias')

        random_state = np.random.RandomState(seed=42)
        perm = torch.from_numpy(random_state.permutation(np.arange(130831)))
        perm = perm.long()
        train_idx = perm[:110000]
        val_idx = perm[110000:120000]
        test_idx = perm[120000:]

        return model, (dataset[train_idx], dataset[val_idx], dataset[test_idx])
pytorch/torch_geometric/nn/models/dimenet_utils.py000066400000000000000000000110031444262304300230350ustar00rootroot00000000000000# Shameless steal from: https://github.com/klicperajo/dimenet

import numpy as np
from scipy import special as sp
from scipy.optimize import brentq

try:
    import sympy as sym
except ImportError:
    sym = None


def Jn(r, n):
    return np.sqrt(np.pi / (2 * r)) * sp.jv(n + 0.5, r)


def Jn_zeros(n, k):
    zerosj = np.zeros((n, k), dtype='float32')
    zerosj[0] = np.arange(1, k + 1) * np.pi
    points = np.arange(1, k + n) * np.pi
    racines = np.zeros(k + n - 1, dtype='float32')
    for i in range(1, n):
        for j in range(k + n - 1 - i):
            foo = brentq(Jn, points[j], points[j + 1], (i, ))
            racines[j] = foo
        points = racines
        zerosj[i][:k] = racines[:k]

    return zerosj


def spherical_bessel_formulas(n):
    x = sym.symbols('x')

    f = [sym.sin(x) / x]
    a = sym.sin(x) / x
    for i in range(1, n):
        b = sym.diff(a, x) / x
        f += [sym.simplify(b * (-x)**i)]
        a = sym.simplify(b)
    return f


def bessel_basis(n, k):
    zeros = Jn_zeros(n, k)
    normalizer = []
    for order in range(n):
        normalizer_tmp = []
        for i in range(k):
            normalizer_tmp += [0.5 * Jn(zeros[order, i], order + 1)**2]
        normalizer_tmp = 1 / np.array(normalizer_tmp)**0.5
        normalizer += [normalizer_tmp]

    f = spherical_bessel_formulas(n)
    x = sym.symbols('x')
    bess_basis = []
    for order in range(n):
        bess_basis_tmp = []
        for i in range(k):
            bess_basis_tmp += [
                sym.simplify(normalizer[order][i] *
                             f[order].subs(x, zeros[order, i] * x))
            ]
        bess_basis += [bess_basis_tmp]
    return bess_basis


def sph_harm_prefactor(k, m):
    return ((2 * k + 1) * np.math.factorial(k - abs(m)) /
            (4 * np.pi * np.math.factorial(k + abs(m))))**0.5


def associated_legendre_polynomials(k, zero_m_only=True):
    z = sym.symbols('z')
    P_l_m = [[0] * (j + 1) for j in range(k)]

    P_l_m[0][0] = 1
    if k > 0:
        P_l_m[1][0] = z

        for j in range(2, k):
            P_l_m[j][0] = sym.simplify(((2 * j - 1) * z * P_l_m[j - 1][0] -
                                        (j - 1) * P_l_m[j - 2][0]) / j)
        if not zero_m_only:
            for i in range(1, k):
                P_l_m[i][i] = sym.simplify((1 - 2 * i) * P_l_m[i - 1][i - 1])
                if i + 1 < k:
                    P_l_m[i + 1][i] = sym.simplify(
                        (2 * i + 1) * z * P_l_m[i][i])
                for j in range(i + 2, k):
                    P_l_m[j][i] = sym.simplify(
                        ((2 * j - 1) * z * P_l_m[j - 1][i] -
                         (i + j - 1) * P_l_m[j - 2][i]) / (j - i))

    return P_l_m


def real_sph_harm(k, zero_m_only=True, spherical_coordinates=True):
    if not zero_m_only:
        S_m = [0]
        C_m = [1]
        for i in range(1, k):
            x = sym.symbols('x')
            y = sym.symbols('y')
            S_m += [x * S_m[i - 1] + y * C_m[i - 1]]
            C_m += [x * C_m[i - 1] - y * S_m[i - 1]]

    P_l_m = associated_legendre_polynomials(k, zero_m_only)
    if spherical_coordinates:
        theta = sym.symbols('theta')
        z = sym.symbols('z')
        for i in range(len(P_l_m)):
            for j in range(len(P_l_m[i])):
                if type(P_l_m[i][j]) != int:
                    P_l_m[i][j] = P_l_m[i][j].subs(z, sym.cos(theta))
        if not zero_m_only:
            phi = sym.symbols('phi')
            for i in range(len(S_m)):
                S_m[i] = S_m[i].subs(x,
                                     sym.sin(theta) * sym.cos(phi)).subs(
                                         y,
                                         sym.sin(theta) * sym.sin(phi))
            for i in range(len(C_m)):
                C_m[i] = C_m[i].subs(x,
                                     sym.sin(theta) * sym.cos(phi)).subs(
                                         y,
                                         sym.sin(theta) * sym.sin(phi))

    Y_func_l_m = [['0'] * (2 * j + 1) for j in range(k)]
    for i in range(k):
        Y_func_l_m[i][0] = sym.simplify(sph_harm_prefactor(i, 0) * P_l_m[i][0])

    if not zero_m_only:
        for i in range(1, k):
            for j in range(1, i + 1):
                Y_func_l_m[i][j] = sym.simplify(
                    2**0.5 * sph_harm_prefactor(i, j) * C_m[j] * P_l_m[i][j])
        for i in range(1, k):
            for j in range(1, i + 1):
                Y_func_l_m[i][-j] = sym.simplify(
                    2**0.5 * sph_harm_prefactor(i, -j) * S_m[j] * P_l_m[i][j])

    return Y_func_l_m
pytorch/torch_geometric/nn/models/gnnff.py000066400000000000000000000172631444262304300213040ustar00rootroot00000000000000import torch
from torch import Tensor
from torch.nn import BatchNorm1d, Embedding, Linear, ModuleList, Sequential

from torch_geometric.nn import radius_graph
from torch_geometric.nn.inits import reset
from torch_geometric.nn.models.dimenet import triplets
from torch_geometric.nn.models.schnet import ShiftedSoftplus
from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter


class GaussianFilter(torch.nn.Module):
    def __init__(self, start=0.0, stop=5.0, num_gaussians=50):
        super().__init__()
        offset = torch.linspace(start, stop, num_gaussians)
        self.coeff = -0.5 / (float(offset[1]) - float(offset[0]))**2
        self.register_buffer('offset', offset)

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        pass

    def forward(self, dist: Tensor) -> Tensor:
        dist = dist.view(-1, 1) - self.offset.view(1, -1)
        return torch.exp(self.coeff * dist.pow(2))


class NodeBlock(torch.nn.Module):
    def __init__(self, hidden_node_channels: int, hidden_edge_channels: int):
        super().__init__()
        self.lin_c1 = Linear(hidden_node_channels + hidden_edge_channels,
                             2 * hidden_node_channels)

        # BN was added based on previous studies.
        # ref: https://github.com/txie-93/cgcnn/blob/master/cgcnn/model.py
        self.bn_c1 = BatchNorm1d(2 * hidden_node_channels)
        self.bn = BatchNorm1d(hidden_node_channels)

    def reset_parameters(self):
        self.lin_c1.reset_parameters()
        self.bn_c1.reset_parameters()
        self.bn.reset_parameters()

    def forward(self, node_emb: Tensor, edge_emb: Tensor, i: Tensor) -> Tensor:
        c1 = torch.cat([node_emb[i], edge_emb], dim=1)
        c1 = self.bn_c1(self.lin_c1(c1))
        c1_filter, c1_core = c1.chunk(2, dim=1)
        c1_filter = c1_filter.sigmoid()
        c1_core = c1_core.tanh()
        c1_emb = scatter(c1_filter * c1_core, i, dim=0,
                         dim_size=node_emb.size(0), reduce='sum')
        c1_emb = self.bn(c1_emb)

        return (node_emb + c1_emb).tanh()


class EdgeBlock(torch.nn.Module):
    def __init__(self, hidden_node_channels: int, hidden_edge_channels: int):
        super().__init__()
        self.lin_c2 = Linear(hidden_node_channels, 2 * hidden_edge_channels)
        self.lin_c3 = Linear(
            3 * hidden_node_channels + 2 * hidden_edge_channels,
            2 * hidden_edge_channels,
        )

        # BN was added based on previous studies.
        # ref: https://github.com/txie-93/cgcnn/blob/master/cgcnn/model.py
        self.bn_c2 = BatchNorm1d(2 * hidden_edge_channels)
        self.bn_c3 = BatchNorm1d(2 * hidden_edge_channels)
        self.bn_c2_2 = BatchNorm1d(hidden_edge_channels)
        self.bn_c3_2 = BatchNorm1d(hidden_edge_channels)

    def reset_parameters(self):
        self.lin_c2.reset_parameters()
        self.lin_c3.reset_parameters()
        self.bn_c2.reset_parameters()
        self.bn_c3.reset_parameters()
        self.bn_c2_2.reset_parameters()
        self.bn_c3_2.reset_parameters()

    def forward(
        self,
        node_emb: Tensor,
        edge_emb: Tensor,
        i: Tensor,
        j: Tensor,
        idx_i: Tensor,
        idx_j: Tensor,
        idx_k: Tensor,
        idx_ji: Tensor,
        idx_kj: Tensor,
    ) -> Tensor:
        c2 = node_emb[i] * node_emb[j]
        c2 = self.bn_c2(self.lin_c2(c2))
        c2_filter, c2_core = c2.chunk(2, dim=1)
        c2_filter = c2_filter.sigmoid()
        c2_core = c2_core.tanh()
        c2_emb = self.bn_c2_2(c2_filter * c2_core)

        c3 = torch.cat([
            node_emb[idx_i],
            node_emb[idx_j],
            node_emb[idx_k],
            edge_emb[idx_ji],
            edge_emb[idx_kj],
        ], dim=1)
        c3 = self.bn_c3(self.lin_c3(c3))
        c3_filter, c3_core = c3.chunk(2, dim=1)
        c3_filter = c3_filter.sigmoid()
        c3_core = c3_core.tanh()
        c3_emb = scatter(c3_filter * c3_core, idx_ji, dim=0,
                         dim_size=edge_emb.size(0), reduce='sum')
        c3_emb = self.bn_c3_2(c3_emb)

        return (edge_emb + c2_emb + c3_emb).tanh()


class GNNFF(torch.nn.Module):
    r"""The Graph Neural Network Force Field (GNNFF) from the
    `"Accurate and scalable graph neural network force field and molecular
    dynamics with direct force architecture"
    `_ paper.
    :class:`GNNFF` directly predicts atomic forces from automatically
    extracted features of the local atomic environment that are
    translationally-invariant, but rotationally-covariant to the coordinate of
    the atoms.

    Args:
        hidden_node_channels (int): Hidden node embedding size.
        hidden_edge_channels (int): Hidden edge embedding size.
        num_layers (int): Number of message passing blocks.
        cutoff (float, optional): Cutoff distance for interatomic
            interactions. (default: :obj:`5.0`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            collect for each node within the :attr:`cutoff` distance.
            (default: :obj:`32`)
    """
    def __init__(
        self,
        hidden_node_channels: int,
        hidden_edge_channels: int,
        num_layers: int,
        cutoff: float = 5.0,
        max_num_neighbors: int = 32,
    ):
        super().__init__()

        self.cutoff = cutoff
        self.max_num_neighbors = max_num_neighbors

        self.node_emb = Sequential(
            Embedding(95, hidden_node_channels),
            ShiftedSoftplus(),
            Linear(hidden_node_channels, hidden_node_channels),
            ShiftedSoftplus(),
            Linear(hidden_node_channels, hidden_node_channels),
        )
        self.edge_emb = GaussianFilter(0.0, 5.0, hidden_edge_channels)

        self.node_blocks = ModuleList([
            NodeBlock(hidden_node_channels, hidden_edge_channels)
            for _ in range(num_layers)
        ])
        self.edge_blocks = ModuleList([
            EdgeBlock(hidden_node_channels, hidden_edge_channels)
            for _ in range(num_layers)
        ])

        self.force_predictor = Sequential(
            Linear(hidden_edge_channels, hidden_edge_channels),
            ShiftedSoftplus(),
            Linear(hidden_edge_channels, hidden_edge_channels),
            ShiftedSoftplus(),
            Linear(hidden_edge_channels, 1),
        )

    def reset_parameters(self):
        reset(self.node_emb)
        self.edge_emb.reset_parameters()
        for node_block in self.node_blocks:
            node_block.reset_parameters()
        for edge_block in self.edge_blocks:
            edge_block.reset_parameters()
        reset(self.force_predictor)

    def forward(self, z: Tensor, pos: Tensor,
                batch: OptTensor = None) -> Tensor:
        """"""
        edge_index = radius_graph(pos, r=self.cutoff, batch=batch,
                                  max_num_neighbors=self.max_num_neighbors)

        i, j, idx_i, idx_j, idx_k, idx_kj, idx_ji = triplets(
            edge_index, num_nodes=z.size(0))

        # Calculate distances and unit vector:
        dist = (pos[i] - pos[j]).pow(2).sum(dim=-1).sqrt()
        unit_vec = (pos[i] - pos[j]) / dist.view(-1, 1)

        # Embedding blocks:
        node_emb = self.node_emb(z)
        edge_emb = self.edge_emb(dist)

        # Message passing blocks:
        for node_block, edge_block in zip(self.node_blocks, self.edge_blocks):
            node_emb = node_block(node_emb, edge_emb, i)
            edge_emb = edge_block(node_emb, edge_emb, i, j, idx_i, idx_j,
                                  idx_k, idx_ji, idx_kj)

        # Force prediction block:
        force = self.force_predictor(edge_emb) * unit_vec

        return scatter(force, i, dim=0, reduce='sum')
pytorch/torch_geometric/nn/models/graph_mixer.py000066400000000000000000000032031444262304300225000ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.utils import scatter


class NodeEncoder(torch.nn.Module):
    r"""The node encoder module from the `"Do We Really Need Complicated
    Model Architectures for Temporal Networks?"
    `_ paper.
    :class:`NodeEncoder` captures the 1-hop temporal neighborhood information
    via mean pooling.

    .. math::
        \mathbf{x}_v^{\prime}(t_0) = \mathbf{x}_v + \textrm{mean} \left\{
        \mathbf{x}_w : w \in \mathcal{N}(v, t_0 - T, t_0) \right\}

    Args:
        time_window (int): The temporal window size :math:`T` to define the
            1-hop temporal neighborhood.
    """
    def __init__(self, time_window: int):
        super().__init__()
        self.time_window = time_window

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_time: Tensor,
        seed_time: Tensor,
    ) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The input node features.
            edge_index (torch.Tensor): The edge indices.
            edge_time (torch.Tensor): The timestamp attached to every edge.
            seed_time (torch.Tensor): The seed time :math:`t_0` for every
                destination node.
        """
        mask = ((edge_time <= seed_time[edge_index[1]]) &
                (edge_time > seed_time[edge_index[1]] - self.time_window))

        src, dst = edge_index[:, mask]
        mean = scatter(x[src], dst, dim=0, dim_size=x.size(0), reduce='mean')
        return x + mean

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(time_window={self.time_window})'
pytorch/torch_geometric/nn/models/graph_unet.py000066400000000000000000000124051444262304300223330ustar00rootroot00000000000000from typing import Callable, List, Union

import torch
from torch import Tensor

from torch_geometric.nn import GCNConv, TopKPooling
from torch_geometric.nn.resolver import activation_resolver
from torch_geometric.typing import OptTensor, PairTensor
from torch_geometric.utils import (
    add_self_loops,
    remove_self_loops,
    to_torch_csr_tensor,
)
from torch_geometric.utils.repeat import repeat


class GraphUNet(torch.nn.Module):
    r"""The Graph U-Net model from the `"Graph U-Nets"
    `_ paper which implements a U-Net like
    architecture with graph pooling and unpooling operations.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Size of each hidden sample.
        out_channels (int): Size of each output sample.
        depth (int): The depth of the U-Net architecture.
        pool_ratios (float or [float], optional): Graph pooling ratio for each
            depth. (default: :obj:`0.5`)
        sum_res (bool, optional): If set to :obj:`False`, will use
            concatenation for integration of skip connections instead
            summation. (default: :obj:`True`)
        act (torch.nn.functional, optional): The nonlinearity to use.
            (default: :obj:`torch.nn.functional.relu`)
    """
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        out_channels: int,
        depth: int,
        pool_ratios: Union[float, List[float]] = 0.5,
        sum_res: bool = True,
        act: Union[str, Callable] = 'relu',
    ):
        super().__init__()
        assert depth >= 1
        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.out_channels = out_channels
        self.depth = depth
        self.pool_ratios = repeat(pool_ratios, depth)
        self.act = activation_resolver(act)
        self.sum_res = sum_res

        channels = hidden_channels

        self.down_convs = torch.nn.ModuleList()
        self.pools = torch.nn.ModuleList()
        self.down_convs.append(GCNConv(in_channels, channels, improved=True))
        for i in range(depth):
            self.pools.append(TopKPooling(channels, self.pool_ratios[i]))
            self.down_convs.append(GCNConv(channels, channels, improved=True))

        in_channels = channels if sum_res else 2 * channels

        self.up_convs = torch.nn.ModuleList()
        for i in range(depth - 1):
            self.up_convs.append(GCNConv(in_channels, channels, improved=True))
        self.up_convs.append(GCNConv(in_channels, out_channels, improved=True))

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for conv in self.down_convs:
            conv.reset_parameters()
        for pool in self.pools:
            pool.reset_parameters()
        for conv in self.up_convs:
            conv.reset_parameters()

    def forward(self, x: Tensor, edge_index: Tensor,
                batch: OptTensor = None) -> Tensor:
        """"""
        if batch is None:
            batch = edge_index.new_zeros(x.size(0))
        edge_weight = x.new_ones(edge_index.size(1))

        x = self.down_convs[0](x, edge_index, edge_weight)
        x = self.act(x)

        xs = [x]
        edge_indices = [edge_index]
        edge_weights = [edge_weight]
        perms = []

        for i in range(1, self.depth + 1):
            edge_index, edge_weight = self.augment_adj(edge_index, edge_weight,
                                                       x.size(0))
            x, edge_index, edge_weight, batch, perm, _ = self.pools[i - 1](
                x, edge_index, edge_weight, batch)

            x = self.down_convs[i](x, edge_index, edge_weight)
            x = self.act(x)

            if i < self.depth:
                xs += [x]
                edge_indices += [edge_index]
                edge_weights += [edge_weight]
            perms += [perm]

        for i in range(self.depth):
            j = self.depth - 1 - i

            res = xs[j]
            edge_index = edge_indices[j]
            edge_weight = edge_weights[j]
            perm = perms[j]

            up = torch.zeros_like(res)
            up[perm] = x
            x = res + up if self.sum_res else torch.cat((res, up), dim=-1)

            x = self.up_convs[i](x, edge_index, edge_weight)
            x = self.act(x) if i < self.depth - 1 else x

        return x

    def augment_adj(self, edge_index: Tensor, edge_weight: Tensor,
                    num_nodes: int) -> PairTensor:
        edge_index, edge_weight = remove_self_loops(edge_index, edge_weight)
        edge_index, edge_weight = add_self_loops(edge_index, edge_weight,
                                                 num_nodes=num_nodes)
        adj = to_torch_csr_tensor(edge_index, edge_weight,
                                  size=(num_nodes, num_nodes))
        adj = (adj @ adj).to_sparse_coo()
        edge_index, edge_weight = adj.indices(), adj.values()
        edge_index, edge_weight = remove_self_loops(edge_index, edge_weight)
        return edge_index, edge_weight

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.hidden_channels}, {self.out_channels}, '
                f'depth={self.depth}, pool_ratios={self.pool_ratios})')
pytorch/torch_geometric/nn/models/jumping_knowledge.py000066400000000000000000000065051444262304300237130ustar00rootroot00000000000000from typing import List, Optional

import torch
from torch import Tensor
from torch.nn import LSTM, Linear


class JumpingKnowledge(torch.nn.Module):
    r"""The Jumping Knowledge layer aggregation module from the
    `"Representation Learning on Graphs with Jumping Knowledge Networks"
    `_ paper based on either
    **concatenation** (:obj:`"cat"`)

    .. math::

        \mathbf{x}_v^{(1)} \, \Vert \, \ldots \, \Vert \, \mathbf{x}_v^{(T)}

    **max pooling** (:obj:`"max"`)

    .. math::

        \max \left( \mathbf{x}_v^{(1)}, \ldots, \mathbf{x}_v^{(T)} \right)

    or **weighted summation**

    .. math::

        \sum_{t=1}^T \alpha_v^{(t)} \mathbf{x}_v^{(t)}

    with attention scores :math:`\alpha_v^{(t)}` obtained from a bi-directional
    LSTM (:obj:`"lstm"`).

    Args:
        mode (str): The aggregation scheme to use
            (:obj:`"cat"`, :obj:`"max"` or :obj:`"lstm"`).
        channels (int, optional): The number of channels per representation.
            Needs to be only set for LSTM-style aggregation.
            (default: :obj:`None`)
        num_layers (int, optional): The number of layers to aggregate. Needs to
            be only set for LSTM-style aggregation. (default: :obj:`None`)
    """
    def __init__(self, mode: str, channels: Optional[int] = None,
                 num_layers: Optional[int] = None):
        super().__init__()
        self.mode = mode.lower()
        assert self.mode in ['cat', 'max', 'lstm']

        if mode == 'lstm':
            assert channels is not None, 'channels cannot be None for lstm'
            assert num_layers is not None, 'num_layers cannot be None for lstm'
            self.lstm = LSTM(channels, (num_layers * channels) // 2,
                             bidirectional=True, batch_first=True)
            self.att = Linear(2 * ((num_layers * channels) // 2), 1)
            self.channels = channels
            self.num_layers = num_layers
        else:
            self.lstm = None
            self.att = None
            self.channels = None
            self.num_layers = None

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        if self.lstm is not None:
            self.lstm.reset_parameters()
        if self.att is not None:
            self.att.reset_parameters()

    def forward(self, xs: List[Tensor]) -> Tensor:
        r"""
        Args:
            xs (List[torch.Tensor]): List containing the layer-wise
                representations.
        """
        if self.mode == 'cat':
            return torch.cat(xs, dim=-1)
        elif self.mode == 'max':
            return torch.stack(xs, dim=-1).max(dim=-1)[0]
        else:  # self.mode == 'lstm'
            assert self.lstm is not None and self.att is not None
            x = torch.stack(xs, dim=1)  # [num_nodes, num_layers, num_channels]
            alpha, _ = self.lstm(x)
            alpha = self.att(alpha).squeeze(-1)  # [num_nodes, num_layers]
            alpha = torch.softmax(alpha, dim=-1)
            return (x * alpha.unsqueeze(-1)).sum(dim=1)

    def __repr__(self) -> str:
        if self.mode == 'lstm':
            return (f'{self.__class__.__name__}({self.mode}, '
                    f'channels={self.channels}, layers={self.num_layers})')
        return f'{self.__class__.__name__}({self.mode})'
pytorch/torch_geometric/nn/models/label_prop.py000066400000000000000000000075411444262304300223230ustar00rootroot00000000000000from typing import Callable, Optional

import torch
from torch import Tensor

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.conv.gcn_conv import gcn_norm
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import one_hot, spmm


class LabelPropagation(MessagePassing):
    r"""The label propagation operator, firstly introduced in the
    `"Learning from Labeled and Unlabeled Data with Label Propagation"
    `_ paper

    .. math::
        \mathbf{Y}^{\prime} = \alpha \cdot \mathbf{D}^{-1/2} \mathbf{A}
        \mathbf{D}^{-1/2} \mathbf{Y} + (1 - \alpha) \mathbf{Y},

    where unlabeled data is inferred by labeled data via propagation.
    This concrete implementation here is derived from the `"Combining Label
    Propagation And Simple Models Out-performs Graph Neural Networks"
    `_ paper.

    .. note::

        For an example of using the :class:`LabelPropagation`, see
        `examples/label_prop.py
        `_.

    Args:
        num_layers (int): The number of propagations.
        alpha (float): The :math:`\alpha` coefficient.
    """
    def __init__(self, num_layers: int, alpha: float):
        super().__init__(aggr='add')
        self.num_layers = num_layers
        self.alpha = alpha

    @torch.no_grad()
    def forward(
        self,
        y: Tensor,
        edge_index: Adj,
        mask: OptTensor = None,
        edge_weight: OptTensor = None,
        post_step: Optional[Callable[[Tensor], Tensor]] = None,
    ) -> Tensor:
        r"""
        Args:
            y (torch.Tensor): The ground-truth label information
                :math:`\mathbf{Y}`.
            edge_index (torch.Tensor or SparseTensor): The edge connectivity.
            mask (torch.Tensor, optional): A mask or index tensor denoting
                which nodes are used for label propagation.
                (default: :obj:`None`)
            edge_weight (torch.Tensor, optional): The edge weights.
                (default: :obj:`None`)
            post_step (callable, optional): A post step function specified
                to apply after label propagation. If no post step function
                is specified, the output will be clamped between 0 and 1.
                (default: :obj:`None`)
        """
        if y.dtype == torch.long and y.size(0) == y.numel():
            y = one_hot(y.view(-1))

        out = y
        if mask is not None:
            out = torch.zeros_like(y)
            out[mask] = y[mask]

        if isinstance(edge_index, SparseTensor) and not edge_index.has_value():
            edge_index = gcn_norm(edge_index, add_self_loops=False)
        elif isinstance(edge_index, Tensor) and edge_weight is None:
            edge_index, edge_weight = gcn_norm(edge_index, num_nodes=y.size(0),
                                               add_self_loops=False)

        res = (1 - self.alpha) * out
        for _ in range(self.num_layers):
            # propagate_type: (y: Tensor, edge_weight: OptTensor)
            out = self.propagate(edge_index, x=out, edge_weight=edge_weight,
                                 size=None)
            out.mul_(self.alpha).add_(res)
            if post_step is not None:
                out = post_step(out)
            else:
                out.clamp_(0., 1.)

        return out

    def message(self, x_j: Tensor, edge_weight: OptTensor) -> Tensor:
        return x_j if edge_weight is None else edge_weight.view(-1, 1) * x_j

    def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:
        return spmm(adj_t, x, reduce=self.aggr)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_layers={self.num_layers}, '
                f'alpha={self.alpha})')
pytorch/torch_geometric/nn/models/lightgcn.py000066400000000000000000000260221444262304300217760ustar00rootroot00000000000000from typing import Optional, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Embedding, ModuleList
from torch.nn.modules.loss import _Loss

from torch_geometric.nn.conv import LGConv
from torch_geometric.typing import Adj, OptTensor
from torch_geometric.utils import is_sparse, to_edge_index


class LightGCN(torch.nn.Module):
    r"""The LightGCN model from the `"LightGCN: Simplifying and Powering
    Graph Convolution Network for Recommendation"
    `_ paper.

    :class:`~torch_geometric.nn.models.LightGCN` learns embeddings by linearly
    propagating them on the underlying graph, and uses the weighted sum of the
    embeddings learned at all layers as the final embedding

    .. math::
        \textbf{x}_i = \sum_{l=0}^{L} \alpha_l \textbf{x}^{(l)}_i,

    where each layer's embedding is computed as

    .. math::
        \mathbf{x}^{(l+1)}_i = \sum_{j \in \mathcal{N}(i)}
        \frac{1}{\sqrt{\deg(i)\deg(j)}}\mathbf{x}^{(l)}_j.

    Two prediction heads and training objectives are provided:
    **link prediction** (via
    :meth:`~torch_geometric.nn.models.LightGCN.link_pred_loss` and
    :meth:`~torch_geometric.nn.models.LightGCN.predict_link`) and
    **recommendation** (via
    :meth:`~torch_geometric.nn.models.LightGCN.recommendation_loss` and
    :meth:`~torch_geometric.nn.models.LightGCN.recommend`).

    .. note::

        Embeddings are propagated according to the graph connectivity specified
        by :obj:`edge_index` while rankings or link probabilities are computed
        according to the edges specified by :obj:`edge_label_index`.

    Args:
        num_nodes (int): The number of nodes in the graph.
        embedding_dim (int): The dimensionality of node embeddings.
        num_layers (int): The number of
            :class:`~torch_geometric.nn.conv.LGConv` layers.
        alpha (float or torch.Tensor, optional): The scalar or vector
            specifying the re-weighting coefficients for aggregating the final
            embedding. If set to :obj:`None`, the uniform initialization of
            :obj:`1 / (num_layers + 1)` is used. (default: :obj:`None`)
        **kwargs (optional): Additional arguments of the underlying
            :class:`~torch_geometric.nn.conv.LGConv` layers.
    """
    def __init__(
        self,
        num_nodes: int,
        embedding_dim: int,
        num_layers: int,
        alpha: Optional[Union[float, Tensor]] = None,
        **kwargs,
    ):
        super().__init__()

        self.num_nodes = num_nodes
        self.embedding_dim = embedding_dim
        self.num_layers = num_layers

        if alpha is None:
            alpha = 1. / (num_layers + 1)

        if isinstance(alpha, Tensor):
            assert alpha.size(0) == num_layers + 1
        else:
            alpha = torch.tensor([alpha] * (num_layers + 1))
        self.register_buffer('alpha', alpha)

        self.embedding = Embedding(num_nodes, embedding_dim)
        self.convs = ModuleList([LGConv(**kwargs) for _ in range(num_layers)])

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        torch.nn.init.xavier_uniform_(self.embedding.weight)
        for conv in self.convs:
            conv.reset_parameters()

    def get_embedding(
        self,
        edge_index: Adj,
        edge_weight: OptTensor = None,
    ) -> Tensor:
        r"""Returns the embedding of nodes in the graph."""
        x = self.embedding.weight
        out = x * self.alpha[0]

        for i in range(self.num_layers):
            x = self.convs[i](x, edge_index, edge_weight)
            out = out + x * self.alpha[i + 1]

        return out

    def forward(
        self,
        edge_index: Adj,
        edge_label_index: OptTensor = None,
        edge_weight: OptTensor = None,
    ) -> Tensor:
        r"""Computes rankings for pairs of nodes.

        Args:
            edge_index (torch.Tensor or SparseTensor): Edge tensor specifying
                the connectivity of the graph.
            edge_label_index (torch.Tensor, optional): Edge tensor specifying
                the node pairs for which to compute rankings or probabilities.
                If :obj:`edge_label_index` is set to :obj:`None`, all edges in
                :obj:`edge_index` will be used instead. (default: :obj:`None`)
            edge_weight (torch.Tensor, optional): The weight of each edge in
                :obj:`edge_index`. (default: :obj:`None`)
        """
        if edge_label_index is None:
            if is_sparse(edge_index):
                edge_label_index, _ = to_edge_index(edge_index)
            else:
                edge_label_index = edge_index

        out = self.get_embedding(edge_index, edge_weight)

        out_src = out[edge_label_index[0]]
        out_dst = out[edge_label_index[1]]

        return (out_src * out_dst).sum(dim=-1)

    def predict_link(
        self,
        edge_index: Adj,
        edge_label_index: OptTensor = None,
        edge_weight: OptTensor = None,
        prob: bool = False,
    ) -> Tensor:
        r"""Predict links between nodes specified in :obj:`edge_label_index`.

        Args:
            prob (bool, optional): Whether probabilities should be returned.
                (default: :obj:`False`)
        """
        pred = self(edge_index, edge_label_index, edge_weight).sigmoid()
        return pred if prob else pred.round()

    def recommend(
        self,
        edge_index: Adj,
        edge_weight: OptTensor = None,
        src_index: OptTensor = None,
        dst_index: OptTensor = None,
        k: int = 1,
    ) -> Tensor:
        r"""Get top-:math:`k` recommendations for nodes in :obj:`src_index`.

        Args:
            src_index (torch.Tensor, optional): Node indices for which
                recommendations should be generated.
                If set to :obj:`None`, all nodes will be used.
                (default: :obj:`None`)
            dst_index (torch.Tensor, optional): Node indices which represent
                the possible recommendation choices.
                If set to :obj:`None`, all nodes will be used.
                (default: :obj:`None`)
            k (int, optional): Number of recommendations. (default: :obj:`1`)
        """
        out_src = out_dst = self.get_embedding(edge_index, edge_weight)

        if src_index is not None:
            out_src = out_src[src_index]

        if dst_index is not None:
            out_dst = out_dst[dst_index]

        pred = out_src @ out_dst.t()
        top_index = pred.topk(k, dim=-1).indices

        if dst_index is not None:  # Map local top-indices to original indices.
            top_index = dst_index[top_index.view(-1)].view(*top_index.size())

        return top_index

    def link_pred_loss(self, pred: Tensor, edge_label: Tensor,
                       **kwargs) -> Tensor:
        r"""Computes the model loss for a link prediction objective via the
        :class:`torch.nn.BCEWithLogitsLoss`.

        Args:
            pred (torch.Tensor): The predictions.
            edge_label (torch.Tensor): The ground-truth edge labels.
            **kwargs (optional): Additional arguments of the underlying
                :class:`torch.nn.BCEWithLogitsLoss` loss function.
        """
        loss_fn = torch.nn.BCEWithLogitsLoss(**kwargs)
        return loss_fn(pred, edge_label.to(pred.dtype))

    def recommendation_loss(
        self,
        pos_edge_rank: Tensor,
        neg_edge_rank: Tensor,
        node_id: Optional[Tensor] = None,
        lambda_reg: float = 1e-4,
        **kwargs,
    ) -> Tensor:
        r"""Computes the model loss for a ranking objective via the Bayesian
        Personalized Ranking (BPR) loss.

        .. note::

            The i-th entry in the :obj:`pos_edge_rank` vector and i-th entry
            in the :obj:`neg_edge_rank` entry must correspond to ranks of
            positive and negative edges of the same entity (*e.g.*, user).

        Args:
            pos_edge_rank (torch.Tensor): Positive edge rankings.
            neg_edge_rank (torch.Tensor): Negative edge rankings.
            node_id (torch.Tensor): The indices of the nodes involved for
                deriving a prediction for both positive and negative edges.
                If set to :obj:`None`, all nodes will be used.
            lambda_reg (int, optional): The :math:`L_2` regularization strength
                of the Bayesian Personalized Ranking (BPR) loss.
                (default: :obj:`1e-4`)
            **kwargs (optional): Additional arguments of the underlying
                :class:`torch_geometric.nn.models.lightgcn.BPRLoss` loss
                function.
        """
        loss_fn = BPRLoss(lambda_reg, **kwargs)
        emb = self.embedding.weight
        emb = emb if node_id is None else emb[node_id]
        return loss_fn(pos_edge_rank, neg_edge_rank, emb)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.num_nodes}, '
                f'{self.embedding_dim}, num_layers={self.num_layers})')


class BPRLoss(_Loss):
    r"""The Bayesian Personalized Ranking (BPR) loss.

    The BPR loss is a pairwise loss that encourages the prediction of an
    observed entry to be higher than its unobserved counterparts
    (see `here `__).

    .. math::
        L_{\text{BPR}} = - \sum_{u=1}^{M} \sum_{i \in \mathcal{N}_u}
        \sum_{j \not\in \mathcal{N}_u} \ln \sigma(\hat{y}_{ui} - \hat{y}_{uj})
        + \lambda \vert\vert \textbf{x}^{(0)} \vert\vert^2

    where :math:`lambda` controls the :math:`L_2` regularization strength.
    We compute the mean BPR loss for simplicity.

    Args:
        lambda_reg (float, optional): The :math:`L_2` regularization strength
            (default: 0).
        **kwargs (optional): Additional arguments of the underlying
            :class:`torch.nn.modules.loss._Loss` class.
    """
    __constants__ = ['lambda_reg']
    lambda_reg: float

    def __init__(self, lambda_reg: float = 0, **kwargs):
        super().__init__(None, None, "sum", **kwargs)
        self.lambda_reg = lambda_reg

    def forward(self, positives: Tensor, negatives: Tensor,
                parameters: Tensor = None) -> Tensor:
        r"""Compute the mean Bayesian Personalized Ranking (BPR) loss.

        .. note::

            The i-th entry in the :obj:`positives` vector and i-th entry
            in the :obj:`negatives` entry should correspond to the same
            entity (*.e.g*, user), as the BPR is a personalized ranking loss.

        Args:
            positives (Tensor): The vector of positive-pair rankings.
            negatives (Tensor): The vector of negative-pair rankings.
            parameters (Tensor, optional): The tensor of parameters which
                should be used for :math:`L_2` regularization
                (default: :obj:`None`).
        """
        log_prob = F.logsigmoid(positives - negatives).mean()

        regularization = 0
        if self.lambda_reg != 0:
            regularization = self.lambda_reg * parameters.norm(p=2).pow(2)
            regularization = regularization / positives.size(0)

        return -log_prob + regularization
pytorch/torch_geometric/nn/models/linkx.py000066400000000000000000000173351444262304300213330ustar00rootroot00000000000000import math

import torch
from torch import Tensor
from torch.nn import BatchNorm1d, Parameter

from torch_geometric.nn import inits
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.models import MLP
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import spmm


class SparseLinear(MessagePassing):
    def __init__(self, in_channels: int, out_channels: int, bias: bool = True):
        super().__init__(aggr='add')
        self.in_channels = in_channels
        self.out_channels = out_channels

        self.weight = Parameter(torch.Tensor(in_channels, out_channels))
        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        inits.kaiming_uniform(self.weight, fan=self.in_channels,
                              a=math.sqrt(5))
        inits.uniform(self.in_channels, self.bias)

    @torch.jit._overload_method
    def forward(self, edge_index, edge_weight=None):
        # type: (SparseTensor, OptTensor) -> Tensor
        pass

    @torch.jit._overload_method
    def forward(self, edge_index, edge_weight=None):
        # type: (Tensor, OptTensor) -> Tensor
        pass

    def forward(
        self,
        edge_index: Adj,
        edge_weight: OptTensor = None,
    ) -> Tensor:
        # propagate_type: (weight: Tensor, edge_weight: OptTensor)
        out = self.propagate(edge_index, weight=self.weight,
                             edge_weight=edge_weight, size=None)

        if self.bias is not None:
            out = out + self.bias

        return out

    def message(self, weight_j: Tensor, edge_weight: OptTensor) -> Tensor:
        if edge_weight is None:
            return weight_j
        else:
            return edge_weight.view(-1, 1) * weight_j

    def message_and_aggregate(self, adj_t: SparseTensor,
                              weight: Tensor) -> Tensor:
        return spmm(adj_t, weight, reduce=self.aggr)


class LINKX(torch.nn.Module):
    r"""The LINKX model from the `"Large Scale Learning on Non-Homophilous
    Graphs: New Benchmarks and Strong Simple Methods"
    `_ paper

    .. math::
        \mathbf{H}_{\mathbf{A}} &= \textrm{MLP}_{\mathbf{A}}(\mathbf{A})

        \mathbf{H}_{\mathbf{X}} &= \textrm{MLP}_{\mathbf{X}}(\mathbf{X})

        \mathbf{Y} &= \textrm{MLP}_{f} \left( \sigma \left( \mathbf{W}
        [\mathbf{H}_{\mathbf{A}}, \mathbf{H}_{\mathbf{X}}] +
        \mathbf{H}_{\mathbf{A}} + \mathbf{H}_{\mathbf{X}} \right) \right)

    .. note::

        For an example of using LINKX, see `examples/linkx.py `_.

    Args:
        num_nodes (int): The number of nodes in the graph.
        in_channels (int): Size of each input sample, or :obj:`-1` to derive
            the size from the first input(s) to the forward method.
        hidden_channels (int): Size of each hidden sample.
        out_channels (int): Size of each output sample.
        num_layers (int): Number of layers of :math:`\textrm{MLP}_{f}`.
        num_edge_layers (int, optional): Number of layers of
            :math:`\textrm{MLP}_{\mathbf{A}}`. (default: :obj:`1`)
        num_node_layers (int, optional): Number of layers of
            :math:`\textrm{MLP}_{\mathbf{X}}`. (default: :obj:`1`)
        dropout (float, optional): Dropout probability of each hidden
            embedding. (default: :obj:`0.0`)
    """
    def __init__(
        self,
        num_nodes: int,
        in_channels: int,
        hidden_channels: int,
        out_channels: int,
        num_layers: int,
        num_edge_layers: int = 1,
        num_node_layers: int = 1,
        dropout: float = 0.0,
    ):
        super().__init__()

        self.num_nodes = num_nodes
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.num_edge_layers = num_edge_layers

        self.edge_lin = SparseLinear(num_nodes, hidden_channels)

        if self.num_edge_layers > 1:
            self.edge_norm = BatchNorm1d(hidden_channels)
            channels = [hidden_channels] * num_edge_layers
            self.edge_mlp = MLP(channels, dropout=0., act_first=True)
        else:
            self.edge_norm = None
            self.edge_mlp = None

        channels = [in_channels] + [hidden_channels] * num_node_layers
        self.node_mlp = MLP(channels, dropout=0., act_first=True)

        self.cat_lin1 = torch.nn.Linear(hidden_channels, hidden_channels)
        self.cat_lin2 = torch.nn.Linear(hidden_channels, hidden_channels)

        channels = [hidden_channels] * num_layers + [out_channels]
        self.final_mlp = MLP(channels, dropout=dropout, act_first=True)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.edge_lin.reset_parameters()
        if self.edge_norm is not None:
            self.edge_norm.reset_parameters()
        if self.edge_mlp is not None:
            self.edge_mlp.reset_parameters()
        self.node_mlp.reset_parameters()
        self.cat_lin1.reset_parameters()
        self.cat_lin2.reset_parameters()
        self.final_mlp.reset_parameters()

    @torch.jit._overload_method
    def forward(self, x, edge_index, edge_weight=None):
        # type: (OptTensor, SparseTensor, OptTensor) -> Tensor
        pass

    @torch.jit._overload_method
    def forward(self, x, edge_index, edge_weight=None):
        # type: (OptTensor, Tensor, OptTensor) -> Tensor
        pass

    def forward(
        self,
        x: OptTensor,
        edge_index: Adj,
        edge_weight: OptTensor = None,
    ) -> Tensor:
        """"""
        out = self.edge_lin(edge_index, edge_weight)

        if self.edge_norm is not None and self.edge_mlp is not None:
            out = out.relu_()
            out = self.edge_norm(out)
            out = self.edge_mlp(out)

        out = out + self.cat_lin1(out)

        if x is not None:
            x = self.node_mlp(x)
            out = out + x
            out = out + self.cat_lin2(x)

        return self.final_mlp(out.relu_())

    def jittable(self, typing: str) -> torch.nn.Module:  # pragma: no cover
        edge_index_type = typing.split(',')[1].strip()

        class EdgeIndexJittable(torch.nn.Module):
            def __init__(self, child):
                super().__init__()
                self.child = child

            def reset_parameters(self):
                self.child.reset_parameters()

            def forward(self, x: Tensor, edge_index: Tensor,
                        edge_weight: OptTensor = None) -> Tensor:
                return self.child(x, edge_index, edge_weight)

        class SparseTensorJittable(torch.nn.Module):
            def __init__(self, child):
                super().__init__()
                self.child = child

            def reset_parameters(self):
                self.child.reset_parameters()

            def forward(self, x: Tensor, edge_index: SparseTensor,
                        edge_weight: OptTensor = None):
                return self.child(x, edge_index, edge_weight)

        if self.edge_lin.jittable is not None:
            self.edge_lin = self.edge_lin.jittable()

        if 'Tensor' == edge_index_type:
            jittable_module = EdgeIndexJittable(self)
        elif 'SparseTensor' == edge_index_type:
            jittable_module = SparseTensorJittable(self)
        else:
            raise ValueError(f"Could not parse types '{typing}'")

        return jittable_module

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_nodes={self.num_nodes}, '
                f'in_channels={self.in_channels}, '
                f'out_channels={self.out_channels})')
pytorch/torch_geometric/nn/models/mask_label.py000066400000000000000000000050061444262304300222700ustar00rootroot00000000000000import torch
from torch import Tensor


class MaskLabel(torch.nn.Module):
    r"""The label embedding and masking layer from the `"Masked Label
    Prediction: Unified Message Passing Model for Semi-Supervised
    Classification" `_ paper.

    Here, node labels :obj:`y` are merged to the initial node features :obj:`x`
    for a subset of their nodes according to :obj:`mask`.

    .. note::

        For an example of using :class:`MaskLabel`, see
        `examples/unimp_arxiv.py `_.


    Args:
        num_classes (int): The number of classes.
        out_channels (int): Size of each output sample.
        method (str, optional): If set to :obj:`"add"`, label embeddings are
            added to the input. If set to :obj:`"concat"`, label embeddings are
            concatenated. In case :obj:`method="add"`, then :obj:`out_channels`
            needs to be identical to the input dimensionality of node features.
            (default: :obj:`"add"`)
    """
    def __init__(self, num_classes: int, out_channels: int,
                 method: str = "add"):
        super().__init__()

        self.method = method
        if method not in ["add", "concat"]:
            raise ValueError(
                f"'method' must be either 'add' or 'concat' (got '{method}')")

        self.emb = torch.nn.Embedding(num_classes, out_channels)
        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.emb.reset_parameters()

    def forward(self, x: Tensor, y: Tensor, mask: Tensor) -> Tensor:
        """"""
        if self.method == "concat":
            out = x.new_zeros(y.size(0), self.emb.weight.size(-1))
            out[mask] = self.emb(y[mask])
            return torch.cat([x, out], dim=-1)
        else:
            x = torch.clone(x)
            x[mask] += self.emb(y[mask])
            return x

    @staticmethod
    def ratio_mask(mask: Tensor, ratio: float):
        r"""Modifies :obj:`mask` by setting :obj:`ratio` of :obj:`True`
        entries to :obj:`False`. Does not operate in-place.

        Args:
            mask (torch.Tensor): The mask to re-mask.
            ratio (float): The ratio of entries to keep.
        """
        n = int(mask.sum())
        out = mask.clone()
        out[mask] = torch.rand(n, device=mask.device) < ratio
        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/models/meta.py000066400000000000000000000146011444262304300211250ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor


class MetaLayer(torch.nn.Module):
    r"""A meta layer for building any kind of graph network, inspired by the
    `"Relational Inductive Biases, Deep Learning, and Graph Networks"
    `_ paper.

    A graph network takes a graph as input and returns an updated graph as
    output (with same connectivity).
    The input graph has node features :obj:`x`, edge features :obj:`edge_attr`
    as well as graph-level features :obj:`u`.
    The output graph has the same structure, but updated features.

    Edge features, node features as well as global features are updated by
    calling the modules :obj:`edge_model`, :obj:`node_model` and
    :obj:`global_model`, respectively.

    To allow for batch-wise graph processing, all callable functions take an
    additional argument :obj:`batch`, which determines the assignment of
    edges or nodes to their specific graphs.

    Args:
        edge_model (torch.nn.Module, optional): A callable which updates a
            graph's edge features based on its source and target node features,
            its current edge features and its global features.
            (default: :obj:`None`)
        node_model (torch.nn.Module, optional): A callable which updates a
            graph's node features based on its current node features, its graph
            connectivity, its edge features and its global features.
            (default: :obj:`None`)
        global_model (torch.nn.Module, optional): A callable which updates a
            graph's global features based on its node features, its graph
            connectivity, its edge features and its current global features.
            (default: :obj:`None`)

    .. code-block:: python

        from torch.nn import Sequential as Seq, Linear as Lin, ReLU
        from torch_geometric.utils import scatter
        from torch_geometric.nn import MetaLayer

        class EdgeModel(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.edge_mlp = Seq(Lin(..., ...), ReLU(), Lin(..., ...))

            def forward(self, src, dest, edge_attr, u, batch):
                # src, dest: [E, F_x], where E is the number of edges.
                # edge_attr: [E, F_e]
                # u: [B, F_u], where B is the number of graphs.
                # batch: [E] with max entry B - 1.
                out = torch.cat([src, dest, edge_attr, u[batch]], 1)
                return self.edge_mlp(out)

        class NodeModel(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.node_mlp_1 = Seq(Lin(..., ...), ReLU(), Lin(..., ...))
                self.node_mlp_2 = Seq(Lin(..., ...), ReLU(), Lin(..., ...))

            def forward(self, x, edge_index, edge_attr, u, batch):
                # x: [N, F_x], where N is the number of nodes.
                # edge_index: [2, E] with max entry N - 1.
                # edge_attr: [E, F_e]
                # u: [B, F_u]
                # batch: [N] with max entry B - 1.
                row, col = edge_index
                out = torch.cat([x[row], edge_attr], dim=1)
                out = self.node_mlp_1(out)
                out = scatter(out, col, dim=0, dim_size=x.size(0),
                              reduce='mean')
                out = torch.cat([x, out, u[batch]], dim=1)
                return self.node_mlp_2(out)

        class GlobalModel(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.global_mlp = Seq(Lin(..., ...), ReLU(), Lin(..., ...))

            def forward(self, x, edge_index, edge_attr, u, batch):
                # x: [N, F_x], where N is the number of nodes.
                # edge_index: [2, E] with max entry N - 1.
                # edge_attr: [E, F_e]
                # u: [B, F_u]
                # batch: [N] with max entry B - 1.
                out = torch.cat([
                    u,
                    scatter(x, batch, dim=0, reduce='mean'),
                ], dim=1)
                return self.global_mlp(out)

        op = MetaLayer(EdgeModel(), NodeModel(), GlobalModel())
        x, edge_attr, u = op(x, edge_index, edge_attr, u, batch)
    """
    def __init__(
        self,
        edge_model: Optional[torch.nn.Module] = None,
        node_model: Optional[torch.nn.Module] = None,
        global_model: Optional[torch.nn.Module] = None,
    ):
        super().__init__()
        self.edge_model = edge_model
        self.node_model = node_model
        self.global_model = global_model

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for item in [self.node_model, self.edge_model, self.global_model]:
            if hasattr(item, 'reset_parameters'):
                item.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_attr: Optional[Tensor] = None,
        u: Optional[Tensor] = None,
        batch: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Optional[Tensor], Optional[Tensor]]:
        r"""
        Args:
            x (torch.Tensor): The node features.
            edge_index (torch.Tensor): The edge indices.
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
            u (torch.Tensor, optional): The global graph features.
                (default: :obj:`None`)
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific graph. (default: :obj:`None`)
        """
        row = edge_index[0]
        col = edge_index[1]

        if self.edge_model is not None:
            edge_attr = self.edge_model(x[row], x[col], edge_attr, u,
                                        batch if batch is None else batch[row])

        if self.node_model is not None:
            x = self.node_model(x, edge_index, edge_attr, u, batch)

        if self.global_model is not None:
            u = self.global_model(x, edge_index, edge_attr, u, batch)

        return x, edge_attr, u

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(\n'
                f'  edge_model={self.edge_model},\n'
                f'  node_model={self.node_model},\n'
                f'  global_model={self.global_model}\n'
                f')')
pytorch/torch_geometric/nn/models/metapath2vec.py000066400000000000000000000241161444262304300225640ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Embedding
from torch.utils.data import DataLoader

from torch_geometric.typing import EdgeType, NodeType, OptTensor
from torch_geometric.utils import sort_edge_index
from torch_geometric.utils.sparse import index2ptr

EPS = 1e-15


class MetaPath2Vec(torch.nn.Module):
    r"""The MetaPath2Vec model from the `"metapath2vec: Scalable Representation
    Learning for Heterogeneous Networks"
    `_ paper where random walks based
    on a given :obj:`metapath` are sampled in a heterogeneous graph, and node
    embeddings are learned via negative sampling optimization.

    .. note::

        For an example of using MetaPath2Vec, see
        `examples/hetero/metapath2vec.py
        `_.

    Args:
        edge_index_dict (Dict[Tuple[str, str, str], torch.Tensor]): Dictionary
            holding edge indices for each
            :obj:`(src_node_type, rel_type, dst_node_type)` edge type present
            in the heterogeneous graph.
        embedding_dim (int): The size of each embedding vector.
        metapath (List[Tuple[str, str, str]]): The metapath described as a list
            of :obj:`(src_node_type, rel_type, dst_node_type)` tuples.
        walk_length (int): The walk length.
        context_size (int): The actual context size which is considered for
            positive samples. This parameter increases the effective sampling
            rate by reusing samples across different source nodes.
        walks_per_node (int, optional): The number of walks to sample for each
            node. (default: :obj:`1`)
        num_negative_samples (int, optional): The number of negative samples to
            use for each positive sample. (default: :obj:`1`)
        num_nodes_dict (Dict[str, int], optional): Dictionary holding the
            number of nodes for each node type. (default: :obj:`None`)
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to the
            weight matrix will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        edge_index_dict: Dict[EdgeType, Tensor],
        embedding_dim: int,
        metapath: List[EdgeType],
        walk_length: int,
        context_size: int,
        walks_per_node: int = 1,
        num_negative_samples: int = 1,
        num_nodes_dict: Optional[Dict[NodeType, int]] = None,
        sparse: bool = False,
    ):
        super().__init__()

        if num_nodes_dict is None:
            num_nodes_dict = {}
            for keys, edge_index in edge_index_dict.items():
                key = keys[0]
                N = int(edge_index[0].max() + 1)
                num_nodes_dict[key] = max(N, num_nodes_dict.get(key, N))

                key = keys[-1]
                N = int(edge_index[1].max() + 1)
                num_nodes_dict[key] = max(N, num_nodes_dict.get(key, N))

        self.rowptr_dict, self.col_dict, self.rowcount_dict = {}, {}, {}
        for keys, edge_index in edge_index_dict.items():
            sizes = (num_nodes_dict[keys[0]], num_nodes_dict[keys[-1]])
            row, col = sort_edge_index(edge_index, num_nodes=max(sizes)).cpu()
            rowptr = index2ptr(row, size=sizes[0])
            self.rowptr_dict[keys] = rowptr
            self.col_dict[keys] = col
            self.rowcount_dict[keys] = rowptr[1:] - rowptr[:-1]

        assert walk_length + 1 >= context_size
        if walk_length > len(metapath) and metapath[0][0] != metapath[-1][-1]:
            raise AttributeError(
                "The 'walk_length' is longer than the given 'metapath', but "
                "the 'metapath' does not denote a cycle")

        self.embedding_dim = embedding_dim
        self.metapath = metapath
        self.walk_length = walk_length
        self.context_size = context_size
        self.walks_per_node = walks_per_node
        self.num_negative_samples = num_negative_samples
        self.num_nodes_dict = num_nodes_dict

        types = set([x[0] for x in metapath]) | set([x[-1] for x in metapath])
        types = sorted(list(types))

        count = 0
        self.start, self.end = {}, {}
        for key in types:
            self.start[key] = count
            count += num_nodes_dict[key]
            self.end[key] = count

        offset = [self.start[metapath[0][0]]]
        offset += [self.start[keys[-1]] for keys in metapath
                   ] * int((walk_length / len(metapath)) + 1)
        offset = offset[:walk_length + 1]
        assert len(offset) == walk_length + 1
        self.offset = torch.tensor(offset)

        # + 1 denotes a dummy node used to link to for isolated nodes.
        self.embedding = Embedding(count + 1, embedding_dim, sparse=sparse)
        self.dummy_idx = count

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.embedding.reset_parameters()

    def forward(self, node_type: str, batch: OptTensor = None) -> Tensor:
        r"""Returns the embeddings for the nodes in :obj:`batch` of type
        :obj:`node_type`."""
        emb = self.embedding.weight[self.start[node_type]:self.end[node_type]]
        return emb if batch is None else emb.index_select(0, batch)

    def loader(self, **kwargs):
        r"""Returns the data loader that creates both positive and negative
        random walks on the heterogeneous graph.

        Args:
            **kwargs (optional): Arguments of
                :class:`torch.utils.data.DataLoader`, such as
                :obj:`batch_size`, :obj:`shuffle`, :obj:`drop_last` or
                :obj:`num_workers`.
        """
        return DataLoader(range(self.num_nodes_dict[self.metapath[0][0]]),
                          collate_fn=self._sample, **kwargs)

    def _pos_sample(self, batch: Tensor) -> Tensor:
        batch = batch.repeat(self.walks_per_node)

        rws = [batch]
        for i in range(self.walk_length):
            edge_type = self.metapath[i % len(self.metapath)]
            batch = sample(
                self.rowptr_dict[edge_type],
                self.col_dict[edge_type],
                self.rowcount_dict[edge_type],
                batch,
                num_neighbors=1,
                dummy_idx=self.dummy_idx,
            ).view(-1)
            rws.append(batch)

        rw = torch.stack(rws, dim=-1)
        rw.add_(self.offset.view(1, -1))
        rw[rw > self.dummy_idx] = self.dummy_idx

        walks = []
        num_walks_per_rw = 1 + self.walk_length + 1 - self.context_size
        for j in range(num_walks_per_rw):
            walks.append(rw[:, j:j + self.context_size])
        return torch.cat(walks, dim=0)

    def _neg_sample(self, batch: Tensor) -> Tensor:
        batch = batch.repeat(self.walks_per_node * self.num_negative_samples)

        rws = [batch]
        for i in range(self.walk_length):
            keys = self.metapath[i % len(self.metapath)]
            batch = torch.randint(0, self.num_nodes_dict[keys[-1]],
                                  (batch.size(0), ), dtype=torch.long)
            rws.append(batch)

        rw = torch.stack(rws, dim=-1)
        rw.add_(self.offset.view(1, -1))

        walks = []
        num_walks_per_rw = 1 + self.walk_length + 1 - self.context_size
        for j in range(num_walks_per_rw):
            walks.append(rw[:, j:j + self.context_size])
        return torch.cat(walks, dim=0)

    def _sample(self, batch: List[int]) -> Tuple[Tensor, Tensor]:
        if not isinstance(batch, Tensor):
            batch = torch.tensor(batch, dtype=torch.long)
        return self._pos_sample(batch), self._neg_sample(batch)

    def loss(self, pos_rw: Tensor, neg_rw: Tensor) -> Tensor:
        r"""Computes the loss given positive and negative random walks."""

        # Positive loss.
        start, rest = pos_rw[:, 0], pos_rw[:, 1:].contiguous()

        h_start = self.embedding(start).view(pos_rw.size(0), 1,
                                             self.embedding_dim)
        h_rest = self.embedding(rest.view(-1)).view(pos_rw.size(0), -1,
                                                    self.embedding_dim)

        out = (h_start * h_rest).sum(dim=-1).view(-1)
        pos_loss = -torch.log(torch.sigmoid(out) + EPS).mean()

        # Negative loss.
        start, rest = neg_rw[:, 0], neg_rw[:, 1:].contiguous()

        h_start = self.embedding(start).view(neg_rw.size(0), 1,
                                             self.embedding_dim)
        h_rest = self.embedding(rest.view(-1)).view(neg_rw.size(0), -1,
                                                    self.embedding_dim)

        out = (h_start * h_rest).sum(dim=-1).view(-1)
        neg_loss = -torch.log(1 - torch.sigmoid(out) + EPS).mean()

        return pos_loss + neg_loss

    def test(self, train_z: Tensor, train_y: Tensor, test_z: Tensor,
             test_y: Tensor, solver: str = "lbfgs", multi_class: str = "auto",
             *args, **kwargs) -> float:
        r"""Evaluates latent space quality via a logistic regression downstream
        task."""
        from sklearn.linear_model import LogisticRegression

        clf = LogisticRegression(solver=solver, multi_class=multi_class, *args,
                                 **kwargs).fit(train_z.detach().cpu().numpy(),
                                               train_y.detach().cpu().numpy())
        return clf.score(test_z.detach().cpu().numpy(),
                         test_y.detach().cpu().numpy())

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'{self.embedding.weight.size(0) - 1}, '
                f'{self.embedding.weight.size(1)})')


def sample(rowptr: Tensor, col: Tensor, rowcount: Tensor, subset: Tensor,
           num_neighbors: int, dummy_idx: int) -> Tensor:

    rand = torch.rand((subset.size(0), num_neighbors), device=subset.device)
    rand *= rowcount[subset].to(rand.dtype).view(-1, 1)
    rand = rand.to(torch.long) + rowptr[subset].view(-1, 1)

    col = col[rand]
    col[(subset >= dummy_idx) | (rowcount[subset] == 0)] = dummy_idx
    return col
pytorch/torch_geometric/nn/models/mlp.py000066400000000000000000000214241444262304300207700ustar00rootroot00000000000000import warnings
from typing import Any, Callable, Dict, List, Optional, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Identity

from torch_geometric.nn.dense.linear import Linear
from torch_geometric.nn.resolver import (
    activation_resolver,
    normalization_resolver,
)
from torch_geometric.typing import NoneType


class MLP(torch.nn.Module):
    r"""A Multi-Layer Perception (MLP) model.
    There exists two ways to instantiate an :class:`MLP`:

    1. By specifying explicit channel sizes, *e.g.*,

       .. code-block:: python

          mlp = MLP([16, 32, 64, 128])

       creates a three-layer MLP with **differently** sized hidden layers.

    1. By specifying fixed hidden channel sizes over a number of layers,
       *e.g.*,

       .. code-block:: python

          mlp = MLP(in_channels=16, hidden_channels=32,
                    out_channels=128, num_layers=3)

       creates a three-layer MLP with **equally** sized hidden layers.

    Args:
        channel_list (List[int] or int, optional): List of input, intermediate
            and output channels such that :obj:`len(channel_list) - 1` denotes
            the number of layers of the MLP (default: :obj:`None`)
        in_channels (int, optional): Size of each input sample.
            Will override :attr:`channel_list`. (default: :obj:`None`)
        hidden_channels (int, optional): Size of each hidden sample.
            Will override :attr:`channel_list`. (default: :obj:`None`)
        out_channels (int, optional): Size of each output sample.
            Will override :attr:`channel_list`. (default: :obj:`None`)
        num_layers (int, optional): The number of layers.
            Will override :attr:`channel_list`. (default: :obj:`None`)
        dropout (float or List[float], optional): Dropout probability of each
            hidden embedding. If a list is provided, sets the dropout value per
            layer. (default: :obj:`0.`)
        act (str or Callable, optional): The non-linear activation function to
            use. (default: :obj:`"relu"`)
        act_first (bool, optional): If set to :obj:`True`, activation is
            applied before normalization. (default: :obj:`False`)
        act_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective activation function defined by :obj:`act`.
            (default: :obj:`None`)
        norm (str or Callable, optional): The normalization function to
            use. (default: :obj:`"batch_norm"`)
        norm_kwargs (Dict[str, Any], optional): Arguments passed to the
            respective normalization function defined by :obj:`norm`.
            (default: :obj:`None`)
        plain_last (bool, optional): If set to :obj:`False`, will apply
            non-linearity, batch normalization and dropout to the last layer as
            well. (default: :obj:`True`)
        bias (bool or List[bool], optional): If set to :obj:`False`, the module
            will not learn additive biases. If a list is provided, sets the
            bias per layer. (default: :obj:`True`)
        **kwargs (optional): Additional deprecated arguments of the MLP layer.
    """
    def __init__(
        self,
        channel_list: Optional[Union[List[int], int]] = None,
        *,
        in_channels: Optional[int] = None,
        hidden_channels: Optional[int] = None,
        out_channels: Optional[int] = None,
        num_layers: Optional[int] = None,
        dropout: Union[float, List[float]] = 0.,
        act: Union[str, Callable, None] = "relu",
        act_first: bool = False,
        act_kwargs: Optional[Dict[str, Any]] = None,
        norm: Union[str, Callable, None] = "batch_norm",
        norm_kwargs: Optional[Dict[str, Any]] = None,
        plain_last: bool = True,
        bias: Union[bool, List[bool]] = True,
        **kwargs,
    ):
        super().__init__()

        # Backward compatibility:
        act_first = act_first or kwargs.get("relu_first", False)
        batch_norm = kwargs.get("batch_norm", None)
        if batch_norm is not None and isinstance(batch_norm, bool):
            warnings.warn("Argument `batch_norm` is deprecated, "
                          "please use `norm` to specify normalization layer.")
            norm = 'batch_norm' if batch_norm else None
            batch_norm_kwargs = kwargs.get("batch_norm_kwargs", None)
            norm_kwargs = batch_norm_kwargs or {}

        if isinstance(channel_list, int):
            in_channels = channel_list

        if in_channels is not None:
            if num_layers is None:
                raise ValueError("Argument `num_layers` must be given")
            if num_layers > 1 and hidden_channels is None:
                raise ValueError(f"Argument `hidden_channels` must be given "
                                 f"for `num_layers={num_layers}`")
            if out_channels is None:
                raise ValueError("Argument `out_channels` must be given")

            channel_list = [hidden_channels] * (num_layers - 1)
            channel_list = [in_channels] + channel_list + [out_channels]

        assert isinstance(channel_list, (tuple, list))
        assert len(channel_list) >= 2
        self.channel_list = channel_list

        self.act = activation_resolver(act, **(act_kwargs or {}))
        self.act_first = act_first
        self.plain_last = plain_last

        if isinstance(dropout, float):
            dropout = [dropout] * (len(channel_list) - 1)
            if plain_last:
                dropout[-1] = 0.
        if len(dropout) != len(channel_list) - 1:
            raise ValueError(
                f"Number of dropout values provided ({len(dropout)} does not "
                f"match the number of layers specified "
                f"({len(channel_list)-1})")
        self.dropout = dropout

        if isinstance(bias, bool):
            bias = [bias] * (len(channel_list) - 1)
        if len(bias) != len(channel_list) - 1:
            raise ValueError(
                f"Number of bias values provided ({len(bias)}) does not match "
                f"the number of layers specified ({len(channel_list)-1})")

        self.lins = torch.nn.ModuleList()
        iterator = zip(channel_list[:-1], channel_list[1:], bias)
        for in_channels, out_channels, _bias in iterator:
            self.lins.append(Linear(in_channels, out_channels, bias=_bias))

        self.norms = torch.nn.ModuleList()
        iterator = channel_list[1:-1] if plain_last else channel_list[1:]
        for hidden_channels in iterator:
            if norm is not None:
                norm_layer = normalization_resolver(
                    norm,
                    hidden_channels,
                    **(norm_kwargs or {}),
                )
            else:
                norm_layer = Identity()
            self.norms.append(norm_layer)

        self.reset_parameters()

    @property
    def in_channels(self) -> int:
        r"""Size of each input sample."""
        return self.channel_list[0]

    @property
    def out_channels(self) -> int:
        r"""Size of each output sample."""
        return self.channel_list[-1]

    @property
    def num_layers(self) -> int:
        r"""The number of layers."""
        return len(self.channel_list) - 1

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for lin in self.lins:
            lin.reset_parameters()
        for norm in self.norms:
            if hasattr(norm, 'reset_parameters'):
                norm.reset_parameters()

    def forward(
        self,
        x: Tensor,
        return_emb: NoneType = None,
    ) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            return_emb (bool, optional): If set to :obj:`True`, will
                additionally return the embeddings before execution of the
                final output layer. (default: :obj:`False`)
        """
        # `return_emb` is annotated here as `NoneType` to be compatible with
        # TorchScript, which does not support different return types based on
        # the value of an input argument.
        emb: Optional[Tensor] = None

        for i, (lin, norm) in enumerate(zip(self.lins, self.norms)):
            x = lin(x)
            if self.act is not None and self.act_first:
                x = self.act(x)
            x = norm(x)
            if self.act is not None and not self.act_first:
                x = self.act(x)
            x = F.dropout(x, p=self.dropout[i], training=self.training)
            if isinstance(return_emb, bool) and return_emb is True:
                emb = x

        if self.plain_last:
            x = self.lins[-1](x)
            x = F.dropout(x, p=self.dropout[-1], training=self.training)

        return (x, emb) if isinstance(return_emb, bool) else x

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({str(self.channel_list)[1:-1]})'
pytorch/torch_geometric/nn/models/node2vec.py000066400000000000000000000167311444262304300217120ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Embedding
from torch.utils.data import DataLoader

from torch_geometric.typing import WITH_PYG_LIB, WITH_TORCH_CLUSTER
from torch_geometric.utils import sort_edge_index
from torch_geometric.utils.num_nodes import maybe_num_nodes
from torch_geometric.utils.sparse import index2ptr


class Node2Vec(torch.nn.Module):
    r"""The Node2Vec model from the
    `"node2vec: Scalable Feature Learning for Networks"
    `_ paper where random walks of
    length :obj:`walk_length` are sampled in a given graph, and node embeddings
    are learned via negative sampling optimization.

    .. note::

        For an example of using Node2Vec, see `examples/node2vec.py
        `_.

    Args:
        edge_index (torch.Tensor): The edge indices.
        embedding_dim (int): The size of each embedding vector.
        walk_length (int): The walk length.
        context_size (int): The actual context size which is considered for
            positive samples. This parameter increases the effective sampling
            rate by reusing samples across different source nodes.
        walks_per_node (int, optional): The number of walks to sample for each
            node. (default: :obj:`1`)
        p (float, optional): Likelihood of immediately revisiting a node in the
            walk. (default: :obj:`1`)
        q (float, optional): Control parameter to interpolate between
            breadth-first strategy and depth-first strategy (default: :obj:`1`)
        num_negative_samples (int, optional): The number of negative samples to
            use for each positive sample. (default: :obj:`1`)
        num_nodes (int, optional): The number of nodes. (default: :obj:`None`)
        sparse (bool, optional): If set to :obj:`True`, gradients w.r.t. to the
            weight matrix will be sparse. (default: :obj:`False`)
    """
    def __init__(
        self,
        edge_index: Tensor,
        embedding_dim: int,
        walk_length: int,
        context_size: int,
        walks_per_node: int = 1,
        p: float = 1.0,
        q: float = 1.0,
        num_negative_samples: int = 1,
        num_nodes: Optional[int] = None,
        sparse: bool = False,
    ):
        super().__init__()

        if WITH_PYG_LIB and p == 1.0 and q == 1.0:
            self.random_walk_fn = torch.ops.pyg.random_walk
        elif WITH_TORCH_CLUSTER:
            self.random_walk_fn = torch.ops.torch_cluster.random_walk
        else:
            if p == 1.0 and q == 1.0:
                raise ImportError(f"'{self.__class__.__name__}' "
                                  f"requires either the 'pyg-lib' or "
                                  f"'torch-cluster' package")
            else:
                raise ImportError(f"'{self.__class__.__name__}' "
                                  f"requires the 'torch-cluster' package")

        self.num_nodes = maybe_num_nodes(edge_index, num_nodes)

        row, col = sort_edge_index(edge_index, num_nodes=self.num_nodes).cpu()
        self.rowptr, self.col = index2ptr(row, self.num_nodes), col

        self.EPS = 1e-15
        assert walk_length >= context_size

        self.embedding_dim = embedding_dim
        self.walk_length = walk_length - 1
        self.context_size = context_size
        self.walks_per_node = walks_per_node
        self.p = p
        self.q = q
        self.num_negative_samples = num_negative_samples

        self.embedding = Embedding(self.num_nodes, embedding_dim,
                                   sparse=sparse)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.embedding.reset_parameters()

    def forward(self, batch: Optional[Tensor] = None) -> Tensor:
        """Returns the embeddings for the nodes in :obj:`batch`."""
        emb = self.embedding.weight
        return emb if batch is None else emb.index_select(0, batch)

    def loader(self, **kwargs) -> DataLoader:
        return DataLoader(range(self.num_nodes), collate_fn=self.sample,
                          **kwargs)

    @torch.jit.export
    def pos_sample(self, batch: Tensor) -> Tensor:
        batch = batch.repeat(self.walks_per_node)
        rw = self.random_walk_fn(self.rowptr, self.col, batch,
                                 self.walk_length, self.p, self.q)
        if not isinstance(rw, Tensor):
            rw = rw[0]

        walks = []
        num_walks_per_rw = 1 + self.walk_length + 1 - self.context_size
        for j in range(num_walks_per_rw):
            walks.append(rw[:, j:j + self.context_size])
        return torch.cat(walks, dim=0)

    @torch.jit.export
    def neg_sample(self, batch: Tensor) -> Tensor:
        batch = batch.repeat(self.walks_per_node * self.num_negative_samples)

        rw = torch.randint(self.num_nodes, (batch.size(0), self.walk_length))
        rw = torch.cat([batch.view(-1, 1), rw], dim=-1)

        walks = []
        num_walks_per_rw = 1 + self.walk_length + 1 - self.context_size
        for j in range(num_walks_per_rw):
            walks.append(rw[:, j:j + self.context_size])
        return torch.cat(walks, dim=0)

    @torch.jit.export
    def sample(self, batch: Tensor) -> Tuple[Tensor, Tensor]:
        if not isinstance(batch, Tensor):
            batch = torch.tensor(batch)
        return self.pos_sample(batch), self.neg_sample(batch)

    @torch.jit.export
    def loss(self, pos_rw: Tensor, neg_rw: Tensor) -> Tensor:
        r"""Computes the loss given positive and negative random walks."""

        # Positive loss.
        start, rest = pos_rw[:, 0], pos_rw[:, 1:].contiguous()

        h_start = self.embedding(start).view(pos_rw.size(0), 1,
                                             self.embedding_dim)
        h_rest = self.embedding(rest.view(-1)).view(pos_rw.size(0), -1,
                                                    self.embedding_dim)

        out = (h_start * h_rest).sum(dim=-1).view(-1)
        pos_loss = -torch.log(torch.sigmoid(out) + self.EPS).mean()

        # Negative loss.
        start, rest = neg_rw[:, 0], neg_rw[:, 1:].contiguous()

        h_start = self.embedding(start).view(neg_rw.size(0), 1,
                                             self.embedding_dim)
        h_rest = self.embedding(rest.view(-1)).view(neg_rw.size(0), -1,
                                                    self.embedding_dim)

        out = (h_start * h_rest).sum(dim=-1).view(-1)
        neg_loss = -torch.log(1 - torch.sigmoid(out) + self.EPS).mean()

        return pos_loss + neg_loss

    def test(
        self,
        train_z: Tensor,
        train_y: Tensor,
        test_z: Tensor,
        test_y: Tensor,
        solver: str = 'lbfgs',
        multi_class: str = 'auto',
        *args,
        **kwargs,
    ) -> float:
        r"""Evaluates latent space quality via a logistic regression downstream
        task."""
        from sklearn.linear_model import LogisticRegression

        clf = LogisticRegression(solver=solver, multi_class=multi_class, *args,
                                 **kwargs).fit(train_z.detach().cpu().numpy(),
                                               train_y.detach().cpu().numpy())
        return clf.score(test_z.detach().cpu().numpy(),
                         test_y.detach().cpu().numpy())

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.embedding.weight.size(0)}, '
                f'{self.embedding.weight.size(1)})')
pytorch/torch_geometric/nn/models/pmlp.py000066400000000000000000000067041444262304300211540ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn import SimpleConv
from torch_geometric.nn.dense.linear import Linear


class PMLP(torch.nn.Module):
    r"""The P(ropagational)MLP model from the `"Graph Neural Networks are
    Inherently Good Generalizers: Insights by Bridging GNNs and MLPs"
    `_ paper.
    :class:`PMLP` is identical to a standard MLP during training, but then
    adopts a GNN architecture during testing.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Size of each hidden sample.
        out_channels (int): Size of each output sample.
        num_layers (int): The number of layers.
        dropout (float, optional): Dropout probability of each hidden
            embedding. (default: :obj:`0.`)
        norm (bool, optional): If set to :obj:`False`, will not apply batch
            normalization. (default: :obj:`True`)
        bias (bool, optional): If set to :obj:`False`, the module
            will not learn additive biases. (default: :obj:`True`)
    """
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        out_channels: int,
        num_layers: int,
        dropout: float = 0.,
        norm: bool = True,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.out_channels = out_channels
        self.num_layers = num_layers
        self.dropout = dropout
        self.bias = bias

        self.lins = torch.nn.ModuleList()
        self.lins.append(Linear(in_channels, hidden_channels, self.bias))
        for _ in range(self.num_layers - 2):
            lin = Linear(hidden_channels, hidden_channels, self.bias)
            self.lins.append(lin)
        self.lins.append(Linear(hidden_channels, out_channels, self.bias))

        self.norm = None
        if norm:
            self.norm = torch.nn.BatchNorm1d(
                hidden_channels,
                affine=False,
                track_running_stats=False,
            )

        self.conv = SimpleConv(aggr='mean', combine_root='self_loop')

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for lin in self.lins:
            torch.nn.init.xavier_uniform_(lin.weight, gain=1.414)
            if self.bias:
                torch.nn.init.zeros_(lin.bias)

    def forward(
        self,
        x: torch.Tensor,
        edge_index: Optional[Tensor] = None,
    ) -> torch.Tensor:
        """"""
        if not self.training and edge_index is None:
            raise ValueError(f"'edge_index' needs to be present during "
                             f"inference in '{self.__class__.__name__}'")

        for i in range(self.num_layers):
            x = x @ self.lins[i].weight.t()
            if not self.training:
                x = self.conv(x, edge_index)
            if self.bias:
                x = x + self.lins[i].bias
            if i != self.num_layers - 1:
                if self.norm is not None:
                    x = self.norm(x)
                x = x.relu()
                x = F.dropout(x, p=self.dropout, training=self.training)

        return x

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, num_layers={self.num_layers})')
pytorch/torch_geometric/nn/models/re_net.py000066400000000000000000000214231444262304300214530ustar00rootroot00000000000000import math
from typing import Callable, List, Tuple

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import GRU, Linear, Parameter

from torch_geometric.data.data import Data
from torch_geometric.utils import scatter


class RENet(torch.nn.Module):
    r"""The Recurrent Event Network model from the `"Recurrent Event Network
    for Reasoning over Temporal Knowledge Graphs"
    `_ paper

    .. math::
        f_{\mathbf{\Theta}}(\mathbf{e}_s, \mathbf{e}_r,
        \mathbf{h}^{(t-1)}(s, r))

    based on a RNN encoder

    .. math::
        \mathbf{h}^{(t)}(s, r) = \textrm{RNN}(\mathbf{e}_s, \mathbf{e}_r,
        g(\mathcal{O}^{(t)}_r(s)), \mathbf{h}^{(t-1)}(s, r))

    where :math:`\mathbf{e}_s` and :math:`\mathbf{e}_r` denote entity and
    relation embeddings, and :math:`\mathcal{O}^{(t)}_r(s)` represents the set
    of objects interacted with subject :math:`s` under relation :math:`r` at
    timestamp :math:`t`.
    This model implements :math:`g` as the **Mean Aggregator** and
    :math:`f_{\mathbf{\Theta}}` as a linear projection.

    Args:
        num_nodes (int): The number of nodes in the knowledge graph.
        num_rels (int): The number of relations in the knowledge graph.
        hidden_channels (int): Hidden size of node and relation embeddings.
        seq_len (int): The sequence length of past events.
        num_layers (int, optional): The number of recurrent layers.
            (default: :obj:`1`)
        dropout (float): If non-zero, introduces a dropout layer before the
            final prediction. (default: :obj:`0.`)
        bias (bool, optional): If set to :obj:`False`, all layers will not
            learn an additive bias. (default: :obj:`True`)
    """
    def __init__(
        self,
        num_nodes: int,
        num_rels: int,
        hidden_channels: int,
        seq_len: int,
        num_layers: int = 1,
        dropout: float = 0.,
        bias: bool = True,
    ):
        super().__init__()

        self.num_nodes = num_nodes
        self.hidden_channels = hidden_channels
        self.num_rels = num_rels
        self.seq_len = seq_len
        self.dropout = dropout

        self.ent = Parameter(torch.Tensor(num_nodes, hidden_channels))
        self.rel = Parameter(torch.Tensor(num_rels, hidden_channels))

        self.sub_gru = GRU(3 * hidden_channels, hidden_channels, num_layers,
                           batch_first=True, bias=bias)
        self.obj_gru = GRU(3 * hidden_channels, hidden_channels, num_layers,
                           batch_first=True, bias=bias)

        self.sub_lin = Linear(3 * hidden_channels, num_nodes, bias=bias)
        self.obj_lin = Linear(3 * hidden_channels, num_nodes, bias=bias)

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.ent, gain=math.sqrt(2.0))
        torch.nn.init.xavier_uniform_(self.rel, gain=math.sqrt(2.0))

        self.sub_gru.reset_parameters()
        self.obj_gru.reset_parameters()
        self.sub_lin.reset_parameters()
        self.obj_lin.reset_parameters()

    @staticmethod
    def pre_transform(seq_len: int) -> Callable:
        r"""Precomputes history objects

        .. math::
            \{ \mathcal{O}^{(t-k-1)}_r(s), \ldots, \mathcal{O}^{(t-1)}_r(s) \}

        of a :class:`torch_geometric.datasets.icews.EventDataset` with
        :math:`k` denoting the sequence length :obj:`seq_len`.
        """
        class PreTransform:
            def __init__(self, seq_len: int):
                self.seq_len = seq_len
                self.inc = 5000
                self.t_last = 0
                self.sub_hist = self.increase_hist_node_size([])
                self.obj_hist = self.increase_hist_node_size([])

            def increase_hist_node_size(self, hist: List[int]) -> List[int]:
                hist_inc = torch.zeros((self.inc, self.seq_len + 1, 0))
                return hist + hist_inc.tolist()

            def get_history(
                self,
                hist: List[int],
                node: int,
                rel: int,
            ) -> Tuple[Tensor, Tensor]:
                hists, ts = [], []
                for s in range(seq_len):
                    h = hist[node][s]
                    hists += h
                    ts.append(torch.full((len(h), ), s, dtype=torch.long))
                node, r = torch.tensor(hists, dtype=torch.long).view(
                    -1, 2).t().contiguous()
                node = node[r == rel]
                t = torch.cat(ts, dim=0)[r == rel]
                return node, t

            def step(self, hist: List[int]) -> List[int]:
                for i in range(len(hist)):
                    hist[i] = hist[i][1:]
                    hist[i].append([])
                return hist

            def __call__(self, data: Data) -> Data:
                sub, rel, obj, t = data.sub, data.rel, data.obj, data.t

                if max(sub, obj) + 1 > len(self.sub_hist):  # pragma: no cover
                    self.sub_hist = self.increase_hist_node_size(self.sub_hist)
                    self.obj_hist = self.increase_hist_node_size(self.obj_hist)

                # Delete last timestamp in history.
                if t > self.t_last:
                    self.sub_hist = self.step(self.sub_hist)
                    self.obj_hist = self.step(self.obj_hist)
                    self.t_last = t

                # Save history in data object.
                data.h_sub, data.h_sub_t = self.get_history(
                    self.sub_hist, sub, rel)
                data.h_obj, data.h_obj_t = self.get_history(
                    self.obj_hist, obj, rel)

                # Add new event to history.
                self.sub_hist[sub][-1].append([obj, rel])
                self.obj_hist[obj][-1].append([sub, rel])

                return data

            def __repr__(self) -> str:  # pragma: no cover
                return f'{self.__class__.__name__}(seq_len={self.seq_len})'

        return PreTransform(seq_len)

    def forward(self, data: Data) -> Tuple[Tensor, Tensor]:
        """Given a :obj:`data` batch, computes the forward pass.

        Args:
            data (torch_geometric.data.Data): The input data, holding subject
                :obj:`sub`, relation :obj:`rel` and object :obj:`obj`
                information with shape :obj:`[batch_size]`.
                In addition, :obj:`data` needs to hold history information for
                subjects, given by a vector of node indices :obj:`h_sub` and
                their relative timestamps :obj:`h_sub_t` and batch assignments
                :obj:`h_sub_batch`.
                The same information must be given for objects (:obj:`h_obj`,
                :obj:`h_obj_t`, :obj:`h_obj_batch`).
        """

        assert 'h_sub_batch' in data and 'h_obj_batch' in data
        batch_size, seq_len = data.sub.size(0), self.seq_len

        h_sub_t = data.h_sub_t + data.h_sub_batch * seq_len
        h_obj_t = data.h_obj_t + data.h_obj_batch * seq_len

        h_sub = scatter(self.ent[data.h_sub], h_sub_t, dim=0,
                        dim_size=batch_size * seq_len,
                        reduce='mean').view(batch_size, seq_len, -1)
        h_obj = scatter(self.ent[data.h_obj], h_obj_t, dim=0,
                        dim_size=batch_size * seq_len,
                        reduce='mean').view(batch_size, seq_len, -1)

        sub = self.ent[data.sub].unsqueeze(1).repeat(1, seq_len, 1)
        rel = self.rel[data.rel].unsqueeze(1).repeat(1, seq_len, 1)
        obj = self.ent[data.obj].unsqueeze(1).repeat(1, seq_len, 1)

        _, h_sub = self.sub_gru(torch.cat([sub, h_sub, rel], dim=-1))
        _, h_obj = self.obj_gru(torch.cat([obj, h_obj, rel], dim=-1))
        h_sub, h_obj = h_sub.squeeze(0), h_obj.squeeze(0)

        h_sub = torch.cat([self.ent[data.sub], h_sub, self.rel[data.rel]],
                          dim=-1)
        h_obj = torch.cat([self.ent[data.obj], h_obj, self.rel[data.rel]],
                          dim=-1)

        h_sub = F.dropout(h_sub, p=self.dropout, training=self.training)
        h_obj = F.dropout(h_obj, p=self.dropout, training=self.training)

        log_prob_obj = F.log_softmax(self.sub_lin(h_sub), dim=1)
        log_prob_sub = F.log_softmax(self.obj_lin(h_obj), dim=1)

        return log_prob_obj, log_prob_sub

    def test(self, logits: Tensor, y: Tensor) -> Tensor:
        """Given ground-truth :obj:`y`, computes Mean Reciprocal Rank (MRR)
        and Hits at 1/3/10."""

        _, perm = logits.sort(dim=1, descending=True)
        mask = (y.view(-1, 1) == perm)

        nnz = mask.nonzero(as_tuple=False)
        mrr = (1 / (nnz[:, -1] + 1).to(torch.float)).mean().item()
        hits1 = mask[:, :1].sum().item() / y.size(0)
        hits3 = mask[:, :3].sum().item() / y.size(0)
        hits10 = mask[:, :10].sum().item() / y.size(0)

        return torch.tensor([mrr, hits1, hits3, hits10])
pytorch/torch_geometric/nn/models/rect.py000066400000000000000000000132351444262304300211360ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear

from torch_geometric.nn import GCNConv
from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import scatter


class RECT_L(torch.nn.Module):
    r"""The RECT model, *i.e.* its supervised RECT-L part, from the
    `"Network Embedding with Completely-imbalanced Labels"
    `_ paper.
    In particular, a GCN model is trained that reconstructs semantic class
    knowledge.

    .. note::

        For an example of using RECT, see `examples/rect.py
        `_.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Intermediate size of each sample.
        normalize (bool, optional): Whether to add self-loops and compute
            symmetric normalization coefficients on-the-fly.
            (default: :obj:`True`)
        dropout (float, optional): The dropout probability.
            (default: :obj:`0.0`)
    """
    def __init__(self, in_channels: int, hidden_channels: int,
                 normalize: bool = True, dropout: float = 0.0):
        super().__init__()
        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.dropout = dropout

        self.conv = GCNConv(in_channels, hidden_channels, normalize=normalize)
        self.lin = Linear(hidden_channels, in_channels)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.conv.reset_parameters()
        self.lin.reset_parameters()
        torch.nn.init.xavier_uniform_(self.lin.weight.data)

    @torch.jit._overload_method
    def forward(self, x, edge_index, edge_weight=None):
        # type: (Tensor, SparseTensor, OptTensor) -> Tensor
        pass

    @torch.jit._overload_method
    def forward(self, x, edge_index, edge_weight=None):
        # type: (Tensor, Tensor, OptTensor) -> Tensor
        pass

    def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -> Tensor:
        """"""
        x = self.conv(x, edge_index, edge_weight)
        x = F.dropout(x, p=self.dropout, training=self.training)
        return self.lin(x)

    @torch.jit._overload_method
    def embed(self, x, edge_index, edge_weight=None):
        # type: (Tensor, SparseTensor, OptTensor) -> Tensor
        pass

    @torch.jit._overload_method
    def embed(self, x, edge_index, edge_weight=None):
        # type: (Tensor, Tensor, OptTensor) -> Tensor
        pass

    def embed(self, x: Tensor, edge_index: Adj,
              edge_weight: OptTensor = None) -> Tensor:
        with torch.no_grad():
            return self.conv(x, edge_index, edge_weight)

    def get_semantic_labels(self, x: Tensor, y: Tensor,
                            mask: Tensor) -> Tensor:
        r"""Replaces the original labels by their class-centers."""
        with torch.no_grad():
            y = y[mask]
            mean = scatter(x[mask], y, dim=0, reduce='mean')
            return mean[y]

    def jittable(self, typing: str) -> torch.nn.Module:  # pragma: no cover
        edge_index_type = typing.split(',')[1].strip()

        class EdgeIndexJittable(torch.nn.Module):
            def __init__(self, child):
                super().__init__()
                self.child = child

            def reset_parameters(self):
                self.child.reset_parameters()

            def forward(self, x: Tensor, edge_index: Tensor,
                        edge_weight: OptTensor = None) -> Tensor:
                return self.child(x, edge_index, edge_weight)

            @torch.jit.export
            def embed(self, x: Tensor, edge_index: Tensor,
                      edge_weight: OptTensor = None) -> Tensor:
                return self.child.embed(x, edge_index, edge_weight)

            @torch.jit.export
            def get_semantic_labels(self, x: Tensor, y: Tensor,
                                    mask: Tensor) -> Tensor:
                return self.child.get_semantic_labels(x, y, mask)

        class SparseTensorJittable(torch.nn.Module):
            def __init__(self, child):
                super().__init__()
                self.child = child

            def reset_parameters(self):
                self.child.reset_parameters()

            def forward(self, x: Tensor, edge_index: SparseTensor,
                        edge_weight: OptTensor = None):
                return self.child(x, edge_index, edge_weight)

            @torch.jit.export
            def embed(self, x: Tensor, edge_index: SparseTensor,
                      edge_weight: OptTensor = None) -> Tensor:
                return self.child.embed(x, edge_index, edge_weight)

            @torch.jit.export
            def get_semantic_labels(self, x: Tensor, y: Tensor,
                                    mask: Tensor) -> Tensor:
                return self.child.get_semantic_labels(x, y, mask)

        if self.conv.jittable is not None:
            self.conv = self.conv.jittable()

        if 'Tensor' == edge_index_type:
            jittable_module = EdgeIndexJittable(self)
        elif 'SparseTensor' == edge_index_type:
            jittable_module = SparseTensorJittable(self)
        else:
            raise ValueError(f"Could not parse types '{typing}'")

        return jittable_module

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.hidden_channels})')
pytorch/torch_geometric/nn/models/rev_gnn.py000066400000000000000000000270521444262304300216410ustar00rootroot00000000000000import copy
from abc import ABC, abstractmethod
from typing import Any, List, Optional, Union

import numpy as np
import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.typing import Adj


class InvertibleFunction(torch.autograd.Function):
    r"""An invertible autograd function. This allows for automatic
    backpropagation in a reversible fashion so that the memory of intermediate
    results can be freed during the forward pass and be constructed on-the-fly
    during the bachward pass.

    Args:
        ctx (torch.autograd.function.InvertibleFunctionBackward):
            A context object that can be used to stash information for backward
            computation.
        fn (torch.nn.Module): The forward function.
        fn_inverse (torch.nn.Module): The inverse function to recompute the
            freed input.
        num_bwd_passes (int): Number of backward passes to retain a link
            with the output. After the last backward pass the output is
            discarded and memory is freed.
        num_inputs (int): The number of inputs to the forward function.
        *args (tuple): Inputs and weights.
    """
    @staticmethod
    def forward(ctx, fn: torch.nn.Module, fn_inverse: torch.nn.Module,
                num_bwd_passes: int, num_inputs: int, *args):
        ctx.fn = fn
        ctx.fn_inverse = fn_inverse
        ctx.weights = args[num_inputs:]
        ctx.num_bwd_passes = num_bwd_passes
        ctx.num_inputs = num_inputs
        inputs = args[:num_inputs]
        ctx.input_requires_grad = []

        with torch.no_grad():  # Make a detached copy which shares the storage:
            x = []
            for element in inputs:
                if isinstance(element, torch.Tensor):
                    x.append(element.detach())
                    ctx.input_requires_grad.append(element.requires_grad)
                else:
                    x.append(element)
                    ctx.input_requires_grad.append(None)
            outputs = ctx.fn(*x)

        if not isinstance(outputs, tuple):
            outputs = (outputs, )

        # Detaches outputs in-place, allows discarding the intermedate result:
        detached_outputs = tuple(element.detach_() for element in outputs)

        # Clear memory of node features:
        if torch_geometric.typing.WITH_PT2:
            inputs[0].untyped_storage().resize_(0)
        else:  # pragma: no cover
            inputs[0].storage().resize_(0)

        # Store these tensor nodes for backward passes:
        ctx.inputs = [inputs] * num_bwd_passes
        ctx.outputs = [detached_outputs] * num_bwd_passes

        return detached_outputs

    @staticmethod
    def backward(ctx, *grad_outputs):
        if len(ctx.outputs) == 0:
            raise RuntimeError(
                f"Trying to perform a backward pass on the "
                f"'InvertibleFunction' for more than '{ctx.num_bwd_passes}' "
                f"times. Try raising 'num_bwd_passes'.")

        inputs = ctx.inputs.pop()
        outputs = ctx.outputs.pop()

        # Recompute input by swapping out the first argument:
        with torch.no_grad():
            inputs_inverted = ctx.fn_inverse(*(outputs + inputs[1:]))
            if len(ctx.outputs) == 0:  # Clear memory from outputs:
                for element in outputs:
                    if torch_geometric.typing.WITH_PT2:
                        element.untyped_storage().resize_(0)
                    else:  # pragma: no cover
                        element.storage().resize_(0)

            if not isinstance(inputs_inverted, tuple):
                inputs_inverted = (inputs_inverted, )

            for elem_orig, elem_inv in zip(inputs, inputs_inverted):
                if torch_geometric.typing.WITH_PT2:
                    elem_orig.untyped_storage().resize_(
                        int(np.prod(elem_orig.size())) *
                        elem_orig.element_size())
                else:  # pragma: no cover
                    elem_orig.storage().resize_(int(np.prod(elem_orig.size())))
                elem_orig.set_(elem_inv)

        # Compute gradients with grad enabled:
        with torch.set_grad_enabled(True):
            detached_inputs = []
            for element in inputs:
                if isinstance(element, torch.Tensor):
                    detached_inputs.append(element.detach())
                else:
                    detached_inputs.append(element)
            detached_inputs = tuple(detached_inputs)
            for x, req_grad in zip(detached_inputs, ctx.input_requires_grad):
                if isinstance(x, torch.Tensor):
                    x.requires_grad = req_grad
            tmp_output = ctx.fn(*detached_inputs)

        if not isinstance(tmp_output, tuple):
            tmp_output = (tmp_output, )

        filtered_detached_inputs = tuple(
            filter(
                lambda x: x.requires_grad
                if isinstance(x, torch.Tensor) else False,
                detached_inputs,
            ))
        gradients = torch.autograd.grad(
            outputs=tmp_output,
            inputs=filtered_detached_inputs + ctx.weights,
            grad_outputs=grad_outputs,
        )

        input_gradients = []
        i = 0
        for rg in ctx.input_requires_grad:
            if rg:
                input_gradients.append(gradients[i])
                i += 1
            else:
                input_gradients.append(None)

        gradients = tuple(input_gradients) + gradients[-len(ctx.weights):]

        return (None, None, None, None) + gradients


class InvertibleModule(torch.nn.Module, ABC):
    r"""An abstract class for implementing invertible modules.

    Args:
        disable (bool, optional): If set to :obj:`True`, will disable the usage
            of :class:`InvertibleFunction` and will execute the module without
            memory savings. (default: :obj:`False`)
        num_bwd_passes (int, optional): Number of backward passes to retain a
            link with the output. After the last backward pass the output is
            discarded and memory is freed. (default: :obj:`1`)
    """
    def __init__(self, disable: bool = False, num_bwd_passes: int = 1):
        super().__init__()
        self.disable = disable
        self.num_bwd_passes = num_bwd_passes

    def forward(self, *args):
        """"""
        return self._fn_apply(args, self._forward, self._inverse)

    def inverse(self, *args):
        return self._fn_apply(args, self._inverse, self._forward)

    @abstractmethod
    def _forward(self):
        pass

    @abstractmethod
    def _inverse(self):
        pass

    def _fn_apply(self, args, fn, fn_inverse):
        if not self.disable:
            out = InvertibleFunction.apply(
                fn,
                fn_inverse,
                self.num_bwd_passes,
                len(args),
                *args,
                *tuple(p for p in self.parameters() if p.requires_grad),
            )
        else:
            out = fn(*args)

        # If the layer only has one input, we unpack the tuple:
        if isinstance(out, tuple) and len(out) == 1:
            return out[0]

        return out


class GroupAddRev(InvertibleModule):
    r"""The Grouped Reversible GNN module from the `"Graph Neural Networks with
    1000 Layers" `_ paper.
    This module enables training of arbitary deep GNNs with a memory complexity
    independent of the number of layers.

    It does so by partitioning input node features :math:`\mathbf{X}` into
    :math:`C` groups across the feature dimension. Then, a grouped reversible
    GNN block :math:`f_{\theta(i)}` operates on a group of inputs and produces
    a group of outputs:

    .. math::

        \mathbf{X}^{\prime}_0 &= \sum_{i=2}^C \mathbf{X}_i

        \mathbf{X}^{\prime}_i &= f_{\theta(i)} ( \mathbf{X}^{\prime}_{i - 1},
        \mathbf{A}) + \mathbf{X}_i

    for all :math:`i \in \{ 1, \ldots, C \}`.

    .. note::

        For an example of using :class:`GroupAddRev`, see `examples/rev_gnn.py
        `_.

    Args:
        conv (torch.nn.Module or torch.nn.ModuleList]): A seed GNN. The input
            and output feature dimensions need to match.
        split_dim (int, optional): The dimension across which to split groups.
            (default: :obj:`-1`)
        num_groups (int, optional): The number of groups :math:`C`.
            (default: :obj:`None`)
        disable (bool, optional): If set to :obj:`True`, will disable the usage
            of :class:`InvertibleFunction` and will execute the module without
            memory savings. (default: :obj:`False`)
        num_bwd_passes (int, optional): Number of backward passes to retain a
            link with the output. After the last backward pass the output is
            discarded and memory is freed. (default: :obj:`1`)
    """
    def __init__(
        self,
        conv: Union[torch.nn.Module, torch.nn.ModuleList],
        split_dim: int = -1,
        num_groups: Optional[int] = None,
        disable: bool = False,
        num_bwd_passes: int = 1,
    ):
        super().__init__(disable, num_bwd_passes)
        self.split_dim = split_dim

        if isinstance(conv, torch.nn.ModuleList):
            self.convs = conv
        else:
            assert num_groups is not None, "Please specific 'num_groups'"
            self.convs = torch.nn.ModuleList([conv])
            for i in range(num_groups - 1):
                conv = copy.deepcopy(self.convs[0])
                if hasattr(conv, 'reset_parameters'):
                    conv.reset_parameters()
                self.convs.append(conv)

        if len(self.convs) < 2:
            raise ValueError(f"The number of groups should not be smaller "
                             f"than '2' (got '{self.num_groups}'))")

    @property
    def num_groups(self) -> int:
        return len(self.convs)

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        for conv in self.convs:
            conv.reset_parameters()

    def _forward(self, x: Tensor, edge_index: Adj, *args):
        channels = x.size(self.split_dim)
        xs = self._chunk(x, channels)
        args = list(zip(*[self._chunk(arg, channels) for arg in args]))
        args = [[]] * self.num_groups if len(args) == 0 else args

        ys = []
        y_in = sum(xs[1:])
        for i in range(self.num_groups):
            y_in = xs[i] + self.convs[i](y_in, edge_index, *args[i])
            ys.append(y_in)
        return torch.cat(ys, dim=self.split_dim)

    def _inverse(self, y: Tensor, edge_index: Adj, *args):
        channels = y.size(self.split_dim)
        ys = self._chunk(y, channels)
        args = list(zip(*[self._chunk(arg, channels) for arg in args]))
        args = [[]] * self.num_groups if len(args) == 0 else args

        xs = []
        for i in range(self.num_groups - 1, -1, -1):
            if i != 0:
                y_in = ys[i - 1]
            else:
                y_in = sum(xs)
            x = ys[i] - self.convs[i](y_in, edge_index, *args[i])
            xs.append(x)

        return torch.cat(xs[::-1], dim=self.split_dim)

    def _chunk(self, x: Any, channels: int) -> List[Any]:
        if not isinstance(x, Tensor):
            return [x] * self.num_groups

        try:
            if x.size(self.split_dim) != channels:
                return [x] * self.num_groups
        except IndexError:
            return [x] * self.num_groups

        return torch.chunk(x, self.num_groups, dim=self.split_dim)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.convs[0]}, '
                f'num_groups={self.num_groups})')
pytorch/torch_geometric/nn/models/schnet.py000066400000000000000000000403271444262304300214670ustar00rootroot00000000000000import os
import os.path as osp
import warnings
from math import pi as PI
from typing import Callable, Dict, Optional, Tuple

import numpy as np
import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Embedding, Linear, ModuleList, Sequential

from torch_geometric.data import Dataset, download_url, extract_zip
from torch_geometric.data.makedirs import makedirs
from torch_geometric.nn import MessagePassing, SumAggregation, radius_graph
from torch_geometric.nn.resolver import aggregation_resolver as aggr_resolver
from torch_geometric.typing import OptTensor

qm9_target_dict: Dict[int, str] = {
    0: 'dipole_moment',
    1: 'isotropic_polarizability',
    2: 'homo',
    3: 'lumo',
    4: 'gap',
    5: 'electronic_spatial_extent',
    6: 'zpve',
    7: 'energy_U0',
    8: 'energy_U',
    9: 'enthalpy_H',
    10: 'free_energy',
    11: 'heat_capacity',
}


class SchNet(torch.nn.Module):
    r"""The continuous-filter convolutional neural network SchNet from the
    `"SchNet: A Continuous-filter Convolutional Neural Network for Modeling
    Quantum Interactions" `_ paper that uses
    the interactions blocks of the form

    .. math::
        \mathbf{x}^{\prime}_i = \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j \odot
        h_{\mathbf{\Theta}} ( \exp(-\gamma(\mathbf{e}_{j,i} - \mathbf{\mu}))),

    here :math:`h_{\mathbf{\Theta}}` denotes an MLP and
    :math:`\mathbf{e}_{j,i}` denotes the interatomic distances between atoms.

    .. note::

        For an example of using a pretrained SchNet variant, see
        `examples/qm9_pretrained_schnet.py
        `_.

    Args:
        hidden_channels (int, optional): Hidden embedding size.
            (default: :obj:`128`)
        num_filters (int, optional): The number of filters to use.
            (default: :obj:`128`)
        num_interactions (int, optional): The number of interaction blocks.
            (default: :obj:`6`)
        num_gaussians (int, optional): The number of gaussians :math:`\mu`.
            (default: :obj:`50`)
        interaction_graph (callable, optional): The function used to compute
            the pairwise interaction graph and interatomic distances. If set to
            :obj:`None`, will construct a graph based on :obj:`cutoff` and
            :obj:`max_num_neighbors` properties.
            If provided, this method takes in :obj:`pos` and :obj:`batch`
            tensors and should return :obj:`(edge_index, edge_weight)` tensors.
            (default :obj:`None`)
        cutoff (float, optional): Cutoff distance for interatomic interactions.
            (default: :obj:`10.0`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            collect for each node within the :attr:`cutoff` distance.
            (default: :obj:`32`)
        readout (str, optional): Whether to apply :obj:`"add"` or :obj:`"mean"`
            global aggregation. (default: :obj:`"add"`)
        dipole (bool, optional): If set to :obj:`True`, will use the magnitude
            of the dipole moment to make the final prediction, *e.g.*, for
            target 0 of :class:`torch_geometric.datasets.QM9`.
            (default: :obj:`False`)
        mean (float, optional): The mean of the property to predict.
            (default: :obj:`None`)
        std (float, optional): The standard deviation of the property to
            predict. (default: :obj:`None`)
        atomref (torch.Tensor, optional): The reference of single-atom
            properties.
            Expects a vector of shape :obj:`(max_atomic_number, )`.
    """

    url = 'http://www.quantum-machine.org/datasets/trained_schnet_models.zip'

    def __init__(
        self,
        hidden_channels: int = 128,
        num_filters: int = 128,
        num_interactions: int = 6,
        num_gaussians: int = 50,
        cutoff: float = 10.0,
        interaction_graph: Optional[Callable] = None,
        max_num_neighbors: int = 32,
        readout: str = 'add',
        dipole: bool = False,
        mean: Optional[float] = None,
        std: Optional[float] = None,
        atomref: OptTensor = None,
    ):
        super().__init__()

        self.hidden_channels = hidden_channels
        self.num_filters = num_filters
        self.num_interactions = num_interactions
        self.num_gaussians = num_gaussians
        self.cutoff = cutoff
        self.dipole = dipole
        self.sum_aggr = SumAggregation()
        self.readout = aggr_resolver('sum' if self.dipole else readout)
        self.mean = mean
        self.std = std
        self.scale = None

        if self.dipole:
            import ase

            atomic_mass = torch.from_numpy(ase.data.atomic_masses)
            self.register_buffer('atomic_mass', atomic_mass)

        # Support z == 0 for padding atoms so that their embedding vectors
        # are zeroed and do not receive any gradients.
        self.embedding = Embedding(100, hidden_channels, padding_idx=0)

        if interaction_graph is not None:
            self.interaction_graph = interaction_graph
        else:
            self.interaction_graph = RadiusInteractionGraph(
                cutoff, max_num_neighbors)

        self.distance_expansion = GaussianSmearing(0.0, cutoff, num_gaussians)

        self.interactions = ModuleList()
        for _ in range(num_interactions):
            block = InteractionBlock(hidden_channels, num_gaussians,
                                     num_filters, cutoff)
            self.interactions.append(block)

        self.lin1 = Linear(hidden_channels, hidden_channels // 2)
        self.act = ShiftedSoftplus()
        self.lin2 = Linear(hidden_channels // 2, 1)

        self.register_buffer('initial_atomref', atomref)
        self.atomref = None
        if atomref is not None:
            self.atomref = Embedding(100, 1)
            self.atomref.weight.data.copy_(atomref)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.embedding.reset_parameters()
        for interaction in self.interactions:
            interaction.reset_parameters()
        torch.nn.init.xavier_uniform_(self.lin1.weight)
        self.lin1.bias.data.fill_(0)
        torch.nn.init.xavier_uniform_(self.lin2.weight)
        self.lin2.bias.data.fill_(0)
        if self.atomref is not None:
            self.atomref.weight.data.copy_(self.initial_atomref)

    @staticmethod
    def from_qm9_pretrained(
        root: str,
        dataset: Dataset,
        target: int,
    ) -> Tuple['SchNet', Dataset, Dataset, Dataset]:  # pragma: no cover
        r"""Returns a pre-trained :class:`SchNet` model on the
        :class:`~torch_geometric.datasets.QM9` dataset, trained on the
        specified target :obj:`target`."""
        import ase
        import schnetpack as spk  # noqa

        assert target >= 0 and target <= 12
        is_dipole = target == 0

        units = [1] * 12
        units[0] = ase.units.Debye
        units[1] = ase.units.Bohr**3
        units[5] = ase.units.Bohr**2

        root = osp.expanduser(osp.normpath(root))
        makedirs(root)
        folder = 'trained_schnet_models'
        if not osp.exists(osp.join(root, folder)):
            path = download_url(SchNet.url, root)
            extract_zip(path, root)
            os.unlink(path)

        name = f'qm9_{qm9_target_dict[target]}'
        path = osp.join(root, 'trained_schnet_models', name, 'split.npz')

        split = np.load(path)
        train_idx = split['train_idx']
        val_idx = split['val_idx']
        test_idx = split['test_idx']

        # Filter the splits to only contain characterized molecules.
        idx = dataset.data.idx
        assoc = idx.new_empty(idx.max().item() + 1)
        assoc[idx] = torch.arange(idx.size(0))

        train_idx = assoc[train_idx[np.isin(train_idx, idx)]]
        val_idx = assoc[val_idx[np.isin(val_idx, idx)]]
        test_idx = assoc[test_idx[np.isin(test_idx, idx)]]

        path = osp.join(root, 'trained_schnet_models', name, 'best_model')

        with warnings.catch_warnings():
            warnings.simplefilter('ignore')
            state = torch.load(path, map_location='cpu')

        net = SchNet(
            hidden_channels=128,
            num_filters=128,
            num_interactions=6,
            num_gaussians=50,
            cutoff=10.0,
            dipole=is_dipole,
            atomref=dataset.atomref(target),
        )

        net.embedding.weight = state.representation.embedding.weight

        for int1, int2 in zip(state.representation.interactions,
                              net.interactions):
            int2.mlp[0].weight = int1.filter_network[0].weight
            int2.mlp[0].bias = int1.filter_network[0].bias
            int2.mlp[2].weight = int1.filter_network[1].weight
            int2.mlp[2].bias = int1.filter_network[1].bias
            int2.lin.weight = int1.dense.weight
            int2.lin.bias = int1.dense.bias

            int2.conv.lin1.weight = int1.cfconv.in2f.weight
            int2.conv.lin2.weight = int1.cfconv.f2out.weight
            int2.conv.lin2.bias = int1.cfconv.f2out.bias

        net.lin1.weight = state.output_modules[0].out_net[1].out_net[0].weight
        net.lin1.bias = state.output_modules[0].out_net[1].out_net[0].bias
        net.lin2.weight = state.output_modules[0].out_net[1].out_net[1].weight
        net.lin2.bias = state.output_modules[0].out_net[1].out_net[1].bias

        mean = state.output_modules[0].atom_pool.average
        net.readout = 'mean' if mean is True else 'add'

        dipole = state.output_modules[0].__class__.__name__ == 'DipoleMoment'
        net.dipole = dipole

        net.mean = state.output_modules[0].standardize.mean.item()
        net.std = state.output_modules[0].standardize.stddev.item()

        if state.output_modules[0].atomref is not None:
            net.atomref.weight = state.output_modules[0].atomref.weight
        else:
            net.atomref = None

        net.scale = 1.0 / units[target]

        return net, (dataset[train_idx], dataset[val_idx], dataset[test_idx])

    def forward(self, z: Tensor, pos: Tensor,
                batch: OptTensor = None) -> Tensor:
        r"""
        Args:
            z (torch.Tensor): Atomic number of each atom with shape
                :obj:`[num_atoms]`.
            pos (torch.Tensor): Coordinates of each atom with shape
                :obj:`[num_atoms, 3]`.
            batch (torch.Tensor, optional): Batch indices assigning each atom
                to a separate molecule with shape :obj:`[num_atoms]`.
                (default: :obj:`None`)
        """
        batch = torch.zeros_like(z) if batch is None else batch

        h = self.embedding(z)
        edge_index, edge_weight = self.interaction_graph(pos, batch)
        edge_attr = self.distance_expansion(edge_weight)

        for interaction in self.interactions:
            h = h + interaction(h, edge_index, edge_weight, edge_attr)

        h = self.lin1(h)
        h = self.act(h)
        h = self.lin2(h)

        if self.dipole:
            # Get center of mass.
            mass = self.atomic_mass[z].view(-1, 1)
            M = self.sum_aggr(mass, batch, dim=0)
            c = self.sum_aggr(mass * pos, batch, dim=0) / M
            h = h * (pos - c.index_select(0, batch))

        if not self.dipole and self.mean is not None and self.std is not None:
            h = h * self.std + self.mean

        if not self.dipole and self.atomref is not None:
            h = h + self.atomref(z)

        out = self.readout(h, batch, dim=0)

        if self.dipole:
            out = torch.norm(out, dim=-1, keepdim=True)

        if self.scale is not None:
            out = self.scale * out

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'hidden_channels={self.hidden_channels}, '
                f'num_filters={self.num_filters}, '
                f'num_interactions={self.num_interactions}, '
                f'num_gaussians={self.num_gaussians}, '
                f'cutoff={self.cutoff})')


class RadiusInteractionGraph(torch.nn.Module):
    r"""Creates edges based on atom positions :obj:`pos` to all points within
    the cutoff distance.

    Args:
        cutoff (float, optional): Cutoff distance for interatomic interactions.
            (default: :obj:`10.0`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            collect for each node within the :attr:`cutoff` distance with the
            default interaction graph method.
            (default: :obj:`32`)
    """
    def __init__(self, cutoff: float = 10.0, max_num_neighbors: int = 32):
        super().__init__()
        self.cutoff = cutoff
        self.max_num_neighbors = max_num_neighbors

    def forward(self, pos: Tensor, batch: Tensor) -> Tuple[Tensor, Tensor]:
        r"""
        Args:
            pos (Tensor): Coordinates of each atom.
            batch (LongTensor, optional): Batch indices assigning each atom to
                a separate molecule.

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        edge_index = radius_graph(pos, r=self.cutoff, batch=batch,
                                  max_num_neighbors=self.max_num_neighbors)
        row, col = edge_index
        edge_weight = (pos[row] - pos[col]).norm(dim=-1)
        return edge_index, edge_weight


class InteractionBlock(torch.nn.Module):
    def __init__(self, hidden_channels: int, num_gaussians: int,
                 num_filters: int, cutoff: float):
        super().__init__()
        self.mlp = Sequential(
            Linear(num_gaussians, num_filters),
            ShiftedSoftplus(),
            Linear(num_filters, num_filters),
        )
        self.conv = CFConv(hidden_channels, hidden_channels, num_filters,
                           self.mlp, cutoff)
        self.act = ShiftedSoftplus()
        self.lin = Linear(hidden_channels, hidden_channels)

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.mlp[0].weight)
        self.mlp[0].bias.data.fill_(0)
        torch.nn.init.xavier_uniform_(self.mlp[2].weight)
        self.mlp[2].bias.data.fill_(0)
        self.conv.reset_parameters()
        torch.nn.init.xavier_uniform_(self.lin.weight)
        self.lin.bias.data.fill_(0)

    def forward(self, x: Tensor, edge_index: Tensor, edge_weight: Tensor,
                edge_attr: Tensor) -> Tensor:
        x = self.conv(x, edge_index, edge_weight, edge_attr)
        x = self.act(x)
        x = self.lin(x)
        return x


class CFConv(MessagePassing):
    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        num_filters: int,
        nn: Sequential,
        cutoff: float,
    ):
        super().__init__(aggr='add')
        self.lin1 = Linear(in_channels, num_filters, bias=False)
        self.lin2 = Linear(num_filters, out_channels)
        self.nn = nn
        self.cutoff = cutoff

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.xavier_uniform_(self.lin1.weight)
        torch.nn.init.xavier_uniform_(self.lin2.weight)
        self.lin2.bias.data.fill_(0)

    def forward(self, x: Tensor, edge_index: Tensor, edge_weight: Tensor,
                edge_attr: Tensor) -> Tensor:
        C = 0.5 * (torch.cos(edge_weight * PI / self.cutoff) + 1.0)
        W = self.nn(edge_attr) * C.view(-1, 1)

        x = self.lin1(x)
        x = self.propagate(edge_index, x=x, W=W)
        x = self.lin2(x)
        return x

    def message(self, x_j: Tensor, W: Tensor) -> Tensor:
        return x_j * W


class GaussianSmearing(torch.nn.Module):
    def __init__(
        self,
        start: float = 0.0,
        stop: float = 5.0,
        num_gaussians: int = 50,
    ):
        super().__init__()
        offset = torch.linspace(start, stop, num_gaussians)
        self.coeff = -0.5 / (offset[1] - offset[0]).item()**2
        self.register_buffer('offset', offset)

    def forward(self, dist: Tensor) -> Tensor:
        dist = dist.view(-1, 1) - self.offset.view(1, -1)
        return torch.exp(self.coeff * torch.pow(dist, 2))


class ShiftedSoftplus(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.shift = torch.log(torch.tensor(2.0)).item()

    def forward(self, x: Tensor) -> Tensor:
        return F.softplus(x) - self.shift
pytorch/torch_geometric/nn/models/signed_gcn.py000066400000000000000000000231601444262304300222770ustar00rootroot00000000000000from typing import Optional, Tuple

import scipy.sparse
import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.nn import SignedConv
from torch_geometric.utils import (
    coalesce,
    negative_sampling,
    structured_negative_sampling,
)


class SignedGCN(torch.nn.Module):
    r"""The signed graph convolutional network model from the `"Signed Graph
    Convolutional Network" `_ paper.
    Internally, this module uses the
    :class:`torch_geometric.nn.conv.SignedConv` operator.

    Args:
        in_channels (int): Size of each input sample.
        hidden_channels (int): Size of each hidden sample.
        num_layers (int): Number of layers.
        lamb (float, optional): Balances the contributions of the overall
            objective. (default: :obj:`5`)
        bias (bool, optional): If set to :obj:`False`, all layers will not
            learn an additive bias. (default: :obj:`True`)
    """
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        num_layers: int,
        lamb: float = 5,
        bias: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.hidden_channels = hidden_channels
        self.num_layers = num_layers
        self.lamb = lamb

        self.conv1 = SignedConv(in_channels, hidden_channels // 2,
                                first_aggr=True)
        self.convs = torch.nn.ModuleList()
        for i in range(num_layers - 1):
            self.convs.append(
                SignedConv(hidden_channels // 2, hidden_channels // 2,
                           first_aggr=False))

        self.lin = torch.nn.Linear(2 * hidden_channels, 3)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.conv1.reset_parameters()
        for conv in self.convs:
            conv.reset_parameters()
        self.lin.reset_parameters()

    def split_edges(
        self,
        edge_index: Tensor,
        test_ratio: float = 0.2,
    ) -> Tuple[Tensor, Tensor]:
        r"""Splits the edges :obj:`edge_index` into train and test edges.

        Args:
            edge_index (LongTensor): The edge indices.
            test_ratio (float, optional): The ratio of test edges.
                (default: :obj:`0.2`)
        """
        mask = torch.ones(edge_index.size(1), dtype=torch.bool)
        mask[torch.randperm(mask.size(0))[:int(test_ratio * mask.size(0))]] = 0

        train_edge_index = edge_index[:, mask]
        test_edge_index = edge_index[:, ~mask]

        return train_edge_index, test_edge_index

    def create_spectral_features(
        self,
        pos_edge_index: Tensor,
        neg_edge_index: Tensor,
        num_nodes: Optional[int] = None,
    ) -> Tensor:
        r"""Creates :obj:`in_channels` spectral node features based on
        positive and negative edges.

        Args:
            pos_edge_index (LongTensor): The positive edge indices.
            neg_edge_index (LongTensor): The negative edge indices.
            num_nodes (int, optional): The number of nodes, *i.e.*
                :obj:`max_val + 1` of :attr:`pos_edge_index` and
                :attr:`neg_edge_index`. (default: :obj:`None`)
        """
        from sklearn.decomposition import TruncatedSVD

        edge_index = torch.cat([pos_edge_index, neg_edge_index], dim=1)
        N = edge_index.max().item() + 1 if num_nodes is None else num_nodes
        edge_index = edge_index.to(torch.device('cpu'))

        pos_val = torch.full((pos_edge_index.size(1), ), 2, dtype=torch.float)
        neg_val = torch.full((neg_edge_index.size(1), ), 0, dtype=torch.float)
        val = torch.cat([pos_val, neg_val], dim=0)

        row, col = edge_index
        edge_index = torch.cat([edge_index, torch.stack([col, row])], dim=1)
        val = torch.cat([val, val], dim=0)

        edge_index, val = coalesce(edge_index, val, num_nodes=N)
        val = val - 1

        # Borrowed from:
        # https://github.com/benedekrozemberczki/SGCN/blob/master/src/utils.py
        edge_index = edge_index.detach().numpy()
        val = val.detach().numpy()
        A = scipy.sparse.coo_matrix((val, edge_index), shape=(N, N))
        svd = TruncatedSVD(n_components=self.in_channels, n_iter=128)
        svd.fit(A)
        x = svd.components_.T
        return torch.from_numpy(x).to(torch.float).to(pos_edge_index.device)

    def forward(
        self,
        x: Tensor,
        pos_edge_index: Tensor,
        neg_edge_index: Tensor,
    ) -> Tensor:
        """Computes node embeddings :obj:`z` based on positive edges
        :obj:`pos_edge_index` and negative edges :obj:`neg_edge_index`.

        Args:
            x (torch.Tensor): The input node features.
            pos_edge_index (torch.Tensor): The positive edge indices.
            neg_edge_index (torch.Tensor): The negative edge indices.
        """
        z = F.relu(self.conv1(x, pos_edge_index, neg_edge_index))
        for conv in self.convs:
            z = F.relu(conv(z, pos_edge_index, neg_edge_index))
        return z

    def discriminate(self, z: Tensor, edge_index: Tensor) -> Tensor:
        """Given node embeddings :obj:`z`, classifies the link relation
        between node pairs :obj:`edge_index` to be either positive,
        negative or non-existent.

        Args:
            x (torch.Tensor): The input node features.
            edge_index (torch.Tensor): The edge indices.
        """
        value = torch.cat([z[edge_index[0]], z[edge_index[1]]], dim=1)
        value = self.lin(value)
        return torch.log_softmax(value, dim=1)

    def nll_loss(
        self,
        z: Tensor,
        pos_edge_index: Tensor,
        neg_edge_index: Tensor,
    ) -> Tensor:
        """Computes the discriminator loss based on node embeddings :obj:`z`,
        and positive edges :obj:`pos_edge_index` and negative nedges
        :obj:`neg_edge_index`.

        Args:
            z (torch.Tensor): The node embeddings.
            pos_edge_index (torch.Tensor): The positive edge indices.
            neg_edge_index (torch.Tensor): The negative edge indices.
        """

        edge_index = torch.cat([pos_edge_index, neg_edge_index], dim=1)
        none_edge_index = negative_sampling(edge_index, z.size(0))

        nll_loss = 0
        nll_loss += F.nll_loss(
            self.discriminate(z, pos_edge_index),
            pos_edge_index.new_full((pos_edge_index.size(1), ), 0))
        nll_loss += F.nll_loss(
            self.discriminate(z, neg_edge_index),
            neg_edge_index.new_full((neg_edge_index.size(1), ), 1))
        nll_loss += F.nll_loss(
            self.discriminate(z, none_edge_index),
            none_edge_index.new_full((none_edge_index.size(1), ), 2))
        return nll_loss / 3.0

    def pos_embedding_loss(
        self,
        z: Tensor,
        pos_edge_index: Tensor,
    ) -> Tensor:
        """Computes the triplet loss between positive node pairs and sampled
        non-node pairs.

        Args:
            z (torch.Tensor): The node embeddings.
            pos_edge_index (torch.Tensor): The positive edge indices.
        """
        i, j, k = structured_negative_sampling(pos_edge_index, z.size(0))

        out = (z[i] - z[j]).pow(2).sum(dim=1) - (z[i] - z[k]).pow(2).sum(dim=1)
        return torch.clamp(out, min=0).mean()

    def neg_embedding_loss(self, z: Tensor, neg_edge_index: Tensor) -> Tensor:
        """Computes the triplet loss between negative node pairs and sampled
        non-node pairs.

        Args:
            z (torch.Tensor): The node embeddings.
            neg_edge_index (torch.Tensor): The negative edge indices.
        """
        i, j, k = structured_negative_sampling(neg_edge_index, z.size(0))

        out = (z[i] - z[k]).pow(2).sum(dim=1) - (z[i] - z[j]).pow(2).sum(dim=1)
        return torch.clamp(out, min=0).mean()

    def loss(
        self,
        z: Tensor,
        pos_edge_index: Tensor,
        neg_edge_index: Tensor,
    ) -> Tensor:
        """Computes the overall objective.

        Args:
            z (torch.Tensor): The node embeddings.
            pos_edge_index (torch.Tensor): The positive edge indices.
            neg_edge_index (torch.Tensor): The negative edge indices.
        """
        nll_loss = self.nll_loss(z, pos_edge_index, neg_edge_index)
        loss_1 = self.pos_embedding_loss(z, pos_edge_index)
        loss_2 = self.neg_embedding_loss(z, neg_edge_index)
        return nll_loss + self.lamb * (loss_1 + loss_2)

    def test(
        self,
        z: Tensor,
        pos_edge_index: Tensor,
        neg_edge_index: Tensor,
    ) -> Tuple[float, float]:
        """Evaluates node embeddings :obj:`z` on positive and negative test
        edges by computing AUC and F1 scores.

        Args:
            z (torch.Tensor): The node embeddings.
            pos_edge_index (torch.Tensor): The positive edge indices.
            neg_edge_index (torch.Tensor): The negative edge indices.
        """
        from sklearn.metrics import f1_score, roc_auc_score

        with torch.no_grad():
            pos_p = self.discriminate(z, pos_edge_index)[:, :2].max(dim=1)[1]
            neg_p = self.discriminate(z, neg_edge_index)[:, :2].max(dim=1)[1]
        pred = (1 - torch.cat([pos_p, neg_p])).cpu()
        y = torch.cat(
            [pred.new_ones((pos_p.size(0))),
             pred.new_zeros(neg_p.size(0))])
        pred, y = pred.numpy(), y.numpy()

        auc = roc_auc_score(y, pred)
        f1 = f1_score(y, pred, average='binary') if pred.sum() > 0 else 0

        return auc, f1

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.hidden_channels}, num_layers={self.num_layers})')
pytorch/torch_geometric/nn/models/tgn.py000066400000000000000000000265061444262304300207760ustar00rootroot00000000000000import copy
from typing import Callable, Dict, Tuple

import torch
from torch import Tensor
from torch.nn import GRUCell, Linear

from torch_geometric.nn.inits import zeros
from torch_geometric.utils import scatter
from torch_geometric.utils.scatter import scatter_argmax

TGNMessageStoreType = Dict[int, Tuple[Tensor, Tensor, Tensor, Tensor]]


class TGNMemory(torch.nn.Module):
    r"""The Temporal Graph Network (TGN) memory model from the
    `"Temporal Graph Networks for Deep Learning on Dynamic Graphs"
    `_ paper.

    .. note::

        For an example of using TGN, see `examples/tgn.py
        `_.

    Args:
        num_nodes (int): The number of nodes to save memories for.
        raw_msg_dim (int): The raw message dimensionality.
        memory_dim (int): The hidden memory dimensionality.
        time_dim (int): The time encoding dimensionality.
        message_module (torch.nn.Module): The message function which
            combines source and destination node memory embeddings, the raw
            message and the time encoding.
        aggregator_module (torch.nn.Module): The message aggregator function
            which aggregates messages to the same destination into a single
            representation.
    """
    def __init__(self, num_nodes: int, raw_msg_dim: int, memory_dim: int,
                 time_dim: int, message_module: Callable,
                 aggregator_module: Callable):
        super().__init__()

        self.num_nodes = num_nodes
        self.raw_msg_dim = raw_msg_dim
        self.memory_dim = memory_dim
        self.time_dim = time_dim

        self.msg_s_module = message_module
        self.msg_d_module = copy.deepcopy(message_module)
        self.aggr_module = aggregator_module
        self.time_enc = TimeEncoder(time_dim)
        self.gru = GRUCell(message_module.out_channels, memory_dim)

        self.register_buffer('memory', torch.empty(num_nodes, memory_dim))
        last_update = torch.empty(self.num_nodes, dtype=torch.long)
        self.register_buffer('last_update', last_update)
        self.register_buffer('_assoc', torch.empty(num_nodes,
                                                   dtype=torch.long))

        self.msg_s_store = {}
        self.msg_d_store = {}

        self.reset_parameters()

    @property
    def device(self) -> torch.device:
        return self.time_enc.lin.weight.device

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        if hasattr(self.msg_s_module, 'reset_parameters'):
            self.msg_s_module.reset_parameters()
        if hasattr(self.msg_d_module, 'reset_parameters'):
            self.msg_d_module.reset_parameters()
        if hasattr(self.aggr_module, 'reset_parameters'):
            self.aggr_module.reset_parameters()
        self.time_enc.reset_parameters()
        self.gru.reset_parameters()
        self.reset_state()

    def reset_state(self):
        """Resets the memory to its initial state."""
        zeros(self.memory)
        zeros(self.last_update)
        self._reset_message_store()

    def detach(self):
        """Detaches the memory from gradient computation."""
        self.memory.detach_()

    def forward(self, n_id: Tensor) -> Tuple[Tensor, Tensor]:
        """Returns, for all nodes :obj:`n_id`, their current memory and their
        last updated timestamp."""
        if self.training:
            memory, last_update = self._get_updated_memory(n_id)
        else:
            memory, last_update = self.memory[n_id], self.last_update[n_id]

        return memory, last_update

    def update_state(self, src: Tensor, dst: Tensor, t: Tensor,
                     raw_msg: Tensor):
        """Updates the memory with newly encountered interactions
        :obj:`(src, dst, t, raw_msg)`."""
        n_id = torch.cat([src, dst]).unique()

        if self.training:
            self._update_memory(n_id)
            self._update_msg_store(src, dst, t, raw_msg, self.msg_s_store)
            self._update_msg_store(dst, src, t, raw_msg, self.msg_d_store)
        else:
            self._update_msg_store(src, dst, t, raw_msg, self.msg_s_store)
            self._update_msg_store(dst, src, t, raw_msg, self.msg_d_store)
            self._update_memory(n_id)

    def _reset_message_store(self):
        i = self.memory.new_empty((0, ), device=self.device, dtype=torch.long)
        msg = self.memory.new_empty((0, self.raw_msg_dim), device=self.device)
        # Message store format: (src, dst, t, msg)
        self.msg_s_store = {j: (i, i, i, msg) for j in range(self.num_nodes)}
        self.msg_d_store = {j: (i, i, i, msg) for j in range(self.num_nodes)}

    def _update_memory(self, n_id: Tensor):
        memory, last_update = self._get_updated_memory(n_id)
        self.memory[n_id] = memory
        self.last_update[n_id] = last_update

    def _get_updated_memory(self, n_id: Tensor) -> Tuple[Tensor, Tensor]:
        self._assoc[n_id] = torch.arange(n_id.size(0), device=n_id.device)

        # Compute messages (src -> dst).
        msg_s, t_s, src_s, dst_s = self._compute_msg(n_id, self.msg_s_store,
                                                     self.msg_s_module)

        # Compute messages (dst -> src).
        msg_d, t_d, src_d, dst_d = self._compute_msg(n_id, self.msg_d_store,
                                                     self.msg_d_module)

        # Aggregate messages.
        idx = torch.cat([src_s, src_d], dim=0)
        msg = torch.cat([msg_s, msg_d], dim=0)
        t = torch.cat([t_s, t_d], dim=0)
        aggr = self.aggr_module(msg, self._assoc[idx], t, n_id.size(0))

        # Get local copy of updated memory.
        memory = self.gru(aggr, self.memory[n_id])

        # Get local copy of updated `last_update`.
        dim_size = self.last_update.size(0)
        last_update = scatter(t, idx, 0, dim_size, reduce='max')[n_id]

        return memory, last_update

    def _update_msg_store(self, src: Tensor, dst: Tensor, t: Tensor,
                          raw_msg: Tensor, msg_store: TGNMessageStoreType):
        n_id, perm = src.sort()
        n_id, count = n_id.unique_consecutive(return_counts=True)
        for i, idx in zip(n_id.tolist(), perm.split(count.tolist())):
            msg_store[i] = (src[idx], dst[idx], t[idx], raw_msg[idx])

    def _compute_msg(self, n_id: Tensor, msg_store: TGNMessageStoreType,
                     msg_module: Callable):
        data = [msg_store[i] for i in n_id.tolist()]
        src, dst, t, raw_msg = list(zip(*data))
        src = torch.cat(src, dim=0)
        dst = torch.cat(dst, dim=0)
        t = torch.cat(t, dim=0)
        raw_msg = torch.cat(raw_msg, dim=0)
        t_rel = t - self.last_update[src]
        t_enc = self.time_enc(t_rel.to(raw_msg.dtype))

        msg = msg_module(self.memory[src], self.memory[dst], raw_msg, t_enc)

        return msg, t, src, dst

    def train(self, mode: bool = True):
        """Sets the module in training mode."""
        if self.training and not mode:
            # Flush message store to memory in case we just entered eval mode.
            self._update_memory(
                torch.arange(self.num_nodes, device=self.memory.device))
            self._reset_message_store()
        super().train(mode)


class IdentityMessage(torch.nn.Module):
    def __init__(self, raw_msg_dim: int, memory_dim: int, time_dim: int):
        super().__init__()
        self.out_channels = raw_msg_dim + 2 * memory_dim + time_dim

    def forward(self, z_src: Tensor, z_dst: Tensor, raw_msg: Tensor,
                t_enc: Tensor):
        return torch.cat([z_src, z_dst, raw_msg, t_enc], dim=-1)


class LastAggregator(torch.nn.Module):
    def forward(self, msg: Tensor, index: Tensor, t: Tensor, dim_size: int):
        argmax = scatter_argmax(t, index, dim=0, dim_size=dim_size)
        out = msg.new_zeros((dim_size, msg.size(-1)))
        mask = argmax < msg.size(0)  # Filter items with at least one entry.
        out[mask] = msg[argmax[mask]]
        return out


class MeanAggregator(torch.nn.Module):
    def forward(self, msg: Tensor, index: Tensor, t: Tensor, dim_size: int):
        return scatter(msg, index, dim=0, dim_size=dim_size, reduce='mean')


class TimeEncoder(torch.nn.Module):
    def __init__(self, out_channels: int):
        super().__init__()
        self.out_channels = out_channels
        self.lin = Linear(1, out_channels)

    def reset_parameters(self):
        self.lin.reset_parameters()

    def forward(self, t: Tensor) -> Tensor:
        return self.lin(t.view(-1, 1)).cos()


class LastNeighborLoader:
    def __init__(self, num_nodes: int, size: int, device=None):
        self.size = size

        self.neighbors = torch.empty((num_nodes, size), dtype=torch.long,
                                     device=device)
        self.e_id = torch.empty((num_nodes, size), dtype=torch.long,
                                device=device)
        self._assoc = torch.empty(num_nodes, dtype=torch.long, device=device)

        self.reset_state()

    def __call__(self, n_id: Tensor) -> Tuple[Tensor, Tensor, Tensor]:
        neighbors = self.neighbors[n_id]
        nodes = n_id.view(-1, 1).repeat(1, self.size)
        e_id = self.e_id[n_id]

        # Filter invalid neighbors (identified by `e_id < 0`).
        mask = e_id >= 0
        neighbors, nodes, e_id = neighbors[mask], nodes[mask], e_id[mask]

        # Relabel node indices.
        n_id = torch.cat([n_id, neighbors]).unique()
        self._assoc[n_id] = torch.arange(n_id.size(0), device=n_id.device)
        neighbors, nodes = self._assoc[neighbors], self._assoc[nodes]

        return n_id, torch.stack([neighbors, nodes]), e_id

    def insert(self, src: Tensor, dst: Tensor):
        # Inserts newly encountered interactions into an ever-growing
        # (undirected) temporal graph.

        # Collect central nodes, their neighbors and the current event ids.
        neighbors = torch.cat([src, dst], dim=0)
        nodes = torch.cat([dst, src], dim=0)
        e_id = torch.arange(self.cur_e_id, self.cur_e_id + src.size(0),
                            device=src.device).repeat(2)
        self.cur_e_id += src.numel()

        # Convert newly encountered interaction ids so that they point to
        # locations of a "dense" format of shape [num_nodes, size].
        nodes, perm = nodes.sort()
        neighbors, e_id = neighbors[perm], e_id[perm]

        n_id = nodes.unique()
        self._assoc[n_id] = torch.arange(n_id.numel(), device=n_id.device)

        dense_id = torch.arange(nodes.size(0), device=nodes.device) % self.size
        dense_id += self._assoc[nodes].mul_(self.size)

        dense_e_id = e_id.new_full((n_id.numel() * self.size, ), -1)
        dense_e_id[dense_id] = e_id
        dense_e_id = dense_e_id.view(-1, self.size)

        dense_neighbors = e_id.new_empty(n_id.numel() * self.size)
        dense_neighbors[dense_id] = neighbors
        dense_neighbors = dense_neighbors.view(-1, self.size)

        # Collect new and old interactions...
        e_id = torch.cat([self.e_id[n_id, :self.size], dense_e_id], dim=-1)
        neighbors = torch.cat(
            [self.neighbors[n_id, :self.size], dense_neighbors], dim=-1)

        # And sort them based on `e_id`.
        e_id, perm = e_id.topk(self.size, dim=-1)
        self.e_id[n_id] = e_id
        self.neighbors[n_id] = torch.gather(neighbors, 1, perm)

    def reset_state(self):
        self.cur_e_id = 0
        self.e_id.fill_(-1)
pytorch/torch_geometric/nn/module_dict.py000066400000000000000000000036451444262304300212120ustar00rootroot00000000000000from typing import Iterable, Mapping, Optional, Tuple, Union

import torch
from torch.nn import Module

Key = Union[str, Tuple[str, ...]]


# `torch.nn.ModuleDict` doesn't allow `.` to be used in key names.
# This `ModuleDict` will support it by converting the `.` to `#` in the
# internal representation and converts it back to `.` in the external
# representation. It also allows passing tuples as keys.
class ModuleDict(torch.nn.ModuleDict):
    def __init__(
        self,
        modules: Optional[Mapping[Union[str, Tuple[str, ...]], Module]] = None,
    ):
        if modules is not None:  # Replace the keys in modules:
            modules = {
                self.to_internal_key(key): module
                for key, module in modules.items()
            }
        super().__init__(modules)

    @staticmethod
    def to_internal_key(key: Key) -> str:
        if isinstance(key, tuple):
            assert len(key) > 1
            key = f"<{'___'.join(key)}>"
        assert isinstance(key, str)
        return key.replace('.', '#')

    @staticmethod
    def to_external_key(key: str) -> Key:
        key = key.replace('#', '.')
        if key.startswith('<') and key.endswith('>') and '___' in key:
            key = tuple(key[1:-1].split('___'))
        return key

    def __getitem__(self, key: Key) -> Module:
        return super().__getitem__(self.to_internal_key(key))

    def __setitem__(self, key: Key, module: Module):
        return super().__setitem__(self.to_internal_key(key), module)

    def __delitem__(self, key: Key):
        return super().__delitem__(self.to_internal_key(key))

    def __contains__(self, key: Key) -> bool:
        return super().__contains__(self.to_internal_key(key))

    def keys(self) -> Iterable[Key]:
        return [self.to_external_key(key) for key in super().keys()]

    def items(self) -> Iterable[Tuple[Key, Module]]:
        return [(self.to_external_key(k), v) for k, v in super().items()]
pytorch/torch_geometric/nn/norm/000077500000000000000000000000001444262304300173135ustar00rootroot00000000000000pytorch/torch_geometric/nn/norm/__init__.py000066400000000000000000000011761444262304300214310ustar00rootroot00000000000000from .batch_norm import BatchNorm, HeteroBatchNorm
from .instance_norm import InstanceNorm
from .layer_norm import LayerNorm, HeteroLayerNorm
from .graph_norm import GraphNorm
from .graph_size_norm import GraphSizeNorm
from .pair_norm import PairNorm
from .mean_subtraction_norm import MeanSubtractionNorm
from .msg_norm import MessageNorm
from .diff_group_norm import DiffGroupNorm

__all__ = [
    'BatchNorm',
    'HeteroBatchNorm',
    'InstanceNorm',
    'LayerNorm',
    'HeteroLayerNorm',
    'GraphNorm',
    'GraphSizeNorm',
    'PairNorm',
    'MeanSubtractionNorm',
    'MessageNorm',
    'DiffGroupNorm',
]

classes = __all__
pytorch/torch_geometric/nn/norm/batch_norm.py000066400000000000000000000201021444262304300217740ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.aggr.fused import FusedAggregation


class BatchNorm(torch.nn.Module):
    r"""Applies batch normalization over a batch of features as described in
    the `"Batch Normalization: Accelerating Deep Network Training by
    Reducing Internal Covariate Shift" `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{\mathbf{x} -
        \textrm{E}[\mathbf{x}]}{\sqrt{\textrm{Var}[\mathbf{x}] + \epsilon}}
        \odot \gamma + \beta

    The mean and standard-deviation are calculated per-dimension over all nodes
    inside the mini-batch.

    Args:
        in_channels (int): Size of each input sample.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        momentum (float, optional): The value used for the running mean and
            running variance computation. (default: :obj:`0.1`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`True`)
        track_running_stats (bool, optional): If set to :obj:`True`, this
            module tracks the running mean and variance, and when set to
            :obj:`False`, this module does not track such statistics and always
            uses batch statistics in both training and eval modes.
            (default: :obj:`True`)
        allow_single_element (bool, optional): If set to :obj:`True`, batches
            with only a single element will work as during in evaluation.
            That is the running mean and variance will be used.
            Requires :obj:`track_running_stats=True`. (default: :obj:`False`)
    """
    def __init__(
        self,
        in_channels: int,
        eps: float = 1e-5,
        momentum: Optional[float] = 0.1,
        affine: bool = True,
        track_running_stats: bool = True,
        allow_single_element: bool = False,
    ):
        super().__init__()

        if allow_single_element and not track_running_stats:
            raise ValueError("'allow_single_element' requires "
                             "'track_running_stats' to be set to `True`")

        self.module = torch.nn.BatchNorm1d(in_channels, eps, momentum, affine,
                                           track_running_stats)
        self.in_channels = in_channels
        self.allow_single_element = allow_single_element

    def reset_running_stats(self):
        r"""Resets all running statistics of the module."""
        self.module.reset_running_stats()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.module.reset_parameters()

    def forward(self, x: Tensor) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
        """
        if self.allow_single_element and x.size(0) <= 1:
            return torch.nn.functional.batch_norm(
                x,
                self.module.running_mean,
                self.module.running_var,
                self.module.weight,
                self.module.bias,
                False,  # bn_training
                0.0,  # momentum
                self.module.eps,
            )
        return self.module(x)

    def __repr__(self):
        return f'{self.__class__.__name__}({self.module.num_features})'


class HeteroBatchNorm(torch.nn.Module):
    r"""Applies batch normalization over a batch of heterogeneous features as
    described in the `"Batch Normalization: Accelerating Deep Network Training
    by Reducing Internal Covariate Shift" `_
    paper.
    Compared to :class:`BatchNorm`, :class:`HeteroBatchNorm` applies
    normalization individually for each node or edge type.

    Args:
        in_channels (int): Size of each input sample.
        num_types (int): The number of types.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        momentum (float, optional): The value used for the running mean and
            running variance computation. (default: :obj:`0.1`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`True`)
        track_running_stats (bool, optional): If set to :obj:`True`, this
            module tracks the running mean and variance, and when set to
            :obj:`False`, this module does not track such statistics and always
            uses batch statistics in both training and eval modes.
            (default: :obj:`True`)
    """
    def __init__(
        self,
        in_channels: int,
        num_types: int,
        eps: float = 1e-5,
        momentum: Optional[float] = 0.1,
        affine: bool = True,
        track_running_stats: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.num_types = num_types
        self.eps = eps
        self.momentum = momentum
        self.affine = affine
        self.track_running_stats = track_running_stats

        if self.affine:
            self.weight = Parameter(torch.Tensor(num_types, in_channels))
            self.bias = Parameter(torch.Tensor(num_types, in_channels))
        else:
            self.register_parameter('weight', None)
            self.register_parameter('bias', None)

        if self.track_running_stats:
            self.register_buffer('running_mean',
                                 torch.Tensor(num_types, in_channels))
            self.register_buffer('running_var',
                                 torch.Tensor(num_types, in_channels))
            self.register_buffer('num_batches_tracked', torch.tensor(0))
        else:
            self.register_buffer('running_mean', None)
            self.register_buffer('running_var', None)
            self.register_buffer('num_batches_tracked', None)

        self.mean_var = FusedAggregation(['mean', 'var'])

        self.reset_parameters()

    def reset_running_stats(self):
        r"""Resets all running statistics of the module."""
        if self.track_running_stats:
            self.running_mean.zero_()
            self.running_var.fill_(1)
            self.num_batches_tracked.zero_()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.reset_running_stats()
        if self.affine:
            torch.nn.init.ones_(self.weight)
            torch.nn.init.zeros_(self.bias)

    def forward(self, x: Tensor, type_vec: Tensor) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The input features.
            type_vec (torch.Tensor): A vector that maps each entry to a type.
        """
        if not self.training and self.track_running_stats:
            mean, var = self.running_mean, self.running_var
        else:
            with torch.no_grad():
                mean, var = self.mean_var(x, type_vec, dim_size=self.num_types)

        if self.training and self.track_running_stats:
            if self.momentum is None:
                self.num_batches_tracked.add_(1)
                exp_avg_factor = 1.0 / float(self.num_batches_tracked)
            else:
                exp_avg_factor = self.momentum

            with torch.no_grad():  # Update running mean and variance:
                type_index = torch.unique(type_vec)

                self.running_mean[type_index] = (
                    (1.0 - exp_avg_factor) * self.running_mean[type_index] +
                    exp_avg_factor * mean[type_index])
                self.running_var[type_index] = (
                    (1.0 - exp_avg_factor) * self.running_var[type_index] +
                    exp_avg_factor * var[type_index])

        out = (x - mean[type_vec]) / var.clamp(self.eps).sqrt()[type_vec]

        if self.affine:
            out = out * self.weight[type_vec] + self.bias[type_vec]

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'num_types={self.num_types})')
pytorch/torch_geometric/nn/norm/diff_group_norm.py000066400000000000000000000111421444262304300230430ustar00rootroot00000000000000import torch
from torch import Tensor
from torch.nn import BatchNorm1d, Linear


class DiffGroupNorm(torch.nn.Module):
    r"""The differentiable group normalization layer from the `"Towards Deeper
    Graph Neural Networks with Differentiable Group Normalization"
    `_ paper, which normalizes node features
    group-wise via a learnable soft cluster assignment

    .. math::

        \mathbf{S} = \text{softmax} (\mathbf{X} \mathbf{W})

    where :math:`\mathbf{W} \in \mathbb{R}^{F \times G}` denotes a trainable
    weight matrix mapping each node into one of :math:`G` clusters.
    Normalization is then performed group-wise via:

    .. math::

        \mathbf{X}^{\prime} = \mathbf{X} + \lambda \sum_{i = 1}^G
        \text{BatchNorm}(\mathbf{S}[:, i] \odot \mathbf{X})

    Args:
        in_channels (int): Size of each input sample :math:`F`.
        groups (int): The number of groups :math:`G`.
        lamda (float, optional): The balancing factor :math:`\lambda` between
            input embeddings and normalized embeddings. (default: :obj:`0.01`)
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        momentum (float, optional): The value used for the running mean and
            running variance computation. (default: :obj:`0.1`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`True`)
        track_running_stats (bool, optional): If set to :obj:`True`, this
            module tracks the running mean and variance, and when set to
            :obj:`False`, this module does not track such statistics and always
            uses batch statistics in both training and eval modes.
            (default: :obj:`True`)
    """
    def __init__(
        self,
        in_channels: int,
        groups: int,
        lamda: float = 0.01,
        eps: float = 1e-5,
        momentum: float = 0.1,
        affine: bool = True,
        track_running_stats: bool = True,
    ):
        super().__init__()

        self.in_channels = in_channels
        self.groups = groups
        self.lamda = lamda

        self.lin = Linear(in_channels, groups, bias=False)
        self.norm = BatchNorm1d(groups * in_channels, eps, momentum, affine,
                                track_running_stats)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin.reset_parameters()
        self.norm.reset_parameters()

    def forward(self, x: Tensor) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
        """
        F, G = self.in_channels, self.groups

        s = self.lin(x).softmax(dim=-1)  # [N, G]
        out = s.unsqueeze(-1) * x.unsqueeze(-2)  # [N, G, F]
        out = self.norm(out.view(-1, G * F)).view(-1, G, F).sum(-2)  # [N, F]

        return x + self.lamda * out

    @staticmethod
    def group_distance_ratio(x: Tensor, y: Tensor, eps: float = 1e-5) -> float:
        r"""Measures the ratio of inter-group distance over intra-group
        distance

        .. math::
            R_{\text{Group}} = \frac{\frac{1}{(C-1)^2} \sum_{i!=j}
            \frac{1}{|\mathbf{X}_i||\mathbf{X}_j|} \sum_{\mathbf{x}_{iv}
            \in \mathbf{X}_i } \sum_{\mathbf{x}_{jv^{\prime}} \in \mathbf{X}_j}
            {\| \mathbf{x}_{iv} - \mathbf{x}_{jv^{\prime}} \|}_2 }{
            \frac{1}{C} \sum_{i} \frac{1}{{|\mathbf{X}_i|}^2}
            \sum_{\mathbf{x}_{iv}, \mathbf{x}_{iv^{\prime}} \in \mathbf{X}_i }
            {\| \mathbf{x}_{iv} - \mathbf{x}_{iv^{\prime}} \|}_2 }

        where :math:`\mathbf{X}_i` denotes the set of all nodes that belong to
        class :math:`i`, and :math:`C` denotes the total number of classes in
        :obj:`y`.
        """
        num_classes = int(y.max()) + 1

        numerator = 0.
        for i in range(num_classes):
            mask = y == i
            dist = torch.cdist(x[mask].unsqueeze(0), x[~mask].unsqueeze(0))
            numerator += (1 / dist.numel()) * float(dist.sum())
        numerator *= 1 / (num_classes - 1)**2

        denominator = 0.
        for i in range(num_classes):
            mask = y == i
            dist = torch.cdist(x[mask].unsqueeze(0), x[mask].unsqueeze(0))
            denominator += (1 / dist.numel()) * float(dist.sum())
        denominator *= 1 / num_classes

        return numerator / (denominator + eps)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'groups={self.groups})')
pytorch/torch_geometric/nn/norm/graph_norm.py000066400000000000000000000052331444262304300220240ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.nn.inits import ones, zeros
from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter


class GraphNorm(torch.nn.Module):
    r"""Applies graph normalization over individual graphs as described in the
    `"GraphNorm: A Principled Approach to Accelerating Graph Neural Network
    Training" `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{\mathbf{x} - \alpha \odot
        \textrm{E}[\mathbf{x}]}
        {\sqrt{\textrm{Var}[\mathbf{x} - \alpha \odot \textrm{E}[\mathbf{x}]]
        + \epsilon}} \odot \gamma + \beta

    where :math:`\alpha` denotes parameters that learn how much information
    to keep in the mean.

    Args:
        in_channels (int): Size of each input sample.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
    """
    def __init__(self, in_channels: int, eps: float = 1e-5):
        super().__init__()

        self.in_channels = in_channels
        self.eps = eps

        self.weight = torch.nn.Parameter(torch.Tensor(in_channels))
        self.bias = torch.nn.Parameter(torch.Tensor(in_channels))
        self.mean_scale = torch.nn.Parameter(torch.Tensor(in_channels))

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        ones(self.weight)
        zeros(self.bias)
        ones(self.mean_scale)

    def forward(self, x: Tensor, batch: OptTensor = None,
                batch_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        if batch is None:
            batch = x.new_zeros(x.size(0), dtype=torch.long)
            batch_size = 1

        if batch_size is None:
            batch_size = int(batch.max()) + 1

        mean = scatter(x, batch, 0, batch_size, reduce='mean')
        out = x - mean.index_select(0, batch) * self.mean_scale
        var = scatter(out.pow(2), batch, 0, batch_size, reduce='mean')
        std = (var + self.eps).sqrt().index_select(0, batch)
        return self.weight * out / std + self.bias

    def __repr__(self):
        return f'{self.__class__.__name__}({self.in_channels})'
pytorch/torch_geometric/nn/norm/graph_size_norm.py000066400000000000000000000027241444262304300230600ustar00rootroot00000000000000from typing import Optional

import torch
import torch.nn as nn
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import degree


class GraphSizeNorm(nn.Module):
    r"""Applies Graph Size Normalization over each individual graph in a batch
    of node features as described in the
    `"Benchmarking Graph Neural Networks" `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{\mathbf{x}_i}{\sqrt{|\mathcal{V}|}}
    """
    def __init__(self):
        super().__init__()

    def forward(self, x: Tensor, batch: OptTensor = None,
                batch_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        if batch is None:
            batch = torch.zeros(x.size(0), dtype=torch.long, device=x.device)
            batch_size = 1

        inv_sqrt_deg = degree(batch, batch_size, dtype=x.dtype).pow(-0.5)
        return x * inv_sqrt_deg.index_select(0, batch).view(-1, 1)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/norm/instance_norm.py000066400000000000000000000110761444262304300225310ustar00rootroot00000000000000from typing import Optional

import torch.nn.functional as F
from torch import Tensor
from torch.nn.modules.instancenorm import _InstanceNorm

from torch_geometric.typing import OptTensor
from torch_geometric.utils import degree, scatter


class InstanceNorm(_InstanceNorm):
    r"""Applies instance normalization over each individual example in a batch
    of node features as described in the `"Instance Normalization: The Missing
    Ingredient for Fast Stylization" `_
    paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{\mathbf{x} -
        \textrm{E}[\mathbf{x}]}{\sqrt{\textrm{Var}[\mathbf{x}] + \epsilon}}
        \odot \gamma + \beta

    The mean and standard-deviation are calculated per-dimension separately for
    each object in a mini-batch.

    Args:
        in_channels (int): Size of each input sample.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        momentum (float, optional): The value used for the running mean and
            running variance computation. (default: :obj:`0.1`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`False`)
        track_running_stats (bool, optional): If set to :obj:`True`, this
            module tracks the running mean and variance, and when set to
            :obj:`False`, this module does not track such statistics and always
            uses instance statistics in both training and eval modes.
            (default: :obj:`False`)
    """
    def __init__(
        self,
        in_channels: int,
        eps: float = 1e-5,
        momentum: float = 0.1,
        affine: bool = False,
        track_running_stats: bool = False,
    ):
        super().__init__(in_channels, eps, momentum, affine,
                         track_running_stats)

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        super().reset_parameters()

    def forward(self, x: Tensor, batch: OptTensor = None,
                batch_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        if batch is None:
            out = F.instance_norm(
                x.t().unsqueeze(0), self.running_mean, self.running_var,
                self.weight, self.bias, self.training
                or not self.track_running_stats, self.momentum, self.eps)
            return out.squeeze(0).t()

        if batch_size is None:
            batch_size = int(batch.max()) + 1

        mean = var = unbiased_var = x  # Dummies.

        if self.training or not self.track_running_stats:
            norm = degree(batch, batch_size, dtype=x.dtype).clamp_(min=1)
            norm = norm.view(-1, 1)
            unbiased_norm = (norm - 1).clamp_(min=1)

            mean = scatter(x, batch, dim=0, dim_size=batch_size,
                           reduce='sum') / norm

            x = x - mean.index_select(0, batch)

            var = scatter(x * x, batch, dim=0, dim_size=batch_size,
                          reduce='sum')
            unbiased_var = var / unbiased_norm
            var = var / norm

            momentum = self.momentum
            if self.running_mean is not None:
                self.running_mean = (
                    1 - momentum) * self.running_mean + momentum * mean.mean(0)
            if self.running_var is not None:
                self.running_var = (
                    1 - momentum
                ) * self.running_var + momentum * unbiased_var.mean(0)
        else:
            if self.running_mean is not None:
                mean = self.running_mean.view(1, -1).expand(batch_size, -1)
            if self.running_var is not None:
                var = self.running_var.view(1, -1).expand(batch_size, -1)

            x = x - mean.index_select(0, batch)

        out = x / (var + self.eps).sqrt().index_select(0, batch)

        if self.weight is not None and self.bias is not None:
            out = out * self.weight.view(1, -1) + self.bias.view(1, -1)

        return out

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num_features})'
pytorch/torch_geometric/nn/norm/layer_norm.py000066400000000000000000000171761444262304300220500ustar00rootroot00000000000000from typing import List, Optional, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.inits import ones, zeros
from torch_geometric.typing import OptTensor
from torch_geometric.utils import degree, scatter


class LayerNorm(torch.nn.Module):
    r"""Applies layer normalization over each individual example in a batch
    of features as described in the `"Layer Normalization"
    `_ paper

    .. math::
        \mathbf{x}^{\prime}_i = \frac{\mathbf{x} -
        \textrm{E}[\mathbf{x}]}{\sqrt{\textrm{Var}[\mathbf{x}] + \epsilon}}
        \odot \gamma + \beta

    The mean and standard-deviation are calculated across all nodes and all
    node channels separately for each object in a mini-batch.

    Args:
        in_channels (int): Size of each input sample.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`True`)
        mode (str, optinal): The normalization mode to use for layer
            normalization (:obj:`"graph"` or :obj:`"node"`). If :obj:`"graph"`
            is used, each graph will be considered as an element to be
            normalized. If `"node"` is used, each node will be considered as
            an element to be normalized. (default: :obj:`"graph"`)
    """
    def __init__(
        self,
        in_channels: int,
        eps: float = 1e-5,
        affine: bool = True,
        mode: str = 'graph',
    ):
        super().__init__()

        self.in_channels = in_channels
        self.eps = eps
        self.affine = affine
        self.mode = mode

        if affine:
            self.weight = Parameter(torch.Tensor(in_channels))
            self.bias = Parameter(torch.Tensor(in_channels))
        else:
            self.register_parameter('weight', None)
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        ones(self.weight)
        zeros(self.bias)

    def forward(self, x: Tensor, batch: OptTensor = None,
                batch_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        if self.mode == 'graph':
            if batch is None:
                x = x - x.mean()
                out = x / (x.std(unbiased=False) + self.eps)

            else:
                if batch_size is None:
                    batch_size = int(batch.max()) + 1

                norm = degree(batch, batch_size, dtype=x.dtype).clamp_(min=1)
                norm = norm.mul_(x.size(-1)).view(-1, 1)

                mean = scatter(x, batch, dim=0, dim_size=batch_size,
                               reduce='sum').sum(dim=-1, keepdim=True) / norm

                x = x - mean.index_select(0, batch)

                var = scatter(x * x, batch, dim=0, dim_size=batch_size,
                              reduce='sum').sum(dim=-1, keepdim=True)
                var = var / norm

                out = x / (var + self.eps).sqrt().index_select(0, batch)

            if self.weight is not None and self.bias is not None:
                out = out * self.weight + self.bias

            return out

        if self.mode == 'node':
            return F.layer_norm(x, (self.in_channels, ), self.weight,
                                self.bias, self.eps)

        raise ValueError(f"Unknow normalization mode: {self.mode}")

    def __repr__(self):
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'affine={self.affine}, mode={self.mode})')


class HeteroLayerNorm(torch.nn.Module):
    r"""Applies layer normalization over each individual example in a batch
    of heterogeneous features as described in the `"Layer Normalization"
    `_ paper.
    Compared to :class:`LayerNorm`, :class:`HeteroLayerNorm` applies
    normalization individually for each node or edge type.

    Args:
        in_channels (int): Size of each input sample.
        num_types (int): The number of types.
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
        affine (bool, optional): If set to :obj:`True`, this module has
            learnable affine parameters :math:`\gamma` and :math:`\beta`.
            (default: :obj:`True`)
        mode (str, optinal): The normalization mode to use for layer
            normalization (:obj:`"node"`). If `"node"` is used, each node will
            be considered as an element to be normalized.
            (default: :obj:`"node"`)
    """
    def __init__(
        self,
        in_channels: int,
        num_types: int,
        eps: float = 1e-5,
        affine: bool = True,
        mode: str = 'node',
    ):
        super().__init__()
        assert mode == 'node'

        self.in_channels = in_channels
        self.num_types = num_types
        self.eps = eps
        self.affine = affine

        if affine:
            self.weight = Parameter(torch.Tensor(num_types, in_channels))
            self.bias = Parameter(torch.Tensor(num_types, in_channels))
        else:
            self.register_parameter('weight', None)
            self.register_parameter('bias', None)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        if self.affine:
            torch.nn.init.ones_(self.weight)
            torch.nn.init.zeros_(self.bias)

    def forward(
        self,
        x: Tensor,
        type_vec: OptTensor = None,
        type_ptr: Optional[Union[Tensor, List[int]]] = None,
    ) -> Tensor:
        r"""
        .. note::
            Either :obj:`type_vec` or :obj:`type_ptr` needs to be specified.
            In general, relying on :obj:`type_ptr` is more efficient in case
            the input tensor is sorted by types.

        Args:
            x (torch.Tensor): The input features.
            type_vec (torch.Tensor, optional): A vector that maps each entry to
                a type. (default: :obj:`None`)
            type_ptr (torch.Tensor or List[int]): A vector denoting the
                boundaries of types. (default: :obj:`None`)
        """
        if type_vec is None and type_ptr is None:
            raise ValueError("Either 'type_vec' or 'type_ptr' must be given")

        out = F.layer_norm(x, (self.in_channels, ), None, None, self.eps)

        if self.affine:
            # TODO Revisit this logic completely as it performs worse than just
            # operating on a dictionary of tensors
            # (especially the `type_vec` code path)
            if type_ptr is not None:
                h = torch.empty_like(out)
                for i, (s, e) in enumerate(zip(type_ptr[:-1], type_ptr[1:])):
                    h[s:e] = out[s:e] * self.weight[i] + self.bias[i]
                out = h
            else:
                out = out * self.weight[type_vec] + self.bias[type_vec]

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'num_types={self.num_types})')
pytorch/torch_geometric/nn/norm/mean_subtraction_norm.py000066400000000000000000000024371444262304300242630ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.utils import scatter


class MeanSubtractionNorm(torch.nn.Module):
    r"""Applies layer normalization by subtracting the mean from the inputs
    as described in the  `"Revisiting 'Over-smoothing' in Deep GCNs"
    `_ paper

    .. math::
        \mathbf{x}_i = \mathbf{x}_i - \frac{1}{|\mathcal{V}|}
        \sum_{j \in \mathcal{V}} \mathbf{x}_j
    """
    def forward(self, x: Tensor, batch: Optional[Tensor] = None,
                dim_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            dim_size (int, optional): The number of examples :math:`B` in case
                :obj:`batch` is given. (default: :obj:`None`)
        """
        if batch is None:
            return x - x.mean(dim=0, keepdim=True)

        mean = scatter(x, batch, dim=0, dim_size=dim_size, reduce='mean')
        return x - mean[batch]

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/norm/msg_norm.py000066400000000000000000000031661444262304300215140ustar00rootroot00000000000000import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Parameter


class MessageNorm(torch.nn.Module):
    r"""Applies message normalization over the aggregated messages as described
    in the `"DeeperGCNs: All You Need to Train Deeper GCNs"
    `_ paper

    .. math::

        \mathbf{x}_i^{\prime} = \mathrm{MLP} \left( \mathbf{x}_{i} + s \cdot
        {\| \mathbf{x}_i \|}_2 \cdot
        \frac{\mathbf{m}_{i}}{{\|\mathbf{m}_i\|}_2} \right)

    Args:
        learn_scale (bool, optional): If set to :obj:`True`, will learn the
            scaling factor :math:`s` of message normalization.
            (default: :obj:`False`)
    """
    def __init__(self, learn_scale: bool = False):
        super().__init__()

        self.scale = Parameter(torch.Tensor([1.0]), requires_grad=learn_scale)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.scale.data.fill_(1.0)

    def forward(self, x: Tensor, msg: Tensor, p: float = 2.0) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            msg (torch.Tensor): The message tensor :math:`\mathbf{M}`.
            p (float, optional): The norm :math:`p` to use for normalization.
                (default: :obj:`2.0`)
        """
        msg = F.normalize(msg, p=p, dim=-1)
        x_norm = x.norm(p=p, dim=-1, keepdim=True)
        return msg * x_norm * self.scale

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}'
                f'(learn_scale={self.scale.requires_grad})')
pytorch/torch_geometric/nn/norm/pair_norm.py000066400000000000000000000053711444262304300216610ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter


class PairNorm(torch.nn.Module):
    r"""Applies pair normalization over node features as described in the
    `"PairNorm: Tackling Oversmoothing in GNNs"
    `_ paper

    .. math::
        \mathbf{x}_i^c &= \mathbf{x}_i - \frac{1}{n}
        \sum_{i=1}^n \mathbf{x}_i \\

        \mathbf{x}_i^{\prime} &= s \cdot
        \frac{\mathbf{x}_i^c}{\sqrt{\frac{1}{n} \sum_{i=1}^n
        {\| \mathbf{x}_i^c \|}^2_2}}

    Args:
        scale (float, optional): Scaling factor :math:`s` of normalization.
            (default, :obj:`1.`)
        scale_individually (bool, optional): If set to :obj:`True`, will
            compute the scaling step as :math:`\mathbf{x}^{\prime}_i = s \cdot
            \frac{\mathbf{x}_i^c}{{\| \mathbf{x}_i^c \|}_2}`.
            (default: :obj:`False`)
        eps (float, optional): A value added to the denominator for numerical
            stability. (default: :obj:`1e-5`)
    """
    def __init__(self, scale: float = 1., scale_individually: bool = False,
                 eps: float = 1e-5):
        super().__init__()

        self.scale = scale
        self.scale_individually = scale_individually
        self.eps = eps

    def forward(self, x: Tensor, batch: OptTensor = None,
                batch_size: Optional[int] = None) -> Tensor:
        r"""
        Args:
            x (torch.Tensor): The source tensor.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        scale = self.scale

        if batch is None:
            x = x - x.mean(dim=0, keepdim=True)

            if not self.scale_individually:
                return scale * x / (self.eps + x.pow(2).sum(-1).mean()).sqrt()
            else:
                return scale * x / (self.eps + x.norm(2, -1, keepdim=True))

        else:
            mean = scatter(x, batch, dim=0, dim_size=batch_size, reduce='mean')
            x = x - mean.index_select(0, batch)

            if not self.scale_individually:
                return scale * x / torch.sqrt(self.eps + scatter(
                    x.pow(2).sum(-1, keepdim=True), batch, dim=0,
                    dim_size=batch_size, reduce='mean').index_select(0, batch))
            else:
                return scale * x / (self.eps + x.norm(2, -1, keepdim=True))

    def __repr__(self):
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/parameter_dict.py000066400000000000000000000036761444262304300217110ustar00rootroot00000000000000from typing import Iterable, Mapping, Optional, Tuple, Union

import torch
from torch.nn import Parameter

Key = Union[str, Tuple[str, ...]]


# `torch.nn.ParameterDict` doesn't allow `.` to be used in key names.
# This `ParameterDict` will support it by converting the `.` to `#` in the
# internal representation and converts it back to `.` in the external
# representation. It also allows passing tuples as keys.
class ParameterDict(torch.nn.ParameterDict):
    def __init__(
        self,
        parameters: Optional[Mapping[Key, Parameter]] = None,
    ):
        # Replace the keys in modules.
        if parameters:
            parameters = {
                self.to_internal_key(key): parameter
                for key, parameter in parameters.items()
            }
        super().__init__(parameters)

    @staticmethod
    def to_internal_key(key: str) -> Key:
        if isinstance(key, tuple):
            assert len(key) > 1
            key = f"<{'___'.join(key)}>"
        assert isinstance(key, str)
        return key.replace('.', '#')

    @staticmethod
    def to_external_key(key: str) -> Key:
        key = key.replace('#', '.')
        if key.startswith('<') and key.endswith('>') and '___' in key:
            key = tuple(key[1:-1].split('___'))
        return key

    def __getitem__(self, key: Key) -> Parameter:
        return super().__getitem__(self.to_internal_key(key))

    def __setitem__(self, key: Key, parameter: Parameter):
        return super().__setitem__(self.to_internal_key(key), parameter)

    def __delitem__(self, key: Key):
        return super().__delitem__(self.to_internal_key(key))

    def __contains__(self, key: Key) -> bool:
        return super().__contains__(self.to_internal_key(key))

    def keys(self) -> Iterable[Key]:
        return [self.to_external_key(key) for key in super().keys()]

    def items(self) -> Iterable[Tuple[Key, Parameter]]:
        return [(self.to_external_key(k), v) for k, v in super().items()]
pytorch/torch_geometric/nn/pool/000077500000000000000000000000001444262304300173115ustar00rootroot00000000000000pytorch/torch_geometric/nn/pool/__init__.py000066400000000000000000000317731444262304300214350ustar00rootroot00000000000000import warnings
from typing import Optional
from torch import Tensor

import torch_geometric.typing
from torch_geometric.typing import OptTensor

from .asap import ASAPooling
from .avg_pool import avg_pool, avg_pool_neighbor_x, avg_pool_x
from .edge_pool import EdgePooling
from .glob import global_add_pool, global_max_pool, global_mean_pool
from .graclus import graclus
from .max_pool import max_pool, max_pool_neighbor_x, max_pool_x
from .mem_pool import MemPooling
from .pan_pool import PANPooling
from .sag_pool import SAGPooling
from .topk_pool import TopKPooling
from .voxel_grid import voxel_grid
from .approx_knn import approx_knn, approx_knn_graph

try:
    import torch_cluster
except ImportError:
    torch_cluster = None


def fps(
    x: Tensor,
    batch: OptTensor = None,
    ratio: float = 0.5,
    random_start: bool = True,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""A sampling algorithm from the `"PointNet++: Deep Hierarchical Feature
    Learning on Point Sets in a Metric Space"
    `_ paper, which iteratively samples the
    most distant point with regard to the rest points.

    .. code-block:: python

        import torch
        from torch_geometric.nn import fps

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch = torch.tensor([0, 0, 0, 0])
        index = fps(x, batch, ratio=0.5)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        batch (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        ratio (float, optional): Sampling ratio. (default: :obj:`0.5`)
        random_start (bool, optional): If set to :obj:`False`, use the first
            node in :math:`\mathbf{X}` as starting node. (default: obj:`True`)
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    if not torch_geometric.typing.WITH_TORCH_CLUSTER_BATCH_SIZE:
        return torch_cluster.fps(x, batch, ratio, random_start)
    return torch_cluster.fps(x, batch, ratio, random_start, batch_size)


def knn(
    x: Tensor,
    y: Tensor,
    k: int,
    batch_x: OptTensor = None,
    batch_y: OptTensor = None,
    cosine: bool = False,
    num_workers: int = 1,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""Finds for each element in :obj:`y` the :obj:`k` nearest points in
    :obj:`x`.

    .. code-block:: python

        import torch
        from torch_geometric.nn import knn

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch_x = torch.tensor([0, 0, 0, 0])
        y = torch.Tensor([[-1, 0], [1, 0]])
        batch_y = torch.tensor([0, 0])
        assign_index = knn(x, y, 2, batch_x, batch_y)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        y (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{M \times F}`.
        k (int): The number of neighbors.
        batch_x (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        batch_y (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^M`, which assigns each
            node to a specific example. (default: :obj:`None`)
        cosine (bool, optional): If :obj:`True`, will use the cosine
            distance instead of euclidean distance to find nearest neighbors.
            (default: :obj:`False`)
        num_workers (int, optional): Number of workers to use for computation.
            Has no effect in case :obj:`batch_x` or :obj:`batch_y` is not
            :obj:`None`, or the input lies on the GPU. (default: :obj:`1`)
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    if not torch_geometric.typing.WITH_TORCH_CLUSTER_BATCH_SIZE:
        return torch_cluster.knn(x, y, k, batch_x, batch_y, cosine,
                                 num_workers)
    return torch_cluster.knn(x, y, k, batch_x, batch_y, cosine, num_workers,
                             batch_size)


def knn_graph(
    x: Tensor,
    k: int,
    batch: OptTensor = None,
    loop: bool = False,
    flow: str = 'source_to_target',
    cosine: bool = False,
    num_workers: int = 1,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""Computes graph edges to the nearest :obj:`k` points.

    .. code-block:: python

        import torch
        from torch_geometric.nn import knn_graph

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch = torch.tensor([0, 0, 0, 0])
        edge_index = knn_graph(x, k=2, batch=batch, loop=False)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        k (int): The number of neighbors.
        batch (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        loop (bool, optional): If :obj:`True`, the graph will contain
            self-loops. (default: :obj:`False`)
        flow (str, optional): The flow direction when using in combination with
            message passing (:obj:`"source_to_target"` or
            :obj:`"target_to_source"`). (default: :obj:`"source_to_target"`)
        cosine (bool, optional): If :obj:`True`, will use the cosine
            distance instead of euclidean distance to find nearest neighbors.
            (default: :obj:`False`)
        num_workers (int, optional): Number of workers to use for computation.
            Has no effect in case :obj:`batch` is not :obj:`None`, or the input
            lies on the GPU. (default: :obj:`1`)
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    if batch is not None and x.device != batch.device:
        warnings.warn("Input tensor 'x' and 'batch' are on different devices "
                      "in 'knn_graph'. Performing blocking device transfer")
        batch = batch.to(x.device)

    if not torch_geometric.typing.WITH_TORCH_CLUSTER_BATCH_SIZE:
        return torch_cluster.knn_graph(x, k, batch, loop, flow, cosine,
                                       num_workers)
    return torch_cluster.knn_graph(x, k, batch, loop, flow, cosine,
                                   num_workers, batch_size)


def radius(
    x: Tensor,
    y: Tensor,
    r: float,
    batch_x: OptTensor = None,
    batch_y: OptTensor = None,
    max_num_neighbors: int = 32,
    num_workers: int = 1,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""Finds for each element in :obj:`y` all points in :obj:`x` within
    distance :obj:`r`.

    .. code-block:: python

        import torch
        from torch_geometric.nn import radius

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch_x = torch.tensor([0, 0, 0, 0])
        y = torch.Tensor([[-1, 0], [1, 0]])
        batch_y = torch.tensor([0, 0])
        assign_index = radius(x, y, 1.5, batch_x, batch_y)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        y (torch.Tensor): Node feature matrix
            :math:`\mathbf{Y} \in \mathbb{R}^{M \times F}`.
        r (float): The radius.
        batch_x (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        batch_y (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^M`, which assigns each
            node to a specific example. (default: :obj:`None`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            return for each element in :obj:`y`. (default: :obj:`32`)
        num_workers (int, optional): Number of workers to use for computation.
            Has no effect in case :obj:`batch_x` or :obj:`batch_y` is not
            :obj:`None`, or the input lies on the GPU. (default: :obj:`1`)
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    if not torch_geometric.typing.WITH_TORCH_CLUSTER_BATCH_SIZE:
        return torch_cluster.radius(x, y, r, batch_x, batch_y,
                                    max_num_neighbors, num_workers)
    return torch_cluster.radius(x, y, r, batch_x, batch_y, max_num_neighbors,
                                num_workers, batch_size)


def radius_graph(
    x: Tensor,
    r: float,
    batch: OptTensor = None,
    loop: bool = False,
    max_num_neighbors: int = 32,
    flow: str = 'source_to_target',
    num_workers: int = 1,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""Computes graph edges to all points within a given distance.

    .. code-block:: python

        import torch
        from torch_geometric.nn import radius_graph

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch = torch.tensor([0, 0, 0, 0])
        edge_index = radius_graph(x, r=1.5, batch=batch, loop=False)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        r (float): The radius.
        batch (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        loop (bool, optional): If :obj:`True`, the graph will contain
            self-loops. (default: :obj:`False`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            return for each element in :obj:`y`. (default: :obj:`32`)
        flow (str, optional): The flow direction when using in combination with
            message passing (:obj:`"source_to_target"` or
            :obj:`"target_to_source"`). (default: :obj:`"source_to_target"`)
        num_workers (int, optional): Number of workers to use for computation.
            Has no effect in case :obj:`batch` is not :obj:`None`, or the input
            lies on the GPU. (default: :obj:`1`)
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    if batch is not None and x.device != batch.device:
        warnings.warn("Input tensor 'x' and 'batch' are on different devices "
                      "in 'radius_graph'. Performing blocking device transfer")
        batch = batch.to(x.device)

    if not torch_geometric.typing.WITH_TORCH_CLUSTER_BATCH_SIZE:
        return torch_cluster.radius_graph(x, r, batch, loop, max_num_neighbors,
                                          flow, num_workers)
    return torch_cluster.radius_graph(x, r, batch, loop, max_num_neighbors,
                                      flow, num_workers, batch_size)


def nearest(
    x: Tensor,
    y: Tensor,
    batch_x: OptTensor = None,
    batch_y: OptTensor = None,
) -> Tensor:
    r"""Finds for each element in :obj:`y` the :obj:`k` nearest point in
    :obj:`x`.

    .. code-block:: python

        import torch
        from torch_geometric.nn import nearest

        x = torch.Tensor([[-1, -1], [-1, 1], [1, -1], [1, 1]])
        batch_x = torch.tensor([0, 0, 0, 0])
        y = torch.Tensor([[-1, 0], [1, 0]])
        batch_y = torch.tensor([0, 0])
        cluster = nearest(x, y, batch_x, batch_y)

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        y (torch.Tensor): Node feature matrix
            :math:`\mathbf{Y} \in \mathbb{R}^{M \times F}`.
        batch_x (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        batch_y (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^M`, which assigns each
            node to a specific example. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    return torch_cluster.nearest(x, y, batch_x, batch_y)


__all__ = [
    'global_add_pool',
    'global_mean_pool',
    'global_max_pool',
    'TopKPooling',
    'SAGPooling',
    'EdgePooling',
    'ASAPooling',
    'PANPooling',
    'MemPooling',
    'max_pool',
    'avg_pool',
    'max_pool_x',
    'max_pool_neighbor_x',
    'avg_pool_x',
    'avg_pool_neighbor_x',
    'graclus',
    'voxel_grid',
    'fps',
    'knn',
    'knn_graph',
    'approx_knn',
    'approx_knn_graph',
    'radius',
    'radius_graph',
    'nearest',
]

classes = __all__
pytorch/torch_geometric/nn/pool/approx_knn.py000066400000000000000000000075771444262304300220620ustar00rootroot00000000000000import torch
from torch import Tensor


def approx_knn(
    x: Tensor,
    y: Tensor,
    k: int,
    batch_x: Tensor = None,
    batch_y: Tensor = None,
) -> Tensor:  # pragma: no cover
    r"""Finds for each element in :obj:`y` the :obj:`k` approximated nearest
    points in :obj:`x`.

    .. note::

        Approximated :math:`k`-nearest neighbor search is performed via the
        `pynndescent `_ library.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        y (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{M \times F}`.
        k (int): The number of neighbors.
        batch_x (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        batch_y (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^M`, which assigns each
            node to a specific example. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """
    from pynndescent import NNDescent

    if batch_x is None:
        batch_x = x.new_zeros(x.size(0), dtype=torch.long)
    if batch_y is None:
        batch_y = y.new_zeros(y.size(0), dtype=torch.long)

    x = x.view(-1, 1) if x.dim() == 1 else x
    y = y.view(-1, 1) if y.dim() == 1 else y

    assert x.dim() == 2 and batch_x.dim() == 1
    assert y.dim() == 2 and batch_y.dim() == 1
    assert x.size(1) == y.size(1)
    assert x.size(0) == batch_x.size(0)
    assert y.size(0) == batch_y.size(0)

    min_xy = min(x.min(), y.min())
    x, y = x - min_xy, y - min_xy

    max_xy = max(x.max(), y.max())
    x, y, = x / max_xy, y / max_xy

    # Concat batch/features to ensure no cross-links between examples exist:
    x = torch.cat([x, 2 * x.size(1) * batch_x.view(-1, 1).to(x.dtype)], dim=-1)
    y = torch.cat([y, 2 * y.size(1) * batch_y.view(-1, 1).to(y.dtype)], dim=-1)

    index = NNDescent(x.detach().cpu().numpy())
    col, dist = index.query(y.detach().cpu().numpy(), k=k)
    dist = torch.from_numpy(dist).view(-1).to(x.device, x.dtype)
    col = torch.from_numpy(col).view(-1).to(x.device, torch.long)
    row = torch.arange(y.size(0), device=x.device, dtype=torch.long)
    row = row.repeat_interleave(k)
    mask = ~torch.isinf(dist)
    row, col = row[mask], col[mask]

    return torch.stack([row, col], dim=0)


def approx_knn_graph(
    x: Tensor,
    k: int,
    batch: Tensor = None,
    loop: bool = False,
    flow: str = 'source_to_target',
) -> Tensor:  # pragma: no cover
    r"""Computes graph edges to the nearest approximated :obj:`k` points.

    .. note::

        Approximated :math:`k`-nearest neighbor search is performed via the
        `pynndescent `_ library.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        k (int): The number of neighbors.
        batch (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        loop (bool, optional): If :obj:`True`, the graph will contain
            self-loops. (default: :obj:`False`)
        flow (str, optional): The flow direction when using in combination with
            message passing (:obj:`"source_to_target"` or
            :obj:`"target_to_source"`). (default: :obj:`"source_to_target"`)

    :rtype: :class:`torch.Tensor`
    """
    assert flow in ['source_to_target', 'target_to_source']
    row, col = approx_knn(x, x, k if loop else k + 1, batch, batch)
    row, col = (col, row) if flow == 'source_to_target' else (row, col)
    if not loop:
        mask = row != col
        row, col = row[mask], col[mask]
    return torch.stack([row, col], dim=0)
pytorch/torch_geometric/nn/pool/asap.py000066400000000000000000000153261444262304300206160ustar00rootroot00000000000000import copy
from typing import Callable, Optional, Tuple, Union

import torch
import torch.nn.functional as F
from torch import Tensor
from torch.nn import Linear

from torch_geometric.nn import LEConv
from torch_geometric.nn.pool.select.topk import topk
from torch_geometric.utils import (
    add_remaining_self_loops,
    remove_self_loops,
    scatter,
    softmax,
    to_edge_index,
    to_torch_coo_tensor,
    to_torch_csr_tensor,
)


class ASAPooling(torch.nn.Module):
    r"""The Adaptive Structure Aware Pooling operator from the
    `"ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical
    Graph Representations" `_ paper.

    Args:
        in_channels (int): Size of each input sample.
        ratio (float or int): Graph pooling ratio, which is used to compute
            :math:`k = \lceil \mathrm{ratio} \cdot N \rceil`, or the value
            of :math:`k` itself, depending on whether the type of :obj:`ratio`
            is :obj:`float` or :obj:`int`. (default: :obj:`0.5`)
        GNN (torch.nn.Module, optional): A graph neural network layer for
            using intra-cluster properties.
            Especially helpful for graphs with higher degree of neighborhood
            (one of :class:`torch_geometric.nn.conv.GraphConv`,
            :class:`torch_geometric.nn.conv.GCNConv` or
            any GNN which supports the :obj:`edge_weight` parameter).
            (default: :obj:`None`)
        dropout (float, optional): Dropout probability of the normalized
            attention coefficients which exposes each node to a stochastically
            sampled neighborhood during training. (default: :obj:`0`)
        negative_slope (float, optional): LeakyReLU angle of the negative
            slope. (default: :obj:`0.2`)
        add_self_loops (bool, optional): If set to :obj:`True`, will add self
            loops to the new graph connectivity. (default: :obj:`False`)
        **kwargs (optional): Additional parameters for initializing the
            graph neural network layer.
    """
    def __init__(self, in_channels: int, ratio: Union[float, int] = 0.5,
                 GNN: Optional[Callable] = None, dropout: float = 0.0,
                 negative_slope: float = 0.2, add_self_loops: bool = False,
                 **kwargs):
        super().__init__()

        self.in_channels = in_channels
        self.ratio = ratio
        self.negative_slope = negative_slope
        self.dropout = dropout
        self.GNN = GNN
        self.add_self_loops = add_self_loops

        self.lin = Linear(in_channels, in_channels)
        self.att = Linear(2 * in_channels, 1)
        self.gnn_score = LEConv(self.in_channels, 1)
        if self.GNN is not None:
            self.gnn_intra_cluster = GNN(self.in_channels, self.in_channels,
                                         **kwargs)
        else:
            self.gnn_intra_cluster = None
        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin.reset_parameters()
        self.att.reset_parameters()
        self.gnn_score.reset_parameters()
        if self.gnn_intra_cluster is not None:
            self.gnn_intra_cluster.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_weight: Optional[Tensor] = None,
        batch: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Tensor, Optional[Tensor], Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): The node feature matrix.
            edge_index (torch.Tensor): The edge indices.
            edge_weight (torch.Tensor, optional): The edge weights.
                (default: :obj:`None`)
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example. (default: :obj:`None`)

        Return types:
            * **x** (*torch.Tensor*): The pooled node embeddings.
            * **edge_index** (*torch.Tensor*): The coarsened edge indices.
            * **edge_weight** (*torch.Tensor, optional*): The coarsened edge
              weights.
            * **batch** (*torch.Tensor*): The coarsened batch vector.
            * **index** (*torch.Tensor*): The top-:math:`k` node indices of
              nodes which are kept after pooling.
        """
        N = x.size(0)

        edge_index, edge_weight = add_remaining_self_loops(
            edge_index, edge_weight, fill_value=1., num_nodes=N)

        if batch is None:
            batch = edge_index.new_zeros(x.size(0))

        x = x.unsqueeze(-1) if x.dim() == 1 else x

        x_pool = x
        if self.gnn_intra_cluster is not None:
            x_pool = self.gnn_intra_cluster(x=x, edge_index=edge_index,
                                            edge_weight=edge_weight)

        x_pool_j = x_pool[edge_index[0]]
        x_q = scatter(x_pool_j, edge_index[1], dim=0, reduce='max')
        x_q = self.lin(x_q)[edge_index[1]]

        score = self.att(torch.cat([x_q, x_pool_j], dim=-1)).view(-1)
        score = F.leaky_relu(score, self.negative_slope)
        score = softmax(score, edge_index[1], num_nodes=N)

        # Sample attention coefficients stochastically.
        score = F.dropout(score, p=self.dropout, training=self.training)

        v_j = x[edge_index[0]] * score.view(-1, 1)
        x = scatter(v_j, edge_index[1], dim=0, reduce='sum')

        # Cluster selection.
        fitness = self.gnn_score(x, edge_index).sigmoid().view(-1)
        perm = topk(fitness, self.ratio, batch)
        x = x[perm] * fitness[perm].view(-1, 1)
        batch = batch[perm]

        # Graph coarsening.
        A = to_torch_csr_tensor(edge_index, edge_weight, size=(N, N))
        S = to_torch_coo_tensor(edge_index, score, size=(N, N))
        S = S.index_select(1, perm).to_sparse_csr()
        A = S.t().to_sparse_csr() @ (A @ S)

        if edge_weight is None:
            edge_index, _ = to_edge_index(A)
        else:
            edge_index, edge_weight = to_edge_index(A)

        if self.add_self_loops:
            edge_index, edge_weight = add_remaining_self_loops(
                edge_index, edge_weight, num_nodes=A.size(0))
        else:
            edge_index, edge_weight = remove_self_loops(
                edge_index, edge_weight)

        return x, edge_index, edge_weight, batch, perm

    @torch.jit.unused
    def jittable(self) -> 'ASAPooling':
        out = copy.deepcopy(self)
        out.gnn_score = out.gnn_score.jittable()
        if out.gnn_intra_cluster is not None:
            out.gnn_intra_cluster = out.gnn_intra_cluster.jittable()
        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'ratio={self.ratio})')
pytorch/torch_geometric/nn/pool/avg_pool.py000066400000000000000000000075761444262304300215100ustar00rootroot00000000000000from typing import Callable, Optional, Tuple

from torch import Tensor

from torch_geometric.data import Batch, Data
from torch_geometric.nn.pool.consecutive import consecutive_cluster
from torch_geometric.nn.pool.pool import pool_batch, pool_edge, pool_pos
from torch_geometric.utils import add_self_loops, scatter


def _avg_pool_x(
    cluster: Tensor,
    x: Tensor,
    size: Optional[int] = None,
) -> Tensor:
    return scatter(x, cluster, dim=0, dim_size=size, reduce='mean')


def avg_pool_x(
    cluster: Tensor,
    x: Tensor,
    batch: Tensor,
    batch_size: Optional[int] = None,
    size: Optional[int] = None,
) -> Tuple[Tensor, Optional[Tensor]]:
    r"""Average pools node features according to the clustering defined in
    :attr:`cluster`.
    See :meth:`torch_geometric.nn.pool.max_pool_x` for more details.

    Args:
        cluster (torch.Tensor): The cluster vector
            :math:`\mathbf{c} \in \{ 0, \ldots, N - 1 \}^N`, which assigns each
            node to a specific cluster.
        x (Tensor): The node feature matrix.
        batch (torch.Tensor): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example.
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)
        size (int, optional): The maximum number of clusters in a single
            example. (default: :obj:`None`)

    :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`) if :attr:`size` is
        :obj:`None`, else :class:`torch.Tensor`
    """
    if size is not None:
        if batch_size is None:
            batch_size = int(batch.max().item()) + 1
        return _avg_pool_x(cluster, x, batch_size * size), None

    cluster, perm = consecutive_cluster(cluster)
    x = _avg_pool_x(cluster, x)
    batch = pool_batch(perm, batch)

    return x, batch


def avg_pool(
    cluster: Tensor,
    data: Data,
    transform: Optional[Callable] = None,
) -> Data:
    r"""Pools and coarsens a graph given by the
    :class:`torch_geometric.data.Data` object according to the clustering
    defined in :attr:`cluster`.
    Final node features are defined by the *average* features of all nodes
    within the same cluster.
    See :meth:`torch_geometric.nn.pool.max_pool` for more details.

    Args:
        cluster (torch.Tensor): The cluster vector
            :math:`\mathbf{c} \in \{ 0, \ldots, N - 1 \}^N`, which assigns each
            node to a specific cluster.
        data (Data): Graph data object.
        transform (callable, optional): A function/transform that takes in the
            coarsened and pooled :obj:`torch_geometric.data.Data` object and
            returns a transformed version. (default: :obj:`None`)

    :rtype: :class:`torch_geometric.data.Data`
    """
    cluster, perm = consecutive_cluster(cluster)

    x = None if data.x is None else _avg_pool_x(cluster, data.x)
    index, attr = pool_edge(cluster, data.edge_index, data.edge_attr)
    batch = None if data.batch is None else pool_batch(perm, data.batch)
    pos = None if data.pos is None else pool_pos(cluster, data.pos)

    data = Batch(batch=batch, x=x, edge_index=index, edge_attr=attr, pos=pos)

    if transform is not None:
        data = transform(data)

    return data


def avg_pool_neighbor_x(
    data: Data,
    flow: Optional[str] = 'source_to_target',
) -> Data:
    r"""Average pools neighboring node features, where each feature in
    :obj:`data.x` is replaced by the average feature values from the central
    node and its neighbors.
    """
    x, edge_index = data.x, data.edge_index

    edge_index, _ = add_self_loops(edge_index, num_nodes=data.num_nodes)

    row, col = edge_index
    row, col = (row, col) if flow == 'source_to_target' else (col, row)

    data.x = scatter(x[row], col, dim=0, dim_size=data.num_nodes,
                     reduce='mean')
    return data
pytorch/torch_geometric/nn/pool/connect/000077500000000000000000000000001444262304300207425ustar00rootroot00000000000000pytorch/torch_geometric/nn/pool/connect/__init__.py000066400000000000000000000001341444262304300230510ustar00rootroot00000000000000from .base import Connect, ConnectOutput

__all__ = [
    'Connect',
    'ConnectOutput',
]
pytorch/torch_geometric/nn/pool/connect/base.py000066400000000000000000000076411444262304300222360ustar00rootroot00000000000000from dataclasses import dataclass
from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.nn.pool.select import SelectOutput


@torch.jit.script
@dataclass(init=False)
class ConnectOutput:
    r"""The output of the :class:`Connect` method, which holds the coarsened
    graph structure, and optional pooled edge features and batch vectors.

    Args:
        edge_index (torch.Tensor): The edge indices of the cooarsened graph.
        edge_attr (torch.Tensor, optional): The pooled edge features of the
            coarsened graph. (default: :obj:`None`)
        batch (torch.Tensor, optional): The pooled batch vector of the
            coarsened graph. (default: :obj:`None`)
    """
    edge_index: Tensor
    edge_attr: Optional[Tensor] = None
    batch: Optional[Tensor] = None

    def __init__(
        self,
        edge_index: Tensor,
        edge_attr: Optional[Tensor] = None,
        batch: Optional[Tensor] = None,
    ):
        if edge_index.dim() != 2:
            raise ValueError(f"Expected 'edge_index' to be two-dimensional "
                             f"(got {edge_index.dim()} dimensions)")

        if edge_index.size(0) != 2:
            raise ValueError(f"Expected 'edge_index' to have size '2' in the "
                             f"first dimension (got '{edge_index.size(0)}')")

        if edge_attr is not None and edge_attr.size(0) != edge_index.size(1):
            raise ValueError(f"Expected 'edge_index' and 'edge_attr' to "
                             f"hold the same number of edges (got "
                             f"{edge_index.size(1)} and {edge_attr.size(0)} "
                             f"edges)")

        self.edge_index = edge_index
        self.edge_attr = edge_attr
        self.batch = batch


class Connect(torch.nn.Module):
    r"""An abstract base class for implementing custom edge connection
    operators as described in the `"Understanding Pooling in Graph Neural
    Networks" `_ paper.

    Specifically, :class:`Connect` determines for each pair of supernodes the
    presence or abscene of an edge based on the existing edges between the
    nodes in the two supernodes.
    The operator also computes pooled edge features and batch vectors
    (if present).
    """
    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        pass

    def forward(
        self,
        select_output: SelectOutput,
        edge_index: Tensor,
        edge_attr: Optional[Tensor] = None,
        batch: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Optional[Tensor]]:
        r"""
        Args:
            select_output (SelectOutput): The output of :class:`Select`.
            edge_index (torch.Tensor): The edge indices.
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific graph. (default: :obj:`None`)
        """
        raise NotImplementedError

    @staticmethod
    def get_pooled_batch(
        select_output: SelectOutput,
        batch: Optional[Tensor],
    ) -> Optional[Tensor]:
        r"""Returns the batch vector of the coarsened graph.

        Args:
            select_output (SelectOutput): The output of :class:`Select`.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each element to a specific example. (default: :obj:`None`)
        """
        if batch is None:
            return batch

        out = torch.arange(select_output.num_clusters, device=batch.device)
        return out.scatter_(0, select_output.cluster_index,
                            batch[select_output.node_index])

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/pool/consecutive.py000066400000000000000000000004211444262304300222070ustar00rootroot00000000000000import torch


def consecutive_cluster(src):
    unique, inv = torch.unique(src, sorted=True, return_inverse=True)
    perm = torch.arange(inv.size(0), dtype=inv.dtype, device=inv.device)
    perm = inv.new_empty(unique.size(0)).scatter_(0, inv, perm)
    return inv, perm
pytorch/torch_geometric/nn/pool/decimation.py000066400000000000000000000031001444262304300217710ustar00rootroot00000000000000from typing import Tuple, Union

import torch
from torch import LongTensor, Tensor


def decimation_indices(
    ptr: LongTensor,
    decimation_factor: Union[int, float],
) -> Tuple[Tensor, LongTensor]:
    """Gets indices which downsample each point cloud by a decimation factor.

    Decimation happens separately for each cloud to prevent emptying smaller
    point clouds. Empty clouds are prevented: clouds will have a least
    one node after decimation.

    Args:
        ptr (LongTensor): The indices of samples in the batch.
        decimation_factor (int or float): The value to divide number of nodes
            with. Should be higher than (or equal to) :obj:`1` for
            downsampling.

    :rtype: (:class:`LongTensor`, :class:`LongTensor`): The indices and
        updated :obj:`ptr` after downsampling.
    """
    if decimation_factor < 1:
        raise ValueError(
            f"The argument `decimation_factor` should be higher than (or "
            f"equal to) 1 for downsampling. (got {decimation_factor})")

    batch_size = ptr.size(0) - 1
    count = ptr[1:] - ptr[:-1]
    decim_count = torch.div(count, decimation_factor, rounding_mode='floor')
    decim_count.clamp_(min=1)  # Prevent empty examples.

    decim_indices = [
        ptr[i] + torch.randperm(count[i], device=ptr.device)[:decim_count[i]]
        for i in range(batch_size)
    ]
    decim_indices = torch.cat(decim_indices, dim=0)

    # Get updated ptr (e.g., for future decimations):
    decim_ptr = torch.cat([decim_count.new_zeros(1), decim_count.cumsum(0)])

    return decim_indices, decim_ptr
pytorch/torch_geometric/nn/pool/edge_pool.py000066400000000000000000000205671444262304300216320ustar00rootroot00000000000000from typing import Callable, List, NamedTuple, Optional, Tuple

import torch
import torch.nn.functional as F
from torch import Tensor

from torch_geometric.utils import coalesce, scatter, softmax


class UnpoolInfo(NamedTuple):
    edge_index: Tensor
    cluster: Tensor
    batch: Tensor
    new_edge_score: Tensor


class EdgePooling(torch.nn.Module):
    r"""The edge pooling operator from the `"Towards Graph Pooling by Edge
    Contraction" `__ and
    `"Edge Contraction Pooling for Graph Neural Networks"
    `__ papers.

    In short, a score is computed for each edge.
    Edges are contracted iteratively according to that score unless one of
    their nodes has already been part of a contracted edge.

    To duplicate the configuration from the `"Towards Graph Pooling by Edge
    Contraction" `__ paper, use
    either :func:`EdgePooling.compute_edge_score_softmax`
    or :func:`EdgePooling.compute_edge_score_tanh`, and set
    :obj:`add_to_edge_score` to :obj:`0.0`.

    To duplicate the configuration from the `"Edge Contraction Pooling for
    Graph Neural Networks" `__ paper,
    set :obj:`dropout` to :obj:`0.2`.

    Args:
        in_channels (int): Size of each input sample.
        edge_score_method (callable, optional): The function to apply
            to compute the edge score from raw edge scores. By default,
            this is the softmax over all incoming edges for each node.
            This function takes in a :obj:`raw_edge_score` tensor of shape
            :obj:`[num_nodes]`, an :obj:`edge_index` tensor and the number of
            nodes :obj:`num_nodes`, and produces a new tensor of the same size
            as :obj:`raw_edge_score` describing normalized edge scores.
            Included functions are
            :func:`EdgePooling.compute_edge_score_softmax`,
            :func:`EdgePooling.compute_edge_score_tanh`, and
            :func:`EdgePooling.compute_edge_score_sigmoid`.
            (default: :func:`EdgePooling.compute_edge_score_softmax`)
        dropout (float, optional): The probability with
            which to drop edge scores during training. (default: :obj:`0.0`)
        add_to_edge_score (float, optional): A value to be added to each
            computed edge score. Adding this greatly helps with unpooling
            stability. (default: :obj:`0.5`)
    """
    def __init__(
        self,
        in_channels: int,
        edge_score_method: Optional[Callable] = None,
        dropout: Optional[float] = 0.0,
        add_to_edge_score: float = 0.5,
    ):
        super().__init__()
        self.in_channels = in_channels
        if edge_score_method is None:
            edge_score_method = self.compute_edge_score_softmax
        self.compute_edge_score = edge_score_method
        self.add_to_edge_score = add_to_edge_score
        self.dropout = dropout

        self.lin = torch.nn.Linear(2 * in_channels, 1)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.lin.reset_parameters()

    @staticmethod
    def compute_edge_score_softmax(
        raw_edge_score: Tensor,
        edge_index: Tensor,
        num_nodes: int,
    ) -> Tensor:
        r"""Normalizes edge scores via softmax application."""
        return softmax(raw_edge_score, edge_index[1], num_nodes=num_nodes)

    @staticmethod
    def compute_edge_score_tanh(
        raw_edge_score: Tensor,
        edge_index: Optional[Tensor] = None,
        num_nodes: Optional[int] = None,
    ) -> Tensor:
        r"""Normalizes edge scores via hyperbolic tangent application."""
        return torch.tanh(raw_edge_score)

    @staticmethod
    def compute_edge_score_sigmoid(
        raw_edge_score: Tensor,
        edge_index: Optional[Tensor] = None,
        num_nodes: Optional[int] = None,
    ) -> Tensor:
        r"""Normalizes edge scores via sigmoid application."""
        return torch.sigmoid(raw_edge_score)

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        batch: Tensor,
    ) -> Tuple[Tensor, Tensor, Tensor, UnpoolInfo]:
        r"""
        Args:
            x (torch.Tensor): The node features.
            edge_index (torch.Tensor): The edge indices.
            batch (torch.Tensor): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example.

        Return types:
            * **x** *(torch.Tensor)* - The pooled node features.
            * **edge_index** *(torch.Tensor)* - The coarsened edge indices.
            * **batch** *(torch.Tensor)* - The coarsened batch vector.
            * **unpool_info** *(UnpoolInfo)* - Information that is
              consumed by :func:`EdgePooling.unpool` for unpooling.
        """
        e = torch.cat([x[edge_index[0]], x[edge_index[1]]], dim=-1)
        e = self.lin(e).view(-1)
        e = F.dropout(e, p=self.dropout, training=self.training)
        e = self.compute_edge_score(e, edge_index, x.size(0))
        e = e + self.add_to_edge_score

        x, edge_index, batch, unpool_info = self._merge_edges(
            x, edge_index, batch, e)

        return x, edge_index, batch, unpool_info

    def _merge_edges(
        self,
        x: Tensor,
        edge_index: Tensor,
        batch: Tensor,
        edge_score: Tensor,
    ) -> Tuple[Tensor, Tensor, Tensor, UnpoolInfo]:

        cluster = torch.empty_like(batch)
        perm: List[int] = torch.argsort(edge_score, descending=True).tolist()

        # Iterate through all edges, selecting it if it is not incident to
        # another already chosen edge.
        mask = torch.ones(x.size(0), dtype=torch.bool)

        i = 0
        new_edge_indices: List[int] = []
        edge_index_cpu = edge_index.cpu()
        for edge_idx in perm:
            source = int(edge_index_cpu[0, edge_idx])
            if not bool(mask[source]):
                continue

            target = int(edge_index_cpu[1, edge_idx])
            if not bool(mask[target]):
                continue

            new_edge_indices.append(edge_idx)

            cluster[source] = i
            mask[source] = False

            if source != target:
                cluster[target] = i
                mask[target] = False

            i += 1

        # The remaining nodes are simply kept:
        j = int(mask.sum())
        cluster[mask] = torch.arange(i, i + j, device=x.device)
        i += j

        # We compute the new features as an addition of the old ones.
        new_x = scatter(x, cluster, dim=0, dim_size=i, reduce='sum')
        new_edge_score = edge_score[new_edge_indices]
        if int(mask.sum()) > 0:
            remaining_score = x.new_ones(
                (new_x.size(0) - len(new_edge_indices), ))
            new_edge_score = torch.cat([new_edge_score, remaining_score])
        new_x = new_x * new_edge_score.view(-1, 1)

        new_edge_index = coalesce(cluster[edge_index], num_nodes=new_x.size(0))
        new_batch = x.new_empty(new_x.size(0), dtype=torch.long)
        new_batch = new_batch.scatter_(0, cluster, batch)

        unpool_info = UnpoolInfo(edge_index=edge_index, cluster=cluster,
                                 batch=batch, new_edge_score=new_edge_score)

        return new_x, new_edge_index, new_batch, unpool_info

    def unpool(
        self,
        x: Tensor,
        unpool_info: UnpoolInfo,
    ) -> Tuple[Tensor, Tensor, Tensor]:
        r"""Unpools a previous edge pooling step.

        For unpooling, :obj:`x` should be of same shape as those produced by
        this layer's :func:`forward` function. Then, it will produce an
        unpooled :obj:`x` in addition to :obj:`edge_index` and :obj:`batch`.

        Args:
            x (torch.Tensor): The node features.
            unpool_info (UnpoolInfo): Information that has been produced by
                :func:`EdgePooling.forward`.

        Return types:
            * **x** *(torch.Tensor)* - The unpooled node features.
            * **edge_index** *(torch.Tensor)* - The new edge indices.
            * **batch** *(torch.Tensor)* - The new batch vector.
        """
        new_x = x / unpool_info.new_edge_score.view(-1, 1)
        new_x = new_x[unpool_info.cluster]
        return new_x, unpool_info.edge_index, unpool_info.batch

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.in_channels})'
pytorch/torch_geometric/nn/pool/glob.py000066400000000000000000000066661444262304300206240ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.utils import scatter


def global_add_pool(x: Tensor, batch: Optional[Tensor],
                    size: Optional[int] = None) -> Tensor:
    r"""Returns batch-wise graph-level-outputs by adding node features
    across the node dimension, so that for a single graph
    :math:`\mathcal{G}_i` its output is computed by

    .. math::
        \mathbf{r}_i = \sum_{n=1}^{N_i} \mathbf{x}_n.

    Functional method of the
    :class:`~torch_geometric.nn.aggr.SumAggregation` module.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times F}`.
        batch (torch.Tensor, optional): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each node to a specific example.
        size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)
    """
    dim = -1 if isinstance(x, Tensor) and x.dim() == 1 else -2

    if batch is None:
        return x.sum(dim=dim, keepdim=x.dim() <= 2)
    return scatter(x, batch, dim=dim, dim_size=size, reduce='sum')


def global_mean_pool(x: Tensor, batch: Optional[Tensor],
                     size: Optional[int] = None) -> Tensor:
    r"""Returns batch-wise graph-level-outputs by averaging node features
    across the node dimension, so that for a single graph
    :math:`\mathcal{G}_i` its output is computed by

    .. math::
        \mathbf{r}_i = \frac{1}{N_i} \sum_{n=1}^{N_i} \mathbf{x}_n.

    Functional method of the
    :class:`~torch_geometric.nn.aggr.MeanAggregation` module.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times F}`.
        batch (torch.Tensor, optional): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each node to a specific example.
        size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)
    """
    dim = -1 if isinstance(x, Tensor) and x.dim() == 1 else -2

    if batch is None:
        return x.mean(dim=dim, keepdim=x.dim() <= 2)
    return scatter(x, batch, dim=dim, dim_size=size, reduce='mean')


def global_max_pool(x: Tensor, batch: Optional[Tensor],
                    size: Optional[int] = None) -> Tensor:
    r"""Returns batch-wise graph-level-outputs by taking the channel-wise
    maximum across the node dimension, so that for a single graph
    :math:`\mathcal{G}_i` its output is computed by

    .. math::
        \mathbf{r}_i = \mathrm{max}_{n=1}^{N_i} \, \mathbf{x}_n.

    Functional method of the
    :class:`~torch_geometric.nn.aggr.MaxAggregation` module.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times F}`.
        batch (torch.Tensor, optional): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each element to a specific example.
        size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)
    """
    dim = -1 if isinstance(x, Tensor) and x.dim() == 1 else -2

    if batch is None:
        return x.max(dim=dim, keepdim=x.dim() <= 2)[0]
    return scatter(x, batch, dim=dim, dim_size=size, reduce='max')
pytorch/torch_geometric/nn/pool/graclus.py000066400000000000000000000024141444262304300213240ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

try:
    from torch_cluster import graclus_cluster
except ImportError:
    graclus_cluster = None


def graclus(edge_index: Tensor, weight: Optional[Tensor] = None,
            num_nodes: Optional[int] = None):
    r"""A greedy clustering algorithm from the `"Weighted Graph Cuts without
    Eigenvectors: A Multilevel Approach" `_ paper of picking an unmarked
    vertex and matching it with one of its unmarked neighbors (that maximizes
    its edge weight).
    The GPU algorithm is adapted from the `"A GPU Algorithm for Greedy Graph
    Matching" `_
    paper.

    Args:
        edge_index (torch.Tensor): The edge indices.
        weight (torch.Tensor, optional): One-dimensional edge weights.
            (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """

    if graclus_cluster is None:
        raise ImportError('`graclus` requires `torch-cluster`.')

    return graclus_cluster(edge_index[0], edge_index[1], weight, num_nodes)
pytorch/torch_geometric/nn/pool/max_pool.py000066400000000000000000000102461444262304300215040ustar00rootroot00000000000000from typing import Callable, Optional, Tuple

from torch import Tensor

from torch_geometric.data import Batch, Data
from torch_geometric.nn.pool.consecutive import consecutive_cluster
from torch_geometric.nn.pool.pool import pool_batch, pool_edge, pool_pos
from torch_geometric.utils import add_self_loops, scatter


def _max_pool_x(
    cluster: Tensor,
    x: Tensor,
    size: Optional[int] = None,
) -> Tensor:
    return scatter(x, cluster, dim=0, dim_size=size, reduce='max')


def max_pool_x(
    cluster: Tensor,
    x: Tensor,
    batch: Tensor,
    batch_size: Optional[int] = None,
    size: Optional[int] = None,
) -> Tuple[Tensor, Optional[Tensor]]:
    r"""Max-Pools node features according to the clustering defined in
    :attr:`cluster`.

    Args:
        cluster (torch.Tensor): The cluster vector
            :math:`\mathbf{c} \in \{ 0, \ldots, N - 1 \}^N`, which assigns each
            node to a specific cluster.
        x (Tensor): The node feature matrix.
        batch (torch.Tensor): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example.
        batch_size (int, optional): The number of examples :math:`B`.
            Automatically calculated if not given. (default: :obj:`None`)
        size (int, optional): The maximum number of clusters in a single
            example. This property is useful to obtain a batch-wise dense
            representation, *e.g.* for applying FC layers, but should only be
            used if the size of the maximum number of clusters per example is
            known in advance. (default: :obj:`None`)

    :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`) if :attr:`size` is
        :obj:`None`, else :class:`torch.Tensor`
    """
    if size is not None:
        if batch_size is None:
            batch_size = int(batch.max().item()) + 1
        return _max_pool_x(cluster, x, batch_size * size), None

    cluster, perm = consecutive_cluster(cluster)
    x = _max_pool_x(cluster, x)
    batch = pool_batch(perm, batch)

    return x, batch


def max_pool(
    cluster: Tensor,
    data: Data,
    transform: Optional[Callable] = None,
) -> Data:
    r"""Pools and coarsens a graph given by the
    :class:`torch_geometric.data.Data` object according to the clustering
    defined in :attr:`cluster`.
    All nodes within the same cluster will be represented as one node.
    Final node features are defined by the *maximum* features of all nodes
    within the same cluster, node positions are averaged and edge indices are
    defined to be the union of the edge indices of all nodes within the same
    cluster.

    Args:
        cluster (torch.Tensor): The cluster vector
            :math:`\mathbf{c} \in \{ 0, \ldots, N - 1 \}^N`, which assigns each
            node to a specific cluster.
        data (Data): Graph data object.
        transform (callable, optional): A function/transform that takes in the
            coarsened and pooled :obj:`torch_geometric.data.Data` object and
            returns a transformed version. (default: :obj:`None`)

    :rtype: :class:`torch_geometric.data.Data`
    """
    cluster, perm = consecutive_cluster(cluster)

    x = None if data.x is None else _max_pool_x(cluster, data.x)
    index, attr = pool_edge(cluster, data.edge_index, data.edge_attr)
    batch = None if data.batch is None else pool_batch(perm, data.batch)
    pos = None if data.pos is None else pool_pos(cluster, data.pos)

    data = Batch(batch=batch, x=x, edge_index=index, edge_attr=attr, pos=pos)

    if transform is not None:
        data = transform(data)

    return data


def max_pool_neighbor_x(
    data: Data,
    flow: Optional[str] = 'source_to_target',
) -> Data:
    r"""Max pools neighboring node features, where each feature in
    :obj:`data.x` is replaced by the feature value with the maximum value from
    the central node and its neighbors.
    """
    x, edge_index = data.x, data.edge_index

    edge_index, _ = add_self_loops(edge_index, num_nodes=data.num_nodes)

    row, col = edge_index
    row, col = (row, col) if flow == 'source_to_target' else (col, row)

    data.x = scatter(x[row], col, dim=0, dim_size=data.num_nodes, reduce='max')
    return data
pytorch/torch_geometric/nn/pool/mem_pool.py000066400000000000000000000123621444262304300214760ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.nn import Conv2d, KLDivLoss, Linear, Parameter

from torch_geometric.utils import to_dense_batch

EPS = 1e-15


class MemPooling(torch.nn.Module):
    r"""Memory based pooling layer from `"Memory-Based Graph Networks"
    `_ paper, which learns a coarsened graph
    representation based on soft cluster assignments

    .. math::
        S_{i,j}^{(h)} &= \frac{
        (1+{\| \mathbf{x}_i-\mathbf{k}^{(h)}_j \|}^2 / \tau)^{
        -\frac{1+\tau}{2}}}{
        \sum_{k=1}^K (1 + {\| \mathbf{x}_i-\mathbf{k}^{(h)}_k \|}^2 / \tau)^{
        -\frac{1+\tau}{2}}}

        \mathbf{S} &= \textrm{softmax}(\textrm{Conv2d}
        (\Vert_{h=1}^H \mathbf{S}^{(h)})) \in \mathbb{R}^{N \times K}

        \mathbf{X}^{\prime} &= \mathbf{S}^{\top} \mathbf{X} \mathbf{W} \in
        \mathbb{R}^{K \times F^{\prime}}

    Where :math:`H` denotes the number of heads, and :math:`K` denotes the
    number of clusters.

    Args:
        in_channels (int): Size of each input sample :math:`F`.
        out_channels (int): Size of each output sample :math:`F^{\prime}`.
        heads (int): The number of heads :math:`H`.
        num_clusters (int): number of clusters :math:`K` per head.
        tau (int, optional): The temperature :math:`\tau`. (default: :obj:`1.`)
    """
    def __init__(self, in_channels: int, out_channels: int, heads: int,
                 num_clusters: int, tau: float = 1.):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.num_clusters = num_clusters
        self.tau = tau

        self.k = Parameter(torch.Tensor(heads, num_clusters, in_channels))
        self.conv = Conv2d(heads, 1, kernel_size=1, padding=0, bias=False)
        self.lin = Linear(in_channels, out_channels, bias=False)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        torch.nn.init.uniform_(self.k.data, -1., 1.)
        self.conv.reset_parameters()
        self.lin.reset_parameters()

    @staticmethod
    def kl_loss(S: Tensor) -> Tensor:
        r"""The additional KL divergence-based loss

        .. math::
            P_{i,j} &= \frac{S_{i,j}^2 / \sum_{n=1}^N S_{n,j}}{\sum_{k=1}^K
            S_{i,k}^2 / \sum_{n=1}^N S_{n,k}}

            \mathcal{L}_{\textrm{KL}} &= \textrm{KLDiv}(\mathbf{P} \Vert
            \mathbf{S})
        """
        S_2 = S**2
        P = S_2 / S.sum(dim=1, keepdim=True)
        denom = P.sum(dim=2, keepdim=True)
        denom[S.sum(dim=2, keepdim=True) == 0.0] = 1.0
        P /= denom

        loss = KLDivLoss(reduction='batchmean', log_target=False)
        return loss(S.clamp(EPS).log(), P.clamp(EPS))

    def forward(
        self,
        x: Tensor,
        batch: Optional[Tensor] = None,
        mask: Optional[Tensor] = None,
        max_num_nodes: Optional[int] = None,
        batch_size: Optional[int] = None,
    ) -> Tuple[Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): The node feature tensor of shape
                :math:`\mathbf{X} \in \mathbb{R}^{N \times F}` or
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example.
                Should not be provided in case node features already have shape
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`.
                (default: :obj:`None`)
            mask (torch.Tensor, optional): A mask matrix
                :math:`\mathbf{M} \in {\{ 0, 1 \}}^{B \times N}`, which
                indicates valid nodes for each graph when using
                node features of shape
                :math:`\mathbf{X} \in \mathbb{R}^{B \times N \times F}`.
                (default: :obj:`None`)
            max_num_nodes (int, optional): The size of the :math:`B` node
                dimension. Automatically calculated if not given.
                (default: :obj:`None`)
            batch_size (int, optional): The number of examples :math:`B`.
                Automatically calculated if not given. (default: :obj:`None`)
        """
        if x.dim() <= 2:
            x, mask = to_dense_batch(x, batch, max_num_nodes=max_num_nodes,
                                     batch_size=batch_size)
        elif mask is None:
            mask = x.new_ones((x.size(0), x.size(1)), dtype=torch.bool)

        (B, N, _), H, K = x.size(), self.heads, self.num_clusters

        dist = torch.cdist(self.k.view(H * K, -1), x.view(B * N, -1), p=2)**2
        dist = (1. + dist / self.tau).pow(-(self.tau + 1.0) / 2.0)

        dist = dist.view(H, K, B, N).permute(2, 0, 3, 1)  # [B, H, N, K]
        S = dist / dist.sum(dim=-1, keepdim=True)

        S = self.conv(S).squeeze(dim=1).softmax(dim=-1)  # [B, N, K]
        S = S * mask.view(B, N, 1)

        x = self.lin(S.transpose(1, 2) @ x)

        return x, S

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.in_channels}, '
                f'{self.out_channels}, heads={self.heads}, '
                f'num_clusters={self.num_clusters})')
pytorch/torch_geometric/nn/pool/pan_pool.py000066400000000000000000000104601444262304300214730ustar00rootroot00000000000000from typing import Callable, Optional, Tuple, Union

import torch
from torch import Tensor
from torch.nn import Parameter

from torch_geometric.nn.pool.select.topk import topk
from torch_geometric.nn.pool.topk_pool import filter_adj
from torch_geometric.typing import OptTensor, SparseTensor
from torch_geometric.utils import scatter, softmax


class PANPooling(torch.nn.Module):
    r"""The path integral based pooling operator from the
    `"Path Integral Based Convolution and Pooling for Graph Neural Networks"
    `_ paper.
    PAN pooling performs top-:math:`k` pooling where global node importance is
    measured based on node features and the MET matrix:

    .. math::
        {\rm score} = \beta_1 \mathbf{X} \cdot \mathbf{p} + \beta_2
        {\rm deg}(\mathbf{M})

    Args:
        in_channels (int): Size of each input sample.
        ratio (float): Graph pooling ratio, which is used to compute
            :math:`k = \lceil \mathrm{ratio} \cdot N \rceil`.
            This value is ignored if min_score is not None.
            (default: :obj:`0.5`)
        min_score (float, optional): Minimal node score :math:`\tilde{\alpha}`
            which is used to compute indices of pooled nodes
            :math:`\mathbf{i} = \mathbf{y}_i > \tilde{\alpha}`.
            When this value is not :obj:`None`, the :obj:`ratio` argument is
            ignored. (default: :obj:`None`)
        multiplier (float, optional): Coefficient by which features gets
            multiplied after pooling. This can be useful for large graphs and
            when :obj:`min_score` is used. (default: :obj:`1.0`)
        nonlinearity (str or callable, optional): The non-linearity to use.
            (default: :obj:`"tanh"`)
    """
    def __init__(
        self,
        in_channels: int,
        ratio: float = 0.5,
        min_score: Optional[float] = None,
        multiplier: float = 1.0,
        nonlinearity: Union[str, Callable] = 'tanh',
    ):
        super().__init__()

        if isinstance(nonlinearity, str):
            nonlinearity = getattr(torch, nonlinearity)

        self.in_channels = in_channels
        self.ratio = ratio
        self.min_score = min_score
        self.multiplier = multiplier
        self.nonlinearity = nonlinearity

        self.p = Parameter(torch.Tensor(in_channels))
        self.beta = Parameter(torch.Tensor(2))

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.p.data.fill_(1)
        self.beta.data.fill_(0.5)

    def forward(
        self,
        x: Tensor,
        M: SparseTensor,
        batch: OptTensor = None,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): The node feature matrix.
            M (SparseTensor): The MET matrix :math:`\mathbf{M}`.
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example. (default: :obj:`None`)
        """
        if batch is None:
            batch = x.new_zeros(x.size(0), dtype=torch.long)

        row, col, edge_weight = M.coo()
        assert edge_weight is not None

        score1 = (x * self.p).sum(dim=-1)
        score2 = scatter(edge_weight, col, 0, dim_size=x.size(0), reduce='sum')
        score = self.beta[0] * score1 + self.beta[1] * score2

        if self.min_score is None:
            score = self.nonlinearity(score)
        else:
            score = softmax(score, batch)

        perm = topk(score, self.ratio, batch, self.min_score)
        x = x[perm] * score[perm].view(-1, 1)
        x = self.multiplier * x if self.multiplier != 1 else x

        edge_index = torch.stack([col, row], dim=0)
        edge_index, edge_weight = filter_adj(edge_index, edge_weight, perm,
                                             num_nodes=score.size(0))
        assert edge_weight is not None

        return x, edge_index, edge_weight, batch[perm], perm, score[perm]

    def __repr__(self) -> str:
        if self.min_score is None:
            ratio = f'ratio={self.ratio}'
        else:
            ratio = f'min_score={self.min_score}'

        return (f'{self.__class__.__name__}({self.in_channels}, {ratio}, '
                f'multiplier={self.multiplier})')
pytorch/torch_geometric/nn/pool/pool.py000066400000000000000000000011671444262304300206410ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.utils import coalesce, remove_self_loops, scatter


def pool_edge(cluster, edge_index, edge_attr: Optional[torch.Tensor] = None):
    num_nodes = cluster.size(0)
    edge_index = cluster[edge_index.view(-1)].view(2, -1)
    edge_index, edge_attr = remove_self_loops(edge_index, edge_attr)
    if edge_index.numel() > 0:
        edge_index, edge_attr = coalesce(edge_index, edge_attr, num_nodes)
    return edge_index, edge_attr


def pool_batch(perm, batch):
    return batch[perm]


def pool_pos(cluster, pos):
    return scatter(pos, cluster, dim=0, reduce='mean')
pytorch/torch_geometric/nn/pool/sag_pool.py000066400000000000000000000135161444262304300214740ustar00rootroot00000000000000from typing import Callable, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.nn import GraphConv
from torch_geometric.nn.pool.select.topk import topk
from torch_geometric.nn.pool.topk_pool import filter_adj
from torch_geometric.typing import OptTensor
from torch_geometric.utils import softmax


class SAGPooling(torch.nn.Module):
    r"""The self-attention pooling operator from the `"Self-Attention Graph
    Pooling" `_ and `"Understanding
    Attention and Generalization in Graph Neural Networks"
    `_ papers.

    If :obj:`min_score` :math:`\tilde{\alpha}` is :obj:`None`, computes:

        .. math::
            \mathbf{y} &= \textrm{GNN}(\mathbf{X}, \mathbf{A})

            \mathbf{i} &= \mathrm{top}_k(\mathbf{y})

            \mathbf{X}^{\prime} &= (\mathbf{X} \odot
            \mathrm{tanh}(\mathbf{y}))_{\mathbf{i}}

            \mathbf{A}^{\prime} &= \mathbf{A}_{\mathbf{i},\mathbf{i}}

    If :obj:`min_score` :math:`\tilde{\alpha}` is a value in :obj:`[0, 1]`,
    computes:

        .. math::
            \mathbf{y} &= \mathrm{softmax}(\textrm{GNN}(\mathbf{X},\mathbf{A}))

            \mathbf{i} &= \mathbf{y}_i > \tilde{\alpha}

            \mathbf{X}^{\prime} &= (\mathbf{X} \odot \mathbf{y})_{\mathbf{i}}

            \mathbf{A}^{\prime} &= \mathbf{A}_{\mathbf{i},\mathbf{i}}.

    Projections scores are learned based on a graph neural network layer.

    Args:
        in_channels (int): Size of each input sample.
        ratio (float or int): Graph pooling ratio, which is used to compute
            :math:`k = \lceil \mathrm{ratio} \cdot N \rceil`, or the value
            of :math:`k` itself, depending on whether the type of :obj:`ratio`
            is :obj:`float` or :obj:`int`.
            This value is ignored if :obj:`min_score` is not :obj:`None`.
            (default: :obj:`0.5`)
        GNN (torch.nn.Module, optional): A graph neural network layer for
            calculating projection scores (one of
            :class:`torch_geometric.nn.conv.GraphConv`,
            :class:`torch_geometric.nn.conv.GCNConv`,
            :class:`torch_geometric.nn.conv.GATConv` or
            :class:`torch_geometric.nn.conv.SAGEConv`). (default:
            :class:`torch_geometric.nn.conv.GraphConv`)
        min_score (float, optional): Minimal node score :math:`\tilde{\alpha}`
            which is used to compute indices of pooled nodes
            :math:`\mathbf{i} = \mathbf{y}_i > \tilde{\alpha}`.
            When this value is not :obj:`None`, the :obj:`ratio` argument is
            ignored. (default: :obj:`None`)
        multiplier (float, optional): Coefficient by which features gets
            multiplied after pooling. This can be useful for large graphs and
            when :obj:`min_score` is used. (default: :obj:`1`)
        nonlinearity (str or callable, optional): The non-linearity to use.
            (default: :obj:`"tanh"`)
        **kwargs (optional): Additional parameters for initializing the graph
            neural network layer.
    """
    def __init__(
        self,
        in_channels: int,
        ratio: Union[float, int] = 0.5,
        GNN: torch.nn.Module = GraphConv,
        min_score: Optional[float] = None,
        multiplier: float = 1.0,
        nonlinearity: Union[str, Callable] = 'tanh',
        **kwargs,
    ):
        super().__init__()

        if isinstance(nonlinearity, str):
            nonlinearity = getattr(torch, nonlinearity)

        self.in_channels = in_channels
        self.ratio = ratio
        self.gnn = GNN(in_channels, 1, **kwargs)
        self.min_score = min_score
        self.multiplier = multiplier
        self.nonlinearity = nonlinearity

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.gnn.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_attr: OptTensor = None,
        batch: OptTensor = None,
        attn: OptTensor = None,
    ) -> Tuple[Tensor, Tensor, OptTensor, Tensor, Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): The node feature matrix.
            edge_index (torch.Tensor): The edge indices.
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example. (default: :obj:`None`)
            attn (torch.Tensor, optional): Optional node-level matrix to use
                for computing attention scores instead of using the node
                feature matrix :obj:`x`. (default: :obj:`None`)
        """
        if batch is None:
            batch = edge_index.new_zeros(x.size(0))

        attn = x if attn is None else attn
        attn = attn.unsqueeze(-1) if attn.dim() == 1 else attn
        score = self.gnn(attn, edge_index).view(-1)

        if self.min_score is None:
            score = self.nonlinearity(score)
        else:
            score = softmax(score, batch)

        perm = topk(score, self.ratio, batch, self.min_score)
        x = x[perm] * score[perm].view(-1, 1)
        x = self.multiplier * x if self.multiplier != 1 else x

        batch = batch[perm]
        edge_index, edge_attr = filter_adj(edge_index, edge_attr, perm,
                                           num_nodes=score.size(0))

        return x, edge_index, edge_attr, batch, perm, score[perm]

    def __repr__(self) -> str:
        if self.min_score is None:
            ratio = f'ratio={self.ratio}'
        else:
            ratio = f'min_score={self.min_score}'

        return (f'{self.__class__.__name__}({self.gnn.__class__.__name__}, '
                f'{self.in_channels}, {ratio}, multiplier={self.multiplier})')
pytorch/torch_geometric/nn/pool/select/000077500000000000000000000000001444262304300205705ustar00rootroot00000000000000pytorch/torch_geometric/nn/pool/select/__init__.py000066400000000000000000000002071444262304300227000ustar00rootroot00000000000000from .base import Select, SelectOutput
from .topk import SelectTopK

__all__ = [
    'Select',
    'SelectOutput',
    'SelectTopK',
]
pytorch/torch_geometric/nn/pool/select/base.py000066400000000000000000000062101444262304300220530ustar00rootroot00000000000000from dataclasses import dataclass
from typing import Optional

import torch
from torch import Tensor


@torch.jit.script
@dataclass(init=False)
class SelectOutput:
    r"""The output of the :class:`Select` method, which holds an assignment
    from selected nodes to their respective cluster(s).

    Args:
        node_index (torch.Tensor): The indices of the selected nodes.
        num_nodes (int): The number of nodes.
        cluster_index (torch.Tensor): The indices of the clusters each node in
            :obj:`node_index` is assigned to.
        num_clusters (int): The number of clusters.
        weight (torch.Tensor, optional): A weight vector, denoting the strength
            of the assignment of a node to its cluster. (default: :obj:`None`)
    """
    node_index: Tensor
    num_nodes: int
    cluster_index: Tensor
    num_clusters: int
    weight: Optional[Tensor] = None

    def __init__(
        self,
        node_index: Tensor,
        num_nodes: int,
        cluster_index: Tensor,
        num_clusters: int,
        weight: Optional[Tensor] = None,
    ):
        if node_index.dim() != 1:
            raise ValueError(f"Expected 'node_index' to be one-dimensional "
                             f"(got {node_index.dim()} dimensions)")

        if cluster_index.dim() != 1:
            raise ValueError(f"Expected 'cluster_index' to be one-dimensional "
                             f"(got {cluster_index.dim()} dimensions)")

        if node_index.numel() != cluster_index.numel():
            raise ValueError(f"Expected 'node_index' and 'cluster_index' to "
                             f"hold the same number of values (got "
                             f"{node_index.numel()} and "
                             f"{cluster_index.numel()} values)")

        if weight is not None and weight.dim() != 1:
            raise ValueError(f"Expected 'weight' vector to be one-dimensional "
                             f"(got {weight.dim()} dimensions)")

        if weight is not None and weight.numel() != node_index.numel():
            raise ValueError(f"Expected 'weight' to hold {node_index.numel()} "
                             f"values (got {weight.numel()} values)")

        self.node_index = node_index
        self.num_nodes = num_nodes
        self.cluster_index = cluster_index
        self.num_clusters = num_clusters
        self.weight = weight


class Select(torch.nn.Module):
    r"""An abstract base class for implementing custom node selections as
    described in the `"Understanding Pooling in Graph Neural Networks"
    `_ paper, which maps the nodes of an
    input graph to supernodes in the coarsened graph.

    Specifically, :class:`Select` returns a :class:`SelectOutput` output, which
    holds a (sparse) mapping :math:`\mathbf{C} \in {[0, 1]}^{N \times C}` that
    assigns selected nodes to one or more of :math:`C` super nodes.
    """
    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        pass

    def forward(self, *args, **kwargs) -> SelectOutput:
        raise NotImplementedError

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/nn/pool/select/topk.py000066400000000000000000000141661444262304300221270ustar00rootroot00000000000000from typing import Callable, Optional, Union

import torch
from torch import Tensor

from torch_geometric.nn.inits import uniform
from torch_geometric.nn.resolver import activation_resolver
from torch_geometric.utils import scatter, softmax

from .base import Select, SelectOutput


# TODO (matthias) Benchmark and document this method.
def topk(
    x: Tensor,
    ratio: Optional[Union[float, int]],
    batch: Tensor,
    min_score: Optional[float] = None,
    tol: float = 1e-7,
) -> Tensor:
    if min_score is not None:
        # Make sure that we do not drop all nodes in a graph.
        scores_max = scatter(x, batch, reduce='max')[batch] - tol
        scores_min = scores_max.clamp(max=min_score)

        perm = (x > scores_min).nonzero().view(-1)
        return perm

    if ratio is not None:
        num_nodes = scatter(batch.new_ones(x.size(0)), batch, reduce='sum')
        batch_size, max_num_nodes = num_nodes.size(0), int(num_nodes.max())

        cum_num_nodes = torch.cat(
            [num_nodes.new_zeros(1),
             num_nodes.cumsum(dim=0)[:-1]], dim=0)

        index = torch.arange(batch.size(0), dtype=torch.long, device=x.device)
        index = (index - cum_num_nodes[batch]) + (batch * max_num_nodes)

        dense_x = x.new_full((batch_size * max_num_nodes, ), -60000.0)
        dense_x[index] = x
        dense_x = dense_x.view(batch_size, max_num_nodes)

        _, perm = dense_x.sort(dim=-1, descending=True)

        perm = perm + cum_num_nodes.view(-1, 1)
        perm = perm.view(-1)

        if ratio >= 1:
            k = num_nodes.new_full((num_nodes.size(0), ), int(ratio))
            k = torch.min(k, num_nodes)
        else:
            k = (float(ratio) * num_nodes.to(x.dtype)).ceil().to(torch.long)

        if isinstance(ratio, int) and (k == ratio).all():
            # If all graphs have exactly `ratio` or more than `ratio` entries,
            # we can just pick the first entries in `perm` batch-wise:
            index = torch.arange(batch_size, device=x.device) * max_num_nodes
            index = index.view(-1, 1).repeat(1, ratio).view(-1)
            index += torch.arange(ratio, device=x.device).repeat(batch_size)
        else:
            # Otherwise, compute indices per graph:
            index = torch.cat([
                torch.arange(k[i], device=x.device) + i * max_num_nodes
                for i in range(batch_size)
            ], dim=0)

        perm = perm[index]
        return perm

    raise ValueError("At least one of the 'ratio' and 'min_score' parameters "
                     "must be specified")


class SelectTopK(Select):
    r"""Selects the top-:math:`k` nodes with highest projection scores from the
    `"Graph U-Nets" `_, `"Towards Sparse
    Hierarchical Graph Classifiers" `_
    and `"Understanding Attention and Generalization in Graph Neural
    Networks" `_ papers.

    If :obj:`min_score` :math:`\tilde{\alpha}` is :obj:`None`, computes:

        .. math::
            \mathbf{y} &= \sigma \left( \frac{\mathbf{X}\mathbf{p}}{\|
            \mathbf{p} \|} \right)

            \mathbf{i} &= \mathrm{top}_k(\mathbf{y})

    If :obj:`min_score` :math:`\tilde{\alpha}` is a value in :obj:`[0, 1]`,
    computes:

        .. math::
            \mathbf{y} &= \mathrm{softmax}(\mathbf{X}\mathbf{p})

            \mathbf{i} &= \mathbf{y}_i > \tilde{\alpha}

    where :math:`\mathbf{p}` is the learnable projection vector.

    Args:
        in_channels (int): Size of each input sample.
        ratio (float or int): The graph pooling ratio, which is used to compute
            :math:`k = \lceil \mathrm{ratio} \cdot N \rceil`, or the value
            of :math:`k` itself, depending on whether the type of :obj:`ratio`
            is :obj:`float` or :obj:`int`.
            This value is ignored if :obj:`min_score` is not :obj:`None`.
            (default: :obj:`0.5`)
        min_score (float, optional): Minimal node score :math:`\tilde{\alpha}`
            which is used to compute indices of pooled nodes
            :math:`\mathbf{i} = \mathbf{y}_i > \tilde{\alpha}`.
            When this value is not :obj:`None`, the :obj:`ratio` argument is
            ignored. (default: :obj:`None`)
        act (str or callable, optional): The non-linearity :math:`\sigma`.
            (default: :obj:`"tanh"`)
    """
    def __init__(
        self,
        in_channels: int,
        ratio: Union[int, float] = 0.5,
        min_score: Optional[float] = None,
        act: Union[str, Callable] = 'tanh',
    ):
        super().__init__()

        if ratio is None and min_score is None:
            raise ValueError(f"At least one of the 'ratio' and 'min_score' "
                             f"parameters must be specified in "
                             f"'{self.__class__.__name__}'")

        self.in_channels = in_channels
        self.ratio = ratio
        self.min_score = min_score
        self.act = activation_resolver(act)

        self.weight = torch.nn.Parameter(torch.Tensor(1, in_channels))

        self.reset_parameters()

    def reset_parameters(self):
        uniform(self.in_channels, self.weight)

    def forward(
        self,
        x: Tensor,
        batch: Optional[Tensor] = None,
    ) -> SelectOutput:
        """"""
        if batch is None:
            batch = x.new_zeros(x.size(0), dtype=torch.long)

        x.view(-1, 1) if x.dim() == 1 else x
        score = (x * self.weight).sum(dim=-1)

        if self.min_score is None:
            score = self.act(score / self.weight.norm(p=2, dim=-1))
        else:
            score = softmax(score, batch)

        node_index = topk(score, self.ratio, batch, self.min_score)

        return SelectOutput(
            node_index=node_index,
            num_nodes=x.size(0),
            cluster_index=torch.arange(node_index.size(0), device=x.device),
            num_clusters=node_index.size(0),
            weight=score[node_index],
        )

    def __repr__(self) -> str:
        if self.min_score is None:
            arg = f'ratio={self.ratio}'
        else:
            arg = f'min_score={self.min_score}'
        return f'{self.__class__.__name__}({self.in_channels}, {arg})'
pytorch/torch_geometric/nn/pool/topk_pool.py000066400000000000000000000132001444262304300216650ustar00rootroot00000000000000from typing import Callable, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.nn.pool.select import SelectTopK
from torch_geometric.utils.num_nodes import maybe_num_nodes


def filter_adj(
    edge_index: Tensor,
    edge_attr: Optional[Tensor],
    perm: Tensor,
    num_nodes: Optional[int] = None,
) -> Tuple[Tensor, Optional[Tensor]]:
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    mask = perm.new_full((num_nodes, ), -1)
    i = torch.arange(perm.size(0), dtype=torch.long, device=perm.device)
    mask[perm] = i

    row, col = edge_index[0], edge_index[1]
    row, col = mask[row], mask[col]
    mask = (row >= 0) & (col >= 0)
    row, col = row[mask], col[mask]

    if edge_attr is not None:
        edge_attr = edge_attr[mask]

    return torch.stack([row, col], dim=0), edge_attr


class TopKPooling(torch.nn.Module):
    r""":math:`\mathrm{top}_k` pooling operator from the `"Graph U-Nets"
    `_, `"Towards Sparse
    Hierarchical Graph Classifiers" `_
    and `"Understanding Attention and Generalization in Graph Neural
    Networks" `_ papers.

    If :obj:`min_score` :math:`\tilde{\alpha}` is :obj:`None`, computes:

        .. math::
            \mathbf{y} &= \sigma \left( \frac{\mathbf{X}\mathbf{p}}{\|
            \mathbf{p} \|} \right)

            \mathbf{i} &= \mathrm{top}_k(\mathbf{y})

            \mathbf{X}^{\prime} &= (\mathbf{X} \odot
            \mathrm{tanh}(\mathbf{y}))_{\mathbf{i}}

            \mathbf{A}^{\prime} &= \mathbf{A}_{\mathbf{i},\mathbf{i}}

    If :obj:`min_score` :math:`\tilde{\alpha}` is a value in :obj:`[0, 1]`,
    computes:

        .. math::
            \mathbf{y} &= \mathrm{softmax}(\mathbf{X}\mathbf{p})

            \mathbf{i} &= \mathbf{y}_i > \tilde{\alpha}

            \mathbf{X}^{\prime} &= (\mathbf{X} \odot \mathbf{y})_{\mathbf{i}}

            \mathbf{A}^{\prime} &= \mathbf{A}_{\mathbf{i},\mathbf{i}},

    where nodes are dropped based on a learnable projection score
    :math:`\mathbf{p}`.

    Args:
        in_channels (int): Size of each input sample.
        ratio (float or int): The graph pooling ratio, which is used to compute
            :math:`k = \lceil \mathrm{ratio} \cdot N \rceil`, or the value
            of :math:`k` itself, depending on whether the type of :obj:`ratio`
            is :obj:`float` or :obj:`int`.
            This value is ignored if :obj:`min_score` is not :obj:`None`.
            (default: :obj:`0.5`)
        min_score (float, optional): Minimal node score :math:`\tilde{\alpha}`
            which is used to compute indices of pooled nodes
            :math:`\mathbf{i} = \mathbf{y}_i > \tilde{\alpha}`.
            When this value is not :obj:`None`, the :obj:`ratio` argument is
            ignored. (default: :obj:`None`)
        multiplier (float, optional): Coefficient by which features gets
            multiplied after pooling. This can be useful for large graphs and
            when :obj:`min_score` is used. (default: :obj:`1`)
        nonlinearity (str or callable, optional): The non-linearity
            :math:`\sigma`. (default: :obj:`"tanh"`)
    """
    def __init__(
        self,
        in_channels: int,
        ratio: Union[int, float] = 0.5,
        min_score: Optional[float] = None,
        multiplier: float = 1.,
        nonlinearity: Union[str, Callable] = 'tanh',
    ):
        super().__init__()

        self.in_channels = in_channels
        self.ratio = ratio
        self.min_score = min_score
        self.multiplier = multiplier

        self.select = SelectTopK(in_channels, ratio, min_score, nonlinearity)

        self.reset_parameters()

    def reset_parameters(self):
        r"""Resets all learnable parameters of the module."""
        self.select.reset_parameters()

    def forward(
        self,
        x: Tensor,
        edge_index: Tensor,
        edge_attr: Optional[Tensor] = None,
        batch: Optional[Tensor] = None,
        attn: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Tensor, Optional[Tensor], Tensor, Tensor, Tensor]:
        r"""
        Args:
            x (torch.Tensor): The node feature matrix.
            edge_index (torch.Tensor): The edge indices.
            edge_attr (torch.Tensor, optional): The edge features.
                (default: :obj:`None`)
            batch (torch.Tensor, optional): The batch vector
                :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
                each node to a specific example. (default: :obj:`None`)
            attn (torch.Tensor, optional): Optional node-level matrix to use
                for computing attention scores instead of using the node
                feature matrix :obj:`x`. (default: :obj:`None`)
        """
        if batch is None:
            batch = edge_index.new_zeros(x.size(0))

        attn = x if attn is None else attn
        select_output = self.select(attn, batch)

        perm = select_output.node_index
        score = select_output.weight
        assert score is not None

        x = x[perm] * score.view(-1, 1)
        x = self.multiplier * x if self.multiplier != 1 else x

        batch = batch[perm]
        edge_index, edge_attr = filter_adj(edge_index, edge_attr, perm,
                                           num_nodes=select_output.num_nodes)

        return x, edge_index, edge_attr, batch, perm, score

    def __repr__(self) -> str:
        if self.min_score is None:
            ratio = f'ratio={self.ratio}'
        else:
            ratio = f'min_score={self.min_score}'

        return (f'{self.__class__.__name__}({self.in_channels}, {ratio}, '
                f'multiplier={self.multiplier})')
pytorch/torch_geometric/nn/pool/voxel_grid.py000066400000000000000000000052611444262304300220310ustar00rootroot00000000000000from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.utils.repeat import repeat

try:
    from torch_cluster import grid_cluster
except ImportError:
    grid_cluster = None


def voxel_grid(
    pos: Tensor,
    size: Union[float, List[float], Tensor],
    batch: Optional[Tensor] = None,
    start: Optional[Union[float, List[float], Tensor]] = None,
    end: Optional[Union[float, List[float], Tensor]] = None,
) -> Tensor:
    r"""Voxel grid pooling from the, *e.g.*, `Dynamic Edge-Conditioned Filters
    in Convolutional Networks on Graphs `_
    paper, which overlays a regular grid of user-defined size over a point
    cloud and clusters all points within the same voxel.

    Args:
        pos (torch.Tensor): Node position matrix
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times D}`.
        size (float or [float] or Tensor): Size of a voxel (in each dimension).
        batch (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots,B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        start (float or [float] or Tensor, optional): Start coordinates of the
            grid (in each dimension). If set to :obj:`None`, will be set to the
            minimum coordinates found in :attr:`pos`. (default: :obj:`None`)
        end (float or [float] or Tensor, optional): End coordinates of the grid
            (in each dimension). If set to :obj:`None`, will be set to the
            maximum coordinates found in :attr:`pos`. (default: :obj:`None`)

    :rtype: :class:`torch.Tensor`
    """

    if grid_cluster is None:
        raise ImportError('`voxel_grid` requires `torch-cluster`.')

    pos = pos.unsqueeze(-1) if pos.dim() == 1 else pos
    dim = pos.size(1)

    if batch is None:
        batch = pos.new_zeros(pos.size(0), dtype=torch.long)

    pos = torch.cat([pos, batch.view(-1, 1).to(pos.dtype)], dim=-1)

    if not isinstance(size, Tensor):
        size = torch.tensor(size, dtype=pos.dtype, device=pos.device)
    size = repeat(size, dim)
    size = torch.cat([size, size.new_ones(1)])  # Add additional batch dim.

    if start is not None:
        if not isinstance(start, Tensor):
            start = torch.tensor(start, dtype=pos.dtype, device=pos.device)
        start = repeat(start, dim)
        start = torch.cat([start, start.new_zeros(1)])

    if end is not None:
        if not isinstance(end, Tensor):
            end = torch.tensor(end, dtype=pos.dtype, device=pos.device)
        end = repeat(end, dim)
        end = torch.cat([end, batch.max().unsqueeze(0)])

    return grid_cluster(pos, size, start, end)
pytorch/torch_geometric/nn/reshape.py000066400000000000000000000006341444262304300203440ustar00rootroot00000000000000import torch
from torch import Tensor


class Reshape(torch.nn.Module):
    def __init__(self, *shape):
        super().__init__()
        self.shape = shape

    def forward(self, x: Tensor) -> Tensor:
        """"""
        x = x.view(*self.shape)
        return x

    def __repr__(self) -> str:
        shape = ', '.join([str(dim) for dim in self.shape])
        return f'{self.__class__.__name__}({shape})'
pytorch/torch_geometric/nn/resolver.py000066400000000000000000000140131444262304300205520ustar00rootroot00000000000000import inspect
from typing import Any, Optional, Union

import torch
from torch import Tensor
from torch.optim import Optimizer
from torch.optim.lr_scheduler import ReduceLROnPlateau

from torch_geometric.nn.lr_scheduler import (
    ConstantWithWarmupLR,
    CosineWithWarmupLR,
    CosineWithWarmupRestartsLR,
    LinearWithWarmupLR,
    PolynomialWithWarmupLR,
)
from torch_geometric.resolver import normalize_string, resolver

try:
    from torch.optim.lr_scheduler import LRScheduler
except ImportError:  # PyTorch < 2.0
    from torch.optim.lr_scheduler import _LRScheduler as LRScheduler

# Activation Resolver #########################################################


def swish(x: Tensor) -> Tensor:
    return x * x.sigmoid()


def activation_resolver(query: Union[Any, str] = 'relu', *args, **kwargs):
    base_cls = torch.nn.Module
    base_cls_repr = 'Act'
    acts = [
        act for act in vars(torch.nn.modules.activation).values()
        if isinstance(act, type) and issubclass(act, base_cls)
    ]
    acts += [
        swish,
    ]
    act_dict = {}
    return resolver(acts, act_dict, query, base_cls, base_cls_repr, *args,
                    **kwargs)


# Normalization Resolver ######################################################


def normalization_resolver(query: Union[Any, str], *args, **kwargs):
    import torch_geometric.nn.norm as norm
    base_cls = torch.nn.Module
    base_cls_repr = 'Norm'
    norms = [
        norm for norm in vars(norm).values()
        if isinstance(norm, type) and issubclass(norm, base_cls)
    ]
    norm_dict = {}
    return resolver(norms, norm_dict, query, base_cls, base_cls_repr, *args,
                    **kwargs)


# Aggregation Resolver ########################################################


def aggregation_resolver(query: Union[Any, str], *args, **kwargs):
    import torch_geometric.nn.aggr as aggr
    if isinstance(query, (list, tuple)):
        return aggr.MultiAggregation(query, *args, **kwargs)

    base_cls = aggr.Aggregation
    aggrs = [
        aggr for aggr in vars(aggr).values()
        if isinstance(aggr, type) and issubclass(aggr, base_cls)
    ]
    aggr_dict = {
        'add': aggr.SumAggregation,
    }
    return resolver(aggrs, aggr_dict, query, base_cls, None, *args, **kwargs)


# Optimizer Resolver ##########################################################


def optimizer_resolver(query: Union[Any, str], *args, **kwargs):
    base_cls = Optimizer
    optimizers = [
        optimizer for optimizer in vars(torch.optim).values()
        if isinstance(optimizer, type) and issubclass(optimizer, base_cls)
    ]
    return resolver(optimizers, {}, query, base_cls, None, *args, **kwargs)


# Learning Rate Scheduler Resolver ############################################


def lr_scheduler_resolver(
    query: Union[Any, str],
    optimizer: Optimizer,
    warmup_ratio_or_steps: Optional[Union[float, int]] = 0.1,
    num_training_steps: Optional[int] = None,
    **kwargs,
) -> Union[LRScheduler, ReduceLROnPlateau]:
    r"""A resolver to obtain a learning rate scheduler implemented in either
    PyG or PyTorch from its name or type.

    Args:
        query (Any or str): The query name of the learning rate scheduler.
        optimizer (Optimizer): The optimizer to be scheduled.
        warmup_ratio_or_steps (float or int, optional): The number of warmup
            steps. If given as a `float`, it will act as a ratio that gets
            multiplied with the number of training steps to obtain the number
            of warmup steps. Only required for warmup-based LR schedulers.
            (default: :obj:`0.1`)
        num_training_steps (int, optional): The total number of training steps.
            (default: :obj:`None`)
        **kwargs (optional): Additional arguments of the LR scheduler.
    """
    if not isinstance(query, str):
        return query

    if isinstance(warmup_ratio_or_steps, float):
        if warmup_ratio_or_steps < 0 or warmup_ratio_or_steps > 1:
            raise ValueError(f"`warmup_ratio_or_steps` needs to be between "
                             f"0.0 and 1.0 when given as a floating point "
                             f"number (got {warmup_ratio_or_steps}).")
        if num_training_steps is not None:
            warmup_steps = round(warmup_ratio_or_steps * num_training_steps)
    elif isinstance(warmup_ratio_or_steps, int):
        if warmup_ratio_or_steps < 0:
            raise ValueError(f"`warmup_ratio_or_steps` needs to be positive "
                             f"when given as an integer "
                             f"(got {warmup_ratio_or_steps}).")
        warmup_steps = warmup_ratio_or_steps
    else:
        raise ValueError(f"Found invalid type of `warmup_ratio_or_steps` "
                         f"(got {type(warmup_ratio_or_steps)})")

    base_cls = LRScheduler
    classes = [
        scheduler for scheduler in vars(torch.optim.lr_scheduler).values()
        if isinstance(scheduler, type) and issubclass(scheduler, base_cls)
    ] + [ReduceLROnPlateau]

    customized_lr_schedulers = [
        ConstantWithWarmupLR,
        LinearWithWarmupLR,
        CosineWithWarmupLR,
        CosineWithWarmupRestartsLR,
        PolynomialWithWarmupLR,
    ]
    classes += customized_lr_schedulers

    query_repr = normalize_string(query)
    base_cls_repr = normalize_string('LR')

    for cls in classes:
        cls_repr = normalize_string(cls.__name__)
        if query_repr in [cls_repr, cls_repr.replace(base_cls_repr, '')]:
            if inspect.isclass(cls):
                if cls in customized_lr_schedulers:
                    cls_keys = inspect.signature(cls).parameters.keys()
                    if 'num_warmup_steps' in cls_keys:
                        kwargs['num_warmup_steps'] = warmup_steps
                    if 'num_training_steps' in cls_keys:
                        kwargs['num_training_steps'] = num_training_steps
                obj = cls(optimizer, **kwargs)
                return obj
            return cls

    choices = set(cls.__name__ for cls in classes)
    raise ValueError(f"Could not resolve '{query}' among choices {choices}")
pytorch/torch_geometric/nn/sequential.jinja000066400000000000000000000020571444262304300215330ustar00rootroot00000000000000from typing import *
from torch_geometric.typing import *

import torch
from torch import Tensor


class {{cls_name}}(torch.nn.Module):
    def reset_parameters(self):
        for name in self._names:
            module = getattr(self, name)
            if hasattr(module, 'reset_parameters'):
                module.reset_parameters()

    def forward(self, {{ input_args|join(', ') }}):
        """"""{% for name, module, in_desc, out_desc in calls %}
        {% for output in out_desc %}{{output}}{{ ", " if not loop.last }}{% endfor %} = self.{{name}}({% for input in in_desc %}{{input}}{{ ", " if not loop.last }}{% endfor %}){% endfor %}
        return {% for output in calls[-1][-1] %}{{output}}{{ ", " if not loop.last }}{% endfor %}

    def __getitem__(self, idx: int) -> torch.nn.Module:
        return getattr(self, self._names[idx])

    def __len__(self) -> int:
        return {{calls|length}}

    def __repr__(self) -> str:
        return 'Sequential(\n{}\n)'.format('\n'.join(
            [f'  ({idx}): ' + str(self[idx]) for idx in range(len(self))]))
pytorch/torch_geometric/nn/sequential.py000066400000000000000000000107411444262304300210670ustar00rootroot00000000000000import os
import os.path as osp
from typing import Callable, List, Tuple, Union
from uuid import uuid1

import torch

from torch_geometric.nn.conv.utils.jit import class_from_module_repr


def Sequential(
    input_args: str,
    modules: List[Union[Tuple[Callable, str], Callable]],
) -> torch.nn.Module:
    r"""An extension of the :class:`torch.nn.Sequential` container in order to
    define a sequential GNN model.
    Since GNN operators take in multiple input arguments,
    :class:`torch_geometric.nn.Sequential` expects both global input
    arguments, and function header definitions of individual operators.
    If omitted, an intermediate module will operate on the *output* of its
    preceding module:

    .. code-block:: python

        from torch.nn import Linear, ReLU
        from torch_geometric.nn import Sequential, GCNConv

        model = Sequential('x, edge_index', [
            (GCNConv(in_channels, 64), 'x, edge_index -> x'),
            ReLU(inplace=True),
            (GCNConv(64, 64), 'x, edge_index -> x'),
            ReLU(inplace=True),
            Linear(64, out_channels),
        ])

    where :obj:`'x, edge_index'` defines the input arguments of :obj:`model`,
    and :obj:`'x, edge_index -> x'` defines the function header, *i.e.* input
    arguments *and* return types, of :class:`~torch_geometric.nn.conv.GCNConv`.

    In particular, this also allows to create more sophisticated models,
    such as utilizing :class:`~torch_geometric.nn.models.JumpingKnowledge`:

    .. code-block:: python

        from torch.nn import Linear, ReLU, Dropout
        from torch_geometric.nn import Sequential, GCNConv, JumpingKnowledge
        from torch_geometric.nn import global_mean_pool

        model = Sequential('x, edge_index, batch', [
            (Dropout(p=0.5), 'x -> x'),
            (GCNConv(dataset.num_features, 64), 'x, edge_index -> x1'),
            ReLU(inplace=True),
            (GCNConv(64, 64), 'x1, edge_index -> x2'),
            ReLU(inplace=True),
            (lambda x1, x2: [x1, x2], 'x1, x2 -> xs'),
            (JumpingKnowledge("cat", 64, num_layers=2), 'xs -> x'),
            (global_mean_pool, 'x, batch -> x'),
            Linear(2 * 64, dataset.num_classes),
        ])

    Args:
        input_args (str): The input arguments of the model.
        modules ([(str, Callable) or Callable]): A list of modules (with
            optional function header definitions). Alternatively, an
            :obj:`OrderedDict` of modules (and function header definitions) can
            be passed.
    """
    try:
        from jinja2 import Template
    except ImportError:
        raise ModuleNotFoundError(
            "No module named 'jinja2' found on this machine. "
            "Run 'pip install jinja2' to install the library.")

    input_args = [x.strip() for x in input_args.split(',')]

    if not isinstance(modules, dict):
        modules = {f'module_{i}': module for i, module in enumerate(modules)}

    # We require the first entry of the input list to define arguments:
    assert len(modules) > 0
    first_module = list(modules.values())[0]
    assert isinstance(first_module, (tuple, list))

    # A list holding the callable function and the input and output names:
    calls: List[Tuple[str, Callable, List[str], List[str]]] = []

    for name, module in modules.items():
        if isinstance(module, (tuple, list)) and len(module) >= 2:
            module, desc = module[:2]
            in_desc, out_desc = parse_desc(desc)
        elif isinstance(module, (tuple, list)):
            module = module[0]
            in_desc = out_desc = calls[-1][-1]
        else:
            in_desc = out_desc = calls[-1][-1]

        calls.append((name, module, in_desc, out_desc))

    root = os.path.dirname(osp.realpath(__file__))
    with open(osp.join(root, 'sequential.jinja'), 'r') as f:
        template = Template(f.read())

    cls_name = f'Sequential_{uuid1().hex[:6]}'
    module_repr = template.render(
        cls_name=cls_name,
        input_args=input_args,
        calls=calls,
    )

    # Instantiate a class from the rendered module representation.
    module = class_from_module_repr(cls_name, module_repr)()
    module._names = list(modules.keys())
    for name, submodule, _, _ in calls:
        setattr(module, name, submodule)
    return module


def parse_desc(desc: str) -> Tuple[List[str], List[str]]:
    in_desc, out_desc = desc.split('->')
    in_desc = [x.strip() for x in in_desc.split(',')]
    out_desc = [x.strip() for x in out_desc.split(',')]
    return in_desc, out_desc
pytorch/torch_geometric/nn/summary.py000066400000000000000000000127601444262304300204150ustar00rootroot00000000000000from collections import defaultdict
from typing import Any, List, Optional, Union

import torch
from torch.jit import ScriptModule
from torch.nn import Module

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.typing import SparseTensor


def summary(
    model: torch.nn.Module,
    *args,
    max_depth: int = 3,
    leaf_module: Optional[Union[Module, List[Module]]] = 'MessagePassing',
    **kwargs,
) -> str:
    r"""Summarizes a given :class:`torch.nn.Module`.
    The summarized information includes (1) layer names, (2) input and output
    shapes, and (3) the number of parameters.

    .. code-block:: python

        import torch
        from torch_geometric.nn import GCN, summary

        model = GCN(128, 64, num_layers=2, out_channels=32)
        x = torch.randn(100, 128)
        edge_index = torch.randint(100, size=(2, 20))

        print(summary(model, x, edge_index))

    .. code-block::

        +---------------------+---------------------+--------------+--------+
        | Layer               | Input Shape         | Output Shape | #Param |
        |---------------------+---------------------+--------------+--------|
        | GCN                 | [100, 128], [2, 20] | [100, 32]    | 10,336 |
        | β”œβ”€(act)ReLU         | [100, 64]           | [100, 64]    | --     |
        | β”œβ”€(convs)ModuleList | --                  | --           | 10,336 |
        | β”‚    └─(0)GCNConv   | [100, 128], [2, 20] | [100, 64]    | 8,256  |
        | β”‚    └─(1)GCNConv   | [100, 64], [2, 20]  | [100, 32]    | 2,080  |
        +---------------------+---------------------+--------------+--------+

    Args:
        model (torch.nn.Module): The model to summarize.
        *args: The arguments of the :obj:`model`.
        max_depth (int, optional): The depth of nested layers to display.
            Any layers deeper than this depth will not be displayed in the
            summary. (default: :obj:`3`)
        leaf_module (torch.nn.Module or [torch.nn.Module], optional): The
            modules to be treated as leaf modules, whose submodules are
            excluded from the summary.
            (default: :class:`~torch_geometric.nn.conv.MessagePassing`)
        **kwargs: Additional arguments of the :obj:`model`.
    """
    # NOTE This is just for the doc-string to render nicely:
    if leaf_module == 'MessagePassing':
        leaf_module = MessagePassing

    def register_hook(info):
        def hook(module, inputs, output):
            info['input_shape'].append(get_shape(inputs))
            info['output_shape'].append(get_shape(output))

        return hook

    hooks = {}
    depth = 0
    stack = [(model.__class__.__name__, model, depth)]

    info_list = []
    input_shape = defaultdict(list)
    output_shape = defaultdict(list)
    while stack:
        name, module, depth = stack.pop()
        module_id = id(module)

        if name.startswith('(_'):  # Do not summarize private modules.
            continue

        if module_id in hooks:  # Avoid duplicated hooks.
            hooks[module_id].remove()

        info = {}
        info['name'] = name
        info['input_shape'] = input_shape[module_id]
        info['output_shape'] = output_shape[module_id]
        info['depth'] = depth
        num_params = sum(p.numel() for p in module.parameters())
        info['#param'] = f'{num_params:,}' if num_params > 0 else '--'
        info_list.append(info)

        if not isinstance(module, ScriptModule):
            hooks[module_id] = module.register_forward_hook(
                register_hook(info))

        if depth >= max_depth:
            continue

        if (leaf_module is not None and isinstance(module, leaf_module)):
            continue

        module_items = reversed(module._modules.items())
        stack += [(f"({name}){mod.__class__.__name__}", mod, depth + 1)
                  for name, mod in module_items if mod is not None]

    training = model.training
    model.eval()

    with torch.no_grad():
        model(*args, **kwargs)

    model.train(training)

    for h in hooks.values():  # Remove hooks.
        h.remove()

    info_list = postprocess(info_list)
    return make_table(info_list, max_depth=max_depth)


def get_shape(inputs: Any) -> str:
    if not isinstance(inputs, (tuple, list)):
        inputs = (inputs, )

    out = []
    for x in inputs:
        if isinstance(x, SparseTensor):
            out.append(str(list(x.sizes())))
        elif hasattr(x, 'size'):
            out.append(str(list(x.size())))
    return ', '.join(out)


def postprocess(info_list: List[dict]) -> List[dict]:
    for idx, info in enumerate(info_list):
        depth = info['depth']
        if idx > 0:  # root module (0) is exclued
            if depth == 1:
                prefix = 'β”œβ”€'
            else:
                prefix = f"{'β”‚    '*(depth-1)}└─"
            info['name'] = prefix + info['name']

        if info['input_shape']:
            info['input_shape'] = info['input_shape'].pop(0)
            info['output_shape'] = info['output_shape'].pop(0)
        else:
            info['input_shape'] = '--'
            info['output_shape'] = '--'
    return info_list


def make_table(info_list: List[dict], max_depth: int) -> str:
    from tabulate import tabulate
    content = [['Layer', 'Input Shape', 'Output Shape', '#Param']]
    for info in info_list:
        content.append([
            info['name'],
            info['input_shape'],
            info['output_shape'],
            info['#param'],
        ])
    return tabulate(content, headers='firstrow', tablefmt='psql')
pytorch/torch_geometric/nn/to_fixed_size_transformer.py000066400000000000000000000024021444262304300241650ustar00rootroot00000000000000from typing import Any

from torch.nn import Module

from torch_geometric.nn.fx import Transformer

try:
    from torch.fx import Graph, GraphModule, Node
except (ImportError, ModuleNotFoundError, AttributeError):
    GraphModule, Graph, Node = 'GraphModule', 'Graph', 'Node'


def to_fixed_size(module: Module, batch_size: int,
                  debug: bool = False) -> GraphModule:
    r"""Converts a model and injects a pre-computed and fixed batch size to all
    global pooling operators.

    Args:
        module (torch.nn.Module): The model to transform.
        batch_size (int): The fixed batch size used in global pooling modules.
        debug (bool, optional): If set to :obj:`True`, will perform
            transformation in debug mode. (default: :obj:`False`)
    """
    transformer = ToFixedSizeTransformer(module, batch_size, debug)
    return transformer.transform()


class ToFixedSizeTransformer(Transformer):
    def __init__(self, module: Module, batch_size: int, debug: bool = False):
        super().__init__(module, debug=debug)
        self.batch_size = batch_size

    def call_global_pooling_module(self, node: Node, target: Any, name: str):
        kwargs = node.kwargs.copy()
        kwargs['dim_size'] = self.batch_size
        node.kwargs = kwargs
pytorch/torch_geometric/nn/to_hetero_module.py000066400000000000000000000145261444262304300222570ustar00rootroot00000000000000import copy
import warnings
from typing import Dict, List, Optional, Union

import torch
import torch.nn.functional as F
from torch import Tensor

import torch_geometric
from torch_geometric.typing import EdgeType, NodeType, OptTensor
from torch_geometric.utils import scatter


class ToHeteroLinear(torch.nn.Module):
    def __init__(
        self,
        module: torch.nn.Module,
        types: Union[List[NodeType], List[EdgeType]],
    ):
        from torch_geometric.nn import HeteroLinear, Linear

        super().__init__()

        self.types = types

        if isinstance(module, Linear):
            in_channels = module.in_channels
            out_channels = module.out_channels
            bias = module.bias is not None

        elif isinstance(module, torch.nn.Linear):
            in_channels = module.in_features
            out_channels = module.out_features
            bias = module.bias is not None

        else:
            raise ValueError(f"Expected 'Linear' module (got '{type(module)}'")

        # TODO We currently assume that `x` is sorted according to `type`.
        self.hetero_module = HeteroLinear(
            in_channels,
            out_channels,
            num_types=len(types),
            is_sorted=True,
            bias=bias,
        )

    def fused_forward(self, x: Tensor, type_vec: Tensor) -> Tensor:
        return self.hetero_module(x, type_vec)

    def dict_forward(
        self,
        x_dict: Dict[Union[NodeType, EdgeType], Tensor],
    ) -> Dict[Union[NodeType, EdgeType], Tensor]:

        if not torch_geometric.typing.WITH_PYG_LIB:
            return {
                key: F.linear(x_dict[key], self.hetero_module.weight[i].T) +
                self.hetero_module.bias[i]
                for i, key in enumerate(self.types)
            }

        x = torch.cat([x_dict[key] for key in self.types], dim=0)
        sizes = [x_dict[key].size(0) for key in self.types]
        type_vec = torch.arange(len(self.types), device=x.device)
        size = torch.tensor(sizes, device=x.device)
        type_vec = type_vec.repeat_interleave(size)
        outs = self.hetero_module(x, type_vec).split(sizes)
        return {key: out for key, out in zip(self.types, outs)}

    def forward(
        self,
        x: Union[Tensor, Dict[Union[NodeType, EdgeType], Tensor]],
        type_vec: Optional[Tensor] = None,
    ) -> Union[Tensor, Dict[Union[NodeType, EdgeType], Tensor]]:

        if isinstance(x, dict):
            return self.dict_forward(x)

        elif isinstance(x, Tensor) and type_vec is not None:
            return self.fused_forward(x, type_vec)

        raise ValueError(f"Encountered invalid forward types in "
                         f"'{self.__class__.__name__}'")


class ToHeteroMessagePassing(torch.nn.Module):
    def __init__(
        self,
        module: torch.nn.Module,
        node_types: List[NodeType],
        edge_types: List[NodeType],
        aggr: str = 'sum',
    ):
        from torch_geometric.nn import HeteroConv, MessagePassing

        super().__init__()

        self.node_types = node_types
        self.node_type_to_index = {key: i for i, key in enumerate(node_types)}
        self.edge_types = edge_types

        if not isinstance(module, MessagePassing):
            raise ValueError(f"Expected 'MessagePassing' module "
                             f"(got '{type(module)}'")

        if (not hasattr(module, 'reset_parameters')
                and sum([p.numel() for p in module.parameters()]) > 0):
            warnings.warn(f"'{module}' will be duplicated, but its parameters "
                          f"cannot be reset. To suppress this warning, add a "
                          f"'reset_parameters()' method to '{module}'")

        convs = {edge_type: copy.deepcopy(module) for edge_type in edge_types}
        self.hetero_module = HeteroConv(convs, aggr)
        self.hetero_module.reset_parameters()

    def fused_forward(self, x: Tensor, edge_index: Tensor, node_type: Tensor,
                      edge_type: Tensor) -> Tensor:
        # TODO This currently does not fuse at all :(
        # TODO We currently assume that `x` and `edge_index` are both sorted
        # according to `type`.

        node_sizes = scatter(torch.ones_like(node_type), node_type, dim=0,
                             dim_size=len(self.node_types), reduce='sum')
        edge_sizes = scatter(torch.ones_like(edge_type), edge_type, dim=0,
                             dim_size=len(self.edge_types), reduce='sum')

        cumsum = torch.cat([node_type.new_zeros(1), node_sizes.cumsum(0)[:1]])

        xs = x.split(node_sizes.tolist())
        x_dict = {node_type: x for node_type, x in zip(self.node_types, xs)}

        # TODO Consider out-sourcing to its own function.
        edge_indices = edge_index.clone().split(edge_sizes.tolist(), dim=1)
        for (src, _, dst), index in zip(self.edge_types, edge_indices):
            index[0] -= cumsum[self.node_type_to_index[src]]
            index[1] -= cumsum[self.node_type_to_index[dst]]

        edge_index_dict = {
            edge_type: edge_index
            for edge_type, edge_index in zip(self.edge_types, edge_indices)
        }

        out_dict = self.hetero_module(x_dict, edge_index_dict)
        return torch.cat([out_dict[key] for key in self.node_types], dim=0)

    def dict_forward(
        self,
        x_dict: Dict[NodeType, Tensor],
        edge_index_dict: Dict[EdgeType, Tensor],
        **kwargs,
    ) -> Dict[NodeType, Tensor]:
        return self.hetero_module(x_dict, edge_index_dict, **kwargs)

    def forward(
        self,
        x: Union[Tensor, Dict[NodeType, Tensor]],
        edge_index: Union[Tensor, Dict[EdgeType, Tensor]],
        node_type: OptTensor = None,
        edge_type: OptTensor = None,
        **kwargs,
    ) -> Union[Tensor, Dict[NodeType, Tensor]]:

        if isinstance(x, dict) and isinstance(edge_index, dict):
            return self.dict_forward(x, edge_index, **kwargs)

        elif (isinstance(x, Tensor) and isinstance(edge_index, Tensor)
              and node_type is not None and edge_type is not None):

            if len(kwargs) > 0:
                raise ValueError("Additional forward arguments not yet "
                                 "supported in fused mode")

            return self.fused_forward(x, edge_index, node_type, edge_type)

        raise ValueError(f"Encountered invalid forward types in "
                         f"'{self.__class__.__name__}'")
pytorch/torch_geometric/nn/to_hetero_transformer.py000066400000000000000000000437471444262304300233430ustar00rootroot00000000000000import copy
import warnings
from collections import defaultdict, deque
from typing import Any, Dict, Optional, Tuple, Union

import torch
from torch.nn import Module

from torch_geometric.nn.dense.linear import is_uninitialized_parameter
from torch_geometric.nn.fx import Transformer, get_submodule
from torch_geometric.typing import EdgeType, Metadata, NodeType
from torch_geometric.utils.hetero import (
    check_add_self_loops,
    get_unused_node_types,
)

try:
    from torch.fx import Graph, GraphModule, Node
except (ImportError, ModuleNotFoundError, AttributeError):
    GraphModule, Graph, Node = 'GraphModule', 'Graph', 'Node'


def get_dict(mapping: Optional[Dict[str, Any]]) -> Dict[str, Any]:
    return mapping if mapping is not None else {}


def to_hetero(module: Module, metadata: Metadata, aggr: str = "sum",
              input_map: Optional[Dict[str, str]] = None,
              debug: bool = False) -> GraphModule:
    r"""Converts a homogeneous GNN model into its heterogeneous equivalent in
    which node representations are learned for each node type in
    :obj:`metadata[0]`, and messages are exchanged between each edge type in
    :obj:`metadata[1]`, as denoted in the `"Modeling Relational Data with Graph
    Convolutional Networks" `_ paper:

    .. code-block:: python

        import torch
        from torch_geometric.nn import SAGEConv, to_hetero

        class GNN(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.conv1 = SAGEConv((-1, -1), 32)
                self.conv2 = SAGEConv((32, 32), 32)

            def forward(self, x, edge_index):
                x = self.conv1(x, edge_index).relu()
                x = self.conv2(x, edge_index).relu()
                return x

        model = GNN()

        node_types = ['paper', 'author']
        edge_types = [
            ('paper', 'cites', 'paper'),
            ('paper', 'written_by', 'author'),
            ('author', 'writes', 'paper'),
        ]
        metadata = (node_types, edge_types)

        model = to_hetero(model, metadata)
        model(x_dict, edge_index_dict)

    where :obj:`x_dict` and :obj:`edge_index_dict` denote dictionaries that
    hold node features and edge connectivity information for each node type and
    edge type, respectively.

    The below illustration shows the original computation graph of the
    homogeneous model on the left, and the newly obtained computation graph of
    the heterogeneous model on the right:

    .. figure:: ../_figures/to_hetero.svg
      :align: center
      :width: 90%

      Transforming a model via :func:`to_hetero`.

    Here, each :class:`~torch_geometric.nn.conv.MessagePassing` instance
    :math:`f_{\theta}^{(\ell)}` is duplicated and stored in a set
    :math:`\{ f_{\theta}^{(\ell, r)} : r \in \mathcal{R} \}` (one instance for
    each relation in :math:`\mathcal{R}`), and message passing in layer
    :math:`\ell` is performed via

    .. math::

        \mathbf{h}^{(\ell)}_v = \bigoplus_{r \in \mathcal{R}}
        f_{\theta}^{(\ell, r)} ( \mathbf{h}^{(\ell - 1)}_v, \{
        \mathbf{h}^{(\ell - 1)}_w : w \in \mathcal{N}^{(r)}(v) \}),

    where :math:`\mathcal{N}^{(r)}(v)` denotes the neighborhood of :math:`v \in
    \mathcal{V}` under relation :math:`r \in \mathcal{R}`, and
    :math:`\bigoplus` denotes the aggregation scheme :attr:`aggr` to use for
    grouping node embeddings generated by different relations
    (:obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"` or :obj:`"mul"`).

    Args:
        module (torch.nn.Module): The homogeneous model to transform.
        metadata (Tuple[List[str], List[Tuple[str, str, str]]]): The metadata
            of the heterogeneous graph, *i.e.* its node and edge types given
            by a list of strings and a list of string triplets, respectively.
            See :meth:`torch_geometric.data.HeteroData.metadata` for more
            information.
        aggr (str, optional): The aggregation scheme to use for grouping node
            embeddings generated by different relations
            (:obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"mul"`). (default: :obj:`"sum"`)
        input_map (Dict[str, str], optional): A dictionary holding information
            about the type of input arguments of :obj:`module.forward`.
            For example, in case :obj:`arg` is a node-level argument, then
            :obj:`input_map['arg'] = 'node'`, and
            :obj:`input_map['arg'] = 'edge'` otherwise.
            In case :obj:`input_map` is not further specified, will try to
            automatically determine the correct type of input arguments.
            (default: :obj:`None`)
        debug (bool, optional): If set to :obj:`True`, will perform
            transformation in debug mode. (default: :obj:`False`)
    """
    transformer = ToHeteroTransformer(module, metadata, aggr, input_map, debug)
    return transformer.transform()


class ToHeteroTransformer(Transformer):

    aggrs = {
        'sum': torch.add,
        # For 'mean' aggregation, we first sum up all feature matrices, and
        # divide by the number of matrices in a later step.
        'mean': torch.add,
        'max': torch.max,
        'min': torch.min,
        'mul': torch.mul,
    }

    def __init__(
        self,
        module: Module,
        metadata: Metadata,
        aggr: str = 'sum',
        input_map: Optional[Dict[str, str]] = None,
        debug: bool = False,
    ):
        super().__init__(module, input_map, debug)

        self.metadata = metadata
        self.aggr = aggr
        assert len(metadata) == 2
        assert len(metadata[0]) > 0 and len(metadata[1]) > 0
        assert aggr in self.aggrs.keys()

        self.validate()

    def validate(self):
        unused_node_types = get_unused_node_types(*self.metadata)
        if len(unused_node_types) > 0:
            warnings.warn(
                f"There exist node types ({unused_node_types}) whose "
                f"representations do not get updated during message passing "
                f"as they do not occur as destination type in any edge type. "
                f"This may lead to unexpected behavior.")

        names = self.metadata[0] + [rel for _, rel, _ in self.metadata[1]]
        for name in names:
            if not name.isidentifier():
                warnings.warn(
                    f"The type '{name}' contains invalid characters which "
                    f"may lead to unexpected behavior. To avoid any issues, "
                    f"ensure that your types only contain letters, numbers "
                    f"and underscores.")

    def placeholder(self, node: Node, target: Any, name: str):
        # Adds a `get` call to the input dictionary for every node-type or
        # edge-type.
        if node.type is not None:
            Type = EdgeType if self.is_edge_level(node) else NodeType
            node.type = Dict[Type, node.type]

        self.graph.inserting_after(node)

        dict_node = self.graph.create_node('call_function', target=get_dict,
                                           args=(node, ), name=f'{name}_dict')
        self.graph.inserting_after(dict_node)

        for key in self.metadata[int(self.is_edge_level(node))]:
            out = self.graph.create_node('call_method', target='get',
                                         args=(dict_node, key, None),
                                         name=f'{name}__{key2str(key)}')
            self.graph.inserting_after(out)

    def get_attr(self, node: Node, target: Any, name: str):
        raise NotImplementedError

    def call_message_passing_module(self, node: Node, target: Any, name: str):
        # Add calls to edge type-wise `MessagePassing` modules and aggregate
        # the outputs to node type-wise embeddings afterwards.

        module = get_submodule(self.module, target)
        check_add_self_loops(module, self.metadata[1])

        # Group edge-wise keys per destination:
        key_name, keys_per_dst = {}, defaultdict(list)
        for key in self.metadata[1]:
            keys_per_dst[key[-1]].append(key)
            key_name[key] = f'{name}__{key[-1]}{len(keys_per_dst[key[-1]])}'

        for dst, keys in dict(keys_per_dst).items():
            # In case there is only a single edge-wise connection, there is no
            # need for any destination-wise aggregation, and we can already set
            # the intermediate variable name to the final output name.
            if len(keys) == 1:
                key_name[keys[0]] = f'{name}__{dst}'
                del keys_per_dst[dst]

        self.graph.inserting_after(node)
        for key in self.metadata[1]:
            args, kwargs = self.map_args_kwargs(node, key)
            out = self.graph.create_node('call_module',
                                         target=f'{target}.{key2str(key)}',
                                         args=args, kwargs=kwargs,
                                         name=key_name[key])
            self.graph.inserting_after(out)

        # Perform destination-wise aggregation.
        # Here, we aggregate in pairs, popping the first two elements of
        # `keys_per_dst` and append the result to the list.
        for dst, keys in keys_per_dst.items():
            queue = deque([key_name[key] for key in keys])
            i = 1
            while len(queue) >= 2:
                key1, key2 = queue.popleft(), queue.popleft()
                args = (self.find_by_name(key1), self.find_by_name(key2))

                new_name = f'{name}__{dst}'
                if self.aggr == 'mean' or len(queue) > 0:
                    new_name = f'{new_name}_{i}'

                out = self.graph.create_node('call_function',
                                             target=self.aggrs[self.aggr],
                                             args=args, name=new_name)
                self.graph.inserting_after(out)
                queue.append(new_name)
                i += 1

            if self.aggr == 'mean':
                key = queue.popleft()
                out = self.graph.create_node(
                    'call_function', target=torch.div,
                    args=(self.find_by_name(key), len(keys_per_dst[dst])),
                    name=f'{name}__{dst}')
                self.graph.inserting_after(out)

    def call_global_pooling_module(self, node: Node, target: Any, name: str):
        # Add calls to node type-wise `GlobalPooling` modules and aggregate
        # the outputs to graph type-wise embeddings afterwards.
        self.graph.inserting_after(node)
        for key in self.metadata[0]:
            args, kwargs = self.map_args_kwargs(node, key)
            out = self.graph.create_node('call_module',
                                         target=f'{target}.{key2str(key)}',
                                         args=args, kwargs=kwargs,
                                         name=f'{node.name}__{key2str(key)}')
            self.graph.inserting_after(out)

        # Perform node-wise aggregation.
        queue = deque(
            [f'{node.name}__{key2str(key)}' for key in self.metadata[0]])
        i = 1
        while len(queue) >= 2:
            key1, key2 = queue.popleft(), queue.popleft()
            args = (self.find_by_name(key1), self.find_by_name(key2))
            out = self.graph.create_node('call_function',
                                         target=self.aggrs[self.aggr],
                                         args=args, name=f'{name}_{i}')
            self.graph.inserting_after(out)
            queue.append(f'{name}_{i}')
            i += 1

        if self.aggr == 'mean':
            key = queue.popleft()
            out = self.graph.create_node(
                'call_function', target=torch.div,
                args=(self.find_by_name(key), len(self.metadata[0])),
                name=f'{name}_{i}')
            self.graph.inserting_after(out)
        self.replace_all_uses_with(node, out)

    def call_module(self, node: Node, target: Any, name: str):
        if self.is_graph_level(node):
            return

        # Add calls to node type-wise or edge type-wise modules.
        self.graph.inserting_after(node)
        for key in self.metadata[int(self.is_edge_level(node))]:
            args, kwargs = self.map_args_kwargs(node, key)
            out = self.graph.create_node('call_module',
                                         target=f'{target}.{key2str(key)}',
                                         args=args, kwargs=kwargs,
                                         name=f'{name}__{key2str(key)}')
            self.graph.inserting_after(out)

    def call_method(self, node: Node, target: Any, name: str):
        if self.is_graph_level(node):
            return

        # Add calls to node type-wise or edge type-wise methods.
        self.graph.inserting_after(node)
        for key in self.metadata[int(self.is_edge_level(node))]:
            args, kwargs = self.map_args_kwargs(node, key)
            out = self.graph.create_node('call_method', target=target,
                                         args=args, kwargs=kwargs,
                                         name=f'{name}__{key2str(key)}')
            self.graph.inserting_after(out)

    def call_function(self, node: Node, target: Any, name: str):
        if self.is_graph_level(node):
            return

        # Add calls to node type-wise or edge type-wise functions.
        self.graph.inserting_after(node)
        for key in self.metadata[int(self.is_edge_level(node))]:
            args, kwargs = self.map_args_kwargs(node, key)
            out = self.graph.create_node('call_function', target=target,
                                         args=args, kwargs=kwargs,
                                         name=f'{name}__{key2str(key)}')
            self.graph.inserting_after(out)

    def output(self, node: Node, target: Any, name: str):
        # Replace the output by dictionaries, holding either node type-wise or
        # edge type-wise data.
        def _recurse(value: Any) -> Any:
            if isinstance(value, Node):
                if self.is_graph_level(value):
                    return value
                return {
                    key: self.find_by_name(f'{value.name}__{key2str(key)}')
                    for key in self.metadata[int(self.is_edge_level(value))]
                }
            elif isinstance(value, dict):
                return {k: _recurse(v) for k, v in value.items()}
            elif isinstance(value, list):
                return [_recurse(v) for v in value]
            elif isinstance(value, tuple):
                return tuple(_recurse(v) for v in value)
            else:
                return value

        if node.type is not None and isinstance(node.args[0], Node):
            output = node.args[0]
            if self.is_node_level(output):
                node.type = Dict[NodeType, node.type]
            elif self.is_edge_level(output):
                node.type = Dict[EdgeType, node.type]
        else:
            node.type = None

        node.args = (_recurse(node.args[0]), )

    def init_submodule(self, module: Module, target: str) -> Module:
        # Replicate each module for each node type or edge type.
        has_node_level_target = bool(
            self.find_by_target(f'{target}.{key2str(self.metadata[0][0])}'))
        has_edge_level_target = bool(
            self.find_by_target(f'{target}.{key2str(self.metadata[1][0])}'))

        if not has_node_level_target and not has_edge_level_target:
            return module

        module_dict = torch.nn.ModuleDict()
        for key in self.metadata[int(has_edge_level_target)]:
            module_dict[key2str(key)] = copy.deepcopy(module)
            if len(self.metadata[int(has_edge_level_target)]) <= 1:
                continue
            if hasattr(module, 'reset_parameters'):
                module_dict[key2str(key)].reset_parameters()
            elif sum([
                    is_uninitialized_parameter(p) or p.numel()
                    for p in module.parameters()
            ]) > 0:
                warnings.warn(
                    f"'{target}' will be duplicated, but its parameters "
                    f"cannot be reset. To suppress this warning, add a "
                    f"'reset_parameters()' method to '{target}'")

        return module_dict

    # Helper methods ##########################################################

    def map_args_kwargs(self, node: Node,
                        key: Union[NodeType, EdgeType]) -> Tuple[Tuple, Dict]:
        def _recurse(value: Any) -> Any:
            if isinstance(value, Node):
                out = self.find_by_name(f'{value.name}__{key2str(key)}')
                if out is not None:
                    return out
                elif isinstance(key, tuple) and key[0] == key[-1]:
                    name = f'{value.name}__{key2str(key[0])}'
                    return self.find_by_name(name)
                elif isinstance(key, tuple) and key[0] != key[-1]:
                    return (
                        self.find_by_name(f'{value.name}__{key2str(key[0])}'),
                        self.find_by_name(f'{value.name}__{key2str(key[-1])}'),
                    )
                else:
                    raise ValueError(f"Cannot generate a graph node '{node}' "
                                     f"for type '{key}' since it does not "
                                     f"exist. Please make sure that all "
                                     f"node types get updated during message "
                                     f"passing.")
            elif isinstance(value, dict):
                return {k: _recurse(v) for k, v in value.items()}
            elif isinstance(value, list):
                return [_recurse(v) for v in value]
            elif isinstance(value, tuple):
                return tuple(_recurse(v) for v in value)
            else:
                return value

        args = tuple(_recurse(v) for v in node.args)
        kwargs = {k: _recurse(v) for k, v in node.kwargs.items()}
        return args, kwargs


def key2str(key: Union[NodeType, EdgeType]) -> str:
    key = '__'.join(key) if isinstance(key, tuple) else key
    return key.replace(' ', '_').replace('-', '_').replace(':', '_')
pytorch/torch_geometric/nn/to_hetero_with_bases_transformer.py000066400000000000000000000550771444262304300255520ustar00rootroot00000000000000import copy
import warnings
from typing import Any, Dict, List, Optional, Union

import torch
from torch import Tensor
from torch.nn import Module, Parameter

from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.dense import Linear
from torch_geometric.nn.fx import Transformer
from torch_geometric.typing import EdgeType, Metadata, NodeType, SparseTensor
from torch_geometric.utils.hetero import get_unused_node_types

try:
    from torch.fx import Graph, GraphModule, Node
except (ImportError, ModuleNotFoundError, AttributeError):
    GraphModule, Graph, Node = 'GraphModule', 'Graph', 'Node'


def to_hetero_with_bases(module: Module, metadata: Metadata, num_bases: int,
                         in_channels: Optional[Dict[str, int]] = None,
                         input_map: Optional[Dict[str, str]] = None,
                         debug: bool = False) -> GraphModule:
    r"""Converts a homogeneous GNN model into its heterogeneous equivalent
    via the basis-decomposition technique introduced in the
    `"Modeling Relational Data with Graph Convolutional Networks"
    `_ paper:
    For this, the heterogeneous graph is mapped to a typed homogeneous graph,
    in which its feature representations are aligned and grouped to a single
    representation.
    All GNN layers inside the model will then perform message passing via
    basis-decomposition regularization.
    This transformation is especially useful in highly multi-relational data,
    such that the number of parameters no longer depend on the number of
    relations of the input graph:

    .. code-block:: python

        import torch
        from torch_geometric.nn import SAGEConv, to_hetero_with_bases

        class GNN(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.conv1 = SAGEConv((16, 16), 32)
                self.conv2 = SAGEConv((32, 32), 32)

            def forward(self, x, edge_index):
                x = self.conv1(x, edge_index).relu()
                x = self.conv2(x, edge_index).relu()
                return x

        model = GNN()

        node_types = ['paper', 'author']
        edge_types = [
            ('paper', 'cites', 'paper'),
            ('paper', 'written_by', 'author'),
            ('author', 'writes', 'paper'),
        ]
        metadata = (node_types, edge_types)

        model = to_hetero_with_bases(model, metadata, num_bases=3,
                                     in_channels={'x': 16})
        model(x_dict, edge_index_dict)

    where :obj:`x_dict` and :obj:`edge_index_dict` denote dictionaries that
    hold node features and edge connectivity information for each node type and
    edge type, respectively.
    In case :obj:`in_channels` is given for a specific input argument, its
    heterogeneous feature information is first aligned to the given
    dimensionality.

    The below illustration shows the original computation graph of the
    homogeneous model on the left, and the newly obtained computation graph of
    the regularized heterogeneous model on the right:

    .. figure:: ../_figures/to_hetero_with_bases.svg
      :align: center
      :width: 90%

      Transforming a model via :func:`to_hetero_with_bases`.

    Here, each :class:`~torch_geometric.nn.conv.MessagePassing` instance
    :math:`f_{\theta}^{(\ell)}` is duplicated :obj:`num_bases` times and
    stored in a set :math:`\{ f_{\theta}^{(\ell, b)} : b \in \{ 1, \ldots, B \}
    \}` (one instance for each basis in
    :obj:`num_bases`), and message passing in layer :math:`\ell` is performed
    via

    .. math::

        \mathbf{h}^{(\ell)}_v = \sum_{r \in \mathcal{R}} \sum_{b=1}^B
        f_{\theta}^{(\ell, b)} ( \mathbf{h}^{(\ell - 1)}_v, \{
        a^{(\ell)}_{r, b} \cdot \mathbf{h}^{(\ell - 1)}_w :
        w \in \mathcal{N}^{(r)}(v) \}),

    where :math:`\mathcal{N}^{(r)}(v)` denotes the neighborhood of :math:`v \in
    \mathcal{V}` under relation :math:`r \in \mathcal{R}`.
    Notably, only the trainable basis coefficients :math:`a^{(\ell)}_{r, b}`
    depend on the relations in :math:`\mathcal{R}`.

    Args:
        module (torch.nn.Module): The homogeneous model to transform.
        metadata (Tuple[List[str], List[Tuple[str, str, str]]]): The metadata
            of the heterogeneous graph, *i.e.* its node and edge types given
            by a list of strings and a list of string triplets, respectively.
            See :meth:`torch_geometric.data.HeteroData.metadata` for more
            information.
        num_bases (int): The number of bases to use.
        in_channels (Dict[str, int], optional): A dictionary holding
            information about the desired input feature dimensionality of
            input arguments of :obj:`module.forward`.
            In case :obj:`in_channels` is given for a specific input argument,
            its heterogeneous feature information is first aligned to the given
            dimensionality.
            This allows handling of node and edge features with varying feature
            dimensionality across different types. (default: :obj:`None`)
        input_map (Dict[str, str], optional): A dictionary holding information
            about the type of input arguments of :obj:`module.forward`.
            For example, in case :obj:`arg` is a node-level argument, then
            :obj:`input_map['arg'] = 'node'`, and
            :obj:`input_map['arg'] = 'edge'` otherwise.
            In case :obj:`input_map` is not further specified, will try to
            automatically determine the correct type of input arguments.
            (default: :obj:`None`)
        debug (bool, optional): If set to :obj:`True`, will perform
            transformation in debug mode. (default: :obj:`False`)
    """
    transformer = ToHeteroWithBasesTransformer(module, metadata, num_bases,
                                               in_channels, input_map, debug)
    return transformer.transform()


class ToHeteroWithBasesTransformer(Transformer):
    def __init__(
        self,
        module: Module,
        metadata: Metadata,
        num_bases: int,
        in_channels: Optional[Dict[str, int]] = None,
        input_map: Optional[Dict[str, str]] = None,
        debug: bool = False,
    ):
        super().__init__(module, input_map, debug)

        self.metadata = metadata
        self.num_bases = num_bases
        self.in_channels = in_channels or {}
        assert len(metadata) == 2
        assert len(metadata[0]) > 0 and len(metadata[1]) > 0

        self.validate()

        # Compute IDs for each node and edge type:
        self.node_type2id = {k: i for i, k in enumerate(metadata[0])}
        self.edge_type2id = {k: i for i, k in enumerate(metadata[1])}

    def validate(self):
        unused_node_types = get_unused_node_types(*self.metadata)
        if len(unused_node_types) > 0:
            warnings.warn(
                f"There exist node types ({unused_node_types}) whose "
                f"representations do not get updated during message passing "
                f"as they do not occur as destination type in any edge type. "
                f"This may lead to unexpected behavior.")

        names = self.metadata[0] + [rel for _, rel, _ in self.metadata[1]]
        for name in names:
            if not name.isidentifier():
                warnings.warn(
                    f"The type '{name}' contains invalid characters which "
                    f"may lead to unexpected behavior. To avoid any issues, "
                    f"ensure that your types only contain letters, numbers "
                    f"and underscores.")

    def transform(self) -> GraphModule:
        self._node_offset_dict_initialized = False
        self._edge_offset_dict_initialized = False
        self._edge_type_initialized = False
        out = super().transform()
        del self._node_offset_dict_initialized
        del self._edge_offset_dict_initialized
        del self._edge_type_initialized
        return out

    def placeholder(self, node: Node, target: Any, name: str):
        if node.type is not None:
            Type = EdgeType if self.is_edge_level(node) else NodeType
            node.type = Dict[Type, node.type]

        out = node

        # Create `node_offset_dict` and `edge_offset_dict` dictionaries in case
        # they are not yet initialized. These dictionaries hold the cumulated
        # sizes used to create a unified graph representation and to split the
        # output data.
        if self.is_edge_level(node) and not self._edge_offset_dict_initialized:
            self.graph.inserting_after(out)
            out = self.graph.create_node('call_function',
                                         target=get_edge_offset_dict,
                                         args=(node, self.edge_type2id),
                                         name='edge_offset_dict')
            self._edge_offset_dict_initialized = True

        elif not self._node_offset_dict_initialized:
            self.graph.inserting_after(out)
            out = self.graph.create_node('call_function',
                                         target=get_node_offset_dict,
                                         args=(node, self.node_type2id),
                                         name='node_offset_dict')
            self._node_offset_dict_initialized = True

        # Create a `edge_type` tensor used as input to `HeteroBasisConv`:
        if self.is_edge_level(node) and not self._edge_type_initialized:
            self.graph.inserting_after(out)
            out = self.graph.create_node('call_function', target=get_edge_type,
                                         args=(node, self.edge_type2id),
                                         name='edge_type')
            self._edge_type_initialized = True

        # Add `Linear` operation to align features to the same dimensionality:
        if name in self.in_channels:
            self.graph.inserting_after(out)
            out = self.graph.create_node('call_module',
                                         target=f'align_lin__{name}',
                                         args=(node, ),
                                         name=f'{name}__aligned')
            self._state[out.name] = self._state[name]

            lin = LinearAlign(self.metadata[int(self.is_edge_level(node))],
                              self.in_channels[name])
            setattr(self.module, f'align_lin__{name}', lin)

        # Perform grouping of type-wise values into a single tensor:
        if self.is_edge_level(node):
            self.graph.inserting_after(out)
            out = self.graph.create_node(
                'call_function', target=group_edge_placeholder,
                args=(out if name in self.in_channels else node,
                      self.edge_type2id,
                      self.find_by_name('node_offset_dict')),
                name=f'{name}__grouped')
            self._state[out.name] = 'edge'

        else:
            self.graph.inserting_after(out)
            out = self.graph.create_node(
                'call_function', target=group_node_placeholder,
                args=(out if name in self.in_channels else node,
                      self.node_type2id), name=f'{name}__grouped')
            self._state[out.name] = 'node'

        self.replace_all_uses_with(node, out)

    def call_message_passing_module(self, node: Node, target: Any, name: str):
        # Call the `HeteroBasisConv` wrapper instead instead of a single
        # message passing layer. We need to inject the `edge_type` as first
        # argument in order to do so.
        node.args = (self.find_by_name('edge_type'), ) + node.args

    def output(self, node: Node, target: Any, name: str):
        # Split the output to dictionaries, holding either node type-wise or
        # edge type-wise data.
        def _recurse(value: Any) -> Any:
            if isinstance(value, Node) and self.is_edge_level(value):
                self.graph.inserting_before(node)
                return self.graph.create_node(
                    'call_function', target=split_output,
                    args=(value, self.find_by_name('edge_offset_dict')),
                    name=f'{value.name}__split')

                pass
            elif isinstance(value, Node):
                self.graph.inserting_before(node)
                return self.graph.create_node(
                    'call_function', target=split_output,
                    args=(value, self.find_by_name('node_offset_dict')),
                    name=f'{value.name}__split')

            elif isinstance(value, dict):
                return {k: _recurse(v) for k, v in value.items()}
            elif isinstance(value, list):
                return [_recurse(v) for v in value]
            elif isinstance(value, tuple):
                return tuple(_recurse(v) for v in value)
            else:
                return value

        if node.type is not None and isinstance(node.args[0], Node):
            output = node.args[0]
            Type = EdgeType if self.is_edge_level(output) else NodeType
            node.type = Dict[Type, node.type]
        else:
            node.type = None

        node.args = (_recurse(node.args[0]), )

    def init_submodule(self, module: Module, target: str) -> Module:
        if not isinstance(module, MessagePassing):
            return module

        # Replace each `MessagePassing` module by a `HeteroBasisConv` wrapper:
        return HeteroBasisConv(module, len(self.metadata[1]), self.num_bases)


###############################################################################


class HeteroBasisConv(torch.nn.Module):
    # A wrapper layer that applies the basis-decomposition technique to a
    # heterogeneous graph.
    def __init__(self, module: MessagePassing, num_relations: int,
                 num_bases: int):
        super().__init__()

        self.num_relations = num_relations
        self.num_bases = num_bases

        # We make use of a post-message computation hook to inject the
        # basis re-weighting for each individual edge type.
        # This currently requires us to set `conv.fuse = False`, which leads
        # to a materialization of messages.
        def hook(module, inputs, output):
            assert isinstance(module._edge_type, Tensor)
            if module._edge_type.size(0) != output.size(0):
                raise ValueError(
                    f"Number of messages ({output.size(0)}) does not match "
                    f"with the number of original edges "
                    f"({module._edge_type.size(0)}). Does your message "
                    f"passing layer create additional self-loops? Try to "
                    f"remove them via 'add_self_loops=False'")
            weight = module.edge_type_weight.view(-1)[module._edge_type]
            weight = weight.view([-1] + [1] * (output.dim() - 1))
            return weight * output

        params = list(module.parameters())
        device = params[0].device if len(params) > 0 else 'cpu'

        self.convs = torch.nn.ModuleList()
        for _ in range(num_bases):
            conv = copy.deepcopy(module)
            conv.fuse = False  # Disable `message_and_aggregate` functionality.
            # We learn a single scalar weight for each individual edge type,
            # which is used to weight the output message based on edge type:
            conv.edge_type_weight = Parameter(
                torch.Tensor(1, num_relations, device=device))
            conv.register_message_forward_hook(hook)
            self.convs.append(conv)

        if self.num_bases > 1:
            self.reset_parameters()

    def reset_parameters(self):
        for conv in self.convs:
            if hasattr(conv, 'reset_parameters'):
                conv.reset_parameters()
            elif sum([p.numel() for p in conv.parameters()]) > 0:
                warnings.warn(
                    f"'{conv}' will be duplicated, but its parameters cannot "
                    f"be reset. To suppress this warning, add a "
                    f"'reset_parameters()' method to '{conv}'")
            torch.nn.init.xavier_uniform_(conv.edge_type_weight)

    def forward(self, edge_type: Tensor, *args, **kwargs) -> Tensor:
        out = None
        # Call message passing modules and perform aggregation:
        for conv in self.convs:
            conv._edge_type = edge_type
            res = conv(*args, **kwargs)
            del conv._edge_type
            out = res if out is None else out.add_(res)
        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_relations='
                f'{self.num_relations}, num_bases={self.num_bases})')


class LinearAlign(torch.nn.Module):
    # Aligns representions to the same dimensionality. Note that this will
    # create lazy modules, and as such requires a forward pass in order to
    # initialize parameters.
    def __init__(self, keys: List[Union[NodeType, EdgeType]],
                 out_channels: int):
        super().__init__()
        self.out_channels = out_channels
        self.lins = torch.nn.ModuleDict()
        for key in keys:
            self.lins[key2str(key)] = Linear(-1, out_channels, bias=False)

    def forward(
        self, x_dict: Dict[Union[NodeType, EdgeType], Tensor]
    ) -> Dict[Union[NodeType, EdgeType], Tensor]:
        return {key: self.lins[key2str(key)](x) for key, x in x_dict.items()}

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_relations={len(self.lins)}, '
                f'out_channels={self.out_channels})')


###############################################################################

# These methods are used in order to receive the cumulated sizes of input
# dictionaries. We make use of them for creating a unified homogeneous graph
# representation, as well as to split the final output data once again.


def get_node_offset_dict(
    input_dict: Dict[NodeType, Union[Tensor, SparseTensor]],
    type2id: Dict[NodeType, int],
) -> Dict[NodeType, int]:
    cumsum = 0
    out: Dict[NodeType, int] = {}
    for key in type2id.keys():
        out[key] = cumsum
        cumsum += input_dict[key].size(0)
    return out


def get_edge_offset_dict(
    input_dict: Dict[EdgeType, Union[Tensor, SparseTensor]],
    type2id: Dict[EdgeType, int],
) -> Dict[EdgeType, int]:
    cumsum = 0
    out: Dict[EdgeType, int] = {}
    for key in type2id.keys():
        out[key] = cumsum
        value = input_dict[key]
        if isinstance(value, SparseTensor):
            cumsum += value.nnz()
        elif value.dtype == torch.long and value.size(0) == 2:
            cumsum += value.size(-1)
        else:
            cumsum += value.size(0)
    return out


###############################################################################

# This method computes the edge type of the final homogeneous graph
# representation. It will be used in the `HeteroBasisConv` wrapper.


def get_edge_type(
    input_dict: Dict[EdgeType, Union[Tensor, SparseTensor]],
    type2id: Dict[EdgeType, int],
) -> Tensor:

    inputs = [input_dict[key] for key in type2id.keys()]
    outs = []

    for i, value in enumerate(inputs):
        if value.size(0) == 2 and value.dtype == torch.long:  # edge_index
            out = value.new_full((value.size(-1), ), i, dtype=torch.long)
        elif isinstance(value, SparseTensor):
            out = torch.full((value.nnz(), ), i, dtype=torch.long,
                             device=value.device())
        else:
            out = value.new_full((value.size(0), ), i, dtype=torch.long)
        outs.append(out)

    return outs[0] if len(outs) == 1 else torch.cat(outs, dim=0)


###############################################################################

# These methods are used to group the individual type-wise components into a
# unfied single representation.


def group_node_placeholder(input_dict: Dict[NodeType, Tensor],
                           type2id: Dict[NodeType, int]) -> Tensor:

    inputs = [input_dict[key] for key in type2id.keys()]
    return inputs[0] if len(inputs) == 1 else torch.cat(inputs, dim=0)


def group_edge_placeholder(
    input_dict: Dict[EdgeType, Union[Tensor, SparseTensor]],
    type2id: Dict[EdgeType, int],
    offset_dict: Dict[NodeType, int] = None,
) -> Union[Tensor, SparseTensor]:

    inputs = [input_dict[key] for key in type2id.keys()]

    if len(inputs) == 1:
        return inputs[0]

    # In case of grouping a graph connectivity tensor `edge_index` or `adj_t`,
    # we need to increment its indices:
    elif inputs[0].size(0) == 2 and inputs[0].dtype == torch.long:
        if offset_dict is None:
            raise AttributeError(
                "Can not infer node-level offsets. Please ensure that there "
                "exists a node-level argument before the 'edge_index' "
                "argument in your forward header.")

        outputs = []
        for value, (src_type, _, dst_type) in zip(inputs, type2id):
            value = value.clone()
            value[0, :] += offset_dict[src_type]
            value[1, :] += offset_dict[dst_type]
            outputs.append(value)

        return torch.cat(outputs, dim=-1)

    elif isinstance(inputs[0], SparseTensor):
        if offset_dict is None:
            raise AttributeError(
                "Can not infer node-level offsets. Please ensure that there "
                "exists a node-level argument before the 'SparseTensor' "
                "argument in your forward header.")

        # For grouping a list of SparseTensors, we convert them into a
        # unified `edge_index` representation in order to avoid conflicts
        # induced by re-shuffling the data.
        rows, cols = [], []
        for value, (src_type, _, dst_type) in zip(inputs, type2id):
            col, row, value = value.coo()
            assert value is None
            rows.append(row + offset_dict[src_type])
            cols.append(col + offset_dict[dst_type])

        row = torch.cat(rows, dim=0)
        col = torch.cat(cols, dim=0)
        return torch.stack([row, col], dim=0)

    else:
        return torch.cat(inputs, dim=0)


###############################################################################

# This method is used to split the output tensors into individual type-wise
# components:


def split_output(
    output: Tensor,
    offset_dict: Union[Dict[NodeType, int], Dict[EdgeType, int]],
) -> Union[Dict[NodeType, Tensor], Dict[EdgeType, Tensor]]:

    cumsums = list(offset_dict.values()) + [output.size(0)]
    sizes = [cumsums[i + 1] - cumsums[i] for i in range(len(offset_dict))]
    outputs = output.split(sizes)
    return {key: output for key, output in zip(offset_dict, outputs)}


###############################################################################


def key2str(key: Union[NodeType, EdgeType]) -> str:
    key = '__'.join(key) if isinstance(key, tuple) else key
    return key.replace(' ', '_').replace('-', '_').replace(':', '_')
pytorch/torch_geometric/nn/unpool/000077500000000000000000000000001444262304300176545ustar00rootroot00000000000000pytorch/torch_geometric/nn/unpool/__init__.py000066400000000000000000000001461444262304300217660ustar00rootroot00000000000000from .knn_interpolate import knn_interpolate

__all__ = [
    'knn_interpolate',
]

classes = __all__
pytorch/torch_geometric/nn/unpool/knn_interpolate.py000066400000000000000000000050321444262304300234220ustar00rootroot00000000000000import torch

from torch_geometric.nn import knn
from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter


def knn_interpolate(x: torch.Tensor, pos_x: torch.Tensor, pos_y: torch.Tensor,
                    batch_x: OptTensor = None, batch_y: OptTensor = None,
                    k: int = 3, num_workers: int = 1):
    r"""The k-NN interpolation from the `"PointNet++: Deep Hierarchical
    Feature Learning on Point Sets in a Metric Space"
    `_ paper.
    For each point :math:`y` with position :math:`\mathbf{p}(y)`, its
    interpolated features :math:`\mathbf{f}(y)` are given by

    .. math::
        \mathbf{f}(y) = \frac{\sum_{i=1}^k w(x_i) \mathbf{f}(x_i)}{\sum_{i=1}^k
        w(x_i)} \textrm{, where } w(x_i) = \frac{1}{d(\mathbf{p}(y),
        \mathbf{p}(x_i))^2}

    and :math:`\{ x_1, \ldots, x_k \}` denoting the :math:`k` nearest points
    to :math:`y`.

    Args:
        x (torch.Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{N \times F}`.
        pos_x (torch.Tensor): Node position matrix
            :math:`\in \mathbb{R}^{N \times d}`.
        pos_y (torch.Tensor): Upsampled node position matrix
            :math:`\in \mathbb{R}^{M \times d}`.
        batch_x (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b_x} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each node from :math:`\mathbf{X}` to a specific example.
            (default: :obj:`None`)
        batch_y (torch.Tensor, optional): Batch vector
            :math:`\mathbf{b_y} \in {\{ 0, \ldots, B-1\}}^N`, which assigns
            each node from :math:`\mathbf{Y}` to a specific example.
            (default: :obj:`None`)
        k (int, optional): Number of neighbors. (default: :obj:`3`)
        num_workers (int, optional): Number of workers to use for computation.
            Has no effect in case :obj:`batch_x` or :obj:`batch_y` is not
            :obj:`None`, or the input lies on the GPU. (default: :obj:`1`)
    """

    with torch.no_grad():
        assign_index = knn(pos_x, pos_y, k, batch_x=batch_x, batch_y=batch_y,
                           num_workers=num_workers)
        y_idx, x_idx = assign_index[0], assign_index[1]
        diff = pos_x[x_idx] - pos_y[y_idx]
        squared_distance = (diff * diff).sum(dim=-1, keepdim=True)
        weights = 1.0 / torch.clamp(squared_distance, min=1e-16)

    y = scatter(x[x_idx] * weights, y_idx, 0, pos_y.size(0), reduce='sum')
    y = y / scatter(weights, y_idx, 0, pos_y.size(0), reduce='sum')

    return y
pytorch/torch_geometric/profile/000077500000000000000000000000001444262304300173655ustar00rootroot00000000000000pytorch/torch_geometric/profile/__init__.py000066400000000000000000000014431444262304300215000ustar00rootroot00000000000000from .profile import profileit, timeit, get_stats_summary
from .profile import (
    trace_handler,
    print_time_total,
    rename_profile_file,
    torch_profile,
)
from .utils import count_parameters
from .utils import get_model_size
from .utils import get_data_size
from .utils import get_cpu_memory_from_gc
from .utils import get_gpu_memory_from_gc
from .utils import get_gpu_memory_from_nvidia_smi
from .benchmark import benchmark

__all__ = [
    'profileit',
    'timeit',
    'get_stats_summary',
    'trace_handler',
    'print_time_total',
    'rename_profile_file',
    'torch_profile',
    'count_parameters',
    'get_model_size',
    'get_data_size',
    'get_cpu_memory_from_gc',
    'get_gpu_memory_from_gc',
    'get_gpu_memory_from_nvidia_smi',
    'benchmark',
]

classes = __all__
pytorch/torch_geometric/profile/benchmark.py000066400000000000000000000116521444262304300216760ustar00rootroot00000000000000import time
from typing import Any, Callable, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.utils import is_torch_sparse_tensor


def require_grad(x: Any, requires_grad: bool = True) -> Any:
    if (isinstance(x, Tensor) and x.is_floating_point()
            and not is_torch_sparse_tensor(x)):
        return x.detach().requires_grad_(requires_grad)
    elif isinstance(x, list):
        return [require_grad(v, requires_grad) for v in x]
    elif isinstance(x, tuple):
        return tuple(require_grad(v, requires_grad) for v in x)
    elif isinstance(x, dict):
        return {k: require_grad(v, requires_grad) for k, v in x.items()}
    return x


def benchmark(
    funcs: List[Callable],
    args: Union[Tuple[Any], List[Tuple[Any]]],
    num_steps: int,
    func_names: Optional[List[str]] = None,
    num_warmups: int = 10,
    backward: bool = False,
    per_step: bool = False,
    progress_bar: bool = False,
):
    r"""Benchmark a list of functions :obj:`funcs` that receive the same set
    of arguments :obj:`args`.

    Args:
        funcs ([Callable]): The list of functions to benchmark.
        args ((Any, ) or [(Any, )]): The arguments to pass to the functions.
            Can be a list of arguments for each function in :obj:`funcs` in
            case their headers differ.
        num_steps (int): The number of steps to run the benchmark.
        func_names ([str], optional): The names of the functions. If not given,
            will try to infer the name from the function itself.
            (default: :obj:`None`)
        num_warmups (int, optional): The number of warmup steps.
            (default: :obj:`10`)
        backward (bool, optional): If set to :obj:`True`, will benchmark both
            forward and backward passes. (default: :obj:`False`)
        per_step (bool, optional): If set to :obj:`True`, will report runtimes
            per step. (default: :obj:`False`)
        progress_bar (bool, optional): If set to :obj:`True`, will print a
            progress bar during benchmarking. (default: :obj:`False`)
    """
    from tabulate import tabulate

    if num_steps <= 0:
        raise ValueError(f"'num_steps' must be a positive integer "
                         f"(got {num_steps})")

    if num_warmups <= 0:
        raise ValueError(f"'num_warmups' must be a positive integer "
                         f"(got {num_warmups})")

    if func_names is None:
        func_names = [get_func_name(func) for func in funcs]

    if len(funcs) != len(func_names):
        raise ValueError(f"Length of 'funcs' (got {len(funcs)}) and "
                         f"'func_names' (got {len(func_names)}) must be equal")

    # Zero-copy `args` for each function (if necessary):
    args_list = [args] * len(funcs) if isinstance(args, tuple) else args

    iterator = zip(funcs, args_list, func_names)
    if progress_bar:
        from tqdm import tqdm
        iterator = tqdm(iterator, total=len(funcs))

    ts: List[List[str]] = []
    for func, args, name in iterator:
        t_forward = t_backward = 0
        for i in range(num_warmups + num_steps):
            args = require_grad(args, backward)

            if torch.cuda.is_available():
                torch.cuda.synchronize()
            t_start = time.perf_counter()

            out = func(*args)

            if torch.cuda.is_available():
                torch.cuda.synchronize()
            if i >= num_warmups:
                t_forward += time.perf_counter() - t_start

            if backward:
                if isinstance(out, (tuple, list)):
                    out = sum(o.sum() for o in out if isinstance(o, Tensor))
                elif isinstance(out, dict):
                    out = out.values()
                    out = sum(o.sum() for o in out if isinstance(o, Tensor))

                out_grad = torch.randn_like(out)
                t_start = time.perf_counter()

                out.backward(out_grad)

                if torch.cuda.is_available():
                    torch.cuda.synchronize()
                if i >= num_warmups:
                    t_backward += time.perf_counter() - t_start

        if per_step:
            ts.append([name, f'{t_forward/num_steps:.6f}s'])
        else:
            ts.append([name, f'{t_forward:.4f}s'])
        if backward:
            if per_step:
                ts[-1].append(f'{t_backward/num_steps:.6f}s')
                ts[-1].append(f'{(t_forward + t_backward)/num_steps:.6f}s')
            else:
                ts[-1].append(f'{t_backward:.4f}s')
                ts[-1].append(f'{t_forward + t_backward:.4f}s')

    header = ['Name', 'Forward']
    if backward:
        header.extend(['Backward', 'Total'])

    print(tabulate(ts, headers=header, tablefmt='psql'))


def get_func_name(func: Callable) -> str:
    if hasattr(func, '__name__'):
        return func.__name__
    elif hasattr(func, '__class__'):
        return func.__class__.__name__
    raise ValueError("Could not infer name for function '{func}'")
pytorch/torch_geometric/profile/profile.py000066400000000000000000000241741444262304300214070ustar00rootroot00000000000000import os
import pathlib
import time
from contextlib import ContextDecorator, contextmanager
from typing import Any, List, NamedTuple, Tuple

import torch
from torch.autograd.profiler import EventList
from torch.profiler import ProfilerActivity, profile

from torch_geometric.profile.utils import (
    byte_to_megabyte,
    get_gpu_memory_from_nvidia_smi,
)


class Stats(NamedTuple):
    time: float
    max_allocated_cuda: float
    max_reserved_cuda: float
    max_active_cuda: float
    nvidia_smi_free_cuda: float
    nvidia_smi_used_cuda: float


class StatsSummary(NamedTuple):
    time_mean: float
    time_std: float
    max_allocated_cuda: float
    max_reserved_cuda: float
    max_active_cuda: float
    min_nvidia_smi_free_cuda: float
    max_nvidia_smi_used_cuda: float


def profileit():  # pragma: no cover
    r"""A decorator to facilitate profiling a function, *e.g.*, obtaining
    training runtime and memory statistics of a specific model on a specific
    dataset.
    Returns a :obj:`Stats` object with the attributes :obj:`time`,
    :obj:`max_active_cuda`, :obj:`max_reserved_cuda`, :obj:`max_active_cuda`,
    :obj:`nvidia_smi_free_cuda`, :obj:`nvidia_smi_used_cuda`.

    .. code-block:: python

        @profileit()
        def train(model, optimizer, x, edge_index, y):
            optimizer.zero_grad()
            out = model(x, edge_index)
            loss = criterion(out, y)
            loss.backward()
            optimizer.step()
            return float(loss)

        loss, stats = train(model, x, edge_index, y)
    """
    def decorator(func):
        def wrapper(*args, **kwargs) -> Tuple[Any, Stats]:
            from pytorch_memlab import LineProfiler

            model = args[0]
            if not isinstance(model, torch.nn.Module):
                raise AttributeError(
                    'First argument for profiling needs to be torch.nn.Module')

            device = None
            for arg in list(args) + list(kwargs.values()):
                if isinstance(arg, torch.Tensor):
                    device = arg.get_device()
                    break
            if device is None:
                raise AttributeError(
                    "Could not infer CUDA device from the args in the "
                    "function being profiled")
            if device == -1:
                raise RuntimeError(
                    "The profiling decorator does not support profiling "
                    "on non CUDA devices")

            # Init `pytorch_memlab` for analyzing the model forward pass:
            line_profiler = LineProfiler(target_gpu=device)
            line_profiler.enable()
            line_profiler.add_function(args[0].forward)

            start = torch.cuda.Event(enable_timing=True)
            end = torch.cuda.Event(enable_timing=True)
            start.record()

            out = func(*args, **kwargs)

            end.record()
            torch.cuda.synchronize()
            time = start.elapsed_time(end) / 1000

            # Get the global memory statistics collected by `pytorch_memlab`:
            memlab = read_from_memlab(line_profiler)
            max_allocated_cuda, max_reserved_cuda, max_active_cuda = memlab
            line_profiler.disable()

            # Get additional information from `nvidia-smi`:
            free_cuda, used_cuda = get_gpu_memory_from_nvidia_smi(
                device=device)

            stats = Stats(time, max_allocated_cuda, max_reserved_cuda,
                          max_active_cuda, free_cuda, used_cuda)

            return out, stats

        return wrapper

    return decorator


class timeit(ContextDecorator):
    r"""A context decorator to facilitate timing a function, *e.g.*, obtaining
    the runtime of a specific model on a specific dataset.

    .. code-block:: python

        @torch.no_grad()
        def test(model, x, edge_index):
            return model(x, edge_index)

        with timeit() as t:
            z = test(model, x, edge_index)
        time = t.duration

    Args:
        log (bool, optional): If set to :obj:`False`, will not log any runtime
            to the console. (default: :obj:`True`)
        avg_time_divisor (int, optional): If set to a value greater than
            :obj:`1`, will divide the total time by this value. Useful for
            calculating the average of runtimes within a for-loop.
            (default: :obj:`0`)
    """
    def __init__(self, log: bool = True, avg_time_divisor: int = 0):
        self.log = log
        self.avg_time_divisor = avg_time_divisor

    def __enter__(self):
        if torch.cuda.is_available():
            torch.cuda.synchronize()
        self.t_start = time.time()
        return self

    def __exit__(self, *args):
        if torch.cuda.is_available():
            torch.cuda.synchronize()
        self.t_end = time.time()
        self.duration = self.t_end - self.t_start
        if self.avg_time_divisor > 1:
            self.duration = self.duration / self.avg_time_divisor
        if self.log:  # pragma: no cover
            print(f'Time: {self.duration:.8f}s', flush=True)

    def reset(self):
        r"""Prints the duration and resets current timer."""
        if self.t_start is None:
            raise RuntimeError("Timer wasn't started.")
        else:
            self.__exit__()
            self.__enter__()


def get_stats_summary(stats_list: List[Stats]):  # pragma: no cover
    r"""Creates a summary of collected runtime and memory statistics.
    Returns a :obj:`StatsSummary` object with the attributes :obj:`time_mean`,
    :obj:`time_std`,
    :obj:`max_active_cuda`, :obj:`max_reserved_cuda`, :obj:`max_active_cuda`,
    :obj:`min_nvidia_smi_free_cuda`, :obj:`max_nvidia_smi_used_cuda`.

    Args:
        stats_list (List[Stats]): A list of :obj:`Stats` objects, as returned
            by :meth:`~torch_geometric.profile.profileit`.
    """
    return StatsSummary(
        time_mean=float(torch.tensor([s.time for s in stats_list]).mean()),
        time_std=float(torch.tensor([s.time for s in stats_list]).std()),
        max_allocated_cuda=max([s.max_allocated_cuda for s in stats_list]),
        max_reserved_cuda=max([s.max_reserved_cuda for s in stats_list]),
        max_active_cuda=max([s.max_active_cuda for s in stats_list]),
        min_nvidia_smi_free_cuda=min(
            [s.nvidia_smi_free_cuda for s in stats_list]),
        max_nvidia_smi_used_cuda=max(
            [s.nvidia_smi_used_cuda for s in stats_list]),
    )


###############################################################################


def read_from_memlab(line_profiler: Any) -> List[float]:  # pragma: no cover
    from pytorch_memlab.line_profiler.line_records import LineRecords

    # See: https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_stats

    track_stats = [  # Different statistic can be collected as needed.
        'allocated_bytes.all.peak',
        'reserved_bytes.all.peak',
        'active_bytes.all.peak',
    ]

    records = LineRecords(line_profiler._raw_line_records,
                          line_profiler._code_infos)
    stats = records.display(None, track_stats)._line_records
    return [byte_to_megabyte(x) for x in stats.values.max(axis=0).tolist()]


def trace_handler(p):
    print_time_total(p)
    profile_dir = str(pathlib.Path.cwd()) + '/'
    timeline_file = profile_dir + 'timeline' + '.json'
    p.export_chrome_trace(timeline_file)


def print_time_total(p):
    if torch.cuda.is_available():
        profile_sort = 'self_cuda_time_total'
    else:
        profile_sort = 'self_cpu_time_total'
    output = p.key_averages().table(sort_by=profile_sort)
    print(output)


def rename_profile_file(*args):
    profile_dir = str(pathlib.Path.cwd()) + '/'
    timeline_file = profile_dir + 'profile'
    for arg in args:
        timeline_file += '-' + arg
    timeline_file += '.json'
    os.rename('timeline.json', timeline_file)


@contextmanager
def torch_profile(export_chrome_trace=True, csv_data=None, write_csv=None):
    use_cuda = torch.cuda.is_available()

    activities = [ProfilerActivity.CPU]
    if use_cuda:
        activities.append(ProfilerActivity.CUDA)

    if export_chrome_trace:
        p_trace_handler = trace_handler
    else:
        p_trace_handler = print_time_total

    p = profile(activities=activities, on_trace_ready=p_trace_handler)

    with p:
        yield
        p.step()

    if csv_data is not None and write_csv == 'prof':
        if use_cuda:
            profile_sort = 'self_cuda_time_total'
        else:
            profile_sort = 'self_cpu_time_total'
        events = EventList(
            sorted(
                p.key_averages(),
                key=lambda evt: getattr(evt, profile_sort),
                reverse=True,
            ), use_cuda=use_cuda)

        save_profile_data(csv_data, events, use_cuda)


def format_prof_time(time):
    # Profile time is in micro seconds, so format it appropriately:
    return round(time / 1e6, 3)


def save_profile_data(csv_data, events, use_cuda):
    sum_self_cpu_time_total = sum(
        [event.self_cpu_time_total for event in events])
    sum_cpu_time_total = sum([event.self_cpu_time_total for event in events])
    sum_self_cuda_time_total = sum(
        [event.self_cuda_time_total for event in events]) if use_cuda else 0

    for e in events[:5]:  # Save top 5 most time consuming operations:
        csv_data['NAME'].append(e.key)
        csv_data['SELF CPU %'].append(
            round(e.self_cpu_time_total * 100.0 / sum_self_cpu_time_total, 3))
        csv_data['SELF CPU'].append(format_prof_time(e.self_cpu_time_total))
        csv_data['CPU TOTAL %'].append(
            round(e.cpu_time_total * 100.0 / sum_cpu_time_total, 3))
        csv_data['CPU TOTAL'].append(format_prof_time(e.cpu_time_total))
        csv_data['CPU TIME AVG'].append(format_prof_time(e.cpu_time_total))
        if use_cuda:
            csv_data['SELF CUDA %'].append(e.self_cuda_time_total * 100.0 /
                                           sum_self_cuda_time_total)
            csv_data['SELF CUDA'].append(
                format_prof_time(e.self_cuda_time_total))
            csv_data['CUDA TOTAL'].append(format_prof_time(e.cpu_time_total))
            csv_data['CUDA TIME AVG'].append(format_prof_time(
                e.cpu_time_total))
        csv_data['# OF CALLS'].append(e.count)
pytorch/torch_geometric/profile/profiler.py000066400000000000000000000404311444262304300215630ustar00rootroot00000000000000import functools
from collections import OrderedDict, defaultdict, namedtuple
from typing import Any, List, NamedTuple, Optional, Tuple

import torch
import torch.profiler as torch_profiler

# predefined namedtuple for variable setting (global template)
Trace = namedtuple('Trace', ['path', 'leaf', 'module'])

# the metrics returned from the torch profiler
Measure = namedtuple('Measure', [
    'self_cpu_total',
    'cpu_total',
    'self_cuda_total',
    'cuda_total',
    'self_cpu_memory',
    'cpu_memory',
    'self_cuda_memory',
    'cuda_memory',
    'occurrences',
])


class Profiler:
    r"""Layer by layer profiling of PyTorch models, using the PyTorch profiler
    for memory profiling. Parts of the code are adapted from :obj:`torchprof`
    for layer-wise grouping.

    Args:
        model (torch.nn.Module): The underlying model to be profiled.
        enabled (bool, optional): If set to :obj:`True`, turn on the profiler.
            (default: :obj:`False`)
        use_cuda (bool, optional): Whether to profile CUDA execution.
            (default: :obj:`False`)
        profile_memory (bool, optional): If set to :obj:`True`, also profile
            memory usage. (default: :obj:`False`)
        paths ([str], optional): Pre-defined paths for fast loading.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        model: torch.nn.Module,
        enabled: bool = True,
        use_cuda: bool = False,
        profile_memory: bool = False,
        paths: Optional[List[str]] = None,
    ):
        self._model = model
        self.enabled = enabled
        self.use_cuda = use_cuda
        self.profile_memory = profile_memory
        self.paths = paths

        self.entered = False
        self.exited = False
        self.traces = ()
        self._ids = set()
        self.trace_profile_events = defaultdict(list)

    def __enter__(self):
        if not self.enabled:
            return self
        if self.entered:
            raise RuntimeError("the profiler can be initialized only once")
        self.entered = True
        self._forwards = {}  # store the original forward functions

        # generate the trace and conduct profiling
        self.traces = tuple(map(self._hook_trace, _walk_modules(self._model)))
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if not self.enabled:
            return
        tuple(map(self._remove_hook_trace, self.traces))
        del self._forwards  # remove unnecessary forwards
        self.exited = True

    def get_trace(self):
        return _layer_trace(self.traces, self.trace_profile_events)

    def __repr__(self) -> str:
        return self.get_trace()[0]

    def __call__(self, *args, **kwargs):
        return self._model(*args, **kwargs)

    def _hook_trace(self, trace):
        """Add hooks to torch modules for profiling. The underlying model's
        forward pass is hooked/decorated here.
        """
        [path, leaf, module] = trace

        # the id of the model  is guaranteed to be unique
        _id = id(module)
        if (self.paths is not None
                and path in self.paths) or (self.paths is None and leaf):
            if _id in self._ids:
                # already wrapped
                return trace
            self._ids.add(_id)
            _forward = module.forward
            self._forwards[path] = _forward

            @functools.wraps(_forward)
            def wrap_forward(*args, **kwargs):
                """The forward pass is decorated and profiled here
                """
                # only torch 1.8.1+ is supported
                torch_version = torch.__version__
                if torch_version <= '1.8.1':
                    raise NotImplementedError(
                        "Profiler requires at least torch 1.8.1")

                activities = [torch.profiler.ProfilerActivity.CPU]
                if self.use_cuda:
                    activities.append(torch.profiler.ProfilerActivity.CUDA)
                with torch_profiler.profile(
                        activities=activities,
                        profile_memory=self.profile_memory,
                ) as prof:
                    res = _forward(*args, **kwargs)

                event_list = prof.events()

                # each profile call should be contained in its own list
                self.trace_profile_events[path].append(event_list)
                return res

            # decorate the underlying model's forward pass
            module.forward = wrap_forward
        return trace

    def _remove_hook_trace(self, trace):
        """Clean it up after the profiling is done.
        """
        [path, leaf, module] = trace
        _id = id(module)
        if _id in self._ids:
            self._ids.discard(_id)
        else:
            return
        if (self.paths is not None
                and path in self.paths) or (self.paths is None and leaf):
            module.forward = self._forwards[path]


def _layer_trace(
        traces: NamedTuple,
        trace_events: Any,
        show_events: bool = True,
        paths: List[str] = None,
        use_cuda: bool = False,
        profile_memory: bool = False,
        dt: Tuple[str, ...] = ('-', '-', '-', ' '),
) -> object:
    """Construct human readable output of the profiler traces and events. The
    information is presented in layers, and each layer contains its underlying
    operators.

    Args:
        traces (trace object): Raw trace to be parsed.
        trace_events (trace object): Raw events to be parsed.
        show_events (bool, optional): If True, show detailed event information.
            (default: :obj:`True`)
        paths (str, optional): Predefine path for fast loading. By default, it
            will not be used.
            (default: :obj:`False`)
        use_cuda (bool, optional): Enables timing of CUDA events.
            (default: :obj:`False`)
        profile_memory (bool, optional): If True, also profile for the memory
            usage information.
            (default: :obj:`False`)
        dt (object, optional): Delimiters for showing the events.
    """
    tree = OrderedDict()

    for trace in traces:
        [path, leaf, module] = trace
        current_tree = tree
        # unwrap all of the events, in case model is called multiple times
        events = [te for t_events in trace_events[path] for te in t_events]
        for depth, name in enumerate(path, 1):
            if name not in current_tree:
                current_tree[name] = OrderedDict()
            if depth == len(path) and ((paths is None and leaf) or
                                       (paths is not None and path in paths)):
                # tree measurements have key None, avoiding name conflict
                if show_events:
                    for event_name, event_group in _group_by(
                            events, lambda e: e.name):
                        event_group = list(event_group)
                        current_tree[name][event_name] = {
                            None:
                            _build_measure_tuple(event_group, len(event_group))
                        }
                else:
                    current_tree[name][None] = _build_measure_tuple(
                        events, len(trace_events[path]))
            current_tree = current_tree[name]
    tree_lines = _flatten_tree(tree)

    format_lines = []
    has_self_cuda_total = False
    has_self_cpu_memory = False
    has_cpu_memory = False
    has_self_cuda_memory = False
    has_cuda_memory = False

    raw_results = {}
    for idx, tree_line in enumerate(tree_lines):
        depth, name, measures = tree_line

        next_depths = [pl[0] for pl in tree_lines[idx + 1:]]
        pre = "-"
        if depth > 0:
            pre = dt[1] if depth in next_depths and next_depths[
                0] >= depth else dt[2]
            depth -= 1
        while depth > 0:
            pre = (dt[0] + pre) if depth in next_depths else (dt[3] + pre)
            depth -= 1

        format_lines.append([pre + name, *_format_measure_tuple(measures)])
        if measures:
            has_self_cuda_total = (has_self_cuda_total
                                   or measures.self_cuda_total is not None)
            has_self_cpu_memory = (has_self_cpu_memory
                                   or measures.self_cpu_memory is not None)
            has_cpu_memory = has_cpu_memory or measures.cpu_memory is not None
            has_self_cuda_memory = (has_self_cuda_memory
                                    or measures.self_cuda_memory is not None)
            has_cuda_memory = (has_cuda_memory
                               or measures.cuda_memory is not None)

            raw_results[name] = [
                measures.self_cpu_total, measures.cpu_total,
                measures.self_cuda_total, measures.cuda_total,
                measures.self_cpu_memory, measures.cpu_memory,
                measures.self_cuda_memory, measures.cuda_memory,
                measures.occurrences
            ]

    # construct the table (this is pretty ugly and can probably be optimized)
    heading = (
        "Module",
        "Self CPU total",
        "CPU total",
        "Self CUDA total",
        "CUDA total",
        "Self CPU Mem",
        "CPU Mem",
        "Self CUDA Mem",
        "CUDA Mem",
        "Number of Calls",
    )

    # get the output aligned
    max_lens = [max(map(len, col)) for col in zip(*([heading] + format_lines))]

    # not all columns should be displayed, specify kept indexes
    keep_indexes = [0, 1, 2, 9]
    if profile_memory:
        if has_self_cpu_memory:
            keep_indexes.append(5)
        if has_cpu_memory:
            keep_indexes.append(6)
    if use_cuda:
        if has_self_cuda_total:
            keep_indexes.append(3)
        keep_indexes.append(4)
        if profile_memory:
            if has_self_cuda_memory:
                keep_indexes.append(7)
            if has_cuda_memory:
                keep_indexes.append(8)

    # the final columns to be shown
    keep_indexes = tuple(sorted(keep_indexes))

    heading_list = list(heading)

    display = (  # table heading
        " | ".join([
            "{:<{}s}".format(heading[keep_index], max_lens[keep_index])
            for keep_index in keep_indexes
        ]) + "\n")
    display += (  # separator
        "-|-".join([
            "-" * max_len for val_idx, max_len in enumerate(max_lens)
            if val_idx in keep_indexes
        ]) + "\n")
    for format_line in format_lines:  # body
        display += (" | ".join([
            "{:<{}s}".format(value, max_lens[val_idx])
            for val_idx, value in enumerate(format_line)
            if val_idx in keep_indexes
        ]) + "\n")
    # layer information readable
    key_dict = {}
    layer_names = []
    layer_stats = []
    for format_line in format_lines:  # body
        if format_line[1] == '':  # key line
            key_dict[format_line[0].count("-")] = format_line[0]
        else:  # must print
            # get current line's level
            curr_level = format_line[0].count("-")
            par_str = ""
            for i in range(1, curr_level):
                par_str += key_dict[i]
            curr_key = par_str + format_line[0]
            layer_names.append(curr_key)
            layer_stats.append(format_line[1:])

    return display, heading_list, raw_results, layer_names, layer_stats


def _flatten_tree(t, depth=0):
    flat = []
    for name, st in t.items():
        measures = st.pop(None, None)
        flat.append([depth, name, measures])
        flat.extend(_flatten_tree(st, depth=depth + 1))
    return flat


def _build_measure_tuple(events: List, occurrences: List) -> NamedTuple:
    # memory profiling supported in torch >= 1.6
    self_cpu_memory = None
    has_self_cpu_memory = any(
        hasattr(e, "self_cpu_memory_usage") for e in events)
    if has_self_cpu_memory:
        self_cpu_memory = sum(
            [getattr(e, "self_cpu_memory_usage", 0) for e in events])
    cpu_memory = None
    has_cpu_memory = any(hasattr(e, "cpu_memory_usage") for e in events)
    if has_cpu_memory:
        cpu_memory = sum([getattr(e, "cpu_memory_usage", 0) for e in events])
    self_cuda_memory = None
    has_self_cuda_memory = any(
        hasattr(e, "self_cuda_memory_usage") for e in events)
    if has_self_cuda_memory:
        self_cuda_memory = sum(
            [getattr(e, "self_cuda_memory_usage", 0) for e in events])
    cuda_memory = None
    has_cuda_memory = any(hasattr(e, "cuda_memory_usage") for e in events)
    if has_cuda_memory:
        cuda_memory = sum([getattr(e, "cuda_memory_usage", 0) for e in events])

    # self CUDA time supported in torch >= 1.7
    self_cuda_total = None
    has_self_cuda_time = any(
        hasattr(e, "self_cuda_time_total") for e in events)
    if has_self_cuda_time:
        self_cuda_total = sum(
            [getattr(e, "self_cuda_time_total", 0) for e in events])

    return Measure(
        self_cpu_total=sum([e.self_cpu_time_total for e in events]),
        cpu_total=sum([e.cpu_time_total for e in events]),
        self_cuda_total=self_cuda_total,
        cuda_total=sum([e.cuda_time_total for e in events]),
        self_cpu_memory=self_cpu_memory,
        cpu_memory=cpu_memory,
        self_cuda_memory=self_cuda_memory,
        cuda_memory=cuda_memory,
        occurrences=occurrences,
    )


def _format_measure_tuple(measure: NamedTuple) -> NamedTuple:
    self_cpu_total = (format_time(measure.self_cpu_total) if measure else "")
    cpu_total = format_time(measure.cpu_total) if measure else ""
    self_cuda_total = (format_time(measure.self_cuda_total) if measure
                       and measure.self_cuda_total is not None else "")
    cuda_total = format_time(measure.cuda_total) if measure else ""
    self_cpu_memory = (format_memory(measure.self_cpu_memory) if measure
                       and measure.self_cpu_memory is not None else "")
    cpu_memory = (format_memory(measure.cpu_memory)
                  if measure and measure.cpu_memory is not None else "")
    self_cuda_memory = (format_memory(measure.self_cuda_memory) if measure
                        and measure.self_cuda_memory is not None else "")
    cuda_memory = (format_memory(measure.cuda_memory)
                   if measure and measure.cuda_memory is not None else "")
    occurrences = str(measure.occurrences) if measure else ""

    return Measure(
        self_cpu_total=self_cpu_total,
        cpu_total=cpu_total,
        self_cuda_total=self_cuda_total,
        cuda_total=cuda_total,
        self_cpu_memory=self_cpu_memory,
        cpu_memory=cpu_memory,
        self_cuda_memory=self_cuda_memory,
        cuda_memory=cuda_memory,
        occurrences=occurrences,
    )


def _group_by(events, keyfn):
    event_groups = OrderedDict()
    for event in events:
        key = keyfn(event)
        key_events = event_groups.get(key, [])
        key_events.append(event)
        event_groups[key] = key_events
    return event_groups.items()


def _walk_modules(module, name: str = "", path=()):
    # Walk through a PyTorch model and output trace tuples (its path, leafe
    # node, model).
    if not name:
        name = module.__class__.__name__

    # This will track the children of the module (layers)
    # for instance, [('conv1', GCNConv(10, 16)), ('conv2', GCNConv(16, 3))]
    named_children = list(module.named_children())

    # it builds the path of the structure
    # for instance, ('GCN', 'conv1', 'lin')
    path = path + (name, )

    # create namedtuple [path, (whether has) leaf, module]
    yield Trace(path, len(named_children) == 0, module)

    # recursively walk into all submodules
    for name, child_module in named_children:
        yield from _walk_modules(child_module, name=name, path=path)


def format_time(time_us: int) -> str:
    r"""Returns a formatted time string."""
    US_IN_SECOND = 1000.0 * 1000.0
    US_IN_MS = 1000.0
    if time_us >= US_IN_SECOND:
        return '{:.3f}s'.format(time_us / US_IN_SECOND)
    if time_us >= US_IN_MS:
        return '{:.3f}ms'.format(time_us / US_IN_MS)
    return '{:.3f}us'.format(time_us)


def format_memory(nbytes: int) -> str:
    """Returns a formatted memory size string."""
    KB = 1024
    MB = 1024 * KB
    GB = 1024 * MB
    if (abs(nbytes) >= GB):
        return '{:.2f} Gb'.format(nbytes * 1.0 / GB)
    elif (abs(nbytes) >= MB):
        return '{:.2f} Mb'.format(nbytes * 1.0 / MB)
    elif (abs(nbytes) >= KB):
        return '{:.2f} Kb'.format(nbytes * 1.0 / KB)
    else:
        return str(nbytes) + ' b'
pytorch/torch_geometric/profile/utils.py000066400000000000000000000107231444262304300211020ustar00rootroot00000000000000import gc
import os
import os.path as osp
import random
import subprocess as sp
import sys
import warnings
from collections.abc import Mapping, Sequence
from typing import Any, Tuple

import torch
from torch import Tensor

from torch_geometric.data.data import BaseData
from torch_geometric.typing import SparseTensor


def count_parameters(model: torch.nn.Module) -> int:
    r"""Given a :class:`torch.nn.Module`, count its trainable parameters.

    Args:
        model (torch.nn.Model): The model.
    """
    return sum([p.numel() for p in model.parameters() if p.requires_grad])


def get_model_size(model: torch.nn.Module) -> int:
    r"""Given a :class:`torch.nn.Module`, get its actual disk size in bytes.

    Args:
        model (torch model): The model.
    """
    path = f'{random.randrange(sys.maxsize)}.pt'
    torch.save(model.state_dict(), path)
    model_size = osp.getsize(path)
    os.remove(path)
    return model_size


def get_data_size(data: BaseData) -> int:
    r"""Given a :class:`torch_geometric.data.Data` object, get its theoretical
    memory usage in bytes.

    Args:
        data (torch_geometric.data.Data or torch_geometric.data.HeteroData):
            The :class:`~torch_geometric.data.Data` or
            :class:`~torch_geometric.data.HeteroData` graph object.
    """
    data_ptrs = set()

    def _get_size(obj: Any) -> int:
        if isinstance(obj, Tensor):
            if obj.data_ptr() in data_ptrs:
                return 0
            data_ptrs.add(obj.data_ptr())
            return obj.numel() * obj.element_size()
        elif isinstance(obj, SparseTensor):
            return _get_size(obj.csr())
        elif isinstance(obj, Sequence) and not isinstance(obj, str):
            return sum([_get_size(x) for x in obj])
        elif isinstance(obj, Mapping):
            return sum([_get_size(x) for x in obj.values()])
        else:
            return 0

    return sum([_get_size(store) for store in data.stores])


def get_cpu_memory_from_gc() -> int:
    r"""Returns the used CPU memory in bytes, as reported by the Python garbage
    collector.
    """
    warnings.filterwarnings('ignore', '.*torch.distributed.reduce_op.*')

    mem = 0
    for obj in gc.get_objects():
        try:
            if isinstance(obj, Tensor) and not obj.is_cuda:
                mem += obj.numel() * obj.element_size()
        except:  # noqa
            pass
    return mem


def get_gpu_memory_from_gc(device: int = 0) -> int:  # pragma: no cover
    r"""Returns the used GPU memory in bytes, as reported by the Python garbage
    collector.

    Args:
        device (int, optional): The GPU device identifier. (default: :obj:`1`)
    """
    warnings.filterwarnings('ignore', '.*torch.distributed.reduce_op.*')

    mem = 0
    for obj in gc.get_objects():
        try:
            if isinstance(obj, Tensor) and obj.get_device() == device:
                mem += obj.numel() * obj.element_size()
        except:  # noqa
            pass
    return mem


def get_gpu_memory_from_nvidia_smi(  # pragma: no cover
    device: int = 0,
    digits: int = 2,
) -> Tuple[float, float]:
    r"""Returns the free and used GPU memory in megabytes, as reported by
    :obj:`nivdia-smi`.

    .. note::

        :obj:`nvidia-smi` will generally overestimate the amount of memory used
        by the actual program, see `here `__.

    Args:
        device (int, optional): The GPU device identifier. (default: :obj:`1`)
        digits (int): The number of decimals to use for megabytes.
            (default: :obj:`2`)
    """
    CMD = 'nvidia-smi --query-gpu=memory.free --format=csv'
    free_out = sp.check_output(CMD.split()).decode('utf-8').split('\n')[1:-1]

    CMD = 'nvidia-smi --query-gpu=memory.used --format=csv'
    used_out = sp.check_output(CMD.split()).decode('utf-8').split('\n')[1:-1]

    if device < 0 or device >= len(free_out):
        raise AttributeError(
            f'GPU {device} not available (found {len(free_out)} GPUs)')

    free_mem = medibyte_to_megabyte(int(free_out[device].split()[0]), digits)
    used_mem = medibyte_to_megabyte(int(used_out[device].split()[0]), digits)

    return free_mem, used_mem


###############################################################################


def byte_to_megabyte(value: int, digits: int = 2) -> float:
    return round(value / (1024 * 1024), digits)


def medibyte_to_megabyte(value: int, digits: int = 2) -> float:
    return round(1.0485 * value, digits)
pytorch/torch_geometric/resolver.py000066400000000000000000000023431444262304300201420ustar00rootroot00000000000000import inspect
from typing import Any, Dict, List, Optional, Union


def normalize_string(s: str) -> str:
    return s.lower().replace('-', '').replace('_', '').replace(' ', '')


def resolver(classes: List[Any], class_dict: Dict[str, Any],
             query: Union[Any, str], base_cls: Optional[Any],
             base_cls_repr: Optional[str], *args, **kwargs):

    if not isinstance(query, str):
        return query

    query_repr = normalize_string(query)
    if base_cls_repr is None:
        base_cls_repr = base_cls.__name__ if base_cls else ''
    base_cls_repr = normalize_string(base_cls_repr)

    for key_repr, cls in class_dict.items():
        if query_repr == key_repr:
            if inspect.isclass(cls):
                obj = cls(*args, **kwargs)
                return obj
            return cls

    for cls in classes:
        cls_repr = normalize_string(cls.__name__)
        if query_repr in [cls_repr, cls_repr.replace(base_cls_repr, '')]:
            if inspect.isclass(cls):
                obj = cls(*args, **kwargs)
                return obj
            return cls

    choices = set(cls.__name__ for cls in classes) | set(class_dict.keys())
    raise ValueError(f"Could not resolve '{query}' among choices {choices}")
pytorch/torch_geometric/sampler/000077500000000000000000000000001444262304300173705ustar00rootroot00000000000000pytorch/torch_geometric/sampler/__init__.py000066400000000000000000000007411444262304300215030ustar00rootroot00000000000000from .base import (BaseSampler, NodeSamplerInput, EdgeSamplerInput,
                   SamplerOutput, HeteroSamplerOutput, NegativeSampling,
                   NumNeighbors)
from .neighbor_sampler import NeighborSampler
from .hgt_sampler import HGTSampler

__all__ = classes = [
    'BaseSampler',
    'NodeSamplerInput',
    'EdgeSamplerInput',
    'SamplerOutput',
    'HeteroSamplerOutput',
    'NumNeighbors',
    'NegativeSampling',
    'NeighborSampler',
    'HGTSampler',
]
pytorch/torch_geometric/sampler/base.py000066400000000000000000000565111444262304300206640ustar00rootroot00000000000000import copy
import math
import warnings
from abc import ABC
from collections import defaultdict
from dataclasses import dataclass
from enum import Enum
from typing import Any, Dict, List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, FeatureStore, GraphStore, HeteroData
from torch_geometric.sampler.utils import to_bidirectional
from torch_geometric.typing import EdgeType, EdgeTypeStr, NodeType, OptTensor
from torch_geometric.utils.mixin import CastMixin


class DataType(Enum):
    r"""The data type a sampler is operating on."""
    homogeneous = 'homogeneous'
    heterogeneous = 'heterogeneous'
    remote = 'remote'

    @classmethod
    def from_data(cls, data: Any):
        if isinstance(data, Data):
            return cls.homogeneous
        elif isinstance(data, HeteroData):
            return cls.heterogeneous
        elif (isinstance(data, (list, tuple)) and len(data) == 2
              and isinstance(data[0], FeatureStore)
              and isinstance(data[1], GraphStore)):
            return cls.remote

        raise ValueError(f"Expected a 'Data', 'HeteroData', or a tuple of "
                         f"'FeatureStore' and 'GraphStore' "
                         f"(got '{type(data)}')")


class SubgraphType(Enum):
    r"""The type of the returned subgraph."""
    directional = 'directional'
    bidirectional = 'bidirectional'
    induced = 'induced'


@dataclass(init=False)
class NodeSamplerInput(CastMixin):
    r"""The sampling input of
    :meth:`~torch_geometric.sampler.BaseSampler.sample_from_nodes`.

    Args:
        input_id (torch.Tensor, optional): The indices of the data loader input
            of the current mini-batch.
        node (torch.Tensor): The indices of seed nodes to start sampling from.
        time (torch.Tensor, optional): The timestamp for the seed nodes.
            (default: :obj:`None`)
        input_type (str, optional): The input node type (in case of sampling in
            a heterogeneous graph). (default: :obj:`None`)
    """
    input_id: OptTensor
    node: Tensor
    time: OptTensor = None
    input_type: Optional[NodeType] = None

    def __init__(
        self,
        input_id: OptTensor,
        node: Tensor,
        time: OptTensor = None,
        input_type: Optional[NodeType] = None,
    ):
        if input_id is not None:
            input_id = input_id.cpu()
        node = node.cpu()
        if time is not None:
            time = time.cpu()

        self.input_id = input_id
        self.node = node
        self.time = time
        self.input_type = input_type

    def __getitem__(self, index: Union[Tensor, Any]) -> 'NodeSamplerInput':
        if not isinstance(index, Tensor):
            index = torch.tensor(index, dtype=torch.long)

        return NodeSamplerInput(
            self.input_id[index] if self.input_id is not None else index,
            self.node[index],
            self.time[index] if self.time is not None else None,
            self.input_type,
        )


@dataclass(init=False)
class EdgeSamplerInput(CastMixin):
    r"""The sampling input of
    :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges`.

    Args:
        input_id (torch.Tensor, optional): The indices of the data loader input
            of the current mini-batch.
        row (torch.Tensor): The source node indices of seed links to start
            sampling from.
        col (torch.Tensor): The destination node indices of seed links to start
            sampling from.
        label (torch.Tensor, optional): The label for the seed links.
            (default: :obj:`None`)
        time (torch.Tensor, optional): The timestamp for the seed links.
            (default: :obj:`None`)
        input_type (Tuple[str, str, str], optional): The input edge type (in
            case of sampling in a heterogeneous graph). (default: :obj:`None`)
    """
    input_id: OptTensor
    row: Tensor
    col: Tensor
    label: OptTensor = None
    time: OptTensor = None
    input_type: Optional[EdgeType] = None

    def __init__(
        self,
        input_id: OptTensor,
        row: Tensor,
        col: Tensor,
        label: OptTensor = None,
        time: OptTensor = None,
        input_type: Optional[EdgeType] = None,
    ):
        if input_id is not None:
            input_id = input_id.cpu()
        row = row.clone().cpu()
        col = col.clone().cpu()
        if label is not None:
            label = label.cpu()
        if time is not None:
            time = time.cpu()

        self.input_id = input_id
        self.row = row
        self.col = col
        self.label = label
        self.time = time
        self.input_type = input_type

    def __getitem__(self, index: Union[Tensor, Any]) -> 'EdgeSamplerInput':
        if not isinstance(index, Tensor):
            index = torch.tensor(index, dtype=torch.long)

        return EdgeSamplerInput(
            self.input_id[index] if self.input_id is not None else index,
            self.row[index],
            self.col[index],
            self.label[index] if self.label is not None else None,
            self.time[index] if self.time is not None else None,
            self.input_type,
        )


@dataclass
class SamplerOutput(CastMixin):
    r"""The sampling output of a :class:`~torch_geometric.sampler.BaseSampler`
    on homogeneous graphs.

    Args:
        node (torch.Tensor): The sampled nodes in the original graph.
        row (torch.Tensor): The source node indices of the sampled subgraph.
            Indices must be re-indexed to :obj:`{ 0, ..., num_nodes - 1 }`
            corresponding to the nodes in the :obj:`node` tensor.
        col (torch.Tensor): The destination node indices of the sampled
            subgraph.
            Indices must be re-indexed to :obj:`{ 0, ..., num_nodes - 1 }`
            corresponding to the nodes in the :obj:`node` tensor.
        edge (torch.Tensor, optional): The sampled edges in the original graph.
            This tensor is used to obtain edge features from the original
            graph. If no edge attributes are present, it may be omitted.
        batch (torch.Tensor, optional): The vector to identify the seed node
            for each sampled node. Can be present in case of disjoint subgraph
            sampling per seed node. (default: :obj:`None`)
        num_sampled_nodes (List[int], optional): The number of sampled nodes
            per hop. (default: :obj:`None`)
        num_sampled_edges (List[int], optional): The number of sampled edges
            per hop. (default: :obj:`None`)
        metadata: (Any, optional): Additional metadata information.
            (default: :obj:`None`)
    """
    node: Tensor
    row: Tensor
    col: Tensor
    edge: OptTensor
    batch: OptTensor = None
    num_sampled_nodes: Optional[List[int]] = None
    num_sampled_edges: Optional[List[int]] = None
    # TODO(manan): refine this further; it does not currently define a proper
    # API for the expected output of a sampler.
    metadata: Optional[Any] = None

    def to_bidirectional(self) -> 'SamplerOutput':
        r"""Converts the sampled subgraph into a bidirectional variant, in
        which all sampled edges are guaranteed to be bidirectional."""
        out = copy.copy(self)

        out.row, out.col, out.edge = to_bidirectional(
            row=self.row,
            col=self.col,
            rev_row=self.row,
            rev_col=self.col,
            edge_id=self.edge,
            rev_edge_id=self.edge,
        )
        out.num_sampled_nodes = out.num_sampled_edges = None

        return out


@dataclass
class HeteroSamplerOutput(CastMixin):
    r"""The sampling output of a :class:`~torch_geometric.sampler.BaseSampler`
    on heterogeneous graphs.

    Args:
        node (Dict[str, torch.Tensor]): The sampled nodes in the original graph
            for each node type.
        row (Dict[Tuple[str, str, str], torch.Tensor]): The source node indices
            of the sampled subgraph for each edge type.
            Indices must be re-indexed to :obj:`{ 0, ..., num_nodes - 1 }`
            corresponding to the nodes in the :obj:`node` tensor of the source
            node type.
        col (Dict[Tuple[str, str, str], torch.Tensor]): The destination node
            indices of the sampled subgraph for each edge type.
            Indices must be re-indexed to :obj:`{ 0, ..., num_nodes - 1 }`
            corresponding to the nodes in the :obj:`node` tensor of the
            destination node type.
        edge (Dict[Tuple[str, str, str], torch.Tensor], optional): The sampled
            edges in the original graph for each edge type.
            This tensor is used to obtain edge features from the original
            graph. If no edge attributes are present, it may be omitted.
        batch (Dict[str, torch.Tensor], optional): The vector to identify the
            seed node for each sampled node for each node type. Can be present
            in case of disjoint subgraph sampling per seed node.
            (default: :obj:`None`)
        num_sampled_nodes (Dict[str, List[int]], optional): The number of
            sampled nodes for each node type and each layer.
            (default: :obj:`None`)
        num_sampled_edges (Dict[EdgeType, List[int]], optional): The number of
            sampled edges for each edge type and each layer.
            (default: :obj:`None`)
        metadata: (Any, optional): Additional metadata information.
            (default: :obj:`None`)
    """
    node: Dict[NodeType, Tensor]
    row: Dict[EdgeType, Tensor]
    col: Dict[EdgeType, Tensor]
    edge: Dict[EdgeType, OptTensor]
    batch: Optional[Dict[NodeType, Tensor]] = None
    num_sampled_nodes: Optional[Dict[NodeType, List[int]]] = None
    num_sampled_edges: Optional[Dict[EdgeType, List[int]]] = None
    # TODO(manan): refine this further; it does not currently define a proper
    # API for the expected output of a sampler.
    metadata: Optional[Any] = None

    def to_bidirectional(self) -> 'SamplerOutput':
        r"""Converts the sampled subgraph into a bidirectional variant, in
        which all sampled edges are guaranteed to be bidirectional."""
        out = copy.copy(self)
        out.row = copy.copy(self.row)
        out.col = copy.copy(self.col)
        out.edge = copy.copy(self.edge)

        src_dst_dict = defaultdict(list)
        edge_types = self.row.keys()
        edge_types = [k for k in edge_types if not k[1].startswith('rev_')]
        for edge_type in edge_types:
            src, rel, dst = edge_type
            rev_edge_type = (dst, f'rev_{rel}', src)

            if src == dst and rev_edge_type not in self.row:
                out.row[edge_type], out.col[edge_type], _ = to_bidirectional(
                    row=self.row[edge_type],
                    col=self.col[edge_type],
                    rev_row=self.row[edge_type],
                    rev_col=self.col[edge_type],
                )
                if out.edge is not None:
                    out.edge[edge_type] = None

            elif rev_edge_type in self.row:
                out.row[edge_type], out.col[edge_type], _ = to_bidirectional(
                    row=self.row[edge_type],
                    col=self.col[edge_type],
                    rev_row=self.row[rev_edge_type],
                    rev_col=self.col[rev_edge_type],
                )
                out.row[rev_edge_type] = out.col[edge_type]
                out.col[rev_edge_type] = out.row[edge_type]
                if out.edge is not None:
                    out.edge[edge_type] = None
                    out.edge[rev_edge_type] = None

            else:  # Find the reverse edge type (if it is unique):
                if len(src_dst_dict) == 0:  # Create mapping lazily.
                    for key in self.row.keys():
                        v1, _, v2 = key
                        src_dst_dict[(v1, v2)].append(key)

                if len(src_dst_dict[(dst, src)]) == 1:
                    rev_edge_type = src_dst_dict[(dst, src)][0]
                    row, col, _ = to_bidirectional(
                        row=self.row[edge_type],
                        col=self.col[edge_type],
                        rev_row=self.row[rev_edge_type],
                        rev_col=self.col[rev_edge_type],
                    )
                    out.row[edge_type] = row
                    out.col[edge_type] = col
                    if out.edge is not None:
                        out.edge[edge_type] = None

                else:
                    warnings.warn(f"Cannot convert to bidirectional graph "
                                  f"since the edge type {edge_type} does not "
                                  f"seem to have a reverse edge type")

        out.num_sampled_nodes = out.num_sampled_edges = None

        return out


@dataclass(frozen=True)
class NumNeighbors:
    r"""The number of neighbors to sample in a homogeneous or heterogeneous
    graph. In heterogeneous graphs, may also take in a dictionary denoting
    the amount of neighbors to sample for individual edge types.

    Args:
        values (List[int] or Dict[Tuple[str, str, str], List[int]]): The
            number of neighbors to sample.
            If an entry is set to :obj:`-1`, all neighbors will be included.
            In heterogeneous graphs, may also take in a dictionary denoting
            the amount of neighbors to sample for individual edge types.
        default (List[int], optional): The default number of neighbors for edge
            types not specified in :obj:`values`. (default: :obj:`None`)
    """
    values: Union[List[int], Dict[EdgeTypeStr, List[int]]]
    default: Optional[List[int]] = None

    def __init__(
        self,
        values: Union[List[int], Dict[EdgeType, List[int]]],
        default: Optional[List[int]] = None,
    ):
        if isinstance(values, (tuple, list)) and default is not None:
            raise ValueError(f"'default' must be set to 'None' in case a "
                             f"single list is given as the number of "
                             f"neighbors (got '{type(default)})'")

        if isinstance(values, dict):
            values = {EdgeTypeStr(key): value for key, value in values.items()}

        # Write to `__dict__` since dataclass is annotated with `frozen=True`:
        self.__dict__['values'] = values
        self.__dict__['default'] = default

    def _get_values(
        self,
        edge_types: Optional[List[EdgeType]] = None,
        mapped: bool = False,
    ) -> Union[List[int], Dict[Union[EdgeType, EdgeTypeStr], List[int]]]:

        if edge_types is not None:
            if isinstance(self.values, (tuple, list)):
                default = self.values
            elif isinstance(self.values, dict):
                default = self.default
            else:
                assert False

            out = {}
            for edge_type in edge_types:
                edge_type_str = EdgeTypeStr(edge_type)
                if edge_type_str in self.values:
                    out[edge_type_str if mapped else edge_type] = (
                        self.values[edge_type_str])
                else:
                    if default is None:
                        raise ValueError(f"Missing number of neighbors for "
                                         f"edge type '{edge_type}'")
                    out[edge_type_str if mapped else edge_type] = default

        elif isinstance(self.values, dict) and not mapped:
            out = {key.to_tuple(): value for key, value in self.values.items()}

        else:
            out = copy.copy(self.values)

        if isinstance(out, dict):
            num_hops = set(len(v) for v in out.values())
            if len(num_hops) > 1:
                raise ValueError(f"Number of hops must be the same across all "
                                 f"edge types (got {len(num_hops)} different "
                                 f"number of hops)")

        return out

    def get_values(
        self,
        edge_types: Optional[List[EdgeType]] = None,
    ) -> Union[List[int], Dict[EdgeType, List[int]]]:
        r"""Returns the number of neighbors.

        Args:
            edge_types (List[Tuple[str, str, str]], optional): The edge types
                to generate the number of neighbors for. (default: :obj:`None`)
        """
        if '_values' in self.__dict__:
            return self.__dict__['_values']

        values = self._get_values(edge_types, mapped=False)

        self.__dict__['_values'] = values
        return values

    def get_mapped_values(
        self,
        edge_types: Optional[List[EdgeType]] = None,
    ) -> Union[List[int], Dict[str, List[int]]]:
        r"""Returns the number of neighbors.
        For heterogeneous graphs, a dictionary is returned in which edge type
        tuples are converted to strings.

        Args:
            edge_types (List[Tuple[str, str, str]], optional): The edge types
                to generate the number of neighbors for. (default: :obj:`None`)
        """
        if '_mapped_values' in self.__dict__:
            return self.__dict__['_mapped_values']

        values = self._get_values(edge_types, mapped=True)

        self.__dict__['_mapped_values'] = values
        return values

    @property
    def num_hops(self) -> int:
        r"""Returns the number of hops."""
        if '_num_hops' in self.__dict__:
            return self.__dict__['_num_hops']

        if isinstance(self.values, (tuple, list)):
            num_hops = max(len(self.values), len(self.default or []))
        else:  # isinstance(self.values, dict):
            num_hops = max([0] + [len(v) for v in self.values.values()])
            num_hops = max(num_hops, len(self.default or []))

        self.__dict__['_num_hops'] = num_hops
        return num_hops

    def __len__(self) -> int:
        r"""Returns the number of hops."""
        return self.num_hops


class NegativeSamplingMode(Enum):
    # 'binary': Randomly sample negative edges in the graph.
    binary = 'binary'
    # 'triplet': Randomly sample negative destination nodes for each positive
    # source node.
    triplet = 'triplet'


@dataclass
class NegativeSampling(CastMixin):
    r"""The negative sampling configuration of a
    :class:`~torch_geometric.sampler.BaseSampler` when calling
    :meth:`~torch_geometric.sampler.BaseSampler.sample_from_edges`.

    Args:
        mode (str): The negative sampling mode
            (:obj:`"binary"` or :obj:`"triplet"`).
            If set to :obj:`"binary"`, will randomly sample negative links
            from the graph.
            If set to :obj:`"triplet"`, will randomly sample negative
            destination nodes for each positive source node.
        amount (int or float, optional): The ratio of sampled negative edges to
            the number of positive edges. (default: :obj:`1`)
        weight (torch.Tensor, optional): A node-level vector determining the
            sampling of nodes. Does not necessariyl need to sum up to one.
            If not given, negative nodes will be sampled uniformly.
            (default: :obj:`None`)
    """
    mode: NegativeSamplingMode
    amount: Union[int, float] = 1
    weight: Optional[Tensor] = None

    def __init__(
        self,
        mode: Union[NegativeSamplingMode, str],
        amount: Union[int, float] = 1,
        weight: Optional[Tensor] = None,
    ):
        self.mode = NegativeSamplingMode(mode)
        self.amount = amount
        self.weight = weight

        if self.amount <= 0:
            raise ValueError(f"The attribute 'amount' needs to be positive "
                             f"for '{self.__class__.__name__}' "
                             f"(got {self.amount})")

        if self.is_triplet():
            if self.amount != math.ceil(self.amount):
                raise ValueError(f"The attribute 'amount' needs to be an "
                                 f"integer for '{self.__class__.__name__}' "
                                 f"with 'triplet' negative sampling "
                                 f"(got {self.amount}).")
            self.amount = math.ceil(self.amount)

    def is_binary(self) -> bool:
        return self.mode == NegativeSamplingMode.binary

    def is_triplet(self) -> bool:
        return self.mode == NegativeSamplingMode.triplet

    def sample(self, num_samples: int,
               num_nodes: Optional[int] = None) -> Tensor:
        r"""Generates :obj:`num_samples` negative samples."""
        if self.weight is None:
            if num_nodes is None:
                raise ValueError(
                    f"Cannot sample negatives in '{self.__class__.__name__}' "
                    f"without passing the 'num_nodes' argument")
            return torch.randint(num_nodes, (num_samples, ))

        if num_nodes is not None and self.weight.numel() != num_nodes:
            raise ValueError(
                f"The 'weight' attribute in '{self.__class__.__name__}' "
                f"needs to match the number of nodes {num_nodes} "
                f"(got {self.weight.numel()})")
        return torch.multinomial(self.weight, num_samples, replacement=True)


class BaseSampler(ABC):
    r"""An abstract base class that initializes a graph sampler and provides
    :meth:`sample_from_nodes` and :meth:`sample_from_edges` routines.

    .. note ::

        Any data stored in the sampler will be *replicated* across data loading
        workers that use the sampler since each data loading worker holds its
        own instance of a sampler.
        As such, it is recommended to limit the amount of information stored in
        the sampler.
    """
    def sample_from_nodes(
        self,
        index: NodeSamplerInput,
        **kwargs,
    ) -> Union[HeteroSamplerOutput, SamplerOutput]:
        r"""Performs sampling from the nodes specified in :obj:`index`,
        returning a sampled subgraph in the specified output format.

        The :obj:`index` is a tuple holding the following information:

        1. The example indices of the seed nodes
        2. The node indices to start sampling from
        3. The timestamps of the given seed nodes (optional)

        Args:
            index (NodeSamplerInput): The node sampler input object.
        """
        raise NotImplementedError

    def sample_from_edges(
        self,
        index: EdgeSamplerInput,
        neg_sampling: Optional[NegativeSampling] = None,
    ) -> Union[HeteroSamplerOutput, SamplerOutput]:
        r"""Performs sampling from the edges specified in :obj:`index`,
        returning a sampled subgraph in the specified output format.

        The :obj:`index` is a tuple holding the following information:

        1. The example indices of the seed links
        2. The source node indices to start sampling from
        3. The destination node indices to start sampling from
        4. The labels of the seed links (optional)
        5. The timestamps of the given seed nodes (optional)

        Args:
            index (EdgeSamplerInput): The edge sampler input object.
            neg_sampling (NegativeSampling, optional): The negative sampling
                configuration. (default: :obj:`None`)
        """
        raise NotImplementedError

    @property
    def edge_permutation(self) -> Union[OptTensor, Dict[EdgeType, OptTensor]]:
        r"""If the sampler performs any modification of edge ordering in the
        original graph, this function is expected to return the permutation
        tensor that defines the permutation from the edges in the original
        graph and the edges used in the sampler. If no such permutation was
        applied, :obj:`None` is returned. For heterogeneous graphs, the
        expected return type is a permutation tensor for each edge type."""
        return None
pytorch/torch_geometric/sampler/hgt_sampler.py000066400000000000000000000052561444262304300222570ustar00rootroot00000000000000from typing import Dict, List, Union

import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.sampler import (
    BaseSampler,
    HeteroSamplerOutput,
    NodeSamplerInput,
)
from torch_geometric.sampler.utils import remap_keys, to_hetero_csc
from torch_geometric.typing import (
    WITH_TORCH_SPARSE,
    EdgeType,
    NodeType,
    OptTensor,
)


class HGTSampler(BaseSampler):
    r"""An implementation of an in-memory heterogeneous layer-wise sampler
    user by :class:`~torch_geometric.loader.HGTLoader`."""
    def __init__(
        self,
        data: HeteroData,
        num_samples: Union[List[int], Dict[NodeType, List[int]]],
        is_sorted: bool = False,
        share_memory: bool = False,
    ):
        if not WITH_TORCH_SPARSE:
            raise ImportError(
                f"'{self.__class__.__name__}' requires 'torch-sparse'")

        if isinstance(data, Data) or isinstance(data, tuple):
            raise NotImplementedError(
                f'{self.__class__.__name__} does not support a data object of '
                f'type {type(data)}.')

        if isinstance(num_samples, (list, tuple)):
            num_samples = {key: num_samples for key in data.node_types}

        self.node_types, self.edge_types = data.metadata()
        self.num_samples = num_samples
        self.num_hops = max([len(v) for v in num_samples.values()])

        # Conversion to/from C++ string type (see `NeighborSampler`):
        self.to_rel_type = {k: '__'.join(k) for k in self.edge_types}
        self.to_edge_type = {v: k for k, v in self.to_rel_type.items()}

        # Convert the graph data into a suitable format for sampling:
        colptr_dict, row_dict, self.perm = to_hetero_csc(
            data, device='cpu', share_memory=share_memory, is_sorted=is_sorted)
        self.row_dict = remap_keys(row_dict, self.to_rel_type)
        self.colptr_dict = remap_keys(colptr_dict, self.to_rel_type)

    def sample_from_nodes(
        self,
        inputs: NodeSamplerInput,
        **kwargs,
    ) -> HeteroSamplerOutput:

        node, row, col, edge = torch.ops.torch_sparse.hgt_sample(
            self.colptr_dict,
            self.row_dict,
            {inputs.input_type: inputs.node},
            self.num_samples,
            self.num_hops,
        )

        return HeteroSamplerOutput(
            node=node,
            row=remap_keys(row, self.to_edge_type),
            col=remap_keys(col, self.to_edge_type),
            edge=remap_keys(edge, self.to_edge_type),
            batch=None,
            metadata=(inputs.input_id, inputs.time),
        )

    @property
    def edge_permutation(self) -> Union[OptTensor, Dict[EdgeType, OptTensor]]:
        return self.perm
pytorch/torch_geometric/sampler/neighbor_sampler.py000066400000000000000000000635601444262304300232740ustar00rootroot00000000000000import copy
import math
import sys
import warnings
from typing import Callable, Dict, List, Optional, Tuple, Union

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data import (
    Data,
    FeatureStore,
    GraphStore,
    HeteroData,
    remote_backend_utils,
)
from torch_geometric.data.graph_store import EdgeLayout
from torch_geometric.sampler import (
    BaseSampler,
    EdgeSamplerInput,
    HeteroSamplerOutput,
    NegativeSampling,
    NodeSamplerInput,
    SamplerOutput,
)
from torch_geometric.sampler.base import DataType, NumNeighbors, SubgraphType
from torch_geometric.sampler.utils import remap_keys, to_csc, to_hetero_csc
from torch_geometric.typing import EdgeType, NodeType, OptTensor

NumNeighborsType = Union[NumNeighbors, List[int], Dict[EdgeType, List[int]]]


class NeighborSampler(BaseSampler):
    r"""An implementation of an in-memory (heterogeneous) neighbor sampler used
    by :class:`~torch_geometric.loader.NeighborLoader`."""
    def __init__(
        self,
        data: Union[Data, HeteroData, Tuple[FeatureStore, GraphStore]],
        num_neighbors: NumNeighborsType,
        subgraph_type: Union[SubgraphType, str] = 'directional',
        replace: bool = False,
        disjoint: bool = False,
        temporal_strategy: str = 'uniform',
        time_attr: Optional[str] = None,
        is_sorted: bool = False,
        share_memory: bool = False,
        # Deprecated:
        directed: bool = True,
    ):
        if not directed:
            subgraph_type = SubgraphType.induced
            warnings.warn(f"The usage of the 'directed' argument in "
                          f"'{self.__class__.__name__}' is deprecated. Use "
                          f"`subgraph_type='induced'` instead.")

        if (not torch_geometric.typing.WITH_PYG_LIB and sys.platform == 'linux'
                and subgraph_type != SubgraphType.induced):
            warnings.warn(f"Using '{self.__class__.__name__}' without a "
                          f"'pyg-lib' installation is deprecated and will be "
                          f"removed soon. Please install 'pyg-lib' for "
                          f"accelerated neighborhood sampling")

        self.data_type = DataType.from_data(data)

        if self.data_type == DataType.homogeneous:
            self.num_nodes = data.num_nodes
            self.node_time = data[time_attr] if time_attr else None

            # Convert the graph data into CSC format for sampling:
            self.colptr, self.row, self.perm = to_csc(
                data, device='cpu', share_memory=share_memory,
                is_sorted=is_sorted, src_node_time=self.node_time)

        elif self.data_type == DataType.heterogeneous:
            self.node_types, self.edge_types = data.metadata()

            self.num_nodes = {k: data[k].num_nodes for k in self.node_types}
            self.node_time = data.collect(time_attr) if time_attr else None

            # Conversion to/from C++ string type: Since C++ cannot take
            # dictionaries with tuples as key as input, edge type triplets need
            # to be converted into single strings.
            self.to_rel_type = {k: '__'.join(k) for k in self.edge_types}
            self.to_edge_type = {v: k for k, v in self.to_rel_type.items()}

            # Convert the graph data into CSC format for sampling:
            colptr_dict, row_dict, self.perm = to_hetero_csc(
                data, device='cpu', share_memory=share_memory,
                is_sorted=is_sorted, node_time_dict=self.node_time)
            self.row_dict = remap_keys(row_dict, self.to_rel_type)
            self.colptr_dict = remap_keys(colptr_dict, self.to_rel_type)

        else:  # self.data_type == DataType.remote
            feature_store, graph_store = data

            # Obtain graph metadata:
            node_attrs = feature_store.get_all_tensor_attrs()
            self.node_types = list(set(attr.group_name for attr in node_attrs))

            edge_attrs = graph_store.get_all_edge_attrs()
            self.edge_types = list(set(attr.edge_type for attr in edge_attrs))

            self.num_nodes = {
                node_type: remote_backend_utils.size(*data, node_type)
                for node_type in self.node_types
            }

            self.node_time: Optional[Dict[str, Tensor]] = None
            if time_attr is not None:
                # If the `time_attr` is present, we expect that `GraphStore`
                # holds all edges sorted by destination, and within local
                # neighborhoods, node indices should be sorted by time.
                # TODO (matthias, manan) Find an alternative way to ensure.
                for edge_attr in edge_attrs:
                    if edge_attr.layout == EdgeLayout.CSR:
                        raise ValueError(
                            "Temporal sampling requires that edges are stored "
                            "in either COO or CSC layout")
                    if not edge_attr.is_sorted:
                        raise ValueError(
                            "Temporal sampling requires that edges are "
                            "sorted by destination, and by source time "
                            "within local neighborhoods")

                # We obtain all features with `node_attr.name=time_attr`:
                time_attrs = [
                    copy.copy(attr) for attr in node_attrs
                    if attr.attr_name == time_attr
                ]
                for attr in time_attrs:  # Reset the index to obtain full data.
                    attr.index = None
                time_tensors = feature_store.multi_get_tensor(time_attrs)
                self.node_time = {
                    time_attr.group_name: time_tensor
                    for time_attr, time_tensor in zip(time_attrs, time_tensors)
                }

            # Conversion to/from C++ string type (see above):
            self.to_rel_type = {k: '__'.join(k) for k in self.edge_types}
            self.to_edge_type = {v: k for k, v in self.to_rel_type.items()}

            # Convert the graph data into CSC format for sampling:
            row_dict, colptr_dict, self.perm = graph_store.csc()
            self.row_dict = remap_keys(row_dict, self.to_rel_type)
            self.colptr_dict = remap_keys(colptr_dict, self.to_rel_type)

        self.num_neighbors = num_neighbors
        self.replace = replace
        self.subgraph_type = SubgraphType(subgraph_type)
        self.disjoint = disjoint
        self.temporal_strategy = temporal_strategy

    @property
    def num_neighbors(self) -> NumNeighbors:
        return self._num_neighbors

    @num_neighbors.setter
    def num_neighbors(self, num_neighbors: NumNeighborsType):
        if isinstance(num_neighbors, NumNeighbors):
            self._num_neighbors = num_neighbors
        else:
            self._num_neighbors = NumNeighbors(num_neighbors)

    @property
    def is_temporal(self) -> bool:
        return self.node_time is not None

    @property
    def disjoint(self) -> bool:
        return self._disjoint or self.is_temporal

    @disjoint.setter
    def disjoint(self, disjoint: bool):
        self._disjoint = disjoint

    # Node-based sampling #####################################################

    def sample_from_nodes(
        self,
        inputs: NodeSamplerInput,
    ) -> Union[SamplerOutput, HeteroSamplerOutput]:
        out = node_sample(inputs, self._sample)
        if self.subgraph_type == SubgraphType.bidirectional:
            out = out.to_bidirectional()
        return out

    # Edge-based sampling #####################################################

    def sample_from_edges(
        self, inputs: EdgeSamplerInput,
        neg_sampling: Optional[NegativeSampling] = None
    ) -> Union[SamplerOutput, HeteroSamplerOutput]:
        out = edge_sample(inputs, self._sample, self.num_nodes, self.disjoint,
                          self.node_time, neg_sampling)
        if self.subgraph_type == SubgraphType.bidirectional:
            out = out.to_bidirectional()
        return out

    # Other Utilities #########################################################

    @property
    def edge_permutation(self) -> Union[OptTensor, Dict[EdgeType, OptTensor]]:
        return self.perm

    # Helper functions ########################################################

    def _sample(
        self,
        seed: Union[Tensor, Dict[NodeType, Tensor]],
        seed_time: Optional[Union[Tensor, Dict[NodeType, Tensor]]] = None,
        **kwargs,
    ) -> Union[SamplerOutput, HeteroSamplerOutput]:
        r"""Implements neighbor sampling by calling either :obj:`pyg-lib` (if
        installed) or :obj:`torch-sparse` (if installed) sampling routines."""
        if isinstance(seed, dict):  # Heterogeneous sampling:
            # TODO Support induced subgraph sampling in `pyg-lib`.
            if (torch_geometric.typing.WITH_PYG_LIB
                    and self.subgraph_type != SubgraphType.induced):
                # TODO (matthias) `return_edge_id` if edge features present
                # TODO (matthias) Ideally, `seed` inherits dtype from `colptr`
                colptrs = list(self.colptr_dict.values())
                dtype = colptrs[0].dtype if len(colptrs) > 0 else torch.int64
                seed = {k: v.to(dtype) for k, v in seed.items()}

                out = torch.ops.pyg.hetero_neighbor_sample(
                    self.node_types,
                    self.edge_types,
                    self.colptr_dict,
                    self.row_dict,
                    seed,
                    self.num_neighbors.get_mapped_values(self.edge_types),
                    self.node_time,
                    seed_time,
                    True,  # csc
                    self.replace,
                    self.subgraph_type != SubgraphType.induced,
                    self.disjoint,
                    self.temporal_strategy,
                    True,  # return_edge_id
                )
                row, col, node, edge, batch = out[:4] + (None, )

                # `pyg-lib>0.1.0` returns sampled number of nodes/edges:
                num_sampled_nodes = num_sampled_edges = None
                if len(out) == 6:
                    num_sampled_nodes, num_sampled_edges = out[4:]

                if self.disjoint:
                    node = {k: v.t().contiguous() for k, v in node.items()}
                    batch = {k: v[0] for k, v in node.items()}
                    node = {k: v[1] for k, v in node.items()}

            elif torch_geometric.typing.WITH_TORCH_SPARSE:
                if self.disjoint:
                    raise ValueError("'disjoint' sampling not supported for "
                                     "neighbor sampling via 'torch-sparse'. "
                                     "Please install 'pyg-lib' for improved "
                                     "and optimized sampling routines.")

                out = torch.ops.torch_sparse.hetero_neighbor_sample(
                    self.node_types,
                    self.edge_types,
                    self.colptr_dict,
                    self.row_dict,
                    seed,  # seed_dict
                    self.num_neighbors.get_mapped_values(self.edge_types),
                    self.num_neighbors.num_hops,
                    self.replace,
                    self.subgraph_type != SubgraphType.induced,
                )
                node, row, col, edge, batch = out + (None, )
                num_sampled_nodes = num_sampled_edges = None

            else:
                raise ImportError(f"'{self.__class__.__name__}' requires "
                                  f"either 'pyg-lib' or 'torch-sparse'")

            if num_sampled_edges is not None:
                num_sampled_edges = remap_keys(
                    num_sampled_edges,
                    self.to_edge_type,
                )

            return HeteroSamplerOutput(
                node=node,
                row=remap_keys(row, self.to_edge_type),
                col=remap_keys(col, self.to_edge_type),
                edge=remap_keys(edge, self.to_edge_type),
                batch=batch,
                num_sampled_nodes=num_sampled_nodes,
                num_sampled_edges=num_sampled_edges,
            )

        else:  # Homogeneous sampling:
            # TODO Support induced subgraph sampling in `pyg-lib`.
            if (torch_geometric.typing.WITH_PYG_LIB
                    and self.subgraph_type != SubgraphType.induced):
                # TODO (matthias) `return_edge_id` if edge features present
                # TODO (matthias) Ideally, `seed` inherits dtype from `colptr`
                out = torch.ops.pyg.neighbor_sample(
                    self.colptr,
                    self.row,
                    seed.to(self.colptr.dtype),  # seed
                    self.num_neighbors.get_mapped_values(),
                    self.node_time,
                    seed_time,
                    True,  # csc
                    self.replace,
                    self.subgraph_type != SubgraphType.induced,
                    self.disjoint,
                    self.temporal_strategy,
                    True,  # return_edge_id
                )
                row, col, node, edge, batch = out[:4] + (None, )

                # `pyg-lib>0.1.0` returns sampled number of nodes/edges:
                num_sampled_nodes = num_sampled_edges = None
                if len(out) == 6:
                    num_sampled_nodes, num_sampled_edges = out[4:]

                if self.disjoint:
                    batch, node = node.t().contiguous()

            elif torch_geometric.typing.WITH_TORCH_SPARSE:
                if self.disjoint:
                    raise ValueError("'disjoint' sampling not supported for "
                                     "neighbor sampling via 'torch-sparse'. "
                                     "Please install 'pyg-lib' for improved "
                                     "and optimized sampling routines.")

                out = torch.ops.torch_sparse.neighbor_sample(
                    self.colptr,
                    self.row,
                    seed,  # seed
                    self.num_neighbors.get_mapped_values(),
                    self.replace,
                    self.subgraph_type != SubgraphType.induced,
                )
                node, row, col, edge, batch = out + (None, )
                num_sampled_nodes = num_sampled_edges = None

            else:
                raise ImportError(f"'{self.__class__.__name__}' requires "
                                  f"either 'pyg-lib' or 'torch-sparse'")

            return SamplerOutput(
                node=node,
                row=row,
                col=col,
                edge=edge,
                batch=batch,
                num_sampled_nodes=num_sampled_nodes,
                num_sampled_edges=num_sampled_edges,
            )


# Sampling Utilities ##########################################################


def node_sample(
    inputs: NodeSamplerInput,
    sample_fn: Callable,
) -> Union[SamplerOutput, HeteroSamplerOutput]:
    r"""Performs sampling from a :class:`NodeSamplerInput`, leveraging a
    sampling function that accepts a seed and (optionally) a seed time as
    input. Returns the output of this sampling procedure."""
    if inputs.input_type is not None:  # Heterogeneous sampling:
        seed = {inputs.input_type: inputs.node}
        seed_time = None
        if inputs.time is not None:
            seed_time = {inputs.input_type: inputs.time}
    else:  # Homogeneous sampling:
        seed = inputs.node
        seed_time = inputs.time

    out = sample_fn(seed, seed_time)
    out.metadata = (inputs.input_id, inputs.time)

    return out


def edge_sample(
    inputs: EdgeSamplerInput,
    sample_fn: Callable,
    num_nodes: Union[int, Dict[NodeType, int]],
    disjoint: bool,
    node_time: Optional[Union[Tensor, Dict[str, Tensor]]] = None,
    neg_sampling: Optional[NegativeSampling] = None,
) -> Union[SamplerOutput, HeteroSamplerOutput]:
    r"""Performs sampling from an edge sampler input, leveraging a sampling
    function of the same signature as `node_sample`."""
    input_id = inputs.input_id
    src = inputs.row
    dst = inputs.col
    edge_label = inputs.label
    edge_label_time = inputs.time
    input_type = inputs.input_type

    src_time = dst_time = edge_label_time
    assert edge_label_time is None or disjoint

    assert isinstance(num_nodes, (dict, int))
    if not isinstance(num_nodes, dict):
        num_src_nodes = num_dst_nodes = num_nodes
    else:
        num_src_nodes = num_nodes[input_type[0]]
        num_dst_nodes = num_nodes[input_type[-1]]

    num_pos = src.numel()
    num_neg = 0

    # Negative Sampling #######################################################

    if neg_sampling is not None:
        # When we are doing negative sampling, we append negative information
        # of nodes/edges to `src`, `dst`, `src_time`, `dst_time`.
        # Later on, we can easily reconstruct what belongs to positive and
        # negative examples by slicing via `num_pos`.
        num_neg = math.ceil(num_pos * neg_sampling.amount)

        if neg_sampling.is_binary():
            # In the "binary" case, we randomly sample negative pairs of nodes.
            if isinstance(node_time, dict):
                src_node_time = node_time.get(input_type[0])
            else:
                src_node_time = node_time

            src_neg = neg_sample(src, neg_sampling, num_src_nodes, src_time,
                                 src_node_time)
            src = torch.cat([src, src_neg], dim=0)

            if isinstance(node_time, dict):
                dst_node_time = node_time.get(input_type[-1])
            else:
                dst_node_time = node_time

            dst_neg = neg_sample(dst, neg_sampling, num_dst_nodes, dst_time,
                                 dst_node_time)
            dst = torch.cat([dst, dst_neg], dim=0)

            if edge_label is None:
                edge_label = torch.ones(num_pos)
            size = (num_neg, ) + edge_label.size()[1:]
            edge_neg_label = edge_label.new_zeros(size)
            edge_label = torch.cat([edge_label, edge_neg_label])

            if edge_label_time is not None:
                src_time = dst_time = edge_label_time.repeat(
                    1 + math.ceil(neg_sampling.amount))[:num_pos + num_neg]

        elif neg_sampling.is_triplet():
            # In the "triplet" case, we randomly sample negative destinations.
            if isinstance(node_time, dict):
                dst_node_time = node_time.get(input_type[-1])
            else:
                dst_node_time = node_time

            dst_neg = neg_sample(dst, neg_sampling, num_dst_nodes, dst_time,
                                 dst_node_time)
            dst = torch.cat([dst, dst_neg], dim=0)

            assert edge_label is None

            if edge_label_time is not None:
                dst_time = edge_label_time.repeat(1 + neg_sampling.amount)

    # Heterogeneus Neighborhood Sampling ######################################

    if input_type is not None:
        seed_time_dict = None
        if input_type[0] != input_type[-1]:  # Two distinct node types:

            if not disjoint:
                src, inverse_src = src.unique(return_inverse=True)
                dst, inverse_dst = dst.unique(return_inverse=True)

            seed_dict = {input_type[0]: src, input_type[-1]: dst}

            if edge_label_time is not None:  # Always disjoint.
                seed_time_dict = {
                    input_type[0]: src_time,
                    input_type[-1]: dst_time,
                }

        else:  # Only a single node type: Merge both source and destination.

            seed = torch.cat([src, dst], dim=0)

            if not disjoint:
                seed, inverse_seed = seed.unique(return_inverse=True)

            seed_dict = {input_type[0]: seed}

            if edge_label_time is not None:  # Always disjoint.
                seed_time_dict = {
                    input_type[0]: torch.cat([src_time, dst_time], dim=0),
                }

        out = sample_fn(seed_dict, seed_time_dict)

        # Enhance `out` by label information ##################################
        if disjoint:
            for key, batch in out.batch.items():
                out.batch[key] = batch % num_pos

        if neg_sampling is None or neg_sampling.is_binary():
            if disjoint:
                if input_type[0] != input_type[-1]:
                    edge_label_index = torch.arange(num_pos + num_neg)
                    edge_label_index = edge_label_index.repeat(2).view(2, -1)
                else:
                    edge_label_index = torch.arange(2 * (num_pos + num_neg))
                    edge_label_index = edge_label_index.view(2, -1)
            else:
                if input_type[0] != input_type[-1]:
                    edge_label_index = torch.stack([
                        inverse_src,
                        inverse_dst,
                    ], dim=0)
                else:
                    edge_label_index = inverse_seed.view(2, -1)

            out.metadata = (input_id, edge_label_index, edge_label, src_time)

        elif neg_sampling.is_triplet():
            if disjoint:
                src_index = torch.arange(num_pos)
                if input_type[0] != input_type[-1]:
                    dst_pos_index = torch.arange(num_pos)
                    # `dst_neg_index` needs to be offset such that indices with
                    # offset `num_pos` belong to the same triplet:
                    dst_neg_index = torch.arange(
                        num_pos, seed_dict[input_type[-1]].numel())
                    dst_neg_index = dst_neg_index.view(-1, num_pos).t()
                else:
                    dst_pos_index = torch.arange(num_pos, 2 * num_pos)
                    dst_neg_index = torch.arange(
                        2 * num_pos, seed_dict[input_type[-1]].numel())
                    dst_neg_index = dst_neg_index.view(-1, num_pos).t()
            else:
                if input_type[0] != input_type[-1]:
                    src_index = inverse_src
                    dst_pos_index = inverse_dst[:num_pos]
                    dst_neg_index = inverse_dst[num_pos:]
                else:
                    src_index = inverse_seed[:num_pos]
                    dst_pos_index = inverse_seed[num_pos:2 * num_pos]
                    dst_neg_index = inverse_seed[2 * num_pos:]

            dst_neg_index = dst_neg_index.view(num_pos, -1).squeeze(-1)

            out.metadata = (input_id, src_index, dst_pos_index, dst_neg_index,
                            src_time)

    # Homogeneus Neighborhood Sampling ########################################

    else:

        seed = torch.cat([src, dst], dim=0)
        seed_time = None

        if not disjoint:
            seed, inverse_seed = seed.unique(return_inverse=True)

        if edge_label_time is not None:  # Always disjoint.
            seed_time = torch.cat([src_time, dst_time])

        out = sample_fn(seed, seed_time)

        # Enhance `out` by label information ##################################
        if neg_sampling is None or neg_sampling.is_binary():
            if disjoint:
                out.batch = out.batch % num_pos
                edge_label_index = torch.arange(2 * seed.numel()).view(2, -1)
            else:
                edge_label_index = inverse_seed.view(2, -1)

            out.metadata = (input_id, edge_label_index, edge_label, seed_time)

        elif neg_sampling.is_triplet():
            if disjoint:
                out.batch = out.batch % num_pos
                src_index = torch.arange(num_pos)
                dst_pos_index = torch.arange(num_pos, 2 * num_pos)
                # `dst_neg_index` needs to be offset such that indices with
                # offset `num_pos` belong to the same triplet:
                dst_neg_index = torch.arange(2 * num_pos, seed.numel())
                dst_neg_index = dst_neg_index.view(-1, num_pos).t()
            else:
                src_index = inverse_seed[:num_pos]
                dst_pos_index = inverse_seed[num_pos:2 * num_pos]
                dst_neg_index = inverse_seed[2 * num_pos:]
            dst_neg_index = dst_neg_index.view(num_pos, -1).squeeze(-1)

            out.metadata = (input_id, src_index, dst_pos_index, dst_neg_index,
                            src_time)

    return out


def neg_sample(seed: Tensor, neg_sampling: NegativeSampling, num_nodes: int,
               seed_time: Optional[Tensor],
               node_time: Optional[Tensor]) -> Tensor:
    num_neg = math.ceil(seed.numel() * neg_sampling.amount)

    # TODO: Do not sample false negatives.
    if node_time is None:
        return neg_sampling.sample(num_neg, num_nodes)

    # If we are in a temporal-sampling scenario, we need to respect the
    # timestamp of the given nodes we can use as negative examples.
    # That is, we can only sample nodes for which `node_time <= seed_time`.
    # For now, we use a greedy algorithm which randomly samples negative
    # nodes and discard any which do not respect the temporal constraint.
    # We iteratively repeat this process until we have sampled a valid node for
    # each seed.
    # TODO See if this greedy algorithm here can be improved.
    assert seed_time is not None
    num_samples = math.ceil(neg_sampling.amount)
    seed_time = seed_time.view(1, -1).expand(num_samples, -1)

    out = neg_sampling.sample(num_samples * seed.numel(), num_nodes)
    out = out.view(num_samples, seed.numel())
    mask = node_time[out] > seed_time  # holds all invalid samples.
    neg_sampling_complete = False
    for i in range(5):  # pragma: no cover
        num_invalid = int(mask.sum())
        if num_invalid == 0:
            neg_sampling_complete = True
            break

        # Greedily search for alternative negatives.
        out[mask] = tmp = neg_sampling.sample(num_invalid, num_nodes)
        mask[mask.clone()] = node_time[tmp] >= seed_time[mask]

    if not neg_sampling_complete:  # pragma: no cover
        # Not much options left. In that case, we set remaining negatives
        # to the node with minimum timestamp.
        out[mask] = node_time.argmin()

    return out.view(-1)[:num_neg]
pytorch/torch_geometric/sampler/utils.py000066400000000000000000000122041444262304300211010ustar00rootroot00000000000000from typing import Any, Dict, List, Optional, Tuple, TypeVar, Union

import numpy as np
import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.storage import EdgeStorage
from torch_geometric.typing import NodeType, OptTensor
from torch_geometric.utils import coalesce, index_sort
from torch_geometric.utils.sparse import index2ptr

# Edge Layout Conversion ######################################################


def sort_csc(
    row: Tensor,
    col: Tensor,
    src_node_time: OptTensor = None,
) -> Tuple[Tensor, Tensor, Tensor]:
    if src_node_time is None:
        col, perm = index_sort(col)
        return row[perm], col, perm
    else:
        # We use `np.lexsort` to sort based on multiple keys.
        # TODO There does not seem to exist a PyTorch equivalent yet :(
        perm = np.lexsort([
            src_node_time[row].detach().cpu().numpy(),
            col.detach().cpu().numpy()
        ])
        perm = torch.from_numpy(perm).to(col.device)

        return row[perm], col[perm], perm


# TODO(manan) deprecate when FeatureStore / GraphStore unification is complete
def to_csc(
    data: Union[Data, EdgeStorage],
    device: Optional[torch.device] = None,
    share_memory: bool = False,
    is_sorted: bool = False,
    src_node_time: Optional[Tensor] = None,
) -> Tuple[Tensor, Tensor, OptTensor]:
    # Convert the graph data into a suitable format for sampling (CSC format).
    # Returns the `colptr` and `row` indices of the graph, as well as an
    # `perm` vector that denotes the permutation of edges.
    # Since no permutation of edges is applied when using `SparseTensor`,
    # `perm` can be of type `None`.
    perm: Optional[Tensor] = None

    if hasattr(data, 'adj'):
        if src_node_time is not None:
            raise NotImplementedError("Temporal sampling via 'SparseTensor' "
                                      "format not yet supported")
        colptr, row, _ = data.adj.csc()

    elif hasattr(data, 'adj_t'):
        if src_node_time is not None:
            # TODO (matthias) This only works when instantiating a
            # `SparseTensor` with `is_sorted=True`. Otherwise, the
            # `SparseTensor` will by default re-sort the neighbors according to
            # column index.
            # As such, we probably want to consider re-adding error:
            # raise NotImplementedError("Temporal sampling via 'SparseTensor' "
            #                           "format not yet supported")
            pass
        colptr, row, _ = data.adj_t.csr()

    elif data.edge_index is not None:
        row, col = data.edge_index
        if not is_sorted:
            row, col, perm = sort_csc(row, col, src_node_time)

        colptr = index2ptr(col, data.size(1))

    else:
        row = torch.empty(0, dtype=torch.long, device=device)
        colptr = torch.zeros(data.num_nodes + 1, dtype=torch.long,
                             device=device)

    colptr = colptr.to(device)
    row = row.to(device)
    perm = perm.to(device) if perm is not None else None

    if not colptr.is_cuda and share_memory:
        colptr.share_memory_()
        row.share_memory_()
        if perm is not None:
            perm.share_memory_()

    return colptr, row, perm


def to_hetero_csc(
    data: HeteroData,
    device: Optional[torch.device] = None,
    share_memory: bool = False,
    is_sorted: bool = False,
    node_time_dict: Optional[Dict[NodeType, Tensor]] = None,
) -> Tuple[Dict[str, Tensor], Dict[str, Tensor], Dict[str, OptTensor]]:
    # Convert the heterogeneous graph data into a suitable format for sampling
    # (CSC format).
    # Returns dictionaries holding `colptr` and `row` indices as well as edge
    # permutations for each edge type, respectively.
    colptr_dict, row_dict, perm_dict = {}, {}, {}

    for edge_type, store in data.edge_items():
        src_node_time = (node_time_dict or {}).get(edge_type[0], None)
        out = to_csc(store, device, share_memory, is_sorted, src_node_time)
        colptr_dict[edge_type], row_dict[edge_type], perm_dict[edge_type] = out

    return colptr_dict, row_dict, perm_dict


def to_bidirectional(
    row: Tensor,
    col: Tensor,
    rev_row: Tensor,
    rev_col: Tensor,
    edge_id: OptTensor = None,
    rev_edge_id: OptTensor = None,
) -> Tuple[Tensor, Tensor, OptTensor]:

    assert row.numel() == col.numel()
    assert rev_row.numel() == rev_col.numel()

    edge_index = row.new_empty(2, row.numel() + rev_row.numel())
    edge_index[0, :row.numel()] = row
    edge_index[1, :row.numel()] = col
    edge_index[0, row.numel():] = rev_col
    edge_index[1, row.numel():] = rev_row

    if edge_id is not None:
        edge_id = torch.cat([edge_id, rev_edge_id], dim=0)

    (row, col), edge_id = coalesce(edge_index, edge_id, reduce='any')

    return row, col, edge_id


###############################################################################

X, Y = TypeVar('X'), TypeVar('Y')


def remap_keys(
    inputs: Dict[X, Any],
    mapping: Dict[X, Y],
    exclude: Optional[List[X]] = None,
) -> Dict[Union[X, Y], Any]:
    exclude = exclude or []
    return {(k if k in exclude else mapping.get(k, k)): v
            for k, v in inputs.items()}
pytorch/torch_geometric/seed.py000066400000000000000000000005421444262304300172200ustar00rootroot00000000000000import random

import numpy as np
import torch


def seed_everything(seed: int):
    r"""Sets the seed for generating random numbers in :pytorch:`PyTorch`,
    :obj:`numpy` and Python.

    Args:
        seed (int): The desired seed.
    """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
pytorch/torch_geometric/testing/000077500000000000000000000000001444262304300174025ustar00rootroot00000000000000pytorch/torch_geometric/testing/__init__.py000066400000000000000000000013061444262304300215130ustar00rootroot00000000000000from .decorators import (
    is_full_test,
    onlyFullTest,
    onlyLinux,
    onlyPython,
    onlyCUDA,
    onlyGraphviz,
    onlyNeighborSampler,
    withPackage,
    withCUDA,
    disableExtensions,
)
from .asserts import assert_module
from .feature_store import MyFeatureStore
from .graph_store import MyGraphStore
from .data import FakeHeteroDataset, get_random_edge_index

__all__ = [
    'is_full_test',
    'onlyFullTest',
    'onlyLinux',
    'onlyPython',
    'onlyCUDA',
    'onlyGraphviz',
    'onlyNeighborSampler',
    'withPackage',
    'withCUDA',
    'disableExtensions',
    'assert_module',
    'MyFeatureStore',
    'MyGraphStore',
    'get_random_edge_index',
    'FakeHeteroDataset',
]
pytorch/torch_geometric/testing/asserts.py000066400000000000000000000104201444262304300214350ustar00rootroot00000000000000import copy
import warnings
from typing import Any, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import WITH_TORCH_SPARSE, SparseTensor
from torch_geometric.utils import to_torch_coo_tensor, to_torch_csc_tensor

SPARSE_LAYOUTS = ['torch_sparse', torch.sparse_csc, torch.sparse_coo]


def assert_module(
    module: torch.nn.Module,
    x: Any,
    edge_index: Tensor,
    *,
    expected_size: Tuple[int, ...],
    test_edge_permutation: bool = True,
    test_node_permutation: bool = False,
    test_sparse_layouts: List[Union[str, int]] = SPARSE_LAYOUTS,
    sparse_size: Optional[Tuple[int, int]] = None,
    atol: float = 1e-08,
    rtol: float = 1e-05,
    equal_nan: bool = False,
    **kwargs,
):
    r"""Asserts that the output of a :obj:`module` is correct. Specifically,
    this method tests that:

    1. The module output has the correct shape.
    2. The module is invariant to the permutation of edges.
    3. The module is invariant to the permutation of nodes.
    4. The module is invariant to the layout of :obj:`edge_index`.

    Args:
        module (torch.nn.Module): The module to test.
        x (Any): The input features to the module.
        edge_index (torch.Tensor): The input edge indices.
        expected_size (Tuple[int, ...]): The expected output size.
        test_edge_permutation (bool, optional): If set to :obj:`False`, will
            not test the module for edge permutation invariance.
        test_node_permutation (bool, optional): If set to :obj:`False`, will
            not test the module for node permutation invariance.
        test_sparse_layouts (List[str or int], optional): The sparse layouts to
            test for module invariance. (default: :obj:`["torch_sparse",
            torch.sparse_csc, torch.sparse_coo]`)
        sparse_size (Tuple[int, int], optional): The size of the sparse
            adjacency matrix. If not given, will try to automatically infer it.
            (default: :obj:`None`)
        atol (float, optional): Absolute tolerance. (default: :obj:`1e-08`)
        rtol (float, optional): Relative tolerance. (default: :obj:`1e-05`)
        equal_nan (bool, optional): If set to :obj:`True`, then two :obj:`NaN`s
            will be considered equal. (default: :obj:`False`)
    """
    if sparse_size is None:
        if 'size' in kwargs:
            sparse_size = kwargs['size']
        elif isinstance(x, Tensor):
            sparse_size = (x.size(0), x.size(0))
        elif (isinstance(x, (tuple, list)) and isinstance(x[0], Tensor)
              and isinstance(x[1], Tensor)):
            sparse_size = (x[0].size(0), x[1].size(0))

    if len(test_sparse_layouts) > 0 and sparse_size is None:
        raise ValueError(f"Got sparse layouts {test_sparse_layouts}, but no "
                         f"'sparse_size' were specified")

    expected = module(x, edge_index=edge_index, **kwargs)
    assert expected.size() == expected_size

    if test_edge_permutation:
        perm = torch.randperm(edge_index.size(1))
        perm_kwargs = copy.copy(kwargs)
        for key, value in kwargs.items():
            if isinstance(value, Tensor) and value.size(0) == perm.numel():
                perm_kwargs[key] = value[perm]
        out = module(x, edge_index[:, perm], **perm_kwargs)
        assert torch.allclose(out, expected, rtol, atol, equal_nan)

    if test_node_permutation:
        raise NotImplementedError

    for layout in (test_sparse_layouts or []):
        # TODO Add support for values.
        if layout == 'torch_sparse':
            if not WITH_TORCH_SPARSE:
                continue

            adj = SparseTensor.from_edge_index(
                edge_index,
                sparse_sizes=sparse_size,
            )
            adj_t = adj.t()

        elif layout == torch.sparse_csc:
            adj = to_torch_csc_tensor(edge_index, size=sparse_size)
            adj_t = adj.t()

        elif layout == torch.sparse_coo:
            warnings.filterwarnings('ignore', ".*to CSR format.*")
            adj = to_torch_coo_tensor(edge_index, size=sparse_size)
            adj_t = adj.t().coalesce()

        else:
            raise ValueError(f"Got invalid sparse layout '{layout}'")

        out = module(x, adj_t, **kwargs)
        assert torch.allclose(out, expected, rtol, atol, equal_nan)

    return expected
pytorch/torch_geometric/testing/data.py000066400000000000000000000036151444262304300206720ustar00rootroot00000000000000from typing import Callable, Optional

import torch
from torch import Tensor

from torch_geometric.data import HeteroData, InMemoryDataset
from torch_geometric.utils import coalesce as coalesce_fn


def get_random_edge_index(
    num_src_nodes: int,
    num_dst_nodes: int,
    num_edges: int,
    dtype: Optional[torch.dtype] = None,
    device: Optional[torch.device] = None,
    coalesce: bool = False,
) -> Tensor:
    row = torch.randint(num_src_nodes, (num_edges, ), dtype=dtype,
                        device=device)
    col = torch.randint(num_dst_nodes, (num_edges, ), dtype=dtype,
                        device=device)
    edge_index = torch.stack([row, col], dim=0)

    if coalesce:
        edge_index = coalesce_fn(edge_index)

    return edge_index


class FakeHeteroDataset(InMemoryDataset):
    def __init__(self, transform: Optional[Callable] = None):
        super().__init__(transform=transform)

        data = HeteroData()

        num_papers = 100
        num_authors = 10

        data['paper'].x = torch.randn(num_papers, 16)
        data['author'].x = torch.randn(num_authors, 8)

        edge_index = get_random_edge_index(
            num_src_nodes=num_papers,
            num_dst_nodes=num_authors,
            num_edges=300,
        )
        data['paper', 'author'].edge_index = edge_index
        data['author', 'paper'].edge_index = edge_index.flip([0])

        data['paper'].y = torch.randint(0, 4, (num_papers, ))

        perm = torch.randperm(num_papers)
        data['paper'].train_mask = torch.zeros(num_papers, dtype=torch.bool)
        data['paper'].train_mask[perm[0:60]] = True
        data['paper'].val_mask = torch.zeros(num_papers, dtype=torch.bool)
        data['paper'].val_mask[perm[60:80]] = True
        data['paper'].test_mask = torch.zeros(num_papers, dtype=torch.bool)
        data['paper'].test_mask[perm[80:100]] = True

        self.data, self.slices = self.collate([data])
pytorch/torch_geometric/testing/decorators.py000066400000000000000000000074021444262304300221240ustar00rootroot00000000000000import os
import sys
from importlib import import_module
from importlib.util import find_spec
from typing import Callable

import torch
from packaging.requirements import Requirement

from torch_geometric.typing import WITH_PYG_LIB, WITH_TORCH_SPARSE
from torch_geometric.visualization.graph import has_graphviz


def is_full_test() -> bool:
    r"""Whether to run the full but time-consuming test suite."""
    return os.getenv('FULL_TEST', '0') == '1'


def onlyFullTest(func: Callable) -> Callable:
    r"""A decorator to specify that this function belongs to the full test
    suite."""
    import pytest
    return pytest.mark.skipif(
        not is_full_test(),
        reason="Fast test run",
    )(func)


def onlyLinux(func: Callable) -> Callable:
    r"""A decorator to specify that this function should only execute on
    Linux systems."""
    import pytest
    return pytest.mark.skipif(
        sys.platform != 'linux',
        reason="No Linux system",
    )(func)


def onlyPython(*args) -> Callable:
    r"""A decorator to skip tests for any Python version not listed."""
    def decorator(func: Callable) -> Callable:
        import pytest

        python_version = f'{sys.version_info.major}.{sys.version_info.minor}'
        return pytest.mark.skipif(
            python_version not in args,
            reason=f"Python {python_version} not supported",
        )(func)

    return decorator


def onlyCUDA(func: Callable) -> Callable:
    r"""A decorator to skip tests if CUDA is not found."""
    import pytest
    return pytest.mark.skipif(
        not torch.cuda.is_available(),
        reason="CUDA not available",
    )(func)


def onlyGraphviz(func: Callable) -> Callable:
    r"""A decorator to specify that this function should only execute in case
    :obj:`graphviz` is installed."""
    import pytest
    return pytest.mark.skipif(
        not has_graphviz(),
        reason="Graphviz not installed",
    )(func)


def onlyNeighborSampler(func: Callable):
    r"""A decorator to skip tests if no neighborhood sampler package is
    installed."""
    import pytest
    return pytest.mark.skipif(
        not WITH_PYG_LIB and not WITH_TORCH_SPARSE,
        reason="No neighbor sampler installed",
    )(func)


def withPackage(*args) -> Callable:
    r"""A decorator to skip tests if certain packages are not installed.
    Also supports version specification."""
    def is_installed(package: str) -> bool:
        req = Requirement(package)
        if find_spec(req.name) is None:
            return False
        module = import_module(req.name)
        if not hasattr(module, '__version__'):
            return True

        version = module.__version__
        # `req.specifier` does not support `.dev` suffixes, e.g., for
        # `pyg_lib==0.1.0.dev*`, so we manually drop them:
        if '.dev' in version:
            version = '.'.join(version.split('.dev')[:-1])

        return version in req.specifier

    na_packages = set(package for package in args if not is_installed(package))

    def decorator(func: Callable) -> Callable:
        import pytest
        return pytest.mark.skipif(
            len(na_packages) > 0,
            reason=f"Package(s) {na_packages} are not installed",
        )(func)

    return decorator


def withCUDA(func: Callable):
    r"""A decorator to test both on CPU and CUDA (if available)."""
    import pytest

    devices = [torch.device('cpu')]
    if torch.cuda.is_available():
        devices.append(torch.device('cuda:0'))

    return pytest.mark.parametrize('device', devices)(func)


def disableExtensions(func: Callable):
    r"""A decorator to temporarily disable the usage of the
    :obj:`torch_scatter`, :obj:`torch_sparse` and :obj:`pyg_lib` extension
    packages."""
    import pytest

    return pytest.mark.usefixtures('disable_extensions')(func)
pytorch/torch_geometric/testing/feature_store.py000066400000000000000000000034771444262304300226360ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.data import FeatureStore, TensorAttr
from torch_geometric.typing import FeatureTensorType


class MyFeatureStore(FeatureStore):
    def __init__(self):
        super().__init__()
        self.store: Dict[Tuple[str, str], Tensor] = {}

    @staticmethod
    def key(attr: TensorAttr) -> str:
        return (attr.group_name, attr.attr_name)

    def _put_tensor(self, tensor: FeatureTensorType, attr: TensorAttr) -> bool:
        index = attr.index

        # None indices define the obvious index:
        if index is None:
            index = torch.arange(0, tensor.shape[0])

        # Store the index:
        self.store[self.key(attr)] = (index, tensor)

        return True

    def _get_tensor(self, attr: TensorAttr) -> Optional[FeatureTensorType]:
        index, tensor = self.store.get(self.key(attr), (None, None))
        if tensor is None:
            return None

        # None indices return the whole tensor:
        if attr.index is None:
            return tensor

        # Empty slices return the whole tensor:
        if (isinstance(attr.index, slice)
                and attr.index == slice(None, None, None)):
            return tensor

        idx = (torch.cat([(index == v).nonzero() for v in attr.index]).view(-1)
               if attr.index.numel() > 0 else [])
        return tensor[idx]

    def _remove_tensor(self, attr: TensorAttr) -> bool:
        return self.store.pop(self.key(attr), None) is not None

    def _get_tensor_size(self, attr: TensorAttr) -> Tuple[int, ...]:
        return self._get_tensor(attr).size()

    def get_all_tensor_attrs(self) -> List[TensorAttr]:
        return [self._tensor_attr_cls.cast(*key) for key in self.store.keys()]

    def __len__(self):
        raise NotImplementedError
pytorch/torch_geometric/testing/graph_store.py000066400000000000000000000017611444262304300222760ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple

from torch import Tensor

from torch_geometric.data import EdgeAttr, GraphStore
from torch_geometric.typing import EdgeTensorType


class MyGraphStore(GraphStore):
    def __init__(self):
        super().__init__()
        self.store: Dict[Tuple, Tuple[Tensor, Tensor]] = {}

    @staticmethod
    def key(attr: EdgeAttr) -> Tuple:
        return (attr.edge_type, attr.layout.value, attr.is_sorted, attr.size)

    def _put_edge_index(self, edge_index: EdgeTensorType,
                        edge_attr: EdgeAttr) -> bool:
        self.store[self.key(edge_attr)] = edge_index

    def _get_edge_index(self, edge_attr: EdgeAttr) -> Optional[EdgeTensorType]:
        return self.store.get(self.key(edge_attr), None)

    def _remove_edge_index(self, edge_attr: EdgeAttr) -> bool:
        return self.store.pop(self.key(edge_attr), None) is not None

    def get_all_edge_attrs(self) -> List[EdgeAttr]:
        return [EdgeAttr(*key) for key in self.store.keys()]
pytorch/torch_geometric/transforms/000077500000000000000000000000001444262304300201235ustar00rootroot00000000000000pytorch/torch_geometric/transforms/__init__.py000066400000000000000000000100501444262304300222300ustar00rootroot00000000000000# flake8: noqa

from .base_transform import BaseTransform
from .compose import Compose, ComposeFilters
from .to_device import ToDevice
from .to_sparse_tensor import ToSparseTensor
from .constant import Constant
from .normalize_features import NormalizeFeatures
from .svd_feature_reduction import SVDFeatureReduction
from .remove_training_classes import RemoveTrainingClasses
from .random_node_split import RandomNodeSplit
from .random_link_split import RandomLinkSplit
from .node_property_split import NodePropertySplit
from .mask import IndexToMask, MaskToIndex
from .pad import Pad

from .to_undirected import ToUndirected
from .one_hot_degree import OneHotDegree
from .target_indegree import TargetIndegree
from .local_degree_profile import LocalDegreeProfile
from .add_self_loops import AddSelfLoops
from .add_remaining_self_loops import AddRemainingSelfLoops
from .remove_isolated_nodes import RemoveIsolatedNodes
from .remove_duplicated_edges import RemoveDuplicatedEdges
from .knn_graph import KNNGraph
from .radius_graph import RadiusGraph
from .to_dense import ToDense
from .two_hop import TwoHop
from .line_graph import LineGraph
from .laplacian_lambda_max import LaplacianLambdaMax
from .gdc import GDC
from .sign import SIGN
from .gcn_norm import GCNNorm
from .add_metapaths import AddMetaPaths, AddRandomMetaPaths
from .rooted_subgraph import RootedEgoNets, RootedRWSubgraph
from .largest_connected_components import LargestConnectedComponents
from .virtual_node import VirtualNode
from .add_positional_encoding import AddLaplacianEigenvectorPE, AddRandomWalkPE
from .feature_propagation import FeaturePropagation

from .distance import Distance
from .cartesian import Cartesian
from .local_cartesian import LocalCartesian
from .polar import Polar
from .spherical import Spherical
from .point_pair_features import PointPairFeatures
from .center import Center
from .normalize_rotation import NormalizeRotation
from .normalize_scale import NormalizeScale
from .random_jitter import RandomJitter
from .random_flip import RandomFlip
from .linear_transformation import LinearTransformation
from .random_scale import RandomScale
from .random_rotate import RandomRotate
from .random_shear import RandomShear
from .face_to_edge import FaceToEdge
from .sample_points import SamplePoints
from .fixed_points import FixedPoints
from .generate_mesh_normals import GenerateMeshNormals
from .delaunay import Delaunay
from .to_superpixels import ToSLIC
from .grid_sampling import GridSampling

general_transforms = [
    'BaseTransform',
    'Compose',
    'ComposeFilters',
    'ToDevice',
    'ToSparseTensor',
    'Constant',
    'NormalizeFeatures',
    'SVDFeatureReduction',
    'RemoveTrainingClasses',
    'RandomNodeSplit',
    'RandomLinkSplit',
    'NodePropertySplit',
    'IndexToMask',
    'MaskToIndex',
    'Pad',
]

graph_transforms = [
    'ToUndirected',
    'OneHotDegree',
    'TargetIndegree',
    'LocalDegreeProfile',
    'AddSelfLoops',
    'AddRemainingSelfLoops',
    'RemoveIsolatedNodes',
    'RemoveDuplicatedEdges',
    'KNNGraph',
    'RadiusGraph',
    'ToDense',
    'TwoHop',
    'LineGraph',
    'LaplacianLambdaMax',
    'GDC',
    'SIGN',
    'GCNNorm',
    'AddMetaPaths',
    'AddRandomMetaPaths',
    'RootedEgoNets',
    'RootedRWSubgraph',
    'LargestConnectedComponents',
    'VirtualNode',
    'AddLaplacianEigenvectorPE',
    'AddRandomWalkPE',
    'FeaturePropagation',
]

vision_transforms = [
    'Distance',
    'Cartesian',
    'LocalCartesian',
    'Polar',
    'Spherical',
    'PointPairFeatures',
    'Center',
    'NormalizeRotation',
    'NormalizeScale',
    'RandomJitter',
    'RandomFlip',
    'LinearTransformation',
    'RandomScale',
    'RandomRotate',
    'RandomShear',
    'FaceToEdge',
    'SamplePoints',
    'FixedPoints',
    'GenerateMeshNormals',
    'Delaunay',
    'ToSLIC',
    'GridSampling',
]

__all__ = general_transforms + graph_transforms + vision_transforms

from torch_geometric.deprecation import deprecated

RandomTranslate = deprecated("use 'transforms.RandomJitter' instead",
                             'transforms.RandomTranslate')(RandomJitter)
pytorch/torch_geometric/transforms/add_metapaths.py000066400000000000000000000332141444262304300232760ustar00rootroot00000000000000import warnings
from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.typing import EdgeType, SparseTensor
from torch_geometric.utils import coalesce, degree


@functional_transform('add_metapaths')
class AddMetaPaths(BaseTransform):
    r""" Adds additional edge types to a
    :class:`~torch_geometric.data.HeteroData` object between the source node
    type and the destination node type of a given :obj:`metapath`, as described
    in the `"Heterogenous Graph Attention Networks"
    `_ paper
    (functional name: :obj:`add_metapaths`).
    Meta-path based neighbors can exploit different aspects of structure
    information in heterogeneous graphs.
    Formally, a metapath is a path of the form

    .. math::

        \mathcal{V}_1 \xrightarrow{R_1} \mathcal{V}_2 \xrightarrow{R_2} \ldots
        \xrightarrow{R_{\ell-1}} \mathcal{V}_{\ell}

    in which :math:`\mathcal{V}_i` represents node types, and :math:`R_j`
    represents the edge type connecting two node types.
    The added edge type is given by the sequential multiplication  of
    adjacency matrices along the metapath, and is added to the
    :class:`~torch_geometric.data.HeteroData` object as edge type
    :obj:`(src_node_type, "metapath_*", dst_node_type)`, where
    :obj:`src_node_type` and :obj:`dst_node_type` denote :math:`\mathcal{V}_1`
    and :math:`\mathcal{V}_{\ell}`, repectively.

    In addition, a :obj:`metapath_dict` object is added to the
    :class:`~torch_geometric.data.HeteroData` object which maps the
    metapath-based edge type to its original metapath.

    .. code-block:: python

        from torch_geometric.datasets import DBLP
        from torch_geometric.data import HeteroData
        from torch_geometric.transforms import AddMetaPaths

        data = DBLP(root)[0]
        # 4 node types: "paper", "author", "conference", and "term"
        # 6 edge types: ("paper","author"), ("author", "paper"),
        #               ("paper, "term"), ("paper", "conference"),
        #               ("term, "paper"), ("conference", "paper")

        # Add two metapaths:
        # 1. From "paper" to "paper" through "conference"
        # 2. From "author" to "conference" through "paper"
        metapaths = [[("paper", "conference"), ("conference", "paper")],
                     [("author", "paper"), ("paper", "conference")]]
        data = AddMetaPaths(metapaths)(data)

        print(data.edge_types)
        >>> [("author", "to", "paper"), ("paper", "to", "author"),
             ("paper", "to", "term"), ("paper", "to", "conference"),
             ("term", "to", "paper"), ("conference", "to", "paper"),
             ("paper", "metapath_0", "paper"),
             ("author", "metapath_1", "conference")]

        print(data.metapath_dict)
        >>> {("paper", "metapath_0", "paper"): [("paper", "conference"),
                                                ("conference", "paper")],
             ("author", "metapath_1", "conference"): [("author", "paper"),
                                                      ("paper", "conference")]}

    Args:
        metapaths (List[List[Tuple[str, str, str]]]): The metapaths described
            by a list of lists of
            :obj:`(src_node_type, rel_type, dst_node_type)` tuples.
        drop_orig_edge_types (bool, optional): If set to :obj:`True`, existing
            edge types will be dropped. (default: :obj:`False`)
        keep_same_node_type (bool, optional): If set to :obj:`True`, existing
            edge types between the same node type are not dropped even in case
            :obj:`drop_orig_edge_types` is set to :obj:`True`.
            (default: :obj:`False`)
        drop_unconnected_node_types (bool, optional): If set to :obj:`True`,
            will drop node types not connected by any edge type.
            (default: :obj:`False`)
        max_sample (int, optional): If set, will sample at maximum
            :obj:`max_sample` neighbors within metapaths. Useful in order to
            tackle very dense metapath edges. (default: :obj:`None`)
        weighted (bool, optional): If set to :obj:`True` compute weights for
            each metapath edge and store them in :obj:`edge_weight`. The weight
            of each metapath edge is computed as the number of metapaths from
            the start to the end of the metapath edge.
            (default :obj:`False`)
    """
    def __init__(
        self,
        metapaths: List[List[EdgeType]],
        drop_orig_edge_types: bool = False,
        keep_same_node_type: bool = False,
        drop_unconnected_node_types: bool = False,
        max_sample: Optional[int] = None,
        weighted: bool = False,
        **kwargs,
    ):

        if 'drop_orig_edges' in kwargs:
            warnings.warn("'drop_orig_edges' is dprecated. Use "
                          "'drop_orig_edge_types' instead")
            drop_orig_edge_types = kwargs['drop_orig_edges']

        if 'drop_unconnected_nodes' in kwargs:
            warnings.warn("'drop_unconnected_nodes' is dprecated. Use "
                          "'drop_unconnected_node_types' instead")
            drop_unconnected_node_types = kwargs['drop_unconnected_nodes']

        for path in metapaths:
            assert len(path) >= 2, f"Invalid metapath '{path}'"
            assert all([
                j[-1] == path[i + 1][0] for i, j in enumerate(path[:-1])
            ]), f"Invalid sequence of node types in '{path}'"

        self.metapaths = metapaths
        self.drop_orig_edge_types = drop_orig_edge_types
        self.keep_same_node_type = keep_same_node_type
        self.drop_unconnected_node_types = drop_unconnected_node_types
        self.max_sample = max_sample
        self.weighted = weighted

    def forward(self, data: HeteroData) -> HeteroData:
        edge_types = data.edge_types  # save original edge types
        data.metapath_dict = {}

        for j, metapath in enumerate(self.metapaths):
            for edge_type in metapath:
                assert data._to_canonical(
                    edge_type) in edge_types, f"'{edge_type}' not present"

            edge_type = metapath[0]
            edge_weight = self._get_edge_weight(data, edge_type)
            adj1 = SparseTensor.from_edge_index(
                edge_index=data[edge_type].edge_index,
                sparse_sizes=data[edge_type].size(), edge_attr=edge_weight)

            if self.max_sample is not None:
                adj1 = self.sample_adj(adj1)

            for i, edge_type in enumerate(metapath[1:]):
                edge_weight = self._get_edge_weight(data, edge_type)
                adj2 = SparseTensor.from_edge_index(
                    edge_index=data[edge_type].edge_index,
                    sparse_sizes=data[edge_type].size(), edge_attr=edge_weight)

                adj1 = adj1 @ adj2

                if self.max_sample is not None:
                    adj1 = self.sample_adj(adj1)

            row, col, edge_weight = adj1.coo()
            new_edge_type = (metapath[0][0], f'metapath_{j}', metapath[-1][-1])
            data[new_edge_type].edge_index = torch.vstack([row, col])
            if self.weighted:
                data[new_edge_type].edge_weight = edge_weight
            data.metapath_dict[new_edge_type] = metapath

        postprocess(data, edge_types, self.drop_orig_edge_types,
                    self.keep_same_node_type, self.drop_unconnected_node_types)

        return data

    def sample_adj(self, adj: SparseTensor) -> SparseTensor:
        row, col, _ = adj.coo()
        deg = degree(row, num_nodes=adj.size(0))
        prob = (self.max_sample * (1. / deg))[row]
        mask = torch.rand_like(prob) < prob
        return adj.masked_select_nnz(mask, layout='coo')

    def _get_edge_weight(self, data: HeteroData,
                         edge_type: EdgeType) -> torch.Tensor:
        if self.weighted:
            edge_weight = data[edge_type].get('edge_weight', None)
            if edge_weight is None:
                edge_weight = torch.ones(
                    data[edge_type].num_edges,
                    device=data[edge_type].edge_index.device)
            assert edge_weight.ndim == 1
        else:
            edge_weight = None
        return edge_weight


@functional_transform('add_random_metapaths')
class AddRandomMetaPaths(BaseTransform):
    r""" Adds additional edge types similar to :class:`AddMetaPaths`.
    The key difference is that the added edge type is given by
    multiple random walks along the metapath.
    One might want to increase the number of random walks
    via :obj:`walks_per_node` to achieve competitive performance with
    :class:`AddMetaPaths`.

    Args:
        metapaths (List[List[Tuple[str, str, str]]]): The metapaths described
            by a list of lists of
            :obj:`(src_node_type, rel_type, dst_node_type)` tuples.
        drop_orig_edge_types (bool, optional): If set to :obj:`True`, existing
            edge types will be dropped. (default: :obj:`False`)
        keep_same_node_type (bool, optional): If set to :obj:`True`, existing
            edge types between the same node type are not dropped even in case
            :obj:`drop_orig_edge_types` is set to :obj:`True`.
            (default: :obj:`False`)
        drop_unconnected_node_types (bool, optional): If set to :obj:`True`,
            will drop node types not connected by any edge type.
            (default: :obj:`False`)
        walks_per_node (int, List[int], optional): The number of random walks
            for each starting node in a metapth. (default: :obj:`1`)
        sample_ratio (float, optional): The ratio of source nodes to start
            random walks from. (default: :obj:`1.0`)
    """
    def __init__(
        self,
        metapaths: List[List[EdgeType]],
        drop_orig_edge_types: bool = False,
        keep_same_node_type: bool = False,
        drop_unconnected_node_types: bool = False,
        walks_per_node: Union[int, List[int]] = 1,
        sample_ratio: float = 1.0,
    ):

        for path in metapaths:
            assert len(path) >= 2, f"Invalid metapath '{path}'"
            assert all([
                j[-1] == path[i + 1][0] for i, j in enumerate(path[:-1])
            ]), f"Invalid sequence of node types in '{path}'"

        self.metapaths = metapaths
        self.drop_orig_edge_types = drop_orig_edge_types
        self.keep_same_node_type = keep_same_node_type
        self.drop_unconnected_node_types = drop_unconnected_node_types
        self.sample_ratio = sample_ratio
        if isinstance(walks_per_node, int):
            walks_per_node = [walks_per_node] * len(metapaths)
        assert len(walks_per_node) == len(metapaths)
        self.walks_per_node = walks_per_node

    def forward(self, data: HeteroData) -> HeteroData:
        edge_types = data.edge_types  # save original edge types
        data.metapath_dict = {}

        for j, metapath in enumerate(self.metapaths):
            for edge_type in metapath:
                assert data._to_canonical(
                    edge_type) in edge_types, f"'{edge_type}' not present"

            src_node = metapath[0][0]
            num_nodes = data[src_node].num_nodes
            num_starts = round(num_nodes * self.sample_ratio)
            row = start = torch.randperm(num_nodes)[:num_starts].repeat(
                self.walks_per_node[j])

            for i, edge_type in enumerate(metapath):
                edge_index = data[edge_type].edge_index
                adj = SparseTensor(row=edge_index[0], col=edge_index[1],
                                   sparse_sizes=data[edge_type].size())
                col, mask = self.sample(adj, start)
                row, col = row[mask], col[mask]
                start = col

            new_edge_type = (metapath[0][0], f'metapath_{j}', metapath[-1][-1])
            data[new_edge_type].edge_index = coalesce(torch.vstack([row, col]))
            data.metapath_dict[new_edge_type] = metapath

        postprocess(data, edge_types, self.drop_orig_edge_types,
                    self.keep_same_node_type, self.drop_unconnected_node_types)

        return data

    @staticmethod
    def sample(adj: SparseTensor, subset: Tensor) -> Tuple[Tensor, Tensor]:
        """Sample a neighbor form :obj:`adj` for each node in :obj:`subset`."""

        rowcount = adj.storage.rowcount()[subset]
        mask = rowcount > 0
        offset = torch.zeros_like(subset)
        offset[mask] = adj.storage.rowptr()[subset[mask]]

        rand = torch.rand((rowcount.size(0), 1), device=subset.device)
        rand.mul_(rowcount.to(rand.dtype).view(-1, 1))
        rand = rand.to(torch.long)
        rand.add_(offset.view(-1, 1))
        col = adj.storage.col()[rand].squeeze()
        return col, mask

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'sample_ratio={self.sample_ratio}, '
                f'walks_per_node={self.walks_per_node})')


def postprocess(data: HeteroData, edge_types: List[EdgeType],
                drop_orig_edge_types: bool, keep_same_node_type: bool,
                drop_unconnected_node_types: bool):

    if drop_orig_edge_types:
        for i in edge_types:
            if keep_same_node_type and i[0] == i[-1]:
                continue
            else:
                del data[i]

    # remove nodes not connected by any edge type.
    if drop_unconnected_node_types:
        new_edge_types = data.edge_types
        node_types = data.node_types
        connected_nodes = set()
        for i in new_edge_types:
            connected_nodes.add(i[0])
            connected_nodes.add(i[-1])
        for node in node_types:
            if node not in connected_nodes:
                del data[node]
pytorch/torch_geometric/transforms/add_positional_encoding.py000066400000000000000000000113441444262304300253370ustar00rootroot00000000000000from typing import Any, Optional

import numpy as np
import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import (
    get_laplacian,
    get_self_loop_attr,
    scatter,
    to_edge_index,
    to_scipy_sparse_matrix,
    to_torch_csr_tensor,
)


def add_node_attr(data: Data, value: Any,
                  attr_name: Optional[str] = None) -> Data:
    # TODO Move to `BaseTransform`.
    if attr_name is None:
        if 'x' in data:
            x = data.x.view(-1, 1) if data.x.dim() == 1 else data.x
            data.x = torch.cat([x, value.to(x.device, x.dtype)], dim=-1)
        else:
            data.x = value
    else:
        data[attr_name] = value

    return data


@functional_transform('add_laplacian_eigenvector_pe')
class AddLaplacianEigenvectorPE(BaseTransform):
    r"""Adds the Laplacian eigenvector positional encoding from the
    `"Benchmarking Graph Neural Networks" `_
    paper to the given graph
    (functional name: :obj:`add_laplacian_eigenvector_pe`).

    Args:
        k (int): The number of non-trivial eigenvectors to consider.
        attr_name (str, optional): The attribute name of the data object to add
            positional encodings to. If set to :obj:`None`, will be
            concatenated to :obj:`data.x`.
            (default: :obj:`"laplacian_eigenvector_pe"`)
        is_undirected (bool, optional): If set to :obj:`True`, this transform
            expects undirected graphs as input, and can hence speed up the
            computation of eigenvectors. (default: :obj:`False`)
        **kwargs (optional): Additional arguments of
            :meth:`scipy.sparse.linalg.eigs` (when :attr:`is_undirected` is
            :obj:`False`) or :meth:`scipy.sparse.linalg.eigsh` (when
            :attr:`is_undirected` is :obj:`True`).
    """
    def __init__(
        self,
        k: int,
        attr_name: Optional[str] = 'laplacian_eigenvector_pe',
        is_undirected: bool = False,
        **kwargs,
    ):
        self.k = k
        self.attr_name = attr_name
        self.is_undirected = is_undirected
        self.kwargs = kwargs

    def forward(self, data: Data) -> Data:
        from scipy.sparse.linalg import eigs, eigsh
        eig_fn = eigs if not self.is_undirected else eigsh

        num_nodes = data.num_nodes
        edge_index, edge_weight = get_laplacian(
            data.edge_index,
            data.edge_weight,
            normalization='sym',
            num_nodes=num_nodes,
        )

        L = to_scipy_sparse_matrix(edge_index, edge_weight, num_nodes)

        eig_vals, eig_vecs = eig_fn(
            L,
            k=self.k + 1,
            which='SR' if not self.is_undirected else 'SA',
            return_eigenvectors=True,
            **self.kwargs,
        )

        eig_vecs = np.real(eig_vecs[:, eig_vals.argsort()])
        pe = torch.from_numpy(eig_vecs[:, 1:self.k + 1])
        sign = -1 + 2 * torch.randint(0, 2, (self.k, ))
        pe *= sign

        data = add_node_attr(data, pe, attr_name=self.attr_name)
        return data


@functional_transform('add_random_walk_pe')
class AddRandomWalkPE(BaseTransform):
    r"""Adds the random walk positional encoding from the `"Graph Neural
    Networks with Learnable Structural and Positional Representations"
    `_ paper to the given graph
    (functional name: :obj:`add_random_walk_pe`).

    Args:
        walk_length (int): The number of random walk steps.
        attr_name (str, optional): The attribute name of the data object to add
            positional encodings to. If set to :obj:`None`, will be
            concatenated to :obj:`data.x`.
            (default: :obj:`"random_walk_pe"`)
    """
    def __init__(
        self,
        walk_length: int,
        attr_name: Optional[str] = 'random_walk_pe',
    ):
        self.walk_length = walk_length
        self.attr_name = attr_name

    def forward(self, data: Data) -> Data:
        row, col = data.edge_index
        N = data.num_nodes

        value = data.edge_weight
        if value is None:
            value = torch.ones(data.num_edges, device=row.device)
        value = scatter(value, row, dim_size=N, reduce='sum').clamp(min=1)[row]
        value = 1.0 / value

        adj = to_torch_csr_tensor(data.edge_index, value, size=data.size())

        out = adj
        pe_list = [get_self_loop_attr(*to_edge_index(out), num_nodes=N)]
        for _ in range(self.walk_length - 1):
            out = out @ adj
            pe_list.append(get_self_loop_attr(*to_edge_index(out), N))
        pe = torch.stack(pe_list, dim=-1)

        data = add_node_attr(data, pe, attr_name=self.attr_name)
        return data
pytorch/torch_geometric/transforms/add_remaining_self_loops.py000066400000000000000000000040471444262304300255100ustar00rootroot00000000000000from typing import Optional, Union

from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import add_remaining_self_loops


@functional_transform('add_remaining_self_loops')
class AddRemainingSelfLoops(BaseTransform):
    r"""Adds remaining self-loops to the given homogeneous or heterogeneous
    graph (functional name: :obj:`add_remaining_self_loops`).

    Args:
        attr (str, optional): The name of the attribute of edge weights
            or multi-dimensional edge features to pass to
            :meth:`torch_geometric.utils.add_remaining_self_loops`.
            (default: :obj:`"edge_weight"`)
        fill_value (float or Tensor or str, optional): The way to generate
            edge features of self-loops (in case :obj:`attr != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`1.`)
    """
    def __init__(self, attr: Optional[str] = 'edge_weight',
                 fill_value: Union[float, Tensor, str] = 1.0):
        self.attr = attr
        self.fill_value = fill_value

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.edge_stores:
            if store.is_bipartite() or 'edge_index' not in store:
                continue

            store.edge_index, edge_weight = add_remaining_self_loops(
                store.edge_index, getattr(store, self.attr, None),
                fill_value=self.fill_value, num_nodes=store.size(0))

            setattr(store, self.attr, edge_weight)

        return data
pytorch/torch_geometric/transforms/add_self_loops.py000066400000000000000000000037421444262304300234600ustar00rootroot00000000000000from typing import Optional, Union

from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import add_self_loops


@functional_transform('add_self_loops')
class AddSelfLoops(BaseTransform):
    r"""Adds self-loops to the given homogeneous or heterogeneous graph
    (functional name: :obj:`add_self_loops`).

    Args:
        attr (str, optional): The name of the attribute of edge weights
            or multi-dimensional edge features to pass to
            :meth:`torch_geometric.utils.add_self_loops`.
            (default: :obj:`"edge_weight"`)
        fill_value (float or Tensor or str, optional): The way to generate
            edge features of self-loops (in case :obj:`attr != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`1.`)
    """
    def __init__(self, attr: Optional[str] = 'edge_weight',
                 fill_value: Union[float, Tensor, str] = 1.0):
        self.attr = attr
        self.fill_value = fill_value

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.edge_stores:
            if store.is_bipartite() or 'edge_index' not in store:
                continue

            store.edge_index, edge_weight = add_self_loops(
                store.edge_index, getattr(store, self.attr, None),
                fill_value=self.fill_value, num_nodes=store.size(0))

            setattr(store, self.attr, edge_weight)

        return data
pytorch/torch_geometric/transforms/base_transform.py000066400000000000000000000024221444262304300235020ustar00rootroot00000000000000import copy
from abc import ABC
from typing import Any


class BaseTransform(ABC):
    r"""An abstract base class for writing transforms.

    Transforms are a general way to modify and customize
    :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData` objects, either by implicitly
    passing them as an argument to a :class:`~torch_geometric.data.Dataset`, or
    by applying them explicitly to individual
    :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData` objects:

    .. code-block:: python

        import torch_geometric.transforms as T
        from torch_geometric.datasets import TUDataset

        transform = T.Compose([T.ToUndirected(), T.AddSelfLoops()])

        dataset = TUDataset(path, name='MUTAG', transform=transform)
        data = dataset[0]  # Implicitly transform data on every access.

        data = TUDataset(path, name='MUTAG')[0]
        data = transform(data)  # Explicitly transform data.
    """
    def __call__(self, data: Any) -> Any:
        # Shallow-copy the data so that we prevent in-place data modification.
        return self.forward(copy.copy(data))

    def forward(self, data: Any) -> Any:
        pass

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}()'
pytorch/torch_geometric/transforms/cartesian.py000066400000000000000000000036201444262304300224470ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('cartesian')
class Cartesian(BaseTransform):
    r"""Saves the relative Cartesian coordinates of linked nodes in its edge
    attributes (functional name: :obj:`cartesian`).

    Args:
        norm (bool, optional): If set to :obj:`False`, the output will not be
            normalized to the interval :math:`{[0, 1]}^D`.
            (default: :obj:`True`)
        max_value (float, optional): If set and :obj:`norm=True`, normalization
            will be performed based on this value instead of the maximum value
            found in the data. (default: :obj:`None`)
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(
        self,
        norm: bool = True,
        max_value: Optional[float] = None,
        cat: bool = True,
    ):
        self.norm = norm
        self.max = max_value
        self.cat = cat

    def forward(self, data: Data) -> Data:
        (row, col), pos, pseudo = data.edge_index, data.pos, data.edge_attr

        cart = pos[row] - pos[col]
        cart = cart.view(-1, 1) if cart.dim() == 1 else cart

        if self.norm and cart.numel() > 0:
            max_value = cart.abs().max() if self.max is None else self.max
            cart = cart / (2 * max_value) + 0.5

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, cart.type_as(pseudo)], dim=-1)
        else:
            data.edge_attr = cart

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(norm={self.norm}, '
                f'max_value={self.max})')
pytorch/torch_geometric/transforms/center.py000066400000000000000000000012001444262304300217460ustar00rootroot00000000000000from typing import Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('center')
class Center(BaseTransform):
    r"""Centers node positions :obj:`data.pos` around the origin
    (functional name: :obj:`center`)."""
    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.node_stores:
            if hasattr(store, 'pos'):
                store.pos = store.pos - store.pos.mean(dim=-2, keepdim=True)
        return data
pytorch/torch_geometric/transforms/compose.py000066400000000000000000000031721444262304300221450ustar00rootroot00000000000000from typing import Callable, List, Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.transforms import BaseTransform


class Compose(BaseTransform):
    r"""Composes several transforms together.

    Args:
        transforms (List[Callable]): List of transforms to compose.
    """
    def __init__(self, transforms: List[Callable]):
        self.transforms = transforms

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for transform in self.transforms:
            if isinstance(data, (list, tuple)):
                data = [transform(d) for d in data]
            else:
                data = transform(data)
        return data

    def __repr__(self) -> str:
        args = [f'  {transform}' for transform in self.transforms]
        return '{}([\n{}\n])'.format(self.__class__.__name__, ',\n'.join(args))


class ComposeFilters:
    r"""Composes several filters together.

    Args:
        filters (List[Callable]): List of filters to compose.
    """
    def __init__(self, filters: List[Callable]):
        self.filters = filters

    def __call__(
        self,
        data: Union[Data, HeteroData],
    ) -> bool:
        for filter_fn in self.filters:
            if isinstance(data, (list, tuple)):
                if not all([filter_fn(d) for d in data]):
                    return False
            elif not filter_fn(data):
                return False
        return True

    def __repr__(self) -> str:
        args = [f'  {filter_fn}' for filter_fn in self.filters]
        return '{}([\n{}\n])'.format(self.__class__.__name__, ',\n'.join(args))
pytorch/torch_geometric/transforms/constant.py000066400000000000000000000036501444262304300223320ustar00rootroot00000000000000from typing import List, Optional, Union

import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('constant')
class Constant(BaseTransform):
    r"""Appends a constant value to each node feature :obj:`x`
    (functional name: :obj:`constant`).

    Args:
        value (float, optional): The value to add. (default: :obj:`1.0`)
        cat (bool, optional): If set to :obj:`False`, existing node features
            will be replaced. (default: :obj:`True`)
        node_types (str or List[str], optional): The specified node type(s) to
            append constant values for if used on heterogeneous graphs.
            If set to :obj:`None`, constants will be added to each node feature
            :obj:`x` for all existing node types. (default: :obj:`None`)
    """
    def __init__(
        self,
        value: float = 1.0,
        cat: bool = True,
        node_types: Optional[Union[str, List[str]]] = None,
    ):
        if isinstance(node_types, str):
            node_types = [node_types]

        self.value = value
        self.cat = cat
        self.node_types = node_types

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:

        for store in data.node_stores:
            if self.node_types is None or store._key in self.node_types:
                num_nodes = store.num_nodes
                c = torch.full((num_nodes, 1), self.value, dtype=torch.float)

                if hasattr(store, 'x') and self.cat:
                    x = store.x.view(-1, 1) if store.x.dim() == 1 else store.x
                    store.x = torch.cat([x, c.to(x.device, x.dtype)], dim=-1)
                else:
                    store.x = c

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(value={self.value})'
pytorch/torch_geometric/transforms/delaunay.py000066400000000000000000000023061444262304300223000ustar00rootroot00000000000000import scipy.spatial
import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('delaunay')
class Delaunay(BaseTransform):
    r"""Computes the delaunay triangulation of a set of points
    (functional name: :obj:`delaunay`)."""
    def forward(self, data: Data) -> Data:
        if data.pos.size(0) < 2:
            data.edge_index = torch.tensor([], dtype=torch.long,
                                           device=data.pos.device).view(2, 0)
        if data.pos.size(0) == 2:
            data.edge_index = torch.tensor([[0, 1], [1, 0]], dtype=torch.long,
                                           device=data.pos.device)
        elif data.pos.size(0) == 3:
            data.face = torch.tensor([[0], [1], [2]], dtype=torch.long,
                                     device=data.pos.device)
        if data.pos.size(0) > 3:
            pos = data.pos.cpu().numpy()
            tri = scipy.spatial.Delaunay(pos, qhull_options='QJ')
            face = torch.from_numpy(tri.simplices)

            data.face = face.t().contiguous().to(data.pos.device, torch.long)

        return data
pytorch/torch_geometric/transforms/distance.py000066400000000000000000000034211444262304300222670ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('distance')
class Distance(BaseTransform):
    r"""Saves the Euclidean distance of linked nodes in its edge attributes
    (functional name: :obj:`distance`).

    Args:
        norm (bool, optional): If set to :obj:`False`, the output will not be
            normalized to the interval :math:`[0, 1]`. (default: :obj:`True`)
        max_value (float, optional): If set and :obj:`norm=True`, normalization
            will be performed based on this value instead of the maximum value
            found in the data. (default: :obj:`None`)
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(self, norm: bool = True, max_value: Optional[float] = None,
                 cat: bool = True):
        self.norm = norm
        self.max = max_value
        self.cat = cat

    def forward(self, data: Data) -> Data:
        (row, col), pos, pseudo = data.edge_index, data.pos, data.edge_attr

        dist = torch.norm(pos[col] - pos[row], p=2, dim=-1).view(-1, 1)

        if self.norm and dist.numel() > 0:
            dist = dist / (dist.max() if self.max is None else self.max)

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, dist.type_as(pseudo)], dim=-1)
        else:
            data.edge_attr = dist

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(norm={self.norm}, '
                f'max_value={self.max})')
pytorch/torch_geometric/transforms/face_to_edge.py000066400000000000000000000020121444262304300230540ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_undirected


@functional_transform('face_to_edge')
class FaceToEdge(BaseTransform):
    r"""Converts mesh faces :obj:`[3, num_faces]` to edge indices
    :obj:`[2, num_edges]` (functional name: :obj:`face_to_edge`).

    Args:
        remove_faces (bool, optional): If set to :obj:`False`, the face tensor
            will not be removed.
    """
    def __init__(self, remove_faces: bool = True):
        self.remove_faces = remove_faces

    def forward(self, data: Data) -> Data:
        if hasattr(data, 'face'):
            face = data.face
            edge_index = torch.cat([face[:2], face[1:], face[::2]], dim=1)
            edge_index = to_undirected(edge_index, num_nodes=data.num_nodes)

            data.edge_index = edge_index
            if self.remove_faces:
                data.face = None

        return data
pytorch/torch_geometric/transforms/feature_propagation.py000066400000000000000000000056461444262304300245460ustar00rootroot00000000000000from torch import Tensor

import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import is_torch_sparse_tensor, to_torch_csc_tensor


@functional_transform('feature_propagation')
class FeaturePropagation(BaseTransform):
    r"""The feature propagation operator from the `"On the Unreasonable
    Effectiveness of Feature propagation in Learning on Graphs with Missing
    Node Features" `_ paper
    (functional name: :obj:`feature_propagation`)

    .. math::
        \mathbf{X}^{(0)} &= (1 - \mathbf{M}) \cdot \mathbf{X}

        \mathbf{X}^{(\ell + 1)} &= \mathbf{X}^{(0)} + \mathbf{M} \cdot
        (\mathbf{D}^{-1/2} \mathbf{A} \mathbf{D}^{-1/2} \mathbf{X}^{(\ell)})

    where missing node features are inferred by known features via propagation.

    .. code-block:: python

        from torch_geometric.transforms import FeaturePropagation

        transform = FeaturePropagation(missing_mask=torch.isnan(data.x))
        data = transform(data)

    Args:
        missing_mask (torch.Tensor): Mask matrix
            :math:`\mathbf{M} \in {\{ 0, 1 \}}^{N\times F}` indicating missing
            node features.
        num_iterations (int, optional): The number of propagations.
            (default: :obj:`40`)
    """
    def __init__(self, missing_mask: Tensor, num_iterations: int = 40):
        self.missing_mask = missing_mask
        self.num_iterations = num_iterations

    def forward(self, data: Data) -> Data:
        assert 'edge_index' in data or 'adj_t' in data
        assert data.x.size() == self.missing_mask.size()
        gcn_norm = torch_geometric.nn.conv.gcn_conv.gcn_norm

        missing_mask = self.missing_mask.to(data.x.device)
        known_mask = ~missing_mask

        if 'edge_index' in data:
            edge_weight = data.edge_attr
            if 'edge_weight' in data:
                edge_weight = data.edge_weight
            edge_index = data.edge_index
            adj_t = to_torch_csc_tensor(edge_index, edge_weight,
                                        size=data.size()).t()
            adj_t, _ = gcn_norm(adj_t, add_self_loops=False)
        elif is_torch_sparse_tensor(data.adj_t):
            adj_t, _ = gcn_norm(data.adj_t, add_self_loops=False)
        else:
            adj_t = gcn_norm(data.adj_t, add_self_loops=False)

        x = data.x.clone()
        x[missing_mask] = 0.

        out = x
        for _ in range(self.num_iterations):
            out = adj_t @ out
            out[known_mask] = x[known_mask]  # Reset.
        data.x = out

        return data

    def __repr__(self) -> str:
        na_values = int(self.missing_mask.sum()) / self.missing_mask.numel()
        return (f'{self.__class__.__name__}('
                f'missing_features={100 * na_values:.1f}%, '
                f'num_iterations={self.num_iterations})')
pytorch/torch_geometric/transforms/fixed_points.py000066400000000000000000000044771444262304300232040ustar00rootroot00000000000000import math
import re

import numpy as np
import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('fixed_points')
class FixedPoints(BaseTransform):
    r"""Samples a fixed number of points and features from a point cloud
    (functional name: :obj:`fixed_points`).

    Args:
        num (int): The number of points to sample.
        replace (bool, optional): If set to :obj:`False`, samples points
            without replacement. (default: :obj:`True`)
        allow_duplicates (bool, optional): In case :obj:`replace` is
            :obj`False` and :obj:`num` is greater than the number of points,
            this option determines whether to add duplicated nodes to the
            output points or not.
            In case :obj:`allow_duplicates` is :obj:`False`, the number of
            output points might be smaller than :obj:`num`.
            In case :obj:`allow_duplicates` is :obj:`True`, the number of
            duplicated points are kept to a minimum. (default: :obj:`False`)
    """
    def __init__(
        self,
        num: int,
        replace: bool = True,
        allow_duplicates: bool = False,
    ):
        self.num = num
        self.replace = replace
        self.allow_duplicates = allow_duplicates

    def forward(self, data: Data) -> Data:
        num_nodes = data.num_nodes

        if self.replace:
            choice = np.random.choice(num_nodes, self.num, replace=True)
            choice = torch.from_numpy(choice).to(torch.long)
        elif not self.allow_duplicates:
            choice = torch.randperm(num_nodes)[:self.num]
        else:
            choice = torch.cat([
                torch.randperm(num_nodes)
                for _ in range(math.ceil(self.num / num_nodes))
            ], dim=0)[:self.num]

        for key, item in data:
            if key == 'num_nodes':
                data.num_nodes = choice.size(0)
            elif bool(re.search('edge', key)):
                continue
            elif (torch.is_tensor(item) and item.size(0) == num_nodes
                  and item.size(0) != 1):
                data[key] = item[choice]

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num}, replace={self.replace})'
pytorch/torch_geometric/transforms/gcn_norm.py000066400000000000000000000025651444262304300223070ustar00rootroot00000000000000import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('gcn_norm')
class GCNNorm(BaseTransform):
    r"""Applies the GCN normalization from the `"Semi-supervised Classification
    with Graph Convolutional Networks" `_
    paper (functional name: :obj:`gcn_norm`).

    .. math::
        \mathbf{\hat{A}} = \mathbf{\hat{D}}^{-1/2} (\mathbf{A} + \mathbf{I})
        \mathbf{\hat{D}}^{-1/2}

    where :math:`\hat{D}_{ii} = \sum_{j=0} \hat{A}_{ij} + 1`.
    """
    def __init__(self, add_self_loops: bool = True):
        self.add_self_loops = add_self_loops

    def forward(self, data: Data) -> Data:
        gcn_norm = torch_geometric.nn.conv.gcn_conv.gcn_norm
        assert 'edge_index' in data or 'adj_t' in data

        if 'edge_index' in data:
            data.edge_index, data.edge_weight = gcn_norm(
                data.edge_index, data.edge_weight, data.num_nodes,
                add_self_loops=self.add_self_loops)
        else:
            data.adj_t = gcn_norm(data.adj_t,
                                  add_self_loops=self.add_self_loops)

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}('
                f'add_self_loops={self.add_self_loops})')
pytorch/torch_geometric/transforms/gdc.py000066400000000000000000000572271444262304300212470ustar00rootroot00000000000000from typing import Any, Dict, List, Tuple

import numpy as np
import torch
from scipy.linalg import expm
from torch import Tensor

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import (
    add_self_loops,
    coalesce,
    is_undirected,
    scatter,
    to_dense_adj,
)
from torch_geometric.utils.sparse import index2ptr


@functional_transform('gdc')
class GDC(BaseTransform):
    r"""Processes the graph via Graph Diffusion Convolution (GDC) from the
    `"Diffusion Improves Graph Learning" `_
    paper (functional name: :obj:`gdc`).

    .. note::

        The paper offers additional advice on how to choose the
        hyperparameters.
        For an example of using GCN with GDC, see `examples/gcn.py
        `_.

    Args:
        self_loop_weight (float, optional): Weight of the added self-loop.
            Set to :obj:`None` to add no self-loops. (default: :obj:`1`)
        normalization_in (str, optional): Normalization of the transition
            matrix on the original (input) graph. Possible values:
            :obj:`"sym"`, :obj:`"col"`, and :obj:`"row"`.
            See :func:`GDC.transition_matrix` for details.
            (default: :obj:`"sym"`)
        normalization_out (str, optional): Normalization of the transition
            matrix on the transformed GDC (output) graph. Possible values:
            :obj:`"sym"`, :obj:`"col"`, :obj:`"row"`, and :obj:`None`.
            See :func:`GDC.transition_matrix` for details.
            (default: :obj:`"col"`)
        diffusion_kwargs (dict, optional): Dictionary containing the parameters
            for diffusion.
            `method` specifies the diffusion method (:obj:`"ppr"`,
            :obj:`"heat"` or :obj:`"coeff"`).
            Each diffusion method requires different additional parameters.
            See :func:`GDC.diffusion_matrix_exact` or
            :func:`GDC.diffusion_matrix_approx` for details.
            (default: :obj:`dict(method='ppr', alpha=0.15)`)
        sparsification_kwargs (dict, optional): Dictionary containing the
            parameters for sparsification.
            `method` specifies the sparsification method (:obj:`"threshold"` or
            :obj:`"topk"`).
            Each sparsification method requires different additional
            parameters.
            See :func:`GDC.sparsify_dense` for details.
            (default: :obj:`dict(method='threshold', avg_degree=64)`)
        exact (bool, optional): Whether to exactly calculate the diffusion
            matrix.
            Note that the exact variants are not scalable.
            They densify the adjacency matrix and calculate either its inverse
            or its matrix exponential.
            However, the approximate variants do not support edge weights and
            currently only personalized PageRank and sparsification by
            threshold are implemented as fast, approximate versions.
            (default: :obj:`True`)

    :rtype: :class:`torch_geometric.data.Data`
    """
    def __init__(
        self,
        self_loop_weight: float = 1.,
        normalization_in: str = 'sym',
        normalization_out: str = 'col',
        diffusion_kwargs: Dict[str, Any] = dict(method='ppr', alpha=0.15),
        sparsification_kwargs: Dict[str, Any] = dict(method='threshold',
                                                     avg_degree=64),
        exact: bool = True,
    ):

        self.__calc_ppr__ = get_calc_ppr()

        self.self_loop_weight = self_loop_weight
        self.normalization_in = normalization_in
        self.normalization_out = normalization_out
        self.diffusion_kwargs = diffusion_kwargs
        self.sparsification_kwargs = sparsification_kwargs
        self.exact = exact

        if self_loop_weight:
            assert exact or self_loop_weight == 1

    @torch.no_grad()
    def forward(self, data: Data) -> Data:
        N = data.num_nodes
        edge_index = data.edge_index
        if data.edge_attr is None:
            edge_weight = torch.ones(edge_index.size(1),
                                     device=edge_index.device)
        else:
            edge_weight = data.edge_attr
            assert self.exact
            assert edge_weight.dim() == 1

        if self.self_loop_weight:
            edge_index, edge_weight = add_self_loops(
                edge_index, edge_weight, fill_value=self.self_loop_weight,
                num_nodes=N)

        edge_index, edge_weight = coalesce(edge_index, edge_weight, N)

        if self.exact:
            edge_index, edge_weight = self.transition_matrix(
                edge_index, edge_weight, N, self.normalization_in)
            diff_mat = self.diffusion_matrix_exact(edge_index, edge_weight, N,
                                                   **self.diffusion_kwargs)
            edge_index, edge_weight = self.sparsify_dense(
                diff_mat, **self.sparsification_kwargs)
        else:
            edge_index, edge_weight = self.diffusion_matrix_approx(
                edge_index, edge_weight, N, self.normalization_in,
                **self.diffusion_kwargs)
            edge_index, edge_weight = self.sparsify_sparse(
                edge_index, edge_weight, N, **self.sparsification_kwargs)

        edge_index, edge_weight = coalesce(edge_index, edge_weight, N)
        edge_index, edge_weight = self.transition_matrix(
            edge_index, edge_weight, N, self.normalization_out)

        data.edge_index = edge_index
        data.edge_attr = edge_weight

        return data

    def transition_matrix(
        self,
        edge_index: Tensor,
        edge_weight: Tensor,
        num_nodes: int,
        normalization: str,
    ) -> Tuple[Tensor, Tensor]:
        r"""Calculate the approximate, sparse diffusion on a given sparse
        matrix.

        Args:
            edge_index (LongTensor): The edge indices.
            edge_weight (Tensor): One-dimensional edge weights.
            num_nodes (int): Number of nodes.
            normalization (str): Normalization scheme:

                1. :obj:`"sym"`: Symmetric normalization
                   :math:`\mathbf{T} = \mathbf{D}^{-1/2} \mathbf{A}
                   \mathbf{D}^{-1/2}`.
                2. :obj:`"col"`: Column-wise normalization
                   :math:`\mathbf{T} = \mathbf{A} \mathbf{D}^{-1}`.
                3. :obj:`"row"`: Row-wise normalization
                   :math:`\mathbf{T} = \mathbf{D}^{-1} \mathbf{A}`.
                4. :obj:`None`: No normalization.

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        if normalization == 'sym':
            row, col = edge_index
            deg = scatter(edge_weight, col, 0, num_nodes, reduce='sum')
            deg_inv_sqrt = deg.pow(-0.5)
            deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0
            edge_weight = deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]
        elif normalization == 'col':
            _, col = edge_index
            deg = scatter(edge_weight, col, 0, num_nodes, reduce='sum')
            deg_inv = 1. / deg
            deg_inv[deg_inv == float('inf')] = 0
            edge_weight = edge_weight * deg_inv[col]
        elif normalization == 'row':
            row, _ = edge_index
            deg = scatter(edge_weight, row, 0, num_nodes, reduce='sum')
            deg_inv = 1. / deg
            deg_inv[deg_inv == float('inf')] = 0
            edge_weight = edge_weight * deg_inv[row]
        elif normalization is None:
            pass
        else:
            raise ValueError(
                f"Transition matrix normalization '{normalization}' unknown")

        return edge_index, edge_weight

    def diffusion_matrix_exact(
        self,
        edge_index: Tensor,
        edge_weight: Tensor,
        num_nodes: int,
        method: str,
        **kwargs,
    ) -> Tensor:
        r"""Calculate the (dense) diffusion on a given sparse graph.
        Note that these exact variants are not scalable. They densify the
        adjacency matrix and calculate either its inverse or its matrix
        exponential.

        Args:
            edge_index (LongTensor): The edge indices.
            edge_weight (Tensor): One-dimensional edge weights.
            num_nodes (int): Number of nodes.
            method (str): Diffusion method:

                1. :obj:`"ppr"`: Use personalized PageRank as diffusion.
                   Additionally expects the parameter:

                   - **alpha** (*float*) - Return probability in PPR.
                     Commonly lies in :obj:`[0.05, 0.2]`.

                2. :obj:`"heat"`: Use heat kernel diffusion.
                   Additionally expects the parameter:

                   - **t** (*float*) - Time of diffusion. Commonly lies in
                     :obj:`[2, 10]`.

                3. :obj:`"coeff"`: Freely choose diffusion coefficients.
                   Additionally expects the parameter:

                   - **coeffs** (*List[float]*) - List of coefficients
                     :obj:`theta_k` for each power of the transition matrix
                     (starting at :obj:`0`).

        :rtype: (:class:`Tensor`)
        """
        if method == 'ppr':
            # Ξ± (I_n + (Ξ± - 1) A)^-1
            edge_weight = (kwargs['alpha'] - 1) * edge_weight
            edge_index, edge_weight = add_self_loops(edge_index, edge_weight,
                                                     fill_value=1,
                                                     num_nodes=num_nodes)
            mat = to_dense_adj(edge_index, edge_attr=edge_weight).squeeze()
            diff_matrix = kwargs['alpha'] * torch.inverse(mat)

        elif method == 'heat':
            # exp(t (A - I_n))
            edge_index, edge_weight = add_self_loops(edge_index, edge_weight,
                                                     fill_value=-1,
                                                     num_nodes=num_nodes)
            edge_weight = kwargs['t'] * edge_weight
            mat = to_dense_adj(edge_index, edge_attr=edge_weight).squeeze()
            undirected = is_undirected(edge_index, edge_weight, num_nodes)
            diff_matrix = self.__expm__(mat, undirected)

        elif method == 'coeff':
            adj_matrix = to_dense_adj(edge_index,
                                      edge_attr=edge_weight).squeeze()
            mat = torch.eye(num_nodes, device=edge_index.device)

            diff_matrix = kwargs['coeffs'][0] * mat
            for coeff in kwargs['coeffs'][1:]:
                mat = mat @ adj_matrix
                diff_matrix += coeff * mat
        else:
            raise ValueError(f"Exact GDC diffusion '{method}' unknown")

        return diff_matrix

    def diffusion_matrix_approx(
        self,
        edge_index: Tensor,
        edge_weight: Tensor,
        num_nodes: int,
        normalization: str,
        method: str,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        r"""Calculate the approximate, sparse diffusion on a given sparse
        graph.

        Args:
            edge_index (LongTensor): The edge indices.
            edge_weight (Tensor): One-dimensional edge weights.
            num_nodes (int): Number of nodes.
            normalization (str): Transition matrix normalization scheme
                (:obj:`"sym"`, :obj:`"row"`, or :obj:`"col"`).
                See :func:`GDC.transition_matrix` for details.
            method (str): Diffusion method:

                1. :obj:`"ppr"`: Use personalized PageRank as diffusion.
                   Additionally expects the parameters:

                   - **alpha** (*float*) - Return probability in PPR.
                     Commonly lies in :obj:`[0.05, 0.2]`.

                   - **eps** (*float*) - Threshold for PPR calculation stopping
                     criterion (:obj:`edge_weight >= eps * out_degree`).
                     Recommended default: :obj:`1e-4`.

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        if method == 'ppr':
            if normalization == 'sym':
                # Calculate original degrees.
                _, col = edge_index
                deg = scatter(edge_weight, col, 0, num_nodes, reduce='sum')

            edge_index_np = edge_index.cpu().numpy()

            # Assumes sorted and coalesced edge indices:
            indptr = index2ptr(edge_index[0], num_nodes).cpu().numpy()
            out_degree = indptr[1:] - indptr[:-1]

            neighbors, neighbor_weights = self.__calc_ppr__(
                indptr, edge_index_np[1], out_degree, kwargs['alpha'],
                kwargs['eps'])
            ppr_normalization = 'col' if normalization == 'col' else 'row'
            edge_index, edge_weight = self.__neighbors_to_graph__(
                neighbors, neighbor_weights, ppr_normalization,
                device=edge_index.device)
            edge_index = edge_index.to(torch.long)

            if normalization == 'sym':
                # We can change the normalization from row-normalized to
                # symmetric by multiplying the resulting matrix with D^{1/2}
                # from the left and D^{-1/2} from the right.
                # Since we use the original degrees for this it will be like
                # we had used symmetric normalization from the beginning
                # (except for errors due to approximation).
                row, col = edge_index
                deg_inv = deg.sqrt()
                deg_inv_sqrt = deg.pow(-0.5)
                deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0
                edge_weight = deg_inv[row] * edge_weight * deg_inv_sqrt[col]
            elif normalization in ['col', 'row']:
                pass
            else:
                raise ValueError(
                    f"Transition matrix normalization '{normalization}' not "
                    f"implemented for non-exact GDC computation")

        elif method == 'heat':
            raise NotImplementedError(
                ('Currently no fast heat kernel is implemented. You are '
                 'welcome to create one yourself, e.g., based on '
                 '"Kloster and Gleich: Heat kernel based community detection '
                 '(KDD 2014)."'))
        else:
            raise ValueError(f"Approximate GDC diffusion '{method}' unknown")

        return edge_index, edge_weight

    def sparsify_dense(
        self,
        matrix: Tensor,
        method: str,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        r"""Sparsifies the given dense matrix.

        Args:
            matrix (Tensor): Matrix to sparsify.
            method (str): Method of sparsification. Options:

                1. :obj:`"threshold"`: Remove all edges with weights smaller
                   than :obj:`eps`.
                   Additionally expects one of these parameters:

                   - **eps** (*float*) - Threshold to bound edges at.

                   - **avg_degree** (*int*) - If :obj:`eps` is not given,
                     it can optionally be calculated by calculating the
                     :obj:`eps` required to achieve a given :obj:`avg_degree`.

                2. :obj:`"topk"`: Keep edges with top :obj:`k` edge weights per
                   node (column).
                   Additionally expects the following parameters:

                   - **k** (*int*) - Specifies the number of edges to keep.

                   - **dim** (*int*) - The axis along which to take the top
                     :obj:`k`.

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        assert matrix.shape[0] == matrix.shape[1]
        N = matrix.shape[1]

        if method == 'threshold':
            if 'eps' not in kwargs.keys():
                kwargs['eps'] = self.__calculate_eps__(matrix, N,
                                                       kwargs['avg_degree'])

            edge_index = (matrix >= kwargs['eps']).nonzero(as_tuple=False).t()
            edge_index_flat = edge_index[0] * N + edge_index[1]
            edge_weight = matrix.flatten()[edge_index_flat]

        elif method == 'topk':
            k, dim = min(N, kwargs['k']), kwargs['dim']
            assert dim in [0, 1]
            sort_idx = torch.argsort(matrix, dim=dim, descending=True)
            if dim == 0:
                top_idx = sort_idx[:k]
                edge_weight = torch.gather(matrix, dim=dim,
                                           index=top_idx).flatten()

                row_idx = torch.arange(0, N, device=matrix.device).repeat(k)
                edge_index = torch.stack([top_idx.flatten(), row_idx], dim=0)
            else:
                top_idx = sort_idx[:, :k]
                edge_weight = torch.gather(matrix, dim=dim,
                                           index=top_idx).flatten()

                col_idx = torch.arange(
                    0, N, device=matrix.device).repeat_interleave(k)
                edge_index = torch.stack([col_idx, top_idx.flatten()], dim=0)
        else:
            raise ValueError(f"GDC sparsification '{method}' unknown")

        return edge_index, edge_weight

    def sparsify_sparse(
        self,
        edge_index: Tensor,
        edge_weight: Tensor,
        num_nodes: int,
        method: str,
        **kwargs,
    ) -> Tuple[Tensor, Tensor]:
        r"""Sparsifies a given sparse graph further.

        Args:
            edge_index (LongTensor): The edge indices.
            edge_weight (Tensor): One-dimensional edge weights.
            num_nodes (int): Number of nodes.
            method (str): Method of sparsification:

                1. :obj:`"threshold"`: Remove all edges with weights smaller
                   than :obj:`eps`.
                   Additionally expects one of these parameters:

                   - **eps** (*float*) - Threshold to bound edges at.

                   - **avg_degree** (*int*) - If :obj:`eps` is not given,
                     it can optionally be calculated by calculating the
                     :obj:`eps` required to achieve a given :obj:`avg_degree`.

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        if method == 'threshold':
            if 'eps' not in kwargs.keys():
                kwargs['eps'] = self.__calculate_eps__(edge_weight, num_nodes,
                                                       kwargs['avg_degree'])

            remaining_edge_idx = (edge_weight >= kwargs['eps']).nonzero(
                as_tuple=False).flatten()
            edge_index = edge_index[:, remaining_edge_idx]
            edge_weight = edge_weight[remaining_edge_idx]
        elif method == 'topk':
            raise NotImplementedError(
                'Sparse topk sparsification not implemented')
        else:
            raise ValueError(f"GDC sparsification '{method}' unknown")

        return edge_index, edge_weight

    def __expm__(self, matrix: Tensor, symmetric: bool) -> Tensor:
        r"""Calculates matrix exponential.

        Args:
            matrix (Tensor): Matrix to take exponential of.
            symmetric (bool): Specifies whether the matrix is symmetric.

        :rtype: (:class:`Tensor`)
        """
        if symmetric:
            e, V = torch.linalg.eigh(matrix, UPLO='U')
            diff_mat = V @ torch.diag(e.exp()) @ V.t()
        else:
            diff_mat_np = expm(matrix.cpu().numpy())
            diff_mat = torch.Tensor(diff_mat_np).to(matrix.device)
        return diff_mat

    def __calculate_eps__(
        self,
        matrix: Tensor,
        num_nodes: int,
        avg_degree: int,
    ) -> float:
        r"""Calculates threshold necessary to achieve a given average degree.

        Args:
            matrix (Tensor): Adjacency matrix or edge weights.
            num_nodes (int): Number of nodes.
            avg_degree (int): Target average degree.

        :rtype: (:class:`float`)
        """
        sorted_edges = torch.sort(matrix.flatten(), descending=True).values
        if avg_degree * num_nodes > len(sorted_edges):
            return -np.inf

        left = sorted_edges[avg_degree * num_nodes - 1]
        right = sorted_edges[avg_degree * num_nodes]
        return (left + right) / 2.0

    def __neighbors_to_graph__(
        self,
        neighbors: List[List[int]],
        neighbor_weights: List[List[float]],
        normalization: str = 'row',
        device: torch.device = 'cpu',
    ) -> Tuple[Tensor, Tensor]:
        r"""Combine a list of neighbors and neighbor weights to create a sparse
        graph.

        Args:
            neighbors (List[List[int]]): List of neighbors for each node.
            neighbor_weights (List[List[float]]): List of weights for the
                neighbors of each node.
            normalization (str): Normalization of resulting matrix
                (options: :obj:`"row"`, :obj:`"col"`). (default: :obj:`"row"`)
            device (torch.device): Device to create output tensors on.
                (default: :obj:`"cpu"`)

        :rtype: (:class:`LongTensor`, :class:`Tensor`)
        """
        edge_weight = torch.Tensor(np.concatenate(neighbor_weights)).to(device)
        i = np.repeat(np.arange(len(neighbors)),
                      np.fromiter(map(len, neighbors), dtype=int))
        j = np.concatenate(neighbors)
        if normalization == 'col':
            edge_index = torch.Tensor(np.vstack([j, i])).to(device)
            N = len(neighbors)
            edge_index, edge_weight = coalesce(edge_index, edge_weight, N, N)
        elif normalization == 'row':
            edge_index = torch.Tensor(np.vstack([i, j])).to(device)
        else:
            raise ValueError(
                f"PPR matrix normalization {normalization} unknown.")
        return edge_index, edge_weight


def get_calc_ppr():
    import numba

    @numba.jit(nopython=True, parallel=True)
    def calc_ppr(
        indptr: np.ndarray,
        indices: np.ndarray,
        out_degree: np.ndarray,
        alpha: float,
        eps: float,
    ) -> Tuple[List[List[int]], List[List[float]]]:
        r"""Calculate the personalized PageRank vector for all nodes
        using a variant of the Andersen algorithm
        (see Andersen et al. :Local Graph Partitioning using PageRank Vectors.)

        Args:
            indptr (np.ndarray): Index pointer for the sparse matrix
                (CSR-format).
            indices (np.ndarray): Indices of the sparse matrix entries
                (CSR-format).
            out_degree (np.ndarray): Out-degree of each node.
            alpha (float): Alpha of the PageRank to calculate.
            eps (float): Threshold for PPR calculation stopping criterion
                (:obj:`edge_weight >= eps * out_degree`).

        :rtype: (:class:`List[List[int]]`, :class:`List[List[float]]`)
        """

        alpha_eps = alpha * eps
        js = [[0]] * len(out_degree)
        vals = [[0.]] * len(out_degree)
        for inode_uint in numba.prange(len(out_degree)):
            inode = numba.int64(inode_uint)
            p = {inode: 0.0}
            r = {}
            r[inode] = alpha
            q = [inode]
            while len(q) > 0:
                unode = q.pop()

                res = r[unode] if unode in r else 0
                if unode in p:
                    p[unode] += res
                else:
                    p[unode] = res
                r[unode] = 0
                for vnode in indices[indptr[unode]:indptr[unode + 1]]:
                    _val = (1 - alpha) * res / out_degree[unode]
                    if vnode in r:
                        r[vnode] += _val
                    else:
                        r[vnode] = _val

                    res_vnode = r[vnode] if vnode in r else 0
                    if res_vnode >= alpha_eps * out_degree[vnode]:
                        if vnode not in q:
                            q.append(vnode)
            js[inode] = list(p.keys())
            vals[inode] = list(p.values())
        return js, vals

    return calc_ppr
pytorch/torch_geometric/transforms/generate_mesh_normals.py000066400000000000000000000017721444262304300250450ustar00rootroot00000000000000import torch
import torch.nn.functional as F

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import scatter


@functional_transform('generate_mesh_normals')
class GenerateMeshNormals(BaseTransform):
    r"""Generate normal vectors for each mesh node based on neighboring
    faces (functional name: :obj:`generate_mesh_normals`)."""
    def forward(self, data: Data) -> Data:
        assert 'face' in data
        pos, face = data.pos, data.face

        vec1 = pos[face[1]] - pos[face[0]]
        vec2 = pos[face[2]] - pos[face[0]]
        face_norm = F.normalize(vec1.cross(vec2), p=2, dim=-1)  # [F, 3]

        idx = torch.cat([face[0], face[1], face[2]], dim=0)
        face_norm = face_norm.repeat(3, 1)

        norm = scatter(face_norm, idx, 0, pos.size(0), reduce='sum')
        norm = F.normalize(norm, p=2, dim=-1)  # [N, 3]

        data.norm = norm

        return data
pytorch/torch_geometric/transforms/grid_sampling.py000066400000000000000000000047171444262304300233250ustar00rootroot00000000000000import re
from typing import List, Optional, Union

import torch
from torch import Tensor

import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import one_hot, scatter


@functional_transform('grid_sampling')
class GridSampling(BaseTransform):
    r"""Clusters points into fixed-sized voxels
    (functional name: :obj:`grid_sampling`).
    Each cluster returned is a new point based on the mean of all points
    inside the given cluster.

    Args:
        size (float or [float] or Tensor): Size of a voxel (in each dimension).
        start (float or [float] or Tensor, optional): Start coordinates of the
            grid (in each dimension). If set to :obj:`None`, will be set to the
            minimum coordinates found in :obj:`data.pos`.
            (default: :obj:`None`)
        end (float or [float] or Tensor, optional): End coordinates of the grid
            (in each dimension). If set to :obj:`None`, will be set to the
            maximum coordinates found in :obj:`data.pos`.
            (default: :obj:`None`)
    """
    def __init__(self, size: Union[float, List[float], Tensor],
                 start: Optional[Union[float, List[float], Tensor]] = None,
                 end: Optional[Union[float, List[float], Tensor]] = None):
        self.size = size
        self.start = start
        self.end = end

    def forward(self, data: Data) -> Data:
        num_nodes = data.num_nodes

        batch = data.get('batch', None)

        c = torch_geometric.nn.voxel_grid(data.pos, self.size, batch,
                                          self.start, self.end)
        c, perm = torch_geometric.nn.pool.consecutive.consecutive_cluster(c)

        for key, item in data:
            if bool(re.search('edge', key)):
                raise ValueError(
                    'GridSampling does not support coarsening of edges')

            if torch.is_tensor(item) and item.size(0) == num_nodes:
                if key == 'y':
                    item = scatter(one_hot(item), c, dim=0, reduce='sum')
                    data[key] = item.argmax(dim=-1)
                elif key == 'batch':
                    data[key] = item[perm]
                else:
                    data[key] = scatter(item, c, dim=0, reduce='mean')

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(size={self.size})'
pytorch/torch_geometric/transforms/knn_graph.py000066400000000000000000000047571444262304300224610ustar00rootroot00000000000000import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_undirected


@functional_transform('knn_graph')
class KNNGraph(BaseTransform):
    r"""Creates a k-NN graph based on node positions :obj:`data.pos`
    (functional name: :obj:`knn_graph`).

    Args:
        k (int, optional): The number of neighbors. (default: :obj:`6`)
        loop (bool, optional): If :obj:`True`, the graph will contain
            self-loops. (default: :obj:`False`)
        force_undirected (bool, optional): If set to :obj:`True`, new edges
            will be undirected. (default: :obj:`False`)
        flow (str, optional): The flow direction when used in combination with
            message passing (:obj:`"source_to_target"` or
            :obj:`"target_to_source"`).
            If set to :obj:`"source_to_target"`, every target node will have
            exactly :math:`k` source nodes pointing to it.
            (default: :obj:`"source_to_target"`)
        cosine (bool, optional): If :obj:`True`, will use the cosine
            distance instead of euclidean distance to find nearest neighbors.
            (default: :obj:`False`)
        num_workers (int): Number of workers to use for computation. Has no
            effect in case :obj:`batch` is not :obj:`None`, or the input lies
            on the GPU. (default: :obj:`1`)
    """
    def __init__(
        self,
        k: int = 6,
        loop: bool = False,
        force_undirected: bool = False,
        flow: str = 'source_to_target',
        cosine: bool = False,
        num_workers: int = 1,
    ):
        self.k = k
        self.loop = loop
        self.force_undirected = force_undirected
        self.flow = flow
        self.cosine = cosine
        self.num_workers = num_workers

    def forward(self, data: Data) -> Data:
        data.edge_attr = None
        batch = data.batch if 'batch' in data else None

        edge_index = torch_geometric.nn.knn_graph(
            data.pos,
            self.k,
            batch,
            loop=self.loop,
            flow=self.flow,
            cosine=self.cosine,
            num_workers=self.num_workers,
        )

        if self.force_undirected:
            edge_index = to_undirected(edge_index, num_nodes=data.num_nodes)

        data.edge_index = edge_index

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(k={self.k})'
pytorch/torch_geometric/transforms/laplacian_lambda_max.py000066400000000000000000000046411444262304300245730ustar00rootroot00000000000000from typing import Optional

from scipy.sparse.linalg import eigs, eigsh

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import get_laplacian, to_scipy_sparse_matrix


@functional_transform('laplacian_lambda_max')
class LaplacianLambdaMax(BaseTransform):
    r"""Computes the highest eigenvalue of the graph Laplacian given by
    :meth:`torch_geometric.utils.get_laplacian`
    (functional name: :obj:`laplacian_lambda_max`).

    Args:
        normalization (str, optional): The normalization scheme for the graph
            Laplacian (default: :obj:`None`):

            1. :obj:`None`: No normalization
            :math:`\mathbf{L} = \mathbf{D} - \mathbf{A}`

            2. :obj:`"sym"`: Symmetric normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1/2} \mathbf{A}
            \mathbf{D}^{-1/2}`

            3. :obj:`"rw"`: Random-walk normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1} \mathbf{A}`
        is_undirected (bool, optional): If set to :obj:`True`, this transform
            expects undirected graphs as input, and can hence speed up the
            computation of the largest eigenvalue. (default: :obj:`False`)
    """
    def __init__(
        self,
        normalization: Optional[str] = None,
        is_undirected: bool = False,
    ):
        assert normalization in [None, 'sym', 'rw'], 'Invalid normalization'
        self.normalization = normalization
        self.is_undirected = is_undirected

    def forward(self, data: Data) -> Data:
        edge_weight = data.edge_attr
        if edge_weight is not None and edge_weight.numel() != data.num_edges:
            edge_weight = None

        edge_index, edge_weight = get_laplacian(data.edge_index, edge_weight,
                                                self.normalization,
                                                num_nodes=data.num_nodes)

        L = to_scipy_sparse_matrix(edge_index, edge_weight, data.num_nodes)

        eig_fn = eigs
        if self.is_undirected and self.normalization != 'rw':
            eig_fn = eigsh

        lambda_max = eig_fn(L, k=1, which='LM', return_eigenvectors=False)
        data.lambda_max = float(lambda_max.real)

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(normalization={self.normalization})'
pytorch/torch_geometric/transforms/largest_connected_components.py000066400000000000000000000037201444262304300264270ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_scipy_sparse_matrix


@functional_transform('largest_connected_components')
class LargestConnectedComponents(BaseTransform):
    r"""Selects the subgraph that corresponds to the
    largest connected components in the graph
    (functional name: :obj:`largest_connected_components`).

    Args:
        num_components (int, optional): Number of largest components to keep
            (default: :obj:`1`)
        connection (str, optional): Type of connection to use for directed
            graphs, can be either :obj:`'strong'` or :obj:`'weak'`.
            Nodes `i` and `j` are strongly connected if a path
            exists both from `i` to `j` and from `j` to `i`. A directed graph
            is weakly connected if replacing all of its directed edges with
            undirected edges produces a connected (undirected) graph.
            (default: :obj:`'weak'`)
    """
    def __init__(self, num_components: int = 1, connection: str = 'weak'):
        assert connection in ['strong', 'weak'], 'Unknown connection type'
        self.num_components = num_components
        self.connection = connection

    def forward(self, data: Data) -> Data:
        import numpy as np
        import scipy.sparse as sp

        adj = to_scipy_sparse_matrix(data.edge_index, num_nodes=data.num_nodes)

        num_components, component = sp.csgraph.connected_components(
            adj, connection=self.connection)

        if num_components <= self.num_components:
            return data

        _, count = np.unique(component, return_counts=True)
        subset = np.in1d(component, count.argsort()[-self.num_components:])

        return data.subgraph(torch.from_numpy(subset).to(torch.bool))

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num_components})'
pytorch/torch_geometric/transforms/line_graph.py000066400000000000000000000072131444262304300226100ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import coalesce, remove_self_loops, scatter


@functional_transform('line_graph')
class LineGraph(BaseTransform):
    r"""Converts a graph to its corresponding line-graph
    (functional name: :obj:`line_graph`):

    .. math::
        L(\mathcal{G}) &= (\mathcal{V}^{\prime}, \mathcal{E}^{\prime})

        \mathcal{V}^{\prime} &= \mathcal{E}

        \mathcal{E}^{\prime} &= \{ (e_1, e_2) : e_1 \cap e_2 \neq \emptyset \}

    Line-graph node indices are equal to indices in the original graph's
    coalesced :obj:`edge_index`.
    For undirected graphs, the maximum line-graph node index is
    :obj:`(data.edge_index.size(1) // 2) - 1`.

    New node features are given by old edge attributes.
    For undirected graphs, edge attributes for reciprocal edges
    :obj:`(row, col)` and :obj:`(col, row)` get summed together.

    Args:
        force_directed (bool, optional): If set to :obj:`True`, the graph will
            be always treated as a directed graph. (default: :obj:`False`)
    """
    def __init__(self, force_directed: bool = False):
        self.force_directed = force_directed

    def forward(self, data: Data) -> Data:
        N = data.num_nodes
        edge_index, edge_attr = data.edge_index, data.edge_attr
        edge_index, edge_attr = coalesce(edge_index, edge_attr, num_nodes=N)
        row, col = edge_index

        if self.force_directed or data.is_directed():
            i = torch.arange(row.size(0), dtype=torch.long, device=row.device)

            count = scatter(torch.ones_like(row), row, dim=0,
                            dim_size=data.num_nodes, reduce='sum')
            cumsum = torch.cat([count.new_zeros(1), count.cumsum(0)], dim=0)

            cols = [
                i[cumsum[col[j]]:cumsum[col[j] + 1]]
                for j in range(col.size(0))
            ]
            rows = [row.new_full((c.numel(), ), j) for j, c in enumerate(cols)]

            row, col = torch.cat(rows, dim=0), torch.cat(cols, dim=0)

            data.edge_index = torch.stack([row, col], dim=0)
            data.x = data.edge_attr
            data.num_nodes = edge_index.size(1)

        else:
            # Compute node indices.
            mask = row < col
            row, col = row[mask], col[mask]
            i = torch.arange(row.size(0), dtype=torch.long, device=row.device)

            (row, col), i = coalesce(
                torch.stack([
                    torch.cat([row, col], dim=0),
                    torch.cat([col, row], dim=0)
                ], dim=0),
                torch.cat([i, i], dim=0),
                N,
            )

            # Compute new edge indices according to `i`.
            count = scatter(torch.ones_like(row), row, dim=0,
                            dim_size=data.num_nodes, reduce='sum')
            joints = torch.split(i, count.tolist())

            def generate_grid(x):
                row = x.view(-1, 1).repeat(1, x.numel()).view(-1)
                col = x.repeat(x.numel())
                return torch.stack([row, col], dim=0)

            joints = [generate_grid(joint) for joint in joints]
            joints = torch.cat(joints, dim=1)
            joints, _ = remove_self_loops(joints)
            N = row.size(0) // 2
            joints = coalesce(joints, num_nodes=N)

            if edge_attr is not None:
                data.x = scatter(edge_attr, i, dim=0, dim_size=N, reduce='sum')
            data.edge_index = joints
            data.num_nodes = edge_index.size(1) // 2

        data.edge_attr = None
        return data
pytorch/torch_geometric/transforms/linear_transformation.py000066400000000000000000000037141444262304300251020ustar00rootroot00000000000000from typing import Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('linear_transformation')
class LinearTransformation(BaseTransform):
    r"""Transforms node positions :obj:`data.pos` with a square transformation
    matrix computed offline (functional name: :obj:`linear_transformation`)

    Args:
        matrix (Tensor): Tensor with shape :obj:`[D, D]` where :obj:`D`
            corresponds to the dimensionality of node positions.
    """
    def __init__(self, matrix: Tensor):
        if not isinstance(matrix, Tensor):
            matrix = torch.tensor(matrix)
        assert matrix.dim() == 2, (
            'Transformation matrix should be two-dimensional.')
        assert matrix.size(0) == matrix.size(1), (
            f'Transformation matrix should be square (got {matrix.size()})')

        # Store the matrix as its transpose.
        # We do this to enable post-multiplication in `forward`.
        self.matrix = matrix.t()

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.node_stores:
            if not hasattr(store, 'pos'):
                continue

            pos = store.pos.view(-1, 1) if store.pos.dim() == 1 else store.pos
            assert pos.size(-1) == self.matrix.size(-2), (
                'Node position matrix and transformation matrix have '
                'incompatible shape')
            # We post-multiply the points by the transformation matrix instead
            # of pre-multiplying, because `pos` attribute has shape `[N, D]`,
            # and we want to preserve this shape.
            store.pos = pos @ self.matrix.to(pos.device, pos.dtype)

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(\n{self.matrix.cpu().numpy()}\n)'
pytorch/torch_geometric/transforms/local_cartesian.py000066400000000000000000000032421444262304300236210ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import scatter


@functional_transform('local_cartesian')
class LocalCartesian(BaseTransform):
    r"""Saves the relative Cartesian coordinates of linked nodes in its edge
    attributes (functional name: :obj:`local_cartesian`). Each coordinate gets
    *neighborhood-normalized* to the interval :math:`{[0, 1]}^D`.

    Args:
        norm (bool, optional): If set to :obj:`False`, the output will not be
            normalized to the interval :math:`{[0, 1]}^D`.
            (default: :obj:`True`)
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(self, norm: bool = True, cat: bool = True):
        self.norm = norm
        self.cat = cat

    def forward(self, data: Data) -> Data:
        (row, col), pos, pseudo = data.edge_index, data.pos, data.edge_attr

        cart = pos[row] - pos[col]
        cart = cart.view(-1, 1) if cart.dim() == 1 else cart

        max_value = scatter(cart.abs(), col, 0, pos.size(0), reduce='max')
        max_value = max_value.max(dim=-1, keepdim=True)[0]

        if self.norm:
            cart = cart / (2 * max_value[col]) + 0.5
        else:
            cart = cart / max_value[col]

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, cart.type_as(pseudo)], dim=-1)
        else:
            data.edge_attr = cart

        return data
pytorch/torch_geometric/transforms/local_degree_profile.py000066400000000000000000000025741444262304300246320ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import degree


@functional_transform('local_degree_profile')
class LocalDegreeProfile(BaseTransform):
    r"""Appends the Local Degree Profile (LDP) from the `"A Simple yet
    Effective Baseline for Non-attribute Graph Classification"
    `_ paper
    (functional name: :obj:`local_degree_profile`)

    .. math::
        \mathbf{x}_i = \mathbf{x}_i \, \Vert \, (\deg(i), \min(DN(i)),
        \max(DN(i)), \textrm{mean}(DN(i)), \textrm{std}(DN(i)))

    to the node features, where :math:`DN(i) = \{ \deg(j) \mid j \in
    \mathcal{N}(i) \}`.
    """
    def __init__(self):
        from torch_geometric.nn.aggr.fused import FusedAggregation
        self.aggr = FusedAggregation(['min', 'max', 'mean', 'std'])

    def forward(self, data: Data) -> Data:
        row, col = data.edge_index
        N = data.num_nodes

        deg = degree(row, N, dtype=torch.float).view(-1, 1)
        xs = [deg] + self.aggr(deg[col], row, dim_size=N)

        if data.x is not None:
            data.x = data.x.view(-1, 1) if data.x.dim() == 1 else data.x
            data.x = torch.cat([data.x] + xs, dim=-1)
        else:
            data.x = torch.cat(xs, dim=-1)

        return data
pytorch/torch_geometric/transforms/mask.py000066400000000000000000000107661444262304300214420ustar00rootroot00000000000000from typing import List, Optional, Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.data.storage import BaseStorage
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import index_to_mask, mask_to_index

AnyData = Union[Data, HeteroData]


def get_attrs_with_suffix(
    attrs: Optional[List[str]],
    store: BaseStorage,
    suffix: str,
) -> List[str]:
    if attrs is not None:
        return attrs
    return [key for key in store.keys() if key.endswith(suffix)]


def get_mask_size(attr: str, store: BaseStorage, size: Optional[int]) -> int:
    if size is not None:
        return size
    return store.num_edges if store.is_edge_attr(attr) else store.num_nodes


@functional_transform('index_to_mask')
class IndexToMask(BaseTransform):
    r"""Converts indices to a mask representation
    (functional name: :obj:`index_to_mask`).

    Args:
        attrs (str, [str], optional): If given, will only perform index to mask
            conversion for the given attributes. If omitted, will infer the
            attributes from the suffix :obj:`_index`. (default: :obj:`None`)
        sizes (int, [int], optional): The size of the mask. If set to
            :obj:`None`, an automatically sized tensor is returned. The number
            of nodes will be used by default, except for edge attributes which
            will use the number of edges as the mask size.
            (default: :obj:`None`)
        replace (bool, optional): if set to :obj:`True` replaces the index
            attributes with mask tensors. (default: :obj:`False`)
    """
    def __init__(
        self,
        attrs: Optional[Union[str, List[str]]] = None,
        sizes: Optional[Union[int, List[int]]] = None,
        replace: bool = False,
    ):
        self.attrs = [attrs] if isinstance(attrs, str) else attrs
        self.sizes = sizes
        self.replace = replace

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.stores:
            attrs = get_attrs_with_suffix(self.attrs, store, '_index')

            sizes = self.sizes or ([None] * len(attrs))
            if isinstance(sizes, int):
                sizes = [self.sizes] * len(attrs)

            if len(attrs) != len(sizes):
                raise ValueError(
                    f"The number of attributes (got {len(attrs)}) must match "
                    f"the number of sizes provided (got {len(sizes)}).")

            for attr, size in zip(attrs, sizes):
                if 'edge_index' in attr:
                    continue
                if attr not in store:
                    continue
                size = get_mask_size(attr, store, size)
                mask = index_to_mask(store[attr], size=size)
                store[f'{attr[:-6]}_mask'] = mask
                if self.replace:
                    del store[attr]

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(attrs={self.attrs}, '
                f'sizes={self.sizes}, replace={self.replace})')


@functional_transform('mask_to_index')
class MaskToIndex(BaseTransform):
    r"""Converts a mask to an index representation
    (functional name: :obj:`mask_to_index`).

    Args:
        attrs (str, [str], optional): If given, will only perform mask to index
            conversion for the given attributes.  If omitted, will infer the
            attributes from the suffix :obj:`_mask` (default: :obj:`None`)
        replace (bool, optional): if set to :obj:`True` replaces the mask
            attributes with index tensors. (default: :obj:`False`)
    """
    def __init__(
        self,
        attrs: Optional[Union[str, List[str]]] = None,
        replace: bool = False,
    ):
        self.attrs = [attrs] if isinstance(attrs, str) else attrs
        self.replace = replace

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.stores:
            attrs = get_attrs_with_suffix(self.attrs, store, '_mask')

            for attr in attrs:
                if attr not in store:
                    continue
                index = mask_to_index(store[attr])
                store[f'{attr[:-5]}_index'] = index
                if self.replace:
                    del store[attr]

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(attrs={self.attrs}, '
                f'replace={self.replace})')
pytorch/torch_geometric/transforms/node_property_split.py000066400000000000000000000137131444262304300246060ustar00rootroot00000000000000from typing import Any, Dict, List

import torch
from torch import Tensor

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_networkx


@functional_transform('node_property_split')
class NodePropertySplit(BaseTransform):
    r"""Creates a node-level split with distributional shift based on a given
    node property, as proposed in the `"Evaluating Robustness and Uncertainty
    of Graph Models Under Structural Distributional Shifts"
    `__ paper
    (functional name: :obj:`node_property_split`).

    It splits the nodes in a given graph into five non-intersecting parts
    based on their structural properties.
    This can be used for transductive node prediction tasks with distributional
    shifts.
    It considers the in-distribution (ID) and out-of-distribution (OOD) subsets
    of nodes.
    The ID subset includes training, validation and testing parts, while
    the OOD subset includes validation and testing parts.
    As a result, it creates five associated node mask vectors for each graph,
    three which are for the ID nodes (:obj:`id_train_mask`,
    :obj:`id_val_mask`, :obj:`id_test_mask`), and two which are for the OOD
    nodes (:obj:`ood_val_mask`, :obj:`ood_test_mask`).

    This class implements three particular strategies for inducing
    distributional shifts in a graph β€” based on **popularity**, **locality**
    or **density**.

    Args:
        property_name (str): The name of the node property to be used
            (:obj:`"popularity"`, :obj:`"locality"`, :obj:`"density"`).
        ratios ([float]): A list of five ratio values for ID training,
            ID validation, ID test, OOD validation and OOD test parts.
            The values must sum to :obj:`1.0`.
        ascending (bool, optional): Whether to sort nodes in ascending order
            of the node property, so that nodes with greater values of the
            property are considered to be OOD (default: :obj:`True`)

    Example:

    .. code-block:: python

        from torch_geometric.transforms import NodePropertySplit
        from torch_geometric.datasets.graph_generator import ERGraph

        data = ERGraph(num_nodes=1000, edge_prob=0.4)()

        property_name = 'popularity'
        ratios = [0.3, 0.1, 0.1, 0.3, 0.2]
        tranaform = NodePropertySplit(property_name, ratios)

        data = transform(data)
    """
    def __init__(
        self,
        property_name: str,
        ratios: List[float],
        ascending: bool = True,
    ):
        if property_name not in {'popularity', 'locality', 'density'}:
            raise ValueError(f"Unexpected 'property_name' "
                             f"(got '{property_name}')")

        if len(ratios) != 5:
            raise ValueError(f"'ratios' must contain 5 values "
                             f"(got {len(ratios)})")

        if sum(ratios) != 1.0:
            raise ValueError(f"'ratios' must sum to 1.0 (got {sum(ratios)})")

        self.property_name = property_name
        self.compute_fn = _property_name_to_compute_fn[property_name]
        self.ratios = ratios
        self.ascending = ascending

    def forward(self, data: Data) -> Data:
        G = to_networkx(data, to_undirected=True, remove_self_loops=True)
        property_values = self.compute_fn(G, self.ascending)
        mask_dict = self._mask_nodes_by_property(property_values, self.ratios)

        for key, mask in mask_dict.items():
            data[key] = mask

        return data

    @staticmethod
    def _compute_popularity_property(G: Any, ascending: bool = True) -> Tensor:
        import networkx.algorithms as A

        property_values = torch.tensor(list(A.pagerank(G).values()))
        property_values *= -1 if ascending else 1
        return property_values

    @staticmethod
    def _compute_locality_property(G: Any, ascending: bool = True) -> Tensor:
        import networkx.algorithms as A

        pagerank_values = torch.tensor(list(A.pagerank(G).values()))

        num_nodes = G.number_of_nodes()
        personalization = dict(zip(range(num_nodes), [0.0] * num_nodes))
        personalization[int(pagerank_values.argmax())] = 1.0

        property_values = torch.tensor(
            list(A.pagerank(G, personalization=personalization).values()))
        property_values *= -1 if ascending else 1
        return property_values

    @staticmethod
    def _compute_density_property(G: Any, ascending: bool = True) -> Tensor:
        import networkx.algorithms as A

        property_values = torch.tensor(list(A.clustering(G).values()))
        property_values *= -1 if ascending else 1
        return property_values

    @staticmethod
    def _mask_nodes_by_property(
        property_values: Tensor,
        ratios: List[float],
    ) -> Dict[str, Tensor]:

        num_nodes = property_values.size(0)
        sizes = (num_nodes * torch.tensor(ratios)).round().long()
        sizes[-1] -= sizes.sum() - num_nodes

        perm = torch.randperm(num_nodes)
        id_size = int(sizes[:3].sum())
        perm = perm[property_values[perm].argsort()]
        perm[:id_size] = perm[:id_size][torch.randperm(id_size)]

        node_splits = perm.split(sizes.tolist())
        names = [
            'id_train_mask',
            'id_val_mask',
            'id_test_mask',
            'ood_val_mask',
            'ood_test_mask',
        ]

        split_masks = {}
        for name, node_split in zip(names, node_splits):
            split_mask = torch.zeros(num_nodes, dtype=torch.bool)
            split_mask[node_split] = True
            split_masks[name] = split_mask
        return split_masks

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.property_name})'


_property_name_to_compute_fn = {
    'popularity': NodePropertySplit._compute_popularity_property,
    'locality': NodePropertySplit._compute_locality_property,
    'density': NodePropertySplit._compute_density_property,
}
pytorch/torch_geometric/transforms/normalize_features.py000066400000000000000000000020041444262304300243670ustar00rootroot00000000000000from typing import List, Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('normalize_features')
class NormalizeFeatures(BaseTransform):
    r"""Row-normalizes the attributes given in :obj:`attrs` to sum-up to one
    (functional name: :obj:`normalize_features`).

    Args:
        attrs (List[str]): The names of attributes to normalize.
            (default: :obj:`["x"]`)
    """
    def __init__(self, attrs: List[str] = ["x"]):
        self.attrs = attrs

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.stores:
            for key, value in store.items(*self.attrs):
                if value.numel() > 0:
                    value = value - value.min()
                    value.div_(value.sum(dim=-1, keepdim=True).clamp_(min=1.))
                    store[key] = value
        return data
pytorch/torch_geometric/transforms/normalize_rotation.py000066400000000000000000000033121444262304300244130ustar00rootroot00000000000000import torch
import torch.nn.functional as F

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('normalize_rotation')
class NormalizeRotation(BaseTransform):
    r"""Rotates all points according to the eigenvectors of the point cloud
    (functional name: :obj:`normalize_rotation`).
    If the data additionally holds normals saved in :obj:`data.normal`, these
    will be rotated accordingly.

    Args:
        max_points (int, optional): If set to a value greater than :obj:`0`,
            only a random number of :obj:`max_points` points are sampled and
            used to compute eigenvectors. (default: :obj:`-1`)
        sort (bool, optional): If set to :obj:`True`, will sort eigenvectors
            according to their eigenvalues. (default: :obj:`False`)
    """
    def __init__(self, max_points: int = -1, sort: bool = False):
        self.max_points = max_points
        self.sort = sort

    def forward(self, data: Data) -> Data:
        pos = data.pos

        if self.max_points > 0 and pos.size(0) > self.max_points:
            perm = torch.randperm(pos.size(0))
            pos = pos[perm[:self.max_points]]

        pos = pos - pos.mean(dim=0, keepdim=True)
        C = torch.matmul(pos.t(), pos)
        e, v = torch.linalg.eig(C)  # v[:,j] is j-th eigenvector
        e, v = torch.view_as_real(e), v.real

        if self.sort:
            indices = e[:, 0].argsort(descending=True)
            v = v.t()[indices].t()

        data.pos = torch.matmul(data.pos, v)

        if 'normal' in data:
            data.normal = F.normalize(torch.matmul(data.normal, v))

        return data
pytorch/torch_geometric/transforms/normalize_scale.py000066400000000000000000000011541444262304300236450ustar00rootroot00000000000000from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform, Center


@functional_transform('normalize_scale')
class NormalizeScale(BaseTransform):
    r"""Centers and normalizes node positions to the interval :math:`(-1, 1)`
    (functional name: :obj:`normalize_scale`).
    """
    def __init__(self):
        self.center = Center()

    def forward(self, data: Data) -> Data:
        data = self.center(data)

        scale = (1 / data.pos.abs().max()) * 0.999999
        data.pos = data.pos * scale

        return data
pytorch/torch_geometric/transforms/one_hot_degree.py000066400000000000000000000027751444262304300234560ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import degree, one_hot


@functional_transform('one_hot_degree')
class OneHotDegree(BaseTransform):
    r"""Adds the node degree as one hot encodings to the node features
    (functional name: :obj:`one_hot_degree`).

    Args:
        max_degree (int): Maximum degree.
        in_degree (bool, optional): If set to :obj:`True`, will compute the
            in-degree of nodes instead of the out-degree.
            (default: :obj:`False`)
        cat (bool, optional): Concat node degrees to node features instead
            of replacing them. (default: :obj:`True`)
    """
    def __init__(
        self,
        max_degree: int,
        in_degree: bool = False,
        cat: bool = True,
    ):
        self.max_degree = max_degree
        self.in_degree = in_degree
        self.cat = cat

    def forward(self, data: Data) -> Data:
        idx, x = data.edge_index[1 if self.in_degree else 0], data.x
        deg = degree(idx, data.num_nodes, dtype=torch.long)
        deg = one_hot(deg, num_classes=self.max_degree + 1)

        if x is not None and self.cat:
            x = x.view(-1, 1) if x.dim() == 1 else x
            data.x = torch.cat([x, deg.to(x.dtype)], dim=-1)
        else:
            data.x = deg

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.max_degree})'
pytorch/torch_geometric/transforms/pad.py000066400000000000000000000527271444262304300212560ustar00rootroot00000000000000import numbers
from abc import ABC, abstractmethod
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, List, Optional, Tuple, Union

import torch
import torch.nn.functional as F

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.data.storage import EdgeStorage, NodeStorage
from torch_geometric.transforms import BaseTransform
from torch_geometric.typing import EdgeType, NodeType


class Padding(ABC):
    r"""An abstract class for specifying padding values."""
    @abstractmethod
    def get_value(
        self,
        store_type: Optional[Union[NodeType, EdgeType]] = None,
        attr_name: Optional[str] = None,
    ) -> Union[int, float]:
        pass


@dataclass(init=False)
class UniformPadding(Padding):
    r"""Uniform padding independent of attribute name or node/edge type.

    Args:
        value (int or float, optional): The value to be used for padding.
            (default: :obj:`0.0`)
    """
    value: Union[int, float] = 0.0

    def __init__(self, value: Union[int, float] = 0.0):
        self.value = value

        if not isinstance(self.value, (int, float)):
            raise ValueError(f"Expected 'value' to be an integer or float "
                             f"(got '{type(value)}'")

    def get_value(
        self,
        store_type: Optional[Union[NodeType, EdgeType]] = None,
        attr_name: Optional[str] = None,
    ) -> Union[int, float]:
        return self.value


@dataclass(init=False)
class MappingPadding(Padding):
    r"""An abstract class for specifying different padding values."""
    values: Dict[Any, Padding] = field(default_factory=dict)
    default: UniformPadding = 0.0

    def __init__(
        self,
        values: Dict[Any, Union[int, float, Padding]],
        default: Union[int, float] = 0.0,
    ):
        if not isinstance(values, dict):
            raise ValueError(f"Expected 'values' to be a dictionary "
                             f"(got '{type(values)}'")

        self.values = {
            key: UniformPadding(val) if isinstance(val, (int, float)) else val
            for key, val in values.items()
        }
        self.default = UniformPadding(default)

        for key, value in self.values.items():
            self.validate_key_value(key, value)

    def validate_key_value(self, key: Any, value: Any):
        pass


class AttrNamePadding(MappingPadding):
    r"""Padding dependent on attribute names.

    Args:
        values (dict): The mapping from attribute names to padding values.
        default (int or float, optional): The padding value to use for
            attribute names not specified in :obj:`values`.
            (default: :obj:`0.0`)
    """
    def validate_key_value(self, key: Any, value: Any):
        if not isinstance(key, str):
            raise ValueError(f"Expected the attribute name '{key}' to be a "
                             f"string (got '{type(key)}')")

        if not isinstance(value, UniformPadding):
            raise ValueError(f"Expected the value of '{key}' to be of "
                             f"type 'UniformPadding' (got '{type(value)}')")

    def get_value(
        self,
        store_type: Optional[Union[NodeType, EdgeType]] = None,
        attr_name: Optional[str] = None,
    ) -> Union[int, float]:
        padding = self.values.get(attr_name, self.default)
        return padding.get_value()


class NodeTypePadding(MappingPadding):
    r"""Padding dependent on node types.

    Args:
        values (dict): The mapping from node types to padding values.
        default (int or float, optional): The padding value to use for node
            types not specified in :obj:`values`. (default: :obj:`0.0`)
    """
    def validate_key_value(self, key: Any, value: Any):
        if not isinstance(key, str):
            raise ValueError(f"Expected the node type '{key}' to be a string "
                             f"(got '{type(key)}')")

        if not isinstance(value, (UniformPadding, AttrNamePadding)):
            raise ValueError(f"Expected the value of '{key}' to be of "
                             f"type 'UniformPadding' or 'AttrNamePadding' "
                             f"(got '{type(value)}')")

    def get_value(
        self,
        store_type: Optional[NodeType] = None,
        attr_name: Optional[str] = None,
    ) -> Union[int, float]:
        padding = self.values.get(store_type, self.default)
        return padding.get_value(attr_name=attr_name)


class EdgeTypePadding(MappingPadding):
    r"""Padding dependent on node types.

    Args:
        values (dict): The mapping from edge types to padding values.
        default (int or float, optional): The padding value to use for edge
            types not specified in :obj:`values`. (default: :obj:`0.0`)
    """
    def validate_key_value(self, key: Any, value: Any):
        if not isinstance(key, tuple):
            raise ValueError(f"Expected the edge type '{key}' to be a tuple "
                             f"(got '{type(key)}')")

        if len(key) != 3:
            raise ValueError(f"Expected the edge type '{key}' to hold exactly "
                             f"three elements (got {len(key)})")

        if not isinstance(value, (UniformPadding, AttrNamePadding)):
            raise ValueError(f"Expected the value of '{key}' to be of "
                             f"type 'UniformPadding' or 'AttrNamePadding' "
                             f"(got '{type(value)}')")

    def get_value(
        self,
        store_type: Optional[EdgeType] = None,
        attr_name: Optional[str] = None,
    ) -> Union[int, float]:
        padding = self.values.get(store_type, self.default)
        return padding.get_value(attr_name=attr_name)


@functional_transform('pad')
class Pad(BaseTransform):
    r"""Applies padding to enforce consistent tensor shapes
    (functional name: :obj:`pad`).

    This transform will pad node and edge features up to a maximum allowed size
    in the node or edge feature dimension. By default :obj:`0.0` is used as the
    padding value and can be configured by setting :obj:`node_pad_value` and
    :obj:`edge_pad_value`.

    In case of applying :class:`Pad` to a :class:`~torch_geometric.data.Data`
    object, the :obj:`node_pad_value` value (or :obj:`edge_pad_value`) can be
    either:

    * an int, float or object of :class:`UniformPadding` class for cases when
      all attributes are going to be padded with the same value;
    * an object of :class:`AttrNamePadding` class for cases when padding is
      going to differ based on attribute names.

    In case of applying :class:`Pad` to a
    :class:`~torch_geometric.data.HeteroData` object, the :obj:`node_pad_value`
    value (or :obj:`edge_pad_value`) can be either:

    * an int, float or object of :class:`UniformPadding` class for cases when
      all attributes of all node (or edge) stores are going to be padded with
      the same value;
    * an object of :class:`AttrNamePadding` class for cases when padding is
      going to differ based on attribute names (but not based on node or edge
      types);
    * an object of class :class:`NodeTypePadding` or :class:`EdgeTypePadding`
      for cases when padding values are going to differ based on node or edge
      types. Padding values can also differ based on attribute names for a
      given node or edge type by using :class:`AttrNamePadding` objects as
      values of its `values` argument.

    Note that in order to allow for consistent padding across all graphs in a
    dataset, below conditions must be met:

    * if :obj:`max_num_nodes` is a single value, it must be greater than or
      equal to the maximum number of nodes of any graph in the dataset;
    * if :obj:`max_num_nodes` is a dictionary, value for every node type must
      be greater than or equal to the maximum number of this type nodes of any
      graph in the dataset.

    Example below shows how to create a :class:`Pad` transform for an
    :class:`~torch_geometric.data.HeteroData` object. The object is padded to
    have :obj:`10` nodes of type :obj:`v0`, :obj:`20` nodes of type :obj:`v1`
    and :obj:`30` nodes of type :obj:`v2`.
    It is padded to have :obj:`80` edges of type :obj:`('v0', 'e0', 'v1')`.
    All the attributes of the :obj:`v0` nodes are padded using a value of
    :obj:`3.0`.
    The :obj:`x` attribute of the :obj:`v1` node type is padded using a value
    of :obj:`-1.0`, and the other attributes of this node type are padded using
    a value of :obj:`0.5`.
    All the attributes of node types other than :obj:`v0` and :obj:`v1` are
    padded using a value of :obj:`1.0`.
    All the attributes of the :obj:`('v0', 'e0', 'v1')` edge type are padded
    usin a value of :obj:`3.5`.
    The :obj:`edge_attr` attributes of the :obj:`('v1', 'e0', 'v0')` edge type
    are padded using a value of :obj:`-1.5`, and any other attributes of this
    edge type are padded using a value of :obj:`5.5`.
    All the attributes of edge types other than these two are padded using a
    value of :obj:`1.5`.

    Example:

    .. code-block::

        num_nodes = {'v0': 10, 'v1': 20, 'v2':30}
        num_edges = {('v0', 'e0', 'v1'): 80}

        node_padding = NodeTypePadding({
            'v0': 3.0,
            'v1': AttrNamePadding({'x': -1.0}, default=0.5),
        }, default=1.0)

        edge_padding = EdgeTypePadding({
            ('v0', 'e0', 'v1'): 3.5,
            ('v1', 'e0', 'v0'): AttrNamePadding({'edge_attr': -1.5},
                                                default=5.5),
        }, default=1.5)

        transform = Pad(num_nodes, num_edges, node_padding, edge_padding)

    Args:
        max_num_nodes (int or dict): The number of nodes after padding.
            In heterogeneous graphs, may also take in a dictionary denoting the
            number of nodes for specific node types.
        max_num_edges (int or dict, optional): The number of edges after
            padding.
            In heterogeneous graphs, may also take in a dictionary denoting the
            number of edges for specific edge types. (default: :obj:`None`)
        node_pad_value (int or float or Padding, optional): The fill value to
            use for node features. (default: :obj:`0.0`)
        edge_pad_value (int or float or Padding, optional): The fill value to
            use for edge features. (default: :obj:`0.0`)
            The :obj:`edge_index` tensor is padded with with the index of the
            first padded node (which represents a set of self-loops on the
            padded node). (default: :obj:`0.0`)
        mask_pad_value (bool, optional): The fill value to use for
            :obj:`train_mask`, :obj:`val_mask` and :obj:`test_mask` attributes
            (default: :obj:`False`).
        add_pad_mask (bool, optional): If set to :obj:`True`, will attach
            node-level :obj:`pad_node_mask` and edge-level :obj:`pad_edge_mask`
            attributes to the output which indicates which elements in the data
            are real (represented by :obj:`True`) and which were added as a
            result of padding (represented by :obj:`False`).
            (default: :obj:`False`)
        exclude_keys ([str], optional): Keys to be removed
            from the input data object. (default: :obj:`None`)
    """
    def __init__(
        self,
        max_num_nodes: Union[int, Dict[NodeType, int]],
        max_num_edges: Optional[Union[int, Dict[EdgeType, int]]] = None,
        node_pad_value: Union[int, float, Padding] = 0.0,
        edge_pad_value: Union[int, float, Padding] = 0.0,
        mask_pad_value: bool = False,
        add_pad_mask: bool = False,
        exclude_keys: Optional[List[str]] = None,
    ):
        self.max_num_nodes = self._NumNodes(max_num_nodes)
        self.max_num_edges = self._NumEdges(max_num_edges, self.max_num_nodes)

        self.node_pad = node_pad_value
        if not isinstance(self.node_pad, Padding):
            self.node_pad = UniformPadding(self.node_pad)

        self.edge_pad = edge_pad_value
        if not isinstance(self.edge_pad, Padding):
            self.edge_pad = UniformPadding(self.edge_pad)

        self.node_additional_attrs_pad = {
            key: mask_pad_value
            for key in ['train_mask', 'val_mask', 'test_mask']
        }

        self.add_pad_mask = add_pad_mask
        self.exclude_keys = set(exclude_keys or [])

    class _IntOrDict(ABC):
        def __init__(self, value):
            self.value = value
            self.is_number = isinstance(value, numbers.Number)

        @abstractmethod
        def get_value(self, key: Optional[Any] = None):
            pass

        def is_none(self):
            return self.value is None

    class _NumNodes(_IntOrDict):
        def __init__(self, value):
            assert isinstance(value, (int, dict)), \
                f'Parameter `max_num_nodes` must be of type int or dict ' \
                f'but is {type(value)}.'
            super().__init__(value)

        def get_value(self, key: Optional[NodeType] = None) -> int:
            if self.is_number or self.value is None:
                # Homodata case.
                return self.value

            assert isinstance(key, str)
            assert key in self.value.keys(), \
                f'The number of {key} nodes was not specified for padding.'
            return self.value[key]

    class _NumEdges(_IntOrDict):
        def __init__(self, value: Union[int, Dict[EdgeType, int], None],
                     num_nodes: '_NumNodes'):  # noqa: F821
            assert value is None or isinstance(value, (int, dict)), \
                f'If provided, parameter `max_num_edges` must be of type ' \
                f'int or dict but is {type(value)}.'

            if value is not None:
                if isinstance(value, int):
                    num_edges = value
                else:
                    num_edges = defaultdict(lambda: defaultdict(int))
                    for k, v in value.items():
                        src_node, edge_type, dst_node = k
                        num_edges[src_node, dst_node][edge_type] = v
            elif num_nodes.is_number:
                num_edges = num_nodes.get_value() * num_nodes.get_value()
            else:
                num_edges = defaultdict(lambda: defaultdict(int))

            self.num_nodes = num_nodes
            super().__init__(num_edges)

        def get_value(self, key: Optional[EdgeType] = None) -> int:
            if self.is_number or self.value is None:
                # Homodata case.
                return self.value

            assert isinstance(key, tuple) and len(key) == 3
            src_v, edge_type, dst_v = key
            if (src_v, dst_v) in self.value.keys():
                if edge_type in self.value[src_v, dst_v]:
                    return self.value[src_v, dst_v][edge_type]

            max_num_edges = self.num_nodes.get_value(
                src_v) * self.num_nodes.get_value(dst_v)
            self.value[src_v, dst_v][edge_type] = max_num_edges
            return self.value[src_v, dst_v][edge_type]

    def __should_pad_node_attr(self, attr_name: str) -> bool:
        if attr_name in self.node_additional_attrs_pad:
            return True
        if self.exclude_keys is None or attr_name not in self.exclude_keys:
            return True
        return False

    def __should_pad_edge_attr(self, attr_name: str) -> bool:
        if self.max_num_edges.is_none():
            return False
        if attr_name == 'edge_index':
            return True
        if self.exclude_keys is None or attr_name not in self.exclude_keys:
            return True
        return False

    def __get_node_padding(
            self, attr_name: str,
            node_type: Optional[NodeType] = None) -> Union[int, float]:
        if attr_name in self.node_additional_attrs_pad:
            return self.node_additional_attrs_pad[attr_name]
        return self.node_pad.get_value(node_type, attr_name)

    def __get_edge_padding(
            self, attr_name: str,
            edge_type: Optional[EdgeType] = None) -> Union[int, float]:
        return self.edge_pad.get_value(edge_type, attr_name)

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:

        if isinstance(data, Data):
            assert isinstance(self.node_pad, (UniformPadding, AttrNamePadding))
            assert isinstance(self.edge_pad, (UniformPadding, AttrNamePadding))

            for store in data.stores:
                for key in self.exclude_keys:
                    del store[key]
                self.__pad_edge_store(store, data.__cat_dim__, data.num_nodes)
                self.__pad_node_store(store, data.__cat_dim__)
            data.num_nodes = self.max_num_nodes.get_value()
        else:
            assert isinstance(
                self.node_pad,
                (UniformPadding, AttrNamePadding, NodeTypePadding))
            assert isinstance(
                self.edge_pad,
                (UniformPadding, AttrNamePadding, EdgeTypePadding))

            for edge_type, store in data.edge_items():
                for key in self.exclude_keys:
                    del store[key]

                src_node_type, _, dst_node_type = edge_type
                self.__pad_edge_store(store, data.__cat_dim__,
                                      (data[src_node_type].num_nodes,
                                       data[dst_node_type].num_nodes),
                                      edge_type)

            for node_type, store in data.node_items():
                for key in self.exclude_keys:
                    del store[key]
                self.__pad_node_store(store, data.__cat_dim__, node_type)
                data[node_type].num_nodes = self.max_num_nodes.get_value(
                    node_type)

        return data

    def __pad_node_store(self, store: NodeStorage, get_dim_fn: Callable,
                         node_type: Optional[str] = None):

        attrs_to_pad = [key for key in store if store.is_node_attr(key)]

        if not attrs_to_pad:
            return

        num_target_nodes = self.max_num_nodes.get_value(node_type)
        assert num_target_nodes >= store.num_nodes, \
            f'The number of nodes after padding ({num_target_nodes}) cannot ' \
            f'be lower than the number of nodes in the data object ' \
            f'({store.num_nodes}).'
        num_pad_nodes = num_target_nodes - store.num_nodes

        if self.add_pad_mask:
            pad_node_mask = torch.ones(num_target_nodes, dtype=torch.bool)
            pad_node_mask[store.num_nodes:] = False
            store.pad_node_mask = pad_node_mask

        for attr_name in attrs_to_pad:
            attr = store[attr_name]
            pad_value = self.__get_node_padding(attr_name, node_type)
            dim = get_dim_fn(attr_name, attr)
            store[attr_name] = self._pad_tensor_dim(attr, dim, num_pad_nodes,
                                                    pad_value)

    def __pad_edge_store(self, store: EdgeStorage, get_dim_fn: Callable,
                         num_nodes: Union[int, Tuple[int, int]],
                         edge_type: Optional[str] = None):
        attrs_to_pad = set(
            attr for attr in store.keys()
            if store.is_edge_attr(attr) and self.__should_pad_edge_attr(attr))
        if not attrs_to_pad:
            return
        num_target_edges = self.max_num_edges.get_value(edge_type)
        assert num_target_edges >= store.num_edges, \
            f'The number of edges after padding ({num_target_edges}) cannot ' \
            f'be lower than the number of edges in the data object ' \
            f'({store.num_edges}).'
        num_pad_edges = num_target_edges - store.num_edges

        if self.add_pad_mask:
            pad_edge_mask = torch.ones(num_target_edges, dtype=torch.bool)
            pad_edge_mask[store.num_edges:] = False
            store.pad_edge_mask = pad_edge_mask

        if isinstance(num_nodes, tuple):
            src_pad_value, dst_pad_value = num_nodes
        else:
            src_pad_value = dst_pad_value = num_nodes

        for attr_name in attrs_to_pad:
            attr = store[attr_name]
            dim = get_dim_fn(attr_name, attr)
            if attr_name == 'edge_index':
                store[attr_name] = self._pad_edge_index(
                    attr, num_pad_edges, src_pad_value, dst_pad_value)
            else:
                pad_value = self.__get_edge_padding(attr_name, edge_type)
                store[attr_name] = self._pad_tensor_dim(
                    attr, dim, num_pad_edges, pad_value)

    @staticmethod
    def _pad_tensor_dim(input: torch.Tensor, dim: int, length: int,
                        pad_value: float) -> torch.Tensor:
        r"""Pads the input tensor in the specified dim with a constant value of
        the given length.
        """
        pads = [0] * (2 * input.ndim)
        pads[-2 * dim - 1] = length
        return F.pad(input, pads, 'constant', pad_value)

    @staticmethod
    def _pad_edge_index(input: torch.Tensor, length: int, src_pad_value: float,
                        dst_pad_value: float) -> torch.Tensor:
        r"""Pads the edges :obj:`edge_index` feature with values specified
        separately for src and dst nodes.
        """
        pads = [0, length, 0, 0]
        padded = F.pad(input, pads, 'constant', src_pad_value)
        if src_pad_value != dst_pad_value:
            padded[1, input.shape[1]:] = dst_pad_value
        return padded

    def __repr__(self) -> str:
        s = f'{self.__class__.__name__}('
        s += f'max_num_nodes={self.max_num_nodes.value}, '
        s += f'max_num_edges={self.max_num_edges.value}, '
        s += f'node_pad_value={self.node_pad}, '
        s += f'edge_pad_value={self.edge_pad})'
        return s
pytorch/torch_geometric/transforms/point_pair_features.py000066400000000000000000000034011444262304300245350ustar00rootroot00000000000000import torch

import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('point_pair_features')
class PointPairFeatures(BaseTransform):
    r"""Computes the rotation-invariant Point Pair Features
    (functional name: :obj:`point_pair_features`)

    .. math::
        \left( \| \mathbf{d_{j,i}} \|, \angle(\mathbf{n}_i, \mathbf{d_{j,i}}),
        \angle(\mathbf{n}_j, \mathbf{d_{j,i}}), \angle(\mathbf{n}_i,
        \mathbf{n}_j) \right)

    of linked nodes in its edge attributes, where :math:`\mathbf{d}_{j,i}`
    denotes the difference vector between, and :math:`\mathbf{n}_i` and
    :math:`\mathbf{n}_j` denote the surface normals of node :math:`i` and
    :math:`j` respectively.

    Args:
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(self, cat: bool = True):
        self.cat = cat

    def forward(self, data: Data) -> Data:
        ppf_func = torch_geometric.nn.conv.ppf_conv.point_pair_features

        assert data.edge_index is not None
        assert data.pos is not None and data.norm is not None
        assert data.pos.size(-1) == 3
        assert data.pos.size() == data.norm.size()

        row, col = data.edge_index
        pos, norm, pseudo = data.pos, data.norm, data.edge_attr

        ppf = ppf_func(pos[row], pos[col], norm[row], norm[col])

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, ppf.type_as(pseudo)], dim=-1)
        else:
            data.edge_attr = ppf

        return data
pytorch/torch_geometric/transforms/polar.py000066400000000000000000000041161444262304300216140ustar00rootroot00000000000000from math import pi as PI
from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('polar')
class Polar(BaseTransform):
    r"""Saves the polar coordinates of linked nodes in its edge attributes
    (functional name: :obj:`polar`).

    Args:
        norm (bool, optional): If set to :obj:`False`, the output will not be
            normalized to the interval :math:`{[0, 1]}^2`.
            (default: :obj:`True`)
        max_value (float, optional): If set and :obj:`norm=True`, normalization
            will be performed based on this value instead of the maximum value
            found in the data. (default: :obj:`None`)
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(
        self,
        norm: bool = True,
        max_value: Optional[float] = None,
        cat: bool = True,
    ):
        self.norm = norm
        self.max = max_value
        self.cat = cat

    def forward(self, data: Data) -> Data:
        (row, col), pos, pseudo = data.edge_index, data.pos, data.edge_attr
        assert pos.dim() == 2 and pos.size(1) == 2

        cart = pos[col] - pos[row]

        rho = torch.norm(cart, p=2, dim=-1).view(-1, 1)

        theta = torch.atan2(cart[..., 1], cart[..., 0]).view(-1, 1)
        theta = theta + (theta < 0).type_as(theta) * (2 * PI)

        if self.norm:
            rho = rho / (rho.max() if self.max is None else self.max)
            theta = theta / (2 * PI)

        polar = torch.cat([rho, theta], dim=-1)

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, polar.type_as(pos)], dim=-1)
        else:
            data.edge_attr = polar

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(norm={self.norm}, '
                f'max_value={self.max})')
pytorch/torch_geometric/transforms/radius_graph.py000066400000000000000000000040021444262304300231410ustar00rootroot00000000000000import torch_geometric
from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('radius_graph')
class RadiusGraph(BaseTransform):
    r"""Creates edges based on node positions :obj:`data.pos` to all points
    within a given distance (functional name: :obj:`radius_graph`).

    Args:
        r (float): The distance.
        loop (bool, optional): If :obj:`True`, the graph will contain
            self-loops. (default: :obj:`False`)
        max_num_neighbors (int, optional): The maximum number of neighbors to
            return for each element in :obj:`y`.
            This flag is only needed for CUDA tensors. (default: :obj:`32`)
        flow (str, optional): The flow direction when using in combination with
            message passing (:obj:`"source_to_target"` or
            :obj:`"target_to_source"`). (default: :obj:`"source_to_target"`)
        num_workers (int): Number of workers to use for computation. Has no
            effect in case :obj:`batch` is not :obj:`None`, or the input lies
            on the GPU. (default: :obj:`1`)
    """
    def __init__(
        self,
        r: float,
        loop: bool = False,
        max_num_neighbors: int = 32,
        flow: str = 'source_to_target',
        num_workers: int = 1,
    ):
        self.r = r
        self.loop = loop
        self.max_num_neighbors = max_num_neighbors
        self.flow = flow
        self.num_workers = num_workers

    def forward(self, data: Data) -> Data:
        data.edge_attr = None
        batch = data.batch if 'batch' in data else None

        data.edge_index = torch_geometric.nn.radius_graph(
            data.pos,
            self.r,
            batch,
            self.loop,
            max_num_neighbors=self.max_num_neighbors,
            flow=self.flow,
            num_workers=self.num_workers,
        )

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(r={self.r})'
pytorch/torch_geometric/transforms/random_flip.py000066400000000000000000000017341444262304300227740ustar00rootroot00000000000000import random

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('random_flip')
class RandomFlip(BaseTransform):
    """Flips node positions along a given axis randomly with a given
    probability (functional name: :obj:`random_flip`).

    Args:
        axis (int): The axis along the position of nodes being flipped.
        p (float, optional): Probability that node positions will be flipped.
            (default: :obj:`0.5`)
    """
    def __init__(self, axis: int, p: float = 0.5):
        self.axis = axis
        self.p = p

    def forward(self, data: Data) -> Data:
        if random.random() < self.p:
            pos = data.pos.clone()
            pos[..., self.axis] = -pos[..., self.axis]
            data.pos = pos
        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(axis={self.axis}, p={self.p})'
pytorch/torch_geometric/transforms/random_jitter.py000066400000000000000000000027461444262304300233470ustar00rootroot00000000000000import numbers
from itertools import repeat
from typing import Sequence, Union

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('random_jitter')
class RandomJitter(BaseTransform):
    r"""Translates node positions by randomly sampled translation values
    within a given interval (functional name: :obj:`random_jitter`).
    In contrast to other random transformations,
    translation is applied separately at each position

    Args:
        translate (sequence or float or int): Maximum translation in each
            dimension, defining the range
            :math:`(-\mathrm{translate}, +\mathrm{translate})` to sample from.
            If :obj:`translate` is a number instead of a sequence, the same
            range is used for each dimension.
    """
    def __init__(self, translate: Union[float, int, Sequence]):
        self.translate = translate

    def forward(self, data: Data) -> Data:
        (n, dim), t = data.pos.size(), self.translate
        if isinstance(t, numbers.Number):
            t = list(repeat(t, times=dim))
        assert len(t) == dim

        ts = []
        for d in range(dim):
            ts.append(data.pos.new_empty(n).uniform_(-abs(t[d]), abs(t[d])))

        data.pos = data.pos + torch.stack(ts, dim=-1)
        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.translate})'
pytorch/torch_geometric/transforms/random_link_split.py000066400000000000000000000337311444262304300242140ustar00rootroot00000000000000import warnings
from copy import copy
from typing import List, Optional, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.data.storage import EdgeStorage
from torch_geometric.transforms import BaseTransform
from torch_geometric.typing import EdgeType
from torch_geometric.utils import negative_sampling


@functional_transform('random_link_split')
class RandomLinkSplit(BaseTransform):
    r"""Performs an edge-level random split into training, validation and test
    sets of a :class:`~torch_geometric.data.Data` or a
    :class:`~torch_geometric.data.HeteroData` object
    (functional name: :obj:`random_link_split`).
    The split is performed such that the training split does not include edges
    in validation and test splits; and the validation split does not include
    edges in the test split.

    .. code-block::

        from torch_geometric.transforms import RandomLinkSplit

        transform = RandomLinkSplit(is_undirected=True)
        train_data, val_data, test_data = transform(data)

    Args:
        num_val (int or float, optional): The number of validation edges.
            If set to a floating-point value in :math:`[0, 1]`, it represents
            the ratio of edges to include in the validation set.
            (default: :obj:`0.1`)
        num_test (int or float, optional): The number of test edges.
            If set to a floating-point value in :math:`[0, 1]`, it represents
            the ratio of edges to include in the test set.
            (default: :obj:`0.2`)
        is_undirected (bool): If set to :obj:`True`, the graph is assumed to be
            undirected, and positive and negative samples will not leak
            (reverse) edge connectivity across different splits. Note that this
            only affects the graph split, label data will not be returned
            undirected.
            (default: :obj:`False`)
        key (str, optional): The name of the attribute holding
            ground-truth labels.
            If :obj:`data[key]` does not exist, it will be automatically
            created and represents a binary classification task
            (:obj:`1` = edge, :obj:`0` = no edge).
            If :obj:`data[key]` exists, it has to be a categorical label from
            :obj:`0` to :obj:`num_classes - 1`.
            After negative sampling, label :obj:`0` represents negative edges,
            and labels :obj:`1` to :obj:`num_classes` represent the labels of
            positive edges. (default: :obj:`"edge_label"`)
        split_labels (bool, optional): If set to :obj:`True`, will split
            positive and negative labels and save them in distinct attributes
            :obj:`"pos_edge_label"` and :obj:`"neg_edge_label"`, respectively.
            (default: :obj:`False`)
        add_negative_train_samples (bool, optional): Whether to add negative
            training samples for link prediction.
            If the model already performs negative sampling, then the option
            should be set to :obj:`False`.
            Otherwise, the added negative samples will be the same across
            training iterations unless negative sampling is performed again.
            (default: :obj:`True`)
        neg_sampling_ratio (float, optional): The ratio of sampled negative
            edges to the number of positive edges. (default: :obj:`1.0`)
        disjoint_train_ratio (int or float, optional): If set to a value
            greater than :obj:`0.0`, training edges will not be shared for
            message passing and supervision. Instead,
            :obj:`disjoint_train_ratio` edges are used as ground-truth labels
            for supervision during training. (default: :obj:`0.0`)
        edge_types (Tuple[EdgeType] or List[EdgeType], optional): The edge
            types used for performing edge-level splitting in case of
            operating on :class:`~torch_geometric.data.HeteroData` objects.
            (default: :obj:`None`)
        rev_edge_types (Tuple[EdgeType] or List[Tuple[EdgeType]], optional):
            The reverse edge types of :obj:`edge_types` in case of operating
            on :class:`~torch_geometric.data.HeteroData` objects.
            This will ensure that edges of the reverse direction will be
            split accordingly to prevent any data leakage.
            Can be :obj:`None` in case no reverse connection exists.
            (default: :obj:`None`)
    """
    def __init__(
        self,
        num_val: Union[int, float] = 0.1,
        num_test: Union[int, float] = 0.2,
        is_undirected: bool = False,
        key: str = 'edge_label',
        split_labels: bool = False,
        add_negative_train_samples: bool = True,
        neg_sampling_ratio: float = 1.0,
        disjoint_train_ratio: Union[int, float] = 0.0,
        edge_types: Optional[Union[EdgeType, List[EdgeType]]] = None,
        rev_edge_types: Optional[Union[EdgeType, List[EdgeType]]] = None,
    ):
        if isinstance(edge_types, list):
            if rev_edge_types is None:
                rev_edge_types = [None] * len(edge_types)

            assert isinstance(rev_edge_types, list)
            assert len(edge_types) == len(rev_edge_types)

        self.num_val = num_val
        self.num_test = num_test
        self.is_undirected = is_undirected
        self.key = key
        self.split_labels = split_labels
        self.add_negative_train_samples = add_negative_train_samples
        self.neg_sampling_ratio = neg_sampling_ratio
        self.disjoint_train_ratio = disjoint_train_ratio
        self.edge_types = edge_types
        self.rev_edge_types = rev_edge_types

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        edge_types = self.edge_types
        rev_edge_types = self.rev_edge_types

        train_data, val_data, test_data = copy(data), copy(data), copy(data)

        if isinstance(data, HeteroData):
            if edge_types is None:
                raise ValueError(
                    "The 'RandomLinkSplit' transform expects 'edge_types' to "
                    "be specified when operating on 'HeteroData' objects")

            if not isinstance(edge_types, list):
                edge_types = [edge_types]
                rev_edge_types = [rev_edge_types]

            stores = [data[edge_type] for edge_type in edge_types]
            train_stores = [train_data[edge_type] for edge_type in edge_types]
            val_stores = [val_data[edge_type] for edge_type in edge_types]
            test_stores = [test_data[edge_type] for edge_type in edge_types]
        else:
            rev_edge_types = [None]
            stores = [data._store]
            train_stores = [train_data._store]
            val_stores = [val_data._store]
            test_stores = [test_data._store]

        for item in zip(stores, train_stores, val_stores, test_stores,
                        rev_edge_types):
            store, train_store, val_store, test_store, rev_edge_type = item

            is_undirected = self.is_undirected
            is_undirected &= not store.is_bipartite()
            is_undirected &= (rev_edge_type is None
                              or store._key == data[rev_edge_type]._key)

            edge_index = store.edge_index
            if is_undirected:
                mask = edge_index[0] <= edge_index[1]
                perm = mask.nonzero(as_tuple=False).view(-1)
                perm = perm[torch.randperm(perm.size(0), device=perm.device)]
            else:
                device = edge_index.device
                perm = torch.randperm(edge_index.size(1), device=device)

            num_val = self.num_val
            if isinstance(num_val, float):
                num_val = int(num_val * perm.numel())
            num_test = self.num_test
            if isinstance(num_test, float):
                num_test = int(num_test * perm.numel())

            num_train = perm.numel() - num_val - num_test

            if num_train <= 0:
                raise ValueError("Insufficient number of edges for training")

            train_edges = perm[:num_train]
            val_edges = perm[num_train:num_train + num_val]
            test_edges = perm[num_train + num_val:]
            train_val_edges = perm[:num_train + num_val]

            num_disjoint = self.disjoint_train_ratio
            if isinstance(num_disjoint, float):
                num_disjoint = int(num_disjoint * train_edges.numel())
            if num_train - num_disjoint <= 0:
                raise ValueError("Insufficient number of edges for training")

            # Create data splits:
            self._split(train_store, train_edges[num_disjoint:], is_undirected,
                        rev_edge_type)
            self._split(val_store, train_edges, is_undirected, rev_edge_type)
            self._split(test_store, train_val_edges, is_undirected,
                        rev_edge_type)

            # Create negative samples:
            num_neg_train = 0
            if self.add_negative_train_samples:
                if num_disjoint > 0:
                    num_neg_train = int(num_disjoint * self.neg_sampling_ratio)
                else:
                    num_neg_train = int(num_train * self.neg_sampling_ratio)
            num_neg_val = int(num_val * self.neg_sampling_ratio)
            num_neg_test = int(num_test * self.neg_sampling_ratio)

            num_neg = num_neg_train + num_neg_val + num_neg_test

            size = store.size()
            if store._key is None or store._key[0] == store._key[-1]:
                size = size[0]
            neg_edge_index = negative_sampling(edge_index, size,
                                               num_neg_samples=num_neg,
                                               method='sparse')

            # Adjust ratio if not enough negative edges exist
            if neg_edge_index.size(1) < num_neg:
                num_neg_found = neg_edge_index.size(1)
                ratio = num_neg_found / num_neg
                warnings.warn(
                    f"There are not enough negative edges to satisfy "
                    "the provided sampling ratio. The ratio will be "
                    f"adjusted to {ratio:.2f}.")
                num_neg_train = int((num_neg_train / num_neg) * num_neg_found)
                num_neg_val = int((num_neg_val / num_neg) * num_neg_found)
                num_neg_test = num_neg_found - num_neg_train - num_neg_val

            # Create labels:
            if num_disjoint > 0:
                train_edges = train_edges[:num_disjoint]
            self._create_label(
                store,
                train_edges,
                neg_edge_index[:, num_neg_val + num_neg_test:],
                out=train_store,
            )
            self._create_label(
                store,
                val_edges,
                neg_edge_index[:, :num_neg_val],
                out=val_store,
            )
            self._create_label(
                store,
                test_edges,
                neg_edge_index[:, num_neg_val:num_neg_val + num_neg_test],
                out=test_store,
            )

        return train_data, val_data, test_data

    def _split(
        self,
        store: EdgeStorage,
        index: Tensor,
        is_undirected: bool,
        rev_edge_type: EdgeType,
    ) -> EdgeStorage:

        edge_attrs = {key for key in store.keys() if store.is_edge_attr(key)}
        for key, value in store.items():
            if key == 'edge_index':
                continue

            if key in edge_attrs:
                value = value[index]
                if is_undirected:
                    value = torch.cat([value, value], dim=0)
                store[key] = value

        edge_index = store.edge_index[:, index]
        if is_undirected:
            edge_index = torch.cat([edge_index, edge_index.flip([0])], dim=-1)
        store.edge_index = edge_index

        if rev_edge_type is not None:
            rev_store = store._parent()[rev_edge_type]
            for key in rev_store.keys():
                if key not in store:
                    del rev_store[key]  # We delete all outdated attributes.
                elif key == 'edge_index':
                    rev_store.edge_index = store.edge_index.flip([0])
                else:
                    rev_store[key] = store[key]

        return store

    def _create_label(
        self,
        store: EdgeStorage,
        index: Tensor,
        neg_edge_index: Tensor,
        out: EdgeStorage,
    ) -> EdgeStorage:

        edge_index = store.edge_index[:, index]

        if hasattr(store, self.key):
            edge_label = store[self.key]
            edge_label = edge_label[index]
            # Increment labels by one. Note that there is no need to increment
            # in case no negative edges are added.
            if neg_edge_index.numel() > 0:
                assert edge_label.dtype == torch.long
                assert edge_label.size(0) == edge_index.size(1)
                edge_label.add_(1)
            if hasattr(out, self.key):
                delattr(out, self.key)
        else:
            edge_label = torch.ones(index.numel(), device=index.device)

        if neg_edge_index.numel() > 0:
            neg_edge_label = edge_label.new_zeros((neg_edge_index.size(1), ) +
                                                  edge_label.size()[1:])

        if self.split_labels:
            out[f'pos_{self.key}'] = edge_label
            out[f'pos_{self.key}_index'] = edge_index
            if neg_edge_index.numel() > 0:
                out[f'neg_{self.key}'] = neg_edge_label
                out[f'neg_{self.key}_index'] = neg_edge_index

        else:
            if neg_edge_index.numel() > 0:
                edge_label = torch.cat([edge_label, neg_edge_label], dim=0)
                edge_index = torch.cat([edge_index, neg_edge_index], dim=-1)
            out[self.key] = edge_label
            out[f'{self.key}_index'] = edge_index

        return out

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(num_val={self.num_val}, '
                f'num_test={self.num_test})')
pytorch/torch_geometric/transforms/random_node_split.py000066400000000000000000000132101444262304300241720ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.data.storage import NodeStorage
from torch_geometric.transforms import BaseTransform


@functional_transform('random_node_split')
class RandomNodeSplit(BaseTransform):
    r"""Performs a node-level random split by adding :obj:`train_mask`,
    :obj:`val_mask` and :obj:`test_mask` attributes to the
    :class:`~torch_geometric.data.Data` or
    :class:`~torch_geometric.data.HeteroData` object
    (functional name: :obj:`random_node_split`).

    Args:
        split (str, optional): The type of dataset split (:obj:`"train_rest"`,
            :obj:`"test_rest"`, :obj:`"random"`).
            If set to :obj:`"train_rest"`, all nodes except those in the
            validation and test sets will be used for training (as in the
            `"FastGCN: Fast Learning with Graph Convolutional Networks via
            Importance Sampling" `_ paper).
            If set to :obj:`"test_rest"`, all nodes except those in the
            training and validation sets will be used for test (as in the
            `"Pitfalls of Graph Neural Network Evaluation"
            `_ paper).
            If set to :obj:`"random"`, train, validation, and test sets will be
            randomly generated, according to :obj:`num_train_per_class`,
            :obj:`num_val` and :obj:`num_test` (as in the `"Semi-supervised
            Classification with Graph Convolutional Networks"
            `_ paper).
            (default: :obj:`"train_rest"`)
        num_splits (int, optional): The number of splits to add. If bigger
            than :obj:`1`, the shape of masks will be
            :obj:`[num_nodes, num_splits]`, and :obj:`[num_nodes]` otherwise.
            (default: :obj:`1`)
        num_train_per_class (int, optional): The number of training samples
            per class in case of :obj:`"test_rest"` and :obj:`"random"` split.
            (default: :obj:`20`)
        num_val (int or float, optional): The number of validation samples.
            If float, it represents the ratio of samples to include in the
            validation set. (default: :obj:`500`)
        num_test (int or float, optional): The number of test samples in case
            of :obj:`"train_rest"` and :obj:`"random"` split. If float, it
            represents the ratio of samples to include in the test set.
            (default: :obj:`1000`)
        key (str, optional): The name of the attribute holding ground-truth
            labels. By default, will only add node-level splits for node-level
            storages in which :obj:`key` is present. (default: :obj:`"y"`).
    """
    def __init__(
        self,
        split: str = "train_rest",
        num_splits: int = 1,
        num_train_per_class: int = 20,
        num_val: Union[int, float] = 500,
        num_test: Union[int, float] = 1000,
        key: Optional[str] = "y",
    ):
        assert split in ['train_rest', 'test_rest', 'random']
        self.split = split
        self.num_splits = num_splits
        self.num_train_per_class = num_train_per_class
        self.num_val = num_val
        self.num_test = num_test
        self.key = key

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.node_stores:
            if self.key is not None and not hasattr(store, self.key):
                continue

            train_masks, val_masks, test_masks = zip(
                *[self._split(store) for _ in range(self.num_splits)])

            store.train_mask = torch.stack(train_masks, dim=-1).squeeze(-1)
            store.val_mask = torch.stack(val_masks, dim=-1).squeeze(-1)
            store.test_mask = torch.stack(test_masks, dim=-1).squeeze(-1)

        return data

    def _split(self, store: NodeStorage) -> Tuple[Tensor, Tensor, Tensor]:
        num_nodes = store.num_nodes

        train_mask = torch.zeros(num_nodes, dtype=torch.bool)
        val_mask = torch.zeros(num_nodes, dtype=torch.bool)
        test_mask = torch.zeros(num_nodes, dtype=torch.bool)

        if isinstance(self.num_val, float):
            num_val = round(num_nodes * self.num_val)
        else:
            num_val = self.num_val

        if isinstance(self.num_test, float):
            num_test = round(num_nodes * self.num_test)
        else:
            num_test = self.num_test

        if self.split == 'train_rest':
            perm = torch.randperm(num_nodes)
            val_mask[perm[:num_val]] = True
            test_mask[perm[num_val:num_val + num_test]] = True
            train_mask[perm[num_val + num_test:]] = True
        else:
            y = getattr(store, self.key)
            num_classes = int(y.max().item()) + 1
            for c in range(num_classes):
                idx = (y == c).nonzero(as_tuple=False).view(-1)
                idx = idx[torch.randperm(idx.size(0))]
                idx = idx[:self.num_train_per_class]
                train_mask[idx] = True

            remaining = (~train_mask).nonzero(as_tuple=False).view(-1)
            remaining = remaining[torch.randperm(remaining.size(0))]

            val_mask[remaining[:num_val]] = True

            if self.split == 'test_rest':
                test_mask[remaining[num_val:]] = True
            elif self.split == 'random':
                test_mask[remaining[num_val:num_val + num_test]] = True

        return train_mask, val_mask, test_mask

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(split={self.split})'
pytorch/torch_geometric/transforms/random_rotate.py000066400000000000000000000035571444262304300233450ustar00rootroot00000000000000import math
import numbers
import random
from typing import Tuple, Union

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform, LinearTransformation


@functional_transform('random_rotate')
class RandomRotate(BaseTransform):
    r"""Rotates node positions around a specific axis by a randomly sampled
    factor within a given interval (functional name: :obj:`random_rotate`).

    Args:
        degrees (tuple or float): Rotation interval from which the rotation
            angle is sampled. If :obj:`degrees` is a number instead of a
            tuple, the interval is given by :math:`[-\mathrm{degrees},
            \mathrm{degrees}]`.
        axis (int, optional): The rotation axis. (default: :obj:`0`)
    """
    def __init__(self, degrees: Union[Tuple[float, float], float],
                 axis: int = 0):
        if isinstance(degrees, numbers.Number):
            degrees = (-abs(degrees), abs(degrees))
        assert isinstance(degrees, (tuple, list)) and len(degrees) == 2
        self.degrees = degrees
        self.axis = axis

    def forward(self, data: Data) -> Data:
        degree = math.pi * random.uniform(*self.degrees) / 180.0
        sin, cos = math.sin(degree), math.cos(degree)

        if data.pos.size(-1) == 2:
            matrix = [[cos, sin], [-sin, cos]]
        else:
            if self.axis == 0:
                matrix = [[1, 0, 0], [0, cos, sin], [0, -sin, cos]]
            elif self.axis == 1:
                matrix = [[cos, 0, -sin], [0, 1, 0], [sin, 0, cos]]
            else:
                matrix = [[cos, sin, 0], [-sin, cos, 0], [0, 0, 1]]
        return LinearTransformation(torch.tensor(matrix))(data)

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}({self.degrees}, '
                f'axis={self.axis})')
pytorch/torch_geometric/transforms/random_scale.py000066400000000000000000000022771444262304300231340ustar00rootroot00000000000000import random
from typing import Tuple

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('random_scale')
class RandomScale(BaseTransform):
    r"""Scales node positions by a randomly sampled factor :math:`s` within a
    given interval, *e.g.*, resulting in the transformation matrix
    (functional name: :obj:`random_scale`)

    .. math::
        \begin{bmatrix}
            s & 0 & 0 \\
            0 & s & 0 \\
            0 & 0 & s \\
        \end{bmatrix}

    for three-dimensional positions.

    Args:
        scales (tuple): scaling factor interval, e.g. :obj:`(a, b)`, then scale
            is randomly sampled from the range
            :math:`a \leq \mathrm{scale} \leq b`.
    """
    def __init__(self, scales: Tuple[float, float]):
        assert isinstance(scales, (tuple, list)) and len(scales) == 2
        self.scales = scales

    def forward(self, data: Data) -> Data:
        scale = random.uniform(*self.scales)
        data.pos = data.pos * scale
        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.scales})'
pytorch/torch_geometric/transforms/random_shear.py000066400000000000000000000024471444262304300231460ustar00rootroot00000000000000from typing import Union

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform, LinearTransformation


@functional_transform('random_shear')
class RandomShear(BaseTransform):
    r"""Shears node positions by randomly sampled factors :math:`s` within a
    given interval, *e.g.*, resulting in the transformation matrix
    (functional name: :obj:`random_shear`)

    .. math::
        \begin{bmatrix}
            1      & s_{xy} & s_{xz} \\
            s_{yx} & 1      & s_{yz} \\
            s_{zx} & z_{zy} & 1      \\
        \end{bmatrix}

    for three-dimensional positions.

    Args:
        shear (float or int): maximum shearing factor defining the range
            :math:`(-\mathrm{shear}, +\mathrm{shear})` to sample from.
    """
    def __init__(self, shear: Union[float, int]):
        self.shear = abs(shear)

    def forward(self, data: Data) -> Data:
        dim = data.pos.size(-1)

        matrix = data.pos.new_empty(dim, dim).uniform_(-self.shear, self.shear)
        eye = torch.arange(dim, dtype=torch.long)
        matrix[eye, eye] = 1

        return LinearTransformation(matrix)(data)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.shear})'
pytorch/torch_geometric/transforms/remove_duplicated_edges.py000066400000000000000000000034151444262304300253420ustar00rootroot00000000000000from typing import List, Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import coalesce


@functional_transform('remove_duplicated_edges')
class RemoveDuplicatedEdges(BaseTransform):
    r"""Removes duplicated edges from a given homogeneous or heterogeneous
    graph. Useful to clean-up known repeated edges/self-loops in common
    benchmark datasets, *e.g.*, in :obj:`ogbn-products`.
    (functional name: :obj:`remove_duplicated_edges`).

    Args:
        key (str or [str], optional): The name of edge attribute(s) to merge in
            case of duplication. (default: :obj:`["edge_weight", "edge_attr"]`)
        reduce (str, optional): The reduce operation to use for merging edge
            attributes (:obj:`"add"`, :obj:`"mean"`, :obj:`"min"`,
            :obj:`"max"`, :obj:`"mul"`). (default: :obj:`"add"`)
    """
    def __init__(
        self,
        key: Union[str, List[str]] = ['edge_attr', 'edge_weight'],
        reduce: str = "add",
    ):
        if isinstance(key, str):
            key = [key]

        self.keys = key
        self.reduce = reduce

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:

        for store in data.edge_stores:
            keys = [key for key in self.keys if key in store]

            store.edge_index, edge_attrs = coalesce(
                edge_index=store.edge_index,
                edge_attr=[store[key] for key in keys],
                num_nodes=max(store.size()),
                reduce=self.reduce,
            )

            for key, edge_attr in zip(keys, edge_attrs):
                store[key] = edge_attr

        return data
pytorch/torch_geometric/transforms/remove_isolated_nodes.py000066400000000000000000000043531444262304300250530ustar00rootroot00000000000000import copy
from collections import defaultdict
from typing import Union

import torch

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('remove_isolated_nodes')
class RemoveIsolatedNodes(BaseTransform):
    r"""Removes isolated nodes from the graph
    (functional name: :obj:`remove_isolated_nodes`)."""
    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        # Gather all nodes that occur in at least one edge (across all types):
        n_id_dict = defaultdict(list)
        for store in data.edge_stores:
            if 'edge_index' not in store:
                continue

            if store._key is None:
                src = dst = None
            else:
                src, _, dst = store._key

            n_id_dict[src].append(store.edge_index[0])
            n_id_dict[dst].append(store.edge_index[1])

        n_id_dict = {k: torch.cat(v).unique() for k, v in n_id_dict.items()}

        n_map_dict = {}
        for store in data.node_stores:
            if store._key not in n_id_dict:
                n_id_dict[store._key] = torch.empty((0, ), dtype=torch.long)

            idx = n_id_dict[store._key]
            mapping = idx.new_zeros(data.num_nodes)
            mapping[idx] = torch.arange(idx.numel(), device=mapping.device)
            n_map_dict[store._key] = mapping

        for store in data.edge_stores:
            if 'edge_index' not in store:
                continue

            if store._key is None:
                src = dst = None
            else:
                src, _, dst = store._key

            row = n_map_dict[src][store.edge_index[0]]
            col = n_map_dict[dst][store.edge_index[1]]
            store.edge_index = torch.stack([row, col], dim=0)

        old_data = copy.copy(data)
        for out, store in zip(data.node_stores, old_data.node_stores):
            for key, value in store.items():
                if key == 'num_nodes':
                    out.num_nodes = n_id_dict[store._key].numel()
                elif store.is_node_attr(key):
                    out[key] = value[n_id_dict[store._key]]

        return data
pytorch/torch_geometric/transforms/remove_training_classes.py000066400000000000000000000016771444262304300254150ustar00rootroot00000000000000from typing import List

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('remove_training_classes')
class RemoveTrainingClasses(BaseTransform):
    r"""Removes classes from the node-level training set as given by
    :obj:`data.train_mask`, *e.g.*, in order to get a zero-shot label scenario
    (functional name: :obj:`remove_training_classes`).

    Args:
        classes (List[int]): The classes to remove from the training set.
    """
    def __init__(self, classes: List[int]):
        self.classes = classes

    def forward(self, data: Data) -> Data:
        data.train_mask = data.train_mask.clone()
        for i in self.classes:
            data.train_mask[data.y == i] = False
        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.classes})'
pytorch/torch_geometric/transforms/rooted_subgraph.py000066400000000000000000000137371444262304300236770ustar00rootroot00000000000000import copy
from abc import ABC, abstractmethod
from typing import Any, Tuple

import torch
from torch import Tensor

from torch_geometric.data import Data
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_torch_csc_tensor


class RootedSubgraphData(Data):
    r"""A data object describing a homogeneous graph together with each node's
    rooted subgraph. It contains several additional properties that hold the
    information to map to batch of every node's rooted subgraph:

    * :obj:`sub_edge_index` (Tensor): The edge indices of all combined rooted
      subgraphs.
    * :obj:`n_id` (Tensor): The indices of nodes in all combined rooted
      subgraphs.
    * :obj:`e_id` (Tensor): The indices of edges in all combined rooted
      subgraphs.
    * :obj:`n_sub_batch` (Tensor): The batch vector to distinguish nodes across
      different subgraphs.
    * :obj:`e_sub_batch` (Tensor): The batch vector to distinguish edges across
      different subgraphs.
    """
    def __inc__(self, key, value, *args, **kwargs) -> Any:
        if key == 'sub_edge_index':
            return self.n_id.size(0)
        if key in ['n_sub_batch', 'e_sub_batch']:
            return 1 + int(self.n_sub_batch[-1])
        elif key == 'n_id':
            return self.num_nodes
        elif key == 'e_id':
            return self.edge_index.size(1)
        return super().__inc__(key, value, *args, **kwargs)

    def map_data(self) -> Data:
        # Maps all feature information of the :class:`Data` object to each
        # rooted subgraph.
        data = copy.copy(self)

        for key, value in self.items():
            if key in ['sub_edge_index', 'n_id', 'e_id', 'e_sub_batch']:
                del data[key]
            elif key == 'n_sub_batch':
                continue
            elif key == 'num_nodes':
                data.num_nodes = self.n_id.size(0)
            elif key == 'edge_index':
                data.edge_index = self.sub_edge_index
            elif self.is_node_attr(key):
                dim = self.__cat_dim__(key, value)
                data[key] = value.index_select(dim, self.n_id)
            elif self.is_edge_attr(key):
                dim = self.__cat_dim__(key, value)
                data[key] = value.index_select(dim, self.e_id)

        return data


class RootedSubgraph(BaseTransform, ABC):
    r"""Base class for implementing rooted subgraph transformations."""
    @abstractmethod
    def extract(
        self,
        data: Data,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:
        # Returns the tuple:
        # :obj:`(sub_edge_index, n_id, e_id, n_sub_batch, e_sub_batch)`
        # of the :class:`RootedSubgraphData` object.
        pass

    def map(
        self,
        data: Data,
        n_mask: Tensor,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:

        n_sub_batch, n_id = n_mask.nonzero().t()
        e_mask = n_mask[:, data.edge_index[0]] & n_mask[:, data.edge_index[1]]
        e_sub_batch, e_id = e_mask.nonzero().t()

        sub_edge_index = data.edge_index[:, e_id]
        arange = torch.arange(n_id.size(0), device=data.edge_index.device)
        node_map = data.edge_index.new_ones(data.num_nodes, data.num_nodes)
        node_map[n_sub_batch, n_id] = arange
        sub_edge_index += (arange * data.num_nodes)[e_sub_batch]
        sub_edge_index = node_map.view(-1)[sub_edge_index]

        return sub_edge_index, n_id, e_id, n_sub_batch, e_sub_batch

    def forward(self, data: Data) -> RootedSubgraphData:
        out = self.extract(data)
        d = RootedSubgraphData.from_dict(data.to_dict())
        d.sub_edge_index, d.n_id, d.e_id, d.n_sub_batch, d.e_sub_batch = out
        return d


class RootedEgoNets(RootedSubgraph):
    r"""Collects rooted :math:`k`-hop EgoNets for each node in the graph, as
    described in the `"From Stars to Subgraphs: Uplifting Any GNN with Local
    Structure Awareness" `_ paper.

    Args:
        num_hops (int): the number of hops :math:`k`.
    """
    def __init__(self, num_hops: int):
        super().__init__()
        self.num_hops = num_hops

    def extract(
        self,
        data: Data,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:

        adj_t = to_torch_csc_tensor(data.edge_index, size=data.size()).t()
        n_mask = torch.eye(data.num_nodes, device=data.edge_index.device)
        for _ in range(self.num_hops):
            n_mask += adj_t @ n_mask

        return self.map(data, n_mask > 0)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(num_hops={self.num_hops})'


class RootedRWSubgraph(RootedSubgraph):
    """Collects rooted random-walk based subgraphs for each node in the graph,
    as described in the `"From Stars to Subgraphs: Uplifting Any GNN with Local
    Structure Awareness" `_ paper.

    Args:
        walk_length (int): the length of the random walk.
        repeat (int, optional): The number of times of repeating the random
            walk to reduce randomness. (default: :obj:`1`)
    """
    def __init__(self, walk_length: int, repeat: int = 1):
        super().__init__()
        self.walk_length = walk_length
        self.repeat = repeat

    def extract(
        self,
        data: Data,
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:
        from torch_cluster import random_walk

        start = torch.arange(data.num_nodes, device=data.edge_index.device)
        start = start.view(-1, 1).repeat(1, self.repeat).view(-1)
        walk = random_walk(data.edge_index[0], data.edge_index[1], start,
                           self.walk_length, num_nodes=data.num_nodes)

        n_mask = torch.zeros((data.num_nodes, data.num_nodes),
                             dtype=torch.bool, device=walk.device)
        start = start.view(-1, 1).repeat(1, (self.walk_length + 1)).view(-1)
        n_mask[start, walk.view(-1)] = True

        return self.map(data, n_mask)

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(walk_length={self.walk_length})'
pytorch/torch_geometric/transforms/sample_points.py000066400000000000000000000041341444262304300233540ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('sample_points')
class SamplePoints(BaseTransform):
    r"""Uniformly samples a fixed number of points on the mesh faces according
    to their face area (functional name: :obj:`sample_points`).

    Args:
        num (int): The number of points to sample.
        remove_faces (bool, optional): If set to :obj:`False`, the face tensor
            will not be removed. (default: :obj:`True`)
        include_normals (bool, optional): If set to :obj:`True`, then compute
            normals for each sampled point. (default: :obj:`False`)
    """
    def __init__(
        self,
        num: int,
        remove_faces: bool = True,
        include_normals: bool = False,
    ):
        self.num = num
        self.remove_faces = remove_faces
        self.include_normals = include_normals

    def forward(self, data: Data) -> Data:
        pos, face = data.pos, data.face
        assert pos.size(1) == 3 and face.size(0) == 3

        pos_max = pos.abs().max()
        pos = pos / pos_max

        area = (pos[face[1]] - pos[face[0]]).cross(pos[face[2]] - pos[face[0]])
        area = area.norm(p=2, dim=1).abs() / 2

        prob = area / area.sum()
        sample = torch.multinomial(prob, self.num, replacement=True)
        face = face[:, sample]

        frac = torch.rand(self.num, 2, device=pos.device)
        mask = frac.sum(dim=-1) > 1
        frac[mask] = 1 - frac[mask]

        vec1 = pos[face[1]] - pos[face[0]]
        vec2 = pos[face[2]] - pos[face[0]]

        if self.include_normals:
            data.normal = torch.nn.functional.normalize(vec1.cross(vec2), p=2)

        pos_sampled = pos[face[0]]
        pos_sampled += frac[:, :1] * vec1
        pos_sampled += frac[:, 1:] * vec2

        pos_sampled = pos_sampled * pos_max
        data.pos = pos_sampled

        if self.remove_faces:
            data.face = None

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.num})'
pytorch/torch_geometric/transforms/sign.py000066400000000000000000000041201444262304300214320ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import scatter, to_torch_csc_tensor


@functional_transform('sign')
class SIGN(BaseTransform):
    r"""The Scalable Inception Graph Neural Network module (SIGN) from the
    `"SIGN: Scalable Inception Graph Neural Networks"
    `_ paper (functional name: :obj:`sign`),
    which precomputes the fixed representations

    .. math::
        \mathbf{X}^{(i)} = {\left( \mathbf{D}^{-1/2} \mathbf{A}
        \mathbf{D}^{-1/2} \right)}^i \mathbf{X}

    for :math:`i \in \{ 1, \ldots, K \}` and saves them in
    :obj:`data.x1`, :obj:`data.x2`, ...

    .. note::

        Since intermediate node representations are pre-computed, this operator
        is able to scale well to large graphs via classic mini-batching.
        For an example of using SIGN, see `examples/sign.py
        `_.

    Args:
        K (int): The number of hops/layer.
    """
    def __init__(self, K: int):
        self.K = K

    def forward(self, data: Data) -> Data:
        assert data.edge_index is not None
        row, col = data.edge_index
        N = data.num_nodes

        edge_weight = data.edge_weight
        if edge_weight is None:
            edge_weight = torch.ones(data.num_edges, device=row.device)

        deg = scatter(edge_weight, col, dim_size=N, reduce='sum')
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
        edge_weight = deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]
        adj = to_torch_csc_tensor(data.edge_index, edge_weight, size=(N, N))
        adj_t = adj.t()

        assert data.x is not None
        xs = [data.x]
        for i in range(1, self.K + 1):
            xs += [adj_t @ xs[-1]]
            data[f'x{i}'] = xs[-1]

        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}(K={self.K})'
pytorch/torch_geometric/transforms/spherical.py000066400000000000000000000043011444262304300224450ustar00rootroot00000000000000from math import pi as PI
from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('spherical')
class Spherical(BaseTransform):
    r"""Saves the spherical coordinates of linked nodes in its edge attributes
    (functional name: :obj:`spherical`).

    Args:
        norm (bool, optional): If set to :obj:`False`, the output will not be
            normalized to the interval :math:`{[0, 1]}^3`.
            (default: :obj:`True`)
        max_value (float, optional): If set and :obj:`norm=True`, normalization
            will be performed based on this value instead of the maximum value
            found in the data. (default: :obj:`None`)
        cat (bool, optional): If set to :obj:`False`, all existing edge
            attributes will be replaced. (default: :obj:`True`)
    """
    def __init__(
        self,
        norm: bool = True,
        max_value: Optional[float] = None,
        cat: bool = True,
    ):
        self.norm = norm
        self.max = max_value
        self.cat = cat

    def forward(self, data: Data) -> Data:
        (row, col), pos, pseudo = data.edge_index, data.pos, data.edge_attr
        assert pos.dim() == 2 and pos.size(1) == 3

        cart = pos[col] - pos[row]

        rho = torch.norm(cart, p=2, dim=-1).view(-1, 1)

        theta = torch.atan2(cart[..., 1], cart[..., 0]).view(-1, 1)
        theta = theta + (theta < 0).type_as(theta) * (2 * PI)

        phi = torch.acos(cart[..., 2] / rho.view(-1)).view(-1, 1)

        if self.norm:
            rho = rho / (rho.max() if self.max is None else self.max)
            theta = theta / (2 * PI)
            phi = phi / PI

        spher = torch.cat([rho, theta, phi], dim=-1)

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, spher.type_as(pos)], dim=-1)
        else:
            data.edge_attr = spher

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(norm={self.norm}, '
                f'max_value={self.max})')
pytorch/torch_geometric/transforms/svd_feature_reduction.py000066400000000000000000000017521444262304300250650ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('svd_feature_reduction')
class SVDFeatureReduction(BaseTransform):
    r"""Dimensionality reduction of node features via Singular Value
    Decomposition (SVD) (functional name: :obj:`svd_feature_reduction`).

    Args:
        out_channels (int): The dimensionlity of node features after
            reduction.
    """
    def __init__(self, out_channels: int):
        self.out_channels = out_channels

    def forward(self, data: Data) -> Data:
        if data.x.size(-1) > self.out_channels:
            U, S, _ = torch.linalg.svd(data.x)
            data.x = torch.mm(U[:, :self.out_channels],
                              torch.diag(S[:self.out_channels]))
        return data

    def __repr__(self) -> str:
        return f'{self.__class__.__name__}({self.out_channels})'
pytorch/torch_geometric/transforms/target_indegree.py000066400000000000000000000031071444262304300236260ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import degree


@functional_transform('target_indegree')
class TargetIndegree(BaseTransform):
    r"""Saves the globally normalized degree of target nodes
    (functional name: :obj:`target_indegree`)

    .. math::

        \mathbf{u}(i,j) = \frac{\deg(j)}{\max_{v \in \mathcal{V}} \deg(v)}

    in its edge attributes.

    Args:
        cat (bool, optional): Concat pseudo-coordinates to edge attributes
            instead of replacing them. (default: :obj:`True`)
    """
    def __init__(
        self,
        norm: bool = True,
        max_value: Optional[float] = None,
        cat: bool = True,
    ):
        self.norm = norm
        self.max = max_value
        self.cat = cat

    def forward(self, data: Data) -> Data:
        col, pseudo = data.edge_index[1], data.edge_attr

        deg = degree(col, data.num_nodes)

        if self.norm:
            deg = deg / (deg.max() if self.max is None else self.max)

        deg = deg[col]
        deg = deg.view(-1, 1)

        if pseudo is not None and self.cat:
            pseudo = pseudo.view(-1, 1) if pseudo.dim() == 1 else pseudo
            data.edge_attr = torch.cat([pseudo, deg.type_as(pseudo)], dim=-1)
        else:
            data.edge_attr = deg

        return data

    def __repr__(self) -> str:
        return (f'{self.__class__.__name__}(norm={self.norm}, '
                f'max_value={self.max})')
pytorch/torch_geometric/transforms/to_dense.py000066400000000000000000000044271444262304300223040ustar00rootroot00000000000000from typing import Optional

import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('to_dense')
class ToDense(BaseTransform):
    r"""Converts a sparse adjacency matrix to a dense adjacency matrix with
    shape :obj:`[num_nodes, num_nodes, *]` (functional name: :obj:`to_dense`).

    Args:
        num_nodes (int, optional): The number of nodes. If set to :obj:`None`,
            the number of nodes will get automatically inferred.
            (default: :obj:`None`)
    """
    def __init__(self, num_nodes: Optional[int] = None):
        self.num_nodes = num_nodes

    def forward(self, data: Data) -> Data:
        assert data.edge_index is not None

        orig_num_nodes = data.num_nodes
        if self.num_nodes is None:
            num_nodes = orig_num_nodes
        else:
            assert orig_num_nodes <= self.num_nodes
            num_nodes = self.num_nodes

        if data.edge_attr is None:
            edge_attr = torch.ones(data.edge_index.size(1), dtype=torch.float)
        else:
            edge_attr = data.edge_attr

        size = torch.Size([num_nodes, num_nodes] + list(edge_attr.size())[1:])
        adj = torch.sparse_coo_tensor(data.edge_index, edge_attr, size)
        data.adj = adj.to_dense()
        data.edge_index = None
        data.edge_attr = None

        data.mask = torch.zeros(num_nodes, dtype=torch.bool)
        data.mask[:orig_num_nodes] = 1

        if data.x is not None:
            size = [num_nodes - data.x.size(0)] + list(data.x.size())[1:]
            data.x = torch.cat([data.x, data.x.new_zeros(size)], dim=0)

        if data.pos is not None:
            size = [num_nodes - data.pos.size(0)] + list(data.pos.size())[1:]
            data.pos = torch.cat([data.pos, data.pos.new_zeros(size)], dim=0)

        if data.y is not None and (data.y.size(0) == orig_num_nodes):
            size = [num_nodes - data.y.size(0)] + list(data.y.size())[1:]
            data.y = torch.cat([data.y, data.y.new_zeros(size)], dim=0)

        return data

    def __repr__(self) -> str:
        if self.num_nodes is None:
            return f'{self.__class__.__name__}()'
        return f'{self.__class__.__name__}(num_nodes={self.num_nodes})'
pytorch/torch_geometric/transforms/to_device.py000066400000000000000000000026571444262304300224500ustar00rootroot00000000000000from typing import List, Optional, Union

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('to_device')
class ToDevice(BaseTransform):
    r"""Performs tensor device conversion, either for all attributes of the
    :obj:`~torch_geometric.data.Data` object or only the ones given by
    :obj:`attrs` (functional name: :obj:`to_device`).

    Args:
        device (torch.device): The destination device.
        attrs (List[str], optional): If given, will only perform tensor device
            conversion for the given attributes. (default: :obj:`None`)
        non_blocking (bool, optional): If set to :obj:`True` and tensor
            values are in pinned memory, the copy will be asynchronous with
            respect to the host. (default: :obj:`False`)
    """
    def __init__(
        self,
        device: Union[int, str],
        attrs: Optional[List[str]] = None,
        non_blocking: bool = False,
    ):
        self.device = device
        self.attrs = attrs or []
        self.non_blocking = non_blocking

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        return data.to(self.device, *self.attrs,
                       non_blocking=self.non_blocking)

    def __repr__(self):
        return f'{self.__class__.__name__}({self.device})'
pytorch/torch_geometric/transforms/to_sparse_tensor.py000066400000000000000000000123511444262304300240700ustar00rootroot00000000000000from typing import Optional, Union

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.typing import SparseTensor
from torch_geometric.utils import (
    sort_edge_index,
    to_torch_coo_tensor,
    to_torch_csr_tensor,
)


@functional_transform('to_sparse_tensor')
class ToSparseTensor(BaseTransform):
    r"""Converts the :obj:`edge_index` attributes of a homogeneous or
    heterogeneous data object into a **transposed**
    :class:`torch_sparse.SparseTensor` or :pytorch:`PyTorch`
    :class:`torch.sparse.Tensor` object with key :obj:`adj_t`
    (functional name: :obj:`to_sparse_tensor`).

    .. note::

        In case of composing multiple transforms, it is best to convert the
        :obj:`data` object via :class:`ToSparseTensor` as late as possible,
        since there exist some transforms that are only able to operate on
        :obj:`data.edge_index` for now.

    Args:
        attr (str, optional): The name of the attribute to add as a value to
            the :class:`~torch_sparse.SparseTensor` or
            :class:`torch.sparse.Tensor` object (if present).
            (default: :obj:`edge_weight`)
        remove_edge_index (bool, optional): If set to :obj:`False`, the
            :obj:`edge_index` tensor will not be removed.
            (default: :obj:`True`)
        fill_cache (bool, optional): If set to :obj:`True`, will fill the
            underlying :class:`torch_sparse.SparseTensor` cache (if used).
            (default: :obj:`True`)
        layout (torch.layout, optional): Specifies the layout of the returned
            sparse tensor (:obj:`None`, :obj:`torch.sparse_coo` or
            :obj:`torch.sparse_csr`).
            If set to :obj:`None` and the :obj:`torch_sparse` dependency is
            installed, will convert :obj:`edge_index` into a
            :class:`torch_sparse.SparseTensor` object.
            If set to :obj:`None` and the :obj:`torch_sparse` dependency is
            not installed, will convert :obj:`edge_index` into a
            :class:`torch.sparse.Tensor` object with layout
            :obj:`torch.sparse_csr`. (default: :obj:`None`)
    """
    def __init__(
        self,
        attr: Optional[str] = 'edge_weight',
        remove_edge_index: bool = True,
        fill_cache: bool = True,
        layout: Optional[int] = None,
    ):
        if layout not in {None, torch.sparse_coo, torch.sparse_csr}:
            raise ValueError(f"Unexpected sparse tensor layout "
                             f"(got '{layout}')")

        self.attr = attr
        self.remove_edge_index = remove_edge_index
        self.fill_cache = fill_cache
        self.layout = layout

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:

        for store in data.edge_stores:
            if 'edge_index' not in store:
                continue

            keys, values = [], []
            for key, value in store.items():
                if key == 'edge_index':
                    continue

                if store.is_edge_attr(key):
                    keys.append(key)
                    values.append(value)

            store.edge_index, values = sort_edge_index(
                store.edge_index,
                values,
                sort_by_row=False,
            )

            for key, value in zip(keys, values):
                store[key] = value

            layout = self.layout
            size = store.size()[::-1]
            value: Optional[Tensor] = None
            if self.attr is not None and self.attr in store:
                value = store[self.attr]

            if layout is None and torch_geometric.typing.WITH_TORCH_SPARSE:
                store.adj_t = SparseTensor(
                    row=store.edge_index[1],
                    col=store.edge_index[0],
                    value=value,
                    sparse_sizes=size,
                    is_sorted=True,
                    trust_data=True,
                )

            # TODO Multi-dimensional edge attributes only supported for COO.
            elif ((value is not None and value.dim() > 1)
                  or layout == torch.sparse_coo):
                store.adj_t = to_torch_coo_tensor(
                    store.edge_index.flip([0]),
                    edge_attr=value,
                    size=size,
                )

            elif layout is None or layout == torch.sparse_csr:
                store.adj_t = to_torch_csr_tensor(
                    store.edge_index.flip([0]),
                    edge_attr=value,
                    size=size,
                )

            if self.remove_edge_index:
                del store['edge_index']
                if self.attr is not None and self.attr in store:
                    del store[self.attr]

            if self.fill_cache and isinstance(store.adj_t, SparseTensor):
                # Pre-process some important attributes.
                store.adj_t.storage.rowptr()
                store.adj_t.storage.csr2csc()

        return data

    def __repr__(self):
        return (f'{self.__class__.__name__}(attr={self.attr}, '
                f'layout={self.layout})')
pytorch/torch_geometric/transforms/to_superpixels.py000066400000000000000000000050751444262304300235710ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import scatter


@functional_transform('to_slic')
class ToSLIC(BaseTransform):
    r"""Converts an image to a superpixel representation using the
    :meth:`skimage.segmentation.slic` algorithm, resulting in a
    :obj:`torch_geometric.data.Data` object holding the centroids of
    superpixels in :obj:`data.pos` and their mean color in :obj:`data.x`
    (functional name: :obj:`to_slic`).

    This transform can be used with any :obj:`torchvision` dataset.

    .. code-block:: python

        from torchvision.datasets import MNIST
        import torchvision.transforms as T
        from torch_geometric.transforms import ToSLIC

        transform = T.Compose([T.ToTensor(), ToSLIC(n_segments=75)])
        dataset = MNIST('/tmp/MNIST', download=True, transform=transform)

    Args:
        add_seg (bool, optional): If set to `True`, will add the segmentation
            result to the data object. (default: :obj:`False`)
        add_img (bool, optional): If set to `True`, will add the input image
            to the data object. (default: :obj:`False`)
        **kwargs (optional): Arguments to adjust the output of the SLIC
            algorithm. See the `SLIC documentation
            `_ for an overview.
    """
    def __init__(self, add_seg: bool = False, add_img: bool = False, **kwargs):
        self.add_seg = add_seg
        self.add_img = add_img
        self.kwargs = kwargs

    def forward(self, img: Tensor) -> Data:
        from skimage.segmentation import slic

        img = img.permute(1, 2, 0)
        h, w, c = img.size()

        seg = slic(img.to(torch.double).numpy(), start_label=0, **self.kwargs)
        seg = torch.from_numpy(seg)

        x = scatter(img.view(h * w, c), seg.view(h * w), dim=0, reduce='mean')

        pos_y = torch.arange(h, dtype=torch.float)
        pos_y = pos_y.view(-1, 1).repeat(1, w).view(h * w)
        pos_x = torch.arange(w, dtype=torch.float)
        pos_x = pos_x.view(1, -1).repeat(h, 1).view(h * w)

        pos = torch.stack([pos_x, pos_y], dim=-1)
        pos = scatter(pos, seg.view(h * w), dim=0, reduce='mean')

        data = Data(x=x, pos=pos)

        if self.add_seg:
            data.seg = seg.view(1, h, w)

        if self.add_img:
            data.img = img.permute(2, 0, 1).view(1, c, h, w)

        return data
pytorch/torch_geometric/transforms/to_undirected.py000066400000000000000000000056341444262304300233350ustar00rootroot00000000000000from typing import Union

import torch
from torch import Tensor

from torch_geometric.data import Data, HeteroData
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import to_undirected


@functional_transform('to_undirected')
class ToUndirected(BaseTransform):
    r"""Converts a homogeneous or heterogeneous graph to an undirected graph
    such that :math:`(j,i) \in \mathcal{E}` for every edge
    :math:`(i,j) \in \mathcal{E}` (functional name: :obj:`to_undirected`).
    In heterogeneous graphs, will add "reverse" connections for *all* existing
    edge types.

    Args:
        reduce (str, optional): The reduce operation to use for merging edge
            features (:obj:`"add"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"mul"`). (default: :obj:`"add"`)
        merge (bool, optional): If set to :obj:`False`, will create reverse
            edge types for connections pointing to the same source and target
            node type.
            If set to :obj:`True`, reverse edges will be merged into the
            original relation.
            This option only has effects in
            :class:`~torch_geometric.data.HeteroData` graph data.
            (default: :obj:`True`)
    """
    def __init__(self, reduce: str = "add", merge: bool = True):
        self.reduce = reduce
        self.merge = merge

    def forward(
        self,
        data: Union[Data, HeteroData],
    ) -> Union[Data, HeteroData]:
        for store in data.edge_stores:
            if 'edge_index' not in store:
                continue

            nnz = store.edge_index.size(1)

            if isinstance(data, HeteroData) and (store.is_bipartite()
                                                 or not self.merge):
                src, rel, dst = store._key

                # Just reverse the connectivity and add edge attributes:
                row, col = store.edge_index
                rev_edge_index = torch.stack([col, row], dim=0)

                inv_store = data[dst, f'rev_{rel}', src]
                inv_store.edge_index = rev_edge_index
                for key, value in store.items():
                    if key == 'edge_index':
                        continue
                    if isinstance(value, Tensor) and value.size(0) == nnz:
                        inv_store[key] = value

            else:
                keys, values = [], []
                for key, value in store.items():
                    if key == 'edge_index':
                        continue

                    if store.is_edge_attr(key):
                        keys.append(key)
                        values.append(value)

                store.edge_index, values = to_undirected(
                    store.edge_index, values, reduce=self.reduce)

                for key, value in zip(keys, values):
                    store[key] = value

        return data
pytorch/torch_geometric/transforms/two_hop.py000066400000000000000000000022761444262304300221630ustar00rootroot00000000000000import torch

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform
from torch_geometric.utils import (
    coalesce,
    remove_self_loops,
    to_edge_index,
    to_torch_csr_tensor,
)


@functional_transform('two_hop')
class TwoHop(BaseTransform):
    r"""Adds the two hop edges to the edge indices
    (functional name: :obj:`two_hop`)."""
    def forward(self, data: Data) -> Data:
        edge_index, edge_attr = data.edge_index, data.edge_attr
        N = data.num_nodes

        adj = to_torch_csr_tensor(edge_index, size=(N, N))
        edge_index2, _ = to_edge_index(adj @ adj)
        edge_index2, _ = remove_self_loops(edge_index2)

        edge_index = torch.cat([edge_index, edge_index2], dim=1)

        if edge_attr is not None:
            # We treat newly added edge features as "zero-features":
            edge_attr2 = edge_attr.new_zeros(edge_index2.size(1),
                                             *edge_attr.size()[1:])
            edge_attr = torch.cat([edge_attr, edge_attr2], dim=0)

        data.edge_index, data.edge_attr = coalesce(edge_index, edge_attr, N)

        return data
pytorch/torch_geometric/transforms/virtual_node.py000066400000000000000000000053371444262304300232000ustar00rootroot00000000000000import copy

import torch
from torch import Tensor

from torch_geometric.data import Data
from torch_geometric.data.datapipes import functional_transform
from torch_geometric.transforms import BaseTransform


@functional_transform('virtual_node')
class VirtualNode(BaseTransform):
    r"""Appends a virtual node to the given homogeneous graph that is connected
    to all other nodes, as described in the `"Neural Message Passing for
    Quantum Chemistry" `_ paper
    (functional name: :obj:`virtual_node`).
    The virtual node serves as a global scratch space that each node both reads
    from and writes to in every step of message passing.
    This allows information to travel long distances during the propagation
    phase.

    Node and edge features of the virtual node are added as zero-filled input
    features.
    Furthermore, special edge types will be added both for in-coming and
    out-going information to and from the virtual node.
    """
    def forward(self, data: Data) -> Data:
        num_nodes, (row, col) = data.num_nodes, data.edge_index
        edge_type = data.get('edge_type', torch.zeros_like(row))

        arange = torch.arange(num_nodes, device=row.device)
        full = row.new_full((num_nodes, ), num_nodes)
        row = torch.cat([row, arange, full], dim=0)
        col = torch.cat([col, full, arange], dim=0)
        edge_index = torch.stack([row, col], dim=0)

        new_type = edge_type.new_full((num_nodes, ), int(edge_type.max()) + 1)
        edge_type = torch.cat([edge_type, new_type, new_type + 1], dim=0)

        old_data = copy.copy(data)
        for key, value in old_data.items():
            if key == 'edge_index' or key == 'edge_type':
                continue

            if isinstance(value, Tensor):
                dim = old_data.__cat_dim__(key, value)
                size = list(value.size())

                fill_value = None
                if key == 'edge_weight':
                    size[dim] = 2 * num_nodes
                    fill_value = 1.
                elif key == 'batch':
                    size[dim] = 1
                    fill_value = int(value[0])
                elif old_data.is_edge_attr(key):
                    size[dim] = 2 * num_nodes
                    fill_value = 0.
                elif old_data.is_node_attr(key):
                    size[dim] = 1
                    fill_value = 0.

                if fill_value is not None:
                    new_value = value.new_full(size, fill_value)
                    data[key] = torch.cat([value, new_value], dim=dim)

        data.edge_index = edge_index
        data.edge_type = edge_type

        if 'num_nodes' in data:
            data.num_nodes = old_data.num_nodes + 1

        return data
pytorch/torch_geometric/typing.py000066400000000000000000000225471444262304300176230ustar00rootroot00000000000000import warnings
from typing import Dict, List, Optional, Tuple, Union

import numpy as np
import torch
from torch import Tensor

WITH_PT2 = int(torch.__version__.split('.')[0]) >= 2

try:
    import pyg_lib  # noqa
    WITH_PYG_LIB = True
    WITH_GMM = WITH_PT2 and hasattr(pyg_lib.ops, 'grouped_matmul')
    WITH_SAMPLED_OP = hasattr(pyg_lib.ops, 'sampled_add')
    WITH_INDEX_SORT = hasattr(pyg_lib.ops, 'index_sort')
    WITH_METIS = hasattr(pyg_lib, 'partition')
except (ImportError, OSError) as e:
    if isinstance(e, OSError):
        warnings.warn(f"An issue occurred while importing 'pyg-lib'. "
                      f"Disabling its usage. Stacktrace: {e}")
    pyg_lib = object
    WITH_PYG_LIB = False
    WITH_GMM = False
    WITH_SAMPLED_OP = False
    WITH_INDEX_SORT = False
    WITH_METIS = False

try:
    import torch_scatter  # noqa
    WITH_TORCH_SCATTER = True
except (ImportError, OSError) as e:
    if isinstance(e, OSError):
        warnings.warn(f"An issue occurred while importing 'torch-scatter'. "
                      f"Disabling its usage. Stacktrace: {e}")
    torch_scatter = object
    WITH_TORCH_SCATTER = False

try:
    import torch_cluster  # noqa
    WITH_TORCH_CLUSTER = True
    WITH_TORCH_CLUSTER_BATCH_SIZE = 'batch_size' in torch_cluster.knn.__doc__
except (ImportError, OSError) as e:
    if isinstance(e, OSError):
        warnings.warn(f"An issue occurred while importing 'torch-cluster'. "
                      f"Disabling its usage. Stacktrace: {e}")
    WITH_TORCH_CLUSTER = False

try:
    import torch_spline_conv  # noqa
    WITH_TORCH_SPLINE_CONV = True
except (ImportError, OSError) as e:
    if isinstance(e, OSError):
        warnings.warn(
            f"An issue occurred while importing 'torch-spline-conv'. "
            f"Disabling its usage. Stacktrace: {e}")
    WITH_TORCH_SPLINE_CONV = False

try:
    import torch_sparse  # noqa
    from torch_sparse import SparseStorage, SparseTensor
    WITH_TORCH_SPARSE = True
except (ImportError, OSError) as e:
    if isinstance(e, OSError):
        warnings.warn(f"An issue occurred while importing 'torch-sparse'. "
                      f"Disabling its usage. Stacktrace: {e}")
    WITH_TORCH_SPARSE = False

    class SparseStorage:
        def __init__(
            self,
            row: Optional[Tensor] = None,
            rowptr: Optional[Tensor] = None,
            col: Optional[Tensor] = None,
            value: Optional[Tensor] = None,
            sparse_sizes: Optional[Tuple[Optional[int], Optional[int]]] = None,
            rowcount: Optional[Tensor] = None,
            colptr: Optional[Tensor] = None,
            colcount: Optional[Tensor] = None,
            csr2csc: Optional[Tensor] = None,
            csc2csr: Optional[Tensor] = None,
            is_sorted: bool = False,
            trust_data: bool = False,
        ):
            raise ImportError("'SparseStorage' requires 'torch-sparse'")

    class SparseTensor:
        def __init__(
            self,
            row: Optional[Tensor] = None,
            rowptr: Optional[Tensor] = None,
            col: Optional[Tensor] = None,
            value: Optional[Tensor] = None,
            sparse_sizes: Optional[Tuple[Optional[int], Optional[int]]] = None,
            is_sorted: bool = False,
            trust_data: bool = False,
        ):
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        @classmethod
        def from_edge_index(
            self,
            edge_index: Tensor,
            edge_attr: Optional[Tensor] = None,
            sparse_sizes: Optional[Tuple[Optional[int], Optional[int]]] = None,
            is_sorted: bool = False,
            trust_data: bool = False,
        ) -> 'SparseTensor':
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        @classmethod
        def from_dense(self, mat: Tensor,
                       has_value: bool = True) -> 'SparseTensor':
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def size(self, dim: int) -> int:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def nnz(self) -> int:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def is_cuda(self) -> bool:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def has_value(self) -> bool:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def set_value(self, value: Optional[Tensor],
                      layout: Optional[str] = None) -> 'SparseTensor':
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def fill_value(self, fill_value: float,
                       dtype: Optional[torch.dtype] = None) -> 'SparseTensor':
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def coo(self) -> Tuple[Tensor, Tensor, Optional[Tensor]]:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def csr(self) -> Tuple[Tensor, Tensor, Optional[Tensor]]:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def requires_grad(self) -> bool:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

        def to_torch_sparse_csr_tensor(
            self,
            dtype: Optional[torch.dtype] = None,
        ) -> Tensor:
            raise ImportError("'SparseTensor' requires 'torch-sparse'")

    class torch_sparse:
        @staticmethod
        def matmul(src: SparseTensor, other: Tensor,
                   reduce: str = "sum") -> Tensor:
            raise ImportError("'matmul' requires 'torch-sparse'")

        @staticmethod
        def sum(src: SparseTensor, dim: Optional[int] = None) -> Tensor:
            raise ImportError("'sum' requires 'torch-sparse'")

        @staticmethod
        def mul(src: SparseTensor, other: Tensor) -> SparseTensor:
            raise ImportError("'mul' requires 'torch-sparse'")

        @staticmethod
        def set_diag(src: SparseTensor, values: Optional[Tensor] = None,
                     k: int = 0) -> SparseTensor:
            raise ImportError("'set_diag' requires 'torch-sparse'")

        @staticmethod
        def fill_diag(src: SparseTensor, fill_value: float,
                      k: int = 0) -> SparseTensor:
            raise ImportError("'fill_diag' requires 'torch-sparse'")

        @staticmethod
        def masked_select_nnz(src: SparseTensor, mask: Tensor,
                              layout: Optional[str] = None) -> SparseTensor:
            raise ImportError("'masked_select_nnz' requires 'torch-sparse'")


# Types for accessing data ####################################################

# Node-types are denoted by a single string, e.g.: `data['paper']`:
NodeType = str

# Edge-types are denotes by a triplet of strings, e.g.:
# `data[('author', 'writes', 'paper')]
EdgeType = Tuple[str, str, str]

DEFAULT_REL = 'to'
EDGE_TYPE_STR_SPLIT = '__'


class EdgeTypeStr(str):
    r"""A helper class to construct serializable edge types by merging an edge
    type tuple into a single string."""
    def __new__(cls, *args):
        if isinstance(args[0], (list, tuple)):
            # Unwrap `EdgeType((src, rel, dst))` and `EdgeTypeStr((src, dst))`:
            args = tuple(args[0])

        if len(args) == 1 and isinstance(args[0], str):
            args = args[0]  # An edge type string was passed.

        elif len(args) == 2 and all(isinstance(arg, str) for arg in args):
            # A `(src, dst)` edge type was passed - add `DEFAULT_REL`:
            args = (args[0], DEFAULT_REL, args[1])
            args = EDGE_TYPE_STR_SPLIT.join(args)

        elif len(args) == 3 and all(isinstance(arg, str) for arg in args):
            # A `(src, rel, dst)` edge type was passed:
            args = EDGE_TYPE_STR_SPLIT.join(args)

        else:
            raise ValueError(f"Encountered invalid edge type '{args}'")

        return str.__new__(cls, args)

    def to_tuple(self) -> EdgeType:
        r"""Returns the original edge type."""
        out = tuple(self.split(EDGE_TYPE_STR_SPLIT))
        if len(out) != 3:
            raise ValueError(f"Cannot convert the edge type '{self}' to a "
                             f"tuple since it holds invalid characters")
        return out


# There exist some short-cuts to query edge-types (given that the full triplet
# can be uniquely reconstructed, e.g.:
# * via str: `data['writes']`
# * via Tuple[str, str]: `data[('author', 'paper')]`
QueryType = Union[NodeType, EdgeType, str, Tuple[str, str]]

Metadata = Tuple[List[NodeType], List[EdgeType]]

# A representation of a feature tensor
FeatureTensorType = Union[Tensor, np.ndarray]

# A representation of an edge index, following the possible formats:
#   * COO: (row, col)
#   * CSC: (row, colptr)
#   * CSR: (rowptr, col)
EdgeTensorType = Tuple[Tensor, Tensor]

# Types for message passing ###################################################

Adj = Union[Tensor, SparseTensor]
OptTensor = Optional[Tensor]
PairTensor = Tuple[Tensor, Tensor]
OptPairTensor = Tuple[Tensor, Optional[Tensor]]
PairOptTensor = Tuple[Optional[Tensor], Optional[Tensor]]
Size = Optional[Tuple[int, int]]
NoneType = Optional[Tensor]

MaybeHeteroNodeTensor = Union[Tensor, Dict[NodeType, Tensor]]
MaybeHeteroEdgeTensor = Union[Tensor, Dict[EdgeType, Tensor]]

# Types for sampling ##########################################################

InputNodes = Union[OptTensor, NodeType, Tuple[NodeType, OptTensor]]
InputEdges = Union[OptTensor, EdgeType, Tuple[EdgeType, OptTensor]]
pytorch/torch_geometric/utils/000077500000000000000000000000001444262304300170655ustar00rootroot00000000000000pytorch/torch_geometric/utils/__init__.py000066400000000000000000000107051444262304300212010ustar00rootroot00000000000000import copy

from .scatter import scatter
from .segment import segment
from .sort import index_sort
from .degree import degree
from .softmax import softmax
from .dropout import dropout_adj, dropout_node, dropout_edge, dropout_path
from .augmentation import shuffle_node, mask_feature, add_random_edge
from .sort_edge_index import sort_edge_index
from .coalesce import coalesce
from .undirected import is_undirected, to_undirected
from .loop import (contains_self_loops, remove_self_loops,
                   segregate_self_loops, add_self_loops,
                   add_remaining_self_loops, get_self_loop_attr)
from .isolated import contains_isolated_nodes, remove_isolated_nodes
from .subgraph import (get_num_hops, subgraph, k_hop_subgraph,
                       bipartite_subgraph)
from .homophily import homophily
from .assortativity import assortativity
from .get_laplacian import get_laplacian
from .get_mesh_laplacian import get_mesh_laplacian
from .mask import mask_select, index_to_mask, mask_to_index
from .select import select, narrow
from .to_dense_batch import to_dense_batch
from .to_dense_adj import to_dense_adj
from .nested import to_nested_tensor, from_nested_tensor
from .sparse import (dense_to_sparse, is_sparse, is_torch_sparse_tensor,
                     to_torch_coo_tensor, to_torch_csr_tensor,
                     to_torch_csc_tensor, to_torch_sparse_tensor,
                     to_edge_index)
from .spmm import spmm
from .unbatch import unbatch, unbatch_edge_index
from .one_hot import one_hot
from .normalized_cut import normalized_cut
from .grid import grid
from .geodesic import geodesic_distance
from .convert import to_scipy_sparse_matrix, from_scipy_sparse_matrix
from .convert import to_networkx, from_networkx
from .convert import to_networkit, from_networkit
from .convert import to_trimesh, from_trimesh
from .convert import to_cugraph, from_cugraph
from .convert import to_dgl, from_dgl
from .smiles import from_smiles, to_smiles
from .random import (erdos_renyi_graph, stochastic_blockmodel_graph,
                     barabasi_albert_graph)
from .negative_sampling import (negative_sampling, batched_negative_sampling,
                                structured_negative_sampling,
                                structured_negative_sampling_feasible)
from .tree_decomposition import tree_decomposition
from .embedding import get_embeddings
from .trim_to_layer import trim_to_layer
from .train_test_split_edges import train_test_split_edges

__all__ = [
    'scatter',
    'segment',
    'index_sort',
    'degree',
    'softmax',
    'dropout_node',
    'dropout_edge',
    'dropout_path',
    'dropout_adj',
    'shuffle_node',
    'mask_feature',
    'add_random_edge',
    'sort_edge_index',
    'coalesce',
    'is_undirected',
    'to_undirected',
    'contains_self_loops',
    'remove_self_loops',
    'segregate_self_loops',
    'add_self_loops',
    'add_remaining_self_loops',
    'get_self_loop_attr',
    'contains_isolated_nodes',
    'remove_isolated_nodes',
    'get_num_hops',
    'subgraph',
    'bipartite_subgraph',
    'k_hop_subgraph',
    'homophily',
    'assortativity',
    'get_laplacian',
    'get_mesh_laplacian',
    'mask_select',
    'index_to_mask',
    'mask_to_index',
    'select',
    'narrow',
    'to_dense_batch',
    'to_dense_adj',
    'to_nested_tensor',
    'from_nested_tensor',
    'dense_to_sparse',
    'is_torch_sparse_tensor',
    'is_sparse',
    'to_torch_coo_tensor',
    'to_torch_csr_tensor',
    'to_torch_csc_tensor',
    'to_torch_sparse_tensor',
    'to_edge_index',
    'spmm',
    'unbatch',
    'unbatch_edge_index',
    'one_hot',
    'normalized_cut',
    'grid',
    'geodesic_distance',
    'to_scipy_sparse_matrix',
    'from_scipy_sparse_matrix',
    'to_networkx',
    'from_networkx',
    'to_networkit',
    'from_networkit',
    'to_trimesh',
    'from_trimesh',
    'to_cugraph',
    'from_cugraph',
    'to_dgl',
    'from_dgl',
    'from_smiles',
    'to_smiles',
    'erdos_renyi_graph',
    'stochastic_blockmodel_graph',
    'barabasi_albert_graph',
    'negative_sampling',
    'batched_negative_sampling',
    'structured_negative_sampling',
    'structured_negative_sampling_feasible',
    'tree_decomposition',
    'get_embeddings',
    'trim_to_layer',
    'train_test_split_edges',
]

# `structured_negative_sampling_feasible` is a long name and thus destroys the
# documentation rendering. We remove it for now from the documentation:
classes = copy.copy(__all__)
classes.remove('structured_negative_sampling_feasible')
pytorch/torch_geometric/utils/assortativity.py000066400000000000000000000044531444262304300223720ustar00rootroot00000000000000import torch
from torch import Tensor

from torch_geometric.typing import Adj, SparseTensor
from torch_geometric.utils import coalesce, degree
from torch_geometric.utils.to_dense_adj import to_dense_adj


def assortativity(edge_index: Adj) -> float:
    r"""The degree assortativity coefficient from the
    `"Mixing patterns in networks"
    `_ paper.
    Assortativity in a network refers to the tendency of nodes to
    connect with other similar nodes over dissimilar nodes.
    It is computed from Pearson correlation coefficient of the node degrees.

    Args:
        edge_index (Tensor or SparseTensor): The graph connectivity.

    Returns:
        The value of the degree assortativity coefficient for the input
        graph :math:`\in [-1, 1]`

    Example:

        >>> edge_index = torch.tensor([[0, 1, 2, 3, 2],
        ...                            [1, 2, 0, 1, 3]])
        >>> assortativity(edge_index)
        -0.666667640209198
    """
    if isinstance(edge_index, SparseTensor):
        adj: SparseTensor = edge_index
        row, col, _ = adj.coo()
    else:
        assert isinstance(edge_index, Tensor)
        row, col = edge_index

    device = row.device
    out_deg = degree(row, dtype=torch.long)
    in_deg = degree(col, dtype=torch.long)
    degrees = torch.unique(torch.cat([out_deg, in_deg]))
    mapping = row.new_zeros(degrees.max().item() + 1)
    mapping[degrees] = torch.arange(degrees.size(0), device=device)

    # Compute degree mixing matrix (joint probability distribution) `M`
    num_degrees = degrees.size(0)
    src_deg = mapping[out_deg[row]]
    dst_deg = mapping[in_deg[col]]

    pairs = torch.stack([src_deg, dst_deg], dim=0)
    occurrence = torch.ones(pairs.size(1), device=device)
    pairs, occurrence = coalesce(pairs, occurrence)
    M = to_dense_adj(pairs, edge_attr=occurrence, max_num_nodes=num_degrees)[0]
    # normalization
    M /= M.sum()

    # numeric assortativity coefficient, computed by
    # Pearson correlation coefficient of the node degrees
    x = y = degrees.float()
    a, b = M.sum(0), M.sum(1)

    vara = (a * x**2).sum() - ((a * x).sum())**2
    varb = (b * x**2).sum() - ((b * x).sum())**2
    xy = torch.outer(x, y)
    ab = torch.outer(a, b)
    out = (xy * (M - ab)).sum() / (vara * varb).sqrt()
    return out.item()
pytorch/torch_geometric/utils/augmentation.py000066400000000000000000000215701444262304300221370ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.utils import scatter
from torch_geometric.utils.num_nodes import maybe_num_nodes


def shuffle_node(x: Tensor, batch: Optional[Tensor] = None,
                 training: bool = True) -> Tuple[Tensor, Tensor]:
    r"""Randomly shuffle the feature matrix :obj:`x` along the
    first dimmension.

    The method returns (1) the shuffled :obj:`x`, (2) the permutation
    indicating the orders of original nodes after shuffling.

    Args:
        x (FloatTensor): The feature matrix.
        batch (LongTensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. Must be ordered. (default: :obj:`None`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`FloatTensor`, :class:`LongTensor`)

    Example:

        >>> # Standard case
        >>> x = torch.tensor([[0, 1, 2],
        ...                   [3, 4, 5],
        ...                   [6, 7, 8],
        ...                   [9, 10, 11]], dtype=torch.float)
        >>> x, node_perm = shuffle_node(x)
        >>> x
        tensor([[ 3.,  4.,  5.],
                [ 9., 10., 11.],
                [ 0.,  1.,  2.],
                [ 6.,  7.,  8.]])
        >>> node_perm
        tensor([1, 3, 0, 2])

        >>> # For batched graphs as inputs
        >>> batch = torch.tensor([0, 0, 1, 1])
        >>> x, node_perm = shuffle_node(x, batch)
        >>> x
        tensor([[ 3.,  4.,  5.],
                [ 0.,  1.,  2.],
                [ 9., 10., 11.],
                [ 6.,  7.,  8.]])
        >>> node_perm
        tensor([1, 0, 3, 2])
    """
    if not training:
        perm = torch.arange(x.size(0), device=x.device)
        return x, perm
    if batch is None:
        perm = torch.randperm(x.size(0), device=x.device)
        return x[perm], perm
    num_nodes = scatter(batch.new_ones(x.size(0)), batch, dim=0, reduce='sum')
    cumsum = torch.cat([batch.new_zeros(1), num_nodes.cumsum(dim=0)])
    perm = torch.cat([
        torch.randperm(n, device=x.device) + offset
        for offset, n in zip(cumsum[:-1], num_nodes)
    ])
    return x[perm], perm


def mask_feature(x: Tensor, p: float = 0.5, mode: str = 'col',
                 fill_value: float = 0.,
                 training: bool = True) -> Tuple[Tensor, Tensor]:
    r"""Randomly masks feature from the feature matrix
    :obj:`x` with probability :obj:`p` using samples from
    a Bernoulli distribution.

    The method returns (1) the retained :obj:`x`, (2) the feature
    mask broadcastable with :obj:`x` (:obj:`mode='row'` and :obj:`mode='col'`)
    or with the same shape as :obj:`x` (:obj:`mode='all'`),
    indicating where features are retained.

    Args:
        x (FloatTensor): The feature matrix.
        p (float, optional): The masking ratio. (default: :obj:`0.5`)
        mode (str, optional): The masked scheme to use for feature masking.
            (:obj:`"row"`, :obj:`"col"` or :obj:`"all"`).
            If :obj:`mode='col'`, will mask entire features of all nodes
            from the feature matrix. If :obj:`mode='row'`, will mask entire
            nodes from the feature matrix. If :obj:`mode='all'`, will mask
            individual features across all nodes. (default: :obj:`'col'`)
        fill_value (float, optional): The value for masked features in the
            output tensor. (default: :obj:`0`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`FloatTensor`, :class:`BoolTensor`)

    Examples:

        >>> # Masked features are column-wise sampled
        >>> x = torch.tensor([[1, 2, 3],
        ...                   [4, 5, 6],
        ...                   [7, 8, 9]], dtype=torch.float)
        >>> x, feat_mask = mask_feature(x)
        >>> x
        tensor([[1., 0., 3.],
                [4., 0., 6.],
                [7., 0., 9.]]),
        >>> feat_mask
        tensor([[True, False, True]])

        >>> # Masked features are row-wise sampled
        >>> x, feat_mask = mask_feature(x, mode='row')
        >>> x
        tensor([[1., 2., 3.],
                [0., 0., 0.],
                [7., 8., 9.]]),
        >>> feat_mask
        tensor([[True], [False], [True]])

        >>> # Masked features are uniformly sampled
        >>> x, feat_mask = mask_feature(x, mode='all')
        >>> x
        tensor([[0., 0., 0.],
                [4., 0., 6.],
                [0., 0., 9.]])
        >>> feat_mask
        tensor([[False, False, False],
                [True, False,  True],
                [False, False,  True]])
    """
    if p < 0. or p > 1.:
        raise ValueError(f'Masking ratio has to be between 0 and 1 '
                         f'(got {p}')
    if not training or p == 0.0:
        return x, torch.ones_like(x, dtype=torch.bool)
    assert mode in ['row', 'col', 'all']

    if mode == 'row':
        mask = torch.rand(x.size(0), device=x.device) >= p
        mask = mask.view(-1, 1)
    elif mode == 'col':
        mask = torch.rand(x.size(1), device=x.device) >= p
        mask = mask.view(1, -1)
    else:
        mask = torch.rand_like(x) >= p

    x = x.masked_fill(~mask, fill_value)
    return x, mask


def add_random_edge(edge_index, p: float, force_undirected: bool = False,
                    num_nodes: Optional[Union[Tuple[int], int]] = None,
                    training: bool = True) -> Tuple[Tensor, Tensor]:
    r"""Randomly adds edges to :obj:`edge_index`.

    The method returns (1) the retained :obj:`edge_index`, (2) the added
    edge indices.

    Args:
        edge_index (LongTensor): The edge indices.
        p (float): Ratio of added edges to the existing edges.
        force_undirected (bool, optional): If set to :obj:`True`,
            added edges will be undirected.
            (default: :obj:`False`)
        num_nodes (int, Tuple[int], optional): The overall number of nodes,
            *i.e.* :obj:`max_val + 1`, or the number of source and
            destination nodes, *i.e.* :obj:`(max_src_val + 1, max_dst_val + 1)`
            of :attr:`edge_index`. (default: :obj:`None`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`LongTensor`, :class:`LongTensor`)

    Examples:

        >>> # Standard case
        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> edge_index, added_edges = add_random_edge(edge_index, p=0.5)
        >>> edge_index
        tensor([[0, 1, 1, 2, 2, 3, 2, 1, 3],
                [1, 0, 2, 1, 3, 2, 0, 2, 1]])
        >>> added_edges
        tensor([[2, 1, 3],
                [0, 2, 1]])

        >>> # The returned graph is kept undirected
        >>> edge_index, added_edges = add_random_edge(edge_index, p=0.5,
        ...                                           force_undirected=True)
        >>> edge_index
        tensor([[0, 1, 1, 2, 2, 3, 2, 1, 3, 0, 2, 1],
                [1, 0, 2, 1, 3, 2, 0, 2, 1, 2, 1, 3]])
        >>> added_edges
        tensor([[2, 1, 3, 0, 2, 1],
                [0, 2, 1, 2, 1, 3]])

        >>> # For bipartite graphs
        >>> edge_index = torch.tensor([[0, 1, 2, 3, 4, 5],
        ...                            [2, 3, 1, 4, 2, 1]])
        >>> edge_index, added_edges = add_random_edge(edge_index, p=0.5,
        ...                                           num_nodes=(6, 5))
        >>> edge_index
        tensor([[0, 1, 2, 3, 4, 5, 3, 4, 1],
                [2, 3, 1, 4, 2, 1, 1, 3, 2]])
        >>> added_edges
        tensor([[3, 4, 1],
                [1, 3, 2]])
    """
    if p < 0. or p > 1.:
        raise ValueError(f'Ratio of added edges has to be between 0 and 1 '
                         f'(got {p}')
    if force_undirected and isinstance(num_nodes, (tuple, list)):
        raise RuntimeError('`force_undirected` is not supported for'
                           ' heterogeneous graphs')

    device = edge_index.device
    if not training or p == 0.0:
        edge_index_to_add = torch.tensor([[], []], device=device)
        return edge_index, edge_index_to_add

    if not isinstance(num_nodes, (tuple, list)):
        num_nodes = (num_nodes, num_nodes)
    num_src_nodes = maybe_num_nodes(edge_index, num_nodes[0])
    num_dst_nodes = maybe_num_nodes(edge_index, num_nodes[1])

    num_edges_to_add = round(edge_index.size(1) * p)
    row = torch.randint(0, num_src_nodes, size=(num_edges_to_add, ))
    col = torch.randint(0, num_dst_nodes, size=(num_edges_to_add, ))

    if force_undirected:
        mask = row < col
        row, col = row[mask], col[mask]
        row, col = torch.cat([row, col]), torch.cat([col, row])
    edge_index_to_add = torch.stack([row, col], dim=0).to(device)
    edge_index = torch.cat([edge_index, edge_index_to_add], dim=1)
    return edge_index, edge_index_to_add
pytorch/torch_geometric/utils/coalesce.py000066400000000000000000000116551444262304300212250ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import index_sort, scatter
from torch_geometric.utils.num_nodes import maybe_num_nodes

MISSING = '???'


@torch.jit._overload
def coalesce(edge_index, edge_attr, num_nodes, reduce, is_sorted, sort_by_row):
    # type: (Tensor, str, Optional[int], str, bool, bool) -> Tensor  # noqa
    pass


@torch.jit._overload
def coalesce(edge_index, edge_attr, num_nodes, reduce, is_sorted, sort_by_row):
    # type: (Tensor, Optional[Tensor], Optional[int], str, bool, bool) -> Tuple[Tensor, Optional[Tensor]]  # noqa
    pass


@torch.jit._overload
def coalesce(edge_index, edge_attr, num_nodes, reduce, is_sorted, sort_by_row):
    # type: (Tensor, List[Tensor], Optional[int], str, bool, bool) -> Tuple[Tensor, List[Tensor]]  # noqa
    pass


def coalesce(
    edge_index: Tensor,
    edge_attr: Union[OptTensor, List[Tensor], str] = MISSING,
    num_nodes: Optional[int] = None,
    reduce: str = 'add',
    is_sorted: bool = False,
    sort_by_row: bool = True,
) -> Union[Tensor, Tuple[Tensor, OptTensor], Tuple[Tensor, List[Tensor]]]:
    """Row-wise sorts :obj:`edge_index` and removes its duplicated entries.
    Duplicate entries in :obj:`edge_attr` are merged by scattering them
    together according to the given :obj:`reduce` option.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor or List[Tensor], optional): Edge weights or multi-
            dimensional edge features.
            If given as a list, will re-shuffle and remove duplicates for all
            its entries. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        reduce (str, optional): The reduce operation to use for merging edge
            features (:obj:`"add"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"mul"`, :obj:`"any"`). (default: :obj:`"add"`)
        is_sorted (bool, optional): If set to :obj:`True`, will expect
            :obj:`edge_index` to be already sorted row-wise.
        sort_by_row (bool, optional): If set to :obj:`False`, will sort
            :obj:`edge_index` column-wise.

    :rtype: :class:`LongTensor` if :attr:`edge_attr` is not passed, else
        (:class:`LongTensor`, :obj:`Optional[Tensor]` or :obj:`List[Tensor]]`)

    .. warning::

        From :pyg:`PyG >= 2.3.0` onwards, this function will always return a
        tuple whenever :obj:`edge_attr` is passed as an argument (even in case
        it is set to :obj:`None`).

    Example:

        >>> edge_index = torch.tensor([[1, 1, 2, 3],
        ...                            [3, 3, 1, 2]])
        >>> edge_attr = torch.tensor([1., 1., 1., 1.])
        >>> coalesce(edge_index)
        tensor([[1, 2, 3],
                [3, 1, 2]])

        >>> # Sort `edge_index` column-wise
        >>> coalesce(edge_index, sort_by_row=False)
        tensor([[2, 3, 1],
                [1, 2, 3]])

        >>> coalesce(edge_index, edge_attr)
        (tensor([[1, 2, 3],
                [3, 1, 2]]),
        tensor([2., 1., 1.]))

        >>> # Use 'mean' operation to merge edge features
        >>> coalesce(edge_index, edge_attr, reduce='mean')
        (tensor([[1, 2, 3],
                [3, 1, 2]]),
        tensor([1., 1., 1.]))
    """
    nnz = edge_index.size(1)
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    idx = edge_index.new_empty(nnz + 1)
    idx[0] = -1
    idx[1:] = edge_index[1 - int(sort_by_row)]
    idx[1:].mul_(num_nodes).add_(edge_index[int(sort_by_row)])

    if not is_sorted:
        idx[1:], perm = index_sort(idx[1:], max_value=num_nodes * num_nodes)
        edge_index = edge_index[:, perm]
        if isinstance(edge_attr, Tensor):
            edge_attr = edge_attr[perm]
        elif isinstance(edge_attr, (list, tuple)):
            edge_attr = [e[perm] for e in edge_attr]

    mask = idx[1:] > idx[:-1]

    # Only perform expensive merging in case there exists duplicates:
    if mask.all():
        if edge_attr is None or isinstance(edge_attr, (Tensor, list, tuple)):
            return edge_index, edge_attr
        return edge_index

    edge_index = edge_index[:, mask]

    dim_size: Optional[int] = None
    if isinstance(edge_attr, (Tensor, list, tuple)) and len(edge_attr) > 0:
        dim_size = edge_index.size(1)
        idx = torch.arange(0, nnz, device=edge_index.device)
        idx.sub_(mask.logical_not_().cumsum(dim=0))

    if edge_attr is None:
        return edge_index, None
    if isinstance(edge_attr, Tensor):
        edge_attr = scatter(edge_attr, idx, 0, dim_size, reduce)
        return edge_index, edge_attr
    if isinstance(edge_attr, (list, tuple)):
        if len(edge_attr) == 0:
            return edge_index, edge_attr
        edge_attr = [scatter(e, idx, 0, dim_size, reduce) for e in edge_attr]
        return edge_index, edge_attr

    return edge_index
pytorch/torch_geometric/utils/convert.py000066400000000000000000000462561444262304300211340ustar00rootroot00000000000000from collections import defaultdict
from typing import Any, Iterable, List, Optional, Tuple, Union

import scipy.sparse
import torch
from torch import Tensor
from torch.utils.dlpack import from_dlpack, to_dlpack

import torch_geometric
from torch_geometric.utils.num_nodes import maybe_num_nodes


def to_scipy_sparse_matrix(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    num_nodes: Optional[int] = None,
) -> scipy.sparse.coo_matrix:
    r"""Converts a graph given by edge indices and edge attributes to a scipy
    sparse matrix.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`index`. (default: :obj:`None`)

    Examples:

        >>> edge_index = torch.tensor([
        ...     [0, 1, 1, 2, 2, 3],
        ...     [1, 0, 2, 1, 3, 2],
        ... ])
        >>> to_scipy_sparse_matrix(edge_index)
        <4x4 sparse matrix of type ''
            with 6 stored elements in COOrdinate format>
    """
    row, col = edge_index.cpu()

    if edge_attr is None:
        edge_attr = torch.ones(row.size(0))
    else:
        edge_attr = edge_attr.view(-1).cpu()
        assert edge_attr.size(0) == row.size(0)

    N = maybe_num_nodes(edge_index, num_nodes)
    out = scipy.sparse.coo_matrix(
        (edge_attr.numpy(), (row.numpy(), col.numpy())), (N, N))
    return out


def from_scipy_sparse_matrix(
        A: scipy.sparse.spmatrix) -> Tuple[Tensor, Tensor]:
    r"""Converts a scipy sparse matrix to edge indices and edge attributes.

    Args:
        A (scipy.sparse): A sparse matrix.

    Examples:

        >>> edge_index = torch.tensor([
        ...     [0, 1, 1, 2, 2, 3],
        ...     [1, 0, 2, 1, 3, 2],
        ... ])
        >>> adj = to_scipy_sparse_matrix(edge_index)
        >>> # `edge_index` and `edge_weight` are both returned
        >>> from_scipy_sparse_matrix(adj)
        (tensor([[0, 1, 1, 2, 2, 3],
                [1, 0, 2, 1, 3, 2]]),
        tensor([1., 1., 1., 1., 1., 1.]))
    """
    A = A.tocoo()
    row = torch.from_numpy(A.row).to(torch.long)
    col = torch.from_numpy(A.col).to(torch.long)
    edge_index = torch.stack([row, col], dim=0)
    edge_weight = torch.from_numpy(A.data)
    return edge_index, edge_weight


def to_networkx(
    data: 'torch_geometric.data.Data',
    node_attrs: Optional[Iterable[str]] = None,
    edge_attrs: Optional[Iterable[str]] = None,
    graph_attrs: Optional[Iterable[str]] = None,
    to_undirected: Optional[Union[bool, str]] = False,
    remove_self_loops: bool = False,
) -> Any:
    r"""Converts a :class:`torch_geometric.data.Data` instance to a
    :obj:`networkx.Graph` if :attr:`to_undirected` is set to :obj:`True`, or
    a directed :obj:`networkx.DiGraph` otherwise.

    Args:
        data (torch_geometric.data.Data): The data object.
        node_attrs (iterable of str, optional): The node attributes to be
            copied. (default: :obj:`None`)
        edge_attrs (iterable of str, optional): The edge attributes to be
            copied. (default: :obj:`None`)
        graph_attrs (iterable of str, optional): The graph attributes to be
            copied. (default: :obj:`None`)
        to_undirected (bool or str, optional): If set to :obj:`True` or
            "upper", will return a :obj:`networkx.Graph` instead of a
            :obj:`networkx.DiGraph`. The undirected graph will correspond to
            the upper triangle of the corresponding adjacency matrix.
            Similarly, if set to "lower", the undirected graph will correspond
            to the lower triangle of the adjacency matrix. (default:
            :obj:`False`)
        remove_self_loops (bool, optional): If set to :obj:`True`, will not
            include self loops in the resulting graph. (default: :obj:`False`)

    Examples:

        >>> edge_index = torch.tensor([
        ...     [0, 1, 1, 2, 2, 3],
        ...     [1, 0, 2, 1, 3, 2],
        ... ])
        >>> data = Data(edge_index=edge_index, num_nodes=4)
        >>> to_networkx(data)
        

    """
    import networkx as nx

    G = nx.Graph() if to_undirected else nx.DiGraph()

    G.add_nodes_from(range(data.num_nodes))

    node_attrs = node_attrs or []
    edge_attrs = edge_attrs or []
    graph_attrs = graph_attrs or []

    values = {}
    for key, value in data(*(node_attrs + edge_attrs + graph_attrs)):
        if torch.is_tensor(value):
            value = value if value.dim() <= 1 else value.squeeze(-1)
            values[key] = value.tolist()
        else:
            values[key] = value

    to_undirected = "upper" if to_undirected is True else to_undirected
    to_undirected_upper = True if to_undirected == "upper" else False
    to_undirected_lower = True if to_undirected == "lower" else False

    for i, (u, v) in enumerate(data.edge_index.t().tolist()):

        if to_undirected_upper and u > v:
            continue
        elif to_undirected_lower and u < v:
            continue

        if remove_self_loops and u == v:
            continue

        G.add_edge(u, v)

        for key in edge_attrs:
            G[u][v][key] = values[key][i]

    for key in node_attrs:
        for i, feat_dict in G.nodes(data=True):
            feat_dict.update({key: values[key][i]})

    for key in graph_attrs:
        G.graph[key] = values[key]

    return G


def from_networkx(
    G: Any,
    group_node_attrs: Optional[Union[List[str], all]] = None,
    group_edge_attrs: Optional[Union[List[str], all]] = None,
) -> 'torch_geometric.data.Data':
    r"""Converts a :obj:`networkx.Graph` or :obj:`networkx.DiGraph` to a
    :class:`torch_geometric.data.Data` instance.

    Args:
        G (networkx.Graph or networkx.DiGraph): A networkx graph.
        group_node_attrs (List[str] or all, optional): The node attributes to
            be concatenated and added to :obj:`data.x`. (default: :obj:`None`)
        group_edge_attrs (List[str] or all, optional): The edge attributes to
            be concatenated and added to :obj:`data.edge_attr`.
            (default: :obj:`None`)

    .. note::

        All :attr:`group_node_attrs` and :attr:`group_edge_attrs` values must
        be numeric.

    Examples:

        >>> edge_index = torch.tensor([
        ...     [0, 1, 1, 2, 2, 3],
        ...     [1, 0, 2, 1, 3, 2],
        ... ])
        >>> data = Data(edge_index=edge_index, num_nodes=4)
        >>> g = to_networkx(data)
        >>> # A `Data` object is returned
        >>> from_networkx(g)
        Data(edge_index=[2, 6], num_nodes=4)
    """
    import networkx as nx

    from torch_geometric.data import Data

    G = G.to_directed() if not nx.is_directed(G) else G

    mapping = dict(zip(G.nodes(), range(G.number_of_nodes())))
    edge_index = torch.empty((2, G.number_of_edges()), dtype=torch.long)
    for i, (src, dst) in enumerate(G.edges()):
        edge_index[0, i] = mapping[src]
        edge_index[1, i] = mapping[dst]

    data = defaultdict(list)

    if G.number_of_nodes() > 0:
        node_attrs = list(next(iter(G.nodes(data=True)))[-1].keys())
    else:
        node_attrs = {}

    if G.number_of_edges() > 0:
        edge_attrs = list(next(iter(G.edges(data=True)))[-1].keys())
    else:
        edge_attrs = {}

    for i, (_, feat_dict) in enumerate(G.nodes(data=True)):
        if set(feat_dict.keys()) != set(node_attrs):
            raise ValueError('Not all nodes contain the same attributes')
        for key, value in feat_dict.items():
            data[str(key)].append(value)

    for i, (_, _, feat_dict) in enumerate(G.edges(data=True)):
        if set(feat_dict.keys()) != set(edge_attrs):
            raise ValueError('Not all edges contain the same attributes')
        for key, value in feat_dict.items():
            key = f'edge_{key}' if key in node_attrs else key
            data[str(key)].append(value)

    for key, value in G.graph.items():
        if key == 'node_default' or key == 'edge_default':
            continue  # Do not load default attributes.
        key = f'graph_{key}' if key in node_attrs else key
        data[str(key)] = value

    for key, value in data.items():
        if isinstance(value, (tuple, list)) and isinstance(value[0], Tensor):
            data[key] = torch.stack(value, dim=0)
        else:
            try:
                data[key] = torch.tensor(value)
            except (ValueError, TypeError):
                pass

    data['edge_index'] = edge_index.view(2, -1)
    data = Data.from_dict(data)

    if group_node_attrs is all:
        group_node_attrs = list(node_attrs)
    if group_node_attrs is not None:
        xs = []
        for key in group_node_attrs:
            x = data[key]
            x = x.view(-1, 1) if x.dim() <= 1 else x
            xs.append(x)
            del data[key]
        data.x = torch.cat(xs, dim=-1)

    if group_edge_attrs is all:
        group_edge_attrs = list(edge_attrs)
    if group_edge_attrs is not None:
        xs = []
        for key in group_edge_attrs:
            key = f'edge_{key}' if key in node_attrs else key
            x = data[key]
            x = x.view(-1, 1) if x.dim() <= 1 else x
            xs.append(x)
            del data[key]
        data.edge_attr = torch.cat(xs, dim=-1)

    if data.x is None and data.pos is None:
        data.num_nodes = G.number_of_nodes()

    return data


def to_networkit(
    edge_index: Tensor,
    edge_weight: Optional[Tensor] = None,
    num_nodes: Optional[int] = None,
    directed: bool = True,
) -> Any:
    r"""Converts a :obj:`(edge_index, edge_weight)` tuple to a
    :class:`networkit.Graph`.

    Args:
        edge_index (torch.Tensor): The edge indices of the graph.
        edge_weight (torch.Tensor, optional): The edge weights of the graph.
            (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes in the graph.
            (default: :obj:`None`)
        directed (bool, optional): If set to :obj:`False`, the graph will be
            undirected. (default: :obj:`True`)
    """
    import networkit as nk

    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    g = nk.graph.Graph(
        num_nodes,
        weighted=edge_weight is not None,
        directed=directed,
    )

    if edge_weight is None:
        edge_weight = torch.ones(edge_index.size(1))

    if not directed:
        mask = edge_index[0] <= edge_index[1]
        edge_index = edge_index[:, mask]
        edge_weight = edge_weight[mask]

    for (u, v), w in zip(edge_index.t().tolist(), edge_weight.tolist()):
        g.addEdge(u, v, w)

    return g


def from_networkit(g: Any) -> Tuple[Tensor, Optional[Tensor]]:
    r"""Converts a :class:`networkit.Graph` to a
    :obj:`(edge_index, edge_weight)` tuple.
    If the :class:`networkit.Graph` is not weighted, the returned
    :obj:`edge_weight` will be :obj:`None`.

    Args:
        g (networkkit.graph.Graph): A :obj:`networkit` graph object.
    """
    is_directed = g.isDirected()
    is_weighted = g.isWeighted()

    edge_indices, edge_weights = [], []
    for u, v, w in g.iterEdgesWeights():
        edge_indices.append([u, v])
        edge_weights.append(w)
        if not is_directed:
            edge_indices.append([v, u])
            edge_weights.append(w)

    edge_index = torch.tensor(edge_indices).t().contiguous()
    edge_weight = torch.tensor(edge_weights) if is_weighted else None

    return edge_index, edge_weight


def to_trimesh(data):
    r"""Converts a :class:`torch_geometric.data.Data` instance to a
    :obj:`trimesh.Trimesh`.

    Args:
        data (torch_geometric.data.Data): The data object.

    Example:

        >>> pos = torch.tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 0]],
        ...                    dtype=torch.float)
        >>> face = torch.tensor([[0, 1, 2], [1, 2, 3]]).t()

        >>> data = Data(pos=pos, face=face)
        >>> to_trimesh(data)
        
    """
    import trimesh
    return trimesh.Trimesh(vertices=data.pos.detach().cpu().numpy(),
                           faces=data.face.detach().t().cpu().numpy(),
                           process=False)


def from_trimesh(mesh):
    r"""Converts a :obj:`trimesh.Trimesh` to a
    :class:`torch_geometric.data.Data` instance.

    Args:
        mesh (trimesh.Trimesh): A :obj:`trimesh` mesh.

Example:

    Example:

        >>> pos = torch.tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 0]],
        ...                    dtype=torch.float)
        >>> face = torch.tensor([[0, 1, 2], [1, 2, 3]]).t()

        >>> data = Data(pos=pos, face=face)
        >>> mesh = to_trimesh(data)
        >>> from_trimesh(mesh)
        Data(pos=[4, 3], face=[3, 2])
    """
    from torch_geometric.data import Data

    pos = torch.from_numpy(mesh.vertices).to(torch.float)
    face = torch.from_numpy(mesh.faces).t().contiguous()

    return Data(pos=pos, face=face)


def to_cugraph(edge_index: Tensor, edge_weight: Optional[Tensor] = None,
               relabel_nodes: bool = True, directed: bool = True):
    r"""Converts a graph given by :obj:`edge_index` and optional
    :obj:`edge_weight` into a :obj:`cugraph` graph object.

    Args:
        edge_index (torch.Tensor): The edge indices of the graph.
        edge_weight (torch.Tensor, optional): The edge weights of the graph.
            (default: :obj:`None`)
        relabel_nodes (bool, optional): If set to :obj:`True`,
            :obj:`cugraph` will remove any isolated nodes, leading to a
            relabeling of nodes. (default: :obj:`True`)
        directed (bool, optional): If set to :obj:`False`, the graph will be
            undirected. (default: :obj:`True`)
    """
    import cudf
    import cugraph

    g = cugraph.Graph(directed=directed)
    df = cudf.from_dlpack(to_dlpack(edge_index.t()))

    if edge_weight is not None:
        assert edge_weight.dim() == 1
        df['2'] = cudf.from_dlpack(to_dlpack(edge_weight))

    g.from_cudf_edgelist(
        df,
        source=0,
        destination=1,
        edge_attr='2' if edge_weight is not None else None,
        renumber=relabel_nodes,
    )

    return g


def from_cugraph(g: Any) -> Tuple[Tensor, Optional[Tensor]]:
    r"""Converts a :obj:`cugraph` graph object into :obj:`edge_index` and
    optional :obj:`edge_weight` tensors.

    Args:
        g (cugraph.Graph): A :obj:`cugraph` graph object.
    """
    df = g.view_edge_list()

    src = from_dlpack(df['src'].to_dlpack()).long()
    dst = from_dlpack(df['dst'].to_dlpack()).long()
    edge_index = torch.stack([src, dst], dim=0)

    edge_weight = None
    if 'weights' in df:
        edge_weight = from_dlpack(df['weights'].to_dlpack())

    return edge_index, edge_weight


def to_dgl(
    data: Union['torch_geometric.data.Data', 'torch_geometric.data.HeteroData']
) -> Any:
    r"""Converts a :class:`torch_geometric.data.Data` or
    :class:`torch_geometric.data.HeteroData` instance to a :obj:`dgl` graph
    object.

    Args:
        data (torch_geometric.data.Data or torch_geometric.data.HeteroData):
            The data object.

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 3, 0], [1, 0, 2, 1, 4, 4]])
        >>> x = torch.randn(5, 3)
        >>> edge_attr = torch.randn(6, 2)
        >>> data = Data(x=x, edge_index=edge_index, edge_attr=y)
        >>> g = to_dgl(data)
        >>> g
        Graph(num_nodes=5, num_edges=6,
            ndata_schemes={'x': Scheme(shape=(3,))}
            edata_schemes={'edge_attr': Scheme(shape=(2, ))})

        >>> data = HeteroData()
        >>> data['paper'].x = torch.randn(5, 3)
        >>> data['author'].x = torch.ones(5, 3)
        >>> edge_index = torch.tensor([[0, 1, 2, 3, 4], [0, 1, 2, 3, 4]])
        >>> data['author', 'cites', 'paper'].edge_index = edge_index
        >>> g = to_dgl(data)
        >>> g
        Graph(num_nodes={'author': 5, 'paper': 5},
            num_edges={('author', 'cites', 'paper'): 5},
            metagraph=[('author', 'paper', 'cites')])
    """
    import dgl

    from torch_geometric.data import Data, HeteroData

    if isinstance(data, Data):
        if data.edge_index is not None:
            row, col = data.edge_index
        else:
            row, col, _ = data.adj_t.t().coo()

        g = dgl.graph((row, col))

        for attr in data.node_attrs():
            g.ndata[attr] = data[attr]
        for attr in data.edge_attrs():
            if attr in ['edge_index', 'adj_t']:
                continue
            g.edata[attr] = data[attr]

        return g

    if isinstance(data, HeteroData):
        data_dict = {}
        for edge_type, store in data.edge_items():
            if store.get('edge_index') is not None:
                row, col = store.edge_index
            else:
                row, col, _ = store['adj_t'].t().coo()

            data_dict[edge_type] = (row, col)

        g = dgl.heterograph(data_dict)

        for node_type, store in data.node_items():
            for attr, value in store.items():
                g.nodes[node_type].data[attr] = value

        for edge_type, store in data.edge_items():
            for attr, value in store.items():
                if attr in ['edge_index', 'adj_t']:
                    continue
                g.edges[edge_type].data[attr] = value

        return g

    raise ValueError(f"Invalid data type (got '{type(data)}')")


def from_dgl(
    g: Any,
) -> Union['torch_geometric.data.Data', 'torch_geometric.data.HeteroData']:
    r"""Converts a :obj:`dgl` graph object to a
    :class:`torch_geometric.data.Data` or
    :class:`torch_geometric.data.HeteroData` instance.

    Args:
        g (dgl.DGLGraph): The :obj:`dgl` graph object.

    Example:

        >>> g = dgl.graph(([0, 0, 1, 5], [1, 2, 2, 0]))
        >>> g.ndata['x'] = torch.randn(g.num_nodes(), 3)
        >>> g.edata['edge_attr'] = torch.randn(g.num_edges(), 2)
        >>> data = from_dgl(g)
        >>> data
        Data(x=[6, 3], edge_attr=[4, 2], edge_index=[2, 4])

        >>> g = dgl.heterograph({
        >>> g = dgl.heterograph({
        ...     ('author', 'writes', 'paper'): ([0, 1, 1, 2, 3, 3, 4],
        ...                                     [0, 0, 1, 1, 1, 2, 2])})
        >>> g.nodes['author'].data['x'] = torch.randn(5, 3)
        >>> g.nodes['paper'].data['x'] = torch.randn(5, 3)
        >>> data = from_dgl(g)
        >>> data
        HeteroData(
        author={ x=[5, 3] },
        paper={ x=[3, 3] },
        (author, writes, paper)={ edge_index=[2, 7] }
        )
    """
    import dgl

    from torch_geometric.data import Data, HeteroData

    if not isinstance(g, dgl.DGLGraph):
        raise ValueError(f"Invalid data type (got '{type(g)}')")

    if g.is_homogeneous:
        data = Data()
        data.edge_index = torch.stack(g.edges(), dim=0)

        for attr, value in g.ndata.items():
            data[attr] = value
        for attr, value in g.edata.items():
            data[attr] = value

        return data

    data = HeteroData()

    for node_type in g.ntypes:
        for attr, value in g.nodes[node_type].data.items():
            data[node_type][attr] = value

    for edge_type in g.canonical_etypes:
        row, col = g.edges(form="uv", etype=edge_type)
        data[edge_type].edge_index = torch.stack([row, col], dim=0)
        for attr, value in g.edge_attr_schemes(edge_type).items():
            data[edge_type][attr] = value

    return data
pytorch/torch_geometric/utils/cross_entropy.py000066400000000000000000000044111444262304300223500ustar00rootroot00000000000000from typing import Tuple

import torch
from torch import Tensor

from torch_geometric.utils import scatter


class SparseCrossEntropy(torch.autograd.Function):
    # We implement our own custom autograd function for this to avoid the
    # double gradient computation to `inputs`.
    @staticmethod
    def forward(ctx, inputs: Tensor, edge_label_index: Tensor) -> Tensor:
        assert inputs.dim() == 2

        logsumexp = inputs.logsumexp(dim=-1)
        ctx.save_for_backward(inputs, edge_label_index, logsumexp)

        out = inputs[edge_label_index[0], edge_label_index[1]]
        out.neg_().add_(logsumexp[edge_label_index[0]])

        return out.sum() / inputs.size(0)

    @staticmethod
    @torch.autograd.function.once_differentiable
    def backward(ctx, grad_out: Tensor) -> Tuple[Tensor, None]:
        inputs, edge_label_index, logsumexp = ctx.saved_tensors

        grad_out = grad_out / inputs.size(0)

        grad_logsumexp = scatter(grad_out.expand(edge_label_index.size(1)),
                                 edge_label_index[0], dim=0,
                                 dim_size=inputs.size(0), reduce='sum')

        # Gradient computation of `logsumexp`: `grad * (self - result).exp()`
        grad_input = (inputs - logsumexp.view(-1, 1))
        grad_input.exp_()
        grad_input.mul_(grad_logsumexp.view(-1, 1))

        grad_input[edge_label_index[0], edge_label_index[1]] -= grad_out

        return grad_input, None


def sparse_cross_entropy(inputs: Tensor, edge_label_index: Tensor) -> Tensor:
    r"""A sparse-label variant of :func:`torch.nn.functional.cross_entropy`.
    In particular, the binary target matrix is solely given by sparse indices
    :obj:`edge_label_index`.

    Args:
        inputs (torch.Tensor): The predicted unnormalized logits of shape
            :obj:`[batch_size, num_classes]`.
        edge_index (torch.Tensor): The sparse ground-truth indices of
            shape :obj:`[2, num_labels]`.

    :rtype: :class:`torch.Tensor`

    Example:

        >>> inputs = torch.randn(2, 3)
        >>> edge_label_index = torch.tensor([[0, 0, 1],
        ...                                  [0, 1, 2]])
        >>> sparse_cross_entropy(inputs, edge_label_index)
        tensor(1.2919)
    """
    return SparseCrossEntropy.apply(inputs, edge_label_index)
pytorch/torch_geometric/utils/degree.py000066400000000000000000000017721444262304300207010ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.utils.num_nodes import maybe_num_nodes


def degree(index: Tensor, num_nodes: Optional[int] = None,
           dtype: Optional[torch.dtype] = None) -> Tensor:
    r"""Computes the (unweighted) degree of a given one-dimensional index
    tensor.

    Args:
        index (LongTensor): Index tensor.
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`index`. (default: :obj:`None`)
        dtype (:obj:`torch.dtype`, optional): The desired data type of the
            returned tensor.

    :rtype: :class:`Tensor`

    Example:

        >>> row = torch.tensor([0, 1, 0, 2, 0])
        >>> degree(row, dtype=torch.long)
        tensor([3, 1, 1])
    """
    N = maybe_num_nodes(index, num_nodes)
    out = torch.zeros((N, ), dtype=dtype, device=index.device)
    one = torch.ones((index.size(0), ), dtype=out.dtype, device=out.device)
    return out.scatter_add_(0, index, one)
pytorch/torch_geometric/utils/dropout.py000066400000000000000000000260731444262304300211430ustar00rootroot00000000000000from typing import Optional, Tuple

import torch

try:
    import torch_cluster  # noqa
    random_walk = torch.ops.torch_cluster.random_walk
except ImportError:
    random_walk = None
from torch import Tensor

from torch_geometric.deprecation import deprecated
from torch_geometric.typing import OptTensor
from torch_geometric.utils.degree import degree
from torch_geometric.utils.num_nodes import maybe_num_nodes
from torch_geometric.utils.sort_edge_index import sort_edge_index
from torch_geometric.utils.subgraph import subgraph


def filter_adj(row: Tensor, col: Tensor, edge_attr: OptTensor,
               mask: Tensor) -> Tuple[Tensor, Tensor, OptTensor]:
    return row[mask], col[mask], None if edge_attr is None else edge_attr[mask]


@deprecated("use 'dropout_edge' instead")
def dropout_adj(
    edge_index: Tensor,
    edge_attr: OptTensor = None,
    p: float = 0.5,
    force_undirected: bool = False,
    num_nodes: Optional[int] = None,
    training: bool = True,
) -> Tuple[Tensor, OptTensor]:
    r"""Randomly drops edges from the adjacency matrix
    :obj:`(edge_index, edge_attr)` with probability :obj:`p` using samples from
    a Bernoulli distribution.

    .. warning::

        :class:`~torch_geometric.utils.dropout_adj` is deprecated and will
        be removed in a future release.
        Use :class:`torch_geometric.utils.dropout_edge` instead.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)
        p (float, optional): Dropout probability. (default: :obj:`0.5`)
        force_undirected (bool, optional): If set to :obj:`True`, will either
            drop or keep both edges of an undirected edge.
            (default: :obj:`False`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> edge_attr = torch.tensor([1, 2, 3, 4, 5, 6])
        >>> dropout_adj(edge_index, edge_attr)
        (tensor([[0, 1, 2, 3],
                [1, 2, 3, 2]]),
        tensor([1, 3, 5, 6]))

        >>> # The returned graph is kept undirected
        >>> dropout_adj(edge_index, edge_attr, force_undirected=True)
        (tensor([[0, 1, 2, 1, 2, 3],
                [1, 2, 3, 0, 1, 2]]),
        tensor([1, 3, 5, 1, 3, 5]))
    """

    if p < 0. or p > 1.:
        raise ValueError(f'Dropout probability has to be between 0 and 1 '
                         f'(got {p}')

    if not training or p == 0.0:
        return edge_index, edge_attr

    row, col = edge_index

    mask = torch.rand(row.size(0), device=edge_index.device) >= p

    if force_undirected:
        mask[row > col] = False

    row, col, edge_attr = filter_adj(row, col, edge_attr, mask)

    if force_undirected:
        edge_index = torch.stack(
            [torch.cat([row, col], dim=0),
             torch.cat([col, row], dim=0)], dim=0)
        if edge_attr is not None:
            edge_attr = torch.cat([edge_attr, edge_attr], dim=0)
    else:
        edge_index = torch.stack([row, col], dim=0)

    return edge_index, edge_attr


def dropout_node(edge_index: Tensor, p: float = 0.5,
                 num_nodes: Optional[int] = None,
                 training: bool = True) -> Tuple[Tensor, Tensor, Tensor]:
    r"""Randomly drops nodes from the adjacency matrix
    :obj:`edge_index` with probability :obj:`p` using samples from
    a Bernoulli distribution.

    The method returns (1) the retained :obj:`edge_index`, (2) the edge mask
    indicating which edges were retained. (3) the node mask indicating
    which nodes were retained.

    Args:
        edge_index (LongTensor): The edge indices.
        p (float, optional): Dropout probability. (default: :obj:`0.5`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`LongTensor`, :class:`BoolTensor`, :class:`BoolTensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> edge_index, edge_mask, node_mask = dropout_node(edge_index)
        >>> edge_index
        tensor([[0, 1],
                [1, 0]])
        >>> edge_mask
        tensor([ True,  True, False, False, False, False])
        >>> node_mask
        tensor([ True,  True, False, False])
    """
    if p < 0. or p > 1.:
        raise ValueError(f'Dropout probability has to be between 0 and 1 '
                         f'(got {p}')

    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    if not training or p == 0.0:
        node_mask = edge_index.new_ones(num_nodes, dtype=torch.bool)
        edge_mask = edge_index.new_ones(edge_index.size(1), dtype=torch.bool)
        return edge_index, edge_mask, node_mask

    prob = torch.rand(num_nodes, device=edge_index.device)
    node_mask = prob > p
    edge_index, _, edge_mask = subgraph(node_mask, edge_index,
                                        num_nodes=num_nodes,
                                        return_edge_mask=True)
    return edge_index, edge_mask, node_mask


def dropout_edge(edge_index: Tensor, p: float = 0.5,
                 force_undirected: bool = False,
                 training: bool = True) -> Tuple[Tensor, Tensor]:
    r"""Randomly drops edges from the adjacency matrix
    :obj:`edge_index` with probability :obj:`p` using samples from
    a Bernoulli distribution.

    The method returns (1) the retained :obj:`edge_index`, (2) the edge mask
    or index indicating which edges were retained, depending on the argument
    :obj:`force_undirected`.

    Args:
        edge_index (LongTensor): The edge indices.
        p (float, optional): Dropout probability. (default: :obj:`0.5`)
        force_undirected (bool, optional): If set to :obj:`True`, will either
            drop or keep both edges of an undirected edge.
            (default: :obj:`False`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`LongTensor`, :class:`BoolTensor` or :class:`LongTensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> edge_index, edge_mask = dropout_edge(edge_index)
        >>> edge_index
        tensor([[0, 1, 2, 2],
                [1, 2, 1, 3]])
        >>> edge_mask # masks indicating which edges are retained
        tensor([ True, False,  True,  True,  True, False])

        >>> edge_index, edge_id = dropout_edge(edge_index,
        ...                                    force_undirected=True)
        >>> edge_index
        tensor([[0, 1, 2, 1, 2, 3],
                [1, 2, 3, 0, 1, 2]])
        >>> edge_id # indices indicating which edges are retained
        tensor([0, 2, 4, 0, 2, 4])
    """
    if p < 0. or p > 1.:
        raise ValueError(f'Dropout probability has to be between 0 and 1 '
                         f'(got {p}')

    if not training or p == 0.0:
        edge_mask = edge_index.new_ones(edge_index.size(1), dtype=torch.bool)
        return edge_index, edge_mask

    row, col = edge_index

    edge_mask = torch.rand(row.size(0), device=edge_index.device) >= p

    if force_undirected:
        edge_mask[row > col] = False

    edge_index = edge_index[:, edge_mask]

    if force_undirected:
        edge_index = torch.cat([edge_index, edge_index.flip(0)], dim=1)
        edge_mask = edge_mask.nonzero().repeat((2, 1)).squeeze()

    return edge_index, edge_mask


def dropout_path(edge_index: Tensor, p: float = 0.2, walks_per_node: int = 1,
                 walk_length: int = 3, num_nodes: Optional[int] = None,
                 is_sorted: bool = False,
                 training: bool = True) -> Tuple[Tensor, Tensor]:
    r"""Drops edges from the adjacency matrix :obj:`edge_index`
    based on random walks. The source nodes to start random walks from are
    sampled from :obj:`edge_index` with probability :obj:`p`, following
    a Bernoulli distribution.

    The method returns (1) the retained :obj:`edge_index`, (2) the edge mask
    indicating which edges were retained.

    Args:
        edge_index (LongTensor): The edge indices.
        p (float, optional): Sample probability. (default: :obj:`0.2`)
        walks_per_node (int, optional): The number of walks per node, same as
            :class:`~torch_geometric.nn.models.Node2Vec`. (default: :obj:`1`)
        walk_length (int, optional): The walk length, same as
            :class:`~torch_geometric.nn.models.Node2Vec`. (default: :obj:`3`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        is_sorted (bool, optional): If set to :obj:`True`, will expect
            :obj:`edge_index` to be already sorted row-wise.
            (default: :obj:`False`)
        training (bool, optional): If set to :obj:`False`, this operation is a
            no-op. (default: :obj:`True`)

    :rtype: (:class:`LongTensor`, :class:`BoolTensor`)

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> edge_index, edge_mask = dropout_path(edge_index)
        >>> edge_index
        tensor([[1, 2],
                [2, 3]])
        >>> edge_mask # masks indicating which edges are retained
        tensor([False, False,  True, False,  True, False])
    """

    if p < 0. or p > 1.:
        raise ValueError(f'Sample probability has to be between 0 and 1 '
                         f'(got {p}')

    num_edges = edge_index.size(1)
    edge_mask = edge_index.new_ones(num_edges, dtype=torch.bool)
    if not training or p == 0.0:
        return edge_index, edge_mask

    if random_walk is None:
        raise ImportError('`dropout_path` requires `torch-cluster`.')

    num_nodes = maybe_num_nodes(edge_index, num_nodes)
    edge_orders = None
    ori_edge_index = edge_index
    if not is_sorted:
        edge_orders = torch.arange(num_edges, device=edge_index.device)
        edge_index, edge_orders = sort_edge_index(edge_index, edge_orders,
                                                  num_nodes=num_nodes)

    row, col = edge_index
    sample_mask = torch.rand(row.size(0), device=edge_index.device) <= p
    start = row[sample_mask].repeat(walks_per_node)

    deg = degree(row, num_nodes=num_nodes)
    rowptr = row.new_zeros(num_nodes + 1)
    torch.cumsum(deg, 0, out=rowptr[1:])
    n_id, e_id = random_walk(rowptr, col, start, walk_length, 1.0, 1.0)
    e_id = e_id[e_id != -1].view(-1)  # filter illegal edges

    if edge_orders is not None:
        # permute edge ids
        e_id = edge_orders[e_id]
    edge_mask[e_id] = False
    edge_index = ori_edge_index[:, edge_mask]

    return edge_index, edge_mask
pytorch/torch_geometric/utils/embedding.py000066400000000000000000000031711444262304300213570ustar00rootroot00000000000000import warnings
from typing import Any, List

import torch
from torch import Tensor


def get_embeddings(
    model: torch.nn.Module,
    *args,
    **kwargs,
) -> List[Tensor]:
    """Returns the output embeddings of all
    :class:`~torch_geometric.nn.conv.MessagePassing` layers in
    :obj:`model`.

    Internally, this method registers forward hooks on all
    :class:`~torch_geometric.nn.conv.MessagePassing` layers of a :obj:`model`,
    and runs the forward pass of the :obj:`model` by calling
    :obj:`model(*args, **kwargs)`.

    Args:
        model (torch.nn.Module): The message passing model.
        *args: Arguments passed to the model.
        **kwargs (optional): Additional keyword arguments passed to the model.
    """
    from torch_geometric.nn import MessagePassing

    embeddings: List[Tensor] = []

    def hook(model: torch.nn.Module, inputs: Any, outputs: Any):
        # Clone output in case it will be later modified in-place:
        outputs = outputs[0] if isinstance(outputs, tuple) else outputs
        assert isinstance(outputs, Tensor)
        embeddings.append(outputs.clone())

    hook_handles = []
    for module in model.modules():  # Register forward hooks:
        if isinstance(module, MessagePassing):
            hook_handles.append(module.register_forward_hook(hook))

    if len(hook_handles) == 0:
        warnings.warn("The 'model' does not have any 'MessagePassing' layers")

    training = model.training
    model.eval()
    with torch.no_grad():
        model(*args, **kwargs)
    model.train(training)

    for handle in hook_handles:  # Remove hooks:
        handle.remove()

    return embeddings
pytorch/torch_geometric/utils/geodesic.py000066400000000000000000000077121444262304300212300ustar00rootroot00000000000000import multiprocessing as mp
from typing import Optional

import numpy as np
import torch
from torch import Tensor


def geodesic_distance(
    pos: Tensor,
    face: Tensor,
    src: Optional[Tensor] = None,
    dest: Optional[Tensor] = None,
    norm: bool = True,
    max_distance: Optional[float] = None,
    num_workers: int = 0,
) -> Tensor:
    r"""Computes (normalized) geodesic distances of a mesh given by :obj:`pos`
    and :obj:`face`. If :obj:`src` and :obj:`dest` are given, this method only
    computes the geodesic distances for the respective source and target
    node-pairs.

    .. note::

        This function requires the :obj:`gdist` package.
        To install, run :obj:`pip install cython && pip install gdist`.

    Args:
        pos (Tensor): The node positions.
        face (LongTensor): The face indices.
        src (LongTensor, optional): If given, only compute geodesic distances
            for the specified source indices. (default: :obj:`None`)
        dest (LongTensor, optional): If given, only compute geodesic distances
            for the specified target indices. (default: :obj:`None`)
        norm (bool, optional): Normalizes geodesic distances by
            :math:`\sqrt{\textrm{area}(\mathcal{M})}`. (default: :obj:`True`)
        max_distance (float, optional): If given, only yields results for
            geodesic distances less than :obj:`max_distance`. This will speed
            up runtime dramatically. (default: :obj:`None`)
        num_workers (int, optional): How many subprocesses to use for
            calculating geodesic distances.
            :obj:`0` means that computation takes place in the main process.
            :obj:`-1` means that the available amount of CPU cores is used.
            (default: :obj:`0`)

    :rtype: :class:`Tensor`

    Example:

        >>> pos = torch.Tensor([[0, 0, 0],
        ...                     [2, 0, 0],
        ...                     [0, 2, 0],
        ...                     [2, 2, 0]])
        >>> face = torch.tensor([[0, 0],
        ...                      [1, 2],
        ...                      [3, 3]])
        >>> geodesic_distance(pos, face)
        [[0, 1, 1, 1.4142135623730951],
        [1, 0, 1.4142135623730951, 1],
        [1, 1.4142135623730951, 0, 1],
        [1.4142135623730951, 1, 1, 0]]
    """
    import gdist

    max_distance = float('inf') if max_distance is None else max_distance

    if norm:
        area = (pos[face[1]] - pos[face[0]]).cross(pos[face[2]] - pos[face[0]])
        norm = (area.norm(p=2, dim=1) / 2).sum().sqrt().item()
    else:
        norm = 1.0

    dtype = pos.dtype

    pos = pos.detach().cpu().to(torch.double).numpy()
    face = face.detach().t().cpu().to(torch.int).numpy()

    if src is None and dest is None:
        out = gdist.local_gdist_matrix(pos, face,
                                       max_distance * norm).toarray() / norm
        return torch.from_numpy(out).to(dtype)

    if src is None:
        src = np.arange(pos.shape[0], dtype=np.int32)
    else:
        src = src.detach().cpu().to(torch.int).numpy()

    dest = None if dest is None else dest.detach().cpu().to(torch.int).numpy()

    def _parallel_loop(pos, face, src, dest, max_distance, norm, i, dtype):
        s = src[i:i + 1]
        d = None if dest is None else dest[i:i + 1]
        out = gdist.compute_gdist(pos, face, s, d, max_distance * norm) / norm
        return torch.from_numpy(out).to(dtype)

    num_workers = mp.cpu_count() if num_workers <= -1 else num_workers
    if num_workers > 0:
        with mp.Pool(num_workers) as pool:
            outs = pool.starmap(
                _parallel_loop,
                [(pos, face, src, dest, max_distance, norm, i, dtype)
                 for i in range(len(src))])
    else:
        outs = [
            _parallel_loop(pos, face, src, dest, max_distance, norm, i, dtype)
            for i in range(len(src))
        ]

    out = torch.cat(outs, dim=0)

    if dest is None:
        out = out.view(-1, pos.shape[0])

    return out
pytorch/torch_geometric/utils/get_laplacian.py000066400000000000000000000072531444262304300222310ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import add_self_loops, remove_self_loops, scatter
from torch_geometric.utils.num_nodes import maybe_num_nodes


def get_laplacian(
    edge_index: Tensor,
    edge_weight: OptTensor = None,
    normalization: Optional[str] = None,
    dtype: Optional[torch.dtype] = None,
    num_nodes: Optional[int] = None,
) -> Tuple[Tensor, OptTensor]:
    r""" Computes the graph Laplacian of the graph given by :obj:`edge_index`
    and optional :obj:`edge_weight`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_weight (Tensor, optional): One-dimensional edge weights.
            (default: :obj:`None`)
        normalization (str, optional): The normalization scheme for the graph
            Laplacian (default: :obj:`None`):

            1. :obj:`None`: No normalization
            :math:`\mathbf{L} = \mathbf{D} - \mathbf{A}`

            2. :obj:`"sym"`: Symmetric normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1/2} \mathbf{A}
            \mathbf{D}^{-1/2}`

            3. :obj:`"rw"`: Random-walk normalization
            :math:`\mathbf{L} = \mathbf{I} - \mathbf{D}^{-1} \mathbf{A}`
        dtype (torch.dtype, optional): The desired data type of returned tensor
            in case :obj:`edge_weight=None`. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1, 2],
        ...                            [1, 0, 2, 1]])
        >>> edge_weight = torch.tensor([1., 2., 2., 4.])

        >>> # No normalization
        >>> lap = get_laplacian(edge_index, edge_weight)

        >>> # Symmetric normalization
        >>> lap_sym = get_laplacian(edge_index, edge_weight,
                                    normalization='sym')

        >>> # Random-walk normalization
        >>> lap_rw = get_laplacian(edge_index, edge_weight, normalization='rw')
    """

    if normalization is not None:
        assert normalization in ['sym', 'rw']  # 'Invalid normalization'

    edge_index, edge_weight = remove_self_loops(edge_index, edge_weight)

    if edge_weight is None:
        edge_weight = torch.ones(edge_index.size(1), dtype=dtype,
                                 device=edge_index.device)

    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    row, col = edge_index[0], edge_index[1]
    deg = scatter(edge_weight, row, 0, dim_size=num_nodes, reduce='sum')

    if normalization is None:
        # L = D - A.
        edge_index, _ = add_self_loops(edge_index, num_nodes=num_nodes)
        edge_weight = torch.cat([-edge_weight, deg], dim=0)
    elif normalization == 'sym':
        # Compute A_norm = -D^{-1/2} A D^{-1/2}.
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
        edge_weight = deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]

        # L = I - A_norm.
        edge_index, tmp = add_self_loops(edge_index, -edge_weight,
                                         fill_value=1., num_nodes=num_nodes)
        assert tmp is not None
        edge_weight = tmp
    else:
        # Compute A_norm = -D^{-1} A.
        deg_inv = 1.0 / deg
        deg_inv.masked_fill_(deg_inv == float('inf'), 0)
        edge_weight = deg_inv[row] * edge_weight

        # L = I - A_norm.
        edge_index, tmp = add_self_loops(edge_index, -edge_weight,
                                         fill_value=1., num_nodes=num_nodes)
        assert tmp is not None
        edge_weight = tmp

    return edge_index, edge_weight
pytorch/torch_geometric/utils/get_mesh_laplacian.py000066400000000000000000000105071444262304300232410ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.utils import add_self_loops, scatter, to_undirected


def get_mesh_laplacian(
    pos: Tensor,
    face: Tensor,
    normalization: Optional[str] = None,
) -> Tuple[Tensor, Tensor]:
    r"""Computes the mesh Laplacian of a mesh given by :obj:`pos` and
    :obj:`face`. Computation is based on the cotangent matrix defined as

    .. math::
        \mathbf{C}_{ij} = \begin{cases}
            \frac{\cot \angle_{ikj}~+\cot \angle_{ilj}}{2} &
            \text{if } i, j \text{ is an edge} \\
            -\sum_{j \in N(i)}{C_{ij}} &
            \text{if } i \text{ is in the diagonal} \\
            0 & \text{otherwise}
      \end{cases}

    Normalization depends on the mass matrix defined as

    .. math::
        \mathbf{M}_{ij} = \begin{cases}
            a(i) & \text{if } i \text{ is in the diagonal} \\
            0 & \text{otherwise}
      \end{cases}

    where :math:`a(i)` is obtained by joining the barycenters of the
    triangles around vertex :math:`i`.

    Args:
        pos (Tensor): The node positions.
        face (LongTensor): The face indices.
        normalization (str, optional): The normalization scheme for the mesh
            Laplacian (default: :obj:`None`):

            1. :obj:`None`: No normalization
            :math:`\mathbf{L} = \mathbf{C}`

            2. :obj:`"sym"`: Symmetric normalization
            :math:`\mathbf{L} = \mathbf{M}^{-1/2} \mathbf{C}\mathbf{M}^{-1/2}`

            3. :obj:`"rw"`: Row-wise normalization
            :math:`\mathbf{L} = \mathbf{M}^{-1} \mathbf{C}`
    """
    assert pos.size(1) == 3 and face.size(0) == 3

    num_nodes = pos.shape[0]

    def get_cots(left, centre, right):
        left_pos, central_pos, right_pos = pos[left], pos[centre], pos[right]
        left_vec = left_pos - central_pos
        right_vec = right_pos - central_pos
        dot = torch.einsum('ij, ij -> i', left_vec, right_vec)
        cross = torch.norm(torch.cross(left_vec, right_vec, dim=1), dim=1)
        cot = dot / cross  # cot = cos / sin
        return cot / 2.0  # by definition

    # For each triangle face, get all three cotangents:
    cot_021 = get_cots(face[0], face[2], face[1])
    cot_102 = get_cots(face[1], face[0], face[2])
    cot_012 = get_cots(face[0], face[1], face[2])
    cot_weight = torch.cat([cot_021, cot_102, cot_012])

    # Face to edge:
    cot_index = torch.cat([face[:2], face[1:], face[::2]], dim=1)
    cot_index, cot_weight = to_undirected(cot_index, cot_weight)

    # Compute the diagonal part:
    cot_deg = scatter(cot_weight, cot_index[0], 0, num_nodes, reduce='sum')
    edge_index, _ = add_self_loops(cot_index, num_nodes=num_nodes)
    edge_weight = torch.cat([cot_weight, -cot_deg], dim=0)

    if normalization is not None:

        def get_areas(left, centre, right):
            central_pos = pos[centre]
            left_vec = pos[left] - central_pos
            right_vec = pos[right] - central_pos
            cross = torch.norm(torch.cross(left_vec, right_vec, dim=1), dim=1)
            area = cross / 6.0  # one-third of a triangle's area is cross / 6.0
            return area / 2.0  # since each corresponding area is counted twice

        # Like before, but here we only need the diagonal (the mass matrix):
        area_021 = get_areas(face[0], face[2], face[1])
        area_102 = get_areas(face[1], face[0], face[2])
        area_012 = get_areas(face[0], face[1], face[2])
        area_weight = torch.cat([area_021, area_102, area_012])
        area_index = torch.cat([face[:2], face[1:], face[::2]], dim=1)
        area_index, area_weight = to_undirected(area_index, area_weight)
        area_deg = scatter(area_weight, area_index[0], 0, num_nodes, 'sum')

        if normalization == 'sym':
            area_deg_inv_sqrt = area_deg.pow_(-0.5)
            area_deg_inv_sqrt[area_deg_inv_sqrt == float('inf')] = 0.0
            edge_weight = (area_deg_inv_sqrt[edge_index[0]] * edge_weight *
                           area_deg_inv_sqrt[edge_index[1]])
        elif normalization == 'rw':
            area_deg_inv = 1.0 / area_deg
            area_deg_inv[area_deg_inv == float('inf')] = 0.0
            edge_weight = area_deg_inv[edge_index[0]] * edge_weight

    return edge_index, edge_weight
pytorch/torch_geometric/utils/grid.py000066400000000000000000000047501444262304300203720ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.utils.coalesce import coalesce


def grid(
    height: int,
    width: int,
    dtype: Optional[torch.dtype] = None,
    device: Optional[torch.device] = None,
) -> Tuple[Tensor, Tensor]:
    r"""Returns the edge indices of a two-dimensional grid graph with height
    :attr:`height` and width :attr:`width` and its node positions.

    Args:
        height (int): The height of the grid.
        width (int): The width of the grid.
        dtype (:obj:`torch.dtype`, optional): The desired data type of the
            returned position tensor.
        dtype (:obj:`torch.device`, optional): The desired device of the
            returned tensors.

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Example:

        >>> (row, col), pos = grid(height=2, width=2)
        >>> row
        tensor([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3])
        >>> col
        tensor([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3])
        >>> pos
        tensor([[0., 1.],
                [1., 1.],
                [0., 0.],
                [1., 0.]])
    """
    edge_index = grid_index(height, width, device)
    pos = grid_pos(height, width, dtype, device)
    return edge_index, pos


def grid_index(
    height: int,
    width: int,
    device: Optional[torch.device] = None,
) -> Tensor:

    w = width
    kernel = [-w - 1, -1, w - 1, -w, 0, w, -w + 1, 1, w + 1]
    kernel = torch.tensor(kernel, device=device)

    row = torch.arange(height * width, dtype=torch.long, device=device)
    row = row.view(-1, 1).repeat(1, kernel.size(0))
    col = row + kernel.view(1, -1)
    row, col = row.view(height, -1), col.view(height, -1)
    index = torch.arange(3, row.size(1) - 3, dtype=torch.long, device=device)
    row, col = row[:, index].view(-1), col[:, index].view(-1)

    mask = (col >= 0) & (col < height * width)
    row, col = row[mask], col[mask]

    edge_index = torch.stack([row, col], dim=0)
    edge_index = coalesce(edge_index, num_nodes=height * width)
    return edge_index


def grid_pos(
    height: int,
    width: int,
    dtype: Optional[torch.dtype] = None,
    device: Optional[torch.device] = None,
) -> Tensor:

    dtype = torch.float if dtype is None else dtype
    x = torch.arange(width, dtype=dtype, device=device)
    y = (height - 1) - torch.arange(height, dtype=dtype, device=device)

    x = x.repeat(height)
    y = y.unsqueeze(-1).repeat(1, width).view(-1)

    return torch.stack([x, y], dim=-1)
pytorch/torch_geometric/utils/hetero.py000066400000000000000000000231061444262304300207270ustar00rootroot00000000000000from typing import Dict, List, Optional, Set, Tuple

import torch
from torch import Tensor
from torch.nn import ParameterDict

from torch_geometric.typing import Adj, EdgeType, NodeType, SparseTensor
from torch_geometric.utils import is_sparse, to_edge_index
from torch_geometric.utils.num_nodes import maybe_num_nodes_dict


def learn_sklearn_heuristic():
    import os
    import time

    from torch_geometric.nn.dense import HeteroLinear, Linear
    os.environ['NVIDIA_TF32_OVERRIDE'] = '0'
    fused_times = {}
    loop_times = {}
    try:
        for num_nodes_per_type in [10**2, 10**3, 10**4, 10**5]:
            for out_feats in [2, 4, 8, 16, 32, 64, 128, 256]:
                for n_feats in [4, 8, 16, 32, 64, 128, 256, 512]:
                    for num_types in [4, 8, 16, 32, 64, 128, 256, 512]:
                        try:
                            if n_feats < out_feats:
                                continue
                            print("benchmarking", num_types, "types w/",
                                  num_nodes_per_type, "nodes per type and",
                                  n_feats, "input features and", out_feats,
                                  "outuput feats")
                            x_dict = {
                                'v' + str(i): torch.randn(
                                    (num_nodes_per_type, n_feats)).cuda()
                                for i in range(num_types)
                            }
                            x = torch.cat(list(x_dict.values()), dim=0)
                            node_type = torch.cat([
                                (j * torch.ones(x_j.shape[0])).long()
                                for j, x_j in enumerate(x_dict.values())
                            ]).cuda()
                            lin = Linear(n_feats, out_feats).cuda()
                            heterolin = HeteroLinear(n_feats, out_feats,
                                                     len(list(x_dict.keys())),
                                                     True).cuda()
                            for i in range(60):
                                if i == 10:
                                    since = time.time()
                                heterolin(x=x, type_vec=node_type)
                            key = (num_types, num_nodes_per_type, n_feats,
                                   out_feats)
                            fused_times[key] = ((time.time() - since) / 50.0)
                            print("Avg time for fuse based=", fused_times[key])
                            for i in range(60):
                                if i == 10:
                                    since = time.time()
                                o = x.new_empty(x.size(0), out_feats)
                                for j in range(num_types):
                                    mask = j == node_type
                                    o[mask] = lin(x[mask])
                            loop_times[key] = ((time.time() - since) / 50.0)
                            print("Avg time for for-loop=", loop_times[key])
                        except:  # noqa
                            continue
    except:  # noqa
        pass
    import numpy as np
    X = np.zeros((len(loop_times), 4))
    y = np.zeros(len(loop_times))
    for i, key in enumerate(loop_times.keys()):
        X[i, :] = key
        loop_time, fused_time = loop_times[key], fused_times[key]
        y[i] = int(fused_time <= loop_time)
    from sklearn.pipeline import make_pipeline
    from sklearn.preprocessing import StandardScaler
    from sklearn.svm import LinearSVC
    scaler = StandardScaler()
    svm = LinearSVC()
    clf = make_pipeline(scaler, svm)
    clf.fit(X, y)

    print("scaler mean=", scaler.mean_)
    print("scaler scale=", scaler.scale_)
    print("svm weights=", svm.coef_)
    print("svm bias=", svm.intercept_)
    # results on A100:
    # scaler mean=
    #     [  125.11603189 12133.21523472   163.81222321    32.43755536]
    # scaler scale=
    #     [  163.34480422 27572.94543809   177.6426489     56.82103934]
    # svm weights=
    #     [[ 2.43877659e+00  1.67583047e+00 -5.20527282e-04  3.43925501e-01]]
    # svm bias=
    #     [1.20236999]


def segmatmul_heuristic(inputs: Tensor, type_ptr, weight: Tensor):
    num_types = len(type_ptr) - 1
    max_num_nodes_per_types = (type_ptr[1:] - type_ptr[:-1]).max()
    in_feat = inputs.size(1)
    out_feat = weight.size(-1)
    # this heuristic was learned with learn_sklearn_heuristic on an A100
    x = torch.tensor([
        int(num_types),
        int(max_num_nodes_per_types),
        int(in_feat),
        int(out_feat)
    ])
    scale_mean = torch.tensor(
        [125.11603189, 12133.21523472, 163.81222321, 32.43755536])
    scale_scale = torch.tensor(
        [163.34480422, 27572.94543809, 177.6426489, 56.82103934])
    svm_weights = torch.tensor(
        [2.43877659e+00, 1.67583047e+00, -5.20527282e-04, 3.43925501e-01])
    bias = 1.20236999
    x = (x - scale_mean) / scale_scale
    return int(x.dot(svm_weights) >= bias)


def group_hetero_graph(edge_index_dict, num_nodes_dict=None):
    num_nodes_dict = maybe_num_nodes_dict(edge_index_dict, num_nodes_dict)

    tmp = list(edge_index_dict.values())[0]

    key2int = {}

    cumsum, offset = 0, {}  # Helper data.
    node_types, local_node_indices = [], []
    local2global = {}
    for i, (key, N) in enumerate(num_nodes_dict.items()):
        key2int[key] = i
        node_types.append(tmp.new_full((N, ), i))
        local_node_indices.append(torch.arange(N, device=tmp.device))
        offset[key] = cumsum
        local2global[key] = local_node_indices[-1] + cumsum
        local2global[i] = local2global[key]
        cumsum += N

    node_type = torch.cat(node_types, dim=0)
    local_node_idx = torch.cat(local_node_indices, dim=0)

    edge_indices, edge_types = [], []
    for i, (keys, edge_index) in enumerate(edge_index_dict.items()):
        key2int[keys] = i
        inc = torch.tensor([offset[keys[0]], offset[keys[-1]]]).view(2, 1)
        edge_indices.append(edge_index + inc.to(tmp.device))
        edge_types.append(tmp.new_full((edge_index.size(1), ), i))

    edge_index = torch.cat(edge_indices, dim=-1)
    edge_type = torch.cat(edge_types, dim=0)

    return (edge_index, edge_type, node_type, local_node_idx, local2global,
            key2int)


def get_unused_node_types(node_types: List[NodeType],
                          edge_types: List[EdgeType]) -> Set[NodeType]:
    dst_node_types = set(edge_type[-1] for edge_type in edge_types)
    return set(node_types) - set(dst_node_types)


def check_add_self_loops(module: torch.nn.Module, edge_types: List[EdgeType]):
    is_bipartite = any([key[0] != key[-1] for key in edge_types])
    if is_bipartite and getattr(module, 'add_self_loops', False):
        raise ValueError(
            f"'add_self_loops' attribute set to 'True' on module '{module}' "
            f"for use with edge type(s) '{edge_types}'. This will lead to "
            f"incorrect message passing results.")


def construct_bipartite_edge_index(
    edge_index_dict: Dict[EdgeType, Adj],
    src_offset_dict: Dict[EdgeType, int],
    dst_offset_dict: Dict[NodeType, int],
    edge_attr_dict: Optional[Dict[EdgeType, Tensor]] = None,
) -> Tuple[Adj, Optional[Tensor]]:
    """Constructs a tensor of edge indices by concatenating edge indices
    for each edge type. The edge indices are increased by the offset of the
    source and destination nodes.

    Args:
        edge_index_dict (Dict[Tuple[str, str, str], torch.Tensor]): A
            dictionary holding graph connectivity information for each
            individual edge type, either as a :class:`torch.Tensor` of
            shape :obj:`[2, num_edges]` or a
            :class:`torch_sparse.SparseTensor`.
        src_offset_dict (Dict[Tuple[str, str, str], int]): A dictionary of
            offsets to apply to the source node type for each edge type.
        dst_offset_dict (Dict[str, int]): A dictionary of offsets to apply for
            destination node types.
        edge_attr_dict (Dict[Tuple[str, str, str], torch.Tensor]): A
            dictionary holding edge features for each individual edge type.
            (default: :obj:`None`)
    """
    is_sparse_tensor = False
    edge_indices: List[Tensor] = []
    edge_attrs: List[Tensor] = []
    for edge_type, src_offset in src_offset_dict.items():
        edge_index = edge_index_dict[edge_type]
        dst_offset = dst_offset_dict[edge_type[-1]]

        # TODO Add support for SparseTensor w/o converting.
        is_sparse_tensor = isinstance(edge_index, SparseTensor)
        if is_sparse(edge_index):
            edge_index, _ = to_edge_index(edge_index)
            edge_index = edge_index.flip([0])
        else:
            edge_index = edge_index.clone()

        edge_index[0] += src_offset
        edge_index[1] += dst_offset
        edge_indices.append(edge_index)

        if edge_attr_dict is not None:
            if isinstance(edge_attr_dict, ParameterDict):
                edge_attr = edge_attr_dict['__'.join(edge_type)]
            else:
                edge_attr = edge_attr_dict[edge_type]
            if edge_attr.size(0) != edge_index.size(1):
                edge_attr = edge_attr.expand(edge_index.size(1), -1)
            edge_attrs.append(edge_attr)

    edge_index = torch.cat(edge_indices, dim=1)

    edge_attr: Optional[Tensor] = None
    if edge_attr_dict is not None:
        edge_attr = torch.cat(edge_attrs, dim=0)

    if is_sparse_tensor:
        # TODO Add support for `SparseTensor.sparse_sizes()`.
        edge_index = SparseTensor(
            row=edge_index[1],
            col=edge_index[0],
            value=edge_attr,
        )

    return edge_index, edge_attr
pytorch/torch_geometric/utils/homophily.py000066400000000000000000000113221444262304300214460ustar00rootroot00000000000000from typing import Union

import torch
from torch import Tensor

from torch_geometric.typing import Adj, OptTensor, SparseTensor
from torch_geometric.utils import degree, scatter


def homophily(edge_index: Adj, y: Tensor, batch: OptTensor = None,
              method: str = 'edge') -> Union[float, Tensor]:
    r"""The homophily of a graph characterizes how likely nodes with the same
    label are near each other in a graph.
    There are many measures of homophily that fits this definition.
    In particular:

    - In the `"Beyond Homophily in Graph Neural Networks: Current Limitations
      and Effective Designs" `_ paper, the
      homophily is the fraction of edges in a graph which connects nodes
      that have the same class label:

      .. math::
        \frac{| \{ (v,w) : (v,w) \in \mathcal{E} \wedge y_v = y_w \} | }
        {|\mathcal{E}|}

      That measure is called the *edge homophily ratio*.

    - In the `"Geom-GCN: Geometric Graph Convolutional Networks"
      `_ paper, edge homophily is normalized
      across neighborhoods:

      .. math::
        \frac{1}{|\mathcal{V}|} \sum_{v \in \mathcal{V}} \frac{ | \{ (w,v) : w
        \in \mathcal{N}(v) \wedge y_v = y_w \} |  } { |\mathcal{N}(v)| }

      That measure is called the *node homophily ratio*.

    - In the `"Large-Scale Learning on Non-Homophilous Graphs: New Benchmarks
      and Strong Simple Methods" `_ paper,
      edge homophily is modified to be insensitive to the number of classes
      and size of each class:

      .. math::
        \frac{1}{C-1} \sum_{k=1}^{C} \max \left(0, h_k - \frac{|\mathcal{C}_k|}
        {|\mathcal{V}|} \right),

      where :math:`C` denotes the number of classes, :math:`|\mathcal{C}_k|`
      denotes the number of nodes of class :math:`k`, and :math:`h_k` denotes
      the edge homophily ratio of nodes of class :math:`k`.

      Thus, that measure is called the *class insensitive edge homophily
      ratio*.

    Args:
        edge_index (Tensor or SparseTensor): The graph connectivity.
        y (Tensor): The labels.
        batch (LongTensor, optional): Batch vector\
            :math:`\mathbf{b} \in {\{ 0, \ldots,B-1\}}^N`, which assigns
            each node to a specific example. (default: :obj:`None`)
        method (str, optional): The method used to calculate the homophily,
            either :obj:`"edge"` (first formula), :obj:`"node"` (second
            formula) or :obj:`"edge_insensitive"` (third formula).
            (default: :obj:`"edge"`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 2, 3],
        ...                            [1, 2, 0, 4]])
        >>> y = torch.tensor([0, 0, 0, 0, 1])
        >>> # Edge homophily ratio
        >>> homophily(edge_index, y, method='edge')
        0.75

        >>> # Node homophily ratio
        >>> homophily(edge_index, y, method='node')
        0.6000000238418579

        >>> # Class insensitive edge homophily ratio
        >>> homophily(edge_index, y, method='edge_insensitive')
        0.19999998807907104
    """
    assert method in {'edge', 'node', 'edge_insensitive'}
    y = y.squeeze(-1) if y.dim() > 1 else y

    if isinstance(edge_index, SparseTensor):
        row, col, _ = edge_index.coo()
    else:
        row, col = edge_index

    if method == 'edge':
        out = torch.zeros(row.size(0), device=row.device)
        out[y[row] == y[col]] = 1.
        if batch is None:
            return float(out.mean())
        else:
            dim_size = int(batch.max()) + 1
            return scatter(out, batch[col], 0, dim_size, reduce='mean')

    elif method == 'node':
        out = torch.zeros(row.size(0), device=row.device)
        out[y[row] == y[col]] = 1.
        out = scatter(out, col, 0, dim_size=y.size(0), reduce='mean')
        if batch is None:
            return float(out.mean())
        else:
            return scatter(out, batch, dim=0, reduce='mean')

    elif method == 'edge_insensitive':
        assert y.dim() == 1
        num_classes = int(y.max()) + 1
        assert num_classes >= 2
        batch = torch.zeros_like(y) if batch is None else batch
        num_nodes = degree(batch, dtype=torch.int64)
        num_graphs = num_nodes.numel()
        batch = num_classes * batch + y

        h = homophily(edge_index, y, batch, method='edge')
        h = h.view(num_graphs, num_classes)

        counts = batch.bincount(minlength=num_classes * num_graphs)
        counts = counts.view(num_graphs, num_classes)
        proportions = counts / num_nodes.view(-1, 1)

        out = (h - proportions).clamp_(min=0).sum(dim=-1)
        out /= num_classes - 1
        return out if out.numel() > 1 else float(out)

    else:
        raise NotImplementedError
pytorch/torch_geometric/utils/isolated.py000066400000000000000000000067661444262304300212620ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.utils import remove_self_loops, segregate_self_loops
from torch_geometric.utils.num_nodes import maybe_num_nodes


def contains_isolated_nodes(
    edge_index: Tensor,
    num_nodes: Optional[int] = None,
) -> bool:
    r"""Returns :obj:`True` if the graph given by :attr:`edge_index` contains
    isolated nodes.

    Args:
        edge_index (LongTensor): The edge indices.
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: bool

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> contains_isolated_nodes(edge_index)
        False

        >>> contains_isolated_nodes(edge_index, num_nodes=3)
        True
    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)
    edge_index, _ = remove_self_loops(edge_index)
    return torch.unique(edge_index.view(-1)).numel() < num_nodes


def remove_isolated_nodes(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    num_nodes: Optional[int] = None,
) -> Tuple[Tensor, Optional[Tensor], Tensor]:
    r"""Removes the isolated nodes from the graph given by :attr:`edge_index`
    with optional edge attributes :attr:`edge_attr`.
    In addition, returns a mask of shape :obj:`[num_nodes]` to manually filter
    out isolated node features later on.
    Self-loops are preserved for non-isolated nodes.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: (LongTensor, Tensor, BoolTensor)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> edge_index, edge_attr, mask = remove_isolated_nodes(edge_index)
        >>> mask # node mask (2 nodes)
        tensor([True, True])

        >>> edge_index, edge_attr, mask = remove_isolated_nodes(edge_index,
        ...                                                     num_nodes=3)
        >>> mask # node mask (3 nodes)
        tensor([True, True, False])
    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    out = segregate_self_loops(edge_index, edge_attr)
    edge_index, edge_attr, loop_edge_index, loop_edge_attr = out

    mask = torch.zeros(num_nodes, dtype=torch.bool, device=edge_index.device)
    mask[edge_index.view(-1)] = 1

    assoc = torch.full((num_nodes, ), -1, dtype=torch.long, device=mask.device)
    assoc[mask] = torch.arange(mask.sum(), device=assoc.device)
    edge_index = assoc[edge_index]

    loop_mask = torch.zeros_like(mask)
    loop_mask[loop_edge_index[0]] = 1
    loop_mask = loop_mask & mask
    loop_assoc = torch.full_like(assoc, -1)
    loop_assoc[loop_edge_index[0]] = torch.arange(loop_edge_index.size(1),
                                                  device=loop_assoc.device)
    loop_idx = loop_assoc[loop_mask]
    loop_edge_index = assoc[loop_edge_index[:, loop_idx]]

    edge_index = torch.cat([edge_index, loop_edge_index], dim=1)

    if edge_attr is not None:
        assert loop_edge_attr is not None
        loop_edge_attr = loop_edge_attr[loop_idx]
        edge_attr = torch.cat([edge_attr, loop_edge_attr], dim=0)

    return edge_index, edge_attr, mask
pytorch/torch_geometric/utils/loop.py000066400000000000000000000367571444262304300204320ustar00rootroot00000000000000from typing import Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter
from torch_geometric.utils.num_nodes import maybe_num_nodes
from torch_geometric.utils.sparse import (
    is_torch_sparse_tensor,
    to_edge_index,
    to_torch_coo_tensor,
    to_torch_csr_tensor,
)


def contains_self_loops(edge_index: Tensor) -> bool:
    r"""Returns :obj:`True` if the graph given by :attr:`edge_index` contains
    self-loops.

    Args:
        edge_index (LongTensor): The edge indices.

    :rtype: bool

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> contains_self_loops(edge_index)
        True

        >>> edge_index = torch.tensor([[0, 1, 1],
        ...                            [1, 0, 2]])
        >>> contains_self_loops(edge_index)
        False
    """
    mask = edge_index[0] == edge_index[1]
    return mask.sum().item() > 0


def remove_self_loops(
    edge_index: Tensor,
    edge_attr: OptTensor = None,
) -> Tuple[Tensor, OptTensor]:
    r"""Removes every self-loop in the graph given by :attr:`edge_index`, so
    that :math:`(i,i) \not\in \mathcal{E}` for every :math:`i \in \mathcal{V}`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Example:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> edge_attr = [[1, 2], [3, 4], [5, 6]]
        >>> edge_attr = torch.tensor(edge_attr)
        >>> remove_self_loops(edge_index, edge_attr)
        (tensor([[0, 1],
                [1, 0]]),
        tensor([[1, 2],
                [3, 4]]))
    """
    size: Optional[Tuple[int, int]] = None
    layout: Optional[int] = None

    if is_torch_sparse_tensor(edge_index):
        assert edge_attr is None
        layout = edge_index.layout
        size = (edge_index.size(0), edge_index.size(1))
        edge_index, edge_attr = to_edge_index(edge_index)

    mask = edge_index[0] != edge_index[1]
    edge_index = edge_index[:, mask]

    if layout is not None:
        assert edge_attr is not None
        edge_attr = edge_attr[mask]
        if str(layout) == 'torch.sparse_coo':  # str(...) for TorchScript :(
            return to_torch_coo_tensor(edge_index, edge_attr, size, True), None
        elif str(layout) == 'torch.sparse_csr':
            return to_torch_csr_tensor(edge_index, edge_attr, size, True), None
        raise ValueError(f"Unexpected sparse tensor layout (got '{layout}')")

    if edge_attr is None:
        return edge_index, None
    else:
        return edge_index, edge_attr[mask]


def segregate_self_loops(
    edge_index: Tensor, edge_attr: OptTensor = None
) -> Tuple[Tensor, OptTensor, Tensor, OptTensor]:
    r"""Segregates self-loops from the graph.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`, :class:`LongTensor`,
        :class:`Tensor`)

    Example:

        >>> edge_index = torch.tensor([[0, 0, 1],
        ...                            [0, 1, 0]])
        >>> (edge_index, edge_attr,
        ...  loop_edge_index,
        ...  loop_edge_attr) = segregate_self_loops(edge_index)
        >>>  loop_edge_index
        tensor([[0],
                [0]])
    """

    mask = edge_index[0] != edge_index[1]
    inv_mask = ~mask

    loop_edge_index = edge_index[:, inv_mask]
    loop_edge_attr = None if edge_attr is None else edge_attr[inv_mask]
    edge_index = edge_index[:, mask]
    edge_attr = None if edge_attr is None else edge_attr[mask]

    return edge_index, edge_attr, loop_edge_index, loop_edge_attr


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, Optional[float], Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, Optional[float], Optional[Tuple[int, int]]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, OptTensor, Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, OptTensor, Optional[Tuple[int, int]]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, Optional[str], Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_self_loops(edge_index, edge_attr, fill_value, num_nodes):
    # type: (Tensor, OptTensor, Optional[str], Optional[Tuple[int, int]]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


def add_self_loops(
    edge_index: Tensor,
    edge_attr: OptTensor = None,
    fill_value: Optional[Union[float, Tensor, str]] = None,
    num_nodes: Optional[Union[int, Tuple[int, int]]] = None,
) -> Tuple[Tensor, OptTensor]:
    r"""Adds a self-loop :math:`(i,i) \in \mathcal{E}` to every node
    :math:`i \in \mathcal{V}` in the graph given by :attr:`edge_index`.
    In case the graph is weighted or has multi-dimensional edge features
    (:obj:`edge_attr != None`), edge features of self-loops will be added
    according to :obj:`fill_value`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional edge
            features. (default: :obj:`None`)
        fill_value (float or Tensor or str, optional): The way to generate
            edge features of self-loops (in case :obj:`edge_attr != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`1.`)
        num_nodes (int or Tuple[int, int], optional): The number of nodes,
            *i.e.* :obj:`max_val + 1` of :attr:`edge_index`.
            If given as a tuple, then :obj:`edge_index` is interpreted as a
            bipartite graph with shape :obj:`(num_src_nodes, num_dst_nodes)`.
            (default: :obj:`None`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> edge_weight = torch.tensor([0.5, 0.5, 0.5])
        >>> add_self_loops(edge_index)
        (tensor([[0, 1, 0, 0, 1],
                [1, 0, 0, 0, 1]]),
        None)

        >>> add_self_loops(edge_index, edge_weight)
        (tensor([[0, 1, 0, 0, 1],
                [1, 0, 0, 0, 1]]),
        tensor([0.5000, 0.5000, 0.5000, 1.0000, 1.0000]))

        >>> # edge features of self-loops are filled by constant `2.0`
        >>> add_self_loops(edge_index, edge_weight,
        ...                fill_value=2.)
        (tensor([[0, 1, 0, 0, 1],
                [1, 0, 0, 0, 1]]),
        tensor([0.5000, 0.5000, 0.5000, 2.0000, 2.0000]))

        >>> # Use 'add' operation to merge edge features for self-loops
        >>> add_self_loops(edge_index, edge_weight,
        ...                fill_value='add')
        (tensor([[0, 1, 0, 0, 1],
                [1, 0, 0, 0, 1]]),
        tensor([0.5000, 0.5000, 0.5000, 1.0000, 0.5000]))
    """
    layout: Optional[int] = None
    is_sparse = is_torch_sparse_tensor(edge_index)

    if is_sparse:
        assert edge_attr is None
        layout = edge_index.layout
        size = (edge_index.size(0), edge_index.size(1))
        edge_index, edge_attr = to_edge_index(edge_index)
    elif isinstance(num_nodes, (tuple, list)):
        size = (num_nodes[0], num_nodes[1])
    else:
        N = maybe_num_nodes(edge_index, num_nodes)
        size = (N, N)

    N = min(size)

    loop_index = torch.arange(0, N, dtype=torch.long, device=edge_index.device)
    loop_index = loop_index.unsqueeze(0).repeat(2, 1)

    if edge_attr is not None:
        if fill_value is None:
            loop_attr = edge_attr.new_full((N, ) + edge_attr.size()[1:], 1.)

        elif isinstance(fill_value, (int, float)):
            loop_attr = edge_attr.new_full((N, ) + edge_attr.size()[1:],
                                           fill_value)
        elif isinstance(fill_value, Tensor):
            loop_attr = fill_value.to(edge_attr.device, edge_attr.dtype)
            if edge_attr.dim() != loop_attr.dim():
                loop_attr = loop_attr.unsqueeze(0)
            sizes = [N] + [1] * (loop_attr.dim() - 1)
            loop_attr = loop_attr.repeat(sizes)

        elif isinstance(fill_value, str):
            col = edge_index[0] if is_sparse else edge_index[1]
            loop_attr = scatter(edge_attr, col, 0, N, fill_value)
        else:
            raise AttributeError("No valid 'fill_value' provided")

        edge_attr = torch.cat([edge_attr, loop_attr], dim=0)

    edge_index = torch.cat([edge_index, loop_index], dim=1)
    if is_sparse:
        if str(layout) == 'torch.sparse_coo':  # str(...) for TorchScript :(
            return to_torch_coo_tensor(edge_index, edge_attr, size), None
        elif str(layout) == 'torch.sparse_csr':
            return to_torch_csr_tensor(edge_index, edge_attr, size), None
        raise ValueError(f"Unexpected sparse tensor layout (got '{layout}')")
    return edge_index, edge_attr


@torch.jit._overload
def add_remaining_self_loops(edge_index, edge_attr=None, fill_value=None,
                             num_nodes=None):
    # type: (Tensor, OptTensor, Optional[float], Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


# @torch.jit._overload
def add_remaining_self_loops(edge_index, edge_attr=None, fill_value=None,
                             num_nodes=None):
    # type: (Tensor, OptTensor, OptTensor, Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


@torch.jit._overload
def add_remaining_self_loops(edge_index, edge_attr=None, fill_value=None,
                             num_nodes=None):
    # type: (Tensor, OptTensor, Optional[str], Optional[int]) -> Tuple[Tensor, OptTensor]  # noqa
    pass


def add_remaining_self_loops(
    edge_index: Tensor,
    edge_attr: OptTensor = None,
    fill_value: Optional[Union[float, Tensor, str]] = None,
    num_nodes: Optional[int] = None,
) -> Tuple[Tensor, OptTensor]:
    r"""Adds remaining self-loop :math:`(i,i) \in \mathcal{E}` to every node
    :math:`i \in \mathcal{V}` in the graph given by :attr:`edge_index`.
    In case the graph is weighted or has multi-dimensional edge features
    (:obj:`edge_attr != None`), edge features of non-existing self-loops will
    be added according to :obj:`fill_value`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional edge
            features. (default: :obj:`None`)
        fill_value (float or Tensor or str, optional): The way to generate
            edge features of self-loops (in case :obj:`edge_attr != None`).
            If given as :obj:`float` or :class:`torch.Tensor`, edge features of
            self-loops will be directly given by :obj:`fill_value`.
            If given as :obj:`str`, edge features of self-loops are computed by
            aggregating all features of edges that point to the specific node,
            according to a reduce operation. (:obj:`"add"`, :obj:`"mean"`,
            :obj:`"min"`, :obj:`"max"`, :obj:`"mul"`). (default: :obj:`1.`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Example:

        >>> edge_index = torch.tensor([[0, 1],
        ...                            [1, 0]])
        >>> edge_weight = torch.tensor([0.5, 0.5])
        >>> add_remaining_self_loops(edge_index, edge_weight)
        (tensor([[0, 1, 0, 1],
                [1, 0, 0, 1]]),
        tensor([0.5000, 0.5000, 1.0000, 1.0000]))
    """
    N = maybe_num_nodes(edge_index, num_nodes)
    mask = edge_index[0] != edge_index[1]

    loop_index = torch.arange(0, N, dtype=torch.long, device=edge_index.device)
    loop_index = loop_index.unsqueeze(0).repeat(2, 1)

    if edge_attr is not None:
        if fill_value is None:
            loop_attr = edge_attr.new_full((N, ) + edge_attr.size()[1:], 1.)

        elif isinstance(fill_value, (int, float)):
            loop_attr = edge_attr.new_full((N, ) + edge_attr.size()[1:],
                                           fill_value)
        elif isinstance(fill_value, Tensor):
            loop_attr = fill_value.to(edge_attr.device, edge_attr.dtype)
            if edge_attr.dim() != loop_attr.dim():
                loop_attr = loop_attr.unsqueeze(0)
            sizes = [N] + [1] * (loop_attr.dim() - 1)
            loop_attr = loop_attr.repeat(*sizes)

        elif isinstance(fill_value, str):
            loop_attr = scatter(edge_attr, edge_index[1], dim=0, dim_size=N,
                                reduce=fill_value)
        else:
            raise AttributeError("No valid 'fill_value' provided")

        inv_mask = ~mask
        loop_attr[edge_index[0][inv_mask]] = edge_attr[inv_mask]

        edge_attr = torch.cat([edge_attr[mask], loop_attr], dim=0)

    edge_index = torch.cat([edge_index[:, mask], loop_index], dim=1)
    return edge_index, edge_attr


def get_self_loop_attr(edge_index: Tensor, edge_attr: OptTensor = None,
                       num_nodes: Optional[int] = None) -> Tensor:
    r"""Returns the edge features or weights of self-loops
    :math:`(i, i)` of every node :math:`i \in \mathcal{V}` in the
    graph given by :attr:`edge_index`. Edge features of missing self-loops not
    present in :attr:`edge_index` will be filled with zeros. If
    :attr:`edge_attr` is not given, it will be the vector of ones.

    .. note::
        This operation is analogous to getting the diagonal elements of the
        dense adjacency matrix.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional edge
            features. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: :class:`Tensor`

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                            [1, 0, 0]])
        >>> edge_weight = torch.tensor([0.2, 0.3, 0.5])
        >>> get_self_loop_attr(edge_index, edge_weight)
        tensor([0.5000, 0.0000])

        >>> get_self_loop_attr(edge_index, edge_weight, num_nodes=4)
        tensor([0.5000, 0.0000, 0.0000, 0.0000])
    """
    loop_mask = edge_index[0] == edge_index[1]
    loop_index = edge_index[0][loop_mask]

    if edge_attr is not None:
        loop_attr = edge_attr[loop_mask]
    else:  # A vector of ones:
        loop_attr = torch.ones_like(loop_index, dtype=torch.float)

    num_nodes = maybe_num_nodes(edge_index, num_nodes)
    full_loop_attr = loop_attr.new_zeros((num_nodes, ) + loop_attr.size()[1:])
    full_loop_attr[loop_index] = loop_attr

    return full_loop_attr
pytorch/torch_geometric/utils/map.py000066400000000000000000000121201444262304300202100ustar00rootroot00000000000000import warnings
from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.utils.dlpack import from_dlpack


def map_index(
    src: Tensor,
    index: Tensor,
    max_index: Optional[int] = None,
    inclusive: bool = False,
) -> Tuple[Tensor, Optional[Tensor]]:
    r"""Maps indices in :obj:`src` to the positional value of their
    corresponding occurence in :obj:`index`.
    Indices must be strictly positive.

    Args:
        src (torch.Tensor): The source tensor to map.
        index (torch.Tensor): The index tensor that denotes the new mapping.
        max_index (int, optional): The maximum index value.
            (default :obj:`None`)
        inclusive (bool, optional): If set to :obj:`True`, it is assumed that
            every entry in :obj:`src` has a valid entry in :obj:`index`.
            Can speed-up computation. (default: :obj:`False`)

    :rtype: (:class:`torch.Tensor`, :class:`torch.BoolTensor`)

    Examples:

        >>> src = torch.tensor([2, 0, 1, 0, 3])
        >>> index = torch.tensor([3, 2, 0, 1])

        >>> map_index(src, index)
        (tensor([1, 2, 3, 2, 0]), tensor([True, True, True, True, True]))

        >>> src = torch.tensor([2, 0, 1, 0, 3])
        >>> index = torch.tensor([3, 2, 0])

        >>> map_index(src, index)
        (tensor([1, 2, 2, 0]), tensor([True, True, False, True, True]))
    """
    if src.is_floating_point():
        raise ValueError(f"Expected 'src' to be an index (got '{src.dtype}')")
    if index.is_floating_point():
        raise ValueError(f"Expected 'index' to be an index (got "
                         f"'{index.dtype}')")
    if src.device != index.device:
        raise ValueError(f"Both 'src' and 'index' must be on the same device "
                         f"(got '{src.device}' and '{index.device}')")

    if max_index is None:
        max_index = max(src.max(), index.max())

    # If the `max_index` is in a reasonable range, we can accelerate this
    # operation by creating a helper vector to perform the mapping.
    # NOTE This will potentially consumes a large chunk of memory
    # (max_index=10 million => ~75MB), so we cap it at a reasonable size:
    THRESHOLD = 40_000_000 if src.is_cuda else 10_000_000
    if max_index <= THRESHOLD:
        if inclusive:
            assoc = src.new_empty(max_index + 1)
        else:
            assoc = src.new_full((max_index + 1, ), -1)
        assoc[index] = torch.arange(index.numel(), dtype=src.dtype,
                                    device=src.device)
        out = assoc[src]

        if inclusive:
            return out, None
        else:
            mask = out != -1
            return out[mask], mask

    WITH_CUDF = False
    if src.is_cuda:
        try:
            import cudf
            WITH_CUDF = True
        except ImportError:
            import pandas as pd
            warnings.warn("Using CPU-based processing within 'map_index' "
                          "which may cause slowdowns and device "
                          "synchronization. Consider installing 'cudf' to "
                          "accelerate computation")
    else:
        import pandas as pd

    if not WITH_CUDF:
        left_ser = pd.Series(src.cpu().numpy(), name='left_ser')
        right_ser = pd.Series(
            index=index.cpu().numpy(),
            data=pd.RangeIndex(0, index.size(0)),
            name='right_ser',
        )

        result = pd.merge(left_ser, right_ser, how='left', left_on='left_ser',
                          right_index=True)

        out = torch.from_numpy(result['right_ser'].values).to(index.device)

        if out.is_floating_point() and inclusive:
            raise ValueError("Found invalid entries in 'src' that do not have "
                             "a corresponding entry in 'index'. Set "
                             "`inclusive=False` to ignore these entries.")

        if out.is_floating_point():
            mask = torch.isnan(out).logical_not_()
            out = out[mask].to(index.dtype)
            return out, mask

        if inclusive:
            return out, None
        else:
            mask = out != -1
            return out[mask], mask

    else:
        left_ser = cudf.Series(src, name='left_ser')
        right_ser = cudf.Series(
            index=index,
            data=cudf.RangeIndex(0, index.size(0)),
            name='right_ser',
        )

        result = cudf.merge(left_ser, right_ser, how='left',
                            left_on='left_ser', right_index=True, sort=True)

        if inclusive:
            try:
                out = from_dlpack(result['right_ser'].to_dlpack())
            except ValueError:
                raise ValueError("Found invalid entries in 'src' that do not "
                                 "have a corresponding entry in 'index'. Set "
                                 "`inclusive=False` to ignore these entries.")
        else:
            out = from_dlpack(result['right_ser'].fillna(-1).to_dlpack())

        out = out[src.argsort().argsort()]  # Restore original order.

        if inclusive:
            return out, None
        else:
            mask = out != -1
            return out[mask], mask
pytorch/torch_geometric/utils/mask.py000066400000000000000000000035341444262304300203770ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor


def mask_select(src: Tensor, dim: int, mask: Tensor) -> Tensor:
    r"""Returns a new tensor which masks the :obj:`src` tensor along the
    dimension :obj:`dim` according to the boolean mask :obj:`mask`.

    Args:
        src (torch.Tensor): The input tensor.
        dim (int): The dimension in which to mask.
        mask (torch.BoolTensor): The 1-D tensor containing the binary mask to
            index with.
    """
    assert mask.dim() == 1
    assert src.size(dim) == mask.numel()
    dim = dim + src.dim() if dim < 0 else dim
    assert dim >= 0 and dim < src.dim()

    size = [1] * src.dim()
    size[dim] = mask.numel()

    out = src.masked_select(mask.view(size))

    size = list(src.size())
    size[dim] = -1

    return out.view(size)


def index_to_mask(index: Tensor, size: Optional[int] = None) -> Tensor:
    r"""Converts indices to a mask representation.

    Args:
        idx (Tensor): The indices.
        size (int, optional). The size of the mask. If set to :obj:`None`, a
            minimal sized output mask is returned.

    Example:

        >>> index = torch.tensor([1, 3, 5])
        >>> index_to_mask(index)
        tensor([False,  True, False,  True, False,  True])

        >>> index_to_mask(index, size=7)
        tensor([False,  True, False,  True, False,  True, False])
    """
    index = index.view(-1)
    size = int(index.max()) + 1 if size is None else size
    mask = index.new_zeros(size, dtype=torch.bool)
    mask[index] = True
    return mask


def mask_to_index(mask: Tensor) -> Tensor:
    r"""Converts a mask to an index representation.

    Args:
        mask (Tensor): The mask.

    Example:

        >>> mask = torch.tensor([False, True, False])
        >>> mask_to_index(mask)
        tensor([1])
    """
    return mask.nonzero(as_tuple=False).view(-1)
pytorch/torch_geometric/utils/mixin.py000066400000000000000000000011401444262304300205570ustar00rootroot00000000000000from typing import Iterator


class CastMixin:
    @classmethod
    def cast(cls, *args, **kwargs):  # TODO Can we apply this recursively?
        if len(args) == 1 and len(kwargs) == 0:
            elem = args[0]
            if elem is None:
                return None
            if isinstance(elem, CastMixin):
                return elem
            if isinstance(elem, tuple):
                return cls(*elem)
            if isinstance(elem, dict):
                return cls(**elem)
        return cls(*args, **kwargs)

    def __iter__(self) -> Iterator:
        return iter(self.__dict__.values())
pytorch/torch_geometric/utils/negative_sampling.py000066400000000000000000000344631444262304300231450ustar00rootroot00000000000000import random
from typing import Optional, Tuple, Union

import numpy as np
import torch
from torch import Tensor

from torch_geometric.utils import coalesce, degree, remove_self_loops
from torch_geometric.utils.num_nodes import maybe_num_nodes


def negative_sampling(edge_index: Tensor,
                      num_nodes: Optional[Union[int, Tuple[int, int]]] = None,
                      num_neg_samples: Optional[int] = None,
                      method: str = "sparse",
                      force_undirected: bool = False) -> Tensor:
    r"""Samples random negative edges of a graph given by :attr:`edge_index`.

    Args:
        edge_index (LongTensor): The edge indices.
        num_nodes (int or Tuple[int, int], optional): The number of nodes,
            *i.e.* :obj:`max_val + 1` of :attr:`edge_index`.
            If given as a tuple, then :obj:`edge_index` is interpreted as a
            bipartite graph with shape :obj:`(num_src_nodes, num_dst_nodes)`.
            (default: :obj:`None`)
        num_neg_samples (int, optional): The (approximate) number of negative
            samples to return.
            If set to :obj:`None`, will try to return a negative edge for every
            positive edge. (default: :obj:`None`)
        method (str, optional): The method to use for negative sampling,
            *i.e.* :obj:`"sparse"` or :obj:`"dense"`.
            This is a memory/runtime trade-off.
            :obj:`"sparse"` will work on any graph of any size, while
            :obj:`"dense"` can perform faster true-negative checks.
            (default: :obj:`"sparse"`)
        force_undirected (bool, optional): If set to :obj:`True`, sampled
            negative edges will be undirected. (default: :obj:`False`)

    :rtype: LongTensor

    Examples:

        >>> # Standard usage
        >>> edge_index = torch.as_tensor([[0, 0, 1, 2],
        ...                               [0, 1, 2, 3]])
        >>> negative_sampling(edge_index)
        tensor([[3, 0, 0, 3],
                [2, 3, 2, 1]])

        >>> # For bipartite graph
        >>> negative_sampling(edge_index, num_nodes=(3, 4))
        tensor([[0, 2, 2, 1],
                [2, 2, 1, 3]])
    """
    assert method in ['sparse', 'dense']

    size = num_nodes
    bipartite = isinstance(size, (tuple, list))
    size = maybe_num_nodes(edge_index) if size is None else size
    size = (size, size) if not bipartite else size
    force_undirected = False if bipartite else force_undirected

    idx, population = edge_index_to_vector(edge_index, size, bipartite,
                                           force_undirected)

    if idx.numel() >= population:
        return edge_index.new_empty((2, 0))

    if num_neg_samples is None:
        num_neg_samples = edge_index.size(1)
    if force_undirected:
        num_neg_samples = num_neg_samples // 2

    prob = 1. - idx.numel() / population  # Probability to sample a negative.
    sample_size = int(1.1 * num_neg_samples / prob)  # (Over)-sample size.

    neg_idx = None
    if method == 'dense':
        # The dense version creates a mask of shape `population` to check for
        # invalid samples.
        mask = idx.new_ones(population, dtype=torch.bool)
        mask[idx] = False
        for _ in range(3):  # Number of tries to sample negative indices.
            rnd = sample(population, sample_size, idx.device)
            rnd = rnd[mask[rnd]]  # Filter true negatives.
            neg_idx = rnd if neg_idx is None else torch.cat([neg_idx, rnd])
            if neg_idx.numel() >= num_neg_samples:
                neg_idx = neg_idx[:num_neg_samples]
                break
            mask[neg_idx] = False

    else:  # 'sparse'
        # The sparse version checks for invalid samples via `np.isin`.
        idx = idx.to('cpu')
        for _ in range(3):  # Number of tries to sample negative indices.
            rnd = sample(population, sample_size, device='cpu')
            mask = np.isin(rnd, idx)
            if neg_idx is not None:
                mask |= np.isin(rnd, neg_idx.to('cpu'))
            mask = torch.from_numpy(mask).to(torch.bool)
            rnd = rnd[~mask].to(edge_index.device)
            neg_idx = rnd if neg_idx is None else torch.cat([neg_idx, rnd])
            if neg_idx.numel() >= num_neg_samples:
                neg_idx = neg_idx[:num_neg_samples]
                break

    return vector_to_edge_index(neg_idx, size, bipartite, force_undirected)


def batched_negative_sampling(
    edge_index: Tensor,
    batch: Union[Tensor, Tuple[Tensor, Tensor]],
    num_neg_samples: Optional[int] = None,
    method: str = "sparse",
    force_undirected: bool = False,
) -> Tensor:
    r"""Samples random negative edges of multiple graphs given by
    :attr:`edge_index` and :attr:`batch`.

    Args:
        edge_index (LongTensor): The edge indices.
        batch (LongTensor or Tuple[LongTensor, LongTensor]): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example.
            If given as a tuple, then :obj:`edge_index` is interpreted as a
            bipartite graph connecting two different node types.
        num_neg_samples (int, optional): The number of negative samples to
            return. If set to :obj:`None`, will try to return a negative edge
            for every positive edge. (default: :obj:`None`)
        method (str, optional): The method to use for negative sampling,
            *i.e.* :obj:`"sparse"` or :obj:`"dense"`.
            This is a memory/runtime trade-off.
            :obj:`"sparse"` will work on any graph of any size, while
            :obj:`"dense"` can perform faster true-negative checks.
            (default: :obj:`"sparse"`)
        force_undirected (bool, optional): If set to :obj:`True`, sampled
            negative edges will be undirected. (default: :obj:`False`)

    :rtype: LongTensor

    Examples:

        >>> # Standard usage
        >>> edge_index = torch.as_tensor([[0, 0, 1, 2], [0, 1, 2, 3]])
        >>> edge_index = torch.cat([edge_index, edge_index + 4], dim=1)
        >>> edge_index
        tensor([[0, 0, 1, 2, 4, 4, 5, 6],
                [0, 1, 2, 3, 4, 5, 6, 7]])
        >>> batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])
        >>> batched_negative_sampling(edge_index, batch)
        tensor([[3, 1, 3, 2, 7, 7, 6, 5],
                [2, 0, 1, 1, 5, 6, 4, 4]])

        >>> # For bipartite graph
        >>> edge_index1 = torch.as_tensor([[0, 0, 1, 1], [0, 1, 2, 3]])
        >>> edge_index2 = edge_index1 + torch.tensor([[2], [4]])
        >>> edge_index3 = edge_index2 + torch.tensor([[2], [4]])
        >>> edge_index = torch.cat([edge_index1, edge_index2,
        ...                         edge_index3], dim=1)
        >>> edge_index
        tensor([[ 0,  0,  1,  1,  2,  2,  3,  3,  4,  4,  5,  5],
                [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11]])
        >>> src_batch = torch.tensor([0, 0, 1, 1, 2, 2])
        >>> dst_batch = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2])
        >>> batched_negative_sampling(edge_index,
        ...                           (src_batch, dst_batch))
        tensor([[ 0,  0,  1,  1,  2,  2,  3,  3,  4,  4,  5,  5],
                [ 2,  3,  0,  1,  6,  7,  4,  5, 10, 11,  8,  9]])
    """
    if isinstance(batch, Tensor):
        src_batch, dst_batch = batch, batch
    else:
        src_batch, dst_batch = batch[0], batch[1]

    split = degree(src_batch[edge_index[0]], dtype=torch.long).tolist()
    edge_indices = torch.split(edge_index, split, dim=1)

    num_src = degree(src_batch, dtype=torch.long)
    cum_src = torch.cat([src_batch.new_zeros(1), num_src.cumsum(0)[:-1]])

    if isinstance(batch, Tensor):
        num_nodes = num_src.tolist()
        cumsum = cum_src
    else:
        num_dst = degree(dst_batch, dtype=torch.long)
        cum_dst = torch.cat([dst_batch.new_zeros(1), num_dst.cumsum(0)[:-1]])

        num_nodes = torch.stack([num_src, num_dst], dim=1).tolist()
        cumsum = torch.stack([cum_src, cum_dst], dim=1).unsqueeze(-1)

    neg_edge_indices = []
    for i, edge_index in enumerate(edge_indices):
        edge_index = edge_index - cumsum[i]
        neg_edge_index = negative_sampling(edge_index, num_nodes[i],
                                           num_neg_samples, method,
                                           force_undirected)
        neg_edge_index += cumsum[i]
        neg_edge_indices.append(neg_edge_index)

    return torch.cat(neg_edge_indices, dim=1)


def structured_negative_sampling(edge_index, num_nodes: Optional[int] = None,
                                 contains_neg_self_loops: bool = True):
    r"""Samples a negative edge :obj:`(i,k)` for every positive edge
    :obj:`(i,j)` in the graph given by :attr:`edge_index`, and returns it as a
    tuple of the form :obj:`(i,j,k)`.

    Args:
        edge_index (LongTensor): The edge indices.
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        contains_neg_self_loops (bool, optional): If set to
            :obj:`False`, sampled negative edges will not contain self loops.
            (default: :obj:`True`)

    :rtype: (LongTensor, LongTensor, LongTensor)

    Example:

        >>> edge_index = torch.as_tensor([[0, 0, 1, 2],
        ...                               [0, 1, 2, 3]])
        >>> structured_negative_sampling(edge_index)
        (tensor([0, 0, 1, 2]), tensor([0, 1, 2, 3]), tensor([2, 3, 0, 2]))

    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    row, col = edge_index.cpu()
    pos_idx = row * num_nodes + col
    if not contains_neg_self_loops:
        loop_idx = torch.arange(num_nodes) * (num_nodes + 1)
        pos_idx = torch.cat([pos_idx, loop_idx], dim=0)

    rand = torch.randint(num_nodes, (row.size(0), ), dtype=torch.long)
    neg_idx = row * num_nodes + rand

    mask = torch.from_numpy(np.isin(neg_idx, pos_idx)).to(torch.bool)
    rest = mask.nonzero(as_tuple=False).view(-1)
    while rest.numel() > 0:  # pragma: no cover
        tmp = torch.randint(num_nodes, (rest.size(0), ), dtype=torch.long)
        rand[rest] = tmp
        neg_idx = row[rest] * num_nodes + tmp

        mask = torch.from_numpy(np.isin(neg_idx, pos_idx)).to(torch.bool)
        rest = rest[mask]

    return edge_index[0], edge_index[1], rand.to(edge_index.device)


def structured_negative_sampling_feasible(
    edge_index: Tensor,
    num_nodes: Optional[int] = None,
    contains_neg_self_loops: bool = True,
) -> bool:
    r"""Returns :obj:`True` if
    :meth:`~torch_geometric.utils.structured_negative_sampling` is feasible
    on the graph given by :obj:`edge_index`.
    :meth:`~torch_geometric.utils.structured_negative_sampling` is infeasible
    if atleast one node is connected to all other nodes.

    Args:
        edge_index (LongTensor): The edge indices.
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        contains_neg_self_loops (bool, optional): If set to
            :obj:`False`, sampled negative edges will not contain self loops.
            (default: :obj:`True`)

    :rtype: bool

    Examples:

        >>> edge_index = torch.LongTensor([[0, 0, 1, 1, 2, 2, 2],
        ...                                [1, 2, 0, 2, 0, 1, 1]])
        >>> structured_negative_sampling_feasible(edge_index, 3, False)
        False

        >>> structured_negative_sampling_feasible(edge_index, 3, True)
        True
    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)
    max_num_neighbors = num_nodes

    edge_index = coalesce(edge_index, num_nodes=num_nodes)

    if not contains_neg_self_loops:
        edge_index, _ = remove_self_loops(edge_index)
        max_num_neighbors -= 1  # Reduce number of valid neighbors

    deg = degree(edge_index[0], num_nodes)
    # True if there exists no node that is connected to all other nodes.
    return bool(torch.all(deg < max_num_neighbors))


###############################################################################


def sample(population: int, k: int, device=None) -> Tensor:
    if population <= k:
        return torch.arange(population, device=device)
    else:
        return torch.tensor(random.sample(range(population), k), device=device)


def edge_index_to_vector(
    edge_index: Tensor,
    size: Tuple[int, int],
    bipartite: bool,
    force_undirected: bool = False,
) -> Tuple[Tensor, int]:

    row, col = edge_index

    if bipartite:  # No need to account for self-loops.
        idx = (row * size[1]).add_(col)
        population = size[0] * size[1]
        return idx, population

    elif force_undirected:
        assert size[0] == size[1]
        num_nodes = size[0]

        # We only operate on the upper triangular matrix:
        mask = row < col
        row, col = row[mask], col[mask]
        offset = torch.arange(1, num_nodes, device=row.device).cumsum(0)[row]
        idx = row.mul_(num_nodes).add_(col).sub_(offset)
        population = (num_nodes * (num_nodes + 1)) // 2 - num_nodes
        return idx, population

    else:
        assert size[0] == size[1]
        num_nodes = size[0]

        # We remove self-loops as we do not want to take them into account
        # when sampling negative values.
        mask = row != col
        row, col = row[mask], col[mask]
        col[row < col] -= 1
        idx = row.mul_(num_nodes - 1).add_(col)
        population = num_nodes * num_nodes - num_nodes
        return idx, population


def vector_to_edge_index(idx: Tensor, size: Tuple[int, int], bipartite: bool,
                         force_undirected: bool = False) -> Tensor:

    if bipartite:  # No need to account for self-loops.
        row = idx.div(size[1], rounding_mode='floor')
        col = idx % size[1]
        return torch.stack([row, col], dim=0)

    elif force_undirected:
        assert size[0] == size[1]
        num_nodes = size[0]

        offset = torch.arange(1, num_nodes, device=idx.device).cumsum(0)
        end = torch.arange(num_nodes, num_nodes * num_nodes, num_nodes,
                           device=idx.device)
        row = torch.bucketize(idx, end.sub_(offset), right=True)
        col = offset[row].add_(idx) % num_nodes
        return torch.stack([torch.cat([row, col]), torch.cat([col, row])], 0)

    else:
        assert size[0] == size[1]
        num_nodes = size[0]

        row = idx.div(num_nodes - 1, rounding_mode='floor')
        col = idx % (num_nodes - 1)
        col[row <= col] += 1
        return torch.stack([row, col], dim=0)
pytorch/torch_geometric/utils/nested.py000066400000000000000000000064201444262304300207230ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.utils import scatter


def to_nested_tensor(
    x: Tensor,
    batch: Optional[Tensor] = None,
    ptr: Optional[Tensor] = None,
    batch_size: Optional[int] = None,
) -> torch.Tensor:
    r"""Given a contiguous batch of tensors
    :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times *}`
    (with :math:`N_i` indicating the number of elements in example :math:`i`),
    creates a `nested PyTorch tensor
    `__.
    Reverse operation of :meth:`from_nested_tensor`.

    Args:
        x (torch.Tensor): The input tensor
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times *}`.
        batch (torch.Tensor, optional): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            element to a specific example. Must be ordered.
            (default: :obj:`None`)
        ptr (torch.Tensor, optional): Alternative representation of
            :obj:`batch` in compressed format. (default: :obj:`None`)
        batch_size (int, optional): The batch size :math:`B`.
            (default: :obj:`None`)
    """
    if ptr is not None:
        offsets = ptr[1:] - ptr[:-1]
        sizes: List[int] = offsets.tolist()
        xs = list(torch.split(x, sizes, dim=0))
    elif batch is not None:
        offsets = scatter(torch.ones_like(batch), batch, dim_size=batch_size)
        sizes: List[int] = offsets.tolist()
        xs = list(torch.split(x, sizes, dim=0))
    else:
        xs = [x]

    # This currently copies the data, although `x` is already contiguous.
    # Sadly, there does not exist any (public) API to preven this :(
    return torch.nested.as_nested_tensor(xs)


def from_nested_tensor(
    x: Tensor,
    return_batch: bool = False,
) -> Union[Tensor, Tuple[Tensor, Tensor]]:
    r"""Given a `nested PyTorch tensor
    `__, creates a contiguous
    batch of tensors
    :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times *}`, and
    optionally a batch vector which assigns each element to a specific example.
    Reverse operation of :meth:`to_nested_tensor`.

    Args:
        x (torch.Tensor): The nested input tensor. The size of nested tensors
            need to match except for the first dimension.
        return_batch (bool, optional): If set to :obj:`True`, will also return
            the batch vector :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`.
            (default: :obj:`False`)
    """
    if not x.is_nested:
        raise ValueError("Input tensor in 'from_nested_tensor' is not nested")

    sizes = x._nested_tensor_size()

    for dim, (a, b) in enumerate(zip(sizes[0, 1:], sizes.t()[1:])):
        if not torch.equal(a.expand_as(b), b):
            raise ValueError(f"Not all nested tensors have the same size "
                             f"in dimension {dim + 1} "
                             f"(expected size {a.item()} for all tensors)")

    out = x.contiguous().values()
    out = out.view(-1, *sizes[0, 1:].tolist())

    if not return_batch:
        return out

    batch = torch.arange(x.size(0), device=x.device)
    batch = batch.repeat_interleave(sizes[:, 0].to(batch.device))

    return out, batch
pytorch/torch_geometric/utils/normalized_cut.py000066400000000000000000000022211444262304300224530ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.utils import degree


def normalized_cut(
    edge_index: Tensor,
    edge_attr: Tensor,
    num_nodes: Optional[int] = None,
) -> Tensor:
    r"""Computes the normalized cut :math:`\mathbf{e}_{i,j} \cdot
    \left( \frac{1}{\deg(i)} + \frac{1}{\deg(j)} \right)` of a weighted graph
    given by edge indices and edge attributes.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor): Edge weights or multi-dimensional edge features.
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: :class:`Tensor`

    Example:

        >>> edge_index = torch.tensor([[1, 1, 2, 3],
        ...                            [3, 3, 1, 2]])
        >>> edge_attr = torch.tensor([1., 1., 1., 1.])
        >>> normalized_cut(edge_index, edge_attr)
        tensor([1.5000, 1.5000, 2.0000, 1.5000])
    """

    row, col = edge_index[0], edge_index[1]
    deg = 1. / degree(col, num_nodes, edge_attr.dtype)
    deg = deg[row] + deg[col]
    cut = edge_attr * deg
    return cut
pytorch/torch_geometric/utils/num_nodes.py000066400000000000000000000035751444262304300214400ustar00rootroot00000000000000from copy import copy
from typing import Dict, Optional, Union

import torch
from torch import Tensor

import torch_geometric
from torch_geometric.typing import EdgeType, NodeType, SparseTensor


@torch.jit._overload
def maybe_num_nodes(edge_index, num_nodes):
    # type: (Tensor, Optional[int]) -> int
    pass


@torch.jit._overload
def maybe_num_nodes(edge_index, num_nodes):
    # type: (SparseTensor, Optional[int]) -> int
    pass


def maybe_num_nodes(
    edge_index: Union[Tensor, SparseTensor],
    num_nodes: Optional[int] = None,
) -> int:
    if num_nodes is not None:
        return num_nodes
    elif isinstance(edge_index, Tensor):
        if torch_geometric.utils.is_torch_sparse_tensor(edge_index):
            return max(edge_index.size(0), edge_index.size(1))

        if torch.jit.is_tracing():
            # Avoid non-traceable if-check for empty `edge_index` tensor:
            tmp = torch.concat([
                edge_index.view(-1),
                edge_index.new_full((1, ), fill_value=-1)
            ])
            return tmp.max() + 1

        return int(edge_index.max()) + 1 if edge_index.numel() > 0 else 0
    else:
        return max(edge_index.size(0), edge_index.size(1))


def maybe_num_nodes_dict(
    edge_index_dict: Dict[EdgeType, Tensor],
    num_nodes_dict: Optional[Dict[NodeType, int]] = None,
) -> Dict[NodeType, int]:
    num_nodes_dict = {} if num_nodes_dict is None else copy(num_nodes_dict)

    found_types = list(num_nodes_dict.keys())

    for keys, edge_index in edge_index_dict.items():

        key = keys[0]
        if key not in found_types:
            N = int(edge_index[0].max() + 1)
            num_nodes_dict[key] = max(N, num_nodes_dict.get(key, N))

        key = keys[-1]
        if key not in found_types:
            N = int(edge_index[1].max() + 1)
            num_nodes_dict[key] = max(N, num_nodes_dict.get(key, N))

    return num_nodes_dict
pytorch/torch_geometric/utils/one_hot.py000066400000000000000000000025741444262304300211020ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor


def one_hot(
    index: Tensor,
    num_classes: Optional[int] = None,
    dtype: Optional[torch.dtype] = None,
) -> Tensor:
    r"""Taskes a one-dimensional :obj:`index` tensor and returns a one-hot
    encoded representation of it with shape :obj:`[*, num_classes]` that has
    zeros everywhere except where the index of last dimension matches the
    corresponding value of the input tensor, in which case it will be :obj:`1`.

    .. note::
        This is a more memory-efficient version of
        :meth:`torch.nn.functional.one_hot` as you can customize the output
        :obj:`dtype`.

    Args:
        index (torch.Tensor): The one-dimensional input tensor.
        num_classes (int, optional): The total number of classes. If set to
            :obj:`None`, the number of classes will be inferred as one greater
            than the largest class value in the input tensor.
            (default: :obj:`None`)
        dtype (torch.dtype, optional): The :obj:`dtype` of the output tensor.
    """
    if index.dim() != 1:
        raise ValueError("'index' tensor needs to be one-dimensional")

    if num_classes is None:
        num_classes = int(index.max()) + 1

    out = torch.zeros((index.size(0), num_classes), dtype=dtype,
                      device=index.device)
    return out.scatter_(1, index.unsqueeze(1), 1)
pytorch/torch_geometric/utils/random.py000066400000000000000000000120421444262304300207160ustar00rootroot00000000000000import warnings
from typing import List, Union

import numpy as np
import torch

from torch_geometric.utils import remove_self_loops, to_undirected


def erdos_renyi_graph(
    num_nodes: int,
    edge_prob: float,
    directed: bool = False,
) -> torch.Tensor:
    r"""Returns the :obj:`edge_index` of a random Erdos-Renyi graph.

    Args:
        num_nodes (int): The number of nodes.
        edge_prob (float): Probability of an edge.
        directed (bool, optional): If set to :obj:`True`, will return a
            directed graph. (default: :obj:`False`)

    Examples:

        >>> erdos_renyi_graph(5, 0.2, directed=False)
        tensor([[0, 1, 1, 4],
                [1, 0, 4, 1]])

        >>> erdos_renyi_graph(5, 0.2, directed=True)
        tensor([[0, 1, 3, 3, 4, 4],
                [4, 3, 1, 2, 1, 3]])
    """

    if directed:
        idx = torch.arange((num_nodes - 1) * num_nodes)
        idx = idx.view(num_nodes - 1, num_nodes)
        idx = idx + torch.arange(1, num_nodes).view(-1, 1)
        idx = idx.view(-1)
    else:
        warnings.filterwarnings('ignore', '.*pass the indexing argument.*')
        idx = torch.combinations(torch.arange(num_nodes), r=2)

    # Filter edges.
    mask = torch.rand(idx.size(0)) < edge_prob
    idx = idx[mask]

    if directed:
        row = idx.div(num_nodes, rounding_mode='floor')
        col = idx % num_nodes
        edge_index = torch.stack([row, col], dim=0)
    else:
        edge_index = to_undirected(idx.t(), num_nodes=num_nodes)

    return edge_index


def stochastic_blockmodel_graph(
    block_sizes: Union[List[int], torch.Tensor],
    edge_probs: Union[List[List[float]], torch.Tensor],
    directed: bool = False,
) -> torch.Tensor:
    r"""Returns the :obj:`edge_index` of a stochastic blockmodel graph.

    Args:
        block_sizes ([int] or LongTensor): The sizes of blocks.
        edge_probs ([[float]] or FloatTensor): The density of edges going
            from each block to each other block. Must be symmetric if the
            graph is undirected.
        directed (bool, optional): If set to :obj:`True`, will return a
            directed graph. (default: :obj:`False`)

    Examples:

        >>> block_sizes = [2, 2, 4]
        >>> edge_probs = [[0.25, 0.05, 0.02],
        ...               [0.05, 0.35, 0.07],
        ...               [0.02, 0.07, 0.40]]
        >>> stochastic_blockmodel_graph(block_sizes, edge_probs,
        ...                             directed=False)
        tensor([[2, 4, 4, 5, 5, 6, 7, 7],
                [5, 6, 7, 2, 7, 4, 4, 5]])

        >>> stochastic_blockmodel_graph(block_sizes, edge_probs,
        ...                             directed=True)
        tensor([[0, 2, 3, 4, 4, 5, 5],
                [3, 4, 1, 5, 6, 6, 7]])
    """

    size, prob = block_sizes, edge_probs

    if not isinstance(size, torch.Tensor):
        size = torch.tensor(size, dtype=torch.long)
    if not isinstance(prob, torch.Tensor):
        prob = torch.tensor(prob, dtype=torch.float)

    assert size.dim() == 1
    assert prob.dim() == 2 and prob.size(0) == prob.size(1)
    assert size.size(0) == prob.size(0)
    if not directed:
        assert torch.allclose(prob, prob.t())

    node_idx = torch.cat([size.new_full((b, ), i) for i, b in enumerate(size)])
    num_nodes = node_idx.size(0)

    if directed:
        idx = torch.arange((num_nodes - 1) * num_nodes)
        idx = idx.view(num_nodes - 1, num_nodes)
        idx = idx + torch.arange(1, num_nodes).view(-1, 1)
        idx = idx.view(-1)
        row = idx.div(num_nodes, rounding_mode='floor')
        col = idx % num_nodes
    else:
        row, col = torch.combinations(torch.arange(num_nodes), r=2).t()

    mask = torch.bernoulli(prob[node_idx[row], node_idx[col]]).to(torch.bool)
    edge_index = torch.stack([row[mask], col[mask]], dim=0)

    if not directed:
        edge_index = to_undirected(edge_index, num_nodes=num_nodes)

    return edge_index


def barabasi_albert_graph(num_nodes: int, num_edges: int) -> torch.Tensor:
    r"""Returns the :obj:`edge_index` of a Barabasi-Albert preferential
    attachment model, where a graph of :obj:`num_nodes` nodes grows by
    attaching new nodes with :obj:`num_edges` edges that are preferentially
    attached to existing nodes with high degree.

    Args:
        num_nodes (int): The number of nodes.
        num_edges (int): The number of edges from a new node to existing nodes.

    Example:

        >>> barabasi_albert_graph(num_nodes=4, num_edges=3)
        tensor([[0, 0, 0, 1, 1, 2, 2, 3],
                [1, 2, 3, 0, 2, 0, 1, 0]])
    """

    assert num_edges > 0 and num_edges < num_nodes

    row, col = torch.arange(num_edges), torch.randperm(num_edges)

    for i in range(num_edges, num_nodes):
        row = torch.cat([row, torch.full((num_edges, ), i, dtype=torch.long)])
        choice = np.random.choice(torch.cat([row, col]).numpy(), num_edges)
        col = torch.cat([col, torch.from_numpy(choice)])

    edge_index = torch.stack([row, col], dim=0)
    edge_index, _ = remove_self_loops(edge_index)
    edge_index = to_undirected(edge_index, num_nodes=num_nodes)

    return edge_index
pytorch/torch_geometric/utils/repeat.py000066400000000000000000000014571444262304300207260ustar00rootroot00000000000000import itertools
import numbers
from typing import Any

import torch
from torch import Tensor


def repeat(src: Any, length: int) -> Any:
    if src is None:
        return None

    if isinstance(src, Tensor):
        if src.numel() == 1:
            return src.repeat(length)

        if src.numel() > length:
            return src[:length]

        if src.numel() < length:
            last_elem = src[-1].unsqueeze(0)
            padding = last_elem.repeat(length - src.numel())
            return torch.cat([src, padding])

        return src

    if isinstance(src, numbers.Number):
        return list(itertools.repeat(src, length))

    if (len(src) > length):
        return src[:length]

    if (len(src) < length):
        return src + list(itertools.repeat(src[-1], length - len(src)))

    return src
pytorch/torch_geometric/utils/scatter.py000066400000000000000000000202111444262304300211000ustar00rootroot00000000000000import warnings
from typing import Optional

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.typing import torch_scatter

major, minor, _ = torch.__version__.split('.', maxsplit=2)
major, minor = int(major), int(minor)
has_pytorch112 = major > 1 or (major == 1 and minor >= 12)

if has_pytorch112:  # pragma: no cover

    warnings.filterwarnings('ignore', '.*is in beta and the API may change.*')

    def scatter(src: Tensor, index: Tensor, dim: int = 0,
                dim_size: Optional[int] = None, reduce: str = 'sum') -> Tensor:
        r"""Reduces all values from the :obj:`src` tensor at the indices
        specified in the :obj:`index` tensor along a given dimension
        :obj:`dim`. See the `documentation
        `__ of the :obj:`torch_scatter` package for more
        information.

        Args:
            src (torch.Tensor): The source tensor.
            index (torch.Tensor): The index tensor.
            dim (int, optional): The dimension along which to index.
                (default: :obj:`0`)
            dim_size (int, optional): The size of the output tensor at
                dimension :obj:`dim`. If set to :obj:`None`, will create a
                minimal-sized output tensor according to
                :obj:`index.max() + 1`. (default: :obj:`None`)
            reduce (str, optional): The reduce operation (:obj:`"sum"`,
                :obj:`"mean"`, :obj:`"mul"`, :obj:`"min"` or :obj:`"max"`,
                :obj:`"any"`). (default: :obj:`"sum"`)
        """
        if isinstance(index, Tensor) and index.dim() != 1:
            raise ValueError(f"The `index` argument must be one-dimensional "
                             f"(got {index.dim()} dimensions)")

        dim = src.dim() + dim if dim < 0 else dim

        if isinstance(src, Tensor) and (dim < 0 or dim >= src.dim()):
            raise ValueError(f"The `dim` argument must lay between 0 and "
                             f"{src.dim() - 1} (got {dim})")

        if dim_size is None:
            dim_size = int(index.max()) + 1 if index.numel() > 0 else 0

        # For now, we maintain various different code paths, based on whether
        # the input requires gradients and whether it lays on the CPU/GPU.
        # For example, `torch_scatter` is usually faster than
        # `torch.scatter_reduce` on GPU, while `torch.scatter_reduce` is faster
        # on CPU.
        # `torch.scatter_reduce` has a faster forward implementation for
        # "min"/"max" reductions since it does not compute additional arg
        # indices, but is therefore way slower in its backward implementation.
        # More insights can be found in `test/utils/test_scatter.py`.

        size = src.size()[:dim] + (dim_size, ) + src.size()[dim + 1:]

        # For "any" reduction, we use regular `scatter_`:
        if reduce == 'any':
            index = broadcast(index, src, dim)
            return src.new_zeros(size).scatter_(dim, index, src)

        # For "sum" and "mean" reduction, we make use of `scatter_add_`:
        if reduce == 'sum' or reduce == 'add':
            index = broadcast(index, src, dim)
            return src.new_zeros(size).scatter_add_(dim, index, src)

        if reduce == 'mean':
            count = src.new_zeros(dim_size)
            count.scatter_add_(0, index, src.new_ones(src.size(dim)))
            count = count.clamp(min=1)

            index = broadcast(index, src, dim)
            out = src.new_zeros(size).scatter_add_(dim, index, src)

            return out / broadcast(count, out, dim)

        # For "min" and "max" reduction, we prefer `scatter_reduce_` on CPU or
        # in case the input does not require gradients:
        if reduce == 'min' or reduce == 'max':
            if (not torch_geometric.typing.WITH_TORCH_SCATTER
                    or not src.is_cuda or not src.requires_grad):

                if src.is_cuda and src.requires_grad:
                    warnings.warn(f"The usage of `scatter(reduce='{reduce}')` "
                                  f"can be accelerated via the 'torch-scatter'"
                                  f" package, but it was not found")

                index = broadcast(index, src, dim)
                return src.new_zeros(size).scatter_reduce_(
                    dim, index, src, reduce=f'a{reduce}', include_self=False)

            return torch_scatter.scatter(src, index, dim, dim_size=dim_size,
                                         reduce=reduce)

        # For "mul" reduction, we prefer `scatter_reduce_` on CPU:
        if reduce == 'mul':
            if (not torch_geometric.typing.WITH_TORCH_SCATTER
                    or not src.is_cuda):

                if src.is_cuda:
                    warnings.warn(f"The usage of `scatter(reduce='{reduce}')` "
                                  f"can be accelerated via the 'torch-scatter'"
                                  f" package, but it was not found")

                index = broadcast(index, src, dim)
                # We initialize with `one` here to match `scatter_mul` output:
                return src.new_ones(size).scatter_reduce_(
                    dim, index, src, reduce='prod', include_self=True)

            return torch_scatter.scatter(src, index, dim, dim_size=dim_size,
                                         reduce='mul')

        raise ValueError(f"Encountered invalid `reduce` argument '{reduce}'")

else:  # pragma: no cover

    def scatter(src: Tensor, index: Tensor, dim: int = 0,
                dim_size: Optional[int] = None, reduce: str = 'sum') -> Tensor:
        r"""Reduces all values from the :obj:`src` tensor at the indices
        specified in the :obj:`index` tensor along a given dimension
        :obj:`dim`. See the `documentation
        `_ of the :obj:`torch_scatter` package for more
        information.

        Args:
            src (torch.Tensor): The source tensor.
            index (torch.Tensor): The index tensor.
            dim (int, optional): The dimension along which to index.
                (default: :obj:`0`)
            dim_size (int, optional): The size of the output tensor at
                dimension :obj:`dim`. If set to :obj:`None`, will create a
                minimal-sized output tensor according to
                :obj:`index.max() + 1`. (default: :obj:`None`)
            reduce (str, optional): The reduce operation (:obj:`"sum"`,
                :obj:`"mean"`, :obj:`"mul"`, :obj:`"min"` or :obj:`"max"`).
                (default: :obj:`"sum"`)
        """
        if reduce == 'any':
            dim = src.dim() + dim if dim < 0 else dim

            if dim_size is None:
                dim_size = int(index.max()) + 1 if index.numel() > 0 else 0

            size = src.size()[:dim] + (dim_size, ) + src.size()[dim + 1:]

            index = broadcast(index, src, dim)
            return src.new_zeros(size).scatter_(dim, index, src)

        if not torch_geometric.typing.WITH_TORCH_SCATTER:
            raise ImportError("'scatter' requires the 'torch-scatter' package")
        return torch_scatter.scatter(src, index, dim, dim_size=dim_size,
                                     reduce=reduce)


def broadcast(src: Tensor, ref: Tensor, dim: int) -> Tensor:
    size = ((1, ) * dim) + (-1, ) + ((1, ) * (ref.dim() - dim - 1))
    return src.view(size).expand_as(ref)


def scatter_argmax(src: Tensor, index: Tensor, dim: int = 0,
                   dim_size: Optional[int] = None) -> Tensor:

    if torch_geometric.typing.WITH_TORCH_SCATTER:
        out = torch_scatter.scatter_max(src, index, dim=dim, dim_size=dim_size)
        return out[1]

    # Only implemented under certain conditions for now :(
    assert dim == 0
    assert src.dim() == 1 and index.dim() == 1

    if dim_size is None:
        dim_size = index.max() + 1 if index.numel() > 0 else 0

    res = src.new_empty(dim_size)
    res.scatter_reduce_(0, index, src.detach(), reduce='amax',
                        include_self=False)

    out = index.new_full((dim_size, ), fill_value=dim_size - 1)
    nonzero = (src == res[index]).nonzero().view(-1)
    out[index[nonzero]] = nonzero

    return out
pytorch/torch_geometric/utils/segment.py000066400000000000000000000021261444262304300211020ustar00rootroot00000000000000from torch import Tensor

import torch_geometric.typing
from torch_geometric.typing import torch_scatter


def segment(src: Tensor, ptr: Tensor, reduce: str = 'sum') -> Tensor:
    r"""Reduces all values in the first dimension of the :obj:`src` tensor
    within the ranges specified in the :obj:`ptr`. See the `documentation
    `__ of the :obj:`torch_scatter` package for more
    information.

    Args:
        src (torch.Tensor): The source tensor.
        ptr (torch.Tensor): A monotonically increasing pointer tensor that
            refers to the boundaries of segments such that :obj:`ptr[0] = 0`
            and :obj:`ptr[-1] = src.size(0)`.
        reduce (str, optional): The reduce operation (:obj:`"sum"`,
            :obj:`"mean"`, :obj:`"mul"`, :obj:`"min"` or :obj:`"max"`).
            (default: :obj:`"sum"`)
    """
    if not torch_geometric.typing.WITH_TORCH_SCATTER:
        raise ImportError("'segment' requires the 'torch-scatter' package")
    return torch_scatter.segment_csr(src, ptr, reduce=reduce)
pytorch/torch_geometric/utils/select.py000066400000000000000000000041451444262304300207220ustar00rootroot00000000000000from typing import Any, List, Union

import torch
from torch import Tensor

from torch_geometric.utils.mask import mask_select
from torch_geometric.utils.sparse import is_torch_sparse_tensor


def select(src: Union[Tensor, List[Any]], index_or_mask: Tensor,
           dim: int) -> Union[Tensor, List[Any]]:
    r"""Selects the input tensor or input list according to a given index or
    mask vector.

    Args:
        src (torch.Tensor or list): The input tensor or list.
        index_or_mask (torch.Tensor): The index or mask vector.
        dim (int): The dimension along which to select.
    """
    if isinstance(src, Tensor):
        if index_or_mask.dtype == torch.bool:
            return mask_select(src, dim, index_or_mask)
        return src.index_select(dim, index_or_mask)

    if isinstance(src, (tuple, list)):
        if dim != 0:
            raise ValueError("Cannot select along dimension other than 0")
        if index_or_mask.dtype == torch.bool:
            return [src[i] for i, m in enumerate(index_or_mask) if m]
        return [src[i] for i in index_or_mask]

    raise ValueError(f"Encountered invalid input type (got '{type(src)}')")


def narrow(src: Union[Tensor, List[Any]], dim: int, start: int,
           length: int) -> Union[Tensor, List[Any]]:
    r"""Narrows the input tensor or input list to the specified range.

    Args:
        src (torch.Tensor or list): The input tensor or list.
        dim (int): The dimension along which to narrow.
        start (int): The starting dimension.
        length (int): The distance to the ending dimension.
    """
    if is_torch_sparse_tensor(src):
        # TODO Sparse tensors in `torch.sparse` do not yet support `narrow`.
        index = torch.arange(start, start + length, device=src.device)
        return src.index_select(dim, index)

    if isinstance(src, Tensor):
        return src.narrow(dim, start, length)

    if isinstance(src, list):
        if dim != 0:
            raise ValueError("Cannot narrow along dimension other than 0")
        return src[start:start + length]

    raise ValueError(f"Encountered invalid input type (got '{type(src)}')")
pytorch/torch_geometric/utils/smiles.py000066400000000000000000000136001444262304300207330ustar00rootroot00000000000000from typing import Any

import torch

import torch_geometric

x_map = {
    'atomic_num':
    list(range(0, 119)),
    'chirality': [
        'CHI_UNSPECIFIED',
        'CHI_TETRAHEDRAL_CW',
        'CHI_TETRAHEDRAL_CCW',
        'CHI_OTHER',
        'CHI_TETRAHEDRAL',
        'CHI_ALLENE',
        'CHI_SQUAREPLANAR',
        'CHI_TRIGONALBIPYRAMIDAL',
        'CHI_OCTAHEDRAL',
    ],
    'degree':
    list(range(0, 11)),
    'formal_charge':
    list(range(-5, 7)),
    'num_hs':
    list(range(0, 9)),
    'num_radical_electrons':
    list(range(0, 5)),
    'hybridization': [
        'UNSPECIFIED',
        'S',
        'SP',
        'SP2',
        'SP3',
        'SP3D',
        'SP3D2',
        'OTHER',
    ],
    'is_aromatic': [False, True],
    'is_in_ring': [False, True],
}

e_map = {
    'bond_type': [
        'UNSPECIFIED',
        'SINGLE',
        'DOUBLE',
        'TRIPLE',
        'QUADRUPLE',
        'QUINTUPLE',
        'HEXTUPLE',
        'ONEANDAHALF',
        'TWOANDAHALF',
        'THREEANDAHALF',
        'FOURANDAHALF',
        'FIVEANDAHALF',
        'AROMATIC',
        'IONIC',
        'HYDROGEN',
        'THREECENTER',
        'DATIVEONE',
        'DATIVE',
        'DATIVEL',
        'DATIVER',
        'OTHER',
        'ZERO',
    ],
    'stereo': [
        'STEREONONE',
        'STEREOANY',
        'STEREOZ',
        'STEREOE',
        'STEREOCIS',
        'STEREOTRANS',
    ],
    'is_conjugated': [False, True],
}


def from_smiles(smiles: str, with_hydrogen: bool = False,
                kekulize: bool = False) -> 'torch_geometric.data.Data':
    r"""Converts a SMILES string to a :class:`torch_geometric.data.Data`
    instance.

    Args:
        smiles (str): The SMILES string.
        with_hydrogen (bool, optional): If set to :obj:`True`, will store
            hydrogens in the molecule graph. (default: :obj:`False`)
        kekulize (bool, optional): If set to :obj:`True`, converts aromatic
            bonds to single/double bonds. (default: :obj:`False`)
    """
    from rdkit import Chem, RDLogger

    from torch_geometric.data import Data

    RDLogger.DisableLog('rdApp.*')

    mol = Chem.MolFromSmiles(smiles)

    if mol is None:
        mol = Chem.MolFromSmiles('')
    if with_hydrogen:
        mol = Chem.AddHs(mol)
    if kekulize:
        Chem.Kekulize(mol)

    xs = []
    for atom in mol.GetAtoms():
        x = []
        x.append(x_map['atomic_num'].index(atom.GetAtomicNum()))
        x.append(x_map['chirality'].index(str(atom.GetChiralTag())))
        x.append(x_map['degree'].index(atom.GetTotalDegree()))
        x.append(x_map['formal_charge'].index(atom.GetFormalCharge()))
        x.append(x_map['num_hs'].index(atom.GetTotalNumHs()))
        x.append(x_map['num_radical_electrons'].index(
            atom.GetNumRadicalElectrons()))
        x.append(x_map['hybridization'].index(str(atom.GetHybridization())))
        x.append(x_map['is_aromatic'].index(atom.GetIsAromatic()))
        x.append(x_map['is_in_ring'].index(atom.IsInRing()))
        xs.append(x)

    x = torch.tensor(xs, dtype=torch.long).view(-1, 9)

    edge_indices, edge_attrs = [], []
    for bond in mol.GetBonds():
        i = bond.GetBeginAtomIdx()
        j = bond.GetEndAtomIdx()

        e = []
        e.append(e_map['bond_type'].index(str(bond.GetBondType())))
        e.append(e_map['stereo'].index(str(bond.GetStereo())))
        e.append(e_map['is_conjugated'].index(bond.GetIsConjugated()))

        edge_indices += [[i, j], [j, i]]
        edge_attrs += [e, e]

    edge_index = torch.tensor(edge_indices)
    edge_index = edge_index.t().to(torch.long).view(2, -1)
    edge_attr = torch.tensor(edge_attrs, dtype=torch.long).view(-1, 3)

    if edge_index.numel() > 0:  # Sort indices.
        perm = (edge_index[0] * x.size(0) + edge_index[1]).argsort()
        edge_index, edge_attr = edge_index[:, perm], edge_attr[perm]

    return Data(x=x, edge_index=edge_index, edge_attr=edge_attr, smiles=smiles)


def to_smiles(data: 'torch_geometric.data.Data',
              kekulize: bool = False) -> Any:
    """Converts a :class:`torch_geometric.data.Data` instance to a SMILES
    string.

    Args:
        data (torch_geometric.data.Data): The molecular graph.
        kekulize (bool, optional): If set to :obj:`True`, converts aromatic
            bonds to single/double bonds. (default: :obj:`False`)
    """
    from rdkit import Chem

    mol = Chem.RWMol()

    for i in range(data.num_nodes):
        atom = Chem.Atom(data.x[i, 0].item())
        atom.SetChiralTag(Chem.rdchem.ChiralType.values[data.x[i, 1].item()])
        atom.SetFormalCharge(x_map['formal_charge'][data.x[i, 3].item()])
        atom.SetNumExplicitHs(x_map['num_hs'][data.x[i, 4].item()])
        atom.SetNumRadicalElectrons(
            x_map['num_radical_electrons'][data.x[i, 5].item()])
        atom.SetHybridization(
            Chem.rdchem.HybridizationType.values[data.x[i, 6].item()])
        atom.SetIsAromatic(data.x[i, 7].item())
        mol.AddAtom(atom)

    edges = [tuple(i) for i in data.edge_index.t().tolist()]
    visited = set()

    for i in range(len(edges)):
        src, dst = edges[i]
        if tuple(sorted(edges[i])) in visited:
            continue

        bond_type = Chem.BondType.values[data.edge_attr[i, 0].item()]
        mol.AddBond(src, dst, bond_type)

        # Set stereochemistry:
        stereo = Chem.rdchem.BondStereo.values[data.edge_attr[i, 1].item()]
        if stereo != Chem.rdchem.BondStereo.STEREONONE:
            db = mol.GetBondBetweenAtoms(src, dst)
            db.SetStereoAtoms(dst, src)
            db.SetStereo(stereo)

        # Set conjugation:
        is_conjugated = bool(data.edge_attr[i, 2].item())
        mol.GetBondBetweenAtoms(src, dst).SetIsConjugated(is_conjugated)

        visited.add(tuple(sorted(edges[i])))

    mol = mol.GetMol()

    if kekulize:
        Chem.Kekulize(mol)

    Chem.SanitizeMol(mol)
    Chem.AssignStereochemistry(mol)

    return Chem.MolToSmiles(mol, isomericSmiles=True)
pytorch/torch_geometric/utils/softmax.py000066400000000000000000000052361444262304300211260ustar00rootroot00000000000000from typing import Optional

from torch import Tensor

from torch_geometric.utils import scatter, segment
from torch_geometric.utils.num_nodes import maybe_num_nodes


def softmax(
    src: Tensor,
    index: Optional[Tensor] = None,
    ptr: Optional[Tensor] = None,
    num_nodes: Optional[int] = None,
    dim: int = 0,
) -> Tensor:
    r"""Computes a sparsely evaluated softmax.
    Given a value tensor :attr:`src`, this function first groups the values
    along the first dimension based on the indices specified in :attr:`index`,
    and then proceeds to compute the softmax individually for each group.

    Args:
        src (Tensor): The source tensor.
        index (LongTensor, optional): The indices of elements for applying the
            softmax. (default: :obj:`None`)
        ptr (LongTensor, optional): If given, computes the softmax based on
            sorted inputs in CSR representation. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`index`. (default: :obj:`None`)
        dim (int, optional): The dimension in which to normalize.
            (default: :obj:`0`)

    :rtype: :class:`Tensor`

    Examples:

        >>> src = torch.tensor([1., 1., 1., 1.])
        >>> index = torch.tensor([0, 0, 1, 2])
        >>> ptr = torch.tensor([0, 2, 3, 4])
        >>> softmax(src, index)
        tensor([0.5000, 0.5000, 1.0000, 1.0000])

        >>> softmax(src, None, ptr)
        tensor([0.5000, 0.5000, 1.0000, 1.0000])

        >>> src = torch.randn(4, 4)
        >>> ptr = torch.tensor([0, 4])
        >>> softmax(src, index, dim=-1)
        tensor([[0.7404, 0.2596, 1.0000, 1.0000],
                [0.1702, 0.8298, 1.0000, 1.0000],
                [0.7607, 0.2393, 1.0000, 1.0000],
                [0.8062, 0.1938, 1.0000, 1.0000]])
    """
    if ptr is not None:
        dim = dim + src.dim() if dim < 0 else dim
        size = ([1] * dim) + [-1]
        count = ptr[1:] - ptr[:-1]
        ptr = ptr.view(size)
        src_max = segment(src.detach(), ptr, reduce='max')
        src_max = src_max.repeat_interleave(count, dim=dim)
        out = (src - src_max).exp()
        out_sum = segment(out, ptr, reduce='sum') + 1e-16
        out_sum = out_sum.repeat_interleave(count, dim=dim)
    elif index is not None:
        N = maybe_num_nodes(index, num_nodes)
        src_max = scatter(src.detach(), index, dim, dim_size=N, reduce='max')
        out = src - src_max.index_select(dim, index)
        out = out.exp()
        out_sum = scatter(out, index, dim, dim_size=N, reduce='sum') + 1e-16
        out_sum = out_sum.index_select(dim, index)
    else:
        raise NotImplementedError

    return out / out_sum
pytorch/torch_geometric/utils/sort.py000066400000000000000000000017711444262304300204340ustar00rootroot00000000000000from typing import Optional, Tuple

import torch

import torch_geometric.typing
from torch_geometric.typing import pyg_lib


def index_sort(
    inputs: torch.Tensor,
    max_value: Optional[int] = None,
) -> Tuple[torch.Tensor, torch.Tensor]:
    r"""Sorts the elements of the :obj:`inputs` tensor in ascending order.
    It is expected that :obj:`inputs` is one-dimensional and that it only
    contains positive integer values. If :obj:`max_value` is given, it can
    be used by the underlying algorithm for better performance.

    Args:
        inputs (torch.Tensor): A vector with positive integer values.
        max_value (int, optional): The maximum value stored inside
            :obj:`inputs`. This value can be an estimation, but needs to be
            greater than or equal to the real maximum.
            (default: :obj:`None`)
    """
    if not torch_geometric.typing.WITH_INDEX_SORT:  # pragma: no cover
        return inputs.sort()
    return pyg_lib.ops.index_sort(inputs, max_value=max_value)
pytorch/torch_geometric/utils/sort_edge_index.py000066400000000000000000000057721444262304300226140ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import index_sort
from torch_geometric.utils.num_nodes import maybe_num_nodes

MISSING = '???'


@torch.jit._overload
def sort_edge_index(edge_index, edge_attr, num_nodes, sort_by_row):  # noqa
    # type: (Tensor, str, Optional[int], bool) -> Tensor  # noqa
    pass


@torch.jit._overload
def sort_edge_index(edge_index, edge_attr, num_nodes, sort_by_row):  # noqa
    # type: (Tensor, Optional[Tensor], Optional[int], bool) -> Tuple[Tensor, Optional[Tensor]]  # noqa
    pass


@torch.jit._overload
def sort_edge_index(edge_index, edge_attr, num_nodes, sort_by_row):  # noqa
    # type: (Tensor, List[Tensor], Optional[int], bool) -> Tuple[Tensor, List[Tensor]]  # noqa
    pass


def sort_edge_index(  # noqa
    edge_index: Tensor,
    edge_attr: Union[OptTensor, List[Tensor], str] = MISSING,
    num_nodes: Optional[int] = None,
    sort_by_row: bool = True,
) -> Union[Tensor, Tuple[Tensor, OptTensor], Tuple[Tensor, List[Tensor]]]:
    """Row-wise sorts :obj:`edge_index`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor or List[Tensor], optional): Edge weights or multi-
            dimensional edge features.
            If given as a list, will re-shuffle and remove duplicates for all
            its entries. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        sort_by_row (bool, optional): If set to :obj:`False`, will sort
            :obj:`edge_index` column-wise.

    :rtype: :class:`LongTensor` if :attr:`edge_attr` is not passed, else
        (:class:`LongTensor`, :obj:`Optional[Tensor]` or :obj:`List[Tensor]]`)

    .. warning::

        From :pyg:`PyG >= 2.3.0` onwards, this function will always return a
        tuple whenever :obj:`edge_attr` is passed as an argument (even in case
        it is set to :obj:`None`).

    Examples:

        >>> edge_index = torch.tensor([[2, 1, 1, 0],
                                [1, 2, 0, 1]])
        >>> edge_attr = torch.tensor([[1], [2], [3], [4]])
        >>> sort_edge_index(edge_index)
        tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]])

        >>> sort_edge_index(edge_index, edge_attr)
        (tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]]),
        tensor([[4],
                [3],
                [2],
                [1]]))
    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    idx = edge_index[1 - int(sort_by_row)] * num_nodes
    idx += edge_index[int(sort_by_row)]

    _, perm = index_sort(idx, max_value=num_nodes * num_nodes)

    edge_index = edge_index[:, perm]

    if edge_attr is None:
        return edge_index, None
    if isinstance(edge_attr, Tensor):
        return edge_index, edge_attr[perm]
    if isinstance(edge_attr, (list, tuple)):
        return edge_index, [e[perm] for e in edge_attr]

    return edge_index
pytorch/torch_geometric/utils/sparse.py000066400000000000000000000350141444262304300207370ustar00rootroot00000000000000from typing import Any, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import SparseTensor
from torch_geometric.utils import coalesce


def dense_to_sparse(adj: Tensor) -> Tuple[Tensor, Tensor]:
    r"""Converts a dense adjacency matrix to a sparse adjacency matrix defined
    by edge indices and edge attributes.

    Args:
        adj (Tensor): The dense adjacency matrix of shape
            :obj:`[num_nodes, num_nodes]` or
            :obj:`[batch_size, num_nodes, num_nodes]`.

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Examples:

        >>> # Forr a single adjacency matrix
        >>> adj = torch.tensor([[3, 1],
        ...                     [2, 0]])
        >>> dense_to_sparse(adj)
        (tensor([[0, 0, 1],
                [0, 1, 0]]),
        tensor([3, 1, 2]))

        >>> # For two adjacency matrixes
        >>> adj = torch.tensor([[[3, 1],
        ...                      [2, 0]],
        ...                     [[0, 1],
        ...                      [0, 2]]])
        >>> dense_to_sparse(adj)
        (tensor([[0, 0, 1, 2, 3],
                [0, 1, 0, 3, 3]]),
        tensor([3, 1, 2, 1, 2]))
    """
    if adj.dim() < 2 or adj.dim() > 3:
        raise ValueError(f"Dense adjacency matrix 'adj' must be 2- or "
                         f"3-dimensional (got {adj.dim()} dimensions)")

    edge_index = adj.nonzero().t()

    if edge_index.size(0) == 2:
        edge_attr = adj[edge_index[0], edge_index[1]]
        return edge_index, edge_attr
    else:
        edge_attr = adj[edge_index[0], edge_index[1], edge_index[2]]
        row = edge_index[1] + adj.size(-2) * edge_index[0]
        col = edge_index[2] + adj.size(-1) * edge_index[0]
        return torch.stack([row, col], dim=0), edge_attr


def is_torch_sparse_tensor(src: Any) -> bool:
    r"""Returns :obj:`True` if the input :obj:`src` is a
    :class:`torch.sparse.Tensor` (in any sparse layout).

    Args:
        src (Any): The input object to be checked.
    """
    if isinstance(src, Tensor):
        if src.layout == torch.sparse_coo:
            return True
        if src.layout == torch.sparse_csr:
            return True
        if src.layout == torch.sparse_csc:
            return True
    return False


def is_sparse(src: Any) -> bool:
    r"""Returns :obj:`True` if the input :obj:`src` is of type
    :class:`torch.sparse.Tensor` (in any sparse layout) or of type
    :class:`torch_sparse.SparseTensor`.

    Args:
        src (Any): The input object to be checked.
    """
    return is_torch_sparse_tensor(src) or isinstance(src, SparseTensor)


def to_torch_coo_tensor(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    size: Optional[Union[int, Tuple[int, int]]] = None,
    is_coalesced: bool = False,
) -> Tensor:
    r"""Converts a sparse adjacency matrix defined by edge indices and edge
    attributes to a :class:`torch.sparse.Tensor` with layout
    `torch.sparse_coo`.
    See :meth:`~torch_geometric.utils.to_edge_index` for the reverse operation.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): The edge attributes.
            (default: :obj:`None`)
        size (int or (int, int), optional): The size of the sparse matrix.
            If given as an integer, will create a quadratic sparse matrix.
            If set to :obj:`None`, will infer a quadratic sparse matrix based
            on :obj:`edge_index.max() + 1`. (default: :obj:`None`)
        is_coalesced (bool): If set to :obj:`True`, will assume that
            :obj:`edge_index` is already coalesced and thus avoids expensive
            computation. (default: :obj:`False`)

    :rtype: :class:`torch.sparse.Tensor`

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> to_torch_coo_tensor(edge_index)
        tensor(indices=tensor([[0, 1, 1, 2, 2, 3],
                               [1, 0, 2, 1, 3, 2]]),
               values=tensor([1., 1., 1., 1., 1., 1.]),
               size=(4, 4), nnz=6, layout=torch.sparse_coo)

    """
    if size is None:
        size = int(edge_index.max()) + 1
    if not isinstance(size, (tuple, list)):
        size = (size, size)

    if not is_coalesced:
        edge_index, edge_attr = coalesce(edge_index, edge_attr, max(size))

    if edge_attr is None:
        edge_attr = torch.ones(edge_index.size(1), device=edge_index.device)

    adj = torch.sparse_coo_tensor(
        indices=edge_index,
        values=edge_attr,
        size=tuple(size) + edge_attr.size()[1:],
        device=edge_index.device,
    )
    adj = adj._coalesced_(True)

    return adj


def to_torch_csr_tensor(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    size: Optional[Union[int, Tuple[int, int]]] = None,
    is_coalesced: bool = False,
) -> Tensor:
    r"""Converts a sparse adjacency matrix defined by edge indices and edge
    attributes to a :class:`torch.sparse.Tensor` with layout
    `torch.sparse_csr`.
    See :meth:`~torch_geometric.utils.to_edge_index` for the reverse operation.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): The edge attributes.
            (default: :obj:`None`)
        size (int or (int, int), optional): The size of the sparse matrix.
            If given as an integer, will create a quadratic sparse matrix.
            If set to :obj:`None`, will infer a quadratic sparse matrix based
            on :obj:`edge_index.max() + 1`. (default: :obj:`None`)
        is_coalesced (bool): If set to :obj:`True`, will assume that
            :obj:`edge_index` is already coalesced and thus avoids expensive
            computation. (default: :obj:`False`)

    :rtype: :class:`torch.sparse.Tensor`

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> to_torch_csr_tensor(edge_index)
        tensor(crow_indices=tensor([0, 1, 3, 5, 6]),
               col_indices=tensor([1, 0, 2, 1, 3, 2]),
               values=tensor([1., 1., 1., 1., 1., 1.]),
               size=(4, 4), nnz=6, layout=torch.sparse_csr)

    """
    adj = to_torch_coo_tensor(edge_index, edge_attr, size, is_coalesced)
    return adj.to_sparse_csr()


def to_torch_csc_tensor(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    size: Optional[Union[int, Tuple[int, int]]] = None,
    is_coalesced: bool = False,
) -> Tensor:
    r"""Converts a sparse adjacency matrix defined by edge indices and edge
    attributes to a :class:`torch.sparse.Tensor` with layout
    `torch.sparse_csc`.
    See :meth:`~torch_geometric.utils.to_edge_index` for the reverse operation.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): The edge attributes.
            (default: :obj:`None`)
        size (int or (int, int), optional): The size of the sparse matrix.
            If given as an integer, will create a quadratic sparse matrix.
            If set to :obj:`None`, will infer a quadratic sparse matrix based
            on :obj:`edge_index.max() + 1`. (default: :obj:`None`)
        is_coalesced (bool): If set to :obj:`True`, will assume that
            :obj:`edge_index` is already coalesced and thus avoids expensive
            computation. (default: :obj:`False`)

    :rtype: :class:`torch.sparse.Tensor`

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> to_torch_csc_tensor(edge_index)
        tensor(ccol_indices=tensor([0, 1, 3, 5, 6]),
               row_indices=tensor([1, 0, 2, 1, 3, 2]),
               values=tensor([1., 1., 1., 1., 1., 1.]),
               size=(4, 4), nnz=6, layout=torch.sparse_csc)

    """
    if size is None:
        size = int(edge_index.max()) + 1
    if not isinstance(size, (tuple, list)):
        size = (size, size)

    if not is_coalesced:
        edge_index, edge_attr = coalesce(edge_index, edge_attr, max(size),
                                         sort_by_row=False)

    if edge_attr is None:
        edge_attr = torch.ones(edge_index.size(1), device=edge_index.device)

    adj = torch.sparse_csc_tensor(
        ccol_indices=index2ptr(edge_index[1], size[1]),
        row_indices=edge_index[0],
        values=edge_attr,
        size=tuple(size) + edge_attr.size()[1:],
        device=edge_index.device,
    )

    return adj


def to_torch_sparse_tensor(
    edge_index: Tensor,
    edge_attr: Optional[Tensor] = None,
    size: Optional[Union[int, Tuple[int, int]]] = None,
    is_coalesced: bool = False,
    layout: torch.layout = torch.sparse_coo,
):
    r"""Converts a sparse adjacency matrix defined by edge indices and edge
    attributes to a :class:`torch.sparse.Tensor` with custom :obj:`layout`.
    See :meth:`~torch_geometric.utils.to_edge_index` for the reverse operation.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): The edge attributes.
            (default: :obj:`None`)
        size (int or (int, int), optional): The size of the sparse matrix.
            If given as an integer, will create a quadratic sparse matrix.
            If set to :obj:`None`, will infer a quadratic sparse matrix based
            on :obj:`edge_index.max() + 1`. (default: :obj:`None`)
        is_coalesced (bool): If set to :obj:`True`, will assume that
            :obj:`edge_index` is already coalesced and thus avoids expensive
            computation. (default: :obj:`False`)
        layout (torch.layout, optional): The layout of the output sparse tensor
            (:obj:`torch.sparse_coo`, :obj:`torch.sparse_csr`,
            :obj:`torch.sparse_csc`). (default: :obj:`torch.sparse_coo`)

    :rtype: :class:`torch.sparse.Tensor`
    """
    if layout == torch.sparse_coo:
        return to_torch_coo_tensor(edge_index, edge_attr, size, is_coalesced)
    if layout == torch.sparse_csr:
        return to_torch_csr_tensor(edge_index, edge_attr, size, is_coalesced)
    if layout == torch.sparse_csc:
        return to_torch_csc_tensor(edge_index, edge_attr, size, is_coalesced)

    raise ValueError(f"Unexpected sparse tensor layout (got '{layout}')")


def to_edge_index(adj: Union[Tensor, SparseTensor]) -> Tuple[Tensor, Tensor]:
    r"""Converts a :class:`torch.sparse.Tensor` or a
    :class:`torch_sparse.SparseTensor` to edge indices and edge attributes.

    Args:
        adj (torch.sparse.Tensor or SparseTensor): The adjacency matrix.

    :rtype: (:class:`torch.Tensor`, :class:`torch.Tensor`)

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
        ...                            [1, 0, 2, 1, 3, 2]])
        >>> adj = to_torch_coo_tensor(edge_index)
        >>> to_edge_index(adj)
        (tensor([[0, 1, 1, 2, 2, 3],
                [1, 0, 2, 1, 3, 2]]),
        tensor([1., 1., 1., 1., 1., 1.]))
    """
    if isinstance(adj, SparseTensor):
        row, col, value = adj.coo()
        if value is None:
            value = torch.ones(row.size(0), device=row.device)
        return torch.stack([row, col], dim=0).long(), value

    if adj.layout == torch.sparse_coo:
        return adj.indices().detach().long(), adj.values()

    if adj.layout == torch.sparse_csr:
        row = ptr2index(adj.crow_indices().detach())
        col = adj.col_indices().detach()
        return torch.stack([row, col], dim=0).long(), adj.values()

    if adj.layout == torch.sparse_csc:
        col = ptr2index(adj.ccol_indices().detach())
        row = adj.row_indices().detach()
        return torch.stack([row, col], dim=0).long(), adj.values()

    raise ValueError(f"Unexpected sparse tensor layout (got '{adj.layout}')")


# Helper functions ############################################################


def get_sparse_diag(
    size: int,
    fill_value: float = 1.0,
    layout: Optional[int] = None,
    dtype: Optional[torch.dtype] = None,
    device: Optional[torch.device] = None,
) -> Tensor:
    return torch.sparse.spdiags(
        torch.full((1, size), fill_value, dtype=dtype, device=device),
        offsets=torch.zeros(1, dtype=torch.long, device=device),
        shape=(size, size),
        layout=layout,
    )


def set_sparse_value(adj: Tensor, value: Tensor) -> Tensor:
    size = adj.size()

    if value.dim() > 1:
        size = size + value.size()[1:]

    if adj.layout == torch.sparse_coo:
        return torch.sparse_coo_tensor(
            indices=adj.indices(),
            values=value,
            size=size,
            device=value.device,
        ).coalesce()

    if adj.layout == torch.sparse_csr:
        return torch.sparse_csr_tensor(
            crow_indices=adj.crow_indices(),
            col_indices=adj.col_indices(),
            values=value,
            size=size,
            device=value.device,
        )

    if adj.layout == torch.sparse_csc:
        return torch.sparse_csc_tensor(
            ccol_indices=adj.ccol_indices(),
            row_indices=adj.row_indices(),
            values=value,
            size=size,
            device=value.device,
        )

    raise ValueError(f"Unexpected sparse tensor layout (got '{adj.layout}')")


def ptr2index(ptr: Tensor) -> Tensor:
    ind = torch.arange(ptr.numel() - 1, dtype=ptr.dtype, device=ptr.device)
    return ind.repeat_interleave(ptr[1:] - ptr[:-1])


def index2ptr(index: Tensor, size: int) -> Tensor:
    return torch._convert_indices_from_coo_to_csr(
        index, size, out_int32=index.dtype == torch.int32)


def cat(tensors: List[Tensor], dim: Union[int, Tuple[int, int]]) -> Tensor:
    # TODO (matthias) We can make this more efficient by directly operating on
    # the individual sparse tensor layouts.
    assert dim in {0, 1, (0, 1)}

    size = [0, 0]
    edge_indices = []
    edge_attrs = []
    for tensor in tensors:
        assert is_torch_sparse_tensor(tensor)
        edge_index, edge_attr = to_edge_index(tensor)
        edge_index = edge_index.clone()

        if dim == 0:
            edge_index[0] += size[0]
            size[0] += tensor.size(0)
            size[1] = max(size[1], tensor.size(1))
        elif dim == 1:
            edge_index[1] += size[1]
            size[0] = max(size[0], tensor.size(0))
            size[1] += tensor.size(1)
        else:
            edge_index[0] += size[0]
            edge_index[1] += size[1]
            size[0] += tensor.size(0)
            size[1] += tensor.size(1)

        edge_indices.append(edge_index)
        edge_attrs.append(edge_attr)

    return to_torch_sparse_tensor(
        edge_index=torch.cat(edge_indices, dim=1),
        edge_attr=torch.cat(edge_attrs, dim=0),
        size=size,
        is_coalesced=dim == (0, 1),
        layout=tensors[0].layout,
    )
pytorch/torch_geometric/utils/spmm.py000066400000000000000000000130561444262304300204200ustar00rootroot00000000000000import warnings

import torch
from torch import Tensor

import torch_geometric.typing
from torch_geometric.typing import Adj, SparseTensor, torch_sparse
from torch_geometric.utils import is_torch_sparse_tensor, scatter


@torch.jit._overload
def spmm(src, other, reduce):
    # type: (Tensor, Tensor, str) -> Tensor
    pass


@torch.jit._overload
def spmm(src, other, reduce):
    # type: (SparseTensor, Tensor, str) -> Tensor
    pass


def spmm(src: Adj, other: Tensor, reduce: str = "sum") -> Tensor:
    """Matrix product of sparse matrix with dense matrix.

    Args:
        src (torch.Tensor or torch_sparse.SparseTensor): The input sparse
            matrix, either a :pyg:`PyG` :class:`torch_sparse.SparseTensor` or a
            :pytorch:`PyTorch` :class:`torch.sparse.Tensor`.
        other (torch.Tensor): The input dense matrix.
        reduce (str, optional): The reduce operation to use
            (:obj:`"sum"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`).
            (default: :obj:`"sum"`)

    :rtype: :class:`Tensor`
    """
    reduce = 'sum' if reduce == 'add' else reduce

    if reduce not in ['sum', 'mean', 'min', 'max']:
        raise ValueError(f"`reduce` argument '{reduce}' not supported")

    if isinstance(src, SparseTensor):
        if src.nnz() == 0:
            return other.new_zeros(src.size(0), other.size(1))

        if (torch_geometric.typing.WITH_PT2 and other.dim() == 2
                and not src.is_cuda() and not src.requires_grad()):
            # Use optimized PyTorch `torch.sparse.mm` path:
            csr = src.to_torch_sparse_csr_tensor()
            return torch.sparse.mm(csr, other, reduce)
        return torch_sparse.matmul(src, other, reduce)

    if not is_torch_sparse_tensor(src):
        raise ValueError("`src` must be a `torch_sparse.SparseTensor` "
                         f"or a `torch.sparse.Tensor` (got {type(src)}).")

    # `torch.sparse.mm` only supports reductions on CPU for PyTorch>=2.0.
    # This will currently throw on error for CUDA tensors.
    if torch_geometric.typing.WITH_PT2:

        if src.is_cuda and (reduce == 'min' or reduce == 'max'):
            raise NotImplementedError(f"`{reduce}` reduction is not yet "
                                      f"supported for 'torch.sparse.Tensor' "
                                      f"on device '{src.device}'")

        # Always convert COO to CSR for more efficient processing:
        if src.layout == torch.sparse_coo:
            warnings.warn(f"Converting sparse tensor to CSR format for more "
                          f"efficient processing. Consider converting your "
                          f"sparse tensor to CSR format beforehand to avoid "
                          f"repeated conversion (got '{src.layout}')")
            src = src.to_sparse_csr()

        # Warn in case of CSC format without gradient computation:
        if src.layout == torch.sparse_csc and not other.requires_grad:
            warnings.warn(f"Converting sparse tensor to CSR format for more "
                          f"efficient processing. Consider converting your "
                          f"sparse tensor to CSR format beforehand to avoid "
                          f"repeated conversion (got '{src.layout}')")

        # Use the default code path for `sum` reduction (works on CPU/GPU):
        if reduce == 'sum':
            return torch.sparse.mm(src, other)

        # Use the default code path with custom reduction (works on CPU):
        if src.layout == torch.sparse_csr and not src.is_cuda:
            return torch.sparse.mm(src, other, reduce)

        # Simulate `mean` reduction by dividing by degree:
        if reduce == 'mean':
            if src.layout == torch.sparse_csr:
                ptr = src.crow_indices()
                deg = ptr[1:] - ptr[:-1]
            else:
                assert src.layout == torch.sparse_csc
                deg = scatter(torch.ones_like(src.values()), src.row_indices(),
                              dim=0, dim_size=src.size(0), reduce='sum')

            return torch.sparse.mm(src, other) / deg.view(-1, 1).clamp_(min=1)

        # TODO The `torch.sparse.mm` code path with the `reduce` argument does
        # not yet support CSC :(
        if src.layout == torch.sparse_csc:
            warnings.warn(f"Converting sparse tensor to CSR format for more "
                          f"efficient processing. Consider converting your "
                          f"sparse tensor to CSR format beforehand to avoid "
                          f"repeated conversion (got '{src.layout}')")
            src = src.to_sparse_csr()

        return torch.sparse.mm(src, other, reduce)

    # pragma: no cover
    # PyTorch < 2.0 only supports sparse COO format:
    if reduce == 'sum':
        return torch.sparse.mm(src, other)
    elif reduce == 'mean':
        if src.layout == torch.sparse_csr:
            ptr = src.crow_indices()
            deg = ptr[1:] - ptr[:-1]
        elif src.layout == torch.sparse_csc:
            assert src.layout == torch.sparse_csc
            deg = scatter(torch.ones_like(src.values()), src.row_indices(),
                          dim=0, dim_size=src.size(0), reduce='sum')
        else:
            assert src.layout == torch.sparse_coo
            src = src.coalesce()
            deg = scatter(torch.ones_like(src.values()),
                          src.indices()[0], dim=0, dim_size=src.size(0),
                          reduce='sum')

        return torch.sparse.mm(src, other) / deg.view(-1, 1).clamp_(min=1)

    raise ValueError(f"`{reduce}` reduction is not supported for "
                     f"'torch.sparse.Tensor' on device '{src.device}'")
pytorch/torch_geometric/utils/subgraph.py000066400000000000000000000301221444262304300212500ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor, PairTensor
from torch_geometric.utils.mask import index_to_mask
from torch_geometric.utils.num_nodes import maybe_num_nodes


def get_num_hops(model: torch.nn.Module) -> int:
    r"""Returns the number of hops the model is aggregating information
    from.

    Example:

        >>> class GNN(torch.nn.Module):
        ...     def __init__(self):
        ...         super().__init__()
        ...         self.conv1 = GCNConv(3, 16)
        ...         self.conv2 = GCNConv(16, 16)
        ...         self.lin = Linear(16, 2)
        ...
        ...     def forward(self, x, edge_index):
        ...         x = torch.F.relu(self.conv1(x, edge_index))
        ...         x = self.conv2(x, edge_index)
        ...         return self.lin(x)
        >>> get_num_hops(GNN())
        2
    """
    from torch_geometric.nn.conv import MessagePassing
    num_hops = 0
    for module in model.modules():
        if isinstance(module, MessagePassing):
            num_hops += 1
    return num_hops


def subgraph(
    subset: Union[Tensor, List[int]],
    edge_index: Tensor,
    edge_attr: OptTensor = None,
    relabel_nodes: bool = False,
    num_nodes: Optional[int] = None,
    return_edge_mask: bool = False,
) -> Union[Tuple[Tensor, OptTensor], Tuple[Tensor, OptTensor, OptTensor]]:
    r"""Returns the induced subgraph of :obj:`(edge_index, edge_attr)`
    containing the nodes in :obj:`subset`.

    Args:
        subset (LongTensor, BoolTensor or [int]): The nodes to keep.
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)
        relabel_nodes (bool, optional): If set to :obj:`True`, the resulting
            :obj:`edge_index` will be relabeled to hold consecutive indices
            starting from zero. (default: :obj:`False`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        return_edge_mask (bool, optional): If set to :obj:`True`, will return
            the edge mask to filter out additional edge features.
            (default: :obj:`False`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6],
        ...                            [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5]])
        >>> edge_attr = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
        >>> subset = torch.tensor([3, 4, 5])
        >>> subgraph(subset, edge_index, edge_attr)
        (tensor([[3, 4, 4, 5],
                [4, 3, 5, 4]]),
        tensor([ 7.,  8.,  9., 10.]))

        >>> subgraph(subset, edge_index, edge_attr, return_edge_mask=True)
        (tensor([[3, 4, 4, 5],
                [4, 3, 5, 4]]),
        tensor([ 7.,  8.,  9., 10.]),
        tensor([False, False, False, False, False, False,  True,
                True,  True,  True,  False, False]))
    """

    device = edge_index.device

    if isinstance(subset, (list, tuple)):
        subset = torch.tensor(subset, dtype=torch.long, device=device)

    if subset.dtype != torch.bool:
        num_nodes = maybe_num_nodes(edge_index, num_nodes)
        node_mask = index_to_mask(subset, size=num_nodes)
    else:
        num_nodes = subset.size(0)
        node_mask = subset

    edge_mask = node_mask[edge_index[0]] & node_mask[edge_index[1]]
    edge_index = edge_index[:, edge_mask]
    edge_attr = edge_attr[edge_mask] if edge_attr is not None else None

    if relabel_nodes:
        node_idx = torch.zeros(node_mask.size(0), dtype=torch.long,
                               device=device)
        node_idx[subset] = torch.arange(node_mask.sum().item(), device=device)
        edge_index = node_idx[edge_index]

    if return_edge_mask:
        return edge_index, edge_attr, edge_mask
    else:
        return edge_index, edge_attr


def bipartite_subgraph(
    subset: Union[PairTensor, Tuple[List[int], List[int]]],
    edge_index: Tensor,
    edge_attr: OptTensor = None,
    relabel_nodes: bool = False,
    size: Optional[Tuple[int, int]] = None,
    return_edge_mask: bool = False,
) -> Union[Tuple[Tensor, OptTensor], Tuple[Tensor, OptTensor, OptTensor]]:
    r"""Returns the induced subgraph of the bipartite graph
    :obj:`(edge_index, edge_attr)` containing the nodes in :obj:`subset`.

    Args:
        subset (Tuple[Tensor, Tensor] or tuple([int],[int])): The nodes
            to keep.
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor, optional): Edge weights or multi-dimensional
            edge features. (default: :obj:`None`)
        relabel_nodes (bool, optional): If set to :obj:`True`, the resulting
            :obj:`edge_index` will be relabeled to hold consecutive indices
            starting from zero. (default: :obj:`False`)
        size (tuple, optional): The number of nodes.
            (default: :obj:`None`)
        return_edge_mask (bool, optional): If set to :obj:`True`, will return
            the edge mask to filter out additional edge features.
            (default: :obj:`False`)

    :rtype: (:class:`LongTensor`, :class:`Tensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 5, 2, 3, 3, 4, 4, 3, 5, 5, 6],
        ...                            [0, 0, 3, 2, 0, 0, 2, 1, 2, 3, 1]])
        >>> edge_attr = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
        >>> subset = (torch.tensor([2, 3, 5]), torch.tensor([2, 3]))
        >>> bipartite_subgraph(subset, edge_index, edge_attr)
        (tensor([[2, 3, 5, 5],
                [3, 2, 2, 3]]),
        tensor([ 3,  4,  9, 10]))

        >>> bipartite_subgraph(subset, edge_index, edge_attr,
        ...                    return_edge_mask=True)
        (tensor([[2, 3, 5, 5],
                [3, 2, 2, 3]]),
        tensor([ 3,  4,  9, 10]),
        tensor([False, False,  True,  True, False, False, False, False,
                True,  True,  False]))
    """

    device = edge_index.device

    src_subset, dst_subset = subset
    if not isinstance(src_subset, Tensor):
        src_subset = torch.tensor(src_subset, dtype=torch.long, device=device)
    if not isinstance(dst_subset, Tensor):
        dst_subset = torch.tensor(dst_subset, dtype=torch.long, device=device)

    if src_subset.dtype != torch.bool:
        src_size = int(edge_index[0].max()) + 1 if size is None else size[0]
        src_node_mask = index_to_mask(src_subset, size=src_size)
    else:
        src_size = src_subset.size(0)
        src_node_mask = src_subset

    if dst_subset.dtype != torch.bool:
        dst_size = int(edge_index[1].max()) + 1 if size is None else size[1]
        dst_node_mask = index_to_mask(dst_subset, size=dst_size)
    else:
        dst_size = dst_subset.size(0)
        dst_node_mask = dst_subset

    edge_mask = src_node_mask[edge_index[0]] & dst_node_mask[edge_index[1]]
    edge_index = edge_index[:, edge_mask]
    edge_attr = edge_attr[edge_mask] if edge_attr is not None else None

    if relabel_nodes:
        node_idx_i = edge_index.new_zeros(src_node_mask.size(0))
        node_idx_j = edge_index.new_zeros(dst_node_mask.size(0))
        node_idx_i[src_subset] = torch.arange(int(src_node_mask.sum()),
                                              device=node_idx_i.device)
        node_idx_j[dst_subset] = torch.arange(int(dst_node_mask.sum()),
                                              device=node_idx_j.device)
        edge_index = torch.stack([
            node_idx_i[edge_index[0]],
            node_idx_j[edge_index[1]],
        ], dim=0)

    if return_edge_mask:
        return edge_index, edge_attr, edge_mask
    else:
        return edge_index, edge_attr


def k_hop_subgraph(
    node_idx: Union[int, List[int], Tensor],
    num_hops: int,
    edge_index: Tensor,
    relabel_nodes: bool = False,
    num_nodes: Optional[int] = None,
    flow: str = 'source_to_target',
    directed: bool = False,
) -> Tuple[Tensor, Tensor, Tensor, Tensor]:
    r"""Computes the induced subgraph of :obj:`edge_index` around all nodes in
    :attr:`node_idx` reachable within :math:`k` hops.

    The :attr:`flow` argument denotes the direction of edges for finding
    :math:`k`-hop neighbors. If set to :obj:`"source_to_target"`, then the
    method will find all neighbors that point to the initial set of seed nodes
    in :attr:`node_idx.`
    This mimics the natural flow of message passing in Graph Neural Networks.

    The method returns (1) the nodes involved in the subgraph, (2) the filtered
    :obj:`edge_index` connectivity, (3) the mapping from node indices in
    :obj:`node_idx` to their new location, and (4) the edge mask indicating
    which edges were preserved.

    Args:
        node_idx (int, list, tuple or :obj:`torch.Tensor`): The central seed
            node(s).
        num_hops (int): The number of hops :math:`k`.
        edge_index (LongTensor): The edge indices.
        relabel_nodes (bool, optional): If set to :obj:`True`, the resulting
            :obj:`edge_index` will be relabeled to hold consecutive indices
            starting from zero. (default: :obj:`False`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        flow (str, optional): The flow direction of :math:`k`-hop aggregation
            (:obj:`"source_to_target"` or :obj:`"target_to_source"`).
            (default: :obj:`"source_to_target"`)
        directed (bool, optional): If set to :obj:`False`, will include all
            edges between all sampled nodes. (default: :obj:`True`)

    :rtype: (:class:`LongTensor`, :class:`LongTensor`, :class:`LongTensor`,
             :class:`BoolTensor`)

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 2, 3, 4, 5],
        ...                            [2, 2, 4, 4, 6, 6]])

        >>> # Center node 6, 2-hops
        >>> subset, edge_index, mapping, edge_mask = k_hop_subgraph(
        ...     6, 2, edge_index, relabel_nodes=True)
        >>> subset
        tensor([2, 3, 4, 5, 6])
        >>> edge_index
        tensor([[0, 1, 2, 3],
                [2, 2, 4, 4]])
        >>> mapping
        tensor([4])
        >>> edge_mask
        tensor([False, False,  True,  True,  True,  True])
        >>> subset[mapping]
        tensor([6])

        >>> edge_index = torch.tensor([[1, 2, 4, 5],
        ...                            [0, 1, 5, 6]])
        >>> (subset, edge_index,
        ...  mapping, edge_mask) = k_hop_subgraph([0, 6], 2,
        ...                                       edge_index,
        ...                                       relabel_nodes=True)
        >>> subset
        tensor([0, 1, 2, 4, 5, 6])
        >>> edge_index
        tensor([[1, 2, 3, 4],
                [0, 1, 4, 5]])
        >>> mapping
        tensor([0, 5])
        >>> edge_mask
        tensor([True, True, True, True])
        >>> subset[mapping]
        tensor([0, 6])
    """

    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    assert flow in ['source_to_target', 'target_to_source']
    if flow == 'target_to_source':
        row, col = edge_index
    else:
        col, row = edge_index

    node_mask = row.new_empty(num_nodes, dtype=torch.bool)
    edge_mask = row.new_empty(row.size(0), dtype=torch.bool)

    if isinstance(node_idx, (int, list, tuple)):
        node_idx = torch.tensor([node_idx], device=row.device).flatten()
    else:
        node_idx = node_idx.to(row.device)

    subsets = [node_idx]

    for _ in range(num_hops):
        node_mask.fill_(False)
        node_mask[subsets[-1]] = True
        torch.index_select(node_mask, 0, row, out=edge_mask)
        subsets.append(col[edge_mask])

    subset, inv = torch.cat(subsets).unique(return_inverse=True)
    inv = inv[:node_idx.numel()]

    node_mask.fill_(False)
    node_mask[subset] = True

    if not directed:
        edge_mask = node_mask[row] & node_mask[col]

    edge_index = edge_index[:, edge_mask]

    if relabel_nodes:
        node_idx = row.new_full((num_nodes, ), -1)
        node_idx[subset] = torch.arange(subset.size(0), device=row.device)
        edge_index = node_idx[edge_index]

    return subset, edge_index, inv, edge_mask
pytorch/torch_geometric/utils/to_dense_adj.py000066400000000000000000000065571444262304300220720ustar00rootroot00000000000000from typing import Optional

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import scatter


def to_dense_adj(
    edge_index: Tensor,
    batch: OptTensor = None,
    edge_attr: OptTensor = None,
    max_num_nodes: Optional[int] = None,
    batch_size: Optional[int] = None,
) -> Tensor:
    r"""Converts batched sparse adjacency matrices given by edge indices and
    edge attributes to a single dense batched adjacency matrix.

    Args:
        edge_index (LongTensor): The edge indices.
        batch (LongTensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. (default: :obj:`None`)
        edge_attr (Tensor, optional): Edge weights or multi-dimensional edge
            features. (default: :obj:`None`)
        max_num_nodes (int, optional): The size of the output node dimension.
            (default: :obj:`None`)
        batch_size (int, optional) The batch size. (default: :obj:`None`)

    :rtype: :class:`Tensor`

    Examples:

        >>> edge_index = torch.tensor([[0, 0, 1, 2, 3],
        ...                            [0, 1, 0, 3, 0]])
        >>> batch = torch.tensor([0, 0, 1, 1])
        >>> to_dense_adj(edge_index, batch)
        tensor([[[1., 1.],
                [1., 0.]],
                [[0., 1.],
                [1., 0.]]])

        >>> to_dense_adj(edge_index, batch, max_num_nodes=4)
        tensor([[[1., 1., 0., 0.],
                [1., 0., 0., 0.],
                [0., 0., 0., 0.],
                [0., 0., 0., 0.]],
                [[0., 1., 0., 0.],
                [1., 0., 0., 0.],
                [0., 0., 0., 0.],
                [0., 0., 0., 0.]]])

        >>> edge_attr = torch.Tensor([1, 2, 3, 4, 5])
        >>> to_dense_adj(edge_index, batch, edge_attr)
        tensor([[[1., 2.],
                [3., 0.]],
                [[0., 4.],
                [5., 0.]]])
    """
    if batch is None:
        num_nodes = int(edge_index.max()) + 1 if edge_index.numel() > 0 else 0
        batch = edge_index.new_zeros(num_nodes)

    if batch_size is None:
        batch_size = int(batch.max()) + 1 if batch.numel() > 0 else 1

    one = batch.new_ones(batch.size(0))
    num_nodes = scatter(one, batch, dim=0, dim_size=batch_size, reduce='sum')
    cum_nodes = torch.cat([batch.new_zeros(1), num_nodes.cumsum(dim=0)])

    idx0 = batch[edge_index[0]]
    idx1 = edge_index[0] - cum_nodes[batch][edge_index[0]]
    idx2 = edge_index[1] - cum_nodes[batch][edge_index[1]]

    if max_num_nodes is None:
        max_num_nodes = int(num_nodes.max())

    elif ((idx1.numel() > 0 and idx1.max() >= max_num_nodes)
          or (idx2.numel() > 0 and idx2.max() >= max_num_nodes)):
        mask = (idx1 < max_num_nodes) & (idx2 < max_num_nodes)
        idx0 = idx0[mask]
        idx1 = idx1[mask]
        idx2 = idx2[mask]
        edge_attr = None if edge_attr is None else edge_attr[mask]

    if edge_attr is None:
        edge_attr = torch.ones(idx0.numel(), device=edge_index.device)

    size = [batch_size, max_num_nodes, max_num_nodes]
    size += list(edge_attr.size())[1:]
    flattened_size = batch_size * max_num_nodes * max_num_nodes

    idx = idx0 * max_num_nodes * max_num_nodes + idx1 * max_num_nodes + idx2
    adj = scatter(edge_attr, idx, dim=0, dim_size=flattened_size, reduce='sum')
    adj = adj.view(size)

    return adj
pytorch/torch_geometric/utils/to_dense_batch.py000066400000000000000000000110241444262304300223760ustar00rootroot00000000000000from typing import Optional, Tuple

import torch
from torch import Tensor

from torch_geometric.experimental import (
    disable_dynamic_shapes,
    is_experimental_mode_enabled,
)
from torch_geometric.utils import scatter


@disable_dynamic_shapes(required_args=['batch_size', 'max_num_nodes'])
def to_dense_batch(
    x: Tensor,
    batch: Optional[Tensor] = None,
    fill_value: float = 0.0,
    max_num_nodes: Optional[int] = None,
    batch_size: Optional[int] = None,
) -> Tuple[Tensor, Tensor]:
    r"""Given a sparse batch of node features
    :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times F}` (with
    :math:`N_i` indicating the number of nodes in graph :math:`i`), creates a
    dense node feature tensor
    :math:`\mathbf{X} \in \mathbb{R}^{B \times N_{\max} \times F}` (with
    :math:`N_{\max} = \max_i^B N_i`).
    In addition, a mask of shape :math:`\mathbf{M} \in \{ 0, 1 \}^{B \times
    N_{\max}}` is returned, holding information about the existence of
    fake-nodes in the dense representation.

    Args:
        x (Tensor): Node feature matrix
            :math:`\mathbf{X} \in \mathbb{R}^{(N_1 + \ldots + N_B) \times F}`.
        batch (LongTensor, optional): Batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. Must be ordered. (default: :obj:`None`)
        fill_value (float, optional): The value for invalid entries in the
            resulting dense output tensor. (default: :obj:`0`)
        max_num_nodes (int, optional): The size of the output node dimension.
            (default: :obj:`None`)
        batch_size (int, optional) The batch size. (default: :obj:`None`)

    :rtype: (:class:`Tensor`, :class:`BoolTensor`)

    Examples:

        >>> x = torch.arange(12).view(6, 2)
        >>> x
        tensor([[ 0,  1],
                [ 2,  3],
                [ 4,  5],
                [ 6,  7],
                [ 8,  9],
                [10, 11]])

        >>> out, mask = to_dense_batch(x)
        >>> mask
        tensor([[True, True, True, True, True, True]])

        >>> batch = torch.tensor([0, 0, 1, 2, 2, 2])
        >>> out, mask = to_dense_batch(x, batch)
        >>> out
        tensor([[[ 0,  1],
                [ 2,  3],
                [ 0,  0]],
                [[ 4,  5],
                [ 0,  0],
                [ 0,  0]],
                [[ 6,  7],
                [ 8,  9],
                [10, 11]]])
        >>> mask
        tensor([[ True,  True, False],
                [ True, False, False],
                [ True,  True,  True]])

        >>> out, mask = to_dense_batch(x, batch, max_num_nodes=4)
        >>> out
        tensor([[[ 0,  1],
                [ 2,  3],
                [ 0,  0],
                [ 0,  0]],
                [[ 4,  5],
                [ 0,  0],
                [ 0,  0],
                [ 0,  0]],
                [[ 6,  7],
                [ 8,  9],
                [10, 11],
                [ 0,  0]]])

        >>> mask
        tensor([[ True,  True, False, False],
                [ True, False, False, False],
                [ True,  True,  True, False]])
    """
    if batch is None and max_num_nodes is None:
        mask = torch.ones(1, x.size(0), dtype=torch.bool, device=x.device)
        return x.unsqueeze(0), mask

    if batch is None:
        batch = x.new_zeros(x.size(0), dtype=torch.long)

    if batch_size is None:
        batch_size = int(batch.max()) + 1

    num_nodes = scatter(batch.new_ones(x.size(0)), batch, dim=0,
                        dim_size=batch_size, reduce='sum')
    cum_nodes = torch.cat([batch.new_zeros(1), num_nodes.cumsum(dim=0)])

    filter_nodes = False
    dynamic_shapes_disabled = is_experimental_mode_enabled(
        'disable_dynamic_shapes')

    if max_num_nodes is None:
        max_num_nodes = int(num_nodes.max())
    elif not dynamic_shapes_disabled and num_nodes.max() > max_num_nodes:
        filter_nodes = True

    tmp = torch.arange(batch.size(0), device=x.device) - cum_nodes[batch]
    idx = tmp + (batch * max_num_nodes)
    if filter_nodes:
        mask = tmp < max_num_nodes
        x, idx = x[mask], idx[mask]

    size = [batch_size * max_num_nodes] + list(x.size())[1:]
    out = torch.as_tensor(fill_value, dtype=x.dtype,
                          device=x.device).repeat(size)
    out[idx] = x
    out = out.view([batch_size, max_num_nodes] + list(x.size())[1:])

    mask = torch.zeros(batch_size * max_num_nodes, dtype=torch.bool,
                       device=x.device)
    mask[idx] = 1
    mask = mask.view(batch_size, max_num_nodes)

    return out, mask
pytorch/torch_geometric/utils/train_test_split_edges.py000066400000000000000000000066411444262304300242040ustar00rootroot00000000000000import math

import torch

import torch_geometric
from torch_geometric.deprecation import deprecated
from torch_geometric.utils import to_undirected


@deprecated("use 'transforms.RandomLinkSplit' instead")
def train_test_split_edges(
    data: 'torch_geometric.data.Data',
    val_ratio: float = 0.05,
    test_ratio: float = 0.1,
) -> 'torch_geometric.data.Data':
    r"""Splits the edges of a :class:`torch_geometric.data.Data` object
    into positive and negative train/val/test edges.
    As such, it will replace the :obj:`edge_index` attribute with
    :obj:`train_pos_edge_index`, :obj:`train_pos_neg_adj_mask`,
    :obj:`val_pos_edge_index`, :obj:`val_neg_edge_index` and
    :obj:`test_pos_edge_index` attributes.
    If :obj:`data` has edge features named :obj:`edge_attr`, then
    :obj:`train_pos_edge_attr`, :obj:`val_pos_edge_attr` and
    :obj:`test_pos_edge_attr` will be added as well.

    .. warning::

        :meth:`~torch_geometric.utils.train_test_split_edges` is deprecated and
        will be removed in a future release.
        Use :class:`torch_geometric.transforms.RandomLinkSplit` instead.

    Args:
        data (Data): The data object.
        val_ratio (float, optional): The ratio of positive validation edges.
            (default: :obj:`0.05`)
        test_ratio (float, optional): The ratio of positive test edges.
            (default: :obj:`0.1`)

    :rtype: :class:`torch_geometric.data.Data`
    """

    assert 'batch' not in data  # No batch-mode.

    num_nodes = data.num_nodes
    row, col = data.edge_index
    edge_attr = data.edge_attr
    data.edge_index = data.edge_attr = None

    # Return upper triangular portion.
    mask = row < col
    row, col = row[mask], col[mask]

    if edge_attr is not None:
        edge_attr = edge_attr[mask]

    n_v = int(math.floor(val_ratio * row.size(0)))
    n_t = int(math.floor(test_ratio * row.size(0)))

    # Positive edges.
    perm = torch.randperm(row.size(0))
    row, col = row[perm], col[perm]
    if edge_attr is not None:
        edge_attr = edge_attr[perm]

    r, c = row[:n_v], col[:n_v]
    data.val_pos_edge_index = torch.stack([r, c], dim=0)
    if edge_attr is not None:
        data.val_pos_edge_attr = edge_attr[:n_v]

    r, c = row[n_v:n_v + n_t], col[n_v:n_v + n_t]
    data.test_pos_edge_index = torch.stack([r, c], dim=0)
    if edge_attr is not None:
        data.test_pos_edge_attr = edge_attr[n_v:n_v + n_t]

    r, c = row[n_v + n_t:], col[n_v + n_t:]
    data.train_pos_edge_index = torch.stack([r, c], dim=0)
    if edge_attr is not None:
        out = to_undirected(data.train_pos_edge_index, edge_attr[n_v + n_t:])
        data.train_pos_edge_index, data.train_pos_edge_attr = out
    else:
        data.train_pos_edge_index = to_undirected(data.train_pos_edge_index)

    # Negative edges.
    neg_adj_mask = torch.ones(num_nodes, num_nodes, dtype=torch.uint8)
    neg_adj_mask = neg_adj_mask.triu(diagonal=1).to(torch.bool)
    neg_adj_mask[row, col] = 0

    neg_row, neg_col = neg_adj_mask.nonzero(as_tuple=False).t()
    perm = torch.randperm(neg_row.size(0))[:n_v + n_t]
    neg_row, neg_col = neg_row[perm], neg_col[perm]

    neg_adj_mask[neg_row, neg_col] = 0
    data.train_neg_adj_mask = neg_adj_mask

    row, col = neg_row[:n_v], neg_col[:n_v]
    data.val_neg_edge_index = torch.stack([row, col], dim=0)

    row, col = neg_row[n_v:n_v + n_t], neg_col[n_v:n_v + n_t]
    data.test_neg_edge_index = torch.stack([row, col], dim=0)

    return data
pytorch/torch_geometric/utils/tree_decomposition.py000066400000000000000000000114031444262304300233310ustar00rootroot00000000000000from itertools import chain
from typing import Any, Tuple, Union

import torch
from scipy.sparse.csgraph import minimum_spanning_tree
from torch import Tensor

from torch_geometric.utils import (
    from_scipy_sparse_matrix,
    to_scipy_sparse_matrix,
    to_undirected,
)


def tree_decomposition(
    mol: Any,
    return_vocab: bool = False,
) -> Union[Tuple[Tensor, Tensor, int], Tuple[Tensor, Tensor, int, Tensor]]:
    r"""The tree decomposition algorithm of molecules from the
    `"Junction Tree Variational Autoencoder for Molecular Graph Generation"
    `_ paper.
    Returns the graph connectivity of the junction tree, the assignment
    mapping of each atom to the clique in the junction tree, and the number
    of cliques.

    Args:
        mol (rdkit.Chem.Mol): An :obj:`rdkit` molecule.
        return_vocab (bool, optional): If set to :obj:`True`, will return an
            identifier for each clique (ring, bond, bridged compounds, single).
            (default: :obj:`False`)

    :rtype: :obj:`(LongTensor, LongTensor, int)` if :obj:`return_vocab` is
        :obj:`False`, else :obj:`(LongTensor, LongTensor, int, LongTensor)`
    """
    import rdkit.Chem as Chem

    # Cliques = rings and bonds.
    cliques = [list(x) for x in Chem.GetSymmSSSR(mol)]
    xs = [0] * len(cliques)
    for bond in mol.GetBonds():
        if not bond.IsInRing():
            cliques.append([bond.GetBeginAtomIdx(), bond.GetEndAtomIdx()])
            xs.append(1)

    # Generate `atom2clique` mappings.
    atom2clique = [[] for i in range(mol.GetNumAtoms())]
    for c in range(len(cliques)):
        for atom in cliques[c]:
            atom2clique[atom].append(c)

    # Merge rings that share more than 2 atoms as they form bridged compounds.
    for c1 in range(len(cliques)):
        for atom in cliques[c1]:
            for c2 in atom2clique[atom]:
                if c1 >= c2 or len(cliques[c1]) <= 2 or len(cliques[c2]) <= 2:
                    continue
                if len(set(cliques[c1]) & set(cliques[c2])) > 2:
                    cliques[c1] = set(cliques[c1]) | set(cliques[c2])
                    xs[c1] = 2
                    cliques[c2] = []
                    xs[c2] = -1
    cliques = [c for c in cliques if len(c) > 0]
    xs = [x for x in xs if x >= 0]

    # Update `atom2clique` mappings.
    atom2clique = [[] for i in range(mol.GetNumAtoms())]
    for c in range(len(cliques)):
        for atom in cliques[c]:
            atom2clique[atom].append(c)

    # Add singleton cliques in case there are more than 2 intersecting
    # cliques. We further compute the "initial" clique graph.
    edges = {}
    for atom in range(mol.GetNumAtoms()):
        cs = atom2clique[atom]
        if len(cs) <= 1:
            continue

        # Number of bond clusters that the atom lies in.
        bonds = [c for c in cs if len(cliques[c]) == 2]
        # Number of ring clusters that the atom lies in.
        rings = [c for c in cs if len(cliques[c]) > 4]

        if len(bonds) > 2 or (len(bonds) == 2 and len(cs) > 2):
            cliques.append([atom])
            xs.append(3)
            c2 = len(cliques) - 1
            for c1 in cs:
                edges[(c1, c2)] = 1

        elif len(rings) > 2:
            cliques.append([atom])
            xs.append(3)
            c2 = len(cliques) - 1
            for c1 in cs:
                edges[(c1, c2)] = 99

        else:
            for i in range(len(cs)):
                for j in range(i + 1, len(cs)):
                    c1, c2 = cs[i], cs[j]
                    count = len(set(cliques[c1]) & set(cliques[c2]))
                    edges[(c1, c2)] = min(count, edges.get((c1, c2), 99))

    # Update `atom2clique` mappings.
    atom2clique = [[] for i in range(mol.GetNumAtoms())]
    for c in range(len(cliques)):
        for atom in cliques[c]:
            atom2clique[atom].append(c)

    if len(edges) > 0:
        edge_index_T, weight = zip(*edges.items())
        edge_index = torch.tensor(edge_index_T).t()
        inv_weight = 100 - torch.tensor(weight)
        graph = to_scipy_sparse_matrix(edge_index, inv_weight, len(cliques))
        junc_tree = minimum_spanning_tree(graph)
        edge_index, _ = from_scipy_sparse_matrix(junc_tree)
        edge_index = to_undirected(edge_index, num_nodes=len(cliques))
    else:
        edge_index = torch.empty((2, 0), dtype=torch.long)

    rows = [[i] * len(atom2clique[i]) for i in range(mol.GetNumAtoms())]
    row = torch.tensor(list(chain.from_iterable(rows)))
    col = torch.tensor(list(chain.from_iterable(atom2clique)))
    atom2clique = torch.stack([row, col], dim=0).to(torch.long)

    if return_vocab:
        vocab = torch.tensor(xs, dtype=torch.long)
        return edge_index, atom2clique, len(cliques), vocab
    else:
        return edge_index, atom2clique, len(cliques)
pytorch/torch_geometric/utils/trim_to_layer.py000066400000000000000000000151121444262304300223100ustar00rootroot00000000000000from typing import Dict, List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import (
    Adj,
    EdgeType,
    MaybeHeteroEdgeTensor,
    MaybeHeteroNodeTensor,
    NodeType,
    SparseStorage,
    SparseTensor,
)


def trim_to_layer(
    layer: int,
    num_sampled_nodes_per_hop: Union[List[int], Dict[NodeType, List[int]]],
    num_sampled_edges_per_hop: Union[List[int], Dict[EdgeType, List[int]]],
    x: MaybeHeteroNodeTensor,
    edge_index: MaybeHeteroEdgeTensor,
    edge_attr: Optional[MaybeHeteroEdgeTensor] = None,
) -> Tuple[MaybeHeteroNodeTensor, MaybeHeteroEdgeTensor,
           Optional[MaybeHeteroEdgeTensor]]:
    r"""Trims the :obj:`edge_index` representation, node features :obj:`x` and
    edge features :obj:`edge_attr` to a minimal-sized representation for the
    current GNN layer :obj:`layer` in directed
    :class:`~torch_geometric.loader.NeighborLoader` scenarios.

    This ensures that no computation is performed for nodes and edges that are
    not included in the current GNN layer, thus avoiding unnecessary
    computation within the GNN when performing neighborhood sampling.

    Args:
        layer (int): The current GNN layer.
        num_sampled_nodes_per_hop (List[int] or Dict[NodeType, List[int]]): The
            number of sampled nodes per hop.
        num_sampled_edges_per_hop (List[int] or Dict[EdgeType, List[int]]): The
            number of sampled edges per hop.
        x (torch.Tensor or Dict[NodeType, torch.Tensor]): The homogeneous or
            heterogeneous (hidden) node features.
        edge_index (torch.Tensor or Dict[EdgeType, torch.Tensor]): The
            homogeneous or heterogeneous edge indices.
        edge_attr (torch.Tensor or Dict[EdgeType, torch.Tensor], optional): The
            homogeneous or heterogeneous (hidden) edge features.
    """
    if layer <= 0:
        return x, edge_index, edge_attr

    if isinstance(num_sampled_edges_per_hop, dict):
        x = {
            k: trim_feat(v, layer, num_sampled_nodes_per_hop[k])
            for k, v in x.items()
        }
        edge_index = {
            k: trim_adj(
                v,
                layer,
                num_sampled_nodes_per_hop[k[0]],
                num_sampled_nodes_per_hop[k[-1]],
                num_sampled_edges_per_hop[k],
            )
            for k, v in edge_index.items()
        }
        if edge_attr is not None:
            edge_attr = {
                k: trim_feat(v, layer, num_sampled_edges_per_hop[k])
                for k, v in edge_attr.items()
            }
        return x, edge_index, edge_attr

    x = trim_feat(x, layer, num_sampled_nodes_per_hop)
    edge_index = trim_adj(
        edge_index,
        layer,
        num_sampled_nodes_per_hop,
        num_sampled_nodes_per_hop,
        num_sampled_edges_per_hop,
    )

    if edge_attr is not None:
        edge_attr = trim_feat(edge_attr, layer, num_sampled_edges_per_hop)

    return x, edge_index, edge_attr


class TrimToLayer(torch.nn.Module):
    def forward(
        self,
        layer: int,
        num_sampled_nodes_per_hop: Optional[List[int]],
        num_sampled_edges_per_hop: Optional[List[int]],
        x: Tensor,
        edge_index: Adj,
        edge_attr: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Tensor, Optional[Tensor]]:

        if (not isinstance(num_sampled_nodes_per_hop, list)
                and isinstance(num_sampled_edges_per_hop, list)):
            raise ValueError("'num_sampled_nodes_per_hop' needs to be given")
        if (not isinstance(num_sampled_edges_per_hop, list)
                and isinstance(num_sampled_nodes_per_hop, list)):
            raise ValueError("'num_sampled_edges_per_hop' needs to be given")

        if num_sampled_nodes_per_hop is None:
            return x, edge_index, edge_attr
        if num_sampled_edges_per_hop is None:
            return x, edge_index, edge_attr

        return trim_to_layer(
            layer,
            num_sampled_nodes_per_hop,
            num_sampled_edges_per_hop,
            x,
            edge_index,
            edge_attr,
        )


# Helper functions ############################################################


def trim_feat(x: Tensor, layer: int, num_samples_per_hop: List[int]) -> Tensor:
    if layer <= 0:
        return x

    return x.narrow(
        dim=0,
        start=0,
        length=x.size(0) - num_samples_per_hop[-layer],
    )


def trim_adj(
    edge_index: Adj,
    layer: int,
    num_sampled_src_nodes_per_hop: List[int],
    num_sampled_dst_nodes_per_hop: List[int],
    num_sampled_edges_per_hop: List[int],
) -> Adj:

    if layer <= 0:
        return edge_index

    if isinstance(edge_index, Tensor):
        return edge_index.narrow(
            dim=1,
            start=0,
            length=edge_index.size(1) - num_sampled_edges_per_hop[-layer],
        )

    elif isinstance(edge_index, SparseTensor):
        size = (
            edge_index.size(0) - num_sampled_dst_nodes_per_hop[-layer],
            edge_index.size(1) - num_sampled_src_nodes_per_hop[-layer],
        )

        num_seed_nodes = size[0] - num_sampled_dst_nodes_per_hop[-(layer + 1)]

        return trim_sparse_tensor(edge_index, size, num_seed_nodes)

    raise ValueError(f"Unsupported 'edge_index' type '{type(edge_index)}'")


def trim_sparse_tensor(src: SparseTensor, size: Tuple[int, int],
                       num_seed_nodes: int) -> SparseTensor:
    r"""Trims a :class:`SparseTensor` along both dimensions to only contain
    the upper :obj:`num_nodes` in both dimensions.

    It is assumed that :class:`SparseTensor` is obtained from BFS traversing,
    starting from the nodes that have been initially selected.

    Args:
        src (SparseTensor): The sparse tensor.
        size (Tuple[int, int]): The number of source and destination nodes to
            keep.
        num_seed_nodes (int): The number of seed nodes to compute
            representations.
    """
    rowptr, col, value = src.csr()

    rowptr = torch.narrow(rowptr, 0, 0, size[0] + 1).clone()
    rowptr[num_seed_nodes + 1:] = rowptr[num_seed_nodes]

    col = torch.narrow(col, 0, 0, rowptr[-1])

    if value is not None:
        value = torch.narrow(value, 0, 0, rowptr[-1])

    csr2csc = src.storage._csr2csc
    if csr2csc is not None:
        csr2csc = csr2csc[csr2csc < len(col)]

    storage = SparseStorage(
        row=None,
        rowptr=rowptr,
        col=col,
        value=value,
        sparse_sizes=size,
        rowcount=None,
        colptr=None,
        colcount=None,
        csr2csc=csr2csc,
        csc2csr=None,
        is_sorted=True,
        trust_data=True,
    )
    return src.from_storage(storage)
pytorch/torch_geometric/utils/unbatch.py000066400000000000000000000041151444262304300210640ustar00rootroot00000000000000from typing import List

import torch
from torch import Tensor

from torch_geometric.utils import degree


def unbatch(src: Tensor, batch: Tensor, dim: int = 0) -> List[Tensor]:
    r"""Splits :obj:`src` according to a :obj:`batch` vector along dimension
    :obj:`dim`.

    Args:
        src (Tensor): The source tensor.
        batch (LongTensor): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            entry in :obj:`src` to a specific example. Must be ordered.
        dim (int, optional): The dimension along which to split the :obj:`src`
            tensor. (default: :obj:`0`)

    :rtype: :class:`List[Tensor]`

    Example:

        >>> src = torch.arange(7)
        >>> batch = torch.tensor([0, 0, 0, 1, 1, 2, 2])
        >>> unbatch(src, batch)
        (tensor([0, 1, 2]), tensor([3, 4]), tensor([5, 6]))
    """
    sizes = degree(batch, dtype=torch.long).tolist()
    return src.split(sizes, dim)


def unbatch_edge_index(edge_index: Tensor, batch: Tensor) -> List[Tensor]:
    r"""Splits the :obj:`edge_index` according to a :obj:`batch` vector.

    Args:
        edge_index (Tensor): The edge_index tensor. Must be ordered.
        batch (LongTensor): The batch vector
            :math:`\mathbf{b} \in {\{ 0, \ldots, B-1\}}^N`, which assigns each
            node to a specific example. Must be ordered.

    :rtype: :class:`List[Tensor]`

    Example:

        >>> edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 4, 5, 5, 6],
        ...                            [1, 0, 2, 1, 3, 2, 5, 4, 6, 5]])
        >>> batch = torch.tensor([0, 0, 0, 0, 1, 1, 1])
        >>> unbatch_edge_index(edge_index, batch)
        (tensor([[0, 1, 1, 2, 2, 3],
                [1, 0, 2, 1, 3, 2]]),
        tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]]))
    """
    deg = degree(batch, dtype=torch.int64)
    ptr = torch.cat([deg.new_zeros(1), deg.cumsum(dim=0)[:-1]], dim=0)

    edge_batch = batch[edge_index[0]]
    edge_index = edge_index - ptr[edge_batch]
    sizes = degree(edge_batch, dtype=torch.int64).cpu().tolist()
    return edge_index.split(sizes, dim=1)
pytorch/torch_geometric/utils/undirected.py000066400000000000000000000132121444262304300215640ustar00rootroot00000000000000from typing import List, Optional, Tuple, Union

import torch
from torch import Tensor

from torch_geometric.typing import OptTensor
from torch_geometric.utils import coalesce, sort_edge_index
from torch_geometric.utils.num_nodes import maybe_num_nodes

MISSING = '???'


@torch.jit._overload
def is_undirected(edge_index, edge_attr, num_nodes):
    # type: (Tensor, Optional[Tensor], Optional[int]) -> bool  # noqa
    pass


@torch.jit._overload
def is_undirected(edge_index, edge_attr, num_nodes):
    # type: (Tensor, List[Tensor], Optional[int]) -> bool  # noqa
    pass


def is_undirected(
    edge_index: Tensor,
    edge_attr: Union[OptTensor, List[Tensor]] = None,
    num_nodes: Optional[int] = None,
) -> bool:
    r"""Returns :obj:`True` if the graph given by :attr:`edge_index` is
    undirected.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor or List[Tensor], optional): Edge weights or multi-
            dimensional edge features.
            If given as a list, will check for equivalence in all its entries.
            (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)

    :rtype: bool

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 0],
        ...                         [1, 0, 0]])
        >>> weight = torch.tensor([0, 0, 1])
        >>> is_undirected(edge_index, weight)
        True

        >>> weight = torch.tensor([0, 1, 1])
        >>> is_undirected(edge_index, weight)
        False

    """
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    edge_attrs: List[Tensor] = []
    if isinstance(edge_attr, Tensor):
        edge_attrs.append(edge_attr)
    elif isinstance(edge_attr, (list, tuple)):
        edge_attrs = edge_attr

    edge_index1, edge_attrs1 = sort_edge_index(
        edge_index,
        edge_attrs,
        num_nodes=num_nodes,
        sort_by_row=True,
    )
    edge_index2, edge_attrs2 = sort_edge_index(
        edge_index,
        edge_attrs,
        num_nodes=num_nodes,
        sort_by_row=False,
    )

    if not torch.equal(edge_index1[0], edge_index2[1]):
        return False
    if not torch.equal(edge_index1[1], edge_index2[0]):
        return False
    for edge_attr1, edge_attr2 in zip(edge_attrs1, edge_attrs2):
        if not torch.equal(edge_attr1, edge_attr2):
            return False
    return True


@torch.jit._overload
def to_undirected(edge_index, edge_attr, num_nodes, reduce):
    # type: (Tensor, str, Optional[int], str) -> Tensor  # noqa
    pass


@torch.jit._overload
def to_undirected(edge_index, edge_attr, num_nodes, reduce):
    # type: (Tensor, Optional[Tensor], Optional[int], str) -> Tuple[Tensor, Optional[Tensor]]  # noqa
    pass


@torch.jit._overload
def to_undirected(edge_index, edge_attr, num_nodes, reduce):
    # type: (Tensor, List[Tensor], Optional[int], str) -> Tuple[Tensor, List[Tensor]]  # noqa
    pass


def to_undirected(
    edge_index: Tensor,
    edge_attr: Union[OptTensor, List[Tensor], str] = MISSING,
    num_nodes: Optional[int] = None,
    reduce: str = 'add',
) -> Union[Tensor, Tuple[OptTensor, Tensor], Tuple[Tensor, List[Tensor]]]:
    r"""Converts the graph given by :attr:`edge_index` to an undirected graph
    such that :math:`(j,i) \in \mathcal{E}` for every edge :math:`(i,j) \in
    \mathcal{E}`.

    Args:
        edge_index (LongTensor): The edge indices.
        edge_attr (Tensor or List[Tensor], optional): Edge weights or multi-
            dimensional edge features.
            If given as a list, will remove duplicates for all its entries.
            (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`edge_index`. (default: :obj:`None`)
        reduce (str, optional): The reduce operation to use for merging edge
            features (:obj:`"add"`, :obj:`"mean"`, :obj:`"min"`, :obj:`"max"`,
            :obj:`"mul"`). (default: :obj:`"add"`)

    :rtype: :class:`LongTensor` if :attr:`edge_attr` is not passed, else
        (:class:`LongTensor`, :obj:`Optional[Tensor]` or :obj:`List[Tensor]]`)

    .. warning::

        From :pyg:`PyG >= 2.3.0` onwards, this function will always return a
        tuple whenever :obj:`edge_attr` is passed as an argument (even in case
        it is set to :obj:`None`).

    Examples:

        >>> edge_index = torch.tensor([[0, 1, 1],
        ...                            [1, 0, 2]])
        >>> to_undirected(edge_index)
        tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]])

        >>> edge_index = torch.tensor([[0, 1, 1],
        ...                            [1, 0, 2]])
        >>> edge_weight = torch.tensor([1., 1., 1.])
        >>> to_undirected(edge_index, edge_weight)
        (tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]]),
        tensor([2., 2., 1., 1.]))

        >>> # Use 'mean' operation to merge edge features
        >>>  to_undirected(edge_index, edge_weight, reduce='mean')
        (tensor([[0, 1, 1, 2],
                [1, 0, 2, 1]]),
        tensor([1., 1., 1., 1.]))
    """
    # Maintain backward compatibility to `to_undirected(edge_index, num_nodes)`
    if isinstance(edge_attr, int):
        edge_attr = MISSING
        num_nodes = edge_attr

    row, col = edge_index[0], edge_index[1]
    row, col = torch.cat([row, col], dim=0), torch.cat([col, row], dim=0)
    edge_index = torch.stack([row, col], dim=0)

    if isinstance(edge_attr, Tensor):
        edge_attr = torch.cat([edge_attr, edge_attr], dim=0)
    elif isinstance(edge_attr, (list, tuple)):
        edge_attr = [torch.cat([e, e], dim=0) for e in edge_attr]

    return coalesce(edge_index, edge_attr, num_nodes, reduce)
pytorch/torch_geometric/visualization/000077500000000000000000000000001444262304300206265ustar00rootroot00000000000000pytorch/torch_geometric/visualization/__init__.py000066400000000000000000000001731444262304300227400ustar00rootroot00000000000000from .graph import visualize_graph
from .influence import influence

__all__ = [
    'visualize_graph',
    'influence',
]
pytorch/torch_geometric/visualization/graph.py000066400000000000000000000100651444262304300223030ustar00rootroot00000000000000from math import sqrt
from typing import Any, Optional

import torch
from torch import Tensor

BACKENDS = {'graphviz', 'networkx'}


def has_graphviz() -> bool:
    try:
        import graphviz
    except ImportError:
        return False

    try:
        graphviz.Digraph().pipe()
    except graphviz.backend.ExecutableNotFound:
        return False

    return True


def visualize_graph(
    edge_index: Tensor,
    edge_weight: Optional[Tensor] = None,
    path: Optional[str] = None,
    backend: Optional[str] = None,
) -> Any:
    r"""Visualizes the graph given via :obj:`edge_index` and (optional)
    :obj:`edge_weight`.

    Args:
        edge_index (torch.Tensor): The edge indices.
        edge_weight (torch.Tensor, optional): The edge weights.
        path (str, optional): The path to where the plot is saved.
            If set to :obj:`None`, will visualize the plot on-the-fly.
            (default: :obj:`None`)
        backend (str, optional): The graph drawing backend to use for
            visualization (:obj:`"graphviz"`, :obj:`"networkx"`).
            If set to :obj:`None`, will use the most appropriate
            visualization backend based on available system packages.
            (default: :obj:`None`)
    """
    if edge_weight is not None:  # Normalize edge weights.
        edge_weight = edge_weight - edge_weight.min()
        edge_weight = edge_weight / edge_weight.max()

    if edge_weight is not None:  # Discard any edges with zero edge weight:
        mask = edge_weight > 1e-7
        edge_index = edge_index[:, mask]
        edge_weight = edge_weight[mask]

    if edge_weight is None:
        edge_weight = torch.ones(edge_index.size(1))

    if backend is None:
        backend = 'graphviz' if has_graphviz() else 'networkx'

    if backend.lower() == 'networkx':
        return _visualize_graph_via_networkx(edge_index, edge_weight, path)
    elif backend.lower() == 'graphviz':
        return _visualize_graph_via_graphviz(edge_index, edge_weight, path)

    raise ValueError(f"Expected graph drawing backend to be in "
                     f"{BACKENDS} (got '{backend}')")


def _visualize_graph_via_graphviz(
    edge_index: Tensor,
    edge_weight: Tensor,
    path: Optional[str] = None,
) -> Any:
    import graphviz

    suffix = path.split('.')[-1] if path is not None else None
    g = graphviz.Digraph('graph', format=suffix)
    g.attr('node', shape='circle', fontsize='11pt')

    for node in edge_index.view(-1).unique().tolist():
        g.node(str(node))

    for (src, dst), w in zip(edge_index.t().tolist(), edge_weight.tolist()):
        hex_color = hex(255 - round(255 * w))[2:]
        hex_color = f'{hex_color}0' if len(hex_color) == 1 else hex_color
        g.edge(str(src), str(dst), color=f'#{hex_color}{hex_color}{hex_color}')

    if path is not None:
        path = '.'.join(path.split('.')[:-1])
        g.render(path, cleanup=True)
    else:
        g.view()

    return g


def _visualize_graph_via_networkx(
    edge_index: Tensor,
    edge_weight: Tensor,
    path: Optional[str] = None,
) -> Any:
    import matplotlib.pyplot as plt
    import networkx as nx

    g = nx.DiGraph()
    node_size = 800

    for node in edge_index.view(-1).unique().tolist():
        g.add_node(node)

    for (src, dst), w in zip(edge_index.t().tolist(), edge_weight.tolist()):
        g.add_edge(src, dst, alpha=w)

    ax = plt.gca()
    pos = nx.spring_layout(g)
    for src, dst, data in g.edges(data=True):
        ax.annotate(
            '',
            xy=pos[src],
            xytext=pos[dst],
            arrowprops=dict(
                arrowstyle="->",
                alpha=data['alpha'],
                shrinkA=sqrt(node_size) / 2.0,
                shrinkB=sqrt(node_size) / 2.0,
                connectionstyle="arc3,rad=0.1",
            ),
        )

    nodes = nx.draw_networkx_nodes(g, pos, node_size=node_size,
                                   node_color='white', margins=0.1)
    nodes.set_edgecolor('black')
    nx.draw_networkx_labels(g, pos, font_size=10)

    if path is not None:
        plt.savefig(path)
    else:
        plt.show()

    plt.close()
pytorch/torch_geometric/visualization/influence.py000066400000000000000000000006041444262304300231500ustar00rootroot00000000000000import torch
from torch.autograd import grad


def influence(model, src, *args):
    x = src.clone().requires_grad_()
    out = model(x, *args).sum(dim=-1)

    influences = []
    for j in range(src.size(0)):
        influence = grad([out[j]], [x], retain_graph=True)[0].abs().sum(dim=-1)
        influences.append(influence / influence.sum())

    return torch.stack(influences, dim=0)