pax_global_header00006660000000000000000000000064147535740150014525gustar00rootroot0000000000000052 comment=adf1e7e9124825dc4303f3ef2383f27a1dac4d0b clip-vip/000077500000000000000000000000001475357401500126145ustar00rootroot00000000000000clip-vip/.gitignore000066400000000000000000000034261475357401500146110ustar00rootroot00000000000000# ---> Python # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ clip-vip/README.md000066400000000000000000000000731475357401500140730ustar00rootroot00000000000000# CLIP-VIP 使用昇腾平台部署CLIP-VIP，简单尝试clip-vip/XPretrain-main/000077500000000000000000000000001475357401500154525ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/000077500000000000000000000000001475357401500167355ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/LICENSE000066400000000000000000000020471475357401500177450ustar00rootroot00000000000000MIT License Copyright (c) 2021 Jie Lei Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.clip-vip/XPretrain-main/CLIP-Vip/README.md000066400000000000000000000140051475357401500202140ustar00rootroot00000000000000# CLIP-ViP (ICLR 2023) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/clip-vip-adapting-pre-trained-image-text/video-retrieval-on-activitynet)](https://paperswithcode.com/sota/video-retrieval-on-activitynet?p=clip-vip-adapting-pre-trained-image-text) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/clip-vip-adapting-pre-trained-image-text/video-retrieval-on-didemo)](https://paperswithcode.com/sota/video-retrieval-on-didemo?p=clip-vip-adapting-pre-trained-image-text) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/clip-vip-adapting-pre-trained-image-text/video-retrieval-on-lsmdc)](https://paperswithcode.com/sota/video-retrieval-on-lsmdc?p=clip-vip-adapting-pre-trained-image-text) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/clip-vip-adapting-pre-trained-image-text/video-retrieval-on-msr-vtt-1ka)](https://paperswithcode.com/sota/video-retrieval-on-msr-vtt-1ka?p=clip-vip-adapting-pre-trained-image-text) By [Hongwei Xue](https://hellwayxue.github.io/)\*, [Yuchong Sun](https://scholar.google.com/citations?user=DuSxNqgAAAAJ&hl=en)\*, [Bei Liu](https://www.microsoft.com/en-us/research/people/libei/), [Jianlong Fu](https://www.microsoft.com/en-us/research/people/jianf/), [Ruihua Song](https://scholar.google.com/citations?user=v5LctN8AAAAJ&hl=en), [Houqiang Li](http://staff.ustc.edu.cn/~lihq/en/), [Jiebo Luo](https://www.cs.rochester.edu/u/jluo/). This repo is the official pytorch implementation of [CLIP-ViP: Adapting Image-Text Pre-training to Video-Language Representation Learning](https://arxiv.org/abs/2209.06430), accepted by [ICLR 2023](https://iclr.cc/Conferences/2023). CLIP-ViP is a video-language model which is based on a pre-trained image-text model [CLIP](https://openai.com/blog/clip/) then further pre-trained (post-pretraining) on a large-scale video-text dataset [HD-VILA-100M](https://github.com/microsoft/XPretrain/tree/main/hd-vila-100m). This repo consists of the code of CLIP-ViP model, the post-pretraining method, finetuning on text-to-video retrieval. ## Requirements We provide a Docker image for easier reproduction: `tiankaihang/azureml_docker:horovod` We use mixed-precision training hence GPUs with Tensor Cores are recommended. ## Getting Started ### General 1. Download Data. Download HD-VILA-100M and other required data following the instruction of [HD-VILA](https://github.com/microsoft/XPretrain/tree/main/hd-vila). Also download auxiliary captions from: [Azure Blob Link](https://hdvila.blob.core.windows.net/dataset/hdvila_ofa_captions_db.zip?sp=r&st=2023-03-16T04:58:26Z&se=2026-03-01T12:58:26Z&spr=https&sv=2021-12-02&sr=b&sig=EYE%2Bj11VWfQ6G5dZ8CKlOOpL3ckmmNqpAtUgBy3OGDM%3D) 2. Download pretrained models. We release the CLIP-ViP model under two settings: CLIP-ViP-B/32: [Azure Blob Link](https://hdvila.blob.core.windows.net/dataset/pretrain_clipvip_base_32.pt?sp=r&st=2023-03-16T05:02:41Z&se=2027-05-31T13:02:41Z&spr=https&sv=2021-12-02&sr=b&sig=91OEG2MuszQmr16N%2Bt%2FLnvlwY3sc9CNhbyxYT9rupw0%3D) CLIP-ViP-B/16: [Azure Blob Link](https://hdvila.blob.core.windows.net/dataset/pretrain_clipvip_base_16.pt?sp=r&st=2023-03-16T05:02:05Z&se=2026-07-31T13:02:05Z&spr=https&sv=2021-12-02&sr=b&sig=XNd7fZSsUhW7eesL3hTfYUMiAvCCN3Bys2TadXlWzFU%3D) 3. Set up the environment for running the experiments. Create a folder that stores pretrained models, all the data, and results. ```bash PATH_TO_STORAGE=/path/to/your/data/ ``` Clone this repo and launch the Docker container for running the experiments. If you want to pre-train on your own dataset, please prepare the environment with `horovod`. It is a better choice to use the pre-built docker image `tiankaihang/azureml_docker:horovod`. Or you can build from the [dockerfile](./docker/Dockerfile). We use mixed-precision training hence GPUs with Tensor Cores are recommended. ```bash # command to get into the container, docker image should be automatically pulled. cd HD-VILA source launch_container.sh $PATH_TO_STORAGE # update the transformers package pip install --upgrade transformers ``` ### Pre-training ```bash #inside the container horovodrun -np $NUM_GPUS python src/pretrain/run_pretrain.py \ --config $CONFIG_PATH ``` `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs/pretrain](src/configs/pretrain). Currently, `pretrain_vip_base_32.json` and `pretrain_vip_base_16.json` are supported ### Text-to-Video Retrieval Finetuning ```bash # inside the container horovodrun -np $NUM_GPUS python src/tasks/run_video_retrieval.py \ --config $CONFIG_PATH ``` `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs](src/configs) postfixed with `_retrieval.json`. For example, you can use `src/configs/msrvtt_retrieval/msrvtt_retrieval_vip_base_32.json` for finetuning CLIP-ViP-B/32 on MSRVTT retrieval. For model, currently, `pretrain_vip_base_32.json` and `pretrain_vip_base_16.json` are supported. For dataset, MSR-VTT, DiDemo, LSMDC, ActivityNet Captions are supported. ## Citation If you find the code and pre-trained models useful for your research, please consider citing our paper: ``` @article{xue2022clip, title={CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment}, author={Xue, Hongwei and Sun, Yuchong and Liu, Bei and Fu, Jianlong and Song, Ruihua and Li, Houqiang and Luo, Jiebo}, journal={arXiv preprint arXiv:2209.06430}, year={2022} } @inproceedings{xue2022advancing, title={Advancing high-resolution video-language representation with large-scale video transcriptions}, author={Xue, Hongwei and Hang, Tiankai and Zeng, Yanhong and Sun, Yuchong and Liu, Bei and Yang, Huan and Fu, Jianlong and Guo, Baining}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={5036--5045}, year={2022} } ``` ## Acknowledgements The code is based on [HD-VILA](https://github.com/microsoft/XPretrain/tree/main/hd-vila). clip-vip/XPretrain-main/CLIP-Vip/docker/000077500000000000000000000000001475357401500202045ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/docker/Dockerfile000066400000000000000000000111431475357401500221760ustar00rootroot00000000000000FROM nvidia/cuda:10.1-devel-ubuntu18.04 # TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully ENV CUDNN_VERSION=7.6.5.32-1+cuda10.1 ENV NCCL_VERSION=2.7.8-1+cuda10.1 ENV PYSPARK_PACKAGE=pyspark==2.4.7 ENV SPARK_PACKAGE=spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz # Python 3.7 is supported by Ubuntu Bionic out of the box ARG python=3.7 ENV PYTHON_VERSION=${python} # Set default shell to /bin/bash SHELL ["/bin/bash", "-cu"] RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \ build-essential \ cmake \ g++-7 \ git \ curl \ vim \ wget \ ca-certificates \ libcudnn7=${CUDNN_VERSION} \ libnccl2=${NCCL_VERSION} \ libnccl-dev=${NCCL_VERSION} \ libjpeg-dev \ libpng-dev \ python${PYTHON_VERSION} \ python${PYTHON_VERSION}-dev \ python${PYTHON_VERSION}-distutils \ librdmacm1 \ libibverbs1 \ ibverbs-providers RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python RUN curl -O https://bootstrap.pypa.io/get-pip.py && \ python get-pip.py && \ rm get-pip.py # Install TensorFlow, Keras, PyTorch and MXNet RUN pip install future typing packaging RUN pip install torch==1.8.0+cu101 torchvision==0.9.0+cu101 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html RUN git clone https://github.com/NVIDIA/apex.git &&\ cd apex &&\ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" . &&\ rm -rf ../apex # Install Spark stand-alone cluster. RUN wget --progress=dot:giga "https://www.apache.org/dyn/closer.lua/spark/${SPARK_PACKAGE}?action=download" -O - | tar -xzC /tmp; \ archive=$(basename "${SPARK_PACKAGE}") bash -c "mv -v /tmp/\${archive/%.tgz/} /spark" # Install PySpark. RUN apt-get update -qq && apt install -y openjdk-8-jdk-headless RUN pip install ${PYSPARK_PACKAGE} # Install Open MPI RUN mkdir /tmp/openmpi && \ cd /tmp/openmpi && \ wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-4.0.0.tar.gz && \ tar zxf openmpi-4.0.0.tar.gz && \ cd openmpi-4.0.0 && \ ./configure --enable-orterun-prefix-by-default && \ make -j $(nproc) all && \ make install && \ ldconfig && \ rm -rf /tmp/openmpi # Install Horovod, temporarily using CUDA stubs RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \ HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_PYTORCH=1 \ pip install --no-cache-dir horovod[pytorch] && \ ldconfig # Install OpenSSH for MPI to communicate between containers RUN apt-get install -y --no-install-recommends openssh-client openssh-server && \ mkdir -p /var/run/sshd # Allow OpenSSH to talk to containers without asking for confirmation RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \ echo " StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \ mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config # Download examples RUN apt-get install -y --no-install-recommends subversion && \ svn checkout https://github.com/horovod/horovod/trunk/examples && \ rm -rf /examples/.svn # RUN pip install decord # mmcv RUN pip install mmcv-full==1.3.10 -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.8.0/index.html RUN pip install lpips RUN pip install ujson RUN pip install einops RUN pip install transformers==2.11.0 easydict tensorboardX==2.0 av==8.0.2 lz4==2.1.9 lmdb==0.97 RUN apt install -y libgl1-mesa-glx RUN apt install -y libglib2.0-0 # WORKDIR "/examples" # RUN apt-get install -y software-properties-common --fix-missing # RUN add-apt-repository ppa:jonathonf/ffmpeg-4 # RUN apt-get update RUN apt-get install -y build-essential python3-dev python3-setuptools make cmake RUN apt-get install -y ffmpeg libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev RUN git clone https://github.com/TiankaiHang/FileStorage.git RUN git clone --recursive https://github.com/dmlc/decord WORKDIR /decord RUN mkdir build RUN cp /FileStorage/video-lib-related/libnvcuvid.so.1 /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1 RUN ln -s /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1 /usr/local/cuda/lib64/libnvcuvid.so WORKDIR /decord/build RUN cmake .. -DUSE_CUDA=ON -DCMAKE_BUILD_TYPE=Release RUN make RUN cd ../python && python setup.py install --user RUN pip install --extra-index-url https://developer.download.nvidia.com/compute/redist --upgrade nvidia-dali-cuda100 RUN echo "FINISHING!!" clip-vip/XPretrain-main/CLIP-Vip/launch_container.sh000066400000000000000000000007241475357401500226100ustar00rootroot00000000000000DATA_DIR=$1 if [ -z $CUDA_VISIBLE_DEVICES ]; then CUDA_VISIBLE_DEVICES='all' fi docker run --gpus '"'device=$CUDA_VISIBLE_DEVICES'"' --ipc=host --rm -it \ --mount src=$(pwd),dst=/VidCLIP,type=bind \ --mount src=$DATA_DIR,dst=/blob_mount,type=bind \ -e NVIDIA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \ -w /VidCLIP tiankaihang/azureml_docker:horovod \ bash -c "source /VidCLIP/setup.sh && export OMPI_MCA_btl_vader_single_copy_mechanism=none && bash" clip-vip/XPretrain-main/CLIP-Vip/setup.sh000066400000000000000000000002451475357401500204320ustar00rootroot00000000000000#!/usr/bin/env bash # source setup.sh export DIR_PWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" export PYTHONPATH="$PYTHONPATH:$DIR_PWD" echo $PYTHONPATH clip-vip/XPretrain-main/CLIP-Vip/src/000077500000000000000000000000001475357401500175245ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/_init_.py000066400000000000000000000000001475357401500213250ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs /000077500000000000000000000000001475357401500212145ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs / actnet_retrieval/000077500000000000000000000000001475357401500246075ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs / actnet_retrieval/actnet_retrieval_vip_base_16.json000066400000000000000000000041001475357401500332060ustar00rootroot00000000000000{ "train_datasets": { "name": "actnet-train", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/train.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" }, "val_datasets": [ { "name": "actnet-test", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/val1.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" } ], "inference_datasets": [ { "name": "actnet-test", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/val1.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" } ], "train_n_clips": 1, "train_num_frms": 32, "test_n_clips": 1, "test_num_frms": 32, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 70, "e2e_weights_path": "path/to/CLIP-ViP-B/16/checkpoint", "clip_weights": "openai/clip-vit-base-patch16", "clip_config": "openai/clip-vit-base-patch16", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 8, "test_batch_size": 8, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 20, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/actnet_retrieval/actnet_retrieval_vip_base_16", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs / actnet_retrieval/actnet_retrieval_vip_base_32.json000066400000000000000000000041021475357401500332060ustar00rootroot00000000000000{ "train_datasets": { "name": "actnet-train", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/train.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" }, "val_datasets": [ { "name": "actnet-test", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/val1.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" } ], "inference_datasets": [ { "name": "actnet-test", "vis_format": "frame", "txt": "clip_data/vis_db/anet_retrieval/val1.jsonl", "vis": "datasets/activitynet/ActivityNetVideoData2020Nov/video_frames_lr" } ], "train_n_clips": 1, "train_num_frms": 32, "test_n_clips": 1, "test_num_frms": 32, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 70, "e2e_weights_path": "path/to/CLIP-ViP-B/32/checkpoint", "clip_weights": "openai/clip-vit-base-patch32", "clip_config": "openai/clip-vit-base-patch32", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 20, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/actnet_retrieval/actnet_retrieval_vip_base_32", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs / didemo_retrieval/000077500000000000000000000000001475357401500245725ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs / didemo_retrieval/didemo_retrieval_vip_base_16.json000066400000000000000000000037701475357401500331700ustar00rootroot00000000000000{ "train_datasets": { "name": "didemo-train", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/train.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" }, "val_datasets": [ { "name": "didemo-val", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/val.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" } ], "inference_datasets": [ { "name": "didemo-test", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/test.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 70, "e2e_weights_path": "path/to/CLIP-ViP-B/16/checkpoint", "clip_weights": "openai/clip-vit-base-patch16", "clip_config": "openai/clip-vit-base-patch16", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 10, "lr_mul_prefix": "logit_scale", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 20, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/didemo_retrieval/didemo_retrieval_vip_base_16", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs / didemo_retrieval/didemo_retrieval_vip_base_32.json000066400000000000000000000037541475357401500331700ustar00rootroot00000000000000{ "train_datasets": { "name": "didemo-train", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/train.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" }, "val_datasets": [ { "name": "didemo-val", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/val.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" } ], "inference_datasets": [ { "name": "didemo-test", "vis_format": "video", "txt": "datasets/lfvideo_data/task/didemo/test.jsonl", "vis": "datasets/didemo/didemo_video_xfps/" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 50, "e2e_weights_path": "path/to/CLIP-ViP-B/32/checkpoint", "clip_weights": "openai/clip-vit-base-patch32", "clip_config": "openai/clip-vit-base-patch32", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 20, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/didemo_retrieval/didemo_retrieval_vip_base_32", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs / lsmdc_retrieval/000077500000000000000000000000001475357401500244335ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs / lsmdc_retrieval/lsmdc_retrieval_vip_base_16.json000066400000000000000000000036441475357401500326720ustar00rootroot00000000000000{ "train_datasets": { "name": "lsmdc-101k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/train_101k_frame.jsonl", "vis": "datasets/lsmdc" }, "val_datasets": [ { "name": "lsmdc-1k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/test_1k_frame.jsonl", "vis": "datasets/lsmdc" } ], "inference_datasets": [ { "name": "lsmdc-1k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/test_1k_frame.jsonl", "vis": "datasets/lsmdc" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 50, "e2e_weights_path": "path/to/CLIP-ViP-B/16/checkpoint", "clip_weights": "openai/clip-vit-base-patch16", "clip_config": "openai/clip-vit-base-patch16", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 10, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/lsmdc_retrieval/lsmdc_retrieval_vip_base_16", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs / lsmdc_retrieval/lsmdc_retrieval_vip_base_32.json000066400000000000000000000036441475357401500326700ustar00rootroot00000000000000{ "train_datasets": { "name": "lsmdc-101k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/train_101k_frame.jsonl", "vis": "datasets/lsmdc" }, "val_datasets": [ { "name": "lsmdc-1k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/test_1k_frame.jsonl", "vis": "datasets/lsmdc" } ], "inference_datasets": [ { "name": "lsmdc-1k", "vis_format": "video", "txt": "clip_data/vis_db/lsmdc/test_1k_frame.jsonl", "vis": "datasets/lsmdc" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 50, "e2e_weights_path": "path/to/CLIP-ViP-B/32/checkpoint", "clip_weights": "openai/clip-vit-base-patch32", "clip_config": "openai/clip-vit-base-patch32", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 10, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/lsmdc_retrieval/lsmdc_retrieval_vip_base_32", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs /config.py000066400000000000000000000500221475357401500230320ustar00rootroot00000000000000""" Modified from UNITER code """ import os import sys import json import argparse from easydict import EasyDict as edict def parse_with_config(parsed_args): """This function will set args based on the input config file. (1) it only overwrites unset parameters, i.e., these parameters not set from user command line input (2) it also sets configs in the config file but declared in the parser """ # convert to EasyDict object, enabling access from attributes even for nested config # e.g., args.train_datasets[0].name args = edict(vars(parsed_args)) if args.config is not None: config_args = json.load(open(args.config)) override_keys = {arg[2:].split("=")[0] for arg in sys.argv[1:] if arg.startswith("--")} for k, v in config_args.items(): if k not in override_keys: setattr(args, k, v) del args.config return args class SharedConfigs(object): """Shared options for pre-training and downstream tasks. For each downstream task, implement a get_*_args function, see `get_pretraining_args()` Usage: >>> shared_configs = SharedConfigs() >>> pretraining_config = shared_configs.get_pretraining_args() """ def __init__(self, desc="shared config for pretraining and finetuning"): parser = argparse.ArgumentParser(description=desc) # debug parameters parser.add_argument( "--blob_mount_dir", default="/blob_mount", type=str, help="blob mount dir") parser.add_argument( "--debug", type=int, choices=[0, 1], default=0, help="debug mode, output extra info & break all loops." "0: disable, 1 enable") parser.add_argument( "--data_ratio", type=float, default=1.0, help="portion of train/val exampels to use," "e.g., overfit a small set of data") # Required parameters parser.add_argument( "--model_config", type=str, help="path to model structure config json") parser.add_argument( "--tokenizer_dir", type=str, help="path to tokenizer dir") parser.add_argument( "--output_dir", type=str, help="dir to store model checkpoints & training meta.") # data preprocessing parameters parser.add_argument( "--max_txt_len", type=int, default=20, help="max text #tokens ") parser.add_argument( "--max_img_size", type=int, default=448, help="max image longer side size, shorter side will be padded with zeros") parser.add_argument( "--img_pixel_mean", type=float, default=None, nargs=3, help="image pixel mean") parser.add_argument( "--img_pixel_std", type=float, default=None, nargs=3, help="image pixel std") parser.add_argument( "--img_input_format", type=str, default="RGB", choices=["BGR", "RGB"], help="image input format is BGR for detectron2") parser.add_argument( "--max_n_example_per_group", type=int, default=2, help="max #examples (e.g., captions) paired with each image/video in an input group." "1: each image is paired with a single sent., equivalent to sample by sent.;" "X (X>1): each image can be paired with a maximum of X sent.; X>1 can be used " "to reduce image processing time, including basic transform (resize, etc) and CNN encoding" ) # video specific parameters parser.add_argument("--fps", type=int, default=1, help="video frame rate to use") parser.add_argument("--num_frm", type=int, default=3, help="#frames to use per clip -- we first sample a clip from a video, " "then uniformly sample num_frm from the clip. The length of the clip " "will be fps * num_frm") parser.add_argument("--frm_sampling_strategy", type=str, default="rand", choices=["rand", "uniform", "start", "middle", "end"], help="see src.datasets.data_utils.extract_frames_from_video_binary for details") # MLL training settings parser.add_argument("--train_n_clips", type=int, default=3, help="#clips to sample from each video for MIL training") parser.add_argument("--score_agg_func", type=str, default="mean", choices=["mean", "max", "lse"], help="score (from multiple clips) aggregation function, lse = LogSumExp") parser.add_argument("--random_sample_clips", type=int, default=1, choices=[0, 1], help="randomly sample clips for training, otherwise use uniformly sampled clips.") # training parameters parser.add_argument( "--train_batch_size", default=128, type=int, help="Single-GPU batch size for training for Horovod.") parser.add_argument( "--val_batch_size", default=128, type=int, help="Single-GPU batch size for validation for Horovod.") parser.add_argument( "--gradient_accumulation_steps", type=int, default=1, help="#updates steps to accumulate before performing a backward/update pass." "Used to simulate larger batch size training. The simulated batch size " "is train_batch_size * gradient_accumulation_steps for a single GPU.") parser.add_argument("--learning_rate", default=5e-5, type=float, help="initial learning rate.") parser.add_argument( "--num_valid", default=20, type=int, help="Run validation X times during training and checkpoint.") parser.add_argument( "--min_valid_steps", default=100, type=int, help="minimum #steps between two validation runs") parser.add_argument( "--save_steps_ratio", default=0.01, type=float, help="save every 0.01*global steps to resume after preemption," "not used for checkpointing.") parser.add_argument("--num_train_epochs", default=10, type=int, help="Total #training epochs.") parser.add_argument("--optim", default="adamw", choices=["adam", "adamax", "adamw"], help="optimizer") parser.add_argument("--betas", default=[0.9, 0.98], nargs=2, help="beta for adam optimizer") parser.add_argument("--decay", default="linear", choices=["linear", "invsqrt"], help="learning rate decay method") parser.add_argument("--dropout", default=0.1, type=float, help="tune dropout regularization") parser.add_argument("--weight_decay", default=1e-3, type=float, help="weight decay (L2) regularization") parser.add_argument("--grad_norm", default=2.0, type=float, help="gradient clipping (-1 for no clipping)") parser.add_argument( "--warmup_ratio", default=0.1, type=float, help="to perform linear learning rate warmup for. (invsqrt decay)") parser.add_argument("--transformer_lr_mul", default=1.0, type=float, help="lr_mul for transformer") parser.add_argument( "--transformer_lr_mul_prefix", default="", type=str, help="lr_mul param prefix for transformer") parser.add_argument("--step_decay_epochs", type=int, nargs="+", help="cnn multi_step decay epochs") # CNN parameters parser.add_argument("--cnn_optim", default="adamw", type=str, choices=["adam", "adamax", "adamw", "sgd"], help="optimizer for CNN") parser.add_argument("--cnn_learning_rate", default=5e-5, type=float, help="learning rate for CNN") parser.add_argument("--cnn_weight_decay", default=1e-3, type=float, help="weight decay for CNN") parser.add_argument("--cnn_sgd_momentum", default=0.9, type=float, help="momentum for SGD") parser.add_argument("--cnn_lr_mul", default=1.0, type=float, help="lr_mul for CNN") parser.add_argument( "--cnn_lr_mul_prefix", default="grid_encoder", type=str, help="lr_mul param prefix for CNN") parser.add_argument("--cnn_lr_decay", default="linear", choices=["linear", "invsqrt", "multi_step", "constant"], help="learning rate decay method") parser.add_argument("--cnn_step_decay_epochs", type=int, nargs="+", help="cnn multi_step decay epochs") parser.add_argument( "--freeze_cnn", default=0, choices=[0, 1], type=int, help="freeze CNN by setting the requires_grad=False for CNN parameters.") # model arch parser.add_argument( "--detectron2_model_cfg", type=str, default="", help="path to detectron2 model cfg yaml") # checkpoint parser.add_argument("--e2e_weights_path", type=str, help="path to e2e model weights") parser.add_argument("--detectron2_weights_path", type=str, help="path to detectron2 weights, only use for pretraining") parser.add_argument("--bert_weights_path", type=str, help="path to BERT weights, only use for pretraining") # inference only, please include substring `inference' # in the option to avoid been overwrite by loaded options, # see start_inference() in run_vqa_w_hvd.py parser.add_argument("--inference_model_step", default=-1, help="pretrained model checkpoint step") parser.add_argument( "--do_inference", default=0, type=int, choices=[0, 1], help="perform inference run. 0: disable, 1 enable") parser.add_argument( "--inference_split", default="val", help="For val, the data should have ground-truth associated it." "For test*, the data comes with no ground-truth.") parser.add_argument("--inference_txt_db", type=str, help="path to txt_db file for inference") parser.add_argument("--inference_img_db", type=str, help="path to img_db file for inference") parser.add_argument("--inference_batch_size", type=int, default=64, help="single-GPU batch size for inference") parser.add_argument("--inference_n_clips", type=int, default=1, help="uniformly sample `ensemble_n_clips` clips, " "each contains `num_frm` frames. When it == 1, " "use the frm_sampling_strategy to sample num_frm frames." "When it > 1, ignore frm_sampling_strategy, " "uniformly sample N clips, each clips num_frm frames.") # device parameters parser.add_argument("--seed", type=int, default=42, help="random seed for initialization") parser.add_argument( "--fp16", type=int, choices=[0, 1], default=0, help="Use 16-bit float precision instead of 32-bit." "0: disable, 1 enable") parser.add_argument("--amp_level", type=str, default='O2', help="Apex Level: O2, O1, O0, O3") parser.add_argument("--n_workers", type=int, default=4, help="#workers for data loading") parser.add_argument("--pin_mem", type=int, choices=[0, 1], default=1, help="pin memory. 0: disable, 1 enable") parser.add_argument("--img_aug", type=int, choices=[0, 1], default=0, help="image augmentation. 0: disable, 1 enable") parser.add_argument('--local_rank', type=int, default=-1, help='local rank passed from distributed launcher') parser.add_argument('--n_gpu', type=int, default=2, help='local rank passed from distributed launcher') parser.add_argument('--deepspeed_config', type=str, default="src/configs/deepspeed_test.json", help='local rank passed from distributed launcher') # can use config files, will only overwrite unset parameters parser.add_argument("--config", help="JSON config files") self.parser = parser def parse_args(self): parsed_args = self.parser.parse_args() args = parse_with_config(parsed_args) # convert to all [0, 1] options to bool, including these task specific ones zero_one_options = [ "fp16", "pin_mem", "use_itm", "use_mlm", "debug", "freeze_cnn", "do_inference", ] for option in zero_one_options: if hasattr(args, option): setattr(args, option, bool(getattr(args, option))) # basic checks # This is handled at TrainingRestorer # if exists(args.output_dir) and os.listdir(args.output_dir): # raise ValueError(f"Output directory ({args.output_dir}) " # f"already exists and is not empty.") if args.cnn_step_decay_epochs and args.cnn_lr_decay != "multi_step": Warning( f"--cnn_step_decay_epochs set to {args.cnn_step_decay_epochs}" f"but will not be effective, as --cnn_lr_decay set to be {args.cnn_lr_decay}") if args.step_decay_epochs and args.decay != "multi_step": Warning( f"--step_decay_epochs epochs set to {args.step_decay_epochs}" f"but will not be effective, as --decay set to be {args.decay}") assert args.gradient_accumulation_steps >= 1, \ f"Invalid gradient_accumulation_steps parameter: {args.gradient_accumulation_steps} " assert 1 >= args.data_ratio > 0, \ f"--data_ratio should be [1.0, 0), but get {args.data_ratio}" assert args.max_img_size > 0, "max_img_size need to be set > 0" if args.score_agg_func == "lse": assert args.loss_type == "ce",\ f"lse method can only work with ce loss, not {args.loss_type}" return args def get_pretraining_args(self): self.parser.add_argument( "--itm_neg_prob", default=0.5, type=float, help="probability to make negative examples in ITM training") self.parser.add_argument( "--use_itm", type=int, choices=[0, 1], default=0, help="enable itm loss. 0: disable, 1 enable") self.parser.add_argument( "--use_mlm", type=int, choices=[0, 1], default=0, help="enable mlm loss. 0: disable, 1 enable") self.parser.add_argument( "--use_itc", type=int, choices=[0, 1], default=0, help="enable itc loss. 0: disable, 1 enable") self.parser.add_argument( "--pixel_random_sampling_size", type=int, default=0, help="use pixel_random_sampling at pre-training, " "0: disable, positive int: enable. In Pixel-BERT, it is 100") args = self.parse_args() return args def get_video_retrieval_args(self): self.parser.add_argument( "--itm_neg_size", default=1, type=int, help="#negative captions to sample for each image") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") self.parser.add_argument( "--margin", default=0.2, type=float, help="ranking loss margin") self.parser.add_argument("--loss_type", type=str, default="ce", choices=["ce", "rank"], help="loss type") self.parser.add_argument("--eval_retrieval_batch_size", type=int, default=256, help="batch size for retrieval, since each batch will only have one image, " "retrieval allows larger batch size") args = self.parse_args() args.num_labels = 1 if args.loss_type == "rank" else 2 return args def get_vqa_args(self): self.parser.add_argument("--ans2label_path", type=str, help="path to {answer: label} file") self.parser.add_argument("--loss_type", type=str, default="bce", help="loss type") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") self.parser.add_argument("--num_labels", type=int, default=3129, help="#labels/output-dim for classifier") return self.parse_args() def get_video_qa_args(self): self.parser.add_argument( "--task", type=str, choices=["action", "transition", "frameqa", "msrvtt_qa"], help="TGIF-QA tasks and MSRVTT-QA") self.parser.add_argument("--loss_type", type=str, default="ce", help="loss type, will be overwritten later") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") # for frameQA msrvtt_qa self.parser.add_argument("--ans2label_path", type=str, default=None, help="path to {answer: label} file") # manually setup config by task type args = self.parse_args() if args.max_n_example_per_group != 1: Warning(f"For TGIF-QA, most GIF is only paired with a single example, no need to" f"use max_n_example_per_group={args.max_n_example_per_group}" f"larger than 1. Automatically reset to 1.") args.max_n_example_per_group = 1 if os.path.exists(args.ans2label_path): num_answers = len(json.load(open(args.ans2label_path, "r"))) else: num_answers = 0 if args.task in ["action", "transition"]: args.num_labels = 5 args.loss_type = "ce" elif args.task == "frameqa": args.num_labels = max(num_answers, 1540) args.loss_type = "ce" elif args.task == "msrvtt_qa": args.num_labels = max(num_answers, 1500) args.loss_type = "ce" else: raise NotImplementedError return args shared_configs = SharedConfigs() clip-vip/XPretrain-main/CLIP-Vip/src/configs /msrvtt_retrieval/000077500000000000000000000000001475357401500246305ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs /msrvtt_retrieval/msrvtt_retrieval_vip_base_16.json000066400000000000000000000040371475357401500333210ustar00rootroot00000000000000{ "train_datasets": { "name": "msrvtt-9k", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/train9k.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" }, "val_datasets": [ { "name": "msrvtt-1ka", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" } ], "inference_datasets": [ { "name": "msrvtt-1ka", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 50, "e2e_weights_path": "path/to/CLIP-ViP-B/16/checkpoint", "clip_weights": "openai/clip-vit-base-patch16", "clip_config": "openai/clip-vit-base-patch16", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 100, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/msrvtt_retrieval/msrvtt_retrieval_vip_base_16", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs /msrvtt_retrieval/msrvtt_retrieval_vip_base_32.json000066400000000000000000000040371475357401500333170ustar00rootroot00000000000000{ "train_datasets": { "name": "msrvtt-9k", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/train9k.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" }, "val_datasets": [ { "name": "msrvtt-1ka", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" } ], "inference_datasets": [ { "name": "msrvtt-1ka", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 50, "e2e_weights_path": "path/to/CLIP-ViP-B/32/checkpoint", "clip_weights": "openai/clip-vit-base-patch32", "clip_config": "openai/clip-vit-base-patch32", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 1e-6, "weight_decay": 0.2, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 1.0, "num_train_epochs": 100, "min_valid_steps": 1, "num_valid": 1, "only_valid_steps": 100, "save_steps_ratio": 0.9, "output_dir": "vidclip_data/output/msrvtt_retrieval/msrvtt_retrieval_vip_base_32", "if_tb_log": 0, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs /pretrain/000077500000000000000000000000001475357401500230405ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs /pretrain/pretrain_vip_base_16.json000066400000000000000000000047021475357401500277400ustar00rootroot00000000000000{ "train_datasets": [ { "name": "hdvila", "vis_format": "videoframe", "txt": "datasets/hdvila/hdvila_subtitles_92m_db", "vis": "youtube_data/ytt180m/video_clips_3fps", "vid_cap_path": "datasets/hdvila/hdvila_captions_db", "vid_txt": "subtitle", "img_dir": "", "cap_path": "", "img_source": "", "img_ratio": 0 } ], "val_datasets": [ { "name": "msrvtt", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" }, { "name": "how2", "vis_format": "video", "txt": "clip_data/vis_db/pretrain_data/test_howto_1k.jsonl", "vis": "youtube_data/ytt180m/video_clips_3fps" }, { "name": "ours", "vis_format": "video", "txt": "clip_data/vis_db/pretrain_data/test_full_1k.jsonl", "vis": "youtube_data/ytt180m/video_clips_3fps" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 70, "e2e_weights_path": null, "clip_weights": "openai/clip-vit-base-patch16", "clip_config": "openai/clip-vit-base-patch16", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 16, "test_batch_size": 16, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 5e-6, "weight_decay": 0.05, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss_vsc_fc", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 5.0, "num_train_epochs": 5, "min_valid_steps": 1, "num_valid": 100, "only_valid_steps": 1000, "save_steps_ratio": 0.01, "output_dir": "vidclip_data/output/pretrain/pretrain_vip_base_16/", "if_tb_log": 1, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs /pretrain/pretrain_vip_base_32.json000066400000000000000000000047021475357401500277360ustar00rootroot00000000000000{ "train_datasets": [ { "name": "hdvila", "vis_format": "videoframe", "txt": "datasets/hdvila/hdvila_subtitles_92m_db", "vis": "youtube_data/ytt180m/video_clips_3fps", "vid_cap_path": "datasets/hdvila/hdvila_captions_db", "vid_txt": "subtitle", "img_dir": "", "cap_path": "", "img_source": "", "img_ratio": 0 } ], "val_datasets": [ { "name": "msrvtt", "vis_format": "video", "txt": "clip_data/vis_db/msrvtt_video_clips/test1ka.jsonl", "vis": "clip_data/vis_db/msrvtt_video_clips/videos_6fps" }, { "name": "how2", "vis_format": "video", "txt": "clip_data/vis_db/pretrain_data/test_howto_1k.jsonl", "vis": "youtube_data/ytt180m/video_clips_3fps" }, { "name": "ours", "vis_format": "video", "txt": "clip_data/vis_db/pretrain_data/test_full_1k.jsonl", "vis": "youtube_data/ytt180m/video_clips_3fps" } ], "train_n_clips": 1, "train_num_frms": 12, "test_n_clips": 1, "test_num_frms": 12, "sample_rate": 0, "sample_jitter": 1, "video_res": [240, 320], "input_res": [224, 224], "max_txt_len": 70, "e2e_weights_path": null, "clip_weights": "openai/clip-vit-base-patch32", "clip_config": "openai/clip-vit-base-patch32", "clip_vision_additional_config": { "type": "ViP", "temporal_size": 12, "if_use_temporal_embed": 1, "logit_scale_init_value": 4.60, "add_cls_num": 3 }, "train_batch_size": 32, "test_batch_size": 32, "max_n_example_per_group": 1, "gradient_accumulation_steps": 1, "n_workers": 8, "pin_mem": 1, "fp16": 1, "amp_level": "O2", "seed": 42, "optim": "adamw", "betas": [0.9, 0.98], "learning_rate": 5e-6, "weight_decay": 0.05, "lr_mul": 1, "lr_mul_prefix": "", "loss_config": { "loss_name": "NCELearnableTempLoss_vsc_fc", "if_gather": 1 }, "warmup_ratio": 0.01, "decay": "cosine", "grad_norm": 5.0, "num_train_epochs": 5, "min_valid_steps": 1, "num_valid": 100, "only_valid_steps": 1000, "save_steps_ratio": 0.01, "output_dir": "vidclip_data/output/pretrain/pretrain_vip_base_32/", "if_tb_log": 1, "if_model_saver": 1, "if_log2file": 1, "dummy_data": 0 } clip-vip/XPretrain-main/CLIP-Vip/src/configs/000077500000000000000000000000001475357401500211545ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/configs/config.py000066400000000000000000000500221475357401500227720ustar00rootroot00000000000000""" Modified from UNITER code """ import os import sys import json import argparse from easydict import EasyDict as edict def parse_with_config(parsed_args): """This function will set args based on the input config file. (1) it only overwrites unset parameters, i.e., these parameters not set from user command line input (2) it also sets configs in the config file but declared in the parser """ # convert to EasyDict object, enabling access from attributes even for nested config # e.g., args.train_datasets[0].name args = edict(vars(parsed_args)) if args.config is not None: config_args = json.load(open(args.config)) override_keys = {arg[2:].split("=")[0] for arg in sys.argv[1:] if arg.startswith("--")} for k, v in config_args.items(): if k not in override_keys: setattr(args, k, v) del args.config return args class SharedConfigs(object): """Shared options for pre-training and downstream tasks. For each downstream task, implement a get_*_args function, see `get_pretraining_args()` Usage: >>> shared_configs = SharedConfigs() >>> pretraining_config = shared_configs.get_pretraining_args() """ def __init__(self, desc="shared config for pretraining and finetuning"): parser = argparse.ArgumentParser(description=desc) # debug parameters parser.add_argument( "--blob_mount_dir", default="/blob_mount", type=str, help="blob mount dir") parser.add_argument( "--debug", type=int, choices=[0, 1], default=0, help="debug mode, output extra info & break all loops." "0: disable, 1 enable") parser.add_argument( "--data_ratio", type=float, default=1.0, help="portion of train/val exampels to use," "e.g., overfit a small set of data") # Required parameters parser.add_argument( "--model_config", type=str, help="path to model structure config json") parser.add_argument( "--tokenizer_dir", type=str, help="path to tokenizer dir") parser.add_argument( "--output_dir", type=str, help="dir to store model checkpoints & training meta.") # data preprocessing parameters parser.add_argument( "--max_txt_len", type=int, default=20, help="max text #tokens ") parser.add_argument( "--max_img_size", type=int, default=448, help="max image longer side size, shorter side will be padded with zeros") parser.add_argument( "--img_pixel_mean", type=float, default=None, nargs=3, help="image pixel mean") parser.add_argument( "--img_pixel_std", type=float, default=None, nargs=3, help="image pixel std") parser.add_argument( "--img_input_format", type=str, default="RGB", choices=["BGR", "RGB"], help="image input format is BGR for detectron2") parser.add_argument( "--max_n_example_per_group", type=int, default=2, help="max #examples (e.g., captions) paired with each image/video in an input group." "1: each image is paired with a single sent., equivalent to sample by sent.;" "X (X>1): each image can be paired with a maximum of X sent.; X>1 can be used " "to reduce image processing time, including basic transform (resize, etc) and CNN encoding" ) # video specific parameters parser.add_argument("--fps", type=int, default=1, help="video frame rate to use") parser.add_argument("--num_frm", type=int, default=3, help="#frames to use per clip -- we first sample a clip from a video, " "then uniformly sample num_frm from the clip. The length of the clip " "will be fps * num_frm") parser.add_argument("--frm_sampling_strategy", type=str, default="rand", choices=["rand", "uniform", "start", "middle", "end"], help="see src.datasets.data_utils.extract_frames_from_video_binary for details") # MLL training settings parser.add_argument("--train_n_clips", type=int, default=3, help="#clips to sample from each video for MIL training") parser.add_argument("--score_agg_func", type=str, default="mean", choices=["mean", "max", "lse"], help="score (from multiple clips) aggregation function, lse = LogSumExp") parser.add_argument("--random_sample_clips", type=int, default=1, choices=[0, 1], help="randomly sample clips for training, otherwise use uniformly sampled clips.") # training parameters parser.add_argument( "--train_batch_size", default=128, type=int, help="Single-GPU batch size for training for Horovod.") parser.add_argument( "--val_batch_size", default=128, type=int, help="Single-GPU batch size for validation for Horovod.") parser.add_argument( "--gradient_accumulation_steps", type=int, default=1, help="#updates steps to accumulate before performing a backward/update pass." "Used to simulate larger batch size training. The simulated batch size " "is train_batch_size * gradient_accumulation_steps for a single GPU.") parser.add_argument("--learning_rate", default=5e-5, type=float, help="initial learning rate.") parser.add_argument( "--num_valid", default=20, type=int, help="Run validation X times during training and checkpoint.") parser.add_argument( "--min_valid_steps", default=100, type=int, help="minimum #steps between two validation runs") parser.add_argument( "--save_steps_ratio", default=0.01, type=float, help="save every 0.01*global steps to resume after preemption," "not used for checkpointing.") parser.add_argument("--num_train_epochs", default=10, type=int, help="Total #training epochs.") parser.add_argument("--optim", default="adamw", choices=["adam", "adamax", "adamw"], help="optimizer") parser.add_argument("--betas", default=[0.9, 0.98], nargs=2, help="beta for adam optimizer") parser.add_argument("--decay", default="linear", choices=["linear", "invsqrt"], help="learning rate decay method") parser.add_argument("--dropout", default=0.1, type=float, help="tune dropout regularization") parser.add_argument("--weight_decay", default=1e-3, type=float, help="weight decay (L2) regularization") parser.add_argument("--grad_norm", default=2.0, type=float, help="gradient clipping (-1 for no clipping)") parser.add_argument( "--warmup_ratio", default=0.1, type=float, help="to perform linear learning rate warmup for. (invsqrt decay)") parser.add_argument("--transformer_lr_mul", default=1.0, type=float, help="lr_mul for transformer") parser.add_argument( "--transformer_lr_mul_prefix", default="", type=str, help="lr_mul param prefix for transformer") parser.add_argument("--step_decay_epochs", type=int, nargs="+", help="cnn multi_step decay epochs") # CNN parameters parser.add_argument("--cnn_optim", default="adamw", type=str, choices=["adam", "adamax", "adamw", "sgd"], help="optimizer for CNN") parser.add_argument("--cnn_learning_rate", default=5e-5, type=float, help="learning rate for CNN") parser.add_argument("--cnn_weight_decay", default=1e-3, type=float, help="weight decay for CNN") parser.add_argument("--cnn_sgd_momentum", default=0.9, type=float, help="momentum for SGD") parser.add_argument("--cnn_lr_mul", default=1.0, type=float, help="lr_mul for CNN") parser.add_argument( "--cnn_lr_mul_prefix", default="grid_encoder", type=str, help="lr_mul param prefix for CNN") parser.add_argument("--cnn_lr_decay", default="linear", choices=["linear", "invsqrt", "multi_step", "constant"], help="learning rate decay method") parser.add_argument("--cnn_step_decay_epochs", type=int, nargs="+", help="cnn multi_step decay epochs") parser.add_argument( "--freeze_cnn", default=0, choices=[0, 1], type=int, help="freeze CNN by setting the requires_grad=False for CNN parameters.") # model arch parser.add_argument( "--detectron2_model_cfg", type=str, default="", help="path to detectron2 model cfg yaml") # checkpoint parser.add_argument("--e2e_weights_path", type=str, help="path to e2e model weights") parser.add_argument("--detectron2_weights_path", type=str, help="path to detectron2 weights, only use for pretraining") parser.add_argument("--bert_weights_path", type=str, help="path to BERT weights, only use for pretraining") # inference only, please include substring `inference' # in the option to avoid been overwrite by loaded options, # see start_inference() in run_vqa_w_hvd.py parser.add_argument("--inference_model_step", default=-1, help="pretrained model checkpoint step") parser.add_argument( "--do_inference", default=0, type=int, choices=[0, 1], help="perform inference run. 0: disable, 1 enable") parser.add_argument( "--inference_split", default="val", help="For val, the data should have ground-truth associated it." "For test*, the data comes with no ground-truth.") parser.add_argument("--inference_txt_db", type=str, help="path to txt_db file for inference") parser.add_argument("--inference_img_db", type=str, help="path to img_db file for inference") parser.add_argument("--inference_batch_size", type=int, default=64, help="single-GPU batch size for inference") parser.add_argument("--inference_n_clips", type=int, default=1, help="uniformly sample `ensemble_n_clips` clips, " "each contains `num_frm` frames. When it == 1, " "use the frm_sampling_strategy to sample num_frm frames." "When it > 1, ignore frm_sampling_strategy, " "uniformly sample N clips, each clips num_frm frames.") # device parameters parser.add_argument("--seed", type=int, default=42, help="random seed for initialization") parser.add_argument( "--fp16", type=int, choices=[0, 1], default=0, help="Use 16-bit float precision instead of 32-bit." "0: disable, 1 enable") parser.add_argument("--amp_level", type=str, default='O2', help="Apex Level: O2, O1, O0, O3") parser.add_argument("--n_workers", type=int, default=4, help="#workers for data loading") parser.add_argument("--pin_mem", type=int, choices=[0, 1], default=1, help="pin memory. 0: disable, 1 enable") parser.add_argument("--img_aug", type=int, choices=[0, 1], default=0, help="image augmentation. 0: disable, 1 enable") parser.add_argument('--local_rank', type=int, default=-1, help='local rank passed from distributed launcher') parser.add_argument('--n_gpu', type=int, default=2, help='local rank passed from distributed launcher') parser.add_argument('--deepspeed_config', type=str, default="src/configs/deepspeed_test.json", help='local rank passed from distributed launcher') # can use config files, will only overwrite unset parameters parser.add_argument("--config", help="JSON config files") self.parser = parser def parse_args(self): parsed_args = self.parser.parse_args() args = parse_with_config(parsed_args) # convert to all [0, 1] options to bool, including these task specific ones zero_one_options = [ "fp16", "pin_mem", "use_itm", "use_mlm", "debug", "freeze_cnn", "do_inference", ] for option in zero_one_options: if hasattr(args, option): setattr(args, option, bool(getattr(args, option))) # basic checks # This is handled at TrainingRestorer # if exists(args.output_dir) and os.listdir(args.output_dir): # raise ValueError(f"Output directory ({args.output_dir}) " # f"already exists and is not empty.") if args.cnn_step_decay_epochs and args.cnn_lr_decay != "multi_step": Warning( f"--cnn_step_decay_epochs set to {args.cnn_step_decay_epochs}" f"but will not be effective, as --cnn_lr_decay set to be {args.cnn_lr_decay}") if args.step_decay_epochs and args.decay != "multi_step": Warning( f"--step_decay_epochs epochs set to {args.step_decay_epochs}" f"but will not be effective, as --decay set to be {args.decay}") assert args.gradient_accumulation_steps >= 1, \ f"Invalid gradient_accumulation_steps parameter: {args.gradient_accumulation_steps} " assert 1 >= args.data_ratio > 0, \ f"--data_ratio should be [1.0, 0), but get {args.data_ratio}" assert args.max_img_size > 0, "max_img_size need to be set > 0" if args.score_agg_func == "lse": assert args.loss_type == "ce",\ f"lse method can only work with ce loss, not {args.loss_type}" return args def get_pretraining_args(self): self.parser.add_argument( "--itm_neg_prob", default=0.5, type=float, help="probability to make negative examples in ITM training") self.parser.add_argument( "--use_itm", type=int, choices=[0, 1], default=0, help="enable itm loss. 0: disable, 1 enable") self.parser.add_argument( "--use_mlm", type=int, choices=[0, 1], default=0, help="enable mlm loss. 0: disable, 1 enable") self.parser.add_argument( "--use_itc", type=int, choices=[0, 1], default=0, help="enable itc loss. 0: disable, 1 enable") self.parser.add_argument( "--pixel_random_sampling_size", type=int, default=0, help="use pixel_random_sampling at pre-training, " "0: disable, positive int: enable. In Pixel-BERT, it is 100") args = self.parse_args() return args def get_video_retrieval_args(self): self.parser.add_argument( "--itm_neg_size", default=1, type=int, help="#negative captions to sample for each image") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") self.parser.add_argument( "--margin", default=0.2, type=float, help="ranking loss margin") self.parser.add_argument("--loss_type", type=str, default="ce", choices=["ce", "rank"], help="loss type") self.parser.add_argument("--eval_retrieval_batch_size", type=int, default=256, help="batch size for retrieval, since each batch will only have one image, " "retrieval allows larger batch size") args = self.parse_args() args.num_labels = 1 if args.loss_type == "rank" else 2 return args def get_vqa_args(self): self.parser.add_argument("--ans2label_path", type=str, help="path to {answer: label} file") self.parser.add_argument("--loss_type", type=str, default="bce", help="loss type") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") self.parser.add_argument("--num_labels", type=int, default=3129, help="#labels/output-dim for classifier") return self.parse_args() def get_video_qa_args(self): self.parser.add_argument( "--task", type=str, choices=["action", "transition", "frameqa", "msrvtt_qa"], help="TGIF-QA tasks and MSRVTT-QA") self.parser.add_argument("--loss_type", type=str, default="ce", help="loss type, will be overwritten later") self.parser.add_argument("--classifier", type=str, default="mlp", choices=["mlp", "linear"], help="classifier type") self.parser.add_argument( "--cls_hidden_scale", type=int, default=2, help="scaler of the intermediate linear layer dimension for mlp classifier") # for frameQA msrvtt_qa self.parser.add_argument("--ans2label_path", type=str, default=None, help="path to {answer: label} file") # manually setup config by task type args = self.parse_args() if args.max_n_example_per_group != 1: Warning(f"For TGIF-QA, most GIF is only paired with a single example, no need to" f"use max_n_example_per_group={args.max_n_example_per_group}" f"larger than 1. Automatically reset to 1.") args.max_n_example_per_group = 1 if os.path.exists(args.ans2label_path): num_answers = len(json.load(open(args.ans2label_path, "r"))) else: num_answers = 0 if args.task in ["action", "transition"]: args.num_labels = 5 args.loss_type = "ce" elif args.task == "frameqa": args.num_labels = max(num_answers, 1540) args.loss_type = "ce" elif args.task == "msrvtt_qa": args.num_labels = max(num_answers, 1500) args.loss_type = "ce" else: raise NotImplementedError return args shared_configs = SharedConfigs() clip-vip/XPretrain-main/CLIP-Vip/src/datasets/000077500000000000000000000000001475357401500213345ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/datasets/data_utils.py000066400000000000000000000354741475357401500240540ustar00rootroot00000000000000import torch import random import torchvision.transforms as transforms from torchvision.transforms.functional import pad as img_pad from torchvision.transforms.functional import resize as img_resize from torch.nn.functional import interpolate as img_tensor_resize from torch.nn.functional import pad as img_tensor_pad from torch.nn.modules.utils import _quadruple from src.utils.basic_utils import flat_list_of_lists import numbers import numpy as np from PIL import Image _pil_interpolation_to_str = { Image.NEAREST: 'PIL.Image.NEAREST', Image.BILINEAR: 'PIL.Image.BILINEAR', Image.BICUBIC: 'PIL.Image.BICUBIC', Image.LANCZOS: 'PIL.Image.LANCZOS', Image.HAMMING: 'PIL.Image.HAMMING', Image.BOX: 'PIL.Image.BOX', } def mask_batch_text_tokens( inputs, tokenizer, mlm_probability=0.15, is_train=True): """ modified from transformers.data.data_collator Args: inputs: (B, L), 2D torch.Tensor, does not work for 1D. It has already been padded. tokenizer: mlm_probability: float is_train: if True use random masking, else mask tokens at fixed position to remove randomness in evaluation. """ if tokenizer.mask_token is None: raise ValueError( "This tokenizer does not have a mask token which is necessary for masked language modeling. " "Remove the --mlm flag if you want to use this tokenizer." ) labels = inputs.clone() # We sample a few tokens in each sequence for masked-LM training # (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa) probability_matrix = torch.full(labels.shape, mlm_probability) special_tokens_mask = [ tokenizer.get_special_tokens_mask( val, already_has_special_tokens=True) for val in labels.tolist() ] probability_matrix.masked_fill_(torch.tensor( special_tokens_mask, dtype=torch.bool), value=0.0) if tokenizer._pad_token is not None: padding_mask = labels.eq(tokenizer.pad_token_id) probability_matrix.masked_fill_(padding_mask, value=0.0) masked_indices = torch.bernoulli(probability_matrix).bool() labels[~masked_indices] = -100 # We only compute loss on masked tokens # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK]) indices_replaced = torch.bernoulli( torch.full(labels.shape, 0.8)).bool() & masked_indices inputs[indices_replaced] = tokenizer.convert_tokens_to_ids( tokenizer.mask_token) # 10% of the time, we replace masked input tokens with random word indices_random = torch.bernoulli( torch.full(labels.shape, 0.5) ).bool() & masked_indices & ~indices_replaced random_words = torch.randint( len(tokenizer), labels.shape, dtype=torch.long) # len(tokenizer) == #vocab inputs[indices_random] = random_words[indices_random] # The rest of the time (10% of the time) we keep the masked input tokens unchanged return inputs, labels def image_to_tensor(image: np.ndarray, keepdim: bool = True) -> torch.Tensor: """Converts a numpy image to a PyTorch 4d tensor image. Args: image (numpy.ndarray): image of the form :math:`(H, W, C)`, :math:`(H, W)` or :math:`(B, H, W, C)`. keepdim (bool): If ``False`` unsqueeze the input image to match the shape :math:`(B, H, W, C)`. Default: ``True`` Returns: torch.Tensor: tensor of the form :math:`(B, C, H, W)` if keepdim is ``False``, :math:`(C, H, W)` otherwise. """ if not isinstance(image, (np.ndarray,)): raise TypeError("Input type must be a numpy.ndarray. Got {}".format( type(image))) if len(image.shape) > 4 or len(image.shape) < 2: raise ValueError( "Input size must be a two, three or four dimensional array") input_shape = image.shape tensor: torch.Tensor = torch.from_numpy(image) if len(input_shape) == 2: # (H, W) -> (1, H, W) tensor = tensor.unsqueeze(0) elif len(input_shape) == 3: # (H, W, C) -> (C, H, W) tensor = tensor.permute(2, 0, 1) elif len(input_shape) == 4: # (B, H, W, C) -> (B, C, H, W) tensor = tensor.permute(0, 3, 1, 2) keepdim = True # no need to unsqueeze else: raise ValueError( "Cannot process image with shape {}".format(input_shape)) return tensor.unsqueeze(0) if not keepdim else tensor def get_padding(image, max_w, max_h, pad_all=False): # keep the images to upper-left corner if isinstance(image, torch.Tensor): h, w = image.shape[-2:] else: w, h = image.size h_padding, v_padding = max_w - w, max_h - h if pad_all: h_padding /= 2 v_padding /= 2 l_pad = h_padding if h_padding % 1 == 0 else h_padding+0.5 t_pad = v_padding if v_padding % 1 == 0 else v_padding+0.5 r_pad = h_padding if h_padding % 1 == 0 else h_padding-0.5 b_pad = v_padding if v_padding % 1 == 0 else v_padding-0.5 else: l_pad, t_pad = 0, 0 r_pad, b_pad = h_padding, v_padding if isinstance(image, torch.Tensor): padding = (int(l_pad), int(r_pad), int(t_pad), int(b_pad)) else: padding = (int(l_pad), int(t_pad), int(r_pad), int(b_pad)) return padding class ImagePad(object): def __init__(self, max_w, max_h, fill=0, padding_mode='constant'): assert isinstance(fill, (numbers.Number, str, tuple)) assert padding_mode in ['constant', 'edge', 'reflect', 'symmetric'] self.max_w = max_w self.max_h = max_h self.fill = fill self.padding_mode = padding_mode def __call__(self, img): """ Args: img (PIL Image): Image to be padded. Returns: PIL Image: Padded image. """ if isinstance(img, torch.Tensor): paddings = _quadruple(get_padding(img, self.max_w, self.max_h)) return img_tensor_pad( img, paddings, self.padding_mode, self.fill) return img_pad( img, get_padding(img, self.max_w, self.max_h), self.fill, self.padding_mode) def __repr__(self): return self.__class__.__name__ + '(padding={0}, fill={1}, padding_mode={2})'.\ format(self.fill, self.padding_mode) def get_resize_size(image, max_size): """ Args: image: PIL Image or torch.tensor max_size: Returns: Note the height/width order difference >>> pil_img = Image.open("raw_img_tensor.jpg") >>> pil_img.size (640, 480) # (width, height) >>> np_img = np.array(pil_img) >>> np_img.shape (480, 640, 3) # (height, width, 3) """ # note the order of height and width for different inputs if isinstance(image, torch.Tensor): # width, height = image.shape[-2:] height, width = image.shape[-2:] else: width, height = image.size if height >= width: ratio = width*1./height new_height = max_size new_width = new_height * ratio else: ratio = height*1./width new_width = max_size new_height = new_width * ratio size = (int(new_height), int(new_width)) return size class ImageResize(object): """Resize the input image (torch.tensor) to the given size. Args: max_size (int): Desired output size. If size is a sequence like (h, w), output size will be matched to this. If size is an int, smaller edge of the image will be matched to this number. i.e, if height > width, then image will be rescaled to (size * height / width, size) interpolation (int, optional): Desired interpolation. Default is ``PIL.Image.BILINEAR`` """ def __init__(self, max_size, interpolation=Image.BILINEAR): assert isinstance(max_size, int) self.max_size = max_size self.interpolation = interpolation def __call__(self, img): """ Args: img (torch.tensor): Image to be scaled. Returns: torch.tensor: Rescaled image. """ if isinstance(img, torch.Tensor): assert isinstance(self.interpolation, str) return img_tensor_resize( img, size=get_resize_size(img, self.max_size), mode=self.interpolation, align_corners=False) return img_resize( img, get_resize_size(img, self.max_size), self.interpolation) def __repr__(self): interpolate_str = _pil_interpolation_to_str[self.interpolation] return self.__class__.__name__ + '(size={0}, interpolation={1})'.format( self.size, interpolate_str) def get_imagenet_transform(min_size=600, max_size=1000): """parameters from https://github.com/pytorch/examples/blob/master/imagenet/main.py This simply crop the center square from the image """ if min_size != 600: import warnings warnings.warn(f'Warning: min_size is not used in image transform, ' f'setting min_size will have no effect.') return transforms.Compose([ ImageResize(max_size, Image.BILINEAR), # longer side will be resized to 1000 ImagePad(max_size, max_size), # pad to 1000 * 1000 ]) class ImageNorm(object): """Apply Normalization to Image Pixels on GPU """ def __init__(self, mean, std): self.mean_value = mean self.std_value = std self.mean = torch.tensor(mean).cuda().view(1, 1, 1, 3, 1, 1) self.std = torch.tensor(std).cuda().view(1, 1, 1, 3, 1, 1) # assert max(std) <= 1 and min(std) >= 0\ # or max(mean) <= 1 and min(mean) >= 0,\ # "Please provide mean or std within range [0, 1]" def __call__(self, img): """ Args: img: float image tensors, (B, N, 3, H, W) Returns: img: normalized float image tensors """ if len(img.shape)==5 and img.shape[2]==3 and len(self.mean.shape)==6: self.mean = torch.tensor(self.mean_value).cuda().view(1, 1, 3, 1, 1) self.std = torch.tensor(self.std_value).cuda().view(1, 1, 3, 1, 1) if torch.max(img) > 1 and self.mean.max() <= 1: img.div_(255.) return img.sub_(self.mean).div_(self.std) def chunk_list(examples, chunk_size=2, pad_to_divisible=True): """ Args: examples: iterable, examples grouped by image/video chunk_size: int, number of examples in each chunk. pad_to_divisible: bool, pad the examples to be divisible by chunk_size. >>> test_examples = [3, 4, 5, 6, 7] >>> chunk_list(test_examples, chunk_size=2, pad_to_divisible=True) [[3, 4], [5, 6], [7, 7]] # the lst element has some randomness >>> chunk_list(test_examples, chunk_size=2, pad_to_divisible=False) [[3, 4], [5, 6], [7]] """ n_examples = len(examples) remainder = n_examples % chunk_size if pad_to_divisible and remainder > 0: n_pad = chunk_size - remainder pad = random.choices(examples, k=n_pad) # with replacement examples = examples + pad n_examples = len(examples) remainder = 0 chunked_examples = [] n_chunks = int(n_examples / chunk_size) n_chunks = n_chunks + 1 if remainder > 0 else n_chunks for i in range(n_chunks): chunked_examples.append(examples[i*chunk_size: (i+1)*chunk_size]) return chunked_examples def mk_input_group(key_grouped_examples, max_n_example_per_group=2, is_train=True, example_unique_key=None): """ Re-organize examples into groups. Each input group will have a single image paired with X (X=max_n_example_per_img) examples. Images with total #examples > X will be split into multiple groups. In the case a group has < X examples, we will copy the examples to make the group has X examples. Args: key_grouped_examples: dict, each key is image/video id, each value is a list(example) associated with this image/video max_n_example_per_group: int, pair max #examples with each image/video. Note that each image can have multiple groups. is_train: bool, if True, copy the examples to make sure each input group has max_n_example_per_group examples. example_unique_key: str, used to make sure no inputs are discarded by matching the input and output ids specified by `example_unique_key` """ input_groups = [] # each element is (id, list(example)) for k, examples in key_grouped_examples.items(): chunked_examples = chunk_list(examples, chunk_size=max_n_example_per_group, pad_to_divisible=is_train) for c in chunked_examples: # if len(c) == 0: # continue input_groups.append((k, c)) if example_unique_key is not None: print(f"Using example_unique_key {example_unique_key} to check whether input and output ids m") # sanity check: make sure we did not discard any input example by accident. input_question_ids = flat_list_of_lists( [[sub_e[example_unique_key] for sub_e in e] for e in key_grouped_examples.values()]) output_question_ids = flat_list_of_lists( [[sub_e[example_unique_key] for sub_e in e[1]] for e in input_groups]) assert set(input_question_ids) == set(output_question_ids), "You are missing " return input_groups def repeat_tensor_rows(raw_tensor, row_repeats): """ repeat raw_tensor[i] row_repeats[i] times. Args: raw_tensor: (B, *) row_repeats: list(int), len(row_repeats) == len(raw_tensor) """ assert len(raw_tensor) == len(raw_tensor), "Has to be the same length" if sum(row_repeats) == len(row_repeats): return raw_tensor else: indices = torch.LongTensor( flat_list_of_lists([[i] * r for i, r in enumerate(row_repeats)]) ).to(raw_tensor.device) return raw_tensor.index_select(0, indices) def img_collate(imgs): """ Args: imgs: Returns: torch.tensor, (B, 3, H, W) """ w = imgs[0].width h = imgs[0].height tensor = torch.zeros( (len(imgs), 3, h, w), dtype=torch.uint8).contiguous() for i, img in enumerate(imgs): nump_array = np.array(img, dtype=np.uint8) if (nump_array.ndim < 3): nump_array = np.expand_dims(nump_array, axis=-1) # (H, W, 3) --> (3, H, W) nump_array = np.rollaxis(nump_array, 2) tensor[i] += torch.from_numpy(nump_array) return tensor clip-vip/XPretrain-main/CLIP-Vip/src/datasets/dataloader.py000066400000000000000000000235711475357401500240160ustar00rootroot00000000000000""" modified from UNITER codebase A meta data loader for sampling from different datasets / training tasks A prefetch loader to speedup data loading """ import random import torch from torchvision import transforms from torch.utils.data import DataLoader from src.utils.distributed import any_broadcast class MetaLoader(object): """ wraps multiple data loader """ def __init__(self, loaders, accum_steps=1, distributed=False, epoch=2): assert isinstance(loaders, dict) self.name2loader = {} self.name2iter = {} self.name2epoch = {} self.sampling_pools = [] n_batches_in_epoch = 0 for n, l in loaders.items(): if isinstance(l, tuple): l, r = l elif isinstance(l, DataLoader): r = 1 else: raise ValueError() n_batches_in_epoch += len(l.dataset) * r / l.batch_size self.name2loader[n] = l self.name2epoch[n] = epoch self.name2loader[n].sampler.set_epoch(self.name2epoch[n]) self.name2iter[n] = iter(l) self.sampling_pools.extend([n]*r) self.n_batches_in_epoch = n_batches_in_epoch self.accum_steps = accum_steps self.distributed = distributed self.step = 0 def __iter__(self): """ this iterator will run indefinitely """ task = self.sampling_pools[0] while True: if self.step % self.accum_steps == 0: task = random.choice(self.sampling_pools) if self.distributed: # make sure all process is training same task task = any_broadcast(task, 0) self.step += 1 iter_ = self.name2iter[task] try: batch = next(iter_) except StopIteration: self.name2epoch[task] += 1 self.name2loader[task].sampler.set_epoch(self.name2epoch[task]) iter_ = iter(self.name2loader[task]) batch = next(iter_) self.name2iter[task] = iter_ yield task, batch def move_to_cuda(batch): if isinstance(batch, torch.Tensor): return batch.cuda(non_blocking=True) elif isinstance(batch, list): new_batch = [move_to_cuda(t) for t in batch] elif isinstance(batch, tuple): new_batch = tuple(move_to_cuda(t) for t in batch) elif isinstance(batch, dict): new_batch = {n: move_to_cuda(t) for n, t in batch.items()} else: return batch return new_batch def record_cuda_stream(batch): if isinstance(batch, torch.Tensor): batch.record_stream(torch.cuda.current_stream()) elif isinstance(batch, list) or isinstance(batch, tuple): for t in batch: record_cuda_stream(t) elif isinstance(batch, dict): for t in batch.values(): record_cuda_stream(t) else: pass class PrefetchLoader(object): """ overlap compute and cuda data transfer (copied and then modified from nvidia apex) """ def __init__(self, loader, img_normalize=None): self.loader = loader self.stream = torch.cuda.Stream() self.img_normalize = img_normalize def __iter__(self): loader_it = iter(self.loader) self.preload(loader_it) batch = self.next(loader_it) while batch is not None: is_tuple = isinstance(batch, tuple) if is_tuple: task, batch = batch # batch["images"] = batch["images"].float() # if self.img_normalize is not None: # batch["images"] = self.img_normalize(batch["images"]) if is_tuple: yield task, batch else: yield batch batch = self.next(loader_it) def __len__(self): return len(self.loader) def preload(self, it): try: self.batch = next(it) except StopIteration: self.batch = None return # if record_stream() doesn't work, another option is to make sure # device inputs are created on the main stream. # self.next_input_gpu = torch.empty_like(self.next_input, # device='cuda') # self.next_target_gpu = torch.empty_like(self.next_target, # device='cuda') # Need to make sure the memory allocated for next_* is not still in use # by the main stream at the time we start copying to next_*: # self.stream.wait_stream(torch.cuda.current_stream()) with torch.cuda.stream(self.stream): self.batch = move_to_cuda(self.batch) # more code for the alternative if record_stream() doesn't work: # copy_ will record the use of the pinned source tensor in this # side stream. # self.next_input_gpu.copy_(self.next_input, non_blocking=True) # self.next_target_gpu.copy_(self.next_target, non_blocking=True) # self.next_input = self.next_input_gpu # self.next_target = self.next_target_gpu def next(self, it): torch.cuda.current_stream().wait_stream(self.stream) batch = self.batch if batch is not None: record_cuda_stream(batch) self.preload(it) return batch def __getattr__(self, name): method = self.loader.__getattribute__(name) return method class InfiniteIterator(object): """iterate an iterable oobject infinitely""" def __init__(self, iterable): self.iterable = iterable self.epoch = 0 self.iterable.sampler.set_epoch(self.epoch) self.iterator = iter(iterable) def __iter__(self): while True: try: batch = next(self.iterator) except StopIteration: self.epoch += 1 self.iterable.sampler.set_epoch(self.epoch) self.iterator = iter(self.iterable) batch = next(self.iterator) yield batch def init_transform_dict(video_res=(240, 320), input_res=(224, 224), randcrop_scale=(0.8, 1.0), color_jitter=(0, 0, 0), norm_mean=(0.48145466, 0.4578275, 0.40821073), norm_std=(0.26862954, 0.26130258, 0.27577711)): normalize = transforms.Normalize(mean=norm_mean, std=norm_std) transform_dict = { 'train': transforms.Compose([ transforms.RandomResizedCrop(input_res, scale=randcrop_scale, interpolation=transforms.InterpolationMode.BICUBIC), transforms.RandomHorizontalFlip(), transforms.ColorJitter(brightness=color_jitter[0], saturation=color_jitter[1], hue=color_jitter[2]), normalize, ]), 'val': transforms.Compose([ transforms.Resize([video_res[0], video_res[1]]), transforms.CenterCrop([int(video_res[0]*0.9), int(video_res[1]*0.9)]), transforms.Resize(input_res), normalize, ]), 'test': transforms.Compose([ transforms.Resize([video_res[0], video_res[1]]), transforms.CenterCrop([int(video_res[0]*0.9), int(video_res[1]*0.9)]), transforms.Resize(input_res), normalize, ]) } return transform_dict def init_transform_dict_simple(video_res=(240, 320), input_res=(224, 224), randcrop_scale=(0.8, 1.0), color_jitter=(0, 0, 0), norm_mean=(0.48145466, 0.4578275, 0.40821073), norm_std=(0.26862954, 0.26130258, 0.27577711)): normalize = transforms.Normalize(mean=norm_mean, std=norm_std) transform_dict = { 'train': transforms.Compose([ transforms.Resize(input_res, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(input_res), normalize, ]), 'val': transforms.Compose([ transforms.Resize(input_res, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(input_res), normalize, ]), 'test': transforms.Compose([ transforms.Resize(input_res, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(input_res), normalize, ]) } return transform_dict def init_transform_dict_image(video_res=(240, 320), input_res=(224, 224), randcrop_scale=(0.8, 1.0), color_jitter=(0, 0, 0), norm_mean=(0.48145466, 0.4578275, 0.40821073), norm_std=(0.26862954, 0.26130258, 0.27577711)): normalize = transforms.Normalize(mean=norm_mean, std=norm_std) transform_dict = { 'train': transforms.Compose([ transforms.RandomResizedCrop(input_res, scale=randcrop_scale, interpolation=transforms.InterpolationMode.BICUBIC), transforms.RandomHorizontalFlip(), transforms.ColorJitter(brightness=color_jitter[0], saturation=color_jitter[1], hue=color_jitter[2]), normalize, ]), 'val': transforms.Compose([ transforms.Resize(input_res, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(input_res), normalize, ]), 'test': transforms.Compose([ transforms.Resize(input_res, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(input_res), normalize, ]) } return transform_dictclip-vip/XPretrain-main/CLIP-Vip/src/datasets/dataset_pretrain_stage1_all_source.py000066400000000000000000000257021475357401500307210ustar00rootroot00000000000000import horovod.torch as hvd import torch from torch.utils.data import Dataset import random import os import json from torch.utils.data.dataloader import default_collate from src.utils.logger import LOGGER from src.utils.basic_utils import flat_list_of_lists from src.datasets.data_utils import mask_batch_text_tokens, img_collate from src.datasets.dataloader import init_transform_dict, init_transform_dict_simple import decord from decord import VideoReader from decord import cpu, gpu decord.bridge.set_bridge("torch") import math import torch.nn.functional as F import numpy as np import cv2 import lmdb import glob import src.utils.stop_words as stop_words from PIL import Image from src.datasets.sample_frames import SampleFrames class HDVILAPretrainDataset(Dataset): """ vis_dir: video data root path anno_path: video meta data path (json, jsonl, lmdb) vid_cap_path: generated video caption path vid_txt: "subtitle" "caption" "caption_subtitle_concat" "caption_subtitle_random" vis_format: ["video", "videoframe"] """ def __init__(self, cfg, vis_dir, anno_path, \ vid_cap_path="", img_dir="", cap_path="", \ img_source="", img_ratio=0, vid_txt='subtitle', vis_format='video', mode="train"): self.vis_format = vis_format self.cfg = cfg self.vis_dir = vis_dir self.anno_path = anno_path self.vid_cap_path = vid_cap_path self.vid_txt = vid_txt self.mode = mode self.n_clips = cfg.train_n_clips if mode == "train" else cfg.test_n_clips self.num_frm = cfg.train_num_frms if mode == "train" else cfg.test_num_frms self.sample_rate = cfg.sample_rate self.transform = init_transform_dict_simple(video_res=cfg.video_res, input_res=cfg.input_res)[mode] self.frame_sampler = SampleFrames(clip_len=self.num_frm, frame_interval=self.sample_rate, num_clips=self.n_clips, temporal_jitter=True) self.video_init_dataset() if self.vis_format == "videoframe" or "caption" in self.vid_txt: self.caption_init_dataset() def caption_init_dataset(self): self.caption_use_lmdb = False json_type = os.path.splitext(self.vid_cap_path)[-1] assert json_type in ['.json', '.jsonl', ''] if json_type == '': print('begin cap lmdb open') cap_env = lmdb.open(self.vid_cap_path, readonly=True, create=False) print('finish cap lmdb open') self.cap_txn = cap_env.begin() print('finish cap env begin') self.caption_use_lmdb = True else: captions = json.load(open(self.vid_cap_path)) cap_dict = {} for item in captions: cap_dict[item["image_id"]] = item["caption"] del captions self.cap_dict = cap_dict def video_init_dataset(self): self.use_lmdb = False json_type = os.path.splitext(self.anno_path)[-1] assert json_type in ['.json', '.jsonl', ''] if json_type == '': self.use_lmdb = True print('begin lmdb open') env = lmdb.open(self.anno_path, readonly=True, create=False) print('finish lmdb open') self.txn = env.begin() print('finish env begin') self.anno_path = self.anno_path + ".jsonl" json_type = os.path.splitext(self.anno_path)[-1] if json_type == '.jsonl': data = [] with open(self.anno_path) as f: for line in f: data.append(json.loads(line)) else: data = json.load(open(self.anno_path)) self.datalist = data def video_idx2path(self, index): if 'clip_id' in self.datalist[index]: vis_id = self.datalist[index]['clip_id'] elif 'video_id' in self.datalist[index]: vis_id = self.datalist[index]['video_id'] clip_name = vis_id video_name = vis_id.split('.')[0] return os.path.join(self.vis_dir, video_name, clip_name) def video_get_text(self, index): if self.use_lmdb: texts = self.txn.get(self.datalist[index]['clip_id'].encode()).decode() else: texts = self.datalist[index]['text'] if "caption" in self.vid_txt: if self.caption_use_lmdb: caption = self.cap_txn.get(self.datalist[index]['clip_id'].encode(), None) if caption: caption = caption.decode() else: caption = self.cap_dict.get(self.datalist[index]['clip_id'], None) if caption and self.vid_txt == "caption": texts = caption elif caption and self.vid_txt == "caption_subtitle_concat": texts = caption + ". with subtitle: " + texts elif caption and self.vid_txt == "caption_subtitle_random": if_caption = random.randint(0, 1) if if_caption: texts = caption if self.vis_format == "videoframe": if self.caption_use_lmdb: mid_caption = self.cap_txn.get(self.datalist[index]['clip_id'].encode(), None) if mid_caption: mid_caption = mid_caption.decode() else: mid_caption = "None" LOGGER.info(f"Failed to load caption of video: {self.datalist[index]['clip_id']}. " f"The caption will be None.") else: mid_caption = self.cap_dict.get(self.datalist[index]['clip_id'], "None") else: mid_caption = None return [texts], [mid_caption] def __len__(self): return len(self.datalist) def get_sample_idx(self, total_frame_num): """ sample rate > 0: use SampleFrames, loop default sample rate = 0: uniform sampling, temporal jittering """ if self.sample_rate > 0: results = {"total_frames": total_frame_num, "start_index": 0} results = self.frame_sampler(results) return results["frame_inds"] elif self.sample_rate == 0: if hasattr(self.cfg, "sample_jitter") and self.cfg.sample_jitter and self.mode == "train": interval = int(total_frame_num / (self.n_clips*self.num_frm - 1)) start = np.random.randint(0, interval+1) end = np.random.randint(total_frame_num-1-interval, total_frame_num) return np.linspace(start, end, self.n_clips*self.num_frm).astype(int) else: return np.linspace(0, total_frame_num-1, self.n_clips*self.num_frm).astype(int) def load_video(self, vis_path): vr = VideoReader(vis_path, ctx=cpu(0)) total_frame_num = len(vr) frame_idx = self.get_sample_idx(total_frame_num) img_array = vr.get_batch(frame_idx) # (n_clips*num_frm, H, W, 3) img_array = img_array.permute(0, 3, 1, 2).float() / 255. img_array = self.transform(img_array) if self.vis_format == "videoframe": mid_frame = vr.get_batch([int(total_frame_num/2)]) mid_frame = mid_frame.permute(0, 3, 1, 2).float() / 255. mid_frame = self.transform(mid_frame) else: mid_frame = None return img_array, mid_frame def __getitem__(self, index): num_retries = 10 # skip error videos for _ in range(num_retries): try: vis_path = self.video_idx2path(index) video, mid_frame = self.load_video(vis_path) texts, mid_caption = self.video_get_text(index) except: LOGGER.info(f"Failed to load examples with video: {os.path.basename(vis_path)}. " f"Will select an random sample as a replacement.") index = random.randint(0, len(self) - 1) continue else: break else: raise RuntimeError(f"Failed to fetch video after {num_retries} retries.") if self.vis_format == "videoframe": image = mid_frame captions = mid_caption else: image = None captions = None return dict( video = video, # [clips*num_frm, C, H_crop, W_crop] texts = texts, image = image, captions = captions ) class PretrainCollator(object): def __init__(self, tokenizer, max_length=40, is_train=True): self.tokenizer = tokenizer self.max_length = max_length self.is_train = is_train def collate_batch(self, batch): batch_size = len(batch) video = default_collate([d["video"] for d in batch]) if "image" in batch[0] and batch[0]["image"] is not None: image = default_collate([d["image"] for d in batch]) else: image = None # subtitle data text_str_list = flat_list_of_lists([d["texts"] for d in batch]) batch_enc = self.tokenizer.batch_encode_plus( text_str_list, max_length=self.max_length, padding="max_length", truncation=True, return_tensors="pt" ) text_input_ids = batch_enc.input_ids # (B, L) text_input_mask = batch_enc.attention_mask # (B, L) # caption data if "captions" in batch[0] and batch[0]["captions"] is not None: text_str_list = flat_list_of_lists([d["captions"] for d in batch]) batch_enc = self.tokenizer.batch_encode_plus( text_str_list, max_length=self.max_length, padding="max_length", truncation=True, return_tensors="pt" ) caption_ids = batch_enc.input_ids # (B*img_num, L) caption_masks = batch_enc.attention_mask # (B*img_num, L) caption_ids = caption_ids.view(batch_size, -1, caption_ids.shape[-1]) caption_masks = caption_masks.view(batch_size, -1, caption_masks.shape[-1]) else: caption_ids = None caption_masks = None collated_batch = dict( video=video, # [B, n_clips*num_frms, C, H, W] text_input_ids=text_input_ids, # [B, L] text_input_mask=text_input_mask, image=image, # [B, img_num, C, H, W] caption_ids=caption_ids, # [B, img_num, L] caption_masks=caption_masks ) return collated_batch clip-vip/XPretrain-main/CLIP-Vip/src/datasets/dataset_video_retrieval.py000066400000000000000000000155651475357401500266120ustar00rootroot00000000000000import torch from torch.utils.data import Dataset import random import os import json from torch.utils.data.dataloader import default_collate from src.utils.logger import LOGGER from src.utils.basic_utils import flat_list_of_lists from src.datasets.data_utils import mask_batch_text_tokens, img_collate from src.datasets.dataloader import init_transform_dict, init_transform_dict_simple import decord from decord import VideoReader from decord import cpu, gpu decord.bridge.set_bridge("torch") import math import torch.nn.functional as F import numpy as np import cv2 import lmdb import glob import src.utils.stop_words as stop_words from PIL import Image from src.datasets.sample_frames import SampleFrames class HDVILAVideoRetrievalDataset(Dataset): """ datalist """ def __init__(self, cfg, vis_dir, anno_path, vis_format='video', mode="train"): assert vis_format in ["video", "frame"] self.cfg = cfg self.vis_dir = vis_dir self.anno_path = anno_path self.mode = mode self.vis_format = vis_format self.n_clips = cfg.train_n_clips if mode == "train" else cfg.test_n_clips self.num_frm = cfg.train_num_frms if mode == "train" else cfg.test_num_frms self.sample_rate = cfg.sample_rate if hasattr(cfg, "text_pos_num"): self.pos_num = cfg.pos_num else: self.pos_num = 1 self.transform = init_transform_dict_simple(video_res=cfg.video_res, input_res=cfg.input_res)[mode] self.frame_sampler = SampleFrames(clip_len=self.num_frm, frame_interval=self.sample_rate, num_clips=self.n_clips, temporal_jitter=True) self.init_dataset_process() def init_dataset_process(self): json_type = os.path.splitext(self.anno_path)[-1] assert json_type in ['.json', '.jsonl'] if json_type == '.jsonl': data = [] with open(self.anno_path) as f: for line in f: data.append(json.loads(line)) else: data = json.load(open(self.anno_path)) self.datalist = data def id2path(self, id): clip_name = id if self.vis_format == 'video': name = os.path.join(self.vis_dir, clip_name.split('/')[-1]+".mp4") if "lsmdc" in self.vis_dir: name = os.path.join(self.vis_dir, clip_name + ".avi") else: name = os.path.join(self.vis_dir, clip_name) return name def __len__(self): return len(self.datalist) def get_sample_idx(self, total_frame_num): """ sample rate > 0: use SampleFrames, loop default sample rate = 0: uniform sampling, temporal jittering """ if self.sample_rate > 0: results = {"total_frames": total_frame_num, "start_index": 0} results = self.frame_sampler(results) return results["frame_inds"] elif self.sample_rate == 0: if hasattr(self.cfg, "sample_jitter") and self.cfg.sample_jitter and self.mode == "train": interval = int(total_frame_num / (self.n_clips*self.num_frm - 1)) start = np.random.randint(0, interval+1) end = np.random.randint(total_frame_num-1-interval, total_frame_num) return np.linspace(start, end, self.n_clips*self.num_frm).astype(int) else: return np.linspace(0, total_frame_num-1, self.n_clips*self.num_frm).astype(int) def load_video(self, vis_path): vr = VideoReader(vis_path, ctx=cpu(0)) total_frame_num = len(vr) frame_idx = self.get_sample_idx(total_frame_num) img_array = vr.get_batch(frame_idx) # (n_clips*num_frm, H, W, 3) img_array = img_array.permute(0, 3, 1, 2).float() / 255. img_array = self.transform(img_array) return img_array def load_frames(self, vis_path, total_frame_num): # print('total_frame_num',total_frame_num) frame_idx = self.get_sample_idx(total_frame_num) img_array = [] for i in frame_idx: img = Image.open(os.path.join(vis_path, \ vis_path.split('/')[-1] + '_{0:03d}.jpg'.format(i))).convert("RGB") img_array.append(np.array(img)) img_array = torch.from_numpy(np.array(img_array)) # (n_clips*num_frm, H, W, 3) img_array = img_array.permute(0, 3, 1, 2).float() / 255. img_array = self.transform(img_array) return img_array def __getitem__(self, index): if self.cfg.dummy_data: return dict( video = torch.randn(self.n_clips*self.num_frm, 3, self.cfg.input_res[0], self.cfg.input_res[1]), # [clips, num_frm, C, H_crop, W_crop] texts = ["This is a dummy sentence, which contains nothing meaningful."] ) vis_id = self.datalist[index]['clip_id'] texts = self.datalist[index]['text'] if isinstance(texts, list): texts = random.sample(self.datalist[index]['text'], self.pos_num) if 'didemo' in self.anno_path: texts = [' '.join(self.datalist[index]['text'])] else: texts = [texts] vis_path = self.id2path(vis_id) video = self.load_video(vis_path) if self.vis_format=='video' else self.load_frames(vis_path, self.datalist[index]['num_frame']) return dict( video = video, # [clips*num_frm, C, H_crop, W_crop] texts = texts ) class VideoRetrievalCollator(object): def __init__(self, tokenizer, max_length=40, is_train=True): self.tokenizer = tokenizer self.max_length = max_length self.is_train = is_train def collate_batch(self, batch): if isinstance(batch[0]["video"], torch.Tensor): v_collate = default_collate else: v_collate = img_collate video = v_collate([d["video"] for d in batch]) text_examples = flat_list_of_lists([d["texts"] for d in batch]) text_str_list = [d for d in text_examples] # (B, ) batch_enc = self.tokenizer.batch_encode_plus( text_str_list, max_length=self.max_length, padding="max_length", truncation=True, return_tensors="pt" ) text_input_ids = batch_enc.input_ids # (B, L) text_input_mask = batch_enc.attention_mask # (B, L) collated_batch = dict( video=video, # [B, clips, num_frm, C, H_crop, W_crop] text_input_ids=text_input_ids, text_input_mask=text_input_mask ) return collated_batch clip-vip/XPretrain-main/CLIP-Vip/src/datasets/sample_frames.py000066400000000000000000000205011475357401500245220ustar00rootroot00000000000000# Copyright (c) OpenMMLab. All rights reserved. import copy as cp import io import os import os.path as osp import shutil import warnings import numpy as np class SampleFrames: """Sample frames from the video. Required keys are "total_frames", "start_index" , added or modified keys are "frame_inds", "frame_interval" and "num_clips". Args: clip_len (int): Frames of each sampled output clip. frame_interval (int): Temporal interval of adjacent sampled frames. Default: 1. num_clips (int): Number of clips to be sampled. Default: 1. temporal_jitter (bool): Whether to apply temporal jittering. Default: False. twice_sample (bool): Whether to use twice sample when testing. If set to True, it will sample frames with and without fixed shift, which is commonly used for testing in TSM model. Default: False. out_of_bound_opt (str): The way to deal with out of bounds frame indexes. Available options are 'loop', 'repeat_last'. Default: 'loop'. test_mode (bool): Store True when building test or validation dataset. Default: False. start_index (None): This argument is deprecated and moved to dataset class (``BaseDataset``, ``VideoDatset``, ``RawframeDataset``, etc), see this: https://github.com/open-mmlab/mmaction2/pull/89. keep_tail_frames (bool): Whether to keep tail frames when sampling. Default: False. """ def __init__(self, clip_len, frame_interval=1, num_clips=1, temporal_jitter=False, twice_sample=False, out_of_bound_opt='loop', test_mode=False, start_index=None, keep_tail_frames=False): self.clip_len = clip_len self.frame_interval = frame_interval self.num_clips = num_clips self.temporal_jitter = temporal_jitter self.twice_sample = twice_sample self.out_of_bound_opt = out_of_bound_opt self.test_mode = test_mode self.keep_tail_frames = keep_tail_frames assert self.out_of_bound_opt in ['loop', 'repeat_last'] if start_index is not None: warnings.warn('No longer support "start_index" in "SampleFrames", ' 'it should be set in dataset class, see this pr: ' 'https://github.com/open-mmlab/mmaction2/pull/89') def _get_train_clips(self, num_frames): """Get clip offsets in train mode. It will calculate the average interval for selected frames, and randomly shift them within offsets between [0, avg_interval]. If the total number of frames is smaller than clips num or origin frames length, it will return all zero indices. Args: num_frames (int): Total number of frame in the video. Returns: np.ndarray: Sampled frame indices in train mode. """ ori_clip_len = self.clip_len * self.frame_interval if self.keep_tail_frames: avg_interval = (num_frames - ori_clip_len + 1) / float( self.num_clips) if num_frames > ori_clip_len - 1: base_offsets = np.arange(self.num_clips) * avg_interval clip_offsets = (base_offsets + np.random.uniform( 0, avg_interval, self.num_clips)).astype(np.int64) else: clip_offsets = np.zeros((self.num_clips, ), dtype=np.int64) else: avg_interval = (num_frames - ori_clip_len + 1) // self.num_clips if avg_interval > 0: base_offsets = np.arange(self.num_clips) * avg_interval clip_offsets = base_offsets + np.random.randint( avg_interval, size=self.num_clips) elif num_frames > max(self.num_clips, ori_clip_len): clip_offsets = np.sort( np.random.randint( num_frames - ori_clip_len + 1, size=self.num_clips)) elif avg_interval == 0: ratio = (num_frames - ori_clip_len + 1.0) / self.num_clips clip_offsets = np.around(np.arange(self.num_clips) * ratio) else: clip_offsets = np.zeros((self.num_clips, ), dtype=np.int64) return clip_offsets def _get_test_clips(self, num_frames): """Get clip offsets in test mode. Calculate the average interval for selected frames, and shift them fixedly by avg_interval/2. If set twice_sample True, it will sample frames together without fixed shift. If the total number of frames is not enough, it will return all zero indices. Args: num_frames (int): Total number of frame in the video. Returns: np.ndarray: Sampled frame indices in test mode. """ ori_clip_len = self.clip_len * self.frame_interval avg_interval = (num_frames - ori_clip_len + 1) / float(self.num_clips) if num_frames > ori_clip_len - 1: base_offsets = np.arange(self.num_clips) * avg_interval clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int64) if self.twice_sample: clip_offsets = np.concatenate([clip_offsets, base_offsets]) else: clip_offsets = np.zeros((self.num_clips, ), dtype=np.int64) return clip_offsets def _sample_clips(self, num_frames): """Choose clip offsets for the video in a given mode. Args: num_frames (int): Total number of frame in the video. Returns: np.ndarray: Sampled frame indices. """ if self.test_mode: clip_offsets = self._get_test_clips(num_frames) else: clip_offsets = self._get_train_clips(num_frames) return clip_offsets def __call__(self, results): """Perform the SampleFrames loading. Args: results (dict): The resulting dict to be modified and passed to the next transform in pipeline. """ total_frames = results['total_frames'] clip_offsets = self._sample_clips(total_frames) frame_inds = clip_offsets[:, None] + np.arange( self.clip_len)[None, :] * self.frame_interval frame_inds = np.concatenate(frame_inds) if self.temporal_jitter: perframe_offsets = np.random.randint( self.frame_interval, size=len(frame_inds)) frame_inds += perframe_offsets frame_inds = frame_inds.reshape((-1, self.clip_len)) if self.out_of_bound_opt == 'loop': frame_inds = np.mod(frame_inds, total_frames) elif self.out_of_bound_opt == 'repeat_last': safe_inds = frame_inds < total_frames unsafe_inds = 1 - safe_inds last_ind = np.max(safe_inds * frame_inds, axis=1) new_inds = (safe_inds * frame_inds + (unsafe_inds.T * last_ind).T) frame_inds = new_inds else: raise ValueError('Illegal out_of_bound option.') start_index = results['start_index'] frame_inds = np.concatenate(frame_inds) + start_index results['frame_inds'] = frame_inds.astype(np.int64) results['clip_len'] = self.clip_len results['frame_interval'] = self.frame_interval results['num_clips'] = self.num_clips return results def __repr__(self): repr_str = (f'{self.__class__.__name__}(' f'clip_len={self.clip_len}, ' f'frame_interval={self.frame_interval}, ' f'num_clips={self.num_clips}, ' f'temporal_jitter={self.temporal_jitter}, ' f'twice_sample={self.twice_sample}, ' f'out_of_bound_opt={self.out_of_bound_opt}, ' f'test_mode={self.test_mode})') return repr_str if __name__ == "__main__": sampler = SampleFrames(clip_len=8, frame_interval=4, num_clips=3, temporal_jitter=True) results = {"total_frames":20, "start_index": 0} print(type(sampler(results)["frame_inds"]))clip-vip/XPretrain-main/CLIP-Vip/src/modeling/000077500000000000000000000000001475357401500213225ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/modeling/CLIP.py000066400000000000000000001366101475357401500224320ustar00rootroot00000000000000# coding=utf-8 # Copyright 2021 The OpenAI Team Authors and The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ PyTorch CLIP model.""" from dataclasses import dataclass from typing import Any, Optional, Tuple, Union from easydict import EasyDict as edict import torch import torch.utils.checkpoint from torch import nn import torch.nn.functional as F from transformers.activations import ACT2FN from transformers.modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling from transformers.modeling_utils import PreTrainedModel from transformers.utils import ( ModelOutput, add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings, ) from transformers.models.clip.configuration_clip import CLIPConfig, CLIPTextConfig, CLIPVisionConfig from functools import partial logger = logging.get_logger(__name__) _CHECKPOINT_FOR_DOC = "openai/clip-vit-base-patch32" CLIP_PRETRAINED_MODEL_ARCHIVE_LIST = [ "openai/clip-vit-base-patch32", # See all CLIP models at https://huggingface.co/models?filter=clip ] # Copied from transformers.models.bart.modeling_bart._expand_mask def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): """ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. """ bsz, src_len = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) inverted_mask = 1.0 - expanded_mask return inverted_mask.masked_fill(inverted_mask.bool(), torch.finfo(dtype).min) # contrastive loss function, adapted from # https://sachinruk.github.io/blog/pytorch/pytorch%20lightning/loss%20function/gpu/2021/03/07/CLIP.html def contrastive_loss(logits: torch.Tensor) -> torch.Tensor: return nn.functional.cross_entropy(logits, torch.arange(len(logits), device=logits.device)) def clip_loss(similarity: torch.Tensor) -> torch.Tensor: caption_loss = contrastive_loss(similarity) image_loss = contrastive_loss(similarity.T) return (caption_loss + image_loss) / 2.0 @dataclass class CLIPOutput(ModelOutput): """ Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. """ loss: Optional[torch.FloatTensor] = None logits_per_image: torch.FloatTensor = None logits_per_text: torch.FloatTensor = None text_embeds: torch.FloatTensor = None image_embeds: torch.FloatTensor = None text_model_output: BaseModelOutputWithPooling = None vision_model_output: BaseModelOutputWithPooling = None def to_tuple(self) -> Tuple[Any]: return tuple( self[k] if k not in ["text_model_output", "vision_model_output"] else getattr(self, k).to_tuple() for k in self.keys() ) class CLIPVisionEmbeddings(nn.Module): def __init__(self, config: CLIPVisionConfig): super().__init__() self.config = config self.embed_dim = config.hidden_size self.image_size = config.image_size self.patch_size = config.patch_size self.class_embedding = nn.Parameter(torch.randn(self.embed_dim)) self.patch_embedding = nn.Conv2d( in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size, bias=False ) self.num_patches = (self.image_size // self.patch_size) ** 2 self.num_positions = self.num_patches + 1 self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim) self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1))) def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor: batch_size = pixel_values.shape[0] patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid] patch_embeds = patch_embeds.flatten(2).transpose(1, 2) class_embeds = self.class_embedding.expand(batch_size, 1, -1) embeddings = torch.cat([class_embeds, patch_embeds], dim=1) embeddings = embeddings + self.position_embedding(self.position_ids) return embeddings class CLIPVisionSTEmbeddings(nn.Module): def __init__(self, config: CLIPVisionConfig, additional_vision_config=None): super().__init__() self.config = config self.embed_dim = config.hidden_size self.image_size = config.image_size self.temporal_size = additional_vision_config.temporal_size self.patch_size = config.patch_size self.class_embedding = nn.Parameter(torch.randn(self.embed_dim)) self.patch_embedding = nn.Conv2d( in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size, bias=False ) self.num_patches = (self.image_size // self.patch_size) ** 2 self.num_positions = self.num_patches + 1 self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim) self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1))) self.temporal_embedding = nn.Parameter(torch.zeros(1, self.temporal_size, self.embed_dim)) def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor: B, T, C, H, W = pixel_values.shape if T != self.temporal_embedding.shape[1]: time_embed = self.temporal_embedding.transpose(1, 2) time_embed = F.interpolate(time_embed, size=(T), mode='linear') time_embed = time_embed.transpose(1, 2) else: time_embed = self.temporal_embedding patch_embeds = self.patch_embedding(pixel_values.reshape(-1, C, H, W)) patch_embeds = patch_embeds.flatten(2).transpose(1, 2) # [B*T, H*W, C] C = patch_embeds.shape[-1] patch_embeds = patch_embeds.reshape(B, T, -1, C) patch_embeds = patch_embeds + time_embed.unsqueeze(2) # [B, T, H*W, C] patch_embeds = patch_embeds + self.position_embedding(self.position_ids[:, 1:]).unsqueeze(1) patch_embeds = patch_embeds.reshape(B, -1, C) class_embeds = self.class_embedding.expand(B, 1, -1) class_embeds = class_embeds + self.position_embedding(self.position_ids[:, 0:1]) embeddings = torch.cat([class_embeds, patch_embeds], dim=1) return embeddings class CLIPTextEmbeddings(nn.Module): def __init__(self, config: CLIPTextConfig): super().__init__() embed_dim = config.hidden_size self.token_embedding = nn.Embedding(config.vocab_size, embed_dim) self.position_embedding = nn.Embedding(config.max_position_embeddings, embed_dim) # position_ids (1, len position emb) is contiguous in memory and exported when serialized self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1))) def forward( self, input_ids: Optional[torch.LongTensor] = None, position_ids: Optional[torch.LongTensor] = None, inputs_embeds: Optional[torch.FloatTensor] = None, ) -> torch.Tensor: seq_length = input_ids.shape[-1] if input_ids is not None else inputs_embeds.shape[-2] if position_ids is None: position_ids = self.position_ids[:, :seq_length] if inputs_embeds is None: inputs_embeds = self.token_embedding(input_ids) position_embeddings = self.position_embedding(position_ids) embeddings = inputs_embeds + position_embeddings return embeddings class CLIPAttention(nn.Module): """Multi-headed attention from 'Attention Is All You Need' paper""" def __init__(self, config): super().__init__() self.config = config self.embed_dim = config.hidden_size self.num_heads = config.num_attention_heads self.head_dim = self.embed_dim // self.num_heads assert ( self.head_dim * self.num_heads == self.embed_dim ), f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads})." self.scale = self.head_dim**-0.5 self.dropout = config.attention_dropout self.k_proj = nn.Linear(self.embed_dim, self.embed_dim) self.v_proj = nn.Linear(self.embed_dim, self.embed_dim) self.q_proj = nn.Linear(self.embed_dim, self.embed_dim) self.out_proj = nn.Linear(self.embed_dim, self.embed_dim) def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, causal_attention_mask: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = False, ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: """Input shape: Batch x Time x Channel""" bsz, tgt_len, embed_dim = hidden_states.size() # get query proj query_states = self.q_proj(hidden_states) * self.scale key_states = self._shape(self.k_proj(hidden_states), -1, bsz) value_states = self._shape(self.v_proj(hidden_states), -1, bsz) proj_shape = (bsz * self.num_heads, -1, self.head_dim) query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape) key_states = key_states.view(*proj_shape) value_states = value_states.view(*proj_shape) src_len = key_states.size(1) attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len): raise ValueError( f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is {attn_weights.size()}" ) # apply the causal_attention_mask first if causal_attention_mask is not None: if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {causal_attention_mask.size()}" ) attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len) if attention_mask is not None: if attention_mask.size() != (bsz, 1, tgt_len, src_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}" ) attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len) attn_weights = nn.functional.softmax(attn_weights, dim=-1) if output_attentions: # this operation is a bit akward, but it's required to # make sure that attn_weights keeps its gradient. # In order to do so, attn_weights have to reshaped # twice and have to be reused in the following attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len) else: attn_weights_reshaped = None attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training) attn_output = torch.bmm(attn_probs, value_states) if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim): raise ValueError( f"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is {attn_output.size()}" ) attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim) attn_output = attn_output.transpose(1, 2) attn_output = attn_output.reshape(bsz, tgt_len, embed_dim) attn_output = self.out_proj(attn_output) return attn_output, attn_weights_reshaped class CLIPMLP(nn.Module): def __init__(self, config): super().__init__() self.config = config self.activation_fn = ACT2FN[config.hidden_act] self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size) self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: hidden_states = self.fc1(hidden_states) hidden_states = self.activation_fn(hidden_states) hidden_states = self.fc2(hidden_states) return hidden_states class CLIPEncoderLayer(nn.Module): def __init__(self, config: CLIPConfig): super().__init__() self.embed_dim = config.hidden_size self.self_attn = CLIPAttention(config) self.layer_norm1 = nn.LayerNorm(self.embed_dim) self.mlp = CLIPMLP(config) self.layer_norm2 = nn.LayerNorm(self.embed_dim) def forward( self, hidden_states: torch.Tensor, attention_mask: torch.Tensor, causal_attention_mask: torch.Tensor, output_attentions: Optional[bool] = False, ) -> Tuple[torch.FloatTensor]: """ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. """ residual = hidden_states hidden_states = self.layer_norm1(hidden_states) hidden_states, attn_weights = self.self_attn( hidden_states=hidden_states, attention_mask=attention_mask, causal_attention_mask=causal_attention_mask, output_attentions=output_attentions, ) hidden_states = residual + hidden_states residual = hidden_states hidden_states = self.layer_norm2(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = residual + hidden_states outputs = (hidden_states,) if output_attentions: outputs += (attn_weights,) return outputs class CLIPPreTrainedModel(PreTrainedModel): """ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. """ config_class = CLIPConfig base_model_prefix = "clip" supports_gradient_checkpointing = True _keys_to_ignore_on_load_missing = [r"position_ids"] def _init_weights(self, module): """Initialize the weights""" factor = self.config.initializer_factor if isinstance(module, CLIPTextEmbeddings): module.token_embedding.weight.data.normal_(mean=0.0, std=factor * 0.02) module.position_embedding.weight.data.normal_(mean=0.0, std=factor * 0.02) elif isinstance(module, CLIPVisionEmbeddings) or isinstance(module, CLIPVisionSTEmbeddings): factor = self.config.initializer_factor nn.init.normal_(module.class_embedding, mean=0.0, std=module.embed_dim**-0.5 * factor) nn.init.normal_(module.patch_embedding.weight, std=module.config.initializer_range * factor) nn.init.normal_(module.position_embedding.weight, std=module.config.initializer_range * factor) elif isinstance(module, CLIPAttention): factor = self.config.initializer_factor in_proj_std = (module.embed_dim**-0.5) * ((2 * module.config.num_hidden_layers) ** -0.5) * factor out_proj_std = (module.embed_dim**-0.5) * factor nn.init.normal_(module.q_proj.weight, std=in_proj_std) nn.init.normal_(module.k_proj.weight, std=in_proj_std) nn.init.normal_(module.v_proj.weight, std=in_proj_std) nn.init.normal_(module.out_proj.weight, std=out_proj_std) elif isinstance(module, CLIPMLP): factor = self.config.initializer_factor in_proj_std = ( (module.config.hidden_size**-0.5) * ((2 * module.config.num_hidden_layers) ** -0.5) * factor ) fc_std = (2 * module.config.hidden_size) ** -0.5 * factor nn.init.normal_(module.fc1.weight, std=fc_std) nn.init.normal_(module.fc2.weight, std=in_proj_std) elif isinstance(module, CLIPModel): nn.init.normal_( module.text_projection.weight, std=module.text_embed_dim**-0.5 * self.config.initializer_factor, ) nn.init.normal_( module.visual_projection.weight, std=module.vision_embed_dim**-0.5 * self.config.initializer_factor, ) if isinstance(module, nn.LayerNorm): module.bias.data.zero_() module.weight.data.fill_(1.0) if isinstance(module, nn.Linear) and module.bias is not None: module.bias.data.zero_() def _set_gradient_checkpointing(self, module, value=False): if isinstance(module, CLIPEncoder): module.gradient_checkpointing = value CLIP_START_DOCSTRING = r""" This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. """ CLIP_TEXT_INPUTS_DOCSTRING = r""" Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ CLIP_VISION_INPUTS_DOCSTRING = r""" Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`CLIPFeatureExtractor`]. See [`CLIPFeatureExtractor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ CLIP_INPUTS_DOCSTRING = r""" Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`CLIPFeatureExtractor`]. See [`CLIPFeatureExtractor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ class CLIPEncoder(nn.Module): """ Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig """ def __init__(self, config: CLIPConfig): super().__init__() self.config = config self.layers = nn.ModuleList([CLIPEncoderLayer(config) for _ in range(config.num_hidden_layers)]) self.gradient_checkpointing = False def forward( self, inputs_embeds, attention_mask: Optional[torch.Tensor] = None, causal_attention_mask: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutput]: r""" Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict encoder_states = () if output_hidden_states else None all_attentions = () if output_attentions else None hidden_states = inputs_embeds for idx, encoder_layer in enumerate(self.layers): if output_hidden_states: encoder_states = encoder_states + (hidden_states,) if self.gradient_checkpointing and self.training: def create_custom_forward(module): def custom_forward(*inputs): return module(*inputs, output_attentions) return custom_forward layer_outputs = torch.utils.checkpoint.checkpoint( create_custom_forward(encoder_layer), hidden_states, attention_mask, causal_attention_mask, ) else: layer_outputs = encoder_layer( hidden_states, attention_mask, causal_attention_mask, output_attentions=output_attentions, ) hidden_states = layer_outputs[0] if output_attentions: all_attentions = all_attentions + (layer_outputs[1],) if output_hidden_states: encoder_states = encoder_states + (hidden_states,) if not return_dict: return tuple(v for v in [hidden_states, encoder_states, all_attentions] if v is not None) return BaseModelOutput( last_hidden_state=hidden_states, hidden_states=encoder_states, attentions=all_attentions ) class CLIPTextTransformer(nn.Module): def __init__(self, config: CLIPTextConfig): super().__init__() self.config = config embed_dim = config.hidden_size self.embeddings = CLIPTextEmbeddings(config) self.encoder = CLIPEncoder(config) self.final_layer_norm = nn.LayerNorm(embed_dim) @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPTextConfig) def forward( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict if input_ids is None: raise ValueError("You have to specify either input_ids") input_shape = input_ids.size() input_ids = input_ids.view(-1, input_shape[-1]) hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids) bsz, seq_len = input_shape # CLIP's text model uses causal mask, prepare it here. # https://github.com/openai/CLIP/blob/cfcffb90e69f37bf2ff1e988237a0fbe41f33c04/clip/model.py#L324 if_fp16 = hidden_states.dtype == torch.float16 causal_attention_mask = self._build_causal_attention_mask(bsz, seq_len, fp16=if_fp16).to(hidden_states.device) # expand attention_mask if attention_mask is not None: # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] attention_mask = _expand_mask(attention_mask, hidden_states.dtype) encoder_outputs = self.encoder( inputs_embeds=hidden_states, attention_mask=attention_mask, causal_attention_mask=causal_attention_mask, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) last_hidden_state = encoder_outputs[0] last_hidden_state = self.final_layer_norm(last_hidden_state) # text_embeds.shape = [batch_size, sequence_length, transformer.width] # take features from the eot embedding (eot_token is the highest number in each sequence) pooled_output = last_hidden_state[torch.arange(last_hidden_state.shape[0]), input_ids.argmax(dim=-1)] if not return_dict: return (last_hidden_state, pooled_output) + encoder_outputs[1:] return BaseModelOutputWithPooling( last_hidden_state=last_hidden_state, pooler_output=pooled_output, hidden_states=encoder_outputs.hidden_states, attentions=encoder_outputs.attentions, ) def _build_causal_attention_mask(self, bsz, seq_len, fp16=False): # lazily create causal attention mask, with full attention between the vision tokens # pytorch uses additive attention mask; fill with -inf mask = torch.empty(bsz, seq_len, seq_len) mask.fill_(float("-inf")) mask.triu_(1) # zero out the lower diagonal mask = mask.unsqueeze(1) # expand mask if fp16: mask = mask.half() return mask class CLIPTextModel(CLIPPreTrainedModel): config_class = CLIPTextConfig def __init__(self, config: CLIPTextConfig): super().__init__(config) self.text_model = CLIPTextTransformer(config) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self) -> nn.Module: return self.text_model.embeddings.token_embedding def set_input_embeddings(self, value): self.text_model.embeddings.token_embedding = value @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPTextConfig) def forward( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: Examples: ```python >>> from transformers import CLIPTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```""" return self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) class CLIPVisionTransformer(nn.Module): def __init__(self, config: CLIPVisionConfig, additional_vision_config=None): super().__init__() self.config = config embed_dim = config.hidden_size self.embeddings = CLIPVisionEmbeddings(config) self.pre_layrnorm = nn.LayerNorm(embed_dim) self.encoder = CLIPEncoder(config) self.post_layernorm = nn.LayerNorm(embed_dim) @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPVisionConfig) def forward( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict if pixel_values is None: raise ValueError("You have to specify pixel_values") hidden_states = self.embeddings(pixel_values) hidden_states = self.pre_layrnorm(hidden_states) encoder_outputs = self.encoder( inputs_embeds=hidden_states, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) last_hidden_state = encoder_outputs[0] pooled_output = last_hidden_state[:, 0, :] pooled_output = self.post_layernorm(pooled_output) if not return_dict: return (last_hidden_state, pooled_output) + encoder_outputs[1:] return BaseModelOutputWithPooling( last_hidden_state=last_hidden_state, pooler_output=pooled_output, hidden_states=encoder_outputs.hidden_states, attentions=encoder_outputs.attentions, ) class CLIPVisionModel(CLIPPreTrainedModel): config_class = CLIPVisionConfig main_input_name = "pixel_values" def __init__(self, config: CLIPVisionConfig): super().__init__(config) self.vision_model = CLIPVisionTransformer(config) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self) -> nn.Module: return self.vision_model.embeddings.patch_embedding @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPVisionConfig) def forward( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```""" return self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) @add_start_docstrings(CLIP_START_DOCSTRING) class CLIPModel(CLIPPreTrainedModel): config_class = CLIPConfig def __init__(self, config: CLIPConfig): super().__init__(config) if not isinstance(config.text_config, CLIPTextConfig): raise ValueError( f"config.text_config is expected to be of type CLIPTextConfig but is of type {type(config.text_config)}." ) if not isinstance(config.vision_config, CLIPVisionConfig): raise ValueError( f"config.vision_config is expected to be of type CLIPVisionConfig but is of type {type(config.vision_config)}." ) text_config = config.text_config vision_config = config.vision_config if hasattr(config, "vision_additional_config"): additional_vision_config = config.vision_additional_config else: additional_vision_config = None self.projection_dim = config.projection_dim self.text_embed_dim = text_config.hidden_size self.vision_embed_dim = vision_config.hidden_size self.text_model = CLIPTextTransformer(text_config) self.vision_model = CLIPVisionTransformer(vision_config, additional_vision_config) self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False) self.text_projection = nn.Linear(self.text_embed_dim, self.projection_dim, bias=False) self.logit_scale = nn.Parameter(torch.ones([]) * self.config.logit_scale_init_value) # Initialize weights and apply final processing self.post_init() @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) def get_text_features( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> torch.FloatTensor: r""" Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> from transformers import CLIPTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict text_outputs = self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) pooled_output = text_outputs[1] text_features = self.text_projection(pooled_output) return text_features @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) def get_image_features( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> torch.FloatTensor: r""" Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict vision_outputs = self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) pooled_output = vision_outputs[1] # pooled_output image_features = self.visual_projection(pooled_output) return image_features @add_start_docstrings_to_model_forward(CLIP_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=CLIPOutput, config_class=CLIPConfig) def forward( self, input_ids: Optional[torch.LongTensor] = None, pixel_values: Optional[torch.FloatTensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, return_loss: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, CLIPOutput]: r""" Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict vision_outputs = self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) text_outputs = self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) image_embeds = vision_outputs[1] image_embeds = self.visual_projection(image_embeds) text_embeds = text_outputs[1] text_embeds = self.text_projection(text_embeds) # normalized features image_embeds = image_embeds / image_embeds.norm(dim=-1, keepdim=True) text_embeds = text_embeds / text_embeds.norm(dim=-1, keepdim=True) # cosine similarity as logits logit_scale = self.logit_scale.exp() logits_per_text = torch.matmul(text_embeds, image_embeds.t()) * logit_scale logits_per_image = logits_per_text.T loss = None if return_loss: loss = clip_loss(logits_per_text) if not return_dict: output = (logits_per_image, logits_per_text, text_embeds, image_embeds, text_outputs, vision_outputs) return ((loss,) + output) if loss is not None else output return CLIPOutput( loss=loss, logits_per_image=logits_per_image, logits_per_text=logits_per_text, text_embeds=text_embeds, image_embeds=image_embeds, text_model_output=text_outputs, vision_model_output=vision_outputs, )clip-vip/XPretrain-main/CLIP-Vip/src/modeling/CLIP_ViP.py000066400000000000000000001510271475357401500232070ustar00rootroot00000000000000# coding=utf-8 # Copyright 2021 The OpenAI Team Authors and The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ PyTorch CLIP model.""" from dataclasses import dataclass from typing import Any, Optional, Tuple, Union import torch import torch.utils.checkpoint from torch import nn import torch.nn.functional as F from transformers.activations import ACT2FN from transformers.modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling from transformers.modeling_utils import PreTrainedModel from transformers.utils import ( ModelOutput, add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings, ) from transformers.models.clip.configuration_clip import CLIPConfig, CLIPTextConfig, CLIPVisionConfig logger = logging.get_logger(__name__) _CHECKPOINT_FOR_DOC = "openai/clip-vit-base-patch32" CLIP_PRETRAINED_MODEL_ARCHIVE_LIST = [ "openai/clip-vit-base-patch32", # See all CLIP models at https://huggingface.co/models?filter=clip ] # Copied from transformers.models.bart.modeling_bart._expand_mask def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): """ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. """ bsz, src_len = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) inverted_mask = 1.0 - expanded_mask return inverted_mask.masked_fill(inverted_mask.bool(), torch.finfo(dtype).min) # contrastive loss function, adapted from # https://sachinruk.github.io/blog/pytorch/pytorch%20lightning/loss%20function/gpu/2021/03/07/CLIP.html def contrastive_loss(logits: torch.Tensor) -> torch.Tensor: return nn.functional.cross_entropy(logits, torch.arange(len(logits), device=logits.device)) def clip_loss(similarity: torch.Tensor) -> torch.Tensor: caption_loss = contrastive_loss(similarity) image_loss = contrastive_loss(similarity.T) return (caption_loss + image_loss) / 2.0 @dataclass class CLIPOutput(ModelOutput): """ Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. """ loss: Optional[torch.FloatTensor] = None logits_per_image: torch.FloatTensor = None logits_per_text: torch.FloatTensor = None text_embeds: torch.FloatTensor = None image_embeds: torch.FloatTensor = None text_model_output: BaseModelOutputWithPooling = None vision_model_output: BaseModelOutputWithPooling = None def to_tuple(self) -> Tuple[Any]: return tuple( self[k] if k not in ["text_model_output", "vision_model_output"] else getattr(self, k).to_tuple() for k in self.keys() ) class CLIPVisionEmbeddings(nn.Module): def __init__(self, config: CLIPVisionConfig): super().__init__() self.config = config self.embed_dim = config.hidden_size self.image_size = config.image_size self.patch_size = config.patch_size self.class_embedding = nn.Parameter(torch.randn(self.embed_dim)) self.patch_embedding = nn.Conv2d( in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size, bias=False ) self.num_patches = (self.image_size // self.patch_size) ** 2 self.num_positions = self.num_patches + 1 self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim) self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1))) def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor: batch_size = pixel_values.shape[0] patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid] patch_embeds = patch_embeds.flatten(2).transpose(1, 2) class_embeds = self.class_embedding.expand(batch_size, 1, -1) embeddings = torch.cat([class_embeds, patch_embeds], dim=1) embeddings = embeddings + self.position_embedding(self.position_ids) return embeddings class CLIPVisionViPEmbeddings(nn.Module): def __init__(self, config: CLIPVisionConfig, additional_vision_config=None): super().__init__() self.config = config self.embed_dim = config.hidden_size self.image_size = config.image_size self.temporal_size = additional_vision_config.temporal_size self.if_use_temporal_embed = additional_vision_config.if_use_temporal_embed self.patch_size = config.patch_size self.add_cls_num = additional_vision_config.add_cls_num self.added_cls = nn.Parameter(torch.randn(self.add_cls_num, self.embed_dim)) self.class_embedding = nn.Parameter(torch.randn(self.embed_dim)) self.patch_embedding = nn.Conv2d( in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size, bias=False ) self.num_patches = (self.image_size // self.patch_size) ** 2 self.num_positions = self.num_patches + 1 self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim) self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1))) if self.if_use_temporal_embed: self.temporal_embedding = nn.Parameter(torch.zeros(1, self.temporal_size, self.embed_dim)) def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor: B, T, C, H, W = pixel_values.shape if self.if_use_temporal_embed: if T != self.temporal_embedding.shape[1]: time_embed = self.temporal_embedding.transpose(1, 2) time_embed = F.interpolate(time_embed, size=(T), mode='linear') time_embed = time_embed.transpose(1, 2) else: time_embed = self.temporal_embedding patch_embeds = self.patch_embedding(pixel_values.reshape(-1, C, H, W)) patch_embeds = patch_embeds.flatten(2).transpose(1, 2) # [B*T, H*W, C] C = patch_embeds.shape[-1] patch_embeds = patch_embeds.reshape(B, T, -1, C) if self.if_use_temporal_embed: patch_embeds = patch_embeds + time_embed.unsqueeze(2) # [B, T, H*W, C] patch_embeds = patch_embeds + self.position_embedding(self.position_ids[:, 1:]).unsqueeze(1) class_embeds = self.class_embedding.expand(B, 1, -1) class_embeds = class_embeds + self.position_embedding(self.position_ids[:, 0:1]) added_cls = self.added_cls.expand(B, self.add_cls_num, -1) added_cls = added_cls + self.position_embedding(self.position_ids[:, 0:1]) N, L = patch_embeds.shape[1], patch_embeds.shape[2] embeds = torch.cat([class_embeds, added_cls, patch_embeds.reshape(patch_embeds.shape[0], -1, patch_embeds.shape[-1])], dim=1) M = 1 + self.add_cls_num return (embeds, (M, N, L)) # [B, M+N*L, C] class CLIPTextEmbeddings(nn.Module): def __init__(self, config: CLIPTextConfig): super().__init__() embed_dim = config.hidden_size self.token_embedding = nn.Embedding(config.vocab_size, embed_dim) self.position_embedding = nn.Embedding(config.max_position_embeddings, embed_dim) # position_ids (1, len position emb) is contiguous in memory and exported when serialized self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1))) def forward( self, input_ids: Optional[torch.LongTensor] = None, position_ids: Optional[torch.LongTensor] = None, inputs_embeds: Optional[torch.FloatTensor] = None, ) -> torch.Tensor: seq_length = input_ids.shape[-1] if input_ids is not None else inputs_embeds.shape[-2] if position_ids is None: position_ids = self.position_ids[:, :seq_length] if inputs_embeds is None: inputs_embeds = self.token_embedding(input_ids) position_embeddings = self.position_embedding(position_ids) embeddings = inputs_embeds + position_embeddings return embeddings class CLIPAttention(nn.Module): """Multi-headed attention from 'Attention Is All You Need' paper""" def __init__(self, config): super().__init__() self.config = config self.embed_dim = config.hidden_size self.num_heads = config.num_attention_heads self.head_dim = self.embed_dim // self.num_heads assert ( self.head_dim * self.num_heads == self.embed_dim ), f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads})." self.scale = self.head_dim**-0.5 self.dropout = config.attention_dropout self.k_proj = nn.Linear(self.embed_dim, self.embed_dim) self.v_proj = nn.Linear(self.embed_dim, self.embed_dim) self.q_proj = nn.Linear(self.embed_dim, self.embed_dim) self.out_proj = nn.Linear(self.embed_dim, self.embed_dim) def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() def forward( self, hidden_states: torch.Tensor, inputs_size, attention_mask: Optional[torch.Tensor] = None, causal_attention_mask: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = False, ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: """Input shape: Batch x Time x Channel""" if inputs_size is not None: return self.forward2(hidden_states, inputs_size), None bsz, tgt_len, embed_dim = hidden_states.size() # get query proj query_states = self.q_proj(hidden_states) * self.scale key_states = self._shape(self.k_proj(hidden_states), -1, bsz) value_states = self._shape(self.v_proj(hidden_states), -1, bsz) proj_shape = (bsz * self.num_heads, -1, self.head_dim) query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape) key_states = key_states.view(*proj_shape) value_states = value_states.view(*proj_shape) src_len = key_states.size(1) attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len): raise ValueError( f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is {attn_weights.size()}" ) # apply the causal_attention_mask first if causal_attention_mask is not None: if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {causal_attention_mask.size()}" ) attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len) if attention_mask is not None: if attention_mask.size() != (bsz, 1, tgt_len, src_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}" ) attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len) attn_weights = nn.functional.softmax(attn_weights, dim=-1) if output_attentions: # this operation is a bit akward, but it's required to # make sure that attn_weights keeps its gradient. # In order to do so, attn_weights have to reshaped # twice and have to be reused in the following attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len) else: attn_weights_reshaped = None attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training) attn_output = torch.bmm(attn_probs, value_states) if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim): raise ValueError( f"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is {attn_output.size()}" ) attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim) attn_output = attn_output.transpose(1, 2) attn_output = attn_output.reshape(bsz, tgt_len, embed_dim) attn_output = self.out_proj(attn_output) return attn_output, attn_weights_reshaped def forward2(self, hidden_states, inputs_size): """ hidden_states: [B, M+N*L, C] inputs_size: (M, N, L) """ M, N, L = inputs_size bsz, tgt_len, embed_dim = hidden_states.size() # get query proj query_states = self.q_proj(hidden_states) * self.scale key_states = self._shape(self.k_proj(hidden_states), -1, bsz) value_states = self._shape(self.v_proj(hidden_states), -1, bsz) proj_shape = (bsz * self.num_heads, -1, self.head_dim) query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape) key_states = key_states.view(*proj_shape) value_states = value_states.view(*proj_shape) # qkv: [B*num_heads, M+N*L, head_dim] # in-frame attention: q = query_states[:, M:].reshape(-1, L, self.head_dim) #[B*num_heads*N, L, head_dim] k = key_states[:, :M].repeat(1, N, 1).reshape(-1, M, self.head_dim) #[B*num_heads*N, M, head_dim] k = torch.cat([k, key_states[:, M:].reshape(-1, L, self.head_dim)], dim=1) #[B*num_heads*N, M+L, head_dim] v = value_states[:, :M].repeat(1, N, 1).reshape(-1, M, self.head_dim) #[B*num_heads*N, M, head_dim] v = torch.cat([v, value_states[:, M:].reshape(-1, L, self.head_dim)], dim=1) #[B*num_heads*N, M+L, head_dim] attn_weights = torch.bmm(q, k.transpose(1, 2)) attn_weights = nn.functional.softmax(attn_weights, dim=-1) attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training) attn_output = torch.bmm(attn_probs, v) # [B*num_heads*N, L, head_dim] attn_output = attn_output.view(bsz, self.num_heads, N, L, self.head_dim) attn_output = attn_output.permute(0, 2, 3, 1, 4) attn_output_frames = attn_output.reshape(bsz, N*L, embed_dim) # [B, N*L, C] # cls divided attention: q = query_states[:, :M] # [B*num_heads, M, head_dim] k = key_states # [B*num_heads, M+N*L, head_dim] v = value_states # [B*num_heads, M+N*L, head_dim] attn_weights = torch.bmm(q, k.transpose(1, 2)) attn_weights = nn.functional.softmax(attn_weights, dim=-1) attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training) attn_output = torch.bmm(attn_probs, v) # [B*num_heads, M, head_dim] attn_output = attn_output.view(bsz, self.num_heads, M, self.head_dim) attn_output = attn_output.transpose(1, 2) attn_output_cls = attn_output.reshape(bsz, M, embed_dim) # [B, M, C] attn_output = torch.cat([attn_output_cls, attn_output_frames], dim=1) attn_output = self.out_proj(attn_output) return attn_output class CLIPMLP(nn.Module): def __init__(self, config): super().__init__() self.config = config self.activation_fn = ACT2FN[config.hidden_act] self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size) self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: hidden_states = self.fc1(hidden_states) hidden_states = self.activation_fn(hidden_states) hidden_states = self.fc2(hidden_states) return hidden_states class CLIPEncoderLayer(nn.Module): def __init__(self, config: CLIPConfig): super().__init__() self.embed_dim = config.hidden_size self.self_attn = CLIPAttention(config) self.layer_norm1 = nn.LayerNorm(self.embed_dim) self.mlp = CLIPMLP(config) self.layer_norm2 = nn.LayerNorm(self.embed_dim) def forward( self, hidden_states: torch.Tensor, inputs_size, attention_mask: torch.Tensor, causal_attention_mask: torch.Tensor, output_attentions: Optional[bool] = False, ) -> Tuple[torch.FloatTensor]: """ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. """ if isinstance(hidden_states, tuple): residual = hidden_states hidden_states = (self.layer_norm1(hidden_states[0]), self.layer_norm1(hidden_states[1])) hidden_states, attn_weights = self.self_attn( hidden_states=hidden_states, inputs_size=inputs_size, attention_mask=attention_mask, causal_attention_mask=causal_attention_mask, output_attentions=output_attentions, ) hidden_states = residual + hidden_states residual = hidden_states hidden_states = self.layer_norm2(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = residual + hidden_states else: residual = hidden_states hidden_states = self.layer_norm1(hidden_states) hidden_states, attn_weights = self.self_attn( hidden_states=hidden_states, inputs_size=inputs_size, attention_mask=attention_mask, causal_attention_mask=causal_attention_mask, output_attentions=output_attentions, ) hidden_states = residual + hidden_states residual = hidden_states hidden_states = self.layer_norm2(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = residual + hidden_states outputs = (hidden_states,) if output_attentions: outputs += (attn_weights,) return outputs class CLIPPreTrainedModel(PreTrainedModel): """ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. """ config_class = CLIPConfig base_model_prefix = "clip" supports_gradient_checkpointing = True _keys_to_ignore_on_load_missing = [r"position_ids"] def _init_weights(self, module): """Initialize the weights""" factor = self.config.initializer_factor if isinstance(module, CLIPTextEmbeddings): module.token_embedding.weight.data.normal_(mean=0.0, std=factor * 0.02) module.position_embedding.weight.data.normal_(mean=0.0, std=factor * 0.02) elif isinstance(module, CLIPVisionEmbeddings) or isinstance(module, CLIPVisionViPEmbeddings): factor = self.config.initializer_factor nn.init.normal_(module.class_embedding, mean=0.0, std=module.embed_dim**-0.5 * factor) nn.init.normal_(module.patch_embedding.weight, std=module.config.initializer_range * factor) nn.init.normal_(module.position_embedding.weight, std=module.config.initializer_range * factor) elif isinstance(module, CLIPAttention): factor = self.config.initializer_factor in_proj_std = (module.embed_dim**-0.5) * ((2 * module.config.num_hidden_layers) ** -0.5) * factor out_proj_std = (module.embed_dim**-0.5) * factor nn.init.normal_(module.q_proj.weight, std=in_proj_std) nn.init.normal_(module.k_proj.weight, std=in_proj_std) nn.init.normal_(module.v_proj.weight, std=in_proj_std) nn.init.normal_(module.out_proj.weight, std=out_proj_std) elif isinstance(module, CLIPMLP): factor = self.config.initializer_factor in_proj_std = ( (module.config.hidden_size**-0.5) * ((2 * module.config.num_hidden_layers) ** -0.5) * factor ) fc_std = (2 * module.config.hidden_size) ** -0.5 * factor nn.init.normal_(module.fc1.weight, std=fc_std) nn.init.normal_(module.fc2.weight, std=in_proj_std) elif isinstance(module, CLIPModel): nn.init.normal_( module.text_projection.weight, std=module.text_embed_dim**-0.5 * self.config.initializer_factor, ) nn.init.normal_( module.visual_projection.weight, std=module.vision_embed_dim**-0.5 * self.config.initializer_factor, ) if isinstance(module, nn.LayerNorm): module.bias.data.zero_() module.weight.data.fill_(1.0) if isinstance(module, nn.Linear) and module.bias is not None: module.bias.data.zero_() def _set_gradient_checkpointing(self, module, value=False): if isinstance(module, CLIPEncoder): module.gradient_checkpointing = value CLIP_START_DOCSTRING = r""" This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. """ CLIP_TEXT_INPUTS_DOCSTRING = r""" Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ CLIP_VISION_INPUTS_DOCSTRING = r""" Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`CLIPFeatureExtractor`]. See [`CLIPFeatureExtractor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ CLIP_INPUTS_DOCSTRING = r""" Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`CLIPFeatureExtractor`]. See [`CLIPFeatureExtractor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ class CLIPEncoder(nn.Module): """ Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig """ def __init__(self, config: CLIPConfig): super().__init__() self.config = config self.layers = nn.ModuleList([CLIPEncoderLayer(config) for _ in range(config.num_hidden_layers)]) self.gradient_checkpointing = False def forward( self, inputs_embeds, inputs_size = None, attention_mask: Optional[torch.Tensor] = None, causal_attention_mask: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutput]: r""" Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict encoder_states = () if output_hidden_states else None all_attentions = () if output_attentions else None hidden_states = inputs_embeds for idx, encoder_layer in enumerate(self.layers): if output_hidden_states: encoder_states = encoder_states + (hidden_states,) if self.gradient_checkpointing and self.training: def create_custom_forward(module): def custom_forward(*inputs): return module(*inputs, output_attentions) return custom_forward layer_outputs = torch.utils.checkpoint.checkpoint( create_custom_forward(encoder_layer), hidden_states, inputs_size, attention_mask, causal_attention_mask, ) else: layer_outputs = encoder_layer( hidden_states, inputs_size, attention_mask, causal_attention_mask, output_attentions=output_attentions, ) hidden_states = layer_outputs[0] if output_attentions: all_attentions = all_attentions + (layer_outputs[1],) if output_hidden_states: encoder_states = encoder_states + (hidden_states,) if not return_dict: return tuple(v for v in [hidden_states, encoder_states, all_attentions] if v is not None) return BaseModelOutput( last_hidden_state=hidden_states, hidden_states=encoder_states, attentions=all_attentions ) class CLIPTextTransformer(nn.Module): def __init__(self, config: CLIPTextConfig): super().__init__() self.config = config embed_dim = config.hidden_size self.embeddings = CLIPTextEmbeddings(config) self.encoder = CLIPEncoder(config) self.final_layer_norm = nn.LayerNorm(embed_dim) @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPTextConfig) def forward( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict if input_ids is None: raise ValueError("You have to specify either input_ids") input_shape = input_ids.size() input_ids = input_ids.view(-1, input_shape[-1]) hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids) bsz, seq_len = input_shape # CLIP's text model uses causal mask, prepare it here. # https://github.com/openai/CLIP/blob/cfcffb90e69f37bf2ff1e988237a0fbe41f33c04/clip/model.py#L324 if_fp16 = hidden_states.dtype == torch.float16 causal_attention_mask = self._build_causal_attention_mask(bsz, seq_len, fp16=if_fp16).to(hidden_states.device) # expand attention_mask if attention_mask is not None: # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] attention_mask = _expand_mask(attention_mask, hidden_states.dtype) encoder_outputs = self.encoder( inputs_embeds=hidden_states, attention_mask=attention_mask, causal_attention_mask=causal_attention_mask, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) last_hidden_state = encoder_outputs[0] last_hidden_state = self.final_layer_norm(last_hidden_state) # text_embeds.shape = [batch_size, sequence_length, transformer.width] # take features from the eot embedding (eot_token is the highest number in each sequence) pooled_output = last_hidden_state[torch.arange(last_hidden_state.shape[0]), input_ids.argmax(dim=-1)] if not return_dict: return (last_hidden_state, pooled_output) + encoder_outputs[1:] return BaseModelOutputWithPooling( last_hidden_state=last_hidden_state, pooler_output=pooled_output, hidden_states=encoder_outputs.hidden_states, attentions=encoder_outputs.attentions, ) def _build_causal_attention_mask(self, bsz, seq_len, fp16=False): # lazily create causal attention mask, with full attention between the vision tokens # pytorch uses additive attention mask; fill with -inf mask = torch.empty(bsz, seq_len, seq_len) mask.fill_(float("-inf")) mask.triu_(1) # zero out the lower diagonal mask = mask.unsqueeze(1) # expand mask if fp16: mask = mask.half() return mask class CLIPTextModel(CLIPPreTrainedModel): config_class = CLIPTextConfig def __init__(self, config: CLIPTextConfig): super().__init__(config) self.text_model = CLIPTextTransformer(config) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self) -> nn.Module: return self.text_model.embeddings.token_embedding def set_input_embeddings(self, value): self.text_model.embeddings.token_embedding = value @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPTextConfig) def forward( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: Examples: ```python >>> from transformers import CLIPTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```""" return self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) class CLIPVisionTransformer(nn.Module): def __init__(self, config: CLIPVisionConfig, additional_vision_config=None): super().__init__() self.config = config embed_dim = config.hidden_size self.embeddings = CLIPVisionViPEmbeddings(config, additional_vision_config) self.pre_layrnorm = nn.LayerNorm(embed_dim) self.encoder = CLIPEncoder(config) self.post_layernorm = nn.LayerNorm(embed_dim) @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPVisionConfig) def forward( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: """ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict if pixel_values is None: raise ValueError("You have to specify pixel_values") hidden_states, inputs_size = self.embeddings(pixel_values) # tuple hidden_states = self.pre_layrnorm(hidden_states) encoder_outputs = self.encoder( inputs_embeds=hidden_states, inputs_size=inputs_size, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) last_hidden_state = encoder_outputs[0] pooled_output = last_hidden_state[:, 0, :] pooled_output = self.post_layernorm(pooled_output) if not return_dict: return (last_hidden_state, pooled_output) + encoder_outputs[1:] return BaseModelOutputWithPooling( last_hidden_state=last_hidden_state, pooler_output=pooled_output, hidden_states=encoder_outputs.hidden_states, attentions=encoder_outputs.attentions, ) class CLIPVisionModel(CLIPPreTrainedModel): config_class = CLIPVisionConfig main_input_name = "pixel_values" def __init__(self, config: CLIPVisionConfig): super().__init__(config) self.vision_model = CLIPVisionTransformer(config) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self) -> nn.Module: return self.vision_model.embeddings.patch_embedding @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=CLIPVisionConfig) def forward( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPooling]: r""" Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```""" return self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) @add_start_docstrings(CLIP_START_DOCSTRING) class CLIPModel(CLIPPreTrainedModel): config_class = CLIPConfig def __init__(self, config: CLIPConfig): super().__init__(config) if not isinstance(config.text_config, CLIPTextConfig): raise ValueError( f"config.text_config is expected to be of type CLIPTextConfig but is of type {type(config.text_config)}." ) if not isinstance(config.vision_config, CLIPVisionConfig): raise ValueError( f"config.vision_config is expected to be of type CLIPVisionConfig but is of type {type(config.vision_config)}." ) text_config = config.text_config vision_config = config.vision_config if hasattr(config, "vision_additional_config"): additional_vision_config = config.vision_additional_config else: additional_vision_config = None self.projection_dim = config.projection_dim self.text_embed_dim = text_config.hidden_size self.vision_embed_dim = vision_config.hidden_size self.text_model = CLIPTextTransformer(text_config) self.vision_model = CLIPVisionTransformer(vision_config, additional_vision_config) self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False) self.text_projection = nn.Linear(self.text_embed_dim, self.projection_dim, bias=False) self.logit_scale = nn.Parameter(torch.ones([]) * self.config.logit_scale_init_value) # Initialize weights and apply final processing self.post_init() @add_start_docstrings_to_model_forward(CLIP_TEXT_INPUTS_DOCSTRING) def get_text_features( self, input_ids: Optional[torch.Tensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.Tensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, if_norm: Optional[bool] = None, ) -> torch.FloatTensor: r""" Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> from transformers import CLIPTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict text_outputs = self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) pooled_output = text_outputs[1] text_features = self.text_projection(pooled_output) if_norm = if_norm if if_norm is not None else False if if_norm: text_features = text_features / text_features.norm(dim=-1, keepdim=True) return text_features @add_start_docstrings_to_model_forward(CLIP_VISION_INPUTS_DOCSTRING) def get_image_features( self, pixel_values: Optional[torch.FloatTensor] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, if_norm: Optional[bool] = None, ) -> torch.FloatTensor: r""" Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict vision_outputs = self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) pooled_output = vision_outputs[1] # pooled_output image_features = self.visual_projection(pooled_output) if_norm = if_norm if if_norm is not None else False if if_norm: image_features = image_features / image_features.norm(dim=-1, keepdim=True) return image_features @add_start_docstrings_to_model_forward(CLIP_INPUTS_DOCSTRING) @replace_return_docstrings(output_type=CLIPOutput, config_class=CLIPConfig) def forward( self, input_ids: Optional[torch.LongTensor] = None, pixel_values: Optional[torch.FloatTensor] = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, return_loss: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, CLIPOutput]: r""" Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import CLIPProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```""" # Use CLIP model's config for some fields (if specified) instead of those of vision & text components. output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict vision_outputs = self.vision_model( pixel_values=pixel_values, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) text_outputs = self.text_model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) image_embeds = vision_outputs[1] image_embeds = self.visual_projection(image_embeds) text_embeds = text_outputs[1] text_embeds = self.text_projection(text_embeds) # normalized features image_embeds = image_embeds / image_embeds.norm(dim=-1, keepdim=True) text_embeds = text_embeds / text_embeds.norm(dim=-1, keepdim=True) # cosine similarity as logits logit_scale = self.logit_scale.exp() logits_per_text = torch.matmul(text_embeds, image_embeds.t()) * logit_scale logits_per_image = logits_per_text.T loss = None if return_loss: loss = clip_loss(logits_per_text) if not return_dict: output = (logits_per_image, logits_per_text, text_embeds, image_embeds, text_outputs, vision_outputs) return ((loss,) + output) if loss is not None else output return CLIPOutput( loss=loss, logits_per_image=logits_per_image, logits_per_text=logits_per_text, text_embeds=text_embeds, image_embeds=image_embeds, text_model_output=text_outputs, vision_model_output=vision_outputs, ) def split_cls(x, size=(8, 49)): """ x [B, 1+N*L, C] size [N, L] return [B, 1, C], [B, N, L, C] """ B = x.shape[0] N, L = size return x[:, 0:1], x[:, 1:].reshape(B, N, L, -1) def merge_cls(x1, x2): return torch.cat([x1, x2.reshape(x2.shape[0], -1, x2.shape[-1])], dim=1) clip-vip/XPretrain-main/CLIP-Vip/src/modeling/VidCLIP.py000066400000000000000000000106021475357401500230650ustar00rootroot00000000000000import torch import torch.nn as nn from functools import partial from transformers.models.clip.configuration_clip import CLIPConfig, CLIPTextConfig, CLIPVisionConfig from src.modeling.CLIP_ViP import CLIPModel, clip_loss from src.modeling.CLIP import CLIPModel as CLIP class VidCLIP(nn.Module): def __init__(self, args): super(VidCLIP, self).__init__() clipconfig = CLIPConfig.from_pretrained(args.clip_config) setattr(clipconfig, "vision_additional_config", args.clip_vision_additional_config) self.vision_additional_config = args.clip_vision_additional_config if args.clip_weights: if self.vision_additional_config.type == "ViP": self.clipmodel = CLIPModel.from_pretrained(args.clip_weights, config=clipconfig) else: self.clipmodel = CLIP.from_pretrained(args.clip_weights, config=clipconfig) else: if self.vision_additional_config.type == "ViP": self.clipmodel = CLIPModel(clipconfig) else: self.clipmodel = CLIP(clipconfig) # init logit scale from logit_scale_value = self.vision_additional_config.logit_scale_init_value self.clipmodel.logit_scale.data.fill_(logit_scale_value) def overload_logit_scale(self, overload_logit_scale): self.clipmodel.logit_scale.data.fill_(overload_logit_scale) def forward(self, video, text_input_ids, text_input_mask, \ image=None, caption_ids=None, caption_masks=None): """ video [B, n_clips*num_frms, C, H, W] text_input_ids [B, L] text_input_mask [B, L] image [B, img_num, C, H, W] caption_ids [B, img_num, L] caption_masks [B, img_num, L] """ B, N, C, H, W = video.shape if self.vision_additional_config.type == "ViP": inputs = {"input_ids": text_input_ids, "attention_mask": text_input_mask, "pixel_values": video, "return_loss": False} outputs = self.clipmodel(**inputs) results = {} results["text_features"] = outputs["text_embeds"] results["vis_features"] = outputs["image_embeds"] # results["loss"] = outputs["loss"] else: video = video.reshape(-1, C, H, W) inputs = {"input_ids": text_input_ids, "attention_mask": text_input_mask, "pixel_values": video} outputs = self.clipmodel(**inputs) vis_features = outputs["vision_model_output"][1] vis_features = self.clipmodel.visual_projection(vis_features) vis_features = vis_features / vis_features.norm(dim=-1, keepdim=True) vis_features = vis_features.reshape(B, N, -1).mean(1) vis_features = vis_features / vis_features.norm(dim=-1, keepdim=True) results = {} results["text_features"] = outputs["text_embeds"] results["vis_features"] = vis_features if image is not None: B, img_num, C, H, W = image.shape L = caption_ids.shape[-1] inputs = {"input_ids": caption_ids.reshape(-1, L), "attention_mask": caption_masks.reshape(-1, L), "pixel_values": image.reshape(-1, 1, C, H, W), "return_loss": False} outputs = self.clipmodel(**inputs) results["img_features"] = outputs["image_embeds"] results["cap_features"] = outputs["text_embeds"] return results def forward_video(self, video): inputs = {"pixel_values": video, "if_norm": True} video_features = self.clipmodel.get_image_features(**inputs) return video_features def forward_text(self, text_input_ids, text_input_mask): inputs = {"input_ids": text_input_ids, "attention_mask": text_input_mask, "if_norm": True} text_features = self.clipmodel.get_text_features(**inputs) return text_features def freeze_text_encoder(self, freeze_text_proj): freeze_list = [self.clipmodel.text_model] if freeze_text_proj: freeze_list.append(self.clipmodel.text_projection) for m in freeze_list: m.eval() for param in m.parameters(): param.requires_grad = False clip-vip/XPretrain-main/CLIP-Vip/src/modeling/__init__.py000066400000000000000000000020001475357401500234230ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/optimization/000077500000000000000000000000001475357401500222525ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/optimization/__init__.py000066400000000000000000000020001475357401500243530ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/optimization/adamw.py000066400000000000000000000104631475357401500237210ustar00rootroot00000000000000""" AdamW optimizer (weight decay fix) copied from hugginface """ import math import torch from torch.optim import Optimizer class AdamW(Optimizer): """ Implements Adam algorithm with weight decay fix. Parameters: lr (float): learning rate. Default 1e-3. betas (tuple of 2 floats): Adams beta parameters (b1, b2). Default: (0.9, 0.999) eps (float): Adams epsilon. Default: 1e-6 weight_decay (float): Weight decay. Default: 0.0 correct_bias (bool): can be set to False to avoid correcting bias in Adam (e.g. like in Bert TF repository). Default True. """ def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-6, weight_decay=0.0, correct_bias=True): if lr < 0.0: raise ValueError( "Invalid learning rate: {} - should be >= 0.0".format(lr)) if not 0.0 <= betas[0] < 1.0: raise ValueError("Invalid beta parameter: {} - " "should be in [0.0, 1.0[".format(betas[0])) if not 0.0 <= betas[1] < 1.0: raise ValueError("Invalid beta parameter: {} - " "should be in [0.0, 1.0[".format(betas[1])) if not 0.0 <= eps: raise ValueError("Invalid epsilon value: {} - " "should be >= 0.0".format(eps)) defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, correct_bias=correct_bias) super(AdamW, self).__init__(params, defaults) def step(self, closure=None): """Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. """ loss = None if closure is not None: loss = closure() for group in self.param_groups: for p in group['params']: if p.grad is None: continue grad = p.grad.data if grad.is_sparse: raise RuntimeError( 'Adam does not support sparse ' 'gradients, please consider SparseAdam instead') state = self.state[p] # State initialization if len(state) == 0: state['step'] = 0 # Exponential moving average of gradient values state['exp_avg'] = torch.zeros_like(p.data) # Exponential moving average of squared gradient values state['exp_avg_sq'] = torch.zeros_like(p.data) exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] beta1, beta2 = group['betas'] state['step'] += 1 # Decay the first and second moment running average coefficient # In-place operations to update the averages at the same time exp_avg.mul_(beta1).add_(1.0 - beta1, grad) exp_avg_sq.mul_(beta2).addcmul_(1.0 - beta2, grad, grad) denom = exp_avg_sq.sqrt().add_(group['eps']) step_size = group['lr'] if group['correct_bias']: # No bias correction for Bert bias_correction1 = 1.0 - beta1 ** state['step'] bias_correction2 = 1.0 - beta2 ** state['step'] step_size = (step_size * math.sqrt(bias_correction2) / bias_correction1) p.data.addcdiv_(-step_size, exp_avg, denom) # Just adding the square of the weights to the loss function is # *not* the correct way of using L2 regularization/weight decay # with Adam, since that will interact with the m and v # parameters in strange ways. # # Instead we want to decay the weights in a manner that doesn't # interact with the m/v parameters. This is equivalent to # adding the square of the weights to the loss with plain # (non-momentum) SGD. # Add weight decay at the end (fixed version) if group['weight_decay'] > 0.0: p.data.add_(-group['lr'] * group['weight_decay'], p.data) return loss clip-vip/XPretrain-main/CLIP-Vip/src/optimization/loss.py000066400000000000000000000316321475357401500236110ustar00rootroot00000000000000import torch import torch.nn.functional as F from torch import nn from torch.autograd import Variable def cosine_sim(im, s): """Cosine similarity between all the image and sentence pairs """ return im.mm(s.t()) def order_sim(im, s): """Order embeddings similarity measure $max(0, s-im)$ """ YmX = (s.unsqueeze(1).expand(s.size(0), im.size(0), s.size(1)) - im.unsqueeze(0).expand(s.size(0), im.size(0), s.size(1))) score = -YmX.clamp(min=0).pow(2).sum(2).sqrt().t() return score class TripletContrastiveLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(TripletContrastiveLoss, self).__init__() self.margin = cfg.margin if cfg.measure == 'order': self.sim = order_sim else: self.sim = cosine_sim self.max_violation = cfg.max_violation def forward(self, im, s): # compute image-sentence score matrix scores = self.sim(im, s) diagonal = scores.diag().view(im.size(0), 1) d1 = diagonal.expand_as(scores) d2 = diagonal.t().expand_as(scores) # compare every diagonal score to scores in its column # caption retrieval cost_s = (self.margin + scores - d1).clamp(min=0) # compare every diagonal score to scores in its row # image retrieval cost_im = (self.margin + scores - d2).clamp(min=0) # clear diagonals mask = torch.eye(scores.size(0)) > .5 I = Variable(mask) if torch.cuda.is_available(): I = I.cuda() cost_s = cost_s.masked_fill_(I, 0) cost_im = cost_im.masked_fill_(I, 0) # keep the maximum violating negative for each query if self.max_violation: cost_s = cost_s.max(1)[0] cost_im = cost_im.max(0)[0] return cost_s.sum() + cost_im.sum() class NCEContrastiveLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(NCEContrastiveLoss, self).__init__() self.temp = cfg.temp def forward(self, vis_feat, text_feat): t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) / self.temp # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label)).mean() return loss class HardNegLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(HardNegLoss, self).__init__() self.hard_negative_num = cfg.hard_negative_num def forward(self, vis_feat, text_feat): sim_matrix = torch.matmul(text_feat, vis_feat.permute(1, 0)) # temperature bsz = sim_matrix.shape[0] retrieval_mask = torch.eye(bsz, dtype=torch.long).to(device=sim_matrix.device) hard_neg_t2v = torch.topk(sim_matrix-10000*retrieval_mask, self.hard_negative_num, dim=1)[0] hard_neg_v2t = torch.topk(sim_matrix.transpose(0, 1)-10000*retrieval_mask, self.hard_negative_num, dim=1)[0] sample_t2v = torch.cat([sim_matrix.diag().view(-1, 1), hard_neg_t2v], -1) sample_v2t = torch.cat([sim_matrix.diag().view(-1, 1), hard_neg_v2t], -1) retrieval_label = torch.zeros(bsz, dtype=torch.long).to(device=sim_matrix.device) loss = (F.cross_entropy(sample_t2v, retrieval_label) + F.cross_entropy(sample_v2t, retrieval_label)).mean() return loss class MILNCEContrastiveLoss(nn.Module): def __init__(self,cfg): super(MILNCEContrastiveLoss, self).__init__() self.temp = cfg.temp def forward(self, video_embd, text_embd): x = torch.matmul(video_embd, text_embd.t()) / self.temp x = x.view(video_embd.shape[0], video_embd.shape[0], -1) nominator = x * torch.eye(x.shape[0])[:,:,None].to(x.device) nominator = nominator.sum(dim=1) nominator = torch.logsumexp(nominator, dim=1) mask = (1-torch.eye(x.shape[0])[:,:,None].to(x.device).repeat(1,1,x.shape[-1])) denominator = torch.cat((x[mask>0].reshape(x.shape[0], x.shape[1]-1, x.shape[2]), x.permute(1,0,2)), dim=1).view(x.shape[0], -1) denominator = torch.logsumexp(denominator, dim=1) return torch.mean(denominator - nominator) class NCELearnableTempLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(NCELearnableTempLoss, self).__init__() def forward(self, vis_feat, text_feat, temp): logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label)).mean() return loss class VidImgNCELearnableTempLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(VidImgNCELearnableTempLoss, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): vis_feat = torch.cat([vis_feat, img_feat], dim=0) text_feat = torch.cat([text_feat, cap_feat], dim=0) logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label)).mean() return loss class VidImgDivideNCELearnableTempLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(VidImgDivideNCELearnableTempLoss, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label t2v_2 = torch.matmul(img_feat, cap_feat.permute(1, 0)) * logit_scale # temperature v2t_2 = t2v_2.permute(1, 0) t2v_label_2 = torch.arange(t2v_2.shape[0], device=t2v_2.device) v2t_label_2 = t2v_label_2 loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label) +\ F.cross_entropy(t2v_2, t2v_label_2) + F.cross_entropy(v2t_2, v2t_label_2)).mean() return loss class NCELearnableTempDSLLoss(nn.Module): """ Compute contrastive loss """ def __init__(self, cfg): super(NCELearnableTempDSLLoss, self).__init__() def forward(self, vis_feat, text_feat, temp): logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v = t2v * F.softmax(t2v, dim=0) v2t = v2t * F.softmax(v2t, dim=0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label)).mean() return loss class NCELearnableTempLoss_vs_vc(nn.Module): """ Compute contrastive loss: video-sub + video-cap """ def __init__(self, cfg): super(NCELearnableTempLoss_vs_vc, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label t2v_2 = torch.matmul(vis_feat, cap_feat.permute(1, 0)) * logit_scale # temperature v2t_2 = t2v_2.permute(1, 0) t2v_label_2 = torch.arange(t2v_2.shape[0], device=t2v_2.device) v2t_label_2 = t2v_label_2 loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label) + \ F.cross_entropy(t2v_2, t2v_label_2) + F.cross_entropy(v2t_2, v2t_label_2)).mean() return loss class NCELearnableTempLoss_vs_vc_fc(nn.Module): """ Compute contrastive loss: video-sub + video-cap """ def __init__(self, cfg): super(NCELearnableTempLoss_vs_vc_fc, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): logit_scale = temp.exp() t2v = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature v2t = t2v.permute(1, 0) t2v_label = torch.arange(t2v.shape[0], device=t2v.device) v2t_label = t2v_label t2v_2 = torch.matmul(vis_feat, cap_feat.permute(1, 0)) * logit_scale # temperature v2t_2 = t2v_2.permute(1, 0) t2v_label_2 = torch.arange(t2v_2.shape[0], device=t2v_2.device) v2t_label_2 = t2v_label_2 t2v_3 = torch.matmul(img_feat, cap_feat.permute(1, 0)) * logit_scale # temperature v2t_3 = t2v_3.permute(1, 0) t2v_label_3 = torch.arange(t2v_3.shape[0], device=t2v_3.device) v2t_label_3 = t2v_label_3 loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(v2t, v2t_label) + \ F.cross_entropy(t2v_2, t2v_label_2) + F.cross_entropy(v2t_2, v2t_label_2) +\ F.cross_entropy(t2v_3, t2v_label_3) + F.cross_entropy(v2t_3, v2t_label_3)).mean() return loss class NCELearnableTempLoss_vsc(nn.Module): """ Compute contrastive loss: video-(sub,cap) """ def __init__(self, cfg): super(NCELearnableTempLoss_vsc, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): assert text_feat.shape[0] == cap_feat.shape[0] logit_scale = temp.exp() v2t = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature t2v = v2t.permute(1, 0) t2v_label = torch.arange(v2t.shape[0], device=v2t.device) v2t_2 = torch.matmul(vis_feat, cap_feat.permute(1, 0)) * logit_scale # temperature t2v_2 = v2t_2.permute(1, 0) t2v_label_2 = torch.arange(v2t_2.shape[0], device=v2t_2.device) diag = torch.eye(v2t.shape[0], dtype=torch.bool).to(v2t.device) v2t_pos = v2t[diag].reshape(v2t.shape[0], 1) v2t_neg = v2t[~diag].reshape(v2t.shape[0], -1) v2t_pos_2 = v2t_2[diag].reshape(v2t_2.shape[0], 1) v2t_neg_2 = v2t_2[~diag].reshape(v2t_2.shape[0], -1) v2t = torch.cat([v2t_pos, v2t_neg, v2t_neg_2], dim=1) v2t_2 = torch.cat([v2t_pos_2, v2t_neg, v2t_neg_2], dim=1) v2t_label = torch.zeros(v2t.shape[0], dtype=torch.long).to(v2t.device) loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(t2v_2, t2v_label_2) + \ F.cross_entropy(v2t, v2t_label) + F.cross_entropy(v2t_2, v2t_label)).mean() return loss class NCELearnableTempLoss_vsc_fc(nn.Module): """ Compute contrastive loss: video-(sub,cap) """ def __init__(self, cfg): super(NCELearnableTempLoss_vsc_fc, self).__init__() def forward(self, vis_feat, text_feat, img_feat, cap_feat, temp): assert text_feat.shape[0] == cap_feat.shape[0] logit_scale = temp.exp() v2t = torch.matmul(vis_feat, text_feat.permute(1, 0)) * logit_scale # temperature t2v = v2t.permute(1, 0) t2v_label = torch.arange(v2t.shape[0], device=v2t.device) v2t_2 = torch.matmul(vis_feat, cap_feat.permute(1, 0)) * logit_scale # temperature t2v_2 = v2t_2.permute(1, 0) t2v_label_2 = torch.arange(v2t_2.shape[0], device=v2t_2.device) diag = torch.eye(v2t.shape[0], dtype=torch.bool).to(v2t.device) v2t_pos = v2t[diag].reshape(v2t.shape[0], 1) v2t_neg = v2t[~diag].reshape(v2t.shape[0], -1) v2t_pos_2 = v2t_2[diag].reshape(v2t_2.shape[0], 1) v2t_neg_2 = v2t_2[~diag].reshape(v2t_2.shape[0], -1) v2t = torch.cat([v2t_pos, v2t_neg, v2t_neg_2], dim=1) v2t_2 = torch.cat([v2t_pos_2, v2t_neg, v2t_neg_2], dim=1) v2t_label = torch.zeros(v2t.shape[0], dtype=torch.long).to(v2t.device) v2t_3 = torch.matmul(img_feat, cap_feat.permute(1, 0)) * logit_scale # temperature t2v_3 = v2t_3.permute(1, 0) t2v_label_3 = torch.arange(t2v_3.shape[0], device=t2v_3.device) v2t_label_3 = t2v_label_3 loss = (F.cross_entropy(t2v, t2v_label) + F.cross_entropy(t2v_2, t2v_label_2) + \ F.cross_entropy(v2t, v2t_label) + F.cross_entropy(v2t_2, v2t_label) + \ F.cross_entropy(t2v_3, t2v_label_3) + F.cross_entropy(v2t_3, v2t_label_3)).mean() return loss def build_loss_func(cfg): loss_func = globals()[cfg.loss_name](cfg) return loss_func if __name__ == '__main__': from easydict import EasyDict as edict cfg = edict({'loss_name':'MILNCELoss', 'temp':0.05}) print(cfg.loss_name) loss_func = build_loss_func(cfg) print(loss_func.temp) video_embd = torch.randn(64,1024) text_embd = torch.randn(1280,1024) print(loss_func(video_embd, text_embd)) clip-vip/XPretrain-main/CLIP-Vip/src/optimization/sched.py000066400000000000000000000055001475357401500237120ustar00rootroot00000000000000""" optimizer learning rate scheduling helpers """ import math from math import ceil from collections import Counter def noam_schedule(step, warmup_step=4000): if step <= warmup_step: return step / warmup_step return (warmup_step ** 0.5) * (step ** -0.5) def warmup_linear(step, warmup_step, tot_step): if step < warmup_step: return step / warmup_step return max(0, (tot_step-step)/(tot_step-warmup_step)) def warmup_cosine(step, warmup_step, tot_step): if step < warmup_step: return step / warmup_step progress = (step - warmup_step) / (tot_step - warmup_step) return 0.5 * (1.0 + math.cos(math.pi * progress)) def multi_step_schedule(n_epoch, milestones, step, warmup_step,gamma=0.5): if step <= warmup_step: return step / warmup_step milestones = list(sorted(milestones)) for i, m in enumerate(milestones): if n_epoch < m: return gamma**i return gamma**(len(milestones)+1) class AutoStep(): def __init__(self, tolerance, gamma): self.tolerance = tolerance self.coeff_mem = 1 self.gamma = gamma self.best_score = 0. self.count = 0 def step(self, score): if score <= self.best_score: self.count += 1 else: self.count = 0 self.best_score = score if self.count > self.tolerance: self.count = 0 self.coeff_mem = self.coeff_mem * self.gamma def get_lr(self, global_step, learning_rate, num_train_steps, warmup_ratio=0.1): warmup_steps = int(warmup_ratio * num_train_steps) if global_step <= warmup_steps: return learning_rate * global_step / warmup_steps return max(self.coeff_mem * learning_rate, 1e-8) def get_lr_sched(global_step, decay, learning_rate, num_train_steps, warmup_ratio=0.1, decay_epochs=[], multi_step_epoch=-1): warmup_steps = int(warmup_ratio*num_train_steps) if decay == 'linear': lr_this_step = learning_rate * warmup_linear( global_step, warmup_steps, num_train_steps) elif decay == 'cosine': lr_this_step = learning_rate * warmup_cosine( global_step, warmup_steps, num_train_steps) elif decay == 'invsqrt': lr_this_step = learning_rate * noam_schedule( global_step, warmup_steps) elif decay == 'constant': lr_this_step = learning_rate elif decay == "multi_step": assert multi_step_epoch >= 0 lr_this_step = learning_rate * multi_step_schedule( multi_step_epoch, decay_epochs, global_step, warmup_steps) if lr_this_step <= 0: # save guard for possible miscalculation of train steps lr_this_step = 1e-8 return lr_this_step clip-vip/XPretrain-main/CLIP-Vip/src/optimization/utils.py000066400000000000000000000143361475357401500237730ustar00rootroot00000000000000from torch.optim import Adam, Adamax, SGD from src.optimization.adamw import AdamW def setup_optimizer(model, opts, model_type="transformer"): """model_type: str, one of [transformer, cnn]""" if model_type == "transformer": param_optimizer = list(model.named_parameters()) no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] optimizer_grouped_parameters = build_optimizer_w_lr_mul( param_optimizer, opts.learning_rate, opts.weight_decay, no_decay=no_decay, lr_mul=opts.transformer_lr_mul, lr_mul_prefix=opts.transformer_lr_mul_prefix) if opts.optim == 'adam': OptimCls = Adam elif opts.optim == 'adamax': OptimCls = Adamax elif opts.optim == 'adamw': OptimCls = AdamW else: raise ValueError('invalid optimizer') optimizer = OptimCls(optimizer_grouped_parameters, lr=opts.learning_rate, betas=opts.betas) else: assert model_type == "cnn" parameters = list(model.named_parameters()) if opts.cnn_optim == "sgd": optimizer_grouped_parameters = build_optimizer_w_lr_mul( parameters, opts.cnn_learning_rate, opts.cnn_weight_decay, lr_mul=opts.cnn_lr_mul, lr_mul_prefix=opts.cnn_lr_mul_prefix) optimizer = SGD(optimizer_grouped_parameters, lr=opts.cnn_learning_rate, momentum=opts.cnn_sgd_momentum, weight_decay=opts.cnn_weight_decay) elif opts.cnn_optim == "adamw": no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] optimizer_grouped_parameters = build_optimizer_w_lr_mul( parameters, opts.cnn_learning_rate, opts.cnn_weight_decay, no_decay=no_decay, lr_mul=opts.cnn_lr_mul, lr_mul_prefix=opts.cnn_lr_mul_prefix) optimizer = AdamW( optimizer_grouped_parameters, lr=opts.cnn_learning_rate, betas=opts.betas) else: raise ValueError("Only support SGD/adamW for cnn.") return optimizer def build_optimizer_w_lr_mul(model_param_optimizer, learning_rate, weight_decay, no_decay=[], lr_mul=1, lr_mul_prefix=""): # Prepare optimizer if lr_mul_prefix == "": param_optimizer = model_param_optimizer param_top = [] else: # top layer has larger learning rate param_top = [(n, p) for n, p in model_param_optimizer if lr_mul_prefix in n and p.requires_grad] param_optimizer = [(n, p) for n, p in model_param_optimizer if lr_mul_prefix not in n and p.requires_grad] optimizer_grouped_parameters = [] if len(param_top): optimizer_grouped_parameters.append( {'params': [p for n, p in param_top if not any(nd in n for nd in no_decay)], 'lr': lr_mul*learning_rate, 'weight_decay': weight_decay}) if len(no_decay): optimizer_grouped_parameters.append( {'params': [p for n, p in param_top if any(nd in n for nd in no_decay)], 'lr': lr_mul*learning_rate, 'weight_decay': 0.0}) if len(param_optimizer): optimizer_grouped_parameters.append( {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': weight_decay}) if len(no_decay): optimizer_grouped_parameters.append( {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}) return optimizer_grouped_parameters def setup_e2e_optimizer(model, opts): """model_type: str, one of [transformer, cnn]""" all_param_optimizer = [ (n, p) for n, p in list(model.named_parameters())] all_grouped_parameters = build_e2e_optimizer_w_lr_mul( all_param_optimizer, opts.learning_rate, opts.weight_decay, lr_mul=opts.lr_mul, lr_mul_prefix=opts.lr_mul_prefix) optimizer_grouped_parameters = [] optimizer_grouped_parameters.extend(all_grouped_parameters) if opts.optim == 'adam': OptimCls = Adam elif opts.optim == 'adamax': OptimCls = Adamax elif opts.optim == 'adamw': OptimCls = AdamW else: raise ValueError('invalid optimizer') optimizer = OptimCls(optimizer_grouped_parameters, lr=opts.learning_rate, betas=opts.betas) return optimizer def build_e2e_optimizer_w_lr_mul( model_param_optimizer, learning_rate, weight_decay, lr_mul=1, lr_mul_prefix=""): no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight', 'logit_scale'] # Prepare optimizer if lr_mul_prefix == "": param_optimizer = model_param_optimizer param_top = [] else: # top layer has larger learning rate param_top = [(n, p) for n, p in model_param_optimizer if lr_mul_prefix in n and p.requires_grad] param_optimizer = [(n, p) for n, p in model_param_optimizer if lr_mul_prefix not in n and p.requires_grad] optimizer_grouped_parameters = [ {'params': [p for n, p in param_top if not any(nd in n for nd in no_decay)], 'lr': lr_mul*learning_rate, 'weight_decay': weight_decay}, {'params': [p for n, p in param_top if any(nd in n for nd in no_decay)], 'lr': lr_mul*learning_rate, 'weight_decay': 0.0}, {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': weight_decay}, {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}] return optimizer_grouped_parameters clip-vip/XPretrain-main/CLIP-Vip/src/pretrain/000077500000000000000000000000001475357401500213505ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/pretrain/run_pretrain.py000066400000000000000000000501171475357401500244360ustar00rootroot00000000000000import os import time import random import math from collections import defaultdict import numpy as np from tqdm import tqdm from os.path import join, exists from easydict import EasyDict as edict import pprint import torch from torch.utils.data import DataLoader import torch.nn.functional as F from torch.nn.utils import clip_grad_norm_ import torchvision.utils as tutils from torch.utils.data.distributed import DistributedSampler from apex import amp import horovod.torch as hvd from transformers import CLIPTokenizerFast from src.modeling.VidCLIP import VidCLIP from src.datasets.dataset_pretrain_stage1_all_source import HDVILAPretrainDataset, PretrainCollator from src.datasets.dataloader import MetaLoader, PrefetchLoader from src.datasets.data_utils import ImageNorm, mk_input_group from src.configs.config import shared_configs from src.utils.misc import set_random_seed, NoOp, zero_none_grad from src.utils.logger import LOGGER, TB_LOGGER, add_log_to_file, RunningMeter from src.utils.basic_utils import ( load_jsonl, load_json, save_json, get_rounded_percentage) from src.utils.load_save import (ModelSaver, BestModelSaver, save_training_meta, load_state_dict_with_mismatch) from src.utils.load_save import E2E_TrainingRestorer as TrainingRestorer from src.optimization.sched import get_lr_sched from src.optimization.utils import setup_e2e_optimizer from src.optimization.loss import build_loss_func from src.utils.distributed import all_gather_list from src.utils.metrics import cal_cossim, compute_metrics def mk_pretrain_dataloader(dataset_name, vis_format, anno_path, vis_dir, vid_cap_path,\ img_dir, cap_path, img_source, img_ratio, vid_txt, cfg, tokenizer, mode): """""" is_train = mode == "train" dataset = HDVILAPretrainDataset( cfg=cfg, vis_dir=vis_dir, anno_path=anno_path, vid_cap_path=vid_cap_path, img_dir=img_dir, cap_path=cap_path, img_source=img_source, img_ratio=img_ratio, vid_txt=vid_txt, vis_format=vis_format, mode=mode ) LOGGER.info(f"[{dataset_name}] is_train {is_train} " f"dataset size {len(dataset)}, ") batch_size = cfg.train_batch_size if is_train else cfg.test_batch_size sampler = DistributedSampler( dataset, num_replicas=hvd.size(), rank=hvd.rank(), shuffle=is_train) data_collator = PretrainCollator( tokenizer=tokenizer, max_length=cfg.max_txt_len, is_train=is_train) dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False, sampler=sampler, num_workers=cfg.n_workers, pin_memory=cfg.pin_mem, collate_fn=data_collator.collate_batch) return dataloader def setup_dataloaders(cfg, tokenizer): LOGGER.info("Init. train_loader and val_loader...") train_loaders = {} for db in cfg.train_datasets: train_loaders[db.name] = mk_pretrain_dataloader( dataset_name=db.name, vis_format=db.vis_format, anno_path=db.txt, vis_dir=db.vis, vid_cap_path=db.vid_cap_path, img_dir=db.img_dir, cap_path=db.cap_path, img_source=db.img_source, img_ratio=db.img_ratio, vid_txt=db.vid_txt, cfg=cfg, tokenizer=tokenizer, mode="train" ) if "ratio" in db: train_loaders[db.name] = (train_loaders[db.name], db.ratio) val_loaders = {} for db in cfg.val_datasets: val_loaders[db.name] = mk_pretrain_dataloader( dataset_name=db.name, vis_format=db.vis_format, anno_path=db.txt, vis_dir=db.vis, vid_cap_path="", img_dir="", cap_path="", img_source="", img_ratio=0, vid_txt='subtitle', cfg=cfg, tokenizer=tokenizer, mode="val" ) return train_loaders, val_loaders def setup_model(cfg, device=None): LOGGER.info("Setup model...") model = VidCLIP(cfg) if cfg.e2e_weights_path: LOGGER.info(f"Loading e2e weights from {cfg.e2e_weights_path}") load_state_dict_with_mismatch(model, cfg.e2e_weights_path) if hasattr(cfg, "freeze_text_model") and cfg.freeze_text_model: freeze_text_proj = hasattr(cfg, "freeze_text_proj") and cfg.freeze_text_proj LOGGER.info(f"Freeze CLIP text model and the status of freezing proj is: {freeze_text_proj}") model.freeze_text_encoder(freeze_text_proj) model.to(device) LOGGER.info("Setup model done!") return model @torch.no_grad() def validate(model, val_loaders, cfg): """use eval_score=False when doing inference on test sets where answers are not available""" model.eval() st = time.time() for loader_name, val_loader in val_loaders.items(): LOGGER.info(f"Loop val_loader {loader_name}.") val_log = {f'valid/{loader_name}_t2v_recall_1': 0, f'valid/{loader_name}_t2v_recall_5': 0, f'valid/{loader_name}_t2v_recall_10': 0, f'valid/{loader_name}_t2v_recall_median': 0, f'valid/{loader_name}_t2v_recall_mean': 0, f'valid/{loader_name}_v2t_recall_1': 0, f'valid/{loader_name}_v2t_recall_5': 0, f'valid/{loader_name}_v2t_recall_10': 0, f'valid/{loader_name}_v2t_recall_median': 0, f'valid/{loader_name}_v2t_recall_mean': 0} text_feats = [] vis_feats = [] for val_step, batch in enumerate(val_loader): feats = model(**batch) # dict # print('feats vis_features', feats['vis_features'].shape) vis_feat = hvd.allgather(feats['vis_features']) text_feat = hvd.allgather(feats['text_features']) # print('allgather vis_features', vis_feat.shape) text_feats.append(text_feat.cpu().numpy()) vis_feats.append(vis_feat.cpu().numpy()) # # Gather across all processes # text_feats = all_gather_list(text_feats) # vis_feats = all_gather_list(vis_feats) text_feats = np.vstack(text_feats) vis_feats = np.vstack(vis_feats) sim_matrix = cal_cossim(text_feats, vis_feats) v2tr1,v2tr5,v2tr10,v2tmedr,v2tmeanr = compute_metrics(sim_matrix.T) t2vr1,t2vr5,t2vr10,t2vmedr,t2vmeanr = compute_metrics(sim_matrix) val_log.update({f'valid/{loader_name}_t2v_recall_1': t2vr1, f'valid/{loader_name}_t2v_recall_5': t2vr5, f'valid/{loader_name}_t2v_recall_10': t2vr10, f'valid/{loader_name}_t2v_recall_median': t2vmedr, f'valid/{loader_name}_t2v_recall_mean': t2vmeanr, f'valid/{loader_name}_v2t_recall_1': v2tr1, f'valid/{loader_name}_v2t_recall_5': v2tr5, f'valid/{loader_name}_v2t_recall_10': v2tr10, f'valid/{loader_name}_v2t_recall_median': v2tmedr, f'valid/{loader_name}_v2t_recall_mean': v2tmeanr }) TB_LOGGER.log_scalar_dict(val_log) LOGGER.info(f"validation finished in {int(time.time() - st)} seconds, " f"validated on {vis_feats.shape[0]} videos" f"{loader_name} t2v recall@1: {val_log['valid/%s_t2v_recall_1'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall@5: {val_log['valid/%s_t2v_recall_5'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall@10: {val_log['valid/%s_t2v_recall_10'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall_med: {val_log['valid/%s_t2v_recall_median'%(loader_name)] :.1f} " f"{loader_name} t2v recall_mean: {val_log['valid/%s_t2v_recall_mean'%(loader_name)] :.1f} " f"{loader_name} v2t recall@1: {val_log['valid/%s_v2t_recall_1'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall@5: {val_log['valid/%s_v2t_recall_5'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall@10: {val_log['valid/%s_v2t_recall_10'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall_med: {val_log['valid/%s_v2t_recall_median'%(loader_name)] :.1f} " f"{loader_name} v2t recall_mean: {val_log['valid/%s_v2t_recall_mean'%(loader_name)] :.1f} " ) model.train() return val_log, t2vr1 def start_training(): cfg = shared_configs.get_pretraining_args() blob_mount(cfg) set_random_seed(cfg.seed) n_gpu = hvd.size() cfg.n_gpu = n_gpu device = torch.device("cuda", hvd.local_rank()) torch.cuda.set_device(hvd.local_rank()) if hvd.rank() != 0: LOGGER.disabled = True LOGGER.info(f"device: {device} n_gpu: {n_gpu}, " f"rank: {hvd.rank()}, 16-bits training: {cfg.fp16}") if hvd.rank() != 0: LOGGER.disabled = True model = setup_model(cfg, device=device) model.train() optimizer = setup_e2e_optimizer(model, cfg) # Horovod: (optional) compression algorithm.compressin compression = hvd.Compression.none optimizer = hvd.DistributedOptimizer( optimizer, named_parameters=model.named_parameters(), compression=compression) # Horovod: broadcast parameters & optimizer state. hvd.broadcast_parameters(model.state_dict(), root_rank=0) hvd.broadcast_optimizer_state(optimizer, root_rank=0) model, optimizer = amp.initialize( model, optimizer, enabled=cfg.fp16, opt_level=cfg.amp_level, keep_batchnorm_fp32=True if cfg.amp_level=='O2' else None) # prepare data tokenizer = CLIPTokenizerFast.from_pretrained(cfg.clip_config) train_loaders, val_loaders = setup_dataloaders(cfg, tokenizer) train_loader = MetaLoader(train_loaders, accum_steps=cfg.gradient_accumulation_steps, distributed=n_gpu > 1) img_norm = None train_loader = PrefetchLoader(train_loader, img_norm) val_loaders = {k: PrefetchLoader(v, img_norm) for k, v in val_loaders.items()} n_batches_in_epoch = train_loader.n_batches_in_epoch # compute the number of steps and update cfg total_train_batch_size = int( n_gpu * cfg.train_batch_size * cfg.gradient_accumulation_steps * cfg.max_n_example_per_group) total_n_epochs = cfg.num_train_epochs cfg.num_train_steps = int(math.ceil( 1. * n_batches_in_epoch * total_n_epochs / (n_gpu * cfg.gradient_accumulation_steps))) cfg.valid_steps = int(math.ceil( 1. * cfg.num_train_steps / cfg.num_valid / cfg.min_valid_steps)) * cfg.min_valid_steps actual_num_valid = int(math.floor( 1. * cfg.num_train_steps / cfg.valid_steps)) + 1 # restore restorer = TrainingRestorer(cfg, model, optimizer) global_step = restorer.global_step TB_LOGGER.global_step = global_step if hvd.rank() == 0: LOGGER.info("Saving training meta...") save_training_meta(cfg) LOGGER.info("Saving training done...") if cfg.if_tb_log: TB_LOGGER.create(join(cfg.output_dir, 'log')) # pbar = tqdm(total=cfg.num_train_steps) if cfg.if_model_saver: model_saver = ModelSaver(join(cfg.output_dir, "ckpt")) best_model_saver = BestModelSaver(join(cfg.output_dir, "ckpt")) else: model_saver = NoOp() restorer = NoOp() best_model_saver = NoOp() if cfg.if_log2file: add_log_to_file(join(cfg.output_dir, "log", "log.txt")) else: LOGGER.disabled = True # pbar = NoOp() model_saver = NoOp() restorer = NoOp() best_model_saver = NoOp() if global_step > 0: pass # pbar.update(global_step) LOGGER.info(cfg) LOGGER.info("Starting training...") LOGGER.info(f"***** Running training with {n_gpu} GPUs *****") LOGGER.info(f" Single-GPU Non-Accumulated batch size = {cfg.train_batch_size}") LOGGER.info(f" max_n_example_per_group = {cfg.max_n_example_per_group}") LOGGER.info(f" Accumulate steps = {cfg.gradient_accumulation_steps}") LOGGER.info(f" Total batch size = #GPUs * Single-GPU batch size * " f"max_n_example_per_group * Accumulate steps [Image] = {total_train_batch_size}") LOGGER.info(f" Total #batches - single epoch = {n_batches_in_epoch}.") LOGGER.info(f" Total #epochs = {total_n_epochs}.") LOGGER.info(f" Total #steps = {cfg.num_train_steps}") LOGGER.info(f" Validate and Save every {cfg.valid_steps} steps, in total {actual_num_valid} times") LOGGER.info(f" Only Validate every {cfg.only_valid_steps} steps") # quick hack for amp delay_unscale bug with optimizer.skip_synchronize(): optimizer.zero_grad() if global_step == 0: optimizer.step() tasks = ["contrastive", "logit_scale"] task2loss = {t: RunningMeter(f'train_loss/{t}', smooth=0) for t in tasks} task2loss["loss"] = RunningMeter('train_loss/loss', smooth=0) LOGGER.info(f'Step zero: start inference') validate(model, val_loaders, cfg) loss_func = build_loss_func(cfg.loss_config) train_loader_iter = iter(train_loader) step = 0 while global_step < cfg.num_train_steps: if hasattr(cfg, "break_step") and global_step == cfg.break_step: break task, batch = next(train_loader_iter) # clamp logit_scale if hasattr(model, 'module'): torch.clamp_(model.module.clipmodel.logit_scale.data, 0, np.log(200)) logit_scale_ = model.module.clipmodel.logit_scale.data else: torch.clamp_(model.clipmodel.logit_scale.data, 0, np.log(200)) logit_scale_ = model.clipmodel.logit_scale.data outputs = model(**batch) if cfg.loss_config.if_gather: vis_feat = hvd.allgather(outputs['vis_features']) text_feat = hvd.allgather(outputs['text_features']) if hasattr(model, 'module'): logit_scale = model.module.clipmodel.logit_scale else: logit_scale = model.clipmodel.logit_scale if "img_features" in outputs and outputs["img_features"] is not None: img_feat = hvd.allgather(outputs['img_features']) if "cap_features" in outputs and outputs["cap_features"] is not None: cap_feat = hvd.allgather(outputs['cap_features']) if cfg.loss_config.loss_name == "NCELearnableTempLoss": loss = loss_func(vis_feat, text_feat, logit_scale) elif cfg.loss_config.loss_name in ["VidImgNCELearnableTempLoss", \ "VidImgDivideNCELearnableTempLoss", "NCELearnableTempLoss_vsc", \ "NCELearnableTempLoss_vs_vc", "NCELearnableTempLoss_vs_vc_fc", \ "NCELearnableTempLoss_vsc_fc"]: if global_step == 1: print(vis_feat.shape, text_feat.shape, img_feat.shape, cap_feat.shape) loss = loss_func(vis_feat, text_feat, img_feat, cap_feat, logit_scale) else: loss = loss_func(vis_feat, text_feat) else: loss = outputs['loss'] task2loss["logit_scale"](logit_scale_.item()) task2loss["contrastive"](loss.item()) task2loss["loss"](loss.item()) delay_unscale = (step + 1) % cfg.gradient_accumulation_steps != 0 with amp.scale_loss( loss, optimizer, delay_unscale=delay_unscale ) as scaled_loss: scaled_loss.backward() # zero_none_grad(model) optimizer.synchronize() # optimizer if (step + 1) % cfg.gradient_accumulation_steps == 0: global_step += 1 TB_LOGGER.log_scalar_dict({l.name: l.val for l in task2loss.values() if l.val is not None}) n_epoch = int(1. * n_gpu * cfg.gradient_accumulation_steps * global_step / n_batches_in_epoch) # learning rate scheduling transformer lr_this_step = get_lr_sched( global_step, cfg.decay, cfg.learning_rate, cfg.num_train_steps, warmup_ratio=cfg.warmup_ratio, decay_epochs=cfg.step_decay_epochs, multi_step_epoch=n_epoch) for pg_n, param_group in enumerate( optimizer.param_groups): if pg_n in [0, 1]: param_group['lr'] = ( cfg.lr_mul * lr_this_step) elif pg_n in [2, 3]: param_group['lr'] = lr_this_step TB_LOGGER.add_scalar( "train/lr", lr_this_step, global_step) # update model params if cfg.grad_norm != -1: grad_norm = clip_grad_norm_( amp.master_params(optimizer), cfg.grad_norm) TB_LOGGER.add_scalar("train/grad_norm", grad_norm, global_step) TB_LOGGER.step() # Check if there is None grad none_grads = [ p[0] for p in model.named_parameters() if p[1].requires_grad and p[1].grad is None] assert len(none_grads) == 0, f"{none_grads}" with optimizer.skip_synchronize(): optimizer.step() optimizer.zero_grad() restorer.step() # checkpoint if global_step % cfg.valid_steps == 0: LOGGER.info(f'Step {global_step}: start validation and Save') _, t2vr1 = validate(model, val_loaders, cfg) model_saver.save(step=global_step, model=model) # if hvd.rank() == 0 and cfg.if_model_saver and t2vr1 > best_model_saver.bestr1: # best_model_saver.save(step=global_step, model=model) # best_model_saver.bestr1 = t2vr1 else: if global_step % cfg.only_valid_steps == 0: LOGGER.info(f'Step {global_step} Training Loss: {loss.item()} LR: {lr_this_step} Logit Scale: {logit_scale_.item()}') LOGGER.info(f'Step {global_step}: start validation') _, t2vr1 = validate(model, val_loaders, cfg) step += 1 if global_step % cfg.valid_steps != 0: LOGGER.info(f'Step {global_step}: start validation') validate(model, val_loaders, cfg) model_saver.save(step=global_step, model=model) def blob_mount(cfg): keys = ["e2e_weights_path", "output_dir"] for key in keys: if cfg[key] is not None: cfg[key] = os.path.join(cfg.blob_mount_dir, cfg[key]) for db in cfg.train_datasets: db.txt = os.path.join(cfg.blob_mount_dir, db.txt) db.vis = os.path.join(cfg.blob_mount_dir, db.vis) if hasattr(db, "img_dir") and db.img_dir: db.img_dir = os.path.join(cfg.blob_mount_dir, db.img_dir) if hasattr(db, "cap_path") and db.cap_path: db.cap_path = os.path.join(cfg.blob_mount_dir, db.cap_path) if hasattr(db, "vid_cap_path") and db.vid_cap_path: db.vid_cap_path = os.path.join(cfg.blob_mount_dir, db.vid_cap_path) for db in cfg.val_datasets: db.txt = os.path.join(cfg.blob_mount_dir, db.txt) db.vis = os.path.join(cfg.blob_mount_dir, db.vis) if __name__ == '__main__': # Initialize Horovod hvd.init() start_training() # horovodrun -np 4 python src/pretrain/run_pretrain_contrast_hvd.py --config src/configs/pretrain_contrast.json --blob_mount_dir /blob_mount/ clip-vip/XPretrain-main/CLIP-Vip/src/tasks/000077500000000000000000000000001475357401500206515ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/tasks/run_video_retrieval.py000066400000000000000000000456501475357401500253040ustar00rootroot00000000000000import os import time import random import math from collections import defaultdict import numpy as np from tqdm import tqdm from os.path import join, exists from easydict import EasyDict as edict import torch from torch.utils.data import DataLoader import torch.nn.functional as F from torch.nn.utils import clip_grad_norm_ import torchvision.utils as tutils from torch.utils.data.distributed import DistributedSampler from apex import amp import horovod.torch as hvd from transformers import CLIPTokenizerFast from src.modeling.VidCLIP import VidCLIP from src.datasets.dataset_video_retrieval import ( HDVILAVideoRetrievalDataset, VideoRetrievalCollator) from src.datasets.dataloader import InfiniteIterator, PrefetchLoader from src.datasets.data_utils import ImageNorm, mk_input_group from src.configs.config import shared_configs from src.utils.misc import set_random_seed, NoOp, zero_none_grad from src.utils.logger import LOGGER, TB_LOGGER, add_log_to_file, RunningMeter from src.utils.basic_utils import ( load_jsonl, load_json, save_json, get_rounded_percentage) from src.utils.basic_utils import flat_list_of_lists from src.utils.load_save import (ModelSaver, BestModelSaver, save_training_meta, load_state_dict_with_mismatch) from src.utils.load_save import E2E_TrainingRestorer as TrainingRestorer from src.optimization.sched import get_lr_sched from src.optimization.utils import setup_e2e_optimizer from src.optimization.loss import build_loss_func from src.utils.distributed import all_gather_list from src.utils.metrics import cal_cossim, compute_metrics, compute_metrics_multi, np_softmax def mk_video_ret_dataloader(dataset_name, vis_format, anno_path, vis_dir, cfg, tokenizer, mode): """""" is_train = mode == "train" dataset = HDVILAVideoRetrievalDataset( cfg=cfg, vis_dir=vis_dir, anno_path=anno_path, vis_format=vis_format, mode=mode ) LOGGER.info(f"[{dataset_name}] is_train {is_train} " f"dataset size {len(dataset)}, ") batch_size = cfg.train_batch_size if is_train else cfg.test_batch_size sampler = DistributedSampler( dataset, num_replicas=hvd.size(), rank=hvd.rank(), shuffle=is_train) vret_collator = VideoRetrievalCollator( tokenizer=tokenizer, max_length=cfg.max_txt_len, is_train=is_train) dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False, sampler=sampler, num_workers=cfg.n_workers, pin_memory=cfg.pin_mem, collate_fn=vret_collator.collate_batch) return dataloader def setup_dataloaders(cfg, tokenizer): LOGGER.info("Init. train_loader and val_loader...") db = cfg.train_datasets train_loader = mk_video_ret_dataloader( dataset_name=db.name, vis_format=db.vis_format, anno_path=db.txt, vis_dir=db.vis, cfg=cfg, tokenizer=tokenizer, mode="train" ) val_loaders = {} for db in cfg.val_datasets: val_loaders[db.name] = mk_video_ret_dataloader( dataset_name=db.name, vis_format=db.vis_format, anno_path=db.txt, vis_dir=db.vis, cfg=cfg, tokenizer=tokenizer, mode="val" ) inference_loaders = {} for db in cfg.inference_datasets: inference_loaders[db.name] = mk_video_ret_dataloader( dataset_name=db.name, vis_format=db.vis_format, anno_path=db.txt, vis_dir=db.vis, cfg=cfg, tokenizer=tokenizer, mode="test" ) return train_loader, val_loaders, inference_loaders def setup_model(cfg, device=None): LOGGER.info("Setup model...") model = VidCLIP(cfg) if cfg.e2e_weights_path: LOGGER.info(f"Loading e2e weights from {cfg.e2e_weights_path}") load_state_dict_with_mismatch(model, cfg.e2e_weights_path) if hasattr(cfg, "overload_logit_scale"): model.overload_logit_scale(cfg.overload_logit_scale) model.to(device) LOGGER.info("Setup model done!") return model @torch.no_grad() def validate(model, val_loaders, cfg): model.eval() st = time.time() for loader_name, val_loader in val_loaders.items(): LOGGER.info(f"Loop val_loader {loader_name}.") valid_len = len(val_loader.dataset) text_feats = [] vis_feats = [] for val_step, batch in enumerate(val_loader): feats = model(**batch) # dict # print('feats vis_features', feats['vis_features'].shape) vis_feat = hvd.allgather(feats['vis_features']) text_feat = hvd.allgather(feats['text_features']) # print('allgather vis_features', vis_feat.shape) text_feats.append(text_feat.cpu().numpy()) vis_feats.append(vis_feat.cpu().numpy()) # # Gather across all processes # text_feats = all_gather_list(text_feats) # vis_feats = all_gather_list(vis_feats) text_feats = np.vstack(text_feats) vis_feats = np.vstack(vis_feats) text_feats = text_feats[:valid_len] vis_feats = vis_feats[:valid_len] sim_matrix = cal_cossim(text_feats, vis_feats) for type in ["simple", "DSL"]: LOGGER.info(f"Evaluate under setting: {type}.") val_log = {f'valid/{loader_name}_t2v_recall_1': 0, f'valid/{loader_name}_t2v_recall_5': 0, f'valid/{loader_name}_t2v_recall_10': 0, f'valid/{loader_name}_t2v_recall_median': 0, f'valid/{loader_name}_t2v_recall_mean': 0, f'valid/{loader_name}_v2t_recall_1': 0, f'valid/{loader_name}_v2t_recall_5': 0, f'valid/{loader_name}_v2t_recall_10': 0, f'valid/{loader_name}_v2t_recall_median': 0, f'valid/{loader_name}_v2t_recall_mean': 0} if type == "DSL": sim_matrix = sim_matrix * np_softmax(sim_matrix*100, axis=0) v2tr1,v2tr5,v2tr10,v2tmedr,v2tmeanr = compute_metrics(sim_matrix.T) t2vr1,t2vr5,t2vr10,t2vmedr,t2vmeanr = compute_metrics(sim_matrix) val_log.update({f'valid/{loader_name}_t2v_recall_1': t2vr1, f'valid/{loader_name}_t2v_recall_5': t2vr5, f'valid/{loader_name}_t2v_recall_10': t2vr10, f'valid/{loader_name}_t2v_recall_median': t2vmedr, f'valid/{loader_name}_t2v_recall_mean': t2vmeanr, f'valid/{loader_name}_v2t_recall_1': v2tr1, f'valid/{loader_name}_v2t_recall_5': v2tr5, f'valid/{loader_name}_v2t_recall_10': v2tr10, f'valid/{loader_name}_v2t_recall_median': v2tmedr, f'valid/{loader_name}_v2t_recall_mean': v2tmeanr }) LOGGER.info(f"validation finished in {int(time.time() - st)} seconds, " f"validated on {vis_feats.shape[0]} videos" f"{loader_name} t2v recall@1: {val_log['valid/%s_t2v_recall_1'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall@5: {val_log['valid/%s_t2v_recall_5'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall@10: {val_log['valid/%s_t2v_recall_10'%(loader_name)] * 100:.4f} " f"{loader_name} t2v recall_med: {val_log['valid/%s_t2v_recall_median'%(loader_name)] :.1f} " f"{loader_name} t2v recall_mean: {val_log['valid/%s_t2v_recall_mean'%(loader_name)] :.1f} " f"{loader_name} v2t recall@1: {val_log['valid/%s_v2t_recall_1'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall@5: {val_log['valid/%s_v2t_recall_5'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall@10: {val_log['valid/%s_v2t_recall_10'%(loader_name)] * 100:.4f} " f"{loader_name} v2t recall_med: {val_log['valid/%s_v2t_recall_median'%(loader_name)] :.1f} " f"{loader_name} v2t recall_mean: {val_log['valid/%s_v2t_recall_mean'%(loader_name)] :.1f} " ) TB_LOGGER.log_scalar_dict(val_log) model.train() return val_log, t2vr1 def start_training(): cfg = shared_configs.get_pretraining_args() blob_mount(cfg) set_random_seed(cfg.seed) n_gpu = hvd.size() cfg.n_gpu = n_gpu device = torch.device("cuda", hvd.local_rank()) torch.cuda.set_device(hvd.local_rank()) if hvd.rank() != 0: LOGGER.disabled = True LOGGER.info(f"device: {device} n_gpu: {n_gpu}, " f"rank: {hvd.rank()}, 16-bits training: {cfg.fp16}") if hvd.rank() != 0: LOGGER.disabled = True model = setup_model(cfg, device=device) model.train() optimizer = setup_e2e_optimizer(model, cfg) # Horovod: (optional) compression algorithm.compressin compression = hvd.Compression.none optimizer = hvd.DistributedOptimizer( optimizer, named_parameters=model.named_parameters(), compression=compression) # Horovod: broadcast parameters & optimizer state. hvd.broadcast_parameters(model.state_dict(), root_rank=0) hvd.broadcast_optimizer_state(optimizer, root_rank=0) model, optimizer = amp.initialize( model, optimizer, enabled=cfg.fp16, opt_level=cfg.amp_level, keep_batchnorm_fp32=True if cfg.amp_level=='O2' else None) # prepare data tokenizer = CLIPTokenizerFast.from_pretrained(cfg.clip_config) train_loader, val_loaders, inference_loaders = setup_dataloaders(cfg, tokenizer) img_norm = None train_loader = PrefetchLoader(train_loader, img_norm) val_loaders = {k: PrefetchLoader(v, img_norm) for k, v in val_loaders.items()} inference_loaders = {k: PrefetchLoader(v, img_norm) for k, v in inference_loaders.items()} # compute the number of steps and update cfg total_train_batch_size = int( n_gpu * cfg.train_batch_size * cfg.gradient_accumulation_steps * cfg.max_n_example_per_group) total_n_examples = len(train_loader.dataset) * cfg.max_n_example_per_group print('total_n_examples', total_n_examples) cfg.num_train_steps = int(math.ceil( 1. * cfg.num_train_epochs * total_n_examples / total_train_batch_size)) cfg.valid_steps = int(math.ceil( 1. * cfg.num_train_steps / cfg.num_valid / cfg.min_valid_steps)) * cfg.min_valid_steps actual_num_valid = int(math.floor( 1. * cfg.num_train_steps / cfg.valid_steps)) + 1 n_steps_in_epoch = int(math.ceil(1. * total_n_examples / total_train_batch_size)) # restore restorer = TrainingRestorer(cfg, model, optimizer) global_step = restorer.global_step TB_LOGGER.global_step = global_step if hvd.rank() == 0: LOGGER.info("Saving training meta...") save_training_meta(cfg) LOGGER.info("Saving training done...") if cfg.if_tb_log: TB_LOGGER.create(join(cfg.output_dir, 'log')) # pbar = tqdm(total=cfg.num_train_steps) if cfg.if_model_saver: model_saver = ModelSaver(join(cfg.output_dir, "ckpt")) best_model_saver = BestModelSaver(join(cfg.output_dir, "ckpt")) else: model_saver = NoOp() restorer = NoOp() best_model_saver = NoOp() if cfg.if_log2file: add_log_to_file(join(cfg.output_dir, "log", "log.txt")) else: LOGGER.disabled = True # pbar = NoOp() model_saver = NoOp() restorer = NoOp() best_model_saver = NoOp() if global_step > 0: pass # pbar.update(global_step) LOGGER.info(cfg) LOGGER.info("Starting training...") LOGGER.info(f"***** Running training with {n_gpu} GPUs *****") LOGGER.info(f" Single-GPU Non-Accumulated batch size = {cfg.train_batch_size}") LOGGER.info(f" max_n_example_per_group = {cfg.max_n_example_per_group}") LOGGER.info(f" Accumulate steps = {cfg.gradient_accumulation_steps}") LOGGER.info(f" Total batch size = #GPUs * Single-GPU batch size * " f"max_n_example_per_group * Accumulate steps [Image] = {total_train_batch_size}") LOGGER.info(f" Total #epochs = {cfg.num_train_epochs}") LOGGER.info(f" Total #steps = {cfg.num_train_steps}") LOGGER.info(f" Validate and Save every {cfg.valid_steps} steps, in total {actual_num_valid} times") LOGGER.info(f" Only Validate every {cfg.only_valid_steps} steps") # quick hack for amp delay_unscale bug with optimizer.skip_synchronize(): optimizer.zero_grad() if global_step == 0: optimizer.step() running_loss = RunningMeter('train_loss', smooth=0) LOGGER.info(f'Step zero: start inference') validate(model, inference_loaders, cfg) loss_func = build_loss_func(cfg.loss_config) for step, batch in enumerate(InfiniteIterator(train_loader)): outputs = model(**batch) if cfg.loss_config.if_gather: vis_feat = hvd.allgather(outputs['vis_features']) text_feat = hvd.allgather(outputs['text_features']) if cfg.loss_config.loss_name in ["NCELearnableTempLoss", "NCELearnableTempDSLLoss"]: if hasattr(model, 'module'): logit_scale = model.module.clipmodel.logit_scale else: logit_scale = model.clipmodel.logit_scale loss = loss_func(vis_feat, text_feat, logit_scale) else: loss = loss_func(vis_feat, text_feat) else: loss = outputs['loss'] if hasattr(model, 'module'): torch.clamp_(model.module.clipmodel.logit_scale.data, 0, np.log(200)) logit_scale_ = model.module.clipmodel.logit_scale.data else: torch.clamp_(model.clipmodel.logit_scale.data, 0, np.log(200)) logit_scale_ = model.clipmodel.logit_scale.data if step % 10 == 0: lr_ = optimizer.param_groups[0]['lr'] LOGGER.info(f'Step {global_step}: loss {loss} lr {lr_} logit_scale {logit_scale_}') running_loss(loss.item()) delay_unscale = (step + 1) % cfg.gradient_accumulation_steps != 0 with amp.scale_loss( loss, optimizer, delay_unscale=delay_unscale ) as scaled_loss: scaled_loss.backward() # zero_none_grad(model) optimizer.synchronize() # optimizer if (step + 1) % cfg.gradient_accumulation_steps == 0: global_step += 1 TB_LOGGER.log_scalar_dict({'vtc_loss': running_loss.val}) n_epoch = int(1.* cfg.gradient_accumulation_steps * global_step / n_steps_in_epoch) # learning rate scheduling transformer lr_this_step = get_lr_sched( global_step, cfg.decay, cfg.learning_rate, cfg.num_train_steps, warmup_ratio=cfg.warmup_ratio, decay_epochs=cfg.step_decay_epochs, multi_step_epoch=n_epoch) for pg_n, param_group in enumerate( optimizer.param_groups): if pg_n in [0, 1]: param_group['lr'] = ( cfg.lr_mul * lr_this_step) elif pg_n in [2, 3]: param_group['lr'] = lr_this_step TB_LOGGER.add_scalar( "train/lr", lr_this_step, global_step) # update model params if cfg.grad_norm != -1: grad_norm = clip_grad_norm_( amp.master_params(optimizer), cfg.grad_norm) TB_LOGGER.add_scalar("train/grad_norm", grad_norm, global_step) TB_LOGGER.step() # Check if there is None grad none_grads = [ p[0] for p in model.named_parameters() if p[1].requires_grad and p[1].grad is None] assert len(none_grads) == 0, f"{none_grads}" with optimizer.skip_synchronize(): optimizer.step() optimizer.zero_grad() restorer.step() # checkpoint if global_step % cfg.valid_steps == 0: LOGGER.info(f'Step {global_step}: start validation and Save') _, t2vr1 = validate(model, inference_loaders, cfg) model_saver.save(step=global_step, model=model) if hvd.rank() == 0 and cfg.if_model_saver and t2vr1 > best_model_saver.bestr1: best_model_saver.save(step=global_step, model=model) best_model_saver.bestr1 = t2vr1 else: if global_step % cfg.only_valid_steps == 0: LOGGER.info(f'Step {global_step}: start inference') _, t2vr1 = validate(model, inference_loaders, cfg) if hvd.rank() == 0 and cfg.if_model_saver and t2vr1 > best_model_saver.bestr1: best_model_saver.save(step=global_step, model=model) best_model_saver.bestr1 = t2vr1 if global_step >= cfg.num_train_steps: break if global_step % cfg.valid_steps != 0: LOGGER.info(f'Step {global_step}: start validation') _, t2vr1 = validate(model, inference_loaders, cfg) model_saver.save(step=global_step, model=model) if hvd.rank() == 0 and cfg.if_model_saver and t2vr1 > best_model_saver.bestr1: best_model_saver.save(step=global_step, model=model) best_model_saver.bestr1 = t2vr1 def blob_mount(cfg): keys = ["e2e_weights_path", "output_dir"] for key in keys: if cfg[key] is not None: cfg[key] = os.path.join(cfg.blob_mount_dir, cfg[key]) db = cfg.train_datasets db.txt = os.path.join(cfg.blob_mount_dir, db.txt) db.vis = os.path.join(cfg.blob_mount_dir, db.vis) for db in cfg.val_datasets: db.txt = os.path.join(cfg.blob_mount_dir, db.txt) db.vis = os.path.join(cfg.blob_mount_dir, db.vis) for db in cfg.inference_datasets: db.txt = os.path.join(cfg.blob_mount_dir, db.txt) db.vis = os.path.join(cfg.blob_mount_dir, db.vis) if __name__ == '__main__': # Initialize Horovod hvd.init() start_training() # CUDA_VISIBLE_DEVICES=2,3 horovodrun -np 2 python src/tasks/run_video_retrieval.py --config src/configs/msrvtt_retrieval_debug.json --blob_mount_dir /blob_mount/ clip-vip/XPretrain-main/CLIP-Vip/src/utils/000077500000000000000000000000001475357401500206645ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/utils/__init__.py000066400000000000000000000020001475357401500227650ustar00rootroot00000000000000clip-vip/XPretrain-main/CLIP-Vip/src/utils/basic_utils.py000066400000000000000000000126771475357401500235540ustar00rootroot00000000000000import os import ujson as json import zipfile import numpy as np import pickle def load_pickle(filename): with open(filename, "rb") as f: return pickle.load(f) def save_pickle(data, filename): with open(filename, "wb") as f: pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL) def load_json(filename): with open(filename, "r") as f: return json.load(f) def save_json(data, filename, save_pretty=False, sort_keys=False): with open(filename, "w") as f: if save_pretty: f.write(json.dumps(data, indent=4, sort_keys=sort_keys)) else: json.dump(data, f) def load_jsonl(filename): with open(filename, "r") as f: return [json.loads(l.strip("\n")) for l in f.readlines()] def save_jsonl(data, filename): """data is a list""" with open(filename, "w") as f: f.write("\n".join([json.dumps(e) for e in data])) def concat_json_list(filepaths, save_path): json_lists = [] for p in filepaths: json_lists += load_json(p) save_json(json_lists, save_path) def save_lines(list_of_str, filepath): with open(filepath, "w") as f: f.write("\n".join(list_of_str)) def read_lines(filepath): with open(filepath, "r") as f: return [e.strip("\n") for e in f.readlines()] def mkdirp(p): if not os.path.exists(p): os.makedirs(p) def flat_list_of_lists(l): """flatten a list of lists [[1,2], [3,4]] to [1,2,3,4]""" return [item for sublist in l for item in sublist] def convert_to_seconds(hms_time): """ convert '00:01:12' to 72 seconds. :hms_time (str): time in comma separated string, e.g. '00:01:12' :return (int): time in seconds, e.g. 72 """ times = [float(t) for t in hms_time.split(":")] return times[0] * 3600 + times[1] * 60 + times[2] def get_video_name_from_url(url): return url.split("/")[-1][:-4] def merge_dicts(list_dicts): merged_dict = list_dicts[0].copy() for i in range(1, len(list_dicts)): merged_dict.update(list_dicts[i]) return merged_dict def l2_normalize_np_array(np_array, eps=1e-5): """np_array: np.ndarray, (*, D), where the last dim will be normalized""" return np_array / (np.linalg.norm(np_array, axis=-1, keepdims=True) + eps) def make_zipfile(src_dir, save_path, enclosing_dir="", exclude_dirs=None, exclude_extensions=None, exclude_dirs_substring=None): """make a zip file of root_dir, save it to save_path. exclude_paths will be excluded if it is a subdir of root_dir. An enclosing_dir is added is specified. """ abs_src = os.path.abspath(src_dir) with zipfile.ZipFile(save_path, "w") as zf: for dirname, subdirs, files in os.walk(src_dir): if exclude_dirs is not None: for e_p in exclude_dirs: if e_p in subdirs: subdirs.remove(e_p) if exclude_dirs_substring is not None: to_rm = [] for d in subdirs: if exclude_dirs_substring in d: to_rm.append(d) for e in to_rm: subdirs.remove(e) arcname = os.path.join(enclosing_dir, dirname[len(abs_src) + 1:]) zf.write(dirname, arcname) for filename in files: if exclude_extensions is not None: if os.path.splitext(filename)[1] in exclude_extensions: continue # do not zip it absname = os.path.join(dirname, filename) arcname = os.path.join(enclosing_dir, absname[len(abs_src) + 1:]) zf.write(absname, arcname) class AverageMeter(object): """Computes and stores the average and current/max/min value""" def __init__(self): self.val = 0 self.avg = 0 self.sum = 0 self.count = 0 self.max = -1e10 self.min = 1e10 self.reset() def reset(self): self.val = 0 self.avg = 0 self.sum = 0 self.count = 0 self.max = -1e10 self.min = 1e10 def update(self, val, n=1): self.max = max(val, self.max) self.min = min(val, self.min) self.val = val self.sum += val * n self.count += n self.avg = self.sum / self.count def dissect_by_lengths(np_array, lengths, dim=0, assert_equal=True): """Dissect an array (N, D) into a list a sub-array, np_array.shape[0] == sum(lengths), Output is a list of nd arrays, singlton dimention is kept""" if assert_equal: assert len(np_array) == sum(lengths) length_indices = [0, ] for i in range(len(lengths)): length_indices.append(length_indices[i] + lengths[i]) if dim == 0: array_list = [np_array[length_indices[i]:length_indices[i+1]] for i in range(len(lengths))] elif dim == 1: array_list = [np_array[:, length_indices[i]:length_indices[i + 1]] for i in range(len(lengths))] elif dim == 2: array_list = [np_array[:, :, length_indices[i]:length_indices[i + 1]] for i in range(len(lengths))] else: raise NotImplementedError return array_list def get_ratio_from_counter(counter_obj, threshold=200): keys = counter_obj.keys() values = counter_obj.values() filtered_values = [counter_obj[k] for k in keys if k > threshold] return float(sum(filtered_values)) / sum(values) def get_rounded_percentage(float_number, n_floats=2): return round(float_number * 100, n_floats) clip-vip/XPretrain-main/CLIP-Vip/src/utils/distributed.py000066400000000000000000000150601475357401500235620ustar00rootroot00000000000000""" Copyright (c) Microsoft Corporation. Licensed under the MIT license. distributed API using Horovod Modified from OpenNMT's native pytorch distributed utils (https://github.com/OpenNMT/OpenNMT-py) """ import math import pickle import torch from horovod import torch as hvd def all_reduce_and_rescale_tensors(tensors, rescale_denom): """All-reduce and rescale tensors at once (as a flattened tensor) Args: tensors: list of Tensors to all-reduce rescale_denom: denominator for rescaling summed Tensors """ # buffer size in bytes, determine equiv. # of elements based on data type sz = sum(t.numel() for t in tensors) buffer_t = tensors[0].new(sz).zero_() # copy tensors into buffer_t offset = 0 for t in tensors: numel = t.numel() buffer_t[offset:offset+numel].copy_(t.view(-1)) offset += numel # all-reduce and rescale hvd.allreduce_(buffer_t[:offset]) buffer_t.div_(rescale_denom) # copy all-reduced buffer back into tensors offset = 0 for t in tensors: numel = t.numel() t.view(-1).copy_(buffer_t[offset:offset+numel]) offset += numel def all_reduce_and_rescale_tensors_chunked(tensors, rescale_denom, buffer_size=10485760): """All-reduce and rescale tensors in chunks of the specified size. Args: tensors: list of Tensors to all-reduce rescale_denom: denominator for rescaling summed Tensors buffer_size: all-reduce chunk size in bytes """ # buffer size in bytes, determine equiv. # of elements based on data type buffer_t = tensors[0].new( math.ceil(buffer_size / tensors[0].element_size())).zero_() buffer = [] def all_reduce_buffer(): # copy tensors into buffer_t offset = 0 for t in buffer: numel = t.numel() buffer_t[offset:offset+numel].copy_(t.view(-1)) offset += numel # all-reduce and rescale hvd.allreduce_(buffer_t[:offset]) buffer_t.div_(rescale_denom) # copy all-reduced buffer back into tensors offset = 0 for t in buffer: numel = t.numel() t.view(-1).copy_(buffer_t[offset:offset+numel]) offset += numel filled = 0 for t in tensors: sz = t.numel() * t.element_size() if sz > buffer_size: # tensor is bigger than buffer, all-reduce and rescale directly hvd.allreduce_(t) t.div_(rescale_denom) elif filled + sz > buffer_size: # buffer is full, all-reduce and replace buffer with grad all_reduce_buffer() buffer = [t] filled = sz else: # add tensor to buffer buffer.append(t) filled += sz if len(buffer) > 0: all_reduce_buffer() def broadcast_tensors(tensors, root_rank, buffer_size=10485760): """broadcast tensors in chunks of the specified size. Args: tensors: list of Tensors to broadcast root_rank: rank to broadcast buffer_size: all-reduce chunk size in bytes """ # buffer size in bytes, determine equiv. # of elements based on data type buffer_t = tensors[0].new( math.ceil(buffer_size / tensors[0].element_size())).zero_() buffer = [] def broadcast_buffer(): # copy tensors into buffer_t offset = 0 for t in buffer: numel = t.numel() buffer_t[offset:offset+numel].copy_(t.view(-1)) offset += numel # broadcast hvd.broadcast_(buffer_t[:offset], root_rank) # copy all-reduced buffer back into tensors offset = 0 for t in buffer: numel = t.numel() t.view(-1).copy_(buffer_t[offset:offset+numel]) offset += numel filled = 0 for t in tensors: sz = t.numel() * t.element_size() if sz > buffer_size: # tensor is bigger than buffer, broadcast directly hvd.broadcast_(t, root_rank) elif filled + sz > buffer_size: # buffer is full, broadcast and replace buffer with tensor broadcast_buffer() buffer = [t] filled = sz else: # add tensor to buffer buffer.append(t) filled += sz if len(buffer) > 0: broadcast_buffer() def all_gather_list(data, max_size=4096): """Gathers arbitrary data from all nodes into a list.""" world_size = hvd.size() if not hasattr(all_gather_list, '_in_buffer') or \ max_size != all_gather_list._in_buffer.size(): all_gather_list._in_buffer = torch.cuda.ByteTensor(max_size) in_buffer = all_gather_list._in_buffer enc = pickle.dumps(data) enc_size = len(enc) if enc_size + 2 > max_size: raise ValueError( 'encoded data exceeds max_size: {}'.format(enc_size + 2)) assert max_size < 255*256 in_buffer[0] = enc_size // 255 # this encoding works for max_size < 65k in_buffer[1] = enc_size % 255 in_buffer[2:enc_size+2] = torch.ByteTensor(list(enc)) # FIXME cannot create buffer out = hvd.allgather(in_buffer.cuda()) results = [] for i in range(0, max_size*world_size, max_size): out_buffer = out[i:i+max_size] size = (255 * out_buffer[0].item()) + out_buffer[1].item() bytes_list = bytes(out_buffer[2:size+2].tolist()) result = pickle.loads(bytes_list) results.append(result) return results def any_broadcast(data, root_rank, max_size=4096): """broadcast arbitrary data from root_rank to all nodes.""" if not hasattr(any_broadcast, '_in_buffer') or \ max_size != any_broadcast._in_buffer.size(): any_broadcast._buffer = torch.cuda.ByteTensor(max_size) buffer_ = any_broadcast._buffer enc = pickle.dumps(data) enc_size = len(enc) if enc_size + 2 > max_size: raise ValueError( 'encoded data exceeds max_size: {}'.format(enc_size + 2)) assert max_size < 255*256 buffer_[0] = enc_size // 255 # this encoding works for max_size < 65k buffer_[1] = enc_size % 255 buffer_[2:enc_size+2] = torch.ByteTensor(list(enc)) hvd.broadcast_(buffer_, root_rank) size = (255 * buffer_[0].item()) + buffer_[1].item() bytes_list = bytes(buffer_[2:size+2].tolist()) result = pickle.loads(bytes_list) return resultclip-vip/XPretrain-main/CLIP-Vip/src/utils/load_save.py000066400000000000000000000347651475357401500232120ustar00rootroot00000000000000""" saving utilities """ import json import os from os.path import dirname, exists, join, realpath import subprocess from apex import amp from easydict import EasyDict as edict import torch from src.utils.basic_utils import save_json, make_zipfile, load_json from src.utils.logger import LOGGER from typing import Any, Dict, Union def save_training_meta(args): # args is an EasyDict object, treat it the same as a normal dict os.makedirs(join(args.output_dir, 'log'), exist_ok=True) os.makedirs(join(args.output_dir, 'ckpt'), exist_ok=True) # training args save_args_path = join(args.output_dir, 'log', 'args.json') save_json(args, save_args_path, save_pretty=True) # save a copy of the codebase. !!!Do not store heavy file in your codebase when using it. code_dir = dirname(dirname(dirname(os.path.realpath(__file__)))) code_zip_filename = os.path.join(args.output_dir, "code.zip") LOGGER.info(f"Saving code from {code_dir} to {code_zip_filename}...") make_zipfile(code_dir, code_zip_filename, enclosing_dir="code", exclude_dirs_substring="results", exclude_dirs=["results", "debug_results", "__pycache__"], exclude_extensions=[".pyc", ".ipynb", ".swap"]) LOGGER.info(f"Saving code done.") class ModelSaver(object): def __init__(self, output_dir): self.output_dir = output_dir self.max_save_load_trial = 10 def save(self, step, model, optimizer=None, prefix="model"): model_path = join(self.output_dir, f"{prefix}_step_{step}.pt") state_dict = {k: v.cpu() if isinstance(v, torch.Tensor) else v for k, v in model.state_dict().items()} # with retrial, as azure blob fails occasionally. save_trial = 0 while save_trial < self.max_save_load_trial: try: LOGGER.info(f"ModelSaver save trial NO. {save_trial}") torch.save(state_dict, model_path) if optimizer is not None: optimizer_state_dict = \ {k: v.cpu() if isinstance(v, torch.Tensor) else v for k, v in optimizer.state_dict().items()} dump = {'step': step, 'optimizer': optimizer_state_dict} torch.save( dump, f'{self.output_dir}/{prefix}_step_{step}_train_state.pt') break except Exception as e: save_trial += 1 class BestModelSaver(object): def __init__(self, output_dir): self.output_dir = output_dir self.max_save_load_trial = 10 self.bestr1 = 0 def save(self, step, model, prefix="model_best"): model_path = join(self.output_dir, f"{prefix}.pt") state_dict = {k: v.cpu() if isinstance(v, torch.Tensor) else v for k, v in model.state_dict().items()} # with retrial, as azure blob fails occasionally. save_trial = 0 while save_trial < self.max_save_load_trial: try: LOGGER.info(f"BestModelSaver save trial NO. {save_trial}") torch.save(state_dict, model_path) break except Exception as e: save_trial += 1 def load_state_dict_with_mismatch(model, loaded_state_dict_or_path): """operated in-place, no need to return `model`""" if isinstance(loaded_state_dict_or_path, str): loaded_state_dict = torch.load( loaded_state_dict_or_path, map_location="cpu") else: loaded_state_dict = loaded_state_dict_or_path model_keys = set([k for k in list(model.state_dict().keys())]) load_keys = set(loaded_state_dict.keys()) toload = {} mismatched_shape_keys = [] for k in model_keys: if k in load_keys: if model.state_dict()[k].shape != loaded_state_dict[k].shape: mismatched_shape_keys.append(k) else: toload[k] = loaded_state_dict[k] LOGGER.info("You can ignore the keys with `num_batches_tracked` or from task heads") LOGGER.info("Keys in loaded but not in model:") diff_keys = load_keys.difference(model_keys) LOGGER.info(f"In total {len(diff_keys)}, {sorted(diff_keys)}") LOGGER.info("Keys in model but not in loaded:") diff_keys = model_keys.difference(load_keys) LOGGER.info(f"In total {len(diff_keys)}, {sorted(diff_keys)}") LOGGER.info("Keys in model and loaded, but shape mismatched:") LOGGER.info(f"In total {len(mismatched_shape_keys)}, {sorted(mismatched_shape_keys)}") model.load_state_dict(toload, strict=False) def compare_dict_difference(dict1, dict2, dict1_name="dict1", dict2_name="dict2", print_value_diff=True, verbose=False): """ Args: dict1: dict2: dict1_name: dict2_name: print_value_diff: bool, output dict value difference within shared keys for dict1 and dict2. In effect only when verbose == True verbose: """ keys1 = set(dict1.keys()) keys2 = set(dict2.keys()) shared_keys = keys1.intersection(keys2) keys1_unique = keys1.difference(shared_keys) keys2_unique = keys2.difference(shared_keys) key_diff_list = list(keys1_unique) + list(keys2_unique) # value difference in the shared keys in dict1 and dict2 value_diff_dict = {} for k in shared_keys: if dict1[k] != dict2[k]: value_diff_dict[k] = [(dict1_name, dict1[k]), (dict2_name, dict2[k])] if verbose: LOGGER.info("=" * 30 + "key difference") LOGGER.info(f"keys in {dict1_name} but not in {dict2_name}: " f"total {len(keys1_unique)}, {sorted(keys1_unique)}") LOGGER.info(f"keys in {dict2_name} but not in {dict1_name}: " f"total {len(keys2_unique)}, {sorted(keys2_unique)}") if verbose and print_value_diff: LOGGER.info("=" * 30 + "value difference") LOGGER.info(f"{json.dumps(value_diff_dict, indent=4)}") return value_diff_dict, key_diff_list def _to_cuda(state): """ usually load from cpu checkpoint but need to load to cuda """ if isinstance(state, torch.Tensor): ret = state.cuda() # assume propoerly set py torch.cuda.set_device if 'Half' in state.type(): ret = ret.float() # apex O2 requires it return ret elif isinstance(state, list): new_state = [_to_cuda(t) for t in state] elif isinstance(state, tuple): new_state = tuple(_to_cuda(t) for t in state) elif isinstance(state, dict): new_state = {n: _to_cuda(t) for n, t in state.items()} else: return state return new_state def _to_cpu(state): """ store in cpu to avoid GPU0 device, fp16 to save space """ if isinstance(state, torch.Tensor): ret = state.cpu() if 'Float' in state.type(): ret = ret.half() return ret elif isinstance(state, list): new_state = [_to_cpu(t) for t in state] elif isinstance(state, tuple): new_state = tuple(_to_cpu(t) for t in state) elif isinstance(state, dict): new_state = {n: _to_cpu(t) for n, t in state.items()} else: return state return new_state class TrainingRestorer(object): """ckpt_dict: a dict contains all optimizers/models""" def __init__(self, opts, **ckpt_dict): if exists(opts.output_dir): restore_opts = json.load(open( f'{opts.output_dir}/log/args.json', 'r')) assert opts == edict(restore_opts) # keep 2 checkpoints in case of corrupted self.save_path = f'{opts.output_dir}/restore.pt' self.backup_path = f'{opts.output_dir}/restore_backup.pt' self.ckpt_dict = ckpt_dict self.save_steps = opts.save_steps self.amp = opts.fp16 # since saving to or loading from azure blob fails sometimes self.max_save_load_trial = 10 if exists(self.save_path) or exists(self.backup_path): LOGGER.info('found previous checkpoint. try to resume...') # with retrial, as azure blob fails occasionally. restore_trial = 0 while restore_trial < self.max_save_load_trial: LOGGER.info(f"TrainingRestorer restore trial NO. {restore_trial}") try: self.restore() break except Exception as e: restore_trial += 1 else: self.global_step = 0 def step(self): self.global_step += 1 if self.global_step % self.save_steps == 0: # with retrial, as azure blob fails occasionally. save_trial = 0 while save_trial < self.max_save_load_trial: LOGGER.info(f"TrainingRestorer save trial NO. {save_trial}") try: self.save() break except Exception as e: save_trial += 1 def save(self): checkpoint_to_save = {'global_step': self.global_step} for k in self.ckpt_dict: checkpoint_to_save[k] = _to_cpu(self.ckpt_dict[k].state_dict()) if self.amp: checkpoint_to_save['amp_state_dict'] = amp.state_dict() if exists(self.save_path): os.rename(self.save_path, self.backup_path) torch.save(checkpoint_to_save, self.save_path) def restore(self): try: checkpoint = torch.load(self.save_path) except Exception: checkpoint = torch.load(self.backup_path) self.global_step = checkpoint['global_step'] for k in self.ckpt_dict: self.ckpt_dict[k].load_state_dict(_to_cuda(checkpoint[k])) if self.amp: amp.load_state_dict(checkpoint['amp_state_dict']) LOGGER.info(f'resume training from step {self.global_step}') class E2E_TrainingRestorer(object): def __init__(self, opts, model, optimizer): if exists(f"{opts.output_dir}/log/args.json"): restore_opts = json.load( open(f'{opts.output_dir}/log/args.json', 'r')) with open(join( opts.output_dir, 'log', 'restore_args.json'), 'w') as writer: json.dump(vars(opts), writer, indent=4) # assert opts == edict(restore_opts) # keep 2 checkpoints in case of corrupted self.save_path = f'{opts.output_dir}/restore.pt' self.backup_path = f'{opts.output_dir}/restore_backup.pt' self.model = model self.optimizer = optimizer self.save_steps = int(opts.save_steps_ratio * opts.num_train_steps) self.amp = opts.fp16 # since saving to or loading from azure blob fails sometimes self.max_save_load_trial = 10 if exists(self.save_path) or exists(self.backup_path): LOGGER.info('found previous checkpoint. try to resume...') # with retrial, as azure blob fails occasionally. restore_trial = 0 while restore_trial < self.max_save_load_trial: LOGGER.info(f"TrainingRestorer restore trial NO. {restore_trial}") try: self.restore(opts) break except Exception as e: restore_trial += 1 else: self.global_step = 0 def step(self): self.global_step += 1 if self.global_step % self.save_steps == 0: # with retrial, as azure blob fails occasionally. save_trial = 0 while save_trial < self.max_save_load_trial: LOGGER.info(f"TrainingRestorer save trial NO. {save_trial}") try: self.save() break except Exception as e: save_trial += 1 def save(self): checkpoint = {'global_step': self.global_step, 'model_state_dict': _to_cpu(self.model.state_dict()), 'optim_state_dict': _to_cpu(self.optimizer.state_dict())} if self.amp: checkpoint['amp_state_dict'] = amp.state_dict() if exists(self.save_path): os.rename(self.save_path, self.backup_path) torch.save(checkpoint, self.save_path) def restore(self, opts): try: checkpoint = torch.load(self.save_path) except Exception: checkpoint = torch.load(self.backup_path) self.global_step = checkpoint['global_step'] self.model.load_state_dict(_to_cuda(checkpoint['model_state_dict'])) self.optimizer.load_state_dict( _to_cuda(checkpoint['optim_state_dict'])) if self.amp: amp.load_state_dict(checkpoint['amp_state_dict']) LOGGER.info(f'resume training from step {self.global_step}') def convert_torchvision_ckpt_to_detectron2(ckpt_path) -> Dict[str, Any]: r""" Return state dict of visual backbone which can be loaded with `Detectron2 `_. This is useful for downstream tasks based on Detectron2 (such as object detection and instance segmentation). This method renames certain parameters from Torchvision-style to Detectron2-style. Returns ------- Dict[str, Any] A dict with three keys: ``{"model", "author", "matching_heuristics"}``. These are necessary keys for loading this state dict properly with Detectron2. """ assert os.path.exists(ckpt_path) input_state_dict = torch.load(ckpt_path) # Detectron2 backbones have slightly different module names, this mapping # lists substrings of module names required to be renamed for loading a # torchvision model into Detectron2. DETECTRON2_RENAME_MAPPING: Dict[str, str] = { "layer1": "res2", "layer2": "res3", "layer3": "res4", "layer4": "res5", "bn1": "conv1.norm", "bn2": "conv2.norm", "bn3": "conv3.norm", "downsample.0": "shortcut", "downsample.1": "shortcut.norm", } # Populate this dict by renaming module names. d2_backbone_dict: Dict[str, torch.Tensor] = {} for name, param in input_state_dict.items(): for old, new in DETECTRON2_RENAME_MAPPING.items(): name = name.replace(old, new) # First conv and bn module parameters are prefixed with "stem.". if not name.startswith("res"): name = f"stem.{name}" d2_backbone_dict[name] = param return { "model": d2_backbone_dict, "__author__": "Karan Desai", "matching_heuristics": True, } clip-vip/XPretrain-main/CLIP-Vip/src/utils/logger.py000066400000000000000000000047661475357401500225320ustar00rootroot00000000000000""" references: UNITER """ import logging from tensorboardX import SummaryWriter import os _LOG_FMT = '%(asctime)s - %(levelname)s - %(name)s - %(message)s' _DATE_FMT = '%m/%d/%Y %H:%M:%S' logging.basicConfig(format=_LOG_FMT, datefmt=_DATE_FMT, level=logging.INFO) LOGGER = logging.getLogger('__main__') # this is the global logger def add_log_to_file(log_path): fh = logging.FileHandler(log_path) formatter = logging.Formatter(_LOG_FMT, datefmt=_DATE_FMT) fh.setFormatter(formatter) LOGGER.addHandler(fh) class TensorboardLogger(object): def __init__(self): self._logger = None self._global_step = 0 def create(self, path): if "AZUREML_TB_PATH" in os.environ: self._logger = SummaryWriter(os.environ["AZUREML_TB_PATH"]) else: self._logger = SummaryWriter(path) def noop(self, *args, **kwargs): return def step(self): self._global_step += 1 @property def global_step(self): return self._global_step @global_step.setter def global_step(self, step): self._global_step = step def log_scalar_dict(self, log_dict, prefix=''): """ log a dictionary of scalar values""" if self._logger is None: return if prefix: prefix = f'{prefix}_' for name, value in log_dict.items(): if isinstance(value, dict): self.log_scalar_dict(value, self._global_step, prefix=f'{prefix}{name}') else: self._logger.add_scalar(f'{prefix}{name}', value, self._global_step) def __getattr__(self, name): if self._logger is None: return self.noop return self._logger.__getattribute__(name) TB_LOGGER = TensorboardLogger() class RunningMeter(object): """ running meteor of a scalar value (useful for monitoring training loss) """ def __init__(self, name, val=None, smooth=0.99): self._name = name self._sm = smooth self._val = val def __call__(self, value): self._val = (value if self._val is None else value*(1-self._sm) + self._val*self._sm) def __str__(self): return f'{self._name}: {self._val:.4f}' @property def val(self): return self._val @property def name(self): return self._name clip-vip/XPretrain-main/CLIP-Vip/src/utils/metrics.py000066400000000000000000000042271475357401500227110ustar00rootroot00000000000000import numpy as np def cal_cossim(feats1, feats2): sim_matrix = np.dot(feats1, feats2.T) return sim_matrix def np_softmax(X, theta = 1.0, axis = None): """ Compute the softmax of each element along an axis of X. Parameters ---------- X: ND-Array. Probably should be floats. theta (optional): float parameter, used as a multiplier prior to exponentiation. Default = 1.0 axis (optional): axis to compute values along. Default is the first non-singleton axis. Returns an array the same size as X. The result will sum to 1 along the specified axis. """ # make X at least 2d y = np.atleast_2d(X) # find axis if axis is None: axis = next(j[0] for j in enumerate(y.shape) if j[1] > 1) # multiply y against the theta parameter, y = y * float(theta) # subtract the max for numerical stability y = y - np.expand_dims(np.max(y, axis = axis), axis) # exponentiate y y = np.exp(y) # take the sum along the specified axis ax_sum = np.expand_dims(np.sum(y, axis = axis), axis) # finally: divide elementwise p = y / ax_sum # flatten if X was 1D if len(X.shape) == 1: p = p.flatten() return p def compute_metrics(x): sx = np.sort(-x, axis=1) d = np.diag(-x) d = d[:, np.newaxis] ind = sx - d ind = np.where(ind == 0) ind = ind[1] r1 = float(np.sum(ind == 0)) / len(ind) r5 = float(np.sum(ind < 5)) / len(ind) r10 = float(np.sum(ind < 10)) / len(ind) medr = np.median(ind) + 1 meanr = np.mean(ind) + 1 return r1, r5, r10, medr, meanr def compute_metrics_multi(x, t2v_labels_list): sx = np.sort(-x, axis=1) t2v_labels_list = np.array(t2v_labels_list) arg = np.arange(x.shape[0]) d = -x[arg, t2v_labels_list] d = d[:, np.newaxis] ind = sx - d ind = np.where(ind == 0) ind = ind[1] r1 = float(np.sum(ind == 0)) / len(ind) r5 = float(np.sum(ind < 5)) / len(ind) r10 = float(np.sum(ind < 10)) / len(ind) medr = np.median(ind) + 1 meanr = np.mean(ind) + 1 return r1, r5, r10, medr, meanr if __name__ == '__main__': sim_matrix = np.random.random((5,5)) clip-vip/XPretrain-main/CLIP-Vip/src/utils/misc.py000066400000000000000000000011071475357401500221700ustar00rootroot00000000000000""" modified from UNITER """ import json import random import sys import torch import numpy as np class NoOp(object): """ useful for distributed training No-Ops """ def __getattr__(self, name): return self.noop def noop(self, *args, **kwargs): return def set_random_seed(seed): random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed_all(seed) def zero_none_grad(model): for p in model.parameters(): if p.grad is None and p.requires_grad: p.grad = p.data.new(p.size()).zero_() clip-vip/XPretrain-main/CLIP-Vip/src/utils/stop_words.py000066400000000000000000000073011475357401500234420ustar00rootroot00000000000000"""List of stop words.""" # This list of English stop words is taken from the "Glasgow Information # Retrieval Group". The original list can be found at # http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words ENGLISH_STOP_WORDS = frozenset([ "a", "about", "above", "across", "actually", "after", "afterwards", "again", "against", "all", "almost", "alone", "along", "already", "also", "although", "always", "am", "among", "amongst", "amoungst", "amount", "an", "and", "another", "any", "anyhow", "anyone", "anything", "anyway", "anywhere", "are", "around", "as", "at", "back", "be", "became", "because", "become", "becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", "between", "beyond", "bill", "both", "bottom", "but", "by", "call", "can", "cannot", "cant", "can't", "co", "con", "could", "couldnt", "cry", "de", "describe", "detail", "do", "done", "don't", "down", "due", "during", "each", "easy", "eg", "eight", "either", "eleven", "else", "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", "fifteen", "fifty", "find", "fire", "first", "five", "for", "former", "formerly", "forty", "found", "four", "from", "further", "give", "had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his", "how", "however", "hundred", "i", "ie", "if", "i'm", "i'll", "i've", "in", "inc", "indeed", "interest", "is", "it", "it'll", "its", "it's", "itself", "just", "keep", "last", "latter", "latterly", "least", "less", "like", "ltd", "made", "many", "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "much", "must", "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "ok", "okay", "on", "once", "one", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own", "part", "per", "perhaps", "please", "put", "rather", "re", "really", "same", "see", "seem", "seemed", "seeming", "seems", "serious", "several", "she", "should", "show", "side", "since", "sincere", "six", "sixty", "so", "some", "somehow", "someone", "something", "sometime", "sometimes", "somewhere", "still", "such", "take", "ten", "than", "thank", "thanks", "that", "that's", "the", "their", "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "thereupon", "these", "they", "third", "this", "those", "though", "three", "through", "throughout", "thru", "thus", "to", "together", "too", "top", "toward", "towards", "twelve", "twenty", "two", "un", "until", "up", "upon", "us", "very", "via", "view", "viewing", "viewer", "was", "we", "we'll", "well", "welcome", "were", "what", "whatever", "when", "whence", "whenever", "where", "whereafter", "whereas", "whereby", "wherein", "whereupon", "wherever", "whether", "which", "while", "whither", "who", "whoever", "whole", "whom", "whose", "why", "will", "with", "within", "without", "would", "wont", "won't", "yet", "you", "your", "yours", "you've", "you'll", "yourself", "yourselves", "youtube", "going", "want", "right", "you're", "we're", "know", "gonna", "need", "bit", "look", "yeah", "guys", "sure", "let's", "video", "oh", "let", "today", "they're", "did", "looks", "different", "great", "different", "say", "um", "probably", "kind", "doesn't", "does", "maybe", "hey", "we've", "better", "hope", "there's", "try" ])clip-vip/XPretrain-main/CODE_OF_CONDUCT.md000066400000000000000000000007051475357401500202530ustar00rootroot00000000000000# Microsoft Open Source Code of Conduct This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Resources: - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns clip-vip/XPretrain-main/LICENSE000066400000000000000000000140621475357401500164620ustar00rootroot00000000000000Microsoft Research License Agreement for XPretrain This Microsoft Research License Agreement, including all exhibits (“Agreement”) is a legal agreement between you and Microsoft Corporation (or based on where you live, one of its affiliates). Please read them. They apply to the Microsoft Research software named above, which may include source code and any associated materials, text or speech files, associated media and “online” or electronic documentation and any updates we provide in our discretion (together, the “Software”). The terms also apply to any Microsoft (i) updates, (ii) supplements, (iii) internet-based services, and (iv) support services for this Software, unless other terms accompany those items. If so, those terms apply. By agreeing to this Agreement and/or by using the Software, you accept these terms. If you do not accept them, do not use the Software. If you comply with these license terms, you have the rights below. 1. SCOPE OF RIGHTS. a. License Grant. You may use, copy, modify, and create derivative works of the Software for non-commercial purposes, subject to the restrictions in this Agreement. Examples of non-commercial uses are teaching, academic research, public demonstrations and personal experimentation. b. Publication. You may publish (or present papers or articles) on your results from using the Software, provided that you have made the appropriate attribution to Microsoft. c. Third Party Programs. The Software may include third party programs that Microsoft, not the third party, licenses to you under this Agreement. Notices, if any, for the third party program are included for your information only. 2. DISTRIBUTION RESTRICTIONS. You may not: a. Distribute the Software or derivative works in any form; b. Alter any copyright, trademark or patent notice in the software; c. Use Microsoft’s trademarks in your programs’ names or in a way that suggests your derivative works or modifications come from or are endorsed by Microsoft; or d. Include the software in malicious, deceptive or unlawful programs. 3. SCOPE OF LICENSE. The Software is licensed, not sold. This Agreement only gives you some rights to use the software. Microsoft reserves all other rights. The patent rights, if any, granted to you in this Agreement only apply to the Software, not to any derivative works you make. In using the Software, you must comply with any technical limitations in the Software that may only allow you to use it in certain ways. You may not: a. Work around any technical limitations in the Software; b. Reverse engineer, decompile or disassemble the Software, except and only to the extent that applicable law expressly permits, despite this limitation; c. Use the Software for commercial software hosting services; d. Publish the Software for others to copy; e. Make more copies of the Software than specified in this Agreement or allowed by applicable law, despite this limitation; f. Rent, lease or lend the Software; or g. Transfer the Software or this Agreement to any third party. 4. FEEDBACK. Any feedback about the Software provided by you to us is voluntarily given, and Microsoft shall be free to use the feedback as it sees fit without obligation or restriction of any kind, even if the feedback is designated by you as confidential. 5. TERM; TERMINATION. The term of this Agreement will commence upon your acceptance of these license terms and will continue indefinitely unless terminated earlier as provided herein. If you breach this Agreement or if you sue Microsoft or any other party over patents that you think may apply to or read on the Software or anyone's use of the Software, this Agreement (and your license and rights obtained herein) terminate automatically. If this Agreement is terminated, you must cease using and distributing any derivative works or modifications of the Software. Any sections that are intended to survive termination of this Agreement shall survive. 6. EXPORT RESTRICTIONS. The Software is subject to United States export laws and regulations. You must comply with all domestic and international export laws and regulations that apply to the Software. These laws include restrictions on destinations, end users and end use. For additional information, see www.microsoft.com/exporting. 7. ENTIRE AGREEMENT. This Agreement, any exhibits, and the terms for supplements, updates, Internet-based services and support services that you use, are the entire agreement for the Software and support services. 8. SEVERABILITY. If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, the remaining provisions will remain in full force and effect. 9. GOVERNING LAW AND VENUE. This Agreement is governed by and construed in accordance with the laws of the state of Washington, without reference to its choice of law principles to the contrary. Each party hereby consents to the jurisdiction and venue of the state and federal courts located in King County, Washington, with regard to any suit or claim arising under or by reason of this Agreement. 10. LEGAL EFFECT. This Agreement describes certain legal rights. You may have other rights under the laws of your country. You may also have rights with respect to the party from whom you acquired the dataset. This Agreement does not change your rights under the laws of your country if the laws of your country do not permit it to do so. 11. NO ASSIGNMENT. You may not assign this Agreement or any rights or obligations hereunder, except with Microsoft’s express written consent. Any attempted assignment in violation of this section will be void. 12. DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS-IS.” YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES OR CONDITIONS. YOU MAY HAVE ADDITIONAL CONSUMER RIGHTS OR STATUTORY GUARANTEES UNDER YOUR LOCAL LAWS WHICH THIS AGREEMENT CANNOT CHANGE. TO THE EXTENT PERMITTED UNDER YOUR LOCAL LAWS, MICROSOFT EXCLUDES THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. clip-vip/XPretrain-main/RAI.md000066400000000000000000000050421475357401500164100ustar00rootroot00000000000000Responsible AI Considerations The proposed video-language dataset and pre-training model shows the capacity and generalization of learned VL representation which could benefit many applications of CV and NLP with a large range of uses across many domains. Each one of the uses has potential benefits and societal impacts. While we foresee that our technology could be used to find key information and improve efficiency and effectiveness for helpdesks, recommendation, retail and sales, we realize that it could also be used, in combination with new data, to fine-tune models to mislead, or otherwise harm people. We are also aware that this work uses considerable computation resources which itself, has environmental impacts. Therefore reducing the model size and computing effort is essential for future research. Machine learning systems can display unfair behavior for different individuals or groups. This is a multi-dimensional, socio-technical challenge and is not explicitly addressed or captured in the current accuracy metrics for this research technology. In general, standardized fairness measures have not yet been agreed upon in academia or industry. We see opportunities for more work in this area to develop methods and benchmarks for measuring fairness aspects. Given that user generated data is used, it is possible that certain demographic groups may not have enough representation. While we have balanced various video categories to mitigate for disparities, it is still likely that bias and fairness issues exist; this is an area of potential future work. There may be a Western heteronormative bias, stereotypical depictions of historically marginalized populations and/or lack of representation among some groups. Although we have filtered the input data for explicit and violent content, it is possible that it hasn’t been totally eliminated in the training data and could have impacts on the results. With visual generation techniques it is particularly important to do further work to prevent malicious use to misinform or harm people. While some mitigations for potential harms can be done in the base model, it’s important to recognize that considering risks for fine tuning data for particular scenarios is critical as well. Ultimately, choosing the application scenario of any final model used in a production system will require careful consideration of potential harms specific to the scenario. For help or issues using the pre-trained models, please submit an issue or contact Bei Liu (bei.liu@microsoft.com) and Jianlong Fu (jianf@microsoft.com). clip-vip/XPretrain-main/README.md000066400000000000000000000077521475357401500167440ustar00rootroot00000000000000# XPretrain This repo includes some recent research works in **multi-modality learning**, especially with **pre-training** method from [MSM group](https://www.microsoft.com/en-us/research/group/multimedia-search-and-mining/) of Microsoft Research. ## Multi-modality Learning ### ***** Video & Language ***** #### Dataset > [**HD-VILA-100M dataset**](https://github.com/microsoft/XPretrain/tree/main/hd-vila-100m): high-resolution and diversified video-language dataset #### Pre-training model > [**HD-VILA (CVPR 2022)**](https://github.com/microsoft/XPretrain/tree/main/hd-vila): high-resolution and diversified video-language pre-training model > [**LF-VILA (NeurIPS 2022)**](https://github.com/microsoft/XPretrain/tree/main/LF-VILA): long-form video-language pre-training model > [**CLIP-ViP (ICLR 2023)**](https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP): adapting image-language pre-training to video-language pretraining model ### ***** Image & Language ***** #### Pre-training model > [**Pixel-BERT**](https://arxiv.org/pdf/2004.00849.pdf): end-to-end image and language pre-training model > [**SOHO (CVPR 2021 oral)**](https://github.com/researchmm/soho): improved end-to-end image and language pre-training model with quantized visual tokens > [**VisualParsing (NeurIPS 2021)**](https://github.com/microsoft/XPretrain/tree/main/visualparsing): Transformer-based end-to-end image and language pre-training model ## News - :smiley:**March, 2023: the code of [**CLIP-ViP**](https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP) and [**LF-VILA**](https://github.com/microsoft/XPretrain/tree/main/LF-VILA) was released.** - January, 2023: our paper [**CLIP-ViP**](https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP) to adapt image-language pre-training model to video-language pretraining was accepted by ICLR 2023. - September, 2022: our paper [**LF-VILA**](https://github.com/microsoft/XPretrain/tree/main/LF-VILA) on long-form video-language pre-training was accepted by NeurIPS 2022. - September, 2022: the code of [**HD-VILA**](https://github.com/microsoft/XPretrain/tree/main/hd-vila) was released. - March, 2022: [**HD-VILA-100M dataset**](https://github.com/microsoft/XPretrain/tree/main/hd-vila-100m) was released publicly. - March, 2022: [**HD-VILA**](https://github.com/microsoft/XPretrain/tree/main/hd-vila) was accepted by CVPR 2022. ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies. ## Contact Information For help or issues using the pre-trained models, please submit an issue. For other communications, please contact [Bei Liu]() (`bei.liu@microsoft.com`) and [Jianlong Fu]() (`jianf@microsoft.com`). clip-vip/XPretrain-main/SECURITY.md000066400000000000000000000054041475357401500172460ustar00rootroot00000000000000 ## Security Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. ## Reporting Security Issues **Please do not report security vulnerabilities through public GitHub issues.** Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) * Full paths of source file(s) related to the manifestation of the issue * The location of the affected source code (tag/branch/commit or direct URL) * Any special configuration required to reproduce the issue * Step-by-step instructions to reproduce the issue * Proof-of-concept or exploit code (if possible) * Impact of the issue, including how an attacker might exploit the issue This information will help us triage your report more quickly. If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. ## Preferred Languages We prefer all communications to be in English. ## Policy Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). clip-vip/XPretrain-main/SUPPORT.md000066400000000000000000000024431475357401500171530ustar00rootroot00000000000000# TODO: The maintainer of this repo has not yet edited this file **REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project? - **No CSS support:** Fill out this template with information about how to file issues and get help. - **Yes CSS support:** Fill out an intake form at [aka.ms/spot](https://aka.ms/spot). CSS will work with/help you to determine next steps. More details also available at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). - **Not sure?** Fill out a SPOT intake as though the answer were "Yes". CSS will help you decide. *Then remove this first heading from this SUPPORT.MD file before publishing your repo.* # Support ## How to file issues and get help This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as a new Issue. For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER CHANNEL. WHERE WILL YOU HELP PEOPLE?**. ## Microsoft Support Policy Support for this **PROJECT or PRODUCT** is limited to the resources listed above. clip-vip/XPretrain-main/hd-vila/000077500000000000000000000000001475357401500167765ustar00rootroot00000000000000clip-vip/XPretrain-main/hd-vila/LICENSE000066400000000000000000000140621475357401500200060ustar00rootroot00000000000000Microsoft Research License Agreement for XPretrain This Microsoft Research License Agreement, including all exhibits (“Agreement”) is a legal agreement between you and Microsoft Corporation (or based on where you live, one of its affiliates). Please read them. They apply to the Microsoft Research software named above, which may include source code and any associated materials, text or speech files, associated media and “online” or electronic documentation and any updates we provide in our discretion (together, the “Software”). The terms also apply to any Microsoft (i) updates, (ii) supplements, (iii) internet-based services, and (iv) support services for this Software, unless other terms accompany those items. If so, those terms apply. By agreeing to this Agreement and/or by using the Software, you accept these terms. If you do not accept them, do not use the Software. If you comply with these license terms, you have the rights below. 1. SCOPE OF RIGHTS. a. License Grant. You may use, copy, modify, and create derivative works of the Software for non-commercial purposes, subject to the restrictions in this Agreement. Examples of non-commercial uses are teaching, academic research, public demonstrations and personal experimentation. b. Publication. You may publish (or present papers or articles) on your results from using the Software, provided that you have made the appropriate attribution to Microsoft. c. Third Party Programs. The Software may include third party programs that Microsoft, not the third party, licenses to you under this Agreement. Notices, if any, for the third party program are included for your information only. 2. DISTRIBUTION RESTRICTIONS. You may not: a. Distribute the Software or derivative works in any form; b. Alter any copyright, trademark or patent notice in the software; c. Use Microsoft’s trademarks in your programs’ names or in a way that suggests your derivative works or modifications come from or are endorsed by Microsoft; or d. Include the software in malicious, deceptive or unlawful programs. 3. SCOPE OF LICENSE. The Software is licensed, not sold. This Agreement only gives you some rights to use the software. Microsoft reserves all other rights. The patent rights, if any, granted to you in this Agreement only apply to the Software, not to any derivative works you make. In using the Software, you must comply with any technical limitations in the Software that may only allow you to use it in certain ways. You may not: a. Work around any technical limitations in the Software; b. Reverse engineer, decompile or disassemble the Software, except and only to the extent that applicable law expressly permits, despite this limitation; c. Use the Software for commercial software hosting services; d. Publish the Software for others to copy; e. Make more copies of the Software than specified in this Agreement or allowed by applicable law, despite this limitation; f. Rent, lease or lend the Software; or g. Transfer the Software or this Agreement to any third party. 4. FEEDBACK. Any feedback about the Software provided by you to us is voluntarily given, and Microsoft shall be free to use the feedback as it sees fit without obligation or restriction of any kind, even if the feedback is designated by you as confidential. 5. TERM; TERMINATION. The term of this Agreement will commence upon your acceptance of these license terms and will continue indefinitely unless terminated earlier as provided herein. If you breach this Agreement or if you sue Microsoft or any other party over patents that you think may apply to or read on the Software or anyone's use of the Software, this Agreement (and your license and rights obtained herein) terminate automatically. If this Agreement is terminated, you must cease using and distributing any derivative works or modifications of the Software. Any sections that are intended to survive termination of this Agreement shall survive. 6. EXPORT RESTRICTIONS. The Software is subject to United States export laws and regulations. You must comply with all domestic and international export laws and regulations that apply to the Software. These laws include restrictions on destinations, end users and end use. For additional information, see www.microsoft.com/exporting. 7. ENTIRE AGREEMENT. This Agreement, any exhibits, and the terms for supplements, updates, Internet-based services and support services that you use, are the entire agreement for the Software and support services. 8. SEVERABILITY. If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, the remaining provisions will remain in full force and effect. 9. GOVERNING LAW AND VENUE. This Agreement is governed by and construed in accordance with the laws of the state of Washington, without reference to its choice of law principles to the contrary. Each party hereby consents to the jurisdiction and venue of the state and federal courts located in King County, Washington, with regard to any suit or claim arising under or by reason of this Agreement. 10. LEGAL EFFECT. This Agreement describes certain legal rights. You may have other rights under the laws of your country. You may also have rights with respect to the party from whom you acquired the dataset. This Agreement does not change your rights under the laws of your country if the laws of your country do not permit it to do so. 11. NO ASSIGNMENT. You may not assign this Agreement or any rights or obligations hereunder, except with Microsoft’s express written consent. Any attempted assignment in violation of this section will be void. 12. DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS-IS.” YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES OR CONDITIONS. YOU MAY HAVE ADDITIONAL CONSUMER RIGHTS OR STATUTORY GUARANTEES UNDER YOUR LOCAL LAWS WHICH THIS AGREEMENT CANNOT CHANGE. TO THE EXTENT PERMITTED UNDER YOUR LOCAL LAWS, MICROSOFT EXCLUDES THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. clip-vip/XPretrain-main/hd-vila/README.md000066400000000000000000000211541475357401500202600ustar00rootroot00000000000000# HD-VILA By [Hongwei Xue](https://hellwayxue.github.io/)\*, [Tiankai Hang](https://tiankaihang.github.io/)\*, [Yanhong Zeng](https://1900zyh.github.io/)\*, [Yuchong Sun](https://github.com/ycsun1972)\*, [Bei Liu](https://www.microsoft.com/en-us/research/people/libei/), [Huan Yang](https://www.microsoft.com/en-us/research/people/huayan/), [Jianlong Fu](https://www.microsoft.com/en-us/research/people/jianf/), and [Baining Guo](https://www.microsoft.com/en-us/research/people/bainguo/). The repo is the official implemenation of ["Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions"](https://arxiv.org/abs/2111.10337). It currently contains code and models for > Large-scale pre-training on [HD-VILA-100M](../hd-vila-100m/README.md) > Downstream tasks including video-text retrieval, video question answering. ## Introdution We propose a novel **H**igh-resolution and **D**iversified **VI**deo-**LA**nguage pre-training model (HD-VILA) for many visual tasks. To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts. Our pre-training model achieves new state-of-the-art results in **10** VL understanding tasks and **2** more novel text-to-visual generation tasks. We outperform SOTA models with relative increases of 40.4% R@1 in zero-shot MSR-VTT text-to-video retrieval task, and 55.4% in high-resolution dataset LSMDC. The learned VL embedding is also effective in generating visually pleasing and semantically relevant results in text-to-visual editing and super-resolution tasks.

statistics

The framework of HD-VILA.

## Getting Started ### General 1. Create a folder that stores pretrained models, all the data, and results. ```bash PATH_TO_STORAGE=/path/to/your/data/ ``` 2. Clone this repo and launch the Docker container for running the experiments. If you want to pre-train on your own dataset, please prepare the environment with `horovod`. It is a better choice to use the pre-built docker image `tiankaihang/azureml_docker:horovod`. Or you can build from the [dockerfile](./docker/Dockerfile). We use mixed-precision training hence GPUs with Tensor Cores are recommended. ```bash # command to get into the container, docker image should be automatically pulled. cd HD-VILA source launch_container.sh $PATH_TO_STORAGE ``` ### Prepare data and pretrained model 1. Use [scripts/download_data.sh](scripts/download_data.sh) to download our pretrained model. In stage-one pretraining, we load pre-trained ResNet50 and BERT-large. Our scripts will also download these model weights. ```bash bash scripts/download_data.sh $PATH_TO_STORAGE ``` 2. The above script will also download the annotations for all downstream tasks. + For Pre-training dataset, please refer to [HD-VILA-100M](../hd-vila-100m/README.md) to prepare pre-training data. Please download the annotations and videos, then put them to $PATH_TO_STORAGE/data/hdvila. + For MSRVTT, the official data and video links can be found in [link](http://ms-multimedia-challenge.com/2017/dataset). For convience, you can download the raw videos from the sharing by [Frozen️ in Time](https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip). Please download the raw videos and put them to $PATH_TO_STORAGE/data/msrvtt_retrieval/videos. We compresse the frame rate to speed up data reading, the compressed videos will be stored in $PATH_TO_STORAGE/data/msrvtt_retrieval/videos_6fps. + For ActivityNet, you can download the raw videos refer to the [official website](http://activity-net.org/download.html). We use the split as [collaborative-experts](https://github.com/albanie/collaborative-experts/tree/master/misc/datasets/activity-net). Please download the raw videos and put them to $PATH_TO_STORAGE/data/activitynet_retrieval/videos. We decode the frame from the original video to speed up data reading. The decoded frames will be stored in $PATH_TO_STORAGE/data/activitynet_retrieval/video_frames and $PATH_TO_STORAGE/data/activitynet_retrieval/video_frames_lr. + For DiDeMo, the raw videos can be download from [LisaAnne/LocalizingMoments](https://github.com/LisaAnne/LocalizingMoments). We use the split as [collaborative-experts](https://github.com/albanie/collaborative-experts/tree/master/misc/datasets/activity-net). We decode the frame from the original video to speed up data reading the same as ActivityNet. + For LSMDC, the raw videos can be download from the [official website](https://sites.google.com/site/describingmovies/download). We use the official split. We decode the frame from the original video to speed up data reading the same as ActivityNet. + For TGIF-QA, you can download the raw videos from the [official page](https://github.com/YunseokJANG/tgif-qa/blob/master/dataset/README.md). Please download the raw videos and put them to $PATH_TO_STORAGE/data/tgif_qa, and then use [gifmp4.py] to convert the videos to MP4(scripts/process_raw_video/gifmp4.py). + For MSRVTT-QA and MSRVTT-MC, we use the videos the same as MSRVTT Retrieval task. + To speed up data reading, we compresse the frame rate or decode the frame from the original video. We provide the processing code [here](scripts/process_raw_video). ### Pre-training Our pre-training consists of two stages. Stage one use contrastive learning to train the dual encoder. Then stage two use the masked modeling to train the cross-modal Transformer. During stage two, the parameters in stage one are frozen. For stage one: ```bash #inside the container horovodrun -np $NUM_GPUS python src/pretrain/run_pretrain_stage1_group.py \ --config src/configs/pretrain_stage1.json ``` For stage two: ```bash #inside the container horovodrun -np $NUM_GPUS python src/pretrain/run_pretrain_stage2_group.py \ --config src/configs/pretrain_stage2.json ``` Notice that replace the `e2e_weights_path` in pretrain_stage2 with your stage one path. ### Finetune the model for downstream tasks Tasks: MSRVTT, DiDeMo, LSMDC and ActivityNet video-text retrieval, MSRVTT MC Test. MSRVTT and T-GIF QA. Please replace the data path in [src/configs](src/configs) with your own data path. 1. Finetuning Retrieval. ```bash # inside the container horovodrun -np $NUM_GPUS python src/tasks/run_video_retrieval.py \ --config $CONFIG_PATH ``` `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs](src/configs) postfixed with `_retrieval.json`. For example, you can use `src/configs/msrvtt_retrieval.json` for MSRVTT retrieval. 2. Finetuning Video QA. ```bash # inside the container horovodrun -np $NUM_GPUS python src/tasks/run_video_qa.py \ --config $CONFIG_PATH ``` `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs](src/configs) postfixed with `_qa.json`. For example, you can use `src/configs/msrvtt_qa.json` for MSRVTT Video QA. 3. MSRVTT Multi-choice Inference. After MSRVTT retrieval model is trained, you can use it for inference for the MSRVTT MC Test task as well, which is essentially a retrieval task in a multiple-choice task setup. ```bash # inside the container horovodrun -np $NUM_GPUS python src/tasks/run_msrvtt_mc.py \ --config src/configs/msrvtt_retrieval.json --do_inference 1 --output_dir $OUTPUT_DIR \ --inference_split val --inference_model_step $STEP \ --inference_txt_db /path/to/msrvtt_retrieval/mc_test.jsonl \ --inference_img_db /path/to/msrvtt_video --inference_batch_size 64 \ --inference_n_clips $INFERENCE_N_CLIPS ``` ## License The license of the code and pre-trained models is [here](./LICENSE). ## Citing HD-VILA If you find the code and pre-trained models useful for your research, please consider citing our paper. :blush: ```bibtex @inproceedings{xue2022advancing, title={Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions}, author={Xue, Hongwei and Hang, Tiankai and Zeng, Yanhong and Sun, Yuchong and Liu, Bei and Yang, Huan and Fu, Jianlong and Guo, Baining}, booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2022} } ``` ## Acknowledgements The code is based on [ClipBERT](https://github.com/jayleicn/ClipBERT), thanks to them! clip-vip/XPretrain-main/hd-vila/launch_container.sh000066400000000000000000000007411475357401500226500ustar00rootroot00000000000000DATA_DIR=$1 if [ -z $CUDA_VISIBLE_DEVICES ]; then CUDA_VISIBLE_DEVICES='all' fi docker run --gpus '"'device=$CUDA_VISIBLE_DEVICES'"' --ipc=host --rm -it \ --mount src=$(pwd),dst=/HD-VILA,type=bind \ --mount src=$DATA_DIR,dst=/data_mount,type=bind \ -e NVIDIA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \ -w /HD-VILA tiankaihang/azureml_docker:horovod \ bash -c "source /HD-VILA/setup.sh && export OMPI_MCA_btl_vader_single_copy_mechanism=none && bash" clip-vip/XPretrain-main/hd-vila/setup.sh000066400000000000000000000002541475357401500204730ustar00rootroot00000000000000#!/usr/bin/env bash # source setup.sh export DIR_PWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" export PYTHONPATH="$PYTHONPATH:$DIR_PWD" echo $PYTHONPATH