40 lines
10 KiB
JSON
40 lines
10 KiB
JSON
{
|
|
"summary": "Both comments discuss using CLIP, a pre-trained model, for image and text feature extraction. Comment A focuses on CIFAR100 feature extraction and similarity computation, while Comment B covers logistic regression implementation, hyperparameter tuning, and utilizes OpenCLIP and Hugging Face CLIP implementations.",
|
|
"details": [
|
|
{
|
|
"comment": "The code provides a brief introduction to CLIP, a neural network trained on various image-text pairs. It can predict relevant text based on an image without directly optimizing for the task and matches the performance of ResNet50 on ImageNet \"zero-shot\" without using any labeled examples. The code also explains how to install necessary dependencies and set up the environment to use CLIP.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":0-16",
|
|
"content": "# CLIP\n[[Blog]](https://openai.com/blog/clip/) [[Paper]](https://arxiv.org/abs/2103.00020) [[Model Card]](model-card.md) [[Colab]](https://colab.research.google.com/github/openai/clip/blob/master/notebooks/Interacting_with_CLIP.ipynb)\nCLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet \u201czero-shot\u201d without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.\n## Approach\n\n## Usage\nFirst, [install PyTorch 1.7.1](https://pytorch.org/get-started/locally/) (or later) and torchvision, as well as small additional dependencies, and then install this repo as a Python package. On a CUDA GPU machine, the following will do the trick:"
|
|
},
|
|
{
|
|
"comment": "Code installs necessary packages for running CLIP and loads the model.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":18-54",
|
|
"content": "```bash\n$ conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0\n$ pip install ftfy regex tqdm\n$ pip install git+https://github.com/openai/CLIP.git\n```\nReplace `cudatoolkit=11.0` above with the appropriate CUDA version on your machine or `cpuonly` when installing on a machine without a GPU.\n```python\nimport torch\nimport clip\nfrom PIL import Image\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel, preprocess = clip.load(\"ViT-B/32\", device=device)\nimage = preprocess(Image.open(\"CLIP.png\")).unsqueeze(0).to(device)\ntext = clip.tokenize([\"a diagram\", \"a dog\", \"a cat\"]).to(device)\nwith torch.no_grad():\n image_features = model.encode_image(image)\n text_features = model.encode_text(text)\n logits_per_image, logits_per_text = model(image, text)\n probs = logits_per_image.softmax(dim=-1).cpu().numpy()\nprint(\"Label probs:\", probs) # prints: [[0.9927937 0.00421068 0.00299572]]\n```\n## API\nThe CLIP module `clip` provides the following methods:\n#### `clip.available_models()`\nReturns the names of the available CLIP models."
|
|
},
|
|
{
|
|
"comment": "This code snippet is for the CLIP library, which provides a model for visual-textual similarity. It includes two primary functions: `clip.load()` and `clip.tokenize()`. The `clip.load()` function loads a pre-trained CLIP model specified by the `name` parameter or downloads it if necessary. The `clip.tokenize()` function tokenizes input text(s) and returns LongTensor containing the tokenized sequences. The loaded model also supports two methods: `model.encode_image()` to encode image features and `model.encode_text()` to encode text features.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":56-76",
|
|
"content": "#### `clip.load(name, device=..., jit=False)`\nReturns the model and the TorchVision transform needed by the model, specified by the model name returned by `clip.available_models()`. It will download the model as necessary. The `name` argument can also be a path to a local checkpoint.\nThe device to run the model can be optionally specified, and the default is to use the first CUDA device if there is any, otherwise the CPU. When `jit` is `False`, a non-JIT version of the model will be loaded.\n#### `clip.tokenize(text: Union[str, List[str]], context_length=77)`\nReturns a LongTensor containing tokenized sequences of given text input(s). This can be used as the input to the model\n---\nThe model returned by `clip.load()` supports the following methods:\n#### `model.encode_image(image: Tensor)`\nGiven a batch of images, returns the image features encoded by the vision portion of the CLIP model.\n#### `model.encode_text(text: Tensor)`\nGiven a batch of text tokens, returns the text features encoded by the language portion of the CLIP model."
|
|
},
|
|
{
|
|
"comment": "The code snippet loads the CLIP model (ViT-B/32) and prepares inputs for zero-shot prediction using an image from the CIFAR-100 dataset.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":78-105",
|
|
"content": "#### `model(image: Tensor, text: Tensor)`\nGiven a batch of images and a batch of text tokens, returns two Tensors, containing the logit scores corresponding to each image and text input. The values are cosine similarities between the corresponding image and text features, times 100.\n## More Examples\n### Zero-Shot Prediction\nThe code below performs zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), and predicts the most likely labels among the 100 textual labels from the dataset.\n```python\nimport os\nimport clip\nimport torch\nfrom torchvision.datasets import CIFAR100\n# Load the model\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel, preprocess = clip.load('ViT-B/32', device)\n# Download the dataset\ncifar100 = CIFAR100(root=os.path.expanduser(\"~/.cache\"), download=True, train=False)\n# Prepare the inputs\nimage, class_id = cifar100[3637]\nimage_input = preprocess(image).unsqueeze(0).to(device)"
|
|
},
|
|
{
|
|
"comment": "This code calculates the similarity between image features and text features using dot product and softmax, then prints the top 5 most similar labels for the given image.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":106-137",
|
|
"content": "text_inputs = torch.cat([clip.tokenize(f\"a photo of a {c}\") for c in cifar100.classes]).to(device)\n# Calculate features\nwith torch.no_grad():\n image_features = model.encode_image(image_input)\n text_features = model.encode_text(text_inputs)\n# Pick the top 5 most similar labels for the image\nimage_features /= image_features.norm(dim=-1, keepdim=True)\ntext_features /= text_features.norm(dim=-1, keepdim=True)\nsimilarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)\nvalues, indices = similarity[0].topk(5)\n# Print the result\nprint(\"\\nTop predictions:\\n\")\nfor value, index in zip(values, indices):\n print(f\"{cifar100.classes[index]:>16s}: {100 * value.item():.2f}%\")\n```\nThe output will look like the following (the exact numbers may be slightly different depending on the compute device):\n```\nTop predictions:\n snake: 65.31%\n turtle: 12.29%\n sweet_pepper: 3.83%\n lizard: 1.88%\n crocodile: 1.75%\n```\nNote that this example uses the `encode_image()` and `encode_text()` methods that return the encoded features of given inputs."
|
|
},
|
|
{
|
|
"comment": "This code loads a pre-trained CLIP model, then applies it to CIFAR100 dataset for feature extraction using logistic regression.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":140-176",
|
|
"content": "### Linear-probe evaluation\nThe example below uses [scikit-learn](https://scikit-learn.org/) to perform logistic regression on image features.\n```python\nimport os\nimport clip\nimport torch\nimport numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom torch.utils.data import DataLoader\nfrom torchvision.datasets import CIFAR100\nfrom tqdm import tqdm\n# Load the model\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel, preprocess = clip.load('ViT-B/32', device)\n# Load the dataset\nroot = os.path.expanduser(\"~/.cache\")\ntrain = CIFAR100(root, download=True, train=True, transform=preprocess)\ntest = CIFAR100(root, download=True, train=False, transform=preprocess)\ndef get_features(dataset):\n all_features = []\n all_labels = []\n with torch.no_grad():\n for images, labels in tqdm(DataLoader(dataset, batch_size=100)):\n features = model.encode_image(images.to(device))\n all_features.append(features)\n all_labels.append(labels)\n return torch.cat(all_features).cpu().numpy(), torch.cat(all_labels).cpu().numpy()"
|
|
},
|
|
{
|
|
"comment": "Calculating image features, performing logistic regression, and evaluating using a logistic regression classifier.\nHyperparameter C should be determined via validation split.\nSee also OpenCLIP and Hugging Face CLIP implementation.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/README.md\":178-198",
|
|
"content": "# Calculate the image features\ntrain_features, train_labels = get_features(train)\ntest_features, test_labels = get_features(test)\n# Perform logistic regression\nclassifier = LogisticRegression(random_state=0, C=0.316, max_iter=1000, verbose=1)\nclassifier.fit(train_features, train_labels)\n# Evaluate using the logistic regression classifier\npredictions = classifier.predict(test_features)\naccuracy = np.mean((test_labels == predictions).astype(float)) * 100.\nprint(f\"Accuracy = {accuracy:.3f}\")\n```\nNote that the `C` value should be determined via a hyperparameter sweep using a validation split.\n## See Also\n* [OpenCLIP](https://github.com/mlfoundations/open_clip): includes larger and independently trained CLIP models up to ViT-G/14\n* [Hugging Face implementation of CLIP](https://huggingface.co/docs/transformers/model_doc/clip): for easier integration with the HF ecosystem"
|
|
}
|
|
]
|
|
} |