45 lines
12 KiB
JSON
45 lines
12 KiB
JSON
{
|
|
"summary": "OpenAI's CLIP model is a multimodal AI for computer vision and zero-shot image classification. It uses ResNet50 or Vision Transformer as encoders but has limitations like dataset building, performance variability, and potential biases. Training data includes website crawling and YFCC100M datasets. The code provides a Google Form link for feedback on model performance and risks.",
|
|
"details": [
|
|
{
|
|
"comment": "Storage location: \"model-card.md\":0-14\nCode description: This code is a model card for CLIP, a multimodal model developed by OpenAI researchers. The model aims to understand what contributes to robustness in computer vision tasks and test generalization abilities in zero-shot image classification tasks. It was not designed for general deployment and requires careful study before being used in specific contexts. The model card provides details on the development date, model type (ResNet50 with modifications as an image encoder and a masked self-attention Transformer as a text encoder), and that the encoders are trained to maximize similarity of inputs.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":0-14",
|
|
"content": "# Model Card: CLIP\nInspired by [Model Cards for Model Reporting (Mitchell et al.)](https://arxiv.org/abs/1810.03993) and [Lessons from Archives (Jo & Gebru)](https://arxiv.org/pdf/1912.10389.pdf), we\u2019re providing some accompanying information about the multimodal model.\n## Model Details\nThe CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they\u2019re being deployed within.\n### Model Date\nJanuary 2021\n### Model Type\nThe base model uses a ResNet50 with several modifications as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (i"
|
|
},
|
|
{
|
|
"comment": "This code describes the CLIP model, a contrastive image-text model with variants using Vision Transformer or ResNet image encoder. It mentions the different released versions of the model and provides links to relevant documents such as the blog post and paper for further details on specifications and intended use.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":14-35",
|
|
"content": "mage, text) pairs via a contrastive loss. There is also a variant of the model where the ResNet image encoder is replaced with a Vision Transformer.\n### Model Versions\nInitially, we\u2019ve released one CLIP model based on the Vision Transformer architecture equivalent to ViT-B/32, along with the RN50 model, using the architecture equivalent to ResNet-50.\nAs part of the staged release process, we have also released the RN101 model, as well as RN50x4, a RN50 scaled up 4x according to the [EfficientNet](https://arxiv.org/abs/1905.11946) scaling rule. In July 2021, we additionally released the RN50x16 and ViT-B/16 models, and in January 2022, the RN50x64 and ViT-L/14 models were released. Lastly, the ViT-L/14@336px model was released in April 2022.\nPlease see the paper linked below for further details about their specification.\n### Documents\n- [Blog Post](https://openai.com/blog/clip/)\n- [CLIP Paper](https://arxiv.org/abs/2103.00020)\n## Model Use\n### Intended Use\nThe model is intended as a research outp"
|
|
},
|
|
{
|
|
"comment": "This code snippet provides information about the intended use and out-of-scope use cases for a specific model. It explains that the primary audience is AI researchers who will use it to study various aspects of computer vision models, such as robustness, generalization, capabilities, biases, and constraints. Deployed use cases are currently out of scope, while non-deployed use cases should only be considered after thorough in-domain testing with a fixed class taxonomy.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":35-45",
|
|
"content": "ut for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.\n#### Primary intended uses\nThe primary intended users of these models are AI researchers.\nWe primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.\n### Out-of-Scope Use Cases\n**Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonst"
|
|
},
|
|
{
|
|
"comment": "The code highlights the need for task-specific testing due to CLIP's performance variability and cautions against unconstrained deployment in certain use cases. It also emphasizes the model's English language limitations and provides information on the data used for training, including crawling websites and using pre-existing datasets like YFCC100M.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":45-55",
|
|
"content": "rated a high need for task specific testing especially given the variability of CLIP\u2019s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. \nCertain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.\nSince the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.\n## Data\nThe model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the"
|
|
},
|
|
{
|
|
"comment": "The code is describing the data used in building a dataset, its mission statement, and discussing performance and limitations. The data comes from internet crawling, mainly focusing on more developed nations and younger male users. The goal was to test robustness and generalizability in computer vision tasks. The dataset will not be released for commercial or deployed use. Performance is evaluated across various benchmarks and computer vision datasets like OCR to text.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":55-67",
|
|
"content": " data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.\n### Data Mission Statement\nOur goal with building this dataset was to test out robustness and generalizability in computer vision tasks. As a result, the focus was on gathering large quantities of data from different publicly-available internet data sources. The data was gathered in a mostly non-interventionist manner. However, we only crawled websites that had policies against excessively violent and adult images and allowed us to filter out such content. We do not intend for this dataset to be used as the basis for any commercial or deployed model and will not be releasing the dataset.\n## Performance and Limitations\n### Performance\nWe have evaluated the performance of CLIP on a wide range of benchmarks across a variety of computer vision datasets such as OCR to textu"
|
|
},
|
|
{
|
|
"comment": "The code lists various datasets used in the evaluation of the model's performance.\n\nIt highlights that CLIP has limitations, such as difficulties with fine-grained classification and counting objects. It also addresses issues related to fairness and bias, while noting a limitation in their approach by using linear probes for evaluation, which may underestimate model performance.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":67-105",
|
|
"content": "re recognition to fine-grained classification. The paper describes model performance on the following datasets:\n- Food101\n- CIFAR10 \n- CIFAR100 \n- Birdsnap\n- SUN397\n- Stanford Cars\n- FGVC Aircraft\n- VOC2007\n- DTD\n- Oxford-IIIT Pet dataset\n- Caltech101\n- Flowers102\n- MNIST \n- SVHN \n- IIIT5K \n- Hateful Memes \n- SST-2\n- UCF101\n- Kinetics700\n- Country211\n- CLEVR Counting\n- KITTI Distance\n- STL-10\n- RareAct\n- Flickr30\n- MSCOCO\n- ImageNet\n- ImageNet-A\n- ImageNet-R\n- ImageNet Sketch\n- ObjectNet (ImageNet Overlap)\n- Youtube-BB\n- ImageNet-Vid\n## Limitations\nCLIP and our analysis of it have a number of limitations. CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which we discuss in the paper and briefly in the next section. Additionally, our approach to testing CLIP also has an important limitation- in many cases we have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance."
|
|
},
|
|
{
|
|
"comment": "Discusses the impact of class design on CLIP's biases, highlights disparities based on race and gender using Fairface dataset, and mentions accuracy over 96% for gender classification across all races.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":107-111",
|
|
"content": "### Bias and Fairness\nWe find that the performance of CLIP - and the specific biases it exhibits - can depend significantly on class design and the choices one makes for categories to include and exclude. We tested the risk of certain kinds of denigration with CLIP by classifying images of people from [Fairface](https://arxiv.org/abs/1908.04913) into crime-related and non-human animal categories. We found significant disparities with respect to race and gender. Additionally, we found that these disparities could shift based on how the classes were constructed. (Details captured in the Broader Impacts Section in the paper).\nWe also tested the performance of CLIP on gender, race and age classification using the Fairface dataset (We default to using race categories as they are constructed in the Fairface dataset.) in order to assess quality of performance across different demographics. We found accuracy >96% across all races for gender classification with \u2018Middle Eastern\u2019 having the highest"
|
|
},
|
|
{
|
|
"comment": "This code is providing the accuracy of the model for various classifications and emphasizing that these evaluations are to test performance and identify potential risks, not to endorse such tasks. It also provides a link to a Google Form for questions or comments about the model.",
|
|
"location": "\"/media/root/Toshiba XG3/works/CLIP/docs/src/model-card.md\":111-119",
|
|
"content": " accuracy (98.4%) and \u2018White\u2019 having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification. Our use of evaluations to test for gender, race and age classification as well as denigration harms is simply to evaluate performance of the model across people and surface potential risks and not to demonstrate an endorsement/enthusiasm for such tasks.\n## Feedback\n### Where to send questions or comments about the model\nPlease use [this Google Form](https://forms.gle/Uv7afRH5dvY34ZEs9)"
|
|
}
|
|
]
|
|
} |