A newly set up index by Stanford researchers has shown the most transparent large language model (LLM) among the ten evaluated was LLaMA 2, achieving a transparency score of 54 percent.
Notably, GPT-4, which powers ChatGPT’s paid version, and Google’s PaLM 2 both received identical transparency scores, securing the third-highest ranking at 40 percent.
The unveiling of the Foundation Model Transparency Index promises to set new standards and expectations for transparency in the realm of AI, with the potential to influence how these powerful language models are developed and utilized in the future.
Stanford researchers have unveiled a revolutionary scoring system, the Foundation Model Transparency Index, which is poised to bring about a significant shift in the world of artificial intelligence. This innovative index aims to evaluate the transparency of ten prominent AI language models, often referred to as “foundation models.” Among the models scrutinized in the index are heavyweights such as OpenAI’s GPT-4, which powers the premium version of ChatGPT, Google’s PaLM 2, the engine behind Bard, and Meta’s LLaMA 2. The evaluation also encompasses lesser-known models, including Amazon’s Titan Text and Inflection AI’s Inflection-1, the driving force behind the Pi chatbot.
The researchers meticulously examined each of these models, assessing them based on a comprehensive set of 100 criteria. These criteria encompassed vital aspects, such as the disclosure of the sources of training data, details about the hardware utilized in the training process, and the labor involved in their development.
Additionally, the rankings included insights into the labor and data used in creating the models, coupled with what the researchers have coined “downstream indicators,” which pertain to the model’s usage post-release. For instance, one of the criteria probed whether the developer disclosed their protocols for storing, accessing, and sharing user data.
Image credit: The Foundation Model Transparency Index
