Democratizing AI with open-source language models

When OpenAI made its chatbot ChatGPT available to the public in November 2022, it immediately became a hit. However, despite the company's name, the underlying algorithm isn't open. Furthermore, ChatGPT users require a connection to OpenAI's cloud service and face usage restrictions. In the meantime, several open-source or freely available alternatives have emerged, with some even able to run on consumer hardware. Although they can't match ChatGPT's performance yet, rapid advancements are occurring in this field, to the extent that some people at the companies developing these artificial intelligence (AI) models have begun to worry.

ChatGPT is presented as a bot that users interact with, which generates human-like text based on their input. OpenAI has fine-tuned it specifically for engaging in conversations and providing contextually appropriate responses. It's capable of handling a variety of tasks, such as generating content or ideas, translating languages, answering questions, and even providing suggestions for code in various programming languages. ChatGPT also responds to follow-up questions, challenges incorrect premises, and rejects inappropriate requests. How does this work? Under the hood, ChatGPT uses a neural network trained on vast amounts of text to generate new content based on the input it receives. Think of it as an advanced form of autocomplete suggestions.

A neural network is a learning algorithm inspired by how our brains function. It consists of a large number of nodes, known as neurons, that receive input from other neurons and perform a mathematical function to calculate their output, which then goes to other neurons. Each input has a weight attached to it that determines how much that value contributes to the result. The neural network's architecture (how the layers of neurons are connected) and its weights determine its functionality.

The network starts with random weights; therefore, when it receives text as input (we're glossing over some details, such as how text is encoded in numbers), its output is also random. As a result, the network has to be trained using training data: an input text with a corresponding output text. Each time the training data input enters the network, its output is compared with the training data's corresponding output. The weights are then adjusted to decrease the difference between the predicted and correct output. In this way, the network undergoes a learning process until it becomes proficient at predicting text.

Such a large neural network capable of generating human-like text is called a large language model (LLM). It typically involves billions to several hundred billion weights, also known as parameters. ChatGPT is based on some large language models developed by OpenAI, with names like GPT-3.5 or (the most recent version) GPT-4. GPT stands for Generative Pre-trained Transformer and is a specific type of large language model, introduced by OpenAI in 2018, and based on the Transformer architecture invented by Google in 2017. Since GPT-3.5, OpenAI hasn't disclosed the size of its models; GPT-3 (released in May 2020) had 175 billion parameters and was trained on 570GB of text.

When large language models are trained on a broad range of data, as is the case with GPT-3.5 and GPT-4, they are also known as foundational models. Their broad training makes them adaptable to various tasks. A foundational model can be fine-tuned by training the model (or part of it) with new data for a specific task or a specific subject-matter domain. This is what OpenAI has done with ChatGPT: it has fine-tuned its foundational GPT models with conversations in which humans played both the user and the AI role. The result is a model specifically fine-tuned to follow a user's instructions and provide human-like responses.

BLOOM

Companies developing large language models lack incentives to open-source their models and the code to run them since training the models requires significant computing power and financial investment. To make the development of these models sustainable, companies need to be able to build a profitable business around it. OpenAI aims to do this by offering the paid ChatGPT Plus subscription and its pay-per-use API access.

Last year, the situation changed with BLOOM (BigScience Large Open-science Open-access Multilingual Language Model), which is freely available. This large language model was the result of a global collaboration involving over a thousand scientists from more than 250 institutions participating as volunteers in the BigScience collective. The project was started by Hugging Face, a company that provides a machine-learning platform, with contributions from NVIDIA, Microsoft, and the French research institution CNRS.

Development of BLOOM occurred entirely in public. The model was trained for 3.5 months on the Jean Zay supercomputer in Paris, utilizing 384 NVIDIA A100 Tensor Core GPUs each with 80GB of RAM. The data set comprised 1.6TB of text (341 billion words) in 46 human languages and 13 programming languages, and the model had 176 billion parameters, which is comparable to GPT-3.

BigScience developed a new license to publish BLOOM: the Responsible AI Licence (RAIL). Its main purpose is to minimize risks arising from irresponsible use of the model. According to the license, users are not allowed to use the model to generate false information with the intent to harm others; nor may they neglect to disclose that the generated text was machine-generated. RAIL is not an open-source license and does not appear on the list of OSI-approved licenses. Developers can use the BLOOM model with Hugging Face's Apache-2-licensed transformers library in their own code, subject to the terms of RAIL.

Smaller large language models

A downside of BLOOM is that it's still too large for convenient local use. In principle, anyone could download the 330GB model, but using it would require substantial hardware. BigScience also released smaller versions of the model, and this trend of smaller models was continued by others. In February, Meta announced a language model called LLaMA, which is available in versions with sizes of seven billion, 13 billion, 33 billion, and 65 billion parameters. According to the developers, the 13B version performs as well as OpenAI's GPT-3, while being a factor of ten smaller. And unlike GPT-3, which requires multiple A100 GPUs to operate, LLaMA-13B needs only one GPU to achieve the same performance.

Meta trained LLaMA on publicly available data sets, such as Wikipedia and Common Crawl. The code to run LLaMA is GPLv3-licensed, but to obtain the full weights of the model, users were required to fill out a form and agree to a "non-commercial bespoke license". Moreover, Meta proved to be quite selective in granting access. But within a week, the weights were leaked on BitTorrent, and LLaMA kickstarted the development of a lot of derivatives. Stanford University introduced Alpaca 7B, based on the LLaMA model with seven billion parameters and supplemented with instructions based on OpenAI's text-davinci-003 model of the GPT-3.5 family. Both the data set and the model were released under the CC BY-NC 4.0 license and thus do not permit commercial use. One reason for this is that OpenAI's terms of use disallow the development of models that compete with OpenAI.

Subsequently, the open research organization Large Model Systems Organization published Vicuna, a LLaMA-based model fine-tuned on 70,000 conversations of users with ChatGPT. This was accomplished using ShareGPT, which is a browser extension for Google Chrome that is designed to easily share ChatGPT conversations. According to the researchers, Vicuna achieves 90% of ChatGPT's quality and outperforms LLaMA and Alpaca in 90% of cases. Both the code (Apache 2 license) and the weights (13 billion parameters subject to LLaMA's license) have been made public. However, since Vicuna is based on LLaMA and on output from ChatGPT, commercial use is not allowed.

This issue prompted US software company Databricks to develop an open-source large language model suitable for commercial use: Dolly 2.0. It is based on EleutherAI's Pythia model with 12 billion parameters and was trained on the Pile text data set, then fine-tuned on 15,000 instructions with answers. To achieve this, the company engaged more than 5,000 of its employees. Dolly is trained on open-ended questions, closed questions, extracting factual information from texts, summarizing texts, brainstorming, classification, and creative writing tasks, all of this in English only. The 23.8GB dolly-v2-12b model can be downloaded from Databricks' page on Hugging Face. The model is using the MIT license, while the databricks-dolly-15k data set is published under the CC BY-SA 3.0 license.

Following this, Stability AI, the creators of the Stable Diffusion open-source model for generating images, published its own family of large language models: StableLM, under a CC BY-SA 4.0 license. Additionally, MosaicML introduced its MPT-7B family of open-source commercially usable large language models (some of them Apache 2 licensed). Another interesting development is BigCode, a project kickstarted by ServiceNow Research and Hugging Face to develop large language models for completing and writing code from other code and natural language descriptions. Their first model, StarCoder, has been trained on permissively licensed data from GitHub and is using the OpenRAIL license, an updated version of the Responsible AI License.

Crowdsourcing open data sets

With new open-source (or freely available) language models emerging regularly (many of which can be found in the awesome-totally-open-chatgpt repository), various organizations have started considering ways to streamline the development of data sets for training models. One such organization is LAION (Large-scale Artificial Intelligence Open Network), a non-profit research organization aiming to democratize AI. With Open Assistant, it plans to develop large language models capable of running on consumer hardware.

Open Assistant is still under development, and currently focuses mainly on collecting data sets with the help of users. The project already boasts a data set of 600,000 interactions, contributed by 13,000 volunteers. Everyone can lend a hand in this endeavor, as explained in the documentation. For example, users are tasked with grading an answer provided by another person, based on parameters such as quality or politeness. Another task involves offering an answer in the role of a chatbot to a user's request. Volunteers may also be asked to select the best response from two possible answers. Tasks are available not only in English, but in many other languages as well. The researchers intend to train language models using the data set generated by these volunteer tasks. It's worth noting that OpenAI has a similar approach with ChatGPT: the company pays (low-wage) contractors to assist in training its language model and to help identify toxic content.

Running language models on consumer hardware

Running large language models with tens to hundreds of billions of parameters on consumer hardware is not feasible. However, with the trend of smaller language models, initiated by LLaMA, operating ChatGPT-like software on a PC becomes possible. An important project for running these models is Georgi Gerganov's MIT-licensed llama.cpp. It enables users to run LLaMa, Alpaca, and other LLaMA-based models locally on their computers. It runs entirely on the CPU, which is possible by applying 4-bit quantization of the models. This reduces the accuracy of the weights to four bits to reduce memory consumption and computational complexity. Llama.cpp supports Linux, macOS, and Windows. Users can chat with the model via a command-line interface.

The model has to be downloaded first, for example from Hugging Face's model hub, and then converted into a specific format. Instructions to do this are provided in llama.cpp's README file. As a test, I downloaded the 8.14GB model ggml-vic13b-q4_0.bin, which was already in the correct format for llama.cpp. The program loads the model fully in RAM, so it's crucial to ensure the computer has sufficient free RAM.

After compiling llama.cpp, running the model in interactive mode is as simple as:

    $ ./main -m models/ggml-vic13b-q4_0.bin -i

On my somewhat dated laptop with four-core Intel Core i7-10510U 1.8GHz CPU and 24GB RAM, this took 20 seconds to start up. After posing the question "What is the most common way of transportation in Amsterdam?", the Vicuna model began to generate its response word by word, taking 15 minutes to complete the task. The resulting answer was vague but not incorrect, resembling a text from a travel guide:

Amsterdam, the capital city of the Netherlands, is known for its picturesque canals and bustling bike culture. It's no surprise that the most common way to get around Amsterdam is by bicycle. However, there are also plenty of other transportation options available in the city, including buses, trams, trains, and even boats. Whether you prefer two wheels or four, there's a mode of transportation in Amsterdam that will suit your needs.

Various projects have emerged that aim to make using open-source language models more akin to a ChatGPT-like experience. For example, Nathan Sarrazin has been working on Serge, a web-based interface for llama.cpp. It runs in a Docker container, allowing users to download a model from a list, choose its settings, and then initiate a chat. Another similar undertaking is oobabooga's text-generation-webui, which supports both CPUs and GPUs for running models. Using GPUs can significantly improve the performance.

Is open-source AI gaining an edge?

In early May, a document penned by Google's Luke Sernau was leaked: "We have no moat, and neither does OpenAI". The author contends that the open-source community has been rapidly catching up with commercial efforts: "They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months." Sernau also clearly recognizes the advantages of the open-source development model. After detailing the numerous innovations that have occurred within a month of LLaMA's weights being leaked, he notes that anyone can tinker: "Many of the new ideas are from ordinary people." The barrier to entry to contribute to these open-source large language models is just "one person, an evening, and a beefy laptop".

Sernau continues with lessons for Google (and OpenAI), focusing on LoRA, a technique that accelerates fine-tuning of models and is already used by a lot of open-source projects in this domain. Thanks to LoRA, almost anyone with an idea can generate a fine-tuned model in under a day and for around $100. "At that pace, it doesn't take long before the cumulative effect of all of these fine-tunings overcomes starting off at a size disadvantage", he said, adding that focusing on maintaining some of the largest models on the planet actually puts Google at a disadvantage. At the end, he even made the case for opening Google's large language models. He perceives Meta as the clear winner in all of this, because most open-source innovation is happening on top of its LLaMA model architecture.

If Sernau is right, it means that language models could become a commodity, fueled by the innovative nature and fast-paced development model of the open-source AI community. This would enable researchers, non-profit organizations, and small businesses to access AI capabilities without depending on cloud-based services or expensive subscription fees. Looking back at the numerous language models that have been published during the last few months, we can wonder how long it will take before we have some usable, completely open-source, AI assistants to help us with our daily tasks.