Artificial intelligence (AI) is becoming more and more common in science, as researchers use it to analyze data, generate hypotheses, and perform experiments. But how does AI learn so much from so little, and what are the implications for scientific discovery?
A recent study by researchers at MIT, Stanford, and Google sheds some light on this question, by exploring a phenomenon called “in-context learning”. This is the ability of AI systems, especially large language models (LLMs), to learn new tasks from only a few examples, without any explicit training or feedback.
LLMs are AI systems that can produce natural language outputs, such as text, speech, or code, based on the patterns they learn from massive amounts of data. Examples of LLMs include GPT-3, BERT, and LaMDA, which have been used for various applications, such as writing, summarizing, translating, and conversing.
The researchers found that LLMs can also perform tasks that they have not been specifically trained for, such as arithmetic, logic, or programming, by simply providing them with a few input-output pairs as examples. For instance, given the examples “2+2=4” and “3+5=8”, an LLM can infer the rule of addition and produce the correct answer for “4+6=?”.
The researchers also found that LLMs can generalize and extrapolate from the examples, and handle variations and complexities that are not present in the examples. For example, given the examples “A and B -> AB” and “C and D -> CD”, an LLM can infer the rule of concatenation and produce the correct answer for “E and F -> EF”, even though it has never seen the letters E and F before.
The researchers explained that LLMs can learn from examples because they have a large and diverse knowledge base, which they acquire from their training data. This knowledge base contains information about various domains, concepts, and rules, which the LLMs can access and manipulate when faced with a new task. The researchers also suggested that LLMs can learn from examples because they have a powerful and flexible generative mechanism, which allows them to produce outputs that are consistent and coherent with the examples and the task.
The researchers argued that in-context learning is a remarkable and useful feature of LLMs, as it enables them to adapt to new situations and challenges, and to perform tasks that are beyond their original scope and purpose. The researchers also claimed that in-context learning is relevant and important for science, as it mimics the way that humans learn and discover new knowledge, by using examples, analogies, and induction.
The researchers concluded that in-context learning is a promising and exciting direction for AI research, and that it could lead to new breakthroughs and innovations in science and technology. However, the researchers also cautioned that in-context learning is not perfect or reliable, and that it has some limitations and risks, such as producing incorrect or biased outputs, or being vulnerable to manipulation or deception. The researchers recommended that in-context learning should be used with care and caution, and that it should be complemented and verified by other methods and sources of evidence.
