THE COST OF GPT-3
GPT-3 is a massive deep-learning model. Deep
learning is a type of AI system that develops its
behavior through experience. Every deep-
learning model is composed of many layers of
parameters that start at random values and
gradually tune themselves as the model is trained
on examples.
Before deep learning, programmers and domain
experts had to manually write the commands that
GH¿QHGWKHORJLFDQGUXOHVWRSDUVHDQGPDNH
sense of text. With deep learning, you provide a
model with a large corpus of text—say, Wikipedia
articles—and it adjusts its parameters to capture
WKHUHODWLRQVEHWZHHQWKHGL̆HUHQWZRUGV<RX
can then use the model for a variety of language
tasks such as answering questions, automatic
email-reply suggestions, and advanced search.
Research and development in the past few years
has shown that in general, the performance of
deep-learning models improves as they are given
larger numbers of parameters and trained on
bigger data sets.
In this respect, GPT-3 has broken all records: It is
composed of 175 billion parameters, which makes
it more than a hundred times larger than its
predecessor, GPT-2. And the data set used to
train the AI is at least 10 times larger than GPT-
2’s 40-gigabyte training corpus. Although there’s
much debate about whether larger neural
networks will solve the fundamental problem of
understanding the context of language, GPT-3
has outperformed all of its predecessors in
language-related tasks.
@bendee983
B
e
n
D
i
c
k
s
o
n
