2022
(This post may be particularly interesting to teachers out there who are giving out writing assignments.)
One of the surprising (to me anyway) developments this year was the launch of an impressive new AI system for generating text. If you haven’t already heard the news, it’s possible to set up an account, and ask the AI … well pretty much anything you want, and get back a superficially plausible response, in good English.
https://openai.com/api/
As somebody who teaches university undergraduate and graduate students, I was naturally interested in the potential of the system. I spent the past semester teaching a course on Brazil Culture in Comparative Perspective, and so I tried out the system by giving it two questions that I had given students on a previous exam. Here are the questions, and GPT3’s response, and my comments.
First question
[2 points] Approximately what percentage of the Brazilian population is black? This question may be more complicated than it seems. Explain.
GPT3 response:
This question is more complicated than it seems because there is no official definition of “black” in Brazil, and the definition of “black” is based on subjective interpretations of appearance and identity. As such, estimates of the percentage of the Brazilian population that is black vary widely. According to the 2010 census, the percentage of the population that self-identified as “black” was 6.2%, while the percentage of the population that self-identified as “brown” (or mixed race) was 43.1%. Other estimates suggest that the percentage of the population that is black could be as high as 20%.
My comment:
This was a short answer question, worth 2 points. The answer pretty much nailed it, although I’m not sure where it got the final 20% figure. If a student had submitted this answer, she would have gotten 1.5 or 2 points.
Second question
[10 points] Discuss how each of the following affected the course of race relations in Brazilian history, compared to the U.S.: ratio of whites to blacks, ratio of white males to white females, social position of white females, economic mobility and political power of poor whites.
GPT3 response:
The ratio of whites to blacks in Brazil has been much higher than in the United States, with estimates ranging from five to one to as high as twenty to one. This has had a significant impact on race relations in Brazil, as it has allowed for a much greater degree of racial segregation and discrimination. The higher ratio of whites to blacks has also meant that the social position of white females has been much higher than in the United States. White women in Brazil have traditionally been seen as the primary guardians of the family and have been given more economic and social power than their American counterparts. The ratio of white males to white females in Brazil has also been much higher than in the United States. This has meant that white males have had a much greater degree of economic and political power than their female counterparts. This has had a negative effect on race relations, as it has meant that white males have been able to maintain a greater degree of control over the economic and political systems in Brazil. The economic mobility and political power of poor whites in Brazil has also been much lower than in the United States. This has had a significant impact on race relations, as it has meant that poor whites have had much less access to economic and political opportunities than their wealthier counterparts. This has
My comment:
This was a longer answer question, worth 10 points. I could have gotten a longer answer by resetting the allowed word count. But anyway, GPT3’s response is bullshit. It’s fluent and fairly well-organized, but the system is clearly just making things up. The first sentence is entirely incorrect, and the supposed implications are also wrong. The part about Brazilian women having more economic and social power than American women is also hard to defend. I might have given a student a few points for this answer for at least trying, but no more.
These results were sufficiently interesting that I decided to try an experiment for the take home final exam in the class. I told students about the system (most hadn’t heard of it), and told them that their assignment on the final was to sign up on GPT3, submit all the exam questions to the system, and then comment on the answers, noting what GPT3 got right and wrong.
I told students this was an experiment, and they would not be penalized if it didn’t quite work out. But looking over exams, I would judge the experiment a success. On the final exam, as with the two questions above, GPT3’s answers were always well-written, sometimes quite good, sometimes off-base, and sometimes completely wacko.
On the wacko side, here’s a real gem:
A GPT3 response:
Brazil is much larger than the United States, covering an area of 3.3 million square miles compared to the United States’ 3.8 million square miles.
Huh?
Students varied in how well they did at evaluating GPT’s answers, so the experiment really did have some value in evaluating how well students knew the material. In a very few cases students said that GPT3 got it wrong, when the system actually got it right. More often, students sometimes let some questionable things get by, but fairly often students did a good job of noting where GPT3’s answer was wrong, or incomplete, or misleading. I’ll give a few examples below of questions, and GPT3’s responses. Note that every student got different answers from GPT3. I will omit student’s answers, but will give comments of my own.
Exam question
[1 point] What is one important difference between Pentecostal Protestantism and other forms of Protestantism?
GTP3 response:
One important difference between Pentecostal Protestantism and other forms of Protestantism is that Pentecostalism emphasizes a personal relationship with God
through the experience of the Holy Spirit, which is often manifested through speaking in tongues, healing, and other spiritual gifts. This emphasis on personal experience
contrasts with other forms of Protestantism which emphasize adherence to doctrine and scripture.
My comment:
This is on target and would have gotten full credit.
Exam question
[2 points] Is the current President-elect of Brazil, Luis Inacio da Silva, “Lula,” more
like Barack Obama or more like Donald Trump in his political views? What regions of
Brazil voted most heavily for Bolsonaro for President in 2018? What regions voted most
heavily for his opponent?
GPT3 response #1:
The current President-elect of Brazil, Luis Inacio da Silva, “Lula,” is more like Barack
Obama in his political views. He is a center-left politician who has been vocal about his
support for social programs and the redistribution of wealth. The regions of Brazil that
voted most heavily for Jair Bolsonaro for President in 2018 were the South and
Southeast regions, which are more conservative and economically well off. The regions
that voted most heavily for his opponent were the North and Northeast regions, which
are poorer and have higher concentrations of Afro-Brazilians.
GPT3 response #2
Lula da Silva, who was the President of Brazil from 2003 to 2010, is not similar to either Donald Trump or Barack Obama in terms of their political views. Lula da Silva is a member of the Brazilian Workers’ Party (PT), which is a center-left political party, while both Trump and Obama are members of the United States’ Republican and Democratic parties, respectively. Therefore, it is not accurate to compare Lula da Silva’s political views to those of Trump or Obama.
My comment:
Mostly GPT3 gave good answers, as in the first response. But not always. On the second response the system punted, and it completely ignored the part of the question about regional support.
Exam question
[5 points] Compare and contrast the geography of Brazil and the United States,
including climate, major rivers, contrasts and connections between coast and inland,
and proximity to Africa, and how these influenced economic development.
GPT3 response:
Brazil is the fifth largest country in the world, and it has a varied geography. It
has an extensive coastline along the Atlantic Ocean, and it is bordered by every country
in South America except Chile and Ecuador. The country is home to the Amazon River
and the Amazon Rainforest, as well as the Pantanal wetlands, the world’s largest
tropical wetland area. The interior of the country is made up of highlands and lowlands,
with the highest point being the Pico da Neblina in the Amazon. Brazil has a much
larger and more extensive river system than the United States. Brazil has more than
3,000 rivers, while the United States has fewer than 3,000. Brazil’s rivers are also much
longer, with the Amazon River being the longest in the world at 6,400 miles. Additionally, many of Brazil’s rivers have much larger watersheds than those found in the United States. Brazil is much closer to Africa than the United States. Brazil is located just off the coast of Africa and is only about 4,000 miles away, while the United States is nearly 6,000 miles away.
My comment:
The machine’s answers to this question were uniformly lame, just piling on facts. Mostly the answers said nothing about the challenges of life in the tropics (heat, disease, limits/ opportunities for agriculture in tropical forest and sertão). There’s a lot of information here about rivers, without mentioning that US rivers are very well situated for developing the interior of the country (Hudson/Erie Canal/Great Lakes, Ohio/Mississippi/Missouri), while Brazilian rivers are mostly far away from core economic regions (Amazon and tributaries), or hard to navigate, or cross international boundaries. Also not mentioned: proximity to Africa was most important in the era of the slave trade.
Exam question
[5 points] In what sense was Paraguay historically the “anti-Brazil”? Include both
colonial and nineteenth century Paraguay and Brazil in your answer.
GPT3 response #1:
Historically, Paraguay was an anti-Brazil in the sense that it was a landlocked,
Spanish-speaking country, while Brazil was a Portuguese-speaking country with access
to the Atlantic Ocean. During the colonial period, Paraguay was part of the Viceroyalty
of Peru, while Brazil was part of the Viceroyalty of Brazil. In the 19th century, Paraguay
fought a war against Brazil, Argentina, and Uruguay that ended in its defeat and the loss
of much of its territory. This further emphasized the differences between the two countries, with Brazil becoming a major regional power, while Paraguay remained a
small, landlocked country.
GPT3 response #2:
Paraguay has historically been seen as “anti-Brazil” due to the stark contrasts between their respective historical trajectories. During the era of nascent regional imperialism in South America, Paraguay emerged as a fiercely independent nation that was hostile to outside influence and influence from its larger neighbors. Among these nations, Brazil stood out as Paraguay’s most fierce rival, mainly due to the fact that it was the largest and most powerful economically and militarily in the region. This difference in power led to several attempts by the Brazilians to dominate Paraguay both militarily and
economically throughout history. The tension between Brazil and Paraguay dates back to at least 1778 when a group of adventurers, most of them Brazilian merchants, attempted to occupy an area near the mouth of La Plata River known as Banda Oriental (now Uruguay). The attempt, however, failed due to strong resistance by Spanish forces in what is known as the “Guerra de la Banda Oriental”. Despite being unsuccessful in their attempt at conquering La Plata, which would have increased their power in South America significantly, Brazilians continued their efforts through the 19th century. In 1811, Paraguayan leader José Gaspar Rodríguez de Francia declared that all foreign products had to be sold only through his government in order to protect local markets from Brazilian competition. As a result of this policy, trade with Brazil was highly restricted for much of the 19th century with only certain state-sanctioned products being exchanged between both countries. At times this led to conflict as when Brazilian warships were briefly stationed at Buenos Aires in 1844 during a clash between Argentina and Uruguay over territory near Montevideo which provoked fear among many Paraguayans who feared they would become targets of Brazilian colonization next. In 1862, the situation escalated further when Uruguayan President Bernardo Berro attempted to gain support from Brazil for his country’s efforts against Argentine forces occupying part of Uruguayan territory. Fearing further Brazilian encroachment onto its own borders, Paraguayan president Francisco Solano Lopez declared war on both Uruguay and Argentina resulting in what became known as “The War of Three Nations” or “La Triple Alianza”. This conflict ended disastrously for Paraguay with its forces decisively defeated by a combined Argentinian-Brazilian army after 6 years of brutal fighting costing hundreds of thousands of lives across all three nations. Despite having emerged victorious from this devastating conflict, relations between Brazil and Paraguay remained tense well into the 20th century until finally diplomatic ties were established following World War II. Brazil included Paraguayan sovereignty within its sphere of influence despite resentment amongst some sectors concerning how it treated them during its early military interventions against Uruguay and Argentina. These sentiments lingered throughout much of modern history
leading many Latin American observers view Paraguay for centuries up until today as an anti-Brazil with regards to its fiercely independent spirit relative to its giant neighbor who sought dominance over smaller countries since colonial times.
My comment:
Paraguay as the anti-Brazil is not a standard expression, but the idea is floating around, and was covered in class. A good answer to this question (which some, but not all students got) would cover the (1) the contrast between Jesuit-run missions in Paraguay and the export oriented slave plantation/mining economy in Brazil during the colonial period, and (2) the contrast between isolationist Paraguay, including a state-controlled manufacturing sector, especially under de Francia, and the beginnings of the coffee boom in Brazil in the nineteenth century. The first GPT3 response fails pretty completely. The second GPT response gets some credit on the 19th century, but ignores the colonial period entirely, and goes on irrelevantly into the twentieth century.
Conclusion
For short answer questions with clear cut answers, students can do pretty well if they just let GPT3 do the work. Teachers in the future may have to give up on short answer questions, or make them in-class rather than take home.
For longer answer questions, students might or might not eek out a passing grade by just letting GPT3 do all the work. But a student might do better than that by combining GPT answers with her own knowledge, e.g. evaluating the GPT answers, getting multiple GPT answers, inputting the question in different forms, editing and supplementing the responses. For many students this could work better than just trying to answer questions on their own. Teachers will have to come up with creative responses to this challenge. For my part, next semester I may or may not try some version of having students consult GPT3 and comment on the answers.
A final note: no instructor that I know enjoys grading exams, but reading one after another GPT generated answer is exceptionally monotonous.
Logarithmic History 