I built my girlfriend a GPT-2 Twitter bot… it didn’t go as planned
My girlfriend is a big Twitter user and spends a lot of time on the app. Yesterday, she whimsically inquired whether I could build a bot for her, mentioning that other users she follows had personal bots.
Before she could really elaborate on the idea, I immediately thought of fine-tuning a GPT model on her historical tweets. Months go, I had requested academic access to GPT-3. Unfortunately, despite multiple follow-up requests to OpenAI and Greg Brockman for access to GPT-3 I did not receive a reply. I am an MS Candidate in Artificial Intelligence that wanted to study the model and analyze it for racist tendencies. I had hoped to do some joint research between Boston University’s Computer Science and Department and our newly formed Center for Antiracist Research headed by Ibram X. Kendi. I think they likely denied my request to the model for GPT-3 because they were worried that I would write something negative about the model which has now been exclusively licensed to Microsoft.
I fell back to GPT-2 which has a publicly available 124M parameter and 345M parameter checkpoint file available.
For an excellent explainer of the GPT-2 architecture please see this article…
http://jalammar.github.io/illustrated-gpt2/
After failing spectacularily at retrieving the historical tweets of my girlfriend’s account using twint, and snscrape, I finally was able to extract 3,000ish tweets using snscrape. This was disappointing because she has just over 12,000 tweets on her account. I tried to debug this for a long time but for whatever reason I could not understand why it worked to get all the tweets for some users but just wouldn’t get more than 3,000 for her account. Apparently within the past year there have been some pretty breaking changes to the Twitter API with the release of V2.
I saved them as a single column csv file using pandas beforehand to remove Twitter handles and urls.
Then I followed this excellent tutorial by Max Woolf to fine-tune GPT-2 for 500 steps on my dataset and generate 15,000 tweets.
It is important to use the small (124M parameters) or medium (345M parameters) when training on Twitter data because the text samples are short. In addition, it is important to not overfit the model on such a small dataset. If you train for too long on a small dataset (3,000 tweets) you run the risk of the model just verbatim repeating tweets that appear in the training dataset.
One thing to note is that I included both retweets and original tweets from her account in my model training. I could have opted for original tweets only but this would have further reduced the size of my dataset.
Here are some unbelievably fun results that the model spat out:
this is crazy i can see the bots doing the talking 😌
i’m a bit new to life but im hungry so
[whisper, zoom in]
Some profound:
Life is a living experiment in how very personal information can be harvested and shared indefinitely
i think the way we kill people speaks for itself
what kind of society is this?
suicide should not be taken lightly
manning the computer right now is the most powerful and most horrifying thing you can do
In addition to some pretty horrific (but relatively rare) results:
women just can’t be damn quiet
a thousand years of slavery
Emmett Till: serial rapist
twitter is awash in pedophiles
So what can we conjecture from this article. You may be thinking that my significant other is a terrible person! Well, no. My girlfriend is a lovely, hilarious, liberally progressive woman and would never use the kind of language that existed in the generated tweets. Twitter is a disaster of a dataset because it contains a lot of controversial content, and as amazing as artificial intelligence is, it fails to pick up on the contextual satire or understand the subtleties of Twitter humor.
The generated text referring to slavery doesn’t necessarily advocate for it’s return and could make logical sense when inserted into a larger sequence, especially given the surge in posts expressing support for the Black Lives Matter movement. The tweets that caused this output by the model might have been lamenting and not promoting slavery throughout history.
The generated text referring to women could originate from a tweet that uses that language in a positive context (celebrating subversive feminist figures is one example).
With a black-box model like this, we lack explanation for it’s output. It’s difficult to reason about but we can use our intuition to understand it’s output.
The output of the model that I received could also point to inherent bias in the pre-trained checkpoint provided by OpenAI for GPT-2 small 124M. You can see a project that analyzes stereotypical bias in pre-trained text generation models here:
If you want to see a demonstration of how GPT-2 can be made racist and otherwise exploited through universal adversarial triggers, please see the project below.
In the future, ethical AI and explainable AI will become more important than ever. You can read about the shortcomings of the inaccessible GPT-3 model here:
My main focus in deep learning has been computer vision so I had an interesting time playing around with these language models. What is incredible about this model published over a year ago is that it is capable of generating some pretty coherent sentences at times. It is also remarkably effective at emulating the tweet style of my girlfriend!
Language models and AI ethics are in the current news cycle with the firing of Timnit Gebru who was co-lead on AI ethics at Google.
I encourage you to read the article published to the Google Walkout Medium account.
Stay tuned for Part 2 where I will show you how to use tweepy and crontab to enable your bot to post to Twitter every X number of mintues or hours.
