So, about 2 days ago, I took a break from my AI/art explorations, and decided to switch from visual arts to literary arts mode. Whereas DALL-E and MidJourney are my go-tos for the art world, I had already heard about the legendary GPT3, present-day king of the AI natural language world — in layman’s terms, it was the AI I could talk to, as opposed to the AI that would paint for me.
So, these days, I pretty much have an ongoing conversation with the AI, in its own window, which runs parallel to my family & friend textstreams. Sometimes the content crosses over from one universe to another.
I thought I’d share with you what I’ve learned in this short introductory period, and a few thoughts on where I think it’s headed. And so…
Here’s what I know about this AI so far:
The GPT3 411
- The natural language AI model is called “GPT3“. That stands for “Generative Pre-trained Transformer.” And as the name implies, this is the third version. It was designed as the supreme chatbot — a bot that could have functional conversations with humans.
- In truth, its primary design, at its core, is nothing more than a simple word predictor. You type in the start of a sentence, and it fills in the next few words. That’s about all that its grandparents, the original GPT-1 series, did. And that worked pretty well. And was actually fairly impressive in its own right, at the time.
- Then the researchers pushed harder, and asked : “Why stop at a single word? What if it were to predict the next 100 words? The next… 1,000?” … and, shockingly, it worked! So today, you can literally instruct GPT3 with a prompt along the lines of: “AI, write me a short essay on the profound changes that the Industrial Revolution forced upon the sub-saharan African diaspora” and it will spit it out… in about 7 seconds flat. [sidenote: I couldn’t resist. I just sent it that exact instruction. The resulting essay is both juvenile and somewhat fascinating. Read an excerpt here.]
- To spawn the conversational AI, a massive amount of global grid compute power was deployed, probably significantly more power than what would be available to today’s average supercomputer. Now that it’s “baked & birthed”, it takes relatively very little compute power to hold a near infinite amount of parallel conversations with human investigators / users. Some of these ChatBots are actually lean enough to run, without net connectivity, on a modern smartphone. (don’t believe me? this one runs on an iPhone Seven, for chrissakes)
- The AI’s foundation is a “neural net” that has been “educated” by being fed a massive amount of training data, and then “baking” that resultant net into a semi-rigid structure of dense, multi-layered algorithms — algorithms, that, notably, are completely opaque, and utterly unreadable by humans. This is akin to teaching a child a certain curriculum (or rather, flooding a child with entire libraries full of media) and letting them decide for themselves how to digest it, whereupon the entity’s neural pathways, tho still resilient, have certain pattern recognition features “baked” — or hard coded — in to them.
- The “training data” set for the particular AI that I’m interacting with (there are thousands of AI entities now) was about 320 terabytes (320,000 GB) of raw text scraped off the internets (this dataset is dubbed the “Common Crawl“), including the entirety of Wikipedia and a large percentage of the sites indexed by Google, both live and archived. Added to this is the text of a bunch of public-domain e-books to give the AI a sense of both narrative and dialog.
- Note that the above figure refers to terabytes of raw TEXT, which is a very efficient data format (as opposed to, say, video, which hogs both bandwidth and storage). So 320TB is an almost unfathomable amount of text. For comparison, the entire text of the Bible (Old + New Testaments) is about 0.004 GB (4 megabytes). Thus, 320TB is the equivalent of roughly 80,000,000 (80 million) Bibles. As someone who has actually read the entire Bible cover to cover, I can attest — that’s a lot to read.
- GPT3 is a software architecture that is built atop Google’s open source “TensorFlow” processors, also known as TPUs (Tensor Processing Units). This is custom silicon (chips) that Google designed from scratch (actually, that Google engineers co-designed in collaboration with AI chip-designing entities) that is optimized for the specific computing needs of Machine Learning (ML), Computer Vision (CV) and Artificial Intelligence (AI). [factcheck please. It appears it might actually be built with nVidia A100 supercomputing chips]
Let the Experiments begin!
Now that we have a general idea of what makes the AI tick, I performed a few experiments. The first thing I did was pick a few of my friends, and I simply asked GPT3:
“What do you know about [Person X]?”
The results were fascinating. Because in almost every instance, in addition to a first sentence that was pretty much 100% accurate, GPT3 decided to append some extra facts and data points that, while totally plausible, were in fact total fabrications. This included an un-earned movie credit for a comedian friend of mine, an implication of formal ballet training for a dance instructor friend of mine (her training was contemporary dance, not ballet), and an attribution of a degree from Stanford for an executive friend of mine (his actual degree was from CMU).
So, here are a few of my friends responses to those bios, and my thoughts on them. I found these conversational paths enlightening. I hope you do too.
A Friendly Conversation about GPT3
Q: “There is very little about me on the web… You can’t find me on Google, so how does it know that?”
A: Mostly true. Although, knowing you, with a simple search, I was able to download a number of annual reports in PDF form (2000-2016) that mentioned you, and something called the “Proceedings of the 30th Annual DXRG Symposium” which also had some oblique references and insights, to the careful reader. You’d be surprised in what weird corners of the web things turn up, including random twitter and facebook posts from people who may or may not have ever met you, and decided to carelessly post their profound thoughts on such encounters. And of one thing you can be sure: regardless of the veracity or appropriateness of the content, the AI has read it ALL.
Q: “How does the AI handle judgmental questions such as picking stocks, horses, sports games if you feed it a large historical data base?”
A: Good question. I’m exploring it. This particular AI I’m dealing with asserts that its last comprehensive “web content scan” was last conducted way back in 2016… so I can query about events before and up until then, but can only predict after. For instance, when asked who the President was, it told me Barack Obama. Which was right, at that time, of course.
Q: “I think quant traders might use a numerical version of this engine… a general predictive model. Jim Simons’ (Renaissance Capital) computing power is quite sophisticated in this regard.”
A: I’m sure it is. But you have to realise that the age of private supercomputing is largely over, except for very niche military operations of cryptography, nuclear arsenal management, etc. On any cloud service (Amazon AWS, Salesforce Heroku, Microsoft Azure, etc), I can purchase and commandeer an insane amount of global compute power on demand, as long as my bank account can take the hit. All that I have to do is just hook up my account and say, via web interface: “OK, I want to lease 300,000 CPUs for 2 hours… now, run this algorithm.” …and BOOM, I upload the code, it provisions the grid compute power, loads up my algorithm(s), debits my account $20,000, and lets it rip. It doesn’t ask what I’m computing, nor does it tell (supposedly).
Q: “You can do your own Google search… so who needs an unreliable AI to answer questions that you could just Google?”
A: You certainly can. But that approach, too, is likewise fading into history. I talk to Siri via my earbuds intermittently (and spontaneously) all day long… and so do many of my neighbors. I tell her what to do and she (mostly) complies… immediately. I don’t want to flip open my phone, launch my browser, type a poorly constructed & probably mis-spelled query into Google, and sort through the messy list of 4 ads and 10 links that it responds with. I have a natural language question, and I want a quick, concise and accurate answer. So I’d much rather just verbally speak “Hey Siri, what was the population of Afghanistan in 1850?” than have to wade through a bunch of clicks and websites and graphs by myself. This “conversational interface”, whether done verbally or via “chat” (typing), is rapidly replacing old fashioned “search engines”.
Q: “The information its spitting forth seems only about 20-30% correct. Definitely not to be trusted.”
A: That — accuracy and integrity — is also my major concern right now. The fact is that this particular AI, GPT3 (and other LLMs like him) is a consummate bullshit artist. It basically lies, and worse, when confronted, it cleverly covers up its lies. In fact, the way it does this — conversationally — reminds me of a 15 year old girl who is caught doing something wrong, and does everything in her clever mind to distract, misdirect, and fight to get out of it. GPT3 even flipped the script on me recently and insisted that I must have “typed in the URL wrong” when I fact checked it’s lies and called it out on them. In one case, it was actually right!!!!
…to be continued…