I have been using PerplexityAI for the past month instead of Google for many of my searches. In general, I prefer it to Google. First, it doesn’t fill my screen with sponsored links and ads. Second, it doesn’t track my views and sell them to others for spam marketing.
Also, since it seems inevitable that AI is here to stay, I’ve been messing around with Perplexity, Gemini, Copilot, ChatGPT and others to test their capabilities.
AI has gotten better since I first tested it a year ago – but that doesn’t make it great. Today I ran two tests. My results are detailed below.
- Give me a list of authors who won the Shamus Award or the Edgar Award since 2010.
I gave this task to Perplexity. It gave me a list of authors sorted by year. But th list seemed awfully short to me. I expected to see 30 authors (15 for each award for 15 years). Instead there were about 12 names total. One of my new favorite authors who had won the Shamus award was missing from the list entirely. I had to ask 3 different follow up questions before Perplexity returned a list that “appeared” comprehensive.
2. Write a Wikipedia page for Steve Ainslie.
I wanted to see what my Wikipedia page would look like. On the first attempt, it gave me nothing saying there wasn’t public info on Steve Ainslie. The second attempt, it gave me a generic Wikipedia template (with words like <insert accomplishment here>, <insert career info here>,). I then prompted it to use Steve Ainslie the blogger as a source. This time it returned a generic, nonspecific Wikipedia page that said I was an accomplished blogger, internet personality, influencer and writer (all wrong). Then I gave it my blog link and asked it to be more specific. This time it gave a very truncated, high level summary of my “about” page and a few sentences on my most recent posts. It neglected any information over 1 year old. It said I retired in 2012 as a “professional” (Wrong – I retired in 2019 after being a sales manager and VP. I didn’t even start my blog until 2017). It never mentioned sales, my dogs, swimming, being a husband, stepfather or manager despite those being referenced 100s of times in my blog.
Summary
My Wikipedia entry was first generically nonspecific. When asked to be more specific, it provided a few details with some glaringly incorrect and others cherry picked from recent posts.
My book list was incomplete by nearly 50%. It only returned more of the items requested when I made two more inquiries I eluding telling the AI it was missing information.
So AI is generating plausible sounding information that includes wrong information, is incomplete or has a recency bias. If I hadn’t been familiar with the content (a list of my favorite authors and my won blog) I could easily have thought both results were correct and comprehensive.
It makes me wonder how many people are using AI today for business and relying on half-baked answers that may be only partially true.
I suspect many people are doing this.
As AI use becomes more ubiquitous across government, customer service, business and public service I’m sure it will get better. But, just as certain, I’m sure it will also generate a cluster**ck of misinformation and just plain incorrect answers.
Here we go. Get ready. It’s coming.
Postscript written the next day…
It occurred to me this morning that how truthful, factual, accurate or complete AI becomes won’t matter at all. Once people (and business and government) rely upon AI, we will be using it regardless. This seems inevitable.