Ego test for Claude 4

5/22/2025 12:45 pm | Share to:

The following post makes mention of my work, so I will start this post off with a reminding disclaimer: I work for Hasbro/Wizards of the Coast, the makers of Magic: The Gathering.

So this is interesting. Claude just put out their new model and as I tend to do, I asked it about me. It noted it didn't have info about me in its information, so it would search online. It then proceeded to be completely factually correct (not true of past instances with Claude and other LLMs.)

However, that's not the interesting part.

At the end of what it wrote, it said:

Game Involvement: Jarrett was on the design team for Doctor Who (referring to the Magic: The Gathering Doctor Who cards), and he's an ardent Commander player and lover of the game he's played since Ice Age.

It cited two links for this section, one of which when I clicked it returned a 404 error for me. Which makes me ask, if it got this information from an online search, how is it trying to cite a page which isn't available? Conceivably the fact could have come from that page, but it isn't immediately obvious from the article title, etc.

I submitted feedback for the response, I am somewhat dubious about how much attention those replies get, but who knows.

My feedback to Anthropic was:

It said it searched online to get this information, but the last link it cites is an article which returns a 404. Either, it references data from an internal archive of the internet which was made prior to that article going offline, or it isn't actually searching online.