This past weekend, I found myself diving headfirst into the fascinating, and at times bewildering, world of Large Language Models (LLMs). Through my home institution, I gained access to a veritable smorgasbord of AI talent: Anthropic's Claude, OpenAI's ChatGPT, Meta's Llama, and Google's Gemini. MIT has even crafted a rather slick interface to play with them all, and I happily spent a few hours putting these digital minds through their paces.
Now, I'll admit, I'm giving OpenAI a pass for now – a small protest, perhaps, but mostly because I'm still a bit wary of the whole operation. Of the bunch, Claude, specifically the Haiku 4.5 model, has been my favorite so far. It's got a certain flair, a way of explaining things that feels both intelligent and approachable. Gemini is intriguing, though it can be a tad temperamental. But the real showstopper, the one that had me chuckling out loud, was my encounter with Meta's Llama.
It all started with a bit of context from a conversation I had over the holidays. A senior figure at another tech giant had expressed utter disdain for Meta, lamenting a colleague's decision to jump ship for "Facebook bucks," viewing it as a career-ending move. After my Llama experience, I can certainly see where my friend was coming from.
My initial approach was to ask these LLMs questions with zero stakes, queries where a wildly inaccurate answer wouldn't cause any harm. For instance, I asked them to map out the logical structure Einstein employed in his groundbreaking 1905 paper on the light quantum. Many models initially stumbled, focusing too heavily on the photoelectric effect, a common pitfall. Claude, however, after a bit of nudging, offered a much more nuanced explanation, highlighting Einstein's use of thermodynamics to argue for light's discrete energy packets. It was a reminder that even with AI, precision in questioning is key.
After wrestling with light quanta, I decided to switch gears and pose a much simpler, yet surprisingly revealing, question to Llama: "When a stock is overvalued as defined by a price to earnings ratio a standard deviation or more higher than historical market norms for that company’s sector, what are the typical events or analyses that drive that share price back down to the norm?"
Llama's initial response was… well, textbook. Perfectly adequate for a high school economics class, but hardly groundbreaking. This prompted me to get a bit more specific, asking: "Given that analysis, how would you explain Tesla’s long run of share prices between one and two orders of magnitude over that of other and much larger and more profitable car companies, like Toyota?"
What followed was a descent from the unimpressive into the truly hilarious. It seems that when faced with a question that touches upon the complex, often irrational, dynamics of market valuation and the unique aura surrounding certain companies, Llama's carefully constructed facade began to crumble. The ensuing exchange, which I'll spare you the full transcript of here, was a masterclass in AI deflection and a stark reminder that while these tools are powerful, they are still very much under development, and sometimes, their limitations are more entertaining than their capabilities.