TIL cool math. Also we need knowledge engines that we can trust (that is: not chatgpt)

06 Oct, 2024

TIL = today I learned.

This morning I watched a video by Numberphile and Matt Parker about the prime constant.

What jumped out to me: representing fractions in different bases.

I'm not a mathematician so what was more interesting than the main topic (although also more basic) was the idea of representing fractions in different bases.

I surmised that 1/3 or 0.3333... repeating would be 0.1 in base 3, which is awesome if true (spoiler it is). really simple way to describe it; we've tamed a beast!

After realising that, I wanted to confirm it

I went duckgo-ing (googling) to find more and see if my intuition was right (spoiler: it was) but I couldn't find a clear explanation of it online. Perhaps because it was so basic (yet also a bit obtuse) there wasn't one immediately obvious. (I'm not familiar with much math notation these days, so if explanations contain too many unclear symbols it gets hard and slow to parse so I kept looking).

I wrote down some examples to test my hypothesis and now to verify I just needed to calculate 1/3 in base 3 using something more objective than myself. However any base converters I found online could only convert decimal numbers and not fractions. The whole point of this exercise is that 0.33333 is only an approximation of a 1/3, so I was stumped.

Is there a way to combine language and math into a calculator?

My first thought was an LLM! I knew LLMs weren't good with math but at the very least they should be able to understand language. I went with Claude 3.5.

"What is 1/3 in base 3?". It confidently gave me the wrong answer. I knew because I prodded and it self contradicted. At this point I felt stuck again (at least I could ask a Mathematician friend). Thankfully I remembered Wolfram Alpha could handle both language and numbers at once! I forgot this site existed.

With a few examples I could quickly confirm my intuition was correct.

This is really cool. Math is fun ya'll. And even better I can trust Wolfram Alpha, so I know it's right, haha.

Trust is a glaring issue with LLMs

In writing this piece I retried the question with ChatGPT and Claude and this time they both got it right. But I only know they got it right because Wolfram Alpha's answers were trustworthy and didn't change.

The problem with LLM hype is that the people behind it themselves just don't know how much they don't know.

Comparing to game design

Tangentially, people who play games or care a lot about games may think they know game design, they might have watched some game design videos on yt, but 9 times out of 10 they are generalising, simplifying or misapplying what they know. (And that's fine and great, at least until they start harrassing people they think know less). They don't know just how much they don't know. And now with AI we have people who don't know how much they don't know and will speak confidently on it, but on literally every domain of knowledge there is. Maybe they don't even realise it's a domain of knowledge in the first place, for example...

Coming back to LLMs I think about this slide often. The prompt is asking for an oil painting and while it's lower res and contains more generative oddities the response that Dall-E 2 gives is to me not only more artistically interesting but definitively a more apt response to "oil painting". However, because of their lack of visual knowledge and education understanding the presenters (and anyone who checked the slides) think the second flashier image is a better response. While I originally saw this on twitter from a presentation OpenAI gave in looking for it I discovered it's actually near the top of their webpage dedicated to Dall-E 3. Increasing the number of people who thought this was a good example of improvement. Link.

While I can't say whether the issue with this cookie example came from worse training data or worse fine-tuning (which I would suspect), it clearly demonstrates the issue with people not knowing how much they don't know.

Are LLMs worth it?

The amount of money going into it LLMs is mind boggling, and maybe there are benefits, but it clearly is a lot of smoke and mirrors too. Imagine those resources going into almost anything else!

Wolfram Alpha might have bugs and might make mistakes 1 in a million, but I am infinitely more confident in it. And what it can do is astounding! the amount of knowledge.

It is a shame that Wolfram Alpha is a for profit enterprise, as this is the kind of knowledge engine that should be a public good. Wikipedia for its flaws in funding is at least dedicated to very different goals which means it can justify initiatives and reach much more than Wolfram Alpha which must prioritize profit.

Do we even value what we could?

I would love to see publicly owned "knowledge engines" but considering that we haven't yet remotely convinced governments that the internet is a human right and should be a public resource, I think doing the same for knowledge engines is unfortunately a ways off.

In conclusion

Writing fractions in different bases is cool but also, we really need less LLM hype and more Wolfram Alpha hype.

Last edited 6th of October

Le meas, Llaura