they grow up so fast
Context windows are rising. Cost of inference is falling. What does this mean for products that are built on top of LLMs?
I’ve been using Lovable to build web apps for the last 6 months. Pretty much ever since it launched. Recently, I was browsing through my prompt history to contextualize how rapidly LLMs become smarter.
The first web app I built on Lovable was Expensify: a simple tool to track my expenses. It took me 78 prompts to build a basic CRUD application with bare minimum functionality.
A couple of months ago, I started building SoulSpace: a social journaling app with an AI therapist chatbot that uses user journal entries as the RAG layer.
With models getting faster, cheaper, and smarter, not only could I be less stingy with my prompt credits but also get much more effective one-shot results. Crucially, with growing context length, I could also unlock further capabilities. For instance, enhancing the prompt with long journal entries as context.
Now I’m not a trained therapist, so I wouldn’t know how to instruct a chatbot to act like one. However, by simply feeding an LLM journal entries from users, I could now work towards creating an AI therapist, without really fine-tuning my own model (which of course would be infeasibly expensive).
So what does this mean for the app layer on top of LLMs? Well, it's rather abstract and hard to imagine but it comes down to one question:
how can we create interaction loops between LLMs and humans that non-invasively extract as much context as possible from humans to feed to LLMs?
As the likes of openAI and Anthropic focus on increasing context window and making cost of inference cheaper, answering the above question becomes the real deal. That's where value lies.