With Drop 3 of our AI Renaissance chapter, we doubled context lengths for every tier on AI Dungeon. As AI models become more powerful and intelligent, increasing context length has become one of the clearest ways we can improve your experience on AI Dungeon. Doubling context length doubles the cost of every AI call, so it was a change we had to approach carefully.
Here we want to share more about this change, and especially why and how we fought hard to make it happen for all of you.
Have I told you lately that I love you?—Rod Stewart
We talk about you a lot. Like, a lot a lot. We’ve become obsessed with finding new ways we can make AI Dungeon better for you. The context doubling was borne out of that obsession, and we wanted to share a behind-the-scenes look into how we approach serving you, especially how we balance giving you value while also running a sustainable business.
The Heartbeat of Latitude
Over the last few years we’ve gone through a lot as a team. We’ve had successes and failures, we’ve made mistakes (some very large) and we’ve learned a huge amount from all of it. Through all of that we’ve had to confront and decide some big questions “Who are we? How do we makes choices in hard situations? And what do we want to focus on as a company?”.
In the past we’ve gotten off course, we’ve over focused on monetization or on how we were perceived. We thought that by trying to optimize funnels and metrics we’d be successful. But that never really brought AI Dungeon to what we wanted it to be.
As we’ve grappled with some of these questions, and learned from past mistakes we finally started to see clearly the answers to those questions. We realized that if we focus on delivering value to all of you above all else, everything else will fall into place.
This has become the heartbeat of Latitude. “How can we give our users more value?” This is the question we prioritize and ask ourselves more than any other. As we’ve tried to hammer this pillar into how we think and work, we’ve looked for other companies that have a similar mindset that we can learn from and be inspired by.
Recently, we’ve been particularly impressed by Costco after we listened to an Acquired podcast episode detailing their focus on customer value. Costco consistently fights for their customers even when it won’t benefit them. For example, Costco tracks the prices of raw materials so that when those prices drop, they can insist that their suppliers lower the prices of their products to reflect the lower cost. But they don’t keep those savings for themselves. The most Costco will ever let itself make on a product is a 15% margin. Instead these savings get passed on to customers, making sure that prices are as cheap as possible for their members.
Their extremely generous refund policy also recently inspired us to change our approach to subscription refunds. Now if you forget to unsubscribe from AI Dungeon we’ll refund you all the way back to the last time you played, no matter how many months it’s been [Link to that blog post here]. These are just a few examples where you can clearly see Costco’s obsessive focus on delivering more value.
Sustainable Business
To deliver value we also need to make sure we are running a sustainable business. Giving everyone unlimited access to GPT-4 Turbo, the most expensive AI model we offer, would create a ton of player value…but it would put us out of business. We need to operate sustainably to provide and build the best AI Dungeon experience possible.
This is particularly challenging for companies who rely on AI like we do. Although costs are coming down, AI is still relatively expensive compared to tech used by traditional games and platforms.
Thanks to all of you, Latitude is fortunate to be operating sustainably today. We’re not on the VC-funded hamster wheel that many companies get caught in, where they need to fundraise regularly to stay afloat. We’re in control of our future and destiny. Without investors pressuring us for higher profits, we’re able to stay true to our values and stay focused on serving you.
We might be hamsters, but we’re not part of the startup rat race.
Commitment to Free Players
Our free tier is where the tension between player value and sustainability has often been the most challenging. It’s often easy to focus on your paying customers, but we work hard to make our free tier as compelling for players as we can.
Because of how expensive AI is, our free tier continues to be unique in the AI space. This hasn’t always been easy. In the past we’ve had to limit free player use through energy, and then later, ads. Today however free players can enjoy unlimited AI Dungeon without restrictions.
We’ve also significantly expanded the features available on the free tier. Just in the last year, we’ve been able to go from one free AI model to three. We’ve added promotional actions you can get a taste of premium AI models without paying. And Advanced settings are available now for everyone, so you can get the best experience possible.
So why offer a free tier at all? Because it actually adds value to ALL players, both free and subscribed.
The free tier means more players use AI Dungeon, and in turn, that leads to more content being created, more people for you to interact with, a better social experience, and overall more fun. Even though it makes being sustainable more challenging, a free tier results in much more value for the entire community. Plus it allows anyone to experience the magic of AI Dungeon even if they aren’t at a place where they can afford a subscription.
Our goal is to give players as much value as possible, while still being able to operate sustainably. As the AI space continues to evolve and improve, we expect to find new ways to give you more value.
Our Unique Positioning
Now, let’s get into the details of strategies we used to give you more context in AI Dungeon. This is the first time we’re sharing some of these publicly since, on the surface, these strategies might not seem like they’d have a direct impact on player value. Please tell us if you like hearing about this kind of work, and we’ll share updates like this more often if you do!
Provider Agnostic Architecture and Negotiating Power
One of the most painful lessons in our company’s history was learning how dangerous it is to be dependent on a single technology provider. In the past this had a profoundly negative impact on players’ experience, and we’ve changed how AI Dungeon is architected since then to avoid being caught in a similar situation again.
AI Dungeon is set up to be provider agnostic. This means that, at any time and with minimal effort, we can change AI providers. For example, we’ve hosted Mixtral on three different providers since launching the model to players in December. There have been instances where we’ve had outages on one provider, and been able to switch to another provider to keep AI Dungeon running.
Being provider agnostic allows us to evaluate dozens of different AI providers and score them on dimensions like cost, strength of partnership, privacy and security policies, content policies, tooling, and server uptime and stability. We’re careful in evaluating potential partners to make sure we choose partners who can provide the best overall experience for our players. We’ve also negotiated changes in policies to align with your values around content freedom and privacy.
Since we have clear insights into our traffic and AI use, we’ve been able to negotiate discounted rates on AI compute by committing to large amounts of traffic with our technology partners to receive volume-based pricing. Like Costco, we’re passing these savings on to you in the form of increased context length for no additional charge.
We’re particularly grateful to our two newest partners, together.ai and octo.ai, who have made these recent changes possible. We’re also grateful to our other providers—Azure, AI21, and Coreweave—who continue to be good partners.
Model Agnostic Strategy and Robust Evaluations
Not only are we agnostic to providers, we’re also model agnostic. Our AI systems are database driven, allowing us to quickly add new AI models, control access, and run comparison tests. The architecture is flexible enough that when a new AI model becomes available, we’re able to evaluate it without even having to write any new code.
We’ve deliberately decided NOT to be in the business of creating our own custom AI models (although we are finetuning models for specific tasks). By leveraging models available commercially and through open source, we’ve been able to take advantage of the wave of innovation happening right now in the AI space. We’re building AI Dungeon (and Heroes) to be incredible experiences that can leverage the best AI models available on the market. We’re convinced that with a small team like ours, building models in-house won’t let us provide value to you as quickly as we can by leveraging third party models. We love being built on the backs of giants like Meta AI, Kobold.ai, Wizard LM, Azure (and OpenAI), Mistral, AI21, and more.
And, oh boy, have we been busy evaluating new models. We regularly get questions from some of you about whether we’re testing new models. The answer is almost always “yes”. We’ve evaluated nearly every promising new model that’s been introduced recently. As a result, our process for evaluating AI models has become quite robust, blending both qualitative and quantitative feedback into our process.
We’ve always relied on our AI Comparisons tool for qualitative evaluation, and now robust industry benchmarks and leaderboards are providing additional metrics to look at. We also look at model properties like parameter counts, response times, and supported context lengths.
We also do qualitative tests by playtesting ourselves, and for models with more potential, opening them up to Alpha testers for more testing and feedback. We look at things like storytelling ability, following instructions, creativity, and coherence. Moralizing is an issue we look out for, and is more common from commercial providers like Azure and Google. Some of these models reject harmless content like fantasy violence, making them poor models for a role play game like AI Dungeon.
Of course, we also look at costs to see whether models are viable and compete with other models at their price point. For instance, we’ve had some models perform similarly to Mixtral 8x7b, but at a much higher cost. If a model performs the same as Mixtral, but we can only offer 1/4 as much context length, this doesn’t seem valuable to offer players. Mixtral, for instance, is a better model that costs less than our outgoing Dragon model. It’s affordable enough that we can offer larger context sizes at each premium tier than ever before. Being model agnostic allows us to quickly test, evaluate, and introduce new models like Mixtral that give more value to players.
Our goal is to have a small portfolio of some of the best models available. We only add new models to our lineup if it’s clear they’ll offer significant value to you. Most models don’t pass our evaluations. For instance, we’ve been surprised that none of the Google models have met our expectations due to their heavy moralizing and below-average storytelling abilities. We’re hopeful that they’ll introduce future models that will be on par with offerings from others.
Being model agnostic means we’ve been able to have better models at better pricing. This means better AI and larger context lengths for you. And the good news is, it’s only going to continue to get better over time!
The outcome of us being provider and model agnostic, is that we’re able to easily take advantage of newer models that offer better performance at a lower cost. We’re also able to negotiate incredible pricing terms to further lower our costs, enabling higher context lengths and a better experience for all of you.
Putting It All Together
We’ve gone deep on the heartbeat of Latitude, a constant focus on how we can deliver more user value. We’ve also shared how we set up our AI architecture to enable that. Now let’s summarize how that has all come together to make double context possible.
In short, to support giving everyone double context, we needed to find a way to sustainably support (roughly) 2x AI costs. We did that in a few ways:
- We architected AI Dungeon to be provider agnostic, allowing us to find the best providers at the cheapest prices, especially as new models and providers have come out in the past few months.
- We leveraged the high traffic volume of AI Dungeon to negotiate discounted pricing on AI compute (so we could give that back to all of you)
- By being model agnostic, we’re able to evaluate and deploy the highest quality and most affordable models
- We’re passing these cost savings on to you by providing double context length for each tier, for no additional charge.
- These changes will still allow us to operate sustainably, ensuring AI Dungeon will be around for the foreseeable future
As we mentioned earlier, this is the first time we’ve shared some of these details publicly. Please let us know if you enjoyed this post, and we can share more updates like this in the future. We’re incredibly grateful for your feedback and support, and we are working hard to give you more and more value in AI Dungeon.