We’re thrilled to introduce a brand new Memory System as an Experimental feature for AI Dungeon with AI Renaissance Drop #3! We’re expecting to release it to Beta in the next few business days, with a release to Production a week later.

Better AI memory was the top requested feature in our latest player survey, with 78% of you saying you’d be excited about it. The experiences and feedback you shared with us on Discord, Reddit, and support further emphasized how important memory is to improving your experience. Because of all your feedback, along with doubling context lengths, we decided to devote the majority of Drop #3 to improving AI memory in AI Dungeon.

The Memory System stores and retrieves key information from your adventure, no matter the context length. It keeps the AI on track and lets you create deeper, richer stories on AI Dungeon. We believe it’s one of the most significant improvements to AI Dungeon in a long time, and we’re excited for you to try it out!

Why We Built The Memory System

Why Memory Matters in AI Dungeon

A while back, Nick posted a Heroes Dev Log about why memory is such an important part of AI adventures. The core promise of AI Dungeon is that you can have ultimate freedom to be who you want to be and that the choices you make truly matter. But if the AI forgets those choices after a few thousand tokens, then those choices actually become meaningless, breaking the power of that promise.

If the AI forgets the names or details of characters you meet, how can you build deep meaningful relationships with them? If you set out on an epic quest, but halfway through, the AI forgets why you even started on it, then how does that quest have any meaning?

I’ve been realizing that to build AI Dungeon into the game our players truly deserve we have to solve this problem. —Nick Walton

It became clear to us that solving this long-term memory problem would be one of the most powerful ways we could make your AI Dungeon experience deeper, richer, and more meaningful.

At first, we solved this problem only in Heroes, building a new memory system, which we wrote about here. We originally weren’t going to bring that improvement to AID since it’s a very complex system to build, but after hearing from alpha testers just how powerful it was, we knew it had the potential to significantly improve AI Dungeon as well. It took quite a bit of work to get it right (Alpha testers can attest to the number of iterations we went through), but we finally figured out the version that we believe will transform AI Dungeon into a much better experience.

AI Context Length, a Major Technical Constraint

So why do we need a memory system?

One of the biggest limitations of using AI large language models is limited context length, which is the amount of information the AI can process when generating its next output. Each model has its own max context length limit, and we also have tier limits in AI Dungeon since the longer your context length is, the higher the cost for the AI to generate a response. Context length is a real, unavoidable constraint that we have to work within.

In AI Dungeon, we construct the context we send to the AI by combining text from your story with AI Instructions, Plot Essentials (formerly called “Memory”), Author’s Note, and relevant Story Cards. The AI processes this information and generates the next action in your adventure.

Eventually, the amount of content in your story exceeds the amount of text we can send to the AI. We have to cut portions of text, usually by removing the oldest parts of your story text. Since text is cut, it can feel like the AI is “forgetting” important details of your story since those details were unable to be sent to the AI.

How Players have Managed the Context Problem

If you’re one of our more experienced players, you know that through careful manual editing of Plot Essentials, Story Cards, and Author’s Note, you can make sure important details are still being included in the context.

Or perhaps you decided to subscribe to higher tiers on AI Dungeon, which offer longer context lengths and extend how long stories can be played before text starts getting trimmed. Our free tier gives players up to 2000 tokens of context (about 8,000-10,000 characters of text). Mythic, our highest tier, has models that support 16x more context than the free tier, a whopping 32,000 tokens of context (and GPT-4 Turbo can support 128k with credits). Every doubling of context approximately doubles the cost we have to pay for your AI call. That’s why additional context is a paid feature of our different tiers.

Why We Want to Do Better

While these solutions can help, we don’t think you should have to pay for the most expensive plan to have a good experience on AI Dungeon. We also don’t think you should have to break immersion to constantly manage the context being sent to the AI.

We’d like you to be able to create immersive stories where you don’t have to worry whether the AI will forget important details you worked so hard to create. We want you to have adventures where you’re not afraid to build deep investment in your character, your relationships with the characters around you, and the lore of the whole world, where you can play a single adventure for months on end in a world that just gets richer and richer with each passing hour.

With the new Memory system, we’re taking an important step toward providing an experience in which the AI can more reliably recall important details of your plot and story, giving you a deeper, richer AI Dungeon experience.

How the Memory System works

Memory System Overview

What Information Should We Store and Use?

As we considered what information should be part of the memory system, we found ourselves seeking inspiration from how the human brain works. When our brains store and recall information, we use two strategies. The first is by compressing memories, where we take a large chunk of information and distill it to remember the most important parts. The second is memory retrieval, where some memories stored in our brain only come to mind when they are relevant to our current context.

For instance, once you finish reading this blog post, it’s unlikely that you’ll be able to recall it word for word in its entirety, but you’ll probably be able to recall the high-level concepts, big changes, and parts that matter to you. This is compressed memory.

You also have a retrieval system that will bring important memories to your mind when they’re relevant. If you hear the word “fire hydrant,” your brain retrieves memories about what a fire hydrant is. You might think of its color and shape, its function and purpose, or any specific memories you have involving one.

Both memory compression and memory retrieval are important parts of enabling your brain to remember what’s important and have the right context when it needs it.

These two features of our memory system are extremely useful in AI Dungeon, too. The story equivalent to a compressed memory is summarization, which will compress a section of the story to just the high-level information about the setting, characters, and events. Being able to retrieve more detailed memories is important, too, since that lets the AI remember important details when they’re relevant in the current context.

What are AI Dungeon Memories?

The purpose of the Memory System is to compress, store, and retrieve Memories to give the AI both a high-level perspective and the ability to recall specific details.

For AI Dungeon, a Memory is an AI-generated summary of a small set of your previous story actions. The summarized Memory retains key plot details but without the illustrative prose that is common in AI Dungeon adventures, making it more information-dense than the text of your adventure.

A new Memory is created by taking four of your previous actions (and the AI’s responses) and sending them to an AI model trained to summarize stories. This summary output from the AI is a Memory.

When starting a new adventure, we’ll wait until you are 8 actions deep into your adventure, and then we’ll summarize the oldest four actions (actions 1-4 of your adventure) into your first Memory. This means your first four actions will be summarized and stored as a Memory, but your most recent four actions are not. Then, after you take four more actions (now you’re up to 12 actions), we’ll summarize the previous 4 as a new memory (actions 5-8). This means your first 8 actions have been converted into two separate memories and your 4 most recent actions have not been. You are free to make edits or undo your last four actions without it impacting Memories.

This cycle repeats indefinitely, and a new memory is created every four actions.

Two Complimentary Features: Auto Summarization and Memory Bank

Memories are used by the two major features that make up the new Memory System: Auto Summarization and the Memory Bank. These two features work together to recall important information, giving the AI the ability to remember the high-level overview and to remember specific relevant details, just like our brains do.

Auto Summarization keeps a running overview of your story’s plot to help the AI track the overall direction of your story. We do this by appending new Memories to your Story Summary, a new Plot Component we’re adding to support this feature. After several Memories are added to the Story Summary, we’ll re-summarize it by sending it to our summarization AI model. Because the summary will have the overall context of your story, it will help keep the AI on track and make sure it doesn’t lose the plot.

The Memory Bank stores and intelligently retrieves relevant Memories and includes them in the context. In practice, it acts like an automatic Story Card system since it saves important details from the story and then dynamically inserts them into the context when they are relevant to your current action. Whereas the Auto Summarization feature is meant to keep track of the overall plot and story direction, the Memory Bank might recall important details like your dog’s name, where you found the Sword of Demons, or that Rangers are allergic to horses and only ride cows.

How Auto Summarization Works

A New Story Summary Plot Component

As we mentioned, there is a new Plot Component called Story Summary where the Auto Summarization feature will store the summary it maintains for your story.

Although Auto Summarization is a paid feature, free players can still use this new plot component to curate a summary of their adventure manually.

Like other Plot Components, the Story Summary can be used by our content creators for published Scenarios. It could be used to include backstory information for your players to use, and it will be utilized by the Auto Summarization feature if enabled by the player.

We’ve added Story Summary to the context inspector so you can track how many tokens of context it’s using for each call.

Auto-updating the Story Summary with Memories

With Auto Summarization enabled, each time a new Memory has been created, it will be appended to the Story Summary plot component. Once the Story Summary gets long enough, we’ll pass it to our summarization model and compress the summary. This will ensure that the Story Summary continues to be an information-dense overview of the entire story plot.

This process will continue throughout your adventure, with new Memories appended and then re-summarized automatically.

Auto Summarization for Existing Adventures

The Auto Summarization will work best for new adventures since the summary feature will dynamically update from the beginning of the adventure.

For existing stories, we’ll summarize the last 8000 tokens of your adventure when you take your first action with Auto Summarization enabled. Or, if you prefer, you could manually summarize the story, and the AI will take that into account for all future summarizations.

Summaries and Editing

Even with Auto Summarization enabled, you can edit the Story Summary manually. You may want to edit the automatically created summary to correct any errors or add additional clarifying details. Although the Auto Summarization overwrites your existing summary, your edits ARE sent to the summarization AI, so they should be incorporated in the new summary.

Only edits to your most recent four actions, including changes, undos, redos, and erases, will be considered by the AI for the Auto Summarization. Any changes to previous actions, including the use of Erase to Here, will not update the summary. We will, however, update any corresponding Memories stored in the Memory Bank by summarizing the action set that creates that memory. If you make changes earlier in your adventure, and those changes would have an impact on the summary, you’ll need to update the Story Summary manually.

This is due to the limitations and costs of the summarization AI model. Like all models, we are constrained by context size for the summarization. Many adventures would extend far beyond the available context limit, so regenerating a summary based on edits wouldn't be possible in many cases. Making new summarization calls for every historical change could also become very expensive.

How the Memory Bank Works

Let’s Get Technical: Embeddings and Vectors

The Memory Bank uses AI technologies that will be new to many of you. To explain how the Memory Bank stores and retrieves relevant memories, you’ll need a basic understanding of embeddings, vectors, and embedding models.

Embeddings represent how similar words and phrases are by mapping them in a multidimensional space. They are created by passing text into specialized embedding language models, which return a vector of numbers as a result.

You can then do simple math to compare how similar different vectors are to each other. For example “water” would be more similar to “liquid” than it would be to “solid”. You can also compare phrases and questions. For example, “Who is Sir Theo?” might match closest with the phrase, “Sir Theo introduces himself to you as the knight who slew the evil dragon of Larion”. These two phrases might also match closely to other passages about Sir Theo or the dragon of Larion. But none of those phrases would likely match very closely to the phrase “Sarah was a forest ranger.”

This lets us build a memory retrieval system. First, we store a bunch of memories in the memory bank with both their text and the embedding vector of that text. Then, when we want to find relevant memories, we decide what text we want to use to look up memories (for our system, we use the most recent action). We get the embedding vector for that lookup query, and we compare it against all the vectors in our memory bank and get a relevance score.

A higher score means it’s more similar to the query, and a lower score means it’s less similar. This lets us decide which memories are relevant to the current events in your story and should be included in the context sent to the AI when generating the story.

Now, we’re ready to talk about AI Dungeon’s new Memory Bank Feature.

Storing and Retrieving AI Dungeon Memories Using Embedding Models

With the Memory Bank, each Memory (an AI-generated summary of four past actions) is embedded and stored. As you continue to take more actions, your Memory Bank starts to fill with more Memories.

When you start a new story, your Memory Bank will be empty. As you take actions, and Memories are created, they will begin to fill your available memory slots. Initially, no memories will be used since your entire adventure history will fit in the context. Once your story has reached the point where it can no longer fit your whole adventure into the context window, we start to retrieve memories from the Memory Bank. We do this by using an embedding model to rank all of your memories by relevance to your current story.

Then, we take the most relevant memories and include them in the context being sent to the AI for the next action of the story. For example, if you are just returning to the town of Castlebrook, the AI will find all the memories related to Castlebrook, remember what happened the last time you were here, and include the most relevant in the context. These are called “Used Memories.”

Before including a Memory in the context, we also check to see whether the Memory is included in its full text within the Story Summary. This check prevents the text of a Memory from appearing twice in the context.

The Memory Bank will be allocated a portion of the context window, similar to how we allocate space to Plot Essentials (formerly called Memory), Author’s Note, and Story Cards. Based on the context space available, we’ll include as many of the top memories as will fit from your memory bank.

Once your Memory Bank has been filled and a new Memory has been created, we remove the least used memories to make room for the new addition. These removed memories are called “Forgotten Memories”. Just like with human memory, very old memories might stay in the Memory Bank forever if they have been used frequently.

The larger your Memory Bank, the more room you have to store memories that could be used in your story, increasing the accuracy of the AI storyteller.

As you play, the AI will continually curate the most important and relevant memories from your story and intelligently include them in the context being sent to the AI when they are relevant, improving the overall experience.

Launch Details

Opt-in Experimental Feature

The Memory System could be one of the most foundational changes to AI Dungeon in recent history. Although we’ve tested it thoroughly internally and with our Alpha testers, we anticipate that as you all start playing with it and give us feedback, we may need to adjust parts of the experience. There could also be unintended behaviors we haven’t encountered yet, or edge cases we didn’t consider. The design might need adjustments to be clearer or more helpful to you.

To avoid any unintended player frustration, we’re launching this first into Beta and then into Production as an opt-in Experimental feature. You can enable it under the AI Models tab on the game screen settings sidebar. There’s a new section there called “Memory System.”

How Your Membership Tier Affects the Memory System

The Memory System will be a Premium feature. When we initially announced it, we expected to make it available to Free players. Our initial prototype was based entirely on the embedding system, and our projected costs were much lower. As we continued to iterate and improve the system, we realized summarization played an important role for both Auto Summarization and the Memory Bank. Those summary AI calls increased the cost of running the Memory System for players and, as a result, it’s currently cost-prohibitive to offer the Memory System to Free players.

Our hope is that we can offer the Memory System to free players in the future. We’ve done this before with other features that were Premium only, then made available to free players over time. Examples include unlimited AI calls, image generation, multiple AI model options, advanced settings, and even Premium AI (with our promo actions).

All subscribers will have access to the Auto Summarization feature. For the Memory Bank, each tier will have a set number of memories that can be stored. You can view the Memory Bank sizes of each tier on our Membership Page.

Key Differences From the Heroes Memory System

Those familiar with the Heroes memory system will see similarities with this new Memory System for AI Dungeon. The work done on Heroes directly inspired and informed the new memory system. There are some important differences and adaptations we made to implement it for AI Dungeon.

The Heroes experience is a much more traditional RPG experience. We track important stats like health, quests, inventory, levels, and player characteristics, which are stored in a “game state.” We didn't bring this part over since AI Dungeon is a collaborative storytelling experience.

We also made some adjustments to account for the different gameplay in AI Dungeon. We know you all value editing, a concept that isn’t as prevalent in Heroes, and supporting editing with the AI Dungeon Memory System required some adaptations.

We’re Excited for You to Try Long Term Memory!

You’ve repeatedly told us that the AI forgetting important information is one of the most frustrating parts of playing AI Dungeon. The new Memory System is built to address that issue. As you play with it, please share feedback and let us know how it affects your stories and adventures. This is a new system, and that means there’s opportunity for improvement and iteration.

We know many of you have been excited about this new feature. Thanks for being patient as we went through additional iteration cycles to get the experience right. We hope the Memory System will help you have better, deeper, and longer adventures.