We’re planning to begin reading all adventure actions from our new S3 architecture instead of the database today. This is a significant architecture change that we’ve been working on for several months to reduce the traffic on our database. With this new change, new actions will be written to BOTH the new S3 architecture, as well as the database, for extra safety and redundancy. This also allows us to roll back to reading from the database, if needed.
This is a more aggressive timeline than we were planning on and, frankly, it’s more aggressive than we would normally consider for a change this significant. However, the multiple slowdowns and outages in recent weeks were either caused by or amplified by the load on the database.
At this point, it seems like we’re doing you a disservice by not being aggressive with this transition—the pain of the old system is worse than the potential pain of moving fast (even if we hit some bugs and issues). This infrastructure has been tested, and we’re confident it’ll be a better solution than continuing with the database approach. As we transition, our team will be paying close attention to community reports of issues and preparing any needed fixes.
We will set Beta to read from the database (the old architecture) so that any players experiencing issues can switch to Beta to use the old architecture. At this point, Production and Beta will be functionally identical except for the storage location that actions are being read from. Once again, both Production and Beta will write to both S3 and the database for redundancy.
If you’re on prod and you see any issues with your adventures, please let us know. Your data is safe and you’ve likely just hit a bug. Examples of past (and resolved) bugs we’ve seen with this new architecture include actions being loaded out of order, context being out of order, adventures not loading fully. Please keep your eye out for any issues like this and switch to Beta if you experience these. We’re also aware that adventures with more than 1000 actions may not be working well (yet) in this new architecture so you may want to use Beta to play large adventures.
Please let us know if you have any questions. Our team will be on high alert as we navigate this transition.