Recent News on Recent News
Posted by Doug Tue, 15 Apr 2008 17:07:00 GMT
[with thanks to Charlie Savage for contributing to this post]
We've had a news feeds on the site for some time now. Unfortunately, as the site has grown, we realized our original design wasn't fast enough, particularly for large communities like the Colorado Mountain Club and NTEN. So it was time for a redesign.
Our news system consists of two concepts - events and notifications. Events happen when a user performs some action on the site, say, creates a map. Notifications are the individual items that make up a news feed. The big difference is that events are generic while notifications are targeted to specific users. To understand the need for this distinction, its easiest to go through an example. When you create a map, the notification in your news feed will be different than the notification your friends see.
Ideally, notifications can be generated on the fly from events, allowing you to have full and accurate information for every situation. But that turns out be slow. Very slow. Since notifications are viewed many times more than they are created, our original implementation was designed for fast retrieval and slow creation. To do that, we created all the notifications at the time of each event and store them into tables for later retrieval.
Of course, this ultimately ended up being a problem. Over time, we started using notifications in ways that we didn't expect. For example, they became a big part of our integration with Facebook and Twitter. We also added new ways of viewing them on the site that the original design didn't envision. Most of all, we didn't plan on having communities with thousands and thousands of users quite as soon as we did. As these problems came up, we fixed the quick and easy things - such as generating notifications in background processes so as to not slow down user requests. But it wasn't enough, and it was time for a major redesign.
Our basic premise remained the same - fast read and slower creation. But we needed to significantly improve notification creation times so that we could generate thousands of them at a time. We did that by storing the original events along with the notifications. That allowed us to move all the notification creation code into SQL, thereby vastly speeding up creation times.
Although it seems, and is, a simple change, it of course rippled throughout the system since notifications are used in so many places. Luckily one of our mantras from the beginning of MapBuzz is as complete test coverage as we can manage. So although it turned out to be more painful change than we thought, we had a good idea at most times of what still needed to be fixed.
And now that we've done it, the notification system is once again fast. It's not perfect, of course (no one can ever really afford perfection), but it's good enough for now.