Tip:
Highlight text to annotate it
X
Yes, so that was one thing that was interesting that we learned on Reddit over the years.
In the beginning it just had a small amount of content,
but as we grew the content kind of turned into two flavors.
Not just hot versus saved but was kind of new versus old.
If you go to your profile page, that content isn't accessed very often.
Before we did the precomputing thing, the hot pages were highly optimized.
The queries were fast. Everything worked nicely. It was cached perfectly.
But somebody will hit a user page that hadn't been accessed in a month.
All of a sudden it's just stale, cold data, it's not in the cache and the whole system just like chugs to bring that out.
That's when we started precomputing those and you guys are taking it to the next level. >>Right.
One thing that we actually do there is there's a special-- comment is the really bad one.
There's just so many comments, so much data and those databases get overloaded,
especially when you pull up an old comment page from two years ago,
and nobody's seen it in like 3 months and it has to load
500 comments and that's a disk hit for every one of those.
We actually have a dedicated comment slave that is just for the likes of Google.
This a whole separate stack for Google, right? >>Yeah, and it's read-only.
Yeah. I would draw a picture, but basically--I think we started adding this just as I was leaving.
There's a whole separate set of databases and app servers, maybe one app server-- >>Two or three.
A couple of app servers that's just for Google,
because Google will come through and index everything.
Reddit gets a lot of traffic these days from Google.
You can take any two or more combination of the front page of Reddit.
Do a Google search for it and it will be in the top 10.
And enough of those, Google is just murdering Reddit.
So we had to build a whole separate infrastructure just for Google,
because they hit the site in a completely different way from users
in a way that it's really hard to cache for.