Tip:
Highlight text to annotate it
X
PETE: So, what about scalability in your architecture? How do you handle that and what are some considerations
that you're looking at in terms of making sure your application architecture is set
that you can scale, because you guys are, you're not huge right now, right? But obviously
the goal is to processing, I'm sure hundreds of thousands are more accost than you are
and you keep scaling that up.
VIDA: Right, So, it's also naturally partitioned right now, where our Simple DB store wanted
to be global. So, we always want all of our users to be on just one database, to be readily
available to us. On the other hand, since we're set up with retailers, we could actually
partition, every retailer's data can actually be completely separated and partitioned from
the other. So, we can serve traffic for specific clients on specific servers with a different
database. So, I'm never worried about one particular database having to handle a humungous
load. For our consumer data, I didn't build simple DB so I don't know how it works but
there's always things you can do for a consumer, you could separate our their data and modulo
it according to certain keys, put all your users at that end with this number on this
particular user data. So, there's still ways to partition the user data. My guess is that
Simple DB probably already does that and kind of hides that from the developer without having
to figure that out by serving it automatically on separate servers without you noticing and
labeling the whole thing as just one big service.
PETE: So, you anticipate pushing Simple DB as far as you possibly can, you're just going
to keep throwing users at it and if it starts to cry and scream then you'll reconsider.
But right now, you think that's probably going to suit your scale.
VIDA: I hope so. I would definitely go with the key value store rather than a SQL store.
I think that SQL has a lot of overhead because you need to process queries, where you need
to query across it and build indexes across it and all the other stuff. And if you want
scalability just being able to store data in different places and partition it out makes
a lot of sense. I mean at Google we didn't use SQL at all in our back end servers anyways
for speech recognition or even in our web server. So, just not really used to thinking
of SQL as the automatic database choice as I think most of the industry does.
PETE: What about the breath of social data that you have at your disposal from, say,
a Facebook API, one of the challenges there is that, that data is always changing, right?
And Facebook is basically changing the schema behind their API and adding fields and deleting
fields and probably merging stuff and whatnot, they're known to do that right?
VIDA: Right.
PETE: Does that give you a lot of headaches and problems?
VIDA: No, because we didn't choose to store the social data in a SQL store. But it is
true, I think you wanted to have, we could originally define a user profile and assume
that their DB certain fields, we'd always have to be updating schema and things like
that. So, Simple DB is really nice because it doesn't assume a strict user schema. You
can still label parts of it but you don't have to strictly label it that way and predefine
it.
PETE: Okay.
VIDA: So, that allows us to be really flexible.
PETE: So, those are really great compliment to each other. The social data you're getting
from a “not always what you'd expect” Facebook API and the schema-less format of
Simple DB.
VIDA: Yeah.