Tip:
Highlight text to annotate it
X
>>Tony Voellm: So this next speaker up, I believe his spoken at more GTACs than anybody
else. He probably does not need an introduction but I will do one.
This is Simon Stewart from Facebook, and he is going to talk to us about how Facebook
tests Facebook on Android. And so with that, I hand it off to you.
Thank you. >>Simon Stewart: Brilliant.
[ Applause ] >>Simon Stewart: Thank you very much for that
introduction. Yes, I think I have spoken at more of these
than anyone else. This is the one where I snatched the crown from James Whittaker.
[ Laughter ] >>Simon Stewart: Sorry, James, if you're watching.
[ Laughter ] >>Simon Stewart: Before I begin, I've been
having a fantastic day today. The quality of the talks has been amazing. And the thing
that I have really noticed is just the awesome work the stenographers have been doing. Like
when I haven't been paying attention. [ Applause ]
>>Simon Stewart: And the signing down here is phenomenal. Just blown away by how well
organized this is. So well done, Google. Keep up the good work.
So, yeah, how Facebook tests Facebook on Android. If I'd been thinking carefully when I named
this presentation, what I would have done is I would have called it how Facebook bakes
quality into Facebook on Android, because we don't just test for the sake of testing;
right? It isn't something that we sit there and go, you know what? I really fancy doing
some pointless testing today. There's nothing I'd rather do.
Actually, sometimes I do feel like that, but it's only when it's a really slow day.
Why are we doing this testing? We're doing it to bake quality into our apps; right? We
can't bolt quality onto an application at the very, very last minute.
We can't bake security in at the very last minute either. We can't bake performance in
at the last minute. And these are things that we need to be thinking of all the way through
the life cycle of the application. So, yes. How Facebook tests Facebook on Android.
So let me take you back to 2012. It was a long, long time ago in a galaxy right here.
The Facebook app on Android. It worked; right? It was kind of fun. And it also had really,
really, really lousy user experience. You know, here's a quote from Mark here going,
"We were never able to get the quality we wanted. Looking back, that's probably one
of the biggest if not the biggest mistake we made."
And why was the performance of the application so appalling?
Well, actually, it wasn't that bad, but what we were doing is a pattern that's fairly common
in the world of mobile applications and that's to use the WebViews extensively. Facebook
was a Web company. We liked to be able to release software. We do it twice a day now.
We push to our live Web site twice a day. That allows us to iterate on things really
quickly. It allows us to make a series of chronic and awful and terrible mistake, realize
we've done something and fix it the next day. The problem is that you can't do that on a
mobile app; right? Someone downloads the app, and then they're stuck with it.
If you've got a bug in that application, they may never update again, which is a horrific
place to be in. So we'd like to retain some flexibility. And
the way we did that is we used Web views. And the Web views on Android and on iOS are
suboptimal, I think is the way to describe them.
You know, there's a number of problems with them, not least of which is they're like no
other browser out there; right? The Android one is based on the stock Android browser
which is not Chrome, unless you're on Jelly Bean and then the Web browser is something
completely different. The JavaScript engines tend to behave appallingly. They're very slow.
They're tedious. So what we were doing is we were making use of these WebViews to allow
us to iterate more quickly, and it was just leading to a really poor experience for our
users. So we had to stop. We reassessed, and what
we did is we decided we were going to make mobile a first-class experience.
That went the wrong way. I could do the entire talk in reverse, by the way.
[ Laughter ] >>Simon Stewart: Start on the final slide.
Begin with your questions and just make it up from there.
[ Laughter ] >>Simon Stewart: Hands up if you'd like me
to do that. There's one hand! Two hands. Brilliant! Next year; right? I'll be invited back for
sure after that. So what we did is we decided we're going to
take our engineers and we were going to train them up on mobile. And this graph here has
the numbers removed but there's a total of about 1400 engineers at Facebook, not all
of them contribute to our mobile code base. But what happened is it was all going along
fairly regularly and then we went what we need to do, we need to put mobile first. We
took our PHP developers, we ran them through training courses, we educated them, we hired
people in who were skilled at Android and mobile development, and that's the result.
We had this massive uptick in the number of engineers committing code to our mobile codebases.
That's a fantastic thing. We're now moving a lot faster.
The important thing, though, is its engineers contributing to our mobile code base. There
are no Android team. What I mean to say is there is no longer an Android team. We tried
to do that. We tried to have a group of people who are responsible, we called them the core
Android team and they were going to be awesome and they were going to somehow keep up with
the flood of new features coming into the Web. And that didn't work. So what we did
is we took a step back, we reassessed again, and now what happens is each of the teams
producing a feature own that feature on every platform it's on.
So if you own pictures, you own it on the Web, the mobile Web, iOS, Android.
So it's far easier for people to keep things in sync; right? There's probably people on
those teams who are specialized on Android or iOS, but there isn't a group of people
who are responsible for desperately trying to ape features from the Web in a mobile client.
And that allows us to keep parity in place; right? And that means users get a far better
experience. If something appears on the Web, chances are it will also appear on the mobile
client relatively soon after that. We do have spoons, so there is no Android team. We do
have spoons. So what's a developer's day like? Let's say
you're on one of these feature teams and we're going to do something. How do we bake quality
into the application? It's done all the way through the process.
So to begin with, how we build. That's seems to be quite an important thing to do with
software. You need to build it. So we get -- at the moment for source control,
it's a DVCS. And so what we do, we have a central repo. And that seemed like a good
idea. Everyone works on their local machine. There are two branches that we care about
day to day. Google, I know, only have one, and that's because none of their tests fail,
because they've got the presubmit, and it's awesome. We haven't got the presubmit, so
people check in. So there are two branches that people care
about. The first one is master. That represents not a villain from Dr. Who, but the most recent
version -- You can see who enjoys British sci-fi here.
[ Laughter ] >>Simon Stewart: It's the latest version of
the code. So master has everything in it that has been
committed to the codebase so far. It's a linear history.
What we also have is a moving tag called stable that represents the last time that we were
sure that our unit tests were passing. So there's master and there's stable.
Anyway, that's kind of fun. And all this code is checked out for Android at least in a single
tree, with, effectively, three or four directories up at the top: Java, Java test, because you
never mix test and Java code; it's just not the thing that you do. Third-party libraries,
we keep those out of the tree. And resources used for Android.
Okay. Get my drink of water. So it's all one tree.
We have more than one mobile app that we make for Android. There's Instagram in there, there's
a Facebook app, there's a messenger app, there's a camera app. You know, if you download an
app for Android from mobile -- from Facebook -- from mobile -- from Facebook, it's all
being built from that tree. Throw that all over my laptop later.
So -- Yes, people know. Murphy's law. So what does it look like when you begin working
on a feature? Well, what you do is you probably -- you're
probably on the stable branch. You create a new branch for your fancy feature, hackety
hack, hackety hack, hackety hack, you git, commit, and you create a new feature. While
you've been working, there have been "N" other developers also checking in code. So, periodically,
you want to grab the work they're doing and rebase.
So you're always working on the top of the tree.
You go hackety hack, hackety hack, hackety hack.
When you're ready, you commit. And then you do a rebase.
So what we like to do is, while you're developing locally on a branch of your own, you can have
as many branches as you want. You can have your commits all organized any way you want.
But when we push it into master, we want a logical change to go into our codebase. We
don't want the individual steps you did. Now, there's a number of reasons for doing
that. The first one, a logical change is a lot easier for other people to comprehend.
If you want people to understand what you're doing, what you need to do is explain why
you're doing it. That helps increase the quality of the code when someone comes and has a look
at things. And second of all, if we need to yank your
changes, it's a lot easier for us to do that if we can see the one logical change that
went in rather than the intermediate steps, particularly if you're like me and you work
on, like, two or three things at the same time.
So we always just squash our commits before we run this command, arc diff.
So how do we write all this code? We use IntelliJ. It's Java; right? So there's
two choices of IDE: Vi- -- No. Hang on. IntelliJ or Eclipse.
Both of them are perfectly reasonable choices. But the way that we've structured our codebase
means that IntelliJ is actually a really good fit for us. And from a massively biased point
of view, I prefer it. And Facebook is small enough that, actually, an individual choice
and someone advocating can really make a difference there.
So, you know, you're using IntelliJ. You want to make sure the code is good. You want to
make sure the code is testable. How do you make sure the code is testable?
You use what -- You use dependency injection. So when you're doing object-oriented -- or
object-oriented software, OO software, you've got collaborators; right? The class that you're
working on collaborates with a bunch of things. There are two ways that it can get hold of
its collaborators. The first way was popularized by J2E back in the late '90s, early 2000s,
and was service location. You had this big thing, and you went into it in the middle
of your method, and you went, "Thing, give me a database connection."
And thing would look at you and go, "Here's a database connection."
And that would be fine. And then sort of, I don't know, 2003, 2002,
early 2000s, this dependency injection phrase started becoming more popular. And that was
-- the major difference with that is that rather than having the method go, "Thing,
give me one of these," the method signature for the class under test somehow accepted
its collaborators from the thing that created it. So there's constructor-based dependency
injection, where all your collaborators are passed in at construction time. And there's
also setter-based dependency injection, also some magic-based dependency injection where
you annotate things and then Guava or Spring or whatever it is that you like suddenly populates
fields that you have no idea how it did it, but it's done it somehow. Right?
Now, the advantage with dependency injection over service location is it makes the collaborators
really, really obvious. It encourages you to think about who your class is collaborating
with. I remember I was on one project, not at Facebook,
and they were using service location. And I ripped out all the singletons and I added
every single thing they depended on to their constructor signature. And 14 arguments later,
somebody went, "That's a really big signature." It's like, "Yes. Yes, it is."
"Do you think we've got too many roles and responsibilities?"
"I don't know. We should probably slim that signature down somehow, but, yes, how are
we going to do that?" Thinking what we'll do is split the class up and we'll have fewer
responsibilities. And what they said to me is, they went, "Ah,
we'll pass on a map." [ Laughter ]
>>Simon Stewart: So, yes, dependency injection, a really, really useful way of ensuring that
your code is testable. It makes it really easy to swap out collaborators at test time.
And that's vital. So you've written your code in IntelliJ. That's
nice. You've used dependency injection, like a good developer should. You probably want
to build your code. Now, we use a tool called BUCK at Facebook
for building our code. Why do we use a tool that nobody else in the
world yet uses? There's a number of reasons for it. What we
want is, we wanted a tool that had minimal overhead when we added a new library or module
to the project. Our original Ant solution had a lot of boilerplate code. It had lots
of submodules. If you've used Maven, the way that you would
do a new module is probably you'd have a top-level library, and you would have some sort of palm
that would point to it, and who knows? We want to keep boilerplate out of our codebase.
Boilerplate is things that machines should do for us.
But we also wanted the build tool to be friendly to the IDEs. We had structured our code in
a way that actually plays really nicely with IntelliJ, and we don't want to lose that.
We want really, really fast, clean builds. But we want our incremental builds to be even
faster, because as was -- as Ari said right at the beginning of the day, doing a clean
is a bug in your build system. You should never, ever need to do that. So, yes, we want
faster incremental builds. And what we need to do is, we ahead of time don't know all
the things that people are going to try and build. So we want to support adding ad hoc
build steps. So BUCK, our build system, has three really
important concepts in the middle of it. The first is a build rule. Now, that's a procedure
for -- You can all read, can't you? Yeah, okay. It's a procedure for producing output
files from input files. You list a series of sources and dependencies and the build
rule runs, and you get the outputs. You get one output. That's the invariant to the system.
You can have at most one output from a build group.
A build file contains a series of these rules. And, typically, it's called BUCK, in all capitals,
which is great, unless you are on a case-insensitive file system.
So we declare all the rules in that. And if we want to have a dependency between two rules,
we use a label to get to the build rule, and we call that a build target. Okay? So that's
a string identifier for a build rule. This is what a target looks like. I like to
read the slash, slash as from the root of the repository. So from the top level, there
is a directory called Java/com/Facebook/share. Excellent. Colon. In which there is a BUCK
file, UI, with a build rule called UI. So this gives us a completely declarative path
to the thing that we're attempting to build, which is nice.
By the way, if you've ever used the Selenium build system, it uses exactly the same syntax
as that. It's called Crazy Fun. If you work for Twitter, pants uses, I think, the exact
same syntax. Is that right? >>> Yes.
>>Simon Stewart: Yes, it is. Brilliant. Good. I think it's because there is a sort of diaspora
of ex-Googlers who pine for the amazing build system that Ari talked about.
[ Applause ] >>Simon Stewart: So, yeah, what does a build
rule look like? [ Laughter ]
>>Simon Stewart: It looks like that. This is how we build a library for Android
that calls UI, the one that we had before. We do a recursive glob over the directory
structure underneath, where this rule is declared. So it's not over the entire tree; it's just
over the subtree and beneath this. We have a series of dependencies, which are
declared there. And visibility. So, by default, no one else can see a build rule outside of
the build file you're declared in. So you can have dependencies within a build file,
but if you step outside, you need to make it visible.
You can either make it public, anyone can use this thing, or you can limit the visibility
to certain subsets of rules. In order to make it nice and easy to have
a library that you can depend on in the rest of your code without needing to be aware of
how that library is structured, with a Java library, you can also export your dependencies.
So you can say, "Look, if you depend on me, then transitively, you get all my dependencies
as well," which is a nice way of composing everything.
Who knows Python? What language does that look like? It's a
leading question. It's Ruby. No. Hang on. It's Python.
Our build file is Python. We interpret the build file using the Python interpreter. We're
moving to Jython, because it would be nice not to have to rely on Python being installed
on the local system. But, yes. So it's Python. And that allows
us to do some really, really nice things. Like, if we're missing the ability to build
a certain type of target, you know, if a build rule should exist to build, you know, iOS
library, for example, we could add it to the system using a glob of rather unpleasant Python
and then just use an iOS library in the rest of our build rules. And then when we modify
BUCK, we can fix it, and we can bake that into the platform. And that's kind of nice.
So we've got this ability to modify the build system on the fly in a tiering complete language
in a completely declarative build system. I'm not sure how I feel about that, but it
is awfully convenient. So what we do, you run your builds. We build
a directed acyclic graph. I was really pleased when I started working on BUCK, because it
allowed me to use some of the computer science that I struggled through all the way through
university and to use terms like "directed acyclic graph" with a straight face.
And what we do is we analyze that graph and we build subparts of the tree in parallel.
So we try and build everything as massively parallel as we can. Right? And that enables
our builds to be significantly faster. And then because we know that if the inputs
to one of the previous steps haven't changed, the output won't have changed, we can skip
building that step. We can just use the cached output. And that allows our incremental builds
to be nice and fast. How does IntelliJ feel about us monkeying
around on the file system like this? Apparently, it likes it.
We run a command called BUCK Project. And what that will do is it will scan through
the entire file system for BUCK files, and it will generate the IntelliJ configs. So
what you do is you go, "I haven't got an IntelliJ config." BUCK Project points IntelliJ at it,
and suddenly everything works out quite nicely. That will also set up some exports for build
rules for test runners and things like that. So that's nice.
And it's fast, guys. This thing is quick. Because if you want people to write tests,
if you want them to be using dependency injection, if you want them to be doing all these things
that we want to encourage people to do, being slow doesn't help. Again, Ari's slide, the
developer who was sleeping, going, "Longer is better."
Longer isn't better. Shorter is better. Instant is best. Actually, if it would somehow leap
forward in time and give you the results before you finish writing the code, that would be
better still. But assuming we're constrained by the laws of physics, instant is best.
So BUCK, by default, spins up as many threads as it can 1.25 times the number of cores in
your machine as reported by Java. So if you've got hyperthreading enabled, on a modern laptop,
that means you've got ten parallel builders running, you know, four physical cores, eight
with hyperthreading times 1.25, ten. And we found that by doing this, we can slash
our build times. Our build time, when we were running in single-threaded mode, used to take
about, I don't know, 20 minutes. And we ran things in parallel, and we dropped that down
to four minutes. That's quite nice. It's an incremental system.
So we only rebuild what's necessary. There are going to be some changes that we're going
to make in the future, like at the moment, we need to scan the file system every time
we do a build to figure out what's changed. If we demonize something to watch the file
system, as in send it into a daemon, rather than just told it it was bad all the time.
We could go faster. And the other thing as well is, like, it would
be nice to be able to do a distributed build, and if someone's built the thing you need
with the same signature of the inputs, you should be able to get the same outputs, you
shouldn't need to do a build. So in 2012, what happened was, we flipped
everyone from using Ant to using BUCK. And I'm just going to hide that. Pull that
up to the front. Can we flip to the Mac, please.
Excellent. So there are two interesting things here.
I've checked out the source code for BUCK, and we build BUCK with Ant, because we're
insane. [ Laughter ]
>>Simon Stewart: You can see the activity monitor from OS X just down here. CPU usage
is the thing to look. What I'm going to do is I'm going to do -- let's
do a clean. Going to just time a clean build of BUCK, and then running all its tests. So
it's doing the compilation. Yep. Excellent. Now it's running the test cases.
It's still running the test cases. And it continues to run the test cases.
And we're not going to wait, because it takes about a minute and a bit to do everything
that it needs to do. So now what I'll do is I'll use BUCK to run
every single test in the file system. And to make it a fair comparison, I'll do a BUCK
clean. This will -- because I'm actually in the directory that BUCK is installed from,
we may see a bit of Ant (indiscernible) as it bootstraps itself, which should be fine.
But look at the CPU usage over here. What it's doing now is it's passing everything.
It's testing all the tests. It doesn't do a meme; right? And, bam, in the time it took
for me to describe what was going on, it's run every single test in the BUCK system.
And that took significantly less than the minute it would have taken if we were using
Ant. So that's pretty cool; right?
Can we go back to the slides, please. So at the time we did the switchover, it was
quicker to pop up a warning on the Ant build saying, Why don't you go and download it?
You could download BUCK, install it on the system, and do a complete build far more quickly
than you could actually do -- than if you finished your Ant build, which is crazy.
And the good news is, BUCK is open source. You can go and download it. You can go and
have a play with the build system that we use at Facebook. We accept pull requests.
Please send pull requests. They'd be lovely. It's pretty nice.
If you want to see the documentation, just head over to Facebook.gitHub.com and took
that at that user-friendly. It's under an Apache 2 license, so if you're working in
a bank, and you're worried about having to make all that source code available, don't
worry. It is written in Java, and a small amount of Python, which makes the Pythonists
happy. And it works on OS X and Linux. Now, the reason it doesn't work on Windows is because
nobody builds on Windows. [ Laughter ]
>>> We wish. >>Simon Stewart: Well, none of our team currently
build on Windows. I'm sure at some point there will be a need for it. That's a flippant comment,
and it was fine. And, of course, we build BUCK with Ant.
So, yes, we've got this amazingly fast build time; right? And that allows us to test. Testing
is good; right? We get our developers to write the tests.
At Facebook, there is no SET, no SDET, no TE, no QAE, nobody -- no test department,
no QA department. We just have engineers. Now, some of the engineers have a passion
for testing. In the same way that some engineers have a passion for databases, some have a
passion for security, and some of them have a passion for finding pictures of kittens
on the Internet and posting it to their time line.
[ Laughter ] >>Simon Stewart: But we get our developers
to write tests. Everyone who writes a line of code is expected to also be writing tests
for that line of code. Which is nice; right? Seems to be working out all right for us so
far. And I quite like it; right? I always find that companies that have a test team,
the test team tends to be looked down on and not treated with the love and respect that
they ought to be. Right? I remember -- [ Applause ]
>>Simon Stewart: Yeah! I love you guys. It's okay.
Yeah, I remember sort of getting interviews from people going, "Oh, he wasn't good enough
to be a coder, but he might be okay as a tester." It's like, "No, no, no, no. You've got it
all wrong. They're not good enough to be a tester. They might be all right as a coder."
And they just go -- It wasn't appreciated in the way I thought it would be. Funny that,
really. So what kind of tests do we ask people to
write? We start with unit tests. Unit tests run using
JUnit. Kent Beck works at Facebook. We're not using TestNG. Cedric doesn't work there;
Kent does. For those of you who don't know, Kent Beck wrote JUnit. Cedric wrote TestNG.
We encourage people to write on the local JVM, because we appreciate fast. Fast is really
important. Like, move fast, break tests. That isn't the Facebook mantra, but it almost is.
In order to enable people to move quickly, they're using dependency injection. We can
replace collaborators at test time using EasyMock. That's one of the libraries that we use. Mockito
is another choice. But we're building an Android app. And if
you include the Android jar from the Android development kit and you run a test using it,
what tends to happen is, anytime you touch anything included in that jar, you get an
exception with stub being the only message in that exception. And that's because what
they've done is they've provided an ABI-compatible jar with no implementations, because the implementation
runs a Dalvik on the device. That makes writing tests incredibly difficult.
That's a bad thing; right? We don't like that. So what we do is, we use a library called
Robolectric. Again, it's open source. And what that does is, it provides stub implementations
or fake implementations of each of the Android methods; right? So what we can do is we can
compile and test and run in the IDE on your desktop for really fast developer feedback
cycle. And that's nice, because there's no compilation to Dalvik, no pushing apps on
the machine, the device, or the emulator, and you can get that nice, fast cycle time.
And we appreciate fast. But unit tests don't cut the mustard; right?
You will occasionally want to make sure that all the unit tests function correctly and
the software when you plug everything together doesn't fall over.
I was on a project once which had slightly over two and a half thousand tests. It was
a .NET project. The tests ran in about two seconds. It was amazing. And the app would
consistently not start up. [ Laughter ]
>>Simon Stewart: And it's because they had forgotten to write integration tests and larger
tests, right, and they were just using mock objects and going, "Yeah, it will return null
here and throw an exception here for the exact same method."
No, guys. You need larger tests. Larger tests, there are a handful of frameworks
that you can use. UI automator, which came in API level 16, so, basically, Jelly Bean
and above. I think it's very nice, yes, but Jelly Bean and above, it starts becoming useful.
We use that quite a lot. We're writing our own test framework to allow
you to write tests out of process and out of device using UI automator. And we call
it Bully because it pushes things around. I may have to come out with a better name
before we open source it. Or maybe not. Who knows.
We also use Robolectric. So the older devices, there are tests using that. We use monkey
runner. We use a whole series of tools that come with the Android Development Kit, and
they're all quite nice. But Robolectric, UI automator, those are two of the key ones that
we use. By the way, we also run things like power
tests and -- on device tests and things like that. But these are tests that the standard,
normal developers write rather than crazy ones who really enjoy testing.
You've built the code. You've tested the code. You kind of think it works; right? So you're
ready. You're going to push it into master. So you remember way back in that black slide,
I was arc diff; right? What does arc diff actually do? It kicks off the code review
process; right? You write a description of what you do. You write a test plan. And if
the test plan goes (noise) or the written equivalent, you tend to be told off and told
to improve your test plan. And we send it up to a tool called Fabricator,
which is like a Swiss Army chain saw of project management. So it does tasks, it does code
reviews, it does browsing of repositories, allows you to do all sorts of clever, wonderful
things. And we like Fabricator at Facebook. This is what a code review looks like. You
can see here that the chap has submitted a reasonable test plan. It has been grayed out,
but it's in purple. The reviewer has come back and asked him to make some changes. The
developer has made some changes. That's what the little icon second from the bottom does.
And then the reviewer has gone, "This is good. You can check it in."
Now, -- [ Laughter ]
>>Simon Stewart: -- Fabricator has a killer feature, and I don't know if you can tell
what it is. It's image macros. In the middle of your code review, you can
just type a little line, and it will insert an image into the code review. This is American
Obama, which is President Obama playing an electric guitar shaped like a bald eagle while
wearing an American hat. I quite like that one. I stare at it endlessly.
It's fascinating. [ Laughter ]
>>Simon Stewart: And ship it. It's like, it's all done. Just ship it. Land the code. Do
something interesting. To land it, we use a command line tool, Arc,
short for Arcanist. It's the command line counterpart to Fabricator.
And landing does two things: It runs Lint, which can be run separately. And we use the
Android Linter with additional Lint tools that we've added to check for things like
null pointer exceptions. So we use @nullable quite a lot, from Guava. Optional as well.
The Guava library is fantastic. If you're not using it, just download it now and use
it. It's one of the best things to come out of Google for developers for a long, long
time. It's awesome. We also make sure that all the APIs for the
versions of Android that we support are all present, and other bits and pieces. And we
do an Arc land. What Arc land does is it takes the changes in your local branch and rebases
them onto master and then pushes your version of master into the central one.
So if you took a look at the history of the project in GIT on master, it's a single linear
history. We don't have weird branching. It's a straight down linear history of logical
changes. Right. So you've landed your code. You've
tested it. It's all wonderful. Right? No. Hang on. We also have a continuous build
system. If you haven't gotten a continuous build system yet, just use one. I don't care
what you use. Cruise Control, Go. We use Buildbot. And you can see here that most of the time
things are pretty good. One developer has caused the test to fail, and so we're not
going to move the stable branch. But when these tests all go green, then we go, "Excellent."
These are sort of longer-running tests. So these there integration tests, end-to-end
tests that we don't what a developer to run on their local workstation because there just
isn't time in the day. If you're not breaking the continuous build server periodically,
you're probably not working hard enough. One of my friends used to say -- I think he's
right, actually -- I think it's okay to break the build periodically as long as you fix
it immediately afterwards, possibly by yanking your change out, reworking it, and resubmitting
it. So Buildbot runs all these longer-running
tests, these (indiscernible) tests, makes sure there are no regressions in performance
or anything like that. We're done; right? Oh, crap.
Sorry, Asim, and you in the back. He betted that I wouldn't profane. Does that count as
profanity? I don't know. ***. There we go. Done it properly now.
[ Laughter ] >>Simon Stewart: So, yes, it's built. We need
to dogfood this thing. As I said before, when you get this application
on your phone, if it falls over, a user may never download it again. That would be an
awful, awful thing to happen. So we dogfood extensively.
One of the platforms that we make sure we dogfood on is Gingerbread.
Problem is, at Facebook, many of our developers are paid well enough to be able to afford
an ICS or Jelly Bean device. And we can ask for, like, a Galaxy Nexus for testing and
things like that. But Gingerbread is still incredibly popular.
They're still churning out devices with Gingerbread on it. These are the latest numbers from the
dashboard that Android have on their Web site. And you can see Gingerbread is about 40, 45%
of the market. So what we do is we make sure that people
are voluntarily using Gingerbread as their primary device and they're dogfooding our
application. So they install the beta builds on their device and they keep on using that.
One of the key people using Gingerbread as their day-to-day device is the release manager
for Android. He's not always the happiest person.
[ Laughter ] >>Simon Stewart: But he often flags up pretty
nasty regressions that no one else has caught yet, because they only show up as sort of
outer memory exceptions on Gingerbread, where other devices are more constrained.
So this is the PHP hammer, by the way. I don't know if you have seen it.
[ Laughter ] [ Applause ]
>>Simon Stewart: It has nothing to do with the talk. I just love the picture.
So you've downloaded the beta version of the app. You've side loaded it. What it does is
it periodically -- at Facebook, at least -- contacts the build services and goes, is there a new
version of the application? And if there is, it nags people to update it. So it downloads
it in the background and then goes, "Would you like to install it?" "Would you? Would
you? Would you?" And you go, "Yeah, all right."
And so once you're on the testing track at Facebook and you're always using the beta
build, you're always using the most recent beta build, which prevents sort of old bugs
being filed by people. And filing bugs is incredibly easy, because
what people do is they take their device and they shake it. Now, we call it a rage shake,
because you normally only shake your device when you're going, "It's not working!" Oh,
hello. I can file a bug. Excellent. And that's kind of a really nice feature;
right? So we get good, instant bug reports from people as they're experiencing it, using
the latest version of the application, which allows us to do our releases.
Releases. Never thought I'd get there, did you?
So you can pick two when you're doing a release. You can have features, quality, or regular
schedule. Wow. That's kind of a tough choice. If there are any project managers in the audience,
features, quality, schedule is not two. That's three.
So we decided that we were going to do time-based releases which are high quality. That means
that occasionally we ship without features that we've been working on. Right?
Now, the advantage of a time-based release, and our releases are about four weeks long.
So what we do is we cut a branch, we do stability work for about three weeks. We have a release
candidate for about a week, where we push it out to the beta testers and anyone in Facebook
who volunteers for it. And then we ship to the Play Store. So once a month, there's an
update to the Facebook application. And it's really nice having that regularity, because
it means if you're working on a feature and you missed this release, you know that you're
not going to have to wait forever before the next release. Right?
It means that you can get this stuff as quickly as possible.
And it means, you know, you've got that consistency, that regularity.
You also know that your feature isn't going to be blocked as somebody else tries to frantically
finish their thing. So time-based releases are hugely, hugely
valuable to us. And done isn't just, hey, it compiles and
the test passed and, you know, nobody's filed a bug with it yet in -- on the release candidates.
You know, done is things like, is the design work all finished? We see changes going into
the release branch around the design of a feature, then we know that the piece hasn't
been finished. If the feature hasn't been finished, we yank
it from the release. We either hide it behind a flag and strip it out using Dex, ProGuard
even, or, you know, we just remove the particular feature and we ask people to keep on working
on it. Logging. How do we know when we release a
feature that it isn't causing a million crashes? Right?
We get metrics and feedback and information from the versions of Facebook that you're
running on your phone. If something goes wrong, we get told about it somehow. Sometimes it
depends. You know, and we can look at that data and
we can go, oh, crikey, there's been a sudden spike in crashes on this particular type of
device. Let's fix that. Right? So we get information from the world. If you haven't put logging
in, we can't get that information. Sometimes there are service-side changes that
need to be made. We still use WebViews in a few places. So if you're pushing a new feature,
we need to make sure it works for the old and the new version of the app.
And, obviously, privacy and legal review are really, really important. Facebook cares about
privacy. Like, we're a social network. If people don't trust us with their data, we're
dead in the water; right? So we need to really, really be concerned with privacy, and we are.
And legal review, just in case. So they're just as important as the code review.
And that's what done means. So that's basically how we bake quality into
the Facebook application; right? We go all the way through. We run tests consistently.
We encourage people to use design patterns for testability. We have defense in depth
for the unit tests, integration tests, the continuous integration server. We have people
dogfooding all the time. We encourage people to use devices that aren't commonly used by
people who have got well-paid jobs but are used widely in the real world, particularly
in the smaller markets, the emerging markets. Great.
So thank you very much for your time. We've got time for some questions.
[ Applause ] >>Simon Stewart: Thank you.
>>Tony Voellm: Thank you. Thank you, Simon. That was fantastic. I really liked seeing
the end to end. We have a couple live questions. How about
we go to the left first. Go ahead and state your name, where you're from.
>>> Hi, my name is Christian from Google. I actually work on the Guava team.
>>Simon Stewart: Awesome work. >>> Yeah. Well, that's more my other teammates,
but -- >>Simon Stewart: The JDK5 back port. Fantastic.
>>> Working on that today. >>Simon Stewart: Ship it!
>>> But -- So I wanted to thank you for some really innovative work on build systems.
[ Laughter ] >>> Putting that out to the community is a
really awesome thing, so I'm glad to see it. I had a question about BUCK, particularly
because not all companies have access to all the source code they will ever build, so how
do you handle binary dependencies of things that are, you know, third-party libraries
-- >>Simon Stewart: Yeah, there's a rule called
prebuilt jar where you can just check in. So we've got third-party dependencies. We
don't build Guava from source, for example. We just check in the jars.
>>> Okay; you check in jars. >>Simon Stewart: Yeah, we check in the jars.
We have prebuilt binary rules and you can depend on those. The binary rules allow you
to also attach source jars and Java dot jars, and if you do that, when you run BUCK project
it will actually hook in the sources so that you can browse into the source code while
you're working on your project. >>> Very sweet.
And obviously you have an Eclipse plug-in coming soon; right?
>>Simon Stewart: We accept pull requests. [ Laughter ]
>>> All right. There you go. >>Tony Voellm: Okay. Looks like we have time
for one more question, so we'll go over here. >>> I'm Dave. I'm from Family Search.
So you mentioned that the code's in a single tree. Is that like all of the code at Facebook
or is it just the code for Android? And the other question that kind of goes along with
it is does that mean it's all in one git repo and how does that perform?
>>Simon Stewart: So at the moment the Android code is in one repo, the iOS code is in another
and www the Web site is in a third repo. I think, you know, Google have demonstrated
you can smoosh everything together and that works pretty well for Google. I mean, we saw
all the stats earlier which were amazing; right? Just totally mind blowing.
So we could do that way, but at the moment we have separate repos for the various things.
But all the Android code lives in one tree. Cool.
>>Tony Voellm: Actually, maybe we'll just grab one more, and then I have some announcements
about dinner and things like that right after this.
So go ahead. Last question, please. >>> Noah Sussman. I am an independent consultant
but formerly of Etsy where I designed the CI system with precommit because it works
so awesomely for, like, Google and Mozilla, and so why not? Why no precommit? I'm fascinated.
>>Simon Stewart: Just because we haven't had the need for it.
The other thing as well is sort of we're a relatively small company. Like I know 1400
engineers sounds like an awful lot, and that's 1400 engineers spread over the Web, the iOS
app, the Android app, working on our internal tooling, working on stuff for our data centers.
Just, you know, everything; right? Facebook does more than just the news feed; right?
We've got messaging infrastructure, we've got groups, we've got all sort of fun things.
So 1400 engineers isn't actually that many. The team working on BUCK is -- there's about
three consistent contributors, and then there are other people who join the team for a month
or two and then they roll back out again. So, you know, there just isn't a huge amount
of manpower. And it turns out that if you run all the unit
tests before you check in, that catches most of the problems. So there hasn't been a sort
of pressing need to have a presubmit thing. When there is a need, then I think we'll do
it. But YAGNI is a really good mentor, right?
If you've not heard of YAGNI, it comes from the Agile community. You Ain't Gonna Need
It. So don't write it, don't use it until you actually have a need for it, right? And
then put it in. So, yeah, that's why we don't do it.
>>Tony Voellm: With that, thank you, Simon. Hopefully you will join us for dinner.
>>Simon Stewart: Thank you very much, guys. [ Applause ]