Tip:
Highlight text to annotate it
X
[applause]
Reid Burke: Hi. I'm Reid Burke, and I've been on the YUI team for a little over two years.
I spend a lot of time working on testing. This talk is about writing code that works,
but before I get into exactly what that means, this talk is mostly about testing software.
When you think about what the purpose of testing is though, I think that the title, instead
of having the title of this talk be something about Yeti or be something about testing software,
I thought it would make more sense that for what testing is actually all about.
To that end, I work... This is actually the alternate title that I was also considering
besides calling it something about Yeti, was when not to write tests. Because as I'm going
to explain later, I think testing is something that is a tool. I'll get to explain more about
that. It's a tool because your time is limited, and that's something to think about through
the rest of this talk is that I'm going to talk about tools, I'm going to talk about...
For folks if you're new to testing or if you're already using tests as a part of your development
strategy, for ways to improve that and to continue doing that. All this stuff is all
supposed to help you write better software, and it's supposed to help you make the best
use of your time because it is limited. I'll get to explain more about that in this talk.
Mostly what I've been doing for the last year is writing and improving Yeti. Yeti is something
that we use at YUI to test it before we release it. This is a project that's been very old.
In fact I talked about this two YUIConfs ago, back when it was a much simpler tool. I'll
get to show you how it's changed in the last two years, the different things that we're
hoping to do now with Yeti that wasn't really part of it when I first wrote it. Also an
appeal for if this kind of thing excites you then for how you can help and help make it
better.
The whole point of Yeti, if you're not familiar, is it's very simple, it's simply to test JavaScript
with any browser. It's not for testing JavaScript on the server or testing server-side JavaScript.
It's made so that... Or it's not even for testing JavaScript powered apps, which are
actually pretty neat. If you've ever used Windows 8 or if you're ever familiar with
how either writing an app like that or writing an app with Appcelerator or any of the hybrid
kind of technologies that allow you to take JavaScript and write it into an application,
that's pretty neat.
What this is for is for web applications. It's trying to do that one thing really well,
and it's trying to do it in a way that isn't quite like the other stuff that we found that
was available for us for testing, including what we were using before.
The biggest thing that's different from what we were using before with Selenium was that
it required a driver. What that basically is is say if you wanted to test something
on your computer, then you would have to have a program that would automate that browser,
so you would need another program, the Selenium driver or web driver], to open up a browser
and to then basically open the test that you want to do and then collect the results and
then move on to the next test. This is all done outside of the browser instead of inside
of the browser.
Something that's problematic is for testing devices. We have all these devices and if
we wanted to write a driver that's possible, but we didn't think that was the best approach,
and I'll get to explain that in a little bit. Basically all the things that we want to do
with Yeti as compared to Selenium is possible, and that's why we chose to do it.
We also chose to do it, or I chose to do it, with Node. Node makes this very easy for me
to do, and I'm hoping it makes it easy for you, the users of Yeti, because it's what
all of our YUI tools for the last year or so have all been built on top of Node, which
is very easy to install nowadays on the three major platforms, desktop computing platforms.
You just get an installer and then you're good to go with a command line tool that easily
lets you install all of YUI's build tools including Yeti if you choose to use it by
itself.
Of course the big thing about Yeti is that it's trying to do one thing well, and that's
in contrast to something like Yogi which is supposed to be the interface that's consistent
that lets you build YUI applications. Yeti is a part of Yogi. The things that I'm talking
about this are just a part of what we're doing with our YUI tools. If you're using Yogi you
get to benefit from a lot of this as a YUI contributor, which is pretty neat. Node's
also just really well suited for orchestrating the interaction between all these different
devices that we have, which I'll get to explain more about later.
How we use this is for right now we use it before releases. Historically we've used it
to test while we're in development, and I'll get to show that to you in just a minute.
What we're hoping to do in the future is actually to have this a part of our CI system. This
is a very ambitious thing. It's something that I'm working on. What the bulk of my work
until the end of the year is going to be on is getting this a part of what we do to test
YUI in an automated way. I'll show you some of the progress that we've made towards that.
To reiterate some things about Selenium, the really big things that we've been focusing
on is testing on real devices. When you think about how with Selenium you have, on the iOS
at least, people have wrote these drivers that are iPhone apps. You as an organization
can take the web driver iOS app from the web driver project, or the Selenium project, and
you can throw that, if you have a developer account that costs $100, you can then throw
that on an iPhone. Now my iPhone is now has, say, this app that I can launch web driver.
I can launch that and then I could say connect this to my web driver hub, or my web driver
server, and then it can listen for tests and then run them on this device.
That seems okay, but then you can also do that for Android too, so that's an option.
Most of these devices are covered. When it comes to this guy, this surface, you're kind
of screwed. There's no driver for it yet. What we want to do is to make sure even if
we're not supporting each environment, we want to make sure that we can test on each
environment to understand and have our people who use the library, even if it's not something
that we support, for them to understand how their code works on whatever they decide to
support.
A really big push in the recent history of YUI is that we're encouraging for folks to
test on different devices, not necessarily to have the same experience on a lot of different
devices. It's really good for us to understand. Even if this isn't on our supported devices
list yet, with every release are we introducing regressions, what kind of problems are we
seeing with devices like these?
We don't want to have to write any more code than we have to, and it turns out with Selenium
a lot of times the kinds of things that we care about for automating tests are just a
very small subset of what Selenium does. Selenium can take screenshots, it can manipulate form
controls. The only thing we care about is getting the test results from a test. That's
pretty much it. It turns out you can do all that in the browser.
We're using... To the actual browser. That's kind of a pain point for this guy. If you
are writing an iOS app that is automating with Selenium, the problem with that is you're
using a UI web view. UI web views are great, you should be testing on them too. The problem
is that it's not exactly the same as, and you're not going to get the same results as
Safari. It's using, for security reasons it uses a different way of running JavaScript.
It actually is using a different engine. That's something to keep in mind, and something that
we wanted to just eliminate altogether by running on the actual browsers that folks
are using and not something that is just built on top of WebKit or something that's an approximation
of it.
This is to compliment, by the way, what we do already with Grover and with PhantomJS.
Those things are really great for smoke testing. It's something that makes it really easy for
you without doing any set up at all. This, I think, is pretty easy, and I'll get to show
that. But that still is something that we use when we're iterating or working on things
and even testing in Travis. Our ultimate goal is to provide things like this where we actually
don't need that, or this gets so good that we can test on real devices instead of having
to use something that's a little bit faster but isn't what our users actually use.
I present to you the Dav Glass Device Laboratory of Southern Illinois.
[laughter]
This is our QA lab. I have this iPad and this iPhone, but everything else... And this isn't
even all the things that Dav has. When I was looking at Yeti and really working on Yeti
earlier this year, I was thinking about how a lot of the people on our team, even though
we can't buy tons of devices... Like most folks on our team do have these devices, and
most people in the community have devices like this, so if I could make a tool that
would make it easy for folks to contribute these devices and to... If all you needed
to do is spend a couple of minute setting up the device to never sleep and then go have
it idle on a website, that would be really neat. That's kind of a really big long term
goal that I would like to have for Yeti.
Shorter term though, what we had was Dav had all these devices and he's in Southern Illinois.
We work from the communities everywhere. Some of us are in Sunnyvale but there's some of
us that are all over the place, but we wanted to be able to use these devices on our computers
and have it work like magic. That's kind of what Yeti will get us to do. Without talking
too much about it I'd like to present some of this stuff to you.
What I have here is all these devices. They are all connected through a WiFi network to
this computer. This computer is running a Yeti server. All I had to do is yeti --server.
If you're trying to install Yeti, that's something you do with NPM, npm install yeti. You get
the Yeti command line, start of server.
Then it gives me two helpful URLs. One of them is something that shows up for just convenience,
like I can launch a browser just by going to local host and then open selection. That
makes it handy for starting stuff on my local machine. Here I have it as a local area network
of devices. Then this could ultimately be put through... You could put a hole through
your firewall and then get all this stuff on the public internet as well, and that's
what's at hub.davglass.com and the unofficial thing that you see inside of Yeti.
When Dav was trying to use that too, it turns out Yeti in order to use it requires WebSockets
right now. We're actually, the YUIConf network is behind a proxy that strips the upgrade
header of HTTP requests which is kind of unfortunate. But that's something that we can definitely
improve on.
We have this yeti --server. Let's see. Now you're pretty much able to run tests. If I
wanted to run something, say like widgets test, then I can just run Yeti and that command
is actually all that I needed to do. What it's doing right now is there are 7 tests
inside of this component. You could see that I wanted to make it as big as possible for
you to see, but it shows all the different browsers that are connected.
Some of them you see are duplicates. Right in the middle of the screen you'll see agent
connected Safari 6.0 twice. What's actually happening is that when you connect two of
the same browser together they can share the load. What's happening is that this device
is iOS 6, Safari 6, and then so is the iPad over there. What happened is they had 7 HTML
files to run and each of them probably, one of them ran 3 and the other one ran 4, so
that's pretty neat.
We have in 15 seconds, across all of these devices, I know what my component is like.
I can do this with any component I'd like. This one's just one, so will be pretty simple.
You can see all of them work.
While that's happening I'm going to go and show you what some of this stuff looks like
for you in the back. It's just basically like you can see that changing, hopefully. This
is an app framework test, so it's going through everything. You can see there's a failure
up there and now it's returning the capture page. If I go back over here... Let's see.
I'm sorry? Oh yeah. Everybody, Dav.
[applause]
Up here you can see the failure that you saw on the Windows 8 tablet is showing up on here.
So that's pretty neat. This ran 2,130 tests, so I'll run it again because Dav's standing
there now. Let me switch over to the Yeti TV.
What's happening here is basically this stuff is getting served from the Node.js server
that you saw in the first terminal. That server, it is on the same computer but it doesn't
have to be. In Dav's case he's actually running it on a computer that just acts as a server.
So yeah, this is complete and we see that. I can run other things while Dav's up here.
Keep him doing stuff. Let's see, app. What's another component that I should test guys?
Anim. Do them all?
Dav Glass: Do them all.
Reid: I can do that. Anim will take a while. Anim's pretty neat to see because it shows
the animations. I'm sorry?
Dav: Switch the feed over.
Reid: Oh yeah, I'm sorry. Cool. So there is the animation. You can see some tablets are
better than others at rendering some of this stuff. It's pretty neat to watch.
What's happening is that all of this is getting served from your computer, so your computer
is submitting these tests to Yeti's server that's somewhere else, and then these devices
are connected, I have a persistent connection to that Yeti server. What's really neat is
that you can share these devices. They're all connected to, say, one hub, that one Yeti
server. Then everyone on your team can then connect to this Yeti server submitting tests
to it and can just use all these devices without any more set up. As you can see, every time
I finish running tests, all these are returning back to a page that then waits for more tests.
Yeah, basically once that's done I can continue and do more things. I will test everything
I believe. Go crazy. Yeah, that'll take a long time. There are 1,078 HTML files in the
library. I'll switch over to that. This will take a while, this might take the rest of
my talk, so I'll have this running in the background.
[laughter]
While I do the rest of my talk. But yeah, this is pretty neat. You can do a lot with
this. It still has a long way to go for getting through everything that we want to do with
it, but as you can see it's pretty capable. You'll see when I return later in the talk
to the results that come back from all these devices, you'll see all the results and see
yeah, it actually gives us really good information on... You'll see quite a few test failures.
This is actually running the absolute latest up-to-date YUI3 code on master. The big thing
that I noticed was how we have a lot of.... Yeah, it's still running. You can see this
here. I'll switch back so you can see what this looks like as results come in.
Yeah, you see this at the bottom. You'll see this kind of bar that updates at the bottom.
That's telling you how fast... The beats per second is pretty much every time that there
is a test function or test method that either passes or fails. This isn't counting assertions,
it's counting each test method. That can vary by framework. I actually, later on in the
talk if someone reminds me I'll run the JQuery test framework as well and you can see that
that also doesn't quite correspond to assertions or whatever, so I call it beats. But it gives
you an idea of how fast things are running. Right now there are a lot of tests here that
might be slower or faster in different phases, so it gives you an idea of how quick things
are rolling.
Anyway. I talked about some of these things already. Sharing browsers and devices on your
team. The other thing that I don't have that's kind of harder to demo, but it's still really
neat, is that you can add more browsers. More of the same tablets means more of the same
software means that your tests will run faster. If you split them up into separate HTML file
compartments for your test to run in, if you take advantage of that that can be really
useful.
It works with Jenkins. We actually, a few weeks ago I did work for the first time to
kind of get this integrated into our CI system. It still has some issues that I want to work
out. But that is possible, and it does work.
It works with QUnit. I mentioned I'm going to run JQuery later on if folks remind me.
It works with YUI Test. I've actually heard a lot of interest from folks that say hey
I use Jasmine, or I want to add test framework X, and that's great, awesome. It's pretty
easy to do that now, and I'm wanting to make it so that you don't even have to submit a
pull request, it's just something that you can plug into Yeti. You could just write,
if you want to use a different code coverage instrument or whatever, you can do that.
I release Yeti every week unless I go on vacation or I'm doing YUIConf, because it's just me
right now. But that goes to the next point: there's a contributing document in the Yeti
repository and it's pretty clear, and right now though there's only four total contributors
in the three year history of Yeti. One was a one-line change by a guy which I'm really
thankful for. Another one was our intern, Clarence. And then Dav, and then me. Yeah,
we could definitely use more help. If this excites you then everything you need to do
is in there, and I would love to have everyone work on this. It's just fantastic.
So anyway, while this stuff runs right now... Let's see. Yeah, we're at the DOM test so
this'll take a while. Think about it though, you're not working on YUI perhaps, or you're
maybe not doing this level of testing, but if you're in this room you're building something,
you're building this stuff with code. What does that code do? Well, we're writing tests
but we want it to work. How does it work? Well, even though we have all these devices
that are running our tests and most of them are passing, just because that's happening
doesn't mean our code works. You don't make your code work just by writing tests. Testing
really only helps you. Just because your code is tested doesn't mean it actually does what
the people using your code expect it to do.
That reminds me of this quote that was written long before I started programing but it still
holds true today, that no matter how much testing you do, no matter how sexy your test
framework is or the number of tests that run or how fast they run, the only thing you can
be assured of is that it shows you that a bug exists. But if everything's green at the
end of the day, that doesn't mean that you don't have bugs.
When you think about it, you get to define what bugs are, right? We on YUI have all this
documentation, we have API documentation, people file bugs when our documentation doesn't
match up with people's expectations. But testing really only gets you to kind of help you verify
that, but it isn't the verification in and of itself because you're ultimately defining
what works. And then also testing can only show you that there's a problem with it, not
that there is no problem at all.
It's a tool, but the goal is to have people use your code that aligns with their expectations
of it. You should be focused on setting up that expectation and also thinking about how
testing works with that, and not get into the trend of thinking like oh my gosh, this
is awesome that I can run all these tests, I can see that everything works, or that everything
is fine because all my tests are passing. Because you can actually write tests that
are really terrible and you can make your numbers look good, you can have a green dot
on travisci.org, you can have 90 per cent, 100 per cent line coverage, but that isn't
telling the whole story.
Some quick antipatterns. Just because you wrote tests first doesn't mean you have working
code. Just because you have a lot of them doesn't mean your code works. Even 100 per
cent line coverage doesn't tell the story for branches, for statements. It doesn't even
tell you if, like I said before, if it's fulfilling what the expectations of... You could write
something that has all the wrong tests or has tests that don't really help you find
something when something goes wrong. But hey, this really does help, so you should be using
Istanbul and Yogi to help you with getting code coverage.
You think about it, the big takeaway from this section is that you are not paid to write
tests. If you don't want to listen to me you can listen to a really smart guy, and this
really smart guy said this: I get paid for code that works, not for tests, so my philosophy
is to test as little as possible to reach a given level of confidence. If I don't typically
make a kind of mistake, like setting the wrong variables in a constructor, I don't test for
it. When coding on a team, I modify my strategy to carefully test code that we collectively
tend to get wrong.
This was written by a guy named Kent Beck. You can be forgiven if you don't know who
he is. What's interesting about Kent Beck is he's been probably the guy most responsible
for advancing the principle of test driven development, which I was really surprised
to read about and learn more about.
This guy has talked a lot about the benefits of writing tests first. But then you hear
this from this guy and he wrote this really great stack overflow comment a couple of years
ago. It's still gotten a lot of attention, a lot of views, even though this has been
locked down so no one keeps voting this up anymore. I love the first comment that says
the world does not think that Kent Beck would say this. There are legions of developers
dutifully pursuing 100 per cent coverage because they think it's what Kent Beck would do. I
have told many... And he goes on.
You always hear test first as a religion. That kind of goes into what we learned in
why not to use YUI, or rather when not to use YUI. When saying that you shouldn't look
at any part of our project as a religion, and I think Ryan had a really good point when
he was saying that. I think that that's true. All this stuff is kind of used to showcase
that. This stuff is a tool. YUI is a tool that helps you get to what your users want.
So is testing. I would encourage you all to think about it that way.
Kent was also one of the 17 original signatories of the Agile Manifesto. I had to look that
up because I kept hearing about Agile and I wonder what that is. If you're like me and
you don't know what the heck I'm talking about, I think it's interesting when I went back
and read it. These guys all met in Utah in 2001 and they all came up with this statement
that they all agreed on. You can take it or leave it, but I think it's interesting to
think about because we hear this all the time.
We are uncovering better ways of developing software by doing it and helping others do
it. Through this work we have come to value: individuals and interactions over processes
and tools, working software over comprehensive documentation, customer collaboration over
contract negotiation, and responding to change over following a plan. That is, while there's
value in the items on the right, we value the items on the left more. He's saying that
all these things are important. This group of people are saying all these things are
kind of important.
Some of these you might think don't apply or they're kind of enterprise-y sounding.
But when you think about it, especially the first two points, individuals and interactions
over processes and tools, you need process when you're dealing with a community this
big. You need something that says who gets to commit, who gets to review, who is going
to review your pull request. But ultimately what's that for? It's for individuals and
how they interact with our code. That's the purpose of that.
There's value to both sides. There's value to comprehensive documentation, because it
lets people at scale know what the expectation and what your software will do. It's very,
very important. It's very important that you speak that really clearly. But what's most
important is that your code works. Instead of being in a meeting all day talking about
it, instead of me standing up here all day and just talking about how awesome this is,
I should be focusing on making this work really well. That's what I hope to do.
The restate this simply, working software is better than comprehensive docs. Working
software is better than just talking about it, or even metrics about it, and that's where
testing comes in. Also just having green lights show up on your CI system does not mean that
you have working software.
Going back to the concept that you can do too much with this, or even just introducing
that I suppose. You think about it, like I mentioned in the beginning of the talk, your
time is very limited. That's kind of the first thing that you can think of, is that your
time is limited so you want to make the best use of the time you spend testing. That's
important. But then say you had all the time in the world to test. If that isn't a problem,
and you can just say well we're going to take half of our team and get 90, 100 per cent
line coverage in the next month and we're going to stop everything and do it, and you're
going to do it right and everything, you could still take this too far. Because even if you
do that, I would say that you're still missing out on things.
But I think this can be illustrated better than an explanation that I can make, and I
think it can relate to a lot of you. Who traveled here to YUIConf? A lot of you. How many of
you traveled by plane, or traveled by plane at all this year? You've probably dealt with
how our popular or infamous Government organization deals with trying to verify that you're safe.
They've done a lot of work in testing because what they're trying to do is make sure that
you're safe, and that's a very important thing. I want to make sure that I'm safe when I'm
traveling home to see my family.
But the way that they've done this is kind of forgetting that when they find something
it shows that there was a threat, there was a knife, there was someone trying to get a
bomb on a plane or what have you. But just because they don't find something doesn't
mean it's not there. They take this to a point where they do a lot and they find more and
more, and it's kind of what encourages this feedback loop of doing more to detect and
more to find things.
But when you get to the point where you have folks being put through conditions that are
harming the people who are meant to be traveling, you're starting to get to the point where
you're not actually doing what you originally set out to do. You're getting to a point where
you have folks who are now more inconvenienced by the process of making sure that they're
safe then what they would have been by maybe using a more reasonable approach.
We've all experienced this, and this is something that I thought was explained really well on
the Signal vs. Noise blog on 37signals.com about how when you first discover test driven
development, you kind of get into feeling like you should be testing everything. I don't
think that's really... This really stood out to me when I read it, and is something that
can really be used by folks to write better software when they think about it this way.
Because whenever you're writing tests, just like whenever the TSA is implementing something
new in how they verify the security for folks who are coming in and screening, it has a
cost associated with it. It takes time, it's inconvenient, it adds more layers. This is
all things that you should be thinking about. When we're talking about code though, it takes
time to write these tests, to update them whenever the code that's behind them changes,
to read and understand this stuff, and it also takes time to actually run them.
When you think about it, it's just kind of... Even if you had all the time in the world
to do this stuff, it's still something that would concern me because this is time I would
probably say is better spent working on features that will make sure that your code works in
delivering more for the people who use your code.
In the Signal vs. Noise blog David talks about quality theater. With things that we started
to care a lot about, and this is just things to think about. Total test count, line coverage,
branch coverage, statement coverage, speed of test runs. Those are all great things,
but we forgot that this stuff, there's a lot of it, there could be too much test code.
Right now we still have browsers that are still running through our test suite. We have
great coverage, that's good. I would say that this stuff is important considering how important
YUI is, or the framework, and how the things that we should have in the core long term
should be absolutely necessary and things in there. When it takes a long time to write
I can make a tool that helps us get that tested faster, but ultimately it might make more
sense to us to have less code in our core and by nature of that, by extension, less
tests that we have to run.
Then high coverage by useless tests. You can get 100 per cent line coverage pretty simply
but still have the wrong kinds of tests. That's something to think about. So there's a quick
checklist that you can take with you and fold in your pocket and take out next time you're
writing code.
Consider how much code you have to how much tests you have. I don't know. I would think
if code was one and tests was two, you're good. If it starts getting to three or even
higher than that, I would start caring a little bit more about it. Not that that doesn't make
sense, but you should probably start looking into it more for why you're doing that in
your app.
You should also be testing the smallest possible thing first. I said this instead of saying
how it was told to me when I first was learning testing from people who I thought were really
smart, and that was that you should do unit tests first and then integration tests. But
that's basically the plain English way to say it, is to test the smallest possible thing
you can before moving on to things that work together. Whether or not that winds up fitting
into what your test framework of choice calls a unit test or not, that doesn't really matter.
You should just try to test the smallest thing first because the benefit you get from that
is that whenever that test will fail later on, it will show you exactly where the failure
is and not just this subsystem failed somewhere but you don't know where that failure is.
Reserve the kinds of things that test subsystems or things that work together for the things
that test a whole bunch of moving parts together for after you get the smallest thing tested.
Then if you're spending too much time writing tests... If you're spending more than a third
or even half, I would say probably about half your time writing tests for a feature or a
bug fix or something, you may be okay. That might be okay, but I would encourage you then
to spend time and look at how other people have solved the problem that you're trying
to solve to see if you could make that faster, make that easier on yourself. There could
be a tool out there that makes testing stuff a lot easier. I would encourage you to look
at that if you're spending too much time writing tests or doing a lot of boilerplate. Finding
a way to make that easier will make your testing time overall much more effective.
Consider the metrics that you don't see when you're running things in your test runner.
Whenever you see how much tests you have and everything, that's great, but just consider
that you're also spending a lot of time not just writing them but the time that it takes
to run them and to understand them and update them later on.
The other thing is we may have forgotten about our own productivity. In making tests more
like a religion, or more like rather something that you adhere to at all costs, putting that
above all things, you can forget about why you do it, which I've talked about a little
bit. But also we almost forgot about what the benefits are for ourselves for not putting
this up as first and most important thing.
Kent Beck and his stack overflow answer went on to say a few things. Different people have
different testing strategies based on this philosophy, but it seems reasonable to me,
given the immature state of understanding how tests can best fit into the inner loop
of coding. Ten or twenty years from now we'll likely have a more universal theory of which
tests to write, which tests not to write, and how to tell the difference. In the meantime,
experimentation seems in order.
Test driven development is for experimenting. Just because it's a good idea, it doesn't
mean you should use it all the time. In fact I actually spendů Most of the time when I
do test driven development I'm doing that in response to fixing a bug. It's because
that tends to make sense, it helps you understand the bug when you write the test, and now you
know that it's not meeting expectations. But just like Agile is not just something fixed,
it's just people met up at a resort in Utah and they decided these things are important,
and a lot of people can agree with them, but that doesn't mean that you have to write software
in a certain way to do it. It is a kind of experimental thing.
So is your creative process. I would encourage you all to think about that when thinking
about how to apply testing with your code, because you do it to make great code.
The question I get after all this when I've explained Yeti and showed that off is, well,
we have limited time but we still want to test, so what should we test and what order
should we do it in? Should we stop everything and do it? It's a really good question to
ask.
You should prioritize what you do. That's because you can't do everything all at once,
nor do I think you should. I don't think it's wise for a team to stop doing everything just
as the knee-jerk reaction to we don't have tests then obviously we must stop everything
and write tests. Not necessarily, but if you are going to do that, or even if you don't,
you should be thinking about what order you should do those things in. I think a good
question to ask for that is what code is changing the most. Because if the coding is changing
quite a bit but you don't have tests for it yet, it may not be... It depends on your application,
but that could be a factor in what you decide to test first.
Something that isn't related to testing, but I thought would illustrate my next point better,
is semantic versioning. When you think about what code's changing you could think about
it as how stable is this section of code, and how relied upon is it. This isn't really
saying that you should use semantic versioning and that helps you, but rather it's kind of
an interesting idea.
If you just take away the human readable part of it and just think about when you look at
version numbers, which is kind of this, but you can look at the version numbers that you
have X, Y, and Z, this lets you know that if I look at this project I can see that this
is all the releases it ever had. I could tell you that it was kind of experimental for some
time and then it became relied upon, and then they made a breaking change. But ultimately
depending on the time it took between these things, this gives me an idea of how stable
this project is or how relied upon it is or how fast it's changing. If I see the X number
change too much over time, it could be that this is something that people keep breaking
compatibility all the time, or they could be doing that over a period of years which
makes that more reasonable because software's always changing.
What we need for our task of testing your own project is something that isn't at the
project level but at the section level inside of your project. We need a way for you to
think about how to categorize each section of your code. When I looked at that and how
to think about that, I looked at Node and I saw that they did a really good job of this
in their documentation because every part of their documentation has been covered by
something they call the stability index, the Node.js stability index.
It accomplishes that goal by doing this. This is straight from their docs. Throughout the
documentation you will see indications of a section's stability. The Node.js API is
still somewhat changing, and as it matures, certain parts are more reliable than others.
Some are so proven, and so relied upon, that they are unlikely to ever change at all. Others
are brand new and experimental, or known to be hazardous and in the process of being redesigned.
What this answers that doesn't help you but helps people who use your code is: how reliable
is this? Because really, even though you can use something like symver or even just well-written
documentation or a blog post or read mes or what have you that give people the right expectations
for how stable your code is, I think that also makes sense for you to do that at a section
level, and to tell people that at each section. When you make that public it gives other people
an idea of how reliable your software is.
But for you, what that gives you is if you stop and take a look at all of your code it
helps you prioritize how you write your tests. This is what Node.js uses. You can invent
your own labels, but I thought they were pretty good. Deprecated code is code that you shouldn't
bother writing tests for at all because the code's about to go away. Whereas locked is
it's so important that no change to this is even planned. API frozen means none without
good reason, and so on. I'm not going to spend too much time on this, but you can read more
about that at the Node.js website and check that out.
I think it's really useful for you to find out and take a good look at where things are
moving quickly in your code and come up with what's more important, and maybe what deserves
more test coverage when you're starting from scratch.
If you don't do that then you're kind of liable then to go and look at and maybe write tests
for things that you most recently wrote, and that may not be where you get the most benefit.
If you're writing tests for experimental stuff when you have poor coverage and the stable
API frozen kind of stuff that's more relied upon, you may not be using your time as wisely
as you can. And like I said, your time is limited.
Going back to Kent Beck and thinking about how to test what matters. His philosophy is
to test as little as possible to reach a given level of confidence. That's kind of the takeaway,
is you want to use testing but you want it just enough that it gives you the confidence
level that you need. If you don't make a kind of mistake then he doesn't test for it. But
this is important, when he's on a team he doesn't do that anymore. He said he modifies
that strategy to carefully test code that as a collective they tend to get wrong. That's
something too, that you can kind of take away is when you're thinking about what kind of
tests to write, if you see people constantly making the same kind of mistake, that's something
you probably want to write tests for.
I'm going to now return back to the Dav Glass Device Laboratory of Southern Illinois which
is now visiting here and check out how we're doing. Let's take a look. Still running, but
we had quite a few failures. That can scroll up here. This gives you just an idea of that.
We don't normally test on this many devices, and this is testing with everything and on
our latest code.
Some of the stuff that we have, our failures are like... Some of these are things that
we look at this kind of thing for scollInfo and it's on the default Android browser, not
Chrome but the default one, and that's kind of going away too. You think about, this is
not something we should be focusing on when thinking about what my priorities should be.
Because we have a lot of these failures, you can see that there are a lot of things that
aren't working.
Some of these things are kind of caused by things that we may not want to fix. We have
this failure here, or rather there was another one that was on Safari 17 where it's failing
on a lot of different kinds of tests, but that isn't something that we... We might just
run this and then understand that Firefox 17 isn't really well behaved, but it's good
for us to understand that and for us to know about these kinds of things.
Anyway, everything has finished except for this Asus. Well, now it's finished though.
I can return back to it and see if we get a result at the end. Yeah, there you go. In
20 minutes 35 seconds we tested 38,137 tests. I think that's pretty cool. We only had 207
of those fail. I think that's pretty neat.
[applause]
Yeah, thank you.
[applause]
For fun I'm going to run JQuery to finish things out. Cool. Oh yeah, thank you Dav.
Let's go back to the... This is the JQuery test suite. Something to keep in mind too
if you decide to use Yeti is how it has... Yeti was kind of made for YUI. We decided
to use HTML files that were pretty small and we wanted to break those out into HTML files
that were small so that folks didn't have to use Yeti. They could just double click
on a file, it would automatically run and show if their tests are working or not. They
didn't need to use install Node or anything else. If someone wanted to contribute a fix
they didn't have to install any tools, they could just use whatever browser they had.
We decided to do that and that kind of pays off for us because the smallest unit is kind
of a HTML file and that can be spread out to all these different machines pretty well.
But JQuery doesn't have that, so they have one test file. What they can do is there's
a drop down menu on each thing that lets you select just to run a subset, but by default
it runs everything. That's what we're seeing here.
If I go back real quick to show what this looks like, some of these timed out, but it's
running at about 117 for NPM. Yeah, I'm done. For QUnit it's assertions and not test methods.
If you're wanting to get started on this you can go to yeti.cx. It's basically just a tool
that helps you build working code. But your code is ultimately up to you. All this stuff
is opinion. It's opinion that I've found by talking with people that know what they're
doing, but the whole goal of this talk is to kind of show you that you should take this
and then find out how testing can work to make your code work for the people that care
about it, the people who actually use it. I think a user focus instead of a focus on
what the best or coolest way to get great metrics that make ourselves feel better is
always the right kind of decision.
Like Kent Beck says, experimentation seems to be in order until we find the right way
to write tests, so I'd encourage you all to experiment. If you're curious about where
this talk came from, it came from a blog post on my website which you can check out by going
to Lanyrd. Everyone go experiment, and thank you so much. I'm @reid on Twitter, App.net,
and GitHub and IRC. I hope to talk with you guys afterwards. Thanks so much.
[applause]
Jenny Han Donnelly: Any questions for Reid?
Audience member: A long time ago in the dawn of time, a year ago, I remember Yeti had this
capacity for, similar to a test framework I totally forget but it was the green one
where it would just watch the file system and run the tests as files changed and give
you that output. Has that kind of shifted away from that?
Reid: Run tests as files change?
Audience member: Yeah.
Reid: It never has done that, but I was talking about that with other folks. It'd be really
interesting to add. Just having it in a separate tab and seeing the results as you change the
file would be really cool, so yeah.
I'd love help. Most of the things I'm working on for the next few weeks are to help YUI
get all the tests that we run right now that we publish on yui.github.com, whenever folks
make changes or commit code, that's now going to a public list. What we're hoping to do
is for those tests right now, for all those to use Yeti. That's where a lot of my focus
is, but I've heard a lot of folks who asked for things like this, like a watch or having
more test frameworks added. I've done a lot of work to make that easier for people to
get introduced to the Yeti code with the contributing file and with some of the rework I've done
with collecting test results in the page. I would love for people to help out with that
if they'd like to see it sooner than I can do it. Thank you.
Jenny: Okay, thank you Reid.
Reid: Thanks.
[applause]