Tip:
Highlight text to annotate it
X
PAUL IRISH: Hi.
So we're back here at GDL Presents, where we interview
unique and talented developers who are developing really
compelling stuff.
Today we have with us Mark Danks, who you might be
familiar with his work at "My Robot Nation."
How's it going, Mark?
MARK DANKS: It's going well.
Good to see you.
PAUL IRISH: Cool.
Great.
So you led development of "My Robot Nation," which has this
builder, which is this incredible app that uses WebGL
to kind of create a rich experience.
Do you want to talk a little bit about, kind of from a
consumer perspective, how this whole thing works?
MARK DANKS: Sure.
So what we were looking to do was really make it easy and
accessible for people to get access to 3D printing.
And we sort of were scratching our heads at what would be
this awesome thing, and really realized that little
customizable robot sort of figurines
would be pretty cool.
Because each one would be totally unique.
And so we thought about writing an app, or a desktop
app, or something, and realized that we could do it
all in a browser.
And it just seems to make the most sense to do that.
So basically, you go to the browser.
It runs full WebGL, so it's all in 3D.
But my background's all in video games.
And so I sort of approached it of, how do you make these
complicated things really easy to understand, where it's sort
of just drag and drop?
You don't have to know anything about 3D graphics.
You don't have to know how to model.
But literally, it's an infinite--
billions and billions of combinations of these
different robot shapes that you can get.
And they're all in full color.
So you can put a hat on your robot.
You can pick different body parts.
And once you like it, and got it all posed, and ready to go,
you can go ahead and we'll 3D print it, and send it to you
in the mail.
PAUL IRISH: Great.
Yeah.
It's really powerful.
And it's pretty rad to create your own robot and then get an
actual physical version in a few weeks.
So why don't you talk a little bit about how this was
constructed?
Because it's really fantastic.
So one of the things that has always been really interesting
to me is seeing someone from the video game industry, and
bringing them into the web, and kind of seeing what their
development experience is like there.
How did you kind of approach just getting into it, of kind
of ascertaining what your development stack was going to
be like, and how you were actually going to construct
with the new primitives available on the web platform?
MARK DANKS: Yeah.
So the way I approached it first was really thinking
about, you know--
there are two main aspects to the engine itself and sort of
the interface.
One of it is the 3D side of it.
And so that's some really heavy JavaScript sitting in
the background.
And that's basically--
it's a full-on game engine.
We do collision detection, all kinds of stuff.
And that, luckily, I sort of--
I have some experience on how to handle that, how to write
all the shaders and that kind of thing.
So I was able to basically translate what, for me, used
to be hardcore C++ code into JavaScript.
And was really--
at the time, honestly, I was kind of skeptical if this
could work.
You know, how do you [? do array ?] test against
10,000 triangles kind of stuff.
And it just does.
It works, totally interactive, 60 frames a second, no
problems at all.
So that was pretty awesome for just getting the graphic side
of it, the WebGL side.
And then one thing that we did differently, I think, is
really going, we're on the web.
The web is all about interfaces.
So rather than try to create the interface that you
interact with where you pick your attachments, and your
stamps, and your robot pieces--
rather than making that more 3D, which is really a pain in
the neck to develop for, we actually do all
that in normal HTML.
Now again, we're using things like jQuery and some pretty
complex CSS, but the idea was we could bring the two
together in a way that you really can't
do on a lot of platforms.
So that was a really nice thing.
So I could spend all my 3D efforts and all my
visualization in JavaScript and kind of treating it like a
game engine.
But then when we wanted to deal with how people were
going to actually interact with it and select all their
stuff, we could just treat it as a normal page, and just
have the two parts talk to each other.
PAUL IRISH: Yeah.
And so a challenge for a lot of people with WebGL is that
dealing with the actual GL primitives is pretty foreign.
And so a lot of people just hop into using a library.
Now you luckily have a lot of experience, which makes
working inside the OpenGL ES 2 kind of environment a bit more
comfortable.
Was there any third-party code that you kind of used in the
construction of the application to kind of assist,
in probably not that, but other parts?
MARK DANKS: Yeah.
For the actual 3D WebGL stuff, we wrote it all.
And again, from my background, I'm a video game
developer in the past.
So I'm used to writing on PlayStation 2 and 3, for Xbox,
things like that.
So it's pretty much OpenGL, DirectX, all no problems.
So for me, that was no problem dealing with shaders and how
you handle all the actual low-level
vertex buffers and that.
It was pretty easy.
So for that, that's all completely ours.
On the interface side, I mean, the one thing we are using is
jQuery, very heavily.
So sort of all of the state transformations, and
animations, and movement you see on that UI are
all done that way.
And so that was a pretty powerful thing, so we didn't
have to reinvent all those wheels.
And again, that's kind of what--
when we were looking at it--
we're looking at how the slice it of the web side versus the
3D side, and really bringing the two together.
So the main one was probably the jQuery
library that we used.
PAUL IRISH: So inside the application, I get to kind of
choose the head, and the torso, and
the legs of my robot.
There's a bunch of different models.
I can choose specific ones.
I can randomize.
I can then color them all.
For these actual components--
so these are all 3D models.
How did you basically take that from a 3D model that was
designed in a modeling application, and bring it out
into something that you can now see in Web GL?
Kind of, what's the process for taking that there?
MARK DANKS: Yeah.
So we had a bunch of modelers who worked with us.
And they're standard game modelers or
worked in special effects.
So they're used to using programs like Maya and Max.
They're the high-end modeling packages.
And in that case, we just exported to sort of a generic
format called OBJ.
It's kind of a lingua franca for the industry.
And then I just wrote a converter that basically spit
out all the stuff, WebGL ready to go.
So I could just load it directly in the vertex buffers
and things like that.
So for me, it's a fairly straightforward pipeline.
The kind of interesting thing about how we have to do, is
since we have the additional wrinkle of 3D printing, what
we 3D print and what you interact with on the screen
are actually different geometries.
So on the screen, your robot's about 10,000 polygons.
We print the robot at about half a million.
So we actually go through a pretty big translation step on
the back end, where we literally recreate everything
you've done on the screen, and convert it to
a print-ready model.
So that was actually one of the bigger challenges, of sort
of how do you-- what you'd see on the screen, if you print
it, like nothing would be round.
Everything would be faceted and angular.
You can't pull tricks with normals and shading in the
real world type stuff.
So yeah, so we actually have this entire translation
process that occurs.
It's kind of an interesting thing that, when it's running
actually in the browser, every frame, basically, I destroy
the robot and re-create it from scratch.
So every single texture, every single model, every single
component--
I rebuild from these instructions.
So every frame is [INAUDIBLE]
redoing all of that.
And then we spit out that same sort of instruction set.
And that's the part then, that we take these high-resolution
3D models, run it through another tool that reconstructs
that, and makes a printable model.
PAUL IRISH: Wow.
Oh, that's fantastic.
So one question I just had, the [? OBJ ?] format that you
export the models as--
is that sent down in a serialized format to the
browser, which then interprets it, and brings it up?
MARK DANKS: Yeah.
No, we convert it to a JSON format.
So that basically with that, I use this for two reasons.
One is that we need to just load it directly into the
WebGL vertex buffers and texture coordinates and
normals and all that kind of stuff.
So that's just a straight--
load in the buffer, we're done.
The other thing that is--
since we do accurate collision tests against the models--
I actually use that exact same data.
It's basically massaged in a way that the WebGL can handle
it, but then my collision code can handle it as well.
So we can then go off and do all the collision on
the exact same data.
And there, I have to get the normals.
I have to get the UV coordinates.
So I need all the data anyways.
So rather than try to split it apart, send double the data
down, I sort of have a little custom format that we made
that just encapsulates it all in one type.
PAUL IRISH: That's fantastic.
So I'm curious--
a lot--
typically with applications like this, what people might
run into is that the amount of data, like the models that
they're sending down or everything else--
it gets quite big.
So on the web, you have the challenge of, you're sending
out all the bytes to the user that they're going to see.
And they can be sizable.
So kind of like, what are the challenges that
you ran into there?
And how did you solve performance, load
time, things like that?
MARK DANKS: Yeah.
So we started doing it in two ways--
well three, maybe.
The honest truth is, we just went, we hope people have a
fast net connection.
And just all of the data down the pipeline.
But that being said, then what we did is we did a lot of
optimization.
So we do everything from welding our vertices, removing
dead things.
My JSON format is fairly compressed.
So like, if I'm sending a float down, I'll look at how
many digits am I actually sending, and I'll strip it
back, and things like that.
So we were able to get it from sort of a brain dead, F print
float type mindset.
We usually able to reduce it down about a
third from that size.
So that was good.
The other thing is we're not doing any
tricks with the textures.
They're just straight up PNGs we send through.
But they are compressed.
The other thing though, that we do do there, is when you're
interacting in the browser, the textures, the PNG ,
stamps, are just 128 by 128.
PAUL IRISH: Uh.
MALE SPEAKER: We can edit this in post.
FEMALE SPEAKER: What happened?
Did we lose him?
PAUL IRISH: Mark's back.
MARK DANKS: Oh--
how much of that was lost?
MALE SPEAKER: Still there?
MARK DANKS: Yeah, we're still here.
PAUL IRISH: We lost the last third of it.
MARK DANKS: OK.
I'll say it again.
PAUL IRISH: OK.
MARK DANKS: OK, so--
PAUL IRISH: Here, let me--
I'll prompt you--
MARK DANKS: You'll ask the question?
PAUL IRISH: Yep.
So Mark, typically you run into a problem with
applications like this, that you're sending down this model
data, you're sending out a lot of other things.
And you have to deal with the fact that it adds up to quite
a few bytes.
Now what kind of challenges did you run into in regard to
like performance and load time, and how
did you solve them?
MARK DANKS: Yeah.
So there are sort of two main places where we're sending
lots of data.
One is the geometry itself.
And in that one, since the sort of internal data for
collision and all that is the exact same as the rendering,
we don't have to double that up.
So that was one nice thing.
Then I did a lot of optimizations on-- the data we
send through is in JSON format, and I actually really
massage it, so the floating point numbers, we use as
smallest numbers as possible.
And I was able to get down from sort of a naive export
out of my tool, down about a third of the size, really
looking at how much resolution in the UV coordinates do I
actually need?
OK, I don't need out to the millions place, only send two
digits kind of thing.
Then the other thing is on the textures, those are actually
128 by 128 inside the engine.
So they're pretty small thumbnails, basically.
But then when we generate the print model on the back end,
we actually use 1024 by 1024.
So again, we're able to sort of trick it, and only send
through the data we need.
So you can see what it will look like, but then when we go
to the print model, we can use these really high-resolution
textures that are just awesome-looking.
The other thing that we do heavily, heavily, heavily, is
the entire engine is asynchronous loading.
So we don't load anything until it's needed.
And in fact, if you select a part, we don't even load it
until it would appear on the robot.
So for instance, if you select the top hat, you can
immediately select it.
We don't do any loading.
We just wait.
And then you actually move over the model and now, OK, we
have to see the top hat-- that's when we
finally load it through.
PAUL IRISH: I see.
That's really smart.
So yeah, this is really powerful.
I'm wondering, when you were building it, was there any
issues where you kind of wished that you had some extra
feature that was just not available in the browsers you
were targeting, or any challenges like that?
MARK DANKS: Well, yeah.
I mean, I wish WebGL was a little more ubiquitous.
But that was something that we decided early on that we
needed to go for the best experience, and we
just go with it.
Interestingly enough, I've actually made change that's on
some of our other apps now, where if you don't have WebGL,
we actually render and canvas 2D.
And it turned out to be really easy.
I kind of wrote it in a weekend.
And the reason was because I had all the collision data,
which was the triangles, and so all I do is I just project
it into canvas as tri's.
I do a depth sort on all the triangles, and you're set.
So we actually have a fallback now.
And so I can actually run-- not "My Robot Nation"-- but
some of our other apps, I can run on my iPhone and my iPad
without a problem.
PAUL IRISH: Oh, that's really cool.
MARK DANKS: So that was kind of cool to see.
I think the general thing that we were seeing with it is just
that we're so JavaScript heavy.
And so it was--
over the development--
it was in development for six or nine months kind of thing.
It was really cool to see the browsers getting better and
better on the JavaScript side.
And literally, we started telling-- by the end, we could
tell the modelers "add more polygons." You can do it now.
Whereas in the beginning, we were really restricted on what
was possible there.
A lot of the stuff that--
I was doing some pretty weird optimizations on my math
libraries, just trying to trick the
browser into going fast--
and by the end, it was sort of, good, I don't have to do
that anymore.
And so that was really a nice thing that--
if you're doing a little one-off, give, show, hide kind
of thing, you don't really care.
But when you're doing something like this, that's
trying to run at 60 frames, suddenly, your performance
matters a lot.
PAUL IRISH: Yep.
Absolutely.
MARK DANKS: Yeah.
I think other things that I look at and look forward to
are a lot more binary support for data coming to the system.
So that's the kind of thing where, if I had
an easy way to encode--
basically my JSON file is binary--
that would be pretty cool.
A lot of the stuff-- that there's a lot of people doing
work, I know, on texture compression, and like
decompressing it in a shader.
So stuff like that is--
I've been watching pretty closely and kind of waiting
for it to become a little more mainstream
before putting it in.
But the new projects we're looking at are even bigger
than this one.
So we're trying to find ways that we can keep on pushing
the content down.
I think definitely our next big effort is really on, how
do we get the bandwidth requirements down?
Because once it's on the client, it seems
to run pretty well.
But getting there is the problem.
PAUL IRISH: Yeah.
I'm curious about kind of-- staying in tune with some of
these conversations and the features that are being
standardized and being imposed, it's a little tricky
sometimes to kind of monitor where these
conversations are happening.
How are you tracking, for instance, texture compression,
or things like that?
MARK DANKS: Yeah.
So thankfully, it's really awesome.
We got hooked in with the WebGL community pretty early.
And we actually presented it to [? SIGGRAPH ?]
Birds of a Feather a couple years ago.
And that's sort of where we sort of burst on the scene and
got introduced to everyone.
So now we do it-- it's everything from the mailing
list-- which is really helpful and really, really good.
A lot of stuff happens there.
Most of the people know I kind of lurk, but I read every
single thing that goes through.
And then--
I'm based in San Francisco--
and so we have WebGL meet-ups once a month.
So those are pretty good, where lots of presentations
are given and talking about what's going on.
I presented there.
Actually, my company's hosting the next one on October 25th.
So that kind of thing is a good place to meet people who
are doing this, and sort of find out what's going on.
And also it's really--
all the people who are helping make WebGL real are really
open, and accessible, and really easy to talk with.
So that's been a really nice thing as well.
PAUL IRISH: Awesome.
Well I encourage everyone to go check out "My Robot Nation"
and dig under the covers a little bit.
It's a really well-developed app, and creates a great
experience--
one of the first great, solid WebGL apps that's definitely
not a demo.
And it's real, creates a real thing that you can hold, even.
So I encourage everyone to go check it out, see
what's going on.
And Mark, thank you very much for joining us and
telling us about it.
MARK DANKS: So one other thing, if I can
make a plug, here--
we're going to actually expand it a bit.
And I'm actually in charge of a another group called
cubify.com.
And one of the things we're going to be releasing here in
the next couple months is actually an SDK.
So people can use this engine to make their
own 3D printed models.
So that's something which--
keep an eye open for it-- probably end of October,
November time-frame, this code's going to be out and
sort of fully supported.
PAUL IRISH: Great.
And say the name of it again?
MARK DANKS: It's cubify.com.
And if you go there, you'll see a bunch of other apps.
In fact, just last week, we released an Android
customization app.
So you can make your own Android with the same
techniques.
PAUL IRISH: Cool.
That's rad.
All right.
Nice.
Well, thanks again, Mark.
This has been great.
MARK DANKS: Thank you so much.
PAUL IRISH: All right, cheers.