Tip:
Highlight text to annotate it
X
GREG DALESANDRE: Hi.
Welcome to the App Engine Fireside Chat.
Apparently we have a big firepit.
So I think what we'll do is--
the people who we are waving to are members of the team, so
if you think we're waving to you to come up on stage--
So we'll do a quick round of introductions so you know who
all the people are on stage--
when I say quick, pretty quick.
And then start to answer questions from the audience.
Talk about topics that you're interested
in us talking about.
Let us know what you think.
Please use the microphones on the side so
that it can be recorded.
I guess I'll start intros, and then I'll go to Alfred next.
So my name is Greg Dalesandre.
I'm a senior product manager on App Engine.
ALFRED FULLER: I'm Alfred Fuller.
I'm a software engineer on the App Engine data store.
MATT WILDER: I'm Matt Wilder.
I'm a site reliability engineer for App Engine.
BRETT SLATKIN: Brett Slatkin, software engineer.
MAX ROSS: Max Ross, software engineer on the data store,
Java run time, various management function.
ALON LEVI: Alon Levi, software engineer on the serving
infrastructure team and various management
functions as well.
PETER MAGNUSSON: Peter Magnusson, engineering
director, mostly management functions, I guess.
SEAN LYNCH: I'm Sean Lynch.
I'm one of the product managers on App Engine.
*** VON ROSSUM: I'm *** von Rossum.
I'm a software engineer on the tools team and I have an
opinion on everything.
ANDREW DURAND:: I'm Andrew Durand, and I
work on the Go team.
MIKE AIZATSKY: I'm Mike Aizatsky.
I'm a software engineer.
I produce files [UNINTELLIGIBLE].
NICK BURTON: I'm Nick Burton, on the task queue team.
GREG DOCK: I'm Greg Dock, and I"m on the task queue and
[UNINTELLIGIBLE]
team.
WESLEY CHUN: Wesley Chun, Developer Advocate.
PETER MCKENZIE: Peter McKenzie, engineering manager.
MICKEY KATARIA: Mickey Kataria, I'm
another product manager.
ERIC MORRIS: Eric Morris, the odd duck from sales and
business development.
JUSTIN HA: Justin Ha, serving
infrastructure, software engineer.
TOM EDWARD HAZEN: Tom [? Edward Hazen, ?]
I do documentation.
MICHAEL HANDLER: I'm Michael Handler.
I'm the tech lead for the site realizing engineering team for
app engine.
GREG DALESANDRE: All right.
So this is actually a pretty good portion of the
team you have here.
And so we're happy to talk about the topics that you're
interested in us talking about.
We could talk amongst ourselves.
That might not be as interesting.
Does anyone have any questions or topics they want us to get
started with?
I heard something about new pricing.
Yes?
MAX ROSS: Yes, there is new pricing.
GREG DALESANDRE: So you might have seen yesterday that we
announced that we are going to be leaving Preview later on
this year, which we're very excited about.
Leaving Preview also comes with a new pricing model.
Let's see.
There's a lot of questions on our external list. I guess I
could talk about some of those in person, if
that would be useful.
Whoever yelled it out.
What sort of thing would you like to hear about?
AUDIENCE: Differences!
GREG DALESANDRE: Differences.
I can't see who it is who's yelling.
There's just a voice back there.
So the main differences in the pricing model
are going to be--
where before there were free apps, and then apps where you
could turn billing on, and then you would pay
for what you used.
In the new model, there's going to be
essentially three types.
There's going to be free apps, still.
We believe very strongly that we want to give people free
apps that they can use forever, so that they can get
started, they can try things.
There are then going to be paid apps.
So instead of apps where you have billing turned on, paid
apps are $9 per app, per month, in addition to the
usage that you use.
But with this $9, you get an SLA, a 99.95% SLA.
And then there's going to be premiere accounts.
And premiere accounts are $500 a month.
You can have as many apps as you want.
I guess I should have said--
the paid apps, you can also start paying
for additional usage.
So you can start scaling.
With the premier accounts, it's $500 per month, but you
get operational support, as well as all of
your apps can scale.
You don't have to pay an additional fee per app or
anything like that.
So that's the usage side of things.
Then one of the big changes is we're going from CPU hours to
instance hours for billing.
So CPU hours is broken into two parts now.
Before what we were doing was charging everyone just based
on the CPU that they were using now.
For your code that's running, we're going to be charging
based on the number of instances that are running,
and how long they're running.
And so essentially, one of the main reasons we did this-- and
this is something we talked about a little in the group--
is right now, because we just used CPU, there's no way to
essentially account for people who want to use high-memory
applications or high-latency applications.
And so what happens is we just don't scale it.
I don't know if any of you have run into this.
If your latency is two seconds in your app, what ends up
happening is, it just doesn't scale in App Engine.
Because essentially, we don't know how to how to treat
applications like that.
So charging for instances essentially now opens up this
possibility for everyone who wants to write applications
and essentially doesn't want to be constrained by, it only
has to be based on CPU.
They can write high-latency applications, high
memory-intensive applications, and essentially pay for their
usage of those.
The other thing that CPU hours were used for is
for all of the APIs.
We've essentially taken those APIs and converted what we
were charging before into operations.
So before, we would charge you some amount of CPU hour per
XMPP stanza, and now we just call it a stanza and charge
per stanza.
And again, this was done for sort of better transparency in
terms of allowing people to estimate and figure out what
it is they're using, and how much it will
cost them to use it.
I think I saw a hand go up at some point.
ALON LEVI: Feel free to queue up at the mike.
Questions don't have to be about billing,
but they can be.
AUDIENCE: I just have two kind of specific questions.
One, with the XMPP in the stanza.
What about presence notifications?
GREG DALESANDRE: So I think with presence notifications--
MAX ROSS: You're asking about pricing for presence
notifications?
GREG DALESANDRE: Yeah, yeah for presence notifications.
AUDIENCE: I mean, I suspect that I know what a stanza
constitutes, but I'd just like to be clear on presence
notifications that I saw was relatively recently added.
If users go on- and offline, et cetera, it [UNINTELLIGIBLE]
the app.
So is that additional charge, or?
BRETT SLATKIN: We're still figuring out little
details like that.
That's a great example, because it's a very
high-frequency event that costs basically nothing,
because there's very little information.
So I think that these are the kind of things we're still
trying to work out, before--
GREG DALESANDRE: Our XMPP person is not here today to
talk in more detail about it.
But I'm in the middle of putting an FAQ together to
sort of post to the list, and I'll add that into it.
AUDIENCE: So just one more small question.
And that is, currently the HR data store has what is it, a
3X multiple on API CPU usage.
Is that going to be eliminated as well so that HR and
master/slave are just the same, as far as cost goes in
the operation?
GREG DALESANDRE: Do you want to take it [INAUDIBLE]?
MAX ROSS: Yes.
We want everyone to use high replication data store.
And we've got other things planned that will encourage
you to move in that direction.
Yes.
Good things.
Yes, carrots.
GREG DALESANDRE: Not beatings.
MAX ROSS: Good things.
Carrots, not sticks.
AUDIENCE: So the growth of the platform is kind of
phenomenal.
So maybe you could talk about how you're dealing with the
technical complexities, as well as the growth in your
team, and how to manage that sort of complexity?
GREG DALESANDRE: Someone from SRE maybe want to talk about
some of the growth and technical complexity?
MATT WILDER: Well, probably the biggest example of this is
building the high-replication data store, right?
Something that brings more liability and more
scalability.
And one of the things that we learned is, when you have a
service that's really big, a very small percent of errors
are really painful so our users.
So this is kind of one of the first things that I think that
we did to really address the scalability of the product.
MAX ROSS: Just in terms of growing the team, one other
thing I would say is, you know, last year we did one of
these fireside chats.
I don't think any of our SREs--
you guys weren't up here with us.
We didn't make a big announcement that hey, the App
Engine team now has SREs.
But in the lifecycle of a product within Google, that's
actually a pretty big milestone, because SREs are in
very short supply.
And so you have to be important to have these
skilled people helping support your service.
And that was relatively new for us around
this time this year.
It's probably been, I don't know, closer to fifteen or
sixteen months.
AUDIENCE: SRE means what?
MAX ROSS: Site reliability engineers.
So yeah, thank you.
[INTERPOSING VOICES]
MICHAEL HANDLER: Yeah.
So there's been SRE engagement with App Engine
since January of 2010.
And I'm giving a talk on it at 3 o'clock over in the other
hallway, Life in Production.
And you know, I'm not going to just reveal everything.
I'll just take over the rest of this and give
the talk right now.
Just get it done.
Dry run.
But It's fantastic.
Google is a moving environment in same way that app engine
scaling is a moving environment.
So we're getting tremendous growth.
People are using the platform in ways that we never
anticipated.
And we're thrilled about that, but at the same time, Google's
not standing still saying, well, this is what we offer.
Google is growing.
We've got other products coming online.
Infrastructure changes, disruptions that are caused by
changes in the infrastructure, surprises that come to us.
And this is all what I end up talking about, and the high
replication data store is based very explicitly on what
we learned, and I talk very explicitly about some of the
things that happened to us, some of the things that we
know could happen to us, some of the things that we're
afraid could happen to us, and that HR data store handles way
better than the master/slave data store ever could.
So come see my talk.
GREG DALESANDRE: The other thing I just wanted to add is,
we're also seeing a wider variety of customer types.
Right?
You know, it's kind of like when you're just getting
started and people are just sort of trying something out
that's new, there's a certain sort of type of developer who
will kind of give it a shot.
Now we're seeing a very wide range of people to--
enterprises you wouldn't expect who are looking into
using it, as well as one person startups who are our
doing it on the weekend.
And so I think it's been interesting trying to support
and handle such a wide variety of customers.
AUDIENCE: I had a more technical question, and that
was, is there is a good way to sink the memcache across
different instances?
BRETT SLATKIN: I mean, every instance should have the same
view of memcache?
AUDIENCE: Right.
BRETT SLATKIN: So could you clarify?
AUDIENCE: Well, yeah.
One application that I've got is collaborative
story-writing, and we'll have 50 or 100 people come in and
they're all exchanging messages, and stuff like that.
But I've got a thing that's checking to see if their
browser needs to refresh based on what it's
giving for the memcache.
And I found the when it was 20 new instances, it didn't have
that same information.
They had to--
BRETT SLATKIN: Yeah, I mean, memcache is supposed to be a
consistent view shared by all instances.
Sometimes you get evicted, and then you lose the information
so it can go away.
But I think that you should definitely check out the new
backend feature that they launched yesterday, which kind
of lets you write your own stateful memory resident
backend that has a consistent view.
And maybe that'd be a better fit for what
you're trying to do.
But if you could stop by our office hours and kind of give
us some more detail, I think that would be really helpful.
Because that's not how it should work.
It should be there.
That's the point.
AUDIENCE: Thanks.
AUDIENCE: I was curious about the integration between Google
Apps and Google App Engine, and what's there now and what
do you see as far as being able to access corporate data
stored in docs that's related to Google Apps?
PETER MAGNUSSON: We plan to improve that.
Great question.
ERIC MORRIS: Echo the great question from the sales side
of the house, as well.
We need a better answer.
We're actively working to make better and tighter
integrations between those, because it's something that
people ask us about all the time.
AUDIENCE: We use your [UNINTELLIGIBLE]
to interact with other services
outside of app engine.
Unfortunately, we work with Apple for its notifications,
and that has to connect on a non-HTTP protocol.
Is there any plan to make something which is
non-HTTP-based, URL patch or is this something that doesn't
work with Google infrastructure, or
something like that?
SEAN LYNCH: One of the things that we're currently working
on is adding support for sockets.
So you will be able to reach out on non-HTTP connections.
AUDIENCE: Cool.
Any timeline on that?
Because we're actually [INAUDIBLE]
notification.
SEAN LYNCH: No timeline at the moment.
AUDIENCE: OK.
Cool.
Thank you.
AUDIENCE: There is a [UNINTELLIGIBLE]
initiative back in India, which is probably the world's
largest private cloud that we're building.
And currently, they are planning to use a
[? nautical ?]
database, which is based on the Bigtable whitepaper
because App Engine is not open source, and you don't have
support for private clouds.
Is there any initiative from that [UNINTELLIGIBLE]
other option for such kind of large private cloud
initiative?
SEAN LYNCH: So the whole discussion of private cloud
and things like that--
first options, there are actually two separate open
source implementations of the App Engine API, a Tycoon AE
and AppScale.
And they're using projects like Cassandra to sort of back
the API equivalent of a data store in terms of
functionality.
From our perspective, when we're talking about things
like HRD, there's just not a way that we
can bottle that up.
The fact that we have Mike and Matt working behind the scenes
to do sort of failovers between multiple data centers
at once isn't sort of something that we can put in a
box and let you throw on a rack.
And that's sort of the fundamental difference when
people talk about private clouds as sort of an option.
You don't have these guys, you don't have this team actually
maintaining the data store or the database, whatever
technology you choose.
So for private clouds, there are a lot of really
interesting open source projects in the no-sequel
space that provide similar or complimentary functionality,
but you can't get a lot of the things that the public cloud
gives you, especially when you look at examples like HRD,
behind your firewall.
Unless you hire these guys and--
GREG DALESANDRE: Don't hire these guys.
That's not an option.
AUDIENCE: Google as a company doesn't have any plans to do
open source or create services?
Are other companies creating service of
their own at open source?
GREG DALESANDRE: Well, as Sean mentioned, our APIs are open
sourced, and there are two external, third party
companies that are building App Engine compatible servers.
SEAN LYNCH: Yeah.
I mean, at Google, our proficiency is the cloud.
That's what we know.
And we do very, very well at that.
And so that's where we will continue to
do very, very well.
AUDIENCE: So one of the great things about App Engine is,
there's no [UNINTELLIGIBLE]
it.
You don't have to pay.
Will the change to [UNINTELLIGIBLE]
the billing, change that?
If I have more instances, am I been billed for that?
GREG DALESANDRE: So one of the things, again, on the
[UNINTELLIGIBLE],
we talked about a lot--
it's still designed to be pay as you go, in terms of, if
there are instances serving traffic, you're paying for
those instances that are serving traffic.
One of the things that we have-- and there was a talk
about this yesterday--
we have a scheduler which essentially decides how many
instances should be running based on the amount of traffic
that you have coming in.
And that's what essentially spins up new instances when
the traffic demands it, and takes down instances when
traffic is no longer needed.
We're in the process of essentially improving that
scheduler or, to be more precise, making that scheduler
fit with the instance model.
And so that essentially is going to choose how many
instances you have running.
But the point is, for instance, if you have no
traffic at all, you shouldn't be paying for instances unless
you've made a decision that you want to have instances
running just in case new traffic comes along.
But again, that would be sort of a decision you'd make of, I
want to leave instances up, just in case new
traffic comes up.
Does that answer the question?
OK.
AUDIENCE: Should we expect to see tighter BigQuery
integration?
GREG DALESANDRE: Oh, BigQuery.
Anyone on stage who can talk to this?
So I think the short answer is, it's definitely an area
where we're looking at.
You probably saw the Google Storage announcement
yesterday, as well.
And all of these parts, we definitely want to and are
working on playing more closely with.
As I'm sure you're used to hearing this answer, we can't
give timelines as to when they're going to be done.
But it's certainly absolutely on our roadmap of things that
we're looking at doing, is integrating with some of the
other sort of cloud services from Google.
ALON LEVI: Yeah.
Another way to answer that is that we want all the
integrations.
Right?
We think all this stuff is really cool.
We'd like to have it all integrated.
It's just a matter of
prioritizing everything, again.
Stability improvement and other things like that.
So yeah, we definitely want it to happen.
AUDIENCE: Yeah.
And that's really the gist of the question.
It's priority for me, but--
GREG DALESANDRE: SSL is a higher priority on
our list right now.
WESLEY CHUN: The ability to have an App Engine application
as an input source to analysis is certainly something that
we're looking at as a use case for the platform.
BRETT SLATKIN: I was just going to add that I think also
a lot of this fall under the OAuth2 kind of heading.
So we're starting to deploy OAuth2 for services, which
makes these integrations a lot easier.
The docs integrations that someone else asked about
earlier using the Google Apps or Google Docs APIs, is hard
now for a variety of reasons.
One of them is Auth.
So authorization authentication is hard, and
OAuth 2 simplifies that a lot.
So as we move more and more of our services to use OAuth2,
then stuff like integrating App Engine with BigQuery or
App Engine with Google Apps will get easier.
So the other Auth teams at Google have been laying the
foundation to let us simplify the problem.
So I think that's kind of a bigger picture answer, too.
GREG DALESANDRE: Right.
And I think that'll help.
But that being said, we also want to make sure that it's
easy to run App Engine with non-Google--
you know, I think we try in general not to do anything too
crazy custom just to have a single point of integration.
I think we do, in general, as much as possible, try to make
the integrations sort of generalized.
And so for instance OAuth 2 is great example of that.
OAuth 2 is something that parts of Google are moving to,
and so it'll work well together, but it's also
something that third parties outside of Google can use
OAuth2 and then we'll work well with them, too.
AUDIENCE: Thanks, gentlemen.
GREG DALESANDRE: Sure.
AUDIENCE: All right.
Thank you very much for App Engine.
It's a really fun product form to develop with.
My question is, in the last two days, we've seen a little
[UNINTELLIGIBLE]
that it's going to make the product form way bigger and
maybe less simple.
And one thing that I've noticed with App Engine is,
there's like a very good best practice.
But if you don't follow it because you don't know it or
whatever, and then you push the production, migrating data
or anything kind of complicated.
So my question is, what resources or guidelines or
ideas can you give us so we can keep up to date with the
best practice as a whole [UNINTELLIGIBLE]?
Just the mailing list?
GREG DALESANDRE: I can answer, unless somebody else wants to.
WESLEY CHUN: The mailing list is good.
Our blog is good.
A number of our Dev Advocates and DevRel people
also blog, as well.
I think our documentation is excellent.
So the documentation is fairly sparse, but it's enough
information to get you started.
So part of the members of the developer relations team, our
job is to take the stuff and build more realistic examples
that you would use.
So keep an eye out for a blog post, as well as articles
under the documentation that give you more of that.
And we're also working on a best
practices article, as well.
MIKE AIZATSKY: We also have been talking for years about
having some kind of community-driven documentation
project like wiki.
Unfortunately, [UNINTELLIGIBLE]
is like SSL stuff and other things.
So if you guys write App Engine app which is a wiki,
we'll be happy to contribute, certainly.
If someone tells me, hey, we need two lines about best
practices to use memcashe, I would just
go and write something.
So just collaborate together guys.
Give us a place to contribute that kind of best practices.
TOM EDWARD HAZEN: And two things.
First is that we're planning some specific best practices
documents, one related to security.
I'd like to take some of the stuff that *** and Justin
talked about in their scaling talk yesterday and create a
document about best practices in regard to scaling.
Second is that if anybody wants to actively contribute
to App Engine documentation, we're
looking for another writer.
So if you'd like to write some best practices documentation,
please see me after the talk.
ALON LEVI: I should also add that as you see places where
it's a little murkier, it's not clear what the best
practice is, please come on the forums and ask questions
for clarification and things like that, you know?
There's a lot more of you than there are of us, and so when
you point out something that we can take a look at and
provide an answer to, it's pretty helpful.
GREG DALESANDRE: And we do monitor the forums, and
respond when other people aren't responding or when it's
important for Google to respond.
But yeah, I would say the two main places to look are our
blog, which is quite active, as well as the forms.
AUDIENCE: OK.
Thank you very much.
AUDIENCE: What sorts of services are you looking to
provide in the $500 a month kind of enterprise tier?
And as an agency, we're kind of continually running up
against the like ten apps per user count.
Is that the sort of thing that would go away once the
pricing's in place?
GREG DALESANDRE: Yeah.
So for the premier accounts that are $500 per month, one
of the main things that you'll get is operational support,
which a lot of our customers have been asking us for, so
that they know that they can contact somebody.
So asking a question on the list will almost always get
you a response, but a lot of times you want essentially a
guaranteed response on it, not sort of waiting when somebody
gets to it, essentially.
But the other thing it does give you is as many apps as
you want, all of which can scale.
So if you're trying to run hundreds of apps, it's
probably a good way for you to go.
Those are the two sort of main aspects that you get with it.
There are things that, over time, we might start adding
in, that are kind of much more of the sort of like
enterprise-y type features.
But for the most part, that's what it is right now.
Does that make sense?
Great.
No more questions?
Come on.
There's got to be more questions.
ALON LEVI: Come on, guys.
We're not even halfway done.
AUDIENCE: SSL?
GREG DALESANDRE: SSL?
So what do you mean by--
I'm just kidding.
SSL is a very high priority for us.
We don't have a date that we can announce it--
PETER MCKENZIE: I can say something.
So I'm responsible for SSL.
GREG DALESANDRE: I was just trying to protect him!
That's all.
PETER MCKENZIE: We are actively working on it and
making good progress.
We can't give a date, but it's certainly high priority.
To be honest, we would have liked to have launched it by
now, but it's certainly foremost in my thoughts, and
in the thoughts of several others of my team.
BRETT SLATKIN: Two things that would speed it up.
Get everyone to install an S&I browser.
So upgrade everyone to IE9 or Chrome or whatever.
If you do that, that'd be cool.
Other thing is, everyone move to IPV6.
That would also help us get the shift sooner.
So if you could help us out there, that'd be great.
We're working on our side.
AUDIENCE: Is there a way to do the first with [INAUDIBLE]?
BRETT SLATKIN: No.
You get this warning prompt that says you're going to die.
I mean, it doesn't work, yeah.
So that's a problem.
If you don't have a browser that's compatible to SNI, then
you can't use it.
It looks like you're being phished or
hacked or stolen from.
AUDIENCE: Triple w dot domain, and then do a little browser
test, and direct to [INAUDIBLE]?
BRETT SLATKIN: You can't even test. It just freaks out.
When you try to get somebody to go to a site, it just
rejects it outright.
So that's the issue.
You can't even test for it.
MICHAEL HANDLER: I'd just like to say operationally that I'm
thrilled that all the mistakes we made with HTTP, we managed
to make again with HTTPS as a community, that it's great,
because it means full employment for me.
So when we're designing protocols, let's keep it up,
make the same mistakes over and over again, and I can't
wait to have a funeral for all of the hacks that we need
system Y to deal with this.
GREG DALESANDRE: There are a lot more questions now!
AUDIENCE: So you guys just announced support for Go.
What other languages would you like to see App Engine run on?
GREG DALESANDRE: What would we like to see, or--
MAX ROSS: What would you like to see?
GREG DALESANDRE: We don't really
matter in this equation.
So, well, you got what you wanted.
No.
AUDIENCE: I want Python 3!
GREG DALESANDRE: Right, OK.
So definitely Python 3.
One or two people might have asked for PHP.
I don't know if we've, have we heard?
Just kidding.
Lots of people ask for PHP all the time.
And that's certainly another language we know people are
asking for.
I think people have talked about like Node.js,
that sort of thing.
You know, we announced the experimental Go run time,
which we have Andrew here from the Go team who's going to
answer questions about that, if people are interested in
asking about that.
We don't have any other run times we can
announce at this point.
AUDIENCE: Would you support languages that weren't
internally used at Google?
GREG DALESANDRE: Would we?
Sure.
I mean, there's nothing to stop us from doing that.
Again, it's a matter of prioritization, integrations,
and all these other things.
MICKEY KATARIA: And just a quick comment.
For those of you who have been paying attention to our public
roadmap, Python 2.7's on there.
We are working on support for 2.7.
PETER MAGNUSSON: Yeah.
So I wanted to go to respond to the notion of if it's got
to do with internal languages.
It's not that there's a policy that no, we're not going to
support other languages.
It's that we have a Go team internally, and we do Go work
internally.
I'm hoping that we'll be able to figure out ways to allow
external communities to participate in adding more
languages and so forth.
We'll see what happens down the road.
I do want to emphasize, though, that with Go, we're
adding something that's fundamentally different than
we would have with the other language options, which is
that that's a generally compiled, low-level language
that we'll be able to run in App Engine.
So that will be very useful for services and so forth.
ANDREW DURAND: I was just going to say that it was
actually at the Go team's kind of impetus that we started
this project.
And we didn't kind of detract resources from the App Engine
team proper and pull their engineers onto it.
It was actually, we got more people on the Go team, and
expanded our efforts to make this happen with the help of
the App Engine team.
So none of the bugs on the issue tracker that you're
feverishly waiting to be fixed didn't get
fixed because of Go.
So don't hold that against us.
BRETT SLATKIN: And to add to that, Go as a language has an
unsafe package.
PHP does not have an unsafe package.
I think that's just dot PHP is the unsafe
package for PHP, right?
So I think part of this is about isolation, security.
When people say PHP, they don't mean the language.
They mean the environment.
They mean it running in Apache.
They mean a very specific set of C extensions that are
community-driven.
So I think that if you try to understand how easy is it for
a language to be brought into App Engine, you need to think
about the full picture of what it means, right?
A language is not just a language.
You've been able to run PHP as just a pure language on App
Engine through [? Quirkus ?],
through the Java run time, for a very long time.
But that's not what people want, and that's not what they
consider PHP.
So that's another thing to remember, is that when people
ask for language, they're asking for how
they program now.
MIKE AIZATSKY: And the other thing to keep in mind is that
we'd like to have all languages, but I'm part of run
times and API teams, as we call it, but the whole process
does not care.
So just let me give you a picture now.
When you want to implement an API, you have to implement
[UNINTELLIGIBLE].
You have to implement API in type.
You have to implement API in Java.
You have to implement API in Python developer server, and
you have to implement API in Java developer server.
Now the Go comes in.
So you now have to implement it Go API.
Fortunately they use Python developer server.
But every extra language adds a tremendous maintenance
complexity for us.
And the whole process does not scale.
If you threw in five languages more, we would be like 100% of
the time just catching up the languages with each other.
So we'll have to start somewhere.
But we would definitely like to see more languages.
ANDREW DURAND: And just one to add.
Late last year, we added a page to the Go website which
allows you to run [UNINTELLIGIBLE]
Go code in a sandbox environment on Google servers.
And so we have the Go expertise on the Go team to
make that happen, and we already demonstrated that we
could do that.
And so taking that sandbox and showing that to the App Engine
team, saying, hey, we've already done this, it was much
easier for them to take us on board than to look at a
totally unfamiliar language, which we don't have domain
expertise in and try and approach
that colossal problem.
AUDIENCE: Are there any plans to support sharing a data
store between apps?
MAX ROSS: Yes.
We talk about that all the time.
I think I've probably even answered that same question up
here in years past. But it's actually getting closer to
becoming a reality.
There's some infrastructure improvements that are going to
make that a lot easier for us.
It's somewhat similar to what Brett was talking about with
OAuth, although the details are different.
But there are pieces that make it hard today that are going
to become easier very soon.
You may have seen--
well, actually, I don't want to get too much into the
details of it.
But the types of things that we have in mind are the
ability to use the same data store service API that you're
used to in whatever language that you program against. But
have some way to have those calls just transparently
redirected someplace else.
And we've got to get the plumbing right, and we've got
to make sure that we are connecting things in a safe
way, we've got to get the apples right.
I think we understand it pretty well, but we need a
couple of infrastructure pieces in place, and then we
need to prioritize it appropriately.
But it's closer this year than it was last year.
And we do have some very concrete ideas about how this
might work.
That's the best I can tell you.
AUDIENCE: Just a quick question about
[UNINTELLIGIBLE].
How will the certificate management work?
Will it be bring your own certificate, or does Google
have a CA that will issue--
PETER MCKENZIE: It will be bring your own certificate.
You'll be able to upload your certificate, and we'll serve
that for you.
GREG DALESANDRE: That's what I like to see.
More questions.
AUDIENCE: Two questions.
What are you doing with regard to roles for administration?
I noticed recently there were little enhancements there.
Can you talk about what you might be doing going forward,
and then also, just any discussion you can give us
around some of the experimental sequel server
stuff that's out there?
GREG DALESANDRE: So for role--
***, you want to talk to it, or?
*** VON ROSSUM: We don't have any specific plans to
sort of add more roles or more probations.
But the sort of infrastructure we have for roles certainly
would support more roles.
So if we get sufficient feedback saying, well, there's
a particular category of administrator in our system
for which we would like to have a role that can do this
but not that, and yet sort of--
then we can certainly relatively easily add that to
the system.
So write up exactly what kind of role you would like to have
added, and we'll look at it.
GREG DALESANDRE: And in response
to the sequel question--
so sequel is actually currently in trusted tester.
It has been for a little while.
This answer is going to sound similar to SSL.
We're working *** it.
We know it's really important.
Although sequel actually is in trusted tester.
So if you're interested in trying it out and seeing what
it's like, you can certainly join our
trusted tester program.
We know how critical it is, and it's a high priority to
get it out there, but things take time.
AUDIENCE: This is probably more of an office hours
question, but since you're running out of questions.
It's a dumb debugging question.
I've got a Python App Engine app that's been running fine.
It's not production yet, so I'm not worried about the bug.
But what's happening is I'm getting very intermittent--
it uses OpenID OpenAuth to connect up to fusion tables.
And very intermittently, I'm getting 403 Forbidden errors
when I try to connect to the app.
It runs fine for days, and then all of a sudden, I'm
getting 403 Forbiddens for a while.
And it'll be on one browser but not another browser.
It's not cookies.
You don't have to look at all that stuff.
Take a brand new browser, and if VM has never seen the app
before, it'll get it, or not.
It's just pretty intermittent.
Anything jump out at anybody, or I'll do some more
troubleshooting on my own.
No?
[INAUDIBLE]
the team!
AUDIENCE: It's hard to debug when you don't
[INTERPOSING VOICES].
GREG DALESANDRE: Debugging on stage, huh?
MICKEY KATARIA: One possibility, and I don't know
off the top of my head for fusion tables, is that a lot
of the APIs do sort of IP-based limiting or some kind
of limiting which [UNINTELLIGIBLE]
maybe occasionally you're running into some of these.
So that's one area to look at.
AUDIENCE: Possibly.
Yeah, I can't even connect to my app, is what's happening.
But I'll do some more troubleshooting.
GREG DALESANDRE: Or you can come by office hours.
AUDIENCE: Again, this question is about the asynchronous API
that recently came up for Data Store that's
currently in Java.
I want to understand whether it's there on Python already.
And a follow-up question to whether there is any
[UNINTELLIGIBLE]
in using async API.
Is it ready to [UNINTELLIGIBLE PHRASE]
where sync merely a waiting call, or is satellite
[UNINTELLIGIBLE]
you create keys with [UNINTELLIGIBLE]
new, fresh entities?
Two different [INAUDIBLE].
ALFRED FULLER: So the async API for Python is out.
Was that 150?
GREG DALESANDRE: 150, yeah.
ALFRED FULLER: Yeah.
It was out in 150.
It's in every level.
So it's in ext.db, and datastore.py, and even lower.
So I recommend you check it out.
And synchronous calls call the async calls
and then call block.
AUDIENCE: OK.
So no worry.
ALFRED FULLER: Yeah.
You should totally use it.
It's awesome.
AUDIENCE: Any word on expanding the one megabyte
limit for entities?
MAX ROSS: You know, we've got some ideas about how maybe we
could split it up.
But we have some internal limits on transaction ties
that we would need to clear out first. And those are
limits that are not just in our layer, but in layers that
go further down the stack.
So we would love to get rid of that one, but we've got our
work cut out for us to do it.
BRETT SLATKIN: I'd just add that Mike has deployed the
file API that just came out.
It's a little experimental, but it's deployed in the
Mapper framework, [? mapperdev.appspot.com. ?]
And it solved this problem for me.
I've been doing a bunch of things with it where I have to
write, you know, 100 megs of random data, and then index it
based on other properties.
So that's what I've been doing.
So the size of my entity only really needs the blob key,
blob reference property, and then a bunch of other stuff.
And that scales out as big as I want.
Because of relation indexes, I can go as large as I possibly
could imagine.
So I think that it requires you to kind of move things
around a little bit, but I don't find this to be a
problem anymore because of the file API.
So that's still experimental, but I encourage you to try
that and to use that instead.
If you need your indexes to be larger than 1 meg, you can add
multiple trial entities to do those queries.
That works really well.
If you need even more data to be indexed, maybe the full
text search thing that they were talking about earlier
today is a better fit.
I don't know beyond that.
But that's how I've been working around this, and I've
had success with it.
AUDIENCE: Are there any plans to allow copying an app in
production, including production data, for like
testing purposes?
MIKE AIZATSKY: If you deploy at least one Python
[UNINTELLIGIBLE],
which might be absolutely empty, then you're going to
have the data store admin pages.
And there is two functions and they're just
through admin right now.
The first one is to delete data and the other
one is to copy data.
So you could set up another app, open remote API.
There are like some instructions how to do that.
But you can actually pick these entities
and copy them over.
And it will start the Mapper job, and it will copy the data
to the other app.
And it can override data, so you can actually catch up on
changes, whatever.
GREG DALESANDRE: Just as a side note on that.
We often get questions about sort of what's kind of the
intended deployment process for App Engine, right?
If you want to have like a dev app, a test app, and a
production app, something like that.
And this is something that we've been talking about sort
of putting more effort into around trying to, again,
prioritize against all the other stuff, too.
But just because, I think this question comes up a lot,
because I imagine that's what you were
trying to get at, right?
You want to sort of have a clone of these things, and if
you use live versions, then it's all in the same data
store, and so you want multiple apps.
And so we'll look into sort of figuring out what the right
way to do that is, and then sort of making that easier and
making kind of the tools.
We might try to do a best practices before that, just so
that people are sort of heading on
down the right path.
MATT WILDER: So in terms of copying data from one
application to another, this is the primary way we're
working on having applications migrate to high replication.
Because we have to copy your data from the master/slave
data store to the high replication data store.
And so we're hard at work building out
the tools to do that.
It's very likely that those new tools, will be able to use
to do other sorts of migrations, as well.
GREG DALESANDRE: Peter, did you have something?
PETER MCKENZIE: Well, I'm not sure I'm
answering the same question.
But in terms of testing and different app versions, name
spaces could also play into that, and how you manage your
roll-out process.
AUDIENCE: I'm curious to know about data source import and
export tools that you can pull out in your roadmap.
Data tools, import and export tools, how
that's coming along?
BRETT SLATKIN: Well, we have the bulk loader, and there's
an experimental version.
I think it's in SVN?
Yeah, it is.
So there's a newer version of it.
I think it's the old prototype.
It's experimental.
It's in the bulk loader SVN repository.
But effectively, it does all of the stuff on the server
side, exports to using the file APIs to some giant blob.
You download the whole blob, and then you can reupload it,
and then import from that blob again.
So those kind of tools are really good for backups, or
for saving and restore, or for copying between apps, or for
saving golden data that you then upload
for staging and testing.
So we've made progress in that direction.
The tools are not ready for primetime.
I think this fits in also with the tools we're using for high
replication data store.
So a lot of effort in that direction, and I think all
these tools will kind of converge really soon.
AUDIENCE: I'm curious if you see adding official support to
any number of open source projects, or if that actually
enables you to use resources for other things that aren't
covered by existing open source projects.
Basically, how do open source projects fit into the grand
scheme of things?
BRETT SLATKIN: I feel like you're talking to me.
Well, we have a bunch of places where we talk about
open source projects.
Especially like Objectify and some other kind of specialized
ORMs that are really--
or like in Japan, there's something called flim3.
That's the biggest thing for writing
web apps on App Engine.
So these are not officially supported.
But the community has built tools that are, for certain
styles of programming, much better than the tools that
we've built, depending on the kind of developer you are.
Like if you're a spring guy, there's some tool that's
better for spring guys.
So I think maybe we should officially support these.
But we have the list of tools, will it work on App Engine,
and stuff like that.
And we try to maintain those kind of
things the best we can.
But maybe a rubber stamp would be useful to you, like
recommended or something like that?
AUDIENCE: Where integration [INTERPOSING VOICES]
a large part--
sorry.
Where integration of projects that still some missing,
functionality gap that maybe is not general enough to meet
everyone's needs.
Things like sessions or RPC.
GREG DALESANDRE: So the other thing I was going to say as
well is that it's hard to say what kind of
official support means.
Although I think we want to encourage the open source
community as much as possible.
We talked earlier about the other open source, some
limitations of App Engine itself.
And I think we like that, we want to encourage it.
It takes some amount of time on our side when we open
source APIs and those sort of things, for instance, and so
it's, of course, all a balance.
But over the course of the year, I think, we'll start to
figure out if there's more things we can do to sort of
encourage some of these projects.
MIKE AIZATSKY: The one thing which I wanted to
[UNINTELLIGIBLE]
by our 20% project is, I wanted to create, and I still
want to create an open source community to fulfill those
smaller parts which do not have their own project.
And I call this operation afterburner, because
afterburner makes you run faster.
And I always tend to [UNINTELLIGIBLE]
produce [UNINTELLIGIBLE]
Google [UNINTELLIGIBLE],
but I want to announce it really soon.
But it's not going to be the place for us.
It's not going to be like official support.
We just want to have one single space for people.
If you've got some use [UNINTELLIGIBLE]
to it, you know there is a place to go to look at, or a
place to contribute it to.
So I want to announce it maybe in a month,
something like that.
AUDIENCE: Hey, if I want to use my new Chromebook for
development of an app engine in the cloud, do I have any
options for that?
BRETT SLATKIN: I think it's control T.
It'll go to the shell, and then you can just--
no.
[LAUGHTER]
*** VON ROSSUM: There will be something.
It's not ready.
SEAN LYNCH: There's a couple of third party ones available
right now, too.
I think Cloud9 just launched sort of an IDE, and they do
deployment into App Engine.
I think if you look around, there's actually a couple of
different options that are sort of cloud-based or
web-based development environments that we'll sort
of hook into App Engine.
AUDIENCE: The Python SDK is 100% open source, but is there
any thought of opening the Java SDK in the same way?
MAX ROSS: Yes.
I believe with 150, there's a little bit of it that's open
source, and I think you can expect to see a lot more in
the next release.
So, yeah.
We want all of it open source, and it's taken us a little
while to get our ducks in a row.
But just that little bit of open source in 150, that means
all of the hard work has been done, and now we've got some
busy work to do to get the rest of it out
there, but it's coming.
SEAN LYNCH: One plug is that the Go run time is already
open sourced.
So they beat the Java guys.
ANDREW DURAND: We are an open source project, so we've got
an unfair advantage.
AUDIENCE: There was quite a huge outage this March.
Who broke Python run time, and what did you do to make sure
that this won't happen again?
[LAUGHTER]
MICHAEL HANDLER: There's always surprises coming.
And you should come to my talk.
And things break in ways that we didn't think that we ever
had to monitor.
And the first thing that we do the minute something happens
that we don't expect, after we fix it, is we say, OK.
How can we detect how that would happen again?
How can we detect those classes of problems
beforehand, before it hits production?
How can we detect that in devel, in QA, in our early
internal instance, so that we find that beforehand and go,
wait, stop.
We broke that.
And let's not make that happen to production.
And we have really good coverage, but reality is
always ready to surprise you.
So that's unfortunately as much as I can say about that
particular incident.
SEAN LYNCH: There is also a public postmortem.
I'm assuming it was like the March 8--?
So there's a public postmortem that we posted as well.
So I don't know if you got a chance to see that.
MAX ROSS: It took us a little bit longer than usual to get
that one out there.
But we take these public postmortems really seriously.
We're not trying to blow smoke in anybody's eyes.
If you think that we are not being transparent enough, we
want to know.
Because we want to be as transparent as we possibly can
with you about what went wrong and we're doing to prevent it
from happening again.
AUDIENCE: Is there any specific reason why the
planned outages, or is that like 5 PM in the area most of
your developers would live?
That's sort of in the middle of our work day, and sort of
affects our revenue at a very serious peak
time for our users.
GREG DALESANDRE: That sounds like someone not running HR.
MATT WILDER: I was going to say two things.
One is, you don't have to worry about
these if you use HR.
Two is that the procedure involved in performing that
maintenance--
we talked about this a little bit in our HR [UNINTELLIGIBLE]
talk--
requires an engineer to do a whole bunch of very specific
steps, and we don't want to be up at five in the morning.
Also, there are things that can go wrong, so we want more
than one person around when those things happen.
GREG DALESANDRE: We also are, you know, people are using App
Engine across the world.
And so it's always going to be inconvenient for someone.
MATT WILDER: It's always 5 PM somewhere, right?
ALFRED FULLER: Also, the data store, which is what causes
these planned maintenances, for the most part, to take an
hour, as we talked to this morning, uses common
infrastructure.
And so if something goes wrong, we want those teams to
be available and around to make sure that you experience
as little down time as possible in those rare
situations.
So one of the reasons why we do that is because it doesn't
just involve us.
It involves a lot of other teams. And in those times,
they're more available than they would be if we did it at
midnight, or some other time.
AUDIENCE: Thank you.
BRETT SLATKIN: I'm just going to plug the
HR data store again.
I have two apps that are very dear to me that I run.
One is on master/slave, and one is on the HR data store.
I launched something on the HR data store an hour before the
maintenance this last week.
I didn't even know.
I had no idea.
And I'm like pushing code and doing everything.
So I think that I've just forgotten about it.
Meanwhile, I'm getting paged on my phone because my other
app was having some issues.
So I think that, just in my own experience, it's totally
changed my frame of mind to have the HR data store.
I've had to work around the ways that it's different, but
really, see if you can move to it.
It's not a gimmick.
It's really serious, what it gives you.
AUDIENCE: I have a follow up question, but I'll
[INAUDIBLE].
AUDIENCE: This is the additional question on OpenID
implementation on App Engine.
You know the single sign on for social networks like
Facebook and Twitter is very small, is you
really look at it.
But the OpenID implementation for Google,
it's a little shaky.
Probably because Google accounts and the Google app
are two different log ins, and if you have to do
a single sign on--
BRETT SLATKIN: So Twitter is everybody but OAuth, and
Facebook is OpenID.
I'm saying, they're not, right?
So could be more specific about what the--
AUDIENCE: For example, if you have a Like button, you know,
once you have logged in into Facebook, you don't have to
really worry about when you go to a different site, click on
Like if you want to be logged in.
But in Google, you don't have [UNINTELLIGIBLE]
in Gmail or in, let's say, you're in a domain like
[UNINTELLIGIBLE],
you don't know which one you are in, right?
So.
BRETT SLATKIN: So this is by design.
This is because if it weren't like that, if there were paths
of login everywhere, then people could drive by, steal
your information.
So they could put a widget in the bottom of a page, you go
to the page, and then you send a request off to their site,
you're automatically logged in, and now they know who you
are, or your ID, et cetera, et cetera.
It works great for Facebook and for a lot of single sign
on things, because they're the center of it, so
they are OK with that.
Publishers pull that too.
But if you apply that to every App Engine app, and there's
200,000 of them, then you've got a security problem.
And so I think that we're trying to strike the right
balance there.
If you can think of some good ideas of how to work around
that, we'd love to hear them.
But that's, I think, the best balance we've
been able to find.
SEAN LYNCH: Specifically on the OpenID implementation,
we've got some--
I don't think they made it to IO, but some changes, or some
new features coming along that way that should also smooth
out some of the integration bits.
So that might help you address your problem, as well.
AUDIENCE: Hi.
When you're deciding how to price App Engine, I'm curious,
going forward, how do you view the typical
customer of app engine?
Given that there's an enormous variance of price sensitivity
between the enterprising folks that would be otherwise
spending millions of dollars on Oracle and then the other
people that are writing the Facebook fad of the week app.
GREG DALESANDRE: So we're relatively transparent about
that by our sort of three different usage types of free
apps, paid apps, and premiere accounts.
Where we really sort of thought about it from the
perspective of free apps are designed to allow people to
try out App Engine, if they want to try out App Engine, or
if they want to use App Engine for a long period of time
without paying anything.
It's designed to do that.
But you know, you get to a point where--
you know, you're all trying to run businesses, and we're
trying to run a business as well, right?
And so the paid app is then, for kind of the profile of it,
we're sort of small people--
not small people, but small businesses.
Like a couple people who were looking to run their apps, but
don't necessarily need operational support, it's OK
for them to go to the list. $9 a month.
I think somebody on the list said that it was two cups of
coffee a month.
Actually, these days, depending where you go, it's
one cup of coffee a month.
But the goal was, it's important that there's a
monthly fee in there to pay for the SLA, and also to
essentially establish a contract with us.
It's very difficult to have an SLA, and a lot of them-- we
got the feedback from customers before where they
really, really wanted to just have a relationship with us,
where they wanted to make sure that we felt accountable to
them, but their usage wasn't that large, even though they
felt like what they were doing was important to them.
And so it's kind of hard, when you're getting something for
free, to come into Google and be like, I can't believe it
went down, this thing that you're letting me do for free.
So the paid apps are really for small businesses, and then
the premier accounts are for enterprises, large businesses,
or even like consulting organizations, those kind of
people, who are using a lot of apps to do a lot
of different things.
AUDIENCE: I guess what I'm wondering is about the part of
the ecosystem that has very high traffic apps with very,
very small margins.
PETER MAGNUSSON: So just to be clear, the price
[UNINTELLIGIBLE]
that we're doing now is targeted at the self-selected
group of current App Engine users.
So it was pretty much entirely driven off of analysis of
current usage, current apps.
And we did extensive--
you know, we're a metric-oriented company.
We did extensive slicing and dicing of all those numbers,
and looked at how this impacted different
kinds of use cases.
And we targeted so that for the vast, vast majority, a
very high percentage of current users, would look at
the price as being, OK, well, this is still a really good
deal, and it's cheaper than any alternative.
There's going to be pathological cases where
you've basically been riding on the fact that we've had a
subsidized service, and that you might have
to look at it again.
I've talked to some developers that have said, well, OK, we
just needed to go in and rearchitect a little bit, and
optimize a little bit, and guess what, then the usage
numbers went down by a certain percentage.
But it should be a very small percentage who are
impacted in that way.
Conversely, for most apps that are paying money, the
consequence is going to be that you're going
to be paying more.
I mean, there's no ifs or buts about that.
But that is also by intention, because we want this to be a
really successful commercial product for Google so we can
build on it.
And it also should not be compared
with unmanaged services.
I've seen some online discussions about, you know,
it's cheaper to do x over here.
When you do the aggregate comparison, you should look at
the aggregate of your whole service, and you should
consider the fact that on App Engine, you're getting a fully
managed service.
So we're happy to look at any particular benchmarks.
We had internal, a dozen or so different models for different
use cases, where we wanted to be competitive on all of them.
So I think at the end of the day, the community is going to
still find it's a really, really good deal.
AUDIENCE: Thank you.
GREG DALESANDRE: So I think we're almost out of time.
So maybe one last question?
AUDIENCE: I've just got a really simple one.
Is there any prospect of being able to map a custom domain to
a specific version of an app?
ALON LEVI: Not at present, no.
But that's an interesting feature [UNINTELLIGIBLE]
we can look into.
Yeah, exactly.
Or have it mapped to a back end, or have it--
Yeah, that's definitely something we'd
like to look into.
GREG DALESANDRE: This is an area where we're actually
looking at right now anyway, of what's the best way to do
some of that.
And so I'm glad you mentioned that.
It was something we hadn't specifically talked about.
ALON LEVI: Good feedback.
Thank you.
GREG DALESANDRE: Thank you.
ALON LEVI: So I guess that's it for us.
Thank you for attending, and feel free to find any of us
afterwards.