Google Prediction API and all apps

Now your apps can get smarter with as little as a single line of code. They can learn to continually adapt to changing conditions and to integrate new information. This week at Google I/O, we’re making the Google Prediction API generally available, meaning you can create apps with these capabilities for yourself. Additionally, we’re introducing several significant new features, including:

  • The ability to stream data and tune your predictive models
  • A forthcoming gallery of user-developed, pre-built models to add smarts even faster.

The Google Prediction API can be used by almost any app to recommend the useful, extract the essential, and automate the repetitive. For example:

  • Recommend a new movie to a customer.
  • Identify most important customers.
  • Automatically tag posts with relevant flags.

For example, Ford Motor Co. Research is working to use the Prediction API to optimize plug-in hybrid vehicle fuel efficiency by optionally providing users with likely destinations to choose from, and soon, optimizing driving controls to conserve fuel. Because the API is a cloud-hosted RESTful service, Ford has been able to access its computationally-intensive machine learning algorithms to find patterns that rank potential destinations based on previous driving paths. Ford will be demonstrating their work at the API’s I/O Session.

Here’s a summary of the features we added to the API today:

  • Streaming training data: Continually incorporate feedback for fast-adapting systems (e.g. user-chosen tags vs predicted ones, final purchases vs expected).
  • General availability: Anyone can now sign up to use the API. Paid users also receive a 99.9% SLA with increased quota.
  • New JavaScript library: Now deploy the Prediction API in your JavaScript – in addition to our updated Python and Java libraries.

Today, we are also announcing the Prediction API’s forthcoming gallery of pre-trained third party predictive models (try these demo models right now), and we will be adding more constantly (maybe yours – waitlist). Once complete, all Prediction API users will be able to:

  • Subscribe to others’ models: improve your apps with others’ predictive data tools.
  • Sell access to your models (e.g. sentiment analysis on social media).
  • Import customized models through the open-standard PMML encoding.

See our recent blog post for even more ideas, and get started at the Google APIs Console.

Thanks to our community of preview developers, who have played a crucial role in helping us make the Google Prediction API simpler and more powerful since its announcement last year at I/O 2010. We are thrilled to invite all developers to join them.

Announcing App Engine 1.4.3. Release

Just in time for spring, we have a new App Engine release to bring our Java and Python runtimes even closer to parity. For Python, we’ve launched a test library to match the existing Java testing framework, and for Java we’ve introduced the Deferred and Remote APIs. This release also introduces a new Blobstore writing feature, the experimental release of the Prospective Search API in Python, and a few goodies for Task Queue and Cron users.

Python

  • Prospective Search API: The experimental Prospective Search API allows Python runtime users to detect and take action on datastore entities that match certain criteria when they are written. For the experimental release, users will be allowed 10,000 subscriptions with the Prospective Search API. Pricing will be announced once the feature is fully launched.
  • Testbed Unit Test Framework: The Testbed suite for Python provides an easy interface for using App Engine API stubs in integration tests similar to the previously existing Java Testing Framework. You can create tests for your application that do not rely on calling App Engine production services, which speeds up the time your tests take to complete, and eliminates dependencies for your test on external services. Here at Google, we’re well known for our testing culture, and we hope that this API will allow you to develop more stable code more quickly.

Java

  • Concurrent Requests: Until now, Java applications relied on starting additional instances to dynamically scale up for higher traffic levels. Now with support for concurrent requests, each application instance may serve multiple user requests at the same time. To start, ensure your application’s code is threadsafe, then enable concurrent requests by adding the flag to your appengine-web.xml.
  • Java Remote API and Deferred API support: The Remote API and Deferred API libraries have been supported in Python for awhile, and now they can be used with Java, too! The Remote API allows you to perform operations on your application’s datastore from your local machine. This is particularly useful for work that is not well-suited to App Engine’s request/response model. The Deferred API allows users to more easily write and execute ad hoc tasks. Our docs contain more information and examples on how to use the Remote API and Deferred API in Java App Engine.

New and changed APIs

  • Files API: The new Files API in Python and Java allow you to pragmatically read and write data using Blobstore. This API can be used to generate reports, export data, or do anything that your heart desires that requires large, binary objects.

Serving Changes

  • Task Queue and Cron update: We’ve addressed some of your top requested items for Task Queues and Cron. With this release, you can now configure the specific application version to which a task queue or cron job will send requests. For those wanting to schedule cron jobs with a range syntax like “every 5 minutes from 11:00 to 17:00”, that too is available in the 1.4.3 release. Last, but not least, the Admin Console Task Queues page now displays a more accurate estimate of queue size for queues containing more than 2000 tasks.

Coming soon

Finally, we have a pre-announcement about the 1.4.4 release. In 1.4.0, we introduced a feature that allowed users to download code that they’ve deployed to App Engine. In 1.4.2 we released admin roles allowing for Owners. When 1.4.4 is released, we plan on including a change that allows both the person who uploaded the code to download it, as well as the Owner(s) of the project (as listed in the Admin Console). In preparation for this, be sure to properly assign roles to all developers on your App Engine application in the Admin Console. Alternatively, you will still be able to permanently disable the code download feature for your application.

That’s it for now, for additional information read the full release notes, including all new features and issues fixed for Java and Python. All feedback is welcome and encouraged in our groups.

Enomaly chooses Google App Engine for SpotCloud.com


SpotCloud.com is the realization of a dream originating more than 7 years ago when we first started Enomaly. SpotCloud was born from a desire to change the way cloud computing is bought and sold for the better by overcoming the inefficiencies (including large capital commitments and low utilization rates) that beset the traditional models. SpotCloud is the first multi-supplier, multi-buyer model of cloud computing, all built on Google App Engine.

SpotCloud is the first marketplace where service providers can sell their unutilized or under-utilized IaaS cloud services, and where buyers can shop competitively for these services on the basis of quality, price, and geography across a federated global pool of capacity with a single, consistent set of access and management mechanisms.

SpotCloud provides its users with the following main services:

  • VM image repository & VM image provisioning services
  • REST API for cloud service users, with management and monitoring functionality
  • REST API for cloud service providers
  • REST client to access service provider API endpoints
  • Single-point unified access & authentication to all underlying cloud services
  • Federated management portal, providing management & monitoring over all connected cloud services
  • Cloud service quality, reliability, and performance measurement instrumentation and rating
  • Runtime management service (to enable time-limited instances)
  • Billing, payment, and settlement services

Why Google App Engine?

We realized early on that a traditional data center infrastructure — even one built on our own ECP technology — was not a fit for a platform which needs to be global from day one. We also had a strong preference for Python, given that large parts of our existing IaaS software stack is built using it. Cost was also very important: SpotCloud needs to manage potentially very large Virtual Machines, delivering them to a globally distributed group of providers and managing them in near-real time across this extended footprint. With Google App Engine we have the power, flexibility, and global breadth of the Google infrastructure at our disposal. This was a key selling feature for us. Also the cost was practically impossible to beat.

How App Engine powers SpotCloud

As you can see from the list of services provided by SpotCloud, most of our application actually runs in the background. This is made possible by App Engine task queues and cron jobs. We need to poll for stage changes across hundreds (soon thousands) of service providers — we need a near-real-time view of instance states, available virtual hardware profiles, cloud utilization data, etc. These things are triggered by cron schedules. Each of these tasks goes into a queue. We’re able to execute these tasks to allow us to keep our view of service provider resources up to date without interrupting the front-end UI and API workflows.

Another challenge is executing tasks at a particular point in the future. Running instances deployed out to a provider through SpotCloud may expire, and renewal may or may not be allowed depending on the seller’s settings. So we need a way to terminate them if their time is up. We can use the task queue API to execute this in the future. We’re able to ensure our data is consistent by adding tasks to the queue in a transactional way.

Buyers upload the appliances they want to provision to the market. So we obviously have a big storage requirement. App Engine’s data store allows us to do this in a highly performant, cost-effective manner. SpotCloud can then distribute these appliances out to providers on demand. We’re able to do this because of the low-cost bandwidth Google provides.

We’re big fans of Django at Enomaly, and we use it in SpotCloud. We’ve been able to loosely couple Django with the App Engine data store, along with other App Engine components and services like urlfetch. This means that we’re not “locked-in” with App Engine. The App Engine API is structured in such a way that we’re able to extend any web framework we want with App Engine components that help us scale.

SpotCloud Success

Since announcing the SpotCloud private beta at the end of last year, we’ve been overwhelmed by the amount of interest we’ve gathered, and have had large numbers of service providers from all over the globe join our beta. We now have more than 10,000 servers from 40+ countries represented in the marketplace.

Since we opened the marketplace to buyers and sellers early in 2011, the market has been off to a roaring start — as noted by The Economist, ComputerWorld, and many others. Google App Engine has performed up to our (high!) expectations, serving an ever-increasing transaction rate in the market without a hiccup. We’re looking forward, as the market ramps up even further, to seeing if we can make App Engine break a sweat! Sign up today as a cloud capacity buyer or seller at www.spotcloud.com.