Building applications on top of Google Apps

February 25, 2010

Editor's Note: This post was written by Michael Cohn and Steve Ziegler from Cloud Sherpas, a software development and professional services company. SherpaTools™ for Google Apps extends Google Apps capabilities. We invited Cloud Sherpas to share their experiences building an application on top of Google Apps utilizing some of our APIs. Cloud Sherpas will also be participating in the Developer Sandbox at Google I/O this May where they'll be demoing their use of Google technologies and answering questions from attendees.

The Directory Manager module of SherpaTools allows administrators to easily manage User Profiles and Shared Contacts in the Google Apps directory. SherpaTools is built on Google App Engine (GAE) utilizing the Google Web Toolkit (GWT), and makes heavy use of the Google Apps Management and Application APIs to provide new administrator and end-user features.

Some customers have tens of thousands of member accounts in their domain. SherpaTools Directory Manager makes it easy to retrieve, edit, and manage user accounts. It also allows you to import or export data in bulk from a Google Docs spreadsheet. This post explains how Directory Manager works with Google Apps, and how we built it.

Authentication to SherpaTools is achieved by using Google Apps as the OpenID identity provider. If a user is already logged into Google Apps, they can access SherpaTools without ever providing SherpaTools their credentials. This Single Sign-On experience greatly enhances user adoption and provides an added security benefit. If the user is not already logged into Google Apps, they are routed to a Google login page. With OpenID, SherpaTools never handles the user's credentials.

Once logged in, SherpaTools securely requests authorization from Google Apps using 2-legged OAuth (2LO) to make API service calls on behalf of the user. Since the app has both an end-user and administrator view, it first retrieves the logged in user's information from Google Apps via a 2LO-authorized call to the UserService of the Provisioning API. Depending on the information sent in the API response, the user is either presented with the administrator application or the end-user screen.

Two-legged OAuth (2LO) allows 3rd-party applications like SherpaTools to make authorized API calls to Google Apps on behalf of a user. Here is how we set up our Google Data API ContactsService that will be fetching User Profiles to use 2LO:

ContactsService contactsService =
   new ContactsService(GlobalConstants.APPLICATION_NAME);
GoogleOAuthParameters parameters = new GoogleOAuthParameters();
parameters.setOAuthConsumerKey(GlobalConstants.CONSUMER_KEY);
parameters.setOAuthConsumerSecret(GlobalConstants.CONSUMER_SECRET);
OAuthHmacSha1Signer signer = new OAuthHmacSha1Signer();
try {
   contactsService.setOAuthCredentials(parameters, signer);
} catch (OAuthException e) {
   // not expected if secret is up-to-date
}

As long as our key/secret pair is correct and the Google Apps customer has entitled our OAuth key to have access to their Contacts API feed, Google authorizes SherpaTools to continue to make API calls. There are two other settings that should be mentioned in configuring the service to work well on GAE. First, since we are dealing with somewhat sensitive data, all calls to Google Apps are made over SSL. To ensure this, we simply set the useSSL flag for the contacts service. Next, the default request/response timeout on GAE for these API calls is only five seconds out of a possible ten. Since we will be retrieving as much data as we can within that ten second window to reduce the total number of operations to complete the work, we raise our connection timeout up to just short of that maximum, 9500 milliseconds:

contactsService.useSsl();
contactsService.setConnectTimeout(9500);

Cloud Sherpas embraced a number of Google Web Toolkit best practices to ensure scalability of SherpaTools. For example, once the app determines which screen the user should see, SherpaTools employs GWT CodeSplitting to optimize and reduce the amount of javascript that needs to be downloaded by the browser client. The app also uses the GWT RPC framework designed according to the command pattern to transparently communicate with the server, and was architected using the model-view-presenter (MVP) design pattern to allow multiple developers to work on the app simultaneously.

After a Google Apps administrator logs into SherpaTools for the first time, the app caches some key information for better performance. For example, to populate the User Profile and Shared Contacts lists, the app retrieves the IDs and names of all contacts using the User Profiles API and Shared Contacts API respectively, and writes this information to the data store and memcache. And for domains with large data sets, SherpaTools uses task queues to break up operations into smaller chunks.

Since we have to scale the export to handle the contact information of tens of thousands of contact entries, there is no way we can retrieve all of those entries in one request. Retrieval of this set of information requires breaking the operation up into smaller sub-tasks. Fortunately, both the GAE Task Queue API and the Google Datastore APIs make it easy to divide the retrieval into smaller chunks.

GAE Task Queues enable background queueing of HTTP operations (GET, POST, etc.) to arbitrary URLs within our application. Each of these queued operations are subject to the same restrictions of any other GAE HTTP operation request. Of particular note is the aforementioned 10 second window to perform our remote service call and the overall 30 second window to complete the total work within a task request. Also, since the Task Queue works from the same set of URLs as are made available to the rest of our application, we need to make sure that there is no ability for unwanted external attempts to execute tasks. We followed the recommended way of eliminating this possibility by restricting outside access to just our application's admins.

This constraint restricts all urls starting with /task/ to only be accessible either from system calls such as from the Task Queue or by admins. The NONE transport guarantee is also important to mention. We initially attempted to encrypt our task calls using SSL with a transport guaranteed of CONFIDENTIAL, but, at the time we attempted this, execution ceased to function properly. Since all of the traffic of all of these calls are strictly on Google's internal network we had no issue with making these calls without SSL.

Now that we have our tasks properly secured, we can create a method for sending our User Profiles fetch task request to the Task Queue:

public void fetchUserProfilesPageTask(String spreadsheetTitle,
   String loggedInEmailAddress, String nextLink, String memcacheKey) {
  Queue queue = QueueFactory.getQueue(USER_PROFILES_QUEUE);
  TaskOptions options =
   TaskOptions.Builder.url("/task/"+USER_PROFILES_FETCH_URL);
  options.param("spreadsheetTitle", spreadsheetTitle);
  options.param("loggedInEmailAddress", loggedInEmailAddress);
  options.param("nextLink", nextLink);
  options.param("memcacheKey", memcacheKey);
  queue.add(options);
}

The url points to a Java HttpServlet that handles the task's HTTP POST, parses the sent parameters, and calls a method to perform the work:

protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
   String loggedInEmailAddress = req.getParameter("loggedInEmailAddress");
   String spreadsheetTitle = req.getParameter("title");
   String nextLink = req.getParameter("nextLink");
   String memcacheKey = req.getParameter("memcacheKey");
   // do the work:
   fetchUserProfilesPage(spreadsheetTitle, loggedInEmailAddress, nextLink, memcacheKey);
}

Summary

This post explains how SherpaTools Directory Manager uses Two Legged OAuth for authentication, and GAE Task Queue API and the Google Datastore APIs to make it easy to divide large retrievals over long intervals into smaller chunks. Other long-running, divisible operations, can use this same approach to spread work across a string of tasks queued in the GAE Task Queue. We would love to hear what you think of this approach and if you have come up with your own solution for similar issues.

Next week we will discuss how we used the User Profile API to retrieve the data sets and the Document List Data API to populate a Google Spreadsheet.

Thanks to the Cloud Sherpas team for authoring this post. Check out SherpaTools at www.sherpatools.com

Using Google Document List APIs to build Memeo Connect for Google Docs

February 16, 2010

Building Memeo Connect™ for Google Docs

Editor's Note: This post was written by our partner Memeo, a digital media management company. Memeo Connect™ for Google Docs enhances Google Docs capabilities. We invited Memeo to share their experiences building an application on top of Google Apps utilizing some of our APIs. Other partners will share their experiences in future posts.

Hi I'm Matthew Tonkin, Mac Applications Architect at Memeo. Google has been a great partner in helping us bring Memeo Connect to market. We're excited about the new additions to the Google Documents List API's and the products we've been working on for them.

Google Apps is a great way for businesses to share and collaborate on documents, but more importantly, it has allowed businesses to move much of their office and IT infrastructure online for dramatic operating cost reductions. Until recently, Google Docs users had no way of uploading, storing, syncing and sharing files of any type. The big news is that Google Docs now supports all file types and there are Documents List API's to prove it. January saw the release of the updated Documents List API with this new arbitrary file support, but no desktop client software to manage file syncing between the desktop and the cloud. Memeo Connect for Google Apps is that missing link. Memeo set out to bridge Google Docs cloud storage with desktop software. A simple goal that has big implications for where users store documents and how they are shared.

The end result is a desktop application that is fully supported by Google Docs for online storage, synchronization, sharing and data management. With Memeo Connect, instant access to your important documents while online or offline is now very possible. Memeo Connect is available on both Windows and Mac platforms.

Our timeline for Memeo Connect was 'impossibly' short due to Google’s aggressive timeline for launch. How did we do it? Apart from the obvious late nights, junk food and beer, we received help from Google to take this product from concept to reality. Some of it came in the form of new API's, some directly from the Documents List API team, some from the client libraries and some from the discussion groups.

1. Keeping native file formats

Google Docs now supports keeping files in their native format. Previously Microsoft Office files and images were converted to the online Google Document formats, removing much of the application specific formatting options used in the Office file formats. However, it's now possible to upload Microsoft Office files and images without conversion, meaning users can keep all of their custom file formatting.

You can get all of this with a simple parameter in the post URL when uploading the new document.

Google Documents List API

https://docs.google.com/feeds/default/private/full?convert=false

If like us, you use the client libraries, then it's just as easy.

Google Data Objective-C client library**

NSURL *uploadURL = [[mDocListFeed postLink] URL];
GDataQueryDocs *uploadQuery = [GDataQueryDocs queryWithFeedURL:uploadURL];
[uploadQuery setShouldConvertUpload:NO];

Google Data .NET client library

string uploadURL = DocumentsListQuery.documentsBaseUri + "?convert=false";
docService.StreamSend(new Uri(uploadURL), fileStreamHandle, GDataRequestType.Insert, contentType, docTitle);

The new features don't just handle Microsoft Office files. Any file type is supported with no additional parameters or changes to client code, however the support for these features is only available through the Google Documents List API for users that have a Premier account.

2. Bigger file size limits and resumable uploads

Google Docs now supports file uploads up to 1GB and is resumable, so failed or paused uploads can pick up where they left off.

Google Documents List API
http://code.google.com/apis/documents/docs/3.0/developers_guide_protocol.html#ResumableUpload

Resumable post link:

resumable-create-media" type="application/atom+xml"
   href="https://docs.google.com/feeds/upload/create-session/default/private/full"/>

Request:

POST /feeds/upload/create-session/default/private/full HTTP/1.1
Host: docs.google.com
GData-Version: 3.0
Authorization:
Content-Length: 0
Content-Type: application/pdf
Slug: MyTitle
X-Upload-Content-Type: application/pdf
X-Upload-Content-Length: 1234567

Resuming:

POST HTTP/1.1
Host: docs.google.com
Content-Length: 100000
Content-Type: application/pdf
Content-Range: bytes 0-99999/1234567

Google Data Objective-C client library**

Resumable uploads are automatically enabled when switching to the uploadLink URL

NSURL *uploadURL = [[mDocListFeed uploadLink] URL];
GDataQueryDocs * uploadQuery = [GDataQueryDocs queryWithFeedURL:uploadURL];

Pausing and resuming uploads is also made easy.

[mUploadTicket pauseUpload];
[mUploadTicket resumeUpload];

Google Data .NET client library

ResumableUploader resumableUploader = new ResumableUploader();
if (newFile)
   resumableUploader.Insert(authentication, docEntry);
else if (partialFile)
   resumableUploader.Resume(authentication, resumeUri, httpMethod, fileStream, cotentType);
else if (updateExistingFile)
   resumableUploader.Update(authentication, docEntry);

** Objective-C code samples have been taken from the Docs Sample code

3. Client Libraries for Mac & Windows

The Google Documents List API provides all of the backend server functionality but isn't ideal for rapid client application development. As client developers, we prefer the efficiencies provided by a wrapper in our native client languages. This is where the client libraries for Objective-C and .NET allowed us to shorten our application development time significantly. Without the client libraries, we simply would not have been able to achieve our goals for Memeo Connect in the time we had available.

Many of the Documents List API features we worked with in our early development were not available through the client libraries. This was initially quite daunting because of the risk that we would have to drop down to raw server calls and miss out on the efficiencies we gained through using the client libraries. However as we got to know the client libraries better, we found they were written flexibly enough that we never had to do that - there was always a way to bend them enough to get what we needed. A sign of great design by their architects.

For example uploading without conversion was simple in the Objective-C client library even before it was officially updated to support it.

NSURL *uploadURL = [[mDocListFeed postLink] URL];
GDataQueryDocs *uploadQuery = [GDataQueryDocs queryWithFeedURL:uploadURL];
[uploadQuery addCustomParameterWithName:@"convert" value:@"false"];

4. Support from Google

The Documents List API team were of enormous help throughout the course of this project. As with most of Google's public API's, there's always an avenue to ask questions about how to best use the technology, pursue bugs or request new features. The new Google Apps Discussion Groups are simply the best way to get an answer quickly and is invaluable in getting past whatever is blocking your progress. Google Developer Relations people monitor the discussion groups and frequently answer questions, link to other relevant resources, and provide code samples.

What's next?

We had a sizable list of things we really wanted to do with Memeo Connect 1.0 and some of that had to wait for 1.x or 2.0. We're looking forward to continuing to work with Google and file as many feature requests as we can to make some of these new features a reality. Anyone can file a feature request for Google Apps API's and I can assure you they all get considered.

In future releases we'll be adding more support for direct file system integration, better tools to manage sharing and we're really excited about supporting new devices. Get ready to see more Google Docs in more places.

New Google App Engine SDK 1.3.1 released

February 11, 2010

announcedfull blog post
Today we announce the release of version 1.3.1 of the App Engine SDK for both Python and Java. While this release contains plenty of new features and fixes, we've concentrated on using our very first SDK release of 2010 to improve the heart of many App Engine applications: the Datastore.

Here are the three major improvements that 1.3.1 has in store for datastore users:

Datastore Query Cursors - Cursors allow applications to save and 'bookmark' their progress through a query, so that it can be resumed later. This works great in combination with paging URLs, as well as processing in the Task Queue API, but there are many other uses. Watch for an upcoming blog post that explores Cursors in the near future. They're also really handy in the context of the next change...
No more 1000 result limit - That's right: with addition of Cursors and the culmination of many smaller Datastore stability and performance improvements over the last few months, we're now confident enough to remove the maximum result limit altogether. Whether you're doing a fetch, iterating, or using a Cursor, there's no limits on the number of results.
Reduced error rate with Automatic Datastore Retries - We've heard a lot of feedback that you don't want to deal with the Datastore's sporadic errors. In response, App Engine now automatically retries all datastore calls (with the exception of transaction commits) when your applications encounters a datastore error caused by being unable to reach Bigtable. Datastore retries automatically builds in what many of you have been doing in your code already, and our tests have shown it drastically reduces the number of errors your application experiences (by up to 3-4x error reduction for puts, 10-30x for gets).

Read the full post here.

Follow @GoogleAppsDev on Twitter.

Control your Google Sites from Apps Script

February 10, 2010

Google Apps Scriptjust launchedHenry Lau, on the Google Apps Script team, has written a great tutorial on Creating a Soccer Club Homepage. In short, his script pulls upcoming Soccer matches from Google Calendar, creates a Google Site about the team and adds the Soccer team from Contacts as collaborators. Here's some snippets of the code.// create a site, given the domain, site URL, site title and summary

var site = SitesApp.createSite("example.com", "rover", "Team Rover", "We'll be the divisional champs this year!");

// create a page for each member of the team
var webpage = site.createWebPage("Jimmy's Page", "JimmysPage", "Welcome to Jimmy's site");

// create a page for team announcements
var annPage = site.createAnnouncementsPage("Team Announcements", "Announcements", "New announcements for the team will be posted here.");

// given one of the Google Calendar events, create an announcement

var message = "There will be a soccer match from " + event.getStartTime() + " until " + event.getEndTime() + "!";
site.createAnnouncement("Soccer Match #1", message, annPage);

It's that easy! Check out the documentation and get started building your scripts.Of course, if you decide that you'd rather create Google Sites via a REST interface, the recently-announced enhancements to the Sites Data API will allow you to write code in your language of choice to create and manage sites.Developer Relations - Google Apps

Google Apps Developer Blog - for developers, by developers

February 8, 2010

code snippets and samples

reviews of customer integration and deployment cases

interviews with developers on best practices for developing in Apps

voting on most-requested developer extensions in Apps

discussion of OAuth roadmap

references to OpenID

smart ways to do logging (and analysis/reporting) in AppEngine, etc.

Storing JSON in AppEngine

register for Google I/OGoogle Apps Discussion ForumGoogle Apps Client Libraries and Sample CodeGoogle Apps API OverviewGoogle Apps API Help ForumGoogle Enterprise Blog

G Suite Developers Blog

Building applications on top of Google Apps

Using Google Document List APIs to build Memeo Connect for Google Docs

New Google App Engine SDK 1.3.1 released

Control your Google Sites from Apps Script

Google Apps Developer Blog - for developers, by developers

Free Trial

Labels

Archive

Feed

Company-wide

Products

Developers