Bridging OAuth 2.0 objects between GData and Discovery

My colleague +Takashi Matsuo and I recently gave a talk about using OAuth2Decorator (from the google-api-python-client library) with request handlers in Google App Engine. Shortly after, a Stack Overflow question sprung up asking about the right way to use the decorator and, as a follow up,  if the decorator could be used with the Google Apps Provisioning API. As I mentioned in my answer,

The Google Apps Provisioning API is a Google Data API...As a result, you'll need to use the gdata-python-client library to use the Provisioning API. Unfortunately, you'll need to manually convert from a oauth2client.client.OAuth2Credentials object to a gdata.gauth.OAuth2Token object to use the same token for either one.

Instead of making everyone and their brother write their own, I thought I'd take a stab at it and write about it here. The general philosophy I took was that the token subclass should be 100% based on an OAuth2Credentials object:

• the token constructor simply takes an OAuth2Credentials object
• the token refresh updates the OAuth2Credentials object set on the token
• values of the current token can be updated directly from the OAuth2Credentials object set on the token

Starting from the top, we'll use two imports:

import httplib2
from gdata.gauth import OAuth2Token

The first is needed to refresh an OAuth2Credentials object using the mechanics native to google-api-python-client, and the second is needed so we may subclass the gdata-python-client native token class.

As I mentioned, the values should be updated directly from an OAuth2Credentials object, so in our constructor, we first initialize the values to None and then call our update method to actual set the values. This allows us to write less code, because, repeating is bad (I think someone told me that once?).

class OAuth2TokenFromCredentials(OAuth2Token):
  def __init__(self, credentials):
    self.credentials = credentials
    super(OAuth2TokenFromCredentials, self).__init__(None, None, None, None)
    self.UpdateFromCredentials()

We can get away with passing four Nones to the superclass constructor, as it only has four positional arguments: client_id, client_secret, scope,  and user_agent. Three of those have equivalents on the OAuth2Credentials object, but there is no place for scope because that part of the token exchange handled elsewhere (OAuth2WebServerFlow) in the google-api-python-client library.

  def UpdateFromCredentials(self):
    self.client_id = self.credentials.client_id
    self.client_secret = self.credentials.client_secret
    self.user_agent = self.credentials.user_agent
    ...

Similarly, the OAuth2Credentials object only implements the refresh part of the OAuth 2.0 flow, so only has the token URI, hence auth_uri, revoke_uri and redirect_uri will not be set either. However, the token URI and the token data are the same for both.

    ...
    self.token_uri = self.credentials.token_uri
    self.access_token = self.credentials.access_token
    self.refresh_token = self.credentials.refresh_token
    ...

Finally, we copy the extra fields which may be set outside of a constructor:

    ...
    self.token_expiry = self.credentials.token_expiry
    self._invalid = self.credentials.invalid

Since OAuth2Credentials doesn't deal with all parts of the OAuth 2.0 process, we disable those methods from OAuth2Token that do.

  def generate_authorize_url(self, *args, **kwargs): raise NotImplementedError
  def get_access_token(self, *args, **kwargs): raise NotImplementedError
  def revoke(self, *args, **kwargs): raise NotImplementedError
  def _extract_tokens(self, *args, **kwargs): raise NotImplementedError

Finally, the last method which needs to be implemented is _refresh, which should refresh the OAuth2Credentials object and then update the current GData token after the refresh. Instead of using the passed in request object, we use one from httplib2 as we mentioned in imports.

  def _refresh(self, unused_request):
    self.credentials._refresh(httplib2.Http().request)
    self.UpdateFromCredentials()

After refreshing the OAuth2Credentials object, we can update the current token using the same method called in the constructor.

Using this class, we can simultaneously call a discovery-based API and a GData API:

from apiclient.discovery import build
from gdata.contacts.client import ContactsClient

service = build('calendar', 'v3', developerKey='...')

class MainHandler(webapp2.RequestHandler):
  @decorator.oauth_required
  def get(self):
    auth_token = OAuth2TokenFromCredentials(decorator.credentials)
    contacts_client = ContactsClient()
    auth_token.authorize(contacts_client)
    contacts = contacts_client.get_contacts()
    ...
    events = service.events().list(calendarId='primary').execute(
        http=decorator.http())
    ...

About Bossy Lobster

A Decorator for App Engine Deferred Tasks

I happen to be a big fan of the deferred library for both Python runtimes in Google App Engine. If an application needs to queue up work, breaking the work into easy to understand units by writing worker methods and then deferring the work into tasks is a breeze using the deferred library. For the majority of cases, if fine grained control over the method of execution is not needed, using the deferred library is a great (and in my opinion, the correct) abstraction.

Maybe I am just biased because I have made a few changes to the deferred library over the past few months? One such change I made added a feature that allows a task to fail once without having an impact on subsequent retries; this can be accomplished by raising a SingularTaskFailure. Over this weekend, I found that I wanted to use this feature for a special type* of worker. Since I wanted to utilize this unique exception, I wanted to make sure that this worker only ran in a deferred task.

Initially I thought I was lost, since any pickled method wouldn't directly have access to the task queue specific headers from the request. But luckily, many of these headers persist as environment variables, so can be accessed via os.environ or os.getenv, yippee! Being a good little (Python) boy, I decided to abstract this requirement into a decorator and let the function do it's own work in peace.

Upon realizing the usefulness of such a decorator, I decided to write about it, so here it is:

import functools
import os

from google.appengine.ext.deferred import defer
from google.appengine.ext.deferred.deferred import _DEFAULT_QUEUE as DEFAULT_QUEUE
from google.appengine.ext.deferred.deferred import _DEFAULT_URL as DEFERRED_URL

QUEUE_KEY = 'HTTP_X_APPENGINE_QUEUENAME'
URL_KEY = 'PATH_INFO'

def DeferredWorkerDecorator(method):
  @functools.wraps(method)
  def DeferredOnlyMethod(*args, **kwargs):
    path_info = os.environ.get(URL_KEY, '')
    if path_info != DEFERRED_URL:
      raise EnvironmentError('Wrong path of execution: {}'.format(path_info))
    queue_name = os.environ.get(QUEUE_KEY, '')
    if queue_name != DEFAULT_QUEUE:
      raise EnvironmentError('Wrong queue name: {}'.format(queue_name))

    return method(*args, **kwargs)

  return DeferredOnlyMethod

This decorator first checks if the environment variable PATH_INFO is set to the default value for the deferred queue: /_ah/queue/deferred. If this is not the case (or if the environment variable is not set), an EnvironmentError is raised. Then the environment variable HTTP_X_APPENGINE_QUEUENAME is checked against the name of the default queue: default. Again, if this is incorrect or unset, an EnvironmentError is raised. If both these checks pass, the decorated method is called with its arguments and the value is returned.

To use this decorator:

import time

from google.appengine.ext.deferred import SingularTaskFailure

@DeferredWorkerDecorator
def WorkerMethod():
  if too_busy():
    time.sleep(30)
    raise SingularTaskFailure

   # do work

WorkerMethod()  # This will fail with an EnvironmentError
defer(WorkerMethod)  # This will perform the work, but in it's own task

In case you want to extend this, here is a more "complete" list of some helpful values that you may be able to retrieve from environment variables:

HTTP_X_APPENGINE_TASKRETRYCOUNT
HTTP_X_APPENGINE_QUEUENAME
HTTP_X_APPENGINE_TASKNAME
HTTP_X_APPENGINE_TASKEXECUTIONCOUNT
HTTP_X_APPENGINE_TASKETA
HTTP_X_APPENGINE_COUNTRY
HTTP_X_APPENGINE_CURRENT_NAMESPACE
PATH_INFO

*Specialized Worker: I had two different reasons to raise a SingularTaskFailure in my worker. First, I was polling for resources that may not have been online, so wanted the task to sleep and then restart (after raising the one time failure). Second, I was using a special sentinel in the datastore to determine if the current user had any other job in progress. Again, I wanted to sleep and try again until the current user's other job had completed.

A Python Metaclass for "extra bad" errors in Google App Engine

So now here we are, having tried to handle errors in Google App Engine...and failed all because silly DeadlineExceededError jumps over Exception in the inheritance chain and goes right for BaseException. How can we catch these in our handlers while staying Pythonic*?

First and foremost, in the case of a timeout, we need to explicitly catch a DeadlineExceededError. To do so, we can use a decorator (hey, that's Pythonic) in each and every handler for each and every HTTP verb. (Again, prepare yourselves, a bunch of code is about to happen. See the necessary imports at the bottom of the post.)

def deadline_decorator(method):

    def wrapped_method(self, *args, **kwargs):
        try:
            method(self, *args, **kwargs)
        except DeadlineExceededError:
            traceback_info = ''.join(format_exception(*sys.exc_info()))
            email_admins(traceback_info, defer_now=True)

            serve_500(self)

    return wrapped_method

Unfortunately, having to manually

is not so Pythonic. At this point I was stuck and wanted to give up, but asked for some advice on G+ and actually got what I needed from the all knowing Ali Afshar. What did I need? Metaclasses.

Before showing the super simple metaclass I wrote, you need to know one thing from StackOverflow user Kevin Samuel:

The main purpose of a metaclass is to change the class automatically, when it's created.

With the __new__ method, the type object in Python actually constructs a class (which is also an object) by taking into account the name of the class, the parents (or bases) and the class attritubutes. So, we can make a metaclass by subclassing type and overriding __new__:

class DecorateHttpVerbsMetaclass(type):

    def __new__(cls, name, bases, cls_attr):
        verbs = ['get', 'post', 'put', 'delete']
        for verb in verbs:
            if verb in cls_attr and isinstance(cls_attr[verb], function):
                cls_attr[verb] = deadline_decorator(cls_attr[verb])

        return super(DecorateHttpVerbsMetaclass, cls).__new__(cls, name,
                                                              bases, cls_attr)

In DecorateHttpVerbsMetaclass, we look for four (of the nine) HTTP verbs, because heck, only seven are supported in RequestHandler, and we're not that crazy. If the class has one of the verbs as an attribute and if the attribute is a function, we decorate it with deadline_decorator.

Now, we can rewrite our subclass of RequestHandler with one extra line:

class ExtendedHandler(RequestHandler):
    __metaclass__ = DecorateHttpVerbsMetaclass

    def handle_exception(self, exception, debug_mode):
        traceback_info = ''.join(format_exception(*sys.exc_info()))
        email_admins(traceback_info, defer_now=True)

        serve_500(self)

By doing this, when the class ExtendedHandler is built (as an object), all of its attributes and all of its parent classes (or bases) attributes are checked and possibly updated by our metaclass.

And now you and James Nekbehrd can feel like a boss when your app handles errors.

Imports:

from google.appengine.api import mail
from google.appengine.ext.deferred import defer
from google.appengine.ext.webapp import RequestHandler
from google.appengine.runtime import DeadlineExceededError
import sys
from traceback import format_exception
from SOME_APP_SPECIFIC_LIBRARY import serve_500
from LAST_POST import email_admins

*Pythonic:

An idea or piece of code which closely follows the most common idioms of the Python language, rather than implementing code using concepts common to other languages.

Notes:

• Using grep -r "Exception)" . | grep "class " I have convinced myself (for now) that the only errors AppEngine will throw that do not inherit from Exception are DeadlineExceededError, SystemExit, and KeyboardInterrupt so that is why I only catch the timeout.
• You can also use webapp2 to catch 500 errors, even when handle_exception fails to catch them.

Disclaimer: Just because you know what a metaclass is doesn't mean you should use one:

• "Don't do stuff like this though, what is your use case?" -Ali Afshar
• "Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why)." -Python Guru Tim Peters
• "The main use case for a metaclass is creating an API." -Kevin Samuel

About Bossy Lobster