Showing posts with label Pythonic. Show all posts
Showing posts with label Pythonic. Show all posts

Wednesday, November 30, 2011

A Python Metaclass for "extra bad" errors in Google App Engine

So now here we are, having tried to handle errors in Google App Engine...and failed all because silly DeadlineExceededError jumps over Exception in the inheritance chain and goes right for BaseException. How can we catch these in our handlers while staying Pythonic*?

First and foremost, in the case of a timeout, we need to explicitly catch a DeadlineExceededError. To do so, we can use a decorator (hey, that's Pythonic) in each and every handler for each and every HTTP verb. (Again, prepare yourselves, a bunch of code is about to happen. See the necessary imports at the bottom of the post.)
def deadline_decorator(method):

    def wrapped_method(self, *args, **kwargs):
        try:
            method(self, *args, **kwargs)
        except DeadlineExceededError:
            traceback_info = ''.join(format_exception(*sys.exc_info()))
            email_admins(traceback_info, defer_now=True)

            serve_500(self)

    return wrapped_method
Unfortunately, having to manually
is not so Pythonic. At this point I was stuck and wanted to give up, but asked for some advice on G+ and actually got what I needed from the all knowing Ali Afshar. What did I need? Metaclasses.

Before showing the super simple metaclass I wrote, you need to know one thing from StackOverflow user Kevin Samuel:
The main purpose of a metaclass is to change the class automatically, when it's created.
With the __new__ method, the type object in Python actually constructs a class (which is also an object) by taking into account the name of the class, the parents (or bases) and the class attritubutes. So, we can make a metaclass by subclassing type and overriding __new__:
class DecorateHttpVerbsMetaclass(type):

    def __new__(cls, name, bases, cls_attr):
        verbs = ['get', 'post', 'put', 'delete']
        for verb in verbs:
            if verb in cls_attr and isinstance(cls_attr[verb], function):
                cls_attr[verb] = deadline_decorator(cls_attr[verb])

        return super(DecorateHttpVerbsMetaclass, cls).__new__(cls, name,
                                                              bases, cls_attr)
In DecorateHttpVerbsMetaclass, we look for four (of the nine) HTTP verbs, because heck, only seven are supported in RequestHandler, and we're not that crazy. If the class has one of the verbs as an attribute and if the attribute is a function, we decorate it with deadline_decorator.

Now, we can rewrite our subclass of RequestHandler with one extra line:
class ExtendedHandler(RequestHandler):
    __metaclass__ = DecorateHttpVerbsMetaclass

    def handle_exception(self, exception, debug_mode):
        traceback_info = ''.join(format_exception(*sys.exc_info()))
        email_admins(traceback_info, defer_now=True)

        serve_500(self)
By doing this, when the class ExtendedHandler is built (as an object), all of its attributes and all of its parent classes (or bases) attributes are checked and possibly updated by our metaclass.

And now you and James Nekbehrd can feel like a boss when your app handles errors.

Imports:
from google.appengine.api import mail
from google.appengine.ext.deferred import defer
from google.appengine.ext.webapp import RequestHandler
from google.appengine.runtime import DeadlineExceededError
import sys
from traceback import format_exception
from SOME_APP_SPECIFIC_LIBRARY import serve_500
from LAST_POST import email_admins
*Pythonic:
An idea or piece of code which closely follows the most common idioms of the Python language, rather than implementing code using concepts common to other languages.
Notes:
  • Using grep -r "Exception)" . | grep "class " I have convinced myself (for now) that the only errors AppEngine will throw that do not inherit from Exception are DeadlineExceededError, SystemExit, and KeyboardInterrupt so that is why I only catch the timeout.
  • You can also use webapp2 to catch 500 errors, even when handle_exception fails to catch them.

Disclaimer: Just because you know what a metaclass is doesn't mean you should use one:
  • "Don't do stuff like this though, what is your use case?" -Ali Afshar
  • "Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why)." -Python Guru Tim Peters
  • "The main use case for a metaclass is creating an API." -Kevin Samuel

Sunday, November 27, 2011

Handling errors in Google App Engine...and failing

After spending a nontrivial amount of my nights and weekends working on an AppEngine app, I wanted a good way to monitor the logs without checking in on them every day. After a particularly frustrating weekend of updates that exposed unnoticed bugs that had yet to be triggered by the app, I set out to find such a way. I set out to find a Pythonic* way.

Since I knew the App Engine Mail API was super easy to configure,  I figured I would just email myself every time there was an exception, before serving my default 500 error page. To do so, I just needed to subclass the default RequestHandler with my own handle_exception method. (OK, prepare yourselves, a bunch of code is about to happen. See the necessary imports at the bottom of the post.)
class ExtendedHandler(RequestHandler):

    def handle_exception(self, exception, debug_mode):
        traceback_info = ''.join(format_exception(*sys.exc_info()))
        email_admins(traceback_info, defer_now=True)

        serve_500(self)
Awesome! By making all my handlers inherit from ExtendedHandler, I can use the native Python modules traceback and sys to get the traceback and my handy dandy
def email_admins(error_msg, defer_now=False):
    if defer_now:
        defer(email_admins, error_msg, defer_now=False)
        return

    sender = 'YOUR APP Errors <errors@your_app_id_here.appspotmail.com>'
    to = 'Robert Admin <bob@example.com>, James Nekbehrd <jim@example.com>'
    subject = 'YOUR APP Error: Admin Notify'
    body = '\n'.join(['Dearest Admin,',
                      '',
                      'An error has occurred in YOUR APP:',
                      error_msg,
                      ''])

    mail.send_mail(sender=sender, to=to,
                   subject=subject, body=body)
to send out the email in the deferred queue** so as not to hold up the handler serving the page. Mission accomplished, right? WRONG!

Unfortunately, handle_exception only handles the "right" kind of exceptions. That is, exceptions which inherit directly from Python's Exception. From the horse's mouth:
Exceptions should typically be derived from the Exception class, either directly or indirectly.
But. But! If the app fails because a request times out, a DeadlineExceededError is thrown and handle_exception falls on its face. Why? Because DeadlineExceededError inherits directly from Exception's parent class: BaseException.  (Gasp)

It's OK little ones, in my next post I explain how I did it while keeping my code Pythonic by using metaclasses.

Imports:
from google.appengine.api import mail
from google.appengine.ext.deferred import defer
from google.appengine.ext.webapp import RequestHandler
import sys
from traceback import format_exception
from SOME_APP_SPECIFIC_LIBRARY import serve_500
*Pythonic:
An idea or piece of code which closely follows the most common idioms of the Python language, rather than implementing code using concepts common to other languages.
**Deferred Queue: Make sure to enable the deferred library in your app.yaml by using deferred: on in your builtins.