Thursday, May 17, 2012

Life of π: Continued Fractions and Infinite Series

This is from a talk I gave to the UC Santa Cruz Math Club back in February. I have had the slides cluttering my desk since I gave the talk as a reminder of putting them up on the web, and today I finally cleaned them off!

An interested tidbit about these slides: I gave the same talk one other time as an undergraduate. As an undergraduate I gave the talk the day after a root canal. This February (at UCSC), one day after the talk I also had a root canal. What's the moral of the story? Don't give this talk.

No seriously, don't give this talk. Though the fact is cool and the math is not too bad to grasp, the stuff from slides 106 to 113 goes way over an audience's collective head just because they have stopped paying attention at that point. Too many variables!

Tuesday, May 15, 2012

Reverse Calculating An Interest Rate

I was recently playing around with some loan data and only happened to have the term (or length, or duration) of the loan, the amount of the recurring payment (in this case monthly) and the remaining principal owed on the loan. I figured there was an easy way to get at the interest rate, but wasn't sure how. After some badgering from my coworker +Paul, I searched the web and found a tool from CALCAmo (a site just for calculating amortizations).

Problem solved, right? Wrong. I wanted to know why; I had to go deeper. So I did a bit of math and a bit of programming and I was where I needed to be. I'll break the following down into parts before going on full steam.
  • Break down the amortization schedule in terms of the variables we have and the one we want
  • Determine a function we want to find zeroes of
  • Write some code to implement the Newton-Raphson method
  • Utilize the Newton-Raphson code to find an interest rate
  • Bonus: Analyze the function to make sure we are right
Step I: Break Down the Amortization Schedule

We can do this using the series \(\left\{P_i\right\}_i\) of principal owed, which varies over time and will go to zero once paid off. In this series, \(P_0\) is the principal owed currently and \(P_i\) is the principal owed after \(i\) payments have been made. (Assuming monthly payments, this will be after \(i\) months.) If the term is \(T\) periods, then we have \(P_T = 0\).

We have already introduced the term (\(T\)); we also need the value of the recurring (again, usually monthly) payment \(R\), the interest rate \(r\) and the initial principal owed \(P_0 = P\).

Time-Relationship between Principal Values

If after \(i\) periods, \(P_i\) is owed, then after one period has elapsed, we will owe \(P_i \cdot m\) where \(m = m(r)\) is some multiplier based on the length of the term. For example if each period is one month, then we divide our rate by \(12\) for the interest and add \(1\) to note that we are adding to existing principal: \[m(r) = 1 + \frac{r}{12}.\] In addition to the interest, we will have paid off \(R\) hence \[P_{i + 1} = P_i \cdot m - R.\] Formula for \(P_i\)

Using this, we can actually determine \(P_i\) strictly in terms of \(m, R\) and \(P\). First, note that \[P_2 = P_1 \cdot m - R = (P_0 \cdot m - R) \cdot m - R = P \cdot m^2 - R(m + 1)\] since \(P_0 = P\). We can show inductively that \[P_i = P \cdot m^i - R \cdot \sum_{j = 0}^{i - 1} m^j.\] We already have the base case \(i = 1\), by definition. Assuming it holds for \(i\), we see that \[P_{i + 1} = P_i \cdot m - R =  m \cdot \left(P \cdot m^i - R \cdot \sum_{j = 0}^{i - 1} m^j\right) - R = P \cdot m^{i + 1} - R \cdot \sum_{j = 1}^{i} m^j - R,\] and our induction is complete. (We bump the index \(j\) since we are multiplying each \(m^j\) by \(m\).)
Each term in the series is related to the previous one (except \(P_0\), since time can't be negative in this case).

Step II: Determine a Function we want to find Zeroes of

Since we know \(P_T = 0\) and \(P_T = P \cdot m^T - R \cdot \sum_{j = 0}^{T - 1} m^j\), we actually have a polynomial in place that will let us solve for \(m\) and in so doing, solve for \(r\).

To make our lives a tad easier, we'll do some rearranging. First, note that \[\sum_{j = 0}^{T - 1} m^j = m^{T - 1} + \cdots + m + 1 = \frac{m^T - 1}{m - 1}.\] We calculate this sum of a geometric series here, but I'll just refer you to the Wikipedia page instead. With this reduction we want to solve \[0 = P \cdot m^T - R \cdot \frac{m^T - 1}{m - 1} \Longleftrightarrow P \cdot m^T \cdot (m - 1) = R \cdot (m^T - 1).\] With that, we have accomplished Step II, we have found a function (parameterized by \(P, T\) and \(R\) which we can use zeroes from to find our interest rate: \[f_{P, T, R}(m) = P \cdot m^T \cdot (m - 1) - R \cdot (m^T - 1) = P \cdot m^{T + 1} - (P + R) \cdot m^T + R.\] Step III: Write some code to implement the Newton-Raphson method

We use the Newton-Raphson method to get super-duper-close to a zero of the function. For in-depth coverage, see the Wikipedia page on the Newton-Raphson method, but I'll give some cursory coverage below. The methods used to show that a fixed point is found are not necessary for the intuition behind the method.

Intuition behind the method

For the intuition, assume we know (and can compute) a function \(f\), its derivative \(f'\) and a value \(x\). Assume there is some zero \(y\) nearby \(x\). Since they are close, we can approximate the slope of the line between the points \((x, f(x)\) and \((y, f(y)\) with the derivative nearby. Since we know \(x\), we use \(f'(x)\) and intuit that \[f'(x) = \text{slope} = \frac{f(y) - f(x)}{y - x} \Rightarrow y - x = \frac{f(y) - f(x)}{f'(x)}.\] But, since we know that \(y\) is a zero, \(f(y) - f(x) = -f(x)\) hence \[y - x = \frac{-f(x)}{f'(x)} \Rightarrow y = x - \frac{f(x)}{f'(x)}.\] Using this method, one can start with a given value \(x_0 = x\) and compute better and better approximations of a zero via the iteration above that determines \(y\). We use a sequence to do so: \[x_{i + 1} = x_i - \frac{f(x_i)}{f'(x_i)}\] and stop calculating the \(x_i\) either after \(f(x_i)\) is below a preset threshold or after the fineness of the approximation \(\left|x_i - x_{i + 1}\right|\) goes below a (likely different) preset threshold. Again, there is much that can be said about these approximations, but we are trying to accomplish things today, not theorize.

Programming Newton-Raphson

To perform Newton-Raphson, we'll implement a Python function that takes the initial guess (\(x_0\)) and the functions \(f\) and \(f'\). We'll also (arbitrarily) stop after the value \(f(x_i)\) drops below \(10^{-8}\) in absolute value.
def newton_raphson_method(guess, f, f_prime):
    def next_value(value):
        return value - f(value)*1.0/f_prime(value)

    current = guess
    while abs(f(current)) > 10**(-8):
        current = next_value(current)

    return current
As you can see, once we have f  and f_prime, everything else is easy because all the work in calculating the next value (via next_value) is done by the functions.

Step IV: Utilize the Newton-Raphson code to find an Interest Rate

We first need to implement \(f_{P, T, R}(m) = P \cdot m^{T + 1} - (P + R) \cdot m^T + R\) and \(f'_{P, T, R}\) in Python. Before doing so, we do a simple derivative calculation: \[f_{P, T, R}'(m) = P \cdot (T + 1) \cdot m^T - (P + R) \cdot T \cdot m^{T - 1}.\] With these formulae in hand, we write a function which will spit out the corresponding f and f_prime given the parameters \(P\) (principal), \(T\) (term) and \(R\) (payment):
def generate_polynomials(principal, term, payment):
    def f(m):
        return (principal*(m**(term + 1)) - (principal + payment)*(m**term) +
                payment)

    def f_prime(m):
        return (principal*(term + 1)*(m**term) -
                (principal + payment)*term*(m**(term - 1)))

    return (f, f_prime)
Note that these functions only take a single argument (m), but we are able to use the other parameters from the parent scope beyond the life of the call to generate_polynomials due to closure in Python.

In order to solve, we need an initial guess, but we need to know the relationship between \(m\) and \(r\) before we can determine what sort of guess makes sense. In addition, once a value for \(m\) is returned from Newton-Raphson, we need to be able to turn it into an \(r\) value so functions m and m_inverse should be implemented. For our dummy case here, we'll assume monthly payments (and compounding):
def m(r):
    return 1 + r/12.0

def m_inverse(m_value):
    return 12.0*(m_value - 1)
Using these, and assuming that an interest rate of 10% is a good guess, we can put all the pieces together:
def solve_for_interest_rate(principal, term, payment, m, m_inverse):
    f, f_prime = generate_polynomials(principal, term, payment)

    guess_m = m(0.10)  # ten percent as a decimal
    m_value = newton_raphson_method(guess_m, f, f_prime)
    return m_inverse(m_value)
To check that this makes sense, let's plug in some values. Using the bankrate.com loan calculator, if we have a 30-year loan (with \(12 \cdot 30 = 360\) months of payments) of $100,000 with an interest rate of 7%, the monthly payment would be $665.30. Plugging this into our pipeline:
>>> principal = 100000
>>> term = 360
>>> payment = 665.30
>>> solve_for_interest_rate(principal, term, payment, m, m_inverse)
0.0699996284703
And we see the rate of 7% is approximated quite well!

Bonus: Analyze the function to make sure we are right

Coming soon. We will analyze the derivate and concavity to make sure that our guess yield the correct (and unique) zero.

Saturday, April 7, 2012

Silly Pranks on your Friends

Disclaimer: These are silly little pranks, but I don't encourage messing with someone's computing environment without letting them know you have done so.

First Prank:

I have a friend who really likes to read people.com, so I figured I would "enrich" her life a bit with another source of daily news :)

I decided to play around with her hosts file, so that when she visited people.com, she really got the New York Times (the realest news I could think of at that time, though there are plenty of fine candidates).

To quote the Wikipedia article on hosts files:
"The hosts file...assists in addressing network nodes in a computer network. It is a common part of an operating system's Internet Protocol (IP) implementation, and serves the function of translating human-friendly hostnames into numeric protocol addresses, called IP addresses, that identify and locate a host in an IP network."
More importantly: "the /etc/hosts file...allows you to add entries that traditionally your computer will look up first before trying your server DNS." (source) This means that even though the DNS Lookup provided by her ISP could resolve people.com, her browser would get an IP address from the hosts file first and hence will render the New York Times page for people.com.

The first step to do this was to find the IP address for the replacement site:
$ ping www.nytimes.com
PING www.nytimes.com (199.239.136.200): 56 data bytes
64 bytes from 199.239.136.200: icmp_seq=0 ttl=64 time=0.062 ms
64 bytes from 199.239.136.200: icmp_seq=1 ttl=64 time=0.054 ms
...
For the second (and final) step, I just needed to add an entry to the hosts file. After locating the file on her Macbook in /etc/hosts, I updated the contents:
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
fe80::1%lo0     localhost
199.239.136.200 people.com  # New entry
Voilà! With that, the prank was complete and the next time she visited people.com, the got the contents of nytimes.com in her browser.

Second Prank coming soon.

Wednesday, March 28, 2012

Where have I been?

Well, it's been a bit crazy and I haven't written a blog post in ages. I have several brewing, but had just been too busy at work (and a ton of travel for personal fun) to really have the excess time to write.

This return post will not have much content but will announce that I'm a big boy now.

In the 1.6.3 release of the App Engine SDK (and hence runtime), three nifty changes of mine were included. Two of them even made the Release Notes:
In addition, the one that was most confusing to fix didn't make it into any set of Release Notes, but I "closed" the issue that I originally opened on StackOverflow. Checkout the diff to see the details. I'll follow up with a summary of each of the fixes in a later post.

Wednesday, November 30, 2011

A Python Metaclass for "extra bad" errors in Google App Engine

So now here we are, having tried to handle errors in Google App Engine...and failed all because silly DeadlineExceededError jumps over Exception in the inheritance chain and goes right for BaseException. How can we catch these in our handlers while staying Pythonic*?

First and foremost, in the case of a timeout, we need to explicitly catch a DeadlineExceededError. To do so, we can use a decorator (hey, that's Pythonic) in each and every handler for each and every HTTP verb. (Again, prepare yourselves, a bunch of code is about to happen. See the necessary imports at the bottom of the post.)
def deadline_decorator(method):

    def wrapped_method(self, *args, **kwargs):
        try:
            method(self, *args, **kwargs)
        except DeadlineExceededError:
            traceback_info = ''.join(format_exception(*sys.exc_info()))
            email_admins(traceback_info, defer_now=True)

            serve_500(self)

    return wrapped_method
Unfortunately, having to manually
is not so Pythonic. At this point I was stuck and wanted to give up, but asked for some advice on G+ and actually got what I needed from the all knowing Ali Afshar. What did I need? Metaclasses.

Before showing the super simple metaclass I wrote, you need to know one thing from StackOverflow user Kevin Samuel:
The main purpose of a metaclass is to change the class automatically, when it's created.
With the __new__ method, the type object in Python actually constructs a class (which is also an object) by taking into account the name of the class, the parents (or bases) and the class attritubutes. So, we can make a metaclass by subclassing type and overriding __new__:
class DecorateHttpVerbsMetaclass(type):

    def __new__(cls, name, bases, cls_attr):
        verbs = ['get', 'post', 'put', 'delete']
        for verb in verbs:
            if verb in cls_attr and isinstance(cls_attr[verb], function):
                cls_attr[verb] = deadline_decorator(cls_attr[verb])

        return super(DecorateHttpVerbsMetaclass, cls).__new__(cls, name,
                                                              bases, cls_attr)
In DecorateHttpVerbsMetaclass, we look for four (of the nine) HTTP verbs, because heck, only seven are supported in RequestHandler, and we're not that crazy. If the class has one of the verbs as an attribute and if the attribute is a function, we decorate it with deadline_decorator.

Now, we can rewrite our subclass of RequestHandler with one extra line:
class ExtendedHandler(RequestHandler):
    __metaclass__ = DecorateHttpVerbsMetaclass

    def handle_exception(self, exception, debug_mode):
        traceback_info = ''.join(format_exception(*sys.exc_info()))
        email_admins(traceback_info, defer_now=True)

        serve_500(self)
By doing this, when the class ExtendedHandler is built (as an object), all of its attributes and all of its parent classes (or bases) attributes are checked and possibly updated by our metaclass.

And now you and James Nekbehrd can feel like a boss when your app handles errors.

Imports:
from google.appengine.api import mail
from google.appengine.ext.deferred import defer
from google.appengine.ext.webapp import RequestHandler
from google.appengine.runtime import DeadlineExceededError
import sys
from traceback import format_exception
from SOME_APP_SPECIFIC_LIBRARY import serve_500
from LAST_POST import email_admins
*Pythonic:
An idea or piece of code which closely follows the most common idioms of the Python language, rather than implementing code using concepts common to other languages.
Notes:
  • Using grep -r "Exception)" . | grep "class " I have convinced myself (for now) that the only errors AppEngine will throw that do not inherit from Exception are DeadlineExceededError, SystemExit, and KeyboardInterrupt so that is why I only catch the timeout.
  • You can also use webapp2 to catch 500 errors, even when handle_exception fails to catch them.

Disclaimer: Just because you know what a metaclass is doesn't mean you should use one:
  • "Don't do stuff like this though, what is your use case?" -Ali Afshar
  • "Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why)." -Python Guru Tim Peters
  • "The main use case for a metaclass is creating an API." -Kevin Samuel

Sunday, November 27, 2011

Handling errors in Google App Engine...and failing

After spending a nontrivial amount of my nights and weekends working on an AppEngine app, I wanted a good way to monitor the logs without checking in on them every day. After a particularly frustrating weekend of updates that exposed unnoticed bugs that had yet to be triggered by the app, I set out to find such a way. I set out to find a Pythonic* way.

Since I knew the App Engine Mail API was super easy to configure,  I figured I would just email myself every time there was an exception, before serving my default 500 error page. To do so, I just needed to subclass the default RequestHandler with my own handle_exception method. (OK, prepare yourselves, a bunch of code is about to happen. See the necessary imports at the bottom of the post.)
class ExtendedHandler(RequestHandler):

    def handle_exception(self, exception, debug_mode):
        traceback_info = ''.join(format_exception(*sys.exc_info()))
        email_admins(traceback_info, defer_now=True)

        serve_500(self)
Awesome! By making all my handlers inherit from ExtendedHandler, I can use the native Python modules traceback and sys to get the traceback and my handy dandy
def email_admins(error_msg, defer_now=False):
    if defer_now:
        defer(email_admins, error_msg, defer_now=False)
        return

    sender = 'YOUR APP Errors <errors@your_app_id_here.appspotmail.com>'
    to = 'Robert Admin <bob@example.com>, James Nekbehrd <jim@example.com>'
    subject = 'YOUR APP Error: Admin Notify'
    body = '\n'.join(['Dearest Admin,',
                      '',
                      'An error has occurred in YOUR APP:',
                      error_msg,
                      ''])

    mail.send_mail(sender=sender, to=to,
                   subject=subject, body=body)
to send out the email in the deferred queue** so as not to hold up the handler serving the page. Mission accomplished, right? WRONG!

Unfortunately, handle_exception only handles the "right" kind of exceptions. That is, exceptions which inherit directly from Python's Exception. From the horse's mouth:
Exceptions should typically be derived from the Exception class, either directly or indirectly.
But. But! If the app fails because a request times out, a DeadlineExceededError is thrown and handle_exception falls on its face. Why? Because DeadlineExceededError inherits directly from Exception's parent class: BaseException.  (Gasp)

It's OK little ones, in my next post I explain how I did it while keeping my code Pythonic by using metaclasses.

Imports:
from google.appengine.api import mail
from google.appengine.ext.deferred import defer
from google.appengine.ext.webapp import RequestHandler
import sys
from traceback import format_exception
from SOME_APP_SPECIFIC_LIBRARY import serve_500
*Pythonic:
An idea or piece of code which closely follows the most common idioms of the Python language, rather than implementing code using concepts common to other languages.
**Deferred Queue: Make sure to enable the deferred library in your app.yaml by using deferred: on in your builtins.

Thursday, November 17, 2011

Quick and Dirty: Santa's Coming

I have been wanting to write a post for awhile, but was travelling for a work event and while on the road I decided to be lazy.

Since I just so happen to use a few GData APIs occasionally in my day to day work, most of the post ideas revolve around quirky things I have done or want to do with the APIs. Also, due to my obscene love for Python, all my mashups seem to end up using the Python Client for GData.

Back-story: As I was finalizing travel and gifts for my winter holiday back home, I called an old friend to let him know I'd be home in 40 days. After relaying this information to a few other people, I noted to my girlfriend that it would be nice if a computer would remind me of the count every day. This is where this quick and dirty pair of scripts come in to remind me when Santa is coming.

Pre-work — Account Settings: To allow an app to make requests on my behalf, I signed up to Manage my Domain for use with Google Apps, etc. For illustration purposes, let's say I used http://example.com (in reality, I used a pre-existing App of mine, I really just needed an OAuth token for one time use, no real safety concerns there). After adding this domain in the management page, I am able to get my "OAuth Consumer Key" and "OAuth Consumer Secret" which we'll say are EXAMPLE_KEY and EXAMPLE_SECRET in this example. Also in the management page, I set my "OAuth 2.0 Redirect URIs" and made sure my app can serve that page (even if it is a 404). Again for illustration, let's pretend I used http://example.com/verify.

After doing this settings pre-work, I have two scripts to do the work for me.

First script — get the OAuth Token:
import gdata.calendar.client
import gdata.gauth

gcal = gdata.calendar.client.CalendarClient()
oauth_callback = 'http://example.com/verify'
scopes = ['https://www.google.com/calendar/feeds/']
consumer_key = 'EXAMPLE_KEY'
consumer_secret = 'EXAMPLE_SECRET'
request_token = gcal.get_oauth_token(scopes, oauth_callback,
                                     consumer_key, consumer_secret)
out_str = ('Please visit https://www.google.com/accounts/OAuthAuthorize'
           'Token?hd=default&oauth_token=%s' % request_token.token)
print out_str
follow = raw_input('Please entry the follow link after authorizing:\n')
gdata.gauth.authorize_request_token(request_token, follow)
gcal.auth_token = gcal.get_access_token(request_token)
print 'TOKEN:', gcal.auth_token.token
print 'TOKEN_SECRET:', gcal.auth_token.token_secret
This script "spoofs" the OAuth handshake by asking the user (me) to go directly to the OAuth Authorize page. After doing so and authorizing the App, I am redirected to http://example.com/verify with query parameters for oauth_verifier and oauth_token. These are then used by the gauth section of the GData library to finish the OAuth handshake. Once the handshake is complete, the script prints out a necessary token and token secret to be used by the second script. I would advise piping the output to a file, augmenting the script to write them to a file, or writing these down (this is a joke, please don't write down 40 plus character goop that was produced by your computer). For the next script, let's pretend our token is FAKE_TOKEN and our token secret is FAKE_TOKEN_SECRET.

Second script — insert the events:
# General libraries
from datetime import date
from datetime import timedelta

# Third-party libraries
import atom
import gdata.gauth
import gdata.calendar.client
import gdata.calendar.data

gcal = gdata.calendar.client.CalendarClient()
auth_token = gdata.gauth.OAuthHmacToken(consumer_key='EXAMPLE_KEY',
                                        consumer_secret='EXAMPLE_SECRET',
                                        token='FAKE_TOKEN',
                                        token_secret='FAKE_TOKEN_SECRET',
                                        auth_state=3)
gcal.auth_token = auth_token

today = date.today()
days_left = (date(year=2011, month=12, day=23) - today).days

while days_left >= 0:
    event = gdata.calendar.data.CalendarEventEntry()
    if days_left > 1:
        msg = '%s Days Until Home for Christmas' % days_left
    elif days_left == 1:
        msg = '1 Day Until Home for Christmas'
    elif days_left == 0:
        msg = 'Going Home for Christmas'
    event.title = atom.data.Title(msg)

    # When
    start_time = '2011-%02d-%02dT08:00:00.000-08:00' % (today.month, today.day)
    end_time = '2011-%02d-%02dT09:00:00.000-08:00' % (today.month, today.day)
    event.when.append(gdata.calendar.data.When(
        start=start_time,
        end=end_time,
        reminder=[gdata.data.Reminder(hours='1')]))

    gcal.InsertEvent(event)

    today += timedelta(days=1)
    days_left -= 1
This script first authenticates by using the key/secret pair for the application (retrieved from the settings page) and the key/secret pair for the user token (that we obtained from the first script). To authenticate, we explicitly construct an HMAC-SHA1 signed token in the final auth state (3) of two-legged OAuth and then set the token on our calendar client (gcal).

After authenticating, we start with today and figure out the number of days in the countdown given my return date of December 23, 2011. With these in hand, we can loop through until there are no days left, creating a CalendarEventEntry with title as the number of days left in the countdown and occurring from 8am to 9am PST (UTC -8). Notice also I include a gdata.data.Reminder so I get an email at 7am every morning (60 minutes before the event) updating my brain on the length of the countdown!

Cleanup: Be sure to go to your issued tokens page and revoke access to the App (e.g. http://example.com) after doing this to avoid any unwanted security issues.

References: I have never read this, but I'm sure the documentation on Registration for Web-Based Applications is very helpful.

Notes:
  • You can annoy other people by inviting them to these events for them as well. To do so, simply add the following two lines before inserting the event
    who_add = gdata.calendar.data.EventWho(email='name@mail.com')
    event.who.append(who_add)
  • Sometimes inserting an item results in a RedirectError, so it may be safer to try the insert multiple times with a helper function such as the following:
    def try_insert(attempts, gcal, event):
        from time import sleep
        from gdata.client import RedirectError
    
        while attempts > 0:
          try:
              gcal.InsertEvent(event)
              break
          except RedirectError:
              attempts -= 1
              sleep(3)
              pass
    
        if attempts == 0:
            print 'Insert "%s" failed' % event.title.text
  • In what I swear was a complete coincidence, v3 of the Calendar API was announced today. I will try to use the new documentation to redo this quick and dirty example with v3.