Django: Generators are your friend

Attribution: http://www.flickr.com/photos/dorkomatic/ / CC BY 2.0

Pro tip: if you need to shove a lot of data through a Django view, DO NOT attempt to create a big string-- use a generator.


Here's something that seems sensible on the surface:

body = ""
for region in regions:
  for zip in region.zipcode.iterator():
    body = body + "\t".join(
           [region.region, zip.zipcode]
           ) + "\n"
resp = HttpResponse(body, mimetype='application/ms-excel')
resp['Content-Disposition'] = 'attachment; filename=%s.xls' % (unicode("Regions"),)

Except when the set of regions and zipcodes gets large!  Let's consider the following replacement that uses a generator:

def dump_it():
    for region in regions:
       for zipcode in region.zipcode.iterator():
           yield "\t".join(
              [region.region, zipcode.zipcode]
           ) + "\n"

resp = HttpResponse(dump_it(), mimetype='application/ms-excel')

resp['Content-Disposition'] = 'attachment; filename=%s.xls' % (unicode("Regions"),)

There is a slight performance difference; the latter takes a few seconds (2 seconds for almost 70K resulting rows on my dog of a laptop).  However, I attempted to benchmark the former on the same dataset, and it took almost 13 MINUTES (775 seconds).  So, slight, meaning within 3 orders of magnitude.
Read more...

Django Forms: Alternate Date Handling

For usability on the "score" page at Sage Steps (free registration required if you want to check it out), we decided that the best method to present a date was a simple drop-down with, e.g. "February 2010" as the text.

I tried out several permutations with mixed luck, but then happened upon the following recipe:

def month_year():
    today = date.today()
    today = date(today.year, today.month, 1)
    dates = []
    for i in range(1,13):
        if (today.month > i):
            month = today.month - i
            year = today.year
        else:
            month = 12-(i-today.month)
            year = today.year - 1
        mon = date(year, month, 1)
        dates.append((mon, mon.strftime("%B %Y")))
    dates.reverse()
    return dates

This creates a set of tuples, e.g.: (date(2009,3,1), 'March 2009'),

month = DateField(widget=Select(choices=month_year()))

This will produce the drop-down as above, and

form.cleaned_data['month']

Will actually return a datetime.date object (trust me, that's a good thing). Read more...

Numeric Formatting in Django

You might have seen my previous post lamenting the weirdnesses around number formatting in python.  This spills over into Django; if you have DecimalFields in your models (especially if you are using the ModelForm object to create your forms.


The previous "format Decimal object as a string that any 10-year-old would expect" method is:

def dec_string(dec):
    if isinstance(dec, Decimal):
        if dec:
            if dec.normalize().as_tuple().exponent > 0:
                return "%d"%dec
            else:
                return "%s"%dec.normalize()
        else:
            return 0
    else:
        return dec

(I added the isinstance() bit to protect me from myself =)

Since we know how to format this, all that remains to be done is to create a widget that automatically formats DecimalFields correctly.

class DecimalInput(TextInput):
  def render(self, name, value, attrs=None):
    value = dec_string(value)
    return super(DecimalInput, self).render(name, value, attrs)

And then just use that in your form:

class EnergyForm(ModelForm):
    kwh = DecimalField(widget=DecimalInput(attrs={'size':'6'}))

There is more to the ModelForm, but that's what the docs are for, right? Read more...

Numeric Formatting in Python

Sometimes I hate Python. Here's the difference between Perl and Python: Perl accepts, even embraces the fact that language is messy. Python sometimes gets this wrong.

Here are some samples of dealing with formatting numbers:

from decimal import Decimal

>>> f= Decimal('1200.1')
>>> "%s"%f.normalize()
'1200.1'

Huzzah, it does the right thing!

>>> f= Decimal('1200')
>>> "%s"%f.normalize()
'1.2E+3'

 Doh!  But clearly, removing the normalize() will fix things, right?

>>> "%s"%f
'1200'

Yee-haw! Now we're cooking!

>>> f= Decimal('1200.100')
>>> "%s"%f
'1200.100'

Hm, users don't want the trailing zeroes.  The only thing that I can think to do is use a regular expression, and now I have two problems.

 But wait! If the number doesn't have anything after the decimal, then we do one, and if it does, we do the other!

def dec_string(dec):
    if dec:
        if dec.normalize().as_tuple().exponent > 0:
            return "%d"%dec
        else:
            return "%s"%dec.normalize()
    else:
        return 0

 And that seems to be working.  Note that the final version uses "%d"%dec, rather than %s, since we want to make sure and truncate past the decimal.

Update: check out my post about integrating it with Django forms to make it seamless. Read more...

Why do I have physical infrastructure?

I've been off on one of my ADD fueled research binges lately-- investigating Cloud Computing. I looked at several offerings:

1. Nimbus
2. Amazon EC2
3. Google AppEngine

And here are the conclusions that I came to:

Nimbus:
A nice layer on top of Xen to manage VM instances on a set of machines. This is really targeted at scientific computing, and does have a bit of a learning curve. If you have a pile of hardware that is being underutilized, some technical know-how and elbow grease (a.k.a. 'Round Tuits'), this is a nice looking solution to run a mess of VMs on. Especially if there are many images, but they don't necessarily run all the time (e.g. testing machines, support/demo environments). Turning VMs on and off is fairly straight forward. It offers an emulation of the Amazon EC2 WSDL interfaces, which is nice.

Amazon EC2:
This is an online cloud computing platform. Upload your VM(s), and use the Web Services interface to activate machines. Pay per hour, or per year. I'd have a hard time justifying capital expenditure (on server hardware) in a start up with this service available. The user can activate as many VMs as they need to service demand. The amount of CPU and disk available is quite elastic (hence the name, Elastic Compute Cloud).

Google AppEngine:
This is an interesting service-- it basically offers a stripped Python (and now Java) environment. Applications need to be specifically ported to the environment, but it does utilize (require) a number of best practices so that your application should be somewhat "forced" to be scalable. Probably a good option for "green field" web 2.0 applications, but if you have code already, might not be the best choice. Integration seems like it might be a little problematic (e.g. no MySQL/Postgres/SQL Server or C Library support).

Summary
Of these, Amazon EC2 is the one that fascinates me. We regularly build VMs that mimic customer environments, and these might only get used for a couple dozens of hours per year. EC2 would be a great solution.

We also have regular requirements to do performance testing under specific circumstances. We can allocate 12 hours on a fairly beefy machine with very low cost, get our testing done, and have results for under $20-ish. That is amazing. Read more...

Singletons in Python

Since the project that I'm working on has a whole lot of nasty, boil the oceans, SQL queries, I thought it'd be best to cache information. Caching is great, but what I really need is an application-wide cache (not tied to a specific instance object).

Normally I use Singletons (from _Design Patterns_) to provide this functionality. Here's how to implement them in Python:

from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66531 (in the comments):

class Singleton(object):
   def __new__(cls, *p, **k):
      if not '_the_instance' in cls.__dict__:
          cls._the_instance = object.__new__(cls)
      return cls._the_instance

all you have to do then is inherit from Singleton, and it just works. This is actually cleaner than some other languages I've implemented Singletons in.

The real bonus, however, is the actual article that the posting above was about:

class Borg(object):
  _state = {}
  def __new__(cls, *p, **k):
      self = object.__new__(cls, *p, **k)
      self.__dict__ = cls._state
      return self

That will share the state of the object across instances. No thinking about it.
Read more...

New Classes vs. Old Classes in Python

I found a bit of daftness in Python today:

class A:
def __init__(self):
print "Rutabaga!"

class B(A):
def __init__(self):
super(B, self)

causes:

TypeError: super() argument 1 must be type, not classobj

Changing the first line to:

class A(object):

fixes it.

Now, forgive me if I think that's completely stunted. You could have at least said:

TypeError: to use super(), the superclass must have one of the built-in base classes in its inheritance path.

Read more...

Learning Python

I've avoided it for at least 10 years, but I finally bit the bullet.

I am developing an application in Python.

Read on for my take on what I like about Python, and what I didn't like so much.

Great stuff:

• Functions are first class


• Great support for object oriented programming


• Basic feature parity with Perl


• Great library support (I'll cover some of these in a later post)


• Closures (all the cool kids have them)

Not so great stuff:

• Scoping is wierd, but maybe I just need to get used to it


• Feels wrong to not close blocks explicitly


• Wierd __foo__ special functions, etc. (although I don't have much room to talk since I like Perl)


• No autovivification

Random thoughts:

• Anyone who says it has "cleaner" object orientation than Perl probably just doesn't "get" Perl.


• Somehow, it reminds me strongly of Javascript. They are just somehow similar.

Read more...