Category: Gentoo

Harmonious goes aspect oriented

(Harmonious is a web framework I’m working on for my degree, see previous blog entries for more info)

So after mulling over various problems (I’m in no hurry so I’m not forcing myself to make decisions about design issues on the spot. Instead I’m letting them sit in my head for however long it takes for the best solution to make itself known) related to implementing Aspect-Oriented Programming techniques elegantly, I’ve finally come to a conclusion and laid out the basic code needed.

So, wtf? It goes like this…

URLs are mapped to methods (see previous blog entries) and these methods can be decorated. The new ‘aspect’ decorator lets you decorate your method with, umm.. aspects. I like to call them Operational Aspects, as they relate to the various aspects of operation required to process a request.

e.g:

@public
@aspect(MyAspect)
def mypage(self):

return “hi!”

The aspects to which you decorate your method with are grouped, this is so that the framework knows what to expect from the aspect and what the aspect is expecting from the framework. There are also 3 execution modes: before your method is executed, after and in parallel with your method.

The reasons for implementing this all stem from my ponderings about how best to implement CRUD (Create Read Update Delete) support. My main qualm was that your method may only want the framework to handle some of these operations for you, the others you may want to handle yourself. So instead of an all or nothing situation you can now decorate your method with aspects (operations) of the CRUD model 🙂

Creating custom aspects is pretty easy too, here is a simple aspect:

class MyAspect:

class aspectmeta:

execution = AspectExecution.AFTER
keep_instance = True # Cache an instance of this aspect as we’re going to use it for more than one execution mode.
returns = AspectKeyword.RESPONSE
group = AspectGroup.OUTPUT

def __init__(self, harm, response):

self.harm = harm
self.response = response

def run(self, event):

notice(“MyAspect got event %s:” % event)

self.response.set_header(‘Status’, ‘groovy’)

return response

As you can probably tell, this aspect is executed after the requested method is executed and it returns a Response object. The framework determines what arguments __init__ accepts using introspection and supplies them from the current scope, most things will be imported though.

There are a number of things these aspects are going to make implementing really easy, input/output filters are pretty much a done job already 🙂

By the by, I’m looking for a new name. The reason being that harmonious.(org,net,com) are all taken 🙁

b2evo’s code tag sucks!

Ciao

Of URLs and Methods

For a long time now I’ve been hacking around on a web framework I started from scratch (mainly because they’re fun to code), but recently I started from scratch again using all I’d learnt from trying things out in the earlier prototype and what I’d learnt from other frameworks. So suddenly it’s all got a bit serious and I’m slowly coming together with something that’s looking very promising and for which I have big ambitions for.

Yeah, I know what you’re all thinking, but I think I understand the framework “scene” pretty well now and what it takes for a framework to stand out from the crowd. Time will tell 😉

Anyway, today I’m going to show you how the URL to method resolver works and what you can do with it.

First of all a little example:

class MyTestController:

@public
def index(self, a, b, c):

return “I like twigs”

def page1(self, a, b, c=”w00t”):

return “PHIL COLLINS IS GOD” # I actually got an anonymous letter in the post containing a badge with this on it! o_O

OK, pretty basic stuff and you can probably tell already what the URLs are going to look like for such mappings:

www.mysite.co.uk/MyTest/arg_a/arg_b/arg_c/ maps to the index method
www.mysite.co.uk/MyTest/page1/arg_a/arg_b/arg_c/ maps to the page1 method, so does
www.mysite.co.uk/MyTest/page1/arg_a/arg_b/

Joy.

Time for full complexity mode:

class MyTestController:

@public
def index(self, a, b, *varargs):

return “Hi mum!”

@public
def page1(self, a, b, c=”t00t”, *varargs, **kwargs):

return “burzum”

@public
@constraint(‘a’, r’^[a-zA-Z]*$’)
@constraint(‘c’, r'(some|thing|other)’)
def page2(self, a, b, c=”123″, **kwargs):

return ” ”

Look! you can accept variable arguments(an unknown number of) and keyword arguments(key/value pairs); groovy.

and some URL examples for those:
www.mysite.co.uk/MyTest/arg_a/arg_b/on/and/on/with/ariston maps to the index method
www.mysite.co.uk/MyTest/page1/arg_a/arg_b/arg_c/blah/a_key/a_value/another_key/another_value maps to the page1 method

(I’ll come to page2 in just a moment)

Well done if you noticed that for the page1 URL we’ll only ever have 1 or no variable in varargs as we need the unmapped URL parts (i.e not arg_a, b or c) to be a multiple of 2 so that we can satisfy the key/value pair requirement of the **kwargs. So, chances are you probably wont want to use *varargs and **kwargs in the same method.

page2…. I hope the decorator name (the @foo thing) and the regular expressions gave away what’s going on here. You can choose to constrain some of your arguments if you so whish, it’ll save you having to do some input validation at least.

So there you have it, you can use Python to its full extent as you would in any normal method and the framework does the leg work for you. I _am_ aware that using regular expressions to constrain URL mappings is standard stuff for frameworks these days, yet I think my using them in decorators is far nicer then having a separate messy mappings file.

Oh and by the way… using **kwargs effectively allows you to never create a URL containing ?foo=bar&ugly=yes again! I’m not saying something like www.mysite.co.uk/blog/06/01/2006/sessionid/df7d7sa5hf7g/size/100/ makes much sense to the user (maybe it’s less confusing for the average Joe?) but however you design your URLs is _totally_ up to you 🙂

p.s Please don’t kill me for any grammar/spelling mistakes — I’ve literally been looking at code all day long (the C apps at work were coded by the devil himself, I’m positive)

p.p.s b2evo’s tags don't seem to work too great and I'm too tired to play around with it.

Templating engines, who’d have ’em?

So what do you see as the purpose of a templating engine?

I’d say it’s a method of separating paradigms and abstracting the backend. Some engines enforce this by implementing their own minimal set of directives, others seem to follow the idea of restricting use, yet implement a vast amount of directives. Take Cheetah for example. Others just allow you to drop code in wherever and however you like.

It would be far less complicated to simply allow inline code, but then there is the argument that if developers are allowed to mix up the paradigms, they probably will.

So when it comes to writing your own engine, do you simply allow inline code, implement just a handful of directives or implement a huge set? If you implement just a handful, people may say “well look at engine XYZ, it has 3 times more features”, if you go for straight inline code, people will say it’s insecure and mixes up the paradigms. Yet I look at Cheetah’s list of directives and can’t help but think it’d so much simpler to just allow inline code. Note that Cheetah as more than enough directives to royally screw up the paradigm.

So at this point I’m thinking I’d rather get rid of the templating engine all together. One solution could be XSLT. XSLT has all the conditional and looping goodness you’d ever have needed from a templating engine. With a simple script to convert Python data structures into XML and XPath to access them, you’re all set. XSLT is a damn sight more complex than your average templating engine syntax though, and this would deter many developers who’d never used it before.

So what’s the solution to all of this? I don’t know, though there are 2 methods I prefer.

The first method is just simply allow inline python code, and then try to encourage developers not to mix up the paradigms and restrain themselves to using only the most basic of code in the templates.

Method #2 is to use a series of regular expressions to restrict usage.

However, I don’t believe in restricting what developers can do. There are after all plenty of other places the developer could screw up the paradigms.

I was hoping this post would help me make my mind up and pick a solution, though really I’m non the wiser.
It’d be great to hear peoples thoughts on this.

Harmonious and GLSR in harmony

It’s been a while since my last entry about harmonious and like-wise; a lot of work has been done.

A large majority of the server basics have been implemented (some only partially): sessions, domain module loading, serving up static files and the templating engine. A couple of extras have also been added: restricted directories (very similar to Apache’s htaccess) and support for virtual hosts.

Oddly enough, I’ve started to enjoy going to bed so I can lay there thinking up weird and wonderful ideas. That doesn’t mean I’m going to bed any earlier than usual though 😉 No doubt I’ll implement many in the future, persistent connections and SSL are both a must.

Next up is getting the thing all threaded up, my current ideas for how to do this goes a little something this…

As the server starts up, it starts a base number of worker threads that sit there waiting to processes requests that enter the queue (this is how CherryPy works). If a request comes in and there is another request waiting in the queue for a thread to process it, the server will create a new worker thread. At scheduled intervals the server will check each worker thread to see if it’s processing a request or not, if two or more are not doing anything, the server will kill one or two of them. The server can either have a max-threads configuration option set or allow the server to find the optimal amount of worker threads by monitoring the average time taken to process requests.

I have a feeling the server will be need to be under some pretty intense loads (and hence a large number of threads) before it decides on an optimal amount of threads. Heh, I’m not even too sure the server *will* be able to discern an optimal amount, I don’t know enough about the effects threading has on the system. Comments are welcome on this.

As for GLSR (GLEP 15), we have it running on Harmonious which was a rather painless process. Scott Hadfield (hadfield) even commented that he enjoyed coding with the Harmonious framework. I couldn’t agree more, it feels a lot cleaner and more organised than that pure CGI crap we were using before. Though I’m biased 😉

On a personal note, I’m off on holiday for 1 week in the Italian alps, tomorrow. My first in 3 years! I’ll try my hardest to uphold the typical drunken-Englishman stereotype 😉

Introducing: Harmonious – Python Web Framework

In parallel to the development of the GLSR (Gentoo Linux Script Repository a.k.a GLEP 15) myself, Scott (hadfield) and John (AllanonJL) have been working on developing a new Python based web framework. The decision to do this came about after it became apparent that we weren’t going to achieve any decent response times with CGI. We then spent a while working with FastCGI, but that turned out to suck also. With quite some code already laid down, we were reluctant to go down the mod_python roue and existing python frameworks either didn’t suit our style or looked rather sucky aswell.

So i’ll give you the blurb about Harmonious, if I may!

Harmonious consists of 5 distinct parts: A web server, a page dispatching agent, a templating engine, session handling routines and a database layer.

I’ll tell you about these in reverse order:

The database layer is still undergoing design and code changes so nothing is final yet. Basically, you’ll be able to easily use different database back ends much like phpBB can. Scott is the guy to talk to about this.

Session handling… pretty standard, not a lot of room for innovation in this area. Only thing worth mentioning is that it’ll be tightly integrated with the rest of the framework.

The templating engine has been around for some time now and is reaching the first stages of maturity. Its a fairly simple engine with nice, clean syntax:

{IF VAR == “fu”} fu {ELSE} bar {!IF}
{LOOP ITER_OBJ} ITER_OBJ.current {!LOOP}
{VARIABLE}
{IMPORT header.tpl}

It does caching also.

The page dispatching agent is where it gets a little more involved. To better explain the agent i’ll give you a quick lo down on the definition of a ‘domain’. A domain is simply a directory which contains sub domains, which in turn can contain sub domains of its own, for example: http://host/domain1/subdomain1/subsubdomain1/. The key feature is that domain modules can inherit ‘parent’ modules contained within the current domain or the domain above it, furthermore, parent modules can also inherit other parent modules from the domain above and so forth.

So, when the dispatching agent is told to load a specific domain module, it searches the hierachy for a module whos name matches the requested path and imports it. Hey presto! Only the code needed to handle the request has been loaded.

We also have ‘site’ modules which contain various classes and functions specific to domain modules (as little code as possible is contained within a domain module. As much as possible is abstracted into a site module), but i’ll not bore you any longer with those.

And finally the web server. This daddy does all the handing off to the page dispatcher and not a lot else in terms of servicing a quest, but wait, I haven’t told you about it’s threading model yet.

*deep breath*
As I mentioned earlier, one of the reasons for not using CGI was the speed factor, I couldn’t just implement a standard web server now could I (even though having the interpreter in memory is the main benefactor)?

The web server starts off with a base number of 4 worker threads ready and waiting to handle a request. If 4 proves to be a little too few, the server will create a new worker thread and add it to the queue. If 5 isn’t quite enough… etc, etc.

While all this is going on, another part of the web server is timing how long each request took to complete and is keeping a n point moving average of request times. If the past few averages got larger after adding more threads to the queue, the server will hold off and not allow any more worker threads to be created. As the load goes down it’ll gradualy kill any threads not doing anything.

I’m sure some of you are sitting there thinking ‘DoS’ to yourself, well no need to worry, as the server will not allow threads to spawn too quickly and no more than n in the last i minutes.

This threading module is all theoretical at the moment and if it actually improves performance under heavy loads will remain to be seen.

With the end of the exam period now in sight, i’ll be working on this as much as I can. A beta release shouldn’t be too far off, with a public release of GLSR following shortly afterward.