Google Summer of Code update #1

So far in the Google Summer of Code 2013 I’ve been working on Ganeti Web Manager.

My job was to refactor Ganeti Web Manager so that instead of one huge, bloated Django application (ganeti_web), the project would consist of many smaller apps.

I shared more details in this blog post.

Now I can say my work is mostly done. I’ve got smaller apps extracted and (this one’s big) I worked on database migrations.

Now (hopefully!) any user can checkout my branch and successfully run it against their codebase.

Why are you doing this, one can ask. There’s no visible benefit for end users. Actually even worse: they for example have to use settings file which is now stored completely elsewhere.

This work brings hope and sunlight to all the GWM developers. It helps new ones “see” and understand code, as the code is now simpler! And it helps old devs navigate through the code, while fixing or enhancing it.

I hope to get my work merged soon, although it’s quite big change so the proper code review is really necessary.

Next on the GSOC agenda: taking care of Ganeti Web Manager dependencies. I.e. documenting, testing, uploading to PyPI, maybe even setting up a CI.

Radio controlled robot

Since last October I’ve been involved into Focus Scientific Circle (or group) at my university. (Yeah, I know our webpage is horrible…)

From the very beginning we wanted to make a robot. A cool one, precisely. So the idea we had was: let’s take Kinect, some mobile platform/chassis, remote control, Raspberry Pi and mix it all together.

I took a long, long time, but finally we have a semi-working prototype. This week my friends made it possible to control our platform via remote control (the same used to control high-end RC aircrafts). That’s a huge progress for our project.

Meanwhile I’ve been working on special algorithm, which is going to help our Kinect orientate in space.

The idea is to make 3-D maps of insides. The thing is, Kinect doesn’t revolve around any solid axis. Thus I think we could track (via Kinect’s camera) a set of points (features) and measure by which angle our robot turns.

If we know that angle, we could make those 3-D views of insides.

I’m going to use OpenCV and code everything in C++. Hopefully it will work quite fast on our Raspberry Pi.

Ganeti Web Manager Modularization

I’ve got accepted in for this year’s Google Summer of Code. My project requires me not only to program, but also to improve automation and work flow for GWM developers.

The least Django version Ganeti Web Manager is compatible with is 1.4.5. However, GWM doesn’t use the new project layout introduced with Django 1.4.

The result is GWM’s main directory being very cluttered. As for 0.10 release, there are 25 files and/or directories. With just the switch to newer project layout, only 21 files/directories exists now in top level dir. That number will soon decrease, as more files get moved to appropriate places.

I’m sure setting files and files containing URLs will get tossed out into related Django application directories, too.

Speaking of which: originally GWM was written as one single Django application. Since commit dde1fdf1cc (WIP) it’s now splitted into many smaller applications: auth (I might change that name in future), clusters (containing code related to clusters; surprise!), ganeti_web (original app), jobs, nodes, utils (or doesn’t-fit-anything-else category), and virtualmachines. Two additional apps are moved from top-level directory and both relate to Muddle. I’m not sure about their fate; it’s possible that they’re gonna get sentenced.

The rationale for such splitting is mainly the construction of Django models in ganeti_web application. That single file itself changed its size from 1,958 down to 295 lines!

Next steps in GWM modularization will involve moving forms, templates, tests, views, models, and URLs to appropriate places. And testing, lots and lots of testing is required.

I also hope to document my work and general progress at least here. GWM recently started hosting its documentation on the ReadTheDocs, I’ll update it as well.

All in all, I’m very happy with my project and GSoC 2013. :) Its timeline has shifted so now GSoC covers more of my summer holiday. #win

PyCon US 2013 afterthoughts

It’s been almost two months after PyCon US and I finally have some time to share my thoughts and experiences from that trip.

What a journey was that, boy, what a journey! The biggest in my life so far. I started in Kraków, Poland, then travelled by bus to Wrocław, Poland (circa 3-4 hours drive). I’ve met with my friend Kacper from Wrocław and my future travel companion Filip (who’s PyCon PL organizer).

The following day, 12/03/2013, we had our flight to Münich, Germany first thing in the morning. After such a short night, I fell asleep really easy.

From Münich we had our transcontinental flight to Philadelphia, Pennsylvania, US. That was such a long flight. In eco class you don’t get anything fancy…

However, as it turned out, the airplane had many conveniences and was quite luxurious (when compared to the airplane I was sitting in during flight to Europe). Every passanger had his/her own TV with games and music and so on. It wasn’t snappy, but it worked.

And there was more space for legs than in our family car! I was totally surprised by this fact.

From Philly we flew to San Francisco. California, finally! At the time, I couldn’t believe I was there. I still can’t. That feeling of being in a beautiful, old city with great history, great buildings and great weather (it was snowing back then in Poland! Horrible!).

San Francisco has many tourist attractions. I could only afford to spend two days sightseeing it, but those were the most interesting days of my life.

I visited Market Street, Union Square, Lombard Street, Fisherman’s Wharf, Golden Gate Bridge, Golden Gate Park, beach by the ocean, Haight-Ashbury and maybe some I don’t remember. Oh, and of course I drove in a Cable Car.

I must admit, I never thought Americans are so positive and kind to foreigners. Or even to everyone! They often say “Hi!” and start a conversation… This is absolutely lovely and amazing. No-one in Poland behaves like this. I hope I will in some little part be as positive as Americans are.

But apart from sightseeing, I was there to enjoy the conference. And it was undoubtedly stunning! I met so many great developers, including my former GSoC fellow Corbin and my mentor Ken.

Now I find PyCon so wonderful, it’s hard to describe in words. I’m certainly looking forward to Montreal 2014 edition. If you want to know what PyCon is like, you should totally come. (Hopefully it’s warm in Montreal in April!)

I was in States for almost a week. Five or six days. I went home without Filip the same route I had come in.

San Francisco, I miss you…

Late for Christmas

I know, I know. Christmas was one month ago. But I though I’d share my personal favorite christmas tree me and my roommates made.

We used empty pizza box (Capricciosa was the choice, I believe), about 14 green LED lights and Netduino platform.

I mainly took care of soldering almost all the pieces together, and my friend wrote small application able to turn on and off random lights, or make a little lightshow.

Here should be photos. Enjoy like we did!

View post on imgur.com

First ever academic semester ends

This time next week I’m going to be after my first ever end term exam (Physics in this case). Should’ve started learning to it waaay before…

My Raspberry Pi was finally shipped. I received it over two months later, with 3 weeks delay. So far it works as a server with no content. I hope to buy some magical wireless keyboard (with touchpad on it) and make my very own media center.

I’ve already tested Xbian and it worked, although not as snappy as I hoped. My guess is that SD card I used was too slow.

The next academic semester will be SO HARD. It’ll end with 4 horrible exams. My scientific group Focus also received financial grant, so we’re gonna build some awesome robot.

All this will make me so much more busy than current, ending semester. Not good.

By the way: I applied for PyCon US financial aid, but the demand was so huge, that they had to give everyone partial funding. Thus I am not likely to attend this years Pycon. Maybe next year, when I save some more money. “You can’t always get what you want”, right? ;)

First academic semester

After this summer work with OSUOSL (during Google Summer of Code) I started my actually first ever ( ;) ) semester in university. In comparison with high school, in university everything is simply 3x harder.

Physics is 3 times harder, calculus is 3 times harder, learning all that stuff is 3 times harder, and even getting up at 8AM is 3 times harder.

Thankfully I have more spare time due to the lack of humanities… STOP! Nonsense! Someone apparently wiser than me knows that I need TWO totally useless humanity courses. Not that I’m trying to become an engineer, because I should be fricking Shakespear as well!

Okay, enough rant. :)

I always pursue some additional challenges. This year with a couple of friends we decided to join Focus scientific group at AGH University of Science and Technology. It’s primary field of work is vision systems, including some Artificial Intelligence.

(Apparently I became Focus president, by the way.)

Me and my friends have some idea of a RC car (with Kinect and Raspberry Pi onboard), ideally autonomous.

My very own Raspberry Pi is on its way from somewhere (Asia?), should get it in 3—4 weeks. I hope I can compile and use OpenKinect and OpenCV software on it.

This also means that I’ll most probably not use my beloved programming language Python for some time. :( But what can you do.

Last very imporant news: at yesterday’s Google Pizza Event for AGH I discovered how cool were Google Internships. Now I want to go for one soo much :)

Google Summer of Code 2012 ends :(

These 3 months passed quicker than I thought. I’m very glad I had this opportunity to actually write code, earn money and make something useful. However, a small group of people who will likely use my code in their work, and who was supporting me from beginning, is what I’m mostly glad of.

During these 3 months I managed to get valuable experience, find out new technologies (like Vagrant, Flask) and fulfill most of my project’s goals. I want to sincerely thank my GSoC mentor, Ken Lett, and our sysadmin, Lance Albertson, for great work, patience and support. I wouldn’t do this all without your help, guys :) Thank you.

Now some highlights of the visible part of my project. Here are some screenshots available.

Serialize SQLAlchemy query results to JSON

Consider you have following model and you want to create super—duper RESTful interface to it.

from application.database import db

class BlogPost(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(50))
    content = db.Column(db.Text)

And here’s one of your views, which returns all the available blog posts:

from flask import jsonify
from application.models import BlogPost

@app.route("/posts")
def blog_posts():
    return jsonify(posts=list(BlogPost.query.all()))

But unfortunately SQLAlchemy results cannot be serialized to JSON. What to do?

You can spend some time and extend SQLAlchemy’s pickle serializer extension (it’s awkward to extend what’s already extended, but nevermind), however there’s a quicker way. Just make your model class inherit from this base class as well:

from collections import OrderedDict

class DictSerializable(object):
    def _asdict(self):
        result = OrderedDict()
        for key in self.__mapper__.c.keys():
            result[key] = getattr(self, key)
        return result

simplejson looks for _asdict() method when iterating through objects to be serialized. We can cheat and return our model and its fields as an ordered dictionary, which is then easily parsed to JSON.

This might not be the quickest method, but certainly is the easiest one.

Flask and SQLAlchemy: init_app

I’ve been recently refactoring my Flask application and one of the most important changes I’ve made was an application factory. You can find out many different (but similar) syntax constructions to use SQLAlchemy with it. Here one and most common:

from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app():
    app = Flask(__name__)
    db.init_app(app)
    db.create_all()
    return app

However, you might stumple upon this exception: RuntimeError: application not registered on db instance and no application bound to current context, perhaps in your tests on during db.create_all().

Documentation isn’t that clear on that matter. After a lot of searching I found out three options to solve this issue.

Option #1

Create your db object with app as a parameter:

def create_app():
    app = Flask(__name__)
    db = SQLAlchemy(app)
    db.create_all()
    return app

The drawback is that you don’t have a global db object and thus your blueprints won’t work. And you won’t be able to easily create models.

(BTW: setting global db and then db = SQLAlchemy(app) did not work for me.)

Option #2

Assign created application to db.app:

db = SQLAlchemy()
def create_app():
    app = Flask(__name__)
    db.init_app()
    db.app = app
    db.create_all()
    return app

The possible drawback: you might mess something up in db’s internals. You don’t want that, do you?

Option #3

Call db.create_all() (that’s the thing we had troubles with) within application context:

db = SQLAlchemy()
def create_app():
    app = Flask(__name__)
    db.init_app(app)
    with app.test_request_context():
        db.create_all()
    return app

The drawback: it’s test_request_context (test!). But, after all, it works fine for me. And all my tests pass! ;)

Use any method you want.

If you had this issue and you solved it another way, please let me know in the comments.

Additional tips

You will need to import models themselves before issuing db.create_all:

with app.test_request_context():
    from application.models import Post
    db.create_all()

It’s a good idea to keep your SQLAlchemy object instance in separate file, to avoid circular imports:

# application/database.py
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()

# application factory
from application.database import db
def create_app():
    ...
    db.init_app(app)
    ...

# application/models.py
from application.database import db
class Post(db.Model):
    ...