Blog by Piotr Banaszkiewicz

Archive About

Peer instruction: update 5

This is gonna be a short one. Because of exams, I have to study a lot. And because there was not much progress on Pitt during this week.

Current release

Current version is v0.3.1, and it’s tagged in git repository. I’m incorporating the git-flow model.

What I wanted to get in was two major features:

  1. countdown before switch from group split to broadcasting mode,
  2. small groups of students of variable-defined size.

I can tell that I was successful and these features are now part of v0.3.1 release.

Bugs

While testing with Greg and Mike at their office and me here in Kraków, I couldn’t connect to them and vice versa. Connections between Mike and Greg worked as intended.

This was strange, because I didn’t change the code so much that it’d break. More so: last week while testing with Greg alone it worked!

So I conducted few more tests:

1) Pitt: Kraków ↔ Kraków 2) Pitt: Kraków ↔ Toronto Mozilla Office 3) Pitt older version: Kraków ↔ Toronto Mozilla Office 4) Other WebRTC application: Kraków ↔ Toronto Mozilla Office

In the previous week, I’m guessing Greg worked from his home. We wanted to try Pitt in this configuration (Kraków ↔ Greg’s home), but due to holidays in Canada we were not able to.

From what I can tell, the issue lies in the client IP addresses resolution. Here’s a lengthy article about how hard it is to do WebRTC properly: http://www.html5rocks.com/en/tutorials/webrtc/basics/#toc-signaling

Anyway, I hopefully will be able to easily overcome this issue.

Peer instruction: update 4

This week me and Greg Wilson, my mentor, we decided to test run Pitt. It was a success and, while revealing some annoying bugs, it made Greg really excited.

Current release

Current version is v0.3, altough I might have tagged it wrong in git. Fortunately it seems that tag was not published to the GitHub repository.

Current features

This is a quick recap on what Pitt is capable of right now.

  1. Users coming in and quitting: thanks to signalling, as soon as someone appears online, they’re being logged in and everyone can see them.
  2. Quick switch mode: the instructor can easily and very quickly switch modes from broadcasting (to all students) to work in small groups (students are split into pairs or groups of 3).

The code is decent. I presume there are some bugs with saved state of the application. For example, what to do if the instructor is broadcasting and someone joins in? What if another instructor joins in? What if there are two instructors and one starts broadcasting, should other instructors see their stream too?

These kind of questions will be addressed for the v0.4 release. Right now I’m working to get v0.3.1 release by Friday, 27.06.2014.

Exposed bugs

During my test run with Greg we quickly found out one super annoying bug.

There’s a very interesting human behavior: people slow down their talk when they hear themselves with small delay. It’s really hard to overcome this basic reflex.

Back to the bug: we found out that local stream video (ie. the one where you can see yourself, because browser plays the video from your webcam) is not muted by default. Add the playback delay to the mix and as a result Greg was struggling to speak.

This bug was really easy to fix (in HTML5: <video src="..." muted="muted">) and the fix is included in the recent code.

There was also issue with the state of the application - it somehow broke for Greg (when he was a student). I have yet to discover the actual bug.

Missing features

Greg wants me to add these two new features to the upcoming release:

  1. Countdown. The instructor presses “back to broadcast mode” button and the switch doesn’t happen immediately. First there’s 30 seconds countdown so that no students are interrupted in the middle of the sentence.
  2. Variable group size. The instructor can specify if the students are split into pairs or groups of 3, or groups of 6. This task is more challenging than it seems: I have to ensure that no students are left alone (what is often the case when there’s odd number of students).

Next release

I’m planning to release next version, v0.3.1, on Friday 27.06.2014 or earlier.

I have 3 exams before Friday and 2 after it. Ouch…

Peer instruction: update 3

The decision has been made, and I switched to a different codebase.

I had one exam this week (Aparatura Automatyzacji, eng. Automation Systems) and five more in next 2-3 weeks, so it is quite challenging for me to find time for peer instruction right now.

I did, however, spend most of this weekend trying to make Pitt broadcasting mode for instructors.

It works! Proof below:

Me as instructor and two students. Having dual monitor setup helps getting this kind of screenshots. (click to enlarge)

As you can see, the layout and design in general is very rough. And so is the code.

This code is mostly based upon multiple events being propagated through sockets.

But I really don’t like the design of these events, and, frankly speaking, I much more prefer a “pub/sub” (publication - subscription) architecture.

At this moment, when the instructor triggers an event, the server has to repropagate it to every student.

In “pub/sub” arch, however, there’s no repropagation in between.

Some people use redis for pub/sub, but I found some really cool protocol: WAMP (don’t confuse it with Windows, Apache, MySQL, PHP stack!).

WAMP authors clearly explain why it is good so please go there and read. Let me just point out a few cool features of WAMP:

  • WAMP is not a NodeJS module, it’s a protocol
  • it can be (and indeed is!) implemented in various languages
  • therefore it’s a solid foundation for a quickly changing service.

What I dislike about WAMP is that at the moment there’s only one leading implementation: Autobahn. And I’m not yet sure if I want to drop NodeJS backend in the future, but if so, then there’s already fast Autobahn|Python.

Peer instruction: update 2

In this week’s videoconference between my mentors, Greg Wilson and Mike Hoye, Software Carpentry’s instructor David Rio, and me, we decided to temporarily drop my current codebase and what I’ve done with Licode and Erizo.

Instead, my current efforts will be focused on developing and enhancing David’s excellent work that uses PeerJS.

This has some interesting consequences.

Firstly, we can’t have broadcasting using MCU in teacher’s mode. Instead, in this mode, we’ll have to open P2P connections, what’s probably going to be quite challenging for teacher’s internet connection.

Note: In teacher’s mode, the teacher broadcasts his stream to all of their students. In student’s mode, students are split into smaller groups (6 people per group in real life, at most 4 people via Internet - more is less efficient) and talk with each other.

But for now we want to ship first version as soon as possible, so we’ll care about that later.

Secondly, we’ll have a lot clearer and “closer to the metal” code. Opening sockets and handling all data transfers will be much easier, and I like that fact a lot!

On my personal side of news, I’m currently preparing for exams (first one: next Thursday), which leaves me very little time for GSOC #sadface…

Peer instruction: update 1

I always avoided JavaScript. With my long exposure to Python, JS never appealed to me. And yet my whole Google Summer of Code project consists mainly of JS.

Let me quickly reintroduce basic ideas and plans for Peer Instruction project.

Peer instruction concepts

The most important feature of Peer Instruction is the ability to, from the technical point of view, quickly switch between teacher and student modes.

Teacher’s mode

Teacher mode is basicly a broadcast sent to every student in a classroom. This might be quite challenging: you can have many direct connections and suffer from capped bandwidth and high CPU usage.

Teacher broadcasts their stream to multiple students Teacher broadcasts their stream to multiple students. (click to enlarge)

The solution is to introduce additional server that helps to “spread” the stream to multiple students. It essentially looks like this:

Teacher broadcasts their stream to multiple students through Multipoint Control Unit Teacher broadcasts their stream to multiple students through Multipoint Control Unit. (click to enlarge)

What’s really good about this design? There’s an excellent open-source MCU available: Licode. I must admit Licode is a little too complicated to set up, but authors’ support and continuous development make it a great option if someone wants, for example, to host their own alternative to Google Hangouts or Skype.

Me testing teacher's mode Me testing teacher’s mode. (click to enlarge)

As you can see above, the interface is pretty simple, but it works (at least for 1 student). I have yet to test it for more students, but I haven’t received a server account at Mozilla to do so.

Student’s mode

This is the second mode of broadcasting. A very crucial from the point of science of peer instruction.

In this mode students are split into smaller groups so that they can talk with each other.

Right now I’m thinking how to implement this mode. One of the ideas is to create one Erizo Room for every group of students.

The way events are propagated within Erizo makes it really easy to subscribe to other students’ streams within one particular room. Using multiple rooms also helps manage all these smaller groups.

I should also mention that Erizo can create “full MCU” rooms - ie. rooms where streams are going through the server - and Peer-to-Peer rooms, where the server only handles sessions, signalling and events, but the streams are actually transferred between the room participants.

I think for small groups of 2-4 students it’s a good idea to use P2P rooms and save resources on the server.

Unfortunately, this solution has some drawbacks, too. I found out that Erizo can only handle so much rooms at once (it’s specified in configuration, but I don’t know the retionale behind it).

Current issues

My main focus for next week(s) is working PoC of students mode. Right now I’m working on a protocol used by Licode events.

It’s possible to send data through Erizo streams. I want to use this “channel” to communicate with students’ browsers.

For now I’ll only send data like “join room ABC for small group discussion” and “leave room ABC”. In future Peer Instruction might be more advanced - quite important feature is chat, and I will likely leverage the existing protocol for that.

Anyway, that’s the update status on my project. I highly welcome any contribution or comments.

Here’s project GIT repository: https://github.com/pbanaszkiewicz/peer-instruction

Peer instruction - a summer project for Mozilla

I was accepted for Google Summer of Code 2014. My project is called “Peer instruction”, and I’m doing it under the wings of Mozilla foundation (or, more specifically, Software Carpentry).

Peer instruction

Science tells us that students in groups learn more effectively 1. This concept is widely used in peer instruction.

Peer instruction is a teaching methodology developed over 20 years ago. It’s different from typical learning classes in today’s colleges or universities. The basic workflow goes more or less like this:

  1. Teacher asks a question.
  2. Students individually answer it.
  3. Students are split into small groups to discuss their answers.
  4. Teacher reveals the good answer.

The key feature of this methodology is splitting students into small groups.

Summer project

If you take an online classes by Udacity, Coursera, Khan Academy or whoever there is, you may notice they don’t deploy peer instruction.

But with web technologies of today, it is possible to create a virtual classroom that leverages this promising teaching methodology! And my task is to do it.

The minimum outcome

No, I don’t want to create a “full-stack” virtual classroom. The minimum viable product of my work will be a proof-of-concept working web service that lets you quickly switch from broadcasting (or “teacher mode”) to many few-to-few multiplexed broadcasts (or “small-groups-talk mode”).

Of course, if I can, I’ll implement as many additional features as possible.

The obstacles

The only two difficulties I foresee right now are:

  1. The lack of fast MCU technology.
  2. The lack of fast internet.

MCU stands for “Multipoint Control Unit”, a software or hardware used to bridge media transmissions.

The only one open source software MCU for WebRTC (technology used for media transmission in modern browsers) I found so far is Erizo. I may have to implement an Erizo API module for Python if I decide to stick to Python.

So that’s the only issue I can somehow overcome. I cannot fix the internet speeds, though…

Possible features

If I have enough time and knowledge, I’ll implement:

  • whiteboard
  • voting system
  • quizes
  • translations

It’s going to be exciting summer. I’m looking forward to it! :)

Contributing

I welcome and highly appreciate any feedback, ideas, suggestions, or complaints. I released publicly my proposal, so you can comment it or even fork and make a better one.

I also want to share the link to the Git repository, where my work will reside. Feel free to send pull requests :)

  1. Peer tutoring is ranked as having 0.55 influence on student’s achievement, while 0.0 is none. ↩

How to bite Flask, SQLAlchemy and pytest all at once

As a person who started with Django, I had some hard time figuring out how application and request contexts work in Flask. I think I finally got to understand them. I also learnt how to use SQLAlchemy’s scoped sessions properly and how to test my REST applications with pytest efficiently.

Here’s what I got to know, split into topics.

Warning: Flask and SQLAlchemy are very flexible, compared to Django. This article presents only one approach out of many on how to deal with thread-local stuff in your code. But I think that it’s one of the best approaches.

  1. How does SQLAlchemy handle sessions
    1. Scoped sessions
  2. Transactions in SQLAlchemy
  3. What are Flask ap­pli­ca­tion and request contexts
    1. Ap­pli­ca­tion context
    2. Request context
  4. Zipping Flask request contexts and SQLAlchemy scoped sessions together
  5. Testing everything
    1. Fixtures
    2. Fixture scope
    3. Trans­ac­tions in tests
    4. Cooler fixtures
  6. Sewing it all together: Flask, SQLAlchemy and pytest
    1. Actual ap­pli­ca­tion
    2. Tests with pytest
    3. Boil­er­plate

How does SQLAlchemy handle sessions

Your SQLAlchemy Session is the main place where you usually talk to your database.

# somewhere globally in your application
from sqlalchemy.orm import sessionmaker
session = sessionmaker()

# somewhere in your views
users = session.query(User).filter(User.name == "Piotr").all()

This code is perfectly fine if used non-concurrently. Non-concurrently, that means only one user can use session at once; that means, only one user connects to your website at once.

In reality that’s not always a case. I bet you want your site to be safely accessible by many users all at once.

If so, then you can’t use sessionmaker alone. It’s not designed to create safe Session objects for multiple threads.

So what’s the solution? It’s actually pretty clever. It’s called scoped sessions: sessions, that are bound to the specific “scope” of your application, for example: the scope of one user’s connection.

Notes: “Scoped sessions” is a different programming pattern than sessionmaker. The former is a registry pattern, whereas the latter is a factory.

Scoped sessions

It’s quite easy to understand how session for application scope is being constructed. For every incoming request, a different Session object is being served.

So user Mark gets session A, user Ellen gets session B and user Sam gets session C. The key is that all these sessions are accessible in your code via the very same line:

# somewhere globally in your application
from sqlalchemy.orm import scoped_session, sessionmaker
session = scoped_session(sessionmaker())

# somewhere in your views
users = session.query(User).filter(User.name == "Piotr").all()

(Look at the previous snippet; they’re almost the same!)

All the required setup is this one little object, scoped_session. You feed it with some session-factory-maker, like sessionmaker, and voilà.

There’s only one part missing… how does scoped session know when to “spawn” a different session? It somehow has to recognize that a new user is requesting your views.

I’ll come back to that in later after explaining what are Flask application and request contexts and how to work with them.

Transactions in SQLAlchemy

SQLAlchemy supports at least two different kinds of transactions. The most popular type is Session based transaction:

u1 = User(name="Piotr", email="test@example.org")
session.add(u1)
try:
    session.commit()
except sqlalchemy.exc.IntegrityError:
    session.rollback()

The second type is more superior. It can roll back even committed session changes! It’s really powerful for testing purposes.

What are Flask application and request contexts

Application context

I like to think about Flask application context as being bound to one thread of your actual application (website). That context might be a set of global objects, like database connection and app settings. These objects should only exist once per your application, right? (I don’t see a point in duplicating app settings or database connections all over the place).

Note: SQLAlchemy provides a pool of connections to the database. You can pop a connection any time and push it back after you’re done. This, however, doesn’t mean you have to pop two or more connections at once!

In Flask, current_app is aware of the active application context. If you have your web application running on two threads, and one user accesses the first thread, they’ll use different Flask application than the other user accessing second thread.

Request context

Request context is very similar to the application context. Every time anyone goes to some page on your site (ie. sends request), a new context is created.

This new context holds information that should only be available within that particular second when the user is being served. I’m assuming you can serve your user within one second :)

For example, imagine you have a view that adds a new blog post to your site:

@app.route("/blogpost". methods=["POST", ])
def blogpost_view():
    return "New blog post: {}".format(request.form)

Flask internals ensure that you do not access a different request’s data. Two requests may be simultaneous, yet you will access exactly the correct request in your code.

Note: New request context creates new application context, if the latter is not available.

Zipping Flask request contexts and SQLAlchemy scoped sessions together

So now you know what powers Flask contexts and that you should choose scoped SQLAlchemy sessions over “normal” ones. But how to make a scoped_session that works with Flask contexts?

Take a closer look at scoped_session. You can see it has a scopefunc argument:

scopefunc – optional function which defines the current scope. If not passed, the scoped_session object assumes “thread-local” scope, and will use a Python threading.local() in order to maintain the current Session. If passed, the function should return a hashable token; this token will be used as the key in a dictionary in order to store and retrieve the current Session.

So… scopefunc has to unambiguously represent each individual context. I was looking for a good way of handling that, and found one in Flask-SQLAlchemy. This Flask extension uses internal context stack to build hashable context tokens. The code looks like this:

# somewhere globally in your application
from flask import _app_ctx_stack
from sqlalchemy.orm import scoped_session, sessionmaker
session = scoped_session(sessionmaker(),
                         scopefunc=_app_ctx_stack.\_\_ident_func__)

Testing everything

Because of the aforementioned flexibility that Flask and SQLAlchemy have, I had really hard time figuring how to test the whole thing. Testing is very important, and with the help of wonderful Python libraries like pytest it actually becomes a pleasure.

Still, when trying out pytest for a first time, there is a small learning curve if you come from Java-based unittest world.

The biggest change is in the ideology: now you don’t have to write classes (test cases) to test your code. You can write a lot simpler functions instead.

The important feature of pytest is fixtures. Use them when you want to set up or tear down your tests.

Fixtures

A fixture is a function that, for example, returns a database session object, which can be leveraged by your tests.

Or it can return a file descriptor to the file in /tmp/random_name. Or your application object. Or Redis connection object.

Look at fixtures docs for more examples.

Fixture scope

Every fixture can be set for a session scope, module scope, or function scope. This means, that the fixture is only run once per testing session, or once per whole module (containing tests), or once for every test function.

Take for example this db_connect fixture.

import pytest

@pytest.fixture(scope="session")
def db_connect(request):
    db = sql.connect()

    def teardown():
        db.close()

    # There's no "add_teardown", see comments below
    # request.add_teardown(teardown)
    request.addfinalizer(teardown)

    return db

It’s dumb and won’t work, but I hope you get the gist. Even if you have a thousand tests that use this fixture, it will be invoked only once, then memorized (cached).

Note: This small fixture uses another fixture! request is a built-in pytest fixture that helps you with teardowns.

I suggest to (at least) create a session-scoped fixture that builds your Flask application object (using application factory), and a session-scoped fixture that builds your SQLAlchemy session and manages transactions.

Transactions in tests

Shortly: it’s way faster to roll back all the changes from database than to recreate whole database from scratch on every new test.

Cooler fixtures

I really like the fixtures that leverage Python’s yield statement.

With yield, the above fixture example looks a lot clearer now:

import pytest

@pytest.yield_fixture(scope="session")
def db_connect(request):
    db = sql.connect()

    yield db

    # everything after yield statement works as a teardown code
    db.close()

Sewing it all together: Flask, SQLAlchemy and pytest

Actual application

  1. Use application factory to easily create Flask application object. This will be used in different parts of your codebase, like tests or dev server.
  2. Create global scoped_session object that spawns actual SQLAlchemy sessions when accessed. Use scopefunc keyword argument to provide hashable function that’s used to recognize context switches.
  3. Don’t bind that scoped_session object to any engine yet. Bind it in your application factory using scoped_session.configure().
  4. During app.teardown_appcontext remove database sessions.

Tests with pytest

This one’s more complicated, so I’ll paste some boilerplate below.

  1. In your conftest.py prepare one session-scoped fixture that creates your app (using factory), creates all the tables, explicitely pops a connection, binds global scoped_session to that connection and yields that app
  2. Prepare second fixture, that creates a new transaction, new application context and yields database session.

Boilerplate

Here’s that promised boilerplate. First application factory create_app:

# database session registry object, configured from
# create_app factory
DbSession = scoped_session(
    sessionmaker(),
    # __ident_func__ should be hashable, therefore used
    # for recognizing different incoming requests
    scopefunc=_app_ctx_stack.\_\_ident_func__
)

def create_app(name_handler, config_object):
    """
    Application factory

    :param name_handler: name of the application.
    :param config_object: the configuration object.
    """
    app = Flask(name_handler)
    app.config.from_object(config_object)
    app.engine = create_engine(app.config["SQLALCHEMY_DATABASE_URI"])

    global DbSession
    # BaseQuery class provides some additional methods like
    # first_or_404() or get_or_404() -- borrowed from
    # mitsuhiko's Flask-SQLAlchemy
    DbSession.configure(bind=app.engine, query_cls=BaseQuery)

    @app.teardown_appcontext
    def teardown(exception=None):
        global DbSession
        if DbSession:
            DbSession.remove()

    return app

And test configuration from conftest.py:

from your_application.global import create_app, DbSession

@pytest.yield_fixture(scope="session")
def app():
    """
    Creates a new Flask application for a test duration.
    Uses application factory `create_app`.
    """
    _app = create_app("testingsession", config_object=TestConfig)

    # Base is declarative_base()
    Base.metadata.create_all(bind=_app.engine)
    _app.connection = _app.engine.connect()

    # No idea why, but between this app() fixture and session()
    # fixture there is being created a new session object
    # somewhere.  And in my tests I found out that in order to
    # have transactions working properly, I need to have all these
    # scoped sessions configured to use current connection.
    DbSession.configure(bind=_app.connection)

    yield _app

    # the code after yield statement works as a teardown
    _app.connection.close()
    Base.metadata.drop_all(bind=_app.engine)


@pytest.yield_fixture(scope="function")
def session(app):
    """
    Creates a new database session (with working transaction)
    for a test duration.
    """
    app.transaction = app.connection.begin()

    # pushing new Flask application context for multiple-thread
    # tests to work
    ctx = app.app_context()
    ctx.push()

    session = DbSession()

    yield session

    # the code after yield statement works as a teardown
    app.transaction.close()
    session.close()
    ctx.pop()

Coffee addiction

I love coffee. I really do. But, unfortunately for me, I don’t drink it anymore.

When I did drink it, though, it was a time of a day just for me. A moment for me and coffee. I sat happily relaxed, eating some cookie or even bigger sweets, sipping quickly while the mug was hot and reading news or watching movies from YouTube.

However, due to my heart condition (I have arrhythmia), my doctor ordered me to live more healthy (sports and such) and stop drinking coffee and alcohol.

I’m sure I wasn’t abusing the latter, but coffee? I drank two mugs a day, sometimes even more.

A little background on me: I’m a student, I’m often exhausted from different activities at my university that are 12 hours long, parties with friends and the lack of sleep.

I sometimes woke up, ate breakfast and went to sleep again because I was too tired to be able to do anything.

Sometimes after eating lunch I had to take a nap, because my body wanted to sleep! So that’s how I started drinking second coffee in the afternoon or after lunch.

It helped for a little bit, but then I started again taking afternoon naps.

Until… I was forced to stop drinking coffee. Now I’m feeling better in the mornings, I started paying closer attention at early 8AM lectures, I don’t even take an afternoon nap anymore! With the same amount of sleep (circa 6hrs daily… I know, it’s killing me slowly), I feel better without coffee.

Why?

I think the reason is simple. Morning or afternoon coffee was making me asleep.

How so?

Because of sugar. I was eating a lot of it (in cake form) when drinking coffee. Do you know how your body knows it’s time to sleep? Because the level of glucose in your blood increases, sending a signal “stop everything and start consuming food” to the brain.

What instead?

I have to drink something… So I started drinking mineral water and my favorite tea: green tee!

I always liked it for being sparser and lighter than coffee, while still keeping me up in the morning.

One can say: Tea contains caffeine! It’s almost like drinking coffee!

No, not really. Studies show that green tea has about 4-7 times less caffeine than coffee.

And I’m drinking poor quality green tea, so it may have even less caffeine. But I like it, I won’t switch. Until the next appointment with my doctor…

I’ve been coffee and alcohol abstinent for 24 days now. And I’m feeling a whole lot better.

Ganeti Web Manager installation

Due to my efforts, Ganeti Web Manager is now a proper Python Package.

The main reasons for this change are:

  • minimizing necessary dependencies (no more fabric!)
  • keeping GWM codebase clean
  • easing development

What does this change mean?

For end users

With GWM being Python package comes one huge advantage: easier installation. It’s actually one line to install whole GWM with dependencies!

Because my GSoC project is to make GWM installation easier for end users, I’m writing a setup script, that will create virtual environment (helps with separation), install GWM’s dependencies and then GWM itself.

To install Ganeti Web Manager, simply get setup.sh script, make it executable and run setup.sh -h to get familiar with it’s command line arguments.

$ wget -c https://github.com/pbanaszkiewicz/ganeti_webmgr-setup/blob/develop/setup.sh
$ chmod +x setup.sh
$ ./setup.sh -h
$ ./setup.sh -d /opt/gwm

Note: As of 2013-08-29, this setup script is Work In Progress (TM). Some parts of it may not be completed and simply won’t work. Be careful. Or wait for Ganeti Web Manager next release, which should be 0.11.

For developers

To start working on GWM, you have to:

  1. make virtual environment
  2. git clone GWM repository
  3. install GWM as a development package

(I suggest using virtualenvwrapper for #1, as it keeps your directory with code clean.)

In shell terms it looks like this:

$ mkvirtualenv gwm
(gwm)$ git clone git://git.osuosl.org/gitolite/ganeti/ganeti_webmgr
(gwm)$ cd ganeti_webmgr
(gwm)$ python setup.py develop

And that’s it, now you can work on GWM. (Alternatively you could install GWM development dependencies instead of whole package:)

(gwm)$ pip install -r requirements/development.txt

Sublime Text 3 for Python development

I’ve been heavy vim user for a couple of years. I had a very cool configuration, really got all these fancy vim key shortcuts muscle-memorized.

But then vim plugins (bundles) started annoying me. Often my configuration became obsolete after an upgrade. I had to go through documentation and set everything yet again.

So I decided to give Sublime Text 2 a try, and then Sublime Text 3. And in August 2013 I switched almost completely. I still love vim, but now I program mainly in Sublime Text 3.

  1. Why switch to Sublime Text 3?
  2. Brief history
  3. Configuration
    1. Theme
    2. Typeface
    3. Python
    4. Behavior
    5. Configuration file
  4. Plugins
    1. Package Control
    2. Git
    3. GitGutter
    4. SideBarEnhancements
    5. Anaconda
    6. Gist
  5. Final words
  6. Two years later update

Why switch to Sublime Text 3?

For me, these are the reasons:

  • Sublime Text 3 is sooo pretty
  • it’s also so freaking fast, way faster than Sublime Text 2
  • and has vast range of plugins, and easy configuration.

Not much, you say? It was enough to made me switch editors.

Brief history

Sublime Text 2 was using Python 2.x as an API interpreter. Early in 2013, first build of Sublime Text 3 appeared, not even remotely complete. And it shipped with Python 3.3 interpreter.

Since then, Sublime Text 3 (now beta build 3047 or bleeding edge dev build 3052) has come a long way and can be used by everyday programmer.

What’s important is that many Sublime Text 2 plugins have been either ported or made Python2/3 compatible.

Configuration

Theme

There’s lots and lots of themes for Sublime Text 3. I currently use Monokai Extended, with Soda dark theme.

Perv Orange is also very good, but a bit darker than Monokai.

Both schemes have better restructuredText coloring. Default Sublime Text 3 syntax highlighting for rsT is… in a poor state.

Soda theme with Monokai Extended color scheme Visual preview of my current configuration. (click to enlarge)

Some visual settings:

{
    "always_show_minimap_viewport": true,
    "bold_folder_labels": true,
    "draw_minimap_border": true,
    "highlight_modified_tabs": true,
    "indent_guide_options":
    [
        "draw_active",
        "draw_normal"
    ],
    "soda_classic_tabs": true,
    "soda_folder_icons": false,
}

Typeface

I’ve tested Consolas, Monaco, Ubuntu Mono and Meslo. Last two are my favorite, I switch them back and forth very often.

Configuration for Meslo:

{
    "font_face": "Meslo LG L",
    "font_size": 10,
    "line_padding_bottom": 0,
    "line_padding_top": 0,
}

Configuration for Ubuntu Mono:

{
    "font_face": "Ubuntu Mono",
    "font_size": 12,
    "line_padding_bottom": 1,
    "line_padding_top": 1,
}

And some additional typeface-agnostic but font-related settings:

{
    "caret_style": "solid",
    "font_options":
    [
        "subpixel_antialias"
    ],
    "highlight_line": true,
}

Python

This part of configuration helps with writing Python.

{
    "ensure_newline_at_eof_on_save": true,
    "folder_exclude_patterns":
    [
        ".svn",
        ".git",
        ".hg",
        "CVS",
        "__pycache__"
    ],
    "indent_to_bracket": true,
    "rulers":
    [
        79
    ],
    "shift_tab_unindent": true,
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true,
    "wrap_width": 80
}

Behavior

Fixes some minor annoyings, like opening a preview after selecting file in side bar.

{
    "enable_telemetry": false,
    "preview_on_click": false,
    "shift_tab_unindent": true,
    "show_panel_on_build": false,
}

You can also change some settings on a per-project basis. Simply open your project.sublime-project file (Project → Edit Project) and add settings section:

{
    "folders":
    [
        ...
    ],
    "settings":
    {
        "python_interpreter": "/home/piotr/.virtualenvs/project/bin/python"
    }
}

Configuration file

For lazy people: https://gist.github.com/pbanaszkiewicz/6351258.

Plugins

These are the plugins I can’t live (and program) without. Absolutely necessary.

Package Control

A package manager for Sublime Text 3. Helps you search for, install, upgrade and remove most of available Sublime Text 3 packages.

Git

Very popular and quite easy for basic application. There’s also some payable SublimeGit plugin I haven’t tried out yet. (It seems nice, though!)

Update 30.08.2013: I switched to SublimeGit plugin. It’s very similar to vim’s fugitive, so I’m already feeling good about it. Definitely well spent money on an alternative to Git.

GitGutter

Adds small icons in left margin indicating which lines have been added, modified or deleted.

GitGutter icons example GitGutter icons: for deleted, added and modified lines. (click to enlarge)

SideBarEnhancements

Adds obvious (but missing from pure Sublime Text 3!) context menu options for side bar.

Additional entries in sidebar's context menu. Additional items in sidebar’s context. (click to enlarge)

Anaconda

So far, the best Python completion and linter for Sublime Text 3. You can believe me, I tested other ones, too.

Anaconda is actively developed, and its’ author is very responsive.

I highly recommend installing this package from git, as I’m not sure if recent critical patches were already pulled in by Package Control.

Warning: do not mistake with Continuum.io’s excellent Anaconda suite for Python.

Gist

This plugins helps managing (adding, editing and removing) GitHub gists. Needs a little bit of configuration efforts, but it’s generally worthwhile.

Warning: After installing each plugin, check if your configuration and favorite key bindings still work. It will be harmful for you to discover after pulling in four plugins that one of them disabled selecting multiple cosecutive lines separately (Shift+Alt+Up/Down) or inserting multiple carets in EOLs in your selection (Ctrl+Shift+L). I even recall some plugin changing behavior of inserting brackets around selection! That was just mean…

Final words

I hope I was somehow able to help you boost your Python development or encourage to use Sublime Text 3. Have a good time and nothing to debug!

Two years later update

It’s December 2015, so over 2 years have passed since this post was originally published. What changed in Sublime Text 3? Am I still using it?

The answer is: yes! Even though the development of Sublime Text 3 is stalled (if only it was open source) I’m still using it. It’s a great product that “just works”.

Here are changes in my configuration:

  • I switched my theme to Solarized Light, because I’m programming a lot in a daylight and I need a good, light background.
  • I’m using a new font: Hack. It works great, but as I mentioned in the original article, I like to switch fonts now and then. I’ll probably switch to something else really soon.
  • Now I also use a bigger font size, and it works better for my eyes.

Screenshot of some code. Solarized Light, Hack font face, and some of AMY’s code. (click to enlarge)

Here is a list of plugins that I got rid of:

  • Gist: I’m not sharing so much code to be in need of plugin for that service.

New plugins:

  • SublimeGit: it was mentioned in update to the original article, but this plugin works excellent. It may not work great during rebases (doesn’t show conflicting files), but I’d not trust any tool except git in such time.
  • SublimeLinter: excellent for checking my Python and JavaScript syntax correctness, and standards (like PEP8) compliance.

I’m really interested to see how my usage of Sublime Text 3 develops in future.