From concept to completion in four hours

The concept: “Hey, I wonder what everyone on twitter is wishing for,” around 5pm yesterday as my mind began to wander from the walloping my Dolphins recieved.

The completed project: Right here., committed at 9.27pm. The project is running on Flask on help from jQuery, hosted by Heroku.

Now, at first glance — and second glance — and any glance thereafter — this isn’t a particularly complex or difficult project. I agree, and that was by design — I wanted this to be something I could move from start to finish, as quickly as possible. Still, I’m very happy with it; not only was I much more productive than I usually am on a Sunday evening (over Winter Break, no less) but I got to learn a lot of things that I had never really touched before:

Twitter’s GET API
Masonry.js
Infinite scrolling
Heroku’s database infrastructure

Below is my rough catalog of my workflow and the various hiccups I encountered. This’ll probably be a longer post, and if you haven’t I recommend checking out the product itself just so you get a better feel for where I was coming from.

First steps: the Twitter API

I knew the core of this thing was going to be the tweets themselves, so I decided to tackle that first.

I hadn’t used the Twitter API before, but I had dealt with more than enough JSON APIs and GET methods, so I wasn’t entirely too concerned. I’d heard a lot about the apparent chaos of Twitter’s API, but for the simple nature of this project all I needed was the search method, which queries the Twitter corpus and returns tweets which hold a given string.

With requests: this is a breeze:

payload = {'q': "i wish",
           'rpp': 100,
           'page': page}
tweets_url = 'http://search.twitter.com/search.json

request = requests.get(tweets_url, params=payload)
tweets_dict = request.json['results']

Tada! We have a tuple of tweets. I knew, at minimum, I’d need the text of the tweet and it’s permalink. The first one, was easy:

text = tweet[‘text’]

But the permalink was nowhere to be found in the JSON. Luckily, I was able to reconstruct it from spare parts, after noticing that tweets have a specific URI pattern:

permalink = “http://www.twitter.com/“ + tweet[‘from_user’] + “/statuses/“ + tweet[‘id’]

After iterating over a few of these to make sure there weren’t any weird edge cases, I began running a ten-minute cronjob and storing them in an sqlite database using Peewee, an excellent Python ORM.

Flask and the easy stuff

Seriously, this was the easy part for me. I had a data source and I wanted to display it as a list and make it look fancy. I decided to go with Flask, as I was comfortable with it, knew how to easily hook it into Peewee, and felt that it’d handle pagination well down the line. The soul of the Flask code is this:

@app.route('/<page>')
def page(page):
    if page == 'favicon.ico':
        return ''

    page = int(page)
    tweets = get_tweets(page)
    return render_template('index.html', tweets = tweets, index = page)

(The root returns page(0), and get_tweets returns a set of Tweet objects by querying the sqlite db.)

The template was similarly lightweight; Jinja2 (Flask’s templating framework) has builtin for loops, so I basically made a massive unordered list and created a new li for each tweet. Applied some basic css (list-style-type:none; float: left to make the lists play nice with each other) and suddenly I had like 95% of my little idea working!

The problem? It looked hideous.

I used Google Webfonts and Subtle Patterns to spruce it up a bit — both are great resources for developers on a budget — but it was still looking ratchety. I was starting to get a little despondent (I’ve got low standards when it comes to MVP design, but this thing was an eyesore even to me!) when I realized this gave me a great reason to bust a new toy out of its plastic: Masonry.js

Masonry.js

You’ve probably seen Masonry dozens of times by now. It’s a simple little jQuery package that organizes things into columns in a very pretty manner. It got uber-popularized by Pinterest, but I had actually seen it before then, on a variety of tumblr themes. I thought it was crazy cool, but didn’t really have any solid reason to use it.

The tutorial it gives you is really all it takes to get the thing up and running. I added a few configuration tweaks to make it play a bit nicer with my use case:

I liked the idea of a fluid design — I wanted five columns, no matter how big the canvas. so I changed the columnWidth option from:

columnWidth: 240

columnWidth: function( containerWidth) {
    return containerWidth / 5;
    }

A more serious issue I ran into was that Masonry was organizing the elements as soon as the tweets were fetched but before the webfont (in this case, Open Sans) was fetched and applied, giving me some gross overlaps. This, too, was a simple fix: instead of invoking the main Masonry function on $(document).ready, invoking it on $(window).loadmade everything peachy.

Infinite scrolling and pagination

So Masonry was working great! I was an hour or so in and had made much more progress than I was expecting. Still, 100 tweets spread over five columns in relatively dinky text does not take up much space: I knew I wanted to have infinite scrolling (again, you’ve probably seen it on Pinterest; you scroll to the end of the page, it automatically loads the next page) but had no idea how.

Masonry has a pretty hands-off infinite scrolling plugin as well. The frontend aspect of it was a pretty easy copy and paste job: since I handled pagination with the Flask (as shown in page(page)), all I had to do was throw in an invisible navigation element. And by passing the current page to the Jinja2 template as index, getting the link to the next page was a snap:

<nav id=‘nextpage’>
    <a href=‘/{{ index + 1 }}’></a>
</nav>

At this point, I was ecstatic. I had it running on localhost perfectly, and I was ready to deploy to Heroku.

Deploying

An abridged version of my troubles:

Heroku doesn’t support sqlite3.
Peewee only barely supports postgres, and requires psycopg2.
My MacBook Air couldn’t handle psycopg2 for some reason.
Heroku supports PyMongo! My MacBook supports PyMongo!
I can’t connect to Heroku’s MongoDB add-on for some reason!
Fuck it, let’s ditch the database and live query the JSON.

So I ended up ditching the entire ORM structure and refactoring: instead of querying a database, get_tweets() now queried Twitter itself by borrowing the original code I used to populate the database.

This was by far the most frustrating and time-consuming part of my four hours, and I feel guilty because my solution is so blatantly wrong: Twitter has a pretty harsh rate limit for GET requests, and if this were anything more serious than an evening project than I would have worked on getting a nicer solution since hundreds of users making many, many redundant requests would have imploded pretty quickly.

Miscellany

Data troubles behind me, I had finally gotten a successful deployment. (As a side note: I love heroku app names. “Tranquil-tundra”? How feng shui.)

There were a few things left to take care of:

I wanted to throw in social media icons, if for no other reason than to throw in social media icons. The creation of these things is clearly optimized to be used by non-developers (seriously, having a page tell me to ‘copy and paste this code’ gives me Vietnam flashbacks to applying CSS on my MySpace page), but it was by no means difficult.
A name/domain. My original choice was to call it Velle, which is French for ‘wish’: I ended up deciding against this, as it was unnecessarily pretentious — only after I had purchased ‘velle.cc’ for 18.99. I decided this is something I might as well put as a jmduke subdomain and left it at that.
Further progress; I thought of maybe having a Like-esque counter for each tweet; maybe having a ranking system; maybe having a bunch of things. Ultimately I decided I liked the simplicity of it as it stood, and left it as is.

And that’s it! A productive four hours: I learned a lot and made something to show for it. I couldn’t be happier.

What I should have done

Javascript backend. In retrospect, this would’ve been a project tailor-made for Backbone or even raw javascript/jquery — once I decided to ditch the ORM, there was no real reason to stick around with Flask beyond not wanting to double back and rewrite code. While Heroku makes Flask deployment crazy easy, it’s still not as easy as just uploading a page.
Took screenshots during development. I was wise enough to scribble down notes as I was working on the project, but nothing stirs my memory better than a screenshot of a pesky bug (such as the Masonry issues due to webfonts).

Thanks for reading and I hope you learned something! If you have any questions about any aspect of the project — or want to take a look at the finished source — shoot me an email. I want to throw it on GitHub at some point — once I do, I’ll update the post. You should probably follow me on Twitter as well.

Justin Duke