Tweeted Links: An introduction, and a review

If you follow me on Twitter you have most likely seen my rants about my new service, Tweeted Links. The method behind the madness of it is basically this: I post a lot of links to Twitter, and often want to revisit them at a later time. I usually don’t Tweet anything meaningful with them that will help searching (e.g. KK) and usually shorten the URLs. Tweeted Links solves that problem! It keeps the links in a single location (a la del.icio.us) and fetches the title of the page. It also solves the problem of shortened URLs by attempting to expand those too.

Another big point that I rather liked about it, was that from design, through development to deployment it took about 6 hours to get the first release out! I fixed a few bugs a couple of days later, so there has altogether been about 8 hours work put into it. Not too shabby, for something useful! I decided to use something new to build it, so used the Monk ‘glue’ framework which by default combines Sinatra, Ohm and Redis. It is a pretty nice system, however like most Ruby software the default skeletons (and at least the ones mentioned on the site) are rather opinionated. I myself prefer Bundler however none of them had this for dependency management. I wouldn’t have thought it would be terribly difficult to have a nice system that lets you add or remove different gems from the skeleton so you can customise it to your liking, that would be a nice feature for a future revision of the website. I then rounded it up with running on Ruby 1.9.2 and Unicorn.

In terms of actually using the tools, I have used Sinatra before and played around with Redis and Ohm a bit, however for those I mainly relearnt everything I (thought I) knew. This was the first time I had properly used a key-value storage engine, so something things were a bit strange. As an example I wanted to ‘expire’ old users after 6 hours so that their statuses would be refetched. In SQL this is dead simple, however how to replicate that in Redis? In the end I ended up fetching all the records and comparing the created_at field in code, I have doubts this is the optimum solution however….

I also had a few teething problems that went unnoticed (for about 15 days >_<) which I recently resolved. I’m not sure what happened however the expiry daemon just stopped working, even restarting it didn’t help. In the end I restarted Redis which sorted it, so I’m not sure what was to blame there. JNo was the last user to be added to the system, so I blame him. Everything seems to be working again now, at least for a while….

Another major problem (which I still haven’t resolved) is an issue with the Twitter search. Certain users just return no search results. The first user I found with this was the BBC News Twitter account, it had over 70,000 tweets so I thought maybe the search just didn’t index large accounts. However I then found a few more accounts that only had a couple of hundred tweets. I still haven’t got this resolved, mainly because I have yet to ask for help from the gods of Twitter 😛

So that about sums up Tweeted Links. I’m quite happy with the results so far, and am looking to expand it a bit further when I get time (I’m currently focussing on some Android work, which I’m sure there will be a few posts on over the next couple of weeks). I expect I will probably rewrite the system from scratch though, as to be honest, I’m not really too happy about how this key-value stuff has worked out so far. The rewrites will also involve some big data restructuring, so I don’t think it is a bad time to do so. Back to MySQL I say, yep I just said no to NoSQL!