Meet Marvin, our Twitter meta-bot
August 26th, 2010 by gerIt all started with a small Twitter project for a client. Then we began thinking about different ideas for twitter bots (not many of them became an actual product but we had a lot of ideas at that time).
It didn’t take too much time to realize that implementing each bot from scratch didn’t make sense. Specially when changes from Twitter’s side have an important impact on each developed bot (like deprecating basic authentication in favor of OAuth).
So we decided to create Marvin: our own meta-bot in charge of handling everything except creating the tweet itself.
Design and Implementation
The idea was to simplify the process of creating a new Twitter bot. Marvin will handle all the repetitive tasks and let each bot concentrate on doing what needs to be done and nothing else.
What does Marvin offer to us:
- OAuth authentication
- Logging
- Data persistence
- Debugging
- More to come…
We implemented Marvin using oauth-python-twitter which provides us with a simple API to communicate with Twitter. Instead of wrapping this API within Marvin, we let each bot use it directly since that gives us a lot of flexibility and it’s very simple to use. We might review this in the future to allow a more powerful debug mechanism.
A bot is implemented as a simple python module which must define the consumer key and consumer secret for the application (defined in Twitter) and implement a run method that will be executed by Marvin:
# Twitter application configuration
# These are fake keys!
CONSUMER_KEY = "j28Rup5fr4thUwruXAP2f"
CONSUMER_SECRET = "j28Rup5fr4thUwruXAP2fj28Rup5fr4thUwruXAP2f"
from datetime import datetime
def run(twitter, data, log):
# Create your tweet, probably getting information
# from some external source
count = data.get("count", 1)
message = "Tweet number %d: Marvin has something to tell you..." % count
# Post the message using oauth-python-twitter API
log.info("Tweetting: " + message)
twitter.PostUpdate(message)
# Save the count number for the next execution
data["count"] = count + 1
As you can see, implementing a bot is very simple since we only have to worry about the tweet and nothing else!
One of the features implemented by Marvin is data persistence. Normally, a bot will have to save information from one execution to the next and we need to provide a way of doing that easily. In the previous example we stored the count number just to show you how it’s done.
To solve that problem we use pickle, a Python module for serializing and deserializing objects. As expected, Marvin will handle saving and restoring the data for you. You just need to store what you need there and read it in the next executing…
Running Marvin
Once Marvin has successfully logged in to an account (for a particular bot instance) we tell him to execute the actual bot process using the following command:
ger@piazzolla:~/projects/marvin$ python marvin.py run bot
In this example, our bot is implemented in bot.py module.
To execute the bot periodically we decided to use cron, which led us to the next section…
Why cron?
We thought about using an in-process task scheduler for Python, like APScheduler but cron’s simplicity and stability was a really apealing reason for us.
With cron running Marvin is as simple as adding this line to the cron’s table:
*/1 * * * * cd $MARVIN_HOME; python marvin.py run bot
In this example Marvin will execute the bot every single minute (don’t worry, we won’t do that!)
We haven’t dismissed APScheduler for future versions, but for now we are very happy with the results we get from using cron.
How to handle OAuth authentication with a command line bot?
Using OAuth from command line is not as simple as it seams. From a web application you are normally redirected to Twitter were you authorize it to access your Twitter account. Everything is simple and with a couple of clicks you are ready to go. But we don’t have a web application or a web browser to be redirected!
The solution for us was to use PIN-based OAuth mechanism created by Twitter. We just tell Marvin that we want to login using the following command:
ger@piazzolla:~/projects/marvin$ python marvin.py login bot Authorization URL: http://twitter.com/oauth/authorize?&oauth_consumer_key=FjZYXIe7z8X6rRHvXjGzrA&oauth_signature_method=HMAC-SHA1 Enter OAuth PIN:
Using this approach we need to open the authorization URL, allow the application to access Twitter (defined by CONSUMER_KEY and CONSUMER_SECRET) and enter the given PIN as shown below:
Enter OAuth PIN: 9823410 Marvin logged in successfully. You can now start using your bot!
Marvin will use this PIN to retrieve the access token from Twitter and will save it for us. Using this access token we will be able to access the authorized Twitter account transparently.
Is this all?
Definitely not. We are planning to add more features to Marvin, like autofollow based on different queries, automated DM responses and much more. You’ll be hearing more about this once we implemented these new features.
Also, based on the reaction we get from this post, we might consider publishing Marvin as the first devartis’ open-source project.
We now invite you all to follow @futbolar, our first bot implemented using Marvin.
