Selenium Webdriver and Jenkins

This post has been hold for quite long time because several reasons: there are bunch of articles covering this setup, there are a lot of questions posted in Q&A site on the similar topic. But then I decided to post it with hope it will help other who face similar problem with selenium web driver.

Version

Firefox version versus webdriver version (selenium-standalone-server.jar) is somehow overlapping each other in a way that firefox version increases every now and then and most of the time our firefox automatic update is enabled, while even the newest web driver does not support the newest firefox. So, just take this recipe: Selenium server 2.31 only support firefox up to  version 17.0. I don’t really know what could it be with the older version of both, most likely it will work.

Jenkins Master Setup

Normally I have several jenkins’ executor (slave/node) to run specific task. Especially for web automation test, I use one dedicated node just to run the tied jenkins’ job only. Following is the setup for the slave machine:

OS      : Ubuntu 12.04.1 LTS
X       : Xvfb
Firefox : Firefox 16.0
Java    : 1.7.0_15
          OpenJDK Runtime Environment (IcedTea7 2.3.7) (7u15-2.3.7-0ubuntu1~12.04.1)
          OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

Here is the Jenkins’ slave setup:

jenkins-slave

It’s also important to take note on the slave launch method, after some consideration and learning from experiences, I choose to launch slave agent via SSH. I used to use root account for this, but I found out somehow firefox has an issue whenever executed by root. I don’t think that this is known problem, so I choose to use non-root user for this purpose (jenkins).

launch-slave

Jenkins Slave Setup

On the slave you need to have `jenkins` user with /var/lib/jenkins as its HOME directory.

# adduser --home /var/lib/jenkins --disabled-password jenkins
# chown -R jenkins:jenkins /var/lib/jenkins

SSH key pair also needed and make sure that master’s public key is put inside slave’s authorized_keys. Also you need to install ssh-askpass

# ssh-keygen
# cat /path/to/master_public_key >> /var/lib/jenkins/.ssh/authorized_keys
# sudo apt-get install ssh-askpass

As you may see, DISPLAY is set to :1, this is related with the display used by Xvfb. This is how Xvfb is started:

/usr/bin/Xvfb :1 -screen 0 1024x768x24

There is also a little note that is important if you use ThoughtWorks’s Twist, you will need to define the hostname in the host file.

# /etc/hosts
127.0.0.1       localhost qa-auto-1 qa-auto-1.localfoo.com

From my trials, I found some keys to make the testing setup stable enough for the long run:

  1. Make sure firefox version is not updated unless selenium server is updated, and make sure the version mix-match for compatibility. For my own setup, I stick with Firefox 16.0 and Selenium Server 2.31
  2. Do not execute testing job with root account. There is either known or unknown problem when firefox is executed with root.
  3. In particular if you use Twist, you will need to define your hostname to your host file.
  4. Make sure X is always available, I put a cron job to monitor whether X is running or not, then do the task respectively.

Vim – entering special character

It’s been a question for me for years, but usually I ignore it. Sometimes, I want to put a special character on my code, for example; a copyright symbol (©) or registered sign (®) or a latin character, or other. Actually it is quiet simple to do using Vim.

First thing to know is RFC1345: Character Mnemonics and Character Sets. You need to know the character mnemonic, then how to put it on your code. Vim provides :digraph. Simply open Vim and type :digraph, you should find a table of special characters which are available in Vim. There are several way of using :digraph, in INSERT mode you can use CTRL+K followed by _two letter combination_, or :set digraph (:set dg), or by entering character value after pressing CTRL+V.

Consider this image:

Vim :digraph

There are three columns described as follow:

1. magenta block (#1): SH, D1, Nb, NH, TS ….

2. green block (#2): ^A, ^Q, #, , , …

3. orange block (#3): 1, 17, 35, 131, 147 …

The first column (#1) represent two letter combination to show the special character in digraph after pressing CTRL+K

The second column (#2) is the special character you want to type

The third columnd (#3) is the character’s decimal value based on RFC1345

See what are the value you need related with REGISTERED SIGN from :digraph extract

Rg  ® 174

So, you can proceed by:

1. CTRL+K Rg

2. CTRL+V 174

3. R <BS> g

For option #3, this can only work when you :set digraph (or :set dg)

<BS> is CTRL+H in Vim

For complete reference, you can see Vim: digraph documentation

NgenTwit – Tweet like no one follows

Well, the “description” part of the title is not a feature at all of a complete application. That’s merely my lack of knowledge in Python, and definitely I don’t have enough time to learn Python. As an old ode says “there isn’t `too late to learn`” and I do believe in my personal inner intelligent and spirit for the sake of the harmony of this universe … I started to learn Python.

As my usual learning scheme, I can’t start if there is no challenge. So, I did challenge myself to write a new twitter desktop application. That was the birth of NgenTwit (credit to http://twitter.com/3072 for the name). The first thing came into my mind is, this application must have a nice GUI. Then, it has the ability to do automatic tweet. For the former spec, choice is PyGTK.

I don’t want to have a big confusion in the beginning, so I look for Python twitter framework, and I choose Tweepy. From it’s documentation, it is clear, clean, and easy to use. Then I start to write.

The first thing to do is to make sure that Tweepy is correctly installed, please refer to Tweepy documentation. You will need Tweepy-example package too, to test Tweepy installation.

Next is twitter related administration. Twitter provide two types of authentication; Basic auth and OAuth. Tweepy provides a very cool API to use respectively. For my case, I will use OAuth. It allows me to tell me follower from where I send my tweet.

Assume that I have an application registered to twitter, I will have consumer key and consumer secret as follow:

consumer_key = "xxxxxxxx"

consumer_secret = "yyyyyyyy"

Each user need access token and access token key. This is persistent once you are authenticated. We can get them by using one of tweepy example: tweepy-examples/oauth/getaccesstoken.py

So now I have four variables needed to access Twitter API; consumer_key, consumer_secret, access_token_key, access_token_secret.

First, I will need Twitter to authenticate me.

 auth = tweepy.OAuthHandler("xxxxxxxx", "yyyyyyyy")

as we have access_token_key and access_token_secret, these variables are set to Tweepy as follow:
assumes their values are “aaaaaaaa” and “bbbbbbb” respectively.

auth.set_access_token("aaaaaaaa", "bbbbbbb")

then initialize Tweepy API with this authenticated access token.

api = tweepy.API(auth)

Now I can do anything I want thanks to tweepy.API.

Now, done with twitter stuffs, I am ready to write the GUI. I am using PyGTK. This is not a rich GUI, it is merely intended to catch my tweet and send it to Twitter once I press return. No way to see the timeline, replies, friends, etc. So, basically only two things needed.
1. User input
2. update_status of Tweepy API

The result is as simply as follow:

Here is the code:

#!/usr/bin/env python
# ngentwit.py

import pygtk
pygtk.require('2.0')
import gtk
import tweepy

class Ngentwit:
    def enter_callback(self, widget, entry):
        entry_text = entry.get_text()
        entry.set_text("Sending to twitter...")
        auth = tweepy.OAuthHandler("xxxxxxxx", "yyyyyyyy")
        auth.set_access_token("aaaaaaaa", "bbbbbbb")
        api = tweepy.API(auth)
        api.update_status(entry_text)
        entry.set_text("")

    def __init__(self):  # create a new window
        window = gtk.Window(gtk.WINDOW_TOPLEVEL)
        window.set_size_request(300, 50)
        window.set_title("NgenTwit")
        window.connect("delete_event", lambda w,e: gtk.main_quit())
        vbox = gtk.VBox(False, 0)  window.add(vbox)
        frame = gtk.Frame("Tweet? No one follows")  

        entry = gtk.Entry()
        entry.set_max_length(140)
        entry.connect("activate", self.enter_callback, entry)
        entry.select_region(0, len(entry.get_text()))  

        frame.add(entry)
        vbox.pack_start(frame, False, False, 0)
        vbox.show()
        window.show_all()

def main():
        gtk.main()
        return 0

if __name__ == "__main__":
        Ngentwit()
        main()

PECL and twitter oauth

Currently, I am trying to make batbet to be able to communicate with twitter API. Twitter uses oauth as authentication method. The first thing comes in mind is to use liboauth for C (available here). In fact I am not so familiar and I am studying how twitter’s oauth works. I got clue from PECL sample (available here).

So, the first thing to do is how to make PECL works then port it to C. Why C? because this is the only language I know 😛

How to install PECL oauth extension?

$ pecl search oauth

Retrieving data…0%Matched packages, channel pecl.php.net:
=======================================
Package Stable/(Latest) Local
oauth   0.99.9 (beta)   0.99.9 oauth consumer extension

then

$ sudo pecl install oauth-0.99.9

Now PECL oauth should be installed to your machine.

or you can also build manually the extension using phpize.

download oauth-0.99.9, extract then go inside oauth directory, invoke phpize

oauth-0.99.9 $ phpize

oauth-0.99.9 $ ./configure; make

oauth-0.99.9 $ sudo make install

To make your new extension works you should adjust your php.ini. On ubuntu there is /etc/php5/cli/php.ini then add following:

extension=oauth.so

Once you’re done, go to twitter example of oauth PECL package.

$ php updateStatus.php

you’ll see …

I was wondering how does the twitter authentication method go, from the beginning to the end. I might be wrong but it works for me, following is the workflow:

1. After you get consumer key and consumer secret, you need to request_token(). For instance, using PECL liboauth:

$request_token_info = $oauth->getRequestToken(https://twitter.com/oauth/request_token);

then you will get “oauth_token” reply.

2. You need to ask authorization from user, by giving a link to:

https://twitter.com/oauth/authorize?oauth_token=oauth_token -> you get this on the step #1

User will have to allow the access.

3. Right after authorization is granted by user, you need to request a new oauth token by accessing:

https://twitter/oauth/access_token

You will get oauth_token and oauth_token_secret to be used to call twitter REST API. This oauth_token and oauth_token_secret is persistent. I save these variables to a plaintext to be used as future request, so user does not need to do step #2 anymore.

4. Use Twitter REST API to do anything you want.

NOTE: I am open to any suggestion.