Bootstrap Templer

Create buildout directory for Templer installation:

$ mkdir templer-buildout
$ cd templer-buildout/

Get bootstrap.py:

$ curl -o http://downloads.buildout.org/2/bootstrap.py

Create buildout.cfg:

[buildout]
parts = templer

[templer]
recipe = zc.recipe.egg
eggs =
    templer.core
    templer.plone

Run bootstrap and buildout to install Templer:

$ python bootstrap.py
$ bin/buildout

Create a new product with Templer

Call the buildout-installed to create a new product with Robot Framework test example:

$ templer-buildout/bin/templer plone_basic example.product

Be careful to answer true for the following question about including Robot test templates:

robot tests (should the default robot test be included) [false]: true

Run buildout:

$ cd example.product
$ python bootstrap.py --distribute
$ bin/buildout

Update Robot Framework tests to be Selenium grid ready

Using Sauce Labs with Robot Framework (Selenium library) is similar to using robot with your own selenium grid. It mainly requires making the browser opening keyword configurable with a few selected variables.

Update src/example/product/tests/robot_test.txt with:

*** Settings ***

Library  Selenium2Library  timeout=10  implicit_wait=0.5

Suite Setup  Start browser
Suite Teardown  Close All Browsers

*** Variables ***

${ZOPE_HOST}  localhost
${ZOPE_PORT}  55001
${ZOPE_URL}  http://${ZOPE_HOST}:${ZOPE_PORT}

${PLONE_SITE_ID}  plone
${PLONE_URL}  ${ZOPE_URL}/${PLONE_SITE_ID}

${BROWSER}  Firefox
${REMOTE_URL}
${DESIRED_CAPABILITIES}  platform:Linux
${BUILD_NUMBER}  manual

*** Test Cases ***

Plone site
    [Tags]  start
    Go to  ${PLONE_URL}
    Page should contain  Plone site

*** Keywords ***

Start browser
    ${BUILD_INFO} =  Set variable
    ...           build:${BUILD_NUMBER},name:${SUITE_NAME} | ${TEST_NAME}
    Open browser  ${PLONE_URL}  ${BROWSER}
    ...           remote_url=${REMOTE_URL}
    ...           desired_capabilities=${DESIRED_CAPABILITIES},${BUILD_INFO}

Let me explain what all those variables are about:

• ZOPE_HOST should match the host for which ZServer is started during the test setup (ZServer host is configured with ZSERVER_HOST-environment variable. It defaults to localhost.
• ZOPE_PORT should match the port number which ZServer is started to listen during the test setup (ZServer pot is configured with ZSERVER_PORT-environment variable. It defaults to 55001, but we reconfigure it later by environment variables with one of the ports currently supported by Sauce Labs.
• ZOPE_URL is a convenience variable for accessing Zope application root.
• PLONE_SITE_ID is the Plone portal object id (and path name) for the test site. It default to plone, but it can be configured with PLONE_SITE_ID-environment variable. The default should be ok for most cases.
• PLONE_URL is a convenience variable for accessing the Plone site front-page.
• BROWSER selects the browser to run the tests with. The supported values depend on Selenium Python-package and can also be read from the documentation of Open browser-keyword in Selenium2Library keywordsdocumentation.
• REMOTE_URL enables testing with Selenium grid by defining the url of the Selenium hub to use.
• DESIRED_CAPABILITIES is used to pass various extra parameters for Selenium hub (e.g. the browser version to use or test metadata).
• BUILD_NUMBER is used to identify the Travis-CI build on Sauce Labs.

When robot tests for Plone are run using bin/test, all the variables above can be overridden by defining corresponding ROBOT_-prefixed environment variable (e.g. ROBOT_REMOTE_URL).

Add Travis-CI configuration with Sauce Labs -support

There are a few steps in adding Travis-CI-support into your product.

At first, create travis.cfg to do the required magic for minimizing buildout-time and setting a few required environment variables. Thanks to the great community, we can just extend a public template:

[buildout]
extends =
    https://raw.github.com/collective/buildout.plonetest/master/travis-4.x.cfg

package-name = example.product
package-extras = [test]

allow-hosts +=
    code.google.com
    robotframework.googlecode.com

[versions]
plone.app.testing = 4.2.2

[environment]
ZSERVER_PORT= 8080
ROBOT_ZOPE_PORT= 8080

[test]
environment = environment

Create .travis.yml for letting Travis-CI to know how the environment should be set up and the tests run:

---
language: python
python: '2.7'
install:
- mkdir -p buildout-cache/downloads
- python bootstrap.py -c travis.cfg
- bin/buildout -N -t 3 -c travis.cfg
- curl -O http://saucelabs.com/downloads/Sauce-Connect-latest.zip
- unzip Sauce-Connect-latest.zip
- java -jar Sauce-Connect.jar $SAUCE_USERNAME $SAUCE_ACCESS_KEY -i $TRAVIS_JOB_ID
  -f CONNECTED &
- JAVA_PID=$!
before_script:
- bash -c "while [ ! -f CONNECTED ]; do sleep 2; done"
script: bin/test
after_script:
- kill $JAVA_PID
env:
  global:
  - ROBOT_BUILD_NUMBER=travis-$TRAVIS_BUILD_NUMBER
  - ROBOT_REMOTE_URL=http://$SAUCE_USERNAME:$SAUCE_ACCESS_KEY@ondemand.saucelabs.com:80/wd/hub
  matrix:
  - ROBOT_BROWSER=firefox ROBOT_DESIRED_CAPABILITIES=tunnel-identifier:$TRAVIS_JOB_ID
  - ROBOT_BROWSER=ie ROBOT_DESIRED_CAPABILITIES=tunnel-identifier:$TRAVIS_JOB_ID
  - ROBOT_DESIRED_CAPABILITIES="platform:OS X 10.8,browserName:iPad,version:6,tunnel-identifier:$TRAVIS_JOB_ID"

Let me describe:

1. Lines 4-7: Run buildout.
2. Lines 8-14: Download and start Sauce Connect.
3. Line 15: Run tests.
4. Lines 16-17: Shutdown Sauce Connect.
5. Lines 18-21: Define required environment variables for letting robot to use Sauce Labs.
6. Lines 22-25: Define build matrix for running the tests with Sauce Labs' default Firefox, Internet Explorer and Mobile Safari. tunnel-identifier-stuff is required for Sauce Labs to allow more than one simultaneous tunnels for the same user account.

Next, define your Sauce Labs username and access key as secret, encrypted, environment variables SAUCE_USERNAME and SAUCE_ACCESS_KEY.

Currently, Sauce Labs offers unlimited free subscription with three simultaneous connections (e.g. running tests for three different browsers at the same time) for Open Source projects. Just make sure to register the account for your project, not yourself. Public repository url is required for the creating the account and it cannot be changed afterwards.

1. Install Travis gem for Ruby (and install Ruby before that when required):

   
   $ gem install travis  # or sudo gem ...
   

2. use travis-command to insert encrypted environment variables into the product's .travis.yml:

   
   $ travis encrypt SAUCE_USERNAME=myusername -r mygithubname/example.product --add env.global
   $ travis encrypt SAUCE_ACCESS_KEY=myaccesskey -r mygithubname/example.product --add env.global
   

Make sure to use your own Sauce Labs username and access key, and your product's Github-repository path (with format username/repo).

Finally, enable Travis-CI-tests for you product either at Travis-CI.org or at GitHub.

Done. If I forgot something, I'll update this post.

Behind the basics: Test level status reporting for Sauce Labs

By default, Sauce Labs doesn't really know did the Selenium tests on it pass or fail. To pass that information from our test runner on Travis-CI to Sauce Labs, we need to add some extra code into our test setup.

At first, append the following into the end of src/example/product/testing.py:

import re
import os
import httplib
import base64
try:
    import json
    assert json  # pyflakes
except ImportError:
    import simplejson as json

from robot.libraries.BuiltIn import BuiltIn

USERNAME_ACCESS_KEY = re.compile('^(http|https):\/\/([^:]+):([^@]+)@')


class Keywords:
    def report_sauce_status(self, status, tags=[], remote_url=''):
        job_id = BuiltIn().get_library_instance(
            'Selenium2Library')._current_browser().session_id

        if USERNAME_ACCESS_KEY.match(remote_url):
            username, access_key =\
                USERNAME_ACCESS_KEY.findall(remote_url)[0][1:]
        else:
            username = os.environ.get('SAUCE_USERNAME')
            access_key = os.environ.get('SAUCE_ACCESS_KEY')

        if not job_id:
            return u"No Sauce job id found. Skipping..."
        elif not username or not access_key:
            return u"No Sauce environment variables found. Skipping..."

        token = base64.encodestring('%s:%s' % (username, access_key))[:-1]
        body = json.dumps({'passed': status == 'PASS',
                           'tags': tags})

        connection = httplib.HTTPConnection('saucelabs.com')
        connection.request('PUT', '/rest/v1/%s/jobs/%s' % (
            username, job_id), body,
            headers={'Authorization': 'Basic %s' % token}
        )
        return connection.getresponse().status

This code defines a custom Robot Framework keyword library with a keyword for passing the test status (and other information) back to Sauce Labs.

Next, we update src/example/product/tests/robot_test.txt to store the session id during the setup of every test and send the test result back to Sauce Labs during the teardown of every test:

*** Settings ***

Library  Selenium2Library  timeout=10  implicit_wait=0.5
Library  example.product.testing.Keywords

Test Setup  Start browser
Test Teardown  Run keywords  Report test status  Close All Browsers

*** Variables ***

${ZOPE_HOST} =  localhost
${ZOPE_PORT} =  55001
${ZOPE_URL} =  http://${ZOPE_HOST}:${ZOPE_PORT}

${PLONE_SITE_ID} =  plone
${PLONE_URL} =  ${ZOPE_URL}/${PLONE_SITE_ID}

${BROWSER} =  Firefox
${REMOTE_URL} =
${DESIRED_CAPABILITIES} =  platform:Linux
${BUILD_NUMBER} =  manual

*** Test Cases ***

Plone site
    [Tags]  start
    Go to  ${PLONE_URL}
    Page should contain  Plone site

*** Keywords ***

Start browser
    ${BUILD_INFO} =  Set variable
    ...           build:${BUILD_NUMBER},name:${SUITE_NAME} | ${TEST_NAME}
    Open browser  ${PLONE_URL}  ${BROWSER}
    ...           remote_url=${REMOTE_URL}
    ...           desired_capabilities=${DESIRED_CAPABILITIES},${BUILD_INFO}

Report test status
    Report sauce status  ${TEST_STATUS}  ${TEST_TAGS}  ${REMOTE_URL}

Please, note how we had to replace suite setup and teardown with test setup and teardown) to open a new Selenium session for every test.

This worked for me. I hope it works for you too.

example.product is available at: https://github.com/datakurre/example.product

buildout.cfg

[buildout]
extends = http://dist.plone.org/release/4.2.1/versions.cfg
develop = .
parts = test
versions = versions

[test]
recipe = zc.recipe.testrunner
eggs = my_package[test]

Nothing special here. I expect setup.py of the tested package to include complete extras_require={'test': ... } with all the required dependencies for testing.

So, on a local machine, python bootstrap.py, bin/buildout and bin/test combo should run tests for a freshly cloned package repository just as expected.

travis.cfg

[buildout]
extends = buildout.cfg
parts =
    download
    install
    test
eggs-directory = buildout-cache/eggs
download-cache = buildout-cache/downloads

[download]
recipe = hexagonit.recipe.download
url = https://launchpad.net/plone/4.2/4.2.1/+download/Plone-4.2.1-UnifiedInstaller.tgz

[install]
recipe = collective.recipe.cmd
on_install = true
cmds = tar jxvf ${download:location}/Plone-4.2.1-UnifiedInstaller/packages/buildout-cache.tar.bz2 1>/dev/null

Here's the magic for re-using Plone Unified Installer for your test buildout:

1. At first, download and unpack the installer in [download] part
2. then extract its buildout-cache in [install] part into the locations defined in [buildout] part.

As you might guessed, after this, buildout needs to download only the extra requirements of the tested package! Long live Plone Unified Installer!

.travis.yml

language: python
python: "2.7"
install:
  - mkdir -p buildout-cache/downloads
  - python bootstrap.py -c travis.cfg
  - bin/buildout -N -t 3 -c travis.cfg
script: bin/test

Note, how we need to create a buildout-cache-directory for downloads as defined earlier in travis.cfg. The rest should be easy: we just do the bootstrap and run our tests with sane buildout-options, and... that's all.

.travis.yml for robotsuite

Oh, in the beginning, I mentiond about learning something important from Godefroid. Well, if you have followed me on creating zope.testrunner-compatible Robot Framework -tests with plone.app.testing, you only need to add a few extra lines to make your Robot Framework tests runnable on Travis CI:

language: python
python: "2.7"
install:
  - mkdir -p buildout-cache/downloads
  - python bootstrap.py -c travis.cfg
  - bin/buildout -N -t 3 -c travis.cfg
before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
script: bin/test

If you think, this is cool, please, give some love for the Travis CI team!

Environment

Here's our dummy Plone add-on package with its testing buildout:

bootstrap.py
buildout.cfg
CHANGES.txt
README.txt
setup.py
src
src/my
src/my_package
src/my_package/__init__.py
src/my_package/tests
src/my_package/tests/__init__.py
src/my_package/tests/test_robot.py
src/my_package/tests/test_accessibility.robot

We've got bootstrap.py, empty text files for README and CHANGES, and the following setup.pyto define our (empty) add-on package:

from setuptools import setup, find_packages

version = "1.0.0"

setup(
    name="my-package",
    version=version,
    description="An example Plone add-on",
    long_description=(open("README.txt").read() + "\n" +
                      open("CHANGES.txt").read()),
    # Get more strings from
    # http://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        "Programming Language :: Python",
    ],
    keywords="",
    author="",
    author_email="",
    url="",
    license="GPL",
    packages=find_packages("src", exclude=["ez_setup"]),
    package_dir={"": "src"},
    include_package_data=True,
    zip_safe=False,
    install_requires=[
        "setuptools",
    ],
    extras_require={"test": [
        "plone.app.testing",
        "rootsuite",
        "robotframework-selenium2library",
    ]},
    entry_points="""
    # -*- Entry points: -*-
    [z3c.autoinclude.plugin]
    target = plone
    """
)

Note, how we've defined test-extras for our package to require robotsuite and robotframework-selenium2librarypackages in addition to the usual plone.app.testing.

And here's our buildout.cfg to set up the test runner:

[buildout]
extends = http://dist.plone.org/release/4.2-latest/versions.cfg
parts = test
develop = .

[test]
recipe = zc.recipe.testrunner
eggs = my-package [test]

Test suite

Let's write our first test suite in Robot Framework syntax into src/my_package/tests/test_accessibility.robot:

*** Settings ***

Library  Selenium2Library  timeout=10  implicit_wait=0.5

Suite Setup  Start browser
Suite Teardown  Close All Browsers

*** Test Cases ***

Plone Accessibility
    Goto homepage
    Click link  Accessibility
    Page should contain  Accessibility Statement

*** Keywords ***

Start browser
    Open browser  http://localhost:55001/plone/

Goto homepage
    Go to  http://localhost:55001/plone/
    Page should contain  Plone site

Note, how we import and configure Selenium2Library, and how we expect Plone to be found at http://localhost:55001/plone/. That's how plone.app.testing serves it.

Robotsuite

The last step is to glue our Robot Framework test suite and plone.app.testing together. That's done with robotsuite-package by defining new a RobotTestSuite with the default PLONE_ZSERVER-layer from plone.app.testing in src/my_package/tests/test_robot.py:

import unittest

import robotsuite
from plone.app.testing import PLONE_ZSERVER
from plone.testing import layered


def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(robotsuite.RobotTestSuite("test_accessibility.robot"),
                layer=PLONE_ZSERVER),
    ])
    return suite

If you have ever defined a Python doctest test suite to be used with plone.app.testing, the above should look very familiar.

Running

With everything above in place, just run:

1. bootstrap (with a Plone-compatible Python or virtualenv)

   
   $ python bootstrap.py
   

2. buildout

   
   $ bin/buildout
   

3. and test

   
   $ bin/test
   

and you should see something like:

$ python bootstrap.py
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz
Extracting in /var/folders/b1/mld_r9wj1jbfwf2jcfl_d61sc6kdnb/T/tmp902seC
Now working in /var/folders/b1/mld_r9wj1jbfwf2jcfl_d61sc6kdnb/T/tmp902seC/distribute-0.6.35
Building a Distribute egg in /var/folders/b1/mld_r9wj1jbfwf2jcfl_d61sc6kdnb/T/tmp69NImk
/var/folders/b1/mld_r9wj1jbfwf2jcfl_d61sc6kdnb/T/tmp69NImk/distribute-0.6.35-py2.7.egg
Creating directory '/.../bin'.
Creating directory '/.../parts'.
Creating directory '/.../develop-eggs'.
Generated script '/.../bin/buildout'.

$ bin/buildout
Develop: '/.../.'
Installing test.
...
Generated script '/.../bin/test'.

$ bin/test
Running plone.app.testing.layers.Plone:ZServer tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.398 seconds.
  Set up plone.app.testing.layers.PloneFixture in 9.921 seconds.
  Set up plone.testing.z2.ZServer in 0.506 seconds.
  Set up plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 2.969 seconds.
Tearing down left over layers:
  Tear down plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Tear down plone.app.testing.layers.PloneFixture in 0.088 seconds.
  Tear down plone.testing.z2.ZServer in 5.151 seconds.
  Tear down plone.testing.z2.Startup in 0.009 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.005 seconds.

You should also find Robot Framework logs and reports being generated into your buildout directory under parts/test.

Custom layer

Obviously, we'd like to run our test against a Plone with our own add-on installed. That requires a custom test layer, as described at plone.app.testing.

Let's start by adding a few more files:

src/my_package/configure.zcml
src/my_package/hello_world.pt
src/my_package/testing.py
src/my_package/tests/test_hello_world.robot

At first we define our custom view to be tested in src/my_package/configure.zcml:

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:browser="http://namespaces.zope.org/browser">

    <browser:page
        name="hello-world"
        for="Products.CMFCore.interfaces.ISiteRoot"
        template="hello_world.pt"
        permission="zope2.View"
        />

</configure>

and in src/my_package/hello_world.pt:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="context/main_template/macros/master"
      i18n:domain="plone">
<body>

<metal:content-core fill-slot="content-core">
    <metal:content-core define-macro="content-core">
      <p>Hello World!</p>
    </metal:content-core>
</metal:content-core>

</body>
</html>

Then we define our custom test layer in src/my_package/testing.py:

from plone.app.testing import (
    PloneSandboxLayer,
    FunctionalTesting,
    PLONE_FIXTURE,
)

from plone.testing.z2 import ZSERVER_FIXTURE


class MyPackageLayer(PloneSandboxLayer):
    defaultBases = (PLONE_FIXTURE,)

    def setUpZope(self, app, configurationContext):
        import my_package
        self.loadZCML(package=my_package)


MY_PACKAGE_FIXTURE = MyPackageLayer()

MY_PACKAGE_ROBOT_TESTING = FunctionalTesting(
    bases=(MY_PACKAGE_FIXTURE, ZSERVER_FIXTURE),
    name="MyPackage:Robot")

Note, how we build on top of PloneSandboxLayer and how we create our final acceptance test layer by combining our custom MY_PACKAGE_FIXTURE and ZSERVER_FIXTURE. The latter would make our Plone sandbox served at http://localhost:55001/. Finally, FunctionalTesting gives us a clean isolated Plone site to be played with for each test case.

Finally, we write a new Robot Framework test suite into src/my_package/tests/test_hello_world.robot:

*** Settings ***

Library  Selenium2Library  timeout=10  implicit_wait=0.5

Suite Setup  Start browser
Suite Teardown  Close All Browsers

*** Test Cases ***

Hello World
    Go to  http://localhost:55001/plone/hello-world
    Page should contain  Hello World!

*** Keywords ***

Start browser
    Open browser  http://localhost:55001/plone/

We can now include our new test suite in src/my_package/tests/test_robot.py:

import unittest

import robotsuite
from my_package.testing import MY_PACKAGE_ROBOT_TESTING
from plone.app.testing import PLONE_ZSERVER
from plone.testing import layered


def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(robotsuite.RobotTestSuite("test_accessibility.robot"),
                layer=PLONE_ZSERVER),
        layered(robotsuite.RobotTestSuite("test_hello_world.robot"),
                layer=MY_PACKAGE_ROBOT_TESTING),
    ])
    return suite

and re-run our tests:

$ bin/test --list-tests
Listing my_package.testing.MyPackage:Robot tests:
  Hello_World (test_hello_world.robot)
Listing plone.app.testing.layers.Plone:ZServer tests:
  Plone_Accessibility (test_accessibility.robot)

$ bin/test
Running my_package.testing.MyPackage:Robot tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.219 seconds.
  Set up plone.app.testing.layers.PloneFixture in 7.204 seconds.
  Set up my_package.testing.MyPackageLayer in 0.028 seconds.
  Set up plone.testing.z2.ZServer in 0.503 seconds.
  Set up my_package.testing.MyPackage:Robot in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 2.493 seconds.
Running plone.app.testing.layers.Plone:ZServer tests:
  Tear down my_package.testing.MyPackage:Robot in 0.000 seconds.
  Tear down my_package.testing.MyPackageLayer in 0.002 seconds.
  Set up plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 2.213 seconds.
Tearing down left over layers:
  Tear down plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Tear down plone.app.testing.layers.PloneFixture in 0.091 seconds.
  Tear down plone.testing.z2.ZServer in 5.155 seconds.
  Tear down plone.testing.z2.Startup in 0.009 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.005 seconds.
Total: 2 tests, 0 failures, 0 errors in 18.305 seconds.

Logging in

plone.app.testing defines a test user for our test site, but how could our Robot Framework test know her login credentials? Well, we have to make our test to ask for the credentials by defining custom Robot Framework test keywords in Python.

Let's add a couple of more files, as in:

src/my_package/testing_keywords.py
src/my_package/tests/test_login.robot

At first, we type our custom Robot Framework keyword library with test keywords for retrieving the test users credentials into src/my_package/testing_keywords.py:

class Keywords(object):
    """Robot Framework keyword library
    """

    def get_test_user_name(self):
        import plone.app.testing
        return plone.app.testing.interfaces.TEST_USER_NAME

    def get_test_user_password(self):
        import plone.app.testing
        return plone.app.testing.interfaces.TEST_USER_PASSWORD

Then, we can write our new login test into src/my_package/tests/test_login.robot:

*** Settings ***

Library  Selenium2Library  timeout=10  implicit_wait=0.5
Library  my_package.testing_keywords.Keywords

Suite Setup  Start browser
Suite Teardown  Close All Browsers

*** Test Cases ***

Log in
    ${TEST_USER_NAME} =  Get test user name
    ${TEST_USER_PASSWORD} =  Get test user password
    Go to  http://localhost:55001/plone/login_form
    Page should contain element  __ac_name
    Input text  __ac_name  ${TEST_USER_NAME}
    Input text  __ac_password  ${TEST_USER_PASSWORD}
    Click Button  Log in
    Page should contain element  css=#user-name

*** Keywords ***

Start browser
    Open browser  http://localhost:55001/plone/

Note, how we can import our custom keyword library right after Selenium2Libary. Also, see how we use our custom keywords to retrieve test user's login credentials into Robot Framework test variables and how we use them later in the test.

We can now include our new test suite in src/my_package/tests/test_robot.py:

import unittest

import robotsuite
from my_package.testing import MY_PACKAGE_ROBOT_TESTING
from plone.app.testing import PLONE_ZSERVER
from plone.testing import layered


def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(robotsuite.RobotTestSuite("test_accessibility.robot"),
                layer=PLONE_ZSERVER),
        layered(robotsuite.RobotTestSuite("test_hello_world.robot"),
                layer=MY_PACKAGE_ROBOT_TESTING),
        layered(robotsuite.RobotTestSuite("test_login.robot"),
                layer=PLONE_ZSERVER),
    ])
    return suite

and re-run our tests:

$ bin/test --list-tests
Listing my_package.testing.MyPackage:Robot tests:
  Hello_World (test_hello_world.robot)
Listing plone.app.testing.layers.Plone:ZServer tests:
  Plone_Accessibility (test_accessibility.robot)
  Log_in (test_login.robot)

$ bin/test
Running my_package.testing.MyPackage:Robot tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.217 seconds.
  Set up plone.app.testing.layers.PloneFixture in 7.132 seconds.
  Set up my_package.testing.MyPackageLayer in 0.026 seconds.
  Set up plone.testing.z2.ZServer in 0.503 seconds.
  Set up my_package.testing.MyPackage:Robot in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 2.473 seconds.
Running plone.app.testing.layers.Plone:ZServer tests:
  Tear down my_package.testing.MyPackage:Robot in 0.000 seconds.
  Tear down my_package.testing.MyPackageLayer in 0.002 seconds.
  Set up plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Ran 2 tests with 0 failures and 0 errors in 7.766 seconds.
Tearing down left over layers:
  Tear down plone.app.testing.layers.Plone:ZServer in 0.000 seconds.
  Tear down plone.app.testing.layers.PloneFixture in 0.088 seconds.
  Tear down plone.testing.z2.ZServer in 5.156 seconds.
  Tear down plone.testing.z2.Startup in 0.009 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.005 seconds.
Total: 3 tests, 0 failures, 0 errors in 23.765 seconds.

Debugging

There's one catch in debugging your code while running Robot Framework tests. It eats your standard input and output, which prevents you to just import pdb; pdb.set_trace(). Instead, you have to add a few more lines to reclaim your I/O at first, and only then let your debugger in:

import sys
for attr in ('stdin', 'stdout', 'stderr'):
    setattr(sys, attr, getattr(sys, '__%s__' % attr))
import pdb; pdb.set_trace()

Resources

• plone.act(Plone keyword library candidate)
• Selenium2Library keywords
• Robot Framework built-in keywords

That's all about it to get started. Have fun!

Introduction

Recently I looked into JavaScript dependency management and wrote a long post about it where I was overwhelmed and found some solutions. Since then I've learned a few new things and have thought a bit more, and I thought I'd share.

My goal is still to be able to develop Obviel with smaller modules than I do now. Obviel is a client-side web framework, but besides that you should not need to know anything about it in order to understand this post.

So, I'm looking for a JavaScript dependency framework for the client-side.

AMD versus CommonJS

Last time I mentioned Asynchronous Module Definition (AMD) and how it contrasts with the CommonJS module definition. AMD wraps everything in your module in a function:

define([jquery], function($) {
   var myModule = {
      foo: function() { ... };
   };
   return myModule;
});

Whereas CommonJS uses this pattern:

var $ = require('jquery');

module.exports = { foo: function() { ... }};

Though AMD has sugar to make it look much like CommonJS plus the define() function wrapper.

AMD was designed to work in web-browsers, client-side, whereas CommonJS modules are mostly used on the server, typically in a NodeJS environment. AMD modules can be directly loaded into the browser without build step, which is an important advantage during development. AMD modules like CommonJS modules can also refer to individual modules in other packages. You could install such an AMD-based package using Bower, like you'd install a CommonJS-based package using npm.

The most well known AMD implementation is called RequireJS. There is a small implementation of it called Almond that RequireJS can use in JavaScript modules that are packaged together for release.

Recently I also learned about RaptorJS, which is a larger framework that features an extended AMD implementation as well as a separate server-side module loader. It contains some interesting ideas such as "adaptive packaging" which helps adjust codebases to different target environments (client, servers, different browsers, etc).

CommonJS on the client

Of course people have worked on bringing CommonJS modules to the client too. And have they! I ran into these so far:

• browserify
• OneJS
• commonjs-everywhere
• browser-build
• component

May a thousand flowers bloom! Overwhelmed, me?

I've done a cursory review of these, so I apologize if I get something wrong, but here goes:

• browserify is a tool that can take a file with a CommonJS module (and its dependencies) and bundle them all up in a .js file you can use in the browser.

• OneJS seems to do something very similar. The docs don't make it immediately obvious what its distinctive features are.

• commonjs-everywhere does the same again (I think... the docs are a bit technical...), but has more features.

  One cool thing is source maps support. Source maps I just found out about: they are basically a way to add debugging symbols to a minified, bundled .js file so the debugger can find out about the original source. This is handy if you bundle plain JS, and also makes it possible to offer better debugger support for languages such as CoffeeScript which compile to JavaScript. Source maps are only supported in Chrome right now, with Firefox support coming up.

  [update: a commenter pointed out that browserify supports source maps too,

  pass a --debug flag during building to enable this]

• browser-build is a very recent implementation that does the same as the others, but is focused on performance: producing a browser version of CommonJS modules of your project really fast, so you never have to wait for your tool during development. It has support for source maps.

  But it also does something more: if you write your code in plain JavaScript (as opposed to CoffeeScript, say), it makes your original modules available to the browser in only very slightly edited form (the line numbers are the same). This should help debugging a great deal in browsers that don't support source maps.

  But I'm unsure about the details, as browser-build needs to have CoffeeScript available to compile the sources and run it and I too lazy to try this right now.

• component is also a package manager (I'll mention it again later in that context), but also contains a build system to generate a single .js file from CommonJS modules.

All of these approaches need a build step during development, which makes debugging harder, though source maps will help. browser-build minimizes the build step during development to the bare minimum, however, and this will help debugging.

Both? uRequire

Then there's uRequire.

uRequire allows you to use CommonJS or AMD or both and converts them to whatever is needed. It talks about a Universal Module Definition (UMD) I haven't studied yet, but apparently its not necessary to use its boilerplate when using uRequire. From what I understand in order to use code in the browser a build step is required.

JS frameworks with a module system

There are lots of JavaScript client-side frameworks that have grown their own module systems. I'll only talk about a few here that seem relevant.

• Dojo had its own module system, but has started to use AMD in recent versions and has been pushing this forward. You can use Dojo modules directly in your own codebase - this kind of inter-package reuse is something I will go into more detail later.
• Closure Tools by Google contains a lot of things, such as a powerful JavaScript compiler and various JavaScript libraries. It also features its own module system, which I'll also talk about more later.

JavaScript Packages and Modules Redux

In the previous blog entry I explored some concepts surrounding dependencies and modules. I've had some new insights on how these concepts apply to the JavaScript world. I'll review some of this again, this time with a focus on what these concepts look like in JavaScript.

• a project is the codebase you're hacking on. In open-source JS land, typically it's something published on github. It's in hackable form, so contains a lot of separate .js files to separate concerns: modules.

• a module is a JavaScript file that provides some functionality by exposing an API.

• a module may depend on another module. A module dependency is expressed in the module itself, in source code. In JavaScript there are multiple systems for expressing modules and their dependencies, such as CommonJS, AMD and Google Closure.

• a package is a collection of one or more modules that is published somewhere so others may use it (this may be published on the internet, or internal to a project). It has metadata that describes the package, its version number, who wrote it, and what other published packages it depends on.

  CommonJS packages on the server-side are distributed as essentially an archive of a CommonJS project: a lib directory full of modules, with package.json for metadata.

  Traditionally client-side JavaScript packages are just distributed as URLs pointing to a .js file that people can download. So, to get jQuery, you download a version at the jQuery website. This is a very large difference between browser and server.

  Bower packages are a formalization of this traditional pattern: there is a single .js file with bower.json metadata to describe it. Bower adds metadata (bower.json) and a package index to the original story.

  In fact the Bower story is more complicated: it does allow you to package up a directory of multiple modules too, which you could then use using, say, RequireJS. This is an entirely different way to use modules, but Bower is agnostic and just installs the stuff. Bower also supports listing more than one .js file in its bower.json configuration file; it's unclear to me what the semantics of this is exactly.

• Package generation. This is something I skipped in the previous discussion of concepts, but is very important in the JavaScript world especially.

  CommonJS packages are just archived versions of a particular project layout: a directory with a package.json, with a lib subdirectory which contains the individual .js modules.

  Browser-targeted packages are most commonly shipped as a single .js file as mentioned before. In the most simple case you maintain this .js file by hand directly and give it to others to use.

  But you can also generate a single .js package file by compiling a bunch of .js module files together. This is what the CommonJS generators described above do, except for browser-build, which actually maintains a tree of unbundled .js modules.

  The realization I had, perhaps obvious, is that a client-side JavaScript package is often shipped as a single compiled .js file. It's like how a C linker works - it bundles individually units into a larger library file (.so, .dll).

• A package manager is a tool that installs a package into your system so you can start using it. npm is popular for NodeJS, Bower is focused on client-side packages and tries to be very package format agnostic. component contains a package manager too, centered around CommonJS (and also the build tool I mentioned earlier).

• A package registry is a system where packages can be registered so that others may find and download them. npm has an index, and so do Bower and component.

MantriJS

Another dependency system I ran into since my last post is MantriJS. MantriJS is built around the Google Closure Tools but hides them from the developer, except for the dependency system.

You define a module that depends on others like this in Mantri/Closure Tools:

goog.provide('obviel.template');
goog.require('obviel.util');

obviel.template.foo = function() { ... };

Here you say you are defining a module called obviel.template and that in this module obviel.util needs to be available. Once you require something you have that available to use in your module, so you can now do this:

obviel.util.blah();

Mantri has a build step for development, but only to build a deps.js file and only when you've changed a dependency. The modules themselves are directly exposed to the browser during development, meaning you can debug them. In this it looks quite similar to browser-build, though browser-build does touch the individual modules in a minor way, something MantriJS does not.

MantriJS does offer a build step to generate a single file .js package from your modules, using the Closure Tools.

I tried to see whether MantriJS was easy to integrate with the Buster.JS test runner; I had to wrestle quite a bit to get RequireJS running properly before. It turned out to be very easy (it Just Worked ™!). See the jspakmantri example project and compare it with the original RequireJS-based jspak project if you like.

Thinking about MantriJS I realized something: MantriJS actually allows you to have modules the way many existing client libraries do it: create a namespace object and fill it with properties you want to expose. This is important to me because that's how Obviel does it now, and I'd prefer not to break that client API.

Global Namespace Modules

So what is this client library module definition pattern MantriJS supports? Everybody is familiar with it. It's what jQuery does for instance: it fills the jQuery object ($) with the jQuery API and exposes this.

For example, to make a module, you create an empty object, perhaps listed in another object to namespace it, and make it globally available:

var obviel = {};
obviel.template = {};

You then fill it with the API you want somehow, for instance like this:

obviel.template.foo = function() { ... };

or like this:

(function(module) {
   module.foo = function() { ... };
}(obviel.template));

To use a module from another one, you simply refer to it:

obviel.template.foo();

That's all that's needed, but there are also frameworks that help you declare and use modules like this, such as MantriJS mentioned earlier; YUI has another one. The primary benefit these add is the ability to express module dependencies better, avoiding the need to mess around with <script> tags.

So this pattern is neither CommonJS or AMD. But it is very widely used on the client-side. Obviel uses it for instance, and Backbone too, and Ember, and Knockout, and Mustache, and YUI, and Google Closure Tools. To just list a few. Let's call it the Global Namespace Modules pattern (GNM).

GNM is not a module definition pattern like CommonJS or AMD. Instead it is defined by how modules are used: you refer to the API of a module using a global namespace that the module exposes (jQuery, obviel, Backbone, Mustache, etc).

GNM assumes that modules are loaded in a particular order, synchronously. You ensure this order by listing <script> tags in a particular order, or by using a smart module loader like MantriJS, or by bundling modules in order into a single .js package file.

Getting this more clear for myself is quite important to me. It had been bugging me for a while after reviewing RequireJS: if I start using it for Obviel, do I need to to break the Obviel API, which assumes GNM. Or do I tell all developers to start using AMD for their code that uses Obviel too?

[update: here is a post with more on this pattern; here's another]

Requirements

After thinking about all this, here are some varying requirements for a JavaScript module dependency system. Ideally Obviel can adopt one that has all of these properties, or as close as possible:

• automated loader: no <script> tag management. (loader)

• encourage fine-grained modules. (fine)

• being able to use browser debuggers like Firebug or the Chrome Dev tools. (debug)

• source maps not required: being able to use these debuggers without relying on new source map technology. (nosm)

• no build step needed during development. (nobuild)

• support for exposing modules using the GNM pattern. Is this really important? Yes, as it's a very popular pattern on the web. Dojo went the way of telling people to use AMD for their own code, and that does help with fine-grained reuse between packages... (gnm)

• compilation tools: bundling, minification to deliver easy to use .js files. This way the browser can load a package efficiently and it becomes easy for people to start using the API the package exposes: just drop in a file. (comp)

• inter-package reuse: being able to require just one module from another package without having to load all of them. (reuse)

  There is some tension here with the bundling into a single .js package approach - if there's a module in a package that I don't use, why does it still get shipped to a web browser? On the server installing a bit more JS code in a package is not a problem, but on browsers people tend to start counting bytes.

  This tension can be reduced in various ways: jQuery now offers various smaller builds with less code. Build tools can cleverly only include modules that are really required, though for inter-package reuse this can defeat the benefit of caching.

• integration with BusterJS test runner. As this is the test runner I use for Obviel. Preferably with the least hassle. (bjs)

• CommonJS everywhere: client definition of modules same as on server, so CommonJS packages can be used on the client too. There is after all potentially a lot of useful code available as a CommonJS package that can be used on the client too, and potentially some of my Obviel code can be run on the server too. (cjs)

Review

Let's review some of the systems mentioned in the light of these requirements. If I get it wrong, please let me know!

[update: source maps are also a browserify feature]

A few notes from the perspective of Obviel:

Nothing ticks all the boxes for Obviel from what I can see. RequireJS, MantriJS and browser-build come closest.

The manual system involves maintaining <script> tags yourself. That is what I'm doing with Obviel now. It involves no build step, so debugging is easy during development. It supports the popular global namespace modules pattern. If a framework exposes multiple modules that users are to include using <script> tags, like Obviel currently does, then inter-package reuse is possible. Compilation into a single .js file is not needed but there are tools that can do it for you. But it's not fine-grained at all, breaking a fundamental requirement for Obviel.

RequireJS is quite nice; script tag management goes away, no build step is needed but compilation to a .js file is still possible. It allows fine-grained reuse of modules in other RequireJS based packages, which is very nice. After some effort it integrates with BusterJS. But it doesn't offer Global Namespace Modules support out of the box. It shouldn't be too hard to make it do that, though, by simply exposing some modules myself, possibly during a build step.

The various CommonJS approaches are interesting. It is attractive is to be able to use same approach on the browser as on the server. But most tools require a bundling build step and I'd like to avoid having to rely on still uncommon source maps to do debugging. That's why browser-build is one of the more interesting ones, as it minimizes the build step required and makes debugging easier.

I still a bit unclear to me whether fine-grained module reuse of other npm-installed packages is possible - do these modules get exposed to the browser too (in a bundle or directly for browser-build?). From what I've read here and there I think so. I also haven't explored how easy it is to integrate these with client-side Buster (server-side Buster integration is supported by Buster), but I get the impression it's posible.

The CommonJS approaches don't offer Global Namespace Modules support so I'd have to hack that up as for RequireJS.

MantriJS was quite a revelation to me as it helped me come clarify my thinking about the Global Namespace Modules pattern. I've contacted the author and he's very responsive to my questions, also nice. It turned out to be dead-easy to integrate with Buster.JS. MantriJS assumes that external JS packages are bundled up in a single .js file for reuse however, so fine-grained module reuse of other packages is not possible.

Still overwhelmed

I'm still overwhelmed by the choices available as well as all the details. But I know a bit more about what's possible and what I want now. Are there any players in this field that I missed? Undoubtedly more will come out of the woodwork soon. What do you think about my requirements? Should I just give up on GNM, or forget about not having a build step during development? Am I missing an important requirement? Please let me know!