How to show stacktrace, standard output and standard error of each test failure in jenkins mail

Email-ext plugin in jenkins allows customizing jenkins build mail at a programmer's will.


Find you jelly script and go to JUnit TEMPLATE portion of the script.

Here's an example how you can change the script to show stacktrace, standard output and standard error.

 <!-- JUnit TEMPLATE -->  
 <j:set var="junitResultList" value="${it.JUnitTestResult}" />  
 <j:if test="${junitResultList.isEmpty()!=true}">  
  <TABLE width="100%">  
   <TR><TD class="bg1" colspan="2"><B>JUnit Tests</B></TD></TR>  
   <j:forEach var="junitResult" items="${it.JUnitTestResult}">  
    <j:forEach var="packageResult" items="${junitResult.getChildren()}">  
     <TR><TD class="bg2" colspan="2"> Name: ${packageResult.getName()} Failed: ${packageResult.getFailCount()} test(s), Passed: ${packageResult.getPassCount()} test(s), Skipped: ${packageResult.getSkipCount()} test(s), Total: ${packageResult.getPassCount()+packageResult.getFailCount()+packageResult.getSkipCount()} test(s)</TD></TR>  
     <j:forEach var="failed_test" items="${packageResult.getFailedTests()}">  
      <TR bgcolor="white">  
        <TD class="test_failed" colspan="1">  
          <B><li><h2>Failed: ${failed_test.getFullName()} </h2></li></B>  
        </TD>  
      </TR>  
      <TR>  
         <TD class="test_failed_log" colspan="2">  
          <li><h4>Stacktrace</h4></li><li>${failed_test.getErrorStackTrace()} </li>  
          <li><h4>Standard Output</h4></li><li>${failed_test.getStdout()} </li>  
          <li><h4>Standard Error</h4></li><li>${failed_test.getStderr()} </li>  
        </TD>  
      </TR>  
     </j:forEach>  
    </j:forEach>  
   </j:forEach>  
  </TABLE>  
 <BR/>  
 </j:if>  

You can further change the visual effects through css classes.

To know more options of the ${failed_test} object see CaseResult

Create Test Suite for Tycho after login

Sometimes tests fail when they are not actually running. This typically happens in case of integration-tests. When something goes wrong(like login failed) the consequent tests all get failed and the build server starts flooding with the infamous All tests failed! mails!

What to do in such case?
Should we run the tests if the login failed even if they were supposed to run after a successful login?

We faced this recently with our RCP application which connects to a remote server to properly function.
We have around 50 SWTBot ui-tests for testing the features. The tests are run in our CI server.
SWTBot first logs onto the server using a login dialog.
But when the login fails we don't want to run the tests at all, here's what we did:

We created a TestSuite which returns all tests only when login succeeds.
Of course it can be customized further to give more logic to load each test case (like different test cases for different platforms!)

package package.of.test.suite;

import junit.framework.JUnit4TestAdapter;
import junit.framework.TestSuite;

import org.apache.log4j.Logger;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swtbot.eclipse.finder.SWTWorkbenchBot;

/*
 * A test suite which creates tests depending on whether the login succeeded or not.
 * Return all tests if login succeeds
 * Else return a suite containing no tests.
 */
public class AllTests extends TestSuite {

    private static boolean loggedIn = initializeProductForTesting();
    private static Logger  mLogger  = Logger.getLogger(AllTests.class);
    private static SWTWorkbenchBot bot;
    public static boolean initializeProductForTesting() {
        System.out.println("in Beforeclass method before all tests!");
        Utils.login();
        return Utils.isLoggedIn();
    }

    public static TestSuite suite() {
        TestSuite suite = new TestSuite();
        if (loggedIn) {
            suite.addTest(new JUnit4TestAdapter(Test1.class));
            suite.addTest(new JUnit4TestAdapter(Test2.class));
            suite.addTest(new JUnit4TestAdapter(Test3.class));
        }
        else {
            mLogger.debug("Can't log into server!");
            mLogger.debug("No tests will run !");
            Display.getDefault().close();
            System.exit(0);
        }
        return suite;
    }
}

We configure tycho-surefire plugin to run this suite
<configuration>
                    ...
                    <testSuite>bundle.symbolic.name.of.test.plugin</testSuite>
                    <testClass>package.of.test.suite.AllTests</testClass>
                    ...
</configuration>

stitch eclipse + swt + swtbot + junit + maven + tycho + surefire

In jist a high level view of running swtbot ui tests with maven/tycho

1. Specify proper packaging i.e eclipse-test-plugin
2. Specify src directory to maven build, usually it’s different in eclipse than what maven thinks
3. Specify various parameters(memory, language, arch, ws, os, ..) correctly to vm and program
4. Add necessary elements (UIHarness, UIThread…, application…) to tycho-surefire configuration
5. Add all implicit dependencies of application/product as explicit to manifest file of the test plugin. Because tycho cannot resolve implicit dependencies.
6. Add p2 repositories to pom so that target platform can be resolved into pom dependencies.
7. Add Declarative Services library/ dependency to test plugin if it uses any declarative service to initialize application/product

All above steps assume at least mid level knowledge on maven/tycho/swtbot/eclipse/rcp and those who have tried the integration already.