SpecFlow on VS11

Today was an exciting day. It started with Visual Studio 11 beta install and finished with uploading a working SpecFlow VS11 package to github downloads. Cool.

Currently I have created a hand-made package for v1.8.1, to test whether the integration components work well on VS11. Later I’ll include the integration package to the standard deployment process.

Currently the things seem to work well. The “Run SpecFlow Scenarios” feature does not seem to work with the MsTest runner, but with the ReSharper EAP this works too.

If you see any problems with the integration, please post it here.

It’s yet unclear to me, how the installation process will look like in the future. On one hand it’s tempting to put the VS11 (and maybe also the VS2010) integration to the VS gallery, but this way the file association (.feature) cannot be set. Currently the SpecFlow installer installs the tools and the runtime for users who don’t use NuGet. If the installation is done through the gallery, we need to find a solution for this too. If you have suggestions on this, please let me know.

SpecFlow 1.8

We have released SpecFlow v1.8 recently (just published 1.8.1 with a few fixes) and I would like to take the opportunity to give a subjective summary of the new features and also some outlook for the upcoming releases.

Run & debug scenarios from context menu in VS2010

Maybe the most interesting (most wanted) feature is to make scenario execution more convenient. Now it is here: in the context menu of the editor, the project, folder or file nodes of the solution explorer, you will find new commands to run or debug the scenarios of the current context: the scenario of the cursor position (or all from the file if you stand in the header or between scenarios) or the scenarios in the files matching to the selected solution explorer node.

Since SpecFlow does not run the scenarios directly, but you run them with a unit test runner, this feature is a little bit tricky and needs a short explanation.

When you select the “Run SpecFlow Scenarios” command, SpecFlow tries to run the selected scenarios with your favorite unit test runner. You can either tell SpecFlow which one is your favorite or let it find out automatically based on some built-in ranking. This can be done in the SpecFlow section of the Visual Studio Tools/Options dialog.

From the picture you can also see the list of supported unit test runners (ReSharper (5 and 6), MsTest and SpecRun). The TDD.NET is in the list, but the gateway is not yet implemented. Contributions for the TDD.NET gateway or any other that is not listed yet is welcome.

Cucumber harmonization

As you’ve probably heard, Richard Lawrence has announced that the Cuke4Nuke project has been retired and we decided to merge our efforts to further improve SpecFlow.

We have been working for a while with Richard, Paul, Christian, Jonas and Darren for the preparation of this and we have started to implement features to be better compatible with Cucumber.

Currently we are at half-way. We have consolidated the terms (e.g. using “hook”) and validated/fixed some obvious compatibility problem. The remaining stuff is mainly concerning the table manipulation API, that is currently covered by the Assist namespace in SpecFlow. Regarding this, we would like to

• make the table API extensions better discoverable by merging them to the main “TechTalk.SpecFlow” namespace
• consolidate the different CompareToX methods into one
• improve the output of the table comparison methods

Improvements in the SpecFlow architecture

SpecFlow has an evolving architecture so there are always improvements here as the features grow. 1.8 is maybe sill an interesting milestone, because we have fully refactored the class-dependency model of the runtime code and have replaced the custom object factory solution with a DI tool.

This step was important, because this consolidated and solved issues with the context injection feature (e.g. disposing injected objects) and also because this has opened up new ways for customizing SpecFlow. Here is an example for the customization:

Let’s say you want to implement a custom way of reporting SpecFlow user errors (e.g. for doing something special for pending steps). For this you need to create a class that implements the IErrorProvider interface. You can let SpecFlow use your custom provider by changing the default dependencies in the configuration like this:

Also the context injection feature can resolve the SpecFlow classes as well, so for example if you need to apply the same conversion logic as SpecFlow uses in your code (including step argument conversions), you can just let an IStepArgumentTypeConverter instance be injected to your step definition class like this:

SpecFlow tests for SpecFlow

Also a new improvement in the SpecFlow solution that now the specifications and the integration tests have got a new platform. The projects TechTalk.SpecFlow.Specs and TechTalk.SpecFlow.IntegrationTests contain SpecFlow tests for SpecFlow itself. The existing non-unit tests have been converted to the new model and I have also added a couple of additional tests, but the coverage is not yet close to complete. My hope is that with the established environment, it will be easier for the contributors to also provide SpecFlow end-to-end tests for the new features.

There is a similar work also in Cucumber, the cucumber-features. The goal is to make the necessary step definitions that SpecFlow can also run (and pass) these scenarios. But this is something to do still.

SpecRun

If you have followed the posts and tweets of TechTalk, you might know that we are also working on an integration test runner tool, that provides smarter (better, faster, more integrated) feedback for integration test execution.

The SpecRun beta has been updated to be compatible with SpecFlow 1.8.1. Take a look at the website: http://www.specrun.com or just grab the nuget package into your project:

Install-Package SpecRun -ProjectName MyProject

SpecFlow next steps

This post is really getting too long, so just a few bullet point for the future ideas:

• continue with Cucumber harmonization
• improve VS2010 step analysis (performance, stability)
• improve step definition skeleton generation (based on the pull request of pmchugh)
• .NET 4.0 specific enhancements
• simplify customizations with plugin infrastructure

Download it now!

Don’t forget, that the VS integration is independent from the project dependencies, so you don’t have to wait until all your projects can upgrade to 1.8. Go ahead and download the installer for 1.8.

Please also take a look at the documentation. It’s getting better and better, but if you see any glitches in it: it’s a wiki, so just fix it!

Have fun!

SpecRun – because integration tests are not unit tests

We have started the SpecFlow project in 2009 to provide a tool for .NET developers to perform BDD-style functional testing. SpecFlow uses unit testing frameworks, like NUnit, MsTest or MbUnit to execute the Gherkin scenarios. While this was a pragmatic choice, it has turned out that for executing functional integration tests, these unit test frameworks are not optimal (quite understandable, as they are unit testing frameworks and not integration testing frameworks). While they can execute the tests very well, they lack features like

• (html) report as primary output
• fast integration test execution by parallelization
• advanced execution metrics: execution time, memory usage, memory leaking, benchmarking
• detection of “random failures”
• test variations (e.g. running the same suite for IE and Firefox)
• support for applying different test configurations
• provide fast feedback about failures (prioritize failing tests first, stop after a number of failures, etc.)

In addition to that, we were missing a test runner that treats Gherkin features, scenarios and scenario outlines as first-class citizens.

To address these problems, we have decided to launch a new product that fits into the SpecFlow-SpecLog context. It has been publicly announced today, so it’s here: SpecRun

We have been working for this tool for a while now and have tested it in our projects. It is really promising. It seems that with parallel execution, the local test execution time (that is more critical from the cost perspective, as developers can’t progress while the tests are running) can be dropped to ca. 60%, that is almost half time. This and the smarter feedback about the errors were received very positively by the developers on the selected projects. They were more willing to run the tests locally reducing the long round-trips and they were able to detect the root cause of the errors better.

SpecRun is also using SpecRun for testing itself obviously, and our own dog food tastes pretty good!

SpecRun will be a commercial product (with free versions for OSS projects of course). This was a hard decision. We have a wonderful community around SpecFlow and the support of TechTalk, but even this way we could not afford the costs of building up so many new things. I think this is a good combination. While we believe that SpecRun is a great help and provides quick ROI for the users, it is not the main concern of SpecFlow. With the separation of the two aspects, we can have still our SpecFlow efforts focused on the main concerns (editor, runtime API, documentation) and also provide an extension for the smarter execution experience.

SpecRun is currently in a closed beta stage, but we are planning to have the commercial release soon. Have a look at our website: http://www.specrun.com, stay tuned on our twitter account: @specrun and contact info@specrun.com if you are interested in participating in the beta program.

Enhanced upgrading options in SpecFlow v1.7

We have just released SpecFlow v1.7 over the weekend. In this release there are extended integration options for the Silverlight async testing framework and also for SharpDevelop 4.

In addition to that, we have tried to make the upgrade process easier for the SpecFlow users, especially if they work in bigger teams or on different projects in parallel.

The decision to upgrade a tool for a project is usually made carefully. You have to consider your close deadlines, check the breaking changes, and discuss it with the team.

If you work on multiple projects in parallel (e.g. you have to support and older project too), it can be quite hard to synchronize that all projects upgrade at the same time.

As SpecFlow is (was) a single msi installer and the different versions cannot be easily installed side-by-side, the upgrade process was sometimes challenging.

From v1.7 we have managed to have better separation of the components that are directly used by the project (the generator and the runtime) and the IDE integration. With this, you can upgrade SpecFlow on your machine and defer the upgrade for the individual projects.

For that to work, please ensure that your project is using SpecFlow 1.6 or later and it is setup as one of the following (see further advanced options in the documentation):

• SpecFlow is configured in the project as a NuGet package (this is the easiest and the recommended option).
• SpecFlow assemblies (not only TechTalk.SpecFlow.dll, but (almost) all the assemblies from the SpecFlow installation folder) are copied into your project folder structure and the reference for the TechTalk.SpecFlow.dll is taken from this folder. See the detailed list of required assemblies in this documentation page (last section).

If you have this setup, you can safely install a newer SpecFlow version on your machine. The projects will still use the configured generator and runtime.

When you feel confortable to upgrade SpecFlow on your project as well, you can do that project-by-project (the easiest it to use NuGet upgrade).

Note 1: Please make sure that the SpecFlow version on the machine is the same or newer than the SpecFlow version used by the projects.

Note 2: This feature works currently with Visual Studio 2010 only.

 

SpecFlow is and open-source project sponsored by TechTalk. You can also support the project by contributing on github or by “adopting a feature” ;-). TechTalk also provides commercial support for SpecFlow.

Learn SpecFlow at the the Progressive .NET Tutorials: http://skillsmatter.com/event/open-source-dot-net/progressive-dot-net-tutorials-2011

Tracing Web Automation Test Errors with SpecFlow

If you try to run your automated UI tests on the build server, you can quickly run into problems that are quite hard to analyze.

The most common scenario is that one of your assertion like “Then my name should be displayed” fails, because the web driver does not find your name in the resulting page. It works locally, but what’s the problem on the build server? Permission problems? Timing issues? Browser compatibility? Different locale? Web compilation or configuration errors? All of these are potential candidates, but it is hard to find out quickly which one.

We are using the following SpecFlow event binding to trace the content of the current page in case of an error. In many cases from the page content it’s easy to say what the problem was.

[AfterScenario]
public void AfterWebScenario()
{
    if (ScenarioContext.Current.TestError != null)
    {
        Console.WriteLine("Page src for failing test: {0}",
            BrowserContext.SeleniumWebDriver.PageSource);
    }

    //... (if you stop your browser session after
    //     each scenario, this comes here probably, 
    //     to ensure that the tracing runs before)
}

Git & Dropbox for preparing live demos

I have arrived from Oslo where we did two sessions with Jonas Bandi on NDC 2011 about SpecFlow.

As Jonas lives in Switzerland and I live in Hungary, we have used Dropbox to synchronize the content we are working on. This was fine for slides and other notes, but for the code demo we did not have the right toolset so far.

Actually I like to do demos where I modify and evolve the code during the session. But if you do so, you need to maintain different versions of the code base. You need a baseline to start the demo from and probably also a “done” version if something goes wrong. Additionally you need to be able to revert back to the baseline while practicing the demo. This begs for a SCC and Git is a perfect choice.

To have a little bit of “surprise” effect, I don’t like to publish my demos upfront so did not wanted to put it into a github repository. To be able to share the code with Jonas, this time we’ve tried to combine Git with Dropbox like this:

• create a “bare” git repository into a dropbox folder
• clone it to your working folder
• work on the code & commit changes
• push the changes to the dropbox folder (the origin)

Though I’m pretty sure that this is not a well scaling solution for real development, but for two persons it was just fine & simple.

Agile Developer Course in Vienna

In summer everyone is on holidays, the projects are running on low frequency, so maybe it’s a good time to meet others and improve your agile development skills.

July 6-8, 2011 there will be a three-day course in Vienna instructed by me (in English) that focuses on agile engineering practices and team collaboration. The attendees can also practice behavior driven development (with SpecFlow) and TDD. This is all done through a lot of hands-on exercises and group discussions.

There are two days left for the discounted price!

Details & registration: http://www.techtalk.at/scrum-trainings/certified-scrum-developer.html

(Successful completion of the course can be credited towards the Certified Scrum Developer (CSD) certification of Scrum Alliance.)