Difference between revisions of "CddBranch"

(→‎Development: TODO)
m (Reverted edits by Jacksonsophia (Talk) to last version by Aleitner)
 
(242 intermediate revisions by 8 users not shown)
Line 1: Line 1:
−
== Summary ==
+
[[Category:Testing]]
−
CDD is an extension to EiffelStudio that provides support for both automatic test case extraction and test execution.
+
[[Category:EiffelDebugger]]
−
It is undistruptive to the development process of the user. The tool is in its earlys stages. Feedback and bug reports are
+
[[Category:CDD]]
−
very much appreciated. Please send them to mailto:andreas.leitner@inf.ethz.ch and or use the  discussion page for this article.
+
  
  
−
[[Image:video_still.png|center]]
+
== What is the CDD EiffelStudio? ==
  
 +
[[Image:cdd_logo.png|center]]
 
<center>
 
<center>
−
<h1>
+
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]
−
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]
+
−
</h1>
+
 
</center>
 
</center>
 +
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can
  
−
== Installation ==
+
* Write test cases
 +
* Manage test cases (using tags)
 +
* Run test cases
 +
* View test outcomes
 +
* '''Automatically extract test cases'''
  
−
=== Requirements ===
+
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum].  
−
* ISE Eiffel 5.7 (below version is recommended, other versions may or may not work)
+
−
** Linux-x86: http://se.ethz.ch/people/leitner/cdd/Eiffel57_gpl_64493-linux-x86.tar.gz
+
−
** Win32: http://se.ethz.ch/people/leitner/cdd/Eiffel57_gpl_64493-windows.msi
+
  
−
=== Install CDD-Extension for EiffelStudio ===
+
CDD EiffelStudio adds the following panel to regular EiffelStudio:
  
−
* Linux
+
[[Image:cdd_panel.png|center]]
−
** Delete the directory $HOME/.es
+
−
** Download http://se.ethz.ch/people/leitner/cdd/cdd_linux_x86_0_1_0.tgz
+
−
** Unpack it over your existing EiffelStudio installation. (You have done it right when you are overwriting a few files in the process.)
+
  
−
* Windows
 
−
** Start the registry editor (regedit.exe) and delete the key "HKEY_CURRENT_USER\Software\ISE\Eiffel57"
 
−
** Download http://se.ethz.ch/people/leitner/cdd/cdd_win32_0_1_0.zip
 
−
** Unpack it over your existing EiffelStudio installation. (You have done it right when you are overwriting a few files in the process.)
 
  
−
== How to Use ==
+
== Publications ==
−
Make sure you also have assertion checking enabled.
+
−
This can be done easily in: "Project->Project Settings..." select assertions and select "True" for all of them (the more the better cdd will work).
+
  
−
=== Step 1: Show Testing Window ===
+
Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", in Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf pdf] [http://se.ethz.ch/people/leitner/publications/ESEC_FSE2007.bib.txt bibtex]
  
−
If not already visible, you need to make sure the testing windows are all properly displayed. You can make them appear by following the menu "View->Tools->Testing":
+
Leitner, A., Pretschner A., Mori S., Meyer B., Oriol M., "On the Effectiveness of Test Extraction without Overhead", under submission
  
−
[[Image:show_testing.png]]
+
== News ==
 +
14.May 2008: Added patch for Windows and Linux that fixes a bug that can cause a crash during extraction.
  
−
=== Step 2: Enable CDD-Testing ===
+
10.April 2008: If installation exits with cryptic error codes, please have a look at the [[CDD Common Problems|Common Problems]] page.
−
To enable CDD-testing for your project, open your project and then enable CDD-testing, by clicking on the half-black / half-white button in the testing window. You can disable testing again at any time by pressing this button again.
+
  
−
[[Image:enable_testing.png]]
+
08.April 2008: CDD EiffelStudio final 7 available
 +
* Important bug fix: EiffelStudio no longer freezes when opening projects with many test cases
  
−
Now continue developing your project as usual.  
+
02.April 2008: CDD EiffelStudio Final 6 available
−
You will notice that for every exception that occurs a new test case shows up in the testing window. Test cases are automatically executed on each compilation to see whether you fixed that particular bug or not. A test case can be in one of three states:
+
* Less mess: Redundant or duplicate test cases are no longer extracted
 +
* Smoother upgrade: Cleaning a project will automatically clean the test suite
 +
* Fixed bug in invarant checking
 +
* Fixed bug that caused EiffelStudio to freeze in some situations
 +
* Improved logging
 +
* Installer ''might'' work with mingw on Windows now. (Not yet tested, reports welcome)
  
−
* "F" for failing - last time the test case was executed the original problem was reproduced
+
== Download CDD ==
−
* "OK" for passing - last time the test case was executed the original problem appeared fixed
+
−
* "INV" for invalid - the test case is out of date (it can no longer run your program in the expected way)
+
  
−
[[Image:test_case_states.png]]
+
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.
  
−
You can now delete test cases with the delete button and debug test cases with the debug button as shown below:
+
=== Linux ===
 +
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7.tar.bz2
 +
* Full Linux version (for Debian stable): http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7_debian_stable.tar.bz2
 +
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html
 +
Notes:
 +
* If you are '''upgrading''' from a previous version, make sure you delete the old version first (rm -rf)
  
−
[[Image:testing_buttons.png]]
+
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.
  
−
=== "bank_account" example walkthrough ===
+
=== Linux Patch ===
 +
* Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec and replace it with the '$ISE_EIFFEL/studio/spec/linux-x86/bin/ec' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.
  
−
TODO: ...
+
=== Windows ===
  
−
=== How Test Cases Are Extracted ===
+
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows.msi
 +
Notes:
 +
* Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those)
 +
* If you are '''upgrading''' from a previous version, you first have to uninstall the old version then delete the existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/) and then install the new version.
 +
* Which C compiler to install?
 +
** Use the Microsoft C compiler either from Visual Studio or the Windows SDK:
 +
*** Visual Studio (up to Visual Studio 2005, but no later, and only the non-express version)
 +
*** Windows SDK (up to version 6.0, but no later)
 +
** Do not use the gcc/mingw.
 +
** Do not use the .NET compiler backend.
  
−
TODO: ...
+
Have a look at http://dev.eiffel.com/Installing_Microsoft_C_compiler_6.1_and_older to learn how to install either Visual C++ or the Windows SDK for use with EiffelStudio.  
  
−
=== How to modify test cases by hand ===
 
  
−
TODO: ...
+
==== Windows Patch ====
  
−
=== Re-extracting test cases ===
+
* Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec.exe and replace it with the '$ISE_EIFFEL/studio/spec/windows/bin/ec.exe' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.
  
−
TODO: ...
+
== Documentation ==
  
−
== Release Notes ==
+
=== Manually Written Test Cases ===
 +
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.
  
−
* CDD can be quite fragile, if there you encounter problems you can do the following:
+
=== Extracted Test Cases ===
−
** Look into the testing-log-window (bottom half of screen) and see what went wrong
+
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.
−
** Remove the directory EIFGENs/cdd_tester from your projects directory.
+
−
** Disable precompiles if having problems
+
−
** Have a look at the test cases (you can just pick&drop them into the editor)
+
−
** Make sure your root cluster is not recursive.
+
−
** Do enable assertion checking!
+
  
 +
=== Test outcomes ===
 +
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.
  
  
−
== Source Code ==
+
[[Image:cdd_buttons.png|center]]
  
−
The source can be found on the cdd branch of the EiffelStudio svn repository (https://origo.ethz.ch/eiffelsoftware/es/branches/cdd/es).
+
=== Debug Test Case ===
−
After compiling it you need to do the following to create a delivery
+
Select a test case and press this button to run a test case in the debugger.
  
−
* Take an existing delivery (5.7 final should be fine)
+
=== Enable/Disable Execution of Test Cases in Background ===
−
* Copy the resulting executable ec.exe resp. ec to $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin/
+
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.
−
* Copy the the file Delivery/eifinit/studio/default.xml to $ISE_EIFFEL/eifinit/studio/default.xml
+
−
* Copy the directory Src/library/cdd to $ISE_EIFFEL/library/cdd
+
  
 +
=== Enable/Disable automatic extraction of Test Cases ===
 +
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.
  
−
== Development: TODO ==
+
=== Clean up/Delete ===
 +
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.
  
−
TODO:
+
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].
−
* Unit test suite for extraction
+
−
* Sample apps with errors
+
−
* Extract test case
+
−
* Compile and run to see if we reproduce failure
+
−
* Features under test, that are not exported to ANY are not supported.
+
−
* When an inv violation occurs after a feature has been executed, the debugger does not give access to routine args.
+
−
* Postcondition/Inv violation in creation procedures not covered yet?
+
−
* object_under_test.withdraw ("Unable to get the arguments") ? when does this happen?
+
−
* Test case result OK->Pass
+
−
* POINTER in test case -> default_pointer
+
−
* logging
+
−
* Capture Prestate
+
−
* ask for rerun (verification)
+
−
* delivery (linux/windows)
+
−
* better icons
+
  
−
* when not possible to disable cdd, gray-out enable/disable button
+
=== Create new manual test class ===
−
* be easy on the processor in the presence of many test cases
+
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:
−
* run relevant test cases first, show results as they tickle in
+
−
* give lower priority to bg testing
+
−
* always also capture prestate of routines down the stack -> new tc if current one goes invalid
+
−
* BackInTimeDebuggerLight
+
−
* Generate test suite root class already when enabling cdd (and compile test suite for the first time already too)
+
  
−
Things we need from estudio:
+
<eiffel>
−
* Invariant violation on creaton proc exit blames routine that contains create instruction not creation procedure
+
class TEST_BANK_ACCOUNT
−
* When a postcondition violation is thrown in a procedure that is also a creation procedure, we cannot don't know if current invokation was a creation or regular call.
+
inherit CDD_TEST_CASE
−
* When a violation occured in the invariant we don't know if this happened before or after the routine invokation
+
feature
−
* When a violation occured after the routine execution (post or inv) we would benefit from access to  the values of the old expressions used in the post of the called routine.
+
  test_deposit
 +
    local
 +
      ba: BANK_ACCOUNT
 +
    do
 +
      create ba.make_with_balance (0)
 +
      ba.deposit (100)
 +
      check
 +
        money_depisited: ba.balance = 100
 +
      end
 +
    end
 +
end
 +
</eiffel>
  
−
=== Design rework ===
+
Redefine routine `set_up' or `tear_down' if you want something to be executed before resp. after every test routine of a class.
−
* currently a test case knows if it is running or not.
+
 
−
** a test case should not know this (it could be executed by several processes)
+
[[Image:cdd_buttons_2.png|center]]
−
*** a bg tester is executing a test case
+
 
−
*** -> move status indication from test case to runner
+
=== Search for tags ===
−
* dont pour everything through output
+
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:
−
** compiler stuff
+
 
−
** test exe stuff
+
<eiffel>
−
* CDD_PRINTER, have a look at gobo printer for et
+
indexing
−
** specifically the variable indentation support
+
  tag: "fixme"
 +
class TEST_BANK_ACCOUNT
 +
inherit CDD_TEST_CASE
 +
feature
 +
  ...
 +
end
 +
</eiffel>
 +
 
 +
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.
 +
 
 +
=== Restrict execution of test cases ===
 +
 
 +
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.
 +
 
 +
=== Change test case view ===
 +
 
 +
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.
 +
 
 +
== Further Documentation and Common Problems ==
 +
Please visit [[Using CDD]] for further documentation or look at  
 +
[[CDD Common Problems|Common Problems]] if you run into problems.
 +
 
 +
== Related Publications ==
 +
 
 +
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]
 +
 
 +
== Project Internal Stuff ==
 +
 
 +
[[CddBranchInternal]]

Latest revision as of 06:23, 31 May 2010


What is the CDD EiffelStudio?

Cdd logo.png

Play CDD Video, click here

CDD (short for Contract Driven Development) is a project developed at ETH Zurich. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can

  • Write test cases
  • Manage test cases (using tags)
  • Run test cases
  • View test outcomes
  • Automatically extract test cases

If you have questions, feedback, or would like to report a bug please visit the CDD forum.

CDD EiffelStudio adds the following panel to regular EiffelStudio:

Cdd panel.png


Publications

Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", in Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 pdf bibtex

Leitner, A., Pretschner A., Mori S., Meyer B., Oriol M., "On the Effectiveness of Test Extraction without Overhead", under submission

News

14.May 2008: Added patch for Windows and Linux that fixes a bug that can cause a crash during extraction.

10.April 2008: If installation exits with cryptic error codes, please have a look at the Common Problems page.

08.April 2008: CDD EiffelStudio final 7 available

  • Important bug fix: EiffelStudio no longer freezes when opening projects with many test cases

02.April 2008: CDD EiffelStudio Final 6 available

  • Less mess: Redundant or duplicate test cases are no longer extracted
  • Smoother upgrade: Cleaning a project will automatically clean the test suite
  • Fixed bug in invarant checking
  • Fixed bug that caused EiffelStudio to freeze in some situations
  • Improved logging
  • Installer might work with mingw on Windows now. (Not yet tested, reports welcome)

Download CDD

The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.

Linux

Notes:

  • If you are upgrading from a previous version, make sure you delete the old version first (rm -rf)

Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.

Linux Patch

  • Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec and replace it with the '$ISE_EIFFEL/studio/spec/linux-x86/bin/ec' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.

Windows

Notes:

  • Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those)
  • If you are upgrading from a previous version, you first have to uninstall the old version then delete the existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/) and then install the new version.
  • Which C compiler to install?
    • Use the Microsoft C compiler either from Visual Studio or the Windows SDK:
      • Visual Studio (up to Visual Studio 2005, but no later, and only the non-express version)
      • Windows SDK (up to version 6.0, but no later)
    • Do not use the gcc/mingw.
    • Do not use the .NET compiler backend.

Have a look at http://dev.eiffel.com/Installing_Microsoft_C_compiler_6.1_and_older to learn how to install either Visual C++ or the Windows SDK for use with EiffelStudio.


Windows Patch

  • Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec.exe and replace it with the '$ISE_EIFFEL/studio/spec/windows/bin/ec.exe' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.

Documentation

Manually Written Test Cases

CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.

Extracted Test Cases

CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.

Test outcomes

A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.


Cdd buttons.png

Debug Test Case

Select a test case and press this button to run a test case in the debugger.

Enable/Disable Execution of Test Cases in Background

If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.

Enable/Disable automatic extraction of Test Cases

If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.

Clean up/Delete

You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.

Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the clean-cdd project.

Create new manual test class

Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like: 

class TEST_BANK_ACCOUNT
inherit CDD_TEST_CASE
feature
  test_deposit
    local
      ba: BANK_ACCOUNT
    do
      create ba.make_with_balance (0)
      ba.deposit (100)
      check
         money_depisited: ba.balance = 100
      end
    end
end

Redefine routine `set_up' or `tear_down' if you want something to be executed before resp. after every test routine of a class.

Cdd buttons 2.png

Search for tags

Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine: 

indexing
  tag: "fixme"
class TEST_BANK_ACCOUNT
inherit CDD_TEST_CASE
feature
   ...
end

Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.

Restrict execution of test cases

Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.

Change test case view

Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.

Further Documentation and Common Problems

Please visit Using CDD for further documentation or look at Common Problems if you run into problems.

Related Publications

  • Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. (pdf)

Project Internal Stuff

CddBranchInternal

Retrieved from "https://dev.eiffel.com/index.php?title=CddBranch&oldid=13869"