Revision as of 07:41, 22 March 2007

Summary

CDD is an extension to EiffelStudio that provides support for both automatic test case extraction and test execution. It is undistruptive to the development process of the user. The tool is in its earlys stages. Feedback and bug reports are very much appreciated. Please send them to mailto:andreas.leitner@inf.ethz.ch and or use the discussion page for this article.

Installation

Requirements

• ISE Eiffel 5.7 (below version is recommended, other versions may or may not work)
  • Linux-x86: http://se.ethz.ch/people/leitner/cdd/Eiffel57_gpl_64493-linux-x86.tar.gz
  • Win32: http://se.ethz.ch/people/leitner/cdd/Eiffel57_gpl_64493-windows.msi

Install CDD-Extension for EiffelStudio

• Linux
  • Delete the directory $HOME/.es
  • Download http://se.ethz.ch/people/leitner/cdd/cdd_linux_x86_0_2_0.tgz
  • Unpack it over your existing EiffelStudio installation. (You have done it right when you are overwriting a few files in the process.)

• Windows
  • Start the registry editor (regedit.exe) and delete the key "HKEY_CURRENT_USER\Software\ISE\Eiffel57"
  • Download http://se.ethz.ch/people/leitner/cdd/cdd_win32_0_2_1.zip
  • Unpack it over your existing EiffelStudio installation. (You have done it right when you are overwriting a few files in the process.)

How to Use

Make sure you also have assertion checking enabled. This can be done easily in: "Project->Project Settings..." select assertions and select "True" for all of them (the more the better cdd will work).

Step 1: Show Testing Window

If not already visible, you need to make sure the testing windows are all properly displayed. You can make them appear by following the menu "View->Tools->Testing":

Step 2: Enable CDD-Testing

To enable CDD-testing for your project, open your project and then enable CDD-testing, by clicking on the half-black / half-white button in the testing window. You can disable testing again at any time by pressing this button again.

Now continue developing your project as usual. You will notice that for every exception that occurs a new test case shows up in the testing window. Test cases are automatically executed on each compilation to see whether you fixed that particular bug or not. A test case can be in one of three states:

• "F" for failing - last time the test case was executed the original problem was reproduced
• "OK" for passing - last time the test case was executed the original problem appeared fixed
• "INV" for invalid - the test case is out of date (it can no longer run your program in the expected way)

You can now delete test cases with the delete button and debug test cases with the debug button as shown below:

"bank_account" example walkthrough

TODO: ...

How Test Cases Are Extracted

TODO: ...

How to modify test cases by hand

TODO: ...

Re-extracting test cases

TODO: ...

Release Notes

• CDD can be quite fragile, if there you encounter problems you can do the following:
  • Look into the testing-log-window (bottom half of screen) and see what went wrong
  • Remove the directory EIFGENs/cdd_tester from your projects directory.
  • Disable precompiles if having problems
  • Have a look at the test cases (you can just pick&drop them into the editor)
  • Do enable assertion checking!
  • Make sure you don't open your project with the 'cdd_tester' target, this target is only used internaly.

Source Code

The source of the last stable version can be found on the cdd branch of the EiffelStudio svn repository (https://origo.ethz.ch/eiffelsoftware/es/branches/cdd/Eiffel_57). After compiling it you need to do the following to create a delivery

• Take an existing delivery (5.7 final should be fine)
• Copy the resulting executable ec.exe resp. ec to $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin/
• Copy the the file Delivery/eifinit/studio/default.xml to $ISE_EIFFEL/eifinit/studio/default.xml
• Copy the directory Src/library/cdd to $ISE_EIFFEL/library/cdd

Development:

Plan

• Find out whether once fields are writable
• Manual test case integration
• New test case representation
  • Tags (todo-tag)
    • dynamic grouping of test cases
  • context as string
• Be ready for capture/replay
• Extract test suite for failure (not only test case)
• Allow extraction of test suites for passing routine applications

Bugs

• Tests generated for infix features do not compile

TODO:

• check invariant of all objects in the input space (not only the target object) before the fut is executed
• Use re-extraction stuff to allow extraction of test cases from non-failing runs.
• Unit test suite for extraction
  • Sample apps with errors
  • Extract test case
  • Compile and run to see if we reproduce failure
• Features under test, that are not exported to ANY are not supported.
  • Create test case for first exported routine down the stack instead.
• When an inv violation occurs after a feature has been executed, the debugger does not give access to routine args.
• Postcondition/Inv violation in creation procedures not covered yet
• Test case result OK->Pass
• POINTER/TYPED_POINTER in test case -> default_pointer,
• logging
• Capture Prestate
  • ask for rerun (verification)
• delivery (linux/windows)
• better icons
• quick fix support for failing test cases

• when not possible to disable cdd, gray-out enable/disable button
• be easy on the processor in the presence of many test cases
  • run relevant test cases first, show results as they tickle in
  • give lower priority to bg testing
• always also capture prestate of routines down the stack -> new tc if current one goes invalid
• BackInTimeDebuggerLight
• Generate test suite root class already when enabling cdd (and compile test suite for the first time already too)
• When debugging a test case, the test case should stop at the begining to let the user set breakpoints.
• What about global state (once!)?

Things we need from estudio:

• Invariant violation on creaton proc exit blames routine that contains create instruction not creation procedure
• When a postcondition violation is thrown in a procedure that is also a creation procedure, we cannot don't know if current invokation was a creation or regular call.
• When a violation occured in the invariant we don't know if this happened before or after the routine invokation
• When a violation occured after the routine execution (post or inv) we would benefit from access to the values of the old expressions used in the post of the called routine.
• Support for multiple open targets