Difference between revisions of "CddBranch"

(→‎Source Code)
(→‎Development:)
Line 100: Line 100:
 
NOTE: To compile the Eiffel_57 version of cdd, you will need the latest 5.7 release of estudio. Same for the Eiffel_60 version, there you will need the latest 6.0 release of estudio.
 
NOTE: To compile the Eiffel_57 version of cdd, you will need the latest 5.7 release of estudio. Same for the Eiffel_60 version, there you will need the latest 6.0 release of estudio.
  
−
== Development: ==
+
== Development (Todo's): ==
  
−
=== Plan ===
+
=== CDD_MANAGER class ===
  
−
* Find out whether once fields are writable
+
* Improve interaction between estudio and the current test suite (violation notification, etc...)
−
* Manual test case integration
+
* Improve logging and status handling
−
* New test case representation
+
−
** Tags (todo-tag)
+
−
*** dynamic grouping of test cases
+
−
** context as string
+
−
* Be ready for capture/replay
+
−
* Extract test suite for failure (not only test case)
+
−
* Allow extraction of test suites for passing routine applications
+
  
−
=== Bugs ===
+
=== Capturing ===
  
−
* Tests generated for infix features do not compile
+
* Use cdd reflection and test case creation classes from Eiffel_57 branch
 +
* Save context data in test case class as string (XML format?)
 +
* Create test case for each routine in current call stack (attach test cases through a linked list)
 +
** Only reflect the state of routines which are exported to ANY
 +
* Add option for extracting context of non failing routines
 +
* Handle postconditions/invariants within creation procedures
 +
* POINTERS?
 +
* restore once values
  
−
=== TODO: ===
+
=== GUI (Tool) ===
−
* check invariant of all objects in the input space (not only the target object) before the fut is executed
+
−
* Use re-extraction stuff to allow extraction of test cases from non-failing runs.
+
−
* Unit test suite for extraction
+
−
** Sample apps with errors
+
−
** Extract test case
+
−
** Compile and run to see if we reproduce failure
+
−
* Features under test, that are not exported to ANY are not supported.
+
−
** Create test case for first exported routine down the stack instead.
+
−
* When an inv violation occurs after a feature has been executed, the debugger does not give access to routine args.
+
−
* Postcondition/Inv violation in creation procedures not covered yet
+
−
* Test case result OK->Pass
+
−
* POINTER/TYPED_POINTER in test case -> default_pointer,
+
−
* logging
+
−
* Capture Prestate
+
−
** ask for rerun (verification)
+
−
* delivery (linux/windows)
+
−
* better icons
+
−
* quick fix support for failing test cases
+
  
−
* when not possible to disable cdd, gray-out enable/disable button
+
* Improve Model/View abstraction for displaying test suite (especially for the GRID)
−
* be easy on the processor in the presence of many test cases
+
* New views (through tabs):
−
** run relevant test cases first, show results as they tickle in
+
** Cluster view (with option for only showing failing test cases)
−
** give lower priority to bg testing
+
** Call stack view
−
* always also capture prestate of routines down the stack -> new tc if current one goes invalid
+
−
* BackInTimeDebuggerLight
+
−
* Generate test suite root class already when enabling cdd (and compile test suite for the first time already too)
+
−
* When debugging a test case, the test case should stop at the begining to let the user set breakpoints.
+
−
* What about global state (once!)?
+
  
−
=== Things we need from estudio: ===
+
=== Execution (Background) ===
−
* Invariant violation on creaton proc exit blames routine that contains create instruction not creation procedure
+
 
−
* When a postcondition violation is thrown in a procedure that is also a creation procedure, we cannot don't know if current invokation was a creation or regular call.
+
* Parse context string in class file for rebuilding state
−
* When a violation occured in the invariant we don't know if this happened before or after the routine invokation
+
* check invariants of all objects in context before running fut
−
* When a violation occured after the routine execution (post or inv) we would benefit from access to the values of the old expressions used in the post of the called routine.
+
* run test cases depending on processor load (run relevant test cases first)
 +
* compile tester after setting up cdd in a new project (speeds up first test execution)
 +
 
 +
 
 +
=== Execution (Debugger) ===
 +
 
 +
* Set breakpoint at feature beginning before executing (so user can actually debug the featue)
 +
 
 +
 
 +
=== GUI (Context) ===
 +
 
 +
* Clickable Compiler/Tester output
 +
 
 +
=== GUI (Refinements) ===
 +
 
 +
* New icons!
 +
 
 +
=== Things we need from estudio ===
 +
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)
 +
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)
 
* Support for multiple open targets
 
* Support for multiple open targets

Revision as of 09:37, 22 March 2007

Summary

CDD is an extension to EiffelStudio that provides support for both automatic test case extraction and test execution. It is undistruptive to the development process of the user. The tool is in its earlys stages. Feedback and bug reports are very much appreciated. Please send them to mailto:andreas.leitner@inf.ethz.ch and or use the discussion page for this article.


Video still.png

Play Video!

Installation

Requirements

Install CDD-Extension for EiffelStudio

  • Windows
    • Start the registry editor (regedit.exe) and delete the key "HKEY_CURRENT_USER\Software\ISE\Eiffel57"
    • Download http://se.ethz.ch/people/leitner/cdd/cdd_win32_0_2_1.zip
    • Unpack it over your existing EiffelStudio installation. (You have done it right when you are overwriting a few files in the process.)

How to Use

Make sure you also have assertion checking enabled. This can be done easily in: "Project->Project Settings..." select assertions and select "True" for all of them (the more the better cdd will work).

Step 1: Show Testing Window

If not already visible, you need to make sure the testing windows are all properly displayed. You can make them appear by following the menu "View->Tools->Testing":

Show testing.png

Step 2: Enable CDD-Testing

To enable CDD-testing for your project, open your project and then enable CDD-testing, by clicking on the half-black / half-white button in the testing window. You can disable testing again at any time by pressing this button again.

Enable testing.png

Now continue developing your project as usual. You will notice that for every exception that occurs a new test case shows up in the testing window. Test cases are automatically executed on each compilation to see whether you fixed that particular bug or not. A test case can be in one of three states:

  • "F" for failing - last time the test case was executed the original problem was reproduced
  • "OK" for passing - last time the test case was executed the original problem appeared fixed
  • "INV" for invalid - the test case is out of date (it can no longer run your program in the expected way)

Test case states.png

You can now delete test cases with the delete button and debug test cases with the debug button as shown below:

Testing buttons.png

"bank_account" example walkthrough

TODO: ...

How Test Cases Are Extracted

TODO: ...

How to modify test cases by hand

TODO: ...

Re-extracting test cases

TODO: ...

Release Notes

  • CDD can be quite fragile, if there you encounter problems you can do the following:
    • Look into the testing-log-window (bottom half of screen) and see what went wrong
    • Remove the directory EIFGENs/cdd_tester from your projects directory.
    • Disable precompiles if having problems
    • Have a look at the test cases (you can just pick&drop them into the editor)
    • Do enable assertion checking!
    • Make sure you don't open your project with the 'cdd_tester' target, this target is only used internaly.

Source Code

The source of the last stable version can be found on the cdd branch of the EiffelStudio svn repository (https://origo.ethz.ch/eiffelsoftware/es/branches/cdd/Eiffel_57). After compiling it you need to do the following to create a delivery

  • Take an existing delivery (5.7 final should be fine)
  • Copy the resulting executable ec.exe resp. ec to $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin/
  • Copy the the file Delivery/eifinit/studio/default.xml to $ISE_EIFFEL/eifinit/studio/default.xml
  • Copy the directory Src/library/cdd to $ISE_EIFFEL/library/cdd

If you would like to have a look at the latest source use https://origo.ethz.ch/eiffelsoftware/es/branches/cdd/Eiffel_60. Attention: this code is still under developement. Most functionality is not implemented yet.

NOTE: To compile the Eiffel_57 version of cdd, you will need the latest 5.7 release of estudio. Same for the Eiffel_60 version, there you will need the latest 6.0 release of estudio.

Development (Todo's):

CDD_MANAGER class

  • Improve interaction between estudio and the current test suite (violation notification, etc...)
  • Improve logging and status handling

Capturing

  • Use cdd reflection and test case creation classes from Eiffel_57 branch
  • Save context data in test case class as string (XML format?)
  • Create test case for each routine in current call stack (attach test cases through a linked list)
    • Only reflect the state of routines which are exported to ANY
  • Add option for extracting context of non failing routines
  • Handle postconditions/invariants within creation procedures
  • POINTERS?
  • restore once values

GUI (Tool)

  • Improve Model/View abstraction for displaying test suite (especially for the GRID)
  • New views (through tabs):
    • Cluster view (with option for only showing failing test cases)
    • Call stack view

Execution (Background)

  • Parse context string in class file for rebuilding state
  • check invariants of all objects in context before running fut
  • run test cases depending on processor load (run relevant test cases first)
  • compile tester after setting up cdd in a new project (speeds up first test execution)


Execution (Debugger)

  • Set breakpoint at feature beginning before executing (so user can actually debug the featue)


GUI (Context)

  • Clickable Compiler/Tester output

GUI (Refinements)

  • New icons!

Things we need from estudio

  • Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)
  • The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)
  • Support for multiple open targets
Retrieved from "https://dev.eiffel.com/index.php?title=CddBranch&oldid=7694"