https://dev.eiffel.com/api.php?action=feedcontributions&user=Mogh&feedformat=atomEiffelStudio: an EiffelSoftware project - User contributions [en]2026-04-13T07:52:49ZUser contributionsMediaWiki 1.24.1https://dev.eiffel.com/index.php?title=CddBranch&diff=11059CddBranch2008-05-14T13:45:22Z<p>Mogh: /* News */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== News ==<br />
14.May 2008: Added patch for Windows and Linux that fixes a bug that can cause a crash during extraction.<br />
<br />
10.April 2008: If installation exits with cryptic error codes, please have a look at the [[CDD Common Problems|Common Problems]] page.<br />
<br />
08.April 2008: CDD EiffelStudio final 7 available<br />
* Important bug fix: EiffelStudio no longer freezes when opening projects with many test cases<br />
<br />
02.April 2008: CDD EiffelStudio Final 6 available<br />
* Less mess: Redundant or duplicate test cases are no longer extracted<br />
* Smoother upgrade: Cleaning a project will automatically clean the test suite<br />
* Fixed bug in invarant checking<br />
* Fixed bug that caused EiffelStudio to freeze in some situations<br />
* Improved logging<br />
* Installer ''might'' work with mingw on Windows now. (Not yet tested, reports welcome)<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7.tar.bz2<br />
* Full Linux version (for Debian stable): http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7_debian_stable.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
Notes:<br />
* If you are '''upgrading''' from a previous version, make sure you delete the old version first (rm -rf)<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Linux Patch ===<br />
* Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec and replace it with the '$ISE_EIFFEL/studio/spec/linux-x86/bin/ec' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows.msi<br />
Notes:<br />
* Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* If you are '''upgrading''' from a previous version, you first have to uninstall the old version then delete the existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/) and then install the new version.<br />
* Which C compiler to install?<br />
** Use the Microsoft C compiler either from Visual Studio or the Windows SDK:<br />
*** Visual Studio (up to Visual Studio 2005, but no later, and only the non-express version)<br />
*** Windows SDK (up to version 6.0, but no later)<br />
** Do not use the gcc/mingw.<br />
** Do not use the .NET compiler backend. <br />
<br />
Have a look at http://dev.eiffel.com/Installing_Microsoft_C_compiler_6.1_and_older to learn how to install either Visual C++ or the Windows SDK for use with EiffelStudio. <br />
<br />
<br />
==== Windows Patch ====<br />
<br />
* Patch version for existing EiffelStudio CDD Edition Final 5 or Final 6 installation (or final 7, fixes bug that causes crash under uncommon circumstances): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows_patch_01.zip<br />
* does NOT work for official EiffelStudio 6.1 installation (or pre-final 5 versions of CDD Edition)! <br />
* Replace <INSTALL_DIRECTORY>/studio/spec/windows/bin/ec.exe with ec.exe contained in archive<br />
* Delete existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/)<br />
* Recompile existing projects from scratch (use the "clean" option in the project load screen)<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
Redefine routine `set_up' or `tear_down' if you want something to be executed before resp. after every test routine of a class.<br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10997CddBranch2008-04-28T15:03:20Z<p>Mogh: /* Windows Patch */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== News ==<br />
10.April.2008: If installation exits with cryptic error codes, please have a look at the [[CDD Common Problems|Common Problems]] page.<br />
<br />
8.April.2008: CDD EiffelStudio final 7 available<br />
* Important bug fix: EiffelStudio no longer freezes when opening projects with many test cases<br />
<br />
2.April.2008: CDD EiffelStudio Final 6 available<br />
* Less mess: Redundant or duplicate test cases are no longer extracted<br />
* Smoother upgrade: Cleaning a project will automatically clean the test suite<br />
* Fixed bug in invarant checking<br />
* Fixed bug that caused EiffelStudio to freeze in some situations<br />
* Improved logging<br />
* Installer ''might'' work with mingw on Windows now. (Not yet tested, reports welcome)<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7.tar.bz2<br />
* Full Linux version (for Debian stable): http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7_debian_stable.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
Notes:<br />
* If you are '''upgrading''' from a previous version, make sure you delete the old version first (rm -rf)<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Linux Patch ===<br />
* Patch for CDD EiffelStudio Final 7: Download http://se.ethz.ch/people/leitner/cdd/ec and replace it with the '$ISE_EIFFEL/studio/spec/linux-x86/bin/ec' from Final 7. This patch fixes a crash that sometimes occurs during extraction of test cases. You may need to delete the precompile and EIFGENs of your project after applying the patch.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows.msi<br />
Notes:<br />
* Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* If you are '''upgrading''' from a previous version, you first have to uninstall the old version then delete the existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/) and then install the new version.<br />
* Which C compiler to install?<br />
** Use the Microsoft C compiler either from Visual Studio or the Windows SDK:<br />
*** Visual Studio (up to Visual Studio 2005, but no later, and only the non-express version)<br />
*** Windows SDK (up to version 6.0, but no later)<br />
** Do not use the gcc/mingw.<br />
** Do not use the .NET compiler backend. <br />
<br />
Have a look at http://dev.eiffel.com/Installing_Microsoft_C_compiler_6.1_and_older to learn how to install either Visual C++ or the Windows SDK for use with EiffelStudio. <br />
<br />
<br />
==== Windows Patch ====<br />
<br />
* Patch version for existing EiffelStudio CDD Edition Final 5 or Final 6 installation (or final 7, fixes bug that causes crash under uncommon circumstances): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows_patch_01.zip<br />
* does NOT work for official EiffelStudio 6.1 installation (or pre-final 5 versions of CDD Edition)! <br />
* Replace <INSTALL_DIRECTORY>/studio/spec/windows/bin/ec.exe with ec.exe contained in archive<br />
* Delete existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/)<br />
* Recompile existing projects from scratch (use the "clean" option in the project load screen)<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
Redefine routine `set_up' or `tear_down' if you want something to be executed before resp. after every test routine of a class.<br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CDD_Common_Problems&diff=10950CDD Common Problems2008-04-15T09:21:21Z<p>Mogh: </p>
<hr />
<div>[[Category:CDD]]<br />
<br />
Problem: ''''My system suddenly has the wrong root class / doesn't compile anymore because it doesn't find the strange wrong root class''''<br />
* This issue arises if EiffelStudio crashes / is killed during the (foreground) debugging of a test case (via the corresponding button in the testing tool). On Windows, crashes should no happen, but there is an issue on Linux with the foreground debugging of test cases we haven't been able to solve. If you are debugging a test case that passes, OR you are debugging a test case that fails, but you press "Continue" after the exception occured (instead of "Stop", to end the execution immediately), EiffelStudio will crash.<br />
* Solution: You have to manually set the root class of the system back to the correct class (via Project Settings, or by directly editing the .ecf file)<br />
* Prevention: On Linux, use test case debugging ONLY for FAILING test cases, and after exception occured, STOP the execution (with the "Stop" button)!<br />
<br />
<br />
Problem: ''''My test cases time out''''<br />
* CDD automatically aborts test cases that run longer than a given period of time. The default value (which is a few seconds) can be changed by setting the environment variable ''''CDD_TESTER_TIMEOUT''''. On Linux ''''"export CDD_TESTER_TIMEOUT=120"'''' sets the timeout to 120 seconds. On Windows the same thing is acomplished with ''''"set CDD_TESTER_TIMEOUT=120"''''.<br />
<br />
Problem: '''Installing EiffelStudio reports error codes like '2908' and '2909' '''<br />
* This seems to be a rare problem. The following might help<br />
<blockquote><br />
<<<br />
I finally found the solution. Have a look at the following URLs:<br />
- http://tinyurl.com/2rj93p<br />
- http://tinyurl.com/3ey6fe<br />
<br />
I believe you should have the same issue. In my case, I think it is because<br />
I<br />
had just installed the .NET 3.0 runtime which is part of the Windows Update.<br />
<br />
If you rename the `Components' registry key, assuming that like me you only<br />
have one child key, the installation should progress fine.<br />
>><br />
<br />
This often happens when uninstalling and then reinstalling EiffelStudio<br />
although it is clearly not a bug in our installer.<br />
<br />
</blockquote><br />
<br />
<br />
Problem: ''''I cannot see the testing window.''''<br />
* Make it visible via ''''View -> Tools -> CDD Output''''<br />
<br />
Problem: ''''Is there some kind of CDD log window?''''<br />
* Yes there is it is called ''''CDD Output''''. If you don't see this window, you can make it visible via ''''"View -> Tools -> CDD Output"''''<br />
<br />
Problem: ''''I am getting a duplicate class error'''' or ''''I am trying to use CDD on an existing project but it doesn't work''''<br />
* If you have a look at the ''''CDD output'''' window you will probably find the error message:<br />
<br />
<pre><br />
Error code: VSCN<br />
Configuration error: cluster has two classes with the same name.<br />
What to do: if both classes are needed, change name of one of them.<br />
<br />
Cluster name: erl_g_tests<br />
First class: CDD_INTERPRETER<br />
First file: "/home/aleitner/src/erl_g/src/erl_g/cdd_tests/erl_g/cdd_interpreter.e"<br />
Second class: CDD_INTERPRETER<br />
Second file: "/home/aleitner/src/erl_g/library/cdd_tests/erl_g_library/cdd_interpreter.e"<br />
</pre><br />
<br />
To fix this go to the project settings, to your active target, '''Groups -> Clusters''' and then select your root cluster. Open the '''Advanced''' tree-item and click on the value part of the '''Exclude Rules'''. Now add rule ''''/cdd_tests$'''' and click ''''OK''''. <br />
<br />
Alternatively you can open your ecf file with a text editor. You will find a stance that looks like this:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Edit edit to make it looks like the following:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
<exclude>/cdd_tests$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Problem: ''''I have written or extracted a test case. How can I execute it?''''<br />
* All tests are executed in the background right after compiling automatically. You can choose to enable or disable background testing via the ''''Enable/Disable automatic background execution of tests''''.<br />
<br />
[[Image:enable_testing.png|center]]<br />
<br />
Problem: ''''The environment is unstable when using the .Net backend.''''<br />
Solution: CDD does currently not support the .Net backend. Please use the C backend.<br />
<br />
<br />
Problem: ''''How to disable the CDD extension in EiffelStudio''''<br />
Solution: Make sure the push-buttons for Extraction and Execution are not pressed. The buttons are shown in the below picture.<br />
[[Image:enable_exec_and_extract.png|center]]<br />
<br />
<br />
Problem: ''''My extracted test cases get overwritten in the SVN Repository, because my colleagues commit their extracted test cases which have the same name.''''<br />
* Solution: Define a new environment variable "CDD_TESTER_ID". The value of the variable will be used as suffix for all extracted test class names. So if all members of the project define a unique "CDD_TESTER_ID", the individually extracted test cases can be commited to the svn without collisions.<br />
* ATTENTION: The value of CDD_TESTER_ID has to conform to a valid Eiffel class name, e.g. it has to consist of (uppercase) letters and '_' only!<br />
* EXAMPLE: You define CDD_TESTER_ID and set it to value "MARKUS". Each time a new test case gets exracted for a class ROOT_CLASS, the generated class name for the test case will be CDD_TEST_ROOT_CLASS_xyz_MARKUS. Likewise it will get stored in a file called cdd_test_root_class_xyz_markus.e</div>Moghhttps://dev.eiffel.com/index.php?title=CDD_Common_Problems&diff=10929CDD Common Problems2008-04-09T08:48:02Z<p>Mogh: </p>
<hr />
<div>[[Category:CDD]]<br />
<br />
Problem: ''''My test cases time out''''<br />
* CDD automatically aborts test cases that run longer than a given period of time. The default value (which is a few seconds) can be changed by setting the environment variable ''''CDD_TESTER_TIMEOUT''''. On Linux ''''"export CDD_TESTER_TIMEOUT=120"'''' sets the timeout to 120 seconds. On Windows the same thing is acomplished with ''''"set CDD_TESTER_TIMEOUT=120"''''.<br />
<br />
Problem: ''''I cannot see the testing window.''''<br />
* Make it visible via ''''View -> Tools -> CDD Output''''<br />
<br />
Problem: ''''Is there some kind of CDD log window?''''<br />
* Yes there is it is called ''''CDD Output''''. If you don't see this window, you can make it visible via ''''"View -> Tools -> CDD Output"''''<br />
<br />
Problem: ''''I am getting a duplicate class error'''' or ''''I am trying to use CDD on an existing project but it doesn't work''''<br />
* If you have a look at the ''''CDD output'''' window you will probably find the error message:<br />
<br />
<pre><br />
Error code: VSCN<br />
Configuration error: cluster has two classes with the same name.<br />
What to do: if both classes are needed, change name of one of them.<br />
<br />
Cluster name: erl_g_tests<br />
First class: CDD_INTERPRETER<br />
First file: "/home/aleitner/src/erl_g/src/erl_g/cdd_tests/erl_g/cdd_interpreter.e"<br />
Second class: CDD_INTERPRETER<br />
Second file: "/home/aleitner/src/erl_g/library/cdd_tests/erl_g_library/cdd_interpreter.e"<br />
</pre><br />
<br />
To fix this go to the project settings, to your active target, '''Groups -> Clusters''' and then select your root cluster. Open the '''Advanced''' tree-item and click on the value part of the '''Exclude Rules'''. Now add rule ''''/cdd_tests$'''' and click ''''OK''''. <br />
<br />
Alternatively you can open your ecf file with a text editor. You will find a stance that looks like this:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Edit edit to make it looks like the following:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
<exclude>/cdd_tests$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Problem: ''''I have written or extracted a test case. How can I execute it?''''<br />
* All tests are executed in the background right after compiling automatically. You can choose to enable or disable background testing via the ''''Enable/Disable automatic background execution of tests''''.<br />
<br />
[[Image:enable_testing.png|center]]<br />
<br />
Problem: ''''The environment is unstable when using the .Net backend.''''<br />
Solution: CDD does currently not support the .Net backend. Please use the C backend.<br />
<br />
<br />
Problem: ''''How to disable the CDD extension in EiffelStudio''''<br />
Solution: Make sure the push-buttons for Extraction and Execution are not pressed. The buttons are shown in the below picture.<br />
[[Image:enable_exec_and_extract.png|center]]<br />
<br />
<br />
Problem: ''''My extracted test cases get overwritten in the SVN Repository, because my colleagues commit their extracted test cases which have the same name.''''<br />
* Solution: Define a new environment variable "CDD_TESTER_ID". The value of the variable will be used as suffix for all extracted test class names. So if all members of the project define a unique "CDD_TESTER_ID", the individually extracted test cases can be commited to the svn without collisions.<br />
* ATTENTION: The value of CDD_TESTER_ID has to conform to a valid Eiffel class name, e.g. it has to consist of (uppercase) letters and '_' only!<br />
* EXAMPLE: You define CDD_TESTER_ID and set it to value "MARKUS". Each time a new test case gets exracted for a class ROOT_CLASS, the generated class name for the test case will be CDD_TEST_ROOT_CLASS_xyz_MARKUS. Likewise it will get stored in a file called cdd_test_root_class_xyz_markus.e</div>Moghhttps://dev.eiffel.com/index.php?title=CDD_Common_Problems&diff=10928CDD Common Problems2008-04-09T08:46:14Z<p>Mogh: </p>
<hr />
<div>[[Category:CDD]]<br />
<br />
Problem: ''''My test cases time out''''<br />
* CDD automatically aborts test cases that run longer than a given period of time. The default value (which is a few seconds) can be changed by setting the environment variable ''''CDD_TESTER_TIMEOUT''''. On Linux ''''"export CDD_TESTER_TIMEOUT=120"'''' sets the timeout to 120 seconds. On Windows the same thing is acomplished with ''''"set CDD_TESTER_TIMEOUT=120"''''.<br />
<br />
Problem: ''''I cannot see the testing window.''''<br />
* Make it visible via ''''View -> Tools -> CDD Output''''<br />
<br />
Problem: ''''Is there some kind of CDD log window?''''<br />
* Yes there is it is called ''''CDD Output''''. If you don't see this window, you can make it visible via ''''"View -> Tools -> CDD Output"''''<br />
<br />
Problem: ''''I am getting a duplicate class error'''' or ''''I am trying to use CDD on an existing project but it doesn't work''''<br />
* If you have a look at the ''''CDD output'''' window you will probably find the error message:<br />
<br />
<pre><br />
Error code: VSCN<br />
Configuration error: cluster has two classes with the same name.<br />
What to do: if both classes are needed, change name of one of them.<br />
<br />
Cluster name: erl_g_tests<br />
First class: CDD_INTERPRETER<br />
First file: "/home/aleitner/src/erl_g/src/erl_g/cdd_tests/erl_g/cdd_interpreter.e"<br />
Second class: CDD_INTERPRETER<br />
Second file: "/home/aleitner/src/erl_g/library/cdd_tests/erl_g_library/cdd_interpreter.e"<br />
</pre><br />
<br />
To fix this go to the project settings, to your active target, '''Groups -> Clusters''' and then select your root cluster. Open the '''Advanced''' tree-item and click on the value part of the '''Exclude Rules'''. Now add rule ''''/cdd_tests$'''' and click ''''OK''''. <br />
<br />
Alternatively you can open your ecf file with a text editor. You will find a stance that looks like this:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Edit edit to make it looks like the following:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
<exclude>/cdd_tests$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Problem: ''''I have written or extracted a test case. How can I execute it?''''<br />
* All tests are executed in the background right after compiling automatically. You can choose to enable or disable background testing via the ''''Enable/Disable automatic background execution of tests''''.<br />
<br />
[[Image:enable_testing.png|center]]<br />
<br />
Problem: ''''The environment is unstable when using the .Net backend.''''<br />
Solution: CDD does currently not support the .Net backend. Please use the C backend.<br />
<br />
<br />
Problem: ''''How to disable the CDD extension in EiffelStudio''''<br />
Solution: Make sure the push-buttons for Extraction and Execution are not pressed. The buttons are shown in the below picture.<br />
[[Image:enable_exec_and_extract.png|center]]<br />
<br />
<br />
Problem: ''''My extracted test cases get overwritten in the SVN Repository, because my colleagues commit their extracted test cases which have the same name.''''<br />
Solution: Define a new environment variable "CDD_TESTER_ID". The value of the variable will be used as suffix for all extracted test class names. So if all members of the project define a unique "CDD_TESTER_ID", the individually extracted test cases can be commited to the svn without collisions.<br />
ATTENTION: The value of CDD_TESTER_ID has to conform to a valid Eiffel class name, e.g. it has to consist of (uppercase) letters and '_' only!<br />
EXAMPLE: You define CDD_TESTER_ID and set it to value "MARKUS". Each time a new test case gets exracted for a class ROOT_CLASS, the generated class name for the test case will be CDD_TEST_ROOT_CLASS_xyz_MARKUS. Likewise it will get stored in a file called cdd_test_root_class_xyz_markus.e</div>Moghhttps://dev.eiffel.com/index.php?title=CDD_Common_Problems&diff=10927CDD Common Problems2008-04-09T08:39:27Z<p>Mogh: </p>
<hr />
<div>[[Category:CDD]]<br />
<br />
Problem: ''''My test cases time out''''<br />
* CDD automatically aborts test cases that run longer than a given period of time. The default value (which is a few seconds) can be changed by setting the environment variable ''''CDD_TESTER_TIMEOUT''''. On Linux ''''"export CDD_TESTER_TIMEOUT=120"'''' sets the timeout to 120 seconds. On Windows the same thing is acomplished with ''''"set CDD_TESTER_TIMEOUT=120"''''.<br />
<br />
Problem: ''''I cannot see the testing window.''''<br />
* Make it visible via ''''View -> Tools -> CDD Output''''<br />
<br />
Problem: ''''Is there some kind of CDD log window?''''<br />
* Yes there is it is called ''''CDD Output''''. If you don't see this window, you can make it visible via ''''"View -> Tools -> CDD Output"''''<br />
<br />
Problem: ''''I am getting a duplicate class error'''' or ''''I am trying to use CDD on an existing project but it doesn't work''''<br />
* If you have a look at the ''''CDD output'''' window you will probably find the error message:<br />
<br />
<pre><br />
Error code: VSCN<br />
Configuration error: cluster has two classes with the same name.<br />
What to do: if both classes are needed, change name of one of them.<br />
<br />
Cluster name: erl_g_tests<br />
First class: CDD_INTERPRETER<br />
First file: "/home/aleitner/src/erl_g/src/erl_g/cdd_tests/erl_g/cdd_interpreter.e"<br />
Second class: CDD_INTERPRETER<br />
Second file: "/home/aleitner/src/erl_g/library/cdd_tests/erl_g_library/cdd_interpreter.e"<br />
</pre><br />
<br />
To fix this go to the project settings, to your active target, '''Groups -> Clusters''' and then select your root cluster. Open the '''Advanced''' tree-item and click on the value part of the '''Exclude Rules'''. Now add rule ''''/cdd_tests$'''' and click ''''OK''''. <br />
<br />
Alternatively you can open your ecf file with a text editor. You will find a stance that looks like this:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Edit edit to make it looks like the following:<br />
<br />
<pre><br />
<file_rule><br />
<exclude>/.svn$</exclude><br />
<exclude>/EIFGENs$</exclude><br />
<exclude>/cdd_tests$</exclude><br />
</file_rule><br />
</pre><br />
<br />
Problem: ''''I have written or extracted a test case. How can I execute it?''''<br />
* All tests are executed in the background right after compiling automatically. You can choose to enable or disable background testing via the ''''Enable/Disable automatic background execution of tests''''.<br />
<br />
[[Image:enable_testing.png|center]]<br />
<br />
Problem: ''''The environment is unstable when using the .Net backend.''''<br />
Solution: CDD does currently not support the .Net backend. Please use the C backend.<br />
<br />
<br />
Problem: ''''How to disable the CDD extension in EiffelStudio''''<br />
Solution: Make sure the push-buttons for Extraction and Execution are not pressed. The buttons are shown in the below picture.<br />
[[Image:enable_exec_and_extract.png|center]]<br />
<br />
Problem: ''''My extracted test cases get overwritten in the SVN Repository, because my colleagues commit their extracted test cases which have the same name.''''<br />
Solution: Define da new environment variable "CDD_TESTER_ID". The value of the variable will be used as suffix for all extracted test class names. So if all members of the project define a unique "CDD_TESTER_ID", the individually extracted test cases can be commited to the svn without collisions.</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10924CddBranch2008-04-08T12:11:10Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== News ==<br />
<br />
2.April.2008: CDD EiffelStudio Final 6 available<br />
* Less mess: Redundant or duplicate test cases are no longer extracted<br />
* Smoother upgrade: Cleaning a project will automatically clean the test suite<br />
* Fixed bug in invarant checking<br />
* Fixed bug that caused EiffelStudio to freeze in some situations<br />
* Improved logging<br />
* Installer ''might'' work with mingw on Windows now. (Not yet tested, reports welcome)<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_7.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
Notes:<br />
* If you are '''upgrading''' from a previous version, make sure you delete the old version first (rm -rf)<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows.msi<br />
Notes:<br />
* Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* If you are '''upgrading''' from a previous version, you first have to uninstall the old version then delete the existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/) and then install the new version.<br />
* Which C compiler to install?<br />
** Use the Microsoft C compiler either from Visual Studio or the Windows SDK:<br />
*** Visual Studio (up to Visual Studio 2005, but no later, and only the non-express version)<br />
*** Windows SDK (up to version 6.0, but no later)<br />
** Do not use the gcc/mingw.<br />
** Do not use the .NET compiler backend. <br />
<br />
Have a look at http://dev.eiffel.com/Installing_Microsoft_C_compiler_6.1_and_older to learn how to install either Visual C++ or the Windows SDK for use with EiffelStudio. <br />
<br />
<br />
==== Windows Patch ====<br />
<br />
* Patch version for existing EiffelStudio CDD Edition Final 5 or Final 6 installation (does NOT work for official EiffelStudio 6.1 installation!): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_7-windows.zip<br />
* Replace <INSTALL_DIRECTORY>/studio/spec/windows/bin/ec.exe with ec.exe contained in archive<br />
* Delete existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/)<br />
* Recompile existing projects from scratch (use the "clean" option in the project load screen)<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
Redefine routine `set_up' or `tear_down' if you want something to be executed before resp. after every test routine of a class.<br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10884CddBranch2008-04-01T20:16:01Z<p>Mogh: /* Windows Patch */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_6.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
==== Windows Patch ====<br />
<br />
* Patch version for existing EiffelStudio CDD Edition Final 5 installation (does NOT work for official EiffelStudio 6.1 installation!): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.zip<br />
* Replace <INSTALL_DIRECTORY>/studio/spec/windows/bin/ec.exe with ec.exe contained in archive<br />
* Delete existing precompilations (= delete EIFGENs in subdirectories of <INSTALL_DIRECTORY>/precomp/spec/windows/)<br />
* Recompile existing projects from scratch (use the "clean" option in the project load screen)<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10883CddBranch2008-04-01T20:07:13Z<p>Mogh: /* Windows Patch */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_6.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
==== Windows Patch ====<br />
<br />
* Patch version: http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.zip<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10882CddBranch2008-04-01T20:06:48Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_6.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Windows Patch ==<br />
<br />
* Patch version: http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.zip<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10881CddBranch2008-04-01T20:05:45Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD EiffelStudio? ==<br />
<br />
[[Image:cdd_logo.png|center]]<br />
<center><br />
[http://se.ethz.ch/people/leitner/cdd/video/ '''Play CDD Video''', click here]<br />
</center><br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. It adds advanced support for unit testing to EiffelStudio. With CDD EiffelStudio you can<br />
<br />
* Write test cases<br />
* Manage test cases (using tags)<br />
* Run test cases<br />
* View test outcomes<br />
* '''Automatically extract test cases'''<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]. <br />
<br />
CDD EiffelStudio adds the following panel to regular EiffelStudio:<br />
<br />
[[Image:cdd_panel.png|center]]<br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_6.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_6-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Documentation ==<br />
<br />
=== Manually Written Test Cases ===<br />
CDD Eiffelstudio allows you to create a new “empty” test case. These test cases are similar to jUnit test cases. A manually written test class must start with the word “TEST” and all test routines also have to start with the word “test”. It also has to inherit from class CDD_TEST_CASE.<br />
<br />
=== Extracted Test Cases ===<br />
CDD EiffelStudio automatically extracts test cases whenever you run your program and an exception is triggered. This feature is novel and not yet part of any other testing environment. You will be the first to try it out.<br />
<br />
=== Test outcomes ===<br />
A test case checks whether your program contains a particular bug. A test cases can fail indicating that the bug is present in your program, or pass indicating your program does not contain this bug. Sometimes test cases will be unresolved, in which case the testing framework was unable to find out whether the test case passed or failed. A test case can be unresolved for several reasons.<br />
<br />
<br />
[[Image:cdd_buttons.png|center]]<br />
<br />
=== Debug Test Case === <br />
Select a test case and press this button to run a test case in the debugger.<br />
<br />
=== Enable/Disable Execution of Test Cases in Background === <br />
If enabled, all test cases are retested every time you press compile. If disabled no test cases are executed.<br />
<br />
=== Enable/Disable automatic extraction of Test Cases === <br />
If enabled every time an exception is triggered a set of test cases that try to reproduce this exception is extracted. If disabled no test cases are extracted.<br />
<br />
=== Clean up/Delete === <br />
You can use the “Clean Up/Delete” button in two different ways. By simply pressing it you will delete all unresolved test cases. By pick and dropping a test case to the “Clean up/Delete” button (right click on test case, move mouse to button and right click again) you can delete a test case. By the way, test cases are just regular classes. So you can use all existing tools that apply to classes in EiffelStudio too.<br />
<br />
Update: To remove duplicate test cases (until the next cdd update), please use the command-line tool from the [http://clean-cdd.origo.ethz.ch/ clean-cdd project].<br />
<br />
=== Create new manual test class ===<br />
Press this button to create a empty test class. You can then edit the class to add manually written test cases. This is how a manually written test case can look like:<br />
<br />
<eiffel><br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
test_deposit<br />
local<br />
ba: BANK_ACCOUNT<br />
do<br />
create ba.make_with_balance (0)<br />
ba.deposit (100)<br />
check<br />
money_depisited: ba.balance = 100<br />
end<br />
end<br />
end<br />
</eiffel><br />
<br />
[[Image:cdd_buttons_2.png|center]]<br />
<br />
=== Search for tags ===<br />
Enter keywords to search for paritcular test cases. Some tags are automatically set for you like the name of the test case. You can also easily add your own tags by adding an indexing item "tag" to your test class or routine:<br />
<br />
<eiffel><br />
indexing<br />
tag: "fixme"<br />
class TEST_BANK_ACCOUNT<br />
inherit CDD_TEST_CASE<br />
feature<br />
...<br />
end<br />
</eiffel><br />
<br />
Tags can be adde to all test routines and classes. Whether they are extracted or manually written does not matter.<br />
<br />
=== Restrict execution of test cases ===<br />
<br />
Once you have many test cases, you will run into situation where you don't want to execute all of them. The restrict button will help you to achieve this. As long as the "Restrict" button is pushed test cases that don't show up in the test case view will not be tested. Execution is restricted to those test cases that do show up.<br />
<br />
=== Change test case view ===<br />
<br />
Select one of several predefined test case views. For example you can group test cases by their outcome to quickly see only failing test cases.<br />
<br />
== Further Documentation and Common Problems ==<br />
Please visit [[Using CDD]] for further documentation or look at <br />
[[CDD Common Problems|Common Problems]] if you run into problems.<br />
<br />
== Related Publications ==<br />
<br />
* Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development - Writing Test Cases", Proceedings of ESEC/FSE'07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007 [http://se.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf (pdf)]<br />
* Sunghun Kim, Shay Artzi, and Michael D. Ernst, "reCrash: Making Crash Reproducible" MIT Computer Science and Artificial Intelligence Laboratory technical report MIT-CSAIL-TR-2007-054, (Cambridge, MA), November 20, 2007. [http://recrash.googlecode.com/files/MIT-CSAIL-TR-2007-054.pdf (pdf)]<br />
<br />
== Project Internal Stuff ==<br />
<br />
[[CddBranchInternal]]</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10880CddMeeting 13 02 20082008-04-01T19:31:56Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Don't store test case if exactly the same one exists already.<br />
* [DONE] Look at test case from Manu that doesn't compile (ARRAY is printed instead of ARRAY[STRING]<br />
* [DONE] Add link to "common problems" website in context window for test cases that time out<br />
<br />
* [DONE]Invariant violations occurring before routine invocation make a test case fail. It should be invalid. How to reproduce: Launch bank account example with final 2. Implement `BANK_ACCOUNT.withdraw' correctly. Replace postcondition with "withdrawn: balance = old balance + an_amount". Launch app, try to withdraw 700. You will get a test case that fails. It should be invalid though.<br />
<br />
* [DONE] Inspect cannot read file problem that occured in lecture. I can reproduce it also on my Linux box now.<br />
* [DONE] Log:<br />
** Playing around (duration and number of times)<br />
** FG-test case execution (duration, number of times, exception thrown, which test case, outcome)<br />
* [DONE] Clean cdd target when main target is cleaned too.<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10875CddMeeting 13 02 20082008-03-26T17:25:34Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Don't store test case if exactly the same one exists already.<br />
* [DONE] Look at test case from Manu that doesn't compile (ARRAY is printed instead of ARRAY[STRING]<br />
* [DONE] Add link to "common problems" website in context window for test cases that time out<br />
<br />
* [DONE]Invariant violations occurring before routine invocation make a test case fail. It should be invalid. How to reproduce: Launch bank account example with final 2. Implement `BANK_ACCOUNT.withdraw' correctly. Replace postcondition with "withdrawn: balance = old balance + an_amount". Launch app, try to withdraw 700. You will get a test case that fails. It should be invalid though.<br />
<br />
* Inspect cannot read file problem that occured in lecture. I can reproduce it also on my Linux box now.<br />
* Log:<br />
** Playing around (duration and number of times)<br />
** FG-test case execution (duration, number of times, exception thrown, which test case, outcome)<br />
* Clean cdd target when main target is cleaned too.<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10873CddMeeting 13 02 20082008-03-26T15:15:09Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Don't store test case if exactly the same one exists already.<br />
* [DONE] Look at test case from Manu that doesn't compile (ARRAY is printed instead of ARRAY[STRING]<br />
* [DONE] Add link to "common problems" website in context window for test cases that time out<br />
<br />
* Invariant violations occurring before routine invocation make a test case fail. It should be invalid. How to reproduce: Launch bank account example with final 2. Implement `BANK_ACCOUNT.withdraw' correctly. Replace postcondition with "withdrawn: balance = old balance + an_amount". Launch app, try to withdraw 700. You will get a test case that fails. It should be invalid though.<br />
<br />
* Inspect cannot read file problem that occured in lecture. I can reproduce it also on my Linux box now.<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Log:<br />
** Playing around (duration and number of times)<br />
** FG-test case execution (duration, number of times, exception thrown, which test case, outcome)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10872CddMeeting 13 02 20082008-03-26T14:50:12Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Don't store test case if exactly the same one exists already.<br />
* [DONE] Look at test case from Manu that doesn't compile (ARRAY is printed instead of ARRAY[STRING]<br />
<br />
* Invariant violations occurring before routine invocation make a test case fail. It should be invalid. How to reproduce: Launch bank account example with final 2. Implement `BANK_ACCOUNT.withdraw' correctly. Replace postcondition with "withdrawn: balance = old balance + an_amount". Launch app, try to withdraw 700. You will get a test case that fails. It should be invalid though.<br />
<br />
* Add link to "common problems" website in context window for test cases that time out<br />
* Inspect cannot read file problem that occured in lecture. I can reproduce it also on my Linux box now.<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Log:<br />
** Playing around (duration and number of times)<br />
** FG-test case execution (duration, number of times, exception thrown, which test case, outcome)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10871CddMeeting 13 02 20082008-03-26T14:48:11Z<p>Mogh: /* Bug Fixing */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* Invariant violations occurring before routine invocation make a test case fail. It should be invalid. How to reproduce: Launch bank account example with final 2. Implement `BANK_ACCOUNT.withdraw' correctly. Replace postcondition with "withdrawn: balance = old balance + an_amount". Launch app, try to withdraw 700. You will get a test case that fails. It should be invalid though.<br />
<br />
* Add link to "common problems" website in context window for test cases that time out<br />
* Inspect cannot read file problem that occured in lecture. I can reproduce it also on my Linux box now.<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Don't store test case if exactly the same one exists already.<br />
* Look at test case from Manu that doesn't compile (ARRAY is printed instead of ARRAY[STRING]<br />
* Log:<br />
** Playing around (duration and number of times)<br />
** FG-test case execution (duration, number of times, exception thrown, which test case, outcome)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10739CddBranch2008-03-12T14:38:49Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_5.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_5-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10736CddBranch2008-03-11T20:31:42Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_3.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_4-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10731CddBranch2008-03-10T18:39:19Z<p>Mogh: /* Windows */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at [http://se.ethz.ch ETH Zurich]. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [http://eiffelstudio.origo.ethz.ch/forum/20 CDD forum]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download CDD ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_3.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_final_3-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10711CddBranch2008-03-05T19:05:31Z<p>Mogh: /* Download CDD Final 1 */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download CDD Final 2 ==<br />
<br />
The following packages contain the full EiffelStudio 6.1 plus the CDD extension. You do not need to have EiffelStudio installed already in order to install below packages. On Windows you do have to have either the Platform SDK or Visual C++ installed. Do not use EiffelStudio with the gcc/mingw or the .Net backend.<br />
<br />
=== Linux ===<br />
* Full Linux version: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_1.tar.bz2<br />
* Installation instructions: http://docs.eiffel.com/eiffelstudio/installation/studio/060_linux.html<br />
<br />
Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD". Make sure you set/update the environment variables PATH, ISE_EIFFEL, and ISE_PLATFORM according to the installation instructions.<br />
<br />
=== Windows ===<br />
<br />
* Full Windows version (with installer): http://n.ethz.ch/~moris/download/Eiffel61_cdd_edition_gpl_72290-windows.msi<br />
* Note 1: Installation is independent of installations of official EiffelStudio 6.1 (neither overwrites nor invalidates nor is influenced by those) <br />
* Note 2: If you have a previous installation of the CDD Edition of EiffelStudio installed, you need to uninstall it first. If you want to reuse the installation directory, you need to manually delete all EIFGENs in its subdirectories after the uninstall procedure.<br />
* Note 3: Do not use the gcc/mingw or the .NET compiler backend. You will have to use the Microsoft C compiler. You can get it either by installing Visual C++, or via (the freely available) Microsoft Platform SDK.Have a look at http://eiffelsoftware.origo.ethz.ch/Installing_Microsoft_C_compiler to learn how to install either compiler.<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10688CDDHowtoRollARelease2008-03-03T21:40:11Z<p>Mogh: /* List of CDD specific files */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* ec(.exe) -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
* Src/library/base/ise/support/cdd -> $ISE_EIFFEL/library/base/ise/support/cdd<br />
* Src/examples/cdd -> $ISE_EIFFEL/examples/cdd<br />
* Delivery/studio/help/defaults -> $ISE_EIFFEL/studio/help/defaults<br />
* Delivery/studio/bitmaps/png/16x16.png -> $ISE_EIFFEL/studio/bitmaps/png/16x16.png<br />
* Delivery/studio/bitmaps/png/splash_shadow.png -> $ISE_EIFFEL/studio/bitmaps/png/splash_shadow.png<br />
* Delivery/studio/bitmaps/png/splash.png -> $ISE_EIFFEL/studio/bitmaps/png/splash.png<br />
* [For Windows Only] estudio.exe -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
RUNTIME patch<br />
* eif_except.h, eif_types.h: Src/C/run-time -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/include/<br />
* finalized.lib, mtfinalized.lib, mtwkbench.lib, wkbench.lib: Src/C/run-time/LIB -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/lib(/msc)/ [not sure about linux!]<br />
** NOTE: the libs mentioned above can be build with the geant script, geant -b %EIFFEL_SRC%/build.eant prepare (this will create the LIB folder and its content)<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10687CDDHowtoRollARelease2008-03-03T21:20:19Z<p>Mogh: /* List of CDD specific files */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* ec(.exe) -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
* Src/library/base/ise/support/cdd -> $ISE_EIFFEL/library/base/ise/support/cdd<br />
* Src/examples/cdd -> $ISE_EIFFEL/examples/cdd<br />
* Delivery/studio/help/defaults -> $ISE_EIFFEL/studio/help/defaults<br />
* Delivery/studio/bitmaps/png/16x16.png -> $ISE_EIFFEL/studio/bitmaps/png/16x16.png<br />
* Delivery/studio/bitmaps/png/splash_shadow.png -> $ISE_EIFFEL/studio/bitmaps/png/splash_shadow.png<br />
* Delivery/studio/bitmaps/png/splash.png -> $ISE_EIFFEL/studio/bitmaps/png/splash.png<br />
* [For Windows Only] estudio.exe -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
RUNTIME patch<br />
* eif_except.h, eif_types.h: Src/C/run-time -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/include/<br />
* finalized.lib, mtfinalized.lib, mtwkbench.lib, wkbench.lib: Src/C/run-time/LIB -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/lib(/msc)/ [not sure about linux!]<br />
** NOTE: the libs mentioned above can be build with the geant script, geant -b %EIFFEL_SRC%/build.eant prepare<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10677CddMeeting 13 02 20082008-03-01T15:45:23Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* [DONE!] CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** [DONE] Log original exception (make it part of test routine's state)<br />
** [DONE] Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** [DONE] Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* [DONE - WORKS ONLY WITH PRECONDITION CHECK ENABLED!] Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* [DONE] Fix config files for system level tests (remove cdd tag)<br />
* [DONE] Revive system level test suite<br />
* [DONE] Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
* [DONE] "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10676CddMeeting 13 02 20082008-03-01T15:03:45Z<p>Mogh: /* Bug Fixing */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* [DONE!] CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** [DONE] Log original exception (make it part of test routine's state)<br />
** [DONE] Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** [DONE] Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* [DONE - WORKS ONLY WITH PRECONDITION CHECK ENABLED!] Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* [DONE] Fix config files for system level tests (remove cdd tag)<br />
* [DONE] Revive system level test suite<br />
* [DONE] Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10675CddMeeting 13 02 20082008-03-01T15:02:53Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** [DONE] Log original exception (make it part of test routine's state)<br />
** [DONE] Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** [DONE] Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* [DONE - WORKS ONLY WITH PRECONDITION CHECK ENABLED!] Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* [DONE] Fix config files for system level tests (remove cdd tag)<br />
* [DONE] Revive system level test suite<br />
* [DONE] Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10673CddMeeting 13 02 20082008-02-29T14:38:04Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* [DONE] Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** [DONE] Log original exception (make it part of test routine's state)<br />
** [DONE] Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** [DONE] Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* [DONE - WORKS ONLY WITH PRECONDITION CHECK ENABLED!] Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* [DONE] Fix config files for system level tests (remove cdd tag)<br />
* [DONE] Revive system level test suite<br />
* [DONE] Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10672CddMeeting 13 02 20082008-02-29T10:09:37Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** Log original exception (make it part of test routine's state)<br />
** Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* Pressing manual test case button may cause error. <br />
* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
* Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10671CddMeeting 13 02 20082008-02-29T10:05:45Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** Log original exception (make it part of test routine's state)<br />
** Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* Pressing manual test case button may cause error. <br />
* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
* Adding a new test case and, changing it, saving it and recompiling it results in an error message about a ".swp" file. Make message go away.<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10663CddMeeting 13 02 20082008-02-27T09:45:43Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
<br />
===Arno===<br />
<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
<br />
* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
** Log original exception (make it part of test routine's state)<br />
** Logging all thrown exceptions (no matter whether extraction is enabled)<br />
** Split log files<br />
* [DONE] Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* Sometimes when a CDD breakpoint is hit no routine invocation is extracted. Find out why and fix it.<br />
* [DONE] Fix problem where exe of target is non standard (Anders report)<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
<br />
* Pressing manual test case button may cause error. <br />
* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Cache debug values when extracting several test cases.<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
* When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)<br />
<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10632CddBranch2008-02-22T15:19:37Z<p>Mogh: /* Download */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download ==<br />
<br />
* Download CDD Final 1<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_1.tar.bz2 (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Installer: http://n.ethz.ch/~moris/download/Eiffel61cdd_gpl_final1-windows.msi (Note: Installation is independent of installations of official EiffelStudio 6.1)<br />
*** Patch 1 for final 1 (fixes critical issue for windows): http://n.ethz.ch/~moris/download/EiffelStudio61cdd_final_1_patch_1.zip (replace existing ec.exe in <install_dir>\studio\spec\windows\bin with ec.exe found in archive. Delete EIFGENs in subdirectories of <install_dir>\precomp, since precompilations might not be compatible. Also build your projects from scratch by deleting their EIFGENs)<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10631CddBranch2008-02-22T15:18:16Z<p>Mogh: /* Download */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download ==<br />
<br />
* Download CDD Final 1<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_final_1.tar.bz2 (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Installer: http://n.ethz.ch/~moris/download/Eiffel61cdd_gpl_final1-windows.msi (Note: Installation is independent of installations of official EiffelStudio 6.1)<br />
*** Patch 1 for final 1 (fixes critical issue for windows): http://n.ethz.ch/~moris/download/EiffelStudio61cdd_final_1_patch_1.zip (replace existing ec.exe int <install_dir>\studio\spec\windows\bin with ec.exe found in archive. Delete EIFGENs in subdirectories of <install_dir>\precomp, since precompilations might not be compatible. Also build your projects from scratch)<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 20.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10628CddMeeting 13 02 20082008-02-20T13:51:18Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
<br />
===Arno===<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
* [DONE, temporary solution] Log original exception (make it part of test routine's state)<br />
* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)<br />
* Fix identation bug in test class printer (not properly reset)<br />
* When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)<br />
* Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
* Aborting the C compilation of the interpreter may leave the interpreter EIFGEN in an invalid state.<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10623CddMeeting 13 02 20082008-02-20T09:31:55Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* 15.02.2008; 14:15<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
<br />
===Arno===<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
* [DONE, temporary solution] Log original exception (make it part of test routine's state)<br />
* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message<br />
* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)<br />
* Refactor CDD Manager-Routine invocation cache-CDD Capturer<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
* Aborting the C compilation of the interpreter may leave the interpreter EIFGEN in an invalid state.<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===Comparisons===<br />
<br />
{| class="wikitable"<br />
|-<br />
!<br />
! contracts<br />
! man. tests<br />
! playing<br />
! extr.<br />
! synth.<br />
|-<br />
|<br />
| A<br />
| B<br />
| A B<br />
| <br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| A<br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| A<br />
| <br />
| <br />
| <br />
|-<br />
| experiment<br />
| A B<br />
| A<br />
| A B<br />
| A<br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| B<br />
| <br />
| <br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| B<br />
| <br />
|-<br />
|<br />
| A B<br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
|<br />
| <br />
| <br />
| <br />
| A<br />
| B<br />
|-<br />
| ISSTA<br />
| A B<br />
| A<br />
| <br />
| <br />
| B<br />
|-<br />
|<br />
| <br />
| A<br />
| <br />
| <br />
| B<br />
|}<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10621CddBranch2008-02-18T21:11:11Z<p>Mogh: /* Download */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download ==<br />
<br />
* Download CDD Beta 3<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_beta_3.tar.gz (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Installer: http://n.ethz.ch/~moris/download/Eiffel61cdd_gpl_final1-windows.msi (Note: Installation is independent of installations of official EiffelStudio 6.1)<br />
*** Patch: Not available for Final 1. (The CDD Edition of EiffelStudio is no longer overwriting installations of official EiffelStudio 6.1. So it does need its own registry entries which are set using the installer. For further releases a patch might be available again for patching installations produced by the installer below)<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final 1 release<br />
* 19.02.2008: Initial Questionnaire<br />
* 22.02.2008: Final 1 handover to ISG<br />
* 22.02.2008: Final 1 installed on students machines<br />
* 29.02.2008: Final 2 release<br />
* 05.03.2008: Final 2 handover to ISG<br />
* 07.03.2008: Final 2 installed on students machines<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10620CDDHowtoRollARelease2008-02-18T20:29:42Z<p>Mogh: /* List of CDD specific files */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* ec(.exe) -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
* Src/library/base/ise/support/cdd -> $ISE_EIFFEL/library/base/ise/support/cdd<br />
* [OBSOLETE, file has been reverted to version of official ES 6.1] Src/library/base/ise/runtime/debug/classic/rt_extension.e -> $ISE_EIFFEL/library/base/ise/runtime/debug/classic/rt_extension.e<br />
* Src/examples/cdd -> $ISE_EIFFEL/examples/cdd<br />
* Delivery/studio/help/defaults -> $ISE_EIFFEL/studio/help/defaults<br />
* Delivery/studio/bitmaps/png/16x16.png -> $ISE_EIFFEL/studio/bitmaps/png/16x16.png<br />
* Delivery/studio/bitmaps/png/splash_shadow.png -> $ISE_EIFFEL/studio/bitmaps/png/splash_shadow.png<br />
* Delivery/studio/bitmaps/png/splash.png -> $ISE_EIFFEL/studio/bitmaps/png/splash.png<br />
* [For Windows Only] estudio.exe -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10609CddMeeting 13 02 20082008-02-14T19:10:11Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* ?<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
<br />
===Arno===<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* [RECURRENT, currently valid]Check log for XML validity<br />
* [DONE] Add timeout information to log<br />
* [DONE] Distinguish extracted, synthesized and manual test cases in logs<br />
* [DONE] Log TS Snapshot after execution (only executed routines)<br />
* [DONE] Log TS Snapshot after compilation (complete test suite)<br />
* [DONE] Log when ES starts up and shuts down<br />
* [DONE] Log time it takes to extract test case<br />
* [DONE] Log time it takes to compile SUT<br />
* [DONE] Log time it takes to compile test suite<br />
* [DONE, temporary solution] Log original exception (make it part of test routine's state)<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* [WAITING FOR ANSWER TO EMAIL] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&diff=10608CddMeeting 13 02 20082008-02-14T13:42:33Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 14.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* ?<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* [delayed] Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* [done] Add timeout judgement<br />
<br />
===Arno===<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
* Make newly extracted test cases show up "expanded" in GUI treeview<br />
* "Cleanup button" deletes filtered unresolved extracted test cases<br />
====Bug Fixing====<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* Check log for XML validity<br />
* [DONE] Add timeout information to log<br />
* [DONE] Distinguish extracted, synthesized and manual test cases in logs<br />
* [DONE] Log TS Snapshot after execution (only executed routines)<br />
* [DONE] Log TS Snapshot after compilation (complete test suite)<br />
* Log when ES starts up and shuts down<br />
* Log time it takes to extract test case<br />
* [DONE] Log time it takes to compile SUT<br />
* [DONE] Log time it takes to compile test suite<br />
* Log original exception (make it part of test routine's state)<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10601CddBranch2008-02-14T09:37:27Z<p>Mogh: /* Download */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Download ==<br />
<br />
* Download CDD Beta 2<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_beta_2.tar.gz (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Patch: There is no Patch release available for Beta3, since the CDD Edition of EiffelStudio is no longer overwriting installations of official EiffelStudio 6.1. So it does need its own registry entries which are set using the installer. (for further releases a patch might be available again for patching installations produced by the installer below)<br />
*** Installer: http://n.ethz.ch/~moris/download/Eiffel61cdd_gpl_beta3-windows.msi (Note: Installation is independent of installations of official EiffelStudio 6.1)<br />
<br />
== Documentation ==<br />
<br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 11.02.2008: Beta 3<br />
* 18.02.2008: Final CDD release for experiment online<br />
* 19.02.2008: Initial Questionnaire<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10600CDDHowtoRollARelease2008-02-14T08:13:28Z<p>Mogh: /* List of CDD specific files */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* ec(.exe) -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
* Src/library/base/ise/support/cdd -> $ISE_EIFFEL/library/base/ise/support/cdd<br />
* Src/library/base/ise/runtime/debug/classic/rt_extension.e -> $ISE_EIFFEL/library/base/ise/runtime/debug/classic/rt_extension.e<br />
* Src/examples/cdd -> $ISE_EIFFEL/examples/cdd<br />
* Delivery/studio/help/defaults -> $ISE_EIFFEL/studio/help/defaults<br />
* Delivery/studio/bitmaps/png/16x16.png -> $ISE_EIFFEL/studio/bitmaps/png/16x16.png<br />
* Delivery/studio/bitmaps/png/splash_shadow.png -> $ISE_EIFFEL/studio/bitmaps/png/splash_shadow.png<br />
* Delivery/studio/bitmaps/png/splash.png -> $ISE_EIFFEL/studio/bitmaps/png/splash.png<br />
* [For Windows Only] estudio.exe -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10599CDDHowtoRollARelease2008-02-14T08:09:12Z<p>Mogh: /* List of CDD specific files */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* ec(.exe) -> $ISE_EIFFEL/studio/spec/$ISE_PLATFORM/bin<br />
* Src/library/base/ise/support/cdd -> $ISE_EIFFEL/library/base/ise/support/cdd<br />
* Src/library/base/ise/runtime/debug/classic/rt_extension.e -> $ISE_EIFFEL/library/base/ise/runtime/debug/classic/rt_extension.e<br />
* Src/examples/cdd -> $ISE_EIFFEL/examples/cdd<br />
* Delivery/studio/help/defaults -> $ISE_EIFFEL/studio/help/defaults<br />
* Delivery/studio/bitmaps/png/16x16.png -> $ISE_EIFFEL/studio/bitmaps/png/16x16.png<br />
* Delivery/studio/bitmaps/png/splash_shadow.png -> $ISE_EIFFEL/studio/bitmaps/png/splash_shadow.png<br />
* Delivery/studio/bitmaps/png/splash.png -> $ISE_EIFFEL/studio/bitmaps/png/splash.png<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_05_02_2008&diff=10581CddMeeting 05 02 20082008-02-07T09:17:29Z<p>Mogh: /* Stefan */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 06.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* Tuesday, 12.02.2008, 10:00<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* Add timeout judgement<br />
* Timeout -> 5 sec<br />
<br />
===Arno===<br />
* When test class gets removed manually, update test suite<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Display ignored test class compilation errors (looks like we will have this for free in 6.1)<br />
* Red bg for failing test cases in view <br />
* When debugging extracted test case, set first breakpoint in "covers." feature<br />
* Extraction for inline agents not currently working (at least not always)<br />
** Create inline agent test case<br />
** Fix extraction for inline agents<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
====Bug Fixing====<br />
* Result type (like Current) produces syntax error in new test class<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* Distinguish extracted, synthesized and manual test cases in logs<br />
* Log TS Snapshot after compilation<br />
* Log TS Snapshot after testing<br />
* Log when ES starts up and shuts down<br />
* Log time it takes to extract test case<br />
* Log time it takes to compile SUT<br />
* Log time it takes to compile test suite<br />
* Log original exception (make it part of test routine's state)<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [Jocelyn]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [Jocelyn]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [Jocelyn]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Jocelyn]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_05_02_2008&diff=10580CddMeeting 05 02 20082008-02-07T09:14:03Z<p>Mogh: /* Beta Tester Feedback */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Wednesday, 06.02.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* Tuesday, 12.02.2008, 10:00<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* Finish tuple_002 test case<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* Add timeout judgement<br />
* Timeout -> 5 sec<br />
<br />
===Arno===<br />
* When test class gets removed manually, update test suite<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Display ignored test class compilation errors (looks like we will have this for free in 6.1)<br />
* Red bg for failing test cases in view <br />
* When debugging extracted test case, set first breakpoint in "covers." feature<br />
* Extraction for inline agents not currently working (at least not always)<br />
** Create inline agent test case<br />
** Fix extraction for inline agents<br />
* Make general encoding/decoding routines for special feature names<br />
** e.g. infix "+" (ES) <=> infix_plus (CDD)<br />
====Bug Fixing====<br />
* Result type (like Current) produces syntax error in new test class<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [RECURRENT] Build releasable delivery on Windows<br />
* Distinguish extracted, synthesized and manual test cases in logs<br />
* Log TS Snapshot after compilation<br />
* Log TS Snapshot after testing<br />
* Log when ES starts up and shuts down<br />
* Log time it takes to extract test case<br />
* Log time it takes to compile SUT<br />
* Log time it takes to compile test suite<br />
* Log original exception (make it part of test routine's state)<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
* Revive system level test suite<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
=== Bernd ===<br />
* Define Project for SoftEng<br />
** Find test suite for us to test students code<br />
** Find project with pure functional part<br />
<br />
===Unassigned===<br />
<br />
* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)<br />
* Cache debug values when extracting several test cases.<br />
* Fix config files for system level tests (remove cdd tag)<br />
<br />
====Beta Tester Feedback====<br />
(Please put your name so we can get back to you in the case of questions)<br />
* (Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [Jocelyn]<br />
** home directory? application_data directory? [Jocelyn]<br />
* (Jocelyn) There should be UI support for deletion of Test Case [Jocelyn]<br />
* (Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [Jocelyn]<br />
* (Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Jocelyn]<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* Task 1: Implement VCard API<br />
* Task 2: Implement Mime API<br />
* Task 3: Write test cases to reveal faults in foreign VCard implementations<br />
* Task 4: Write test cases to reveal faults in foreign Mime implementations<br />
<br />
* Group A:<br />
** Task 1, Manual Tests<br />
** Task 2, Extracted Tests<br />
** Task 3, Manual Tests<br />
** Task 4, Extracted Tests<br />
* Group B:<br />
** Task 1, Extracted Tests<br />
** Task 2, Manual Tests<br />
** Task 3, Extracted Tests<br />
** Task 4, Manual Tests<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
===How reliably can we extract test cases that reproduce the original failure?===<br />
* Log original exception and exception received from first test execution<br />
<br />
===Are the extracted tests useful for debugging?===<br />
* Ask developers, using CDD<br />
<br />
===What is the (time and memory) overhead of enabling extraction?===<br />
<br />
===What is the size of the extracted test cases?===<br />
<br />
===Are we able to reproduce bugs from industry?===<br />
Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.<br />
<br />
===Does it make a difference in the quality of the code, whether one tests manually or extracts them?===<br />
* Compare projects using extracted tests and manual tests to ref test suite<br />
<br />
===Do contracts replace traditional testing oracles?===<br />
* Original API without contracts<br />
* Run failing test cases (the ones we get from second part) with reference API with contracts<br />
* How many times does the contract replace the testing oracle?</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_31_01_2008&diff=10564CddMeeting 31 01 20082008-02-05T16:01:46Z<p>Mogh: /* Beta Tester Feedback */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Tuesday, 31.1.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* Tuesday, 5.2.2008, 10:00<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* [done] Commit dangling patch from 6.0 to 6.1<br />
* [done] Make it so that tester target never has extraction or execution enabled<br />
** remove hack from CDD_MANAGER.schedule_testing_restart<br />
* [done] Make CDD Windows apear by default<br />
* Finish tuple_002 test case<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
<br />
===Arno===<br />
* When test class gets removed manually, update test suite<br />
* Clean up test case in interpreter after each execution (through garbage collection?)<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Display ignored test class compilation errors (looks like we will have this for free in 6.1)<br />
* Make sure CDD Tools are visible by default (what layout would you prefer?)<br />
** Main tool shares tabs with clusters/features tool, output tool after C output tool<br />
* Red bg for failing test cases in view <br />
* Write new simple "New Manual Test Case" dialog<br />
* Test case for (user defined) expanded types<br />
* Test case containing feature names with underscores and "like Current"<br />
* When debugging extracted test case, set first breakpoint in "covers." feature<br />
<br />
<br />
====Bug Fixing====<br />
* Result type (like Current) produces syntax error in new test class<br />
* Fix interpreter hang after runtime crash<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations<br />
* Check if interpreter compilation errors are propagated correctly (seems to start interpreter even though compilation has failed)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [DONE] Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings<br />
* [DONE, except probably for bad memory corruption bugs which didn't occur anymore since agents are ignored] Make popup on interpreter crash go away (win32 only)<br />
<br />
* [RECURRENT] Build releasable delivery on Windows<br />
<br />
* Logging<br />
** What data to log?<br />
** Implement storing<br />
** Define how students should submit logs<br />
* Data Gathering<br />
** Define what data to gather<br />
** Define how to process gather data<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Is it still necessary to ever call the routine update actions with argument "void"?<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Define Project for SoftEng (due by next meeting)<br />
** Find System level test suite for us to test students code<br />
** Find project with pure functional part<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
===Unassigned===<br />
<br />
* Cache debug values when extracting several test cases.<br />
* Enable execution and extraction by default for new projects.<br />
* Make CDD Window and CDD Log Window visiable by default<br />
* "Debug selected test routine" should be grayed out if no test case is currently selected<br />
* Testing V2 Application should not interupt flow<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* Extraction for inline agents not currently working (at least not always)<br />
** Create inline agent test case<br />
** Fix extraction for inline agents<br />
* Revive system level test suite<br />
* There is a performance problem when compiling a test suite for 'ec' itself. There are indications that the parent ec process consumes too much CPU cycles. Maybe the CDD output refresh is too eager. Find out what the problem is and fix it.<br />
<br />
====Beta Tester Feedback====<br />
* It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?)<br />
** home directory? application_data directory? <br />
* There should be UI support for deletion of Test Case<br />
* [BUG] the manual test case creation dialog should check if class with chosen name is already in the system<br />
* It would be nice if there was a way to configure the timeout for the interpreter<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
==Do Contracts improve Tests?==<br />
<br />
* Is there a correlation between Tests quantity or quality and the quantity or quality of contracts?<br />
<br />
== Corellation between failure/fault type and test type? ==<br />
<br />
* Do certain kind of tests find certain kind of failures/faults?<br />
<br />
===Use of CDD increases development productivity===<br />
* Did the use of testing decrease development time?<br />
<br />
* Meassures:<br />
** Number of compilations<br />
** Number of saves<br />
** Number of revisions<br />
** IDE time<br />
** Asking the students<br />
<br />
Emphasis on quetionnair result. Correlation with logs only if it makes sense<br />
<br />
===Use of CDD increases code correctness===<br />
* Is there a relation between code correctness of project (vs. some system level test suite) and test activity?<br />
<br />
* Measures:<br />
** number of tests<br />
** number of times test were run<br />
** Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)<br />
** Secret test suite<br />
<br />
===Developer Profile: Is there a correlation between Developer Profile and the way they use testing tools===<br />
* How did students use the testing tools?<br />
* Are ther clusters of similar use? <br />
* What is charactersitic for these clusters?<br />
* Meassures:<br />
** Aksing students before and after<br />
** Are there projects where tests initially always fail resp. pass<br />
** How often do they test?<br />
** How correct is their project?<br />
<br />
Midterm questionnaire will be used to phrase questions for final questionnaire.<br />
====Example profiles====<br />
* Waldundwiesen Hacker<br />
** No explicit structure. Does whatever seems appriorate at the time. No QA plan.<br />
* Agile<br />
** Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.<br />
* Waterfall inspired<br />
** Explicit process model. Phases don't interleave.<br />
* ?<br />
<br />
===How do extracted, synthesized and manually written test cases compare?===<br />
* Which tests are the most useful to students?<br />
* How many tests are there in each category?<br />
* What's the test suite quality of each category?<br />
* Were some excluded from testing more often than others?<br />
* How many red/green and green/red transitions are there in each category?<br />
* Which had compile-time errors most often that did not get fixed?<br />
* Meassures:<br />
** LOC<br />
** Number of tests<br />
** Number of executions<br />
** Outcome transitions</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_31_01_2008&diff=10563CddMeeting 31 01 20082008-02-05T14:17:20Z<p>Mogh: /* Beta Tester Feedback */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Tuesday, 31.1.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* Tuesday, 5.2.2008, 10:00<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* [done] Commit dangling patch from 6.0 to 6.1<br />
* [done] Make it so that tester target never has extraction or execution enabled<br />
** remove hack from CDD_MANAGER.schedule_testing_restart<br />
* [done] Make CDD Windows apear by default<br />
* Finish tuple_002 test case<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
<br />
===Arno===<br />
* When test class gets removed manually, update test suite<br />
* Clean up test case in interpreter after each execution (through garbage collection?)<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Display ignored test class compilation errors (looks like we will have this for free in 6.1)<br />
* Make sure CDD Tools are visible by default (what layout would you prefer?)<br />
** Main tool shares tabs with clusters/features tool, output tool after C output tool<br />
* Red bg for failing test cases in view <br />
* Write new simple "New Manual Test Case" dialog<br />
* Test case for (user defined) expanded types<br />
* Test case containing feature names with underscores and "like Current"<br />
* When debugging extracted test case, set first breakpoint in "covers." feature<br />
<br />
<br />
====Bug Fixing====<br />
* Result type (like Current) produces syntax error in new test class<br />
* Fix interpreter hang after runtime crash<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations<br />
* Check if interpreter compilation errors are propagated correctly (seems to start interpreter even though compilation has failed)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [DONE] Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings<br />
* [DONE, except probably for bad memory corruption bugs which didn't occur anymore since agents are ignored] Make popup on interpreter crash go away (win32 only)<br />
<br />
* [RECURRENT] Build releasable delivery on Windows<br />
<br />
* Logging<br />
** What data to log?<br />
** Implement storing<br />
** Define how students should submit logs<br />
* Data Gathering<br />
** Define what data to gather<br />
** Define how to process gather data<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Is it still necessary to ever call the routine update actions with argument "void"?<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Define Project for SoftEng (due by next meeting)<br />
** Find System level test suite for us to test students code<br />
** Find project with pure functional part<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
===Unassigned===<br />
<br />
* Cache debug values when extracting several test cases.<br />
* Enable execution and extraction by default for new projects.<br />
* Make CDD Window and CDD Log Window visiable by default<br />
* "Debug selected test routine" should be grayed out if no test case is currently selected<br />
* Testing V2 Application should not interupt flow<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* Extraction for inline agents not currently working (at least not always)<br />
** Create inline agent test case<br />
** Fix extraction for inline agents<br />
* Revive system level test suite<br />
* There is a performance problem when compiling a test suite for 'ec' itself. There are indications that the parent ec process consumes too much CPU cycles. Maybe the CDD output refresh is too eager. Find out what the problem is and fix it.<br />
<br />
====Beta Tester Feedback====<br />
* It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?)<br />
** home directory? application_data directory? <br />
* There should be UI support for deletion of Test Case<br />
* [BUG] the manual test case creation dialog should check if class with chosen name is already in the system<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
==Do Contracts improve Tests?==<br />
<br />
* Is there a correlation between Tests quantity or quality and the quantity or quality of contracts?<br />
<br />
== Corellation between failure/fault type and test type? ==<br />
<br />
* Do certain kind of tests find certain kind of failures/faults?<br />
<br />
===Use of CDD increases development productivity===<br />
* Did the use of testing decrease development time?<br />
<br />
* Meassures:<br />
** Number of compilations<br />
** Number of saves<br />
** Number of revisions<br />
** IDE time<br />
** Asking the students<br />
<br />
Emphasis on quetionnair result. Correlation with logs only if it makes sense<br />
<br />
===Use of CDD increases code correctness===<br />
* Is there a relation between code correctness of project (vs. some system level test suite) and test activity?<br />
<br />
* Measures:<br />
** number of tests<br />
** number of times test were run<br />
** Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)<br />
** Secret test suite<br />
<br />
===Developer Profile: Is there a correlation between Developer Profile and the way they use testing tools===<br />
* How did students use the testing tools?<br />
* Are ther clusters of similar use? <br />
* What is charactersitic for these clusters?<br />
* Meassures:<br />
** Aksing students before and after<br />
** Are there projects where tests initially always fail resp. pass<br />
** How often do they test?<br />
** How correct is their project?<br />
<br />
Midterm questionnaire will be used to phrase questions for final questionnaire.<br />
====Example profiles====<br />
* Waldundwiesen Hacker<br />
** No explicit structure. Does whatever seems appriorate at the time. No QA plan.<br />
* Agile<br />
** Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.<br />
* Waterfall inspired<br />
** Explicit process model. Phases don't interleave.<br />
* ?<br />
<br />
===How do extracted, synthesized and manually written test cases compare?===<br />
* Which tests are the most useful to students?<br />
* How many tests are there in each category?<br />
* What's the test suite quality of each category?<br />
* Were some excluded from testing more often than others?<br />
* How many red/green and green/red transitions are there in each category?<br />
* Which had compile-time errors most often that did not get fixed?<br />
* Meassures:<br />
** LOC<br />
** Number of tests<br />
** Number of executions<br />
** Outcome transitions</div>Moghhttps://dev.eiffel.com/index.php?title=CddMeeting_31_01_2008&diff=10562CddMeeting 31 01 20082008-02-05T14:12:47Z<p>Mogh: /* Unassigned */</p>
<hr />
<div>[[Category:CDD]]<br />
=CDD Meeting, Tuesday, 31.1.2008, 10:00=<br />
<br />
== Next Meeting ==<br />
* Tuesday, 5.2.2008, 10:00<br />
<br />
== Tasks ==<br />
<br />
===Andreas===<br />
* Forumulate Experiment Hypothesis (Andreas)<br />
* Fix AutoTest for courses<br />
** New release<br />
* Write documentation and videos tutorials (together with final release)<br />
* [done] Commit dangling patch from 6.0 to 6.1<br />
* [done] Make it so that tester target never has extraction or execution enabled<br />
** remove hack from CDD_MANAGER.schedule_testing_restart<br />
* [done] Make CDD Windows apear by default<br />
* Finish tuple_002 test case<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
<br />
===Arno===<br />
* When test class gets removed manually, update test suite<br />
* Clean up test case in interpreter after each execution (through garbage collection?)<br />
* Build releasable delivery for Linux (after each Beta I guess...)<br />
* Display ignored test class compilation errors (looks like we will have this for free in 6.1)<br />
* Make sure CDD Tools are visible by default (what layout would you prefer?)<br />
** Main tool shares tabs with clusters/features tool, output tool after C output tool<br />
* Red bg for failing test cases in view <br />
* Write new simple "New Manual Test Case" dialog<br />
* Test case for (user defined) expanded types<br />
* Test case containing feature names with underscores and "like Current"<br />
* When debugging extracted test case, set first breakpoint in "covers." feature<br />
<br />
<br />
====Bug Fixing====<br />
* Result type (like Current) produces syntax error in new test class<br />
* Fix interpreter hang after runtime crash<br />
* Check why EiffelStudio quits after debugging a test routine and ignoring violations<br />
* Check if interpreter compilation errors are propagated correctly (seems to start interpreter even though compilation has failed)<br />
<br />
===Ilinca===<br />
<br />
* Integrate variable declarations into AutoTest trunk (by 8.2.2008)<br />
<br />
===Stefan===<br />
* [DONE] Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings<br />
* [DONE, except probably for bad memory corruption bugs which didn't occur anymore since agents are ignored] Make popup on interpreter crash go away (win32 only)<br />
<br />
* [RECURRENT] Build releasable delivery on Windows<br />
<br />
* Logging<br />
** What data to log?<br />
** Implement storing<br />
** Define how students should submit logs<br />
* Data Gathering<br />
** Define what data to gather<br />
** Define how to process gather data<br />
* Second Chance re-run to find true prestate (with Jocelyn)<br />
* Allow for test case extraction of passing routine invocations (with Jocelyn)<br />
<br />
* Rebuilding manual test suite through extraction and synthesizing<br />
* Find performance bottleneck of test case extraction and propose extraction method for second chance<br />
<br />
====Bugs/Things to look at====<br />
<br />
* For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES<br />
* Is it still necessary to ever call the routine update actions with argument "void"?<br />
* Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)<br />
<br />
===Manu===<br />
* Define Project for SoftEng (due by next meeting)<br />
** Find System level test suite for us to test students code<br />
** Find project with pure functional part<br />
* Install CDD in student labs (Manu)<br />
* Devise questionnaires<br />
** Initial (due next meeting after Manu's vacation)<br />
** Midterm<br />
** Final<br />
* Analyze questionnaires<br />
* Rework example profiles<br />
* Assis will use CDD to get a feel for it and create a test suite for the students to start with<br />
<br />
===Unassigned===<br />
<br />
* Cache debug values when extracting several test cases.<br />
* Enable execution and extraction by default for new projects.<br />
* Make CDD Window and CDD Log Window visiable by default<br />
* "Debug selected test routine" should be grayed out if no test case is currently selected<br />
* Testing V2 Application should not interupt flow<br />
* Retest if test cases with errors are properly ignored (after 6.1 port)<br />
* Extraction for inline agents not currently working (at least not always)<br />
** Create inline agent test case<br />
** Fix extraction for inline agents<br />
* Revive system level test suite<br />
* There is a performance problem when compiling a test suite for 'ec' itself. There are indications that the parent ec process consumes too much CPU cycles. Maybe the CDD output refresh is too eager. Find out what the problem is and fix it.<br />
<br />
====Beta Tester Feedback====<br />
* It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?)<br />
* There should be UI support for deletion of Test Case<br />
<br />
== Questionnaires ==<br />
<br />
* Use ELBA<br />
<br />
== Software Engineering Project ==<br />
<br />
* One large project, but divided into testable subcomponents<br />
* Students required to write test cases<br />
* Fixed API to make things uniformly testable<br />
* Public/Secret test cases (similar to Zeller course)<br />
* Competitions:<br />
** Group A test cases applied to Group A project<br />
** Group A test cases applied to Groupt B project<br />
<br />
* Idea how to cancel out bias while allowing fair grading:<br />
** Subtasks 1 and 2, Students divided into groups A and B<br />
** First both groups do 1, A is allowed to use tool, B not<br />
** Then both groups do 2, B is allowed to use tool, A not<br />
** Bias cancelation:<br />
*** Project complexity<br />
*** Experience of students<br />
*** Experience gained in first subtask, when developing second<br />
*** Risk: One task might be better suited for the tool than the other<br />
<br />
== Data to harvest ==<br />
* IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled<br />
* Test Case Source (just final version, or all versions?)<br />
** Use Profiler to get coverage approximation<br />
* TC Meta Data (with timestamps -> Evolution of Test Case)<br />
** TC Added/Removed/Changed<br />
** TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])<br />
** TC execution time<br />
** Modificiations to a testcase (compiler needs to recompile)<br />
* Development Session Data<br />
** IDE Startup<br />
** File save<br />
* Questionnairs<br />
** Initial<br />
** Final<br />
<br />
<br />
== Logging ==<br />
<br />
* "Meta" log entries<br />
** Project opened (easy)<br />
** CDD enable/disable (easy)<br />
** general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)<br />
<br />
* CDD actions log entries<br />
** Compilation of interpreter (start, end, duration)<br />
** Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)<br />
** Extraction of new test case (extraction time)<br />
<br />
* Test Suite Status<br />
** Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)<br />
** Test class: (do we need info on this level)<br />
** Test routine: status (basically as you see it in the tool)<br />
<br />
==Experiment Hypotheses==<br />
<br />
==Do Contracts improve Tests?==<br />
<br />
* Is there a correlation between Tests quantity or quality and the quantity or quality of contracts?<br />
<br />
== Corellation between failure/fault type and test type? ==<br />
<br />
* Do certain kind of tests find certain kind of failures/faults?<br />
<br />
===Use of CDD increases development productivity===<br />
* Did the use of testing decrease development time?<br />
<br />
* Meassures:<br />
** Number of compilations<br />
** Number of saves<br />
** Number of revisions<br />
** IDE time<br />
** Asking the students<br />
<br />
Emphasis on quetionnair result. Correlation with logs only if it makes sense<br />
<br />
===Use of CDD increases code correctness===<br />
* Is there a relation between code correctness of project (vs. some system level test suite) and test activity?<br />
<br />
* Measures:<br />
** number of tests<br />
** number of times test were run<br />
** Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)<br />
** Secret test suite<br />
<br />
===Developer Profile: Is there a correlation between Developer Profile and the way they use testing tools===<br />
* How did students use the testing tools?<br />
* Are ther clusters of similar use? <br />
* What is charactersitic for these clusters?<br />
* Meassures:<br />
** Aksing students before and after<br />
** Are there projects where tests initially always fail resp. pass<br />
** How often do they test?<br />
** How correct is their project?<br />
<br />
Midterm questionnaire will be used to phrase questions for final questionnaire.<br />
====Example profiles====<br />
* Waldundwiesen Hacker<br />
** No explicit structure. Does whatever seems appriorate at the time. No QA plan.<br />
* Agile<br />
** Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.<br />
* Waterfall inspired<br />
** Explicit process model. Phases don't interleave.<br />
* ?<br />
<br />
===How do extracted, synthesized and manually written test cases compare?===<br />
* Which tests are the most useful to students?<br />
* How many tests are there in each category?<br />
* What's the test suite quality of each category?<br />
* Were some excluded from testing more often than others?<br />
* How many red/green and green/red transitions are there in each category?<br />
* Which had compile-time errors most often that did not get fixed?<br />
* Meassures:<br />
** LOC<br />
** Number of tests<br />
** Number of executions<br />
** Outcome transitions</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10560CddBranch2008-02-05T11:44:01Z<p>Mogh: /* Documentation */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Documentation ==<br />
<br />
* Download CDD Beta 2<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_beta_2.tar.gz (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Patch: http://n.ethz.ch/~moris/download/Eiffel61_cdd_beta_2_patch.zip (follow the instructions of the readme.txt contained in the archive)<br />
*** Installer: http://n.ethz.ch/~moris/download/Eiffel_61_gpl_cdd_beta2-windows.msi (Note that the installer will overwrite an existing installation of Eiffel Studio 6.1.) <br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 18.02.2008: Final CDD release for experiment online<br />
* 19.02.2008: Initial Questionnaire<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10551CddBranch2008-02-04T22:45:47Z<p>Mogh: /* Documentation */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Documentation ==<br />
<br />
* Download CDD Beta 2<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_beta_2.tar.gz (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Patch: http://n.ethz.ch/~moris/download/Eiffel61_cdd_beta_2_patch.zip (follow the instructions of the readme.txt contained in the archive)<br />
*** Installer: (TODO) (Note that the installer will overwrite an existing installation of Eiffel Studio 6.1.) <br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 18.02.2008: Final CDD release for experiment online<br />
* 19.02.2008: Initial Questionnaire<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CddBranch&diff=10550CddBranch2008-02-04T22:41:48Z<p>Mogh: /* Documentation */</p>
<hr />
<div>[[Category:Testing]]<br />
[[Category:EiffelDebugger]]<br />
[[Category:CDD]]<br />
<br />
<br />
== What is the CDD extension for EiffelStudio? ==<br />
CDD (short for Contract Driven Development) is a project developed at ETH Zurich. The extension adds full support for unit testing to EiffelStudio. It also introduces the new idea of extracting test cases automatically from failures observed via the debugger. The following lists the main features of CDD:<br />
<br />
* Automated extraction of test cases from failures (For every exception thrown a new test case is created)<br />
* Visualization of test cases and their outcomes<br />
* One button creation of manual test cases<br />
* Automated execution of test cases in the background<br />
* Limit visible test cases via predefined filters and custom tags<br />
* Testing occurs in the background and is undisruptive to the developer<br />
* Easy test case management through tags<br />
<br />
If you have questions, feedback, or would like to report a bug please visit the [[http://eiffelstudio.origo.ethz.ch/forum/20| CDD forum]]<br />
<br />
<br />
[[Image:video_still.png|center]]<br />
<center><br />
<h3><br />
[http://se.ethz.ch/people/leitner/cdd/video Play Video!]<br />
</h3><br />
(TODO: Update with more recent screenshot and video)<br />
</center><br />
<br />
== Documentation ==<br />
<br />
* Download CDD Beta 2<br />
** Linux: http://se.ethz.ch/people/leitner/cdd/Eiffel61_cdd_beta_2.tar.gz (Download the above file and install it just like you would install a EiffelStudio tar ball. Afterwards proceed to section "Using CDD".)<br />
** Windows<br />
*** Patch: http://n.ethz.ch/~moris/download/EiffelStudio_6.1_CDD_BETA_2_patch.zip (follow the instructions of the readme.txt contained in the archive)<br />
*** Installer: (TODO) (Note that the installer will overwrite an existing installation of Eiffel Studio 6.1.) <br />
* [[Using CDD]]<br />
* [[CDD Common Problems|Common Problems]]<br />
<br />
== Old Documentation == <br />
<br />
Documentation for the release of CDD for EiffelStudio version 5.7 is available from [[CddOldDocumentation]].<br />
<br />
= Project Internal Stuff =<br />
<br />
== Schedule ==<br />
<br />
<br />
* 04.01.2008: Final experiment definition (questions to ask, how to conduct experiment)<br />
* 08.01.2008: Finalized list of features to go in release (including logging and log submission)<br />
* 27.01.2008: Beta 1 (feature complete version online)<br />
* 04.02.2008: Beta 2 (designated testers test release, # > 3)<br />
* 11.02.2008: Beta tester feedback in<br />
* 18.02.2008: Final CDD release for experiment online<br />
* 19.02.2008: Initial Questionnaire<br />
* 25.03.2008: Midterm Questionnaire<br />
* 19.05.2008: Final Questionnaire<br />
* 20.05.2008: Having all data<br />
* 06.06.2008: Finished analysis<br />
<br />
== Stefans Master Plan == <br />
<br />
* MA Start ca 17.12.2007<br />
* MA End ca 17.6.2008<br />
<br />
* Testing the tester<br />
** System level test for CDD (incl. framework)<br />
** Recreating existing unit test suite with CDD<br />
** Large scale validation of CDD<br />
*** Info 4 and/or Software Engineering<br />
*** Questions<br />
**** Does testing (manual/extracted) increase developer productivity?<br />
**** How many tests do ppl end up with (manual/extracted)?<br />
**** ...<br />
<br />
== Various ==<br />
* Regression testing: [[CddRegressionTesting]]<br />
* TreeView Specification: [[CddTreeViewSpec]]<br />
* [[CDDHowtoRollARelease]]<br />
<br />
=== Things we need from estudio ===<br />
* Invariants should be checked during debugging equally to pre- and post conditions (they could also be visualised in the flat view the same way like pre- and post conditions are)<br />
* The information whether some call is a creation call or a normal routine call (Not sure if this is really necessary, what if we assume every call to some creation procedure is always a creation call?)<br />
* Support for multiple open targets</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10535CDDHowtoRollARelease2008-02-04T15:18:35Z<p>Mogh: /* How to roll a release on Windows */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* cdd base library classes -> Src/library/base/ise/support/cdd<br />
* the manual_test_class.cls file -> Delivery/studio/help/defaults<br />
* the new 16x16.png -> Delivery/studio/bitmaps/png/<br />
* the cdd examples folder -> Src/examples/cdd<br />
* Splash screens -> Delivery/studio/bitmaps/png/splash_shadow.png Delivery/studio/bitmaps/png/splash.png<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
*** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
**** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
**** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
**** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
*** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
*** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
**** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
**** <INSTALL_DIR>/releases/enterprise_version/ needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Moghhttps://dev.eiffel.com/index.php?title=CDDHowtoRollARelease&diff=10534CDDHowtoRollARelease2008-02-04T15:17:26Z<p>Mogh: /* How to roll a release on Windows */</p>
<hr />
<div>[[Category:CDD]]<br />
== List of CDD specific files ==<br />
The following files have to be overwritten or added to a stock EiffelStudio release:<br />
* cdd base library classes -> Src/library/base/ise/support/cdd<br />
* the manual_test_class.cls file -> Delivery/studio/help/defaults<br />
* the new 16x16.png -> Delivery/studio/bitmaps/png/<br />
* the cdd examples folder -> Src/examples/cdd<br />
* Splash screens -> Delivery/studio/bitmaps/png/splash_shadow.png Delivery/studio/bitmaps/png/splash.png<br />
<br />
== How to roll a release on Windows ==<br />
* prepare some directory <INSTALL_DIR> like this:<br />
** create 3 subdirectories <INSTALL_DIR>/EiffelStudio, <INSTALL_DIR>/gcc, <INSTALL_DIR>/releases. Fill them according to the following description<br />
** [<INSTALL_DIR>/EiffelStudio] has to contain a complete delivery without the ec.exe binaries. Without a working delivery script, this can be achieved by "patching" an official installation:<br />
*** copy the contents of the installation directory of a fresh offcial installation into <INSTALL_DIR>/EiffelStudio (fresh = NO EIFGENs produced. To be sure, uninstall the official version if alrady existing, manually delete the remains in the installation directory, and reinstall it WITHOUT building any precompilations)<br />
*** Remove the ec.exe from <INSTALL_DIR>/EiffelStudio/studio/spec/windows/bin<br />
*** Replace/Add the cdd specific files (see list above). Watch out for the proper place to add/replace them (the above list is svn-specific)<br />
** [<INSTALL_DIR>/gcc] needs to contain content of <svn-eiffel-branch>/free_add_ons/gcc<br />
** [<INSTALL_DIR>/releases] needs to contain the ec.exe binaries (the cdd versions of course)<br />
*** <INSTALL_DIR>/releases/gpl_version/ needs to contain ec.exe<br />
*** <INSTALL_DIR>/releases/enterprise_version needs to contain ec.exe (take the same ec.exe, it's a dummy for using the scripts later on)<br />
<br />
* let env variable %INSTALL_DIR% point to the <INSTALL_DIR><br />
* let env variable %INIT_DIR% point to the <svn-eiffel-branch>/Delivery/scripts/windows folder<br />
* finalize the "hallow" tool (<svn-eiffel-branch>/Src/tools/hallow/hallow.ecf)<br />
* create if not exists directory %INIT_DIR%/install/bin<br />
* copy content of <svn-eiffel-branch>/Src/tools/hallow/EIFGENs/hallow/F_code to INIT_DIR/install/bin<br />
* create if not exists directory %INIT_DIR%/install/binaries/x86<br />
* get a proper setup.dll (from manus probably, or build yourself, instructions for this will be added soon) and put it into %INIT_DIR%/install/binaries/x86/<br />
* start command line, go to %INIT_DIR%/install/content/eiffelstudio and run:<br />
** nmake /nologo clean<br />
** nmake /nologo<br />
** nmake /nologo gpl_x86<br />
* wait some minutes .... and pray :-)</div>Mogh