Difference between revisions of "CddMeeting 13 02 2008"

Latest revision as of 04:42, 7 April 2008

1 CDD Meeting, Wednesday, 14.02.2008, 10:00

CDD Meeting, Wednesday, 14.02.2008, 10:00

Next Meeting

15.02.2008; 14:15

Tasks

Andreas

Forumulate Experiment Hypothesis (Andreas)
Fix AutoTest for courses
- New release
Write documentation and videos tutorials (together with final release)
Finish tuple_002 test case
[delayed] Retest if test cases with errors are properly ignored (after 6.1 port)
[done] Add timeout judgement
Build releasable delivery for Linux (after each Beta I guess...)

Arno

Bug Fixing

[ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)

Ilinca

Integrate variable declarations into AutoTest trunk (by 8.2.2008)

Stefan

[RECURRENT] Build releasable delivery on Windows
[RECURRENT, currently valid]Check log for XML validity

Opening Geneaology solution makes CDD ES freeze.
FG Debugging Crash
- Wenn ich im CDD Eiffel den Debug the selected test case button drücke und dann während dem debuggen eine assertion violation auftritt, crasht das ganze Eiffel Studio. Das ist mit der CDD Version die ich am Samstag frisch vom SVN server gebaut habe.
When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)

Make newly extracted test cases show up "expanded" in GUI treeview
Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)
Make general encoding/decoding routines for special feature names
- e.g. infix "+" (ES) <=> infix_plus (CDD)
Cache debug values when extracting several test cases.
Allow for test case extraction of passing routine invocations (with Jocelyn)
Rebuilding manual test suite through extraction and synthesizing
Find performance bottleneck of test case extraction and propose extraction method for second chance
[LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)
[LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.

Bugs/Things to look at

For big projects (like ES itself) background compilation of the interpreter leads to completely unresponsive ES
[NO EASY SOLUTION POSSIBLE] Crash upon closing of EiffelStudio (feature call on void target in breakpoint tool)

Manu

Install CDD in student labs (Manu)
Devise questionnaires
- Initial (due next meeting after Manu's vacation)
- Midterm
- Final
Analyze questionnaires
Rework example profiles
Assis will use CDD to get a feel for it and create a test suite for the students to start with

Bernd

Define Project for SoftEng
- Find test suite for us to test students code
- Find project with pure functional part

Unassigned

Beta Tester Feedback

(Please put your name so we can get back to you in the case of questions)

(Jocelyn) It should be possible to set the location of the cdd_tests directory (what if location of .ecf file is not readable?) [delayed]
- home directory? application_data directory? [Jocelyn]
(Jocelyn) There should be UI support for deletion of Test Case [difficult for test routines, possible for classes. documentation added.]
(Jocelyn) [BUG] the manual test case creation dialog should check if class with chosen name is already in the system [delayed]
(Jocelyn) It would be nice if there was a way to configure the timeout for the interpreter [Stefan will ask Jocelyn for concrete example]

Questionnaires

Use ELBA

Software Engineering Project

Task 1: Implement VCard API
Task 2: Implement Mime API
Task 3: Write test cases to reveal faults in foreign VCard implementations
Task 4: Write test cases to reveal faults in foreign Mime implementations

Group A:
- Task 1, Manual Tests
- Task 2, Extracted Tests
- Task 3, Manual Tests
- Task 4, Extracted Tests
Group B:
- Task 1, Extracted Tests
- Task 2, Manual Tests
- Task 3, Extracted Tests
- Task 4, Manual Tests

One large project, but divided into testable subcomponents
Students required to write test cases
Fixed API to make things uniformly testable
Public/Secret test cases (similar to Zeller course)
Competitions:
- Group A test cases applied to Group A project
- Group A test cases applied to Groupt B project

Idea how to cancel out bias while allowing fair grading:
- Subtasks 1 and 2, Students divided into groups A and B
- First both groups do 1, A is allowed to use tool, B not
- Then both groups do 2, B is allowed to use tool, A not
- Bias cancelation:
  - Project complexity
  - Experience of students
  - Experience gained in first subtask, when developing second
  - Risk: One task might be better suited for the tool than the other

Data to harvest

IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled
Test Case Source (just final version, or all versions?)
- Use Profiler to get coverage approximation
TC Meta Data (with timestamps -> Evolution of Test Case)
- TC Added/Removed/Changed
- TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])
- TC execution time
- Modificiations to a testcase (compiler needs to recompile)
Development Session Data
- IDE Startup
- File save
Questionnairs
- Initial
- Final

Logging

"Meta" log entries
- Project opened (easy)
- CDD enable/disable (easy)
- general EiffelStudio action log entries for Developer Behaviour (harder... what do we need??)

CDD actions log entries
- Compilation of interpreter (start, end, duration)
- Execution of test cases (start, end, do we need individual duration of each test cases that gets executed?)
- Extraction of new test case (extraction time)

Test Suite Status
- Test suite: after each refresh log list of all test cases (class level, needed because it's not possible to know when manual test cases get added...)
- Test class: (do we need info on this level)
- Test routine: status (basically as you see it in the tool)

Experiment Hypotheses

Comparisons

	contracts	man. tests	playing	extr.	synth.
	A	B	A B
	A B		A	B
	A B	A
experiment	A B	A	A B	A
		A	B
		A		B
	A B			A	B
				A	B
ISSTA	A B	A			B
		A			B

How reliably can we extract test cases that reproduce the original failure?

Log original exception and exception received from first test execution

Are the extracted tests useful for debugging?

Ask developers, using CDD

What is the (time and memory) overhead of enabling extraction?

What is the size of the extracted test cases?

Are we able to reproduce bugs from industry?

Take bug repository (say EiffelStudio, Gobo, eposix, ...). Use buggy version, try to reproduce bug, see if extracted test case is good.

Does it make a difference in the quality of the code, whether one tests manually or extracts them?

Compare projects using extracted tests and manual tests to ref test suite

Do contracts replace traditional testing oracles?

Original API without contracts
Run failing test cases (the ones we get from second part) with reference API with contracts
How many times does the contract replace the testing oracle?

Retrieved from "https://dev.eiffel.com/index.php?title=CddMeeting_13_02_2008&oldid=10908"

Category:

@@ Line 20: / Line 20: @@
 ====Bug Fixing====
-* Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)
+* [ON WINDOWS NO CRASH OCCURS] Check why EiffelStudio quits after debugging a test routine and ignoring violations (ask Jocelyn)
-* CDD Output Tool window display is not saved (on initial start after installation it is there. But after closing and reopening it is gone)
 ===Ilinca===
@@ Line 30: / Line 29: @@
 * [RECURRENT] Build releasable delivery on Windows
 * [RECURRENT, currently valid]Check log for XML validity
-* [DONE, temporary solution] Log original exception (make it part of test routine's state)
-* Clean up Logging, add information that allows for associating extracted test case with its "extraced" message
+* Opening Geneaology solution makes CDD ES freeze.
-** Logging all thrown exceptions (no matter whether extraction is enabled)
+* FG Debugging Crash
-** Split log files
+** Wenn ich im CDD Eiffel den Debug the selected test case button drücke und dann während dem debuggen eine assertion violation auftritt, crasht das ganze Eiffel Studio. Das ist mit der CDD Version die ich am Samstag frisch vom SVN server gebaut habe.
-* Fix CDD BP bug (in some cases there is a visible, disabled user bp generated which DOES make the application stop!)
 * When detecting if compilation succeeded, inlude "corrupted" as keyword when parsing output message (as in the "project corrupted. cannot continue message)
-* Refactor CDD Manager-Routine invocation cache-CDD Capturer
-* Allow for test case extraction of passing routine invocations (with Jocelyn)
-* Revive system level test suite
-* Fix config files for system level tests (remove cdd tag)
+* Make newly extracted test cases show up "expanded" in GUI treeview
+* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)
+* Make general encoding/decoding routines for special feature names
+** e.g. infix "+" (ES) <=> infix_plus (CDD)
+* Cache debug values when extracting several test cases.
+* Allow for test case extraction of passing routine invocations (with Jocelyn)
 * Rebuilding manual test suite through extraction and synthesizing
 * Find performance bottleneck of test case extraction and propose extraction method for second chance
+* [LOW PRIORITY since inv prob solved] When runtime of interpreter is severly damaged a popup box shows up on Windows. Popup can be ignored and goes away. This popup does not come up when we shoot down interpreter, but when a faulty instruction is executed. (E.g. the inv bug can trigger this dialog)
+* [LOW PRIORITY since unlikely situation] Test cases with a 'False' (not satisfieable) invariant makes interpreter hang and ES crash. Add test case and fix it.
 ====Bugs/Things to look at====
@@ Line 65: / Line 68: @@
 ===Unassigned===
-* Only execute unresolved test cases once. Disable them afterwards. (Needs discussion)
-* Cache debug values when extracting several test cases.
-* Aborting the C compilation of the interpreter may leave the interpreter EIFGEN in an invalid state.
-* Pressing manual test case button may cause error.
-* Make general encoding/decoding routines for special feature names
-** e.g. infix "+" (ES) <=> infix_plus (CDD)
-* Make newly extracted test cases show up "expanded" in GUI treeview
-* "Cleanup button" deletes filtered unresolved extracted test cases
-* Fix problem where exe of target is non standard (Anders report)