Difference between revisions of "CddMeeting01112008"

(Use of CDD increases development productivity)
 
(15 intermediate revisions by 2 users not shown)
Line 7: Line 7:
  
 
===Andreas===
 
===Andreas===
 +
* Forumulate Experiment Hypothesis (Andreas)
 +
* Fix AutoTest for courses
 +
** Integrate AUT_TEST_CASE into CDD_TEST_CASE hierarchy
 +
** Variable declaration for failing test cases
 +
** New release
  
 
===Arno===
 
===Arno===
Line 16: Line 21:
 
** http://tango.freedesktop.org/Tango_Icon_Library
 
** http://tango.freedesktop.org/Tango_Icon_Library
 
* Status bar (Arno)
 
* Status bar (Arno)
 +
* Restore open nodes and selection after grid update (Arno)
 +
** Maybe better/easier solved via incremental updates from tree
 +
* Add failure context window (Arno)
 +
** Maybe also additional information such as previous outcomes?
 +
* Check why Gobo slows down compilation of project not using gobo when melting (performance issue for compiling interpreter)
 +
* Environment variable (or better user preference) for qualifying class names (to avoid svn conflicts)
  
  
 
===Stefan===
 
===Stefan===
* Add filters and tags for extracted, manual tests and automated tests
+
* Add filters and tags for extracted, manual tests and synthesized tests
 
* Look at/fix test case execution for agents
 
* Look at/fix test case execution for agents
 
+
* Second Chance re-run to find true prestate (with Jocelyn)
===Manu===
+
* Allow for test case extraction of passing routine invocations (with Jocelyn)
 
+
* Logging
===Jocelyn===
+
 
+
* Port to 6.1 (?, probably only after Beta 1)
+
* Manual re-run to find true prestate (Jocelyn, Stefan)
+
* Logging (Stefan)
+
 
** What data to log?
 
** What data to log?
 
** Implement storing
 
** Implement storing
 
** Define how students should submit logs
 
** Define how students should submit logs
* Data Gathering (Stefan)
+
* Data Gathering
 
** Define what data to gather
 
** Define what data to gather
 
** Define how to process gather data
 
** Define how to process gather data
* Forumulate Experiment Hypothesis (Andreas)
+
* Automate CDD System level tests  
* Define Project for SoftEng (Manu)
+
** Find System level test suite for us to test students code
+
** Find project with pure functional part
+
* "Execute visible test cases only" Button (?)
+
* Restore open nodes and selection after grid update (Arno)
+
** Maybe better/easier solved via incremental updates from tree
+
* Automate CDD System level tests (Stefan)
+
* Install CDD in student labs (Manu)
+
* Pause test execution and compilation during regular compilation and execution (Arno)
+
 
* Add most important convenience routine to CDD_TEST_CASE (Stefan)
 
* Add most important convenience routine to CDD_TEST_CASE (Stefan)
* Add failure context window (Arno)
 
** Maybe also additional information such as previous outcomes?
 
* Check why Gobo slows down compilation of project not using gobo when melting (performance issue for compiling interpreter)
 
* Fix AutoTest for courses
 
** Integrate AUT_TEST_CASE into CDD_TEST_CASE hierarchy
 
** Variable declaration for failing test cases
 
** New release
 
 
* Move logs below cdd_tests
 
* Move logs below cdd_tests
* Environment variable (or better user preference) for qualifying class names (to avoid svn conflicts)
 
 
* Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings
 
* Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings
 
* While extracting test cases, flag objects that are target to a currently executing routine
 
* While extracting test cases, flag objects that are target to a currently executing routine
 
* During setup check inv of all objects that are not flaged
 
* During setup check inv of all objects that are not flaged
 +
 +
===Manu===
 +
* Define Project for SoftEng (due by next meeting)
 +
** Find System level test suite for us to test students code
 +
** Find project with pure functional part
 +
* Install CDD in student labs (Manu)
 +
* Free form mid-term questions on how students use CDD (will help us define usage groups and questions for final quenstionnair)
 +
 +
===Unassigned===
 +
* Port to 6.1 (?, probably only after Beta 1)
 +
* Filter should be case insensitive
 +
* Log when test case has changed
 +
* Add info to indexing clause
 +
** "This class has been automatically created by CDD"
 +
** "Visit ... to learn more about extracted test cases"
 +
** Creation date
 +
* Write documentation and videos tutorials
 +
* Build releasable delivery on Windows and Linux
 +
* Rebuilding manual test suite through extraction and synthesizing
 +
* Devise questionnaires
 +
** Initial
 +
** Midterm
 +
** Final
 +
* Analyze questionnaires
  
 
== Software Engineering Project ==
 
== Software Engineering Project ==
Line 89: Line 102:
 
===Use of CDD increases development productivity===
 
===Use of CDD increases development productivity===
 
* Did the use of testing decrease development time?
 
* Did the use of testing decrease development time?
* This can be meassured by either looking at
+
 
 +
* Meassures:
 
** Number of compilations
 
** Number of compilations
 
** Number of saves
 
** Number of saves
Line 95: Line 109:
 
** IDE time
 
** IDE time
 
** Asking the students
 
** Asking the students
 
None of the above strikes me as particualry reliable though. Also, it is easy to develop quickly if you do a bad job.
 
In order to compare apples to apples we must be careful to compare projects with a similar correcntess and completeness. We could use an external test suite to assess correctness, or the grade of the students.
 
 
  
 
===Use of CDD increases code correctness===
 
===Use of CDD increases code correctness===
 
* Is there a relation between code correctness of project (vs. some system level test suite) and test activity?
 
* Is there a relation between code correctness of project (vs. some system level test suite) and test activity?
  
Measures for test activity:
+
* Measures:
* number of tests
+
** number of tests
* number of times test were run
+
** number of times test were run
* Number of pass/fail, fail/pass transitions
+
** Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)
  
 
===Developer Profile===
 
===Developer Profile===
* How did students use the testing tools.
+
* How did students use the testing tools?
 
* Are ther clusters of similar use?  
 
* Are ther clusters of similar use?  
 
* What is charactersitic for these clusters?
 
* What is charactersitic for these clusters?
Line 118: Line 128:
 
** How correct is their project?
 
** How correct is their project?
  
I am not completely sure yet what to assess here.
+
Midterm questionnaire will be used to phrase questions for final questionnaire.
 +
====Example profiles====
 +
* Waldundwiesen Hacker
 +
** No explicit structure. Does whatever seems appriorate at the time. No QA plan.
 +
* Agile
 +
** Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.
 +
* Waterfall inspired
 +
** Explicit process model. Phases don't interleave.
 +
* ?
  
 
===How do extracted, synthesized and manually written test cases compare?===
 
===How do extracted, synthesized and manually written test cases compare?===
Line 127: Line 145:
 
* How many red/green and green/red transitions are there in each category?
 
* How many red/green and green/red transitions are there in each category?
 
* Which had compile-time errors most often that did not get fixed?
 
* Which had compile-time errors most often that did not get fixed?
 +
* Meassures:
 +
** LOC
 +
** Number of tests
 +
** Number of executions
 +
** Outcome transitions

Latest revision as of 04:51, 17 January 2008

CDD Meeting, Tuesday, 11.1.2008, 10:00

Next Meeting

  • Friday, 17.1.2008, 14:00

Tasks

Andreas

  • Forumulate Experiment Hypothesis (Andreas)
  • Fix AutoTest for courses
    • Integrate AUT_TEST_CASE into CDD_TEST_CASE hierarchy
    • Variable declaration for failing test cases
    • New release

Arno

  • Right click clears CDD log window
  • Add CDD IDE log entry when new test case is extracted
  • Implement "New Manual Test Case" Button
  • Better Icons for GUI (Arno)
  • Status bar (Arno)
  • Restore open nodes and selection after grid update (Arno)
    • Maybe better/easier solved via incremental updates from tree
  • Add failure context window (Arno)
    • Maybe also additional information such as previous outcomes?
  • Check why Gobo slows down compilation of project not using gobo when melting (performance issue for compiling interpreter)
  • Environment variable (or better user preference) for qualifying class names (to avoid svn conflicts)


Stefan

  • Add filters and tags for extracted, manual tests and synthesized tests
  • Look at/fix test case execution for agents
  • Second Chance re-run to find true prestate (with Jocelyn)
  • Allow for test case extraction of passing routine invocations (with Jocelyn)
  • Logging
    • What data to log?
    • Implement storing
    • Define how students should submit logs
  • Data Gathering
    • Define what data to gather
    • Define how to process gather data
  • Automate CDD System level tests
  • Add most important convenience routine to CDD_TEST_CASE (Stefan)
  • Move logs below cdd_tests
  • Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings
  • While extracting test cases, flag objects that are target to a currently executing routine
  • During setup check inv of all objects that are not flaged

Manu

  • Define Project for SoftEng (due by next meeting)
    • Find System level test suite for us to test students code
    • Find project with pure functional part
  • Install CDD in student labs (Manu)
  • Free form mid-term questions on how students use CDD (will help us define usage groups and questions for final quenstionnair)

Unassigned

  • Port to 6.1 (?, probably only after Beta 1)
  • Filter should be case insensitive
  • Log when test case has changed
  • Add info to indexing clause
    • "This class has been automatically created by CDD"
    • "Visit ... to learn more about extracted test cases"
    • Creation date
  • Write documentation and videos tutorials
  • Build releasable delivery on Windows and Linux
  • Rebuilding manual test suite through extraction and synthesizing
  • Devise questionnaires
    • Initial
    • Midterm
    • Final
  • Analyze questionnaires

Software Engineering Project

  • One large project, but divided into testable subcomponents
  • Students required to write test cases
  • Fixed API to make things uniformly testable
  • Public/Secret test cases (similar to Zeller course)
  • Competitions:
    • Group A test cases applied to Group A project
    • Group A test cases applied to Groupt B project

Data to harvest

  • IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled
  • Test Case Source (just final version, or all versions?)
    • Use Profiler to get coverage approximation
  • TC Meta Data (with timestamps -> Evolution of Test Case)
    • TC Added/Removed
    • TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])
    • TC execution time
    • Modificiations to a testcase (compiler needs to recompile)
  • Development Session Data
    • IDE Startup
    • File save
  • Questionnairs
    • Initial
    • Final

Experiment Hypotheses

Use of CDD increases development productivity

  • Did the use of testing decrease development time?
  • Meassures:
    • Number of compilations
    • Number of saves
    • Number of revisions
    • IDE time
    • Asking the students

Use of CDD increases code correctness

  • Is there a relation between code correctness of project (vs. some system level test suite) and test activity?
  • Measures:
    • number of tests
    • number of times test were run
    • Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)

Developer Profile

  • How did students use the testing tools?
  • Are ther clusters of similar use?
  • What is charactersitic for these clusters?
  • Meassures:
    • Aksing students before and after
    • Are there projects where tests initially always fail resp. pass
    • How often do they test?
    • How correct is their project?

Midterm questionnaire will be used to phrase questions for final questionnaire.

Example profiles

  • Waldundwiesen Hacker
    • No explicit structure. Does whatever seems appriorate at the time. No QA plan.
  • Agile
    • Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.
  • Waterfall inspired
    • Explicit process model. Phases don't interleave.
  •  ?

How do extracted, synthesized and manually written test cases compare?

  • Which tests are the most useful to students?
  • How many tests are there in each category?
  • What's the test suite quality of each category?
  • Were some excluded from testing more often than others?
  • How many red/green and green/red transitions are there in each category?
  • Which had compile-time errors most often that did not get fixed?
  • Meassures:
    • LOC
    • Number of tests
    • Number of executions
    • Outcome transitions