- 1 CDD Meeting, Tuesday, 11.1.2008, 10:00
- 1.1 Next Meeting
- 1.2 Tasks
- 1.3 Software Engineering Project
- 1.4 Data to harvest
- 1.5 Experiment Hypotheses
CDD Meeting, Tuesday, 11.1.2008, 10:00
- Friday, 17.1.2008, 14:00
- Forumulate Experiment Hypothesis (Andreas)
- Fix AutoTest for courses
- Integrate AUT_TEST_CASE into CDD_TEST_CASE hierarchy
- Variable declaration for failing test cases
- New release
- Right click clears CDD log window
- Add CDD IDE log entry when new test case is extracted
- Implement "New Manual Test Case" Button
- Better Icons for GUI (Arno)
- Status bar (Arno)
- Restore open nodes and selection after grid update (Arno)
- Maybe better/easier solved via incremental updates from tree
- Add failure context window (Arno)
- Maybe also additional information such as previous outcomes?
- Check why Gobo slows down compilation of project not using gobo when melting (performance issue for compiling interpreter)
- Environment variable (or better user preference) for qualifying class names (to avoid svn conflicts)
- Add filters and tags for extracted, manual tests and synthesized tests
- Look at/fix test case execution for agents
- Second Chance re-run to find true prestate (with Jocelyn)
- Allow for test case extraction of passing routine invocations (with Jocelyn)
- What data to log?
- Implement storing
- Define how students should submit logs
- Data Gathering
- Define what data to gather
- Define how to process gather data
- Automate CDD System level tests
- Add most important convenience routine to CDD_TEST_CASE (Stefan)
- Move logs below cdd_tests
- Uniqe id to tag test cases with. To be used in logs. So test logs are resiliant to test class renamings
- While extracting test cases, flag objects that are target to a currently executing routine
- During setup check inv of all objects that are not flaged
- Define Project for SoftEng (due by next meeting)
- Find System level test suite for us to test students code
- Find project with pure functional part
- Install CDD in student labs (Manu)
- Free form mid-term questions on how students use CDD (will help us define usage groups and questions for final quenstionnair)
- Port to 6.1 (?, probably only after Beta 1)
- Filter should be case insensitive
- Log when test case has changed
- Add info to indexing clause
- "This class has been automatically created by CDD"
- "Visit ... to learn more about extracted test cases"
- Creation date
- Write documentation and videos tutorials
- Build releasable delivery on Windows and Linux
- Rebuilding manual test suite through extraction and synthesizing
- Devise questionnaires
- Analyze questionnaires
Software Engineering Project
- One large project, but divided into testable subcomponents
- Students required to write test cases
- Fixed API to make things uniformly testable
- Public/Secret test cases (similar to Zeller course)
- Group A test cases applied to Group A project
- Group A test cases applied to Groupt B project
Data to harvest
- IDE Time with CDD(extraction) enabled / IDE Time with CDD(extraction) disabled
- Test Case Source (just final version, or all versions?)
- Use Profiler to get coverage approximation
- TC Meta Data (with timestamps -> Evolution of Test Case)
- TC Added/Removed
- TC Outcome (transitions from FAIL/PASS/UNRESOLVED[bad_communication <-> does_not_compile <-> bad_input])
- TC execution time
- Modificiations to a testcase (compiler needs to recompile)
- Development Session Data
- IDE Startup
- File save
Use of CDD increases development productivity
- Did the use of testing decrease development time?
- Number of compilations
- Number of saves
- Number of revisions
- IDE time
- Asking the students
Use of CDD increases code correctness
- Is there a relation between code correctness of project (vs. some system level test suite) and test activity?
- number of tests
- number of times test were run
- Number of pass/fail, fail/pass transitions, (also consider unresolved/* transitions ?)
- How did students use the testing tools?
- Are ther clusters of similar use?
- What is charactersitic for these clusters?
- Aksing students before and after
- Are there projects where tests initially always fail resp. pass
- How often do they test?
- How correct is their project?
Midterm questionnaire will be used to phrase questions for final questionnaire.
- Waldundwiesen Hacker
- No explicit structure. Does whatever seems appriorate at the time. No QA plan.
- Processes interleave. Conscionsness for QA. Maybe even Test First or TDD.
- Waterfall inspired
- Explicit process model. Phases don't interleave.
How do extracted, synthesized and manually written test cases compare?
- Which tests are the most useful to students?
- How many tests are there in each category?
- What's the test suite quality of each category?
- Were some excluded from testing more often than others?
- How many red/green and green/red transitions are there in each category?
- Which had compile-time errors most often that did not get fixed?
- Number of tests
- Number of executions
- Outcome transitions