Dear Keith, We regret to inform you that your paper Using Software Maintenance and Evolution on Large Open Source Projects\\ In An Introductory Software Engineering Course was not selected for presentation at ICSME 2016. We received 127 submissions, of which 125 were considered for review. Each paper under consideration received three independent reviews, which were subsequently discussed on-line by members of the program committee. In some cases where a paper was actively discussed, we also asked the leading reviewers to summarize the outcome of the discussion in a meta-review. Following this process, eventually 37 papers (29%) were selected for presentation at ICSME 2016. Despite this news, we still hope to see you at ICSME 2016 in Raleigh, North Carolina, USA this October. In particular, please consider the other tracks at ICSME 2016: · ERA Track: http://icsme2016.github.io/cfp/era-track.html · Industry Track: http://icsme2016.github.io/cfp/industry-track.html · Tool Demos Track: http://icsme2016.github.io/cfp/tool-demos-track.html · Artifacts Track: http://icsme2016.github.io/cfp/artifacts-track.html · Doctoral Symposium: http://icsme2016.github.io/cfp/doctoral-symposium.html · Technology Briefings: http://icsme2016.github.io/cfp/technology-briefings.html Be sure to check http://icsme2016.github.io for the latest news about the conference, and please feel free to contact us with any questions/concerns. Sincerely, Bram Adams and Denys Poshyvanyk ICSME 2016 Program Committee Co-Chairs ----------------------- REVIEW 1 --------------------- PAPER: 74 TITLE: Using Software Maintenance and Evolution on Large Open Source Projects\\ In An Introductory Software Engineering Course AUTHORS: Keith Gallagher, Mark Fioravanti and Alyssa Marcoux ----------- Review ----------- Paper summary The paper describes the authors’ experience with using software maintenance on a large open source project as part of an introductory software engineering course. The experiences described are predominantly positive. Points for the paper: -- the topic of integrating maintenance and evolution activities in academic curricula is important stemming form the societal obligation of the academia to prepare software engineers for the industry. Points against the paper: -- the paper seems to miss several important details (see below) -- motivation with respect to the related work should be improved I really appreciate the idea behind this work but in the current form I cannot support acceptance (see below). Hence, I would like to encourage the authors to revise this paper and resubmit it to one of the forthcoming software engineering/education meetings. Supporting argumentation The paper glosses over a number of important points which make it more difficult to understand and apply the approach. -- Setting: how many students are participating in the course? how many instructors are available to support the course? how much time are the students expected to spend on the course? how many additional courses are the students expected to follow simultaneously? how much time are the instructors expected to spend on the course? how are the six weekly hours distributed over lecture/lab? Obviously, a course with 30 students and multiple instructors is very different from courses with 300 students and one instructor. -- Educational: what are the learning objectives of the course? how does the teacher ensure that they are achieved? In particular, Section VII-B states that the course is about the process and not about the product. How is the process assessed---“audits through out the rest of the term are to check progress, which we know anyway” is vague, and while part of the final deliverables (comments, management issues, summaries of team meetings) can be seen as reflecting the process it is still not clear how the assessment is done. Do you use some kind of rubrics? How are the individual students from the “crashed and burn” teams assessed? Would it be possible to differentiate between different students working together, should this be required by the educational institution? -- Course-specific: ----- how would the instructor know the exact location of the issue identified by the students? Do you expect the instructors to have a perfect knowledge of the systems under study? ----- Discussion of Rational/Intuitive and Extrovert/Introvert reminds of psychological profiling. Are the students expected to participate in some kind of formal psychological assessment? What happens if the students decline to do so? Presentation. -- The paper is not self-contained, e.g., such statement as “link editing and object file management are covered in more depth than in Chapter 3” makes sense only if one knows what is contained in Chapter 3. By the similar token the entire abstract about the Monkeysort program is included in the paper *without* stating what the program does. -- Limitations of the proposed teaching solution are barely discussed: report of the Graduate student assistant (Section IX) is the nice exception. The “Limitations” paragraph in the abstract describes albeit briefly the setting of the course. Motivation and related work. -- While section X presents a nice overview of the related work and stresses the differences with the approach described, this discussion does not provide enough motivation for implementing the approach as proposed. What shortcomings do you see in Rajlich’s team-based project course [42]—yes, in his case the students are graded individually but why is this necessarily a problem? -- Additional papers on project-like work in software engineering curricula: David A. Umphress, T. Dean Hendrix, James H. Cross II: Software Process in the Classroom: The Capstone Project Experience. IEEE Software 19(5): 78-85 (2002) A. T. Chamillard, Kim A. Braun: The software engineering capstone: structure and tradeoffs. SIGCSE 2002: 227-231 Wouter Poncin, Alexander Serebrenik, Mark van den Brand: Mining student capstone projects with FRASR and ProM. OOPSLA Companion 2011: 87-96 Jane Huffman Hayes, Timothy Lethbridge, Daniel Port: Evaluating Individual Contribution Toward Group Software Engineering Projects. ICSE 2003: 622-627 Jari Vanhanen, Timo O. A. Lehtinen, Casper Lassenius: Teaching real-world software engineering through a capstone project course with industrial customers. EduRex@ICSE 2012: 29-32 Questions: -- what is “other work” referred to on p.2? Final exam? -- is the notion of a supplier slice taught in the course or is it merely used to illustrate the notion of a “Performance question”? Minor improvements -- LaTeX issues (em on p.1, him in Fig.1, double quote in the acknowledgements) -- confusion of figures and tables ----------------------- REVIEW 2 --------------------- PAPER: 74 TITLE: Using Software Maintenance and Evolution on Large Open Source Projects\\ In An Introductory Software Engineering Course AUTHORS: Keith Gallagher, Mark Fioravanti and Alyssa Marcoux ----------- Review ----------- ## Summary The authors recount their experience teaching a Software Engineering course structured around a maintenance project rather than a "greenfield" project. Instead of the project being structured around the lectures, the lectures are given based on the needs of the project. The authors argue that the common practice of teaching software engineering "bottom-up" is not ideal, as it inevitably promotes the waterfall model. In their course, the authors let the students work with existing open-source projects and are given the task of selecting and working on a number of bugs from the project's bug tracker. The paper covers the complete course and elaborates about all components of the course, including group composition, project selection, grading and evaluation. ## Pros + focusing on maintenance reflects reality and exposes students to real-world software issues + the student tasks come from real issue tracker databases + having the lectures follow the project is innovative and intriguing + there are many nice ideas about how to reverse of "flip" the usual way of running a software engineering course + Hypothesis that SE topics can better be taught "top-down" instead of "bottom-up" + Paper covers all aspects of the course ## Cons - Confusing structure, no clear narrative - Inconsistent language - There are many typos and grammatical errors - there is no “customer” -- the student teams only address pre-defined issues in the issue tracker database - the course seems to focus very heavily on technical (i.e., programming and development) issues rather than broader software engineering issues - the paper reads a bit like a "dump" of random observations and anecdotes; it's not clear what the take-home message(s) should be - Outcome unclear; evaluation/comparison to similar and other approaches missing, conclusion mentions goals, not outcomes - missing some key references to related work (especially "flipped" courses) ## Detailed remarks I was both excited and frustrated by this paper. I would really like to see it revised and presented, however. The main idea behind this course, inverting the usual relationship between lecture and project (where the project follows the lecture topics, which often and inevitably results in a "waterfall model approach") is an interesting proposal. The abstract promises to answer what happens when this inversion is done in an introductory SE course when the project consists of working on existing open-source projects and when the project is used as a way to cover and learn the usual topics in SE. I like the idea of this way of organising a course. I really, really like the idea of structuring an introductory software engineering course around maintenance tasks, rather than on a greenfield project, but your paper convinced me that I should not do it the way you have described, mostly for practical reasons. No customer: for me, the key difference between "programming" and "software engineering" is that the latter must explicitly take into account the needs of and value towards the customer. By focusing on predefined maintenance tasks taken from an issue tracker you eliminate the most important and challenging aspect of software engineering. Who plays the role of "the customer" in your students' projects? This aspects needs to be discussed in some detail. Too technical: it is clear from your paper that the students get exposed to a large number of nasty technical issues that they must resolve. That is certainly fine and is part of the software engineering story, but it sends the wrong message to the students: "If you get the technical problems under control, then you will be ok". This is just plain wrong. [Aside: I am reminded of a paper I read (and reviewed) some years ago called "Twenty Dirty Tricks to Train Software Engineers" -- https://www.seas.gwu.edu/~mlancast/cs254/ppt/p209-dawson.pdf I encourage you to read and cite it. My problem with that paper is that, if you expose students to a "real" software engineering context, then you don't need to play any "dirty tricks" -- the usual problems will arise by themselves. However, many of the interesting problems have to do with customer communication, so focusing on the technical issues is just plain misleading.] How to implement this in practice? I love the idea of making the lectures follow the project, but I do not see how to do this in practice unless you can afford to dedicate considerable resources to the course. Can you reuse prepared lectures with little adaptation, or do you need to tailor special lectures to the issues at hand? How much effort is required from the professor in charge? From the assistants? What are your tips and advice to make this work smoothly? The structure and style of writing in this paper is confusing for me. While I like the first part of the paper (starting with a basic course and project description, followed by the lab review and examination sections), I often miss a clear narrative in the paper that joins the numerous sections together. Occasionally, whole sections or paragraphs are too isolated. In Sections VI there are 11 (!) subsections describing the digressions during the lectures. They are simple descriptions of what has been covered, but there are no comments on why these topics were selected. Are they selected based on the problems individual teams are facing during the project? Are they pre-selected topics that do not change over the years? I don't see the value of Section VIII (Student Stories). The stories are simply pasted as a list, without any comments or a discussion from the authors. While they are interesting to read, I fail to see how they are relevant to the main statements of this paper. What are you learning from these comments? Do they differ from comments you obtain in different courses (e.g., an introductory SE course that follows a more traditional model)? Sections X and XI, titled "Related Work and Comparison" and "Perspectives and Conclusion" respectively, are seriously lacking. While a wide range of sources are quoted in Section X, there is no comparison made with this work. I often fail to see how the related work is relevant as there is often no comparison with the presented work made at all. Section XI is very short and does not, in my opinion, entail a real conclusion. A short statement about what the course is is made, but no concluding remarks. The sentence "The goal is to establish an environment for learning, not an environment for teaching." describes both the goal of the course and this paper, and I was hoping to find an answer on whether this goal has been achieved and whether the presented course is suitable for an introductory SE course. Due to the lack of evaluation and comparison (e.g. with other courses), these questions were not answered and I'm not sure what the core statement of this paper is. Overall, I think that the paper presents an interesting approach for an introductory SE course. Unfortunately, it is written in far too anecdotal a way. The course is described in detail, but the descriptions often lack reasons for course design decisions. Apart from anecdotal evidence, there is no evaluation of the course and a comparison with other approaches is almost completely missing, and the conclusions are weak at best and do not answer the questions posed in the abstract. What's the take-home message? The paper reads too much like a grab bag of random observations rather than a serious reflection on how to improve the "introduction to software engineering" experience. I also have the feeling the paper was written in a rush. If the paper is accepted, please step back and make explicit the "lessons learned". ## Further remarks p 4. I love the idea of making the mid-term an evaluation of the instructor. p 5. The reference to Weinberg's “Kill That Code!” is both obscure and frustrating. I was not able to track this article down, so a summary would certainly have been welcome. p 5. I don't know MonkeySort and I don't see why this is pedagogically interesting. This whole paragraph is both obscure (not well explained) and unconvincing. The "Related work" section seems kind of random and half-hearted. Are these really the most important references? How do they relate to your work? Nowhere do you mention the trend towards "flipped" courses. Surely this is relevant. See e.g., https://en.wikipedia.org/wiki/Flipped_classroom ## Typos - p 1 "Computer Science [an -> and] Software Engineering students" - I.A.: Missing dot after first paragraph ("... C++"). - "An em environment for learning" -- What's "em"? - p 2 "Compares and contrast with others" -> "Comparisons and contrasts" (it's a list of *things*) - II.B: Line 4: "The response is thus. When ...". Maybe replace dot with colon? - II.B.1): Final sentence is missing closing quotation marks. - III: "The measures are admittedly [course -> coarse]" - III: "Of the 968 total [x total] number of files" - p 3 / Fig 1 heading "him Possible Projects" -- huh? - "Mixing in “one from every tribe and nation” further complicates [^ things]." - "At every [x each] fortnightly audit meeting" - "This person was [Incredibly -> incredibly] involved." - p 4 "it must be uploaded onto the server and compiled [x it] there." - "As the term winds down, we remind them [of they -> that] must present at the final, discussed below." - p 5 "A [x a] badly sourced lecture on team structures" -- I don't understand what you mean by a "badly sourced" lecture. Do you just mean that inadequate literature references are provided? - p 8 "From the perspective of the GSA, there are a number of issues that the assistant must be prepared to support students [^ with] throughout the semester." - "Sometimes students have issues in which one student fixes an issue" -- avoid repetition of "issue" - "they do not document their progress [x it] at all or they record too much detail" - "Students can often be convinced to [x being] [documenting -> document] their progress" - "As [x as] the Internet and open source communities" ----------------------- REVIEW 3 --------------------- PAPER: 74 TITLE: Using Software Maintenance and Evolution on Large Open Source Projects\\ In An Introductory Software Engineering Course AUTHORS: Keith Gallagher, Mark Fioravanti and Alyssa Marcoux ----------- Review ----------- The authors propose a neat idea that I often use in my own studio SE courses: teach SE through maintaining an open source software system. Rather than focusing on breadth in SE, the students learn through experience of maintaining a realistic system. I think this work would be of value to the ICSME community, but in its current form is difficult to evaluate. The authors admit that the idea is not novel, but the relationship to prior work is unclear. The related literature is summarized, but not clearly differentiated. Is it an experience report? Is the work a replication? If so, I would expect to see a more detailed comparison with at least one of the techniques. I would have appreciated less focus on how the course was run and more details about how the course improved student learning on maintenance tasks. Do students understand maintenance or SE skills any better using this model as opposed to a different model? A more formal evaluation, or even more formal qualitative analysis of the anecdotal survey data would make the case study much more convincing. The paper reads more like a stream of consciousness about the class, rather than a more typical ICSME research paper. + interesting approach to SE education + useful details for replication - education outcomes in terms of SE or SoftEvo unclear - contribution & relationship to prior work unclear Typos: - p 1 "Computer Science anD Software Engineering" ------------------------- METAREVIEW ------------------------ PAPER: 74 TITLE: Using Software Maintenance and Evolution on Large Open Source Projects\\ In An Introductory Software Engineering Course The authors describe their experiences running a software engineering course around a maintenance project, where the lectures follow the needs of the project, rather than the other way around. Pros + The focus on maintenance is important and refreshing. + Experience reports on "flipped" courses are welcome. Cons - Presentation lacks a clear narrative. - Important details are skipped over. - The paper is not self-contained. - Learning objectives are not clearly stated. - Important related work is not considered. In summary, although the topic is clearly of interest, the paper needs a serious rewrite to actually be useful to readers interested in pursuing such an approach. We strongly encourage the authors to prepare a major revision for another venue.