How Concept Selection Tools Impact the Development of Creative Ideas in Engineering Design Education

Concept selection has been shown to be the gatekeeper of the product design and development process due to its impact on the quality, cost, and desirability of the final product [1], as well as its impact on the development time and cost of later design stages [2]. During this process, concepts generated earlier in the design process are evaluated, selected, and synthesized into a final solution for further development in order to address the design goal [3,4]. In industry, selecting highly creative concepts increases the likelihood of radical innovation and product success [5], while selecting poor concepts can result in large expenses including redesign costs and production postponement. Because of this, creativity, or ideas that are both of high quality and novelty [6–9], has been considered an essential factor in product development [10] and training engineering students to be creative has become a necessary component of engineering education [11].

While creativity is an important factor throughout the engineering design process, it is typically only emphasized during idea generation in engineering design education [12]. This is problematic, because Rietzschel et al. [13] have pointed out that, “The advantages of having many creative ideas at one's disposal can be easily undone by a suboptimal selection process. Instead of simply making groups more productive, it may be more fruitful to make them more effective in all stages of the creative process.” Therefore, it is important to study what impacts the development of creative ideas after idea generation in order to better educate students in innovation practices. To date, only a few studies have investigated the evolution of creative ideas or the factors that lead to the promotion or filtering of these ideas after idea generation in engineering design education [14–18]. For example, Starkey et al. identified a reduction in the creativity of student design ideas from the idea generation stage to the students' final conceptual design, regardless of the design task being explored [19]. This study, however, did not look at the underlying factors that led to the filtering of these creative ideas or the impact of the concept selection tool used.

The evaluation of the utility of concept selection tools in the engineering design process is important because these tools were developed as a means to aid decision makers in the fuzzy-front end of the design process—a stage of design that is rife with ambiguities and uncertainties [20]. Tools that have been routinely introduced in engineering design courses in order to provide a systematic structure to lead decision making [21] include Pugh Chart [22], quality function deployment [23], and the analytic hierarchy process (AHP) [24,25]. These traditional tools have been considered as efficient ways to aid novices (e.g., engineering design students) in identifying sensible concepts and potentially mitigating judgment errors due to their transparency and high repeatability [26,27]. However, they have also been criticized because of their transparency to incite confirmation bias [28] and their inability to measure global creativity (the creativity of an idea with respect to existing products and ideas on the market [29,30]), which may contribute to the lack of innovation in the final product [13,31]. In order to mitigate some deficits of these traditional concept selection tools and provide a more practical tool to measure the global creativity, Gosnell and Miller [32] developed the tool for assessing semantic creativity (TASC) to help a group of novices achieve expert level creativity ratings based on single adjective selection and semantic similarity.

No matter what decision tool is used in the design process, these concept selection instruments can provide only directions or suggestions on the designs to select for further development [33]. In other words, these tools do not make the decision for decision makers. Instead, an imperfect human decision maker makes this decision. Even if a concept selection tool is optimized to identify the most creative (high quality and novelty) ideas within a set, the tool used to inform this decision must be trusted by the individual using this tool. Therefore, when investigating the impact of concept selection tools on the flow of creative ideas, it is important to not only explore the effectiveness of a decision tool for providing recommendations on the selection of creative ideas, but also to identify the concept selection decisions made by human decision makers align with these recommendations.

While there has been practice of applying concept selection tools in engineering education settings, the impact of these tools on students' selection of creative ideas as part of the engineering design process has yet to be investigated. For the purpose of filling this research gap, the current study was developed to understand how the utilization of concept selection tools impact the creativity of ideas selected through an empirical investigation with 60 engineering students in an introduction to engineering design course. Specifically, the students in this study were monitored for their team informal screening, selection, and final conceptual design during an 8-week grade-dependent course project that emphasized creative idea development. Half of the students in the study utilized a traditional concept selection tool (the concept selection matrix (CSM)) while the other half utilized the newly developed TASC method. The results of this study are used to identify the impact of these tools on the creativity of ideas selected by student teams and provide recommendations to educators in the adoption of concept selection tools in classrooms and to researchers on modifying existing concept selection tools or developing new tools.

In order to discover the impact of concept selection tools on student team design outcomes, previous research was explored. This section summarizes this prior work and provides support for the current investigation.

Creativity has been regarded as an essential factor to consider during the product design process in both engineering design education and industry. In order to boost the creativity of ideas generated, researchers have developed ideation tools to encourage students to generate a large amount of ideas (see, for example, Refs. [34–37]). Even though the effectiveness of idea generation can be improved using these tools, a reduction in creativity throughout the design process and the abandonment of novel ideas have been observed in engineering design class projects regardless of the design task being explored [19]. In other words, merely generating creative ideas is not enough to promote the development of creative concepts in an educational context [13]. However, it is unclear what factors may lead to students' abandonment of creativity.

While not specifically studied in the context of creativity throughout the design process, previous research on students' concept selection processes has shown that students often value technical feasibility during the concept selection process [18,38] and select feasible and desirable ideas at the cost of originality [39]. In addition, concept selection has been shown to be largely subject to individual attributes [40], risk taking attitudes [16,41,42], and various cognitive biases and heuristics, such as design fixation [43–45], ownership bias [17,46], and the bias against creativity [47]. Specifically, individuals have been said to have an inherent bias against creative ideas due to the risk and uncertainties of creative ideas [39,48], and their judgment of originality has been found to be negatively related to judgments of appropriateness [49]. It has also been argued that students tend to be less creative in class projects when there is a risk of receiving poor grades [50], since engineering students believe getting good grades is more important than engaging with the learning process [38]. Even though previous research has shown that explicitly instructing students to select creative ideas can improve students' concept selection effectiveness in selecting novel ideas, students in this prior work were not satisfied with the team informal discussion of idea selection and perceived selected ideas with low effectiveness [39].

In order to eliminate the effect the factors mentioned earlier and mitigate the risks and uncertainties associated with the fuzzy front end of the design process [51], many formal concept selection tools have been created to provide a framework to guide concept selection (see, for example, Refs. [22–25]). One concept selection tool that has been heavily integrated throughout engineering education is the CSM [52] due to its transparent decision process and high repeatability [21]. Specifically, CSM was created to aid novices in comparing alternatives through a combination of the criteria developed in the AHP [25,53,54] and through the subjective ratings of candidate ideas [52]. As an example, Drake introduced the method into undergraduate and postgraduate student projects and found it helpful in providing insights of students' reasoning [55]. On one hand, the CSM method helps enhance group consensus and commitment to the decision [56] by structuring a framework to systematically direct the pattern and content of discussion by encouraging team members to analyze the problem as a collaborative unit [57,58]. On the contrary, utilizing the CSM in a group setting has been shown to expose designers to team level biases that have been shown to impact decision making [59], such as conformity to majority opinions [60], the halo effect [61], and social loafing [62]. In addition, the CSM method requires users to setup a problem hierarchy, determine paired comparisons, establish the priorities, and calculate aggregated scores from relative priorities and weights of criteria [25]. Accordingly, the complexities of the CSM method make various implementations problematic in the increasingly fast paced industry [63,64]. The CSM method is also limited by inherent shortcomings of the AHP, such as the restriction to solve hierarchical structured problems [65], rank reversal when adding an irrelevant alternative result [66,67] or an indifferent criterion [68], and the convergence of alternatives [33].

While research on these methods has focused on the benefits and deficits of the effectiveness of their decision-making powers, little research has explored the impact of these concept selection tools on student teams' decision-making process or outcomes. In other words, few researchers have tracked whether students use the recommendations provided by the formal concept selection tools in class projects or how this affects the creativity of the ideas developed throughout this process. Without this information, we do not know how, if at all, concept selection tools play a role in students' abandonment of creativity throughout the conceptual design process. The current study seeks to fill this research void and provide a preliminary understanding of the impact of formal concept selection tools on the quality and novelty of ideas selected by student teams' at several decision points.

In order to overcome the limitations of existing concept selection tools, particularly with reference to ratings of idea creativity, the TASC was developed [15,32]. This method uses natural language processing and semantic similarity to quantify product originality and feasibility by taking advantage of both subjective opinions and computational power [15,32]. Specifically, the TASC method provides a website² to process creativity evaluations which requires each team to upload their candidate ideas (no limitation in the number compared), and each member in the team to select 3–5 adjectives that best describe each idea individually (see Fig. 1 for example) [32]. Once these ratings have been completed, novelty ratings for each idea are calculated by adding the novelty weights for each of the words chosen by each participant for each design idea, where the weights are determined using Wordnet:: Similarity to analyze the semantic similarities between the adjective words selected and the word “innovative.” Similarly, quality ratings for each idea are calculated by adding the quality weights for each of the words chosen by each participant for each design idea based on the semantic similarities between the adjective words selected and the word “feasible.” Importantly, the weights of each of the words selected to rate each idea for novelty and quality are blind to the participants.

The TASC method helps teams make decisions through crowdsourcing or by aggregating individual decisions together without intervention from other team members [69]. In this way, the impact of the group size and group interaction in collective judgments can be prevented, and the time and money required to find, meet, and train skilled raters can be reduced [70]. This type of aggregation is also in line with Amabile's [71] definition of creativity which identifies a product (or idea) as creative to the extent that appropriate observers independently agree it is creative. Prior research has pointed out that the TASC method provides input that allows designers to consider creativity more thoughtfully [15], so that creative ideas can be considered longer in the design process. Moreover, it has been illustrated that aggregate TASC ratings of 11 novice designers can be used to mimic expert ratings [15].

While both traditional concept selection tools like the CSM and new tools like the TASC method can provide design decision-making directions for designers, there are significant differences in their purpose and the underlying approach for informing human decision-making during the design process. For example, the TASC method utilizes semantic creativity ratings leading to evaluations of idea creativity on a global scale as an effort to promote the selection of creative ideas [32]. Both methods, though, can serve as concept selection tools to provide recommendations during the concept selection process and aid designers, especially novices to make decisions.

Even though practical designers prefer intuitive methods better than formal decision-making tools [72–75], these formal tools like the CSM method are often introduced in engineering design classrooms. The transparency of the CSM method's rating process [55] may help students learn and develop trust on the tool, but it may also allow student decision makers to easily manipulate the criteria, weights, and the evaluation results to get the answer they want (confirmation bias [76,77]). This is problematic because researchers have recommended that concept selection tools should be used only as a “decision consultant” and not as a means for deriving the final answer [33]. In addition, this confirmation bias can cause a fixation on initial ideas and block decision from identifying other, perhaps better, design alternatives [72]. Therefore, it is very important to investigate and compare the impact of the CSM method and newly developed tools such as TASC on the decision students make after the concept selection tool ranks an idea set, in order to provide recommendations on how to incorporate concept selection tools in engineering design curricula.

In order to fill this research gap, the current study was developed to compare and contrast the impacts of the CSM method and the TASC method on the quality and novelty of ideas selected by student teams at several decision points throughout the conceptual design process in an engineering design course.

The previous work brought to light that the decision making process in engineering design education on which ideas to keep and which to be abandoned can be impacted by both the concept selection tool utilized as well as the human making the decision. However, the influence of concept selection tools on student team decision-making and design outcomes, particularly as they relate to the creativity of the ideas, is still unclear. Therefore, the current study was developed to answer the following questions:

1. (1)

   How does the average creativity (quality and novelty) of student design teams' ideas change from team informal screening to the final conceptual design? We hypothesized that the average quality (AQL) of the student design team's ideas would increase and that the novelty of the student design team's ideas would decrease during the conceptual design process, since students are more likely to select feasible and desirable ideas [18,78] at the cost of originality [39].

2. (2)

   What impact does the concept selection tool have on the evolution of the creativity (quality and novelty) of a student team's design ideas? We hypothesized that student teams who used the CSM would be more likely to see increases in the feasibility of their design ideas throughout the process over the TASC method since the CSM method specifically assesses the quality of the ideas based on design requirements and technical feasibility [25]. We also hypothesized that the use of the CSM method would have no impact on the novelty of ideas throughout the design process, since the CSM method does not typically include the novelty of ideas as a criterion [25]. On the contrary, we hypothesized that student teams who used the TASC would see an increase in the novelty of their design outcomes throughout the design process, since the TASC method evaluates the creativity of the ideas through measuring both the novelty and quality of ideas [32].

3. (3)

   Do student teams select ideas based on the recommendations of the concept selection tools? We hypothesized that students would be more likely to select ideas recommended by the CSM method over the TASC method due to the transparency of the CSM method in its decision making process [1] and the fact that it has been largely integrated in engineering design education building a larger sense of trust [21]. Alternatively, the TASC method is new and unfamiliar to students [32], and the weights of each of the words selected to rate each idea for novelty and quality is blind to the participants, which may lead students to feel the method is less trustworthy due to a lack of transparency in the decision criterion.

In order to answer these research questions, a study was conducted with two sections of a first-year undergraduate engineering design course with student teams working on the same, graded, 8-week design project. Each section of the course was randomly assigned at the start of the project to one of two conditions: the CSM and the TASC method. Details regarding the methodology used in this study are found in the remainder of this section.

The participants in this study were undergraduate students in a first-year engineering design course taught at a large northeastern university. In all, 60 students (19 females and 41 males) from two sections of the course taught by the same instructor participated in the study, with 30 students in each section. In each section, students formed a total of 8 teams (6 four-member and 2 three-member teams) based on student proficiencies in three-dimensional modeling, sketching, and engineering design. First-year engineering design students were selected because they received little information about the concept selection tools prior to this course so that CSM was not rooted in students' mind.

The design study presented here was a part of a graded 8-week design project conducted in a first-year engineering design course (see Fig. 2 for the timetable of the 8-week project). At the start of the project, students were given the following design problem based on a fictional location:

Suggestions, such as upgrading the locomotive fleet or adopting alternate freight shipping methods, were given at the start of the project (see the Appendix for complete task description). After receiving the problem description, the students performed preliminary research on available transportation options during the first 2 weeks of the project. They used this research to develop design criteria and conduct an AHP analysis to determine the weights of those criteria. Following preliminary data gathering and as part of the current study, students participated in a 20 min individual idea generation session following the rules of brainstorming [79] where they were asked to individually sketch out as many ideas as possible for the freight shipping system and write notes on the sketches to help others understand the concept's features, such as the transportation and the route utilized, see Table 1 for sample ideas and the final conceptual design generated by a student team. Importantly, creativity was emphasized during the individual idea generation session and throughout the course project.

Upon completion of the individual idea generation session, each student team was given 20 min to screen the ideas generated by all team members. During this process, the students were allowed to alter or combine their generated ideas or create new ideas freely. Each team was then instructed to categorize the ideas into two piles: “consider” for ideas that had any useful elements for further development either in whole or in part and “do not consider” for designs that the team no longer wanted to consider as part of the process. The team informal screening process was supposed to simulate the fast screening process in the actual design process in industry in order to reduce the number of ideas that would enter the concept selection tools.

Next, students evaluated ideas from the consider pile using the CSM or TASC method depending on their sections conditional assignment. Specifically, students in the CSM section of the course were given a 10 min instruction on why and how to use the CSM method. After that, the students were asked to go through the ideas in their consider pile and rank those ideas as a team using their previously developed AHP weights and a CSM template in microsoftexcel. In order to use the CSM, students were asked to follow the process set forth by Ulrich and Eppinger [52]. Specifically, the students were asked to rate the candidate ideas based on the criteria previously developed using the AHP using a five-point scale, where one indicated the idea failed to meet the criterion while five indicated the idea successfully met the criterion. Through synthesizing the criteria weights and the corresponding ratings, an overall score was obtained for each candidate idea. While, students in the TASC section of the course were given a 10 min introduction on why and how to use the TASC method and the TASC website.² Then, teams used the TASC website to upload their ideas and rate and rank them individually. See Fig. 3 for an example result using the TASC method. Importantly, the weights of the adjective words in the word bank were blind to students and the calculation of the quality, novelty, and overall creativity scores happened in the back end; therefore, students were not able to change the weights of the adjective words or manipulate the final recommendations. In addition, students in the TASC section were also taught how to calculate customer needs weights using the AHP method as part of the curriculum design. Students in the TASC section, however, did not use or apply these weights during the concept selection process to inform their decision making during the project studied.

Once the evaluation of the candidate ideas was complete, the teams in both sections of the course were given 10 min to discuss the evaluation results and complete a team survey that included three questions: (1) What idea(s) does your team think should be further developed after today's activity; (2) What factors did your team consider when selecting these ideas and; (3) Are the ideas that your team chose to further develop ranked highly in the formal concept selection tool? If not, why did you select these concepts? Specifically, students were instructed that these tools should be used only as a “decision consultant” and not as a means for deriving the final answer [33]. Each student was then asked to individually fill out a survey that consisted of eight Likert scale statements about their experience using their respective concept selection tool (CSM or TASC) and 12 Likert scale statements about their personal preferences on characteristics of candidate ideas; these results were reported in a prior conference proceeding [78]. Finally, at the end of the project, approximately four weeks later, each team was asked to write a report, including a detailed description of their final conceptual design.

In order to quantify the novelty and quality of the ideas at different stages in the conceptual design process, the Shahet al. novelty metrics [9] and Linsey's method [80] to evaluate idea quality were used. Specifically, the average novelty (ANV) and AQL of ideas were calculated at each stage of the concept selection process: team informal screening, team informal discussion, and final conceptual design, see Fig. 4. As a reminder, at the team informal screening stage, all of the ideas that were filtered into the consider pile were evaluated while at the team informal screening stage, all of the ideas that student teams selected for further development as the result of team informal discussion were evaluated. Finally, at the final conceptual design stage, all of the final conceptual designs that were either combined or revised from existing ideas were evaluated. Importantly, the novelty and quality ratings were conducted by two raters: one who had a Ph.D. in an engineering design related field and more than 4 years of experience and one who had completed graduate course work in engineering design and had a minimum of two publications in the field of engineering design creativity. The details of the rating process are described in detail in the remainder of this section.

The novelty of the ideas was defined as how unusual or unexpected an idea is compared to other ideas [9] and was calculated using the Shah et al. novelty metric [9]. Specifically, the two raters used a design rating survey to assess the novelty and of each idea, see the website link³ for full design rating survey question list. This survey helped raters classify features of each design idea addressed, similar to the approach used in previous studies [16,41]. The inter-rater reliability (percent of agreement) between the two raters reached 0.88. Once the ratings were complete, the novelty of the ideas was calculated.

where $T$ is the total number of the ideas and final designs and $Ciisthe$ total number of the ideas and final designs that contain feature $i$⁠.

where $fi$ is the feature novelty of feature $iandn$ is the number of features in total of idea $j$⁠.

where $Nj$ is the novelty of team $p$'s $j$th idea and $m$ is the total number of ideas in team $p$⁠.

where $qcost,j,p$ is the quality of idea $j$ depending on the cost, $qemission,j,p$ is the quality of idea $j$ depending on the emission, $qcapacity,j,p$ is the quality of idea $j$ depending on the capacity, and $qefficiency,j,p$ is the quality of idea $j$ depending on the efficiency.

where $Qj,p$ is the quality of team $p$'s $j$th idea and $m$ is the total number of ideas in team $p$⁠.

During the study, a total of 259 ideas were generated, in which 97 ideas were categorized into consider (mean = 6.06 ideas/team, SD = 1.73) by the student teams with a mean AQL of −0.14 (SD = 0.30) and a mean ANV of 0.68 (SD = 0.14). As a reminder, idea quality ranged from −1 to 1 where positive scores meant that the idea was better than existing solutions, and negative scores meant that it was not as good as existing solutions. Idea novelty, differently, ranged from 0 to 1, where higher scores indicated higher levels of novelty. The remainder of this section presents of our analyses with respect to our research questions.

Our first research question was developed to identify how, or to what effect, the quality and novelty of a student team's idea set changed throughout the course of their design project. We hypothesized that the AQL of student design teams' ideas would increase, while the ANV of design teams' ideas would decrease since students are more likely to select feasible and desirable ideas at the cost of originality [19,39]. In order to address this research question, the quality and novelty of each student team's ideas were compared at three design stages: team informal screening, team informal discussion of idea selection, and final conceptual design, see Fig. 4. Thus, the dependent variables in this research question were the ANV and quality (AQL) of a team's idea set, while the independent variable was the stage of the design process. Prior to conducting our analysis on the data, assumptions for the repeated measure ANOVA were tested. Specifically, Mauchly's test of sphericity indicated that the assumption of sphericity was violated for AQL (χ²(2) = 6.46, p = 0.04) and ANV (χ²(2) = 6.61, p = 0.04); therefore, a Greenhouse–Geisser correction was used in this research question.

The corrected, repeated measures ANOVA revealed that there was no significant difference between the ANV of a team's ideas throughout the design process (F(1.43, 20.02) = 3.16, p = 0.08, η_p² = 0.18), see Fig. 5. However, there was a significant difference between the AQL of the team's ideas during the three stages of the design process (F(1.437, 20.120) = 30.68, p < 0.01, η_p² = 0.69). Specifically, post-hoc tests using the Bonferroni correction revealed that the AQL of a team's final conceptual design (mean = 0.33, SD = 0.25) was significantly higher than both the AQL of their ideas during team informal screening (mean = −0.12, SD = 0.15, p < 0.01, η_p² = 0.76) and their ideas during team concept selection (mean = −0.07, SD = 0.20, p < 0.01, η_p² = 0.76), see Fig. 6. These results indicate that the AQL significantly increased during the detailed design stage. There were no other significant differences.

These results met part of our hypothesis that the AQL would increase over the design process, but contradicted part of our hypothesis that the ANV would decrease over the design process. The increase of the AQL may be a result of students' concentration on the technical feasibility of the system [18,38] and the enriched details in the final conceptual design compared to the early stage concepts. The lack of increase in the average novelty of design ideas throughout the process may be due to the fact that novelty is rarely stressed in the later phases of design, especially during the concept selection process.

The first research question showed that the AQL of a student team's ideas changed significantly over the course of their design project, but there was no significant change in the ANV of their ideas. This question, though, did not compare the impact of different concept selection tools on these changes. Therefore, our second research question was developed in order to identify if, or to what effect, the concept selection tool used during the design process impacted the AQL and ANV of a student team's ideas and final conceptual design. Specifically, we hypothesized that student teams who used the CSM method would be more likely to see more increase in the AQL of their ideas, while student teams who used the TASC method would be more likely to see more increase in the ANV of their ideas throughout the conceptual design process.

In order to test this question, a repeated measures ANOVA was computed with the dependent variables being the ANV and quality (AQL) of a team's idea set and the independent variables being the stage of the design process and the concept selection tool used. The between-groups effect of the concept selection tools from the repeated measures ANOVA revealed no significant difference in the ANV (F(1, 14) = 3.10, p = 0.10, η_p² = 0.18) or AQL (F(1, 14) = 3.28, p = 0.09, η_p² = 0.19) of a team's ideas throughout the design process between the TASC and CSM section, see Figs. 7 and 8. These results reject our hypotheses by finding that there was no difference in the changes in the average novelty and quality of the design ideas developed throughout the design process based on the concept selection tool used. In other words, this finding indicates that the CSM and TASC method did not impact the decisions made by student teams differently, even though there are fundamental differences in the way these decision tools rank ideas.

While the second research question showed no difference in the ANV or AQL of ideas selected between the CSM and TASC section throughout the design process, it was unclear if students actually selected the ideas highly recommended by the tools. Thus, the third research question was developed to identify if student teams' selection of ideas aligned with concept selection tools' recommendations. This is important because if a creative idea is ranked highly by the concept selection tool, but not selected by the team, the utility of this tool is minimal. Based on prior research, we hypothesized that students would be more likely to select ideas ranked highly in CSM, because they trusted the CSM method more than the TASC method. Therefore, in this research question, principles of directed content analysis [81] were used to analyze student teams' responses to postsurvey questions. It is important to note that only seven teams from the CSM section were in the analysis of the first question and seven teams from the TASC section in the analysis of the second question due to illegible responses.

The first question on the postsurvey asked student teams which idea(s) their team thought should be further developed after using the concept selection activity and if these ideas were ranked highly in the concept selection tool. The results revealed that 5 out of 8 (62.5%) teams from the TASC section and 6 out of 7 (85.7%) teams from the CSM section selected the ideas ranked first in the corresponding concept selection tools, respectively, see Fig. 9 for a full exploration of the ideas selected by each team. Importantly, all of the student teams in the CSM section selected ideas ranked in the top 3. On the contrary, student teams in the TASC section selected ideas ranked anywhere from first to fifth. This suggested that student teams were less likely to use the recommendations of TASC than those of CSM. However, student teams' likelihood of selecting ideas in both sections decreased as the ranking decreased, indicating that students were more likely to select ideas ranked highly in these tools.

When asked if the ideas their team chose to further develop were ranked highly in the formal concept selection tool and if not, why their team selected them, student teams from the CSM section mentioned they would choose the ideas ranked first by the CSM method, because the ideas met their self-created criteria. For instance, one of the teams from the CSM section wrote, “The ideas we chose further to develop did rank highly in the formal concept selection method. This is important as it agrees with the factors we decided” [emphasis added]. Similarly, student teams who chose the ideas ranked first in the TASC method wrote that “…we chose them because they met our ideal designs and needs” and “The formal concept selection agreed with our reasoning as to which concepts were the most effective when implemented in our situation.” While, student teams who did not select ideas ranked first by the TASC method stated they did not agree with some; “… the ideas that were highly ranked according to the TASC website were not as viable as the idea we ended up choosing.” In addition, 6 out of 8 teams (75%) from the CSM section and 6 out of 7 (85.7%) teams from the TASC section emphasized different aspects of customer needs, such as reducing smog, EPA requirements, and cost. Additionally, 3 out of 8 (37.5%) teams from the CSM section and 4 out of 7 (57.1%) teams from the TASC section mentioned technical feasibility. Interestingly, none of the teams from the CSM section mentioned “creativity,” while 3 out of 7 (42.9%) teams from the TASC section emphasized that they specifically considered creativity during their concept selection.

The results of this research question confirmed our hypothesis that students would be more likely to select ideas ranked highly in CSM than those in TASC. One potential reason for this difference may be that the transparency of the ratings in the CSM method allowed student teams to manipulate the relative priorities of the criteria and subjectively rate the ideas so that the evaluation results were more consistent with their expectations [25]. On the contrary, the weights of the adjective words in the TASC method were blind to students [32].

These results lead us back to the purpose of this study, which was to understand the impact of traditional, and newly developed concept selection tools on the evolution of creative ideas and the decision-making processes of engineering students. Specifically, the main findings were

• The AQL of the ideas was significantly improved from the team informal screening stage to the final conceptual design stage. However, there was no change in the ANV of the ideas.

• There were no significant differences between the impact of CSM and TASC methods on the ANV or quality (AQL) of student team ideas throughout the design process.

• Student teams were more likely to select ideas ranked highly in the CSM method over the newly developed TASC method.

• Student teams were more likely to select ideas after using concept selection tools if these ideas matched their preconceived expectations.

Specifically, our results showed that the AQL of student teams' ideas increased from team informal screening to the final conceptual design, but there was no change in ANV. This reflects the fact that students often emphasize customer needs and technical feasibility throughout the design process [18,78], which is a construct of the quality measurement. It shows an overall improvement in the quality of ideas as a student team tackles a design challenge, which is preferred due to the significant influence of idea quality on later design stages [5]. However, there was no significant change in the ANV of the student teams' ideas. This indicates student teams did not give as much value later on in the design process to attaining new ways of solving the problem. Based on previous analysis on individual survey, it was concluded that this failure of improving the novelty of ideas was not due to the likelihood to get a good grade, ownership of the ideas, easiness to prototype, or instructor's preferences [78]. Instead, this might be due to students' preference of customer needs, design criteria, and technical feasibility over the novelty of ideas and the recommendations of concept selection tools [78]. This also aligns with previous research that found students' subjective ratings of novelty and feasibility were inversely related [49], which means that the move toward more feasible design alternatives in the later stages of the design process may come at the cost of design originality. The current study examined the impact of formal concept selection tools in the conceptual design process as a complement to a previous study that tracked the changes of best novelty and quality of ideas generated and selected by students during a class project without the aid of formal concept selection tools [19]. It was found that even though concept selection tools were not a reason of students' abandoning creativity, but the benefits of concept selection tools were not fully taken by the students, either. This finding contributes in identifying factors that lead to decrease of idea novelty throughout the conceptual design process and would require a shift in the way how the later stages of the design process are taught, which is often geared toward evaluating, selecting, and synthesizing the original ideas into a final solution for further development [3,4]. It also urges design educators to integrate training that not only focuses on developing feasible design alternatives, but also focuses on continuing to develop novel alternatives throughout the design process [13].

The second finding showed that there were no significant differences between the impact of CSM and TASC methods on the ANV or quality (AQL) of student team ideas throughout the design process, despite the fundamental differences between these tools. This is surprising since the CSM method was designed to considering customer needs [25] and not necessarily to evaluate the novelty of a solution [9,29,30,82]. Differently, the TASC method utilizes semantic creativity ratings that lead to evaluations of idea creativity on a global scale [32]. While these tools are fundamentally different, the lack of differences in the impact of these tools on the average novelty and quality of ideas throughout the design process may be due in part to the fact that student teams were less likely to use the recommendations from the TASC method compared to the CSM method, as reported in the third research finding. In other words, even if the TASC method was recommending ideas that were more novel and more feasible than the CSM method, students would not necessarily take these recommendations, which would minimize the impact of these kinds of decision tools. A potential reason that student teams in the CSM section were more likely to select ideas ranked highly by CSM method may be attributed to the fact that they had used this approach in a previous project. On the contrary, students in the TASC section had never utilized this approach before and the method was neither familiar nor transparent to them, which may have impacted their likelihood of selecting ideas ranking highly by this approach. Future work is needed to explore exactly why students are biased for or against these approaches.

In addition, the results showed that student teams were more likely to select ideas ranked highly in the concept selection tools when these ideas matched their expectations of which ideas would meet their predefined criteria the best. In other words, regardless of the recommendations the decision tool provided, students in the current study were likely to exhibit confirmation bias during the concept selection process where they only looked for evidence (i.e., ratings) that supported their beliefs. This type of bias can cause a fixation on initial ideas and block students from identifying other, perhaps better, design alternatives [72]. This confirmation bias may also have been a reason that students were more likely to select alternatives ranked highly in the CSM method as the CSM method has a highly transparent rating process [55], which allows students to easily manipulate the criteria, weights, and the evaluation results to get the answer they want. The issue that student teams would only select ideas ranked highly in concept selection tools when these ideas met their expectations indicated that students might not fully understand the benefits of using formal concept selection tools. This means that the judgments students make on which ideas to consider and thus put into the concept selection tool can bias the decision making very early on in the design process. While this finding points educators to be more thoughtful in the teaching and introduction of concept selection tools and informal decision making processes into the engineering design classroom, more work is needed to both identify the impact of decision biases in this process as well as to develop tools and methods to enhance the flow of creative ideas in this process.

The current study was developed to understand how the utilization of concept selection tools impacted the selection of creative ideas during the conceptual design process in engineering education through an experimental evaluation of 60 first-year students. The results of this study indicated that the AQL of student teams' design ideas increased significantly over the course of an 8-week design process, while there was no change in the ANV of their ideas, and that the evolution of the AQL and ANV of ideas in the conceptual design process was not significantly impacted by the concept selection tool used. In addition, students showed a confirmation bias that they would select ideas recommended by concept selection tools only when highly ranked ideas met their expectations. These findings indicate that there are other factors impacting student decision making and the development of creative ideas during this process. These results call for the need for engineering design educators to emphasize on the importance of developing not only feasible design alternatives in the later stages of design, but also stressing the importance of original solutions in an effort to drive creative idea development. In addition, the phenomenon that student teams would only select ideas ranked highly in concept selection tools when these ideas met their expectations indicated that students indicates that they may not fully understand the benefits of using formal concept selection tools. This calls for changes in engineering design courses in order to emphasize on not only how to use the design tools, but also why to use these tools. It also requires more research into the cognitive biases associated with student team decision-making during the concept selection process, and the modification or development of design tools that mitigate these biases in an effort to promote the flow of creative ideas.

While this study provides insights into the use of concept selection tools in engineering design education, some limitations still exist. First, the design problem used in the current study was a transportation or systems design problem which may have influenced the creativity of ideas developed [19]. Since the study was embedded in an engineering design course, the design of the study was limited by the curriculum design, course requirements, and the time frame of the design process (8 weeks). Because the informal screening of concepts happens early in the design process, it is possible that student teams may have abandoned their most novel ideas immediately in the design process, as has been found in previous studies [19]. In this way, the current study, along with this prior work, suggests a need to explore ways of teaching for supporting novel idea development in engineering education in order to better maintain creative potential during the design process.

Furthermore, student teams' selections might be skewed due to their bias toward their previous experience with the CSM or TASC methods or the transparency of the ranking systems. In addition, only two fundamentally different concept selection tools were used in the current study. Future work should expand this work to include explorations of a wider variety of decision aids in different design scenarios including a control group in which students do not use any formal concept selection tools. That being said, the current work strongly points to the need for further investigations into the use of formal and informal decision processes in engineering design education and their ultimate impact on idea development. On top of that, the current study does not provide exact answers to why students do not use the recommendations of the concept selection tools or how should concept selection process be taught in engineering design classrooms. Further exploration is needed to reveal the answers to these questions through both behavioral and attitudinal studies. Importantly, the current study conducted in engineering design classrooms also serves as the first step of investigating the impact of concept selection tools. Future studies are needed to identify if these same challenges exist in engineering design industry. These studies are of particular interest because they would allow for the identification of the long-term impact of concept selection tools on engineered solutions where the time frame of the design process does not have to be limited to an educational semester.

We would like to thank our undergraduate research assistant Lisa Miele, and our participants for their help in this project.

• National Science Foundation, Division of Civil, Mechanical and Manufacturing Innovation (Grant No. 1351493).

Pittsadelphia is looking for the design of a cost-effective freight shipping system that reduces smog and meets EPA requirements, while maintaining or increasing freight capacity into and out of this important port city.

Every day into and out of the port city of Pittsadelphia, approximately 165,000 tons of freight or minerals (coal, etc.) per day travel via rail. Smog from locomotive emissions is a key complaint of city residents. Smog is generated from engine-emitted NO_x. Tier 2 locomotives used to haul freight are approaching age for overhaul, at which time investments will be required to meet EPA tier 3 (or higher) requirements.

Suggestions have been made to address locomotive emissions (i.e., smog) by

1. (1)

   Upgrade the locomotive fleet to meet more recent emissions guidelines set by the EPA. A few options may exist to meet the new guidelines:

   • Sell existing fleet and purchase new locomotives

   • Upgrade fleet with exhaust after-treatment hardware

   • Utilize alternate fuels (Biodiesel, CNG, LNG, etc.), which may produce less NO_x

2. (2)

   Alternate freight shipping methods:

   • By water

   • By air

   • By ground, i.e., trucking

GE Transportation, a unit of GE (NYSE: GE), solves the world's toughest transportation challenges. GE Transportation builds equipment that moves the rail, mining, and marine industries. GE's fuel-efficient and lower-emissions freight and passenger locomotives; diesel engines for rail; marine and stationary power applications; signaling and software solutions; drive systems for mining trucks; and value-added services help customers grow. GE Transportation is headquartered in Chicago, IL, and employs approximately 13,000 employees worldwide.

Each design team should research and evaluate the suggestions made for fleet upgrade or alternate shipping methods. For upgrades, consider physical constraints of new hardware, as well as fuel storage requirements. Provide your recommendations, commenting on impact to:

1. (1)

   Emissions/regulatory requirements

2. (2)

   Costs: fuel, infrastructure, etc.

3. (3)

   Freight throughput/capacity

4. (4)

   Public opinion

5. (5)

   On-time delivery

Note: Your instructor will clarify her or his expectations for these deliverables and respective due dates.

• Technical report containing the following elements

• Rationale for the recommendation

• Description of alternative concepts and their evaluation

• Systems diagram

• Concept of operations

• Environmental analysis

• Assessment of important aspects of your system for feasibility and adoption, including public opinion

• Economic viability of the system o CAD drawings

• Model or prototype of a component of the overall system