Contents
METRICS UTILIZED TO MEASURE CREATIVITY
Shah, Vargas-Hernandez, and Smith (SVS)
The SVS literature defines and justifies four metrics, namely Novelty, Variety, Quality and Quantity, to measure ideation effectiveness.
Novelty
Novelty is a measure of how distinguished or unexpected an idea is compared to the other ideas generated. Once all the ideas have been generated, they are differentiated based on their key characteristics (Shah, Vargas-Hernandez and Smith 2003). Each idea is analyzed based on its functionality, how the function is achieved at the abstract level. Overall novelty is measured by the following equation (1)
(1)
N is the novelty score calculated for the idea with m functions or attributes and n stages. Weights (fj) are assigned based on the importance of each attribute or characteristic in order to compute an overall score. Further, each function may be addressed at the conceptual and/or embodiment stage and weights (pk) assigned according to the importance of each stage.
Variety
Expanding and exploring the hypothetical design space in the early stages of the design process is necessary to develop successful designs (Julie Hirtz, Robert B. Stone, Daniel A. McAdams, Simon Szykman, Kristin L. Wood 2001). Variety is used to assess how dissimilar ideas are to each other within a given sample (Shah, Vargas-Hernandez and Smith 2003). Because of this, engineering researchers have sought to capture the variety of design ideas generated throughout engineering design activities and how the breadth of solution space is explored is during the idea generation process (Shah, Vargas-Hernandez and Smith 2003).
In order to calculate variety, a genealogy tree approach was utilized. The variety metrics commonly adopted in the engineering literature break design variety into four hierarchical branches: the physical principle, followed by the working principle, embodiment and detail. Specifically, variety is calculated using the following equation [2].
(2)
V is the variety score, m is the number of functions solved by the design, fj is a weight assigned to relative importance of function j, Sk is the score for hierarchical level k, bk is the number of branches at hierarchical level k, and N is the total number of ideas in the set.
Quality
Quality is the measure of the feasibility of an idea and how well the design requirements are met by the concept generated (Shah, Vargas-Hernandez and Smith 2003). The quality metric is calculated by the following formula (3).
(3)
Sjk is the score for quality for function j at stage k; m is the total number of functions; fj is the eight for function j; pk is the weight for stage k. The denominator is for normalizing to a scale of 10.
Quantity
Quantity is the volume of ideas generated by a participant or a group within an allocated period of time (Shah, Vargas-Hernandez and Smith 2003).
Consensual Assessment Technique
The Consensual Assessment Technique is a powerful tool used by creativity researchers to rate the creativity of creative products such as stories, collages, poems, and other artefacts. The Consensual Assessment Technique (CAT) is a creativity validation method largely adopted by the fields of social sciences and education. This method considers an idea to be creative if appropriate raters independently agree that it is creative. The raters assigned are usually experts/quasi-experts in the field of interest with domain-specific knowledge and experience. CAT method is independent of the validity of a particular theory of creativity. The raters are required to rate aspects including drawing abilities, overall creativity, usefulness, uniqueness and elegance on a six-point Likert scale.
Nelson Variety Metric
The Nelson variety metric was a refinement to the existing SVS Variety metric to correct its flaws and shortcomings (Nelson, B. A., Wilson, J. O., Rosen, D., and Yen, J., 2009). The Nelson metric aimed to correct two flaws in particular:
- The first flaw was double counting design ideas. The SVS metric counted the number of design ideas at each level rather than the number of differentiations.
- The second flaw was normalizing a group score on measuring variety. Variety, unlike novelty, can be measured only for a set of ideas and not for an idea alone. A non-normalized variety score is a better representation of the actual design space explored.
Correcting these flaws, the Nelson metric refines the SVS variety metrics as follows (4).
(4)
V is the variety score, m is the total number of required functions solved by the design, fj is a weighting factor for the relative importance of function j, Sk is the score for hierarchical level k (scores of 10, 5, 2, and 1 for the four levels, respectively, bk is the number of branches at hierarchical level k, the first term inside the parenthesis is the score for differentiation at the
physical principle level, dl is the number of differentiations at node l.
References
Shah, J. J., Vargas Hernandez, N., & Smith, S. M. (2003). Metrics for measuring ideation effectiveness. Design Studies, 24, 111-134. Citation Details.
Hirtz, J., Stone, R. B., McAdams, D. A., Szykman, S., and Wood, K. L., 2002, “A functional basis for engineering design: reconciling and evolving previous efforts,” Research in Engineering Design, 13(2), pp. 65-82. Citation Details.
Henderson, D., Helm, K., Jablokow, K., McKilligan, S., Daly, S., and Silk, E., “A Comparison of Variety Metrics in Engineering Design” Proc. ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers. Citation Details.
Nelson, B. A., Wilson, J. O., Rosen, D., and Yen, J., 2009, “Refined metrics for measuring ideation effectiveness,” Design Studies, 30(6), pp. 737-743. Citation details.