Abstract

Computationally studying team discourse can provide valuable, real-time insights into the state of design teams and design cognition during problem-solving. The particular experimental design, adopted from previous work by the authors, places one of the design team conditions under the guidance of a human process manager. In that work, teams under this process management outperformed the unmanaged teams in terms of their design performance. This opens the opportunity to not only model design discourse during problem-solving, but more critically, to explore process manager interventions and their impact on design cognition. Utilizing this experimental framework, a topic model is trained on the discourse of human designers of both managed and unmanaged teams collaboratively solving a conceptual engineering design task. Results show that the two team conditions significantly differ in a number of the extracted topics and, in particular, those topics that most pertain to the manager interventions. A dynamic look during the design process reveals that the largest differences between the managed and unmanaged teams occur during the latter half of problem-solving. Furthermore, a before and after analysis of the topic-motivated interventions reveals that the process manager interventions significantly shift the topic mixture of the team members’ discourse immediately after intervening. Taken together, these results from this work not only corroborate the effect of the process manager interventions on design team discourse and cognition but provide promise for the computational detection and facilitation of design interventions based on real-time, discourse data.

References

1.
Ball
,
L. J.
, and
Christensen
,
B. T.
,
2019
, “
Advancing an Understanding of Design Cognition and Design Metacognition: Progress and Prospects
,”
Des. Stud.
,
65
(
1
), pp.
35
59
.
2.
Cross
,
N.
,
2001
, “Design Cognition: Results From Protocol and Other Empirical Studies of Design Activity,”
Design Knowing and Learning: Cognition in Design Education
,
C
.
Eastman
,
W
.
Newstatter
, and
M
.
McCracken
, ed.,
Elsevier
,
Oxford, UK
, pp.
79
103
.
3.
Den Otter
,
A.
, and
Emmitt
,
S.
,
2007
, “
Exploring Effectiveness of Team Communication: Balancing Synchronous and Asynchronous Communication in Design Teams
,”
Eng. Constr. Archit. Manage.
,
14
(
5
), pp.
408
419
.
4.
Lloyd
,
P.
,
Lawson
,
B.
, and
Scott
,
P.
,
1995
, “
Can Concurrent Verbalization Reveal Design Cognition?
,”
Des. Stud.
,
16
(
2
), pp.
237
259
.
5.
Stempfle
,
J.
, and
Badke-Schaub
,
P.
,
2002
, “
Thinking in Design Teams—An Analysis of Team Communication
,”
Des. Stud.
,
23
(
5
), pp.
473
496
.
6.
Martin
,
M. J.
, and
Foltz
,
P. W.
,
2004
, “
Automated Team Discourse Annotation and Performance Prediction Using LSA
,”
Proceedings of Human Language Technology and North American Association for Computational Linguistics Conference 2004: Short Papers. Association for Computational Linguistics
,
Boston, MA
,
May 2–7
,
Association for Computational Linguistics
, pp.
97
100
.
7.
Dong
,
A.
,
Hill
,
A. W.
, and
Agogino
,
A. M.
,
2004
, “
A Document Analysis Method for Characterizing Design Team Performance
,”
ASME J. Mech. Des.
,
126
(
3
), pp.
378
385
.
8.
Dong
,
A.
,
2005
, “
The Latent Semantic Approach to Studying Design Team Communication
,”
Des. Stud.
,
26
(
5
), pp.
445
461
.
9.
Fu
,
K.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2010
, “
Design Team Convergence: The Influence of Example Solution Quality
,”
ASME J. Mech. Des.
,
132
(
11
), p.
111005
.
10.
Dong
,
A.
,
Kleinsmann
,
M. S.
, and
Deken
,
F.
,
2013
, “
Investigating Design Cognition in the Construction and Enactment of Team Mental Models
,”
Des. Stud.
,
34
(
1
), pp.
1
33
.
11.
Linsey
,
J. S.
,
Markman
,
A. B.
, and
Wood
,
K. L.
,
2012
, “
Design by Analogy: A Study of the WordTree Method for Problem Re-representation
,”
ASME J. Mech. Des.
,
134
(
4
), p.
041009
.
12.
Agogué
,
M.
,
Kazakçi
,
A.
,
Hatchuel
,
A.
,
Le Masson
,
P.
,
Weil
,
B.
,
Poirel
,
N.
, and
Cassotti
,
M.
,
2014
, “
The Impact of Type of Examples on Originality: Explaining Fixation and Stimulation Effects
,”
J. Creat. Behav.
,
48
(
1
), pp.
1
12
.
13.
Song
,
B.
,
Srinivasan
,
V.
, and
Luo
,
J.
,
2017
, “
Patent Stimuli Search and Its Influence on Ideation Outcomes
,”
Des. Sci.
,
3
(
1
), p.
e25
.
14.
Fu
,
K.
,
Chan
,
J.
,
Cagan
,
J.
,
Kotovsky
,
K.
,
Schunn
,
C.
, and
Wood
,
K.
,
2013
, “
The Meaning of ‘Near’ and ‘Far’: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output
,”
ASME J. Mech. Des.
,
135
(
2
), p.
021007
.
15.
Fu
,
K.
,
Cagan
,
J.
,
Kotovsky
,
K.
, and
Wood
,
K.
,
2013
, “
Discovering Structure in Design Databases Through Functional and Surface Based Mapping
,”
ASME J. Mech. Des.
,
135
(
3
), p.
031006
.
16.
Goucher-Lambert
,
K.
,
Gyory
,
J. T.
,
Kotovsky
,
K.
, and
Cagan
,
J.
,
2020
, “
Adaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation
,”
ASME J. Mech. Des.
,
142
(
9
), p.
091401
.
17.
Gyory
,
J. T.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2019
, “
Are You Better off Alone? Mitigating the Underperformance of Engineering Teams During Conceptual Design Through Adaptive Process Management
,”
Res. Eng. Des.
,
30
(
1
), pp.
85
102
.
18.
Gyory
,
J. T.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2018
, “
Should Teams Collaborate During Conceptual Engineering Design? An Experimental Study
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Quebec City, Canada
,
Aug. 26–29
,
American Society of Mechanical Engineers
, Vol.
51845
.
19.
Mohr
,
J. W.
, and
Bogdanov
,
P.
,
2013
, “
Introduction-Topic Models: What They Are and Why They Matter
,”
Poetics
,
41
(
6
), pp.
545
569
.
20.
Blei
,
D. M.
,
2012
, “
Probabilistic Topic Models
,”
Commun. ACM
,
55
(
4
), pp.
77
84
.
21.
Rosen-Zvi
,
M.
,
Griffiths
,
T.
,
Steyvers
,
M.
, and
Smyth
,
P.
,
2004
, “
The Author-Topic Model for Authors and Documents
,”
Conference on Uncertainty in Artificial Intelligence
,
Banff, Canada
,
July 7–11
, pp.
487
494
.
22.
Blei
,
D. M.
, and
Lafferty
,
J. D.
,
2007
, “
A Correlated Topic Model of Science
,”
Ann. Appl. Stat.
,
1
(
1
), pp.
17
35
.
23.
Ball
,
Z.
, and
Lewis
,
K.
,
2020
, “
Predicting Design Performance Utilizing Automated Topic Discovery
,”
ASME. J. Mech. Des.
,
142
(
12
), p.
121703
.
24.
Ahmed
,
F.
,
Fuge
,
M.
, and
Gorbunov
,
L. D.
,
2016
, “
Discovering Diverse, High Quality Design Ideas From a Large Corpus
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
,
Aug. 21–24
,
American Society of Mechanical Engineers
, Vol.
50190
.
25.
Ahmed
,
F.
, and
Fuge
,
M.
,
2018
, “
Creative Exploration Using Topic-Based Bisociative Networks
,”
Des. Sci.
,
4
(
12
), pp.
1
30
.
26.
Bhowmik
,
T.
,
Niu
,
N.
,
Savolainen
,
J.
, and
Mahmoud
,
A.
,
2015
, “
Leveraging Topic Modeling and Part-of-Speech Tagging to Support Combinational Creativity in Requirements Engineering
,”
Requir. Eng.
,
20
(
3
), pp.
253
280
.
27.
Joung
,
J.
, and
Kim
,
H. M.
,
2021
, “
Automated Keyword Filtering in Latent Dirichlet Allocation for Identifying Product Attributes From Online Reviews
,”
ASME J. Mech. Des.
,
143
(
8
), p.
084501
.
28.
Suryadi
,
D.
, and
Kim
,
H. M.
,
2019
, “
A Data-Driven Approach to Product Usage Context Identification From Online Customer Reviews
,”
ASME J. Mech. Des.
,
141
(
12
), p.
121004
.
29.
Nijstad
,
B. A.
, and
Stroebe
,
W.
,
2006
, “
How the Group Affects the Mind: A Cognitive Model of Idea Generation in Groups
,”
Personal. Soc. Psychol. Rev.
,
10
(
3
), pp.
186
213
.
30.
Hey
,
J.
,
Linsey
,
J.
,
Agogino
,
A. M.
, and
Wood
,
K. L.
,
2008
, “
Analogies and Metaphors in Creative Design
,”
Int. J. Eng. Educ.
,
24
(
2
), pp.
283
294
.
31.
Yilmaz
,
S.
, and
Seifert
,
C.
,
2010
, “
Cognitive Heuristics in Design Ideation
,”
Proceedings of 11th International Design Conference
,
Dubrovnik, Croatia
,
May 17–20
, pp.
1
11
.
32.
Isaksen
,
S. G.
,
2013
, “
Facilitating Creative Problem-Solving Groups
,” http://www.cpsb.com/research/articles/creative-problem-solving/Facilitating-CPS-Groups.html.
33.
Gyory
,
J. T.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2018
, “
An Exploration of the Effects of Managerial Intervention on Engineering Design Team Performance
,”
International Conference on-Design Computing and Cognition
,
Como, Italy
,
July 2–4
.
34.
Blei
,
D. M.
,
Ng
,
A. Y.
, and
Jordan
,
M. I.
,
2003
, “
Latent Dirichlet Allocation
,”
J. Mach. Learn. Res.
,
3
(
1
), pp.
993
1022
.
35.
Deerwester
,
S.
,
Dumais
,
S. T.
,
Furnas
,
G. W.
,
Landauer
,
T. K.
, and
Harshman
,
R.
,
1990
, “
Indexing by Latent Semantic Analysis
,”
J. Am. Soc. Inf. Sci.
,
41
(
6
), pp.
391
407
.
36.
Hofmann
,
T.
,
2017
, “
Probabilistic Latent Semantic Indexing
,”
ACM SIGIR Forum
,
51
(
2
), pp.
211
218
.
37.
Ramos
,
J.
,
2003
, “
Using TF-IDF to Determine Word Relevance in Document Queries
,”
Proceedings of the First Instructional Conference on Machine Learning
,
242
(
1
), pp.
29
48
.
38.
Kim
,
T. Y.
,
Min
,
M.
,
Yoon
,
T.
, and
Lee
,
J. H.
,
2010
, “
Semantic Analysis of Twitter contents using PLSA, and LDA
,”
SCIS & ISIS
,
Okayama, Japan
,
Dec. 8–12
.
39.
Rosa
,
K. D.
,
Shah
,
R.
,
Lin
,
B.
,
Gershman
,
A.
, and
Frederking
,
R.
,
2011
, “
Topical Clustering of Tweets
,”
Proceedings of the ACM SIGIR: SWSM
,
Beijing, China
,
July 28
.
40.
Mehrotra
,
R.
,
Sanner
,
S.
,
Buntine
,
W.
, and
Xie
,
L.
,
2013
, “
Improving lda Topic Models for Microblogs via Tweet Pooling and Automatic Labeling
,”
Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
,
Dublin, Ireland
,
July 28–Aug. 1
.
41.
Nguyen
,
V.-A.
,
Boyd-Graber
,
J.
, and
Resnik
,
P.
,
2012
, “
SITS: A Hierarchical Nonparametric Model Using Speaker Identity for Topic Segmentation in Multiparty Conversations
,”
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers – Vol. 1
,
Jeju Island, South Korea
,
July 8–14
, pp.
78
87
.
42.
Purver
,
M.
,
Kording
,
K.
,
Griffiths
,
T. L.
, and
Tenenbaum
,
J. B.
,
2006
, “
Unsupervised topic modelling for multi-party spoken discourse
,”
Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics
,
Sydney, Australia
,
July 17–20
, pp.
17
24
.
43.
Crain
,
S. P.
,
Zhou
,
K.
,
Yang
,
S. H.
, and
Zha
,
H.
,
2012
, “Dimensionality Reduction and Topic Modeling: From Latent Semantic Indexing to Latent Dirichlet Allocation and Beyond,”
Mining Text Data
,
Springer
,
Boston, MA
.
44.
Handler
,
A.
,
Denny
,
M.
,
Wallach
,
H.
, and
O’Connor
,
B.
,
2016
, “
Bag of What? Simple Noun Phrase Extraction for Text Analysis
,”
Proceedings of 2016 EMNLP Workshop on Natural Language Processing and Computational Social Science
,
Austin, TX
,
Nov. 5
, pp.
114
124
.
45.
Loper
,
E.
, and
Bird
,
S.
,
2004
, “
NLTK: The Natural Language Toolkit
,”
Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, Association for Computational Linguistics
,
Barcelona, Spain
,
July 21–26
.
46.
Huang
,
L.
,
Ma
,
J.
, and
Chen
,
C.
,
2017
, “
Topic Detection From Microblogs Using T-LDA and Perplexity
,”
Proceedings—2017 24th Asia-Pacific Software Engineering Conference Workshops (APSECW)
,
Nanjing, China
,
Dec. 4–8
, pp.
71
77
.
47.
AlSumait
,
L.
,
Barbará
,
D.
, and
Domeniconi
,
C.
,
2008
, “
On-Line LDA: Adaptive Topic Models for Mining Text Streams with Applications to Topic Detection and Tracking
,”
Proceedings—IEEE International Conference on Data Mining, ICDM
,
Pisa, Italy
,
Dec. 15–19
, IEEE, pp.
3
12
.
48.
Heinrich
,
G.
,
2005
, “
Parameter Estimation for Text Analysis
,”
Technical Report
,
1
(
1
), pp.
1
36
.
49.
Merriam-Webster
,
M. W. S. C.
,
1977
,
Dictionary
,
G & C Merriam Company, 830., Springfield
,
MA
.
50.
Kullback
,
S.
, and
Leibler
,
R. A.
,
2010
, “
On Information and Sufficiency
,”
Ann. Math. Stat.
,
22
(
1
), pp.
79
86
.
51.
Pérez-Cruz
,
F.
,
2008
, “
Kullback-Leibler Divergence Estimation of Continuous Distributions
,”
IEEE International Symposium on Information Theory—Proceedings
,
Toronto, ON, Canada
,
July 6–11
, pp.
1666
1670
.
52.
Shlens
,
J.
,
2014
, “
Notes on Kullback-Leibler Divergence and Likelihood
.”
arXiv preprint arXiv:1404.2000
.
53.
Kapoor
,
R.
,
Gupta
,
R.
,
Son
,
L. H.
,
Jha
,
S.
, and
Kumar
,
R.
,
2018
, “
Boosting Performance of Power Quality Event Identification With KL Divergence Measure and Standard Deviation
,”
Measurement
,
126
(
1
), pp.
134
142
.
54.
Foulds
,
J.
,
Boyles
,
L.
,
DuBois
,
C.
,
Smyth
,
P.
, and
Welling
,
M.
,
2013
, “
Stochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation
,”
Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
,
Chicago, IL
,
Aug. 11–14
, ACM, pp.
446
454
.
55.
Hoffman
,
M. D.
,
Blei
,
D. M.
,
Wang
,
C.
, and
Paisley
,
J.
,
2013
, “
Stochastic Variational Inference
,”
J. Mach. Learn. Res.
,
14
(
1
), pp.
1303
1347
.
56.
Griffiths
,
T. L.
, and
Steyvers
,
M.
,
2004
, “
Finding Scientific Topics
,”
Proc. Natl. Acad. Sci. USA
,
101
(
1
), pp.
5228
5235
.
57.
Asuncion
,
A.
,
Welling
,
M.
,
Smyth
,
P.
, and
Teh
,
Y. W.
,
2009
, “
On Smoothing and Inference for Topic Models
,”
Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence
,
Montreal, Quebec, Canada
,
June 18–21
,
AUAI Press
, pp.
27
34
.
58.
Teh
,
Y. W.
,
Newman
,
D.
, and
Welling
,
M.
,
2007
, “
A collapsed variational bayesian inference algorithm for latent dirichlet allocation
,”
Adv. Neur. Infor. Proc. Sys.
,
19
(
1
), pp.
1353
1360
.
59.
Airoldi
,
E. M.
,
Blei
,
D.
,
Erosheva
,
E. A.
, and
Fienberg
,
S. E.
,
2014
,
Handbook of Mixed Membership Models and Their Applications
,
CRC Press
,
Boca Raton, FL
.
60.
Newman
,
D.
,
Baldwin
,
T.
,
Cavedon
,
L.
,
Huang
,
E.
,
Karimi
,
S.
,
Martinez
,
D.
,
Scholer
,
F.
, and
Zobel
,
J.
,
2010a
, “
Visualizing Search Results and Document Collections Using Topic Maps
,”
Web Semantics
,
8
(
2–3
), pp.
169
175
.
61.
Newman
,
D.
,
Lau
,
J. H.
,
Grieser
,
K.
, and
Baldwin
,
T.
,
2010b
, “
Automatic Evaluation of Topic Coherence
,”
Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
,
Los Angeles, CA
,
June 2–4
, pp.
100
108
.
62.
Blei
,
D. M.
, and
Lafferty
,
J. D.
,
2006
, “
Dynamic Topic Models
,”
Proceedings of the 23rd International Conference on Machine Learning
,
Pittsburgh, PA
,
June 25–29
.
63.
Pujara
,
J.
, and
Skomoroch
,
P.
,
2012
, “
Large-Scale Hierarchical Topic Models
.”
NIPS Workshop on Big Learning
. Vol.
128
.
64.
Liu
,
Y.
,
Liu
,
Z.
,
Chua
,
T. S.
, and
Sun
,
M.
,
2015
, “
Topical Word Embeddings
,”
Twenty-Ninth NAAI Conference on Artificial Intelligence
,
Austin, TX
,
Jan. 25–30
.
You do not currently have access to this content.