Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields
<h3>Background</h3><p dir="ltr">Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , , , |
| منشور في: |
2022
|
| الموضوعات: | |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1864513519695691776 |
|---|---|
| author | Joni Salminen (7434770) |
| author2 | Soon-gyo Jung (7434773) Ahmed Kamel (18279025) Willemien Froneman (4448308) Bernard J. Jansen (7434779) |
| author2_role | author author author author |
| author_facet | Joni Salminen (7434770) Soon-gyo Jung (7434773) Ahmed Kamel (18279025) Willemien Froneman (4448308) Bernard J. Jansen (7434779) |
| author_role | author |
| dc.creator.none.fl_str_mv | Joni Salminen (7434770) Soon-gyo Jung (7434773) Ahmed Kamel (18279025) Willemien Froneman (4448308) Bernard J. Jansen (7434779) |
| dc.date.none.fl_str_mv | 2022-10-21T03:00:00Z |
| dc.identifier.none.fl_str_mv | 10.7717/peerj-cs.1136 |
| dc.relation.none.fl_str_mv | https://figshare.com/articles/journal_contribution/Who_is_in_the_sample_An_analysis_of_real_and_surrogate_users_as_participants_in_user_study_research_in_the_information_technology_fields/25514062 |
| dc.rights.none.fl_str_mv | CC BY 4.0 info:eu-repo/semantics/openAccess |
| dc.subject.none.fl_str_mv | Information and computing sciences Computer vision and multimedia computation Human-computer interaction Software engineering User studies Study participants Effect of user studies Value of user types Sampling External validity |
| dc.title.none.fl_str_mv | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| dc.type.none.fl_str_mv | Text Journal contribution info:eu-repo/semantics/publishedVersion text contribution to journal |
| description | <h3>Background</h3><p dir="ltr">Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice.</p><h3>Objective</h3><p dir="ltr">To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants.</p><h3>Method</h3><p dir="ltr">We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users).</p><h3>Results</h3><p dir="ltr">Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were.</p><h3>Conclusions</h3><p dir="ltr">Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers.</p><h3>Implications</h3><p dir="ltr">The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.</p><h2>Other Information</h2><p dir="ltr">Published in: PeerJ Computer Science<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.7717/peerj-cs.1136" target="_blank">https://dx.doi.org/10.7717/peerj-cs.1136</a></p> |
| eu_rights_str_mv | openAccess |
| id | Manara2_5fe5645f4745e347ab1fdf950b5c7ccf |
| identifier_str_mv | 10.7717/peerj-cs.1136 |
| network_acronym_str | Manara2 |
| network_name_str | Manara2 |
| oai_identifier_str | oai:figshare.com:article/25514062 |
| publishDate | 2022 |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | CC BY 4.0 |
| spelling | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fieldsJoni Salminen (7434770)Soon-gyo Jung (7434773)Ahmed Kamel (18279025)Willemien Froneman (4448308)Bernard J. Jansen (7434779)Information and computing sciencesComputer vision and multimedia computationHuman-computer interactionSoftware engineeringUser studiesStudy participantsEffect of user studiesValue of user typesSamplingExternal validity<h3>Background</h3><p dir="ltr">Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice.</p><h3>Objective</h3><p dir="ltr">To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants.</p><h3>Method</h3><p dir="ltr">We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users).</p><h3>Results</h3><p dir="ltr">Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were.</p><h3>Conclusions</h3><p dir="ltr">Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers.</p><h3>Implications</h3><p dir="ltr">The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.</p><h2>Other Information</h2><p dir="ltr">Published in: PeerJ Computer Science<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.7717/peerj-cs.1136" target="_blank">https://dx.doi.org/10.7717/peerj-cs.1136</a></p>2022-10-21T03:00:00ZTextJournal contributioninfo:eu-repo/semantics/publishedVersiontextcontribution to journal10.7717/peerj-cs.1136https://figshare.com/articles/journal_contribution/Who_is_in_the_sample_An_analysis_of_real_and_surrogate_users_as_participants_in_user_study_research_in_the_information_technology_fields/25514062CC BY 4.0info:eu-repo/semantics/openAccessoai:figshare.com:article/255140622022-10-21T03:00:00Z |
| spellingShingle | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields Joni Salminen (7434770) Information and computing sciences Computer vision and multimedia computation Human-computer interaction Software engineering User studies Study participants Effect of user studies Value of user types Sampling External validity |
| status_str | publishedVersion |
| title | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| title_full | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| title_fullStr | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| title_full_unstemmed | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| title_short | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| title_sort | Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields |
| topic | Information and computing sciences Computer vision and multimedia computation Human-computer interaction Software engineering User studies Study participants Effect of user studies Value of user types Sampling External validity |