|Year : 2021 | Volume
| Issue : 4 | Page : 339-344
Assessing information on YouTube™ as a quality source for the treatment of varicoceles
H Stephen Hong, J Jacob Lang, Shivashankar Damodaran, Puneet Sindhwani
Department of Urology, University of Toledo College of Medicine and Life Sciences, Toledo, Ohio, USA
|Date of Submission||19-May-2021|
|Date of Decision||04-Sep-2021|
|Date of Acceptance||23-Sep-2021|
|Date of Web Publication||1-Oct-2021|
H Stephen Hong
Department of Urology, University of Toledo College of Medicine and Life Sciences, Toledo, Ohio
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Introduction: YouTube™ has grown into one of the largest disseminators of health care information. We assessed the quality of information on varicoceles and their treatment, available on YouTube™.
Methods: Using a YouTube™ search query with the keyword “varicocele,” the quality of the first 50, nonrepeat videos in English were assessed as a representative group for the topic. DISCERN and Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV) standardized tools were utilized by three independent reviewers to grade the quality of these videos based on content, understandability, and actionability.
Results: The average and median DISCERN score was 31.34 (±9.37) and 31 (interquartile range 25–35), respectively, indicating poor quality. The interrater reliability (IRR) scores ranged from 0.51 to 0.93, indicating fair to excellent reliability. The average PEMAT-AV understandability and actionability scores were 69.8% ±15.4% and 11.0% ±24.6%, respectively, indicating mostly understandable but poor actionability. The t-test results showed that international videos scored higher without statistical significance in the DISCERN or PEMAT-AV scores (P = 0.18, 0.59, and 0.20).
Conclusions: The current quality of videos on YouTube™ on the topic of varicoceles is of poor quality due to a lack of a holistic approach in explaining the wide range of treatment options available. With the ease of access to produce and disseminate health information, there is a need to create high-quality videos on varicoceles that empower a patient to make an informed decision.
|How to cite this article:|
Hong H S, Lang J J, Damodaran S, Sindhwani P. Assessing information on YouTube™ as a quality source for the treatment of varicoceles. Indian J Urol 2021;37:339-44
|How to cite this URL:|
Hong H S, Lang J J, Damodaran S, Sindhwani P. Assessing information on YouTube™ as a quality source for the treatment of varicoceles. Indian J Urol [serial online] 2021 [cited 2022 Jan 20];37:339-44. Available from: https://www.indianjurol.com/text.asp?2021/37/4/339/327369
| Introduction|| |
With the explosion of online content, it is now common for physicians to encounter patients who have performed their own research on the internet to determine the treatment option for their perceived need.,, This ease of access to information has been shown to be beneficial to the patient-physician relationship; however, this benefit is realized only if the information is shared with the physician and a further discussion is had about how best to proceed. However, this largely relies on the assumption that patients are accessing quality information related to their condition, an area in which YouTubeTM is minimally regulated. Studies analyzing the quality of videos on YouTubeTM of other common health conditions have shown quality of information is lacking.,,,, Urological diseases and procedures addressed on YouTubeTM are not immune to these issues as a study found that there is a negative correlation with scientific quality and viewer engagement. This can be alarming as the percentage of urologist using YouTubeTM and other social media platforms is increasing.
Research has already been conducted on assessing information on urological conditions;,, however, information related to varicoceles has yet to be addressed. Varicocele is prevalent in up to 15% of healthy men and up to 35% of men with primary infertility. Varicocele is widely accepted to be the most common identifiable cause of male infertility through a mechanism that is not completely understood. Not all patients who are diagnosed with varicocele require treatment, placing more onus on the patient to choose whether or not to treat. In particular, patients who are trying to address issues of fertility or with mild scrotal pain will most likely try to supplement their knowledge before approaching more invasive treatments. The aim of this study is to the examine the quality, understandability, and accessibility of varicocele-related videos on YouTubeTM from the patient viewpoint to better understand how this information could affect patient decision-making.
| Methods|| |
Video selection requirements
A YouTube™ video search was conducted on October 1st, 2020, using the keyword “varicocele.” This search was conducted under an incognito webpage to limit YouTube's optimize search that is specific to a reviewer's past video history. Since the IP address used to access the YouTube™ server was in the United States, the search was used to represent a standard United States user. Previous research has shown that 41% of viewers click on entries on the first page, with only 20% searching beyond the third page., However, YouTube™ has updated its search layout by continuously adding new videos as a user reaches the bottom of the page, dubbed the “infinite scroll.” Therefore, the first 50 videos, representing 2.5 pages of videos on the older layout (20 videos per page × 2.5 pages = 50 videos) that fulfilled any of the following requirements had its title and URL recorded:
- General description of varicoceles
- Pathophysiology of varicoceles
- Diagnosis of varicoceles
- Treatment options of varicoceles
- Complications of varicoceles
- Personal experiences with varicoceles or treatment of varicoceles.
Videos were excluded if they were duplicates, not in English, or if the primary goal was unrelated but mentioned varicocele (e.g., videos about vasectomy or female infertility). Other video data including source category (e.g., private practice, vlogs), video category (e.g., Educational, personal experience), upload date, duration, likes, and dislikes were also recorded.
Two medical students and one senior urological resident examined the included videos utilizing two tools: The DISCERN tool to assess quality and the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V) for understandability and accessibility.
DISCERN is a standardized survey usable by both untrained laypersons and health professionals that is typically used for the assessment of written information on treatment choices for a health problem,, which was modified to reflect the media used in this study. The 16-question survey has 8 questions that address the reliability of the source, 7 questions that assess the quality of the treatment options, and 1 question regarding the rater's evaluation of the entire publication. Each question is scored between 1 and 5, with 5 defined as a definite “yes” in accomplishing the goal of the question and 1 as a definite “no.” Partial credit of a 2–4 is given if it meets criterion to some extent. The score from each category, except the rater's evaluation, is summed (total of 15 categories), giving a score between 15 and 75. Scores between 63 and 75 are considered as “excellent,” 51–62 as “good,” 39–50 as “fair,” 28–38 as “poor,” and <27 as “very poor.” Summary statistics for each question were calculated and sorted to assess the weaknesses and strengths of the current information on varicoceles.
PEMAT-A/V is another standardized survey to analyze audiovisual media on the understandability and actionability of the information provided. The PEMAT-AV contains 17 criteria, with the first 13 assessing understandability and the last 4 assessing actionability. One of the PEMAT-A/V criteria was completed excluded due to media being assessed are videos, with others excluded in some videos as they did not apply. These criteria are scored as either agree or disagree using a 1 or 0, respectively, without partial credit. Summary statistics were also calculated and sorted to assess the weaknesses and strengths of the current information on varicoceles.
In addition, a large number of the videos were found to be produced by authors outside the United States. The United States versus World production analysis was conducted using a homoscedastic t-test to compare the means of the two groups for the DISCERN, PEMAT-AV understandability, and PEMAT-AV actionability scores.
Statistical analyses were perform using Microsoft Excel (Microsoft Excel 16.0, Microsoft, Redmond, WA, USA) and Google Sheets (Google LLC, Mountain View, California, USA). An IRR was calculated to validate the DISCERN and PEMAT-AV scores between the three raters. Linear regressions between run time to mean DISCERN scores and number of views to mean DISCERN scores were created. A one-sample hypothesis testing for correlation was conducted on both linear regressions.
| Results|| |
For general characteristics of the 50 videos, the average length of each video was 6 min and 17 s with an average of 100,546 views, 460 likes, 44 dislikes and 3 years and 5-month-old. Fifty-six percent of the videos featured a US licensed physician. Of note, 24% of the videos featured non-US licensed physicians. Only 4% of the videos contained false statements, all of which came from the same channel that promoted nonscientific-based claims of natural resolving varicoceles such as “a proper diet will help minimize the (renal) nutcracker syndrome.”
The descriptions and average with standard deviation of each question are found on Table 1. The videos achieved a median DISCERN score of 31 (interquartile range 25–35) and an average of 31.34 (±9.37), which represents “poor quality.” 48% of the videos received “very poor,” 38% received “poor,” 10% received “fair” and 4% received “good” ratings, with none achieving an “excellent” rating. Overall, the highest rated question was “was it relevant?” and the lowest was “does it provide support for shared decision-making?” [Table 1]. The linear regression showed a positive correlation between the run time and DISCERN score (r = 0.71, P < 0.001) and a negligible correlation between the number of views and mean DISCERN score (r = −0.03799, P = 0.79).
|Table 1: Mean DISCERN scores with standard deviations of all 50 videos based on the 3 evaluators in this study|
Click here to view
Patient education materials assessment tool for audiovisual materials
To examine the understandability and actionability of the videos, PEMAT-AV was used. The overall average understandability of the videos was 69.8% ±15.4%, with 70% representing the cut-off for achieving understandability. A little over half of videos (56%) achieved this standard. [Table 2] shows the results of the PEMAT-AV understandability ratings. Most of the videos did not have a summary nor had informative headers as most of these videos had little to no postproduction. In addition, some of the videos had difficulty using only common, everyday language, or not explaining the medical terms used in the videos.
|Table 2: Mean Patient Education Materials Assessment Tool for Audiovisual Material understandability scores with standard deviations of all 50 videos based on the 3 evaluators in this study|
Click here to view
[Table 3] displays the results of the PEMAT-AV actionability. PEMAT-AV actionability ratings scored much poorer, with an average of 11.0% ±24.6%, with 60% as the cut-off for actionability. This cut-off was adapted because the last section of the actionability tool was not applicable (“The material explains how to use the charts, graphs, tables, or diagrams to take actions.”) Only 5 (10%) of the videos achieved this score. Actions such as subscribing to the channel were not included as pertinent actionability items as they have no clinical significance.
|Table 3: Mean Patient Education Materials Assessment Tool for Audiovisual Material actionability scores with standard deviations of all 50 videos based on the 3 evaluators in this study|
Click here to view
To assess the robustness of our study, the IRR score was calculated. [Table 4] represents all of the IRR scores based on the two tools used with the PEMAT-AV understandability and actionability scores separated.
|Table 4: The interrater reliability scores for the DISCERN, Patient Education Materials Assessment Tool for Audiovisual Material understandability, and Patient Education Materials Assessment Tool for Audiovisual Material actionability scores between the 3 evaluators for all 50 videos in this study|
Click here to view
Because of the large percentage of videos made outside of the United States (36%), a comparison between the origins of these producers is represented in [Figure 1]. The majority of these international producers were from India (83%). A comparison of the two sources of information shows that the international videos had a mean run time of 7 min and 2 s, while the US videos ran for 5 min and 49 s. Although the analysis showed that the international producers had higher averages in all three categories, none of the P values from the homoscedastic t-test were statistically significant. These P values were 0.28, 0.67, and 0.20 for the DISCERN, PEMAT-AV understandability, and PEMAT-AV actionability scores, respectively.
|Figure 1: A comparison of the mean PEMAT-AV and DISCERN scores with standard deviations of video producers from the United States versus the rest of the World, showing a statistically insignificant higher score in the international PEMAT-AV and DISCERN scores|
Click here to view
| Discussion|| |
There has been a growing movement in the urological community to address the increase in use of social and digital media to engage the patient population., Although patients still highly trust the information from their physicians, there is an increasing reliance on the internet to gather health-related information. Therefore, it is imperative that an objective analysis of medical information, including that in the urological field, be conducted on a periodic basis. This is the first study to examine the quality of the information on varicoceles provided on YouTube for the consumption by the general public.
Utilizing the first 50 nonduplicated videos, this study encompasses more than 5 h of videos with over 5 million views. The overwhelming majority of these videos were meant to be educational with only 4% of the videos representing “personal experiences” or “blogs.” Based on this study, the quality of the information provided in these videos were rated as “poor” on the DISCERN tool. Understandably, longer video scored higher in the DISCERN tool because of the time needed to provide reliable information. However, there was no correlation between the number of views and the quality of the video. This is of concern as it would be ideal that higher-quality videos would be accessed more often than those of lower quality.
Many of these videos seemed to lead patients into actively seeking treatment through the various methods available for varicocele, often focusing on research indicating varicocele as the most common known causes of male infertility within the videos.,, With a large portion (60%) of the content creators being physicians from a private practice, they may have a more vested interest in convincing their patients that surgical treatment is the only and best method improving male fertility. In addition, they often fail to provide opportunities for the viewer to search additional resources or provide the support for shared-decision making with their health provider. Luckily, all but one content creator provided accurate information. It will be an ethical challenge to combat misleading health information as the goal of providing ease of access to information is meant to be inclusive to the entire population. Alienation to certain groups because of health misinformation can be extremely polarizing, as seen with “anti-vaxxers.”
Although there are major pitfalls in the informational quality of the videos, a strength was found in that they generally rated well in the understandability of information provided. The high PEMAT-AV understandability scores of these videos can be attributed to the visual appeal, the abundance of cheap, high-quality audio and video equipment, and the ease of use for postproduction programs on the market. Many of these videos could have been recorded using a mobile telephone device, edited using free programs like iMovie on Apple products, and still be able to achieve high scores on DISCERN and PEMAT-AV.
With YouTube's international market penetration, it seems that the same pitfalls that the American medical video producers suffer from are shared with their international counterparts, represented by the statistically insignificant difference between the mean DISCERN and PEMAT-AV scores. However, the quality of these videos tended to be slightly higher, which can be correlated to the longer run times in the international videos. The overwhelming majority of the international videos were from India. With China having a ban on all Alphabet products, the parent company of YouTube, India is the largest country by population to upload and view content on YouTube's platform. There are an estimated 342.3 million viewers in India, which surpasses the current US population, making it unsurprising that Indian producers made up the majority of the international population.
This study is not without limitations, most pertinently the massive amount of information being constantly uploaded on the YouTube platform and some of the subjectivity of the DISCERN tool. This study must be defined on a specific timeline, while the amount of information and the YouTube algorithm to rank videos on its website remains fluid. This fluidity will require periodic checks to assess the evolution of information quality. An additional limitation is the DISCERN tool's flexibility to allow some rater interpretation of a video's ability to satisfy each quality criterion by giving partial scores between 2 and 4. Regardless, these scores and the PEMAT-AV scores were shown to have good robustness as the IRR scores ranged from 0.513 to 0.925, indicating a “moderate” to “almost perfect” kappa scores. This and future studies that continue to examine the quality of new information being uploaded on the platform can provide guidance for content creators to address the drawbacks of the current information and take in considerations of providing a more holistic view of the topic of varicoceles.
As YouTube's popularity increases for health education and information, it is important that individual medical specialties and the medical community as a whole work to assess the quality of this information. This is not usually done by YouTube's parent company Google, as their videos are listed usually on popularity and other proprietary algorithms that do not take scientific analysis into account. It is in the opinion of the authors that if the most subscribed individual YouTube content creator, PewDiePie, were to create a video on varicoceles, it would be the first video option in searches related to this topic due to his over 108 million subscribers, regardless of the validity of the content in this hypothesized video. This creates challenges in the patient population to accessing quality and verified information about health problems. However, this provides an opportunity for the medical and scientific community to embrace this method of information dissemination and as an opening to work with these large broadcasting platforms.
| Conclusions|| |
We found that the quality of the information on varicoceles on YouTube is poor. There is a need to develop videos that approache varicoceles more holistically that not only inform patients, but also empower them to make the decision that best reflects their goals in treatment.
Financial support and sponsorship: Nil.
Conflicts of interest: There are no conflicts of interest.
| References|| |
Ahluwalia S, Murray E, Stevenson F, Kerr C, Burns J. 'A heartbeat moment': Qualitative study of GP views of patients bringing health information from the internet to a consultation. Br J Gen Pract 2010;60:88-94.
Caiata-Zufferey M, Schulz PJ. Physicians' communicative strategies in interacting with Internet-informed patients: Results from a qualitative study. Health Commun 2012;27:738-49.
Sommerhalder K, Abraham A, Zufferey MC, Barth J, Abel T. Internet information and medical consultations: Experiences from patients' and physicians' perspectives. Patient Educ Couns 2009;77:266-71.
Tan SS, Goonawardene N. Internet health information seeking and the patient-physician relationship: A systematic review. J Med Internet Res 2017;19:e9.
Drozd B, Couvillon E, Suarez A. Medical YouTube videos and methods of evaluation: Literature review. JMIR Med Educ 2018;4:e3.
Goobie GC, Guler SA, Johannson KA, Fisher JH, Ryerson CJ. YouTube videos as a source of misinformation on idiopathic pulmonary fibrosis. Ann Am Thorac Soc 2019;16:572-9.
ReFaey K, Tripathi S, Yoon JW, Justice J, Kerezoudis P, Parney IF, et al
. The reliability of YouTube videos in patients education for Glioblastoma treatment. J Clin Neurosci 2018;55:1-4.
MacLeod MG, Hoppe DJ, Simunovic N, Bhandari M, Philippon MJ, Ayeni OR. YouTube as an information source for femoroacetabular impingement: A systematic review of video content. Arthroscopy 2015;31:136-42.
Loeb K, Sengupta S, Butaney M, Macaluso Jr J, Czarniecki S, Robbins R, et al
. Dissemination of misinformative and biased information about prostate cancer on YouTube. Eur Urol 2019;75:564-7.
Loeb S, Carrick T, Frey C, Titus T. Increasing social media use in urology: 2017 American Urological Association survey. Eur Urol Focus 2020;6:605-8.
Baydilli N, Selvi I. Is social media reliable as a source of information on Peyronie's disease treatment? Int J Impot Res. 2021. doi: 10.1038/s41443-021-00454-3. Epub ahead of print.
Duran MB, Kizilkan Y. Quality analysis of testicular cancer videos on YouTube. Andrologia 2021;53:e14118.
Ji L, Sebesta EM, Stumbar SE, Rutman MP, Chung DE. Evaluating the quality of overactive bladder patient education material on YouTube: A pilot study using the patient education materials assessment tool. Urology 2020;145:90-3.
Alsaikhan B, Alrabeeah K, Delouya G, Zini A. Epidemiology of varicocele. Asian J Androl 2016;18:179-81.
] [Full text]
Agarwal A, Hamada A, Esteves SC. Insight into oxidative stress in varicocele-associated male infertility: Part 1. Nat Rev Urol 2012;9:678-90.
Morahan-Martin JM. How internet users find, evaluate, and use online health information: A cross-cultural review. Cyberpsychol Behav 2004;7:497-510.
Charnock D, Shepperd S. Learning to DISCERN online: Applying an appraisal tool to health websites in a workshop setting. Health Educ Res 2004;19:440-6.
Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105-11.
Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns 2014;96:395-403.
Teoh JY, Mackenzie G, Tortolero L, Rivas JG. Social media analytics: What you need to know as a urologist. Eur Urol Focus 2020;6:434-6.
Rivas JG, Socarras MR, Patruno G, Uvin P, Esperto F, Dinis PJ, et al.
Perceived role of social media in urologic knowledge acquisition among young urologists: A European survey. Eur Urol Focus 2018;4:768-73.
Hesse BW, Moser RP, Rutten LJ. Surveys of physicians and electronic health information. N Engl J Med 2010;362:859-60.
Kupis Ł, Dobroński PA, Radziszewski P. Varicocele as a source of male infertility – Current treatment techniques. Cent European J Urol 2015;68:365-70.
Kantartzi PD, Goulis ChD, Goulis GD, Papadimas I. Male infertility and varicocele: Myths and reality. Hippokratia 2007;11:99-104.
Bradshaw AS, Shelton SS, Wollney E, Treise D, Auguste K. Pro-vaxxers get out: Anti-vaccination advocates influence undecided first-time, pregnant, and new mothers on facebook. Health Commun 2021;36:693-702.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.
[Table 1], [Table 2], [Table 3], [Table 4]