|Year : 2022 | Volume
| Issue : 1 | Page : 29-33
Utility of WhatsApp in emergency urological practice: An interrater reliability study
Aditya Prakash Sharma, Saket Singh, Sudheer Kumar Devana, Kapil Chaudhary, Tarun Pareek, Shrawan K Singh
Department of Urology, PGIMER, Chandigarh, India
|Date of Submission||24-Mar-2021|
|Date of Decision||20-Sep-2021|
|Date of Acceptance||06-Dec-2021|
|Date of Web Publication||1-Jan-2022|
Department of Urology, PGIMER, Chandigarh
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Introduction: The messaging application 'WhatsApp' is used in clinical practice, often for communication between a medical trainee and a consultant. We designed this study to find the interrater reliability of the data transmitted through this application and validating its use in urological practice.
Materials and Methods: Clinical details and computerized tomographic (CT) images of 30 patients visiting the urology emergency were posted in a closed WhatsApp group involving three consultants (SKD, APS, and KC). The CT images were posted in the WhatsApp group as Whole Image (WI) and Image of Interest (IOI) format and rated on a scale of 1–5. The consultants formulated a provisional diagnosis and initial management strategy. The interrater reliability of these responses was analyzed in the study.
Results: Mean WI rating ranged from 3.03 ± 0.61 to 3.73 ± 0.64 (Cronbach alfa [α]-0.494, P = 0.006). Mean IOI rating ranged from 3.4 ± 0.56 to 4.13 ± 0.73 (α-0.824, P < 0.0001). For diagnosis, the proportion of observed agreement (P0) was 83.3% for SKD and APS, 76.6% for SKD and KC, and 73.3% for APS and KC. For management, P0 was 86.6% for APS and KC, 86.6% for SKD and APS, and 80% for SKD and KC.
Conclusions: WhatsApp Messenger serves to transmit good quality pictures of CT scan images. A reasonable diagnosis and management strategy can be formulated using this app with fair inter-rater reliability.
|How to cite this article:|
Sharma AP, Singh S, Devana SK, Chaudhary K, Pareek T, Singh SK. Utility of WhatsApp in emergency urological practice: An interrater reliability study. Indian J Urol 2022;38:29-33
|How to cite this URL:|
Sharma AP, Singh S, Devana SK, Chaudhary K, Pareek T, Singh SK. Utility of WhatsApp in emergency urological practice: An interrater reliability study. Indian J Urol [serial online] 2022 [cited 2023 Feb 6];38:29-33. Available from: https://www.indianjurol.com/text.asp?2022/38/1/29/334595
| Introduction|| |
WhatsApp is a smartphone-based application used frequently for telecommunication., As of July 2019, WhatsApp has reached over 2000 million users worldwide and 400 million users in India, thus having a firm digital footprint., Despite this heavy base of digital communication platforms in India, WhatsApp was not officially approved for providing telemedical consultation until recently. Still, this app has increasingly been used as an “off-label” means of communication for teleconsultation in clinical medical practice before COVID (BC) era., Due to the COVID-19 pandemic, its use was approved in the guidelines provided by the Indian Medical Council and vetted by the Ministry of Health and Family Welfare, India, for its use in teleconsultation within 3 days of the first lockdown on March 26, 2020. With this official approval, the app came to used as a means of teleconsultation.
Another prime use of WhatsApp in clinical practice is communication between a medical trainee/resident and a consultant, where the resident seeks opinion regarding a particular case for proper management. However, studies validating the use of WhatsApp for this purpose, especially in urological practice, are lacking. We are unsure whether the use of WhatsApp as a mode of transmission of information can have the same reproducibility as that of physical consultation. Is there a potential loss of data or misinformation or misinterpretation of data sent through WhatsApp? It also needs to be determined whether this makes any difference in decision-making for patients and clinical care. Hence, we designed this study to find the inter-observer variation and inter-rater reliability of the data transmitted through this app in real-world setting and thus validating its use in urological practice.
| Materials and Methods|| |
The prospective study was conducted from June 1, 2020, to July 15, 2020 at a tertiary care center after taking approval from the institute ethics committee (NK/6316/Study/190). Thirty patients presenting for emergency urological consultations under care of three urology consultants (APS, SKD, and KC) of a single unit were included.
All the emergency consultations to the urology department under the three consultants of single unit (APS, SKD, and KC) with relevant cross-sectional imaging in the form of computed tomography (CT) scan were initially evaluated by the resident posted for emergency urology care. The clinical details and radiological images were posted in a closed WhatsApp group involving the three consultants and a senior resident (SR) after taking informed consent from the patient. The relevant radiological images were posted in the WhatsApp group as a Whole Image (WI) and Image of Interest (IOI) format. WI included the photo of whole CT scan sheet with multiple cross-sectional images and IOI included the specific cross-sectional image with the possible abnormality or pathology [Figure 1]. It may be prudent to mention here that the facility of Picture Archiving and Communication System is still not well established at our center.
|Figure 1: Screenshot of CT scan images sent by the resident to consultant (a) Whole Image (b) Image of Interest|
Click here to view
All three consultants independently reviewed the clinical details and the images provided within 15–30 min of receiving the images and data. They rated the quality of image both WI and IOI separately on Likert scale of 1–5 (1– very poor, 2 – poor, 3 – good, 4 – very good, and 5 – excellent). The resident rated the image quality of original image and noted it separately. He also formulated his provisional diagnosis and line of management. Similarly, the consultants after looking at the images formulated a provisional diagnosis and initial management strategy. They sent these responses (Quality of WI, Quality of IOI, provisional diagnosis, and management) as a personalized WhatsApp message to the resident separately to maintain blinding from each other's responses. All the responses were recorded by the resident in the case report form and later entered in a Microsoft Excel Sheet.
In case of gross discrepancy in the responses, the initial response was recorded for analysis, and then management options were discussed among the three consultants before making the final decision of line of treatment for a particular patient. Respective phone models used were as follows: Consultant 1-SKD (iPhone 11, Apple, USA), Consultant 2-APS (One plus 5T, China), Consultant 3-KC (iPhone 8, Apple, USA), SR-(iPhone 6s, Apple, USA). All the phones of consultants and resident are meant for personal use and are password protected. WhatsApp Messenger safeguarded the information by providing end-to-end encryption. The images were not shared out of this closed WhatsApp group and were archived in a safe encrypted folder on a computer and deleted from all the smartphones after completion of the study.
Data were entered in Microsoft Excel 2010 and were analyzed using IBM SPSS Statistics for Windows, version 21 (IBM Corp., Armonk, New York, USA). The data were expressed as number and percentage. Categorical data between the groups were analyzed from a 2 × 2 contingency table. The concordance for the rating of image was represented using Cronbach alpha, and interrater reliability was represented by the interclass correlation coefficient (ICC) for rating of imaging. For diagnosis, coding was done in binary fashion, 0 for observed disagreement and 1 for observed agreement to the diagnosis of resident. For management, coding was done as 0 for observed disagreement and 1 for observed agreement to the final management of the patient. P0 was defined as a proportion of observed agreement, which is given as sum total of agreements divided by the total response (a + d/N) where a = total positive agreements, d = total negative agreements, and N = Total responses (30). The interobserver agreement between two raters was calculated using kappa (κ) statistics. P < 0.05 was considered statistically significant.
| Results|| |
A total of 30 patients (20 males and 10 females) were included in the study. Mean age of the patients was 45.47 ± 13.68 years. The demographic details, provisional diagnosis, and imaging details are provided in [Supplementary Table 1][Additional file 1]. The mean WI rating was as follows: resident – 3.27 ± 0.58, SKD – 3.70 ± 0.75, APS – 3.73 ± 0.64, and KC – 3.03 ± 0.61. The Cronbach alpha for WI rating was 0.494 and ICC was 0.196 (P = 0.006). The mean IOI rating by resident (the resident rated the actual imaging) was 4.07 ± 0.78. The IOI rating by the three consultants was as follows: SKD – 4.07 ± 0.87, APS – 4.13 ± 0.73, and KC – 3.4 ± 0.56. The Cronbach alpha for IOI was 0.824 and ICC was 0.540 (P < 0.0001). None of the consultants asked for any added IOI to be sent separately.
For diagnosis, proportion of observed agreement P0 was 83.3% for SKD and APS, 76.6% for SKD and KC, and 73.3% for APS and KC. For initial management, P0 was 80% for APS and KC, 70% for SKD and APS, and 70% for SKD and KC. As compared to he resident, the P0 was 86.67% for resident and KC, 80% for resident and APS, and 76.67% for resident and SKD. The corresponding kappa values are provided in [Table 1].
|Table 1: Inter observer agreement for management decisions: Kappa values for various readers|
Click here to view
There was difference in formulating the management strategy in terms of the modality of diversion used for obstructive uropathy such as placement of double J stent versus a percutaneous nephrostomy (4/30). There also was difference in terms of approach to a particular procedure such as laparoscopic versus open radical nephrectomy (1/30). We clubbed the modalities “DJ stenting” and “PCN placement” as “Urinary diversion.” The modalities “laparoscopic radical nephrectomy” and “open radical nephrectomy” were combined as “radical nephrectomy.” After this recoding, the P0 and interrater reliability increased further and is shown in [Table 2].
|Table 2: Interobserver agreement for management decision: Kappa values for various readers after adjustment|
Click here to view
| Discussion|| |
In the era of social distancing, the use of teleconsultation is likely to increase., Hence, studies regarding the validation of tools for teleconsultation are needed to optimize their use. In this study, we intended to determine the inter observer variation and calculate inter-rater reliability of use of clinical data transmitted through WhatsApp in emergency urological setting. In our study, the mean scores of the image rating reflected “good (3)” to “very good (4)” rating for all transmitted radiological images. The mean scores for the WI were less than that given for the IOI by all the observers. This entails from the fact that there is always a need to zoom the WI to look for the abnormality/pathology, which leads to loss of pixels of the WI, resulting in blurring of image when seen on a mobile phone [Figure 2]b. However, the IOI is transmitted after clicking an individual image which does not require zooming and hence has no such problems of pixel loss [Figure 1] and [Figure 2]a. Thus, there is better perception and readability for IOI than WI.
|Figure 2: Image showing a section sent as Image of Interest (a) and the same section of CT scan after zooming in the Whole Image (b). Note the blurring of image after zooming and thus causing loss of information|
Click here to view
There was wide inter-rater variability in rating the quality of WI when compared to rating of quality of IOI as is seen from the values of Cronbach alpha (0.494 vs. 0.824) and ICC (0.196 vs. 0.540). The plausible explanation for the same could be difference in the rater's subjective interpretation of the quality of WI and the phone model used. Scanning the whole CT image on phone and interpreting it leads to the difference in ratings assigned to WI. Furthermore, as mentioned earlier, the need for zooming in and consequential loss of pixels leads to blurring of image [Figure 2]. The perception of this blurred and zoomed image leads to a significant difference in subjective rating by the observer, while for IOI alone such zooming is infrequently needed.
Regarding diagnoses, it was seen that the proportion of observed agreement P0 values among consultants were good and ranged from 73.3% to 83.3%. Thus, it may be inferred that despite the difference in rating the quality of images, the data may be interpreted to reach a reasonable urological diagnosis. On further analyzing the data on nonagreement cases, we noticed that the consultants found additional findings on imaging other than those pointed by the resident in 5/30 occasions (SKD and APS) and 4/30 occasions (KC), respectively. This amounts to a very high likelihood (>80%) of users finding the said diagnosis, based upon the clinical data and imaging using WhatsApp, irrespective of the image quality.
The final and the most important outcome of any teleconsultation is its utility in providing a management strategy after reviewing the clinical data. In our study, when we looked at the formulation of management strategy, we found high P0 ranging from 70% to 86.7%. Despite this high proportion of observed agreement, we had low kappa values (0.270–0.627). This paradoxical low κ values despite the high observed agreement is due to prevalence dependency of κ, which has been described in detail by Cicchetti and Feinstein and must be interpreted with caution., In cases where exact prevalence or the “gold standard” is unknown the calculation of prevalence is done using marginal totals from a 2 × 2 table for observed agreements and disagreements. In case the marginal totals (observed agreement or disagreement) become very low either vertically or horizontally as in our study, the κ lowers drastically for the same value of the proportion of observed agreement (Po). Thus, it is the proportion of observed agreement which become more relevant than κ alone in such situations.
As mentioned in the results, the observed agreement and κ increased further after recoding to “Urinary diversion” and “radical nephrectomy” as mentioned in the result section. This recoding was done since approaches to a particular procedure are likely to differ even in physical consultation, and hence the recoded values possibly reflect the true agreement among the observers. This difference of opinion can be sorted after mutual discussion, as was done in our series, and these differences are likely to arise even in cases where the clinical data and the imaging are presented physically to a group of physicians.
In a study by Sener et al. on evaluation of interrater reliability of WhatsApp for evaluation of hematuria, 212 patients were evaluated for hematuria by two groups of urologists. One group having direct access to the patients, while the other group comprising urologist blinded to patients' data and received image on WhatsApp. The grade of hematuria was evaluated by them as follows: Hematuria with the following rating: 0 – no hematuria, 1 – hematuria that does not require invasive treatment, and 2 – hematuria requiring bladder drainage or any form of active treatments. They found almost perfect agreement between two groups, (kappa-0.992). Another study for cystoscopic/ureteroscopic image was conducted by Arada et al., who found significant agreement between consultant and attending plans. The reply was in the form of “agree” or “disagree” to the formulated management plan by the attending. However, this being a conference abstract, the detailed methodology and results could not be ascertained. Contrary to these studies, the kappa values in our study are low (0.280–0.672) as previously mentioned. This difference may also be accounted for by the complexity of the problem being evaluated. Reading and interpreting CT scan images on phone is a complex task as compared to the interpretation of hematuria using color of urine and cystoscopic images.
In this current era of COVID-19, the use of telemedicine has increased exponentially. WhatsApp has provided a universal tool for teleconsults and telemedicine in resource-limited countries like India. The major advantages of using this app for telemedical consultation was its widespread user base, no extra cost for sending messages, calls (both audio and video), ability to share photos, videos and message using a single platform and with end to end encryption facility which maintained confidentiality of patient's identity and data., Amid the call for social/physical distancing the use of this tool for emergency consults and communication between the residents and consultants in each field is also bound to increase. The same app can now be used for seeking such consults and has been more so useful in after COVID-19 (AC) era where physical distancing is a norm. Thus, this study was designed to validate the use and assess the inter-rater reproducibility of teleconsults. This present study is the first of its kind addressing the use of WhatsApp for entire decision-making of the patients visiting emergency. The images and the clinical scenarios are much more complex than those in the already published literature. The study is a small step toward the incorporation of social media applications (such as Telegram, Facebook, Twitter, etc.) being used off label for clinical use into mainstream clinical practice. Of course, excessive use of such apps comes at the expense of a precious commodity, i.e., time and they have associated side effects related to increase in screen time such as eye and neck strains.
An important limitation of the study was the small sample size. The study had descriptive data for comparison (diagnosis and management); this derives from the fact that we wished to replicate the day-to-day clinical practice in our system. In this study, all the phones used can click and reproduce high-quality images. The quality of mobile phones used can also affect the interpretation of the image. We did not address the picture-taking skill of a particular trainee through this study. A comparator arm of physical control was not kept in view of COVID-19 situation.
| Conclusions|| |
WhatsApp messenger serves to transmit good quality pictures of cross-sectional imaging modality such as CT scans. These images along with appropriate clinical history can be used to formulate a reasonable diagnosis and management strategy with fair to substantial inter-rater reliability. WhatsApp can be used in emergency urological setting with significant agreement among the resident and consultant.
| References|| |
De Benedictis A, Lettieri E, Masella C, Gastaldi L, Macchini G, Santu C, et al.
WhatsApp in hospital? An empirical investigation of individual and organizational determinants to use. PLoS One 2019;14:e0209873.
Di Maida F, Scalici Gesolfo C, Fazio I, Mortellaro G, Blasi L, Borsellino N, Spada M, et al.
Whatsapp messenger as a tool for the multidisciplinary management in everyday clinical practice. Eur Urol Suppl 2017;16:e1445-6.
Whatsapp Hits 400 Million Monthly Active Users in India Out of 1.5 Billion User. Available from: http://www. Firstpost.com
. [Last accessed on 2020 Aug 28].
Sharma AP, Mavuduru RS, Singh SK, Mandal AK. WhatsApp use in urological practice: Yin and Yang! Indian J Urol 2019;35:172-3.
Martin G, Khajuria A, Arora S, King D, Ashrafian H, Darzi A. The impact of mobile technology on teamwork and communication in hospitals: A systematic review. J Am Med Inform Assoc 2019;26:339-55.
Luciani LG, Mattevi D, Cai T, Giusti G, Proietti S, Malossini G. Teleurology in the time of Covid-19 pandemic: Here to stay? Urology 2020;140:4-6.
Devana SK, Chaudhary K, Sharma AP, Singh SK. Changing urological practice during COVID-19. Indian J Urol 2020;36:153-8. [Full text]
Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990;43:551-8.
Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43:543-9.
Donker DK, Hasman A, van Geijn HP. Interpretation of low kappa values. Int J Biomed Comput 1993;33:55-64.
Sener TE, Butticè S, Sahin B, Netsch C, Dragos L, Pappalardo R, et al.
WhatsApp use in the evaluation of hematuria. Int J Med Inform 2018;111:17-23.
Arada EI, Florencio L, Macalalag M, Mendiola F, Dy J, Ballesteros C, et al.
Mp23-01 Using android smartphones to take cystoscopic and ureterosopic images for Exclusive online internet referrals. J Urol 2015;193:e267.
Kamel Boulos MN, Giustini DM, Wheeler S. Instagram and WhatsApp in health and healthcare: An overview. Futur Internet 2016;8:1-14.
Bellote MC, Santamaria HT, Pelayo-Nieto M, Heman Prasad ES, Gadzhiev N, Gudaru K. Social media in the urology practice | Opinion: YES. Int Braz J Urol 2019;45:877-81.
[Figure 1], [Figure 2]
[Table 1], [Table 2]