Reliability of a Newly Developed Tool to Assess and Classify Work-related Stress (TAWS-16) for Indian Workforce
Article information
Abstract
Objectives
Work stress is associated with non-communicable diseases, increased healthcare costs, and decreased work productivity among employees in the information technology sector. There is a need for regular work-stress screening among employees using valid and reliable tools. The Tool to Assess and Classify Work Stress (TAWS-16) was developed to overcome limitations in existing stress assessment tools in India. This study aimed to test the reliability of TAWS-16 in a sample of managerial-supervisory employees.
Methods
This observational reliability study included data from 62 employees. Test-retest and inter-method reliability were investigated using a TAWS-16 web application and interview by telephone, respectively. Kappa values and intra-class correlation coefficients were calculated. Internal consistency was assessed through Cronbach’s alpha.
Results
For both test-retest and inter-method reliability, the agreement for both work-related factors and symptoms suggestive of work stress exceeded 80%, and all kappa values were 0.40 or higher. Cronbach’s alpha for test-retest and inter-method reliability was 0.983 and 0.941, respectively.
Conclusions
TAWS-16 demonstrated acceptable reliability. It measured stressors, coping abilities, and psychosomatic symptoms associated with work stress. We recommend using TAWS-16 to holistically identify work stress among employees during periodical health check-ups in India.
INTRODUCTION
An estimated 44 million people are employed in organised sector workplaces in India (Census 2011) [1]. Within the organised sector, information technology (IT) and IT-enabled service sector employees form a key workforce from an economic perspective. The IT sector’s contribution to India’s gross domestic product (GDP) increased from 1.2% in 1998 to 7.4% in 2022, and it is expected to contribute 10% to India’s GDP by 2025 [2]. India’s IT sector is the leading sourcing destination globally, accounting for approximately 55% of the global market share of the services sourcing business (year 2019–2020). During the fiscal year 2021–2022, the IT industry employed 5.1 million people in India [3], and the good health of these employees is vital for sustained economic growth and development [4].
With rapidly evolving work ecosystems, the health profile of relatively young employees is changing. Interactions with medical officers indicate that obesity, hypertension, impaired glucose tolerance, mental disorders, substance use, and repetitive stress injuries are commonly observed in the IT workforce. Work-related stress is often reported to be closely associated with many health and productivity issues in IT employees [5].
Work stress was observed in an estimated 51% of IT professionals [6] and in 55% of managerial employees [7]. Despite methodological limitations and variations, it is generally accepted by stakeholders that work stress is widely prevalent and is an important issue to be addressed because the Indian workforce is huge and consists of the young, productive segment of the population. Work-related stress is a response people may have when presented with work demands and pressures that are not matched to their knowledge and abilities and that challenge their ability to cope [8]. Low job security, excessive work demands, lack of job control, monotonous work, lack of organisational support, adverse physical working conditions, inflexible work hours, relationships at work, role conflict and ambiguity, and work-life imbalance are known work stressors [9].
Stress in harmful proportions increases the risk of non-communicable diseases (NCDs) and subsequent complications, which may increase healthcare costs and decrease work productivity. It is essential that harmful levels of work-related stress should be detected early and managed for optimum health and work efficiency. However, regular screening for work stress is non-existent in most workplaces because it is not a mandated part of the periodic medical examination process. In addition, few tools or questionnaires to screen for harmful work stress are available, and they are lengthy, not validated for Indian workplaces, or limited in scope, without comprehensively covering the experience of health-related symptoms due to harmful stress levels. Copyright and the costs of study instruments limit their application in low-resource settings and by researchers.
There is an unmet need for a short screening tool to identify harmful work stress and its related symptoms and to evaluate them amid the regular examination of workers. Towards this goal, Tool to Assess and Classify Work-related Stress (TAWS-16) was developed by the Centre for Public Health, National Institute of Mental Health and Neurosciences (NIMHANS) and used in many workplaces. The overview, content, criteria, and construct validity of TAWS-16 have been published [5], and the current paper presents the results for the instrument’s reliability and internal consistency.
METHODS
This reliability assessment study is part of a larger validation study of TAWS-16 conducted on 356 managerial-supervisory staff affiliated with select IT companies in Bengaluru, Karnataka, India, from October 2020 to December 2020. Reliability assessments were conducted on a random sample of 62 study subjects drawn from the set of 356 IT staff. Sampling details are available at https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0280189 . The sample size for the reliability assessment was estimated based on an expected intra-class correlation coefficient (ICC) of 0.8 [10], a significance level of 0.05, a test power of 80%, and a 20% non-response rate. The final sample size was calculated to be 62 subjects for the assessment of test-retest, inter-method reliability, and internal consistency.
TAWS-16 consists of 2 sections, A and B, with 16 items each. Items in Section A investigate the work-stressor experience of employees over the past 6 months of employment and categorize their work-stress levels. Table 1 presents these items and their domains of assessment. Details regarding items, responses, scoring for each item, and cut-off scores for categorization of work-stress levels are available in the Instruction Manual of TAWS-16 [11]. Employees reporting experience of work stressors affirmatively in the last 6 months (for each item in Section A) are asked further regarding their ability to cope or efficiently manage each reported stressor. Subjects scoring >48 points in Section A are classified as having work stress, with different cut-off scores for mild, moderate, and severe stress. Items in Section B investigate commonly reported psychosomatic symptoms suggestive of work stress. Employees experiencing symptoms are asked further about their frequency. A cross-tabulation of different categories of work-stress levels and symptom levels is used to provide a color code and intervention guide for occupational health managers, from an NCD risk-reduction perspective [5].
The summarized domains of assessment in the study instrument
Data Collection
Data collection was initially planned to occur through face-to-face interviews with the study participants, but we changed the approach due to the spread of coronavirus disease 2019 (COVID-19) in Bengaluru. A web application was developed for TAWS-16, following the complete life cycle of software development. Participants from consenting IT companies in Bengaluru were contacted, and a web link ( https://app.esamiksha.in/stress_assessment) was provided to gather their responses. We received a total of 356 responses.
Assessing inter-method reliability
A simple random sample of 62 respondents was contacted by telephone and interviewed 1 week later, with the interview completed within 7 days of contact, to assess reliability between the self-administered web method and the telephone interview method. Inter-method reliability was considered acceptable if the ICC >0.7.
Test-retest reliability
Test-retest reliability provides an indication of stability over time. A TAWS-16 link was provided to participants on 2 separate occasions within 1 week for self-administration and reporting. Test-retest reliability was considered acceptable if the ICC >0.7.
Internal consistency
Internal consistency concerns the extent to which items on the test or instrument measure the same thing. It was assessed by computing Cronbach’s alpha for each item, each sub-scale, and the entire study instrument after data collection from all study subjects, with 0.8 considered minimally acceptable.
Data from the application was provided to the investigator as a comma-separated values file, which was converted to Excel format (Microsoft, Redmond, WA, USA) and checked for duplicate values, outlier issues, and consistency in responses. Coding and data were analyzed using SPSS version 25 (IBM Corp., Armonk, NY, USA). Data privacy and confidentiality were ensured because the web application captured anonymous, de-linked data, with each subject creating his/her own user ID and password. Identifiers such as personal name, phone number, company name, and residence details were not collected.
Statistical Analysis
Test-retest and inter-method reliability were assessed for agreement beyond chance using unweighted percentage agreement (for 2×2 contingency tables) and kappa statistics [12]. Both ICCs and Spearman’s rank correlation coefficients [13] were computed as a measure of test-retest reliability and inter-method reliability. ICCs were classified as follows: “excellent” (≥0.81), “good” (0.61–0.80), “moderate” (0.41–0.60), and “poor” (≤0.40). The values of kappa were characterized as follows: “slight” (0.0–0.2), “moderate” (0.2–0.4), “fair” (0.4–0.6), “substantial” (0.6–0.8), “almost perfect” (0.8–1.0) [10]. Cronbach’s alpha was computed to assess internal consistency for both inter-method and test-retest reliability, with 0.8 [14] considered minimally acceptable.
Ethics Statement
Scientific and ethical clearance was obtained by NIMHANS Ethics Committee No. NIMH/DO/IEC (BS & NS DIV) 2020–21. Informed consent (verbal and through web-link) was obtained from each study participant. Participants were informed that they could withdraw from the study at any time.
RESULTS
We studied 62 participants, of whom 50 respondents (80.6%) did not reveal any work stress in either method (M1 and M2). Statistically significant inter-method reliability was observed between the self-administered and telephone interview methods for assessing work-related stress using TAWS-16 (kappa= 0.525, p<0.001; Table 2). Statistically significant test-retest reliability was observed for TAWS-16 as well (kappa=0.475, p<0.001).
Reliability assessment of TAWS-16 for work-related stress
For assessing psychosomatic symptoms suggestive of work-stress using TAWS-16, both inter-method and test-retest reliability assessments were conducted. Of the 62 participants, 35 (56.5%) did not reveal any symptoms suggestive of stress in either the self-administered or telephone interview methods. A statistically significant agreement (p<0.001) was observed for the ability of TAWS-16 to assess psychosomatic symptoms suggestive of work stress in both the self-administered and telephone interview methods (kappa=0.572, p<0.001; Table 3). Similar results were observed for the test-retest reliability assessment.
Reliability assessment of TAWS-16 for symptoms suggestive of work stress
The reliability of TAWS-16 was assessed through test-retest and inter-method reliability tests. Table 4 shows the ICC values for test-retest (0.96) and inter-method reliability (0.88), indicating excellent reliability. Variation due to systematic errors in the inter-method assessment of TAWS-16 was around 12%, and that of the test-retest assessment was 3.6%, as per ICC values. The highest ICC value was observed for test-retest reliability (0.96) regarding work-stress assessment. All other ICC values showed excellent scores. Significant positive correlations were observed for both test-retest and inter-method reliability. Spearman’s rho ranged from 0.88 to 0.95, with the highest value (0.95) found for test-retest reliability for work-stress assessment.
Internal Consistency
Cronbach’s alpha for both test-retest and inter-method reliability showed good internal consistency. For the work-stress assessment items, the values of alpha for test-retest and inter-method reliability were 0.983 and 0.941, respectively. Similarly, for the items of symptoms suggestive of work-stress, the values of alpha for test-retest and inter-method reliability were 0.918 and 0.924, respectively (Table 4).
DISCUSSION
The reliability of TAWS-16 was investigated with a sample of 62 IT employees in Bengaluru, extracted from our overall validation study. A strength of the study was that it tested the reliability of a much-needed work-stress assessment tool, with a focus on increasing the scope of work-stress screening by assessing reliability between the self-administered and interview-administered methods. The study was completed despite the COVID-19 pandemic among IT professionals, who are a key working population in India. The web application ensured accuracy and completeness in data collection and has the potential to be customized for different workplaces and industries. The COVID situation provoked various aspects of work stress that invite further research and interventions.
TAWS-16 not only assesses and quantifies work stress, but also measures coping abilities and the experience of psychosomatic symptoms suggestive of work stress. TAWS-16 is brief, easy to administer, and easily understood by respondents. This tool has already been used in various industries, and implementing it has proven to be feasible.
The study results indicate that TAWS-16 has acceptable to good test-retest reliability. Because of methodological differences, it would not be epidemiologically correct to make direct comparisons with the findings of other studies, but the reliability characteristics of TAWS-16 appear to support its suitability for use.
For both test-retest and inter-method reliability, the agreement for work-related factors and symptoms suggestive of work stress exceeded 80%, and all kappa values were 0.40 or higher, suggesting acceptable test-retest reliability. The results for both test-retest and inter-method reliability were quite consistent. Similarly, almost all values of the ICCs and Rs regarding reliability exceeded 0.85 for both work-related factors and symptoms suggestive of work stress.
It is entirely plausible that work-related factors causing stress and the physical symptoms suggestive of work stress are consistent over the period of time studied, and hence our findings revealed a similar level of agreement. An underlying assumption of studies of test-retest reliability is that the object of measurement is very stable and is unlikely to change between measurements. It should also be noted that categorical measures (e.g., measures describing respondents as experiencing or not experiencing stress) will lower the values of the reliability statistics of those measures.
Though we have not undertaken a formal comparison in the study, a review of reliability measures indicates that TAWS-16 is as reliable as existing tools in terms of psychometric reliability scores, such as the Occupational Stress Scale [15], Job Description Index [16], Work Role Inventory [17], Work Ability Index and Work Ability Score [18], and Job Content Questionnaire [19].
The study’s limitations were closely related to the ongoing COVID-19 situation, since we were unable to conduct an inter-observer reliability measurement. Working from home might produce intermittent variations in stress responses for IT employees, which might interfere with test-retest and inter-method reliability as well. The questionnaire is presently available in English. For wider application in informal sectors, it would be necessary to translate TAWS-16 into other Indian languages and conduct cross-linguistic validation studies as well.
Since publicly available tools specific to measuring and classifying work stress in Indian work settings are limited, TAWS-16 will help to bridge this unmet need in occupational health services in India. The validity of TAWS-16 has been established and published. The findings of this study indicate that TAWS-16 has acceptable to excellent inter-method and test-retest reliability and internal consistency. TAWS-16 can be used for both self-report and interview-based screening for work stress. Although further assessment using other methods and in other demographic groups is required, TAWS-16 shows promise as a useful research and evaluation tool for work-stress assessment and classification.
Notes
CONFLICT OF INTEREST
The authors have no conflicts of interest associated with the material presented in this paper.
FUNDING
None.
AUTHOR CONTRIBUTIONS
Conceptualization: Gururaj G, Sukumar GM. Data curation: Roy R. Formal analysis: Roy R, Philip M. Funding acquisition: None. Methodology: Roy R, Sukumar GM, Philip M. Project administration: Roy R, Sukumar GM. Visualization: Roy R. Writing – original draft: Roy R. Writing – review & editing: Roy R, Sukumar GM, Philip M, Gururaj G.
ACKNOWLEDGEMENTS
The authors would like to express their sincere gratitude to the AKA Labs team for their support and development of the TAWS-16 web application. Most significantly, our sincere thanks to the domain experts and IT professionals for their cooperation and participation in the study.