Delyth Lloyd1* MA MPH, Tracey Varker1 PhD, Thi Pham1 BSc, John O’Connor1 PhD, Andrea Phelps1PhD
Abstract
Online mental healthcare resources have proliferated at a greater pace than evidence for their effectiveness. They may nevertheless be an attractive alternative for contemporary veterans and serving personnel who are reluctant to engage in traditional face to face treatment. This has created an urgent need to evaluate the effectiveness of online mental health care for the military community. This paper reports on the two-stage evaluation of the Wellbeing Toolbox, a self-guided website for ex-serving members and their families. Stage 1 evaluated the reach and acceptability of the website. Results from user experience interviews and a survey of 291 open access users indicated that the site reached a relevant audience and was accessible and acceptable for the ex-service community. Stage 2 investigated the effectiveness of the Wellbeing Toolbox in achieving wellbeing goals (the primary outcome) and other mental health outcomes (secondary outcomes). All 30 participants in the effectiveness trial achieved at least some of their individual wellbeing goals, with most success in “getting active”, “building support” and “keeping calm” goals. There was no corresponding improvement in overall mental health status. The value and role of self-guided online help is discussed.
Conflict of Interest: None for any author. The Australian Centre for Posttraumatic Mental Health is partially funded by the Department of Veterans Affairs.
Introduction
After a decade of escalated military activity, particularly in the Middle East, we can expect to see an increase in the rate of service-related mental health and wellbeing issues. These include significant health and wellbeing problems, depression, anxiety, PTSD, and associated social, relationship and occupational challenges. However, many returnees are reluctant to seek professional help 1. In 2010, only 55% of Australian Department of Veterans Affairs (DVA) veteran clients reporting a mental health or wellbeing problem had sought professional support 2. The unmet need for mental health care is a burden on the community with family, social and economic costs. Engaging in early intervention could assist the adjustment of returning military personnel to post-deployment life and help prevent the development of chronic Problems.
In recent years, health services internationally have faced the challenge of adapting to serve the needs of a new generation of military returnees with different preferences to earlier veterans, and reluctance to engage in traditional face-to-face health services 3. Among military personnel, stigma has been identified as one of the main barriers to accessing mental health care 1,4. There is a prevailing attitude amongst many military personnel that getting help for a psychological or behavioural problem would be perceived as a sign of personal weakness or that their careers could be adversely affected 5,6.
On-line self-help tools tailored to the military experience may be particularly powerful in their reach because of the privacy benefits, as well as convenience and flexibility to fit around other commitments. Social trends internationally suggest an increasing appetite for personal health control, and preference for the internet as a source of health information7. In Australia, younger veterans (up to 55 years of age) are known to be the age group most inclined to use the web to find health information 2.
On-line self-help tools, particularly those with a cognitive or problem-solving approach can be very effective interventions 8. There is mounting evidence that online tools can be as effective as face-to-face psychotherapy for some disorders, especially depression and anxiety 9 10. There is preliminary evidence for the effectiveness of online support in reducing posttraumatic stress symptoms in returning veterans 11. Some online resources are designed to be used in a stand alone self-help context, whereas others, and those most prominent in the effectiveness literature, provide guided or supported self-help. Many resources studied incorporate elements of therapist contact such as email support, telephone calls or face-to-face counselling. A recent review found that stand alone self-help interventions were effective for motivated clients with sub-threshold mood disorders, while therapist assistance was more important to treatment effectiveness when dealing with clinical levels of symptomatology 12. The current study targeted sub threshold problems which commonly occur following a traumatic experience, using a self-help website designed for stand alone use by members of the military community. Although the importance of early intervention in mental health is well established, few of the existing self-help websites focus on early intervention and this was, therefore, a novel approach.
The Wellbeing Toolbox (www.wellbeingtoolbox.net.au) was developed by the Australian Centre for Posttraumatic Mental Health on behalf of the Australian DVA. The Wellbeing Toolbox consists of six topics (or modules), a self-management plan and an optional self-assessment questionnaire. The six skill building modules are Problem solving, Building support, Helpful thinking, Getting active, Keeping calm and Sleeping better. These were adapted from an established program ‘Skills for Psychological Recovery’ (SPR)13. SPR, originally designed as an early intervention for the post-disaster context, has an established theoretical background, and face validity for mental health providers 14. The Wellbeing Toolbox is pitched at the level of generic psychosocial skills for early intervention for challenges arising from transitioning and other life issues, and to prevent difficulties escalating into severe mental health problems.
This paper presents the results of a two-stage evaluation of the Wellbeing Toolbox. First, we evaluated data from the naturalistic users who came to the site in the first 12 months after its launch, to examine the extent to which the site was accessible and acceptable to the target audience. We used Program Logic (i.e. a ‘road map’ that presents the theory behind and expected outcome of a program’s actions) to map the intended outcomes of the website, to identify research questions and align this with data which could feasibly be obtained by manipulating aspects of the website (e.g. building in analytics and page tracking, incorporating questionnaires and a data extraction portal). Methodological challenges included how to obtain rigorous evidence whilst retaining user privacy which is part of the appeal for online self-help audiences.
In addition to satisfaction with the website, the extent to which users benefit from their interaction with the site remained an important unanswered question. Whilst the availability of on-line resources is a positive first step toward improving health in the target population, further evidence was required to determine whether website use leads to measurable improvements in wellbeing. This led to a second study testing the hypotheses that site users would achieve improvements in individual wellbeing goals (primary outcome) and mental health status (secondary outcome) after 3 months of use.
Study 1: Accessibility and acceptability of the Wellbeing Toolbox
This study aimed to determine accessibility and acceptability of the Wellbeing Toolbox for the target audience of current and former military personnel in Australia.
Materials and Methods
Participants
Initial satisfaction data was obtained via an on-line survey completed by 291 naturalistic, open access users of the site. From this group, 15 randomly selected and consenting veteran users participated in telephone interviews.
Measures
An online satisfaction survey asked basic demographic information and four structured satisfaction questions relating to ease of use, helpfulness, whether the user would return if they needed further help and whether they would recommend the site to others. Site usage information was extracted via Google Analytics15. A structured 30 minute interview enquired about the way the site was used (e.g. frequency and duration of visits, self-reported impact and outcomes of using the website), and the extent to which this self-help sample used other face-to-face health services.
Procedure
The satisfaction survey was posted on the Wellbeing Toolbox website for the first year of operation. During this period the extent of site usage was measured on a monthly basis by data extracted from Google Analytics. Users willing to be contacted for telephone interview registered their contact details online. Fifteen of these users were randomly selected using a computer generated random number list and were contacted to provide more detail about their experience of using the website in a structured interview. Interviewers used a checklist to record responses. Open ended explanatory comments and examples were transcribed verbatim. Interview record sheets were categorically analysed by one researcher and verified by a second.
Results
During the first year of use the Wellbeing Toolbox received 7477 unique visitors. Visitors who completed the online survey (n=291) were predominantly male veterans with a mean age of 60 years (range 28-88). A breakdown of participants by age category revealed that the age profile of the sample was similar to that of DVA clients overall: approximately 60% between 60 and 69 years of age; 20% between 50 and 59; 10% between 40 and 49; 6% between 30 and 39; and just over 1% under 29 years. Among the 291 users who completed the online satisfaction questionnaire, 68% found the site easy to use, 52% found the site helpful, 70% said that they would return to the site if they needed help in the future and 61% intended to recommend it to others. Among the 15 randomly selected veteran users who were interviewed, the majority (80%) reported that they were benefiting in meaningful ways and could give an example of the way site usage had improved an aspect of their wellbeing or their self-management of wellbeing issues. Just over half of the interviewees (53%) were not currently connected to mental health services and one third (33%) had never received help from a psychologist, social worker or counsellor in the past. As such, the site had reached veterans who were not currently using other mental health services and some who had never used DVA mental health services to which they were entitled.
Study 2: Effectiveness of the Wellbeing Toolbox
The aim of Study 2 was to examine the effectiveness of the Wellbeing Toolbox as a means of improving individual wellbeing and mental health issues for current and former military personnel. The funders and host of the Wellbeing Toolbox, DVA, had a particular interest in whether the site was effective for current and former serving military personnel up to the age of 55. As such, the effectiveness trial targeted this age group. We hypothesized that users who set specific wellbeing goals would achieve improvements in these areas after 3 months of use. It was hypothesized that the use of the Wellbeing Toolbox would also lead to a significant improvement in mental health outcomes on standard mental health measures post-intervention.
Methods
Participants
Thirty participants were recruited to participate in this study. Participants were required to be Australian serving or exserving members or their families, have access to a computer, be interested in improving aspects of their wellbeing, and not to be currently engaged in active psychological treatment. Although the site is intended for subclinical problems, participants were not excluded on the basis of mental health symptom severity. Demographics of the trial participants are shown in Table 1.
[ Table 1 available in download]
Measures
Prior to using the website, participants took part in a telephone interview to discuss and set individual wellbeing goals. The interviewer used a structured interview guide to check if each wellbeing domain was relevant to the participant, and if so elicited and clarified a goal the participant hoped to achieve within three months. Post-intervention, participants reviewed and rated the extent to which their goals had been achieved and attained over the three month period of the trial, and provided general feedback on their experience of using the website.
Participants also completed a self-report booklet before using the website and post-intervention. Self-report instruments measured psychological distress, alcohol use, quality of life, and depression, anxiety and stress.
Goal achievement
Achievement of each individual goal was rated on a 5-point scale (1 = ‘not achieved at all’, 5 = ‘totally achieved’), developed specifically for this trial.
Psychological distress
The Kessler 10 16 is a widely used brief screening tool for psychological distress. It is already available on-line within Wellbeing Toolbox site as an optional questionnaire. The cut-offs are 10-15 mild distress; 16-29 moderate; 30 or over severe.
Depression, anxiety and stress
The Depression Anxiety and Stress Scale (DASS 21)17is a 21 item screening tool, it has three sub-scales for depression, anxiety and stress, and is widely used in research and clinical practice. It has good psychometric properties and is routinely used by community veteran mental health clinics for clients presenting to counselling in Australia.
Alcohol use
The Alcohol Use Disorders Identification Test (AUDIT)18 is a 10-item scale developed by the World Health Organisation as a screening instrument for hazardous and harmful alcohol consumption. The scale has demonstrated strong internal reliability (0.86). This is the gold standard alcohol measure used in many veteran programs and research.
Quality of life
The WHOQOL-Brèf (the short form of the World Health Organization’s Quality of Life instrument) is an internationally recognized instrument 19,20, which has been validated for Australian use 21. The WHOQOL-Brèf comprises 26 questions on four scales measuring physical health, psychological wellbeing, social relationships and the environment. Higher scores indicate higher quality of life.
Procedure
The research was approved by the Department of Veterans’ Affairs Human Research Ethics Committee. Advertisements for trial participation were circulated via advocacy and ex-service support organisations, and on social media via the Australian Department of Veterans’ Affairs Facebook and Twitter accounts. Participants expressed an interest online, then after receiving study information, those eligible for inclusion in the trial provided informed consent and were admitted to the trial by the interviewer. The initial assessment included a structured interview and completion of self-report measures. Participants, together with the interviewer, developed individual wellbeing goals, which could feasibly be achieved in three months. After the assessment, participants were directed to the Wellbeing Toolbox website (www.wellbeingtoolbox.net.au) and encouraged to use it over the coming 3 months to improve those aspects of their wellbeing using the interactive modules and self-management plan within the site. They were given a log sheet and asked to record the amount of time spent on the site for each visit.
Data analysis
This study utilized a mixed methods approach and a pre- post design. Group means for goal attainment were calculated. Paired t-tests were used to examine whether participant’s scores on mental health measures were significantly different over time. Qualitative data including verbatim comments and examples were thematically categorised by one researcher and verified by a second.
Results
We hypothesized that those site users who set specific wellbeing goals would achieve improvements in these areas after 3 months of use. In addition, we examined whether use of the website led to significant improvement in mental health outcomes during this period.
Achievement of wellbeing goals
All trial participants identified areas of their wellbeing that they would like to improve. Examples of wellbeing goals associated with each of the six wellbeing modules are shown in Table 2. At 3 months when the goals were reviewed, participants rated to what extent their goals had been achieved. Over 75% of participants’ goals were achieved in all topic areas. Participants were asked to record how long they spent on the website and to rate the extent to which their use of the website had contributed to achieving their goals. Unfortunately, the site usage logs were rarely used by participants, leaving them to rely on memory. Participants tended to give vague or uncertain estimations of the time spent on site, and to rate lowly the extent to which the website had contributed to their goal attainment. These areas were followed up in the final interview. Participants reported that the process of setting the goals together with the interviewer had been a very helpful part of their experience but that their internet usage was variable and hard to remember to what extent they had used which modules.
[ Table 2 available in download]
Mental health outcomes
As a secondary outcome measure, we examined whether use of the website led to significant improvement in mental health outcomes after three months of use. Despite the achievement of personal wellbeing goals reported above, widespread improvements were not found in response to standard mental health outcome measures. Results are presented in Table 3.
[ Table 3 available in download]
As shown in Table 3, participants were experiencing significant mental health problems at initial assessment. Mean psychological distress was in the high range M=29.40 (8.07), means from the depression anxiety and stress subscales from the DASS were all in the severe range. Twenty percent of the participants scored positively for alcohol abuse or dependence.
There was a small non-significant reduction in K10 between baseline and follow-up assessment, a small non-significant reduction in DASS subscale scores for depression, anxiety and stress. No domains of quality of life shifted significantly between baseline and the post-intervention assessment. It was interesting to note that the only significant change in mental health outcomes was a reduction in alcohol use, which was not directly targeted in the Wellbeing Toolbox (t(13)=2.31, p<.05) . This may have been a secondary benefit of achieving well-being goals but this was not explored in the current investigation.
Discussion
The Wellbeing Toolbox evaluation consisted of two studies. The first examined the accessibility, acceptability and reach of the website. During its first year of use, the Wellbeing Toolbox website was accessed by a significant number of Australian veterans who were generally satisfied with their experience on the site. The second examined the effectiveness of using the site to achieve wellbeing goals and improvement in mental health. The effectiveness trial succeeded in recruiting 30 ex-serving members who sought wellbeing improvements and were willing and interested in participating in an online intervention. Individual wellbeing goals were achieved during three months of site use, but these improvements were not accompanied by reductions in scores on mental health instruments.
Since the site did not aim to treat mental health conditions, the lack of significant results on standard outcome measures is perhaps unsurprising. Meta-analysis and review findings suggest that for clinical levels of disorder, the best treatment outcomes from online interventions require some form of therapist contact 12,22. However the data trended toward improvement across a range of measures and there was a significant reduction in alcohol use. This points to the potential of ‘wellbeing’ oriented content providing at least a foundation level of benefit which could be augmented by face-to-face or tele-therapy psychological treatment specifically targeting mental health problems. It was somewhat concerning in this study that participants reported severe mental health problems, given that none were currently receiving other mental health care.
On a related point, few participants were able to set specific, measurable, achievable, realistic and time-related goals without interviewer assistance. Participants struggled to identify wellbeing improvements which could reasonably be achieved in 3 months. Once set, goals were generally achieved independently of further input. This points to opportunities either to strengthen goal setting ability of self-guided site users (some means of helping them to refine their own goals) or to engage therapist support in the goal setting process.
This study points to the potential value of resources like the Wellbeing Toolbox as an appealing entry point or adjunct to more intensive face-to-face treatment. In practice this means digital resources having features such as push button access to local mental health services, capabilities to send assessment results to a provider, or request a call back from a supporting clinician. In addition, there is room to develop clinicians’ awareness of and engagement with digital resources 23 to the extent that they can confidently direct appropriate clients toward digital resources (websites or phone or tablet applications), and support their use of these tools whilst actively referring more severely affected people (or those for whom self-guided strategies have failed) toward more involved care. These features should be a higher priority in future digital resource design, and in practitioner training and professional development 24.
The study has several limitations.
1. The trial was designed as a pilot to indicate if a larger RCT was warranted, and as such was conducted with relatively small numbers and did not involve a comparison treatment arm such as face to face Skills for Psychological Recovery. These factors inevitably limit the interpretation of the findings.
2. It would have been useful to have more reliable information regarding site usage, and to relate the amount of time spent on site, pages viewed and activities undertaken with achievement of goals, but this data was not available as participants tended not to complete the site usage logs. Future study designs may consider, with participants’ consent, the use of automated usage tracking.
3. Severity of mental health problems, with the exception of current risk to self or others, was not used as an exclusion criterion because we were interested in how effective the Wellbeing Toolbox would be for participants who were attracted to on-line self-help. However we did not anticipate that almost all participants would be experiencing severe mental health problems, limiting our capacity to assess its effectiveness for sub-clinical problems.
Despite these limitations, this study illustrates the value of online resources in reaching populations in need, their acceptability to people who have not sought help in other forms, and their effectiveness in leading to meaningful improvements in wellbeing of the ex-serving community. For those with more severe mental health concerns, online resources should be part of a suite of support options that also includes face to face treatment.
