Validitet og reliabilitet

Alle udgaver af Outcomes Star er inden udgivelse blevet testet i workshops og gennem pilotprojekter med frontlinjemedarbejdere og borgere i organisationer, som arbejder med den pågældende målgruppe. Tilknyttet hver stjerne er en organisationsguide, hvori pågældende stjernes pilotprojekt vil være beskrevet.

Herunder findes et kort resume af en undersøgelse af stjernernes validitet og reliabilitet. Den fulde undersøgelse, som indeholder yderligere information, referencer og forskellige videnskabsteoretiske perspektiver på Outcomes Star, kan findes på engelsk her.

Klik her for at hente den fulde rapport.

1) Usability and face validity

This aspect of validity covers questions such as:

Does the tool measure those things that are most important and relevant to the service and service users?
Do the descriptions within the tool resonate with service user and worker experiences of change?
Does the tool support the consistency and effectiveness of keywork /casework and help service users to make progress?
Is the data helpful in assessing the effectiveness of the service, identifying service strengths and weaknesses as a basis for on-going improvement?

A study by Burns, MacKeith and Graham (2008) looked at 25 organisations that had been using the homelessness version of the Outcomes Star for two years or more and focussed on the interpretative (or practical) aspects of validity. This study found that:

All 25 organisations reported that key-work had improved as a result of using the Outcomes Star.
Key-work had become more focussed on service user change, more systematic and consistent and covered a wider range of issues than before the Outcomes Star was used.
Those organisations that analysed the data that they collected found that the data was helpful in assessing the effectiveness of the service and identifying areas for improvement.

Dr Helen Killaspy and colleagues at the University College London in the UK have recently completed a study to investigate the validity and reliability of the Mental Health Recovery Star. The study looks at the length of time the tool took to complete, how easy it was to decide on a score, staff and service user views of its usefulness in care planning and as a clinical outcome measure. The findings have been submitted for publication but the article is not yet in the public domain.

2) Reliability

This aspect covers whether the Stars provide consistent reliable data, in other words, are staff consistent in where they place someone on the different ladders. This includes the questions:

Is an individual member of staff consistent over time? (test:re-test reliability)
Do different members of staff use the ladders in the same way? (inter-rater reliability)

As well as looking at staff and service user experiences of using the Recovery Star, the study carried out by the University College London also looked at the test:re-test reliability, inter-rater reliability. However the results of this study are not yet in the public domain.

3) Other aspects of validity

There are a number of other aspects of validity that can be investigated, including looking at:

Whether items on the tool measured a single underlying construct relating to recovery (internal consistency) and the nature of any underlying factors (factor validity);
Whether the tool measures efficiently (item redundancy)
Whether the tool detects reported change over time (responsiveness)
Whether results on the tool correlate with results on other tools which have previously been validated (convergent validity)

Geoff Dickens, Head of Nursing Research & Research Manager, St Andrew’s Academic Centre, King’s College London, Institute of Psychiatry and colleagues carried out a study looking 203 adults who had completed the Recovery Star two or three times (Dickens et al (2012)). They found that the tool had very good internal consistency (Cronbach’s α greater than 0.8), little obvious item redundancy, and most item scores moved in a positive direction over time indicating good responsiveness. They conclude that "Further research on the psychometric properties of Recovery Star is clearly warranted, but this study provides good preliminary evidence for its adoption in recovery-focused mental health services."

In the development process for the Alcohol Star the pilot service providers used both the Outcomes Star and other more established measures of the severity of the service user’s drinking problem along with a simple measure of alcohol usage. The findings were that of all of the measures, change on the Alcohol Star correlated most strongly with reduction in Alcohol usage. This finding indicates strong convergent reliability of this tool with measures of alcohol use.

We are keen to carry out further validity and reliability testing on our existing Outcomes Stars. If you have links with a university or research body that is interested in working with us on this then please contact joy(at)triangleconsulting.co.uk