‘Measuring positive changes in a psychologically informed environment — Is a successful outcome only ‘A home and a job’?

29.04.2022: After a wonderful restful few weeks of annual leave over the Easter holidays, I am sitting down to write this PIE blog as I commence my third year of working as the lead for Psychologically Informed Environments (PIE), at the national youth homeless charity — Centrepoint. This is therefore a natural point in which to pause and reflect on the PIE journey so far, but more importantly, considering how we are continuing to ‘evaluate’ this moving forward. One of the key ‘ingredients’ of a PIE is the ‘evaluation of outcomes’ (Keats et al, 2012 — PIE Good Practice Guide: https://eprints.soton.ac.uk/340022/1/Good%2520practice%2520guide%2520-%2520%2520Psychologically%2520informed%2520services%2520for%2520homeless%2520people%2520.pdf).

Specifically, the PIE Good Practice Guide notes that ‘evaluation of outcomes is crucial in the development of psychologically informed services. Evaluations are crucial because they are a cornerstone of reflective practice, which in turn is a cornerstone of psychologically informed environments’ (p.26). Put another way, ‘if you do not know what impact what you do or say is having, how can you know whether it is positive and how can you improve it?’ (p.26). As Keats et al (2012) further argue ‘there is not a great mystique about evaluation: it is the opportunity to know which things you do or say are effective, in what situations and with whom’ (p.26). Fortunately, as psychologists are scientists or evidence based practitioners, research is a cornerstone of our professional training and part of our ongoing psychology work in practice alongside any more clinical or support work.

Moreover, the PIE Implementation and Assessment Guide (Westminster Council, 2015) notes that evaluation is a critical part of PIE, which allows the development of evidence generating practice in this sector. This argues that evaluation can be used to understand and verify ‘what works’, support continuous learning and improvement, evidence progress to key stakeholders (e.g. service users, staff, commissioners etc.) and demonstrate the impact of PIE’s on improving the services for those experiencing homelessness. It can also contribute to a wider understanding of the psychological factors that underpin homelessness, in order to determine what interventions are required to #EndYouthHomelessness in the future.

As a result, throughout our Centrepoint journey we have been evaluating our PIE training with pre and post training questionnaire measures looking at staff PIE skills and knowledge, attitudes, satisfaction and applicability to working role. We have also evaluated our reflective practice offer, which has been written up and submitted for academic peer review publication (Tussis & Miles, 2022; in review) and we are collecting ongoing visual records of the changes to the environments in our supported accommodation services via the PIE Physical Environment Fund. We are also currently considering a wider evaluation of PIE later this year, which we already have baseline data for, but our follow-up phase of data collection has been delayed by the onset of the COVID-19 pandemic. For more details about some of our PIE research projects, please see previous blog here: https://drhelenmiles.medium.com/not-just-numbers-why-data-and-research-matters-in-a-psychologically-informed-environment-db7da280ecd2.

However, research on PIE’s in the homeless sector to date still remains somewhat limited. Partly this is because this is a relatively new area and it can take time to set up a PIE within an organisation before it is then possible to evaluate it effectiveness. The psychology of homelessness has also been a somewhat neglected area in the past decades relative to housing and social theories (Maguire & Richie, 2015). However, there has been some research from those who have been working in this field since the early days of the approach. For example, a research study by Stronge & Williamson (2014) in homeless services provided by South London & Maudsley NHS Trust (SLAM) found over 80% of hostel residents in South London engaged in psychology interventions and that support staff found reflective practice and PIE training sessions very helpful.

Further research studies within St. Mungo’s adult homelessness services, such as Cockersell (2011), found that the additional of psychotherapeutic approaches to service delivery resulted in improvements in attendance rates to the service and motivation for change (e.g. participants were three times more likely to move from pre-contemplation to action on the Outcome Star). There were also improvements in Education, Employment and Training (EET) outcomes (e.g. participants were twice as likely to be in employment, voluntary placements, education or training). Moreover, Phipps et al (2017) also conducted a qualitative study on staff and residents’ experiences of PIE within St. Mungo’s. This research found that a PIE approach helped create a ‘home’, increased awareness of residents’ mental health needs, helped build and manage relationships between staff and residents, and that reflective practice sessions were valued as helpful by staff in order to modify their practice to lead to better outcomes for residents.

So evaluations of a PIE should be simple right? Well annoyingly actually not always. Psychologists are all too aware of the challenges of measuring outcomes that relate to ‘people’. Unlike the natural sciences such as chemistry or biology, our substances of measurement — people — are complex, unique and often changing. If you put two chemicals together, you know what you will get! However, if you put two people together there may be many conscious and unconscious factors that affect the dynamic or outcome. This is of course what makes psychology — ‘The scientific study of the human mind and behaviour’ so interesting — if a bit frustrating at times! As a result, this week I have been reflecting on the complexity of ‘whole system’ evaluations of PIE and how many factors can influence what may be considered a PIE outcome. Take the example of ‘evictions’ — this might seem a clear objective measure of how successful a PIE might be? The aim of course in this case would be for a PIE approach to support staff to work more effectively with challenging behaviours displayed by homeless young people, utilising a person centred ‘elastic tolerance’ approach, and thereby reducing the need for warnings and evictions. However, how this maybe internally applied in one service setting by one staff members can vary from another. Moreover, external factors can influence outcomes. For example, during COVID-19 there was a UK wide ban on evictions, so any data from this period is likely to be biased and thus needs to be interpreted within this wider context.

To manage this complexity, some psychologists have evaluated particular aspects or ‘ingredients of a PIE separately (as outlined above in our approach to evaluating our Centrepoint PIE so far). Other psychologists have also researched or evaluated specific PIE based interventions. For example, Buckley et al (2020) looked at the specific impact of psychological case formulation in reflective practice sessions. They found that staff that engaged in these sessions perceived this approach to increase their understanding of homeless service users and encouraged staff to take a different (and more positive) approach to their work, although they found that the usefulness of the approach was restricted by the systemic limitations within the wider organisation. Another recent case study based research project by Schwaiger & Williamson (2021) evaluated a PIE informed mentalization-based art initiative with homeless people. They observed positive benefits from the programme to include engagement, tenancy sustainment, social inclusion, and reduced substance use.

Consequently, when considering how we can evaluate whether a PIE approach is effective, the Good Practice Guide (Keats et al, 2012) note that there need to be a range of measures that may need to be creatively employed to capture all the outcomes. This can be a mix of quantitative (data) and qualitative methodologies, but also need to operate at multiple levels including policy level (e.g. what required by local commissioners), service level and individual outcomes. This importance of not accepting that a ‘one size fits all’ approach to outcomes has been highlighted to me on several occasions over the past week during reflective practice sessions across the organisation.

Firstly, it has been ‘appraisal season’ within the organisation, a period each year where staff stop and reflect on what they have achieved over the past year. What has been interesting to talk with staff about is how much they perhaps haven’t realised just how much they have achieved, and how much their approach to working with young people may have changed or been modified by their attendance at reflective practice sessions or PIE training sessions over the past year. Capturing this anecdotal feedback and reflection as clear evidence for the success of our Centrepoint PIE approach is difficult. However, I have seen examples where staff’s approach to working with a young person has changed following an exploration for the underlying reasons for their challenging behaviour and/or staff’s self-reported stress and risk of burnout or attrition has reduced (e.g. as they may be taking a situation less personally and they are engaging in regular self-care).

Secondly, I have also been thinking more about our outcomes for young people following some feedback from a team I have been supporting in reflective practice over the past 18 months. This team were supporting a homeless young person with an extremely high level of suicidal ideation consequent of significant early traumatic experiences. Their mental health issues had resulted in them making several attempts on their own life in the past year, which have only been thwarted by direct staff intervention. This homeless young person has dominated our reflective practice sessions, which we have used to formulate and understand their issues, develop action and safety plans, and plan how to advocate for them to move to more appropriate supported accommodation that can provide specialist treatment for them.

Due to the consistent hard work of the amazing staff team in this service, this young person’s mental state has been stabilised, they have been supported to maintain their tenancy and most importantly, they have been prevented from taking their own life and are now motivated to engage in mental health treatment. As they continue their recovery journey within suitable mental health services and leave us in Centrepoint, arguably we have not achieved our organisational outcome of ‘a home and a job’ for them. However, this was never going to be a realistic outcome for them at this stage in their life. Rather, the fact that they have moved on from our services to access the care they need is arguably a successful outcome for this individual. Perhaps not one that would be captured by our Key Performance Indicators (KPIs) of a tenancy or being in EET but arguably a successful outcome nevertheless.

Of course, we should always aim to support every homeless young person that comes to Centrepoint to achieve ‘a home and a job’, thereby reducing their risk of homelessness in the future. However, due to their often-significant traumatic early experiences or even their individual circumstances, a successful outcome might not be this straightforward. Another example of this may be within our young parents’ services, where perhaps an appropriate outcome for a mother with a baby is not a job (just yet) but instead that they have improved their parenting skills and are therefore not at risk of having their child taken into the care of local social services. I would argue that this outcome might be significant for the individual homeless young person, and reflect ‘distance travelled’ rather than end destination.

Taken together, all of this means that as we enter our third year of our PIE journey at Centrepoint, we need to remain flexible in how we evaluate whether it is ‘working’ and consider ensure our outcomes goals are individual as well as organisational. We need to consider both objective ‘hard’ outcome measures (e.g. staff training / reflective practice attendance, staff attrition, incidents, evictions, progress towards move on or EET) alongside ‘soft’ outcome measures (e.g. a wide variety of individual outcomes for homeless young people that reflect positive change, staff and young people feedback, wider cultural change). We also might need to be creative and flexible in how we look for evidence of success. As Keats et al (2012) highlight ‘evaluation and monitoring [in a PIE], particularly at the individual level can also enable staff and clients to see progress that is potentially significant but might otherwise be hidden’ (p.27). As the picture at the beginning of this blog illustrates, measuring outcomes in psychology can be ‘messy’ or challenging because we are dealing with ‘people’. Therefore, we need to remain alert to notice all the positive impacts of a psychologically informed approach in homelessness whilst continuing to appreciate the complexity of evidencing this moving forward…