The Reframing of Methodology: Revisiting a PhD Study

Introduction

1.1 This paper, which draws on my PhD study, explores some methodological dilemmas by focusing on issues associated with the execution of qualitative research when framed within a positivist research design. Importantly it suggests that revisiting the methodology of previous studies may lead to re-evaluation and deeper understanding of methodological promises and their possibilities. I acknowledge how revisiting, in research terms, usually entails the return, by an original researcher, to a previous research site to see primarily what has happened, or to explore new issues or ideas. It does not usually entail the reformulation or re-identification of methodological perspectives in respect of an original study.

1.2 Discussion around revisiting is found in debates about open access, data archiving and data re-analysis more generally. In terms of qualitative studies, the revisiting debate has highlighted issues relating to epistemological, moral, legal and practical concerns (Parry & Mauthner 2004). It is out-with the scope of the paper to document these debates, but suffice to note they have drawn much interest from proponents as well as 'revisiting sceptics' among the qualitative research community (see for exampleBishop 2005; Broom et al. 2009; Corti 2000; Fielding 2004; Heaton 2004; Thompson 2000). Central to these debates is the assertion that, in a revisiting situation, new conditions or contexts apply (even where the researcher/analyst has not) and this will inevitably have implications for the production of knowledge (Mauthner et al. 1998). Notwithstanding this, it is generally not contentious that revisiting may offer researchers methodological insights (Mauthner et al. 1998), and it is in this spirit that the methodological revisit and re-evaluation, described here, was undertaken.

The PhD

2.1 The PhD, which comprised a qualitative study of young people in the UK secure estate, who were participating in the Duke of Edinburgh's Award (DofE), was linked to a funded evaluation of the DofE delivered to young people in custody. Whereas the funded study used a quantitative method, which involved implementation of an attitudes to offending behaviour (ATOB) instrument pre and post DofE programme implementation, the PhD comprised a qualitative study drawing on the perceptions of young people in custody. The PhD study, while separate from the funded study, used focus groups and in-depth interviews to explore perceptions of intervention participants, pre and post intervention, alongside implementation of the ATOB instrument at both study phases. The PhD focus differed from the funded study objectives in that its interests were far broader than the intervention and it included a subsequent data collection phase which involved visits to Youth Offending Teams and interviews and focus groups with a purposive sample of their members.

2.2 There were a number advantages afforded by the link between the two studies. Not least of these was that it allowed me access to the secure estate, and young people residing therein. The DofE had a generally good relationship with the secure estate, and the majority of those institutions approached agreed for the study to proceed. Moreover, my relationship with DofE, and the funded element of the study, assisted approval for the PhD study from the Youth Justice Board, who were key gatekeepers regarding my access both to the secure estate and to Youth Offending Teams. The funders were happy because I was able to provide more process evaluation data (which would have not been possible within the original project budget) and they got to associate their relatively short term study with a PhD, which yielded a wealth of in-depth data on interventions in the secure estate, and the community, in the long term. Notwithstanding these advantages, the association of the PhD with a funded study was also the source of some dilemmas, most notably of a methodological nature although inevitably there were also practical implications. Before outlining these it is necessary to set out the (funded and PhD) study designs in a little more detail.

Study Design

3.1 The funded study employed a pre and post intervention design, whereby secure estate establishments were visited at two separate research phases, before and following implementation of an intervention for young incarcerated people (which comprised delivery of DofE components). The sample, which comprised seven secure estate establishments in England and Wales delivering the DofE, was selected to include a range of different types of secure estate establishments. It comprised of: Secure Children's Homes, Secure Training Centres and Young Offenders' Institutions (including those funded and not funded by the Youth Justice Board). The sample establishments together covered an age range of young people between 12 and 21 years. Establishments were also sampled to represent geographical areas and length of experience in running the programme. Of the original eight establishments approached, three declined to participate and another three were recruited to replace them. Of these, access to two was successfully negotiated, but access negotiations with the third (although agreement in principle was achieved) were not successful. Of the seven eventually recruited, at one establishment access to young people, whilst originally agreed, was subsequently not permitted, although the researchers were granted permission to interview staff who delivered the programme.

Crime-Pics

4.1 At each phase of the funded study (pre and post intervention), an 'attitudes to offending behaviour' (ATOB) instrument 'CRIME-PICS ll' was administered to a sample of intervention participants. This instrument is designed to capture changes in attitudes towards offending. CRIME-PICS II is a 35-item structured questionnaire designed by Michael and Associates in 1994 as a successor to their previous tested CRIME-PICS instrument, which was developed with the support of the Probation Service. The questionnaire is designed to measure attitudes to offending including offenders' attitudes and perceptions of their problems, both at the outset of the order and again towards the end, in order to assess whether there has been any change. To this end it has been used to measure improvements in attitudes to crime (Harper & Chitty 2005). It consists of two components. First, twenty items that measure the extent to which attitudes are pro-criminal, i.e. supportive of crime; Second, a five item problem inventory that measures the number and extent of self-perceived problems (Rex et al. 2004).

4.2 The scales included in the questionnaire are: general attitude, anticipation of future offending, victim empathy, evaluation of crime as worthwhile and problem inventory. The instrument, which takes only a few minutes to administer and score, has been adopted by both the Probation and Prison services in England and Wales as a standard measure for the impact of interventions with offenders. It has also been used routinely in evaluations of rehabilitation projects in the private and voluntary sectors and comprised one of a raft of measures used by the National Probation Directorate (NPD) to evaluate the impact of nationally accredited general offending behaviour programmes (Feasey et al. 2005).

4.3 In the past, the effectiveness of interventions with offenders has been assessed through either simple activity measures such as compliance levels, or by measuring outcomes such as reconviction rates. The first of these measures, however, tell nothing about the impact of the intervention, while the second comprises a blunt instrument which involves an appreciable time lapse before outcomes can be evaluated (Frude et al. 1998). As CRIME-PICS II focuses on attitude change, it potentially offered the study an evaluative measure of any changes in offenders' thinking about crime, and victims of crime, occurring between two points in time (pre and post intervention) (Miers et al. 2001). Reconviction rates can be statistically correlated with questionnaire scores so that a project's likely impact on reconviction can be assessed. This potentially makes the instrument a useful measure relating to reconviction (Raynor 1998). The instrument can also be used diagnostically to monitor the progress of offenders. As it is easily understood by offenders, the results can be shared and discussed with them (Frude et al. 1998). For all these reasons, and the perceived association with what was understood as 'scientific' method, the funders selected our research proposal for funding.

4.4 To reiterate, the ATOB instrument was administered to young people aged between 14 and 21 years residing in six young offenders' establishments at baseline (phase one) and after 6 months, (phase two of the study). At baseline, respondents had minimal, if any, experience of the DofE, whereas at phase two of the study all respondents had participated in the programme. It was anticipated that as young people gained experience of the DofE and became embedded in the programme, a difference in scores would be apparent between the two phases of the study.

4.5 The ATOB instrument was completed by young people after taking part in focus groups typically involving participation of eight to twelve young people, a member of prison staff, myself and a DofE worker. The focus groups which formed part of data collection for the PhD study explored gaps in current understanding about the implementation of interventions in the secure estate through the examination of how young people in the secure estate responded to the DofE intervention, how different aspects of the programme and its delivery were perceived by participants, perceived facilitators and barriers to delivery, how the programme sat in the wider contact of youth justice provision and the implications of the programme for perceived future opportunities of young people upon release from custody.

4.6 Focus group discussion at phase one explored young people's perceptions of the DofE and their reasons for, and anticipations of, involvement in the intervention prior to (or in the early stages of) delivery of DofE activities. At phase two, discussion focussed on young people's receipt of intervention activities in which they had participated during the preceding six months. During these groups, discussion focussed on participant perceptions of (and barriers to and facilitators of) learning associated with the intervention, and their anticipations for the future. Focus groups were audio recorded at five institutions with participants' agreement. At the sixth institution, where audio recording equipment was not allowed, detailed notes were taken during the focus group.

Design problems

5.1 Several practical problems were attendant on the study design. Some of these had been anticipated because they are routinely encountered by those carrying out research on those in custody. That the implementation of interventions in prisons is problematic partly explains why the evidence base concerning the efficacy of interventions targeting young offenders in custodial and community settings is thin (Douglas & Plugge 2006; Moony et al. 2007). Evaluation of custodial interventions is difficult because studies often focus on change in the short term and are unable to capture long term effects. Linked to this, outcome measures against which the success of programmes are measured may not capture adequately unintended benefits (Nichols & Crow 2004; Hunter & Boyce 2009). Moreover, tackling re-offending is a long-term process, and participation in programmes whilst in custody form only part of the rehabilitation process (Harper & Chitty 2005). While similar issues affect much evaluation design, research in prisons has its own unique problems, largely stemming from issues around custody and risk.

5.2 These issues, which affected both implementation and evaluation of the DofE programme, had implications for both the funded and PhD study. Participant selection to the programme, and the evaluation, was dependent on their designated risk status. Several risks were attendant on young people's selection to the DoE programme. First, was the risk associated with reputation of the institution if, for example, young people involved in the programme should abscond while engaged in DofE activities, some of which take place outside of the prison premises. It has been evidenced how the media tend to focus more upon youth deviance rather than on their accomplishments, who may collude in and exacerbate the 'demonisation' of young people (Scraton 2009). The data indicated awareness of, and concern about, media interest in activities of young offenders, among secure estate staff, and this issue is expanded in a previous publication (Dubberley 2010). Second, was the (physical) risk to staff (from young people themselves) particularly when engaged in activities outside of the prison, and any potential risks to the public. Third was the risk posed by young people to themselves (and their future prospects) should they attempt to abscond during DofE activities. Fourth was the risk that young people pose to each other inside the secure estate. Those implementing the programme said that in the case of interventions which draw a range of participants across the institution (who are accommodated in separate areas of the prison) the risks are heightened. Over and above this, the designated level of risk associated with those ineligible for release on temporary license (ROTL) in many, but not all, cases prevented them from being selected to the DofE programme in the first place (Hagell & Newburn 1994). Hence, young people's designated risk status determined their eligibility for intervention participation.

5.3 In addition, usage of the offending behaviour questionnaires in this context was particularly problematic, Although CRIME-PICS II has been used successfully with young offenders it has critics (CRG 2003). The main problem arises from its use as a before and after measure, requiring two sets of measurements which will be some considerable distance apart in terms of both time and setting. Often, implementation of follow-up questionnaires can be difficult because young people may have moved on and lost contact with workers (CRG 2003). Such loss of data impacts on the conclusions that can be drawn from the findings. Furthermore, in situations where only some young people complete the test on both occasions, there may be differences, between them and others who did not repeat the test. Thus, the findings may not be a result of the intervention (Rex et al. 2004).

5.4 Implementing CRIME-PICS at two phases to the same respondents was problematic. Exigencies of life (where, for example, prisoners are moved to other institutions at short notice or their risk status is reclassified preventing participation) in the secure estate meant that there was little certainty of engaging the same young people twice. In addition, because of the imperative to anonymise participants, even where a young person did complete the instrument at both phases, it was not always possible to match the questionnaire to a particular respondent. This prevented comparison of individual instrument scores across time. Moreover, the group composition changed, with new participants (who had participated in the DofE during the preceding six months) joining the focus groups at phase two. For these reasons, the aim of comparing individual before and after scores was abandoned. Instead, a group measure, or average score of participants, was compared across the two study phases.

5.5 Overall, sixty four young people (at a total of six secure estate establishments) took part in focus groups at phase one of the study and 46 at follow up, with approximately 20 respondents completing the instrument at both study phases. Although the ATOB instrument data indicated some changes in group response patterns between the two study periods, these were not for the most part statistically significant. While descriptive statistical analysis did suggest less positive attitudes to crime at the second phase of the study, which were then 'explained' using some of the process evaluation data, publishing the findings proved difficult. Of the criminology journals approached, the majority were "not interested in small scale evaluations", while others critiqued the use of the ATOB instrument in a context which constrained its deployment to such an extent. When the process evaluation data were used to make sense of the ATOB data, quantitative reviewers did not appreciate the emphasis placed on the qualitative data, while qualitative reviewers critiqued the positivist framework in which the study sat. This latter point is poignant as, while I may argue that the qualitative PhD findings stand on their own, when presenting the study methodology it is clear that a positivist study design framed the research.

Methodological Distinctions

6.1 The problem of positioning the study overall, arose because it accommodated two distinct methodological traditions. While the funded study was underpinned by positivist assumptions, the PhD study claimed an interpretivist approach, drawing on relativist ontology and associated epistemological understandings (Denzin & Lincoln 1994). Two different sets of research methods (namely the attitude of offending instrument and qualitative interviewing) reflected the methodological distinction.

6.2 Qualitative process evaluation combined with quantitative methods is sometimes described as post-positivist in that it engages with some of the heuristic insights associated with interpretive social science, while retaining a positivist epistemological framework (Denzin & Lincoln 1998;Parry et al. 2001). Such a combined or pluralist approach is not uncommon especially in evaluation research (Pawson & Tilley 1997), where an attempt is made to make the best of both worlds by incorporating the reliability afforded by quantitative measurement while celebrating a measure of subjective understanding of participants. My PhD study, however, was treated as separate to the evaluation, and while I did collect some process evaluation data, I was keen to distance it from the aims of the funded study. Before exploring this issue further it should be noted that post-positivist evaluation designs themselves are not without their practical problems, particularly where evaluation outcomes identified by the positivist endeavour do not represent or capture the complexity of implementation, or its context, nor accommodate the (often longer term) unanticipated outcomes for participants (Ritchie et al. 2008).

6.3 While my role as researcher on the funded study included the collection of some process data, it was necessary that the PhD comprised a study in its own right which did not simply draw on the funded study data, nor reproduce its aims and objectives. The use of ATOB instrument, as a quantitative method, was kept separate from the PhD. However, some process data from the focus groups were used to explain the overall scores from the instrument, which did indicate some positive changes between the two phases of the study. Because the sole intention of the process data was their anticipated use in explaining the instrument scores, they were narrowly focused. In contrast, The PhD study was designed as a qualitative inquiry which would be theoretically driven by an interpretive approach, through which deeper understanding of the issues important to young people participating in the programme would emerge.

6.4 It is important to elaborate briefly my own methodological position in respect of the PhD study, which drew theoretically on phenomenological ideas. I was primarily interested in a phenomenological approach because of the emphasis upon individuals' lived experiences and shared meanings (Dowling 2004). At the same time the study was influenced by hermeneutic concerns about the nature of understanding and the interpretation of experience (Draper 1996). I was drawn to interpretivist understanding because it rejects a definitive version of external social reality in favour of the understanding that individuals find and create meaning through subjective interpretation of contexts which they inhabit (Denzin & Lincoln 2005; Parahoo 2006). Hence, the PhD set out to explore the experiences of study participants (young people in custody doing the DoE) which I felt could not be reached solely through the observation of situations or events. Rather it was the experience of participants, and my interpretations of these, which framed data collection as joint endeavour through conversation between myself and the study participants. Most importantly I determined that the PhD study would not be framed by positivist parameters which are concerned to articulate the 'right' (and only) way of examining phenomena and discovering 'truth'. Hence, the PhD work started from the premise that reality is socially constructed, rather than grounded in objective fact, a position which resonated with that of interpretivist phenomenologists who argue that natural scientific method is applicable to the investigation of physical objects and not 'self-reflecting human beings' (Flyvbjerg 2001: 32).

Revisiting a lost Cause

7.1 My rejection of positivism, in, respect of the PhD study, meant that any potential contribution of the data collected via the ATOB instrument was forfeited. Whilst it was clear that the instrument was problematic, particularly in the prison context, the data it produced appeared in some measure, to resonate with the qualitative findings. That is, descriptive statistical analysis suggested that attitudes towards offending behaviour changed over the two study phases and this was borne out by the qualitative process data. While for the most part, the quantitative findings were not statistically significant, they were nevertheless interesting and potentially important. In not acknowledging this aspect of the evaluation in the PhD the issue of causation, which is at the heart of positivist tradition, was side stepped. Because of this, the quantitative data were not be used as indication of change and the qualitative data were simply mined for their meaning rather than in relation to anything else. This at the time seemed appropriate, resonating as it did with my understanding that social science differs from natural science because social actions should be interpreted in relation to intentions, meanings, and contexts, rather in terms of causal explanation (Ekstrom 1992).

7.2 In retrospect, and with the aim of reconciling the distinct aspects of the overall endeavour, I set out to identify a methodological position which would accommodate causation without losing the emphasis on interpretation and meaning. This led me to critical realism, as a paradigm which celebrates qualitative interpretation but at the same time speaks unashamedly to issues of causality. Critical realists agree with positivists there are causes for social phenomena, just as there are in the natural world (Bhaskar 2004). However, they have their own take on causation which differs from positivism. That is it is characterised by a generative view of causality, which denotes a realist ontology and epistemology based on the understanding that social science should uncover processes and mechanisms which exist, but which (unlike in positivism) are not directly observable (Outhwaite 1987; Ekstrom 1992). Indeed the generative theory of causality is an implicit criticism of the succession view of causality favoured by positivism whereby events are understood to successively follow each other with some regularity, and where causal explanations are expressible as a set of universal laws. Conversely, in generative causality, it is the exposition of processes generating phenomena, events and actions which is central, and it is in the properties of the things and relations themselves where causal power is found. Causes are thus properties underpinning sequences of events and changes, which themselves are observable (Sayer 1984;Outhwaite 1987).

7.3 Generative views of causation support what has been described as an explanatory approach to social theorising (Therborn 1991). Because interpretive understanding renders it possible to identify motives behind actions, and contexts of meaning in which the actions are embedded, social scientists may, it is argued, elucidate the processes linking cause and effect (Ekstrom 1992). Most critical realists regard motives and intentions as central causal mechanisms in social-scientific research. These motives and intentions are developed within sociocultural frameworks (Sayer 1984). Hence, key to causal explanations of social actions is interpretation and understanding of the meaning invested by individuals in different situations. In other words, causal explanations of social action are found in the social properties and meanings indexing different contexts as well as in the processes whereby social phenomena are produced out of pre-existing conditions (as opposed to in the conjunction of events). While structure and agency are understood to be interdependent, it is possible to unpick them analytically, with the emergent structure furnishing a similar context of action for future individuals (Archer 1995). Through the isolation of structural and/or cultural factors which provide a context of action it is possible to investigate how those factors shape subsequent interactions and how in turn they reproduce or transform the initial context.

7.4 Causal-explanatory research calls for both quantitative and qualitative strategies that are much more sensitive to complex and changeable contexts than mainstream variable-oriented research (Pawson 1988; Sayer 1984). From this perspective interpretive and causal analysis are not opposed but inter-dependent (Outhwaite 1987; Sayer 1984). Critical realism accepts that social research is similar in some ways to natural scientific inquiry, but differs from it in its understanding of the active role of individuals in constructing the social world, and the complex and changeable nature of that world (Collier 1994). Its adherents agree with positivists in that, like the natural sciences, they argue there are causes for social phenomena (Bhaskar 2004). However, they do not think that social causation is discoverable using the same methods as positivists. So, for example, while critical realists may use experimental methods when generalising causation, the social sphere is not understood as a closed experimental environment, but rather an open system where complex variables cannot be controlled (Collier 1994). Moreover, because critical realist research begins with individuals' interpretation of social events proponents advocate using qualitative methods which explore meaning, perceptions and experience (Archer 2003). Some critical realists argue that social research should avoid reproducing the statistical methods of the 'hard' sciences' and/or that quantitative research is not necessary or desirable for social research (Archer 2003; Collier 1994). Indeed, Collier (1994) asserts that quantitative research is only able to provide very limited superficial knowledge, and therefore should be avoided. However, others reject the simple dichotomy drawn between qualitative and quantitative approaches, arguing that it is based on an erroneous division of the idealist and subjective, on the one hand, and the empirical and objective on the other (Sayer 1984; Danermark et al. 2002). While both types of research are considered important by many critical realists, however, qualitative research is more often privileged (Sayer 1984).

Designing a critical realist study

8.1 The process of working through the methodological dilemmas associated with my PhD has led me to imagine a different type of study informed by critical realism. This study would be designed to produce detailed understanding of why an intervention such as the DofE may have the potential to work (and for whom and in what circumstances) and effect actual changes (Carlsson 2004). In doing so it would endeavour to grasp structural and institutional aspects which are independent of the individual's reasoning but nevertheless influence process and outcomes, based on the understanding that it is the action of stakeholders and contextual conditions that facilitate change and not necessarily the intervention per se. A post-positivist approach is rejected because it mixes method in the absence of a cohesive methodology, and as consequence causation slips through the net.

8.2 This new study would not use an 'attitudes to offending behaviour' (ATOB) instrument, because it would endeavour to ascertain actual outcomes for participants rather than simply dispositions. In addition, while ATOB's do have their uses, in a custody context they are shown not to be suitable for the reasons described earlier. In preference, a system of data monitoring would be deployed, drawing on the Scottish experience of Transitional Care Monitoring. The Transitional Care initiative was established to address issues associated with the uneven provision of services throughout custody, for those with an identified drug problem, and to co-ordinate and enhance ex-prisoners' access to community based services on their release from prison. At the outset of the transitional care initiative, prisoners and service providers entered into a contract which formed the basis for a process and outcome evaluation which used a system of participant logs to record data from a range of stakeholder sources (MacRae et al 2006).

8.3 In brief, the proposed DoE study would similarly involve a contract, this time between the young person, YJS, DofE and the researcher, which would set out participation in (and effectively sign stakeholders up to) a longitudinal panel study. Data collection phases would comprise before and during implementation of the DoE programme delivery in custody, plus immediately following custody and (ideally) at both one and two year points of time following release into the community. Data collection would involve qualitative interviews, at each data collection phase, with a sample of young people undergoing the programme, in addition to data from the electronic progress records maintained by the DoE on all their participants and YJS data on recidivism. Qualitative interviews with young people would focus on specific elements of the programme content, delivery and receipt, tying the different programme components, closely into data collection. While it is acknowledged that such a study would inevitably face difficulties, particularly in maintaining involvement of study participants in qualitative interviews following release from custody, in principle outcomes in terms of programme completion and reoffending behaviour could be monitored respectively through the DoE and the YJS. Potentially, the study would therefore yield both qualitative a quantitative data to a serve a common aim.

8.4 Key elements of this proposal outline are informed by critical realist imperatives. That is, the study draws on both qualitative (interviews) and quantitative (from electronic records) data to closely monitor participant progress through, and subsequent to, custody, enabling the identification of both structural and cultural factors which make up the context of respondent actions. This renders possible an in-depth investigation of how these factors influence interactions and in turn reproduce or transform the context.

8.5 In conclusion, the revisiting exercise has served to address some personal concerns about post-positivist evaluation where qualitative process data are used to shore up positivist research design (Ritchie et al. 2008). A new study design informed by critical realism, which offers some ontological boldness tempered by epistemological caution (Danermark et al 2002) would, it is anticipated, place an emphasis on generative causation, where interpretation and understanding of the meaning invested by individuals in different contexts is central to the explanation of social actions.

References