Show me the money! Or so the saying goes. Implying that talk is cheap and facts are the only thing that matters. But there is another thing we are being asked to do with money and that is put it where our mouth is. So evidence is not quite enough. We have to also be willing to act on it and demonstrate a personal commitment to our facts.
As so often, folk wisdom has outlined two contradictory dictums that get applied based on the parameters of a given situation. On one account, evidence is all that is needed. Wheras an alternative interpretation claims that evidence is only sufficient when backed up by personal commitment.
I’ve recently come across two approaches to the same issue that resonate with my own, namely claiming that evidence cannot be divorced from the personal commitment of its wielder. As such it needs to be constantly interrogated from all perspectives, including the ones that come from “scientific method” but not relying solely on them.
This is crucial for anybody claiming to adhere to evidence-based practice. Because the evidence is not just out there. It is always in the hands of someone caught (as we all are) within a complex web of ethical commitments. Sure, we can point to a number of contexts where lack of evidence was detrimental (vaccinations). But we can equally easily find examples of evidence being the wrong thing to have (UK school league tables, eugenics). History of science demonstrates both incorrect conlusions being drawn from correct facts (ether), and correct conclusions being drawn form incorrect facts (including aircraft construction as Ira Flatow showed). I am not aware of any research quantifying the proportion of these respective positions. Intuitively, one would feel inclined to conclude that mostly the correct facts have led to the correct conclusions and incorrect ones to the wrong ones but we have been burned before.
That’s why I particularly liked what Mark Kleiman had to say at the end of his lecture on Evidence-based practice in policing:
“The notion that we can substitute method for common sense, a very wide-spread notion, is wrong. We’re eventually going to have to make judgments. Evidence-based practice is a good slogan, but it’s not a method in the Cartesian sense. It does not guarantee that what we’re going to do is right.”
Of course, neither is the opposite true. I’m always a bit worried about appeals to “common sense”. I used to tell participants on cross-cultural training courses: there is nothing common about common sense. It always belongs to someone. And it does not come naturally to us. It is only common because someone told someone else about it. How often, have we heard a myth debunked only to conclude that the debunking had really been common sense. That makes no sense. It was the myth that was common sense. The debunking will only become common sense after we tell everyone about it. How many people would figure out even the simplest things such as looking right and then left before crossing the road without someone telling them? We spend a lot of time (as a society and individuals) maintaining the commonality of our senses. Clifford Geertz showed in “Common sense as a cultural system” the complex processes involved in common sense and I would not want to entrust anything of importance to common sense. But I agree that judgments are essential. We always make judgements whether we realise that or not.
This resonates wonderfully with what I consider to be the central thesis of the “Science’s First Mistake” a wonderful challenge to the assumptions of the universality of science by Ian Angell and Dionysios Demetis.
“…different ages have different perceptions of uncertainty; and so there are different approaches to theory construction and application, delivering different risk assessments and prompting different decisions. Note this book stresses decisions not solutions, because from its position there are no solutions, only contingent decisions. And each decision is itself a start of a new journey, not the end of an old one.
Indeed, there is no grander delusion than the production of a solution, with its linear insistence on cause and effect.”
All our solutions are really decisions. Decisions contingent on a myriad of factors both related to the data and our personal situation. We couch these decisions in the guise of solutions to avoid personal responsibility. Not perhaps often in a conscious way but in a way that abstracts away from the situation. We do not always have to ask qui bono when presented with a solution but we should always ask where from. What is the situation in which a solution was born? Not because we have a suspicious mind but because we have an inquisitive one.
But decisons are just what they are. They are on their own neither good or bad. They are just inevitable. And here I’d like to present the conclusions to a paper on the epistemological basis of case study in educational policy making I co-wrote a few years ago with John Elliott. It’s longer (by about a mile) than the two extracts above. But I highlighted in bold the bits that are relevant. The rest is there just to provide sufficient context to understand the key theses:
from the Summary and Conclusions of “Epistemology as ethics in research and policy: Under what terms might case studies yield useful knowledge to policy makers. by John Elliott and Dominik Lukeš. In: Evidence-based Education Policy: What Evidence? What Basis? Whose Policy?, 2008.”
Overall, we have informed our inquiry with three perspectives on case study. One rooted in ethnography and built around the metaphor of the understanding of cultures one is not familiar with (these are presented by the work of scholars like Becker, Willis, Lacey, Wallace, Ball, and many others). Another strand revolves around the tradition of responsive and democratic evaluation and portrayal represented by the work of scholars such as MacDonald, Simons, Stake, Parlett, and others. This tradition (itself not presenting a unified front) aims to break down the barriers between the researcher, the researched and the audience. It recognizes the situated nature of all actors in the process (cf. Kemmis 1980) and is particularly relevant to the concerns of the policy maker. Finally, we need to add Stenhouse’s approach of contemporary history into the mix. Stenhouse provides a perspective that allows the researcher to marry the responsibility to data with the responsibility to his or her environment in a way that escapes the stereotypes associated with the ‘pure forms’ of both quantitative and qualitative research. A comprehensive historian-like approach to data collection, retention and dissemination that allows multiple interpretations by multiple actors accounts both for the complexity of the data and the situation in the context of which it is collected and interpreted.
However, we can ask ourselves whether a policy-maker faced with examples of one of these traditions could tell these perspectives apart? Should it not perhaps be the lessons from the investigation of case study that matter rather than an ability to straightforwardly classify it as an example of one or the other? In many ways, in the policy context, it is the act of choice, such as choosing on which case study a policy should be based, that is of real importance.
In that case, instead of transcendental arbitration we can provide an alternative test: Does the case study change the prejudices of the reader? Does it provide a challenge? Perhaps our notion of comprehensiveness can include the question: Is the case study opening the mind of the reader to factors that they would have otherwise ignored? This reminds us of Gadamer’s “fusion of horizons” (“Understanding […] is always the fusion of […] horizons [present and past] which we imagine to exist by themselves.” Gadamer 1975, p. 273).
However, this could be seen as suggesting that case study automatically leads to a state of illumination. In fact, the interpretation of case study in this way requires a purposeful and active approach on the part of the reader. What, then, is the role of the philosopher? Should each researcher have a philosopher by his or her side? Or is it necessary, as John Elliott has long argued, to locate the philosopher in the practitioner? Should we expect teachers and/or policy-makers to go to the philosopher for advice or would a better solution be to strive for philosopher teachers and philosopher practitioners? Being a philosopher is of course determined not by speaking of Plato and Rousseau but by the constant challenge to personal and collective prejudices.
In that case, we can conclude that case study would have a great appeal to the politician and policy maker as a practical philosopher but that it would be a mistake to elevate it above other ways of doing practical philosophy. In this, following Gadamer, we advocate an antimethodological approach. The idea of the policy maker as philosopher and policy maker as researcher (i.e. underscoring the individual ethical agency of the policy maker) should be the proper focus of discussion of reliability and generalization. Since the policy maker is the one making the judgement, the type of research and study is then not as important as a primary focus.
And, in a way, the truthfulness of the case arises out of the practitioners’ use of the study. The judgment of warrant as well as the universalizing and revelatory nature of a particular study should become apparent to anybody familiar with the complexities of the environment. An abstract standard of quality reminiscent of statistical methods (number of interviewees, questions asked, sampling) is ultimately not a workable basis for decision making and action although that does not exclude the process of seeking a shared metaphorical perspective both on the process of data gathering and interpretation (cf. Kennedy 1979, Fox 1982). Gadamer’s words seem particularly relevant in this context: “[t]he understanding and interpretation of texts is not merely a concern of science, but is obviously part of the total human experience of the world.” (1975, p. xi)
We cannot discount the situation of the researcher no more than we can discount the situation of the researched. One constitutive element of the situation is the academic tribe: “[w]e are pursuing a ‘scholarly’ identity through our case studies rather than an intrinsic fascination with the phenomena under investigation” warns Fox (1982). To avoid any such accusations of impropriety social science cultivates a ‘prejudice against prejudice’, a distancing from experience and valuing in order to achieve objectivity whereas the condition of our understanding is that we have prejudices and any inquiry undertaken by ‘us’ needs to be approached in the spirit of a conversation with others; the conversation alerts participants to their prejudices. In a sense, the point of conversation is to reconstruct prejudices, which is an alternative view of understanding itself.
It should be stressed however, that this conversation does not automatically lead to a “neater picture” of the situation nor does it necessarily produce a “social good”. There is the danger of viewing ‘disciplined conversation’ as an elevated version of the folk theory on ideal policy: ‘if only everyone talked to one another, the world would be a nicer place’. Academic conversation (just like any democratic dialectic process) is often contentious and not quite the genteel affair it tries to present itself as. Equally, any given method of inquiry, including and perhaps headed by case study, can be both constitutive and disruptive of our prejudices.
Currently the culture of politeness aimed at avoiding others’ and one’s own discomfort at any cost contributes to the problem. Can one structure research that enables people to reflect about prejudices that they inevitably bring to the situation and reconstruct their biases to open up the possibility of action and not cause discomfort to themselves or others? We could say that concern with generalization and method is a consequence of academic discourse and culture and one of the ways in which questions of personal responsibility are argued away. Abstraction, the business of academia, is seen as antithetical to the process of particularization, the business of policy implementation. But given some of the questions raised in this paper, we should perhaps be asking whether abstraction and particularization are parallel processes to which we ascribe polar directionality only ex post facto. In this sense we can further Nussbaum’s distinction between generalization and universalization by rephrasing the dichotomy in the following terms: generalization is assumed to be internal to the data whereas universalization is a situated human cognitive and affective act.
A universalizable case study is of such quality that the philosopher policy maker can discern its relevance for the process of policy making (similarly to Stake’s naturalistic generalization). This is a different way of saying the same thing as Kemmis (1980, p. 136): “Case study cannot claim authority, it must demonstrate it.” The power of case study in this context can be illustrated by anecdotes from the field where practitioners had been convinced that a particular case study describes their situation and berating colleagues or staff for revealing intimate details of their situation, whereas the case study had been based on research of an entirely unrelated entity. In cases like these, the universal nature of the case study is revealed to the practitioner. Its public aspects often engender action where idle rumination and discontent would be otherwise prevalent. Even when this kind of research tells people what they already “know”, it can inject accountability by rendering heretofore private knowledge public. The notion of case study as method with transcendental epistemology therefore cannot be rescued even by attempts like Bassey’s (1999) to offer ‘fuzzy generalizations’ since while cognition and categorization is fuzzy, action involves a commitment to boundaries. This focus on situated judgement over transcendental rationality in no way denies the need for rigour and instrumentalism. We agree with Stenhouse that there needs to be a space in which the quality of a particular case study can be assessed. But such judgments will be different when made by case study practitioners and when made by policy-makers or teachers. The epistemological philosopher will apply yet another set of criteria. All these agents would do well to familiarize themselves with the criteria applied by the others but they would be unwise to assume that they can ever fully transcend the situated parameters of their community of practice and make all boundaries (such as those described by Kushner 1993) disappear.
Philosophy often likes to position itself in the role of an independent arbiter but it must not forget that it too is an embedded practice with its own community rules. That does not mean there’s no space for it in this debate. If nothing else, it can provide a space (not unlike the liminal space of Turner’s rituals) in which the normal assumptions are suspended and transformation of the prejudice can occur. In this context, we should perhaps investigate the notion of therapeutic reading of philosophy put forth by the New Wittgensteinians (see Crary and Read 2000).
This makes the questions of ethics alluded to earlier even more prominent. We propose that given their situated nature, questions of generalization are questions of ethics rather than inquiries into some disembodied transcendent rationality. Participants in the complex interaction between practitioners of education, educational researchers and educational policy makers are constantly faced with ethical decisions asking themselves: how do I act in ways that are consonant with my values and goals? Questions of warrant are internal to them and their situation rather than being easily resolvable by external expert arbitration. This does not exclude instrumental expertise in research design and evaluation of results but the role of such expertise is limited to the particular. MacDonald and Walker (1977) point out the importance of apprenticeship in the training of case study practitioners and we should bear in mind that this experience cannot be distilled into general rules for research training as we find them laid out in a statistics textbook.
Herein can lie the contribution of philosophy: An inquiry into the warrant and generalization of case study should be an inquiry into the ethics surrounding the creation and use of research, not an attempt to provide an epistemologically transcendent account of the representativeness of sampled data.
Related articles
- Towards Community-based Research for Education (researchity.net)
