Chapter I provided an overview of this study, its research question, definitions, and research heritage. In particular, it noted that this study of differences in the legal reasoning of first and third year law students is preceded most closely by the work of David Bryden (1984). Therefore legal literature, and especially Bryden's work, will be the starting point of this review. Reviewing the legal literature will contribute to this study both a definition of legal reasoning and a basis for coding its presence in essay question answers. Then this review will place the present study within the larger body of research on thinking, reasoning, and problem solving more generally. In designing this study to go beyond what Bryden had done, especially the technique of using thinking aloud protocols as a data collection method will be drawn from that larger field of research.
Formal, empirical study of law school learning is scarce. Teich (1986) and the report of the Special Committee for a Study of Legal Education (American Bar Association, 1980) are good overviews of prior work. Much of that work has consisted of surveys of opinion regarding legal education (e.g., Kahn, 1980). Some studies compared the effectiveness of various teaching methods (e.g., Kimball and Farmer, 1979; Lorensen, 1968) or testing methods (e.g., Wood, 1924; 1925; 1927). Closer to this study is the work of Philips (1982) in using ethnographic techniques to trace the learning of legal jargon or "cant." However, there is one prior study on which this study can build directly. That study was done by Bryden (1984).
Bryden studied law school learning by comparing the performance of students who were at early and late stages of their law school program. Bryden's participants were drawn from law students at three excellent American law schools. Random sampling was attempted in selecting students in their last semester of their third year of law school. Since about half of those asked declined to participate, the eighty-seven students who did volunteer cannot be considered a representative sample as Bryden himself notes (1984, p. 482). Beginning first year students were given a general invitation and Bryden then selected those whose law school entrance examination scores most closely matched the already selected third year students. To encourage students to participate, prize money was offered for excellent performance in the study.
Bryden's research procedure was to administer a four question, three hour essay examination to all the participants. The third year students took the examination during their last semester in law school and the first year students took it early in their first semester.
The essay questions were similar to those used for regular law school examinations. Fact patterns, new to the participants, described legal problems such as the possibility of making a legally binding contract in certain circumstances or the possible consequences for a pet cat that was alleged to have bitten two neighbors. Selected legal authorities, both cases and statutes, were included in an appendix.
Students were to find "issues," which are problems usually arising from gaps, ambiguities, or conflicts in the interrelationship of existing law with given facts. The questions were designed to raise certain types of issues that law students would be equipped to isolate and solve. For example, a statute was ambiguous in its possible application to a set of facts. Students were expected to know that interpreting the statute according to its purpose would be one way to resolve the issue. Likewise two cases in the appendix could have been applied to another set of facts. Students were expected to know how to use the cases based on the distinction between a case's "holding" (its binding legal authority) and its "dicta" (words which might be persuasive but which were not necessary to the decision and thus not binding on a future court).
Bryden and a student each coded the essay answers by counting the number of legal issues which were raised by the examinees and in addition they noted whether the examinees resolved each issue that was raised. Bryden then compared the examinees' responses to what he, as a law school professor, considered to be the issues raised in the questions. Bryden's scoring procedure is consistent with typical law school practice (Delaney, 1982; Kelso & Kelso, 1984; Nickles, 1977; Swygert, 1983; Wood, 1924, 1925, 1927).Bryden's findings were that "[t]he seniors were nearly always more proficient than the entering freshmen, yet hardly anything was said by a majority even of the seniors." In other words, even the third year law students often failed to raise and discuss issues that Bryden felt were presented by the questions. Bryden supports this summary conclusion by giving footnotes detailing what percentage of first and third year students raised each issue. No inferential statistics were applied to the data. He does, however, provide extensive examples of how the first and third year students responded to some of the questions.
Bryden admits that there are a number of methodological problems with the study. He acknowledges, for example, that his coding techniques were subjective but he compensates by sharing much of the data so that the reader can draw independent conclusions. A similar technique would have been helpful in regard to the sampling difficulties which he encountered; unfortunately no explanation is given for the large number of students who declined to participate.
One further research question that stands out in Bryden's work is the question adopted for pursuit in the present study. Bryden compared the legal reasoning ability of persons starting law school with those almost finished with the three years of law school. His finding of relatively small differences between those two groups not only was troubling to him as a legal educator but also led him to question whether any difference at all would be found if the group to be compared to third year students were not starting students, as he had done, but rather students who had completed one year of law school. That question, of course, is the one taken up by the present study.
This study will take from Bryden, not only the research question comparing first and third year law students, but also his dependence upon essay questions as a source of data. As will be clearer when the results of this study are discussed, the use of essay questions, for better or worse, was the single most important design decision that was made. Therefore, this review will devote a fair amount of space to explaining why essay questions seem to be an appropriate data gathering source when studying law school learning.
Should essay questions be the heart of this study's data collection? On the one hand, other tasks have proven fruitful in the analysis of analogous problem solving tasks. For example, Elstein, Shulman, and Sprafka (1978) used actors as patients to study the diagnostic skills of physicians. Legal clinics associated with most law schools would present similar research opportunities. However, if the goal is examining changes in legal thinking, then essay questions are closely tied to the law school educational methods for teaching thinking. The reasons for this close tie are developed in the following paragraphs.
The principal tie between essay questions and law school's teaching of legal thinking comes from the idea that essay questions are like a written continuation of the typical classroom process. As noted in the introduction, the traditional casebook method uses sequences of appellate court opinions. The facts, rules, and reasons of each opinion are compared and contrasted with the others as well as being tested against new hypotheticals proposed by the professor. Discussions typically are between the professor and individual students but all students are expected to follow along vicariously and be prepared to step in and carry the discussion if called upon. This process is described in more detail in, for example, Kelso (1972) or Turow (1977). An essay question can be seen as a written hypothetical to which each student must explicitly respond individually rather than implicitly as part of the classroom group. Thus essay questions closely fit the classroom teaching method.
The second part of the tie between essay questions and law school's teaching of thinking comes from the idea that the casebook method is the primary method of teaching legal thinking. Law schools use a number of teaching methods including lectures, simulations, and clinics (Levin, 1989). However, since about 1870, the casebook method has had a special place in the teaching of legal thinking. The following paragraphs will show this special place by briefly tracing the development of the casebook method.
These materials serve a function in addition to tracing the casebook method. The materials to be quoted are not the product of formal research efforts. All could be dismissed as speculation based upon anecdotal data. However, all the casebook method discussion comes from law professors who have made important contributions to legal literature. What they say, therefore, at least is data about the professors themselves and their own beliefs about the teaching of legal reasoning. Reading quotations such as those that follow is perhaps the only way to appreciate the depth of the professors shared belief that their teaching dramatically impacts the way that their students think. This belief about the possibility of teaching thinking is important for the present study even if what the professors say about the casebook method is flawed, at least from a research methodology standpoint.
The casebook method began at Harvard as a way of teaching, not so much legal thinking, but doctrine. In the words of Christopher Langdell, its founder:
Law, considered as a science, consists of certain principles or doctrines. To have such a mastery of these as to be able to apply them with constant facility and certainty to the ever-tangled skein of human affairs, is what constitutes a true lawyer; and hence to acquire that mastery should be the business of every earnest student of law. Each of these doctrines has arrived at its present state by slow degrees; in other words, it is growth, extending in many cases through centuries. This growth is to be traced in the main through a series of cases; and much the shortest and best, if not the only way of mastering the doctrine effectually is by studying the cases in which it is embodied. But the cases which are useful and necessary for this purpose at the present day bear an exceedingly small proportion to all that have been reported. The vast majority are useless, and worse than useless, for any purpose of systematic study. Moreover, the number of fundamental legal doctrines is much less
than is commonly supposed; the many guises in which the same doctrine is constantly making its appearance, and the great extent to which legal treatises are a repetition of each other, being the cause of much misapprehension. If these doctrines could be so classified and arranged that each should be found in its proper place, and nowhere else, they would cease to be formidable from their number (pp. vi, vii. 1871).
Langdell's basic message was that law could be a science if it restricted itself to the development of doctrines drawn from the primary data of decided cases. The commonplace analogy was that law would be like chemistry with the library as the laboratory. In Langdell's method, students of the law would learn, not from treatises or apprenticeships, but from casebooks of court opinions under the guidance of, not practicing attorneys, but professional teachers. This had the theoretical advantage of making legal education more suited for a university setting and the practical advantage of allowing a few teachers to instruct large numbers of students.
If law schools still professed to use Langdell's case method primarily to teach doctrine, then the method would have little relevance for the present study of legal thinking. However, as noted by Stevens (1983, pp. 119-120), Langdell's emphasis on doctrine was not totally accepted once it became apparent that the number of legal doctrines to be learned was greater than that predicted by Langdell. Gradually the emphasis turned to producing a "trained mind."
At first the changes were small as the case method's purpose moved from doctrine to mind training. In 1892, William Keener, Dean of Columbia Law School, wrote that at least one modification of the method seemed appropriate:
I think that time can be saved without the sacrifice of substantial benefit in the first year, by discussing in some subjects the elementary principles as stated by text writers before discussing the cases on a given topic. This simply for the purpose of acquainting the student with legal modes of thought and expression (p. 149).
After about forty more years, the change from conveying doctrine to teaching legal thinking had gone much further. In 1931, Thomas Konop, based upon his twenty-six years of experience, wrote the following:
The purpose of a law school is not merely to impart knowledge of the principles of the law - to simply stuff the memory of the students, but to train students to think legally.... The decided cases are the authoritative sources of the Common Law. The study of the law should proceed from these sources (p. 282).
Likewise in 1952, after quoting the previously quoted passage from Langdell, Morgan says:
I have had considerable experience in the classroom with second and third year students whose instruction in the first year has been of varied sorts. And that experience has demonstrated to my satisfaction that men who have not had rigid training by the case method in their first year are likely to have developed habits of superficial thought and a tendency to accept plausible generalizations and to support attractive fireside solutions of difficult problems by ethically appealing but inapplicable legal maxims; they still lack the capacity to discriminate between the legally important and the legally immaterial elements in a complicated situation, although many of them can write a pretty essay on the general principles which might be but probably are not pertinent upon careful analysis. Unless such habits are eliminated and unless such a capacity is developed, the student will continue to be unfit to do the kind of work required of every really good lawyer in any branch of professional activity (p. 387).
Morgan goes on to acknowledge that other teaching methods, including lectures, have a place in law school. However, his conclusion still is that the case method is the "best available method" for giving a student "a sure foundation" to acquire the capacities and qualities that the student needs (p. 391). In a comment that foreshadows the results of the present study, Morgan's conclusion also notes that a foundation is given because three or four years of education are insufficient to equip a student with the needed capacities and qualities.
Reference to more recent articles shows that the case method now is subject to more criticism but it continues to be thought of as the primary way to teach legal thinking. For example, Watson (1968) uses his experience both as a law professor and as a psychiatrist to address the case method of teaching. In his view, the common application of the method has detrimental effects for the student both in law school and in future legal practice. In law school, the method can lead to psychologically defensive reactions which hinder learning. Although some students adapt successfully, others are driven by stress to perform less well than they are able. In legal practice, the after effects of the case method may leave a student "unemotional" (p. 131) and thus less able to work with people in a service profession. This tracks with the critique of, for example, Redmount (1981).
Watson, however, seeks modifications to the case method, not its abolition:
Let me emphasize that it is a superb teaching technique which must remain a fundamental part of good legal education. There is no question of its effectiveness in developing enormous intellectual skill. However, modifications could be made which would greatly enhance its value (p. 145).
Principal among Watson's suggested modifications are various ways for students to see both the extent of their progress and the need for working with their emotions. Watson also suggests, among other things, that students be given heroes to emulate, that law professors receive interdisciplinary training to sharpen their teaching and counseling skills, that practitioners be involved in the teaching process, and that apprenticeship arrangements be considered to provide what law school cannot.
Finally, Wade (1984) is a good example of a more recent article that supports the use of the case method to teach legal thinking. However, these references to Langdell, Keener, Konop, Morgan, Watson and Wade are not meant to be taken as saying that all legal writers agree with the case method as traditionally used. To help place the preceding references in perspective, the following paragraphs will cite examples of writers who urge more dramatic changes in the case method. However, even while these writers debate particular aspects of the case method, the implicit assumption will continue to be that the case method plays a central role in the teaching of legal reasoning. This latter point shows why even persons seeking modifications to the case method are still important to this present study. If legal writers across a wide spectrum agree that the case method plays such a role in teaching legal reasoning, then the foundation is laid for an argument to be made shortly that it is advisable to continue to use essay questions as part of the data collection due to the close association that essay questions have with the case method.
The first example of a person seeking more sweeping changes to the case method system is Llewellyn. For example, Llewellyn (1948) noted that: "For it is obvious that man could hardly devise a more wasteful method of imparting information about subject matter [emphasis in original] than the case-class. Certainly man never has" (p. 215). Llewellyn's proposed remedy was to supplement the cases with outside readings. Llewellyn also proposed that the current definition of the case method was too narrow:
It is not the judicial decision which is the essence of the "case"; it is instead the concrete problem-raising situation - so that, as I see it, any introduction of the so-called "problem method" into law teaching is really but an expansion of the essential merits of case-teaching, an expansion obscured only by a current mis-emphasis upon the idea of a "case" as being at best the official report of a judicially decided cause [emphasis in original] (p. 217).
Finally, Llewellyn (p. 218) stated that law schools have not given attention to the type of books which could be substituted for casebooks nor have they combined the available teaching approaches to form "a reasonably rounded, reasonably reliable body of training for a whole student body" [emphasis in original]. This Llewellyn linked to a social responsibility to make a law degree "a reliable mint-mark" (p. 218) [emphasis in original].
Llewellyn suggested changes go further than any of the others because he would substitute hypothetical problems for appellate court cases. This goes to the heart of Langdell's desire to treat cases as original data. However, even in suggesting that the law school case method become more like the case method of, for example, a business school, Llewellyn at least keeps much of the classroom teaching method in place. The discussion materials change from appellate cases to hypothetical problems but much of the rest stays the same.
Another who seeks fundamental changes in the case method is Gross (1984). Gross' concern is with the much larger question of what conception of the legal system should law school convey to students. Much of his discussion rests on the premise that law schools rely too greatly on a nomological model of knowledge and rationality, that it is too often assumed that phenomena are completely determined by laws that can be precisely identified, defined, and proved (p. 393). However, as part of that larger issue, Gross argues (pp. 426-427) that the case method of teaching is too narrow for two reasons. The first reason is that the case method teaches only some of the skills used in legal practice. As Gross notes, the teaching of these broader skills has been treated by, for example, Kreiling (1981). Even if the clinical model of instruction described by Kreiling is not adopted, Gross would address the teaching of those broader skills, in part, by including in the curriculum more writing projects. The second reason for arguing that the case method is too narrow is that the case method, and especially the essay examinations used in conjunction with it, teach students to reason with overly abstracted rules rather than with the legal authority itself. In other words, students learn, especially in preparation for the time pressure imposed by examinations, to reduce cases to simple rules rather than to reason with them in the fuller process described by Levi. Gross argues that addressing this problem could be one part of making students more reflective learners.
As fundamental as Llewellyn's and Gross' suggested changes might appear to be, they are not the most extreme position in current law school thought about the case method. The most caustic critiques flow from writers like Kennedy and Boyle who would turn the method upon itself. For example, Kennedy admits that law school teaches some important skills (1982, p. 595) but he is extremely critical of current teaching, including the use of the case method:
Law schools teach these rather rudimentary, essentially instrumental skills in a way that almost completely mystifies them for almost all law students. The mystification has three parts. First, the schools teach skills through class discussions of cases in which it is asserted that the law emerges from a rigorous analytical procedure called "legal reasoning," which is unintelligible to the layman but somehow both explains and validates the great majority of the rules in force in our system. At the same time, the class context and the materials present every legal issue as distinct from every other, as a tub on its own bottom, so to speak, with no hope or even any reason to hope that from law study one might derive an integrating vision of what law is, how it works, or how it might be changed (other than in an incremental, case-by-case, reformist way).
Second, the teaching of skills in the mystified context of legal reasoning about utterly unconnected legal problems means that skills are taught badly, unselfconsciously, to be absorbed by osmosis as one picks up the knack of "thinking like a lawyer." Bad or only randomly good teaching generates and then accentuates real differences and imagined differences in student capabilities. But it does so in such a way that students don't know when they are learning and when they aren't and have no way of improving or even understanding their own learning processes. They experience skills training as the gradual emergence of differences among themselves, as a process of ranking that reflects something that is just "there" inside them.
Third, the schools teach skills in isolation from actual lawyering experience. "Legal reasoning" is sharply distinguished from law practice, and one learns nothing about practice....
Teachers convince students that legal reasoning exists, and is different from policy analysis, by bullying them into accepting as valid in particular cases arguments about legal correctness that are circular, question-begging, incoherent, or so vague as to be meaningless (p. 596).
Boyle (1985) demonstrates how Kennedy's ideas can be put into practice in the classroom. According to Boyle:
If there is one issue on which there is general consensus among law teachers, it is that legal education is even sicker than legal scholarship. First year angst, second and third year burnout, not enough theory, not enough practical skills - these themes come up again and again (p. 1005).
The tone of this quotation conveys the passion with which Boyle approaches the topic of law school teaching. Boyle argues, along with Kennedy, that legal reasoning techniques should be taught explicitly rather than simply being modelled. To illustrate how that can be done, Boyle provides descriptions of his day to day classroom teaching. Thus his typical practice is to give students a handout which explains the usual types of arguments based on precedent and policy. He argues that students can thereby learn in thirty minutes how to interpret words either by their dictionary meaning or their purpose, how to phrase a rule broadly or narrowly, and how to select among common pairs of policy objectives such as morality as freedom versus morality as security or the need for a firm rule versus the need for a flexible standard.
On the one hand, Boyle's use of a handout is consistent with the previous remarks of Keener and even the texts of Langdell's other followers included material other than just collections of appellate case reports (e.g., Ames, 1893, 1894, 1904). On the other hand, the handout is a call for a dramatic change. The summary of his handout states:
Neither the precedential nor the "policy" arguments are capable of providing a neutral, "correct" answer. You must learn to use these arguments, if you wish to convince decision makers. You cannot rely on them, or the system they justify, to provide you with moral or political guidance. In fact, the most dangerous thing about these arguments is that they tend to discourage committed moral or political thought about the legal system by making you feel insecure or powerless (until you learn them) and at the same time by offering tools to knock down any suggestion for change. But this cuts both ways. Remember that it also means that the arguments which people give you about why the status quo is "simultaneously natural, inevitable and just" - that these arguments are also vulnerable to deconstruction. The choice is yours (p. 1063).
The preceding references to Llewellyn, Gross, Kennedy, and Boyle demonstrate forcefully that not all law professors today agree with the case method of teaching as it has evolved from the time of Langdell. However, for the purposes of this study, the point again is that the case method, and therefore essay testing, are an integral part of the law school experience even in the eyes of those who seek major modifications in the method. In addition, the preceding references demonstrate the depth of these law professors' belief that their teaching has a major impact on their students' legal thinking.
The preceding discussion presents an indirect argument for the importance of essay questions as a source of data on the learning of legal thinking. The argument was that the case method of instruction has historically played a central role in the teaching of legal thinking. Therefore it was argued that the learning of legal thinking should be measured by a test that was very similar to the case method of instruction. Essay questions filled that role since they were like written hypotheticals forming the next step beyond the classroom discussion.
Another way to demonstrate the importance of essay questions is more direct but less explanatory. Reference will again be made to the work of Wood (1924; 1925; 1927) at Columbia Law School. Wood worked with law school professors to construct twenty-two objective tests in twelve courses over three years. The purpose of the tests was to study the ability of those tests to measure reasoning ability as compared with the like ability of traditional essay tests. Although some of the tests consisted of all objective questions, other tests were part objective and part essay. The objective test portion of a combined examination consisted of from 60 to 200 questions and the essay portion typically was three or four questions (1924, p. 247). In discussing the ability of the objective tests to predict future performance in law school, Wood drew this conclusion from some of the 1927 data:
Every correlation presented, without exception, clearly and consistently favors the conclusion that the new-type [objective] examinations give more accurate and more reliable measures of reasoning ability than old-type [essay] examinations of equal or greater time allowance, both when we compare the new and old-type examinations singly or in combination. In the face of such data, it seems impossible that the idea could persist that the objective examinations measure only factual knowledge and isolated bits of information (1927, p. 795).
Despite Wood's conclusions, law school examinations still are dominated by essay questions. More specifically, essay questions dominate at the institution where this research was conducted. The persistent use of essay questions, in the face of Wood's critique, more directly shows how deeply embedded essay questions are in legal education practice.
This study continued the use of essay questions due to both the direct and indirect arguments which have been detailed here. Although alternative data collection methods had been fruitful in other studies, particularly the use of live actors as simulated clients or patients, still the use of essay questions appeared reasonable in light of the connections between essay questions and legal education. Due to the importance of the decision to use essay questions, those connections with legal education have been developed at some length. However, as mentioned earlier, the decision to use essay questions will be revisited when the results of this study are discussed. The decision will be revisited because the use of essay questions may be the single most important factor underlying the results of this study. The reasons for the importance of using essay questions will include the insights of Gross presented above, especially his idea that students learn, not Levi's reasoning by analogy, but an oversimplified method of using abstracted rules.
Even if Bryden's research question about first and third year students is adopted, and even if essay questions are used to collect data, the further question remains of whether Bryden's coding scheme should be maintained. Bryden coded "issues" based upon interpreting a statue according to its purpose, distinguishing between the "holding" and "dicta" of a case, and the like. Although this approach provides indicators of legal reasoning which are clearly tied to the case method of instruction, a review of legal literature shows that "legal reasoning" has been defined in ways that permit other coding methods.
As an overview, legal reasoning, sometimes called "thinking like a lawyer", has been examined from a number of directions. Some have focused on the legal education process either from the perspective of law professors (e.g., Boyle, 1985; Brown, 1983; Cox, 1976; Friedland, 1986; Kelso, 1972; Llewellyn, 1960) or law students (e.g., Turow, 1977; White, 1983). Kahn (1980) constructed a taxonomy of legal thinking skills based largely on survey data from law students, professors, and practitioners. Moreover the American Bar Association has sponsored numerous studies of legal education. A good example is the 1980 report of the Special Committee for a Study of Legal Education. Gee and Jackson (1982) provide a good overview of legal education studies.
Legal education and legal reasoning have also been studied from other perspectives, most of which could have been used as the basis of this study. Teich (1986), as mentioned earlier, provides a good summary of prior empirical research on law school teaching methods, including the previously referenced studies of Kimball and Farmer (1979), Lorensen (1968), and Wood (1924, 1925, 1927). Crombag, de Wijkerslooth, and Serooskerken (1975) attempted to reduce legal problem solving to a flow chart. Schwartz (1980) and Thielens (1980) describe the socialization process of which thinking like a lawyer is a part. Philips (1982) used ethnographic techniques to study socialization from the aspect of learning the particular "cant" or language in which legal thinking is expressed. Sarat and Felstiner (1986) used similar techniques to describe what an attorney does during an actual client conference. Frank (1947) wrote about clinical legal education. Boyum and Mather (1983), Jensen and Horvitz (1979), Nagel and Neef (1979), and Spaeth (1979) all use decision theory to model legal reasoning. Brkic (1985) describes it from the perspective of formal logic. Gardner (1987) and Tito (1987) explain legal reasoning in the context of artificial intelligence. Stevens (1983) provides a historical perspective by tracing how thinking like a lawyer has been taught in America since 1850 and Consalus (1978) covers the Colonial period. Parallels have been drawn between legal reasoning and taking photographs (Conniff, 1985), writing history (Berlin, 1960), and interpreting literary works (Dworkin, 1985).
In addition to the preceding, two other perspectives could have been used as the basis of this study. One would be a philosophical perspective. This is an attractive possibility since this is the most elemental level on which to consider what thinking like a lawyer really means. Burton (1985), Golding (1984), Goldstein (1987), and MacCormick (1978) give good introductions to legal reasoning in a philosophical context. Dworkin (1978), Fuller (1964), Hart (1961), and Kelman (1987) are good examples of the vast literature developing the concepts in more detail.
The other attractive perspective from which to work would be based upon the lists or catalogues of legal education objectives developed by, for example, Cort and Sammons (1980), Holmes (1976), and Strong (1950). Wangerin (1986) reviews much of this work and adds some contributions of his own. Each of these lists attempts to present an overview of the capacities which a law student needs to become a lawyer. Thus Holmes includes descriptions not only of legal analysis and synthesis, but also such skills as legal drafting, counseling, and negotiation (p. 579). He also attempts to present common ground for all the different law school teaching approaches.
Cort and Sammons developed detailed definitions of generic competencies for use in evaluating law students performance in a law school clinical setting. Their approach also received support when the State Bar of California Bar Examiners (1988) adopted parts of it in the practice competency section of their bar examination.
However, for a study of the changes taking place in law students during law school, one definition of "legal reasoning" seems especially attractive because it is so closely tied to the essential ideas behind the casebook method. The definition was developed by a law school classroom professor, Edward Levi, a long-time professor of law at University of Chicago. Based upon his own experience in teaching from appellate opinions, Levi (1949, pp. 1-2) said:
The basic pattern of legal reasoning is reasoning by example. It is reasoning from case to case. It is a three-step process described by the doctrine of precedent in which a proposition descriptive of the first case is made into a rule of law and then applied to a next similar situation. The steps are these: similarity is seen between the cases; next the rule of law inherent in the first case is announced; then the rule of law is made applicable to the second case.
The balance of Levi's book consists of extended examples of legal reasoning drawn from sequences of appellate opinions. Part of one of those sequences, the one on the law of product liability, is included in the torts textbook (Prosser, Wade, and Schwartz, 1982) which all participants in this research project had studied. Jaff (1986) and Suber (1988) provide examples of how Levi's definition fits closely with law school teaching.
Levi's approach also has the advantage of describing a process which makes it more suitable for this study than the doctrine oriented approach of Langdell (1871), the closely associated definition of Holmes (1976), or Cort and Sammons' (1980) product oriented analysis. Levi's description also is at a convenient level of abstraction. On the one hand, Levi himself stresses (e.g., 1949, p. 7) that legal reasoning is not primarily a deductive process applied to an existing rule; reasoning by example is the more fundamental level of logic which is used to select rules or premises to be applied in a given situation. On the other hand, it does not propose to settle which of two conflicting analogies necessarily must be selected. That selection might well involve philosophical analysis on which there is no present consensus in the legal field. In other words, using Levi's definition provides the advantage of allowing a middle level of process analysis to which law students would have received repeated indirect exposure in their case method study of appellate opinions.
As will be developed in more detail in Chapter III, Levi's definition provides the basis for a system of coding different from the issue counting of Bryden. Levi's definition focuses on the role of case analogies in legal reasoning. Thus, if a person is using legal reasoning as described by Levi, they will mention cases. References to cases might prove to be easier or less ambiguous to code than the categories used by Bryden. Thus, although counting case references would be an indirect measure of legal reasoning, it still might be a better method than Bryden's counting of issues. In addition, Levi's definition has the advantage of describing legal reasoning even when the reasoning does not complete all the steps described by Levi. If a student uses only a rule, as Gross (1984) would predict, this still can be described in Levi's terms as being a truncated version of the more complete process. The use of the rule is counted even if the rest of the process is absent. Thus the coding still can work even if, for example, a deductive reasoning process is being used.
Levi's definition of legal reasoning, although developed in the legal context, also has the advantage of being paralleled by problem solving findings in other contexts. These parallels help link the concept of legal reasoning to the more general field of problem solving. It is not difficult, for example, to make connections between Levi's definition and the works of Polya or Sternberg. One of the closest parallels comes from the protocol analysis work of Schon (1983; 1987). Schon studied, among other things, the interactions between a teacher and student of architectural design. Based upon his observations, Schon isolates three levels or conceptions of thinking during problem solving. The first level is the use of facts, rules, and procedures to solve instrumental problems nonproblematically (1987, p. 39). Teaching this level of thinking is a form of technical training in which, as Schon puts it:
[s]tudents would be expected to acquire the material by reading, listening and watching, familiarizing themselves with examples of practice problems matched to the appropriate categories of theory and technique. Coaching would consist in observing student performance, detecting errors of application, pointing out correct responses. (1987, p. 39)
The second level of thinking is where problem solving is viewed as "thinking like a _____". On this level, Schon states:
students will still learn relevant facts and operations but will also learn the forms of inquiry by which competent practitioners reason their way, in problematic instances, to clear connections between general knowledge and particular cases. The standard drills of law school classroom and the medical clinic exemplify this view. In a practicum of this kind, there is presumed to be a right answer for every situation, some item in the corpus of professional knowledge that can be seen, eventually, to fit the case at hand. (1987, p. 39).
The third level of thinking is an extension of the second level but concentrates not on rules of inquiry assumed to be available to solve all problems but instead focuses on what Schon calls "reflection-in-action". This concept is developed in both of his books but he states the general idea as follows:
We will see students as having to learn a kind of reflection-in-action that goes beyond statable rules - not only by devising new methods of reasoning, as above, but also by constructing and testing new categories of understanding, strategies of action, and ways of framing problems. Coaches will emphasize indeterminate zones of practice and reflective conversations with the materials of a situation. (1987, pp. 39-40).
Both Levi (1949) and Schon (1987) leave open the question of how a problem solver finds an analogy or sees a new way of framing a problem. In the legal context, some insight on this process may be provided by the work of Collins (1985). His studies also used protocols to study legal teaching. His subjects included well-known professors like Max Beberman, Richard Anderson, Arthur Miller and Roger Schank.
Collins identified ten major strategies which inquiry teachers use in choosing cases (1985, pp. 580-581). The ten can be arranged in five pairs of cases: positive and negative paradigm; near miss and near hit; maximal and minimal relation; independent or dependent variable range; and insufficient or unnecessary factor counter example. Although Collins warned that the list is incomplete, especially for law schools where the cases may be in the form of hypotheticals, still the pairs add detail to optimal learning sequences such as that suggested by Gagne (1977, pp. 119-120) for concept learning.
In addition to the listing of types of cases used in inquiry teaching, Collins listed eight strategies that are used to force students to learn reasoning skills. Those eight were: asking students to form hypotheses; asking students to test hypotheses; asking students to make predictions; asking students to consider alternative predictions; entrapping students into revealing their misconception; tracing consequences to a contradiction; encouraging students to formulate alternative hypotheses; and encouraging students to question authority.
Collins' (1985) findings supplement the findings of Bryden, Levi, and Schon. Bryden counted the number of issues raised by his subjects. Levi referred to the use of analogies as central to legal thinking. Schon made reflective thinking the center of his discussion. However, as noted earlier, Bryden, Levi, and Schon all left much unsaid about how the subject originally would isolate an issue or pick an analogy or begin a new cycle of reflective thought. More generally, Dillon (1982) has noted that not much is known about "problem finding" as opposed to problem solving.
In this regard, Collins (1985) might have helped. Although a distinction might be drawn between techniques of teaching problem solving and techniques of problem solving, still Collins' list of case types and teaching strategies might have served as guides when measuring problem solving learning. This possibility arose from the research of those like Bandura (1965) who found that modeled behavior was likely to be imitated. In the present study, law school professors will have modeled certain techniques in the process of teaching problem solving. When student problem solving is observed, one could hypothesize that some of those techniques might appear even though it must be granted that the task facing the student is very different from that which faced the professor. For the professor, the techniques might help in selecting the next question to ask in order to assist a learner. For the student, the techniques might help in finding an appropriate analogy to solve a legal problem. Thus Collins' list of techniques might have provided clues on what to look for when legal reasoning is examined in protocols. However, as will become clearer in Chapters IV and V, the data actually gathered in the present study did not lend itself to analysis using Collins' lists.
Despite the nice meshing of Levi's definition with the work of Schon and Collin's, not everyone would agree that reasoning by analogy is the best definition of what students learn in law school. In part this has been foreshadowed in the exchange between King James and Lord Coke which was quoted in Chapter I. Just as King James felt that he had science and reason and therefore he did not need specially educated judges to decide cases, so some scholars would support the possibility that law students come to law school already equipped with the reasoning process described by Levi and therefore the scholars question how time is spent in law school. On a very general level, this would be supported by Minsky's (1986) observation that our language is riddled and stitched with analogies and metaphors. In the legal context, Mudd (1983, pp. 705-706) has questioned how legal thinking differs from what students learn in undergraduate courses. Hornstein (1981) likewise argues that legal reasoning is good reasoning applied to legal materials. Fejfar (1986), relying upon Lonergan (1957), states that: "Much of a law student's education, as well as a lawyer's practice, involves common sense, rather than purely logical or scientific understanding." Part of Lonergan's description of common sense sounds much like Levi's description of legal reasoning:
Common sense, on the other hand, never aspires to universally valid knowledge and it never attempts exhaustive communication. Its concern is the concrete and particular. Its function is to master each situation as it arises. Its procedure is to reach an incomplete set of insights that is to be completed only by adding on each occasion the further insights that scrutiny of the occasion reveals. (Lonergan, 1957, pp. 176-177)
If legal reasoning is common sense, then interesting linkages might exist with work currently being done on informal logic (e.g., Blair & Johnson, 1980). However, if legal reasoning is common sense already possessed by the students, then this could have a dramatic effect on a study, such as this one, which attempts to measure law school learning. This topic will come up again in Chapter V during the discussion of the results of this study. In any case, the debate over the nature of legal reasoning helps to illustrate how a review of the legal literature soon leads to literature on thinking, reasoning, and problem solving more generally. It is to that literature that this review now turns.
General literature on thinking, reasoning, and problem solving.
Reviewing legal education literature provides not only a starting point in Bryden but also a basis for using essay questions in the study of legal reasoning. However, further insights into the present research question are provided by work that has been done in regard to thinking, reasoning, and problem solving more generally. This has already been demonstrated by, for example, reference to the work of Schon and Lonergan. In addition, this section of the review will draw from that body of work especially the idea of using thinking aloud protocols to supplement the data being gathered in the essay questions.
It is not the purpose of this review to exhaustively cover prior work on thinking, reasoning, and problem solving. That literature stretches back even beyond the dialogue of Socrates with Menon (Plato, ____B.C./1956). For a person who wishes to review more of that literature, at least the more recent portions have been surveyed in the works of a number of authors. Good examples include the works of Mayer (1983), Nickerson (1988), Nickerson, Perkins, and Smith (1985), and Shuell (1986). Edited works include: Baron and Sternberg (1987); Johnson-Laird and Wason (1977); and the two volume set by Segal, Chipman, and Glaser (1985). Frederiksen (1984), Hill's (1979) bibliography, and Rowe (1985) concentrate on problem solving. Edited works on problem solving include Groner, Groner, and Bischof (1983) and Tuma and Reif (1980). Bloom and Broder (1950) discussed the problem solving process of college students. Chi, Glaser, and Farr (1988) edit a collection on the nature and acquisition of expertise. Kennedy (1987) concentrates on the development of expertise in the context of professional education.
The preceding reviews show that, as was the case for legal education, some programs which teach thinking have been the subject of relatively little formal research. For example, the works of de Bono (1967; 1976; 1986), Polya (1954; 1957), and Whimbey and Lochhead (1981) have been widely accepted and used but much of that acceptance and use appears to be based upon intuitive appeal rather than extensive formal evaluation. Of course this does not mean that no evaluation of the programs has been done. Lochhead (1985), for example, reports on how parts of his program have been evaluated. However, some programs have been subjected to much more evaluation. For example, IE or FIE (Feuerstein Instrumental Enrichment) program of Feuerstein (Feuerstein, Hoffman, Jensen, & Rand, 1985) is a program which has been formally tested a number of times as illustrated by the review of Savell, Twohig, and Rachford (1986).
The reviews also show that, compared to the legal literature, there is an even greater split between studies which support or undermine the proposition that thinking, reasoning or problem solving can be taught. On the supporting side are studies like Nisbett, Fong, Lehman, and Cheng's (1987) success in teaching inferential rules, Ghatala, Levin, Pressley, and Lodico's (1985) results in teaching cognitive strategy monitoring to second graders, and Woodward, Carnine, and Gersten's (1988) reported improvements to problem solving skills through the use of structured teaching and computer simulations.
Even closer to this study would be successful programs based on Sternberg's (1977; 1982) work on analogies. Polya's work would have fit into this category but, as Newell (1983) notes, Polya's work is often overlooked. In any case, Alexander, White, Haensly, and Crimmins-Jeanes (1987) is a good example of a successful program based upon Sternberg's work. It is especially interesting for the law school context because, although based upon work with fourth graders, it supports the proposition that direct classroom instruction can help with analogical reasoning.
On the other hand, there are also studies which undermine the proposition that thinking, reasoning, and problem solving can be taught. For example, if behavioral decision theory studies are consulted, a number would call into question the very kind of learning which law school attempts to provide. Johnson (1988) is a recent summary of the implications of those studies across contexts such as graduate admissions office decisions (Dawes, 1971), economic forecasting (Armstrong, 1978), and clinical psychology (Goldberg, 1970). Likewise the studies of Elstein et al., (1978) and Feltovich, Johnson, Moller, and Swanson (1984) indicate that learning a knowledge base is the key to expert problem solving.
Reference to the wider field of thinking, reasoning, and problem solving thus does not provide unequivocal support for the proposition that thinking, reasoning, and problem solving can be taught in schools, even if the school is a law school. However, the studies do provide something else for this study. Many of the studies in other areas use, not the essay questions of law school, but thinking aloud protocols produced by making a transcript of what a person says when asked to say what they are thinking while problem solving. Newell and Simon (1972) demonstrated the use of the method with puzzle solving but it soon was used in expert-novice (Elstein et al., 1978; Feltovich et al., 1984) and naive-novice (Voss, Blais, Means, Greene, and Ahwesh, 1986) comparisons.
Collection of thinking aloud protocols has not met universal approval. Nisbett and Wilson (1977) present detailed arguments against them, including the idea that thinking aloud interferes with the task being studied. However, Ericsson and Simon (1984) answer many of those arguments, at least in a context where concurrent verbal explanations are close to the mental verbalizations likely to be involved in the task. For example, concurrent verbalization might not interfere with execution of a mental task like a Tower of Hanoi problem but might well interfere with a task like hitting a baseball.
Likewise Johnston and Afflerbach (1985) addressed the problem of experts not verbalizing what they were doing because the processes appeared to be elemental. Johnston and Afflerbach believed that the processes were not elemental but rather were automated from long practice. They found that selecting the appropriate level of difficulty for a question could increase the amount and quality of data gathered. Increasing the difficulty of the questions apparently caused the experts to move more toward novice performance. When operating closer to the novice level, the automated processes disappeared and the steps in the thinking process were rendered accessible.
Although studies such as those of Ericsson and Simon and Johnston and Afflerbach address some of the criticisms of thinking aloud protocol methodology, they do not answer all the questions, particularly in regard to combining thinking aloud with the task of writing answers to essay questions. In addition, alternative research methodologies are available. As mentioned earlier, Johnson (1988) collects a number of studies which have used judgment and decision theory to study experts. Shulman and Elstein (1975) explain various judgment and decision theory approaches. They also distinguish those approaches from process tracing approaches, including thinking aloud protocols, and give examples of contributions attained by each approach.
No one research approach has emerged as the method of choice. However, given the emphasis of this study on reasoning rather than, for example, decision making, it would appear that the use of thinking aloud protocols would be a good research approach to supplement the data collection from essay questions.
Summary.
The goal of this study is to examine the differences in the legal reasoning of first and third year law students. Because the starting point for this study would be Bryden's work based in the legal research tradition, this review first examined works in that scholarly tradition. That review showed that the use of essay questions was deeply embedded in both law school teaching theory and practice. The key to the wide-spread use of essay questions appeared to be their close connection to the type of discussion typically conducted in a law school class using the case method of teaching.
Regarding the definition of the key phrase "legal reasoning", a number of options presented themselves in the literature. The basis for these ranged from the Langdell's doctrinal emphasis to the practice oriented descriptions of Cort and Sammons. For purposes of studying the effect of classroom learning, the definition developed by a classroom professor was selected as most closely tied to the case method of instruction. The process described in Levi's definition also had the advantage of facilitating coding and of paralleling the processes described by, for example, Schon.
Although Bryden's study provided the research question and initial outline for this study, and although review of other legal research both provided a definition of legal reasoning and reinforced the advisability of using essay questions to elicit data, the more general literature also made contributions. The impetus for conducting this study was increased by the possibility of contributing to the ongoing controversy about whether thinking, reasoning, and problem solving can be taught. In addition, the general literature provided an additional research tool in the form of thinking aloud protocols. This tool would assist data collection on participants' thinking processes at the same time that the written products in the form of essay answers would be gathered.
Much of this study's design has taken shape during this review of the literature. However, the more detailed description of the resulting research plan now will be taken up in Chapter III.