Educational Data Mining

A recent Education Week article on “educational data mining” (EDM) highlights several key resources and insights.

To wit:

  • “the first international conference on the subject [was] held in 2008 and the first academic journal [was] launched a year later.”
  • the article reports on a study that demonstrated how aggregating large amounts of student data helped researchers make better evaluations about whether students were guessing while doing their work
  • several other centers focused on the field are referenced

A key quote: “Analysis of massive databases isn’t new to fields like finance and physics, but it has started to gain traction in education only recently.”  The expansion of large scale data mining into the human services field is just beginning to explode and will have great consequences for how we conceive of education.


Texas Two-Step: A Pair of Trends that Undermine the Textbook Influence of the Lonestar State (Pt. II)

A friend forwarded me an article about the Texas textbook market that validates some of the observations/predictions I made in March.  Specifically, it highlights trends that could be undermining Texas’ influence on the textbook market (while also questioning the validity of the theory to begin with).


1) Texas’ actions re: curricular content could be giving rise to opposition in other states.

“The debates in Texas only heighten the sensitivity” in states and districts elsewhere to review those materials more closely before signing off, Mr. Diskey [executive director of the school division of the Association of American Publishers] added.

2) Texas is participating in breaking up its own influence by responding to incentives to provide for digital instructional materials.

In any case, amid concerns about the high cost of printed textbooks and the rapidity with which they become outdated, the Texas market for instructional materials is poised for a potential sea change. The recent legislation is expected to provide districts with new sources of digital textbooks and other electronic classroom materials.

“Now we have all of these new ways of acquiring instructional materials in addition to the traditional process,” said Anita G. Givens, an associate commissioner at the Texas Education Agency.

For instance, the state education commissioner was given authority to approve a list of digital textbooks that districts may buy with state textbook aid, providing them with new options beyond the materials adopted by the state board. Also, districts for the first time will be able to use a portion of that aid to pay for hardware, such as laptop computers, to access digital content.

“That is a big shift,” Ms. Givens said, “because one of the cost drivers in terms of whether electronic [material] makes sense is whether [schools] have the infrastructure and the access points.”

A friend today summed up the new economy: “It’s about information and networks.”  Changes in how instructional content is delivered remain one of the best symbols of this evolution.

Usability and Usefulness: Teachers and the Data They Love

I promised I’d return to the question of usefulness and usability in Instructional Improvement Systems, as it relates to the Wireless Generation report I discussed in an earlier post. Here are their conclusions, interspersed with my comments:

“First, data must be fresh: between a day and a week old . . . One large district discovered, soon after the launch of its teacher-facing system, that teachers started to call the help desk to complain as soon as the data are even three days stale.

This teacher behavior happily dovetails with research (discussed previously) that shows that short-term data analysis is most convincingly correlated with improved instruction.

“Second, data must be rich, providing multiple sources so that educators can ‘triangulate’—home in on a particular problem with the confidence that different measures agree. Many standardized assessments (including those sold as ‘formative’) are tuned for the middle of the curve, not for below-proficient students; they may be able to pick out at-risk students but do a poor job diagnosing what is causing at-riskness.

Third, data must be fine-grained enough to be instructionally actionable . . . for instance, if standards do not differentiate two-digit multiplication items that are cast as computation versus word problems, teachers may not uncover the students who need extra support in approaching word problems.

The two points above outline the classic problem of moving from a data system that provides general descriptive information (about the whole dataset) to one that provides actionable information about particular challenges.

“Fourth, if users are truly to explore data, access tools must be Google-fast and Apple-simple, with response times of, at most, a few seconds.

Who can argue?

“Finally, data needs to be clean and accurate. Happily, the best way to establish accurate education data for a student is to show it to that student’s teacher—or, of course, the student—and provide him or her with a way to address errors, for instance, by calling a help line or clicking a ‘report a problem’ link.”

Understanding variations in user behavior is an often-neglected aspect of information systems design and is a defining pursuit, as I see it, of the field of knowledge management. Concrete, human, details, such as these, increase my confidence in the expertise of the analyst.

So far, so good.

Next time we take on this issue, we’ll look at some of the peculiar dynamics that arise when initiating systems change in a non-competitive environment, such as characterizes good chunks of the human services world.

A Systems-Based Look at How Your Child Is Being Evaluated

The question of when to aggregate, and when to distribute, innovative energy in the development of data systems is an omnipresent one.  I’ve discussed before how user-friendliness, especially important in the human services field, depends upon the robustness of the underlying information system.  A recent Wireless Generation (an organization covered in an earlier post) report (a) discussed the history of information system development in state school systems and (b) provides suggestions about how to make such systems accessible, engaging, and useful.

Today we’ll cover (a), the history and will spin out some of its implications for the strategic aspects of data systems development (We’ll take on (b), usability and usefulness, at a later time.). Continue reading

Do Information Management Systems Help Teachers Teach?

From a recent paper on how teachers use data from assessments (CPRE, 2009):

“Overall, we found that teachers who focused on students’ conceptual understanding using one type of assessment were more likely to do so for all types of assessment, including interim assessments. This suggests that analytic or diagnostic capacity underlies effective formative assessment, regardless of whether those assessments are embedded within instruction, developed by teachers, or externally designed.”

The authors recommend providing better training to teachers on how to use data: “professional development for interim assessment use should go beyond using ‘point and click’ to locate and organize data and should emphasize analysis of student results in the context of standards and curriculum.”

Whether teacher training can fill this gap, or whether the pool of teachers needs to be adjusted, the paper provides a necessary reminder of the dangers of installing new technology without first understanding how it will generate the desired results.

(Thanks to Sarah Tantillo, NJ charter school pioneer and literacy expert, for bringing this study to my attention.)

Texas Texts and E-Media: The True Story of How the Texas BoE May Revise Education History

You may have read about the recent textbook revisions proposed in Texas (I include both a link to The Economist‘s take and a link to the detail-full, though more partisan take of The Huffington Post.).  While The Economist, The New York Times, and others highlight the economic weight of textbook-related decisions by the State of Texas, I would argue that a more dramatic outcome may eventually involve the relative dimunition of Texas’ influence.  Namely: Texas’ potential decision creates a perfect storm to increase interest in electronic and/or open source textbooks, which do not require economies of scale as large as does the current publishing regime.

Electronic innovators and open source writers tend to be (much more often than not) precisely the sort of folks who would most object to the Texas Board of Education recommendations.  I would not be surprised to see an “alternative textbooks movement” take root in the short-term (and a less-covered Texas controversy took place on this front in recent months); I would, however, be quite surprised if such a movement does not materialize in the mid-term.

Upstream, Downstream: How Schools Districts Should Think about Data

Most school district data regimes are focused on collecting information from the classroom and “funneling it up”: providing statistics to management-level decision-makers.   One challenge of the coming decade(s) will be to learn how to “funnel down” our data: to make it accessible, and meaningful, in real-time, to our teachers and students.

Wireless Generation (disclosure: an organization where I have sought employment), which designs hand-held devices that can instantaneously analyze student work, and provide feedback to educators, is therefore one my favorite ed tech companies.  But the kind of thinking they epitomize is hardly ascendant in most school districts.  I recently met with a high-ranking official in a New Jersey school district and was surprised to find that she had not ever considered the idea of channeling detailed, data-derived information to teachers on a regular basis.  She understood that data richer than test scores could potentially be gleaned from classrooms but saw this first as an asset to management, rather than an asset to teachers.

The fact is that we don’t respect our teachers enough to think of them as data users and analyzers.  Of course, it’s true that we will have to tailor the presentation of information to the needs of those not accustomed to evaluating classroom data but the challenge there belongs to management: to provide training and to work with developers to create friendly and accessible interfaces.