educational data mining <-?-> psychometrics

I have just returned from my first Psychometric Society meeting (IMPS 2012), which followed close on the heels of my first Educational Data Mining meeting . Both were excellent, intellectually stimulating conferences. I am attaching the slides from my presentation IRT in the Style of Collaborative Filtering. In it, I expose myself to the psychometrician’s revulsion to Joint Maximum Likelihood Estimation, but not without a few defensive slides. I presented similar material–naturally from a different perspective–at EDM.

This is largely why I came to IMPS, and I succeeded in getting a conversation started with Alina von Davier at ETS, who appears to be interested in the same issues: EDM, driven by computer scientists and cognitive scientists with an artificial intelligence bent, is bringing out exciting, new models and learning algorithms that crunch (big) data and are measured by their ability to predict, i.e. by cross-validation.  Psychometricians, fueled by measurement theorists, statisticians and psychologists, worry (rightly) about reliability and construct validity. Especially for high-stakes implementations, they can’t afford to take a risk on a dynamic predictive model that may change the way it classifies a student from one day to the next. They need repeatable models with statistical guarantees behind them. In the middle ground (forthcoming title: When does cross-validation imply construct validity?) there has to be some interesting work to do.

For interesting notes from a grad student who is grappling with the tension between knowledge engineering vs learning-models-from-data in CS education.

Article by: https://zinia.ai/