Which comes first, the Theta or the Q?

As I continue to try to reconcile EDM community thinking (largely influenced by the CMU cognitive community) with psychometric thinking, the more I suspect a philosophical roadblock. Q-matrix aka rule space theory was indeed influenced by AIED and diagnostic modeling ideas and has subsequently been reabsorbed by the EDM world as a stand-in for item response models and psychometrics as a whole. Which is odd, since the same cognitive EDM community is generally not thinking in terms of latent trait models, but rather in terms of mastery of some O(100) individual domain skills. In other words, from an increased likelihood of success in answering a question correctly, one makes an inference about which skills the examinee has mastered not an inference of how skilled the examinee is. (This reminds me of the distinction between guilt and shame…)

Since “the truth” is some description of the knowledge space in terms of knowledge components, the cogEDM practitioner want to assess the value of the Q-matrix, perhaps improve it (in the same vein as dividing a KC into subKCs), but learning the Q-matrix from the data is a bit disingenuous since the number of KCs must jibe with the philosophical assumptions of how learning works. Learning a Q-matrix with only 2 factors is never done.

Latent trait modeling tends toward the opposite extreme, perhaps due to historical reasons: one might typically assume that one overall skill should explain all the observations unless the model-data fit is really bad, in which case consider adding a dimension. The latent trait model serves the purpose of explaining the variance/covariance in the observed data, and therefore the more parsimonious the model, the better.

We may know that a domain expert would classify the questions on an instrument into 4 or 5 groups, but that doesn’t mean of course that 4 or 5 skills (KCs, whatever) are necessary to explain student response data. Of course if a single factor could explain the data, and if the items, thus parameterized do not cluster at all, then we might safely say the expert is just plain wrong. A compromise however is that it is possible to find 5 clusters with only 3 skills, just for example. Which brings me to the question in the title of this post.

Post by : https://zinia.ai/