Notes from Learning at Scale - Day 1
Robust Evaluation Matrix
Doroudi et al. A/B testing? Instructional policy : flowchart
Calibrate a best policy for one student model => Might not be good for other models ! Not try to be good everywhere, otherwise overfitting
Shayan Doroudi, Vincent Aleven, Emma Brunskill of @CSDatCMU on "Robust Evaluation Matrix" #las17ed pic.twitter.com/mer7cZbprM
— ACM Learning Center (@acmeducation) 20 avril 2017
Visual approach
Design space unlimited Solution space underdefined
Compromise between top-down expert et bottom-up data-driven
Their tool, tempr is on GitHub.
Allows to merge actions and see the amount of these buckets of actions in the logs over time (for high learners and low learners), potentially to verify theories.
Tested on a complex interface for learning electrical engineering:
Combining to down and bottom up analyses to understand complex learning at #las17ed. @PhETsims pic.twitter.com/7henvfrUDF
— Ido Roll (@hummus_monster) 20 avril 2017
Next Step Recommendation
Navigational efficiency: jump directly to resource
Use LSTMs (certain type of recurrent neural networks) to model the behavior of a student given their history, and predict the next page on which they will spent more time (in order to recommend it to them).
Do not only train on certified students, in order to get more data.
- Baseline (next lesson): 52%
- n-gram: 61%
- LSTM: 64%
They had to log the visited pages via JavaScript.
A new way for MOOCs to automatically adapt themselves based on learners’ behavior, from @zpardos: https://t.co/eUfVE5bvrV
— Berkeley I School (@BerkeleyISchool) 13 avril 2017
Computational Approaches to Human Learning (CAHL) Research is on GitHub.
Questions
- Why not recommend several pages?
- Indeed we should, more relevant course, more relevant discussion post, etc.
-
For adults: several choices is better, for kids: one choice is better.
- You did collaborative filtering, why not content-based?
-
We should try, yes.
- Why LSTM, not other approaches?
-
Not matrix factorization because there is a high temporal component (but we could do tensor factorization, right).
- What if it hurts? Because you’re not optimizing anything.
- Maybe yes.
Learning to program with localization
Are learning outcomes associated with the presence or absence of localization in educational software?
Did users of Scratch learned programming better if the blocks were in their language?
TL;DR: No.
Source code has been translated in @scratch even if the original project has been built in English. #crowdsourced process #las17ed pic.twitter.com/9pjF6klNZi
— Ella Hamonic (@Ella_Hmc) 20 avril 2017
Statistics
- 20.8M projects shared
- 17M users
- 107M comments
- 3M studios created
See also scratch.mit.edu/statistics
Study done on Portugal, Italy, Brazil, Germany, Norway.
They discovered: for 80% of the users, language does not change.
Measure of learning: block repertoire (~ vocabulary).
Main advantage: localization may save users that might not learn code at all.
Teaching Students to Recognize and Implement Good Coding Style
They built AutoStyle, a tutor that automatically generates hints and feedback regarding the style of Python code.
Goal: Optimize readability of the reviewers.
Goal of the research: see if multiple-choice questions / hints can help acquire good style.
- Use of built-in functions
- Avoid unnecessary control structures
Metrics: ABC Score (Assignment, Branch, Conditional)
- We want to avoid that students code in FORTRAN. At least they should learn some idioms. The question is: Is that medium skill learnable?
How to teach writing code at scale? Commenting, coding styles... how to assess those skills? #las17ed pic.twitter.com/YIflBhtzBN
— Ella Hamonic (@Ella_Hmc) 20 avril 2017
An interesting study of AutoStyle #Tutor with auto generated coding style hints by Eliane Wiese at #las17ed pic.twitter.com/UIDt3t7d13
— Peter Brusilovsky (@peterpaws) 20 avril 2017
Detecting Diligence with Online Behaviors on Intelligent Tutoring Systems
Diligence: Working on academic tasks that are beneficial in the long-run instead of immediate incentives
Academic Diligence Task (ADT) (Galla et al., 2015)
- Study where students can either lay off or work. Surprisingly, it has predictive power on their standardized test scores and whether they will pursue 4-year college.
How to adapt it to adaptive online learning?
TL;DR: Diligence is correlated with success.
- Isn’t this an old-fashioned idea?
- Is it a good idea that I’m being distracted or not?
Epistemic Cognition: A Promising and Necessary Construct for Enriching Large-scale Online Learning Analysis
Additional Explanatory Power for Learner States and Learning Tasks
Add some epistemic constructs in the predicting, knowledge-tracing approaches may add explanatory power.
Additional Insight into for Learner and Environment Development
Involve demographic data in order to improve inclusion and diversity in the tasks provided online.
Epistemic cognition in online learning, emerging and necessary research topic! #las17ed pic.twitter.com/lFSP5gydt7
— carina211 (@carina211) 20 avril 2017