Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve our rating average algorithm and display #1602

Open
Josh-Cena opened this issue Feb 12, 2024 · 2 comments
Open

Improve our rating average algorithm and display #1602

Josh-Cena opened this issue Feb 12, 2024 · 2 comments
Labels
Catalog Course relationships Related to the ongoing research about course graph; https://docs.google.com/document/d/1mIsanCz1U3M6 difficult enhancement New feature or request New data structure New information that's currently not present in the catalog data. Probably needs Ferry/API updates

Comments

@Josh-Cena
Copy link
Member

Josh-Cena commented Feb 12, 2024

Following our discussion with Miki, the average ratings for many courses do not fairly represent the quality of the instructor/course. This is especially true for average instructor ratings, which tend to "stick" because for courses we only calculate average within the same instructor where possible. For examples of unfair instructor ratings:

  • A junior instructor got a poor rating in their first/second semester of teaching but improved later; there's no reflection on this general trend
  • Professors who also teach intro courses (service courses is a tradition in the math department) tend to have lower averages than those who only teach electives because of different audience backgrounds/expectations, etc.

I think there are several things we could do here:

  1. We should weigh ratings by recency. Courses taught more then 2–3 years ago should be weighted drastically less in the average.
  2. Miki pointed out that weighing average by enrollment (which we planned to do) actually puts intro course instructors at a disadvantage because those courses have more students and those students tend to be more unhappy.
  3. We need to have explicit documentation on how these averages are computed.
  4. If possible, whenever we show an average, we should show other statistics such as sample size, standard deviation, and maybe display a graph against time to better put the number into context.
@Josh-Cena Josh-Cena added enhancement New feature or request Catalog difficult New data structure New information that's currently not present in the catalog data. Probably needs Ferry/API updates labels Feb 12, 2024
@bearsyankees
Copy link
Member

I also would add I think we should give the user a choice here in what data they are seeing, whether that be a toggle for weighted ratings or not, or a slider for how far back they want data to go, or other options

@Josh-Cena
Copy link
Member Author

Sounds really, really advanced for most users, I guess...

@Josh-Cena Josh-Cena added the Course relationships Related to the ongoing research about course graph; https://docs.google.com/document/d/1mIsanCz1U3M6 label Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Catalog Course relationships Related to the ongoing research about course graph; https://docs.google.com/document/d/1mIsanCz1U3M6 difficult enhancement New feature or request New data structure New information that's currently not present in the catalog data. Probably needs Ferry/API updates
Projects
None yet
Development

No branches or pull requests

2 participants