IMDb User Ratings Page Redesign

Hi @ACT_1​ -

If you are not yet seeing the ratings page changes, you should see them shortly!

I think I see a way out of this conundrum.

First though I want to review some critical assumptions.

That the privacy laws and so forth are at odds with storing demographic data on a user's profile.

That demographic data is in fact stored on the users profile in IMDB's database and not stamped on to a movie's vote data row (a database term).

That users, especially top "critics" want to be able to share their votes with others and let others see them. 

That users want to be able to see a demographic breakdown of movie votes across a variety of demographic variables including gender and age. That users have yearned for even more demographic data say for example ethnicity, religion, or politics/class. 

The way out. Store two datasets that can not be related to each other. 

1) Remove all demographic data from a users profile and any table that can be related to that user

2) Offer a client side advanced voting demographic data entry survey experience. This data does not get stored in the user profile or a table associated with the user in IMDB's database.

3) when a user votes for a movie they can choose to fill out an extensive demographic data survey along with their vote. Most importantly they can click save for later and consent to storing these answers on their Browsers cookies. You can sort out whether that should be encrypted somehow SHA256? etc. 

4) this demographic survey data is combined with the movie vote itself but stored in a completely different store of movie rating data that has NO association to a user profile

5) a user when reviewing a movie can click on something to see the demographic survey results 

I believe the correct way to refer to this survey data would be "anonymous" demographic survey data. 

6) You can consider whether you want to mark on a user's vote that they have already submitted a demographic survey. Its only a bit field 0/1. This bit field is stored on the user's movie vote table not the anonymous demographic vote table. If they have previously submitted the survey you can decide whether users should be allowed to submit one again and perhaps how long they should be allowed to wait before adding another one. Say for example flip the bit field every 10 years as a user might shift from one age related demographic tier to another? 

7) a user will never be able to see the demographic data they assigned to a particular movie vote. this data is stored anonymously in a separate data store

8) a user can change their demographic answers stored on their client side before submitting their vote (and they certainly may want to do so as their age demographic changes (slowly) over decades).

9) additional demographic survey questions can be offered in this advanced survey submission and be displayed on the anonymous demographic report. 

10) over time updates to the app can offer these additional survey questions to the user in that survey experience to enrich the survey reporting

So overall the idea is to continue offering rich demographic reporting on movie votes but introduce a "fire wall" between the user data and the demographic data but prevent a user from flooding the demographic survey data set with more than 1 vote over at least say a 10 year period of time. The demographic report would be able to be referred to as anonymous and as such should not trip any privacy laws. You would expect the total number of demographic survey results to perhaps be vastly less than the regular votes submitted without them but the use of the client side data storage should make it far less painful for users to submit demographic survey answers in the future once they have taken the time to select answers to the questions the first time.

I've suggested a path for us to have our cake and eat it too. Keep the privacy fans and lawyers happy and keep the movie collection analysis of demographic bias transparent. The more the demographic reports can offer us ability to see how these movie ratings split along various demographic lines the more we can understand how various bias impacted the movie ratings. 

I think its one thing to complain but quite another to provide constructive criticism by offering alternative solutions. Please consider this path carefully. Thanks for you taking the time to review this proposal as a path forward. I think you will find plenty of users who are willing to fill out the survey once they appreciate that their participation is what allows the anonymous report to add value to their own research. Perhaps a bit of training and marketing emphasizing the benefit of filling out the surveys and submitting them along with the votes. Even 500-1000 users can offer enormous value in an anonymous demographic report. of course the more the better.

The fun part about this path is that you can start enhancing the survey questions offered and offer even more fascinating reporting than we have ever experienced before.

The bottom line is that you have to be quite strict about never associating this survey data to a particular user's id in the database. Once you start down this path, all sorts of doors should open for you to present a very rich and valuable data set to your consumers. You should no longer have to fear legal action if you have approached this from an anonymous survey gathering perspective. Good luck and do not be afraid of the undiscovered country.

@SidneyLeeJohnson​ 

First though I want to review some critical assumptions.

That the privacy laws and so forth are at odds with storing demographic data on a user's profile.

No sorry, that’s not it at all.  It is perfectly possible for us to securely store and manage this data (and we do).  The two issues are that:

1. A decreasing number of customers are willing to provide this data directly (anywhere, this is not specifically an IMDb issue).
2. Increasing consents will be required to maintain the data from the third party account linkage (and therefore see also #1). 

Thanks for the write-up though. 

@Col_Needham​ Can't get age/gender but CAN get country. For some reason this doesn't quite wash.

@Col_Needham​ I'd pay to have the top1000 return.  I don't care about age and gender ratings but I relied on the top 1000 to help when a movie is hyperregional or prone to some sort of brigading.  

Maybe add it as a feature to imdb pro.  I'd also pay for a second decimal point in the weighted ratings. 

@Col_Needham & was ist bitte das Problem mit den Einwilligungen, für Cookies braucht es diese ja auch... & die meisten User wären wohl bereit, diese zu geben... & was spricht gegen die Darstellung der top1000 voters? Die sind/waren ja immer anonym...

Ich werde den Eindruck nicht los, das dies alles Ausflüchte sind! Die Datenschutzgeseze der EU wurden ja z.B. bereits 2018 verschärft!

Bringen sie zumindest die top1000 zurück oder erklären sie mir, was gegen dies spricht!

@cannon​ Thanks for the feedback.  We will miss the demographic breakdowns too, however, we had little choice in this.  A combination of factors forced the decision, but mostly due to declining access to the demographic data itself -- fewer customers were willing to provide their own demographic data directly to IMDb, and similarly access to the same via third-party account linkage was becoming more difficult as various online privacy laws around the world (understandably) require increasing levels of consent. 

This led to a declining percentage of votes being included in the demographic breakdowns and the difficult decision to retire those breakdowns, sorry. 

The breakdown by country for all titles and by episode for TV-series / podcasts are intended to offset some of this inconvenience.  We welcome ideas on other views of the ratings data which might be helpful here. 

Sad. 
I hope you bring it back and add the ability to see the exact number of grades. 
For now it looks like we'll have to look for another service 

@Col_Needham​ 

I understand the demographic breakdown is gone. Indeed I noticed that often the sum of votes in the demo table was far from the total number of votes. 

But please, restore the top 1000 voters. It was very helpful, specially when there was a big gap with respect to the rest of voters.

If you are willing to further breakdown the ratings, a feasible idea could be to breakdown per year of rating. So that we can see the evolution of the rating. That can even be a static view just showing a line running over the years on the x-axis and with the (average or weighted average) rating on the y-axis. Then, if you hover the mouse over the line in a point corresponding to a given year, you also see the number of votes on that specific year. 

That can be interesting because some films receive a very high score on the first years, but eventually, after 10 or 20 years the movie become old and the score declines. Sometimes the style or the theme is very specific to a given historical period, or the movie is kind of 'generational' so that after a while it is even hard to understand why it was a success or it was praised by the critic.

We see for instance that the TOP250 changed a lot over time, not only because of new movies coming in.

The feature of breaking down the rating per year of vote will be helpful specially in the future, as clearly IMDB has votes only from 1996 onward and we cannot know how the Godfather or a Buster Keaton movie was liked at the release date.

@Col_Needham 

Is this a change just in the past few weeks? Even looking at a new show like The Last of Us, you can see that there are tens of thousands of ratings with demographic data. It might not be perfect, but it's still a significant sample size and clearly important to many of the users of your site. I hope you will reconsider.

@Col_Needham​ This was a staggering loss of transparency. I think I would take an opposite path and see if you can go the other direction and require the consent and gathering of demographic data by IMDB users to vote. This is a survey after all and in order to detect bias you need the demographic data. I would even go further and suggest you start collecting ethnic demographic data as well and show the aggregate breakdown transparently. I have long suspected some kind of racism is present in voting and seeing the ethnic breakdown of voting would go a long way to proving that. Now we will add sexism and ageism to the hidden biases that IMDB can't address with a one vote serves all purpose. Circle the wagons and go entirely in the opposite direction. A particular users demographic data isn't visible to other users (as far as I can tell). The data is only used in aggregate reporting. Have privacy laws really gone so far as to prevent bias in survey data from being discovered and reported? Do you really need umpteen thousands of no-demographic voters to be more useful or a richer set of demographic consenting voters? I would take the richer data set any day. I can get the other from RT and MC. I could only get the transparent detail from IMDB and now its gone. VERY SAD DAY.

@gstine99​ The change is rolling out starting this week. Unfortunately, the past volume and availability of demographic data are no indication of future availability due to expiring permissions and on-going privacy changes. 

@Col_Needham​  hi Col, I do like the country by country data but it is a particular shame to lose any semblance of age and gender demographics - it was never about specifics for me but just where something sat on a vague spectrum. Will it remain in IMDBPro? 

@Col_Needham​ Where is TOP 1000 VOTERS? I liked this feature very much.

@Col_Needham please bring back top 1000. This was a much more discerning score than the overall and was very helpful to me. 

@haleywill​ Thanks for the feedback … 

Will it remain in IMDBPro? 

No, unfortunately not.  We simply will not have access to the same data so we cannot use it anywhere, sorry.  We are just waiting for the work to remove it from IMDbPro to be scheduled. 

@Col_Needham​ I have worked in tech. You just make birthdate a required field in user profiles.

@Col_Needham​ This really destroyed the IMDb ratings. If nothing else, PLEASE bring back the Top 1000 voters statistics. Those were very Interesting because it gives an idea of how frequent users voted as opposed to casual users or people who are trying to manipulate a movie’s rating due to either ideological reasons or being involved in the production. Seeing that a movie had, say 8,5 but then 5,0 from Top 1000 voters was very Interesting information. 

I’ve been on IMDb for 20+ years and been Top Contributor several of those and of all the changes made during that time this is by far the worst. 

@Col_Needham​ 

SORRY SORRY SORRY!? UNBELIEVABLE!

@gstine99​ 

I hope the IMDb team will reconsider their decision to remove it.

The issue is that this was not our decision, sorry. 

Could you please make it so that you can see the exact number of votes instead of "3.2k" or something. The exact numbers were helpful. That's not confidential information, is it? It's about user comfort

@Col_Needham​ Not your decision....? whose then?

@klnaco​ This will not be possible, sorry.  Please see our reply above at https://community-imdb.sprinklr.com/conversations/imdbcom/imdb-user-ratings-page-redesign/64220f80ac2f711237475865?commentId=6422251f82212822057e4b28&replyId=6422a9d382212822057e6c99

@Col_Needham​ IMDB's dismissive attitude about this will lead to your going defunct. Not being hyperbolic; it's just true. You-all have shot yourselves in the leg with this nonsensical quality-degrading change. No one cares how people in Sweden rate a film. We want to know by gender and age. If this is a gender issue then allow people to pick Male, Female, and Other

It's pretty clear from this and other changes that IMDb doesn't care what users found helpful or what we'd actually like from the site. They'll give us what they want and tell us how much we enjoy the changes.

I do love this buck passing though. Gosh, BIG TECH can tell when a surfer hunted for obscure information five years ago and still barrage us with ads BUT AMAZON/IMDb can't assign basic demographic data to users? It's really hard to believe. Especially since they CAN tell us the top countries. Something just doesn't make sense.

Additionally, guess they could have decided to do a male/female/not specified type breakdown.

@bderoes​ 

I like the idea of the Top 10,000.

Or even, one might consider the vote of reviewers.

I think it's nice to have more granularity in the ratings. Nationality is certainly not the most revealing variable. Is a UK voter any different than a US voter? I think that it is not.

On the contrary, men and women are different, an 18 yo is different from a 45 yo.

And also, votes change in time: a movie can be valuable when released, but eventually loose its interest (or the other way around).

So for me the point is: do we want to have nice views with useless statistics or do we want to see something helping us to give an a priori evaluation of a movie?

@bderoes​ Thanks for the feedback …

It was never clear whether the age demographic was based on the voter at the age when they voted or the age when you look at the report. 

It was the age when you looked at the report. For example, I have been voting since the ratings poll launched in 1989 (when I was 22) so for a title I have not rewatched (and therefore not revoted) since then, my vote for that title would have moved upwards into older age groups across the years, and finally have been placed in the 45+ age group ever since 2012. 

As for the Top 1000 voters, I hope that IMDb has not only stopped displaying their vote, but is completely ignoring it in their weighting. They either need to expand that to Top 10,000, or dump it altogether. Those numbers were often significantly biased. You don't have to watch a title to rate it, and I strongly suspected that many of those Top 1000 were NOT honest reviewers who appreciate a large variety of genres/content.

This is why the Top 1,000 breakdown has been removed.  You do not need to worry about them having any outsized influence in the overall weighted average, but by definition, we could not filter the ratings from the Top 1,000 breakdown itself, which reduced its accuracy and usefulness. 

For series, create an average rating per season.

Display the actual number of votes, not the abbreviated **k.

Thanks for these.

@SidneyLeeJohnson​ exactly - I am with you! I had no idea this shift was afoot. It is a radical degradation of IMDB. The excuses for making these horrific changes are illogical. I use IMDB demographics ALL the time. I will literally never come to IMDB to find out how people in Turkey rated a film.

I literally created an account to agree with you.

Now ratings are only useful if you're a young white man.

Finding films that appeal to me as ablack woman has now become incredibly difficult and I worry about how things like this will affect the wider media industry.

It's basically further institutionalising white men as the default and resigning everyone else to an other category. In fact it's not doing that, It's basically now saying "everyone agrees that movies featuring POC/not men objectively aren't as good as movies about white men and explosions"

This combined with the genuine sexism and racism in with the review bombing of media heavily featuring people who aren't white or male paints a very dystopian picture.

I'd ask who these changes and this website are supposed to be for but the answer is becoming disturbingly obvious.

@jupiterLILY​ "Ratings are only useful to a young white man".  Why do you single out this group as though they are somehow advantaged.  I'm a baby boomer male so in the past I looked at ratings for the older age range and male. I can't do that any more and miss the feature. What I see in entertainment media is far from " institutionalizing white men as the default". My mind goes to Sunny Hostin on The View who will make any topic about race. Entertainment media is hypervigilant going out of it's way to make minorities, women, and LGBQT folks happy. The issue here is removing demographic filters that we used to have and found useful.  That's all.

How does making up the majority of the voting body not advantage a group?

Somewhat lacking in imagination with that one.

If you're a white dude coming to this website, it's great and it's highly likely that you will not see the problem. You're going to be given a list of movies that appeal to you and all the reccommendations are going to be pretty spot on.

And entertainment media is hypervigilant about making themselves rich. If you think they're trying to make *anyone* happy then I have a bridge to sell you.

There has been a slight push in recent years when studies have realised that not just white men have money, but the vast vast majority of media is still produced by and for white men.

Minorities being able to make some content that appeals to them isn't a bad thing, it's not pandering, it's just people participating in an art form. Not really sure why you're presenting that as a problem.

@jupiterLILY​ 

@nospam77​ 

You know something has been bothering me about this. Why would a forum get rid of what is/was its most popular feature? To borrow from a movie’s tagline let’s follow the money. 
Let’s suppose it’s in a studio’s interest not to have too much detailed rating information out there. As an older male I would avoid movies that were popular with young females. IMDB used to allow me to know this consequently I would avoid seeing  them. 
But if I didn’t know chances are I might go to/rent the movie. 
Please excuse my cynical mind but is too big a stretch to think that  studios might lay on IMDB to withhold demographic information in order to increase viewership? They are THE source of information for IMDB after all and wield a crucial hammer. 

Don't forget that IMDb is owned by Amazon which also now owns MGM -- merged into Amazon MGM Studios. This includes United Artists.

I largely agree with you and I'm cynical for similar reasons. Although, I'm not sure how intentional it is or if it's just like it is in other industries like where it's just the result of not enough of the individuals not questioning their implicit bias. White men are assumed to be the defualt population in science and medicine, it's still not mandatory for them to also use female crash test dummies and don't even get me started on bias in pharmeceutical testing. Why would something as like movies be any different.

Studios definitely have and want this information, they want movies that poll highly with everyone because that equals more money. It feels more like companies are now trying to tell people what they like, instead of finding content that's tailored to them.

Concealing that information from users just reinforces systemic bias though. I found this, and it's from quite a while ago, but it shows you just how many more men are voting on movies than women. Only 2 movies have more than 100,000 votes by women and they squeaked it, the average is 65,000 women voting for every 374,000 men.

Given that they give the impression of being the go to place for movie ratings, that's some really biased data. I know there are similar problems with the countries being surveyed. I'd love to see some adjusted ratings that accounted for the different weights in gender, country etc.

@SidneyLeeJohnson​ Well-stated. I shared similar statements. I cannot imagine what forces led IMDB to compromise their product so profoundly negatively.

The IMDbPro brand is focused on neither the experiences of movie goers, entertainment seekers, movie reviewers nor the audiences of those evaluators, so even invoking the name "IMDbPro" is pointless in this context.

The age and gender breakdown is already available on IMDb Pro, in a different design, and is so far still displayed there.

I think professionals may well be interested in how projects are received by different demographics. (Isn't that all entertainment executives think about?)

Yes, that's a good point, Peter. The demographic data can certainly be of interest to professionals in show business. I was just thinking about how much IMDbPro is so oriented around casting calls, employment opportunities, talent representation, public relations management and, to a small degree, networking. I've observed one person on here, maybe more, who proposed making an ad-free video-on-demand service a feature of IMDbPro, which I thought would be going too far.

@SidneyLeeJohnson​ 

As now noted above, no, unfortunately not.  We simply will not have access to the same data so we cannot use it anywhere, sorry.  We are just waiting for the work to remove it from IMDbPro to be scheduled.

@Col_Needham​ are you passing along to upper level decision makers how horrified people are by this ruination of your product?

Hi @mturulova​ -

You are not seeing the breakdown because the "Rating By Demographic" section has been removed and is no longer available.

Sorry, but this is a really terrible idea.  Why did you do it?

- Ratings by age and by gender were really helpful in deciding which movie to watch.

For example, if some movie has significantly higher rating in older or younger age group, or in female group, it gives you a pretty good picture if you would like it or not, depending which group you are in.

- You removed it and replaced it with ratings for users in specific country (?!?)

How does it help?

Do you really think that _anybody_ might say "Oh, this movie is popular in Turkey or Germany, so that means I might like it"?  Or "well, users in France generally like specific type of movies, so this country rating for France gives me good information what this movie might look like" ?

So, you removed the info that was very valuable to users, and replacing it with something meaningless that has no value for users at all?

Really?

@V7Fabio​ The reasons are covered above -> https://community-imdb.sprinklr.com/conversations/imdbcom/imdb-user-ratings-page-redesign/64220f80ac2f711237475865?commentId=6422251f82212822057e4b28&replyId=6422a9d382212822057e6c99  

We had little choice in the matter, sorry. 

@ron3​ Thanks for the bug report.  We are unable to reproduce this. Do you have a link to the above page and please can you let us know on which browser (including version) and operating system you are seeing this? 

@Col_Needham​ I don't get it--who is "we" and why wouldn't you have a choice?

@Peter_pbn​ Thanks for the suggestion. 

I like the breakdown by country. Otherwise I agree with the comments so far that the old design was more useful, as it provided insight whether a show appealed to young and old, or was perferred by males or females. The gender category might be adjusted to be helpful to a broader non-binary demographic; notwithstanding, the differentiation between those who identify as male/female was very helpful.

I used to consult the breakdown regularly. I miss it.