Duplicate and Redundant Keywords to be deleted or merged

@keyword_expert​ Yes, I realised that we were displaying it in opposite ways after I typed this up.

A couple other pointers: First, one of my rules of thumb (which I do sometimes break) is to only post lists like this if a keyword (or combination of keywords) to be merged is currently assigned to at least 50 titles. Anything less than 50 titles and I feel like I could be doing the mergers myself rather than using up valuable staff time to make these changes as mass mergers.

This is true, but I can't delete keywords at the moment. They just get denied. And it doesn't solve the problem down the line as new people could just turn up and make the same keyword mistakes.

And finally, a lot of the keywords on your list are not true duplicates. For example, "prison" and "prison-film" are not the same thing. One is a film focused on one or more prisons, while the other is a prison that just happens to appear in a film, possibly even in just one scene (without being the focus of the film).

I was under the impression that "prison-film" is basically just a prison movie. Same with the other "-films" in the list. Are you suggesting that "prison-film" is basically used as an AKA for "prison-drama"? The problem there is this doesn't give an adequate way to distinguish TV shows focused around prisons vs. a TV show that just happens to have a prison in it for a time. Keep in mind that "prison-film" literally only has 5 uses, so its usage in the sense that you're proposing isn't really common.

@Skavau​ Are you able to modify keywords, though? That can often be easier than deleting keywords. I sometimes get deletions denied, particularly on "protected" keywords or titles. But modifying keywords is almost never denied.

And yeah, I understand your point about preventing these keywords from being re-created in the future. That is why auto-conversions (as performed by staff) are great, but again, I focus my time and effort (and the time and effort of IMDb staff) on the keywords with the greatest urgency, which typically means keywords (or keyword combinations) with at least 50 titles. Since I have compiled hundreds of keywords in my internal list, there is never any shortage of such keywords that meet my 50-keyword criterion.

Finally, "prison-film" and "prison-movie" are duplicates, but "prison" is not the same thing. 

Are you able to modify keywords, though? That can often be easier than deleting keywords. I sometimes get deletions denied, particularly on "protected" keywords or titles. But modifying keywords is almost never denied.

I will try.

And yeah, I understand your point about preventing these keywords from being re-created in the future. That is why auto-conversions (as performed by staff) are great, but again, I focus my time and effort (and the time and effort of IMDb staff) on the keywords with the greatest urgency, which typically means keywords (or keyword combinations) with at least 50 titles. Since I have compiled hundreds of keywords in my internal list, there is never any shortage of such keywords that meet my 50-keyword criterion.

Well, fair enough. I get the emphasis on larger keywords, but at the same time - if an errant keyword only has 1 or 2 titles attached to it, then it could be quickly dispatched of and blocked before it slowly grows into a bigger problem. Cutting it all out at the root etc.

Finally, "prison-film" and "prison-movie" are duplicates, but "prison" is not the same thing. 

Would you say that "prison-film" and "prison-movie" are duplicates of "prison-drama"?

@Skavau​ 

Would you say that "prison-film" and "prison-movie" are duplicates of "prison-drama"?

I would not agree with that proposition. For example, there are documentaries that are prison films without being prison dramas. And many documentaries are not dramas.

https://www.imdb.com/search/title/?title=prison&title_type=documentary&genres=!drama&keywords=prison

@keyword_expert​ So "prison-documentary" for distinction for prison documentaries?

How would you distinguish a TV show based around a prison setting vs. a TV show that shows a prison?

@Skavau​ I also keep a private internal list of duplicate keywords with 40 to 49 titles. Even that is a sizable list. My plan is to monitor those keywords to see if they ever get over 50 titles, and then post them.

I understand your point about cutting down keywords with fewer titles before they take root, but the same thing can be said about keywords with 50 titles: they can be re-created unless they are set up for permanent auto-conversion by IMDb staff. And they are much more likely to be re-created, since we already know from their numbers that they are popular. 

Overall, for me it's just a question of efficiency. There is simply a greater "bang for the buck" of merging a title with 50 keywords versus 5 keywords. 

Fair enough.

@Skavau​ 

How would you distinguish a TV show based around a prison setting vs. a TV show that shows a prison?

The former gets the keyword "prison-show" and "prison," while the latter just gets the keyword "prison."

Edit: corrected to mean "prison-show," not "prison-film."

It doesn't seem intuitive to identify a tv-series as a film, and given that "prison-film" has 5 uses in total, and "prison-movie" seems to have none - I'd argue that it's clearly not seen that way.

@Skavau​ That was a mistake on my part. I meant to say "prison-show," not "prison-film."

prison-show (5 titles)

Well, fair enough. What is "prison-drama" to you then?

@keyword_expert​ I can modify keywords, perhaps, it goes into the queue for processing. Every "processing" I've ever had has been declined so far though.

@Skavau​ "prison-drama" is a title focused on life in a prison that also meets the genre definition of a drama:

By the way, "prison-drama" is listed as an accepted "subgenre" keyword here:

https://help.imdb.com/article/contribution/titles/keywords/GXQ22G5Y72TH8MJ5?ref_=helpart_nav_30#

I suppose I don't really see the descriptive difference between "prison-film" or "prison-show" and "prison-drama" on the assumption that "prison-documentary" exists (which it should do). It just makes for more tag splitting.

@Skavau​ Well, this is another specific criticism of a subgenre keyword. The subgenre keywords have been the subject of a lot of complaints. Just search on this forum for "subgenre keywords":

https://community-imdb.sprinklr.com/conversations/search?q=subgenre%20keywords

Indirectly related, but I made a thread about "drama" subgenres.

@Skavau​ For your questions about films or shows set in prisons, I just came across these keywords:

set-in-prison (2 titles)

prison-setting (4 titles)

life-in-prison (19 titles)

jail-setting (1 title)

@keyword_expert​ Well "prison-drama" seems to be an officially endorsed term by the site.

@Skavau​ Of course, not all titles set in prisons are dramas.

Are you referring to prison documentaries? "Prison-series" or "Prison-show" would perhaps be somewhat better, but IMDB needs to set out (or allow us to brainstorm guidelines so we can make coherent distinctions).

like, even with 'detective' - I wouldn't say "detective-protagonist" works because I know of a TV series where technically the one of main characters is a detective... but the show is about surviving a zombie pandemic. 

@Skavau​  Several different genres can be set in prisons: horror films, romance films, even comedy films, for example.

Ah, fair enough. Well I guess in a TV series context they're all prominently dramas. But probably should add "Prison-series" to all of them additionally.

@Skavau​ In researching the "prison" keywords, I came across some pretty bad keywords. These are bad because they are vague. Some of them mean "locked-in-a-room," some mean "incarceration," and some mean "captivity." I modified a few of them, but only made a small dent.

locked-up (65 titles)

lockup (34 titles)

lock-up (13 titles)

Honestly I don't think it should be on us to trim these errant keywords. Someone else will just add them back.

We should discover and then, frankly, the system should merge and then automatically reroute the old keyword when someone else inevitably attempts to add it again.

If you mean they're used inaccurately, then sure, I agree. But that goes for any keyword on the site being honest.

There's a whole mess on here.

@Skavau​ It depends on the keyword. With keywords like this that cannot be mass-merged (because they could mean more than one keyword), sometimes it is possible to "audit" the keywords, trim them down to zero, and ask IMDb staff to block the keyword from future use. Or sometimes if a large percentage of instances of a keyword mean one specific other keyword, it can be possible to modify all other instances of the keyword and then ask staff for a mass merger (and then block the keyword). But in other cases it isn't worth the time or can be impossible to do. Again, it all depends on the keyword in question.

Fair enough. I agree that the vagueness of them means they should be blocked entirely though. But sometimes you simply can't know about what the keyword is referring to without watching the film or show.

@Skavau​ Yeah, it's usually not possible to figure it out completely for 100% of titles.

Here are some examples where I was able to figure it out:

A few vague keywords that are ready to be merged and/or blocked (trump, roosevelt, washington, vancouver)

Do they ever freeze keywords even if they have some uses? Like does it have to go to zero before they can freeze them?

@Skavau​ They have the capability to mass delete, mass merge, block keywords from future use, and establish future auto-conversion (which automatically converts keywords when they are added in the future). They can do all of these things regardless of the number of instances of a keyword. 

I do believe that "future-time-travel" is not necessarily a synonym to those other keywords. As much as I would like this keyword to mean the same thing as "forwards-time-travel," I also recognize that in at least some instances it might mean time travel that takes place in the future. For example, it might be a movie made in today's time, with a plot that originates 1,000 years from now, where time travel takes place. In other words, it might be the "future of time travel" for some titles, while "time travel into the future" for most other titles.

I suppose that "future-time-travel" could mean something different from "forwards-time-travel" in some cases. But it's still a keyword that reads in an odd way to me. It's not obvious from reading it what the specific intent is, and in my opinion, that makes it a keyword that should be removed. But that is a separate discussion.

Also, "zombie-sex" is not necessarily the same thing as "human-zombie-sex." The former might be two zombies having sex with each other.

This is true, and it's so hilariously specific.

Even "zombie-bite" is not necessarily the same as "bitten-by-a-zombie." This is a close call, but I regard "bitten-by-a-zombie" as a keyword for titles that show the zombie taking the bite, while "zombie-bite" might be best used for titles where the bite mark is shown on a person (oftentimes without showing the act of biting). 

I think in this case you might be reading too much into the intent behind it being posted. If "zombie-bite" is specifically used to refer to an instance where a bite is uncovered on a person, as opposed to a bite happening, then it is simply not obvious that it might be applied that way.

"bitten by a zombie" vs. "revealed zombie bite" or "offscreen zombie bite" would have more clarity, and I would support "zombie-bite" being merged into that.

The police officer keywords are an entirely different story. They were the focus of a massive manual editing campaign by @DataOrganizer and he spent so much time on it that he got burnt out and took a permanent break from IMDb. As a result of all of that, the police keywords are in a serious state of disarray, and there is some improper English grammar and poorly formed keywords, like "female-police-officer-knockout" and "male-police-officer-deceased" just to name a couple. I think these keywords deserve a separate, future post all on their own. That has been on my list of things to do eventually. Stay tuned....

The problem with "police" and "cop" so far as I can see is that IMDB needs to choose between "cop" and "police" as nearly every "x-police" has an "x-cop" parallel. Once all they've all been merged together into one, it'll be somewhat easier to spot the duplicates.

"Police" seems to have more general usage, where as cop is a very Americentric term.

@Skavau​ 

I dislike the keyword "future-time-travel" as well. It's annoying and I agree it shouldn't exist.

I totally see your point about "zombie-bite." Like I said, this is a close call. But another one of my rules of thumb is, when in doubt, don't mass merge. Because mass mergers cannot ever be reversed.

I forgot to mention earlier that another one of my rules of thumb is to allow at least a 7-day comment period on proposed keyword mergers from fellow contributors before asking staff to act on a list of keywords.  Because again, these mass changes cannot be undone. Occasionally I have made mistakes in my proposals that are not caught before the merger goes through, and each time that happens I regret it. But it is a learning experience that hopefully leads to fewer mistakes in the future.

And yeah, "cop" versus "police" is a related problem that should eventually be dealt with. That was part of the work of @DataOrganizer.

@keyword_expert​ But we also have "space opera" and "alien-invasion" without "sci-fi". They are both by definition sci-fi without the suffix.

I don't get the logic there

@Skavau​ Not really. It is possible to have an alien invasion in a title (for example, as a brief part of a comedy TV episode) without dominating the plot so that it makes the entire title fit within the Sci-Fi genre.

Well that's possible, but alien-invasion simply has much more usage than "alien-invasion-sci-fi".

And it makes even less sense for "space opera" - which is a long-established science fiction genre (without reference to the 'sci-fi' suffix). You can't just have "space opera" as a background reference in an episode. You either are or are not a space opera.

I would encourage you to post your criticisms of "space-opera-sci-fi," "alien-invasion-sci-fi," "prison-drama," etc. in this thread:

https://community-imdb.sprinklr.com/conversations/data-issues-policy-discussions/subgenre-keywords/61d477ea59e8f360a0ce4e18

Looks like they did it to help new users, not for emphasis. So they wanted to encourage "alien invasion sci-fi" not because they want to distinguish between a film that has alien invasion as a small part of its narrative vs. a film chiefly about an alien invasion, but because they want new users to able to find the tag easier on the keyword search.

Really weird.

@keyword_expert​ Problem is that it's just going to cause more tag confusion.

@keyword_expert​ You know IMDB needs a hierarchical system, where if someone put in "girl-detective", the system would autoapply "female detective" as a parent tag.

This would solve a lot of problems.

@keyword_expert​ This is why I don't really distinguish between a keyword with 100 uses vs. 2 uses. It's still a duplicate and shutting that path down and merging it into another is still useful.

Honestly I think the answer is, if IMDB staff are too busy/disinterested in organising this is to empower you, me and others with keyword (just keyword) merge/deletion/block powers. Plenty of database sites use volunteers for things like this.

@Skavau​ 

This is why I don't really distinguish between a keyword with 100 uses vs. 2 uses. It's still a duplicate and shutting that path down and merging it into another is still useful.

I still disagree with you on that. There is a major difference between a keyword with 100 uses vs. 2 uses. Based on the numbers alone, a "bad" keyword with 100 uses has already been demonstrated to be very popular, and therefore much more deserving of a permanent solution. I have seen from experience many popular keywords like this be mass-merged but not set up for auto-conversion, and then the keyword just gets created again soon thereafter. This message board is littered with old posts from years ago proving that this exact phenomenon happens with popular keywords. It is much less likely to happen with a keyword with only 2 uses.

Honestly I think the answer is, if IMDB staff are too busy/disinterested in organising this is to empower you, me and others with keyword (just keyword) merge/deletion/block powers. Plenty of database sites use volunteers for things like this.

Believe it or not, there has actually been some discussion between IMDb employees and volunteers (not on this message board) of setting up some kind of arrangement like this. I don't know if it will ever happen, though.

@keyword_expert​ Well if we could just do it ourselves, we'd honestly soon clear through them through merging and deletion. It's only a priority to focus on more commonly used akas because 'staffs time is precious'.

@Skavau​ True.