Add music videos to the IMDb Dataset title.basics.tsv.gz

@Michelle​ Hi

Have you heard anything back about this yet?

Thanks!

Hi @brianrisselada​ -

Thanks for this additional comment, I have added this to the ticket.  I can confirm that the ticket is currently still being researched, as soon as I have an update I will be sure to post the details here.

Thanks again for your patience!

Hi @brianrisselada​ -

The ticket is still being investigated by the applicable tech team.  As soon as I have an update on the status I will let you know here.

Thanks for your patience!

Hi @Michelle​ 

I hope you don't mind me just checking in again to just confirm if there have been any new updates on this or time estimates for fixing.

Thanks.

Hi @brianrisselada​ -

I just checked the ticket status and can see that it is still pending review by our tech team.  Unfortunately, I cannot provide a timeline as to when it will be resolved, but I have given a nudge on the ticket to see if I can help speed up the investigation.

Thanks again for your patience!

@Michelle​ Thank you!

@Michelle​ Just popping in again to see if you think it would be helpful to give another nudge on this ticket since it's been a couple more weeks now.

Hi @brianrisselada​ -

I reviewed the ticket and unfortunately, it is still pending review.  Unfortunately I cannot provide an exact timeframe as to when it will be handled by the tech team, although I will continue to nudge the ticket for progress.

@Michelle​ I appreciate nudging it.

It is frustrating that this data that has been freely available that we have been using has been missing a bunch of information for a while now.

@Michelle​ Just checking in again.

I just downloaded the latest file again and the data is still missing.

Do you know why this is taking so long or why it does not seem to be a high priority?

I like that this new category of Music Video has been created, but to provide these files but not have all of the titles showing up in them makes the files useless in many ways. And it has been like this for several months at least.

Thanks for your help.

@Col_Needham Do you have any insight into this?

Thank you.

@Michelle or anyone else? Is anyone reading this?

Is there any other way to easily extract this information?

@Michelle​ Thought I'd just check in again to see if there is any movement on this.

Thanks.

Just checking in again. Haven't heard from @Michelle in over a month so just wondering if any other employees know anything about this.

@Bethanny @Maya @Sally @Fran @Edward @Jon 

@brianrisselada​ It seems we hear from IMDb staff less and less these days on this forum. I am waiting for numerous pending posts to be resolved by staff.

@keyword_expert​ Thanks for your comment. I actually was wondering if it was just this issue that was no longer being given attention or if it was happening more in general.

I'm not sure what else to do. I seem to have been abandoned.

I guess I'll see if there are any other ways to contact employees at IMDb other than through this board.

If anyone reading this is aware of any other ways like a phone number or email or other people who work at IMDb that might be able to help, please let me know.

Thank you.

@brianrisselada​ 

I recently received this response from @Michelle on this thread:

Again, I apologize for the delay on handling these larger-scale Keyword updates.  After this coming US Holiday we are hoping to systematically handle these older requests on a more consistent basis.

@keyword_expert​ Thank you for sharing.

What IMDb staff is encouraging with having an incomplete dataset available here is for people to start mass scraping the site for data now. I know they don't like data scraping on their site but if they don't provide the data any other way that's about the only way available to get it.

@Michelle​ It's been three months since I've heard from you. Can you give me a status update on this?

Thanks.

@keyword_expert​ Thanks for asking. Let me clarify.

The "Music Video category" I'm referring to is neither a genre category nor a keyword category.

Rather it's in the category that IMDb seems to define as a "Title Type"

For more clarification, the list of categories under Title Type is short enough to list here (you can see them in the Advanced Title Search form https://www.imdb.com/search/title/):

• Feature Film
• TV Movie
• TV Series
• TV Episode
• TV Special
• Mini-Series
• Documentary
• Video Game
• Short Film
• Video
• TV Short
• Podcast Series
• Podcast Episode
• Music Video

The category of "Music Video" was recently created. However many of the titles that are currently designated as this category have been in the database for a long time, but used to be designated in other categories in the past, such as "Short Film" or "Video".

The following search URL shows the titles that are missing from the file: https://www.imdb.com/search/title/?title_type=music_video

@brianrisselada​ Yes, we have been discussing this issue (and whether it affects the total title count) in this thread as well:

https://community-imdb.sprinklr.com/conversations/data-issues-policy-discussions/whats-going-to-happen-when-imdb-reaches-10-million-titles/5f4a7a008815453dba94e72f?commentId=5f4a7bad8815453dbae2c126&replyId=6233848948f90a6784748434

@keyword_expert​ Do you know anything else about this issue? I haven't heard back from anyone in a while.

I don't know anything except what has been posted in this thread.

@keyword_expert​ From your experience do many people use these dataset files? They don't seem too concerned with fixing this very quickly. I thought this was a resource that a lot of people used.

@brianrisselada​ I really wouldn't know; I only learned of the existence of these files around the time you posted this thread.

@Michelle​ Apology accepted. Thank you for the response.

If I don't hear back after a certain amount of time, is it OK if I check in again? If so, what would that amount of time be?

Thank you!

@Michelle​ It doesn't seem right that @brianrisselada should have to wait more than a year to learn anything about why this problem exists and what can be done to fix it.

@keyword_expert​ Thanks.  I just wish I knew if it was an issue of difficulty or prioritization.  It's hard to imagine why it would be so difficult to fix, but of course I'm no expert. If it's not that difficult, why isn't it a prioritization as this used to be a feature that the staff seemed to emphasize. But maybe no one else is using it, or they don't care that it's now missing data?  I just wish I could get some more info so I knew what to expect and how to keep planning going forward regarding this.

@brianrisselada​ It feels like it is taking more effort for staff to read this thread and provide non-update updates than it might take to just fix the problem. Very frustrating.

@keyword_expert​ I just downloaded the latest file and it is still not fixed. Music videos are still not appearing in the file.

Hi @brianrisselada​ -

I am profoundly sorry it has taken over a year to confirm the status on this issue.  I am personally escalating further to the team responsible to confirm either if there is a planned fix or if the datasets will remain as-is without Music Video titles.

@Michelle​ I am glad to hear from you! I am incredibly grateful for you following up and and escalating the issue.

Hi @brianrisselada​ -

I have just confirmed with the applicable team manager that due to several factors, including scalability and company focus, unfortunately we will not be updating the IMDb Dataset title.basics.tsv.gz to include Music Video titles.

Sorry it has taken so long to provide you with and answer, especially as it's likely not what you were hoping to hear.

@Michelle​ That's extremely unfortunate as the datasets are now incomplete and actually lacking titles that used to be there before.

Contributors were responsible for this data and yet the amount of it they can get back keeps shrinking and shrinking over the years.

@Michelle​ Would I have access to this data if I were a top contributor?

I think, the folks who run IMDb are worried about rogue websites (and beyond) harvesting data from IMDb. Regardless, the mitigation observed over the past half decade may in turn be exacerbating the problem.

@jeorj_euler​ if that were the case then why not just get rid of sharing the data at all instead of advertising it as a resource for contributors but then not fixing it after they make changes that break it?

So, as not to lose too many contributors on the crowd source end. The amount of utilization of particular data types is also taken into consideration. Public access to underutilized ones tend to be locked off if not the data itself being discarded entirely.

@jeorj_euler​ well that data used to be there. All that happened was they added a new type but apparently didn't update their program that spits out the dataset files to include it. I can't imagine it would even take that much time to update. They don't even care about giving the smallest amount of time for our sake though.

@brianrisselada​ 

It is disappointing to see IMDb staff reward your diligence and patience with complete disregard of the problem.

They have been doing the same thing on keyword problems the past few months. They even  wrongly closed most of my open threads on this message board, including threads where they haven't even yet decided whether to take action.

All of this is really disappointing, and it has caused me to become disenchanted with the IMDb website and this forum in 2023.