IMDb Data – Now available in Amazon S3

In regards to usecases, a lot of things that cannot be achieved through the advanced search tools, can be achieved through downloading files like certificates.list.gz, keywords.list.gz, so on and so forth, and then processing the data found within them.

So, I'm now in the position of urging for a much more comprehensive data set to be made available or for the advanced search tools to be more advanced. In addition, basically, Luca Canali is right. Amazon/IMDb is taking things away from us. Whatever it is that we will be gaining in return remains to be seen.

That is a huge drawback for me. I'm not only a film buff, I also write articles about old and rare films for a non-commercial blog, and I do extensive research for that purpose. I'm using the IMDb list files for more than 15 years, and I download the updates almost every week (in the first years the diff files, then, since I had DSL, the list files). I wrote a script (which makes use of wget) that determines if new files are available, so the download is started automatically.

And yes, I need ALL list files. Not every file every day, but every file from time to time. I'm using the program AMDbFront (don't look for it - it has disappeared from the Internet since its author didn't develop it any more) to convert the files into a MySQL database. AMDbFront is also the viewer for the data. I'm using it in GUI mode almost daily, but sometimes I make complex queries using SQL. One example: A few years ago, scientists from Northwestern University developed a method to determine automatically which are the most culturally significant films (the winner was THE WIZARD OF OZ), and they used the IMDb list files for that purpose (movie-links.list in particular). Here is their paper:

http://www.pnas.org/content/112/5/1281.full

I managed to reproduce their most important result (the long-gap citation count) with my local IMDb data, using a SQL query I wrote. The cited article only covers US films, but I used the method to create respective lists for many other countries, and I published my results in the above-mentioned blog.

I also wrote a script (in VBScript) which adds a table to the MySQL database that contains all films I have on DVD or Blu-ray. The table contains the title (exactly as it's in movies.list) and flags for seen/unseen, region code and short/long films. That information is taken from a text file I maintain for that purpose.  With appropriate SQL queries, I can answer questions like "how many short films from France from the 1930s do I have on DVD" or "who is the actor/actress with whom I have the most films on DVD"?

Well, this all will become impossible with the new dataset format. I surely won't switch to it (even if it would be free of costs), but I will freeze my installation at the current state. That's the lesser of two evils for me.

Thank you Mr Needham, this is indeed reassuring, as mentioned before.
The HTTP access should be a good alternative way to obtain the data i guess.
And i'm sure that adding the AKA titles will help a lot of users, including myself.
Will by any chance, the languages of the movies, be included as well ?
Because AKA titles are mainly important when dealing with non English movies, but i think there is no possibility with the new data files to determine which movie is English or non-English.

This reply was created from a merged topic originally titled Proper channel to request authorisation to extract and use information for resear....

I was wondering if there is a channel I could ask IMDB authorisation for using IMDB movie synopsis data in my thesis.
I am aware of the 'IMDb Data – Now available in Amazon S3' announcement, but I was not able to find an interface that would publish movie synopsis.

Your response is greatly appreciated. 

Best regards,

"In our continued effort to best serve our Contributors, we are streamlining the datasets and making them available in a more useful and structured format in S3" == "on a short notice we are destroying many projects and initiatives that were using our data, by withholding most of it and providing the rest in a new incompatible format"

I can understand why a transition needs to be made and that its not easy to achieve parity in the new system. My use case is a little different than the others on this list so I thought I'd chime in.

I do economic research on the television industry; asking questions like how characteristics of production companies affects the quality of the show. To that end I use a bunch of data that's not included in the new subset on S3. AKA titles definitely as mentioned by others; they are useful for matching across different datasets. The distributor and production company lists files helps me track which shows were on which networks as well as affiliated with each production company. Producer and writer lists lets me for example connect a show with an Emmy winning executive producer or creator. Lists like language and runtime help me screen out noise from the data, especially for shows that were not very popular and may be incorrectly labelled in other variables. And the full set of genres is important to capture all the shows in a category; if a show has more than 3 genres it may not be included if for example I try to understand what is relevant for ratings in comedies. For ratings, knowing the distribution of ratings is useful to understand how targeted a show was.

Anyway I hope that is useful. Happy to expand on this more if it would help with your triaging of features.

I appreciate that Col and sv are stating that they want to hear from us, our reactions to this, and how we have been using the data, so that is why I am sharing this.

I just checked and I have submitted 1,014 updates to IMDB, which I suspect would appear to be a pittance compared to the top contributors, but I hope it is still evidence that I care about the completeness and accuracy of the IMDB database.

One of my favorite things on the IMDB site is the advanced search, but I wanted to do so much more and I wanted to integrate the data with my own custom algorithms and personal logs. I discovered the .list files provided by IMDB and this has been my basis for an exciting adventure. I am not a professional IT person or programmer. But to do what I wanted I taught myself Access and eventually SQL and Python and Django. I only download the files again about every year because I don't watch a lot of new films but I love adding and updating the data for older and more obscure films.

I'll admit that when I first started, even though I didn't have any database or programming experience, I found the .list files seemed antiquated, but I wrote my own programs to extract the data and place them into an SQL database to work the way I wanted it to. So I am kind of glad that IMDB is moving to some new structures that will hopefully be easier to use even though it will probably mean countless hours for me to rewrite a lot of my program for getting the updated data into my database.

The thing I am most concerned about though is making sure all of the data is still available. I use almost all of it, and what is currently in the S3 files would not be worth me using anymore. So I am glad that it has been indicated that there may be means for the rest of the data to be available still. I will list the information below that I use and how I use it.

What I use the most:
Movie (with stats like release year, number of votes, and rating)
AKA
Company (production and distribution)
Country of production
Genre
Keyword
Language spoken
Person (all persons of all roles, and all of their credits on all movies, but most of all directors)
Running time

What I also use often:
Aspect ratio
Certificate
Cinematrographic process
Color
Film negative format
Printed film format
Location
Movie connections
Release date

What I also use but rarely:
Camera
Film length
Laboratory
Sound mix

Also I used ALL OF THE ATTRIBUTES of these items

All of this is strictly for personal use. It is primarily for me to log films I've seen and track data about them and to run interesting queries in the database to find interesting results and patterns within the database regarding connections between all of these stats listed above. However it also can result in me identifying information that needs to be updated in the IMDB database that I can then submit updates for. I have had thoughts before about some day creating something available for public use, but that may be a pipe dream, and if I did ever do it I would of course pay for the license to do so.

So I hope that through this transition that all of this information will be made available in a complete and easy way for contributes like me who wish to have the information.

One other thing I'd like to request: the most useful thing you could add to the datasets you make available would be for movies and for the people to list the IMDB ID used in the IMDB URL for each of the movies. For instance in the dataset for movies for the entry for "The Godfather" it would list the ID as tt0068646 which corresponds to the webpage for "The Godfather" which is http://www.imdb.com/title/tt0068646/. Or for instance for the person Alfred Hitchcock the ID nm0000033 which refers to his page at http://www.imdb.com/name/nm0000033/

Thank you for considering my situation. 

The switch, in its currently implementation is horrible.

IMDb is a community-driven website that relies on the mass of users for nearly everything, from reviews to ratings, to episode and movie release dates yet somehow most of those things are missing from the data dumps. You owe it to the community to give back and 'complete' data dumps in the form of an S3 bucket where the devs pay for bandwidth is the least you could do.

The tens of thousands of users that left reviews or ratings didn't do so for the benefit of a corporation. We contribute information to large repositories like wikipedia or IMDb because we want people to have access to it, and we do so hoping that the gatekeepers will do their best to keep all out there and easily available...but instead you guys have gone the opposite way. Everything needs to be accessed through your interfaces or apps, what you do give back is anorexic in comparison to what you take, and yet you still rely on users to feed you information for your business model to even work...

I urge you to seriously reconsider this philosophy or at the very least have a moment of honesty with the developer community and explain yourselves better. There is no reason to have omitted all of this information and I'm starting to think that there is also no reason to contribute or rely on your website.

You guys have spent the past 30 years harvesting your users for data while providing decent dumps of your database, and now that we've all learned to rely on you guys, you're taking that away. Take a page from Google: "Don't be evil".

Hello,

Thank you for your continued feedback. As we review your feedback and work on the HTTP solutions described by Col and sv above, we are revising the shutdown date of the IMDb FTP sites to December 28, 2017.  

More updates will be provided closer to that time. Thank you.

Chris

This reply was created from a merged topic originally titled Flat files/data sets availability ?.

Hi,

Could some IMDb staff confirm when will the last batch of flat files/data sets be issued and made available on the FTP sites?

The Berlin one still mentions 2017-09-10, a Col Needham post mentions November, 7, and the latest set available was issued on November 24...

Thank you in advance for some clarification of the roadmap!

V.

I have been accessing the Amazon S3 for a few months and am now getting "Access Denied" from the S3 bucket, started this week. I see no one upset here, is this just me? How do I resolve this?

"This more robust and reliable solution will replace the IMDb FTP sites, which will be retired on December 28, 2017."

How about actors.list.gz? It is not updated after Sep 22 2017?

-timon

This is my first look at the data IMDb are now providing and I'm extremely disappointed. Maybe I've just been spoiled for the last 20 years with the depth and quality of data IMDb have historically provided. Sad to say, I foresee lots of screen scrapping in my future :-(

The final build of the data that gets published to the FTP mirrors occurred yesterday so those mirrors contain the final FTP snapshot.

Will that final build be pushed to the FTP site(s)? The dates on most of the files are from several days before that final build.  Thanks.

As I only updated my local cache of the IMDb data dumps twice per year, I am just now learning of the retirement of the FTP dumps.  I have been using these dumps for years for personal use, mostly for data mining and to create lists of movies that I want to track down on cable.

I've invested a tremendous amount of time in creating code to parse these dumps.  At my current bill rate, it's tens of thousands of dollars.  As a film enthusiast and software professional, these dumps were a great way to do both at the same time.

While this is very sad for my home, for-fun project, I think it is unfair to think that Amazon is doing this as a money grab.  Amazon has world class data infrastructure in AWS, and it's only natural that eventually they would want to move people onto AWS and away from legacy systems that were built in the 90s.  I don't know much about the internals of IMDb, but I would expect that the system that created the FTP dumps is at least a couple generations away from the system that feeds IMDb.com.

That said, are there any tutorials on specifically getting IMDb data through S3?  I am pretty comfortable with the programming involved, but have not worked with Amazon S3 before.  imdb.com/interfaces states the entry point is https://datasets.imdbws.com/, but I need details on how to construct the SOAP or REST calls.