sv_6654070's profile

15 Messages

 • 

820 Points

Thu, Jun 29, 2017 8:49 PM

IMDb Data – Now available in Amazon S3

This is an announcement for customers of the IMDb bulk data available via FTP.

We are pleased to announce, starting today IMDb datasets are now available in Amazon S3 via an HTTPS link. Using the new interface, customers can bulk-access IMDb title and name data.

For details on the S3 solution, file format and access guidelines, see www.imdb.com/interfaces.


In our continued effort to best serve our Contributors, we are streamlining the datasets and making them available in a more useful and structured format in S3. Notably:


  • Data refresh frequency is now daily (previously weekly).
  • IMDb title and name identifiers are included in all the files for ease of matching and linking back to IMDb.
  • The files are in tab separated values (TSV) format.
  • The sets of data we provide are updated to only include the essential ones that help with matching and linking to an IMDb title or name.
As part of housekeeping the FTP site, the data files will no longer be updated. The list data files will continue to be available at two locations (see below) until February 28, 2017. We strongly encourage FTP site users to switch to the S3 solution at the earliest to ensure their applications continue to work without interruption.

ftp://ftp.funet.fi/pub/mirrors/ftp.imdb.com/pub/frozendata

ftp://ftp.fu-berlin.de/pub/misc/movies/database/frozendata

 If you are not an IMDb Contributor and wish to obtain IMDb content for commercial use, we offer a content license.  The license grants you access to our content via an XML web service, plus the right to use the content in your product or service.  If that interests you, please email licensing@imdb.com.

 If you have any questions or concerns, please share your feedback in this thread.

 Thank you for your continued support.

Responses

8.1K Messages

 • 

184.5K Points

4 y ago

In regards to usecases, a lot of things that cannot be achieved through the advanced search tools, can be achieved through downloading files like certificates.list.gz, keywords.list.gz, so on and so forth, and then processing the data found within them.

So, I'm now in the position of urging for a much more comprehensive data set to be made available or for the advanced search tools to be more advanced. In addition, basically, Luca Canali is right. Amazon/IMDb is taking things away from us. Whatever it is that we will be gaining in return remains to be seen.

5 Messages

 • 

240 Points

4 y ago

That is a huge drawback for me. I'm not only a film buff, I also write articles about old and rare films for a non-commercial blog, and I do extensive research for that purpose. I'm using the IMDb list files for more than 15 years, and I download the updates almost every week (in the first years the diff files, then, since I had DSL, the list files). I wrote a script (which makes use of wget) that determines if new files are available, so the download is started automatically.

And yes, I need ALL list files. Not every file every day, but every file from time to time. I'm using the program AMDbFront (don't look for it - it has disappeared from the Internet since its author didn't develop it any more) to convert the files into a MySQL database. AMDbFront is also the viewer for the data. I'm using it in GUI mode almost daily, but sometimes I make complex queries using SQL. One example: A few years ago, scientists from Northwestern University developed a method to determine automatically which are the most culturally significant films (the winner was THE WIZARD OF OZ), and they used the IMDb list files for that purpose (movie-links.list in particular). Here is their paper:

http://www.pnas.org/content/112/5/1281.full

I managed to reproduce their most important result (the long-gap citation count) with my local IMDb data, using a SQL query I wrote. The cited article only covers US films, but I used the method to create respective lists for many other countries, and I published my results in the above-mentioned blog.

I also wrote a script (in VBScript) which adds a table to the MySQL database that contains all films I have on DVD or Blu-ray. The table contains the title (exactly as it's in movies.list) and flags for seen/unseen, region code and short/long films. That information is taken from a text file I maintain for that purpose.  With appropriate SQL queries, I can answer questions like "how many short films from France from the 1930s do I have on DVD" or "who is the actor/actress with whom I have the most films on DVD"?

Well, this all will become impossible with the new dataset format. I surely won't switch to it (even if it would be free of costs), but I will freeze my installation at the current state. That's the lesser of two evils for me.

3 Messages

 • 

180 Points

4 y ago

Thank you Mr Needham, this is indeed reassuring, as mentioned before.
The HTTP access should be a good alternative way to obtain the data i guess.
And i'm sure that adding the AKA titles will help a lot of users, including myself.
Will by any chance, the languages of the movies, be included as well ?
Because AKA titles are mainly important when dealing with non English movies, but i think there is no possibility with the new data files to determine which movie is English or non-English.

2 Messages

 • 

100 Points

4 y ago

This reply was created from a merged topic originally titled Proper channel to request authorisation to extract and use information for resear....

I was wondering if there is a channel I could ask IMDB authorisation for using IMDB movie synopsis data in my thesis.
I am aware of the 'IMDb Data – Now available in Amazon S3' announcement, but I was not able to find an interface that would publish movie synopsis.

Your response is greatly appreciated. 

Best regards,

1 Message

 • 

102 Points

4 y ago

"In our continued effort to best serve our Contributors, we are streamlining the datasets and making them available in a more useful and structured format in S3" == "on a short notice we are destroying many projects and initiatives that were using our data, by withholding most of it and providing the rest in a new incompatible format"

1 Message

 • 

80 Points

4 y ago

I can understand why a transition needs to be made and that its not easy to achieve parity in the new system. My use case is a little different than the others on this list so I thought I'd chime in.

I do economic research on the television industry; asking questions like how characteristics of production companies affects the quality of the show. To that end I use a bunch of data that's not included in the new subset on S3. AKA titles definitely as mentioned by others; they are useful for matching across different datasets. The distributor and production company lists files helps me track which shows were on which networks as well as affiliated with each production company. Producer and writer lists lets me for example connect a show with an Emmy winning executive producer or creator. Lists like language and runtime help me screen out noise from the data, especially for shows that were not very popular and may be incorrectly labelled in other variables. And the full set of genres is important to capture all the shows in a category; if a show has more than 3 genres it may not be included if for example I try to understand what is relevant for ratings in comedies. For ratings, knowing the distribution of ratings is useful to understand how targeted a show was.

Anyway I hope that is useful. Happy to expand on this more if it would help with your triaging of features.

36 Messages

 • 

1.2K Points

4 y ago

I appreciate that Col and sv are stating that they want to hear from us, our reactions to this, and how we have been using the data, so that is why I am sharing this.

I just checked and I have submitted 1,014 updates to IMDB, which I suspect would appear to be a pittance compared to the top contributors, but I hope it is still evidence that I care about the completeness and accuracy of the IMDB database.

One of my favorite things on the IMDB site is the advanced search, but I wanted to do so much more and I wanted to integrate the data with my own custom algorithms and personal logs. I discovered the .list files provided by IMDB and this has been my basis for an exciting adventure. I am not a professional IT person or programmer. But to do what I wanted I taught myself Access and eventually SQL and Python and Django. I only download the files again about every year because I don't watch a lot of new films but I love adding and updating the data for older and more obscure films.

I'll admit that when I first started, even though I didn't have any database or programming experience, I found the .list files seemed antiquated, but I wrote my own programs to extract the data and place them into an SQL database to work the way I wanted it to. So I am kind of glad that IMDB is moving to some new structures that will hopefully be easier to use even though it will probably mean countless hours for me to rewrite a lot of my program for getting the updated data into my database.

The thing I am most concerned about though is making sure all of the data is still available. I use almost all of it, and what is currently in the S3 files would not be worth me using anymore. So I am glad that it has been indicated that there may be means for the rest of the data to be available still. I will list the information below that I use and how I use it.

What I use the most:
Movie (with stats like release year, number of votes, and rating)
AKA
Company (production and distribution)
Country of production
Genre
Keyword
Language spoken
Person (all persons of all roles, and all of their credits on all movies, but most of all directors)
Running time

What I also use often:
Aspect ratio
Certificate
Cinematrographic process
Color
Film negative format
Printed film format
Location
Movie connections
Release date

What I also use but rarely:
Camera
Film length
Laboratory
Sound mix

Also I used ALL OF THE ATTRIBUTES of these items

All of this is strictly for personal use. It is primarily for me to log films I've seen and track data about them and to run interesting queries in the database to find interesting results and patterns within the database regarding connections between all of these stats listed above. However it also can result in me identifying information that needs to be updated in the IMDB database that I can then submit updates for. I have had thoughts before about some day creating something available for public use, but that may be a pipe dream, and if I did ever do it I would of course pay for the license to do so.

So I hope that through this transition that all of this information will be made available in a complete and easy way for contributes like me who wish to have the information.

One other thing I'd like to request: the most useful thing you could add to the datasets you make available would be for movies and for the people to list the IMDB ID used in the IMDB URL for each of the movies. For instance in the dataset for movies for the entry for "The Godfather" it would list the ID as tt0068646 which corresponds to the webpage for "The Godfather" which is http://www.imdb.com/title/tt0068646/. Or for instance for the person Alfred Hitchcock the ID nm0000033 which refers to his page at http://www.imdb.com/name/nm0000033/

Thank you for considering my situation. 

4 Messages

 • 

256 Points

4 y ago

The switch, in its currently implementation is horrible.

IMDb is a community-driven website that relies on the mass of users for nearly everything, from reviews to ratings, to episode and movie release dates yet somehow most of those things are missing from the data dumps. You owe it to the community to give back and 'complete' data dumps in the form of an S3 bucket where the devs pay for bandwidth is the least you could do.

The tens of thousands of users that left reviews or ratings didn't do so for the benefit of a corporation. We contribute information to large repositories like wikipedia or IMDb because we want people to have access to it, and we do so hoping that the gatekeepers will do their best to keep all out there and easily available...but instead you guys have gone the opposite way. Everything needs to be accessed through your interfaces or apps, what you do give back is anorexic in comparison to what you take, and yet you still rely on users to feed you information for your business model to even work...

I urge you to seriously reconsider this philosophy or at the very least have a moment of honesty with the developer community and explain yourselves better. There is no reason to have omitted all of this information and I'm starting to think that there is also no reason to contribute or rely on your website.

You guys have spent the past 30 years harvesting your users for data while providing decent dumps of your database, and now that we've all learned to rely on you guys, you're taking that away. Take a page from Google: "Don't be evil".

Employee

 • 

66 Messages

 • 

3K Points

4 y ago

Hello,

Thank you for your continued feedback. As we review your feedback and work on the HTTP solutions described by Col and sv above, we are revising the shutdown date of the IMDb FTP sites to December 28, 2017.  

More updates will be provided closer to that time. Thank you.

Chris

8.1K Messages

 • 

184.5K Points

Sounds good, I guess... for now.

8.1K Messages

 • 

184.5K Points

WTF?! Time is running out!

36 Messages

 • 

1.2K Points

Yes, I'd also like to hear more updates. Thank you.

2.4K Messages

 • 

81.2K Points

Count me in as well...

Employee

 • 

66 Messages

 • 

3K Points

Hello,
     We are still on track for the FTP sites to stop being updated after December 28. Before that date I will share information about the datasets published on S3, addressing the feedback we have received about requiring an S3 account to access those datasets.
Thank you for your continued patience,
Chris.

16 Messages

 • 

792 Points

I have been watching these comments, and I was hoping you would address the failures with the actors file.  

You haven't updated the actors file on the ftp site(s) in almost three months now.  It still has a date of September 22nd.   Were you guys aware of that?

Will there be any opportunity at all to get a single consistent picture of the last data handoff before you go live? 

And are you seriously saying you are making significant changes to your operational systems between Christmas and New Year's day?  And your operations staff is on board with this plan?

What possible motive is there for not saying which of the next few days:  Dec 20, 21, 22, 26, or 27 you will be communicating?  Those are the only 5 days that are not holidays or weekends.

If you tried to schedule a riskier date in the entire year to convert, or one more likely to cause problems, I think you have chosen the second worst date, with the 29th also being a Friday before a 3 day holiday weekend during Christmas-New Years being slightly worse.

8.1K Messages

 • 

184.5K Points

It's going to be a busy next two weeks.

16 Messages

 • 

792 Points

not so much for me, I've moved my data sources over to tmdb, as far as I can tell i mostly lose minor uncredited roles plus the mpaa rating, everything else i used i have via APIs rather than parsing a download file, so Im happy.

I still harbor resentment that imdb made me do this, and is so uncommunicative.  It is clear that imdb has no respect whatsoever for people using these files, their unwillingness to communicate their plans speaks volumes for the disregard they have for users here.

You would think a company that built its data on the back of volunteers and public submission of data would see the value in giving back, but it appears they are in full amazon mode now.  Petty, overworked, and customers are $.

I wonder would Col have done it the same again, if he could do it over.  Hes probably covered by a non-disparagement, so I doubt he can really respond as he might like to.

Anyway, failing to provide the promised data, and witholding the information about what is changing next tuesday is pretty much par for the course for the new amazon.

4 Messages

 • 

256 Points

So the data dumps in their current state...is that final too? It's missing a ton of information we've been relying on.

@Gardner von Holt
I've nearly moved over to tmdb but it's been a bit complicated trying to get data dumps from them (though possible). I was wondering, are you recreating their entire database or just querying as needed? I'm writing a script to download and update a database in-place for tmdb and thought I'd check if others are doing it too.

16 Messages

 • 

792 Points

No the API actually models my use case far better, as I watch a movie i have not see before I make api calls and add a small subset of the new movie's data to my local database.

2.4K Messages

 • 

81.2K Points

It sounds like a very interesting use case. Thanks for the feedback (I just need to learn to master APIs!)

2.4K Messages

 • 

81.2K Points

4 y ago

This reply was created from a merged topic originally titled Flat files/data sets availability ?.

Hi,

Could some IMDb staff confirm when will the last batch of flat files/data sets be issued and made available on the FTP sites?

The Berlin one still mentions 2017-09-10, a Col Needham post mentions November, 7, and the latest set available was issued on November 24...

Thank you in advance for some clarification of the roadmap!

V.

2.4K Messages

 • 

81.2K Points

Unless mistaken, the FTP site on Berlin Freie Universität is now dead (it prompts me for username/password.
I received no answer to my question above (I was only indicated that the question was forwarded to the "FTP team")

That is the end of a 27 y old story, which started at the university of Cardiff, building the biggest film-related database with contributions from all over the world, but in return, making the data available on those FTP servers for free.

Col N. repeated still recently that they are working on a http solution, on top of the S3 service proposed on Amazon infrastructure, which is rather complex, above all is not free and last but not least: partial.
And now FTP is closed.

Hey you guys, obviously, you have a vacant project manager position (I mean: a real IT project manager). I can make myself available if you wish.

Otherwise, sayonara, IMDb.

16 Messages

 • 

792 Points

Vincent, I am able to access both the German and Finland ftp sites still as of 8-Dec from the USA

2.4K Messages

 • 

81.2K Points

Thank god I started my post with "Unless mistaken"!!
But I confirm, from France, I am prompted a user/pwd on Berlin and connection with the finnish site keeps getting reinitialized.
I will give it another try later.

6 Messages

 • 

172 Points

4 y ago

I have been accessing the Amazon S3 for a few months and am now getting "Access Denied" from the S3 bucket, started this week. I see no one upset here, is this just me? How do I resolve this?

Employee

 • 

66 Messages

 • 

3K Points

Hello Robert,

    Is there specific data or tables that your are seeing this issue with?

Can you access earlier versions of the tables by using an earlier date in the path?

To help us track down if something has changed, can you tell us the exact date when you stopped being able to access the datasets on S3?

Many thanks,
Chris.

6 Messages

 • 

172 Points

I found the issue. It started last week intermittently. I was getting all of the object keys in a list and downloading those containing "current". There is a 1000 object limit on the ListObjectRequest and I suppose you recently went over that limit. I added a prefix to my request and now get a list of objects with "documents/v1/current/" as prefix.

Thank you for your response.

8 Messages

 • 

270 Points

4 y ago

"This more robust and reliable solution will replace the IMDb FTP sites, which will be retired on December 28, 2017."

How about actors.list.gz? It is not updated after Sep 22 2017?

-timon

8.1K Messages

 • 

184.5K Points

They're making off like bandits with a whole lot of customer-generated content.

8 Messages

 • 

270 Points

Who own this content? I have not red any "small print" carefully (att all).

4 Messages

 • 

310 Points

4 y ago

This is my first look at the data IMDb are now providing and I'm extremely disappointed. Maybe I've just been spoiled for the last 20 years with the depth and quality of data IMDb have historically provided. Sad to say, I foresee lots of screen scrapping in my future :-(

Employee

 • 

4.4K Messages

 • 

144.5K Points

From our 19 August update:

"On the point about contributors, we are looking at extending the range of data available via the http solution based on your contribution history and volume. For top contributors and those people using the data to help us clean it via bulk corrections, this is likely to extend far beyond the current set of data even on the FTP site.  It is not our intention to deprive access to the data by those people who have genuinely helped to build it over the years and who want to continue to improve IMDb. We aim to also be able to grant specific permissions to specific customers for specific extra subsets of data as required on a case by case basis. This latter part may take some time to become a fully formed solution so please bear with us. "

4 Messages

 • 

256 Points

This doesn't answer the real question that is, why is information being walled off? IMDB's role up until now is that of a repository built up by millions of contributors. Have you guys run out of ways to monetize the data?

It doesn't matter anyways to me as I've moved on and will be contributing data to other, more transparent organizations.

6 Messages

 • 

278 Points

I've also had my first look at the data that is available.  Getting the IMDb IDs is welcome, and the files are much easier to parse.  I'd already solved both these problems, but this will be good news for anyone looking at IMDb data for the first time.

However, there are some big gaps in the dataset.  There's no way to get the complete cast and crew for a title.  There is only the top, above-the-title, "principal" actors, and even then, the role and credit order is missing.  The actors file has "known for" data, which is fine, but limited.  For example, according to IMDb, Harrison Ford is known for Raiders of the Lost Ark, Witness, Air Force One and The Fugitive.  All of these are fine, notable films, but all the Star Wars films are missing, let alone minor works like American Graffiti.

There is no replacement for aka-names, certificates, plot, release-dates, running-times.  The crew credits don't qualify contributions the way the old data did.  You don't know if a writing credit is the credited screenwriter, or "story by", or the author of the original work for an adapted screenplay. 

I'm hopeful that IMDb will decide to fill in the gaps.  The alternatives are to scrape IMDb.com or to move to crowdsourced alternatives like tmdb.  I'd much rather stay with IMDb, even if it means paying for access.

4 Messages

 • 

256 Points

We've been going on about this for months, and their only response is that only top contributors will have access to a separate, more comprehensive dataset.

That and the fact that they want everyone's contact info and names for data dumps tells me that they're not happy about sharing that much anymore.

What they're completely forgetting is that the public shared with them first.

16 Messages

 • 

792 Points

David, please try to engage them, and if you are successful, please report back, if I could get a guaranteed source of data I might return as I would prefer the quality here.  I attempted to engage the licensing team months ago, and the licensing team returned my initial email, but never actually made any offer as to the pricing.  Like you, I was willing to pay, but no one ever named a price.

4 Messages

 • 

310 Points

In the old datasets the primary key was, what I call, the Fully Qualified IMDb Title (FQIT). My entire home library as based on FQIT, which is human readable. FQIT is still supported by IMDb in the .../title/tt9999999/reference page.

In the new datasets, the primary key is TCONST, which is not human readable. Like Hostnames and IP addresses, Hostnames are much easier to remember and more meaningful than the IP address.

I hope IMDb could update title.basics.tsv, or create a new dataset, to relate TCONST (new primary key) with FQIT (old primary key). This would go a long way to addressing mine (and I assume others) issues.

I understand a few FQIT's change over time and have developed processes to address it. Not really an issue for me.

2.4K Messages

 • 

81.2K Points

FQITs (and names) have proven not stable enough to my taste: the accumulated data sets over the years have lead to unreconciled titles by the tens of thousands, and I am only concerned by feature films, as I discard all series and most of TV films and videos.
So I am really glad to be able to peg the data model on TCONST and NCONST, and I am goingg to take this opportunity to create a major evolution of my database. But a dataset reconciling FQIT and TCONST would be welcome for sure!

Nevertheless, I am going to hold it for a while, as the name IDs, NCONST, come up to 95000000, so I guess that IMDB will soon roll out a longer lasting code/structure/syntax.

Among the other side wishes, pending the availability of extended datasets, I would appreciate that https://datasets.imdbws.com/ indicates the update dates of the available sets. But I am sure it will come in time.

7 Messages

 • 

362 Points

I'm working on the FQIT (title) and FQIN (name) stuff because I want to use the old data as long as possible with my IMDb application that is around for 18 years now (with several updates along the way). 

Currently it is (still) not clear which additional data will be available to match the old LIST file content. We've only heared some vague promises.

For the access to the extended dataset I've only provided some minor corrections to the IMDb dataset itself which in my eyes would not qualify me to get access to the full data. On the other hand I'm a developer of an application which is around for years (18 year as I said above) supporting the LIST file format. This should allow me to get full access.
We will see what happens in the future.

I hope to release a new version of my application soon (the new TSV file format is supported as in 'it is read' but not used on the query side yet).

258 Messages

 • 

7.8K Points

4 y ago

The final build of the data that gets published to the FTP mirrors occurred yesterday so those mirrors contain the final FTP snapshot.

Will that final build be pushed to the FTP site(s)? The dates on most of the files are from several days before that final build.  Thanks.

2.4K Messages

 • 

81.2K Points

The weekly builds are pushed on Fridays, so I guess we will get them by tomorrow. Hopefully the bug on the actor extraction will have been solved (stuck to September so far)

7 Messages

 • 

362 Points

I hope so too. I stumled over another bug is in the business.list.gz file from 2017-12-22. It contains a publicate entry which causes my application to fail due to an error (duplicate key) thrown by the database.

The error in the business.list.gz file is the title "Akemarropa" (2009) which can be found in line 2.264.972 and also in line 2.265.001.

2.4K Messages

 • 

81.2K Points

Thanks for sharing. The FTP files have never been faultless, there has always been a handful of bugs and mistakes. But I have always preferred to correct them manually compared with the millions of available data.

In case you missed it, just note that the Dec 22, 2017 publication is known as the last and final one.

6 Messages

 • 

278 Points

4 y ago

As I only updated my local cache of the IMDb data dumps twice per year, I am just now learning of the retirement of the FTP dumps.  I have been using these dumps for years for personal use, mostly for data mining and to create lists of movies that I want to track down on cable.

I've invested a tremendous amount of time in creating code to parse these dumps.  At my current bill rate, it's tens of thousands of dollars.  As a film enthusiast and software professional, these dumps were a great way to do both at the same time.

While this is very sad for my home, for-fun project, I think it is unfair to think that Amazon is doing this as a money grab.  Amazon has world class data infrastructure in AWS, and it's only natural that eventually they would want to move people onto AWS and away from legacy systems that were built in the 90s.  I don't know much about the internals of IMDb, but I would expect that the system that created the FTP dumps is at least a couple generations away from the system that feeds IMDb.com.

That said, are there any tutorials on specifically getting IMDb data through S3?  I am pretty comfortable with the programming involved, but have not worked with Amazon S3 before.  imdb.com/interfaces states the entry point is https://datasets.imdbws.com/, but I need details on how to construct the SOAP or REST calls.

16 Messages

 • 

792 Points

The S3 files are accessible using the Amazon AWS SDK, there are sdks for most languages.  You can also access 3s files using a tool like Panic's Transmit.  Transmit is an ftp tool that also knows how to access files on s3 and the various cloud providers (google, etc.)

6 Messages

 • 

212 Points

You don't need libraries and you certainly don't need to construct the raw HTTP requests yourself. You can use the s3cmd CLI tool: https://getsatisfaction.com/imdb/topics/imdb-data-now-available-in-amazon-s3?topic-reply-list[settin... 

If you do want to construct the HTTP requests directly, then you need to go read Amazon's S3 docs.

6 Messages

 • 

278 Points

@Andrew - thanks for the pointer.  s3cmd is straightforward to use and is probably the optimal solution for this problem.