IMDb Data – Now available in Amazon S3

Thank you Mr Needham for this much welcome answer, which is somewhat reassuring. V.

Thanks, for your feedback, Col Needham. I'm pleased to know that you have big plans. I hope the things of the future wind up having more merits than demerits.

Thank you for responding, at least the timing and some of the motivation for this is clearer. However, without the ability to answer the simple question "who is in this move", the data is essentially useless to me. Sad day for imdb, when it switches from "here is some awesome data, we are excited to see what you do with it" to "Convince us why we should let you see our data" Reluctantly I am switching to another data provider, themoviedb.org

“ Sad dayfor imdb, when it switches from "here is some awesome data, we are excitedto see what you do with it" to "Convince us why we should let you seeour data" “ You said it best, right there. There are 2 main ways to "consume" IMDb data(files): 1 - The Static Mode: People already have an use for the data and gothere and grab it for their usual needs; 2 - The Discovery/Explorer/ResearchMode: people get some data, and then detect some patterns, get/try differentdata (files) and see some more patterns, get new ideas and connections, testtheories, invent new uses for the data, detect errors/inconsistencies, get yetanother batch of different files to retest other hypothesis... While the first type of Users is perfectly fine and has animportant place (plus, those same Users could already have been or they havethe potential to become a type 2 User) the real "power" of (IMDb)data use is in the 2nd type. IMDb wants to categorize/"lock" people in"typical", static/monolithic, use cases (which they can't becausethere is no such thing, when one gets access to such a rich and diverse qualitydata – the richer the data the more unlimited are the possibilities) withoutthe understanding that is in exploration the true power and possibility of theunimpeded access people enjoyed until now. This is also the mode where moreerrors and problems with the data are bound to be discovered and, thus,reported/corrected. So, they are (re)enabling type 1 Users (which is good), but type 2 Users(where the true value, for both IMDb and the Users/Contributors, really lies)are shackled and constrained in typical use-case boxes that has little or noreal use to them (because the most desirable "consumer" of data, forIMDb, is the one that has no [pre}set idea what she/he'll do tomorrow with thatdata; the ones that ask themselves "What if...?" and then go aboutchecking that out). And these Type 2 are also the main Contributors (if not toIMDb directly, at least indirectly) to the Film/TV community. They make all ofus appreciate and understand all the interconnected nature of the art form. Andthat'll always return, in the end, to IMDb, in one form or another, because themore (quality) information and the bigger the community, the more people will turnto IMDb (because is one of the best, more popular, places to know more).Curtailing, impeding, Type 2 users, in any way, is a substantial self-inflictedwound (the proverbial "shot in the foot"). In spite of building one of the most successful stories ofdata gathering/maintenance in history they seem to lack a basic understandingof how this was achieved, how the whole direct/indirect feedback loop works,how the ample availability of their (almost) raw data was like seed(s) forfertile ground(s) -- where they could/would reap the benefits, severalfold,later, down the line, directly or indirectly. This kind of decisions seems toblissfully ignore how IMDb arrived at this point in time. This is akin to the Captain of the Titanic failing to acknowledgethe “invisible” 90% of the iceberg that lied below the line.

I don't think I can or ever will understand or be able to wrap my head around this line of logic. We created this awesome movie database, lets make an APIWe made this awesome API, lets give everyone access to all the data! 20 years passNow that we've given everyone all the data for 20 years, and the API is old lets update it!People can't be using ALL our data right? No way that people are actually using all the data we provided them over the course of 20 years just because we can't see analytics for it. Not possible Nope.Well just remove access to 9/10ths the data then. Its fine if you guys change the format and work to update the way its parsed, removed redundancy etc, I am ALL for that, trust me you. But plain and simple, not giving me something I already use, is going to destroy so many projects, and development ecosystems, I don't think that we have a number that could go high enough to represent the amount you are killing by removing access to so much of the data. You're literally killing an un-ending amount of (infinity) projects.

On the point about contributors, we are looking at extending the range of data available via the http solution based on your contribution history and volume. We aim to also be able to grant specific permissions to specific customers for specific extra subsets of data as required on a case by case basis. This latter part may take some time to become a fully formed solution so please bear with us Any update(s) on the above?

My takeaway from Col Needham's recent post was that the primary issue standing between providing complete data and not was the significant development effort to make the data available from the new systems. But now, It sounds like from this post that IMDB will have the logic in place to provide fuller datasets, and will provide them to contributors. Now it appears to be purely that IMDB wishes simply to prevent the public from accessing the collected data for free. This appears to no longer be about development efforts, rather it appears to be exclusively about cutting off free data. Frankly, IMDB needs to get its story straight. If the data is in fact available, and more complete data sets would be there for us ... if we provide IMDB with free (aka donated) labor, then the entire premise of Col's justification looks shaky .. at best. I hope you guys come to your senses, but I no longer believe we are being told the real story here, so I have now completed moving my data sources away from IMDB, and will donate both time and money elsewhere. I've enjoyed using IMDB's data since well before it sold out to Amazon, and its sad to see so many years of cooperation trampled by this ill considered project.

Thanks for postponing the shutdown, IMDb staff. We can at least grant y'all that.

Due to my notification settings, I was informed: "Nobody liked your comment". That's kind of funny like a pun. Thanks, Nobody.

With the revised shutdown date now days away, I note that no promised updates have not occured. Really a shame to see an organization lose its way.

I can foresee the IMDb contributor base potentially splintering into "patriots" and "loyalists". I may be no patriot, but the loyalists are really starting to disgust me.

A first quick look at title.basics.tsv.gz is very promising, and much more satisfactory than the former FTP offer. Could you please state the charset used to published the text files?

Hello Vincent, The files are in the UTF-8 character set. I have pushed out an update to the http://www.imdb.com/interfaces/ page to add that to the file details section. Best regards, Chris.

My quick look at title.akas.tsv.gz suggests it is UTF-8 - tt0000008,5 is Cyrillic and looks sensible, emacs reports that the underlying file was UTF-8 and displays the characters correctly. That Cyrillic aka title does not appear in the aka-titles.list.gz file on the FTP site. It looks to me as if the new system is including data that the old system could not handle.

The latest and final FTP snapshot is online now. But once again there is no updated actors.list! It's still from September 22. That's very annoying and embarrasing - 3 months should have been time enough to fix this problem. At least the final dump should contain recent files only. By the way, on the German FTP server there is a new directory "frozendata". The files are now available both in ftp://ftp.fu-berlin.de/pub/misc/movie... and ftp://ftp.fu-berlin.de/pub/misc/movie..., but I guess that in the long run, only "frozendata" will remain.

Thanks for providing the HTTPS interface. It's really appreciated. Something I don't understand is about the genre information. For all the movies that have more than 3 genres, only the first 3 in alphabetical order are reported. For example for "Dunkirk" (tt5013056) the War and Thriller ones are omitted. And Dunkirk really is a "War" film... I cannot get the reason for this limitation. In fact, also imdb.com has this problem. The top bar lists only 3 genres, but then in the "Genres:" tag has all of them. I would be also really great if you can add also the "Country" and the "Keywords" information. This info is really important to categorize movies. Without it I have to resort to scraping your site, that is something I would like to avoid. Thanks, Luca