Missing entries (tconst) in title.basics.tsv with regard to other files

When checking in the main file (title.basics.tsv) all the tconst (tt....) found in the other files (akas, episodes and principals) there seems to be incoherencies and missing movies or TV shows.

The parent series of some TV show episodes found in the title.episodes.tsv (the parentTconst value) are absent in the title.basics.tsv
For instance, tt0086748 show 8 episodes in the title.episodes.tsv

+-----------+--------------+--------------+---------------+
| tconst    | parentTconst | seasonNumber | episodeNumber |
+-----------+--------------+--------------+---------------+
| tt0630512 | tt0086748    | 1            | 6             |
| tt0630513 | tt0086748    | 1            | 8             |
| tt0630514 | tt0086748    | 1            | 7             |
| tt0630515 | tt0086748    | 1            | 4             |
| tt0630516 | tt0086748    | 1            | 2             |
| tt0630517 | tt0086748    | 1            | 5             |
| tt0630518 | tt0086748    | 1            | 1             |
| tt0630519 | tt0086748    | 1            | 3             |
+-----------+--------------+--------------+---------------+

but tt0086748 is not present on title.basics.tsv

while the TV show exists on IMdb:
https://www.imdb.com/title/tt0086748/

What is even more surprising, is that for some cases the webpage of the parent TV show doesn't exists on the file nor on the website:

+-----------+--------------+--------------+---------------+
| tconst    | parentTconst | seasonNumber | episodeNumber |
+-----------+--------------+--------------+---------------+
| tt7153894 | tt4824012    | 6            | 21            |
+-----------+--------------+--------------+---------------+

tt4824012 is also absent from tile.basics.tsv and it returns a dead page
but tt6781204 shows the episode with a different ID for the serie (tt2912216 instead of tt4824012)

And for a more extreme case:

+-----------+--------------+--------------+---------------+
| tconst    | parentTconst | seasonNumber | episodeNumber |
+-----------+--------------+--------------+---------------+
| tt4839832 | tt4820982    | 1            | 17            |
+-----------+--------------+--------------+---------------+

here, both IDs (episode and parent serie) are absent in the titles.basics and the website
tt4820982 -> 404 Not Found
tt4839832 -> 404 Not Found

In conclusion, from those 3 files:
- title.akas.tsv
- title.episodes.tsv
- title.principals.tsv
when we extract respectively the titleId, parentTconst and tconst and check them against the tconst in title.basics.tsv
there are 6220 missing entries
1086 seem to be valid pages on IMDb website (although 48 are redirections -> HTTP/1.1 308 Redirect)
but 5134 are dead pages (see tt4820982 above)

The files title.ratings and title.crew doesn't seem to have the same problem.

Responses

Bethanny

Employee

•

5.2K Messages

•

54.7K Points

2 years ago

Hi @Adren -

We have reported this to the team in charge and will give you an update as soon as I have one.

Cheers!

Adren

15 Messages

•

224 Points

2 years ago

[Update 2022-09-29]

The column "knownForTitles" from the character's file (name.basics.tsv), provides a large number of tconst that should also be present in the title.basics.tsv

Unfortunately, there are a large number of IDs that are unavailable such as this 1922 movie (Beyond the Rainbow)
https://www.imdb.com/title/tt0012937/
or the more recent TV show (Joe Scott - TMI)
https://www.imdb.com/title/tt21318080/

both tconst are only present in the names.basics.tsv :

+------------+----------------+-----------------------------------------+
| nconst     | primaryName    | knownForTitles                          |
+------------+----------------+-----------------------------------------+
| nm0150767  | F. Champury    | tt0005330,tt0012937,tt0011624           |
| nm0604384  | Harry T. Morey | tt0010289,tt0012444,tt0185913,tt0012937 |
| nm12640960 | Joe Scott      | tt14797924,tt21318080                   |
+------------+----------------+-----------------------------------------+

but in no other file from the datasets downloaded last week (23 September 2022)

As a conclusion, I found 7895 unique tt (tconst) IMDb absent from the title.basics.tsv file while they are present in one (or many) of those 4 other files (title.akas.tsv title.episodes.tsv title.principals.tsv names.basics)
checked individually, the missing IDs from title.basics are distributed as follows:

name.basics       2578
title.akas        5741
title.crew        0
title.episodes    27
title.principals  928
title.ratings     0

And when those 7895 tconst are checked on the IMDb website, here is the result:
- 1866 are existing films/series/... (returns an "HTTP/1.1 200 OK"), including 535 that are redirections towards another page (308 redirect)
- but an astonishing number doesn't exist: 6029 sends back a "404 Not Found"

(edited)

Bethanny

Employee

•

5.2K Messages

•

54.7K Points

1 year ago

Hi @Adren -

This has now been fixed.

Cheers!

Adren

15 Messages

•

224 Points

Hello @Bethanny

Unfortunately, with the files retrieved yesterday (2023-04-26 @ 15:15) there are still thousands of tconst found in both title.akas.tsv and names.basics that are not present in the title.basics.tsv

for instance:

tt0021006
tt0021453
tt0023019
tt0024677
tt0036165
tt0038098
tt0046142
tt0052041
tt0052206
...
tt7766088
tt7779806
tt7829938
tt7869672
tt7892078
tt7978886
tt8206494
tt8466868
tt8982514
tt9496006

from akas are not in the main title.basics

if I look more precisely at the first example (tt0021006)

here is the line in title.akas.tsv

tt0021006 1       Ja, der Himmel über Wien        AT      \N      \N      \N      \N

but there is no corresponding web page

https://www.imdb.com/title/tt0021006/ (404 not found)

on the other hand there is a movie with this particular title for Austria

https://www.imdb.com/title/tt0023449/releaseinfo/#akas

so it should be tt0023449 instead of tt0021006

To conclude, there are still 5201 tconst missing in akas and 2537 in names

I hope this will help

1 year ago

Bethanny

Employee

•

5.2K Messages

•

54.7K Points

@Adren Hi!

Sorry to hear that, I let our team in charge know about this.

Thanks!

1 year ago

Adren

15 Messages

•

224 Points

1 year ago

If you want to check how many tconst are missing in the titles.basics file compared to the title.akas you can launch the following commands on a Unix system (Linux or else) to get both lists of sorted IDs

cut -d$'\t' -f1 datasets.imdbws.com/title.basics.tsv |sort -u > /tmp/basics_tconst.tsv
cut -d$'\t' -f1 datasets.imdbws.com/title.akas.tsv |sort -u > /tmp/akas_tconst.tsv

and to compute the IDs only in akas but not in basics

comm -13 /tmp/basics_tconst.tsv /tmp/akas_tconst.tsv > /tmp/only_in_akas.tsv

the result is that there are still 5303 missing tconst in the akas file that are not found in basics (files retrieved on the 2023-05-05)

grep -c ^tt /tmp/only_in_akas.tsv

(edited)

Adren

15 Messages

•

224 Points

6 months ago

Hello @Bethanny

I just wanted to let you know as of today, the inconsistency is not present anymore in the files.

This is probably due to the change of mentioned in the message/banner explaining that the "datasets are backed by a new data source as of March 18th, 2024".

You can close this ticket.

Adren

15 Messages

•

224 Points

3 months ago

Hello @Bethanny

Unfortunately, I have to reopen this ticket as that there is again a large number of missing tconst in all the related (sub)files compared to the reference (title.basics.tsv).

Here is a detailed list of missing tconst in the following files

name.basics       236
title.akas        1854
title.crew        189045
title.episode    1512 (tconst) / 10 (parentTconst)
title.principals  1413
title.ratings     0

I checked the first 10 IDs on the title.crew and they are mostly redirects.

Adren

15 Messages

•

224 Points

As of today (11th of July 2024), the incoherencies in title.axas and title.episode are fixed. But title.crew still contains nearly 190k tconst missing from title.basics

Here is the update:

name.basics	314
title.akas	0
title.crew	189820
title.episode	0 / 0
title.principals	1310
title.ratings	0

Here are the first missing tconst (order by number) in title.crew that are not present in title.basics

┌───────────┬─────────────────────┬───────────┐ 
│  tconst   │      directors      │  writers  │ 
├───────────┼─────────────────────┼───────────┤ 
│ tt0000021 │ nm0525910           │           │ 
│ tt0000136 │ nm0525910           │           │ 
│ tt0000311 │                     │           │ 
│ tt0000600 │ nm0488932           │ nm0241414 │ 
│ tt0000635 │ nm0085865,nm0448682 │ nm0000636 │ 
└───────────┴─────────────────────┴───────────┘

(all of them are redirections to another page)

(edited)

2 months ago

Adren

Missing entries (tconst) in title.basics.tsv with regard to other files

Bethanny

Adren

Bethanny

Adren

Bethanny

Adren

Adren

Adren

Adren

Related Conversations

Helpful Widget