4 Messages
•
214 Points
HTTP Headers for IMDb datasets
The HTTP headers for the title.ratings.tsv.gz return different dates for the "Date" and "Last-Modified". It appears "Date" is the correct header based on when new files are available, but I'm not sure. Can you confirm "Date" is the right date to use for when the file was created? If so, does "Last-Modified" have any meaning?
Accepted Solution
Bethanny
Employee
•
5.6K Messages
•
58.9K Points
4 years ago
Hi @burkasaurusrex -
We received an answer from the team in charge:
Date: Current date and time.
Last-Modified: Object creation date or the last modified date, whichever is the latest.
The Last-Modified is right field to use to understand when our dataset’s get updated.
Cheers!
0
burkasaurusrex
4 Messages
•
214 Points
4 years ago
Following up here after a bit more investigation. "Last-Modified" changes with the "ETag", while "Date" seems to change somewhat randomly every few hours even when "ETag" does not change. Both behaviors are more of what I'd expect (maybe the CDN is returning the "Date" it cached the result or something while "Last-Modified" is returning file information).
However, when the "ETag" changes, the new "Last-Modified" date points to ~20hrs earlier. I would expect a date closer to the actual "ETag" change. So is the "Last-Modified" date when the information was pulled actually from IMDb's database? Or is it erroneous?
0
0
Bethanny
Employee
•
5.6K Messages
•
58.9K Points
4 years ago
Hi burkasaurusrex-
I have made the question to the team in charge. As soon as I have an answer I will give you an update here.
Cheers!
0
0
burkasaurusrex
4 Messages
•
214 Points
4 years ago
Thanks @Bethanny! I started doing a HEAD request every hour or so. In case it's helpful to the team, here's the output of unique responses I've observed over the last few days in case it's helpful (all dates UTC):
(edited)
0
0