Metadata Update
So what exactly is metadata? Defined on Wikipedia as “data about data”, it turns our collection of media from a drab list of files into an interlinked web of facts and pictures, and imbues each item with a rich set of properties. Your files might have structure, but painting them with metadata converts that simple collection into a multi-dimensional universe of relationships. You can now set about answering complex questions like “Do I have any romantic comedies from the 1990′s that I haven’t watched in at least a year, starring Julia Roberts, and not featuring Whoopi Goldberg?”
Everyone loves metadata, and we didn’t anticipate the extreme load the Plex/Nine release would put on a number of sites when we launched. Tens of thousands of early downloaders, eagerly rescanning their huge personal media collections, contributed to massive amounts of traffic to multiple sites.
As a result, we’ve had to spend quite a bit of time since the release focusing on stabilizing our sources of metadata, optimizing the metadata agents (the bits of code that go out and get your metadata), and adding infrastructure to support all of our new users. Here’s a summary of what we’ve done:
- We’ve brought up a massively powerful machine to serve as our TheTVDB proxy cache. All requests to TheTVDB go through this machine, and it serves over 99% of all bytes out of its cache, which means we’ve reduced the data load on the parent site by a factor of 100x. At peak, we were serving over 500 requests/second, and sending out 320Mbps. Darrin, one of our super-talented Plex engineers, worked literally day and night to get this running after the release, and we also appreciate the help and support from the TVDB guys!
- For movies, we’ve moved to using data that is accessible with an API or through structured data dumps. Specifically, we’re using metadata from Freebase, Wikipedia, and TheMovieDB (as well as a few others for extra artwork, such as MoviePosterDB). This ensures the best availability and stability of the data.
If you’re not familiar with Freebase, you should check them out. It’s one of the few sites in recent memory that’s totally blown me away. The people who designed it are very, very smart people and the amount of data available is unbelievable. If you check out the page for the movie 300, you’ll see it links to 33 reviews of the movie, 6 other sites (such as Rotten Tomatoes), and then has a veritable cornucopia of data including cast, genres, subjects, filming locations, award lists, and more. All of that data is available via a sophisticated API, or via weekly database dumps.
We’ve processed the most recent Freebase data dump into a form that’s most suitable for our agent to consume. Additionally, we’ve enhanced the Wikipedia agent to support multiple different languages for the summaries. Finally, much more data from TheMovieDB is being pulled in by that agent.
In summary, massive amounts of data, all structured (no more “scraping” sites that can change at a moment’s notice), and all completely up to you as to how you use them. Like TheMovieDB summaries? Drag it to the top of the list of agents. Prefer your summaries in Swedish? Make sure Wikipedia is above TheMovieDB, so its internationalized summaries will take precedence. Have two French movies for your mother-in-law? You can manually set the language preference to French for just those two movies, and she’ll offer to babysit her grandkids while happily reading the summaries in French.
These agent changes have been pushed, and you will have them within the hour, unless you check sooner with Plex Online > More > Check for Updates.
Get your settings exactly how you like them, shift-click the refresh button to get new metadata for all your movies, and then sit back and watch the metadata flow in. (N.B. At this point in time, poster/art selections are “sticky” so once set, it won’t change unless you rescan a section from scratch).
Here’s a summary of what the different movie agents now provide, so as to allow you to prioritize them accordingly, through the settings option shown below:
- Freebase: Genres, content ratings, studio, directors, writers, actors, tag-lines.
- Wikipedia: Multi-language summaries, directors, writers, actors, studio.
- TheMovieDB: Summaries (more plot oriented), content ratings, directors, writers, actors, studio, tag-lines.
- MoviePosterDB: Lots of movie posters, at lower resolution than TheMovieDB.
So as an example, if you hate the Wikipedia summaries, and prefer English plot summaries, drag TheMovieDB above Wikipedia. If you leave Wikipedia enabled, summaries that aren’t found from TheMovieDB will be filled in by Wikipedia.
If you want your summaries in Swedish, you’ll need to enable Wikipedia and have it higher in the list than TheMovieDB. Note that currently, in order to change languages, you’ll need to create a new section with the new language setting. Alternatively you can “fix match” on an individual item and manually set the language.
Lots of you have asked: How can we help? Luckily this is quite easy; let’s say you have a movie that’s missing data, or has incorrect data. You can head to one of those sites above and add the missing data, and then everyone in the community will benefit, including users of other apps that access those sites. This really is a case where each one of you has the power to help hundreds of thousands of other people!
The most immediate “turnaround” from this data would be through TheMovieDB, which we access through a well-designed API. We cache requests for 4 hours, so if you add data, you will not see the new data for at most this amount of time. (Note that we are also working to improve TheTVDB refresh times, which are now between 24-48 hours.)
Also, if you’re a developer, please check out our repository for agents. They are easy and fun to write, and we’re really looking forward to seeing the creative things you come up with. Oncleben31 has already written an agent for Allociné for French users, and the ever talented Sander wrote one for MovieMeter, for our Dutch users.
In the near future, we’ll allow you to fully customize any of the data for your media and lock it in place, so that it won’t be overwritten by new data from the Internet. So, for example, you can lock all your titles and summaries, but let the ratings and genres continue to expand and improve over time.
Your media has a bright future inside Plex, and metadata is the key.
77 comments
77 Comments so far
Leave a reply
Awesome Update Guys!
Loving Plex9!
Keep up the great work you all are doing.
Thanks,
Tony.
Thanks Team Plex!!!!
you, once again, made my day!
Awesome news! Who knew twitter would be so handy?! Great job guys.
Will this fix library items where it shows a clip from the film instead of a poster and the year is listed as -1?
@Mazma, it should help, but if you still have a movie which doesn’t get a poster, please add it as described in this sticky post: http://forums.plexapp.com/index.php?/topic/17010-my-musictv-showsmovies-dont-have-posters/
So, if I move TheMovieDB above Wikipedia (for the movie summary plots) it will automatically request the new metadata and update all my movies? Awesome!
Amazing…keep up the great work!
Elan, great news! But regarding “customizing” the metadata in the future, does this mean that we can manually edit the content for any/all of the fields for each file? I wasn’t sure from your wording if we’ll have complete control or if we could only “lock” certain fields, as you say. Thanks again and all the best!
So did IMDB take their ball and go home? Disappointing not to see some transparency in what happened.
How do we ‘unstick’ artwork.
Great work as always!
@Dave: Easiest way is to remove the section and add it again. As long as you don’t change the language, you won’t lose view count, view offset, or rating.
@Elan: Are the Agents capable of letting you correlate what is being requested? I’m pretty sure your TheTVDB proxy would easily allow it.
Would love to see a series of ‘Most Requested’ and other similar anonymous lists as inadvertently voted by Plex/Nine users!
Additionally, being able to filter this by region/ country would certainly prove interesting. For the most part it would tip me off to content i might otherwise have never heard of.
Would it be also possible to point Freebase for TV show category?
Why movie and not TV shows
@crislaet: That’s the beauty of the architecture, anyone can write a metadata agent
Right now we’re sticking with TheTVDB because it seems to have richer data. Over time, of course, that may change.
Hey Elan! This looks awesome. So happy to seen an update to the Metadata agents.
Quick question though: is there any reason that my media manager isn’t pulling down “Cast” information for movies?
I appreciate the help and greatly appreciate all of the time you put into Plex. NINE is awesome!
seen = see! Wish I could type…
@Elan: it seems quite a waste to rebuild the entire database for a single “stuck” item, does it not? In my case, I have one stuck poster (from TVrage) which has nothing to do with the show, and would dearly love to remove it (without redoing the hundreds of other shows in the section…) Any way? help, thanks
@elan
OK but I was trying to make something special with Tutorials, that’s why I mentioned Freebase
Actually, I’m trying to inform some Training DVD sessions on Photoshop, Illustrator.
The hierarchy is similar to TV shows: Chapters (which would be episodes…)
TheTVDB gently kick my Trainings out from their site (which I understand as it’s not ‘TV shows’ content) so was trying to figure out whether Freebase would be the Site to share those tutorials to other people too.
@vdmsss: Of course, this is easy. Just right click and Fix Match
@JPer: It should be pulling down cast, which movie are you not seeing cast for, and where are you seeing a lack of cast?
Is the freebase agent a separate entity, one which we can see the the info provider settings? Or is it part of the Plex Movie bundle? Cheers on pushing the updates, and for all your hard work
Oops…please disregard that last question. I went into Plex Online and installed everything…I see Freebase in the Info Provider Settings now
@elan: Typically I use Aeon for my primary skin and when I hit “I” on the keyboard to bring up more info about the movie it states that “Cast Info Is Unavailable.” To be sure that this was not an Aeon issue I switched back to Mediastream and when I got to a Movie’s info page the Cast area is completely blank. This has all taken place after using “shift-click” to update my library.
@JPer: Yeah, movie cast is not displaying in the Plex client at the moment, the only way to view would be indirectly via By Director, By Starring Actor, etc.
Ahhhh. I see. My mistake. Thanks a lot for clearing that up!
@elan: the problem is that the only available poster is the wrong one, I seem to have no way to select another one, or to select “none” in order to remove the wrong match.
My case is of a poster, but I suppose that the same problem could happen with a description: what if the only match is a wrong match, how do I undo it?
Thanks
@vdmsss: Are you sure it didn’t match the wrong show? If you paste your filename/directory I can try to reproduce over here.
We’re be adding *much* better support for customizing stuff locally in the event of no online metadata before too long.
btw, the github link you gave doesn’t work, just bounces me back to my dashboard. the correct url is http://github.com/plexinc-agents
Plex is a amazing just went out and bought a mac mini to run it on my tv. However i was wondering if there is anyway to add descriptions or tags to my home movies. When i go to the media manager the edit info button is grayed out
Thank you so much for all your hard work Plex is Awesome!!
Elan: Will the “locking in place” of metadata allow us to lock even unupdated metadata?
For example, I would like to handle ratings entirely on my own (via a site that has a lot more votes to get to their score) and would like to be able to tell Plex to just never try to fill in the ratings from any agents. This way I can just go through unrated items and populate the scores.
Thanks.
@Billy: Thanks for the heads up, I fixed it. BTW, haven’t seen you around in a while, hope all is well with you
@Kyle: Thanks very much, that’s very kind of you to say! The ‘edit info’ is grayed out because we weren’t able to get it completed for the first release. Don’t fear, it’ll be along before too long.
@whymse: That’s *exactly* the sort of thing we’ll be allowing you to do.
Has TheTVDB caching optional and temporary or is it going to stay ? Reason i ask is usually we watch TV show day after it aired and that means no metadata that i enter in TheTVDB is presented to us, which has pretty much defeated purpose of entering it in the first place.
Thanks
SWEET!
@wally007: It needs to stay until/unless TheTVDB has a means to deal with all the extra load we’re placing on it. However, we are looking into ways to reduce the latency on the updates, so stay tuned, it’ll likely get much better.
How can we use the meta agents (http://github.com/plexinc-agents) on our macs?
A wiki page would be nice
Oups,
I guess I found a great blog with great posts about plex…
here is some infos: http://oncleben31.cc/en/plex/cinepassion-metadata-agent-for-plex-9/
@OncleBen, Bienvenue dans mes feeds de Google Reader
thetvdb is clearly the best out there. thanks for update
The new agents work perfectly! Thanks for the update.
Never have heard of freebase before, but they do have a lot of data!
Question: when will we be able to see the movie cast available in Plex when looking at the movie info? And will there be local trailer support soon? (-trailer.mov in the movie dir)
Thanks
Could you guys please add Freebase as an infosource for TV Shows as well? In my opinion and experience, Freebase has a very good quality of information and it (currently) better maintained than TheTVDB. It would be great if we could use both these info sources for all out TV Shows. Thanks.
Willem, Netherlands
@Elan: When does the metadata arrive for the music section?
at this moment I’m not so glad with the new music setup (I re-installed plex 8 for this reason), it works (more or less) the same as Itunes. Too much clicking, no album view and the scanning doesn’t show my Various Artists albums.
For the rest amazing work on Plex 9!!
one question: in your TheMovieDB.bundle you write:
#note: right now, themoviedb only supports “en” for language
this is not correct! a quick test of mine showed me, that the api is working with different languages. eg. http://api.themoviedb.org/2.1/Movie.getInfo/de/xml/a3dc111e66105f6387e99393813ae4d5/19995
http://api.themoviedb.org/2.1/Movie.getInfo/fr/xml/a3dc111e66105f6387e99393813ae4d5/19995
there is even a fallback to english info, if there is no translation for a given language:
http://api.themoviedb.org/2.1/Movie.getInfo/se/xml/a3dc111e66105f6387e99393813ae4d5/19995
you simply need to set them in the request-string…
All looks good…loving Plex 9 but am having issues with the ‘resume from’ feature…if I have to stop a film, I have to start from the beginning again and fast forward. Seems to work fine with TV, just not films.
I’m curious, will finer grained control ever be implemented for the metadata? Eg, I like TheMovieDB’s summaries but poster art from MoviePosterDB so I can set those metadata fields to check those respective sites first. It would basically be like the current arrangement but every metadata field would have it’s own organizeable list of sources.
Hi Elan
Many of my movies are in german. Now how do I name them in order to be recognized from Freebase.com? As the site is in english, should the names for my german titles be in english too?
Bastian
@elan , at the pace theTVDB is moving that means caching is permanent
Looking forward to reducing delays though, hopefully it can get close to other ‘scrapers’ 4 hour window soon.
Just checked out Freebase for couple TV shows my GF watches ( mostly Lifetime and Bravo shows ) and it’s few SEASONS behind. So i dont think it’s replacement for theTVDB, YET.
Thank you!
Any chance of adding support for Flash 10.1 or even the 64 bit beta that was introduced a week or so ago?
I still feel really uncomfortable running software with known security holes on my media server, and I just refuse to downgrade on my non-entertainment Macs.
I don’t see much about this, except a brief note in one of the forums to downgrade.
Thanks!
Just one question:
It seems, that Freebase or MovieDB aren’t able to recognize foreign movie titles. IMDB recognized the titles and relinked to the original ones.
That’s ok for, as long as I can see the original title. I don’t need german plot info or whatever.
Is there any workaround to let the agents keep the german movie title?
By the way:
The work you and your team do is absolutely awesome! Keep up the good work guys!
Hola from the Freebase (and Google)! Thanks for using us. The Freebase client on the website is unilingual, but supports only English at this time. The quad dumps, a new feature in the past month or so, features multi-language support, so you can find that favorite German movies ( http://www.freebase.com/tools/explore2/en/keinohrhasen or the pretty view, http://www.freebase.com/view/en/keinohrhasen ). It would be great if Plex stats using the Quad over the older TSV dumps.
Our dumps will be updated weekly. So if you improve the data there, it will show up in the dumps within a week or so. Editing for non-English will require a bit of work, though it is possible:
http://wiki.freebase.com/wiki/Internationalisation
We’ve been emphasizing obtaining data about films over the past few years, so we hope you find what you are looking for, if not complete, help yourself and others out by adding the missing data/images.
http://www.freebase.com/view/user/gmackenz
@Steebro: We’ll be adding better support for international titles for sure!
@W: Flash 10.1 support is on the list, assuming it’s possible, we just want to get through this first round of bugs at least before we look at it.
@bblu: We’ll definitely be improving support for international titles before too long. As time goes one, the data in general and international support will only get better!
@berylium: Already supported at the lower layers, just not exported to the UI yet
@flow: Doh! Thanks for pointing this out.
@sjobbels: It’s def the weakest of the three right now, so expect us to be putting more energy into it! You should be getting metadata for your artists and albums, however.
And what about the RAR-support?!
Who gives a “!€”#€ about metadata when your whole library are in RAR-format?
ANd no, i dont want to spend several days and hours unpacking and redoing my whole library again.
What were you thinkin!?
/Jack
@RAR: I was honestly thinking that the needs of the many outweigh the needs of the few. But if you insist on keeping your data in a closed format which makes it impossible to access in the majority of media players, and which impacts the performance when playing or seeking, you might want to check out this project: http://forums.plexapp.com/index.php?/topic/17211-transprar-rar-workaround-for-plex-9/
So I dragged TheMovieDB to the top, but it doesn’t appear to be taking hold. That site seems to do a better job with kids movies/straight to DVD summaries. There are numerous examples I have where TheMovieDB has a plot summary, but it doesn’t show up in Plex or the PMM at all…
@CharlesF: If you change the order, you’ll need to right click + Refresh or shift click on the refresh button to *force* a refresh of metadata, otherwise it just refreshes stale data.
Thanks for your answer.
I have another question. I copied all my DVD’s with RipIt to my Mac, but when plex tries to find those, it always looks for them together with the extension “dvdmedia”. It doesn’t make a difference if they are visible or not, it searches for “minority report dvdmedia”.
Any idea? Oh, by the way, it worked before.
Thanks again.
Wahoo! Added a decent poster for Rubin & Ed and added Sakura Killers to TheMovieDB, now all my movies look sweet!
Will we be getting some other metadata sources soon?
Music videos, movie trailers, and even game trailers would be dandy.
@Danny: Thanks for setting such a good example and helping everyone else out!
There’s a lot of comments here already, so I apologise if this has already been covered.
As per your instructions, I’ve lowered the precedence of the wikipedia scanner, so that I get plot info rather than production info. This seems to do the trick for new movies I add, but doesn’t update movies already in the database. Is there a way to refresh previously added metadata?
Also, on a somewhat related note, I see there’s an opensubtitles scanner. I assume (perhaps incorrectly) that this will download the subtitles for each movie, but in fact doesn’t seem to do anything?
@A huge Plex fan:
a) to refresh all the other movies, do a “shift+click” on the refresh button on the lower right of the Plex Media Manager, this will force a reload of the data.
b) the current Opensubtitles agent (scanners look for files on your drive, agents go out and fetch information about the files) just does hash matches to try to help out, it doesn’t do anything with actual subtitles (yet!)
Ah, just read two comments above. I need to shift-click refresh. Got it.
Plex isn’t working at all. Can’t play movies, can’t stream movies. Can’t watch movies on the iPhone either. How the hell do you set this up? On the iPhone it can’t play, on the computer it can’t stream. I have the movies pointed in the plex server under videos.
After playing around with this some more, I have to say I’m left slightly confused. There doesn’t seem to be a way to easily know exactly what metadata a given agent provides. For instance, I expect that if I put rottentomatoes at the top of the list, then I will get their ratings. But will this also bring in summaries, cast information and so on from there?
@vdmsss, I have the same problem with several anime shows that have missing art in TheTVDB getting incorrect art from TVRage. My workaround is to go into Info Provider Settings and uncheck TVRage. Then right-click the TV show, Fix Incorrect Match, select a _different_ match, then repeat selecting the correct match again. (Hopefully with metadata editing in the future it will be easier to clear incorrect metadata directly.)
Still haven’t figured out how to unmatch an incorrect match altogether when there is no correct match in TheTVDB… I guess I need to make a TheTVDB account…
BTW @elan, the search can’t seem to find the “Yami to Bōshi to Hon no Tabibito” (84142) entry in TheTVDB. Maybe it’s because of the accented “o”?
@grgz: You are clever, nice work. You can also just select the same show again with a Fix Incorrect Match. We’ll be looking to add a “forceful” refresh option that refreshes everything. We’ll also be adding an “unmatch” option
If a search fails in TVDB, try with the ID. We’re making improvements locally to searching which will be pushed before long.
Cool! Thanks Elan. And searching by the id did the trick on that tricky search.
I found another way to contribute: I don’t use the iPhone app, but I bought it anyway, I think you deserve the money, the desktop app is great on mac mini above the tv.
Having this problem where it can’t find subtitles since I installed Plex 9, but I’m sure its fixed in the next update. can’t wait
keep up the good work.
(tip: cache the library data in the iPhone app. taking too long to lead it every time you press a button)
While the project is going into the right direction the meta data search is still fishy.
My agent settings has setting for movieposterdb.com enabled just fine. Searching manually for the movie I get lots of posters/cover art for it. But Plex does not find the movie at all. I even dragged it up to the top of the list and shift-clicked refresh.
It’s really too much of a black box.
I totally agree with Torsten! I’ve got a few movies in my
database that get wrongly matched by the agent without apparent
reason. Even the function “Fix Incorrect Match” only turns up the
same wrong results as before. When I actually go to Freebase or
TheMovieDB website I find the movie has the exact same name and
year as my file. I’d like to have the possibility to tell Plex to
get the metadata for a specific file from a specific website. At
the moment that feature doesnt seem to be working. This would also
fix the problem with TV/Movie classification for some files (i.e.
TV Movies that you keep in the Movies folder or vice versa) but i
guess you guys are probably already working on some of that stuff.
thanks a lot for all the good work! After all 99% of my files are
instantly identified correctly which is awesome!
Is there no way to reorder/manage the Agents under Windows?
I can only add/remove Sections in PMM, but i cant setup anything about Metaagents.
So atm it’s sensless for me to use PMS/Plex 9 until it’s possible to sort the Agents in Windows 7.
@Mama: you can do it now by editing the XML files under AppData/Local/Plex Media Server/Plug-in Support/Metadata Combination. Not as easy as a UI (which we are working on as we speak), but at least it’s possible!
Noting that this page begins with the note of how much drag requests for metadata caused on various systems, it is very surprising to me that there is no option in plex to simply turn metadata seeking off. Currently that is not the case, but for myself, I simply would prefer that option as a permanent setting. Please ask the dev team to enable an “all off” setting for metadata.
@chas: Just make a section using a Personal Media agent and there will be no metadata retrieved.