FCP X: Divide Large Libraries [u]

Posted on by Larry

[Updated Aug. 12, 2014, with additional tips on improving performance.]

For smaller projects it is perfectly fine to store all media and projects in the same library. But, for larger projects, this can cause problems. While the specific number of open files varies based upon RAM and project complexity, a rough number that I use to determine when a Library is getting full is around 3,000 clips.

Creating multiple libraries for large projects can simplify media management and improve the performance of Final Cut Pro X.

However, before we start moving media around, we need to define two key terms:

Managed media is media which is stored inside the library bundle on our hard disk. External media is media which is stored in a folder outside the library bundle, but linked to the folder using “symlinks.” Symlinks are a high-power version of an alias.

In general, there is no performance difference between Managed and External media; assuming the hard disks they are stored on are roughly the same speed. However, if you plan to use the same media in more than one Library, external media will decrease the amount of storage space you use. This is because when FCP uses media from one library in a project stored in a different library, it will copy the media file if the media is stored inside a library (Managed) but only copy the link to that file if the media is stored outside the library (External).

So, for very large projects, storing media outside the library will provide greater flexibility and reduced file storage requirements. Storing all media inside the Library on your hard disk is easier and simpler to manage and backup.

Final Cut has several media management functions built-into it:

Consolidate a library is the best option when media is stored in multiple locations and you want to store everything in one spot.

Copying media is the best option when you need to have multiple copies, say to share with another editor.

Moving media is the best option when you want to break up a large library into multiple pieces.

NOTE: The techniques described here require Final Cut Pro X 10.1.2 or later.

WHY THIS MATTERS

First, there is a limit to how many clips FCP X can have open at one time. While this numbers in the thousands, I’ve had several editors contact me who’s clips numbered in the tens of thousands! Finding simple ways to manage media means that we need to limit the number of clips that are open at one time.

Final Cut allows us to have multiple libraries open at once and to edit media from any open library into any project. However, when a clip is edited from, say, Library 1 into a project which is stored in Library 2, FCP makes a copy of the Library 1 media into Library 2. This makes sense because if you deleted Library 1 and all its media because you are done with it, you don’t necessarily want to delete the media which is referenced by projects stored in Library 2.

But, for large projects, copying media between libraries negates the advantage of storing media in separate libraries in the first place AND it takes up more disk space.

So, Final Cut provides a second option: When you edit media which is linked to Library 1 (External media) into a project which is stored in Library 2, only the link to the media is copied, not the media itself.

This means that when we use External media, we can use thousands of clips stored in dozens of libraries and, provided that all media is stored external to any library, performance won’t suffer and our hard disks won’t fill with duplicate media.

NOTE: Again, to obtain the best performance, only open the libraries that you need, and keep the total number of open clips and libraries as small as possible.

UPDATE: THOUGHTS ON PERFORMANCE

Apple made major improvements to system performance in the 10.1.2 update. So, as a start, if you are experiencing slow-downs, be sure to upgrade. After that, while there are many elements that affect the performance of Final Cut, the three big ones are:

Final Cut uses as much RAM as you have installed, so adding more RAM – assuming your system supports it – is always a good idea. As well, other applications or utilities can divert system resources, which also slows FCP down.

However, this article looks at using multiple libraries as a way to improve performance.

NOTE: Another factor affecting performance is network speed, if you are editing clips or accessing libraries store on a server, rather than editing clips or libraries stored locally.

THE IDEAL APPROACH

Ideally, we should think out media management before we start importing our first clip. This is very easy to say and almost impossible to do. Still, if you are someone who can actually plan that far ahead, creating multiple libraries and storing media externally will vastly simplify media management for large projects.

After you create a new library, and BEFORE you import any clips, select the Library and choose File > Library Properties (type Control + Cmd + J). This opens the Library Properties section of the Inspector

Click the Modify Settings button and click the Media menu.

Select Choose from the menu and navigate to the folder where you want all media for that library stored. In this example, I’ve created a folder on my 2nd Drive RAID called “External Media Folder,” and selected that.

NOTE: You can name this external folder anything you want. Pick a folder name that helps you remember what’s inside. Final Cut Pro will work with any folder name that the Finder supports.

Now, when you import clips into that Library, Final Cut will store the media in this external folder and create links to the media in the Library itself.

Cool. But who can plan that far ahead?

CREATE NEW LIBRARIES

If you’re like most editors, you won’t realize you need multiple libraries until you are drowning in files and wondering why FCP X is so slow. The answer is your libraries are too big and need to be divided up.

NOTE: Libraries and the clips they contain require RAM to open. When you have thousands of clips in a single library, you can exceed the RAM that FCP has available. Divide your media into separate libraries, and only open the libraries you need for the work you are doing right then. Get in the habit of closing libraries you don’t need to reduce the amount of RAM FCP uses to track all your clips.

Remember a project can contain media from any library, whether that library is open or not.

We want to create external media, so that we can edit any clip from any library into any project without duplicating media files in the library. This is actually quite easy – but not intuitive. The secret is to create the new libraries first, then move the media into it.

Here’s how.

Let’s say that we have one big library: “Main Library” and we want to divide this into three new Libraries: “Library 1,” “Library 2,” and “Library 3.”

NOTE: Remember, you can use any name for a library. I’m just trying to be really obvious here.

First, create a folder on your external hard drive to store the media for each library you want to create. Here, for example, I’ve created three folders on my 2nd Drive:

Next, create three new libraries in Final Cut Pro X using File > New > Library. Store the Library files wherever you want – again, I recommend never using your internal drive – and name them whatever you want. These Library files will not contain media, just links to the media.

For this example, I created Library 1, Library 2 and Library 3.

Select each of the three new libraries – one at a time – go to Library Properties (Control +  Cmd + J) and change their Storage Locations so that Library 1 Media is stored in Library 1 External.

Set Library 2 to store its media to Library 2 External.

Set Library 3 to store its media to Library 3 External.

UPDATE: IMPROVING PERFORMANCE

Something I just learned is that Final Cut can point multiple libraries to the same media folder. In fact, Apple recommends that when you need to create multiple libraries to minimize the number of clips open at the same time for a specific project, have all the libraries point to the same external media folder.

When importing media, FCP X assigns unique identifiers to distinguish clips from one another.  This means that clips with the same name shot on different days or using different cameras won’t overwrite each other. This is why it’s important to use the media management commands — like Consolidate — within FCP X because the app can properly track all links to media files wherever they are located.

Storing all media files from multiple libraries into one folder is the same as outlined above, except that when you set the Media location using Library Preferences, point each Library to the same external media folder. While storing media in multiple folders works perfectly well, creating a single external folder makes it easy to locate media and assures that there’s no confusion about where new media files should be imported.

While I think the idea of storing all media for a specific project in a single external folder makes a great deal of sense, I will create separate media folders to store media for projects that are not related to each other.

MOVE MEDIA

While you can drag media from one event in one library to another event in a different library, since we want to take a big project and break it into smaller groups, it is MUCH easier to do this by moving media contained in entire events.

Here, in the Main Library, I’ve created three events – one for all the media that I want to move to each of the three libraries.

NOTE: You can create multiple events for each library, because each event name will be retained when you move it to a new library. Also, when you move media files all keywords, ratings and metadata move with them.

Now, organize your media by dragging it into the appropriate event. To keep things simple for this article, I’m dragging three clips from an event in the Main Library, into the event for Library 1.

Let’s pretend that all our media is now organized and ready to move. The process is the same for each event and library.

Select the Event containing the media you want to move to a new Library. Choose File > Move Event to > [Name of Library]. In this example, I’m moving the Event “Media for Library 1” into “Library 1.”

Final Cut gives us the option to move camera original, optimized or proxy files. in general, because you don’t want to keep recreating optimized or proxy media, you should move everything.

NOTE: If the media is inside the library, it will be moved. If the media in the Main Library is linked, only a link will be created.

Open the Background Tasks window to see the status of your media. (This is optional, but fun to watch.)

Because we set Library Preferences to store all media external to the Library, Final Cut moves the media to the external folder – “Library 1 External” – while creating symlinks in Library 1 that point to the media location.

Here we see that the clips were moved from the Main Library to Library 1.

While the media was moved OUT of the Main Library into Library 1 External. We have successfully created external media from an existing library!

SUMMARY

If media is stored externally to the original Main Library, Final Cut will only move symlinks to the new libraries. No media will be moved from its external location. This is still a good idea, because we need to reduce the size of the open Library.

If media is stored inside the original Main Library, Final Cut will move the media to the media location specified in Library Preferences, which, in our case, is an external folder and create symlinks in the new library.

Either way, you end up with as many libraries as you need, the ability to open and close libraries as needed and no duplication of files or hard disk storage space.

Very cool.


Bookmark the permalink.

29 Responses to FCP X: Divide Large Libraries [u]

← Older Comments
  1. Is there a way to extract all my video files from a Library and move them to an external folder after they have been Consolidated or stored in the Main Library?

    • Larry Jordan says:

      Philip:

      Yes.

      Quit FCP X.

      Control-click (right-mouse-click) on the Library icon in the Finder.

      Select “Open package contents.”

      Drag the folder “High Quality Media” out of the Library folder to the location you want your media stored.

      You will then need to reconnect/relink any projects using that media to the new location.

      Larry

  2. Pam Arts says:

    What’s the best way to edit & store huge files (850+g) on FCPX usung my laptop?

    • Larry says:

      Pam:

      (smile…) In these days of 5K images and RAW formats, 850 GB is not that big anymore. The file size isn’t really the issue, its more the codec and bandwidth that are the challenge. Whether you are using a laptop or a desktop, I recommend storing your files on an external drive. If these are HD formats, a single Thunderbolt drive will be fine. If these are images larger than HD – 2K, 4K, or beyond – or you are editing multicam, then I recommend an external RAID of at least 4 drives.

      Editing in FCP X is the same, regardless of the size or number of files.

      Larry

  3. Jason says:

    Hi Larry, How come I have one Library I created called Virtual Consults, but on my external drive Final Cut has saved the library into Virtual Consults and 4 or 5 other Virtual Consult libraries with different numbers attached to the end of the library name followed by EST?

    • Larry says:

      Jason:

      You are seeing the backups that FCP X makes automatically of your library database. Normally, these are stored in [ User Directory ] > Movies > FCP Backups. For whatever reason, this got changed to the same location as your library.

      These are backups. Continue working with the library that does NOT have any numbers after it.

      Larry

  4. Eric says:

    I think I know what to do know after reading this, but to clarify let me tell you my situation. I think in a normal Mac hierarchical folder structure, and i’m trying to apply that to FCPX – but in a way that clips/raw media don’t overwhelm the drives. I created ONE library and run everything off an external drive so I can move it between iMac and macbook. The library is FCP-External. inside that are Events that are title as major “areas” of movies and inside each Event are individual projects – making up the actual movies. I had hoped that by doing that the imported media would reside inside the project, contained within the project, but it doesn’t seem to do that. instead ALL media seems to show up under the primary (initial?) Event in a group called “clips”. And it seems that if I move a project under a different event, then the clips get added to, but aren’t “physically” tied to the Project I’m working on – meaning I can’t wholesale look at a Project and see all it’s clips/media in the browser. Obviously this seems to be driving a massive amount of clips (the media) that is ever-growing without a way to easily see what media is tied to the actual project I’m working on. I’ve taken a Project as the lowest level item where I’m working, hopefully that is right.

    It seems that your above guidance is to actually make different Libraries for the different types of movies I want to work on (school, personal, business, etc) and build events & Projects under them. And I can create those AND move my existing media/Projects/Events to them and have everything work. Am I right??
    thanks

    • Larry says:

      Eric:

      You are assuming that you can organize events and projects in the Finder. You can’t. You can organize Libraries and media, but not events and projects; those are always internal to FCP X. Media is not assigned to an event until it is imported. Media is not assigned to a project until it is edited into the timeline inside Final Cut.

      So, a better way to think is to organize your media by subject / project. Unix allows thousands of files in the same Finder folder, but that doesn’t mean WE can easily find a file stored amongst thousands of files.

      There is every advantage to storing media externally, but don’t try to replicate the internal structure of Final Cut to your media. I find it much easier to create a master folder for a project in the Finder, then subfolders for different subjects / scenes / interviews. I use folder names that make sense so FCP X will assign appropriate keywords.

      And, in answer to your second question, yes, it is very easy to copy or move files from one Library to another. Don’t try to do all your work in a single library, you will very quickly bog your whole system down.

      Larry

  5. berta says:

    Using fcpx, can I save my library and project in one external drive while using a second external drive to retrieve the original 4k media? If so, would this potentially create a problem?

    • Larry says:

      Berta:

      As long as both drives are fast (i.e. not connected via USB 2) and large enough, this would work fine.

      It is a very typical way to work.

      Larry

← Older Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.