Localization
Local copies of files that live on a slow network share: policies, preferences, the Cache menu and the Read node's bar.
Reading plates over the network can be slow and uneven. Localization keeps a copy of the files a comp reads on a local disk and reads that copy instead. It works the same way as in Nuke: the same preferences, the same knob on the Read node, the same Cache menu and the same coloured bar on the node.
The comp is rescanned for Reads to copy only when a Read's file, range or policy, or the current frame, changes, so an idle comp costs nothing.
The copying happens in the background. It never makes the interface or a render wait: until a frame's copy is there, the Read reads the source file.
Turn it on for a Read
Every Read (and DeepRead, and GeoImport for Alembic files) has a localization policy knob under its file name:
| Policy | What happens |
|---|---|
| on | The files are copied, wherever they are. |
| from auto-localize path | The files are copied when they are under one of the auto-localize from folders in the Preferences. This is the default. |
| on demand | The files are copied only when you click Update on the node (or use Force Update). |
| off | The files are never copied. |
Only the frames in the Read's frame range are copied, starting at the current frame and working outwards (image sequences before movie files). The Update button, on the same row as the policy menu, copies the files now, and copies again any file whose source has changed.
New Reads take their policy from Preferences › Performance › Localization › localization policy.
The policy is saved in the comp and in exported .nk scripts as Nuke's localizationPolicy knob,
and .nk scripts that use it (or the older cacheLocal knob) load with it.
The bar on the node
A bar at the top of the Read's thumbnail shows where its files are:
- amber, filling from the left: copying (a movie file shows no bar until it is complete);
- green: every frame in the range is local and up to date;
- red: a source file changed after it was copied;
- striped red: out of date, and the Read is reading the source files until you update it. hide out of date progress bar in the Preferences hides this one.
The colours are theme colours (--oc-graph-loc-*).
When a source file changes
A local copy counts as up to date while its size and modification time match the source's. Each file is checked again after check for updated files every minutes (30 by default), or at once with Cache › Localization › Force Update.
- With read source files when localized files are out of date on (the default here), a Read with any out-of-date frame reads all its frames from the source files until they are copied again. Turned off, it keeps reading the old copies.
- With mode on, Reads whose policy is on or from auto-localize path are copied again by themselves. On demand Reads stay red until you click Update.
- With mode manual, nothing is copied by itself; out-of-date files are shown red and you update them with Update or Force Update.
Preferences › Performance › Localization
| Preference | What it does |
|---|---|
| mode | on: copies Reads set to on or from auto-localize path by itself and checks every localized file for changes. manual: checks for changes but only copies when you ask. off: nothing is copied and every Read reads its source files. The status bar shows the mode at the bottom right. |
| read source files when localized files are out of date | See above. |
| hide out of date progress bar | Hides the striped bar of Reads reading their source files. |
| pause localization on script/project open | Opening a comp pauses localization, so it opens faster. Resume it with Cache › Localization › Pause. |
| localization policy | The policy new Reads get. |
| auto-localize from | The folders whose files are copied by from auto-localize path Reads, for example /mnt/jobs. Separate several folders with ; (Nuke takes one). Empty: those Reads are not copied. |
| localize to | Where the copies go. Empty uses /cache/<app>/localized when the machine has a writable /cache disk, else localized in your user cache folder (~/.cache/<app>/localized). A file keeps its full path inside it, with the leading / turned into _ (as in Nuke): /mnt/jobs/a.exr becomes <localize to>/_mnt/jobs/a.exr. |
| limit to (GB) | The most space the copies may use; 0 means no limit, and a negative number keeps that much disk space free. When a new copy would go over it, the copies no open comp uses are deleted, least recently used first. If that is not enough, localization pauses and a Failed to Localize File dialog says so. The dialog shows how much is in use and free; free space is measured on the disk that holds the localize to folder (or its nearest existing parent folder). |
| check for updated files every (minutes) | How long a copy is trusted before its source is checked again. |
| never check for updated files | Never checks by itself (Force Update still does). |
| limit copy speed (MB/s) | Keeps the copying from filling the network link (ours; 0 = no limit). |
Cache › Localization
- Pause: stops copying (and starts it again). The status bar shows (paused). Force Update still works while paused.
- Force Update › All / Selected / On Demand Only: checks the sources now and copies what is missing or out of date, for every Read, the selected ones, or the Reads set to on demand.
- Clear Unused Local Files...: lists the local files no Read in the open comp uses. Continue deletes them, Cancel keeps them.
Copying stops while the Viewer plays, so playback has the disk and network to itself, and goes on when it stops.
How the copying works
- One background thread in the backend copies one file at a time.
- An interrupted copy (Pause, playback, quitting) is kept as a partial file and continues from where it stopped, as long as the source has not changed.
- A finished copy is checked by size and gets the source's modification time.
- The source files are only ever read, never changed.
- Each Read's set of files has a manifest in
<localize to>/.manifests/, so the copies are known again in the next session.
Rendering from the command line and the farm
render.sh and the LAN farm workers read the source files by default. Add --localize (Nuke's
--use-local-file-cache works too) to read the local copies that match their source instead:
./render.sh --localize -F 1001-1100 shot.nk
./render.sh --worker --localizeNothing is copied during a render; a copy that no longer matches its source is skipped and the source is read.
Differences from Nuke
- auto-localize from takes several folders.
- limit copy speed is ours.
- Least recently used copies that no open comp needs are removed when the limit is reached; Nuke only asks you to clear space.
- Nuke's
NUKE_LOCALIZATION_NUMWATCHERS(threads that check files) has no counterpart: one thread checks and copies.