diff options
| -rw-r--r-- | README.md | 35 | ||||
| -rw-r--r-- | SETUP.md | 96 | ||||
| -rw-r--r-- | TODO.md | 7 | ||||
| -rw-r--r-- | USE.md | 142 |
4 files changed, 194 insertions, 86 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..eafb1a4 --- /dev/null +++ b/README.md @@ -0,0 +1,35 @@ +# Handling digital media masters + +Master files - +either original sources or as close as possible - +are precious and should be carefully tracked. + +Digital media files - +e.g. photos, videos, music - +are often large and binary encoded, +which poses special challenges for their tracking. + + +## Git-annex + +Digital media masters are tracked with git and git-annex. + +Git is a version control system - +a repository for historic versions of file contents of a folder. +Git-annex is an extension to git, +better at handling large files. + +Conceptually, +git-annex tracks changes to data same as git, +but where git tracks the *contents* of files, +git-annex instead tracks file *checksums* into git, +moving aside the actual data (to an "annex") +and tracks which annex currently holds data with which checksums. + + +## References + +* [Source of this document][media-master] + +[media-master]: https://source.redpill.dk/media-master + "Digital media master material" diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..3b88339 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,96 @@ +# Setting up digital media masters + +Digitally stored media master files +are often relatively large, +and are more likely used as-is than edited. + +We use git-annex to track such files. + +As an addon to git, +some of the interaction is done through git commands, +while some is done using git-annex-specific commands. + + +## Special strings + +NB! This documentation uses special strings +which you may want to adapt for your local setup: + +githostref: src.ex... +gitshellhost: git.example.org + + +### Paths + +You are recommended to use path suffix "annex" +for media master projects. +Examples: + + * ~/public_annex/home-videos + * ~/private_annex/friends-snoring + * ~/shared_annex_family/xmas_photos + + +## Create + +To turn a folder into a git repository +(see also alternative of (cloning)[#Clone] an existing project), +go into the folder, +and initialize its git and git-annex databases: + +```shell +git init +git annex init +``` + +To use git-annex only for large files (git for smaller ones), +add e.g. the following to file `.gitattributes`: + +```shell +git config annex.largefiles 'largerthan=100kb and not (mimetype=text/*)' +``` + +Finally, save all content: + +```shell +git add . +git commit -m "Initial commit" +``` + +(final `git commit` is implied by a later `git annex sync` or `git annex move`) + + +## Clone + +To collaborate on a shared git repository, +create a local clone from the shared location, +and tell git-annex to use it: + +```shell +git clone git.example.org:/srv/git/src.ex.../example +git annex init +``` + + +### Publish + +To publish a git repository initially created locally, +first create a new empty git and git annex publicly, +then tell your local git where its new origin will be, +and finally push your local git and git annex into its new public location: + +```shell +ssh git.example.org git init --bare --shared /srv/git/src.ex.../example.git +ssh git.example.org GIT_DIR=/srv/git/src.ex.../example.git git annex init +git remote add origin git.example.org:/srv/git/src.ex.../example.git +git push --set-upstream origin main +git annex sync --content --all +``` + + +## References + +* [Source of this document][media-master] + +[media-master]: https://source.redpill.dk/media-master + "Digital media master material" @@ -1 +1,6 @@ - * separate SETUP from USE + * cover ADMIN tasks + * distributed replicated storage + * purging obsolete assets + <https://git-annex.branchable.com/walkthrough/unused_data/> + * verify data + <https://git-annex.branchable.com/walkthrough/fsck__58___verifying_your_data/> @@ -1,69 +1,28 @@ # Using digital media masters -Master files - -either original sources or as close as possible - -are precious and should be carefully tracked. +Digitally stored media master files +are often relatively large, +and are more likely used as-is than edited. -Digital media files - -e.g. photos, videos, music - -are often large and binary encoded, -which poses special challenges for their tracking. - - -## Git-annex - -Digital media masters are tracked with git and git-annex. - -Git is a version control system - -a repository for historic versions of file contents of a folder. -Git-annex is an extension to git, -better handling large files. - - -### Paths - -You are recommended to use path suffix "annex" -for media master projects. -Examples: - - * ~/public_annex/home-videos - * ~/private_annex/friends-snoring - * ~/shared_annex_family/xmas_photos - - -## Create - -To turn a folder into a git repository -(see also alternative of (cloning)[#Clone] an existing project), -go into the folder, -and initialize its git and git-annex databases: - - git init - git annex init --version=7 - -To use git-annex only for large files (git for smaller ones), -add e.g. the following to file `.gitattributes`: - - * annex.largefiles=((largerthan=100kb)and(not(mimetype=text/*))) - *.svg annex.largefiles=nothing - -Finally (save)[#Save] all content: - - git annex add . - git commit -m "Initial commit" - -(final `git commit` is implied by [`git annex sync` or `git annex move`](#clone)) +We use git-annex to track such files. +As an addon to git, +some of the interaction is done through git commands, +while some is done using git-annex-specific commands. ## Status To check status of metadata, use git: - git status +```shell +git status +``` To check status of file content storage, use git-annex: - git annex info +```shell +git annex info +``` ## Save @@ -73,50 +32,59 @@ for git-annex archival, first mark which files are involved and then archive their (changes to) content: - git annex add foo bar - git annex add baz - git commit -m "Update foo bar, and add baz." - -(final `git commit` is implied by [`git annex sync` or `git annex move`](#clone)) +```shell +git annex add foo bar +git annex add baz +git commit -m "Update foo bar, and add baz." +``` +(final `git commit` is implied by [`git annex sync` or `git annex move`](#sync)) -## Clone -To collaborate on a shared git repository, -first create a local clone from the shared location, -and tell git-annex to use it: +## Sync - git clone git://[[!template id=githost]]/example - git annex init --version=7 +To align a git-annex repository with subscribed annex(es), +request a syncronization: -Then from time to time syncronize, -ensuring that all content exists both locally and remotely: - - git annex sync --content --all +```shell +git annex sync --content --all +``` Alternatively (e.g. on slow/expensive network), syncronize only metadata and only with nearest clones: - git annex sync +```shell +git annex sync +``` Alternatively (e.g. on small host), push the content to only be remote without keeping a local copy: - git annex move . +```shell +git annex move . +``` + + +## Access + +While main purpose of git-annex is to store large media elsewhere, +you obviously want to sometime work with the content locally. + +To fetch content of current folder and all subfolders: +```shell +git annex get . +``` -### Publish +To afterwards relieve local storage of git-annex tracked content +in current folder and all subfolders: -To publish a git repository initially created locally, -first create a new empty git and git annex publicly, -then tell your local git where its new origin will be, -and finally push your local git and git annex into its new public location: +```shell +git annex move . +``` - ssh [[!template id=githost]] git init --bare --shared /srv/git/[[!template id=githost]]/example.git - ssh [[!template id=githost]] GIT_DIR=/srv/git/[[!template id=githost]]/example.git git annex init - git remote add origin [[!template id=githost]]:/srv/git/[[!template id=githost]]/example.git - git push --set-upstream origin master - git annex sync --content --all +To fetch or relieve a single file or another directory, +replace the dot with the path to the file or directory. ## Split @@ -127,13 +95,17 @@ Many audio formats can be split with the package shntool and appropriate helper tools, e.g. like this for a FLAC file with CUE file for splicing hints: - shnsplit -o flac -f CUE_FILE.cue FLAC_FILE.flac +```shell +shnsplit -o flac -f CUE_FILE.cue FLAC_FILE.flac +``` For many other formats, including video content, splitting can be done using the [copy] parameter to FFMpeg, e.g. like this: - ffmpeg -i INFILE -ss 15 -t 60 -acodec copy -vcodec copy OUTFILE +```shell +ffmpeg -i INFILE -ss 15 -t 60 -acodec copy -vcodec copy OUTFILE +``` [copy]: https://ffmpeg.org/ffmpeg.html#Stream-copy "FFMpeg copy parameter" @@ -141,7 +113,7 @@ e.g. like this: ## References -* [media-master][Source of this document] +* [Source of this document][media-master] -[media-master]: <https://source.redpill.dk/media-master>. +[media-master]: https://source.redpill.dk/media-master "Digital media master material" |