From 9d41d19622e18954fda0895c63e1eba09908b31d Mon Sep 17 00:00:00 2001 From: Jonas Smedegaard <dr@jones.dk> Date: Mon, 20 May 2024 09:18:34 +0200 Subject: separate README and SETUP from USE --- README.md | 35 +++++++++++++++++++++++ SETUP.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ TODO.md | 4 ++- USE.md | 98 +++++---------------------------------------------------------- 4 files changed, 138 insertions(+), 92 deletions(-) create mode 100644 README.md create mode 100644 SETUP.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..9ecc72f --- /dev/null +++ b/README.md @@ -0,0 +1,35 @@ +# Handling digital media masters + +Master files - +either original sources or as close as possible - +are precious and should be carefully tracked. + +Digital media files - +e.g. photos, videos, music - +are often large and binary encoded, +which poses special challenges for their tracking. + + +## Git-annex + +Digital media masters are tracked with git and git-annex. + +Git is a version control system - +a repository for historic versions of file contents of a folder. +Git-annex is an extension to git, +better at handling large files. + +Conceptually, +git-annex tracks changes to data same as git, +but where git tracks the *contents* of files, +git-annex instead tracks file *checksums* into git, +moving aside the actual data (to an "annex") +and tracks which annex currently holds data with which checksums. + + +## References + +* [media-master][Source of this document] + +[media-master]: <https://source.redpill.dk/media-master>. + "Digital media master material" diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..cdf3216 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,93 @@ +# Setting up digital media masters + +Digitally stored media master files +are often relatively large, +and are more likely used as-is than edited. + +We use git-annex to track such files. + +As an addon to git, +some of the interaction is done through git commands, +while some is done using git-annex-specific commands. + + +### Paths + +You are recommended to use path suffix "annex" +for media master projects. +Examples: + + * ~/public_annex/home-videos + * ~/private_annex/friends-snoring + * ~/shared_annex_family/xmas_photos + + +## Create + +To turn a folder into a git repository +(see also alternative of (cloning)[#Clone] an existing project), +go into the folder, +and initialize its git and git-annex databases: + + git init + git annex init --version=7 + +To use git-annex only for large files (git for smaller ones), +add e.g. the following to file `.gitattributes`: + + * annex.largefiles=((largerthan=100kb)and(not(mimetype=text/*))) + *.svg annex.largefiles=nothing + +Finally (save)[#Save] all content: + + git annex add . + git commit -m "Initial commit" + +(final `git commit` is implied by [`git annex sync` or `git annex move`](#clone)) + + +## Clone + +To collaborate on a shared git repository, +first create a local clone from the shared location, +and tell git-annex to use it: + + git clone git://[[!template id=githost]]/example + git annex init --version=7 + +Then from time to time syncronize, +ensuring that all content exists both locally and remotely: + + git annex sync --content --all + +Alternatively (e.g. on slow/expensive network), +syncronize only metadata and only with nearest clones: + + git annex sync + +Alternatively (e.g. on small host), +push the content to only be remote without keeping a local copy: + + git annex move . + + +### Publish + +To publish a git repository initially created locally, +first create a new empty git and git annex publicly, +then tell your local git where its new origin will be, +and finally push your local git and git annex into its new public location: + + ssh [[!template id=githost]] git init --bare --shared /srv/git/[[!template id=githost]]/example.git + ssh [[!template id=githost]] GIT_DIR=/srv/git/[[!template id=githost]]/example.git git annex init + git remote add origin [[!template id=githost]]:/srv/git/[[!template id=githost]]/example.git + git push --set-upstream origin master + git annex sync --content --all + + +## References + +* [media-master][Source of this document] + +[media-master]: <https://source.redpill.dk/media-master>. + "Digital media master material" diff --git a/TODO.md b/TODO.md index 9656fe4..805122c 100644 --- a/TODO.md +++ b/TODO.md @@ -1 +1,3 @@ - * separate SETUP from USE + * cover ADMIN tasks + * distributed replicated storage + * purging obsolete assets diff --git a/USE.md b/USE.md index 288f3b9..58b616a 100644 --- a/USE.md +++ b/USE.md @@ -1,59 +1,14 @@ # Using digital media masters -Master files - -either original sources or as close as possible - -are precious and should be carefully tracked. +Digitally stored media master files +are often relatively large, +and are more likely used as-is than edited. -Digital media files - -e.g. photos, videos, music - -are often large and binary encoded, -which poses special challenges for their tracking. - - -## Git-annex - -Digital media masters are tracked with git and git-annex. - -Git is a version control system - -a repository for historic versions of file contents of a folder. -Git-annex is an extension to git, -better handling large files. - - -### Paths - -You are recommended to use path suffix "annex" -for media master projects. -Examples: - - * ~/public_annex/home-videos - * ~/private_annex/friends-snoring - * ~/shared_annex_family/xmas_photos - - -## Create - -To turn a folder into a git repository -(see also alternative of (cloning)[#Clone] an existing project), -go into the folder, -and initialize its git and git-annex databases: - - git init - git annex init --version=7 - -To use git-annex only for large files (git for smaller ones), -add e.g. the following to file `.gitattributes`: - - * annex.largefiles=((largerthan=100kb)and(not(mimetype=text/*))) - *.svg annex.largefiles=nothing - -Finally (save)[#Save] all content: - - git annex add . - git commit -m "Initial commit" - -(final `git commit` is implied by [`git annex sync` or `git annex move`](#clone)) +We use git-annex to track such files. +As an addon to git, +some of the interaction is done through git commands, +while some is done using git-annex-specific commands. ## Status @@ -80,45 +35,6 @@ and then archive their (changes to) content: (final `git commit` is implied by [`git annex sync` or `git annex move`](#clone)) -## Clone - -To collaborate on a shared git repository, -first create a local clone from the shared location, -and tell git-annex to use it: - - git clone git://[[!template id=githost]]/example - git annex init --version=7 - -Then from time to time syncronize, -ensuring that all content exists both locally and remotely: - - git annex sync --content --all - -Alternatively (e.g. on slow/expensive network), -syncronize only metadata and only with nearest clones: - - git annex sync - -Alternatively (e.g. on small host), -push the content to only be remote without keeping a local copy: - - git annex move . - - -### Publish - -To publish a git repository initially created locally, -first create a new empty git and git annex publicly, -then tell your local git where its new origin will be, -and finally push your local git and git annex into its new public location: - - ssh [[!template id=githost]] git init --bare --shared /srv/git/[[!template id=githost]]/example.git - ssh [[!template id=githost]] GIT_DIR=/srv/git/[[!template id=githost]]/example.git git annex init - git remote add origin [[!template id=githost]]:/srv/git/[[!template id=githost]]/example.git - git push --set-upstream origin master - git annex sync --content --all - - ## Access While main purpose of git-annex is to store large media elsewhere, -- cgit v1.2.3