Tracking Huge Files with Git LFS
0 likes21,196 views
Git LFS is a system for managing large files in Git repositories. It allows large files like images, videos and design files to be stored separately from the Git repository while still keeping track of versions and metadata. The document discusses how to set up and use Git LFS, including tracking files, migrating existing repositories, and tips for developers, migrators and administrators. It emphasizes using the latest versions of Git and Git LFS and properly configuring Git LFS for enterprise usage.
![$ cat ~/.gitconfig
[filter "lfs"]
clean = git-lfs clean %f
smudge = git-lfs smudge %f
required = true](https://image.slidesharecdn.com/build-01-smithschneider-161009200441/85/Tracking-Huge-Files-with-Git-LFS-28-320.jpg){kind=tracking}
![@kit3bus
git lfs track "*.png"
git lfs track "*.[pP][nN][gG]"
Case sensitive:
Case in-sensitive:
Git LFS
Gotchas](https://image.slidesharecdn.com/build-01-smithschneider-161009200441/85/Tracking-Huge-Files-with-Git-LFS-57-320.jpg){kind=tracking}
Tracking Huge Files with Git LFS
- 1. Tracking huge files with Git LFS LARS SCHNEIDER • GIT SOLUTIONS LEAD • AUTODESK • @KIT3BUS STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE
- 2. THE GIT LFS MODEL THE PROBLEM WITH BIG FILES Agenda MIGRATION GIT LFS PERSONAS THE GIT DATA MODEL
- 3. data model
- 4. $> git init $> tree .git/objects .git/objects ── info └── pack 2 directories
- 5. $> touch some-file.txt $> git add some-file.txt
- 6. $> tree .git/objects .git/objects ── e6 │ └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391 ── info └── pack 3 directories, 1 file zlib compressed SHA1
- 7. Type Chapter title here master 98ca9.. bab1e.. fad3d.. cat .git/refs/heads/master$ fad3dd41d0cf3d1b6aa2d8ad0549ab2fcb1575d1 “Directed Acyclic Graph”
- 8. master 98ca9.. bab1e.. fad3d.. 434bb..tree bab1e..parent Tim P <kannonboy@…> 1455209277 -0800committer Tim P <kannonboy@…> 1455209277 -0800author My life is my commit message. git cat-file -p 98ca9$
- 9. git cat-file -p 434bb ace23..100644 blob .gitignore dbdbd..100644 blob README.md a0bc3..040000 tree app 33d33..040000 tree config b1de7..100755 blob deploy-prod.sh 7011e..100755 blob deploy-staging.sh typefilemode SHA-1 master 98ca9.. bab1e.. fad3d.. $ 434bb..
- 17. (Large File Storage) Git LFS
- 21. git pull$ LFS store Git host ☞ ☞ ☞ dabad.. 98ca9.. bab1e.. fad3d.. 86753.. 434bb.. 4749d.. bdd12.. 778aa..
- 22. git checkout bab1e$ LFS store Git host ☞ ☞ ☞ dabad.. 98ca9.. bab1e.. fad3d.. 86753.. 434bb.. 4749d.. bdd12.. 778aa.. HEAD
- 23. https://git-lfs.github.com/spec/v1version sha256:325ddfb…oid 29342295size git cat-file -p 4749d$ ☞ ☞ ☞ dabad.. 98ca9.. bab1e.. fad3d.. 86753.. 434bb.. 4749d.. bdd12.. 778aa..
- 24. massive_video.mp4 Work tree dev .git/lfs/objects Clean filter (git-lfs clean) ☞ Index massive_video.mp4 $ .git/objects git add
- 25. $ dev .git/lfs/objects Smudge filter (git-lfs smudge) Work tree massive_video.mp4 ☞ Commit tree massive_video.mp4 .git/objects LFS Store git checkout
- 26. .git/lfs/objects .git/objects Hosted repo LFS store git push / pull
- 27. $ brew install git-lfs $ git lfs install
- 28. $ cat ~/.gitconfig [filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true
- 29. $ git lfs track “*.mp4” $ cat .gitattributes *.mp4 filter=lfs diff=lfs merge=lfs -text
- 30. @kit3bus Lars Schneider Autodesk Inc. Git and Git LFS contributor Technical Lead for Git at Autodesk
- 31. @kit3bus Who are we? • Best known for AutoCAD 2D and 3D computer-aided design • 33 years in business • 4000 engineers, hundreds of products, terabytes of code and asset data
- 32. @kit3bus Architecture, Engineering and Construction Image by Dave Tyner, Autodesk Plant 3D - P&ID
- 35. @kit3bus 3D Printing "Future of Making Things" Image courtesy of Local Motors Inc.
- 36. @kit3bus What do we use Git LFS for? Integration Test Data (3D Models, ...) Auxiliary Data (Documentation, Images, Videos, ...) Build Artifacts (not recommended)
- 37. @kit3bus DEVELOPER MIGRATOR ADMINISTRATOR - Git LFS Usage - What have we learned?
- 39. @kit3bus Migration Process 1 Identify an engineer with deep code knowledge Create a "demo" migration on Git migration server Iterate on "demo" migration until repo and CI are OK Ask broader team to "play" with the "demo" migration Perform migration on Git production server 2 3 4 5
- 41. @kit3bus How to migrate? git-svn / git-p4 / git-tfs ...
- 42. @kit3bus How to migrate? + git-svn / git-p4 / git-tfs ... git filter branch / git-lfs-migrate
- 43. @kit3bus How to migrate? + git-p4 git-svn / git-p4 / git-tfs ... git filter branch / git-lfs-migrate
- 44. @kit3bus Git LFS Migration Gotchas Discard large file history 1998 2007 2016 code code code +++
- 45. @kit3bus Git LFS Migration Gotchas Avoid "orphaned" LFS files after history rewrite LFS Ptr Git Repo LFS Storage
- 46. @kit3bus (INCLUDES DESIGNER, TESTER, ...) Developer
- 47. @kit3bus Teach why "Large" files are a problem! All history is local. Good for source files. Problem for large files.
- 48. @kit3bus What is a "problematic" file? Files that do not compress well...
- 49. @kit3bus What is a "problematic" file? ... and change frequently. Mon Tue Wed
- 50. @kit3bus What is a "problematic" file? Files smaller than 500kb are OK. Rule of Thumb
- 51. @kit3bus How to track Git LFS files? git lfs track "*.png"
- 52. @kit3bus How to track Git LFS files? git lfs track "*.png"
- 53. @kit3bus How to track Git LFS files? git lfs track "*.lfs.*" e.g. /images/elephant.lfs.png
- 54. @kit3bus How to track Git LFS files? git lfs track "/big/*" e.g. /big/elephant.png
- 55. @kit3bus How to track Git LFS files? git lfs track "/xxl.png"
- 56. @kit3bus How to track Git LFS files? Less than 1000 files in LFS are OK. Rule of Thumb Up to 70x speed improvement pending!
- 57. @kit3bus git lfs track "*.png" git lfs track "*.[pP][nN][gG]" Case sensitive: Case in-sensitive: Git LFS Gotchas
- 58. @kit3bus No line ending conversions on LFS files! Git LFS Gotchas
- 59. @kit3bus Use the latest Git / Git LFS version! Git LFS Tips & Tricks
- 60. @kit3bus Setup your Git credential helper (or use SSH)! Watch out for the "administrator" shell! Git LFS Tips & Tricks
- 61. @kit3bus git lfs clone <URL> Use Git 2.9+ if your Submodules contain Git LFS files. Git LFS Tips & Tricks
- 62. @kit3bus Git LFS Tips & Tricks Use Git Sparse Checkout if you have too many LFS files!
- 64. @kit3bus How to make sure Git LFS is used properly? Configure Git LFS on all platforms! Enterprise Config for Git https://git.io/vi1F4
- 65. @kit3bus How to make sure Git LFS is used properly? "What happens in Git, stays in Git."
- 66. @kit3bus How to make sure Git LFS is used properly? Rewriting history can cause a lot of trouble!
- 67. @kit3bus How to make sure Git LFS is used properly? Configure file size limit on Git server!
- 68. @kit3bus How to make sure Git LFS is used properly? Configure file size limit with local Git pre-commit hooks!
- 69. @kit3bus How to make sure Git LFS is used properly? Use code reviews and limit write access to shared branches! At least initially.
- 70. @kit3bus Takeaways • Git LFS works • Use the latest Git/Git LFS version • Use `git lfs clone` • Track problematic files in Git LFS • Reject problematic files in Git • Keep an eye on # of tracked files