github/restic
4
0
Fork 0
mirror of https://github.com/restic/restic.git synced 2026-10-19 16:15:41 +00:00
Code Issues Releases Wiki Activity
9,927 Commits 11 Branches 53 Tags
master
Commit Graph
94 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Weiss 947f0c345e correct typos 2023-05-01 14:56:42 +02:00
Alexander Weiss d23a2e1925 better error handling and correct nil tree behavior 2023-05-01 14:56:42 +02:00
Alexander Weiss 08ae708b3b make linter happy 2023-05-01 14:56:42 +02:00
Alexander Weiss 99a05d5ab2 Update troubleshooting documentation 2023-05-01 14:56:42 +02:00
Alexander Weiss 6557f36f61 Add changelog and docu for #2876 2023-05-01 14:56:42 +02:00
Alexander Weiss 5f58797ba7 Add repair command 2023-05-01 14:56:42 +02:00
Alexander Weiss 9cef6b4c69 Add troubleshooting documentation 2023-05-01 14:56:42 +02:00
Alexander Weiss 1751afae26 Make snapshots dirs in mount command customizable 2022-08-05 23:46:46 +02:00
Alexander Weiss 57f4003f2f Generalize fuse snapshot dirs implemetation 
+ allow "/" in tags and snapshot template
 2022-08-05 23:46:46 +02:00
Alexander Weiss 696c18e031 Add possibility to set snapshot ID (used in test) 2022-08-05 23:46:46 +02:00
Alexander Weiss 7643237da5 prune: separate collecting/printing/pruning 2022-07-30 17:37:07 +02:00
Alexander Weiss 7478cbf70e prune: Enhance treatment of duplicates 2022-07-17 00:22:23 +02:00
Alexander Weiss bce87922c0 Improve recover 
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
 2021-09-03 21:36:57 +02:00
Alexander Weiss 81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Alexander Weiss 780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss d107a2cfdf Separate dry run tests 2021-08-04 21:19:29 +02:00
Alexander Weiss 38a8a48a25 Simplify dry run backend 2021-08-04 21:19:29 +02:00
Alexander Weiss 39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Alexander Weiss 6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss 5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss 573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Alexander Weiss e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Weiss d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Alexander Weiss 34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss 7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss 07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss 3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Weiss f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss f6df94a50e prune: Add self-healing 
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
 2020-12-29 20:20:05 +01:00
Alexander Weiss 2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
Alexander Weiss d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss 68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Alexander Weiss d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Alexander Weiss e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss 26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss 5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
Alexander Weiss 83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Weiss aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss 92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Weiss 67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss 4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
Alexander Weiss cb5ec7ea6b Fix statistics in prune for duplicates 
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
 2020-11-18 22:30:22 +01:00
Alexander Weiss 9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss 3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss 30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss 187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss 1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss 5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss 43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
Alexander Weiss 7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss 17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss 80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
Alexander Weiss 239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
Alexander Weiss 826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
Alexander Weiss d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss 38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Weiss aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Weiss b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss 7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss 3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss 581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Weiss b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
Alexander Weiss 6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
Alexander Weiss 9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
Alexander Weiss b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss 5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
Alexander Weiss 9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss 0eb8553c87 add changelog for #2840 2020-08-01 20:43:18 +02:00
Alexander Weiss d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss 1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss 2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Weiss f8316948d1 Optimize FUSE - make command restic mount faster and consume less memory 
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
 2020-07-28 23:01:18 +02:00
Alexander Weiss 9d1fb94c6c make Lookup() return all blobs 
+ simplify syntax
 2020-07-25 21:18:34 +02:00
Alexander Weiss 9175795fdb Check contents in archiver 
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
 2020-07-25 08:18:28 +02:00
Alexander Weiss a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Alexander Weiss 3b7a3711e6 Add more realistic index benchmarks 
+ reduce test size of BenchmarkMasterIndexLookupParallel
 2020-07-21 07:18:20 +02:00
Alexander Weiss b5543cff5d Loop over index files for 'list blobs' 
=> reduces memory consumption a lot!
 2020-06-14 17:25:34 +02:00
Alexander Weiss 1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss ce4a2f4ca6 save packIDs and duplicates separately 
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
 2020-06-14 07:56:21 +02:00
Alexander Weiss cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Alexander Weiss 7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
Alexander Weiss 70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00
Alexander Weiss 91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
Alexander Weiss 8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
Alexander Weiss 9a9101d144 Support specifying multiple host flags for various commands 
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
 2020-02-26 22:17:59 +01:00