swift errors with rclone?
Posted: Sun Aug 20, 2023 8:39 pm
I'm using rclone to sync my borgbackup files hourly to blomp (and other services). Sometimes I can go for days without an error, sometimes just a few hours. I wiped the blomp directory and started fresh earlier today, and a few hours later I'm already seeing I/O errors and files that can't be deleted. It feels like the filesystem on your end might be corrupt.
Here's a snippet of the rclone output showing both a sync and a check. (Times are in UTC, of course.)
2023-08-20, 23:04:17: rclone to blomp
2023/08/20 23:04:20 INFO : hints.123290: Copied (new)
2023/08/20 23:04:20 INFO : integrity.123290: Copied (new)
2023/08/20 23:04:20 INFO : config: Updated modification time in destination
2023/08/20 23:04:21 INFO : data/123/123287: Copied (new)
2023/08/20 23:04:21 INFO : index.123290: Copied (new)
2023/08/20 23:04:21 INFO : nonce: Copied (replaced existing)
2023/08/20 23:04:21 INFO : data/123/123290: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123289: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123278: Copied (new)
2023/08/20 23:04:34 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:04:45 ERROR : Attempt 1/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:00 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:10 ERROR : Attempt 2/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:19 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 18.386 KiB/s, ETA 0s
Checks: 3020 / 3767, 80%
Transferred: 8 / 8, 100%
Elapsed time: 1m2.2s
Checking:
* data/121/121485: checking
* data/122/122386: checking
* data/122/122437: checking
* data/122/122664: checking
* data/122/122701: checking
* data/122/122778: checking
* data/122/122791: checking
* data/122/122855: checking
2023/08/20 23:05:23 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:34 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:34 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 15.148 KiB/s, ETA 0s
Errors: 2 (retrying may help)
Checks: 3767 / 3767, 100%
Transferred: 8 / 8, 100%
Elapsed time: 1m16.9s
2023/08/20 23:05:34 Failed to sync with 2 errors: last error was: not deleting files as there were IO errors
2023-08-20, 23:05:34: rclone check blomp
2023/08/20 23:05:36 INFO : Using md5 for hash comparisons
2023/08/20 23:05:36 ERROR : hints.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : index.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : integrity.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/122/122959: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123274: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123277: File not in Local file system at /opt/borg
2023/08/20 23:06:01 NOTICE: Local file system at /opt/borg: 6 files missing
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 differences found
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 errors while checking
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 1258 matching files
2023/08/20 23:06:01 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 6 (retrying may help)
Checks: 1258 / 1258, 100%
Elapsed time: 27.4s
2023/08/20 23:06:01 Failed to check with 6 errors: last error was: 6 differences found
Thoughts?
Here's a snippet of the rclone output showing both a sync and a check. (Times are in UTC, of course.)
2023-08-20, 23:04:17: rclone to blomp
2023/08/20 23:04:20 INFO : hints.123290: Copied (new)
2023/08/20 23:04:20 INFO : integrity.123290: Copied (new)
2023/08/20 23:04:20 INFO : config: Updated modification time in destination
2023/08/20 23:04:21 INFO : data/123/123287: Copied (new)
2023/08/20 23:04:21 INFO : index.123290: Copied (new)
2023/08/20 23:04:21 INFO : nonce: Copied (replaced existing)
2023/08/20 23:04:21 INFO : data/123/123290: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123289: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123278: Copied (new)
2023/08/20 23:04:34 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:04:45 ERROR : Attempt 1/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:00 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:10 ERROR : Attempt 2/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:19 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 18.386 KiB/s, ETA 0s
Checks: 3020 / 3767, 80%
Transferred: 8 / 8, 100%
Elapsed time: 1m2.2s
Checking:
* data/121/121485: checking
* data/122/122386: checking
* data/122/122437: checking
* data/122/122664: checking
* data/122/122701: checking
* data/122/122778: checking
* data/122/122791: checking
* data/122/122855: checking
2023/08/20 23:05:23 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:34 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:34 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 15.148 KiB/s, ETA 0s
Errors: 2 (retrying may help)
Checks: 3767 / 3767, 100%
Transferred: 8 / 8, 100%
Elapsed time: 1m16.9s
2023/08/20 23:05:34 Failed to sync with 2 errors: last error was: not deleting files as there were IO errors
2023-08-20, 23:05:34: rclone check blomp
2023/08/20 23:05:36 INFO : Using md5 for hash comparisons
2023/08/20 23:05:36 ERROR : hints.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : index.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : integrity.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/122/122959: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123274: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123277: File not in Local file system at /opt/borg
2023/08/20 23:06:01 NOTICE: Local file system at /opt/borg: 6 files missing
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 differences found
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 errors while checking
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 1258 matching files
2023/08/20 23:06:01 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 6 (retrying may help)
Checks: 1258 / 1258, 100%
Elapsed time: 27.4s
2023/08/20 23:06:01 Failed to check with 6 errors: last error was: 6 differences found
Thoughts?