swift errors with rclone?

[newell]

I'm using rclone to sync my borgbackup files hourly to blomp (and other services). Sometimes I can go for days without an error, sometimes just a few hours. I wiped the blomp directory and started fresh earlier today, and a few hours later I'm already seeing I/O errors and files that can't be deleted. It feels like the filesystem on your end might be corrupt.

Here's a snippet of the rclone output showing both a sync and a check. (Times are in UTC, of course.)

2023-08-20, 23:04:17: rclone to blomp
2023/08/20 23:04:20 INFO : hints.123290: Copied (new)
2023/08/20 23:04:20 INFO : integrity.123290: Copied (new)
2023/08/20 23:04:20 INFO : config: Updated modification time in destination
2023/08/20 23:04:21 INFO : data/123/123287: Copied (new)
2023/08/20 23:04:21 INFO : index.123290: Copied (new)
2023/08/20 23:04:21 INFO : nonce: Copied (replaced existing)
2023/08/20 23:04:21 INFO : data/123/123290: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123289: Copied (new)
2023/08/20 23:04:22 INFO : data/123/123278: Copied (new)
2023/08/20 23:04:34 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:04:45 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:04:45 ERROR : Attempt 1/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:00 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:10 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:10 ERROR : Attempt 2/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:19 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 18.386 KiB/s, ETA 0s
Checks: 3020 / 3767, 80%
Transferred: 8 / 8, 100%
Elapsed time: 1m2.2s
Checking:
* data/121/121485: checking
* data/122/122386: checking
* data/122/122437: checking
* data/122/122664: checking
* data/122/122701: checking
* data/122/122778: checking
* data/122/122791: checking
* data/122/122855: checking

2023/08/20 23:05:23 ERROR : data/28/28877: Failed to set modification time: object not found
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting files as there were IO errors
2023/08/20 23:05:34 ERROR : Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: not deleting directories as there were IO errors
2023/08/20 23:05:34 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
2023/08/20 23:05:34 INFO :
Transferred: 4.608 MiB / 4.608 MiB, 100%, 15.148 KiB/s, ETA 0s
Errors: 2 (retrying may help)
Checks: 3767 / 3767, 100%
Transferred: 8 / 8, 100%
Elapsed time: 1m16.9s

2023/08/20 23:05:34 Failed to sync with 2 errors: last error was: not deleting files as there were IO errors


2023-08-20, 23:05:34: rclone check blomp
2023/08/20 23:05:36 INFO : Using md5 for hash comparisons
2023/08/20 23:05:36 ERROR : hints.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : index.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:36 ERROR : integrity.123277: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/122/122959: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123274: File not in Local file system at /opt/borg
2023/08/20 23:05:44 ERROR : data/123/123277: File not in Local file system at /opt/borg
2023/08/20 23:06:01 NOTICE: Local file system at /opt/borg: 6 files missing
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 differences found
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 6 errors while checking
2023/08/20 23:06:01 NOTICE: Swift container blomp@*REDACTED*.com path tx-*REDACTED*-com: 1258 matching files
2023/08/20 23:06:01 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 6 (retrying may help)
Checks: 1258 / 1258, 100%
Elapsed time: 27.4s

2023/08/20 23:06:01 Failed to check with 6 errors: last error was: 6 differences found

Thoughts?

[BlompSupport]

Hello there, Blomper!

We really appreciate you letting us know about this concern. Our team is already on it and we're here to help.

Thanks a bunch for being such a valuable member of our community.

Wishing you a fantastic day ahead, because you're awesome just the way you are!

[BlompSupport]

We just uploaded a new version that fixed a lot of these issues. Do you have the most recent version?

[BlompSupport]

Hey there, Blomper!

Hope you're having a fantastic Friday!

We've noticed we haven't heard from you in a while, and we just wanted to reach out and remind you that we're always here to assist you with any questions or concerns you might have about our services.

We noticed you opened a ticket regarding an issue with rclone, and we were wondering if you're still experiencing any difficulties with it. If you are, please don't hesitate to let us know. You can share more details and even some screenshots if that helps.

On the other hand, if everything's going smoothly, we want to extend our heartfelt thanks for choosing Blomp! We hope you have an amazing weekend ahead.

Warm regards,