blomp service down (error 500 / 503 / credentials declined / file missing

[BlompSupport]

Hey there, Blomper!

Could you do me a favor and just give me a quick check to see if that pesky issue is still hanging around?

Thanks a bunch!

[BlompSupport]

Hey there, Blomper!

Could you do me a favor and just give a quick check to see if that pesky issue is still hanging around?

Thanks a bunch!

[life8607]

Hi Blomp,

(other account is my 2nd premium, so also mine)
the issue that it crashes has been solves, so cheers to that!

i do seem to have the issue of files not always showing up, and then after sometimes a few minutes and sometimes multiple hours its back.
it seems everytime its weekend issues arise.

im getting the following error:

Code: Select all

2023/08/11 18:32:56 ERROR : IO error: HTTP Error: 503: 503 Service Unavailable
2023/08/11 18:38:20 ERROR : IO error: Object Not Found
2023/08/11 18:38:28 ERROR : IO error: Object Not Found
2023/08/11 18:38:31 ERROR : IO error: Object Not Found
2023/08/11 18:38:32 ERROR : IO error: Object Not Found
2023/08/11 18:39:59 ERROR : IO error: Object Not Found
2023/08/11 18:39:59 ERROR : IO error: Object Not Found
2023/08/11 18:39:59 ERROR : IO error: Object Not Found
2023/08/11 18:39:59 ERROR : IO error: Object Not Found

i know you guys are working hard to solve issues but this is a recurring issue that comes back everytime.
The connections are stable as this happens on 3 different servers on 3 different networks in 3 different data centers.
(a 50GB connection, a 10GB connection and a 1GB connection, all connections are fiber and in different country's with different ISP's)

i dont mind helping with troubleshooting as i like the motto you have with blomp, and the prices are so cheap compared to other storage providers, but if this issue keeps happening without any fix i will have to search for another as this is critical for backups, they keep getting interrupted due to the instabillity at blomp servers.


both authenticion URL's are giving the same error as described above, these are the tokens URL's i found and tried:

Code: Select all

https://authenticate.blomp.com
https://authenticate.ain.net

i am using rclone (i know not officialy supported) but the moment rclone crashes, the blomp dasboard also shows i have 0bytes in use and 0 files or im getting the error 503 service unavailable.

i've tried blomp go and via blomp go files are also unavailable. (purely for testing as all servers use ubuntu / debian kernal.)


the rclone config file looks as followed (and it works when dashboard is available):

Code: Select all

[blomp_subscription_2]
type = swift
user = EMAIL_HERE
key = VALIDATION_PASSWORD_HERE
auth = https://authenticate.blomp.com
#auth = https://authenticate.ain.net
tenant = storage
auth_version = 2
endpoint_type = public
leave_parts_on_error = true
chunk_size = 1P
no_chunk = false

[blomp_subscription_encrypted_2]
type = crypt
remote = blomp_subscription_2:EMAIL_HERE
password = ENCRYPTION_PASSWORD_HERE

the command i use for mounting is as followed:

Code: Select all

rclone mount blomp_subscription_2:EMAIL_HERE LOCAL_PATH_HERE --config rclone_config.conf --vfs-cache-mode writes --dir-cache-time 1h --vfs-cache-poll-interval 30s --vfs-read-chunk-size 1G --vfs-read-chunk-size-limit 50G --no-checksum --transfers 128

--vfs-cache-mode has to be writes, full is not possible with the local HDD capacity
--dir-cache-time 1h to not overload your servers but do detect changes on hourly basis
--vfs-cache-poll-interval is low due to file changs within a folder, so not the structure of it.
--vfs-read-chunk-size is 1GB to lower the amount of queries done to the blomp server
--vfs-read-chunk-size-limit is 50GB as backups will never reach 50GB and can then be pulled in a single request if needed (also to lower the queries towards your servers).
--no-checksum (dont really care for correct hash as everything is fully encrypted anyway)
-- transfers is at 128 due to the 50GB connection on 1 of the datacenter servers, to sync the files to your servers in a few seconds.


if you have some logs i can help you spit through them to find the issue if you like, or we can setup a call or something like that, if you see weird names without extentions in my blomp storage(like .rar, .zip etc) thats because everything is encrypted 4x before uploading and can only be decrypted by our servers with the decryption keys.

[BlompSupport]

Hey Blomper,

Hope you're doing well!

I wanted to drop you a quick note to say a huge thank you for taking the time to report the issue you encountered. Your proactive approach in letting us know is immensely appreciated, as it helps us ensure a smoother experience for everyone.

I wanted to assure you that we've already got our team on the case, diligently working to get things back on track. I'm sorry for any inconvenience this might have caused you. Your patience means a lot to us, and we're committed to resolving this as swiftly as we can.

I also wanted to take a moment to express how grateful we are for your amazing support and involvement within our community. Your contributions truly make a world of difference. Your technical know-how and willingness to assist are like a beacon guiding others through challenges. You're like a real Blomper team member!

Your presence enriches our community, and your insights have a significant impact. We're fortunate to have you on board, and we truly value your role as a valuable member.

If there's anything else you need or any ideas you'd like to share, please don't hesitate to reach out. We're here to make sure your experience is top-notch.

Wishing you a fantastic day ahead!

Warmest regards,

[BlompSupport]

Hi, again Blomper!

Great news coming your way! We've taken a close look at the issue you brought up, and it looks like things are sailing smoothly from our end.

Could you do us a favor and check if everything's back to normal on your side too? Your patience while we sorted this out is highly appreciated.

We're really grateful to have users like you who care about making Blomp an even more user-friendly and awesome place. Your contribution truly counts!

Thanks a bunch for hanging in there. If you ever have any more questions or ideas, we're all ears.

Wishing you a fantastic day ahead!

Warm regards,

[life8607]

Hi Team,

Thanks for coming back with an awnser, i appriciate it!
so the issue seems to be back

currently the error im getting is as followed:

Code: Select all

2023/08/14 07:19:49 ERROR : IO error: Object Not Found
2023/08/14 07:19:49 ERROR : IO error: Object Not Found
2023/08/14 07:19:50 ERROR : IO error: Object Not Found
2023/08/14 07:19:50 ERROR : IO error: Object Not Found
2023/08/14 07:19:51 ERROR : IO error: Object Not Found
2023/08/14 07:19:51 ERROR : IO error: Object Not Found
2023/08/14 07:19:58 ERROR : IO error: Object Not Found
2023/08/14 07:19:58 ERROR : IO error: Object Not Found
2023/08/14 07:20:03 ERROR : IO error: Object Not Found
2023/08/14 07:20:03 ERROR : IO error: Object Not Found
2023/08/14 07:20:05 ERROR : IO error: Object Not Found
2023/08/14 07:20:05 ERROR : IO error: Object Not Found
2023/08/14 07:20:10 ERROR : IO error: Object Not Found
2023/08/14 07:20:10 ERROR : IO error: Object Not Found
2023/08/14 07:20:25 ERROR : IO error: Object Not Found
2023/08/14 07:20:26 ERROR : IO error: Object Not Found
2023/08/14 07:20:32 ERROR : IO error: Object Not Found
2023/08/14 07:20:32 ERROR : IO error: Object Not Found
2023/08/14 07:21:24 ERROR : IO error: Object Not Found
2023/08/14 07:21:24 ERROR : IO error: Object Not Found
2023/08/14 07:21:25 ERROR : IO error: Object Not Found
2023/08/14 07:21:25 ERROR : IO error: Object Not Found
2023/08/14 07:21:25 ERROR : IO error: Object Not Found
2023/08/14 07:21:25 ERROR : IO error: Object Not Found
2023/08/14 07:21:29 ERROR : IO error: Object Not Found
2023/08/14 07:21:29 ERROR : IO error: Object Not Found
2023/08/14 07:21:36 ERROR : IO error: Object Not Found
2023/08/14 07:21:36 ERROR : IO error: Object Not Found
2023/08/14 07:21:41 ERROR : IO error: Object Not Found
2023/08/14 07:21:41 ERROR : IO error: Object Not Found
2023/08/14 07:22:37 ERROR : IO error: Object Not Found
2023/08/14 07:22:37 ERROR : IO error: Object Not Found
2023/08/14 07:22:53 ERROR : IO error: Object Not Found
2023/08/14 07:22:53 ERROR : IO error: Object Not Found
2023/08/14 07:23:19 ERROR : IO error: Object Not Found
2023/08/14 07:23:19 ERROR : IO error: Object Not Found
2023/08/14 07:24:29 ERROR : IO error: Object Not Found
2023/08/14 07:24:29 ERROR : IO error: Object Not Found
2023/08/14 07:55:29 ERROR : IO error: Object Not Found
2023/08/14 07:55:34 ERROR : IO error: Object Not Found
2023/08/14 07:56:12 ERROR : IO error: Object Not Found
2023/08/14 07:56:27 ERROR : ___REDACTED_FILE_NAME___: ReadFileHandle.Read error: low level retry 1/10: Object Not Found
2023/08/14 10:03:58 ERROR : ___REDACTED_FILE_NAME___: ReadFileHandle.Read error: low level retry 1/10: failed to authenticate decrypted block - bad password?
2023/08/14 10:11:11 ERROR : ___REDACTED_FILE_NAME___: ReadFileHandle.Read error: low level retry 1/10: failed to authenticate decrypted block - bad password?

PS: thanks for being on top of this, if you guys ever need some help let me know, i know a thing or to.

[life8607]

Hi just wanted to let you know the above message is stil valid, files have been unreachable for over 24h.

[BlompSupport]

Hello Blomper,

We really appreciate you bringing this issue to our attention. We want to assure you that we're actively addressing it and working hard to find a solution. Rest assured, we'll reach out as soon as we've got everything back on track.

We apologize for any inconvenience this might be causing, and I want to extend a heartfelt thank you for your valuable contribution to enhancing the user-friendliness of Blomp!

Have a good day.

[life8607]

Hi,

Thanks for getting back.
For now i've cancelled both subscriptions due to slow responses and unstable services.

If you guys keep the prices like they are, and you fix the mounting / server instability i'd reconsider you guys.
so for now im saying goodbye, ill keep an eye out on these posts on the forum to see if the issue is still there.

Once it seems fixed ill test it out and if all is stable for longer period ill reconsider coming back.

[life8607]

Hi,

So after being down for almost 36hours the mounts started to work again but im now missing a lot of files, and many files are corrupt.



im unable to transfer them locally due to blomp servers giving out errors.