I am trying to upload a new version of Coastal Callouts, which is around 1 GB total of files. The file upload is excruciatingly slow - it appears to be going about 1.5 Mbps. I have 100 Mbps fiber internet and no issues uploading to other services, so I suspect it's some sort of rate limiting on the site. Can you please look into this, it makes updating mods very painful. 

We have an issue open right now and the issue you are encountering is most likely being caused by this.

https://monitor.g17media.net/

 

It should be resolved shortly.

On 7/27/2020 at 3:27 AM, Cyan said:

We have an issue open right now and the issue you are encountering is most likely being caused by this.

https://monitor.g17media.net/

 

It should be resolved shortly.

 

I'm still having this issue. It's slightly better - uploading at about 5 Mbps now - but still unbearable for a 500 MB upload. 

 

It would also be nice if the upload/update page prompted you with a confirmation dialog when closing the tab. I've wasted hours of slow upload time by accidentally closing a tab that was slowly uploading a massive file. 

Edited August 19, 20205 yr by PNWParksFan

I'm going to try and look at improving these things.

One of the main issues that can cause the speed loss is that if the site has experienced a DDoS attack, as part of the mitigation flow which will persist even a few hours into the attack, individual connections can be limited in upload speed. This is something we can sort out with changes to how uploads are done, but will need some custom logic as our software doesn't support this by default. It's on my roadmap though.

As for the confirmation, this is something I can probably add easily and you should see this soon.

@Cyan I just tried uploading another update to Coastal Callouts, which is a ~600 MB file. I have a 100 Mbps symmetrical fiber connection, so it should be very fast on my end. It took about 45 minutes to actually upload. So whatever issues are causing slow upload speeds still appear to be occurring. 

I think we've improved this as much as we can on the server-side of things. I think we've pin-pointed the issue down to our DDoS protection, which makes use of Cloudflare Magic Transit.

Due to the way it works, inbound traffic is rate-limited and because this cannot easily be inspected by the network-level protection (due to encryption), the rate-limiting is indiscriminately applied to all traffic algorithmically. Unfortunately this is not configurable or tune-able for us.

 

I am investigating a solution that would effectively change the route that file uploads take to the site so that they go through a different solution without the aggressive rate-limiting. The problem is that whilst the ability to change the upload host natively existed in previous versions of the community software we use, this was removed at some point, so this will have to be built in-house. I will be honest I completely forgot about adding the confirmation dialog you suggested, and I'll see if I can rush this through.

1 hour ago, Cyan said:

I think we've improved this as much as we can on the server-side of things. I think we've pin-pointed the issue down to our DDoS protection, which makes use of Cloudflare Magic Transit.

Due to the way it works, inbound traffic is rate-limited and because this cannot easily be inspected by the network-level protection (due to encryption), the rate-limiting is indiscriminately applied to all traffic algorithmically. Unfortunately this is not configurable or tune-able for us.

 

I am investigating a solution that would effectively change the route that file uploads take to the site so that they go through a different solution without the aggressive rate-limiting. The problem is that whilst the ability to change the upload host natively existed in previous versions of the community software we use, this was removed at some point, so this will have to be built in-house. I will be honest I completely forgot about adding the confirmation dialog you suggested, and I'll see if I can rush this through.

 

Thanks for taking a look at it! Fortunately it's only a problem every once in a while - most users probably aren't unloading hundreds of MB frequently - but whenever it does happen it's a big frustration, so I appreciate you continuing to look for solutions. 

There is now confirmation to prevent accidentally navigating away from the file upload process. Although this isn't a complete fix, I'm hoping this at least makes this a little less painful until a full fix is in place.

@Cyan the upload process is still painfully slow for large files. It takes me several hours to get a full update to Coastal Callouts uploaded. It failed about 20% of the way through a 600 MB file today. Looking at the Task Manager, I'm only getting about 2 Mbps upload speed. I have fiber internet with 100 Mbps upload so it's not an issue on my end. Always makes me dread updating. 😭

On 8/8/2021 at 3:05 AM, PNWParksFan said:

@Cyan the upload process is still painfully slow for large files. It takes me several hours to get a full update to Coastal Callouts uploaded. It failed about 20% of the way through a 600 MB file today. Looking at the Task Manager, I'm only getting about 2 Mbps upload speed. I have fiber internet with 100 Mbps upload so it's not an issue on my end. Always makes me dread updating. 😭

I'm sorry this hasn't been fixed yet. The issue is definitely still not on your side and is still related to how the DDoS protection upload rate limiting works.

 

I was working on support for alternative upload endpoints but this was placed on the back burner a few months ago after I was informed that the upload speed issues would be fixed by upstream provider changes.

 

These don't seem to have materialised. We are probably a week or so away from deploying a version upgrade of the site software, I will try and see if I can finish the alternative upload endpoints as part of that update, that will fix this issue.

On 8/8/2021 at 3:05 AM, PNWParksFan said:

@Cyan the upload process is still painfully slow for large files. It takes me several hours to get a full update to Coastal Callouts uploaded. It failed about 20% of the way through a 600 MB file today. Looking at the Task Manager, I'm only getting about 2 Mbps upload speed. I have fiber internet with 100 Mbps upload so it's not an issue on my end. Always makes me dread updating. 😭

I'm happy to say we've implemented the changes I mentioned above in our staging environment and it will be in the next website update (we are just applying finishing touches).

I'm confident this should resolve the majority of the upload speed problems.

 

The changes I talked about are now live, and you should now see a really large performance uplift in uploading. Please let me know if you are still experiencing issues.