Jump to content


Photo

[Website Outages] - Image Attachment Issues


  • Please log in to reply
39 replies to this topic

#1 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 10 May 2017 - 04:01 PM

Hi Folks,

It looks like we've got an issue with image uploads. May have run out of disk space on the server. Will need to look into it tonight to see where all the space is going.

 

It says the site is currently taking up 16gb of space.



#2 Retro Z

Retro Z

    The 1000+ club

  • Members
  • PipPipPipPipPipPip
  • 1,422 posts
  • Location:Melbourne, western suburbs
  • Tagline:73' 240z and 72’ 240z

Posted 10 May 2017 - 04:14 PM

I always use a image hosting website to help out with that.

#3 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 10 May 2017 - 06:41 PM

Yeah problem is third party images often end up broken and that nullifies a thread.
  • Andrew_L26 likes this

#4 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 10 May 2017 - 08:38 PM

Update:

 

/forum/upload folder is using 14GB of data :(.

 

Not sure what options I have, looks like upgrade (higher monthly expense) or move hosts.

 

I think next plan up might be $200 p/month (USD), but offers 100gb of space.

 

https://www.rochen.com/managed/cloud/



#5 Riceburner

Riceburner

    The 2000+ club

  • Members
  • PipPipPipPipPipPipPip
  • 3,284 posts
  • Location:Melbourne, Victoria

Posted 10 May 2017 - 09:39 PM

What about an annual membership fee?


  • Andrew_L26 likes this

#6 Mr Camouflage

Mr Camouflage

    Senior Member

  • Members
  • PipPipPipPipPip
  • 986 posts
  • Website:http://www.wazcars.com
  • Location:Perth
  • Tagline:www.wazcars.com

Posted 11 May 2017 - 02:22 AM

Some of the member galleries have massive files in them 4000px wide ~2mb each.

 

If they were resized to say 2000px wide and 70% quality they'd only be around 600kb.

 

eg:

http://www.viczcar.c...383-perth-240z/

 

http://www.viczcar.c...t50n-rb30et-22/

 

http://www.viczcar.c...sun-450zx-2017/

 

Does this forum have a plugin that can resize the the attachments?

 

If not I have a php script that can do it.



#7 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 11 May 2017 - 08:10 AM

Some of the member galleries have massive files in them 4000px wide ~2mb each.

 

If they were resized to say 2000px wide and 70% quality they'd only be around 600kb.

 

eg:

http://www.viczcar.c...383-perth-240z/

 

http://www.viczcar.c...t50n-rb30et-22/

 

http://www.viczcar.c...sun-450zx-2017/

 

Does this forum have a plugin that can resize the the attachments?

 

If not I have a php script that can do it.

 

Yes I need to review the gallery compression, however I think the gallery is a much smaller portion of space than the uploads in general (forum attachments) and generally I set the resolution of those quite low. I'll need to review again, in the short term I may upgrade our hosting package as we're on an older legacy plan and there is no direct upgrade for this plan, next step up is 100gb. From there every 5gb is another $5 p/month.

 

However I will also look at a local host option, going to need to run some forum updates also so might try and do it all in 1 hit. Just feeling time poor lately.



#8 Rickers

Rickers

    Advanced Member

  • Members
  • PipPipPip
  • 150 posts
  • Location:Melbourne, Australia

Posted 11 May 2017 - 08:59 AM

Have you thought of external storage? Amazon S3's seem like a viable option.

>Write some script to keep track of /forum/upload/ storage space
>When 5gb? 10gb? is exceeded, 'archive' the X images that have been around the longest.
   -(i.e. push the X images to the Amazon S3 that have the earliest upload timestamp. Obviously wait until a downtime (i.e. 2am on Sunday?)).
>Write a simple API to take care of "Post" and "Get" to transfer the images between your current webserver and the S3
>If someone requests an image that isn't on the webserver, "Get" the image from the S3 and store it back on the webserver.

**Could also add a little code to reset the timestamp of an image when it has been accessed (so the most popular images never leave the webserver).

We only use our own physical hardware at work so I might be barking up the wrong tree... (i.e. it may be to harsh on the network to transfer X images).



#9 44014

44014

    Senior Member

  • Members
  • PipPipPipPipPip
  • 817 posts
  • Location:Brisbane
  • Tagline:Garage San Maru

Posted 11 May 2017 - 09:00 AM

Talk to Jake, He basically does this for a living and can probably assist in finding the best deal for you Gav.



#10 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 11 May 2017 - 09:18 AM

Talk to Jake, He basically does this for a living and can probably assist in finding the best deal for you Gav.


Pinged him last night on the Facebook 😁. We chatted in the past about hosting the site.

#11 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 12 May 2017 - 10:22 AM

Have you thought of external storage? Amazon S3's seem like a viable option.

>Write some script to keep track of /forum/upload/ storage space
>When 5gb? 10gb? is exceeded, 'archive' the X images that have been around the longest.
   -(i.e. push the X images to the Amazon S3 that have the earliest upload timestamp. Obviously wait until a downtime (i.e. 2am on Sunday?)).
>Write a simple API to take care of "Post" and "Get" to transfer the images between your current webserver and the S3
>If someone requests an image that isn't on the webserver, "Get" the image from the S3 and store it back on the webserver.

**Could also add a little code to reset the timestamp of an image when it has been accessed (so the most popular images never leave the webserver).

We only use our own physical hardware at work so I might be barking up the wrong tree... (i.e. it may be to harsh on the network to transfer X images).

 

 

So there is a couple of options, Invision Power (the forum software) has their own CDN service I could use, in fact they also offer to host the forum. I originally was hesitant to do that but the benefit of having them manage the forum is that they perform software updates on your behalf (I don't have to run upgrades) and I may be able to place the forum on a sub-domain so forum.auszcar.com which is hosted remotely.

 

You can also leverage Amazons AWS (CDN) via a config in the forum. I've never done it.

 

1 thing that isn't clear to me re: CDN's is if the images are all stored and hosted remotely or do they originally get stored locally (on this web-server) and then shared / sent to different data centers (and mirrored)? Ideally I can have all static images/files stored and retrieved using the third party CDN - since their storage is massive and dirt cheap and backed up for redundancy across data centers.

 

I will need to look into all of this. I suspect it would save A LOT of money in hosting costs..just never configured any of this and don't know exactly how it works in practice.



#12 Rickers

Rickers

    Advanced Member

  • Members
  • PipPipPip
  • 150 posts
  • Location:Melbourne, Australia

Posted 12 May 2017 - 12:22 PM

My statements below are from a good 30 minutes of research - so take them with a grain of salt. Further investigating should be done before attempting anything..

 

I'm not really sure what your best bet is in terms of CDN, I haven't heard anything about maxCDN (IpBoards CDN) and my knowledge of CloudFront (Amazon CDN) is very limited. I think I would be more inclined to go with CloudFront, simply because of how established they are (i.e. customer service should be able to answer any question with ease, probably more data server locations.. etc.).

 

I'm not really sure about this - but I think CDN's are designed to be used alongside storage servers (i.e. Amazon S3 stores the data and then CloudFront takes copies of the files and caches them in data servers around the world for faster access). Storing files locally on the webserver seems a bit counter-intuitive (i.e. it won't solve your storage issues).

 

So the process flow would be something like:

  1. User uploads file - IPBoards config tells the webservice to upload it to a storage server (i.e. Amazon S3 - bucket).
  2. The file is then replicated (or copied) into the caches of the different data servers that you specified when creating your CDN (i.e. CloudFront).
  3. User accesses file - IPBoards config tells the webservice to use the CDN to access the data (so the cache file is retrieved from the closest source).
  4. User is provided with file.

 

Here is a few helpful links:

Setting up Amazon S3 and CloudFront

Setting IPBoards to upload attachments to S3

Discussiong about IPBoards and using CloudFront (mentions how to set CDN).


  • gav240z likes this

#13 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 12 May 2017 - 12:31 PM

Thanks, yes I'll need to do more research and speak with Invision Power themselves about how it all works together.



#14 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 13 May 2017 - 08:41 AM

Hi folks,
The site is being migrated to a new hosting plan this weekend which will cause some minor disruption as things progress.

Once migrated image uploading should work again.

They managed to upgrade us for $25 USD extra per month which was good. However this is just a short term solution and I plan to look at other options locally also.
  • PB260Z likes this

#15 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 14 May 2017 - 05:39 PM

Image upload test.

Attached Thumbnails

  • IMG_20170514_161709.jpg


#16 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,324 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 14 May 2017 - 06:26 PM

Hi

Your garage / workshop is looking very tidy, way tidier than mine anyway.

Cheers

#17 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 14 May 2017 - 06:27 PM

So looks like we're back in business, I had to make changes to the DNS (domain name server) settings to point to the new hosting environment.

 

I don't know if it's just me but it does feel a bit quicker. Here's a run down of the upgraded server.

 

 

Dear Gavin, 

Last year Rochen launched Managed Cloud Servers (MCS) as an upgrade to Managed Virtual Servers (MVS), which you are currently utilizing. While we are no longer offering MVS to new customers we will continue to support the platform for existing customers.
 

MCS does have several significant advantages over MVS though: -

  • High Availability – Unlike with MVS the MCS platform is built on top of VMWare vSphere and takes advantage of a feature called vMotion. This means that if the underlying physical hardware fails then your MCS will simply restart on backup hardware within a few minutes.
  • Highly Scalable – A single MCS can scale up to 32 vCPUs, 156GB DDR4 RAM and almost limitless SSD-Accelerated SAN storage. Not only is the platform more scalable than MVS it is more scalable than most dedicated servers. Plus you can handle scaling yourself (up or down) with a few clicks via the My Rochen Portal. We also include options for automatic and scheduled scaling.
  • Upgraded Resources and Features – The base configuration for MCS provides 100% more CPU power (2 vCPUs), 400% more RAM (4GB DDR4 RAM) and 200% more storage (100GB SSD-Accelerated SAN storage) than MVS. MCS is fully managed and includes all of the features of MVS and more. Scaling beyond the base configuration is also more affordable.
  • Web Application Firewall (WAF) – The additional resources allow for features like a WAF with specific rulesets for Joomla, Drupal and WordPress to operate and further protect your websites.

 

I need to run some website updates / upgrades also, so will make some backup's and try and do a local test first before performing on the live site. Just a matter of finding time. :(.



#18 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,324 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 14 May 2017 - 06:29 PM

Yeah, does seem to be running fastervthan usualy.
But I don't want to jinx things

#19 Jords

Jords

    Member

  • Members
  • PipPip
  • 26 posts
  • Location:Manly

Posted 15 May 2017 - 03:25 PM

Hi, is anyone having issues with not being able to connect to the site through OPTUS broadband? The site was working fine this morning and I can access thru my mobile on Telstra?

If the DNS settings have changed maybe the new settings haven't replicated?

#20 gav240z

gav240z

    Administrator

  • Administrators
  • 13,249 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 15 May 2017 - 03:33 PM

Hi, is anyone having issues with not being able to connect to the site through OPTUS broadband? The site was working fine this morning and I can access thru my mobile on Telstra?

If the DNS settings have changed maybe the new settings haven't replicated?

 

It's possible, I believe it can take anywhere from 24-48 hours to fully propagate through the web. Not sure how often Optus refreshes their DNS cache? If you can access the site through another proxy it's probably down to that? 






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users