Jump to content


Photo

MAKE A DONATION to maintain this site - See the 'Home' page for details


  • Please log in to reply
64 replies to this topic

#21 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 16 May 2015 - 12:58 PM

Migrating a live site to a new bit of software whilst trying to keep the website up is a bit like this...



#22 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,237 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 16 May 2015 - 08:58 PM

Gdday

I had no idea how much work you put in to keep this forum running.

Well done on a top job, I for one appreciate it.

Cheers

PB

#23 boyblunda

boyblunda

    Advanced Member

  • Members
  • PipPipPip
  • 105 posts
  • Location:Youngs Siding Western Australia

Posted 17 May 2015 - 12:08 AM

Quite right PB.

If Gav was to put the time into his own projects that he currently puts into maintaining and developing this site, he would probably be a lot further advanced with the multiple car interests he has.

The same can also be said for the significant contributions made by moderators and people who make appropriate positive comment to the threads eg. I am hugely impressed and grateful with the detail, care and commitment of the comments that HS30-H makes to this forum.

Gav, if you are needing a bit extra for technical development, then let us know. I am sure that many of us would respond to such a call.

To all of those who are donating members, I am sure that every bit of the support we contribute is also appreciated.

Thanks again Gav - great job.

#24 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 17 May 2015 - 11:48 AM

Gdday

I had no idea how much work you put in to keep this forum running.

Well done on a top job, I for one appreciate it.

Cheers

PB


In fairness most of the time the site runs itself, but upgrades and maintenance can be a pain given that the site is using 3 different bits of software all developed by different groups. Probably something that comes up often is I have to keep increasing the space allocated to the server for all the images / attachments. We are at around 7gig worth of attachments and that's with relatively restrictive image sizes for attachments.

The video I posted was more of a joke, at Google I've heard engineers say that making major changes to a bit of software while maintaining uptime is a bit like changing the engine in a Boeing 787 mid flight. In this case it's not quite that bad because I'm doing a lot of the experimentation with the migration process offline, but the trick is to iron out all the kinks before uploading to the live environment. Making sure that when people log in for the first time the experience is relatively seamless (apart from the learning curve of the new software) and that everything is working as intended.

I've been working on it on and off the last couple of weeks as free time allows and as mentioned unfortunately very little has happened on the car, but on the positive side once we are on the new software I'll be able to whittle it down from 3 independent bits of software to 1 piece and all upgrades from that point on will be 1 piece of software. It also means making changes will involve 1 set of templates and I'll be able to more easily delegate roles to different parts of the website. So I'll be able to give some individuals access to publish content to to the homepage more easily etc..

In the long term it will also free up more time. It might seem silly but I'm actually looking forward to the time we move to Auszcar.com - which I've put off forever given the complexity of the move with 3 different bits of software.

#25 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 17 May 2015 - 11:42 PM

Well might have had a break-through in terms of the avatars saga...

http://www.simplemac...?topic=197938.0
http://wiki.simplema...atars_directory

Seems like SMF does some odd storage mechanism for avatars, unfortunately (I wasn't aware) but it looks like all avatar requests cause a database query! Which is ridiculous, had I known I'd have done something about it. Now that I do know I realise the site would have been even faster...

I hope this is the fix I need to make sure all avatars move across with the migration.

#26 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 18 May 2015 - 12:09 AM

Ok after testing this locally, I just ran this command on the server and it seems page load time has increased significantly for forum threads where multiple avatars are displayed. Wondering if anyone else has noticed increased responsiveness?

#27 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 18 May 2015 - 11:52 AM

1 more piece of the puzzle solved, looks like Avatars now move across more consistently, there is a few that are broken but the majority say 98% will move across. Yay!

Attached Thumbnails

  • 46072_3b25ae971355b269637a49ef853081c445b1cf76.png


#28 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,237 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 18 May 2015 - 08:12 PM

In fairness most of the time the site runs itself


Way to modest mate  :)

#29 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 18 May 2015 - 10:28 PM

Way to modest mate  :)


Thanks mate, does the site seem any quicker to you today? In terms of response and page load times?

#30 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 18 May 2015 - 10:41 PM

Gav I work in IT - in corporate lending so worlds apart from web but does involve some database stuff and APIs etc.  Let me know if there's anything I can lend a hand with.


Hi mate,
I have a question for you re: SQL.

I've just converted the Gallery over and it seems to be working fine, except for the fact that each album that was previously created had an owner assigned to it. Now however the owners are blank (as in not defined).

The old Coppermine Gallery table (cpg132_albums) has a couple of columns:

1. title = Album name
2. owner = member id

The new Invision Power Gallery table (ips_gallery_albums) has the following corresponding columns:

1. album_name = Album name
2. album_owner_id = member id

Where the left hand value before the = sign is the database column name.

What I need to do is some kind of lookup for the old table, where I extract the title and owner columns and then insert them into the album_name and album_owner_id columns.

Although interestingly I tried do manually update the album_owner_id value via phpmyadmin UI and it didn't seem to work... I'll look into that.

#31 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,237 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 19 May 2015 - 09:12 PM

Thanks mate, does the site seem any quicker to you today? In terms of response and page load times?


Been out and about today and yes the site does seem faster  :)
Using iPad on Optus

#32 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 19 May 2015 - 11:16 PM

Really just putting this here for my own sake, but just in case anyone else is interested.

I noticed that after converting the photo albums across to the new software that the albums lost their owners, along with comments etc..

Turns out that when you 'bridge' software (in this case Coppermine gallery and SMF forum) Coppermine defers to the SMF users tables (they share the same user tables) rather than creating their own independent tables.

So what's the problem? This means Coppermines own user tables are blank (no users in them) the script to convert Coppermine to Invision Power Suite's Gallery can't convert these users because they don't exist in the table.

Compounding the problem is the fact that when you convert from the old forum software to the new software a members unique ID changes, so a member with ID 2 before becomes 2048 or similar. I've been doing a few join statements in SQL to figure out how I can look up a specific user by ID and name and see how their ID has changed in the old system to the new, I'm hoping to use this information to update the database with the correct owners of albums etc..

Once I figure that out, I think that's the last major hurdle. Then I can upload a test version for people to experiment with and see if any issues come up before uploading and transitioning over.

This looks like something I might need.
http://stackoverflow...n-another-table

#33 andyk_79

andyk_79

    Senior Member

  • Members
  • PipPipPipPipPip
  • 610 posts
  • Website:http://www.viczcar.com/forum/topic/12055-andrews-1972-240/
  • Location:Sydney West
  • Tagline:Useless Pest asking clueless questions

Posted 20 May 2015 - 07:51 AM

Sorry Gav just saw this now.  I might be looking a bit simplicly, but an insert/select would do the trick ?

insert into ips_gallery_albums (album_name, album_owner_id)
select title, owner from cpg132_albums;


Edited

Note your comment about the user tables - do they provide a data dictionary for the DB ?  I'll google if I can find one online

Edit again:

Just realised this will insert when you want to update if the gallery has migrated but the owner is blank.  Update script:

update ips_gallery_albums set album_owner_id = (select owner from cpg132_albums where cpg132_albums.title = ips_gallery_albums.album_name) where album_owner_id is null;

Hi mate,
I have a question for you re: SQL.

I've just converted the Gallery over and it seems to be working fine, except for the fact that each album that was previously created had an owner assigned to it. Now however the owners are blank (as in not defined).

The old Coppermine Gallery table (cpg132_albums) has a couple of columns:

1. title = Album name
2. owner = member id

The new Invision Power Gallery table (ips_gallery_albums) has the following corresponding columns:

1. album_name = Album name
2. album_owner_id = member id

Where the left hand value before the = sign is the database column name.

What I need to do is some kind of lookup for the old table, where I extract the title and owner columns and then insert them into the album_name and album_owner_id columns.

Although interestingly I tried do manually update the album_owner_id value via phpmyadmin UI and it didn't seem to work... I'll look into that.



#34 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 20 May 2015 - 09:31 PM

Thanks mate, I actually discovered that the User IDs have changed between the old software to the new software.

I ran this command.

update ips_gallery_comments set comment_author_id = (select member_id from ips_members where ips_gallery_comments.comment_author_name = ips_members.name) where ips_gallery_comments.comment_author_id = '0';


Which seems to have updated the comments table to include the correct author IDs. Just got to work out how to do it for the owner IDs.

Worst case scenario I'll do it in a manual fashion...there is only about 300 unique rows.

#35 George

George

    The 1000+ club

  • Members
  • PipPipPipPipPipPip
  • 1,846 posts
  • Location:Sydney

Posted 17 June 2015 - 02:46 PM

How is this going Gav? Can anyone help you possibly if you're stuck or lack the time?

#36 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 17 June 2015 - 02:53 PM

I've got a friend at work who was going to help with some SQL queries to make the migration over. Just been a combination of busy, a bit sick the last week and when I have spare time working on the car for fear of it stagnating.

#37 brent012

brent012

    Advanced Member

  • Members
  • PipPipPip
  • 97 posts
  • Location:Sydney
  • Tagline:New Member

Posted 17 June 2015 - 06:48 PM

I'm happy to help with writing some SQL queries if you you need any help.

I've got a fair bit of experience with SQL through work (software developer).

#38 gav240z

gav240z

    Administrator

  • Administrators
  • 12,386 posts
  • Website:http://www.viczcar.com
  • Location:Sydney NSW
  • Tagline:Jack of all trades, master of none.

Posted 13 July 2015 - 10:47 AM

Minor update on the weekend I managed to get the gallery album's associated with the correct members in the new software. A few hiccups where multiple users had given their albums the same name, about 40 or so albums with the album name "new album" for example.

Those will have to be manually adjusted and assigned to the correct users later...

Next steps.

1. Associate individual images with their respective owners (I'm hoping the album table and images table share a common key to make this easy).

2. Associate the comments on images with the correct user (these were lost during the migration process).

3. Once that's done, get the latest database dump from the live site, convert the latest instance over to the new software, upload the new software to a development sub-domain and get Auszcar users to test it out and see what they think...

4. Iron out any kinks and then I'll replace the live site with the new software.

That will take care of 2 pieces of software becoming 1 piece. Then for the front page articles I've have to transition those over so go from 3 pieces of software down to 1. This will make updating and maintaining the site much easier.

I've noticed both Classiczcars.com and Hybridz.org use the same software as we are moving to, so those who use those forums should be able to adapt pretty quickly. I do find that Classizcars.com is a bit harder to use these days. I hope to try and keep things as simple as possible.

Eventually once everything is tied to 1 piece of software moving domains permanently to Auszcar.com should be easy enough..I hope.

#39 PB260Z

PB260Z

    Peter (PB)

  • Moderators
  • PipPipPipPipPipPipPip
  • 3,237 posts
  • Location:Northern Sydney
  • Tagline:Must stop buying new toys for the workshop.

Posted 13 July 2015 - 11:09 AM

Hi Gav

You have been busy.
I for one did not realise just how much work goes on the behind the scenes to keep this forum running.

Thanks for all the effort.

Cheers

PB

#40 andyk_79

andyk_79

    Senior Member

  • Members
  • PipPipPipPipPip
  • 610 posts
  • Website:http://www.viczcar.com/forum/topic/12055-andrews-1972-240/
  • Location:Sydney West
  • Tagline:Useless Pest asking clueless questions

Posted 13 July 2015 - 02:10 PM

Maybe those of us with some tech know how can help through a "working bee" type of get together - tie it in with a drive day/bbq or something ?

ditto to the appreciation for all of your efforts Gav!




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users