Recent forum downtime

[mark]

	Quote:	Stephanwolf
	still not quite so stable I guess

There may be some teething problems still for a while. It's basically a new server, so we have to iron out a lot of things which were ironed out on the old configuration. Occasionally we may restart things, processes get locked up, we have to find out why, etc. Just get a cup of coffee and they'll be gone by the time you get back

[Stephanwolf]

	Quote:	mark
	There may be some teething problems still for a while. It's basically a new server, so we have to iron out a lot of things which were ironed out on the old configuration. Occasionally we may restart things, processes get locked up, we have to find out why, etc. Just get a cup of coffee and they'll be gone by the time you get back

I wasn't complaining

[Woodsie]

	Quote:	mark
	Just get a cup of coffee and they'll be gone by the time you get back

CCccaaannn'ttt sssttoooppp hhhaaandddssesss shhhakkkiinnngg. Tooooo muuuuchhh cofffffeeeee!!!!!!

[Shorrick Mk2]

Aaah, the green blobs are back.

Just in time, as my People's Moderator position was being challenged.

I am having repeated "Database Error" messages when posting... maybe something needs to be fixed to avoid another crash soon

[mark]

	Quote:	Salsa_Lover
	I am having repeated "Database Error" messages when posting... maybe something needs to be fixed to avoid another crash soon

Please read the other posts on this page which discuss this issue.

[Jonathan_hughes76]

Mark

I would be interested to know if you host this forum on a server running under Vmware or MS Virtual Server or such like? under Windoze 2003, Solaris, Linux ? Your initial post suggests that perhaps you do. I've a lot of past experience setting up and maintaining virtual server farms. Dynamic expanding virtual hard disks growing to be greater than the physical hard disk space on the underlying server isn't unheard of!

Regards

[gooner]

	Quote:	Jonathan_hughes76
	Mark I would be interested to know if you host this forum on a server running under Vmware or MS Virtual Server or such like? under Windoze 2003, Solaris, Linux ? Your initial post suggests that perhaps you do. I've a lot of past experience setting up and maintaining virtual server farms. Dynamic expanding virtual hard disks growing to be greater than the physical hard disk space on the underlying server isn't unheard of! Regards

It's a LAMP setup IIRC (Linux Apache MySQL PHP)

[Jonathan_hughes76]

Gooner

Thanks for that. However is the LAMP s/w config running within a VM or not?

regards

[gooner]

	Quote:	Jonathan_hughes76
	Gooner Thanks for that. However is the LAMP s/w config running within a VM or not? regards

It's not a virtual server. Mark has his own physical server. I don't know if he is running VMs on it though, i don't think so. I'm sure he'll be about at some point to supply technical details.

[Lob]

win2k3 with vmware running CentOS in the VM IIRC. But things are now in hand

[mark]

Didn't really want to get technical on this thread, but here goes.

Own physical server (1U, 2.8Ghz Pentium D, 4GB physical ram, 2 x 36GB WD Raptor S-ATA (10,000RPM) running as RAID-1 + 500GB backup disc). Server located in Telehouse 2 Zurich (multiple peerings, etc).

Host runs Win2k3 as base operating system, VMWare Server 1.0.4 runs on top of that. Current server (as you see it) is running a Centos4/Blue Quartz distro in a VM. This VM has 3GB physical ram assisgned, with two virtual disc files running on the mirrored raptors.

Johnathan - I agree with your comments, I guess you've had it happen to you as well, so suffice to say it won't happen again, moves are afoot as we speak. The main changes are that while right now I have about 60 domains running from this server, some are large collections of photos spanning several GB, most are low traffic sites, and EF takes about 95% of the server load.

Very soon this will be split into 3 VMs, each built specifically for the application. For example, EF and one other forum will be split of into their own VM , this machine will be optimised just for forum use. All the large photo sites will get their own VM with no email, no backup, no php, no mysql, etc.

By splitting in this way I hope to increase the reliability, reduce the risk and avoid any single point of failure (ahem, cough, cough).

But probably the biggest change will be that these new VMs will have pre-allocated virtual disc files, so the problem of growing virtual disc files will be a thing of the past. Once the various functions are split off into multiple VMs, disk usage becomes a lot more predictable, so pre-allocation isn't a drama anymore.

Right now these 3 VMs will run together on one physical machine. I can't afford to go to multiple physical servers, since each unit of rack space isn't cheap. But if this were to happen in the future, having multiple VMs will make it easier to move.

Basically, this work is ongoing and began a few days after the crash. We hope to have it completed in a couple of weeks.

[mark]

Minor hiccup, sorry folks. Had a problem while trying to restore some attachments I had in an old backup. We lost 14:40-15:03 today. Any posts done then are gone as we rolled back to 14:40. Sorry about that.

[grumpygrapefruit]

Hi Mark

I'm hitting a little prob, when I click to last post for "Where to find foods from home" it takes me here

[SamC]

	Quote:	mark
	Minor hiccup, sorry folks. Had a problem while trying to restore some attachments I had in an old backup. We lost 14:40-15:03 today. Any posts done then are gone as we rolled back to 14:40. Sorry about that.

Honestly Mark - twice in a couple of weeks. Sort it out FFS!



Seriously, thanks for nursing the forum back to health - much appreciated.

[DaveA]

You ungrateful bastard. Click that donation box immediately, or a salvo of groans is heading your way.

dave

	Quote:	SamC
	Honestly Mark - twice in a couple of weeks. Sort it out FFS! Seriously, thanks for nursing the forum back to health - much appreciated.

[mark]

	Quote:	grumpygrapefruit
	Hi Mark I'm hitting a little prob, when I click to last post for "Where to find foods from home" it takes me here

That was a little aftershock of the hiccup. It should have been resolved pretty soon after that though. If you see the same thing again, just PM me.

[mark]

Quick update - the photo system is back online. We have absolutely all images prior to June 02, 2007. We have *some* images betwee June and November 2007, but I fear most were lost. All comments made on images (providing the images themselves are still there) survived - in other words, no comments were lost.

Apologies to those who may have uploaded an image in the last six months that may have been lost.

Reminder - click on "Photos" on the navigation bar at the top of the page to access the photo system.

[mark]

Scheduled forum downtime

As part of the work to improve the reliability of the forum we will be taking the forum down for about two hours between 23:00 and 01:00 on this coming Tuesday evening.


We’ve chosen this time because it’s a time which isn’t usually as busy as others.


We plan to migrate the entire data on the server to some brand new virtual discs (which look just like the old ones but are better, trust me). Hopefully, the forum will be back online with no problems at all.


If things do go wrong, we will be taking backups prior to doing any work.


The forum will close five minutes before 23:00 and after that you’ll be redirected to a message saying that it is offline.


We apologise in advance for any inconvenience caused and suggest that you put the “early night” to good use, especially if you’ve been staying up late at night on the forum and “neglecting” certain activities.

[DaveA]

Cant you do it on Thursday night instead ? It might be a bit quieter then...

dave

	Quote:	mark
	Scheduled forum downtime As part of the work to avoid landing ourselves in trouble again constantly improve the reliability of the forum we will be taking the forum down for as long as we need to about two hours between 23:00 and 01:00 on this coming Tuesday evening. We’ve chosen this time because we don’t want to stay up all night because it’s a time which isn’t usually as busy as others. We plan to migrate the entire data on the server to some brand new virtual discs (which look just the old ones but are better, trust me). Hopefully, if we don’t manage to mangle something during the process, the forum will be back online with no problems at all. If things do go wrong, we should be able to rustle up a backup from some old DVDs somewhere in my drawer we will be taking backups prior to doing any work. The forum will close five minutes before 23:00 and after that you’ll be redirected to a message saying that it is offline. We apologise in advance for any inconvenience caused and suggest that you put the “early night” to good use, especially if you’ve been staying up late at night on the forum and “neglecting” certain activities.