failure of the system, looking for answers
Need help for the guys. Had a major failure of the system last week, and I'm a bit confused as to what happened and why I was unable to get free. I lost a bit of my confidence in Vsphere, and I'd like to renew that.
A bit on the environment, esxi 5.0 U1, with servers HP Gen 8 3 node cluster running. HP P4500 SAN iscsi for vmfs data storage. ESXi is installed on HP's CF cards in servers. This cluster has been upward and running for about 6 months, not a single hiccup before that.
Wednesday we have migrated 2 of our vm (SQL and progress) to this cluster 5.0 database, the virtual machine went on different hosts in the cluster. The virtual machine was running on a cluster 2 older nodes with esxi 4.1 U2. Machines migrated fine, fine tools updates, reboot, everything seemed fine.
About 2 hours after that the migration, we started to receive calls that our Progress database is down, users could not connect. Then we started getting more calls, other machines were inaccessible. Glancing from vcenter I could see all the virtual machines in question were on the same host, and I was unable to open the console in vsphere for VMs on that host. The host showed he was connected, showed all the vm connected, but I could not open the console or the desktop remotely to one of the VMs on that host. I started studying, and of course this host fell all the SAN Iscsi network connections. The railways have shown dead. Ports of nic card viewed active, the switch ports showed activity, but the connections were down. I could always ping the address of management for the host however, and ports for vmotion were in place.
At this point, I started to try and vmotion VMs on that host data base progress, he would not migrate, just sat at 8% prepare to migrate. I tried other virtual machines with the same result. I started to wonder why HA had not kicked, and why I couldn't move anything. At this stage the host started disconnecting from the cluster. I could always ping on the host, but vsphere showed as disconnected. I couldn't move my VM, and I couldn't go to the host via vcenter, via the vsphere client pointed directly to the host, or by using the DCUI.
So I called VMware support, got an engineer on the line with me, and it became clear we were going to have to power cycle that host and crash all virtual machine running on it. It wasn't a very pleasant for me answer because this database of progress is our main production system, and I was afraid of corruption. We had no choice, so we did. When the host came back upward, fortunately the virtual machine came very well. VMware engineer digging in the newspapers and said that this was with a particular NIC card driver with known issues. He showed me the KB on this issue, and it seemed to be a known issue. We have updated these drivers on all hosts, and that's all.
My problem with this is that, how is the cluster ran fine for 6 months without problem, and how come the redundant path to the SAN did not keep the connection active when with the path with the bad nic card driver failed? I have 2 different, with 2 NICs different paths for the Iscsi SAN. The other card had no known driver issue. Why both paths failed not just in cause of a driver on one of the cards problem? More worrying is also how is it that I could not immigrate anything, and why no HA kick in?
Sorry for the novel here, but without all the details is not part of a story. My biggest concern, that's why I couldn't move anything? In the event of a host failure, what you're supposed to do in order to migrate the machines if they don't migrate via vsphere client? We were down for about 2.5 hours, and a lot of questions were thrown on me the senior management as to why my "system available, redundant high" took hours to retrieve...
Guys here any ideas, thoughts on how I would have handled it differently, reasons why I should be confident everything is fine now?
Thanks for your time
Kevin
trink408 wrote:
Thanks Matt.
I guess some of the problem is not completely understand how the HA or Vmotion. I was under the impression once that the virtual machine was not reachable HA wouldn't kick and move the virtual machine? The virtual machine was not to ping requests or accessible by the Office remotely. I couldn't get their power off or do anything through vcenter either.
If they had no link usable storage, this seems possible. HA simply does not address the failures of storage. Not at all. It IS possible to activate at the level of the VM HA, but it isn't on by default - must be activated on a per-VM basis. You should read the book by Duncan Epping on this - it's the bible of the AH.
So in case something happens with the connection of storage, you have really no way to vmotion anything right here, and your only option is to kill the host / VM running on it, and then migrate or leave HA move them?
Fix.
I didn't know that I wouldn't be able to vmotion virtual machine if the host has lost its connection to storage. The other hosts in the cluster saw storage.
The source host is controlling these VMDK, as far as the other guests are concerned (they see a lock on the file) and when they ask the host if its still alive, he answers (because it has not powered the network lost or down, which are the failure mode THAT HA is designed to handle). So they take charge.
The reason why it took so long was because I didn't kill the host with my concerns for the database. I was hoping that the VMware engineer may have a way to gracefully close things down. So he spent some time looking around and trying to determine what was going on. Ultimately we just power cycle the host, so I could have done much earlier, and if we are facing this again in the future, I would.
As long as you follow the seller advised for more decent databases (keeping the transaction logs, etc.), don't have good backups, theres no real risk of data corruption. VMware does not significantly change the i/o path, in order to have the same exposure than on a physical host.
I still do not understand why the two paths were marked as dead and the host completely lost connection to the SAN, as well as to show finally disconnected in vcenter.
Well the host went offline because it got stuck in an all-paths-down scenario, which is common for 5.0. 5.1 solves this problem a little. I don't know why all roads fell, but I suspect that you have a misconfig somewhere... normally you expect at least 4 paths in a system set up correctly on the left. Check with HP to ensure that you follow best practices.
Maybe all the IP stack has been corrupted or something?
Possible, of course, but unlikely. Never heard before.
I appreciate the help and the preview, I thought that the problem was a lack of understanding on my part and fully accept it.
HA and vMotion are complex. At least now you can go back to mgmt and tell them why he fell more (because it was a failure outside the scope of the software scenario) and may ask for money to build a proper SQL cluster. Definitely recommend the book of Duncan Epping: http://www.amazon.com/VMware-vSphere-Clustering-Technical-Deepdive/dp/1463658133/ref=la_B002YJMRCY_1_2?ie=UTF8&qid=1363622697&sr=1-2
Tags: VMware
Similar Questions
-
Photoshop Elements and the unexpected failure of the system
Hello!
I would ask you to reactivate Photoshop Elements 8, after the unexpected failure of the system. For example: my Windows Vista has a Blue Screen of Death, and I have to format partition and reinstall my OS (my material remains unchanged).
Then install the Photoshop Elements 8 (I bought in the local store), and I'll be able to reactivate my copy of PSE8?
Thanks for the reply
Best regards
Jacek.
PS.
Sorry, my English is terrible
If you have activated only once, then you should be able to reactivate a second time without any problem.
If you have enabled the two copies, then you will need to contact customer service to obtain your activation reset.
-
CC have reinstalled due to the failure of the system. When opened in the image space RAW goes yellow spot, with blue background. I reinstalled without effect. At the opening, in RAW, the image is restored for future treatment. Error occurs with both dng and arw. Ideas.
Try the following:
In the Camera Raw window, press Ctrl + K (Cmd + K on a Mac) to bring up the preferences.
Uncheck the box processor graphics use.
If that suits him, update your graphics driver may allow you to work with the checked graphics processor.
-
I entered my CC number and my monthly payment. When I try to download Photoshop it says 'you are running an operating system that Photoshop no longer supports. See the system requirements for a complete list of the supported platforms. "After watching computer software system preferences is up to date and my google chrome is updated. How can I get photoshop?
Please check the requirements for Photoshop and see if your system meets the requirements:
If your system meets the requirements, please see the FAQ for installation instructions:
FAQ: How to download and install the Adobe Creative Cloud Desktop App?
FAQ: How to download and install creative cloud applications?
-
failure of the system after the installation of windows 7 upgrade advisor
original title: failure of the system after the installation of the Advisor to upgrade to windows 7, which now, I can't delete
My Winows Vista worked fine until I downloaded the windows 7 upgrade advisor - the system crashed, only load windows in safe mode, and when I try to delete the Upgrade Advisor program, it tells me windows service install is not available...
Follow these steps to try to solve your problems of boot.
Restore point:
http://www.howtogeek.com/HOWTO/Windows-Vista/using-Windows-Vista-system-restore/
Do Safe Mode system restore, if it is impossible to do in Normal Mode.
Try typing F8 at startup and in the list of Boot selections, select Mode safe using ARROW top to go there > and then press ENTER.
Try a restore of the system once, to choose a Restore Point prior to your problem...
Click Start > programs > Accessories > system tools > system restore > choose another time > next > etc.
http://www.windowsvistauserguide.com/system_restore.htm
Read the above for a very good graph shows how backward more than 5 days in the System Restore Points by checking the correct box.
.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If restore work not and you do not have a Vista DVD from Microsoft, do a repair disc to do a Startup Repair:
Download the ISO on the link provided and make a record of repair time it starts.
Go to your Bios/Setup, or the Boot Menu at startup and change the Boot order to make the DVD/CD drive 1st in the boot order, then reboot with the disk in the drive.
At the startup/power on you should see at the bottom of the screen either F2 or DELETE, go to Setup/Bios or F12 for the Boot Menu.
When you have changed that, insert the Bootable disk you did in the drive and reboot.
http://www.bleepingcomputer.com/tutorials/tutorial148.html
Link above shows what the process looks like and a manual, it load the repair options.
NeoSmart containing the content of the Windows Vista DVD 'Recovery Centre', as we refer to him. It cannot be used to install or reinstall Windows Vista, and is just a Windows PE interface to recovering your PC. Technically, we could re-create this installation with downloadable media media freely from Microsoft (namely the Microsoft WAIK, several gigabyte download); but it is pretty darn decent of Microsoft to present Windows users who might not be able to create such a thing on their own.
Read all the info on the website on how to create and use:
http://NeoSmart.net/blog/2008/Windows-Vista-recovery-disc-download/
ISO Burner:http://www.snapfiles.com/get/active-isoburner.html
It's a very good Vista startup repair disk.
You can do a system restart tool, system, etc it restore.
It is NOT a disc of resettlement.
And the 32-bit is what normally comes on a computer, unless 64-bit.
See you soon.
Mick Murphy - Microsoft partner
-
Is windows update causes the failure of the system?
I have windows XPsp2 and I've updated with XPsp3 and it takes place in a few errors and the failure of the system to my PC. Unlike my aunts PC she XPsp1 with a slow internet connection so she was not able to update, but it works for more than a year. then the causes updated a few problems in the future?
No, do not keep your computer fully patched all the time causes more problems.
4 steps to protect your computer
http://www.Microsoft.com/security/pypc.aspxNB: If WinXP SP3 is not installed by April 12, 2010, your computer WinXP SP2 will not be offered as all other reviews of security updates and the Windows Update Web site will not be available.er
The above has applied to WinXP SP1 computer your aunt since October 2006.
For you and your aunt...
HOW to get a computer WinXP SP1 or SP2 fully patched
http://groups.Google.com/group/Microsoft.public.WindowsXP.General/MSG/a066ae41add7dd2b~ Robear Dyer (PA bear); MS MVP (that is to say, mail, security, Windows & Update Services) since 2002. DISCLAIMER: I do not represent nor don't work for Microsoft
-
Original title: after a virus, I've lost the system restore for cannot defragment now
I had a virus which caused problems one of the minor being I've lost the system tools. It now says its empty when I open it.
You don't need system defragmentation tools.
You can defragment from the hard drive, then right-click of your hard drive, properties, tools, defragment now.
Of course, you must fix your missing tools too.
Here's the easiest way I know (thank you, Ramesh):
http://WindowsXP.MVPs.org/accessories.htm
Part of the answer to the technical support engineer Microsoft does not too which make me think that the MSE has never actually tested this Board to see if it actually works or not. It's a shame.
In addition, these shortcuts are not part in the folder
in any case. These things must go in the folder for all users - if you don't want your XP to be ignored. It is a source of embarrassment. I don't know what you used to remove the virus, but I could do that at some point (soon):
Download, install, update and do a full scan with these free malware detection programs:Malwarebytes (MMFA): http://malwarebytes.org/SUPERAntiSpyware: (SAS): http://www.superantispyware.com/They can be uninstalled later if you wish. -
Hi all
My question is:
Transaction is written to the log file and it is not written to undo tablespace.
During a failure of the system how oracle rolls back the transaction.
I have already provided the answer, you ignored if well (you seem to only read the responses by people of your country).
Redo log is always written * first * before * writing to the data block (redo log writing is much more aggressive). So it DOESN 'T MATTER if you lose these scriptures of rollback segment.
Valuation: rear roller followed by roll forward, using redo log files and/or archive redo log files.
Sybrand Bakker
Senior Oracle DBA
-
15.0.1 Firefox what are the system requirements for Mac?
I use Mac OS 10.6.8 and I can't install Firefox v 15.0.1. What are the system requirements for this new version of Firefox running on a Mac?
Firefox should work fine on Mac OS X 10.6.8
See:
If he has problems with the update or the permissions then simpler is to download the full version and trash the version currently installed to do a clean install of the new version.
Download a new copy of the Firefox program and save the file on the desktop disk image (dmg)
- Firefox 15.0.x: http://www.mozilla.org/en-US/firefox/all.html
- Basket of the current application of Firefox (for example, open the Applications folder in Finder and drag the Firefox application to the trash) to do a clean install (re) -
- Install the new version you downloaded
- https://support.Mozilla.org/KB/installing+Firefox+on+Mac
Your profile data is stored in the Firefox profile folder, so you will not lose your bookmarks and other personal data if you uninstall and (re) install Firefox.
-
Vista wouldn't boot - the system starts for a few seconds and powers down.
I have a laptop computer HP6915 under Vista Premium. When I turn on the system starts for a few seconds and turns back. I don't know if there is a hardware problem or if Vista is somehow corrupted. I tried to enter the net bios, but the machine does not respond to the F2 command. Ideas?
Hello
Symptoms, especially since it will be impossible to access BIOS, mean that you need a real computer
Shop (not the leeks and the glances at a Best Buy or other stores) or HP support.Support HP & drivers
http://welcome.HP.com/country/us/en/support.htmlContact HP
http://welcome.HP.com/country/us/en/contact_us.htmlHP forums
http://h30434.www3.HP.com/PSG/I hope this helps.
-
When you try to access the tab to pay the buyer work center items get error:
You have encountered an unexpected error. Please contact the system administrator for assistance. (Without any additional information)
Please need help on how to solve this problem.
Thank you
Rambeau
It seems bug-
Please see - Center of the buyer: make an Exception when clicking on PAY tab ELEMENTS (Doc ID 1993378.1)
Kind regards
Kiran
-
I try to install A Indesign
I use a Mac Book Pro with OS X 10.7.5
This happens
You are running an operating system that InDesign can't stand. See the system requirements for a complete list of the supported platforms.
Hello
Please check the version of InDesign and minimum required to make it work:
System requirements for Adobe InDesign CC for Mac OS and Windows
Kind regards
Chirag
-
You are running an operating system that Photoshop no longer supports. See the system requirements for a complete list of the supported platforms.
What should I do?
Hello
What is the operating system you have?
Please refer to the Photoshop system requirements
Creative cloud desktop application download via the link below.
Download Adobe Creative cloud apps | Free trial of Adobe Creative Cloud
* He'd help you download an older version of PS that is compatible for your operating system.
-
m download first pro cc on nut of mac, it is said "or running an operating system that Premiere Pro no longer supports. See the system requirements for a complete list of the supported platforms. "How can I fix
Hello
Please see system requirements below: -.
Premiere Pro system requirements for Mac OS and Windows
I hope this helps you!
-
get this message
You are running an operating system that Photoshop no longer supports. See the system requirements for a complete list of the supported platforms.
Hello
Please see the system requirements for photoshop: -.
Maybe you are looking for
-
When I click on my Firefox I immediately get this message, (image). The problem is that I forgot my password and I can't even click on settings to disable this option, I can't get anywhere.
-
MathType: Insert mathtype equation is disabled. Can you help me?
MathType: Insert mathtype equation is disabled. Can you help me?
-
I would really like to find an answer to this, all keys to work orders, voice work, same intro movie, just doesn't a screen to fly on.
-
computer does not connect to the internet after reinstalling XP SP2 wit
After that the computer we reinstalled windows XP with SP2. The operating system loaded fine, but don't can't get the computer to connect to the internet. Went through all the steps suggested even check the Service Manager to ensure that the approp
-
I have windows xp home edition version 2002 with computer acer bought in 2006 from Bahrain