Periods of high disk latency

Hello - I just moved to a new company and I am working on virtualization of most of the servers here. It's a small IT shop with 20-30 servers. I bought 3 new HP DL380 G7 servers and I have about half of the 20 converted physical servers. I noticed a problem that appears and it seems random times.

There are 2 tables (13 6 disks raid, raid 5 disks 5) 2 luns, a VMFS shared LUN between the 3 servers. This is on a HP P2000 G3 FC SAN (already in place before). Every once and a while I notice latency of very high disk it seems that only on the LUNS on the Raid 5 array.

I've attached a screenshot of ESXTOP running on all servers ESX 3 showing the high latency. I can understand if there was a very high use but is not really yet everything going on at this time, as you can see on the screenshots. The tweak only I made so far is to increase the depth of queue to 64.

Brocade 8 GB HBA using hanging on the HP (Brocade) switches. Everything works to 8 GB and all readers of the San are 6 GB SAS.

I'm new on this and HP options without, came from one world IBM DS4xxx, so I do not know that how still to watch the performance on the SAN itself, the web interface does not have all followed what-so-ever.

Any help is appreciated-

Thank you

Well, I agree that it was a question of ownerhip lun.

The extent of maxing controllers, theres more to the issue than just flow.  Its actually very common on these lowend tables except if you make something trivial like a streaming the CPU controller will not be able to do in the face of a workload of IOPS / s high (or even moderate).  This isn't the bandwidth, it is the CPU on the controller.  His community does even more than you might expect on the bays of large companies (HDS USP, EMC Symm, etc.).

-Matt

VCP, VCDX #52, Unix Geek, Nerd of storage

Tags: VMware

Similar Questions

  • ESXi 5.1 SMV degrade slowly with until this only unusable disk latency issues. How do I troubleshoot and resolve?

    Dell R710 servers

    Reference Dell Perc 6 / i raid controllers

    3 500 GB hard disks in a raid 5 configuration

    2 data stores.

    3 Red Hat linux 5.5 VMS servers.  Database intensive applications with e/s running high.

    Symptom: Servers become very slow or does not respond over time. Each VM restarts does not help. A restart of the entire server ESXi provides immediate relief and the process begins again. I can only run the servers about a week before I have to restart again.

    Disk latency may average in the range of 100 to 300 before a restart. After the reboot, the latency average in the range of 5 to 9 and slowly grows as the days pass.

    Use memory and CPU look normal to me. Uniformly flat even when the servers seem to have trouble.

    Any ideas what would cause this or how to fix?

    Please go to esxtop on the server and press D to see the latency of disk.

    FollowESXTOP - yellow bricks

    to troubleshoot latency.

  • Satellite A500-1DN - high DPC latency resolved

    Hi all!

    All first of all sorry for the bad English!
    I am writing because this problem drove me crazy in the half of the year. I got cracking audio etc. thanks for the high dpc latency time but finally I managed to understand how to clear my A500 these boring of CPD points. I made these:

    1. update of the BIOS (1.90)
    2 use this VGA nVidia Driver: 257.21. Only this one works. I tried every existing. If you have another type of satellite, but a Geforce 230 M GT in there and you have high latency tell him!
    3. use wlan driver atheros version 8002580
    4. use optimzer wlan. Name: wopt021 (disable background scanning and enable streaming mode)
    If you use WiFi, jumping 3. 4.

    Now my A500 works correctly, I am really satisfied with it :)
    I hope that this guide will help all those who have the same problem!

    Good bye

    Hi atibrd,

    I'm happy to hear that you have found a solution to this "problem" but I must say also I never had problems with my Satellite A500 and I put t know what you mean with a DPC latency. I use my Satellite A500 with OS preinstalled Toshiba and the latest version of the BIOS. Everything works fine, even some newer games run great on it.

    In addition, with regard to the nVidia display driver I want to give you a little advice: be careful with 3 third party drivers, even from nVidia. No one from nVidia has tested this driver in particular on the Satellite A500. If the installation goes on your own risk because it doesn t have a protection against overheating.
    That's why I only use the drivers provided on the official website of Toshiba directly. They are tested and working 100% correctly.

    In any case, maybe for some people they have this DPC problem too it s useful. :)

  • High disk usage. What is c?

    I get a text box in the lower RIGHT of my computer screen saying: "high disk usage: host process for windows services.  The computer is slow too, but I don't know what that means. Can someone tell me please?

    Hello elwindouglas,

    If you start your computer in Mode safe, can you reproduce this problem?  What version of Windows you are currently using and you have made any changes to your computer before this problem?  This message appears no matter what applications you are running?

    Below, I've added a few links that I hope will be able to help get the resolution.

    System File Checker: (for Vista and Windows 7)

    http://support.Microsoft.com/kb/929833

    Start your computer in Mode safe:

    http://Windows.Microsoft.com/en-us/Windows-Vista/start-your-computer-in-safe-mode

    Ways to improve your computer's performance:

    http://Windows.Microsoft.com/en-us/Windows-Vista/start-your-computer-in-safe-mode

    Here's another question, with the use of disk high on another forum: (for Vista)

    http://answers.Microsoft.com/en-us/Windows/Forum/windows_vista-performance/high-disc-usage-by-svchostexec/d7ee0136-bbec-4fc7-955d-c281eb786773

    Thank you

    James

  • my internet is used with any one using the computer as I think it's because the svchost.exe is corrupt, as a window comes up saying high disk useage has one an idea how to solve this

    the load down on the internet limit is used in about 2 weeks, that it is with a download of large files. also, a window pops up saying that he caused by svchost.exe high disk usage.

    I had a few a look at the problem on the computer and they seemed remedy and the down load was normal for a while, but has since returned to duty.
      

    Hello BrianWallacexx,

    This thread has been created in the forum Desktop and personalization; the Microsoft moderation team has moved this thread in the Forum user accounts, security, and confidentiality.

  • Demand: Disk latency average VM for VMS powered

    We are interested in finding the average of VM disk latency, as evidenced by the powered on virtual machines.

    We did a group (Group-PoweredOnVMs) in vCenter Ops who has all virtual machines under tension.

    We see this group in the section of the environment on the Overview of the environment.

    We have created a great metric (SM-AvgVMDiskReadLatency) to avg the Lantency of disc (avg (VM: disk |)) Latency (ms) reading the disk())

    This great metric associated with a package (SMPkg-VMDiskMetrics)

    In the Types of resource of the Overview of the environment section, we selected the Group (Group-PoweredOnVMs) and related metric great packaging

    However, when we put that great metric on a metric graph, NO DATA.

    Any ideas on how to do this? It seems that whenever we try to use a dynamic group, we do not get data.

    Thank you!

    You need a reference with ^ N.

    You have a group - wide. However, you want to watch a top-down relationship exactly step 1 below the resource group.

    To do this: avgN (Virtual Machine: disk |) Latency (ms), 1 disc)

  • How to export data from disk latency?

    I want to the latency of disk order data in CSV or XLS format but could not find a way to export.  Its not included in the existing reports.  I can view the data in a nice graph, but I can't go to the numbers.

    Anyone know if this is possible?

    Thank you

    VCops-data-censored.png

    Go to the section "all indicators; Locate this same metric and view it (choose the period that interests us).

    There, you have the ability to save data in CSV or PNG.
    I've attached a snapshot.

  • Using PowerCLI for virtual disk latencies

    Hi all

    I'm trying to get the disk reading and latencies on some virtual machines

    I'm quite a beginner on PowerCLi apart from some ships a writing utility

    I checked this thread

    http://communities.VMware.com/thread/304827

    But these counters do not work for me

    Can someone please help?

    [vSphere PowerCLI] D:\Program Files\VMware\Infrastructure\vSphere PowerCLI > $stats += Get-Stat - entity "USB-SSAPP" - Stat virtualDisk.totalWriteLatency.average
    Start $today1am. AddDays ($intStartDay) - finish $today4am. AddDays ($intEndDay)
    [vSphere PowerCLI] D:\Program Files\VMware\Infrastructure\vSphere PowerCLI > $stats

    Returns nothing,

    I created all the variables that I'm providing in the command.

    Kind regards

    P

    You might start by checking what statistical level, the period you specified is defined in vCenter.

    "Latency" needs to at least level 2 metrics.

    I guess that "USB-SSAPP" is the name of a host ESX (i)?

  • How can I stop high disk activity HARD and blocking of the machine

    At startup, the machine takes 2 to 3 minutes before the login screen is available, with HDD coming out of the machine all the time. The Office takes 1.5 minutes to load up. Then quite randomly and especially if I have access to Internet, the HARD drive starts screaming from afar and I get any program I use, indicate "not responding". Similarly, logging on Internet and then check in a site and outside, Yahoo shows "not responding" then comes good, however at that time that the HARD drive is once more trying to get out of the machine with the blocking of the entire system. This condition may last between 1 and 2 hours.

    I can't get out of the Internet or Vista and my only recourse is to hold down the button "power on" for 5 seconds, stop the machine. The alternative is to turn the machine on and leave it alone for a few hours, but it is still not guaranteed the high activity of the HARD drive will be start at random. At the time of blocking, the CPU is variable between 1 and 10% with 4 GB of ram running at 96% to 100%.

    I am running Windows Vista Home Premium (SP2), on an AMD Athlon (TM) 64 x 2 processor Dual core 5000 + 2.60 GHz, Medion machine with 4.0 GB Ram and 32-bit operating system. The machine is a PC MT7 MED MT469G. The boot drive (C: /) is about 350 GB, with a second backup drive (D: /) about 100 GB and a Data partition (E: /) drive on 400 Gb. I.e. two 350 and 500 GB drives.

    The machine becomes practically unusable and I can't stop this frenzied action of HARD drive. I have to be to destroy the HARD drive whenever I turn on. I tried the advice from the experts by disabling regular departures, stopping at various folders, to nothing does not.

    I came through all products Windows 95Windows with XP and 2000 and when I bought this computer Medion, with Vista, I have never experience so much time lost to fight with a machine with this huge HARD drive activity and constant locking machine.

    I understand, sites that Windows 7 is pretty much the same, God forbid, I will put on my wife's machine and is the reason why I have not upgraded.

    Help! Please, how can I get out of this mess?

    Concerning

    John Mills

    I'm not sure I understand your disk configuration. You have two physical drives with drive non-system, divided into two partitions?

    All other programs are running when this happens outside of the browser?

    I would start by Process Explorer to identify the process generating the IO.

    http://TechNet.Microsoft.com/en-us/Sysinternals/bb896653

    On the file menu, click 'Show details for all processes', and then on the view menu, select the columns to activity e/s of the shoe.

    While you are in the view menu, click on "System information" and check how your 4 GB 32-bit Vista can see. It is normally about 3 GB but can be much less with some video cards.

    There is more information on the use of Process Explorer here:

    Windows Sysinternals Primer: Process Explorer, Process Monitor and more. Tech· Ed 2010 North America | Channel 9

    You can also use the built in by typing perfmon performance monitor in the box "Search" in the Start Menu.

    http://Channel9.msdn.com/events/TechEd/Australia/Tech-Ed-Australia-2011/SVR306

    This should help you identify files that are used.

    For a more detailed study, you must use Windows Performance Toolkit:

    http://msdn.Microsoft.com/en-us/performance/cc825801

    http://Channel9.msdn.com/events/build/BUILD2011/HW-59T

    Download here: http://www.microsoft.com/downloads/dlx/en-us/listdetailsview.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b and select the components you need.

    There is a possibility of a hardware problem or the disc is not not in AHCI mode.

    I would also look at the antivirus you are using.

  • Oh, the irony? High DPC latency with idle CPU - gives more stable, more low-latency DPC CPU usage 100%

    Just like in the title,

    I tried everything, disabling components, elimination of most of my RAM, update BIOS and loading failsafe and optimized defaults, etc. - and one thing I noticed is that Prime95 running x 64 on all hearts eliminates the peaks high in 1.2.0 DPC latency checker. Audio stuttering, what original research for corrections me, brought me to DPC latency checker.

    It's really strange that the 100% CPU usage would cause the DPC latency stabilize (mine hovers around 350-450 microseconds in the full use of the CPU, but in use slowed parameters or moderate apparently something causes a DPC peak at ~ 4100us max).

    Let me know if you figured it out!

    Model name : GA-EP45-UD3P(rev. 1.6)
    --------------------------
    M/B Rev : 1.6
    BIOS Ver : FE
    Purchase garage : Newegg.com
    -------------------------------------------------------------------------------
    Brand VGA : ATI model : 5770
    Brand CPU : Intel model : Quad 9400 speed : 2.66
    Operating system : Win 7 64 bit SP :
    Brand memory : G.SKILL Type : DDRII
    Memory size : 4 GB, speed : 800
    Power supply : Antec Neopower 550 W

    Hi Frankie,

    We understand that you too met the same scenario as this 6 year old thread. Some resolutions are always topical, while others are no longer applicable due to constant updates and corrections in the course of the last years.

    However, to resolve your concern, you can view article utilization of the processor for Windows 7 .

    Let us know how it goes.

  • Disk latency issues

    I'm under ESXi on a Dell PowerEdge r310. I currently have 2 x 250 GB HDDs in a RAID1 for configuring disks hypervisor and 2 x 2 TB in RAID1 for VM data store. I thought keeping them separated would be nice in case there are problems with the picture of the hypervisor. It worked for about 2 days now and last night I advanced and moved our VM during the machine production. Today, around the time look, I get this message:

    Device naa.600508e000000000a54c6f66b1cbaf06
    performance has deteriorated. I/O latency
    passed with an average value of 12049
    microseconds to 242737 microseconds.
    warning
    My WHAT VM production placed at roughly the same time. I turn it off and started upward and has worked very well since.
    2 x disks in RAID1 2 TB is far from ideal. My question is this. I lose all my time keep the hypervisor separate virtual machines? I would be wise to install the hypervisor on a single disc (zero redundancy) and then use the 3 disks in a RAID5 configuration for the vmstore? Should I just use all 4 disks in a single table in RAID10?
    Any advice would be appreciated.

    When ESXi starts, it creates a virtual drive for its basic file system and therefore ends with minimum requirements on the installation disk I/O.   I suggest you these options:

    (1) 4 disks in RAID 10 / 5 array and have the ESXi installation and the data store on this table.

    (2) 4 discs in a table to store data only and ESXi boot from a flash deivce.

  • High write latency

    Hello

    I have two ESXi 4.1 U1 and Huawei S2600 storage and the lantency of command write and physical device is high (see attached files)

    The storage is connected directly to servers through iSCSI with the frames and Round Robin. I use a Raid 5 with four SAS 15 k disks and the virtual machine is in a logic unit number in this Raid

    I have 19 VM distributed between two hosts and 4 of them is using 2 vCPU

    Any ideas where is the problem?

    Thank you

    I doubt that its really active/active.

    Try MRU and see where you get.  Change the political trajectory didn't effect frames jumbo or anything like that.

  • 6735 s his stuttering, high DPC latency

    6735 s aged 8 months.

    A month ago I started to notice the audio stuttering in video to download websites such as youtube. The problem tends to appear after about an hour of computer use / navigation. Recently, however it has started to affect the mpg and mp3 files played locally on my hard drive. The problem is much worse that a download is in progress (i.e. any download, the media files not only). I ran a DPC latency checker that I found on the web and I found that when I'm browsing or downloading files DPC latency goes through the roof.

    About a week before I first noticed this problem I have updated graphics drivers from ATI (Catalyst v9.8, then recently v9.10), so I think it has something to do with it.

    I tried to disable the WIFI / network cards, bluetooth, USB, CDROM drive, ATI drivers uninstall audio drivers, firewall, antivirus, killing all the processes that are unnecessary in the Manager of tasks etc. Only a reboot will solve the problem (for an hour or two before returning)

    Does anyone else have this problem? It drives me nuts having to restart my computer for all hours.

    Update: turns out the culprit being the ZoneAlarm firewall. I had heard of people having problems with the COMODO firewall, so I tried to uninstall ZoneAlarm and all is well again. Note must be a uninstall complete, just disable or turn off ZoneAlarm was not enough.

    See you soon.

  • (Redirected) Computer beeps during use. Maybe high disk or the use of the CPU usage?

    New here and new to Dell computers. 1st stop, I have a Dell XPS 13 "i7 6560u 16 GB and SSD 512. When using the computer, I get a series of 2 beeps. It seems to be related to the high use of the disc or the CPU usage. I guess my question is possible to put these notifications for a longer delay or at least decrease the volume of the beeps. I am aware that when I use one of my CSD programs to open a browser and looking for an answer to the question about what there will be times where I could get high uses a of these, that's all just annoying and not really want to turn them off entirely (option I would) I would delay having more time this will happen to until I get through this ,

    I thank you,

    Ben

    Hi w4qed,

    Please repost this in the laptop computer forum help.

    http://en.community.Dell.com/support-forums/laptop/f/3518

  • Csrss.exe high disk activity

    Csrss.exe is constantly reading e/s. You can hear the churning of disk as well as to see the activity with the Task Manager. I use Windows 7 64. I don't know what he thinks, he must do it, but I just want to stop it. If she did something like disk indexing, if the price is the constant disk activity then I have indexed disk.

    If anyone knows how I can stop this thing I would appreciate it.

    Hello

    You can check this link:

    http://TechNet.Microsoft.com/en-us/Sysinternals/bb896646

Maybe you are looking for