High cache buffers chains

After a data load data has increased 30%. After the charge, one statement now takes 5 h 40 min before the charge. The statement runs during the night shirt lot, so there is no other connections. Statistics for tables and indexes are collected and current.

I compared the DOWNLOAD before loading the data and current AWR:
-locking: buffer cache is moved from 12Mio expected 95Mio expected. As a result, the CPU usage is very high during the period of support.
-compatible gets multiplied by 3
-select count (distinct (hladdr)) from x$ bh. The result is 8192
-J' I checked for hot blocks - there is none there is no other process during this period

The slow statement is a DEC as (the same statement works in an environment with much more data in less time.)
CREATE TABLE XXX NOLOGGING PARALLEL TABLESPACE YYY COMPRESS PARTITION OF RANGE (...) select with outer joins.

What else review / should I?

TanteKaethe wrote:
I put together the plan values real explain for the two environments (see below).

In the meantime, I've added a hint of the query / * + USE_HASH (bs tk_pla) that causes the CBO in the environment that is slow to use a hash instead of the nested loop join. The execution time down to a minute.
The solution with the suspicion is acceptable, I would like to get rid of this suspicion - or at least to (better) understand why the CBO in the slow environment chooses a loop nested without suspicion (= if there is indeed some statistics wrong or other configuration errors;) I also did some checking on the tables and indexes. SAT_INDI a 41100 lines while A-lines in the slow environment shows a difference).

The statistics of the A-LINES are different, because the lower part of the development from Operation plan ID 22 runs 76 000 times due to the LOOP IMBRIQUEE. It is therefore essentially the lines properly estimated 40 k time 76 000, makes a total of 3 G lines generated.

I have already spoken several times, but you have not addressed this yet: why you get a cost estimate of 0 for multiple access to the index in the plan of 'slow '? What you get in the index at the global level statistics (as these seem to be partitioned, very likely local index): btree height (BLEVEL), the number of lines, pads of sheets, separate keys etc. ?

In addition, which seems very strange in terms of slow: the view from Operation card identity 22 seems to be encrypted with 0, although some of the sub-operations such as access to the table SAT_INDI have a cost > 0.

It would be helpful if you could post the output of the plan PLAN EXPLAIN corresponding series for serial execution see the costs, given my conclusions above are based on the parallel execution plan.

This cost of 0 for the view is probably the main reason why the optimizer favors the external operation of LOOP IMBRIQUEE. There could be a bug that the entire view is encrypted with 0 if sub-operations have a cost > 0.

It could be a side effect of the index being encrypted with 0 access, so I think that you should focus on this point first.

Kind regards
Randolf

Oracle related blog stuff:
http://Oracle-Randolf.blogspot.com/

SQLTools ++ for Oracle (Open source Oracle GUI for Windows):
http://www.sqltools-plusplus.org:7676 /.
http://sourceforge.NET/projects/SQLT-pp/

Tags: Database

Similar Questions

  • latch: cache buffers chains

    Hello dear gurus!


    How to identify a hot block in the database buffer Cache. [163424.1 ID]

    This lock is acquired when looking for blocks of data cached in the buffer cache.
    Because the buffer cache is implemented as a sum of channels of blocks, each of these
    channels is protected by a child of this lock when needs to be analyzed.
    Expression protected by that latch children mean that parent latch child lock and buffer to protect protect block and also views
    DBA_HIST_LATCH_PARENT and V$ LATCH_PARENT belongs to the buffers (DBA_HIST_LATCH_CHILDREN, V$ LATCH_CHILDREN belongs to the blocks)?



    Thank you and best regards,
    Pavel

    Hello

    normally when a latch has children, then it doesn't do much on its own: children do most (or all) of the work, for example:

    http://www.freelists.org/post/Oracle-l/difference-between-child-and-parent-LATCHES, 4
    http://learningoracle.WordPress.com/2008/02/19/LATCHES-and-latch-contention/

    I'm not sure if the lock of the buffer cache is also part of this category, but I think your interpretation is wrong.
    Blocks in the buffer cache are organized in a kind of hash buckets, and a child lock protects one or the other or
    several of these buckets, not a specific block of hash. I can't say what function (or even at all) plays exactly the parent latch,
    but the basic logic States that in all cases, it cannot be very restrictive.

    If you need a more precise answer, try the blog of Andrey Nikolaev (andreynikolaev.wordpress.com), or try experimenting with locks
    you on a test system (for example to simulate an activity in the buffer cache and see how the stats for parent and child latches change in time,
    and try to see if stats for parent latches are just the stats are appropriate for their children).

    Best regards
    Nikolai

  • Background DBWR1 and DBWR0 lock wait event: cache buffers lru chain

    Hi all

    We have a Test database 10.2.0.1 having

    db_writer_processes integer 2
    very large integer SGA_MAX_SIZE 1504 M
    Whole large SGA_TARGET 1504M

    When I check the event of the two background processes is to say DBWR1 and DBWR0 is pending in

    latch: cache buffers lru chain

    "bdrbd_lcoal > select program session $ v where event =' latch: cache buffers lru chain."

    PROGRAM
    ------------------------------------------------
    Oracle@DB (DBW1)
    Oracle@DB (DBW0)

    When I try to manual control his account more than 4 minutes to complete the command checkpoint
    and four session are waiting in case of waiting free buffers.

    Please help solve this problem.

    Thank you
    Jamsher

    Edited by: Jamsher February 3, 2011 04:31

    Hello

    First define the sga_max_size, then sga_target

    change the system sga_max_size set = 2000 m; -sga_max_size is not dynamic, closure of the base data, make the changes, and restart the database

    ALTER system set sga_target = 1800 m;

    You can read the following as well

    http://asktom.Oracle.com/pls/asktom/f?p=100:11:0:P11_QUESTION_ID:30011178429375

    Published by: jazz81 on February 3, 2011 13:48

    Published by: jazz81 on February 3, 2011 13:55

  • Cache Buffer Chain latch

    Hi all

    What is the CBC lock?
    When it occurs, and why?

    What is the inner workings of the latch of the CBC? How can we identify latch of CBC and how to fix it?

    -Yasser

    Yasser,

    Database buffer cache is divided into something that is called activities together. Each game has two or three pads inside. These buffers are associated with something called hash strings. When you ask for the buffer for your table, you must acquire the lock of Cache buffer. CBC latch is used to search for your activities of buffer for the necessary buffer based on its Address (DBA) of data block. That's all there is on the latch of the CBC. Rest of the internal components are actually on the cache buffers not on the locking system itself.

    Read this note from Steve Adams for the same institutions,
    http://www.Ixora.com.au/q+a/cache.htm
    HTH
    Aman...

  • Cache buffers DB, file Cache Redo, DBWR, LGWR and log and data files

    Hi all Experts,

    I m very sory for taking your time to a matter of very basic level. in fact, in my mind, I have a confusion about the DB buffer cache and the cache of log buffers
    My question is in the DB buffer cache are three types of dirty, pinned and free i-e data. and in Sales, they are all changed data data and are willing to empty in DBRW. and then he write data files.

    but again bufers also works for the CDC and all the changed data is temporaryliy chacheed bufers redo and the LGWR writes in the log files. and these data can be data are committed or not. and when the log switch is held, then it writes data to data files commited

    My question is that if a log file may have committed data type and stop and when a log switch takes place then only the data are committed are transferred to datafiles, then where are the data no go?


    If dirty pads also contain modified data so wath is the diffrence between the bufers and data log data incorrect.

    I know that this can be funny. but I m maybe wrong abot the concept. Please correct my concept about this

    Thank you very much

    Kind regards

    user12024849 wrote:
    Hi all Experts,

    I m very sory for taking your time to a matter of very basic level. in fact, in my mind, I have a confusion about the DB buffer cache and the cache of log buffers

    The first thing we do not mention this newspaper stamp in the log buffer cache. It won't make a difference to even call it that, but you should stick with the term normally used.

    My question is in the DB buffer cache are three types of dirty, pinned and free i-e data.

    Correction, it is the States of the buffer. These aren't the types. A buffer can be available in all of these three States. Also, note even when you would select a buffer, his first PIN before it can be given to you. Apart from this, there is a type more State called instant capture buffer aka CR (coherent reading) buffer that is created when a select query arrives for a buffer inconsitent.

    and in Sales, they are all changed data data and are willing to empty in DBRW. and then he write data files.

    Yes, that's correct.

    but again bufers also works for the CDC and all the changed data is temporaryliy chacheed bufers redo and the LGWR writes in the log files. and these data can be data are committed or not.

    That is partially right. The buffer which is written by DBWR in the data file is a full buffer while in the log buffer, it is not the entire block that is copied. Oracle enters the log buffer called vector of change . It's actually the representation of this change that you made in the data block. It is the internal representation of the change that is copied into the log block and is much smaller in size. The size of the buffer of paper compared to the buffer cache is much smaller. What you mentioned about the writing of written data room again or not committed is correct. His transactional change that needs to be protected and therefore is almost always written in the restore log file by progression of the LGWR.

    and when the log switch is held, then it writes data to data files commited

    Fake! To the command log, Checkpoint is triggered and if I'm wrong, its called point of control of thread . This causes DBWR write buffers dirty this thread in the data file regardless of whether or not they are committed.

    >

    My question is that if a log file may have committed data type and stop and when a log switch takes place then only the data are committed are transferred to datafiles, then where are the data no go?

    As I explained, it is not only the validated data are written to the file data but committed and not committed is written.

    If dirty pads also contain modified data so wath is the diffrence between the bufers and data log data incorrect.

    Read my response above where I explained the difference.

    I know that this can be funny. but I m maybe wrong abot the concept. Please correct my concept about this

    No sound is not funny, but make sure that you mix in anything else and understand the concept that he told you. Assuming that before understanding can cause serious disasters by getting the concepts clearly.

    Thank you very much

    HTH
    Aman...

  • Simulate the Cache Buffer Chain latch

    Hi all

    Can simulate CBC latch scenario, I want to say I want to create test cases to simulate the latch of the CBC and dig in for different solutions to reduce the latch of the CBC.

    It would be great if someone can give me the test case, link or idea.

    -Yasser

    Published by: YasserRACDBA on December 16, 2009 17:42
  • Need help for analysis "plan and background events waiting" on the report statspack for oracle database 11.2.0.4 on AIX

    HI: I analyze the STATSPACK report: this is the "volume test" on our UAT server for most of entry or "bind variables".  Our shared pool is well used in oracle.  Recovery of Oracle logs is not configured properly on this server, as in "Top 5 events of waiting", there are 2 for Oder.

    I need to know what other information may be digging from of 'waiting in the foreground events' & ' background waiting events ", and which can help us better understand, in combination of ' Top 5 wait event, that how did the server test /?  It could be overwhelming. wait events, so appreciate useful diagnostic or analyses.  Database is oracle 11.2.0.4 updated from 11.2.0.3 on IBM AIX 64-bit, level 6.x system power


    STATSPACK report


    DB Id Instance Inst Num Startup Time Release RAC database


    ~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---

    700000XXX XXX 1 22 April 15 12:12 11.2.0.4.0 no.


    Host name Platform CPU Cores Sockets (G) memory

    ~~~~ ---------------- ---------------------- ----- ----- ------- ------------

    dXXXX_XXX AIX-Based Systems (64-2 1 0 16.0)


    Snapshot Id Snap Snap time Sessions Curs/Sess comment

    ~~~~~~~~    ---------- ------------------ -------- --------- ------------------

    BEGIN Snap: 5635 22 April 15 13:00:02 114 4.6

    End Snap: 5636 22 April 15 14:00:01 128 8.8

    Elapsed time: 59.98 (mins) Av law Sess: 0.6

    DB time: 35,98 (mins) DB CPU: 19,43 (mins)


    Cache sizes Begin End

    ~~~~~~~~~~~       ---------- ----------

    Cache buffer: block 2 064 M Std size: 8 K

    Shared pool: 3 072 M Log Buffer: 13 632 K

    Load profile per second per Transaction per Exec by call

    ~~~~~~~~~~~~      ------------------  ----------------- ----------- -----------

    DB Time (s): 0.0 0.6 0.00 0.00

    DB CPU: 0.0 0.3 0.00 0.00

    Size: 458 720,6 8,755.7

    Logical reads: 245,7 12 874,2

    Block changes: 1 356.4 25.9

    Physical reads: 6.6 0.1

    Physical writings: 61.8 1.2

    The user calls: 38.8 2 033,7

    Analysis: 286,5 5.5

    Hard analysis: 0.5 0.0

    Treated W/A Mo: 1.7 0.0

    Logons: 1.2 0.0

    Runs: 801,1 15.3

    Cancellations: 6.1 0.1

    Operations: 52.4


    Indicators of the instance

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    Buffer % Nowait: 100.00 do NoWait %: 100.00

    Buffer % success: 99.98% W/A optimal, Exec: 100.00

    Library success %: 99,77% soft Parse: 99.82

    Run parse %: 64.24 latch hit %: 99.98

    Analyze the CPU to analyze Elapsd %: 53.15% Non-Parse CPU: 98.03


    Shared pool statistics Begin End

    ------  ------

    % Memory use: 10.50 12.79

    % SQL with executions > 1: 69,98 78,37

    % Memory for SQL w/exec > 1: 70.22 81,96

    Top 5 timed events Avg % Total

    ~~~~~~~~~~~~~~~~~~                                                   wait   Call

    Event waits time (s) (ms) time

    ----------------------------------------- ------------ ----------- ------ ------

    CPU time                                                       847          50.2

    ENQ: TX - 4 480 97 434 25.8 line lock conflict

    Log file sync 284 169 185 1 11.0

    log file parallel write 299 537 164 1 9.7

    log file sequential read 698 16 24 1.0

    Host CPU (processors: 2 hearts: Sockets 1: 0)

    ~ ~ ~ Medium load

    Begin End User System Idle WIO WCPU

    ------- -------   ------- ------- ------- ------- --------

    1.16 1.84 19.28 14.51 66.21 1.20 82.01


    Instance of CPU

    ~~~~~~~~~~~~                                       % Time (seconds)

    -------- --------------

    Host: Time (s) Total: 7,193.8

    Host: Availability of time processor (s): 2,430.7

    % of time host is busy: 33.8

    Instance: Time processor Total (s): 1,203.1

    % Busy CPU used, for example: 49.5

    Instance: Time of database total (s): 2,426.4

    % DB time waiting for CPU (resp. resources): 0.0


    Statistical memory Begin End

    ~~~~~~~~~~~~~~~~~                ------------ ------------

    Host Mem (MB): 16,384.0 16 384,0

    Use of LMS (MB): 7,136.0 7 136,0

    Use of PGA (Mo): 282.5 361.4

    Host % Mem used for SGA + PGA: 45.3 45.8

    Foreground wait events DB/Inst: XXXXXs Snaps: 5635-5636

    -> Only events with wait times Total (s) > =.001 are indicated

    --> sorted by Total desc waiting time, waits desc (idle last events)


    AVG % Total

    % Tim Total wait Wait Wait call

    Event is waiting for the time (s) (ms) /txn times

    ---------------------------- ------------ ---- ---------- ------ -------- ------

    ENQ: TX - line lock 4 480 0 434 97 contentio 0,0 25.8

    284 167 0 185 1 file synchronization log 1.5 11.0

    File I/O 8 741 of disk 0 4 operations 0.0 0.2

    direct path write 0 13 247 3 0.1 0.2

    DB file sequential read 6 058 0 1 0.0 0.1

    buffer busy waits 1 800 0 1 1 0,0.1

    SQL * Net more data to the client 29 161 0 1 0.2 0.1

    direct path read 7 696 0 1 0.0 0.0

    db file scattered read 316 0 1 2 0,0.0

    latch: shared pool 144 0 0 2 0,0.0

    Initialization of 30 0 0 3 0,0.0 CSS

    cursor: hand 10 0 0 9 0,0.0 S

    lock row cache 41 0 0 2 0,0.0

    latch: rank objects cache 19 0 0 3 0,0.0

    log file switch (private 8 0 0 7 0,0.0 str

    library cache: mutex X 28 0 0 2 0,0.0

    latch: cache buffers chains 54 0 0 1 0,0.0

    free lock 290 0 0 0.0 0.0

    sequential control file read 1 568 0 0 0.0 0.0

    switch logfile (4 0 0 6 0,0.0 control point

    Live sync 8 0 0 3 0,0.0 road

    latch: redo allocation 60 0 0 0 0.0.0

    SQL * Net break/reset for 34 0 0 1 0,0.0 customer

    latch: enqueue hash chains 45 0 0 0 0.0.0

    latch: cache buffers lru chain 7 0 0 2 0,0.0

    latch: allowance 5 0 0 1 0,0.0 session

    latch: object queue header 6 0 0 1 0,0.0 o

    Operation of metadata files ASM 30 0 0 0 0.0.0

    latch: in memory of undo latch 15 0 0 0.0 0.0

    latch: cancel the overall data 8 0 0 0 0.0.0

    SQL * Net client message 6 362 536 0 278 225 44 33.7

    jobq slave wait 7 270 100 3 635 500 0.0

    SQL * Net more data to 7 976 0 15 2 0,0 clien

    SQL * Net message to client 6 362 544 0 8 0 33.7

    -------------------------------------------------------------

    Context of the DB/Inst events waiting: XXXXXs clings: 5635-5636

    -> Only events with wait times Total (s) > =.001 are indicated

    --> sorted by Total desc waiting time, waits desc (idle last events)

    AVG % Total

    % Tim Total wait Wait Wait call

    Event is waiting for the time (s) (ms) /txn times

    ---------------------------- ------------ ---- ---------- ------ -------- ------

    log file parallel write 299 537 0 164 1 1.6 9.7

    log file sequential read 698 0 16 24 0.0 1.0

    db file parallel write 9 556 0 13 1 0,1.8

    146 0 10 70 0,0.6 startup operating system thread

    control file parallel write 2 037 0 2 1 0,0.1

    Newspaper archive e/s 35 0 1 30 0,0.1

    LGWR wait for redo copy 2 447 0 0 0.0 0.0

    async file IO DB present 9 556 0 0 0.1 0.0

    DB file sequential read 145 0 0 2 0,0.0

    File I/O disk 349 0 operations 0 0.0 0.0

    db file scattered read 30 0 0 4 0,0.0

    sequential control file read 5 837 0 0 0.0 0.0

    ADR block lu file 19 0 0 4 0,0.0

    Block ADR file write 5 0 0 15 0,0.0

    direct path write 14 0 0 2 0,0.0

    direct path read 3 0 0 7 0,0.0

    latch: shared pool 3 0 0 6 0,0.0

    single log file write 56 0 0 0.0 0.0

    latch: redo allocation 53 0 0 0 0.0.0

    latch: 1 0 0 3 0,0.0 active service list

    free latch 11 0 0 0 0.0.0

    CPI of RDBMS 5 314 523 57 189 182 1.7 message

    Space Manager: slave wa slowed 4 086 88 18 996 4649 0.0

    DIAG idle wait 7 185 100 1000 7 186 0.0

    Streams AQ: waiting time 2 50 4 909 # 0,0

    Streams AQ: qmn slowed slave 129 0 3 612 28002 0.0 w

    Streams AQ: Coordinator of the 258 50 3 612 14001 0,0 qmn

    SMON timer 2 43 3 605 83839 0.0

    PMON timer 99 1 199 2999 3 596 0.0

    SQL * Net client message 17 019 0 31 2 0.1

    SQL * Net message to client 12 762 0 0 0.1 0

    class slaves wait 28 0 0 0 0.0

    Thank you very much!

    Hello

    I think that your CPU is overloaded by your stress tests. You have one VCPU with 2 wires (2 LCPU), right? And the load average is greater than one. You have time DB which is not counted in (CPU time + wait events) and which comes no doubt from time spent in the runqueue.

    > Oracle recovery logs is not properly configured on this server, as in "Top 5 events of waiting", there are 2 for oder

    It is an error in statspack for show "log file parallel write here." This moment is historical and is included in 'log file sync '. And I don't think you have to redo misconfiguration. Waiting for 1ms to commit is ok. In OLTP you should have more than one validation in a user interaction so that the user don't worry not about 1 m in batch mode, unless you commit to each row, 1 DC to commit should not increase the total execution time.

    The fact that you have a lot of line lock (enq: TX - line lock conflict) but very little time (on average 97 ms) is probably a sign that testers are running simultaneously a charge affecting the same data. Their set of test data is perhaps too simple and short. An example: when stress tests of an order entry system if you run 1000 concurrent sessions, ordering the same product to the same customer, you can get this kind of symptoms, but the test we unrealistic.

    It's a high activity of 2000 calls per second, 52 transactions per second, user. But you also have low average active sessions, so the report probably covers a period of non-uniform activity, which makes the averages without meaning.

    So note to tell about the events of waiting here. But we don't have any info about 39% of DB time devoted to the CPU which is where something can be improved.

    Kind regards

    Franck.

  • Help interpret trace HANGANALYZE a file

    Hi friends


    Its my first time on trace HANGANALYZE files. I need some advice. I searched the forum but no thread was my doubt

    I am running on Windows 2003 Server x 64 10.2.0.1.0 Oracle. Simply because I review some grip of the instance. During the grip, we see no blocking, just sometimes a line lock conflict. But sometimes.
    We have 4 or 5 grip during the day. AWR reports very high "library cache lock" and "latch: shared pool ',"free latche"and" latch: cache buffers chains. During the blocking period, these are the Top 5 timed events 4.

    So I could generate a trace HANGANALYZE file since I was able to connect on the sql more see the primary blocker.

    SQL > setmypid oradebug
    SQL > oradebug hanganalyze 3;
    Hang in c:\oracle\product\10.2.0\admin\orcl2000\udump\oraprod_ora_50.trc analysis
    SQL > oradebug hanganalyze 3;
    Hang in c:\oracle\product\10.2.0\admin\orcl2000\udump\oraprod_ora_50.trc analysis

    This is the result I'd like help in interpreting

    ==============
    HANG THE ANALYSIS:
    ==============
    Channels open found:
    Channel 1: < cnode/sid/sess_srno/proc_ptr/ospid/wait_event >: < 0/1408/38922/0x3a63e638/6128/No Wait >-< 0/1547/18677/0x3a627730/6556/latch: cache library >
    Channel 2: < cnode/sid/sess_srno/proc_ptr/ospid/wait_event >: < 0/1510/64865/0x3e60cae8/1264/No Wait >-< 0/1412/11994/0x39634db8/4552/enq: TX - line lock conflict >

    Among other channels found:
    Channel 3: < cnode/sid/sess_srno/proc_ptr/ospid/wait_event >: < 0/1308/44023/0x3e5fe688/2720/No Wait >
    Channel 4: < cnode/sid/sess_srno/proc_ptr/ospid/wait_event >: < 0/1316/41853/0x3e6124a0/4148/No Wait >
    Channel 5: < cnode/sid/sess_srno/proc_ptr/ospid/wait_event >:
    ... and so on...

    Additional information that will be sent at higher levels:
    [level 4]: 2 node discharges-[REMOTE_WT] [SHEETS] [LEAF_NW]
    [level 5]: 18 node-[SINGLE_NODE] [SINGLE_NODE_NW] [IGN_DMP] landfills
    [level 6]: 2 node discharges-[NLEAF]
    [level 10]: 230 landfills node-[IGN]

    State of the nodes
    ([nodenum] / cnode/sid/sess_srno/session/ospid/state/start/finish / [adjlist] /predecessor):
    [1284] / 0/1285/16090/0x3e8fd3f0/5328/IGN/1/2 / / no
    [1285] / 0/1286/48916/0x3994b358/3432/IGN/3/4 / / no
    [1286] / 0/1287/21534/0x3a946c38/5836/IGN/5/6 / / no
    [1288] / 0/1289/35409/0x3994c790/4252/IGN/7/8 / / no
    [1293] / 0/1294/13836/0x3e901098/6176/IGN/9/10 / / no
    [1296] / 0/1297/1361/0x3e9024d0/5176/IGN/11/12 / / no
    [1298] / 0/1299/10760/0x3a94bd18/6028/IGN/13/14 / / no
    [1300] / 0/1301/19757 / 0 x 39951870/5912/IGN/15/16 / / no
    [1305] / 0/1306/29005/0x3e906178/5880/IGN/17/18 / / no
    [1306] / 0/1307/53304/0x399540e0/5900/IGN/19/20 / / no


    ... and so on...



    My doubt is.

    The agroalimentaire1 string may have blocked Chain2 on a library cache lock event. Perfect.


    How to find the id of sql that held the lock to work on?

    TKS a lot

    Edited by: KeenOnOracle May 21, 2013 14:49

    Anything in history of the Session Active (ASH), for this session (1048) for this time?

    Lordane Iotzov

  • DB file scattered read with Free (CBC) latch wait events

    Hi all

    On our production database server, we found about 50 sessions in lock Free (CBC) wait wait event with about 30 sessions waiting for db file scattered read... Also, the CPU load was obviously very high(90-99%).

    Our server has 4 CPU and OS is HP - UX
    Oracle version: 8.1.6.0.0
    The optimizer mode is RULE

    We found all sessions waiting performing this query below:
    (SELECT /*+ORDERED*/
            ud.user_id,
            DECODE (udl.new_user_name,
                    NULL, ud.user_name,
                    udl.new_user_name
                   ) AS user_name,
            udl.old_user_name, ud.PASSWORD, ud.status, au.first_name,
            au.last_name, sd.sd_id, sd.site_id, s.manager_number,
            sd.delivery_code, sd.site_server_id, au.email_address, au.outorg_id,
            udl.action_code, udl.action_category, udl.action_date AS action_date,
            au.admin_users_uid
       FROM user_delivery_log udl,
            user_delivery ud,
            appuser au,
           site_delivery sd,
            site_t s
      WHERE udl.action_date BETWEEN TO_DATE ('04/21/2009 17/46/25',
                                             'MM/DD/YYYY HH24/MI/SS'
                                            )
                                AND TO_DATE ('04/21/2009 17/47/26',
                                             'MM/DD/YYYY HH24/MI/SS'
                                            )
        AND (   (udl.action_category = 'I' AND BITAND (udl.action_code, 8195) != 0
                )
             OR (udl.action_category = 'U' AND BITAND (udl.action_code, 2079) != 0
                )
            )
        AND udl.site_id != 0
        AND ud.user_id = udl.user_id
        AND ud.sd_id = udl.sd_id
       AND ud.user_id = au.user_id
        AND ud.sd_id = sd.sd_id
        AND sd.site_id = s.site_id
        AND (sd.delivery_code = 'AEN')
        AND (   udl.new_user_name IS NOT NULL
            OR 0 =
                   ((SELECT /*+ORDERED*/
                            COUNT (*)
                       FROM user_delivery_log udl1
                      WHERE udl1.action_date
                               BETWEEN TO_DATE ('04/21/2009 17/46/25',
                                                'MM/DD/YYYY HH24/MI/SS'
                                               )
                                   AND TO_DATE ('04/21/2009 17/47/26',
                                               'MM/DD/YYYY HH24/MI/SS'
                                               )
                        AND (   (    udl1.action_category = 'I'
                                 AND BITAND (udl1.action_code, 8195) != 0
                                )
                             OR (    udl1.action_category = 'U'
                                 AND BITAND (udl1.action_code, 2079) != 0
                                )
                            )
                        AND udl1.site_id != 0
                        AND udl1.sd_id = ud.sd_id
                        AND udl1.user_id = ud.user_id
                        AND udl1.new_user_name IS NOT NULL))
            ))
    UNION
    (SELECT /*+ORDERED*/
            ud.user_id,
            DECODE (udl.new_user_name,
                    NULL, ud.user_name,
                    udl.new_user_name
                   ) AS user_name,
            NULL AS old_user_name, ud.PASSWORD, ud.status, au.first_name,
            au.last_name, sd.sd_id, sd.site_id, s.manager_number,
            sd.delivery_code, sd.site_server_id, au.email_address, au.outorg_id,
            1536, 'U', udl.action_date AS action_date, au.admin_users_uid
       FROM user_delivery_log udl,
            user_delivery ud,
            appuser au,
            site_delivery sd,
            site_t s
      WHERE (udl.user_id, udl.action_date) IN (
               SELECT   udl.user_id, MAX (action_date)
                   FROM user_delivery_log udl
                  WHERE udl.action_date <
                           TO_DATE ('04/21/2009 17/46/25',
                                    'MM/DD/YYYY HH24/MI/SS')
                    AND (   (    udl.action_category = 'I'
                             AND BITAND (udl.action_code, 8195) != 0
                            )
                         OR (    udl.action_category = 'U'
                             AND BITAND (udl.action_code, 2079) != 0
                            )
                        )
                    AND udl.user_id IN (
                           SELECT user_id
                             FROM appuser_log aul
                            WHERE aul.action_date
                                     BETWEEN TO_DATE ('04/21/2009 17/46/25',
                                                      'MM/DD/YYYY HH24/MI/SS'
                                                    )
                                         AND TO_DATE ('04/21/2009 17/47/26',
                                                      'MM/DD/YYYY HH24/MI/SS'
                                                     )
                              AND aul.action_category = 'U'
                              AND BITAND (aul.action_code, 5632) != 0)
                    AND udl.new_user_name IS NOT NULL
               GROUP BY user_id)
        AND udl.site_id != 0
        AND 0 =
               (SELECT COUNT (*)
                  FROM user_delivery_log udl, site_delivery sd
                 WHERE udl.action_date BETWEEN TO_DATE ('04/21/2009 17/46/25',
                                                        'MM/DD/YYYY HH24/MI/SS'
                                                       )
                                           AND TO_DATE ('04/21/2009 17/47/26',
                                                        'MM/DD/YYYY HH24/MI/SS'
                                                       )
                   AND (   (    udl.action_category = 'I'
                           AND BITAND (udl.action_code, 8195) != 0
                           )
                        OR (    udl.action_category = 'U'
                            AND BITAND (udl.action_code, 2079) != 0
                           )
                       )
                   AND udl.site_id != 0
                  AND udl.sd_id = sd.sd_id
                   AND udl.user_id = au.user_id
                   AND (sd.delivery_code = 'AEN'))
        AND ud.user_id = au.user_id
        AND ud.user_id = udl.user_id
        AND ud.sd_id = sd.sd_id
        AND sd.site_id = s.site_id
        AND (sd.delivery_code = 'AEN'))
    ORDER BY action_date ASC;
    
    Below is the execution plan:
    Execution Plan
    ----------------------------------------------------------
       0      SELECT STATEMENT Optimizer=RULE (Cost=341 Card=3 Bytes=312)
       1    0   SORT (UNIQUE) (Cost=342 Card=3 Bytes=312)
       2    1     UNION-ALL
       3    2       CONCATENATION
       4    3         FILTER
       5    4           NESTED LOOPS (Cost=12 Card=1 Bytes=100)
       6    5             NESTED LOOPS (Cost=11 Card=1 Bytes=94)
       7    6               NESTED LOOPS (Cost=10 Card=1 Bytes=77)
      8    7                 NESTED LOOPS (Cost=8 Card=1 Bytes=52)
       9    8                   TABLE ACCESS (BY INDEX ROWID) OF 'USER_DEL
              IVERY_LOG' (Cost=4 Card=2 Bytes=58)
      10    9                     INDEX (RANGE SCAN) OF 'IE1_USER_DELIVERY
              _LOG' (NON-UNIQUE) (Cost=3 Card=2)
      11    8                   TABLE ACCESS (BY INDEX ROWID) OF 'USER_DEL
              IVERY' (Cost=2 Card=1192563 Bytes=27428949)
      12   11                     INDEX (UNIQUE SCAN) OF 'IDX_PK_INV_USER_
              SD_ID' (UNIQUE) (Cost=1 Card=1192563)
      13    7                 TABLE ACCESS (BY INDEX ROWID) OF 'APPUSER' (
              Cost=2 Card=863102 Bytes=21577550)
      14   13                   INDEX (UNIQUE SCAN) OF 'PK_APPUSER' (UNIQU
              E) (Cost=1 Card=863102)
      15    6               TABLE ACCESS (BY INDEX ROWID) OF 'SITE_DELIVER
              Y' (Cost=1 Card=3494 Bytes=59398)
      16   15                 INDEX (UNIQUE SCAN) OF 'PK_NEW_SITE_DELIVERY
              ' (UNIQUE)
      17    5             TABLE ACCESS (BY INDEX ROWID) OF 'SITE_T' (Cost=
              1 Card=64113 Bytes=384678)
      18   17               INDEX (UNIQUE SCAN) OF 'PK_SITE_T' (UNIQUE)
      19    4           TABLE ACCESS (BY INDEX ROWID) OF 'USER_DELIVERY_LO
              G' (Cost=6 Card=1 Bytes=27)
      20   19             INDEX (RANGE SCAN) OF 'IE1_USER_DELIVERY_LOG' (N
              ON-UNIQUE) (Cost=3 Card=1)
      21    3         FILTER
      22   21           NESTED LOOPS (Cost=12 Card=1 Bytes=100)
      23   22             NESTED LOOPS (Cost=11 Card=1 Bytes=94)
      24   23               NESTED LOOPS (Cost=10 Card=1 Bytes=77)
      25   24                 NESTED LOOPS (Cost=8 Card=1 Bytes=52)
      26   25                   TABLE ACCESS (BY INDEX ROWID) OF 'USER_DEL
              IVERY_LOG' (Cost=4 Card=2 Bytes=58)
      27   26                     INDEX (RANGE SCAN) OF 'IE1_USER_DELIVERY
             _LOG' (NON-UNIQUE) (Cost=3 Card=2)
      28   25                   TABLE ACCESS (BY INDEX ROWID) OF 'USER_DEL
              IVERY' (Cost=2 Card=1192563 Bytes=27428949)
      29   28                     INDEX (UNIQUE SCAN) OF 'IDX_PK_INV_USER_
              SD_ID' (UNIQUE) (Cost=1 Card=1192563)
      30   24                 TABLE ACCESS (BY INDEX ROWID) OF 'APPUSER' (
              Cost=2 Card=863102 Bytes=21577550)
      31   30                   INDEX (UNIQUE SCAN) OF 'PK_APPUSER' (UNIQU
             E) (Cost=1 Card=863102)
      32   23               TABLE ACCESS (BY INDEX ROWID) OF 'SITE_DELIVER
              Y' (Cost=1 Card=3494 Bytes=59398)
      33   32                 INDEX (UNIQUE SCAN) OF 'PK_NEW_SITE_DELIVERY
              ' (UNIQUE)
      34   22             TABLE ACCESS (BY INDEX ROWID) OF 'SITE_T' (Cost=
              1 Card=64113 Bytes=384678)
      35   34               INDEX (UNIQUE SCAN) OF 'PK_SITE_T' (UNIQUE)
      36    2       NESTED LOOPS (Cost=314 Card=1 Bytes=112)
      37   36         NESTED LOOPS (Cost=313 Card=1 Bytes=106)
      38   37           NESTED LOOPS (Cost=312 Card=1 Bytes=89)
      39   38             NESTED LOOPS (Cost=310 Card=1 Bytes=64)
      40   39               NESTED LOOPS (Cost=306 Card=1 Bytes=41)
      41   40                 VIEW OF 'VW_NSO_1' (Cost=302 Card=1 Bytes=22
              )
      42   41                   SORT (GROUP BY) (Cost=302 Card=1 Bytes=35)
      43   42                     NESTED LOOPS (Cost=299 Card=1 Bytes=35)
      44   43                       TABLE ACCESS (FULL) OF 'APPUSER_LOG' (
              Cost=93 Card=1 Bytes=16)
      45   43                       TABLE ACCESS (FULL) OF 'USER_DELIVERY_
              LOG' (Cost=206 Card=73505 Bytes=1396595)
      46   40                 TABLE ACCESS (BY INDEX ROWID) OF 'USER_DELIV
              ERY_LOG' (Cost=4 Card=429251 Bytes=8155769)
      47   46                   INDEX (RANGE SCAN) OF 'IE1_USER_DELIVERY_L
              OG' (NON-UNIQUE) (Cost=3 Card=429251)
      48   39               TABLE ACCESS (BY INDEX ROWID) OF 'USER_DELIVER
             Y' (Cost=4 Card=1192563 Bytes=27428949)
      49   48                 INDEX (RANGE SCAN) OF 'IDX_PK_INV_USER_SD_ID
              ' (UNIQUE) (Cost=3 Card=1192563)
      50   38             TABLE ACCESS (BY INDEX ROWID) OF 'APPUSER' (Cost
              =2 Card=43156 Bytes=1078900)
      51   50               INDEX (UNIQUE SCAN) OF 'PK_APPUSER' (UNIQUE) (
              Cost=1 Card=43156)
      52   51                 NESTED LOOPS (Cost=7 Card=1 Bytes=31)
      53   52                   TABLE ACCESS (BY INDEX ROWID) OF 'USER_DEL
             IVERY_LOG' (Cost=6 Card=1 Bytes=24)
      54   53                     INDEX (RANGE SCAN) OF 'IE1_USER_DELIVERY
              _LOG' (NON-UNIQUE) (Cost=3 Card=1)
      55   52                   TABLE ACCESS (BY INDEX ROWID) OF 'SITE_DEL
              IVERY' (Cost=1 Card=3494 Bytes=24458)
      56   55                     INDEX (UNIQUE SCAN) OF 'PK_NEW_SITE_DELI
              VERY' (UNIQUE)
      57   37           TABLE ACCESS (BY INDEX ROWID) OF 'SITE_DELIVERY' (
              Cost=1 Card=3494 Bytes=59398)
      58   57             INDEX (UNIQUE SCAN) OF 'PK_NEW_SITE_DELIVERY' (U
              NIQUE)
      59   36         TABLE ACCESS (BY INDEX ROWID) OF 'SITE_T' (Cost=1 Ca
              rd=64113 Bytes=384678)
      60   59           INDEX (UNIQUE SCAN) OF 'PK_SITE_T' (UNIQUE)
    For a clear view of the execution plan:
    Plan Table
    --------------------------------------------------------------------------------------------------------
    | Operation                 |  Name    |  Rows | Bytes|  Cost  | Pstart| Pstop |
    --------------------------------------------------------------------------------
    | SELECT STATEMENT          |          |     3 |  312 |    341 |       |       |
    |  SORT UNIQUE              |          |     3 |  312 |    342 |       |       |
    |   UNION-ALL               |          |       |      |        |       |       |
    |    CONCATENATION          |          |       |      |        |       |       |
    |     FILTER                |          |       |      |        |       |       |
    |      NESTED LOOPS         |          |     1 |  100 |     12 |       |       |
    |       NESTED LOOPS        |          |     1 |   94 |     11 |       |       |
    |        NESTED LOOPS       |          |     1 |   77 |     10 |       |       |
    |         NESTED LOOPS      |          |     1 |   52 |      8 |       |       |
    |          TABLE ACCESS BY I|USER_DELI |     2 |   58 |      4 |       |       |
    |           INDEX RANGE SCAN|IE1_USER_ |     2 |      |      3 |       |       |
    |          TABLE ACCESS BY I|USER_DELI |     1M|   26M|      2 |       |       |
    |           INDEX UNIQUE SCA|IDX_PK_IN |     1M|      |      1 |       |       |
    |         TABLE ACCESS BY IN|APPUSER   |   863K|   20M|      2 |       |       |
    |          INDEX UNIQUE SCAN|PK_APPUSE |   863K|      |      1 |       |       |
    |        TABLE ACCESS BY IND|SITE_DELI |     3K|   58K|      1 |       |       |
    |         INDEX UNIQUE SCAN |PK_NEW_SI |     3K|      |        |       |       |
    |       TABLE ACCESS BY INDE|SITE_T    |    64K|  375K|      1 |       |       |
    |        INDEX UNIQUE SCAN  |PK_SITE_T |    64K|      |        |       |       |
    |      TABLE ACCESS BY INDEX|USER_DELI |     1 |   27 |      6 |       |       |
    |       INDEX RANGE SCAN    |IE1_USER_ |     1 |      |      3 |       |       |
    |     FILTER                |          |       |      |        |       |       |
    |      NESTED LOOPS         |          |     1 |  100 |     12 |       |       |
    |       NESTED LOOPS        |          |     1 |   94 |     11 |       |       |
    |        NESTED LOOPS       |          |     1 |   77 |     10 |       |       |
    |         NESTED LOOPS      |          |     1 |   52 |      8 |       |       |
    |          TABLE ACCESS BY I|USER_DELI |     2 |   58 |      4 |       |       |
    |           INDEX RANGE SCAN|IE1_USER_ |     2 |      |      3 |       |       |
    |          TABLE ACCESS BY I|USER_DELI |     1M|   26M|      2 |       |       |
    |           INDEX UNIQUE SCA|IDX_PK_IN |     1M|      |      1 |       |       |
    |         TABLE ACCESS BY IN|APPUSER   |   863K|   20M|      2 |       |       |
    |          INDEX UNIQUE SCAN|PK_APPUSE |   863K|      |      1 |       |       |
    |        TABLE ACCESS BY IND|SITE_DELI |     3K|   58K|      1 |       |       |
    |         INDEX UNIQUE SCAN |PK_NEW_SI |     3K|      |        |       |       |
    |       TABLE ACCESS BY INDE|SITE_T    |    64K|  375K|      1 |       |       |
    |        INDEX UNIQUE SCAN  |PK_SITE_T |    64K|      |        |       |       |
    |    NESTED LOOPS           |          |     1 |  112 |    314 |       |       |
    |     NESTED LOOPS          |          |     1 |  106 |    313 |       |       |
    |      NESTED LOOPS         |          |     1 |   89 |    312 |       |       |
    |       NESTED LOOPS        |          |     1 |   64 |    310 |       |       |
    |        NESTED LOOPS       |          |     1 |   41 |    306 |       |       |
    |         VIEW              |VW_NSO_1  |     1 |   22 |    302 |       |       |
    |          SORT GROUP BY    |          |     1 |   35 |    302 |       |       |
    |           NESTED LOOPS    |          |     1 |   35 |    299 |       |       |
    |            TABLE ACCESS FU|APPUSER_L |     1 |   16 |     93 |       |       |
    |            TABLE ACCESS FU|USER_DELI |    73K|    1M|    206 |       |       |
    |         TABLE ACCESS BY IN|USER_DELI |   429K|    7M|      4 |       |       |
    |          INDEX RANGE SCAN |IE1_USER_ |   429K|      |      3 |       |       |
    |        TABLE ACCESS BY IND|USER_DELI |     1M|   26M|      4 |       |       |
    |         INDEX RANGE SCAN  |IDX_PK_IN |     1M|      |      3 |       |       |
    |       TABLE ACCESS BY INDE|APPUSER   |    43K|    1M|      2 |       |       |
    |        INDEX UNIQUE SCAN  |PK_APPUSE |    43K|      |      1 |       |       |
    |         NESTED LOOPS      |          |     1 |   31 |      7 |       |       |
    |          TABLE ACCESS BY I|USER_DELI |     1 |   24 |      6 |       |       |
    |           INDEX RANGE SCAN|IE1_USER_ |     1 |      |      3 |       |       |
    |          TABLE ACCESS BY I|SITE_DELI |     3K|   23K|      1 |       |       |
    |           INDEX UNIQUE SCA|PK_NEW_SI |     3K|      |        |       |       |
    |      TABLE ACCESS BY INDEX|SITE_DELI |     3K|   58K|      1 |       |       |
    |       INDEX UNIQUE SCAN   |PK_NEW_SI |     3K|      |        |       |       |
    |     TABLE ACCESS BY INDEX |SITE_T    |    64K|  375K|      1 |       |       |
    |      INDEX UNIQUE SCAN    |PK_SITE_T |    64K|      |        |       |       |
    --------------------------------------------------------------------------------
    All the locks of CBC point to the APPUSER_LOG and USER_DELIVERY_LOG tables.
    Could someone help me please in the setting of this query, I need to avoid latch and scattered read... .as application knows huge slow...

    -Yasser

    YasserRACDBA wrote:

    Oracle version: 8.1.6.0.0
    The optimizer mode is RULE

    We found all sessions waiting performing this query below:

    (SELECT /*+ORDERED*/
    

    Even if you run based on rules in general, this query will run based on costs due to the indicator.
    Do you have statistics in place to support the OBC?

    | VIEW |VW_NSO_1 | 1 | 22 | 302 | | |
    | SORT GROUP BY | | 1 | 35 | 302 | | |
    | NESTED LOOPS | | 1 | 35 | 299 | | |
    | TABLE ACCESS FU|APPUSER_L | 1 | 16 | 93 | | |
    | TABLE ACCESS FU|USER_DELI | 73K| 1M| 206 | | |

    This is typical of an unnested "IN" subquery - it looks like the optimiser has unnested your double-level IN subquery finding the maximum action date for a user who has done something in the last minute into a massive group and aggregate (so we've got unnesting and complex view merging all in one - I don't think I've seen that in 8i before).

    Your db file scattered reads are probably extreme because (a) the USER_DELIxxxx table is big, and (b) the number of rows you are getting from the APPUSER_Lxxx table is more than the one that Oracle expects. You may find that collecting stats on APPUSER_Lxxx addresses this issue

    I can't see any reason why the appuser_log table should be subject to cache buffers chains latch problems - if the plan is true than the only thing it suffers is one full scan. However, given the "one row" estimate that comes out of the VW_NOS_1 line, the optimizer is free to do all sorts of silly things as it works through the chain of nested loops, and if the number of rows from the driver is much more than one then the indexed activity against USER_DELIxxx could be huge.

    For example:

    |         TABLE ACCESS BY IN|USER_DELI |   429K|    7M|      4 |       |       |
    |          INDEX RANGE SCAN |IE1_USER_ |   429K|      |      3 |       |       |
    

    Is the next step in this nested loop (and similar things happen more down and in the previous sections of the plan) suggesting that a systematic index scan rnage will pick up thousands of lines - which, even if only about right, could mean a huge number of buffer visits to the table. ... the numbers don't make real sense, either by the way, but once again, it is probably due to the lack of appropriate statistics.

    Concerning
    Jonathan Lewis
    http://jonathanlewis.WordPress.com
    http://www.jlcomp.demon.co.UK

    "Science is more than a body of knowledge; It's a way of thinking. "
    Carl Sagan

  • physical_read_requests &gt; disk_reads

    Hello

    I was wondering about sqls in v$ sql where physical_read_requests vs disk_reads.
    I know when instead it is likely reading diluvium, but I'm trying to understand some SQLs I see with, for example, 900 k physical_read_requests and disk 4 bed.

    Query your database for:

    select sql_id, disk_reads, physical_read_requests
    from v$sql 
    where physical_read_requests > disk_reads
    /
    
    SQL_ID        DISK_READS PHYSICAL_READ_REQUESTS
    ------------- ---------- ----------------------
    fd9qbhuy1hgrx          0                     14
    2xucgknahj256        167                    181
    934ur8r7tqbjx          1                     15
    934ur8r7tqbjx          0                      7
    fpvps147hqh7g          1                      8
    89p81j073yhv8         26                     40
    akj15bfw7ypt4         41                     48
    bn4b3vjw2mj3u          0                     35
    


    Any clue?

    Salman Qureshi says:

    I here speculates that if even block (for which a physical read request is issued to a SQL), some other SQL/session could also have requested a physical read (and a reading of disc) - occupied buffer waiting happen also. This disc has played for this sql would not increase (because the data has already been read form the disc at the request of the other sql/session). So in this case, physical_read_request for this SQL has increased, but not disk_read.

    Good idea, but I don't think it should work like that.

    The second session who wanted to read the block would be the cache buffers chains get found block one buffer that takes this block pinned exclusively in the State "be read from disk", so do not try to read it, so should not change either of the two stats. (He makes a good point that the session not next - but it's out of scope for this thread.)

    I've been speculating in the sense of an instrumentation error - a task update a single statistic, but not the other. The possibilities include:

    readings for dynamic sampling

    readings of global temporary tables

    readings of materialized "with" subqueries

    readings of tablespace temp for mergers of sorts/axe/bitmap

    journey live readings

    readings of the flash cache

    Exadata ' no data not necessary "read the applications

    However, a much simpler explanation is that not all the readings are bed data block, so eventually the counties (for example) physical_read_requests controlfile read requests while disk_reads number just blocks to read data files.  Try this:

    SQL > select disk_reads, physical_read_requests, sql_text from v$ sql where sql_id = "adnu4f081dhar";

    DISK_READS PHYSICAL_READ_REQUESTS SQL_TEXT

    ---------- ---------------------- --------------------------------------------------------------------------------

    0 28 select current_scn in the database of v$

    SQL > select current_scn from v$ database;

    CURRENT_SCN

    -----------

    1.2670E + 13

    SQL > select disk_reads, physical_read_requests, sql_text from v$ sql where sql_id = "adnu4f081dhar";

    DISK_READS PHYSICAL_READ_REQUESTS SQL_TEXT

    ---------- ---------------------- --------------------------------------------------------------------------------

    0 35 select current_scn in the database of v$

    SQL > select current_scn from v$ database;

    CURRENT_SCN

    -----------

    1.2670E + 13

    SQL > select disk_reads, physical_read_requests, sql_text from v$ sql where sql_id = "adnu4f081dhar";

    DISK_READS PHYSICAL_READ_REQUESTS SQL_TEXT

    ---------- ---------------------- --------------------------------------------------------------------------------

    0 42 select current_scn in the database of v$

    The declaration card 7 physical_read_requests and not disk_reads on each execution and at the same time I see 7 waits 'control file sequential read '.

    Concerning

    Jonathan Lewis

  • Locking in AWR errors

    Hello

    yesterday a fix has been applied in 11.2.0.2 If Oracle Support for a bug, we faced. But today we faced a massive performance problem in the database about lock contention. Compared to AWR my during the previous day, logical reads also rose 5 times if the load was the same. Kindly help.
     DB Name     DB Id     Instance     Inst num     Startup Time     Release     RAC
    ORS     706933460     ORS     1     25-Sep-12 18:09     11.2.0.2.0     NO
    Host Name     Platform     CPUs     Cores     Sockets     Memory (GB)
    orsfo     AIX-Based Systems (64-bit)      12      12            69.50
    Snap Id     Snap Time     Sessions     Cursors/Session
    Begin Snap:     32939     26-Sep-12 15:15:19     2672      6.5
    End Snap:     32940     26-Sep-12 15:30:28     2650      6.7
    Elapsed:            15.15 (mins)            
    DB Time:            4,589.98 (mins)            
    Report Summary
    Cache Sizes
    
    Begin     End          
    Buffer Cache:      10,240M      10,240M     Std Block Size:      8K
    Shared Pool Size:      4,096M      4,096M     Log Buffer:      44,624K
    Load Profile
    
    Per Second     Per Transaction     Per Exec     Per Call
    DB Time(s):      303.1      0.5      0.08      0.14
    DB CPU(s):      11.8      0.0      0.00      0.01
    Redo size:      302,576.0      538.2            
    Logical reads:      1,840,179.4      3,273.2            
    Block changes:      2,405.5      4.3            
    Physical reads:      253.7      0.5            
    Physical writes:      742.9      1.3            
    User calls:      2,200.3      3.9            
    Parses:      770.1      1.4            
    Hard parses:      19.6      0.0            
    W/A MB processed:      9.7      0.0            
    Logons:      1.9      0.0            
    Executes:      3,841.7      6.8            
    Rollbacks:      508.3      0.9            
    Transactions:      562.2                  
    Instance Efficiency Percentages (Target 100%)
    
    Buffer Nowait %:      100.00     Redo NoWait %:      99.99
    Buffer Hit %:      99.99     In-memory Sort %:      100.00
    Library Hit %:      98.63     Soft Parse %:      97.46
    Execute to Parse %:      79.96     Latch Hit %:      99.71
    Parse CPU to Parse Elapsd %:      3.33     % Non-Parse CPU:      98.51
    Shared Pool Statistics
    
    Begin     End
    Memory Usage %:      92.94      92.88
    % SQL with executions>1:      85.15      83.17
    % Memory for SQL w/exec>1:      79.07      75.70
    Top 5 Timed Foreground Events
    
    Event Waits               Time(s)           Avg wait (ms) % DB time      Wait Class
    latch: cache buffers chains     357,012          112,342      315      40.79      Concurrency
    SQL*Net message from dblink     2,900,096     14,069      5      5.11      Network
    DB CPU                      10,722                  3.89      
    latch free                12,947           3,072      237      1.12      Other
    SQL*Net more data from dblink     382,236           1,017      3      0.37      Network 
    Kind regards
    Karan

    853100 wrote:

    When I check my "ordered by gets Section SQL ', all queries on my table increases by a factor of 10 (for example 18 k gets by performance yesterday become 140 k gets by performance today). Coupled with the fact that a key on the table query has executed only 500 times yesterday and 10000 times today. This query is executed every time a response is required from the front.

    Thus, the number of executions has increased by a factor of 20 and the number of receives a performance has increased by a factor of 8 - what anomaly will give you the best reduction of workload, if you can stop what is happening? Conversely - Gets an increase COULD be due to a change in the execution plan, in order to check plan yesterday today against (I use script $ORACLE_HOME/rdbms/admin/awrsqrpt); If not, is there an UPDATE (delete or insert) on the table which is now also past 20 times more often, if so, gets the increase COULD be gets against the rollback segment as a session does more work for generating consistent images in reading, in order to check the activity of the forum stats on "undo records applied.

    >

    PS: Restorations per second were similar compared to these two days. Applied patch was also 10093383.

    Look at the stat "rollback changes - undo records applied" to see if cancellations are "real" or whether a restore 'no' non-essential radiation of the intermediate layer.

    Concerning
    Jonathan Lewis
    http://jonathanlewis.WordPress.com
    Author: core Oracle

    P.S. It might have been helpful see a few examples of the query that runs 20 times as often.

  • Interpreting the Trace file.

    Hi, I use 10.2.0.4.0 oracle version.

    I have some info to trace file as below, for one of the query. So how should I interpret the trace file? What is the problem in the application and scope of the improvements in the query? Please note that I withdrew the request and plans of the trace file, I've posted only the sections of waiting.
    call     count       cpu    elapsed       disk      query    current        rows
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    Parse        1      0.14       0.13          0          0          1           0
    Execute      1      6.63     162.12      33540      72921        383           0
    Fetch    17272    178.89    1933.95     274835    3147603         20      259063
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    total    17274    185.66    2096.21     308375    3220524        404      259063
    
    Misses in library cache during parse: 1
    Optimizer mode: CHOOSE
    Parsing user id: 36  
    
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      control file sequential read                    4        0.00          0.00
      db file sequential read                    302812        0.62       1913.89
      latch: cache buffers chains                     3        0.04          0.04
      direct path write temp                        501        0.01          0.30
      SQL*Net message to client                   17272        0.00          0.04
      db file scattered read                        120        0.02          0.63
      direct path read temp                         608        0.14          1.71
      SQL*Net message from client                 17272       44.81      31865.74
      SQL*Net more data to client                    15        0.00          0.00
      latch: object queue header operation            1        0.00          0.00
      latch: library cache                            3        0.03          0.04
      latch: library cache pin                        1        0.00          0.00
      latch: cache buffer handles                     1        0.00          0.00
    
    
    
    ********************************************************************************
    
    OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
    
    call     count       cpu    elapsed       disk      query    current        rows
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    Parse        1      0.14       0.13          0          0          1           0
    Execute      1      6.63     162.12      33540      72921        383           0
    Fetch    17272    178.89    1933.95     274835    3147603         20      259063
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    total    17274    185.66    2096.21     308375    3220524        404      259063
    
    Misses in library cache during parse: 1
    
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      SQL*Net message to client                   17275        0.00          0.04
      SQL*Net message from client                 17274       75.57      31941.39
      SQL*Net more data from client                   2        0.00          0.01
      db file sequential read                    302812        0.62       1913.89
      control file sequential read                    4        0.00          0.00
      latch: cache buffers chains                     3        0.04          0.04
      direct path write temp                        501        0.01          0.30
      db file scattered read                        120        0.02          0.63
      direct path read temp                         608        0.14          1.71
      SQL*Net more data to client                    15        0.00          0.00
      latch: object queue header operation            1        0.00          0.00
      latch: library cache                            3        0.03          0.04
      latch: library cache pin                        1        0.00          0.00
      latch: cache buffer handles                     1        0.00          0.00
    
    
    OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
    
    call     count       cpu    elapsed       disk      query    current        rows
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    Parse       11      0.02       0.01          0          0          0           0
    Execute    348      0.20       0.17          0          0          1           0
    Fetch      367      0.06       0.37         59       1187          0        3806
    ------- ------  -------- ---------- ---------- ---------- ----------  ----------
    total      726      0.28       0.56         59       1187          1        3806
    
    Misses in library cache during parse: 11
    Misses in library cache during execute: 10
    
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      db file sequential read                        59        0.01          0.32
    
        1  user  SQL statements in session.
      348  internal SQL statements in session.
      349  SQL statements in session.
    ********************************************************************************

    But rewrite the SQL code will probably be irrelevant.

    If most of the numbers in the file plan and execution of trace is correct, then it is not more time is lost somewhere in the network that you send to 260 000 lines back and forward with a size of 15 lines extraction (the default in sql * more)?

    You must increase the size of mining significantly.

    And you can watch some of the writings of Charles Hooper on the monitoring network:
    http://hoopercharles.WordPress.com/category/network-monitoring/

  • Buffer max metadata

    Hello dear gurus!

    According to ducumentation:
    the cache buffers chains latches are used to protect a list of buffers in the buffer cache.
    These locks are used when searching, adding, or removing a buffer from the buffer cache

    Blocks in the buffer cache are placed on linked lists (buffer cache strings) having a stranglehold on a hash table
    how to fit between them, buckets, tampons, handles, blocks, and a hash table?

    I'm usung 10.2.0.5

    Thank you and best regards,
    Pavel

    >

    Hi Pavel,

    How match between them, buckets, tampons, handles, blocks, and a hash table?

    It's a matter of big and certainly not one that can respond to a single post.

    What I would say to you is to get your hands on Tom Kyte Expert database Architecture
    and work there. This will give you a good overview of the architecture of Oracle and others
    of what you mentioned above.

    Then you can watch Jonathan Lewis base Oracle and Steve Adams Oracle 8i internal services.

    Keep an eye on the blog of Jonathan Lewis, Tanel Poder and follow their blog rolls.

    It of a great topic and will require a lot of reading, but start by Tom Kyte - you can't
    need more according to your needs.

    HTH,

    Paul...

    Pavel

  • Slow performance of async IO on Oracle Linux 6.2

    Hello

    The platform is Oracle 11.2.0.3 64-bit on Oracle Linux 6.2 64-bit. The storage is iSCSI.
    After you enable async i/o on the level of database (disk_asynch_io = TRUE and filesystemio_options = SETALL), we notice a significantly lower yield (up to 25% slower).
    AIO-max-nr is set to 1048576.

    No idea what could be the problem? There are no network problems, the CPU load seems ok...

    This is the result of a level 8 10046 trace:

    Asynchronous i/o disabled:
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      Disk file operations I/O                        4        0.00          0.00
      db file sequential read                    166147        2.21         84.01
      db file scattered read                       2406        1.74          7.05
      undo segment extension                          2        0.01          0.01
      log file switch completion                      2        0.02          0.05
      log buffer space                                4        2.86          2.92
      latch free                                      1        0.00          0.00
      db file parallel read                           2        0.15          0.19
      direct path write temp                        577        0.53          2.46
      latch: shared pool                             11        0.00          0.00
      direct path read temp                         577        0.01          0.54
      latch: object queue header operation            1        0.00          0.00
    
           1  session in tracefile.
          11  user  SQL statements in trace file.
           6  internal SQL statements in trace file.
          17  SQL statements in trace file.
          13  unique SQL statements in trace file.
     5583431  lines in trace file.
         409  elapsed seconds in trace file.
    E/s asynchronous active:
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      Disk file operations I/O                        3        0.00          0.00
      db file sequential read                    143264        0.78        242.64
      latch: cache buffers lru chain                  2        0.00          0.00
      latch: object queue header operation            2        0.00          0.00
      log file switch completion                      3        0.06          0.15
      db file scattered read                       2333        0.07          4.93
      asynch descriptor resize                        1        0.00          0.00
      log buffer space                                3        0.13          0.25
      direct path read temp                          97        0.00          0.32
      reliable message                               67        0.00          0.03
      latch: cache buffers chains                     1        0.00          0.00
    
           1  session in tracefile.
          11  user  SQL statements in trace file.
           6  internal SQL statements in trace file.
          17  SQL statements in trace file.
          13  unique SQL statements in trace file.
     5560145  lines in trace file.
         716  elapsed seconds in trace file.
    Kind regards
    Matthias Hoys
    matthiashoys.WordPress.com

    Published by: mhoys on May 16, 2012 11:50

    mhoys wrote:
    I don't see any problems at the level of the disc. Using statspack, I noticed that response time are most often less than 10 m, I run the same statements and I'm the only one with access to the system at the moment, there is no other load.

    Matthias

    Have you checked if this 10ms rose? I means that you set for this base lines?
    Also, it feels like previous database has been able to treat more quickly because it can recover the file system cache blocks. But will directly the database may not be available benefits. This is just my hypothesis in your case. So this review with guys admin OS that async is configured correctly or not.

    BTW definition NONE - disabled both live e/s and e/s asynchronous. So I think that one of them is wrong and you should check that one?
    If its Direct IO-> it would be because of the hidden file system now is not in use, decrease in performance both (assumption)
    If it is Async-> check if async is configured correctly.

    The best tool for you to understand what is happening is to use TRACK through 10046 sessions.

  • PerfMon statement drive dramatic increase in access to Oracle starting times

    Hello

    My Oracle 10 g (10.2.0.4) server is hosted on a windows 2003 server.
    The data files are stored on a disk RAID1 on a dedicated partition array: currently 30 free concerts on 180, that shouldn't be a concern unless I am mistaken, because the data files have been created in the form of 10 GB of files with no automatic growth. I add a new data file whenever I need more space for my tables (alerts when 80% used).

    Since 2 days I feel a loss of theatrical representation:
    The EM console reports nothing special (no alarm storage-related) outside the need for a more paged memory.
    I have a reorg when segmentation Adviser suggests.
    My the optimizer statistics are calculated by the default scheduled task.

    The strange thing I noticed is that as soon as I start the database, there is a huge disk activity increase even if no request at all is submitted to the database.
    PerfMon reports current length of queue of > 1000 disc and disc > 3000 ms access time

    CPU is 2% of the activity on the 4-CPU server.
    I have a lot of spare memory (currently 3 used on the 16 GB).

    It is a server dev for ETL processes, there is very little simultaneous connections.

    Any suggestion is welcome.

    AWR report is available here
    http://min.us/mqnXQhd5Z

    Published by: user10799939 on March 22, 2012 09:30
    
    Cache Sizes
    ~~~~~~~~~~~                       Begin        End
                                 ---------- ----------
                   Buffer Cache:     1,296M     1,296M  Std Block Size:         8K
               Shared Pool Size:       160M       160M      Log Buffer:    14,364K 
    
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                                       ---------------       ---------------
                      Redo size:            460,955.72 ;         2,477,358.63
                  Logical reads:              3,392.16 ;            18,230.80
                  Block changes:              6,451.93 ;            34,675.22
                 Physical reads:                  2.92 ;                15.67
                Physical writes:                394.52 ;             2,120.28
                     User calls:                  1.69 ;                 9.08
                         Parses:                  3.31 ;                17.81
                    Hard parses:                  0.17 ;                 0.90
                          Sorts:                  1.32 ;                 7.09
                         Logons:                  0.06 ;                 0.31
                       Executes:                  7.01 ;                37.68
                   Transactions:                  0.19 
    
      % Blocks changed per Read:  190.20 ;   Recursive Call %:    96.23
    Rollback per transaction %:    0.30 ;      Rows per Sort:    14.41 
    
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:   99.98 ;      Redo NoWait %:   99.86
                Buffer  Hit   %:   99.92 ;   In-memory Sort %:  100.00
                Library Hit   %:   96.30 ;       Soft Parse %:   94.96
             Execute to Parse %:   52.74 ;        Latch Hit %:   99.07
    Parse CPU to Parse Elapsd %:    0.35 ;    % Non-Parse CPU:   99.30 
    
    Shared Pool Statistics        Begin    End
                                  ------  ------
                 Memory Usage %:   75.48 ;  75.51
        % SQL with executions>1:   79.92 ;  85.03
      % Memory for SQL w/exec>1:   77.07 ;  70.09 
    
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    ------------------------------ ------------ ----------- ------ ------ ----------
    db file sequential read               9,052      17,688   1954   51.3 ;  User I/O
    log file switch (checkpoint in        5,303       4,649    877   13.5 Configurat
    log file switch completion            4,245       4,023    948   11.7 Configurat
    wait for a undo record               32,393       3,531    109   10.3 ;     Other
    db file parallel write               18,771       3,437    183   10.0 System I/O 
    

    haven't seen as much waiting on average. For example, 877ms for "log file switch" is above the threshold. And other events waiting too...

    
    Time Model Statistics                DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
    -> Total time in database user-calls (DB Time): 34446.5s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name 
    
    Statistic Name                                       Time (s) % of DB Time
    ------------------------------------------ ------------------ ------------
    sql execute elapsed time                              4,008.5 ;        11.6
    parse time elapsed                                      352.9 ;         1.0
    hard parse elapsed time                                 352.7 ;         1.0
    PL/SQL compilation elapsed time                         120.1 ;          .3
    DB CPU                                                   61.8 ;          .2
    failed parse elapsed time                                21.3 ;          .1
    PL/SQL execution elapsed time                             8.0 ;          .0
    connection management call elapsed time                   0.0 ;          .0
    hard parse (sharing criteria) elapsed time                0.0 ;          .0
    repeated bind elapsed time                                0.0 ;          .0
    hard parse (bind mismatch) elapsed time                   0.0 ;          .0
    DB time                                              34,446.5 ;         N/A
    background elapsed time                              14,889.7 ;         N/A
    background cpu time                                      39.0 ;         N/A
              ------------------------------------------------------------- 
    
    Wait Class                            DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc 
    
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    -------------------- ---------------- ------ ---------------- ------- ---------
    User I/O                       10,515     .1           17,785    1691      15.8
    Configuration                  10,186   79.5 ;           8,865     870      15.3
    System I/O                     27,619     .0            8,774     318      41.6
    Other                          57,768   98.3 ;           6,915     120      87.0
    Commit                          2,634   88.6 ;           2,481     942       4.0
    Concurrency                     2,847   75.4 ;           2,240     787       4.3
    Application                       219    2.3 ;              23     105       0.3
    Network                         4,790     .0                0       0       7.2
              ------------------------------------------------------------- 
    

    once seen it yet, there is an expectation very strong user IO

    
    Wait Events                          DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last) 
    
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    ---------------------------- -------------- ------ ----------- ------- ---------
    db file sequential read               9,052     .0      17,688    1954      13.6
    log file switch (checkpoint           5,303   78.0 ;      4,649     877       8.0
    log file switch completion            4,245   89.2 ;      4,023     948       6.4
    wait for a undo record               32,393   99.8 ;      3,531     109      48.8
    db file parallel write               18,771     .0       3,437     183      28.3
    wait for stopper event to be         24,203   99.8 ;      2,634     109      36.5
    log file sync                         2,634   88.6 ;      2,481     942       4.0
    control file sequential read          7,356     .0       2,431     330      11.1
    buffer busy waits                     2,513   83.1 ;      2,173     865       3.8
    log file parallel write                 520     .0       1,566    3012       0.8
    control file parallel write             840     .0       1,334    1588       1.3
    rdbms ipc reply                         172   91.3 ;        330    1916       0.3
    enq: CF - contention                    309   23.0 ;        268     867       0.5
    log buffer space                        638   28.5 ;        192     301       1.0
    enq: PS - contention                     52   23.1 ;         71    1362       0.1
    db file scattered read                  113     .0          67     590       0.2
    os thread startup                        76   77.6 ;         63     834       0.1
    reliable message                         57   78.9 ;         50     878       0.1
    enq: RO - fast object reuse              22   22.7 ;         23    1038       0.0
    latch free                              537     .0          16      30       0.8
    Streams AQ: qmn coordinator               3  100.0 ;         15    5005       0.0 
    

    Overrides

    
    Background Wait Events               DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
    -> ordered by wait time desc, waits desc (idle events last) 
    
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    ---------------------------- -------------- ------ ----------- ------- ---------
    db file parallel write               18,772     .0       3,437     183      28.3
    events in waitclass Other            24,367   99.5 ;      3,010     124      36.7
    control file sequential read          6,654     .0       2,333     351      10.0
    log file parallel write                 520     .0       1,566    3012       0.8
    control file parallel write             840     .0       1,334    1588       1.3
    buffer busy waits                       899   94.2 ;        884     984       1.4
    log file switch (checkpoint             206   82.0 ;        185     898       0.3
    os thread startup                        76   77.6 ;         63     834       0.1
    log file switch completion               46   93.5 ;         45     982       0.1
    log buffer space                        158   31.0 ;         12      77       0.2
    db file sequential read                  62     .0           7     111       0.1
    db file scattered read                   20     .0           6     318       0.0
    direct path read                        660     .0           5       7       1.0
    log file sequential read                 66     .0           4      65       0.1
    log file single write                    66     .0           1      16       0.1
    enq: RO - fast object reuse               2     .0           0      38       0.0
    latch: cache buffers chains               3     .0           0       6       0.0
    direct path write                       660     .0          -5      -8       1.0
    rdbms ipc message                     9,052   87.5 ;     21,399    2364      13.6
    pmon timer                            1,318   90.4 ;      3,562    2703       2.0
    Streams AQ: qmn coordinator             633   97.6 ;      3,546    5602       1.0
    Streams AQ: waiting for time             77   61.0 ;      3,449   44795       0.1
    PX Deq: Join ACK                         21     .0           0       0       0.0 
    

    Once again exceeded

    
    Tablespace IO Stats                  DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
    -> ordered by IOs (Reads + Writes) desc 
    
    Tablespace
    ------------------------------
                     Av      Av     Av                       Av     Buffer Av Buf
             Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
    -------------- ------- ------ ------- ------------ -------- ---------- ------
    UNDOTBS1
               914       0 ######     1.0 ;   1,368,515      383      2,534  863.2
    MDMREF_INDICES
             6,918       2 ######     1.0 ;      11,086        3          0    0.0
    SYSAUX
               626       0 ######     1.1 ;       1,804        1          0    0.0
    SYSTEM
               850       0 ######     1.7 ;         296        0          0    0.0
    MDMREF_DATA
               293       0  712.3 ;    1.0 ;         274        0          0    0.0
    MDMPRJ_ODS
               198       0   72.1 ;    1.0 ;         198        0          0    0.0
    FEU_VERT
                33       0   61.5 ;    1.0 ;          33        0          0    0.0
    USERS
                33       0   31.5 ;    1.0 ;          33        0          0    0.0
              ------------------------------------------------------------- 
    

    Now have a serious look. AV Dr (ms). For a value of tablespace cannot event now enter the window that's why its display #.

    According to the recommendation of the oracle Av Rd (ms) should not be greater than 20, if his will on 20 so its considered to be a problem with the IO subsystem. But as its seen in your case its passing.

    Now the question on my side
    Did configuration changes?
    I suggest you undo these changes as soon as possible and to communicate with the storage admin guys...

    Hope this helps

Maybe you are looking for