Challenges using dynamic SQL for data quality tests

(published for the wrong forum)

Background:

I'm developing a data quality, mechanism, on Oracle 11 g, which uses five blocks of PL/SQL by test test script:
< li > two to supervise the event test by inserting a record in this regard in a fact table and update after the test with the result of the counts;
table < li > two to create an image "should be" of the tested data set and store it in a scene.
< li > and the remaining block to perform the analysis by selecting a union of the staging table data and the data "such what" of the target table, abandoning all matching files so that only incompatibilities (errors) remain and store this information in a results table, labeled with either recording source "should be" ("should be") or target ("such what").

I intend to do this pilot by the data, so that each block is stored as a value in an array of scripts, associated with one or more test scripts through the record key values and pulled in a SQL statement execution by a stored procedure in a package called by Informatica Oracle dynamic. In this way, a variety of data quality tests can be programmed as automated controls that work every night and trigger alerts if problems are detected.

I have two challenges:
< li > PL/SQL blocks that create the DataSet "should be" can be very long, and I learned through Discussion Forums OTN size maximum of 32767 K for a string of PL/SQL variable cannot be large enough to fit the entire block, and the EXECUTE IMMEDIATE statement accepts not the CLOB.
< li > if there is no anomaly, the trick of COUNT () ON the PARTITION that I use to get the counts of subquery won't work, because the analysis block everything is an INSERT INTO < i > < /i > table (column1, Column2, etc.) SELECT (< i > < /i > comparison query) statement, to avoid that my duty to hit the table target several times. My approach to this is driven by performance problems.

First question: can I I EXECUTE IMMEDIATE nest, so that the SQL block running offers another EXECUTE IMMEDIATE statement that calls another block of SQL? This would solve my problem first.
Second question: what is the way most effective to get the record account of subqueries in an INSERT INTO (SELECT) statement? I feel I'm too close the code to see the obvious answer here.

Here is a shell of the block analysis ((deuxieme question):
DECLARE

StartDate DATE;
EndDate DATE;
TEST_KEY NUMBER;

BEGIN

INSERT INTO TEST_EVENT_DETAIL 
    (TEST_EVENT_KEY,TEST_EVENT_DETAIL_KEY,COLUMN1,COLUMN2,COLUMN3,COLUMN4,
    COLUMN5)
SELECT
    TEST_KEY as TEST_EVENT_KEY,
    TEST_EVENT_DETAIL_SEQ.NEXTVAL AS TEST_EVENT_DETAIL_KEY,
    RESULTS.TABLENAME as COLUMN1,
    RESULTS.COLUMNNAME1 as COLUMN2,
    RESULTS.COLUMNNAME2 as COLUMN3,
    RESULTS.subqry_count as COLUMN4,
    null as COLUMN5  -- there are more generic columns, this is just an example; columns not used by a particular test are filled with nulls
FROM
(SELECT MIN(TABLENAME) as TABLENAME,
    min(subqry_count) as subqry_count, 
    COLUMNNAME1,
    COLUMNNAME2
  FROM 
(SELECT TABLENAME as TABLENAME,
    count(TABLENAME) over (partition by TABLENAME) subqry_count,
    COLUMNNAME1,
    COLUMNNAME2
  from
(
/** Source Data **/
 SELECT 'SOURCE' as TABLENAME,
    COLUMNNAME1,
    COLUMNNAME2
   FROM TEST_EVENT_STAGE A
 WHERE A.TEST_EVENT_KEY=TEST_KEY
  UNION ALL
/** Target Data **/
  SELECT 'TARGET' as TABLENAME,
    COLUMNNAME1,
    COLUMNNAME2
  FROM TABLENAME B
  WHERE ____________
) TMP
)
GROUP BY COLUMNNAME1, COLUMNNAME2
HAVING COUNT(*) = 1 -- this drops out all records that don't match
ORDER BY COLUMNNAME1, COLUMNNAME2, TABLENAME ASC
) RESULTS;

END;
and here's some pseudocode for the stored procedure that would call the PL/SQL (first question) blocks:
Declare

TestProcessStart DATE;
TestKey     NUMBER;
StartDate      DATE;
EndDate      DATE;
BlockStatus        varchar2(200);
BlockSQL        varchar2(32767);

begin
 
Set BlockSQL=(select SCRIPT_BLOCK from DIM_SCRIPTS where SCRIPT_KEY=(select SQL_BLOCK2 from DIM_TEST where TEST_KEY=TestKey));

execute immediate 'begin ' || BlockSQL || '; end;'
   using in BlockSQL, in out BlockStatus ;

 if BlockStatus != 'OK' then
    dbms_output.put_line('error');
 end if;
end;
Any ideas/recommendations?

PL/SQL blocks that create the data set "should be" can be very long, and I learned through the Forums of Discussion OTN 32767 K maximum size for a string of PL/SQL variable cannot be large enough to fit the entire block, and the EXECUTE IMMEDIATE statement does not accept the CLOB.

Not in 11g more: EXECUTE IMMEDIATE statement:

»
dynamic_sql_stmt

Literal string, string variable or string expression representing a SQL statement. Its type must be CHAR, VARCHAR2 or CLOB.
«

Tags: Database

Similar Questions

  • How to write the SQL without using dynamic SQL?

    How can you write this under SQL without using execute immediately?

    You can use static SQL (using something like a CASE statement)?

    test procedure (one in varchar2, b number, each number) is

    v_num_recs pls_integer;

    Start

    Select count (*)

    in v_num

    FROM table1

    where

    col1 = one and

    If b is not null then col2 = b

    If c is not null then col3 = c;

    / * i.e. If b is not null, where condition to add this line only. If c is not null, where condition should add only this line. OR condition No. it * /.

    dbms_output.put_line (v_num);

    end;

    Or should I use dynamic SQL statements for this?

    I was wondering if the two are NOT NULL? Well check this.

    Select count (*) in v_num

    table

    where col1 = one

    and col2 = (CASE WHEN (b is not null) THEN b ELSE END col2)

    and col3 = (CASE WHEN (c is not null) THEN ELSE END col3 c)

  • OutOfMemoryError: Limit superior GC exceeded when loading directly the source using IKM sql for sql. Growing ODI_MAX_HEAP do not solve the problem.

    OutOfMemoryError: GC overhead limit at execution a loading interface directly sql for sql with no work table.

    I get the error message: error: exception OutOfMemoryError: higher GC limit exceeded when executing an interface making a direct using IKM SQL for SQL command load Append, source a 150millions lines table.

    I have increased the ODI_MAX_HEAP and the interface run longer and failed. I'm already at: ODI_MAX_HEAP = 12560 m I tested with ODI_MAX_HEAP = 52560 m and still error.

    I am following up to the memory of the server and I still have available memory...

    Apart from the problem of memory I know that this type of load should be possible because the step of data load on LKM SQL to Oracle is able to load the work table $ CAN. Ideally, I want to emulate this behavior by using SQL for SQL IKM.

    1 - What is the right path to follow here? (change the parameters of memory or modify the IKM?)


    2 - ideas on how to solve the OutOfMemoryError: GC overhead limit exceeded error? (GC means Garbage Collector)

    Execution of the IKM interface in the Simulator generates this code:

    Load (Source) command:

    Select

    source - tbl.col1 COL1,

    source - tbl.col2 COL2,

    source-tbl. "' COL3 ' COL3

    of public.source - tbl AS source-tbl

    where

    (1 = 1)

    Default command (Destination):

    insert into the source-tbl

    (

    col1,

    col2,

    COL3

    )

    values

    (

    : COL1,.

    : COL2.

    : COL3

    )

    My experience is very limited with ODI so I don't know about changing the code to the KMs

    Thanks in advance.

    Find a work around the error of generals limit exceeded GC:

    -in my case I was running without the IDE so that changes made to the odiparams.sh were not useful.

    -This means that I need to change the JVM settings to:

    $ODI_HOME/oracledi/client/odi/bin/odi.conf

    AddVMOption - XX: MaxPermSize = NNNNM

    $$ODI_HOME/oracledi/client/ide/bin/ide.conf

    AddVMOption - XmxNNNNM

    AddVMOption - XmsNNNNM

    Where NNNN is a higher value.

  • link: new values on after the trigger by using dynamic sql.

    Hello all I have a problem I want to solve using dynamic sql on a trigger. The problem is that are around 5 qty fields who each have 18 separate fields. This is because of different sizes (S, M, L, XL, etc., up to 18). So I will try to use the: new values on a trigger to dynamically fill these information on a table, but immediate execution is run is returning the value like: new.open_sz (x) instead of the actual numeric value of the column. See code below. Is there a way to do this.

    Thank you in advance.


    CREATE OR REPLACE TRIGGER T_EXAMPLE
    after update of
    open_sz1, open_sz2, open_sz3, open_sz4, open_sz5, open_sz6,
    open_sz7, open_sz8, open_sz9, open_sz10, open_sz11, open_sz12,
    open_sz13, open_sz14, open_sz15, open_sz16, open_sz17, open_sz18
    on order_li_m for each line
    declare
    Is of TYPE NumArray table indexes number directory;
    NumArray v_orderqty;
    -TYPE VarArray is the table of the varchar2 (200) index directory.
    -v_orderqty K_A2KSTD. VarArray;
    v_col varchar2 (30);
    v_sql varchar2 (1000);
    BEGIN
    for a 1 in... 18 loop
    v_col: = ': new.open_sz' | one;
    v_sql: = ' SELECT: 1 double ';
    run immediately v_sql in v_orderqty (a) using v_col;
    dbms_output.put_line (v_orderqty (a));
    end loop;
    end T_EXAMPLE;
    /

    Raffy Martin wrote:
    Is there a way to do this.

    You may not use: news and: old identifiers in dynamic SQL, the way to do it is to specify the real column names.

  • problem using dynamic sql

    Hi all

    I have to insert the record in a table in one database to another database which requires db_link.so to do this I use run immediately, but when I try to insert by using the procedure below it will directly itno exception block.
    Help, please



    declare
    p_study_to NUMBER: = 1;
    p_study_from NUMBER: = 3;
    P_db_link VARCHAR2 (30): = 'abc ';
    BEGIN

    RUN IMMEDIATELY ' INSERT INTO RRRRR
    (study_to, db_link, dt_created, dt_modified, modified_by)'
    ||' SELECT study_to, db_link, dt_created,
    dt_modified, modified_by
    OF UUUU@:2'
    using p_study_to, p_db_link;
    EXCEPTION
    WHILE OTHERS
    THEN
    dbms_output.put_line ('error when inserting in RRRRR');
    END;
    /


    Thanks in advance!

    Hello

    You can tell only AID like this to give values for expressions, not table names (or their qualifications, as in this case).
    Try:

    declare
         p_study_to     NUMBER := 1;
         p_study_from      NUMBER := 3;
         P_db_link      VARCHAR2(30) := 'abc';
         sql_txt          VARCHAR2 (2000);
    BEGIN
    
         sql_txt := 'INSERT INTO     RRRRR (study_to, db_link, dt_created, dt_modified, modified_by)'
              ||           ' SELECT  study_to, db_link, dt_created, dt_modified, modified_by'
              ||          ' FROM    UUUU@' || p_db_link ;
         dbms_output.put_line (sql_txt);
    
            -- EXECUTE IMMEDIATE  sql_txt;
    END;
    

    Whenever you write dynamic sql code, it displays first.
    Only when it shows something that looks right should you UN-comment the EXECUTE IMMEDIATE statement.
    Before you go into Production, comment out code (or delete) the call to put_line.

    Use an EXCEPTION handler only when you can improve on what is the mechanism of error by default, for example, you plan a particular error, and you have something specail do when this error occurs.

    Using dynamic SQL just to have the same code running on different databases, where each database needs a different connection name? If so, use synonyms instead of dynamic SQL.

  • Making the best use of space for the quality by balancing the video bitrate and audio settings?

    I have a few questions about the first export still / encoding in encore. I am trying to create a dvd single layer, where I maximize overall quality using as much space as possible.  The video is 97 minutes, SD 720 x 480, 29.97 FPS and consists of interviews for the most part, as well as b-roll hd cameras and stills, rare title graphic card.

    1. I can use dynamic link to send the sequence that appears to maximize the space available, however I have heard that using DL is generally frowned upon and I myself cannot tell what settings it uses (for example vbr, 2 pass, etc.). Is my interpretation that DL is not the ideal way to go wrong? If DL is just as good as a separate output, is there a way to know what parameters it uses?

    2. I used a bitrate calculator and tried a Min: 3.5 target: 5.5 Max: 8 audio output PCM and I found myself having a little more than 400 MB of unused space. When someone recommended a Min: 5.5 target: 7 output Max 8.5 the file size is too large with a simple DVD layer. I read that it is recommended to use an exit instead of PCM, if the video is over an hour audio Dolby. My question is is it a best practice to balance needs of flow rates and audio and video? I realize, of course, by using more data PCM is 'better', but it is usually best to go on the road to dolby and have more space to increase your video bitrate? What is the audio difference type that requires a high-end system to discern? Y audio data loss is worth for the benefit of video bitrate higher?

    I know it's a lot, just trying to maximize space to get the best possible performance. Thanks for any help/advice you can provide.

    -Nick

    As most of my videos are closer to 2 hours usually, I always used Dolby Digital, PCM takes so much space and I prefer to use this space to increase video quality.

    Note that by default, if you import a .wav (PCM) file into Encore, it transcode automatically that Dolby, unless you have changed specifically the default settings of transcoding in Encore.

    A decent basis for coding rule is 560/minute = bitrate, so 560/97 = 5.77 and I usually just round off a little for menu overhead and margin of safety. This requires using Dolby to 192 k, so I would use 5.6 average bitrate.

    Thank you

    Jeff Pulera

    Safe Harbor computers

  • Can Equium A100 - I use E drive for data backup?

    My Equium A100-27 has a 80 GB hard drive, it is split in the C drive half and drive E.

    I understand that the E drive is used for system recovery purposes, but can I also use it to store data?

    If I use it for how much room must be left free for system data recovery?

    It's old laptop model, and there is no recovery of data stored on the second partition of HARD drive.
    If you can do what you want with this partition, and also use it for backup of data or backup data.

  • Char invalid when you use dynamic sql statements with cursor

    Hello guys,.
    I'm opening a following cursor with dynamic sql
     v_sql := 'SELECT OS.table_name FROM OBJECT_STATS_CONTROL OS, ALL_TABLES AT
                    WHERE OS.analyze_flag <> ''N''
                    AND OS.last_analyze < sysdate -1 AND AT.last_analyzed < sysdate - 1 '||v_1||';'; 
    my cursor statement is
    BEGIN
           OPEN v_cur FOR v_sql; 
           LOOP
           FETCH v_cur INTO v_tabname;
    ......
    I get an error message
    0911 invalid char
    can any body suggest me where I'm wrong

    Thank you

    Remove the semicolon at the end:

    ||';'  -- remove this.
    
  • Oracle sql for data after the first occurrence of the hyphen

    Hi, I need oracle sql get data after the first occurrence of the hyphen.

    source of the string:
    ABCD - efgh

    I just want everything after the ' - '.

    someone help me please...

    Edited by: 1001076 on 28 may 2013 18:42

    Handle: 1001076
    Status level: Beginner
    Join date: April 18, 2013
    Messages total: 12
    Total Questions: 4 (3 pending)

    I offer you my condolences; Since you rarely get your questions answered here.

  • Error using IKM SQL for Hyperion Essbase (data)

    Get (last Intnermost): file ' < string >, line 75, in?
    Error 'com.hyperion.odi.common.ODIConstants' attribute class has no attribute 'PRE_LOAD_MAXL_SCRIPT '.

    I'm not usiing a pre load a MaxL Script. I even put the 'ABORT_ON_PRE_MAXL_ERROR' to no.

    Any help would be appreciated.

    Hello

    You may be using the old java (jar) files in your directory of driver, in the most recent class ODIConstants has been updated to include "PRE_LOAD_MAXL_SCRIPT".

    I would try to install a patch later or a later version of ODI.

    See you soon

    John
    http://John-Goodwin.blogspot.com/

  • Need help with SQL/PL/SQL for Dates

    Hi Experts - need help with a SQL query.

    I need to insert some date fields in a table called CALENDAR_PERIOD.

    Current data in CALENDAR_PERIOD table with their data types:

    STARTPERIOD (DATE) YEAR (NUMBER) PERIOD_LABEL (Varchar2 255)

    02/11/2014 2014 2014/02/11 SUN

    03/11/2014 2014 14/03/11 MON

    04/11/2014 2014 11/04/14 MAR

    I have to increment above values up to the year 2025. I don't know how to write SQL and increment of these values.

    Ex: My next value should insert: 05/11/2015 2014 11/05/14 WED like that I need to insert data until 12 31, 2025.

    Can you please help me with PL/SQL block?

    Really appreciate your help!

    DB version:

    Oracle Database 11 g Enterprise Edition Release 11.2.0.3.0 - 64 bit Production

    PL/SQL Release 11.2.0.3.0 - Production

    CORE Production 11.2.0.3.0

    AMT for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production

    NLSRTL Version 11.2.0.3.0 - Production

    Thank you
    Sandy

    Hello Sandy,

    Maybe something like

    INSERT INTO calendar_period (startperiod, year, period_label)

    SELECT DATE '' 2014-11-04 + LEVEL

    , TO_NUMBER (TO_CHAR (DATE '' 2014-11-04 + LEVEL, "YYYY"))

    , TO_CHAR (DATE '' 2014-11-04 + LEVEL, "MM/DD/YY DY")

    OF the double

    CONNECT BY LEVEL<= date="" '2025-12-314="" -="" date="">

    ;

    ((mais je ne comprends pas pourquoi nous créons une telle table "année" et "period_label" peuvent être calculé à partir de startperiod))

    Best regards

    Bruno Vroman.

  • drop and re-create the table using dynamic sql

    The following procedure should drop the table and re-create it. Currently, I get the dbms_output of the query. If I run the query only, it works well. but through the procedure, it does not work. Please help me

    PROCEdure emp_backup is
    sql_txt varchar2(10000);
    begin
         begin
         EXECUTE IMMEDIATE 'drop table emp_backup ' ;
         exception
         when others then
         null;
    
         sql_txt:= 'CREATE TABLE emp_backup as ' ||
                     'select * from emp  '||
                     'where dep_no=10 ' ;
         dbms_output.put_line(sql_txt);
         EXECUTE IMMEDIATE sql_txt;
    Exception
    when others then
    Null;
    
    End;

    Daniel wrote:
    I'm getting following error ORA-01031: insufficient privileges

    What is the command to grant privileges to this

    GRANT CREATE TABLE TO <>
    

    Detaching Frank, however, and echoing my previous comment, let them down and to re-create the tables makes no sense. It is logical to truncate a table if you test a load. If you build a data warehouse, it makes sense to delete and re-create a partition if you rerun a load for a particular date and that you have daily partitions. Deletion and recreation of a table in a stored procedure is not a reasonable way to test a system (nor is there a reasonable way to approach change control issues that will inevitably produce when you need to change the structure of a table).

    Justin

  • just curious, using pl.sql for the site

    Hello all;

    I have a small question, I made it using other tools, but I'm just trying to learn a few things again associated with pl/sql. If I have a site similar to below.

    http://www.BBC.co.UK/news/

    and I want to extract only the date of this site which is July 5, 2012 update at 14:38 HE and record that information in a table. Is this possible with pl/sql

    http://www.morganslibrary.org/reference/pkgs/utl_http.html

    If you use 11 GR 1 material or above you will also need this:
    http://www.morganslibrary.org/reference/pkgs/DBMS_NETWORK_ACL_ADMIN.html

  • PL/SQL for data Replcate

    I'm working on a project where I need to transfer records between two bases of Oracle 9i running on separate servers in response to a user who clicks on a button in a Windows form. My intial thoughts were to create a table in a database I could use to create a DML trigger that transfers data to the database. Do we know if it is an option, or should I look at Oracle Streams. Also is that someone could suggest where I might find information detailing how to create a PL/SQL procedure that transfers data between databases. Thanks in advance.

    Hello
    Welcome to the OTN Forums!

    Refer

    DB links

    * 009 *.

  • Problems with Insert by using dynamic Sql

    Hello
    I use the following procedure to read a BLOB after download tab-delimited text files and insert the data in a table called TBL_TMP. The columns of the table are the following: T1, T2, T3, T4, T5, T6, T7, T8, T9. I have different text files.

    Some files could fill the table until the column T8 (v_data_array (8)) and others could fill the columns up to T9 (v_data_array (9)). So I try to change this procedure in order to take account of the text file that will fill in column T9 (v_data_array (9)) but the only thing is fails for text files that fill will alone up to T8.

    The error was: ORA-01403: no data found. I tried using NVL (v_data_array (9), 'NULL'), but it does not work. How the procedure can be modified to accept two text files containing data up to columns T8 and T9.

    Thanks in advance.
    CREATE OR REPLACE PROCEDURE pr_file_upload
      (name_in IN varchar2)
    IS
    v_blob_data       BLOB; --will hold the binary structure
    v_blob_len        NUMBER; --The length of the BLOB in bytes
    v_position        NUMBER; --Will store the current position of the pointer in the blob
    v_raw_chunk       RAW(10000); --Incremental raw chunks of the BLOB will be read in the loop
    v_char            CHAR(1); --Stores the current character of the BLOB
    c_chunk_len       NUMBER := 1; --Stores the current length of the raw chunks in bytes.
    v_line            VARCHAR2 (32767) := NULL;  --Stores the value of the current line in the loop
    v_data_array      wwv_flow_global.vc_arr2;  --apex array size of 32,000
    v_name_in         VARCHAR2 (50);
    BEGIN
    
    
    -- Read data from APEX_APPLICATION_FILES view in APEX
    SELECT blob_content INTO v_blob_data
    FROM APEX_APPLICATION_FILES WHERE filename = name_in;
    
    
    v_blob_len := dbms_lob.getlength(v_blob_data);
    v_position := 1;
    
    -- Read and convert binary to char
    WHILE ( v_position <= v_blob_len ) LOOP
     v_raw_chunk := dbms_lob.substr(v_blob_data,c_chunk_len,v_position);
     v_char :=  chr(fn_hex_to_decimal(rawtohex(v_raw_chunk)));
     v_line := v_line || v_char;
     v_position := v_position + c_chunk_len;
    -- When a whole line is retrieved
     IF v_char = CHR(10) THEN
    -- Convert tab CHR(9) to : to use wwv_flow_utilities
       v_line := REPLACE (v_line, CHR(9), ':');
    -- Convert each column separated by : into array of data
       v_data_array := wwv_flow_utilities.string_to_table (v_line);
    -- Insert data into target table
       EXECUTE IMMEDIATE 'insert into TBL_TMP (t1, t2, t3, t4, t5, t6, t7, t8, t9)
        values (:1,:2,:3,:4,:5,:6,:7,:8,:9)'
        USING
          v_data_array(1),
          v_data_array(2),
          v_data_array(3),
          v_data_array(4),
          v_data_array(5),
          v_data_array(6),
          v_data_array(7),
          v_data_array(8),
          NVL(v_data_array(9),'NULL');   -- Need help here.  This is empty when reading text files that fills up to column T8.
    
    
    -- Clear out
       v_line := NULL;
      END IF;
     END LOOP;
    END;
    /

    NVL (v_data_array (9), 'NULL') will not work. Use:

    CASE WHEN v_data_array.EXISTS(9) THEN v_data_array(9) END
    

    SY.

Maybe you are looking for

  • Problems with iMac 24 "2008

    I am totally beside me how to fix my computer. Here's the story. A year or so, my computer has stopped starting upwards. I blamed it on my husband for the recording of all its video games on my computer and I thought it kind of caught or something wa

  • hpbii +: bad payments

    I keep getting payments hurt when I get home loan information the problem, says, a loan of $60,000 made a nominal interest of 12% for 30 years, what are the constant monthly mortgage payments? My calculator is set to 1 P/year I entered my calculator

  • X 360 HP spectrum: Windows display 1920 x 1080, but it's really 1280 x 720

    Graphics Intel HD 520, pilots the latest. Two display settings and the Intel graphics settings display 1920 x 1080, but http://www.whatismyscreenresolution.com/ watch 1280 x 720 I tried uninstalling the driver, put it back, reset between the two. I t

  • Qosmio is plugged in but not charging

    Hello Yesterday, I spilled my shake on my laptop... I managed to open it clean and everything works well EXCEPT MY battery, it is at 0% and every time I remove my AC my laptop turn off, then, what do I do is my battery died or my charger connector? A

  • Satellite L750D - external use of GPUS

    Hello I have a problem with my laptop, specifically a Satellite L750D-1DQ. When I play a game, my APU AMD A8 - 3520M (with integrated graphics Radeon HD 6620) works fine, but my external GPU (6400M HD) done anything. I tried everything and I managed