Logic of transformation

Hello

I asked about the best way to implement the logic of transformation with OWB. In my case, the source is a database and the amount of data is only a little more than 2 million lines. Extract phase I did a direct load from the source to the table in ITS phase where the table is structured the same as the source. Type of loading is TRUNCATE/INSERT. It takes two minutes to load which I think is pretty well in our environment. I've been making mappings that takes the data from the table - its, makes all the transformations and save a direct result of the dw-table (UPDATE/INSERT). All in a map. But when there is a lot of transformation in several expressions with box-when a lot of joins clauses and the research, the result is very inefficient - I think - can not be solved by tuning the plan of the explain command. I also include the surrogate_id in the Web of mapping (we do not use object dimension or cube objects, a few tables). There are situations where the data used in the expression are handled several times in the other expression which - I think - a lot of handling nested in the package. Time to update the dw table can be found on several hours... I think my mistake is trying to put too many things in the unique mapping when the order of operators can be found on the inefficient code.

I was told that the best way to manage data is to extract the data from the source to the table - its is construced as dw-table. I know people who did the tampering in separate SQL clauses in a PL/SQL procedure. Do a lot of updates for SA table. The only thing to do with OWB is to transfer the data from the table'S ready at the DW table. I don't bite. I tend to think that the transformation is always possible more effective with OWB (if the mapping is designed right way) because overall according to the mode that it generates only one article SQL to do any manipulation in the unique map. Is it? I wonder three different things:

(1) when to split the map to separate mappings.
(2) can I do maps update the original table with loading type updated or are still transferred directly mappings that stores the results in the new table?
(3) the use of tables in ITS region. When I should make transformation multimode (do more sa-tables to store the incremental result of each step)?

I think that it is difficult to clarify me on this. I hope you get my point and can give me some good guidelines to follow.

Kind regards

Juha

Hi Juha,

It is not the order or the use of simple operators that make the mapping of poor performance but operations temporarily stores the data on the disc are those should I avoid - or try to minimize their effect.

Temporary tablespace using Oracle DWH (and perhaps on other RDBMS platforms) is general. I think that it is very difficult to avoid completely the temporary segments creting in environment DWH.

Or maybe I should concentrate on the path I have chosen and separate big joins on their own mappings and reduce the amount of joints in a mapping?

In my view, splitting complex mapping in several mappings is the best approach in your case.

Kind regards
Oleg

Tags: Business Intelligence

Similar Questions

  • Changing the content ID method

    Hi all

    What about the content ID of the content whether active documents/web or etc.

    During export/import content, is there a way to change the content ID to something else batch and no severance of the relationship with the Site that use web/content or other assets?
    for example. Initial content for content ID is ABC_XXXX01 XYZ_XXXX01where ABC and XYZ view the prefix and the XXXX01 is to refer to the sequence number

    Or what is the common/practice method to change the content for an entire site id?

    Thank you

    Or what is the common/practice method to change the content for an entire site id?

    In general, archive them is option one here - you should be able to define export/import content items and to define the logic of transformation as part of it.

    I don't know everything, if it maintains the link also to WCM objects. For example,.
    -you have a static list, referring to a piece of content, which now has a new ID, or
    -you have a region assigned to a content element, now with a new code.

    It should be relatively easy to identify.

    UCM 11 g since, I think, PS3, also contains an export/import feature a Web site. So, you could also try this one.

  • avoiding the multiple connection so that others

    If I nested call of procs such as JAVA p1 calling calling calling p3 p2 calling p4 and each of them have a call to a stored procedure to the error in the log and re-raise so that the java layer is notified of the exception, then the error is recorded several times in each nested proc as the exception propagates upward to the java layer and is trapped by WHEN E.A. and revived each proc. Even if I take Tom Kytes advice and limit the OTHERS WHEN in p1 alone though the outermost proc I would still need to log the exception exactly where it happens because I have all the context information tell in p3. But WHEN OTHERS in p1 would always double check the exception.

    Any successful implementation of suggestions / real to this problem?

    user4900730 wrote:

    That's exactly my question. I want to immediately connect the error say p3. But how should I inform p2 and p1 it has already been saved.

    You can use a global variable (defined in a packet header and usable by any code of PL in this session) for example is a Boolean.

    If TRUE then the LogMessage() procedure called from an exception handler logs an exception. If FALSE, it is not.

    You can now set to true by default. The first time that LogMessage() is called in a session, she saves this exception, and then set the variable false - preventing any future exception being saved.

    So what you want to do, is not feasible. I wonder, however, whether this approach is sensitive for the reasons that I said previously.

    The previous answer by BILLY VERREYNE indicates that multiple logging of the error should not be considered a problem. Maybe I'll settle for that.

    You can always report on an exception. Let's say that your LogMessage() procedure registers the exception with a timestamp of system (using a stand-alone insert) and with a single session or the process identifier, a journal table.

    P3 throws an exception. It connects there. P2 catches, saves her and it triggers again. P3 grabs and it opens a session.

    You have now 3 exceptions to the process. Each with a different timestamp. Your report for people dealing with exceptions can select the first exception (using the time stamp) that took place for this process.

    At the same time, you have a pile of complete call procedures that handled the exception which may shed more light on who called p3 when the 3 p exception occurred.

    Consistencyis a concept important in software engineering *. Consistency in design. Consistency in the code.

    If the p1 exception handler must act differently from p2 and 3 p (because of the requirement of logging) exception handlers, you now have an inconsistency in the design and code. This makes the design more complex and more complex code. Increase of complexity makes the code more difficult to read and to understand and maintain. Increases the likelihood of bugs.

    This is the fundamental reason why I don't like the approach you plan to record an exception only once to avoid a kind of duplication. There is inconsistent behavior of the code and exception handlers.

    Rather deal with this (no exception duplicate) requirement to report on this matter - as it is much more logical to apply the filter and reports logic to transform the data on exceptions in useful information for users.

  • Generated code differs from OWB Client and Deployment Manager...

    Hello

    I have developed mappings using OWB9.2 and the generated code.

    But even when mapping is used to code generated by the Deployment Manager is to have an additional code and the procedures/functions at WB_RT %.

    Why this happens, no idea on that body.

    Please let me know.

    With respect,
    Murielle...

    Hi Ricardo,

    the code generated in the mapping Editor shows you how your mapping logic is transformed into sql and pl/sql.
    When the mapping is deployed some audit is added and some identifiers that reference objects in the runtime environment are defined.
    Which is why you should not deploy the code generated manually, always deploy via the control center.

    Kind regards
    Carsten.

  • Recon creates several resources

    Hi all

    Have an interesting situation.

    We have 1 object of resource, but we have 2 different sources acknowledged. When we run the first recon for this resource, it fits well.
    When we run the second recon for the same resource, it creates duplicate resource objects.

    the only difference in the Recon is a flat file and a DB. the resource has about 7 required fields with only 1. the db recon uses only the necessary
    field. I'm not sure if this is what the origin of this problem or not.

    any help would be appreciated.

    Thanx
    Fred

    Hi Fred

    Nice to see again you.

    As far as I know, there is no way 9as) to the IOM to ignore case. What you can do is to ensure that all channels are in the same case.

    In the GTC connectors, there is a transformer case and in some of the "no TAG" connectors there is an option to add the logic of transformer or specific options allowing you to business transformation.

    If the connector does not support the transformation, you need to write a custom connector that supports the changes.

    Best regards
    / Martin

  • Logic blocks when transformations of midi

    Hello

    I use Logic Pro X and whenever I try to use the any option processing midi asked more than one note logic crashes. What is new, he suddenly began to occur, but it gives my a really hard time in my work.

    I deleted the logic pro x application in the applications folder and downloaded again on the app store with none of the positive changes.

    Could someone help me please know how to fix this, or tell me if I have to completely remove the software to re - install again from scratch?

    Thanks in advance for any information.

    Daniel

    No, resettlement will not help. It rarely does.

    I'm afraid you pouvez have of the stumbed across a bug. Can you describe exactly what you do when the logic crashes? I know that, in a window of transformer, alters numbers by dragging can occasionally crash logic and the simple solution is to not move, but just type the numbers (or the question is the reverse, not sure). So he my being that you just need to avoid a specific command or a set of conditions.

  • Transform midi logical crash 10.2.1

    Hello!

    In the logic 10.2.1 Midi transform window crash the app in the Yosemite and El Capitan.

    I tried on 2 computers, MacBook Pro (with El Capitan) and Macmini (with Yosemite)

    Is a bug?

    My transformation window has disappeared from the 'window' - tab. Can be accessed only via the "Piano Roll" - window: functions > MIDI transform.

    There are several errors of MIDI with this update. My M-Audio oxygen 49 has stopped working after update. Works perfectly in MainStage3 and GarageBand and Logic Pro X receives no signals, with one exception: when the message "Changed number of input/output MIDI. 1 input and 1 output there are"watch after you connect the USB, if I push the logical message will read what I play. But the second I press "OK" on this message, the connection is closed.

    Hope someone can help us. Good luck!

  • Transformation of table names and a column in the logic model (CCM)

    In addition to my previous question on the transformations of the relational model logical I would like to set for the names of tables like school_units to become SchoolUnit when you do: engineer to the logic model. Ditto for the columns for attributes, i. e. I would like to convert a column of class_sign to classSign.
    With the help of Philip Stoyanov, I could set the mixed case to emphasize the shape of logic to the relational model. But now I can't engineer to the logic model without losing all the work accomplished in the logic model (concerning the designation of entities and attributes).

    Best regards, Robert

    Hi Robert,.

    You can try to define "Incomplete modifiers" in your glossary so the name validation will not fail (against the glossary) because it is existing only don't not in the glossary of terms.

    If the name is not recognised as valid then separator is changed, but setting camel case is not considered - I logged a bug about this.

    Philippe

  • How to transform a primary partition to logical partition

    Hello

    I just bought a laptop computer HP Probook 4330 s, product number - A6D89EA #ABB, with 64-bit Windows 7 Professional preinstalled on it.

    My problem is I can not reduce the C Partition and another partition to keep my data or install another OS in a logical partition, because this system has already made four as primary partitions:

    -300 system - partition NTFS Active

    -GO 442,2 (Windows (C) - primary partition NTFS

    -18.6 GB - HP Recovery - NTFS primary partition

    -5 GB - HP-Tools - primary partition FAT32.

    I want to turn one of these partitions in logical Partition and shrink Windows-C, create a partition extended with two logical partitions in it.

    I should mention that I don't need the HP tools.

    Or, I need to contact Microsoft to allow me to download a Win 7 Professional and reinstall on this notebook using the key delivered with this laptop and do what I want the partitions?

  • logic to be transformed into a table sql pl statement and select fill.

    create table emp (eno number, varchar2 (20) ename, sal number (10.2), dno number);

    create table emp_commision (eno number number dno, dname varchar2 (20), number (5.2)) of the commission.

    DECLARE
    TYPE EmpTabTyp1 IS TABLE OF THE emp % ROWTYPE directory INDEX;
    TYPE IS of EmpTabTyp2 TABLE emp_commision % ROWTYPE directory INDEX;
    emp_tab1 EmpTabTyp1;
    emp_tab2 EmpTabTyp2;

    C1 (p_dno number) cursor select eno, dno, dname, commission of emp_commision where dno = p_dno;
    number of l_num_index1;

    Start
    Select eno, ename, sal bulk collect into emp_tab1 from emp;
    I'm in emp_tab1.first... emp_tab1. Last
    loop
    If (emp_tab1 (i) .sal > 10000) then
    for l_rec_commision in c1 (emp_tab1.dno)
    loop

    l_num_index1: = (l_num_index1 + 1);
    emp_tab2 (l_num_index1) .commission: = l_rec_commision.commision;

    end loop;

    end if;
    end loop;
    while others then
    dbms_output.put_line (SQLERRM);
    end;
    /


    Requirement:

    For the same logic, I want to fill the table with the commission plsql emp_tab2, without using a cursor
    and two loops (loop) and the cursor for loop for. Sal > 10000 expected
    also be implemented.

    Others can be done in a single query.

    Thank you
    Vinodh

    ora_1978 wrote:
    Requirement:

    For the same logic, I want to fill the table with the commission plsql emp_tab2, without using a cursor
    and two loops (loop) and the cursor for loop for. Sal > 10000 expected
    also be implemented.

    Others can be done in a single query.

    Thank you
    Vinodh

    In addition there are errors in your code and it won't work, it is unclear what exactly you want as you have not explained with sample data and expected results.

    You select the number of columns to collect your in bulk.
    You collect the commission for an entire Department without regard to the employee for each employee record, so you would end up duplicating your data in your pl/sql collection (don't call them paintings, they are not really tables).
    You have not initialized l_num_index1 so it will never be incremented
    You have added an unnecessary error handler.
    etc.

    I'm guessing that you are looking for something in this direction...

    declare
      type EmpTabType2 is table of emp_commision%ROWTYPE index by binary_integer;
      emp_tab2 EmpTabType2;
    begin
      select ec.*
      bulk collect into emp_tab2
      from   emp e
             join emp_commision ec on (e.eno = ec.eno and e.dno = ec.dno)
      where  e.sal > 10000;
    end;
    

    ... but we can't really help without more information.

  • the custom transformation does not run against logic model

    Here is the screen [http://img849.imageshack.us/img849/7411/20110530174314002.jpg | http://img849.imageshack.us/img849/7411/20110530174314002.jpg]

    In addition to the definition of the logical = object that the script against the relational model. To prove this, have a script to collect all the table names, and have different names in the two models.

    There are scripts works on the logic model - ability to auto create a primary key column when creating an entity?

    Philippe

  • With the selection of a value logic!

    Hello

    I am faced with a simple logic problem!

    I have to choose a letter A, B, C, D, E, F & G.

    What I want is a display on the front panel (figure logic)... and when I click on any button... it sends the letter corresponding to the transformation. The condition is that I don't want only one letter to be send at once... (as in figure logic) and do not like in the figure logic2 or logic3!

    Any idea?

    Thank you
    Ford

    Hello Mike...

    A radio button is a perfect option

    but... I want to choose only one option at a time... for example, if A is WE and click B... then should be ON OFF and B... even if the user clicks on the F it takes IT and left it OUT.

    Ford

  • logical AND with the structure of the event

    Hello world

    The structure of the event can manage several events at once to do the same thing: looks like an OR logical operator. But I have not found a way to sequence events to approach a logic and example: you must click on a button to draw, then enter the image would change the mouse cursor. With a structure of the event, I can handle these 2 events separately or together (i.e. change the cursor), but I can't do a sequence of events.

    Is this possible to do with a structure of the event?

    I hope that I am clear (sorry for English btw). And thanks in advance.

    Christophe

    I don't think this is possible directly. You will need to add status information to your event loop that could enforce the order of events and the rules of logic you want. Where transformation you would have to check whether the required event has occurred before this event. If that were the case, perform your treatment. If this isn't the case, ignore the event. Your first event would need set this status information. You can also include some kind of time-out for the second event were to occur within a specific period.

    This type of logic may be better treated with the help of producer/consumer architecture and a state machine in the task of the consumer.

  • Entity and attribute transformation of names (case-sensitive) to the relational model

    Hello

    For the names of entities and attributes, I would use camel casing, e. g. AccessRightGroup (or the right access group) and its translation to the relational model in the Table name access_right_groups. I found, I can do in the entity properties > General > abbreviation preferred by writing the name of the table you want in this area. But I guess that there is a more elegant way by setting that breaks are transformed into small capital and underlined and entity (set) becomes plural as table. Is there such a possibility?

    How that would make the camel case attributes?

    Best regards, Robert

    Hi Robert,.

    case-sensitive, you must set 'Title case' as a separator for logic model in the design properties dialog box "settings > naming standards '-c' is for this form of the names AccessRightGroup; for group of access rights that you need not change anything - the default separator is set to 'space '.

    For the plural in the relational model, you need a glossary with defined plurals - however it only works for the entity name unique term. I logged an ER to change this.

    Tiny in a relational model - well you must set to relational model - in the Properties dialog box - "(Naming options > Table, cas de caractères...-mis à baisser et le nom sera mis dans cette affaire àle cours de la transformation) - otherwise you always can use the script provided with Data Modeler to convert the names of all the lower/upper cases."

    Phhilip

  • converting string to XML using Xquery transformation in OSB node

    How to convert the string to XML node elementusing built in function by using Xquery transformation in OSB?

    Discover this - http://www.javamonamour.org/2011/06/fn-beainlinedxml.html

    If in SOA (BPEL & mediator), you can use oraext:parseXML.

    You must analyze in depth where to implement your requirement as some good practices advise to implement logic more complex in SOA and let OSB only connect to endpoints of services.

    Hope this helps,

    A.

Maybe you are looking for

  • display zip files... what should I do now to see the content of my zip file?

    Sir ihave downloaded songs on the internet that is downloaded as a zip file format... and I have change the path of the zip opening and is open with internet explore form.and, now when I open my file zip my scenn begin to blink heavlyy after that I h

  • WALLPAPER WILL NOT CHANGE...

    So I'm going to "Choose a wallpaper" and it says"click a pictue to make my wallpaper" Good, and then he said: "to use a picture that is not listed, go to the image and double-click it." Good, and then under that there is a 'Place' boxwith a menu drop

  • Author and not taken into account when packing using Blackberry APK Packer-the author personalized ID

    Hello In order to update my existing app, I use Blackberry APK package to create the update bar my updated APK file. In the package manager, I put "Author" and "Author ID" in the advanced settings, but the bar generated file does not take their accou

  • Compatibility Windows 7 and AOL

    Are there known problems of compatibility of Windows 7 with AOL?

  • How to close a PopupScreen

    Hi, I'm a newbie here. I have 1 button on my PopupScreen.I want to click it then to do sth. To maintain the display, I use pushGlobalScreen as below: UiApplication.getUiApplication().pushGlobalScreen(new pScreen(), 1,true); then: final class pScreen