data cleansing

Similar Questions

• Scripts for data cleansing Perf

  Hello

  We currently use BB 4.4 with the new features enabled PERF. Someone at - it scripts for cleaning or truncate the Perf Data collected? 4.5 is supposed to have the function programmed in. We run the BB server on unix.

  Perf Data under the $BBHOME / bbvar / perf

  The files are in plain text with lines like this:

  Here is a not so elegant way to truncate your files in perl. You will need to set the time as a unix timestamp.

  #Purge $BBPERF
  @files = "grep - Rl. $BBPERF / *';"

  $file (@files) {} foreach
  chomp ($file);
  Print "$file\n";
  @data ='cat $file | AWK ' {if (\$4 > = $startTime) {print \$0}} ";
  If (!) () open (FILE, "> $file"))) {print "cannot open $file: $!"} \n » ; print LOGFILE "cannot open $file: $! \n » ; next ;}
  foreach $line (@data) {print FILE $line ;}
  Close the FILE;
  }

• bunch of table temporary organized vs table for organizing data

  We use the Oracle 10 g on Linux platform.
  
  We have an intermediate table where the data read from the file (containing 1 million rows per day) is loaded, enriched(insert/update/delete) and then finally load in the table of another. I want to know if this table is a global temporary table or normal heap organized to reduce the generation of undo/redo.
  
  I'm not in favor of the temporary table because:
  1. any additional pressure on the temporary tablespace can cause ORA-01652: unable to extend temp segment temporary problems
  2. they are mainly intended to manipulate specific session data.
  3 - statistics do not exist for these work Oracle tables. To do this, we will have to do a dynamic sampling in the query.
  
  The problem with the table organized in piles is that they generate more undo/redo as temporary tables.
  
  Please guide me.

  >
  We have an intermediate table where the data read from the file (containing 1 million rows per day) is loaded, enriched(insert/update/delete) and then finally load in the table of another. I want to know if this table is a global temporary table or normal heap organized to reduce the generation of undo/redo.

  I'm not in favor of the temporary table because:
  1. any additional pressure on the temporary tablespace can cause ORA-01652: unable to extend temp segment temporary problems
  2. they are mainly intended to manipulate specific session data.
  3 - statistics do not exist for these work Oracle tables. To do this, we will have to do a dynamic sampling in the query.

  The problem with the table organized in piles is that they generate more undo/redo as temporary tables.
  >
  Some of you concerns can easily be mitigated.

  1 - temp tablespace

  A common practice for ETL processing to create and use a temporary tablespace customized to use TWG. This prevents the TWG of impacting on the standard temp space and possibly interfere with the rest of the DB.

  See "Creating a temporary Table" in the DBA Guide. This article has sample code that illustrates this.
  http://docs.Oracle.com/CD/B28359_01/server.111/b28310/tables003.htm#i1006400
  >
  By default, rows in a temporary table is stored in the temporary tablespace default of the user who creates it. However, you can assign a temporary table to a different tablespace when creating the temporary table by using the TABLESPACE in CREATE TABLE of TEMPORARY GLOBAL clause. You can use this feature to save the space used by temporary tables. For example, if you need to perform many operations of small temporary table and the default temporary tablespace is configured for sort operations and uses so a large measure, these small operations consume a lot of unnecessary disk space. In this case, it is best to assign a tablespace temporary second with a smaller measure.
  >
  #2 (planned for the session-specific data processing) is correct, that they have the specific session data. But GTT can also be used simply to reduce the amount of REDO normal DML operations during the processing of data which are already isolated from other users and does not need to be shared.
  >
  3 - statistics do not exist for these work Oracle tables. To do this, we will have to do a dynamic sampling in the query.
  >
  Sure - but a working Oracle generally does not collect statistics whenever you do DML on your staging table. And if you do stats new TRUNCATE/LOAD operations should gather anyway once you load the new data.

  You may not have considered one of the factors are that you should design your architecture to be scalable. It is quite possible that you have only a SINGLE step and your current treatment is very simple.

  If so AND you do not need an evolutionary process, so the solution suggested by knani, maybe the best solution for you. But this solution is not very scalable.

  Complext ETL implementations have several steps. And the data still does not move between these steps in one nice, easy step. It is a common requirement that after each stage of data processing can must be discussed or reported on make sure you that he satisfied all the requirements of the company. Then data problems (e.g. lack of data, incorrect, etc.) must be resolved before the data can be processed to the next step. Depending on the severity of the problems that step may need to be rerun.

  If only the TWG is used there is no way to "review" the data. And if ONLY the external tables are used, then you can not tables of several process efficient in parallel and asynchronously.

  Complex implementations that I worked on usually consisted of a range of external and normal tables and TWG.

  External tables and one step simple "data cleansing" are used to load data into tables normal as soon as possible. That allows multiple processes to run asynchronously and in parallel for the detection of problems of data 'stored' as much as possible. Any table can be reloaded/processed without affecting other processes.

  This suggests that your first step should knapen processes in place do the same "serial" cleaning possible.

  The second stage of ETL, when necessary, can perform cleanup of more complex data for data in a table or several tables. TWG can be effectively used here to store the intermediate results that may have large amounts of 'temporary' DML performed on the data. At the end of this stage the TWG data would be transferred to another table of result or permanent staging.

  Start with a simple one-step process (maybe Karthicks). The key is to avoid complicating the process in a way that makes it impassable.

• Vista Business operating system: my E internal drive is almost FULL

  Drive E is almost complete: I can't drive data cleansing E that appears as almost whole, 2.83 MB of free space on GB 54.5. 10.6 GB free of 45 GB C drive after you remove programs and cleaning a. I did a defrag and still get a blue pie chart for drive E, HELP please.

  Hi Hall K,.

  I think I discovered where part of the space has been exhausted. It's in the file MYNAME-PC E and there are more than 100 x 100MB zip weekly backup files to the when I got my laptop at the end of 2008. I am afraid to delete them in case something has a value, but the last March 21 backup did not complete since E is complete. I can send pictures of the screen if an e-mail address is available. I intend to buy an external drive of 320 GB and disabled in the interval of automatic backup of files in the backup and Restore Center. I am getting a space low disk WARNING: drive E that I have to ignore, but on getting the new external drive, I have to transfer these 100 + zip on the external hard drive and then remove drive E and turn on the automatic backup of files. In the future, should I auto file backup or do I just save say monthly manually.

  I look forward to your responses,

  Best regards, Robert

• Canvas vs campaign program Builder - what is the difference?

  I'm fairly new to Eloqua, but I come from a background of automation with solutions like Pardot & Marketo.  No matter which can either answer or direct me in the right direction to understand the fundamental difference between the creation of a campaign against a program?  I have been immersed in Eloqua for the last month and I'm still struggling with this issue.  They seem to offer the same functions and I wonder if campaigns to expand programs or vice versa.  All articles and discussions I've met in my initial research seems to suggest that they can be interchangeable at times so maybe someone knows of a hidden gem article that it responds better than the rest, I lived.  Advice or pointers you can give a new user is greatly appreciated.

  Program Builder vs campaign canvas has strong information.

  About canvas campaign serves what it either externally (e.g. E-mail blasting, campaigns to feed) and the program Builder is more for the internal work (for example, the addition of data, data cleansing, lead scoring, CRM integration). E9 didn't program generator. The canvas of the campaign resulted in a friendly way more to build automated campaigns.

• End of the year catching up... what will you do?

  As we approach the end of the year, many of us (especially those in B2B) will see the last two weeks and the first week (maybe) to slow down a little. With ALL that extra time on your hands (not) what you pick up that you have not had time to do earlier in the year? Metrics? Data cleansing? Deduping files? Drive speed test?

  For me this year it will be all about the lead flow. We have many different processes and groups, and this year I will take the time to look at where everything is happening, how long it takes to get where it needs to go, and where the gaps are.

  What will you do

  We will be transitioning our lead scoring model to the new interface of E10, which will involve a lot of tests and re-evaluation.  We must determine a way to translate the A1, B1, C1, etc. in gross to mesh with some marking campaign that we have in Salesforce.

  I also need to archive some contacts bounced and unsubscribed off Eloqua... because of some new lead gen efforts, we hit our limit of contact. Yay!

• Connectors of clouds are generally extremely slow?

  Greetings... I loaded 86 k contacts on a program last night and only 10 k left the stage with cloud connector.  Cloud connectors usually take this long to deal with?

  Cloud connectors are set to run automatically.  I saw 7 k move in and out of these two steps, 75 k still waiting to step 1 and 3.5 k in step 2.  None are waiting for action (at the time) when I check the connector of cloud.

  I use 2 cloud connectors, connector cloud of eloqua string manipulation.  The second uses the name of eloqua validation cloud connector.

  Is this normal?

  I think the delay is with the connector of string manipulation.  I used the name validation connector and although slow standards of native database, all 95 + records in our instance that is passed through the connector in less than a day.  I had several problems with the strong manipulation connector (connector string concatenation).  I called connectors more 10 times in our data cleansing program, and it took 5 days for all records of 95K to go through the program.

  I think it's the price we pay for the use of free connectors.  I've seen the connectors 3 rd - party that we have in our records to process more than 20K integration in next to no time at all.  Our provider of integration for the connectors is 4Thought Marketing and I think that they just released some data cleaning cloud connectors.

  -Dan

• Remove unsubscribed Contacts of Eloqua to reduce the cost of the subscription

  I asked the customer a question related to my post here on Topliners, on how long is the user cached data stored in Eloqua and what is their status holders remebered?

  I'm still waiting response for cached data; However, it was confirmed to me today by the customer support that unsubscribed status is preserved forever. So, even if you remove all your opt-out of the system, remove contact can be added; However, Eloqua will always have this user marked as not subscribed at the very least. The rest of the cached data is still up in the air (until I can probably answer in the coming days).

  This brings up an interesting question, especially for those who want to renew their subscription Eloqua and exceeded their allocation size of Contact database and do some data cleansing. Ripe fruit would be to remove hard-bouncebacks, employees of your company, etc. (However, there are reasons why you do not want to delete all contacts that is unique to each business rule, as to buy a rejection list of non-net-new prospects, prospects rating rented which cost per each send email to avoid the mix of organic conductors and also in case you intend to suppliers accept or use another tool to make outbound e-mail etc.); However, if these rules do not apply to you, it seems that we can clean up a massive amount of contact records and reduce your allocation of subscription database, which reduces the costs, without negative impact.

  Customer support wrote to me today: "When a contact is deleted, Eloqua maintains the subscription status and bounceback of that contact at all times" and for those of you asking on how long to cache data is stored oudu record to delete (mentioned as a subject of suplimental thereon), I got this response: "now about the question regarding the repopulation of the contact field data when you re-created your contact system. read your eloqua cookie and used data associated with your existing profile yet to fill again the contact data - FYI related to a contact record profile data lasts about four months in the system (even after that contact has been deleted). »

• Find the numbers in the Varchar field

  Hello

  I have a Varchar2 column for data cleansing, that I wish to find here is this column:

  1. pure numbers i.e.

  122

  33445

  2. number of places that is

  222 3333

  3 11

  3. data with any alphabet (not in any of A to Z or z) IE

  3334 11-22

  33 & & 44

  11/99

  Can someone give me please above notebooks?

  Thank you
  Aw j and d

  To retrieve only numbers and numbers with spaces, the conditions could be simple. But for the third, you mentioned as not in a - z or A - Z. This will include only numbers also. So I used characters according to what you mentioned (-, &, /). If you have more models that also include.

  SELECT CASE WHEN (REGEXP_LIKE(C1,'^[0-9]+$')) THEN 'PURE NUMBERS.

  WHEN (REGEXP_LIKE (C1,'^ [0-9] + + [0-9] * $')) THEN 'NUMBERS WITH SPACES'

  WHEN (REGEXP_LIKE(C1,'-|&|/')) THEN 'data with any alphabet.

  VALIDATIONS END AS FROM T1;

  Here's how it goes.

  SET DEFINE OFF

  () AS T1 (C1)

  SELECT '122' DOUBLE UNION ALL

  SELECT '33445' DOUBLE UNION ALL

  SELECT 222 3333' DOUBLE UNION ALL

  SELECT 3 11' FROM DUAL UNION ALL

  SELECT 11-3334-22' FROM DUAL UNION ALL

  SELECT 33 & 44' FROM DUAL UNION ALL

  SELECT ' 11/99 OF DOUBLE)

  SELECT C1, CASE WHEN (REGEXP_LIKE(C1,'^[0-9]+$')) THEN 'PURE NUMBERS.

  WHEN (REGEXP_LIKE (C1,'^ [0-9] + + [0-9] * $')) THEN 'NUMBERS WITH SPACES'

  WHEN (REGEXP_LIKE(C1,'-|&|/')) THEN 'data with any alphabet.

  VALIDATIONS END AS FROM T1;

  OUTPUT:

  NUMBERS OF PURE 122

  NUMBERS OF PURE 33445

  222 3333 NUMBERS WITH SPACES

  3 11 NUMBERS WITH SPACES

  11-3334-22 data with no alphabet

  33 & 44 data with no alphabet

  Data from 11/99 with no alphabet

• Database cleanup

  Hello, I am a DBA Jr. and come to be asked to rub the database that has been inserted between a certain amount of time. I need to delete the data in each table of the schema that was inserted during a given period of time. How can I go about it?
  
  Thanks in advance.

  Wrote:
  Hello, I am a DBA Jr. and come to be asked to rub the database that has been inserted between a certain amount of time. I need to delete the data in each table of the schema that was inserted during a given period of time. How can I go about it?

  Thanks in advance.

  before starting to work on Data Scrubbing / Data Cleansing do of course you take backup of your database, just add to what other guru provide you, you can check this link which offer solutions according to your needs:
  http://www.bardess.com/pages/solutions/data-management-data-cleansing-improvement.html

• SQL to locate certain documents

  Oracle 11R2
  
  Data cleansing. We want to find records in the EQUIPIMENT_BILLS where the STATUS is 'INFO', 'DISABLED', 'NONE '.
  The overflow if another answer (s) exists with the same BILL_NUMBER, its STATUS is also 'INFO', 'DISABLED', 'NONE '.
  
  
  BILL_NUMBER is not unique in the EQUIPIMENT_BILLS table. However most of the time there is only a single record.
  
  With the help of the following query to find the records we can cleanup.
  
  

  SELECT 
     EB.BILL_NUMBER
   FROM
     EQUIPIMENT_BILLS EB 
   LEFT JOIN  
     EQUIPIMENT E
   ON
     EB.EQUIPIMENT = E.EQUIPIMENT
   GROUP BY 
     EB.BILL_NUMBER 
  HAVING SUM(CASE WHEN NVL(E.STATUS, 'NONE') IN ('INFO', 'INVALID', 'NONE')  THEN 0 ELSE 1 END) = 0;

  My question is is there a better way to do it?

  Hello

  Here's another way:

  SELECT  bill_number
  FROM     equipment_bills
  WHERE     equipment  NOT IN (
                      SELECT     equipment
                      FROM     equipment
                      WHERE     status     NOT IN ( 'INFO'
                                       , 'INVALID'
                                       , 'NONE'
                                       )
                   )
  ;
  

  Depending on what mean you by "better", it could be better.
  I think it's a little easier to uderstand, but I'm not one who has to maintain the code.
  Try it on your system to see if there is a significant difference in performance.

• How to get documents and data to the cloud on my Mac desktop?

  After the upgrade to Sierra all my data (documents and desktop) have been transferred to Icloud. I absolutely not want all my data on another server costs me more money. How do I lose all my data safely back on my Mac, without anything? I'm quite t... o... that the decisions of this importance are make with a single click (or without?). Preferably, I do not want to export every file (of the thousands...) by hand, but just change things back to where it was before.

  Thanks in advance!

  Documents are also in your folder on the drive on your Mac iCloud.

  Just disable iCloud drive in iCloud preferences.

  Then go icloud.com and connect, then you remove the files on disc to iCloud.

  8. how to disable iCloud drive?

  To disable iCloud Drive on your iOS device, go to settings > iCloud > iCloud driveand on your Mac, go to the Apple menu > System Preferences > iCloud. If you disable iCloud drive, so you you will manage your files locally, documents are not synchronized or updated your iOS iOS devices 8 or OS X Yosemite or iCloud.com.

  http://www.iphonehacks.com/2015/01/icloud-drive-Guide.html

• Photos shared on iOS devices use data?

  I have created a number of shared albums in the Photos on my Mac and iOS devices.

  When other people visit then shared albums, what data are used to access?

  I invited a person to see a shared album which had mobile data off on their iOS device and no wifi. However, they were able to access a shared album which surprised me because I assume they would have need of mobile data or wifi.

  Shared albums are stored locally on the iOS devices. Photos downloads smaller and optimized versions pictures of an iPhone. iCloud, sharing photos - Apple Support

  Once shared, pictures taken with your standard devices, iOS devices or cameras have up to 2048 pixels on the long edge. Panoramic photos can be up to 5400 pixels wide.

• Create, save data backup

  Hello

  This is probably a fairly common question, but I looked around and either did not answer or did not understand them. I honestly don't know who. So, without further ADO, here is my question.

  My iPhone becomes quite comprehensive, so I was thinking about the removal of some of the games I have on it to make room for the new. I have a few save data on the apps that I would still be there when the application is downloaded again later. Is there a way to do this?

  Thanks in advance.

  App data is stored in the backups of iOS - iTunes or iCloud.

  The backup of your iPhone, iPad and iPod touch - Apple Support

• Remove data from the HD Server?

  I've recently updated my OS to 10.12 Sierra and went to install Server.app. However, when trying to install Mac OS Server, I received the following message:

  "This the server version does not support upgrading the server from data on this volume. To upgrade your database server, you need to install an older version of the server and OS X."

  ***? In what universe would be an acceptable solution? Now, the fact is that I probably had installed Server.app before on a very old version of Mac OS (not sure which) and just never bothered to upgrade during the last major revision of OS X or more.

  What that is, how can I get rid of these old "data server" so I can install a new copy? Thank you.

  Wondering how to remove the data from the server that you had before the upgrade to Sierra? If so, delete the folder/Library/Server and the.app to / applications. Then go to the App Store and buy the latest version of the server. I'm guessing that your last purchase server had Yosemite and that's why try to install the server to the App Store / Shopping does not work. I did purchase the server from Yosemite.