Questions after TimesTen first trial: memory footprint and query performance

Hello!

I'm testing TimesTen In - Memory Database cache to see if it could help with some ad hoc reports questioned this need too long to run in our Oracle database.

Here is the configuration:

1.) TimesTen Server CPU Quad Core 2 with 32 GB of RAM running Windows 2003 x 64.

2.) put in place two cachegroups read-only: a little for a quick test and the real thing that maps to a table of the database as such:


Database table looks like:
  CREATE TABLE "TB_BD" 
   (   
   "VALUE" NUMBER NOT NULL ENABLE, 
   "TIME_UTC" TIMESTAMP (6) NOT NULL ENABLE, 
   "ASSIGNED_TO_ID" NUMBER NOT NULL ENABLE, 
   "EVENT_ID" NUMBER, 
   "ID" NUMBER NOT NULL ENABLE, 
   "ID_LABEL" NUMBER NOT NULL ENABLE, 
   "ID_ALARM" NUMBER, 
    CONSTRAINT "PK_TB_BD" PRIMARY KEY ("ID")  
   );
Oracle database table has 1.367.336.329 lines and table segments are approximately 61 GB, so a medium line takes about 46 bytes.

Since I have 32 GB in the TimesTen machine, I created the Group cache with a where predicate in the ID column that only the 98.191.284 most recent ranks get in the cache group. In the Oracle database, it is around 4.2 GB of data.

After the cache loading dssize group returns:
Command> dssize

  PERM_ALLOCATED_SIZE:      26624000
  PERM_IN_USE_SIZE:         19772852
  PERM_IN_USE_HIGH_WATER:   26622892
  TEMP_ALLOCATED_SIZE:      32768
  TEMP_IN_USE_SIZE:         10570
  TEMP_IN_USE_HIGH_WATER:   14192

  (Note: the high PERM_IN_USE_HIGH_WATER comes from a first test where I tried to cache too many rows)
I then ran on the TimesTen machine:
tisql> select avg(value) from tb_bd;
She is still going after 10 hours, so I can already tell that the query execution time is not really met my expectations. :-)

In the Windows Task Manager, I see that tisql constantly use 13% of CPU (= 100% / 8 cores), so that it uses only a carrot, but even he was using all the hearts and the execution time would be 1/8th, it wouldn't meet my expectation. :-)

I also see in the the Windows Task Manager who becomes slowly higher and higher, currently the 'MemUsage' of my tisq 14FR processl. I believe that it is shared memory mapping that is already mapped by the TimesTen process that has approximately 24 GB mapped. The query is probably 53% through and the total time of queries can be around 20 hours.


My questions:

1.) for what I tested, 1 GB of data in the table Oracle needs about 4-5 gigabytes of memory in the TimesTen database. I read a post on the forum who has explained with ' data are optimized for performance, no space in TT ", but I don't quite buy it. A factor of 4-5 means that the CPU must spend 4 to 5 times the amount of data. The data is not compressed in the Oracle database, but it is in its natural binary form. I would like to understand why data takes much more space in TT - like when you have a numeric in Oracle, which TT do with it to make it 4 - 5 times bigger and why does do that?

2.) regarding the performance of the queries: how long can take even to the base allows to browse about 20 GB of data in memory, number of lines, summarize the NUMBER of a column with a division to get the avg (< column >)? Is there something flawed with my setup?


Thanks for the ideas!

Kind regards
Marcus

Published by: user11973438 on 06.09.2012 23:27

I agree that the use of 4 - 5 times more memory than Oracle is far from optimal. Your drawing is unfortunately a little pathological; normally we see more like 2 - 3 times (which is still too really0. There are many internal differences between Oracle and TimesTen in the way data are stored internally. Some are historical, and some are due to the optimization of performance rather than storage efficiency.

For example:

1 oracle lines are always variable storage length while TimesTen lines are always of fixed length in storage.

2. in Oracle, a column defined as NUMBER only occupies the space needed based on the stored value. In TimesTen SEVERAL column always occupies the space to store the maximum possible precision and therefore takes up 22 bytes. You can reduce it by restricting explicitly using NUMBER (n) or NUMBER(n,p).

3 TimesTen does not support any kind of parallel query within a single data store. All queries will be run using maximum core CPU; Oracle DB supports parallel queries and so it can make a big difference for certain types of application.

4. NUMBER is implemented in software and is relatively ineffective. Calculating the average of almost 100M lines will take time... You can try to change cela a native binary type (TT_INTEGER, TT_BIGINT, BINARY_DOUBLE depending on your data); This will no doubt give a good improvement (but see point 5 below).

5. with a database of this size, it is possible that Windows made a lot of paging, while the query is running. I myself also observed on Windows it seems to be a penalty when a process key/maps a page for the first time. You should monitor the paging activity via the task manager that the query is run. All important pagination will really affect the performance. Also, try to execute the query a second time without disconnecting ttIsql this may also show an advantage. On Unix/Linux platforms, we provide an option (MemoryLock == 4) to lock the entire database in physical memory to prevent any paging, but is not available under Windows.

Chris

Tags: Database

Similar Questions

  • Question about version first submit 1.0 and 1.1, with some corrections?

    Hello

    First of all excuse me if this question has already been answered here.

    If my app (version 1.0) is rejected due some bugs, what's the next step?  Fix it and submit the 1.1 version or do it with 1.0? In my case I presented version 1.1 because I couldn't do it with 1.0 as the application was already signed.

    Thank you

    You can not re - sign with the same version, so change it demanded. There is no need to go back to 1.1, however, if you do not want. There is up to four items to the version number. My last example is 1.0.4.1261... You could make 1.0.0.1 if you wanted.

  • Compression and query performance in data warehouses

    Hello

    Using Oracle 11.2.0.3 have a table made large index bitmap with dimensions would reveal.

    Understanding bitmap indexes are compressed by default, so assume it cannot compress more.

    Is this correct?

    You want to try to compress the large fact table to see if this will reduce the IO on the readings and so give performance benefits.

    Speed ETL fine want just to increase the performance of the report.

    Thoughts - someone has seen significant gains in data warehouse performance report with compression.

    In addition, current PCTFREE on table 10%.

    Insert as only tabel considering making this 1% to redesign performance report.

    Thoughts?

    Thank you

    Yes, you can't compress bitmap indexes, they are already compressed.

    Yes, Compression (obviously depends on the type of data) reduced definetely e physics / s so improves performance. We saw this in our environment.

    There ONLY inserts and updates, you can go with pctfree 0, you need to book ANYTHING for updates.

  • 10 Windows, system and process compressed memory &amp; slow down the performance of Lightroom

    I use Lightroom 6.6 (non - CC version) and for months, I have noticed the degradation in performance on my machine Windows 10 the more Lightroom runs. LR restarts solves the problem for a while, but anywhere from minutes to an hour, Lightroom performance down to the level where it becomes unusable.

    Finally, I think I found a reproducible indicator in the Task Manager which may lead to the reason for this - it seems to be linked to the process "System and compressed memory" which is new for Windows 10. Maybe it's a coincidence, it could be the correlation instead of causation - but that's what I want to know.

    When I start Lightroom and edit some pictures, everything seems normal in the Task Manager:

    17-07-2016 00-59-41.png

    Lightroom uses about 6 GB of memory, which is OK, as I 24GO of RAM in my machine. The mentioned system and compressed memory process currently maintains that a low profile with the use of memory 204 MB and Lightroom performs well at this point.

    Some pictures later, however, things start to get worse:

    17-07-2016 01-09-15.png

    The system and compressed memory process increases its memory footprint and Lightroom memory footprint decreases. At this point, I think Lightroom is already slower - but this could be just imagination. It isn't so bad so I can be sure that it really is.

    But a few minutes and retouching of photos later, system and compressed memory past Lightroom in the memory usage and which are how things always become really bad:

    17-07-2016 01-37-54.png

    System and compressed memory draws a memory more and more away from Lightroom, and meanwhile, Lightrooms performance is killed. As I understand the system and memory tablet, it's intent is to compress the memory instead of disk paging-, but it seems he has high performance here and there of the implications would not even be necessary to page or compress memory because there were a lot:

    17-07-2016 01-38-08.png

    Perhaps a Windows problem, may be a Lightroom problem - chances are, it's a combination of the two. With 24 GB RAM and Lightroom be gourmet applications in memory just running, this should not occur or cause problems. I use other applications in memory such as Visual Studio, or sometimes first - this does not occur when you use this application.

    I can't believe I'm the only one knows than that on Windows 10 - yet I have not found much about it online, just a few remarks on this subject in some support threads here and there, but no solution.

    So-, here is my question d ' others have experienced this too? Someone was able to get this under control? It's really frustrating relaunch Lightroom every 30 to 60 minutes to get a good performance.

    Thanks for the tips

    b.

    There is a leak memory known in Lightroom 6.6. You might want to try to go back to Lightroom 6.5.1 and see if the problem goes away. See How do I restore a previous version of Lightroom? The Lightroom Queen and where it says Lightroom 6.1.1 you insert Lightroom 6.5.1

  • Very bad audio and video performance

    I have a Pavilion dm1 laptop running Windows 7.  When I got all first the machine audio and video performance was very good.  However, towards the end of last year the performance for audio and video has become extremely slow.  Still, the sound stutters even without video.

    I used the tools to clean the registry, updated (or verified) all drivers I can think, but nothing has not improved.

    Maybe related, but less to worry about because it is necessary that in rare occasions, boot has become very slow too.

    I think I have tracked down a part of the problem by chance.

    Normally, I start my DM1 when connected to various devices of office as keyboards etc.  These are all plugged into the USB ports.  As the boot begins there very long delays in the process.  And when the boot ends stuttering audio and video as described.  However, if no USB device is attached during the boot process, start happening quickly and the audio/video all work fine.

  • Type questions NMH405 Impressions after the first few days - and a few beginners

    Hi all, after some research I made the plunge and acquired a new NMH405 (I really wouldn't have no existing "NMH400" i.e. no discs).  I removed the 500 GB drive and added two new drives 1.5 to WD Green Caviar and set up in Raid-1 after a few messages of error and waiting for a very long time, so that readers resynched I managed to get online as z-drive on my vista system.  I played a bit with the built-in backup tool (I like it!).  I tried the builtin player after placing some of my iTunes on the pole media albums and decided that I didn't really like this option for many.  I then simply moved my itunes on the MediaHub libraries and used iTunes on Vista as a front-end application.   It works very well.

    I have a few questions, probably show my limited knowledge of readers connected to the network.

    Q1.  Is there another way to turn off the device by pressing the bottom on the NMH405 (aka my NMH415!)? When turned off, I assume that the only other way to power it on again via the background?

    Q2.  I noticed that the device is offline for centuries after having recycled Windows Vista or the hub itself.  Why is this? And what can I do to put it online in an orderly fashion?  There must be a way to get that ugly red crossed NAS showing green and online as a z-drive in 'My Computer' much faster.

    Q3.  I reformatted the drive of 500 GB that the box came with, added to the drive enclosure box and attached the port USB in front of the NMH405, after awhile he said on the Panel until she had detected the drive and asker if I wanted to import content.  I refused (partly because he was empty!).  Now, how do you get this drive? (Occasionally), I would use it to backup my Raid-1 configuration files (most recent do you know what might happen there...).  How to use this USB disk attached?  Also, I noticed that it does not appear in 'My Computer' - in fact "computer" in windows?  Add directly to my windows system it is detected instantly?

    Thanks for the pointers or responses to improve my user experience.  I'm still learning here.

    Best regards

    Q1. Your correct, there is only one button on the front for the start/stop. There is an option to restart the device inside the web interface through the configuration > system option.

    Q2. It depends on how fast your network can establish a connection, a mapped network drive is fine but I noticed that it slows down the process startup or logon when you turn on the PC. What I did just create a shortcut on the desktop (site \\nmh410\media) that points to my mediahub and did not map the mediahub more to drive z.

    Q3. If it's still a drive of 500 GB fresh with nothing on it, then you will need to create a partition on it firstly the external USB drive by connecting directly to the PC. Read this Microsoft article on partitioning. In my view, that an external USB drive icon will not be displayed on the PC from the external USB drive is plugged into USB port of mediahub. To access this disk external USB which is plugged into the USB port of mediahub, I simply create a shortcut to \\nmh410\media\device . For backup, you can use the NTI Backup software that accompanies your mediahub.

    See you soon.

  • Cannot install After Effect free trial, the page shows "that we are now some questions, please try again later." If the problem persists, contact customer support. "for several days.

    Cannot install After Effect free trial, the page shows "that we are now some questions, please try again later." If the problem persists, contact customer support. "for several days.

    Is this problem will soon be resolved? Thank you!

    You MUST have a 64-bit computer and operating system... What is your computer?

    -Programs are supposed TO not view or download if they are not installed on your computer

    MINIMUM for After Effects and Premiere Pro http://helpx.adobe.com/premiere-pro/system-requirements.html

  • After downloading the trial version free effect page works. It shows "that we are now some questions, please try again later." If the problem persists, contact customer support. "for a few days. This problem will be solved soon?

    After downloading the trial version free effect page works. It shows "that we are now some questions, please try again later." If the problem persists, contact customer support. "for a few days. This problem will be solved soon?

    Hello

    I tried the method via install CreativeCloudInstaller, it shows no AE in the application list.

    It seems that I do not meet the minimum requirement for the installation of AE, my computer is 32-bit instead of 64-bit.

    It would be more useful if they show the more accurate message in installer CC and the free trial link page in the Adobe Web site.

    Thanks a lot for all media!

  • I downloaded lightroom after a free trial and I can't seem to run. I have regitered product?

    I HAV edownloaded lightroom after a free trial TSO and can't make it work?

    @

    Open Lightroom and you should see a message on your trial period that expires and gives you the option to purchase.  Click this option, and you should be given the opportunity to enter your serial number.

    If not, uninstall your trial of lightroom (using adobe uninstaller).  then download and install lr 5.7.

    Downloads available:

    Download and installation help links Adobe

    Help download and installation to Prodesigntools links can be found on the most linked pages.  They are essential; especially steps 1, 2 and 3.  If you click on a link that does not have these listed steps, open a second window by using the link to Lightroom 3 to see these "important Instructions".

  • Is a Pro at the first trial here, and if so, where is it?

    Some of the experts in the forum hardware, after seeing my material, recommended that I try Premier Pro with the trial to see how it works with my computer for 30 days.  I went to trials, and along the way, someone came online with a chat window and asked if he could help me.  I explained, and I could tell he pointed me to a trial of creative cloud (something I don't want to use).  I explained that I was looking for a Premier Pro, not the CC trial, and he said that he undstood.  He put a link in its window, and I pulled it thinking I was where I needed to be.  Unfortunately, he downloaded the app for creative cloud, which is what I didn't need.

    Also separately, I found files for the first Pro program and downloaded (4 files, a little less than 2 gigabytes).  But those who do not resemble a trial, rather they resemble replacement for the purchased program files.

    I've looked everywhere on the Adobe website and read the entries I could find here on a Pro of the first trial and still not put it to the test

    Here can someone help me find the trial for Premier Pro?  Or if it does not exist, should I try the trial CC version, and that is representative of how the program will occur if I buy Premier Pro.

    Terry Quinn

    PS Adobe seems of course difficult to evaluate and buy their product.

    You can download a trial of Premiere Pro CS6 via creative cloud application manager. And you can download and install the application handler without subscribing to what it is. On the applications tab, click filters and Versions and select the previous Version.

    If you have difficulties to go this route, you can also get free trial here: Download Adobe CS6 tests: direct links (without Assistant or Manager) | ProDesignTools

    No matter how do you to get the trial, if you decide to purchase a perpetual license, you can do so through this page: Creative Suite 6. It would also be possible to purchase via the Help menu.

    Edited to add: I think just a place where you get confused is confusing the subscription of creative cloud system and the Manager of creative application Cloud - certainly a point understandable confusion. You can use the application manager CC to install a trial version of any available application threw creative cloud and then buy the perpetual license outside the creative cloud. Another understandable point of confusion is that 'CS6' and 'CC' are different versions, but you can download CS6 via the CC application manager.

  • Have been left in charge all night I get the message "after reboot requires the password" question is why is this automatic restart, and is this normal

    Have been left in charge all night I get the message "after reboot requires the password" question is why is this automatic restart, and is this normal

    Occasionally, it may restart because something was wrong. It is not a problem unless this happens frequently.

  • Firefox will open 4 or 5 thin legs, but then do not load any other sites after the first 4 or 5 or you can update one of these first tabs - including: config and the addon page

    Firefox 5 worked well. I installed Firefox 7, and when I ran, tabs just say «connecting to...» "and hang up. Rebooting did not help. Sites Web open correctly in IE and Chrome. Disabled all firewalls and antivirus, nothing helped. Uninstalled and reinstalled Firefox 5, everything has worked again. It was using Vista 64-bit.

    Upgrading to Windows 7, uninstalled Firefox 5, installed Firefox 7, had the same problem. Uninstalled Firefox 7 completely (including profile information, I saved this information in another file), restarted the computer and installed Firefox 7 using a clean profile. Do not install add-ons, checked to make sure that all plug-ins were updated and updated plugins up-to-date.

    Now when I run Firefox, I can load 3 or 4 or 5 tabs very well - after these first tabs bit are open, I can not open or refresh all other tabs - with about: config and the add-on Manager - I have to restart Firefox. The hangup does not appear to be associated with what websites I try to open, but it seems that the number is the problem. I ran through all the procedures of the FAQ, including changing the maximum number of connections to 48, and the problem does not seem to go away.

    As a side note, I had this same problem when I tried to go from version 5 to version 6 more. Version 5 is the latest version which worked on my system.

    The default value of the pref Network.http.Max - connections has been increased from 30 up to 256 in versions of Firefox 6 +.

    Try to decrease the value of the pref Network.http.Max - connections from 255 to a setting like 30 such as used in versions of Firefox 3.

    See also:

  • URGENT::opening multiple files - i have 3 files in my directory... When I run the code I can open a file and read it... but the while loop get blocked after the first iteration... can someone me help or give some indications

    bbmChatDB of public database;

    public static FileConnection fconnRead = null;
    public static String fileReadData = "";
    public static InputStream is = null;
    data Byte [] = null;
    DataInputStream is = null;

    Here is my code...

    I have 3 files in my directory... When I run the code I can open a file and read it... but the while loop get blocked after the first iteration... can someone me help or give some indications

    try {}

    FileConnection fc = Connector.open("file:///store/home/user/documents/BSM/") (FileConnection); "

    If (fc.exists ()) {}

    Enumeration e = fc.list ();

    While (e.hasMoreElements ()) {}

    System.out.println ("files are:" + (String) e.nextElement ());

    play this file
    StringBuffer stringBuff = new StringBuffer();
    try {}
    System.out.println ("opening file")-;
    System.out.println ("file name is:" + (String) e.nextElement ());
    fconnRead = Connector.open("file:///store/home/user/documents/BSM/(String)e.nextElement(),Connector.READ_WRITE) (FileConnection);
    System.out.println ("data length")-;
    If (fconnRead.exists ()) {}
    is = fconnRead.openDataInputStream ();
    data = IOUtilities.streamToBytes (is);
    Ddd = new String string (data);
    fileReadData = ddd.toString ();
    System.out.println ("length of data:" + fileReadData.length ());
    System.out.println ("read data :" + fileReadData);

    }

    } catch (IOException ee) {}
    ee.printStackTrace ();
    System.out.println ("Exception in the read data :")
    + ee.getMessage ());
    }

    }
    }

    } catch (IOException e) {}

    e.printStackTrace ();
    }

    {Finally

    try {}
    If (is! = null) {}
    is. Close();
    }
    System.out.println ("is closed...");
    } catch (IOException e1) {}

    E1. PrintStackTrace();
    }

    If (fconnRead! = null) {}
    try {}
    fconnRead.close ();
    } catch (Exception e) {}
    System.out.println (try ());
    }

    }

    }

    You can recode this treatment so that it uses only:

    e.nextElement ())

    Once a loop iteration.

    Directly at the start saying something like:

    String fileName = e.nextElement ());

    and use fileName everywhere in your loop.

    Also be aware that printStackTrace() will do nothing in your situation, it only works if you catch Throwable.  So make sure you something output all your catches exception and also have a catch (Throwable t) to catch the things you miss, as follows:

    {} catch (Throwable t)

    t.printStackTrace ();

    System.out.println ("Eception exception:" + t.toString ());

    }

    I think your code is thrown an exception and you don't see it.

  • I want to ask a question about my mobile mouse memory. I googled my problem and arrived at a forum. He has not answered my question, so I want to ask a question, but I couldn't find how to ask.

    I want to ask a question about my mobile mouse memory.  I googled my problem and arrived at a forum.  He has not answered my question, so I want to ask a question, but I couldn't find how to ask.

    You can ask your question here.

  • values based on the following logic 'values after the first values of two '_' and before last '_' values '.

    Hi all

    I need the values according to below 2logics in a single select query using instring and substring

    1 values based on the following logic 'values after the first values of two '_' and before last '_' values '.

    2 values based on the following logic 'values after the first values of two '_' and before last'-'values '.

    EXM:

    Entry: ABCD_EFGH_IJKLM - NOPQ_XYZ output: IJKLM - NOPQ

    Entry:. ABCD_EFGH_IJKLM - NOPQ output:IJKLM

    Thank you.

    Check the following

    WITH DATA1 AS

    (SELECT "ABCD_EFGH_IJKLM - NOPQ_XYZ" double val)

    UNION ALL

    SELECT 'ABCD_EFGH_IJKLM - NOPQ' double val

    )

    SELECT SUBSTR (VAL, INSTR(VAL,'_',1,2) + 1, DECODE (BIGGER (INSTR (VAL, '_',-1, 1), INSTR(VAL,'-',-1,1)), INSTR (VAL,'-', - 1, 1), LENGTH (VAL) + 1, INSTR (VAL, '_',-1, 1))-(INSTR (VAL, '_', 1, 2) + 1))

    OF DATA1;

    Concerning

    Salim

Maybe you are looking for