Smart way to save large amounts of data using the circular buffer

Hello everyone,

I am currently enter LabView that I develop a measurement of five-channel system. Each "channel" will provide up to two digital inputs, up to three analog inputs of CSR (sampling frequency will be around 4 k to 10 k each channel) and up to five analog inputs for thermocouple (sampling frequency will be lower than 100 s/s). According to the determined user events (such as sudden speed fall) the system should save a file of PDM that contains one row for each data channel, store values n seconds before the impact that happened and with a specified user (for example 10 seconds before the fall of rotation speed, then with a length of 10 minutes).

My question is how to manage these rather huge amounts of data in an intelligent way and how to get the case of error on the hard disk without loss of samples and dumping of huge amounts of data on the disc when recording the signals when there is no impact. I thought about the following:

-use a single producer to only acquire the constant and high speed data and write data in the queues

-use consumers loop to process packets of signals when they become available and to identify impacts and save data on impact is triggered

-use the third loop with the structure of the event to give the possibility to control the VI without having to interrogate the front panel controls each time

-use some kind of memory circular buffer in the loop of consumer to store a certain number of data that can be written to the hard disk.

I hope this is the right way to do it so far.

Now, I thought about three ways to design the circular data buffer:

-l' use of RAM as a buffer (files or waiting tables with a limited number of registrations), what is written on disk in one step when you are finished while the rest of the program and DAQ should always be active

-broadcast directly to hard disk using the advanced features of PDM, and re-setting the Position to write of PDM markers go back to the first entry when a specific amount of data entry was written.

-disseminate all data on hard drive using PDM streaming, file sharing at a certain time and deleting files TDMS containing no abnormalities later when running directly.

Regarding the first possibility, I fear that there will be problems with a Crescent quickly the tables/queues, and especially when it comes to backup data from RAM to disk, my program would be stuck for once writes data only on the disk and thus losing the samples in the DAQ loop which I want to continue without interruption.

Regarding the latter, I meet lot with PDM, data gets easily damaged and I certainly don't know if the PDM Set write next Position is adapted to my needs (I need to adjust the positions for (3analog + 2ctr + 5thermo) * 5channels = line of 50 data more timestamp in the worst case!). I'm afraid also the hard drive won't be able to write fast enough to stream all the data at the same time in the worst case... ?

Regarding the third option, I fear that classify PDM and open a new TDMS file to continue recording will be fast enough to not lose data packets.

What are your thoughts here? Is there anyone who has already dealt with similar tasks? Does anyone know some raw criteria on the amount of data may be tempted to spread at an average speed of disk at the same time?

Thank you very much

OK, I'm reaching back four years when I've implemented this system, so patient with me.

We will look at has a trigger and wanting to capture samples before the trigger N and M samples after the outbreak.  The scheme is somewhat complicated, because the goal is not to "Miss" samples.  We came up with this several years ago and it seems to work - there may be an easier way to do it, but never mind.

We have created two queues - one samples of "Pre-event" line of fixed length N and a queue for event of unlimited size.  We use a design of producer/consumer, with State Machines running each loop.  Without worrying about naming the States, let me describe how each of the works.

The producer begins in its state of "Pre Trigger", using Lossy Enqueue to place data in the prior event queue.  If the trigger does not occur during this State, we're staying for the following example.  There are a few details I am forget how do ensure us that the prior event queue is full, but skip that for now.  At some point, relaxation tilt us the State. p - event.  Here we queue in the queue for event, count the number of items we enqueue.  When we get to M, we switch of States in the State of pre-event.

On the consumer side we start in one State 'pending', where we just ignore the two queues.  At some point, the trigger occurs, and we pass the consumer as a pre-event.  It is responsible for the queue (and dealing with) N elements in the queue of pre-event, then manipulate the M the following in the event queue for.  [Hmm - I don't remember how we knew what had finished the event queue for - we count m, or did you we wait until the queue was empty and the producer was again in the State of pre-event?].

There are a few 'holes' in this simple explanation, that which some, I think we filled.  For example, what happens when the triggers are too close together?  A way to handle this is to not allow a relaxation to be processed as long as the prior event queue is full.

Bob Schor

Tags: NI Software

Similar Questions

  • Advice needed on the way to store large amounts of data

    Hi guys,.

    Im not sure what the best way is to put at the disposal of my android application of large amounts of data on the local device.

    For example records of food ingredients, in the 100?

    I have read and managed to create .db using this tutorial.

    http://help.Adobe.com/en_US/air/1.5/devappsflex/WS5b3ccc516d4fbf351e63e3d118666ade46-7d49. HTML

    However, to complete the database, I use flash? If this kind of defeated the purpose of it. No point in me from a massive range of data from flash to a sql database, when I could access the as3 table data live?

    Then maybe I could create the .db with an external program? but then how do I include this .db in the apk file and deploy it for android users device.

    Or maybe I create a class as3 with a xml object initialization and use it as a way to store data?

    Any advice would be appreciated

    You can use any way you want to fill your SQLite database, including using external programs, (temporarily) incorporation of a text file with SQL, executing some statements code SQL from code AS3 etc etc.

    Once you have filled in your db, deploy with your project:

    http://chrisgriffith.WordPress.com/2011/01/11/understanding-bundled-SQLite-databases-in-AI r-for-mobile.

    Cheers, - Jon-

  • Looking for ideas on how to get large amounts of data to the line in via APEX

    Hi all

    I am building a form that will be used to provide large amounts of data in row. Only 1 or 2 columns per line, but potentially dozens or hundreds of lines.

    I was initially looking at using a tabular subform, but this feels like a method heavy since more than an insignificant number of lines.

    So now I'm wondering what are the solutions others have used?

    Theoretically, I could just provide a text box and get the user to paste in a list delimited by lines and use the background to interpret code on submit.

    Another method that I've been thinking is to get the user to save and download a CSV file that gets automatically imported by the form.

    Is there something else? If not, can someone give me any indication of which of the above would be easier to implement?

    Thank you very much

    PT

    Hi PT,.

    I would say that you need a loading data wizard to transfer your data with a CSV file. 17.13 Creating Applications with loading capacity of data

    It is available for apex 4.0 and distributions, later.

    Kind regards

    Vincent

    http://vincentdeelen.blogspot.com

  • Backup and restoration of large amounts of data during the update of the operating system (permanent storage)

    We all have our persistent store backup and restore after updating the OS using Desktop Manager,

    We store the data using intHashtable, how can we implement this?

    While searching, I had the following example, but it shows how to implement custom objects

    http://www.BlackBerry.com/knowledgecenterpublic/livelink.exe/fetch/2000/8067/645045/8655/8656/110625...

    my doubts, is how to manage the tables hash, vectors, Inthashtables...

    You'll need to serialize your data backup and deserialize on restore.  It'll be to you, how to convert these objects into a stream of bytes.  Take a look at the demo of otabackuprestore which comes with the SDK Java BlackBerry and BlackBerry JDE for an example.

  • Memory management by displaying the large amount of data

    Hello

    I have a requirement to display the large amount of data on the front in table 2 & 7 graphic during the time period 100hrs for 3 channels, data read from strings must be written in the binary file, and then converted and displayed in front of the Panel for 3 channels respectively.

    If I get 36 samples after conversion for all hours, up to 83 h 2388 samples displayed in table and graphical data are thin and samples correspond exactly.

    After that 90 hours 45 minutes late is observed after theoretical calculation of samples, what could be the problem

    I have controller dual-core PXI8108 with 1 GB of ram

    As DFGray,

    says there is no problem with the RAM or display, problem with conversion (timming issue) if I am data conversion of large amount it takes even, compared to the conversion less amount of data. So I modifed so that each data point Sec 1 is convereted at once, problem solved

    Thanks for your replies

  • How can I find a large amount of data from a stored procedure?

    How can I find a large amount of data to a stored procedure in an effective way?

    For example do not use a cursor to go through all the lines and then assign values to variables.

    Thanks in advance!

    >
    How can I find a large amount of data to a stored procedure in an effective way?

    For example do not use a cursor to go through all the lines and then assign values to variables.
    >
    Leave the query to create the object back to you.

    Declare a cursor in a package specification than the result set gives you desired. And to declare a TYPE in the package specification which returns a table composed of % rowtype to this cursor.

    Then use this type as the function's return type. Here is the code example that shows how easy it is.

    create or replace
    package pkg4
      as
        CURSOR emp_cur is (SELECT empno, ename, job, mgr, deptno FROM emp);
        type pkg_emp_table_type is table of emp_cur%rowtype;
        function get_emp(
                         p_deptno number
                        )
          return pkg_emp_table_type
          pipelined;
  end;
  / 

 create or replace
    package body pkg4
      as
        function get_emp(
                         p_deptno number
                        )
          return pkg_emp_table_type
          pipelined
          is
            v_emp_rec emp_cur%rowtype;
          begin
              open emp_cur;
              loop
                fetch emp_cur into v_emp_rec;
                exit when emp_cur%notfound;
                pipe row(v_emp_rec);
              end loop;
          end;
  end;
  / 

select * from table(pkg4.get_emp(20));

     EMPNO ENAME      JOB              MGR     DEPTNO
---------- ---------- --------- ---------- ----------
      7369 DALLAS     CLERK2          7902         20
      7566 DALLAS     MANAGER         7839         20
      7788 DALLAS     ANALYST         7566         20
      7876 DALLAS     CLERK           7788         20
      7902 DALLAS     ANALYST         7566         20

    If you return a line an actual table (all columns of the table) so you don't need to create a cursor with the query a copy you can just declare the type like this % rowtype tables table.

     create or replace
    package pkg3
      as
        type emp_table_type
          is
            table of emp%rowtype;
        function get_emp(
                         p_deptno number
                        )
          return emp_table_type
          pipelined;
  end;
  / 

 create or replace
    package body pkg3
      as
        function get_emp(
                         p_deptno number
                        )
          return emp_table_type
          pipelined
          is
          begin
              for v_rec in (select * from emp where deptno = p_deptno) loop
                pipe row(v_rec);
              end loop;
          end;
  end;
  /

  • Transport of large amounts of data from a schema of one database to another

    Hello

    We have large amount of data to a 10.2.0.4 database schema from database to another 11.2.0.3.

    Am currently using datapump but quite slow again - to have done in chunks.

    Also files datapump big enough in order to have to compress and move on the network.

    Is there a quick way to better/more?

    Habe haerd on tablespaces transportable but never used and do not know for speed - if more rapid thana datapump.

    tablespace names in the two databases.

    Also source database on the system of solaris on Sun box opertaing

    target database on aix on the power box series of ibm.

    Any ideas would be great.

    Thank you

    Published by: user5716448 on 08-Sep-2012 03:30

    Published by: user5716448 on 2012-Sep-08 03:31

    >
    Habe haerd on tablespaces transportable but never used and do not know for speed - if more rapid thana datapump.
    >
    Speed? Just copy the data files themselves at the level of the BONE. Of course, you use always EXPDP to export the "metadata" for the tablespace but that takes just seconds.

    See and try the example from Oracle-Base
    http://www.Oracle-base.com/articles/Misc/transportable-tablespaces.php

    You can also use the first two steps listed on your actual DB to see if it is eligible for transport and to see what there could be violations.
    >
    DBMS_TTS EXEC. TRANSPORT_SET_CHECK (ts_list-online 'TEST_DATA', => of incl_constraints, TRUE);

    PL/SQL procedure successfully completed.

    SQL > TRANSPORT_SET_VIOLATIONS The view is used to check violations.

    SELECT * FROM transport_set_violations;

    no selected line

    SQL >

  • Is it better to transfer usb2, eSATA (or usb3) in small quantities or large amounts of data? Corrupted data?

    Not sure if this should be in performance/maintenance or hardware/drivers.

    Hello. I was wondering about the usb2, eSATA and a bit on the usb3. I have usb2 and eSATA on my systems.

    Someone I work with told me that there may be corrupted data if you transfer a large amoutns of data via Usb2. It is best to break your files to move, copy, etc., he said. My colleague told me earlier that anything more than 30 or 40 GB start to be transferred correctly from external factors or for some reason any.

    These issues apply to eSATA or Usb3? I guess not, since these other methods are designed to transfer large amounts of data.

    Is this true? Is this due to material limitations? What is the recommended size of transfer? It's Windows XP, Vista or 7 limits?

    Any info or links are appriciated.

    Thank you.

    I have never heard of something like this before and have done some fairly large data movements in the past.  I would recommend using the program Robocopy in Windows Vista/Windows 7 (and available for Windows XP as a download add-on) to drag the move instead of type / move, given that Robocopy includes a number of features and security provisions that are not present in the case.

    'Brian V V' wrote in the new message: * e-mail address is removed from the privacy... *

    Not sure if this should be in performance/maintenance or hardware/drivers.

    Hello. I was wondering about the usb2, eSATA and a bit on the usb3. I have usb2 and eSATA on my systems.

    Someone I work with told me that there may be corrupted data if you transfer a large amoutns of data via Usb2. It is best to break your files to move, copy, etc., he said. My colleague told me earlier that anything more than 30 or 40 GB start to be transferred correctly from external factors or for some reason any.

    These issues apply to eSATA or Usb3? I guess not, since these other methods are designed to transfer large amounts of data.

    Is this true? Is this due to material limitations? What is the recommended size of transfer? It's Windows XP, Vista or 7 limits?

    Any info or links are appriciated.

    Thank you.

  • Impossible to transfer large amounts of data more than 10 GB

    Original title: the maximum data transfer size?

    I recently installed an eSata controller card in location faster PCIex-1 of my computer to benefit from a transfer of data to and from my Fantom drives GreenDrive, of 2 TB external HARD drive.  When I started to copy the files from that drive to a new drive HARD internal, I recently installed I could not transfer large amounts of data more than 10 GB.  The pop-up message indicating files preparing to copy, then nothing would pass.  When I copy or cut smaller amounts of data all works fine.  Perplexed...

    I think I got the question.  It seems that some of the files I transfer were problematic.  When I transferred in small amounts, I was then invited for what I wanted to do about these files.  Thanks for the reply!

  • Is there a way to save out/export vector art of the Flash to open as a vector art in Illustrator CC?

    Is there a way to save out/export vector art of the Flash to open as a vector art in Illustrator CC?

    Yes Alesclandre, we are looking in the workflows of Flash-Illustrator for the next update and we will have a Flash export vector that can be imported into Adobe Illustrator.

    Kind regards

    Soline

  • How many cell data uses the health app?

    We have little data plans, and my wife intends to use the health app every day.  Should I use to run the cell data health application?  Another reference... If I used the maps on the watch application would use this as much data as it normally does on my iPhone?  Just try to get an idea of the amount of data from the watch uses.  Thank you.

    I have not observed the health app using cellular data at all.

    Regarding the cards on the watch, it's just an extension of maps on your phone. Cards cannot run independently on the watch. If you use maps on the watch, he uses exactly this use from your phone from your phone is running the show.

  • Receiving data using the device RFCOMM Protocol activate bluetooth

    Hi all

    I want to apply that receive the data of bluetooth device using the RFCOMM Protocol.

    I'm pasting some of my code

    Server StreamConnectionNotifier = null, Server1 = null, server2 = null;

    Hi all

    Can some body having any idea or sample code to receive data using the for Bluetooth RFCOMM Protocol. I am able to find a device to activate bluetooth and also connect with it.

    And ther eis a problem plus I only magnified device does not form some sort of link (link Url SPP, OPP Url link, etc.) for bluetooth.

    But I can verify that this device is correct because I am able to receive data from this device to my computer by checking the connection port and to open this port.

  • Issue while loading data using the file Rules Essbase

    Hi all

    I am facing problem while loading data using the Rules file. In the rules file, I rejected several members in two areas (two dimensions). Now if I load the data using the rules file I'm getting errors for all members in the dataload.err file. If I reject mutiple members of a single field, the data load without settling errors in the dataload.err file.

    I want to know how rmany members of several fields of ejection for loading data using the file Rules Essbase? Is it possible?

    Okay, okay... I think that you must assign Global Select / reject Boolean in the parameters of loading data as 'Or':

  • Is it possible to transfer everything to a virtual hard disk that is smaller on a virtual hard disk that is larger in VMWare Workstation, using the same virtual machine?

    Hello

    I was wondering if it is possible to transfer everything to a virtual hard disk that is smaller on a virtual hard disk that is larger in VMWare Workstation, using the same virtual machine? In the affirmative, please could someone advise me on how to do it. Also utility management and disk transfer can be used in virtual machines to do this?

    Thank you very much

    Tim Auluck

    I'm not sure what exactly you are asking. Can you please clarify!

    Basically you can resize a virtual drive or add a second virtual disk to a virtual machine and transfer/copy files. You can also resize the virtual disk using VMware Converter for example.

    André

  • Send large amounts of data

    Hi all

    I did an applet that receives the data, signs and returns the signed data. When the amount of data is too big, I break into blocks of 255 bytes and use the Signature.update method.

    OK, it works fine, but perfomarnce is poor due to the large amount of blocks. Is it possible to increase the size of the blocks?

    Thank you.

    Hello

    You cannot change the size of the block, but you can change what you send.

    You can get better performance by sending multiple of your block size of hash function that the card will not do an internal buffering.

    You could also do as much of the work in your host as possible application and then simply send the data you need to work on with the private key. Generate the hash of the message does not require a private so key can be made in your host application. You then send the result of the hash to the card is encrypted with the private key. This will be the fastest method.

    See you soon,.
    Shane

Maybe you are looking for