How to read a selected part of a large ascii file (~ 200 MB)?

I have several large ascii files I need to read in.  These files are part of a standard test for an application that I wrote. How well my application parses files determines how the program accomplishes its main task. For a real test, the application captures live data in the form of a 2D double table and analyzes the data of this form.  This table is long of 3 million items (1 ms/s @ 3s).  I usually never treat any form of ascii file because all data is stored as TDMS using this method (though I need to update this link with a few critical changes). (Thank you again Ben)

Ascii files I have to read all have two rows of data from the header, followed a ~ 8 million lines of data representing the data I capture in general.  Each line contains point data and a value of accumulated time.  I need to load each file separately, analyze the data within, and communicate the results.  The part that I need help for loading of the file.  So far, I was able to load a file without memory problems.  A stay is similar to a real test, I need actually only 3 million lines for analysis.  But I have to be able to select the 3 million lines by time values in the file.  Technically, I only need the single column of data and the frequency of sampling, represented by time values.

How can I select a specific section of an ascii file and read them in LabVIEW as a double 2D array?  Which is possible for 3 million points of data without crippling the system using all the memory just to accomplish this task?  An alternative version of 'last resort' would be to run a separate program to create a file DDHN that I could then read and go from there.  But I prefer to read the file directly to my request.

I'm running on an HP EliteBook 8540w with Win 7 Enterprise (64-bit), i7 CPU bicoeur (2.67 GHz) with 8 GB of RAM and 32-bit LV2011.

Thank you

Scott


Tags: NI Software

Similar Questions

  • I have CS5 running.  How to read on a Canon EOS 6 d RAW files

    I have CS5 Bridge/Photoshop running... How to read my RAW files on my new Canon EOS 6 d camera?

    you have 7.3 dng converter or better.  You can get the latest version: http://www.adobe.com/downloads/updates/

  • How to change the selected part of the pdf

    I would like to create a contract in pdf format. Is it possible to create the ONLY part of the text allowed to edit? For example: on behalf of the company to fill the street and...
    How to do step by step?

    Use the form fields.

  • HOW TO READ DATA SELECT TO_CHAR (TGL, 'DAY') AS TEMP_SELLOUT_MP_HY TGL;

    This is my code

    SET SERVEROUTPUT ON

    DECLARE

    CURSOR CUR_SELLOUT_MP IS

    SELECT / * + INDEX_JOIN (SO) * / TK.ID,.
    TMP_SELLOUT_MP. TGL,
    TMP_SELLOUT_MP. TOTAL_PENJUALAN,
    TMP_SELLOUT_MP. LIVING ROOM WITH CORNER
    TMP_SELLOUT_MP. NOKIA,
    TMP_SELLOUT_MP.SE,
    TMP_SELLOUT_MP. SS,
    TMP_SELLOUT_MP. LAINNYA,
    TMP_SELLOUT_MP. NAMA_TOKO,
    TMP_SELLOUT_MP. KOTA,
    TMP_SELLOUT_MP. NAMA_MP,


    To_char (TGL, 'Day')
    OF TK REF_TOKO,
    TEMP_SELLOUT_MP_HY TMP_SELLOUT_MP,
    REF_SALES_OFFICER SO
    WHERE SUPERIOR (TK. NAMA) = UPPER (TMP_SELLOUT_MP. NAMA_TOKO)
    AND THE TOP (TMP_SELLOUT_MP. NAMA_MP) = UPPER (SO. NAMA)
    AND SO. SOF_TYPE = "MOBILEPROMOTOR";


    CURSOR CUR_TGL IS

    SELECT DISTINCT (TO_CHAR (TGL, 'Day')) AS TEMP_SELLOUT_MP_HY TANGGAL;

    V_A VARCHAR2 (20);

    BEGIN

    FOR I IN CUR_TGL LOOP

    V_A: = I.TANGGAL;

    IF V_A = THEN 'Friday'
    DBMS_OUTPUT. PUT_LINE (V_A);
    END IF;



    END LOOP;


    END;


    Why dbms_output.put_line does not show?




    These cursor CUR_TGL data:
    Thursday
    Friday
    Wednesday
    Saturday

    Published by: xoops on March 11, 2010 09:21

    What happens when you use:

    SELECT DISTINCT(TO_CHAR(TGL, 'fmDay')) AS TANGGAL FROM TEMP_SELLOUT_MP_HY;
    

    Add 'fm' TSE format, your match does not account the trailing space:

    SQL> select '#'||to_char(sysdate, 'Day')||'#'
      2  from   dual;
    
    '#'||TO_CHA
    -----------
    #Thursday #
    
    SQL> select '#'||to_char(sysdate, 'fmDay')||'#'
      2  from   dual;
    
    '#'||TO_CHA
    -----------
    #Thursday#
    

    Published by: hoek on March 11, 2010 18:43

  • How to read the two columns of data in a file of PDM.

    Hi all

    I am reading two data columns for the 2nd of a PDM file two sheets, as shown below.

    Two columns of data must then be shared so they can be displayed in a xy chart and also apply a linear adjustment VI. I implemented the graph xy and linear adjustment using a txt file (see below), therefore all the outputs work, however this application with a tdms file turns a little more tax.

    So, essentially, that I don't know how read the correct leaf and therefore the columns in the PDM file and then how to produce the graph xy and linear adjustment of the data types produce. Here's my current attempt, which produces several errors of the type of terminal, as well as not being able to select the exact data in the PDM.

    Any help/suggestions/example vi on this asap would be greatly appreciated.

    Thanks in advance,

    Pete

    Also the PDM file viewer. VI helps you easily understand the structure of data files.

  • How to read a combination of strings and data data file

    Hello

    I'm having the combination of strings and datas to read a data file. I'm reading the file name, the time constants and comments in four indicators separate string (lines for comments varies for different files). And read the data in a 2D array. How can I do this? Y at - it a function that can serch of special characters in the spreadsheet file, so I can locate exactly where I should start reading the specific data. Here's how the data file is displayed. Thank you very much.

    Best,

    Richard

    filename.dat
    14:59 12/31/2009
    --------------------------------------------------
    Sample = 2451
    Frequency = 300, time of waiting = 2500
    Temperature = 20
    --------------------------------------------------
    a few comments
    a few comments
    a few comments
    a few comments
    a few comments
    --------------------------------------------------

    7.0000E + 2 1.5810E - 5
    7.0050E + 2 1.5400E - 5
    7.0100E + 2 1.5500E - 5
    7.0150E + 2 1.5180E - 5

    ....

    Here's a (big) hint.

    I would like to give you an excerpt, but I've not updated this machine yet.  It's a pretty boring delimiter, if you know what's good for you, you start to use commas, tabs or simple spaces (my order of preference).  It is far too easy start add or remove spaces and then you need to use more complex methods to obtain data from worksheet.

  • How to read my raw d810 8.6 raw camera files?

    can't figure out how to get camera raw 8.6?  I'm subscribed to CC for $10 per month.  Need to read my files d810

    If you have Photoshop CC, you want to download Camera Raw 9.1.

    Plugin Raw camera setup

    Edit: Make sure that you actually have installed 9.1. Sometimes the ACR 8.0 is installed when you install first PS Si such is the case, you will need to reinstall PS if the direct download link does not work.

  • How to read the value of the selected item in the DropDownList control?

    Hello

    The DropDownList component has a selectedItem property is supposed to return a ListBase.

    But what I want is just the 'chain' which is supposed to be the selected item in the DropDownList control.

    How to read this value?

    Thank you

    -Malena

    You are misreading the documentation. ListBase.selectedItem returns an object. This item comes directly from your data provider, so you just have to go up or go for any property you need.

  • How to read or get the current selected value of selectonechoice

    How to read or get the current selected value of selectonechoice, so that it can be used in the custom query.

    selectOneChoice11.getValue () m:System.NET.SocketAddress.ToString () returns the index of the list.

    from 0 - number of values in selectonechoice.

    I want to read the exact value of selected.

    Concerning

    Niaz M

    Why don't you try this?

    Re: Value of SelectOneChoice in valueChangeListener

    Both approaches will work...

    Julian

  • VBAI: How to read a shared variable?

    I'm doing my inspection to do things that depend on the value of shared Variables. How to set the VBAI Shared variables, it is obvious, but I didn't understand how to read them and then create some kind of a structure of MATTER using this variable.

    Any ideas on how to implement such an architecture?

    I create two different States and use the transitions that are based on the value of the shared variable. You can read the value of the variables or the previous measures in stages that allow this (not all stages, but most don't) or transitions. To create a new State, go to the state diagram (click on the 'toggle window view hand' so the State-transition diagram is in the main part of the window). Right click in an empty part of the state diagram and select "create a new State".

    To create a transition, right-click on a State and select 'Create a new Transition' and drag the transition to the new State.

    To change the transition, double click it and you will be able to select the shared variable as measurement, and comparison operations to make the decision. If this comparison is false, the default transition is taken instead.

    Select help > Show context Help if you change the schema of the State for more details on how to use it.

    Hope this helps,

    Brad

  • Dual Boot with windows XP Pro with 2 HARD drive, error on secondary windows install: "Windows did not start because of a configuration of the disk of the computer problem. Could not read the selected boot disk. Check boot path and disk hardware... »

    Hi, I recently bought a new Hitachi 1 TB internal hard drive. I wanted to do a clean install on the new drive (d), be able to dual boot my computer and leave my old 300 GB disk (C :) as the primary windows installation. After physically installing the new HARD drive, I noticed that "new hardware found" and the the brand name flashed in the bottom right corner. I thought I'd see the new hard drive in my computer, but I did not. I initialized the new HARD drive with the computer management window (right click on my computer - manage) and I also formatted in NTFS (I had to change my printer hard drive letter to assign the new HARD drive with the letter D, but I've made other changes after this step). Now, I could see and use in my main windows on the C: installation. I then installed a windows new copy XP pro with the installation CD I got for my C: but on the D: (Note: I have a Dell computer (and the Dell reinstallation CD) and my current version of windows has SP3 while the installation CD is only SP2) (I also reformatted my HARD drive during the installation of windows to NTFS)

    Now when I try to start on installing the new on my D: I get "Windows did not start because of a configuration of the disk of the computer problem. Could not read the selected boot disk. Check startup disk and hardware access path. "and another line saying to refer to manuals HARD drive and windows.

    My Boot.ini file at this time was:

    [boot loader]
    Timeout = 5
    default = multi (0) disk (0) rdisk (0) partition (2) \WINDOWS
    [operating systems]
    "multi (0) disk (0) rdisk (0) partition (2) \WINDOWS="Microsoft Windows XP Professional "/ noexecute = optin/fastdetect
    "signature (ac516a5c) disk (0) rdisk (0) partition (1) \WINDOWS="Microsoft Windows XP Professional own "/ noexecute = optin/fastdetect

    I looked towards the top of this help page, but for now I'm unlucky:

    http://support.Microsoft.com/kb/314477

    (The next steps I took by throwing from the Windows CD and go into the repair console)

    I tried to use the bootcfg /rebuild but the line to boot.ini for my secondary HARD drive still does not work.

    He gave me something like:

    [boot loader]
    Timeout = 5
    default = multi (0) disk (0) rdisk (0) partition (2) \WINDOWS
    [operating systems]
    "multi (0) disk (0) rdisk (0) partition (2) \WINDOWS="Microsoft Windows XP Professional "/ noexecute = optin/fastdetect
    "signature (ac516a5c) disk (0) rdisk (1) partition (1) \WINDOWS="Microsoft Windows XP Professional own "/ noexecute = optin/fastdetect

    I also tried to change the signature() to multi (0), but it did not work. (or try almost all possible permutations of disk() and rdisk() partition() 0 to 2).

    I also tried to use expand F(dvd-drive):\i386\ntkrnlmp(I_have_a_duo-core).ex_ D:\windows\system32\ntoskrnl.exe and a few other combinations, but I when I login as D:\windows simply, it fails to extend with a message like "could not develop" or C:\windows I get "restricted access".

    I have not yet tried CHKDSK/r, but since I did a clean install on a newly formatted drive... I don't think it would work.

    I've also marked the partition on the D: as active (in computer management), but it did not help me.

    I've done a first installation of windows repair, strangely, I couldn't fix this one... Still does not work...

    My default windows installation yet works very well & I see that windows has been installed on the D: but I'm ideas from how to operate the dual-boot, can someone throw me a BONE on what to do?

    Maybe try to do something with the Boot Manager (whatever it is?) Create a new partition and use it as a boot manager? I don't really want to do this, because if I made a mistake I will be unable to use my computer and dual boot should work without going through all this trouble... But I found this page: http://technet.microsoft.com/fr-fr/library/ee829686%28WS.10%29.aspx

    Could a clean install two HDD help? I hope I won't have to do...

    Someone has an idea?

    PS: I'm not such an advanced user so I need a solution step by step, thanks!

    Edit: I also noted that I can't see my new HARD drive in my BIOS... weird... SATA-0 is my old drive HARD 1 and 2 are my dvd 3-5 readers are 'off' and it doesn't seem to be anything that it is plugged in. I also have 4 bays for internal HARD disks. Now, 2 are busy.

    Edit2: I looked inside my case and enabled the good Sata in the Bios after a reboot it detected the drive correctly. Dual boot doesn't always work well... The fact that it was disable in the BIOS while I was installing the new HARD drive could change anything? Hmm...

    In case I wasn't clear enough: I had a windows XP Pro installed on my C:, bought a new hard drive and installed the same copy of windows on it (d). I think that should allow me to double boot between the two installs different. Even if for some reason I can't boot from one on my D:. Andrew, I do not understand why you so far physically disconnect the other drive, the point of the whole operation is to build a boot.ini file so that the BIOS knows that it y facilities 2 windows and you allow to choose (and not have conflicting problems between the 2 systems), someone correct me if I'm wrong? "Expand" the part of my first post made reference to one of the solutions provided by Microsoft in the first link that I have included in my previous post (but I can't seem to work).

    In addition, it is the procedure that I followed: http://www.ehow.com/how_5950826_boot-two-different-hard-drives.html

    (except I've had the first installation already done).

    Andrew "also, once a new hd is plugged on the pc & running, most of the BIOS will place (them) as 1st priority hd, set the BIOS to do this", you mean that by installing separately from windows on each unit when the other is not plugged in, I could choose what disk to start from defining their priority in the BIOS once they are all plugged back? It's too tedious (as I intend to move frequently between the 2 installs) and requires a computer to other users of the lack of computer knowledge. Unfortunately, it is not a viable option.

    EDIT: I DID IT! Hmmm... Well, I double checked in the recovery console, the path of the new HARD drive with the arc command Scan. Tried to start again with the MULTI (0) in my boot.ini and it worked... I also had my windows install CD in. Maybe that is why I could not start, windows needs the CD to complete the installation? Weird, I don't know what I did to the difficulty it... but hey, it works! Thanks for your ideas guys!

  • How to read the VI Analyzer config file?

    Hello

    How to read the analuzer vi configuration file? I'm getting a lot of errors on the size of the front panel and I don't know what should be the size you want. Where is the maximum expected size of the specified front panel?

    Kaspar says:

    Hello

    How to read the analuzer vi configuration file? I'm getting a lot of errors on the size of the front panel and I don't know what should be the size you want. Where is the maximum expected size of the specified front panel?

    If you have VI Analyzer installed, you can load the configuration file and look at the section select the Tests. There is a test for the maximum size of front under front-> User Interface. The default value is the resolution of the current monitor of the PC (including multiple monitors). You can change to the desired option.

  • How to read 3 analog inputs simultaneously with 6070E?

    Hello

    I wanted to read 3 analog signals simultaneously using cards of acquiring data NI PXI-6070E and Labview 2010.

    I can with success read and record the 1 signal (which you can see in the attachment), but do not know and can not find examples on how to read 3 signals simultaneously. Any help, especially a simple example would be great.

    Thank you!

    Just click on = pull down from the constant physical channel, select Browse and shift-click or Ctl-click here to add other channels. You can also type in new channels with the ai0 syntax: 2 for three continuous channels for example.

  • How to read continuously from the rs232?

    Hello

    I'm new to LabVIEW and my questions would be obvious for some for you. So please help if you know the solution.

    It's for my school project. For your information, we need to schedule a Committee of Altera DE2 to control traffic lights (something like that) and at the same time, the Commission shall forward the States of enlightenment to a computer with LabVIEW installed via an RS-232 cable.

    The rest of the project was fine except for the part of LabVIEW, because as I said earlier, I'm new to LabVIEW.

    We use the "basic series write and Read.vi" in the folder examples of NEITHER. The problem is that we do not know how to change the vi in order to receive the data that come the DE2 Board permanently. Each time, the vi stops reading after receiving 24 bytes of data.

    So, the first question is:
    1.) how to read continuously from the RS-232 port? (We tried to use while "loop" but without success, maybe we were doing it the wrong way.) If you know the solution, please enlighten us for us in a detailed manner.

    (2) Secondly, once we have received the 'chain', how can convert us to bits (0 and 1) so that we might be able to manipulate the bits, for example turn on the LEDs that correspond to the bits.

    Thank you.

    Instead of reading the bytes of the port so to read that many bytes (which may be zero), just do a VISA read for 1 byte.  He will wait for this next byte.

    To get an array of Boolean out of this byte, use this code.  He turns the characters into a byte array.  Takes the first and the only element of this array, and then converts this single U8 number into an array of bytes.

    Which country are you?  I've never seen or heard lights orange before.  Only the yellow lights.

  • How to read several bye a file of text one

    Hi all

    I need to select the files of text several times and read the files one by one

    some how I managed to select the file to a single text and analyze the data in the text file

    I need to do for mutiple files at the same time

    Please help me

    gowthamggk wrote:

    I have sucessfully read a text from the file through this VI

    but like like that... I need for multiple files

    Why do you not use loop For?

    gowthamggk wrote:

    also should read the text line by line

    You can right-click the function "read text file" and select "read lines." See the code snippet below.

Maybe you are looking for

  • repair iphone warranty in Cambodia

    How can I get my iphone repaired when you live in Cambodia? iPhone 6 s, bought the Birmingham UK Airport, August 16, 2016, screen went black 3 weeks ago, does not work, no phone number to ring to Cambodia I can do? John Wharmby p/n + 883754699.

  • How to move photos from the Photos app in El Capitan?

    I have a few dozens of photos that apparently were downloaded from an iPhone in the Photos app in El Capitan on my MacBook Pro. I prefer to manually manage the photos in the form of image files in a directory of classic storage. Is there an easy way

  • Why after an update my Norton 360 registry cleaning is a registry key not valid?

    This time was the in update for Adobe Reader, but I saw it with some other updates as well. I noticed this problem in my computer after that microsoft has passed to outlool.com. From there, I started to get the registry changes any time I open micros

  • Game lag

    Hey guys,.I got Bioshock to my new windows 7 Asus gaming computer and or a reason any after that I turned on the game and learned to play. The game was lagging like crazy. I looked at all the requirements and was unable to understand why it had been

  • Active Sync problem - push slow

    Hello I use the premium new Z5 with Active Sync (Microsoft Exchange Online). The email push notifications come very late, sometimes several minutes later. What could be the problem? Thank you Marco