IonicWind Software

IWBasic => General Questions => Topic started by: Jerry Muelver on January 21, 2009, 03:15:56 AM

Title: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 03:15:56 AM
I've got a simple language translator -- read text in one panel, select a word in the text, right-click to get a translation in a second panel. Works great with a 56k smallish language file, but falls down with a 560k file. The istring "buffer" holding the file is DEF at 700k. The file load routine is:

  buffer = ""
  if(openfile(file1,"idoeng.txt","R") = 0)
   do
     if(read(file1,ln) = 0)
if len(buffer) < (700000-256)
buffer = buffer + ln + chr$(13) + chr$(10)
endif
     endif
   until eof(file1)
   closefile file1
ELSE
   MESSAGEBOX d1,"idoeng.txt not found","No File"
  endif

Is there a way to pull the entire file (with its CRLF line endings) inthe "buffer" VAR in one go, like use binary instead of sequential file, or do that pointer-string thing I used to understand?
Title: Re: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 04:58:16 AM
Okay, got it working. The input var "ln" needed to be enlarged to accommodate strings longer than 256 char in the file. It loads, and works, now. Takes quite a while to load the file in, though. Is there a quicker way? Maybe I should pull it in as a resource and use it from there, instead of loading it when the program starts up.
Title: Re: Loading large text files
Post by: aurelCB on January 21, 2009, 05:38:20 AM
How enlarge ln ?
Like this
def ln[4096] :istring
or something else ?
Title: Re: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 07:18:36 AM
ln[512]:istring is probably enough, but I made it [1024] just in case. I looked at the report file that Windows wanted to send to Microsoft, and found where the file-loading failed. Then I dug into the file to find that point and discovered there were some strings in the dictionary longer than 256 characters. So now, it works!

But it's very slow to load. I need faster loading, or a "Loading...." message in the application. I'll try pulling the file in as a binary file, and see how that goes. Also have to figure out the RESOURCE business for dictionary embedding.
Title: Re: Loading large text files
Post by: aurelCB on January 21, 2009, 07:33:30 AM
Thanks for answer.
Yes it is very slow,hmm i dont know how will be with memory?
Title: Re: Loading large text files
Post by: Ionic Wind Support Team on January 21, 2009, 09:41:03 AM
Jerry,
-Open the file as a binary file
-Get its length with LEN
-read it in one go with   __READ(bf, buffer, length)

Paul.
Title: Re: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 10:23:09 AM
Thanks, Paul. I knew there was something like that available, from back in the Fletchie DynaString days....

I also got it working by loading into a hidden richtext, and doing my search on that instead of a string. It loads instantly, but takes a tad longer than INSTR.

Now I've got to make a choide.... And then, figure out how to handle UniCode so I can show a Japanese (or Greek) dictionary entry. I'm thinking my dictionary display panel has to be richtext....  ::)
Title: Re: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 02:26:28 PM
Trying to load the file as a resource .... no luck.
My declaration for the pointer is:

DEF buffer: pointer
buffer = new(char,600000)

In my initializing sub, right after the menu is set up, I've got:

flen = LOADRESOURCE("idoeng",@RESSTRING, buffer)
MESSAGEBOX d1,"flen = "+str$(flen),"No File"

I get a 0 for "flen", whether I use "1" or "idoeng" for the resource, whether I use "buffer" as a pointer, or go for "#<string>buffer". The EXE is too small to be containing the 500K file. I have the resource as #1, custom, type "idoeng", filename correctly pointing to the dictionary file. What more do I need?
Title: Re: Loading large text files
Post by: LarryMc on January 21, 2009, 03:19:32 PM
Jerry

When I was doing things like that I had problems getting it to work

I wound up using id's > 200 and a type of 324
I used the following routine to extract the resource into a file:

  ER(id:INT, restype:INT, filename:STRING)
BFILE f
   MEMORY m
   INT retc
    retc = 0
   if loadresource(id, restype, m)
      if openfile(f, filename, "w")=0
         retc = __write(f, m, LEN(m)-1)
         closefile f
      endif
      freemem m
   endif
   return
return
endsub


Maybe this will help you.

Larry
Title: Re: Loading large text files
Post by: Jerry Muelver on January 21, 2009, 03:50:31 PM
Where do I send the kisses and hugs, and who shall I send as my proxy?

Perfect!  ;D  ;D  ;D  :-*  :-*  :-*
Title: Re: Loading large text files
Post by: LarryMc on January 21, 2009, 04:56:11 PM
Quote from: Jerry Muelver on January 21, 2009, 03:50:31 PM
....who shall I send as my proxy?

The frog is looking pretty good  ;D
Title: Re: Loading large text files
Post by: Jerry Muelver on January 23, 2009, 08:33:48 AM
Got it working pretty good, now. Still seriously beta, but nonetheless impressive.
Ido->English http://idomondo.org/tradidoeng.exe
Ido->Spanish http://idomondo.org/tradidoesp.exe
Ido->Japanese http://idomondo.org/tradidonip.exe
Ido file to load into top window http://idomondo.org/habemus.txt

Thanks for the tips!