Loading large text files

[Jerry Muelver]

I've got a simple language translator -- read text in one panel, select a word in the text, right-click to get a translation in a second panel. Works great with a 56k smallish language file, but falls down with a 560k file. The istring "buffer" holding the file is DEF at 700k. The file load routine is:

Code Select Expand


  buffer = ""
  if(openfile(file1,"idoeng.txt","R") = 0)
   do
     if(read(file1,ln) = 0)
if len(buffer) < (700000-256)
buffer = buffer + ln + chr$(13) + chr$(10)
endif
     endif
   until eof(file1)
   closefile file1
ELSE
   MESSAGEBOX d1,"idoeng.txt not found","No File"
  endif

Is there a way to pull the entire file (with its CRLF line endings) inthe "buffer" VAR in one go, like use binary instead of sequential file, or do that pointer-string thing I used to understand?

[Jerry Muelver]

Okay, got it working. The input var "ln" needed to be enlarged to accommodate strings longer than 256 char in the file. It loads, and works, now. Takes quite a while to load the file in, though. Is there a quicker way? Maybe I should pull it in as a resource and use it from there, instead of loading it when the program starts up.

[aurelCB]

How enlarge ln ?
Like this
def ln[4096] :istring
or something else ?

[Jerry Muelver]

ln[512]:istring is probably enough, but I made it [1024] just in case. I looked at the report file that Windows wanted to send to Microsoft, and found where the file-loading failed. Then I dug into the file to find that point and discovered there were some strings in the dictionary longer than 256 characters. So now, it works!

But it's very slow to load. I need faster loading, or a "Loading...." message in the application. I'll try pulling the file in as a binary file, and see how that goes. Also have to figure out the RESOURCE business for dictionary embedding.

[aurelCB]

Thanks for answer.
Yes it is very slow,hmm i dont know how will be with memory?

[Ionic Wind Support Team]

Jerry,
-Open the file as a binary file
-Get its length with LEN
-read it in one go with   __READ(bf, buffer, length)

Paul.

[Jerry Muelver]

Thanks, Paul. I knew there was something like that available, from back in the Fletchie DynaString days....

I also got it working by loading into a hidden richtext, and doing my search on that instead of a string. It loads instantly, but takes a tad longer than INSTR.

Now I've got to make a choide.... And then, figure out how to handle UniCode so I can show a Japanese (or Greek) dictionary entry. I'm thinking my dictionary display panel has to be richtext.... 

[Jerry Muelver]

Trying to load the file as a resource .... no luck.
My declaration for the pointer is:

DEF buffer: pointer
buffer = new(char,600000)

In my initializing sub, right after the menu is set up, I've got:

flen = LOADRESOURCE("idoeng",@RESSTRING, buffer)
MESSAGEBOX d1,"flen = "+str$(flen),"No File"

I get a 0 for "flen", whether I use "1" or "idoeng" for the resource, whether I use "buffer" as a pointer, or go for "#<string>buffer". The EXE is too small to be containing the 500K file. I have the resource as #1, custom, type "idoeng", filename correctly pointing to the dictionary file. What more do I need?

[LarryMc]

Jerry

When I was doing things like that I had problems getting it to work

I wound up using id's > 200 and a type of 324
I used the following routine to extract the resource into a file:

Code Select Expand


  ER(id:INT, restype:INT, filename:STRING)
BFILE f
   MEMORY m
   INT retc
    retc = 0
   if loadresource(id, restype, m)
      if openfile(f, filename, "w")=0
         retc = __write(f, m, LEN(m)-1)
         closefile f
      endif
      freemem m
   endif
   return
return
endsub

Maybe this will help you.

Larry

[Jerry Muelver]

Where do I send the kisses and hugs, and who shall I send as my proxy?

Perfect!           

[LarryMc]

....who shall I send as my proxy?

The frog is looking pretty good 

[Jerry Muelver]

Got it working pretty good, now. Still seriously beta, but nonetheless impressive.
Ido->English http://idomondo.org/tradidoeng.exe
Ido->Spanish http://idomondo.org/tradidoesp.exe
Ido->Japanese http://idomondo.org/tradidonip.exe
Ido file to load into top window http://idomondo.org/habemus.txt

Thanks for the tips!