ZeroMemory

[Brian]

Just been reading on MSDN about zeroing memory. Instead of RtlZeroMemory, they recommend using RtlSecureZeroMemory, as this call is guaranteed to work with all flavours of compilers

Thing is, it doesn't seem to be supported in our include files. I've tried a direct DECLARE IMPORT call in my code, but it flags an error. Can't find what DLL it is declared in, as well

Any ideas?

Brian

[jalih]

Just been reading on MSDN about zeroing memory. Instead of RtlZeroMemory, they recommend using RtlSecureZeroMemory, as this call is guaranteed to work with all flavours of compilers

Thing is, it doesn't seem to be supported in our include files. I've tried a direct DECLARE IMPORT call in my code, but it flags an error. Can't find what DLL it is declared in, as well

Any ideas?

Brian

RtlSecureZeroMemory is inline function defined probably somewhere in Microsoft C compiler header files. You can't directly import it, you have to translate it from C (or asm?) to IWB.

If you are not storing passwords or some other sensitive information, then just using RtlZeroMemory is the best choice.

[Brian]

Thanks, Jalih,

I will stick with what I know, then!

Brian

[LarryMc]

rtlFillMemory is also available to us to fill memory with a byte other than zero if we desire....just saying

[Andy]

Am I missing something here....(probably knowing me)?

RtlZeroMemory which I've asked about before doesn't seem to clear out an array - see code:

Code Select Expand


$include "windowssdk.inc"

int x
string a[100]

a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"

openconsole
print
print "Before zero memory"
print
for x = 0 to 4
    print "string ",x," = ",a[x]
next x

print
rtlzeromemory(a,LEN(a))
print
print "After zero memory"
print

for x = 0 to 4
    print "string ",x," = ",a[x]
next x

do:until inkey$ <> ""
closeconsole
end

It only seems to blank out a[0] (zero), not the rest of the array values.

Andy.

[Brian]

Andy, best I can come up with at the moment is:

   RtlZeroMemory(a[0],255*LEN(a[0]))
   RtlZeroMemory(a[1],255*LEN(a[1]))
   RtlZeroMemory(a[2],255*LEN(a[2]))
   RtlZeroMemory(a[3],255*LEN(a[3]))
   RtlZeroMemory(a[4],255*LEN(a[4]))

Don't really know if you need the 255, though - I will have another look

Brian

No, you don't need the 255. I also tried with _memset, which could be quicker:

DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)

_memset(a,0,LEN(a))

Brian (again)

[Egil]

Andy,
The way I see it you have declared an array of five string variables, each 1 byte long. Therefore your code won't work because LEN(a) becomes ONE byte long. You have to clear each member of the array. I know this has been discussed before, but do not remember if it was on this forum or somewhere else.

Code Select Expand

$include "windowssdk.inc"

int x
string a[100]

a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"

openconsole
print
print "Before zero memory"
print
for x = 0 to 4
    print "string ",x," = ",a[x]
next x

print
'rtlzeromemory(a,LEN(a))
for x = 0 to 4
rtlzeromemory(a[x],LEN(a[x]))
next x

print
print "After zero memory"
print

for x = 0 to 4
    print "string ",x," = ",a[x]
next x

do:until inkey$ <> ""
closeconsole
end



[Andy]

Egil,

I think this is a string array isn't it? with space for 100 entries, each 255 characters long.

Unless I've misunderstood all this time?

From the help file....

"An array is a collection of data of the same type. Arrays are referenced by their variable name and an index. Think of a box of cookies. Each row of cookies can be thought of as an array.  IWBASIC supports arrays up to five array dimensions. When using an array the index is enclosed in brackets '[ ]'.

DEF age[20]:INT
age[0] = 40
age[1] = 32"

Andy.

[Brian]

Andy,

That's just what I thought! I must admit, array dimensioning has always been a bit hazy for me, as well

Brian

[Egil]

I think this is a string array isn't it? with space for 100 entries, each 255 characters long.

That's corect Andy, but as soon as you in the example define a this way: a[0] = "0", a[0] becomes 1 byte long.
LEN(a) therefore never becomes larger than the number of bytes in your definition.
And when you call: rtlzeromemory(a,LEN(a)) the length of a is never larger than one.

Just print out the length of each member of the array to see for yourself.



Egil

[Brian]

Egil,

So in your reasoning, these should only contain one character, instead of the ten when you actually print them?

a[0] = "0000000000"
a[1] = "1111111111"
a[2] = "2222222222"
a[3] = "3333333333"
a[4] = "4444444444"

Brian (puzzled, or is that puddled)

Actually, _memset is quite interesting. Try it with: _memset(a,254,LEN(a))

[Egil]

Brian,

In your example each member has ten bytes, but Andy's example was:

Code Select Expand

a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"

Which set the length of each member to one byte. It is just a matter of counting characters in each string.

Egil

[Egil]

Brian,

try the  following code:

Code Select Expand


string a[100]

a[0] = "A"
a[1] = "B"
a[2] = "C"
a[3] = "D"
a[4] = "E"

openconsole
print
print
print "LEN[a] = ", len(a)
Print
print "Contents of a: " + a
print

do:until inkey$ <> ""
closeconsole
end


What I have tried to say is that whatever number of array members and whatever number of characters stored in each array member,
The lenght of variable a in above example becomes one. LEN only "sees" the first member of the array as long as the array index is missing.
And that is what happens in Andy's example when the calls rtlzeromemory(a,LEN(a)). The index to the array member is missing so only the first member is "zeroed".

Hope this makes things a little clearer...


Egil

[jalih]

What I have tried to say is that whatever number of array members and whatever number of characters stored in each array member,
The lenght of variable a in above example becomes one. LEN only "sees" the first member of the array as long as the array index is missing.
And that is what happens in Andy's example when the calls rtlzeromemory(a,LEN(a)). The index to the array member is missing so only the first member is "zeroed".

Hope this makes things a little clearer...

Egil

Egil is right, just substitute LEN() with SIZEOF() and it should work fine...

[billhsln]

An example of how I have always explained Arrays and the Dimensions:

Imagine a single page of a book and the spot where a single character is located, that character has a ROW and a COLUMN (2 dimensions), it also has a Page Number (3 dimensions), it is in a certain book (4 Dimensions), if this book is in a library,  then is exists on a certain shelf in that library (5 Dimensions), it exists on a specific book cabinet (?) (6 Dimensions), it exists in a specific library (7) dimensions, in a specific city (8 Dimensions), on a specific planet (9 Dimensions), in a galaxy (10 Dimensions)..   That should pretty much cover it.

Bill

[ckoehn]

There is a command called STRING$ in IWBASIC.  This will fill your array with 0, deleting previous content. 

a[0] = STRING$(254,0)
a[1] = STRING$(254,0)
a[2] = STRING$(254,0)
a[3] = STRING$(254,0)
a[4] = STRING$(254,0)

RtlZeroMemory(a[0],255)  seems like this should work also.

RtlZeroMemory(a, 255 * Arraysize)  This should work.

Code Select Expand

$include "windowssdk.inc"

int x
string a[100]

for x = 0 to 99
a[x] = str$(x)
next x

openconsole
print
print "Before zero memory"
print
for x = 0 to 99
    print "| ",x," = ",a[x],
next x

print
rtlzeromemory(a,255 * 45)   ' delete the first 45 items 0-44
'for x = 0 to 4
' rtlzeromemory(a[x],LEN(a[x]))
'next x

print
print "After zero memory"
print

for x = 0 to 99
    print "| ",x," = ",a[x],
next x

do:until inkey$ <> ""
closeconsole
end


I think this will work.

[Brian]

Egil, Bill and Jalih,

All well explained, thank you. And either of these two commands work, just don't know which one is the fastest (probably the CDECL):

DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
   _memset(a,0,SIZEOF(a))

   RtlZeroMemory(a,SIZEOF(a))

Brian

[Andy]

Brian,

I've been doing some timing testing, this is what my PC tells me:

No matter how many entries you have in an array, both rtlzeromemory and _memset remain constant.

By that I mean these two commands clear out the whole array - e.g. array "a" has a size of 1 million, but actually contains say 100k entries, or 500k entries - the time to clear array "a" remains the same irrespective of how many entries are actually in the array.

Now interestingly I found using a For/Next loop to clear the array works much quicker on smaller number of entries if you check for the very first empty string. You can then stop clearing the array as all subsequent ones will also be empty.

Code Select Expand

for x = 0 to 1000000
    if a[x] = "" then breakfor
    a[x] = ""
next x

See attached zero100k screenshot.

Timimgs get closer as the array fills up, around half full or a little more (See attached zero500k screenshot).

When you get to nearly full the For/Next method is slower (See attached zero900k screenshot).

Now another interesting thing is that on larger number of entries (around or over half full) the For/Next method still works quicker if we then remove checking for empty strings.

Code Select Expand

for x = 0 to 1000000
    'if a[x] = "" then breakfor   <---- commented out / removed.
    a[x] = ""
next x

So I think the answer is to keep track of how many entries you have in an array and use one of the two For/Next methods (See attached zeroNoCheck screenshot).

Obviously if you were randomly adding entries other than sequentially you would be better using one of the other two commands.

Here is the little program I used to compare timings:

Code Select Expand


$include "windowssdk.inc"
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)

int x
int ArraySize = 500000
string a[1000000]
uint t1,t2

openconsole

for x = 0 to ArraySize
    a[x] = str$(x)
next x
print
print "  Before For/Next = ''    (Array filled with ",ArraySize,"strings.)"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
t1 = gettickcount()
for x = 0 to 1000000
    if a[x] = "" then breakfor
    a[x] = ""
next x
t2 = gettickcount()
print
print "  Array empty"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
print
print "  For/Next = '' took ",t2-t1,"millisecs."
print "  --------------------------------------"

'---------------------------------------------

for x = 0 to ArraySize
    a[x] = str$(x)
next x
print
print "  Before _memset"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
t1 = gettickcount()
_memset(a,0,SIZEOF(a))
t2 = gettickcount()
print
print "  Array empty"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
print
print "  _memset took ",t2-t1,"millisecs."
print "  --------------------------------------"


'---------------------------------------------

for x = 0 to ArraySize
    a[x] = str$(x)
next x
print
print "  Before rtlzeromemory"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
t1 = gettickcount()
rtlzeromemory(a,sizeof(a))
t2 = gettickcount()
print
print "  Array empty"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
print
print "  rtlzeromemory took ",t2-t1,"millisecs."
print "  --------------------------------------"

print
print "  Press any key to exit..."
print

do:until inkey$ <> ""
closeconsole
end


You can change the value of variable "ArraySize" to choose how many entries you want to store and then clear.

Andy.

[Brian]

Andy,

That's a good little testing program. I ran it through several times, and RtlZeroMemory came up Top Trumps every time, even showing zero seconds in some runs, but most times about 16ms

Good stuff,

Brian

[Andy]

Brian,

Are you saying on your machine RtlZeroMemory was the quickest?, that's not the case on my PC.  True that my PC is no where as quick as yours - so is this down to processing speed I wonder?

I would have thought no matter how slow or fast the processor runs, the results should lead to the same conclusion that the For/Next (with / without) checking for empty strings should be the quickest??

Strange 

My results were based on just compiling the exe file THEN browsing to it and running it from there - i.e. not compile and execute - even so I get the same result.

Andy.

[jalih]

Now interestingly I found using a For/Next loop to clear the array works much quicker on smaller number of entries if you check for the very first empty string. You can then stop clearing the array as all subsequent ones will also be empty.

Code Select Expand

for x = 0 to 1000000
    if a[x] = "" then breakfor
    a[x] = ""
next x

I am pretty sure

Code Select Expand

a[x] = """ don't really actually clear the string, it just sets the first byte as zero. Using memset or RtlZeroMemory on other hand clears the entire string buffer.

[Brian]

Andy,
Ran the exe like you, at least 10 times, and consistently got 15-16ms with the RtlZeroMemory, always faster. Sometimes it came back with 0 milliseconds, but I didn't believe that!

Jalih,
Yes, I was wondering myself if it was clearing the variables properly

But, all in all, RtlZeroMemory seems to work for me

Brian

[Andy]

Sometimes it came back with 0 milliseconds, but I didn't believe that!

Brian,

I've amended the program and included a pointer "p1", the pointer is filled up the same as string array "a".

The pointer is then deleted and re-created, this seems to be the fastest on my PC.

I've also included Graham's precision timings shown in brackets e.g. "( 12.34 )" which are different to the gettickcount method.

The pointer sometimes returned 0 using the tick count method, but Graham's method showed 10.54 (See screenshot). A command / process obviously must take "some time" so it cannot be 0.

But still don't understand why rtlzeromemory is quicker for you than it is for me

Code Select Expand


$include "windowssdk.inc"
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)

def Freq,StartCount,StopCount:uint64
def TimingmSec,OverHead:double
def x1,x2,f:pointer

f = Freq: QueryPerformanceFrequency(f)
x1 = StartCount: QueryPerformanceCounter(x1)
' do nothing between calls ..
x2 = StopCount: QueryPerformanceCounter(x2)

OverHead = StopCount - StartCount

int x
int ArraySize = 100000
string a[1000000]
uint t1,t2

int P1size = 1000000
pointer P1
P1 = NEW(string,P1size)
settype P1,STRING

openconsole
print
print

for x = 0 to ArraySize
    #p1[x,0] = str$(x)
next x

print "  Before deleteing pointer           (Array filled with ",ArraySize,"strings.)"
print

for x = 0 to 1
    print "  Pointer string ",x,"   ",#p1[x,0]
next x

x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
delete p1
P1 = NEW(string,P1size)
settype P1,STRING
x2 = StopCount: QueryPerformanceCounter(x2)
t2 = gettickcount()

print
print "  Array empty"
print

for x = 0 to 1
    print "  Pointer string ",x,"   ",#p1[x,0]
next x

TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000

print
print "  Pointer took ",t2-t1,"millisecs. (",TimingmSec," )"
print "  --------------------------------------"

'---------------------------------------------


for x = 0 to ArraySize
    a[x] = str$(x)
next x
print
print "  Before For/Next = ''"   
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x

if ArraySize > 500000 'If number of array entries is greater than half of the array size.
   x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
for x = 0 to 1000000
a[x] = ""
next x
else
   x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
endif

x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
t2 = gettickcount()

print
print "  Array empty"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
print
print "  For/Next = '' took ",t2-t1,"millisecs. (",TimingmSec," )"
print "  --------------------------------------"

'---------------------------------------------

for x = 0 to ArraySize
    a[x] = str$(x)
next x
print
print "  Before rtlzeromemory"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
rtlzeromemory(a,sizeof(a))
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
t2 = gettickcount()
print
print "  Array empty"
print
for x = 0 to 1
    print "  string ",x," = ",a[x]
next x
print
print "  rtlzeromemory took ",t2-t1,"millisecs. (",TimingmSec," )"
print "  --------------------------------------"

print
print "  Press any key to exit..."
print

delete p1

do:until inkey$ <> ""
closeconsole
end


Andy.

[Andy]

I think the difference between mine and Brian's findings lies at the two computers architect.

In particular the FSB (buss speed from the CPU to the Northbride chip, then onto the ram and back).

The race must really be between creating a pointer, using it then deleteing it and finally re-creating it for use again AND the rtlzeromemory command.

So which is fastest? I think it's down to the computer architect - if it's older / slower pointers are the way for sure - if it's new, then it's a close race between them.

 

[Brian]

Well, surprisingly enough on my PC (after running the exe three times):

Pointer 2.43
For/Next 9.56
RtlZero 59.35

Now I am confused, but that's not unusual! Added a little text file of my CPU's capabilities, generated with Piriform's Speccy

Brian