Just been reading on MSDN about zeroing memory. Instead of RtlZeroMemory, they recommend using RtlSecureZeroMemory, as this call is guaranteed to work with all flavours of compilers
Thing is, it doesn't seem to be supported in our include files. I've tried a direct DECLARE IMPORT call in my code, but it flags an error. Can't find what DLL it is declared in, as well
Any ideas?
Brian
Quote from: Brian Pugh on November 09, 2017, 06:55:27 AM
Just been reading on MSDN about zeroing memory. Instead of RtlZeroMemory, they recommend using RtlSecureZeroMemory, as this call is guaranteed to work with all flavours of compilers
Thing is, it doesn't seem to be supported in our include files. I've tried a direct DECLARE IMPORT call in my code, but it flags an error. Can't find what DLL it is declared in, as well
Any ideas?
Brian
RtlSecureZeroMemory is inline function defined probably somewhere in Microsoft C compiler header files. You can't directly import it, you have to translate it from C (or asm?) to IWB.
If you are not storing passwords or some other sensitive information, then just using RtlZeroMemory is the best choice.
Thanks, Jalih,
I will stick with what I know, then!
Brian
rtlFillMemory is also available to us to fill memory with a byte other than zero if we desire....just saying
Am I missing something here....(probably knowing me)?
RtlZeroMemory which I've asked about before doesn't seem to clear out an array - see code:
$include "windowssdk.inc"
int x
string a[100]
a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"
openconsole
print
print "Before zero memory"
print
for x = 0 to 4
print "string ",x," = ",a[x]
next x
print
rtlzeromemory(a,LEN(a))
print
print "After zero memory"
print
for x = 0 to 4
print "string ",x," = ",a[x]
next x
do:until inkey$ <> ""
closeconsole
end
It only seems to blank out a[0] (zero), not the rest of the array values.
Andy.
Andy, best I can come up with at the moment is:
RtlZeroMemory(a[0],255*LEN(a[0]))
RtlZeroMemory(a[1],255*LEN(a[1]))
RtlZeroMemory(a[2],255*LEN(a[2]))
RtlZeroMemory(a[3],255*LEN(a[3]))
RtlZeroMemory(a[4],255*LEN(a[4]))
Don't really know if you need the 255, though - I will have another look
Brian
No, you don't need the 255. I also tried with _memset, which could be quicker:
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
_memset(a,0,LEN(a))
Brian (again)
Andy,
The way I see it you have declared an array of five string variables, each 1 byte long. Therefore your code won't work because LEN(a) becomes ONE byte long. You have to clear each member of the array. I know this has been discussed before, but do not remember if it was on this forum or somewhere else.
$include "windowssdk.inc"
int x
string a[100]
a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"
openconsole
print
print "Before zero memory"
print
for x = 0 to 4
print "string ",x," = ",a[x]
next x
print
'rtlzeromemory(a,LEN(a))
for x = 0 to 4
rtlzeromemory(a[x],LEN(a[x]))
next x
print
print "After zero memory"
print
for x = 0 to 4
print "string ",x," = ",a[x]
next x
do:until inkey$ <> ""
closeconsole
end
Egil,
I think this is a string array isn't it? with space for 100 entries, each 255 characters long.
Unless I've misunderstood all this time?
From the help file....
"An array is a collection of data of the same type. Arrays are referenced by their variable name and an index. Think of a box of cookies. Each row of cookies can be thought of as an array. IWBASIC supports arrays up to five array dimensions. When using an array the index is enclosed in brackets '[ ]'.
DEF age[20]:INT
age[0] = 40
age[1] = 32"
Andy.
Andy,
That's just what I thought! I must admit, array dimensioning has always been a bit hazy for me, as well
Brian
QuoteI think this is a string array isn't it? with space for 100 entries, each 255 characters long.
That's corect Andy, but as soon as you in the example define a this way:
a[0] = "0", a[0] becomes 1 byte long.
LEN(a) therefore never becomes larger than the number of bytes in your definition.
And when you call:
rtlzeromemory(a,LEN(a)) the length of a is never larger than one.
Just print out the length of each member of the array to see for yourself.
Egil
Egil,
So in your reasoning, these should only contain one character, instead of the ten when you actually print them?
a[0] = "0000000000"
a[1] = "1111111111"
a[2] = "2222222222"
a[3] = "3333333333"
a[4] = "4444444444"
Brian (puzzled, or is that puddled)
Actually, _memset is quite interesting. Try it with: _memset(a,254,LEN(a))
Brian,
In your example each member has ten bytes, but Andy's example was:a[0] = "0"
a[1] = "1"
a[2] = "2"
a[3] = "3"
a[4] = "4"
Which set the length of each member to one byte. It is just a matter of counting characters in each string.
Egil
Brian,
try the following code:
string a[100]
a[0] = "A"
a[1] = "B"
a[2] = "C"
a[3] = "D"
a[4] = "E"
openconsole
print
print
print "LEN[a] = ", len(a)
Print
print "Contents of a: " + a
print
do:until inkey$ <> ""
closeconsole
end
What I have tried to say is that whatever number of array members and whatever number of characters stored in each array member,
The lenght of variable a in above example becomes one. LEN only "sees" the first member of the array as long as the array index is missing.
And that is what happens in Andy's example when the calls rtlzeromemory(a,LEN(a)). The index to the array member is missing so only the first member is "zeroed".
Hope this makes things a little clearer...
Egil
Quote from: Egil on November 10, 2017, 08:10:39 AM
What I have tried to say is that whatever number of array members and whatever number of characters stored in each array member,
The lenght of variable a in above example becomes one. LEN only "sees" the first member of the array as long as the array index is missing.
And that is what happens in Andy's example when the calls rtlzeromemory(a,LEN(a)). The index to the array member is missing so only the first member is "zeroed".
Hope this makes things a little clearer...
Egil
Egil is right, just substitute LEN() with SIZEOF() and it should work fine...
An example of how I have always explained Arrays and the Dimensions:
Imagine a single page of a book and the spot where a single character is located, that character has a ROW and a COLUMN (2 dimensions), it also has a Page Number (3 dimensions), it is in a certain book (4 Dimensions), if this book is in a library, then is exists on a certain shelf in that library (5 Dimensions), it exists on a specific book cabinet (?) (6 Dimensions), it exists in a specific library (7) dimensions, in a specific city (8 Dimensions), on a specific planet (9 Dimensions), in a galaxy (10 Dimensions).. That should pretty much cover it.
Bill
There is a command called STRING$ in IWBASIC. This will fill your array with 0, deleting previous content.
a[0] = STRING$(254,0)
a[1] = STRING$(254,0)
a[2] = STRING$(254,0)
a[3] = STRING$(254,0)
a[4] = STRING$(254,0)
RtlZeroMemory(a[0],255) seems like this should work also.
RtlZeroMemory(a, 255 * Arraysize) This should work.
$include "windowssdk.inc"
int x
string a[100]
for x = 0 to 99
a[x] = str$(x)
next x
openconsole
print
print "Before zero memory"
print
for x = 0 to 99
print "| ",x," = ",a[x],
next x
print
rtlzeromemory(a,255 * 45) ' delete the first 45 items 0-44
'for x = 0 to 4
' rtlzeromemory(a[x],LEN(a[x]))
'next x
print
print "After zero memory"
print
for x = 0 to 99
print "| ",x," = ",a[x],
next x
do:until inkey$ <> ""
closeconsole
end
I think this will work.
Egil, Bill and Jalih,
All well explained, thank you. And either of these two commands work, just don't know which one is the fastest (probably the CDECL):
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
_memset(a,0,SIZEOF(a))
RtlZeroMemory(a,SIZEOF(a))
Brian
Brian,
I've been doing some timing testing, this is what my PC tells me:
No matter how many entries you have in an array, both rtlzeromemory and _memset remain constant.
By that I mean these two commands clear out the whole array - e.g. array "a" has a size of 1 million, but actually contains say 100k entries, or 500k entries - the time to clear array "a" remains the same irrespective of how many entries are actually in the array.
Now interestingly I found using a For/Next loop to clear the array works much quicker on smaller number of entries if you check for the very first empty string. You can then stop clearing the array as all subsequent ones will also be empty.
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
See attached zero100k screenshot.
Timimgs get closer as the array fills up, around half full or a little more (See attached zero500k screenshot).
When you get to nearly full the For/Next method is slower (See attached zero900k screenshot).
Now another interesting thing is that on larger number of entries (around or over half full) the For/Next method still works quicker if we then remove checking for empty strings.
for x = 0 to 1000000
'if a[x] = "" then breakfor <---- commented out / removed.
a[x] = ""
next x
So I think the answer is to keep track of how many entries you have in an array and use one of the two For/Next methods (See attached zeroNoCheck screenshot).
Obviously if you were randomly adding entries other than sequentially you would be better using one of the other two commands.
Here is the little program I used to compare timings:
$include "windowssdk.inc"
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
int x
int ArraySize = 500000
string a[1000000]
uint t1,t2
openconsole
for x = 0 to ArraySize
a[x] = str$(x)
next x
print
print " Before For/Next = '' (Array filled with ",ArraySize,"strings.)"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
t1 = gettickcount()
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
print
print " For/Next = '' took ",t2-t1,"millisecs."
print " --------------------------------------"
'---------------------------------------------
for x = 0 to ArraySize
a[x] = str$(x)
next x
print
print " Before _memset"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
t1 = gettickcount()
_memset(a,0,SIZEOF(a))
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
print
print " _memset took ",t2-t1,"millisecs."
print " --------------------------------------"
'---------------------------------------------
for x = 0 to ArraySize
a[x] = str$(x)
next x
print
print " Before rtlzeromemory"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
t1 = gettickcount()
rtlzeromemory(a,sizeof(a))
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
print
print " rtlzeromemory took ",t2-t1,"millisecs."
print " --------------------------------------"
print
print " Press any key to exit..."
print
do:until inkey$ <> ""
closeconsole
end
You can change the value of variable "ArraySize" to choose how many entries you want to store and then clear.
Andy.
Andy,
That's a good little testing program. I ran it through several times, and RtlZeroMemory came up Top Trumps every time, even showing zero seconds in some runs, but most times about 16ms
Good stuff,
Brian
Brian,
Are you saying on your machine RtlZeroMemory was the quickest?, that's not the case on my PC. True that my PC is no where as quick as yours - so is this down to processing speed I wonder?
I would have thought no matter how slow or fast the processor runs, the results should lead to the same conclusion that the For/Next (with / without) checking for empty strings should be the quickest??
Strange ???
My results were based on just compiling the exe file THEN browsing to it and running it from there - i.e. not compile and execute - even so I get the same result.
Andy.
Quote from: Andy on November 10, 2017, 11:55:07 PM
Now interestingly I found using a For/Next loop to clear the array works much quicker on smaller number of entries if you check for the very first empty string. You can then stop clearing the array as all subsequent ones will also be empty.
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
I am pretty sure
a[x] = """ don't really actually clear the string, it just sets the first byte as zero. Using memset or RtlZeroMemory on other hand clears the entire string buffer.
Andy,
Ran the exe like you, at least 10 times, and consistently got 15-16ms with the RtlZeroMemory, always faster. Sometimes it came back with 0 milliseconds, but I didn't believe that!
Jalih,
Yes, I was wondering myself if it was clearing the variables properly
But, all in all, RtlZeroMemory seems to work for me
Brian
Quote from: Brian Pugh on November 11, 2017, 08:45:03 AM
Sometimes it came back with 0 milliseconds, but I didn't believe that!
Brian,
I've amended the program and included a pointer "p1", the pointer is filled up the same as string array "a".
The pointer is then deleted and re-created, this seems to be the fastest on my PC.
I've also included Graham's precision timings shown in brackets e.g. "( 12.34 )" which are different to the gettickcount method.
The pointer sometimes returned 0 using the tick count method, but Graham's method showed 10.54 (See screenshot). A command / process obviously must take "some time" so it cannot be 0.But still don't understand why rtlzeromemory is quicker for you than it is for me ???
$include "windowssdk.inc"
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
def Freq,StartCount,StopCount:uint64
def TimingmSec,OverHead:double
def x1,x2,f:pointer
f = Freq: QueryPerformanceFrequency(f)
x1 = StartCount: QueryPerformanceCounter(x1)
' do nothing between calls ..
x2 = StopCount: QueryPerformanceCounter(x2)
OverHead = StopCount - StartCount
int x
int ArraySize = 100000
string a[1000000]
uint t1,t2
int P1size = 1000000
pointer P1
P1 = NEW(string,P1size)
settype P1,STRING
openconsole
print
print
for x = 0 to ArraySize
#p1[x,0] = str$(x)
next x
print " Before deleteing pointer (Array filled with ",ArraySize,"strings.)"
print
for x = 0 to 1
print " Pointer string ",x," ",#p1[x,0]
next x
x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
delete p1
P1 = NEW(string,P1size)
settype P1,STRING
x2 = StopCount: QueryPerformanceCounter(x2)
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " Pointer string ",x," ",#p1[x,0]
next x
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
print
print " Pointer took ",t2-t1,"millisecs. (",TimingmSec," )"
print " --------------------------------------"
'---------------------------------------------
for x = 0 to ArraySize
a[x] = str$(x)
next x
print
print " Before For/Next = ''"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
if ArraySize > 500000 'If number of array entries is greater than half of the array size.
x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
for x = 0 to 1000000
a[x] = ""
next x
else
x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
endif
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
print
print " For/Next = '' took ",t2-t1,"millisecs. (",TimingmSec," )"
print " --------------------------------------"
'---------------------------------------------
for x = 0 to ArraySize
a[x] = str$(x)
next x
print
print " Before rtlzeromemory"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
x1 = StartCount: QueryPerformanceCounter(x1)
t1 = gettickcount()
rtlzeromemory(a,sizeof(a))
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
t2 = gettickcount()
print
print " Array empty"
print
for x = 0 to 1
print " string ",x," = ",a[x]
next x
print
print " rtlzeromemory took ",t2-t1,"millisecs. (",TimingmSec," )"
print " --------------------------------------"
print
print " Press any key to exit..."
print
delete p1
do:until inkey$ <> ""
closeconsole
end
Andy.
I think the difference between mine and Brian's findings lies at the two computers architect.
In particular the FSB (buss speed from the CPU to the Northbride chip, then onto the ram and back).
The race must really be between creating a pointer, using it then deleteing it and finally re-creating it for use again AND the rtlzeromemory command.
So which is fastest? I think it's down to the computer architect - if it's older / slower pointers are the way for sure - if it's new, then it's a close race between them.
Well, surprisingly enough on my PC (after running the exe three times):
Pointer 2.43
For/Next 9.56
RtlZero 59.35
Now I am confused, but that's not unusual! Added a little text file of my CPU's capabilities, generated with Piriform's Speccy
Brian
Brian,
Thanks for your PC spec, much faster than my AMD setup.
Anyway, I was curious and tried it on Anne's laptop (much faster machine), the results were the same - pointers being by far the quickest.
I've amended the program to test the three methods 10 times and display the slowest, fastest and average time for each - you can clearly see rtlzeromemory is the slowest as you have to wait a second or two before it returns it's results.
Every time pointers win outright no matter on the spec on the machine.
$include "windowssdk.inc"
DECLARE CDECL EXTERN _memset(dst AS POINTER,value AS CHAR,count AS UINT)
def Freq,StartCount,StopCount:uint64
def TimingmSec,OverHead,Average,Fastest,Slowest:float
def x1,x2,f:pointer
setprecision(4)
int x
int ArraySize = 100000
string a[1000000]
int loops = 0
int maxloops = 10
int P1size = 1000000
pointer P1
P1 = NEW(string,P1size)
settype P1,STRING
Slowest = 0.00
Fastest = 99999999.99
f = Freq: QueryPerformanceFrequency(f)
x1 = StartCount: QueryPerformanceCounter(x1)
for x = 1 to 1000
next x
x2 = StopCount: QueryPerformanceCounter(x2)
OverHead = StopCount - StartCount
openconsole
print
print " Array filled with ",ltrim$(USING("#,###,###",ArraySize))," string entries, testing ",maxloops,"times."
print
do
loops ++
for x = 0 to ArraySize
#p1[x,0] = str$(x)
next x
x1 = StartCount: QueryPerformanceCounter(x1)
delete p1
P1 = NEW(string,P1size)
settype P1,STRING
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
if TimingmSec > Slowest
Slowest = TimingmSec
endif
if TimingmSec < Fastest
Fastest = TimingmSec
endif
Average = Average + TimingmSec
until loops = maxloops
print
print " Pointer slowest (",Slowest," ) millisecs"
print " Pointer fastest (",Fastest," ) millisecs"
print
print " Pointer took on average (",Average / maxloops," ) millisecs"
print " ----------------------------------------------------------------------"
'---------------------------------------------
loops = 0
Slowest = 0.00
Fastest = 99999999.99
Average = 0.00
do
loops ++
for x = 0 to ArraySize
a[x] = str$(x)
next x
if ArraySize > 500000 'If number of array entries is greater than half of the array size.
x1 = StartCount: QueryPerformanceCounter(x1)
for x = 0 to 1000000
a[x] = ""
next x
else
x1 = StartCount: QueryPerformanceCounter(x1)
for x = 0 to 1000000
if a[x] = "" then breakfor
a[x] = ""
next x
endif
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
if TimingmSec > Slowest
Slowest = TimingmSec
endif
if TimingmSec < Fastest
Fastest = TimingmSec
endif
Average = Average + TimingmSec
until loops = maxloops
print
print " For/Next slowest (",Slowest," ) millisecs"
print " For/Next fastest (",Fastest," ) millisecs"
print
print " For/Next took on average (",Average / maxloops," ) millisecs"
print " ----------------------------------------------------------------------"
'---------------------------------------------
loops = 0
Slowest = 0.00
Fastest = 99999999.99
Average = 0.00
do
loops ++
for x = 0 to ArraySize
a[x] = str$(x)
next x
x1 = StartCount: QueryPerformanceCounter(x1)
rtlzeromemory(a,sizeof(a))
x2 = StopCount: QueryPerformanceCounter(x2)
TimingmSec = (StopCount - StartCount - OverHead) / Freq * 1000
if TimingmSec > Slowest
Slowest = TimingmSec
endif
if TimingmSec < Fastest
Fastest = TimingmSec
endif
Average = Average + TimingmSec
until loops = maxloops
print
print " rtlzeromemory slowest (",Slowest," ) millisecs"
print " rtlzeromemory fastest (",Fastest," ) millisecs"
print
print " rtlzeromemory took on average (",Average / maxloops," ) millisecs"
print " ----------------------------------------------------------------------"
print
print
print " Press any key to exit..."
print
delete p1
do:until inkey$ <> ""
closeconsole
end
BTW: Remember recently we were looking at sorting and Clint came up with his bubblesort4optimised code which returned 0 millisecs to sort? well I've since gone back to it and timed it again using Grahams code - it actually takes around 1.019 millisecs - but still fantastic.
Fasecero's won the race at 0.026!!!
Quote from: Andy
Anyway, I was curious and tried it on Anne's laptop (much faster machine), the results were the same - pointers being by far the quickest.
I've amended the program to test the three methods 10 times and display the slowest, fastest and average time for each - you can clearly see rtlzeromemory is the slowest...
Note, that security wise the three methods are not the same. Releasing memory thru pointer doesn`t zero the memory content. Same is when assigning string as empty inside a loop.
Good stuff, Andy,
As before, pointers won out at an average of 2.43, for/next at 9.11, and rtlzero at 15.85. Great timing code, as well, I can see a use for that in testing code
Jalih, as Bill pointed out to me, you would only be bothered about securely erasing all the variable if it contained sensitive code, like passwords, or banking codes, perhaps
Brian
Thanks Brian, you can call me Sherlock in future if you wish lol!
Jalih - It's a good point you make too, for the most part we are just using arrays for general ever day use, but something to note if you have sensitive data.
Andy (aka Sherlock Holmes). ;D