December 01, 2024, 10:26:48 AM

News:

IonicWind Snippit Manager 2.xx Released!  Install it on a memory stick and take it with you!  With or without IWBasic!


Speeding up compile time (NT+NTFS)

Started by sapero, January 02, 2008, 06:54:41 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

sapero

January 02, 2008, 06:54:41 AM Last Edit: January 02, 2008, 12:06:45 PM by sapero
Recentry i'm writing large projects with multiple source files, every file includes many more than windows.inc, so the time lost for rebuilding any small modification is very big.
Here comes a Turbo-Booster for the parser - it will not allow to parse if the source file has been not re-saved.

How it works:
1. Instalation: simply run the program, it will create executable image hook in HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options.
Now every time someone will run acparse.exe - the system will run the hook application with hooked application path in first parameter in the command line.

2. Uninstalling: run the program again - it will remove hook from the registry.

3. When Aurora IDE is compiling (parsing), the hook first tries to read a hidden stream from input file (.src).
If it fails or previously parsed output (.asm) does not exists - the source file will be parsed, so the hook is temporary disabled and acparse is executed. When parsing is done, a hidden stream is created in source file.
Else current parsing stage is completed without parsing, and you saved some seconds.

This tool does not check what files are #include'd and modified, when you modify a class or structure in include file, all source files with references to modified object should be recompilled. To force a file to be recompilled - just save it from the IDE, or delete .asm file with same name.
When you save the file - Aurora probably uses CREATE_ALWAYS flag that first deletes any existing file (also streams).

File streams References:
How To Use NTFS Alternate Data Streams
NTFS Multiple Data Streams

So, extract acparseex.exe from attached archive to any location, run it, open multi file Aurora project (create first a backup for any case), then compile your project two times (holding a stoper :) )

Added version for emergence (ebparseex).

ExMember001

January 02, 2008, 02:29:36 PM #1 Last Edit: January 02, 2008, 02:44:53 PM by KrYpT
great work :)
unfortunally it doesnt work on Vista...
the key cant be create in HKLM... maybe a permission problem.

Ionic Wind Support Team

Yep, you have to use HKCU on Vista instead of HKLM. 

Ionic Wind Support Team

sapero

February 07, 2009, 12:43:47 PM #3 Last Edit: February 08, 2009, 05:34:40 AM by sapero
I have extended this util. If you have vista or above, it will create the debugger subkey with access granted to all users, so be sure to run it with administrative privileges for the first registration.  If the key below already exists, please delete it first.HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\acparse.exe
After you register it, it should work even on standard user accounts, because the key is allowed to be modified from any account.

New in this release:
  • try and catch keywords added, to handle exceptions. To use it, you'll need to copy the attached gp.lib to /lib directory.
  • wstrings are properly aligned (to word boundary)
  • you can use escape sequences in wide strings: \n \t \\ \" \x00 \x0000. The \x0000 is to insert two byte wide character
  • diacritics are now supported in wstring's
  • If you compile include file, the asm output will be created in system defined TEMP directory, and the file will be compiled two times, second time with UNICODE macro defined. You'll see an error in the output window:
nasm: fatal: unable to open input file
    Normally you do not compile include files, but if your translating some C headers with ansi/unicode declares then you can check for errors without creating temporary source file and incluging the include.[/li]
  • Unknown or incomplete escape sequences are reported as an error, for example L"abc\d".

Example:import int MessageBoxW(int q,pointer w,pointer e,int r);
import int MessageBoxA(int q,pointer w,pointer e,int r);

sub main()
{
unsigned int code, address;
string *info;

try
{
MessageBoxW(0, L"A\n\t\x0020ÃÆ'Ã,¤ÃÆ'Ã,¶ÃÆ'Ã,¼ÃÆ'Ã...¸ąść\"\\", NULL, 0);
int a = 1/0;
}
catch(&code, &address, &info)
{
MessageBoxA(0, using("& (&) at &", info, hex$(code), hex$(address)),"",0);
}

try
{
int *p = NULL;
*p = 0;
}
catch
{
MessageBoxA(0, "failed to write value to NULL pointer", "", 0);
}

try
{
// something
}
// catch is optional
}


This will also compile correctly:try *(int)p=1;
Notes about try:
* you cannot use RETURN inside TRY {} yet.
* you can nest it.

Unfortunatelly, if you get compile errors, the line number shown in output window may be wrong. This is because the original source file is modified before compiling:
* if you use try/catch, 4 lines with declares are added to the top of source file.
* if you use wstrings (quoted strings L""), 4 additional lines are added for each wstring to the top of source file
.
Fixed, added a temporary #include.

The above code just before compilation:extern int __try();
extern int __leave();
extern int __catch(opt pointer q,opt pointer w,opt pointer e);
#use "gp.lib"
wstring _STR000000;
#asm
%define $_STR000000 _STR000000
#endasm
import int MessageBoxW(int q,pointer w,pointer e,int r);
import int MessageBoxA(int q,pointer w,pointer e,int r);

sub main()
{
unsigned int code, address;
string *info;

if __try()
{
MessageBoxW(0, _STR000000, NULL, 0);
int a = 1/0;
}
if __catch(&code, &address, &info)
{
MessageBoxA(0, using("& (&) at &", info, hex$(code), hex$(address)));
}

if __try()
{
int *p = 0;
*p = 0;
}
if __catch()
{
MessageBoxA(0, "failed to put value at NULL address", "", 0);
}
if __try()
{
// something
} __leave();
// catch is optional
}
#asm
segment .data
align 2
_STR000000: dw 65,13,10,9,0x0020,228,246,252,223,261,347,263,'"','\',0
%undef $_STR000000
%macro $_STR000000 1
%endm
#endasm

Where first three lines declare the exception handling functions, next line includes exception library.
Next is a trick which creates a initialized wstring variable, and at the bottom is next trick to hide the wstring variable from being allocated in .bss section.
The _STR000000 array is created from multiple calls to MultiByteToWideChar, for every single character.

It was tested on XP, and I do not think it will work on Vista, because on XP the original parser creates temporay files ($$$lib.link) always in the /bin directory, and Aurora IDE will not load $$$lib.link from other directory. Later I'll see how it works on Vista.

sapero

The file has been updated. Problem with line number has been solved. I've put all pre-declares into a temporary include file, so the source file after modification is:
#include "acparseexext.inc" import int MessageBoxW(int q,pointer w,pointer e,int r);
...
// post defines here
The temporary include file is deleted after the original parser quits. It is good to know we can put multiple preprocessor directives in single line :)
This version installs also process hook for nasm, to skip assembling unmodified asm files.

Lets do some benchmarks: I have a project with 45 source files. Each file includes windows.inc and other headers.
To compile all files, it needs 171 seconds.
To recompile the project with one modified source, it still needs 171 seconds without this tool, and only 9 seconds with it.

The version for Vista is not finished yet.

sapero

June 12, 2009, 03:57:13 PM #5 Last Edit: June 13, 2009, 03:07:41 AM by sapero
I have created a mini version that resides in system tray, and uses only dll injection instead the registry and debugging.
There is a dll which at process attach time installs hooks at ShellExecute and CreateProcess.

The ShellExecute hook enforces the "valid" working directory for programms started from the IDE (Aurora and Emergence).
The CreateProcess hook does same as the ShellExecute hook, but for debugged programms. It can also "fake" the call to CreateProcess if the current file to be compilled has been not modified. If the application to be excuted is acparse or ebparse, it checks first if the source file (src, eba) contains a stream (NTFS only). If this fails, the compiller will be executed, else the steam will be extracted into $$$lib.link file, and two fake handles (manual reset events) will be returned in PROCESS_INFORMATION...

It hooks also the nasm and gorc tools, so the saved time is maximal. It does not hook the linker, because if you update a library in another directory, the change should be reflected at next compilation.
Because Aurora and Ebasic parsers still write to the Program Files (installation) directory (creating the $$$lib.link file) I have used this location to copy from, or extract the proviously saved libs list.

Tested under XP SP3. The included dll will not load if 7 first bytes in the two hooked api's are not supported.
With this plugin, the compile time is much better than with the debugger version: you save time lost while launching a debugger.

Please note: this tool does not detect modified include files. If you modify the size or add a variable to class definition in include file, you'll need to re-save all source files in order to make this change visible. If your program is crashing after such modification, just re-save all source files, or delete all .a or .asm files and compile again.

EDIT: fixed invalid operator in Compile function, and removed reference to gp.inc.params->dwCreationFlags & ~CREATE_SUSPENDED

sapero

June 15, 2009, 01:17:07 PM #6 Last Edit: June 17, 2009, 04:37:27 PM by sapero
I have added parser extensions from my previous project to this tool, and two days searched why ShellExecute is crashing (the IDE too). Only the Application Verifier from Microsoft helped me to find all serious bugs, and Aurora stopped with crashing :D
So what's new:

* try/catch can be used without IF. To use it, copy gp.lib to your /aurora/libs directory.try
{
int *p = NULL;
*p = 1;
}
catch // is optional
{
print("EXCEPTION");
}


* NEW not requiring parenthesespointer x = new int;
pointer y = new word[5];

* Braces are allowed after a colon and semicolonswitch (a)
{
case 1:
{
// something
}
}

print("brace demo start");
{
// something
}
print("brace demo end");

*typedef allowed without #typedef MYTYPE byte

* #define name value replaces all name tokens with value#define MsgBox MessageBox
MsgBox(0,"body","caption");

#define SEP ," ",
print(a SEP b SEP c SEP "\n");


* additional escape sequence: \nnn (up to 3 digits, in the range 0-255)print("escape1: \656a \0091");

* full support for escape sequences in unicode strings, plus support for diacritical charactersL"abc\09def\x09ąćÃÆ'Ã,¶ÃÆ'Ã,¼ÃÆ'Ã,¤ÃÆ'Ã...¸"

* extended string handling:print("this " "is"
" a string");


* console applications (if launched from Aurora/Emergence IDE) are started from a .bat file, so the window will not disappear until you press a key (idea taken from Visual Studio).
* #region and #endregion tool added.

Before the Aurora parser gets executed, this tool searches for supported extensions. If at least one is found, a temporary source file is created, where all the extensions are translated to parser friendly code. Now we need to tell to the parser to parse the temporary file instead the original one. This is done by additional dll injection - adevFileRedirector.dll. This dll is hooking the CreateFile api. If the file to be opened is your original source file, the path is changed to the temporary file.

In demo_extensions.src you'll find probably all the additional tools.

sapero

June 17, 2009, 12:26:18 PM #7 Last Edit: June 17, 2009, 04:50:21 PM by sapero
Updated to v3.
* You can return or break from any level of nested try block. The required number of __leave() calls is added automatically (does not apply to raw __try function)
sub testing()
{
try
{
return;
}
}
The code above is converted tosub testing()
{
if __try()
{
__leave();return;
}
__leave();
}

* You can break from do-until loop. do {if (error) break;} until !run;
* Continue statement added, can be used inside FOR, DO and WHILE loops.
QuoteThe continue statement passes control to the next iteration of the nearest enclosing do, for, or while statement in which it appears, bypassing any remaining statements in the do, for, or while statement body.
for (a=0; a<=6; a++)
{
if (a==3) continue; // do not print 3; jump to "next a"
print(a);
}

* The try, catch and throw statements are now styled in the scintilla control.
* To view the file(s) that is generated by this extension, just force a syntax error and compile. The files are hidden - one .ext and one .inc

Bug: #ifdef is recognized by this extension only for the UNICODE keyword. If you #define UNICODE in source file, it will be catched by this extension, and all referenced #if's and TEXT macros will be handled correctly.
For Ebasic, the language extensions are not implemented, just the compile speedup.

EDIT - added the missing support for .exports files. When nasm finishes, the optional .exports file is copied to a stream in the 'owning' .asm file, so if you compile again and the .src and .asm files were not modified, the .exports file will be extracted from the stream, and your binary (exe/dll) will not miss any exported symbols.
A small bug fixed - 'break' after 'case' (after a colon) generated unnecessary braces.