March 29, 2024, 01:23:06 AM

News:

IonicWind Snippit Manager 2.xx Released!  Install it on a memory stick and take it with you!  With or without IWBasic!


A sorting mystery.

Started by Egil, June 19, 2016, 07:11:09 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Egil

Hi,

After solving the directory selection problem yesterday ( http://www.ionicwind.com/forums/index.php?topic=5886.msg43511#msg43511 ), I decided to include that code in a program to select and make a list of directories containing sound recordings, and save the results in a textfile to make it easier to find a particular recording.

All the recordings are stored in subdirectories, where Artistname and Title are used as name of each subdirectory. And the length of each such name/title combination varies mainly from approximately 25 to 60 characters. One particular name even had 84 charaters.

The code worked as expected, but when I tried it on a directory containing a lot of audio books, I ran into a mystery problem. Probably because of the great number of subdirectories.

The different directories contain from around 100 to more than 2000 subdirectories. All of them listed in alphabetical order when Windows Explorer is used to select them, and that is also how they appear in the file produced with my code. But when the number of subdirectories becomes higher, the lists loose sorting, and the list is presented in a random manner. This also happen if the list is printed on a console screen. I have no idea why, but suspect it is because of the way Windows use buffers.

I think the best way to get around this problem, is to sort the data before it is written to the file. An internet search came up with numerous sorting algorithms, all claiming to be the fastest and best. So I'm rather confused at the moment. Maybe some of you guys can suggest what is the best way to do this?



Regards,
Egil.



PS: Here is the code used for testing.

'
'----------------------------------------------------------------------------------------
' ListDir.cba
' Lists the content of selected directory and write this to a file called "0DIR.TXT".
' Returns number of items written to the file.
' File "0DIR.TXT" is not counted and is not included in the list.
'----------------------------------------------------------------------------------------
'
AUTODEFINE "OFF"

DECLARE ListDir(mydir:string)
DECLARE GETDIR()
def path:string
def num:int

OPENCONSOLE

path = GETDIR()
num = ListDir(path)

select num
case -2
print "No Directory selected - Operation Cancelled"
num = 0
endselect

if num > 0 then print "Found:",num :'Do not print negative numbers

print:print:print "Press any key to exit"
do:until inkey$<>""
closeconsole
end


SUB ListDir(mydir:string)
'----------------------------------------------------------------------------------------
' Lists the content of selected directory and write this to a file called "0DIR.TXT".
' Pass full path to directory when calling.
' Returns number of items written to the file.
' File "0DIR.TXT" is not counted and is not included in the list.
'----------------------------------------------------------------------------------------
def ofile:file
def cnt,dirhandle:int
def name:string
cnt = 0

IF mydir <> ""

' IF(OPENFILE(ofile,mydir+"\0DIR.TXT","W") = 0) :' USE THIS LINE IN FINAL CODE
IF(OPENFILE(ofile,"0DIR.TXT","W") = 0) :' USE THIS LINE FOR TESTING ONLY

dirhandle = FINDOPEN(mydir+"\*.*")
DO
name = FINDNEXT(dirhandle)
if (name <> ".") & (name <> "..") & (name <> "0DIR.TXT")
cnt = cnt + 1
WRITE(ofile,name)
endif
UNTIL name = ""
FINDCLOSE dirhandle
CLOSEFILE ofile
ENDIF
ELSE
RETURN -2 :'  Selection was cancelled - NO directory was passed
ENDIF
RETURN cnt-1


SUB GETDIR()
'----------------------------------------------------------------------------------------
' Select directory and return full path as string.
' CONSOLE MODE VERSION (Does NOT work in Windows mode!!!!!!!!)
'----------------------------------------------------------------------------------------

def BIF_DONTGOBELOWDOMAIN, BIF_RETURNONLYFSDIRS:int
def BIF_BROWSEFORCOMPUTERS, BIF_BROWSEFORPRINTERS:int

CONST BIF_DONTGOBELOWDOMAIN = &H2
const BIF_RETURNONLYFSDIRS=&H1
const BIF_BROWSEFORCOMPUTERS=&H1000
const BIF_BROWSEFORPRINTERS=&H2000

def pidl:int
def buffer:string

Type BROWSEINFO
    def hWnd:int
    def root:int
    def folder:string
    def title:string
    def flags:int
    def callback:int
    def lparam:int
    def iimage:int
EndType

DECLARE "shell32",SHBrowseForFolder(lpbi:BROWSEINFO),int
declare "shell32",SHGetPathFromIDList(pidl:int,pszPath:string)
DECLARE "ole32",CoTaskMemFree(pidl:int)

def bi:BROWSEINFO
bi.hWnd=0 :'Set to zero in console mode
bi.root=0
bi.folder=buffer
bi.title="Choose a Folder.."
bi.flags=BIF_RETURNONLYFSDIRS

pidl=SHBrowseForFolder(bi)
if pidl<>0
SHGetPathFromIDList(pidl,buffer)
CoTaskMemFree(pidl)
endif

RETURN buffer


Support Amateur Radio  -  Have a ham  for dinner!

Egil

A search on the forum came up with this EB example: http://www.ionicwind.com/forums/index.php?topic=3129.msg25581#msg25581 .

Bill's code does more or less the same as I am trying to do, but when running his code on one of my directories, the resulting file is sorted exactly the same way as with my CB code.


Egil

Support Amateur Radio  -  Have a ham  for dinner!

Andy

June 19, 2016, 11:53:14 PM #2 Last Edit: June 20, 2016, 12:15:13 AM by Andy
Egil,

How about starting with the basics, here is the IWB example program to list files and folders on your C: drive:

DEF count:INT

OPENCONSOLE
'recursively print all directories
count = printdir("c:\\")
PRINT count,"Files"
waitcon

END


SUB printdir(path:STRING),INT
DEF count:INT
DEF dir,attrib=0:INT
DEF filename:STRING
DEF fullname:STRING
count = 0
dir = FINDOPEN(path + "*.*")
IF(dir)
DO
filename = FINDNEXT(dir,attrib)
IF len(filename)
IF attrib & @FILE_DIRECTORY
IF(filename <> ".") & (filename <> "..")
PRINT path + "[" + filename + "]"
fullname = path + filename + "\\"
count = count + printdir(fullname)
ENDIF
ELSE
                                          count = count + 1
PRINT path + filename
ENDIF
ENDIF
'the exit case is when there are no more entries
'in the current directory
UNTIL filename = ""
FINDCLOSE dir
ENDIF
RETURN count
ENDSUB


I amended this code and wrote every entry to a file, and it returned a list of over 151 thousand files / folders.

Try something like this in CB first and see how you get on.

Andy.

Day after day, day after day, we struck nor breath nor motion, as idle as a painted ship upon a painted ocean.

Egil

Hi Andy,

That was the first thing I tried whenI discovvered these resulst. But I only check for one single directory at a time.
When the number of files/subdirs in a directory becomes high, this strange thing occours.

The CB equivalent to your printdir sub is:


SUB ListDir(mydir:string)
def cnt,dirhandle:int
def name:string
cnt = 0

IF mydir <> ""
dirhandle = FINDOPEN(mydir+"\*.*")
DO
name = FINDNEXT(dirhandle)
cnt = cnt + 1
PRINT name
UNTIL name = ""
FINDCLOSE dirhandle
ENDIF
RETURN cnt-1



It gives the same result as with my first try that also printed the found contents to a file.

The first screendump below shows the beginning of the list in Windows Explorer . The next shows what comes out of above code. The directory has 1331 subdirs.
On another directory with 316 records, it listed them in alphabetical order, so this happens only with directories whith a large amount of files. I ddo not know the limit for printing correctly...

But this is no big deal, I was only trying out an idea. And I am not going to use this code for anything serious... :D
But still wonder if anyone else have had the same experience.


Regards,
Egil.


Support Amateur Radio  -  Have a ham  for dinner!

Bill-Bo

Andy,

Just ran your program. 850,308 files/folders. Could you show how to print it to a file?

Bill

Egil

Andy,

Just for fun I compiled your code in IWB and tried it in the same directory (1331 subdirs). Except from the fact that your code also showed the contents in each subdirectory, the program behaved exactly the same way as with the CB code. No alphabetical sorting.

I guess the solution to this "problem" will be to import the directory content into a LV control, sort the list, and print the sorted list to printer or file. This way will be more  flexible as the list can be sorted in numeropus ways before printing.


Regards,
Egil.
Support Amateur Radio  -  Have a ham  for dinner!

Bill-Bo

Egil,

I compiled and ran Bill's program from the link in your first post. It just opens my Excel 2007 and displays a popup indicating it cannot find the C:\!#!Files.CSV file.

Bill

Egil

June 20, 2016, 03:49:51 AM #7 Last Edit: June 20, 2016, 03:51:58 AM by Egil
Quote from: Bill-Bo on June 20, 2016, 03:41:10 AM
Egil,

I compiled and ran Bill's program from the link in your first post. It just opens my Excel 2007 and displays a popup indicating it cannot find the C:\!#!Files.CSV file.

Bill


I have no problems here using the Libre Office Calc. But try changing the filename to C:\Files.CSV.

Since this in fact is an ordinary ASCII file, it should also be able to open it in Notepad. But then you'll lose the list formatting.


Egil
Support Amateur Radio  -  Have a ham  for dinner!

Andy

Bill,

Here's how to print it to a file:

Change the line openfile(MyFile,"C:\\win10\\List.txt","W") to a folder and file name of your choice i.e. openfile(MyFile"C:\\myFolder\\MyList.txt","W")

DEF count:INT
file MyFile
OPENCONSOLE
'recursively print all directories
openfile(MyFile,"C:\\win10\\List.txt","W")
count = printdir("c:\\")
PRINT count,"Files"
closefile MyFile
waitcon

END


SUB printdir(path:STRING),INT
DEF count:INT
DEF dir,attrib=0:INT
DEF filename:STRING
DEF fullname:STRING
count = 0
dir = FINDOPEN(path + "*.*")
IF(dir)
DO
filename = FINDNEXT(dir,attrib)
IF len(filename)
IF attrib & @FILE_DIRECTORY
IF(filename <> ".") & (filename <> "..")
PRINT path + "[" + filename + "]"
               write MyFile,path + "[" + filename + "]"
fullname = path + filename + "\\"
count = count + printdir(fullname)
ENDIF
ELSE
            count = count + 1
PRINT path + filename
            write MyFile,path + filename
ENDIF
ENDIF
'the exit case is when there are no more entries
'in the current directory
UNTIL filename = ""
FINDCLOSE dir
ENDIF
RETURN count
ENDSUB


Run the program, when you've closed it, go to your folder and file and open it with say notepad.
Day after day, day after day, we struck nor breath nor motion, as idle as a painted ship upon a painted ocean.

billhsln

My program in the link does not actually sort the data, it puts it into a CSV that can be loaded into Excel and my wife would sort it in Excel.

Have you thought about storing the data in a data base file and then reading it from the data base file with a sort on the select?

SELECT directory,field2,field3 FROM dbfile ORDER BY 1,2

Should solve the file limit problem.

Bill
When all else fails, get a bigger hammer.

Egil

Quote from: billhsln on June 21, 2016, 09:24:11 AM
Have you thought about storing the data in a data base file and then reading it from the data base file with a sort on the select?


Hi Bill,

That is probably the method I'll go for when get around to tidy up my hughe audio file collection.
I was not aware of any limit on how many files and/or subdirectories a directory can contain before the automatic sorting does not  work any more.
When I try your  code on directories  containing between 200 and 450 entries, the result is a csv file where the filenames and driectories are all sorted in alphabetical order. My own CB code does exactly the same, but makes a text file.

But when I  tried to list a directory containing 1131 entries, the resulting file had all the entries randomly listed, even when file explorer sort them nice and tidy.
I can make a batch file that writes all that directories into textfiles, with the contents sorted  any way I like. (....If I'm able to remember the correct batch language syntax.... ;D)
Therefore I am quite convinced that it's possible to do the same from within code written in one of "our" languages.
Think that will be a nice exercise for the kids that started all this... ::)


All the best,
Egil.

Support Amateur Radio  -  Have a ham  for dinner!

Egil

June 21, 2016, 11:24:50 AM #11 Last Edit: June 21, 2016, 11:26:55 AM by Egil
Just for fun, I made a little experiment.

Paste this into notepad and save as TEST.BAT : dir /O > list.txt
Running this batch file  produce a textfile called list.txt  containing the present directory listing in alphabetical order.
I copied it into the directory with 1331 entries, and the resulting file still had the list sorted aphabetically.


Egil
Support Amateur Radio  -  Have a ham  for dinner!

billhsln

I would recommend:  DIR /O:NED > list.txt

Sorts by Name, Extension and Date.

Bill
Old DOS guy
When all else fails, get a bigger hammer.

Egil

June 21, 2016, 01:07:40 PM #13 Last Edit: June 21, 2016, 01:09:17 PM by Egil
Bravo!

I did not remember the extra switches. Have not used this for ages...
Maybe I should google for the complete list. I'm sure it is out there somewhere. It's a pity that almost nobody use it anymore.


Egil
Support Amateur Radio  -  Have a ham  for dinner!

billhsln

I don't know if it still works, but we used to be able to do:  DIR /?

Bill
When all else fails, get a bigger hammer.

Egil

Yes, of course!

Now I remember another one: HELP

Gives you the full command list.

Brings back memories.... 5-1/2 inch diskettes, 640 kB RAM and ten feet of User Manuals, including a crash course in assembler.
If someone had told me back then that I, in the year 2016, would walk around with a pocket telephone with only a fraction of the computing power of my present cellphone, I'd asked what the heck he was smoking....


Egil
Support Amateur Radio  -  Have a ham  for dinner!