this program reads huge amount of data binary file named data.bin each item in file 1024 bytes long. first 24 bytes of each item key , remaining 1000 bytes random information. , adds these items array list called "items" can sorted using merge sort algorithm.
but im getting outofmemoryerror on line commented error after adding 227475 items. supposed externally sorted not working right. how split huge amount of items smaller sets sorted merged?
public static void main(string[] args) { system.out.println("welcome external merge sorter."); arraylist<entry<biginteger, biginteger>> items = new arraylist<entry<biginteger, biginteger>> (); tentry<biginteger, biginteger> en = null; try { randomaccessfile data = new randomaccessfile("data.bin","rws"); system.out.println("found file data.bin."); long length = data.length(); long recs = length / 1024; long count = 0; byte []b = new byte[1024]; system.out.println("sorting..."); while(count < recs) { count++; data.readfully(b); byte []key = arrays.copyofrange(b, 0, 24); byte []value = arrays.copyofrange(b, 24, 1024); system.out.println(count); //error en = new tentry<biginteger, biginteger>(new biginteger(key), new biginteger(value)); items.add(en); } } catch (exception e) { e.printstacktrace(); } itemcompare compare = new itemcompare(); mergesort sorter = new mergesort(); sorter.sort(items, compare); system.out.println("done!"); }
so, might not option in case, or maybe don't want use other code. but, there several 3rd party libraries external sorting. here 1 i've used before, http://code.google.com/p/externalsortinginjava/. i've found fast , memory efficient.
here example code use
file f = new file("/file/to/sort"); comparator<string> comparator = new comparator<string>() { public int compare(string r1, string r2){ return r1.compareto(r2);}}; int maxnumbertempfiles = 1000; charset cs = charset.defaultcharset(); boolean distinctvalues = true; list<file> l = externalsort.sortinbatch(f, comparator, maxnumbertempfiles,cs,null,distinctvalues) ; file sf = new file("sortedfile."+r.nextint()+".tmp"); externalsort.mergesortedfiles(l, sf, comparator,cs, true);
Comments
Post a Comment