[Vis5d-users] RE: 3G files

Thanks Leigh. Vaidya will look into it and let you know how it goes.

Jason

----------------
Jason Leigh, PhD
Associate Professor, Computer Science Department
sp...@ev...              Electronic Visualization Lab (M/C 152)
EVL Phone (312) 996-3002         University of Illinois at Chicago
EVL FAX   (312) 413-7585         851 S. Morgan St. Room 1120 SEO
http://www.evl.uic.edu/spiff     Chicago, IL 60607-7053 

> -----Original Message-----
> From: or...@or... [mailto:or...@or...] On 
> Behalf Of Leigh Orf
> Sent: Thursday, February 13, 2003 8:08 AM
> To: vaidyasubramanian chandrasekhar
> Cc: Jason Leigh; vis...@ss...
> Subject: Re: 3G files 
> 
> 
> 
> Vaidya,
> 
> Some background:
> 
> Out-of-the box, vis5d does not support files which are > 2 GB 
> in a 32 bit operating system with 32 bit integers. Some 
> operating system/compiler combinations can make "int" be 64 
> bits and it will work fine, but I could not find a way to 
> tell gcc on Linux to treat ints as 64 bits.
> 
> The v5dstruct structure has the following integers which are 
> used to navigate through a vis5d file:
> 
>   int CurPos;              /* current position of file pointer */
>   int FirstGridPos;        /* position of first grid in file */
>   int GridSize[MAXVARS];   /* size of each grid */
>   int SumGridSizes;        /* sum of GridSize[0..NumVars-1] */
> 
> When the size of the vis5d file exceeds 2^31 bytes (2 GB), 
> there is an overflow in CurPos and probably SumGridSizes. I 
> found this out while trying to write a large vis5d file.
> 
> My patch does the following: changes "int" to "int64_t" in 
> variables which refer to positioning within a vis5d file.
> 
> How to compile using my patch:
> 
> Take the patch which was attached to the file you quoted and 
> put it in the src directory. Type
> 
> patch < patch64
> 
> and it will modify some files.
> 
> You must then set the following envrinment variables which 
> will be passed to gcc during compilation. If you are using 
> the bash shell the command is:
> 
> export CFLAGS="-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"
> 
> If you are in some csh derivative it's
> 
> setenv CFLAGS "-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"
> 
> Then, type configure and make.
> 
> You can then link with v5d.o and binio.o in your program 
> which creates vis5d files if you want to create > 2 GB files, 
> and of course the vis5d binary you created can read them.
> 
> I'm pretty sure that the vis5d binary that you create using 
> my method will be able to read vis5d files which are made 
> using the default source code, but I can't be 100% sure. I 
> know it doesn't work the other way around! What will happen 
> if you run the default vis5d with one of my large files is 
> once you get 2 GB into the file all the isosurfaces and 
> contours get all messed up.
> 
> I have sent my patch to the vis5d list but I don't think 
> there is any development going on right now. There may be a 
> "better way" to make vis5d work with these large files, but I 
> think my way is sound.
> 
> Incidentally, just to make sure, you have to be using a 
> filesystem that supports > 2 GB files, such as ext3 or 
> reiserfs... ext2 does not. Most modern Linux distros come 
> with these filesystems, and I would strongly recommend that 
> you use either on on all of your main partitoins.
> 
> Leigh
> 
> vaidyasubramanian chandrasekhar wrote:
> 
> |   Leigh orf,
> |   Iam Vaidya from EVL @uic working on the vis5d and getting 
> it on the geo
> |   wall. I received your mail for runnning huge datasets > 
> 2G on vis5d, I am
> |   not too sure where the patches need to be added and then 
> what to excatly
> |   do,.
> |   
> |   Can you please tell me where the patch needs to be added 
> and where the
> |   changes need to be done .?
> |   
> |   The smaller vis5d data sets load and are animated in the 
> geowall and hence
> |   I persume the problem as you had suggested is due to the 
> large data 
> | sets.
> |   
> |   regards
> |   
> |   vaidya
>