Editing/Browsing Really Big Files

[Ronman]

I have the need to edit, but mostly just browse really big files. A small file would be 1GB. Text Edit, BBEdit, Smultron and most others try to load the entire file into memory. Even with a GB of memory and a lot of time on my hands these files can take 10-15 minutes to load and forget about scrolling through them.

Any ideas.

RonMan

 

[BjarneDM]

you might want to take a look at one of the emacs ports:
http://www.versiontracker.com/dyn/moreinfo/macosx/24218
http://www.versiontracker.com/dyn/moreinfo/macosx/26662
http://www.versiontracker.com/dyn/moreinfo/macosx/17244

I know nothing about how emacs handles files, but with the Unix roots of the program it might be an option.

Now, you are stating that you need to browse these files.
Are you looking for specific pieces of text?
How are files organised: one big line, paragraphs, short lines ... ?
Depending on how you answer the above, the 'grep' and 'sed' utilities in 'Terminal' might be of interest to you.

 

[fryke]

Apart from that I think one has to _also_ accept that working with _really_ big files might also simply take some time. If a file is bigger than your RAM, you can't expect to just "browse through the document" by scrolling down and up, searching "quickly" etc., since you can't hold the document (uncompressed) in RAM.

In graphics, applications work with approximations, which works fine most of the time, since the eye often doesn't need every detail for a certain task. Text's a different monster altogether, though.

But I, too, would look into vim and emacs for tasks like that task.

 

[Ronman]

you might want to take a look at one of the emacs ports:
http://www.versiontracker.com/dyn/moreinfo/macosx/24218
http://www.versiontracker.com/dyn/moreinfo/macosx/26662
http://www.versiontracker.com/dyn/moreinfo/macosx/17244

I know nothing about how emacs handles files, but with the Unix roots of the program it might be an option.

Now, you are stating that you need to browse these files.
Are you looking for specific pieces of text?
How are files organised: one big line, paragraphs, short lines ... ?
Depending on how you answer the above, the 'grep' and 'sed' utilities in 'Terminal' might be of interest to you.

Good ideas, that I have yet to try. 20 years ago I was really good with emacs. I have no idea how it handles large files.

RonMan

 

[Ronman]

Apart from that I think one has to _also_ accept that working with _really_ big files might also simply take some time. If a file is bigger than your RAM, you can't expect to just "browse through the document" by scrolling down and up, searching "quickly" etc., since you can't hold the document (uncompressed) in RAM.

In graphics, applications work with approximations, which works fine most of the time, since the eye often doesn't need every detail for a certain task. Text's a different monster altogether, though.

But I, too, would look into vim and emacs for tasks like that task.

I wish that grep and such would work but these are raw data file that have difficult to see relationships between records. Eventually it all goes into mySQL where I can see it, but I need a way to verify that it went in correctly. My only choice is to look at the raw file.

Here is a place where a WinTel buddy has me beat. He uses something called Lister that does only that. It is fast and only loads what is on the screen (not sure how it does scrolling backwards, but buffering, to a point, makes sense).

RonMan