text editor for large file
Author
Discussion

onomatopoeia

Original Poster:

3,526 posts

248 months

Monday 6th August 2007
quotequote all
Any suggestion for an editor (linux or windows) that can edit a text file without loading the whole thing into memory? I've got an enormous file created by mysqldump to edit that wont fit into memory (RAM + swap).

dilbert

7,741 posts

262 months

Monday 6th August 2007
quotequote all
Out of interest, how big is it?

Edit;

Do bear in mind that swapspace is diskspace...... You could just make your swapfile bigger.

Edited by dilbert on Monday 6th August 12:47

stevieb

5,253 posts

298 months

Monday 6th August 2007
quotequote all
google Textpad.. It is a good flexible editor and i have not come up agaist the limits of that yet...

onomatopoeia

Original Poster:

3,526 posts

248 months

Monday 6th August 2007
quotequote all
Half a gig. Notepad dies at 250MB or so. Wordstar 6.0 for DOS which was always my choice for loading large files back in the day will let me load and edit it but after three hours is still running the NTVDM process to 99% of the CPU trying to save it and I'm bored hehe

The linux server it lives on is "rather old", has a fixed swap partition which is not enough when combined with the RAM, nor does it have enough disk space available to have two copies on the disk at once.

dilbert

7,741 posts

262 months

Monday 6th August 2007
quotequote all
Well given the size of the file I can't be much help. I just wrote an editor, and I know it copes well up to 100M or so, but I figured it unlikely that people would need more, so I've not tested it beyond. TBH I think you're unlikely to find something that will operate on the file "on disk", unless you go really basic, maybe edlin?

The thing is that to get most of the features most people expect, wordwrap, syntax hilights and so on, you have to index the data. If you're going to index, you may as well keep it in memory and let the OS deal with the swapfile. Equally most editors will need space to index the data, and this will be a variable proportion of the size of the file, fixed for a given editor.

I know from experience that you get better performance out of notepad if you leave wordwrap switched off. There is a good reason for this, which I won't bore you with, but it might help.

Failing that, maybe write something that will read the file, and "chunk it". Wouldn't take long.

Edited by dilbert on Monday 6th August 14:19

rich1231

17,340 posts

291 months

Monday 6th August 2007
quotequote all
textpad and PFE in use here

stevieb

5,253 posts

298 months

Monday 6th August 2007
quotequote all
Text Pad should be able to deal with half a gig, we regularly have csv files in excess of 1 gig that it can deal with but we have plenty of RAM in our machines!!

onomatopoeia

Original Poster:

3,526 posts

248 months

Monday 6th August 2007
quotequote all
Textpad did the job, cheers all smile

dilbert

7,741 posts

262 months

Monday 6th August 2007
quotequote all
onomatopoeia said:
Textpad did the job, cheers all smile
On your original, constrained, machine?

LordLanky

68 posts

235 months

Monday 6th August 2007
quotequote all
stream edit it with sed !
"UP THE UNIX!!!"
getmecoat

jimmyjimjim

8,349 posts

269 months

Tuesday 7th August 2007
quotequote all
Ultra-edit32 is probably the best editor I've found, handles just about anything. I'd recommmend it for most tasks, apart from it can get pretty complex, and you have to pay for it after a month, IIRC. Worth a trial, though.
We've got a proprietary editor at work that'll handle very large text (and proprietary) files, but it's mainly for analysis than editing.