Parsing large file/big files with GBs space: shell command fastest way to remove header, trailer, last line, first lines with neither copying nor parsing the file.

1. We have truncate which is part of coreutils package.

this will remove whichever you want. suppose i want to remove last line. I count size of last line and remove it, i am doing  neither copying nor parsing the file.

Useful for large files which are in GBs.


truncate --size=-25 filename
 
 
 
                    

This is thefastest route since it modifies file in-place, and 
therefore doesn't require neither copying nor parsing the file. 
However, you'll still need to check how many bytes to remove... 
 
you need to specify input offset to get last kilobyte and 
then use tail -2 | LANG= wc -c 


2. What is the best way (better performance) to read a specific line of a file?
ruby -ne '$.==10 and (print; exit)'

No comments :

Post a Comment