Linux File Handling Tips & Tricks

About:
Linux File Handling Tips & Tricks

Uploaded:
2013-11-07

Check for updates of this page        Bookmark this freepage         View as text to select more keys at once.


----- heretic@uWpOCr1HNGD+pNSCQJ_3zfu6dWo ----- 2012.12.04 - 22:19:20GMT -----

In the Sierra Request > Jenna Request thread, datmon mentioned a linux command that he uses to generate picture indexes, which made me think of all the other commands that I use frequently to move, rename, zip, or otherwise handle files. (Thanks datmon!) I thought I'd start a thread to share some of the tricks that I've picked up, and hopefully learn some new ones from you guys too :)

The following commands are all for linux, but Windows tricks are welcome too.

First of all, datmon's post from the aforementioned thread:
----- datmon@a89M0Ll7ONMCookIXFxEtes1z20 ----- 2012.12.04 - 14:32:10GMT -----

I'm not sure for windows, but for linux I use this command for pictures:

r="${PWD##*/}" ; montage -background white -title "$r" -label '%f' -size 512x512 '*[256x256]' -auto-orient -geometry +35+35 -tile 4x -shadow index.jpg

Videos:
movie thumbnailer (mtn)
http://moviethumbnail.sourceforge.net/index.en.html

For windows I use Thumbmailme 3.0 for videos.
For linux I run this command:

mtn -b 0.80 -c5 -r4 -h100 -o _index.jpg -f /pathtofont/arial.ttf $PWD

--

For the rest, I'll separate each section with "--".

--

Create folders for image sets, and move the images into those folders:

Many times, I end up with many pictures from a model from multiple sets all in the same folder. For sets that have a good naming convention (Newstar sets are a good example), it is easy to create folders and move the files into those folders with one command.

Consecutive range:
for num in {052..059}; do
mkdir $num
mv *-"$num"-* $num
done

Non-consecutive range:
for num in {191, 245, 246}; do
mkdir $num
mv *-"$num"-* $num
done

Paste one of the above into a text editor, and change the numbers in the range to fit your needs. Then, copy the entire statement, and paste into your terminal (you'll have to right click and choose paste, ctrl+v won't work), and hit enter. It will create a folder with the name of the number of the set (e.g., 001, 002, 003), and move files that follow the naming pattern anything-###-anything into the folder with the same number. You can change the pattern to fit your needs. 'mkdir nameHere' is the command to make a new folder, and 'mv filename foldername' is the command to move a file. The * matches any character, so in this case, the * in the name lets us move all files that contain '-052-' to a folder called '052'.

--

Repair a broken archive with par2 files:

This one is fairly easy. If you only have relevant files and one par2 in a folder, you can simply run:
par2 r -v *par2 *

Otherwise, you can use something like:
par2 r -v *valensiya*par2 *valensiya*

The r specifies repair, -v specifies verbose (logging of what it's doing), then you give it the name of the par2 file, and the names of the files with relevant data. * fills in the gaps and makes it easier to write.

--

Recursively delete a file from child folders of the given folder:

The following will delete all Thumbs.db files in all folders that are in the folder in which you run this command:
find . -type f -name "Thumbs.db" -exec rm -v {} \;

You can give looser specifications as well, if you wish:
find . -type f -name "*copy*" -exec rm -v {} \;

Be careful - again, this will remove all instances that match the given name in all folders that are in the folder that you run the command in.

--

Rename multiple files within a folder:

The following is the command I use to rename thumbnails that I generate for the CandyDoll freesite:
rename -v 's/_[0-9]*\./_preview\./' *

It uses regex to determine a match. The -v specifies verbose. It will only change the names of files that match the given regex pattern. The syntax is:
rename -v 's/regex you are looking for/what to replace it with/' (files to apply this to)

Another example, for changing a bunch of files that were named with '_'s while the normal pattern is with '-'s:
rename -v 's/_/-/' *

This next example is more advanced, and uses a capturing group to move part of the file name to another part of the name:
rename 's/Silver-Alissa_(\w+-1)-\d+.jpg/replacement-tile_silver-alissa_$1.jpg/' *

Parentheses define a capturing group, and its contents can be recalled using a dollar sign, followed by the number of the capturing group (e.g., $1) within the regex string. You can have multiple capturing groups, and the count starts with 1. In terms of the regex itself, thats a whole other topic, but here are a couple basics:

\w any character from a-z, upper or lower case, any digit, or _
\d any digit
[abc%@123] any one of the characters within the brackets
+ repeat 1 or more times
* repeat 0 or more times
[a-z][0-9] any character from a-z (lower case only), followed by any digit ('-' specifies range; if you want that to be a valid option, place it at the end)
[123-] 1, 2, 3, or -
. any character at all (for the purposes of this)
\. a period (\ is an escape character, specifying that the following character should be taken literally, without its usual special meaning)

There's much more you can find about regex that you can find with a simple Google search.

--

Combine multiple parts of a split file:
cat filename* > filename

Not much to explain. Try this if for example you have an avi broken into multiple parts, ending with 001, 002, etc., and no par2 to repair the file automatically.

--

Compress each folder within a given folder into separate rar files, with the same name as the folders:

I have the following in a file called zipFolders.sh:
#!/bin/bash
for folder in */
do
rar a "${folder%/}.rar" "$folder"
done

Simply place the file in the folder that contains folders that you want compressed, and double click the file. I usually choose 'Run in terminal' so I can watch it's progress, but it can run silently as well. You'll need to allow executing of the file as a program. You could also just paste it in the terminal if you wanted. I keep it in a file that stays in the folder that I use as a staging area for CandyDoll inserts since it never really needs changing.

--

Unzip multiple .zip files that were created without folders inside (the kind where if you unzipped a bunch at once you would create a mess) into folders named with the given number:
for num in {110..145}; do
unzip *$num.zip -d $num
done

You may or may not need to add 'mkdir $num' in the line before 'unzip...'; I don't use this often, so I don't remember.

--

Most of these are fairly simple, but can be made more complex if you wanted to. Never underestimate terminal commands. In some cases, they can be much more powerful than programs with GUIs.

--

----- datmon@a89M0Ll7ONMCookIXFxEtes1z20 ----- 2012.12.05 - 02:43:59GMT -----

Thanks, I see a couple that I might use. I'm a linux script newbie, but I like to take existing scripts and modify them for my use. This also means that I can't explain in detail how some of these work.

I like my scripts on one line so I can run them from the terminal and to be able to right-click on a file or folder and run the script from a custom action I created in the file manager.

list the number of files in folders, nicely formatted:
find . -type f | cut -d "/" -f 2 | sort | uniq -c | awk '{print $2 " (" $1 " files)"} input_file'

I would love to get this script to work with the rar script, where it will compress a folder and list the number of files it contains. ex: Amy (40 files).rar. I tried it, but couldn't figure it out. Maybe someone more experienced can.

--

rar folders recursively on one line:

for f in *; do rar a "$f.rar" "$f"; done

---

create empty folders 001 - 009:

for (( i=1; i<10; i++)); do mkdir 00$i; done

create empty folders 010 - 100:

for (( i=10; i<100; i++)); do mkdir 0$i; done

---

For people that like to rename files :)

recursively rename all picture files and movie files to the name of the folder and append original file name. If you have a folder named Amy-001 then all the pictures within the folder will be renamed. D3302.jpg will be renamed to Amy-001-D3302.jpg.

find -type d | while read d; do cd "$d"; c=1; for i in *.jpg *.JPG *.jpeg *.avi *.mpg *.wmv; do if [ ! -f "$i" ]; then continue; fi; r="${PWD##*/}-$i"; mv -i "$i" "$r"; ((c++)); done; cd "$OLDPWD"; done

---

recursively rename all files and folders to lowercase.

find . -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;

---
CSV-Tutorial:

A bit more on Linux

----- Stedman Winfrey Esq.@RTVDrig+ExrU_oMv86e37q_dui8 ----- 2013.11.01 - 23:54:31GMT -----

Hello, does any one have some pro-tips for a Linux user? I have Debian, how is it best to tune up my Freenet? Thank you!

----- Anonymous ----- 2013.11.06 - 02:08:42GMT -----

Use root encryption with luks and a *long* password. Also turn off freenet's own internal datastore encryption. It's slow as hell and totally pointless if your disk is already encrypted.

OpenJDK 7 doesn't have the same problems as Orakle's so you can use it without fear of DB corruption.

Install git and compile your own freenet source + patch.

Run freenet as it's own user in a restricted account. It only needs UDP access to the outside so restrict TCP connections for that user.

Freenet generates a lot of I/O so put the datastore on a different physical encrypted disk and mount it back to where it should be.

Other than that it's bigger datastore, more memory, more bandwidth...


Freepage, made with a modified ShareLink Freenet plugin.