Hello!
I understand everything has it's limits and not insist to get this into primary goal... this is just a remark!
I am seeing a sharp drop in performance using decent amount of stream input and output.
This is only for example but let's assume we have a text file with such content:
replace.txt
0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
So in real life we have a text file encoded with non-english ASCII symbols.
And we need to replace all those 36 values with text and going to do so with following script with a lot of pipes:
replace.ys1
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "0" "-" | REPL -- "1" "-" | REPL -- "2" "-" | REPL -- "3" "-" | REPL -- "4" "-" | REPL -- "5" "-" | REPL -- "6" "-" | REPL -- "7" "-" | REPL -- "8" "-" | REPL -- "9" "-" | REPL -- "A" "-" | REPL -- "B" "-" | REPL -- "C" "-" | REPL -- "D" "-" | REPL -- "E" "-" | REPL -- "F" "-" | REPL -- "G" "-" | REPL -- "H" "-" | REPL -- "I" "-" | REPL -- "J" "-" | REPL -- "K" "-" | REPL -- "L" "-" | REPL -- "M" "-" | REPL -- "N" "-" | REPL -- "O" "-" | REPL -- "P" "-" | REPL -- "Q" "-" | REPL -- "R" "-" | REPL -- "S" "-" | REPL -- "T" "-" | REPL -- "U" "-" | REPL -- "V" "-" | REPL -- "W" "-" | REPL -- "X" "-" | REPL -- "Y" "-" | REPL -- "Z" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
All works... but:
`timethis.exe yori -nouser -c YS "replace.ys1"
Elapsed time: 0:00:11.989
Child CPU time: 0:00:00.156
Child kernel time: 0:00:00.156
Child user time: 0:00:00.000
Tree CPU time: 0:00:00.562
Tree kernel time: 0:00:00.406
Tree user time: 0:00:00.156
12 seconds certainly not a big deal but if process comes to real text file with normal contents speed degrades drastically...
The thing here is if we split those 36 REPL pipes into 36 different lines:
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "0" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "1" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "2" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "3" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "4" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "5" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "6" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "7" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "8" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "9" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "A" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "B" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "C" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "D" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "E" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "F" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "G" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "H" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "I" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "J" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "K" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "L" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "M" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "N" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "O" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "P" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "Q" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "R" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "S" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "T" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "U" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "V" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "W" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "X" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "Y" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
ICONV -i ascii -e utf8 -w -- "replace.txt" | REPL -- "Z" "-" | ICONV -i utf8 -e ascii -w | SPONGE -- "replace.txt"
and do that again with:
`timethis.exe yori -nouser -c YS "replace.ys1"
Elapsed time: 0:00:01.738
Child CPU time: 0:00:00.436
Child kernel time: 0:00:00.421
Child user time: 0:00:00.015
Tree CPU time: 0:00:02.281
Tree kernel time: 0:00:01.656
Tree user time: 0:00:00.625
So despite there was 36 disk reads 72 encoding conversions and 36 disk writes it is 6 times faster than 1st variant... one problem here is we need to check an ERRORLEVEL after each of 36 lines to ensure that all was correctly.
And by the way in real life with 30 lines in script we get speed something around 5 seconds per file and with one line in script with 30 pipes speed drops to 30 seconds per file!
And this is not about REPL
only... I've tried a lot of sequiental pipes around multipe ICONV
and it seems it degrades in the same way!
So it is basically 6 times faster everywhere I've seen, tried this with CMD
YORI
YORIMIN
and ONEYORI
... and the numbers always almost the same!
And just for example tried this in CMD
with external findstr.exe
:
CMD> echo "long long long long long long long long long long long long long long long long long long long text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text" | findstr "text"
So there is 128 pipes and nothing seems to degrade significantly... I am basically want to say that as I see this is not due to Windows piping mechanism!
Or maybe you can advice how to speed things up?
P.S. I'm so sorry to be so annoying =) I believe I am doing useful things for this project