Anyone got any AWK coding experience?
Discussion
TheExcession said:
a bit but its been a while.
Likewise...
What are you trying to do?
Not at work any more but look in tommorrow and I'll explain more. It's quite a simple one really to do with stripping a DOS file of it's separators and converting it to unx format.
More tommorrow - I've had enough of the bloody thing...
Right!
Scenario: An EDI transmission creates a DOS text file, which needs to be read by out BO system, which is a UNIX based. Hence we need to first remove the Line Returns, and then convert the file to unix format ready to be imported into our system (in a csv format but as a text file).
The file is importh.txt and should eventually come out as that.
The AWK code is here (hoping the formatting works):
cd /u/3ex/fw/in
dtox horders.txt > horders.unx
mv horders.unx horders.txt
FILE=horders.txt
awk '
BEGIN{FS=","}
{
if ($1=="HEAD"
{
REF=substr($5,length($5)-9);
print $0 "," REF ",";
}
if ($1=="LINE"
{
print $0 "," REF ",";
}
if ($1=="TEXT"
{
print $0 "," REF ",";
}
}' < $FILE > orders.txt
Can anyone explain why the script isn't working? if I remove the 'cd' bit to just locally run everything, it falls over at the 'dtox' part (removing the separators)...?
Cheers
Neil
Edited to Add: Nope, it removed all the tabs - but the emphasis is the same
>> Edited by neil_cardiff on Wednesday 11th January 09:31
Scenario: An EDI transmission creates a DOS text file, which needs to be read by out BO system, which is a UNIX based. Hence we need to first remove the Line Returns, and then convert the file to unix format ready to be imported into our system (in a csv format but as a text file).
The file is importh.txt and should eventually come out as that.
The AWK code is here (hoping the formatting works):
cd /u/3ex/fw/in
dtox horders.txt > horders.unx
mv horders.unx horders.txt
FILE=horders.txt
awk '
BEGIN{FS=","}
{
if ($1=="HEAD"
{
REF=substr($5,length($5)-9);
print $0 "," REF ",";
}
if ($1=="LINE"
{
print $0 "," REF ",";
}
if ($1=="TEXT"
{
print $0 "," REF ",";
}
}' < $FILE > orders.txt
Can anyone explain why the script isn't working? if I remove the 'cd' bit to just locally run everything, it falls over at the 'dtox' part (removing the separators)...?
Cheers
Neil
Edited to Add: Nope, it removed all the tabs - but the emphasis is the same
>> Edited by neil_cardiff on Wednesday 11th January 09:31
TheExcession said:
Hi Neil,
Any chance you could post the first few lines of the input file and then provide an example of what you want it to end up as?
cheers
Original:
HEAD, KAUFMANN, FOCUS, FOCUS-1229, 001229267167, 06/01/2006, , , , , , , , , , , , , , , , , , , , , , , , , ,
LINE, 1, KL114W, , 1, 5010763042013, 018377, , , 0.0000, TWISTED ROPE WASTE BIN, 2.27, , , , ,
LINE, 2, KB358, , 2, 5010763056614, 018391, , , 0.0000, SGRS & CHRM WASTE BIN, 3.36, , , , ,
LINE, 3, KB234, , 1, 5010763054207, 020162, , , 0.0000, CHOC RAT&CHR DSKTP FIL STG, 6.1, , , , ,
LINE, 4, KB93L, , 2, 5010763025870, 981260, , , 0.0000, LRGE SEA GRSS STORAGE BSKT, 5.82, , , , ,
LINE, 5, KB115, , 1, 5010763041061, 987600, , , 0.0000, SEA GRASS SHELF BASKET, 3.3, , , , ,
LINE, 6, KB135, , 2, 5010763043126, 987624, , , 0.0000, S'GRSS/CHRM EF SML FILE BX, 14.24, , , , ,
RECON, KAUFMANN, FOCUS, 001229267167, 6
Should end up like this:
HEAD, HORWOODS, ALLD, ALL-CENTWH, 008-71404165, 26/12/2004, 02/01/2005, 00:00:00, , , , , , , , , , , , , , , , , , , , , , , , ,8-71404165,
LINE, 1, S102, , 48, 5010763001980, 0038601-1, , , 0.0000, 14CM MILK SAUCE POT, 7.55000019, , , , , ,8-71404165,
LINE, 2, S104, , 12, 5010763001997, 0038442-0, , , 0.0000, 14CM SAUCEPAN, 9.5, , , , , ,8-71404165,
LINE, 3, S105, , 15, 5010763002000, 0038448-7, , , 0.0000, 16CM SAUCEPAN, 10.7200002, , , , , ,8-71404165,
LINE, 4, S106, , 8, 5010763002017, 0038454-5, , , 0.0000, 18CM SAUCEPAN, 11.6999998, , , , , ,8-71404165,
LINE, 5, S107, , 22, 5010763002024, 0038459-4, , , 0.0000, 20CM SAUCEPAN, 12.9099998, , , , , ,8-71404165,
LINE, 6, S119, , 3, 5010763002086, 0038482-6, , , 0.0000, 24CM FRYPAN OPEN, 15.2200002, , , , , ,8-71404165,
END
Hi Neil,
Sorry mate got called away...
Soeesentially you are reading a REF var fr omthe HEAD line and then appending it to every other 'TEXT' or 'LINE' line in the file.
The awk looks fine to me, to remove the cd part you need to provide the full path for the dtox to work.
dtox /u/3ex/fw/in/horders.txt > /u/3ex/fw/in/horders.unx
I'd then make sure you fully path in rteferences to the other files.
Sorry mate got called away...
Soeesentially you are reading a REF var fr omthe HEAD line and then appending it to every other 'TEXT' or 'LINE' line in the file.
The awk looks fine to me, to remove the cd part you need to provide the full path for the dtox to work.
dtox /u/3ex/fw/in/horders.txt > /u/3ex/fw/in/horders.unx
I'd then make sure you fully path in rteferences to the other files.
neil_cardiff said:
Right!
Scenario: An EDI transmission creates a DOS text file, which needs to be read by out BO system, which is a UNIX based. Hence we need to first remove the Line Returns, and then convert the file to unix format ready to be imported into our system (in a csv format but as a text file).
The file is importh.txt and should eventually come out as that.
The AWK code is here (hoping the formatting works):
cd /u/3ex/fw/in
dtox horders.txt > horders.unx
mv horders.unx horders.txt
Don't like this. Depending on file permissions and the value of the shell "noclobber" variable, you may or may not end up with what you thought in your 'horders.txt' file. I would change this to read;
cd /u/3ex/fw/in
dtox horders.txt > horders.unx
Then read the "unx" file directly in the "awk" script;
FILE=horders.unx
neil_cardiff said:
awk '
BEGIN{FS=","}
This sets the field seperator to ",". Each line read form now on is split
into fields, remembering that the first field is number 1, on the commas. Field
'0' is the whole input line.
neil_cardiff said:
{
if ($1=="HEAD"
{
REF=substr($5,length($5)-9);
print $0 "," REF ",";
}
So, if the *first* field is the string "HEAD", set the variable "REF" to
be the last 9 characters of the fifth field, then print the whole line
with ",xxxxx," on the end.
neil_cardiff said:
if ($1=="LINE"
{
print $0 "," REF ",";
}
if ($1=="TEXT"
{
print $0 "," REF ",";
}
If the first field is "LINE" or "TEXT", just repeat the print, without setting
the REF variable.
neil_cardiff said:
}' < $FILE > orders.txt
Can anyone explain why the script isn't working? if I remove the 'cd' bit to just locally run everything, it falls over at the 'dtox' part (removing the separators)...?
"Falls over"? What's the error message?
If you email me your 'phone number, I'm happy to call and try and talk you through this... (Although my suggestion of rewriting it in perl was serious!)
Regards,
Z.
>> Edited by zumbruk on Wednesday 11th January 13:36
neil_cardiff said:
FILE=horders.txt
awk '
BEGIN{FS=","}
{
if ($1=="HEAD"
{
REF=substr($5,length($5)-9);
print $0 "," REF ",";
}
if ($1=="LINE"
{
print $0 "," REF ",";
}
if ($1=="TEXT"
{
print $0 "," REF ",";
}
}' < $FILE > orders.txt
What does this line do? We can't figure out what it is trying to achieve...!
Hey Neil, awk just takes a file and reads it one line at a time, and if you tell it to will act upon the contents of that line.
An awk comand has 3 parts/sections, an BEGIN, MAIN and END, the code in the 'begin' section is execute once when the cammonad is called, normally it's where you initailse variables etc.
Code in the 'MAIN' section is exected for every line in the input file.
Code at the 'END' is executed once after the input file has been processed.
so in your BEGIN section you declare FS=',' FS is the field separator, so when processing lines in the MAIN section awk will separate each line into an array of fields based upon seeing a comma in the input line.
Once you get into main...
Fields from the input line are tested by referencing them using $, $0 is the entire input line, $1 is the first data field, $2 the second etc etc, all determined by pressence of the FS - in your case a ','.
so the lines
if ($1=="HEAD"
{
REF=substr($5,length($5)-9) ;
print $0 "," REF "," ;
}
say's if the first field contain 'HEAD' (in your case this is how the code is identifying the first line of the file) then delcare a variable called REF
assign this var the value of field 5 - but using a substring function, in your data
HEAD, KAUFMANN, FOCUS, FOCUS-1229, 001229267167, 06/01/2006, , , , , , , , , , , , , , , , , , , , , , , , , ,
field 5 is 001229267167, but then it takes a substring of this such that it uses the last 9 characters - 1229267167, so fro mnow on REF=1229267167
the line print $0 "," REF "," ;
prints the enitre line but also adds ',1229267167,' to the end of it.
The next few test lines say
if ($1=="LINE" ) .... or if ($1=="TEXT" ) these are looking at the 1st field of the Input line, if the Value is TEXT or LINE then the line is printed, again with the added REF variable on the end.
so awk is taking the input file, when it sees the HEAD field it outputs the entire line with the extra extracted REF info on the end. For every other line in the file, providing the line begins with LINE or TEXT it will out put that line with the extra REF info.
all output form awk is being redirected intoyour output file...
< $FILE > orders.txt
it uses what ever is in $FILE for input and all output goes to orders.txt
HTH
edit: Doh beaten to it by zumbruk - almost word for word there
>> Edited by TheExcession on Wednesday 11th January 13:39
neil_cardiff said:
Right!
}' < $FILE > orders.txt
That line just means "read input from the file whose name is in the variable $FILE and write it to a file called orders.txt". It's not part of AWK, that's standard Unix shell syntax.
You set the $FILE variable in the START part of the program; AWK programs have 3 parts, a START part which is executed once before any data is read, a main section which is executed once for every line in the input file and a FINISH part which is executed once after all the data is read.
>> Edited by zumbruk on Wednesday 11th January 13:50
GreenV8S said:
To start with, put the awk script in a separate file rather than in-line on the awk command line; this way you will get sensible error messages.
You reckon?
What you'll actually get is awk's famous and utterly useless universal error message "Bailing out near line 1". (Besides, I don't think this is an awk problem - I think he's reading the wrong file cos the d2x is going wrong.)
Anyhow, Neil, if you do that, you need to change your script to something like,
awk -f awkfile < inputfile > outputfile
(It's very hard trying to do this on Pistonheads, 'cos when you're composing an answer you can't see the original unless you quote it and then you're into the [quote][/unquote] nonsense.
zumbruk said:
To start with, put the awk script in a separate file rather than in-line on the awk command line; this way you will get sensible error messages.
You reckon?
Well, sensible in the sense that it will give a line number and you can then go to that line number and see where the problem occurs, rather than the 'somewhere near line 1' mallarky.
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff