Howdy,
What causes this?
2019-08-25 15:34:19 Grunged message (type 2) from 618:618/1 at offset 10769 in packet: /opt/sbbs/fido/inbound/d622a770.pkt
2019-08-25 15:34:19 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
I've seen this a few times from various uplinks while rescanning messages bases (to populate my synchronet).
I know with mystic it has occurred because the last message (in a mail bundle) doesnt have the right number of terminating nulls - but Ive also seen this from some non mystic uplinks.
in packet: /opt/sbbs/fido/inbound/d622a770.pkt2019-08-25 15:34:19 Grunged message (type 2) from 618:618/1 at offset 10769
have some bug(s) that cause this. Not much can be done except to notify the party where it came from.2019-08-25 15:34:19 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pktA bad packet, likely a software bug somewhere.
Some software authors either couldn't follow the fidonet specs or they
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30|.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65
69 73 6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e69 73 00 4e 6f 20 |.Sean Dennis.No |
Re: Grunged Messages
By: Digital Man to Alterego on Sat Aug 24 2019 11:47 pm
2019-08-25 15:34:19 Grunged message (type 2) from 618:618/1 at offset 10769 in packet: /opt/sbbs/fido/inbound/d622a770.pktA bad packet, likely a software bug somewhere.
2019-08-25 15:34:19 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
Some software authors either couldn't follow the fidonet specs or they have some bug(s) that cause this. Not much can be done except to notify the party where it came from.
So I did some checking of this packet - and I think I found the culprit.
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30 |.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65 69 73 6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e 69 73 00 4e 6f 20 |.Sean Dennis.No |
Notice that the date is "04 Feb 119".
I'm guessing you are barfing on the Date being an incorrect date.
Could I make a suggestion? It seems this message in the packet resulted in SBBSecho throwing the packet away (well not processing any more messages in it). The messages preceeding this packet were accepted and tossed, (and it appears that subsequent messages would be OK).
Technically this field is of the right size (20 bytes), and thus the layout of the message is correct (well, I havent checked the other fields) - but this one message has a bad date.
Is it possible to just throw this message away, and continue processing to see if other messages are valid (and if so, toss them)?
10769 in packet: /opt/sbbs/fido/inbound/d622a770.pkt2019-08-25 15:34:19 Grunged message (type 2) from 618:618/1 at offset
/opt/sbbs/fido/inbound/d622a770.pkt2019-08-25 15:34:19 Bad packet detected:
A bad packet, likely a software bug somewhere.
Is it possible to just throw this message away, and continue processing to see if other messages are valid (and if so, toss them)?Sure. I just committed a one-line change to do that.
Re: Grunged Messages
By: Digital Man to Alterego on Mon Aug 26 2019 08:48 pm
2019-08-25 15:34:19 Grunged message (type 2) from 618:618/1 at offset 10769 in packet: /opt/sbbs/fido/inbound/d622a770.pkt
2019-08-25 15:34:19 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
A bad packet, likely a software bug somewhere.
Is it possible to just throw this message away, and continue processing to see if other messages are valid (and if so, toss them)?Sure. I just committed a one-line change to do that.
So, some more feedback - I updated and re-tried the packet.
2019-08-29 15:00:57 Kurt Weiske (618:618/1) To: (3:510/1) Exported to /opt/sbbs/data/netmail/1.msg
2019-08-29 15:00:57 Grunged message (type 21057) from 618:618/1 at offset 10844 in packet: /opt/sbbs/fido/inbound/d622a770.pkt
2019-08-29 15:00:57 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
1) The offset for the bad packet changed (10769->10844).
2) It created a netmail out of the bad messages (it was echomail for area:MIN_HUB).
3) It didnt toss the remaining messages after this bad message :(
No big deal for me - thought you might like to know...
2019-08-29 15:00:57 Kurt Weiske (618:618/1) To: (3:510/1) Exported toSome *other* invalid packet header data was discovered (besides the DateTime field) then.
/opt/sbbs/data/netmail/1.msg
2019-08-29 15:00:57 Grunged message (type 21057) from 618:618/1 at
offset 10844 in packet: /opt/sbbs/fido/inbound/d622a770.pkt
2019-08-29 15:00:57 Bad packet detected:
/opt/sbbs/fido/inbound/d622a770.pkt
1) The offset for the bad packet changed (10769->10844).
2) It created a netmail out of the bad messages (it was echomail for
area:MIN_HUB).
3) It didnt toss the remaining messages after this bad message :(
Well it will toss a message has that particular DateTime corruption you pointed out. Other times of packet data corruption will still cause the packet to be rejected.
Re: Grunged Messages
By: Digital Man to Alterego on Thu Aug 29 2019 01:31 pm
2019-08-29 15:00:57 Kurt Weiske (618:618/1) To: (3:510/1) Exported toSome *other* invalid packet header data was discovered (besides the DateTime field) then.
/opt/sbbs/data/netmail/1.msg
2019-08-29 15:00:57 Grunged message (type 21057) from 618:618/1 at
offset 10844 in packet: /opt/sbbs/fido/inbound/d622a770.pkt
2019-08-29 15:00:57 Bad packet detected:
/opt/sbbs/fido/inbound/d622a770.pkt
1) The offset for the bad packet changed (10769->10844).
OK... I'm curious as to what that was, so I'll see if I can find it...
2) It created a netmail out of the bad messages (it was echomail for
area:MIN_HUB).
This mailpacket was an echomail rescan - so it only has echmail in it. I'll validate that for sure with a packet dump tool that I wrote.
This was the same message that it was barfing on previously (because of the bad time "4 Feb 119"). So instead of discarding it (or tossing it with an invalid date, it converted it to a Netmail
(*.msg) from 618:618/1 to 3:510/1). The 3:510/1 is incorrect (no zone info on echomail for a "to" receiptient?), but the echomail was from 618:618/1 to 618:510/1.
Here is the msg in question (and the beging of the next msg)
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30 |.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65 69 73 6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e 69 73 00 4e 6f 20 |.Sean Dennis.No | 00002a50 62 61 64 20 70 61 63 6b 65 74 73 00 41 52 45 41 |bad packets.AREA| 00002a60 3a 4d 49 4e 5f 48 55 42 0d 01 50 49 44 3a 20 42 |:MIN_HUB..PID: B| 00002a70 57 4d 41 58 32 20 33 2e 31 31 20 5b 45 76 61 6c |WMAX2 3.11 [Eval| 00002a80 5d 0d 01 4d 53 47 49 44 3a 20 36 31 38 3a 36 31 |]..MSGID: 618:61| 00002a90 38 2f 31 2e 30 20 35 63 35 38 66 32 36 32 0d 48 |8/1.0 5c58f262.H| 00002aa0 69 20 4b 75 72 74 2c 0d 0a 0d 0a 49 20 68 61 76 |i Kurt,....I hav| 00002ab0 65 6e 27 74 20 6e 6f 74 69 63 65 64 20 61 6e 79 |en't noticed any| 00002ac0 20 62 61 64 20 70 61 63 6b 65 74 73 20 6c 61 74 | bad packets lat| 00002ad0 65 6c 79 2e 2e 2e 20 3a 29 0d 0a 0d 0a 4c 61 74 |ely... :)....Lat| 00002ae0 65 72 2c 0d 0a 53 65 61 6e 0d 0a 0d 0a 20 0d 0a |er,..Sean.... ..| 00002af0 2d 2d 2d 20 4d 75 6c 74 69 4d 61 69 6c 2f 57 69 |--- MultiMail/Wi| 00002b00 6e 20 76 30 2e 35 31 0d 0a 20 2a 20 4f 72 69 67 |n v0.51.. * Orig| 00002b10 69 6e 3a 20 4d 69 63 72 6f 6e 65 74 20 57 6f 72 |in: Micronet Wor| 00002b20 6c 64 20 48 51 20 2d 20 62 62 73 2e 6f 75 74 70 |ld HQ - bbs.outp| 00002b30 6f 73 74 62 62 73 2e 6e 65 74 20 28 36 31 38 3a |ostbbs.net (618:| 00002b40 36 31 38 2f 31 29 0d 53 45 45 4e 2d 42 59 3a 20 |618/1).SEEN-BY: | 00002b50 31 30 30 2f 31 20 32 20 32 30 30 2f 31 20 32 35 |100/1 2 200/1 25| 00002b60 30 2f 31 20 33 30 30 2f 31 20 35 30 30 2f 31 20 |0/1 300/1 500/1 | 00002b70 35 31 30 2f 31 20 36 31 38 2f 31 20 32 0d 0a 01 |510/1 618/1 2...| 00002b80 50 41 54 48 3a 20 36 31 38 2f 31 0d 00 02 00 01 |PATH: 618/1.....| 00002b90 00 01 00 6a 02 fe 01 00 00 00 00 30 37 20 46 65 |...j.......07 Fe| 00002ba0 62 20 31 39 20 20 31 37 3a 33 34 3a 34 34 00 41 |b 19 17:34:44.A| 00002bb0 6c 6c 00 53 65 61 6e 20 44 65 6e 6e 69 73 00 46 |ll.Sean Dennis.F| 00002bc0 69 6e 61 6c 6c 79 2e 2e 2e 00 41 52 45 41 3a 4d |inally....AREA:M| 00002bd0 49 4e 5f 48 55 42 0d 01 4d 53 47 49 44 3a 20 36 |IN_HUB..MSGID: 6|
3) It didnt toss the remaining messages after this bad message :(
Well it will toss a message has that particular DateTime corruption you pointed out. Other times of packet data corruption will still cause the packet to be rejected.
That bad message wasnt tossed, and as you can see from the dump there is the "next" echo message, and it wasnt tossed either (nor any of the remaining messages).
(The change I suggested was given that this one echomail message is bad in the packet, then it should stop the remaining ones from getting tossed.)
Here is the msg in question (and the beging of the next msg)|.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65 69
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30
734e 6f 20 |.Sean Dennis.No | 00002a50 62 61 64 20 70 61 63 6b 65 74 73 00 41 52 45 41 |bad packets.AREA| 00002a60 3a 4d 49 4e 5f
6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e 69 73 00
48
Re: Grunged Messages
By: Alterego to Digital Man on Fri Aug 30 2019 08:18 am
DM
I originally posted this message in Slyedit on a CP437 terminal.
Posting it, and displaying it afterwards, it looks ok.
But on a UTF terminal it is wrapped horribly.
It was pasted in text 78 chars wide (a hex dump). So re-displaying it on a UTF termial, your display wrapping is not happy about somthing, and the pasted in text is merged to a single long line.
Here is the msg in question (and the beging of the next msg)
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30 |.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65 69 73 6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e 69 73 00 4e 6f 20 |.Sean Dennis.No | 00002a50 62 61 64 20 70 61 63 6b 65 74 73 00 41 52 45 41 |bad packets.AREA| 00002a60 3a 4d 49 4e 5f 48
I've switched out SLYedit for Deuce's FSEditor to see how pasted text would go here (in case it is editor related). This is being posted on a UTF8 terminal...
00002a10 00 02 00 01 00 01 00 6a 02 fe 01 00 00 00 00 30 |.......j.......0| 00002a20 34 20 46 65 62 20 31 31 39 20 20 32 30 3a 32 36 |4 Feb 119 20:26| 00002a30 3a 33 32 04 00 4b 75 72 74 20 57 65 69 73 6b 65 |:32..Kurt Weiske| 00002a40 00 53 65 61 6e 20 44 65 6e 6e 69 73 00 4e 6f 20 |.Sean Dennis.No | 00002a50 62 61 64 20 70 61 63 6b 65 74 73 00 41 52 45 41 |bad packets.AREA| 00002a60 3a 4d 49 4e 5f 48 55 42 0d 01 50 49 44 3a 20 42 |:MIN_HUB..PID: B|
The text above it pasted in at 78 chars wide.
BTW: I'm seeing funky stuff happen with backspace (since my user config is backspace is ^H (ascii 8?)
You said "netmail" there (before). Any message in packet that does notcontain an AREA: line is considered netmail.
You can send me the packet, but if it's corrupted, it's corrupted. How it'scorrupted isn't all that important or interesting. <shrug>
Re: Grunged Messages
By: Digital Man to Alterego on Thu Aug 29 2019 03:50 pm
You said "netmail" there (before). Any message in packet that does not contain an AREA: line is considered netmail.
I know - but SBBSecho created a *.msg file (and the log made me conclude it was a netmail) - but as you can see from the hex dump it is definately an echomail).
You can send me the packet, but if it's corrupted, it's corrupted. How it's corrupted isn't all that important or interesting. <shrug>
But that's my point.
I dont think the packet is corrupt, but the message inside it is invalid - it has bad text representing a date (but its structure is still valid - it was 20 chars). I'm OK with "throw that message away away", but it would be great if it continued on to parse for any more valid messages and toss them (especially since it already parsed 5 or 6 before the bad one - and the structure of the message was valid).
I've since got another bad packet from FSXnet - it had a message with a subject length of 81 characters (null terminated), and while I would agree that this message had a bad structure (and thus any additional ones are likely to be "off" by 9 chars - and thus probably wouldnt be parsed correctly) - it might be appropriate to throw that packet because the data was bad and the structure was bad.
I know my OCD is probably kicking in here - but I thought I'd bring it up in case you werent aware of that scenario.
I'll send you the packet in case you do want to explore it. I have the bad subject line packet too if that is of any value.
Re: Grunged Messages
By: Alterego to Digital Man on Fri Aug 30 2019 09:52 am
Re: Grunged Messages
By: Digital Man to Alterego on Thu Aug 29 2019 03:50 pm
You said "netmail" there (before). Any message in packet that does not contain an AREA: line is considered netmail.
I know - but SBBSecho created a *.msg file (and the log made me conclude it was a netmail) - but as you can see from the hex dump it is definately an echomail).
You can send me the packet, but if it's corrupted, it's corrupted. How it's corrupted isn't all that important or interesting. <shrug>
But that's my point.
I dont think the packet is corrupt, but the message inside it is invalid - it has bad text representing a date (but its structure is still valid - it was 20 chars). I'm OK with "throw that message away away", but it would be great if it continued on to parse for any more valid messages and toss them (especially since it already parsed 5 or 6 before the bad one - and the structure of the message was valid).
I've since got another bad packet from FSXnet - it had a message with a subject length of 81 characters (null terminated), and while I would agree that this message had a bad structure (and thus any additional ones are likely to be "off" by 9 chars - and thus probably wouldnt be parsed correctly) - it might be appropriate to throw that packet because the data was bad and the structure was bad.
I know my OCD is probably kicking in here - but I thought I'd bring it up in case you werent aware of that scenario.
I'll send you the packet in case you do want to explore it. I have the bad subject line packet too if that is of any value.
I think it's of more value to the author of the software that is generating the bad packets. Try going that route.
In the mean-time, I think that example packet you emailed me is a clear indicator that the "20th-byte-of-DateTime-field must be null" is a good check for a valid packed-message header, so I'm going to restore that check to SBBSecho.
But on a UTF terminal it is wrapped horribly.
UTF-8 shouldn't effect wrapping. Perhaps you just mean on a terminal > 80 columns wide?
BTW: I'm seeing funky stuff happen with backspace (since my user config is backspace is ^H (ascii 8?)
So toggle that setting in user defaults.
Re: Message wrapping
By: Digital Man to Alterego on Thu Aug 29 2019 05:31 pm
But on a UTF terminal it is wrapped horribly.
UTF-8 shouldn't effect wrapping. Perhaps you just mean on a terminal > 80 columns wide?
OK, you are right, I probably should have said on "my" UTF terminal which is
80 chars wide. I guess it might be happening on a CP437 terminal too.
Anyway, I dont think it should, since it is not posted that way...
BTW: I'm seeing funky stuff happen with backspace (since my user config is backspace is ^H (ascii 8?)
So toggle that setting in user defaults.
Well, I know I can do that. I thougth you were building Synchronet so that it could work well with both modes. I was just reporting that it doesnt in that scenario. :)
I guess I should just stick to using one terminal ...
The first bytes following the 20-byte DateTime field are supposed to bethe "toUserName", but in this message header first bytes are "04 00", clearly this message is totally messed up and the first
indication (that the 20th byte of the DateTime was non-null) was a goodindicator of that. Trying to parse further is just making things worse as demonstrated by my pktdump utility:
Word/line-wrapping (re-flowing) message text is a tough thing to get rightin all scenarios.
Re: Grunged Messages
By: Digital Man to Alterego on Thu Aug 29 2019 05:49 pm
The first bytes following the 20-byte DateTime field are supposed to be the "toUserName", but in this message header first bytes are "04 00", clearly this message is totally messed up and the first
indication (that the 20th byte of the DateTime was non-null) was a good indicator of that. Trying to parse further is just making things worse as demonstrated by my pktdump utility:
Hmm, the 20th byte should be null? I know you've bought this up a similiar discussion like this in the FTSC echo - but for the date field I didnt think it needed to be.
+-----------------------+-----------------------+
14 E | |
~ DateTime ~
| 20 bytes |
+-----------------------+-----------------------+
34 22 | toUserName |
~ max 36 bytes ~
| null terminated |
+-----------------------+-----------------------+
From FTSC-0001.16 "to" should be null terminated but date isnt shown that way.
I do see earlier in that document this though:
DateTime = (* a character string 20 characters long *)
(* 01 Jan 86 02:34:56 *)
DayOfMonth " " Month " " Year " "
" " HH ":" MM ":" SS
Null
So, I assumed it wasnt null terminated. (But I didnt think that that didnt make sense, because if the date is stored as per the example it is 19 chars).
<humble>
Can I change my argument - the structure is not valid, so I agree it should be thrown away ;)
I guess
I can test
To see if this sentence
Is a single sentence or over 4 lines.
If it isnt wrapped, then I will ask, why the pasted in content was.
Re: Message wrapping
By: Digital Man to Alterego on Thu Aug 29 2019 06:42 pm
Word/line-wrapping (re-flowing) message text is a tough thing to get right in all scenarios.
I know that it would be - but there is something "different" between that hexdump past and normal typing.
There is also something different about the message I posted, where I pasted in from FTSC-0001. It's not being horribly wrapped on my wide UTF terminal.
I guess
I can test
To see if this sentence
Is a single sentence or over 4 lines.
The above sentice, was posted over 4 lines, using normal typing and "return" at the end of the line. Each line starts with a capital.
If it is wrapped, then I guess this IS a difficult problem to solve - can the auto wrapping be "turned off"?
If it isnt wrapped, then I will ask, why the pasted in content was.
Alterego wrote to Digital Man <=-
So, some more feedback - I updated and re-tried the packet.
2019-08-29 15:00:57 Kurt Weiske (618:618/1) To: (3:510/1) Exported to /opt/sbbs/data/netmail/1.msg 2019-08-29 15:00:57 Grunged message (type 21057) from 618:618/1 at offset 10844 in packet: /opt/sbbs/fido/inbound/d622a770.pkt 2019-08-29 15:00:57 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
This was the same message that it was barfing on previously (because ofthe
bad time "4 Feb 119").
especially not from my Micronet address?2019-08-29 15:00:57 Kurt Weiske (618:618/1) To: (3:510/1) Exported to /opt/sbbs/data/netmail/1.msg 2019-08-29 15:00:57 Grunged message (type 21057) from 618:618/1 at offset 10844 in packet:
/opt/sbbs/fido/inbound/d622a770.pkt 2019-08-29 15:00:57 Bad packet detected: /opt/sbbs/fido/inbound/d622a770.pkt
Weird - I'm 618:300/1 and don't recall sending netmail to fido zone 3,
Sysop: | sneaky |
---|---|
Location: | Ashburton,NZ |
Users: | 28 |
Nodes: | 8 (0 / 8) |
Uptime: | 125:13:46 |
Calls: | 1,998 |
Calls today: | 2 |
Files: | 11,111 |
Messages: | 943,007 |