wallace breaks dkim signature

[Skale, Franz]

Hi Jan,
It doesn't matter if the milter doesn't change the message or not.
It has to take care of the encoding to represent the OS the right 
internal character set.
It use spamassassin as well as other content filters.
Of course they reaveive the whole body but only mangle the header.
Nevertheless, the forwarded bytestream must have the right, internal 
character encoding.
Simple testmail q.e.d: (as stated in the mime encoding guidlines !)

Mail from my perl MUA: (snipped):
X-Virus-Scanned: Debian amavisd-new at
Sender: test at example.com
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Date: Sat, 02 Jun 2018 16:23:07 +0200
 From: test <test at example.com>
To: xxxxxxxxxx at gmail.com
Subject: test
Message-ID: <xxxxxxxxxxxxxx at example.com>
X-Sender: test at example.com
Content-Transfer-Encoding: quoted-printable

=C2=B5=E2=88=9E=E2=88=9E=C5=93=C5=93@=E2=88=86=E2=88=86=E2=88=86=CF=80=CF=
=80}=E2=89=A0}}{||=C2=BA=C2=AA=C2=A9=C6=92=C2=A9=E2=88=AB=E2=88=AB=C5=93=C3=
=A6=C3=A6=C3=A4=C3=A4=C3=BC=

Outputting the (sending to another stream, of course, it has to be 
decoded).
perl -E 'use utf8; use MIME::QuotedPrint "encode_qp"; use Mojo::Util 
"encode"; printf("%s\n", encode_qp(encode("UTF-8", 
"µ∞∞œœ@∆∆∆ππ}≠}}{||ºª©ƒ©∫∫œææääü")));'
=C2=B5=E2=88=9E=E2=88=9E=C5=93=C5=93@=E2=88=86=E2=88=86=E2=88=86=CF=80=CF=
=80}=E2=89=A0}}{||=C2=BA=C2=AA=C2=A9=C6=92=C2=A9=E2=88=AB=E2=88=AB=C5=93=C3=
=A6=C3=A6=C3=A4=C3=A4=C3=BC=

So the body was: µ∞∞œœ@∆∆∆ππ}≠}}{||ºª©ƒ©∫∫œææääü


The UTF-8 flow should now be clear !

My own MUA uprades all stream using UTF-8 (utf8::upgrade).

Rgds.
Franz

Am 2018-06-02 16:07, schrieb Jan Kowalsky:
> Hi Franz,
> 
> 
> Am 02.06.2018 um 09:08 schrieb Skale, Franz:
>> Hi Jan,
>> respamd must take care of the data, regardless of which encoding 
>> stream
>> injected.
> 
> dkim signing doesn't change (and shouldn't) anything of the body. If
> it's utf-8 8bit it will be utf-8 8bit after signing.
> 
> Problem is (without milter or not) that wallace seems to change the
> transfer encoding always to qouted printable which in theory is fine.
> Mailclients don't do this always. Some send with utf-8 and 8bit 
> transfer
> encoding. Which should be fine too.
> 
> As wallace comes into play after dkim signing - the problem is, that
> wallace changes the transfer encoding to quoted printable. if the mail
> has non-ascii characters even if the original message (before and after
> signing) have had transfer encoding 8bit.
> 
> This breaks of course signature. There is nothing rspamd can care 
> about.
> The utf-8 8bit is totally fine (and in the responsibility of the user
> agent).
> 
>> As i develop webservices, you've to guess the encoding prior before
>> mangling the data stream.
>> Whit utf-8 you have the ability to call is_utf8  which tells you, if 
>> the
>> the utf8 (2 bytes) representation) is either native or encoded.
>> Input has to be encoded and output decoded.
>> Regarding your problem, i think that respamd doesn't check, if the 
>> input
>> stream is alreasy quoted-printable encoded.
> 
> dkim signing (with rspamd as milter) is proceeded before mail goes to
> wallace.
> 
>> I think you've to file a bug report.
> 
> Yes, I'll do. I'll try to understand how wallace get's data from smtpd 
> -
> but I didn't get it so far.
> 
>> Pykolab only set's encoding for string handling, obviously the way to 
>> go.
>> /usr/lib/python2.7/dist-packages/wallace/module_resources.py:   
>> charset.add_charset('utf-8', charset.SHORTEST, charset.QP)
>> /usr/lib/python2.7/dist-packages/wallace/module_resources.py:        
>> msg
>> = MIMEText(utils.stripped_message(message_text), _charset='utf-8')
>> So, it's utf-8 encoded.
> 
> but why it's still quoted printable if I disable all wallace modules?
> For me it looks like the smtpd.py already get's data in quoted 
> printable
> no matter the original encoding.
> 
> in __init__.py
> 
>     def process_message(self, peer, mailfrom, rcpttos, data):
> 
> 
> if I print data to debug - it's already quoted-printable. But I didn't
> find out where the data variable comes from. So I fail with further
> debugging.
> 
>> Ergo, respamd has to decode the input stream (utf8_decode) when it's 
>> not
>> in native encoded.
>> The client sets the msg_content_type (Content-Type).
>> So, if the client uses wrong localesm than , of course the string
>> representation is wrong.
>> Respamd has to take care about decoding the string according to the
>> Content-Type.
>> I think you've to file a bug report for respamd.
> 
> 
> Well, there is everything fine with encoding. rspamd doesn't touch
> anything on encoding or Content type. Just parsing email and sign's 
> dkim.
> 
> For me it looks totally fine to send emails with utf8 and 8bit (not
> quoted printable) like some clients do. There is no need for changing 
> it
> on the way.
> 
> My guess is, that wallace in principle sets transfer encoding to quoted
> printable as long it's not transferrable with 7bit. But I can't find
> where it does.
> 
> Regards
> Jan
> 
>> 
>> 
>> Rgds.
>> Franz
>> 
>> Am 2018-06-02 03:10, schrieb Jan Kowalsky:
>>> Hi Franz,
>>> 
>>> Am 01.06.2018 um 11:17 schrieb Skale, Franz:
>>>> Hi,
>>>> i see no mangling of the message body other than setting the default
>>>> locale to UTF-8 then encode it quoted printable. The header will be
>>>> parsed and changed (invitation etc.).
>>> 
>>> yes, that's it. It's changing the Content-Transfer-Encoding to
>>> quoted-printable. If this is done after dkim signing it breaks 
>>> signature.
>>> 
>>>> It the message contains html, it will be parsed too.
>>>> Since you didn't supply a debug example i can only urge you to 
>>>> enable
>>>> wallace debugging.
>>> 
>>> I can't see anything in debug log - except if smtplib is called the 
>>> data
>>> is already qouted printable:
>>> 
>>> 
>>> 2018-06-02 01:25:54,004 pykolab.wallace INFO Accepted connection
>>> 2018-06-02 01:25:54,018 pykolab.wallace DEBUG [8771]: Resource
>>> Management called for ('/var/spool/pykolab/wallace/tmpk9DNom',), {}
>>> 2018-06-02 01:25:54,019 pykolab.wallace DEBUG [8771]: Renaming
>>> '/var/spool/pykolab/wallace/tmpk9DNom' to
>>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
>>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Nachricht ist
>>> keine iTip Nachricht (keine Multipart Nachricht)
>>> 2018-06-02 01:25:54,020 pykolab.wallace INFO Message is not an iTip
>>> message or does not contain any (valid) iTip.
>>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: No itips, no
>>> resources, pass along
>>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
>>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Invitation 
>>> policy
>>> called for
>>> ('/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',), {}
>>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Invitation 
>>> policy
>>> executing for 
>>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',
>>> False
>>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Renaming
>>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom' to
>>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Nachricht ist
>>> keine iTip Nachricht (keine Multipart Nachricht)
>>> 2018-06-02 01:25:54,022 pykolab.wallace INFO Message is not an iTip
>>> message or does not contain any (valid) iTip objects.
>>> 2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: No itips, no
>>> users, pass along
>>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>>> 2018-06-02 01:25:54,022 pykolab.wallace INFO Akzeptiere Nachricht in
>>> /var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom (durch
>>> Modul wallace)
>>> 2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: Akzeptiere
>>> Nachricht in:
>>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>>> 2018-06-02 01:25:54,023 pykolab.wallace DEBUG [8771]: recipients:
>>> ['test.user at example.net']
>>> send: 'ehlo mx0.example.net\r\n'
>>> reply: '250-mx0.example.net\r\n'
>>> reply: '250-PIPELINING\r\n'
>>> reply: '250-SIZE 20480000\r\n'
>>> reply: '250-VRFY\r\n'
>>> reply: '250-ETRN\r\n'
>>> reply: '250-STARTTLS\r\n'
>>> reply: '250-XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT\r\n'
>>> reply: '250-ENHANCEDSTATUSCODES\r\n'
>>> reply: '250-8BITMIME\r\n'
>>> reply: '250 DSN\r\n'
>>> reply: retcode (250); Msg: mx0.example.net
>>> PIPELINING
>>> SIZE 20480000
>>> VRFY
>>> ETRN
>>> STARTTLS
>>> XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT
>>> ENHANCEDSTATUSCODES
>>> 8BITMIME
>>> DSN
>>> send: 'mail FROM:<test.user at example.net> size=1123\r\n'
>>> reply: '250 2.1.0 Ok\r\n'
>>> reply: retcode (250); Msg: 2.1.0 Ok
>>> send: 'rcpt TO:<test.user at example.net>\r\n'
>>> reply: '250 2.1.5 Ok\r\n'
>>> reply: retcode (250); Msg: 2.1.5 Ok
>>> send: 'data\r\n'
>>> reply: '354 End data with <CR><LF>.<CR><LF>\r\n'
>>> reply: retcode (354); Msg: End data with <CR><LF>.<CR><LF>
>>> data: (354, 'End data with <CR><LF>.<CR><LF>')
>>> send: 'Sender: test.user at example.net\r\nDKIM-Signature: v=1;
>>> a=rsa-sha256; c=relaxed/relaxed; d=example.net;\r\n s=dkim201805;
>>> t=1527895386;\r\n
>>> h=from:from:sender:sender:reply-to:subject:subject:date:date:\r\n
>>> message-id:message-id:to:to:cc:mime-version:mime-version:\r\n
>>> content-type:content-type:\r\n
>>> content-transfer-encoding:content-transfer-encoding:in-reply-to:\r\n
>>> references; bh=IYC/RDnaNtcM6arkuRe/LIW86LUe+V8zvrkFPp/dOoY=;\r\n
>>> b=XjBEXPP/CfSc9RqxX6G+zVW0gorAevrouaSNdQXIx2GhJVvUvheJszeils1SKtRYV7h3oK\r\n
>>> 
>>> ricUH0upeecCDgQJPyGc90aY/JwsoLs2ZpANomt53fQxOJiSyIiuqGbRAyZgsddK0BoW77\r\n
>>> 
>>> +TpL2Xatf9c5u017mxvzAWJngXzD52hV7txlM/gKcGy3SZR48F74JNyGdIJX3qmMBe0dSo\r\n
>>> 
>>> oGj6g0YHN4nTdtvJ995J7eYYgofUJlUglOezF58rQV7n4Vh44pncZZ+vMDKNaQ2h9eKPw6\r\n
>>> 
>>> AcypUALlZvnssOWyuHRBzHqy7Aet2F9dH8sBvOAZCxcLnuOlC8kruqNAF34/WA==\r\nTo:
>>> Test User <test.user at example.net>\r\nFrom: cu-test
>>> <test.user at example.net>\r\nSubject: test encoding 2\r\nMessage-ID:
>>> <7b2f68db-4e2a-1762-a223-08baaef3a380 at example.net>\r\nDate: Sat, 2 
>>> Jun
>>> 2018 01:25:50 +0200\r\nMIME-Version: 1.0\r\nContent-Type: text/plain;
>>> charset=utf-8\r\nContent-Language: 
>>> de-LU\r\nContent-Transfer-Encoding:
>>> quoted-printable\r\n\r\ntest 
>>> encoding2\r\n=C3=A4=C3=B6=C3=BC\r\n.\r\n'
>>> reply: '250 2.0.0 Ok: queued as 09238B14\r\n'
>>> reply: retcode (250); Msg: 2.0.0 Ok: queued as 09238B14
>>> data: (250, '2.0.0 Ok: queued as 09238B14')
>>> send: 'quit\r\n'
>>> reply: '221 2.0.0 Bye\r\n'
>>> reply: retcode (221); Msg: 2.0.0 Bye
>>> 2018-06-02 01:25:54,153 pykolab.wallace DEBUG [9175]: Worker process
>>> PoolWorker-7 initializing
>>> 
>>> 
>>>> Be sure that your default locale on the server is utf-8 !
>>> 
>>> yes. it is.
>>> 
>>> What I actually don't understand: Where exactly wallace is changing 
>>> the
>>> encoding? I send email with 8bit encoding and utf-8.
>>> 
>>> This is my original mail:
>>> 
>>> 
>>> Date: Sat, 2 Jun 2018 01:25:50 +0200
>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>>  Thunderbird/52.8.0
>>> MIME-Version: 1.0
>>> Content-Type: text/plain; charset=utf-8
>>> Content-Language: de-LU
>>> Content-Transfer-Encoding: 8bit
>>> 
>>> test encoding2
>>> äöü
>>> 
>>> (encoded in utf-8 - not quoted printable)
>>> 
>>> I found the function for converting in the footer and invitation
>>> modules. But even if I disable this modules the encoding still is
>>> changed. It's not if I disable wallace in postfix master.cf. I 
>>> wondering
>>> if this is done by python smtplib.
>>> 
>>> My submission part of master.cf:
>>> 
>>> submission          inet        n       -       n       -       -
>>> smtpd
>>>     -o cleanup_service_name=cleanup_submission
>>>     -o syslog_name=postfix/submission
>>>     -o smtpd_tls_security_level=encrypt
>>>     -o smtpd_sasl_auth_enable=yes
>>>     -o smtpd_sasl_authenticated_header=yes
>>>     -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>>     -o smtpd_helo_restrictions=$mua_helo_restrictions
>>>     -o smtpd_data_restrictions=$submission_data_restrictions
>>>     -o 
>>> smtpd_recipient_restrictions=$submission_recipient_restrictions
>>>     -o smtpd_sender_restrictions=$submission_sender_restrictions
>>>     -o content_filter=smtp-wallace:[127.0.0.1]:10026
>>> 
>>> Because I don't have amavis I call wallace directly in submission.
>>> 
>>> Milter is called in main.cf:
>>> 
>>> smtpd_milters = inet:mailpd.example.net:11332
>>> non_smtpd_milters = inet:mailpd.example.net:11332
>>> milter_protocol = 6
>>> milter_mail_macros =  i {mail_addr} {client_addr} {client_name}
>>> {auth_authen}
>>> milter_default_action = accept
>>> 
>>> 
>>> So the mailflow now is: going to prequeue-miltering and signs dkim.
>>> After that there is the content filter set to wallaces - which 
>>> alteres
>>> the message and breaks signature.
>>> 
>>> 
>>>> Also, reordering of the content_filter directive might help.
>>> 
>>> It's not easy, because wallace always is content filter - and milter 
>>> is
>>> prequeue (opendkim or rspamd doesn't matter).
>>> 
>>> What I tried: using rspamd as a content_filter which is possible. So 
>>> I
>>> could pass mails from submission to wallace:
>>> 
>>> submission          inet        n       -       n       -       -
>>> smtpd
>>>     -o cleanup_service_name=cleanup_submission
>>>     -o syslog_name=postfix/submission
>>>     -o smtpd_tls_security_level=encrypt
>>>     -o smtpd_sasl_auth_enable=yes
>>>     -o smtpd_sasl_authenticated_header=yes
>>>     -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>>     -o smtpd_helo_restrictions=$mua_helo_restrictions
>>>     -o smtpd_data_restrictions=$submission_data_restrictions
>>>     -o 
>>> smtpd_recipient_restrictions=$submission_recipient_restrictions
>>>     -o smtpd_sender_restrictions=$submission_sender_restrictions
>>>     # overwrite the default miter - we can't do that on submission,
>>>       because we have first go to wallace
>>>     -o smtpd_milters=
>>>     -o content_filter=smtp-wallace:[127.0.0.1]:10026
>>> 
>>> And then in wallace:
>>> 
>>> # Listener to re-inject email from Wallace into Postfix
>>> 127.0.0.1:10027     inet        n       -       n       -       100
>>> smtpd
>>>     -o cleanup_service_name=cleanup_internal
>>>     -o content_filter=
>>>     -o local_recipient_maps=
>>>     -o relay_recipient_maps=
>>>     -o smtpd_milters=
>>>     -o smtpd_restriction_classes=
>>>     -o smtpd_client_restrictions=
>>>     -o smtpd_helo_restrictions=
>>>     -o smtpd_sender_restrictions=
>>>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>>>     -o mynetworks=127.0.0.0/8
>>>     -o smtpd_authorized_xforward_hosts=127.0.0.0/8
>>>     -o content_filter=smtp:[127.0.0.1]:2525
>>> 
>>> in the last line I call the rspamd content filter which I define 
>>> hiere:
>>> 
>>> # rspamd as content filter
>>> 127.0.0.1:2525 inet  n       -       n       -       -       smtpd
>>>     -o syslog_name=postfix/content-filter
>>>     -o mynetworks=127.0.0.0/8
>>>     -o content_filter=
>>>     -o smtpd_milters=${rspamd}
>>>     -o smtpd_tls_security_level=none
>>>     -o smtpd_client_restrictions=
>>>     -o smtpd_helo_restrictions=
>>>     -o smtpd_sender_restrictions=
>>>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>>>     -o smtpd_relay_restrictions=permit_mynetworks,reject
>>>     -o smtpd_authorized_xforward_hosts=${mynetworks}
>>> 
>>> 
>>> Now everything works - but on smtp we have the mailflow:
>>> 
>>>   -> rspamd as milter -> wallace -> rspamd as content_filter
>>> 
>>> I haven't got any Idea how to either to call rspamd content filter in
>>> case of submission but not of smtpd. Or have one directive where both
>>> content filters are called. As far as I understand there is no 
>>> possility
>>> in postfix to add more then one content_filter without reinjection. 
>>> But
>>> even in this case - still the problem to distinguish between smtpd 
>>> and
>>> submission.
>>> 
>>> Regard
>>> Jan
>>> 
>>>> Am 2018-06-01 10:08, schrieb Jan Kowalsky:
>>>>> Hi Franz,
>>>>> 
>>>>> thanks for answer,
>>>>> 
>>>>> Am 01.06.2018 um 08:44 schrieb Skale, Franz:
>>>>>> Hi,
>>>>>> DKIM uses non standard quoted printable encoding !
>>>>>> See:
>>>>>> http://dkim.org/specs/rfc4871-dkimbase.html#dkim-qp
>>>>> 
>>>>> But as far as I understand, dkim itselfs doesn't change the mail
>>>>> body at
>>>>> all. The qp encoding in my understanding is only for calculating 
>>>>> hash.
>>>>> 
>>>>>> Wallace, of course, uses standard QP encoding and doesn't take 
>>>>>> care
>>>>>> for
>>>>>> special cases, though it's a feature request.
>>>>> 
>>>>> But wallaces alters the mailbody itself. It changes all to quoted
>>>>> printable and, if configured, adds footer/header.
>>>>> 
>>>>> So dkim signing in my experiences is valid if the email already is
>>>>> quoted printable from the mua. Roundcube is fine, thunderbird in
>>>>> default
>>>>> not. But we do not have control about how users are configure there
>>>>> email programs and 8bit email transfer isn't very uncommon any 
>>>>> more.
>>>>> 
>>>>> The problem is, that wallace alteres mails after dkim milter is
>>>>> applied.
>>>>> Even if the encoding is no problem we run in trubles as far as 
>>>>> wallace
>>>>> adds e.g. footers.
>>>>> 
>>>>> Regards
>>>>> Jan
> _______________________________________________
> users mailing list
> users at lists.kolab.org
> https://lists.kolab.org/mailman/listinfo/users