wallace breaks dkim signature

[Jan Kowalsky]

Hi Franz,


Am 02.06.2018 um 09:08 schrieb Skale, Franz:
> Hi Jan,
> respamd must take care of the data, regardless of which encoding stream
> injected.

dkim signing doesn't change (and shouldn't) anything of the body. If
it's utf-8 8bit it will be utf-8 8bit after signing.

Problem is (without milter or not) that wallace seems to change the
transfer encoding always to qouted printable which in theory is fine.
Mailclients don't do this always. Some send with utf-8 and 8bit transfer
encoding. Which should be fine too.

As wallace comes into play after dkim signing - the problem is, that
wallace changes the transfer encoding to quoted printable. if the mail
has non-ascii characters even if the original message (before and after
signing) have had transfer encoding 8bit.

This breaks of course signature. There is nothing rspamd can care about.
The utf-8 8bit is totally fine (and in the responsibility of the user
agent).

> As i develop webservices, you've to guess the encoding prior before
> mangling the data stream.
> Whit utf-8 you have the ability to call is_utf8  which tells you, if the
> the utf8 (2 bytes) representation) is either native or encoded.
> Input has to be encoded and output decoded.
> Regarding your problem, i think that respamd doesn't check, if the input
> stream is alreasy quoted-printable encoded.

dkim signing (with rspamd as milter) is proceeded before mail goes to
wallace.

> I think you've to file a bug report.

Yes, I'll do. I'll try to understand how wallace get's data from smtpd -
but I didn't get it so far.

> Pykolab only set's encoding for string handling, obviously the way to go.
> /usr/lib/python2.7/dist-packages/wallace/module_resources.py:   
> charset.add_charset('utf-8', charset.SHORTEST, charset.QP)
> /usr/lib/python2.7/dist-packages/wallace/module_resources.py:        msg
> = MIMEText(utils.stripped_message(message_text), _charset='utf-8')
> So, it's utf-8 encoded.

but why it's still quoted printable if I disable all wallace modules?
For me it looks like the smtpd.py already get's data in quoted printable
no matter the original encoding.

in __init__.py

    def process_message(self, peer, mailfrom, rcpttos, data):


if I print data to debug - it's already quoted-printable. But I didn't
find out where the data variable comes from. So I fail with further
debugging.

> Ergo, respamd has to decode the input stream (utf8_decode) when it's not
> in native encoded.
> The client sets the msg_content_type (Content-Type).
> So, if the client uses wrong localesm than , of course the string
> representation is wrong.
> Respamd has to take care about decoding the string according to the
> Content-Type.
> I think you've to file a bug report for respamd.


Well, there is everything fine with encoding. rspamd doesn't touch
anything on encoding or Content type. Just parsing email and sign's dkim.

For me it looks totally fine to send emails with utf8 and 8bit (not
quoted printable) like some clients do. There is no need for changing it
on the way.

My guess is, that wallace in principle sets transfer encoding to quoted
printable as long it's not transferrable with 7bit. But I can't find
where it does.

Regards
Jan

> 
> 
> Rgds.
> Franz
> 
> Am 2018-06-02 03:10, schrieb Jan Kowalsky:
>> Hi Franz,
>>
>> Am 01.06.2018 um 11:17 schrieb Skale, Franz:
>>> Hi,
>>> i see no mangling of the message body other than setting the default
>>> locale to UTF-8 then encode it quoted printable. The header will be
>>> parsed and changed (invitation etc.).
>>
>> yes, that's it. It's changing the Content-Transfer-Encoding to
>> quoted-printable. If this is done after dkim signing it breaks signature.
>>
>>> It the message contains html, it will be parsed too.
>>> Since you didn't supply a debug example i can only urge you to enable
>>> wallace debugging.
>>
>> I can't see anything in debug log - except if smtplib is called the data
>> is already qouted printable:
>>
>>
>> 2018-06-02 01:25:54,004 pykolab.wallace INFO Accepted connection
>> 2018-06-02 01:25:54,018 pykolab.wallace DEBUG [8771]: Resource
>> Management called for ('/var/spool/pykolab/wallace/tmpk9DNom',), {}
>> 2018-06-02 01:25:54,019 pykolab.wallace DEBUG [8771]: Renaming
>> '/var/spool/pykolab/wallace/tmpk9DNom' to
>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Nachricht ist
>> keine iTip Nachricht (keine Multipart Nachricht)
>> 2018-06-02 01:25:54,020 pykolab.wallace INFO Message is not an iTip
>> message or does not contain any (valid) iTip.
>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: No itips, no
>> resources, pass along
>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
>> 2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Invitation policy
>> called for
>> ('/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',), {}
>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Invitation policy
>> executing for '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',
>> False
>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Renaming
>> '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom' to
>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>> 2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Nachricht ist
>> keine iTip Nachricht (keine Multipart Nachricht)
>> 2018-06-02 01:25:54,022 pykolab.wallace INFO Message is not an iTip
>> message or does not contain any (valid) iTip objects.
>> 2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: No itips, no
>> users, pass along
>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>> 2018-06-02 01:25:54,022 pykolab.wallace INFO Akzeptiere Nachricht in
>> /var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom (durch
>> Modul wallace)
>> 2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: Akzeptiere
>> Nachricht in:
>> '/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
>> 2018-06-02 01:25:54,023 pykolab.wallace DEBUG [8771]: recipients:
>> ['test.user at example.net']
>> send: 'ehlo mx0.example.net\r\n'
>> reply: '250-mx0.example.net\r\n'
>> reply: '250-PIPELINING\r\n'
>> reply: '250-SIZE 20480000\r\n'
>> reply: '250-VRFY\r\n'
>> reply: '250-ETRN\r\n'
>> reply: '250-STARTTLS\r\n'
>> reply: '250-XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT\r\n'
>> reply: '250-ENHANCEDSTATUSCODES\r\n'
>> reply: '250-8BITMIME\r\n'
>> reply: '250 DSN\r\n'
>> reply: retcode (250); Msg: mx0.example.net
>> PIPELINING
>> SIZE 20480000
>> VRFY
>> ETRN
>> STARTTLS
>> XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT
>> ENHANCEDSTATUSCODES
>> 8BITMIME
>> DSN
>> send: 'mail FROM:<test.user at example.net> size=1123\r\n'
>> reply: '250 2.1.0 Ok\r\n'
>> reply: retcode (250); Msg: 2.1.0 Ok
>> send: 'rcpt TO:<test.user at example.net>\r\n'
>> reply: '250 2.1.5 Ok\r\n'
>> reply: retcode (250); Msg: 2.1.5 Ok
>> send: 'data\r\n'
>> reply: '354 End data with <CR><LF>.<CR><LF>\r\n'
>> reply: retcode (354); Msg: End data with <CR><LF>.<CR><LF>
>> data: (354, 'End data with <CR><LF>.<CR><LF>')
>> send: 'Sender: test.user at example.net\r\nDKIM-Signature: v=1;
>> a=rsa-sha256; c=relaxed/relaxed; d=example.net;\r\n s=dkim201805;
>> t=1527895386;\r\n
>> h=from:from:sender:sender:reply-to:subject:subject:date:date:\r\n
>> message-id:message-id:to:to:cc:mime-version:mime-version:\r\n
>> content-type:content-type:\r\n
>> content-transfer-encoding:content-transfer-encoding:in-reply-to:\r\n
>> references; bh=IYC/RDnaNtcM6arkuRe/LIW86LUe+V8zvrkFPp/dOoY=;\r\n
>> b=XjBEXPP/CfSc9RqxX6G+zVW0gorAevrouaSNdQXIx2GhJVvUvheJszeils1SKtRYV7h3oK\r\n
>>
>> ricUH0upeecCDgQJPyGc90aY/JwsoLs2ZpANomt53fQxOJiSyIiuqGbRAyZgsddK0BoW77\r\n
>>
>> +TpL2Xatf9c5u017mxvzAWJngXzD52hV7txlM/gKcGy3SZR48F74JNyGdIJX3qmMBe0dSo\r\n
>>
>> oGj6g0YHN4nTdtvJ995J7eYYgofUJlUglOezF58rQV7n4Vh44pncZZ+vMDKNaQ2h9eKPw6\r\n
>>
>> AcypUALlZvnssOWyuHRBzHqy7Aet2F9dH8sBvOAZCxcLnuOlC8kruqNAF34/WA==\r\nTo:
>> Test User <test.user at example.net>\r\nFrom: cu-test
>> <test.user at example.net>\r\nSubject: test encoding 2\r\nMessage-ID:
>> <7b2f68db-4e2a-1762-a223-08baaef3a380 at example.net>\r\nDate: Sat, 2 Jun
>> 2018 01:25:50 +0200\r\nMIME-Version: 1.0\r\nContent-Type: text/plain;
>> charset=utf-8\r\nContent-Language: de-LU\r\nContent-Transfer-Encoding:
>> quoted-printable\r\n\r\ntest encoding2\r\n=C3=A4=C3=B6=C3=BC\r\n.\r\n'
>> reply: '250 2.0.0 Ok: queued as 09238B14\r\n'
>> reply: retcode (250); Msg: 2.0.0 Ok: queued as 09238B14
>> data: (250, '2.0.0 Ok: queued as 09238B14')
>> send: 'quit\r\n'
>> reply: '221 2.0.0 Bye\r\n'
>> reply: retcode (221); Msg: 2.0.0 Bye
>> 2018-06-02 01:25:54,153 pykolab.wallace DEBUG [9175]: Worker process
>> PoolWorker-7 initializing
>>
>>
>>> Be sure that your default locale on the server is utf-8 !
>>
>> yes. it is.
>>
>> What I actually don't understand: Where exactly wallace is changing the
>> encoding? I send email with 8bit encoding and utf-8.
>>
>> This is my original mail:
>>
>>
>> Date: Sat, 2 Jun 2018 01:25:50 +0200
>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>  Thunderbird/52.8.0
>> MIME-Version: 1.0
>> Content-Type: text/plain; charset=utf-8
>> Content-Language: de-LU
>> Content-Transfer-Encoding: 8bit
>>
>> test encoding2
>> äöü
>>
>> (encoded in utf-8 - not quoted printable)
>>
>> I found the function for converting in the footer and invitation
>> modules. But even if I disable this modules the encoding still is
>> changed. It's not if I disable wallace in postfix master.cf. I wondering
>> if this is done by python smtplib.
>>
>> My submission part of master.cf:
>>
>> submission          inet        n       -       n       -       -
>> smtpd
>>     -o cleanup_service_name=cleanup_submission
>>     -o syslog_name=postfix/submission
>>     -o smtpd_tls_security_level=encrypt
>>     -o smtpd_sasl_auth_enable=yes
>>     -o smtpd_sasl_authenticated_header=yes
>>     -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>     -o smtpd_helo_restrictions=$mua_helo_restrictions
>>     -o smtpd_data_restrictions=$submission_data_restrictions
>>     -o smtpd_recipient_restrictions=$submission_recipient_restrictions
>>     -o smtpd_sender_restrictions=$submission_sender_restrictions
>>     -o content_filter=smtp-wallace:[127.0.0.1]:10026
>>
>> Because I don't have amavis I call wallace directly in submission.
>>
>> Milter is called in main.cf:
>>
>> smtpd_milters = inet:mailpd.example.net:11332
>> non_smtpd_milters = inet:mailpd.example.net:11332
>> milter_protocol = 6
>> milter_mail_macros =  i {mail_addr} {client_addr} {client_name}
>> {auth_authen}
>> milter_default_action = accept
>>
>>
>> So the mailflow now is: going to prequeue-miltering and signs dkim.
>> After that there is the content filter set to wallaces - which alteres
>> the message and breaks signature.
>>
>>
>>> Also, reordering of the content_filter directive might help.
>>
>> It's not easy, because wallace always is content filter - and milter is
>> prequeue (opendkim or rspamd doesn't matter).
>>
>> What I tried: using rspamd as a content_filter which is possible. So I
>> could pass mails from submission to wallace:
>>
>> submission          inet        n       -       n       -       -
>> smtpd
>>     -o cleanup_service_name=cleanup_submission
>>     -o syslog_name=postfix/submission
>>     -o smtpd_tls_security_level=encrypt
>>     -o smtpd_sasl_auth_enable=yes
>>     -o smtpd_sasl_authenticated_header=yes
>>     -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>     -o smtpd_helo_restrictions=$mua_helo_restrictions
>>     -o smtpd_data_restrictions=$submission_data_restrictions
>>     -o smtpd_recipient_restrictions=$submission_recipient_restrictions
>>     -o smtpd_sender_restrictions=$submission_sender_restrictions
>>     # overwrite the default miter - we can't do that on submission,
>>       because we have first go to wallace
>>     -o smtpd_milters=
>>     -o content_filter=smtp-wallace:[127.0.0.1]:10026
>>
>> And then in wallace:
>>
>> # Listener to re-inject email from Wallace into Postfix
>> 127.0.0.1:10027     inet        n       -       n       -       100
>> smtpd
>>     -o cleanup_service_name=cleanup_internal
>>     -o content_filter=
>>     -o local_recipient_maps=
>>     -o relay_recipient_maps=
>>     -o smtpd_milters=
>>     -o smtpd_restriction_classes=
>>     -o smtpd_client_restrictions=
>>     -o smtpd_helo_restrictions=
>>     -o smtpd_sender_restrictions=
>>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>>     -o mynetworks=127.0.0.0/8
>>     -o smtpd_authorized_xforward_hosts=127.0.0.0/8
>>     -o content_filter=smtp:[127.0.0.1]:2525
>>
>> in the last line I call the rspamd content filter which I define hiere:
>>
>> # rspamd as content filter
>> 127.0.0.1:2525 inet  n       -       n       -       -       smtpd
>>     -o syslog_name=postfix/content-filter
>>     -o mynetworks=127.0.0.0/8
>>     -o content_filter=
>>     -o smtpd_milters=${rspamd}
>>     -o smtpd_tls_security_level=none
>>     -o smtpd_client_restrictions=
>>     -o smtpd_helo_restrictions=
>>     -o smtpd_sender_restrictions=
>>     -o smtpd_recipient_restrictions=permit_mynetworks,reject
>>     -o smtpd_relay_restrictions=permit_mynetworks,reject
>>     -o smtpd_authorized_xforward_hosts=${mynetworks}
>>
>>
>> Now everything works - but on smtp we have the mailflow:
>>
>>   -> rspamd as milter -> wallace -> rspamd as content_filter
>>
>> I haven't got any Idea how to either to call rspamd content filter in
>> case of submission but not of smtpd. Or have one directive where both
>> content filters are called. As far as I understand there is no possility
>> in postfix to add more then one content_filter without reinjection. But
>> even in this case - still the problem to distinguish between smtpd and
>> submission.
>>
>> Regard
>> Jan
>>
>>> Am 2018-06-01 10:08, schrieb Jan Kowalsky:
>>>> Hi Franz,
>>>>
>>>> thanks for answer,
>>>>
>>>> Am 01.06.2018 um 08:44 schrieb Skale, Franz:
>>>>> Hi,
>>>>> DKIM uses non standard quoted printable encoding !
>>>>> See:
>>>>> http://dkim.org/specs/rfc4871-dkimbase.html#dkim-qp
>>>>
>>>> But as far as I understand, dkim itselfs doesn't change the mail
>>>> body at
>>>> all. The qp encoding in my understanding is only for calculating hash.
>>>>
>>>>> Wallace, of course, uses standard QP encoding and doesn't take care
>>>>> for
>>>>> special cases, though it's a feature request.
>>>>
>>>> But wallaces alters the mailbody itself. It changes all to quoted
>>>> printable and, if configured, adds footer/header.
>>>>
>>>> So dkim signing in my experiences is valid if the email already is
>>>> quoted printable from the mua. Roundcube is fine, thunderbird in
>>>> default
>>>> not. But we do not have control about how users are configure there
>>>> email programs and 8bit email transfer isn't very uncommon any more.
>>>>
>>>> The problem is, that wallace alteres mails after dkim milter is
>>>> applied.
>>>> Even if the encoding is no problem we run in trubles as far as wallace
>>>> adds e.g. footers.
>>>>
>>>> Regards
>>>> Jan