[Mailman-Users] DKIM best practise

Discussion:

Yasir Assam

2015-06-21 01:39:58 UTC

I have a list which all subscribers can post to.

I'm using mailman 2.1.18 on Debian Jessie with exim4. I have full
personlisation and verp turned on.

What should I do about DKIM?

At the moment I preserve the original poster's DKIM header and my list's
MTA also adds DKIM to all outgoing mail.

If I set from_is_list to Munge, hotmail users can't reply to the list,
even when they hit Reply All (if I try doing Reply All from a hotmail
account, I only see the sender's address, not the list address). If it
wasn't for this hotmail problem, I'd probably prefer to have a munged
from header.

If I set from_is_list to No, the hotmail Reply All problem goes away,
but now Yahoo-sent email ends up in Yahoo's spam (i.e. if
***@yahoo.com.au sends to ***@example.com, bob receives the email he
just posted in his spam folder, not in his inbox). I'm specifically
talking about a yahoo.com.au address (I haven't tried yahoo.com yet).

Just to be clear, when from_is_list is No, the DKIM header I'm adding is
for the list domain, e.g. if the list is ***@example.com then
d=example.com in my added DKIM header.

Here's an example Authentication-Results added by a gmail subscriber
receiving a post from a yahoo.com.au subscriber (names changed):

Authentication-Results: mx.google.com;
spf=pass (google.com: domain of
list-bounces+bob=***@example.com designates x.x.x.x as permitted
sender) smtp.mail=list-bounces+bob=***@example.com;
dkim=pass header.i=@example.com;
dmarc=fail (p=NONE dis=NONE) header.from=yahoo.com.au

So what is the recommended way of doing this? Should I not bother adding
a DKIM header to mailman-sent emails? Should I strip the original DKIM
header (REMOVE_DKIM_HEADERS) ?

Is there any way I can get hotmail to reply to the list when the From:
header is munged? Is munging considered bad form (when not mitigating
DMARC reject policies)?

Thanks,
Yasir

------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Mark Sapiro

2015-06-21 15:46:21 UTC

Permalink

Post by Yasir Assam
I'm using mailman 2.1.18 on Debian Jessie with exim4. I have full
personlisation and verp turned on.
What should I do about DKIM?
At the moment I preserve the original poster's DKIM header and my list's
MTA also adds DKIM to all outgoing mail.

This is the good and is the best you can do.

Post by Yasir Assam
If I set from_is_list to Munge, hotmail users can't reply to the list,
even when they hit Reply All (if I try doing Reply All from a hotmail
account, I only see the sender's address, not the list address). If it
wasn't for this hotmail problem, I'd probably prefer to have a munged
from header.

With Munge From and Full Personalization, delivered posts will be From:
the list with Reply-To: the poster and To: the recipient. Hotmail is
taking the Reply-To: as overriding the From: even for reply-all and with
Full Personalization, the From: is the only header (other than
List-Post) with the list address.

Post by Yasir Assam
If I set from_is_list to No, the hotmail Reply All problem goes away,
but now Yahoo-sent email ends up in Yahoo's spam (i.e. if
just posted in his spam folder, not in his inbox). I'm specifically
talking about a yahoo.com.au address (I haven't tried yahoo.com yet).

Yahoo.com.au publishes DMARC p=none. Yahoo.com publishes DMARC p=reject.
Without some Munge From, Wrap Message or anonymous_list transformation,
yahoo.com mail will not be accepted by Yahoo, Hotmail and many other ISPs.

As far as the mail from yahoo.com.au ending up in spam, removing the
broken DKIM sig may help (REMOVE_DKIM_HEADERS = 2). It shouldn't matter
(see below), but it may help.

Post by Yasir Assam
Just to be clear, when from_is_list is No, the DKIM header I'm adding is
d=example.com in my added DKIM header.
Here's an example Authentication-Results added by a gmail subscriber
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of
dmarc=fail (p=NONE dis=NONE) header.from=yahoo.com.au
So what is the recommended way of doing this? Should I not bother adding
a DKIM header to mailman-sent emails? Should I strip the original DKIM
header (REMOVE_DKIM_HEADERS) ?

What you are doing is correct and good practice. Removing incoming DKIM
headers probably won't help. The DKIM standard says an invalid DKIM
signature and no DKIM signature SHOULD be treated the same (RFC 6376/STD
76, sec 6.3)

Post by Yasir Assam
header is munged? Is munging considered bad form (when not mitigating
DMARC reject policies)?

There are a few things you can do.

You can turn off Full Personalization which will leave the list address
in To: and Hotmail's reply-all should include it.

You can set reply_to_list to this list which will put the list address
in Reply-To: (along with the poster's address), but this will make it
more difficult to reply only to the poster as a simple reply will also
include the list.

There are some changes in this area in 2.1.19 (see
<https://bugs.launchpad.net/mailman/+bug/1407098>), but I don't think
they help your situation.

You could modify CookHeaders.py to add the poster's address to Cc:
rather than Reply-To: in your case.

You should also consider using dmarc_moderation_action rather than
from_is_list to only Munge From when 'necessary'.
--
Mark Sapiro <***@msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Yasir Assam

2015-06-22 02:32:23 UTC

Permalink

Many thanks for your response Mark.

Comments below.

Post by Mark Sapiro

This is the good and is the best you can do.

I noticed that this list, mailman-***@python.org, doesn't add a DKIM
header unless the list itself generates the email, i.e. the email you
sent to this list only has your DKIM header (d=msapiro.net), whereas the
original welcome email has DKIM with d=python.org. On my list, I'm
adding a DKIM header for the list domain, even though the From: header
isn't the list. In other words, if ***@gmail.com posts to
***@example.com, my MTA was still adding a d=example.com DKIM header
when resending her email via the list, even when From: is ***@gmail.com.

Is it right to do this?

I subscribed to mailman-***@python.org using a Yahoo address, and
interestingly, 2 emails ended up in spam (one of which was my original
post, which is from a non-yahoo address).

Post by Mark Sapiro

the list with Reply-To: the poster and To: the recipient. Hotmail is
taking the Reply-To: as overriding the From: even for reply-all and with
Full Personalization, the From: is the only header (other than
List-Post) with the list address.

This isn't quite true in my case. You're right about all the headers,
except that Mailman is adding a CC field with the list address. To be
clear, using Full Personalisation and Munge From:

From: list address
Reply-to: poster
To: recipient
CC: list address

Given that CC contains the list address, you'd expect Hotmail to include
it as a recipient when doing Reply All, but it doesn't!

Post by Mark Sapiro

Yes, I read about this recently. My test list doesn't yet contain any
yahoo.com addresses.

Post by Mark Sapiro
As far as the mail from yahoo.com.au ending up in spam, removing the
broken DKIM sig may help (REMOVE_DKIM_HEADERS = 2). It shouldn't matter
(see below), but it may help.

I think I tried this but it didn't make a difference to Yahoo's spam filter.

Post by Mark Sapiro

Post by Yasir Assam
header is munged? Is munging considered bad form (when not mitigating
DMARC reject policies)?

There are a few things you can do.
You can turn off Full Personalization which will leave the list address
in To: and Hotmail's reply-all should include it.

Hotmail doesn't include it. I tried Full Personalisation off, Munge on,
with the following headers:

From: list address
Reply-to: poster
To: list address

In Hotmail, Reply All only includes poster, not list address (despite
list address appearing in To field)

Post by Mark Sapiro
You can set reply_to_list to this list which will put the list address
in Reply-To: (along with the poster's address), but this will make it
more difficult to reply only to the poster as a simple reply will also
include the list.

I used to have this on years ago, and what happened was that repliers
were sending private mail to the whole list when they only intended it
for the original poster. I ended up getting complaints and had to turn
it off.

Post by Mark Sapiro
There are some changes in this area in 2.1.19 (see
<https://bugs.launchpad.net/mailman/+bug/1407098>), but I don't think
they help your situation.
rather than Reply-To: in your case.

If munging is on, and I put the poster address in CC: rather than
Reply-to: won't that mean a single Reply (not Reply All) will go to the
list address?

Post by Mark Sapiro
You should also consider using dmarc_moderation_action rather than
from_is_list to only Munge From when 'necessary'.

The reason I tried from_is_list = Munge is because I wanted valid DKIM
headers (added by my MTA). When I used Munge with DKIM added to all list
mail, yahoo didn't mark any mail as spam. I would have stuck with it if
I didn't have the hotmail Reply All problem.

Thanks,
Yasir

------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Stephen J. Turnbull

2015-06-22 03:58:05 UTC

Permalink

DKIM header unless the list itself generates the email, i.e. the
email you sent to this list only has your DKIM header
(d=msapiro.net), whereas the original welcome email has DKIM with
d=python.org.

IIUC, Mark has input into, but does not control, policy on
mail.python.org. People have different experience with, and therefore
opinions on policy, about these things.

As Mark already said, according to the standards it is correct and
good practice to add a DKIM signature to every message you process
outside of the MTA and then reinject into the Internet mail system.
In more friendly terms, if you simply pass on the message *exactly* as
received except for adding "Received" and 2List-Post" to the front of
the message, you don't need to DKIM sign but it doesn't hurt. But if
you change the message (eg, by adding a list signature or by adding
the list name to the Subject field), you *should* DKIM sign.

On my list, I'm adding a DKIM header for the list domain, even
though the From: header isn't the list. In other words, if
a d=example.com DKIM header when resending her email via the list,
Is it right to do this?

Yes.

interestingly, 2 emails ended up in spam (one of which was my original
post, which is from a non-yahoo address).

Yahoo and Hotmail are a child's garden of diseases when it comes to
their behavior in the mail system. Outlook and Gmail also cause
problems. It would be OK if there was only one 800 pound gorilla
around, you'd just adapt. The problem is that there are several, and
they have conflicting requirements. You can't satisfy them all.

This isn't quite true in my case. You're right about all the headers,
except that Mailman is adding a CC field with the list address.

I don't think that Mailman adds the CC:.

Given that CC contains the list address, you'd expect Hotmail to
include it as a recipient when doing Reply All, but it doesn't!

Then you can't workaround both Hotmail's broken MUA and yahoo.com's
idiotic DMARC policy, and provide full reply functionality to people
with decent MUAs. There may be a workaround for both Hotmail's broken
MUA and yahoo.com.au's spam filtering policy, but we can't design one
without accurate information about yahoo.com.au's policy, and they are
very unlikely to provide it.

The only thing that will satisfy all parties is to turn off all list
decorations: no header or footer in the body, and no change to the
Subject field.

Post by Mark Sapiro
rather than Reply-To: in your case.

If munging is on, and I put the poster address in CC: rather than
Reply-to: won't that mean a single Reply (not Reply All) will go to the
list address?

Yes. It definitely won't go to the poster.

------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Yasir Assam

2015-06-22 06:10:27 UTC

Permalink

Post by Stephen J. Turnbull
As Mark already said, according to the standards it is correct and
good practice to add a DKIM signature to every message you process
outside of the MTA and then reinject into the Internet mail system.
In more friendly terms, if you simply pass on the message *exactly* as
received except for adding "Received" and 2List-Post" to the front of
the message, you don't need to DKIM sign but it doesn't hurt. But if
you change the message (eg, by adding a list signature or by adding
the list name to the Subject field), you *should* DKIM sign.

Got it, thanks.

Post by Stephen J. Turnbull

Post by Yasir Assam
interestingly, 2 emails ended up in spam (one of which was my original
post, which is from a non-yahoo address).

I was hoping there was a way to pacify all these gorillas, but it looks
like there isn't.

Post by Stephen J. Turnbull

Post by Yasir Assam
This isn't quite true in my case. You're right about all the headers,
except that Mailman is adding a CC field with the list address.

I don't think that Mailman adds the CC:.

I think it does in my version of Mailman (2.1.18 on Debian Jessie).

So it sounds like I should do the following:

* DKIM sign all mail sent by mailman
* Don't remove existing DKIM headers
* from_is_list = No
* REMOVE_DKIM_HEADERS = No
* dmarc_moderation_action = wrap message

I'll tell my users to keep checking their spam folders and marking my
mail as not spam, and hope that yahoo and others learn to flag my list
mail as not spam over time.

Thanks,
Yasir

------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Mark Sapiro

2015-06-23 05:02:37 UTC

Permalink

Post by Stephen J. Turnbull

DKIM header unless the list itself generates the email, i.e. the
email you sent to this list only has your DKIM header
(d=msapiro.net), whereas the original welcome email has DKIM with
d=python.org.

IIUC, Mark has input into, but does not control, policy on
mail.python.org. People have different experience with, and therefore
opinions on policy, about these things.

Steve's understanding is correct.

Right.

But, we are actually dealing with two issues here: DKIM signing as a
general practice and DKIM signing specifically to address DMARC issues.

Yes, it is good practice to DKIM sign for your domain all mail which is
sent by servers in your domain. You are essentially saying yes, I made
transformations to this message that broke its original DKIM signature,
but I am taking responsibility for this message and if my DKIM sig is
valid, I vouch for this mail.

DMARC however puts a more stringent requirement on a message. It says
that if a message is From: a domain that publishes a DMARC policy, and
there isn't a valid SPF or DKIM signature whose domain 'aligns' (i.e. is
the same as in some sense) with the domain in the From: address,
recipients should handle the message in accord with the From: domain's
DMARC policy.

Thus, as a mailing list that makes modifications to messages that break
DKIM sigs, it doesn't help a message From: ***@yahoo.com pass DMARC for
me to DKIM sign it with my domain unless I also change the From: address
to my domain or at least to a domain without a DMARC policy other than
"none".
--
Mark Sapiro <***@msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Mark Sapiro

2015-06-22 02:33:10 UTC

Permalink

Post by Yasir Assam
header is munged? Is munging considered bad form (when not mitigating
DMARC reject policies)?

It may not have been clear from my earlier reply, but yes, From Munging
is considered bad form. It violates the basic email RFCs definition of
the meaning of the From: header. It is best not to do it
unconditionally, and a better response to DMARC mitigation is Wrap
Message, but that said we recognize that Wrap Message creates issues for
some (particularly moblie device) MUAs so that in some cases at least,
Munge From is the most acceptable mitigation. It is best if only applied
when needed via dmarc_moderation_action and not unconditionally via
from_is_list.

On a different subject, I have looked a bit further, and with what I
understand your list settings to be, i.e. Full Personalization,
from_is_list = Munge From, reply_goes_to_list = Poster; I think even
with 2.1.18 the list posting address will be put in or added to Cc:, and
the original From: will be in Reply-To:.

Thus any reasonable MUA should 'reply' to the original From: and
'reply-all' to the original From: and the list. If the headers are as I
say and Hotmail is not doing this, then Hotmail is behaving badly as the
Reply-To: should override the list address in the From:, but it
shouldn't affect replying-all to a Cc:.
--
Mark Sapiro <***@msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net

Yasir Assam

2015-06-22 03:00:27 UTC

Permalink

Thanks Mark.

What you describe below makes sense, and I agree hotmail is behaving
badly, but I'm stuck with its bad behaviour.

In any case, if From Munging is not good to do generally then I have to
figure out another way placate Yahoo's spam filter.

Yasir

Post by Mark Sapiro

Post by Yasir Assam
header is munged? Is munging considered bad form (when not mitigating
DMARC reject policies)?

------------------------------------------------------
Mailman-Users mailing list Mailman-***@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/q0nk%40ml-in.narkive.net