[tlug] A Test Message in Japanese

["Stephen J. Turnbull" <turnbull.stephen.fw@example.com>]

BLUF: Unless TLUG is sending different copies of the messages to other
people (Mailman's not doing that because personalization is not on),
this is is a receiver-side problem for this test message, which is
trivial as far as message structure goes.  I'm pretty sure it's not a
sender-side problem in general -- but that doesn't mean TLUG can't (or
shouldn't) do something to help people with non-conforming clients.

If you're having problems with this test message, though, I'd say it's
time to upgrade or switch clients if you care about reading non-ASCII
mail.

Curt J. Sampson writes:

 > This is to test the conversions our list server is doing on
 > foreign-language messages. If you see this message in garbled form,
 > rather than a simple Japanese sentence below, please let me know.
 > (Probably a reply just to me is easier; I'll soon summarize my results
 > to the list.)

People who see it garbled should try to get the raw message (also
called "original" by some MUAs), and send the header.  Here's mine
with commentary.  If you know what you're doing you can restrict to a
few interesting headers as I do, but it's usually simpler to just send
the whole thing unless there's local information you want to hide.  I
reverse the usual convention; quoted material is *not* marked, my
comments are marked with a prefixed "* ".

* MTA fields (Return-Path and trace fields) omitted.
* These should be ignored by MUAs, or perhaps the MUA will blow up if
  incorrect.
* DKIM authentication field omitted.  Same.

X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2HOKAAEGkRe/9pKnoJlBoI9gT4CARF?=
 =?us-ascii?q?VbFhZhBSDSYRcXqFngXQPAQEBAQEBAQEBIAcOAgEBhFmCMyQ8Ag0CEAEBBQE?=
 =?us-ascii?q?BAQEBBQQCAmmFCwgkDIRGLAgFTwVJASYCIBEMAQEECAIeDAIDAQIGAggcAgU?=
 =?us-ascii?q?TCgQCAgIBAQsZBAUWFggLBRgEgwaCegEBBI96mwR1gTKCfwEBBXeBPYUCAwa?=
 =?us-ascii?q?BDigDAQGGAYY0BoIAhXCBVgMCgUqDKIJekFuHM5dsBwOCOoZkaIkrhUwogjg?=
 =?us-ascii?q?QiBKQO5dUk0OBIQyBWHsKO4JsUBgNhwyOCodcZYEIkFkBAQ?=
X-IronPort-AV: E=Sophos;i="5.70,433,1574089200"; 
   d="scan'208";a="18989682"

* Nothing to see here except the identity of the spam filter and
  antivirus.  This is important, because many filters alter the
  message.

MIME-Version: 1.0

* conformant

Content-Disposition: inline

* bizarre but conformant.  I don't think Mailman is inserting it; it
  seems to be the sending client.

User-Agent: NeoMutt/20170113 (1.7.2)

* I remember neodogs from some SF novel or other.  conformant

X-BeenThere: tlug@example.com
X-Mailman-Version: 2.1.26
Precedence: list
List-Unsubscribe: <http://lists.tlug.jp/mailman/options/tlug>,
 <mailto:tlug-request@example.com?subject=unsubscribe>
List-Archive: <https://lists.tlug.jp/ML/>
List-Post: <mailto:tlug@example.com>
List-Help: <mailto:tlug-request@example.com?subject=help>
List-Subscribe: <http://lists.tlug.jp/mailman/listinfo/tlug>,
 <mailto:tlug-request@example.com?subject=subscribe>

* 8 fields of Mailman goodness, conformant

Content-Type: text/plain; charset="utf-8"

* conformant but suspect -- I wouldn't be surprised if the problem
  manifests only in a multipart content-type.

Content-Transfer-Encoding: base64

* conformant

Errors-To: tlug-bounces+turnbull.stephen.fw=u.tsukuba.ac.jp@example.com
List-Id: Tokyo Linux Users Group <tlug.tlug.jp>

* more Mailman goodness.  Bizarre that it follows the Content-Type and
* Content-Transfer-Encoding fields.  I guess that's Mailman's doing.
* various addressing and identification fields omitted
* I realized later that I should have reported Message-ID, mea maxima
  culpa.  Very useful for confirming you're looking at the same message.

VGhpcyBpcyB0byB0ZXN0IHRoZSBjb252ZXJzaW9ucyBvdXIgbGlzdCBzZXJ2ZXIgaXMgZG9pbmcg
b24KZm9yZWlnbi1sYW5ndWFnZSBtZXNzYWdlcy4gSWYgeW91IHNlZSB0aGlzIG1lc3NhZ2UgaW4g
Z2FyYmxlZCBmb3JtLApyYXRoZXIgdGhhbiBhIHNpbXBsZSBKYXBhbmVzZSBzZW50ZW5jZSBiZWxv
dywgcGxlYXNlIGxldCBtZSBrbm93LgooUHJvYmFibHkgYSByZXBseSBqdXN0IHRvIG1lIGlzIGVh
c2llcjsgSSdsbCBzb29uIHN1bW1hcml6ZSBteSByZXN1bHRzCnRvIHRoZSBsaXN0LikKCuOBk+OB
oeOCieOBr+aXpeacrOiqnuOBp+OBlOOBluOBhOOBvuOBmeOAggoKY2pzCi0tIApDdXJ0IEouIFNh
bXBzb24gICAgICA8Y2pzQGN5bmljLm5ldD4gICAgICArODEgOTAgNzczNyAyOTc0CgpUbyBpdGVy
YXRlIGlzIGh1bWFuLCB0byByZWN1cnNlIGRpdmluZS4KICAgIC0gTCBQZXRlciBEZXV0c2NoCgot
LSAKVG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIG1haWxpbmcgbGlzdCwKcGxlYXNlIHNlZSB0aGUg
aW5zdHJ1Y3Rpb25zIGF0IGh0dHBzOi8vbGlzdHMudGx1Zy5qcC9saXN0Lmh0bWwK

* conformant content-transfer-encoding (length of each line is a
  multiple of 4) and the content decodes sanely

* Here's J. Hart's message, which some reported as mojibake:

X-Gm-Message-State: APjAAAUC3sGf7Vc8484Fa65VDCTCRQhZ18F98aOFVUU2TDblfEynOeTQ
 Guq/rQ80a8TIuuiqsZza8ae7DNM9
X-Google-Smtp-Source: APXvYqxI+ayB16ClHas62R6UUpi2yUTUegczkoFPL0EZRKtEpngXbHzIpd1eYbbPhRboBWwLey8Njw==

* Hrm.  If I were QAnon, I'd blame Google.

User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130512 Thunderbird/17.0.5

* Report User-Agent and X-Mailer fields for sure.  The composing MUA
  could be doing something weird.

MIME-Version: 1.0

* conformant

References: <20200212152631.mchqpmoat553elmz@example.com>
In-Reply-To: <20200212152631.mchqpmoat553elmz@example.com>

* conformant threading information

X-BeenThere: tlug@example.com
X-Mailman-Version: 2.1.26
Precedence: list
List-Unsubscribe: <http://lists.tlug.jp/mailman/options/tlug>,
 <mailto:tlug-request@example.com?subject=unsubscribe>
List-Archive: <https://lists.tlug.jp/ML/>
List-Post: <mailto:tlug@example.com>
List-Help: <mailto:tlug-request@example.com?subject=help>
List-Subscribe: <http://lists.tlug.jp/mailman/listinfo/tlug>,
 <mailto:tlug-request@example.com?subject=subscribe>

* conformant Mailman goodness (shouldn't affect display unless
* receiving client is very very very VERY brokked)

Content-Transfer-Encoding: base64
Content-Type: text/plain; charset="utf-8"; Format="flowed"

* Always report these when discussing encoding issues.
* conformant but bizarre.  There is zero flowed content below, and
  capitalizing the parameter tag "Format" is uh, unusual.
  format=flowed doesn't seem to be used much nowadays.  Many clients
  assume that other clients will reformat paragraphs and ignore the
  RFC's strong recommendation that lines be broken to less than 74
  halfwidth characters.  I'm not sure it's an RFC recommendation, but
  in general it's considered a bad idea to use parameters to specify
  defaults (such as "Content-Disposition: inline" for content-type
  "text" above) and features that aren't used ("format=flowed" here)
  because many clients Do Wrong Things with "rare" features.
* NB: my guess about multipart content-type was wrong.

Errors-To: tlug-bounces@example.com
List-Id: Tokyo Linux Users Group <tlug.tlug.jp>
Sender: "Tlug" <tlug-bounces@example.com>

* Mailman goodness.
* addressing info omitted.
* Now the content:

bG9va3MgZ29vZCBoZXJlLgoKT24gMDIvMTMvMjAyMCAxMjoyNiBBTSwgQ3VydCBKLiBTYW1wc29u
IHdyb3RlOgo+IFRoaXMgaXMgdG8gdGVzdCB0aGUgY29udmVyc2lvbnMgb3VyIGxpc3Qgc2VydmVy
IGlzIGRvaW5nIG9uCj4gZm9yZWlnbi1sYW5ndWFnZSBtZXNzYWdlcy4gSWYgeW91IHNlZSB0aGlz
IG1lc3NhZ2UgaW4gZ2FyYmxlZCBmb3JtLAo+IHJhdGhlciB0aGFuIGEgc2ltcGxlIEphcGFuZXNl
IHNlbnRlbmNlIGJlbG93LCBwbGVhc2UgbGV0IG1lIGtub3cuCj4gKFByb2JhYmx5IGEgcmVwbHkg
anVzdCB0byBtZSBpcyBlYXNpZXI7IEknbGwgc29vbiBzdW1tYXJpemUgbXkgcmVzdWx0cwo+IHRv
IHRoZSBsaXN0LikKPgo+IOOBk+OBoeOCieOBr+aXpeacrOiqnuOBp+OBlOOBluOBhOOBvuOBmeOA
ggo+Cj4gY2pzCgoKLS0gClRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBtYWlsaW5nIGxpc3QsCnBs
ZWFzZSBzZWUgdGhlIGluc3RydWN0aW9ucyBhdCBodHRwczovL2xpc3RzLnRsdWcuanAvbGlzdC5o
dG1sCg==

* conformant (line lengths divisible by 4), decodes to:

looks good here.

On 02/13/2020 12:26 AM, Curt J. Sampson wrote:
> This is to test the conversions our list server is doing on
> foreign-language messages. If you see this message in garbled form,
> rather than a simple Japanese sentence below, please let me know.
> (Probably a reply just to me is easier; I'll soon summarize my results
> to the list.)
>
> こちらは日本語でございます。
>
> cjs


-- 
To unsubscribe from this mailing list,
please see the instructions at https://lists.tlug.jp/list.html

* I have no idea where Scott's client is getting the "\#" thing from,
  and the decoded bytes in his message are clearly not UTF-8 (there
  should be no ASCII characters there at all; Japanese represented in
  UTF-8 is all 8-bit-set bytes).  Scott's client doesn't announce
  itself in the header.

* Jim Blackson writes:

I also have seen this problem. First garbled message was the 
[tlug] [announcement] Nomikai - June 14th, 2019 sent by "Justin"
<tlug@example.com>.  The header included the lines: 
Content-Type: text/plain; charset="utf-8"
...
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by altair.a1d.local id x549aTJA090625

This same message received on the TLUG-Admin list had a different
encoding: 
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

* The difference in charsets makes no sense to me, since they're the
  same message.  The only thing I can think of is that Mailman is
  changing the charset on a plain text message when a footer is added
  (tlug does, tlug-admin does not), but as far as I can tell there's
  no need (the footer is 100% ASCII).
* X-MIME-Autoconverted is like spam- and virus-checking: it's altering
  the message content in ways which should not, but in practice could,
  confuse clients.  Mailman is not doing it, and I don't think it's a
  TLUG host (can't tell since the domain name isn't on the Internet).
  It's also possible that it does the conversions wrong.  (Believe it
  or not, there are different versions of BASE64, which use different
  punctuation characters -- 52 letters + 10 ASCII digits = 62, and you
  need 65: 64 base64 digits, and a padding character.  The padding
  character is always "=", and the MIME convention for email is that
  the other two digits are "+" and "/", but "-", "~", and/or "_" are
  used in other applications.  Eg, "/" is not file-system safe, and
  "+" has special meaning in URLs.)

Steve