Characters in email body messed up

Summary

When sending email from MODX3 via either QuickEmail or FormIt, the body’s characters do not display properly when email received. The subject’s characters are fine.

Step to reproduce

Send any email with characters like ö, ü, ó, ú, ő, é, á, ű, or í.

Observed behavior

The character “ü” displays as “ü” for example.

Expected behavior

Characters like “á” should display as “á”.

Environment

MODX 3.0.0., MySQL collation utf8mb4-general-ci (tried utf8-general-ci, too)

Check the content-type header of the received emails in your email client. You can see it with “Show Original” at Gmail.

It should look something like this:

Content-Type: text/html; charset="Something"

Another thing to check is the actual character encoding in the DB. Look at the structure tab in PhpMyAdmin (or the equivalent) to make sure that the tables and any text fields are really set to utf8mb4.

Look especially at the tables that contain the text you’re sending.

Another possibility: If you switched the DB from another character set, it could be that the database charset is utfmb4, but the data in it is in the older encoding. If the characters display properly on the site, this probably isn’t the issue.

Also, check the charset in the ‘core/config/config.inc.php’ file

Thanks for answering so quickly!

Unfortunately, everything is and was as it should be, but the problem persists:

- Content-type header of the received emails in your email client.

X-Mailer: PHPMailer 6.6.0 (https://github.com/PHPMailer/PHPMailer)
MIME-Version: 1.0
Content-Type: text/html; charset=UTF-8

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>Lieber Roland Tomb&Atilde;&iexcl;cz!<br><br>

- The actual character encoding in the DB. Look at the structure tab in PhpMyAdmin (or the equivalent) to make sure that the tables and any text fields are really set to utf8mb4. Look especially at the tables that contain the text you’re sending.

laser2

Database server
Server: Localhost via UNIX socket
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.2.43-MariaDB-log-cll-lve - MariaDB Server
Protocol version: 10
User: nqjnxhqb@localhost
Server charset: UTF-8 Unicode (utf8)
Web server
cpsrvd 11.102.0.8
Database client version: libmysql - mysqlnd 7.4.27
PHP extension: mysqliDocumentation curlDocumentation mbstringDocumentation
PHP version: 7.4.27
phpMyAdmin
Version information: 4.9.7

Checks out.

- Another possibility: If you switched the DB from another character set, it could be that the database charset is utfmb4, but the data in it is in the older encoding. If the characters display properly on the site, this probably isn’t the issue.

They do display properly. :upside_down_face:

  • Also, check the charset in the ‘core/config/config.inc.php’ file
$database_connection_charset = 'utf8';
...
$database_dsn = 'mysql:host=localhost;dbname=nqjnxhqb_laser;charset=utf8';

I don’t know :frowning_face: I have the exact same setup and code in modx2.8.3 working flawlessly on numerous sites. I have even tried using different SMTP services (google included). Same problem. Even your QuickEmail sends bogus characters when I change the default message in your snippet.

Any other ideas?

Also, as I pointed out in my first post, the email subject’s characters are fine. Please check in the image attached that the same “ü” character is OK in the subject, NOK in the body, and both come from chunks. I know it’s obvious, but just to be clear, this FormIt email uses a tpl for the email body, populating variables from a form. It messes up the characters it gets from the form, too.

:confused:

What are the values of the system settings mail_charset and mail_encoding?

Hi @halftrainedharry :slight_smile:

UTF-8, 8bit

Strangely, what displays is actually ISO-8859-1… How can that be possible?

I can reproduce the issue.

Everything is ok if I set $modx->mail->setHTML(false);
But if html is set to true, I get the same messed up characters.

The problem occurs here, when the class InlineStyle is used. But I don’t know how to fix it.


As a temporary fix, try adding a meta tag with the charset to your mail body chunk.

$modx->mail->set(modMail::MAIL_BODY,'<head><meta charset="UTF-8"></head><body>some text with umlauts ä, ü, ö, etc.</body>');
1 Like

@halftrainedharry

Then we identified a bug! :slight_smile:

You were spot on with the HTML issue. Got around the problem by switching off HTML in my FormIt call:

&emailHtml=`0`
&fiarHtml=`0`

Not ideal, though :frowning_face:

I created an issue on github.

1 Like

Does the MODX3 upgrade change the character coding setup in the DB? My 2.8.3 tables are all utf8, not utf8mb4.

It does not. It will work with what you have. But the character encoding will be the least of your worries if you rely on exotic extras. Do check out whether your extras are MODX3-compatible before upgrading here: https://sitedash.app/extras

In a very old extra of mine called emailresource (latest commit 2011), There a class for converting CSS to inline styles.

I can’t guarantee that it would work any better than what you have. But I think the extra is fairly easy to install and use, and you could see if it makes a difference. If it does, and you’re desparate, you could alter your code to use the class.

1 Like

Try to set in the system settings
locale = en_US.UTF-8

Unfortunately, this doesn’t solve the issue.

I have a potential patch here, would very much appreciate some real world testing on that to make sure it works consistently:

2 Likes

It works! Running live on my site as we speak.

1 Like

This topic was automatically closed 2 days after discussion ended and a solution was marked. New replies are no longer allowed. You can open a new topic by clicking the link icon below the original post or solution and selecting “+ New Topic”.