Emails between O365 and On-premises do not work

When sending an e-mail from O365 migrated users to On-premise users the On-premise users  don’t get e-mails.

Failure Message

Delivery has failed to these recipients or groups:

User (User@domain.com.tr)
The server has tried to deliver this message, without success, and has stopped trying. 

Please try sending this message again. If the problem continues, contact your helpdesk.
 

User2 ( Company ) (User2@domain.com.tr)
The server has tried to deliver this message, without success, and has stopped trying. 

Please try sending this message again. If the problem continues, contact your helpdesk.
 

Diagnostic information for administrators:

Generating server: DB4PR03MB532.eurprd03.prod.outlook.com
Receiving server: emea01-internal.map.protection.outlook.com (10.47.216.25)
 

User (User@domain.com.tr)
4/4/2014 7:35:30 PM – Remote Server at emea01-internal.map.protection.outlook.com (10.47.216.25) returned ‘550 4.4.7 QUEUE.Expired; message expired’

4/4/2014 7:27:34 PM – Remote Server at emea01-internal.map.protection.outlook.com (10.47.216.25) returned ‘450 4.7.0 Proxy session setup failed on Frontend with ‘451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was 210.179.31.5:25”

User2 ( Company ) (User2@domain.com.tr)
4/4/2014 7:35:30 PM – Remote Server at emea01-internal.map.protection.outlook.com (10.47.216.25) returned ‘550 4.4.7 QUEUE.Expired; message expired’

4/4/2014 7:27:34 PM – Remote Server at emea01-internal.map.protection.outlook.com (10.47.216.25) returned ‘450 4.7.0 Proxy session setup failed on Frontend with ‘451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was 210.179.31.5:25”

Original message headers:

Received: from DB4PR03MB610.eurprd03.prod.outlook.com (10.141.234.156) by DB4PR03MB532.eurprd03.prod.outlook.com (10.141.235.143) with Microsoft SMTP Server (TLS) id 15.0.908.10; Wed, 2 Apr 2014 19:31:29 +0000 Received: 

from DB4PR03MB610.eurprd03.prod.outlook.com (10.141.233.156) by DB4PR03MB610.eurprd03.prod.outlook.com 

(10.141.234.156) with Microsoft SMTP Server (TLS) id 15.0.898.11; Wed, 2 Apr 2014 12:49:18 +0000 Received: from DB4PR03MB610.eurprd03.prod.outlook.com ([10.141.233.156]) by DB4PR03MB620.eurprd03.prod.outlook.com 

([10.141.233.156]) with mapi id 15.00.0913.002; Wed, 2 Apr 2014 12:49:17 +0000 Content-Type: multipart/mixed; boundary=”_000_2c4cf07ee43e4faab98dc52f068a566fDB4PR03MB620eurprd03pro_” 

 From: test <test@domain.com.tr> To: “User ( Company )” <user@domain.com.tr>, “User2 ( Company )” <User2@domain.com.tr> Subject: deneme Thread-Topic: deneme Thread-Index: Ac9Oce26frtuRTMySYWFyAvAom/lyQ== Date: Wed, 2 Apr 2014 12:49:16 +0000 Message-ID: <2c4cf07ee43e4faab98dc52f068a566f@DB4PR03MB620.eurprd03.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: <2c4cf07ee43e4faab98dc52f068a566f@DB4PR03MB620.eurprd03.prod.outlook.com> x-originating-ip: [78.186.201.28] X-Forefront-Antispam-Report: SFV:SKI;SFS:;DIR:INB;SFP:;SCL:-1;SRVR:DB4PR03MB610;H:DB4PR03MB620.eurprd03.prod.outlook.com;FPR:;LANG:tr;;SKIP:2; MIME-Version: 1.0 X-MS-Exchange-CrossPremises-AuthAs: Internal X-MS-Exchange-CrossPremises-AuthMechanism: 03 X-MS-Exchange-CrossPremises-AuthSource: DB4PR03MB620.eurprd03.prod.outlook.com X-MS-Exchange-CrossPremises-SCL: -1 X-MS-Exchange-CrossPremises-messagesource: StoreDriver X-MS-Exchange-CrossPremises-BCC: X-MS-Exchange-CrossPremises-originalclientipaddress: 78.186.201.28 X-MS-Exchange-CrossPremises-avstamp-service: 1.0 X-MS-Exchange-CrossPremises-antispam-scancontext: DIR:Originating;SFV:SKI;SKIP:0; X-MS-Exchange-CrossPremises-processed-by-journaling: Journal Agent X-MS-Exchange-CrossPremises-ContentConversionOptions: True;00160000;True;; X-OrganizationHeadersPreserved: DB4PR03MB610.eurprd03.prod.outlook.com Return-Path: test@domain.com.tr X-OriginatorOrg: domain.com

Symptoms

When you try to telnet the Office 365 hub transport from Exchange on-premises server it won’t recognize the telnet commands on the SMTP server.

Resolution:

451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS commnd first” Office 365 Hybrid

If you have an Office 365 hybrid configuration you may experience issues sending emails between on premise and cloud users (in either direction).

The Exchange 2013 (or 2010) on premises queue viewer may show:

‘451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was xxx.xxx.xxx.xxx’

The Office 365 Message Trace Console shows the delivery status of ‘None’
 


The errors suggest the TLS connection cannot be made but a TLS certificate IS present and during the Hybrid Connection Wizard the required connectors are automatically created so should not require an additional configuration.

When an email is sent between on premise & cloud (Office 365) users of your SSO domain it is sent across one of the automatically created send connectors. These connectors are secured using TLS.

So, assuming you have ruled out all the normal stuff its now time to get baffled. We know the on premise server can send and receive external email. We also know that the Office 365 service can send and receive email. It is just the email between the two services that does not work.

I was banging my head against a wall for ages until I used Telnet to connect from my on premise Exchange server to Microsoft cloud gateway.

What I got is shown below:

This is not correct. As you can see the server has not recognised the “ehlo” statement and the banner does not “look right”…

A bit of digging around the firewall I noticed that packets were being dropped when TLS was attempted.

The firewall is a Cisco PIX 515. I disabled ESMTP inspection but that made no difference so I discounted this as the cause.

After a lot more digging around and raging I remembered that the PIX was behind another Cisco firewall – this time an ASA 5510. So I accessed this device and sure enough this edge firewall was also inspecting and dropping TLS over SMTP.

Once both firewall were configured not to inspect ESMTP the default configuration that was set by the Hybrid Configuration Wizard started working straight away.

The commands to disable ESMTP inspection are:

pix(config)#policy-map global_policy
pix(config-pmap)#class inspection_default
pix(config-pmap-c)#no inspect esmtp
pix(config-pmap-c)#exit
pix(config-pmap)#exit
Now telnet the cloud server and you should see a correct banner: