Category Archives: Exchange 2016

Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

The Issue:

So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

User12 StalledDueToTarget_BigFunnel

image

Error Message

Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

————–

The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

Database: DB01

Error:

The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

image

Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

To dig deeper I am going to search some of the users reporting the same error by using their GUID

image

The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

Solution: (For a single user)

To resolve the problem, I am going to remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

image

Solution: (For all users)

Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

clip_image001

Some relevant errors you might encounter as you’re moving users to Exchange 2019

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

Search and Delete certain Items/Folders from a Mailbox

The Story

During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

The Error after migration:

Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

image

After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

Solution applied:

First, Let’s see the user’s dumpster and recoverable items

Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

image

To Delete the dumpster only use this

Delete dumpster only

Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

To delete a certain email with certain subject in the dumpster use the following:

Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

image

The cmdlet will search and delete

clip_image001

image

Reference:

https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help

Slow Migration – Office 365

The story:

In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.

One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:

Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server

Error in Office 365

         : 20.

                                           27.04.2016 08:03:17 [DB3PR05MB0778] Transient error DataExportTransientExcep

                                           tion has occurred. The system will retry (2/1280).

                                           27.04.2016 08:04:53 [DB3PR05MB0778] The Microsoft Exchange Mailbox Replicati

                                           on service ‘DB3PR05MB0778.eurprd05.prod.outlook.com’ (15.1.466.25 caps:03FFF

                                           F) is examining the request.

                                           27.04.2016 08:04:55 [DB3PR05MB0778] Connected to target mailbox ‘lcwonline.o

                                           nmicrosoft.com\ec96e315-1059-4710-b358-1c4b42f3edeb (Primary)’, database ‘EU

                                           RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’

                                           Version 15.1 (Build 466.0).RequestExpiryTimestamp                   : 03.04.2116 07:42:38

ObjectState                              : New

Troubleshooting:

To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.

Things that could always come in mind and handy are what you will need to start your troubleshooting:

– Bandwidth Limitations or Performance:

https://technet.microsoft.com/en-us/library/dn592150(v=exchg.150).aspx

https://technet.microsoft.com/en-us/library/jj200581(v=exchg.150).aspx

– Exchange Configuration (MRS)

To troubleshoot the MRs, You need to know what kind of error you’re getting and to see this you can use the following powershell after you connect to Office 365 powershell.

Get-MoveRequest {email} | Get-MoveRequestStatistics -Diagnostic -IncludeReport | Export-Clixml c:\logfile.xml

The resultant report will reveal the error and shows you where is the exact culprit.

– Disk Latency
– Firewall Configuration (IPS/IDS)

From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.

SOLUTION:

===========

1. Some instability was detected in communications as well as saturation by the size of the link.
2. The procedure to increase the timeout for the service through the file MRSProxy

File: MsExchangeMailboxReplication.exe.config

Object / line: DataImportTimeout.

New Value: 00:10:00

clip_image001[4]

New Configuration

clip_image001[6]

3- Unified messaging Integration between Exchange 2016 and Skype for Business

Setting up UM

To setup UM between Exchange and Skype for business server, the most important step is how you configure the Certificates between both servers in order for them to trust each other.

For that you don’t have to use a public Certificate but rather an internal CA certificate that has its root certificate installed on all of the server where you intend to deploy the UM. (Exchange, S4B Servers..etc.).

To claim this certificate, the easiest step would be to get the CSR from Skype for Business’s Deployment Wizard

Run Deployment Wizard and click on the “Install or Update skype for business Server system”

clip_image001

Then click on step 3 (request, install or assign Cert)

clip_image002

I already have certificate deployed for S4B service but I’ll request CSR again to get one trusted certificate for both Exchange and S4B.

I will tick only the services that matters as in the below screenshot (Server default and Web services internal) later also will be used for OWA integration with UM.

clip_image003

Click on Request

clip_image004

Click on Advanced

clip_image005

Next

clip_image006

I’ll continue next until I’ve got to the important part which is “Name and Security settings” I’ll need to tick the “Mark the certificate’s private key as exportable” since we’ll export the certificate to Exchange servers

clip_image007

Next I’ll add Exchange servers’s FQDNs.

clip_image008

clip_image009

Click Next

clip_image010

clip_image011

Here is the CMDLET

Request-CSCertificate -New -Type Default,WebServicesInternal -CA “DC2016.moh10ly.com\moh10ly-DC2016-CA” -Country “TR” -FriendlyName “Skype for Business Server 2015 Default certificate 3/18/2016” -KeySize 2048 -PrivateKeyExportable $True -Organization “moh10ly” -OU “moh10ly” -DomainName “sip.moh10ly.com,ex2016.moh10ly.com,ex2016-2.moh10ly.com,ex2010.moh10ly.com” -AllSipDomain -Verbose -Report “C:\Users\administrator.MOH10LY\AppData\Local\Temp\2\Request-CSCertificate-[2016_03_18][11_16_35].html”

Click Next again and mark the thumbprint for the new Cert as we’ll need to see it later to make sure it’s properly configured for the UM on Exchange.

8BA9A2C4CD926B01C029F6B9A76D75BBEFDDE069

clip_image012

Click next to assign the Cert

clip_image013

clip_image014

Successfully, the certificate has been assigned to the Services

clip_image015

The CMDLET that was applied

Set-CSCertificate -Type Default,WebServicesInternal -Thumbprint 8BA9A2C4CD926B01C029F6B9A76D75BBEFDDE069 -Confirm:$false -Report “C:\Users\administrator.MOH10LY\AppData\Local\Temp\2\Set-CSCertificate-[2016_03_18][11_19_06].html”

Now it’s time to export this certificate and import it to Exchange servers

clip_image016

I’ll find the certificate that I have created today by looking at the expiration date which is 2 years from now with the same day.

clip_image017

Now I’ll right click on the certificate and export it with the private key.

clip_image018

I’ll open Exchange EMC and import the certificate

clip_image019

I’ll have to put the exported cert in a shared folder and provide the path and the password for it

clip_image020

I’ll add the two servers below

clip_image021

clip_image022

I’ll double click on the imported certificate and assign the UM services to it on each of the servers

clip_image023

clip_image024

I have got the below error due to not configuring the service to use TLS instead of TCP on both servers.

clip_image025

To fix this I’ll go on Exchange Management shell and run the following CMDLET

Get-UMService | Set-UMService -UMStartupMode TLS

clip_image026

clip_image027

Now I’ll try to save again

clip_image028

clip_image029

I’ll proceed with YES and continue to do the same to the other Server and restart the UM service on both servers

clip_image030

Now it’s time to create a UM Dial plan

clip_image031

I’ll configure the UM Dial plan according to my Skype for Business settings for users enabled for EV

clip_image032

To use powershell, you can use the following cmdlet

New-UMDialPlan –Name DialplanName –UriType SIPURI –NumberOfDigitsInExtension 4 –VoIPSecurity Secured –CountryOrRegionCode 1 –AccessTelephoneNumber +9012345678

Next, adding a gateway to the UM (NOTE: If configured incorrect, will cause the service not to start and errors with event ID (1057, 4999,1430, 1038) will appear.

Time to configure Gateway

clip_image033

In the gateway I’ll add my PBX (AsteriskNow) and place my already configured UMDP

clip_image034

clip_image035

When you create the dial plan, Exchange automatically creates a new UM mail policy along with it and it also generates a name that’s related to the Dial plan

In order to see this policy, you will have to double click on the new dial plan to view it and you can also change the policy in it .. Which I’m going to apply for the length of the policy to make it shorter

clip_image036

Double click on the Mailbox policy and navigate to Pin Polices and change it to the length you want to allow

clip_image037

Configure Auto Attendant

clip_image038

Set the AA as how you want it to be configured and make sure you add the full E.164 format as it won’t accept otherwise.

clip_image039

Click Save to continue

Now time to configure OVA (Outlook voice access)

Subscriber Access

If you want to configure Outlook Voice Access (OVA) , sometimes also referred to as Subscriber Access, click on the Configure button. Select Outlook Voice Access in the left hand menu and enter the telephone number you want to use to access OVA. This must be in the E.164 notation.

clip_image040

To do so click on Configure

clip_image041

To assign the new dial plan to the UM services, both on the Client Access Server (UM Call Router) as well as on the Mailbox server. In an Exchange Management Shell windows enter the following commands:

1

2

Set-UMCallRouterSettings -DialPlans “Exchangelabs Dial Plan” –Server 2012E15FE04

Get-UMService | Set-UMService -DialPlans “Exchangelabs Dial Plan”

clip_image042

clip_image043

Now I’ll also change the UM call router to TLS and assign Certificate to the service then restart it

clip_image044

clip_image045

Restart the services of the Call router, then associate the service with the dialplan you created.

Set-UMCallRouterSettings -DialPlans “UMDP1” –Server EX2016

Set-UMCallRouterSettings -DialPlans “UMDP1” –Server EX2016-2

Configure Skype for Business Server

To configure the UM Service to be used with Skype for Business Server. Microsoft has a script that will create and configure all necessary components. This scripts is located in the scripts directory C:\Program Files\Microsoft\Exchange Server\V15\Scripts.

Run the following CMDLET

CD $ExScripts

.\ExchUCUtil.ps1

clip_image046

The first time you setup this script it’ll detect the Dial plan and set it up with Skype for Business Server

clip_image047

It will show that no setting has changed but the fact that the dial plan is showing here Not found means that there something has changed .. You’ll notice that if you run the same script again.

clip_image048

Let’s try it again

Here you can see that the dial plan has been assigned to the S4B Front end server.

clip_image049

This script performs the following:

  • Grants Skype for Business Server permission to read Exchange UM Active Directory components, specifically, the SIP URI dial plan that was created in the first step;
  • Creates a UM IP gateway for each Skype for business Server pool that hosts users who will be enabled for Enterprise Voice;
  • Create an Exchange UM hunt group for each UM IP gateway. The hunt group pilot identifier will be the name of the dial plan associated with the corresponding UM IP gateway. The hunt group must specify the UM SIP dial plan used with the UM IP gateway.

When the script has run you’ll see a new UM IP Gateway appear in the EAC. Since this script not only creates the UM IP Gateway but also sets the necessary permissions the UM IP Gateway was not created manually in the first step.

clip_image050

Next we’ll go to Skype for Business FE server and then run the OcsUmUtil.exe tool which creates the contact objects for Outlook Voice Access and for the auto attendants. This tool can be found in C:\Program Files\Common Files\Skype for Business Server 2015\Support

clip_image051

I’ll right click the file to run it as administrator

clip_image052

Click on Load Data

clip_image053

clip_image054

Select the SIP dial plan and click ADD

clip_image055

Click OK

Right after configuring this your Voice mail should be enabled once you enable your user for it

After I enable user for UM and assign a valid dialplan .. Now I can see the user has got his Voice Mail option available.

clip_image056

Hope this was useful

clip_image057

—-

UM gateway

clip_image058

clip_image059

clip_image060