Tuning FIM MA exports during initial load (pre R2)

I wanted to share my findings on following the steps outlined in KB2417774 to improve FIM MA export performance on a pre-R2 implementation I’m working on (build 4.0.3617.2, with 200K metaverse objects under management).

When faced with thousands of pending exports (mostly adds) to the FIM Service and observing an export rate of approximately 5 every 30 seconds, I figured it was time to give this “async” mode a try … given that it now appears to be the default mode in FIM R2 anyhow.

The following config changes resulted in a throughput improvement which varied between 1 and 3 per second.  This was the kind of improvement I was hoping for … it meant that the exports were processed in a couple of hours instead of closer to 24 hours.  However I did find there were some drawbacks … which are all fine now that I know what to expect … so this post is for when you find yourself in the same situation (and also a diary note-to-self!!!).

Here is my current miiserver.exe.config setting for the FIM sync service (this goes immediately below the </startup> node):

<resourceSynchronizationClient 
 exportFetchResultsPollingTimerInSeconds="4"
 exportRequestsInProcessMaximum="100"
 exportWaitingForRequestsToProcessTimeoutInSeconds="600" />
<system.serviceModel>

… and here is the one for the FIM service (Microsoft.ResourceManagement.Service.exe.config … goes immediately below the </system.serviceModel> node):

<resourceManagementService externalHostName="myFIMServer.com" synchronizationExportThrottle="Limited" requestRecoveryMaxPerMinute="60" />

The first time I tried this, however, I used the “Unlimited” option and found that I eventually ran out of resources on my (slightly under-spec’d) SQL instance.  It also made accessing the FIM portal a problem (timeouts), and while I expected this because the KB warned me, I nonetheless found this too much of a hindrance in monitoring progress on the Request history.  I subsequently found that the above setting gave me the through-put I was looking for, whilst making the FIM portal more responsive.

I have since reset the Microsoft.ResourceManagement.Service.exe.config setting back to the following:

<resourceManagementClient resourceManagementServiceBaseAddress="myFIMServer.com" />

… noting that while the R2 notes say that this is now the default, there is nothing to say this is a recommended default pre-R2.  Having said that you may find your environment has more resources than mine, and this may be workable … but for the moment I feel much more comfortable with the default configuration once the initial load is done with.

There are a couple of catches to note when doing this:

1. Make sure your tempdb database on your SQL server has room to grow.  By default this is possibly on your C: drive, as it was for me, and I noticed just in time that tempdb had grown to 17Gb (!!!) and I was running with about 3% free disk space.  Luckily I was able to (semi-) gracefully bring everything to a stop and remedy the situation by relocating this database to where I had diligently placed the FIM ones by following the simple steps outlined in this helpful SQL blog post.
2. Make sure you’ve got plenty of RAM on your SQL server.  With tempdb growing like it does in this mode, you need to have RAM headroom to spare.  I doubled my lab VM (hosting SQL and the FIM sync service) from 8 to 16 Gb and would have doubled it again if I could!  As it was, I followed FIM SQL best practice by setting a RAM limit on SQL (7Gb) and found that this was at least workable.
3. Expect the FIM service to take its time processing the queued requests.  If you chose the “Unlimited” option like I did first, expect FIM to potentially take hours to clear the queue.  Even with the “Limited” option you can expect delays.
4. Recycle the FIMService and FIMSynchronizationService (and SQL if you can, to bring tempdb back to a reasonable size) after you’ve finished the initial load phase.  With resources stretched I found that I needed to do this to get a performing FIM environment as quickly as possible.
5. Re-build the full text catalog on your FIMService database after the initial load (and defrag all the indexes on this db if you want to go one step further like me).

P.S. Thanks to those responsible for awarding me with an MVP this month … it’s a great honour (that’s the English spelling!) and I’m proud to say that with Carol having returned home from Geneva in the last couple of years, Australia (and also UNIFY!) now has 2 of the current headcount of 11 FIM MVPs :).  I appreciate your continuing support.