SharePoint performance debugging

Written by Cornelius J. van Dyk on . Posted in Blog

We released a SharePoint 2010 portal to a new client today.  Part of the portal design included a report center library that contained security trimmer report folders which are updated asynchronously via WCF services.  In addition there was an external report center application that we wanted to link into the SharePoint UI so we simply added a Page Viewer Web part to the page that pointed to the external app.

The problem was that we were seeing major performance lag in rendering the external report app UI.  When we pointed to the development server, the UI rendered slick and fast, but when we pointed back to the production server, we experienced lag again.  And thus the trace/debug efforts began.

My standard rule when hunting bugs/performance issues is to “Assume NOTHING”.  Always compare apples to apples and double, tripper, quadruple check everything.

Our analysis of the differences turned up the following:

  1. The architectural difference between the production server and the development server was that the development server had it’s SQL source local to the server while the production server had a separate SQL server it was targeting.  In order to eliminate SQL Server from the equation, we pointed the development server to the production SQL data source and tested against it from the production environment.  The result was fast and responsive which meant that SQL Server wasn’t the problem.
  2. This narrowed the problem down to the report server and the client computer.  Next we compared the production and development report servers.  Since both servers were virtualized, we checked the resource settings in the vSphere console.  It turned out that the servers were identical except for the development server having 4 CPU cores while the production server only had 2 CPU cores.  We upped the production CPU cores to 4 and rebooted the VM.  Testing against the new configuration still showed slow, lagging performance in the UI rendering.  That pretty much eliminated the reports servers from the equation unless one of the VMs was actually bad.
  3. The next step was to install Fiddler2 on the client laptop and trace the two different calls to see where the traffic was going.  Fiddler did show some minor differences in header and body sizes of the HTTP calls, but not nearly enough to justify the lag we were seeing.  The development server was rendering in under a second while the production server took 5-6 seconds.  What struck me as curious was the fact that Fiddler was reporting that the total execution time of the HTTP call was 0.6 seconds in both cases.  So where exactly did that other 5 seconds go in the case of the production server?  Upon closer inspection, we noticed that the development SharePoint server had it’s properties for the Page Viewer Web Part set to point directly to the IP address of the development report server while the production SharePoint server’s Page Viewer was pointing to the report server using the FQDN instead.  And that’s when it TeaKayGee’d me.  The primary DNS server was flakey and after repointing to the secondary DNS server, performance was back as expected!

I was amazed at how big a difference the DNS request performance could make in this case.  That’s one more item I can add to my checklist when trouble shooting SharePoint performance issues. 



Tags: , , ,

Trackback from your site.

Cornelius J. van Dyk

Born and raised in South Africa during the 70's I got my start in computers when a game on my Sinclair ZX Spectrum crashed, revealing it's BASIC source code. The ZX had a whopping 48K of memory which was considered to be a lot in the Commodore Vic20 era, but more importantly, it had BASIC built into the soft touch keyboard. Teaching myself to program, I coded my first commercial program at age 15.

After graduating high school at 17, I joined the South African Air Force, graduating the Academy and becoming a Pilot with the rank of First Lieutenant by age 20. After serving my country for six years, I made my way back into computer software.

Continuing my education, I graduated Suma Cum Laude from the Computer Training Institute before joining First National Bank where my work won the Smithsonian Award for Technological Innovation in the field of Banking and Insurance. Soon I met Will Coleman from Amdahl SA, who introduced me to a little known programming language named Huron/ObjectStar. As fate would have it, this unknown language and Y2K brought me to the USA in 1998.

I got involved with SharePoint after playing around with the Beta for SharePoint Portal Server 2003. Leaving my career at Rexnord to become a consultant in 2004, I was first awarded the Microsoft Most Valuable Professional Award for SharePoint in 2005, becoming only the 9th MVP for WSS at the time. I fulfilled a life long dream by pledging allegiance to the Flag as a US citizen in 2006. I met the love of my life and became a private consultant in 2008. I was honored to receive my ninth MVP award for SharePoint Server in 2013.

Leave a comment

You must be logged in to post a comment.