But Does It Work?

IE 9.0 Memory Grows by 6+ MB when Refreshing SharePoint Pages

Dear Microsoft,

Could you please fix IE 9.0!  When I use IE 9.0 (x32) on WIndows 7 to view SharePoint (Enterprise Edition) pages (any page), I see that the IE 9 memory process grows by about 6 MB everytime I refresh the page or navigate to a new page.  Since I work with clients that spend a lot of time on the SharePoint pages that I build for them, it means that local machine memory consumption goes up rapidly (100 clicks --> .6GB !) and they believe that SharePoint is slow and causes the browser to crash frequently.

I have done quite a bit of research on this issues and have concluded that the culprit is IE 9.0 (x32), most likely the add-in "NameCtrl Class."  Here is what I have found:

 

1. Important metnions of this probelm
       IE 9 and SharePoint memory usage is extreme Posted (May 23, 2011)
      ?IE 9 and SharePoint 2010 memory leak (Posted June 27, 2011)

      The thing to note is that the problem has been around for a while,
      and then the blog trail went silent.

2. The problem causes IE to crash after some time, in my case every twenty minutes.  The problem also causes my Windows machine to slow down.

3. The prolem only occurs with IE 9:

    Not with Firefox, not with Safari, not with IE 8.

4. I have observed the problem on several corporate enterprise servers (both 2007 & 2010), on a virgin installation of enterprtise server running on my own Virtual Machine, as well as on Office 365.

5. NOTE: The problem does not exist when navigating pages of WSS 3.0 or Founddation Server.

6. If you have a single IE 9.0 session and tab open, when you naviagte to a different domain, e.g. nytimes.com, IE 9.0 releases the memory and the process size goes down.

This is a trivial way to see the problem yourself.

Step 1:  Navigate to the any page that is being served up by SharePoint 2007 or 2010 enterprise server.

Step 2: Launch the Windows Task Manager and click on the Processes tab.

Step 4: Note the process size of iexplorer.exe *32.  (NOTE: There is usually more than one running)

Step 5: Refresh the page from step 1.

Step 6: Note the process size of one of the iexplorer.exe *32 instances should have increased. 

Step 7. Repeat steps 1->6 several times.  You will see that the there is no limit to the growth of the process size for IE 9.

 

The screenshots below show a Before (125,532K) and an After (131,492K) a single refresh, for an increase of 5.96 MB.  Note the screenshots were taken of a site that was built on Office 365.

 

BEFORE

AFTER

If I find a solution to this issue, I will post it here.

The InfoPath Eco System (SharePoint Saturday Boston 4-28-2012)

Some people have had trouble with this link:

Download SharePoint Saturday April 2012 InfoPath 2.pptx (1680.4K)

You should be able to get the file from:

HERE  (You might be asked for credentials, just cancel and the deck should open)

 

Also see my original post on the InfoPath Eco System.

Gas Mileage: 55 mph vs 70 mph - What a huge difference!

A week ago my wife and I were driving a rented Toyota Corolla from Pittsburgh, PA to Cleveland, OH. Since my wife was tired and slept, I was able to run a series of experiments that she would never had let me get away with, if she had been awake.  (She hates it when I drive 55 mph)

The Corolla had a gas mileage gauge that provided both instantaneous and average gas mileage read outs. The instantaneous reading was useless, since the numbers kept oscillating widely. However, the average read out was more useful. It allowed me to reset the gauge, then I could drive for 10 minutes at a given speed, and then see what the gas mileage was. Since there are many variables, I attempted to keep as many things constant as possible for a given experiment.  I also repeated each experiment three times. Specifically, I reset the meter and drove for 10 minutes under the following conditions:

• 55 mph
• 70 mph
• 70 mph (drafting behind a truck)

Each time I got very similar results:

• 55 mph --> 53 mpg
• 70 mph --> 37 mpg
• 70 mph (drafting behind a truck) --> 43 mpg

Of course we have all been told this before, but running the experiment myself and seeing the huge difference made an impact on me.  Driving 55 mph, instead of 70 mph,  decreased fuel consumption by a whopping 30%!  On the trip from Pittsburgh to Cleveland (and back) 250 miles, I could use 4.7 gallons, instead of 6.7 gallons of fuel.

As a result, I started to think about whether I really needed to get places as quickly as I usually think I do.  There are plenty of choices we all make.  Lindsay Leeven in his latest blog post posits that a 50% reduction in vehicle weight reduces the fuel consumption by 30%.  I already drive reasonably small cars, so I am not ready to go out and drive a much lighter car, when safety is at the top of my list.

I began to think what it would take for drivers in the US to make a noticeable difference in the number of gallons of gasoline that are sold for use in automobiles and light trucks and vans.  Suppose 30% of the drivers on the road changed their driving habits for one week, is there a statistic that would show the results? 

I discovered that one can download a spreadsheet from the US government, which summarizes the daily average of fuel sold for use in automobiles.  The data is provided on a weekly basis, starting in 1990.  For kicks I graphed the data for every year, since 1990, bracketing the week of every year that includes Christmas:  For example, in 2010 the US statistics were published for the week ending Friday December 24 2010, Friday December 31 2010, and Friday January 6 2011.  What I noticed over the 20 year period is that the amount of fuel used during the week that includes Christmas has the following characteristics when it is compared to the weeks that precedes and follows Christmas:

1. Week of Christmas never differs by more than 11.11% from the other two weeks.
2. The average difference between the Christmas week and the other two weeks is 3.27%
3. The standard deviation between the Christmas week and the other two weeks is 2.82%

Figure 1 illustrates the data.  The red bars represent the weeks that include Christmas.

Figure 1: The red bars represent the weeks that include Christmas.

So, if 30% of US drivers decided to slow down for the week that includes Christmas during 2011 (December 25, 2011 --> December 31, 2011), and the only difference in their driving habits was that they slowed down to 55 mph on the highway, then I believe that a statically significant observation could be made that showed when people in the US set their minds to making a change, they can do so.  Wouldn't that be nice if people in the US decided to make a point rather than be legislated to slow down!  Everyone knows that legislation usually doesn't work.

 

Exceeding The Amount of Allowable Resources with Browser Based InfoPath Forms

Yesterday I received an emergency call from one of my clients.  One of their customers was filling out a form, when they were greeted by the following critical error (I love these Microsoft errors, **NOT**):

A little googling (yes, I know I am supposed to use Bing, but I do like google better) quickly told me that I needed to talk to the administrators and verify that the issue was most likely due to exceeding the number of allowable postbacks.   When the administrators checked the server logs, this is what they found:



The culprit?  I was using a repeating table in the form.  The repeating table had 5 columns each containing a drop-down and each, except the first was filtering a SharePoint list by using the previous column's selected value. (I refer to this as cascading drop-downs, see my post Browser Based InfoPath 2010 - Creating Cascading Dropdown Fields with no Code .)  More recently I have found that my clients like the term "smart drop-downs," better than cascading.  I refer to the tables that contains smart drop-downs as smart tables. 

So, what was the issue?  Users can add as many rows in the table as they like.  Each row, due to the smart drop-downs, causes postbacks to the server.  When the number of rows results in 76 or more postbacks, the InfoPath Forms Server complains with the ugly message seen above.  The result, the user loses all their work for the form filling session.

So, what can we InfoPath designers do?  Well we can ask the administrator to increase the limit of the number of permissable postbacks.  This will alleviate the situation, but not solve it.  This now allows the table to grow larger, but it will still cause an error when the table gets too large.  The best solution that I could find is that in the InfoPath form, I can add logic, and keep count of the number of rows.  Once this number exceeds a specific number, I can have red warning text appear that alerts the user and tells them not to add any more rows.  I have not found a way to prevent the user from continuing to add more rows to the repeating table.

Dear Microsoft, can you please make SharePoint code updates *NOT* break things?

Dear Microsoft,

Last week I went to the SharePoint conference in Anaheim.  There were two things that you stressed as part of the keynote.  One of them was Office 365, and the other was your focus on hardening SharePoint. Although your demo was focused on a 14 TB MS-SQL database with 7,000+ concurrent users, the implication was clear, you are hardening SharePoint as well as the database.

I must tell you, I am not running into issues with my SharePoint environment going down.  I am however having issues when the IT organization upgrades SharePoint.  I am a SharePoint analyst.  This means that I am often sandwiched between the IT organization and the business users.  Between two and three times a year I find that I am scrambling for several weeks.  Why?  Because the IT organization has updated to the latest Service Pack or Code Release of SharePoint.  Of course the party-line is that organizations should perform tests in a staging environment, before these releases are released to production.  My request to you is, could you please release code that has been tested a little bit better?  At the risk of being called out as a lunatic, how about allowing for a rollback to the previous working state?

I must tell you, trying to test SharePoint in a staging environment, is akin to using a screen door on a submarine.  It is just not feasible to test thoroughly.

How are you handling code releases in Office 365?  Will I wake up one morning to find out that things are broken?  If not, can you please share your secret sauce with the administrators of on premises implementations?

If you improve the reliability of the SharePoint upgrade process, my end users and my administrators will sing your praises.

Creating a Heartbeat SharePoint Site to Monitor Outbound Email Health

I have been consulting at a Fortune 500 company as a SharePoint analyst.  One of the things I do is quickly create electronic forms and the other parts of the InfoPath EcoSystem  (See my blog post.)  The solutions I create improve efficiencies by up to two orders of magnitude.  Problems arise when the underlying infrastructure fails. 

One of the most common infrastructure failures is that outbound emails from workflows fail to be delivered.  I am usually the first to notice and report the issue to the IT group.  The reasons for these failures vary widely, and are usually not due to a SharePoint issue.  My clients get very upset when their notifications and requests for approvals are not delivered.  Therefore, I need to stay on top of this. 

This blog summarizes how I constructed a simple heartbeat^* SharePoint site that provides me with a dashboard which alerts me to an issue.  Of course, ideally I would like to receive an email alert, but I cannot rely on the email system to deliver an alert, when the email system is not working.  (Escher's drawings are fine on paper, but in the real world I can't rely on things that aren't working)

Figure 1 shows the home page of my site.  It shows a summary explanation of the Exception list along with the Email Sender List and Email Receiver List.

  
 

Figure 1: A screen shot of the home page of my heartbeat email health site. 

The site leverages three distinct lists:

• An Email Sender List
• An Email Receiver List
• An Exception List.

The Email Sender list has workflow associated with it.  This workflow is started once and then loops forever.  The loop has a pause in it that last 30 minutes.  30 minutes is plenty of time for the emails to arrive at the Receiver list.  As part of each loop, the Email Sender sends an email to the Email Receiver list.  The subject of each email is unique.  After the 30 minute pause, the workflow looks for the email with the unique subject.  If it finds the email, it continues to loop.  If it does not find the item, it creates an entry in the Exception list.  It is my job to periodically look at the Exception list and ensure that it is empty.  The workflow is summarized by the following list of steps and architectural diagram.

Step 1: Administrator creates an entry in the Email Sender list.

Step 2: The workflow sends an email to the Receiver list.  The subject of the email is unique.
            The subject will become the title field in the Email Receiver list.

Step 3: The Email Receiver list receives the email (hopefully)

Step 4: The workflow waits 30 minutes, and checks to see if the email has arrived. If it has arrived, the workflow goes back to Step2.

Step 5: If the email cannot be found, the workflow adds an entry to the exception list.

Figure 2 shows a summary of the architecture of this system

Figure 2: Architectural Diagram of the Heartbeat System

In summary, I was able to create this SharePoint site in one day.  Although I used Nintex workflows to develop the workflows, it is possible to use basic SharePoint designer workflows.  It is a quick and easy way to develop a dashboard solution that monitors the health of the email delivery system.  The solution does not depend upon any functionality that is not included in SharePoint Foundation (2010) or WSS (2007.)

 

* I refer to this site as a heartbeat site, because it sends out an email on a regular basis, every 30 minutes.  This is analogous to a heart beating once every 60^th of a second.

The InfoPath Ecosystem

Introduction

 There is a huge demand for converting paper based forms to electronic forms.  I use the term "paper based" loosely.  Really, what I mean is any form that is filled out by hand and submitted through any one of the following vehicles:

• Hand delivery (sneaker net)
• Fax
• Email with the form attached as a scanned document. 

By-the-way, most Adobe pdf forms in use today fall into this category.  The reason is not that Adobe doesn't have an adequate electronic forms processing environment, because Adobe does.  Instead, it is because Adobe has not been as successful as Microsoft in engaging the community to use their suite.  This post is not about comparing tools, but rather highlighting that there is significantly more to a form, than the form itself.

In the case of InfoPath, most of my clients come to me with the expectation of converting a paper form to an electronic form.  Many people think that this is a trivial matter.  It would be, but there are a variety of justifiable and implicit expectations that users have of electronic forms that are sometimes subtle and are often overlooked by the uninitiated.  As I have become well versed with these issues, I have coined the phrase "InfoPath ecosystem" as a term to describe the bigger picture. (I should really be referring to it as the "electronic form ecosystem", but since I specialize in SharePoint and InfoPath, I use the product specific names.)  The constituent parts of the model are:

1. The Electronic Form (InfoPath)
2. Launch & Landing pages
3. Reporting
4. Automated workflows together with apropos web pages

Figure 1 illustrates the InfoPath Ecosystem. 

 
Figure 1: The InfoPath Ecosystem

 

1. The Electronic Form (InfoPath)

Some of the subtleties that people don't think of when they are converting "paper" forms to InfoPath forms are:

• Save vs Submit
  Almost every project I have worked on, starts off with the clients telling me they want a very simple form.  When I ask them do they want save functionality, they tell me no.  I explain that once a form is submitted, the data should not be changed.  After people start working with the form for a while, almost everyone asks me to add save functionality.  By save, I mean the user is able to save the form for later modifications, and the recipients of the from are not notified yet.  Furthermore, if there are any automated workflows that are supposed to run, they do not run, until the form is actually submitted.  Once the form is submitted, the form imitator can go back and look at the form, but she can no longer modify the form.
• Intelligent pre-filling
  People have little patience for filling out electronic forms.  Even less so than paper forms.  They get frustrated quickly. Therefore the forms need to pre-fill as much information as possible.  For example, in a system where the user is logged in, a great deal of personal information can be pre-filled.  InfoPath allows one to add sophisticated logic, which allows many of the form fields to be pre-filled with the most likely answers and furthermore make excellent guesses on things like dates and times.
• Intelligent hiding or disabling of controls
  For example, the Submit button should be grayed out, until all required fields are filled in. 
• Showing and hiding different portions of the form as a function of the user and the state of the form
  For example, a process owner may have special privileges that an ordinary user does not have.  Another example is to provide a preview page which shows a summary of all answers before the user submits the form.  The preview page does not allow editing, it just shows the user the answers they have given to the form questions. This is similar to the Print Preview feature in many applications.
• Important Controls to make the user experience easy
  For example, if the user needs to pick an answer from a very large number of choices such as all universities in the U.S.  An ideal control for this is a text box with auto-complete.  Marc Anderson and I worked on such a solution recently and we wrote companion blog posts on the subject. (See: Marcel's Post and Marc's Post)

All of the above mentioned, with the exception of the last item, can be implemented by a seasoned InfoPath power user.   The last item requires a "middle-tier" developer who knows how to write Javascript, calls to Web Services and jQuery (or similar.)

2. Launch vs Landing Pages
You don't want to expose the uninitiated user to a native SharePoint Form Library or List.  An example of a native form library is shown in Figure 2. It is too confusing.   I have found that it is a good idea to provide a Launch page that provides instructions along with a launch button.  Similarly you want to have a page where users can see what forms they own.  By "owning", I mean forms that the user has saved, has submitted, is interested in tracking, or needs to take some action on. For simpler forms, one can use a single page for the purpose of launching and landing.  For complex forms, one might want to create a distinct landing page for each stage of form's life-cycle.  Figure 3 illustrates a combined launch & landing page that my team has created for new badge requests.  In order to build these pages, the required technology is out-of-the-box SharePoint with a little bit of custom html that is saved in a Content Editor Web Part.

 

Figure 2: Is a screenshot of a Form Library that contains applications for carpools.  To the uninitiated user, it is unfriendly and confusing.  This is the reason we need to build a launch and landing page depicted in figure 3.

 
 

Figure 3: A screen shot of a Badge Request launch & landing page.  This page shows the launch buttons, as well as a two nicely formatted
SharePoint lists that show all of my requests.  In this case, I have no pending requests..
 

3. Reporting 

The process owner often wants to go back and see summary statistics of the aggregate data.  In its simplest incantation, this may be a simple list, but it often comes with dashboard requirements as well as nice charts and graphs.  One can really go far here.  The simplest thing to do is use custom SharePoint Web Part Pages with DataView Web Parts, Chart Web Parts, and Excel.  There are numerous third party add-ons that can allow for very rich visualization and data analysis.  Some examples are: Spotfire, Tableau (Visualization tools) and Knime (An open source tool that helps with data analysis.)

4. Automated workflows with apropos web pages  

I have already alluded to the landing pages that one might use in a human intensive workflow.  The workflows, whether reviewer, approval, sophisticated business processes, or just notification workflows can be messy to design and implement.  The workflows should be carefully designed and documented.  Generally I like to write down a traditional state diagram.  These state diagrams resemble a directed graph from mathematics (See figure 4.)  When it comes to implement the workflows, there are many options.   SharePoint Designer Workflows is a free technology and can get you pretty far.  I often use third party tools to help write the workflows, but they can be quite pricey.  Both Nintex & K2 provide excellent workflow add-ons.

 

Figure 4: Directed graph that I use as my company logo (Mathft)

 

Conclusion

People often trivialize the design of electronic forms (in my case InfoPath forms.)  Besides subtleties in the forms themselves, people often don't realize that there is an important ecosystem that is needed to make the forms easy to use, and the aggregate results easy to  analyze.  

InfoPath and Autocomplete and Other Lessons

I have been working on developing a simple approval form for the top level executives at a fortune 500 company.  The form had to look really nice, and since I am quite facile using InfoPath 2010 together with SharePoint 2010, I decided to take that route.

This choice allowed me to develop a prototype, which evolved into the production form rather quickly.  Everything was going along perfectly, until I hit a bump in the road.  For one of the form fields, the project sponsors needed users to pick from a list of some 2000+ entries.  The drop-down list that I had been using, was not going to be adequate, since choosing from a list of 2000+ choices is a terrible user experience.  To solve this issue, several of my colleagues suggested that I use the new SharePoint 2010 Managed  Metadata feature, which provides an autocomplete feature.  Unfortunately it turns out that Microsoft does not support Managed Metadata inside InfoPath 2010.  (Just do a Google search for "managed metadata infopath" and you will find plenty of hate mail!)

The inability to have autocomplete for this key field was going to be a show stopper.  Well I called up my friend and colleague Marc Anderson, MVP and author of SP Services.  Marc suggested that we get together for a day.  We did, and he helped me solve the problem.

Since SharePoint 2010 embeds the form web part into web part pages, we were able to populate a Content Editor Web Part with the appropriate Javascript and leveraged jQuery, jQueryUI, and SPServices.  Marc's blog summarizes how it works.

The picture below illustrates the autocomplete functionality that we wound up using.  In the picture, I used a list with Cities for test data.  In the picture, you can see that I typed new, and all the choices that include the the letter sequence new are provided.

This was a huge win for me.  In the past, this would have been a show stopper and I would have had to start all over again using a different choice of technologies.  The bigger lesson that I took from this is that many of the user interface limitations that are inherent in InfoPath can now be overcome becuase one can easily mix and match InfoPath with Javascript and all the web based standards that can be leveraged.

I have recently dropped off the face of the earth on Facebook. I thought I would share why:

Recently my daughter sent me the following blurb:

"ALL THE PHONE NUMBERS IN YOUR PHONE are now on Facebook!!! go to the top right of the screen, click on ACCOUNT, click on EDIT FRIENDS, left side of screen and click CONTACTS. You will see all phone numbers (FB friends or not) are published that you have stored in your mobile phone. TO REMOVE, go to Right column, click on "this page." Please repost this on your status, so your friends can remove their numbers and thus prevent abuse if they do not want them published"

So I did some research. I found out that this has some truth to it: Here are the facts:

1. If you use Facebook from your smartphone and if you were not aware of this choice, then it is turned on by defualt.
2. Only you can see the phone numbers.
3. There is supposedly a way to remove the phone numbers and make sure that your phone does not add them again.

I do have a Droid, so this may be different for you iPhone users. On the Droid, I could see that the option to post phone numbers was indeed selected, but I could not find any way to turn that option off. My last resort was to remove the application from my phone. Next the instructions on Facebook indicated that I needed to remove my mobile friends. Once I did that, I found I had no friends left. I wonder what everyone is thinking! "Boy he snubbed me"

So.... Why did I want to remove phone numbers if no one can see them, except for myself? Well, I know that my Facebook account was hacked once, and I had to change my password. I really don't want these phone numbers getting into the wrong hands.

Just received an invitation to the new Microsoft online (Office 365 - Beta.)

My first impression is very positive! Leagues better than the old BPOS!

1) Performance is excellent.
2) MS seems to have removed the requirement to install the authentication piece software on every client machine!
3) Still a stripped down version of SharePoint, seems like SharePoint Foundation. (I will investigate more)
4) Administration is much easier than the BPOS administrator interface