Performance issues resolved.
Bill Lee issued his final summary:
This week the team has identified the individual cache (IEntityGroup) that is responsible for the large performance improvement we saw last week. We believe that differences in the number of organisational hierarchy groups in Test (930) and Live (1020) explain why it wasn't possible to originally reproduce the issue on Test. The IEntityGroup cache has been increased to a maximum of 10,000 items and other cache sizes increased to reasonable values. Subsequent loading testing suggests these are suitable for the Live dataset.
For information, we have run a load of 3 logins/second across 3 app server (servers at KB), during which 90% of logins were within 3.2 seconds. Increasing the load to 10 logins/second, 90% of logins were within 4 seconds. It should be noted that 10 logins/second is significantly greater than the expected load. Running the same load test with one app server, instead of 3 results in similar results, so our conclusion is that throughput is being limited by database capacity rather than application server.
We have also identified a significant amount of database write activity related to portal stats logging. Each login results in about 50 rows inserted across a number of stats tables. After our load testing one table stats_events has ~11 million rows and is consuming about 1GB of tablespace. This is generated from about 600,000 logins (roughly a months' worth of Live Logins). Looking at the Uportal documentation, stats recording is off by default (mmm, no guesses why) so we may have to look to see if there is anything we can "turn down" or if we have any plans to prune these stats tables periodically, as any more than a month or so of data is likely to cause pain. This will be followed up with Service Management and Chrisann next week when she returns from leave.
The only other task now remaining against this item is to run a load test with real users. This has been scheduled for 24th September. As I am on leave from next Thursday this will be the last special update on MyEd from me, I don’t think we need any more in any case. Reporting responsibilities can go back to Morna as project manager as usual.