Home

Recent Blogs

Categories

Downloads

Links

Monthly Archives





IM me on the bleedyellow.com sametime community - jeremy.hodge@zetaone.com

Contact Me

Email Me - jeremy.hodge@zetaone.com

« ZetaOne is Looking to hire | Main| »

Ever wondered about the scalability / availability of XPages on Domino ? Well...

Category
0
I can't say this is the most scientific test, but it's real world.

We've got an XPages app that serves up a set of pages to entrants of a well known contest ... The entrants enter their registration information, and we record the data, and direct them to a thank you page.

The server that runs this is a Linux domino server, with 2 processors, 4 cores each, and ample memory (but its only running on a single partition, and since we're talking a 32-bit domino installation, memory is maxed out at 4GB). Server uses a simple single RAID 10 volume with 8 15k SAS 146GB Drives. I've dedicated 512MB of that ram to the JVM for the HTTP Stack in the Notes.ini, but, from the second image below, you can see it only has 51MB allocated to the Java Heap and 66MB allocated to the non-heap memory. We also have the server in a cluster that replicates the data in real time (cluster replication) to another server that is not serving general public requests. Its only for backup. The server does not serve any other NRPC traffic, its strictly for web traffic. For what its worth, all web traffic is anonymous.

A few days ago (February 14) I enabled the domino log to track some statistics to see what the overall load on the server was because I knew we were seeing some "significant" traffic.

Below you can see the server has serviced just under 9.3 million requests for resources from the domino server since midday the 14th. The majority of those (just over 9.2 million) are directly related to the XPages App I describe above.  The third entry shows the number of new entries we record each day, on average about 30,000 - 60,000 new entries every day. From the chart #3 below, the bandwidth graph you can see that the majority of our entries come in from 6 AM throughout the day, then taper of at about 1 AM ...

From the 4th image below, you can see we serve on average about 45,000 XPage Entry requests per day (give or take - its been as high as 60,000 in one day), then a couple of partial refreshes as they complete their entry form, then the submission to save the entry, and a redirect to a final XPage confirming the submission. so 90,000+ full XPage requests in a 19 hour period = roughly 4,730 XPages served per hour or about 79 per minute. or 1.3 full XPage requests every second, plus partial refreshes and cluster replication.  This "jives" with a gut level check i perform when I open a view with the totals and hit F9 every second and watch the record count go up by about 1 every second.

The final screen shot is a sh server from the server's console. Its hardly phased ... Availability Index sits at 100. While the application isn't computationally complex ... Domino and XPages are taking the beating only saying, "Thank you sir may I have another".  What do you think? Are XPages ready for production use?

Click the Read More link to view the images ...

DomLog.nsf statistics
A picture named M2


Java Heap/Non-Heap Memory Allocation
A picture named M3

Timed Bandwidth Graph
A picture named M4

Number of new entries recorded per day
A picture named M5

sh server
A picture named M6
show stats platform.* - ran this morning (2/18/2011) while server was "under load" adding approx 1-2 entries every second...
  Platform.LogicalDisk.1.AssignedName = sda
  Platform.LogicalDisk.1.AvgQueLen = 0.07
  Platform.LogicalDisk.1.AvgQueLen.Avg = 1.58
  Platform.LogicalDisk.1.AvgQueLen.Peak = 23.69
  Platform.LogicalDisk.1.PctUtil = 1.67
  Platform.LogicalDisk.1.PctUtil.Avg = 3.63
  Platform.LogicalDisk.1.PctUtil.Peak = 107.41
  Platform.LogicalDisk.1.ServiceTimeinmsecs = 0.86
  Platform.LogicalDisk.1.ServiceTimeinmsecs.Avg = 0.68
  Platform.LogicalDisk.1.ServiceTimeinmsecs.Peak = 4
  Platform.LogicalDisk.TotalNumofDisks = 1
  Platform.Memory.PagesPerSec = 0
  Platform.Memory.RAM.AvailMBytes = 61
  Platform.Memory.RAM.AvailMBytes.Avg = 303.37
  Platform.Memory.RAM.AvailMBytes.Min = 53.08
  Platform.Memory.RAM.AvailMBytes.Peak = 2,998.72
  Platform.Memory.RAM.PctUtil = 99.62
  Platform.Memory.RAM.TotalMBytes = 16,067.96
  Platform.Network.1.AdapterName = eth0
  Platform.Network.1.BytesRecvdPerSec = 81,047.2
  Platform.Network.1.BytesSentPerSec = 258,962.82
  Platform.Network.1.NetworkBytesPerSec = 340,010.02
  Platform.Network.1.PctCollisionRate = 0
  Platform.Network.2.AdapterName = eth1
  Platform.Network.2.BytesRecvdPerSec = 6.4
  Platform.Network.2.BytesSentPerSec = 0
  Platform.Network.2.NetworkBytesPerSec = 6.4
  Platform.Network.2.PctCollisionRate = 0
  Platform.Network.3.AdapterName = eth2
  Platform.Network.3.BytesRecvdPerSec = 6.4
  Platform.Network.3.BytesSentPerSec = 0
  Platform.Network.3.NetworkBytesPerSec = 6.4
  Platform.Network.3.PctCollisionRate = 0
  Platform.Network.4.AdapterName = eth3
  Platform.Network.4.BytesRecvdPerSec = 435.9
  Platform.Network.4.BytesSentPerSec = 425.17
  Platform.Network.4.NetworkBytesPerSec = 861.07
  Platform.Network.4.PctCollisionRate = 0
  Platform.Network.TotalNumofAdapters = 4
  Platform.Network.Total.BytesRecvdPerSec = 81,495.9
  Platform.Network.Total.BytesSentPerSec = 259,387.98
  Platform.Network.Total.NetworkBytesPerSec = 340,883.88
  Platform.Network.Total.PctCollisionRate = 0
  Platform.PagingFile.Free.SizeMBytes = 1,415.23
  Platform.PagingFile.Total.PctUtil = 31.14
  Platform.PagingFile.Total.PctUtil.Avg = 19.28
  Platform.PagingFile.Total.PctUtil.Peak = 31.14
  Platform.PagingFile.Total.SizeMBytes = 2,055.15
  Platform.Process.ActiveDomino.TotalCpuUtil = 0
  Platform.Process.ActiveNumOfDominoProc = 15
  Platform.Process.adminp.1.DominoLocalFreeKBytes = 249
  Platform.Process.adminp.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.adminp.1.MemFreeKBytes = 3,507,251
  Platform.Process.adminp.1.PctCpuUtil = 0
  Platform.Process.adminp.1.ProcessID = 5264
  Platform.Process.amgr.1.DominoLocalFreeKBytes = 1,233
  Platform.Process.amgr.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.amgr.1.MemFreeKBytes = 3,503,703
  Platform.Process.amgr.1.PctCpuUtil = 0
  Platform.Process.amgr.1.ProcessID = 5838
  Platform.Process.amgr.2.DominoLocalFreeKBytes = 291
  Platform.Process.amgr.2.DominoSharedFreeKBytes = 30,886
  Platform.Process.amgr.2.MemFreeKBytes = 3,506,443
  Platform.Process.amgr.2.PctCpuUtil = 0
  Platform.Process.amgr.2.ProcessID = 5262
  Platform.Process.amgr.3.DominoLocalFreeKBytes = 291
  Platform.Process.amgr.3.DominoSharedFreeKBytes = 31,012
  Platform.Process.amgr.3.MemFreeKBytes = 3,506,443
  Platform.Process.calconn.1.DominoLocalFreeKBytes = 297
  Platform.Process.calconn.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.calconn.1.MemFreeKBytes = 3,508,743
  Platform.Process.calconn.1.PctCpuUtil = 0
  Platform.Process.calconn.1.ProcessID = 5270
  Platform.Process.cldbdir.1.DominoLocalFreeKBytes = 201
  Platform.Process.cldbdir.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.cldbdir.1.MemFreeKBytes = 3,507,631
  Platform.Process.cldbdir.1.PctCpuUtil = 0
  Platform.Process.cldbdir.1.ProcessID = 6018
  Platform.Process.clrepl.1.DominoLocalFreeKBytes = 697
  Platform.Process.clrepl.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.clrepl.1.MemFreeKBytes = 3,497,279
  Platform.Process.clrepl.1.PctCpuUtil = 0
  Platform.Process.clrepl.1.ProcessID = 17588
  Platform.Process.event.1.DominoLocalFreeKBytes = 691
  Platform.Process.event.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.event.1.MemFreeKBytes = 3,501,307
  Platform.Process.event.1.PctCpuUtil = 0
  Platform.Process.event.1.ProcessID = 4732
  Platform.Process.http.1.DominoLocalFreeKBytes = 227,662
  Platform.Process.http.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.http.1.MemFreeKBytes = 2,418,659
  Platform.Process.http.1.PctCpuUtil = 0
  Platform.Process.http.1.ProcessID = 5274
  Platform.Process.replica.1.DominoLocalFreeKBytes = 2,312
  Platform.Process.replica.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.replica.1.MemFreeKBytes = 3,506,355
  Platform.Process.replica.1.PctCpuUtil = 0
  Platform.Process.replica.1.ProcessID = 5256
  Platform.Process.rnrmgr.1.DominoLocalFreeKBytes = 458
  Platform.Process.rnrmgr.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.rnrmgr.1.MemFreeKBytes = 3,508,063
  Platform.Process.rnrmgr.1.PctCpuUtil = 0
  Platform.Process.rnrmgr.1.ProcessID = 5272
  Platform.Process.router.1.DominoLocalFreeKBytes = 1,234
  Platform.Process.router.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.router.1.MemFreeKBytes = 3,494,231
  Platform.Process.router.1.PctCpuUtil = 0
  Platform.Process.router.1.ProcessID = 5257
  Platform.Process.sched.1.DominoLocalFreeKBytes = 777
  Platform.Process.sched.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.sched.1.MemFreeKBytes = 3,508,147
  Platform.Process.sched.1.PctCpuUtil = 0
  Platform.Process.sched.1.ProcessID = 5268
  Platform.Process.server.1.DominoLocalFreeKBytes = 4,067
  Platform.Process.server.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.server.1.MemFreeKBytes = 3,398,275
  Platform.Process.server.1.PctCpuUtil = 0
  Platform.Process.server.1.ProcessID = 4710
  Platform.Process.stats.1.DominoLocalFreeKBytes = 265
  Platform.Process.stats.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.stats.1.MemFreeKBytes = 3,508,727
  Platform.Process.stats.1.PctCpuUtil = 0
  Platform.Process.stats.1.ProcessID = 15786
  Platform.Process.update.1.DominoLocalFreeKBytes = 1,280
  Platform.Process.update.1.DominoSharedFreeKBytes = 30,886
  Platform.Process.update.1.MemFreeKBytes = 3,504,799
  Platform.Process.update.1.PctCpuUtil = 0
  Platform.Process.update.1.ProcessID = 5258
  Platform.System.ContextSwitchesPerSec = 11,136.47
  Platform.System.ContextSwitchesPerSec.Avg = 15,903.06
  Platform.System.ContextSwitchesPerSec.Min = 2,424
  Platform.System.ContextSwitchesPerSec.Peak = 70,794.23
  Platform.System.PctCombinedCpuUtil = 3.8
  Platform.System.PctCombinedCpuUtil.Avg = 3.62
  Platform.System.PctCombinedCpuUtil.Peak = 12.24
  Platform.System.PctTotalPrivilegedCpuUtil = 0.35
  Platform.System.PctTotalPrivilegedCpuUtil.Avg = 0.46
  Platform.System.PctTotalPrivilegedCpuUtil.Peak = 3.18
  Platform.System.PctTotalUserCpuUtil = 3.45
  Platform.System.PctTotalUserCpuUtil.Avg = 3.16
  Platform.System.PctTotalUserCpuUtil.Peak = 11.36
  Platform.Time.LastSample = 02/18/2011 09:15:50 EST
  Platform.Time.SampleRateInMins = 1

Posted by Jeremy G Hodge On 02/17/2011 | | del.icio.us  Technorati  Digg This 

Comments

Gravatar Image10 - If at first you don't succeed, find out if the loser gets anything.

Posted by Haarfarben 2012 At 01:50:15 PM On 10/26/2011 | - Website - |


Gravatar Image9 - Coming home from very lonely places, all of us go a little mad: whether from great personal success, or just an all-night drive, we are the sole survivors of a world no one else has ever seen.

Posted by Männerfrisuren 2012 trend At 02:40:11 AM On 10/25/2011 | - Website - |


Gravatar Image8 - Skill without imagination is craftsmanship and gives us many useful objects such as wickerwork picnic baskets. Imagination without skill gives us modern art.

Posted by Xylitol side effects 2012 At 07:24:02 AM On 10/13/2011 | - Website - |


Gravatar Image7 - No one who cannot rejoice in the discovery of his own mistakes deserves to be called a scholar.

Posted by f150 towing capacity chart At 03:03:14 AM On 10/07/2011 | - Website - |


Gravatar Image6 - The solution is to gradually become free of societal rewards and learn how to substitute for them rewards that are under one's own powers. This is not to say that we should abandon every goal endorsed by society; rather, it means that, in addition to or instead of the goals others use to bribe us with, we develop a set of our own.

Posted by nail polish designs for short nails easy At 06:53:46 AM On 10/01/2011 | - Website - |


Gravatar Image5 - I did not use any managed beans for this app...

Posted by Jeremy Hodge At 04:29:57 PM On 02/21/2011 | - Website - |


Gravatar Image4 - Thanks for sharing this level of detail.

Was this app straight XPages or were you using any managed beans?

Posted by Russell Maher At 02:38:39 PM On 02/19/2011 | - Website - |


Gravatar Image3 - UPDATE: I was able to start the stats engine, and have included the stats above. @Erik - I'm currently not running statistics on that server, and the big promo winds down today ... I have another that will spool up in a month or so, and will turn on stats for that one.

Posted by Jeremy Hodge At 08:59:55 AM On 02/18/2011 | - Website - |


Gravatar Image2 - Cool. Can you run a "show stat server.platform.*" and post that also? The will show hardware usage averages. The Availability Index is a kinda ho-hum statistic.

Posted by Erik Brooks At 08:19:38 PM On 02/17/2011 | - Website - |


Gravatar Image1 - Yowza...... very cool sir! Thank you for sharing. Was there ever ANY doubt that XPages WERE NOT ready for prime time? Um..... no.

Posted by Bruce Elgort At 07:11:44 PM On 02/17/2011 | - Website - |


Post A Comment