John McCann August 6 2011 07:21:34 AM
I was upgrading a major application to a new release. When we deployed on the QA system and started running against the full data load, the overnight processing agent that goes in and touches a good percentage of the 100K records in the application started crashing the server. We got the dreaded "LSXBE: ****** Out of Backend Memory *******"" error. We reported the problem to IBM who asked for all the logs, copies of the database, etc. After having a good laugh that they would not be getting a copy of databases, they eventually pointed us to a posting that effectively said, "The cause of the problem is not recycling Domino Java API objects correctly." Only problem is that the agent was written in LotusScript, not Java.
We had a number of false starts and red herrings (session.getDatabase will not return a database object if consistency check going on - duh). The failing call stack, as shown below, had us focusing on DocumentCollection processing. We were expecting a failure on getting or traversing a collection. We never tracked it down exactly, but are pretty sure the actual failure was on a NotesDocumentCollection.getNextDocument statement. The failing stack was:
### FATAL THREAD 1/3 [ nAMgr: 0f34: 0324]
Exception code: c0000005 (ACCESS_VIOLATION)
@[ 1] 0x60001706 nnotes.OSLockWriteSem@4+22 (19a)
@[ 2] 0x61cf4a78 nlsxbe.ANNote::ANNAddToCollList+56 (0)
@[ 3] 0x61d26127 nlsxbe.ANDocColl::ANDCNavigate+535 (0)
In one of the iterations of trying to isolate exactly where the problem was occurring, IBM support (thank you Charles) pointed out that we just may be running into resource limitations on the machine since it looked like we were running into memory problems. It seems we had all these objects sitting around in memory. We started looking at the code and said, no, we clean up. See, for each record in the primary database, we create an object, process it, then set the object to Nothing at the end of handling the record. The object we create has other objects it creates, but LotusScript should be cleaning up all those when the primary object gets destroyed, right? Seemed logical, only in Java do you have to do your own recycling. Hmm, I mused. Are there any known problems with NotesDateTime variables and memory leaks? The big change in this upgrade was to add using NotesDateTime variables instead of LotusScript date/time variants since we started worrying about time zones and localizing date and times shown. Charles looked it up and indicated nothing known there.
So, just for chagrins, when adding the latest tracing to produce a bunch more messages, I went through and added destructors to all my classes and changed the set to nothing to delete statements. (I also found out that unless a class has a Delete method, don't invoke delete on the class). Lo and behold, the problem goes away.
I have neither the tools nor the time to examine internal memory usage at points in time. I will leave that to the Lotus developers. However, what I think I have discovered empirically is one of two situations. And, I don't know which one it is, so I left the code in for both thinking (perhaps wrongly) that it does no harm.
1) LotusScript will not clean up a list of objects contained within another object
2) A NotesDateTime contained within an object is not properly cleaned up when the object is destroyed.
Simplified code before (real routine is over 8K lines of LotusScript)
Option Explicit
Public Class cFoo
Public ndtEvent as New NotesDateTime("")
End Class
Public Class cBar
Public lstFoo List as cFoo
Sub New
Dim i as Long
Dim oFoo as cFoo
For i = 1 to 10
Set oFoo = New cFoo
Set lstFoo(i) = oFoo
Next I
End Sub
End Class
Sub Initialize
Dim session as new NotesSession
Dim dbCur as NotesDatabase
Dim vwCur as NotesView
Dim docCur as NotesDocument
Dim oBar as cBar
Set dbCur = session.CurrentDatabase()
Set vwCur = dbCur.GetView("SomeView")
Set docCur = vwCur.GetFirstDocument()
While not docCur is Nothing
Set oBar = new cBar
' **** do something with the document
Set oBar = Nothing
Set docCur = vwCur.GetNextDocument(docCur)
Loop
End Sub
I have not compiled the above routine nor tested it, so it may have typos. But, it should provide the basics of what was going on. I assumed (bad choice by me) that the set of oBar to Nothing would clean up the memory of the cBar object as well as the 10 cFoo objects it created. When I changed the code to have my own destructor routines and invoke Delete as appropriate, the memory problems went away. As indicated above, I wasn't able to determine the root cause of the memory problem, so I shotgunned and cleaned up things that had changed (Datetime) and things I knew about.
Option Explicit
Public Class cFoo
Public ndtEvent as New NotesDateTime("")
Sub Delete
Set ndtEvent = Nothing
End Sub
End Class
Public Class cBar
Public lstFoo List as cFoo
Sub New
Dim i as Long
Dim oFoo as cFoo
For i = 1 to 10
Set oFoo = New cFoo
Set lstFoo(i) = oFoo
Next I
End Sub
Sub Delete
Forall Foo in lstFoo
Delete Foo
End Forall
End Sub
End Class
Sub Initialize
Dim session as new NotesSession
Dim dbCur as NotesDatabase
Dim vwCur as NotesView
Dim docCur as NotesDocument
Dim oBar as cBar
Set dbCur = session.CurrentDatabase()
Set vwCur = dbCur.GetView("SomeView")
Set docCur = vwCur.GetFirstDocument()
While not docCur is Nothing
Set oBar = new cBar
' **** do something with the document
Delete oBar
Set docCur = vwCur.GetNextDocument(docCur)
Loop
End Sub
Let me know if this solves a memory problem for you and I will pass the feedback to Lotus support that there really is a problem in this area.
2011-08-18 Update: It has been suggested that delete method for cBar Erase the lstFoo element instead. Not having the time to test all the options, I've updated my production code to do both - issue the delete then Erase lstFoo.
