Duplicate EMail Elimination

John McCann  May 17 2012 10:56:57 AM

Argh, ran into a problem with my inbox filled with duplicates while I was having problems with an IMAP source.   I wrote a duplicate email eliminator that I thought others might be able to use to save themselves some time.

' Agent Duplicate Deleter
' Purpose:  Delete duplicates emails from selected list

' Change History:
' May 17, 2012 - John McCann
' - Initial Creation

Option Public
Option Declare


' Class Msg
' Description: Information to compare and find the email
Class Msg
        Public strUNID As String
        Public strMsgID As String
        Public strOther        As String
        Public strSubject As String
       

End Class
Sub Initialize
       
        Dim session                 As New NotesSession
        Dim dbThis                        As NotesDatabase
        Dim dcThis                        As NotesDocumentCollection
        Dim docThis                        As NotesDocument
        Dim itmMessageID        As NotesItem
        Dim itmOther                As NotesItem
        Dim strUNID                        As String
       
        Dim fRemoved                As Boolean
        Dim lstMsgs                        List As Msg
        Dim lstIDs                        List As String
        Dim vntIDs                        As Variant
        Dim msgThis                        As Msg
        Dim msgBase                        As Msg
        Dim i                                As Long
       
        On Error GoTo This_Error
       
        Set dbThis = session.Currentdatabase
        Set dcThis = dbThis.Unprocesseddocuments
        Set docThis = dcThis.Getfirstdocument()
        While Not docThis Is Nothing
                strUNID = docThis.UniversalID
               
                ' going to match on one of the message IDs
                Set itmMessageID = docThis.GetFirstItem("$MessageID")
                If itmMessageID Is Nothing Then
                        Set itmMessageID = docThis.GetFirstItem("$IMAPUID")
                End If
               
                ' Need at least another field for uniqueness
                Set itmOther = docThis.GetFirstItem("$INetOrig")
                If itmOther Is Nothing Then
                        Set itmOther = docThis.Getfirstitem("$Orig")
                        If itmOther Is Nothing Then
                                Set itmOther = docThis.Getfirstitem("$Abstract")
                                If itmOther Is Nothing Then
                                        Set itmOther = docThis.GetFirstitem("DomainKey_Signature")
                                End If
                        End If
                End If
                ' create the message for our list
                Set msgThis = New Msg
                With msgThis
                        .strMsgID = itmMessageID.Text
                        .strSubject = docThis.Subject(0)
                        .strOther = itmOther.Text
                        .strUNID = strUNID
                End With
               
                ' save the message
                Set lstMsgs(strUNID) = msgThis
               
                ' create a list by IDs for dup elimination
                If IsElement(lstIDs(msgThis.strMsgID)) THen
                        lstIDS(msgThis.strMsgID) = lstIDS(msgThis.strMsgID) & ";" & docThis.UniversalID
                Else
                        lstIDS(msgThis.strMsgID) = docThis.UniversalID
                End if
                Set docThis = dcThis.Getnextdocument(docThis)
        Wend
       
        ' now, figure out which ones to remove
        ForAll msgID In lstIDs
                vntIDs = Split(msgID,";")
                ' only if more than 1
                If UBound(vntIDs) > 0 Then
                        Set msgBase = lstMsgs(vntIDs(0))
                        ' compare each to the first
                         For i = 1 To UBound(vntIDs)
                                 strUNID = vntIDs(i)
                                 If strUNID <> "" Then
                                         Set msgThis = lstMsgs(strUNID)
                                         ' if all three items match, then remove
                                         If msgThis.strSubject = msgBase.strSubject Then
                                                 If msgThis.strOther = msgBase.strOther Then
                                                         If msgThis.strMsgID = msgBase.strMsgID Then
                                                                 Set docThis = dbThis.Getdocumentbyunid(strUNID)
                                                                 Call docThis.Remove(True)
                                                                 Erase lstMsgs(strUNID)
                                                         End If
                                                 End If
                                         End If
                                 End If
                         Next
                End If
        End ForAll
               

       
This_Exit:
        Exit Sub
This_Error:
        MsgBox "Error " & Error & ", Subject=" & docThis.Subject(0) & ", Time=" & CStr(docThis.Created)
        Resume this_Exit
       
End Sub

• Comments [0]

Fun with @DbLookup

John McCann  April 27 2012 09:00:11 PM

I continue to be underwhelmed by IBM's level of documentation and examples.  It reminds me so much of teenagers doing just enough to get by and hoping they won't get caught.  

After spending considerable wasted time on XPages Server Side JavaScript @DbLookup results, I thought I would share my findings.  My results are specific to 8.5.3, though I suspect they are applicable to all 8.5.x versions.

The official IBM documentation indicates that @DbLookup (JavaScript) "Returns view column or field values that correspond to matched keys in a sorted view column".  The "Return value" is "any" and the description is "An array containing the column or item values".  Most of us now know this is incorrect.  If one or zero records matched the key, the value returned is a string, not an array.  

What many don't know are the circumstances where the result is "undefined".  I found two.  If you have specified the database incorrectly, e.g. made a typo, the result will be "undefined".  The second situation is where you don't have access to the database.  Mine was  a keyword lookup type of database that had an ACL with maximum Internet name and password set to "No Access".   The parameter [FAILSILENT] seems to have no affect on this.  

In the days attempting to debug my code, I also confirmed the multiple undocumented database specification techniques available, not just the server name array.  All the following worked (once I increased maximum Internet access):

// the defined way, an array of two elements
var db = @DbName();

// arrays pointing to another database on the same server
var db = [@DbName()[0], "folder\\filename.nsf"];   // note double slashes
var db = new Array(session.getServerName(), "folder/filename.nsf");     // slash the other way not doubled

// C API at lowest level still uses old Notes 2 conventions of double bang
var db = @DbName()[0], + "!!" + "folder\\filename.nsf";

// And, you can specify replicaId either as a string or single element array
var db = ["85256FF7:12345678"];
var db = "85256FF7:12345678";

// all work in this lookup
@DbLookup(db, viewname, "key", column, "[FAILSILENT");


With all this additional information about DbLookup that I uncovered the hard way, I thought it only appropriate to share.

I also decided to update Tom Steenbergen's excellent wrapper routine for DbLookup with this information.  While doing so, I identified that I really needed control over where I wanted the caching to occur.  This lead to the discovery that @Unique affected the cache, and other issues.  Therefore, I came up with the following derivation:

/* *****************************************************************
* Returns @DbLookup results as array and allows for cache  
* Author: John McCann - derived from work by Tom Steenbergen
* @param server -name of the server the database is on (only used if dbname not empty, if omitted, the server of the current database is used)  
* @param dbname -name of the database (if omitted the current database is used)  
* @param cache -empty for nocache, otherwise scope at which to cache  (application, request, view, session)
* @param unique -"unique" for returning only unique values, empty or anything for all results  
* @param sortit -"sort" for returning the values sorted alphabetically  
* @param viewname -name of the view  
* @param keyname -key value to use in lookup  
* @param field -field name in the document or column number to retrieve
* @param keywords - one or more comma separate strings containing [FAILSILENT], [PARTIALMATCH], or [RETURNDOCUMENTUNIQUEID]  
* @return array with requested results  
 ****************************************************************** */
 
function DbLookupArray(server, dbname, cache, unique, sortit, viewname, keyname, field, keywords) {  
       var result;
       try {
           var cachekey = "dblookup_"+dbname+"_"+@ReplaceSubstring(viewname," ","_")+"_"+@ReplaceSubstring(keyname," ","_")+"-"+@ReplaceSubstring(field," ","_");

           // if cache is specified, try to retrieve the cache from the appropriate scope
           switch (cache.toLowerCase()) {
           case "application":
                result = applicationScope.get(cachekey);  
                break;
           case "request":
                result = requestScope.get(cachekey);  
                break;
           case "view":
                result = viewScope.get(cachekey);  
                break;
           case "session":
                result = sessionScope.get(cachekey);  
           }  
           // if the result is empty, no cache was available or not requested,  
           //  do the dblookup, convert to array if not, cache it when requested  
           if (!result) {  
                // determine database to run against  
                var db = "";  
                if (!dbname.equals("")) { // if a database name is passed, build server, dbname array  
                   if (server.equals("")){
                           db = new Array(@DbName()[0],dbname); // no server specified, use server of current database  
                   } else if (dbname.indexOf("!!")!=-1 || dbname.indexOf(":")!=-1){
                           db = dbname;  // string value if double bang or replicaID spec
                   } else {
                                       db = new Array(server, dbname);
                   }  
                }
                var result = @DbLookup(db, viewname, keyname, field, keywords);
                // assume Mark Leusink's debug toolbar installed
                if (result==undefined){
                        dBar.error("DbLookup returned undefined, cachekey=" + cachekey);
                        // have result, process it
                } else if (result) {
                        // cache before manipulating
                             switch (cache.toLowerCase()) {
                                        case "application":
                                        result = applicationScope.put(cachekey,result);  
                                        break;
                                   case "request":
                                        result = requestScope.put(cachekey,result);  
                                        break;
                                   case "view":
                                        result = viewScope.put(cachekey,result);  
                                        break;
                                   case "session":
                                        result = sessionScope.put(cachekey,result);  
                                }  
                        if (typeof result == "string") {
                                result = new Array(result);  
                        } else {
                                // sort and unique only apply if multiple results
                                if (unique.toLowerCase()=="unique") result = @Unique(result);  
                                if (sortit.toLowerCase()=="sort") result.sort();
                        }
                }
           // we cached before operations on result set performed, so redo these if necessary
           }  else {
                if (typeof result == "string") {
                        result = new Array(result);  
                } else {
                                // sort and unique
                       if (unique.toLowerCase()=="unique") result = @Unique(result);  
                       if (sortit.toLowerCase()=="sort") result.sort();
                }
       }
       } catch(e){
               // this is our own error capture routine
               result=jsError(e);
       } finally {
               return result;
       }
}  

• Comments [0]

Day of Week

John McCann  February 29 2012 11:20:27 AM

I picked up someone else's code today and saw a technique that struck me.   The application was set up to need a three character day of week.

I look at the code and see the following construct:

       intDayOfWeek = Weekday(Now)
       Select Case intDayOfWeek
               Case Is = 1
                       strDayOfWeek = "sun"
               Case Is = 2
                       strDayOfWeek = "mon"
               Case Is = 3
                       strDayOfWeek = "tue"
               Case Is = 4
                       strDayOfWeek = "wed"
               Case Is = 5
                       strDayOfWeek = "thu"
               Case Is = 6
                       strDayOfWeek = "fri"
               Case Is = 7
                       strDayOfWeek = "sat"
       End Select

Besides the unnecessary "Is = ", something just nagged at me as there has got to be a better way.  Scratched around for a few minutes and came up with what I think is a simpler solution:

       strDayOfWeek = lcase$(format$(Now(),"ddd"))

The found construct was only in one time initialization code, so performance effect is probably minimal - a slightly smaller module with execution difference smaller than the ticks with which you are measuring.  One could even argue that the first construct is clearer in what is being done so is more maintainable.  Something in me just likes the transforming solution.

• Comments [0]

D’OH moment on getDocument performance

John McCann  February 2 2012 12:23:35 PM

We had just had a discussion concerning performance - an agent that has to collect information on a few thousand documents to present data to a dashboard.   The agent carefully uses a NotesViewNavigator and ReadViewEntries to maximize performance.

I was debugging the agent to fix a few typos in an update I made and had reason to look at the Domino server console for potential error messages.  I saw on the console, warning messages from the anti-virus software about an attachment it couldn't scan.

Then, D'OH, I made the connection.   It isn't just that using the NotesView and ViewEntries to get the information you need is so much more efficient than getting a document collection and opening documents.  I suspect that the real performance gains are the fact that you avoid the antivirus scan of the document and all the attachments.

• Comments [0]

LotusScript needs destructors too

John McCann  August 6 2011 07:21:34 AM

I was upgrading a major application to a new release.  When we deployed on the QA system and started running against the full data load, the overnight processing agent that goes in and touches a good percentage of the 100K records in the application started crashing the server.   We got the dreaded "LSXBE: ****** Out of Backend Memory *******"" error.    We reported the problem to IBM who asked for all the logs, copies of the database, etc.   After having a good laugh that they would not be getting a copy of databases, they eventually pointed us to a posting that effectively said, "The cause of the problem is not recycling Domino Java API objects correctly."   Only problem is that the agent was written in LotusScript, not Java.

We had a number of false starts and red herrings (session.getDatabase will not return a database object if consistency check going on - duh).   The failing call stack, as shown below, had us focusing on DocumentCollection processing.  We were expecting a failure on getting or traversing a collection.  We never tracked it down exactly, but are pretty sure the actual failure was on a NotesDocumentCollection.getNextDocument statement.  The failing stack was:

### FATAL THREAD 1/3 [   nAMgr:  0f34:  0324]
Exception code: c0000005 (ACCESS_VIOLATION)
@[ 1] 0x60001706 nnotes.OSLockWriteSem@4+22 (19a)
@[ 2] 0x61cf4a78 nlsxbe.ANNote::ANNAddToCollList+56 (0)
@[ 3] 0x61d26127 nlsxbe.ANDocColl::ANDCNavigate+535 (0)


In one of the iterations of trying to isolate exactly where the problem was occurring, IBM support (thank you Charles) pointed out that we just may be running into resource limitations on the machine since it looked like we were running into memory problems.  It seems we had all these objects sitting around in memory.   We started looking at the code and said, no, we clean up.  See, for each record in the primary database, we create an object, process it, then set the object to Nothing at the end of handling the record.  The object we create has other objects it creates, but LotusScript should be cleaning up all those when the primary object gets destroyed, right?   Seemed logical, only in Java do you have to do your own recycling.    Hmm, I mused.   Are there any known problems with NotesDateTime variables and memory leaks?  The big change in this upgrade was to add using NotesDateTime variables instead of LotusScript date/time variants since we started worrying about time zones and localizing date and times shown.   Charles looked it up and indicated nothing known there.

So, just for chagrins, when adding the latest tracing to produce a bunch more messages, I went through and added destructors to all my classes and changed the set to nothing to delete statements.  (I also found out that unless a class has a Delete method, don't invoke delete on the class).    Lo and behold, the problem goes away.

I have neither the tools nor the time to examine internal memory usage at points in time.  I will leave that to the Lotus developers.  However, what I think I have discovered empirically is one of two situations.  And, I don't know which one it is, so I left the code in for both thinking (perhaps wrongly) that it does no harm.

1) LotusScript will not clean up a list of objects contained within another object
2) A NotesDateTime contained within an object is not properly cleaned up when the object is destroyed.

Simplified code before (real routine is over 8K lines of LotusScript)

Option Explicit
Public Class cFoo
Public ndtEvent  as New NotesDateTime("")
End Class

Public Class cBar
Public lstFoo List as cFoo
Sub New
  Dim i as Long
  Dim oFoo as cFoo
  For i = 1 to 10
     Set oFoo = New cFoo
     Set lstFoo(i) = oFoo
  Next I
End Sub
End Class

Sub Initialize
Dim session as new NotesSession
Dim dbCur as NotesDatabase
Dim vwCur as NotesView
Dim docCur as NotesDocument
Dim oBar as cBar

Set dbCur = session.CurrentDatabase()
Set vwCur = dbCur.GetView("SomeView")
Set docCur = vwCur.GetFirstDocument()
While not docCur is Nothing
     Set oBar = new cBar
     ' **** do something with the document
    Set oBar = Nothing
    Set docCur = vwCur.GetNextDocument(docCur)
Loop
End Sub

I have not compiled the above routine nor tested it, so it may have typos.  But, it should provide the basics of what was going on.   I assumed (bad choice by me) that the set of oBar to Nothing would clean up the memory of the cBar object as well as the 10 cFoo objects it created.   When I changed the code to have my own destructor routines and invoke Delete as appropriate, the memory problems went away.    As indicated above, I wasn't able to determine the root cause of the memory problem, so I shotgunned and cleaned up things that had changed (Datetime) and things I knew about.

Option Explicit
Public Class cFoo
Public ndtEvent  as New NotesDateTime("")
Sub Delete
  Set ndtEvent = Nothing
End Sub
End Class

Public Class cBar
Public lstFoo List as cFoo
Sub New
  Dim i as Long
  Dim oFoo as cFoo
  For i = 1 to 10
     Set oFoo = New cFoo
     Set lstFoo(i) = oFoo
  Next I
End Sub
Sub Delete
  Forall Foo in lstFoo
      Delete Foo
  End Forall
End Sub
End Class

Sub Initialize
Dim session as new NotesSession
Dim dbCur as NotesDatabase
Dim vwCur as NotesView
Dim docCur as NotesDocument
Dim oBar as cBar

Set dbCur = session.CurrentDatabase()
Set vwCur = dbCur.GetView("SomeView")
Set docCur = vwCur.GetFirstDocument()
While not docCur is Nothing
     Set oBar = new cBar
     ' **** do something with the document
    Delete oBar
    Set docCur = vwCur.GetNextDocument(docCur)
Loop
End Sub

Let me know if this solves a memory problem for you and I will pass the feedback to Lotus support that there really is a problem in this area.

2011-08-18 Update: It has been suggested that delete method for cBar Erase the lstFoo element instead.   Not having the time to test all the options, I've updated my production code to do both - issue the delete then Erase lstFoo.

• Comments [0]

Document Collection loses position in recursive function

John McCann  May 4 2011 12:54:17 PM

As Mr. Pyle would say, surprise, surprise. surprise.

I have a hierarchical construct of organizations.  Each organization has a field giving the organization to which it is subordinate, i.e. the parent organization.

I was attempting to get all the organizations subordinate to a particular organization by traversing the tree with a recursive function within a class construct.    I would create a group and invoke a function within the group to get all directly subordinate groups, passing the NotesView of groups by parents.   Within the function, I would establish a document collection for the groups subordinate to this one, i.e. those that consider this their parent, process all those groups, and recursively tell each to get its subordinates.

To traverse the document collection in the recursive function, I was using the more efficient getfirstdocument/getnextdocument method.   Much to my surprise, the application only traversed down the first leg of the tree.  The getnext was losing position after the first pop of the stack.  I had to change the routine to use getNthDocument to process the document collection and everything worked just fine.

• Comments [0]

Notes and Sametime performance with JVM Heap Size

John McCann  September 3 2010 11:41:34 AM

I was having a problem with Domino Designer in 8.5.2, when I was editing a VERY large script library (LotusScript) and pasted a line of code into it.   Eclipse showed the message about correcting indentation and would do nothing else for minutes at a time.  While researching this problem, I found out two interesting things that might affect you all:

1) In the eclipse editor, if you are in the mode that displays all the code at once, pasting can be VERY slow.  Switch to the mode that displays a single function or class at once and it will be faster.   At least one user reported this to Lotus as a bug.

2) JVM Heap Size can dramatically affect performance.   Lotus Notes 8.5 (and Sametime and Designer 8.5) use the JVM virtual machine.  The default settings for Heap Size for my system at 8.5.2 were start with 48M and grow it to 256M.    For my machine with 6GB of memory, this is ridiculously low.   Lots of time is being spent expanding and purging the Java Heap.    I set mine to 1024M for both and 512/256 for Sametime.  As reported on a number of posts from some skilled Notes folks, it noticeably speeds up performance.

Details

For Notes JVM (and Designer):

[NotesProgramDirectory]\framework\rcp\deploy\jvm.properties

For my system it was
vmarg.Xmx=-Xmx256m
vmarg.Xms=-Xms48m

I changed it to
vmarg.Xmx=-Xmx1024m
vmarg.Xms=-Xms1024m

I have seen recommendations to make this no more than 1/3 or 1/2 of memory size.  Not being memory constrained, I didn't test which is better.  Never exceed physical RAM or performance will really suffer more than it already does..

Sametime has a similar properties file which you can change to improve Sametime startup and performance.   It is in
C:\Program Files\IBM\Lotus\Sametime Connect\rcp\eclipse\plugins\com.ibm.rcp.jcl.desktop.win32.x86_???????????????\jvm.properties

Not sure what values the question marks will have on your system

I changed mine to:
vmarg.Xms=-Xms128M
vmarg.Xmx=-Xmx512m


I hope this helps you, it definitely helped me.

• Comments [0]

Obtaining mail.box in LotusScript

John McCann  July 24 2010 12:00:00 PM

Well, I finally tracked down another bug that has been plaguing me - failures in opening the mail.box database directly.

In a WebQuerySave agent for a form,  I was sending MIME formatted email messages.  As is a commonly documented technique, I open the server mail.box directly and created the Notes document directly there.  When we upgraded from Domino 8.0.2 to 8.5.1 on the development systems (never tried 8.5), these agents failed.  The failure was that the database was not open.    I had been using a construct similar to the following to set the NotesDatabase variable:

Set dbMail = session.GetDatabase(session.currentdatabase.server, "mail.box",false)
If dbMail is Nothing then
      Set dbMail = session.GetDatabase(session.currentdatabase.server, "mail1.box",false)
End if

This code had been running merrily since Domino 6.x and probably before.  In Domino 8.5.1 it fails.  The variable dbMail is "Nothing".     The first time I ran into this, I used the quick workaround of the complete file path:

Set dbMail = session.GetDatabase(session.currentdatabase.server, "c:\lotus\domino\data\mail.box",false)

I knew this was bad, but needed something quick.

I spent some time today exploring the known problem of the format of Notesdatabase.server being unpredictable.  It may be flat, abbreviated, or canonical depending on the phase of the moon.  IBM's announced intention is to never fix the unpredictability.  I found that my problem wasn't dependent on the format of the server name field even though I believed the relationship between moon phases and what is returned has changed between releases.

But, this led me to the "duh" moment.  Why am I passing server name in the first place?  The agent runs on the server.  I only want the mail box on that server.  So, take the server name out.  The code is now as shown below and works like a champ, independent of data directory location, on servers with and without multiple mail boxes.

Set dbMail = session.GetDatabase("", "mail.box",false)
If dbMail is Nothing then
      Set dbMail = session.GetDatabase("", "mail1.box",false)
End if

This also solve a problem I had been seeing on the console

Error connecting to server XYZZY, Remote system no longer responding.

This message was the result of attempting to open the mail.box file on the same server.  Removing the server name from the getDatabase eliminates this message..

• Comments [0]

Creating an ISO Date in Lotus Notes Formula Language

John McCann  July 21 2010 06:00:00 PM

Another posting on the lack of ISO 8601 support in Lotus Domino.  

I had a value in a date variable and had to display it on the web in a locale and timezone independent format.   Dojo has been chosen as the tool to use, so I was trying to display the data using Dojo.   My fun was finding out how to get Lotus to display the date time using ISO standards.  While I can display yyyy-mm-dd for a date only variable, forget the yyyy-mm-ddThh:mm:ssTZ needed for date time.   Oh, and I need to display the UTC, GMT, or Zulu time (all the same, just different names).

My solution is to use a computed for display field or computed text and the following (for field XYZZY):

$txtTime := @If(@IsTime(XYZZY);@Explode(@TimeToTextInZone(XYZZY;"Z=0$DO=0";"D0T1S2");" /:");"");

@If(        $txtTime="";

                "";

        @Elements($txtTime)<6;"";

                $txtTime[3] + "-" +  $txtTime[1] + "-" + $txtTime[2] + "T" +

                @If($txtTime[6]="PM"; @Text(@TextToNumber($txtTime[4])+12); $txtTime[4]) + ":" + $txtTime[5] + "Z"

        )

There is a little more to get dojo to display it in the browser's locale and timezone, but this at least gets the value out of Domino in the proper format.

• Comments [0]