Mailing List
Home
Forum Home
Maven - Project building tool
Axis - Java SOAP implementation
Lucene - Full-featured text search engine APIs
Cocoon - MVC web framework based on XML/XSL
Fop - Create PDF, PCL, PS, SVG, XML driven by XSL formatting objects.
Log4J - A log library
POI - Java Excel, Word and other Microsoft Office files manipulating library
Oracle database error code ...
Subjects
log4j warning: No appenders could be found
java security AccessControlException: access denied (java io FilePermission clie
java lang InstantiationException: org apache tools ant Main
Apache Axis Tutorial
Subject: Struts <logic iterate >
log4j properties How to parse outpu to multiple files
configuring log4j with BEA Weblogic 8 1
How to use XSL FOP Java together
JSP precompile
Proposal: Adding jar manifest classpath in jar and war plugins
Servlet File Download dialog problem (IE6,Adobe 6 0)
java security AccessControlException: access denied (java io FilePermission
Unsupported major minor version 48 0 problem while running the an
   telope task
Subject: axis wsdl2java Ant Task usage
net sf hibernate MappingException: Error reading resource: test/User hbm xml
Building EAR ANT Script for websphere 5 0
CREATING WAR Files
Classpath problem
jsp data into Excel
Jboss 3 2 3+ vs Tomcat Axis Question
RE: How to include jars and add them into the MANIFEST MF/Class Path
attribute
Printing problem
Subject: InstantiationException
Couldn 't find trusted certificate
Please : How can one install ant 1 6 0 under Eclipse 2 1 ?
Excel: Too many different cell formats
Subject: AXIS: tomcat timeout ?
1 3 final: now giving me java io FileNotFoundException (Too many
open files)
XDoclet, Struts and Maven: Where to start? SOLUTION
Subject: Running junit tests fails
 
SV: OutOfMemory-problems with SortComparatorSource / ScoreDocComparator

SV: OutOfMemory-problems with SortComparatorSource / ScoreDocComparator

2007-11-08       - By Tobias Hill

 Back
Hi Tobias,

I had the similar problem with lucene custom sorting about two years ago.

Please take a look at these two email threads:
http://www.gossamer-threads.com/lists/lucene/java-dev/39100
http://www.gossamer-threads.com/lists/lucene/java-user/38016

It seems like they did not make any patch for this problem since then.
But you could do it yourself - check one of the threads above, there
is small patch that force lucene to not cache in case of Custom
sorting.

P.S. I'm not subscribed (just checking RSS), so you could repost it to
mailing list.

Hope it helps,
Alex



__ ____ ____ ____ ____ ____ ____ ____ __
Fr?n: Tobias Hill [Tobias.Hill@(protected)]
Skickat: den 8 november 2007 18:34
Till: java-user@(protected)
?mne: OutOfMemory-problems with SortComparatorSource / ScoreDocComparator

Hi,

We have implemented a custom sort following the pattern in Lucene in Action.
Unfortunately this has led to quite serious memory problems. When analyzing
those (with a profiler) it seems that there are as many remaining instances of
our SortComparatorSource as there have been queries against the index.
Moreover, it seems that those are held by only one reference (per instance)
coming from some cache:ish feature of lucene itself.

Caching does not make sense for our comparator ... and if it has to be this
way (for a reason or other) it seems reasonable that the cache dropped the
reference if available memory is getting low so that gc could do it's thing.
Right?

When digging a bit deeper this also seems to be the intention, since I see
WeakHashMap entries in the reference chain. This leaves me puzzled. Why
are not our instances retained when there is only one incoming reference.
Weak as it seems too.

I would be very thankful if anyone could spot why this custom sorter just
piles up in the cache. See below.

Best regards,
Tobias



/**
* The BlendSorter can sort on any lucene field and blend its value
* with the document-score. The blend can be configured.
*/
public class BlendSorter implements SortComparatorSource {

   private final static Log log = LogFactory.getLog(BlendSorter.class);
   private FieldConverter fieldConverter;
   private float blendFactor;

   /**
    * Constructs a BlendSorter
    * @(protected) fieldConverter    The converter to use wen converting the lucent
field
    * @(protected) blendFactor       The factor to blend (0 means 0% fieldvalue and
100%
    *                          document-score, 1 means 100% fieldvalue and 0%
document-score).
    */
   public BlendSorter(FieldConverter fieldConverter, float blendFactor) {
       this.fieldConverter = fieldConverter;
       this.blendFactor = blendFactor;
   }

   public ScoreDocComparator newComparator(final IndexReader indexReader,
final String field) throws IOException {
       return new ScoreDocComparator() {
           Map<Integer, Float> values = new HashMap<Integer, Float>();

           public int compare(ScoreDoc scoreDoc1, ScoreDoc scoreDoc2) {
               try {
                   Float v1 = getValue(scoreDoc1, indexReader, field);
                   Float v2 = getValue(scoreDoc2, indexReader, field);
                   return v2.compareTo(v1);

               } catch (IOException e) {
                   log.error("Cannot read doc", e);
               }
               return 0;
           }

           public Comparable sortValue(ScoreDoc scoreDoc) {
               return values.get(scoreDoc.doc);
           }

           public int sortType() {
               return SortField.FLOAT;
           }


           // lazily get values, and store them in our value-map
           private Float getValue(ScoreDoc scoreDoc, IndexReader indexReader,
String field) throws IOException {
               Float value = values.get(scoreDoc.doc);
               if (value != null) return value;
               final Document doc = indexReader.document(scoreDoc.doc);
               float fieldValue = fieldConverter.get(doc.get(field));
               float queryValue = scoreDoc.score;
               value = blendFactor * fieldValue + (1 - blendFactor) *
queryValue;
               values.put(scoreDoc.doc, value);
               return value;
           }
       };
   }

}



-- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ------
To unsubscribe, e-mail: java-user-unsubscribe@(protected)
For additional commands, e-mail: java-user-help@(protected)


-- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ------
To unsubscribe, e-mail: java-user-unsubscribe@(protected)
For additional commands, e-mail: java-user-help@(protected)