WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Conversation

@zihanx
Copy link

@zihanx zihanx commented Dec 16, 2025

Summary

This PR enhances the runOnMergeFinished hook in ConcurrentMergeScheduler
to provide access to OneMerge and MergeRateLimiter parameters.

Motivation

Currently, applications that subclass ConcurrentMergeScheduler to monitor
merge behavior have limited context in the runOnMergeFinished hook. The
MergeRateLimiter contains valuable metrics about merge performance
(pause time, throttling, MB/sec) that would be useful for:

  • Monitoring merge performance in production
  • Building custom metrics and dashboards
  • Debugging merge-related issues
  • Optimizing merge scheduler configurations

Changes

  • Changed runOnMergeFinished from package-private to protected to allow subclassing
  • Added OneMerge merge parameter for merge context
  • Added MergeRateLimiter rateLimiter parameter to expose merge rate metrics

Backward Compatibility

This is a breaking change for any code that currently overrides runOnMergeFinished,
but since the method was package-private, external impact should be minimal.

Example Use Case

Applications can now override this method to emit custom metrics:

@Override
protected synchronized void runOnMergeFinished(MergeSource mergeSource, 
                                                OneMerge merge, 
                                                MergeRateLimiter rateLimiter) {
    super.runOnMergeFinished(mergeSource, merge, rateLimiter);
    
    // Emit custom metrics
    recordMetric("merge.paused.ns", rateLimiter.getTotalPausedNS());
    recordMetric("merge.stopped.ns", rateLimiter.getTotalStoppedNS());
    recordMetric("merge.mbps", rateLimiter.getMBPerSec());
}

Changes runOnMergeFinished method signature in ConcurrentMergeScheduler
to provide access to OneMerge and MergeRateLimiter parameters. This
allows applications to monitor merge performance metrics like pause time,
throttling behavior, and throughput.

- Changed runOnMergeFinished from package-private to protected
- Added OneMerge merge parameter for merge context
- Added MergeRateLimiter rateLimiter parameter for metrics access

This enables custom monitoring and observability for merge operations
without requiring internal API access.
@mikemccand
Copy link
Member

[Disclaimer: @zihanx and I work together, at Amazon customer facing product search]

Thanks @zihanx -- these metrics are really helpful merge intensive indexing. Lucene applies backpressure (halts incoming indexing threads to give merge backlog a chance to get back under the hard merge count limit) and you are flying blind if there's no way to monitor it.

Could you add @lucene.experimental to the API javadocs to make it clear it's not necessarily a frozen API?

Copy link
Member

@mikemccand mikemccand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great -- thank you @zihanx!

Oh sorry I forgot this the first time around: how about a CHANGES.txt entry describing this, and how it enables visibility on this important Lucene behavior (backpressure to indexing when there is too much merge debt)?

@github-actions github-actions bot added this to the 11.0.0 milestone Dec 19, 2025
@zihanx
Copy link
Author

zihanx commented Dec 19, 2025

Looks great -- thank you @zihanx!

Oh sorry I forgot this the first time around: how about a CHANGES.txt entry describing this, and how it enables visibility on this important Lucene behavior (backpressure to indexing when there is too much merge debt)?

Thanks @mikemccand! I have included an entry in CHANGES.txt and everything should be in place now :)

Copy link
Member

@mikemccand mikemccand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zihanx -- this looks great. I'll try to merge soon if there are no objections/feeddback...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants