Solr overview and tuning

BOA's Solr integration is per-site and INI-driven. The Solr-side core lifecycle is controlled entirely by three variables in each site's boa_site_control.ini; there is no per-platform Solr state (boa_platform_control.ini Solr keys are ignored). Solr 9 is BOA PRO only.

The authoritative INI contract lives in this page; the daemon that enforces it is documented in Core lifecycle.

The per-site INI contract

In <platform-root>/sites/<domain>/modules/boa_site_control.ini:

solr_integration_module = search_api_solr9
solr_update_config      = NO
solr_custom_config      = NO

manage_solr_config.sh reads these on every four-minute pass and reconciles the site's Solr core against them. A missing site INI is seeded from /data/conf/default.boa_site_control.ini.

solr_integration_module

Selects the integration module and therefore the core type and Solr version.

The search_api_solr / search_api_solr7 / search_api_solr9 labels all resolve under the shared search_api_solr config tree, but the deployed config family is a per-Drupal-version subdirectory of it, not one flat family: search_api_solr9 on a D10 platform is served from solr9_drupal10, while search_api_solr / search_api_solr7 map to the solr7_drupalN families (D7 gets solr7_drupal7, passed to create_core -d at core creation).

Core version is switched by which Solr daemon the label targets (Solr 7 on port 9077, Solr 9 on port 9099).

When the value is removed or no longer names a recognised module, the daemon issues solr delete against every version's data dir for the computed core ID (and its _Old_/_Legacy_ aliases) and removes the site's solr.php. This is the explicit deletion path (BOA cleanly deletes the core within 15 minutes), distinct from the age-based orphan archival in Core lifecycle.

solr_update_config

Controls auto-refresh of schema.xml + solrconfig.xml when the integration module version changes:

Ignored when solr_custom_config = YES.

solr_custom_config

solr_custom_config = YES locks the core's config across upgrades (intentional immutability).

The transient-NO pattern is the supported way to pick up a Drupal-generated config without permanent lock-in:

1. Set NO.
2. Let one pass deploy the new config.
3. Restore YES.

Other lifecycle behaviour (deletion on INI removal, solr.php refresh, optimisation) still runs under YES.

solr.php

Once a core exists the daemon writes sites/<domain>/solr.php with the core's connection details. It is not used to connect to the core — Drupal does not include it; it is information only, for reading off the credentials when you configure the search_api_solr server in the site. It is removed when the core is deleted.

Core continuity across site operations

The Solr core is keyed to the site (oct.<user>.<domain>), not the platform:

• Migration between platforms — the core stays attached. After migration, Verify the site; solr_integration_module and solr.php are preserved; no re-index needed, provided the deployed schema.xml/solrconfig.xml remain compatible with the integration-module version in the codebase.
• Cloning — the clone gets its own fresh core (unique solr.php, empty index) within a few passes. The clone's Drupal config must be pointed at its new Solr server before indexing. Cron is not enabled on the cloned site by default, which prevents the clone from overwriting the original site's index.
• Cross-host migration (xoct/xcopy transfer, xmass sync/cutover) — Solr data (/opt/solr4, /var/solr7/data, /var/solr9/data) is rsynced data-only into the target's existing Solr setup, following the target's own /var/solrN layout (including a /var/solrN symlink onto attached storage). Each Solr transfer is pre-checked for disk space — during the migration DRY run and again at --live: an index that will not fit at its target destination prints DENY, marks the run NOT_CLEAN (so --live is refused until resolved), and that Solr transfer is skipped. See Cross-host migration for the full DRY/--live workflow.

Tuning and diagnostics

A Solr-enabled host accumulates orphan cores (deleted/renamed/cloned-site cores left on disk) and high-write cores stress the JVM heap. The reconciliation daemon automates orphan archival and ratio-driven optimisation; the operator-facing work is diagnosis and high-write core configuration.

Read the GC log

tail -f /var/solr9/logs/solr_gc.log      # Solr 7: /var/solr7/logs/solr_gc.log

Healthy: ParNew (young gen) collecting normally; Old gen declining or stable; CMS cycles infrequent.

Orphan-core accumulation (the key indicator): Old gen identical before/after on every cycle, CMS firing every ~2 s and freeing nothing:

GC(N)  Old: 1642215K->1642215K(2759348K)
GC(N)  Old: 1642215K->1642215K(2759348K)

Each loaded core holds live heap; the Old gen flatlines. Resolution: let the orphan-cleanup pass archive stale cores (or force it — see Core lifecycle). The pass unloads each core from the live JVM before moving it, so its held heap references are released the moment it is archived — no Solr restart is required. Archiving a large backlog of orphan cores typically drops Old gen by hundreds of MB within the same hour and ends the continuous CMS spin; a full restart is at most an optional belt-and-braces step.

Abortable Preclean running for seconds in solr_gc.log:

GC(N)  Concurrent Abortable Preclean 6014.616ms

Unlike the frozen Old gen above, this signals a sudden allocation burst — a large indexing job, a query spike, or (most often) many per-document commits churning searchers — not orphan accumulation. It points at write pressure, so treat it as a cue for the high-write tuning below, not orphan cleanup.

Overlapping onDeckSearchers in /var/solr9/logs/solr.log:

PERFORMANCE WARNING: Overlapping onDeckSearchers=2

Searchers created faster than they warm — per-document commits firing too quickly. Fix via high-write tuning below.

Normal post-restart: brief ~2 s CMS cycling that burns off startup allocation with Old gen decreasing sweep over sweep is not a leak; if Metaspace is also decreasing, classloader cleanup is working.

Index optimisation (ratio-driven, 12 h throttle)

Optimisation is automatic and per deleted-doc ratio — no fixed cadence and no per-core list. _optimize_solr_cores runs via the throttle wrapper _run_optimize_if_due (sentinel /var/backups/solr/.optimize_last_run.pid, at most once per 12 h, _OPTIMIZE_INTERVAL_HOURS=12). For each registered core it reads numDocs/maxDoc from the admin STATUS API, computes the deleted-doc percentage, and decides:

A protected core (conf/.protected.conf) never gets the forced full merge even at >= 30% — it may carry its own tuned merge policy, so it gets expungeDeletes only.

Every update uses waitFlush=false: the request returns immediately and the merge continues in the background, so run time stays bounded regardless of index size.

# What the daemon issues internally (Solr 9; 9077 for Solr 7):
curl 'http://127.0.0.1:9099/solr/<core>/update?expungeDeletes=true&waitFlush=false'
curl 'http://127.0.0.1:9099/solr/<core>/update?optimize=true&maxSegments=1&waitFlush=false'

Low-write and read-only cores naturally fall under the threshold and are skipped; there is no manual weekly schedule.

If you ever run a manual full optimize on a large core instead of leaving it to the daemon, run it inside screen: on a 10 GB+ index it takes 30–120 minutes with heavy I/O, and disk usage temporarily doubles while Lucene writes the merged segments alongside the originals (the old files are swapped and deleted atomically only when the merge completes), so confirm free space first. Never interrupt an in-progress optimize — the writes are atomic, so interrupting only discards the work done so far and may leave temporary merge files to clean up on the next restart.

High-write core tuning

For sites used as an API layer with continuous index churn, stock search_api_solr config underperforms. Apply these in the core's conf/solrconfig.xml, alongside a conf/.protected.conf sentinel so BOA does not overwrite the file on the next config sync. Highest-impact first:

1. ramBufferSizeMB 32 → 256 — fewer flushes, fewer tiny segments, less merge work.
2. Switch to TieredMergePolicy (deletesPctAllowed=20) — handles high delete ratios far better than the default LogByteSizeMergePolicy; keeps the delete ratio self-healing.
3. infoStream → false — stop logging every merge (floods solr.log on a high-write core).

Commit/searcher tunables that specifically cure overlapping onDeckSearchers:

<autoCommit>
  <maxTime>300000</maxTime>          <!-- 5 min; app issues its own commits -->
  <openSearcher>false</openSearcher>  <!-- don't reopen searcher on hard commit -->
</autoCommit>

<autoSoftCommit>
  <maxTime>10000</maxTime>          <!-- 10s near-realtime; drop the maxDocs trigger -->
</autoSoftCommit>

<maxWarmingSearchers>4</maxWarmingSearchers>  <!-- default 2 → 4 -->

Hard commits no longer open searchers, soft commit is time-only (no per-document maxDocs churn), and more warming-searcher headroom keeps frequent soft commits from exhausting the pool. Apply via the solr_custom_config = YES route so the config survives upgrades.

Complete high-write solrconfig.xml

The changes above are the operator-critical deltas. For a full copy-paste drop-in, the complete high-write block also pins the tiered-merge sub-parameters, commit retention, and lock type. Apply it to conf/solrconfig.xml behind a conf/.protected.conf sentinel; every setting not shown stays at the shipped search_api_solr defaults.

<indexConfig>
  <ramBufferSizeMB>256</ramBufferSizeMB>            <!-- from 32: fewer flushes, fewer tiny segments -->
  <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
    <int name="maxMergeAtOnce">10</int>
    <int name="segmentsPerTier">10</int>
    <double name="maxMergedSegmentMB">8192</double>
    <double name="deletesPctAllowed">20</double>    <!-- self-heals the delete ratio -->
  </mergePolicyFactory>
  <lockType>${solr.lock.type:native}</lockType>
  <reopenReaders>true</reopenReaders>
  <deletionPolicy class="solr.SolrDeletionPolicy">
    <str name="maxCommitsToKeep">1</str>
    <str name="maxOptimizedCommitsToKeep">0</str>
  </deletionPolicy>
  <infoStream>false</infoStream>                     <!-- from true: stop per-merge log flooding -->
</indexConfig>

<updateHandler class="solr.DirectUpdateHandler2">
  <autoCommit>
    <maxDocs>${solr.autoCommit.MaxDocs:10000}</maxDocs>
    <maxTime>${solr.autoCommit.MaxTime:300000}</maxTime>   <!-- 5 min; app commits itself -->
    <openSearcher>false</openSearcher>
  </autoCommit>
  <autoSoftCommit>
    <maxTime>${solr.autoSoftCommit.MaxTime:10000}</maxTime>  <!-- 10s; no maxDocs trigger -->
  </autoSoftCommit>
  <updateLog>
    <str name="dir">${solr.data.dir:}</str>
  </updateLog>
</updateHandler>

<query>
  <!-- ... standard settings ... -->
  <maxWarmingSearchers>4</maxWarmingSearchers>       <!-- from 2 -->
</query>

Core health checks

# Per-core stats via Solr admin
curl -s 'http://127.0.0.1:9099/solr/<core>/admin/luke?wt=json&numTerms=0' | jq
# Core status (the CoreAdmin STATUS action the daemon's own health check uses)
curl -s 'http://127.0.0.1:9099/solr/admin/cores?action=STATUS&wt=json' | jq '.status | keys'
# Java process memory
ps -o pid,vsz,rss,cmd -p $(pgrep -f solr) | awk '{print $1, $2/1024 " MB VSZ", $3/1024 " MB RSS"}'
# Cache hit/eviction stats
curl -s 'http://127.0.0.1:9099/solr/admin/mbeans?cat=CACHE&stats=true'
# Facet breakdown of what is indexed — explains an unexpectedly high doc count
curl -s 'http://127.0.0.1:9099/solr/<core>/select?q=*:*&rows=0&facet=true&facet.field=ss_type'

D7 apachesolr_attachments Java path

When apachesolr_attachments cannot find a Java binary, pin it in sites/<domain>/local.settings.php:

$conf['apachesolr_attachments_java'] = '/usr/bin/java11 -Xms32m -Xmx64m';

/usr/bin/java11 exists on every Solr 4/7-capable host (the Solr 4 path runs under Java 11).

Related

• Solr 9 modules — the deployed .mod set and how modules load.
• Core lifecycle — the reconciliation daemon, orphan archival, and forced runs.
• Reference appendix — solr_integration_module, solr_update_config, solr_custom_config.