Solr overview and tuning

BOA's Solr integration is per-site and INI-driven. The Solr-side core lifecycle is controlled entirely by three variables in each site's boa_site_control.ini; there is no per-platform Solr state (boa_platform_control.ini Solr keys are ignored).

The authoritative INI contract lives in this page; the daemon that enforces it is documented in Core lifecycle.

The per-site INI contract

In <platform-root>/sites/<domain>/modules/boa_site_control.ini:

solr_integration_module = search_api_solr9
solr_update_config      = NO
solr_custom_config      = NO

manage_solr_config.sh reads these on every four-minute pass and reconciles the site's Solr core against them. A missing site INI is seeded from /data/conf/default.boa_site_control.ini.

solr_integration_module

Selects the integration module and therefore the core type and Solr version.

The search_api_solr / search_api_solr7 / search_api_solr9 labels all resolve under the shared search_api_solr config tree, but the deployed config family is a per-Drupal-version subdirectory of it, not one flat family: search_api_solr9 on a D10 platform is served from solr9_drupal10, while search_api_solr / search_api_solr7 map to the solr7_drupalN families (D7 gets solr7_drupal7, passed to create_core -d at core creation).

Core version is switched by which Solr daemon the label targets (Solr 7 on port 9077, Solr 9 on port 9099).

When the value is removed or no longer names a recognised module, the daemon removes the computed core ID (and its _Old_/_Legacy_ aliases) from every version's data dir and removes the site's solr.php. The mechanism differs per version: Solr 9 cores are unloaded through the Core Admin API on 127.0.0.1:9099 (action=UNLOAD with deleteIndex, deleteDataDir and deleteInstanceDir), because the Solr 9 CLI ignores -p on delete and would target port 8983; Solr 7 cores are removed with solr delete -p 9077; Solr 4 cores are dropped from solr.xml, their directory deleted and jetty9 restarted. This is the explicit deletion path (BOA cleanly deletes the core within 15 minutes), distinct from the age-based orphan archival in Core lifecycle.

solr_update_config

Controls auto-refresh of schema.xml + solrconfig.xml when the integration module version changes:

Ignored when solr_custom_config = YES.

solr_custom_config

solr_custom_config = YES locks the core's config across upgrades (intentional immutability).

The transient-NO pattern is the supported way to pick up a Drupal-generated config without permanent lock-in:

1. Set NO.
2. Let one pass deploy the new config.
3. Restore YES.

Other lifecycle behaviour (deletion on INI removal, solr.php refresh, optimisation) still runs under YES.

solr.php

Once a core exists the daemon writes sites/<domain>/solr.php with the core's connection details. It is not used to connect to the core — Drupal does not include it; it is information only, for reading off the credentials when you configure the search_api_solr server in the site. It is removed when the core is deleted.

Core continuity across site operations

The Solr core is keyed to the site (oct.<user>.<domain>), not the platform:

• Migration between platforms — the core stays attached. After migration, Verify the site; solr_integration_module and solr.php are preserved; no re-index needed, provided the deployed schema.xml/solrconfig.xml remain compatible with the integration-module version in the codebase.
• Cloning — the clone gets its own fresh core (unique solr.php, empty index) within a few passes. The clone's Drupal config must be pointed at its new Solr server before indexing. Cron is not enabled on the cloned site by default, which prevents the clone from overwriting the original site's index.
• Cross-host migration (xoct/xcopy transfer, xmass sync/cutover) — Solr data (/opt/solr4, /var/solr7/data, /var/solr9/data) is rsynced data-only into the target's existing Solr setup, following the target's own /var/solrN layout (including a /var/solrN symlink onto attached storage). Each Solr transfer is pre-checked for disk space — during the migration DRY run and again at --live: an index that will not fit at its target destination prints DENY, marks the run NOT_CLEAN (so --live is refused until resolved), and that Solr transfer is skipped. See Cross-host migration for the full DRY/--live workflow.

Tuning and diagnostics

A Solr-enabled host accumulates orphan cores (deleted/renamed/cloned-site cores left on disk) and high-write cores stress the JVM heap. The reconciliation daemon automates orphan archival and ratio-driven optimisation; the operator-facing work is diagnosis and high-write core configuration.

Read the GC log

tail -f /var/solr9/logs/solr_gc.log      # Solr 7: /var/solr7/logs/solr_gc.log

Healthy: ParNew (young gen) collecting normally; Old gen declining or stable; CMS cycles infrequent.

Orphan-core accumulation (the key indicator): Old gen identical before/after on every cycle, CMS firing every ~2 s and freeing nothing:

GC(N)  Old: 1642215K->1642215K(2759348K)
GC(N)  Old: 1642215K->1642215K(2759348K)

Each loaded core holds live heap; the Old gen flatlines. Resolution: let the orphan-cleanup pass archive stale cores (or force it — see Core lifecycle). The pass unloads each core from the live JVM before moving it, so its held heap references are released the moment it is archived — no Solr restart is required. Archiving a large backlog of orphan cores typically drops Old gen by hundreds of MB within the same hour and ends the continuous CMS spin; a full restart is at most an optional belt-and-braces step.

Abortable Preclean running for seconds in solr_gc.log:

GC(N)  Concurrent Abortable Preclean 6014.616ms

Unlike the frozen Old gen above, this signals a sudden allocation burst — a large indexing job, a query spike, or (most often) many per-document commits churning searchers — not orphan accumulation. It points at write pressure, so treat it as a cue for the high-write tuning below, not orphan cleanup.

Overlapping onDeckSearchers in /var/solr9/logs/solr.log:

PERFORMANCE WARNING: Overlapping onDeckSearchers=2

Searchers created faster than they warm — per-document commits firing too quickly. Fix via high-write tuning below.

Normal post-restart: brief ~2 s CMS cycling that burns off startup allocation with Old gen decreasing sweep over sweep is not a leak; if Metaspace is also decreasing, classloader cleanup is working.

Index optimisation (ratio-driven, 12 h throttle)

Optimisation is automatic and per deleted-doc ratio — no fixed cadence and no per-core list. _optimize_solr_cores runs via the throttle wrapper _run_optimize_if_due (sentinel /var/backups/solr/.optimize_last_run.pid, at most once per 12 h, _OPTIMIZE_INTERVAL_HOURS=12). For each registered core it reads numDocs/maxDoc from the admin STATUS API, computes the deleted-doc percentage, and decides:

A protected core (conf/.protected.conf) never gets the forced full merge even at >= 30% — it may carry its own tuned merge policy, so it gets expungeDeletes only.

Every update uses waitFlush=false: the request returns immediately and the merge continues in the background, so run time stays bounded regardless of index size.

# What the daemon issues internally (Solr 9; 9077 for Solr 7):
curl 'http://127.0.0.1:9099/solr/<core>/update?expungeDeletes=true&waitFlush=false'
curl 'http://127.0.0.1:9099/solr/<core>/update?optimize=true&maxSegments=1&waitFlush=false'

Low-write and read-only cores naturally fall under the threshold and are skipped; there is no manual weekly schedule.

If you ever run a manual full optimize on a large core instead of leaving it to the daemon, run it inside screen: on a 10 GB+ index it takes 30–120 minutes with heavy I/O, and disk usage temporarily doubles while Lucene writes the merged segments alongside the originals (the old files are swapped and deleted atomically only when the merge completes), so confirm free space first. Never interrupt an in-progress optimize — the writes are atomic, so interrupting only discards the work done so far and may leave temporary merge files to clean up on the next restart.

High-write core tuning

For sites used as an API layer with continuous index churn, stock search_api_solr config underperforms. Apply these in the core's conf/solrconfig.xml, alongside a conf/.protected.conf sentinel so BOA does not overwrite the file on the next config sync. Highest-impact first:

1. ramBufferSizeMB 32 → 256 — fewer flushes, fewer tiny segments, less merge work.
2. Switch to TieredMergePolicy (deletesPctAllowed=20) — handles high delete ratios far better than the default LogByteSizeMergePolicy; keeps the delete ratio self-healing.
3. infoStream → false — stop logging every merge (floods solr.log on a high-write core).

Commit/searcher tunables that specifically cure overlapping onDeckSearchers:

<autoCommit>
  <maxTime>300000</maxTime>          <!-- 5 min; app issues its own commits -->
  <openSearcher>false</openSearcher>  <!-- don't reopen searcher on hard commit -->
</autoCommit>

<autoSoftCommit>
  <maxTime>10000</maxTime>          <!-- 10s near-realtime; drop the maxDocs trigger -->
</autoSoftCommit>

<maxWarmingSearchers>4</maxWarmingSearchers>  <!-- default 2 → 4 -->

Hard commits no longer open searchers, soft commit is time-only (no per-document maxDocs churn), and more warming-searcher headroom keeps frequent soft commits from exhausting the pool. Apply via the solr_custom_config = YES route so the config survives upgrades.

Deploying the changed config to a live core — no Solr restart needed. A solrconfig.xml edit is inert until the core re-reads it; the whole-daemon restart is the blunt way. The precise way: back the current file up, copy the new config in, then re-register it with a core RELOAD (substitute your Solr port and core name):

curl "http://127.0.0.1:9077/solr/admin/cores?action=RELOAD&core=<corename>"

Skip the reload and the high-write tuning you just wrote does nothing — the running core keeps its loaded config until something reloads it.

Complete high-write solrconfig.xml

The changes above are the operator-critical deltas. For a full copy-paste drop-in, the complete high-write block also pins the tiered-merge sub-parameters, commit retention, and lock type. Apply it to conf/solrconfig.xml behind a conf/.protected.conf sentinel; every setting not shown stays at the shipped search_api_solr defaults.

<indexConfig>
  <ramBufferSizeMB>256</ramBufferSizeMB>            <!-- from 32: fewer flushes, fewer tiny segments -->
  <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
    <int name="maxMergeAtOnce">10</int>
    <int name="segmentsPerTier">10</int>
    <double name="maxMergedSegmentMB">8192</double>
    <double name="deletesPctAllowed">20</double>    <!-- self-heals the delete ratio -->
  </mergePolicyFactory>
  <lockType>${solr.lock.type:native}</lockType>
  <reopenReaders>true</reopenReaders>
  <deletionPolicy class="solr.SolrDeletionPolicy">
    <str name="maxCommitsToKeep">1</str>
    <str name="maxOptimizedCommitsToKeep">0</str>
  </deletionPolicy>
  <infoStream>false</infoStream>                     <!-- from true: stop per-merge log flooding -->
</indexConfig>

<updateHandler class="solr.DirectUpdateHandler2">
  <autoCommit>
    <maxDocs>${solr.autoCommit.MaxDocs:10000}</maxDocs>
    <maxTime>${solr.autoCommit.MaxTime:300000}</maxTime>   <!-- 5 min; app commits itself -->
    <openSearcher>false</openSearcher>
  </autoCommit>
  <autoSoftCommit>
    <maxTime>${solr.autoSoftCommit.MaxTime:10000}</maxTime>  <!-- 10s; no maxDocs trigger -->
  </autoSoftCommit>
  <updateLog>
    <str name="dir">${solr.data.dir:}</str>
  </updateLog>
</updateHandler>

<query>
  <!-- ... standard settings ... -->
  <maxWarmingSearchers>4</maxWarmingSearchers>       <!-- from 2 -->
</query>

Core health checks

# Per-core stats via Solr admin
curl -s 'http://127.0.0.1:9099/solr/<core>/admin/luke?wt=json&numTerms=0' | jq
# Core status (the CoreAdmin STATUS action the daemon's own health check uses)
curl -s 'http://127.0.0.1:9099/solr/admin/cores?action=STATUS&wt=json' | jq '.status | keys'
# Java process memory
ps -o pid,vsz,rss,cmd -p $(pgrep -f solr) | awk '{print $1, $2/1024 " MB VSZ", $3/1024 " MB RSS"}'
# Cache hit/eviction stats
curl -s 'http://127.0.0.1:9099/solr/admin/mbeans?cat=CACHE&stats=true'
# Facet breakdown of what is indexed — explains an unexpectedly high doc count
curl -s 'http://127.0.0.1:9099/solr/<core>/select?q=*:*&rows=0&facet=true&facet.field=ss_type'

D7 apachesolr_attachments Java path

When apachesolr_attachments cannot find a Java binary, pin it in sites/<domain>/local.settings.php:

$conf['apachesolr_attachments_java'] = '/usr/bin/java11 -Xms32m -Xmx64m';

/usr/bin/java11 exists on every Solr 4/7-capable host (the Solr 4 path runs under Java 11).

Removing unused Solr versions — solrtrim

A hosted-named box's first full barracuda pass installs all three Solr versions (Solr 4 under Jetty 9, Solr 7, Solr 9); most boxes use one or none, and the unused ones idle as running JVMs. solrtrim (root-only, /opt/local/bin) is the standard way to take a version a box does not use off it:

# Classify each version's real use and forecast what the next
# full barracuda pass would (re)install
solrtrim status

# Remove one or more versions, or everything
solrtrim remove <solr4|solr7|solr9|all> [--force] [--blanket]

status scores use by index-write recency — the one-year bar. Queries leave no filesystem trace (the request logs are empty in practice), so the newest data/index write across a version's real cores (dotted names only — the Solr 4 installer seeds core0–core9 empty) is the only measurable use signal; document counts are reported as evidence but never decide the verdict, because a populated estate can be a years-dead leftover. Each version's newest-write date is printed so the verdict is auditable. Two carve-outs: a site's solr_integration_module binding (or a running service with its SR* token in _XTRAS_LIST) counts as used regardless of write age.

remove refuses two shapes without --force:

• a USED version (an index write inside the last year, or live bindings / an SR* opt-in) — removing it breaks live search;
• an ambiguous one (registered cores, all empty, zero bindings) — that is a per-box operator ruling, never dropped automatically.

A STALE version — populated cores whose newest write is older than a year — is a deny candidate by the same bar and removes without --force. Any removal of a version holding real cores archives the data tree first to /var/backups/solrtrim-<service>-data-<timestamp>.tar.gz and refuses when the archive cannot land. That refusal is not a no-op: the deny, the service stop and the deregistration described next have already happened by the time the archive is attempted, so the version is left down, denied and deregistered with its trees untouched — free the space (the precheck wants the data size plus 1G under /var/backups) or fix the cause, then re-run. Only the USED and ambiguous refusals above change nothing at all. Every removal writes the version's deny in both forms (_DENY_JETTY9/_DENY_SOLR7/_DENY_SOLR9 in /root/.barracuda.cnf plus the /etc/boa/.deny.*.cnf marker) before anything is deleted, then removes the service registration, init script, distribution and data trees, defaults file and system user. remove refuses while a barracuda/octopus run is in flight, and takes a single-flight lock, so a second remove refuses while one is still running.

The deny is what makes removal honest, not what makes it permanent. The per-version deny knobs act only after an install (the 5-minutely autoupboa parks the service), so on a hosted-shaped box the next full barracuda pass reinstalls a removed version — and autoupboa parks it again within minutes. It never serves, but it comes back to disk. solrtrim says this in its end-state report. The only install-time opt-out BOA has is the blanket pair _DENY_JAVA=YES + /root/.deny.java.cnf, which remove all --blanket writes once no version remains installed: that end state survives full barracuda passes with nothing reinstalling, and is exactly the shape a golden template must have — clones then get only the versions their box actually needs. A migration target's Solr set is gated against its source automatically by xmass prep-target.

Related

• Solr 9 modules — the deployed .mod set and how modules load.
• Core lifecycle — the reconciliation daemon, orphan archival, and forced runs.
• Reference appendix — solr_integration_module, solr_update_config, solr_custom_config.