Update YZ sim to address large memory usage by SFBayLaser · Pull Request #905 · SBNSoftware/icaruscode

SFBayLaser · 2026-06-02T18:47:38Z

This PR provides the necessary updates to utilize improvements in the wirecell toolkit to address the ICARUS YZ simulation's large memory usage. While the main changes are in the toolkit itself we have modified the detector simulation at the ICARUS level to break the TPC simulation into four separate producer modules. At the same time we are maintaining the non-YZ simulation to a single producer module. This requires updates at the stage 0 processing level as well for the two cases.

At this point the comparison of non-YZ (top) to YZ (bottom) cpu and memory usage is:

Generally, we are now running the YZ sim with more than a 2x reduction in memory usage

…is broken into four "daq" modules, one per TPC. These files make that definition, for standard YZ sim and also for overlay

… detsim

…yz sim uses 4 modules (daq0-daq3) while the standard sim continues to use only one (daq) so stage 0 will need to recognize this

… inherit from the standard one

Copilot

Pull request overview

This PR updates the ICARUS WireCell-based YZ simulation configuration to reduce peak memory usage by splitting the YZ TPC simulation into four per-TPC producer modules, while also refactoring the Jsonnet graph generation to avoid expensive g.uses() expansion/dedup behavior.

Changes:

Add a new per-TPC YZ Jsonnet graph (wcls-per-tpc-...) driven by tpc_idx, designed to be run by four separate WireCellToolkit modules.
Refactor the existing multi-TPC YZ Jsonnet to use shared YZMap nodes and a memoized DFS-based uses resolver; update sim.jsonnet to derive YZ pipeline counts from tools.anodes.
Update detsim and stage0 FHiCL wiring for the per-TPC YZ module labels (daq0..daq3) and adjust YZ SimChannel merge inputs accordingly.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
icaruscode/TPC/ICARUSWireCell/icarus/wcls-per-tpc-sim-drift-simchannel-yzsim.jsonnet	New per-TPC YZ simulation graph parameterized by `tpc_idx` to enable 4-module execution.
icaruscode/TPC/ICARUSWireCell/icarus/wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet	Refactors YZ graph pieces (YZMap, sink, uses traversal) and speeds up configuration emission.
icaruscode/TPC/ICARUSWireCell/icarus/wcls-multitpc-sim-drift-simchannel-refactored.jsonnet	Replaces `g.uses()` emission with the same memoized DFS resolver (non-YZ path).
icaruscode/TPC/ICARUSWireCell/icarus/sim.jsonnet	Makes YZ transform/reframer/analog pipeline counts depend on `nanodes` for per-TPC slicing.
icaruscode/TPC/ICARUSWireCell/detsimmodules_wirecell_per_TPC_ICARUS.fcl	New detsim module definitions for running YZ as four modules (`daq0..daq3`) plus overlay variants.
icaruscode/TPC/ICARUSWireCell/detsimmodules_wirecell_ICARUS.fcl	Switches the base `icarus_simwire_wirecell` config to the YZ-sim Jsonnet.
fcl/standard/standard_mc_all_stage0_icarus.fcl	Changes the standard stage0 include to the YZ-specific stage0 config.
fcl/reco/Stage0/mc/stage0_run2_wcdnn_icarus_mc.fcl	Changes the stage0+DNN entry point to include the YZ-specific stage0 config.
fcl/reco/Stage0/mc/stage0_run2_wc_yz_icarus_mc.fcl	New stage0 config variant that overrides decoder fragment labels for per-TPC (`daq0..daq3`) inputs.
fcl/reco/Stage0/mc/stage0_run2_wc_raw_icarus_mc.fcl	Switches the “keep raw digits” stage0 config to include the YZ-specific stage0 config.
fcl/reco/Stage0/mc/stage0_run2_wc_icarus_mc_notriggersim.fcl	Switches the “no trigger sim” stage0 config to include the YZ-specific stage0 config.
fcl/g4/mergesimsources_icarus.fcl	Updates YZ SimChannel merge input labels to use `daq0..daq3:simpleSC*`.
fcl/detsim/detsim_2d_icarus_refactored.fcl	Includes the new per-TPC WireCell detsim module definitions (non-YZ still uses single `daq`).
fcl/detsim/detsim_2d_icarus_refactored_yzsim.fcl	Rewires YZ detsim to run `daq0..daq3` instead of a single `daq` producer.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_overlay.fcl	Updates overlay YZ detsim overrides to target `daq0..daq3` producers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+local g = import 'pgraph.jsonnet';
+local f = import 'pgrapher/common/funcs.jsonnet';
+local wc = import 'wirecell.jsonnet';
+
+local io = import 'pgrapher/common/fileio.jsonnet';
+local tools_maker = import 'pgrapher/experiment/icarus/icarus_tools.jsonnet';
+local base = import 'pgrapher/experiment/icarus/simparams.jsonnet';
+


+local tpc_idx = std.parseInt(std.extVar('tpc_idx'));
+local apa_lo = tpc_idx * 2;     // 2 anodes per TPC
+local local_iota = std.range(0, 89);  // 90 per-anode-plane-response branches
+local volname = ["EE", "EW", "WE", "WW"];
+


+// product instances match the single-module reference run.
+// ---------------------------------------------------------------------------
+local overlay_drifter = std.extVar("overlay_drifter");
+local localeLiftime = [std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us];


+      structs: {
+         DL: 4e-9
+         DT: 8.8e-9
+         coh_noise_scale: 1
+         gain0: 1.705212e1


+      structs: {
+         DL: 4e-9
+         DT: 8.8e-9
+         coh_noise_scale: 1
+         gain0: 1.705212e1


@@ -1 +1 @@
-#include "stage0_run2_icarus_mc.fcl"
+#include "stage0_run2_wc_yz_icarus_mc.fcl"


@@ -1,3 +1,3 @@
-#include "stage0_run2_wc_icarus_mc.fcl"
+#include "stage0_run2_wc_yz_icarus_mc.fcl"


 ## This fhicl file takes the standard stage 0 wc mc file and keeps the raw digits
 ##
-#include "stage0_run2_wc_icarus_mc.fcl"
+#include "stage0_run2_wc_yz_icarus_mc.fcl"


cerati · 2026-06-25T15:03:15Z

@@ -1,4 +1,4 @@
-#include "stage0_run2_wc_icarus_mc.fcl"
+#include "stage0_run2_wc_yz_icarus_mc.fcl"


here (and in other places), I think Copilot is pointing out that effectively you are changing the behavior of the fcl file to be "non-YZ" to become "YZ". Is this correct?

        plugins: ["WireCellPgraph", "WireCellGen","WireCellSio","WireCellRoot","WireCellLarsoft", "WireCellHio"]
        // needs to be found via your WIRECELL_PATH 
-        configs: ["pgrapher/experiment/icarus/wcls-multitpc-sim-drift-simchannel-refactored.jsonnet"]
+        configs: ["pgrapher/experiment/icarus/wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet"]


SFBayLaser · 2026-06-02T18:59:34Z

The above PR is compatible with larsoft v10_20_09 and later. It also requires icarus_data v10_20_03.

cerati

I do not fully understand the details of the jsonnet options, but this generally does what expected, i.e. splitting the detsim per tpc and updating the configuration to reflect wire cell changes. I approve pending some minor comments i left.

cerati · 2026-06-25T15:03:15Z

@@ -1,4 +1,4 @@
-#include "stage0_run2_wc_icarus_mc.fcl"
+#include "stage0_run2_wc_yz_icarus_mc.fcl"


here (and in other places), I think Copilot is pointing out that effectively you are changing the behavior of the fcl file to be "non-YZ" to become "YZ". Is this correct?

cerati · 2026-06-25T15:04:51Z

+    // tools.anodes so that callers that narrow tools.anodes to a per-TPC
+    // subset (e.g. the per-module wcls-per-tpc...jsonnet) only build the
+    // relevant slice instead of indexing past the end.  With the full 8
+    // anodes this evaluates to std.range(0, 359), matching the legacy form.


what about defining a local variable as nanodes*45-1 and the using it below?

cerati · 2026-06-25T15:05:24Z

+//      g4_ref_time: -1500 * wc.us, // G4RefTime from detectorclocks_icarus.fcl
+//      use_energy: true,
+//    },
+//  },nin=1, nout=1, uses=tools.anodes);


should this block be deleted or is there a reason to keep it around commented out?

cerati · 2026-06-25T15:06:35Z

+local resolve_uses_unique(roots) =
+  std.foldl(function(s, n) visit(n, s), roots, { seen: {}, result: [] }).result;
+
+resolve_uses_unique(graph.uses) + [app]


not clear to me what this code does, maybe add some documentation? (copilot please help...)

this is referring to lines 369-404

cerati · 2026-06-25T15:07:20Z

 		     name: 'setdrifters%d' %n, 
                     data: {
 		              drifter: wc.tn(drifters[n])
+			      #drifter: "Drifter"


cerati · 2026-06-25T15:08:38Z

+//      g4_ref_time: -1500 * wc.us, // G4RefTime from detectorclocks_icarus.fcl
+//      use_energy: true,
+//    },
+//  },nin=1, nout=1, uses=tools.anodes);


same as above, shall this be removed?

cerati · 2026-06-25T15:10:09Z

+// Pnodes (g.pipeline/g.pnode/g.intern wrappers) are scaffolding whose names
+// can collide (fandrifter creates fanout/fanin/intern all named 'fandrifter'),
+// so we always traverse them and never dedup them.  The pgraph is acyclic
+// by design, so unconditional traversal terminates.


maybe the above documentation is what needs to be added to the other jsonnet as well?

SFBayLaser added 11 commits May 21, 2026 11:58

As per Haiwang Yu, we are moving to a scheme where the YZ simulation …

3bf2452

…is broken into four "daq" modules, one per TPC. These files make that definition, for standard YZ sim and also for overlay

Top level fhicl files for running the regular YZ detsim and the overlay

c5f99ed

Updates some functions needed for new yz scheme

b99e248

Remapping the SimChannel inputs for merging

378210b

The input scheme for stage 0 changes with the move to four modules at…

cb8e876

… detsim

I am not sure if these are necessary?

fbe9e2f

Updates aimed at getting the non-yz simulation to run. Note that the …

fffb353

…yz sim uses 4 modules (daq0-daq3) while the standard sim continues to use only one (daq) so stage 0 will need to recognize this

Oops, this part of previous commit

1d03122

We need a specific stage 0 file when the yz simulation runs which can…

a1fd210

… inherit from the standard one

Returning this to a single det sim producer

5b53499

The "wc" fhicl files now inherit from the "yz sim" file

0c4b9e6

SFBayLaser requested a review from Copilot June 2, 2026 18:51

Copilot started reviewing on behalf of SFBayLaser June 2, 2026 18:52 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

jas1005 force-pushed the develop branch from 6b894cd to 716718e Compare June 24, 2026 21:43

cerati self-requested a review June 25, 2026 14:45

cerati approved these changes Jun 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update YZ sim to address large memory usage#905

Update YZ sim to address large memory usage#905
SFBayLaser wants to merge 11 commits into
developfrom
feature/usher_fourmoduleyzsim

SFBayLaser commented Jun 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

cerati Jun 25, 2026

Uh oh!

SFBayLaser commented Jun 2, 2026

Uh oh!

cerati left a comment

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

cerati Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1 +1 @@
		#include "stage0_run2_icarus_mc.fcl"
		#include "stage0_run2_wc_yz_icarus_mc.fcl"

		@@ -1,3 +1,3 @@
		#include "stage0_run2_wc_icarus_mc.fcl"
		#include "stage0_run2_wc_yz_icarus_mc.fcl"

		@@ -1,4 +1,4 @@
		#include "stage0_run2_wc_icarus_mc.fcl"
		#include "stage0_run2_wc_yz_icarus_mc.fcl"

Uh oh!

Conversation

SFBayLaser commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SFBayLaser commented Jun 2, 2026

Uh oh!

cerati left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SFBayLaser commented Jun 2, 2026 •

edited

Loading