sbn::DuplicateEventTracker Class Reference

Keeps track of duplicate events in the job. More...

#include <DuplicateEventTracker.h>

Classes
struct	Config
	Service configuration. More...

Public Types
using	Parameters = art::ServiceTable< Config >

Public Member Functions
	DuplicateEventTracker (Parameters const &config, art::ActivityRegistry &reg)
	Constructor: reads the configuration. More...
void	printSummary () const
	Prints a summary of the current duplicates. More...

Protected Member Functions
Callback functions
void	postEventReading (art::Event const &event, art::ScheduleContext)
	Records the event and throws an exception depending on the configuration. More...
void	postOpenFile (std::string const &fileName)
	Records the current file name. More...
void	postEndJob ()
	Prints the summary and throws an exception depending on the configuration. More...

Protected Attributes
bool const	fWarningOnly
	Only warning, no exception. More...
bool const	fSkipSummary
	Do not print the summary of duplicate events. More...
bool const	fExceptionAtEnd
	Throw exception as late as possible. More...
std::string const	fLogCategory
	Message service category tag. More...
sbn::EventRegistry::FileID_t	fCurrentInputFileID = sbn::EventRegistry::NoFileID
	ID in the event registry of the current input file. More...
sbn::EventRegistry	fEventRegistry
	Record of all events and their source. More...
std::atomic< unsigned int >	fNDuplicateEvents { 0U }
	Duplicate event count. More...

Detailed Description

Keeps track of duplicate events in the job.

This art service maintains a record of all the processed events, and it takes action when one is processed more than once. The main purpose of the service is to detect the duplication as soon as possible and in such case to interrupt the job with an error, so that the input can be amended.

The default action is to immediately throw an exception. Alternatively, a warning message can be emitted each time an event is reprocessed, and a summary of all duplication can be emitted at the end of the job.

The service also offers an interface, to receive the current list of events that have been processed so far.

See sbn::EventRegistry for a description of the (large) memory usage of this service.

Configuration parameters

• WarningOnly (flag, default: false): this flag has effect on warning messages and on exception throwing:
  • warning messages: if this flag is set to true, the service will emit a warning on each duplicate event encountered, otherwise it will not emit any warning (summary will still describe the duplicates if not skipped);
  • exception in case of duplicate events: if set to false, an exception will be thrown at the first duplicate event, unless ExceptionAtEnd is set, in which case the exception will be thrown at the end; if the flag is set to true, an exception is thrown at the end if ExceptionAtEnd is set, otherwise no exception is thrown at all
• SkipSummary (flag, default: false): when set to true, it disables the printout of the duplicate events at the end of the job; unless WarningOnly is also set, this summary is going to be one-entry only
• ExceptionAtEnd (flag, default: false): if duplicate events are detected, an exception is thrown at the end of the job instead of at the first duplicate event; this allows to assess all the duplicate events at once; if set, it is recommended that SkipSummary be disabled; note that with this option enabled, if duplicate events are detected an exception is always thrown (at the end of the job), regardless the setting of WarningOnly end of the job)
• LogCategory (text, default: "DuplicateEventTracker"): tag of the output category used by this service to the message facility

Multi-threading notes

This service is designed to work with the concurrent processing of events from the same file. Reading multiple files is not supported (but art 3 doesn't do that anyway).

Definition at line 99 of file DuplicateEventTracker.h.

Member Typedef Documentation

using sbn::DuplicateEventTracker::Parameters = art::ServiceTable<Config>

Definition at line 139 of file DuplicateEventTracker.h.

Constructor & Destructor Documentation

sbn::DuplicateEventTracker::DuplicateEventTracker	(	Parameters const &	config,
		art::ActivityRegistry &	reg
	)

Constructor: reads the configuration.

Definition at line 27 of file DuplicateEventTracker_service.cc.

  : fWarningOnly(config().WarningOnly())
  , fSkipSummary(config().SkipSummary())
  , fExceptionAtEnd(config().ExceptionAtEnd())
  , fLogCategory(config().LogCategory())
{
  
  reg.sPostOpenFile.watch(this, &DuplicateEventTracker::postOpenFile);
  
  reg.sPostSourceEvent.watch(this, &DuplicateEventTracker::postEventReading);
  
  reg.sPostEndJob.watch(this, &DuplicateEventTracker::postEndJob);

} // sbn::DuplicateEventTracker::DuplicateEventTracker()

Member Function Documentation

void sbn::DuplicateEventTracker::postEndJob ( )

protected

Prints the summary and throws an exception depending on the configuration.

Definition at line 140 of file DuplicateEventTracker_service.cc.

                                          {
  
  if (!fSkipSummary) printSummary();
  
  if (fExceptionAtEnd && (fNDuplicateEvents > 0U)) {
    throw cet::exception(fLogCategory)
      << "Found " << fNDuplicateEvents << " duplicate events in the job.\n";
  }
  
} // sbn::DuplicateEventTracker::postEndJob()

void sbn::DuplicateEventTracker::postEventReading ( art::Event const &  event, art::ScheduleContext    )

protected

Records the event and throws an exception depending on the configuration.

Definition at line 101 of file DuplicateEventTracker_service.cc.

{
  
  sbn::EventRegistry::EventRecord_t const eventInfo
    = fEventRegistry.recordEvent(event.id(), fCurrentInputFileID);
  assert(eventInfo.sourceFiles.size() > 0U);
  if (eventInfo.sourceFiles.size() == 1U) return; // all done, no new duplicates
  
  ++fNDuplicateEvents;
  
  if (fWarningOnly) {
    mf::LogWarning(fLogCategory) 
      << "WARNING: event " << event.id() << " encountered "
      << eventInfo.sourceFiles.size() << " times,"
      << "\n  now from '"
      << fEventRegistry.sourceNameOr(eventInfo.sourceFiles.back(), "<?>")
      << "',"
      << "\n  first time from '"
      << fEventRegistry.sourceNameOr(eventInfo.sourceFiles.front(), "<?>")
      << "'"
      ;
    return;
  }
  
  if (fExceptionAtEnd) return;
  
  assert(eventInfo.sourceFiles.size() == 2U);
  throw cet::exception(fLogCategory)
    << "Duplicate event " << event.id() << " encountered"
    << "\n  first in '"
    << fEventRegistry.sourceNameOr(eventInfo.sourceFiles.front(), "<?>") << "'"
    << "\n  and now in '"
    << fEventRegistry.sourceNameOr(eventInfo.sourceFiles.back(), "<?>") << "'\n"
    ;
  
} // sbn::DuplicateEventTracker::postEventReading()

void sbn::DuplicateEventTracker::postOpenFile ( std::string const &  fileName )

protected

Records the current file name.

Definition at line 90 of file DuplicateEventTracker_service.cc.

                                                                     {
  
  // register the current input file; if it exists already,
  // we just get its ID again (but that's troublesome anyway)
  fCurrentInputFileID = fEventRegistry.recordSource(fileName);
  
} // sbn::DuplicateEventTracker::postOpenFile()

void sbn::DuplicateEventTracker::printSummary

(

)

const

Prints a summary of the current duplicates.

Definition at line 45 of file DuplicateEventTracker_service.cc.

                                                  {
  
  mf::LogInfo log(fLogCategory);
  log << "Summary of duplicate events encountered:";
  
  std::set<sbn::EventRegistry::FileID_t> duplicateFileIDs;
  unsigned int nDuplicateEvents = 0U;
  auto eventRecords = fEventRegistry.records();
  std::sort(eventRecords.begin(), eventRecords.end(),
    [](auto const& A, auto const& B){ return A.first < B.first; });
  for (auto const& [ eventID, record ]: eventRecords) {
    if (record.sourceFiles.size() <= 1U) continue;
    ++nDuplicateEvents;
    
    log << "\n  " << eventID << " from " << record.sourceFiles.size()
      << " sources:";
    for (auto const fileID: record.sourceFiles) {
      log << " [" << fileID << "]";
      duplicateFileIDs.insert(fileID);
    }
  } // for
  
  if (!duplicateFileIDs.empty()) {
    log << "\nDuplicate events from " << duplicateFileIDs.size() << " files:";
    for (auto const fileID: duplicateFileIDs) {
      // a bit cheating multithreading: we assume that between the copy of the
      // event records and the source file list no source has been moved or
      // removed; with the current interface, this is assured
      auto const sourceName = fEventRegistry.sourceName(fileID);
      log << "\n  [" << fileID << "]";
      if (sourceName) log << " '" << *sourceName << "'";
      else log << " N/A"; // should not happen though
    } // for sources
  } // if duplicates
  
  if (nDuplicateEvents > 0U) {
    log << "\nCounted " << nDuplicateEvents << " duplicate events from "
      << duplicateFileIDs.size() << " source files.";
  }
  else log << "\nNo duplicate events found.";
  
} // sbn::DuplicateEventTracker::printSummary()

Member Data Documentation

sbn::EventRegistry::FileID_t sbn::DuplicateEventTracker::fCurrentInputFileID = sbn::EventRegistry::NoFileID

protected

ID in the event registry of the current input file.

Definition at line 163 of file DuplicateEventTracker.h.

sbn::EventRegistry sbn::DuplicateEventTracker::fEventRegistry

protected

Record of all events and their source.

Definition at line 165 of file DuplicateEventTracker.h.

bool const sbn::DuplicateEventTracker::fExceptionAtEnd

protected

Throw exception as late as possible.

Definition at line 155 of file DuplicateEventTracker.h.

std::string const sbn::DuplicateEventTracker::fLogCategory

protected

Message service category tag.

Definition at line 157 of file DuplicateEventTracker.h.

std::atomic<unsigned int> sbn::DuplicateEventTracker::fNDuplicateEvents { 0U }

protected

Duplicate event count.

Definition at line 167 of file DuplicateEventTracker.h.

bool const sbn::DuplicateEventTracker::fSkipSummary

protected

Do not print the summary of duplicate events.

Definition at line 154 of file DuplicateEventTracker.h.

bool const sbn::DuplicateEventTracker::fWarningOnly

protected

Only warning, no exception.

Definition at line 153 of file DuplicateEventTracker.h.

---

The documentation for this class was generated from the following files:

• DuplicateEventTracker.h
• DuplicateEventTracker_service.cc