All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Public Member Functions | Static Public Member Functions | Public Attributes | Static Private Attributes | List of all members
python.extractor_dict.MetaData Class Reference
Inheritance diagram for python.extractor_dict.MetaData:
python.extractor_dict.expMetaData

Public Member Functions

def __init__
 
def extract_metadata_to_pipe
 
def get_job
 

Static Public Member Functions

def wait_for_subprocess
 
def mdart_gen
 
def md_handle_application
 

Public Attributes

 inputfile
 

Static Private Attributes

 __metaclass__ = abc.ABCMeta
 

Detailed Description

Base class to hold / interpret general metadata

Definition at line 33 of file larbatch/python/extractor_dict.py.

Constructor & Destructor Documentation

def python.extractor_dict.MetaData.__init__ (   self,
  inputfile 
)

Definition at line 38 of file larbatch/python/extractor_dict.py.

38 
39  def __init__(self, inputfile):
40  self.inputfile = inputfile
python.extractor_dict.MetaData.inputfile
inputfile
Definition: larbatch/python/extractor_dict.py:39
python.extractor_dict.MetaData.__init__
def __init__
Definition: larbatch/python/extractor_dict.py:38

Member Function Documentation

def python.extractor_dict.MetaData.extract_metadata_to_pipe (   self) 
Extract metadata from inputfile into a pipe for further processing.

Definition at line 41 of file larbatch/python/extractor_dict.py.

41 
42  def extract_metadata_to_pipe(self):
43  """Extract metadata from inputfile into a pipe for further processing."""
44  local = project_utilities.path_to_local(self.inputfile)
45  if len(local) > 0:
46  proc = Popen(["sam_metadata_dumper", local], stdout=PIPE,
47  stderr=PIPE)
48  else:
49  url = project_utilities.path_to_url(inputfile)
50  proc = Popen(["sam_metadata_dumper", url], stdout=PIPE,
51  stderr=PIPE)
52  if len(local) > 0 and local != self.inputfile:
53  os.remove(local)
54  return proc
python.extractor_dict.MetaData.inputfile
inputfile
Definition: larbatch/python/extractor_dict.py:39
python.extractor_dict.MetaData.extract_metadata_to_pipe
def extract_metadata_to_pipe
Definition: larbatch/python/extractor_dict.py:41
def python.extractor_dict.MetaData.get_job (   self,
  proc 
) 
Run the proc in a 60-sec timeout queue, return stdout, stderr

Definition at line 55 of file larbatch/python/extractor_dict.py.

55 
56  def get_job(self, proc):
57  """Run the proc in a 60-sec timeout queue, return stdout, stderr"""
58  q = queue.Queue()
59  thread = threading.Thread(target=self.wait_for_subprocess, args=[proc, q])
60  thread.start()
61  thread.join(timeout=60)
62  if thread.is_alive():
63  print('Terminating subprocess because of timeout.')
64  proc.terminate()
65  thread.join()
66  rc = q.get()
67  jobout = convert_str(q.get())
68  joberr = convert_str(q.get())
69  if rc != 0:
70  raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
71  return jobout, joberr
format
static std::string format(PyObject *obj, unsigned int pos, unsigned int indent, unsigned int maxlen, unsigned int depth)
Definition: fclmodule.cxx:374
python.extractor_dict.MetaData.wait_for_subprocess
def wait_for_subprocess
Definition: larbatch/python/extractor_dict.py:73
print
do one_file $F done echo for F in find $TOP name CMakeLists txt print
Definition: update_sources.sh:94
python.larbatch_utilities.convert_str
def convert_str
Definition: larbatch_utilities.py:1171
python.extractor_dict.MetaData.get_job
def get_job
Definition: larbatch/python/extractor_dict.py:55
def python.extractor_dict.MetaData.md_handle_application (   md)
static 
If there's no application key in md dict, create the key with a blank dictionary.
Then return md['application'], along with mdval

Definition at line 95 of file larbatch/python/extractor_dict.py.

95 
96  def md_handle_application(md):
97  """If there's no application key in md dict, create the key with a blank dictionary.
98  Then return md['application'], along with mdval"""
99  if 'application' not in md:
100  md['application'] = {}
101  return md['application']
102 
python.extractor_dict.MetaData.md_handle_application
def md_handle_application
Definition: larbatch/python/extractor_dict.py:95
def python.extractor_dict.MetaData.mdart_gen (   jobtuple)
static 
Take Jobout and Joberr (in jobtuple) and return mdart object from that

Definition at line 84 of file larbatch/python/extractor_dict.py.

84 
85  def mdart_gen(jobtuple):
86  """Take Jobout and Joberr (in jobtuple) and return mdart object from that"""
87  mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split('\n') if line[-3:-1] != ' ,')
88  mdtop = json.JSONDecoder().decode(mdtext)
89  if len(list(mdtop.keys())) == 0:
90  print('No top-level key in extracted metadata.')
91  sys.exit(1)
92  file_name = list(mdtop.keys())[0]
93  return mdtop[file_name]
print
do one_file $F done echo for F in find $TOP name CMakeLists txt print
Definition: update_sources.sh:94
opdet::factory::join
S join(S const &sep, Coll const &s)
Returns a concatenation of strings in s separated by sep.
Definition: OpRecoFactoryStuff.h:573
util::quantities::concepts::decode
void decode(std::any const &src, Interval< Args...> &iv)
Decodes an interval.
Definition: intervals_fhicl.h:104
python.extractor_dict.MetaData.mdart_gen
def mdart_gen
Definition: larbatch/python/extractor_dict.py:84
list
list
Definition: file_to_url.sh:28
def python.extractor_dict.MetaData.wait_for_subprocess (   jobinfo,
  q 
)
static 
Run jobinfo, put the return code, stdout, and stderr into a queue

Definition at line 73 of file larbatch/python/extractor_dict.py.

73 
74  def wait_for_subprocess(jobinfo, q):
75  """Run jobinfo, put the return code, stdout, and stderr into a queue"""
76  jobout, joberr = jobinfo.communicate()
77  jobout = convert_str(jobout)
78  joberr = convert_str(joberr)
79  rc = jobinfo.poll()
80  for item in (rc, jobout, joberr):
81  q.put(item)
82  return
python.extractor_dict.MetaData.wait_for_subprocess
def wait_for_subprocess
Definition: larbatch/python/extractor_dict.py:73
python.larbatch_utilities.convert_str
def convert_str
Definition: larbatch_utilities.py:1171

Member Data Documentation

python.extractor_dict.MetaData.__metaclass__ = abc.ABCMeta
staticprivate

Definition at line 35 of file larbatch/python/extractor_dict.py.

python.extractor_dict.MetaData.inputfile

Definition at line 39 of file larbatch/python/extractor_dict.py.

The documentation for this class was generated from the following file:
Generated by   doxygen 1.8.5