Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JP-3737: Handle micrometeorite flashes #308

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

JP-3737: Handle micrometeorite flashes #308

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
22878e6
Add flashfrac
drlaw1558 Oct 1, 2024
495655d
Merge branch 'main' into jp-3737
drlaw1558 Oct 16, 2024
66f0bb6
Code style cleanup
drlaw1558 Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 92 additions & 80 deletions src/stcal/jump/jump.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
min_diffs_single_pass=10,
mask_persist_grps_next_int=True,
persist_grps_flagged=25,
mmflashfrac=1.0,
):
"""
This is the high-level controlling routine for the jump detection process.
Expand Down Expand Up @@ -221,6 +222,11 @@
min_diffs_single_pass : int
The minimum number of groups to switch to flagging all outliers in a single pass.

mmflashfrac: float
Fraction of the array in a given group that has to have a jump detected
in order to flag the entire array as a jump. This is designed to deal with
sporadic micrometeorite flashes.

Returns
-------
gdq : int, 4D array
Expand Down Expand Up @@ -298,46 +304,7 @@
dqflags['DO_NOT_USE']
gdq[gdq == np.bitwise_or(dqflags['SATURATED'], dqflags['JUMP_DET'])] = \
dqflags['SATURATED']
# This is the flag that controls the flagging of snowballs.
if expand_large_events:
gdq, total_snowballs = flag_large_events(
gdq,
jump_flag,
sat_flag,
min_sat_area=min_sat_area,
min_jump_area=min_jump_area,
expand_factor=expand_factor,
sat_required_snowball=sat_required_snowball,
min_sat_radius_extend=min_sat_radius_extend,
edge_size=edge_size,
sat_expand=sat_expand,
max_extended_radius=max_extended_radius,
mask_persist_grps_next_int=mask_persist_grps_next_int,
persist_grps_flagged=persist_grps_flagged,
)
log.info("Total snowballs = %i", total_snowballs)
number_extended_events = total_snowballs
if find_showers:
gdq, num_showers = find_faint_extended(
data,
gdq,
pdq,
readnoise_2d,
frames_per_group,
minimum_sigclip_groups,
dqflags,
snr_threshold=extend_snr_threshold,
min_shower_area=extend_min_area,
inner=extend_inner_radius,
outer=extend_outer_radius,
sat_flag=sat_flag,
jump_flag=jump_flag,
ellipse_expand=extend_ellipse_expand_ratio,
num_grps_masked=grps_masked_after_shower,
max_extended_radius=max_extended_radius,
)
log.info("Total showers= %i", num_showers)
number_extended_events = num_showers

else:
yinc = int(n_rows // n_slices)
slices = []
Expand Down Expand Up @@ -463,46 +430,54 @@
gdq[gdq == np.bitwise_or(dqflags['SATURATED'], dqflags['JUMP_DET'])] = \
dqflags['SATURATED']

# This is the flag that controls the flagging of snowballs.
if expand_large_events:
gdq, total_snowballs = flag_large_events(
gdq,
jump_flag,
sat_flag,
min_sat_area=min_sat_area,
min_jump_area=min_jump_area,
expand_factor=expand_factor,
sat_required_snowball=sat_required_snowball,
min_sat_radius_extend=min_sat_radius_extend,
edge_size=edge_size,
sat_expand=sat_expand,
max_extended_radius=max_extended_radius,
mask_persist_grps_next_int=mask_persist_grps_next_int,
persist_grps_flagged=persist_grps_flagged,
)
log.info("Total snowballs = %i", total_snowballs)
number_extended_events = total_snowballs
if find_showers:
gdq, num_showers = find_faint_extended(
data,
gdq,
pdq,
readnoise_2d,
frames_per_group,
minimum_sigclip_groups,
dqflags,
snr_threshold=extend_snr_threshold,
min_shower_area=extend_min_area,
inner=extend_inner_radius,
outer=extend_outer_radius,
sat_flag=sat_flag,
jump_flag=jump_flag,
ellipse_expand=extend_ellipse_expand_ratio,
num_grps_masked=grps_masked_after_shower,
max_extended_radius=max_extended_radius,
)
log.info("Total showers= %i", num_showers)
number_extended_events = num_showers
# Look for snowballs in near-IR data
if expand_large_events:
gdq, total_snowballs = flag_large_events(

Check warning on line 435 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L435 

Added line #L435 was not covered by tests
gdq,
jump_flag,
sat_flag,
min_sat_area=min_sat_area,
min_jump_area=min_jump_area,
expand_factor=expand_factor,
sat_required_snowball=sat_required_snowball,
min_sat_radius_extend=min_sat_radius_extend,
edge_size=edge_size,
sat_expand=sat_expand,
max_extended_radius=max_extended_radius,
mask_persist_grps_next_int=mask_persist_grps_next_int,
persist_grps_flagged=persist_grps_flagged,
)
log.info("Total snowballs = %i", total_snowballs)
number_extended_events = total_snowballs

Check warning on line 451 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L450-L451 

Added lines #L450 - L451 were not covered by tests

# Look for showers in mid-IR data
if find_showers:
gdq, num_showers = find_faint_extended(

Check warning on line 455 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L455 

Added line #L455 was not covered by tests
data,
gdq,
pdq,
readnoise_2d,
frames_per_group,
minimum_sigclip_groups,
dqflags,
snr_threshold=extend_snr_threshold,
min_shower_area=extend_min_area,
inner=extend_inner_radius,
outer=extend_outer_radius,
sat_flag=sat_flag,
jump_flag=jump_flag,
ellipse_expand=extend_ellipse_expand_ratio,
num_grps_masked=grps_masked_after_shower,
max_extended_radius=max_extended_radius,
)
log.info("Total showers= %i", num_showers)
number_extended_events = num_showers

Check warning on line 474 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L473-L474 

Added lines #L473 - L474 were not covered by tests

# Look for micrometeorite flashes if the triggering
# threshold is less than the entire array
if (mmflashfrac < 1):
gdq = find_micrometeorite_flashes(gdq, jump_flag, mmflashfrac)

Check warning on line 479 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L479 

Added line #L479 was not covered by tests

elapsed = time.time() - start
log.info("Total elapsed time = %g sec", elapsed)

Expand All @@ -514,6 +489,43 @@
# Return the updated data quality arrays
return gdq, pdq, total_primary_crs, number_extended_events, stddev

# Find micrometeorite flashes
def find_micrometeorite_flashes(gdq, jump_flag, mmflashfrac):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing a docstring.

Also, start the function with nints, ngroups, nrows, ncols = gdq.shape, then use these more descriptive variable names, rather than shape[k].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point- done

"""
This routine looks for micrometeorite flashes that light up the entire array in a small number
of groups. It does this by looking for cases where greater than mmflash_pct percent
of the array is flagged as 'JUMP_DET', and flags the entire array as JUMP_DET in such cases.

Parameters
----------
gdq : int, 4D array
Group dq array
jump_flag : int
DQ flag for jump detection.
mmflashfrac : float
Fraction of the array that can be flagged as a jump before the entire array
will be flagged.
Returns
-------
gdq : int, 4D array
Updated group dq array

"""
log.info("Looking for micrometeorite flashes")

Check warning on line 514 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L514 

Added line #L514 was not covered by tests

nints, ngroups, nrows, ncols = gdq.shape

Check warning on line 516 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L516 

Added line #L516 was not covered by tests

npix = nrows * ncols # Number of pixels in array

Check warning on line 518 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L518 

Added line #L518 was not covered by tests
# Loop over integrations
for ii in range(0, nints):

Check warning on line 520 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L520 

Added line #L520 was not covered by tests
# Loop over groups
for jj in range(0, ngroups):
indx = np.where(gdq[ii, jj, :, :] & jump_flag != 0)
fraction = float(len(indx[0])) / npix
if (fraction > mmflashfrac):
gdq[ii, jj, :, :] = gdq[ii, jj, :, :] | jump_flag
log.info(f"Detected a micrometeorite flash in integ = {ii}, group = {jj}")
return gdq

Check warning on line 528 in src/stcal/jump/jump.py

View check run for this annotation

Codecov / codecov/patch

src/stcal/jump/jump.py#L522-L528 

Added lines #L522 - L528 were not covered by tests

def flag_large_events(
gdq,
Expand Down
Loading