Shift Notes

[tanishqaggarwal]

Here's a place to place records of notable spacecraft activity (and potentially our overall response strategy) towards them.

[tanishqaggarwal]

Shift: Jan 17 Night

Rotator: Tanishq Aggarwal

Spacecraft Activity:

• Eclipse (zero cursun) and low battery (~6600 mV)
• Eventually bounced back to 7100 mV
• Got bootcause: 2 (I2C WDT reset) at about 2:30 AM EST 1/18

Commanding

• Reduce radio wait cycles and transceive cycles (PowerSave3)

Concerns:

• Continue to watch behavior of PowerSave3 on spacecraft
• Is bootcount rate correlated with Gomspace voltage? Need to check what we designated as crit power threshold on flight units

[fatimayousuf]

Shift: January 18 Morning

Rotator: Fatima Yousuf

Spacecraft Activity:

• Battery voltage oscillating between 6700 and 7100
• No change in boot count rate
• Got another downlink from follower

Commanding

• Sent PowerSave3 to leader again

[shihaocao]

Shift: Jan 18 Evening

Rotator: Shihao Cao

Spacecraft Activity:

Nothing new. Leader has had no comms for the last 8 hours. The previous longest recorded gap is 12 hours.

Concerns:

Continue to watch the behavior of PowerSave3. It's looking pretty good for the spacecraft, but lets confirm through testing.
Since we have on call right now, let's continue to not have more than one command in the queue at a time. Requeue PowerSave3 whenever we get confirmation it's doing ok. Otherwise, discuss in #mission-ops

[stewartaslan]

Shift: Jan 18/19 Night

Rotator: Stewart Aslan

Spacecraft Activity:

• Received one downlink from Leader at 6:56pm pacific time. Battery voltage of 6.695V is close to the lowest ever measured, which was 6.685V on Jan 18.
• A PowerSave3 command was in the queue at that time, so that command should have been applied to the spacecraft.
• Queued up another PowerSave3 command.
• Queued a second PowerSave3 command to account for the three hour gap in coverage 1-4am pacific time. There are two commands in the queue at the end of this shift.

Concerns:

• Battery voltage is low
• Searched for a really long time through google drive, github, my laptop, my brain.... can't find what the gomspace i2c wdt timeout is set to. Having this info would be very helpful for diagnosing reboots.

[govindchari]

Shift: Jan 19 Morning

Rotator: Govind Chari

Spacecraft Activity:

• Received two downlinks from Leader in the span of one hour. One at 9:00 am EST and one at 9:53am EST

• Battery voltage went up from yesterday's all time low of 6.685V to 6.995V

• Queued PowerSaving3

[shihaocao]

Shift: Jan 19 Noon

Rotator: Shihao Cao

Spacecraft Activity:

Voltage restabilizing, back up to 7.0V

Queued PowerSaving3 with Claudia

[stewartaslan]

Shift: Jan 19/20 Night

Rotator: Stewart Aslan

Spacecraft Activity:

• Received 5 downlinks from Leader (MOMSN 51-55), two sets of which were on the same boot
• Two bootcause data points both have a value of 2 (I2C WDT reset), confirming that this is the cause of frequent reboots
• Queued five PowerSave3 commands (MTMSN 27-31), two commands in the queue at end of shift to account for three hour gap in shift coverage.
• Based on the ignore_sun_vectors flag and GOMSpace bootcause telemetry, MTMSN 26 and 27 were received, however ignore_sun_vectors did not flip from true to false in MOMSN 55, indicating that MTMSN 28 was not received or not applied. Worth keeping an eye on this for future commands.

Concerns:

• Battery voltage appears to be in a somewhat stable oscillation between approximately 6750mV and 7050mV, which is quite low
• Battery temperature is definitely trending downwards and is not clearly stabilizing, however the GOMSpace datasheet indicates an operating temperature range of -40C to 85C, so we should be ok for a while.
• Not all commands are being received, we should make sure to keep track of which commands are received and be diligent about switching the ignore_sun_vectors flag.

[govindchari]

Shift: Jan 20 Morning

Rotator: Govind Chari

Spacecraft Activity:
Have not received a downlink in 11 hours

[fatimayousuf]

Shift: Jan 20 Afternoon/Evening

Rotator: Fatima Yousuf

Spacecraft Activity:

• Received three downlinks from leader, none from follower
• Sent an uplink to leader that shifted flow 36 to 2nd in the flow order so that we can get data on gomspace.wdt_i2c_time_left (MTMSN 32)

[kylekrol]

Shift: Jan 20/21 Night

Rotator: Kyle

Spacecraft Activity:

• Received just one message from the leader and continued queueing the power saving commands (MTMSN 33) in addition to the flow changes Fatima implemented above. Not much else to note.
• Batter reported to be at 7.06 V for the leader.

[samuelqian01]

Shift: Jan 21 Morning

Rotator: Samuel Qian

Spacecraft Activity:

• Received three downlinks (7:28 AM, 9:59 AM, 10:31 AM)
• Queued PowerSaving3 with Shihao
• Battery voltage dropped from 7.180V to 6.785V after briefly fluctuating between ~7 AM est to ~10 AM est

[jer296]

Shift: Jan 21-22 Night

Rotator: Josh

Spacecraft Activity:

• Received one downlink from Follower 10:06PM EST 01/21
• Follower battery voltage reported as 6.720V
• Queued PowerSaving3 with ~24-hour max_wait_time (500k)
• Received one downlink from Leader 12:59AM EST 01/22
• Leader battery voltage reported as 7.015V
• Queued PowerSaving3 with nominal max_wait_time

[afoarce]

Shift: Jan 21 Afternoon

Rotator: Andreea

Spacecraft Activity:

• Received two downlinks from Leader (2:12PM, 6:47PM)
• Battery voltage fairly stable, went from 6.785 to 6.780
• Queued PowerSaving3 each time with flow reordering

[millie-schwartz]

Shift: 21-22 night (thru 1 am, finished by tanishq)

Rotator: Millie

Spacecraft Activity:

• Received 3 packets from Follower and 1 from leader between 10:50-11:20
• Follower vbatt decreased from previous packet, once again faulted
• Queued PS3 for both with ignore_sun_vectors flipped from previous uplink for both

[tanishqaggarwal]

Shift: Jan 22, 1 AM - 4 AM EST

Rotator: Tanishq Aggarwal

Spacecraft Activity:

• No telemetry in these three hours.

Commanding:

• Powersave3 re-queued for leader as a wakeup contingency.

[angelaloh09]

Shift: Jan 23 Morning

Rotator: Angela

Spacecraft Activity:

• Received just 2 messages from the leader
• Battery Reported: 6.71 V for leader.

Commanding:

• Shihao/Fatima queued power saving + ADCS reboot command.

*Note from Jan 22 morning (forgot to document): did not receive any packets. New longest recorded gap between receiving leader downlinks: 14 hrs

[samuelqian01]

Shift: 1/22 Afternoon

Rotator: Samuel
Spacecraft Activity:

• Received 1 message from leader

• Battery reported to be ~7 V for leader

• Qued PowerSaving3 after receiving the packet

[millie-schwartz]

Shift: 1/23 Afternoon

Rotator: Millie

Spacecraft Activity: No downlinks received

[claupiet]

Shift: 1/24 Afternoon

Rotator: Claudia
Spacecraft Activity:

• Received 1 message from leader
• Battery reported: 6.96 V for leader

Commanding:

• Queued PowerSaving3 with ADCS Power Cycle Combo

[tanishqaggarwal]

Shift: 1/24 Night

Rotator: Tanishq
Spacecraft Activity:

• No packets received

Commanding:

• None. Commands queued by last rotator.

[stewartaslan]

Shift: 1/25-1/26 night

Rotator: Stewart

Spacecraft activity:

• No packets received, so did not get a chance to test the new automatic commanding functionality.

[govindchari]

Shift: 1/26 morning

Rotator: Govind

Spacecraft activity:

Have not received downlink in 14 hours

[angelaloh09]

Shift: 1/29 11:00am check-in

Rotator: Angela

Spacecraft Activity:

• No packets received in the past 9 hours.

[claupiet]

Shift: 1/31 10:00pm check-in

Rotator: Claudia

Spacecraft Activity:

• Downlink from leader and from follower
• Leader at 6.83 V
• Follower at 6.89 V

Commanding:

• None by me, queued up for both by others (MTMSN 7 for follower, 75 for leader)

[shihaocao]

Shift: 02/01 10:00 pm check-in

Rotator: Shihao

Spacecraft Activity:
MOMSN: 120 MTMSN: 0

{
"adcs.state": "1",
"adcs_monitor.functional_fault.base": "true",
"adcs_monitor.havt_device0": "false",
"adcs_monitor.havt_device1": "false",
"adcs_monitor.havt_device10": "false",
"adcs_monitor.havt_device11": "false",
"adcs_monitor.havt_device12": "false",
"adcs_monitor.havt_device13": "false",
"adcs_monitor.havt_device14": "false",
"adcs_monitor.havt_device15": "false",
"adcs_monitor.havt_device16": "false",
"adcs_monitor.havt_device17": "false",
"adcs_monitor.havt_device2": "false",
"adcs_monitor.havt_device3": "false",
"adcs_monitor.havt_device4": "false",
"adcs_monitor.havt_device5": "false",
"adcs_monitor.havt_device6": "false",
"adcs_monitor.havt_device7": "false",
"adcs_monitor.havt_device8": "false",
"adcs_monitor.havt_device9": "false",
"adcs_monitor.wheel1_fault.base": "true",
"adcs_monitor.wheel2_fault.base": "true",
"adcs_monitor.wheel3_fault.base": "true",
"adcs_monitor.wheel_pot_fault.base": "true",
"attitude_estimator.L_body": "0.000000,0.000000,0.000000,",
"attitude_estimator.fault.base": "true",
"attitude_estimator.ignore_sun_vectors": "false",
"attitude_estimator.q_body_eci": " -nan,-1.000000,-1.000000,-1.000000,",
"attitude_estimator.valid": "false",
"cursed": "false",
"gomspace.cursun": "996",
"gomspace.cursys": "624",
"gomspace.get_hk.base": "true",
"gomspace.low_batt.base": "true",
"gomspace.temp.output1": "18",
"gomspace.temp.output2": "17",
"gomspace.temp.output3": "18",
"gomspace.temp.output4": "18",
"gomspace.vbatt": "7045",
"orbit.pos": "2475471.759935,5773023.471555,-2806497.619627,",
"orbit.valid": "false",
"orbit.vel": "-5848.085552,344.776099,-4455.404002,",
"pan.bootcount": "163",
"pan.cycle_no": "21128",
"pan.deployed": "true",
"pan.deployment.elapsed": "106",
"pan.sat_designation": "0",
"pan.state": "10",
"piksi.state": "3",
"piksi_fh.dead.base": "false",
"prop.overpressured.base": "false",
"prop.pressurize_fail.base": "false",
"prop.state": "0",
"prop.tank1_temp_high.base": "false",
"prop.tank2_temp_high.base": "false",
"qfh.state": "0",
"radio.state": "3",
"rel_orbit.state": "0",
"time.gps": "2000,282467310,400151",
"time.valid": "true",
"time.downlink_received": "2022-02-02T06:27:46Z"
}

MOMSN: 119 MTMSN: 87

{
"adcs.state": "1",
"adcs_monitor.functional_fault.base": "true",
"adcs_monitor.havt_device0": "true",
"adcs_monitor.havt_device1": "true",
"adcs_monitor.havt_device10": "true",
"adcs_monitor.havt_device11": "true",
"adcs_monitor.havt_device12": "true",
"adcs_monitor.havt_device13": "true",
"adcs_monitor.havt_device14": "true",
"adcs_monitor.havt_device15": "true",
"adcs_monitor.havt_device16": "true",
"adcs_monitor.havt_device17": "true",
"adcs_monitor.havt_device2": "true",
"adcs_monitor.havt_device3": "true",
"adcs_monitor.havt_device4": "true",
"adcs_monitor.havt_device5": "true",
"adcs_monitor.havt_device6": "true",
"adcs_monitor.havt_device7": "true",
"adcs_monitor.havt_device8": "true",
"adcs_monitor.havt_device9": "true",
"adcs_monitor.wheel1_fault.base": "true",
"adcs_monitor.wheel2_fault.base": "true",
"adcs_monitor.wheel3_fault.base": "true",
"adcs_monitor.wheel_pot_fault.base": "true",
"attitude_estimator.L_body": "0.000000,0.000000,0.000000,",
"attitude_estimator.fault.base": "true",
"attitude_estimator.ignore_sun_vectors": "false",
"attitude_estimator.q_body_eci": " -nan,-1.000000,-1.000000,-1.000000,",
"attitude_estimator.valid": "false",
"cursed": "false",
"gomspace.cursun": "978",
"gomspace.cursys": "564",
"gomspace.get_hk.base": "false",
"gomspace.low_batt.base": "true",
"gomspace.temp.output1": "18",
"gomspace.temp.output2": "17",
"gomspace.temp.output3": "18",
"gomspace.temp.output4": "18",
"gomspace.vbatt": "7060",
"orbit.pos": "2475471.759935,5773023.471555,-2806497.619627,",
"orbit.valid": "false",
"orbit.vel": "-5848.085552,344.776099,-4455.404002,",
"pan.bootcount": "163",
"pan.cycle_no": "20497",
"pan.deployed": "true",
"pan.deployment.elapsed": "106",
"pan.sat_designation": "0",
"pan.state": "10",
"piksi.state": "9",
"piksi_fh.dead.base": "false",
"prop.overpressured.base": "false",
"prop.pressurize_fail.base": "false",
"prop.state": "0",
"prop.tank1_temp_high.base": "false",
"prop.tank2_temp_high.base": "false",
"qfh.state": "0",
"radio.state": "3",
"rel_orbit.state": "0",
"time.gps": "2000,282360040,400151",
"time.valid": "true",
"time.downlink_received": "2022-02-02T06:26:32Z"
}

It looks like the ADCS reboot, brought all HAVT devices down, but this could be confounded by get_hk.base being true. All of I2C probably went down. Inconclusive data.

On 02/01 also got many other second frames without rebooting, did not seem to make ADCS faults go away.

[angelaloh09]

Shift: 2/5 11:00AM Morning

Rotator: Angela

Spacecraft Activity:

• Received 1 message from leader at 12:00am. Battery reported to be ~7 V for leader.
• TLM seems nominal

[millie-schwartz]

Shift: 2/6 Night
Rotator: Millie
Spacecraft Activity:

• Received 2 packages from leader, 1 from follower
• Leader Vbatt @ abt 6770
• Follower Vbatt a little under 6900 (nice)
• Things look normal

[claupiet]

Shift: 2/7 Night
Rotator: Claudia
Spacecraft Activity:

• 6 packets from leader since noon EST!
• Leader battery at 7.155 V
• Nothing unusual to report

[afoarce]

Shift 2/11 Night
Rotator: Andreea
Spacecraft Activity:

• No Packets received 2/11
• 2 Follower packets received 2/10
  • Battery 6.795
• 6 Leader packets received 2/10
  • Battery 6.830

[millie-schwartz]

Shift 2/13 Night
Rotator: Millie
Spacecraft Activity:

• Multiple MCT outages (including rn)
• 2 packages from leader around 8
• Waiting for one more comm from leader to reinstate command bot