Multiple Packets for One Array

[jmdelahanty]

Hey PowerBroker!

Things are moving along pretty smoothly with our project! I can send trial sets of 45 or smaller no problem and the lab is pretty happy with it. A new challenge has appeared for me, though. The lab would like it to be possible to double the amount of trials to 90 total as a definite ceiling.

I had thought that I was doing this correctly with what you have taught me so far but, after some testing, I've come to realize that while a second packet is getting sent by Python, my Arduino code doesn't seem to collecting the second packet's content. Here's an example of what I mean:

(bruker_control) C:\Users\jdelahanty>python Documents\gitrepos\headfix_control\bruker_control\bruker_control.py

First Half of Trials Sent
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
First Half of Trials Received
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
First Half Trial Array Transfer Successful!
Second Half of Trials Sent
[0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1]
Second Half of Trials Received
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0] # <--- Arduino sends first packet's contents!
Second Half Trial Array Transfer Failure!
Exiting...
Traceback (most recent call last):
  File "Documents\gitrepos\headfix_control\bruker_control\bruker_control.py", line 522, in serial_transfer_trialArray
    sys.exit()
SystemExit
Experiment Over!
Exiting...

Any advice? I had thought it would collect the packet independently but I'm clearly misunderstanding how multiple packets for the same object work. Is this something I can do even or should I create a second array in the Arduino code to store the second half of trials?

Here's some full Python and Arduino code:
Python

#### Trial Array Generation ####
# Import scipy for statistical distributions
import scipy
# Import scipy.stats truncated normal distribution for ITI Array
from scipy.stats import truncnorm
# Import numpy for trial array generation/manipulation and Harvesters
import numpy as np
# Import numpy default_rng
from numpy.random import default_rng

# -----------------------------------------------------------------------------
# Functions
# -----------------------------------------------------------------------------
#### Trial Generation ####
# Random Trials Array Generation
def gen_trial_array(totalNumberOfTrials):
    # Always initialize trial array with 3 reward trials
    trialArray = [1,1,1]
    # Define number of samples needed from generator
    num_samples = totalNumberOfTrials - len(trialArray)
    # Define probability that the animal will receive sucrose 60% of the time
    sucrose_prob = 0.5
    # Initialize random number generator with default_rng
    rng = np.random.default_rng(2)
    # Generate a random trial array with Generator.binomial
    # Use n=1 to pull one sample at a time, p=.6 as probability of sucrose
    # Use num_samples to fill out accurate number of trials
    # Use .tolist() to convert random_trials from np.array to list
    random_trials = rng.binomial(
    n=1, p=sucrose_prob, size=num_samples
    ).tolist()
    # Append the two arrays together
    for i in random_trials:
        trialArray.append(i)

    if len(trialArray) > 45:
        split_array = np.array_split(trialArray, 2)
        first_trialArray = split_array[0].tolist()
        second_trialArray = split_array[1].tolist()
    else:
        first_trialArray = None
        second_trialArray = None

    ## TODO: Write out the trial array into JSON as part of experiment config
    # Return trialArray or halves of trial arrays (if needed)
    return trialArray, first_trialArray, second_trialArray

#### Serial Transfer ####
# Import pySerialTransfer for serial comms with Arduino
from pySerialTransfer import pySerialTransfer as txfer


# Trial Array Transfer
def serial_transfer_trialArray(trialArray, first_trialArray, second_trialArray):
    # Check if two packets are necessary. If 'first_trialArray' is None,
    # the message is small enough to fit in one packet. There's no need to
    # check the second_trialArray as it too will be None by design in this case.
    if first_trialArray == None:
        try:
            # Initialize COM Port for Serial Transfer
            link = txfer.SerialTransfer('COM12', 115200, debug=True)

            # Send the trial array
            # Initialize trialArray_size of 0
            trialArray_size = 0
            # Stuff packet with size of trialArray
            trialArray_size = link.tx_obj(trialArray)
            # Open communication link
            link.open()
            # Send array
            link.send(trialArray_size, packet_id=0)

            print(trialArray)

            while not link.available():
                pass

            # Receive trial array:
            rxtrialArray = link.rx_obj(obj_type=type(trialArray),
            obj_byte_size=trialArray_size, list_format='i')

            print(rxtrialArray)

            if trialArray == rxtrialArray:
                print("Trial Array transfer successful!")

            else:
                link.close()
                print("Trial Array error! Exiting...")
                sys.exit()

            # Close the communication link
            link.close()

        except KeyboardInterrupt:
            try:
                link.close()
            except:
                pass

        except:
            import traceback
            traceback.print_exc()

            try:
                link.close()
            except:
                pass

    elif first_trialArray != None:
        try:
            # Initialize COM Port for Serial Transfer
            link = txfer.SerialTransfer('COM12', 115200, debug=True)

            # Send the first half of trials with packet_id = 0
            first_trialArray_size = 0
            first_trialArray_size = link.tx_obj(first_trialArray)
            link.open()
            link.send(first_trialArray_size, packet_id=0)

            print("First Half of Trials Sent")
            print(first_trialArray)

            while not link.available():
                pass

            # Receive the first half of trials from Arduino
            rxfirst_trialArray = link.rx_obj(obj_type=type(first_trialArray),
            obj_byte_size=first_trialArray_size, list_format='i')

            print("First Half of Trials Received")
            print(rxfirst_trialArray)

            # Confirm packet was sent correctly
            if first_trialArray == rxfirst_trialArray:
                print("First Half Trial Array Transfer Successful!")
            else:
                link.close()
                print("First Half Trial Array Transfer Failure!")
                print("Exiting...")
                sys.exit()

            # Send second half of trials with packet_id = 0
            second_trialArray_size = 0
            second_trialArray_size = link.tx_obj(second_trialArray)
            link.send(second_trialArray_size, packet_id=1)

            print("Second Half of Trials Sent")
            print(second_trialArray)

            # Receive second half of trials from Arduino
            rxsecond_trialArray = link.rx_obj(obj_type=type(second_trialArray),
            obj_byte_size=second_trialArray_size, list_format='i')

            print("Second Half of Trials Received")
            print(rxsecond_trialArray)

            if second_trialArray == rxsecond_trialArray:
                print("Second Half Trial Array Transfer Successful!")
            else:
                link.close()
                print("Second Half Trial Array Transfer Failure!")
                print("Exiting...")
                sys.exit()

            link.close()

        except KeyboardInterrupt:
            try:
                link.close()
            except:
                pass

        except:
            import traceback
            traceback.print_exc()

            try:
                link.close()
            except:
                pass
    else:
        print("Something is wrong...")
        print("Exiting...")
        sys.exit()

Arduino

// Use package SerialTransfer.h from PowerBroker2 https://github.com/PowerBroker2/SerialTransfer
#include "SerialTransfer.h"

// Rename SerialTransfer to myTransfer
SerialTransfer myTransfer;

const int MAX_NUM_TRIALS = 46; // maximum number of trials possible

int32_t trialArray[MAX_NUM_TRIALS]; // create trial array

boolean acquireTrials = true;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);

  myTransfer.begin(Serial1, true);
}


void loop(){
  trials_rx();
}

int trials_rx() {
  if (acquireTrials) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(trialArray);
      Serial.println("Received Trial Array");

      myTransfer.sendDatum(trialArray);
      Serial.println("Sent Trial Array");

      acquireTrials = false;
    }
  }
}

[PowerBroker2]

It'll take me a while to figure out what is going on Python side. I'm guessing you didn't post your whole program and just the pieces that pertain to the USB logic, which is totally fine but it does make it more difficult to understand. Also, I would suggest using more whitespace, especially between the top of a block (edit I mistook some of the code for comments because of how GitHub color codes Python snippets) comment and the code above it like this:

some_code = "hi"

# block
# comment
more_code = "hello"

For more guidelines on Python styling, check out PEP-8

Lastly, I would suggest finding a way to migrate the error handling up out of the function to the parent. Less redundant code and smaller functions is always better.

I'll try to see about suggestions for preventing the actual transfer errors soon.

[jmdelahanty]

It'll take me a while to figure out what is going on Python side. I'm guessing you didn't post your whole program and just the pieces that pertain to the USB logic, which is totally fine but it does make it more difficult to understand.

Ah okay, I was hoping to just offer what was relevant so it was easier, sorry about that. I included the gen_trial_array function just so you wouldn't have to make them yourself. If it's easier, I can remove it! Since sometimes a user might want to run something quick (ie 10-20 trials) I wanted to try and make it flexible so the same function could send different sizes. Maybe it would be better to just have functions that were specific for each case (as in multiple packet vs. single packet)? <--- Definitely going to do this.

Also, I would suggest using more whitespace, especially between the top of a block (edit I mistook some of the code for comments because of how GitHub color codes Python snippets) comment and the code above it like this:

Okay, got it. I definitely haven't been careful about complying with PEP-8 as you can see haha. I'll try and update my code to comply with it so it's easier to read.

Lastly, I would suggest finding a way to migrate the error handling up out of the function to the parent. Less redundant code and smaller functions is always better.

Okay! I'll think more about this. I'm not so great at error handling yet.

I'll try to see about suggestions for preventing the actual transfer errors soon.

Thank you so much! I'm sorry my code is confusing... If you want to wait until I make it more readable I can try my best to do that!

Edit: I just installed Flake8 as a linter and it's very upset with me.
Edit2: In keeping with the idea of having smaller functions and less redundant code, I'm going to definitely just make different functions for single vs. multipacket arrays.

[jmdelahanty]

Okay, so this morning I've been making individual functions for different numbers of packet transfers. Here's what I have that (I'm hoping!) is a lot nicer. The Arduino code is unchanged from before.

Edit: I'm also working on making a couple functions that will send arbitrary numbers of single packet arrays/multi-packet arrays. But, for now, having a function for each array I want to send is the method I'm taking.

###############################################################################
# Import Packages
###############################################################################
# File Types
# Import JSON for configuration file
import json

# Trial Array Generation
# Import scipy.stats truncated normal distribution for ITI Array
from scipy.stats import truncnorm
# Import numpy for trial array generation/manipulation and Harvesters
import numpy as np
# Import numpy default_rng
from numpy.random import default_rng

# Serial Transfer
# Import pySerialTransfer for serial comms with Arduino
from pySerialTransfer import pySerialTransfer as txfer


# -----------------------------------------------------------------------------
# Multi-packet Generation: Trial Array
# -----------------------------------------------------------------------------


def gen_trialArray_multipacket(totalNumberOfTrials):

    # Always initialize trial array with 3 reward trials
    trialArray = [1, 1, 1]

    # Define number of samples needed from generator
    num_samples = totalNumberOfTrials - len(trialArray)

    # Define probability that the animal will receive sucrose 50% of the time
    sucrose_prob = 0.5

    # Initialize random number generator with default_rng
    rng = default_rng(2)

    # Generate a random trial array with Generator.binomial.  Use n=1 to pull
    # one sample at a time and p=0.5 as probability of sucrose.  Use
    # num_samples to generate the correct number of trials.  Finally, use
    # tolist() to convert random_trials from an np.array to a list.
    random_trials = rng.binomial(n=1, p=sucrose_prob, size=num_samples).tolist()

    # Append the two arrays together
    for i in random_trials:
        trialArray.append(i)

    # Use np.array_split() to divide list into smaller sizes
    split_array = np.array_split(trialArray, 2)

    # First index of split array is content of first packet
    first_trialArray = split_array[0].tolist()

    # Second index of split array is content of second packet
    second_trialArray = split_array[1].tolist()

    # TODO: Write out the trial array into JSON as part of experiment config

    # Return trial arrays
    return first_trialArray, second_trialArray

# -----------------------------------------------------------------------------
# Trial Array Transfer: Multi-packet
# -----------------------------------------------------------------------------


def serialtransfer_trialArray_multipacket(first_trialArray, second_trialArray):

    try:
        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize first packet size of 0
        first_trialArray_size = 0

        # Stuff the packet with the first trial array
        first_trialArray_size = link.tx_obj(first_trialArray)

        # Open the communication link
        link.open()

        # Send the first packet with a packet_id of 0
        link.send(first_trialArray_size, packet_id=0)

        print("First Half of Trials Sent")
        print(first_trialArray)

        while not link.available():
            pass

        # Receive the first half of trials from Arduino
        rxfirst_trialArray = link.rx_obj(obj_type=type(first_trialArray),
                                         obj_byte_size=first_trialArray_size,
                                         list_format='i')

        print("First Half of Trials Received")
        print(rxfirst_trialArray)

        # TODO Move error checking outside this function

        # # Confirm packet was sent correctly
        # if first_trialArray == rxfirst_trialArray:
        #     print("First Half Trial Array Transfer Successful!")
        # else:
        #     link.close()
        #     print("First Half Trial Array Transfer Failure!")
        #     print("Exiting...")
        #     sys.exit()

        # Initialize second packet size of 0
        second_trialArray_size = 0

        # Stuff the packet with second trial array
        second_trialArray_size = link.tx_obj(second_trialArray)

        # Send the second packet with a packet_id of 1
        link.send(second_trialArray_size, packet_id=1)

        print("Second Half of Trials Sent")
        print(second_trialArray)

        # Receive second half of trials from Arduino
        rxsecond_trialArray = link.rx_obj(obj_type=type(second_trialArray),
                                          obj_byte_size=second_trialArray_size,
                                          list_format='i')

        print("Second Half of Trials Received")
        print(rxsecond_trialArray)

        # TODO Put error checking outside this function
        # if second_trialArray == rxsecond_trialArray:
        #     print("Second Half Trial Array Transfer Successful!")
        # else:
        #     link.close()
        #     print("Second Half Trial Array Transfer Failure!")
        #     print("Exiting...")
        #     sys.exit()
        #
        # link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass

###############################################################################
# Main Function
###############################################################################


if __name__ == "__main__":

    # Give config file name
    config_file = 'config.json'

    # read JSON config file
    with open(config_file, 'r') as inFile:
        contents = inFile.read()
        # Convert from JSON to Dictionary
        config = json.loads(contents)

    # Gather total number of trials
    num_trials = config["metadata"]["totalNumberOfTrials"]["value"]

    # If there's more than 45 trials, generate multipacket arrays
    first_trialArray, second_trialArray = gen_trialArray_multipacket(num_trials)

    # Use multipacket serial transfer for trial arrays
    serialtransfer_trialArray_multipacket(first_trialArray, second_trialArray)

    sys.exit()

Edit again: Just realized having the config file here would probably be helpful! Sorry for not including it before.

{"metadata": {
"totalNumberOfTrials": {"value": 46},
"punishTone": {"value": 2000},
"rewardTone": {"value": 18000},
"USDeliveryTime_Sucrose": {"value": 5},
"USDeliveryTime_Air": {"value": 20},
"USConsumptionTime_Sucrose": {"value": 1000}
  }
}

[PowerBroker2]

Thanks - the Python looks much better!

Here is what's most likely causing the issue:

        # Send the second packet with a packet_id of 1
        link.send(second_trialArray_size, packet_id=1)

        print("Second Half of Trials Sent")
        print(second_trialArray)

        # while not link.available(): <------------------------ Missing in original script
        #     pass

        # Receive second half of trials from Arduino
        rxsecond_trialArray = link.rx_obj(obj_type=type(second_trialArray),
                                          obj_byte_size=second_trialArray_size,
                                          list_format='i')

[jmdelahanty]

Hooray! I'm glad the Python looks better haha. I'll try my best to keep in line with PEP8 from now on. Thanks for the advice!

This definitely seems to have been the source of the issue! Now I can send multipacket arrays no problem.

You're awesome PowerBroker2. You should change your name to PowerBroker1 because you're number 1.

One last thing for the error checking part, would it be acceptable to have a different function evaluate whether packets are transmitted correctly where the function returns True if the contents match and False if not? And then I can call this error_checker function at the end of the transmission? Or should I have the transmission function return the values of each rxfirst/second_trialArray and have error_checker outside the transmission function?

[PowerBroker2]

Thanks man, always glad to help!

One last thing for the error checking part, would it be acceptable to have a different function evaluate whether packets are transmitted correctly where the function returns True if the contents match and False if not? And then I can call this error_checker function at the end of the transmission? Or should I have the transmission function return the values of each rxfirst/second_trialArray and have error_checker outside the transmission function?

I think if you localize the transmissions to a single function or so, integrating error handling like you already have is fine. I do think it would be beneficial to have a function that serialtransfer_trialArray_multipacket() could call that would replace the code for a single transmit and echo receive packet processing. That function would be called twice, would prevent copy and pasting errors (like the one we just caught), and would shorten the program. You could then have another function serialtransfer_trialArray_multipacket() could call that would verify the array values match like you described. I hope this makes sense.

[jmdelahanty]

Edit2: I have found a solution for my multipacket problem! Unfortunately, it makes single packet transmission broken. I definitely need to be smarter about this... I removed the code I uploaded yesterday and will post a new comment with what I've attempted so far. Turns out, as seen in my most recent comment, I have not actually solved the multipacket problem.

Hey PowerBroker! So I've gotten to the point where I've made the code a bit more modular and, I think, easier to understand. I've made a function that sends an individual packet and, depending on the size of the arrays I need to send, it either sends just one or multiple packets! I've turned this code into a module for my program so my __main__ file isn't too cluttered. Here's what it looks like:

# Bruker 2-Photon Serial Transfer Utils
# Jeremy Delahanty May 2021
# pySerialTransfer written by PowerBroker2
# https://github.com/PowerBroker2/pySerialTransfer

###############################################################################
# Import Packages
###############################################################################

# Serial Transfer
# Import pySerialTransfer for serial comms with Arduino
from pySerialTransfer import pySerialTransfer as txfer

# Import Numpy for splitting arrays
import numpy as np

###############################################################################
# Functions
###############################################################################


###############################################################################
# Serial Transfer to Arduino: One Packet
###############################################################################


# -----------------------------------------------------------------------------
# Send an Individual Packet
# -----------------------------------------------------------------------------


def transfer_packet(array, packet_id):

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize array_size of 0
        array_size = 0

        # Stuff packet with size of trialArray
        array_size = link.tx_obj(array)

        # Open communication link
        link.open()

        # Send array
        link.send(array_size, packet_id=0)

        print("Sent Array")
        print(array)

        while not link.available():
            pass

        # Receive trial array:
        rxarray = link.rx_obj(obj_type=type(array),
                              obj_byte_size=array_size,
                              list_format='i')

        print("Received Array")
        print(rxarray)

        # Close the communication link
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass


# -----------------------------------------------------------------------------
# Configuration/Metadata File Transfer
# -----------------------------------------------------------------------------


# TODO: Add error checking function for configuration
def transfer_metadata(config):

    try:
        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # stuff TX buffer (https://docs.python.org/3/library/struct.html#format-characters)
        metaData_size = 0
        metaData_size = link.tx_obj(config['metadata']['totalNumberOfTrials']['value'],       metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['punishTone']['value'],                metaData_size, val_type_override='H')
        metaData_size = link.tx_obj(config['metadata']['rewardTone']['value'],                metaData_size, val_type_override='H')
        metaData_size = link.tx_obj(config['metadata']['USDeliveryTime_Sucrose']['value'],    metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['USDeliveryTime_Air']['value'],        metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['USConsumptionTime_Sucrose']['value'], metaData_size, val_type_override='H')

        # Open comms to Arudino
        link.open()

        # Send the metadata to the Arduino
        link.send(metaData_size, packet_id=0)

        # While sending the data, the link is unavailable.  Pass this state
        # until done.
        while not link.available():
            pass

        # Receive packet from Arduino
        # Create rxmetaData dictionary
        rxmetaData = {}

        # Start rxmetaData reception size of 0
        rxmetaData_size = 0

        # Receive each field from the Arduino
        rxmetaData['totalNumberOfTrials'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['punishTone'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['H']
        rxmetaData['rewardTone'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['H']
        rxmetaData['USDeliveryTime_Sucrose'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['USDeliveryTime_Air'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['USConsumptionTime_Sucrose'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)

        print(rxmetaData)

        # Close comms to the Arduino
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass
    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass

        # TODO: Move error handling outside of function; need to learn how...

        # if trialArray == rxtrialArray:
        #     print("Trial Array transfer successful!")
        #
        # else:
        #     link.close()
        #     print("Trial Array error! Exiting...")
        #     sys.exit()

        # Close the communication link


# -----------------------------------------------------------------------------
# Trial Array Transfers: One Packet
# -----------------------------------------------------------------------------


def onepacket_transfers(array_list):

    # Give each new packet an ID of 0.  The link is closed per packet
    # transmission.
    packet_id = 0

    # For each array in the list of arrays defining trial data
    for array in array_list:

        # Transfer the packet
        transfer_packet(array, packet_id)


###############################################################################
# Serial Transfer to Arduino: Multi-packet
###############################################################################

# -----------------------------------------------------------------------------
# Trial Array Splitting for Multipacket Transfers
# -----------------------------------------------------------------------------


def split_multipacket_array(array):

    # This function receives a large list of trial variables.  It needs to be
    # split into two arrays using np.array_split.
    split_ndarray = np.array_split(array, 2)

    # Return the split numpy arrays
    return split_ndarray


# -----------------------------------------------------------------------------
# Trial Array Transfers: Multi-packet
# -----------------------------------------------------------------------------


def multipacket_transfer(array_list):

    # For each array in the list of trial arrays
    for array in array_list:

        # Split the array into packets small enough to transfer
        split_array = split_multipacket_array(array)

        # Transfer the arrays as multiple packets
        transfer_arrays_multipacket(split_array)


def transfer_arrays_multipacket(split_array):

    # Initialize first packet with an ID of 0
    packet_id = 0

    # For each array received by the splitting function
    for array in split_array:

        # Save the array as a list for transfer
        array = array.tolist()

        # Send the array
        transfer_packet(array, packet_id)

        # Increment the packet number for sending next array
        packet_id += 1

[jmdelahanty]

Edit: Here's the super hacky way (at least it feels that way...) that I've handled the problem. I've basically forced single packet transmissions to increment the transmissionStatus as if they were multipacket transmissions by adding this to each transmission block. I've updated the code below so it includes this statement:

 if (metadata.totalNumberOfTrials > 45) {
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }

Here's my attempt to control the Arduino's flow for multipacket transmissions better. I've introduced a transmissionStatus that acts as a counter for each packet received by the Arduino. Since I know exactly how many packets are created for multipacket transmissions, I can set up functions that only move forward when the previous data has been received completely. This is a consequence of my update to my code which transmits a single packet per connection to the Arduino. When the data has all been received, I send an update about Python being done with its transmissions. Here's what that looks like:

###############################################################################
# Serial Transfer to Arduino: Python Status
###############################################################################


def update_python_status():

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize array_size of 0
        array_size = 0

        # Stuff packet with size of trialArray
        array_size = link.tx_obj(1)

        # Open communication link
        link.open()

        # Send array
        link.send(array_size, packet_id=0)

        print("Sent END OF TRANSMISSION Status")

        while not link.available():
            pass

        # Receive trial array:
        rxarray = link.rx_obj(obj_type=type(1),
                              obj_byte_size=array_size,
                              list_format='i')

        print("Received END OF TRANSMISSION Status")
        print(rxarray)

        # Close the communication link
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass

And here's how I've tried writing the Arduino code. INCLUDING UPDATE

// Use package SerialTransfer.h from PowerBroker2 https://github.com/PowerBroker2/SerialTransfer
#include "SerialTransfer.h"

// Rename SerialTransfer to myTransfer
SerialTransfer myTransfer;

const int MAX_NUM_TRIALS = 60; // maximum number of trials possible; much larger than needed but smaller than max value of metadata.totalNumberOfTrials




struct __attribute__((__packed__)) metadata_struct {
  uint8_t totalNumberOfTrials;              // total number of trials for experiment
  uint16_t punishTone;                      // airpuff frequency tone in Hz
  uint16_t rewardTone;                      // sucrose frequency tone in Hz
  uint8_t USDeliveryTime_Sucrose;           // amount of time to open sucrose solenoid
  uint8_t USDeliveryTime_Air;               // amount of time to open air solenoid
  uint16_t USConsumptionTime_Sucrose;       // amount of time to wait for sucrose consumption
} metadata;

int32_t trialArray[MAX_NUM_TRIALS]; // create trial array
int32_t ITIArray[MAX_NUM_TRIALS]; // create ITI array
int32_t noiseArray[MAX_NUM_TRIALS]; // create noise array
int32_t transmissionStatus;
int32_t pythonGo;

boolean acquireMetaData = true;
boolean acquireTrials = false;
boolean acquireITI = false;
boolean acquireNoise = false;
boolean rx = true;
boolean pythonGoSignal = false;
boolean arduinoGoSignal = false;


void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);

  myTransfer.begin(Serial1, true);
}


void loop(){
  rx_function();
  pythonGo_rx();
  go_signal();
}

void rx_function() {
  if (rx) {
  metadata_rx();
  trials_rx();
  iti_rx();
  noise_rx(); 
  }
}

int metadata_rx() {
  if (acquireMetaData && transmissionStatus == 0) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(metadata);
      Serial.println("Received Metadata");

      myTransfer.sendDatum(metadata);
      Serial.println("Sent Metadata");
      
      acquireMetaData = false;
      transmissionStatus++;
      Serial.println(transmissionStatus);
      acquireTrials = true;
    }
  }
}

int trials_rx() {
  if (acquireTrials && transmissionStatus >= 1 && transmissionStatus < 3) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(trialArray);
      Serial.println("Received Trial Array");

      myTransfer.sendDatum(trialArray);
      Serial.println("Sent Trial Array");
      if (metadata.totalNumberOfTrials > 45) {  // Hacky way of forcing fix...
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      acquireITI = true;
    }
  }
}

int iti_rx() {
  if (acquireITI && transmissionStatus >= 3 && transmissionStatus < 5) {
    acquireTrials = false;
    if (myTransfer.available())
    {
      myTransfer.rxObj(ITIArray);
      Serial.println("Received ITI Array");

      myTransfer.sendDatum(ITIArray);
      Serial.println("Sent ITI Array");

      if (metadata.totalNumberOfTrials > 45) { // Hacky way of forcing fix...
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      acquireNoise = true;
    }
  }
}

int noise_rx() {
  if (acquireNoise && transmissionStatus >= 5 && transmissionStatus < 7) {
    acquireITI = false;
    if (myTransfer.available())
    {
      myTransfer.rxObj(noiseArray);
      Serial.println("Received Noise Array");

      myTransfer.sendDatum(noiseArray);
      Serial.println("Sent Noise Array");

      if (metadata.totalNumberOfTrials > 45) { // Hacky way of forcing fix...
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      pythonGoSignal = true;
    }
  }
}

void go_signal() {
  if (arduinoGoSignal) {
    Serial.println("GO!");
    arduinoGoSignal = false;
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(trialArray[i]);
    }
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(ITIArray[i]);
    }
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(noiseArray[i]);
    }
  }
}

int pythonGo_rx() {
  if (pythonGoSignal && transmissionStatus == 7) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(pythonGo);
      Serial.println("Received Python Status");
  
      myTransfer.sendDatum(pythonGo);
      Serial.println("Sent Python Status");

      arduinoGoSignal = true;
    }
  }
}

As I noted in the previous comment, this breaks single packet transmissions on the Arduino side of things since there's fewer packets coming in. A fix - probably very hacky - that I'm going to do for now is to simply make a second set of transmission code that's for one packet arrays.

This hacky fix seems to work, but I feel like I'm just doing something wrong...

[jmdelahanty]

First, I'm sorry for all these new comments today and yesterday, I don't mean to spam you...

I've run into a new problem with the transmission where, although the packets are being transmitted to and from the Arduino, the second set of trials are not being appended to the first one. For example, if I want to do an experiment of 46 trials, I'll split the array into two arrays of 23. The transfer goes along as normal and it says that everything has been parsed successfully.

When I go through the experiment, however, the list runs out after the 23rd trial. It's like the second array is overwriting the first array. This is if I use the multipacket functions written above or if I do it like this:

def multipacket_dev(split_array):

    packet_id = 0

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        for array in split_array:

            array = array.tolist()

            # Initialize array_size of 0
            array_size = 0

            # Stuff packet with size of trialArray
            array_size = link.tx_obj(array)

            # Open communication link
            link.open()

            # Send array
            link.send(array_size, packet_id=packet_id)

            print("Sent Array")
            print(array)

            while not link.available():
                pass

            # Receive trial array:
            rxarray = link.rx_obj(obj_type=type(array),
                                  obj_byte_size=array_size,
                                  list_format='i')

            print("Received Array")
            print(rxarray)

            packet_id += 1

        # Close the communication link
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass

Do you think that it's necessary to initialize two different arrays for each transmission on the Arduino side so things don't get written over on accident? I'm sure it's because I'm doing something wrong for storing the two transmissions to the same object...

Edit: An additional thing that seems to happen is that if the array I'm building is too large, as in above 254 bytes, I can't send the array back to Python properly. It turns out that this maxes out at 60 trials before it's too large! This makes sense, but I'm not sure how to properly split the array into multiple transmissions on the Arduino side. In a previous comment here you mentioned using a specific send size. This seems like what I need to do! So the idea would be to send the max amount and then the leftovers when ready. Is that maybe a solution? Still plugging away.

Edit2: Maybe PowerBroker inspired brain blast. What if I specify the start and end of the larger arrays and have the packets organized that way? What I mean by this is that maybe I can set one object as requiring 2 packets to complete it. Something like array_name[object_id] where the object_id is linked a specified number of packets. I swear that's something you've taught me to do with the configuration file transmission! Looking into it...

[PowerBroker2]

I'll take a look at all this once I get the chance - either tomorrow or this weekend

[PowerBroker2]

To make it a little easier for me, could you sum up all the edits into one reply?

[jmdelahanty]

Hey PowerBroker! I apologize for the delay. I've been busy with some other stuff here in the lab and haven't had a chance yet to put it all into one comment that summarizes things correctly based on where I'm at. I'll do this probably on Tuesday afternoon/evening! In the meantime, I've been very successfully using what we have so far to run behavior with my labmates and have generated some really cool stuff. The lab is pretty happy with the system and it operates smoothly.

I'll update you soon! Thanks again man!

[jmdelahanty]

Hey PowerBroker! Thanks for your patience! Okay, here's a summary of what I have going on. There's a TLDR below too.

I've made a module called serialtransfer_utils that I call in my main function which has a more generalized form of sending packets using a couple different functions. I've also discovered that the largest array of trials I can send in one packet is 60 trials which, for now at least, is enough. Each experiment has 3 arrays: trial type, inter-trial interval (a measure of time between each trial), and a sound duration (how long a tone is played before delivering a stimulus). I put these three arrays into a new list that a function called transfer_packet iterates through. This works nicely! The trouble comes when I try to send an object that's larger than 60 trials.

If an array has more than 60 elements, I divide it in half and make a list consisting of both arrays called split_array. In a function called transfer_arrays_multipacket, I give it the split array list and, for each array in split_array, I send one packet. After sending the first packet, I increment packet_id and then send the second packet. This function successfully sends the split up arrays to the Arduino, but on the Arduino side things aren't getting received quite right. For each transfer of a large array, I only open the link once. In other words, both packets are delivered with one call to link.open().

What I'm seeing is that, although the Arduino is receiving both halves of the large array, the first half is getting overwritten by the second half. For example, if I send 70 trials, the Arduino receives two halves of 35. When I print out each value of the array once the transfer is complete, the first 35 values are from the second packet. The second 35 values are all 0s. The first half seems to have disappeared or been overwritten! I've also had to create a transmission_status variable that counts how many times the loop() function has gone through so I can make sure that all the data has been received for each array.

Another issue that I'm thinking is maybe causing this all is that, when I want to confirm on the Python side that things have been received properly, the Arduino is trying to send the whole 70 element array which doesn't fit! The largest size I can send is 60! I thought I could accomplish this with send_data rather than send_datum, but I didn't get that going properly.

-- TLDR --
Second packet overwrites first one, unsure how to send only the relevant half of a multipacket object back to Python.

Here's the code that I've been working with:
Arduino

// Use package SerialTransfer.h from PowerBroker2 https://github.com/PowerBroker2/SerialTransfer
#include "SerialTransfer.h"

// Rename SerialTransfer to myTransfer
SerialTransfer myTransfer;

const int MAX_NUM_TRIALS = 60; // maximum number of trials possible; much larger than needed but smaller than max value of metadata.totalNumberOfTrials




struct __attribute__((__packed__)) metadata_struct {
  uint8_t totalNumberOfTrials;              // total number of trials for experiment
  uint16_t punishTone;                      // airpuff frequency tone in Hz
  uint16_t rewardTone;                      // sucrose frequency tone in Hz
  uint8_t USDeliveryTime_Sucrose;           // amount of time to open sucrose solenoid
  uint8_t USDeliveryTime_Air;               // amount of time to open air solenoid
  uint16_t USConsumptionTime_Sucrose;       // amount of time to wait for sucrose consumption
} metadata;

int32_t trialArray[MAX_NUM_TRIALS]; // create trial array
int32_t ITIArray[MAX_NUM_TRIALS]; // create ITI array
int32_t noiseArray[MAX_NUM_TRIALS]; // create noise array
int32_t transmissionStatus;
int32_t pythonGo;

boolean acquireMetaData = true;
boolean acquireTrials = false;
boolean acquireITI = false;
boolean acquireNoise = false;
boolean rx = true;
boolean pythonGoSignal = false;
boolean arduinoGoSignal = false;


void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);

  myTransfer.begin(Serial1, true);
}


void loop(){
  rx_function();
  pythonGo_rx();
  go_signal();
}

void rx_function() {
  if (rx) {
  metadata_rx();
  trials_rx();
  iti_rx();
  noise_rx(); 
  }
}

int metadata_rx() {
  if (acquireMetaData && transmissionStatus == 0) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(metadata);
      Serial.println("Received Metadata");

      myTransfer.sendDatum(metadata);
      Serial.println("Sent Metadata");
      
      acquireMetaData = false;
      transmissionStatus++;
      Serial.println(transmissionStatus);
      acquireTrials = true;
    }
  }
}

int trials_rx() {
  if (acquireTrials && transmissionStatus >= 1 && transmissionStatus < 3) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(trialArray);
      Serial.println("Received Trial Array");

      myTransfer.sendDatum(trialArray);
      Serial.println("Sent Trial Array");
      
      if (metadata.totalNumberOfTrials > 60) {
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      acquireITI = true;
    }
  }
}

int iti_rx() {
  if (acquireITI && transmissionStatus >= 3 && transmissionStatus < 5) {
    acquireTrials = false;
    if (myTransfer.available())
    {
      myTransfer.rxObj(ITIArray);
      Serial.println("Received ITI Array");

      myTransfer.sendDatum(ITIArray);
      Serial.println("Sent ITI Array");

  if (metadata.totalNumberOfTrials > 60) {
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      acquireNoise = true;
    }
  }
}

int noise_rx() {
  if (acquireNoise && transmissionStatus >= 5 && transmissionStatus < 7) {
    acquireITI = false;
    if (myTransfer.available())
    {
      myTransfer.rxObj(noiseArray);
      Serial.println("Received Noise Array");

      myTransfer.sendDatum(noiseArray);
      Serial.println("Sent Noise Array");

      if (metadata.totalNumberOfTrials > 60) {
        transmissionStatus++;
      }
      else {
        transmissionStatus++;
        transmissionStatus++;
      }
      Serial.println(transmissionStatus);
      pythonGoSignal = true;
    }
  }
}
// Python status function for flow control
int pythonGo_rx() {
  if (pythonGoSignal && transmissionStatus == 7) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(pythonGo);
      Serial.println("Received Python Status");
  
      myTransfer.sendDatum(pythonGo);
      Serial.println("Sent Python Status");

      arduinoGoSignal = true;
    }
  }
}

void go_signal() {
  if (arduinoGoSignal) {
    Serial.println("GO!");
    arduinoGoSignal = false;
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(trialArray[i]);
    }
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(ITIArray[i]);
    }
    for (byte i = 0; i <metadata.totalNumberOfTrials; i++) {
      Serial.println(noiseArray[i]);
    }
    Serial.println(sizeof(trialArray));
    rx = true;
    acquireMetaData = true;
  }
}

And here's my Python:

# Bruker 2-Photon Serial Transfer Utils
# Jeremy Delahanty May 2021
# pySerialTransfer written by PowerBroker2
# https://github.com/PowerBroker2/pySerialTransfer

###############################################################################
# Import Packages
###############################################################################

# Serial Transfer
# Import pySerialTransfer for serial comms with Arduino
from pySerialTransfer import pySerialTransfer as txfer

# Import Numpy for splitting arrays
import numpy as np

# Import sys for exiting program safely
import sys

###############################################################################
# Functions
###############################################################################


###############################################################################
# Serial Transfer to Arduino: Error Checking
###############################################################################


def array_error_check(transmitted_array, received_array):

    # If the transmitted array and received array are equal
    if transmitted_array == received_array:

        # Tell the user the transfer was successful
        print("Successful Transfer")

    # If the transmission failed
    else:

        # Tell the user an error occured
        print("Transmission Error!")

        # Tell the user the program is exiting
        print("Exiting...")

        # Exit the program
        sys.exit()


###############################################################################
# Serial Transfer to Arduino: One Packet
###############################################################################


# -----------------------------------------------------------------------------
# Send an Individual Packet
# -----------------------------------------------------------------------------


def transfer_packet(array, packet_id):

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize array_size of 0
        array_size = 0

        # Stuff packet with size of trialArray
        array_size = link.tx_obj(array)

        # Open communication link
        link.open()

        # Send array
        link.send(array_size, packet_id=packet_id)

        print("Sent Array")
        print(array)

        while not link.available():
            pass

        # Receive trial array:
        rxarray = link.rx_obj(obj_type=type(array),
                              obj_byte_size=array_size,
                              list_format='i')

        print("Received Array")
        print(rxarray)

        # Close the communication link
        link.close()

        array_error_check(array, rxarray)

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass


# -----------------------------------------------------------------------------
# Configuration/Metadata File Transfer
# -----------------------------------------------------------------------------


# TODO: Add error checking function for configuration
def transfer_metadata(config):

    try:
        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # stuff TX buffer (https://docs.python.org/3/library/struct.html#format-characters)
        metaData_size = 0
        metaData_size = link.tx_obj(config['metadata']['totalNumberOfTrials']['value'],       metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['punishTone']['value'],                metaData_size, val_type_override='H')
        metaData_size = link.tx_obj(config['metadata']['rewardTone']['value'],                metaData_size, val_type_override='H')
        metaData_size = link.tx_obj(config['metadata']['USDeliveryTime_Sucrose']['value'],    metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['USDeliveryTime_Air']['value'],        metaData_size, val_type_override='B')
        metaData_size = link.tx_obj(config['metadata']['USConsumptionTime_Sucrose']['value'], metaData_size, val_type_override='H')

        # Open comms to Arudino
        link.open()

        # Send the metadata to the Arduino
        link.send(metaData_size, packet_id=0)

        # While sending the data, the link is unavailable.  Pass this state
        # until done.
        while not link.available():
            pass

        # Receive packet from Arduino
        # Create rxmetaData dictionary
        rxmetaData = {}

        # Start rxmetaData reception size of 0
        rxmetaData_size = 0

        # Receive each field from the Arduino
        rxmetaData['totalNumberOfTrials'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['punishTone'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['H']
        rxmetaData['rewardTone'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['H']
        rxmetaData['USDeliveryTime_Sucrose'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['USDeliveryTime_Air'] = link.rx_obj(obj_type='B', start_pos=rxmetaData_size)
        rxmetaData_size += txfer.ARRAY_FORMAT_LENGTHS['B']
        rxmetaData['USConsumptionTime_Sucrose'] = link.rx_obj(obj_type='H', start_pos=rxmetaData_size)

        print(rxmetaData)

        # Close comms to the Arduino
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass
    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass


# -----------------------------------------------------------------------------
# Trial Array Transfers: One Packet
# -----------------------------------------------------------------------------


def onepacket_transfers(array_list):

    # Give each new packet an ID of 0.  The link is closed per packet
    # transmission.
    packet_id = 0

    # For each array in the list of arrays defining trial data
    for array in array_list:

        # Transfer the packet
        transfer_packet(array, packet_id)


###############################################################################
# Serial Transfer to Arduino: Multi-packet
###############################################################################


# -----------------------------------------------------------------------------
# Trial Array Splitting for Multipacket Transfers
# -----------------------------------------------------------------------------


def split_multipacket_array(array):

    # This function receives a large list of trial variables.  It needs to be
    # split into two arrays using np.array_split.
    split_ndarray = np.array_split(array, 2)

    # Return the split numpy arrays
    return split_ndarray


# -----------------------------------------------------------------------------
# Trial Array Transfers: Multi-packet
# -----------------------------------------------------------------------------


def multipacket_transfer(array_list):

    packet_id = 0
    # For each array in the list of trial arrays
    for array in array_list:

        # Split the array into packets small enough to transfer
        split_array = split_multipacket_array(array)

        # Transfer the arrays as multiple packets
        transfer_arrays_multipacket(split_array, packet_id)
        # multipacket_dev(split_array, packet_id)

        packet_id = 0


def transfer_arrays_multipacket(split_array, packet_id):

    # For each array received by the splitting function
    for array in split_array:

        # Save the array as a list for transfer
        array = array.tolist()

        # Send the array
        transfer_packet(array, packet_id)

        # Increment the packet number for sending next array
        packet_id += 1


def multipacket_dev(split_array, packet_id):

    print(packet_id)
    new_array = []

    for array in split_array:
        new_array.append(array.tolist())

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize array_size of 0
        first_array_size = 0
        second_array_size = 0

        # Stuff packet with size of trialArray
        first_array_size = link.tx_obj(new_array[0])

        # Open communication link
        link.open()

        # Send array
        link.send(first_array_size, packet_id=packet_id)

        # print("Sent Array")
        # print(new_array[0])

        while not link.available():
            pass

        # Receive trial array:
        first_rxarray = link.rx_obj(obj_type=type(new_array[0]),
                              obj_byte_size=first_array_size,
                              list_format='i')

        # print("Received Array")
        # print(first_rxarray)

        packet_id += 1

        second_array_size = link.tx_obj(new_array[1])

        link.send(second_array_size, packet_id=packet_id)

        while not link.available():
            pass

        second_rxarray = link.rx_obj(obj_type=type(new_array[1]),
        obj_byte_size=second_array_size, list_format='i')

        # print(second_rxarray)
        # Close the communication link
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass


###############################################################################
# Serial Transfer to Arduino: Python Status
###############################################################################


def update_python_status():

    try:

        # Initialize COM Port for Serial Transfer
        link = txfer.SerialTransfer('COM12', 115200, debug=True)

        # Initialize array_size of 0
        array_size = 0

        array = 1

        # Stuff packet with size of trialArray
        array_size = link.tx_obj(array)

        # Open communication link
        link.open()

        # Send array
        link.send(array_size, packet_id=0)

        print("Sent END OF TRANSMISSION Status")

        while not link.available():
            pass

        # Receive trial array:
        rxarray = link.rx_obj(obj_type=type(array),
                              obj_byte_size=array_size,
                              list_format='i')

        print("Received END OF TRANSMISSION Status")

        array_error_check(array, rxarray)

        # Close the communication link
        link.close()

    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass

    except:
        import traceback
        traceback.print_exc()

        try:
            link.close()
        except:
            pass

Finally, here's how I call for sending multi-packet objects in Python:

1. Make "big" arrays (larger than 60 elements), store the arrays in array_list
2. Call multipacket_transfer function
3. Let the magic happen

This is hopefully not too much of a mess, but if what I wrote is trash that's okay! I'll improve it until it's understandable enough for you to not have to hit your head over.

[PowerBroker2]

Ok, I think I have an idea of what you're trying to do. One thing about this library is that it was built for speed, so if it finishes processing a packet and there's still serial data in the UART input buffer, it will basically overwrite the last packet with the newer packet. I'm not 100% sure if this is what's happening in your case, but try adding a small delay between transmissions.

Also, you should only need to open the port once in the beginning of your program, but that's up to you.

[jmdelahanty]

I will try that this week! I'm implementing a couple other things for the experiment that's higher priority for the lab.

Also, you should only need to open the port once in the beginning of your program, but that's up to you.

I will also change my program so it does that, thanks for the tip! Keep brokering all that power.

[jmdelahanty]

Okay, so I tried to introduce delays of different amounts of time on the Python side and I still seem to run into the same problem. I was wondering if it could be that when the Arduino tries to send the data back to Python after receiving the first packet, there's an error that's produced on the Python side. I think that the sendDatum part of the Arduino code is trying to send an array that's too big for one packet. I'm not splitting the array right now into something that sendDatum can actually send correctly.

Since I have two packets on the Python side, should I make two sendDatum statements for each half of the total array that's transmitted? I'm not sure how that would be the problem, but I figured I would ask.

[PowerBroker2]

should I make two sendDatum statements for each half of the total array that's transmitted?

That sounds like the right idea, however, the devil is always in the detail. Whatever the solution is, make sure it's scalable and relatively straight forward. Bandaids and hack solutions are terrible in the long run.

Here's a few things to keep in mind:

1. Always know the length of the data you're trying to stuff in the packets. No exceptions!
2. If you send more than one packet for a given object (i.e. a super long array or struct), you need to keep track of both how many packets it will take to reconstruct the object on the other side and what packet is currently being processed. This data must also be shared with the device on the other side of the transmission. An easy way of doing this is by reserving 2 bytes (one for number of packets in total and one for the current packet number) in the payload, where the transmitter is saying, "hey, this is packet 2 of 4 to reconstruct array x". I do exactly this in my text file transfer examples (TX and RX)

I hope this ramble makes sense

[jmdelahanty]

Whatever the solution is, make sure it's scalable and relatively straight forward. Bandaids and hack solutions are terrible in the long run.

Got it, I'll be extra careful to make sure the solution is scalable.

For your next two points, I'm pretty sure this makes sense. I will do my best to implement this by following your text examples, thanks for linking them! I'm pretty sure I understand the idea.

Over the next few days, I'll try to implement this but it may be a little bit before I get there. There's other things I gotta get done first for the project. I'll keep you posted! Thanks man!

[jmdelahanty]

Hey PowerBroker,

First, I hope that things have been going well and that you have power in abundance!

I just wanted to check in with how things are going. The system is still working great and we're collecting data nicely! I've been told that getting trial sizes greater than 60 isn't much of a priority and so I've had to work on other things over the past couple months. I plan on coming in this weekend to try and make it happen since it'd be cool, fun to learn, and is something I think about often since I feel like I'm so close!

In the meantime, a different enhancement for the system is something I've been working on. I'm trying to get the Arduino to reset after a session is over on its own so I don't have to reupload the .ino file each time. I've used the button on the board and made a reset function that successfully restarts the Arduino, but it seems that it's retaining the previous session's information as if the reset doesn't flush/reinitialize variables. I was looking into how things are stored in Arduinos and I came across the terms EEPROM and flash memory, but I don't know how to tell where things are actually being stored on the board! Do you have any advice for resetting things properly/resources where I can learn where transfers are stored?

[PowerBroker2]

Hey Jeremy, glad to hear things are working well for you!

I'm trying to get the Arduino to reset after a session is over on its own so I don't have to reupload the .ino file each time

Why do you need to reset the Arduino after a session? You should be able to have the code start over, wait for new config input from Python via USB, and then start the next session. I haven't looked at your codebase in a while, but I understand this change could be a hefty effort.

I've used the button on the board and made a reset function that successfully restarts the Arduino, but it seems that it's retaining the previous session's information as if the reset doesn't flush/reinitialize variables.

This is pretty strange. All variables are stored in SRAM unless you specifically tell the compiler to store the variable in flash via the PROGMEM modifier or use the EEPROM.h library to save values in EEPROM. Variables in SRAM get cleared/reinitialized during a reset while variables in flash and EEPROM retain their values.

Are you sure the normal SRAM variables are retaining non-initialized values after button resets? What about hard resets?

[jmdelahanty]

Hello! Things are definitely going well. Maybe I can broker power of my own one day if I continue learning from you!

Why do you need to reset the Arduino after a session? You should be able to have the code start over, wait for new config input from Python via USB, and then start the next session. I haven't looked at your codebase in a while, but I understand this change could be a hefty effort.

I had thought that this is what was supposed to happen, but I couldn't get the code to start over from the very beginning of the sketch. It plows right through all the flags I thought would reset to their starting statuses it seems and starts giving trials. Could it be because I don't have those flags written in the Setup part of the sketch? If it is a hefty effort, I think it would definitely be worth pursuing nonetheless! I'm having a feeling that it's because I'm coding badly and is probably a pretty simple fix.

Are you sure the normal SRAM variables are retaining non-initialized values after button resets?

So I haven't verified that it is for sure retaining non-initialized values after reset which I can try this weekend when I'm hacking around. I assumed it was retaining them because it starts delivering stimuli once the reset is complete. I figured that an array that was allocated but not written to would crash the system rather than keep going (as in when I create trialArray[MAX_NUM_TRIALS] but haven't written anything to it yet with SerialTransfer).

The setup always starts with a sucrose trial and, after reset, it delivers sucrose as well. I'll try printing out the trialArray values after a reset to see what it does. I never let it go very long because I knew something was up.

What about hard resets?

I want to say it just started delivering trials right away after a hard reset now that you mention it, but it's been a while since I've tested. I'll try it out this weekend too. This makes me think that it probably is something wrong with my flags not being changed properly during/after the reset...

[PowerBroker2]

Hey man, sorry for the late reply - I've been busy with a bunch of stuff lately. It seems like you already have the logic in place to reset the Arduino in software, so the hefty effort will likely be in the form of bug hunting instead of refactoring. At least one bug is causing your flags in the logic to be ignored.

Could it be because I don't have those flags written in the Setup part of the sketch?

It might be an issue if the flags aren't properly initialized. You also might need your own "setup" function that is called as soon as the trial is done. The function would then reset all the variables needing reset plus a flag to say "hey, tell Python I'm done with the trial and wait for more input". I think that would be a good way to do it.

[jmdelahanty]

Hey man, sorry for the late reply - I've been busy with a bunch of stuff lately.

No problem at all!

At least one bug is causing your flags in the logic to be ignored... It might be an issue if the flags aren't properly initialized. You also might need your own "setup" function that is called as soon as the trial is done. The function would then reset all the variables needing reset plus a flag to say "hey, tell Python I'm done with the trial and wait for more input".

Okay, that's super helpful! I'll try making a different setup function that reinitializes flags and see how that turns out. I'll keep you posted! Thanks man!

[jmdelahanty]

Hello! Fun quick update, I got the reset function to work properly with your help! Thank you so much! It's going to make the running of our experiments easier to do for sure. I'm in the middle of refactoring things for the codebase and, once that's done, I'll keep working on multi-packet transmissions. You can check it out/show it to future employers if you'd like. Here's the repo!. The branch I'm working on is called "bruker_control_refactor". There's also some docs I tried writing but there's a few typos here.

[PowerBroker2]

Nice work man! I love the auto documentation and website. I actually want to do something similar for both this repo and this other Python library, but it's not working right. I go into the docs folder, open cmd, I do sphinx-quickstart and keep all defaults. Then I make sure the conf.py has the following contents:

# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('..'))


# -- Project information -----------------------------------------------------

project = 'WarThunder'
copyright = '2021, PowerBroker2'
author = 'PowerBroker2'

# The full version, including alpha/beta/rc tags
release = '2.3.0'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [ 
    'sphinx.ext.autodoc',
    'sphinx.ext.napoleon',
    'sphinx_autodoc_typehints'
]


autoclass_content = 'both'
atuodoc_member_order = 'bysource'

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'classic'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []

and make sure the index.rst has the following contents:

.. WarThunder documentation master file, created by
   sphinx-quickstart on Sun Aug 29 16:32:40 2021.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to WarThunder's documentation!
======================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:


telemetry.py
==============

Module to query and access telemetry data during War Thunder matches.

.. currentmodule:: telemetry

.. automodule:: telemetry
  :members:


mapinfo.py
==============

Module to query and access map and non-player object/vehicle data during War Thunder matches.

.. currentmodule:: mapinfo

.. automodule:: mapinfo
  :members:


acmi.py
==============

Module to make Tacview compatible ACMI files - see https://www.tacview.net/documentation/acmi/en/.

.. currentmodule:: acmi

.. automodule:: acmi
  :members:


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

After this, I run in cmd sphinx-apidoc -o . .. --ext-autodoc and make html.

After opening the WarThunder.html in the _build folder doesn't include all of the docstrings and stuff I need in there.

Do you have any advice on how to fix it to make it look like your website? (except that I want to use the "classic" format)

[PowerBroker2]

To be clear: I'm testing this documentation stuff with my other repo before I mess with this one

[PowerBroker2]

Nvm, it was a real pain, but I got it to work with my other repo. Here's the site: https://warthunder.readthedocs.io/en/latest/

[PowerBroker2]

Wait, two of my modules aren't showing up in the website. Bruh, how did you figure out how to do this?

[jmdelahanty]

Hey! It took a while of honestly just doing trial and error rewriting the documentation I had written in the files themselves I think when that happened. Let me take a peek really quick.

From what I'm seeing in your doc strings and stuff everything looks like how I learned online, so I'm not sure why it's not populating all of your modules. I see that your docs have some additional files for what the modules themselves are and it's that modules.rst file that's referenced elsewhere which I didn't do. I think that's the only difference I see really... I can look more this evening and see what I can come up with! It was definitely a frustrating process for it to build "successfully" without errors only to load the page and see half my stuff was missing.

So I'm totally guessing here, but I'm wondering if Sphinx knows how to handle multiple .rst files. Since there's a couple .rst files in the docs folder, maybe Sphinx gets confused?

Another thing that could help is if you use the make clean command before using make html again. At least that's what the internet told me when I was trying to build mine.

Also, I noticed that in your conf.py, the os.path line at the top might be looking in the folder above docs? (sys.path.insert(0, os.path.abspath('..')). I'm not great at path stuff so maybe I'm reading that wrong...

Lastly, there was a while where I was convinced one night that I was never going to get it right when I hit refresh out of frustration and suddenly all my stuff was there. Someone I chatted with mentioned that if you push a lot stuff to the readthedocs site quickly it can take a while to actually populate everything somehow...

Another update!

After updating some stuff for my own docs, I may have discovered an issue! I can't tell if this is what's happening with yours, but read the docs fails to build certain modules if it can't import a package that's used in your files successfully. Sphinx/rtd imports the code modules into the containers hosting your documentation. If there's an issue with getting that package installed on rtd's end, the system passes over it and builds the docs without the code you wrote. It's worth it to look at the Raw output of the build on the site and look it over. I'm betting somewhere in that output there's a section that will describe how an import failed. The way I've been overcoming this is by commenting out code that doesn't import while keeping my docstrings/functions in place. I then commit this to the repo, build the docs, and then uncomment out those parts. This is obviously not a good/sustainable solution, but until I figure out how to import opencv and an appropriate distribution of win32com for read the docs - using import_mock_modules doesn't work for me - that's how I've been solving that issue.