Create an iterator interface for alternating_update

[mtfishman]

An iterator interface for alternating_update could allow advanced users more fine-grained control over iterating through region updates and sweeps, for example allow them to change the sweep pattern based on the current state of the solver, perform custom printing, move objects back and forth between different devices (i.e. CPU, GPU, disk, etc.) based on properties of the current state, etc.

Additionally, higher level function interfaces like alternating_update can be implemented as loops over those iterators, which could improve the code logic.

For example, alternating_update could be implemented like this:

sweep_iterator = AlternatingUpdateIterator(projected_operator, init_state, sweep_plans)
for (; region_iterator, which_sweep) in sweep_iterator # Iterate over sweep updates
  @show which_sweep # Custom printing
  for (; state, region) in region_iterator # Iterate over the region updates of the sweep
    @show region # More custom printing
    # Perform some logic, like if maxlinkdim(state) > 1000, move from GPU to CPU
    if maxlinkdim(state) > 1000
      sweep_iterator.state = cpu(state)
    end
  end
end

That's just a rough outline and example, I'm sure the interface won't be exactly like that.

This design approach could also help us rethink the sweep plan logic, which is a bit of a mess right now, and we'll also have to think about how it interfaces with the observer system (I think it fits in well, and would allow users to inject calls to the observer in a more custom way, while right now the calls are hardcoded at certain points in the code logic).

See these references for examples of iterative algorithms based around iterators:
https://docs.sciml.ai/DiffEqDocs/stable/basics/integrator
https://iterativesolvers.julialinearalgebra.org/dev/iterators
https://discourse.julialang.org/t/ann-optimkit-jl-a-blissfully-ignorant-julia-package-for-gradient-optimization/41063/3

@emstoudenmire @JoeyT1994

[emstoudenmire]

I like this. If I'm thinking about it correctly, the current design is based around an iterable data structure, so this idea would be conceptually a "superset" of the current one.

That is, one could still implement this new iterator design by just filling up a data structure and returning that, but one would not have to do it that way and could also have the freedom to write a set of functions that pass a stateful object to each other, which might be much shorter and cleaner to code and more 'dynamic'. These functions could check certain states and decide to do more steps on the fly or change keywords or other things like that.

Also we would probably make the default sweep_iterator be more of the "stateful function" design since that would be well suited for simple iteration patterns, like just walking a tree.