-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Serialize full pipeline configurations #469
Merged
mdekstrand
merged 24 commits into
lenskit:main
from
mdekstrand:feature/pipeline-serialization
Aug 8, 2024
Merged
Changes from 11 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
4538ba8
add Pydantic dependency
mdekstrand fc61a1d
add string type representations
mdekstrand 7943203
move and split pipeline clone / config
mdekstrand 7577d1e
add pydantic to docs
mdekstrand 176df23
add initial config model + input node + test
mdekstrand ff4d598
store types in sets
mdekstrand 8ee677a
fix type error in pipeline construction
mdekstrand 1ba6aa1
Serialize and round-trip component inputs
mdekstrand 0691cce
parse qualified (colon-separated) types
mdekstrand b0f44d2
serialize to qualified names
mdekstrand 8fbe084
support round-trip configuration
mdekstrand 8c398df
refactor configuration with pattern-matching
mdekstrand 6153b6e
implement fallback node serialization
mdekstrand e43f51b
document lack of support for serializing literal nodes
mdekstrand 4a03762
add pipeline metadata
mdekstrand 1b3c838
add configuration hashing
mdekstrand fa5ad11
clarify from_config comments
mdekstrand ae4b9d8
add pipeline error & warning classes
mdekstrand 2a598cb
update to properly throw and test for pipeline errors
mdekstrand cf2ff62
warn when parameter has no annotation
mdekstrand d218f10
fix deprecated warning method
mdekstrand 6c93561
document @-nodes
mdekstrand 48e4e27
fix SHA docs
mdekstrand 886f35b
only serialize once to insert hash into config
mdekstrand File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
""" | ||
Pydantic models for pipeline configuration and serialization support. | ||
""" | ||
|
||
# pyright: strict | ||
from __future__ import annotations | ||
|
||
from types import FunctionType | ||
|
||
from pydantic import BaseModel, Field | ||
from typing_extensions import Any, Optional, Self | ||
|
||
from .components import ConfigurableComponent | ||
from .nodes import ComponentNode, InputNode | ||
from .types import type_string | ||
|
||
|
||
class PipelineConfig(BaseModel): | ||
""" | ||
Root type for serialized pipeline configuration. A pipeline config contains | ||
the full configuration, components, and wiring for the pipeline, but does | ||
not contain the | ||
""" | ||
|
||
inputs: list[PipelineInput] | ||
components: dict[str, PipelineComponent] = Field(default_factory=dict) | ||
|
||
|
||
class PipelineInput(BaseModel): | ||
name: str | ||
"The name for this input." | ||
types: Optional[set[str]] | ||
"The list of types for this input." | ||
|
||
@classmethod | ||
def from_node(cls, node: InputNode[Any]) -> Self: | ||
if node.types is not None: | ||
types = {type_string(t) for t in node.types} | ||
else: | ||
types = None | ||
|
||
return cls(name=node.name, types=types) | ||
|
||
|
||
class PipelineComponent(BaseModel): | ||
code: str | ||
""" | ||
The path to the component's implementation, either a class or a function. | ||
This is a Python qualified path of the form ``module:name``. | ||
""" | ||
|
||
config: dict[str, object] | None = Field(default=None) | ||
""" | ||
The component configuration. If not provided, the component will be created | ||
with its default constructor parameters. | ||
""" | ||
|
||
inputs: dict[str, str] = Field(default_factory=dict) | ||
""" | ||
The component's input wirings, mapping input names to node names. | ||
""" | ||
|
||
@classmethod | ||
def from_node(cls, node: ComponentNode[Any]) -> Self: | ||
comp = node.component | ||
if isinstance(comp, FunctionType): | ||
ctype = comp | ||
else: | ||
ctype = comp.__class__ | ||
|
||
code = f"{ctype.__module__}:{ctype.__qualname__}" | ||
|
||
config = comp.get_config() if isinstance(comp, ConfigurableComponent) else None | ||
|
||
return cls(code=code, config=config, inputs=node.connections) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# This file is part of LensKit. | ||
# Copyright (C) 2018-2023 Boise State University | ||
# Copyright (C) 2023-2024 Drexel University | ||
# Licensed under the MIT license, see LICENSE.md for details. | ||
# SPDX-License-Identifier: MIT | ||
|
||
import json | ||
|
||
from lenskit.pipeline import Pipeline | ||
from lenskit.pipeline.components import AutoConfig | ||
from lenskit.pipeline.nodes import ComponentNode | ||
|
||
|
||
class Prefixer(AutoConfig): | ||
prefix: str | ||
|
||
def __init__(self, prefix: str = "hello"): | ||
self.prefix = prefix | ||
|
||
def __call__(self, msg: str) -> str: | ||
return self.prefix + msg | ||
|
||
|
||
def test_auto_config_roundtrip(): | ||
comp = Prefixer("FOOBIE BLETCH") | ||
|
||
cfg = comp.get_config() | ||
assert "prefix" in cfg | ||
|
||
c2 = Prefixer.from_config(cfg) | ||
assert c2 is not comp | ||
assert c2.prefix == comp.prefix | ||
|
||
|
||
def test_pipeline_config(): | ||
comp = Prefixer("scroll named ") | ||
|
||
pipe = Pipeline() | ||
msg = pipe.create_input("msg", str) | ||
pipe.add_component("prefix", comp, msg=msg) | ||
|
||
assert pipe.run(msg="FOOBIE BLETCH") == "scroll named FOOBIE BLETCH" | ||
|
||
config = pipe.component_configs() | ||
print(json.dumps(config, indent=2)) | ||
|
||
assert "prefix" in config | ||
assert config["prefix"]["prefix"] == "scroll named " |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
from lenskit.pipeline.config import PipelineInput | ||
from lenskit.pipeline.nodes import InputNode | ||
|
||
|
||
def test_untyped_input(): | ||
node = InputNode("scroll") | ||
|
||
cfg = PipelineInput.from_node(node) | ||
print(cfg) | ||
assert cfg.name == "scroll" | ||
assert cfg.types is None | ||
|
||
|
||
def test_input_with_type(): | ||
node = InputNode("scroll", types={str}) | ||
|
||
cfg = PipelineInput.from_node(node) | ||
print(cfg) | ||
assert cfg.name == "scroll" | ||
assert cfg.types == {"str"} | ||
|
||
|
||
def test_input_with_none(): | ||
node = InputNode("scroll", types={str, type(None)}) | ||
|
||
cfg = PipelineInput.from_node(node) | ||
print(cfg) | ||
assert cfg.name == "scroll" | ||
assert cfg.types == {"None", "str"} | ||
|
||
|
||
def test_input_with_generic(): | ||
node = InputNode("scroll", types={list[str]}) | ||
|
||
cfg = PipelineInput.from_node(node) | ||
print(cfg) | ||
assert cfg.name == "scroll" | ||
assert cfg.types == {"list"} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bugfix for existing code around
None
.