Skip to content

rsrc tagging 3.3 design

Jump to bottom Edit New page
chris grzegorczyk edited this page Jan 11, 2013 · 2 revisions

Table of Contents

Overview

This design covers both tagging and filtering of resources.

Tracking

Document Status Draft
Updated 2013/01/07 Update with details / changes from implementation, add tracking section
Updated 2013/01/09 Add details on filtering for a resource

Out of Scope

Filters related to functional areas that we do not suport are out of scope. These areas are:

  • VPC
  • Spot instances

Feature Dependencies

Support for filters related to the following items depend on these features being included in the release:

Related Features

This feature relates to the following features in this release:

  • CloudWatch
  • Resource Attributes

Analysis

Open Issues

  • How will the eucalyptus account manage tags?, there is no place to add the usual verbose pseudoselector.
  • With filters, does . match newline?
  • Does a tag delete operation succeed if any tag was deleted? (or all tags, or just if there is no error)
  • If using multiple filters are they expected to match the same (sub) item? (persumably not, example below)
  • Should there be an error on an incorrectly escaped wildcard (e.g. "\a")
  • How do wildcards interact with typed filters? (numbers, dates and bools)
  Filters:
    ip-permission.protocol=tcp
    ip-permission.to-port=1
  Match both?:
    Security Group A
      Rule 1 = protocol tcp, port 1
    Security Group B
      Rule 1 = protocol udp, port 1
      Rule 2 = protocol tcp, port 2

Design

Tagging

Eucalyptus Extensions

We will reserve the additional prefix euca for tag names and values. We will support the standard aws prefixes, the euca prefix is for use where there is no aws equivalent (Eucalyptus extensions)

Entity Model

Tags are supplementary information, so must not impact the resource that they tag. Tags can be created by accounts that do not own the tagged resource so must be separately managed.

The entity model aims to use database features for:

  1. Tag lifecycle - delete when tagged resource deleted
  2. Tag constraint - relation to tagged data must be valid
These aims are achieved with the following model:

design-tag-er.png

Note that tags are deleted when the tagged resource is deleted (e.g. an instance), but would not be restored when that instance is restored. This also means that you cannot filter a terminated instance by its tags.

Service Impact

Tag CRUD is part of the EC2 API, internally a new TagManager component is created to handle tags.

Tag functionality must be tested with both SOAP and Query APIs.

Tag Operations

The tag manager implements the following operations.

CreateTags

Tag creation enforces the following:

  • Maximum tags - each account has a limited number of tags per resource
  • Tag key and value format syntactic validation of tag data
  • Tag key and value semantic validation (forbid reserved values / duplicate keys for an account)
Multiple resources and tags can be specified in a request. Every specified tag is added to every specified resource.

Specifying a key that already exists for a resource overwrites the existing value for that tag.

DeleteTags

Tag deletion deletes all matching tags from all specified resources.

Tag values are optional, if specified the value must match.

The operation succeeds if any tag was deleted

DescribeTags

Describe tags for the account with possible filters as outlined below.

You can tag public or shared resources, but the tags you assign are available only to your AWS account and not to the other accounts sharing the resource.

Integration Areas

IAM: tag operations access controlled via IAM policies

Filtering

Filtering should be done by the DB where possible for efficiency. Since filters can have multiple values query by example is not a good fit and criteria will be used.

Modularity

Filtering is an EC2 concept but the implementation should be flexible enough for use with other APIs. Other APIs will often allow a subset of information to be returned from an operation based on some selection parameters.

Discovery

Filters should be discovered, whether individually, per type or per API.

Database Filtering

Filtering will be peformed at query time where possible. It is not always possible to filter using the DB, the following cases have been identified:

  • Type mismatch - Where the DB type differs from the filter type wildard filtering is not possible (e.g. Volume size is an integer in the DB)
  • Translated values - Some values are translated from internal to EC2 (e.g. Volume status)
  • ElementCollections - Hibernate does not support criteria queries for element collections
Database filtering is an optimization, it will be used to restrict the results to the extent possible, but will not reliably filter out all non-matching resources.

Filtering via Hibernate Criterion

Hibernate criterion can be used to filter database listings. A new Entities method is required to support this: 

  List<T> query( T example, 
                 Bool readOnly, 
                 Criterion criterion,
                 Map aliases )

The aliases are required in the case of filters on related entities.

Filters can be translated to criterion as follows: 

  conjunction // And
  - disjunction // Or
    - filter restriction 1
    - filter restriction 2
  - disjunction // Or
    - filter restriction 3

The top level criterion is a conjunction (Restrictions.conjunction()), containing a disjuction for each filter with all the permitted values.

Wildcard Translation

Filters support the following wildcards:

  • *: Matches zero or more characters
  • ?: Matches exactly one character
Hibernate supports the following wildcards when using like:
  • % : Matches zero or more characters
  • _: Matches exactly one character
In both cases wildcards can be escaped (to allow literal values) with a backslash (\).

Translation of wildcards for direct DB filtering must support literal values from each grammar.

Example for Security Groups

An example of using criterion for restriction of query results is: 

  final Junction filters = Restrictions.conjunction();
  final Junction nameFilter = Restrictions.disjunction();
  final Junction ruleTypeFilter = Restrictions.disjunction();
  filters.add( nameFilter );
  filters.add( ruleTypeFilter );
  nameFilter.add( Restrictions.like( "displayName", "A%" ) );
  nameFilter.add( Restrictions.like( "displayName", "B%" ) );
  ruleTypeFilter.add( Restrictions.like( "networkRules.protocol", NetworkRule.Protocol.tcp ) ); // Enum type

In this case the resulting (Hibernate generated) query is: 

  select 
       this_.id as id10_5_, this_.creation_timestamp as creation2_10_5_, this_.last_update_timestamp as last3_10_5_, this_.metadata_perm_uuid as metadata4_10_5_, this_.version as version10_5_, this_.metadata_display_name as metadata6_10_5_, this_.metadata_last_state as metadata7_10_5_, this_.metadata_state as metadata8_10_5_, this_.metadata_state_change_stack as metadata9_10_5_, this_.metadata_account_name as metadata10_10_5_, this_.metadata_account_id as metadata11_10_5_, this_.metadata_unique_name as metadata12_10_5_, this_.metadata_user_id as metadata13_10_5_, this_.metadata_user_name as metadata14_10_5_, this_.metadata_network_group_description as metadata15_10_5_, this_.vm_network_index as vm16_10_5_, extantnetw3_.id as id9_0_, extantnetw3_.creation_timestamp as creation2_9_0_, extantnetw3_.last_update_timestamp as last3_9_0_, extantnetw3_.metadata_perm_uuid as metadata4_9_0_, extantnetw3_.version as version9_0_, extantnetw3_.metadata_display_name as metadata6_9_0_, extantnetw3_.metadata_last_state as metadata7_9_0_, extantnetw3_.metadata_state as metadata8_9_0_, extantnetw3_.metadata_state_change_stack as metadata9_9_0_, extantnetw3_.metadata_account_name as metadata10_9_0_, extantnetw3_.metadata_account_id as metadata11_9_0_, extantnetw3_.metadata_unique_name as metadata12_9_0_, extantnetw3_.metadata_user_id as metadata13_9_0_, extantnetw3_.metadata_user_name as metadata14_9_0_, extantnetw3_.networkGroup_id as network16_9_0_, extantnetw3_.metadata_extant_network_tag as metadata15_9_0_, indexes4_.metadata_extant_network_index_fk as metadata18_9_7_, indexes4_.id as id7_, indexes4_.id as id12_1_, indexes4_.creation_timestamp as creation2_12_1_, indexes4_.last_update_timestamp as last3_12_1_, indexes4_.metadata_perm_uuid as metadata4_12_1_, indexes4_.version as version12_1_, indexes4_.metadata_display_name as metadata6_12_1_, indexes4_.metadata_last_state as metadata7_12_1_, indexes4_.metadata_state as metadata8_12_1_, indexes4_.metadata_state_change_stack as metadata9_12_1_, indexes4_.metadata_account_name as metadata10_12_1_, indexes4_.metadata_account_id as metadata11_12_1_, indexes4_.metadata_unique_name as metadata12_12_1_, indexes4_.metadata_user_id as metadata13_12_1_, indexes4_.metadata_user_name as metadata14_12_1_, indexes4_.metadata_network_index_bogus_id as metadata15_12_1_, indexes4_.metadata_network_index_extant_network_fk as metadata17_12_1_, indexes4_.metadata_network_index as metadata16_12_1_, extantnetw5_.id as id9_2_, extantnetw5_.creation_timestamp as creation2_9_2_, extantnetw5_.last_update_timestamp as last3_9_2_, extantnetw5_.metadata_perm_uuid as metadata4_9_2_, extantnetw5_.version as version9_2_, extantnetw5_.metadata_display_name as metadata6_9_2_, extantnetw5_.metadata_last_state as metadata7_9_2_, extantnetw5_.metadata_state as metadata8_9_2_, extantnetw5_.metadata_state_change_stack as metadata9_9_2_, extantnetw5_.metadata_account_name as metadata10_9_2_, extantnetw5_.metadata_account_id as metadata11_9_2_, extantnetw5_.metadata_unique_name as metadata12_9_2_, extantnetw5_.metadata_user_id as metadata13_9_2_, extantnetw5_.metadata_user_name as metadata14_9_2_, extantnetw5_.networkGroup_id as network16_9_2_, extantnetw5_.metadata_extant_network_tag as metadata15_9_2_, networkgro6_.id as id10_3_, networkgro6_.creation_timestamp as creation2_10_3_, networkgro6_.last_update_timestamp as last3_10_3_, networkgro6_.metadata_perm_uuid as metadata4_10_3_, networkgro6_.version as version10_3_, networkgro6_.metadata_display_name as metadata6_10_3_, networkgro6_.metadata_last_state as metadata7_10_3_, networkgro6_.metadata_state as metadata8_10_3_, networkgro6_.metadata_state_change_stack as metadata9_10_3_, networkgro6_.metadata_account_name as metadata10_10_3_, networkgro6_.metadata_account_id as metadata11_10_3_, networkgro6_.metadata_unique_name as metadata12_10_3_, networkgro6_.metadata_user_id as metadata13_10_3_, networkgro6_.metadata_user_name as metadata14_10_3_, networkgro6_.metadata_network_group_description as metadata15_10_3_, networkgro6_.vm_network_index as vm16_10_3_, networkrul1_.id as id11_4_, networkrul1_.creation_timestamp as creation2_11_4_, networkrul1_.last_update_timestamp as last3_11_4_, networkrul1_.metadata_perm_uuid as metadata4_11_4_, networkrul1_.version as version11_4_, networkrul1_.metadata_network_rule_high_port as metadata6_11_4_, networkrul1_.metadata_network_rule_low_port as metadata7_11_4_, networkrul1_.metadata_network_rule_protocol as metadata8_11_4_ 
  from 
       metadata_network_group this_ left outer join metadata_extant_network extantnetw3_ on this_.vm_network_index=extantnetw3_.id 
           left outer join metadata_network_indices indexes4_ on extantnetw3_.id=indexes4_.metadata_extant_network_index_fk 
           left outer join metadata_extant_network extantnetw5_ on indexes4_.metadata_network_index_extant_network_fk=extantnetw5_.id 
           left outer join metadata_network_group networkgro6_ on extantnetw5_.networkGroup_id=networkgro6_.id 
           inner join metadata_network_rule networkrul1_ on this_.id=networkrul1_.metadata_network_group_rule_fk 
  where 
       (this_.metadata_account_id like ?) and 
       ((this_.metadata_display_name like ? or 
         this_.metadata_display_name like ?) and 
       (networkrul1_.metadata_network_rule_protocol like ?))

Testability

As seen in the above example, use of criterion (as opposed to something like the JPA metamodel) result in textual reference to properties such as: 

  networkRules.protocol

This is not verifiable at compilation time, so these reference will be verified with unit tests, e.g.: 

  class NetworkGroupFilterSupportTest
    testFilteringSupport
      new NetworkGroupFilterSupport().getPersistenceFilters().values().each { ...

Collection Filtering

Filtering on collections of items must be possible for either internal for EC2 formats (TODO pick one format)

Filtering should be implemented using Predicates (Guava)

Similarly to database filter construction we can contruct a Predicate for a given set of filters: 

  Predicates.and
  - Predicates.or
    - filter restriction 1
    - filter restriction 2
  - Predicates.or
    - filter restriction 3

Each predicate will extract the list of values for that filter and check against a distinct given value (or wildcard).

Wildcard Translation

Filters support the following wildcards:

  • *: Matches zero or more characters
  • ?: Matches exactly one character
These can be translated to regular expression wildcards:
  • .*: Matches zero or more characters
  • .: Matches exactly one character
Other regular expression syntax from the filter value must be escaped (e.g. Pattern.quote)

Filter Generation

It will be possible to generate a filter from a filter set, e.g. 

  Filter
  - Map<String,String> getAliases()  // For DB filtering
  - Criterion asCriterion()          // For DB filtering
  - Predicate asPredicate()          // For collection filtering

Support for DB filtering is optional, in this case aliases will be an empty Map and the Criterion will be a no-op (e.g. Restrictions.conjunction())

Resource Filtering Details

Each resource that supports filtering will include the following implementation classes:

  • *FilterSupport - Defines how to filter for a resource
  • Filter*Functions - function enumerations to access values of each item that can be filtered
  • *FilterSupportTest - Unit test for the resource, should cover each filterable item
For the second item there is no requirement to use an enumeration for the functions, but it is suggested that for clarity the function definition is outside of the *FilterSupport constructor.

The support classes declare how to filter using the following approach: 

  public static class NetworkGroupFilterSupport extends FilterSupport<NetworkGroup> {
    public NetworkGroupFilterSupport() {
      super( builderFor( NetworkGroup.class )
          .withStringProperty( "description", FilterFunctions.DESCRIPTION )
          .withUnsupportedProperty( "group-id" )
          .withStringProperty( "group-name", FilterFunctions.NAME )
          .withStringSetProperty( "ip-permission.cidr", FilterSetFunctions.PERMISSION_CIDR )

The resource class is specified (NetworkGroup in the above example) and then filterable items are declared with related details for collection and DB filtering.

For Predicate filtering support, each filter is declared along with the function that is used to extract the value(s) for that filter. If the resource has multiple values for that filter item then one of the *Set* methods is used. The method used should reflect the (filter) type of the resources values for the filter (boolean, date, integer, long, string).

For Criteria filtering support, it is necessary to declare any aliases used (withPersistenceAlias) in addition to declaring each filter and its path (withPersistenceFilter). For non-string values the type must be declared (if Boolean, Date, Integer, Long), else a conversion function must be provided from a String value (a common case here is for Enum values, these can be converted using Guavas Enums.valueOfFunction)

To verify the predicates and criteria a unit test class should be implemented (Groovy recommended), this should extend FilterSupportTest.InstanceTest.

Filter Details

Type Name Supported DB Support Notes
Address (Elastic IP) domain Y - Hardcoded domain standard (no vpc)[1]
  instance-id Y -
  public-ip Y -
  allocation-id - - N/A [1]
  association-id - - N/A [1]
  network-interface-id - - N/A [1]
  network-interface-owner-id - - N/A [1]
  private-ip-address - - N/A [1]
Availability Zone message - - N/A [2]
  region-name - - N/A [2]
  state - - Not compatible [3]
  zone-name Y -
Bundle Tasks bundle-id Y -
  error-code Y -
  error-message Y -
  instance-id Y -
  progress Y -
  s3-bucket Y -
  s3-prefix Y -
  start-time Y -
  state Y -
  update-time Y -
Image architecture Y Y
  block-device-mapping.delete-on-termination Y Y
  block-device-mapping.device-name Y Y
  block-device-mapping.snapshot-id Y Y
  block-device-mapping.volume-size Y Y
  block-device-mapping.volume-type Y Y Hardcoded type standard (no io1)
  description Y Y
  image-id Y Y
  image-type Y Y
  is-public Y Y
  kernel-id Y Y
  manifest-location Y Y
  name Y Y
  owner-alias Y - DB filtering could be possible [4]
  owner-id Y Y Account ID
  platform Y Y As per AWS windows or empty
  product-code Y Y
  product-code.type - - devpay / marketplace, probably not relevant for us [5]
  ramdisk-id Y Y
  root-device-name Y - DB filtering could be possible [4]
  root-device-type Y - DB filtering could be possible [4]
  state Y Y
  state-reason-code - - N/A [2]
  state-reason-message - - N/A [2]
  tag-key Y -
  tag-value Y -
  tag:key Y -
  virtualization-type - - Differ from AWS here [5]
  hypervisor - - Differ from AWS here [5]
Instance architecture Y - DB filtering could be possible [4]
  availability-zone Y Y
  block-device-mapping.attach-time Y - DB filtering could be possible [4]
  block-device-mapping.delete-on-termination Y - DB filtering could be possible [4]
  block-device-mapping.device-name Y - DB filtering could be possible [4]
  block-device-mapping.status Y - DB filtering could be possible [4]
  block-device-mapping.volume-id Y - DB filtering could be possible [4]
  client-token - - N/A [2]
  dns-name Y Y Public name
  group-id - - N/A [2]
  group-name Y - DB filtering could be possible [4]
  image-id Y - DB filtering could be possible [4]
  instance-id Y Y
  instance-lifecycle Y Y Hardcoded with no value (no spot) [6]
  instance-state-code Y - DB filtering could be possible [4]
  instance-state-name Y - DB filtering could be possible [4]
  instance-type Y - DB filtering could be possible [4]
  instance.group-id - - N/A [1]
  instance.group-name - - N/A [1]
  ip-address Y Y Public IP
  kernel-id Y - DB filtering could be possible [4]
  key-name Y - DB filtering could be possible (full key in DB) [4]
  launch-index Y Y DB no wildcard [7]
  launch-time Y Y DB no wildcard [7]
  monitoring-state - - N/A [2]
  owner-id Y Y Account ID
  placement-group-name - - N/A [2]
  platform Y Y
  private-dns-name Y Y
  private-ip-address Y Y
  product-code Y - DB filtering could be possible [4]
  product-code.type - - N/A [2]
  ramdisk-id Y - DB filtering could be possible (full key in DB) [4]
  reason Y - DB filtering could be possible [4]
  requester-id - - N/A [2]
  reservation-id Y Y
  root-device-name - - TODO
  root-device-type - - TODO
  source-dest-check - - TODO
  spot-instance-request-id - - N/A [6]
  state-reason-code - - TODO
  state-reason-message - - TODO
  subnet-id - - N/A [1]
  tag-key Y -
  tag-value Y -
  tag:key Y -
  virtualization-type - - TODO
  vpc-id - - N/A [1]
  hypervisor - - TODO
  network-interface.description - - N/A [1]
  network-interface.subnet-id - - N/A [1]
  network-interface.vpc-id - - N/A [1]
  network-interface.network-interface.id - - N/A [1]
  network-interface.owner-id - - N/A [1]
  network-interface.availability-zone - - N/A [1]
  network-interface.requester-id - - N/A [1]
  network-interface.requester-managed - - N/A [1]
  network-interface.status - - N/A [1]
  network-interface.mac-address - - N/A [1]
  network-interface-private-dns-name - - N/A [1]
  network-interface.source-destination-check - - N/A [1]
  network-interface.group-id - - N/A [1]
  network-interface.group-name - - N/A [1]
  network-interface.attachment.attachment-id - - N/A [1]
  network-interface.attachment.instance-id - - N/A [1]
  network-interface.attachment.instance-owner-id - - N/A [1]
  network-interface.addresses.private-ip-address - - N/A [1]
  network-interface.attachment.device-index - - N/A [1]
  network-interface.attachment.status - - N/A [1]
  network-interface.attachment.attach-time - - N/A [1]
  network-interface.attachment.delete-on-termination - - N/A [1]
  network-interface.addresses.primary - - N/A [1]
  network-interface.addresses.association.public-ip - - N/A [1]
  network-interface.addresses.association.ip-owner-id - - N/A [1]
  association.public-ip - - N/A [1]
  association.ip-owner-id - - N/A [1]
  association.allocation-id - - N/A [1]
  association.association-id - - N/A [1]
Key Pair fingerprint Y Y
  key-name Y Y
Region endpoint Y -
  region-name Y -
Security Group description Y Y
  group-id - - N/A [2]
  group-name Y Y
  ip-permission.cidr Y -
  ip-permission.from-port Y Y
  ip-permission.group-name Y -
  ip-permission.protocol Y Y
  ip-permission.to-port Y Y
  ip-permission.user-id Y -
  owner-id Y Y Account ID
  tag-key Y -
  tag-value Y -
  tag:key Y -
Snapshot description Y Y
  owner-alias Y - DB filtering could be possible [4]
  owner-id Y Y Account ID
  progress Y Y
  snapshot-id Y Y
  start-time Y Y DB no wildcard [7]
  status Y - DB filtering could be possible [4]
  tag-key Y -
  tag-value Y -
  tag:key Y -
  volume-id Y Y
  volume-size Y Y DB no wildcard [7]
Tag key Y Y
  resource-id Y Y
  resource-type Y Y
  value Y Y
Volume attachment.attach-time Y -
  attachment.delete-on-termination Y -
  attachment.device Y -
  attachment.instance-id Y -
  attachment.status Y -
  availability-zone Y Y
  create-time Y Y DB no wildcard [7]
  size Y Y DB no wildcard [7]
  snapshot-id Y Y
  status Y - DB filtering could be possible [4]
  tag-key Y -
  tag-value Y -
  tag:key Y -
  volume-id Y Y
  volume-type Y Y Hardcoded type standard (no io1)
  1. a b c d e f g h i j k l m n o p q r s t u v w x y z ba bb bc bd be bf bg bh bi bj bk bl bm Eucalyptus does not support VPC
  2. a b c d e f g h i j k Eucalyptus does not support this field
  3. ^ Eucalyptus uses this field for other purposes
  4. a b c d e f g h i j k l m n o p q r s t u v DB filtering could be possible
  5. a b c Investigation required
  6. ^ Eucalyptus does not support spot instances
  7. a b c d e f

Configuration

A new configuration property is added to control the permitted number of tags per resource (default 10): 

  tagging.max_tags_per_resource = 10

Upgrade

No upgrade impact noted.

Packaging

No specific packaging requirements.

Documentation

No specific documentation items noted.

Security

No specific security concerns are noted for this design.

Testing

Testing should cover SOAP and Query APIs

References

tag:rls-3.3
Add a custom footer
  • Contact Info
Clone this wiki locally