Internet Engineering Task Force E.H. Haleplidis Internet-Draft University of Patras Intended status: Informational O.C. Cherkaoui Expires: October 27, 2013 University of Quebec in Montreal S.H. Hares Huawei W.M.W. Wang Zhejiang Gongshang University April 25, 2013 Forwarding and Control Element Separation (ForCES) OpenFlow Model Library draft-haleplidis-forces-openflow-lib-03 Abstract This document describes the OpenFlow switch in Logical Function Blocks (LFBs) used in the Forwarding and Control Element Separation (ForCES). The LFB classes are defined according to the ForCES Forwading Element (FE) model and ForCES protocol specifications. The library includes the descriptions of the OpenFlow LFBs and the XML definitions. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on October 27, 2013. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents Haleplidis, et al. Expires October 27, 2013 [Page 1] Internet-Draft ForCES OpenFlow library April 2013 (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1. ForCES . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2. OpenFlow . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Terminology and Conventions . . . . . . . . . . . . . . . . . 5 2.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 2.2. Definitions . . . . . . . . . . . . . . . . . . . . . . . 5 3. OpenFlow ForCES library . . . . . . . . . . . . . . . . . . . 8 3.1. OpenFlow Specification . . . . . . . . . . . . . . . . . 8 3.2. ForCES-based OpenFlow Specification . . . . . . . . . . . 10 4. OpenFlow Base Types . . . . . . . . . . . . . . . . . . . . . 18 4.1. Data Types . . . . . . . . . . . . . . . . . . . . . . . 18 4.1.1. Atomic . . . . . . . . . . . . . . . . . . . . . . . 18 4.1.2. Compound Struct . . . . . . . . . . . . . . . . . . . 19 4.1.3. Compound Array . . . . . . . . . . . . . . . . . . . 21 4.2. Frame Types . . . . . . . . . . . . . . . . . . . . . . . 21 4.3. MetaData Types . . . . . . . . . . . . . . . . . . . . . 21 5. OpenFlow LFBs . . . . . . . . . . . . . . . . . . . . . . . . 23 5.1. OpenFlowSwitch . . . . . . . . . . . . . . . . . . . . . 23 5.1.1. Data Handling . . . . . . . . . . . . . . . . . . . . 23 5.1.2. Components . . . . . . . . . . . . . . . . . . . . . 23 5.1.3. Capabilities . . . . . . . . . . . . . . . . . . . . 24 5.1.4. Events . . . . . . . . . . . . . . . . . . . . . . . 24 5.2. OFFlowTables . . . . . . . . . . . . . . . . . . . . . . 25 5.2.1. Data Handling . . . . . . . . . . . . . . . . . . . . 25 5.2.2. Components . . . . . . . . . . . . . . . . . . . . . 26 5.2.3. Capabilities . . . . . . . . . . . . . . . . . . . . 27 5.2.4. Events . . . . . . . . . . . . . . . . . . . . . . . 27 5.3. OFGroupTable . . . . . . . . . . . . . . . . . . . . . . 27 5.3.1. Data Handling . . . . . . . . . . . . . . . . . . . . 27 5.3.2. Components . . . . . . . . . . . . . . . . . . . . . 28 5.3.3. Capabilities . . . . . . . . . . . . . . . . . . . . 28 5.3.4. Events . . . . . . . . . . . . . . . . . . . . . . . 28 5.4. OFPort . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.4.1. Data Handling . . . . . . . . . . . . . . . . . . . . 28 5.4.2. Components . . . . . . . . . . . . . . . . . . . . . 28 5.4.3. Capabilities . . . . . . . . . . . . . . . . . . . . 29 5.4.4. Events . . . . . . . . . . . . . . . . . . . . . . . 29 5.5. OFQueue . . . . . . . . . . . . . . . . . . . . . . . . . 29 Haleplidis, et al. Expires October 27, 2013 [Page 2] Internet-Draft ForCES OpenFlow library April 2013 5.5.1. Data Handling . . . . . . . . . . . . . . . . . . . . 30 5.5.2. Components . . . . . . . . . . . . . . . . . . . . . 30 5.5.3. Capabilities . . . . . . . . . . . . . . . . . . . . 30 5.5.4. Events . . . . . . . . . . . . . . . . . . . . . . . 30 5.6. OFRedirectIn . . . . . . . . . . . . . . . . . . . . . . 30 5.6.1. Data Handling . . . . . . . . . . . . . . . . . . . . 30 5.6.2. Components . . . . . . . . . . . . . . . . . . . . . 31 5.6.3. Capabilities . . . . . . . . . . . . . . . . . . . . 31 5.6.4. Events . . . . . . . . . . . . . . . . . . . . . . . 31 5.7. OFRedirectOut . . . . . . . . . . . . . . . . . . . . . . 31 5.7.1. Data Handling . . . . . . . . . . . . . . . . . . . . 31 5.7.2. Components . . . . . . . . . . . . . . . . . . . . . 31 5.7.3. Capabilities . . . . . . . . . . . . . . . . . . . . 31 5.7.4. Events . . . . . . . . . . . . . . . . . . . . . . . 31 5.8. OFActionSet . . . . . . . . . . . . . . . . . . . . . . . 31 5.8.1. Data Handling . . . . . . . . . . . . . . . . . . . . 32 5.8.2. Components . . . . . . . . . . . . . . . . . . . . . 32 5.8.3. Capabilities . . . . . . . . . . . . . . . . . . . . 32 5.8.4. Events . . . . . . . . . . . . . . . . . . . . . . . 32 5.9. OFBuffer . . . . . . . . . . . . . . . . . . . . . . . . 32 5.9.1. Data Handling . . . . . . . . . . . . . . . . . . . . 32 5.9.2. Components . . . . . . . . . . . . . . . . . . . . . 33 5.9.3. Capabilities . . . . . . . . . . . . . . . . . . . . 33 5.9.4. Events . . . . . . . . . . . . . . . . . . . . . . . 33 5.10. OFAction . . . . . . . . . . . . . . . . . . . . . . . . 33 5.10.1. Data Handling . . . . . . . . . . . . . . . . . . . 33 5.10.2. Components . . . . . . . . . . . . . . . . . . . . . 34 5.10.3. Capabilities . . . . . . . . . . . . . . . . . . . . 34 5.10.4. Events . . . . . . . . . . . . . . . . . . . . . . . 34 5.11. OFActionLFBs . . . . . . . . . . . . . . . . . . . . . . 34 5.11.1. OFActionOutput . . . . . . . . . . . . . . . . . . . 34 5.11.2. OFActionSetVLANVID . . . . . . . . . . . . . . . . . 35 5.11.3. OFActionSetVLANPriority . . . . . . . . . . . . . . 35 5.11.4. OFActionSetMACSource . . . . . . . . . . . . . . . . 35 5.11.5. OFActionSetMACDestination . . . . . . . . . . . . . 35 5.11.6. OFActionSetIPSource . . . . . . . . . . . . . . . . 36 5.11.7. OFActionSetIPDestination . . . . . . . . . . . . . . 36 5.11.8. OFActionSetIPTOS . . . . . . . . . . . . . . . . . . 36 5.11.9. OFActionSetIPECN . . . . . . . . . . . . . . . . . . 36 5.11.10. OFActionSetTCPSource . . . . . . . . . . . . . . . . 37 5.11.11. OFActionSetTCPDestination . . . . . . . . . . . . . 37 5.11.12. OFActionCopyTTLOut . . . . . . . . . . . . . . . . . 37 5.11.13. OFActionCopyTTLIn . . . . . . . . . . . . . . . . . 37 5.11.14. OFActionSetMPLSLabel . . . . . . . . . . . . . . . . 38 5.11.15. OFActionSetMPLSTC . . . . . . . . . . . . . . . . . 38 5.11.16. OFActionSetMPLSTTL . . . . . . . . . . . . . . . . . 38 5.11.17. OFActionDecrementMPLSTTL . . . . . . . . . . . . . . 38 5.11.18. OFActionPushVLan . . . . . . . . . . . . . . . . . . 39 Haleplidis, et al. Expires October 27, 2013 [Page 3] Internet-Draft ForCES OpenFlow library April 2013 5.11.19. OFActionPopVLAN . . . . . . . . . . . . . . . . . . 39 5.11.20. OFPushMPLSOFAction . . . . . . . . . . . . . . . . . 39 5.11.21. OFPopMPLSOFAction . . . . . . . . . . . . . . . . . 39 5.11.22. OFSetQueueOFAction . . . . . . . . . . . . . . . . . 40 5.11.23. OFSetIPTTLOFAction . . . . . . . . . . . . . . . . . 40 5.11.24. OFDecrementIPTTLOFAction . . . . . . . . . . . . . . 40 5.11.25. OFExperimenterOFAction . . . . . . . . . . . . . . . 40 6. XML for OpenFlow library . . . . . . . . . . . . . . . . . . 41 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 94 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 94 9. Security Considerations . . . . . . . . . . . . . . . . . . . 94 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 94 10.1. Normative References . . . . . . . . . . . . . . . . . . 94 10.2. Informative References . . . . . . . . . . . . . . . . . 95 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 96 1. Introduction The purpose of this document is to create a library of Logical Functional Blocks that are necessary to describe an OpenFlow switch using the ForCES model. This includes DataTypes, MetaData and of course the LFBs. Readers of this document can get a better understanding of what are the internal parts of an OpenFlow switch in a more formal approach. Additionally having a ForCES-defined OpenFlow switch allows developers to build a ForCES FE that is based on the OF switch model or even a middleware so that ForCES-implemented OpenFlow switches may be controlled by an OpenFlow controller, or a ForCES Control Element (CE) may control OpenFlow switches 1.1. ForCES ForCES [RFC3746], focuses on the communication and model necessary to separate control-plane functionality such as routing protocols, signaling protocols, and admission control, from data-forwarding- plane per-packet activities, such as packet forwarding, queuing, and header editing. The modeling of FEs is based on an abstraction using distinct Logical Functional Blocks (LFBs), which are interconnected in a directed graph, and receive, process, modify, and transmit packets along with metadata. An LFB is a block of encapsulated fine-grained operation of the forwarding plane. The ForCES model [RFC5812] additionally includes both a capability and a state model. One of the advantages of the ForCES Model is that it is independent of the actual implementation of the FE; it only provides a view of its capabilities and state that can be acted upon using the ForCES protocol. It is Haleplidis, et al. Expires October 27, 2013 [Page 4] Internet-Draft ForCES OpenFlow library April 2013 left to the forwarding plane developers to define how the FE functionality is represented using the model. As the OpenFlow library document requires complex metadata, the [I-D.haleplidis-forces-model-extension] is used. The ForCES protocol [RFC5810] was developed to allow the CEs to determine the capabilities of each FE expressed by the FE model, to add and remove entries, parameters, query for statistics, and register for and receive events in a scalable fashion over secure and reliable means. The strength of the ForCES protocol stems from the fact that it is agnostic of the model, as a CE can control any Forwarding Element described with the ForCES model. 1.2. OpenFlow OpenFlow [OpenFlowSpec1.1] is conceptually similar to ForCES on separating the control and forwarding plane. It provides a protocol that mediates between the controller and the switch. Unlike ForCES, the OpenFlow switch is statically defined to deal with flows and the protocol is aware of the flow components. An OpenFlow Switch consists of one or more flow tables, a group table that performs packet lookups and forwarding, and an OpenFlow channel to an external controller. A flow table is consisted of flow entries, each containing a set of match fields to match against packets, counters and instructions. The controller manages the switch via the OpenFlow protocol. Using this protocol, the controller can add, update, and delete flow and group entries. 2. Terminology and Conventions 2.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 2.2. Definitions This document follows the terminology defined by ForCES related documents of RFC3654, RFC3746, RFC5810,RFC5811,RFC5812,RFC5812. The definitions are repeated below for clarity. Also additional definitions from the OpenFlow specification 1.1 [OpenFlowSpec1.1] are also included. Haleplidis, et al. Expires October 27, 2013 [Page 5] Internet-Draft ForCES OpenFlow library April 2013 Control Element (CE) - A logical entity that implements the ForCES protocol and uses it to instruct one or more FEs on how to process packets. CEs handle functionality such as the execution of control and signaling protocols. Forwarding Element (FE) - A logical entity that implements the ForCES protocol. FEs use the underlying hardware to provide per- packet processing and handling as directed/controlled by one or more CEs via the ForCES protocol. LFB (Logical Functional Block) - The basic building block that is operated on by the ForCES protocol. The LFB is a well defined, logically separable functional block that resides in an FE and is controlled by the CE via the ForCES protocol. The LFB may reside at the FE's datapath and process packets or may be purely an FE control or configuration entity that is operated on by the CE. Note that the LFB is a functionally accurate abstraction of the FE's processing capabilities, but not a hardware-accurate representation of the FE implementation. LFB Class and LFB Instance - LFBs are categorized by LFB Classes. An LFB Instance represents an LFB Class (or Type) existence. There may be multiple instances of the same LFB Class (or Type) in an FE. An LFB Class is represented by an LFB Class ID, and an LFB Instance is represented by an LFB Instance ID. As a result, an LFB Class ID associated with an LFB Instance ID uniquely specifies an LFB existence. LFB Metadata - Metadata is used to communicate per-packet state from one LFB to another, but is not sent across the network. The FE model defines how such metadata is identified, produced, and consumed by the LFBs. It defines the functionality but not how metadata is encoded within an implementation. LFB Components - Operational parameters of the LFBs that must be visible to the CEs are conceptualized in the FE model as the LFB components. The LFB components include, for example, flags, single-parameter arguments, complex arguments, and tables that the CE can read and/or write via the ForCES protocol (see below). ForCES Protocol - While there may be multiple protocols used within the overall ForCES architecture, the term "ForCES protocol" and "protocol" refer to the "Fp" reference points in the ForCES framework in [RFC3746]. This protocol does not apply to CE-to-CE communication, FE-to-FE communication, or to communication between FE and CE managers. Basically, the ForCES protocol works in a master-slave mode in which FEs are slaves and CEs are masters. Haleplidis, et al. Expires October 27, 2013 [Page 6] Internet-Draft ForCES OpenFlow library April 2013 ForCES Protocol Transport Mapping Layer (ForCES TML) - A layer in ForCES protocol architecture that uses the capabilities of existing transport protocols to specifically address protocol message transportation issues, such as how the protocol messages are mapped to different transport media (like TCP, IP, ATM, Ethernet, etc.), and how to achieve and implement reliability, multicast, ordering, etc. The ForCES TML specifications are detailed in separate ForCES documents, one for each TML. Match Field - a field against which a packet is matched, including packet headers, the ingress port, and the metadata value. Action - an operation that forwards the packet to a port or modifies the packet, such as decrementing the TTL field. Actions may be specified as part of the instruction set associated with a flow entry or in an action bucket associated with a group entry. Flow entry - an element in a flow table used to match and process packets. It contains a set of match fields for matching packets, a set of counters to track packets, and a set of instructions to apply. Instruction - an operation that either contains a set of actions to add to the action set, contains a list of actions to apply immediately to the packet, or modifies pipeline processing. Flow Table - A stage of the pipeline, contains flow entries. OpenFlow pipeline - the set of linked flow tables that provide matching, forwarding, and packet modifications in an OpenFlow switch. Groups - a list of action buckets and some means of choosing one or more of those buckets to apply on a per-packet basis. Group Entry - an element in a group table. It contains a group identifier to distinguish groups, a group type to define the type of the group, a set of counters to track packets, and a set of action buckets. Action Bucket - a set of actions and associated parameters, defined for groups. Action Set - a set of actions associated with the packet that are accumulated while the packet is processed by each table and that are executed when the instruction set instructs the packet to exit the processing pipeline. Haleplidis, et al. Expires October 27, 2013 [Page 7] Internet-Draft ForCES OpenFlow library April 2013 Ports - where packets enter and exit the OpenFlow pipeline. May be a physical port, a logical port defined by the switch, or a reserved port defined by the specification. 3. OpenFlow ForCES library 3.1. OpenFlow Specification An OpenFlow switch as described in the OpenFlow Specification document [OpenFlowSpec1.1] appears in Figure 1 +---------+ +---------+ +---------+ | | | | | | | Port | | Group | | Port | | | | Table | | | +---------+ +---------+ +---------+ | M1 /\ | M2 | \/ | +---------+ +---------+ +---------+ +---------+ | | M1 | | M1 | | M2 | Execute | | Flow | ---> | Flow | -->...---> | Flow | ---> | Action | | Table 0 | M2 | Table 1 | M2 | Table N | | Set | +---------+ M3 +---------+ M3 +---------+ +---------+ Legend M1: Ingress Port M2: Action Set{} M3: Metadata Figure 1: OpenFlow switch datapath A packet enters the switch through a Port and is passed on the first Flow Table along with the Ingress Port as a Metadata (M1). Additionally each frame carries around a list of actions, called Action Set (M2), which have initially no actions in it. The Action Set will be executed at the end of the DataPath in the Execute Action Set block. After the first Flow Table another metadata called Metadata (M3) also accompanies the packet. This data inside the metadata maybe written by the Flow Tables when the Write Metadata instruction is applied. Each Flow Table performs a match based on certain fields (e.g IP Source Address or Source MAC Address) and then perform a specific instruction if there is a match. If no match occurs, the frame is processed based on the Flow Table's configuration. The choices are either: Haleplidis, et al. Expires October 27, 2013 [Page 8] Internet-Draft ForCES OpenFlow library April 2013 a. Forward to the OpenFlow controller b. Send to the next flow table c. Drop the frame The list of instructions a Flow Table may perform upon a match are: o Apply a List of actions o Clear the Action Set o Write actions on the action set o Write Metadata o Go to Flow Table (allows a FlowTable X to send the packet and metadata to any FlowTable Y, provided that X>Y) In OpenFlow there are two types of action executions which are independent of each other. The first one refered to as action list and is programmed into the flow table to be executed immediately within the packet pipeline upon a match on a flow table. The second one gets executed at the end of the pipeline in the execute action set. The second type of actions is collected in a metadata refered to as Action Set during the datapath processing with hte Write Actions instruction. The type of actions the Flow Table can perform or write in the Action Set is: o Setting of a field (e.g. IP address, MAC address) o Push or Pop tags (VLAN, MPLS) o Copy TTL inwards or outwards o Decrease TTLs o Output the packet to ports (a copy of the original packet will be sent to the port) o Apply QoS to a packet o Apply the packet to a group (a copy of the original packet will be sent to the group) Haleplidis, et al. Expires October 27, 2013 [Page 9] Internet-Draft ForCES OpenFlow library April 2013 Additionally a Flow Table may drop the packet. The drop is implicit based on the Flow Table's configuration (e.g. when there are no more instructions). An Action Set MUST contains a maximum of one action of each of the following class of types which MUST be executed in the order specified below regardless of the order they were added to the Action Set. The output action in the action set is executed last. If both an output action and a group action are specified in an action set, the output action is ignored and the group action takes precedence. If no output action and no group action were specified in an action set, the packet is dropped. 1. Copy TTL inwards 2. Pop a tag (maximum one of VLAN tag, MPLS tag) 3. Push a tags (maximum one of VLAN tag, MPLS tag) 4. Copy TTL outwards 5. Decrease TTL 6. Setting of a field (maximum one of set IP address, set VLAN ID, etc...) 7. Apply QoS to a packet 8. Apply the packet to a group 9. Output the packet The Group Table contains a set of Group Entries, each of which contains a set of Action Buckets, each of which contains a set of actions which can be applied to a group of packets that don't have the same set of matching fields. This alleviates the problem of having to set up the same set of actions in flow tables for different set of matching fields by having these set of actions in one place only. 3.2. ForCES-based OpenFlow Specification ForCES models FEs using LFBs, fine-grained operations of the forwarding plane. It is logical to have at least the following LFB classes: 1. OFPort Haleplidis, et al. Expires October 27, 2013 [Page 10] Internet-Draft ForCES OpenFlow library April 2013 2. OFFlowTables 3. OFGroupTable 4. OFActionSet 5. OFActions 6. OFQueue Additionally packets may be sent to the controller or the controller may send packets to the switch to be put on the datapath. RedirectIn and RedirectOut are two LFBs defined in the Base LFB Library [I-D.ietf-forces-lfb-lib]. However as some more metadata are required for the OpenFlow switch, the OFRedirectIn and OFRedirectOut will be defined by extending the initial LFBs. The controller may request the switch, if the switch is capable, instead of redirecting packets to the controller instead to buffer them and send a specific amount of the packet to the controller. For this task we introduce the OFBuffer LFB. While it may seem that having multiple OFFlowTables instances to represent each Flow Table in the OpenFlow, the authors decided to model the OFFlowTables to contain all the Flow Tables in one instance of the OFFlowTables. On of the OFFlowTables's components is an array of Flow Tables entries and each entry contains its own Flow Entries, Flow Table Counter and Miss Behaviour. The index of the Flow Tables entry represents the Flow Table ID. The rationale behind such a decision is the simplification of the model. With multiple ActionLFBs and multiple FlowTables, the resulting connection graph between Flow Tables and FlowTables and ActionLFBs would be very complex. Addionally this simplifies also the metadata Metadata which is invisible to the model as it only passed between Flow Tables.Figure 2 shows an example of how the OFFlowTables is internally. | From /\To OFOutput /\To Group /\To Redirect | OFPorts | Action | Table | Out +----|---------------|--------------------|--------------|----------+ | | | | | | | | +-----------+--------------------+--------------+ | | \/ | | | | | | +--------+ +--------+ +--------+ +--------+ | | | | --> | | | | | | | | | Flow | | Flow |--+---+--> | Flow |-+--+->| Flow | | | | Table 0|--+ | Table 1| | /\ | Table 2| /\ /\ | Table N| | | +--------+ | +--------+ | | +--------+ | | +--------+ | | /\ /\ | /\ | | | /\ | | | /\ | Haleplidis, et al. Expires October 27, 2013 [Page 11] Internet-Draft ForCES OpenFlow library April 2013 | | | | | | +----------|---|----+ | | | | | | | | | | | | | | | | | | +----|---+---------+------|---+-------+ | | | | | | | | | | | +----------+-------------+------+-------------------+ | | | | | +---|----------------------------|----------------------------------+ |From | To/From |Redirect In \/OFActions Figure 2: FlowTable Internal Figure 3 depicts what an OFFlowTables with its own set of ActionLFBs would look like. Figure 4 depicts how the OFFlowTables work with a shared set of ActionLFBs with the OFGroupTable LFB look like. +------+ M1,M2,M3 M1,M2,M3 +---------+ | Push | P(2) P(1) | Set | | Vlan |<----------+ +----------->| IP | |Header| | | | Address | +------+ | | +---------+ | | | | | P(3),M1 +--------+ P(2),M1 | +------------>| |<------------+ | | ---------------------->| OFFlow |----------------------> P | Tables | P(3) | | +---------+ P,M1,M2 | | | | <------- | | Legend: |Decrement| | | P: Packet | IP TTL | +--------+ M1: PacketID | | P(1),M1 /\ M2: LFBClassID +---------+ -------------+ M3: ActionIndex Figure 3: ForCES FlowTable with each own set of ActionLFBs Haleplidis, et al. Expires October 27, 2013 [Page 12] Internet-Draft ForCES OpenFlow library April 2013 A packet P enters the OFFlowTables from an OFPortLFB. If a match occurs for the packet within the OFFlowTables and if the instruction for that match is an "Apply Action List" then the actions must be performed immediately and in the order specified in the action list. Alongside the packet, some metadata are passed as well. M1, the PacketID required by the OFFlowTables to identify the packet and continue exectution from where it stopped when it returns, M2 the LFBClassID so that the ActionLFB knows the ClassID of the LFB to return the packet and M3 the ActionIndex required in some ActionLFBs to determine parameters for the action. For example in Figure 3 let's assume that the action list includes the following four actions: 1. Decrement IP TTL 2. Set IP Address 3. Push VLAN header 4. Final Action The packet P will be first sent to the Decrement IP TTL Action LFB. Upon completion it will be returned as P1 to the OFFlowTables and then will be sent to the Set IP Address LFB. Upon return as P2 it will be sent to the Push VLAN Header and returned finally as P3 to the Flow Table LFB. Then depending upon the final action the packet may: o A copy of P3 will be sent to a Port LFB if it is an output action and the action set will be executed. o A copy of P3 will be sent to the Group LFB if it is an group action and the action set will be executed. o Remain in the OFFlowTables and checked for a match in the Flow Table specified by a Goto action if it is a goto action. +---------+ P(1),M1,M2,M3 | Set IP |<-----------------+ | Address | | | |--------------+ | +---------+ P(2),M1 | | | | | | P(2),M1 \/ | +------+ M2,M3 +---------+ | Push |<-----------| Group | P3 | Vlan | | Table |-------> Haleplidis, et al. Expires October 27, 2013 [Page 13] Internet-Draft ForCES OpenFlow library April 2013 |Header|----------->| | +------+ P(3),M1 +---------+ P /\ | P(1) /\ M1 | | | M2 | |P(1),M1 | M3 | \/ | +--------+ | Legend: P | |----------------+ P: Packet ----->| OFFlow | M1: PacketID | Tables |----------------> M2: LFBClassID +--------+ P(1) M3: ActionIndex Figure 4: ForCES FlowTables and GroupTable with a common ActionLFB A packet enters the OFFlowTables from an OFPortLFB. If a match occurs for the packet within the OFFlowTables and if the instruction for that match is an "Apply Action List" then the actions must be performed immediately and in the order specified in the action list. For example in Figure 4 let's assume that the action list includes the following three actions: 1. Push VLAN Header 2. Group 3. Output The packet P will be first sent to the Push VLAN header Action LFB and upon completion will be returned as P1 to the OFFlowTables. Then a copy of P1 will be sent to the Group Table LFB and then finally a copy of P1 will be sent to the Port LFB for output and the action set will be executed. Let's assume that the following actions will be executed for P1 in the action bucket in the Group Table. 1. Set IP Address 2. Push Vlan Header 3. Output The packet P1 will be sent to the Set IP Address Action LFB and upon completion will be returned as P2 to the Group Table. Then the packet P2 will be sent to the Push Vlan Header Action LFB and be returned as P3 to the Group Table and be sent to the Port LFB for output. Haleplidis, et al. Expires October 27, 2013 [Page 14] Internet-Draft ForCES OpenFlow library April 2013 Regarding the OFActions, when a match occurs in the OFFlowTables or an action bucket must be executed in the OFGroupTable it may contain multiple actions, it seems reasonable to separate each action as an individual LFB that performs that specific action. For every action needed to be executed, the OFFlowTables or the OFGroupTable will send the frame to the appropriate action LFB(s) in the order defined either by the Action Bucket or the instructions in the OFFlowTables's match entry. Once the packet has been processed from an Action LFB, it MUST be returned to the LFB instance that made that call. OFFlowTables and OFGroupTable LFB may either have shared ActionLFBs or separate. Current specified Action LFBs are: Output Actions o OFActionOutput Set Queue Actions o OFActionSetQueue Push/Pop Tag Actions o OFActionPushVLan o OFActionPopVLAN o OFActionPushMPLS o OFActionPopMPLS Set Actions o OFActionSetMACSource o OFActionSetMACDestination o OFActionSetVLANVID o OFActionSetVLANPriority o OFActionSetMPLSLabel o OFActionSetMPLSTC o OFActionSetMPLSTTL Haleplidis, et al. Expires October 27, 2013 [Page 15] Internet-Draft ForCES OpenFlow library April 2013 o OFActionDecrementMPLSTTL o OFActionSetIPSource o OFActionSetIPDestination o OFActionSetIPTOS o OFActionSetIPECN o OFActionSetIPTTL o OFActionDecrementIPTTL o OFActionSetTCPSource o OFActionSetTCPDestination o OFActionCopyTTLOut o OFActionCopyTTLIn Experimenter Actions o OFActionExperimenter Most Action LFBs have data associated with the action, e.g. an IP Address for the SetIPSource or SetIPDestination actions, stored in an array in the LFB. The FlowTable sending the packet needs to send additionally as a metadata an index pointing to the action parameter needed for the execution. Each Action LFB has one group input port that accepts a packet, the LFBClassID of the LFB that sent the packet, so that it can be returned after the action has been performed, and optionally the Action Index. Furthermore, one more metadata is required, the PacketID with which the Flow Table LFB or the Group Table LFB can keep track of a packet's progress. Additionally each Action LFB has one group output port that returns the altered packet to the sender. Since the action LFBs have these ports in common and the ForCES model can support augmentation of LFB classes, similar to inheritance in object oriented programming, an OFActionLFB has been specified from which all Action LFBs are derived from. The Action LFBs can be used also by the OFGroupTable and the OFActionSet using the same input and output port. Additionally each OFFlowTables can output a packet to a specific port through the OFOutputAction LFB. Figure 5 shows an example of a Haleplidis, et al. Expires October 27, 2013 [Page 16] Internet-Draft ForCES OpenFlow library April 2013 topology and how the various LFBs are interconnected. The controller can obtain the topology information by querying the FEObject's LFBTopology. /\ /\ /\ /\ /\ | Out | Out | Out | Out | Out | | | | | +-------+ +-------+ +-------+ +-------+ +-------+ | | | | | | | | | | |OFQueue| |OFQueue| |OFQueue| ... |OFQueue| |OFQueue| | | | | | | | | | | +-------+ +-------+ +-------+ +-------+ +-------+ /\ /\ /\ /\ /\ | | | | | | +--+---------+ | | | | | | +--------+ +--------+ +--------+ +--------+ In | | | |<-- In -->| | | | In -->| OFPort | | OFPort | | OFPort | | OFPort |<-- | | | | ... | | | | +--------+ +--------+ +--------+ +--------+ | /\ | /\ | /\ | /\ | | | | | | | | +--+---|----------+---|-----------------+---|------------+ | | | | | | | | +---------+----+---------------------+---------------+ | | | +----------+ +------------+ | | |------------------> | |To | +---->| OFOutput | | OFRedirect |--> | | | Action | +----------+ | Out |Controller | | | |--->| |--->| | | | +----------+ | OFBuffer | +------------+ | | /\ /\ +--| |<-+ | | | | | +----------+ | +------------+ | | +----------------+ +-| |From | | | | | | OFRedirect |<-- | | | +--------------------------------| In |Controller | | | | | | +------------+ | | | | | | +---------+ \/ | \/ \/ | +-----------| | +--------+ | | OFGroup | | |------------------->| Table | | OFFlow | | | | | Tables | +-------------+ +---------+ | |-->| | /\ /\ +--------+ | OFActionSet | | | /\ | |---+ | Haleplidis, et al. Expires October 27, 2013 [Page 17] Internet-Draft ForCES OpenFlow library April 2013 | +-------------+ | | | | | +------+-------+-------|------+--------------+ | | | | | | | +--------------+----------+---+--------------+ | | | | | | | | | | +--|----------+---|-----------+--|-----------+ | | | | | | | | | | | | | | \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ \/ +--------+ +--------+ +--------+ +--------+ | | | | | | | | |OFAction| |OFAction| |OFAction| |OFAction| | | | | | | | | +--------+ +--------+ +--------+ +--------+ Figure 5: ForCES OpenFlow Switch example LFB connectivity When a packet has finished processing of the Flow Table pipelines, it will be sent along with the ActionSet Metadata to the OFActionSet LFB for processing of the ActionSet. The ActionSet LFB is responsible for performing all actions specified in the ActionSet Metadata in the order specified by the ActionSet requirements in Section 3.1. Buffering packets for OpenFlow PacketIn messages is handled by the OFBuffer LFB. The execution of OpenFlow's PacketOut message which contains a list of actions to be performed on a buffered or a redirected packet in the switch is to be performed internally in the OFFlowTables after receiving the list and the packet from the OFBuffer LFB. OpenFlow 1.1 provides, with the use of the experimenter concept new instruction or action types. In this model, new instruction types can be modeled by expanding the InstructionTypes datatype definition and new action types can be modeled by creating new ActionLFBs. 4. OpenFlow Base Types Some datatypes in this LFB library are imported from Base LFB Library [I-D.ietf-forces-lfb-lib] as they have already been defined there. 4.1. Data Types Data types defined in the OpenFlow library are categorized by types of atomic, compound struct, and compound array. 4.1.1. Atomic Haleplidis, et al. Expires October 27, 2013 [Page 18] Internet-Draft ForCES OpenFlow library April 2013 The following data types are defined as atomic data types in the OpenFlow library: +-----------------------------+---------------------------+---------+ | Data Type Name | Brief Description | Base | | | | Type | +-----------------------------+---------------------------+---------+ | MPLSLabelValue | An MPLS label | uint32 | | | | | | MPLSTrafficClassValues | The MPLS Traffic Class | uchar | | | | | | IPv4ToSbits | TOSBits | uchar | | | | | | ActionType | The possible actions | uint16 | | | | | | InstructionTypes | Instructions supported | uchar | | | | | | FlowTableMissConfigType | Types to configure the | uint32 | | | default behavior of | | | | unmatched packets in a | | | | Flow Table | | | | | | | PacketInTypes | Packet In Types | uchar | | | | | | GroupBucketExecuteType | To determine which Action | uchar | | | Bucket(s) should be | | | | executed (all, select, | | | | indirect, fast failover) | | | | | | | PortNumberType | Port Number values | uint32 | | | | | | QueuePropertyType | Property type for a queue | uint16 | +-----------------------------+---------------------------+---------+ OpenFlow Atomic Types 4.1.2. Compound Struct The following data types are defined as struct data types in the OpenFlow library: +------------------------------+------------------------------------+ | Data Type Name | Brief Description | +------------------------------+------------------------------------+ | SwitchDescriptionType | Fields of the switch description | | | | | WildcardsType | Wildcards for fields | | | | Haleplidis, et al. Expires October 27, 2013 [Page 19] Internet-Draft ForCES OpenFlow library April 2013 | MatchFieldType | A Match Field Type (contains all | | | possible match fields) | | | | | FlowEntry | A Flow entry type | | | | | ActionRowType | An Action Row for the action table | | | | | TableCounterType | Counter per table | | | | | FlowCounterType | Counter per flow | | | | | WriteMetadataTableType | Metadata and mask for the write | | | metadata instruction per row | | | | | GroupCounterType | Counters per group | | | | | BucketCounterType | Counters per bucket | | | | | GroupTableEntry | A Row of the Group Table | | | | | ActionBucket | An Action Bucket | | | | | PortConfigurationType | Types of configuration for the | | | OpenFlow port | | | | | PortStateType | Current State of the port | | | | | PortFeaturesType | Port Features | | | | | PortCounterType | Counter per port | | | | | QueueArrayPropertiesType | Type Definition for property | | | | | QueueCounterType | Counters per queue | | | | | ActionRowType | Row type for the ActionSet and | | | Action List. Contains type of | | | action, LFB Class and Instance ID | | | and ActionTableIndex. | +------------------------------+------------------------------------+ OpenFlow Struct Types Haleplidis, et al. Expires October 27, 2013 [Page 20] Internet-Draft ForCES OpenFlow library April 2013 4.1.3. Compound Array The following data types are defined as an array data type in the OpenFlow library +----------------+------------------------------------------------+ | Data Type Name | Brief Description | +----------------+------------------------------------------------+ | Actions | Actions to perform. An Array of ActionRowTypes | +----------------+------------------------------------------------+ OpenFlow Array Types 4.2. Frame Types No additional frame types are defined in this library 4.3. MetaData Types The following metadata are defined in the OpenFlow type library: +-----------------------+----------------+----------+---------------+ | MetaData Name | Brief | MetaData | Metadata Type | | | Description | ID | | +-----------------------+----------------+----------+---------------+ | IngressPort | The Ingress | 1024 | uint32 | | | port the | | | | | packet has | | | | | arrived from. | | | | | | | | | InPhyPort | The Port Index | 1025 | uint32 | | | of the | | | | | Physical | | | | | interface the | | | | | frame entered | | | | | the switch | | | | | | | | | PacketID | The PacketID | 1026 | uint32 | | | metadata is | | | | | used to | | | | | uniquelly | | | | | identify a | | | | | packet within | | | | | the Flow Table | | | | | or the Group | | | | | Table to | | | | | continue | | | | | processing it | | | Haleplidis, et al. Expires October 27, 2013 [Page 21] Internet-Draft ForCES OpenFlow library April 2013 | | after it has | | | | | been returned | | | | | from an | | | | | OFActionLFB. | | | | | | | | | ActionIndex | The Action | 1027 | uint32 | | | Index metadata | | | | | is used to | | | | | point the row | | | | | in the array | | | | | in an Action | | | | | LFB | | | | | | | | | GroupIndex | The Group | 1028 | uint32 | | | index metadata | | | | | is used to | | | | | point to the | | | | | row of the | | | | | array in an | | | | | Group LFB | | | | | | | | | LFBClassIDMetadata | The LFBClassID | 1029 | uint32 | | | | | | | LFBInstanceIDMetadata | The LFB | 1030 | uint32 | | | Instance ID | | | | | | | | | QueueID | The Queue ID | 1031 | uint32 | | | the packet | | | | | should be sent | | | | | to | | | | | | | | | BufferID | The Buffer ID | 1032 | uint32 | | | of a stored | | | | | packet in the | | | | | switch | | | | | requried for | | | | | the PacketOut | | | | | message | | | | | | | | | RedirectReason | The reason the | 1033 | uchar | | | packet was | | | | | redirected to | | | | | the controller | | | | | | | | | FlowTableID | The FlowTable | 1034 | uchar | | | ID the packet | | | | | was sent to | | | | | the controller | | | Haleplidis, et al. Expires October 27, 2013 [Page 22] Internet-Draft ForCES OpenFlow library April 2013 | | from | | | | | | | | | ActionList | The Action | 1035 | Array of | | | List that may | | ActionRowType | | | come along | | | | | with the | | | | | packet in a | | | | | PacketOut | | | | | message | | | | | | | | | ActionSet | The Action Set | 1036 | Array of | | | to be executed | | ActionRowType | | | at the end of | | | | | the pipeline | | | +-----------------------+----------------+----------+---------------+ OpenFlow Metadata Types 5. OpenFlow LFBs 5.1. OpenFlowSwitch Similar to the concept of the FEProtocol LFB and the FEObject LFB, the OpenFlowSwitchLFB contains information and configuration parameters regarding the functionality of the switch but play no role in the datapath processing. Therefore there are no input or output ports. 5.1.1. Data Handling This LFB does not handle any data. 5.1.2. Components The DatapathID component, a unsigned integer of 64 bits, uniquely identifies a datapath. The lower 48 bits are intended for the switch MAC address, while the top 16 bits are up to the implementer. The HandleFragments component, a Boolean, defines what the switch does with fragments. If true the switch will drop fragments. If false there is no special handling. The ReassembleFragments component, a Boolean, defines if the switch will reassemble fragments. The InvalidTTLtoController component, a Boolean, defines whether the switch will send packets with invalid TTL to the controller. Haleplidis, et al. Expires October 27, 2013 [Page 23] Internet-Draft ForCES OpenFlow library April 2013 The SwitchDescription component, a structure, contains the following information about the switch: o Manufacturer description o Hardware description o Software description o Serial Number o Human readable description of datapath Lastly the Ports component is an array which contains in its rows, all the port numbers. 5.1.3. Capabilities The following capabilities have been defined for the OpenFlowSwitch LFB An assortment of Boolean type capabilities to define: o FlowStatistics. If the switch keeps flow statistics o TableStatistics. If the switch keep table statistics o PortStatistics. If the switch keep port statistics o GroupStatistics. If the switch keep group statistics o IPReassembly. If the switch can reassemble IP fragments o QueueStats. If the switch keeps queue statistics o ARPMatchIP. If the switch matches IP addresses in ARP packets The TablesSupported capability, an unsigned integer of 8 bits, defines the number of tables supported by the switch, each of which can have a different set of supported wildcard bits and number of entries. Additionally the another capability, the ActionSupported, defines the supported actions for the switch. 5.1.4. Events Haleplidis, et al. Expires October 27, 2013 [Page 24] Internet-Draft ForCES OpenFlow library April 2013 Three events have been specified regarding the ports. The first event will be triggered when a new port is added to the switch, the second when a port has been removed from the switch and the third when a port has been modified 5.2. OFFlowTables An LFB that houses all OpenFlow Flow Tables residing in the switch. 5.2.1. Data Handling The OFFlowTables describes the process of selecting packets and classify them into specific flows based on specific match fields assigned by the controller. The LFB is expected to receive all types of Ethernet packets through a group input named InputPort from an OFPort along with the IngressPort and the InPhyPort as metadata. All Flow Tables reside in an array within the LFB, the index of the array being the Flow Table ID. Each Flow Table compares the packet with the MatchFields inside the FlowEntries Table. If there is no match, depending upon the MissBehaviour component, one of the following actions will occur: 1. The packet or part of it will be sent to the controller via the singleton output port RedirectPacketOut along with the IngressPort, InPhyPort, RedirectReason and FlowTableID as metadata and optionally the BufferID if the packet is buffered in the switch. How the packet is buffered in the switch is implementation specific. 2. The packet will be sent it to the next table in the pipeline (the next flow table row entry in the array). 3. The packet will be dropped. If there is a match the LFB will decide based on the InstructionType of the component Instructions inside the matched FlowEntry. If the instruction is Apply Actions, the LFB will use the InstructionIndex to find the Actions inside the ApplyActionList. Each row of the ApplyActionList is an array containing rows of ActionRowTypes. For every ActionRowType, the LFB will send the packet to the corresponding Action LFB through the group output ActionPort port alongside with the LFBClassIDMetadata of the LFB, the PacketID and the ActionIndex, if the specific action has any Haleplidis, et al. Expires October 27, 2013 [Page 25] Internet-Draft ForCES OpenFlow library April 2013 parameters, like the Set MAC Address action. The ActionIndex is used as an index for the table inside the Action LFB. The packet is then returned from the Action LFB through the group input port PacketReturn to continue further processing if exists. The PacketID is an identifier issued by the OFFlowTables LFB that will uniquely identify the packet so that when it returns it will continue processing from whence it stopped. One exception to the rule when applying the action list is in regards to the Group action. The OFGroup LFB handles groups. The OFFlowTables using the ActionIndex locates the Group Identifier in the OFFlowTables's GroupTable component. Then it sends a copy of the packet to the GroupTableLFB using the group output ActionPort and the original packet will continue in the Flow Table. The ActionSet metadata is being populated as a packet traverses this LFB by both instructions Clear Actions and Write Actions. Write Metadata instruction is implementation specific as it is interally to the LFB. If the instruction is Goto Table, the LFB will use the InstructionIndex to find the value OFFlowTables InstanceID in the GoToFlowTable table and send the packet to the FLowTables row entry with the index of the FlowTable ID. At the end of the flow table pipeline, the packet along with the ActionSet metadata will be sent to the OFActionSet via the singleton output port ActionSetPort Additionally the OFFlowTables handle packets incoming from the controller expecting a whole packet either through the RedirectPacketIn group input port from the OFRedirectIn LFB or the OFBuffer LFB along with the ActionList and the IngressPort as metadata. The OFFlowTables MUST perform the ActionList on the packet prior to be sent in the flow tables. 5.2.2. Components The FlowTables, an array holding all Flow Tables of the switch. Each row of the array is a struct of the FlowEntries, the FlowTableCounter and the MissBehaviour for each flow. The FlowEntries component of the struct is an array and each row of the array is a struct containing the cookie, the MatchFields, the Counters, the Instructions, the Timeouts, the Timers and the priority of the specific flow entry. The FlowTableCounter are the counters of the specific FlowTable and the MissBehaviour component of the struct specifies what the specific Flow Table shall do with the packet if there is no match. Haleplidis, et al. Expires October 27, 2013 [Page 26] Internet-Draft ForCES OpenFlow library April 2013 The ApplyActionList is a component to maintain the actions required per flow. It is an array of Actions, which is an array of a struct of ActionType and ActionTableIndex. The WriteActions is a component to maintain the actions to be writen for a write actions instruction. It is an array of Actions, which is an array of a struct of ActionType and ActionTableIndex. The WriteMetadataTable is a component to hold the metadata values required for the write metadata value. It is an array of WriteMetadataTableType, which is a struct of the Metadata value and the MetadataMask. The GotoFlowTable component contains the FlowTable IDs flows should go to for the goto table action. It is an array of uint32. The value is selected using the InstructionIndex. The GroupTable component contains group identifiers. It is an array of group identifiers indexed by the ActionTableIndex. 5.2.3. Capabilities This LFB has no capabilities specified. 5.2.4. Events One event has been defined regarding the Flow Table. The event will be triggered when a flow is deleted from the Flow Table whether due to the idle timeout, or to the hard timeout or a flow was deleted by the controller. 5.3. OFGroupTable The Group LFB contains Action Buckets that can be applied to a packet during its path in the Flow Tables pipeline. 5.3.1. Data Handling The OFGroupTable does not take part in the actual handling of the data. Rather, it contains the action per group which are required by all Flow Tables in the pipeline. Packets initially enter this LFB from an OFActionSet LFB or a OFFlowTables via the group input port PacketIn and using the GroupIndex metadata the LFB finds the group requested for this packet. Then the LFB depending on the requested actions sends the packet to the required OFActionLFBs via the group output ActionPort and expects results via the group input PacketReturn. Haleplidis, et al. Expires October 27, 2013 [Page 27] Internet-Draft ForCES OpenFlow library April 2013 5.3.2. Components The LFB has only one component which is the Group Table. This is an array of GroupTableEntry types. Each GroupTableEntry contains a Group Identifier, the type of Group, the required counters and an array of action buckets. An action bucket is a struct which contains the Group weight required for select groups, the watch port and watch group required for fast failover groups, the bucket counters and the actions for this bucket. The structure of actions in a bucket are identical to the actions in the flow table LFB containing the type of action and an action table index. With the action type and action index the Group LFB can identify the component and index of the action details. 5.3.3. Capabilities This LFB has no capabilities specified. 5.3.4. Events This LFB has no events specified. 5.4. OFPort 5.4.1. Data Handling This LFB abstract the point where packets enter and exit the OpenFlow switch pipeline. May be a physical port, a virtual port defined by the switch. The LFB handles Ethernet frames coming in or going out to/of the OpenFlow switch. Ethernet frames are received and passed to an OFFlowTables through the singleton output port PacketIn, along with the IngressPortID and InPhyPort metadata. When a packet is ready to be send on the wire, it is sent to an OFPort instance through the group input port PacketOut and then depending of whether the packet has been assigned a queue with the QueueID as metadata and then is sent to the OFQueue LFB via the group output port QueueOut. 5.4.2. Components The PortNumber component uniquely identifies the port within a switch. The IEEEMAC component contains the MAC Address of the port. Haleplidis, et al. Expires October 27, 2013 [Page 28] Internet-Draft ForCES OpenFlow library April 2013 The Name component is a human readable name of the port. The Configuration component specifies port behaviour. It's a struct component with the following boolean fields. PortDown, NoReceiving, NoForwarding and NoPacket_In. The State component defines the current state of the OpenFlow Switch. It is a struct component that defines whether the link is down, the port is blocked or the port can be used for live fast failover. The Current Features component describes the current features of the port. It is a struct component and specifies the Speed Types, the Connected Medium, the Auto Negotiation and the Pause Types The Advertised Features component describes the advertised features of the port. The component is of the same structure as the current features. The CurrentSpeed component defines the current port bitrate in kbps. The MaximumSpeed component defines the maximum port bitrate in kbps. The PortCounter component contains the necessary counters for the port. It is a struct component comprised of counters for Packets Received, Packets Transmitted, Bytes Received, Bytes Transmitted, Drops Received, Transmit Drops, Errors in reception, Errors in transmittion, Frame Alignment Errors received, Received Overrun Errors, CRC Errors in received packets, Collisions. 5.4.3. Capabilities Two capabilities has been defined for the Port LFB. Supported Features and Peer Features. These include: o Types of Speed supported o Medium Connected to the port o Auto-negotiation o Pause Types supported of the port 5.4.4. Events This LFB has no events specified. 5.5. OFQueue Haleplidis, et al. Expires October 27, 2013 [Page 29] Internet-Draft ForCES OpenFlow library April 2013 5.5.1. Data Handling This LFB manages the queuing algorithm for handling packets prior to be output from the switch. Multiple OFQueue LFBs can be attached to an OFPort LFB to handle queues. If a packet has been set a QueueID with a Set Queue action, it is sent to the OFQueue LFB after the OFPort LFB and it enters this LFB via the group input port PacketIn where it will be handled according to the LFBs configuration and then be output from the switch. 5.5.2. Components The QueueID component, a uint32, defines the ID for the specific queue. The Properties component, is an array of Property Types an the length of the property, defines the current queue mode. Current specified modes are none and minimum rate. The QueueCounter component, a struct of TransmitPackets, TransmitBytes, TransimtOverrunErrors holds the necessary counter for the LFB. 5.5.3. Capabilities This LFB has no capabilities specified. 5.5.4. Events This LFB has no events specified. 5.6. OFRedirectIn The OFRedirectIn LFB abstracts the process for the controller to inject data packets into the switch to input data packets into the data path. The LFB is derived from the RedirectIn LFB defined in the Base LFB Library [I-D.ietf-forces-lfb-lib]. 5.6.1. Data Handling A packet or a bufferID arrives from the controller depending on whether the packet was buffered or not in the switch. If the packet was not buffered or the controller wishes to inject a packet into the switch, the packet will be output from the singleton output port PacketIn along with an ActionList and an IngressPort metadata to be sent to the OFFlowTables for processing. If the packet was buffered in the switch, the no packet is sent from the singleton output port BufferIn but only the BufferID, the ActionList and the IngressPort Haleplidis, et al. Expires October 27, 2013 [Page 30] Internet-Draft ForCES OpenFlow library April 2013 metadatas which is sent out the BufferIn singleton port to the OFBuffer LFB. 5.6.2. Components This LFB has no components specified. 5.6.3. Capabilities This LFB has no capabilities specified. 5.6.4. Events This LFB has no events specified. 5.7. OFRedirectOut The OFRedirectOut LFB abstracts the process for the switch to deliver data packets to the controller. The LFB is derived from the RedirectIn LFB defined in the Base LFB Library [I-D.ietf-forces-lfb-lib]. 5.7.1. Data Handling A packet arrives from the group input port Outgoing from the OFActionOutput LFBs or part of a packet along with the BufferID metadata arrives from the OFBufferLFB via the Outgoing group port. Besides the optional BufferID metadata, the IngressPort, the InPhyPort and the RedirectReason and the optional FlowTableID are sent to the Outgoing input port required to be sent to the Controller as metadata. 5.7.2. Components This LFB has no components specified. 5.7.3. Capabilities This LFB has no capabilities specified. 5.7.4. Events This LFB has no events specified. 5.8. OFActionSet Haleplidis, et al. Expires October 27, 2013 [Page 31] Internet-Draft ForCES OpenFlow library April 2013 The OFActionSet LFB abstracts the process for the switch to process the Action Set metadata and perform the Actions therein on the received packet. 5.8.1. Data Handling A packet arrives from the singleton input port PacketIn from the OFFlowTables LFB along with the ActionSet metadata. The ActionSet LFB then performs the ActionSet on the packet by sending it to the necessary Action LFBs via the group output port ActionPort and receiving it back via the PacketReturn group input port, or even sending it to the OFGroupTable via the GroupPort singleton output port. If the packet must be sent on an output port, the packet is sent via the OutputPort singelton output port. As all necessary parameters are inside the ActionSEt metadata, this LFB has no configurable or queriable parameters. 5.8.2. Components This LFB has no components specified. 5.8.3. Capabilities This LFB has no capabilities specified. 5.8.4. Events This LFB has no events specified. 5.9. OFBuffer The OFBuffer LFB abstracts the process for the switch to buffer packets and send only part of it to the RedirectOut to be sent to the controller, or receive a BufferID and send buffered packets to the OFFlowTables. 5.9.1. Data Handling Haleplidis, et al. Expires October 27, 2013 [Page 32] Internet-Draft ForCES OpenFlow library April 2013 A packet arrives from the singleton input port FlowTableIn from the OFFlowTables LFB to be Buffered. If there is enough room, based on the in the switch the packet is buffered and the MissSendLen specifies the amount of the packet is sent to the OFRedirectOut LFB to be sent to the controller via the ToRedirectOut singleton output port along with the Buffer ID metadata. Upon receiving the BufferID from the ControllerIn singleton input port from the OFRedirectIn LFB optionally along with the ActionList metadata, the OFBuffer LFB retrieves the buffered packet and sends it along with the ActionList to the OFFlowTables LFB. 5.9.2. Components This LFB has one component specified. The MissSendLen component, an unsigned integer of 16 bits, defines the maximum number of bytes of each packet sent to the controller as a result of both flow table misses and flow table hits with the controller as the destination. 5.9.3. Capabilities This LFB has one capability specified. The MaxBufferedPackets, an unsigned integer of 32 bits, defines the maximum packets the switch can buffer when sending packets to the controller. 5.9.4. Events This LFB has no events specified. 5.10. OFAction This LFB is a template used for create OFActionLFBs. All OFActionLFBs have the input and output port in common but have different components. This LFB defines how input and output port of all OFActionLFBs. Inside OFActionLFBs there is a table with the required attributes where applicable as some OFActionLFBs don't require attributes. 5.10.1. Data Handling A packet arrives in an OFActionLFB via the group input PacketIn from an OFFlowTables or an OFGroupTable, along with the LFBClassID metadata, required to uniquely identify the sender, the PacketID and optionally the ActionIndex metadata if the action requires a specific attributes, the IngressPort, the InPhyPort and the QueueID required Haleplidis, et al. Expires October 27, 2013 [Page 33] Internet-Draft ForCES OpenFlow library April 2013 by the OFActionOutput. Once the packet has been processed it is return to the sender LFB via the group output PacketOut along with the PacketID. 5.10.2. Components This LFB has no components specified. 5.10.3. Capabilities This LFB has no capabilities specified. 5.10.4. Events This LFB has no events specified. 5.11. OFActionLFBs As none of the OFActionLFBs have any capabilities or events, these sections are ommited from the draft. 5.11.1. OFActionOutput 5.11.1.1. Data Handling The OFActionOutputLFB does not modify the packet in any way, rather forwards a packet to a specified OFPort. Additionally there are several virtual ports that the OFActionOutputLFB may send the packet to: All - Group output, sends the packet out all standard ports, but not to the ingress port or ports configured not to forward Controller - Sends the packet to the controller Table - Submit the packet to the first flow table so that the packet can be processed through the regular OpenFlow pipeline. Only valid in the action list of a packet-out message InPort - Sends the packet out the ingress port. Local - Sends the packet to the switch's local networking stack Normal - Processes the packet using the traditional non-OpenFlow pipeline of the switch. Flood - Floods the packet using the normal pipeline of the switch. Haleplidis, et al. Expires October 27, 2013 [Page 34] Internet-Draft ForCES OpenFlow library April 2013 5.11.1.2. Components This LFB has only one component, the OutputActionTable, which is an array of a struct of the port number and optionally the maximum length in bytes, if the receiving end is the controller. 5.11.2. OFActionSetVLANVID 5.11.2.1. Data Handling The OFActionSetVLANVIDLFB replaces the existing VLAN ID. Only applies to packets with an existing VLAN tag. 5.11.2.2. Components This LFB has only one component, the SetVLANVIDActionTable, which is an array of uint16 VLAN tag values. 5.11.3. OFActionSetVLANPriority 5.11.3.1. Data Handling The OFActionSetVLANPriorityLFB replaces the existing VLAN priority. Only applies to packets with an existing VLAN tag. 5.11.3.2. Components This LFB has only one component, the SetVLANPriorityActionTable, which is an array of uchar VLAN priority values. 5.11.4. OFActionSetMACSource 5.11.4.1. Data Handling The OFActionSetMACSourceLFB replaces the existing Ethernet source MAC address. 5.11.4.2. Components This LFB has only one component, the SetMACSourceActionTable, which is an array of IEEEMAC addresses. 5.11.5. OFActionSetMACDestination 5.11.5.1. Data Handling The OFActionSetMACDestinationLFB replaces the existing Ethernet source MAC address. Haleplidis, et al. Expires October 27, 2013 [Page 35] Internet-Draft ForCES OpenFlow library April 2013 5.11.5.2. Components This LFB has only one component, the SetMACSourceActionTable, which is an array of IEEEMAC addresses. 5.11.6. OFActionSetIPSource 5.11.6.1. Data Handling The OFActionSetIPSourceLFB replaces the existing IP source address with new value and update the IP checksum (and TCP/UDP/SCTP checksum if applicable). This action is only applicable to IPv4 packets. 5.11.6.2. Components This LFB has only one component, the SetIPSourceActionTable, which is an array of IPv4 addresses. 5.11.7. OFActionSetIPDestination 5.11.7.1. Data Handling The OFActionSetIPDestinationLFB replaces the existing IP destination address with new value and update the IP checksum (and TCP/UDP/SCTP checksum if applicable). This action is only applicable to IPv4 packets. 5.11.7.2. Components This LFB has only one component, the SetIPDestinationActionTable, which is an array of IPv4 addresses. 5.11.8. OFActionSetIPTOS 5.11.8.1. Data Handling The OFActionSetIPTOSLFB replaces the existing IP TOS value and update the IP checksum. Only applies to IPv4 packets. 5.11.8.2. Components This LFB has only one component, the SetIPTOSActionTable, which is an array of IPv4 uchar TOS values. 5.11.9. OFActionSetIPECN 5.11.9.1. Data Handling Haleplidis, et al. Expires October 27, 2013 [Page 36] Internet-Draft ForCES OpenFlow library April 2013 The OFActionSetIPECNLFB replaces the existing IP ECN value and update the IP checksum. Only applies to IPv4 packets. 5.11.9.2. Components This LFB has only one component, the SetIPECNActionTable, which is an array of IPv4 uchar ECN values. 5.11.10. OFActionSetTCPSource 5.11.10.1. Data Handling The OFActionSetTCPSourceLFB replaces the existing TCP/UDP/SCTP source port with new value and update the TCP/UDP/SCTP checksum. This action is only applicable to TCP, UDP and SCTP packets. 5.11.10.2. Components This LFB has only one component, the SetTCPSourceActionTable, which is an array of uint16 values. 5.11.11. OFActionSetTCPDestination 5.11.11.1. Data Handling The OFActionSetTCPDestinationLFB replaces the existing TCP/UDP/SCTP destination port with new value and update the TCP/UDP/SCTP checksum. This action is only applicable to TCP, UDP and SCTP packets. 5.11.11.2. Components This LFB has only one component, the SetTCPDestinationActionTable, which is an array of uint16 values. 5.11.12. OFActionCopyTTLOut 5.11.12.1. Data Handling The OFActionCopyTTLOutLFB copies the TTL from next-to-outermost to outermost header with TTL. Copy can be IP-to-IP, MPLS-to-MPLS, or IP-to-MPLS. 5.11.12.2. Components This LFB has no components specified. 5.11.13. OFActionCopyTTLIn Haleplidis, et al. Expires October 27, 2013 [Page 37] Internet-Draft ForCES OpenFlow library April 2013 5.11.13.1. Data Handling The OFActionCopyTTLOutLFB copies the TTL from outermost to next-to- outermost header with TTL. Copy can be IP-to-IP, MPLS-to-MPLS, or IP-to-MPLS. 5.11.13.2. Components This LFB has no components specified. 5.11.14. OFActionSetMPLSLabel 5.11.14.1. Data Handling The OFActionSetMPLSLabelLFB replaces the existing MPLS label. Only applies to packets with an existing MPLS shim header. 5.11.14.2. Components This LFB has only one component, the SetMPLSLabelActionTable, which is an array of uint32 MPLS label values. 5.11.15. OFActionSetMPLSTC 5.11.15.1. Data Handling The OFActionSetMPLSTCLFB replaces the existing MPLS traffic class. Only applies to packets with an existing MPLS shim header. 5.11.15.2. Components This LFB has only one component, the SetMPLSTCActionTable, which is an array of uchar MPLS label values. 5.11.16. OFActionSetMPLSTTL 5.11.16.1. Data Handling The OFActionSetMPLSTTLLFB replaces the existing MPLS TTL. Only applies to packets with an existing MPLS shim header. 5.11.16.2. Components This LFB has only one component, the SetMPLSTTLTable, which is an array of uchar MPLS TTL values. 5.11.17. OFActionDecrementMPLSTTL Haleplidis, et al. Expires October 27, 2013 [Page 38] Internet-Draft ForCES OpenFlow library April 2013 5.11.17.1. Data Handling The OFActionDecrementMPLSTTLLFB decrements the MPLS TTL. Only applies to packets with an existing MPLS shim header. 5.11.17.2. Components This LFB has no components specified. 5.11.18. OFActionPushVLan 5.11.18.1. Data Handling The OFActionPushVLanLFB pushes a new VLAN header onto the packet. The Ethertype is used as the Ethertype for the tag. Only Ethertype 0x8100 and 0x88a8 should be used. 5.11.18.2. Components This LFB has only one component, the PushVLANTable, which is an array of uint16 ethertypes. 5.11.19. OFActionPopVLAN 5.11.19.1. Data Handling The OFActionPopVLANLFB pops the outer-most VLAN header from the packet. 5.11.19.2. Components This LFB has no components specified. 5.11.20. OFPushMPLSOFAction 5.11.20.1. Data Handling The OFPushMPLSOFActionLFB pushes a new MPLS shim header onto the packet. The Ethertype is used as the Ethertype for the tag. Only Ethertype 0x8847 and 0x8848 should be used. 5.11.20.2. Components This LFB has only one component, the PushMPLSTable, which is an array of uint16 MPLS header values. 5.11.21. OFPopMPLSOFAction Haleplidis, et al. Expires October 27, 2013 [Page 39] Internet-Draft ForCES OpenFlow library April 2013 5.11.21.1. Data Handling The OFPopMPLSOFActionLFB pops the outer-most MPLS tag or shim header from the packet. The Ethertype is used as the Ethertype for the resulting packet (Ethertype for the MPLS payload). 5.11.21.2. Components This LFB has only one component, the PopMPLSTable, which is an array of uint16 ethertype values. 5.11.22. OFSetQueueOFAction 5.11.22.1. Data Handling The OFSetQueueOFActionLFB sets the queue ID for the packet. This LFB will return the packet along with the QueueID as a metadata. 5.11.22.2. Components This LFB has only one component, the SetQueueTable, which is an array of uint32 queue identifiers. 5.11.23. OFSetIPTTLOFAction 5.11.23.1. Data Handling The OFSetIPTTLOFActionLFB replaces the existing IP TTL and update the IP checksum. Only applies to IPv4 packets. 5.11.23.2. Components This LFB has only one component, the SetIPTTLTable, which is an array of uchar TTL values. 5.11.24. OFDecrementIPTTLOFAction 5.11.24.1. Data Handling The OFDecrementIPTTLOFActionLFB decrements the existing IP TTL and update the IP checksum. Only applies to IPv4 packets. 5.11.24.2. Components This LFB has no components specified. 5.11.25. OFExperimenterOFAction Haleplidis, et al. Expires October 27, 2013 [Page 40] Internet-Draft ForCES OpenFlow library April 2013 5.11.25.1. Data Handling The OFExperimenterOFActionLFB handles experimenter actions. 5.11.25.2. Components This LFB has only one component, the SetIPTTLTable, which is an array of uint32 Experimenter ID values. 6. XML for OpenFlow library Arbitrary Any kind of Frame IEEEMAC An IEEE MAC Address byte[6] SwitchDescriptionType The type of the switch description MFR Manufacturer description string[256] HW Hardware description string[256] SF Software description Haleplidis, et al. Expires October 27, 2013 [Page 41] Internet-Draft ForCES OpenFlow library April 2013 string[256] SerialNum Serial Number string[32] DP Human Readable description of datapath string[256] MPLSLabelValue An MPLS label. uint32 MPLSTrafficClassValues The MPLS Traffic Class uchar IPv4ToSbits TOSBits uchar WildcardsType Wildcards for fields Haleplidis, et al. Expires October 27, 2013 [Page 42] Internet-Draft ForCES OpenFlow library April 2013 InPort Input Port Wildcard boolean VLANID VLan ID Wildcard boolean VLANPCP Vlan priority Wildcard boolean DLType Ethernet frame typ Wildcard boolean IPToS IP ToS Wildcard boolean IPProtocol IP Protocol Wildcard boolean TPSource TCP/UDP/SCTP source port Wildcard boolean TPDestination TCP/UDP/SCTP destination port Wildcard boolean MPLSLabel MPLS label Wildcard boolean Haleplidis, et al. Expires October 27, 2013 [Page 43] Internet-Draft ForCES OpenFlow library April 2013 MPLSTC MPLS TC Wildcard boolean MatchFieldType A Match Field Type IngressPort Numerical representation of incoming port, starting at 1. This may be a physical or switch-defined virtual port. uint32 Wildcards Wildcards for match fields WildcardsType EthernetSourceAddress Ethernet source address IEEEMAC EthernetSourceAddressMask Ethernet source address mask IEEEMAC EthernetDestinationAddress Ethernet destination address IEEEMAC EthernetDestinationAddressMask Ethernet destination address mask IEEEMAC VlanID VLAN identifier of outermost VLAN tag. VlanIDType Haleplidis, et al. Expires October 27, 2013 [Page 44] Internet-Draft ForCES OpenFlow library April 2013 VlanPriority VLAN PCP Field of outermost VLAN tag. VlanPriorityType EtherType Ethernet type of the OpenFlow packet payload after VLAN tags. 802.3 frames have special handling uint16 IPv4TOS Specify as 8-bit value and place ToS in upper 6 bits for match IPv4ToSbits IPProto IP protocol or lower 8 bits of ARP opcode. Only the lower 8 bits of the ARP opcode are used for the match uchar IPv4SourceAddress IPv4 Source Address to match IPv4Addr IPv4SourceAddressMask IPv4 Source Address mask IPv4Addr IPv4DestinationAddress IPv4 Destination Address to match IPv4Addr IPv4DestinationAddressMask IPv4 Destination Address mask IPv4Addr TCPSourcePort Haleplidis, et al. Expires October 27, 2013 [Page 45] Internet-Draft ForCES OpenFlow library April 2013 Source Port for TCP and ICMP to match uint16 TCPDestinationPort Destination Port for TCP and ICMP to match uint16 MPLSlabel Match on outermost MPLS tag. MPLSLabelValue MPLSTrafficClass Match on outermost MPLS tag for traffic class. MPLSTrafficClassValues Metadata MetaData uint64 MetadataMask MetaData Mask uint64 FlowEntry A Flow entry Cookie Opaque data chosen by controller uint64 MatchFields Match Fields: to match against packets. These consist of the ingress port and packet headers, and optionally metadata specified by a previous table Haleplidis, et al. Expires October 27, 2013 [Page 46] Internet-Draft ForCES OpenFlow library April 2013 MatchFieldType Counters Counters: to update for matching packets FlowCounterType Instructions Instruction: what to do with the packet of the flow InstructionType The instruction type InstructionTypes InstructionIndex The instruction index. uint32 InstructionType Timeouts Timeouts for the flow entry IdleTimeout Timeout to expire if no flows are matched for this flow entry uint16 HardTimeout Timeout to expire for this flow entry regardless of idle timeout uint16 Haleplidis, et al. Expires October 27, 2013 [Page 47] Internet-Draft ForCES OpenFlow library April 2013 Timers Timers per flow Duration_Sec Time flow has been alive in seconds uint32 Duration_nSec Time flow has been alive in nanoseconds beyond Duration_Sec uint32 Priority Priority within the specified flow table uint16 ActionRowType An Action Row for the action table Action The type of action ActionType ActionTableIndex Index of the Table this action applies to uint32 ActionType The type of action Haleplidis, et al. Expires October 27, 2013 [Page 48] Internet-Draft ForCES OpenFlow library April 2013 uint16 OUTPUT Output to switch port SetVLANVID Set the 802.1q VLAN id SetVLANPCP Set the 802.1q priority SetDLSrc Set Ethernet source address SetDLDst Set Ethernet destination address SetIPSrc Set IP source address SetIPDst Set IP Destination address SetIPTOS Set ToS field SetIPECN Set ECN field SetTPSource TCP/UDP/SCTP source port SetTPDestination TCP/UDP/SCTP destination port Haleplidis, et al. Expires October 27, 2013 [Page 49] Internet-Draft ForCES OpenFlow library April 2013 CopyTTLOut Copy TTL "outwards" -- from next-to-outermost to outermost CopyTTLIn Copy TTL "inwards" -- from outermost to next-to-outermost SetMPLSLabel Set MPLS label SetMPLSTC Set MPLS TC SetMPLSTTL Set MPLS TTL PushVLANTag Push a new VLAN tag PopVLANTag Pop the outer VLAN tag PushMPLSTag Push a new MPLS tag PopMPLSTag Pop the outer MPLS tag SetQueue Set queue ID when outputting to a port Group Apply group Haleplidis, et al. Expires October 27, 2013 [Page 50] Internet-Draft ForCES OpenFlow library April 2013 SetIPTTL Set IP TTL DecIPTTL Decrement IP TTL TableCounterType Counter per table ReferenceCount Active Entries uint32 PacketLookups Packet Lookups uint64 PacketMatches Packet Matches uint64 Actions Actions to perform. An Array of ActionRowTypes ActionRowType FlowCounterType Counter per flow ReceivedPackets Packets Received uint64 Haleplidis, et al. Expires October 27, 2013 [Page 51] Internet-Draft ForCES OpenFlow library April 2013 ReceivedBytes Bytes Received uint64 DurationS Duration in seconds uint32 DurationNS Duration in nanoseconds uint32 InstructionTypes Instructions supported uchar GotoTable Indicates the next table in the processing pipeline. The table-id must be greater than the current table-id. The flows of last table of the pipeline can not include this instruction WriteMetadata Writes the masked metadata value into the metadata field. The mask specifies which bits of the metadata register should be modified (i.e. new metadata = old metadata and ~mask | value and mask) WriteAction Merges the specifieed action(s) into the current action set. If an action of the given type exists in the current set, overwrite it, otherwise add it. ApplyActions Haleplidis, et al. Expires October 27, 2013 [Page 52] Internet-Draft ForCES OpenFlow library April 2013 Applies the specific action(s) immediately, without any change to the Action Set. This instruction may be used to modify the packet between two tables or to execute multiple actions of the same type. The actions are specified as an action list ClearActions Clears all the actions in the action set immediately. WriteMetadataTableType Metadata and mask for the write metadata instruction per row Metadata The metadata uint64 MetadataMask The metadata mask uint64 FlowTableMissConfigType Types to configure the default behavior of unmatched packets in a Flow Table uint32 Controller Send to the controller Continue Continue to the next table in the pipeline or send to the controller if the FlowTable is the last. Haleplidis, et al. Expires October 27, 2013 [Page 53] Internet-Draft ForCES OpenFlow library April 2013 Drop Drop the packet PacketInTypes Packet-in Types uchar NoMatch No Matching flow Action Explicit action to send to controller GroupCounterType Counters per group ReferenceCount Flow Entries uint32 PacketCount Packet Count uint64 ByteCount Byte Count uint64 Haleplidis, et al. Expires October 27, 2013 [Page 54] Internet-Draft ForCES OpenFlow library April 2013 GroupBucketExecuteType To determine which Action Bucket(s) should be executed uchar all Execute all buckets in the group. This group is used for multicast or broadcast forwarding. The packet is effectively cloned for each bucket; one packet is processed for each bucket of the group. If a bucket directs a packet explicitly out the ingress port, this packet clone is dropped. If the controller writer wants to forward out the ingress port, the group should include an extra bucket which includes an output action to the OFPP_IN_PORT virtual port. select Execute one bucket in the group. Packets are sent to a single bucket in the group, based on a switch-computed selection algorithm (e.g. hash on some user-configured tuple or simple round robin). All configuration and state for the selection algorithm is external to OpenFlow. When a port speciffied in a bucket in a select group goes down, the switch may restrict bucket selection to the remaining set (those with forwarding actions to live ports) instead of dropping packets destined to that port. This behavior may reduce the disruption of a downed link or switch. indirect Execute the one defined bucket in this group. Allows multiple flows or groups to point to a common group identifier, supporting faster, more efficient convergence (e.g. next hops for IP forwarding). This group type is effectively identical to an all group with one bucket. fastfailover Haleplidis, et al. Expires October 27, 2013 [Page 55] Internet-Draft ForCES OpenFlow library April 2013 Execute the first live bucket. Each action bucket is associated with a specific port and/or group that controls its liveness. Enables the switch to change forwarding without requiring a round trip to the controller. If no buckets are live, packets are dropped. This group type must implement a liveness mechanism BucketCounterType Counters per bucket PacketCount Packet Count uint64 ByteCount Byte Count uint64 GroupTableEntry A Row of the Group Table GroupID Group Identifier uniquely identifying the group uint32 GroupType The group type to determine which action bucket will be executed. GroupBucketExecuteType GroupCounters Counters per group GroupCounterType Haleplidis, et al. Expires October 27, 2013 [Page 56] Internet-Draft ForCES OpenFlow library April 2013 ActionBuckets An ordered list of action buckets. Each action bucket is a set of actions similar to a flow table. ActionBucket ActionBucket An Action Bucket Weight Relative weight of bucket. Only defined for select groups. uint16 WatchPort Port whose state affects whether this bucket is live. Required for fast failover group uint32 WatchGroup Group whose state affects whether this group is live. Only required for fast failover groups uint32 Actions Actions for this bucket Actions BucketCounter A counter for this bucket BucketCounterType PortNumberType Haleplidis, et al. Expires October 27, 2013 [Page 57] Internet-Draft ForCES OpenFlow library April 2013 Port Number values uint32 InPort Sent the packet out the input port. This virtual port must be explicitly used in order to send back out of the input port Table Submit the packet to the first flow table. NBL This destination port can only be used in packet-out messages Normal Process with normal L2/L3 switching Flood Send the packet to all physical ports in VLAN, except input port and those blocked or link down. All Send the packet to all physical ports, except input port. Controller Send the packet to the controller. Local Local openflow "port". Any Wildcard port used only for flow mod (delete) and flow stats requests. Selects all flows regardless of output port (including flows with no output port). Haleplidis, et al. Expires October 27, 2013 [Page 58] Internet-Draft ForCES OpenFlow library April 2013 PortConfigurationType Types of configuration for the OpenFlow port PortDown Port is administatively down boolean NoReceiving Drop all packets received by this port boolean NoForwarding Drop packets forwarded to the port boolean NoPacket_In Do not send packet-in messages for port boolean PortStateType Current State of the port LinkDown No physical link present boolean PortBlocked Port is blocked boolean PortLive Haleplidis, et al. Expires October 27, 2013 [Page 59] Internet-Draft ForCES OpenFlow library April 2013 Live for Fast Failover Group boolean PortFeaturesType Port Features SpeedTypes Types of Speed supported 10MB_HD 10 Mb half-duplex rate support. boolean 10MB_FD 10 Mb full-duplex rate support. boolean 100MB_HD 100 Mb half-duplex rate support. boolean 100MB_FD 100 Mb full-duplex rate support. boolean 1GB_HD 1 Gb half-duplex rate support. boolean 1GB_FD 1 Gb full-duplex rate support. boolean Haleplidis, et al. Expires October 27, 2013 [Page 60] Internet-Draft ForCES OpenFlow library April 2013 10GB_FD 10 Gb full-duplex rate support. boolean 40GB_FD 40 Gb full-duplex rate support. boolean 100GB_FD 100 Gb full-duplex rate support. boolean 1TB_FD 1 Tb full-duplex rate support. boolean Other Other rate, not listed. boolean MediumConnected Medium Connected to the port Copper Copper Medium boolean Fiber Fiber Medium boolean Haleplidis, et al. Expires October 27, 2013 [Page 61] Internet-Draft ForCES OpenFlow library April 2013 Auto Auto-negotiation boolean PauseTypes Pause Types supported of the port Pause Pause boolean AsymmetricPause Asymmetric pause boolean PortCounterType Counter per port ReceivedPackets Packets Received uint64 TransmittedPackets Packets Transmitted uint64 ReceivedBytes Bytes Received uint64 TransmittedBytes Bytes Transmitted uint64 Haleplidis, et al. Expires October 27, 2013 [Page 62] Internet-Draft ForCES OpenFlow library April 2013 ReceivedDrops Drops Received uint64 TransmitDrops Transmit Drops uint64 RecieveErrors Errors in reception uint64 TransmitErrors Errors in transmittion uint64 ReceivedFrameAlignmentErrors Frame Alignment Errors received uint64 ReceiveOverrunErrors Received Overrun Errors uint64 ReceivedCRCErrors CRC Errors in received packets uint64 Collisions Collisions uint64 QueuePropertyType Property type for a queue uint16 Haleplidis, et al. Expires October 27, 2013 [Page 63] Internet-Draft ForCES OpenFlow library April 2013 None No property defined MinimumRate Minimum datarate guaranteed QueueArrayPropertiesType Type Definition for property Property One of Queue Priority Types QueuePropertyType Length Length of property uint32 QueueCounterType Counters per queue TransmitPackets Packets Transmitted uint64 TransmitBytes Bytes Transmitted uint64 TransimtOverrunErrors Overrun Errors uint64 Haleplidis, et al. Expires October 27, 2013 [Page 64] Internet-Draft ForCES OpenFlow library April 2013 ActionRowType The ActionSet Row Type ActionTypes The type of action for the action set ActionType LFBClassID The LFBClassID this action is supposed to go uint32 LFBInstanceID The LFBInstanceID this action is supposed to go uint32 ActionTableIndex The index of the action in the LFBClass.LFBInstance. If the LFB class is a group, then the index is the group index. uint32 IngressPort The Ingress port the packet has arrived from. 1024 uint32 InPhyPort The Port Index of the Physical interface the frame entered the switch 1025 uint32 Haleplidis, et al. Expires October 27, 2013 [Page 65] Internet-Draft ForCES OpenFlow library April 2013 PacketID The PacketID metadata is used to uniquely identify a packet within the Flow Table or the Group Table to continue processing it after it has been returned from an OFActionLFB. 1026 uint32 ActionIndex The Action Index metadata is used to point the row in the array in an Action LFB 1027 uint32 GroupIndex The Group index metadata is used to point to the row of the array in an Group LFB 1028 uint32 LFBClassIDMetadata The LFBClassID 1029 uint32 LFBInstanceIDMetadata The LFB Instance ID 1030 uint32 QueueID The Queue ID 1031 uint32 BufferID The Buffer ID of a stored packet in the switch requried for the PacketOut message 1032 uint32 Haleplidis, et al. Expires October 27, 2013 [Page 66] Internet-Draft ForCES OpenFlow library April 2013 RedirectReason The reason the packet was redirected to the controller 1033 uchar NoMatch No match on the Flow Table (table miss) Action Specific Output to controller action. FlowTableID The FlowTable ID the packet was sent to the controller from 1034 uchar ActionList The Action List that may come along with the packet in a PacketOut message 1035 ActionRowType ActionSet The Action Set per packet. 1036 ActionRowType ActionTypes Haleplidis, et al. Expires October 27, 2013 [Page 67] Internet-Draft ForCES OpenFlow library April 2013 OFSwitch Similar to the FEProtocol and FEObject LFB, the OpenFlowSwitch LFB contains information required for the openflow protocol. 1.1 DatapathID Datapath unique ID. The lower 48-bits are for a MAC address, while the upper 16-bits are implementer-defined. uint64 HandleFragments if true drop fragments. If false no special handling. boolean ReassembleFragments If true, reassemble fragments boolean InvalidTTLtoController Send packets with invalid TTL ie. 0 or 1 to controller boolean SwitchDescription Information about the switch SwitchDescriptionType Ports The Ports that this switch has. It is an array of the Port Numbers uint32 Haleplidis, et al. Expires October 27, 2013 [Page 68] Internet-Draft ForCES OpenFlow library April 2013 FlowStatistics Whether the switch keep flow statistics boolean TableStatistics Whether the switch keep table statistics boolean PortStatistics Whether the switch keep port statistics boolean GroupStatistics Whether the switch keep group statistics boolean IPReassembly Whether the switch can reassemble IP fragments boolean QueueStats Whether the switch keeps queue statistics boolean ARPMatchIP Whether the switch matches IP addresses in APR packets boolean ActionsSupported What actions are supported ActionType Haleplidis, et al. Expires October 27, 2013 [Page 69] Internet-Draft ForCES OpenFlow library April 2013 ActionType TablesSupported Number of tables supported by the datapath uchar PortAdded This event is sent when a port is added Ports PortDeleted This event is sent when a port is deleted Ports PortModified This event is sent when a port is modified Ports OFFlowTables An LFB that houses all OpenFlow Flow Tables Haleplidis, et al. Expires October 27, 2013 [Page 70] Internet-Draft ForCES OpenFlow library April 2013 residing in the switch 1.1 InputPort An Input port that expects packets from an OFPort LFB Arbitrary IngressPort InPhyPort PacketReturn A port that expects the packet to be returned from an OFActionLFB. If the OFActionLFB is the OFQueueLFB then the QueueID metadata is expected as well. Arbitrary PacketID QueueID RedirectPacketIn A port that expects a packet from the controller Arbitrary ActionList IngressPort Haleplidis, et al. Expires October 27, 2013 [Page 71] Internet-Draft ForCES OpenFlow library April 2013 OutputPort A port that produces packets leaving the flow table to go to the OFOutputAction (to be sent to an output port) Arbitrary IngressPort InPhyPort QueueID GroupPort A port that produces packets leaving the flow table to go to the OFGroupTable. Arbitrary IngressPort InPhyPort GroupIndex QueueID ActionPort A port that sends the packet to an OFAction LFB Arbitrary LFBClassIDMetadata PacketID ActionIndex Haleplidis, et al. Expires October 27, 2013 [Page 72] Internet-Draft ForCES OpenFlow library April 2013 ActionSetPort A port that sends the packet to the ActionSet LFB Arbitrary IngressPort InPhyPort ActionSet QueueID RedirectPacketOut A port that sends a packet or part of it to the controller Arbitrary IngressPort InPhyPort RedirectReason FlowTableID FlowTables Flow entries inside the FlowTable LFB FlowEntries An array of Flow Entries FlowEntry Haleplidis, et al. Expires October 27, 2013 [Page 73] Internet-Draft ForCES OpenFlow library April 2013 FlowTableCounter A counter for each Flow Table TableCounterType MissBehaviour What should the FlowTable do if a miss occurs FlowTableMissConfigType ApplyActionList Table of actions for each flow Actions WriteActions Table of Actions to write to the ActionSet Actions WriteMetadataTable The write MetaDataTable WriteMetadataTableType GotoFlowTable Containing the FlowTable IDs this flow should go to. uint32 GroupTable Table of group indeces to point a packet to Haleplidis, et al. Expires October 27, 2013 [Page 74] Internet-Draft ForCES OpenFlow library April 2013 uint32 FlowRemoved If a CE subscribes to this event, it will send an event when a flow is removed. FlowEntries FlowEntry FlowTableID FlowEntries FlowEntry Cookie FlowEntries FlowEntry MatchFields FlowEntries FlowEntry Timeouts IdleTimeout FlowEntries FlowEntry Priority OFGroupTable The OpenFlow Group Tables 1.1 Haleplidis, et al. Expires October 27, 2013 [Page 75] Internet-Draft ForCES OpenFlow library April 2013 PacketIn A port to expect packets, the GroupIndex metadata, the IngressPort and InPhyPort and optionally the QueueID if the packet has already been through the OFActionSetQueue LFB. Arbitrary GroupIndex IngressPort InPhyPort QueueID PacketReturn A port that expects the packet to be returned from an OFActionLFB. If the OFActionLFB is the OFQueueLFB then the QueueID metadata is expected as well. Arbitrary PacketID QueueID PacketOut The port to return the packet to caller Arbitrary Haleplidis, et al. Expires October 27, 2013 [Page 76] Internet-Draft ForCES OpenFlow library April 2013 ActionPort A port that sends the packet to an OFActionLFB Arbitrary LFBClassIDMetadata LFBInstanceIDMetadata ActionIndex GroupTable The group table GroupTableEntry OFPort Input or Output port of an OpenFlow switch 1.1 PacketOut The input port of the Port LFB from the OFActionOutput LFB to output packets. Arbitrary QueueID PacketIn Haleplidis, et al. Expires October 27, 2013 [Page 77] Internet-Draft ForCES OpenFlow library April 2013 Sends a packet to the OFFlowTables that is received by the OFPort LFB. Arbitrary IngressPort InPhyPort QueueOut Sends a packet to the OFQueue LFB to be processed and output from the switch Arbitrary PortNumber The port number uniquely identifies a port within a switch. PortNumberType IEEEMAC MAC Address of the port IEEEMAC Name Human readable name of the port string[16] Configuration Configuration of the port PortConfigurationType State State of the OpenFlow Switch Haleplidis, et al. Expires October 27, 2013 [Page 78] Internet-Draft ForCES OpenFlow library April 2013 PortStateType CurrentFeatures Current features of the port PortFeaturesType Advertised Features advertised by the port PortFeaturesType CurrentSpeed Current port bitrate in kbps uint32 MaximumSpeed Maximum port bitrate in kbps uint32 PortCounter Counters for the port PortCounterType Supported Features Supported by the port PortFeaturesType Peer Features advertised by the peer PortFeaturesType OFQueue A queue LFB that can be attached to a port and be used to map flows on it. Flows mapped to a queue will be treated according to that queue's configuration 1.1 Haleplidis, et al. Expires October 27, 2013 [Page 79] Internet-Draft ForCES OpenFlow library April 2013 PacketIn An input port that expects any kind of frame. Arbitrary QueueID ID for the specific queue uint32 Properties List of queue properties QueueArrayPropertiesType QueueCounter Counters for the queue QueueCounterType OFRedirectIn The OFRedirectIn LFB abstracts the process for the controller to inject data packets into the switch to input data packets into the data path. 1.1 RedirectIn PacketIn An output port that sends a packet in the data path Arbitrary Haleplidis, et al. Expires October 27, 2013 [Page 80] Internet-Draft ForCES OpenFlow library April 2013 ActionList IngressPort BufferIn An output port that sends only the buffer id to locate a buffered packet BufferID ActionList IngressPort OFRedirectOut The OFRedirectOut LFB abstracts the process for the switch to deliver data packets to the controller 1.1 RedirectOut Outgoing The input port expects either the whole packet to be sent to the controller or part of it along with the buffer ID Arbitrary IngressPort InPhyPort RedirectReason FlowTableID BufferID Haleplidis, et al. Expires October 27, 2013 [Page 81] Internet-Draft ForCES OpenFlow library April 2013 OFActionSet The ActionSet LFB 1.1 PacketIn A port that expects a packet along with metadata IngressPort InPhyPort ActionSet QueueID PacketReturn A port that expects the packet to be returned from an OFActionLFB. If the OFActionLFB is the OFQueueLFB then the QueueID metadata is expected as well. Arbitrary PacketID QueueID OutputPort A port that produces packets leaving the flow table to go to the OFOutputAction (to be sent to an output port) Arbitrary IngressPort InPhyPort QueueID Haleplidis, et al. Expires October 27, 2013 [Page 82] Internet-Draft ForCES OpenFlow library April 2013 GroupPort A port that produces packets leaving the flow table to go to the OFGroupTable. Arbitrary IngressPort InPhyPort GroupIndex QueueID ActionPort A port that sends the packet to an OFAction LFB Arbitrary LFBClassIDMetadata PacketID ActionIndex OFBuffer An LFB that buffers packets and send them to the controller 1.1 FlowTableIn A packet received by a Flow Table to be buffered and sent to the controller Haleplidis, et al. Expires October 27, 2013 [Page 83] Internet-Draft ForCES OpenFlow library April 2013 Arbitrary IngressPort InPhyPort ControllerIn A Buffer ID received as metadata along with optional Action list from the controller BufferID ActionList ToFlowTable When a buffer ID has been received optionally along with the Action List metadata it is sent to the Flow Tables for process Arbitrary ActionList ToRedirectOut When a packet has been received it will be buffered and the buffer id will be sent to the redirect out. BufferID Haleplidis, et al. Expires October 27, 2013 [Page 84] Internet-Draft ForCES OpenFlow library April 2013 MissSendLen Max bytes of new flow that datapath should send to the controller. uint16 MaxBufferedPackets Maximum packets buffered at once. uint32 OFAction An LFB that performs one specific action on a packet in the OpenFlow switch. The OFActionLFB expects any kind of packet and as metadata the FlowTableInstanceID to know from which Flow Table the packet has arrived from and the Action Index to specify the row in the Action Table, if there is an Action table. 1.1 PacketIn An input port that gets the packet to perform the action on. Expects the ClassID of the LFB that calls it to know to which LFB to return it to. Can accept calls from the OFFlowTables or the OFGroupLFB. Arbitrary PacketID LFBClassIDMetadata ActionIndex IngressPort InPhyPort QueueID Haleplidis, et al. Expires October 27, 2013 [Page 85] Internet-Draft ForCES OpenFlow library April 2013 PacketOut The output port from which the packet will be send back to the LFB (OFFlowTables or OFGroupTable) from which it came from. Arbitrary PacketID OFActionOutput An LFB that performs the Output Action 1.1 OFAction PortOutput Send a copy of the packet to the specified port(s) Arbitrary OutputActionTable Output to switch port Port The port to send the packet out PortNumberType MaxLength If the port is the controller sets Haleplidis, et al. Expires October 27, 2013 [Page 86] Internet-Draft ForCES OpenFlow library April 2013 the maximum number of bytes to send. uint16 OFActionSetVLANVID An LFB that performs the Set VLANID Action 1.1 OFAction SetVLANVIDActionTable Set the 802.1q VLAN ID uint16 SetVLANPriorityActionTable An LFB that performs the Set VLAN Priority Action 1.1 OFAction SetVLANPriorityActionTable Set the 802.1q VLAN Priority uchar OFActionSetMACSource An LFB that performs the Set MAC Source Action 1.1 OFAction Haleplidis, et al. Expires October 27, 2013 [Page 87] Internet-Draft ForCES OpenFlow library April 2013 SetMACSourceActionTable Set MAC source address IEEEMAC OFActionSetMACDestination An LFB that performs the Set MAC Destination Action 1.1 OFAction SetMACDestinationActionTable Set MAC destination address IEEEMAC OFActionSetIPSource An LFB that performs the Set IP Source Action 1.1 OFAction SetIPSourceActionTable Set the IP source address IPv4Addr OFActionSetIPDestination An LFB that performs the Set IP Destination Action 1.1 OFAction Haleplidis, et al. Expires October 27, 2013 [Page 88] Internet-Draft ForCES OpenFlow library April 2013 SetIPDestinationActionTable Set the IP destination address IPv4Addr OFActionSetIPTOS An LFB that performs the Set VLANID Action 1.1 OFAction SetIPTOSActionTable Set IP ToS field uchar OFActionSetIPECN An LFB that performs the Set IP ECN Action 1.1 OFAction SetIPECNActionTable Set IP ECN field uchar OFActionSetTCPSource An LFB that performs the Set TCP/UDP/SCTP Source port Action 1.1 OFAction Haleplidis, et al. Expires October 27, 2013 [Page 89] Internet-Draft ForCES OpenFlow library April 2013 SetTCPSourceActionTable Sets TCP/UDP/SCTP source port uint16 OFActionSetTCPDestination An LFB that performs the Set TCP/UDP/SCTP Destination port Action 1.1 OFAction SetTCPDestinationActionTable Sets TCP/UDP/SCTP destination port uint16 OFActionCopyTTLOut An LFB that performs the copy TTL outwards Action 1.1 OFAction OFActionCopyTTLIn An LFB that performs the copy TTL inwards Action 1.1 OFAction OFActionSetMPLSLabel An LFB that performs the Set MPLS Label Action 1.1 OFAction Haleplidis, et al. Expires October 27, 2013 [Page 90] Internet-Draft ForCES OpenFlow library April 2013 SetMPLSLabelActionTable Sets MPLS Label Table uint32 OFActionSetMPLSTC An LFB that performs the Set MPLS Traffic Class Action 1.1 OFAction SetMPLSTCActionTable Sets MPLS Label Table uchar OFActionSetMPLSTTL An LFB that performs the Set MPLS TTL Action 1.1 OFAction OFActionSetMPLSTTL Sets MPLS TTL Table uchar OFActionDecrementMPLSTTL An LFB that performs the decrementation of the MPLS TTL Action 1.1 OFAction Haleplidis, et al. Expires October 27, 2013 [Page 91] Internet-Draft ForCES OpenFlow library April 2013 OFActionPushVLan An LFB that performs the Push VLAN Action 1.1 OFAction PushVLANTable Push VLAN Table uint16 OFActionPopVLAN An LFB that performs the Pop VLAN Action 1.1 OFAction OFActionPushMPLS An LFB that performs the Push MPLS Action 1.1 OFAction PushMPLSTable Push MPLS Table uint16 OFActionPopMPLS An LFB that performs the Pop MPLS Action 1.1 OFAction PopMPLSTable Pop MPLS Table Haleplidis, et al. Expires October 27, 2013 [Page 92] Internet-Draft ForCES OpenFlow library April 2013 uint16 OFActionSetQueue An LFB that performs the Set Queue Action 1.1 OFAction QueuePacketOut The output port from which the packet will be send back to the Flow Table/GroupTable from which it came from. Arbitrary PacketID QueueID SetQueueTable Sets Queue Table uint32 OFActionSetIPTTL An LFB that performs the Set IP TTL Action 1.1 OFAction SetIPTTLActionTable Haleplidis, et al. Expires October 27, 2013 [Page 93] Internet-Draft ForCES OpenFlow library April 2013 Sets IP TTL Table uchar OFActionDecrementIPTTL An LFB that performs the decrementation of the IP TTL Action 1.1 OFAction OpenFlow XML Library 7. Acknowledgements The authors would like to thank Ahmad N. Quttoum, Zoltan Lajos Kis, Joel Halpern and especially Jamal Hadi Salim, for discussions which helped shape this document. 8. IANA Considerations (TBD) 9. Security Considerations TBD 10. References 10.1. Normative References [I-D.haleplidis-forces-model-extension] Haleplidis, E., "ForCES Model Extension", draft- haleplidis-forces-model-extension-01 (work in progress), October 2012. [I-D.ietf-forces-lfb-lib] Wang, W., Haleplidis, E., Ogawa, K., Li, C., and J. Halpern, "ForCES Logical Function Block (LFB) Library", draft-ietf-forces-lfb-lib-08 (work in progress), February 2012. Haleplidis, et al. Expires October 27, 2013 [Page 94] Internet-Draft ForCES OpenFlow library April 2013 [McKeown] , "McKeown, N., Anderson, T., Balakrishnan, H., et al, "OpenFlow: enabling innovation in campus networks", ACM SIGCOMM Computer Communication Review. 2008, 38(2):69-74.", , . [OpenFlowSpec1.1] http://www.OpenFlow.org/, "The OpenFlow 1.1 Specification.", , . [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation of IP Control and Forwarding", RFC 3654, November 2003. [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, "Forwarding and Control Element Separation (ForCES) Framework", RFC 3746, April 2004. [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and Control Element Separation (ForCES) Protocol Specification", RFC 5810, March 2010. [RFC5811] Hadi Salim, J. and K. Ogawa, "SCTP-Based Transport Mapping Layer (TML) for the Forwarding and Control Element Separation (ForCES) Protocol", RFC 5811, March 2010. [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control Element Separation (ForCES) Forwarding Element Model", RFC 5812, March 2010. [RFC5813] Haas, R., "Forwarding and Control Element Separation (ForCES) MIB", RFC 5813, March 2010. [RFC6053] Haleplidis, E., Ogawa, K., Wang, W., and J. Hadi Salim, "Implementation Report for Forwarding and Control Element Separation (ForCES)", RFC 6053, November 2010. 10.2. Informative References [RFC2629] Rose, M.T., "Writing I-Ds and RFCs using XML", RFC 2629, June 1999. Haleplidis, et al. Expires October 27, 2013 [Page 95] Internet-Draft ForCES OpenFlow library April 2013 Authors' Addresses Evangelos Haleplidis University of Patras Department of Electrical & Computer Engineering Patras 26500 Greece Email: ehalep@ece.upatras.gr Omar Cherkaoui University of Quebec in Montreal Montreal Canada Email: cherkaoui.omar@uqam.ca Susan Hares Huawei USA Email: shares@ndzh.com Weiming Wang Zhejiang Gongshang University 18 Xuezheng Str., Xiasha University Town Hangzhou 310018 P.R.China Phone: +86-571-28877721 Email: wmwang@zjgsu.edu.cn Haleplidis, et al. Expires October 27, 2013 [Page 96]