Event processing built on top of JSON-RPC

Summary

Engine to vdsm communication was always initiated by an engine. Even when we execute long running tasks on vdsm there is polling mechanism to check status of a task. This behavior creates communication overhead and we want to address this issue by sending messages from vdsm and breaking current mechanism of rpc. This feature provides infrastructure to send messages from vdsm and to receive them on an engine side. We are not going to modify existing xmlrpc and it is still supported in 3.6.

Owner

• Name: Piotr Kliczewski (Pkliczewski)
• Email: pkliczew@redhat.com

Current status

• Last updated on – by (WIKI)

Overview

In 3.5 release we introduced jsonrpc which provided asynchronous behavior. We are going to leverage it in order to provide vdsm side messages. We refer to such messages as events generated by vdsm. Constant flow of events is delivered to different parts of an engine by using reactive streams abstractions. Each event can be sent to one or more vdsm clients by using stomp level subscriptions.

Event structure

3.5 implementation uses stomp protocol to send jsonrpc defined messages. As part of this effort we are going to use notification structure as defined in the specification.

Here is the structure of an event:

SEND
destination: <queue/topic>
content-type: text/json
content-length: <length>
content-encoding: <token as defined by IANA>
{
    "jsonrpc": "2.0",
    "method": "`<receiver>`|`<component>`|`<operation_id>`|`<unique_id>`",
    "params": {
        <contents>
    }
}
^@

<queue/topic> defines a destination to which we deliver events. Each event is delivered to all the clients which subscribed to its destination. At the moment event destination is defined in config.py using ‘event_queue’ property and value of it is an engine response: ‘jms.topic.vdsm_responses’

<receiver>|<component>|<operation_id>|<unique_id> defines subscription id which is used to match engine side subscription.

<contents> defines the place where we send data as part of an event.

Subscription identifier

Subscription ID is used to uniquely identify an event, so that the entity receiving it is able to match it to subscribed entities. The idea behind this ID is to use information known by all the parties. Here is how we define each of the sections:

<receiver> contains IP address or host name, and it is provided by the client side (engine) when it is received.

<component> contains information about which component generated the event such as virt/storage/network and etc.

<operation_id> contains information about the operation. In order to ease migration from rpc based communication to events we can use operation names such as Image_move, Volume_copy etc.

<unique_id> contains information about the object on which we perform operation like image, volume or VM uuid.

When subscribing, it is possible to use wildcards in different sections of the subscription ID. For example:

’\*|virt|\*|8839ddac-d833-4b0d-b7e2-4517fd100c8f’ - for this subscription id we are receive events which match component ‘virt’ and are generated for vm id ‘8839ddac-d833-4b0d-b7e2-4517fd100c8f’. Receiving all possible events by specifying ‘\*|\*|\*|\*’ filter is not supported.

Communication infrastructure

During design process we have explored the following communication models.

Internal to vdsm broker

In 3.5 we already had notion of a broker which was responsible for processing of stomp level messages. Due to time constraints we haven’t implemented subscription mechanism which is provided as part of this implementation. Even though subscriptions were not implemented in vdsm, the engine always sends a SUBSCRIBE frame. 3.5 implementation uses queue naming convention which is not supported by brokers such as activemq so we have decided to change it for 3.6 and introduce legacy mode in internal broker. There are following ways internal broker can process messages:

• legacy mode

It is detected when following destinations are used for requests: ‘/queue/_local/vdsm/requests’ and for responses ‘/queue/_local/vdsm/reponses’.

• standard mode

Vds request ‘jms.topic.vdsm_requests’ and response ‘jms.queue.reponses’ destinations Irs request destination ‘jms.topic.vdsm_irs_requests’ and response ‘jms.queue.irsreponses’ destination vdsClient and migration used to use ‘jms.topic.vdsm_requests’ as destination and provide unique response destination in stomp frame level header ‘reply-to’

• broker mode

Above modes let jsonrpc code to process messages and vdsm generates the response. This mode is used when request destinations are different than queue/topic names used for the modes above. Vdsm acts as regular stomp broker supporting most of the spec.

Broker based

We explored 2 possible typologies of a broker. We are going to make sure that we are able to use a broker between vdsm and engine but it won’t be supported in 3.6 release.