Logisland Rest Api Guide
The Logisland REST API for third party applications
1. Introduction
Logisland makes available a standard RESTful API definition to interoperate with any third party application implementing it.
The API should be implemented by a third party application and logisland will regularly poll this endpoint in order to:
-
Ask for configuration changes to be triggered.
-
Report the latest configuration applied (to ease up resynchronization and business continuity).
Both flows can hence be resumed by the following sequence diagram:
2. Usage
In terms of API, two degrees of freedom are possible:
- Dataflow
-
A
dataflowis a set of services and streams allowing a data flowing from one or more sources, being transformed and reach one or more destinations (sinks).Act at dataflow level if you want to:
-
Add/Remove any streaming endpoint
-
Change any active stream configuration (e.g. kafka topic)
-
Create/Remote/Modify any service
-
- Pipeline
-
A
pipelineis a processing chain acting on a data flowing point-to-point.The api gives you the possibility to have a finer-grained control of what is going of any stream pipeline without perturbing the stream itself. This means that the processor chain will be dynamically reconfigured without the need of stopping the stream and reconfigure the whole dataflow.
Act at pipeline level if you want to:
-
Add/Remove processors in the pipeline
-
Change any processor configuration
-
| As a general rule, the changes will be triggered if the lastUpdated field of the object you are going to modify is fresher than the one known by logisland. |
3. Specification
This section resumes the Rest API specification. More details are available on the swagger spec.
4. GET /dataflows/{dataflowName}
4.1. Summary
Retrieves the configuration for a specified dataflow
4.2. Description
A dataflow is a set of services and streams allowing a data flowing from one or more sources, being transformed by a pipeline and reach one or more destinations (sinks). Logisland will call this endpoint to know which configuration should be run.
This endpoint also supports HTTP caching (Last-Updated, If-Modified-Since) as per RFC 7232, section 3.3
4.3. Parameters
| Name | Located in | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|---|
dataflowName |
path |
Yes |
string |
the dataflow name (aka the logisland job name) |
4.4. Request
4.5. Headers
If-Modified-Since: Timestamp of last response
4.6. Responses
4.6.1. 200
Return the dataflow configuration. On logisland side, the following will happen: - At dataflow level:
-
Fully reconfigure a dataflow (stop and then start) if nothing is running (initial state) or if lastUpdated is fresher than the one of the already running dataflow.
In this case be aware that old stream and services will be destroyed and new ones will be created.
-
Do nothing otherwise (keep running the active dataflow)
-
At pipeline level:
-
The processor chain will be fully reconfigured if and only if the pipeline lastUpdated is fresher than the lastUpdated known by the system.
In any case the stream is never stopped.
Type: :ref:`Versioned <d_bcefda54d79a3bedfa83231aed8d38b1>` extended :ref:`inline <i_ae1.1.215667b75409cb3251ba13c032>`
Example:
{
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"services": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
],
"streams": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring",
"pipeline": {
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"processors": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
]
}
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring",
"pipeline": {
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"processors": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
]
}
}
]
}4.6.2. 304
Nothing has been modified since the last call.
In this case the body content will be completely ignored (hence the server can answer with an empty body to save network and resources).
4.6.3. 404
Not found (the server probably does not handle this dataflow)
4.6.4. default
Unexpected error
5. POST /dataflows/{dataflowName}
5.1. Summary
Push the configuration of running dataflows.
5.2. Description
In order to ensure business continuity, Logisland will contact the third party application in order to push a snapshot of the current configuration. The endpoint will be called:
-
On a regular basis (according to logisland configuration).
-
Each time the a dataflow or a pipeline configuration change has been applied.
This service can be seen as well as a liveness ping.
5.3. Parameters
| Name | Located in | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|---|
jobId |
path |
Yes |
string |
logisland job id (aka the engine name) |
||
dataflowName |
path |
Yes |
string |
the dataflow name (aka the logisland job name) |
5.4. Request
-
_d_68b618b2088b15f9f9f912df4be811df:
5.5. Body
A streaming pipeline.
:ref:`Versioned <d_bcefda54d79a3bedfa83231aed8d38b1>` extended :ref:`inline <i_ae1.1.215667b75409cb3251ba13c032>`
-
_i_ae1.1.215667b75409cb3251ba13c032:
5.6. Inline schema:
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
lastModified |
Yes |
string |
date-time |
the last modified timestamp of this pipeline (used to trigger changes). |
|
modificationReason |
No |
string |
Can be used to document latest changeset. |
||
services |
No |
array of :ref:`Component <d_3c2b4cd64485b5f73be7a1facba6ed8c>` |
The service controllers. |
||
streams |
No |
array of :ref:`Component <d_3c2b4cd64485b5f73be7a1facba6ed8c>` extended :ref:`inline <i_09545770fbf157c057309e15e402b2f4>` |
The engine properties. |
{
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"services": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
],
"streams": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring",
"pipeline": {
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"processors": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
]
}
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring",
"pipeline": {
"lastModified": "2015-01-01T15:00:00.000Z",
"modificationReason": "somestring",
"processors": [
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
},
{
"component": "somestring",
"config": [
{
"key": "somestring",
"type": "string",
"value": "somestring"
},
{
"key": "somestring",
"type": "string",
"value": "somestring"
}
],
"documentation": "somestring",
"name": "somestring"
}
]
}
}
]
}5.7. Responses
5.7.1. default
The server should return HTTP 200 OK. By the way, the response is ignored by Logisland since the operation has a fire and forget nature.
6. Data Structures
Here is the specification of all useful structures
6.1. Component
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
component |
Yes |
string |
|||
config |
No |
array of |
|||
documentation |
No |
string |
|||
name |
Yes |
string |
6.2. DataFlow
A streaming pipeline.
Inline schema:
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
lastModified |
Yes |
string |
date-time |
the last modified timestamp of this pipeline (used to trigger changes). |
|
modificationReason |
No |
string |
Can be used to document latest changeset. |
||
services |
No |
array of |
The service controllers. |
||
streams |
No |
array of |
The engine properties. |
-
_d_0752e439d11d3f0d4f6b437e63ea7248:
6.3. Pipeline
Tracks stream processing pipeline configuration
Inline schema:
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
lastModified |
Yes |
string |
date-time |
the last modified timestamp of this pipeline (used to trigger changes). |
|
modificationReason |
No |
string |
Can be used to document latest changeset. |
||
processors |
No |
array of :ref:`Component` |
6.4. Processor
A logisland 'processor'.
6.5. Property
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
key |
Yes |
string |
|||
type |
No |
string |
{'default': 'string'} |
||
value |
Yes |
string |
6.6. Service
A logisland 'controller service'.
6.7. Stream
Inline schema:
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
component |
Yes |
string |
|||
config |
No |
array |
|||
documentation |
No |
string |
|||
name |
Yes |
string |
|||
pipeline |
No |
6.8. Versioned
a versioned component
| Name | Required | Type | Format | Properties | Description |
|---|---|---|---|---|---|
lastModified |
Yes |
string |
date-time |
the last modified timestamp of this pipeline (used to trigger changes). |
|
modificationReason |
No |
string |
Can be used to document latest changeset. |
