The Points Database¶

The Points database is a core component of Normal which nearly every other component interacts with. It provides extremely efficient access to Points, and has a number of features optimized for IoT use cases including:

• Indexed attribute access, allowing clients to look up relevant points in different ways. ** Support for complex queries
• Storage and querying of of time-series data related to the points.
• Efficient replication APIs for data and metadata

Point Structure¶

A Point is simply a structure, identified by a UUID, which generally maps to what controllers would think of as a point -- a physical or virtual I/O value. Point generally are associated with time-series of scalar values and so the Points database has optimized support for this use case.

By default a Point has only a few fields:

• uuid: the globally-unique identifier for the point.
• name: a human-readable name for the point
• device_uuid: if set, the uuid of a point representing the device this point is attached to.
• point_type: either POINT, DEVICE, or VARIABLE.
• period: a duration specifying how often to record data for this point
• cov: settings related to Change-of-Value settings for this point
• latest_value: the most up-to-date value from this point, as added using AddData

In addition to these base fields, additional metadata for points is added by associating it with a Layer. Layers are groups of key-value attributes which can be indexed for efficient queries. For most protocols, the metadata is stored. in layers and not in the base point.

Layers¶

Layers provide a way to structure additional metadata about points. Many protocol drivers and components that create new points will create layers so that the metadata for those points will be indexed. Since the point database is based on RediSearch, attributes can be indexed as text, numeric, or tag values. Attributes which are not defined in a layer are still stored and retrieved; but can't be queried using GetPoints or GetDistinctAttrs.

The example of entity-relation diagram below shows some of the layers and fields which are included by default with Normal. The BACnet and Modbus layers are both populated from their respective protocol drivers, and contain attribute definitions which are specific to those protocols -- for instance, BACnet defines a Device ID, Object Id, and Instance number to define a BACnet object, which is mapped to a Point in Normal. Similarly, the Modbus layer indexes the register address and name defined by the Modbus connection profile.

erDiagram
    POINT || -- o| BACNET : ""
    POINT || -- o| MODBUS : "" 
    POINT || -- o| AUTOMATION: ""
    POINT || -- o| MODEL: ""
    POINT {
        uuid uuid
        uuid device_uuid
        text name
        PointType point_type
        duration  period
        bool cov

    }
    BACNET  {
        text prop_object_name
        text prop_description
        numeric device_id
        numeric object_type
        numeric instance
    }
    MODBUS {
        string device_address
        numeric register_address
        string register_name
        numeric data_type
    }
    MODEL {
        tag id
        text dis
        tag markers
        tag class
        tag equipRef
    }
    AUTOMATION {
        tag application_id
        tag hook_id
        tag group_key
        tag label
    }

Making Queries¶

The main query endpoints for the points database are GetPoints and GetPointsById; although since many other APIs also accept Query objects as a parameter, it is an important concept to learn.

A query object is a recursive definition of and's, or's, and not's, which allows the user to submit logical queries over attributes in the points database.

message Query {
    oneof {
        // filter based on attribute values
        FieldQuery field = 1;
        // filter by properties of a reference
        ReferenceQuery reference = 2;

        // logical combinations of subqueries
        repeated Query and = 3;
        repeated Query or = 4;
        Query not = 5;
    }
};

The field query refers to queries over attributes, and is how you can query attributes within layers. For instance, this query will return points whose device_id attribute is either 260000 or 260001; if you are using the BACnet simulator we provide, this will return some points.

import requests

NFURL = "http://localhost:8080"
requests.post(NFURL+"/api/v1/point/query", json={
    "structured_query": {
        "field": {
            "property": "device_id", 
            "numeric": {"min_value": 260000, "max_value": 260001}
        }
    }
  }).json()
# prints {'points': [], 'totalCount': '49'}

Time Series Data¶

Each Point has a scalar time series associated with it, accessed using the GetData and AddPointsData APIs. This timeseries is typically used to record values retrieved from sensors using one of our native protocol implementations, or using a driver. For replication into other systems, see the Data Adaptor section.

In order to add values to a point's series, you must know the uuids of the points you wish to add data to. Once obtained, you may add values using this call:

import requests
import datetime
requests.post(NFURL+"/api/v1/point/data", json={
     "uuid": "ff2a8df2-2e01-3397-8984-2cff8f1dc944", 
     "values": [{
         "real": 47, 
         "ts":datetime.datetime.now().astimezone().isoformat()
     }]
 }).

Example Queries¶

The SDK repository contains a number of example queries to help demonstrate different features of the query API.