OSDB  OSDB Configuration Guide

Contents

See Also

Setup

After obtaining the OSDB sources

OSDB Server

Starting the Public OSDB Server

npm start shows the general form of the command for launching osdb_server.
To start osdb_server on https://osdb.openlinksw.com:

sudo su -
screen -S osdb_srv
cd osdb
./startup_linux_https_uriburner.sh
Ctrl-a d

If launching on another server, modify one of the config/*.json files and the corresponding startup_*.sh file to reflect your server hostname, port etc.

In General

To start the OSDB server, if not using one of the above mentioned startup_xxx scripts:

. ./osdb_env.sh
osdb_server -h # Shows usage help
osdb_server -r {service registry SPARQL endpoint}

e.g.

osdb_server -r https://linkeddata.uriburner.com/sparql

By default, the server starts on the port identified by the serverURL entry in config/default.json.

{
    "serverURL": "https://osdb.openlinksw.com/",
    "serverKey": "/etc/.../privkey.pem",
    "serverCertificate": "/etc/.../fullchain.pem"
    …
}

To startup using an alternative configuration, use the NODE_ENV environment variable.

. ./osdb_env.sh
NODE_ENV=localhost-http-dev osdb_server -r http://localhost:8896/sparql

config/localhost-http-dev.json

{
    "serverURL": "http://localhost:8000",
    // "serverKey": "config/server.key",
    // "serverCertificate": "config/server.crt"
}

Starting osdb_server on Linux

Because of issues with /bin/env on Linux, you must specify node explicitly.

cd /my/osdb
. ./osdb_env.sh
node bin/osdb_server -r https://linkeddata.uriburner.com/sparql

OSDB Client

A very simple client is provided to interact with the OSDB bot in addition to any other client bindings, for example to Facebook Messenger or Slack.

To start the OSDB client:

. ./osdb_env.sh
osdb_client -s https://osdb.openlinksw.com

To see the OSDB client command set:

osdb> help

exit does just that.

OSDB Registry

The OSDB Registry holds RDF descriptions of services and actions supported by the bot. OSDB should be capable of using any SPARQL supporting RDF database with little or no change. Both HTTP and HTTPS SPARQL endpoints are supported.

Using Virtuoso as a Registry

If Virtuoso is being used as a Registry Server, you may need to make the following grants before being able to load service descriptions into the Registry:

grant execute on DB.DBA.SPARUL_LOAD to "SPARQL";
grant execute on DB.DBA.SPARUL_LOAD to SPARQL_UPDATE;
grant execute on DB.DBA.SPARUL_CLEAR to "SPARQL";
grant execute on DB.DBA.SPARUL_CLEAR to SPARQL_UPDATE;

then grant role SPARQL_UPDATE to user SPARQL.

Using an HTTPS-Protected Registry

The above grants assume the OSDB Server is using a publicly accessible HTTP SPARQL endpoint. No further configuration should be needed for an HTTP endpoint. For details on how to use an HTTPS SPARQL endpoint, please refer to OSDB Configuration Guide - HTTPS Registries

Importing Service Descriptions

When first installed, the OSDB Service Registry will be empty. Backend services that the OSDB server is to interact with must be described to the server by loading their descriptions into the Registry. These service descriptions take the form of RDF imported by the server through load or load as commands.

load {service_description_url}
Loads the RDF service description identified by the given URL

load {service_description_url} as {service_id}
Loads the RDF service description identified by the given URL using service_id as the service name.

In order for a service to be usable by OSDB, the RDF service description must include specific classes and properties from the schema.org or OpenLink Web Services ontologies.

Named Services

Services can either be named or anonymous. Named services are imported using the command:
load {service_description_url}

A named service includes a schema:WebApplication instance with these properties:

@prefix schema: <http://schema.org/> . 

<#FacetedBrowsingService>
  a schema:WebApplication ;
  schema:name "facet" ;
  [ schema:description "Faceted Browsing Service" ; ]
  schema:potentialAction
    <#FacetedBrowsingSearchAction> ;

schema:name provides the service’s ID. This is not an ID in the conventional RDF sense (i.e. not an entity URI), but a one word moniker used to identify the service in the OSDB CLI. schema:description provides an optional summary of the service, as shown in this CLI output below.

osdb> services
OSDB services:

Service ID: facet
   Summary: Faceted Browsing Service

osdb>

Alternatives to schema:name for specifying the service ID are: schema:identifier, skos:prefLabel, skos:altLabel, schema:alternateName, and rdfs:label. The property order of preference when selecting the property providing the service ID is: schema:name, schema:identifier, schema:alternateName, rdfs:label, skos:prefLabel, skos:altLabel.
If a service ID isn’t provided, OSDB will generate one from the service summary if present; so the service ID or the service summary must be present. Other properties which can be used for the service summary are: dcterms:description and rdfs:comment.

Named services specify the actions they support through schema:potentialAction properties which point to schema:Actions describing the service’s actions.

Anonymous Services

An anonymous service does not include a schema:WebApplication in its description. It comprises one or more schema:Action instances.

Anonymous services are imported using:
load {service_description_url} as {service_id}

The supplied service ID should be a single word moniker. No summary is shown in the CLI for an anonymous service.

If load as is used to import a named service, the given service ID overrides any service name supplied in the service description.

Describing Service Actions

Actions

The minimum description of an action is:

<action_url>
  a schema:Action ;
  schema:target {entry_point} ; 
  [ schema:description {action_summary} ; ]
  oplwebsrv:hasActionWord {action_id} ;
  ...

The action ID must be a single word, unique across all actions supported by a service. schema:description supplies a summary of the action for display in the CLI. If an action ID is not supplied, OSDB derives one from the action summary if available; so one or other must be present.

In addition to oplwebsrv:hasActionWord, other properties supported for specifying the action ID are, in order of preference: schema:name, schema:identifier, schema:alternateName, rdfs:label, skos:altLabel and skos:prefLabel. Alternative properties for the action summary are: dcterms:description and rdfs:comment.

This RDF snippet:

<#FacetedBrowsingSearchAction>
  a schema:Action, schema:SearchAction ;
  schema:description "Search — Faceted Browsing Action" ;
  oplwebsrv:hasActionWord "search" ;
  schema:target <#FacetedBrowsingSearchEntryPoint> ;

generates the CLI output below:

osdb> facet actions
facet actions:

Action ID: search
   Action: Search — Faceted Browsing Action

osdb>

An action description omitting oplwebsrv:hasActionWord (or equivalent property for setting the action ID):

ns4:FacetedBrowsingAction
  rdf:type	schema:Action , schema:SearchAction ;
  schema:name	"Search - Faceted Browsing Action" ;
  schema:target	ns1:this ;

produces:

osdb> anonymous_facet actions
anonymous_facet actions:

Action ID: search_faceted_browsing_action
   Action: Search — Faceted Browsing Action

osdb>

EntryPoints

Each Action has an associated EntryPoint, specified through a schema:target property. The EntryPoint describes an API endpoint.

<#FacetedBrowsingSearchEntryPoint>
  a schema:EntryPoint ;
  schema:name "Search — Faceted Browsing Search Endpoint" ;
  schema:contentType "text/html" ;
  schema:contentType "application/rdf+xml" ;
  schema:contentType "text/turtle" ;
  schema:contentType "application/json" ;
  schema:httpMethod "GET" ;
  schema:urlTemplate "http://localhost:8896/fct/search(?q,view:type,same-as,inference,offset,limit)" ; 
  schema:url <http://localhost:8896/fct/search>;
  oplwebsrv:hasParameter <#FacetedBrowsingSearchActionParameterQ> ;
  oplwebsrv:hasParameter <#FacetedBrowsingSearchActionParameterViewType> ;
  oplwebsrv:hasParameter <#FacetedBrowsingSearchActionParameterSameAs> ;
  oplwebsrv:hasParameter <#FacetedBrowsingSearchActionParameterOffset> ;
  oplwebsrv:hasParameter <#FacetedBrowsingSearchActionParameterLimit> ;

schema:httpMethod and schema:contentType are mandatory properties, as is oplwebsrv:hasParameter if the API has any parameters. In the absence of any parameters, or for requests using only header parameters , the request URL is defined by the EntryPoint schema:url property. If the EntryPoint uses URL path parameters, schema:urlTemplate must be specified, in place of schema:url. Requests using query string parameters, but without any URL path parameters, can specify either a schema:url or a schema:urlTemplate. When an Action using query string parameters is invoked, the constructed query string will be appended to either the schema:url or the expanded schema:urlTemplate depending on whether URL path parameters are present.

URL Templates

Three variants of URL template (as defined by RFC 6570) are supported: simple string expansion, form-style query expansion and path segment expansion.

Note that OSDB can use ‘()’ or ‘{}’ as template expression markers. If using nanotation to describe a service for import into OSDB, use ‘()’ for template expressions because nanotation interprets ‘{}’ as nanotation delimiters.

Examples

Simple string expansion - {var} or (var)

Form-style query expansion - {?var} or (?var)

Path segment expansion - {/var} or (/var)

EntryPoint Parameters

Each parameter of an API endpoint must be defined as an oplwebsrv:WebServiceParameter. The OpenLink Web Services ontology is used in preference to schema.org as schema.org currently lacks properties for adequately describing parameters.

<#FacetedBrowsingSearchActionParameterQ>
  a oplwebsrv:WebServiceParameter ;
  schema:name "Faceted Browsing Search Action Parameter 'q'" ;
  schema:description """
   A word or phrase to be used for the Facet free text search. 
   """;
  oplwebsrv:shortDescription "A word or phrase to be used for the Facet free text search. Multiple words should be delimited by '+'"  ;
  oplwebsrv:usageExample "http://localhost:8896/fct/search?q=linked+data" ;
  oplwebsrv:parameterName "q" ; 
  oplwebsrv:parameterExampleValue "linked+data" ;
  oplwebsrv:isRequired true ;

At a minimum, oplwebsrv:parameterName must be supplied. The Action Console uses schema:name as a parameter’s display name in the input form, or uses oplwebsrv:parameterName if schema:name is not present.
oplwebsrv:shortDescription provides a short description of the parameter in the CLI. If not supplied, one of schema:description, ’schema:name or oplwebsrv:parameterName provides the description (in that order of preference). oplwebsrv:isRequired indicates if the parameter must be supplied in the API call. If omitted, the parameter is assumed to be optional. oplwebsrv:parameterExampleValue will populate a parameter’s entry field in the Action Console with the example value. It is useful for providing ‘ready to go’ examples of actions and indicating the type of value expected for a parameter. schema:description also provides the text for a tooltip, displayed when a user hovers over the ‘?’ icon adjacent to the parameter display name in the Action Console. It can be used to display a hint to the user about the nature of the parameter.

In the OSDB CLI, the parameter descriptions are used to build a list of action options (switches).

osdb> facet search -h

Action options:

-o, --offset [offset]
    Sets the result set page size for scrolling
    Required: no
-l, --limit [limit]
    Limits the number of result set entities returned
    Required: no
-v, --view_type [view_type]
    Sets the default view orientation
    Required: no
    Permitted values: entity-types, attribute-names
-s, --same_as [same_as]
    Enables owl:sameAs inferencing for facet search
    Required: no
    Permitted values: yes
-q, --q <q>
    A word or phrase to be used for the Facet free text search. 
    Multiple words should be delimited by '+'
    Required: yes
-u, --url_only [url_only]
    Return only the action URL, not the action results
    Required: no
-r, --response_format [response_format]
    Sets the response media type
    Required: no
    Permitted values: application/json, text/html, application/rdf+xml, text/turtle
-t, --quality_values [quality_values]
    Quality value string. e.g. application/json;q=1.0,text/turtle;q=0.8,*/*;q=0.5. 
    --response_format takes precedence.
    Required: no

osdb>
Restricted Parameter Values

Some parameters may accept only a limited set of values. Such values are enumerated using the optional oplwebsrv:parameterValue property.

<#FacetedBrowsingSearchActionParameterViewType>
  a oplwebsrv:WebServiceParameter ;
  schema:name "Faceted Browsing Search Action Parameter 'view:type'" ;
  schema:description """
   Sets the default view orientation of the initial page returned 
   by the Faceted Browsing Service. 
   """;
  oplwebsrv:shortDescription "Sets the default view orientation" ;
  oplwebsrv:parameterName "view:type" ; 
  oplwebsrv:isRequired false ;
  oplwebsrv:parameterValue "entity-types" , "attribute-names" ;

When present, the CLI will display this list of permitted values and reject an action command containing any invalid parameter values.

Parameter Types

A parameter is one of three types: a query string parameter, a header value or a path parameter. In the service description, the parameter type is specified using the oplwebsrv:parameterType property with a value of "query", "header" or "path". If oplwebsrv:parameterType is omitted, a parameter is assumed to be a query string parameter.

Query String Parameters

A query string parameter adds a name=value pair to the query string of the HTTP request URL.

Header Parameters

A header parameter supplies a value for an HTTP request header.
e.g.

<#CsvTransformParamBaseUri>
  a oplwebsrv:WebServiceParameter ;
  oplwebsrv:parameterName "Content-Location" ; 
  oplwebsrv:parameterType "header" ;
  schema:name "Base URI" ;
  oplwebsrv:shortDescription """
    Base URI to be used as the stem for all generated entity URIs.
    Typically it is set to match the URI of the document providing the CSV data.
    """ ;
  oplwebsrv:isRequired true ;
  wdrs:describedby <#this> .
Path Parameters

Path parameters are used to describe parameterized URL paths, typically seen in RESTful APIs, where some path segments may vary depending of the resource being accessed. A URL template must be defined for path parameters. e.g.

<#LdpRead_EntryPoint>
  a schema:EntryPoint ;
  schema:name "LDP Read - Lookup a container or resource" ;
  schema:description """Retrieve a resource description""" ;
  schema:contentType "text/turtle" ;
  schema:contentType "application/ld+json" ;
  schema:httpMethod "GET" ;
  schema:urlTemplate "http://example.openlinksw.com(/ldprPath)" ; 
  oplwebsrv:isEntryPointOf <#LdpRead> ;
  oplwebsrv:hasParameter 
    <#LdpParamResourcePath> ;
  oplwebsrv:usageExample 
    """curl -iH 'Accept: text/turtle, */*;q=0.1' -u dav:dav http://example.openlinksw.com/DAV/home/fusepool/ldp/"""" ; 
  oplwebsrv:securityScheme <#LdpBasicSecurityScheme> ;
  wdrs:describedby <#this> . 

<#LdpParamResourcePath>
  a oplwebsrv:WebServiceParameter ;
  oplwebsrv:parameterName "ldprPath" ; 
  schema:name "Resource Path" ;
  schema:description "The absolute path to the LDPR or LDPC to look up" ;
  oplwebsrv:parameterType "path" ;
  oplwebsrv:isRequired true ;
  wdrs:describedby <#this> .

Hostnames can be parameterized using a combination of a “path” parameterType and simple string expansion.

<#SPARULPostUpdateEntryPoint>
  a schema:EntryPoint ;
  schema:httpMethod "POST" ;
  schema:urlTemplate "https://(host)/sparql(?default-graph-uri,format,should-sponge,timeout)" ;
  oplwebsrv:hasParameter  
    <#SPARULQueryServiceParameterHost> , 
    <#SPARULQueryServiceParameterDefaultGraph> , 
    ...
  oplwebsrv:usageExample 
    "curl -iL \"https://linkeddata.uriburner.com/sparql?default-graph-uri=&should-sponge=&format=text%2Fturtle&timeout=30000000\" --data-urlencode \"query=with <urn:idehen:sparul:qa:> insert { <#this> <#relatedTo> <#that> . }\" " ;
  wdrs:describedby <#this> .

<#SPARULQueryServiceParameterHost>
  a oplwebsrv:WebServiceParameter ;
  schema:description "The hostname of the SPARUL endpoint" ;
  schema:name "Host" ;
  oplwebsrv:parameterName "host" ;
  oplwebsrv:parameterType "path" ;
  oplwebsrv:isRequired "true"^^xsd:boolean ;
  oplwebsrv:parameterExampleValue "linkeddata.uriburner.com" ;
  wdrs:describedby <#this> .
Parameter Data Type and Default Value

Optional properties oplwebsrv:parameterDataType and oplwebsrv:parameterValueDefault provide additional guidance to users when entering parameter values through the OSDB Action Console UI.

  <rdf:Description rdf:about="http://linkeddata.uriburner.com/about/id/entity/...">
    <rdf:type rdf:resource="http://www.openlinksw.com/ontology/webservices#WebServiceParameter" />
    <schema:name>wfd</schema:name>
    <schema:description>
    Perform Word Frame Disambiguation on input terms in order to provide alignments to WordNet
    synsets, WordNet Super-senses and Dolce classes. By default it is set to false.
    </schema:description>
    <oplwebservices:isRequired
     rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oplwebservices:isRequired>
    <oplwebservices:parameterName>wfd</oplwebservices:parameterName>
    <oplwebservices:parameterValue>true</oplwebservices:parameterValue>
    <oplwebservices:parameterValue>false</oplwebservices:parameterValue>
    <oplwebservices:parameterType>query</oplwebservices:parameterType>
    <oplwebservices:parameterDataType>boolean</oplwebservices:parameterDataType>
  </rdf:Description>
  <rdf:Description rdf:about="http://linkeddata.uriburner.com/about/id/entity/...">
    <rdf:type rdf:resource="http://www.openlinksw.com/ontology/webservices#WebServiceParameter" />
    <schema:name>wfd_profile</schema:name>
    <schema:description>
    The profile associated with the Word Frame Disambiguation
    </schema:description>
    <oplwebservices:isRequired 
     rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oplwebservices:isRequired>
    <oplwebservices:parameterValueDefault>b</oplwebservices:parameterValueDefault>
    <oplwebservices:parameterName>wfd_profile</oplwebservices:parameterName>
    <oplwebservices:parameterValue>b</oplwebservices:parameterValue>
    <oplwebservices:parameterValue>d</oplwebservices:parameterValue>
    <oplwebservices:parameterValue>t</oplwebservices:parameterValue>
    <oplwebservices:parameterType>query</oplwebservices:parameterType>
    <oplwebservices:parameterDataType>string</oplwebservices:parameterDataType>
  </rdf:Description>

oplwebsrv:parameterValueDefault declares the value of the parameter that the target service will use if none is provided. For optional parameters whose values are constrained to a fixed set, the OSDB Action Console displays a select list of permitted values with an additional option ‘use default setting’. oplwebsrv:parameterValueDefault causes the default value to be displayed adjacent to the ‘use default setting’ option.

action console param default val

oplwebsrv:parameterDataType gives an indication of the parameter’s data type in the tooltip adjacent to the parameter label. The Action Console does not act on the data type, but a data type of, say, ‘integer’ in the tooltip gives a user some indication of the type of value expected by the target service.

Suggested values for `oplwebsrv:parameterDataType’ include: ‘string’, ‘boolean’, ‘binary’, ‘numeric’, ‘integer’, ‘float’, ‘date’ etc. A data type prefixed by ‘long’, e.g. ‘long binary’ or ‘long string’, causes the Action Console to display a multi-line textarea, rather than the default single-line text field, for inputting action data.

action console param tooltip

In this example, the allowed boolean values were added the select list as a consequence of oplwebsrv:parameterValue properties, not by the presence of <oplwebservices:parameterDataType>boolean</oplwebservices:parameterDataType>

action console param data type

Default Action Options

Some options are available to every action. They are provided by the OSDB server and need not be included in a service’s RDF description. These options are:

See the CLI output above for the details of these options. --url_only is only supported for actions using HTTP GET, not HTTP POST. If neither --response_format nor --quality_values is supplied, the target service chooses the default output format, assuming --url_only was not used.

Integration with the Virtuoso Sponger Middleware

The Virtuoso Sponger is an extract, load and transform middleware layer for generating RDF-based Linked Data from a variety of HTTP-accessible document types. Architecturally, it includes an Action Sniffer metacartridge. This metacartridge enables the Sponger to extract structured metadata embedded in HTML documents using terms from the schema.org vocabulary that describe Actions. This metacartridge augments what it extracts with additional relations that identify and describe web applications, entry points and parameters as required by a service registered in OSDB.

URL Patterns

If the original web resource being sponged is http://site/path/ and your
Virtuoso instance is at http://virt:port/ then the Sponger can be invoked
at http://virt:port/about/html/http://site/path/&@Lookup@=&refresh=clean
(where the parameters force cleaning the graph and re-running the sponger).

OSDB requires an RDF view of the sponged graph, from which to extract
service definitions. The Sponger provides two URL patterns by which this is
exposed:

URL Pattern 1: /about/rdf/

The “html” fragment in the previous URL is only one of several output
formats the Sponger supports; in particular it can use content-negotiation
to answer requests for RDF in various serializations such as Turtle used by
OSDB.

Therefore, within OSDB, it suffices to run a command such as

osdb> load http://virt:port/about/rdf/http://site/path/

and that will request a Turtle rendition of the data from the Sponger for import as a service description.

URL Pattern 2: /about/data/entity/

The Sponger’s output at
http://virt:port/about/html/http://site/path/?@Lookup@=&refresh=clean has
a header About: and the title of the resource sponged. This title is a
link, characterised by the pattern /about/id/entity; it suffices to run a
command such as

osdb> load http://virt:port/about/data/entity/turtle/proto/host/urlpart

which will also request a Turtle serialization of the data from the Sponger.

URL Pattern 3: /about/id/entity/

Finally this third variant
osdb> load http://virt:port/about/id/entity/proto/host/urlpart
can also be used to load a Sponger generated service description.

Example: Registering PressThink as an OSDB Service via the Sponger

http://pressthink.org/ includes a schema:searchAction embedded as a structured data island in a <script type=“application/ld+json”> element. This can be revealed easily using the OpenLink Structured Data Sniffer (OSDS), as shown in the screenshot below.

OSDS-extracted structured metadata from pressthink.org

Loading the Service Description using the OSDB Client CLI

Loading this page using the Sponger’s import service (at the /about endpoint) using the URI pattern http://linkeddata.uriburner.com/about/rdf/http/pressthink.org/
results in the production of an “Action description document” that can then be loaded into the OSDB Service Registry.

osdb> load http://linkeddata.uriburner/about/rdf/http/pressthink.org
Loaded service description

The PressThink search action can then be exercised through the OSDB Action Console:

pressthink search action viewed in OSDB Action Console

producing:

pressthink search action results

Loading the Service Description using the OSDB REST API

Service descriptions can be loaded by different routes. The same Sponger-generated service description can be loaded through the OSDB REST API:

$ curl -k -X POST https://osdb.openlinksw.com/osdb/api/v1/services -d '{"service_description_url":"http://linkeddata.uriburner.com/about/rdf/http/pressthink.org"}' -H 'Content-Type: application/json'
{"status":"success","method":"load_service","api":"/osdb/api/v1/services","response":"Loaded service description"}

Loading the Service Description using Facebook Messenger

OSDB includes a binding to Facebook Messenger. The OpenLink Linked Data Bot app exposes the OSDB CLI in the Messenger UI. Messenger users can then interact with OSDB, and load services, as depicted below.

pressthink service loading through Facebook Messenger