From the Desktop to User Trials

Margherita Facca

FLAME’s Devops Pipeline

FLAME is enabling trials of innovative media applications with people in real-world locations. By creating and supporting a process to take ideas from the developer’s machine to user trials in a FLAME city or replica we have reduced the cost of testing new services and experiences with the public. This process is FLAME’s devops pipeline.

FLAME’s devops pipeline showing the activities at each stage.
Figure 1: FLAME’s devops pipeline showing the activities at each stage.

The pipeline (Figure 1) takes software from the development machine to deployment for user trials in a FLAME replica such as Bristol or Barcelona. Going from one end to the other increases the complexity, realism and cost, moving from TRL 4 (technology development) to 7 (technology demonstration). Each stage must be completed before moving on to the next, but it is expected that earlier stages will be revisited as lessons are learnt and ideas and concepts are revised and incorporated (Figure 2).

A development process will repeat earlier stages as lessons are incorporated. The TRL increases from the start of the pipeline to the end.
Figure 2: A development process will repeat earlier stages as lessons are incorporated. The TRL increases from the start of the pipeline to the end.

Decomposition of the Service and Initial Development

The initial development of a (media) service often involves the decomposition of an existing software service to create several independently operating service functions (SF) that are communicating over the network with each other ideally over HTTP and composed into a service function chain (SFC) to create the whole service. Once the decomposed service’s unit tests have passed it is packaged for FLAME using a packing toolchain provided by the project for the LXD and KVM hypervisors. As part of the packaging toolchain, Telegraf is installed which allows the collection of various performance data points of the operating system the service container and the service itself. The result of the packaging is a compressed TAR ball which can be uploaded into any FLAME-based platform.

Initial Integration using FLAME-in-a-Box

FLAME provides a virtual appliance, FLAME-in-a-Box, that fits on an ordinary modern office laptop. As illustrated in Figure 3, this appliance comprises a minimal set-up of virtual instances allowing experimenters to familiarise themselves with the TOSCA-based resource descriptor and test it:

  • a UE (user equipment node) for the test or client software;
  • a “cluster” where packaged service functions are deployed;
  • the “sr-ue” which is a service router connecting the ue, cluster and pce-sfemc;
  • the “pce-sfemc” node (path computation element and service function endpoint management and control services) which also includes the FLAME orchestrator;
  • another service router (“sr-ps”); and
  • a “ps” instance for platform services such as DHCP, IP gateway and DNS.

Additionally, the FLAME orchestrator node has been also equipped with a web-based TOSCA parser allowing experimenters to get some help in case they have written an invalid TOSCA descriptor.

Integration & Experimentation in the Sandpit

FLAME’s sandpit is a 72-core server hosted at IT Innovation and accessed remotely through SSH and web interfaces. The sandpit uses a combination of containers and virtual machines to emulate a deployment of the FLAME platform in a physical infrastructure such as that found in Bristol’s Millennium Square.

In contrast to FLAME-in-a-Box, the sandpit provides FLAME’s cross-layer management and control (CLMC) component and sufficient resources (memory, CPU) in four “clusters” for the deployed service functions to execute and be tested. Crucially, the sandpit also includes four “emulated UE” nodes which allow experimenters to install their test clients on the user-equipment (UE) nodes which can then connect to and test their service functions. Each UE is deployed in a different area of the sandpit to be closer to a particular cluster and thus vary their performance.

The sandpit also hosts FLAME’s interactive tutorial which guides a developer through creating a TOSCA resource specification and TOSCA alerts specification, deploying a simple service function chain, viewing its state in the orchestrator, viewing monitoring data in the CLMC, and using one of the UEs to apply load to the service and triggering a change in the deployment.

The topology of the sandpit with clusters (green), emulated UE (red) and SDN switches (blue).
Figure 4: The topology of the sandpit with clusters (green), emulated UE (red) and SDN switches (blue).

In the sandpit, the media service developer uploads their packaged service functions to the sandpit’s repository and adapts their service function chain (TOSCA resource specification) to make use of the four available clusters and, potentially, the foundation media services which are provided by FLAME and made available through the same image repository.

The key tasks in the sandpit are to test the functionality of the deployed service function chain (but not test its performance) and understand and explore the data collected in the CLMC. Data is collected at various levels: metrics from the SDN level describing routes and latencies; metrics from the container of each deployed service function (service function endpoint or SFE) describing the memory, disc, CPU and network; metrics describing the performance of application containers or web servers such as Tomcat or Nginx if they are used; and application-specific metrics where they are implemented. This data may be explored in the CLMC’s Chronograf interface and used to understand the behaviour of the service and to define triggers (encoded in the TOSCA alerts specification) which fire off alerts to the SFEMC or to the media service in order to adapt the service to demand. Note, while the TOSCA-based resource descriptor is given to the FLAME orchestrator, trigger alerts are sent to the SFEMC, as both are operating at different timescales.

Functional tests and experiments to understand a service’s response to demand may be conducted from the emulated UEs. A developer uses SSH to get a terminal on a UE and may install their client software or test suite there. It is even possible to forward an X-server connection from a UE to a developer’s desktop if a graphical interface is required.

Testing & Experimentation at a Replica

Although the sandpit provides a realistic technical environment, moving from the sandpit to a replica does introduce key changes. The performance (network, CPU) in a replica will be different to the sandpit which affects the performance and behaviour of the media service. The physical environment provides new challenges: the connectivity of mobile phones must be tested (both the signal strength, bandwidth and which access point is used in different locations) and the behaviour and positioning of any physical markers (e.g. QR codes, BLE beacons) must be experimented with.

Ideally, the first tests are done remotely so that the developer does not need to travel. A VPN connection to the replica can be used to deploy a service function chain and some tests can then be performed either by including an emulated UE as part of the chain or by a member of the FLAME team local to the deployment physically performing some tests using a mobile phone under the instruction of the developer. Data can be explored in the remote replica’s CLMC interface.

With these remote tests successfully completed, a developer (with the agreement of the host) would travel to the replica site to perform those tests and experiments which are dependent on the physical environment. This is also an opportunity to get an initial impression of the user experience in the environment.

User trials

The ultimate aim is to understand the user experience of a new media service and how FLAME’s innovative features serve to enhance that experience. This is achieved through the user trials. Before conducting user trials there are many administrative tasks to be completed such as participant recruitment and the preparation of data sharing agreements, participant information and consent sheets and so forth but, whilst important, they are not discussed further here.

Small trial

A “small trial” may be around 10 people on site over some short time and might not include all aspects of a media service (it could just focus on part of the experience).

Small scale FLAME trials serve to provide a number of important outcomes:

  • validation of the technical deployment and operation of your media service;
  • generate observations of real use that will drive usability improvements; and
  • capture of a selection of QoE data that can be analysed alongside QoS metrics.

If well-planned, small scale trials will generate critical data (and later, knowledge) that will significantly improve the quality of a FLAME based media service as well as removing or mitigating risks before scaling to a large trial.

A small trial should include time for the participants to reflect on the experience. At its most basic, participants should be asked to respond to a simple questionnaire. However, greater insights can be garnered by actively reviewing experiences with your participants – for example, by discussing particular observations you have made and asking further questions to clarify behaviours.

Feedback from a small trial may cause a developer to rethink the service and its experience and so it may be necessary to return to the sandpit to understand how to capture and respond to additional data or back to the first stage to adapt the service and repackage.

Large trial

A “large trial” would be around 50 people on site, free to use the service as they like. Primary outcomes for a FLAME large trial are:

  • verified, stable deployment and provision of your media service at scale;
  • demonstrable, dynamic management of your media service using FLAME platform behaviours; and
  • empirically supported knowledge supporting the benefits of the FLAME platform and the value of the media service.

The scale of a large trial causes differences in the approach and challenge in comparison to a small trial. For instance, with many people it is not possible to use qualitative analysis methods (such as individual interviews) to gather information about the users’ experiences. The scale of the gathered data set is dramatically increased and, combined with the less-controlled participant behaviour, this may become difficult to analyse.

Opportunities come from having a sufficiently large cohort to do effective A/B testing (if appropriate) and to gather a dataset which can provide significant statistics and evidence for the efficacy of the approach.

Conclusion

FLAME’s innovative networking, monitoring, management and foundation media services deployed in physical spaces across Europe provide a platform for the development and trailing of novel media services and experiences.

FLAME has developed and supports the devops pipeline described here to rapidly move new concepts from initial service development right the way through to an operational environment where members of the public can participate in trials. The tools and processes in this pipeline provide a clear path from idea to impact, reducing costs for the media service developer and gathering evidence throughout to show the value of both the new media service and FLAME itself.

Blog post by Stephen C Phillips (IT Innovation) and Sebastian Robitzsch (IDE)

Glossary

  • CLMC / Cross-Layer Management and Control: the FLAME platform service which collects monitoring data and provides alerts to other components to adapt the service.
  • FLIPS / Flexible IP Services: the underlying networking system used in FLAME
  • FMS / Foundation Media Services: a set of media services developed by FLAME for common tasks such as storage, transcoding, etc.
  • QoE / Quality of Experience: measures (quantitative and qualitative) describing the quality of the end-users’ experience.
  • QoS / Quality of Service: measures (primarily quantitative) describing the performance of the service.
  • SF / Service Function: services are decomposed into service functions; each service function is packaged separately.
  • SFC / Service Function Chain: a service function chain is the composition of service functions into a complete service.
  • SFE / Service Function Endpoint: a deployed instance of a service function.
  • SR / Service Router: the FLIPS instance at the edge translating the IP-based world (HTTP, SSL, TCP and any other IP-based protocol) into FLIPS and vice versa
  • TOSCA alerts specification: a configuration document describing what actions to take when data thresholds are reached.
  • TOSCA resource specification: a configuration document describing the service function chain, the states of the service functions and the resources required for them.
  • TRL / Technology Readiness Level: a scale from 1 to 9 indicating the maturity of a concept, from idea to deployment in an operational environment.
  • UE / User Equipment: the physical equipment such as a mobile phone that the user interacts through.